✨ Move the exports to the queue worker

The exporter becomes a durable-job worker: the shared item and params
contract, the creation command (mixed scope rejected, single-use runs
marked transient), the external job-def, and the worker that consumes
the `:exporter` queue with one management client, serialized beats,
multipart settle and share-aware asset fetches.

The server side it needs lands here too: per-tenant `exporter:<tenant>`
key ids derived from the same secret (the bare legacy id keeps serving
the deployment own tenant), a heartbeat that filters by tenant in its
own write, and the management API with session mint and settle scoped to
the job owner. The artifact upload is multipart-only and confined to the
parser temp areas.

The frontend follows the job on the websocket: milestones, artifact
download and cancel by job id, with the payload validated against the
contract by a test per caller. nginx, devenv and the docs drop the old
exporter endpoint.

AI-assisted-by: deepseek-v4.1-flash
This commit is contained in:
Andrey Antukh 2026-11-04 17:20:22 +00:00
parent 0bba8b1425
commit 676701f74c
139 changed files with 10153 additions and 5064 deletions

3
.gitignore vendored
View File

@ -62,7 +62,7 @@ opencode.json
/exporter/target
/exporter/.shadow-cljs
/exporter/resources/wasm/
/exporter/src/app/wasm/shared.js
/exporter/src/exporter/wasm/shared.js
/frontend/.storybook/preview-body.html
/frontend/.storybook/preview-head.html
/frontend/playwright-report/
@ -99,7 +99,6 @@ opencode.json
# render-wasm build artifacts copied into the exporter (autogenerated)
/exporter/resources/wasm/render-wasm.js
/exporter/resources/wasm/render-wasm.wasm
/exporter/src/app/wasm/shared.js
/media-processor/dist/
/media-processor/target/

View File

@ -70,11 +70,54 @@ Every background job is a job-def: a plain `(defn execute-X [cfg params] ...)` i
- The domain cfg of a binfile request lives in `app.binfile.common` (`export-cfg`, `import-cfg`), so the legacy RPC and the jobs freeze the same ids, resolve the same team features and apply the same anti-zip-bomb limits. `resolve-export-type` is the translation the legacy RPC needs for its deprecated boolean flags; the creation command asks the caller for the type instead of guessing a default.
- Two retention knobs, easy to confuse: `:jobs-retention` sweeps terminal internal rows (the ones without a profile), while `:jobs-user-ttl` is the default expiry a user-facing job gets when it is created; the jobs GC deletes the row once `expires_at` passes and touches its resource. The heavy work of a user job runs on its own `:binfile` queue (`:worker-binfile-parallelism`, default 1) so it does not block the default runner.
- The quote of a family of jobs (`::export-jobs-per-profile`, `::import-jobs-per-profile`) counts the jobs of the profile that are still on their way (`new`/`scheduled`/`running`/`retry`) whose name the registry gives for that family, so a new job of the family is covered without touching the quote.
- A user-facing job is created through `app.rpc.commands.jobs`. One command per job type: `create-export-binfile-job` and `create-import-binfile-job` name the job they create and their body carries only `:params`, whose schema the command imports from the job namespace (`::sm/params`), so the RPC validates the business params up front, before the command runs; the job-def is still the place that owns them and validates them again at run time. The commands check the permission and the quote of the family before the expensive part, submit with `max-retries 0`, an expiry from `:jobs-user-ttl` and the queue the job-def routes to through `::jobs/queue-name` (`:binfile` when the def says nothing; read by the command, never by the substrate), and answer with a summary of the row. The import command has its own creation schema (`import-binfile/schema:create-params`, version optional) because the request is not the stored params: the command fills the version (from the caller or the package header) and the manifest metadata kept for audit. It assembles the upload, reads the manifest of a version 3 package for the audit (never its content) and stores the package in `job-resource` as the resource of the job, releasing it when the job never comes to own it.
- The `:exporter` queue has no runner in the backend: its consumer is the exporter process (a separate Node worker; see `mem:exporter/core`). The backend serves it through `app.rpc.management.jobs`, shared-key authenticated (`X-Shared-Key: exporter[:<tenant>] <key>`, the key the exporter derives from the same secret with a per-tenant salt; the bare `exporter` id is the legacy form): `claim-job {job-id scheduled-at}` (exact `scheduled-at` match; a stale payload answers `{:action :skip :status ...}`; the row's params come back decoded with the job-def, so the worker gets typed values — a row whose def is gone answers the stored shapes), `report-job-progress {job-id progress {stage counters}}` (the worker owns the milestone words; the substrate validates the shape), `create-job-session {job-id}` (a real `http_session_v2` of the owner profile, so the assets fetch as the user; TTL `:job-session-ttl` default 1h), `complete-job` (the params form carries `result`/`resource-id`; the artifact form carries a multipart `content` part, the single source of `filename`/`mtype`, declared with `::rpc/multipart-only-params` so a transit body cannot name a server path — path confined by `media.v/check-upload-path`, allowlist `common.media/export-artifact-types`, cap `:exporter-max-result-size`; the checks run before any bytes are stored and a lost race stores nothing) and `fail-job {job-id error session-id}`. Every method answers `{:action :skip :status ...}` on a row it cannot write and reports the row status, and a settle named with `session-id` closes that render session whatever the write says — only when the session belongs to the job's `profile-id`. Every method also checks the row belongs to the calling exporter's tenant (suffix of a per-tenant key id, own tenant for the bare id): anything else answers a bare `{:action :skip}` (no status, no write, no session close), so no oracle leaks tenant rows. `report-job-progress` carries the tenant in the heartbeat write itself (`:tenant` on `jobs/heartbeat`, filtered in the touch and progress statements), so the hot path needs no row read; a filter that matches nothing raises `:interrupt` with `:reason :foreign-tenant`. The tenant is normalized to lowercase when the configuration is read (`PENPOT_TENANT=ACME` behaves as `acme`), and `app.setup.keys/derive-exporter-key` is the single definition of the exporter key on the backend (registry and middleware both call it).
- A user-facing job is created through `app.rpc.commands.jobs`. One command per job type: `create-export-binfile-job`, `create-export-assets-job` and `create-import-binfile-job` name the job they create and their body carries only `:params`, whose schema the command imports from the job namespace (`::sm/params`), so the RPC validates the business params up front, before the command runs; the job-def is still the place that owns them and validates them again at run time. The commands check the permission and the quote of the family before the expensive part — binfile checks the files it reads by id, and assets checks every item on its own: an item carrying `:share-id` validates through the share-link permissions the same way `get-file-fragment` does (a logged viewer of the share is enough; the command stays authenticated, so the anonymous visitor is refused, the same as the exporter cookie it replaces) —, submit with `max-retries 0`, an expiry from `:jobs-user-ttl` (or ten minutes when the creation says `:transient`, the single-use exports of the clipboard and the Plugin API) and the queue the job-def routes to through `::jobs/queue-name` (`:binfile` when the def says nothing; read by the command, never by the substrate), and answer with a summary of the row. Assets params carry an optional `:kind` (`:shapes`/`:frames`, declared by the frames dialog; absent means shapes): the worker routes on it instead of inferring the export from the items. The import command has its own creation schema (`import-binfile/schema:create-params`, version optional) because the request is not the stored params: the command fills the version (from the caller or the package header) and the manifest metadata kept for audit. It assembles the upload, reads the manifest of a version 3 package for the audit (never its content) and stores the package in `job-resource` as the resource of the job, releasing it when the job never comes to own it.
- `get-job` is the read surface a profile has over its own jobs: it answers with the same summary the creation commands return plus the `result` of a finished job or its public error (through `decode-job-error`, so the keywords come back), and a job of another profile is not distinguishable from one that does not exist: both answer `not-found`. `decode-row` decodes `params`, `result` and `error`, and the result is then decoded with the decoder the substrate compiled from the result schema the job-def the row names declares (`decode-result`; a def without one, or a row whose job-def is gone, keeps the plain JSON the row holds: a uuid as the text it is on disk, a uuid used as a map key as a keyword). The event log still has no read surface: a client that follows a job takes its progress from the websocket.
- `heartbeat` uses named options `[cfg & {:keys [job-id progress] :as options}]` and returns the number of durable writes, so 0 means there was nothing to write: no job context, or the throttle did not allow it. A write that finds the job no longer active (terminal or gone) raises an `:interrupt` (`code :job-interrupted`, carrying the job id and the status it found) through the shared `jobs/check-active` instead of counting zero, so a task in course stops at its next beat; the interrupt is internal and never reaches HTTP (the management progress report answers `:skip` for it). The job id falls back to `::jobs/job-id` on the cfg and then to the runner-bound `*job-id*`. The `modified_at` touch is throttled to ~1s and the progress event to ~200ms, so the check fires at most once per second with no extra query (the read of the row happens only when a write already found nothing). A progress report is a milestone (`schema:progress`): a required `:stage` keyword plus an optional `:counters` map of `scope -> {:current, :total?}`, and no other key. Counters are non-negative, a `total` is positive and never lower than its `current`, and stage and counter keys are names of at most 64 characters. A milestone without counters is valid (a point of the run with no units to count), the stage vocabulary belongs to the worker and not to the substrate, and a client must degrade for a stage it does not know. Read a stored payload back with `jobs/decode-progress`: the database holds the keywords as strings. The 1s beat (down from 60s, so a cancelled heavy job is seen as gone at its next beat) costs one `modified_at` touch per beating job per second: cheap while the only heavy beater runs alone on its queue, revisit if concurrent beating jobs ever grow.
- `job.error` conforms to `app.jobs/schema:job-error` (`type` and `code` keywords, `hint` text, extra details allowed). Read it back with `jobs/decode-job-error`; a plain `db/decode-json-pgobject` leaves the keywords as strings. `complete` takes named options and an optional `resource-id`, which is only ever set, never replaced: it reads the row locked inside its own transaction and applies the rules in a fixed order, skipping a job that is no longer completable before refusing one for its resource, so the answer is a function of the row and not of the order two callers arrive in. It checks the result against the schema the job-def of the row declares before it writes (a violation raises `:invalid-result` and nothing is stored), and only for a transition that is going to write: a terminal job answers `:skip` without validating. `complete` is the single door of a result, so the check covers the runner and the external worker that completes a job through the management RPC (`::complete-job`), which stores a result the substrate would otherwise take blind.
## The wake-up contract of a worker queue (public surface)
What an out-of-JVM process needs to learn about work WITHOUT reading the
dispatcher's guts. Verified against `app.worker/queue-key`,
`encode-payload` (`app.worker.dispatcher`) and the runner's local
`decode-payload`; a change there is a change of this contract.
- The key: `penpot.worker.queue:<tenant>:<queue>` (`app.worker/queue-key`).
`job.queue` stores the bare name, so the prefix is composed at use. The
tenant prefix keeps instances sharing one Redis from handing each other
jobs.
- The payload: a JSON array of exactly two items, `["<job-id>",
"<scheduled-at>"]`, the instant formatted by `ct/format-inst`. Fewer
than two is not a payload; the runner's decoder reads exactly those
two and drops and logs anything it cannot parse as uuid + instant.
- Dispatcher order: the rows are marked `scheduled` FIRST, pushed LAST
(RPUSH). A crash between the two leaves `scheduled` rows that
`reschedule-lost-jobs` re-queues; the reverse order would leave a
duplicate payload nothing deduplicates.
- The `scheduled-at` in the payload must be handed back AS-IS on `claim-job`:
the claim demands an exact match and answers `{:action :skip :status ...}`
for a stale payload. That is what makes a claim idempotent and what turns
the queue into a mere notification: a pop that is not confirmed by a claim
is not work.
- The queue is not the truth: the management API is (`claim-job`,
`report-job-progress`, `complete-job`, `fail-job` in
`app/rpc/management/jobs.clj`), shared-key authenticated by the route
resolver. Everything a consumer decides about a job goes through it.
- The queue has NO owner: anyone with Redis access can BLPOP it. Operational
rule that follows: do NOT wire a JVM runner for a queue an external
process consumes — the dispatcher hands each payload to whoever pops, so
both would start taking the same jobs.
- Cancellation travels the beats: `report-job-progress` answers
`{:action :skip}` once the row stopped being active — the only signal that
distinguishes "cancelled" from "network down". Same mechanism as the
internal `heartbeat` `:interrupt` (see the heartbeat note above); do not
build a second channel.
- Not part of this contract: `app.jobs/request`/`reply` (the ephemeral lane;
in retirement, `P1/F5/T16`). A comment in `app/jobs.clj` (2026-09-17) saying
that lane "waits for a producer" predates that decision and must not guide
anything here.
For worker dispatch, cron, retry semantics (`ex/raise :type ::wrk/retry` with `:delay`/`:strategy`), deduplication, job events, and queue internals: `mem:backend/subtleties`.
## Jobs metrics

View File

@ -1,9 +1,78 @@
# Promesa
# Promesa (funcool, 12.0.1) — verified API shapes
The promise lib of the monorepo: JVM (backend, `promesa.exec` thread
pools) and CLJS (frontend, exporter). Declared in `common/deps.edn`;
heavy users: `frontend/src/app/main/**` (Rx interop), `exporter/src/app/**`
(management client, auth), `backend/src/app/storage/s3.clj` and
`worker/cron.clj` (JVM await paths). Doc: `doc/promises.md` upstream.
## Two API families (verified against the 12.0.1 jar)
Argument order is inconsistent BY DESIGN; each op belongs to one of
two families:
**promise-first `([p f ...])` — pairs with `->` threading** (the
promise arrives as the threaded first argument):
- `then`, `then'`, `bind`, `handle`, `finally`, `catch` (`([p f])`,
2-arity with predicate: `([p pred-or-type f])`), `chain`
`([p f & fs])`.
**fn-first `([f p ...])` — pairs with `->>` threading** (the promise
arrives as the threaded LAST argument):
- `map`/`fmap`, `mapcat`/`mcat`, `hmap`/`hcat`, `fnly`, `merr`.
Every 3-arity in a family inserts its executor FIRST for the fn-first
family (`([executor f p])`) and LAST for the promise-first family
(`([p f executor])`).
**What `f` must return:**
- `then`/`catch`/`fmap`/`hmap`: plain value OK — auto unwrapping.
- the bind-style ops `mcat` and `merr`: **`f` MUST return a promise
instance** (`merr` may also throw). A plain value does NOT get
wrapped: the impl's `fbind` pushes a BIND task that demands a
thenable and crashes with `TypeError: expected thenable`
(first-hand crash in the exporter worker, 2026-10-07).
`finally`/`fnly`: run `f`, mirror the original promise value.
## promesa's own -> / ->> macros (do NOT confuse with clojure.core)
`p/->` and `p/->>` exist in `promesa.core` (12.x). Their purpose is NOT
arg positioning: they expand to a `chain` over `then` and REALIZE the
promise at each step, so every form receives the realized value
(e.g. `(p/-> (js/fetch ...) .-body .-status)`).
## Repo convention
**Prefer `->>` composition** over the map family (fn-first ops); it is
how new promise code should read:
(->> (fetch ...)
(p/fmap (fn [v] ...))
(p/mcat (fn [v] (p/...))))
A `p/do` terminator inside `->>` receives the threaded promise as its
LAST expression — it runs AFTER the earlier exprs, so it can never act
as the chain's continuation; terminate by wrapping the chain in an
explicit `(p/do chain ...)` (`p/let` avoids the same trap).
Legacy code threads promises with `->` over the promise-first
family (`(-> p (p/then ...))`, `(p/then p f)` calls). Read the args in
that order when touching it; do not rewrite it unless the task touches
that code anyway.
For JVM blocking code, `promesa.exec/await!` exists but is deprecated
since 12.0.0 → `p/await`.
## Async test gotcha (CLJS)
An async cljs.test var only proves its chain when the chain is really
awaited: a broken link outside the `p/let`/`p/do` body ends silently as
"Ran N tests containing 0 assertions" with 0 failures. If a test passes
with 0 assertions, the chain broke upstream; add a print in the last
link (`CHAIN RAN`) to prove plumbing before trusting green.
## Misc
Promise library of the monorepo (frontend CLJS and common, `funcool/promesa` 12.0.1 in `common/deps.edn`). Its API has two call styles and mixing them in one expression is the trap: `fmap`/`mcat`/`merr` are **fn first, promise last** (`->>` friendly) and `then`/`then'`/`catch`/`handle` are **promise first, fn last** (`->` friendly). Convention: `fmap`/`mcat`/`merr` inside `->>` pipelines, `then`/`catch` with `->`. A `then` written inside a `->>` reads backwards and hides whether the step carries a value or a promise.
- `fmap` (alias `map`, both `(f p)`) is functor map: `f` returns a plain value.
- `mcat` (alias `mapcat`, both `(f p)`) is monadic bind: `f` must return a promise, and the result mirrors it.
- `then` also unwraps a promise returned by `f`, which is why it covers both cases at the cost of not naming which one it is; `then'` maps without unwrapping.
- `merr` (fn first) and `catch` (promise first) are the error side.
- `all` takes a collection of promises (a lazy seq works); `let` chains sequential bindings.
- Seam with the rx pipelines of the frontend: `beicon.v2.core/from` builds an observable from "arrays, collections, and promise instance" (`external/beicon`), so `(->> p (p/mcat f) (rx/from) (rx/map g))` is the usual bridge (`app.main.data.imports`, `app.util.zip`).

View File

@ -10,6 +10,7 @@
- `app.common.logic.*`: higher-level workflows/algorithms over files, shapes, components, variants, libraries, tokens, etc.
- `app.common.geom.*`: geometry helpers and transformations.
- `app.common.schema` / `app.common.schema.*`: Malli abstraction layer.
- `app.common.export`: asset-export job item/params schemas, the single source of truth the backend job-def and the exporter worker validate and decode against.
- `app.common.math`, `app.common.time`, `app.common.uuid`, `app.common.json`, etc.: cross-runtime utilities.
- `app.common.test_helpers.*`: test builders and production-path helpers.

View File

@ -8,7 +8,7 @@ You are working on the GitHub project `penpot/penpot`, a monorepo.
- Cross-cutting testing principles, TDD workflow, and anti-patterns: `mem:testing`.
- Verified Clojure language behaviors that contradict common assumptions (e.g. `int?` covers `Long`; `integer?` is the general predicate): `mem:clojure/idioms` — read before assuming stdlib predicate semantics.
- Read `mem:clojure/design-rules` (function shape, boundary checks, naming) before writing or modifying Clojure/ClojureScript code — not only when designing a solution or an API.
- Promises: which call style to use where (`fmap`/`mcat`/`merr` vs `then`/`catch`) and how they bridge to rx: `mem:clojure/promesa` — read before chaining promises or mixing them with observables.
- Promise code (JVM + CLJS): promesa's two API families and their arg orders (fmap/mcat fn-first, then/catch promise-first, `mcat` f must return a promise), the repo `->>` preference, and async-test silent-green traps: `mem:clojure/promesa` — read before writing or reading promise chains.
# Development workflow

View File

@ -76,7 +76,7 @@ No `--delete` on the working-tree pass: gitignored caches in the workspace survi
- `run-devenv-shell [--instance 0|wsN|N] [cmd...]`: bash in target instance. (`--instance` flag not yet renamed to `--ws`.)
- `start-devenv` / `log-devenv` / `drop-devenv`: legacy paths around ws0 + shared infra. `drop-devenv` never removes volumes.
`exporter/scripts/run` and `wait-and-start.sh` source `backend/scripts/_env` then `_env.local` if present.
`exporter/scripts/run` and `wait-and-start.sh` source `backend/scripts/_env` then `_env.local` if present, and default `PENPOT_INTERNAL_URI` to the in-container nginx loopback (`http://localhost:4449`, `:-` so an explicit value wins). The loopback is invariant across instances, so it is defined at the exporter start scripts — the only in-container consumer — not in compose or `_env`.
## Image and dependency pins

View File

@ -1,36 +1,77 @@
# Exporter Architecture and Workflow
`exporter/`: CLJS/Node headless export service. Depends on `common/`; uses Playwright plus export JS/CLJS deps for SVG/PDF/assets.
`exporter/`: CLJS/Node **worker process** that consumes the backend's
`:exporter` job queue and renders shapes, frames and files to bitmap, SVG and
PDF. Depends on `common/`; uses Playwright plus export JS/CLJS deps for
SVG/PDF/assets. There is no HTTP surface of its own and nothing is persisted
on this side: the row of the job, its cancel and its result are the backend's.
## Layout and commands
- Source: `exporter/src/`; config: `deps.edn`, `shadow-cljs.edn`, `package.json`; runtime helpers/assets: `vendor/`, `scripts/`.
- `exporter/resources/wasm/render-wasm.{js,wasm}` and `exporter/src/app/wasm/shared.js` are generated render-wasm build output, not source: they are gitignored. Never hand-edit or commit them; regenerate by rebuilding render-wasm.
- From `exporter/`: setup `./scripts/setup`; watch `pnpm run watch` or `pnpm run watch:app`; production build `pnpm run build`; test bundle `pnpm run build:test`; tests `pnpm run test` or `pnpm run test:quiet`; lint `pnpm run lint:clj`; format check/fix `pnpm run check-fmt:clj` / `pnpm run fmt:clj`.
- Source: `exporter/src/` (one tree, `exporter.*`; the legacy `app.*` tree is
gone); config: `deps.edn`, `shadow-cljs.edn`, `package.json`; runtime helpers/assets: `vendor/`, `scripts/`.
- `exporter/resources/wasm/render-wasm.{js,wasm}` and `exporter/src/exporter/wasm/shared.js` are generated render-wasm build output, not source: they are gitignored. Never hand-edit or commit them; regenerate by rebuilding render-wasm.
- From `exporter/`: setup `./scripts/setup`; watch `pnpm run watch` or `pnpm run watch:app` (both build render-wasm first); run what watch built with `pnpm run start:dev` (dev env via `scripts/run`; changes arrive through shadow hot-reload while `watch` runs) or `pnpm run start` (env as-is, same as the container `CMD`); production build `pnpm run build` (wasm + shadow release, same chain `scripts/build <version>` uses for the release bundle before stamping the version and generating `setup`); test bundle `pnpm run build:test`; tests `pnpm run test` or `pnpm run test:quiet`; lint `pnpm run lint:clj`; format check/fix `pnpm run check-fmt:clj` / `pnpm run fmt:clj`. `./scripts/ci exporter` in the repo root is green (test-quiet defaults `PENPOT_SECRET_KEY` like `test` does).
- Because exporter consumes `common/`, shared file/shape/model changes may need exporter verification even when the immediate change is not under `exporter/`.
- Cross-cutting testing principles and anti-patterns: `mem:testing`.
- Exporter test conventions and CI: `mem:exporter/testing`.
- No promesa chains in `src` (only raw JS promise interop where a JS API demands
it); async is `^:async` + `await`. Promise arg orders and the async-test
silent-green trap: `mem:clojure/promesa` — read before touching async code.
## HTTP and browser pool
## The worker anatomy
- POST body limit is about 60 MB. Exporter supports `application/transit+json`; request params merge query params and body params.
- Map response bodies are Transit JSON and force HTTP 200; nil 200 bodies become 204.
- Auth token comes from cookie `auth-token`, then uploads use Bearer auth plus the management shared key.
- Each export job gets a fresh Playwright browser context. On success, the context closes and the browser returns to the pool; on error, the browser is destroyed instead of reused.
- Borrow validates browser connection. Pool acquire timeout is about 10s; font loading timeout logs a warning and continues after about 15s.
- The queue wake-up contract (key shape, the two-item payload, mark-then-push, the claim-back rule) is the backend's public surface now: `mem:backend/core`, "The wake-up contract of a worker queue". This process was its first external consumer; read it before writing another.
- Main thread boots pools + K pollers (`exporter.main`): browser pool, wasm pool, temp area (`:exporter/tmpdir`, which also cleans what a previous process left on its init), then `consumer.worker/start`. Shutdown: SIGTERM/SIGINT run `shutdown` (the worker stops intake first, in-flight runs drain within their grace period, then the system halts in reverse dependency order) before exiting; `stop` alone halts the same way without exiting.
- K pollers (`exporter.consumer.worker`): one Redis connection per poller (connection = slot + backpressure), BLPOP on `penpot.worker.queue:<tenant>:exporter`, payload `[job-id scheduled-at]` JSON; corrupt payload warns and drops, dead Redis delays 1s and retries. `K = PENPOT_EXPORTER_WORKER_CONCURRENCY` (default 2, floored at 1); replicas × K is the total concurrency. Pools size themselves to K (both read `consumer.config/concurrency`); there is no independent pool-max config anymore, and the wasm pool takes `:min` (`PENPOT_WASM_WORKER_POOL_MIN`, clamped to the max) plus `:idle-timeout` (`PENPOT_WASM_WORKER_IDLE_TIMEOUT`, seconds turned into milliseconds). The other legacy pool/queue keys were deleted: nothing reads them.
- Management client (`exporter.consumer.api`): claim/progress/create-job-session/complete-job-with-artifact (multipart via undici FormData)/fail-job, POST to `<internal-uri>/api/management/methods/<method>` with `X-Shared-Key: exporter:<tenant> <key>` (key = `PENPOT_EXPORTER_SHARED_KEY` or HKDF-derived from the secret with a per-tenant salt; the backend only lets the id touch its own tenant's jobs); transit bodies, non-2xx errors carry `:status` and the error body. One module-level undici `Agent` (TLS posture unchanged, closed in `shutdown` via `close-client!`) and a per-call `AbortSignal.timeout`: 10s for claim/beat/session, 120s for the artifact upload, so a hung backend cannot pin a poller slot. `fetch` is a plain def seam for hermetic tests.
- Runner (`exporter.consumer`, `run-export [render tmpdir claim params]`): mints a render session (`create-job-session`), plans via `make-plan` (single → artifact of its type; multi → zip; frames → pdf via pdfunite), settles via `complete-job-with-artifact` multipart or `fail-job`; the settle never rejects. `exporter.consumer.plan` holds the render-plan pieces (name transducers, partition of 50, grouping by `[scale type]`), dispatching on `exporter.renderer/wasm?`. The job kind is declared (`:kind` in the frozen params), never sniffed.
- Beats: per-object milestones `:preparing/:rendering/:packaging` with `objects`/`pages` counters, throttled at 250ms; a watchdog repeats the last milestone every 1s.
- Beats are serialized (`consumer/enqueue-beat`): one report in flight, the next waiting, so the watchdog repeating the last report can never overwrite a newer milestone. The chain is safe because every call carries a timeout.
- Cancellation: a `skip` answer on any beat means the row is terminal (cancelled, aborted, settled). The runner raises `:job-cancelled` between units of work; mid-Skia the per-lease interval fires the SharedArrayBuffer signal and terminates the leased worker thread. The runner owns the local registry entry via `exporter.jobs` and releases it on settle.
- `exporter.jobs` is the **local cancel arm only**: one flag per claimed job (`register`/`mark-cancelled` returning true/nil/`cancelled?`/`release`) plus one zero-arg check constructor per job (`check-cancelled`, raising `:job-cancelled`). No terminate callbacks and no shared-array signal travel through here — the abort belongs to whoever holds the render (the per-lease interval of `exporter.wasm.scope` terminates its own worker).
- Temp files: `exporter.tmpdir/track`/`release` per job id; the `:exporter/tmpdir` init cleans orphans (aged `:job-ttl`). No redis-side store, cancel topic or abandoned-job sweep: an unclaimed-in-time row goes `aborted` by the backend's own lease GC.
- Run artifacts: `exporter.resources` (`create [tmpdir type name]`, `create-zip`/`add-to-zip`/`close-zip`); no per-file deletion timers — the tmpdir registry owns every path.
## Export batching and async behavior
## Running it (devenv and deploy)
- `prepare-exports` groups entries by `[scale type]` and partitions groups into chunks of 50. Each partition uses file/page/share/name from its first item, so be careful if entries might cross those boundaries.
- Single-export response is used only when multiple export is not forced and there is exactly one prepared export containing exactly one object.
- Multi-object export can run async: when `wait` is false it returns a resource immediately and publishes progress/end/error to Redis by profile topic; when `wait` is true it waits for upload and returns the uploaded resource.
- Frame export returns a resource immediately and publishes Redis updates; it does not follow the same `wait` option path.
- ZIP entry names are sanitized and duplicates receive numeric suffixes.
- Devenv: the tmux `exporter` window (`./scripts/watch` + `wait-and-start.sh`) IS the worker now — no separate window, no role env. It needs the backend and valkey of the devenv up; `watch:app` rebuilds with the same shadow build the `main` build uses.
- Deploy: the compose service is one `penpot-exporter` (the same image, `CMD ["node", "app.js"]`); scale by replicas, K is per process. Nginx has no `/api/export` route anymore: the frontend talks to the backend's RPC only (`create-export-assets-job`), and downloads ride `/assets/by-id/?share-id=`/auth of the job resource.
- Useful beats while debugging: the boot log names `:workers K`; each poller logs `running job`/`job settled`; the runner logs `export job settled` with `:outcome`, and a cancel logs `job cancelled by backend` on the watchdog.
## The tree
- Lifecycle is `exporter.utils.system`: async-aware Integrant-style init/halt (`init-key`/`halt-key` multimethods, `ref`s, `weavejester.dependency` ordering, `^:async` + `cljs.core/await`, no promesa chains); tests are `^:async deftest` with `await` directly in the body.
- `exporter.main` mirrors the backend's `app.main`: a `system-config` def reading env at the wiring layer, a public `system` atom with the running map, `start`/`stop`/`restart` (`^:async`, resolving `:started`/`:stopped`/`:restarted`), a thread guard in `start` (a render worker thread runs this same bundle and boots the `exporter.wasm.worker` engine instead of the system — without it a warmed pool worker would nest pools and pollers and never post `ready`), `start-custom` as the test seam, idempotent `install-process-handlers` for the `uncaughtException`/SIGTERM/SIGINT wiring (signals drain through `shutdown`, whose `exit` is a test seam; runs on `start`, never on namespace load), `^:dev/before-load-async`/`^:dev/after-load-async` hooks bridging the promise lifecycle to shadow's `done` callbacks (a hook failure logs, never sticks the reload), and `main` as the entry point: shadow boots `exporter.main/main`. The entry leaves info logging enabled for the new tree (`setup!` on the `exporter` logger); without it the default level filters everything below fatal.
- Service config maps take plain (non-namespaced) param keys (`{:exporter.browser/pool {:max 2}}`): the top-level key already scopes the component, so namespacing the params only adds noise. Component keys and error reasons stay namespaced.
- `exporter.config` is the process configuration: `PENPOT_`-prefixed env over defaults, malli-decoded and validated; `get`/`get-internal-uri`/`management-key` (HKDF)/`flags`/`version`. `prepare-config` over an explicit env object is pure and throws `:invalid-config` (tests exercise it); only the wiring entry `load-config!` reads the live environment and exits on it.
- `exporter.browser` is the browser pool service: the running instance is the raw pool, the public API takes it first (`exec` fails fast on a nil pool), the browser factory is injected through config for hermetic tests.
- `exporter.wasm.pool` follows the backend `db/run!` shape: `run [cfg f & args]` resolves the pool from `cfg` (the `:exporter.wasm/pool` service instance, or a raw pool), acquires a worker, applies `f` to the cfg carrying it under `:exporter.wasm.pool/worker` plus any extra args, and returns the worker — or destroys it when `f` fails, like a rollback. Pure pool mechanics only (`init`/`halt`, `terminate`, `await-ready`); the render protocol lives one level up.
- `exporter.wasm.scope` owns the lease: `run [cfg on-scope]` lends one worker for the whole batch (serializer, per-lease signal buffer, mid-render abort interval), and the public low-level `render-on [cfg params]` renders on the leased worker — asserting `:worker-not-leased`/`:timeout-ms-missing` (`:assertion`, never a silent default: a bare `::key` resolves to the CURRENT ns, never to an alias, so cross-ns keys are literals or `::alias/key`) — with the rest preconfigured and namespaced by setter (`:exporter.renderer/on-object` from the caller, `:exporter.wasm.scope/cancel-buffer` from the lease); the check is mandatory at the lease entry and runs here when configured.
- `exporter.renderer.browser` merges the bitmap/pdf/svg flows into one `render [cfg params on-object check-cancelled]` (branching by type, `on-object` always awaited, unknown types raise); `cfg` carries what the job params do not freeze (`:exporter.browser/pool`, `:base-uri`, `:public-uri`, `:svgo?`).
- `exporter.browser.scope` owns the browser batch: `run [cfg exports]` fires every export through its own checkout, all at once, bounded by the pool. No lease — a browser batch shares nothing but the pool, so fan-out only. The facade validates once, partitions by backend and joins both scopes.
- The wasm driver is `exporter.wasm.scope/run`, called from the facade; there is no `exporter.renderer.wasm` wrapper.
- `exporter.renderer` is the malli-validated facade with one entry point, the `render-fn` closure over `cfg`: it validates the task map (`:exports :on-object :check-cancelled`) and dispatches it through `render [cfg task]`, which assumes a valid task and only partitions and joins — the browser exports through `exporter.browser.scope`, the wasm ones through one leased worker — resolving nil once every export landed (objects travel through `on-object`). The `:exporter/renderer` service builds the closure from pools plus tmpdir (refs) and static config (`:base-uri`, `:public-uri`, `:management-key`, `:image-cache-size`, `:svgo?`). The facade merges the render env (`::wrender/env`) into every wasm export's params before the lease: the worker thread reads no config and no globals. The consumer renders through the injected fn and never requires the facade. `cfg` is task-free infrastructure with process lifetime (pools, uris, key, flags, tmpdir); everything per-task rides in the validated task map. One closed schema validates data plus injections, so a bad task — or a misspelled key — never touches a pool; task validation raises `:data-validation`. Each driver takes the same `[cfg params on-object check-cancelled]` shape (`headless?` is now `wasm?`: the browser also renders headless, so the old name lied) and implements cancel internally: cooperative checkpoints on the browser, checkpoints plus a mid-render abort of its own leased worker on wasm. The lease lives one level down, in `exporter.wasm.scope/run`, which the facade calls with the task keys set once in `cfg`; the browser batch goes through `exporter.browser.scope/run [cfg exports]`.
- `exporter.consumer.worker` is the queue poller (`read-payload`, `process`, `poll-once`, `start`/`stop` with a running flag, no promesa): K connections over the `:exporter` queue, claim → `run-export` through the injected `:renderer` and `:exporter/tmpdir` → settle before the next pop. The worker config carries the queue knobs plus both by ref; no render statics nest inside it. `stop` quits every connection and ends every loop; the temp area boots clean through the tmpdir init.
- `exporter.consumer.config` holds the consumer knobs (`concurrency` floored at 1, `queue-key`, `queue-name`): the runner, the worker and `exporter.main` read it.
- `exporter.consumer.api` is the backend management client (claim/progress/session/settle, transit bodies, multipart artifact), with the same `fetch` def seam for hermetic tests.
- `exporter.wasm` is the headless render-wasm driver under Node (artifact load + headless boot, font provisioning, shape rendering); `exporter.wasm.enums` binds the generated `shared.js` the `render-wasm/build export` step emits next to it (not committed). `exporter.wasm.text`/`serialize` own the browser-free call sequencing over the shared serializers.
- `exporter.wasm.render` is the headless pipeline (fetch scene → serialize → provision fonts/images → relayout → render each object) as `[env params on-object]`: `env` carries `:internal-uri`, `:public-uri`, `:tmpdir`, `:image-cache-size`, and `fetch` is a plain def seam. Asset fetches (images `by-file-media-id`, team fonts `by-id`) carry the item's `:share-id` through `app.common.uri/append-query-param`, so an export started from a share finds its images and fonts; the render headers carry no `X-Shared-Key` (those routes authenticate the session, never the worker). `exporter.wasm.worker` is the worker-thread entry: it splits `::render/env` off the posted params (asserting `:env-missing`), assocs the cancel fn from the shared buffer, runs the pipeline posting `object`/`done`, and posts `error` instead of ever rejecting.
- `exporter.main` wires both pools (`:exporter.browser/pool` and `:exporter.wasm.pool/pool`, each sized to the worker concurrency) plus `:exporter/tmpdir` (path + job ttl from env) plus `:exporter/renderer` (pools and tmpdir as `ref`s, static uris/key/flags) plus `:exporter.consumer/worker` (`:renderer` and `:exporter/tmpdir` by ref, `:concurrency`, `:queue-key`). Service keys follow the domain (`:exporter.wasm/pool`, `:exporter.consumer/worker`), written `::alias/name` where an alias covers them. No test boots the production wiring: the wasm pool warms one worker eagerly and the worker script in a test process would be the test bundle itself; the wiring is asserted as data and the lifecycle rides fake configs.
- Async test vars prove their chain only when really awaited: a test passing with 0 assertions means the chain broke upstream.
## Render details
- Headless engines (wasm/Skia) lease one render worker for the whole run (`scope/run`); browser renders go one DOM page per partition of 50.
- Each export gets a fresh Playwright browser context. On success the context closes and the browser returns to the pool; on error the browser is destroyed instead of reused. Borrow validates the connection; pool acquire timeout about 10s; font loading timeout logs a warning and continues after about 15s.
- Bitmap export differs for WASM vs non-WASM render paths: WASM forces Playwright `deviceScaleFactor` to 1 and passes scale through the render URL; non-WASM uses `deviceScaleFactor = scale`.
- WebP is produced by taking a PNG screenshot and converting it with ImageMagick.
- SVG export rasterizes text foreignObjects to PNG, converts through PPM/color masks/potrace, and reassembles SVG paths. It also replaces non-breaking spaces for SVG compatibility and drops empty defs/paths.
- PDF export injects `@page` sizing through raw browser `evaluate` JavaScript; that code cannot rely on CLJS runtime helpers.
- Temporary resources schedule local deletion, then uploads POST to `/api/management/methods/upload-tempfile` with `X-Shared-Key: exporter <management-key>` and Bearer auth.
- ZIP entry names are sanitized and duplicates receive numeric suffixes.
## On the render engines (do not lose in refactors)
- The wasm engines run on worker threads; `../render-wasm/build export` must run first: it emits the artifact into `exporter/resources/wasm` and `src/exporter/wasm/shared.js` (both uncommitted) before compiling.
- A render thread cannot read a flag: the cancel reaches it through the SharedArrayBuffer signal, and a render that never answers is killed by the silence watchdog of `exporter.wasm.scope` (the same mechanism a hard cancel rides on).
- `exporter.main/start` decides what boots by thread: the main thread consumes the queue, a render worker (`exporter.wasm.worker/main`) renders.

View File

@ -12,5 +12,6 @@
- Focus a test var with `node target/tests/test.js --focus exporter-tests.renderer-svg-test/creates-the-correct-gradient-element`.
- Set app log level by appending `--log-level warn` (or `trace|debug|info|warn|error`).
- `test:quiet` accepts forwarded options but rebuilds the bundle; prefer the direct runner after `build:test` for focused runs.
- The direct runner and `test`/`test:quiet` default `PENPOT_SECRET_KEY=test-secret-key` when the environment says nothing: the config needs a secret to boot.
- From `exporter/`: `pnpm run check-fmt:clj` checks ClojureScript formatting.
- From `exporter/`: `pnpm run lint:clj` runs ClojureScript linting.

View File

@ -5,7 +5,7 @@ Frontend: CLJS SPA; React/Rumext; Potok; RxJS; okulary refs; SCSS modules; share
## Stable namespace map
- `app.main.ui.*`: Rumext/React UI components for workspace, dashboard, viewer, settings, auth, nitrate, etc.
- `app.main.data.*`: Potok event handlers and side effects.
- `app.main.data.*`: Potok event handlers and side effects. Exports live in `app.main.data.exports.*`: assets (shapes and frames) and files (binfile) both run as jobs of the jobs substrate (`create-export-*-job` + `dj/watch-job` on the websocket; the widget `ui.exports.assets/progress-widget` shows milestones `{stage counters}`, downloads on the completed row's `result` and cancels by job id). With render-wasm active the export never leaves the browser: the shared worker WASM module renders it (one file, a ZIP of shapes or a multi-page boards PDF) and the client downloads it locally, with no job.
- `app.main.refs`: reactive refs/lenses over store and derived workspace data.
- `app.main.store`: Potok store and `emit!`.
- `app.plugins.*` and `app.plugins`: CLJS implementation of Plugin JS API proxies.

View File

@ -81,6 +81,8 @@
:jobs-retention (ct/duration {:days 7})
:jobs-user-ttl (ct/duration {:days 7})
:jobs-request-timeout (ct/duration {:minutes 2})
:job-session-ttl (ct/duration {:hours 1})
:exporter-max-result-size (* 1024 1024 350) ; 350MiB
:media-max-file-size (* 1024 1024 30) ; 30MiB
:font-max-file-size (* 1024 1024 30) ; 30MiB
@ -214,6 +216,8 @@
[:jobs-retention {:optional true} ::ct/duration]
[:jobs-user-ttl {:optional true} ::ct/duration]
[:jobs-request-timeout {:optional true} ::ct/duration]
[:job-session-ttl {:optional true} ::ct/duration]
[:exporter-max-result-size {:optional true} ::sm/int]
[:telemetry-enabled {:optional true} ::sm/boolean]
[:default-blob-version {:optional true} ::sm/int]
[:allow-demo-users {:optional true} ::sm/boolean]
@ -504,6 +508,25 @@
(or (c/get config :jobs-user-ttl)
(ct/duration {:days 7})))
(defn get-job-session-ttl
"How long the token minted by `create-job-session` verifies, beyond
the session row's own lifetime: an export never runs for hours, so a
short expiry is defense in depth for the worker that dies without
settling. The session row itself is swept by the sessions idle GC."
[]
(or (c/get config :job-session-ttl)
(ct/duration {:hours 1})))
(defn get-exporter-max-result-size
"How big the artifact a worker sends with `complete-job` may be. The
ceiling is the same the multipart body already has at the HTTP server
(`:http/max-multipart-body-size`); lowering it rejects a result
before any object is stored. Raising it past the server cap is an
operational change of both this and nginx `client_max_body_size`."
[]
(or (c/get config :exporter-max-result-size)
(* 1024 1024 350)))
(defn get
"A configuration getter. Helps code be more testable."
([key]

View File

@ -15,6 +15,7 @@
[app.http :as-alias http]
[app.http.errors :as errors]
[app.metrics :as mtx]
[app.setup.keys :as keys]
[app.tokens :as tokens]
[app.util.inet :as inet]
[app.util.pointer-map :as pmap]
@ -394,16 +395,34 @@
[^String a ^String b]
(MessageDigest/isEqual (.getBytes a "UTF-8") (.getBytes b "UTF-8")))
(def ^:private exporter-key-prefix
"Prefix of the per-tenant exporter key ids (`exporter:<tenant>`). The
setup only registers the ids it knows, but any id with this prefix
the secret derives is a key this deployment minted: the tenant travels
in the id, so no enumeration is needed."
"exporter:")
(defn- derive-tenant-key
"The expected secret of a per-tenant exporter id: the shared derivation
with the tenant the id names, so the registry and the middleware can
never build two different salts."
[secret key-id]
(keys/derive-exporter-key secret (subs (name key-id) (count exporter-key-prefix))))
(defn- wrap-shared-key-auth
[handler keys]
[handler keys secret]
(if (seq keys)
(fn [request]
(if-let [[key-id key] (some-> (yreq/get-header request "x-shared-key")
(str/split #"\s+" 2))]
(let [key-id (-> key-id str/lower keyword)]
(if (and (string? key)
(contains? keys key-id)
(constant-time-eq? key (get keys key-id)))
(or (and (contains? keys key-id)
(constant-time-eq? key (get keys key-id)))
;; Per-tenant exporter ids need no registry entry.
(and (string? secret)
(str/starts-with? (name key-id) exporter-key-prefix)
(constant-time-eq? key (derive-tenant-key secret key-id)))))
(-> request
(assoc ::http/auth-key-id key-id)
(handler))

View File

@ -157,22 +157,34 @@
(declare ^:private assign-session-cookie)
(declare ^:private clear-session-cookie)
(defn generate-token
"The authentication token of a session, with exactly the claims the
authz resolver accepts: the session id (`:sid`) and the profile it
belongs to (`:uid`). The optional `:exp` bounds how long the token
verifies, independent of the session row: without it, the expiry of a
login session is its absolute max-age counted from `:created-at`."
[cfg session & {:keys [exp]}]
(let [absolute-max-age (cf/get :auth-token-cookie-max-age-absolute
default-cookie-max-age-absolute)
claims (cond-> {:iss "authentication"
:aud "penpot"
:sid (:id session)
:iat (:modified-at session)
:uid (:profile-id session)
:sso-provider-id (:sso-provider-id session)
:sso-session-id (:sso-session-id session)}
exp
(assoc :exp exp)
(and (nil? exp) (ct/inst? (:created-at session)))
(assoc :exp (ct/plus (:created-at session)
absolute-max-age)))
header {:kid 1 :ver 1}]
(tokens/generate cfg claims header)))
(defn- assign-token
[cfg session]
(let [absolute-max-age (cf/get :auth-token-cookie-max-age-absolute default-cookie-max-age-absolute)
claims {:iss "authentication"
:aud "penpot"
:sid (:id session)
:iat (:modified-at session)
:uid (:profile-id session)
:sso-provider-id (:sso-provider-id session)
:sso-session-id (:sso-session-id session)}
claims (if (:created-at session)
(assoc claims :exp (ct/plus (:created-at session) absolute-max-age))
claims)
header {:kid 1 :ver 1}
token (tokens/generate cfg claims header)]
(assoc session :token token)))
(assoc session :token (generate-token cfg session)))
(defn create-fn
[{:keys [::manager] :as cfg} {profile-id :id :as profile}

View File

@ -807,15 +807,31 @@
WHERE id = ?
AND status IN ('new', 'scheduled', 'running', 'retry')")
(def ^:private sql:touch-heartbeat-of-tenant
"UPDATE job
SET modified_at = ?
WHERE id = ?
AND tenant = ?
AND status IN ('new', 'scheduled', 'running', 'retry')")
(def ^:private sql:lock-active-job
"SELECT id FROM job
WHERE id = ?
AND status IN ('new', 'scheduled', 'running', 'retry')
FOR UPDATE")
(def ^:private sql:lock-active-job-of-tenant
"SELECT id FROM job
WHERE id = ?
AND tenant = ?
AND status IN ('new', 'scheduled', 'running', 'retry')
FOR UPDATE")
(defn- touch-job
[connectable job-id now]
(-> (db/exec-one! connectable [sql:touch-heartbeat now job-id])
[connectable job-id now tenant]
(-> (if (some? tenant)
(db/exec-one! connectable [sql:touch-heartbeat-of-tenant now job-id tenant])
(db/exec-one! connectable [sql:touch-heartbeat now job-id]))
(db/get-update-count)))
(defn- report-progress
@ -823,14 +839,18 @@
The row is locked first: a concurrent complete, fail or cancel blocks
here, so a progress event can never land on an already terminal job.
With a `tenant`, the lock also filters by it, so a foreign report
matches nothing and writes nothing.
Returns 1 when the event was stored.
Like every lifecycle write it owns its transaction (see `own-transaction`),
so a beat survives the rollback of the work around it."
[cfg job-id progress]
[cfg job-id progress tenant]
(db/tx-run! (own-transaction cfg)
(fn [{:keys [::db/conn] :as tx-cfg}]
(if (db/exec-one! conn [sql:lock-active-job job-id])
(if (if (some? tenant)
(db/exec-one! conn [sql:lock-active-job-of-tenant job-id tenant])
(db/exec-one! conn [sql:lock-active-job job-id]))
(do
(insert-event tx-cfg job-id "progress" progress)
1)
@ -844,19 +864,35 @@
not. The status is read from the row, because a write can only tell
that it matched no row, not why.
With a `tenant`, a row of another tenant is inactive too, and the
interrupt carries `:reason :foreign-tenant` (and no `:status`): the
caller answers it like a missing row, without revealing state.
The interrupt is an internal signal, not an error: the caller is a task
that must stop the work in course. `heartbeat` calls this when one of
its writes finds the job gone, and a caller can also ask explicitly,
which is what the binfile adapter does before and after its run.
Answers with the cfg, so a caller can thread the check into its run."
[cfg job-id]
[cfg job-id & {:keys [tenant]}]
(let [job (get-job cfg job-id)]
(when-not (and (some? job) (contains? active-statuses (:status job)))
(cond
;; a row of another tenant is not this caller's business: same
;; signal, no state
(and (some? job) (some? tenant) (not= tenant (:tenant job)))
(ex/raise :type :interrupt
:code :job-interrupted
:reason :foreign-tenant
:hint "the job belongs to another tenant"
:job-id job-id)
(not (and (some? job) (contains? active-statuses (:status job))))
(ex/raise :type :interrupt
:code :job-interrupted
:reason (if job :terminal :gone)
:hint "the job is no longer active"
:job-id job-id
:status (or (:status job) "gone")))
cfg))
(defn heartbeat
@ -867,6 +903,9 @@
- `:job-id` explicit job id; defaults to `::jobs/job-id` on the cfg
and then to the runner-bound `*job-id*`.
- `:progress` optional progress report (see `schema:progress`).
- `:tenant` optional tenant the beat belongs to: both writes filter
by it, so a foreign beat matches nothing. Internal
runners omit it and beat as their own tenant.
- `::force?` internal: skips only the progress throttle.
Returns the number of durable writes performed, so 0 means there was
@ -892,7 +931,7 @@
forgiving on the handler path, where it is logged and ignored; the
forced path used by the management API propagates it so the external
worker can retry."
[cfg & {:keys [job-id progress] :as options}]
[cfg & {:keys [job-id progress tenant] :as options}]
(let [job-id (or job-id (get cfg ::job-id) *job-id*)
;; a malformed report is a caller bug: never swallow it
progress (some-> progress validate-progress)]
@ -902,16 +941,16 @@
force? (boolean (::force? options))
writes (volatile! 0)]
(when (should-write? heartbeats job-id now heartbeat-interval)
(let [n (touch-job connectable job-id now)]
(let [n (touch-job connectable job-id now tenant)]
;; the touch is conditional on the row still being active: a
;; write that matched no row is what raises the interrupt
(when (zero? n)
(check-active cfg job-id))
(check-active cfg job-id :tenant tenant))
(vswap! writes + n)))
(when (and (some? progress)
(or force?
(should-write? progresses job-id now progress-interval)))
(let [store #(report-progress cfg job-id progress)
(let [store #(report-progress cfg job-id progress tenant)
n (if force?
(store)
(try
@ -925,7 +964,7 @@
;; the raise stays outside the catch above: an interrupt is
;; not a transient failure and must not be swallowed with it
(when (zero? n)
(check-active cfg job-id))
(check-active cfg job-id :tenant tenant))
(vswap! writes + n)))
(int @writes)))))

View File

@ -665,6 +665,7 @@
:offload-file-data (ig/ref :app.tasks.offload-file-data/job-def)
:objects-gc (ig/ref :app.tasks.objects-gc/job-def)
:export-binfile (ig/ref :app.tasks.export-binfile/job-def)
:export-assets (ig/ref :app.tasks.export-assets/job-def)
:import-binfile (ig/ref :app.tasks.import-binfile/job-def)
:storage-gc-deleted (ig/ref ::sto.gc-deleted/job-def)
:storage-gc-touched (ig/ref ::sto.gc-touched/job-def)
@ -731,6 +732,11 @@
::mtx/metrics (ig/ref ::mtx/metrics)
::mbus/msgbus (ig/ref ::mbus/msgbus)}
;; the export of shapes/frames: the work runs on the external exporter
;; worker, so this def carries no handler work and no components
:app.tasks.export-assets/job-def
{}
:app.storage.gc-deleted/job-def
{::db/pool (ig/ref ::db/pool)
::sto/storage (ig/ref ::sto/storage)

View File

@ -370,14 +370,19 @@
"Authorizes a shared-key caller against the method's `::allowed-shared-keys`.
A management method declares the shared-key ids allowed to call it with
`::allowed-shared-keys`. When the metadata is absent any configured shared key is
accepted, preserving the previous behavior. When it is present and empty no
key is accepted."
`::allowed-shared-keys`: a set of ids, or a predicate over the calling
key id (e.g. the exporter family with per-tenant ids). When the metadata
is absent any configured shared key is accepted, preserving the previous
behavior. When it is present and empty no key is accepted."
[_ f mdata]
(if (contains? mdata ::allowed-shared-keys)
(let [allowed-keys (::allowed-shared-keys mdata)]
(let [allowed (::allowed-shared-keys mdata)]
(fn [cfg params]
(if (contains? allowed-keys (::auth-key-id params))
(if (if (set? allowed)
(contains? allowed (::auth-key-id params))
;; A predicate (e.g. the exporter family with per-tenant
;; ids): the set form stays for every other method.
(allowed (::auth-key-id params)))
(f cfg params)
(do
(l/wrn :hint "shared key is not allowed for this method"
@ -541,7 +546,7 @@
["/api"
["/management"
["/methods/:method-name"
{:middleware [[mw/shared-key-auth shared-keys]
{:middleware [[mw/shared-key-auth shared-keys (:secret-key (::setup/props cfg))]
[session/authz cfg]]
:handler (make-rpc-handler management-methods)}]

View File

@ -16,6 +16,7 @@
[app.binfile.v3 :as bf.v3]
[app.common.data :as d]
[app.common.exceptions :as ex]
[app.common.export :as export]
[app.common.schema :as sm]
[app.common.time :as ct]
[app.config :as cf]
@ -44,6 +45,19 @@
[:map {:title "create-export-binfile-job" :closed true}
[:params export-binfile/schema:params]])
(def ^:private schema:create-export-assets-job
"The shapes and frames frozen in the params, rendered as image files
by the external exporter worker; the business params are what the
`:export-assets` job-def declares.
`:transient` marks a single-use export (a clipboard image, a plugin
shape export): the caller consumes the artifact in the same flow and
never comes back for it, so its row and object live minutes, not the
user ledger's days."
[:map {:title "create-export-assets-job" :closed true}
[:params export/schema:params]
[:transient {:optional true} ::sm/boolean]])
(def ^:private schema:create-import-binfile-job
"The `.penpot` package the caller uploaded, imported into a project by
the backend runner; the business params are what the `:import-binfile`
@ -93,19 +107,27 @@
(defn- submit-job
"Create the job row with what every user job shares: the queue its
job-def routes the heavy work to (`::jobs/queue-name`, `:binfile` when
the def says nothing), no automatic retry and the expiry of the ledger."
[cfg job-def name params profile-id & {:keys [resource-id]}]
the def says nothing), no automatic retry and the expiry of the ledger
(or the one the caller names, for a single-use job)."
[cfg job-def name params profile-id & {:keys [resource-id expires-at]}]
(jobs/submit cfg (cond-> {::jobs/name name
::jobs/params params
::jobs/queue (or (::jobs/queue-name job-def)
:binfile)
::jobs/profile-id profile-id
::jobs/max-retries 0
::jobs/expires-at (ct/plus (ct/now)
(cf/get-jobs-user-ttl))}
::jobs/expires-at (or expires-at
(ct/plus (ct/now)
(cf/get-jobs-user-ttl)))}
(some? resource-id)
(assoc ::jobs/resource-id resource-id))))
(def ^:private transient-job-ttl
"How long a single-use export lives. The legacy tempfile paths these
flows replace lived ten minutes, and the artifact is downloaded in the
same flow: nothing reads it later."
(ct/duration {:minutes 10}))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; EXPORT
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
@ -129,6 +151,49 @@
(= 1 (count teams)))
(first teams)))))))
(defn- check-file-permissions
"The files the job will read, checked before anything is stored: a
refusal arrives synchronously. The handler checks it again before
reading, because a permission can be revoked while the job waits.
This is the binfile flavour: whole files, no share ids."
[cfg profile-id file-ids]
(when (empty? file-ids)
(ex/raise :type :validation
:code :no-files-to-export
:hint "expected at least one file to export"))
(doseq [file-id file-ids]
(files/check-read-permissions! cfg profile-id file-id)))
(defn- check-items-scope!
"One job renders one batch: every item names the same file, page and
share (a missing share-id is its own group). The worker groups by
scale and type and renders each group against the page of its first
item, so a mixed batch would render objects against the wrong page
instead of failing. The legacy surface only took one file; refuse
here, synchronously, the way permissions refuse."
[items]
(let [scopes (into #{} (map (juxt :file-id :page-id :share-id)) items)]
(when (> (count scopes) 1)
(ex/raise :type :validation
:code :mixed-export-scope
:hint "all the items of an export belong to the same file, page and share"))))
(defn- check-item-permissions
"The assets flavour: the items name files, and a viewer that reached
them through a share link names the share, so the permission goes the
same way every share fetch goes (a logged viewer of a public share is
allowed; the workspace path stays for the plain items). The command
is not anonymous: an anonymous visitor has no profile to own the
artifact, the same as the exporter cookie this cutover replaces."
[cfg profile-id items]
(check-items-scope! items)
(doseq [item items]
(let [file-id (:file-id item)
share-id (:share-id item)]
(if (some? share-id)
(files/check-read-permissions! cfg profile-id file-id share-id)
(files/check-read-permissions! cfg profile-id file-id)))))
(sv/defmethod ::create-export-binfile-job
"Create a durable job that exports a set of files as a `.penpot`
package.
@ -153,13 +218,7 @@
params (jobs/validate-params job-def (:params envelope))
file-ids (:file-ids params)]
(when (empty? file-ids)
(ex/raise :type :validation
:code :no-files-to-export
:hint "expected at least one file to export"))
(doseq [file-id file-ids]
(files/check-read-permissions! cfg profile-id file-id))
(check-file-permissions cfg profile-id file-ids)
(quotes/check! cfg {::quotes/id ::quotes/export-jobs-per-profile
::quotes/profile-id profile-id})
@ -173,6 +232,42 @@
:export-type (d/name (:export-type params))}
team-id (assoc :team-id team-id))}))))
(sv/defmethod ::create-export-assets-job
"Create a durable job that renders shapes and frames as image files.
Its params freeze the items to render (file, page, object, type,
scale, name and the optional share id) the way the exporter receives
them. The work runs on the external exporter worker, which consumes
the queue the job-def names; the backend ships no runner for it.
The job is created, not run: the caller follows it by its id. The
read permission of every file named in the items is checked here so a
refusal arrives synchronously, and the handler checks it again before
reading."
{::doc/added "2.20"
::webhooks/event? true
::sm/params schema:create-export-assets-job
::sm/result schema:job-summary}
[cfg {:keys [::rpc/profile-id] :as envelope}]
(let [job-def (resolve-job-def cfg :export :export-assets)
;; the RPC layer decoded the params against the job-def schema
;; before the command ran
params (jobs/validate-params job-def (:params envelope))]
(check-item-permissions cfg profile-id (:exports params))
(quotes/check! cfg {::quotes/id ::quotes/export-jobs-per-profile
::quotes/profile-id profile-id})
(let [expires-at (when (:transient envelope)
(ct/plus (ct/now) transient-job-ttl))
summary (get-job-summary cfg (submit-job cfg job-def :export-assets
params profile-id
:expires-at expires-at))]
(with-meta summary
{::audit/props {:job-id (:id summary)
:files (count (:exports params))}}))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; IMPORT
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

View File

@ -15,19 +15,63 @@
(app.http.middleware) to authenticate the `<id> <secret>` header, and
wrap-management applies wrap-shared-key-authorization (app.rpc) to
check the key id against ::rpc/allowed-shared-keys. If the route
middleware or wrap-management changes, revisit these methods."
middleware or wrap-management changes, revisit these methods.
Every write method answers `{:action :run}` when it landed and
`{:action :skip}` when it did not, with the state of the row it found
(read fresh after the failed write) in `:status`: the worker tells a
cancelled job apart from one it was too late for, without guessing."
(:require
[app.common.exceptions :as ex]
[app.common.media :as cm]
[app.common.schema :as sm]
[app.common.time :as ct]
[app.config :as cf]
[app.db :as db]
[app.http.session :as session]
[app.jobs :as jobs]
[app.jobs.storage :as js]
[app.media.validation :as media.v]
[app.rpc :as-alias rpc]
[app.rpc.doc :as doc]
[app.util.services :as sv]))
[app.setup :as-alias setup]
[app.storage :as sto]
[app.util.services :as sv]
[clojure.string :as str]))
(def ^:private allowed-shared-key-ids
"Only the exporter worker claims and settles durable jobs today."
#{:exporter})
(def ^:private exporter-key-prefix
"Prefix of the per-tenant exporter key ids (`exporter:<tenant>`). The
bare `exporter` id is the legacy form, bound to this backend's own
tenant below."
"exporter:")
(defn- exporter-key?
"A key id of the exporter family: the bare legacy id, or one suffixed
with the tenant it renders for (lowercase — the middleware lowercases
every id before comparing)."
[key-id]
(and (some? key-id)
(let [id-name (if (keyword? key-id) (name key-id) (str key-id))]
(or (= id-name "exporter")
(str/starts-with? id-name exporter-key-prefix)))))
(defn- key-tenant
"The tenant the calling exporter renders for: the suffix of a
per-tenant key id, or this backend's own tenant for the bare legacy
id (single-tenant deploys only ever see their own)."
[key-id]
(let [id-name (name key-id)]
(if (str/starts-with? id-name exporter-key-prefix)
(subs id-name (count exporter-key-prefix))
(cf/get :tenant))))
(defn- tenant-ok?
"The job belongs to the calling exporter's tenant. Anything else is
answered exactly like a missing row, so the caller learns nothing
about rows it cannot see."
[key-id job]
(and (some? job)
(= (key-tenant key-id) (:tenant job))))
;; ---- RPC METHOD: CLAIM-JOB
@ -40,27 +84,66 @@
[:map {:title "claim-job-result"}
[:action [:enum :run :skip]]
[:name {:optional true} ::sm/text]
;; Opaque per-job params blob; intentionally untyped.
[:params {:optional true} :any]])
;; Params decoded with the row's job-def (typed values); a row
;; whose job-def is gone answers the stored shapes untouched.
[:params {:optional true} :any]
;; On `:skip`, the state of the row the write lost against.
[:status {:optional true} ::sm/text]])
(defn- decode-claim-params
"The frozen params through the decoder of the row's job-def, so the
worker gets typed values (uuids, keywords, instants) instead of the
stored JSON shapes — transit carries the types natively. A row whose
job-def is gone (retired) answers untouched, mirroring what
`jobs/decode-result-of` does for results."
[cfg row]
(let [params (:params row)]
(if-let [job-def (and (some? params)
(get (jobs/get-defs cfg) (keyword (:name row))))]
(jobs/decode-params job-def params)
params)))
(defn- row-status
"The state of the row after a write that touched nothing: one fresh
read, never the row the method already saw, so the answer names the
state the race produced."
[cfg job-id]
(:status (jobs/get-job cfg job-id)))
(defn- skip-with-status
"The answer of a landed-nowhere write: one fresh read of the row
after the failure, so the worker sees the state the race produced."
[cfg job-id]
(if-let [status (row-status cfg job-id)]
{:action :skip :status status}
{:action :skip}))
(sv/defmethod ::claim-job
{::doc/added "2.20"
::sm/params schema:claim-job-params
::sm/result schema:claim-job-result
::rpc/auth false
::rpc/allowed-shared-keys allowed-shared-key-ids}
[cfg {:keys [job-id scheduled-at]}]
::rpc/allowed-shared-keys exporter-key?}
[cfg {:keys [job-id scheduled-at] :as params}]
(let [row (jobs/get-job cfg job-id)]
;; Race condition: if the job is claimed between get-job and claim! by
;; another worker, claim! returns 0 and we return :skip. The conditional
;; claim (UPDATE ... WHERE status IN ('new','scheduled','retry') AND
;; scheduled_at=?) handles this correctly: only a pending row with an
;; exact scheduled-at match is claimed, a stale payload is skipped.
(if (and row (pos? (jobs/claim cfg job-id scheduled-at)))
(cond
;; A job of another tenant is indistinguishable from a missing one:
;; a bare skip, never the row's state.
(and (some? row) (not (tenant-ok? (::rpc/auth-key-id params) row)))
{:action :skip}
(and row (pos? (jobs/claim cfg job-id scheduled-at)))
{:action :run
:name (:name row)
:params (:params row)}
{:action :skip})))
:params (decode-claim-params cfg row)}
:else
(skip-with-status cfg job-id))))
;; ---- RPC METHOD: REPORT-JOB-PROGRESS
@ -74,15 +157,16 @@
(def ^:private schema:report-job-progress-result
[:map {:title "report-job-progress-result"}
[:action [:enum :run :skip]]])
[:action [:enum :run :skip]]
[:status {:optional true} ::sm/text]])
(sv/defmethod ::report-job-progress
{::doc/added "2.20"
::sm/params schema:report-job-progress-params
::sm/result schema:report-job-progress-result
::rpc/auth false
::rpc/allowed-shared-keys allowed-shared-key-ids}
[cfg {:keys [job-id progress]}]
::rpc/allowed-shared-keys exporter-key?}
[cfg {:keys [job-id progress] :as params}]
;; A lost race (row already terminal) reports :skip so the worker
;; stops retrying a report that can never land, mirroring claim-job.
;; The forced route propagates database errors: the external worker
@ -90,62 +174,237 @@
;; the beat raises when the job is no longer active is caught here for
;; the same reason, and it never reaches the HTTP error mapping: it is
;; an internal signal, not a failure of the request.
;;
;; The tenant rides in the write itself (`:tenant`), so the hot path is
;; one statement and needs no row read. A beat for another tenant's job
;; matches nothing and the substrate says why; the answer is a bare
;; skip, never the row's state.
(let [result (ex/try! (jobs/heartbeat cfg
:job-id job-id
:progress progress
:tenant (key-tenant (::rpc/auth-key-id params))
::jobs/force? true))]
(cond
(ex/exception? result)
(if (= :interrupt (:type (ex-data result)))
{:action :skip}
(throw result))
(let [data (ex-data result)]
(cond
(= :foreign-tenant (:reason data))
{:action :skip}
(= :interrupt (:type data))
(skip-with-status cfg job-id)
:else
(throw result)))
(pos? result)
{:action :run}
:else
{:action :skip})))
(skip-with-status cfg job-id))))
;; ---- RPC METHOD: COMPLETE-JOB
(def ^:private schema:complete-job-params
[:map {:title "complete-job-params"}
[:job-id ::sm/uuid]
;; Per-job result: any JSON-encodable value, nil when the job returns
;; nothing. Nested maps and vectors are allowed; unserializable
;; values are dropped to nil by jobs/encode-result with a warning. A
;; job whose job-def declares a result schema has its result validated
;; against it: one that does not conform is rejected and the job is
;; not completed.
[:result [:maybe :any]]
;; Storage object the worker produced for this job. A resource is only
;; set, never replaced: a job that already has one is rejected.
[:resource-id {:optional true} ::sm/uuid]])
[:and
[:map {:title "complete-job-params"}
[:job-id ::sm/uuid]
;; Per-job result: any JSON-encodable value, nil when the job returns
;; nothing. Nested maps and vectors are allowed; unserializable
;; values are dropped to nil by jobs/encode-result with a warning. A
;; job whose job-def declares a result schema has its result validated
;; against it: one that does not conform is rejected and the job is
;; not completed.
[:result {:optional true} [:maybe :any]]
;; Storage object the worker produced for this job. A resource is only
;; set, never replaced: a job that already has one is rejected.
[:resource-id {:optional true} ::sm/uuid]
;; The multipart form: the worker sends the artifact itself, the
;; server stores it. The part is the single source of what the user
;; will download: its filename and type travel in the part itself
;; (`schema:upload` requires the name; `check-artifact!` enforces
;; the allowlist on the type), never as separate text fields.
[:content {:optional true} media.v/schema:upload]
;; The render session the worker used, closed once the job settles:
;; complete is terminal, the session has no reader after it.
[:session-id {:optional true} ::sm/uuid]]
[:fn {:error/message "one of :content, :result or :resource-id is required"}
(fn [{:keys [content resource-id] :as params}]
(boolean (or (some? content)
(contains? params :result)
(some? resource-id))))]])
(def ^:private schema:complete-job-result
[:map {:title "complete-job-result"}
[:action [:enum :run :skip]]])
[:action [:enum :run :skip]]
[:status {:optional true} ::sm/text]])
(defn- close-job-session!
"The session a worker rendered with, deleted once the job settles, and
only when it really belongs to that job's owner: the id travels in the
request, so a shared-key holder must not be able to delete another
profile's login.
Best-effort: the settle already decided the outcome, and a session that
cannot be deleted now is swept later by the sessions idle GC. With no
job row to match (it was already gone) nothing is deleted."
[cfg job-id session-id]
(when (and (uuid? session-id) (uuid? job-id))
(ex/ignoring
(when-let [job (jobs/get-job cfg job-id)]
(when-let [session (session/read-session (::session/manager cfg) session-id)]
(when (= (:profile-id job) (:profile-id session))
(session/delete-session (::session/manager cfg) session-id)))))))
(defn- check-artifact!
"The bytes a worker offers as the result: the path must be one the
multipart parser wrote (`media.v/check-upload-path`), the type one the
export produces and the size under the configured cap. Every check
refuses to *use* the bytes, not to receive them (the parser already
wrote them to a server tempfile); refusing early costs nothing,
storing early costs a GC cycle."
[content]
(media.v/check-upload-path content)
(media.v/check-media-type content cm/export-artifact-types)
(let [max-size (cf/get-exporter-max-result-size)]
(when (> (:size content) max-size)
(ex/raise :type :validation
:code :request-body-too-large
:hint "the result of the job is over the maximum size"
:size (:size content)
:max-size max-size)))
content)
(defn- completable-job
"The row a completion may land on: running (or retrying), whatever
the JSON form asks for."
[cfg job-id]
(let [job (jobs/get-job cfg job-id)]
(when (and (some? job)
(contains? #{"running" "retry"} (:status job)))
job)))
(defn- artifact-owner
"The job an artifact completion may land on: completable AND owned,
because the object needs an owner and a job without one owns nothing
(it may only settle without an artifact). Read before anything is
stored: a lost race with a cancel never turns the bytes of the worker
into an object."
[cfg job-id]
(let [job (completable-job cfg job-id)]
(when (and (some? job)
(uuid? (:profile-id job)))
job)))
(defn- complete-with-artifact
"The multipart form of a completion, on a job that is still
completable: the artifact is stored out of any transaction (storage
waits for no database), owned by the profile of the job, and the
completion itself writes the `resource_id` and the result descriptor
`{resource-uri filename mtype size}` in its own transaction, where
the first terminal state wins. A completion that lost the race
releases the artifact it stored: no object lingers for the storage GC
to find later."
[cfg job content session-id]
(let [resource (js/put-resource cfg
(:profile-id job)
{:content (sto/content (:path content))
:filename (:filename content)
:mtype (:mtype content)})]
(if (pos? (jobs/complete cfg
:job-id (:id job)
:resource-id (:resource-id resource)
:result {:resource-uri (:resource-uri resource)
:filename (:filename content)
:mtype (:mtype content)
:size (:size resource)}))
(do
(close-job-session! cfg (:id job) session-id)
{:action :run})
(do
(js/release-resource cfg (:resource-id resource))
(close-job-session! cfg (:id job) session-id)
(skip-with-status cfg (:id job))))))
(sv/defmethod ::complete-job
"Complete a job on behalf of the worker that ran it, with the result it
produced and, when it wrote one, the resource it stores.
"Close a running job as completed, in one of two forms.
The params form carries the plain `:result` the job produced and an
optional storage object (`:resource-id`) the worker wrote itself: the
internal binfile path.
The artifact form carries the artifact itself (`:content`, a multipart
part with `:filename` and `:mtype` naming the download): the server
stores it, owned by the profile of the job, and coalesces the resource
into the same completion. The checks (completable, path, type, size)
run before any object is written: a lost race against a cancel never
stores bytes.
The result of a job whose job-def declares a result schema is validated
against it before it is stored: one that does not conform is rejected and
the job is not completed. A job that is no longer completable is skipped,
and nothing is validated for it."
and nothing is validated for it.
Both forms belong to the calling exporter's tenant: a job of another
tenant answers like a missing row, and nothing is stored, written or
revealed for it.
With `:session-id`, the render session the worker minted through
`create-job-session` is deleted once the completion is answered,
whatever it says: the settle of the run is what ends the life of the
session it minted, even a settle that lost against a cancel. The
session is deleted only when it belongs to the job's owner.
A completion of a job that already ended answers `{:action :skip,
:status ...}` and changes nothing."
{::doc/added "2.20"
::sm/params schema:complete-job-params
::sm/result schema:complete-job-result
::rpc/auth false
::rpc/allowed-shared-keys allowed-shared-key-ids}
[cfg {:keys [job-id result resource-id]}]
(if (pos? (jobs/complete cfg
:job-id job-id
:result result
:resource-id resource-id))
{:action :run}
{:action :skip}))
;; The artifact arrives as a multipart part: the parser writes it to a
;; server tempfile and only a request flagged multipart server-side can
;; carry it. Without this, a transit body could name any internal path.
::rpc/multipart-only-params [:content]
::rpc/allowed-shared-keys exporter-key?}
[cfg {:keys [job-id result resource-id content session-id] :as params}]
(cond
;; the multipart form stores the artifact itself: only for a job
;; that can still complete and has an owner, because the object
;; needs one
(some? content)
(if-let [job (artifact-owner cfg job-id)]
(if (tenant-ok? (::rpc/auth-key-id params) job)
(do
(check-artifact! content)
(complete-with-artifact cfg job content session-id))
;; a job of another tenant answers like a missing row, and its
;; session is not ours to close
{:action :skip})
(do
;; a completion that lost the race is not a session to keep either
(close-job-session! cfg job-id session-id)
(skip-with-status cfg job-id)))
;; the params form: nothing to store here, the resource (if any)
;; already lives in storage
:else
(if-let [job (completable-job cfg job-id)]
(if (tenant-ok? (::rpc/auth-key-id params) job)
(if (pos? (jobs/complete cfg
:job-id job-id
:result result
:resource-id resource-id))
(do
(close-job-session! cfg job-id session-id)
{:action :run})
(do
(close-job-session! cfg job-id session-id)
(skip-with-status cfg job-id)))
{:action :skip})
(do
(close-job-session! cfg job-id session-id)
(skip-with-status cfg job-id)))))
;; ---- RPC METHOD: FAIL-JOB
@ -154,19 +413,125 @@
[:job-id ::sm/uuid]
;; Rich error report: type/code/hint are required, the map stays
;; open to worker-defined details (see jobs/fail).
[:error jobs/schema:job-error]])
[:error jobs/schema:job-error]
;; The render session the worker used, closed once the job settles:
;; fail is terminal too.
[:session-id {:optional true} ::sm/uuid]])
(def ^:private schema:fail-job-result
[:map {:title "fail-job-result"}
[:action [:enum :run :skip]]])
[:action [:enum :run :skip]]
[:status {:optional true} ::sm/text]])
(sv/defmethod ::fail-job
{::doc/added "2.20"
::sm/params schema:fail-job-params
::sm/result schema:fail-job-result
::rpc/auth false
::rpc/allowed-shared-keys allowed-shared-key-ids}
[cfg {:keys [job-id error]}]
(if (pos? (jobs/fail cfg job-id error))
{:action :run}
{:action :skip}))
::rpc/allowed-shared-keys exporter-key?}
[cfg {:keys [job-id error session-id] :as params}]
;; A job of another tenant answers like a missing row, before any
;; write lands and without touching its session.
(let [job (jobs/get-job cfg job-id)]
(if (and (some? job)
(not (tenant-ok? (::rpc/auth-key-id params) job)))
{:action :skip}
;; The session is closed whatever the write says: the worker that
;; settles (even a settle that loses against a cancel it cannot
;; see) is what ends the life of the render session it minted. It
;; is deleted only when it belongs to the job's owner.
(let [landed? (pos? (jobs/fail cfg job-id error))]
(close-job-session! cfg job-id session-id)
(if landed?
{:action :run}
(skip-with-status cfg job-id))))))
;; ---- RPC METHOD: CREATE-JOB-SESSION
(def ^:private schema:create-job-session-params
[:map {:title "create-job-session-params"}
[:job-id ::sm/uuid]])
(def ^:private schema:create-job-session-result
[:map {:title "create-job-session-result"}
[:session-id ::sm/uuid]
[:session-token ::sm/text]])
(defn- validate-job!
"The job whose owner will render: it must exist, be `running` (the
claim already passed: a session for a waiting job would outlive a
job that may never run) and have an owner. A job of another tenant
is indistinguishable from a missing one on purpose: the tenant is
the boundary of what the key can touch, and the profile the boundary
of what the session can touch."
[cfg key-id job-id]
(let [job (jobs/get-job cfg job-id)]
(when (nil? job)
(ex/raise :type :not-found
:code :job-not-found
:hint "the job does not exist"
:job-id job-id))
(when-not (tenant-ok? key-id job)
(ex/raise :type :not-found
:code :job-not-found
:hint "the job does not exist"
:job-id job-id))
(when-not (= "running" (:status job))
(ex/raise :type :validation
:code :invalid-job-state
:hint "the session is minted for a running job"
:status (:status job)
:job-id job-id))
(when-not (uuid? (:profile-id job))
(ex/raise :type :validation
:code :job-without-profile
:hint "the job has no owner profile to mint the session for"
:job-id job-id))
(:profile-id job)))
(defn- validate-owner!
"The profile the session will render as: the same one the login
accepts (exists, active, not blocked, not deleted). A session is a
key to everything the owner can touch, so a worker that settles a job
of a dead profile never gets one."
[cfg profile-id]
(let [profile (db/get* cfg :profile {:id profile-id})]
(when (or (nil? profile) (some? (:deleted-at profile)))
(ex/raise :type :not-found
:code :profile-not-found
:hint "the owner profile of the job does not exist"
:profile-id profile-id))
(when-not (:is-active profile)
(ex/raise :type :validation
:code :profile-not-active
:hint "the owner profile of the job is not active"
:profile-id profile-id))
(when (:is-blocked profile)
(ex/raise :type :restriction
:code :profile-blocked
:hint "the owner profile of the job is marked as blocked"
:profile-id profile-id))))
(sv/defmethod ::create-job-session
"Mint an authentication session of the owner of a running job, for
the worker to render as the user: render.html and assets authenticate
by cookie and the RPC by Bearer, through the surfaces that already
exist. Every call mints a fresh session (worker retries never wait on
one a settle may have closed), and the token carries the short `:exp`
of `:job-session-ttl`: a worker that dies without settling leaves
nothing usable for long, and the sessions idle GC sweeps the row."
{::doc/added "2.20"
::sm/params schema:create-job-session-params
::sm/result schema:create-job-session-result
::rpc/auth false
::rpc/allowed-shared-keys exporter-key?}
[cfg {:keys [job-id] :as params}]
(let [profile-id (validate-job! cfg (::rpc/auth-key-id params) job-id)]
(validate-owner! cfg profile-id)
(let [session (session/create-session (::session/manager cfg)
{:profile-id profile-id})
token (session/generate-token cfg session
:exp (ct/plus (ct/now)
(cf/get-job-session-ttl)))]
{:session-id (:id session)
:session-token token})))

View File

@ -103,8 +103,10 @@
(let [secret (get props :secret-key)]
(reduce (fn [keys id]
(let [key (or (get cfg id)
(-> (keys/derive secret :salt (name id))
(bc/bytes->b64-str true)))]
(if (= id :exporter)
(keys/derive-exporter-key secret (cf/get :tenant))
(-> (keys/derive secret :salt (name id))
(bc/bytes->b64-str true))))]
(if (or (str/empty? key)
(str/blank? key))
(do

View File

@ -8,6 +8,7 @@
"Keys derivation service."
(:refer-clojure :exclude [derive])
(:require
[buddy.core.codecs :as bc]
[buddy.core.kdf :as bk]))
(defn derive
@ -20,3 +21,17 @@
:alg :hkdf
:digest :blake2b-512})]
(bk/get-bytes engine size)))
(defn derive-exporter-key
"The management key of the exporter that renders for `tenant`: HKDF
blake2b512 over the salt `exporter:<tenant>`, base64url.
The single definition of the exporter key on the backend: the shared-key
registry (bare `exporter` id, own tenant) and the per-tenant ids of the
middleware both come through here. The Node worker derives the same
value; both sides pin the same golden vector, so a change on one side
fails a test instead of a deployment."
[secret-key tenant]
(assert (string? tenant) "expect string tenant")
(-> (derive secret-key :salt (str "exporter:" tenant))
(bc/bytes->b64-str true)))

View File

@ -0,0 +1,55 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.tasks.export-assets
"The `:export-assets` job: the shapes and frames a caller froze at
creation time, rendered as image files.
This is an EXTERNAL job: the work runs on the exporter worker, a
process outside the JVM, so the job is created with `::jobs/queue-name
:exporter` (read by the creation command; the substrate never looks at
it) and the backend ships no runner for that queue. The handler here
is a guard, not a worker: it is never executed under a correct
configuration, because the only runner that could claim the job is
external, and `claim-job` does not look at the queue. Should a wrong
configuration ever route the job to a JVM runner, it fails visibly
here instead of \"completing\" empty.
Progress vocabulary of the external worker (the substrate validates
the shape, the worker owns the words): stages `:preparing`,
`:rendering` and `:packaging`, with counters `objects` (one per
exported shape) and `pages` (one per exported frame)."
(:require
[app.common.exceptions :as ex]
[app.common.export :as export]
[app.common.schema :as sm]
[app.jobs :as jobs]
[integrant.core :as ig]))
(defn- execute-export-assets
"Plain handler, importable and testable without integrant.
It only raises: the work of this job belongs to the external worker.
See the namespace docstring."
[_context _params]
(ex/raise :type :internal
:code :no-handler-for-external
:hint "this job runs on the external exporter worker"))
(defmethod ig/init-key ::job-def
[_ _cfg]
{::jobs/name :export-assets
::jobs/family :export
::jobs/resource-role :output
;; the queue the creation command routes the job to: an external
;; worker consumes it, the backend ships no runner for it
::jobs/queue-name :exporter
;; the item shapes live in app.common.export, the single source of
;; truth both runtimes validate and decode against
::jobs/schema export/schema:params
::jobs/handler execute-export-assets
::jobs/decoder (sm/decoder export/schema:params sm/json-transformer)
::jobs/validator (sm/validator export/schema:params)})

View File

@ -573,7 +573,11 @@
(:app.http/auth-key-id request)
:else
(first (::rpc/allowed-shared-keys mdata)))
;; A predicate declares a family, not an enumerable
;; set: fall back to the legacy id, which every such
;; method accepts.
(let [allowed (::rpc/allowed-shared-keys mdata)]
(if (set? allowed) (first allowed) :exporter)))
data (if (or (contains? data ::rpc/auth-key-id)
(nil? key-id))
data

View File

@ -212,6 +212,244 @@
(t/is (= #{:job-id :files :export-type} (set (keys props))))
(t/is (not (contains? (:result out) :team-id)))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; ASSETS EXPORT
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defn- export-item
[file-id & {:keys [page-id object-id type name scale suffix share-id]}]
(cond-> {:file-id file-id
:page-id (or page-id (uuid/next))
:object-id (or object-id (uuid/next))
:type (or type :png)
:name (or name "shape")
:scale (or scale 1)}
suffix (assoc :suffix suffix)
share-id (assoc :share-id share-id)))
(defn- create-export-assets-job
[profile-id exports & {:as extra}]
(th/command! (merge {::th/type :create-export-assets-job
::rpc/profile-id profile-id
:params {:exports exports}}
extra)))
(t/deftest create-export-assets-job-answers-with-a-pending-job
(let [profile (th/create-profile* 1)
file-id (first (:file-ids (import-fixture! profile)))
item (export-item file-id)
before (count-storage-objects)
out (create-export-assets-job (:id profile) [item])]
(t/is (th/success? out))
(let [result (:result out)
job (jobs/get-job th/*system* (:id result))]
(t/testing "the answer is enough to follow the job"
(t/is (uuid? (:id result)))
(t/is (= "pending" (:status result)))
(t/is (= "export-assets" (:name result)))
(t/is (ct/inst? (:created-at result))))
(t/testing "and the row routes to the external worker queue"
(let [job-def (jobs/get-job-def (::jobs/defs th/*system*) :export-assets)
params (jobs/decode-params job-def (:params job))]
(t/is (= "exporter" (:queue job)))
(t/is (= 0 (:max-retries job)))
(t/is (= (:id profile) (:profile-id job)))
(t/is (= 1 (count (:exports params))))
(t/is (= :png (:type (first (:exports params)))))))
(t/testing "creating the job stores no artifact"
(t/is (= before (count-storage-objects)))))))
(t/deftest create-export-assets-job-checks-the-read-permission
(let [owner (th/create-profile* 1)
other (th/create-profile* 2)
file-id (first (:file-ids (import-fixture! owner)))
out (create-export-assets-job (:id other) [(export-item file-id)])]
(t/testing "a file the caller cannot read is not exported"
(t/is (not (th/success? out)))
(t/is (= :not-found (th/ex-type (:error out)))))
(t/testing "and no job was created for it"
(t/is (zero? (count-jobs))))))
(t/deftest create-export-assets-job-rejects-an-unknown-type
(let [profile (th/create-profile* 1)
file-id (first (:file-ids (import-fixture! profile)))
out (create-export-assets-job (:id profile)
[(export-item file-id :type :psd)])]
(t/is (not (th/success? out)))
;; refused by the RPC wrapper before the command runs (the same
;; `:params-validation` a real caller sees over HTTP)
(t/is (= :params-validation (th/ex-code (:error out))))
(t/testing "and nothing was created"
(t/is (zero? (count-jobs))))))
(t/deftest create-export-assets-job-transient-expires-in-minutes
;; A single-use export (clipboard image, plugin shape export) is
;; consumed in the same flow: minutes, not the user ledger's days.
(let [profile (th/create-profile* 1)
file-id (first (:file-ids (import-fixture! profile)))
item (export-item file-id)
life (fn [out]
(let [row (th/db-get :job {:id (:id (:result out))}
:expires-at :created-at)]
(- (inst-ms (:expires-at row))
(inst-ms (:created-at row)))))
plain (create-export-assets-job (:id profile) [item])
quick (create-export-assets-job (:id profile) [item] :transient true)]
(t/is (th/success? plain))
(t/is (th/success? quick))
(t/testing "a plain export keeps the user ledger"
;; the expiry is computed before the row is written, so the life is
;; at most the ttl and at least it minus the insert
(t/is (<= (- (inst-ms (cf/get-jobs-user-ttl)) 1000)
(life plain)
(inst-ms (cf/get-jobs-user-ttl)))))
(t/testing "a transient export lives minutes"
(t/is (<= (- (inst-ms (ct/duration {:minutes 10})) 1000)
(life quick)
(inst-ms (ct/duration {:minutes 10})))))))
(t/deftest create-export-assets-job-rejects-a-mixed-scope
;; One job renders one batch: the worker groups by scale and type and
;; renders each group against the page of its first item, so items
;; from two files, pages or shares would render against the wrong
;; page. Refused here, synchronously, before any permission check.
(let [profile (th/create-profile* 1)
file-id (first (:file-ids (import-fixture! profile)))
page-id (uuid/next)
one (export-item file-id :page-id page-id)
other (export-item (uuid/next) :page-id page-id)]
(t/testing "items from two files are refused"
(let [out (create-export-assets-job (:id profile) [one other])]
(t/is (not (th/success? out)))
(t/is (= :mixed-export-scope (th/ex-code (:error out))))))
(t/testing "items from two pages are refused"
(let [out (create-export-assets-job (:id profile) [one (export-item file-id)])]
(t/is (not (th/success? out)))
(t/is (= :mixed-export-scope (th/ex-code (:error out))))))
(t/testing "and nothing was created"
(t/is (zero? (count-jobs))))))
(t/deftest create-export-assets-job-accepts-an-empty-suffix
;; the export form sends `""` when the user types no suffix (it is the
;; normal case, not an edge): the legacy `/api/export` surface took a
;; plain string, so the job must take it too
(let [profile (th/create-profile* 1)
file-id (first (:file-ids (import-fixture! profile)))
item (assoc (export-item file-id) :suffix "")
out (create-export-assets-job (:id profile) [item])]
(t/is (th/success? out))
(t/testing "and the empty suffix is frozen as-is for the worker"
(let [job (jobs/get-job th/*system* (:id (:result out)))
job-def (jobs/get-job-def (::jobs/defs th/*system*) :export-assets)
params (jobs/decode-params job-def (:params job))]
(t/is (= "" (:suffix (first (:exports params)))))))))
(t/deftest create-export-assets-job-accepts-a-frames-kind
;; the frames dialog freezes fully typed pages plus an explicit kind,
;; so the worker joins them into one pdf instead of inferring it
;; from missing keys
(let [profile (th/create-profile* 1)
file-id (first (:file-ids (import-fixture! profile)))
item (assoc (export-item file-id) :type :pdf :scale 1 :suffix "")
out (th/command! {::th/type :create-export-assets-job
::rpc/profile-id (:id profile)
:params {:exports [item]
:kind :frames}})]
(t/is (th/success? out))
(t/testing "and the kind is frozen as-is for the worker"
(let [job (jobs/get-job th/*system* (:id (:result out)))
job-def (jobs/get-job-def (::jobs/defs th/*system*) :export-assets)
params (jobs/decode-params job-def (:params job))]
(t/is (= :frames (:kind params)))))))
(t/deftest create-export-assets-job-accepts-a-file-nameable-through-a-share
;; the viewer that renders a file through a public share exports the
;; same way every share fetch goes: the share id names the file, the
;; share-link permissions answer (a logged viewer is enough), and the
;; workspace membership is never touched
(let [owner (th/create-profile* 1 {:is-active true})
viewer (th/create-profile* 2 {:is-active true})
file-id (first (:file-ids (import-fixture! owner)))
share (th/command! {::th/type :create-share-link
::rpc/profile-id (:id owner)
:file-id file-id
:pages #{(uuid/next)}
:who-comment "team"
:who-inspect "all"})
share-id (:id (:result share))
out (create-export-assets-job (:id viewer)
[(export-item file-id
:share-id share-id)])]
(t/testing "a logged viewer of the share can have the job"
(t/is (th/success? out))
(t/is (= 1 (count-jobs)))))
(t/testing "a share id that names another file is not a pass"
(let [owner (th/create-profile* 3 {:is-active true})
viewer (th/create-profile* 4 {:is-active true})
file-a (first (:file-ids (import-fixture! owner)))
file-b (first (:file-ids (import-fixture! viewer)))
share (th/command! {::th/type :create-share-link
::rpc/profile-id (:id viewer)
:file-id file-b
:pages #{(uuid/next)}
:who-comment "team"
:who-inspect "all"})
share-id (:id (:result share))
out (create-export-assets-job (:id viewer)
[(export-item file-a
:share-id share-id)])]
(t/is (not (th/success? out)))
(t/is (= :not-found (th/ex-type (:error out))))
(t/testing "and nothing was created for it"
;; only the accepted share job of the first case exists
(t/is (= 1 (count-jobs)))))))
(t/deftest create-export-assets-job-adds-the-job-id-to-the-audit-props
(let [profile (th/create-profile* 1)
file-id (first (:file-ids (import-fixture! profile)))
out (create-export-assets-job (:id profile)
[(export-item file-id)])
props (::audit/props (meta (:result out)))]
(t/is (th/success? out))
(t/testing "the audit of the call points at the job and its items"
(t/is (= (:id (:result out)) (:job-id props)))
(t/is (= 1 (:files props)))
(t/is (= #{:job-id :files} (set (keys props)))))))
(t/deftest create-export-assets-job-counts-in-the-export-quote
;; the quote of the family counts every job of it, so a mixed backlog
;; of binfile and assets jobs fills the same quote
(with-mocks [mock {:target 'app.config/get
:return (th/config-get-mock
{:quotes-export-jobs-per-profile 1})}]
(let [profile (th/create-profile* 1)
file-id (first (:file-ids (import-fixture! profile)))]
(t/is (th/success? (create-export-assets-job (:id profile)
[(export-item file-id)])))
(t/testing "the next export job of the family is refused, whatever its type"
(let [out (create-export-binfile-job (:id profile) #{file-id})]
(t/is (not (th/success? out)))
(t/is (= :max-quote-reached (th/ex-code (:error out))))))
(t/is (= 1 (count-jobs))))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; IMPORT
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

View File

@ -8,14 +8,24 @@
(:require
[app.common.logging :as l]
[app.common.time :as ct]
[app.common.transit :as transit]
[app.common.uuid :as uuid]
[app.config :as cf]
[app.db :as db]
[app.http.middleware :as mw]
[app.http.session :as session]
[app.jobs :as jobs]
[app.metrics :as-alias mtx]
[app.rpc :as-alias rpc]
[app.rpc :as rpc]
[app.setup :as-alias setup]
[app.setup.keys :as keys]
[app.storage.tmp :as tmp]
[backend-tests.helpers :as th]
[clojure.test :as t]))
[clojure.test :as t]
[cuerdas.core :as str]
[datoteka.fs :as fs]
[datoteka.io :as io]
[mockery.core :refer [with-mocks]]))
(t/use-fixtures :once th/state-init)
@ -33,7 +43,7 @@
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defn- mk-job
[{:keys [name status params scheduled-at]
[{:keys [name status params scheduled-at profile-id tenant]
:or {name "media-process"
status "new"
params {:x 1}
@ -41,23 +51,24 @@
(let [id (uuid/next)]
(th/db-insert! :job {:id id
:name name
:tenant (cf/get :tenant)
:tenant (or tenant (cf/get :tenant))
:queue "media"
:params (db/json params)
:priority 100
:max-retries 3
:retry-num 0
:status status
:profile-id profile-id
:scheduled-at scheduled-at
:created-at (ct/now)
:modified-at (ct/now)})
:modified-at (ct/now)})
id))
(defn- get-row
[id]
(let [row (th/db-get :job {:id id}
:status :result :error
:started-at :completed-at :modified-at)]
:status :result :error :resource-id
:started-at :completed-at :modified-at :name)]
(reduce (fn [row key]
(update row key #(cond-> % (db/pgobject? %) db/decode-json-pgobject)))
row
@ -85,7 +96,27 @@
(defn- mgmt
[type params]
(th/management-command! (assoc params ::th/type type)))
;; The legacy key id, pinned explicitly: the jobs methods declare a
;; predicate instead of a set now, and the shared fallback cannot pick
;; a first id from it.
(th/management-command! (merge {::th/type type
::rpc/auth-key-id :exporter}
params)))
(defn- mk-profile
([] (mk-profile 1))
([i]
;; the created profile is inactive until the email is verified, and
;; a session can only be minted for an owner the login would accept
(th/create-profile* i {:is-active true})))
(defn- get-session-row
[id]
(th/db-get :http-session-v2 {:id id}))
(defn- decode-token
[token]
(session/decode-token {::setup/props (get th/*system* :app.setup/props)} token))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; TESTS
@ -112,20 +143,22 @@
out (mgmt :claim-job {:job-id job-id
:scheduled-at (ct/in-future {:minutes 5})})]
(t/is (nil? (:error out)))
(t/is (= {:action :skip} (:result out)))
(t/is (= {:action :skip :status "new"} (:result out)))
(t/is (= "new" (:status (get-row job-id))))))
(t/deftest claim-job-skips-terminal-and-cancelled-rows
;; the skip says the state of the row that won: a worker that carries
;; a stale payload sees cancelled/aborted, not a plain skip to retry on
(let [completed-id (mk-job {:status "completed"})
cancelled-id (mk-job {:status "cancelled"})
aborted-id (mk-job {:status "aborted"})]
(t/is (= {:action :skip}
(t/is (= {:action :skip :status "completed"}
(:result (mgmt :claim-job {:job-id completed-id
:scheduled-at (ct/now)}))))
(t/is (= {:action :skip}
(t/is (= {:action :skip :status "cancelled"}
(:result (mgmt :claim-job {:job-id cancelled-id
:scheduled-at (ct/now)}))))
(t/is (= {:action :skip}
(t/is (= {:action :skip :status "aborted"}
(:result (mgmt :claim-job {:job-id aborted-id
:scheduled-at (ct/now)}))))
(t/is (= "completed" (:status (get-row completed-id))))
@ -180,7 +213,7 @@
:progress {:stage :pages
:counters {:pages {:current 100 :total 100}}}})]
(t/is (nil? (:error out)))
(t/is (= {:action :skip} (:result out)))
(t/is (= {:action :skip :status "completed"} (:result out)))
(t/is (= [] (get-progresss job-id)))))
(t/deftest report-job-progress-persists-rapid-reports
@ -322,19 +355,19 @@
(t/testing "complete on a terminal row reports skip"
(let [out (mgmt :complete-job {:job-id done-id :result {:x 1}})]
(t/is (nil? (:error out)))
(t/is (= {:action :skip} (:result out)))))
(t/is (= {:action :skip :status "completed"} (:result out)))))
(t/testing "fail on a terminal row reports skip"
(let [out (mgmt :fail-job {:job-id done-id
:error {:type :internal
:code "processing-error"
:hint "bad image"}})]
(t/is (nil? (:error out)))
(t/is (= {:action :skip} (:result out)))))
(t/is (= {:action :skip :status "completed"} (:result out)))))
(t/testing "progress on a terminal row reports skip"
(let [out (mgmt :report-job-progress {:job-id done-id
:progress {:stage :pages :counters {:pages {:current 4 :total 10}}}})]
(t/is (nil? (:error out)))
(t/is (= {:action :skip} (:result out))))))
(t/is (= {:action :skip :status "completed"} (:result out))))))
(let [cfg (make-cfg)
job-id (mk-job {})
_ (jobs/claim cfg job-id (:scheduled-at (th/db-get :job {:id job-id} :id :scheduled-at)))]
@ -393,7 +426,7 @@
:result {:v 2}
:resource-id second})]
(t/is (nil? (:error out)))
(t/is (= {:action :skip} (:result out)))))
(t/is (= {:action :skip :status "completed"} (:result out)))))
(t/testing "the row keeps the resource it was given and no event is added"
(let [row (get-row job-id)]
@ -448,3 +481,563 @@
:attempt 3})]
(t/is (nil? (:error out)))
(t/is (= 3 (:attempt (jobs/decode-job-error (:error (get-row job-id))))))))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; CREATE-JOB-SESSION
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(t/deftest create-job-session-mints-a-session-of-the-owner
(let [profile (mk-profile)
job-id (mk-job {:status "running"
:profile-id (:id profile)})
out (mgmt :create-job-session {:job-id job-id})
result (:result out)]
(t/is (nil? (:error out)))
(t/testing "the answer is a real session of the job owner"
(t/is (uuid? (:session-id result)))
(t/is (string? (:session-token result)))
(let [row (get-session-row (:session-id result))]
(t/is (some? row))
(t/is (= (:id profile) (:profile-id row)))))
(t/testing "the token verifies and names the session and its owner"
(let [claims (decode-token (:session-token result))]
(t/is (some? claims))
(t/is (= (:session-id result) (:sid claims)))
(t/is (= (:id profile) (:uid claims)))
;; the short :exp of a job session: at most an hour by default,
;; never the absolute max-age of a login session
(t/is (pos? (- (inst-ms (:exp claims)) (inst-ms (ct/now)))))
(t/is (pos? (compare (ct/plus (ct/now) (ct/duration {:hours 1}))
(:exp claims))))))))
(t/deftest create-job-session-honours-the-configured-ttl
(with-mocks [mock {:target 'app.config/get-job-session-ttl
:return (ct/duration {:minutes 5})}]
(let [profile (mk-profile)
job-id (mk-job {:status "running"
:profile-id (:id profile)})
out (mgmt :create-job-session {:job-id job-id})
claims (decode-token (get-in out [:result :session-token]))]
(t/is (nil? (:error out)))
(t/is (pos? (- (inst-ms (:exp claims)) (inst-ms (ct/now)))))
(t/is (pos? (compare (ct/plus (ct/now) (ct/duration {:minutes 5}))
(:exp claims)))))))
(t/deftest create-job-session-mints-a-fresh-session-every-call
;; every call is self-contained: a worker retry never waits on a
;; session that a settle may already have closed
(let [profile (mk-profile)
job-id (mk-job {:status "running"
:profile-id (:id profile)})
first (mgmt :create-job-session {:job-id job-id})
second (mgmt :create-job-session {:job-id job-id})]
(t/is (nil? (:error first)))
(t/is (nil? (:error second)))
(t/is (not= (get-in first [:result :session-id])
(get-in second [:result :session-id])))))
(t/deftest create-job-session-refuses-a-job-that-is-not-running
(let [profile (mk-profile)]
(doseq [status ["new" "scheduled" "completed" "failed" "cancelled" "aborted"]]
(let [job-id (mk-job {:status status :profile-id (:id profile)})
out (mgmt :create-job-session {:job-id job-id})]
(t/is (some? (:error out)))
(t/is (= :validation (th/ex-type (:error out))))
(t/is (= :invalid-job-state (th/ex-code (:error out))))
(t/testing "and no session was minted"
(t/is (zero? (count (th/db-exec! ["SELECT id FROM http_session_v2"])))))))))
(t/deftest create-job-session-refuses-an-unknown-job
(let [out (mgmt :create-job-session {:job-id (uuid/next)})]
(t/is (some? (:error out)))
(t/is (= :not-found (th/ex-type (:error out))))
(t/is (= :job-not-found (th/ex-code (:error out))))))
(t/deftest create-job-session-refuses-a-job-without-owner
(let [job-id (mk-job {:status "running"})
out (mgmt :create-job-session {:job-id job-id})]
(t/is (some? (:error out)))
(t/is (= :validation (th/ex-type (:error out))))
(t/is (= :job-without-profile (th/ex-code (:error out))))))
(t/deftest create-job-session-refuses-an-owner-the-login-would-refuse
(let [profile (mk-profile)
job-id (mk-job {:status "running"
:profile-id (:id profile)})]
(t/testing "a blocked owner is a restriction, not a fault"
(th/db-update! :profile {:is-blocked true} {:id (:id profile)})
(let [out (mgmt :create-job-session {:job-id job-id})]
(t/is (= :restriction (th/ex-type (:error out))))
(t/is (= :profile-blocked (th/ex-code (:error out))))))
(t/testing "an inactive owner is refused like the login does"
(th/db-update! :profile {:is-blocked false :is-active false} {:id (:id profile)})
(let [out (mgmt :create-job-session {:job-id job-id})]
(t/is (= :validation (th/ex-type (:error out))))
(t/is (= :profile-not-active (th/ex-code (:error out)))))))
(t/testing "a deleted owner does not exist"
(let [profile (mk-profile 2)
job-id (mk-job {:status "running"
:profile-id (:id profile)})]
(th/db-update! :profile {:deleted-at (ct/now)} {:id (:id profile)})
(let [out (mgmt :create-job-session {:job-id job-id})]
(t/is (= :not-found (th/ex-type (:error out))))
(t/is (= :profile-not-found (th/ex-code (:error out))))))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; COMPLETE-JOB MULTIPART
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defn- artifact-tempfile
"An on-disk artifact, as the multipart parser would leave one for the
server: a path with the bytes of the produced export in it."
[content]
(let [path (fs/create-tempfile :dir tmp/default-tmp-dir
:prefix "test-artifact-")]
(with-open [out (io/output-stream path)]
(io/write* out (.getBytes ^String content "UTF-8")))
path))
(defn- multipart-complete
[job-id path & {:as extra}]
(-> (merge {:job-id job-id
:content {:path path
:filename "result.zip"
:size (fs/size path)
:mtype "application/zip"}}
extra)
(assoc ::th/type :complete-job
::rpc/auth-key-id :exporter)
(th/with-multipart)
(th/management-command!)))
(t/deftest claim-job-skips-a-job-of-another-tenant-like-a-missing-one
(let [job-id (mk-job {:status "new" :tenant "other-tenant"})]
;; indistinguishable from a missing row: a bare skip, no status
(t/is (= {:action :skip}
(:result (mgmt :claim-job {:job-id job-id
:scheduled-at (ct/now)}))))))
(t/deftest claim-job-accepts-the-suffixed-key-of-its-tenant
(let [scheduled-at (ct/now)
job-id (mk-job {:scheduled-at scheduled-at})
tenant-key (keyword (str "exporter:" (cf/get :tenant)))
out (mgmt :claim-job {:job-id job-id
:scheduled-at scheduled-at
::rpc/auth-key-id tenant-key})]
(t/is (nil? (:error out)))
(t/is (= :run (:action (:result out))))))
(t/deftest report-job-progress-skips-a-job-of-another-tenant-without-writing
(let [job-id (mk-job {:status "running" :tenant "other-tenant"})
out (mgmt :report-job-progress {:job-id job-id
:progress {:stage :pages
:counters {:pages {:current 1
:total 2}}}})]
(t/is (nil? (:error out)))
(t/is (= {:action :skip} (:result out)))
(t/testing "no progress event was stored"
(t/is (zero? (count-job-events job-id))))))
(t/deftest create-job-session-rejects-a-job-of-another-tenant-as-missing
(let [job-id (mk-job {:status "running" :tenant "other-tenant"})
out (mgmt :create-job-session {:job-id job-id})]
(t/is (= :not-found (th/ex-type (:error out))))
(t/is (= :job-not-found (th/ex-code (:error out))))))
(t/deftest complete-job-skips-a-job-of-another-tenant-without-storing
(let [profile (mk-profile)
job-id (mk-job {:status "running"
:profile-id (:id profile)
:tenant "other-tenant"})
path (artifact-tempfile "penpot")
session-id (uuid/next)
out (multipart-complete job-id path :session-id session-id)]
(t/is (nil? (:error out)))
;; bare skip, like a missing row: no status to tell tenants apart
(t/is (= {:action :skip} (:result out)))
(t/testing "no artifact was stored and the row is untouched"
(t/is (zero? (count (th/db-exec! ["SELECT id FROM storage_object"]))))
(t/is (= "running" (:status (get-row job-id)))))))
(t/deftest fail-job-skips-a-job-of-another-tenant
(let [job-id (mk-job {:status "running" :tenant "other-tenant"})
out (mgmt :fail-job {:job-id job-id
:error {:type :internal
:code :boom
:hint "boom"}})]
(t/is (nil? (:error out)))
(t/is (= {:action :skip} (:result out)))
(t/is (= "running" (:status (get-row job-id))))))
(t/deftest complete-job-params-form-skips-a-job-of-another-tenant
;; The form without `:content` carries `:result`: a foreign tenant gets
;; a bare skip, so neither the write nor the row's state leaks.
(let [job-id (mk-job {:status "running" :tenant "other-tenant"})
out (mgmt :complete-job {:job-id job-id
:result {:x 1}})]
(t/is (nil? (:error out)))
(t/is (= {:action :skip} (:result out)))
(t/is (= "running" (:status (get-row job-id))))
(t/is (nil? (:completed-at (get-row job-id))))))
(t/deftest complete-job-refuses-an-upload-outside-multipart
;; The upload map names a server path: only a request the multipart
;; parser flagged may carry one. A transit body is refused before any
;; store.
(let [profile (mk-profile)
job-id (mk-job {:status "running"
:profile-id (:id profile)})
path (artifact-tempfile "penpot")
out (mgmt :complete-job {:job-id job-id
:content {:path path
:filename "result.zip"
:size (fs/size path)
:mtype "application/zip"}})]
(t/is (= :validation (th/ex-type (:error out))))
(t/is (= :params-validation (th/ex-code (:error out))))
(t/testing "nothing was stored and the row is untouched"
(t/is (zero? (count (th/db-exec! ["SELECT id FROM storage_object"]))))
(t/is (= "running" (:status (get-row job-id)))))))
(t/deftest complete-job-refuses-an-artifact-path-outside-the-upload-areas
;; Even over multipart, the parser only writes under its own temp
;; areas: anything else (e.g. an internal config file) is refused by
;; the same media check the other upload methods use.
(let [profile (mk-profile)
job-id (mk-job {:status "running"
:profile-id (:id profile)})
path (fs/create-tempfile :prefix "outside-upload-")]
(with-open [out (io/output-stream path)]
(io/write* out (.getBytes "secret" "UTF-8")))
(let [out (multipart-complete job-id path)]
(t/is (= :validation (th/ex-type (:error out))))
(t/is (= :params-validation (th/ex-code (:error out))))
(t/testing "nothing was stored and the row is untouched"
(t/is (zero? (count (th/db-exec! ["SELECT id FROM storage_object"]))))
(t/is (= "running" (:status (get-row job-id))))))))
(t/deftest close-job-session-leaves-a-session-of-another-profile-alone
;; The session id travels in the request: a shared-key holder must not
;; be able to delete another profile's login through a settle.
(let [owner (mk-profile 1)
other (mk-profile 2)
job-id (mk-job {:status "running"
:profile-id (:id owner)})
stranger (session/create-session (get th/*system* :app.http.session/manager)
{:profile-id (:id other)})
out (multipart-complete job-id (artifact-tempfile "penpot")
:session-id (:id stranger))]
(t/is (nil? (:error out)))
(t/is (= {:action :run} (:result out)))
(t/testing "the stranger session is still there"
(t/is (some? (get-session-row (:id stranger)))))))
(t/deftest derive-exporter-key-matches-the-node-vector
;; The same golden value the exporter test pins: a change on either
;; side of the derivation fails here instead of in a deployment.
(t/is (= "LXzydEEZEVAxbiNAzXV7mPSm6wYbrS7Omp4yMZ4gmOY"
(keys/derive-exporter-key "test-secret-key" "default"))))
(t/deftest complete-job-multipart-stores-the-artifact-and-completes
(let [profile (mk-profile)
job-id (mk-job {:status "running"
:profile-id (:id profile)})
path (artifact-tempfile "penpot")
out (multipart-complete job-id path)
row (get-row job-id)
rid (:resource-id row)]
(t/is (nil? (:error out)))
(t/is (= {:action :run} (:result out)))
(t/is (= "completed" (:status row)))
(t/testing "the artifact lives in the job-resource bucket, owned by the profile"
(let [{:keys [metadata]} (first (th/db-exec! ["SELECT metadata
FROM storage_object WHERE id = ?" rid]))
metadata (cond-> metadata (db/pgobject? metadata) db/decode-transit-pgobject)]
(t/is (some? rid))
(t/is (= "job-resource" (:bucket metadata)))
(t/is (= (:id profile) (:profile-id metadata)))))
(t/testing "the result names the artifact and how to download it"
(let [result (:result row)]
(t/is (= "result.zip" (:filename result)))
(t/is (= "application/zip" (:mtype result)))
(t/is (= (fs/size path) (:size result)))
(t/is (str/ends-with? (:resource-uri result) (str "/assets/by-id/" rid)))))))
(t/deftest complete-job-multipart-ignores-legacy-text-fields
;; The part is the single source: an exporter that still sends the old
;; `filename`/`mtype` text fields completes fine, and the part wins.
(let [profile (mk-profile)
job-id (mk-job {:status "running"
:profile-id (:id profile)})
path (artifact-tempfile "penpot")
out (multipart-complete job-id path
:filename "ignored.zip"
:mtype "image/gif")]
(t/is (nil? (:error out)))
(t/is (= {:action :run} (:result out)))
(let [result (:result (get-row job-id))]
(t/is (= "result.zip" (:filename result)))
(t/is (= "application/zip" (:mtype result))))))
(t/deftest complete-job-multipart-closes-the-render-session
(let [profile (mk-profile)
job-id (mk-job {:status "running"
:profile-id (:id profile)})
session-id (get-in (mgmt :create-job-session {:job-id job-id})
[:result :session-id])
path (artifact-tempfile "penpot")
out (multipart-complete job-id path :session-id session-id)]
(t/is (nil? (:error out)))
(t/is (= {:action :run} (:result out)))
(t/testing "the session is gone, deleted after the commit"
(t/is (nil? (get-session-row session-id))))))
(t/deftest complete-job-multipart-refuses-a-mtype-outside-the-allowlist
(let [profile (mk-profile)
job-id (mk-job {:status "running"
:profile-id (:id profile)})
path (artifact-tempfile "penpot")
out (multipart-complete job-id path
:content {:path path
:filename "result.gif"
:size (fs/size path)
:mtype "image/gif"})]
(t/is (some? (:error out)))
(t/is (= :validation (th/ex-type (:error out))))
(t/is (= :media-type-not-allowed (th/ex-code (:error out))))
(t/testing "no artifact was stored and the row is untouched"
(t/is (zero? (count (th/db-exec! ["SELECT id FROM storage_object"]))))
(t/is (= "running" (:status (get-row job-id)))))))
(t/deftest complete-job-multipart-refuses-a-part-without-filename
;; The part is the single source of the download name: a part without
;; one fails the upload schema before anything is stored.
(let [profile (mk-profile)
job-id (mk-job {:status "running"
:profile-id (:id profile)})
path (artifact-tempfile "penpot")
out (mgmt :complete-job {:job-id job-id
:content {:path path
:size (fs/size path)
:mtype "application/zip"}})]
(t/is (some? (:error out)))
(t/is (= :validation (th/ex-type (:error out))))
(t/testing "no artifact was stored and the row is untouched"
(t/is (zero? (count (th/db-exec! ["SELECT id FROM storage_object"]))))
(t/is (= "running" (:status (get-row job-id)))))))
(t/deftest complete-job-multipart-refuses-a-result-over-the-cap
(let [profile (mk-profile)
job-id (mk-job {:status "running"
:profile-id (:id profile)})
path (artifact-tempfile "penpot")]
(with-mocks [mock {:target 'app.config/get-exporter-max-result-size
:return 0}]
(let [out (multipart-complete job-id path)]
(t/is (some? (:error out)))
(t/is (= :validation (th/ex-type (:error out))))
(t/is (= :request-body-too-large (th/ex-code (:error out))))
(t/testing "no artifact was stored and the row is untouched"
(t/is (zero? (count (th/db-exec! ["SELECT id FROM storage_object"]))))
(t/is (= "running" (:status (get-row job-id)))))))))
(t/deftest complete-job-multipart-closes-the-session-on-a-lost-race
(let [profile (mk-profile)
job-id (mk-job {:status "running"
:profile-id (:id profile)})
session-id (get-in (mgmt :create-job-session {:job-id job-id})
[:result :session-id])
path (artifact-tempfile "penpot")]
(with-mocks [mock {:target 'app.jobs/complete
:return 0}]
(let [out (multipart-complete job-id path :session-id session-id)]
(t/is (nil? (:error out)))
(t/is (= :skip (-> out :result :action)))
(t/testing "the session is gone even though the write lost"
(t/is (nil? (get-session-row session-id))))
(t/testing "the stored artifact was released, the row untouched"
(t/is (nil? (:resource-id (get-row job-id))))
(t/is (= "running" (:status (get-row job-id)))))))))
(t/deftest complete-job-multipart-of-a-terminal-job-skips-and-stores-nothing
(let [profile (mk-profile)
job-id (mk-job {:status "cancelled"
:profile-id (:id profile)})
path (artifact-tempfile "penpot")
out (multipart-complete job-id path)]
(t/is (nil? (:error out)))
;; the bytes are not used: the completion lost the race with the
;; cancel, and no object is left for the storage GC to find
(t/is (= {:action :skip :status "cancelled"} (:result out)))
(t/is (zero? (count (th/db-exec! ["SELECT id FROM storage_object"]))))))
(t/deftest fail-job-closes-the-render-session
(let [profile (mk-profile)
job-id (mk-job {:status "running"
:profile-id (:id profile)})
session-id (get-in (mgmt :create-job-session {:job-id job-id})
[:result :session-id])
out (mgmt :fail-job {:job-id job-id
:error {:type :internal
:code "processing-error"
:hint "bad image"}
:session-id session-id})]
(t/is (nil? (:error out)))
(t/is (= {:action :run} (:result out)))
(t/is (= "failed" (:status (get-row job-id))))
(t/is (nil? (get-session-row session-id)))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; WIRE CONTRACT: what the exporter actually receives and sends
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; The claim carries the row's stored shapes (plain JSON: every id a
;; string, every enum its name). The exporter normalizes them to typed
;; values. These tests pin the backend half of that contract: the claim
;; returns stored shapes untouched, the dispatcher bytes are exact, and
;; the HTTP edges (auth, dispatch, body parsing) run production code.
(t/deftest claim-job-returns-decoded-typed-params
;; The row stores JSON (strings); the claim decodes with the row's
;; job-def, so the worker gets typed values (uuids, keywords) that
;; transit carries natively. A row whose job-def is gone answers
;; untouched (see the media-process claims above).
(let [file-id (uuid/next)
object-id (uuid/next)
scheduled-at (ct/now)
job-id (mk-job {:name "export-assets"
:scheduled-at scheduled-at
:params {:exports [{:file-id (str file-id)
:page-id (str (uuid/next))
:object-id (str object-id)
:type "png"
:name "Board"
:suffix ""
:scale 2}]
:name "the export"}})
out (mgmt :claim-job {:job-id job-id
:scheduled-at scheduled-at})
params (get-in out [:result :params])]
(t/is (nil? (:error out)))
(t/is (= "export-assets" (get-in out [:result :name])))
(t/testing "ids travel as typed uuids"
(t/is (= file-id (get-in params [:exports 0 :file-id])))
(t/is (= object-id (get-in params [:exports 0 :object-id]))))
(t/testing "enums travel as keywords"
(t/is (= :png (get-in params [:exports 0 :type]))))))
(t/deftest claim-params-decode-to-typed-values-with-the-job-def
;; The row stores JSON (strings); the decoder the runner would use
;; turns them into the typed values the exporter validates against.
;; The claim path returns the stored shapes (see above); this pins
;; the decode half both sides rely on.
(let [file-id (uuid/next)
object-id (uuid/next)
stored {:exports [{:file-id (str file-id)
:page-id (str (uuid/next))
:object-id (str object-id)
:type "png"
:name "Board"
:suffix ""
:scale 2}]
:name "the export"}
decoded (jobs/decode-params
(jobs/get-job-def (jobs/get-defs (make-cfg)) :export-assets)
stored)]
(t/is (= file-id (get-in decoded [:exports 0 :file-id])))
(t/is (= object-id (get-in decoded [:exports 0 :object-id])))
(t/is (= :png (get-in decoded [:exports 0 :type])))
(t/is (= "the export" (:name decoded)))))
(t/deftest shared-key-middleware-allows-the-exporter-key
(let [secret "test-secret"
derived (keys/derive-exporter-key secret "acme")
handler (#'mw/wrap-shared-key-auth (fn [request] request)
{:exporter "test-key"}
secret)
allowed (th/make-dummy-request :method :post
:headers {"x-shared-key" "exporter test-key"})
denied (th/make-dummy-request :method :post
:headers {"x-shared-key" "exporter wrong-key"})
missing (th/make-dummy-request :method :post)
tenant (th/make-dummy-request :method :post
:headers {"x-shared-key" (str "exporter:acme " derived)})
foreign (th/make-dummy-request :method :post
:headers {"x-shared-key" "exporter:acme wrong-key"})]
(t/testing "the declared key reaches the handler with its id"
(let [request (handler allowed)]
(t/is (= :exporter (:app.http/auth-key-id request)))))
(t/testing "a wrong key is a 403"
(t/is (= 403 (:yetti.response/status (handler denied)))))
(t/testing "a missing key is a 403"
(t/is (= 403 (:yetti.response/status (handler missing)))))
(t/testing "a per-tenant id the secret derives reaches the handler too"
(let [request (handler tenant)]
(t/is (= :exporter:acme (:app.http/auth-key-id request)))))
(t/testing "a per-tenant id with a wrong key is a 403"
(t/is (= 403 (:yetti.response/status (handler foreign)))))))
(t/deftest management-dispatch-routes-by-method-name
(t/testing "the stub method runs with the multipart flag of the request"
(let [handler (rpc/make-rpc-handler
{:ping [{} (fn [data]
(select-keys data [:app.rpc/auth-key-id
:app.rpc/is-multipart]))]})
request (-> (th/make-dummy-request
:method :post
:headers {"content-type" "multipart/form-data; boundary=test"})
(assoc :path-params {:method-name "ping"}
:params {}
:app.http/auth-key-id :exporter))
response (handler request)]
(t/is (= 200 (:yetti.response/status response)))
(t/is (true? (:app.rpc/is-multipart (:yetti.response/body response))))
(t/is (= :exporter (:app.rpc/auth-key-id (:yetti.response/body response))))))
(t/testing "the real claim method resolves through the same dispatch"
(let [scheduled-at (ct/now)
job-id (mk-job {:scheduled-at scheduled-at})
handler (rpc/make-rpc-handler
(get th/*system* :app.rpc/management-methods))
request (-> (th/make-dummy-request
:method :post
:headers {"content-type" "application/transit+json"})
(assoc :path-params {:method-name "claim-job"}
:params {:job-id job-id :scheduled-at scheduled-at}
:app.http/auth-key-id :exporter))
response (handler request)
body (:yetti.response/body response)]
(t/is (= 200 (:yetti.response/status response)))
(t/is (= :run (:action body)))
(t/is (= "media-process" (:name body))))))
(t/deftest management-body-parses-transit-by-content-type
(let [payload {:job-id (uuid/next)
:scheduled-at (ct/inst "2026-10-07T09:00:00Z")}
request (th/make-dummy-request
:method :post
:headers {"content-type" "application/transit+json"}
:body-bytes (transit/encode-str payload))]
(t/testing "a transit body decodes to the params map"
(t/is (= payload (#'mw/read-body-params
request nil "application/transit+json")))))
(t/testing "a broken body raises malformed, not a 500"
(let [request (th/make-dummy-request
:method :post
:headers {"content-type" "application/transit+json"}
:body-bytes "{nope")
metrics (get th/*system* :app.metrics/metrics)]
(try
(#'mw/read-body-params request metrics "application/transit+json")
(t/is false "should have raised")
(catch clojure.lang.ExceptionInfo cause
(t/is (= :validation (th/ex-type cause)))
(t/is (= :malformed-json (th/ex-code cause))))))))

View File

@ -29,11 +29,21 @@
"every management namespace must be scanned")
(doseq [[method-name [mdata _]] methods]
(let [ns-name (:ns mdata)
expected (get expected-shared-keys ns-name)]
expected (get expected-shared-keys ns-name)
allowed (::rpc/allowed-shared-keys mdata)]
(t/is (some? expected)
(str "unexpected management namespace for " method-name ": " ns-name))
(t/is (= expected (::rpc/allowed-shared-keys mdata))
(str "wrong ::rpc/allowed-shared-keys for " method-name " in " ns-name))))))
(if (= "app.rpc.management.jobs" ns-name)
;; The jobs methods declare a predicate: the bare legacy id
;; plus any `exporter:<tenant>` suffix.
(do
(t/is (fn? allowed))
(t/is (allowed :exporter))
(t/is (allowed :exporter:acme))
(t/is (not (allowed :nexus)))
(t/is (not (allowed nil))))
(t/is (= expected allowed)
(str "wrong ::rpc/allowed-shared-keys for " method-name " in " ns-name)))))))
(t/deftest management-command-uses-the-key-from-the-request-metadata
(let [out (th/management-command!

View File

@ -325,3 +325,15 @@
(t/is (= "running" (:status row)))
(t/is (nil? (:error row)))
(t/is (= [] (:events row)))))))
(t/deftest dispatcher-payload-bytes-are-the-exporter-contract
;; The only bytes the exporter parses: a JSON array of exactly two
;; items, [job-id scheduled-at], the instant in ct/format-inst shape.
;; The worker echoes scheduled-at back byte-identical on claim, so any
;; drift here breaks the claim-back rule (and the exporter test pins
;; the same literal from its side).
(let [payload (#'wdisp/encode-payload
{:id (uuid/parse "03b7300f-11e6-8131-8008-e4b09b780a0e")
:scheduled-at (ct/inst "2026-10-07T09:00:00Z")})]
(t/is (= "[\"03b7300f-11e6-8131-8008-e4b09b780a0e\",\"2026-10-07T09:00:00Z\"]"
payload))))

View File

@ -0,0 +1,56 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.common.export
"The shapes of an asset export job, shared by its producer (the
backend, which freezes them in the job row) and its consumer (the
exporter worker, which renders them).
One source of truth on purpose: the row stores plain JSON (every id
and enum a string) and each side decodes with the repo transformer
(`sm/json-transformer`), so a drift between what the backend writes
and what the exporter reads fails in tests instead of in production."
(:require
[app.common.schema :as sm]))
(def schema:export-item
"One thing to render: where it lives (file, page, object), what to
produce of it (type, scale) and what to call it. The same item shape
the legacy `/api/export` surface received, frozen in the params of
the job; the optional `share-id` marks the item as coming from a
public share."
[:map {:title "export-item" :closed true}
[:file-id ::sm/uuid]
[:page-id ::sm/uuid]
[:object-id ::sm/uuid]
[:type [::sm/one-of #{:png :jpeg :webp :svg :pdf}]]
[:name ::sm/text]
;; a plain string on purpose, empty included: the export form sends
;; `""` when the user types no suffix, and the legacy `/api/export`
;; surface took it that way; the worker concatenates it as-is
[:suffix {:optional true} :string]
[:scale ::sm/number]
[:share-id {:optional true} ::sm/uuid]])
(def schema:export-items
"The items of one assets run, as the exporter validates them after
normalizing the claim."
[:vector {:min 1} schema:export-item])
(def schema:params
"Business params of an asset export: the items to render, frozen when
the job was created, plus the options that shape the output. There is
no `wait` here: a job is followed by its id, not waited on. The
optional `kind` declares the export the caller asked for (`:frames`
joins pages into one pdf); absent it is a shapes export. The worker
reads it as declared instead of inferring it from the items."
[:map {:title "export-assets-params" :closed true}
[:exports [:vector {:min 1} schema:export-item]]
[:kind {:optional true} [::sm/one-of #{:shapes :frames}]]
[:name {:optional true} ::sm/text]
[:skip-children {:optional true} ::sm/boolean]
[:force-multiple {:optional true} ::sm/boolean]
[:is-wasm {:optional true} ::sm/boolean]])

View File

@ -25,6 +25,18 @@
(def tempfile-types
(conj image-types "application/pdf" "application/zip"))
(def export-artifact-types
"The content types an export job may leave as its result artifact:
what the exporter produces, nothing more. The `complete-job` call of
the management API refuses any other type for the bytes a worker
sends."
#{"application/zip"
"application/pdf"
"image/png"
"image/jpeg"
"image/webp"
"image/svg+xml"})
(def storage-object-types
"Every content-type the system stores in storage objects: uploaded
images and fonts, generated thumbnails, temporary files and the

View File

@ -0,0 +1,62 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns common-tests.export-test
"The asset-export shapes both runtimes share: stored JSON (every id
and enum a string) decodes to typed values with the repo transformer,
and typed values decode to themselves. Runs on JVM and JS because the
backend writes the row and the exporter reads the claim."
(:require
[app.common.export :as export]
[app.common.schema :as sm]
[clojure.test :as t]))
(def ^:private stored-item
"One item as the job row holds it: plain JSON, strings everywhere."
{:file-id "c9c0b00d-4f50-8024-8008-d07a282a13ee"
:page-id "c9c0b00d-4f50-8024-8008-d07a282a13ef"
:object-id "1ac0612f-1ee8-8023-8008-d07a2aa96040"
:type "png"
:name "Board"
:suffix ""
:scale 2})
(def ^:private decode
(sm/decoder export/schema:export-item sm/json-transformer))
(t/deftest stored-strings-decode-to-typed-values
(let [item (decode stored-item)]
(t/testing "ids decode to uuids"
(t/is (uuid? (:file-id item)))
(t/is (uuid? (:page-id item)))
(t/is (uuid? (:object-id item)))
(t/is (= #uuid "1ac0612f-1ee8-8023-8008-d07a2aa96040"
(:object-id item))))
(t/testing "the type decodes to a keyword"
(t/is (= :png (:type item))))
(t/testing "plain values ride through untouched"
(t/is (= "Board" (:name item)))
(t/is (= "" (:suffix item)))
(t/is (= 2 (:scale item))))))
(t/deftest typed-values-decode-to-themselves
(let [typed {:file-id #uuid "c9c0b00d-4f50-8024-8008-d07a282a13ee"
:page-id #uuid "c9c0b00d-4f50-8024-8008-d07a282a13ef"
:object-id #uuid "1ac0612f-1ee8-8023-8008-d07a2aa96040"
:type :png
:name "Board"
:suffix ""
:scale 2}]
(t/is (= typed (decode typed)))))
(t/deftest params-schema-owns-kind-and-options
(let [decode-params (sm/decoder export/schema:params sm/json-transformer)
params (decode-params {:exports [stored-item]
:kind "frames"
:name "the export"})]
(t/is (= :frames (:kind params)))
(t/is (= "the export" (:name params)))
(t/is (= 1 (count (:exports params))))))

View File

@ -18,6 +18,7 @@
[common-tests.colors-test]
[common-tests.data-test]
[common-tests.exceptions-test]
[common-tests.export-test]
[common-tests.files-builder-test]
[common-tests.files-changes-test]
[common-tests.files-migrations-0025-test]
@ -126,6 +127,7 @@
'common-tests.colors-test
'common-tests.data-test
'common-tests.exceptions-test
'common-tests.export-test
'common-tests.files-builder-test
#?(:cljs 'common-tests.fonts-test)
'common-tests.files-changes-test

View File

@ -23,6 +23,13 @@ PENPOT_USER_DATA_VOLUME=penpotdev_user_data
# database number (0 for ws0, 1 for ws1, ...) set by instance-env-overrides.
PENPOT_HOST=devenv
PENPOT_PUBLIC_URI=https://localhost:3449
# In-container address of the platform edge (nginx), used to reach the
# backend without leaving through the public URI (which is host-side and
# unreachable from inside the container). Caddy only emulates the external
# HTTPS layer; no service goes outside to come back in. The exporter worker
# uses it for the management API; per-container loopback, so ws1+ overrides
# never touch it.
PENPOT_INTERNAL_URI=http://localhost:4449
PENPOT_DATABASE_URI=postgresql://postgres/penpot
PENPOT_DATABASE_USERNAME=penpot
PENPOT_DATABASE_PASSWORD=penpot

View File

@ -137,10 +137,6 @@ http {
}
# On production, this is controlled by ELB
location /api/export {
proxy_pass http://127.0.0.1:6061;
}
location = /.well-known/oauth-authorization-server/api/mcp/oauth {
proxy_pass http://127.0.0.1:6060;
proxy_http_version 1.1;

View File

@ -225,6 +225,9 @@ services:
# PENPOT_MCP_BACKEND_URI: http://penpot-backend:6060/
penpot-exporter:
# The exporter is a consumer's process: it polls the backend's
# `:exporter` job queue through valkey and calls the backend's
# management API to claim, report and settle every export job.
image: "penpotapp/exporter:${PENPOT_VERSION:-2.18}"
restart: always
@ -239,18 +242,17 @@ services:
<< : [*penpot-flags, *penpot-secret-key, *penpot-public-uri]
# Don't touch it; this uses an internal docker network to
# communicate with the frontend.
# reach the backend's management API and the frontend of the
# assets the exports render.
PENPOT_INTERNAL_URI: http://penpot-frontend:8080
## Valkey (or previously Redis) is used for the websockets notifications
## and for storing the state export jobs
## Valkey (or previously Redis) carries the export jobs queue the
## backend dispatcher pushes to, plus the websockets
## notifications
PENPOT_REDIS_URI: redis://penpot-valkey/0
# PENPOT_EXPORTER_MAX_CONCURRENT_JOBS: 4
# PENPOT_EXPORTER_MAX_JOBS_PER_PROFILE: 2
# PENPOT_EXPORTER_QUEUE_MAX: 64
# PENPOT_EXPORTER_JOB_TTL: 3600
# PENPOT_WASM_WORKER_POOL_MAX: 2
# PENPOT_EXPORTER_WORKER_CONCURRENCY: 2
# PENPOT_WASM_WORKER_POOL_MIN: 1
# PENPOT_WASM_WORKER_IDLE_TIMEOUT: 300
# PENPOT_WASM_WORKER_IMAGE_CACHE_SIZE: 134217728

View File

@ -69,13 +69,12 @@ else
fi
export PENPOT_BACKEND_URI=${PENPOT_BACKEND_URI:-http://penpot-backend:6060}
export PENPOT_EXPORTER_URI=${PENPOT_EXPORTER_URI:-http://penpot-exporter:6061}
export PENPOT_HTTP_SERVER_MAX_BODY_SIZE=${PENPOT_HTTP_SERVER_MAX_BODY_SIZE:-367001600} # Default to 350MiB
export PENPOT_IPV6_LISTEN_DIRECTIVE=${PENPOT_IPV6_LISTEN_DIRECTIVE:-"listen [::]:8080 default_server reuseport backlog=16384;"}
if is_truthy "${PENPOT_DISABLE_IPV6_LISTEN:-}"; then
export PENPOT_IPV6_LISTEN_DIRECTIVE=""
fi
envsubst "\$PENPOT_BACKEND_URI,\$PENPOT_EXPORTER_URI,\$PENPOT_HTTP_SERVER_MAX_BODY_SIZE,\$PENPOT_IPV6_LISTEN_DIRECTIVE,\$PENPOT_PUBLIC_PATH,\$PENPOT_PUBLIC_PATH_DIRECTIVES" \
envsubst "\$PENPOT_BACKEND_URI,\$PENPOT_HTTP_SERVER_MAX_BODY_SIZE,\$PENPOT_IPV6_LISTEN_DIRECTIVE,\$PENPOT_PUBLIC_PATH,\$PENPOT_PUBLIC_PATH_DIRECTIVES" \
< /tmp/nginx.conf.template > /etc/nginx/nginx.conf
if [[ $PENPOT_FLAGS == *"enable-admin-console"* ]]; then

View File

@ -150,11 +150,6 @@ http {
proxy_pass $PENPOT_BACKEND_URI/link-preview$is_args$args;
}
location /api/export {
proxy_set_header Host $proxy_host;
proxy_pass $PENPOT_EXPORTER_URI;
}
location = /.well-known/oauth-authorization-server/api/mcp/oauth {
proxy_set_header Host $proxy_host;
proxy_pass $PENPOT_BACKEND_URI$request_uri;

View File

@ -898,15 +898,14 @@ them.
In case you have a custom setup, you probably need to configure the following environment
variables on the frontend container:
To connect the frontend to the exporter and backend, you need to fill out these environment variables.
To connect the frontend to the backend, you need to fill out this environment variable.
```bash
# Frontend
PENPOT_BACKEND_URI: http://your-penpot-backend:6060
PENPOT_EXPORTER_URI: http://your-penpot-exporter:6061
```
These variables are used for generate correct nginx.conf file on container startup.
This variable is used to generate a correct nginx.conf file on container startup.
### Exporter
@ -917,9 +916,10 @@ The exporter uses this variable:
PENPOT_INTERNAL_URI: http://penpot-frontend:8080
```
- `PENPOT_INTERNAL_URI`: The URI used by the exporter's headless browser to
communicate with the frontend (internal Docker network). Defaults to
`PENPOT_PUBLIC_URI` if not set. The default value
- `PENPOT_INTERNAL_URI`: The URI the exporter uses to reach the backend
management API, where it claims, reports and settles its export jobs, and
the frontend that serves the assets it renders (internal Docker network).
Defaults to `PENPOT_PUBLIC_URI` if not set. The default value
`http://penpot-frontend:8080` used in the docker-compose is a good default and
it is recommended to keep it unchanged.

View File

@ -1,100 +1,74 @@
# Exporter
Node service that renders shapes and files to bitmap, SVG and PDF. Wasm exports
are **jobs**: created over HTTP, admitted by a scheduler with bounded
concurrency, and persisted in Redis so their state can be queried and cancelled.
The legacy entry point, which is what the browser backend still goes through,
runs the export as soon as it is asked for, with no admission control.
Node process that renders shapes, frames and files to bitmap, SVG and PDF. It
is a **consumer**: it polls the backend's `:exporter` job queue on Redis and
drives each export through the backend's management API — claim, report the
milestones, and settle (`complete-job` multipart with the artifact, or
`fail-job` on error). There is no HTTP surface of its own and nothing is
persisted on this side: the row of the job, its cancel button and its result
all live in the backend.
## HTTP API
## How a job runs
Mounted under `/api/export` (the router matches on the path *after* that prefix,
so it also works when the process is hit directly on `/`).
1. The backend dispatcher pushes `[job-id, scheduled-at]` (JSON) into
`penpot.worker.queue:<tenant>:exporter`.
2. One of the `K` pollers pops it and owns that connection for the whole run:
the blocking pop is both the semaphore of the slot and the backpressure, so
the process never runs more exports than pollers. `K` is
`PENPOT_EXPORTER_WORKER_CONCURRENCY` (default 2, and the browser and wasm
pools size themselves to the same figure).
3. The poller claims the job over the management API and, on `:run`, mints a
render session (`create-job-session`) through which every asset fetch
happens as the owner of the job.
4. The milestones are reported as beats (`:preparing`, `:rendering`,
`:packaging`, with the `objects` and `pages` counters). A watchdog re-sends
the last one every second, whatever the run is doing.
5. It settles with `complete-job` (the artifact rides multipart, the session
is closed in the same step) or `fail-job`. The settle never rejects: a
worker that cannot talk leaves the row to the backend's lease GC.
| Method | Path | Description |
|----------|-----------------|----------------------------------------------------|
| `POST` | `/` | Legacy command multiplex; runs unscheduled |
| `POST` | `/jobs` | Create an export job |
| `GET` | `/jobs/{id}` | Job record |
| `DELETE` | `/jobs/{id}` | Request cancellation |
## Cancellation
Job states: `queued` -> `running` -> `ended` | `error` | `cancelled`. The last
three are terminal.
A cancellation reaches the worker through the beats: a `skip` answer means the
backend no longer listens (the row is terminal), so the runner stops between
units of work; mid-Skia the watchdog terminates the leased render worker. The
row was already `cancelled` by the backend's `cancel-job`.
## Redis layout
## Running it
Every key is namespaced with `penpot.exporter.` plus the tenant
(`PENPOT_TENANT`, `default` in code but set to the workspace name in devenv,
e.g. `devenv-ws0`).
Local (devenv): `./scripts/setup` once (installs deps + Playwright
chromium), then `pnpm run watch` in one pane (builds render-wasm and
recompiles on change) and `pnpm run start:dev` in another (loads the
backend dev env and runs `target/app.js`). Code changes arrive through
shadow hot-reload: `on-before-load` halts the running system and
`on-after-load` reboots it in the same process. The tmux `exporter`
window does the same split (`./scripts/watch` +
`./scripts/wait-and-start.sh`).
```
penpot.exporter.{tenant}.job.{job-id} hash field: data (transit blob of the
whole record)
penpot.exporter.{tenant}.job-cancel pubsub payload: the job id, one line
```
There is no index: the keyspace is one self-expiring hash per job and nothing
else. Each hash carries the same TTL as the exported file
(`PENPOT_EXPORTER_JOB_TTL`, default 3600s), refreshed on every write and never
after the job settles.
## Inspecting Redis
Redis is not published on the host, so `redis-cli` from your machine gets
connection refused. Run it **inside the devenv container**, against the `valkey`
host on database 0:
```bash
redis-cli -h valkey -n 0
```
`redis-cli -u "$PENPOT_REDIS_URI"` does the same and follows whatever the env is
set to (`redis://valkey/0` in devenv).
Keys carry the tenant, which in devenv is the **workspace name**
(`$PENPOT_TENANT`, e.g. `devenv-ws0`), not `default`. From the prompt:
```
# every job record
KEYS penpot.exporter.devenv-ws0.job.*
# the whole record, transit-json in the `data` field
HGET penpot.exporter.devenv-ws0.job.<job-id> data
# seconds left before the record expires
TTL penpot.exporter.devenv-ws0.job.<job-id>
# watch cancellations as they are published (blocks the connection)
SUBSCRIBE penpot.exporter.devenv-ws0.job-cancel
# drop one record
DEL penpot.exporter.devenv-ws0.job.<job-id>
```
`KEYS` is fine here -- the keyspace is a handful of job hashes. On a real
deployment use `SCAN 0 MATCH penpot.exporter.<tenant>.job.* COUNT 100` instead.
Do not `FLUSHDB`: the backend shares this database.
The backend debug UI also renders these records: `/dbg` has an *Export jobs*
section, with a `?job-id=` filter.
`pnpm run start` runs the compiled `target/app.js` with the env as-is
(the same `node target/app.js` the container image uses as `CMD`). The
release bundle is `./scripts/build <version>` (wasm + shadow release +
version stamp + generated `setup`); `./manage.sh build-exporter-bundle`
packs it for the image.
## Configuration
| Variable | Default | Description |
|---------------------------------------|---------|--------------------------------------|
| `PENPOT_REDIS_URI` | `redis://redis/0` | Job store and cancel topic |
| `PENPOT_TENANT` | `default` | Key and topic prefix |
| `PENPOT_EXPORTER_JOB_TTL` | `3600` | Lifetime of a job record, in seconds |
| `PENPOT_EXPORTER_MAX_CONCURRENT_JOBS` | `4` | Admission limit |
| `PENPOT_EXPORTER_MAX_JOBS_PER_PROFILE`| `2` | Per-profile admission limit |
| `PENPOT_EXPORTER_QUEUE_MAX` | `64` | Queue cap; over it, `429 :queue-full` |
| `PENPOT_EXPORTER_REPLICA_ID` | hostname | Names this replica; boot cleanup only cancels its own jobs |
| `PENPOT_WASM_WORKER_POOL_MAX` | `2` | Headless render worker threads; min 1 |
| `PENPOT_WASM_WORKER_POOL_MIN` | `1` | Workers kept warm; clamped to the max |
| `PENPOT_WASM_WORKER_IDLE_TIMEOUT` | `300` | Silence before a worker is terminated, in seconds |
| `PENPOT_WASM_WORKER_IMAGE_CACHE_SIZE` | `134217728` | Per-worker image cache budget, in bytes |
| Variable | Default | Description |
|------------------------------------------|---------|----------------------------------------------------|
| `PENPOT_PUBLIC_URI` | — | The frontend the assets render through |
| `PENPOT_INTERNAL_URI` | `public-uri` | Same network's frontend; the management API too |
| `PENPOT_REDIS_URI` | `redis://redis/0` | The queue and the rest of redis |
| `PENPOT_TENANT` | `default` | Queue prefix (workspace name in devenv) |
| `PENPOT_TEMPDIR` | `/tmp/penpot` | Temp area owned by the tmpdir service |
| `PENPOT_SECRET_KEY` | — | Required; the management key derives from it when no shared key is set |
| `PENPOT_EXPORTER_SHARED_KEY` | — | The shared key; derived from the secret otherwise |
| `PENPOT_EXPORTER_JOB_TTL` | `3600` | Lifetime of the temp files a job owns, in seconds |
| `PENPOT_EXPORTER_WORKER_CONCURRENCY` | `2` | `K`: pollers, running exports, pool sizes |
| `PENPOT_WASM_WORKER_POOL_MIN` | `1` | Render workers kept warm; clamped to the max |
| `PENPOT_WASM_WORKER_IDLE_TIMEOUT` | `300` | Silence before a worker is terminated, in seconds |
| `PENPOT_WASM_WORKER_IMAGE_CACHE_SIZE` | `134217728` | Per-worker image cache budget, in bytes |
A headless job leases one render worker for its whole run, so it is admitted
only when a worker is free: `PENPOT_WASM_WORKER_POOL_MAX` is the real limit for
them, and `PENPOT_EXPORTER_MAX_CONCURRENT_JOBS` bounds the browser ones
alongside.
## Inspecting the backend
Nothing about a job is stored here: `get-job` in the backend answers the full
row, and its `app.debug`/admin surfaces render the same records.

View File

@ -32,9 +32,11 @@
"clear:shadow-cache": "rm -rf .shadow-cljs && rm -rf target",
"watch:app": "pnpm run build:wasm && pnpm run clear:shadow-cache && clojure -M:dev:shadow-cljs watch main",
"watch": "pnpm run watch:app",
"start": "node target/app.js",
"start:dev": "./scripts/run",
"build:app": "clojure -M:dev:shadow-cljs release main",
"build:wasm": "../render-wasm/build export",
"build": "pnpm run clear:shadow-cache && pnpm run build:app",
"build": "pnpm run build:wasm && pnpm run clear:shadow-cache && pnpm run build:app",
"fmt:clj": "cljfmt fix --parallel=true src/ test/",
"check-fmt:clj": "cljfmt check --parallel=true src/ test/",
"lint:clj": "clj-kondo --fail-level error --parallel --lint src/ test/",

View File

@ -6,10 +6,12 @@ export CURRENT_VERSION=$1;
export NODE_ENV=production;
pnpm install || exit 1;
pnpm run build:wasm;
# Build the application (wasm artifact first, then the shadow release)
pnpm run build;
WASM_SRC="resources/wasm";
WASM_SHARED="src/app/wasm/shared.js";
WASM_SHARED="src/exporter/wasm/shared.js";
if [ ! -f "$WASM_SRC/render-wasm.wasm" ] || [ ! -f "$WASM_SHARED" ]; then
echo "ERROR: the render-wasm build did not produce:" >&2;
echo " $WASM_SRC/render-wasm.wasm" >&2;
@ -17,11 +19,6 @@ if [ ! -f "$WASM_SRC/render-wasm.wasm" ] || [ ! -f "$WASM_SHARED" ]; then
exit 1;
fi
rm -rf target
# Build the application
pnpm run build;
cp pnpm-lock.yaml target/;
cp pnpm-workspace.yaml target/;
cp package.json target/;

View File

@ -7,4 +7,9 @@ if [ -f $SCRIPT_DIR/../../backend/scripts/_env.local ]; then
source $SCRIPT_DIR/../../backend/scripts/_env.local;
fi
# Same default as wait-and-start.sh: the platform edge is always the
# in-container nginx loopback, so it is defined at the start scripts,
# not in compose or _env; an explicitly exported value always wins.
export PENPOT_INTERNAL_URI="${PENPOT_INTERNAL_URI:-http://localhost:4449}"
exec node target/app.js

View File

@ -24,6 +24,13 @@ progress("Running tests...");
const result = spawnSync(
"node",
["target/tests/test.js", ...process.argv.slice(2)],
{ stdio: "inherit" },
);
{
stdio: "inherit",
// the config of the process needs a secret to even boot: same
// default the `test` script uses when the environment says nothing
env: {
...process.env,
PENPOT_SECRET_KEY: process.env.PENPOT_SECRET_KEY || "test-secret-key",
},
});
process.exit(result.status ?? 1);

View File

@ -7,6 +7,11 @@ if [ -f $SCRIPT_DIR/../../backend/scripts/_env.local ]; then
source $SCRIPT_DIR/../../backend/scripts/_env.local;
fi
# In-container address of the platform edge (nginx). Invariant across
# instances (per-container loopback), so it lives here, not in compose
# or _env; an explicitly exported value always wins.
export PENPOT_INTERNAL_URI="${PENPOT_INTERNAL_URI:-http://localhost:4449}"
bb -i '(babashka.wait/wait-for-port "localhost" 9630)';
bb -i '(babashka.wait/wait-for-path "target/app.js")';
sleep 2;

View File

@ -6,13 +6,11 @@
{:target :esm
:runtime :node
:output-dir "target/"
:devtools {:before-load-async app.core/stop
:after-load app.core/start}
:modules
{:app
{:entries []
:init-fn app.core/main}}
:init-fn exporter.main/main}}
:js-options
{:entry-keys ["module" "browser" "main"]

View File

@ -1,98 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.auth
"Resolves the caller's session cookie to a real profile id.
The export commands take `:profile-id` from the request body, which was
harmless while it only picked a pub/sub topic. The job API can read and
cancel other people's work, so its ownership comes from the session: the
token goes to the backend's `get-profile` command, which answers with the
anonymous profile (`uuid/zero`) when it is not a valid session.
Results are memoized briefly, so a burst of export calls from one client is
one round trip rather than one per request."
(:require
["undici" :as http]
[app.common.exceptions :as ex]
[app.common.logging :as l]
[app.common.transit :as t]
[app.common.uri :as u]
[app.common.uuid :as uuid]
[app.config :as cf]
[promesa.core :as p]))
(def ^:private cache-ttl-ms 60000)
(defonce ^:private cache (atom {}))
(defn- put-in-cache
"Stores the resolution by session token and removes expired entries.
Without cleanup, long-lived exporters accumulate stale entries"
[cache token profile-id now]
(-> (into {} (remove (fn [[_ {:keys [expires-at]}]] (<= expires-at now))) cache)
(assoc token {:profile-id profile-id
:expires-at (+ now cache-ttl-ms)})))
(defn- rpc-uri
[]
(-> (cf/get-internal-uri)
(u/ensure-path-slash)
(u/join "api/rpc/command/get-profile")
(str)))
(defn- fetch-profile-id
[token]
(let [uri (rpc-uri)
headers #js {"Content-Type" "application/transit+json"
"X-Shared-Key" (str "exporter " cf/management-key)
"Cookie" (str "auth-token=" token)}]
(->> (p/do (http/fetch uri #js {:method "POST" :headers headers :body (t/encode-str {})}))
(p/mcat (fn [^js resp]
(if (= 200 (.-status resp))
(.text resp)
(p/resolved nil))))
(p/fmap (fn [body]
(some-> body t/decode-str :id)))
(p/merr (fn [cause]
(l/warn :hint "unable to resolve session profile" :uri uri :cause cause)
(p/resolved nil))))))
(defn resolve-profile-id
"Promise of the authenticated profile id, or nil for an anonymous or absent
session."
[token]
(if (nil? token)
(p/resolved nil)
(let [{:keys [profile-id expires-at]} (get @cache token)]
(if (and expires-at (> expires-at (js/Date.now)))
(p/resolved profile-id)
(->> (fetch-profile-id token)
(p/fmap (fn [profile-id]
(let [profile-id (when (and profile-id (not= uuid/zero profile-id)) profile-id)]
(swap! cache put-in-cache token profile-id (js/Date.now))
profile-id))))))))
(defn require-profile-id
"Like `resolve-profile-id`, but rejects anonymous callers."
[token]
(->> (resolve-profile-id token)
(p/mcat (fn [profile-id]
(if profile-id
(p/resolved profile-id)
(ex/raise :type :authentication
:code :authentication-required
:hint "no valid session for this request"))))))
(defn check-owner!
"Raises unless `profile-id` owns `job`."
[job profile-id]
(when (or (nil? job)
(not= (str (:profile-id job)) (str profile-id)))
(ex/raise :type :not-found
:code :object-not-found
:hint "job does not exist"))
job)

View File

@ -1,217 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.browser
(:require
["generic-pool" :as gp]
["generic-pool/lib/errors.js" :as gpe]
["playwright" :as pw]
[app.common.exceptions :as ex]
[app.common.logging :as l]
[app.config :as cf]
[app.util.object :as obj]
[promesa.core :as p]))
(l/set-level! :trace)
(def TimeoutError gpe/TimeoutError)
;; --- BROWSER API
(def default-timeout 30000)
(def default-viewport-width 1920)
(def default-viewport-height 1080)
(def default-user-agent
(str "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/99.0.3729.169 Safari/537.36"))
(defn create-cookies
[uri {:keys [name token] :or {name "auth-token"}}]
(let [domain (str (:host uri)
(when (:port uri)
(str ":" (:port uri))))]
#js [#js {:domain domain
:path "/"
:name name
:value token}]))
(defn nav!
([page url] (nav! page url nil))
([page url {:keys [wait-until timeout] :or {wait-until "networkidle" timeout 20000}}]
(.goto ^js page (str url) #js {:waitUntil wait-until :timeout timeout})))
(defn sleep
[page ms]
(.waitForTimeout ^js page ms))
(defn wait-for-fonts
"Wait until the browser has finished loading all fonts.
Checking `document.fonts.status === 'loaded'` on its own is not enough:
the render page injects its `@font-face` rules asynchronously and the
browser only loads a face lazily, when some painted text actually uses
it. A face that is declared but not yet requested stays in the `unloaded`
state, which does NOT keep `document.fonts.status` at `loading`, so the
check can pass before any real font glyphs are available. The export is
then captured with a (usually wider) fallback font, and auto-width text
laid out with the real font metrics overflows its bounds and gets clipped.
To avoid that we explicitly request every declared face and await
`document.fonts.ready` before checking the status."
([page] (wait-for-fonts page nil))
([page {:keys [timeout] :or {timeout 15000}}]
(-> (.waitForFunction ^js page
"async () => {
if (!document.fonts) return true;
try {
await Promise.all(Array.from(document.fonts, (face) => face.load()));
} catch (e) {}
await document.fonts.ready;
return document.fonts.status === 'loaded';
}"
nil
#js {:timeout timeout})
(p/catch (fn [cause]
(l/warn :hint "wait-for-fonts timed out; continuing anyway"
:cause (ex-message cause))
(p/resolved nil))))))
(defn wait-for
([locator] (wait-for locator nil))
([locator {:keys [state timeout] :or {state "visible" timeout 10000}}]
(.waitFor ^js locator #js {:state state :timeout timeout})))
(defn screenshot
([frame] (screenshot frame {}))
([frame {:keys [full-page? omit-background? type quality path]
:or {type "png" full-page? false omit-background? false quality 95}}]
(let [options (-> (obj/new)
(obj/set! "type" (name type))
(obj/set! "omitBackground" omit-background?)
(cond-> path (obj/set! "path" path))
(cond-> (= "jpeg" type) (obj/set! "quality" quality))
(cond-> full-page? (-> (obj/set! "fullPage" true)
(obj/set! "clip" nil))))]
(.screenshot ^js frame options))))
(defn emulate-media!
[page {:keys [media]}]
(.emulateMedia ^js page #js {:media media})
page)
(defn pdf
([page] (pdf page {}))
([page {:keys [scale path page-ranges]
:or {page-ranges "1"
scale 1}}]
(.pdf ^js page #js {:path path
:scale scale
:pageRanges page-ranges
:printBackground true
:preferCSSPageSize true})))
(defn eval!
[frame f]
(.evaluate ^js frame f))
(defn select
[frame selector]
(.locator ^js frame selector))
(defn select-all
[frame selector]
(.$$ ^js frame selector))
;; --- BROWSER STATE
(defonce pool (atom nil))
(defonce pool-browser-id (atom 1))
(def browser-pool-factory
(letfn [(create []
(p/let [opts #js {:args #js ["--allow-insecure-localhost" "--font-render-hinting=none"]}
browser (.launch pw/chromium opts)
id (swap! pool-browser-id inc)]
(l/info :origin "factory" :action "create" :browser-id id)
(unchecked-set browser "__id" id)
browser))
(destroy [obj]
(let [id (unchecked-get obj "__id")]
(l/info :origin "factory" :action "destroy" :browser-id id)
(.close ^js obj)))
(validate [obj]
(let [id (unchecked-get obj "__id")]
(l/info :origin "factory" :action "validate" :browser-id id :obj obj)
(p/resolved (.isConnected ^js obj))))]
#js {:create create
:destroy destroy
:validate validate}))
(defn init
[]
(let [opts #js {:max (cf/get :browser-pool-max 5)
:min (cf/get :browser-pool-min 0)
:testOnBorrow true
:evictionRunIntervalMillis 5000
:numTestsPerEvictionRun 5
;; :acquireTimeoutMillis 120000 ; 2min
:acquireTimeoutMillis 10000 ; 10 s
:idleTimeoutMillis 10000}]
(l/info :hint "initializing browser pool" :opts opts)
(reset! pool (gp/createPool browser-pool-factory opts))
(p/resolved nil)))
(defn stop
[]
(when-let [pool (deref pool)]
(l/info :hint "finalizing browser pool")
(p/do!
(.drain ^js pool)
(.clear ^js pool))))
(defn- ex-ignore
[p]
(p/handle p (constantly nil)))
(defn- translate-browser-errors
[cause]
(if (instance? TimeoutError cause)
(ex/raise :type :internal
:code :timeout
:hint (ex-message cause)
:cause cause)
(p/rejected cause)))
(defn exec!
[config handle]
(letfn [(handle-browser [browser]
(p/let [id (unchecked-get browser "__id")
context (.newContext ^js browser config)]
(l/trace :hint "exec:handle:start" :browser-id id)
(p/let [page (.newPage ^js context)
result (handle page)]
(.close ^js context)
(l/trace :hint "exec:handle:end" :browser-id id)
result)))
(on-acquire [pool browser]
(-> (handle-browser browser)
(p/then (fn [result]
(.release ^js pool browser)
result))
(p/catch (fn [cause]
(p/do!
(ex-ignore (.destroy ^js pool browser))
(p/rejected cause))))))]
(when-let [pool (deref pool)]
(-> (p/do! (.acquire ^js pool))
(p/then (partial on-acquire pool))
(p/catch translate-browser-errors)))))

View File

@ -1,105 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.core
(:require
["node:process" :as proc]
["node:worker_threads" :as wt]
[app.browser :as bwr]
[app.common.logging :as l]
[app.config :as cf]
[app.http :as http]
[app.jobs :as jobs]
[app.jobs.utils :as job.utils]
[app.redis :as redis]
[app.wasm :as wasm]
[app.wasm.pool :as wasm.pool]
[app.wasm.worker :as wasm.worker]
[promesa.core :as p]))
(enable-console-print!)
(l/setup! {:app :info})
(defn start
"Render workers run this same bundle, so the thread decides what gets booted:
the http server and its pools, or one render worker."
[& _]
(if-not ^boolean wt/isMainThread
(wasm.worker/main)
(do
(l/info :msg "initializing"
:public-uri (str (cf/get :public-uri))
:internal-uri (str (cf/get-internal-uri))
:replica cf/replica-id
:version (:full cf/version))
(l/info :msg "headless wasm export available"
:wasm-dir wasm/artifact-dir
:workers (cf/get :wasm-worker-pool-max)
:image-cache-size (cf/get :wasm-worker-image-cache-size))
(p/do
(bwr/init)
(redis/init)
(jobs/init)
(job.utils/init)
(wasm.pool/init)
(http/init)))))
(def main start)
;; Draining a pool waits for every checked-out resource to come back, which an
;; export in flight can hold for as long as its own timeout. On a hot reload
;; that would block `start` from ever running again, leaving a drained pool that
;; fails every later job.
(def ^:private shutdown-step-timeout 3000)
(defn- shutdown-step
[label f]
(-> (p/race [(p/do (f))
(p/fmap (constantly ::timeout) (p/delay shutdown-step-timeout))])
(p/handle (fn [result cause]
(when (or (some? cause) (= ::timeout result))
(l/warn :hint "shutdown step did not finish cleanly"
:step label
:cause cause))
nil))))
(defn stop
[done]
;; an empty line for visual feedback of restart
(js/console.log "")
(if-not ^boolean wt/isMainThread
;; A render worker owns no server, pools or connections; nothing to unwind.
(done)
(do
(l/info :msg "stopping")
(p/do
(shutdown-step "browser-pool" bwr/stop)
(shutdown-step "wasm-worker-pool" wasm.pool/stop)
(shutdown-step "redis" redis/stop)
(shutdown-step "http" http/stop)
(done)))))
(.on proc/default "uncaughtException"
(fn [cause]
(js/console.error cause)))
(defonce ^:private exiting? (atom false))
(defn- exit!
"Cancels this process's jobs before exiting, so their records and clients
do not keep waiting on work that dies with it. A second signal exits at once."
[]
(if (compare-and-set! exiting? false true)
(->> (shutdown-step "cancel-jobs" jobs/cancel-all!)
(p/fnly (fn [_ _] (proc/exit 0))))
(proc/exit 0)))
;; Signals are only delivered to the main thread, and `exit` in a worker would
;; take down that worker rather than the process.
(when ^boolean wt/isMainThread
(.on proc/default "SIGTERM" exit!)
(.on proc/default "SIGINT" exit!))

View File

@ -1,110 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.handlers
(:require
[app.auth :as auth]
[app.common.data :as d]
[app.common.logging :as l]
[app.common.spec :as us]
[app.handlers.export :as export]
[app.util.transit :as t]
[clojure.spec.alpha :as s]
[promesa.core :as p]))
(l/set-level! :debug)
(def ^:private error-codes
#{:queue-full})
(defn on-error
[error exchange]
(let [{:keys [type code] :as data} (ex-data error)]
(cond
(and (= :validation type)
(contains? error-codes code))
(let [data {:type :validation
:code code
:hint (ex-message error)}]
(l/warn :hint "rejecting export request" :code code)
(-> exchange
(assoc :response/status 429)
(assoc :response/body (t/encode data))
(assoc :response/headers {"content-type" "application/transit+json"})))
(= :authentication type)
(let [data {:type :authentication
:code code
:hint (ex-message error)}]
(-> exchange
(assoc :response/status 401)
(assoc :response/body (t/encode data))
(assoc :response/headers {"content-type" "application/transit+json"})))
(or (= :validation type)
(= :assertion type))
(let [explain (us/pretty-explain data)
data (-> data
(assoc :explain explain)
(assoc :type :validation)
(dissoc ::s/problems ::s/value ::s/spec))]
(-> exchange
(assoc :response/status 400)
(assoc :response/body (t/encode data))
(assoc :response/headers {"content-type" "application/transit+json"})))
(= :not-found type)
(-> exchange
(assoc :response/status 404)
(assoc :response/body (t/encode data))
(assoc :response/headers {"content-type" "application/transit+json"}))
(and (= :internal type)
(= :browser-not-ready code))
(let [data {:type :server-error
:code :internal
:hint (ex-message error)
:data data}]
(-> exchange
(assoc :response/status 503)
(assoc :response/body (t/encode data))
(assoc :response/headers {"content-type" "application/transit+json"})))
:else
(let [data {:type :server-error
:code code
:hint (ex-message error)
:data data}]
(l/error :hint "unexpected internal error" :cause error)
(-> exchange
(assoc :response/status 500)
(assoc :response/body (t/encode (d/without-nils data)))
(assoc :response/headers {"content-type" "application/transit+json"}))))))
(defn handler
"The original `POST /api/export` entry point, and the one the browser backend
still goes through. The export runs as soon as it is asked for, and the
contract is unchanged: `:wait` answers with the finished resource, otherwise
with the resource handle while the work runs."
[{:keys [:request/params :request/auth-token] :as exchange}]
(let [{:keys [cmd wait] :as params} (export/conform-params params)]
(l/debug :hint "process-request" :cmd cmd)
(->> (auth/resolve-profile-id auth-token)
(p/mcat (fn [profile-id]
;; The session wins when there is one; the body value stays
;; the fallback so nothing that used to work stops working.
(export/export! auth-token (cond-> params
(some? profile-id)
(assoc :profile-id profile-id)))))
(p/mcat (fn [{:keys [resource pending]}]
(if wait
(p/fmap (fn [resource]
(assoc exchange :response/body resource))
pending)
(do
(p/merr (constantly nil) pending)
(p/resolved
(assoc exchange :response/body (dissoc resource :path))))))))))

View File

@ -1,100 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.handlers.export
"Handle export jobs"
(:require
[app.common.spec :as us]
[app.handlers.export-frames :as export-frames]
[app.handlers.export-shapes :as export-shapes]
[app.jobs :as jobs]
[app.jobs.scheduler :as scheduler]
[app.jobs.utils :as job.utils]
[clojure.spec.alpha :as s]
[promesa.core :as p]))
;; --- PARAMS
(defmulti command-spec :cmd)
(s/def ::cmd ::us/keyword)
(s/def ::wait ::us/boolean)
(defmethod command-spec :export-shapes [_] ::export-shapes/params)
(defmethod command-spec :export-frames [_] ::export-frames/params)
(s/def ::params
(s/and (s/keys :req-un [::cmd]
:opt-un [::wait])
(s/multi-spec command-spec :cmd)))
(defn conform-params
[params]
(us/conform ::params params))
(defn- prepare
[cmd auth-token params]
(case cmd
:export-shapes (export-shapes/prepare auth-token params)
:export-frames (export-frames/prepare auth-token params)))
(defn- current
[job]
(or (jobs/lookup (:id job)) job))
(defn- run-and-track
[job run]
(->> (p/do (run job))
(p/mcat (fn [resource]
(->> (jobs/complete! (current job) resource)
(p/fmap (constantly resource)))))
(p/merr (fn [cause]
(if (jobs/cancelled? (:id job))
(p/rejected cause)
(->> (jobs/fail! (current job) cause)
(p/mcat (fn [_] (p/rejected cause)))))))))
(defn- run-now!
"Runs the job as soon as it is created, outside the scheduler."
[job]
(->> (p/do (jobs/start! job))
(p/mcat (fn [job] (p/do ((jobs/run-fn (:id job)) job))))
(p/fnly (fn [_ _]
(jobs/release! (:id job))
(job.utils/release! (:id job))))))
(defn- create!
[auth-token {:keys [cmd profile-id] :as params} start]
(let [{:keys [resource total headless run]} (prepare cmd auth-token params)]
(->> (jobs/create! {:profile-id profile-id
:cmd cmd
;; Backend used for admission and cancel UI: headless
;; WASM when `:is-wasm` is set, otherwise browser.
:backend (if headless "wasm" "browser")
:total total
:name (:name resource)
:resource-id (:id resource)}
(fn [job] (run-and-track job run)))
(p/fmap (fn [job]
(try
{:job job
:resource resource
:pending (start job)}
(catch :default cause
(jobs/fail! job cause)
(jobs/release! (:id job))
(throw cause))))))))
(defn create-job!
"Returns a promise of `{:job :resource :pending}`."
[auth-token params]
(create! auth-token params scheduler/submit!))
(defn export!
"Like `create-job!`, but the work starts right away: This is what
keeps browser exports behaving exactly as they did before there were jobs."
[auth-token params]
(create! auth-token params run-now!))

View File

@ -1,93 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.handlers.export-frames
(:require
[app.common.spec :as us]
[app.handlers.export-shapes :refer [count-objects headless-exports? prepare-exports]]
[app.handlers.resources :as rsc]
[app.jobs :as jobs]
[app.jobs.utils :as job.utils]
[app.renderer :as rd]
[app.util.shell :as sh]
[cljs.spec.alpha :as s]
[cuerdas.core :as str]
[promesa.core :as p]))
(declare ^:private join-pdf)
(declare ^:private move-file)
(s/def ::name ::us/string)
(s/def ::file-id ::us/uuid)
(s/def ::page-id ::us/uuid)
(s/def ::object-id ::us/uuid)
(s/def ::is-wasm ::us/boolean)
(s/def ::export
(s/keys :req-un [::file-id ::page-id ::object-id ::name]))
(s/def ::exports
(s/every ::export :kind vector? :min-count 1))
(s/def ::params
(s/keys :req-un [::exports]
:opt-un [::name ::is-wasm]))
(defn- run-export
[job auth-token resource {:keys [exports is-wasm file-id]}]
(let [rendered (atom [])
on-object
(fn [{:keys [path] :as _object}]
(job.utils/track! (:id job) path)
(jobs/progress! job (count (swap! rendered conj path))))
exports
(map #(assoc % :is-wasm is-wasm :job-id (:id job)) exports)]
(job.utils/track! (:id job) (:path resource))
(->> (rd/with-scope exports
(fn [render]
(jobs/check-cancelled! job)
(->> exports
(map (fn [export] (render export on-object)))
(p/all))))
(p/fmap (fn [_] @rendered))
(p/mcat (partial join-pdf job file-id))
(p/mcat (partial move-file resource))
(p/fmap (constantly resource))
(p/mcat (partial rsc/upload-resource auth-token))
(p/mcat (fn [resource]
(->> (sh/stat (:path resource))
(p/fmap #(merge resource %)))))
(p/fmap (fn [resource] (dissoc resource :path))))))
(defn prepare
[auth-token {:keys [exports name is-wasm] :as _params}]
(let [exports (-> (map #(assoc % :type :pdf :scale 1 :suffix "") exports)
(prepare-exports auth-token is-wasm))
resource (rsc/create :pdf (or name (-> exports first :name)))
file-id (-> exports first :file-id)]
{:resource resource
:total (count-objects exports)
:headless (headless-exports? exports is-wasm)
:run (fn [job] (run-export job auth-token resource
{:exports exports
:is-wasm is-wasm
:file-id file-id}))}))
(defn- join-pdf
[job file-id paths]
(p/let [prefix (str/concat "penpot.pdfunite." file-id ".")
path (job.utils/track! (:id job) (sh/tempfile :prefix prefix :suffix ".pdf"))]
(apply sh/run-cmd! "pdfunite" (conj (vec paths) path))
path))
(defn- move-file
[{:keys [path] :as resource} output-path]
(p/do
(sh/move! output-path path)
resource))

View File

@ -1,197 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.handlers.export-shapes
(:require
[app.common.data :as d]
[app.common.spec :as us]
[app.handlers.resources :as rsc]
[app.jobs :as jobs]
[app.jobs.utils :as job.utils]
[app.renderer :as rd]
[app.util.mime :as mime]
[app.util.shell :as sh]
[cljs.spec.alpha :as s]
[cuerdas.core :as str]
[promesa.core :as p]))
(declare prepare-exports)
;; Regex to clean namefiles
(def sanitize-file-regex #"[\\/:*?\"<>|]")
(s/def ::file-id ::us/uuid)
(s/def ::filename ::us/string)
(s/def ::name ::us/string)
(s/def ::object-id ::us/uuid)
(s/def ::page-id ::us/uuid)
(s/def ::share-id ::us/uuid)
(s/def ::profile-id ::us/uuid)
(s/def ::scale ::us/number)
(s/def ::suffix ::us/string)
(s/def ::type ::us/keyword)
(s/def ::wait ::us/boolean)
(s/def ::is-wasm ::us/boolean)
(s/def ::export
(s/keys :req-un [::page-id ::file-id ::object-id ::type ::suffix ::scale ::name]
:opt-un [::share-id]))
(s/def ::exports
(s/coll-of ::export :kind vector? :min-count 1))
(s/def ::params
(s/keys :req-un [::exports ::profile-id]
:opt-un [::wait ::name ::skip-children ::force-multiple ::is-wasm]))
(defn count-objects
[exports]
(reduce + 0 (map (comp count :objects) exports)))
(defn- render!
[job export on-object]
(jobs/check-cancelled! job)
(rd/render (assoc export :job-id (:id job)) on-object))
(defn- scoped-renders
"Renders every export, the headless ones sharing a single worker."
[job exports on-object]
(rd/with-scope exports
(fn [render]
(jobs/check-cancelled! job)
(->> exports
(map (fn [export] (render export on-object)))
(p/all)))))
(defn- run-single
[job auth-token resource {:keys [export is-wasm skip-children]}]
(job.utils/track! (:id job) (:path resource))
(->> (render! job
(assoc export :skip-children skip-children :is-wasm (boolean is-wasm))
(fn [{:keys [path] :as _object}]
(job.utils/track! (:id job) path)
(sh/move! path (:path resource))))
(p/fmap (constantly resource))
(p/mcat (partial rsc/upload-resource auth-token))
(p/fmap (fn [resource] (dissoc resource :path)))))
(defn- run-multiple
[job auth-token resource {:keys [exports is-wasm]}]
(let [failure (volatile! nil)
zip (rsc/create-zip :resource resource
:on-error (fn [cause] (vreset! failure cause))
:on-progress (fn [{:keys [done]}]
(jobs/progress! job done)))
append (fn [{:keys [filename path] :as _object}]
(job.utils/track! (:id job) path)
(rsc/add-to-zip zip path (str/replace filename sanitize-file-regex "_")))]
(job.utils/track! (:id job) (:path resource))
(->> (scoped-renders job
(map #(assoc % :is-wasm (boolean is-wasm) :job-id (:id job)) exports)
append)
(p/mcat (fn [_]
(if-let [cause @failure]
(p/rejected cause)
(rsc/close-zip zip))))
(p/fmap (constantly resource))
(p/mcat (partial rsc/upload-resource auth-token))
(p/fmap (fn [resource] (dissoc resource :path))))))
(defn headless-exports?
"Whether any of `exports` renders headless, and so whether the job leases a
render worker. Mirrors what `rd/with-scope` decides at run time."
[exports is-wasm]
(boolean (some #(rd/headless? {:is-wasm is-wasm :type (:type %)}) exports)))
(defn prepare
[auth-token {:keys [exports force-multiple name skip-children is-wasm] :as _params}]
(let [exports (prepare-exports exports auth-token is-wasm)
headless? (headless-exports? exports is-wasm)
single? (and (not force-multiple)
(= 1 (count exports))
(= 1 (count (-> exports first :objects))))]
(if single?
(let [export (first exports)
resource (rsc/create (:type export) (or name (:name export)))]
{:resource resource
:total 1
:headless headless?
:run (fn [job] (run-single job auth-token resource
{:export export
:is-wasm is-wasm
:skip-children skip-children}))})
(let [resource (rsc/create :zip (or name (-> exports first :name)))]
{:resource resource
:total (count-objects exports)
:headless headless?
:run (fn [job] (run-multiple job auth-token resource
{:exports exports :is-wasm is-wasm}))}))))
(defn- assoc-file-name
"A transducer that assocs a candidate filename and avoid duplicates"
[]
(letfn [(find-candidate [params used]
(loop [index 0]
(let [candidate (str (:name params)
(:suffix params "")
(when (pos? index)
(str/concat "-" (inc index)))
(mime/get-extension (:type params)))]
(if (contains? used candidate)
(recur (inc index))
candidate))))]
(fn [rf]
(let [used (volatile! #{})]
(fn
([] (rf))
([result] (rf result))
([result params]
(let [candidate (find-candidate params @used)
params (assoc params :filename candidate)]
(vswap! used conj candidate)
(rf result params))))))))
(def ^:const ^:private
default-partition-size 50)
(defn prepare-exports
[exports token is-wasm]
(letfn [(process-group [[part1 :as group]]
;; Browser path renders one DOM page per partition; headless WASM
;; takes the whole group in one render.
(if (rd/headless? {:is-wasm is-wasm :type (:type part1)})
[(build-render group)]
(sequence (comp (partition-all default-partition-size)
(map build-render))
group)))
(build-render [[part1 :as part]]
{:file-id (:file-id part1)
:page-id (:page-id part1)
:share-id (:share-id part1)
:name (:name part1)
:token token
:type (:type part1)
:scale (:scale part1)
:objects (mapv part-entry->object part)})
(part-entry->object [entry]
{:id (:object-id entry)
:filename (:filename entry)
:name (:name entry)
:suffix (:suffix entry)})]
(let [xform (comp
(map #(assoc % :token token))
(assoc-file-name))]
(->> (sequence xform exports)
(d/group-by (juxt :scale :type))
(map second)
(into [] (mapcat process-group))))))

View File

@ -1,60 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.handlers.jobs
"REST surface for export jobs, under `/api/export/jobs`.
Ownership always comes from the session (see `app.auth`), never from the
request body."
(:require
[app.auth :as auth]
[app.common.exceptions :as ex]
[app.common.logging :as l]
[app.handlers.export :as export]
[app.jobs :as jobs]
[promesa.core :as p]))
(defn create
[{:keys [:request/auth-token :request/params] :as exchange}]
(->> (auth/require-profile-id auth-token)
(p/mcat (fn [profile-id]
(let [params (-> params
(assoc :profile-id profile-id)
(export/conform-params))]
(l/dbg :hint "create export job" :cmd (:cmd params) :profile-id (str profile-id))
(export/create-job! auth-token params))))
(p/fmap (fn [{:keys [job resource pending]}]
;; A failure is reported through the job record, so the
;; promise must not surface as an unhandled rejection.
(p/merr (constantly nil) pending)
(-> exchange
(assoc :response/body (-> (or (jobs/lookup (:id job)) job)
(assoc :filename (:filename resource))
(assoc :mtype (:mtype resource)))))))))
(defn fetch
[{:keys [:request/auth-token] :as exchange} job-id]
(->> (auth/require-profile-id auth-token)
(p/mcat (fn [profile-id]
(->> (jobs/fetch job-id)
(p/fmap #(auth/check-owner! % profile-id)))))
(p/fmap (fn [job]
(assoc exchange :response/body job)))))
(defn cancel
[{:keys [:request/auth-token] :as exchange} job-id]
(->> (auth/require-profile-id auth-token)
(p/mcat (fn [profile-id]
(->> (jobs/fetch job-id)
(p/fmap #(auth/check-owner! % profile-id)))))
(p/mcat (fn [job] (jobs/cancel! (:id job))))
(p/mcat (fn [_] (jobs/fetch job-id)))
(p/fmap (fn [job]
(if job
(assoc exchange :response/body job)
(ex/raise :type :not-found
:code :object-not-found
:hint "job does not exist"))))))

View File

@ -1,98 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.handlers.resources
"Temporal resources management."
(:require
["archiver" :as arc]
["node:fs" :as fs]
["node:fs/promises" :as fsp]
["node:path" :as path]
["undici" :as http]
[app.common.exceptions :as ex]
[app.common.transit :as t]
[app.common.uri :as u]
[app.common.uuid :as uuid]
[app.config :as cf]
[app.util.mime :as mime]
[app.util.shell :as sh]
[cljs.core :as c]
[cuerdas.core :as str]
[promesa.core :as p]))
(defn- get-path
[type id]
(path/join sh/tmpdir (str/concat "penpot.resource." (c/name type) "." id)))
(defn create
"Generates ephimeral resource object."
[type name]
(let [task-id (uuid/next)
path (-> (get-path type task-id)
(sh/schedule-deletion))]
{:path path
:mtype (mime/get type)
:name name
:filename (str/concat (str/replace name #"[\\/:*?\"<>|]" "_") (mime/get-extension type))
:id task-id}))
(defn create-zip
[& {:keys [resource on-complete on-progress on-error]}]
(let [^js zip (new arc/ZipArchive)
^js out (fs/createWriteStream (:path resource))
on-complete (or on-complete (constantly nil))
progress (atom 0)]
(.on zip "error" on-error)
(.on zip "end" on-complete)
(.on zip "entry" (fn [data]
(let [name (unchecked-get data "name")
num (swap! progress inc)]
(on-progress {:done num :filename name}))))
(.pipe zip out)
zip))
(defn add-to-zip
[zip path name]
(.file ^js zip path #js {:name name}))
(defn close-zip
[zip]
(p/create (fn [resolve]
(.on ^js zip "close" resolve)
(.finalize ^js zip))))
(defn upload-resource
[auth-token resource]
(->> (fsp/readFile (:path resource))
(p/fmap (fn [buffer]
(new js/Blob #js [buffer] #js {:type (:mtype resource)})))
(p/mcat (fn [blob]
(let [fdata (new http/FormData)
agent (new http/Agent #js {:connect #js {:rejectUnauthorized false}})
headers #js {"X-Shared-Key" (str "exporter " cf/management-key)
"Authorization" (str "Bearer " auth-token)}
request #js {:headers headers
:method "POST"
:body fdata
:dispatcher agent}
uri (-> (cf/get-internal-uri)
(u/ensure-path-slash)
(u/join "api/management/methods/upload-tempfile")
(str))]
(.append fdata "content" blob (:filename resource))
(http/fetch uri request))))
(p/mcat (fn [response]
(if (not= (.-status response) 200)
(ex/raise :type :internal
:code :unable-to-upload-resource
:response-status (.-status response))
(.text response))))
(p/fmap t/decode-str)
(p/fmap (fn [result]
(merge resource (dissoc result :id))))))

View File

@ -1,189 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.http
(:require
["cookies$default" :as Cookies]
["inflation$default" :as inflate]
["node:http" :as http]
["node:stream$default" :as stream]
["raw-body$default" :as raw-body]
[app.common.logging :as l]
[app.common.transit :as t]
[app.config :as cf]
[app.handlers :as handlers]
[app.router :as router]
[cuerdas.core :as str]
[lambdaisland.uri :as u]
[promesa.core :as p]))
(l/set-level! :info)
(defprotocol IStreamableResponseBody
(write-body! [_ response]))
(extend-protocol IStreamableResponseBody
string
(write-body! [data response]
(.write ^js response data)
(.end ^js response))
js/Buffer
(write-body! [data response]
(.write ^js response data)
(.end ^js response))
stream/Stream
(write-body! [data response]
(.pipe ^js data response)
(.on ^js data "error" (fn [cause]
(js/console.error cause)
(.end response)))))
(defn- handle-response
[{:keys [:response/body
:response/headers
:response/status
response]
:as exchange}]
(let [status (or status 200)
headers (clj->js headers)
body (or body "")]
(.writeHead ^js response status headers)
(write-body! body response)))
(defn- parse-headers
[req]
(let [orig (unchecked-get req "headers")]
(persistent!
(reduce #(assoc! %1 (str/lower %2) (unchecked-get orig %2))
(transient {})
(js/Object.keys orig)))))
(defn- wrap-body-params
[handler]
(let [opts #js {:limit "60mb" :encoding "utf8"}]
(fn [{:keys [:request/method :request/headers request] :as exchange}]
(let [ctype (get headers "content-type")]
(if (= method "post")
(-> (raw-body (inflate request) opts)
(p/then (fn [data]
(cond-> data
(= ctype "application/transit+json")
(t/decode-str))))
(p/then (fn [data]
(handler (assoc exchange :request/body-params data)))))
(handler exchange))))))
(defn- wrap-params
[handler]
(fn [{:keys [:request/body-params :request/query-params] :as exchange}]
(handler (assoc exchange :request/params (merge query-params body-params)))))
(defn- wrap-response-format
[handler]
(fn [exchange]
(p/then
(handler exchange)
(fn [{:keys [:response/body :response/status] :as exchange}]
(cond
(map? body)
(let [data (t/encode-str body {:type :json-verbose})
size (js/Buffer.byteLength data "utf-8")]
(-> exchange
(assoc :response/body data)
(assoc :response/status (or status 200))
(update :response/headers assoc "content-type" "application/transit+json")
(update :response/headers assoc "content-length" size)))
(and (nil? body)
(= 200 status))
(-> exchange
(assoc :response/body "")
(assoc :response/status 204)
(assoc :response/headers {"content-length" 0}))
:else
exchange)))))
(defn- wrap-query-params
[handler]
(fn [{:keys [:request/uri] :as exchange}]
(handler (assoc exchange :request/query-params (u/query-string->map (:query uri))))))
(defn- wrap-error
[handler on-error]
(fn [exchange]
(-> (p/do (handler exchange))
(p/catch (fn [cause] (on-error cause exchange))))))
(defn- wrap-auth
[handler cookie-name]
(fn [{:keys [:request/cookies] :as exchange}]
(let [token (.get ^js cookies cookie-name)]
(handler (cond-> exchange token (assoc :request/auth-token token))))))
(defn- wrap-health
"Add /healthz entry point intercept."
[handler]
(fn [{:keys [:request/path] :as exchange}]
(if (= path "/readyz")
(assoc exchange
:response/status 200
:response/body "OK")
(handler exchange))))
(defn- create-adapter
[handler]
(fn [req res]
(let [cookies (Cookies. req res)
headers (parse-headers req)
uri (u/uri (unchecked-get req "url"))
exchange {:request/method (str/lower (unchecked-get req "method"))
:request/path (:path uri)
:request/uri uri
:request/headers headers
:request/cookies cookies
:request req
:response res}]
(-> (p/do (handler exchange))
(p/then handle-response)))))
(defn- create-server
[handler]
(.createServer ^js http (create-adapter handler)))
(def instance (atom nil))
(defn init
[]
(let [handler (-> (router/create handlers/handler)
(wrap-health)
(wrap-auth "auth-token")
(wrap-response-format)
(wrap-params)
(wrap-query-params)
(wrap-body-params)
(wrap-error handlers/on-error))
server (create-server handler)
port (cf/get :http-server-port 6061)]
(.listen server port)
(l/info :hint "welcome to penpot"
:module "exporter"
:flags cf/flags
:version (:full cf/version))
(l/info :hint "starting http server" :port port)
(reset! instance server)))
(defn stop
[]
(if-let [server @instance]
(p/create (fn [resolve]
(.close server (fn []
(l/info :hint "shutdown http server")
(resolve)))))
(p/resolved nil)))

View File

@ -1,307 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.jobs
"Export job model and lifecycle.
The record is persisted in redis (`app.jobs.store`); the runtime bits that
cannot be serialized -- cancel callbacks, the cancel signal shared with a
render worker, the throttling bookkeeping -- stay in this process, keyed by
job id.
Every state change also publishes the same `:export-update` message the
exporter has always published, so websocket clients keep working unchanged."
(:require
[app.common.data :as d]
[app.common.exceptions :as ex]
[app.common.logging :as l]
[app.common.time :as ct]
[app.common.uuid :as uuid]
[app.config :as cf]
[app.jobs.store :as store]
[app.redis :as redis]
[promesa.core :as p]))
(l/set-level! :debug)
;; A large export reports progress per object; persisting each one would be
;; hundreds of writes for information nobody reads at that resolution.
(def ^:private progress-throttle-ms 250)
(def ^:private terminal-states #{"ended" "error" "cancelled"})
(defonce ^:private registry (atom {}))
(defn- now-ms
[]
(inst-ms (ct/now)))
(defn- runtime
[job-id]
(get @registry (str job-id)))
(defn lookup
"The live record of a job running in this process, or nil."
[job-id]
(:job (runtime job-id)))
(defn fetch
"The job record from the shared store."
[job-id]
(store/fetch job-id))
(defn- publish!
[{:keys [id profile-id resource-id state done total name filename mtype
resource-uri error]}]
(redis/pub! (redis/->tenant-key (str profile-id))
(d/without-nils
{:type :export-update
:job-id id
:resource-id resource-id
:status state
:done done
:total total
:name name
:filename filename
:mtype mtype
:resource-uri resource-uri
:cause error})))
(defn- store-job!
[job]
(swap! registry update (str (:id job)) assoc :job job)
job)
(defn create!
"Builds a queued job and persists it. `run-fn` is a 1-arg fn of the job that
performs the export and returns a promise; the caller decides when to run
it."
[{:keys [profile-id cmd backend total name resource-id]} run-fn]
(let [job {:id (uuid/next)
:profile-id profile-id
:replica cf/replica-id
:cmd cmd
:backend backend
:state "queued"
:done 0
:total total
:name name
:resource-id resource-id
:created-at (now-ms)}]
(swap! registry assoc (str (:id job)) {:job job :run-fn run-fn :cancelled? false})
(->> (store/persist! job)
(p/fmap (constantly job)))))
(defn run-fn
[job-id]
(:run-fn (runtime job-id)))
(defn cancelled?
[job-id]
(boolean (:cancelled? (runtime job-id))))
(defn terminal?
[job]
(contains? terminal-states (:state job)))
(defn check-cancelled!
"Raises when the job has been cancelled. Called as each render's turn comes
up, so a cancellation stops the ones that have not started yet."
[job]
(when (cancelled? (:id job))
(ex/raise :type :internal
:code :job-cancelled
:hint "export job was cancelled")))
(defn cancel-signal
"Int32Array over a SharedArrayBuffer, readable from a worker thread: 0 while
the job is live, 1 once it has been cancelled. Nil once the job has been
released -- writing the signal back would leave an entry for a settled job in
the registry that nothing would ever remove."
[job-id]
(let [k (str job-id)]
(when-let [rt (get @registry k)]
(or (:cancel-signal rt)
(let [signal (js/Int32Array. (js/SharedArrayBuffer. 4))]
(swap! registry update k (fn [rt] (some-> rt (assoc :cancel-signal signal))))
(when (cancelled? job-id)
(js/Atomics.store signal 0 1))
signal)))))
(defn on-cancel
"Registers a callback used to abort the job's in-flight work (terminating a
render worker). A job fans out over several renders, so callbacks
accumulate. One registered for a job that already settled is dropped: keeping
it would revive that job's registry entry for good."
[job-id f]
(swap! registry update (str job-id)
(fn [rt] (some-> rt (update :cancel-fns (fnil conj []) f)))))
(defn release!
"Drops the runtime entry once the job settled. The persisted record stays
until its TTL."
[job-id]
(swap! registry dissoc (str job-id)))
(defn- live
"The job as the lifecycle last left it, or nil once it settled. Callers hold
the snapshot handed to them when their work started; writing that back would
resurrect a failed export as running and drop the error with it."
[job]
(when-let [current (:job (runtime (:id job)))]
(when-not (terminal? current)
current)))
(defn- persist-and-publish!
[job]
(store-job! job)
(publish! job)
(store/persist! job))
(defn transition!
"Moves the job on. The first terminal state wins: anything arriving after it
is dropped, so a late failure cannot overwrite a cancellation, nor a straggler
overwrite either."
[job data]
(if-let [job (live job)]
(let [job (merge job data)]
(l/info :hint "export job" :job-id (str (:id job)) :state (:state job)
:replica (:replica job))
(persist-and-publish! job))
(p/resolved job)))
(defn start!
[job]
(transition! job {:state "running" :started-at (now-ms)}))
(defn progress!
"Reports `done` objects completed. Writes are throttled, so the caller need
not care how often it calls this."
[{:keys [id] :as job} done]
(if-let [job (live job)]
(let [k (str id)
now (now-ms)
last (:last-progress-ms (runtime id) 0)
job (assoc job :done done)
write? (>= (- now last) progress-throttle-ms)]
(store-job! job)
(if write?
(do
(swap! registry update k assoc :last-progress-ms now)
(persist-and-publish! job))
(p/resolved job)))
(p/resolved job)))
(defn complete!
[job {:keys [uri filename mtype size] :as _resource}]
(transition! job {:state "ended"
:ended-at (now-ms)
:done (:total job)
:resource-uri uri
:filename filename
:mtype mtype
:size size}))
(defn fail!
[job cause]
(l/error :hint "export job failed" :job-id (str (:id job)) :cause cause)
(transition! job {:state "error"
:ended-at (now-ms)
:error (ex-message cause)}))
(defn- cancel-local!
[job-id & {:as data}]
(let [k (str job-id)
rt (get @registry k)
job (:job rt)]
(if (or (nil? job) (terminal? job))
(p/resolved job)
(do
(swap! registry update k (fn [rt] (some-> rt (assoc :cancelled? true))))
(when-let [signal (:cancel-signal rt)]
(js/Atomics.store signal 0 1))
;; Recorded before the callbacks run, not after: one of them
;; (`scheduler/drop-queued!`) releases the job, and `transition!` on a
;; released job is a no-op, so a queued job would keep claiming to be
;; queued in the store and never publish its `cancelled` update.
(let [result (transition! job (merge {:state "cancelled" :ended-at (now-ms)} data))]
(doseq [f (:cancel-fns rt)]
(try
(f)
(catch :default cause
(l/warn :hint "error on job cancel callback" :job-id k :cause cause))))
result)))))
(defn cancel!
"Cancels a job. One this process does not own is broadcast over the cancel
topic, so whoever runs it acts on it. Idempotent."
[job-id]
(if (some? (runtime job-id))
(cancel-local! job-id)
(->> (fetch job-id)
(p/mcat (fn [job]
(cond
(nil? job) (p/resolved nil)
(terminal? job) (p/resolved job)
:else (p/do
(store/request-cancel! (:id job))
job)))))))
(defn cancel-all!
"Cancels every job this process holds, on shutdown. Nothing in flight survives
the process, and clients learn it now rather than when this replica boots
again, if it ever does under the same id."
[]
(let [job-ids (keys @registry)]
(when (seq job-ids)
(l/info :hint "cancelling jobs on shutdown" :count (count job-ids)))
(->> job-ids
(map #(cancel-local! % :interrupted true))
(p/all))))
(defn- abandoned?
"A job this replica's previous run left mid-flight. A sibling's jobs are its
own business: they may well be running."
[job]
(and (= cf/replica-id (:replica job))
(not (terminal? job))))
(defn- clean-abandoned!
"Marks every job left mid-flight by a previous run of this replica as
cancelled.
A queue and its running jobs live in the memory of the process that owns
them, so nothing in flight when it died can be resumed; without this the
record would keep claiming to be running until its TTL expires. A replica
that never comes back under the same id leaves its records to the TTL."
[]
(->> (store/fetch-all)
(p/mcat (fn [jobs]
(let [abandoned (filter abandoned? jobs)]
(when (seq abandoned)
(l/warn :hint "cancelling jobs abandoned by a previous process"
:replica cf/replica-id
:count (count abandoned)))
(->> abandoned
(map (fn [job]
(store/persist! (assoc job
:state "cancelled"
:interrupted true
:ended-at (now-ms)))))
(p/all)))))
(p/fmap (fn [result] (count result)))
(p/merr (fn [cause]
(l/warn :hint "unable to clean abandoned jobs" :cause cause)
(p/resolved 0)))))
(defn init
[]
(store/on-cancel-request
(fn [job-id]
(when (some? (runtime job-id))
(l/info :hint "remote cancel request" :job-id job-id)
(cancel-local! job-id))))
(clean-abandoned!))

View File

@ -1,150 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.jobs.scheduler
"Admission control for export jobs.
Limits concurrent jobs and rejects work rather than allowing an unbounded backlog.
Every limit is per replica: the queue lives in this process only.
Queue order is FIFO, except jobs whose profile is already at its cap are skipped,
as are headless jobs once every render worker is busy."
(:require
[app.common.exceptions :as ex]
[app.common.logging :as l]
[app.config :as cf]
[app.jobs :as jobs]
[app.jobs.utils :as job.utils]
[app.wasm.pool :as pool]
[promesa.core :as p]))
(l/set-level! :debug)
(defonce ^:private state
(atom {:running {} ;; job-id -> {:profile-id :headless?}
:queue []})) ;; vector of {:job :resolve :reject}
(defn- max-concurrent [] (cf/get :exporter-max-concurrent-jobs 4))
(defn- max-per-profile [] (cf/get :exporter-max-jobs-per-profile 2))
(defn- max-queued [] (cf/get :exporter-queue-max 64))
(defn- headless?
[job]
(= "wasm" (:backend job)))
(defn- running-for
[{:keys [running]} profile-id]
(count (filter #(= profile-id (:profile-id %)) (vals running))))
(defn- running-headless
[{:keys [running]}]
(count (filter :headless? (vals running))))
(defn- eligible?
[state job]
(and (< (count (:running state)) (max-concurrent))
(< (running-for state (:profile-id job)) (max-per-profile))
;; A headless job holds one render worker for its whole run, so admitting
;; more of them than there are workers would only move the wait inside
;; the pool, with the job already reporting itself as running.
(or (not (headless? job))
(< (running-headless state) (pool/capacity)))))
(declare ^:private pump!)
(defn- finish!
[job-id]
(swap! state update :running dissoc (str job-id))
(jobs/release! job-id)
(job.utils/release! job-id)
(pump!))
(defn- execute!
[{:keys [id profile-id] :as job}]
(swap! state update :running assoc (str id) {:profile-id profile-id
:headless? (headless? job)})
(if (jobs/cancelled? id)
(do (finish! id)
(p/resolved job))
(let [run-fn (jobs/run-fn id)]
(->> (p/do (jobs/start! job))
(p/mcat (fn [job] (p/do (run-fn job))))
(p/fnly (fn [_ _] (finish! id)))))))
(defn- drop-queued!
"Removes a queued job and settles its promise, freeing its queue slot on cancellation."
[job-id]
(let [entry (volatile! nil)]
(swap! state (fn [state]
(let [queue (:queue state)
idx (->> (map-indexed vector queue)
(some (fn [[idx entry]]
(when (= (str job-id) (str (-> entry :job :id)))
idx))))]
(if idx
(do (vreset! entry (nth queue idx))
(assoc state :queue (into (subvec queue 0 idx) (subvec queue (inc idx)))))
state))))
(when-let [{:keys [resolve]} @entry]
(jobs/release! job-id)
(job.utils/release! job-id)
(resolve nil))))
(defn- take-eligible
"Pops the first queued entry that can run now, or nil."
[state]
(let [queue (:queue state)
idx (->> (map-indexed vector queue)
(some (fn [[idx entry]]
(when (eligible? state (:job entry))
idx))))]
(when idx
[(assoc state :queue (into (subvec queue 0 idx) (subvec queue (inc idx))))
(nth queue idx)])))
(defn- pump!
[]
(loop []
(let [entry (volatile! nil)]
(swap! state (fn [state]
(if-let [[next-state next-entry] (take-eligible state)]
(do (vreset! entry next-entry) next-state)
(do (vreset! entry nil) state))))
(when-let [{:keys [job resolve reject]} @entry]
(-> (execute! job)
(p/then resolve)
(p/catch reject))
(recur)))))
(defn submit!
"Registers `job` for execution. Returns a promise of the job's result, which
resolves when the export actually finishes; callers that only need the handle
can ignore it. Raises when the exporter is saturated."
[job]
(let [resolve* (volatile! nil)
reject* (volatile! nil)
pending (p/create (fn [resolve reject]
(vreset! resolve* resolve)
(vreset! reject* reject)))]
(if (eligible? @state job)
(-> (execute! job)
(p/then @resolve*)
(p/catch @reject*))
(let [queued (volatile! false)]
(swap! state (fn [state]
(if (< (count (:queue state)) (max-queued))
(do (vreset! queued true)
(update state :queue conj {:job job
:resolve @resolve*
:reject @reject*}))
(do (vreset! queued false) state))))
(when-not @queued
(ex/raise :type :validation
:code :queue-full
:hint "too many queued export jobs"))
(jobs/on-cancel (:id job) (fn [] (drop-queued! (:id job))))
(l/dbg :hint "export job queued" :job-id (str (:id job)))))
pending))

View File

@ -1,87 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.jobs.store
"Redis persistence for export jobs.
Stores each job as a single blob with a TTL matching the exported file,
so records expire with their files.
Reads use Redis. Cancellation requires the process running the job."
(:require
[app.common.logging :as l]
[app.common.transit :as t]
[app.config :as cf]
[app.redis :as redis]
[promesa.core :as p]))
(def ^:private cancel-topic (redis/->key "job-cancel"))
(defn- job-key
[job-id]
(redis/->key "job." job-id))
(defn- ttl
[]
(cf/get :exporter-job-ttl 3600))
(defn persist!
"Writes the job record and refreshes its TTL.
If the write fails, log it and continue. The export still runs
and publishes websocket updates, but the job can't be fetched
afterward (fetch returns nil, REST returns 404)."
[{:keys [id state] :as job}]
(let [jkey (job-key id)]
(->> (p/do
(redis/hset! jkey {:data (t/encode-str job)})
(redis/expire! jkey (ttl))
job)
(p/merr (fn [cause]
(if (= :redis-not-available (:code (ex-data cause)))
(l/warn :hint "job record not persisted, no redis connection"
:job-id (str id) :state state)
(l/error :hint "unable to persist job record"
:job-id (str id) :state state :cause cause))
(p/resolved job))))))
(defn fetch
"The job record, or nil when unknown or expired."
[job-id]
(->> (redis/hgetall (job-key job-id))
(p/fmap (fn [data]
(when-let [blob (get data "data")]
(try
(t/decode-str blob)
(catch :default cause
(l/warn :hint "unable to decode job record" :job-id (str job-id) :cause cause)
nil)))))))
(defn fetch-all
[]
(->> (redis/scan (redis/->key "job.*"))
(p/mcat (fn [keys]
(->> (map (fn [k]
(->> (redis/hgetall k)
(p/fmap (fn [data]
(when-let [blob (get data "data")]
(try
(t/decode-str blob)
(catch :default _ nil)))))))
keys)
(p/all))))
(p/fmap (fn [jobs] (vec (remove nil? jobs))))))
(defn request-cancel!
"Asks every exporter to cancel `job-id`. Only the one running it will act."
[job-id]
(redis/pub! cancel-topic (str job-id)))
(defn on-cancel-request
"Registers `handler` (fn of the job-id string) for cancel requests. Returns an
unsubscribe fn."
[handler]
(redis/sub! cancel-topic handler))

View File

@ -1,86 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.jobs.utils
"Temp file ownership for export jobs.
Temp files used to be cleaned only by the per-file timer in `app.util.shell`,
an hour after creation and lost entirely on restart. Here each job owns the
paths it creates, so they are dropped as soon as it settles and whatever a
crash left behind is cleaned at boot."
(:require
["node:fs/promises" :as fsp]
["node:path" :as path]
[app.common.logging :as l]
[app.config :as cf]
[app.util.shell :as sh]
[cuerdas.core :as str]
[promesa.core :as p]))
(def ^:private managed-prefix "penpot.")
(defonce ^:private tracked (atom {}))
(defn track!
"Registers `path` as owned by `job-id`, so it is removed when the job settles."
[job-id path]
(when (and job-id path)
(swap! tracked update (str job-id) (fnil conj #{}) path))
path)
(defn- remove-path!
[path]
(->> (p/do (fsp/rm path #js {:recursive true :force true}))
(p/merr (fn [cause]
(l/warn :hint "unable to remove job temp file" :path path :cause cause)
(p/resolved nil)))))
(defn release!
"Removes every file the job owns. Called once the job reached a terminal
state and its result has already been uploaded, so nothing else reads them."
[job-id]
(let [k (str job-id)
paths (get @tracked k)]
(swap! tracked dissoc k)
(if (seq paths)
(->> (map remove-path! paths)
(p/all)
(p/fmap (fn [_]
(l/dbg :hint "released job temp files" :job-id k :count (count paths))
nil)))
(p/resolved nil))))
(defn- clean!
"Removes managed temp files older than the job TTL. They can only be leftovers
of a previous process: every live one belongs to a job of this process."
[]
(let [max-age (* 1000 (cf/get :exporter-job-ttl 3600))
now (js/Date.now)]
(->> (p/do (fsp/readdir sh/tmpdir))
(p/mcat (fn [entries]
(->> (filter #(str/starts-with? % managed-prefix) entries)
(map (fn [entry]
(let [fpath (path/join sh/tmpdir entry)]
(->> (p/do (fsp/stat fpath))
(p/mcat (fn [^js stat]
(if (> (- now (inst-ms (.-mtime stat))) max-age)
(->> (remove-path! fpath)
(p/fmap (constantly 1)))
(p/resolved 0))))
(p/merr (fn [_] (p/resolved 0)))))))
(p/all))))
(p/fmap (fn [results]
(let [removed (reduce + 0 results)]
(when (pos? removed)
(l/info :hint "removed orphaned export temp files" :count removed))
removed)))
(p/merr (fn [cause]
(l/warn :hint "temp file cleanup failed" :cause cause)
(p/resolved 0))))))
(defn init
[]
(clean!))

View File

@ -1,177 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.redis
(:require
["ioredis" :as redis]
[app.common.data.macros :as dm]
[app.common.exceptions :as ex]
[app.common.logging :as l]
[app.common.transit :as t]
[app.config :as cf]
[promesa.core :as p]))
(l/set-level! :trace)
(def client (atom nil))
;; A connection in subscriber mode rejects every other command, so the
;; subscriptions need a connection of their own.
(def ^:private subscriber (atom nil))
(def ^:private subscriptions (atom {}))
(defn- create-client
[uri role]
(let [^js client (new redis/default uri)]
(.on client "connect"
(fn [] (l/info :hint "redis connection established" :uri uri :role role)))
(.on client "error"
(fn [cause] (l/error :hint "error on redis connection" :role role :cause cause)))
(.on client "close"
(fn [] (l/warn :hint "connection closed" :role role)))
(.on client "reconnect"
(fn [ms] (l/warn :hint "reconnecting to redis" :role role :ms ms)))
(.on client "end"
(fn [] (l/warn :hint "client ended, no more connections will be attempted" :role role)))
client))
(defn- dispatch-message
[topic payload]
(doseq [handler (get @subscriptions topic)]
(try
(handler payload)
(catch :default cause
(l/error :hint "error on redis subscription handler" :topic topic :cause cause)))))
(defn init
[]
(let [uri (cf/get :redis-uri)]
(swap! client (fn [prev]
(when prev (.disconnect ^js prev))
(create-client uri "commands")))
(swap! subscriber (fn [prev]
(when prev (.disconnect ^js prev))
(let [^js conn (create-client uri "subscriber")]
(.on conn "message" (fn [topic payload] (dispatch-message topic payload)))
;; Reinstate subscriptions after a reconnection.
(.on conn "connect"
(fn []
(doseq [topic (keys @subscriptions)]
(.subscribe conn topic))))
conn)))))
(defn stop
[]
(reset! subscriptions {})
(swap! subscriber (fn [conn]
(when conn (.quit ^js conn))
nil))
(swap! client (fn [client]
(when client (.quit ^js client))
nil)))
(def ^:private tenant (cf/get :tenant))
(defn ->tenant-key
"Namespaces `parts` under the tenant, the prefix the backend msgbus uses."
[& parts]
(dm/str tenant "." (apply str parts)))
(defn ->key
"Namespaces `parts` under the exporter, inside the tenant."
[& parts]
(dm/str "penpot.exporter." tenant "." (apply str parts)))
(defn pub!
"Publishes on `topic`, which must already be namespaced."
[topic payload]
(let [payload (if (map? payload) (t/encode-str payload) payload)]
(when-let [client @client]
(.publish ^js client topic payload))))
(defn sub!
"Subscribes `handler` (fn of the raw payload string) to `topic`, which must
already be namespaced. Returns a 0-arg fn that removes this handler."
[topic handler]
(swap! subscriptions update topic (fnil conj []) handler)
(when-let [conn @subscriber]
(.subscribe ^js conn topic))
(fn []
(swap! subscriptions update topic (fn [handlers] (vec (remove #(= % handler) handlers))))))
(defn- with-client
"Runs `f` against the command connection. Rejects when there is no connection
or the command fails: whether a failure is survivable depends on what the
caller was doing, and only the caller knows."
[f]
(if-let [client @client]
(p/do (f client))
(p/rejected (ex/error :type :internal
:code :redis-not-available
:hint "no redis connection"))))
(defn- with-client-lenient
"For reads, where an unreachable redis is reported as \"nothing there\"."
[f]
(->> (with-client f)
(p/merr (fn [cause]
(l/warn :hint "redis command failed" :cause cause)
(p/resolved nil)))))
(defn hset!
"Writes `data` (a map of string/keyword -> value) as a hash. Nil values are
dropped, since redis has no null."
[k data]
(let [obj (reduce-kv (fn [obj field value]
(if (some? value)
(doto obj (unchecked-set (name field) (str value)))
obj))
#js {}
data)]
(if (zero? (alength (js/Object.keys obj)))
(p/resolved nil)
(with-client (fn [^js client] (.hset client k obj))))))
(defn hgetall
"Returns the hash as a map of string keys, or nil when it does not exist."
[k]
(->> (with-client-lenient (fn [^js client] (.hgetall client k)))
(p/fmap (fn [result]
(when (and result (pos? (alength (js/Object.keys result))))
(persistent!
(reduce (fn [res field]
(assoc! res field (unchecked-get result field)))
(transient {})
(js/Object.keys result))))))))
(defn expire!
[k seconds]
(with-client (fn [^js client] (.expire client k seconds))))
(defn del!
[k]
(with-client (fn [^js client] (.del client k))))
(defn scan
"Every key matching `pattern`, walked in cursor batches so a large keyspace is
never blocked the way `KEYS` would block it.
Batches are accumulated in memory rather than consumed as a stream, which a
promise-returning fn cannot express. Fine for the job keyspace, but reading
redis wants a streaming or reactive interface before it is used for more."
[pattern]
(letfn [(step [cursor found]
(->> (with-client-lenient (fn [^js client] (.scan client cursor "MATCH" pattern "COUNT" 200)))
(p/mcat (fn [result]
(if (nil? result)
(p/resolved found)
(let [next-cursor (aget result 0)
found (into found (aget result 1))]
(if (= "0" next-cursor)
(p/resolved found)
(step next-cursor found))))))))]
(step "0" [])))

View File

@ -1,79 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.renderer
"Common renderer interface."
(:require
[app.common.logging :as l]
[app.common.spec :as us]
[app.renderer.bitmap :as rb]
[app.renderer.pdf :as rp]
[app.renderer.svg :as rs]
[app.renderer.wasm :as rw]
[cljs.spec.alpha :as s]))
(s/def ::name ::us/string)
(s/def ::suffix ::us/string)
(s/def ::type #{:png :jpeg :webp :pdf :svg})
(s/def ::page-id ::us/uuid)
(s/def ::file-id ::us/uuid)
(s/def ::share-id ::us/uuid)
(s/def ::scale ::us/number)
(s/def ::token ::us/string)
(s/def ::filename ::us/string)
(s/def ::is-wasm ::us/boolean)
(s/def ::job-id ::us/uuid)
(s/def ::object
(s/keys :req-un [::id ::name ::suffix ::filename]
:opt-un [::share-id]))
(s/def ::objects
(s/coll-of ::object :min-count 1))
(s/def ::render-params
(s/keys :req-un [::file-id ::page-id ::scale ::token ::type ::objects]
:opt-un [::is-wasm ::job-id]))
(defn headless?
"Whether `params` renders with render-wasm rather than a browser."
[{:keys [is-wasm]}]
(boolean is-wasm))
(defn render
[{:keys [type is-wasm] :as params} on-object]
(us/verify ::render-params params)
(us/verify fn? on-object)
(let [headless? (headless? params)]
(when is-wasm
(l/info :hint "render"
:type type
:backend (if headless? "wasm" "browser")))
(if headless?
(rw/render params on-object)
(case type
:png (rb/render params on-object)
:jpeg (rb/render params on-object)
:webp (rb/render params on-object)
:pdf (rp/render params on-object)
:svg (rs/render params on-object)))))
(defn with-scope
"Runs `f`, a fn of a render fn with the same signature as `render`. Exports
that render headless share one worker for the whole call instead of acquiring
one per render; the browser backend keeps rendering them in parallel."
[exports f]
(if (some headless? exports)
(rw/with-scope (:job-id (first exports))
(fn [render-leased]
(f (fn [params on-object]
(us/verify ::render-params params)
(us/verify fn? on-object)
(if (headless? params)
(render-leased params on-object)
(render params on-object))))))
(f render)))

View File

@ -1,67 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.renderer.bitmap
"A bitmap renderer."
(:require
[app.browser :as bw]
[app.common.logging :as l]
[app.common.uri :as u]
[app.config :as cf]
[app.util.mime :as mime]
[app.util.shell :as sh]
[cuerdas.core :as str]
[promesa.core :as p]))
(defn render
[{:keys [file-id page-id share-id token scale type objects skip-children] :as params} on-object]
(letfn [(prepare-options [uri]
#js {:screen #js {:width bw/default-viewport-width
:height bw/default-viewport-height}
:viewport #js {:width bw/default-viewport-width
:height bw/default-viewport-height}
:locale "en-US"
:storageState #js {:cookies (bw/create-cookies uri {:token token})}
:deviceScaleFactor scale
:userAgent bw/default-user-agent})
(render-object [page {:keys [id] :as object}]
(p/let [path (sh/tempfile :prefix "penpot.tmp.bitmap." :suffix (mime/get-extension type))
node (bw/select page (str/concat "#screenshot-" id))]
(bw/wait-for node)
(case type
:png (bw/screenshot node {:omit-background? true :type type :path path})
:jpeg (bw/screenshot node {:omit-background? false :type type :path path})
:webp (p/let [png-path (sh/tempfile :prefix "penpot.tmp.bitmap." :suffix ".png")]
;; playwright only supports jpg and png, we need to convert it afterwards
(bw/screenshot node {:omit-background? true :type :png :path png-path})
(sh/run-cmd! "convert" png-path "-quality" "100" (str "WEBP:" path))))
(on-object (assoc object :path path))))
(render [uri page]
(l/info :uri uri)
(p/do
;; navigate to the page and perform basic setup
(bw/nav! page (str uri))
(bw/sleep page 1000) ; the good old fix with sleep
(bw/wait-for-fonts page)
(bw/eval! page (js* "() => document.body.style.background = 'transparent'"))
;; take the screnshot of requested objects, one by one
(p/run (partial render-object page) objects)
nil))]
(p/let [params {:file-id file-id
:page-id page-id
:share-id share-id
:object-id (mapv :id objects)
:route "objects"
:skip-children skip-children}
uri (-> (cf/get-internal-uri)
(u/ensure-path-slash)
(u/join "render.html")
(assoc :query (u/map->query-string params)))]
(bw/exec! (prepare-options uri) (partial render uri)))))

View File

@ -1,83 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.renderer.pdf
"A pdf renderer."
(:require
[app.browser :as bw]
[app.common.data.macros :as dm]
[app.common.logging :as l]
[app.common.uri :as u]
[app.config :as cf]
[app.util.mime :as mime]
[app.util.shell :as sh]
[promesa.core :as p]))
(defn render
[{:keys [file-id page-id share-id token scale type objects] :as params} on-object]
(letfn [(prepare-options [uri]
#js {:screen #js {:width bw/default-viewport-width
:height bw/default-viewport-height}
:viewport #js {:width bw/default-viewport-width
:height bw/default-viewport-height}
:locale "en-US"
:storageState #js {:cookies (bw/create-cookies uri {:token token})}
:deviceScaleFactor scale
:userAgent bw/default-user-agent})
(prepare-uri [base-uri object-id]
(let [params {:file-id file-id
:page-id page-id
:share-id share-id
:object-id object-id
:route "objects"}]
(-> base-uri
(u/join "render.html")
(assoc :query (u/map->query-string params)))))
(sync-page-size! [dom]
(bw/eval! dom
(fn [elem]
;; IMPORTANT: No CLJS runtime allowed. Use only JS
;; primitives. This runs in a context without access to
;; cljs.core. Avoid any functions that transpile to
;; cljs.core/* calls, as they will break in the browser
;; runtime.
(let [width (.getAttribute ^js elem "width")
height (.getAttribute ^js elem "height")
style-node (let [node (.createElement js/document "style")]
(.appendChild (.-head js/document) node)
node)]
(set! (.-textContent style-node)
(dm/str "@page { size: " width "px " height "px; margin: 0; }\n"
"html, body, #app { margin: 0; padding: 0; width: " width "px; height: " height "px; overflow: visible; }"))))))
(render-object [page base-uri {:keys [id] :as object}]
(p/let [uri (prepare-uri base-uri id)
path (sh/tempfile :prefix "penpot.tmp.pdf." :suffix (mime/get-extension type))]
(l/info :uri uri)
(bw/nav! page uri)
(p/let [dom (bw/select page (dm/str "#screenshot-" id))]
(bw/wait-for dom)
(sync-page-size! dom)
(bw/screenshot dom {:full-page? true})
(bw/sleep page 2000) ; the good old fix with sleep
(bw/wait-for-fonts page)
(bw/pdf page {:path path})
path)))
(render [base-uri page]
(p/loop [objects (seq objects)]
(when-let [object (first objects)]
(p/let [path (render-object page base-uri object)]
(on-object (assoc object :path path))
(p/recur (rest objects))))))]
(let [base-uri (-> (cf/get-internal-uri)
(u/ensure-path-slash))]
(bw/exec! (prepare-options base-uri)
(partial render base-uri)))))

View File

@ -1,356 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.renderer.svg
(:require
["@penpot/svgo" :as svgo]
["xml-js" :as xml]
[app.browser :as bw]
[app.common.data :as d]
[app.common.exceptions :as ex]
[app.common.logging :as l]
[app.common.types.color :as ctc]
[app.common.uri :as u]
[app.config :as cf]
[app.renderer.svg-gradient :as svg-gradient]
[app.util.mime :as mime]
[app.util.shell :as sh]
[clojure.walk :as walk]
[cuerdas.core :as str]
[promesa.core :as p]))
(l/set-level! :trace)
(defn- xml->clj
[data]
(js->clj (xml/xml2js data)))
(defn- clj->xml
[data]
(xml/js2xml (clj->js data)))
(defn ^boolean element?
[item]
(and (map? item)
(= "element" (get item "type"))))
(defn ^boolean group-element?
[item]
(and (element? item)
(= "g" (get item "name"))))
(defn ^boolean shape-element?
[item]
(and (element? item)
(str/starts-with? (get-in item ["attributes" "id"]) "shape-")))
(defn ^boolean foreign-object-element?
[item]
(and (element? item)
(= "foreignObject" (get item "name"))))
(defn ^boolean empty-defs-element?
[item]
(and (= (get item "name") "defs")
(nil? (get item "attributes"))
(nil? (get item "elements"))))
(defn ^boolean empty-path-element?
[item]
(and (= (get item "name") "path")
(let [d (get-in item ["attributes" "d"])]
(or (str/blank? d)
(nil? d)
(str/empty? d)))))
(defn flatten-toplevel-svg-elements
"Flattens XML data structure if two nested top-side SVG elements found."
[item]
(if (and (= "svg" (get-in item ["elements" 0 "name"]))
(= "svg" (get-in item ["elements" 0 "elements" 0 "name"])))
(update-in item ["elements" 0] assoc "elements" (get-in item ["elements" 0 "elements" 0 "elements"]))
item))
(defn replace-text-nodes
"Function responsible of replace the foreignObject elements on the
provided XML with the previously rasterized PATH's."
[xmldata nodes]
(letfn [(replace-fobject [item]
(if (foreign-object-element? item)
(let [id (get-in item ["attributes" "id"])
node (get nodes id)]
(if node
(:svgdata node)
item))
item))
(process-element [item xform]
(d/update-when item "elements" #(into [] xform %)))]
(let [xform (comp (remove empty-defs-element?)
(remove empty-path-element?)
(map replace-fobject))]
(->> xmldata
(xml->clj)
(flatten-toplevel-svg-elements)
(walk/prewalk (fn [item]
(cond-> item
(element? item)
(process-element xform))))
(clj->xml)))))
(defn parse-viewbox
"Parses viewBox string into width & height map."
[data]
(let [[width height] (->> (str/split data #"\s+")
(drop 2)
(map d/parse-double))]
{:width width
:height height}))
(defn- replace-internal-uris
"Replaces internal-uri references with public-uri in SVG output.
This ensures that font URLs and other resource references in the
exported SVG use the public-facing URI accessible to end users."
[svg-content]
(let [internal-uri (str (cf/get-internal-uri))
public-uri (str (cf/get :public-uri))]
(if (and (not= internal-uri public-uri)
(str/includes? svg-content internal-uri))
(str/replace svg-content internal-uri public-uri)
svg-content)))
(defn render
[{:keys [page-id file-id share-id objects token scale type]} on-object]
(letfn [(convert-to-ppm [pngpath]
(let [ppmpath (str/concat pngpath "origin.ppm")]
(l/trace :fn :convert-to-ppm :path ppmpath)
(-> (sh/run-cmd! "convert" pngpath ppmpath)
(p/then (constantly ppmpath)))))
(trace-color-mask [pbmpath]
(l/trace :fn :trace-color-mask :pbmpath pbmpath)
(let [svgpath (str/concat pbmpath ".svg")]
(-> (sh/run-cmd! "potrace" "--flat" "-b" "svg" pbmpath "-o" svgpath)
(p/then (constantly svgpath)))))
(generate-color-layer [ppmpath color]
(when-not (ctc/hex-color-string? color)
(ex/raise :type :validation
:code :invalid-color
:hint (str "invalid hex color: " color)))
(l/trace :fn :generate-color-layer :ppmpath ppmpath :color color)
(let [pbmpath (str/concat ppmpath ".mask-" (subs color 1) ".pbm")]
(-> (sh/run-cmd! "ppmcolormask" color ppmpath)
(p/then (fn [stdout]
(-> (sh/write-file! pbmpath stdout)
(p/then (constantly pbmpath)))))
(p/then trace-color-mask)
(p/then sh/read-file)
(p/then (fn [data]
(p/let [data (xml->clj data)
data (get-in data ["elements" 1])]
{:color color
:svgdata data}))))))
(set-path-color [id color mapping node]
(let [color-mapping (get mapping color)]
(cond
(and (some? color-mapping)
(= "transparent" (get color-mapping "type")))
(update node "attributes" assoc
"fill" (get color-mapping "hex")
"fill-opacity" (get color-mapping "opacity"))
(and (some? color-mapping)
(= "gradient" (get color-mapping "type")))
(update node "attributes" assoc
"fill" (str "url(#gradient-" id "-" (subs color 1) ")"))
:else
(update node "attributes" assoc "fill" color))))
(get-gradients [id mapping]
(->> mapping
(filter (fn [[_color data]]
(= (get data "type") "gradient")))
(mapv (partial svg-gradient/data->gradient-def id))))
(join-color-layers [{:keys [id x y width height mapping] :as node} layers]
(l/trace :fn :join-color-layers :mapping mapping)
(loop [result (-> (:svgdata (first layers))
(assoc "elements" []))
layers (seq layers)]
(if-let [{:keys [color svgdata]} (first layers)]
(recur (->> (get svgdata "elements")
(filter #(= (get % "name") "g"))
(map (partial set-path-color id color mapping))
(update result "elements" into))
(rest layers))
;; Now we have the result containing the svgdata of a
;; SVG with all text layers. Now we need to transform
;; this SVG to G (Group) and remove unnecessary metadata
;; objects.
(let [vbox (-> (get-in result ["attributes" "viewBox"])
(parse-viewbox))
transform (str/fmt "translate(%s, %s) scale(%s, %s)" x y
(/ width (:width vbox))
(/ height (:height vbox)))
gradient-defs (get-gradients id mapping)
elements
(->> (get result "elements")
(mapv (fn [group]
(let [paths (get group "elements")]
(if (= 1 (count paths))
(let [path (first paths)]
(update path "attributes"
(fn [attrs]
(-> attrs
(d/merge (get group "attributes"))
(update "transform" #(str transform " " %))))))
(update-in group ["attributes" "transform"] #(str transform " " %)))))))
elements (cond->> elements
(seq gradient-defs)
(into [{"type" "element" "name" "defs" "attributes" {}
"elements" gradient-defs}]))]
(-> result
(assoc "name" "g")
(assoc "attributes" {})
(assoc "elements" elements))))))
(convert-to-svg [ppmpath {:keys [colors] :as node}]
(l/trace :fn :convert-to-svg :ppmpath ppmpath :colors colors)
(-> (p/all (map (partial generate-color-layer ppmpath) colors))
(p/then (partial join-color-layers node))))
(trace-node [{:keys [data] :as node}]
(l/trace :fn :trace-node)
(p/let [pngpath (sh/tempfile :prefix "penpot.tmp.render.svg.parse."
:suffix ".origin.png")
_ (sh/write-file! pngpath data)
ppmpath (convert-to-ppm pngpath)
svgdata (convert-to-svg ppmpath node)]
(-> node
(dissoc :data)
(assoc :svgdata svgdata))))
(extract-element-attrs [^js element]
(let [^js attrs (.. element -attributes)
^js colors (.. element -dataset -colors)
^js mapping (.. element -dataset -mapping)]
#js {:id (.. attrs -id -value)
:x (.. attrs -x -value)
:y (.. attrs -y -value)
:width (.. attrs -width -value)
:height (.. attrs -height -value)
:colors (.split colors ",")
:mapping (js/JSON.parse mapping)}))
(extract-single-node [[shot node]]
(l/trace :fn :extract-single-node)
(p/let [attrs (bw/eval! node extract-element-attrs)]
{:id (unchecked-get attrs "id")
:x (unchecked-get attrs "x")
:y (unchecked-get attrs "y")
:width (unchecked-get attrs "width")
:height (unchecked-get attrs "height")
:colors (vec (unchecked-get attrs "colors"))
:mapping (js->clj (unchecked-get attrs "mapping"))
:data shot}))
(resolve-text-node [page node]
(p/let [attrs (bw/eval! node extract-element-attrs)
id (unchecked-get attrs "id")
text-node (bw/select page (str "#screenshot-text-" id " foreignObject"))
shot (bw/screenshot text-node {:omit-background? true :type "png"})]
[shot node]))
(extract-txt-node [page item]
(-> (p/resolved item)
(p/then (partial resolve-text-node page))
(p/then extract-single-node)
(p/then trace-node)))
(extract-txt-nodes [page {:keys [id] :as objects}]
(l/trace :fn :process-text-nodes)
(-> (bw/select-all page (str/concat "#screenshot-" id " foreignObject"))
(p/then (fn [nodes] (p/all (map (partial extract-txt-node page) nodes))))
(p/then (fn [nodes] (d/index-by :id nodes)))))
(extract-svg [page {:keys [id] :as object}]
(let [node (bw/select page (str/concat "#screenshot-" id))]
(bw/wait-for node)
(bw/eval! node (fn [elem] (.-outerHTML ^js elem)))))
(prepare-options [uri]
#js {:screen #js {:width bw/default-viewport-width
:height bw/default-viewport-height}
:viewport #js {:width bw/default-viewport-width
:height bw/default-viewport-height}
:locale "en-US"
:storageState #js {:cookies (bw/create-cookies uri {:token token})}
:deviceScaleFactor scale
:userAgent bw/default-user-agent})
(render-object [page {:keys [id] :as object}]
(p/let [path (sh/tempfile :prefix "penpot.tmp.render.svg." :suffix (mime/get-extension type))
node (bw/select page (str/concat "#screenshot-" id))]
(bw/wait-for node)
(p/let [xmldata (extract-svg page object)
txtdata (extract-txt-nodes page object)
result (replace-text-nodes xmldata txtdata)
;; SVG standard don't allow the entity
;; nbsp. &#160; is equivalent but compatible
;; with SVG.
result (str/replace result "&nbsp;" "&#160;")
result (if (contains? cf/flags :exporter-svgo)
(svgo/optimize result svgo/defaultOptions)
result)
result (replace-internal-uris result)]
;; (println "------- ORIGIN:")
;; (cljs.pprint/pprint (xml->clj xmldata))
;; (println "------- RESULT:")
;; (cljs.pprint/pprint (xml->clj result))
;; (println "-------")
(sh/write-file! path result)
(on-object (assoc object :path path))
path)))
(render [uri page]
(l/info :uri uri)
(p/do
;; navigate to the page and perform basic setup
(bw/nav! page (str uri))
(bw/sleep page 1000) ; the good old fix with sleep
(bw/wait-for-fonts page)
;; take the screnshot of requested objects, one by one
(p/run (partial render-object page) objects)
nil))]
(p/let [params {:file-id file-id
:page-id page-id
:share-id share-id
:render-embed true
:object-id (mapv :id objects)
:route "objects"}
uri (-> (cf/get-internal-uri)
(u/ensure-path-slash)
(u/join "render.html")
(assoc :query (u/map->query-string params)))]
(bw/exec! (prepare-options uri)
(partial render uri)))))

View File

@ -1,50 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.renderer.wasm
"Main-thread side of the headless renderer.
Renders run on pooled workers because Skia calls are synchronous and would block
the HTTP server and other exports. Each job keeps one worker for all its renders,
sharing its caches and pool slot."
(:require
[app.jobs :as jobs]
[app.wasm.pool :as pool]
[promesa.core :as p]))
(defn- serializer
"Chains thunks so a job's renders run one at a time on its worker. A failure
is isolated: it doesn't break the chain for the next one."
[]
(let [queue (atom (p/resolved nil))]
(fn [thunk]
(let [result (p/handle @queue (fn [_ _] (thunk)))]
(reset! queue (p/handle result (fn [_ _] nil)))
result))))
(defn with-scope
"Runs `f`, a fn of a 2-arg render fn. Every render goes to the same worker,
one at a time, so the cancel check runs as each render's turn comes up."
[job-id f]
(pool/with-worker
(fn [worker]
(let [chain (serializer)
live (volatile! worker)
signal (when job-id (jobs/cancel-signal job-id))
opts {:cancel-buffer (some-> signal (.-buffer))
:cancelled? (when job-id #(jobs/cancelled? job-id))}]
(when job-id
;; Between objects the worker sees the flag; inside a render only
;; terminating the thread stops it. Cleared on the way out so a later
;; cancel cannot terminate a worker that is by then somebody else's.
(jobs/on-cancel job-id (fn [] (pool/terminate! @live))))
(->> (p/do (f (fn [params on-object]
(chain #(pool/render-on worker params on-object opts)))))
(p/fnly (fn [_ _] (vreset! live nil))))))))
(defn render
[params on-object]
(with-scope (:job-id params) (fn [render*] (render* params on-object))))

View File

@ -1,61 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.router
"Method + path dispatch.
Requests arrive with whatever prefix the proxy in front uses (`/api/export`
in devenv, `/` when talking to the process directly), so routes are matched
on the remainder after that prefix."
(:require
[app.common.exceptions :as ex]
[app.handlers.jobs :as jobs.handlers]
[cuerdas.core :as str]))
(def ^:private mount-point "/api/export")
(defn- route-path
[path]
(let [path (or path "/")
path (if (str/starts-with? path mount-point)
(subs path (count mount-point))
path)
path (str/rtrim path "/")]
(if (str/empty? path) "/" path)))
(defn- job-id
[path prefix]
(let [id (subs path (count prefix))]
(when-not (or (str/empty? id) (str/includes? id "/"))
id)))
(defn create
"Builds the request handler. `legacy-handler` serves the original
`POST /api/export` command multiplex."
[legacy-handler]
(fn [{:keys [:request/method :request/path] :as exchange}]
(let [path (route-path path)]
(cond
(and (= "post" method) (= "/" path))
(legacy-handler exchange)
(and (= "post" method) (= "/jobs" path))
(jobs.handlers/create exchange)
(and (= "get" method) (str/starts-with? path "/jobs/"))
(if-let [id (job-id path "/jobs/")]
(jobs.handlers/fetch exchange id)
(ex/raise :type :not-found :code :object-not-found :hint "unknown route"))
(and (= "delete" method) (str/starts-with? path "/jobs/"))
(if-let [id (job-id path "/jobs/")]
(jobs.handlers/cancel exchange id)
(ex/raise :type :not-found :code :object-not-found :hint "unknown route"))
:else
(ex/raise :type :not-found
:code :route-not-found
:hint (str "no route for " method " " path))))))

View File

@ -1,107 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.util.shell
"Shell & FS utilities."
(:require
["node:child_process" :as proc]
["node:fs" :as fs]
["node:path" :as path]
[app.common.exceptions :as ex]
[app.common.logging :as l]
[app.common.time :as ct]
[app.common.uuid :as uuid]
[app.config :as cf]
[cuerdas.core :as str]
[promesa.core :as p]))
(l/set-level! :trace)
(def ^:const default-deletion-delay
(* 60 60 1)) ;; 1h
(def tmpdir
(let [path (cf/get :tempdir)]
(l/inf :hint "tmptdir setup" :path path)
(when-not (fs/existsSync path)
(fs/mkdirSync path #js {:recursive true}))
path))
(defn schedule-deletion
([path] (schedule-deletion path default-deletion-delay))
([path delay]
(let [remove-path
(fn []
(try
(when (fs/existsSync path)
(fs/rmSync path #js {:recursive true})
(l/trc :hint "tempfile permanently deleted" :path path))
(catch :default cause
(l/err :hint "error on deleting temporal file"
:path path
:cause cause))))
scheduled-at
(-> (ct/now) (ct/plus #js {:seconds delay}))]
(l/trc :hint "schedule tempfile deletion"
:path path
:scheduled-at (ct/format-inst scheduled-at))
(js/setTimeout remove-path (* delay 1000))
path)))
(defn tempfile
[& {:keys [prefix suffix]
:or {prefix "penpot."
suffix ".tmp"}}]
(loop [i 0]
(if (< i 1000)
(let [path (path/join tmpdir (str/concat prefix (uuid/next) "-" i suffix))]
(if (fs/existsSync path)
(recur (inc i))
(schedule-deletion path)))
(ex/raise :type :internal
:code :unable-to-locate-temporal-file
:hint "unable to find a tempfile candidate"))))
(defn move!
[origin-path dest-path]
(.rename fs/promises origin-path dest-path))
(defn stat
[path]
(->> (.stat fs/promises path)
(p/fmap (fn [data]
{:path path
:created-at (inst-ms (.-ctime ^js data))
:size (.-size data)}))
(p/merr (fn [_cause]
(p/resolved nil)))))
(defn rmdir!
[path]
(.rm fs/promises path #js {:recursive true}))
(defn write-file!
[fpath content]
(.writeFile fs/promises fpath content))
(defn read-file
[fpath]
(.readFile fs/promises fpath))
(defn run-cmd!
[cmd & args]
(p/create
(fn [resolve reject]
(l/trace :fn :run-cmd :cmd cmd :args args)
(proc/execFile cmd (clj->js args) #js {:encoding "buffer"}
(fn [error stdout _stderr]
;; (l/trace :fn :run-cmd :stdout stdout)
(if error
(reject error)
(resolve stdout)))))))

View File

@ -1,29 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.util.transit
(:require
[cognitect.transit :as t]))
;; --- Transit Handlers
(def ^:private +read-handlers+
{"u" uuid})
(def ^:private +write-handlers+
{})
;; --- Public Api
(defn decode
[data]
(let [r (t/reader :json {:handlers +read-handlers+})]
(t/read r data)))
(defn encode
[data]
(let [w (t/writer :json-verbose {:handlers +write-handlers+})]
(t/write w data)))

View File

@ -1,240 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.wasm.pool
"Pool of headless render workers.
Mirrors `app.browser`: a `generic-pool` whose objects are `worker_threads`
instead of browsers, so acquisition and eviction behave the same way for both
render backends. A worker is expensive to build (it boots its own render-wasm
module), hence the pooling.
Acquisition is not capped: the admission scheduler is the backpressure, and
the idle watchdog guarantees a wedged worker gives its slot back.
Workers run the same bundle as the main thread; `app.core/start` branches on
`isMainThread`. There is always at least one worker, since a headless render
has nowhere else to go."
(:require
["generic-pool" :as gp]
["node:path" :as path]
["node:process" :as proc]
["node:worker_threads" :as wt]
[app.common.exceptions :as ex]
[app.common.logging :as l]
[app.common.transit :as t]
[app.config :as cf]
[promesa.core :as p]))
(l/set-level! :info)
(defonce pool (atom nil))
(defonce ^:private worker-id (atom 0))
(def ^:private ready-timeout-ms 60000)
(defn- idle-timeout-ms
"How long a render may go silent before the worker is presumed wedged. Reset
on every message, so a long export keeps its worker as long as it keeps
reporting objects; only a thread stuck inside Skia, which reports nothing and
emits no `exit`, runs it out."
[]
(* 1000 (cf/get :wasm-worker-idle-timeout 300)))
(defn- worker-script
[]
(path/resolve (aget (.-argv proc/default) 1)))
(defn- create-worker
[]
(p/create
(fn [resolve reject]
(let [script (worker-script)
id (swap! worker-id inc)
worker (new wt/Worker script)
timer (js/setTimeout
(fn []
(l/error :hint "render worker did not become ready" :worker-id id)
(.terminate ^js worker)
(reject (ex/error :type :internal
:code :worker-not-ready
:hint "render worker did not become ready")))
ready-timeout-ms)]
(unchecked-set worker "__id" id)
(unchecked-set worker "__alive" true)
(.on ^js worker "error"
(fn [cause]
(l/error :hint "render worker error" :worker-id id :cause cause)
(unchecked-set worker "__alive" false)
;; A worker that dies while booting has to fail its own creation;
;; rejecting after `resolve` is a no-op, so this is safe for the
;; errors that arrive once it is already in the pool.
(js/clearTimeout timer)
(reject cause)))
(.on ^js worker "exit"
(fn [code]
(l/info :hint "render worker exited" :worker-id id :code code)
(unchecked-set worker "__alive" false)))
;; Not `.once`: a stray message before the handshake would consume the
;; listener and leave the worker hanging until `ready-timeout-ms`.
(letfn [(on-ready [data]
(when (= "ready" (unchecked-get data "type"))
(js/clearTimeout timer)
(.off ^js worker "message" on-ready)
(l/info :origin "factory" :action "create" :worker-id id)
(resolve worker)))]
(.on ^js worker "message" on-ready))))))
(def ^:private worker-pool-factory
#js {:create create-worker
:destroy (fn [worker]
(l/info :origin "factory" :action "destroy"
:worker-id (unchecked-get worker "__id"))
(.terminate ^js worker))
:validate (fn [worker]
(p/resolved (true? (unchecked-get worker "__alive"))))})
(defn capacity
"How many renders can run at once, and so how many headless jobs the
scheduler may admit."
[]
;; Clamped rather than rejected: a bad value should not stop the exporter
;; from booting, and a headless render has no other backend to fall back to.
(max 1 (cf/get :wasm-worker-pool-max 2)))
(defn init
[]
(let [configured (cf/get :wasm-worker-pool-max 2)
max-workers (capacity)
opts #js {:max max-workers
:min (min max-workers (cf/get :wasm-worker-pool-min 1))
:testOnBorrow true
:evictionRunIntervalMillis 30000
:numTestsPerEvictionRun 2
:idleTimeoutMillis 300000}]
(when (not= configured max-workers)
(l/warn :hint "wasm-worker-pool-max raised to the minimum of one"
:configured configured))
(l/info :hint "initializing render worker pool" :opts opts)
(reset! pool (gp/createPool worker-pool-factory opts))
(p/resolved nil)))
(defn stop
[]
(when-let [instance @pool]
(l/info :hint "finalizing render worker pool")
(reset! pool nil)
(p/do
(.drain ^js instance)
(.clear ^js instance))))
(defn- run-on-worker
"Settles when the worker reports the render finished, failed, or the thread
went away. That last case matters: a terminated worker (how a cancel stops a
render mid-Skia) emits `exit` and never `error`, and a promise left pending
there would keep its pool slot borrowed for the life of the process."
[^js worker params cancel-buffer on-object]
(p/create
(fn [resolve reject]
(let [timer (volatile! nil)]
(letfn [(disarm []
(when-let [t @timer]
(js/clearTimeout t)
(vreset! timer nil)))
(rearm []
(disarm)
(vreset! timer (js/setTimeout
(fn []
(l/error :hint "render worker went silent, terminating"
:worker-id (unchecked-get worker "__id"))
(cleanup)
;; Terminating is what frees the pool slot:
;; the `exit` it raises has no listener left.
(unchecked-set worker "__alive" false)
(.terminate ^js worker)
(reject (ex/error :type :internal
:code :render-timeout
:hint "render worker stopped responding")))
(idle-timeout-ms))))
(cleanup []
(disarm)
(.off worker "message" on-message)
(.off worker "error" on-error)
(.off worker "exit" on-exit))
(on-error [cause]
(cleanup)
(reject cause))
(on-exit [code]
(cleanup)
(reject (ex/error :type :internal
:code :worker-exited
:hint (str "render worker exited with code " code))))
(on-message [data]
(rearm)
(case (unchecked-get data "type")
;; A failure while the main thread handles the object (moving
;; the file, appending to the zip) has to end the render too,
;; or nothing ever settles this promise.
"object" (try
(on-object (t/decode-str (unchecked-get data "payload")))
(catch :default cause
(cleanup)
(reject cause)))
"done" (do (cleanup) (resolve nil))
"error" (do (cleanup)
(reject (ex/error :type :internal
:code (or (some-> (unchecked-get data "code") keyword)
:wasm-render-error)
:hint (unchecked-get data "message"))))
nil))]
(.on worker "message" on-message)
(.once worker "error" on-error)
(.once worker "exit" on-exit)
(rearm)
(.postMessage worker #js {:type "render"
:params (t/encode-str params)
:cancel cancel-buffer}))))))
(defn with-worker
"Acquires one worker for the whole of `f`, a fn of that worker."
[f]
(let [instance @pool]
(->> (p/do (.acquire ^js instance))
(p/mcat (fn [worker]
(->> (p/do (f worker))
(p/fmap (fn [result]
(.release ^js instance worker)
result))
(p/merr (fn [cause]
;; The module may be aborted or mid-write, and
;; a terminated worker cannot be reused.
(-> (p/do (.destroy ^js instance worker))
(p/handle (fn [_ _] (p/rejected cause))))))))))))
(defn render-on
"Renders `params` on an already acquired worker."
[worker params on-object {:keys [cancel-buffer cancelled?]}]
(if (and cancelled? (cancelled?))
(p/rejected (ex/error :type :internal
:code :job-cancelled
:hint "export job was cancelled"))
(run-on-worker worker params cancel-buffer on-object)))
(defn terminate!
[^js worker]
(when worker
(unchecked-set worker "__alive" false)
(.terminate worker)))

View File

@ -1,502 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.wasm.render
"Headless render pipeline: renders exports with the render-wasm Skia pipeline,
with no browser and no WebGL.
Per request: fetch scene (get-page RPC) -> serialize -> provision fonts and
images -> relayout text with the real fonts -> render each object.
This runs inside a render worker (`app.wasm.worker`), one WASM design state
per worker, so the synchronous Skia calls never block the process that serves
HTTP. `app.renderer.wasm` is the main-thread side that drives it.
Moved here verbatim from `app.renderer.wasm`; git reads it as a new file only
because that namespace still exists as the proxy. Reviewable as a rename:
`git show <base>:exporter/src/app/renderer/wasm.cljs | diff -u - <this file>`.
Handles png/jpeg/webp (Skia encodes all three), pdf and svg."
(:require
["node:fs" :as fs]
["undici" :as http]
[app.common.data :as d]
[app.common.exceptions :as ex]
[app.common.fonts :as cfnt]
;; Required for side effects: these register the transit read handlers and
;; deftype impls the `get-page` response is decoded into.
[app.common.geom.matrix]
[app.common.geom.point]
[app.common.geom.rect]
[app.common.logging :as l]
[app.common.transit :as t]
[app.common.types.fills.impl]
[app.common.types.objects-map]
[app.common.types.path.impl]
[app.common.types.shape]
[app.common.types.shape.images :as images]
[app.common.uri :as u]
[app.common.uuid :as uuid]
[app.config :as cf]
[app.util.mime :as mime]
[app.util.shell :as sh]
[app.wasm :as wasm]
[app.wasm.serialize :as serialize]
[cuerdas.core :as str]
[promesa.core :as p]))
;; --- module lifecycle (one shared, lazily-initialized instance)
(defonce ^:private module* (atom nil))
(defn- ensure-module!
[]
(or @module*
(reset! module* (wasm/init!))))
;; --- backend endpoints
;;
;; Every fetch targets the internal endpoint (falling back to public-uri),
;; in a deployment the exporter reaches the backend over the container network
(defn- internal-uri
"Absolute URI for `path` on the internal (backend) endpoint."
[path]
(-> (cf/get-internal-uri)
(u/ensure-path-slash)
(u/join path)
(str)))
(defn- public-uri
"Absolute URI for `path` on the public endpoint. Whoever opens an exported SVG
resolves its `@font-face` sources, so those cannot use the internal endpoint."
[path]
(-> (cf/get :public-uri)
(u/ensure-path-slash)
(u/join path)
(str)))
(defn- error-detail
"Node's fetch reports every transport failure as a bare `TypeError: fetch
failed`; the actual reason (TLS rejection, DNS, ECONNREFUSED) is buried in a
nested `cause` chain that the logger does not print. Flattens the chain into
one readable string."
[cause]
(->> (iterate (fn [^js e] (unchecked-get e "cause")) cause)
(take-while some?)
(take 5)
(map (fn [^js e]
(let [code (unchecked-get e "code")
msg (or (unchecked-get e "message") (str e))]
(if code (str code ": " msg) msg))))
(str/join " <- ")))
(defn- fetch!
"`undici/fetch` that fails with an ex-info carrying the target uri and the
unwrapped cause chain, so a failed request says what actually went wrong and
against which endpoint."
[uri opts]
(->> (p/do (http/fetch uri opts))
(p/merr (fn [cause]
(p/rejected (ex-info "http fetch failed"
{:uri uri :detail (error-detail cause)}
cause))))))
(defn- explain
"Log-friendly reason for `cause`: the detail `fetch!` already attached, or a
freshly unwrapped chain for anything else (WASM aborts, decode errors)."
[cause]
(or (:detail (ex-data cause))
(error-detail cause)))
(defn- rpc-headers
"Auth headers for backend RPC calls (management key + bearer)."
[token]
#js {"Content-Type" "application/transit+json"
"X-Shared-Key" (str "exporter " cf/management-key)
"Authorization" (str "Bearer " token)})
(defn- asset-headers
"Auth headers for `/assets/*`. Cookie, not Bearer: those endpoints redirect to
a presigned S3/minio URL, and a Bearer header makes S3 400 (\"multiple
authentication types\")."
[token]
#js {"X-Shared-Key" (str "exporter " cf/management-key)
"Cookie" (str "auth-token=" token)})
;; --- shape bundle fetch (backend RPC)
(defn- fetch-objects
"Fetches the exported roots and their children from the backend via the
`get-page` RPC (`:object-id`, as the browser render path does), using the
same auth the exporter uses elsewhere (management key + bearer)."
[{:keys [file-id page-id share-id token objects]}]
(let [headers (rpc-headers token)
root-ids (into #{} (map :id) objects)
body (t/encode-str (cond-> {:file-id file-id
:page-id page-id}
(seq root-ids) (assoc :object-id root-ids)
share-id (assoc :share-id share-id)))
uri (internal-uri "api/rpc/command/get-page")]
(l/dbg :hint "wasm render: get-page"
:uri uri
:file-id (str file-id)
:page-id (str page-id)
:roots (count root-ids))
(->> (fetch! uri #js {:method "POST" :headers headers :body body})
(p/mcat (fn [^js resp]
(if (= 200 (.-status resp))
(.text resp)
(->> (.text resp)
(p/mcat (fn [resp-body]
(l/error :hint "wasm render: get-page failed"
:uri uri
:status (.-status resp)
:body resp-body)
(p/rejected (ex-info "get-page failed"
{:status (.-status resp)
:body resp-body}))))))))
(p/fmap t/decode-str)
(p/fmap :objects))))
;; --- font resolution
;;
;; The text serializer keeps each font's real uuid, so `wasm/fonts-for-shape`
;; reports it. Custom (team) fonts resolve through the file's font variants,
;; google fonts through the shared `app.common.fonts` catalog; builtin
;; fonts through its bundled family + the frontend's static `/fonts/`.
(defn- fetch-font-variants
"Team (custom) font variants for the file, or nil — a failure here degrades
to fallback fonts, it does not fail the export."
[{:keys [file-id share-id token]}]
(let [headers (rpc-headers token)
body (t/encode-str (cond-> {:file-id file-id}
share-id (assoc :share-id share-id)))
uri (internal-uri "api/rpc/command/get-font-variants")]
(->> (fetch! uri #js {:method "POST" :headers headers :body body})
(p/mcat (fn [^js resp]
(if (= 200 (.-status resp))
(.text resp)
(p/resolved nil))))
(p/fmap (fn [s] (when s (t/decode-str s))))
(p/merr (fn [cause]
(l/warn :hint "wasm render: get-font-variants failed"
:uri uri :detail (explain cause) :cause cause)
(p/resolved nil))))))
(defn- fetch-ttf-bytes
"Downloads a TTF, returning a promise of an ArrayBuffer (or nil). A failure
here degrades to fallback fonts, it does not fail the export."
([uri] (fetch-ttf-bytes uri #js {:method "GET"}))
([uri opts]
(->> (fetch! uri opts)
(p/mcat (fn [^js resp]
(if (= 200 (.-status resp))
(.arrayBuffer resp)
(p/resolved nil))))
(p/merr (fn [cause]
(l/warn :hint "wasm render: font fetch failed"
:uri uri :detail (explain cause) :cause cause)
(p/resolved nil))))))
;; TTF bytes cached for the process lifetime, keyed by whatever identifies the
;; variant (a gfont id+weight+style, a builtin file name).
(defonce ^:private font-bytes* (atom {}))
(defn- cached-ttf-bytes
[cache-key fetch-fn]
(if-let [bytes (get @font-bytes* cache-key)]
(p/resolved bytes)
(->> (fetch-fn)
(p/fmap (fn [buf]
(when buf (swap! font-bytes* assoc cache-key buf))
buf)))))
(defn- fetch-asset-bytes
[asset-id {:keys [token]}]
(fetch-ttf-bytes (internal-uri (str "assets/by-id/" asset-id))
#js {:method "GET" :headers (asset-headers token)}))
(defn- fetch-gfont-bytes
[ttf-url]
(fetch-ttf-bytes (cfnt/gstatic->proxy-url ttf-url (internal-uri "internal/gfonts/font"))))
(defn- fetch-builtin-font-bytes
[ttf-file]
(cached-ttf-bytes ttf-file #(fetch-ttf-bytes (internal-uri (str "fonts/" ttf-file)))))
(defn- family-uuid
[id]
(uuid/from-unsigned-parts (aget id 0) (aget id 1) (aget id 2) (aget id 3)))
(defn- find-variant
"Custom variant for a family: uuid+weight+style first, degrading to
uuid+weight then uuid."
[variants font-uuid weight style]
(let [style-str (if (zero? style) "normal" "italic")]
(or (d/seek (fn [v] (and (= (:font-id v) font-uuid)
(= (:font-weight v) weight)
(= (name (:font-style v)) style-str)))
variants)
(d/seek (fn [v] (and (= (:font-id v) font-uuid)
(= (:font-weight v) weight)))
variants)
(d/seek (fn [v] (= (:font-id v) font-uuid)) variants))))
(defn- make-resolve-font
"Builds a `resolve-font` fn (family map -> promise of TTF bytes). Custom
variants first; the bundled fonts for `uuid/zero`, which is what
`font-id->uuid` maps every builtin family to; google catalog otherwise."
[variants params]
(fn [{:keys [id weight style]}]
(let [font-uuid (family-uuid id)
variant (find-variant variants font-uuid weight style)]
(cond
(:ttf-file-id variant)
(fetch-asset-bytes (:ttf-file-id variant) params)
(= uuid/zero font-uuid)
(fetch-builtin-font-bytes (cfnt/resolve-ttf-file weight style))
:else
(if-let [gurl (cfnt/resolve-ttf-url font-uuid weight style)]
(fetch-gfont-bytes gurl)
(p/resolved nil))))))
(defn- make-font-url
"Builds a `font-url` fn (family map -> public URL of its TTF), the same
sources `make-resolve-font` downloads from but addressed publicly. The SVG
export emits one `@font-face` per family from these."
[variants]
(fn [{:keys [id weight style]}]
(let [font-uuid (family-uuid id)
variant (find-variant variants font-uuid weight style)]
(cond
(:ttf-file-id variant)
(public-uri (str "assets/by-id/" (:ttf-file-id variant)))
(= uuid/zero font-uuid)
(public-uri (str "fonts/" (cfnt/resolve-ttf-file weight style)))
:else
(some-> (cfnt/resolve-ttf-url font-uuid weight style)
(cfnt/gstatic->proxy-url (public-uri "internal/gfonts/font")))))))
;; --- fallback fonts (emoji + per-script noto fonts)
;;
;; Emoji and non-latin scripts render through fallback families, not through
;; any span's font family, so `wasm/fonts-for-shape` never reports them and the
;; provisioning above never uploads them. Must run per request, since
;; `clear-fonts!` empties the store; the TTF bytes stay cached per process.
(defn- scene-fallback-fonts
"Fallback font descriptors needed by the scene's text. Deduped because
several languages map to one noto family and provisioning is concurrent —
otherwise they all miss the byte cache at once and refetch the same TTF."
[scene]
(let [texts (for [shape (vals scene)
:when (= :text (:type shape))
node (or (some->> (:content shape) (tree-seq :children :children)) [])
:let [text (:text node)]
:when (string? text)]
text)
emoji? (boolean (some cfnt/contains-emoji? texts))
langs (reduce cfnt/collect-used-languages #{} texts)]
(distinct
(cond-> (cfnt/add-noto-fonts [] langs)
emoji? (cfnt/add-emoji-font)))))
(defn- fetch-fallback-font-bytes
"Downloads one fallback font's TTF. Cached by the whole variant, not just
`font-id`: `resolve-ttf-url` picks a different TTF per weight/style, so a
font-id-only key would serve the first downloaded variant for every other one."
[{:keys [font-id weight style]}]
(if-let [ttf-url (some-> (cfnt/gfont-id->uuid font-id) (cfnt/resolve-ttf-url weight style))]
(cached-ttf-bytes [font-id weight style] #(fetch-gfont-bytes ttf-url))
(p/resolved nil)))
(defn- provision-fallback-fonts!
[scene]
(->> (scene-fallback-fonts scene)
(map (fn [{:keys [font-id weight style is-emoji is-fallback] :as font}]
(if-let [font-uuid (cfnt/gfont-id->uuid font-id)]
(->> (fetch-fallback-font-bytes font)
(p/fmap (fn [buf]
(if buf
(wasm/store-font! {:id (uuid/get-u32 font-uuid)
:weight weight
:style style
:emoji? (boolean is-emoji)
:fallback? (boolean is-fallback)}
buf)
(l/warn :hint "wasm render: fallback font unavailable"
:font-id font-id)))))
(p/resolved nil))))
(p/all)))
;; --- image resolution
;;
;; Image fills reference file-media ids; the encoded bytes go straight to
;; `_store_image` (Skia decodes, no WebGL), keyed by media uuid so this happens
;; once per request rather than per rendered object.
(defn- fetch-file-media-bytes
"Downloads an image fill's encoded bytes by file-media id."
[media-id {:keys [token]}]
(let [headers (asset-headers token)
uri (internal-uri (str "assets/by-file-media-id/" media-id))]
(->> (fetch! uri #js {:method "GET" :headers headers})
(p/mcat (fn [^js resp]
(if (= 200 (.-status resp))
(.arrayBuffer resp)
(do
(l/warn :hint "wasm render: image fetch non-200"
:media-id (str media-id)
:uri uri
:status (.-status resp))
(p/resolved nil)))))
(p/merr (fn [cause]
(l/warn :hint "wasm render: image fetch failed"
:media-id (str media-id) :uri uri
:detail (explain cause) :cause cause)
(p/resolved nil))))))
(defn- provision-images!
"Fetches and stores every image the scene references (shape, stroke and
text-span fills, enumerated by `app.common.types.shape.images`). Unlike fonts,
the image store is not reset per request, so already-held images are skipped
and repeated exports of a file reuse them.
Always registers a public media URL for each id so SVG export can emit linked
`<image href>` even when the encoded bytes were already cached."
[scene params]
(let [all-ids (images/scene-image-ids scene)
new-ids (remove wasm/image-cached? all-ids)]
(l/dbg :hint "wasm render: provisioning images"
:total (count all-ids)
:cached (- (count all-ids) (count new-ids)))
(doseq [image-id all-ids]
(wasm/store-image-url! image-id (public-uri (str "assets/by-file-media-id/" image-id))))
(->> new-ids
(map (fn [image-id]
(->> (fetch-file-media-bytes image-id params)
(p/fmap (fn [buf]
(if buf
(do
(l/dbg :hint "wasm render: image stored"
:media-id (str image-id)
:bytes (.-byteLength ^js buf))
(wasm/store-image! image-id buf))
(l/warn :hint "wasm render: image unavailable"
:media-id (str image-id))))))))
(p/all))))
(defn- relayout-text!
"Recomputes layout for every text shape, once the real fonts are provisioned
(serialize-time layout used the fallback)."
[scene]
(doseq [shape (vals scene)
:when (= :text (:type shape))]
(wasm/update-text-layout! (:id shape))))
;; --- render
(defn- check-cancelled!
"Cancellation is cooperative: a render already inside Skia cannot be
interrupted, so the flag is only observed between objects. Killing a job
mid-object is the caller's job (terminating the worker)."
[{:keys [cancelled?] :as _params}]
(when (and cancelled? (cancelled?))
(ex/raise :type :internal
:code :job-cancelled
:hint "export job was cancelled")))
(defn- render-object-bytes
[type id scale]
(case type
:pdf (let [bytes (wasm/render-shape-pdf id)]
(l/dbg :hint "PDF generated via Skia (render-wasm headless)"
:object-id (str id)
:backend "skia-wasm"
:bytes (.-length bytes))
bytes)
:svg (let [bytes (wasm/render-shape-svg id scale)]
(l/dbg :hint "SVG generated via Skia (render-wasm headless)"
:object-id (str id)
:backend "skia-wasm"
:bytes (.-length bytes))
bytes)
(wasm/render-shape-raster id scale type)))
(defn- render*
[{:keys [scale type objects] :as params} on-object]
(l/dbg :hint "wasm render: start"
:type type
:scale scale
:objects (count objects)
:file-id (str (:file-id params))
:page-id (str (:page-id params)))
(->> (ensure-module!)
(p/mcat (fn [_] (fetch-objects params)))
(p/mcat (fn [scene]
(l/dbg :hint "wasm render: scene fetched" :shapes (count scene))
(serialize/serialize-scene! scene)
(l/dbg :hint "wasm render: scene serialized")
;; So fonts from a previous request don't leak into this one.
(wasm/clear-fonts!)
(->> (p/all [(fetch-font-variants params)
(provision-images! scene params)
(provision-fallback-fonts! scene)])
(p/mcat
(fn [[variants _]]
(let [variants (or variants [])
resolve-font (make-resolve-font variants params)
font-url (make-font-url variants)]
;; Before rendering, so the relayout below sees real
;; font metrics. Deduped across objects: shapes
;; sharing one family download its TTF once.
(wasm/provision-fonts! (map :id objects) resolve-font
:font-url font-url))))
(p/mcat
(fn [_]
(relayout-text! scene)
(p/run
(fn [{:keys [id] :as object}]
(check-cancelled! params)
(let [bytes (render-object-bytes type id scale)
path (sh/tempfile :prefix "penpot.tmp.wasm."
:suffix (mime/get-extension type))]
(l/dbg :hint "wasm render: object rendered"
:object-id (str id) :bytes (.-length bytes))
(fs/writeFileSync path bytes)
;; `on-object` returns a plain value (zip append) or
;; a promise (single export's file move); `p/do`
;; normalizes both to a thenable.
(p/do (on-object (assoc object :path path)))))
objects))))))
(p/fmap (fn [result]
;; After the request, never mid-render, so an image can't
;; disappear under a running export.
(let [evicted (wasm/evict-images! (cf/get :wasm-worker-image-cache-size wasm/image-cache-size))]
(when (pos? evicted)
(l/info :hint "wasm render: evicted cached images" :count evicted)))
result))
(p/merr (fn [cause]
(l/error :hint "wasm render: failed"
:detail (explain cause)
:internal-uri (str (cf/get-internal-uri))
:cause cause)
;; A panic can leave the mem buffer allocated or the instance
;; aborted; drop it so the next request rebuilds a fresh one.
(reset! module* nil)
(p/rejected cause)))))
(defn render
"Public entry. Renders every object of `params`, calling `on-object` with
`{:id :filename :path ...}` as each one is written out."
[params on-object]
(render* params on-object))

View File

@ -1,71 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.wasm.worker
"Render worker entry point.
Owns one render-wasm module and renders one export request at a time. The
Skia calls are synchronous, so running them here is what lets several exports
progress at once: the main thread keeps serving HTTP, zipping and uploading
while this thread is blocked inside a render.
Messages in: {type: \"render\", params: <transit>, cancel: SharedArrayBuffer}
Messages out: {type: \"ready\"}
{type: \"object\", payload: <transit>} one per rendered object
{type: \"done\"} | {type: \"error\", message, code}"
(:require
["node:worker_threads" :as wt]
[app.common.logging :as l]
[app.common.transit :as t]
[app.wasm.render :as render]
[promesa.core :as p]))
(defn- post!
[message]
(.postMessage ^js wt/parentPort message))
(defn- cancelled-fn
[buffer]
(if (some? buffer)
(let [signal (js/Int32Array. buffer)]
(fn [] (pos? (js/Atomics.load signal 0))))
(constantly false)))
(defn- handle-render
[data]
(let [params (-> (unchecked-get data "params")
(t/decode-str)
(assoc :cancelled? (cancelled-fn (unchecked-get data "cancel"))))]
(->> (render/render params
(fn [object]
(post! #js {:type "object" :payload (t/encode-str object)})))
(p/fmap (fn [_] (post! #js {:type "done"})))
(p/merr (fn [cause]
(l/warn :hint "render worker: request failed" :cause cause)
(post! #js {:type "error"
:message (or (ex-message cause) (str cause))
:code (some-> cause ex-data :code name)})
(p/resolved nil))))))
(defn- on-message
[data]
(case (unchecked-get data "type")
"render" (handle-render data)
(l/warn :hint "render worker: unknown message" :type (unchecked-get data "type"))))
(defonce ^:private listening
;; `defonce` survives a hot reload, so a reload does not stack a second
;; listener on the port. The indirection through the var keeps the reloaded
;; `on-message` in play instead of pinning the one captured at boot.
(delay
(.on ^js wt/parentPort "message" (fn [data] (on-message data)))
true))
(defn main
[& _]
@listening
(post! #js {:type "ready"})
(l/info :hint "render worker ready"))

View File

@ -0,0 +1,246 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns exporter.browser
"The browser pool service and the page API its pages speak.
The pool is a system component (`:exporter.browser/pool`): `init-key`
builds a `generic-pool` of Chromium browsers, `halt-key` drains it.
Everything else here is the plain page API the renderers use; none of
it touches the pool, so none of it needs the system.
There are no promesa chains in this namespace: Playwright and
`generic-pool` already speak native promises, and `await` covers the
rest."
(:refer-clojure :exclude [eval])
(:require
["generic-pool" :as gp]
["generic-pool/lib/errors.js" :as gpe]
["playwright" :as pw]
[app.common.data :as d]
[app.common.exceptions :as ex]
[app.common.logging :as l]
[exporter.util.object :as obj]
[exporter.utils.system :as system]))
(l/set-level! :trace)
(def ^:private TimeoutError gpe/TimeoutError)
;; --- PAGE API
(def default-timeout 30000)
(def default-viewport-width 1920)
(def default-viewport-height 1080)
(def default-user-agent
(str "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/99.0.3729.169 Safari/537.36"))
(defn create-cookies
[uri {:keys [name token] :or {name "auth-token"}}]
(let [domain (str (:host uri)
(when (:port uri)
(str ":" (:port uri))))]
#js [#js {:domain domain
:path "/"
:name name
:value token}]))
(defn nav
([page url] (nav page url nil))
([page url {:keys [wait-until timeout] :or {wait-until "networkidle" timeout 20000}}]
(.goto ^js page (str url) #js {:waitUntil wait-until :timeout timeout})))
(defn sleep
[page ms]
(.waitForTimeout ^js page ms))
(defn- ^:async wait-for-fonts-async
[page timeout]
(try
(await (.waitForFunction ^js page
"async () => {
if (!document.fonts) return true;
try {
await Promise.all(Array.from(document.fonts, (face) => face.load()));
} catch (e) {}
await document.fonts.ready;
return document.fonts.status === 'loaded';
}"
nil
#js {:timeout timeout}))
(catch :default cause
(l/warn :hint "wait-for-fonts timed out; continuing anyway"
:cause (ex-message cause))
nil)))
(defn wait-for-fonts
"Wait until the browser has finished loading all fonts.
Checking `document.fonts.status === 'loaded'` on its own is not enough:
the render page injects its `@font-face` rules asynchronously and the
browser only loads a face lazily, when some painted text actually uses
it. A face that is declared but not yet requested stays in the `unloaded`
state, which does NOT keep `document.fonts.status` at `loading`, so the
check can pass before any real font glyphs are available. The export is
then captured with a (usually wider) fallback font, and auto-width text
laid out with the real font metrics overflows its bounds and gets clipped.
To avoid that we explicitly request every declared face and await
`document.fonts.ready` before checking the status."
([page] (wait-for-fonts page nil))
([page {:keys [timeout] :or {timeout 15000}}]
(wait-for-fonts-async page timeout)))
(defn wait-for
([locator] (wait-for locator nil))
([locator {:keys [state timeout] :or {state "visible" timeout 10000}}]
(.waitFor ^js locator #js {:state state :timeout timeout})))
(defn screenshot
([frame] (screenshot frame {}))
([frame {:keys [full-page? omit-background? type quality path]
:or {type "png" full-page? false omit-background? false quality 95}}]
(let [options (-> (obj/new)
(obj/set! "type" (name type))
(obj/set! "omitBackground" omit-background?)
(cond-> path (obj/set! "path" path))
(cond-> (= "jpeg" type) (obj/set! "quality" quality))
(cond-> full-page? (-> (obj/set! "fullPage" true)
(obj/set! "clip" nil))))]
(.screenshot ^js frame options))))
(defn emulate-media
[page {:keys [media]}]
(.emulateMedia ^js page #js {:media media})
page)
(defn pdf
([page] (pdf page {}))
([page {:keys [scale path page-ranges]
:or {page-ranges "1"
scale 1}}]
(.pdf ^js page #js {:path path
:scale scale
:pageRanges page-ranges
:printBackground true
:preferCSSPageSize true})))
(defn eval
[frame f]
(.evaluate ^js frame f))
(defn select
[frame selector]
(.locator ^js frame selector))
(defn select-all
[frame selector]
(.$$ ^js frame selector))
;; --- POOL SERVICE
(def defaults
"Static pool defaults. The sizing (`:max`) mirrors the worker
concurrency default; the wiring layer passes the env-derived value."
{:max 2
:acquire-timeout 10000
:idle-timeout 10000
:launch-args ["--allow-insecure-localhost" "--font-render-hinting=none"]})
(defn- default-create-browser
[launch-args]
(fn []
(.launch pw/chromium #js {:args (clj->js launch-args)})))
(defn- ^:async create-instance
[create-browser id-counter]
(let [browser (await (create-browser))
id (swap! id-counter inc)]
(l/info :origin "factory" :action "create" :browser-id id)
(unchecked-set browser "__id" id)
browser))
(defn- destroy-instance
[browser]
(l/info :origin "factory" :action "destroy"
:browser-id (unchecked-get browser "__id"))
(.close ^js browser))
(defn- validate-instance
[browser]
(l/info :origin "factory" :action "validate"
:browser-id (unchecked-get browser "__id"))
(.isConnected ^js browser))
(defn- make-factory
[create-browser]
(let [id-counter (atom 0)]
#js {:create (fn [] (create-instance create-browser id-counter))
:destroy destroy-instance
:validate validate-instance}))
(defmethod system/init-key ::pool
[_ cfg]
(let [opts (merge defaults (d/without-nils cfg))
create-browser (or (:create-browser opts)
(default-create-browser (:launch-args opts)))]
(l/info :hint "initializing browser pool" :max (:max opts))
(gp/createPool (make-factory create-browser)
#js {:max (:max opts)
:min 0
:testOnBorrow true
:evictionRunIntervalMillis 5000
:numTestsPerEvictionRun 5
:acquireTimeoutMillis (:acquire-timeout opts)
:idleTimeoutMillis (:idle-timeout opts)})))
(defn- ^:async drain-pool
[pool]
(await (.drain ^js pool))
(await (.clear ^js pool))
nil)
(defmethod system/halt-key ::pool
[_ pool]
(when pool
(l/info :hint "finalizing browser pool")
(drain-pool pool)))
(defn- throw-browser-error
[cause]
(if (instance? TimeoutError cause)
(ex/raise :type :internal
:code :timeout
:hint (ex-message cause)
:cause cause)
(throw cause)))
(defn ^:async exec
"Check out a browser, run `handle` with a fresh page, and put the
browser back. `handle` takes the page and may return a plain value or
a promise. On failure the browser is destroyed instead of reused, and
a Playwright timeout surfaces as an `:timeout` error."
[pool context-opts handle]
(when-not pool
(throw (ex-info "browser pool is not started" {:reason ::not-started})))
(let [browser (await (.acquire ^js pool))]
(try
(let [context (await (.newContext ^js browser context-opts))
_ (l/trace :hint "exec:handle:start" :browser-id (unchecked-get browser "__id"))
page (await (.newPage ^js context))
result (await (handle page))]
(await (.close ^js context))
(l/trace :hint "exec:handle:end" :browser-id (unchecked-get browser "__id"))
(await (.release ^js pool browser))
result)
(catch :default cause
(try
(await (.destroy ^js pool browser))
(catch :default destroy-cause
(l/warn :hint "browser destroy failed"
:cause (ex-message destroy-cause))))
(throw-browser-error cause)))))

View File

@ -0,0 +1,26 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns exporter.browser.scope
"One batch through browsers: every export gets its own checkout,
all at once, bounded by the pool. No lease — a browser batch shares
nothing but the pool, so this level owns fan-out only, the mirror of
`exporter.wasm.scope` holding a lease instead. Task keys ride in
`cfg` under `:exporter.renderer/*`, set by whoever built the batch."
(:require
[exporter.renderer.browser :as driver]))
(defn ^:async run
"Renders every export through its own browser checkout. Rejects on
the first failure; progress and cancel flow per export through the
task's own callbacks."
[cfg exports]
(let [on-object (:exporter.renderer/on-object cfg)
check-cancelled (:exporter.renderer/check-cancelled cfg)]
(await (js/Promise.all
(mapv (fn [params]
(driver/render cfg params on-object check-cancelled))
exports)))))

View File

@ -4,14 +4,23 @@
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.config
(ns exporter.config
"The process configuration, read from the environment.
Same shape as the legacy `app.config` it replaces: `PENPOT_`-prefixed
variables become kebab-case keys over the defaults, decoded and
validated against the schema. The one deliberate split: `prepare`
over an explicit env object is pure and throws on invalid input (so
tests exercise it), while only the wiring entry `load-config!` reads
the live environment and exits on it — the process never boots with
a config it cannot derive its management key from."
(:refer-clojure :exclude [get])
(:require
["node:buffer" :as buffer]
["node:crypto" :as crypto]
["node:os" :as os]
["node:process" :as process]
[app.common.data :as d]
[app.common.exceptions :as ex]
[app.common.flags :as flags]
[app.common.logging :as l]
[app.common.schema :as sm]
@ -25,16 +34,12 @@
{:public-uri "http://localhost:3449"
;; :internal-uri nil ;; internal-uri cannot be nil
:tenant "default"
:host "localhost"
:http-server-port 6061
:http-server-host "0.0.0.0"
:tempdir "/tmp/penpot"
:redis-uri "redis://redis/0"
:exporter-max-concurrent-jobs 4
:exporter-max-jobs-per-profile 2
:exporter-queue-max 64
:exporter-job-ttl 3600
:wasm-worker-pool-max 2
;; how many export jobs run at once: every poller owns one Redis
;; connection, and the pools size themselves to the same number
:exporter-worker-concurrency 2
:wasm-worker-pool-min 1
:wasm-worker-idle-timeout 300
:wasm-worker-image-cache-size (* 128 1024 1024)})
@ -45,19 +50,12 @@
[:public-uri {:optional true} ::sm/uri]
[:internal-uri {:optional true} ::sm/uri]
[:exporter-shared-key {:optional true} :string]
[:host {:optional true} :string]
[:tenant {:optional true} :string]
[:flags {:optional true} [::sm/set :keyword]]
[:redis-uri {:optional true} :string]
[:tempdir {:optional true} :string]
[:browser-pool-max {:optional true} ::sm/int]
[:browser-pool-min {:optional true} ::sm/int]
[:exporter-max-concurrent-jobs {:optional true} ::sm/int]
[:exporter-max-jobs-per-profile {:optional true} ::sm/int]
[:exporter-queue-max {:optional true} ::sm/int]
[:exporter-job-ttl {:optional true} ::sm/int]
[:exporter-replica-id {:optional true} :string]
[:wasm-worker-pool-max {:optional true} ::sm/int]
[:exporter-worker-concurrency {:optional true} ::sm/int]
[:wasm-worker-pool-min {:optional true} ::sm/int]
[:wasm-worker-idle-timeout {:optional true} ::sm/int]
[:wasm-worker-image-cache-size {:optional true} ::sm/int]])
@ -75,12 +73,13 @@
[config]
(flags/parse (:flags config)))
(defn- read-env
[prefix]
(let [env (unchecked-get process "env")
kwd (fn [s] (-> (str/kebab s) (str/keyword)))
prefix (str prefix "_")
len (count prefix)]
(defn read-env
"The `PENPOT_`-prefixed variables of `env` as kebab-case keys, the
rest ignored. Takes the env object explicitly so tests pass a fake."
[env]
(let [prefix "penpot_"
len (count prefix)
kwd (fn [s] (-> (str/kebab s) (str/keyword)))]
(reduce (fn [res key]
(let [val (unchecked-get env key)
key (str/lower key)]
@ -90,22 +89,32 @@
{}
(js/Object.keys env))))
(defn- prepare-config
[]
(let [env (read-env "penpot")
env (d/without-nils env)
data (merge defaults env)
data (decode-config data)]
(defn prepare-config
"Defaults plus `env`, decoded and validated. Throws on invalid input
instead of exiting: only the wiring entry exits, on the live
environment."
[env]
(let [data (-> (merge defaults (d/without-nils (read-env env)))
(decode-config))]
(when-not (valid-config? data)
(let [explain (explain-config data)]
(println (sm/humanize-explain explain))
(process/exit -1)))
(throw (ex/error :type :internal
:code :invalid-config
:hint (sm/humanize-explain (explain-config data)))))
data))
(defn load-config!
"The live environment as validated config. Exits the process on
invalid input: without a secret there is no management key to derive,
so there is nothing to boot."
[]
(try
(prepare-config (unchecked-get process "env"))
(catch :default cause
(println (ex-message cause))
(process/exit -1))))
(def config
(prepare-config))
(load-config!))
(def version
(v/parse "%version%"))
@ -126,17 +135,19 @@
(or (c/get config :internal-uri)
(c/get config :public-uri)))
(def replica-id
"Names this exporter among its siblings. It must survive a restart, since
the boot sweep uses it to find the jobs the previous run left behind."
(or (c/get config :exporter-replica-id)
(os/hostname)))
(defn derive-management-key
"The management key for `secret` and `tenant`: HKDF blake2b512 over
\"exporter:<tenant>\", 32 bytes, base64url. Pure so tests pin the
formula the backend derives on its side. The tenant rides in the
salt (and in the key id the client sends), so a key only ever opens
its own tenant's jobs."
[secret tenant]
(-> (.from buffer/Buffer (crypto/hkdfSync "blake2b512" secret (str "exporter:" tenant) "" 32))
(.toString "base64url")))
(def management-key
(let [key (or (c/get config :exporter-shared-key)
(let [secret-key (c/get config :secret-key)
derived-key (crypto/hkdfSync "blake2b512" secret-key, "exporter" "" 32)]
(-> (.from buffer/Buffer derived-key)
(.toString "base64url"))))]
(derive-management-key (c/get config :secret-key)
(c/get config :tenant)))]
(l/inf :hint "exporter key initialized" :key (d/obfuscate-string key))
key))

View File

@ -0,0 +1,406 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns exporter.consumer
"The runner of the `:export-assets` job.
Runs one export end to end: the render plan of
`exporter.consumer.plan` (the name transducers and the partitioning
the export surfaces always kept), the pdf assembly of the frames
run, the beats shaped by the progress contract (`:preparing`,
`:rendering`, `:packaging`, with the `objects` and `pages` counters
of every partition) and one settle: a `complete-job` multipart call
that stores the artifact and closes the session the backend minted
in the same step. The render runs as the job's owner through the
session `create-job-session` opened. The batch renders through the
injected render fn (`:exporter/renderer` once wired): this namespace
never calls the facade directly.
Cancellation arrives through the beats: a `skip` answer from
`report-job-progress` says the job can no longer hear the worker
(cancelled, aborted, settled). Between units of work the runner
raises, with the same check the drivers run; inside a render the
lease holder aborts its own worker. The local registry entry is
released on every settle — unlike the legacy runner, which only
released the temp files and leaked one entry per job.
The run never rejects on an export failure: whatever goes wrong
reaches the backend as `fail-job` (with the session id, so the render
session does not outlive the settle). Only bugs of this process
reject. There are no promesa chains in this namespace."
(:require
[app.common.exceptions :as ex]
[app.common.export :as export]
[app.common.logging :as l]
[app.common.schema :as sm]
[app.common.uuid :as uuid]
[cuerdas.core :as str]
[exporter.consumer.api :as api]
[exporter.consumer.plan :as plan]
[exporter.jobs :as jobs]
[exporter.resources :as rsc]
[exporter.shell :as shell]
[exporter.tmpdir :as tdir]))
(def ^:private report-throttle-ms 250)
(def ^:private watchdog-interval-ms 1000)
;; ---- PROGRESS AND CANCEL (the milestones of the contract)
(defn enqueue-beat
"Chain `f` after the promise in `chain`, so only one beat is in flight
at a time: the watchdog and a unit beat would otherwise race and a
stale milestone could overwrite a newer one. A failed link does not
break the chain. Returns the promise of this link."
[chain f]
(let [prev @chain
next (-> (if (some? prev)
(.catch prev (fn [_] nil))
(js/Promise.resolve))
(.then (fn [_] (f))))]
(reset! chain next)
next))
(defn- start-beats
"The reporter of one run's beats.
The beat reports the stage and the counters and asks the backend to
go on; a `skip` answer means nobody is listening anymore — the job
was cancelled or settled elsewhere — and the export must stop: the
reporter fires `on-cancel` once (what marks the local flag the
checks read; the in-flight render aborts itself) and returns the
answer, so the runner sees it at the next unit of work and raises.
Between forced beats the reporter throttles the way the legacy export
did, since hundreds of objects would otherwise be hundreds of HTTP
calls for information nobody reads at that resolution. An errored
beat answers `run`: the export itself must not fail because the beat
could not fly, and the backend that is unreachable has the lease to
decide what happens to the row. The beat never rejects.
Beats are serialized: the watchdog repeats the last report while a
unit beat may be in flight, and two POSTs arriving out of order would
let the older milestone win."
[job-id on-cancel]
(let [last-beat (atom 0)
last (atom nil)
fired (atom false)
chain (atom nil)
check (fn [answer]
(when (= :skip (:action answer))
(when (compare-and-set! fired false true)
(l/info :hint "job cancelled by backend"
:job-id (str job-id))
(on-cancel)))
answer)
report (^:async fn [stage counters]
(try
(check (await (api/report-job-progress job-id {:stage stage
:counters counters})))
(catch :default cause
(l/warn :hint "beat failed to land"
:job-id (str job-id)
:cause cause)
{:action :run})))
beat (^:async fn [stage counters & {:keys [force?]}]
(let [now (js/Date.now)
now? (or force?
(>= (- now @last-beat) report-throttle-ms))]
(if now?
(do (reset! last-beat now)
(reset! last {:stage stage :counters counters})
(await (enqueue-beat chain #(report stage counters))))
{:action :run :throttled? true})))]
{:beat beat :last last}))
;; ---- THE WATCHDOG
(defn- with-watchdog
"The forced beats of one run: one every `watchdog-interval-ms`, and
they repeat the last report the run made, so a cancellation that
lands mid-render (no object landing, no unit of work ending) is seen
within that interval. `stop` clears it once the run settles."
[beat last]
(let [timer (js/setInterval
#(when (some? @last)
(beat (:stage @last) (:counters @last) :force? true))
watchdog-interval-ms)]
(fn stop [] (js/clearInterval timer))))
(defn- counter
"One counter of the vocabulary: `kind` is `objects` or `pages`."
[kind current total]
{kind {:current current :total total}})
;; ---- THE PARAMS
(defn- ->uuid
"One id of the claim into what the renderer wants: the claim carries
the job row JSON, every id a string, while the render spec only
takes uuid objects. Values that already are uuids (tests, callers in
process) pass through."
[v]
(if (string? v) (uuid/uuid v) v))
(defn- normalize-items
"The items of one claim, decoded with the row's job-def: every id a
uuid object and the type a keyword. Plain JSON shapes (tests, edn
callers) still pass through, so string ids and type names are
tolerated here."
[items]
(->> items
(mapv (fn [item]
(cond-> item
(string? (:type item)) (update :type keyword)
(string? (:file-id item)) (update :file-id ->uuid)
(string? (:page-id item)) (update :page-id ->uuid)
(string? (:object-id item)) (update :object-id ->uuid)
(string? (:share-id item)) (update :share-id ->uuid))))))
(def ^:private check-items
"The normalized claim items against the shared contract: what the
backend froze, decoded. A mistyped backend fails here, at the
boundary, instead of three stages deep in the render."
(sm/check-fn export/schema:export-items :hint "invalid export items"))
(defn- make-plan
"The render plan the legacy handlers prepare: the same transducers of
names and partition size they served the old surface with, the same
`single?` rule (one prepared export with one object, not forced),
and the artifact the whole run fills."
[job-id token tmpdir {:keys [exports kind force-multiple name skip-children is-wasm]}]
(let [items (check-items (normalize-items exports))
;; declared by the caller, never sniffed: typeless items do not
;; exist on the validated contract, so there is nothing to infer
frames? (= :frames (keyword kind))
prepared (plan/prepare-exports items token is-wasm)
single? (and (not frames?)
(not (true? force-multiple))
(= 1 (count prepared))
(= 1 (count (-> prepared first :objects))))
artifact-kind (cond
single? (-> prepared first :type)
frames? :pdf
:else :zip)
;; The artifact parks in the injected area, so it is tracked
;; where it is born: single, multi and frames all settle
;; through it, and the release wipes the job area once the
;; settle uploaded it.
resource (rsc/create tmpdir
artifact-kind
(or name (-> prepared first :name)))]
(tdir/track job-id (:path resource))
{:frames? frames?
:single? single?
:job-id (->uuid job-id)
:prepared prepared
:total (plan/count-objects prepared)
:skip-children skip-children
:is-wasm (boolean is-wasm)
:counter-kind (if frames? :pages :objects)
:resource resource}))
;; ---- THE RUNNERS
(defn- check-beat
"A forced beat after a unit of work: a `skip` there is a cancelled
job, the same exception the legacy surfaces raised, so everything
below unwinds."
[answer]
(when (= :skip (:action answer))
(throw (ex/error :type :internal
:code :job-cancelled
:hint "export job was cancelled")))
answer)
(defn- ^:async run-single
"One render, one object: the artifact IS the object, moved into the
resource path the multipart settle will name."
[render beat plan check-cancelled]
(let [{:keys [job-id resource counter-kind]} plan
export (-> plan :prepared first)
object (atom nil)]
(check-beat (await (beat :rendering (counter counter-kind 0 1) :force? true)))
(await (render {:exports [(cond-> (assoc export
:job-id job-id)
(some? (:skip-children plan))
(assoc :skip-children (:skip-children plan)))]
:on-object (fn [obj]
(reset! object obj)
(tdir/track job-id (:path obj))
nil)
:check-cancelled check-cancelled}))
(await (shell/move (:path @object) (:path resource)))
(check-beat (await (beat :packaging (counter counter-kind 1 1) :force? true)))
resource))
(defn- ^:async run-multiple
"The multi-object render: every prepared export renders through the
one batch of the facade, appending each object to the zip as it
lands, and the progress counts what has landed so far. Any zipping
error surfaces after the renders, so the failures of the writer do
not compete with the ones of the render."
[render beat plan check-cancelled]
(let [{:keys [job-id resource prepared total counter-kind]} plan
failure (volatile! nil)
rendered (volatile! 0)
zip (rsc/create-zip :resource resource
:on-error (fn [cause] (vreset! failure cause))
:on-progress (fn [_] nil))
append (^:async fn [{:keys [filename path]}]
(check-cancelled)
(tdir/track job-id path)
(vswap! rendered inc)
(await (beat :rendering
(counter counter-kind @rendered total)))
(rsc/add-to-zip zip path
(str/replace filename
plan/sanitize-file-regex
"_"))
nil)]
(check-beat (await (beat :rendering (counter counter-kind 0 total) :force? true)))
(await (render {:exports (mapv #(cond-> (assoc % :job-id job-id)
(some? (:skip-children plan))
(assoc :skip-children (:skip-children plan)))
prepared)
:on-object append
:check-cancelled check-cancelled}))
(when-let [cause @failure]
(throw cause))
(await (rsc/close-zip zip))
(check-beat (await (beat :packaging (counter counter-kind total total) :force? true)))
resource))
(defn- ^:async join-pdf
"The pages render one file per page; `pdfunite` stitches one pdf out
of them all, the way the legacy export did."
[job-id tmpdir file-id paths]
(let [path (tdir/track job-id
(shell/tempfile tmpdir
:prefix (str/concat "penpot.pdfunite." file-id ".")
:suffix ".pdf"))]
(await (shell/run-cmd "pdfunite" (into [] (concat (vec paths) [path]))))
path))
(defn- ^:async run-frames
"The frames render: a file per page, joined into the pdf of the file
once every page has landed."
[render tmpdir beat plan check-cancelled]
(let [{:keys [job-id resource prepared total]} plan
file-id (-> prepared first :file-id)
paths (volatile! [])
rendered (volatile! 0)
on-object (^:async fn [{:keys [path]}]
(check-cancelled)
(tdir/track job-id path)
(vswap! paths conj path)
(vswap! rendered inc)
(await (beat :rendering
(counter :pages @rendered total)))
nil)]
(check-beat (await (beat :rendering (counter :pages 0 total) :force? true)))
(await (render {:exports (mapv #(cond-> (assoc % :job-id job-id
:is-wasm (:is-wasm plan))
(some? (:skip-children plan))
(assoc :skip-children (:skip-children plan)))
prepared)
:on-object on-object
:check-cancelled check-cancelled}))
(let [joined (await (join-pdf job-id tmpdir file-id @paths))]
(await (shell/move joined (:path resource)))
(check-beat (await (beat :packaging (counter :pages total total) :force? true)))
resource)))
(defn- run-prepared
[render tmpdir beat check-cancelled plan]
(cond
(:single? plan) (run-single render beat plan check-cancelled)
(:frames? plan) (run-frames render tmpdir beat plan check-cancelled)
:else (run-multiple render beat plan check-cancelled)))
;; ---- THE SETTLE
(defn- failure-report
"The error shape the backend stores for `job.error`: type and code
keywords, hint the human message. A cause that is not a penpot error
becomes an internal one, with only its message as the hint."
[cause]
(let [data (ex-data cause)]
{:type (or (:type data) :internal)
:code (or (:code data) :export-failed)
:hint (or (:hint data) (ex-message cause) "export failed")}))
(defn cancelled?
"Whether `cause` ends a run the user asked to end: a cancelled
export is a normal ending, reported and logged as one, never as a
failure with a trace."
[cause]
(= :job-cancelled (:code (ex-data cause))))
(defn- ^:async settle-failure
"Settles the job as failed, after releasing the temp files the run
owns. The settle never rejects: if the failure cannot reach the
backend, the row keeps running until the lease GC turns it `aborted`,
which is the honest state for a worker that cannot talk."
[job-id session cause]
(await (tdir/release job-id))
(jobs/release job-id)
(try
(await (api/fail-job job-id (failure-report cause)
:session-id (some-> @session :session-id)))
(catch :default settle-cause
(l/warn :hint "unable to settle the failed job"
:job-id job-id :cause settle-cause))))
(defn ^:async run-export
"Runs one claimed export to its settle: first breath, render
session, plan, render batch, multipart complete. `render` is the
injected render fn (one task map per batch: `:exports`, `:on-object`,
`:check-cancelled`) plus `tmpdir`, the directory every temp path is
created under; the claim names the job and `params` carries the
frozen job params."
[render tmpdir {:keys [job-id]} params]
(let [session (atom nil)
stop-watchdog (atom nil)
_ (jobs/register job-id)
stop-cmd (fn [] (jobs/mark-cancelled job-id))
{:keys [beat last]} (start-beats job-id stop-cmd)
check-cancelled (jobs/check-cancelled job-id)]
(try
(reset! stop-watchdog (with-watchdog beat last))
;; the first breath right after the claim, before the
;; session roundtrip: the widget moves from queued to
;; exporting on it, and a cancel that landed meanwhile
;; raises here instead of after wasted work
(check-beat (await (beat :preparing {} :force? true)))
(let [session' (await (api/create-job-session job-id))]
(reset! session session')
(l/info :hint "render session minted"
:job-id (str job-id))
(let [plan (make-plan job-id (:session-token session') tmpdir params)
resource (await (run-prepared render tmpdir beat check-cancelled plan))]
;; the watchdog outlives the render, so the run stops it on
;; the way out: a run that ends badly stops beating too, or
;; its interval would knock on a dead job forever, one HTTP
;; call a second that only ever answers skip
(@stop-watchdog)
(let [artifact {:path (str (:path resource))
:filename (:filename resource)
:mtype (:mtype resource)}
answer (await (api/complete-job-with-artifact
{:job-id job-id :session-id (:session-id session')}
artifact))]
(await (tdir/release job-id))
(jobs/release job-id)
(l/info :hint "export job settled" :job-id (str job-id)
:outcome (some-> answer :action name))
nil)))
(catch :default cause
(when-let [stop @stop-watchdog] (stop))
(if (cancelled? cause)
(l/info :hint "export job cancelled" :job-id (str job-id))
(l/error :hint "export job failed" :job-id (str job-id) :cause cause))
(await (settle-failure job-id session cause))))))

View File

@ -0,0 +1,173 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns exporter.consumer.api
"The backend management API client.
The only place of the worker that speaks HTTP: every method of the
backend ledger (claim, progress, session, settle) travels through
here, with the shared key the exporter was configured with and, when
a call renders as a user, the Bearer token of its session. Request
bodies travel as transit+json (except the artifact form of
`complete-job`, which is multipart, like every other upload) and the
answers come back as transit too. There are no promesa chains in
this namespace."
(:require
["node:fs" :as fs]
["undici" :as http]
[app.common.exceptions :as ex]
[app.common.transit :as t]
[app.common.uri :as u]
[cljs.core :as c]
[exporter.config :as cf]))
(def fetch
"The network entry point of the client: a plain def, redefined by the
tests of the client, which never need a network."
http/fetch)
;; ---- THE REQUEST
;; ---- THE CONNECTION
(def ^:private call-timeout-ms
"A management call that has not answered by then is treated as failed:
the beat path already knows how to carry on without it."
10000)
(def ^:private upload-timeout-ms
"The artifact upload carries the whole render, so it gets room."
120000)
(def ^:private agent*
"The single undici Agent of the process: keep-alive and TLS in one
place. Created on first use so requiring this namespace never opens a
socket."
(delay
(new http/Agent #js {:connect #js {:rejectUnauthorized false}})))
(defn close-client!
"Release the connections of the process. Called from the shutdown path."
[]
(when (realized? agent*)
(.close ^js @agent*)))
(defn- dispatcher
"The agent of every request. Certificate verification is off on the
management calls (shared key plus Bearer tokens ride each beat), the
same way the existing uploads already accept self-signed certificates.
Conscious posture for first-party deployers with self-signed TLS;
revisit if the threat model changes."
[]
@agent*)
(defn- method-uri
[method]
(-> (cf/get-internal-uri)
(u/ensure-path-slash)
(u/join (str "api/management/methods/" (c/name method)))
(str)))
(defn- shared-key
"What every management call carries: the key id of this exporter plus
the key, and the wish to read transit back. The id names the tenant
this worker renders for (`exporter:<tenant>`), so the backend only
lets it touch its own jobs."
[]
#js {"X-Shared-Key" (str "exporter:" (cf/get :tenant) " " cf/management-key)
"Accept" "application/transit+json"})
(defn- ^:async parse-response
"The answer of a management call: transit when it lands. A non-200 is
the failure of the call, with the error body the backend answered;
a body that does not decode stays as the plain text it came as."
[response]
(let [status (.-status response)
text (await (.text response))
body (let [decoded (ex/try! (t/decode-str text))]
(if-not (ex/exception? decoded) decoded text))]
(if (= status 200)
body
(throw (ex/error :type :internal
:code :failed-management-request
:status status
:response body)))))
(defn- ^:async request
"POST `params` to a management method as transit+json. The content
type travels on every call: without it the backend never parses the
body as transit and validates an empty map. A call that does not
answer in time is aborted and rejects like any other failure."
[method params]
(let [response (await (fetch (method-uri method)
#js {:method "POST"
:headers (doto (shared-key)
(unchecked-set "Content-Type" "application/transit+json"))
:body (t/encode-str params)
:signal (js/AbortSignal.timeout call-timeout-ms)
:dispatcher (dispatcher)}))]
(await (parse-response response))))
(defn- ^:async request-multipart
"POST a FormData body to a management method. The content type is
settled by the FormData itself (boundary included), so only the
shared key goes in the headers. The artifact carries the whole render,
so this call gets the longer timeout."
[method form-data]
(let [response (await (fetch (method-uri method)
#js {:method "POST"
:headers (shared-key)
:body form-data
:signal (js/AbortSignal.timeout upload-timeout-ms)
:dispatcher (dispatcher)}))]
(await (parse-response response))))
;; ---- THE METHODS
(defn ^:async claim-job
"Request a queued job from the backend. The answer is either
`{:action :run :name ... :params ...}` (this poller owns the job) or
`{:action :skip ...}` (something else won it, or it is gone)."
[job-id scheduled-at]
(await (request :claim-job {:job-id job-id :scheduled-at scheduled-at})))
(defn ^:async report-job-progress
"The heartbeat of a running job: the stage hitos of the progress
contract plus the counters of the current work. A `:skip` says the
job can no longer hear the worker (cancelled, aborted, settled)."
[job-id progress]
(await (request :report-job-progress {:job-id job-id :progress progress})))
(defn ^:async create-job-session
"Mint an authentication session of the owner of a running job, so the
render views authenticate as the user, not as a machine."
[job-id]
(await (request :create-job-session {:job-id job-id})))
(defn ^:async complete-job-with-artifact
"Close a job whose result is the artifact itself: the backend stores
the bytes and derives the result descriptor. `artifact` says what it
is (`:path`, `:filename`, `:mtype`); `ctx` names the job and, when
the worker rendered with a session, the one to close."
[{:keys [job-id session-id]} {:keys [path filename mtype]}]
;; The part is the single source of name and type: they travel in the
;; part itself (name in the disposition, type in the Blob), never as
;; separate text fields.
(let [blob (await (fs/openAsBlob (str path) #js {:type (str mtype)}))
fdata (new http/FormData)]
(.append fdata "job-id" (str job-id))
(.append fdata "content" blob filename)
(when session-id
(.append fdata "session-id" (str session-id)))
(await (request-multipart :complete-job fdata))))
(defn ^:async fail-job
"Close a job as failed with the rich error the backend stores for
`job.error`. `:session-id` closes the render session the same way."
[job-id error & {:keys [session-id]}]
(await (request :fail-job
(cond-> {:job-id job-id :error error}
(some? session-id) (assoc :session-id session-id)))))

View File

@ -0,0 +1,38 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns exporter.consumer.config
"The knobs of the consumer: how many jobs run at once and which
queue they come from. Read live, so a long-lived process obeys a
configuration change on the next poll."
(:require
[cljs.core :as c]
[exporter.config :as cf]))
(def queue-name
"The queue this worker consumes: the one the `:export-assets` job-def
routes its jobs to through `::jobs/queue-name`."
"exporter")
(defn concurrency
"K pollers: each one owns its Redis connection, and the blocking
connection is the semaphore of the slot, so K is the whole concurrency
of the process. At least one: a worker role with zero pollers is a
misconfiguration, not a rest.
Takes the configured value as an argument, so the tests do not
journey through the global config; the zero-arity reads it live."
([]
(concurrency (c/get cf/config :exporter-worker-concurrency)))
([value]
(max 1 (if (number? value) value 2))))
(defn queue-key
"The Redis list the dispatcher pushes this queue's payloads through:
`penpot.worker.queue:<tenant>:<queue>`."
([] (queue-key (cf/get :tenant)))
([tenant]
(str "penpot.worker.queue:" tenant ":" queue-name)))

View File

@ -0,0 +1,95 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns exporter.consumer.plan
"The render plan of an assets run.
The items the backend froze in the job become the renders this
process runs: file names built once and kept free of duplicates, then
grouped by scale and type, one render per group on the wasm engine
and one DOM page per partition of fifty on the browser engine.
Every item of the job names the same file, page and share: the
creation command refuses mixed scopes, so grouping by scale and
type never mixes batches that render against different pages.
`exporter.consumer` runs whatever this namespace plans, and this
namespace only plans."
(:require
[app.common.data :as d]
[cuerdas.core :as str]
[exporter.renderer :as renderer]
[exporter.util.mime :as mime]))
;; Regex to clean namefiles
(def sanitize-file-regex #"[\\/:*?\"<>|]")
(defn count-objects
[exports]
(reduce + 0 (map (comp count :objects) exports)))
(defn- assoc-file-name
"A transducer that assocs a candidate filename and avoid duplicates"
[]
(letfn [(find-candidate [params used]
(loop [index 0]
(let [candidate (str (:name params)
(:suffix params "")
(when (pos? index)
(str/concat "-" (inc index)))
(mime/get-extension (:type params)))]
(if (contains? used candidate)
(recur (inc index))
candidate))))]
(fn [rf]
(let [used (volatile! #{})]
(fn
([] (rf))
([result] (rf result))
([result params]
(let [candidate (find-candidate params @used)
params (assoc params :filename candidate)]
(vswap! used conj candidate)
(rf result params))))))))
(def ^:const ^:private
default-partition-size 50)
(defn prepare-exports
[exports token is-wasm]
(letfn [(process-group [[part1 :as group]]
;; The browser path renders one DOM page per partition; wasm
;; takes the whole group in one render.
(if (renderer/wasm? {:is-wasm is-wasm :type (:type part1)})
[(build-render group)]
(sequence (comp (partition-all default-partition-size)
(map build-render))
group)))
(build-render [[part1 :as part]]
(cond-> {:file-id (:file-id part1)
:page-id (:page-id part1)
:name (:name part1)
:token token
:type (:type part1)
:scale (:scale part1)
:is-wasm (boolean is-wasm)
:objects (mapv part-entry->object part)}
;; a nil share-id is not an absent one for the render
;; spec: only name it when the item came through one
(some? (:share-id part1)) (assoc :share-id (:share-id part1))))
(part-entry->object [entry]
{:id (:object-id entry)
:filename (:filename entry)
:name (:name entry)
:suffix (:suffix entry)})]
(let [xform (comp
(map #(assoc % :token token))
(assoc-file-name))]
(->> (sequence xform exports)
(d/group-by (juxt :scale :type))
(map second)
(into [] (mapcat process-group))))))

View File

@ -0,0 +1,171 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns exporter.consumer.worker
"The consumer of the backend `:exporter` queue.
K pollers run in parallel, each one owning its Redis connection: the
blocking BLPOP of a poller holds that connection for the whole wait,
and the poller does not come back to the queue until the job it
claimed has settled, so the connection is both the semaphore of the
slot and the backpressure — nothing the queue receives can appear
twice taken, and no process ever runs more exports than it has
connections.
The dispatcher pushes `[job-id, scheduled-at]` as JSON into
`penpot.worker.queue:<tenant>:exporter`. Corrupt payloads are logged
and dropped; a Redis hiccup is a lost turn, not a lost job: the
poller waits a bit and pops again (what was already delivered is
claimed, and the loop does not stop the process). Stopping quits
every connection and ends every loop: unlike the legacy worker,
whose loops kept failing turns after the stop, no poller survives
its handle.
The worker is a system component (`:exporter.consumer/worker`): its
config carries the queue knobs (`:concurrency`, `:queue-key`, the
`:connect` factory) and the `:renderer` it renders through (the
`:exporter/renderer` service, injected by ref). There are no promesa
chains in this namespace."
(:require
["ioredis" :as redis]
[app.common.logging :as l]
[exporter.config :as cf]
[exporter.consumer :as consumer]
[exporter.consumer.api :as api]
[exporter.consumer.config :as ccfg]
[exporter.utils.system :as system]))
(def ^:private poll-timeout-s 5)
(def ^:private reconnect-delay-ms 1000)
(def ^:private drain-poll-ms 100)
(def ^:private default-shutdown-grace-ms 30000)
(defn- wait
[ms]
(js/Promise. (fn [resolve] (js/setTimeout resolve ms))))
(defn read-payload
"The JSON payload of the dispatcher: `job-id` and the `scheduled-at`
of the row when it was pushed. Anything else is a corrupt payload:
none, and the caller drops it."
[payload]
(try
(let [[job-id scheduled-at] (js->clj (js/JSON.parse payload))]
{:job-id job-id :scheduled-at scheduled-at})
(catch :default cause
(l/warn :hint "corrupt queue payload dropped"
:payload (str payload) :cause cause)
nil)))
(defn ^:async process
"One payload of the queue: claim the job, and when the backend gives
it to this poller, run the export to its settle. `skip` means
something else won it (or it is gone): the payload is dropped, with
the state the backend says."
[cfg {:keys [job-id scheduled-at]}]
(let [answer (await (api/claim-job job-id scheduled-at))]
(if (= :skip (:action answer))
(l/info :hint "skipping job" :job-id (str job-id)
:name (:name answer) :status (:status answer))
(do
(l/info :hint "running job" :job-id (str job-id) :name (:name answer))
(await (consumer/run-export (:renderer cfg) (:exporter/tmpdir cfg) {:job-id job-id} (:params answer)))
(l/info :hint "job settled" :job-id (str job-id))
nil))))
;; ---- THE POLLER
(defn ^:async poll-once
"One turn: pop the queue, claim what came, and wait for the settle.
An empty pop and a dropped payload both resolve; a claimed job is
awaited the whole run."
[conn key on-payload]
(let [answer (await (.blpop ^js conn key poll-timeout-s))
payload (second answer)
claim (when (some? payload) (read-payload payload))]
(when (some? claim)
(await (on-payload claim))
nil)))
(defn- ^:async poll-loop
"The wait of one poller, turn after turn, until the handle stops. A
failed turn (Redis down, a bug) is logged, never allowed to end the
loop; a delay keeps the retry from spinning while the connection
rebuilds. An explicit loop, not recursion: a self-call under `await`
would retain one async frame per turn on a process that lives for
weeks."
[handle conn key on-payload]
(loop []
(when @(:running handle)
(try
(await (poll-once conn key on-payload))
(catch :default cause
(l/error :hint "poller turn failed" :key key :cause cause)
(await (wait reconnect-delay-ms))))
(recur))))
(defn- open-connection
"One Redis connection of one poller. ioredis reconnects on its own
strategy, so a fallen Redis comes back with this process doing
nothing but wait (and the failed pop comes right after)."
[]
(new redis/default (cf/get :redis-uri)))
;; ---- THE START
(defn ^:async start
"Launches the pollers over the queue and resolves the handle their
`stop` needs. The temp area boots clean before the first poller: the
tmpdir service drops what a previous process left on its init."
[{:keys [concurrency queue-key connect] :as cfg}]
(let [running (atom true)
inflight (atom 0)
handle {:running running :conns (atom []) :inflight inflight}
on-payload (^:async fn [claim]
(swap! inflight inc)
(try
(await (process cfg claim))
(finally
(swap! inflight dec))))
open (or connect open-connection)
k (or concurrency (ccfg/concurrency))]
(l/info :hint "worker started" :pollers k :queue queue-key)
(doseq [_ (range k)]
(let [conn (open)]
(swap! (:conns handle) conj conn)
(poll-loop handle conn queue-key on-payload)))
handle))
(defn- ^:async drain-inflight
[inflight deadline]
(when (and (pos? @inflight)
(< (js/Date.now) deadline))
(await (wait drain-poll-ms))
(await (drain-inflight inflight deadline))))
(defn ^:async stop
"Ends the pollers, closing their connections and their loops: the
running flag falls first, so a turn that ends after the quit does
not pop again, then the in-flight runs drain within the grace
period before the pools halt behind them. Idempotent: a second stop
quits nothing."
([handle] (stop handle nil))
([handle {:keys [grace-ms] :or {grace-ms default-shutdown-grace-ms}}]
(when (and (some? handle)
(compare-and-set! (:running handle) true false))
(await (js/Promise.allSettled
(mapv (fn [conn] (.quit ^js conn)) @(:conns handle))))
(let [inflight (or (:inflight handle) (atom 0))]
(await (drain-inflight inflight (+ (js/Date.now) grace-ms))))
nil)))
(defmethod system/init-key ::consumer/worker
[_ cfg]
(start cfg))
(defmethod system/halt-key ::consumer/worker
[_ handle]
(stop handle))

View File

@ -0,0 +1,69 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns exporter.jobs
"The local arm of the job life this process runs.
The durable row of a job is the backend's business: the worker claims
it, beats against it and settles it through the management API. This
namespace keeps only the runtime bit that cannot live in a row — the
cancel flag the parts of a run read between renders — keyed by the
job id of the claimed job.
This is deliberately smaller than the legacy `app.jobs`: the abort
itself belongs to whoever holds the render (the per-lease interval of
`exporter.wasm.scope` terminates its own worker when the check
fails), so no terminate callbacks and no shared-array signal travel
through here. The only surface toward the renderers is one zero-arg
check constructor per job."
(:require
[app.common.exceptions :as ex]))
(defonce ^:private registry (atom {}))
(defn register
"The runtime record of a job this process runs: one cancel flag,
down until something marks it."
[job-id]
(swap! registry assoc (str job-id) {:cancelled? false})
nil)
(defn cancelled?
"True once the job has been marked cancelled in this process. The
check between objects reads it; inside a render the leased worker
reads its own abort instead."
[job-id]
(boolean (:cancelled? (get @registry (str job-id)))))
(defn mark-cancelled
"Signals one job of this process as cancelled: the flag the checks
read goes up. A job this process does not own, or a settled one, is
a no-op. Returns true when this call marked it, nil otherwise."
[job-id]
(let [k (str job-id)
rt (get @registry k)]
(when (some? rt)
(when-not (:cancelled? rt)
(swap! registry assoc-in [k :cancelled?] true)
true))))
(defn release
"Drops the local runtime entry once the job settled. The row stays in
the backend until it is read once more or expires."
[job-id]
(swap! registry dissoc (str job-id))
nil)
(defn check-cancelled
"The zero-arg check of one job: silent while it lives, raising
`:job-cancelled` once marked. The same fn travels into the render
task and runs between units of work."
[job-id]
(fn []
(when (cancelled? job-id)
(throw (ex/error :type :internal
:code :job-cancelled
:hint "export job was cancelled")))))

View File

@ -0,0 +1,185 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns exporter.main
"Entry point of the new exporter tree, modeled on the backend's
`app.main`: a `system-config` map wired with refs, and `start`,
`stop` and `restart` managing the running system.
Render workers run this same bundle, so the thread decides what gets
booted: one render worker (`exporter.wasm.worker`, owning the
headless module), or the system of the main thread. Without the guard a
warmed pool worker would boot pools and pollers of its own and never
post its `ready` handshake."
(:require
["node:process" :as proc]
["node:worker_threads" :as wt]
[app.common.logging :as l]
[exporter.browser :as browser]
[exporter.config :as cf]
[exporter.consumer :as-alias consumer]
[exporter.consumer.api :as api]
[exporter.consumer.config :as ccfg]
[exporter.consumer.worker]
[exporter.tmpdir]
[exporter.utils.system :as system]
[exporter.wasm :as-alias wasm]
[exporter.wasm.pool]
[exporter.wasm.worker :as wasm.worker]))
(l/setup! {:exporter :info})
(defn system-config
"The production wiring, built at every call so a reboot picks up
configuration changes without reloading namespaces. Env-derived
values are read here, at the wiring layer; the components themselves
take plain data. The render service owns the render config (pools by
ref plus the static config); the worker only names the renderer it
renders through."
[]
{::browser/pool
{:max (ccfg/concurrency)}
::wasm/pool
{:max (ccfg/concurrency)
:min (cf/get :wasm-worker-pool-min)
;; the knob is in seconds, the pool wants milliseconds
:idle-timeout (* 1000 (cf/get :wasm-worker-idle-timeout))}
:exporter/tmpdir
{:path (cf/get :tempdir)
:job-ttl (cf/get :exporter-job-ttl 3600)}
:exporter/renderer
{::browser/pool (system/ref :exporter.browser/pool)
::wasm/pool (system/ref :exporter.wasm/pool)
:exporter/tmpdir (system/ref :exporter/tmpdir)
:base-uri (cf/get-internal-uri)
:public-uri (cf/get :public-uri)
:image-cache-size (cf/get :wasm-worker-image-cache-size)
:svgo? (contains? cf/flags :exporter-svgo)}
::consumer/worker
{:renderer (system/ref :exporter/renderer)
:exporter/tmpdir (system/ref :exporter/tmpdir)
:concurrency (ccfg/concurrency)
:queue-key (ccfg/queue-key)}})
;; The running system map, or nil when nothing is started.
;; Counterpart of the backend's `app.system/system` var.
(defonce system
(atom nil))
;; The config the running system booted with. `restart` reboots it,
;; so tests can cycle a fake config without touching production.
(defonce ^:private config-used
(atom nil))
(defn ^:async start-custom
"Boot the given config, halting the running system first, like the
backend's `start-custom`. Resolves to `:started`."
[config]
(when-let [sys @system]
(await (system/halt sys)))
(swap! config-used (constantly config))
(reset! system nil)
(let [sys (await (system/init config))]
(reset! system sys)
:started))
(declare shutdown)
;; Install-once guard: `start` runs on every hot reload, and piling
;; process listeners would end in MaxListenersExceeded warnings.
(defonce ^:private handlers-installed? (atom false))
(defn install-process-handlers
"Register the process-level handlers: log uncaught exceptions instead
of dying silently, and drain on SIGTERM/SIGINT. Signals only reach the
main thread; exiting in a render worker would take down that worker
rather than the process. Runs on `start`, never on namespace load,
and only the first call takes effect."
[]
(when (compare-and-set! handlers-installed? false true)
(.on proc/default "uncaughtException"
(fn [cause]
(js/console.error cause)))
(when ^boolean wt/isMainThread
(.on proc/default "SIGTERM" (fn [] (shutdown)))
(.on proc/default "SIGINT" (fn [] (shutdown))))))
(defn ^:async start
"Boot the production wiring. Resolves to `:started`. On a render
worker thread boots the render worker instead: it runs this same
bundle and owns no pools or pollers."
[]
(install-process-handlers)
(if ^boolean wt/isMainThread
(do
(l/info :msg "starting exporter"
:workers (ccfg/concurrency)
:version (:full cf/version))
(await (start-custom (system-config))))
(do
(wasm.worker/main)
:started)))
(defn ^:async stop
"Halt the running system, if any. Resolves to `:stopped`."
[]
(when-let [sys @system]
(await (system/halt sys)))
(reset! system nil)
:stopped)
(defn ^:async shutdown
"Graceful shutdown: halt the running system (the worker stops
intake first, in-flight runs drain within their grace period, then
the pools halt in reverse dependency order) and exit. A halt
failure still exits, so a deploy never leaves the process behind.
`exit` is a seam for the tests, which cannot stub the process."
([] (shutdown proc/exit))
([exit]
(try
(await (stop))
(catch :default cause
(l/warn :hint "graceful shutdown halt failed" :cause cause)))
(api/close-client!)
(exit 0)))
(defn ^:async restart
"Halt and reboot whatever config is running (or the production
wiring when nothing ever started). Resolves to `:restarted`."
[]
(await (stop))
(await (start-custom (or @config-used (system-config))))
:restarted)
(defn ^:dev/before-load-async on-before-load
"Hot reload hook: halt the running system before shadow swaps the
code. `done` always runs, so a halt failure is logged, never a
stuck reload."
[done]
(-> (stop)
(.then (fn [_] (done)))
(.catch (fn [cause]
(l/warn :hint "hot reload stop failed"
:cause (ex-message cause))
(done)))))
(defn ^:dev/after-load-async on-after-load
"Hot reload hook: boot the production wiring after shadow swaps the
code. `done` always runs, so a boot failure is logged, never a
stuck reload."
[done]
(-> (start)
(.then (fn [_] (done)))
(.catch (fn [cause]
(l/warn :hint "hot reload start failed"
:cause (ex-message cause))
(done)))))
(def main start)

View File

@ -0,0 +1,140 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns exporter.renderer
"Common renderer interface, validated with malli from day one.
One entry point: `render` takes `cfg` plus a task map
(`:exports`, `:on-object`, `:check-cancelled`), validates it, and
fires the whole batch at once — the browser exports through
`exporter.browser.scope`, the wasm ones through one leased
worker in `exporter.wasm.scope`. A single export is a batch of one.
Rendered objects travel through `on-object`; the batch itself
resolves nil once every export landed.
`cfg` is task-free infrastructure with process lifetime: the pool
services under their system keys (`:exporter.browser/pool`,
`:exporter.wasm/pool`) plus the static render config
(`:base-uri`, `:public-uri`, `:svgo?`) — a view over the running
system map that each domain reads with its own keys. The task
map holds everything per-task: the `exports` data plus the two
injected indirections, `on-object` (the per-object collector) and
`check-cancelled` (the zero-arg check-cancelled raising
`:job-cancelled`). One closed schema validates the whole task, so a
bad batch — or a non-fn injection — never touches a pool. Each
driver takes the same `[cfg params on-object check-cancelled]` shape and
implements cancel internally: cooperative checkpoints on the
browser, checkpoints plus a mid-render abort of its own leased
worker on wasm.
The renderer is also a system service (`:exporter/renderer`): its
config value carries the pools as refs plus the static config, and
its running instance is the render fn with that config closed over,
which is what the consumer renders through — the consumer never
calls this namespace directly."
(:require
[app.common.schema :as sm]
[exporter.browser.scope :as bscope]
[exporter.utils.system :as system]
[exporter.wasm.render :as wrender]
[exporter.wasm.scope :as scope]))
(def schema:type
[:enum :png :jpeg :webp :pdf :svg])
(def schema:object
[:map
[:id ::sm/uuid]
[:name :string]
[:suffix :string]
[:filename :string]
[:share-id {:optional true} ::sm/uuid]])
(def schema:objects
[:vector {:min 1} schema:object])
(def schema:render-params
[:map {:closed true}
[:file-id ::sm/uuid]
[:page-id ::sm/uuid]
[:scale ::sm/number]
[:token :string]
[:type schema:type]
[:objects schema:objects]
[:name :string]
[:share-id {:optional true} ::sm/uuid]
[:skip-children {:optional true} ::sm/boolean]
[:is-wasm {:optional true} ::sm/boolean]
[:job-id {:optional true} ::sm/uuid]])
(def schema:exports
[:vector {:min 1} schema:render-params])
(def schema:task
[:map {:closed true}
[:exports schema:exports]
[:on-object ::sm/fn]
[:check-cancelled ::sm/fn]])
(def ^:private check-task
(sm/check-fn schema:task :hint "invalid render task"))
(defn wasm?
"Whether `params` renders with render-wasm rather than a browser."
[{:keys [is-wasm]}]
(boolean is-wasm))
(defn- render-env
"The render environment of one batch: internal and public endpoints,
the temp area and the image cache budget. Read from the service config
(wired from the environment); the worker thread never resolves any of
it itself."
[cfg]
{:internal-uri (:base-uri cfg)
:public-uri (:public-uri cfg)
:tmpdir (:exporter/tmpdir cfg)
:image-cache-size (:image-cache-size cfg)})
(defn ^:async render
"Renders every export, firing them all at once: the browser ones
through their own scope, the wasm ones through one leased
worker. Assumes a valid task — validation lives one level up, at
the service entry, so through here a bad task never touches a pool
because it never arrives. Rejects on the first failure."
[cfg {:keys [exports on-object check-cancelled]}]
(let [cfg (assoc cfg
::on-object on-object
::check-cancelled check-cancelled)
;; Everything the worker thread needs beyond the export
;; itself rides in the posted params: the worker reads no
;; config and no globals.
wexports (map #(assoc % ::wrender/env (render-env cfg))
(filter wasm? exports))
bexports (remove wasm? exports)]
(await (js/Promise.all
(cond-> []
(seq bexports)
(conj (bscope/run cfg bexports))
(seq wexports)
(conj (scope/run cfg
(fn [render-fn]
(js/Promise.all (map render-fn wexports))))))))
nil))
(defn render-fn
"The render entry of one running system: validates the task map and
dispatches it through the facade with `cfg` closed over, resolving
nil once every export landed. The single boundary where a bad batch
is rejected before anything is leased — and the dependency the
consumer renders through, so the consumer never calls the facade
directly."
[cfg]
(fn [task]
(render cfg (check-task task))))
(defmethod system/init-key :exporter/renderer
[_ cfg]
(render-fn cfg))

View File

@ -0,0 +1,509 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns exporter.renderer.browser
"Renders through a browser page: bitmap, pdf and svg in one namespace.
Every flow renders one partition of objects per browser checkout:
navigate once, then capture object by object, calling `on-object`
with each object carrying its `:path`. `on-object` may return a
plain value or a promise; it is always awaited.
`cfg` is task-free infrastructure: the `:exporter.browser/pool`, the
render `:base-uri` and `:public-uri`, the `:exporter/tmpdir` area and
the `:svgo?` flag. The
per-task injections (`on-object`, `check-cancelled`) ride positional. There
are no promesa chains in this namespace."
(:require
["@penpot/svgo" :as svgo]
["xml-js" :as xml]
[app.common.data :as d]
[app.common.data.macros :as dm]
[app.common.exceptions :as ex]
[app.common.logging :as l]
[app.common.types.color :as ctc]
[app.common.uri :as u]
[clojure.walk :as walk]
[cuerdas.core :as str]
[exporter.browser :as browser]
[exporter.renderer.svg-gradient :as svg-gradient]
[exporter.shell :as shell]
[exporter.util.mime :as mime]))
(l/set-level! :trace)
(defn- raise-if-cancelled
"Cooperative checkpoint over the injected zero-arg check-cancelled. This
backend never blocks indefinitely (every step yields), so
yielding-point checks are the whole cancel story here. The check-cancelled is
mandatory: without one it raises `:check-cancelled-missing` instead
of running unprotected."
[check-cancelled]
(if (fn? check-cancelled)
(check-cancelled)
(throw (ex/error :type :assertion
:code :check-cancelled-missing
:hint "browser render needs a zero-arg check-cancelled"))))
(defn- prepare-options
[uri token scale]
#js {:screen #js {:width browser/default-viewport-width
:height browser/default-viewport-height}
:viewport #js {:width browser/default-viewport-width
:height browser/default-viewport-height}
:locale "en-US"
:storageState #js {:cookies (browser/create-cookies uri {:token token})}
:deviceScaleFactor scale
:userAgent browser/default-user-agent})
(defn- render-uri
[base-uri query-params]
(-> base-uri
(u/ensure-path-slash)
(u/join "render.html")
(assoc :query (u/map->query-string query-params))))
;; --- BITMAP (png, jpeg, webp)
(defn- ^:async render-bitmap-object
[tmpdir page type object on-object]
(let [path (await (shell/tempfile tmpdir :prefix "penpot.tmp.bitmap." :suffix (mime/get-extension type)))
node (browser/select page (str/concat "#screenshot-" (:id object)))]
(await (browser/wait-for node))
(case type
:png (await (browser/screenshot node {:omit-background? true :type type :path path}))
:jpeg (await (browser/screenshot node {:omit-background? false :type type :path path}))
:webp (let [png-path (await (shell/tempfile tmpdir :prefix "penpot.tmp.bitmap." :suffix ".png"))]
;; playwright only supports jpg and png, we need to convert it afterwards
(await (browser/screenshot node {:omit-background? true :type :png :path png-path}))
(await (shell/run-cmd "convert" png-path "-quality" "100" (str "WEBP:" path)))))
(await (on-object (assoc object :path path)))))
(defn- ^:async render-bitmap-page
[tmpdir uri page type objects on-object]
(l/info :uri (str uri))
;; navigate to the page and perform basic setup
(await (browser/nav page (str uri)))
(await (browser/sleep page 1000)) ; the good old fix with sleep
(await (browser/wait-for-fonts page))
(await (browser/eval page (js* "() => document.body.style.background = 'transparent'")))
;; take the screnshot of requested objects, one by one
(await (js/Promise.all (mapv (fn [object] (render-bitmap-object tmpdir page type object on-object))
objects)))
nil)
(defn- ^:async render-bitmap
[cfg {:keys [file-id page-id share-id token scale type objects skip-children]} on-object]
(let [query {:file-id file-id
:page-id page-id
:share-id share-id
:object-id (mapv :id objects)
:route "objects"
:skip-children skip-children}
uri (render-uri (:base-uri cfg) query)]
(await (browser/exec (:exporter.browser/pool cfg)
(prepare-options uri token scale)
(fn [page] (render-bitmap-page (:exporter/tmpdir cfg) uri page type objects on-object))))))
;; --- PDF
(defn- prepare-pdf-uri
[base-uri {:keys [file-id page-id share-id]} object-id]
(let [query {:file-id file-id
:page-id page-id
:share-id share-id
:object-id object-id
:route "objects"}]
(-> base-uri
(u/join "render.html")
(assoc :query (u/map->query-string query)))))
(defn- sync-page-size
[dom]
(browser/eval dom
(fn [elem]
;; IMPORTANT: No CLJS runtime allowed. Use only JS
;; primitives. This runs in a context without access to
;; cljs.core. Avoid any functions that transpile to
;; cljs.core/* calls, as they will break in the browser
;; runtime.
(let [width (.getAttribute ^js elem "width")
height (.getAttribute ^js elem "height")
style-node (let [node (.createElement js/document "style")]
(.appendChild (.-head js/document) node)
node)]
(set! (.-textContent style-node)
(dm/str "@page { size: " width "px " height "px; margin: 0; }\n"
"html, body, #app { margin: 0; padding: 0; width: " width "px; height: " height "px; overflow: visible; }"))))))
(defn- ^:async render-pdf-object
[tmpdir page base-uri params object on-object]
(let [uri (prepare-pdf-uri base-uri params (:id object))
path (await (shell/tempfile tmpdir :prefix "penpot.tmp.pdf." :suffix (mime/get-extension (:type params))))]
(l/info :uri (str uri))
(await (browser/nav page uri))
(let [dom (browser/select page (dm/str "#screenshot-" (:id object)))]
(await (browser/wait-for dom))
(await (sync-page-size dom))
(await (browser/screenshot dom {:full-page? true}))
(await (browser/sleep page 2000)) ; the good old fix with sleep
(await (browser/wait-for-fonts page))
(await (browser/pdf page {:path path}))
(await (on-object (assoc object :path path))))))
(defn- ^:async render-pdf-page
[tmpdir base-uri page params objects on-object check-cancelled]
(doseq [object objects]
(raise-if-cancelled check-cancelled)
(await (render-pdf-object tmpdir page base-uri params object on-object)))
nil)
(defn- ^:async render-pdf
[cfg {:keys [token scale] :as params} on-object check-cancelled]
(let [base-uri (-> (:base-uri cfg)
(u/ensure-path-slash))]
(await (browser/exec (:exporter.browser/pool cfg)
(prepare-options base-uri token scale)
(fn [page]
(render-pdf-page (:exporter/tmpdir cfg) base-uri page params (:objects params) on-object check-cancelled))))))
;; --- SVG
(defn- xml->clj
[data]
(js->clj (xml/xml2js data)))
(defn- clj->xml
[data]
(xml/js2xml (clj->js data)))
(defn- ^boolean element?
[item]
(and (map? item)
(= "element" (get item "type"))))
(defn- ^boolean foreign-object-element?
[item]
(and (element? item)
(= "foreignObject" (get item "name"))))
(defn- ^boolean empty-defs-element?
[item]
(and (= (get item "name") "defs")
(nil? (get item "attributes"))
(nil? (get item "elements"))))
(defn- ^boolean empty-path-element?
[item]
(and (= (get item "name") "path")
(let [d (get-in item ["attributes" "d"])]
(or (str/blank? d)
(nil? d)
(str/empty? d)))))
(defn- flatten-toplevel-svg-elements
"Flattens XML data structure if two nested top-side SVG elements found."
[item]
(if (and (= "svg" (get-in item ["elements" 0 "name"]))
(= "svg" (get-in item ["elements" 0 "elements" 0 "name"])))
(update-in item ["elements" 0] assoc "elements" (get-in item ["elements" 0 "elements" 0 "elements"]))
item))
(defn- replace-text-nodes
"Function responsible of replace the foreignObject elements on the
provided XML with the previously rasterized PATH's."
[xmldata nodes]
(letfn [(replace-fobject [item]
(if (foreign-object-element? item)
(let [id (get-in item ["attributes" "id"])
node (get nodes id)]
(if node
(:svgdata node)
item))
item))
(process-element [item xform]
(d/update-when item "elements" #(into [] xform %)))]
(let [xform (comp (remove empty-defs-element?)
(remove empty-path-element?)
(map replace-fobject))]
(->> xmldata
(xml->clj)
(flatten-toplevel-svg-elements)
(walk/prewalk (fn [item]
(cond-> item
(element? item)
(process-element xform))))
(clj->xml)))))
(defn- parse-viewbox
"Parses viewBox string into width & height map."
[data]
(let [[width height] (->> (str/split data #"\s+")
(drop 2)
(map d/parse-double))]
{:width width
:height height}))
(defn- replace-internal-uris
"Replaces internal-uri references with public-uri in SVG output.
This ensures that font URLs and other resource references in the
exported SVG use the public-facing URI accessible to end users."
[svg-content internal-uri public-uri]
(let [internal-uri (str internal-uri)
public-uri (str public-uri)]
(if (and (not= internal-uri public-uri)
(str/includes? svg-content internal-uri))
(str/replace svg-content internal-uri public-uri)
svg-content)))
(defn- ^:async convert-to-ppm
[pngpath]
(let [ppmpath (str/concat pngpath "origin.ppm")]
(l/trace :fn :convert-to-ppm :path ppmpath)
(await (shell/run-cmd "convert" pngpath ppmpath))
ppmpath))
(defn- ^:async trace-color-mask
[pbmpath]
(l/trace :fn :trace-color-mask :pbmpath pbmpath)
(let [svgpath (str/concat pbmpath ".svg")]
(await (shell/run-cmd "potrace" "--flat" "-b" "svg" pbmpath "-o" svgpath))
svgpath))
(defn- ^:async generate-color-layer
[ppmpath color]
(when-not (ctc/hex-color-string? color)
(ex/raise :type :validation
:code :invalid-color
:hint (str "invalid hex color: " color)))
(l/trace :fn :generate-color-layer :ppmpath ppmpath :color color)
(let [pbmpath (str/concat ppmpath ".mask-" (subs color 1) ".pbm")
stdout (await (shell/run-cmd "ppmcolormask" color ppmpath))]
(await (shell/write-file pbmpath stdout))
(let [svgpath (await (trace-color-mask pbmpath))
data (await (shell/read-file svgpath))
data (xml->clj data)
data (get-in data ["elements" 1])]
{:color color
:svgdata data})))
(defn- set-path-color
[id color mapping node]
(let [color-mapping (get mapping color)]
(cond
(and (some? color-mapping)
(= "transparent" (get color-mapping "type")))
(update node "attributes" assoc
"fill" (get color-mapping "hex")
"fill-opacity" (get color-mapping "opacity"))
(and (some? color-mapping)
(= "gradient" (get color-mapping "type")))
(update node "attributes" assoc
"fill" (str "url(#gradient-" id "-" (subs color 1) ")"))
:else
(update node "attributes" assoc "fill" color))))
(defn- get-gradients
[id mapping]
(->> mapping
(filter (fn [[_color data]]
(= (get data "type") "gradient")))
(mapv (partial svg-gradient/data->gradient-def id))))
(defn- join-color-layers
[{:keys [id x y width height mapping] :as node} layers]
(l/trace :fn :join-color-layers :mapping mapping)
(loop [result (-> (:svgdata (first layers))
(assoc "elements" []))
layers (seq layers)]
(if-let [{:keys [color svgdata]} (first layers)]
(recur (->> (get svgdata "elements")
(filter #(= (get % "name") "g"))
(map (partial set-path-color id color mapping))
(update result "elements" into))
(rest layers))
;; Now we have the result containing the svgdata of a
;; SVG with all text layers. Now we need to transform
;; this SVG to G (Group) and remove unnecessary metadata
;; objects.
(let [vbox (-> (get-in result ["attributes" "viewBox"])
(parse-viewbox))
transform (str/fmt "translate(%s, %s) scale(%s, %s)" x y
(/ width (:width vbox))
(/ height (:height vbox)))
gradient-defs (get-gradients id mapping)
elements
(->> (get result "elements")
(mapv (fn [group]
(let [paths (get group "elements")]
(if (= 1 (count paths))
(let [path (first paths)]
(update path "attributes"
(fn [attrs]
(-> attrs
(d/merge (get group "attributes"))
(update "transform" #(str transform " " %))))))
(update-in group ["attributes" "transform"] #(str transform " " %)))))))
elements (cond->> elements
(seq gradient-defs)
(into [{"type" "element" "name" "defs" "attributes" {}
"elements" gradient-defs}]))]
(-> result
(assoc "name" "g")
(assoc "attributes" {})
(assoc "elements" elements))))))
(defn- ^:async convert-to-svg
[ppmpath node]
(l/trace :fn :convert-to-svg :ppmpath ppmpath :colors (:colors node))
(let [layers (await (js/Promise.all (mapv (partial generate-color-layer ppmpath)
(:colors node))))]
(join-color-layers node layers)))
(defn- ^:async trace-node
[tmpdir node]
(l/trace :fn :trace-node)
(let [pngpath (await (shell/tempfile tmpdir :prefix "penpot.tmp.render.svg.parse."
:suffix ".origin.png"))]
(await (shell/write-file pngpath (:data node)))
(let [ppmpath (await (convert-to-ppm pngpath))
svgdata (await (convert-to-svg ppmpath node))]
(-> node
(dissoc :data)
(assoc :svgdata svgdata)))))
(defn- extract-element-attrs
[^js element]
(let [^js attrs (.. element -attributes)
^js colors (.. element -dataset -colors)
^js mapping (.. element -dataset -mapping)]
#js {:id (.. attrs -id -value)
:x (.. attrs -x -value)
:y (.. attrs -y -value)
:width (.. attrs -width -value)
:height (.. attrs -height -value)
:colors (.split colors ",")
:mapping (js/JSON.parse mapping)}))
(defn- ^:async extract-single-node
[[shot node]]
(l/trace :fn :extract-single-node)
(let [attrs (await (browser/eval node extract-element-attrs))]
{:id (unchecked-get attrs "id")
:x (unchecked-get attrs "x")
:y (unchecked-get attrs "y")
:width (unchecked-get attrs "width")
:height (unchecked-get attrs "height")
:colors (vec (unchecked-get attrs "colors"))
:mapping (js->clj (unchecked-get attrs "mapping"))
:data shot}))
(defn- ^:async resolve-text-node
[page node]
(let [attrs (await (browser/eval node extract-element-attrs))
id (unchecked-get attrs "id")
text-node (browser/select page (str "#screenshot-text-" id " foreignObject"))
shot (await (browser/screenshot text-node {:omit-background? true :type "png"}))]
[shot node]))
(defn- ^:async extract-txt-node
[tmpdir page item]
(let [[shot node] (await (resolve-text-node page item))
single (await (extract-single-node [shot node]))]
(await (trace-node tmpdir single))))
(defn- ^:async extract-txt-nodes
[tmpdir page {:keys [id]}]
(l/trace :fn :process-text-nodes)
(let [nodes (await (browser/select-all page (str/concat "#screenshot-" id " foreignObject")))
nodes (await (js/Promise.all (mapv (partial extract-txt-node tmpdir page) nodes)))]
(d/index-by :id nodes)))
(defn- ^:async extract-svg
[page {:keys [id]}]
(let [node (browser/select page (str/concat "#screenshot-" id))]
(await (browser/wait-for node))
(await (browser/eval node (fn [elem] (.-outerHTML ^js elem))))))
(defn sanitize-svg-content
"SVG standard don't allow the entity nbsp. &#160; is equivalent but
compatible with SVG."
[content]
(str/replace content "&nbsp;" "&#160;"))
(defn- ^:async render-svg-object
[cfg page type object on-object]
(let [path (await (shell/tempfile (:exporter/tmpdir cfg) :prefix "penpot.tmp.render.svg." :suffix (mime/get-extension type)))
node (browser/select page (str/concat "#screenshot-" (:id object)))]
(await (browser/wait-for node))
(let [xmldata (await (extract-svg page object))
txtdata (await (extract-txt-nodes (:exporter/tmpdir cfg) page object))
result (replace-text-nodes xmldata txtdata)
result (sanitize-svg-content result)
result (if (:svgo? cfg)
(svgo/optimize result svgo/defaultOptions)
result)
result (replace-internal-uris result (:base-uri cfg) (:public-uri cfg))]
(await (shell/write-file path result))
(await (on-object (assoc object :path path))))))
(defn- ^:async render-svg-page
[cfg uri page type objects on-object]
(l/info :uri (str uri))
;; navigate to the page and perform basic setup
(await (browser/nav page (str uri)))
(await (browser/sleep page 1000)) ; the good old fix with sleep
(await (browser/wait-for-fonts page))
;; take the screnshot of requested objects, one by one
(await (js/Promise.all (mapv (fn [object] (render-svg-object cfg page type object on-object))
objects)))
nil)
(defn- ^:async render-svg
[cfg {:keys [file-id page-id share-id token scale type objects]} on-object]
(let [query {:file-id file-id
:page-id page-id
:share-id share-id
:render-embed true
:object-id (mapv :id objects)
:route "objects"}
uri (render-uri (:base-uri cfg) query)]
(await (browser/exec (:exporter.browser/pool cfg)
(prepare-options uri token scale)
(fn [page] (render-svg-page cfg uri page type objects on-object))))))
(defn ^:async render
"Renders one partition of objects through a browser page: bitmap
(png, jpeg, webp), pdf or svg depending on the type in `params`.
`cfg` is task-free infrastructure (pools, uris, flags), while
`on-object` and `check-cancelled` ride positional as the per-task
injections — the per-object collector and the zero-arg check-cancelled
raising `:job-cancelled`.
The check-cancelled runs here at fail-fast entry plus the sequential pdf
loop; the concurrent flows rely on the entry check-cancelled and the caller's
`on-object`. Calls `on-object` with each object carrying its
`:path`, awaiting it every time, and resolves nil."
[cfg {:keys [type] :as params} on-object check-cancelled]
(raise-if-cancelled check-cancelled)
(case type
(:png :jpeg :webp) (await (render-bitmap cfg params on-object))
:pdf (await (render-pdf cfg params on-object check-cancelled))
:svg (await (render-svg cfg params on-object))
(ex/raise :type :validation
:code :unknown-render-type
:hint (str "unknown render type: " type))))

View File

@ -4,7 +4,7 @@
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.renderer.svg-gradient)
(ns exporter.renderer.svg-gradient)
(defn- get-stops
[data]

View File

@ -0,0 +1,76 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns exporter.resources
"The run's local resources: the artifact files the runner parks in
the injected temp area until the settle moves them into the job's
completion, zipped once a run packs more than one.
Same operations as the legacy `app.handlers.resources`, adapted:
the temp area arrives injected (no global), and the zip close is
awaited instead of chained. The per-file deletion timers of the old
shell are gone on purpose: the tmpdir registry owns every path and
releases it on the settle."
(:require
["archiver" :as arc]
["node:fs" :as fs]
["node:path" :as path]
[app.common.uuid :as uuid]
[cljs.core :as c]
[cuerdas.core :as str]
[exporter.util.mime :as mime]))
(defn- resource-path
[tmpdir type id]
(path/join tmpdir (str/concat "penpot.resource." (c/name type) "." id)))
(defn create
"An ephemeral artifact descriptor of `type` named `name`, parked
under the injected `tmpdir`."
[tmpdir type name]
(let [task-id (uuid/next)]
{:path (resource-path tmpdir type task-id)
:mtype (mime/get type)
:name name
:filename (str/concat (str/replace name #"[\\/:*?\"<>|]" "_") (mime/get-extension type))
:id task-id}))
(defn create-zip
[& {:keys [resource on-complete on-progress on-error]}]
(let [^js zip (new arc/ZipArchive)
^js out (fs/createWriteStream (:path resource))
on-complete (or on-complete (constantly nil))
progress (atom 0)]
(.on zip "error" on-error)
(.on zip "end" on-complete)
(.on zip "entry" (fn [data]
(let [name (unchecked-get data "name")
num (swap! progress inc)]
(on-progress {:done num :filename name}))))
(.pipe zip out)
zip))
(defn add-to-zip
[zip path name]
(.file ^js zip path #js {:name name}))
(defn ^:async close-zip
([zip] (close-zip zip nil))
([zip {:keys [timeout-ms] :or {timeout-ms 30000}}]
(await (js/Promise.
(fn [resolve reject]
(let [timer (js/setTimeout
(fn [] (reject (js/Error. "zip close timed out")))
timeout-ms)
done (fn []
(js/clearTimeout timer)
(resolve nil))
fail (fn [cause]
(js/clearTimeout timer)
(reject cause))]
(.on ^js zip "close" (fn [] (done)))
(.on ^js zip "error" (fn [cause] (fail cause)))
(.finalize ^js zip)))))))

View File

@ -0,0 +1,109 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns exporter.shell
"Shell and filesystem utilities of the new tree.
Same operations as the legacy `app.util.shell`, adapted: nothing
runs at namespace load (no log level, no global tmpdir created the
first time anything requires this) and no promesa chains (`^:async`
+ `await`). Paths are explicit everywhere: `tempfile` takes the
directory to create under, so the temp area is injected by whoever
owns it — the `:exporter/tmpdir` service at boot — instead of read
from a global. Legacy operations nobody calls did not survive the
move."
(:require
["node:child_process" :as proc]
["node:fs" :as fs]
["node:path" :as path]
[app.common.exceptions :as ex]
[app.common.logging :as l]
[app.common.time :as ct]
[app.common.uuid :as uuid]
[cuerdas.core :as str]))
(def ^:const default-deletion-delay
(* 60 60 1)) ;; 1h
(defn ensure-dir
"Creates `dir` (and parents) when missing, returning it. The one
place a directory comes into existence in the new tree; a blank path
is a wiring error, not an empty directory."
[dir]
(when (str/blank? dir)
(throw (ex/error :type :assertion
:code :tmpdir-path-missing
:hint "shell needs a directory to ensure")))
(when-not (fs/existsSync dir)
(fs/mkdirSync dir #js {:recursive true}))
dir)
(defn schedule-deletion
([path] (schedule-deletion path default-deletion-delay))
([path delay]
(let [remove-path
(fn []
(try
(when (fs/existsSync path)
(fs/rmSync path #js {:recursive true})
(l/trc :hint "tempfile permanently deleted" :path path))
(catch :default cause
(l/err :hint "error on deleting temporal file"
:path path
:cause cause))))
scheduled-at
(-> (ct/now) (ct/plus #js {:seconds delay}))]
(l/trc :hint "schedule tempfile deletion"
:path path
:scheduled-at (ct/format-inst scheduled-at))
(js/setTimeout remove-path (* delay 1000))
path)))
(defn tempfile
"A fresh path under `dir` that nothing else owns yet.
Deletion is not scheduled here: the tmpdir registry tracks every path a
job writes and releases them on the settle, and the boot sweep removes
what a crashed process left. `schedule-deletion` stays for one-off
callers that own a path the registry does not know."
[dir & {:keys [prefix suffix]
:or {prefix "penpot."
suffix ".tmp"}}]
(loop [i 0]
(if (< i 1000)
(let [path (path/join dir (str/concat prefix (uuid/next) "-" i suffix))]
(if (fs/existsSync path)
(recur (inc i))
path))
(throw (ex/error :type :internal
:code :unable-to-locate-temporal-file
:hint "unable to find a tempfile candidate")))))
(defn ^:async move
[origin-path dest-path]
(await (.rename fs/promises origin-path dest-path)))
(defn ^:async write-file
[fpath content]
(await (.writeFile fs/promises fpath content)))
(defn ^:async read-file
[fpath]
(await (.readFile fs/promises fpath)))
(defn ^:async run-cmd
[cmd & args]
(await
(js/Promise.
(fn [resolve reject]
(l/trace :fn :run-cmd :cmd cmd :args args)
(proc/execFile cmd (clj->js args) #js {:encoding "buffer"}
(fn [error stdout _stderr]
(if error
(reject error)
(resolve stdout))))))))

View File

@ -0,0 +1,100 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns exporter.tmpdir
"The temp-file area as a system component (`:exporter/tmpdir`).
Its config value names the directory (`{:path ...}`, read from the
environment at the wiring layer) plus the age that makes a leftover
an orphan (`:job-ttl`, seconds, same unit as the environment knob);
booting ensures the directory exists and cleans whatever a previous
process left behind, before anything that writes to it boots. The
running instance is the path itself, which renderers and runners
read straight from their config. Fail fast on a blank path instead
of failing the first render that writes. Named for what it owns,
not for the legacy namespace it shadows.
The area also owns what jobs park in it: every path a job tracks is
removed on its release, so temp files die with the settle instead of
an hour later (or never, on a crash)."
(:require
["node:fs/promises" :as fsp]
["node:path" :as path]
[app.common.exceptions :as ex]
[app.common.logging :as l]
[cuerdas.core :as str]
[exporter.shell :as shell]
[exporter.utils.system :as system]))
(def ^:private managed-prefix "penpot.")
(defonce ^:private tracked (atom {}))
(defn track
"Registers `path` as owned by `job-id`, so it is removed when the job
settles. Returns the path, so it wraps the call that creates it."
[job-id path]
(when (and job-id path)
(swap! tracked update (str job-id) (fnil conj #{}) path))
path)
(defn- ^:async remove-path
[path]
(try
(await (fsp/rm path #js {:recursive true :force true}))
(catch :default cause
(l/warn :hint "unable to remove job temp file" :path path :cause cause))))
(defn ^:async release
"Removes every file the job owns. Called once the job reached a
terminal state and its result has already been uploaded, so nothing
else reads them. Unknown jobs resolve at once."
[job-id]
(let [paths (get @tracked (str job-id))]
(swap! tracked dissoc (str job-id))
(when (seq paths)
(await (js/Promise.all (mapv remove-path paths)))
(l/dbg :hint "released job temp files" :job-id (str job-id) :count (count paths)))
nil))
(defn ^:async clean-orphans
"Removes managed temp files older than `job-ttl-s`. They can only be
leftovers of a previous process: every live one belongs to a job of
this process. Resolves the count it removed."
[tmpdir job-ttl-s]
(let [max-age (* 1000 job-ttl-s)
now (js/Date.now)]
(try
(let [entries (await (fsp/readdir tmpdir))
removed (volatile! 0)]
(doseq [entry (filter #(str/starts-with? % managed-prefix) entries)]
(let [fpath (path/join tmpdir entry)]
(try
(let [^js stat (await (fsp/stat fpath))]
(when (> (- now (inst-ms (.-mtime stat))) max-age)
(await (remove-path fpath))
(vswap! removed inc)))
(catch :default _ nil))))
(when (pos? @removed)
(l/info :hint "removed orphaned export temp files" :count @removed))
@removed)
(catch :default cause
(l/warn :hint "temp file cleanup failed" :cause cause)
0))))
(defn- ^:async boot
[path job-ttl]
(let [dir (shell/ensure-dir path)]
(await (clean-orphans dir job-ttl))
dir))
(defmethod system/init-key :exporter/tmpdir
[_ {:keys [path job-ttl]}]
(when (nil? job-ttl)
(throw (ex/error :type :assertion
:code :job-ttl-missing
:hint "tmpdir needs :job-ttl (seconds) in its config")))
(boot path job-ttl))

View File

@ -4,7 +4,7 @@
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.util.mime
(ns exporter.util.mime
"Mimetype and file extension helpers."
(:refer-clojure :exclude [get])
(:require
@ -29,5 +29,3 @@
:jpeg "image/jpeg"
:png "image/png"
:webp "image/webp"))

View File

@ -4,9 +4,9 @@
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.util.object
(ns exporter.util.object
"A collection of helpers for work with javascript objects."
(:refer-clojure :exclude [set! get get-in merge clone contains?])
(:refer-clojure :exclude [set! get get-in merge update clone contains?])
(:require
[cuerdas.core :as str]))
@ -46,24 +46,24 @@
[a]
(js/Object.assign #js {} a))
(defn merge!
(defn merge-into
([a b]
(js/Object.assign a b))
([a b & more]
(reduce merge! (merge! a b) more)))
(reduce merge-into (merge-into a b) more)))
(defn merge
([a b]
(js/Object.assign #js {} a b))
([a b & more]
(reduce merge! (merge a b) more)))
(reduce merge-into (merge a b) more)))
(defn set!
[obj key value]
(unchecked-set obj key value)
obj)
(defn update!
(defn update
[obj key f & args]
(let [found (get obj key ::not-found)]
(if-not (identical? ::not-found found)

View File

@ -0,0 +1,243 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns exporter.utils.system
"Async-aware Integrant-style system lifecycle for the new tree.
The shape mirrors `integrant.core`, trimmed down: a config map of
qualified keys to values, `ref`s between them, and two multimethods,
`init-key` and `halt-key`, that turn values into running components
and back. The dependency graph rides `weavejester.dependency`, the
same library Integrant uses.
The difference is async: `init` and `halt` are `^:async`, every step
is awaited, and an `init-key` method may return a plain value or a
promise. `init` never throws synchronously; every failure, including
invalid config, surfaces as a rejected promise.
Deliberately left out from Integrant: refsets, profiles, vars,
expand/converge, suspend/resume, composite keys, EDN reading and
namespace loading."
(:require
[clojure.set :as set]
[clojure.string :as str]
[clojure.walk :as walk]
[weavejester.dependency :as dep]))
(defrecord Ref [key])
(defn ref
"Create a reference to a top-level key in a config map. The ref
resolves to the running instance, once its target has started."
[key]
(when-not (qualified-keyword? key)
(throw (ex-info (str "Invalid reference: " key ". Must be a qualified keyword.")
{:reason ::invalid-ref
:ref key})))
(->Ref key))
(defn ref?
"Return true if its argument is a ref."
[x]
(instance? Ref x))
(defn- derived-from?
[key candidate]
(or (= key candidate) (isa? key candidate)))
(defn find-derived
"Return a seq of all entries in a map, m, where the key is the
candidate key or derived from it. If there are no matching keys, nil
is returned."
[m k]
(seq (filter #(or (= (key %) k) (derived-from? (key %) k)) m)))
(defn- find-derived-1
[m k]
(let [kvs (find-derived m k)]
(when (next kvs)
(throw (ex-info (str "Ambiguous key: " k ". Found multiple candidates: "
(str/join ", " (map key kvs)))
{:reason ::ambiguous-key
:key k
:matching (mapv key kvs)})))
(first kvs)))
(defn- depth-search
[pred? coll]
(filter pred? (tree-seq coll? seq coll)))
(defn- find-derived-refs
[config v]
(->> (depth-search ref? v)
(map :key)
(mapcat #(map key (find-derived config %)))))
(defn- dependency-graph
[config]
(reduce-kv (fn [g k v]
(reduce #(dep/depend %1 k %2) g (find-derived-refs config v)))
(dep/graph)
config))
(defn- key-comparator
[graph]
(dep/topo-comparator #(compare (str %1) (str %2)) graph))
(defn- config-keyset
[config ks]
(set (mapcat #(map key (find-derived config %)) ks)))
(defn- validate-init
"Check the config and the requested keys, and return the relevant
keys in init order: requested, plus their transitive dependencies.
A circular dependency surfaces the `weavejester.dependency` ex-info."
[config ks]
(doseq [k (keys config)]
(when-not (qualified-keyword? k)
(throw (ex-info (str "Invalid config key: " k ". Keys must be qualified keywords.")
{:reason ::invalid-config
:key k}))))
(doseq [k ks]
(when-not (seq (find-derived config k))
(throw (ex-info (str "Unknown key: " k ". It matches nothing in the config.")
{:reason ::unknown-key
:key k}))))
(let [graph (dependency-graph config)
keyset (config-keyset config ks)
relevant (select-keys config (set/union keyset
(dep/transitive-dependencies-set graph keyset)))
ref-keys (distinct (map :key (depth-search ref? (vals relevant))))
missing (filterv #(nil? (find-derived relevant %)) ref-keys)]
(when (seq missing)
(throw (ex-info (str "Missing definitions for refs: " (str/join ", " missing))
{:reason ::missing-refs
:config config
:missing-refs missing})))
(doseq [k ref-keys]
(find-derived-1 relevant k))
(let [comparator (key-comparator graph)]
(vec (sort comparator (keys relevant))))))
(defn- resolve-refs
[sys v]
(walk/postwalk #(if (ref? %) (val (find-derived-1 sys (:key %))) %) v))
(defmulti init-key
"Turn a config value associated with a key into a running component.
May return a plain value or a promise; the system awaits it either way."
{:arglists '([key config])}
(fn [key _config] key))
(defmethod init-key :default
[k _config]
(throw (ex-info (str "No init-key method for " k)
{:reason ::missing-init-key
:key k})))
(defmulti halt-key
"Stop the running component associated with a key. Receives the
running instance returned by `init-key`, so anything halt needs must
travel inside it. May return a plain value or a promise; the system
awaits it either way. Must be idempotent."
{:arglists '([key instance])}
(fn [key _instance] key))
(defmethod halt-key :default
[_instance-key _instance]
nil)
(defn ^:private ^:async build-key
[sys k v]
(try
(assoc sys k (await (init-key k (resolve-refs sys v))))
(catch :default cause
(throw (ex-info (str "Error on key " k " when building system")
{:reason ::init-failed
:key k
:value v
:system sys}
cause)))))
(defn- halt-order
[sys ks]
(let [config (or (::origin (meta sys)) {})
graph (dependency-graph config)
keyset (config-keyset config ks)
comparator (key-comparator graph)]
(->> (set/union keyset (dep/transitive-dependents-set graph keyset))
(filter #(contains? sys %))
(sort comparator)
(reverse)
(vec))))
(defn ^:private ^:async halt-keys
"Halt every key in ks, collecting failures instead of stopping at the
first one. Returns a vector of `{:key :cause}` maps."
[sys ks]
(loop [failures []
ks ks]
(if (seq ks)
(let [k (first ks)
failures (try
(await (halt-key k (sys k)))
failures
(catch :default cause
(conj failures {:key k :cause cause})))]
(recur failures (rest ks)))
failures)))
(defn ^:private ^:async halt-silently
[sys]
(when (map? sys)
(await (halt-keys sys (halt-order sys (keys sys)))))
nil)
(defn ^:async init
"Turn a config map into a running system map. Keys start in
dependency order, each one awaited, with refs resolved to the running
instances. With two args, only those keys start, plus whatever they
depend on.
Returns a promise of the system map. If a component fails to start,
what already started is halted in reverse order, and the promise
rejects with the original failure carrying `:reason ::init-failed`,
the offending `:key`, and the partial `:system`."
([config]
(init config (keys config)))
([config ks]
(assert (map? config) "init needs a config map")
(let [order (validate-init config ks)]
(try
(loop [sys (with-meta {} {::origin config})
ks order]
(if (seq ks)
(recur (await (build-key sys (first ks) (config (first ks))))
(rest ks))
sys))
(catch :default cause
(await (halt-silently (:system (ex-data cause))))
(throw cause))))))
(defn ^:async halt
"Halt a running system map in reverse dependency order, awaiting each
component. With two args, halt those keys plus whatever depends on them.
Returns a promise of nil. Every component is given its chance to stop:
failures are collected, and once everything ran, the promise rejects
with `:reason ::halt-failed` and the `:failures` vector."
([sys]
(halt sys (keys sys)))
([sys ks]
(when-not (and (map? sys) (::origin (meta sys)))
(throw (ex-info "halt needs a system map built by init."
{:reason ::not-a-system})))
(let [failures (await (halt-keys sys (halt-order sys ks)))]
(when (seq failures)
(throw (ex-info (str "Failed to halt " (count failures) " component(s): "
(str/join ", " (map :key failures)))
{:reason ::halt-failed
:failures failures}))))))

View File

@ -4,13 +4,14 @@
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.wasm
(ns exporter.wasm
"Headless driver for the render-wasm module under Node: the GPU-free
counterpart of `app.render-wasm.api`. Loads the emscripten artifact, boots it
via `init_headless`, and exposes font provisioning + shape rendering.
Serialization is reused from the portable render-wasm leaves, so this
namespace owns only the Node runtime and the headless render calls.
There are no promesa chains in this namespace.
Requires render-wasm built with `-sENVIRONMENT=web,node`."
(:require
@ -23,10 +24,9 @@
[app.common.render-wasm.serializers :as sr]
[app.common.render-wasm.wasm :as wasm]
[app.common.uuid :as uuid]
;; Required for side effects: binds the generated enums.
[app.wasm.enums]
[cuerdas.core :as str]
[promesa.core :as p]
;; Required for side effects: binds the generated enums.
[exporter.wasm.enums]
[shadow.esm :refer [dynamic-import]]))
(def ^:private default-viewport-width 1920)
@ -44,10 +44,6 @@
path in devenv and inside the bundle, so it is a constant."
"resources/wasm")
(def image-cache-size
"Byte budget the image store is trimmed to between requests."
(* 256 1024 1024))
(defn- read-result-bytes
"Reads `len` bytes from the WASM heap starting at `offset`, copying them out
(via `.slice`) before the buffer is freed."
@ -56,39 +52,35 @@
;; --- MODULE LIFECYCLE
(defn init!
(defn ^:async init
"Loads the render-wasm artifact under Node and boots it headless. Sets the
shared `wasm/internal-module` so the portable serialization leaves work.
Idempotent-ish: callers should hold the returned module."
([] (init! default-viewport-width default-viewport-height))
([] (init default-viewport-width default-viewport-height))
([width height]
(let [dir artifact-dir
js-path (path/resolve dir "render-wasm.js")
wasm-path (path/resolve dir "render-wasm.wasm")
(let [dir artifact-dir
js-path (path/resolve dir "render-wasm.js")
wasm-path (path/resolve dir "render-wasm.wasm")
wasm-bytes (fs/readFileSync wasm-path)]
(l/info :hint "loading render-wasm (headless)" :js js-path)
;; shadow-cljs :esm — use its dynamic-import helper (raw `js/import`
;; compiles to an undefined `import$`).
(->> (dynamic-import (str "file://" js-path))
(p/mcat
(fn [mod]
(let [factory (unchecked-get mod "default")]
(factory
#js {;; Bypass the web fetch loader: instantiate from local bytes.
:instantiateWasm
(fn [imports success]
(-> (js/WebAssembly.instantiate wasm-bytes imports)
(.then (fn [result] (success (.-instance result)))))
#js {})
:locateFile (fn [p] (path/resolve dir p))
:printErr (fn [s] (l/warn :wasm s))}))))
(p/fmap
(fn [module]
(set! wasm/internal-module module)
(h/call module "_init_headless" width height)
(set! wasm/context-initialized? true)
(l/info :hint "render-wasm headless module ready" :width width :height height)
module))))))
(let [mod (await (dynamic-import (str "file://" js-path)))
factory (unchecked-get mod "default")
module (await (factory
#js {;; Bypass the web fetch loader: instantiate from local bytes.
:instantiateWasm
(fn [imports success]
(-> (js/WebAssembly.instantiate wasm-bytes imports)
(.then (fn [result] (success (.-instance result)))))
#js {})
:locateFile (fn [p] (path/resolve dir p))
:printErr (fn [s] (l/warn :wasm s))}))]
(set! wasm/internal-module module)
(h/call module "_init_headless" width height)
(set! wasm/context-initialized? true)
(l/info :hint "render-wasm headless module ready" :width width :height height)
module))))
;; --- FONT PROVISIONING (on demand, mirrors the browser)
@ -130,7 +122,7 @@
(d/distinct-xf font-key))
shape-ids))
(defn store-font!
(defn store-font
"Uploads one font's TTF bytes into the WASM font store, keyed by the family
(uuid quartet + weight + style). `font-bytes` is a Uint8Array/Buffer.
@ -147,12 +139,12 @@
(aget id 0) (aget id 1) (aget id 2) (aget id 3)
weight style (boolean emoji?) (boolean fallback?))))
(defn store-font-url!
(defn store-font-url
"Registers the public URL a font family was loaded from. The SVG export emits
one `@font-face` per family from these, and skips families without one, so
this must run for every family `store-font!` uploads.
this must run for every family `store-font` uploads.
Does NOT call `mem/free`, for the same reason as `store-font!`."
Does NOT call `mem/free`, for the same reason as `store-font`."
[{:keys [id weight style]} url]
(let [bytes (js/Buffer.from url "utf-8")
ptr (mem/alloc (.-byteLength bytes))]
@ -161,16 +153,16 @@
(aget id 0) (aget id 1) (aget id 2) (aget id 3)
weight style)))
(defn clear-fonts!
(defn clear-fonts
"Resets the WASM font store. Must be called once per render request because
the shared module would otherwise accumulate fonts across requests."
[]
(h/call wasm/internal-module "_clear_fonts"))
(defn update-text-layout!
(defn update-text-layout
"Recomputes a text shape's layout with the currently provisioned fonts. Text is
laid out at serialize time using the fallback font (real fonts aren't uploaded
yet), so this must run again after `provision-fonts!` or glyph metrics/line
yet), so this must run again after `provision-fonts` or glyph metrics/line
breaks are wrong.
Forced, because provisioning a font changes nothing `update_layout` keys on:
@ -191,13 +183,13 @@
(aget buf 0) (aget buf 1) (aget buf 2) (aget buf 3)
false)))))
(defn store-image-url!
(defn store-image-url
"Registers the public URL an image was loaded from. The SVG export emits
linked `<image href>` from these, and falls back to Skia base64 when missing,
so this should run for every media id the scene references (including
already-cached images).
Does NOT call `mem/free`, for the same reason as `store-font-url!`."
Does NOT call `mem/free`, for the same reason as `store-font-url`."
[image-id url]
(when (and (some? url) (not (str/blank? url)))
(let [bytes (js/Buffer.from url "utf-8")
@ -207,7 +199,7 @@
(h/call wasm/internal-module "_store_image_url"
(aget quart 0) (aget quart 1) (aget quart 2) (aget quart 3)))))
(defn store-image!
(defn store-image
"Uploads one image's *encoded* bytes (PNG/JPEG — Skia decodes, no WebGL) into
the WASM image store via `_store_image`. Buffer layout matches the Rust reader:
[shape uuid 16][image uuid 16][is_thumbnail u32][encoded bytes]. Images are
@ -239,28 +231,30 @@
(.set heap img-u8 (+ ptr 36))
(h/call module "_store_image")))
(defn evict-images!
(defn evict-images
"Evicts least-recently-used images until the store retains at most `max-bytes`
bytes. Returns the number evicted."
[max-bytes]
(h/call wasm/internal-module "_evict_images_to_budget" max-bytes))
(defn provision-fonts!
(defn ^:async provision-fonts
"Resolves and uploads every font needed by `shape-ids`, each family fetched
once. `resolve-font` is an injected fn of the family map -> promise of TTF
bytes (or nil to skip); optional `font-url` is a fn of the family map -> the
public URL those bytes came from. This keeps the font *source* (gfonts proxy
/ custom assets / backend) out of the driver."
[shape-ids resolve-font & {:keys [font-url]}]
(->> (fonts-for-shapes shape-ids)
(map (fn [family]
(->> (resolve-font family)
(p/fmap (fn [bytes]
(when bytes
(store-font! family bytes)
(when-let [url (when font-url (font-url family))]
(store-font-url! family url))))))))
(p/all)))
(let [families (fonts-for-shapes shape-ids)]
(await (js/Promise.all
(mapv (^:async fn [family]
(let [bytes (await (resolve-font family))]
(when bytes
(store-font family bytes)
(when-let [url (when font-url (font-url family))]
(store-font-url family url)))
nil))
families)))
nil))
;; --- RENDER

View File

@ -4,7 +4,7 @@
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.wasm.enums
(ns exporter.wasm.enums
"Binds this build's generated enums into the shared bridge.
`shared.js` is emitted next to this file by `render-wasm/build export` and is

View File

@ -0,0 +1,190 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns exporter.wasm.pool
"Pool of headless render workers.
Mirrors `exporter.browser`: a `generic-pool` whose objects are
`worker_threads` instead of browsers, so acquisition and eviction
behave the same way for both render backends. A worker is expensive
to build (it boots its own render-wasm module), hence the pooling.
Acquisition is not capped: the admission scheduler is the
backpressure, and the idle watchdog guarantees a wedged worker gives
its slot back.
Workers run the same bundle as the main thread. There is always at
least one worker, since a headless render has nowhere else to go.
Unlike the browser service, the running instance is a map: the pool
plus the silence timeout the renders await on it. There are no
promesa chains in this namespace: the pool speaks native promises,
and `await` covers the rest."
(:require
["generic-pool" :as gp]
["node:path" :as path]
["node:process" :as proc]
["node:worker_threads" :as wt]
[app.common.data :as d]
[app.common.exceptions :as ex]
[app.common.logging :as l]
[exporter.utils.system :as system]
[exporter.wasm :as-alias wasm]))
(l/set-level! :info)
(def defaults
"Static pool defaults. The sizing (`:max`) mirrors the worker
concurrency default; the wiring layer passes the env-derived values."
{:max 2
:min 1
:timeout-ms 300000
:ready-timeout-ms 60000
:eviction-interval 30000
:eviction-batch 2
:idle-timeout 300000})
(defn await-ready
"Resolves the worker once it posts its ready handshake. A worker
that stays silent too long is terminated and rejected with
`:worker-not-ready`; a boot error rejects with its own cause."
[worker timeout-ms]
(js/Promise.
(fn [resolve reject]
(let [timer (js/setTimeout
(fn []
(l/error :hint "render worker did not become ready")
(.terminate ^js worker)
(reject (ex/error :type :internal
:code :worker-not-ready
:hint "render worker did not become ready")))
timeout-ms)]
(.on ^js worker "error"
(fn [cause]
(l/error :hint "render worker error" :cause cause)
(unchecked-set worker "__alive" false)
;; Rejecting after `resolve` is a no-op, so this is safe
;; for the errors that arrive once the worker is pooled.
(js/clearTimeout timer)
(reject cause)))
(.on ^js worker "exit"
(fn [code]
(l/info :hint "render worker exited" :code code)
(unchecked-set worker "__alive" false)))
;; Not `.once`: a stray message before the handshake would
;; consume the listener and hang the worker until the timeout.
(letfn [(on-ready [data]
(when (= "ready" (unchecked-get data "type"))
(js/clearTimeout timer)
(.off ^js worker "message" on-ready)
(resolve worker)))]
(.on ^js worker "message" on-ready))))))
(defn- default-script
[]
(path/resolve (aget (.-argv proc/default) 1)))
(defn- default-create-worker
[script ready-timeout-ms]
(fn []
(await-ready (new wt/Worker script) ready-timeout-ms)))
(defn- ^:async create-instance
[create-worker id-counter]
(let [worker (await (create-worker))
id (swap! id-counter inc)]
(l/info :origin "factory" :action "create" :worker-id id)
(unchecked-set worker "__id" id)
(unchecked-set worker "__alive" true)
worker))
(defn- destroy-instance
[worker]
(l/info :origin "factory" :action "destroy"
:worker-id (unchecked-get worker "__id"))
(unchecked-set worker "__alive" false)
(.terminate ^js worker))
(defn- validate-instance
[worker]
(true? (unchecked-get worker "__alive")))
(defn- make-factory
[create-worker]
(let [id-counter (atom 0)]
#js {:create (fn [] (create-instance create-worker id-counter))
:destroy destroy-instance
:validate validate-instance}))
(defmethod system/init-key ::wasm/pool
[_ cfg]
(let [opts (merge defaults (d/without-nils cfg))
max-workers (:max opts)
script (or (:script opts) (default-script))
create-worker (or (:create-worker opts)
(default-create-worker script (:ready-timeout-ms opts)))]
(l/info :hint "initializing render worker pool" :max max-workers)
{:pool (gp/createPool (make-factory create-worker)
#js {:max max-workers
:min (min max-workers (:min opts))
:testOnBorrow true
:evictionRunIntervalMillis (:eviction-interval opts)
:numTestsPerEvictionRun (:eviction-batch opts)
:idleTimeoutMillis (:idle-timeout opts)})
:timeout-ms (:timeout-ms opts)}))
(defn- ^:async drain-pool
[pool]
(await (.drain ^js pool))
(await (.clear ^js pool))
nil)
(defmethod system/halt-key ::wasm/pool
[_ instance]
(when-let [pool (:pool instance)]
(l/info :hint "finalizing render worker pool")
(drain-pool pool)))
(defn- pool?
"A raw generic-pool: anything lendable through acquire and release."
[x]
(and (some? x)
(fn? (unchecked-get x "acquire"))
(fn? (unchecked-get x "release"))))
(defn ^:async run
"Lends one worker: resolves the pool from `cfg` (the `::wasm/pool`
service instance, or a raw pool), acquires a worker, applies `f` to
the cfg carrying it under `::worker` plus any extra `args`, and
returns the worker — or destroys it when `f` fails, the way a failed
transaction rolls back instead of returning its connection: a failed
worker may be aborted or mid-write, and a terminated worker cannot
come back."
[cfg f & args]
(let [cfg (if (pool? cfg) {::wasm/pool {:pool cfg}} cfg)
pool (:pool (::wasm/pool cfg))]
(when-not (pool? pool)
(throw (ex/error :type :assertion
:code :invalid-pool-cfg
:hint "no pool in cfg")))
(let [worker (await (.acquire ^js pool))]
(try
(let [result (await (apply f (assoc cfg ::worker worker) args))]
(await (.release ^js pool worker))
result)
(catch :default cause
(try
(await (.destroy ^js pool worker))
(catch :default cause'
(l/warn :hint "render worker destroy failed"
:cause cause')))
(throw cause))))))
(defn terminate
[worker]
(when worker
(unchecked-set worker "__alive" false)
(.terminate worker)))

View File

@ -0,0 +1,523 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns exporter.wasm.render
"Headless render pipeline: renders exports with the render-wasm Skia pipeline,
with no browser and no WebGL.
Per request: fetch scene (get-page RPC) -> serialize -> provision fonts and
images -> relayout text with the real fonts -> render each object.
This runs inside a render worker (one WASM design state per worker), so the
synchronous Skia calls never block the main thread. Everything the pipeline
needs beyond the export itself travels in `env`
(`:internal-uri`, `:public-uri`, `:tmpdir`, `:image-cache-size`): the
worker thread reads no config and no globals.
There are no promesa chains in this namespace.
Handles png/jpeg/webp (Skia encodes all three), pdf and svg."
(:require
["node:fs" :as fs]
["undici" :as http]
[app.common.data :as d]
[app.common.exceptions :as ex]
[app.common.fonts :as cfnt]
;; Required for side effects: these register the transit read handlers and
;; deftype impls the `get-page` response is decoded into.
[app.common.geom.matrix]
[app.common.geom.point]
[app.common.geom.rect]
[app.common.logging :as l]
[app.common.transit :as t]
[app.common.types.fills.impl]
[app.common.types.objects-map]
[app.common.types.path.impl]
[app.common.types.shape]
[app.common.types.shape.images :as images]
[app.common.uri :as u]
[app.common.uuid :as uuid]
[cuerdas.core :as str]
[exporter.shell :as shell]
[exporter.util.mime :as mime]
[exporter.wasm :as wasm]
[exporter.wasm.serialize :as serialize]))
(def fetch
"The network entry point of the pipeline: a plain def, redefined by
the tests, which never need a network."
http/fetch)
;; --- module lifecycle (one shared, lazily-initialized instance)
(defonce ^:private module* (atom nil))
(defn- ^:async ensure-module
[]
(or @module*
(reset! module* (await (wasm/init)))))
;; --- backend endpoints
;;
;; Every fetch targets the internal endpoint (falling back to public-uri),
;; in a deployment the exporter reaches the backend over the container network
(defn internal-uri
"Absolute URI for `path` on the internal (backend) endpoint."
[env path]
(-> (:internal-uri env)
(u/ensure-path-slash)
(u/join path)
(str)))
(defn public-uri
"Absolute URI for `path` on the public endpoint. Whoever opens an exported SVG
resolves its `@font-face` sources, so those cannot use the internal endpoint."
[env path]
(-> (:public-uri env)
(u/ensure-path-slash)
(u/join path)
(str)))
(defn- share-uri
"An asset URI plus the share the export came through, when it has one:
the asset endpoints decide permissions by that parameter, so a share
viewer without it gets a 404 and the image or font silently vanishes.
The URI helpers do the merging and encoding, so the value is escaped
and an existing query survives."
[uri share-id]
(cond-> uri
(some? share-id) (u/append-query-param "share-id" (str share-id))))
(defn error-detail
"Node's fetch reports every transport failure as a bare `TypeError: fetch
failed`; the actual reason (TLS rejection, DNS, ECONNREFUSED) is buried in a
nested `cause` chain that the logger does not print. Flattens the chain into
one readable string."
[cause]
(->> (iterate (fn [^js e] (unchecked-get e "cause")) cause)
(take-while some?)
(take 5)
(map (fn [^js e]
(let [code (unchecked-get e "code")
msg (or (unchecked-get e "message") (str e))]
(if code (str code ": " msg) msg))))
(str/join " <- ")))
(defn- ^:async request
"`undici/fetch` that fails with an ex-info carrying the target uri and the
unwrapped cause chain, so a failed request says what actually went wrong and
against which endpoint."
[uri opts]
(try
(await (fetch uri opts))
(catch :default cause
(throw (ex-info "http fetch failed"
{:uri uri :detail (error-detail cause)}
cause)))))
(defn- explain
"Log-friendly reason for `cause`: the detail `request` already attached, or a
freshly unwrapped chain for anything else (WASM aborts, decode errors)."
[cause]
(or (:detail (ex-data cause))
(error-detail cause)))
(defn- rpc-headers
"Headers for backend RPC calls: transit in, Bearer out. No shared key
here: those endpoints authenticate the session, not the worker."
[_env token]
#js {"Content-Type" "application/transit+json"
"Authorization" (str "Bearer " token)})
(defn- asset-headers
"Auth headers for `/assets/*`. Cookie, not Bearer: those endpoints redirect to
a presigned S3/minio URL, and a Bearer header makes S3 400 (\"multiple
authentication types\")."
[_env token]
#js {"Cookie" (str "auth-token=" token)})
;; --- shape bundle fetch (backend RPC)
(defn ^:async fetch-objects
"Fetches the exported roots and their children from the backend via the
`get-page` RPC (`:object-id`, as the browser render path does), using the
same auth the exporter uses elsewhere (management key + bearer)."
[env {:keys [file-id page-id share-id token objects]}]
(let [headers (rpc-headers env token)
root-ids (into #{} (map :id) objects)
body (t/encode-str (cond-> {:file-id file-id
:page-id page-id}
(seq root-ids) (assoc :object-id root-ids)
share-id (assoc :share-id share-id)))
uri (internal-uri env "api/rpc/command/get-page")]
(l/dbg :hint "wasm render: get-page"
:uri uri
:file-id (str file-id)
:page-id (str page-id)
:roots (count root-ids))
(let [resp (await (request uri #js {:method "POST" :headers headers :body body}))]
(if (= 200 (.-status resp))
(:objects (t/decode-str (await (.text resp))))
(let [resp-body (await (.text resp))]
(l/error :hint "wasm render: get-page failed"
:uri uri
:status (.-status resp)
:body resp-body)
(throw (ex-info "get-page failed"
{:status (.-status resp)
:body resp-body})))))))
;; --- font resolution
;;
;; The text serializer keeps each font's real uuid, so `wasm/fonts-for-shape`
;; reports it. Custom (team) fonts resolve through the file's font variants,
;; google fonts through the shared `app.common.fonts` catalog; builtin
;; fonts through its bundled family + the frontend's static `/fonts/`.
(defn ^:async fetch-font-variants
"Team (custom) font variants for the file, or nil — a failure here degrades
to fallback fonts, it does not fail the export."
[env {:keys [file-id share-id token]}]
(let [headers (rpc-headers env token)
body (t/encode-str (cond-> {:file-id file-id}
share-id (assoc :share-id share-id)))
uri (internal-uri env "api/rpc/command/get-font-variants")]
(try
(let [resp (await (request uri #js {:method "POST" :headers headers :body body}))]
(when (= 200 (.-status resp))
(t/decode-str (await (.text resp)))))
(catch :default cause
(l/warn :hint "wasm render: get-font-variants failed"
:uri uri :detail (explain cause) :cause cause)
nil))))
(defn- ^:async fetch-ttf-bytes
"Downloads a TTF (an ArrayBuffer, or nil). A failure here degrades to
fallback fonts, it does not fail the export."
[uri opts]
(try
(let [resp (await (request uri opts))]
(when (= 200 (.-status resp))
(await (.arrayBuffer resp))))
(catch :default cause
(l/warn :hint "wasm render: font fetch failed"
:uri uri :detail (explain cause) :cause cause)
nil)))
;; TTF bytes cached for the process lifetime, keyed by whatever identifies the
;; variant (a gfont id+weight+style, a builtin file name).
(defonce ^:private font-bytes* (atom {}))
(defn- ^:async cached-ttf-bytes
[cache-key fetch-fn]
(if-let [bytes (get @font-bytes* cache-key)]
bytes
(let [buf (await (fetch-fn))]
(when buf (swap! font-bytes* assoc cache-key buf))
buf)))
(defn- ^:async fetch-asset-bytes
[env asset-id token share-id]
(fetch-ttf-bytes (share-uri (internal-uri env (str "assets/by-id/" asset-id)) share-id)
#js {:method "GET" :headers (asset-headers env token)}))
(defn- ^:async fetch-gfont-bytes
[env ttf-url]
(fetch-ttf-bytes (cfnt/gstatic->proxy-url ttf-url (internal-uri env "internal/gfonts/font"))
#js {:method "GET"}))
(defn- ^:async fetch-builtin-font-bytes
[env ttf-file]
(cached-ttf-bytes ttf-file #(fetch-ttf-bytes (internal-uri env (str "fonts/" ttf-file))
#js {:method "GET"})))
(defn family-uuid
[id]
(uuid/from-unsigned-parts (aget id 0) (aget id 1) (aget id 2) (aget id 3)))
(defn find-variant
"Custom variant for a family: uuid+weight+style first, degrading to
uuid+weight then uuid."
[variants font-uuid weight style]
(let [style-str (if (zero? style) "normal" "italic")]
(or (d/seek (fn [v] (and (= (:font-id v) font-uuid)
(= (:font-weight v) weight)
(= (name (:font-style v)) style-str)))
variants)
(d/seek (fn [v] (and (= (:font-id v) font-uuid)
(= (:font-weight v) weight)))
variants)
(d/seek (fn [v] (= (:font-id v) font-uuid)) variants))))
(defn make-resolve-font
"Builds a `resolve-font` fn (family map -> TTF bytes). Custom
variants first; the bundled fonts for `uuid/zero`, which is what
`font-id->uuid` maps every builtin family to; google catalog otherwise."
[env variants params]
(^:async fn [{:keys [id weight style]}]
(let [font-uuid (family-uuid id)
variant (find-variant variants font-uuid weight style)]
(cond
(:ttf-file-id variant)
(await (fetch-asset-bytes env (:ttf-file-id variant)
(:token params) (:share-id params)))
(= uuid/zero font-uuid)
(await (fetch-builtin-font-bytes env (cfnt/resolve-ttf-file weight style)))
:else
(if-let [gurl (cfnt/resolve-ttf-url font-uuid weight style)]
(await (fetch-gfont-bytes env gurl))
nil)))))
(defn- make-font-url
"Builds a `font-url` fn (family map -> public URL of its TTF), the same
sources `make-resolve-font` downloads from but addressed publicly. The SVG
export emits one `@font-face` per family from these."
[env variants share-id]
(fn [{:keys [id weight style]}]
(let [font-uuid (family-uuid id)
variant (find-variant variants font-uuid weight style)]
(cond
(:ttf-file-id variant)
(share-uri (public-uri env (str "assets/by-id/" (:ttf-file-id variant)))
share-id)
(= uuid/zero font-uuid)
(public-uri env (str "fonts/" (cfnt/resolve-ttf-file weight style)))
:else
(some-> (cfnt/resolve-ttf-url font-uuid weight style)
(cfnt/gstatic->proxy-url (public-uri env "internal/gfonts/font")))))))
;; --- fallback fonts (emoji + per-script noto fonts)
;;
;; Emoji and non-latin scripts render through fallback families, not through
;; any span's font family, so `wasm/fonts-for-shape` never reports them and the
;; provisioning above never uploads them. Must run per request, since
;; `clear-fonts` empties the store; the TTF bytes stay cached per process.
(defn scene-fallback-fonts
"Fallback font descriptors needed by the scene's text. Deduped because
several languages map to one noto family and provisioning is concurrent —
otherwise they all miss the byte cache at once and refetch the same TTF."
[scene]
(let [texts (for [shape (vals scene)
:when (= :text (:type shape))
node (or (some->> (:content shape) (tree-seq :children :children)) [])
:let [text (:text node)]
:when (string? text)]
text)
emoji? (boolean (some cfnt/contains-emoji? texts))
langs (reduce cfnt/collect-used-languages #{} texts)]
(distinct
(cond-> (cfnt/add-noto-fonts [] langs)
emoji? (cfnt/add-emoji-font)))))
(defn- ^:async fetch-fallback-font-bytes
"Downloads one fallback font's TTF. Cached by the whole variant, not just
`font-id`: `resolve-ttf-url` picks a different TTF per weight/style, so a
font-id-only key would serve the first downloaded variant for every other one."
[env {:keys [font-id weight style]}]
(if-let [ttf-url (some-> (cfnt/gfont-id->uuid font-id) (cfnt/resolve-ttf-url weight style))]
(cached-ttf-bytes [font-id weight style] #(fetch-gfont-bytes env ttf-url))
nil))
(defn- ^:async provision-fallback-fonts
[env scene]
(await (js/Promise.all
(mapv (^:async fn [{:keys [font-id weight style is-emoji is-fallback] :as font}]
(if-let [font-uuid (cfnt/gfont-id->uuid font-id)]
(let [buf (await (fetch-fallback-font-bytes env font))]
(if buf
(wasm/store-font {:id (uuid/get-u32 font-uuid)
:weight weight
:style style
:emoji? (boolean is-emoji)
:fallback? (boolean is-fallback)}
buf)
(l/warn :hint "wasm render: fallback font unavailable"
:font-id font-id)))
nil))
(scene-fallback-fonts scene))))
nil)
;; --- image resolution
;;
;; Image fills reference file-media ids; the encoded bytes go straight to
;; `_store_image` (Skia decodes, no WebGL), keyed by media uuid so this happens
;; once per request rather than per rendered object.
(defn ^:async fetch-file-media-bytes
"Downloads an image fill's encoded bytes by file-media id, or nil when
they cannot be had: the object renders without its image either way.
The share the export came through rides in the URI: a share viewer is
allowed through it, and without it the fetch is a 404."
[env media-id token share-id]
(let [headers (asset-headers env token)
uri (share-uri (internal-uri env (str "assets/by-file-media-id/" media-id))
share-id)]
(try
(let [resp (await (request uri #js {:method "GET" :headers headers}))]
(if (= 200 (.-status resp))
(await (.arrayBuffer resp))
(do
(l/warn :hint "wasm render: image fetch non-200"
:media-id (str media-id)
:uri uri
:status (.-status resp))
nil)))
(catch :default cause
(l/warn :hint "wasm render: image fetch failed"
:media-id (str media-id) :uri uri
:detail (explain cause) :cause cause)
nil))))
(defn- ^:async provision-images
"Fetches and stores every image the scene references (shape, stroke and
text-span fills, enumerated by `app.common.types.shape.images`). Unlike fonts,
the image store is not reset per request, so already-held images are skipped
and repeated exports of a file reuse them.
Always registers a public media URL for each id so SVG export can emit linked
`<image href>` even when the encoded bytes were already cached."
[env scene params]
(let [all-ids (images/scene-image-ids scene)
new-ids (remove wasm/image-cached? all-ids)]
(l/dbg :hint "wasm render: provisioning images"
:total (count all-ids)
:cached (- (count all-ids) (count new-ids)))
(doseq [image-id all-ids]
(wasm/store-image-url image-id
(share-uri (public-uri env (str "assets/by-file-media-id/" image-id))
(:share-id params))))
(await (js/Promise.all
(mapv (^:async fn [image-id]
(let [buf (await (fetch-file-media-bytes env image-id
(:token params)
(:share-id params)))]
(if buf
(do
(l/dbg :hint "wasm render: image stored"
:media-id (str image-id)
:bytes (.-byteLength ^js buf))
(wasm/store-image image-id buf))
(l/warn :hint "wasm render: image unavailable"
:media-id (str image-id))))
nil)
new-ids)))
nil))
(defn- relayout-text
"Recomputes layout for every text shape, once the real fonts are provisioned
(serialize-time layout used the fallback)."
[scene]
(doseq [shape (vals scene)
:when (= :text (:type shape))]
(wasm/update-text-layout (:id shape))))
;; --- render
(defn- check-cancelled
"Cancellation is cooperative: a render already inside Skia cannot be
interrupted, so the flag is only observed between objects. Killing a job
mid-object is the caller's job (terminating the worker)."
[{:keys [cancelled?] :as _params}]
(when (and cancelled? (cancelled?))
(ex/raise :type :internal
:code :job-cancelled
:hint "export job was cancelled")))
(defn- render-object-bytes
[type id scale]
(case type
:pdf (let [bytes (wasm/render-shape-pdf id)]
(l/dbg :hint "PDF generated via Skia (render-wasm headless)"
:object-id (str id)
:backend "skia-wasm"
:bytes (.-length bytes))
bytes)
:svg (let [bytes (wasm/render-shape-svg id scale)]
(l/dbg :hint "SVG generated via Skia (render-wasm headless)"
:object-id (str id)
:backend "skia-wasm"
:bytes (.-length bytes))
bytes)
(wasm/render-shape-raster id scale type)))
(defn invalidate-module-on-error?
"Whether a failed render drops the shared module: every error does,
except the cooperative `:job-cancelled` raised between objects, which
leaves the instance intact for the next render to reuse."
[cause]
(not= :job-cancelled (:code (ex-data cause))))
(defn- ^:async render*
[env {:keys [scale type objects] :as params} on-object]
(l/dbg :hint "wasm render: start"
:type type
:scale scale
:objects (count objects)
:file-id (str (:file-id params))
:page-id (str (:page-id params)))
(try
(await (ensure-module))
(let [scene (await (fetch-objects env params))]
(l/dbg :hint "wasm render: scene fetched" :shapes (count scene))
(serialize/serialize-scene scene)
(l/dbg :hint "wasm render: scene serialized")
;; So fonts from a previous request don't leak into this one.
(wasm/clear-fonts)
(let [[variants] (await (js/Promise.all [(fetch-font-variants env params)
(provision-images env scene params)
(provision-fallback-fonts env scene)]))
variants (or variants [])
resolve-font (make-resolve-font env variants params)
font-url (make-font-url env variants (:share-id params))]
;; Before rendering, so the relayout below sees real
;; font metrics. Deduped across objects: shapes
;; sharing one family download its TTF once.
(await (wasm/provision-fonts (map :id objects) resolve-font
:font-url font-url)))
(relayout-text scene)
;; Sequential, like the synchronous Skia calls underneath:
;; the cancel check lands between objects.
(doseq [{:keys [id] :as object} objects]
(check-cancelled params)
(let [bytes (render-object-bytes type id scale)
path (shell/tempfile (:tmpdir env)
:prefix "penpot.tmp.wasm."
:suffix (mime/get-extension type))]
(l/dbg :hint "wasm render: object rendered"
:object-id (str id) :bytes (.-length bytes))
(fs/writeFileSync path bytes)
(await (on-object (assoc object :path path)))))
;; After the request, never mid-render, so an image can't
;; disappear under a running export.
(let [evicted (wasm/evict-images (:image-cache-size env))]
(when (pos? evicted)
(l/info :hint "wasm render: evicted cached images" :count evicted)))
nil)
(catch :default cause
(l/error :hint "wasm render: failed"
:detail (explain cause)
:internal-uri (str (:internal-uri env))
:cause cause)
;; A panic can leave the mem buffer allocated or the instance
;; aborted; drop it so the next request rebuilds a fresh one. A
;; cooperative cancel leaves the instance intact by design, so it
;; never invalidates the module the next render reuses.
(when (invalidate-module-on-error? cause)
(reset! module* nil))
(throw cause))))
(defn ^:async render
"Public entry. Renders every object of `params` with the render
environment `env` (`:internal-uri`, `:public-uri`, `:tmpdir`,
`:image-cache-size`), calling `on-object` with
`{:id :filename :path ...}` as each one is written out."
[env params on-object]
(await (render* env params on-object)))

View File

@ -0,0 +1,171 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns exporter.wasm.scope
"One leased render worker for a whole batch.
The pool owns lending; this level owns what a lease means: every
headless export runs on the same worker, one at a time, sharing its
caches and pool slot, with the cancel abort armed for the life of
the lease. Task keys arrive in `cfg` under `:exporter.renderer/*`
(set by whoever built the batch); the pool instance under the
pool's own key."
(:require
[app.common.exceptions :as ex]
[app.common.logging :as l]
[app.common.transit :as t]
[exporter.wasm.pool :as pool]))
(defn- serializer
"Chains thunks so a job's renders run one at a time on its worker. A
failure is isolated: it doesn't break the chain for the next one."
[]
(let [queue (atom (js/Promise.resolve nil))]
(fn [thunk]
(let [result (.then @queue (fn [_] (thunk)) (fn [_] (thunk)))]
(reset! queue (.then result (fn [_] nil) (fn [_] nil)))
result))))
(defn- render-on-worker-
"Settles when the worker reports the render finished, failed, or the
thread went away. That last case matters: a terminated worker (how a
cancel stops a render mid-Skia) emits `exit` and never `error`, and a
promise left pending there would keep its pool slot borrowed for the
life of the process."
[^js worker params cancel-buffer on-object timeout-ms]
(js/Promise.
(fn [resolve reject]
(let [timer (volatile! nil)]
(letfn [(disarm []
(when-let [t @timer]
(js/clearTimeout t)
(vreset! timer nil)))
(rearm []
(disarm)
(vreset! timer (js/setTimeout
(fn []
(l/error :hint "render worker went silent, terminating"
:worker-id (unchecked-get worker "__id"))
(cleanup)
;; Terminating is what frees the pool slot:
;; the `exit` it raises has no listener left.
(unchecked-set worker "__alive" false)
(.terminate ^js worker)
(reject (ex/error :type :internal
:code :render-timeout
:hint "render worker stopped responding")))
timeout-ms)))
(cleanup []
(disarm)
(.off worker "message" on-message)
(.off worker "error" on-error)
(.off worker "exit" on-exit))
(on-error [cause]
(cleanup)
(reject cause))
(on-exit [code]
(cleanup)
(reject (ex/error :type :internal
:code :worker-exited
:hint (str "render worker exited with code " code))))
(on-message [data]
(rearm)
(case (unchecked-get data "type")
;; A failure while the main thread handles the object
;; (moving the file, appending to the zip) has to end
;; the render too, or nothing ever settles this promise.
"object" (try
(on-object (t/decode-str (unchecked-get data "payload")))
(catch :default cause
(cleanup)
(reject cause)))
"done" (do (cleanup) (resolve nil))
"error" (do (cleanup)
(reject (ex/error :type :internal
:code (or (some-> (unchecked-get data "code") keyword)
:wasm-render-error)
:hint (unchecked-get data "message"))))
nil))]
(.on worker "message" on-message)
(.once worker "error" on-error)
(.once worker "exit" on-exit)
(rearm)
(.postMessage worker #js {:type "render"
:params (t/encode-str params)
:cancel cancel-buffer}))))))
(defn ^:async render-on
"Low-level render on the worker leased in `cfg`. A missing worker
or timeout raises instead of failing obscurely (`:worker-not-leased`,
`:timeout-ms-missing`): both are programmer errors, and a missing
timeout must never degrade silently into the default silence budget."
[cfg params]
(let [worker (::pool/worker cfg)
timeout-ms (or (:timeout-ms (:exporter.wasm/pool cfg))
(throw (ex/error :type :assertion
:code :timeout-ms-missing
:hint "render-on needs :timeout-ms in cfg")))]
(when-not worker
(throw (ex/error :type :assertion
:code :worker-not-leased
:hint "render-on needs a leased worker in cfg")))
(when-let [check-cancelled (:exporter.renderer/check-cancelled cfg)]
(check-cancelled))
(await (render-on-worker- worker
params
(::cancel-buffer cfg)
(:exporter.renderer/on-object cfg)
timeout-ms))))
(defn ^:async run
"Leases one worker for `on-scope`, a fn of a single-arg render fn.
Every headless export fires through it, one at a time on the same
worker, so the check-cancelled runs as each render's turn comes up. The check-cancelled
is mandatory: without one the lease raises `:check-cancelled-
missing` before acquiring anything. The per lease signal buffer is
born here; the interval dies with the lease. A truly stuck thread
is backstopped by the pool's own silence watchdog."
[cfg on-scope]
(let [check-cancelled (:exporter.renderer/check-cancelled cfg)]
(when-not (fn? check-cancelled)
(throw (ex/error :type :assertion
:code :check-cancelled-missing
:hint "scope/run needs a zero-arg check-cancelled in cfg")))
(await
(pool/run cfg
(^:async fn [cfg]
(let [worker (::pool/worker cfg)
chain (serializer)
live (volatile! worker)
signal (js/Int32Array. (js/SharedArrayBuffer. 4))
timer (volatile! nil)
render-cfg (assoc cfg
::cancel-buffer (.-buffer signal))]
(vreset! timer
(js/setInterval
(fn []
(try
(check-cancelled)
(catch :default _cause
;; One-shot: the lease is doomed anyway.
;; Between objects the worker sees the
;; flag; inside a render only terminating
;; the thread stops it.
(some-> @timer js/clearInterval)
(js/Atomics.store signal 0 1)
(some-> @live pool/terminate))))
(::watch-interval cfg 1000)))
(try
(await (on-scope (fn [params] (chain (fn [] (render-on render-cfg params))))))
(finally
(some-> @timer js/clearInterval)
(vreset! live nil)))))))))

View File

@ -4,7 +4,7 @@
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.wasm.serialize
(ns exporter.wasm.serialize
"Browser-free shape serialization for the headless exporter: the counterpart
of `app.render-wasm.api/set-object`, which cannot be reused directly because
its namespace pulls React/DOM/store. Only the call sequencing lives here —
@ -12,15 +12,15 @@
WASM are the editor's.
Covers everything except svg-raw. Image bytes and fonts are provisioned
separately by `app.renderer.wasm`."
separately by the render pipeline."
(:require
[app.common.render-wasm.api.props :as props]
[app.common.render-wasm.helpers :as h]
[app.common.render-wasm.serialize-shape :as serialize-shape]
[app.common.render-wasm.wasm :as wasm]
[app.wasm.text :as text]))
[exporter.wasm.text :as text]))
(defn set-shape!
(defn set-shape
"Serializes a single shape into the WASM design state. The host-independent
properties (base props, children, blur, shadows, svg-attrs, mask, bool-type,
path geometry, grow-type) go through the shared `serialize-shape` — the same
@ -34,13 +34,13 @@
(when-not (= type :group)
(props/write-shape-strokes! (:strokes prepared)))
(when (= type :text)
(text/set-shape-text! (:content prepared)))))
(text/set-shape-text (:content prepared)))))
(defn serialize-scene!
(defn serialize-scene
"Loads every shape of an `objects` map into the WASM design state. Resets the
shapes pool first so repeated exports don't accumulate into the shared
state. Order is irrelevant: shapes reference each other by id and the tree
is resolved at render time."
[objects]
(h/call wasm/internal-module "_init_shapes_pool" (count objects))
(run! set-shape! (vals objects)))
(run! set-shape (vals objects)))

View File

@ -4,7 +4,7 @@
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.wasm.text
(ns exporter.wasm.text
"Browser-free text-content serialization for the headless exporter. Only the
paragraph walk is local: the binary layout and the font-id -> uuid mapping
both come from `app.common.render-wasm.text-content`."
@ -14,7 +14,7 @@
[app.common.render-wasm.text-content :as tc]
[app.common.render-wasm.wasm :as wasm]))
(defn set-shape-text!
(defn set-shape-text
"Serializes a text shape's content into the current WASM shape. Mirrors the
editor's sequence: clear -> vertical-align -> append each paragraph -> layout.
Byte writing and font resolution are the shared

View File

@ -0,0 +1,85 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns exporter.wasm.worker
"Render worker entry point.
Owns one render-wasm module and renders one export request at a time. The
Skia calls are synchronous, so running them here is what lets several exports
progress at once: the main thread keeps polling the queue and settling jobs
while this thread is blocked inside a render. There are no promesa chains
in this namespace.
Messages in: {type: \"render\", params: <transit>, cancel: SharedArrayBuffer}
Messages out: {type: \"ready\"}
{type: \"object\", payload: <transit>} one per rendered object
{type: \"done\"} | {type: \"error\", message, code}"
(:require
["node:worker_threads" :as wt]
[app.common.exceptions :as ex]
[app.common.logging :as l]
[app.common.transit :as t]
[exporter.wasm.render :as render]))
(def parent-port
"The channel renders report through: the parent port of a render
thread. A plain def so tests — which run on the main thread, where
the port is null — rebind it, the way they rebind `fetch`."
wt/parentPort)
(defn- post
[message]
(.postMessage ^js parent-port message))
(defn- cancelled-fn
[buffer]
(if (some? buffer)
(let [signal (js/Int32Array. buffer)]
(fn [] (pos? (js/Atomics.load signal 0))))
(constantly false)))
(defn ^:async handle-render
"One render message: split the render env off the params, run the
pipeline, and post every object plus the settle. Never rejects: a
failure posts `error`, so the main thread always settles the lease."
[data]
(try
(let [{env ::render/env :as params} (t/decode-str (unchecked-get data "params"))]
(when (nil? env)
(throw (ex/error :type :assertion
:code :env-missing
:hint "worker render needs ::render/env in params")))
(let [params (assoc (dissoc params ::render/env)
:cancelled? (cancelled-fn (unchecked-get data "cancel")))]
(await (render/render env params
(fn [object]
(post #js {:type "object" :payload (t/encode-str object)}))))
(post #js {:type "done"})))
(catch :default cause
(l/warn :hint "render worker: request failed" :cause cause)
(post #js {:type "error"
:message (or (ex-message cause) (str cause))
:code (some-> cause ex-data :code name)}))))
(defn- on-message
[data]
(case (unchecked-get data "type")
"render" (handle-render data)
(l/warn :hint "render worker: unknown message" :type (unchecked-get data "type"))))
(defonce ^:private listening
;; `defonce` survives a hot reload, so a reload does not stack a second
;; listener on the port. The indirection through the var keeps the reloaded
;; `on-message` in play instead of pinning the one captured at boot.
(delay
(.on ^js wt/parentPort "message" (fn [data] (on-message data)))
true))
(defn main
[& _]
@listening
(post #js {:type "ready"})
(l/info :hint "render worker ready"))

View File

@ -0,0 +1,97 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns exporter-tests.browser-scope-test
"The browser batch: every export through its own checkout, all at
once, against a stub pool with stub pages."
(:require
["generic-pool" :as gp]
["node:os" :as os]
[app.common.uuid :as uuid]
[cljs.test :as t :include-macros true]
[exporter.browser.scope :as scope]
[exporter.config :as cf]))
(defn- stub-locator
[]
#js {:waitFor (fn [_opts] (js/Promise.resolve nil))
:screenshot (fn [_opts] (js/Promise.resolve nil))})
(defn- stub-page
[calls]
#js {:goto (fn [url _opts]
(swap! calls conj [:goto (str url)])
(js/Promise.resolve nil))
:waitForTimeout (fn [ms] (swap! calls conj [:sleep ms]) (js/Promise.resolve nil))
:waitForFunction (fn [_src _arg _opts] (js/Promise.resolve nil))
:locator (fn [sel] (swap! calls conj [:select sel]) (stub-locator))
:$$ (fn [sel]
(swap! calls conj [:select-all sel])
(js/Promise.resolve #js []))
:evaluate (fn [_f & _args] (js/Promise.resolve nil))})
(defn- stub-context
[page]
#js {:newPage (fn [] (js/Promise.resolve page))
:close (fn [] (js/Promise.resolve nil))})
(defn- stub-browser
[page]
#js {:newContext (fn [_opts] (js/Promise.resolve (stub-context page)))
:isConnected (fn [] true)
:close (fn [] (js/Promise.resolve nil))})
(defn- stub-pool
[calls]
(let [page (stub-page calls)]
(gp/createPool #js {:create (fn [] (js/Promise.resolve (stub-browser page)))
:destroy (fn [browser] (.close ^js browser))
:validate (fn [_] true)}
#js {:max 2 :min 0})))
(defn- test-cfg
[calls]
{:exporter.browser/pool (stub-pool calls)
:exporter/tmpdir (os/tmpdir)
:base-uri (cf/get-internal-uri)
:public-uri (cf/get :public-uri)
:svgo? false})
(defn- test-params
[id]
{:file-id (uuid/next)
:page-id (uuid/next)
:token "token"
:scale 1
:type :png
:objects [{:id id :name id :suffix ".png" :filename (str id ".png")}]})
(t/deftest ^:async run-fires-every-export-through-its-own-checkout
(try
(let [calls (atom [])
seen (atom [])]
(await (scope/run (assoc (test-cfg calls)
:exporter.renderer/on-object (fn [object] (swap! seen conj object) nil)
:exporter.renderer/check-cancelled (fn [] nil))
[(test-params "a") (test-params "b")]))
(t/is (= ["a" "b"] (mapv :id @seen)))
(t/is (= 2 (count (filter #(and (vector? %) (= :goto (first %))) @calls)))))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))))
(t/deftest ^:async run-rejects-on-the-first-failure
(try
(let [calls (atom [])]
(try
(await (scope/run (assoc (test-cfg calls)
:exporter.renderer/on-object (fn [_] (throw (ex-info "on-object broke" {})))
:exporter.renderer/check-cancelled (fn [] nil))
[(test-params "a")]))
(t/is false "run should have rejected a failing on-object")
(catch :default cause
(t/is (= "on-object broke" (ex-message cause))))))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))))

View File

@ -0,0 +1,108 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns exporter-tests.browser-test
"The browser pool service: lifecycle through the system, and the
acquire/use/release contract of exec, against a fake browser factory."
(:require
["generic-pool/lib/errors.js" :as gpe]
[cljs.test :as t :include-macros true]
[exporter.browser :as browser]
[exporter.utils.system :as system]))
(defn- stub-page
[]
#js {:marker :stub-page})
(defn- stub-context
[calls page]
#js {:newPage (fn [] (swap! calls conj :new-page) (js/Promise.resolve page))
:close (fn [] (swap! calls conj :close-context) (js/Promise.resolve nil))})
(defn- stub-browser
[calls]
(let [page (stub-page)]
#js {:newContext (fn [_opts]
(swap! calls conj :new-context)
(js/Promise.resolve (stub-context calls page)))
:isConnected (fn [] true)
:close (fn []
(swap! calls conj :close-browser)
(js/Promise.resolve nil))}))
(defn- fake-create-browser
[calls]
(fn []
(swap! calls conj :create-browser)
(js/Promise.resolve (stub-browser calls))))
(defn- test-config
[calls]
{:exporter.browser/pool {:max 1
:create-browser (fake-create-browser calls)}})
(t/deftest ^:async pool-starts-and-stops-through-the-system
(try
(let [calls (atom [])
sys (await (system/init (test-config calls)))]
(t/is (some? (:exporter.browser/pool sys)))
(t/is (nil? (await (system/halt sys)))))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))))
(t/deftest ^:async exec-acquires-runs-handle-and-releases
(try
(let [calls (atom [])
sys (await (system/init (test-config calls)))
pool (:exporter.browser/pool sys)
seen (atom nil)
result (await (browser/exec pool {} (fn [page] (reset! seen page) :handle-result)))]
(t/is (= :handle-result result))
(t/is (= :stub-page (.-marker @seen)))
(t/is (= [:create-browser :new-context :new-page :close-context] @calls))
(t/is (not (some #{:close-browser} @calls)))
(t/testing "a second exec reuses the released browser: nothing is created again"
(await (browser/exec pool {} (fn [_] :again)))
(t/is (= 1 (count (filter #{:create-browser} @calls)))))
(await (system/halt sys)))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))))
(t/deftest ^:async exec-destroys-the-browser-on-failure
(try
(let [calls (atom [])
sys (await (system/init (test-config calls)))
pool (:exporter.browser/pool sys)]
(try
(await (browser/exec pool {} (fn [_] (js/Promise.reject (ex-info "render broke" {})))))
(t/is false "exec should have rejected when the handle fails")
(catch :default cause
(t/is (= "render broke" (ex-message cause)))
(t/is (some #{:close-browser} @calls))))
(await (system/halt sys)))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))))
(t/deftest ^:async exec-without-pool-fails-fast
(try
(await (browser/exec nil {} (fn [_] :never)))
(t/is false "exec should have rejected without a pool")
(catch :default cause
(t/is (= ::browser/not-started (-> cause ex-data :reason))))))
(t/deftest ^:async exec-translates-playwright-timeouts
(try
(let [calls (atom [])
sys (await (system/init (test-config calls)))
pool (:exporter.browser/pool sys)]
(try
(await (browser/exec pool {} (fn [_] (js/Promise.reject (gpe/TimeoutError. "timeout exceeded")))))
(t/is false "exec should have rejected on a playwright timeout")
(catch :default cause
(t/is (= :timeout (-> cause ex-data :code)))))
(await (system/halt sys)))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))))

View File

@ -0,0 +1,194 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns exporter-tests.consumer-api-test
"The management client of the new tree against a fake http: what it
sends (method uri, shared key, shape of the body) is the whole
contract this client owns."
(:require
["node:fs/promises" :as fsp]
["node:os" :as os]
["node:path" :as path]
["undici" :as http]
[app.common.transit :as transit]
[app.common.uuid :as uuid]
[cljs.test :as t :include-macros true]
[cuerdas.core :as str]
[exporter.config :as cf]
[exporter.consumer.api :as api]))
(def ^:private original-fetch api/fetch)
(def ^:private calls
"Every fetch the client made, in order."
(atom []))
(defn- fake-fetch
[uri request]
(swap! calls conj [uri request])
(js/Promise.resolve #js {:status 200
:text (constantly
(js/Promise.resolve
(transit/encode-str {:action :run})))}))
(defn- failure-fetch
"A fake that answers like a management endpoint that refuses the key."
[_uri _request]
(js/Promise.resolve #js {:status 403
:text (constantly (js/Promise.resolve "{}"))}))
(defn- install-fetch
([] (install-fetch fake-fetch))
([the-fake]
(set! api/fetch the-fake)
(reset! calls [])
(fn [] (set! api/fetch original-fetch))))
(defn- last-request
[]
(second (last @calls)))
(defn- fetch-header
"One header of the last request, exactly as it would travel."
[name]
(unchecked-get (.-headers (second (last @calls))) name))
(defn- body-value
"One field of the last FormData request."
[name]
(.get (.-body (last-request)) name))
(defn- decode-body
"The body of the last transit request, decoded."
[]
(transit/decode-str (.-body (last-request))))
(defn- last-uri
[]
(first (last @calls)))
(t/deftest ^:async claim-job-posts-the-claim-of-one-poller
(let [restore (install-fetch)]
(try
(let [job-id (uuid/next)]
(await (api/claim-job job-id "2026-10-07T00:00:00Z"))
(t/is (str/ends-with? (last-uri) "/api/management/methods/claim-job"))
(t/is (= (str "exporter:" (cf/get :tenant) " " cf/management-key) (fetch-header "X-Shared-Key")))
(t/is (= {:job-id job-id :scheduled-at "2026-10-07T00:00:00Z"}
(decode-body))))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))
(finally
(restore)))))
(t/deftest ^:async json-calls-declare-the-transit-content-type
;; without it the backend never parses the body as transit and
;; validates an empty map (every key a `missing-key`)
(let [restore (install-fetch)]
(try
(await (api/claim-job (uuid/next) "2026-10-07T00:00:00Z"))
(t/is (= "application/transit+json"
(fetch-header "Content-Type")))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))
(finally
(restore)))))
(t/deftest ^:async report-job-progress-posts-the-beat
(let [restore (install-fetch)]
(try
(let [job-id (uuid/next)
progress {:stage :rendering
:counters {:objects {:current 1 :total 2}}}]
(await (api/report-job-progress job-id progress))
(t/is (str/ends-with? (last-uri) "/api/management/methods/report-job-progress"))
(t/is (= {:job-id job-id :progress progress} (decode-body))))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))
(finally
(restore)))))
(t/deftest ^:async fail-job-carries-the-error-and-the-session
(let [restore (install-fetch)]
(try
(let [job-id (uuid/next)
sid (uuid/next)
error {:type :internal :code :processing-error :hint "boom"}]
(await (api/fail-job job-id error))
(await (api/fail-job job-id error :session-id sid))
(t/is (= {:job-id job-id :error error :session-id sid}
(decode-body))))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))
(finally
(restore)))))
(t/deftest ^:async complete-job-with-artifact-rides-multipart
(let [restore (install-fetch)]
(try
(let [job-id (uuid/next)
sid (uuid/next)
tmpfile (path/join (os/tmpdir) (str "test-artifact." (uuid/next)))]
(await (fsp/writeFile tmpfile "the export bytes"))
(await (api/complete-job-with-artifact
{:job-id job-id :session-id sid}
{:path tmpfile
:filename "export.zip"
:mtype "application/zip"}))
(t/is (instance? http/FormData (.-body (last-request))))
(t/is (= (str job-id) (body-value "job-id")))
(t/is (= (str sid) (body-value "session-id")))
(t/is (= (str "exporter:" (cf/get :tenant) " " cf/management-key)
(fetch-header "X-Shared-Key")))
(t/testing "name and type travel in the part itself, no text fields"
(let [body (.-body (last-request))
content (.get body "content")]
(t/is (nil? (.get body "filename")))
(t/is (nil? (.get body "mtype")))
(t/is (= "export.zip" (.-name content)))
(t/is (= "application/zip" (.-type content)))
(t/is (= 16 (.-size content))))))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))
(finally
(restore)))))
(t/deftest ^:async complete-job-with-artifact-streams-a-large-file
(let [restore (install-fetch)]
(try
(let [job-id (uuid/next)
tmpfile (path/join (os/tmpdir) (str "test-artifact-large." (uuid/next)))
payload (.repeat "x" 262144)]
(await (fsp/writeFile tmpfile payload))
(await (api/complete-job-with-artifact
{:job-id job-id}
{:path tmpfile
:filename "large.zip"
:mtype "application/zip"}))
(let [content (.get (.-body (last-request)) "content")]
(t/is (= 262144 (.-size content)))
(t/is (= "application/zip" (.-type content)))
(t/is (= "large.zip" (.-name content)))))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))
(finally
(restore)))))
(t/deftest ^:async a-failed-management-answer-is-the-failure-of-the-call
(let [restore (install-fetch failure-fetch)]
(try
(try
(await (api/claim-job (uuid/next) "2026-10-07T00:00:00Z"))
(t/is false "the call should have rejected on a 403")
(catch :default cause
(let [data (ex-data cause)]
(t/is (= :internal (:type data)))
(t/is (= :failed-management-request (:code data)))
(t/is (= 403 (:status data))))))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))
(finally
(restore)))))

View File

@ -0,0 +1,489 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns exporter-tests.consumer-test
"The export runner of the new tree: plan → render batch → settle,
over a fake management http and a stub renderer. The beats paint the
stage vocabulary in order and the settle carries the artifact; a
failure or a cancel settles as fail-job and never leaks the beat."
(:require
["node:fs/promises" :as fsp]
["node:os" :as os]
["node:path" :as path]
["undici" :as http]
[app.common.schema :as sm]
[app.common.transit :as transit]
[app.common.uuid :as uuid]
[cljs.test :as t :include-macros true]
[cuerdas.core :as cstr]
[exporter.consumer :as consumer]
[exporter.consumer.api :as api]
[exporter.jobs :as jobs]
[exporter.renderer :as renderer]
[exporter.shell :as shell]))
;; ---- THE MANAGEMENT FAKE
(def ^:private calls
"One entry per management call: [method, body]."
(atom []))
(defn- method-of
[uri]
(last (cstr/split (str uri) "/")))
(defn- steps-of
[]
(mapv first @calls))
(defn- call-of
[method]
(some (fn [[m body]]
(when (= m method) body))
@calls))
(defn- fake-fetch
[respond]
(reset! calls [])
(let [original api/fetch]
(set! api/fetch
(fn [uri request]
(let [method (method-of uri)
body (if (instance? http/FormData (.-body request))
(.-body request)
(transit/decode-str (.-body request)))]
(swap! calls conj [method body])
(js/Promise.resolve #js {:status 200
:text (constantly
(js/Promise.resolve
(transit/encode-str
(respond method body))))}))))
(fn [] (set! api/fetch original))))
;; ---- THE RENDER STUB
(defn- stub-render
"A render fn in place of the service: writes one file of `content`
per export and calls back with it. When `captured` is an atom, it
also keeps the last batch it received."
([content] (stub-render content nil))
([content captured]
(fn [{:keys [exports on-object] :as task}]
(when (some? captured)
(reset! captured task))
(js/Promise.
(fn [resolve reject]
(let [writes (mapv (fn [_]
(let [path (shell/tempfile (os/tmpdir) :prefix "penpot.render."
:suffix ".png")]
(.then (fsp/writeFile path content)
(fn [_]
(on-object {:path path
:filename "rendered.png"})))))
exports)]
(-> (js/Promise.all (clj->js writes))
(.then (fn [_] (resolve nil)) reject))))))))
(defn- stub-render-fail
[]
(fn [& _]
(js/Promise.reject (ex-info "render boom" {}))))
;; ---- THE FIXTURE
(defn- export-params
[]
{:exports [{:file-id (uuid/next)
:page-id (uuid/next)
:object-id (uuid/next)
:type :png
:name "test shape"
:suffix ""
:scale 1}]
:name "the export"})
(defn- answer-with
[sid stages]
(fn [method body]
(when (= method "report-job-progress")
(vswap! stages conj (get-in body [:progress :stage])))
(case method
"create-job-session" {:session-id sid :session-token "session-token"}
{:action :run})))
(t/deftest ^:async run-exports-one-shape-to-a-multipart-settle
(let [job-id (uuid/next)
sid (uuid/next)
stages (volatile! [])
restore-fetch (fake-fetch
(fn [method body]
(when (= method "report-job-progress")
(vswap! stages conj (get-in body [:progress :stage])))
(case method
"create-job-session" {:session-id sid :session-token "session-token"}
{:action :run})))
render (stub-render "rendered!")]
(try
(await (consumer/run-export render (os/tmpdir) {:job-id job-id} (export-params)))
(t/testing "the run went by first breath, session, beats, settle"
(t/is (= ["report-job-progress" "create-job-session"
"report-job-progress" "report-job-progress"
"complete-job"]
(steps-of))))
(t/testing "the settle carried the real artifact, named and typed"
(let [fd (call-of "complete-job")]
(t/is (instance? http/FormData fd))
(t/is (= (str job-id) (.get fd "job-id")))
(t/is (= (str sid) (.get fd "session-id")))
(t/is (= "image/png" (.-type (.get fd "content"))))))
(t/testing "the beats painted the vocabulary in order"
(t/is (= [:preparing :rendering :packaging] @stages)))
(t/testing "the local registry entry is gone with the settle"
(t/is (false? (jobs/cancelled? job-id))))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))
(finally
(restore-fetch)))))
(t/deftest ^:async run-hands-the-renderer-uuids-not-claim-strings
(let [restore-fetch (fake-fetch (answer-with (uuid/next) (volatile! [])))
captured (atom nil)
render (stub-render "rendered!" captured)
job-id (uuid/next)
share-id (uuid/next)]
(try
(await (consumer/run-export
render (os/tmpdir)
{:job-id job-id}
{:exports [{:file-id (str (uuid/next))
:page-id (str (uuid/next))
:object-id (str (uuid/next))
:share-id (str share-id)
:type "png"
:name "test shape"
:suffix ""
:scale 1}]
:name "the export"}))
(let [export (-> @captured :exports first)]
(t/testing "every id arrived as a uuid object"
(t/is (uuid? (:file-id export)))
(t/is (uuid? (:page-id export)))
(t/is (uuid? (:share-id export)))
(t/is (= share-id (:share-id export)))
(t/is (uuid? (-> export :objects first :id))))
(t/testing "the job-id traveled with the render"
(t/is (= job-id (:job-id export))))
(t/testing "the task handed to the renderer satisfies the render contract"
(t/is ((sm/validator renderer/schema:task) @captured)))
(t/testing "the task carried both injections"
(t/is (fn? (:on-object @captured)))
(t/is (fn? (:check-cancelled @captured)))))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))
(finally
(restore-fetch)))))
(t/deftest ^:async run-carries-is-wasm-to-single-and-multiple
(let [mk-params (fn [n]
{:exports (mapv (fn [i]
{:file-id (uuid/next)
:page-id (uuid/next)
:object-id (uuid/next)
:type :png
:name (str "shape-" i)
:suffix ""
:scale 1})
(range n))
:name "the export"
:is-wasm true})]
(doseq [n [1 2]]
(let [restore-fetch (fake-fetch (answer-with (uuid/next) (volatile! [])))
captured (atom nil)
render (stub-render "rendered!" captured)
job-id (uuid/next)]
(try
(await (consumer/run-export render (os/tmpdir) {:job-id job-id} (mk-params n)))
(let [exports (:exports @captured)]
(t/testing (str "every export of a batch of " n " carries :is-wasm true")
(t/is (seq exports))
(t/is (every? #(true? (:is-wasm %)) exports))))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))
(finally
(restore-fetch)))))))
(t/deftest ^:async run-carries-skip-children-to-multiple
(let [mk-params (fn []
{:exports [{:file-id (uuid/next)
:page-id (uuid/next)
:object-id (uuid/next)
:type :png
:name "one"
:suffix ""
:scale 1}
{:file-id (uuid/next)
:page-id (uuid/next)
:object-id (uuid/next)
:type :png
:name "two"
:suffix ""
:scale 1}]
:name "the export"
:skip-children true})
restore-fetch (fake-fetch (answer-with (uuid/next) (volatile! [])))
captured (atom nil)
render (stub-render "rendered!" captured)
job-id (uuid/next)]
(try
(await (consumer/run-export render (os/tmpdir) {:job-id job-id} (mk-params)))
(let [exports (:exports @captured)]
(t/is (seq exports))
(t/is (= 2 (reduce + 0 (map (comp count :objects) exports))))
(t/is (every? #(true? (:skip-children %)) exports)))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))
(finally
(restore-fetch)))))
(defn- one-shape
[n]
{:file-id (uuid/next)
:page-id (uuid/next)
:object-id (uuid/next)
:type :png
:name (str "shape-" n)
:suffix ""
:scale 1})
(t/deftest ^:async run-leaves-no-tracked-files-behind
;; The artifact is tracked where make-plan creates it, so the settle
;; release wipes the job area on every path: without the track the
;; multi zip (and the frames pdf below) stays on disk.
(let [restore-fetch (fake-fetch (answer-with (uuid/next) (volatile! [])))
render (stub-render "rendered!")
check-empty (^:async fn [label params]
(let [area (path/join (os/tmpdir) (str "penpot-consumer-release-test." (uuid/next)))
_ (await (fsp/mkdir area #js {:recursive true}))
job-id (uuid/next)]
(try
(await (consumer/run-export render area {:job-id job-id} params))
(let [entries (await (fsp/readdir area))]
(t/testing (str "the job area is empty after a " label " run")
(t/is (empty? entries))))
(finally
(await (fsp/rm area #js {:recursive true :force true}))))))]
(try
(await (check-empty "single" {:exports [(one-shape "one")] :name "the export"}))
(await (check-empty "multi" {:exports [(one-shape "one") (one-shape "two")] :name "the export"}))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))
(finally
(restore-fetch)))))
(t/deftest ^:async run-frames-leaves-no-tracked-files-behind
(let [restore-fetch (fake-fetch (answer-with (uuid/next) (volatile! [])))
render (stub-render "page!")
original-cmd shell/run-cmd
area (path/join (os/tmpdir) (str "penpot-consumer-release-frames-test." (uuid/next)))
job-id (uuid/next)]
(set! shell/run-cmd
(^:async fn [_cmd & args]
(let [dest (last (filter string? (flatten args)))]
(await (shell/write-file dest "joined!"))
nil)))
(try
(await (fsp/mkdir area #js {:recursive true}))
(await (consumer/run-export render area {:job-id job-id}
{:exports [(assoc (one-shape "page") :type :pdf)]
:kind :frames
:name "the export"}))
(let [entries (await (fsp/readdir area))]
(t/testing "the job area is empty after a frames run"
(t/is (empty? entries))))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))
(finally
(restore-fetch)
(set! shell/run-cmd original-cmd)
(await (fsp/rm area #js {:recursive true :force true}))))))
(t/deftest ^:async run-packs-two-shapes-as-zip
(let [restore-fetch (fake-fetch (answer-with (uuid/next) (volatile! [])))
render (stub-render "rendered!")
job-id (uuid/next)]
(try
(await (consumer/run-export
render (os/tmpdir)
{:job-id job-id}
{:exports [{:file-id (uuid/next)
:page-id (uuid/next)
:object-id (uuid/next)
:type :png
:name "one"
:suffix ""
:scale 1}
{:file-id (uuid/next)
:page-id (uuid/next)
:object-id (uuid/next)
:type :png
:name "two"
:suffix ""
:scale 1}]
:name "the export"}))
(t/testing "the artifact is the zip of both objects"
(let [fd (call-of "complete-job")]
(t/is (instance? http/FormData fd))
(t/is (= "application/zip" (.-type (.get fd "content"))))))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))
(finally
(restore-fetch)))))
(t/deftest ^:async a-failed-run-settles-fail-job-and-stops-beating
(let [restore-fetch (fake-fetch (answer-with (uuid/next) (volatile! [])))
render (stub-render-fail)
job-id (uuid/next)]
(try
(await (consumer/run-export render (os/tmpdir) {:job-id job-id} (export-params)))
(await (js/Promise. (fn [resolve] (js/setTimeout resolve 1400))))
(t/testing "the settle was a fail, not a complete"
(t/is (= "fail-job" (last (steps-of)))))
(t/testing "the failure is the shape the backend stores"
(let [fail (call-of "fail-job")]
(t/is (= :internal (get-in fail [:error :type])))
(t/is (= :export-failed (get-in fail [:error :code])))
(t/is (= "render boom" (get-in fail [:error :hint])))))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))
(finally
(restore-fetch)))))
(t/deftest ^:async cancel-before-the-render-settles-cancelled
(let [restore-fetch (fake-fetch
(fn [method _body]
(case method
"create-job-session" {:session-id (uuid/next)
:session-token "session-token"}
"report-job-progress" {:action :skip}
{:action :run})))
render (stub-render "should never render")
job-id (uuid/next)]
(try
(await (consumer/run-export render (os/tmpdir) {:job-id job-id} (export-params)))
(t/testing "the settle of a cancelled job was a fail the backend answers skip"
(t/is (= "fail-job" (last (steps-of))))
(let [fail (call-of "fail-job")]
(t/is (= :job-cancelled (get-in fail [:error :code])))))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))
(finally
(restore-fetch)))))
(t/deftest ^:async run-reports-progress-in-order
;; The intermediate beats are awaited, so the counters land in the
;; order the objects do: the widget never paints a step backwards.
(let [progresses (volatile! [])
restore-fetch (fake-fetch
(fn [method body]
(when (= method "report-job-progress")
(vswap! progresses conj (get-in body [:progress :counters])))
(case method
"create-job-session" {:session-id (uuid/next)
:session-token "session-token"}
{:action :run})))
render (stub-render "rendered!")
job-id (uuid/next)]
(try
(await (consumer/run-export
render (os/tmpdir)
{:job-id job-id}
{:exports [(one-shape "one") (one-shape "two")]
:name "the export"}))
(let [currents (into [] (comp (map :objects) (map :current) (filter some?)) @progresses)]
(t/is (= currents (sort currents))
"counters never step backwards"))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))
(finally
(restore-fetch)))))
(t/deftest ^:async run-joins-frame-pages-as-pdf
;; the frames path end to end except pdfunite itself: the page
;; renders, the join runs in the injected temp area, the artifact
;; settles as pdf. Frames travel declared (`:kind`), not sniffed:
;; the items are fully typed pages, as the frozen job holds them.
(let [tmpdir (shell/ensure-dir (path/join (os/tmpdir) "penpot-consumer-frames-test"))
restore-fetch (fake-fetch
(fn [method _body]
(case method
"create-job-session" {:session-id (uuid/next)
:session-token "session-token"}
{:action :run})))
render (stub-render "page!")
joined (atom nil)
original-cmd shell/run-cmd
job-id (uuid/next)]
(set! shell/run-cmd
(^:async fn [cmd & args]
(let [dest (last (filter string? (flatten args)))]
(reset! joined [cmd dest])
(await (shell/write-file dest "joined!"))
nil)))
(try
(await (consumer/run-export render tmpdir {:job-id job-id}
{:exports [{:file-id (uuid/next)
:page-id (uuid/next)
:object-id (uuid/next)
:type :pdf
:name "page"
:suffix ""
:scale 1}]
:kind :frames
:name "the export"}))
(t/testing "pdfunite joined inside the injected temp area"
(t/is (= "pdfunite" (first @joined)))
(t/is (cstr/starts-with? (second @joined) tmpdir)))
(t/testing "the artifact settled as pdf"
(let [fd (call-of "complete-job")]
(t/is (instance? http/FormData fd))
(t/is (= "application/pdf" (.-type (.get fd "content"))))))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))
(finally
(restore-fetch)
(set! shell/run-cmd original-cmd)))))
;; ---- SERIALIZED BEATS
(t/deftest ^:async beats-run-one-at-a-time
;; The watchdog repeats the last report while a unit beat may be in
;; flight: serialized, the newest milestone can never be overwritten by
;; an older one arriving late.
(let [chain (atom nil)
live (atom 0)
peak (atom 0)
order (atom [])
task (fn [label]
(^:async fn []
(swap! live inc)
(swap! peak max @live)
(swap! order conj [:start label])
(await (js/Promise.
(fn [resolve] (js/setTimeout resolve 2))))
(swap! order conj [:end label])
(swap! live dec)
label))]
(await (js/Promise.all
[(consumer/enqueue-beat chain (task :a))
(consumer/enqueue-beat chain (task :b))
(consumer/enqueue-beat chain (task :c))]))
(t/testing "never two in flight"
(t/is (= 1 @peak)))
(t/testing "and in the order they were queued"
(t/is (= [[:start :a] [:end :a]
[:start :b] [:end :b]
[:start :c] [:end :c]]
@order)))))

View File

@ -0,0 +1,221 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns exporter-tests.consumer-worker-test
"The queue loop of the new tree: the payload it reads, the claim it
makes, and the handoff to the runner, over a fake redis connection
and a fake management http."
(:require
["node:fs/promises" :as fsp]
["node:os" :as os]
["undici" :as http]
[app.common.transit :as transit]
[app.common.uuid :as uuid]
[cljs.test :as t :include-macros true]
[cuerdas.core :as cstr]
[exporter.consumer.api :as api]
[exporter.consumer.worker :as worker]))
;; ---- THE MANAGEMENT FAKE
(def ^:private calls
"One entry per management call: [method, body]."
(atom []))
(defn- fake-fetch
[respond]
(reset! calls [])
(let [original api/fetch]
(set! api/fetch
(fn [uri request]
(let [method (last (cstr/split (str uri) "/"))
body (if (instance? http/FormData (.-body request))
(.-body request)
(transit/decode-str (.-body request)))]
(swap! calls conj [method body])
(js/Promise.resolve #js {:status 200
:text (constantly
(js/Promise.resolve
(transit/encode-str
(respond method body))))}))))
(fn [] (set! api/fetch original))))
;; ---- THE QUEUE FAKE
(defn- fake-conn
[queue]
#js {:blpop (fn [_key _timeout-s]
(if-let [payload (first @queue)]
(do
(swap! queue rest)
(js/Promise.resolve #js ["the.queue.key" payload]))
(js/Promise.resolve nil)))
:quit (fn [] (js/Promise.resolve nil))})
(defn- encoded-claim
"The dispatcher payload as the backend emits it
(`app.worker.dispatcher/encode-payload`): a JSON array of exactly two
items, `[job-id scheduled-at]` with the instant in `ct/format-inst`
shape. Byte-identical literals are asserted on the backend side;
`scheduled-at` travels back untouched on claim."
[job-id scheduled-at]
(js/JSON.stringify #js [(str job-id) scheduled-at]))
(defn- test-cfg
[extra]
(merge {:concurrency 1
:queue-key "the.queue.key"
:exporter/tmpdir (os/tmpdir)}
extra))
(t/deftest read-decodes-the-dispatcher-payload
(t/testing "a payload in shape names the job and when it was pushed"
(let [job-id (uuid/next)]
(t/is (= {:job-id (str job-id)
:scheduled-at "2026-10-07T09:00:00Z"}
(worker/read-payload (encoded-claim job-id
"2026-10-07T09:00:00Z"))))))
(t/testing "a corrupt payload is dropped, not thrown at the loop"
(t/is (nil? (worker/read-payload "{not a payload")))))
(t/deftest ^:async poll-once-awaits-what-it-claims
(try
(let [queue (atom [(encoded-claim (uuid/next) "2026-10-07T09:00:00Z")])
conn (fake-conn queue)
handled (atom [])]
(await (worker/poll-once conn
"the.queue.key"
(fn [claim]
(swap! handled conj claim)
(js/Promise.resolve nil))))
(t/is (= 1 (count @handled))))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))))
(t/deftest ^:async poll-once-drops-empty-pops-and-corrupt-payloads
(try
(let [queue (atom ["{corrupt"])
conn (fake-conn queue)]
(await (worker/poll-once conn
"the.queue.key"
(fn [_claim]
(t/is false "corrupt must not be handled"))))
(await (worker/poll-once conn
"the.queue.key"
(fn [_claim]
(t/is false "a corrupt pop has nothing to handle"))))
(t/is (empty? @queue)))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))))
(t/deftest ^:async process-drops-a-claim-the-worker-never-won
(let [restore-fetch (fake-fetch
(fn [method _body]
(case method
"claim-job" {:action :skip :status "cancelled"}
{:action :run})))]
(try
(await (worker/process (test-cfg {})
{:job-id (uuid/next)
:scheduled-at "2026-10-07T09:00:00Z"}))
(t/testing "only the claim traveled: no run, no render"
(t/is (= ["claim-job"] (mapv first @calls))))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))
(finally
(restore-fetch)))))
(t/deftest ^:async process-renders-the-claim-through-the-injected-renderer
;; the whole vertical slice over fakes: the claim, the runner, the
;; injected render fn and the multipart settle
(let [restore-fetch (fake-fetch
(fn [method _body]
(case method
"claim-job" {:action :run
:name "export-assets"
:params {:exports [{:file-id (str (uuid/next))
:page-id (str (uuid/next))
:object-id (str (uuid/next))
:type "png"
:name "test shape"
:suffix ""
:scale 1}]
:name "the export"}}
"create-job-session" {:session-id (uuid/next)
:session-token "session-token"}
{:action :run})))
ran (atom nil)
render (fn [{:keys [exports on-object]}]
(reset! ran exports)
(js/Promise.
(fn [resolve reject]
(-> (.then (fsp/writeFile "/tmp/penpot-worker-render.png" "rendered!")
(fn [_]
(on-object {:path "/tmp/penpot-worker-render.png"
:filename "rendered.png"})))
(.then (fn [_] (resolve nil)) reject)))))]
(try
(let [job-id (uuid/next)]
(await (worker/process (test-cfg {:renderer render})
{:job-id (str job-id)
:scheduled-at "2026-10-07T09:00:00Z"}))
(t/testing "the one shape rendered with uuids and the job stamped"
(t/is (= 1 (count @ran)))
(let [export (first @ran)]
(t/is (uuid? (:file-id export)))
(t/is (= job-id (:job-id export)))))
(t/testing "the run settled as a complete"
(t/is (= "complete-job" (last (mapv first @calls))))))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))
(finally
(restore-fetch)))))
(t/deftest ^:async start-opens-one-connection-per-poller-and-stop-quits-them
(try
(let [opened (atom [])
quit (atom [])
conn (fn []
(let [c (fake-conn (atom []))]
(swap! opened conj c)
(set! (.-quit c) (fn [] (swap! quit conj c) (js/Promise.resolve nil)))
;; the first pop never answers: the pollers sit in
;; it, so the test never races their loop
(set! (.-blpop c) (fn [_k _t] (js/Promise. (fn [_res _rej]))))
c))
handle (await (worker/start (test-cfg {:concurrency 2 :connect conn})))]
(t/is (= 2 (count @opened)))
(await (worker/stop handle))
(t/is (= 2 (count @quit)))
(t/testing "a second stop is a no-op"
(await (worker/stop handle))
(t/is (= 2 (count @quit)))))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))))
(t/deftest ^:async stop-waits-for-in-flight-runs-within-the-grace
(try
(let [handle {:running (atom true)
:conns (atom [])
:inflight (atom 1)}]
(js/setTimeout (fn [] (reset! (:inflight handle) 0)) 20)
(await (worker/stop handle {:grace-ms 2000}))
(t/is (zero? @(:inflight handle)))
(t/is (false? @(:running handle))))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))))
(t/deftest ^:async stop-gives-up-after-the-grace-period
(try
(let [handle {:running (atom true)
:conns (atom [])
:inflight (atom 1)}]
(await (worker/stop handle {:grace-ms 30}))
(t/is (= 1 @(:inflight handle))
"the grace expired with work still in flight")
(t/is (false? @(:running handle))))
(catch :default cause
(t/is false (str "unexpected failure: " (ex-message cause))))))

Some files were not shown because too many files have changed in this diff Show More