penpot/exporter
Andrey Antukh 676701f74c ✨ Move the exports to the queue worker
The exporter becomes a durable-job worker: the shared item and params
contract, the creation command (mixed scope rejected, single-use runs
marked transient), the external job-def, and the worker that consumes
the `:exporter` queue with one management client, serialized beats,
multipart settle and share-aware asset fetches.

The server side it needs lands here too: per-tenant `exporter:<tenant>`
key ids derived from the same secret (the bare legacy id keeps serving
the deployment own tenant), a heartbeat that filters by tenant in its
own write, and the management API with session mint and settle scoped to
the job owner. The artifact upload is multipart-only and confined to the
parser temp areas.

The frontend follows the job on the websocket: milestones, artifact
download and cancel by job id, with the payload validated against the
contract by a test per caller. nginx, devenv and the docs drop the old
exporter endpoint.

AI-assisted-by: deepseek-v4.1-flash
2026-11-04 17:20:22 +00:00
..
2026-11-04 17:20:22 +00:00
2026-09-29 09:07:34 +02:00
2026-11-04 17:20:22 +00:00
2026-10-29 15:43:06 +01:00
2026-11-04 17:20:22 +00:00
2020-08-05 11:19:34 +02:00

Exporter

Node process that renders shapes, frames and files to bitmap, SVG and PDF. It is a consumer: it polls the backend's :exporter job queue on Redis and drives each export through the backend's management API — claim, report the milestones, and settle (complete-job multipart with the artifact, or fail-job on error). There is no HTTP surface of its own and nothing is persisted on this side: the row of the job, its cancel button and its result all live in the backend.

How a job runs

  1. The backend dispatcher pushes [job-id, scheduled-at] (JSON) into penpot.worker.queue:<tenant>:exporter.
  2. One of the K pollers pops it and owns that connection for the whole run: the blocking pop is both the semaphore of the slot and the backpressure, so the process never runs more exports than pollers. K is PENPOT_EXPORTER_WORKER_CONCURRENCY (default 2, and the browser and wasm pools size themselves to the same figure).
  3. The poller claims the job over the management API and, on :run, mints a render session (create-job-session) through which every asset fetch happens as the owner of the job.
  4. The milestones are reported as beats (:preparing, :rendering, :packaging, with the objects and pages counters). A watchdog re-sends the last one every second, whatever the run is doing.
  5. It settles with complete-job (the artifact rides multipart, the session is closed in the same step) or fail-job. The settle never rejects: a worker that cannot talk leaves the row to the backend's lease GC.

Cancellation

A cancellation reaches the worker through the beats: a skip answer means the backend no longer listens (the row is terminal), so the runner stops between units of work; mid-Skia the watchdog terminates the leased render worker. The row was already cancelled by the backend's cancel-job.

Running it

Local (devenv): ./scripts/setup once (installs deps + Playwright chromium), then pnpm run watch in one pane (builds render-wasm and recompiles on change) and pnpm run start:dev in another (loads the backend dev env and runs target/app.js). Code changes arrive through shadow hot-reload: on-before-load halts the running system and on-after-load reboots it in the same process. The tmux exporter window does the same split (./scripts/watch + ./scripts/wait-and-start.sh).

pnpm run start runs the compiled target/app.js with the env as-is (the same node target/app.js the container image uses as CMD). The release bundle is ./scripts/build <version> (wasm + shadow release + version stamp + generated setup); ./manage.sh build-exporter-bundle packs it for the image.

Configuration

Variable Default Description
PENPOT_PUBLIC_URI — The frontend the assets render through
PENPOT_INTERNAL_URI public-uri Same network's frontend; the management API too
PENPOT_REDIS_URI redis://redis/0 The queue and the rest of redis
PENPOT_TENANT default Queue prefix (workspace name in devenv)
PENPOT_TEMPDIR /tmp/penpot Temp area owned by the tmpdir service
PENPOT_SECRET_KEY — Required; the management key derives from it when no shared key is set
PENPOT_EXPORTER_SHARED_KEY — The shared key; derived from the secret otherwise
PENPOT_EXPORTER_JOB_TTL 3600 Lifetime of the temp files a job owns, in seconds
PENPOT_EXPORTER_WORKER_CONCURRENCY 2 K: pollers, running exports, pool sizes
PENPOT_WASM_WORKER_POOL_MIN 1 Render workers kept warm; clamped to the max
PENPOT_WASM_WORKER_IDLE_TIMEOUT 300 Silence before a worker is terminated, in seconds
PENPOT_WASM_WORKER_IMAGE_CACHE_SIZE 134217728 Per-worker image cache budget, in bytes

Inspecting the backend

Nothing about a job is stored here: get-job in the backend answers the full row, and its app.debug/admin surfaces render the same records.