penpot/exporter/README.md
Andrey Antukh 2d80945965 🐛 Fix the review findings of the exporter worker
Await the wasm callback chain before settling a render, so a
multi-object export cannot close its zip with entries still pending.
Hand a payload popped during shutdown back to its queue instead of
running an export the stop is about to cut, and let stop wait for the
poller turn before quitting the connections. Destroy the zip write
stream on the failure paths: it leaked one descriptor per cancelled or
failed job, and releasing the temp path does not close it.

Gate complete-job on the tenant in a single read at the entry, so a
foreign row no longer reveals its status. Normalize the exporter tenant
before validating it: an upper-case value derived another key and
popped another queue. Share the queue payload shape between the
dispatcher and the worker in app.common.types.job, require the export
suffix in the contract, and validate the payload before decoding it.

Count dangling promise rejections as failures in the exporter test
runner, release the export flag when the persist fails, and pin the
progress answer of a missing job. Memory, README and the pool docstring
stop describing behaviour the code no longer has.

AI-assisted-by: deepseek-v4.1-flash
2026-11-04 18:52:23 +00:00

4.4 KiB

Exporter

Node process that renders shapes, frames and files to bitmap, SVG and PDF. It is a consumer: it polls the backend's :exporter job queue on Redis and drives each export through the backend's management API — claim, report the milestones, and settle (complete-job multipart with the artifact, or fail-job on error). There is no HTTP surface of its own and nothing is persisted on this side: the row of the job, its cancel button and its result all live in the backend.

How a job runs

  1. The backend dispatcher pushes [job-id, scheduled-at] (JSON) into penpot.worker.queue:<tenant>:exporter.
  2. One of the K pollers pops it and owns that connection for the whole run: the blocking pop is both the semaphore of the slot and the backpressure, so the process never runs more exports than pollers. K is PENPOT_EXPORTER_WORKER_CONCURRENCY (default 2, and the browser and wasm pools size themselves to the same figure).
  3. The poller claims the job over the management API and, on :run, mints a render session (create-job-session) through which every asset fetch happens as the owner of the job.
  4. The milestones are reported as beats (:preparing, :rendering, :packaging, with the objects and pages counters). A watchdog re-sends the last one every second, whatever the run is doing.
  5. It settles with complete-job (the artifact rides multipart, the session is closed in the same step) or fail-job. The settle never rejects: a worker that cannot talk leaves the row to the backend's lease GC.

Cancellation

A cancellation reaches the worker through the beats: a skip answer means the backend no longer listens (the row is terminal), so the runner stops between units of work; mid-Skia the watchdog terminates the leased render worker. The row was already cancelled by the backend's cancel-job.

Running it

Local (devenv): ./scripts/setup once (installs deps + Playwright chromium), then pnpm run watch in one pane (builds render-wasm and recompiles on change) and pnpm run start:dev in another (loads the backend dev env and runs target/app.js). Code changes arrive through shadow hot-reload: on-before-load halts the running system and on-after-load reboots it in the same process. The tmux exporter window does the same split (./scripts/watch + ./scripts/wait-and-start.sh).

pnpm run start runs the compiled target/app.js with the env as-is (the same node target/app.js the container image uses as CMD). The release bundle is ./scripts/build <version> (wasm + shadow release + version stamp + generated setup); ./manage.sh build-exporter-bundle packs it for the image.

Configuration

Variable Default Description
PENPOT_PUBLIC_URI — The frontend the assets render through
PENPOT_INTERNAL_URI public-uri Same network's frontend; the management API too
PENPOT_REDIS_URI redis://redis/0 The queue and the rest of redis
PENPOT_TENANT default Queue prefix (workspace name in devenv); lowercased on read
PENPOT_TEMPDIR /tmp/penpot Temp area owned by the tmpdir service
PENPOT_SECRET_KEY — Required; the management key derives from it when no shared key is set
PENPOT_EXPORTER_SHARED_KEY — The shared key; derived from the secret otherwise
PENPOT_EXPORTER_JOB_TTL 3600 Lifetime of the temp files a job owns, in seconds
PENPOT_EXPORTER_WORKER_CONCURRENCY 2 K: pollers, running exports, pool sizes
PENPOT_WASM_WORKER_POOL_MIN 1 Render workers kept warm; clamped to the max
PENPOT_WASM_WORKER_IDLE_TIMEOUT 300 How long an idle pooled worker is kept before eviction, in seconds
PENPOT_WASM_WORKER_IMAGE_CACHE_SIZE 134217728 Per-worker image cache budget, in bytes

Inspecting the backend

Nothing about a job is stored here: get-job in the backend answers the full row, and its app.debug/admin surfaces render the same records.