Nothing of the old exporter deserves preserving: once the plan ends
there is no HTTP surface, so the default of `PENPOT_EXPORTER_ROLES`
stops being `http` and becomes `worker`. The process that dev and
deploy have always started (the tmux window's recipe and the compose
service, both untouched) now boots the K pollers of the backend queue;
`http` stays selectable by env only until the deletion task takes the
legacy surfaces away.
AI-assisted-by: glm-5.3-flash
A cancellation reaches the worker through its beats: a `skip` answer
from `report-job-progress` says the job can no longer hear it. The
runner raises `:job-cancelled` between units of work (the same
exception the legacy surfaces raised), and the render mid-Skia dies by
terminate: the watchdog of the beats fires the cancel port the wasm
pool already serves (`jobs/mark-cancelled` raises the flag, arms the
shared-array signal and runs the terminate callbacks
`renderer.wasm/with-scope` registered), and the run settles as
`fail-job` with the session id, which the backend answers skip on a row
it already cancelled.
The settle is what ends the render session's life, whatever the write
says: complete and fail now close the named session even when they
lose the write, so no settle of the worker leaves a session behind.
The process boots by roles (`PENPOT_EXPORTER_ROLES`): `http` boots the
server and its pools exactly as today, `worker` boots K pollers of the
backend queue, and both roles coexist during the transition sharing
the engines; the shutdown unwinds the worker before redis, so no poller
pops on a closing connection.
AI-assisted-by: glm-5.3-flash
The `:exporter` queue gets its consumer: K pollers (the new
`:exporter-worker-concurrency`), each one with its own Redis
connection, so the blocking BLPOP is both the semaphore of the slot and
the backpressure: a poller claims one job and does not pop again until
the job settles. The dispatcher's `[job-id, scheduled-at]` payloads are
read per pop; a corrupt payload is logged and dropped, a Redis hiccup
is a failed turn that the loop retries after a small delay, and no
test exercises the forever loop directly.
The run of the `:export-assets` job ports the legacy export machinery
to the substrate: the item transducers and partitioning of
export-shapes, the zip assembly and the pdfunite join of export-frames,
rendering as the session the backend minted. The progress contract
dresses the beats (`:preparing`, `:rendering`, `:packaging` with the
`objects` and `pages` counters, throttled per object) and the settle is
one `complete-job` multipart that stores the artifact and closes the
session; an export failure settles as `fail-job` with the session id,
self-contained, so the loop only ever sees a resolved run.
AI-assisted-by: glm-5.3-flash
Two API families with inconsistent argument order, verified against
the 12.0.1 jar: fmap/mcat are fn-first (they pair with ->>), then/catch
are promise-first (they pair with ->). The bind-style ops require the
fn to return a promise; a plain value crashes with "expected
thenable". The repo convention prefers ->> composition, and legacy ->
chains stay as they are. Routes from critical-info and exporter/core.
AI-assisted-by: glm-5.3-flash
The worker side of the exporter gets its config: `PENPOT_EXPORTER_ROLES`
(`http` by default, `http,worker` during the transition) and
`PENPOT_EXPORTER_WORKER_CONCURRENCY` (K pollers, the concurrency of the
process, floored at one), both read from the live config, never at
import time; every accessor is a plain function so the tests do not
journey through globals. The queue key it polls is
`penpot.worker.queue:<tenant>:exporter`, the same one the dispatcher
pushes.
The management client (`app.consumer.api`) is the only place of the
worker that speaks HTTP: claim, progress, session, complete and fail,
all POSTs with `X-Shared-Key: exporter <key>`, transit+json bodies and
transit answers. The artifact form of `complete-job` rides multipart
(the undici FormData the uploads already use); a non-200 answer arrives
as the failure of the call, carrying the status and the error body.
AI-assisted-by: glm-5.3-flash
`complete-job` accepts a second form: the worker sends the artifact
itself as a multipart body, the server stores it in the `job-resource`
bucket owned by the job profile, and the completion coalesces
`resource_id` and the result descriptor `{resource-uri, filename,
mtype, size}` in its own transaction (first terminal state wins; a
lost race releases the artifact it stored, so nothing lingers for the
storage GC). The checks run before any object is written: a lost race
against a cancel never stores bytes, the artifact type must be one the
export produces (new `cm/export-artifact-types` allowlist) and the size
must fit the new `:exporter-max-result-size` config (default 350 MiB)
below the HTTP multipart ceiling.
`complete-job` and `fail-job` take an optional `session-id`, deleted
best-effort after the commit, and every write method answers `:skip`
with the `:status` of the row it lost against, read fresh after the
failed write, so a worker tells a cancelled job apart from one it was
too late for.
AI-assisted-by: glm-5.3-flash
The management method mints a real `http_session_v2` of the owner of a
running job, so a worker renders as the user through the auth surfaces
that already exist (cookies for render.html and assets, Bearer for the
RPC). It refuses anything the login would not accept: an unknown job,
a job that is not running, a job without owner, and an owner profile
that is inactive, blocked or deleted.
Every call mints a fresh session, so worker retries never wait on one
a settle may have closed, and the token carries the short `:exp` of the
new `:job-session-ttl` config (default 1h): a worker that dies without
settling leaves nothing usable for long, and the sessions idle GC
sweeps the row. The token minting moves to a public `generate-token` in
app.http.session, with an explicit `:exp`, and the login path delegates
to it unchanged.
AI-assisted-by: glm-5.3-flash
The job-def freezes the export items the frontend prepares (file, page,
object, type, scale, name and the optional share-id) and routes the job
to the `:exporter` queue through the new `::jobs/queue-name` metadata,
which the creation command reads and the substrate never does. The
handler only raises: the work runs on the external worker, and a wrong
configuration that claimed the job on a JVM runner fails visibly.
Export has one command per job type: `create-export-binfile-job` keeps
the exact behavior the export command had, and `create-export-assets-job`
checks the read permission of every file named in the items before
anything is stored. The family quote still counts every export job of
either kind.
AI-assisted-by: glm-5.3-flash
* ✨ Run binfile import and export as durable jobs
Import and export of `.penpot` files become durable jobs: two commands
create them and answer at once with a job id the caller follows, while
a worker of the `binfile` queue does the work. Each job keeps its own
ledger (owner, params frozen at creation, result, progress history and
an expiry), and the legacy commands and their SSE keep working.
Why: both operations ran inside the request. A large import held it
open for minutes, progress only existed while the caller kept the
stream open, and a client that went away left work nobody recorded,
retried or could cancel. The unified `job` substrate already dispatched,
leased, retried and logged events, but it had no user-facing ledger.
How:
- The substrate gains the pieces a user-facing job needs: optional
job-def metadata (family, resource-role), an expiry on submit, the
owner in the runner context, and job resources in their own bucket,
written with the profile of the job and never deduplicated.
- One generic command per family: the caller names the job and sends its
business params, the registry checks the family, the job-def decodes
and validates, and permission and quota are checked first.
- Progress taps of the core become milestones of the job: a shared stage
vocabulary with self-contained counters, throttled heartbeats, and an
interrupt that stops the run inside its own transaction on cancel.
- Cancel reaches pending and running jobs from both modals, and closing
a modal mid-run cancels the job it was following.
- The export re-checks the read permission at run time and answers with
the package descriptor; the import re-checks the edition permission,
runs the core in a transaction so a cancellation rolls it back, and
releases the consumed package and its temporary copy.
AI-assisted-by: deepseek-v4.1-flash
* ♻️ Handle externally-cancelled binfile jobs end to end
The `cancelled` outcome of a job nobody followed no longer hangs its
screens: the data layers emit an explicit terminal message for it.
- The export emits `{:cancelled true}` per file and the modal paints a
neutral cancelled row instead of success; a message without an
artifact never reaches the download branch.
- The import closes every entry with a dedicated cancelled message
through the existing error rendering, so the wizard completes.
- Both flows are pinned by data-layer tests, including the red state
they replace.
Also in this change: the progress-contract comment of the management
API, early business-params validation in create-import-job (before
assembling the upload), a `release-resource` helper without fabricated
job ids, a note on the heartbeat write cost, and a memory cleanup.
AI-assisted-by: muse-spark-1.3-contributor
* ♻️ Pre-translate the cancelled import message
The cancelled entry of the import wizard carried its message as a key
for the UI to translate, which needed a rehash marker comment. Like
its neighboring errors, it now translates eagerly in the data layer:
the literal call keeps rehash clean with no marker, and the tests
assert through the same call instead of the raw key.
AI-assisted-by: muse-spark-1.3-contributor
* ♻️ Move cancelled strings under generic labels keys
The dashboard-specific cancelled key never shipped: before use it moves
under `labels.*` next to the generic strings both flows already use,
with Spanish translations following the established wording.
AI-assisted-by: muse-spark-1.3-contributor
* ✨ Add minor improvements
* ♻️ Rename export job command to create-export-binfile-job
Export now has one command per job type, the shape the :export-assets
command needs: create-export-binfile-job freezes exactly the
:export-binfile job, so :name no longer travels in the body and only
:params does. Import keeps the generic envelope.
submit-job reads the queue from the job-def metadata ::jobs/queue-name
(falling back to :binfile), read by the command and never by the
substrate, so a job-def can route its work to another queue without a
change in the command.
The rename reaches the frontend and the tests: the binfile export is
asked for under its own name, and the exporter service keeps
create-export-job.
AI-assisted-by: deepseek-v4.1-flash
* ✨ Polish the import and export status rows
The status row of a file already carries what the file is doing, so the
generic "Uploading file…"/"Downloading file…" lines are gone: the
import wizard drops its own (the export one went with the jobs
refactor), along with the now-unused style.
In the export, every file is queued up front, even the ones whose job
is created only after the previous one is over, and a file stays queued
until its first milestone: a worker picking the job up no longer ends
the queue. The queued text is a step smaller than the milestone.
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Show the cancelled export and fix the import error
The cancelled export was unreachable: mark-file-cancelled sets :loading
false and the label lived inside the loading branch, so a cancelled
file only painted a neutral notification. The terminal state now
carries the label, and the export flags lose the `?` suffix
(predicates keep it; a data prop is not a question).
The import wizard rendered (:error entry) through tr, but every
producer writes user-facing text (a hint or a translated label), never
a key: it renders the text as-is now, and the dynamic-key ignores and
the stale comment are gone.
The total-text-limit comment no longer references the 200MiB it
replaced; the value is 500MiB in develop already.
AI-assisted-by: deepseek-v4.1-flash
* ♻️ Reflect the real job params in the creation commands
The creation commands declared their params as an open empty map, so the
RPC contract and the generated docs said nothing about them. Each command
now imports the schema from the job namespace
(`export-binfile/schema:params`, `import-binfile/schema:create-params`),
and the RPC validates the business params before the command runs. The
job-def still owns the params and re-validates the assembled ones at run
time. The commands no longer decode the params again: the RPC did it.
Import joins export as one command per job type: `create-import-job`
becomes `create-import-binfile-job`, its body no longer carries `:name`
and the job-def is resolved by name like export's. Its creation schema
keeps the version optional, because the command reads it from the package
header when the caller does not say it, and fills the manifest metadata
kept for audit.
A version out of range is now refused by the schema (`:params-validation`)
instead of the command's own `:unsupported-version` check, which is gone.
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Stop the test system from running job runners
The test system is main/system-config plus main/worker-config with a
few components dissoc'ed. The runners of default, webhook and cron were
listed by hand; the binfile runner added with the durable binfile jobs
was not, so a real penpot/job-runner/binfile/0 thread ran during the
whole suite.
That runner races the tests that drive the loop by hand (run-batch plus
run-worker-loop): it takes the job the test just dispatched and
completes it, and the test then reads a state it did not produce. That
is the reported flake of binfile-jobs-integration-test, whose middle
assertion waits for a runner of another queue to not take the job. It
only reproduced in CI: in the devenv PENPOT_TENANT makes the tenant
baked into main/worker-config differ from the one the test builds, so
the two used different queue keys locally.
Drop every [<profile> :app.worker/runner] by key shape instead of
listing the known queues, so a new queue cannot bring the race back.
AI-assisted-by: deepseek-v4.1-flash
* ✨ Upload the package one chunk at a time and as a percentage
The upload milestone counts chunks, not units of work, so the row wrote
"Uploading the package 3/9" over a file the user had uploaded once: it
read as if nine packages were on their way. Write the stage as a
percentage of the file instead ("Uploading file… (33%)"), and make the
English text of the stage say file, not package.
The chunks also go one at a time now (concat-all instead of merge-all
with two in flight), so the percentage moves in a straight line instead
of jumping. upload-blob-chunked is the generic uploader, so media and
font uploads become sequential too.
AI-assisted-by: deepseek-v4.1-flash
* 🌐 Complete the Spanish catalogue
es.po was 25 keys short of en.po: the 15 that the durable binfile jobs
introduced (14 jobs.progress.stage.* and labels.queued) plus 10 gaps
that predate them (errors.connection-error, errors.save-retrying, the
find shortcuts, workspace.header.retrying,
workspace.tokens.stroke-width and the mcp session labels).
sync -l es then sorts the entries, syncs the #: comments from en and
drops the 7 dashboard.import.progress.* keys en no longer has. The
msgid sets of en and es are now identical (0 missing, 0 extra).
AI-assisted-by: deepseek-v4.1-flash
* ♻️ Move the job status keys into the jobs namespace
The three status words the durable import and export rows render were
filed under labels.*, next to the generic labels bucket, while their
progress stages live under jobs.progress.stage.*: the strings of one
feature sat in two places.
Rename them so the whole job vocabulary is one namespace:
labels.queued -> jobs.queued
labels.export-cancelled -> jobs.export-cancelled
labels.import-cancelled -> jobs.import-cancelled
No other locale has these keys yet, so the rename costs no translation.
The convention (a key names the feature that renders it; labels.* holds
the generic reusable words) is written down in the frontend/translations
memory.
AI-assisted-by: deepseek-v4.1-flash
* 🔥 Remove the translation keys nothing references
Six keys of en.po (and of es.po, which now mirrors it) have no (tr ...)
call site left in frontend/src or common/src, and no literal reference
either:
labels.uploading-file a duplicate of jobs.progress.stage.upload
since the upload stage says "Uploading file…"
labels.downloading-file dead since the durable jobs replaced the old
import and export progress rows
inspect.attributes.stroke.alignment.center
inspect.attributes.stroke.alignment.inner
inspect.attributes.stroke.alignment.outer
workspace.toolbar.mcp-connect-here
The first two are leftovers of the durable binfile jobs, the other four
predate them. Both catalogues keep an identical msgid set afterwards.
AI-assisted-by: deepseek-v4.1-flash
Remove per-side stroke widths from both Inspect geometry panels so
Stroke is the only section that lists them. Keep dimensions, position,
corner radii, and transforms in Size and position.
Add rendered-component regressions for boards, rectangles, multiple
selection, uniform strokes, and CSS declarations.
Closes#12026
AI-assisted-by: GPT-6
Signed-off-by: Danial Shirali <danialshirali16@gmail.com>
Signed-off-by: Eva Marco <eva.marco@kaleidos.net>
Co-authored-by: Eva Marco <eva.marco@kaleidos.net>
Wrap every id prop in the comp-processors warn logs
with str so the log shows plain text instead of
the #uuid reader form. Also use :hint instead of
:msg and lowercase text, as the rest of the system
does. No logic changes, logging only.
AI-assisted-by: muse-spark-1.3
* 🐛 Show the size badge on paths and straight lines
The badge was hidden when the shape was smaller than the badge in
either side. Flat paths have a zero width or height, so they never
showed it, and straight two-point paths hid it on purpose.
Paths now only compare their longest side with the badge width. A
straight line compares its length, and the badge runs along the
line, rotated with it and kept upright. Near the bottom of the
viewport the badge moves to the other side of the line.
The badge now reads the size from the shape :selrect, like the
measures inputs, so a rotated path shows the same values in both
places. The three badge branches share a single render block.
AI-assisted-by: claude-opus-5-5
* ♻️ Extract the size badge layout and add tests for it
Move the badge geometry out of selection-size-badge* into the pure
size-badge-layout function, so it can be tested without rendering.
The component now only picks the color and draws the result.
Add unit tests for rects, flat paths and straight lines: when the
badge hides, where it sits, how it rotates and how it moves when it
reaches the bottom of the viewport. Keep the line rotation between
-90 and 90 degrees instead of returning 360 for lines drawn leftward.
The e2e test still expected straight lines to hide the badge. It now
checks that the badge shows and follows the line angle.
AI-assisted-by: claude-opus-5-5
* ✨ Show the length of straight lines in the size badge
A straight two-point path is measured by its length, not by the box
around it, so its size badge now shows the length with two decimals at
most instead of width x height. Other shapes and paths keep width x
height.
The shorter text makes the badge narrower, so it also shows on shorter
lines; the rule is the same, the badge needs the line to be at least as
long as the badge.
Relates to #12064
AI-assisted-by: claude-opus-5-5
* 🐛 Hide the path size badge when the path is lower than it
Paths other than a straight line only hid the size badge when their
longest side was narrower than it, so a wide but low path, or a narrow
but tall one, still showed it. Flat paths no longer need that special
case, since straight lines get their own length badge. Use the same
rule as any other shape: hide the badge when either the width or the
height of the path is smaller than the badge.
AI-assisted-by: claude-opus-5-5
Check a milestone against the Main project board with
scripts/project-anomalies.py (check writes
tmp/<MILESTONE>-ANOMALIES.md): open issues with merged PRs,
milestone mismatches either way, needs triage labels and
unassigned non-community issues. Outsiders resolve in bulk;
only a missing milestone auto-fixes, on confirmation.
Teach scripts/gh.py batched issue lookup (issue subcommand),
milestone on prs output, plus assignees and projects fields,
and retry transient HTTP 504s.
AI-assisted-by: muse-spark-1.3
Move the inline programs out of the update-changelog skill into
scripts/changelog.py (check-merged, cross-ref, report) and shrink
the skill to the workflow alone.
Teach scripts/gh.py to retry transient HTTP 504s, resolve lookups
in batches of 50 (new issue subcommand, milestone on prs output),
and report 💥 entries without the breaking change label.
Add the nine missing 2.17.0 changelog entries found while
trying the new flow.
AI-assisted-by: muse-spark-1.3
In list view the height of each component row depended on its
thumbnail, which renders deferred. When the Assets tab mounted again,
rows were shorter until the thumbnails appeared, so the restored
scrollTop landed several rows further down and scroll anchoring kept
it there. Reserve the thumbnail height in the row grid so rows keep
their size from the first render.
Closes#11971
AI-assisted-by: claude-opus-5-5
Adding a flex or grid layout to a board with children derives the
paddings from the distance between the board and its children. A child
that overflows the board, or floating point noise after "Resize board
to fit content", gave negative paddings. The shape schema rejects them
since paddings must be non-negative, so update-file failed and autosave
stopped. Clamp the derived paddings to 0, as the derived gaps already
are.
Closes#12107
AI-assisted-by: claude-opus-5-5
The tokens list container stays mounted when another token set is
selected; only the restore key changes. With no saved position for the
new set, use-restore-scroll did nothing, so the list kept the scroll
position of the previous set. Scroll to the top when nothing is saved
for the key. Freshly mounted panels are already at the top, so Layers
and Assets do not change.
Closes#11973
AI-assisted-by: claude-opus-5-5
* 🐛 Sanitize smuggled SVG on binfile import
Route imported SVG storage objects through the SVG sanitizer on
all three binfile paths (v3, v1, v2), closing the second bypass
of GHSA-ffhp-m958-qxvr. Integrity checks still run on the raw
bundle bytes first; only the persisted copy is the sanitized one.
Adds a shared sanitize-imported-svg helper with unit tests and a
v3 export-tamper-import test proving smuggled scripts no longer
survive the import.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Normalize SVG content-type on binfile import
Close the residual GHSA-ffhp-m958-qxvr bypass where a crafted
content-type spelling (uppercase, parameters) skipped the import
sanitizer. Detection now uses a shared case-insensitive predicate,
stored values are canonicalized, and the v3 re-read honors the
import size limit.
Extends the tamper tests to obfuscated spellings and adds
exhaustive v1 coverage, including the tempfile branch.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Gate binfile content-type on known storage types
Define the complete set of storage content-types in
app.common.media and enforce it on the binfile import schema, so
unknown types fail closed instead of passing through. Detection
keeps a single normalization at the boundary with an exact
predicate, and the string helpers use cuerdas.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Keep only produced types in storage-object-types
Drop apng, avif, penpot and plain text from the set: no flow
stores them, they only exist in the extension mapping table.
Every member must have a producing code path; unproduced types
stay out until some flow actually stores them.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Reject unknown content-type on v1 binfile import
Enforce the storage content-type allowlist on the v1 import
path, like v3 already does via schema. Bundles declaring types
outside the set, or none at all, fail closed with the same
media-type-not-allowed error as uploads. The sanitize branch is
explicit: SVG bytes are sanitized, anything else passes through.
AI-assisted-by: muse-spark-1.3-contributor
Renaming a token with "Remap tokens" grouped the references by container
and wrote the union of their attributes onto every shape in it. A shape
with a stroke width token on one side got the token on all four sides
when another shape in the same container had it on every side.
Group the references by shape, so each shape is only renamed on the
attributes it already had the token on.
Closes#12075
AI-assisted-by: claude-opus-5-5
Since stroke widths can be set per side, stroke width tokens are stored
in the four per-side attributes, not in :stroke-width. The inspect stroke
panel still read :stroke-width for Border width and had no token lookup
for the per-side widths, so it showed the raw value instead of the token.
Read the token of each side for the per-side widths, and the token shared
by every side for Border width. Files that are not migrated yet keep the
token in :stroke-width, so Border width still falls back to it.
Closes#12029
AI-assisted-by: claude-opus-5-5
* 🐛 Show the top stroke width in the old render
With the old render, the stroke width input showed "Mixed" when the four
sides had different widths or tokens. The old render only draws the top
side, and the input must show it as the global stroke width.
Move the input value and token logic into two pure functions. When per
side strokes are disabled, both return the top side: its width and its
token, or no token when the top has none.
Closes#12063
AI-assisted-by: claude-opus-5-5
* 🐛 Add e2e tests for the stroke width input in both renders
Cover the global stroke width input when the four sides differ. With the
new render it shows Mixed. With the old render it shows the top side:
its width, or its token when the top has one.
The old render cannot create per side data from the UI, and switching
the render reloads the page and drops the test state. The tests patch
the existing stroke token fixture instead, appending the per side fields
to a rectangle and a board.
Relates to #12063
AI-assisted-by: claude-opus-5-5
The nested group wrapper only turned into a wrapping flex row when it
had a direct button child. Since the pills are wrapped in the DS
tooltip (#11900), the button is no longer a direct child, so pills in
nested groups were stacked one per row. Match pills as "any child that
is not a group" instead, so the rule no longer depends on the pill's
own markup.
Also drop the .token-pill rules of this module: CSS modules prefix
them with the token_tree scope, so they never matched the pill button.
Closes#12099
AI-assisted-by: claude-opus-5-5
