The subscription to the bus reaches redis through an asynchronous agent
action, so a message published right after the connection opens can be
lost before redis knows the subscription. The delivery assertions of the
websocket tests published once and bet on winning that race, which fails
on a loaded CI runner.
`poll-published!` publishes, drains a window and publishes again only
when the window saw nothing, up to three attempts. The assertion still
proves that the client receives a message it published, and the negative
assertions keep the single drained window they need. A test pins the
retry so it does not go back to a single publish.
AI-assisted-by: deepseek-v4.1-flash
* ✨ Reference the file data storage object with a foreign key
The file data pointer lived in `file_data.metadata.storage-ref-id`, a JSON
key. The storage GC parsed it to find live references, and nothing stopped it
from physically deleting an object a live row still pointed at: a failure in
that scan lost the blob.
Migration 0159 moves the pointer to a nullable `storage_object_id` column on
`file_data`, backfills it from the metadata key, removes the key from the
rows it backfills, indexes the column and rebuilds the foreign key to
`storage_object(id)` as `NO ACTION DEFERRABLE INITIALLY IMMEDIATE`.
`INITIALLY IMMEDIATE` is deliberate: the GC catches the violation inside the
chunk transaction, while a deferred check would fire at commit, outside it,
and the sweep would never move past the blocked chunk.
The column is the only source of truth. The `storage-ref-id` entry is gone
from the file metadata schema, and the single point that serializes the
metadata drops it, so no write can resurrect it. `upsert!` does not query
the row: the pointer of the row being replaced travels in its params, and
the object is touched before the row is written. Fragments and snapshots
pass nil because a modified pointer always gets a new id; the offload passes
its own row; `insert-file!` drops it, since an INSERT never replaces a
persisted object.
`update-file!` takes the file map read from the database plus the fields to
change and merges them, so the pointer comes from the read map and no caller
can lose it by building a partial map. A map without `:data` raises
`:missing-file-data`: it is not `assert`, which is compiled out in the
deployed binary. The params cannot override the pointer or retarget the
write with `:id`/`:file-id`.
`bfc/file->response` strips the pointer from every response (`get-file`,
`create-file`, `duplicate-file`, the view-only bundle, the team access
request, the comments read, the library summary, the snapshot read and the
debug download). A `file-data` object is authentication-only, so a leaked id
would let any logged-in profile fetch the blob from `/assets/by-id`.
`get-file-fragment` resolves through the file data backend, so a fragment on
cold storage no longer hands nil to `blob/decode`, and answers only the keys
of its result schema.
The storage GC reference scan, the restore union and `objects-gc` read the
column. The offload reads `SELECT fd.*`, so its rows go through a
`decode-row` that decodes jsonb metadata and drops a JSON null, which the row
schema rejected; a row without a readable object raises
`:missing-storage-object` with the file and row ids. A chunk blocked by the
foreign key (SQLState 23503) is a real anomaly, since the reference scan
should have frozen the object: it stays an error with its own hint and the
chunk is retried on the next run until the reference is fixed.
The release is a coordinated restart, not a rolling deploy, and not
rollback-safe for the rows it writes: the previous code writes and scans the
metadata pointer, so a key it writes misses the backfill and removing it
loses the reference. `operations/2.20.0-file-data-storage-object-id.md`
builds the column, the index and the foreign key online and runs the
backfill with every instance stopped, then marks the step as applied; a
fresh or small instance does the whole job in the migration. The runbook
keeps the validation scan out of the stopped window and adds a lock timeout
and retry to the online statements.
Tests cover the foreign key shape, the backfill and its corrupt and orphan
guards, the rebuild of a missing index and of a wrong, non-deferrable or
deferred constraint, the deletion refused by a live reference, the metadata
cleanup, the offload row normalizations, the touched objects of the unshare
and the restore, the response stripping on every call site and the chunk
blocked by the foreign key. The backend memories and the developer guide
document the pointer, the GC behavior and the two migration paths.
Closes#12768
AI-assisted-by: deepseek-v4.1-flash
AI-assisted-by: inkling-2-small-free
* 🐛 Hold the file row lock across the migration write
`get-file` applies the pending migrations in memory and, when it finds
any, persists the result. That write read the row without a lock and
`update-file!` is an unconditional UPDATE, so a save that committed in
between was overwritten: the saved content, `revn` and `modified-at`
went back, the next save from that client failed with `:revn-conflict`
and the object the save left became an orphan the storage GC cannot
revisit once it is frozen.
The read now runs in a transaction of its own (`get-file` calls
`get-file*` inside `db/tx-run!` and accepts a cfg map only), so the
`file` row lock covers the re-check and the write. `get-file*` reads
the row once (`sql:get-file` plus `resolve-applied-migrations`) and
completes it with `attach-file-data`; only when a migration is pending
it reads the file again under `FOR UPDATE`, re-checks `need-migration?`
on that fresh map and persists it (`migrate-file`). The common case
stays at one row read and one migrations resolution, the migration case
keeps a single decoded copy in the heap, and a row that changed in
between is not overwritten. A migration found on the unlocked read is
applied in memory only, for the next read to persist.
`migrate-file` returns the reattached map when it persists, with the
`:storage-object-id` of the row the write left, and the realized map
when it does not; `:realize?` is what returns the read fully plain.
`fdata/upsert!` answers `{:written ... :storage-object-id ...}` instead
of a boolean, `update-file!` propagates it and the migrating read puts
it back in the map it returns, so a caller that persists again touches
the current object instead of the replaced one. `insert-file!` keeps
nil: an INSERT never replaces a persisted object.
The write also compares the stored pointer with the caller's value
(`IS NOT DISTINCT FROM`) and raises `:stale-file-data-pointer` when it
affects no row. Every writer holds the `file` row lock, so a pointer
that no longer matches is a protocol violation, not a race to retry.
The expected pointer is a separate argument because the storage branch
replaces `:storage-object-id` before the upsert, and the check lives in
`upsert-in-database`: the `legacy-db` branches do not use that SQL and
their count can legitimately be zero.
The save, snapshot and restore-backup reads take `FOR UPDATE` instead
of `FOR SHARE`: they inherit the lock, and the migration branch asks
for the exclusive one in the same transaction, so a shared lock that
upgrades deadlocks two concurrent readers. A data read rejects
`:lock-for-share?` for the same reason; only row-only reads accept it.
`sql:get-file-without-data` is a plain select now, not a CTE, which
makes locking legal without loading the data. Realization has a single
owner: `attach-file-data` only resolves and decodes, and `get-file*`
realizes once per path. The applied migrations no longer travel in the
`get-file-with-summary` response, cache hit and cache miss alike.
`embed-assets` reads each library without `:realize?`, so its
`:components` is an unloaded pointer map; `absorb-assets` walks it with
`vals`, `PointerMap.seq` calls `load!` and that raised with no loader
bound, so every export with `:merge-libraries` of a library with
components failed. The absorb runs under the library's own loader now.
The in-place binfile import (`import-file-and-overwrite*`,
`::overwrite`, `::bfc/file-id`) had no product caller and cannot carry
the previous `storage_object_id`, so overwriting a storage-backed row
would leave its object unreclaimed. `import-files!` rejects a cfg with
`::bfc/file-id` (`:not-supported`) and the helpers only that path used
go with it (`invalidate-thumbnails`, `*reference-file*`, the
`::overwrite` id branch of `index-object` and of `save-file!`, the
`on-conflict-do-nothing?` flag of the media insert).
Tests: a new `file-migration-read-test` pins the single read, the
locked re-read, the concurrent save interleaving, the double migration
and the read-only path; the upsert tests cover the return shape and the
stale pointer; the SQL recorder they share lives in
`backend-tests.helpers`, and the save, snapshot, restore and summary
sequences follow the new reads. The backend memories document the read
contract and the pointer.
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Harden the get-file guard and skip its read-only transaction
The cfg guard of `get-file` was an `assert`, which `app.config` elides
when `:backend-asserts` is off, so a deployed binary accepted a bare
pool or connection while the docstring promised it raises. It throws
`IllegalArgumentException` now, the class the option guards of the same
namespace already use, and the existing test expects it instead of
`AssertionError`.
A read with `:read-only?` applies the pending migrations in memory and
writes nothing, so the transaction the migration write needs only costs
two round trips there. `get-file` runs it with `db/run!` instead: with
a cfg that carries a connection it still joins the caller's
transaction, and only the reads that would have opened their own skip
it. A lock option needs a read-write transaction from the caller,
because PostgreSQL rejects row locks in a read-only one; the docstring
states that.
Tests: the guard test expects the new class; a cfg marked
`::db/read-only` reads a pending migration without persisting it (the
config flag that fills that key is legacy, but the key is set by hand);
and a read-only read is pinned as not opening a transaction while the
same call without the option does. The backend memory documents the
split and the guard.
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Revoke websocket subscriptions when access is revoked
GHSA-m53j-2766-6jqw. A websocket subscription was authorized once,
when it opened, and that decision was trusted for the rest of the
connection's life, so a member whose access was revoked kept receiving
live file content as long as the tab stayed open.
The revocation travels over the message bus instead of closing the
socket from the mutation: the connection registry is local to one
backend instance, while the RPC that revokes access runs on whichever
instance received the request. Every instance re-verifies the
subscriptions it owns with the same `has-read-permissions?` predicates
that `:subscribe-file` and `:subscribe-team` authorize with, so a
downgrade to viewer keeps its subscription and a non-member
organization owner keeps read-only access.
Two mechanisms cover each other:
- Access-changing mutations announce after the write. delete-team-
member, leave-team, update-team-member-role, delete-file,
delete-project, move-files, move-project and delete-team announce.
delete-share-link is deliberately excluded: a share-link visitor
never passes the subscribe permission check, so there is no
subscription to revoke. Team-wide changes publish one event per
team; the watcher resolves the members itself, and both publishers
defer through `db/after-commit` so the watcher never decides from
pre-commit state.
- Every relay re-checks access every 5 minutes by default
(`subscription-revalidation-interval`), driven from the relay loop
with `alts!` over the subscription channel and a one-shot tick
channel, so a steady stream of edits cannot push the check away
forever.
Supporting changes:
- The registry holds `:connections` and `:by-profile` in one atom,
kept consistent by register/unregister in a single swap!; a profile
lookup is one map access, and a profile whose last connection
leaves is dropped from the index.
- `close-file-subscription` and `close-team-subscription` announce
the :leave-* message, close the relay channel and purge the bus
subscription, and are idempotent. Closing the channel ends the
relay loop and its re-check task.
- The permission query the re-check uses filters soft-deleted files,
projects and teams, matching its sibling queries.
- A transient database error fails open: it logs and keeps the
subscription; the next tick retries.
- The revocation watcher's params schema declared the nitrate client
as a bare keyword inside [:map ...], which malli resolved as a
registry lookup and which took the whole system build down under
enable-backend-asserts. It is now an explicit pair, optional
because init-key yields nil with the admin-console flag off.
The tests reproduce the leak against the real message bus, including
the negative control for the interval re-check. The written records
live in mem:backend/websocket-subscriptions.
Closes#12155
AI-assisted-by: space-bunny-free
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Mirror the subscribe gates in the revocation re-checks
Addresses the second review round on PR #12189 (pabloalba).
The re-checks only asked whether the profile could read. The membership
legs of sql:file-permissions filter soft-deleted rows since the F2 fix,
but the organization-owner fallback grants read-only access over HTTP
without looking at deleted_at, so a soft-deleted file or team kept an
expelled org owner subscribed indefinitely. The re-checks now gate on
existence as well: file-readable? and team-readable? require a live row
(db/get*, non-throwing, soft-deleted rows excluded) plus the read
permission, mirroring the :subscribe-file gates and extending them to
teams. Both relays and revalidate-profile-subscriptions use them; the
subscribe handlers keep their exact behavior.
Also from the same review:
- watcher-consumes-revocation-events-from-the-bus no longer races the
command's after-commit announcement: it takes access away with a
direct row delete and drives the watcher with a synthetic bus event,
instead of deleting through the command and publishing a second
notify-* that would mask a broken command announcement.
- moving-a-project-closes-source-team-subscriptions pins the
move-project e2e: the file subscription drops, the source-team
subscription survives (membership unchanged).
- soft-deleted-file/team-closes-org-owner-subscription cover the new
gate with red-before/green-after tests.
mem:backend/websocket-subscriptions updated.
Closes#12155
AI-assisted-by: inkling-2-small-free
* 🐛 Refuse dead teams in resolve-team-id (PR #12189 review)
Second review round on PR #12189 (pabloalba).
The file re-check added in 87ca7aca53 stops at the file row, but the
organization-owner fallback grants read-only access to non-member org
owners via Nitrate over HTTP, which knows nothing about Penpot
soft-deletion. resolve-team-id resolved the team from unfiltered SQL,
so an expelled org owner kept the file subscription open for an
indefinitely soft-deleted file, project or team. Now both SQL lookups
filter deleted_at, and resolve-team-id checks the team row itself
(db/get*, which filters soft-deleted rows) for every caller — teams,
projects and files alike.
Also from the same review:
- file-readable? now uses the same sql:get-minimal-file as the
subscribe gate (non-throwing via throw-if-not-exists false), and
team-readable? projects id and deleted-at, so db/get* can filter.
- The test suite extracts the thrice-duplicated raw delete into
delete-team-membership!; a sibling of the fail-open pin now redefs
the existence probe to throw; memory and docstring aligned with the
code.
- docs/technical-guide/developer/architecture/backend.md attributed
the websocket to notifications.clj; it now names app.http.websocket
as the owner and notifications.clj as the revocation publisher.
- mem:backend/auth-permissions-product-domains documents the fallback
mechanism and the resolve-team-id contract.
Closes#12155
AI-assisted-by: inkling-2-small-free
* 🐛 Test resolve-team-id refusal and tidy review findings
Third pass over PR #12189 after review-code on 225eb98de2.
resolve-team-id now has a direct test: all three branches resolve
while rows are live, and file, project and team soft-deletes each
refuse their branch — the two new SQL predicates and the final team-row
check are all pinned by it.
From the same review:
- failing-existence-check-keeps-the-subscription and the sibling
a-failing-check-keeps-the-subscription are deleted: they could not
distinguish a live relay from a dead one. The fail-open policy
remains documented in the docstring and the memory; the memory no
longer claims the pins exist.
- team-readable? projects ::sql/columns like the rest of the backend;
the fully-qualified keyword went away with the new alias.
- The dead when-not guard in get-organization-owner-viewer-team is
removed (resolve-team-id guarantees the row live); the let binding
in resolve-team-id no longer shadows the destructured team-id.
- docs/technical-guide names rpc/notifications.clj as the publisher
and the tree lists the real top-level modules.
Closes#12155
AI-assisted-by: inkling-2-small-free
* 🐛 Make the revocation tests deterministic
Third review round on PR #12189 (pabloalba).
Both tests took access away through a command that announces, so the
live watcher could close the subscriptions before the test looked:
- revocation-closes-subscriptions-that-are-no-longer-authorized
asserted an exact count of 2 closes after delete-team-member, and
that count reads 0 whenever the watcher got there first. It now
removes the member with delete-team-membership!, the raw row delete
that publishes nothing, and asserts both subscriptions are still
open before calling revalidate-profile-subscriptions.
- team-event-closes-every-member-subscription ran delete-team and then
published a second notify-team-permissions-changed, so a missing
announcement in delete-team still passed. It now soft-deletes the
team row (sql:team-members joins team_profile_rel and profile only,
so the watcher still resolves every member), asserts nothing closed,
and drives one synthetic team event.
The command paths stay covered by the announce and end-to-end tests
for delete-team-member and delete-team. The file split the reviewer
suggested stays the optional follow-up, tracked outside this PR and
waiting for the merge.
Closes#12155
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Allow OIDC registration with password login disabled
The register-profile command rejected every registration without the
login-with-password flag, including SSO ones, while the OIDC callback
already allowed :oidc-registration as an alternative, so new OIDC
users were stuck at the register screen with a 400
registration-disabled error.
The gate now branches on the verified token backend: password tokens
keep the previous checks, SSO tokens only require :registration or
:oidc-registration through a predicate shared with the callback, so
the two gates cannot drift apart again. Flags and invitation checks
are split into single-responsibility helpers, and tokens without a
backend claim are rejected loudly by schema instead of defaulting
silently. Covered by regression tests for the SSO matrix, canceled
invitations, the missing-backend case and the no-password-fallback,
plus a configuration doc note.
Closes#12964
AI-assisted-by: Muse Spark 1.3 Contributor
* 🐛 Gate SSO registration by provider type, accept invitations
Follow up the review on #12974. The SSO registration gate now treats a
team invitation token as a substitute for :registration /
:oidc-registration, in both the OIDC callback and register-profile,
mirroring the password flow: the callback redirects to the register
screen when an invitation is present, and register-profile revalidates
the invitation row under a row lock before creating the profile when
no SSO flag is set, raising :invalid-token instead of creating an
account that cannot join a team.
The SSO branch is restricted to OIDC-typed backends
(oidc-provider-types); a prepared-register token carrying any other
backend is rejected with :invalid-token instead of silently taking
the SSO rules. The flags and invitation checks are shared by one
check-registration helper in register-profile; the prepare command
runs flags, then the email-mismatch restriction, then the
invitation row lookup, so the password login flag fails closed first.
The schema defs are private, the create-profile default uses
penpot-backend, the OIDC callback test decodes the redirect token to
assert the :prepared-register claims, and the docs and the auth
memory state the invitation path for SSO registration.
Regression tests cover invitation-only registration, canceled
invitations with and without SSO flags, unknown backends, and the
double-fault precedence in prepare-register.
AI-assisted-by: inkling-2-small-free
* 🐛 Require invitation email match for SSO registration
Close the hole left by treating any invitation token as a substitute
for the SSO registration flags. register-profile now requires the
invitation member-email to match the prepared-register email, raising
:email-does-not-match-invitation otherwise, mirroring the password
flow. The OIDC callback verifies the team-invitation JWT and the
email before redirecting to register instead of trusting raw
presence. Covered by the requested mismatch regression test plus
callback tests for mismatch and garbage tokens.
AI-assisted-by: Muse Spark 1.3 Contributor
* ♻️ Share registration gate between prepare and register-profile
After the email-match fix, check-registration runs flags, email
and invitation row in the same order as validate-register-attempt!.
The prepare command now calls the shared helper instead of inlining
the trio, and the stale docstring rationale is gone, so future
invitation rule changes land in one place.
AI-assisted-by: Muse Spark 1.3 Contributor
Migration 0158 created file_change with kind NOT NULL, which blocks
any emergency rollback to the previous version: the old code inserts
without kind and every save would fail. kind is still a reserve
(reads use label IS NOT NULL), so it stays nullable until the
rollback window closes; the NOT NULL comes in a later migration.
The runbook keeps parity, the schema and migration tests cover the
nullable column and the rollback insert, and the storage memory
records the decision.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Fix font upload crash and preserve variant metadata
opentype.js 2.0.0 returns no fontSubfamily name for fonts whose
Macintosh name table only holds custom records. That made
parse-font-weight/parse-font-style call re-seq on nil and crash the
upload.
A nil or blank variant is now treated as no match in
app.common.media, defaulting to weight 400 and style normal. The
variant parsing also accepts oblique as italic, and the new
parse-font-family helper derives the family from a filename by
stripping the known weight/style tokens, separating a style token
glued to a weight token ("Roboto-BoldItalic" -> "Roboto").
first-nonblank-string returns the first nonblank candidate, so a
blank preferredFamily/preferredSubfamily no longer shadows a valid
fontFamily/fontSubfamily.
app.main.data.fonts gains resolve-font-metadata and
prepare-font-entry. opentype.js values win per field; any missing
value falls back to the filename-derived family, weight and style.
The parsed and woff2 branches share the same helper, so metadata is
preserved instead of collapsing every face to weight 400, and
fonts with missing vertical-metric tables upload with no height
warning instead of failing.
process-upload reports per-file failures (unknown signature,
unparseable file, preparation error) with a single bad-font
notification instead of silently dropping them, and never errors
for per-file failures.
The upload now sends :variant-name to create-font-variant, which
accepts it as an optional param (ships in 2.20), so the foundry
name survives the upload.
Closes#10436
AI-assisted-by: deepseek-v4.1-flash
AI-assisted-by: muse-spark-1.3-contributor
Signed-off-by: CHOI JUNSOO <junsoo1172@gmail.com>
* 🐛 Parse glued font style tokens in the font parsers
EvaMarco's review on #11951: parse-font-family and
parse-font-weight handled glued tokens like "BoldItalic", but
parse-font-style did not. For names such as
"opensans-bolditalic-webfont.woff2" the three parsers disagreed:
the italic face resolved to style normal and collided with the
bold-only file on the (family, weight, style) join key. One
binary silently overwrote the other during upload, losing one
font face without any warning.
A new private normalize-font-name helper is applied by all three
parsers. It splits a style token glued to a weight token, now
with a lookahead boundary so "LightItalic_v2" splits too, and
strips a trailing technical suffix (-VF or _v<number>).
parse-font-family, parse-font-weight and parse-font-style now
interpret the name consistently.
Tests cover the three cases from the review, the bold-only
control case, and an end-to-end process-upload check that a bold
and a bold-italic WOFF2 survive as two variants.
Relates to #10436 and PR #11951.
AI-assisted-by: inkling-2-small-free
* Remove duplicate frontend test registration
---------
Signed-off-by: CHOI JUNSOO <junsoo1172@gmail.com>
The test expected exactly one publication of the :rpc-climit-waiting
gauge inside the with-redefs window. The republisher daemon thread,
started at component init with no delay and rescheduling every 15 s,
publishes the same counter and can land inside that window, so the
recorded sequence could be [0 0] and the assertion failed with
actual [0 +0] (Backend Tests job in #12979).
Assert on the last published value instead of the whole sequence,
the contract waiting-last documents and the sibling tests use.
AI-assisted-by: inkling-2-small-free
* 🐛 Skip circular library links on binfile import
File import recreated circular library references that the
link-file-to-library command already rejects. The v3 importer
now skips manifest relations that would close a cycle, or link
a file to itself, and logs each skipped link. Closes#12052.
AI-assisted-by: muse-spark-1.3-contributor-free
🐛 Address review findings on circular import guard
F1 extracts the cycle check into bfc/library-link-status and reuses
it in link-file-to-library and v3 import, keeping UI errors intact.
F3 orders get-files-rels and manifest relations so the first edge
wins deterministically, and documents it.
F2 adds a three-node transitive cycle test; F4 asserts file_library_sync
has no orphan rows for skipped links and matches rels otherwise.
AI-assisted-by: muse-spark-1.3-contributor
🐛 Pass team-id in binfile v3 circular library tests
The four circular library relation tests (import-skips-circular,
import-skips-transitive-cycle, import-skips-self-referencing,
import-keeps-acyclic) called v3/import-files! without ::bfc/team-id.
resolve-and-link-libraries asserts (uuid? team-id), so with
enable-backend-asserts the tests errored instead of running.
Add (assoc ::bfc/team-id (:default-team-id profile)) to each test,
mirroring the production RPC import path and the earlier fix in
a36193e54b for the other binfile v3 import tests.
Verified: 1338 tests, 6910 assertions, 0 failures.
AI-assisted-by: mistral-large-4
* 🐛 Guard the v1 import and restore the RPC check order
The version 1 importer recreated the circular library references that
the version 3 importer already skips, and a user reaches it through
`import-binfile` and the import job. It now asks
`bfc/library-link-status` before inserting each relation, so a self
link or an edge that would close a cycle is skipped and logged, and it
processes the relations in sorted order like v3 does. Closes#12052.
The version 2 importer stays out on purpose: only the developer REPL
reaches it, and a note in the code records the gap.
`link-file-to-library` keeps the shared helper but restores the order
of the base commit: the self link guard, then the edition permission
and team ownership checks, and only then the cycle check. A caller
without permission gets `not-found` again instead of the circular
error, and a rejected request no longer runs the recursive library
query first.
Tests: a v1 import that recreates a cycle, the self link error of the
command, and the permission check running before the cycle check.
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Tolerate repeated file entries in the v1 binfile import
The v1 exporter unioned the requested files with their libraries
without deduplicating, so a package exported from a file inside a
library cycle (see #12052) lists that file twice in the metadata. The
v1 import created one file per entry, so the second insert hit the
primary key of `file` and aborted the whole import with "file already
exists".
The importer now reads the pair of objects of every entry to keep the
stream in sync but imports the first occurrence only, warns about the
repeats, and answers with one id per imported file.
The v1 export keeps its behavior: it has no caller in the product, and
only the tests and the developer REPL use it.
Tests: a package with a repeated entry, built with the v1 export from
a cyclic library pair, imports the file once and leaves no duplicated
rows.
AI-assisted-by: deepseek-v4.1-flash
* ✨ Partition file_change by file_id
Partition `file_change` by `HASH (file_id)` in 16 partitions, with the
same bounds as `file_data`, so a join between them is resolved partition
by partition. The primary key becomes `(file_id, id)`, which a
partitioned table needs, and every access filters by `file_id`.
The new table carries a `kind` column (1 snapshot, 2 revision) as a
reserve: the queries still tell them apart by the label, and `kind`
takes over when they are rewritten. The payload of a snapshot stays in
`file_data`; the `data` column is kept for reading old `legacy-db`
payloads.
The four snapshot commands require `file-id`, because it prunes the
partition. The objects GC locks and deletes by the pair, in batches. The
old table is kept as `file_change_legacy` as the way back.
The automatic migration refuses to run above 10 GiB or 2M rows and
points to the runbook; big instances partition online with that runbook,
which copies by id, follows the `updated_at` watermark in delta passes
and checks that no row of the source is missing from the target.
Closes#12416
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Address the review of the file_change partition
The production runbook promised a rollback it did not have: the section
was a sketch, and its statements could write to the table about to be
dropped. The rollback is removed, and so is every comment that described
it. `file_change_legacy` stays as a frozen copy of the previous layout.
The cutover check now aborts the transaction when a row of the old table
is missing in the new one, instead of leaving the decision to the
operator.
The five secondary indexes of the new table are built after the last
delta pass: the bulk and the delta passes only need the primary key, so
the batches no longer maintain them.
The copy of 0158 is now replayed in a test against the pre-0158 shape
with a labelled and an unlabelled row, plus an empty instance that must
leave no legacy table behind. The updated_at assertion of the snapshot
mutations is bounded by the test clock, and the repair revision stops
writing the `features` of the file, like the normal save.
AI-assisted-by: deepseek-v4.1-flash
* 📎 Fix linter issues
* 🐛 Fix stale helper call in rpc_file_snapshot_test
The suite does not load: `th/run-pending-tasks!` does not exist in
backend-tests.helpers. The pending jobs runner is `run-pending-jobs`
(no arguments), the name every other call site of the suite uses.
AI-assisted-by: inkling-2-small-free
* 📚 Align the partition change with the 2.20.0 milestone
2.19.0 is already shipped; this work targets 2.20.0. The operations
file of the partition procedure carries the 2.20.0 prefix, the
references that point at it (the migration, the runbook table and the
memory) follow, and the RPC doc/changes tags of the snapshot commands
record 2.20 as the release that makes `file-id` required.
AI-assisted-by: inkling-2-small-free
* 🐛 Fix review findings on the partition rebase
The 0157 legacy test was missed by the rebase and inserted file_change
rows without the new mandatory `kind` column, so the backend suite fails
against the migrated schema. The inserts now carry `:kind`
(snapshot-kind for the labelled rows, revision-kind for the rest).
The post-cutover verification block of the runbook terminates the
first query with a semicolon so the block pastes into psql without a
syntax error.
The guard message of the migration names the step
(0158-partition-file-change) instead of "0158", ambiguous since two
migrations share that number.
Closes#12416
AI-assisted-by: inkling-2-small-free
* 🐛 Fix the stale label in the snapshot restore test
The rename block of snapshots-crud runs before the restore block,
so the restored version is already named "label1-updated" when the
restore publishes its message. The command names the version by the
label the row has at that moment, not by the one it had when it was
created, so the test was asserting a stale value.
Assert the current label, and say why in the comment.
AI-assisted-by: deepseek-v4.1-flash
* ♻️ Split the version event tests into one test per command
The workspace version events and the plugin proxy send the file-id
of the open file on every snapshot command, because the command
needs it to prune the partition of file_change. Both tests checked
that with a doseq and t/testing blocks, so a failure named the whole
group and not the command that broke.
Give each command its own test. The workspace ones share a helper
that runs the event and returns the command and params it sends; the
plugin ones keep their own mock of rp/cmd!.
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Require thread access when marking threads as read
mark-all-threads-as-read took any list of thread ids and wrote the read
status of the caller on each of them, without checking that the caller
can access the file that owns the thread. It was the only comment
mutation that did not check permissions, and it took the ids as an
unbounded array, so a single request could keep a database connection
and a transaction open for a long time.
The rpc now resolves the threads in one query and checks the comment
permissions of every distinct file, answering not-found when a thread
does not exist or is not accessible. Both cases answer the same error
on purpose, so the response cannot be used to tell existing threads
apart.
The status rows are now written with a single insert, keeping the
modified-at date one second in the future that makes the thread count
as read. Repeated ids are deduped, the command returns nil again like
the sibling commands with no result, and the schema limits the number
of threads per call to 500 while the dashboard sends the ids in batches
of that size.
Tests cover access rejection, unknown threads, mixed batches that write
nothing, repeated ids, empty lists, marking twice, and the frontend
batch split.
Relates to GHSA-j4vc-cmf6-jhr7
AI-assisted-by: minimax-m3.1-flash
* 🐛 Refresh the unread list when a batch fails
The dashboard sends the thread ids in batches of 500 and each batch
commits on the server. A failure in a later batch left the unread list
stale: the re-read only ran when every batch succeeded, so the threads
already marked as read stayed visible as unread.
The re-read now runs in both paths. On failure the event is emitted
before the error is propagated, so the error toast still shows and the
success toast stays limited to a full success.
The backend also answers an empty list without opening a transaction,
since there is nothing to check and nothing to write.
Relates to GHSA-j4vc-cmf6-jhr7
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Restrict the team invitation token and list to admins
The token of a team invitation is a bearer credential: whoever holds it
accepts the invitation with the role of the invitation row. The RPC that
mints it only required edition permissions, while the rest of the
invitation commands require admin, so a team editor could get the token of
any pending invitation, including the ones that grant the owner role.
The token now requires admin, and the token of an owner invitation is
restricted to the team owner: an admin cannot create nor promote an owner
invitation, so it cannot keep its token either. It reuses the error code
already used for that rule.
Minting the token now leaves an audit event named as an action, carrying
the granted role and the id of the invitation, so the extraction does not
go unnoticed.
The list of pending invitations is admin only as well: it carries the
email of the invited person and the role offered, and the dashboard was
painting those emails to any member. The frontend catches the permission
error and shows the permission notice instead, and the copy action is not
offered for an owner invitation, which the backend rejects for an admin.
The five invitation commands now answer with the same permission error,
so they speak one language.
Relates to GHSA-4mmc-j8rh-74pf
AI-assisted-by: minimax-m3.1-flash
* 🐛 Clear the cached invitations and gate the list on team admin
The pending invitations are a team admin only list. The dashboard still
offered the entry to every member and keyed the section on the invite
policy of the organization, so a team admin blocked by that policy saw
the no-permission notice although the backend lets them list.
- `fetch-invitations` answers a denied refetch with an
`invitations-fetched` event carrying an empty list, so a demoted admin
does not keep the cached emails and the header no longer paints the
invite button from a stale list.
- The header tabs and the team options dropdown hide the invitations
entry without `:is-admin`, which already includes the team owner.
- `invitations-section-content` keys the section on `:is-admin`, and the
invite policy only gates the invite actions: resending keeps the
`can-invite?` check, deleting and row selection only need team admin.
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Apply the share-link page scope to the comment queries and writes
A share link can be restricted to a subset of the pages of a file, and
the comment queries enforced that scope. But the RPC that resolves the
profiles of the commenters authorized the request with
check-comment-permissions! and dropped the returned perms, so its query
filtered only by file id. The holder of a page scoped link got the id,
email, fullname and photo of every profile that commented on any page
of the file, including the pages the owner left out of the link.
The query now takes the pages of the share link and only selects the
authors of the threads on them. The profile of the caller is still
added, so the viewer keeps its own avatar.
The scope was missing as well on the comment writes that take a page id
or a thread id: create-comment-thread accepted a page outside the link,
create-comment accepted a thread of an unshared page, and
update-comment-thread, update-comment-thread-position and
update-comment-thread-frame accepted a thread of an unshared page. They
now raise not-found, the same way the comment queries already did.
Replying to a thread of an unshared page also sent the emails to the
thread participants.
The page scope rule now lives in one place: share-link-pages returns
the pages granted by a share link and check-page-scope raises not-found
when a page is outside of it, so get-comment-threads reuses the helper
instead of checking the pages inline.
The docstring of get-profiles-for-file-comments said that it returns
the participants of all the comment threads of the file, which is not
true anymore for a share link. The docs are public, so the text and
the changes list are updated.
The tests of the share link page scope are hardened: the setup asserts
that the comment threads are created, the membership case has a second
commenter so it can no longer pass for the wrong reason, and the link
without pages, the allowed paths of the thread updates and the
who-comment=team denial are covered.
Relates to GHSA-89cq-3wv7-987r
AI-assisted-by: minimax-m3.1-flash
* 🐛 Enforce the share-link page scope on the remaining comment writes
Check the page scope of the share link before mutating in the four
comment writes that still skipped it: update-comment-thread-status,
update-comment, delete-comment-thread and delete-comment. The check
runs before the ownership checks, so an out-of-scope write fails with
not-found and the response does not reveal that the comment or thread
exists. Load the file in create-comment only after the permissions and
page scope checks, matching create-comment-thread, so a rejected reply
does not load the file data.
Cover the denial and the in-scope paths of the four commands, and the
fail-closed case of a link without pages on the comment reads. Move
the outsider thread creation after the profiles assertion so the
caller-union assertion fails when the union is removed. Document the
page scope invariant in the backend permissions memory.
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Escape SVG values and sanitize every SVG upload
The SVG sanitizer removed dangerous elements and attributes by name and
then wrote the tree back with clojure.xml/emit, which escapes nothing.
A payload hidden in an attribute value or in a text node became live
markup again: an id of "x' onload='alert(1)'" was stored as a real
onload handler, and an entity encoded script tag was stored as a real
script element.
Write the filtered tree with a serializer that escapes text and
attribute values, and route every path that stores an uploaded SVG
through it. The thumbnail writers (create-file-object-thumbnail and
create-file-thumbnail) and the organization logo stored the upload
untouched, so they could plant live markup in a public bucket.
The filter only knows about SVG elements, so it also accepted
documents that are not SVG at all: a page with an iframe or a meta
refresh was stored as an image. The sanitizer now rejects any document
whose root is not an svg element in the SVG namespace, and it compares
the scheme of an url the way the browser does, so a scheme split with
a tab or a line break no longer slips through the javascript: filter.
The attributes are written sorted by name, because the parser hands
them over in the reverse order of the document and the output changed
on every sanitization.
The profile picture of an identity provider is not stored either when
it is an SVG: the avatar is always rendered as an image, so an SVG
would only add the risk of keeping live markup in the storage. The
dimensions of an SVG are now read from its local name, so an SVG with
a namespace prefix can be uploaded like any other.
The logo had no size limit and the sanitizer reads the whole file, so
it now honors the media max file size.
The output shape changes with it: no line breaks, double quotes, and
the single quote of an attribute value escaped as an entity. The hash
of every stored SVG object changes with it.
The media tests use cuerdas.core instead of clojure.string, as the rest
of the codebase does.
Relates to GHSA-qwhj-rchj-58mg
AI-assisted-by: minimax-m3.1-flash
* 🐛 Drop non-SVG namespace subtrees in the SVG sanitizer
The sanitizer only checked the namespace of the root, so a child could
rebind xmlns to XHTML and smuggle live HTML (iframe/srcdoc/meta) past the
blocklist. The serializer escapes attribute values, but the browser decodes
them again, so srcdoc stays live.
Track the namespace bindings through the recursion and drop any element
that is not in the SVG namespace, with its subtree. The root is validated
up front with the same model, so a document that is not an svg document is
rejected with an honest hint instead of a misleading one.
The default namespace is stored under a namespaced key, because a prefix
literally named `default` collided with the internal key and re-opened the
same hole through a prefixed binding. The attribute filter now keeps
`xmlns`/`xmlns:*` declarations, which are not event handlers, so a prefix
named like one does not end up undeclared in the output.
The blocklist also drops HTML-only elements written in the svg namespace
(iframe, meta, embed, link, base).
Add regression tests for the child rebinding, for the prefixed variants,
for the `default` prefix, for the root local name, for the
create-file-media-object-from-url path (mocked download), and for the
HTML-only tags. Assert double-quoted attributes in the clean content test.
Relates to GHSA-qwhj-rchj-58mg
AI-assisted-by: longcat-2.5-free
* ✨ Strictly validate uploads against file bytes on disk
Sniff the real content type of every upload with simplemagic magic
bytes instead of trusting the client-supplied mtype header. Declared
types outside the allowlist keep the not-allowed code; allowed
declarations whose bytes differ are rejected as mismatch. Covers SVG
by probing its root tag over leading bytes (magic never matches XML
text), TTF/OTF by magic name and WOFF/WOFF2 by four-cc.
Read the real file size from disk instead of trusting the declared
:size. Files over the limit are rejected even when the declared size
lies downwards, and small files pass even when it lies upwards.
Run the path check first everywhere so forged server paths keep the
params-validation code. Recheck from-url downloads after the write,
with a declared-only fast fail at header time. Thread the assembled
path through font validation.
Refresh the affected tests around real and sparse fixture files,
including a PNG-as-font chunked-upload rejection case.
Closes#12796
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Accept BOM-prefixed SVG uploads and cover WOFF/WOFF2/OTF/WEBP
Address review comments on #12801:
- SVG files saved with a UTF-8 BOM were rejected as
:media-type-mismatch: read-head kept U+FEFF and the <svg probe
never ran. A leading BOM is now stripped before the probe, and a
generic XML sniff (application/xml, text/xml) is treated as
"no magic evidence" for declared image/svg+xml, so the root-tag
probe decides whether the file opens with an <svg root.
- sniff-match?/svg-file? docstrings claimed the magic database never
matches SVG; simplemagic 1.17 does match an SVG with an XML prolog
as image/svg+xml. Reworded to match reality.
- Added matching-pass tests for WOFF, WOFF2, OTF and WEBP (they hit
the read-four-cc, magic-name and direct magic branches) and three
BOM SVG regression cases.
AI-assisted-by: inkling-2-small-free
send-template drops nil values before enqueuing the mail job, but the
invite-to-organization schema required the :user-name key. Inviting an
email with no Penpot account left :user-name nil, the key was dropped
and the job failed its schema check, so the invitation email was never
sent. :user-name is now optional in invite-to-organization and in
renewal-notice, which has the same shape.
New test renders both templates through the queue with a nil
:user-name.
AI-assisted-by: claude-opus-5-5
send-template drops nil values before enqueuing the mail job, but the
invite-to-organization schema required the :user-name key. Inviting an
email with no Penpot account left :user-name nil, the key was dropped
and the job failed its schema check, so the invitation email was never
sent. :user-name is now optional in invite-to-organization and in
renewal-notice, which has the same shape.
New test renders both templates through the queue with a nil
:user-name.
AI-assisted-by: claude-opus-5-5
The client calls the duplicate-file command in two places, the dashboard and in
the plugin API. In both places the :data is dropped via `(dissoc file :data)`,
so the client never uses that field.
However the file data takes a while to send over the wire and decode. When
duplicating penpot-app there's a roughly 1s freeze whenever the response is
returned from the server with the file data. We can avoid it by dropping the
data from the response.
I also did a quick check and neither the MCP server nor any public client of
the API on GitHub seem to use the :data field.
Co-authored-by: Andrey Antukh <niwi@niwi.nz>
* 🐛 Return the full profile from the LDAP invitation branch
The invitation branch responded with only {:invitation-token token},
so the session start received a map without id or auth-backend. With
the new auth-origin vocabulary that map fails validation and the login
breaks in development and tests; in production the event would be
emitted without a channel.
Return the profile plus the token, like the regular branch, and cover
both branches with RPC-level tests.
Closes#12849
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Mint an auth token in the OIDC callback with the invitation inside
The callback used the invitation JWE as the session token, so the
frontend landed directly on token verification and never ran the
regular session start: no signin event, no teams, no host hook.
Always mint an :auth token and carry the invitation in its claims.
Token verification returns the claims verbatim, so login-from-token
attaches the invitation to the profile and logged-in does the rest,
the same pattern the verify-email token already uses.
Closes#12849
AI-assisted-by: deepseek-v4.1-flash
* ✨ Keep the session origin in the telemetry shadow rows
The anonymisation filter drops every string unless it is on a short
allowlist. Add source and activation next to lang and auth-backend so
the shadow copy keeps them; the email and the full name are still
dropped and the email domain is still derived from the email.
The filter keeps its two bindings: the transducer discards everything
that is not a UUID, a boolean or a number, so the allowlist is applied
on the filtered map, never on the raw props.
Closes#12849
AI-assisted-by: deepseek-v4.1-flash
* ✨ Describe the signup outcome with activation, not with action
The three success branches of register-profile labelled a new account
as "login", "email-verification" or "accept-invitation", so the event
contradicted its own name and the value only lived in context, where
the analytics consumers never read it.
Add activation to the event props and drop action from those branches:
the name already says it is a signup, and a constant action would only
add noise. The retry branches keep their action in context because
there it distinguishes four different outcomes.
Closes#12849
AI-assisted-by: deepseek-v4.1-flash
* ✨ Describe the session origin in the signin event
The event carried the profile but never said why the session started:
the only field that tried was frontend metadata that never leaves the
browser, and several flows filled it with the wrong value.
Add a small closed vocabulary (source, auth-backend, activation,
from-invitation) and compose it from the profile when signin is
emitted. The assertion fails in development and tests and is elided in
the release build, so a bad value cannot break a real login.
Closes#12849
AI-assisted-by: deepseek-v4.1-flash
* ✨ Emit the session origin from every login entry point
The four emitters stamped a free-form string as metadata, so the
session event could not rely on it: two flows that were signups wrote
the same value as a login.
Route them through origin-meta: password and LDAP logins declare
source login, register declares source register with activation
immediate, and login-from-token derives both from the token issuer:
:auth for a login and :verify-email for a signup that needed the
email confirmation.
Closes#12849
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Send the invitation signups through the regular session start
Registering from a team invitation link navigated by hand to token
verification, so the session start never ran: no signin event, no
teams, no host hook. The same navigation was copied in four places.
Teach login-from-register to carry an optional invitation token,
extract the profile decoration into with-register-origin, and call it
from the four sites instead of the hand-written navigation.
Closes#12849
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Resolve the invitation without waiting for the teams
Registering from an invitation link now goes through the session
start, and that start only navigates after the teams request
completes. fetch-teams has no error handling, so a network failure
there would leave the invitation unaccepted and the user on the signup
screen with a session already minted.
Move the invitation navigation out of the teams path: it emits as soon
as the profile carries a token, and the teams path emits nothing for
an invited profile so it cannot move the user away from the token
verification screen.
Closes#12849
AI-assisted-by: deepseek-v4.1-flash
* 📚 Document the session origin vocabulary and the invitation flow
Add a focused memory with the four fields, where each value comes from,
the table of flows, the LDAP exception and the invitation path through
logged-in, and link it from the frontend routing.
Update the audit-log memory: source and activation are now on the
telemetry allowlist, activation lives in props and action only exists
on the retry rows.
Closes#12849
AI-assisted-by: deepseek-v4.1-flash
* 📎 Cover the invitation session start and fix the auth memories
The invitation flows and the token decoration in login-from-token were
only covered by helper tests, so removing the guard that keeps the
teams path from navigating, or the token attach, broke the behaviour
silently.
Run the events through ptk/watch with a teams-fetched event as input:
an invited profile navigates to the token verification screen and
never to the dashboard, and the signin keeps from-invitation. Add the
login-from-token cases with and without the token, and fix the memory
statements about action, the flow table and from-invitation.
Closes#12849
AI-assisted-by: deepseek-v4.1-flash
New profiles now store renderer `wasm` in their props at creation, so
every signup path (email, OIDC, LDAP, Nitrate, demo, CLI) starts on
WebGL. A renderer passed by the caller still wins.
Existing profiles without a stored renderer keep the frontend fallback
to SVG, so no backfill or version check is needed to leave them as they
are.
The `register-profile` audit event gains `renderer-default` when the
profile is created in that call, so signups before and after this
change can be told apart.
Closes#11855
AI-assisted-by: claude-opus-5-5
* 🐛 Normalize stored SVG attribute keys to kebab-case
The v3 reader already emits kebab-case keys, but birth stored the
camelCase React prop names, so import and export never converged:
fill rules painted wrongly and filter defs lost their blur.
Birth now normalizes every stored attribute key to kebab-case
through the `svg-props` whitelist (`attrs->kebab-props`), applied on
all element paths of the svg importer, on the file builder and on
defs. svg-raw `:content` is kebab-ized in spelling only and keeps
unknown keys, because the sidebar reads them. Non-spec attrs Penpot
must keep live in `penpot-extra-attrs`, seeded with `:stroke-style`.
Render boundaries translate back to camelCase for React and for the
wasm props, and the shape schema declares the closed kebab contract
derived from that same whitelist, so birth and validation cannot
drift.
Migration `0035-fix-shape-svg-attrs-and-defs` converts old files
once with the birth transforms, and carries back the svg-viewbox
rect normalization the withdrawn `0021` had. That old id leaves
`available-migrations`, so files recording it migrate again through
this one, and the binfile cleaner drops its own svg repair.
Review fixes included here: `setup-opacity` and the blend mode read
the stored keys, gradient stops accept `:style` as a CSS string or a
map, `inherit-attributes` reads `:style` so a child's inline style
beats the inherited group attr per CSS, and the svg-attrs menu shows
spec spelling (`viewBox`, `class`) through `stored-attr-display-name`.
Tests cover the migration, the closed svg-attr schemas and the
import/export round trip; the svg-content schema test pins its
sample seed, which dealt no svg-raw shape on unlucky runs.
Closes#11947
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Keep legacy svg node tags and transforms valid in migration 0035
The closed svg node schemas require keyword tags and map attrs, but
stored files hold def and svg-raw content tags as strings, and some
node `:attrs` as nil. Migration 0035 left them as they were, so the
file data check that runs after migrating rejected those files and
they could not be opened. `normalize-node-tree` now turns string tags
into keywords and nil attrs into an empty map, at birth and in 0035.
With keyword tags, the svg defs renderer applies the shape transform
to user-space defs. Legacy `:svg-transform` values are plain maps,
which multiply into NaN matrices. `gmt/coerce` turns them into
matrices, in 0035 and in the binfile cleaner, because every v3 import
decodes the transform as a plain map again. Legacy masks and gradients
now render as designed in the svg renderer.
The binfile test helper calls `export-files!` with the current config
keys.
AI-assisted-by: claude-opus-5-5
---------
Ignore imported library relations that point outside the
package or self-link a file on the v3 import path. Delete
leftover cross-team and self relations from file_library_rel
and their matching file_library_sync rows with migration
0158, sync rows first.
Move the get-files-rels SQL to a var, without behaviour change.
Cover the write-side invariants with state-based tests:
tampered manifests, package containment, self links and the
migration cleanup.
AI-assisted-by: deepseek-v4.1-flash
AI-assisted-by: muse-spark-1.3-contributor
A component whose :main-instance-id points to a missing shape fails
validation on its copies with :ref-shape-not-found. A repair scoped to
the copy's page never sees the main's own error, so it detached the
copy, even when the main was still on the component's page.
The repair of :ref-shape-not-found now looks for the component's main
on its page first and points :main-instance-id back to it, which keeps
the copy linked. A copy is still detached when no main is found.
The notice of a rejected save always said the same thing. It did not
name the file when the user had moved to another one, gave no hint when
the page held data that was already invalid, and replaced the notice of
a backend repair still on screen, since the store holds one toast.
The backend flags a rejection whose first commit touches pages or
components that failed validation before it (:preexisting-damage). The
rejection notice names a file other than the open one, says when the
data was damaged before, and folds into a repair notice still on
screen; a later repair notice folds a rejection notice in the same way.
The notice says that the latest changes were undone, no longer that the
rest of the work is saved.
- Ending a drag inside the frame the shapes already belong to sent an
empty reparent save and bumped the revision. The move commits only
when some shape changes parent.
- When a user restored a version, other sessions reloaded the file with
no notice. They now see a dialog naming who restored which version
(by its name, else its date), saying that later changes are no longer
in the file and, when they had changes not saved yet, that those were
lost.
- Exports drawn in the browser (PDF, SVG, and PNG when the WASM renderer
draws it) and duplicating a file from the dashboard started at once,
so an export could show changes a pending save then rejected, and a
copy could miss queued edits. Both wait for the file's pending saves.
A client export warns that it includes unsaved changes; a copy warns
that it misses them.
- A save that failed on a short platform failure (a lost or refused
database connection, lack of resources, a server shutting down, a
serialization failure or deadlock, a socket or HTTP timeout, a host
that did not resolve, an interruption) answered 500, which stops
autosave. It now answers 503, which the frontend retries. Other I/O
and database errors keep answering 500, timeouts 504.
- A save answered not-found no longer replaces the workspace with the
access page. It is a terminal save error: the workspace stays open
with the failed save notice, report and reload link. When edit rights
are lost, the notice says how many changes could not be saved.
- A library sync marked the library synced three seconds later,
whatever happened to its save. A rejected sync was undone but stayed
marked: copies stayed out of date and the "updates available" notice
never came back. The mark now waits for the save, a rollback of the
sync brings the notice back, and a failed mark (e.g. the user just
became a viewer) is logged instead of leaving the workspace.
- While saves fail the workspace is read-only. The view-mode banner
says editing is disabled because changes can't be saved, with the
Inspect banner's size and bold lead and no Done button. Users can
still switch pages, pan, zoom and inspect. The lock lifts when a save
lands and brings back the mode the user had or picked meanwhile; the
WebGL context lock keeps the workspace read-only if it lasts longer.
The lock lives in app.main.data.persistence.lock.
- A viewer is never locked. A role change that leaves the user a viewer
keeps the edits that lost permission, queued and on screen, and shows
the count notice again. The role change that brings edit rights back
releases them and rolls them back on screen, as a rejected save would.
Any other terminal failure is released by the next role change or file
opened.
The debug faults `io` (socket timeout, 503) and `eof` (500) cover both
sides of the I/O rule.
The update-file tests each built the same rect change and the same
change that fails while applied. The rect change moves to the test
helpers as th/add-rect-change and the failing one gets a named helper.
The test of a database failure during the retry no longer counts calls
to process-changes: it fails every attempt that leaves the failing
commit out.
update-file checked the schema of the whole file on every save, which
loads and decodes every page even when the changes touch one. On files
with several pages that was most of the time of a save.
validate-file-schema-affected! checks the file data with the pages index
reduced to the pages the changes modify; the data outside pages is still
checked whole, and a change type it does not know makes it check the
whole file. update-file uses it for the strict and the soft schema check.
A save of 10 commits on a file of 10 pages of 2k shapes goes from ~1.1 s
to ~240 ms; a single page file costs the same.
A save whose changes failed validation was rejected as a whole: autosave
stopped, the edits made after it piled up unsent, and a reload lost
them all, the valid ones included.
The frontend now sends each commit of a save as its own group with its
commit id (:changes-with-metadata). When applying and validating all of
them at once fails, the backend searches by bisection for the first
commit that fails, keeps the commits before it and answers with
:rejected (that commit and every later one). Only the kept changes are
stored, snapshotted and broadcast, so other users never see the rejected
ones. Flat :changes and groups without ids stay all-or-nothing; database
and network errors still fail the whole request. A rejection without an
error type is reported as an internal error.
The frontend rolls the file back to what the server holds: it undoes
every pending commit of that file, applies again the remote changes that
arrived meanwhile, and redoes the kept commits. Undoing only the
rejected commits would write stale values over a peer's later change to
the same attribute. The rejected and later commits leave the queue, the
buffer and the undo history, and a warning that hides after ten seconds
tells the user. The pure bookkeeping lives in
app.main.data.persistence.commits.
The pending notice of a commit the rollback dropped no longer leaves the
save status at pending, which made "Pin this version" hang and fail.
Debug tools drive these cases by hand: debug.commitInvalidChange() makes a
save fail validation, holdSaves() delays the answers, failNextSave() and
failNextRollback() fail them on purpose (backend flag debug-save-faults),
and persistenceState() shows the queue.
* ✨ Render email templates inside the mail job
The web process rendered each email to full subject and HTML body
before submitting it, so every queued job carried a huge params
payload. The queue now stores only the template id plus the raw
template data, and the new :send-templated-mail job renders and
sends the email inside the worker.
The old :sendmail job stays in place to drain jobs submitted in the
previous format; the quota notice and the REPL test mail keep using
it. All templated callers (auth, profile, feedback, invitations,
comments, Nitrate) submit recipes through eml/send-template.
Closes#12445
AI-assisted-by: muse-spark-1.3-contributor
* ♻️ Unify templated mail on template id strings
The ::eml/template value was a namespaced keyword whose name had to
match the registry key, with one live mismatch (user-feedback vs
feedback). Callers now pass the registry id string itself, so keys
stay ::eml/-namespaced and values stay plain data, and the single
templates set rejects unsendable mail both at submit and in the
worker.
Closes#12445
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Encode templated mail data with the template schema
The job params column is plain JSON, so the URIs of an organization
(:logo, :avatar-bg-url) were stored as record maps and the template
check rejected them in the worker. Invites for organization teams,
organization invites and renewal notices failed without a retry.
Each registry entry now carries the JSON encoder and decoder of its
template schema: send-template encodes the data before submit and the
worker decodes it before rendering. The job schema keeps only the mail
envelope, as the template schemas own the typed data.
Closes#12445
AI-assisted-by: claude-opus-5-5
---------
An operator writing `PENPOT_TENANT=ACME` and one writing `acme` name the
same tenant: the value is compared as an exact string in the keys and
rows it lands in, and one uppercase letter would otherwise split one
tenant into two.
The value is normalized when the configuration is read, so nothing
downstream has to care about case, and the validated form accepts only
lowercase labels from then on.
AI-assisted-by: deepseek-v4.1-flash
* ✨ Migrate legacy file payloads to file_data and delete orphans
Copy file.data and labelled file_change.data to file_data on boot
(type main/snapshot, backend db, deleted_at preserved), then delete
file_change rows with data but no label, which no read path reaches.
Large instances pre-migrate with batched statements and fake this
step. Closes#12784.
AI-assisted-by: muse-spark-1.3-contributor
* 📚 Add 2.19.0 operations runbook for legacy file_data migration
Manual pre-migration procedure for big instances, same statements as
migration 0157 in batched transactions, following the convention of
the file-change partition runbook. Closes#12784.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Address review feedback on the legacy file data migration
Fix the issues raised in the review of the migration that moves the
legacy file payloads to file_data.
The migration deletes the unlabelled file_change rows that carry data,
but restore-files still cleared their date, and the retention test
asserted they came back. A row without a date never reaches objects-gc,
which is the leak fixed in #12419. Restore now unmarks only snapshots,
the rows that carry a label, and the retention test checks that the
legacy payload keeps its date instead of coming back.
Cover the conflict path that had no test: a pre-existing snapshot that
collides with a labelled change row, and a second run that must leave
the legacy columns null and the orphan row absent.
The runbook analyses file_data after the inserts, so the planner does
not wait for autovacuum.
Drop app.srepl.procs.fdata-storage: the copy it performed is what the
migration replaces and nothing requires it.
Closes#12784
AI-assisted-by: deepseek-v4.1-flash