mirror of
https://github.com/penpot/penpot.git
synced 2026-11-16 23:20:37 +00:00
583 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
e23ca02aa2
|
✨ Reference the file data storage object with a foreign key (#12769)
* ✨ Reference the file data storage object with a foreign key The file data pointer lived in `file_data.metadata.storage-ref-id`, a JSON key. The storage GC parsed it to find live references, and nothing stopped it from physically deleting an object a live row still pointed at: a failure in that scan lost the blob. Migration 0159 moves the pointer to a nullable `storage_object_id` column on `file_data`, backfills it from the metadata key, removes the key from the rows it backfills, indexes the column and rebuilds the foreign key to `storage_object(id)` as `NO ACTION DEFERRABLE INITIALLY IMMEDIATE`. `INITIALLY IMMEDIATE` is deliberate: the GC catches the violation inside the chunk transaction, while a deferred check would fire at commit, outside it, and the sweep would never move past the blocked chunk. The column is the only source of truth. The `storage-ref-id` entry is gone from the file metadata schema, and the single point that serializes the metadata drops it, so no write can resurrect it. `upsert!` does not query the row: the pointer of the row being replaced travels in its params, and the object is touched before the row is written. Fragments and snapshots pass nil because a modified pointer always gets a new id; the offload passes its own row; `insert-file!` drops it, since an INSERT never replaces a persisted object. `update-file!` takes the file map read from the database plus the fields to change and merges them, so the pointer comes from the read map and no caller can lose it by building a partial map. A map without `:data` raises `:missing-file-data`: it is not `assert`, which is compiled out in the deployed binary. The params cannot override the pointer or retarget the write with `:id`/`:file-id`. `bfc/file->response` strips the pointer from every response (`get-file`, `create-file`, `duplicate-file`, the view-only bundle, the team access request, the comments read, the library summary, the snapshot read and the debug download). A `file-data` object is authentication-only, so a leaked id would let any logged-in profile fetch the blob from `/assets/by-id`. `get-file-fragment` resolves through the file data backend, so a fragment on cold storage no longer hands nil to `blob/decode`, and answers only the keys of its result schema. The storage GC reference scan, the restore union and `objects-gc` read the column. The offload reads `SELECT fd.*`, so its rows go through a `decode-row` that decodes jsonb metadata and drops a JSON null, which the row schema rejected; a row without a readable object raises `:missing-storage-object` with the file and row ids. A chunk blocked by the foreign key (SQLState 23503) is a real anomaly, since the reference scan should have frozen the object: it stays an error with its own hint and the chunk is retried on the next run until the reference is fixed. The release is a coordinated restart, not a rolling deploy, and not rollback-safe for the rows it writes: the previous code writes and scans the metadata pointer, so a key it writes misses the backfill and removing it loses the reference. `operations/2.20.0-file-data-storage-object-id.md` builds the column, the index and the foreign key online and runs the backfill with every instance stopped, then marks the step as applied; a fresh or small instance does the whole job in the migration. The runbook keeps the validation scan out of the stopped window and adds a lock timeout and retry to the online statements. Tests cover the foreign key shape, the backfill and its corrupt and orphan guards, the rebuild of a missing index and of a wrong, non-deferrable or deferred constraint, the deletion refused by a live reference, the metadata cleanup, the offload row normalizations, the touched objects of the unshare and the restore, the response stripping on every call site and the chunk blocked by the foreign key. The backend memories and the developer guide document the pointer, the GC behavior and the two migration paths. Closes #12768 AI-assisted-by: deepseek-v4.1-flash AI-assisted-by: inkling-2-small-free * 🐛 Hold the file row lock across the migration write `get-file` applies the pending migrations in memory and, when it finds any, persists the result. That write read the row without a lock and `update-file!` is an unconditional UPDATE, so a save that committed in between was overwritten: the saved content, `revn` and `modified-at` went back, the next save from that client failed with `:revn-conflict` and the object the save left became an orphan the storage GC cannot revisit once it is frozen. The read now runs in a transaction of its own (`get-file` calls `get-file*` inside `db/tx-run!` and accepts a cfg map only), so the `file` row lock covers the re-check and the write. `get-file*` reads the row once (`sql:get-file` plus `resolve-applied-migrations`) and completes it with `attach-file-data`; only when a migration is pending it reads the file again under `FOR UPDATE`, re-checks `need-migration?` on that fresh map and persists it (`migrate-file`). The common case stays at one row read and one migrations resolution, the migration case keeps a single decoded copy in the heap, and a row that changed in between is not overwritten. A migration found on the unlocked read is applied in memory only, for the next read to persist. `migrate-file` returns the reattached map when it persists, with the `:storage-object-id` of the row the write left, and the realized map when it does not; `:realize?` is what returns the read fully plain. `fdata/upsert!` answers `{:written ... :storage-object-id ...}` instead of a boolean, `update-file!` propagates it and the migrating read puts it back in the map it returns, so a caller that persists again touches the current object instead of the replaced one. `insert-file!` keeps nil: an INSERT never replaces a persisted object. The write also compares the stored pointer with the caller's value (`IS NOT DISTINCT FROM`) and raises `:stale-file-data-pointer` when it affects no row. Every writer holds the `file` row lock, so a pointer that no longer matches is a protocol violation, not a race to retry. The expected pointer is a separate argument because the storage branch replaces `:storage-object-id` before the upsert, and the check lives in `upsert-in-database`: the `legacy-db` branches do not use that SQL and their count can legitimately be zero. The save, snapshot and restore-backup reads take `FOR UPDATE` instead of `FOR SHARE`: they inherit the lock, and the migration branch asks for the exclusive one in the same transaction, so a shared lock that upgrades deadlocks two concurrent readers. A data read rejects `:lock-for-share?` for the same reason; only row-only reads accept it. `sql:get-file-without-data` is a plain select now, not a CTE, which makes locking legal without loading the data. Realization has a single owner: `attach-file-data` only resolves and decodes, and `get-file*` realizes once per path. The applied migrations no longer travel in the `get-file-with-summary` response, cache hit and cache miss alike. `embed-assets` reads each library without `:realize?`, so its `:components` is an unloaded pointer map; `absorb-assets` walks it with `vals`, `PointerMap.seq` calls `load!` and that raised with no loader bound, so every export with `:merge-libraries` of a library with components failed. The absorb runs under the library's own loader now. The in-place binfile import (`import-file-and-overwrite*`, `::overwrite`, `::bfc/file-id`) had no product caller and cannot carry the previous `storage_object_id`, so overwriting a storage-backed row would leave its object unreclaimed. `import-files!` rejects a cfg with `::bfc/file-id` (`:not-supported`) and the helpers only that path used go with it (`invalidate-thumbnails`, `*reference-file*`, the `::overwrite` id branch of `index-object` and of `save-file!`, the `on-conflict-do-nothing?` flag of the media insert). Tests: a new `file-migration-read-test` pins the single read, the locked re-read, the concurrent save interleaving, the double migration and the read-only path; the upsert tests cover the return shape and the stale pointer; the SQL recorder they share lives in `backend-tests.helpers`, and the save, snapshot, restore and summary sequences follow the new reads. The backend memories document the read contract and the pointer. AI-assisted-by: deepseek-v4.1-flash * 🐛 Harden the get-file guard and skip its read-only transaction The cfg guard of `get-file` was an `assert`, which `app.config` elides when `:backend-asserts` is off, so a deployed binary accepted a bare pool or connection while the docstring promised it raises. It throws `IllegalArgumentException` now, the class the option guards of the same namespace already use, and the existing test expects it instead of `AssertionError`. A read with `:read-only?` applies the pending migrations in memory and writes nothing, so the transaction the migration write needs only costs two round trips there. `get-file` runs it with `db/run!` instead: with a cfg that carries a connection it still joins the caller's transaction, and only the reads that would have opened their own skip it. A lock option needs a read-write transaction from the caller, because PostgreSQL rejects row locks in a read-only one; the docstring states that. Tests: the guard test expects the new class; a cfg marked `::db/read-only` reads a pending migration without persisting it (the config flag that fills that key is legacy, but the key is set by hand); and a read-only read is pinned as not opening a transaction while the same call without the option does. The backend memory documents the split and the guard. AI-assisted-by: deepseek-v4.1-flash |
||
|
|
679e70e48c |
🐛 Keep file_change kind nullable so rollback keeps writing
Migration 0158 created file_change with kind NOT NULL, which blocks any emergency rollback to the previous version: the old code inserts without kind and every save would fail. kind is still a reserve (reads use label IS NOT NULL), so it stays nullable until the rollback window closes; the NOT NULL comes in a later migration. The runbook keeps parity, the schema and migration tests cover the nullable column and the rollback insert, and the storage memory records the decision. AI-assisted-by: muse-spark-1.3-contributor |
||
|
|
92c478b0e2 | Merge remote-tracking branch 'origin/merge-staging' into staging | ||
|
|
dc38eaa843
|
🐛 Fix font upload crash and preserve font metadata (#11951)
* 🐛 Fix font upload crash and preserve variant metadata opentype.js 2.0.0 returns no fontSubfamily name for fonts whose Macintosh name table only holds custom records. That made parse-font-weight/parse-font-style call re-seq on nil and crash the upload. A nil or blank variant is now treated as no match in app.common.media, defaulting to weight 400 and style normal. The variant parsing also accepts oblique as italic, and the new parse-font-family helper derives the family from a filename by stripping the known weight/style tokens, separating a style token glued to a weight token ("Roboto-BoldItalic" -> "Roboto"). first-nonblank-string returns the first nonblank candidate, so a blank preferredFamily/preferredSubfamily no longer shadows a valid fontFamily/fontSubfamily. app.main.data.fonts gains resolve-font-metadata and prepare-font-entry. opentype.js values win per field; any missing value falls back to the filename-derived family, weight and style. The parsed and woff2 branches share the same helper, so metadata is preserved instead of collapsing every face to weight 400, and fonts with missing vertical-metric tables upload with no height warning instead of failing. process-upload reports per-file failures (unknown signature, unparseable file, preparation error) with a single bad-font notification instead of silently dropping them, and never errors for per-file failures. The upload now sends :variant-name to create-font-variant, which accepts it as an optional param (ships in 2.20), so the foundry name survives the upload. Closes #10436 AI-assisted-by: deepseek-v4.1-flash AI-assisted-by: muse-spark-1.3-contributor Signed-off-by: CHOI JUNSOO <junsoo1172@gmail.com> * 🐛 Parse glued font style tokens in the font parsers EvaMarco's review on #11951: parse-font-family and parse-font-weight handled glued tokens like "BoldItalic", but parse-font-style did not. For names such as "opensans-bolditalic-webfont.woff2" the three parsers disagreed: the italic face resolved to style normal and collided with the bold-only file on the (family, weight, style) join key. One binary silently overwrote the other during upload, losing one font face without any warning. A new private normalize-font-name helper is applied by all three parsers. It splits a style token glued to a weight token, now with a lookahead boundary so "LightItalic_v2" splits too, and strips a trailing technical suffix (-VF or _v<number>). parse-font-family, parse-font-weight and parse-font-style now interpret the name consistently. Tests cover the three cases from the review, the bold-only control case, and an end-to-end process-upload check that a bold and a bold-italic WOFF2 survive as two variants. Relates to #10436 and PR #11951. AI-assisted-by: inkling-2-small-free * Remove duplicate frontend test registration --------- Signed-off-by: CHOI JUNSOO <junsoo1172@gmail.com> |
||
|
|
a35dd49b6a | Merge branch 'staging-2.19.x' into staging | ||
|
|
12d73d0f63 |
🐛 Fix flaky assertion in climit waiting-count test
The test expected exactly one publication of the :rpc-climit-waiting gauge inside the with-redefs window. The republisher daemon thread, started at component init with no delay and rescheduling every 15 s, publishes the same counter and can land inside that window, so the recorded sequence could be [0 0] and the assertion failed with actual [0 +0] (Backend Tests job in #12979). Assert on the last published value instead of the whole sequence, the contract waiting-last documents and the sibling tests use. AI-assisted-by: inkling-2-small-free |
||
|
|
bcac14a4a8
|
🐛 Skip circular library links on binfile import (#12062)
* 🐛 Skip circular library links on binfile import File import recreated circular library references that the link-file-to-library command already rejects. The v3 importer now skips manifest relations that would close a cycle, or link a file to itself, and logs each skipped link. Closes #12052. AI-assisted-by: muse-spark-1.3-contributor-free 🐛 Address review findings on circular import guard F1 extracts the cycle check into bfc/library-link-status and reuses it in link-file-to-library and v3 import, keeping UI errors intact. F3 orders get-files-rels and manifest relations so the first edge wins deterministically, and documents it. F2 adds a three-node transitive cycle test; F4 asserts file_library_sync has no orphan rows for skipped links and matches rels otherwise. AI-assisted-by: muse-spark-1.3-contributor 🐛 Pass team-id in binfile v3 circular library tests The four circular library relation tests (import-skips-circular, import-skips-transitive-cycle, import-skips-self-referencing, import-keeps-acyclic) called v3/import-files! without ::bfc/team-id. resolve-and-link-libraries asserts (uuid? team-id), so with enable-backend-asserts the tests errored instead of running. Add (assoc ::bfc/team-id (:default-team-id profile)) to each test, mirroring the production RPC import path and the earlier fix in a36193e54b for the other binfile v3 import tests. Verified: 1338 tests, 6910 assertions, 0 failures. AI-assisted-by: mistral-large-4 * 🐛 Guard the v1 import and restore the RPC check order The version 1 importer recreated the circular library references that the version 3 importer already skips, and a user reaches it through `import-binfile` and the import job. It now asks `bfc/library-link-status` before inserting each relation, so a self link or an edge that would close a cycle is skipped and logged, and it processes the relations in sorted order like v3 does. Closes #12052. The version 2 importer stays out on purpose: only the developer REPL reaches it, and a note in the code records the gap. `link-file-to-library` keeps the shared helper but restores the order of the base commit: the self link guard, then the edition permission and team ownership checks, and only then the cycle check. A caller without permission gets `not-found` again instead of the circular error, and a rejected request no longer runs the recursive library query first. Tests: a v1 import that recreates a cycle, the self link error of the command, and the permission check running before the cycle check. AI-assisted-by: deepseek-v4.1-flash * 🐛 Tolerate repeated file entries in the v1 binfile import The v1 exporter unioned the requested files with their libraries without deduplicating, so a package exported from a file inside a library cycle (see #12052) lists that file twice in the metadata. The v1 import created one file per entry, so the second insert hit the primary key of `file` and aborted the whole import with "file already exists". The importer now reads the pair of objects of every entry to keep the stream in sync but imports the first occurrence only, warns about the repeats, and answers with one id per imported file. The v1 export keeps its behavior: it has no caller in the product, and only the tests and the developer REPL use it. Tests: a package with a repeated entry, built with the v1 export from a cyclic library pair, imports the file once and leaves no duplicated rows. AI-assisted-by: deepseek-v4.1-flash |
||
|
|
34c702bc80
|
✨ Add table partitioning for file-change (#12417)
* ✨ Partition file_change by file_id Partition `file_change` by `HASH (file_id)` in 16 partitions, with the same bounds as `file_data`, so a join between them is resolved partition by partition. The primary key becomes `(file_id, id)`, which a partitioned table needs, and every access filters by `file_id`. The new table carries a `kind` column (1 snapshot, 2 revision) as a reserve: the queries still tell them apart by the label, and `kind` takes over when they are rewritten. The payload of a snapshot stays in `file_data`; the `data` column is kept for reading old `legacy-db` payloads. The four snapshot commands require `file-id`, because it prunes the partition. The objects GC locks and deletes by the pair, in batches. The old table is kept as `file_change_legacy` as the way back. The automatic migration refuses to run above 10 GiB or 2M rows and points to the runbook; big instances partition online with that runbook, which copies by id, follows the `updated_at` watermark in delta passes and checks that no row of the source is missing from the target. Closes #12416 AI-assisted-by: deepseek-v4.1-flash * 🐛 Address the review of the file_change partition The production runbook promised a rollback it did not have: the section was a sketch, and its statements could write to the table about to be dropped. The rollback is removed, and so is every comment that described it. `file_change_legacy` stays as a frozen copy of the previous layout. The cutover check now aborts the transaction when a row of the old table is missing in the new one, instead of leaving the decision to the operator. The five secondary indexes of the new table are built after the last delta pass: the bulk and the delta passes only need the primary key, so the batches no longer maintain them. The copy of 0158 is now replayed in a test against the pre-0158 shape with a labelled and an unlabelled row, plus an empty instance that must leave no legacy table behind. The updated_at assertion of the snapshot mutations is bounded by the test clock, and the repair revision stops writing the `features` of the file, like the normal save. AI-assisted-by: deepseek-v4.1-flash * 📎 Fix linter issues * 🐛 Fix stale helper call in rpc_file_snapshot_test The suite does not load: `th/run-pending-tasks!` does not exist in backend-tests.helpers. The pending jobs runner is `run-pending-jobs` (no arguments), the name every other call site of the suite uses. AI-assisted-by: inkling-2-small-free * 📚 Align the partition change with the 2.20.0 milestone 2.19.0 is already shipped; this work targets 2.20.0. The operations file of the partition procedure carries the 2.20.0 prefix, the references that point at it (the migration, the runbook table and the memory) follow, and the RPC doc/changes tags of the snapshot commands record 2.20 as the release that makes `file-id` required. AI-assisted-by: inkling-2-small-free * 🐛 Fix review findings on the partition rebase The 0157 legacy test was missed by the rebase and inserted file_change rows without the new mandatory `kind` column, so the backend suite fails against the migrated schema. The inserts now carry `:kind` (snapshot-kind for the labelled rows, revision-kind for the rest). The post-cutover verification block of the runbook terminates the first query with a semicolon so the block pastes into psql without a syntax error. The guard message of the migration names the step (0158-partition-file-change) instead of "0158", ambiguous since two migrations share that number. Closes #12416 AI-assisted-by: inkling-2-small-free * 🐛 Fix the stale label in the snapshot restore test The rename block of snapshots-crud runs before the restore block, so the restored version is already named "label1-updated" when the restore publishes its message. The command names the version by the label the row has at that moment, not by the one it had when it was created, so the test was asserting a stale value. Assert the current label, and say why in the comment. AI-assisted-by: deepseek-v4.1-flash * ♻️ Split the version event tests into one test per command The workspace version events and the plugin proxy send the file-id of the open file on every snapshot command, because the command needs it to prune the partition of file_change. Both tests checked that with a doseq and t/testing blocks, so a failure named the whole group and not the command that broke. Give each command its own test. The workspace ones share a helper that runs the event and returns the command and params it sends; the plugin ones keep their own mock of rp/cmd!. AI-assisted-by: deepseek-v4.1-flash |
||
|
|
e4fcc663c0 | Merge remote-tracking branch 'origin/staging-2.19.x' into staging | ||
|
|
1db1487c7f
|
🐛 Require thread access when marking threads as read (#12894)
* 🐛 Require thread access when marking threads as read mark-all-threads-as-read took any list of thread ids and wrote the read status of the caller on each of them, without checking that the caller can access the file that owns the thread. It was the only comment mutation that did not check permissions, and it took the ids as an unbounded array, so a single request could keep a database connection and a transaction open for a long time. The rpc now resolves the threads in one query and checks the comment permissions of every distinct file, answering not-found when a thread does not exist or is not accessible. Both cases answer the same error on purpose, so the response cannot be used to tell existing threads apart. The status rows are now written with a single insert, keeping the modified-at date one second in the future that makes the thread count as read. Repeated ids are deduped, the command returns nil again like the sibling commands with no result, and the schema limits the number of threads per call to 500 while the dashboard sends the ids in batches of that size. Tests cover access rejection, unknown threads, mixed batches that write nothing, repeated ids, empty lists, marking twice, and the frontend batch split. Relates to GHSA-j4vc-cmf6-jhr7 AI-assisted-by: minimax-m3.1-flash * 🐛 Refresh the unread list when a batch fails The dashboard sends the thread ids in batches of 500 and each batch commits on the server. A failure in a later batch left the unread list stale: the re-read only ran when every batch succeeded, so the threads already marked as read stayed visible as unread. The re-read now runs in both paths. On failure the event is emitted before the error is propagated, so the error toast still shows and the success toast stays limited to a full success. The backend also answers an empty list without opening a transaction, since there is nothing to check and nothing to write. Relates to GHSA-j4vc-cmf6-jhr7 AI-assisted-by: deepseek-v4.1-flash |
||
|
|
8510daef52
|
🐛 Restrict the team invitation token and list to admins (#12892)
* 🐛 Restrict the team invitation token and list to admins The token of a team invitation is a bearer credential: whoever holds it accepts the invitation with the role of the invitation row. The RPC that mints it only required edition permissions, while the rest of the invitation commands require admin, so a team editor could get the token of any pending invitation, including the ones that grant the owner role. The token now requires admin, and the token of an owner invitation is restricted to the team owner: an admin cannot create nor promote an owner invitation, so it cannot keep its token either. It reuses the error code already used for that rule. Minting the token now leaves an audit event named as an action, carrying the granted role and the id of the invitation, so the extraction does not go unnoticed. The list of pending invitations is admin only as well: it carries the email of the invited person and the role offered, and the dashboard was painting those emails to any member. The frontend catches the permission error and shows the permission notice instead, and the copy action is not offered for an owner invitation, which the backend rejects for an admin. The five invitation commands now answer with the same permission error, so they speak one language. Relates to GHSA-4mmc-j8rh-74pf AI-assisted-by: minimax-m3.1-flash * 🐛 Clear the cached invitations and gate the list on team admin The pending invitations are a team admin only list. The dashboard still offered the entry to every member and keyed the section on the invite policy of the organization, so a team admin blocked by that policy saw the no-permission notice although the backend lets them list. - `fetch-invitations` answers a denied refetch with an `invitations-fetched` event carrying an empty list, so a demoted admin does not keep the cached emails and the header no longer paints the invite button from a stale list. - The header tabs and the team options dropdown hide the invitations entry without `:is-admin`, which already includes the team owner. - `invitations-section-content` keys the section on `:is-admin`, and the invite policy only gates the invite actions: resending keeps the `can-invite?` check, deleting and row selection only need team admin. AI-assisted-by: deepseek-v4.1-flash |
||
|
|
0420db1ca9
|
🐛 Apply the share-link page scope to the comment queries and writes (#12888)
* 🐛 Apply the share-link page scope to the comment queries and writes A share link can be restricted to a subset of the pages of a file, and the comment queries enforced that scope. But the RPC that resolves the profiles of the commenters authorized the request with check-comment-permissions! and dropped the returned perms, so its query filtered only by file id. The holder of a page scoped link got the id, email, fullname and photo of every profile that commented on any page of the file, including the pages the owner left out of the link. The query now takes the pages of the share link and only selects the authors of the threads on them. The profile of the caller is still added, so the viewer keeps its own avatar. The scope was missing as well on the comment writes that take a page id or a thread id: create-comment-thread accepted a page outside the link, create-comment accepted a thread of an unshared page, and update-comment-thread, update-comment-thread-position and update-comment-thread-frame accepted a thread of an unshared page. They now raise not-found, the same way the comment queries already did. Replying to a thread of an unshared page also sent the emails to the thread participants. The page scope rule now lives in one place: share-link-pages returns the pages granted by a share link and check-page-scope raises not-found when a page is outside of it, so get-comment-threads reuses the helper instead of checking the pages inline. The docstring of get-profiles-for-file-comments said that it returns the participants of all the comment threads of the file, which is not true anymore for a share link. The docs are public, so the text and the changes list are updated. The tests of the share link page scope are hardened: the setup asserts that the comment threads are created, the membership case has a second commenter so it can no longer pass for the wrong reason, and the link without pages, the allowed paths of the thread updates and the who-comment=team denial are covered. Relates to GHSA-89cq-3wv7-987r AI-assisted-by: minimax-m3.1-flash * 🐛 Enforce the share-link page scope on the remaining comment writes Check the page scope of the share link before mutating in the four comment writes that still skipped it: update-comment-thread-status, update-comment, delete-comment-thread and delete-comment. The check runs before the ownership checks, so an out-of-scope write fails with not-found and the response does not reveal that the comment or thread exists. Load the file in create-comment only after the permissions and page scope checks, matching create-comment-thread, so a rejected reply does not load the file data. Cover the denial and the in-scope paths of the four commands, and the fail-closed case of a link without pages on the comment reads. Move the outsider thread creation after the profiles assertion so the caller-union assertion fails when the union is removed. Document the page scope invariant in the backend permissions memory. AI-assisted-by: deepseek-v4.1-flash |
||
|
|
4874da450f
|
🐛 Escape SVG values and sanitize every SVG upload (#12887)
* 🐛 Escape SVG values and sanitize every SVG upload The SVG sanitizer removed dangerous elements and attributes by name and then wrote the tree back with clojure.xml/emit, which escapes nothing. A payload hidden in an attribute value or in a text node became live markup again: an id of "x' onload='alert(1)'" was stored as a real onload handler, and an entity encoded script tag was stored as a real script element. Write the filtered tree with a serializer that escapes text and attribute values, and route every path that stores an uploaded SVG through it. The thumbnail writers (create-file-object-thumbnail and create-file-thumbnail) and the organization logo stored the upload untouched, so they could plant live markup in a public bucket. The filter only knows about SVG elements, so it also accepted documents that are not SVG at all: a page with an iframe or a meta refresh was stored as an image. The sanitizer now rejects any document whose root is not an svg element in the SVG namespace, and it compares the scheme of an url the way the browser does, so a scheme split with a tab or a line break no longer slips through the javascript: filter. The attributes are written sorted by name, because the parser hands them over in the reverse order of the document and the output changed on every sanitization. The profile picture of an identity provider is not stored either when it is an SVG: the avatar is always rendered as an image, so an SVG would only add the risk of keeping live markup in the storage. The dimensions of an SVG are now read from its local name, so an SVG with a namespace prefix can be uploaded like any other. The logo had no size limit and the sanitizer reads the whole file, so it now honors the media max file size. The output shape changes with it: no line breaks, double quotes, and the single quote of an attribute value escaped as an entity. The hash of every stored SVG object changes with it. The media tests use cuerdas.core instead of clojure.string, as the rest of the codebase does. Relates to GHSA-qwhj-rchj-58mg AI-assisted-by: minimax-m3.1-flash * 🐛 Drop non-SVG namespace subtrees in the SVG sanitizer The sanitizer only checked the namespace of the root, so a child could rebind xmlns to XHTML and smuggle live HTML (iframe/srcdoc/meta) past the blocklist. The serializer escapes attribute values, but the browser decodes them again, so srcdoc stays live. Track the namespace bindings through the recursion and drop any element that is not in the SVG namespace, with its subtree. The root is validated up front with the same model, so a document that is not an svg document is rejected with an honest hint instead of a misleading one. The default namespace is stored under a namespaced key, because a prefix literally named `default` collided with the internal key and re-opened the same hole through a prefixed binding. The attribute filter now keeps `xmlns`/`xmlns:*` declarations, which are not event handlers, so a prefix named like one does not end up undeclared in the output. The blocklist also drops HTML-only elements written in the svg namespace (iframe, meta, embed, link, base). Add regression tests for the child rebinding, for the prefixed variants, for the `default` prefix, for the root local name, for the create-file-media-object-from-url path (mocked download), and for the HTML-only tags. Assert double-quoted attributes in the clean content test. Relates to GHSA-qwhj-rchj-58mg AI-assisted-by: longcat-2.5-free |
||
|
|
e1f924cf00 |
🐛 Send organization invitations to unregistered emails (#12937)
send-template drops nil values before enqueuing the mail job, but the invite-to-organization schema required the :user-name key. Inviting an email with no Penpot account left :user-name nil, the key was dropped and the job failed its schema check, so the invitation email was never sent. :user-name is now optional in invite-to-organization and in renewal-notice, which has the same shape. New test renders both templates through the queue with a nil :user-name. AI-assisted-by: claude-opus-5-5 |
||
|
|
4e8f05171e
|
🐛 Forward attribute deletion events to admin console (#12932) | ||
|
|
1df2941576 | Merge remote-tracking branch 'origin/staging' into develop | ||
|
|
0ea6d166f0
|
✨ Add a session origin model to auth events (#12856)
* 🐛 Return the full profile from the LDAP invitation branch The invitation branch responded with only {:invitation-token token}, so the session start received a map without id or auth-backend. With the new auth-origin vocabulary that map fails validation and the login breaks in development and tests; in production the event would be emitted without a channel. Return the profile plus the token, like the regular branch, and cover both branches with RPC-level tests. Closes #12849 AI-assisted-by: deepseek-v4.1-flash * 🐛 Mint an auth token in the OIDC callback with the invitation inside The callback used the invitation JWE as the session token, so the frontend landed directly on token verification and never ran the regular session start: no signin event, no teams, no host hook. Always mint an :auth token and carry the invitation in its claims. Token verification returns the claims verbatim, so login-from-token attaches the invitation to the profile and logged-in does the rest, the same pattern the verify-email token already uses. Closes #12849 AI-assisted-by: deepseek-v4.1-flash * ✨ Keep the session origin in the telemetry shadow rows The anonymisation filter drops every string unless it is on a short allowlist. Add source and activation next to lang and auth-backend so the shadow copy keeps them; the email and the full name are still dropped and the email domain is still derived from the email. The filter keeps its two bindings: the transducer discards everything that is not a UUID, a boolean or a number, so the allowlist is applied on the filtered map, never on the raw props. Closes #12849 AI-assisted-by: deepseek-v4.1-flash * ✨ Describe the signup outcome with activation, not with action The three success branches of register-profile labelled a new account as "login", "email-verification" or "accept-invitation", so the event contradicted its own name and the value only lived in context, where the analytics consumers never read it. Add activation to the event props and drop action from those branches: the name already says it is a signup, and a constant action would only add noise. The retry branches keep their action in context because there it distinguishes four different outcomes. Closes #12849 AI-assisted-by: deepseek-v4.1-flash * ✨ Describe the session origin in the signin event The event carried the profile but never said why the session started: the only field that tried was frontend metadata that never leaves the browser, and several flows filled it with the wrong value. Add a small closed vocabulary (source, auth-backend, activation, from-invitation) and compose it from the profile when signin is emitted. The assertion fails in development and tests and is elided in the release build, so a bad value cannot break a real login. Closes #12849 AI-assisted-by: deepseek-v4.1-flash * ✨ Emit the session origin from every login entry point The four emitters stamped a free-form string as metadata, so the session event could not rely on it: two flows that were signups wrote the same value as a login. Route them through origin-meta: password and LDAP logins declare source login, register declares source register with activation immediate, and login-from-token derives both from the token issuer: :auth for a login and :verify-email for a signup that needed the email confirmation. Closes #12849 AI-assisted-by: deepseek-v4.1-flash * 🐛 Send the invitation signups through the regular session start Registering from a team invitation link navigated by hand to token verification, so the session start never ran: no signin event, no teams, no host hook. The same navigation was copied in four places. Teach login-from-register to carry an optional invitation token, extract the profile decoration into with-register-origin, and call it from the four sites instead of the hand-written navigation. Closes #12849 AI-assisted-by: deepseek-v4.1-flash * 🐛 Resolve the invitation without waiting for the teams Registering from an invitation link now goes through the session start, and that start only navigates after the teams request completes. fetch-teams has no error handling, so a network failure there would leave the invitation unaccepted and the user on the signup screen with a session already minted. Move the invitation navigation out of the teams path: it emits as soon as the profile carries a token, and the teams path emits nothing for an invited profile so it cannot move the user away from the token verification screen. Closes #12849 AI-assisted-by: deepseek-v4.1-flash * 📚 Document the session origin vocabulary and the invitation flow Add a focused memory with the four fields, where each value comes from, the table of flows, the LDAP exception and the invitation path through logged-in, and link it from the frontend routing. Update the audit-log memory: source and activation are now on the telemetry allowlist, activation lives in props and action only exists on the retry rows. Closes #12849 AI-assisted-by: deepseek-v4.1-flash * 📎 Cover the invitation session start and fix the auth memories The invitation flows and the token decoration in login-from-token were only covered by helper tests, so removing the guard that keeps the teams path from navigating, or the token attach, broke the behaviour silently. Run the events through ptk/watch with a teams-fetched event as input: an invited profile navigates to the token verification screen and never to the dashboard, and the signin keeps from-invitation. Add the login-from-token cases with and without the token, and fix the memory statements about action, the flow table and from-invitation. Closes #12849 AI-assisted-by: deepseek-v4.1-flash |
||
|
|
988cde3c9d
|
✨ Make WebGL the default renderer for new accounts (#12880)
New profiles now store renderer `wasm` in their props at creation, so every signup path (email, OIDC, LDAP, Nitrate, demo, CLI) starts on WebGL. A renderer passed by the caller still wins. Existing profiles without a stored renderer keep the frontend fallback to SVG, so no backfill or version check is needed to leave them as they are. The `register-profile` audit event gains `renderer-default` when the profile is created in that call, so signups before and after this change can be told apart. Closes #11855 AI-assisted-by: claude-opus-5-5 |
||
|
|
a77cb2608e
|
🐛 Normalize stored SVG attribute keys to kebab-case (#12006)
* 🐛 Normalize stored SVG attribute keys to kebab-case The v3 reader already emits kebab-case keys, but birth stored the camelCase React prop names, so import and export never converged: fill rules painted wrongly and filter defs lost their blur. Birth now normalizes every stored attribute key to kebab-case through the `svg-props` whitelist (`attrs->kebab-props`), applied on all element paths of the svg importer, on the file builder and on defs. svg-raw `:content` is kebab-ized in spelling only and keeps unknown keys, because the sidebar reads them. Non-spec attrs Penpot must keep live in `penpot-extra-attrs`, seeded with `:stroke-style`. Render boundaries translate back to camelCase for React and for the wasm props, and the shape schema declares the closed kebab contract derived from that same whitelist, so birth and validation cannot drift. Migration `0035-fix-shape-svg-attrs-and-defs` converts old files once with the birth transforms, and carries back the svg-viewbox rect normalization the withdrawn `0021` had. That old id leaves `available-migrations`, so files recording it migrate again through this one, and the binfile cleaner drops its own svg repair. Review fixes included here: `setup-opacity` and the blend mode read the stored keys, gradient stops accept `:style` as a CSS string or a map, `inherit-attributes` reads `:style` so a child's inline style beats the inherited group attr per CSS, and the svg-attrs menu shows spec spelling (`viewBox`, `class`) through `stored-attr-display-name`. Tests cover the migration, the closed svg-attr schemas and the import/export round trip; the svg-content schema test pins its sample seed, which dealt no svg-raw shape on unlucky runs. Closes #11947 AI-assisted-by: muse-spark-1.3-contributor * 🐛 Keep legacy svg node tags and transforms valid in migration 0035 The closed svg node schemas require keyword tags and map attrs, but stored files hold def and svg-raw content tags as strings, and some node `:attrs` as nil. Migration 0035 left them as they were, so the file data check that runs after migrating rejected those files and they could not be opened. `normalize-node-tree` now turns string tags into keywords and nil attrs into an empty map, at birth and in 0035. With keyword tags, the svg defs renderer applies the shape transform to user-space defs. Legacy `:svg-transform` values are plain maps, which multiply into NaN matrices. `gmt/coerce` turns them into matrices, in 0035 and in the binfile cleaner, because every v3 import decodes the transform as a plain map again. Legacy masks and gradients now render as designed in the svg renderer. The binfile test helper calls `export-files!` with the current config keys. AI-assisted-by: claude-opus-5-5 --------- |
||
|
|
2df4d10bd5
|
✨ Enforce attribute-based organization permissions (#12868)
* ✨ Enforce attribute-based organization permissions * 📎 Code review |
||
|
|
6a1b58c310 | Merge remote-tracking branch 'origin/staging' into develop | ||
|
|
0b8a428d6b
|
🐛 Validate binfile library relations against the package (#12862)
Ignore imported library relations that point outside the package or self-link a file on the v3 import path. Delete leftover cross-team and self relations from file_library_rel and their matching file_library_sync rows with migration 0158, sync rows first. Move the get-files-rels SQL to a var, without behaviour change. Cover the write-side invariants with state-based tests: tampered manifests, package containment, self links and the migration cleanup. AI-assisted-by: deepseek-v4.1-flash AI-assisted-by: muse-spark-1.3-contributor |
||
|
|
eadd1cce70 |
🐛 Keep a copy linked when the repair finds its component's main
A component whose :main-instance-id points to a missing shape fails validation on its copies with :ref-shape-not-found. A repair scoped to the copy's page never sees the main's own error, so it detached the copy, even when the main was still on the component's page. The repair of :ref-shape-not-found now looks for the component's main on its page first and points :main-instance-id back to it, which keeps the copy linked. A copy is still detached when no main is found. |
||
|
|
61bf66342b |
✨ Tell repairs, damaged data and other files apart in save notices
The notice of a rejected save always said the same thing. It did not name the file when the user had moved to another one, gave no hint when the page held data that was already invalid, and replaced the notice of a backend repair still on screen, since the store holds one toast. The backend flags a rejection whose first commit touches pages or components that failed validation before it (:preexisting-damage). The rejection notice names a file other than the open one, says when the data was damaged before, and folds into a repair notice still on screen; a later repair notice folds a rejection notice in the same way. The notice says that the latest changes were undone, no longer that the rest of the work is saved. |
||
|
|
6426b17c9c |
🐛 Fix small save bugs found by the rollback test plan
- Ending a drag inside the frame the shapes already belong to sent an empty reparent save and bumped the revision. The move commits only when some shape changes parent. - When a user restored a version, other sessions reloaded the file with no notice. They now see a dialog naming who restored which version (by its name, else its date), saying that later changes are no longer in the file and, when they had changes not saved yet, that those were lost. - Exports drawn in the browser (PDF, SVG, and PNG when the WASM renderer draws it) and duplicating a file from the dashboard started at once, so an export could show changes a pending save then rejected, and a copy could miss queued edits. Both wait for the file's pending saves. A client export warns that it includes unsaved changes; a copy warns that it misses them. |
||
|
|
cb26e8a39b |
✨ Keep autosave and the workspace alive when saves fail
- A save that failed on a short platform failure (a lost or refused database connection, lack of resources, a server shutting down, a serialization failure or deadlock, a socket or HTTP timeout, a host that did not resolve, an interruption) answered 500, which stops autosave. It now answers 503, which the frontend retries. Other I/O and database errors keep answering 500, timeouts 504. - A save answered not-found no longer replaces the workspace with the access page. It is a terminal save error: the workspace stays open with the failed save notice, report and reload link. When edit rights are lost, the notice says how many changes could not be saved. - A library sync marked the library synced three seconds later, whatever happened to its save. A rejected sync was undone but stayed marked: copies stayed out of date and the "updates available" notice never came back. The mark now waits for the save, a rollback of the sync brings the notice back, and a failed mark (e.g. the user just became a viewer) is logged instead of leaving the workspace. - While saves fail the workspace is read-only. The view-mode banner says editing is disabled because changes can't be saved, with the Inspect banner's size and bold lead and no Done button. Users can still switch pages, pan, zoom and inspect. The lock lifts when a save lands and brings back the mode the user had or picked meanwhile; the WebGL context lock keeps the workspace read-only if it lasts longer. The lock lives in app.main.data.persistence.lock. - A viewer is never locked. A role change that leaves the user a viewer keeps the edits that lost permission, queued and on screen, and shows the count notice again. The role change that brings edit rights back releases them and rolls them back on screen, as a rejected save would. Any other terminal failure is released by the next role change or file opened. The debug faults `io` (socket timeout, 503) and `eof` (500) cover both sides of the I/O rule. |
||
|
|
989f68abfd |
♻️ Share the change helpers of the update-file tests
The update-file tests each built the same rect change and the same change that fails while applied. The rect change moves to the test helpers as th/add-rect-change and the failing one gets a named helper. The test of a database failure during the retry no longer counts calls to process-changes: it fails every attempt that leaves the failing commit out. |
||
|
|
2f3b928a60 |
⚡ Check the schema of only the pages a save touches
update-file checked the schema of the whole file on every save, which loads and decodes every page even when the changes touch one. On files with several pages that was most of the time of a save. validate-file-schema-affected! checks the file data with the pages index reduced to the pages the changes modify; the data outside pages is still checked whole, and a change type it does not know makes it check the whole file. update-file uses it for the strict and the soft schema check. A save of 10 commits on a file of 10 pages of 2k shapes goes from ~1.1 s to ~240 ms; a single page file costs the same. |
||
|
|
4f42ab1d2d |
✨ Keep the valid commits of a save that fails validation
A save whose changes failed validation was rejected as a whole: autosave stopped, the edits made after it piled up unsent, and a reload lost them all, the valid ones included. The frontend now sends each commit of a save as its own group with its commit id (:changes-with-metadata). When applying and validating all of them at once fails, the backend searches by bisection for the first commit that fails, keeps the commits before it and answers with :rejected (that commit and every later one). Only the kept changes are stored, snapshotted and broadcast, so other users never see the rejected ones. Flat :changes and groups without ids stay all-or-nothing; database and network errors still fail the whole request. A rejection without an error type is reported as an internal error. The frontend rolls the file back to what the server holds: it undoes every pending commit of that file, applies again the remote changes that arrived meanwhile, and redoes the kept commits. Undoing only the rejected commits would write stale values over a peer's later change to the same attribute. The rejected and later commits leave the queue, the buffer and the undo history, and a warning that hides after ten seconds tells the user. The pure bookkeeping lives in app.main.data.persistence.commits. The pending notice of a commit the rollback dropped no longer leaves the save status at pending, which made "Pin this version" hang and fail. Debug tools drive these cases by hand: debug.commitInvalidChange() makes a save fail validation, holdSaves() delays the answers, failNextSave() and failNextRollback() fail them on purpose (backend flag debug-save-faults), and persistenceState() shows the queue. |
||
|
|
97fe51e08b | Merge remote-tracking branch 'origin/staging' into develop | ||
|
|
9234e829d9
|
✨ Render email templates inside the mail job (#12448)
* ✨ Render email templates inside the mail job The web process rendered each email to full subject and HTML body before submitting it, so every queued job carried a huge params payload. The queue now stores only the template id plus the raw template data, and the new :send-templated-mail job renders and sends the email inside the worker. The old :sendmail job stays in place to drain jobs submitted in the previous format; the quota notice and the REPL test mail keep using it. All templated callers (auth, profile, feedback, invitations, comments, Nitrate) submit recipes through eml/send-template. Closes #12445 AI-assisted-by: muse-spark-1.3-contributor * ♻️ Unify templated mail on template id strings The ::eml/template value was a namespaced keyword whose name had to match the registry key, with one live mismatch (user-feedback vs feedback). Callers now pass the registry id string itself, so keys stay ::eml/-namespaced and values stay plain data, and the single templates set rejects unsendable mail both at submit and in the worker. Closes #12445 AI-assisted-by: muse-spark-1.3-contributor * 🐛 Encode templated mail data with the template schema The job params column is plain JSON, so the URIs of an organization (:logo, :avatar-bg-url) were stored as record maps and the template check rejected them in the worker. Invites for organization teams, organization invites and renewal notices failed without a retry. Each registry entry now carries the JSON encoder and decoder of its template schema: send-template encodes the data before submit and the worker decodes it before rendering. The job schema keeps only the mail envelope, as the template schemas own the typed data. Closes #12445 AI-assisted-by: claude-opus-5-5 --------- |
||
|
|
eef6868e9f
|
✨ Forward attribute update events (#12861) | ||
|
|
e42bae34cf
|
✨ Preserve subscription audit properties (#12850) | ||
|
|
9d898b8b31
|
🔧 Normalize the tenant to lowercase (#12843)
An operator writing `PENPOT_TENANT=ACME` and one writing `acme` name the same tenant: the value is compared as an exact string in the keys and rows it lands in, and one uppercase letter would otherwise split one tenant into two. The value is normalized when the configuration is read, so nothing downstream has to care about case, and the validated form accepts only lowercase labels from then on. AI-assisted-by: deepseek-v4.1-flash |
||
|
|
f532b8c12e
|
✨ Add get-profiles-by-ids management method (#12832) | ||
|
|
35baa06606
|
✨ Migrate legacy file payloads to file_data and delete orphans (#12785)
* ✨ Migrate legacy file payloads to file_data and delete orphans Copy file.data and labelled file_change.data to file_data on boot (type main/snapshot, backend db, deleted_at preserved), then delete file_change rows with data but no label, which no read path reaches. Large instances pre-migrate with batched statements and fake this step. Closes #12784. AI-assisted-by: muse-spark-1.3-contributor * 📚 Add 2.19.0 operations runbook for legacy file_data migration Manual pre-migration procedure for big instances, same statements as migration 0157 in batched transactions, following the convention of the file-change partition runbook. Closes #12784. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Address review feedback on the legacy file data migration Fix the issues raised in the review of the migration that moves the legacy file payloads to file_data. The migration deletes the unlabelled file_change rows that carry data, but restore-files still cleared their date, and the retention test asserted they came back. A row without a date never reaches objects-gc, which is the leak fixed in #12419. Restore now unmarks only snapshots, the rows that carry a label, and the retention test checks that the legacy payload keeps its date instead of coming back. Cover the conflict path that had no test: a pre-existing snapshot that collides with a labelled change row, and a second run that must leave the legacy columns null and the orphan row absent. The runbook analyses file_data after the inserts, so the planner does not wait for autovacuum. Drop app.srepl.procs.fdata-storage: the copy it performed is what the migration replaces and nothing requires it. Closes #12784 AI-assisted-by: deepseek-v4.1-flash |
||
|
|
0e9a9a8950
|
💥 Enable admin console by default (#12475)
Signed-off-by: David Barragán Merino <david.barragan@kaleidos.net> |
||
|
|
be37341f36 | Merge remote-tracking branch 'origin/staging' into develop | ||
|
|
610813281b | Merge remote-tracking branch 'origin/main-2.18.x' into staging | ||
|
|
41654aa1b7
|
✨ Restrict management RPC methods to their declared shared keys (#12736)
* ✨ Restrict management RPC methods to their shared keys Management methods now declare the shared-key ids allowed to call them with ::rpc/shared-keys. A wrapper in wrap-management checks the key id from the request before opening a transaction, so a valid key outside the declared set answers 403 shared-key-not-allowed. Methods without the metadata keep accepting any configured key. Scope upload-tempfile to exporter and media-processor, file analysis to nexus, durable jobs to exporter, and the Nitrate methods to admin-console. The test helper injects a declared key for the allowed path, and a coverage test fails when a management method is missing or declares the wrong set. Closes #12730 AI-assisted-by: deepseek-v4.1-flash * ♻️ Harden shared-key authorization scope and tests Rename the per-method metadata to ::rpc/allowed-shared-keys so it reads as the allowed key ids, not the secrets. Fix the management test helper to resolve the key from the supplied request and to keep an explicit nil, which prevents a rejection test from silently using an allowed key. Strengthen coverage: assert every management namespace is scanned, check all four job methods reject a key they do not declare, verify a rejected complete-job leaves the row and its events untouched, and exercise both keys allowed on upload-tempfile. Log denied keys with the method name and key id, without the secret, and update the audit-log memory for push-audit-events. Closes #12730 AI-assisted-by: deepseek-v4.1-flash |
||
|
|
afe6c1b99c
|
🐛 Confine upload paths to temp dirs (#12772)
* 🐛 Confine upload paths to temp dirs Reject upload maps whose path leaves the server temp dirs. Transit bodies can forge a Path such as /etc/passwd and the handlers open it directly. Centralize the realpath prefix check in media validation and apply it to all nine upload fields. Legit multipart files under /tmp/undertow/ or /tmp/penpot/ still pass. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Rename upload validators to check-* Drop the validate prefix and the bang from the media validation helpers for a consistent check naming. Move the temp dir confinement into validation and cover all nine upload fields. Add unit tests for each helper. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Fix svg upload tests to use penpot tmp dir SVG upload tests wrote tempfiles to the system tmp dir, outside the allowed upload prefixes, so check-upload-path rejected them. Point write-temp-svg at /tmp/penpot like the other upload tests. AI-assisted-by: muse-spark-1.3-contributor |
||
|
|
93d36d04d8
|
✨ Stabilize fragmented file data across snapshots and GC (#12420)
* ✨ Stabilize fragmented file data across snapshots and GC Snapshots, automatic and user, now store pointer-based rows that share fragments with the live file instead of duplicating page content. `get-file-snapshot` resolves those pointers on the server so version previews keep rendering plain data, and `restore!` reuses the shared rows. Export still materializes (the package carries no fragments) and import regenerates them. `file-gc` treats the main file and every unpurged snapshot as liveness roots, reads the file unrealized, and reattaches the original pointer wherever the cleaned value is unchanged, so a pass over an untouched file writes no fragments and a pass over an edited file keeps the ids of the pages it did not touch. Snapshot creation and the restore backup take the file row `FOR SHARE`, which serializes them with the pass and closes the race that let a new snapshot reference fragments the pass was about to mark. All backend pointer bindings go through the `fdata/with-pointers`, `with-tracked`, `with-tracked-pointers` and `with-untracked` macros. The loader reads fragments a previous pass marked until `objects-gc` purges them, and the frontend fails loudly with retry instead of dropping an unresolved fragment. Review fixes in this branch: the automatic snapshot attaches before pending migrations, `realize-pointers` is the single verb, `reattach-unchanged` takes the identity fast path, the restore guard takes its revn from the backup read, and `PointerMap.withMeta` generates a fresh id when metadata changes. Tests cover both snapshot writers (red without the lock), snapshot sharing and pin release through the real delete and purge paths, reattach against orphan marking, preview after offload, the pointer binding contracts and the with-meta id change. Closes #12410 AI-assisted-by: muse-spark-1.3-contributor AI-assisted-by: deepseek-v4.1-flash ♻️ Pin no-churn behavior for untouched pages Pin the behavior the churn verification measured: a save keeps the fragment id of every page it does not touch, with a page of eleven keys (PersistentHashMap), a 46-shape objects map and a nine-page index (hash map). Also pin the mixed case: a data-level objects-map flip rewrites the old pointer pages once and the next save keeps the new ids. The memory records the identity rule behind it: re-assoc with the same value returns the same map in array and hash maps alike. Closes #12410 AI-assisted-by: deepseek-v4.1-flash ⚡ Stop loading untouched pages when enabling objects-map `enable-objects-map` wrapped every page on every save, and reading a page's objects forced its PointerMap to load the fragment. A save of one page in a seven-page file therefore made seven fragment reads. Skip unloaded pointers: their fragment was already wrapped by the encode-file that wrote it, so loading it only to wrap it again is a no-op. The save now loads only the pages it edits. A page with a plain fragment (a data-level feature flip) keeps its plain fragment until it is first loaded or edited; a migration that enables the feature must rewrite the data itself. Cover the change with load-count tests for save and restore, update the flip test to the new contract, and drop the stale restore comment. Closes #12410 AI-assisted-by: deepseek-v4.1-flash ⚡ Keep page fragment ids through pending migrations Applying a pending migration on read realized the file, ran the migration and persisted it again; the persist wrapped every plain page in a new PointerMap, so every page got a new fragment id even when the migration did not change it. That leaves the fragment cache cold and rewrites every fragment on the first read after a deploy. Capture the original pointers before realizing, reattach the unchanged positions after migrating (same value comparison file-gc already uses), persist the reattached file and return the realized one: the read contract is plain data. Read-only connections skip the persist and the reattach. A migration that adds objects-map skips the reattach so the pages get wrapped. Move the three helpers to fdata so file-gc and the migration path share them, and cover the path with a real 0031 migration test: the unchanged page keeps its id, the changed page gets a new one, the caller gets realized data and a second read does not migrate again. Closes #12410 AI-assisted-by: deepseek-v4.1-flash * 🐛 Realize library pointers with each library's own loader Fragment rows are keyed by file id, and every caller binds the fragment loader of the file it works on. get-resolved-file-libraries returned libraries with their pages still as pointers, so the first read of a library page under fdata/pointer-map asked the consuming file for the fragment and failed with :fragment-not-found. That broke migrations and file validation for every file using a pointer-mapped library. The weak-map loader now realizes each library's pointers with the library's own id. The test fails without the change. AI-assisted-by: claude-opus-5-5 Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 🐛 Restore the pointer-map require the repair path still needs The branch dropped `app.util.pointer-map` from the namespace while the auto-repair path (added to develop before this branch) still rebinds `pmap/*tracked*` around validation and repair in `repair-preexisting-damage` and `process-changes-and-validate`. The namespace did not compile, so the whole backend failed to load. Those toggles do not fit the fdata macros: `with-untracked` binds nil and `with-tracked` creates a fresh atom, while the repair path has to turn tracking off and restore the existing atom (and read it before the repair starts). Restore the require and keep the bindings explicit. Also pin the feature-off path with a test: with `fdata/pointer-map` off (the default), a save creates no fragments, file-gc writes nothing and the snapshot preview still returns plain data. Drop a stray `\n` char literal left at the end of one deftest. Update the memories with the exception to the macro rule, the snapshot FOR SHARE locking and the pre-migration automatic snapshot. Closes #12410 AI-assisted-by: deepseek-v4.1-flash * 🐛 Validate debug files with their own pointer loader The `/dbg/actions` file validation passed the file id where `get-resolved-file-libraries` expects the file map, so the library map was seeded with a nil key. It also validated a file read without realizing it, so the first page lookup of a pointer-mapped file raised "load is not supported when *load-fn* is not bind" and the endpoint could not validate exactly the files the feature touches. Read the file realized (validation walks every page and component anyway) and build the library map from the file. The library loader already resolves each library's pointers with its own id, so the realized file and its libraries validate as plain data. Covered by a handler-level test that fails before the change. AI-assisted-by: deepseek-v4.1-flash * ♻️ Bind the snapshot loader through the fdata helper `reduce-snapshots` hand-rolled the same `binding` that `fdata/with-pointers` builds (it also loaded the function's own pointer loader before entering the reduction). Use the macro so the rule holds there too and the namespace no longer needs `app.util.pointer-map`; the loader is identical (same cfg and file-id) and snapshot walking behaves the same. AI-assisted-by: deepseek-v4.1-flash * 📚 Note the pointer-binding exceptions and snapshot ordering Record what the feature review settled: - Two direct `binding`s stay on purpose: the fine-grained `*tracked*` toggles of `app.rpc.commands.files-update` (validation/repair turn tracking off and restore the existing atom) and `binfile.common/encode-file` (fresh tracked set inheriting the current one). Atom-scoped macros would only be worth it if the set grows. - The automatic snapshot is taken before the pending migration, and the pre-migration window is narrow: the `get-file` RPC migrates on read and its ETag hashes the available migrations, so a post-deploy read cannot be a 304 from the pre-deploy state. The old-format snapshot only appears when the save is the first request to touch the file after the deploy, and `get-file-snapshot` previews it as stored. AI-assisted-by: deepseek-v4.1-flash * 🐛 Apply pending migrations when previewing a version Automatic snapshots are attached before the save applies pending file migrations, so a version can store the old format. The preview RPC handed that data to the UI as stored, and the viewport renders it with the current code. Realize the snapshot with the live file id (fragments are keyed by file, not by snapshot), normalize its migrations to a set (the decoder keeps the pgarray as a vector, and `set/difference` would compare indexes) and apply the pending migrations with the pure `fmg/migrate-file` and the live libraries. Nothing is persisted: the stored version keeps its data and restoring it re-migrates on the next read. The response now carries the snapshot's migrations. Use `:features` the way `get-file` does for `fdata/objects-map`: a client that does not declare it gets plain objects. The historical feature set is not checked against the enabled one on purpose: a version may predate a feature change. Cover the preview of a snapshot with a pending migration and the objects realization; both pin that the stored row is untouched. Update the file-data memory with the preview contract and the vector/set trap. AI-assisted-by: deepseek-v4.1-flash * 🐛 Decode snapshot migrations as an ordered set `file-snapshots/decode-snapshot` handed the `file_change.migrations` pgarray as a vector. `set/difference` falls back to index lookups when the first collection is shorter than the second, so the pending set was right only by accident: a stored subset happens to work, while a stored set larger than `available-migrations` (a snapshot taken by a newer build) marks every available migration as pending and re-applies it to already migrated data. Decode it as a `d/ordered-set`, the type the rest of the code and `schema:file` expect. This also covers `restore!` and the snapshot list, and it lets `get-file-snapshot` drop the `(into #{} ...)` normalization it was carrying. Re-wrap the union in `fmg/migrate` so the invariant holds even when a caller hands a plain set. A snapshot with NULL `migrations` predates SQL migration 0140 and keeps the format version of its time. Pin that legacy path: the preview regenerates the applied set from that version, migrates in memory and leaves the stored row untouched. Update the file-data memory with the invariant and the legacy path. AI-assisted-by: deepseek-v4.1-flash * 🐛 Fix flaky waiting-gauge assertion in climit test The throwable-while-waiting test asserted the exact list of waiting-gauge publications, but the component sampler republishes the same count when its first tick lands inside the recorded window under load. Assert only the last value, which is the contract the helper documents. AI-assisted-by: muse-spark-1.3 --------- Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> Co-authored-by: Álvaro Tejero Cantero <alvorithm@teje.ro> |
||
|
|
29373fff9f
|
✨ Rework the progress of the import and export dialogs (#12192)
* ✨ Rework the import dialog The import wizard paints a Check/Upload/Import stepper derived from the dialog status, plus a phase title, a description and a progress bar per phase, and its rows follow the new design: folder icon, file name and a size subtitle. While checking, the dialog shows no rows and no spinner, because the phase header already carries the message. The wizard writes the job milestone once, in the header (`jp/current-milestone-text`), since every file of a container reports the same one. A row keeps its name and its size, says "Ready" once its container finished uploading, and shows the error text when it fails; the button that drops a file only appears while the wizard still offers its import step. The `jobs.queued` label now reads "Waiting job slot". The bar measures the whole import: determinate while the containers are uploaded, weighted by their size (`imp/upload-pct`), and animated otherwise, because the stages of a job are not comparable. The phase title counts the files, and `imp/imported-count` counts them done with the job's own counter, so the number moves while a container rebuilds its files one by one. The analysis reads the entries of a version 3 package in one pass: a file shows what it occupies inside its container, its own entries plus the storage objects its media and thumbnails reference, while what the container weighs stays in `:container-size`, the size the upload bar is weighed on. The ready step waits for the end of the analysis (`imp/ready-to-import?`), not for every known file, so Continue never leaves out the files that arrive later. Tests cover the file sizes, the upload percentage, the done counter, the wizard step and the milestone in course. The change also adds `-XX:NativeMemoryTracking=summary` to the backend devenv so `jcmd` shows native memory, and documents the wizard, the promise call styles (`mem:clojure/promesa`) and two CSS pitfalls: a logical `inline-size` wins over an inline physical `width`, and Rumext leaves a plain `:style` map raw, so inline styles leave CLJS as `#js` objects. AI-assisted-by: glm-5.3-flash AI-assisted-by: deepseek-v4.1-flash * ✨ Rework the export dialog Every exported file is its own job, so each row reports its own state: folder icon, name and a size-plus-`.penpot` subtitle; waiting for a worker; running with the milestone of the job and an animated bar; or exported with a tick. A failed row paints red, takes the close icon and shows the hint of the job, or a generic label, through `files-export-modal.export-error`. Success, failure and cancellation replace each other, so a row never reports two outcomes at once. The ids of the messages are canonicalized to string before they match a row, because they cross serialization boundaries. The export result now carries `:size`, so the subtitle says the format until the job produces the artifact and the size once it exists; a transport failure builds a synthetic error, so a nil `ex-data` no longer leaves the row hanging. The header counts the files and says whether the export runs or ended; since every file has its own job, it paints no global bar. Closing changes too: the Close button is gone. While files are in flight the footer offers Cancel export and the X cancels as well, which marks the loading rows cancelled and leaves them in place, so their outcome can be read; once every row reached a terminal state the footer offers Done, which dismisses the dialog. The retry button of the finished design is left out, because the client has no retry mechanism yet. Styles come along: the rows scroll while the header and the footer stay put, and the new phase header, row and bar follow the tokens of the import wizard. The title key becomes `files-export-modal.title`, and the new keys cover the header, the row states and the buttons. AI-assisted-by: muse-spark-1.3 * ♻️ Inline the binfile job adapters into their task handlers The wrappers in app.binfile.jobs only forwarded to the core, so each task handler now composes its own run: frozen cfg, active check, progress bridge and format dispatch. export-cfg freezes the whole export (file-ids, export-type, output) and v3/export-files! takes it as its single argument; ::bfc/ids becomes ::bfc/file-ids. The bridge keeps one job: bfj/run forwards the cfg to its callback, and check-active answers with it to thread the run. AI-assisted-by: muse-spark-1.3 * 🐛 Migrate binfile tests to the single-arg export-files! The refactor made v3/export-files! take a single cfg carrying ::bfc/file-ids and ::bfc/output, but six calls in binfile_test (added by the tokens handling change) still used the old two-arg form with ::bfc/ids. Move them to the new form, as the rest of the file already does. AI-assisted-by: muse-spark-1.3 * 🐛 Fix export artifact expectations to include file size The export flow now reports the artifact size, but the stale expectations dropped it and failed the frontend suite. Include the size in both assertions so they match the emitted message. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Migrate remaining export-files! test callers to the single-arg form rpc_binfile_test and offload_file_data_test still passed the output apart and used ::bfc/ids; move both into the cfg as ::bfc/file-ids/::bfc/output. * 🐛 Hide the downloads hint when no file was exported After Cancel or when every job failed, the export header said "0 files exported · Check your downloads folder."; drop the hint then. * ♻️ Read the object of a container entry once The index of a container asked twice for the object of the same zip entry: once to test it and once as the key of the update. Resolve it once, in the branch that owns it, so the entries of a file never pay for a question that does not apply to them. Addresses a review note on #12192. AI-assisted-by: deepseek-v4.1-flash * ♻️ Never size an import row with its container's weight A row of a version 3 package took the size of the container whenever the container held no entry for its file, while the comment beside it promised the opposite. An unknown size now stays nil, which the row already understands: it paints no subtitle instead of a number that belongs to the whole package. The test carries a file the manifest lists and the container does not hold, and the memory records what the row does then. Addresses a review note on #12192. AI-assisted-by: deepseek-v4.1-flash * 🐛 Retry pool stats read until snapshot is consistent Hikari exposes each pool counter through a separate MXBean call, so a borrow or return between the reads can leave one snapshot off by one. The test read once and failed in CI with total 2 vs active + idle 3. Read up to 20 times with a 10ms pause until total equals active + idle, then assert on the stable snapshot. The invariant stays strict; only the transient race is tolerated. AI-assisted-by: muse-spark-1.3-contributor --------- Co-authored-by: Elena Torro <elenatorro@gmail.com> |
||
|
|
65a76a21cd | Merge remote-tracking branch 'origin/staging' into develop | ||
|
|
c0f0b14540
|
🎉 Add direct MCP feedback channel via dedicated tool (#12551)
* ♻️ Move devenv-only tools to a separate folder * 🎉 Add an MCP feedback channel Add a user-approved feedback tool and an MCP-token-authenticated backend endpoint that queues plain-text email. Enable the tool through the configured backend URI in both local and multi-user mode. Validate subject and feedback limits, enforce per-profile rate limits, and document configuration. Cover authentication, queueing, tool availability, and failure handling with backend and MCP tests. Closes #12490 AI-assisted-by: gpt-6 * ♻️ Move MCP feedback submission into RPC Use the standard RPC wrappers for feedback and remove the dedicated HTTP endpoint, route wiring, and custom rate limiter. Restrict MCP tokens to opted-in routes and commands. Add a shared RPC call helper to the MCP backend client and adapt the existing tests. AI-assisted-by: gpt-6 * 🐛 Harden MCP feedback auth, validation and messages Accept MCP tokens when either the mcp or the access-tokens flag is on, so instances that enable MCP without personal access tokens can send feedback. Personal tokens still need access-tokens. Keep the issuer in one constant, decode tokens without raising to reject unknown issuers, and name the issuer and type checks. Reject control characters in the subject, and in the feedback all but tabs and line breaks, in both the command and the tool, so a NUL answers 400 instead of failing when the mail job is stored. Answer 403 with profile-blocked or profile-not-active for unusable profiles, record only who sent feedback in the audit event, and add doc metadata. Replace on OAuth approval an MCP token issued before MCP tokens had their own issuer, or one this instance cannot decrypt, since the backend rejects both. In the MCP server, mention disabled access tokens on 401, drop the unused 503 case, and say that delivery happens in the background. Document the flags in the README. AI-assisted-by: claude-opus-5-5 * 📚 Document the MCP feedback tool and MCP token rules List submit_feedback among the MCP tools and add an MCP feedback section to the configuration guide covering PENPOT_MCP_BACKEND_URI, PENPOT_MCP_FEEDBACK_EMAIL, the flags and the email settings. Add commented examples of both variables to the compose file. In the agent memories, record when OAuth approval replaces an MCP token, how the MCP server reaches the backend, and which flags, routes and commands accept MCP tokens. AI-assisted-by: claude-opus-5-5 |
||
|
|
3cc91e8ba2
|
♻️ Take the command limits before the root ones in the RPC limiter (#12686)
* ♻️ Take the command limits before the root ones in the RPC limiter The RPC concurrency limiter took the root limiters (the ones every request passes through) before the ones a command declares, so a request waiting for the limits of its own command held the root permits of its profile while it waited: a profile with a pile-up of heavy requests saw its other requests queue behind them and be rejected with 429. That order was forced by two commands that entered the limiter from inside their handler: a command that declares no limits of its own can only take them deeper than the root ones, so two paths of one profile could take the same limiters in opposite order and park for good. The commands declare their limits in their metadata now and their handlers call the work directly, so no command enters the limiter from inside its handler and every path takes the limiters in the same order. The root limiters are appended to the sequence first, so the limits a command declares end up outermost and are taken first: a request waiting for them does not hold the root permits of its profile. The photo of a team keeps its limits, because it declares them now that the helper does not. A call from inside a handler can only reenter the limits that thread already holds, and it is refused when it asks for a new one, because that would take it after the root ones and invert the order against the other paths. The key of a limit is resolved the same way in every path, so both paths point at the same limiter and the reentrancy check sees it. Every entry of the shipped climit.edn declares a queue, so the number of requests waiting for the same limit is bounded instead of piling up until the timeout. The production file lives outside the repository and has to be mirrored to pick them up. The tests cover the acquisition order of a command with and without limits of its own, the order of the limits as a whole (a ring of any size fails, not only a pair in opposite order), the nested call that asks for a limit the thread does not hold, and the queues and timeouts of the shipped configuration. The import of a binfile declares its request as multipart in its test, the way the commands that take an upload map do. AI-assisted-by: minimax-m3.1-flash AI-assisted-by: deepseek-v4.1-flash * 🔧 Keep the commands' limits and the shipped config in sync `profile-plugin-ops` guarded only an UPDATE on the caller's own profile row, which the row lock already serializes, so its global cap could only reject a burst of different profiles toggling plugins. Both commands declare the by-profile limit alone now, and the example config drops the entry. A declared id without an entry does not limit anything: `wrap` logs a warning and runs the handler with no limiter at all, which is how a command ends up unbounded in production. A test fails now when a command declares an id the config does not list, and the header of the example says the production file mirrors the shape and lists every declared id, not the values. AI-assisted-by: deepseek-v4.1-flash * 📚 Record that background work is not waited for with a sleep A long command moved to the background reports its own completion, so polling it with a sleep inside a foreground command only blocks the session and the notification arrives anyway. The rule says to keep working on something else, or to end the response and wait for it. AI-assisted-by: minimax-m3.1-flash |
||
|
|
e20a0d748e
|
✨ Add OAuth support for MCP server (#12480)
* 🎉 Add OAuth support for remote MCP connections Add MCP-specific discovery, client registration, and approval endpoints. Exchange single-use authorization codes with PKCE for the fixed MCP user token, enabling MCP and creating the token on approval when needed. Use self-contained client IDs without new database tables or stored client records. Keep temporary authorization state in Redis/Valkey. Accept bearer credentials while preserving query-parameter connections. Enable OAuth through PENPOT_MCP_AUTH_URI in multi-user mode. Document the environment variable in the README and implementation details in Serena. Explain protocol and concurrency decisions in the Clojure code. Cover token provisioning, OAuth state, client registration, HTTP flows, and the approval page with tests. AI-assisted-by: gpt-6 * 🐛 Wait for devenv initialization before starting builds Docker reports a container as Running as soon as its main process starts, while the entrypoint may still be preparing the development environment. manage.sh used that status to launch tmux jobs, allowing frontend builds to race with ownership changes to /opt/rustup and Cargo setup. On a fresh container, rustup could fail with Permission denied before the frontend watcher started. Nginx would later serve the existing JavaScript, leaving source changes absent from the running app despite restarting devenv. Wait for nginx's local health endpoint before launching the jobs. Nginx starts after entrypoint setup, so its response confirms that setup has finished. Allow up to 120 seconds and report a timeout without starting the jobs if initialization does not complete. Validation: shell syntax and whitespace checks pass. AI-assisted-by: gpt-6 * ♻️ Share the login modal with MCP authorization Extract the viewer login modal and its styles into a shared auth component, retaining the viewer wrapper and its close behavior. Use the modal on the MCP approval page and preserve the pending request through login. Closing it clears the saved return URL and shows a cancellation message without further actions. Cover return URL handling and cancellation with frontend tests. AI-assisted-by: gpt-6 * ✨ Show client icons in MCP authorization Preserve HTTPS logo URLs from client registration and published metadata and show the icon beside the application name on the approval page. Ignore unusable URLs and hide failed images without blocking approval. Update the access description to refer to MCP connections established from Penpot. Cover logo metadata and approval responses in backend tests. AI-assisted-by: gpt-6 * 🐛 Follow the system theme when no preference is configured Default to the system theme in the shared resolver used by the app and plugins. Preserve explicit light and dark choices, including the legacy default value that means dark. Reuse the reactive theme hook in progress widgets and show the system preference in settings for accounts without a configured theme. Verify preference resolution with regression tests and browser checks. All 1,474 frontend tests pass. AI-assisted-by: gpt-6 * 🐛 Replace expired MCP tokens during OAuth approval Settings can give an MCP token an expiry date. OAuth approval reused the newest MCP token without checking it, so an expired token reached the client and never matched a plugin connection. Approval now treats an expired token as missing and creates a new one. Token exchange rejects a code whose token expired after approval with invalid_grant. AI-assisted-by: claude-opus-5-5 * 🐛 Log unexpected errors on MCP OAuth endpoints The OAuth routes sit outside the shared HTTP middleware, so the shared error handler never saw them, and their own handler turned unexpected failures into a 503 without logging anything. Failures in the session middleware of the approval endpoints had no handler at all. Unexpected failures now also go through the shared error handler for its logging, while clients still get an OAuth error body. The OAuth error handler also wraps the session middleware of the approval endpoints. AI-assisted-by: claude-opus-5-5 * 🔧 Add devenv script to run the MCP server with OAuth start-mcp-devenv-oauth starts the MCP server in multi-user mode with PENPOT_MCP_AUTH_URI derived from PENPOT_PUBLIC_URI, to test OAuth in the devenv. Agentic tooling still uses the single-user start-mcp-devenv. AI-assisted-by: claude-opus-5-5 * 🐛 Fix failing test case --------- Co-authored-by: alonso.torres <alonso.torres@kaleidos.net> |
||
|
|
4dc684dd9c | Merge remote-tracking branch 'origin/staging' into develop | ||
|
|
506923e2e9 | Merge remote-tracking branch 'origin/main' into staging | ||
|
|
ff9506c551
|
✨ Add member profile support for organization attributes (#12652) |