From 58b349fb779b7c9c80332f75cc8751dcc2ecb3e8 Mon Sep 17 00:00:00 2001 From: Filip Sajdak Date: Tue, 10 Nov 2026 11:33:42 +0100 Subject: [PATCH 01/29] :wrench: Keep the CI expect timeout in the default and ds e2e projects (#12922) The Playwright config gives the retrying expects 40 seconds on CI and 5 seconds elsewhere, but the default and ds projects set their own expect block for the screenshot tolerance, and a project block replaces the top-level one instead of merging with it. On CI those projects have run with Playwright's 5 second default. That is too short for the first dashboard load on a busy runner, where the dashboard module is fetched lazily: specs that wait for the dashboard heading time out and pass on the retry (flaky), or fail after the retries. In the shard of one recent run that was 7 flaky and 4 failed specs out of 112, and develop itself failed the same families of specs on several runs the same day. Name the timeout once and repeat it in both projects. A probe config that imports this one and polls a condition that never holds reports 40000 ms for both projects with this change, and 5000 ms without it. The render-wasm project already repeats its own value. AI-assisted-by: claude-opus-5-5 AI-assisted-by: claude-sonnet-5-5 Signed-off-by: Filip Sajdak --- frontend/playwright.config.js | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/frontend/playwright.config.js b/frontend/playwright.config.js index 087e6ba56f..a7e5496a7a 100644 --- a/frontend/playwright.config.js +++ b/frontend/playwright.config.js @@ -11,6 +11,13 @@ const userAgent = platform === 'darwin' ? "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" : undefined; +// Default timeout of the retrying expects (toBeVisible and friends). CI +// runners are slower and the first load of a page fetches its module, so +// they get more time. A project that sets its own `expect` replaces the +// top-level one instead of merging with it, so every such project has to +// repeat this timeout or it falls back to Playwright's 5 seconds. +const expectTimeout = process.env.CI ? 40000 : 5000; + /** * @see https://playwright.dev/docs/test-configuration */ @@ -29,7 +36,7 @@ export default defineConfig({ timeout: 80000, expect: { - timeout: process.env.CI ? 40000 : 5000, + timeout: expectTimeout, }, /* Reporter to use. See https://playwright.dev/docs/test-reporters */ @@ -58,6 +65,7 @@ export default defineConfig({ }, snapshotPathTemplate: "{testDir}/{testFilePath}-snapshots/{arg}.png", expect: { + timeout: expectTimeout, toHaveScreenshot: { maxDiffPixelRatio: 0.001, }, @@ -70,6 +78,7 @@ export default defineConfig({ use: { ...devices["Desktop Chrome"], colorScheme: "dark" }, testDir: "./playwright/ui/visual-specs", expect: { + timeout: expectTimeout, toHaveScreenshot: { maxDiffPixelRatio: 0.005 }, }, }, From 34c702bc805aa3ec368b194b25dc51d659e4e5bd Mon Sep 17 00:00:00 2001 From: Andrey Antukh Date: Tue, 10 Nov 2026 23:27:17 +0100 Subject: [PATCH 02/29] :sparkles: Add table partitioning for file-change (#12417) * :sparkles: Partition file_change by file_id Partition `file_change` by `HASH (file_id)` in 16 partitions, with the same bounds as `file_data`, so a join between them is resolved partition by partition. The primary key becomes `(file_id, id)`, which a partitioned table needs, and every access filters by `file_id`. The new table carries a `kind` column (1 snapshot, 2 revision) as a reserve: the queries still tell them apart by the label, and `kind` takes over when they are rewritten. The payload of a snapshot stays in `file_data`; the `data` column is kept for reading old `legacy-db` payloads. The four snapshot commands require `file-id`, because it prunes the partition. The objects GC locks and deletes by the pair, in batches. The old table is kept as `file_change_legacy` as the way back. The automatic migration refuses to run above 10 GiB or 2M rows and points to the runbook; big instances partition online with that runbook, which copies by id, follows the `updated_at` watermark in delta passes and checks that no row of the source is missing from the target. Closes #12416 AI-assisted-by: deepseek-v4.1-flash * :bug: Address the review of the file_change partition The production runbook promised a rollback it did not have: the section was a sketch, and its statements could write to the table about to be dropped. The rollback is removed, and so is every comment that described it. `file_change_legacy` stays as a frozen copy of the previous layout. The cutover check now aborts the transaction when a row of the old table is missing in the new one, instead of leaving the decision to the operator. The five secondary indexes of the new table are built after the last delta pass: the bulk and the delta passes only need the primary key, so the batches no longer maintain them. The copy of 0158 is now replayed in a test against the pre-0158 shape with a labelled and an unlabelled row, plus an empty instance that must leave no legacy table behind. The updated_at assertion of the snapshot mutations is bounded by the test clock, and the repair revision stops writing the `features` of the file, like the normal save. AI-assisted-by: deepseek-v4.1-flash * :paperclip: Fix linter issues * :bug: Fix stale helper call in rpc_file_snapshot_test The suite does not load: `th/run-pending-tasks!` does not exist in backend-tests.helpers. The pending jobs runner is `run-pending-jobs` (no arguments), the name every other call site of the suite uses. AI-assisted-by: inkling-2-small-free * :books: Align the partition change with the 2.20.0 milestone 2.19.0 is already shipped; this work targets 2.20.0. The operations file of the partition procedure carries the 2.20.0 prefix, the references that point at it (the migration, the runbook table and the memory) follow, and the RPC doc/changes tags of the snapshot commands record 2.20 as the release that makes `file-id` required. AI-assisted-by: inkling-2-small-free * :bug: Fix review findings on the partition rebase The 0157 legacy test was missed by the rebase and inserted file_change rows without the new mandatory `kind` column, so the backend suite fails against the migrated schema. The inserts now carry `:kind` (snapshot-kind for the labelled rows, revision-kind for the rest). The post-cutover verification block of the runbook terminates the first query with a semicolon so the block pastes into psql without a syntax error. The guard message of the migration names the step (0158-partition-file-change) instead of "0158", ambiguous since two migrations share that number. Closes #12416 AI-assisted-by: inkling-2-small-free * :bug: Fix the stale label in the snapshot restore test The rename block of snapshots-crud runs before the restore block, so the restored version is already named "label1-updated" when the restore publishes its message. The command names the version by the label the row has at that moment, not by the one it had when it was created, so the test was asserting a stale value. Assert the current label, and say why in the comment. AI-assisted-by: deepseek-v4.1-flash * :recycle: Split the version event tests into one test per command The workspace version events and the plugin proxy send the file-id of the open file on every snapshot command, because the command needs it to prune the partition of file_change. Both tests checked that with a doseq and t/testing blocks, so a failure named the whole group and not the command that broke. Give each command its own test. The workspace ones share a helper that runs the event and returns the command and params it sends; the plugin ones keep their own mock of rp/cmd!. AI-assisted-by: deepseek-v4.1-flash --- .serena/memories/backend/storage.md | 15 + .serena/memories/clojure/idioms.md | 1 + backend/src/app/db.clj | 5 +- backend/src/app/features/fdata.clj | 5 +- backend/src/app/features/file_changes.clj | 21 + backend/src/app/features/file_snapshots.clj | 118 ++- backend/src/app/migrations.clj | 5 +- .../sql/0158-partition-file-change.sql | 251 +++++ backend/src/app/rpc/commands/files.clj | 4 +- .../src/app/rpc/commands/files_snapshot.clj | 31 +- backend/src/app/rpc/commands/files_update.clj | 9 +- backend/src/app/rpc/quotes.clj | 13 +- backend/src/app/scripts/decode_file_data.clj | 20 +- backend/src/app/tasks/delete_object.clj | 6 +- backend/src/app/tasks/objects_gc.clj | 44 +- backend/test/backend_tests/db_test.clj | 8 + .../test/backend_tests/fdata_resolve_test.clj | 42 + .../test/backend_tests/fdata_upsert_test.clj | 55 ++ .../file_change_retention_test.clj | 5 +- .../backend_tests/file_change_schema_test.clj | 110 +++ .../migrations_legacy_file_data_test.clj | 11 +- .../migrations_partition_file_change_test.clj | 210 ++++ .../test/backend_tests/objects_gc_test.clj | 65 ++ .../restore_storage_objects_test.clj | 27 + .../backend_tests/rpc_file_snapshot_test.clj | 246 ++++- .../rpc_file_update_repair_test.clj | 4 + .../test/backend_tests/rpc_quotes_test.clj | 1 + .../scripts_decode_file_data_test.clj | 11 + .../src/app/main/data/workspace/versions.cljs | 29 +- frontend/src/app/plugins/file.cljs | 19 +- .../data/workspace_versions_test.cljs | 71 ++ .../frontend_tests/plugins/file_test.cljs | 114 ++- operations/2.20.0-file-change-partition.md | 912 ++++++++++++++++++ operations/README.md | 16 + 34 files changed, 2388 insertions(+), 116 deletions(-) create mode 100644 backend/src/app/features/file_changes.clj create mode 100644 backend/src/app/migrations/sql/0158-partition-file-change.sql create mode 100644 backend/test/backend_tests/fdata_resolve_test.clj create mode 100644 backend/test/backend_tests/fdata_upsert_test.clj create mode 100644 backend/test/backend_tests/file_change_schema_test.clj create mode 100644 backend/test/backend_tests/migrations_partition_file_change_test.clj create mode 100644 operations/2.20.0-file-change-partition.md create mode 100644 operations/README.md diff --git a/.serena/memories/backend/storage.md b/.serena/memories/backend/storage.md index 529fcd1cc8..f10e74124b 100644 --- a/.serena/memories/backend/storage.md +++ b/.serena/memories/backend/storage.md @@ -131,6 +131,21 @@ Since `put-object!` uses backend-specific operations (`impl/resolve-backend` + ` - Writes always resolve the backend through `fdata/default-backend` (`cf/get :file-data-backend`), so post-offload edits land back in hot (`db`) storage without extra code; reads stay transparent through `fdata/resolve-file-data` + `decode-file-data` + `realize`. - Deleting an offloaded row touches its storage object first (`objects-gc/delete-file-data-chunk` via `fdata/process-metadata`), so `storage-gc-touched` reclaims the blob with the usual delay; restoring a file unmarks the referenced objects. +## File Change + +- `file_change` is partitioned by `HASH (file_id)` in 16 partitions, with the same bounds as `file_data`, so a join between them is resolved partition by partition (`enable_partitionwise_join` is set on every pooled connection). +- The automatic migration (`0158-partition-file-change`) refuses to run when `file_change` is above 10 GiB total size or 2M rows (`pg_total_relation_size` + `pg_class.reltuples`, catalog reads). It analyzes the table first, because `reltuples` is -1 without statistics. It raises with the path of `operations/2.20.0-file-change-partition.md`, because the copy happens inside the migration transaction and a big instance would restart in a loop. Below the guard it raises `statement_timeout` to 30 minutes for the copy and the index builds, and analyzes the new partitions at the end, after the copy and the indexes, so the planner does not start with empty statistics. +- Its primary key is `(file_id, id)`: a partitioned table needs the partition key in the primary key, so every access filters by `file_id`. From 2.20 the four snapshot commands (`update-file-snapshot`, `delete-file-snapshot`, `lock-file-snapshot`, `unlock-file-snapshot`) require the `file-id` param, because they use it to prune the partition; without it they answer `:params-validation`. +- It has a `kind` column: `1` is a snapshot (it has a label and its data lives in `file_data`) and `2` is a revision (it has no label and it holds the changes). The values are in `app.features.file-changes`. +- `kind` is a deliberate reserve, not a leftover: today no query filters by it, the queries still tell a snapshot from a revision by its label (`label IS NOT NULL`), and `kind` takes over when they are rewritten. No constraint ties `kind` and `label` together on purpose: the day `kind` is the source of truth, that constraint would have to go. +- Every write of the application to a row of `file_change` moves `updated_at` to the moment of the write (`lock-by!`, `unlock!`, the snapshot rename and delete, `delete-object`, the restore of a file, and the `legacy-db` snapshot write of `fdata`). The actions of the foreign keys (`ON DELETE SET NULL` on `locked_by` and `profile_id`) change the row without touching it; during the partition both tables carry the same keys, so both receive them. The runbook of the partition uses it as the watermark of its delta passes: a write of the application that leaves it behind is lost at the cutover. +- `file_change_legacy` is a frozen copy of the previous layout of the partition: it keeps the rows and the indexes of the old table. A new instance (empty table) does not keep it. On an upgraded instance the table stays until the operator drops it by hand days later (`DROP TABLE file_change_legacy;`, see `operations/2.20.0-file-change-partition.md`); nothing drops it automatically. +- A revision carries only the delta of a save (the changes). The features of the file are not repeated on it: nobody read them from there. +- A snapshot carries the features and the migrations of the file. +- It keeps the `data` column: the `legacy-db` backend is still supported (migration `0157-migrate-legacy-file-data` moved the existing blobs, but the backend keeps reading and writing that column). It has no `data_ref_id` or `data_backend` columns: nothing read them since the file storage refactor. +- It has no index for the system snapshots. The old one filtered them by `data IS NOT NULL`, and with the payload living in `file_data` the column no longer tells snapshots apart. The new one could only filter them by `deleted_at IS NULL`, but the system snapshots always carry a deletion date in the future, so it would never match a row. The query that would use it filters that date with an "or", which no partial index can serve. +- The `objects-gc` purge groups each chunk by `file_id` and locks it with `file_id = ?` and `id = ANY(?::uuid[])`: a tuple `IN ((file_id, id), ...)` gives the planner no constant partition key, so it scans all 16 partitions in every chunk. Each group is one `FOR UPDATE SKIP LOCKED` query inside the chunk transaction. + ## Metrics - `bucket` is always the Penpot logical bucket (object metadata), never an S3 bucket. Unknown/absent buckets are labeled `"unknown"`. diff --git a/.serena/memories/clojure/idioms.md b/.serena/memories/clojure/idioms.md index 3fa120aa62..e8c7bc299e 100644 --- a/.serena/memories/clojure/idioms.md +++ b/.serena/memories/clojure/idioms.md @@ -11,4 +11,5 @@ Behaviors confirmed against the language/stdlib — do not re-derive from assump - Inside an `^:async` fn, a nested call that needs a temp binding (e.g. calling a local fn `(g (f x))`, whose arg gets a `G__` binding) compiles to `(await (async function(){…})())`: `await` unwraps a returned promise, so `(settled (upload (make-ctx)))` blocks until the upload promise settles instead of passing the promise on (a test that pushes the answer later hangs, idle). Bind each step in its own `let` binding. - `!` suffix allowed cases (only exceptions to the no-new-`!` rule in `mem:clojure/design-rules`): mutating public `app.db` API (`insert!`, `update!`, `delete!`, `run!`, `tx-run!`, `exec!`, ...) and fns whose contract is to work over transients (stdlib `conj!`, `assoc!`, `dissoc!`, `disj!`, `persistent!` and custom helpers over them); any further case must be registered here with justification before use. - Penpot extends `clojure.core/Inst` in `app.common.time` ( JVM-only section), so `inst-ms` also accepts `Duration` (returns `.toMillis`, NOT epoch millis), `FileTime` and `OffsetDateTime`. It only works where that namespace is loaded; on a bare `Duration`, prefer explicit `.toMillis`, which reads as what it is (an amount, not an instant). +- The threading macros insert the threaded value at DIFFERENT positions: `->` and `some->` put it FIRST (`(-> x (f a))` → `(f x a)`), `->>` and `some->>` put it LAST (`(->> x (f a))` → `(f a x)`). `some->`/`some->>` also short-circuit on nil. With unary calls both nest the same way (`(-> x (f) (g))` and `(->> x (f) (g))` are both `(g (f x))`): what changes between them is the position inside each call, not the order of the steps. A pipeline that calls a fn whose first argument is a config/context (`app.features.fdata/resolve-file-data` is `(cfg, file)`) must use `->>`/`some->>`; with `->`/`some->` the arguments are swapped and the call silently returns the wrong value (it cost a broken `get-snapshot` in `app.features.file-snapshots`). - `cuerdas.core/index-of` returns `nil` (not `-1`) when the substring is absent, like `clojure.string/index-of`. `(pos? nil)` **throws on the JVM** but evaluates to `false` in CLJS (JS coercion), so `(if (pos? (str/index-of s "\n")) …)` works in the browser and crashes the backend. Guard with `some?` before `pos?` (this was `app.common.exceptions/first-line`). diff --git a/backend/src/app/db.clj b/backend/src/app/db.clj index c9998b49b8..23b5ddad7a 100644 --- a/backend/src/app/db.clj +++ b/backend/src/app/db.clj @@ -108,7 +108,10 @@ (def initsql (str "SET statement_timeout = 300000;\n" - "SET idle_in_transaction_session_timeout = 300000;")) + "SET idle_in_transaction_session_timeout = 300000;\n" + ;; lets the planner resolve a join between two partitioned tables + ;; pairwise, when both are partitioned with the same bounds + "SET enable_partitionwise_join = on;")) (defn- create-datasource-config [{:keys [::uri] :as cfg}] diff --git a/backend/src/app/features/fdata.clj b/backend/src/app/features/fdata.clj index d18da11ea6..da8b662514 100644 --- a/backend/src/app/features/fdata.clj +++ b/backend/src/app/features/fdata.clj @@ -205,8 +205,11 @@ (db/delete! cfg :file-data {:id id :file-id file-id :type "snapshot"} {::db/return-keys false}) + ;; every mutation of `file_change` moves `updated_at`: the partition + ;; runbook uses it as the watermark of the delta passes (db/update! cfg :file-change - {:data data} + {:data data + :updated-at (ct/now)} {:file-id file-id :id id} {::db/return-keys false})) diff --git a/backend/src/app/features/file_changes.clj b/backend/src/app/features/file_changes.clj new file mode 100644 index 0000000000..f01b7bfe6c --- /dev/null +++ b/backend/src/app/features/file_changes.clj @@ -0,0 +1,21 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns app.features.file-changes + "The kinds of the rows of the `file_change` table. + + A snapshot is the state of a file at a point: it has a label and its + data lives in `file_data` (old `legacy-db` payloads live in + `file_change.data`). A revision is the delta of a save: it has + no label and it holds the changes.") + +(def snapshot-kind + "The kind of a snapshot row." + 1) + +(def revision-kind + "The kind of a revision row." + 2) diff --git a/backend/src/app/features/file_snapshots.clj b/backend/src/app/features/file_snapshots.clj index 2370758d28..7c1818b743 100644 --- a/backend/src/app/features/file_snapshots.clj +++ b/backend/src/app/features/file_snapshots.clj @@ -19,6 +19,7 @@ [app.db :as db] [app.db.sql :as-alias sql] [app.features.fdata :as fdata] + [app.features.file-changes :as fchanges] [app.jobs :as jobs] [app.storage :as sto] [app.util.blob :as blob] @@ -74,57 +75,71 @@ LEFT JOIN file_data AS fd ON (fd.file_id = f.id AND fd.id = f.id) WHERE f.id = ?") +(def ^:private sql:snapshot-columns + "c.id, + c.label, + c.revn, + c.created_at, + c.modified_at, + c.deleted_at, + c.profile_id, + c.created_by, + c.locked_by, + c.features, + c.metadata, + c.migrations, + c.version, + c.file_id") + +(def ^:private sql:snapshot-visible + "AND CASE WHEN c.created_by = 'user' + THEN c.deleted_at IS NULL + WHEN c.created_by = 'system' + THEN c.deleted_at IS NULL OR c.deleted_at >= ?::timestamptz + END") + (def ^:private sql:get-snapshot-without-data (str "WITH snapshots AS (" sql:snapshots ")" - "SELECT c.id, - c.label, - c.revn, - c.created_at, - c.modified_at, - c.deleted_at, - c.profile_id, - c.created_by, - c.locked_by, - c.features, - c.metadata, - c.migrations, - c.version, - c.file_id - FROM snapshots AS c - WHERE c.id = ? - AND CASE WHEN c.created_by = 'user' - THEN c.deleted_at IS NULL - WHEN c.created_by = 'system' - THEN c.deleted_at IS NULL OR c.deleted_at >= ?::timestamptz - END")) + "SELECT " sql:snapshot-columns + " FROM snapshots AS c + WHERE c.file_id = ? + AND c.id = ? " sql:snapshot-visible)) (defn get-minimal-snapshot - [cfg snapshot-id] + "Get the minimal snapshot info (without the data blob). The query filters + by `file_id`, which lets the planner prune the partitions of `file_change` + when it chooses an index scan. Raises `:snapshot-not-found` when there is + no matching snapshot." + [cfg file-id snapshot-id] (let [now (ct/now)] - (-> (db/get-with-sql cfg [sql:get-snapshot-without-data snapshot-id now] - {::db/remove-deleted false}) - (decode-snapshot)))) + (or (-> (db/get-with-sql cfg [sql:get-snapshot-without-data file-id snapshot-id now] + {::db/remove-deleted false + ::db/throw-if-not-exists false}) + (decode-snapshot)) + (ex/raise :type :not-found + :code :snapshot-not-found + :hint "unable to find snapshot with the provided id" + :snapshot-id snapshot-id + :file-id file-id)))) (def ^:private sql:get-snapshot (str sql:snapshots " AND c.file_id = ? - AND c.id = ? - AND CASE WHEN c.created_by = 'user' - THEN (c.deleted_at IS NULL) - WHEN c.created_by = 'system' - THEN (c.deleted_at IS NULL OR c.deleted_at >= ?::timestamptz) - END")) + AND c.id = ? " + sql:snapshot-visible)) (defn get-snapshot "Get a fully decoded snapshot for read-only preview or restoration. - Returns the snapshot map with decoded :data field." + Returns the snapshot map with decoded :data field, or nil when the file has + no snapshot with that id." [cfg file-id snapshot-id] (let [now (ct/now)] - (->> (db/get-with-sql cfg [sql:get-snapshot file-id snapshot-id now] - {::db/remove-deleted false}) - (decode-snapshot) - (fdata/resolve-file-data cfg) - (fdata/decode-file-data cfg)))) + (some->> (db/get-with-sql cfg [sql:get-snapshot file-id snapshot-id now] + {::db/remove-deleted false + ::db/throw-if-not-exists false}) + (decode-snapshot) + (fdata/resolve-file-data cfg) + (fdata/decode-file-data cfg)))) (def ^:private sql:get-visible-snapshots (str "WITH " @@ -267,6 +282,7 @@ (-> snapshot (update :features into-array) (update :migrations into-array) + (assoc :kind fchanges/snapshot-kind) (assoc :updated-at created-at) (assoc :profile-id profile-id) (assoc :session-id session-id) @@ -313,7 +329,10 @@ (-> (db/update! cfg :file-change {:label label :created-by "user" - :updated-at modified-at + ;; the partition runbook uses updated_at as the + ;; watermark of its delta passes: a rename has to move + ;; it, unlike file_data.modified_at above + :updated-at (ct/now) :deleted-at nil} {:file-id file-id :id id} @@ -346,7 +365,7 @@ (when-not snapshot (ex/raise :type :not-found :code :snapshot-not-found - :hint "unable to find snapshot with the provided label" + :hint "unable to find snapshot with the provided id" :snapshot-id snapshot-id :file-id file-id)) @@ -426,7 +445,8 @@ :file-id file-id :id id}}) (db/update! cfg :file-change - {:deleted-at deleted-at} + {:deleted-at deleted-at + :updated-at (ct/now)} {:id id :file-id file-id} {::db/return-keys false}) true) @@ -435,19 +455,25 @@ (str sql:snapshots " AND c.file_id = ?")) (defn lock-by! - [conn id profile-id] + [conn file-id id profile-id] + (assert (uuid? file-id) "missing file-id") + (assert (uuid? id) "missing id") (-> (db/update! conn :file-change - {:locked-by profile-id} - {:id id} + {:locked-by profile-id + :updated-at (ct/now)} + {:id id :file-id file-id} {::db/return-keys false}) (db/get-update-count) (pos?))) (defn unlock! - [conn id] + [conn file-id id] + (assert (uuid? file-id) "missing file-id") + (assert (uuid? id) "missing id") (-> (db/update! conn :file-change - {:locked-by nil} - {:id id} + {:locked-by nil + :updated-at (ct/now)} + {:id id :file-id file-id} {::db/return-keys false}) (db/get-update-count) (pos?))) diff --git a/backend/src/app/migrations.clj b/backend/src/app/migrations.clj index 9e1a91b636..b886816a1d 100644 --- a/backend/src/app/migrations.clj +++ b/backend/src/app/migrations.clj @@ -538,7 +538,10 @@ :fn (mg/resource "app/migrations/sql/0157-migrate-legacy-file-data.sql")} {:name "0158-del-forbidden-file-library-rels" - :fn (mg/resource "app/migrations/sql/0158-del-forbidden-file-library-rels.sql")}]) + :fn (mg/resource "app/migrations/sql/0158-del-forbidden-file-library-rels.sql")} + + {:name "0158-partition-file-change" + :fn (mg/resource "app/migrations/sql/0158-partition-file-change.sql")}]) (defn apply-migrations! [pool name migrations] diff --git a/backend/src/app/migrations/sql/0158-partition-file-change.sql b/backend/src/app/migrations/sql/0158-partition-file-change.sql new file mode 100644 index 0000000000..f44b12fef2 --- /dev/null +++ b/backend/src/app/migrations/sql/0158-partition-file-change.sql @@ -0,0 +1,251 @@ +-- Partition `file_change` by `file_id`. +-- +-- The table grows with every file change and its TOAST holds the blobs of +-- the changes, so it is the biggest relation of the database. Partitioning it +-- by `file_id` lets the queries that filter by file prune to a single +-- partition, and lets the join with `file_data` (already partitioned with the +-- same modulus) be resolved partition by partition. +-- +-- The primary key has to include the partition key, so it becomes +-- `(file_id, id)`. Every access by id alone would lose its index, so the +-- access paths of the application filter by `file_id` too. +-- +-- The new table is created with the final schema: the columns that nobody +-- used anymore are gone (`data_ref_id` and `data_backend`, dead since the +-- file storage refactor), the `data` column is kept as-is (the `legacy-db` +-- file data backend reads old payloads from it: migration +-- 0157-migrate-legacy-file-data moved the existing blobs), and the new +-- `kind` column tells the snapshots and the revisions apart. +-- +-- The `kind` column is a reserve: the queries still tell a snapshot from a +-- revision by its label (`label IS NOT NULL`), and `kind` takes over when +-- they are rewritten. That is why no constraint ties the two together: the +-- day `kind` is the source of truth, the constraint would have to go. +-- +-- The old table is kept aside as `file_change_legacy`: a frozen copy of the +-- previous layout. It is dropped by hand, days later (or right here, when it +-- is empty: on a new instance there is nothing to keep). +-- +-- This migration is not meant to run on a big instance: the copy of the +-- rows of the table happens inside the transaction of the migration, and +-- the pool sets a 5 minute statement timeout. On those instances mark the +-- migration as applied and partition the table by hand, with the procedure +-- of `operations/2.20.0-file-change-partition.md`. +-- +-- It assumes the service is stopped (a stop-start upgrade): it renames the +-- table, and a process of the previous version would keep writing to +-- `file_change_legacy` through its server-side prepared statements, which +-- hold the OID of the table and survive the rename. A rolling deployment is +-- the scenario of the runbook, not of this path. +-- +-- The GRANTs of the table are not carried over: this creates a new table, +-- so an instance with custom grants on `file_change` has to re-apply them +-- after the upgrade. + +-- 0. The guard for the big instances, before anything is touched: a copy +-- that does not fit keeps the backend down in a restart loop, and the +-- operator has to learn it from a clear message instead of from a +-- `statement timeout`. Both numbers are catalog reads, not table scans. +-- +-- The row count comes from `pg_class.reltuples`, which is -1 on a +-- relation that was never analyzed (a restore without `--analyze`, for +-- instance). Analyze first so the number is real; on a table that fails +-- the guard this costs seconds per boot attempt, not a table scan. +-- +-- The copy, the index builds and the `ANALYZE` of an instance that +-- passes the guard can take longer than the 5 minutes of the pool. The +-- timeout is bounded, so a wrong estimate still fails instead of hanging +-- the boot, and it covers the `ANALYZE`: on a cold, large table it can +-- outlast the pool too, and the guard has to be the one that speaks. +SET LOCAL statement_timeout = '30min'; + +ANALYZE file_change; + +DO $$ +BEGIN + IF pg_total_relation_size('file_change'::regclass) > 10::bigint * 1024 * 1024 * 1024 + OR (SELECT reltuples FROM pg_class WHERE oid = 'file_change'::regclass) > 2000000 + THEN + RAISE EXCEPTION + 'file_change is too large for the automatic partition migration: mark 0158-partition-file-change as applied and follow operations/2.20.0-file-change-partition.md'; + END IF; +END $$; + +-- 1. The legacy blobs of the snapshots were already moved to `file_data` +-- by migration 0157-migrate-legacy-file-data (on the big instances, by +-- hand, with its runbook), so there is nothing to copy here. The `data` +-- column travels with the row in step 4 below: the `legacy-db` backend +-- still reads old payloads from it. + +-- 2. The current table becomes the legacy one. The names of its indexes and +-- constraints live in the schema, not in the table, so they are renamed +-- with a `_legacy` suffix to free the canonical names. +ALTER TABLE file_change RENAME TO file_change_legacy; + +ALTER TABLE file_change_legacy RENAME CONSTRAINT file_change_pkey + TO file_change_pkey_legacy; +ALTER TABLE file_change_legacy RENAME CONSTRAINT file_change_file_id_fkey + TO file_change_file_id_fkey_legacy; +ALTER TABLE file_change_legacy RENAME CONSTRAINT file_change_locked_by_fkey + TO file_change_locked_by_fkey_legacy; +ALTER TABLE file_change_legacy RENAME CONSTRAINT file_change_profile_id_fkey + TO file_change_profile_id_fkey_legacy; + +ALTER INDEX file_change__data_ref_id__idx + RENAME TO file_change__data_ref_id__idx_legacy; +ALTER INDEX file_change__deleted_at__idx + RENAME TO file_change__deleted_at__idx_legacy; +ALTER INDEX file_change__file_id__revn__idx + RENAME TO file_change__file_id__revn__idx_legacy; +ALTER INDEX file_change__locked_by__idx + RENAME TO file_change__locked_by__idx_legacy; +ALTER INDEX file_change__profile_id__idx + RENAME TO file_change__profile_id__idx_legacy; +ALTER INDEX file_change__system_snapshots__idx + RENAME TO file_change__system_snapshots__idx_legacy; + +-- This one comes from another branch, so an instance may not have it. +ALTER INDEX IF EXISTS file_change__user_snapshots__idx + RENAME TO file_change__user_snapshots__idx_legacy; + +-- 3. The new table, partitioned with the same bounds as `file_data`, so both +-- relations can be joined partition by partition. +CREATE TABLE file_change ( + file_id uuid NOT NULL, + id uuid NOT NULL DEFAULT gen_random_uuid(), + created_at timestamptz NOT NULL DEFAULT now(), + session_id uuid, + revn bigint NOT NULL DEFAULT 0, + data bytea, + changes bytea, + profile_id uuid, + features text[], + label text, + version integer, + created_by text NOT NULL DEFAULT 'system', + updated_at timestamptz DEFAULT now(), + deleted_at timestamptz, + migrations text[], + locked_by uuid, + kind smallint NOT NULL, + CONSTRAINT file_change_kind_check CHECK (kind IN (1, 2)) +) PARTITION BY HASH (file_id); + +-- the columns keep the storage of PostgreSQL by default (migration +-- 0156-reset-pg-storage-defaults dropped the explicit EXTERNAL overrides) + +COMMENT ON COLUMN file_change.kind IS + 'The kind of the row: 1 is a snapshot (it has a label and its data lives in file_data) and 2 is a revision (it has no label and it holds the changes).'; +COMMENT ON COLUMN file_change.locked_by IS + 'Profile ID of user who has locked this version. Only the creator can lock/unlock their own versions. Locked versions cannot be deleted by others.'; + +CREATE TABLE file_change_00 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 0); +CREATE TABLE file_change_01 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 1); +CREATE TABLE file_change_02 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 2); +CREATE TABLE file_change_03 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 3); +CREATE TABLE file_change_04 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 4); +CREATE TABLE file_change_05 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 5); +CREATE TABLE file_change_06 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 6); +CREATE TABLE file_change_07 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 7); +CREATE TABLE file_change_08 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 8); +CREATE TABLE file_change_09 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 9); +CREATE TABLE file_change_10 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 10); +CREATE TABLE file_change_11 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 11); +CREATE TABLE file_change_12 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 12); +CREATE TABLE file_change_13 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 13); +CREATE TABLE file_change_14 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 14); +CREATE TABLE file_change_15 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 15); + +-- 4. The rows move to the partitioned table. The column list is explicit +-- because the legacy table has columns that the new one does not have +-- (`data_ref_id` and `data_backend`). +-- The kind comes from the label: a row with a label is a snapshot, a row +-- without one is a revision. +INSERT INTO file_change (id, file_id, created_at, session_id, revn, data, changes, + profile_id, features, label, version, created_by, + updated_at, deleted_at, migrations, locked_by, kind) +SELECT id, file_id, created_at, session_id, revn, data, changes, + profile_id, features, label, version, created_by, + updated_at, deleted_at, migrations, locked_by, + CASE WHEN label IS NOT NULL THEN 1 ELSE 2 END + FROM file_change_legacy; + +-- 5. The primary key includes the partition key now. +ALTER TABLE file_change ADD PRIMARY KEY (file_id, id); + +-- 6. The foreign keys. +ALTER TABLE file_change ADD FOREIGN KEY (file_id) + REFERENCES file(id) DEFERRABLE; +ALTER TABLE file_change ADD FOREIGN KEY (locked_by) + REFERENCES profile(id) ON DELETE SET NULL DEFERRABLE; +ALTER TABLE file_change ADD FOREIGN KEY (profile_id) + REFERENCES profile(id) ON DELETE SET NULL DEFERRABLE; + +-- 7. The indexes, with the canonical names. The one of the deleted rows now +-- carries the `file_id`, so the garbage collector can read the rows it has +-- to delete without touching the table. +-- +-- There is no index for the system snapshots: the old one filtered them by +-- `data IS NOT NULL`, and with the payload living in `file_data` the +-- column no longer tells snapshots apart. The new one could only filter +-- them by `deleted_at IS NULL`, but the system snapshots always carry a +-- deletion date in the future, so it would never match a row (and +-- the query that would use it filters that date with an "or", which no +-- partial index can serve). +CREATE INDEX file_change__deleted_at__idx + ON file_change (deleted_at, file_id, id) WHERE deleted_at IS NOT NULL; +CREATE INDEX file_change__file_id__revn__idx + ON file_change (file_id, revn); +CREATE INDEX file_change__locked_by__idx + ON file_change (locked_by) WHERE locked_by IS NOT NULL; +CREATE INDEX file_change__profile_id__idx + ON file_change (profile_id) WHERE profile_id IS NOT NULL; +CREATE INDEX file_change__user_snapshots__idx + ON file_change (file_id) + WHERE label IS NOT NULL + AND created_by = 'user' + AND deleted_at IS NULL; + +-- 8. Statistics for the new partitions. The `ANALYZE` of step 0 ran against +-- the old table, before the copy: without this one the planner starts +-- with empty statistics on the partitions and picks plans by guesswork +-- until autovacuum gets to them. The backend starts serving right after +-- this migration, so it is the worst moment to leave them unanalyzed. +ANALYZE file_change; + +-- 9. The legacy table is kept as a frozen copy of the previous layout. It is +-- dropped by hand, days after the cutover: +-- DROP TABLE file_change_legacy; +-- +-- Its foreign key on `file_id` goes away with the copy: it is `NO +-- ACTION`, so while the legacy table has rows, the objects GC cannot +-- delete a file (it deletes the rows of the file first, and the legacy +-- ones are not its business). The legacy table is frozen: nothing writes +-- to it again. +ALTER TABLE file_change_legacy + DROP CONSTRAINT file_change_file_id_fkey_legacy; + +-- On a new instance there is nothing to go back to, so the empty table +-- is dropped here. +DO $$ +BEGIN + IF NOT EXISTS (SELECT 1 FROM file_change_legacy) THEN + DROP TABLE file_change_legacy; + END IF; +END $$; diff --git a/backend/src/app/rpc/commands/files.clj b/backend/src/app/rpc/commands/files.clj index b2b465cf78..b355bffc3d 100644 --- a/backend/src/app/rpc/commands/files.clj +++ b/backend/src/app/rpc/commands/files.clj @@ -1401,7 +1401,9 @@ ;; for ever. Rows that still carry a legacy payload without a label are ;; garbage that no read path reaches; 0157-migrate-legacy-file-data deletes ;; them, so they must keep their date and go with the collector. - "UPDATE file_change SET deleted_at = null + ;; Every write moves `updated_at`: the delta passes copy the rows whose + ;; `updated_at` changed, so a restore that left it stale could be missed. + "UPDATE file_change SET deleted_at = null, updated_at = now() WHERE file_id = ANY(?::uuid[]) AND label IS NOT NULL") diff --git a/backend/src/app/rpc/commands/files_snapshot.clj b/backend/src/app/rpc/commands/files_snapshot.clj index 80be2cd189..ede921d523 100644 --- a/backend/src/app/rpc/commands/files_snapshot.clj +++ b/backend/src/app/rpc/commands/files_snapshot.clj @@ -52,6 +52,7 @@ "Retrieve a file bundle with data from a specific snapshot for read-only preview. Does not modify any database state." {::doc/added "2.16" + ::doc/changes [["2.20" "An unknown snapshot is reported with the :snapshot-not-found error code"]] ::sm/params schema:get-file-snapshot ::sm/result files/schema:file-with-permissions ::db/transaction true} @@ -196,7 +197,7 @@ (ex/raise :type :conflict :code :file-modified :hint "the file was modified during the restore process, please retry"))) - (let [snapshot (fsnap/get-minimal-snapshot cfg id) + (let [snapshot (fsnap/get-minimal-snapshot cfg file-id id) vern (fsnap/restore! cfg file-id id)] ;; Peers name the restored version by its label when a ;; user named it, else by its date @@ -214,28 +215,32 @@ (def ^:private schema:update-file-snapshot [:map {:title "update-file-snapshot"} + [:file-id ::sm/uuid] [:id ::sm/uuid] [:label ::sm/text]]) (sv/defmethod ::update-file-snapshot {::doc/added "1.20" + ::doc/changes [["2.20" "The :file-id param is now required and an unknown snapshot answers :snapshot-not-found"]] ::sm/params schema:update-file-snapshot ::db/transaction true} - [{:keys [::db/conn]} {:keys [::rpc/profile-id id label]}] - (let [snapshot (fsnap/get-minimal-snapshot conn id)] + [{:keys [::db/conn]} {:keys [::rpc/profile-id file-id id label]}] + (let [snapshot (fsnap/get-minimal-snapshot conn file-id id)] (files/check-edition-permissions! conn profile-id (:file-id snapshot)) (fsnap/update! conn (assoc snapshot :label label)))) (def ^:private schema:remove-file-snapshot [:map {:title "remove-file-snapshot"} + [:file-id ::sm/uuid] [:id ::sm/uuid]]) (sv/defmethod ::delete-file-snapshot {::doc/added "1.20" + ::doc/changes [["2.20" "The :file-id param is now required and an unknown snapshot answers :snapshot-not-found"]] ::sm/params schema:remove-file-snapshot ::db/transaction true} - [{:keys [::db/conn] :as cfg} {:keys [::rpc/profile-id id]}] - (let [snapshot (fsnap/get-minimal-snapshot conn id)] + [{:keys [::db/conn] :as cfg} {:keys [::rpc/profile-id file-id id]}] + (let [snapshot (fsnap/get-minimal-snapshot conn file-id id)] (files/check-edition-permissions! conn profile-id (:file-id snapshot)) (when (not= (:created-by snapshot) "user") @@ -263,14 +268,16 @@ (def ^:private schema:lock-file-snapshot [:map {:title "lock-file-snapshot"} + [:file-id ::sm/uuid] [:id ::sm/uuid]]) (sv/defmethod ::lock-file-snapshot {::doc/added "1.20" + ::doc/changes [["2.20" "The :file-id param is now required and an unknown snapshot answers :snapshot-not-found"]] ::sm/params schema:lock-file-snapshot ::db/transaction true} - [{:keys [::db/conn]} {:keys [::rpc/profile-id id]}] - (let [snapshot (fsnap/get-minimal-snapshot conn id)] + [{:keys [::db/conn]} {:keys [::rpc/profile-id file-id id]}] + (let [snapshot (fsnap/get-minimal-snapshot conn file-id id)] (files/check-edition-permissions! conn profile-id (:file-id snapshot)) (when (not= (:created-by snapshot) "user") @@ -298,18 +305,20 @@ :profile-id profile-id :locked-by (:locked-by snapshot))) - (fsnap/lock-by! conn id profile-id))) + (fsnap/lock-by! conn (:file-id snapshot) id profile-id))) (def ^:private schema:unlock-file-snapshot [:map {:title "unlock-file-snapshot"} + [:file-id ::sm/uuid] [:id ::sm/uuid]]) (sv/defmethod ::unlock-file-snapshot {::doc/added "1.20" + ::doc/changes [["2.20" "The :file-id param is now required and an unknown snapshot answers :snapshot-not-found"]] ::sm/params schema:unlock-file-snapshot ::db/transaction true} - [{:keys [::db/conn]} {:keys [::rpc/profile-id id]}] - (let [snapshot (fsnap/get-minimal-snapshot conn id)] + [{:keys [::db/conn]} {:keys [::rpc/profile-id file-id id]}] + (let [snapshot (fsnap/get-minimal-snapshot conn file-id id)] (files/check-edition-permissions! conn profile-id (:file-id snapshot)) (when (not= (:created-by snapshot) "user") @@ -336,4 +345,4 @@ :snapshot-id id :profile-id profile-id)) - (fsnap/unlock! conn id))) + (fsnap/unlock! conn (:file-id snapshot) id))) diff --git a/backend/src/app/rpc/commands/files_update.clj b/backend/src/app/rpc/commands/files_update.clj index 266214ec27..6cc2846e9f 100644 --- a/backend/src/app/rpc/commands/files_update.clj +++ b/backend/src/app/rpc/commands/files_update.clj @@ -21,6 +21,7 @@ [app.config :as cf] [app.db :as db] [app.features.fdata :as fdata] + [app.features.file-changes :as fchanges] [app.features.file-snapshots :as fsnap] [app.features.logical-deletion :as ldel] [app.http.errors :as errors] @@ -371,6 +372,7 @@ (when repair (db/insert! conn :file-change {:id (uuid/next) + :kind fchanges/revision-kind :session-id repair-session-id :profile-id profile-id :created-at timestamp @@ -379,7 +381,6 @@ :file-id (:id file) :revn (:revn repair) :version (:version file) - :features (into-array (:features file)) :changes (blob/encode (:changes repair))} {::db/return-keys false}) (l/wrn :hint "file auto-repaired" @@ -391,8 +392,13 @@ ;; Insert change (xlog) with deleted_at in a future data for ;; make them automatically eleggible for GC once they expires + ;; + ;; The revision carries only the delta of the save: the features of the + ;; file are not repeated here (nobody reads them from a revision, and + ;; they are duplicated on every save). (db/insert! conn :file-change {:id (uuid/next) + :kind fchanges/revision-kind :session-id session-id :profile-id profile-id :created-at change-created-at @@ -401,7 +407,6 @@ :file-id (:id file) :revn (:revn file) :version (:version file) - :features (into-array (:features file)) :changes (blob/encode changes)} {::db/return-keys false}) diff --git a/backend/src/app/rpc/quotes.clj b/backend/src/app/rpc/quotes.clj index ac79505d16..1086c2423e 100644 --- a/backend/src/app/rpc/quotes.clj +++ b/backend/src/app/rpc/quotes.clj @@ -423,6 +423,8 @@ (sm/lazy-validator schema:snapshots-per-file)) (def ^:private sql:get-snapshots-per-file + ;; A version is told apart by its label, and the partial index with the same + ;; predicate answers the count without touching the table. "SELECT count(*) AS total FROM file_change AS fc WHERE fc.file_id = ? @@ -454,11 +456,16 @@ (sm/lazy-validator schema:snapshots-per-team)) (def ^:private sql:get-snapshots-per-team + ;; The files of the team are resolved first, so the count filters by + ;; `file_id`: the planner can prune the partitions of `file_change` when it + ;; chooses an index scan (with a sequential scan it reads all of them). A + ;; version is told apart by its label. "SELECT count(*) AS total FROM file_change AS fc - JOIN file AS f ON (f.id = fc.file_id) - JOIN project AS p ON (p.id = f.project_id) - WHERE p.team_id = ? + WHERE fc.file_id = ANY (SELECT f.id + FROM file AS f + JOIN project AS p ON (p.id = f.project_id) + WHERE p.team_id = ?) AND fc.created_by = 'user' AND fc.deleted_at IS NULL AND fc.label IS NOT NULL") diff --git a/backend/src/app/scripts/decode_file_data.clj b/backend/src/app/scripts/decode_file_data.clj index b7d775eafc..a732a34fc1 100644 --- a/backend/src/app/scripts/decode_file_data.clj +++ b/backend/src/app/scripts/decode_file_data.clj @@ -39,6 +39,7 @@ `[.data[\"pages-index\"][] | {id, name}]`, etc." (:require [app.binfile.common :as bfc] + [app.common.exceptions :as ex] [app.common.json :as json] [app.db :as db] [app.features.file-snapshots :as fsnap] @@ -183,16 +184,23 @@ (defn get-snapshot-json "Return a snapshot as JSON (realized). For nREPL. - Takes the same `format` and `output` as `get-file-json`." + Takes the same `format` and `output` as `get-file-json`. It raises when the + file has no snapshot with that id." [system file-id snapshot-id & {:keys [format output] :or {format :full}}] (let [snapshot (db/run! system (fn [cfg] (fsnap/get-snapshot cfg file-id snapshot-id))) - data (case format - :full snapshot - :data (:data snapshot) - :summary (summarize-data (:data snapshot)) - :pages (:pages (summarize-data (:data snapshot))))] + data (if snapshot + (case format + :full snapshot + :data (:data snapshot) + :summary (summarize-data (:data snapshot)) + :pages (:pages (summarize-data (:data snapshot)))) + (ex/raise :type :not-found + :code :snapshot-not-found + :hint "unable to find snapshot with the provided id" + :snapshot-id snapshot-id + :file-id file-id))] (write-output (data->json data) output))) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; diff --git a/backend/src/app/tasks/delete_object.clj b/backend/src/app/tasks/delete_object.clj index 87cacd7a53..b45e0d8ca2 100644 --- a/backend/src/app/tasks/delete_object.clj +++ b/backend/src/app/tasks/delete_object.clj @@ -53,7 +53,8 @@ :deleted-at (ct/format-inst deleted-at)) (db/update! conn :file-change - {:deleted-at deleted-at} + {:deleted-at deleted-at + :updated-at (ct/now)} {:id id :file-id file-id} {::db/return-keys false}) @@ -93,7 +94,8 @@ ;; lifetime; the collector would still take it, just much later than it ;; should. (db/update! conn :file-change - {:deleted-at deleted-at} + {:deleted-at deleted-at + :updated-at (ct/now)} {:file-id id :deleted-at nil} {::db/return-keys false}) diff --git a/backend/src/app/tasks/objects_gc.clj b/backend/src/app/tasks/objects_gc.clj index 1fca7bf321..80f212a6b1 100644 --- a/backend/src/app/tasks/objects_gc.clj +++ b/backend/src/app/tasks/objects_gc.clj @@ -673,41 +673,48 @@ {:fetch-size 5})))))) (def ^:private sql:get-file-change - "SELECT id FROM file_change + "SELECT id, file_id FROM file_change WHERE deleted_at IS NOT NULL AND deleted_at <= ? ORDER BY deleted_at ASC") (def ^:private sql:lock-file-change - "SELECT id, file_id, deleted_at FROM file_change - WHERE id = ANY(?::uuid[]) + "SELECT file_id, id, deleted_at FROM file_change + WHERE file_id = ? + AND id = ANY(?::uuid[]) AND deleted_at IS NOT NULL AND deleted_at <= ? FOR UPDATE SKIP LOCKED") (defn- delete-file-changes-chunk - [cfg ids] + [cfg rows] (try (db/tx-run! cfg (fn [{:keys [::db/conn ::threshold]}] (db/exec-one! conn ["SET LOCAL rules.deletion_protection TO off"]) - (let [ids (db/create-array conn "uuid" ids)] - (->> (db/exec! conn [sql:lock-file-change ids threshold]) - (reduce (fn [total {:keys [id file-id deleted-at]}] - (l/trc :obj "file-change" - :id (str id) - :file-id (str file-id) - :deleted-at (ct/format-inst deleted-at)) + ;; The rows are locked per file, with a constant `file_id`, + ;; so the planner prunes to the partition that holds each + ;; one instead of scanning all of them for a tuple `IN`. + (->> (group-by :file-id rows) + (mapcat (fn [[file-id file-rows]] + (let [ids (db/create-array conn "uuid" (map :id file-rows))] + (db/exec! conn [sql:lock-file-change file-id ids threshold])))) + (reduce (fn [total {:keys [id file-id deleted-at]}] + (l/trc :obj "file-change" + :id (str id) + :file-id (str file-id) + :deleted-at (ct/format-inst deleted-at)) - (let [affected (-> (db/delete! conn :file-change {:id id}) - (db/get-update-count))] - (+ total affected))) - 0))))) + (let [affected (-> (db/delete! conn :file-change + {:id id :file-id file-id}) + (db/get-update-count))] + (+ total affected))) + 0)))) (catch Throwable cause (l/err :obj "file-change" :hint "error on chunk deletion, skipping it" - :chunk ids + :chunk rows :cause cause) 0))) @@ -720,7 +727,10 @@ (+ total deleted))))] (db/tx-run! (assoc cfg ::db/read-only true) (fn [{:keys [::db/conn]}] - (transduce (comp (map :id) (partition-all chunk-size)) rf 0 + (transduce (comp (map (fn [row] {:id (:id row) + :file-id (:file-id row)})) + (partition-all chunk-size)) + rf 0 (db/plan conn [sql:get-file-change threshold] {:fetch-size 5})))))) diff --git a/backend/test/backend_tests/db_test.clj b/backend/test/backend_tests/db_test.clj index aa6732ead5..b0f20cb256 100644 --- a/backend/test/backend_tests/db_test.clj +++ b/backend/test/backend_tests/db_test.clj @@ -80,3 +80,11 @@ (t/deftest uuid->hash-code-stable-for-zero-uuid (t/is (= 3659997967308761462 (db/uuid->hash-code uuid/zero)))) + +(t/deftest pool-enables-the-partitionwise-join + ;; The pool sets the GUC on every connection, so the planner can resolve a + ;; join between two partitioned tables pairwise, partition by partition. + (let [row (db/exec-one! th/*pool* + ["SELECT current_setting('enable_partitionwise_join') AS partitionwise_join"])] + + (t/is (= "on" (:partitionwise-join row))))) diff --git a/backend/test/backend_tests/fdata_resolve_test.clj b/backend/test/backend_tests/fdata_resolve_test.clj new file mode 100644 index 0000000000..e9d777d900 --- /dev/null +++ b/backend/test/backend_tests/fdata_resolve_test.clj @@ -0,0 +1,42 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS INC Sucursal en España SL + +(ns backend-tests.fdata-resolve-test + "Resolution of file data by backend (`app.features.fdata/resolve-file-data`). + + A row without a `file_data` twin carries no backend: the read queries mark + it with `coalesce(fd.backend, 'legacy-db')` and hand its blob over as + `:legacy-data`. The `legacy-db` backend is still supported, so that case + has to resolve to the blob instead of nil." + (:require + [app.features.fdata :as fdata] + [clojure.test :as t])) + +(t/deftest legacy-db-resolves-the-legacy-blob + (let [blob (vec (.getBytes "legacy-blob" "UTF-8")) + file {:backend "legacy-db" + :legacy-data (byte-array blob)} + out (fdata/resolve-file-data nil file)] + (t/is (= blob (vec (:data out)))) + (t/is (not (contains? out :legacy-data))))) + +(t/deftest missing-backend-defaults-to-legacy-db + ;; The queries leave `:backend` nil when the `LEFT JOIN` with `file_data` + ;; finds no twin; the dispatch default keeps that row on `legacy-db`. + (let [blob (vec (.getBytes "legacy-blob" "UTF-8")) + file {:legacy-data (byte-array blob)} + out (fdata/resolve-file-data nil file)] + (t/is (= blob (vec (:data out)))) + (t/is (not (contains? out :legacy-data))))) + +(t/deftest db-backend-drops-the-legacy-key + (let [data (vec (.getBytes "db-blob" "UTF-8")) + file {:backend "db" + :data (byte-array data) + :legacy-data (byte-array (vec (.getBytes "stale" "UTF-8")))} + out (fdata/resolve-file-data nil file)] + (t/is (= data (vec (:data out)))) + (t/is (not (contains? out :legacy-data))))) diff --git a/backend/test/backend_tests/fdata_upsert_test.clj b/backend/test/backend_tests/fdata_upsert_test.clj new file mode 100644 index 0000000000..62fd2913c2 --- /dev/null +++ b/backend/test/backend_tests/fdata_upsert_test.clj @@ -0,0 +1,55 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns backend-tests.fdata-upsert-test + "Covers the invariants of `fdata/upsert!` that the partition runbook relies + on." + + (:require + [app.common.time :as ct] + [app.common.uuid :as uuid] + [app.db :as db] + [app.features.fdata :as fdata] + [app.features.file-changes :as fchanges] + [backend-tests.helpers :as th] + [clojure.test :as t])) + +(t/use-fixtures :once th/state-init) +(t/use-fixtures :each th/database-reset) + +(t/deftest legacy-db-snapshot-moves-the-updated-at-watermark + ;; The runbook of the partition picks the rows changed since the bulk copy + ;; with `updated_at`, so every mutation of `file_change` has to move it. The + ;; `legacy-db` snapshot path writes the payload into the row, and it used to + ;; leave the watermark behind. + (let [profile (th/create-profile* 1) + file (th/create-file* 1 {:profile-id (:id profile) + :project-id (:default-project-id profile)}) + id (uuid/next) + past (ct/in-past "2h") + payload (byte-array [1 2 3])] + + (th/db-insert! :file-change + {:id id + :file-id (:id file) + :revn 1 + :label "v1" + :kind fchanges/snapshot-kind + :updated-at past}) + + (t/is (true? (fdata/upsert! th/*system* + {:id id + :file-id (:id file) + :type "snapshot" + :backend "legacy-db" + :data payload}))) + + (let [row (th/db-get :file-change {:id id} {::db/remove-deleted false})] + (t/testing "the payload lands in the legacy column" + (t/is (= (seq payload) (seq ^bytes (:data row))))) + + (t/testing "the watermark moves to the moment of the mutation" + (t/is (pos? (compare (inst-ms (:updated-at row)) (inst-ms past)))))))) diff --git a/backend/test/backend_tests/file_change_retention_test.clj b/backend/test/backend_tests/file_change_retention_test.clj index 3929e0b129..83246e9226 100644 --- a/backend/test/backend_tests/file_change_retention_test.clj +++ b/backend/test/backend_tests/file_change_retention_test.clj @@ -10,7 +10,7 @@ A file has two kinds of row. Durable history: the snapshots, which a restore brings back. And ephemeral revisions: the xlog of an edit, born already scheduled for deletion one hour later, which nobody needs once the sessions - that were going to read them are gone. + that were going to read them are gone. The `kind` column tells them apart. The invariant under test is that a deletion date is assigned once and never moved: restoring a file must not unmark a revision, and deleting a file must @@ -21,6 +21,7 @@ [app.common.time :as ct] [app.common.uuid :as uuid] [app.db :as db] + [app.features.file-changes :as fchanges] [app.rpc :as-alias rpc] [app.rpc.commands.files :as files] [app.util.blob :as blob] @@ -64,6 +65,7 @@ :version 0 :created-by (or created-by "system") :label label + :kind (if label fchanges/snapshot-kind fchanges/revision-kind) :data data :changes changes} {::db/return-keys false}) @@ -170,6 +172,7 @@ out (th/command! {::th/type :update-file-snapshot ::rpc/profile-id (:id profile) + :file-id file-id :id revision-id :label "renamed"})] diff --git a/backend/test/backend_tests/file_change_schema_test.clj b/backend/test/backend_tests/file_change_schema_test.clj new file mode 100644 index 0000000000..521d056a01 --- /dev/null +++ b/backend/test/backend_tests/file_change_schema_test.clj @@ -0,0 +1,110 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS INC Sucursal en España SL + +(ns backend-tests.file-change-schema-test + (:require + [app.db :as db] + [backend-tests.helpers :as th] + [clojure.test :as t])) + +(t/use-fixtures :once th/state-init) +(t/use-fixtures :each th/database-reset) + +(defn- query + "Runs a query that returns a single value and returns it." + [sql & params] + (-> (db/exec! th/*pool* (into [sql] params)) + first + (vals) + first)) + +(defn- partition-names + [] + (->> (db/exec! th/*pool* ["SELECT c.relname AS name + FROM pg_class c + JOIN pg_partition_tree('file_change') t ON t.relid = c.oid + WHERE t.isleaf + ORDER BY c.relname"]) + (mapv :name))) + +(t/deftest file-change-is-partitioned-in-sixteen-named-partitions + (t/is (= (mapv #(format "file_change_%02d" %) (range 16)) + (partition-names)))) + +(t/deftest file-change-primary-key-is-file-id-and-id + (t/is (= "PRIMARY KEY (file_id, id)" + (query "SELECT pg_get_constraintdef(oid) AS def + FROM pg_constraint + WHERE conrelid = 'file_change'::regclass AND contype = 'p'")))) + +(t/deftest file-change-has-three-foreign-keys-and-one-check + (t/is (= 3 (query "SELECT count(*) AS total + FROM pg_constraint + WHERE conrelid = 'file_change'::regclass AND contype = 'f'"))) + (t/is (= 1 (query "SELECT count(*) AS total + FROM pg_constraint + WHERE conrelid = 'file_change'::regclass AND contype = 'c'"))) + (t/is (re-find #"kind" + (query "SELECT pg_get_constraintdef(oid) AS def + FROM pg_constraint + WHERE conrelid = 'file_change'::regclass AND contype = 'c'")))) + +(t/deftest file-change-has-the-six-canonical-indexes + (t/is (= ["file_change__deleted_at__idx" + "file_change__file_id__revn__idx" + "file_change__locked_by__idx" + "file_change__profile_id__idx" + "file_change__user_snapshots__idx" + "file_change_pkey"] + (->> (db/exec! th/*pool* ["SELECT indexname AS name + FROM pg_indexes + WHERE tablename = 'file_change' + AND schemaname = current_schema() + ORDER BY indexname"]) + (mapv :name))))) + +(t/deftest file-change-keeps-the-data-column-and-a-kind-column + ;; The `data` column is kept: the `legacy-db` backend reads old payloads + ;; from it (migration 0157 moved the existing blobs). The dead + ;; `data_ref_id`/`data_backend` columns stay gone: no code read them since + ;; the file storage refactor. + (t/is (= [] + (->> (db/exec! th/*pool* ["SELECT attname AS name + FROM pg_attribute + WHERE attrelid = 'file_change'::regclass + AND attnum > 0 AND NOT attisdropped + AND attname IN ('data_ref_id', 'data_backend')"]) + (mapv :name)))) + (t/is (= ["bytea"] + (->> (db/exec! th/*pool* ["SELECT pg_catalog.format_type(atttypid, atttypmod) AS type + FROM pg_attribute + WHERE attrelid = 'file_change'::regclass + AND attname = 'data'"]) + (mapv :type)))) + (t/is (true? (query "SELECT attnotnull AS notnull + FROM pg_attribute + WHERE attrelid = 'file_change'::regclass + AND attname = 'kind'")))) + +(t/deftest inserting-a-row-without-a-kind-fails + ;; The column has no default on purpose: an insert that forgets it has to + ;; fail instead of storing the wrong kind silently. + (let [profile (th/create-profile* 1) + file (th/create-file* 1 {:profile-id (:id profile) + :project-id (:default-project-id profile)})] + (t/is (thrown? java.sql.SQLException + (db/exec! th/*pool* ["INSERT INTO file_change (file_id, id) + VALUES (?, gen_random_uuid())" + (:id file)]))))) + +(t/deftest inserting-a-row-with-an-unknown-kind-fails + (let [profile (th/create-profile* 1) + file (th/create-file* 1 {:profile-id (:id profile) + :project-id (:default-project-id profile)})] + (t/is (thrown? java.sql.SQLException + (db/exec! th/*pool* ["INSERT INTO file_change (file_id, id, kind) + VALUES (?, gen_random_uuid(), 3)" + (:id file)]))))) diff --git a/backend/test/backend_tests/migrations_legacy_file_data_test.clj b/backend/test/backend_tests/migrations_legacy_file_data_test.clj index e2a31ce2bf..5fcd1874ec 100644 --- a/backend/test/backend_tests/migrations_legacy_file_data_test.clj +++ b/backend/test/backend_tests/migrations_legacy_file_data_test.clj @@ -13,6 +13,7 @@ [app.common.time :as ct] [app.common.uuid :as uuid] [app.db :as db] + [app.features.file-changes :as fchanges] [app.util.blob :as blob] [backend-tests.helpers :as th] [clojure.java.io :as io] @@ -58,6 +59,7 @@ :revn 1 :data snap-data :label "v1" + :kind fchanges/snapshot-kind :created-at created-at :updated-at updated-at :deleted-at deleted-at}) @@ -69,7 +71,8 @@ {:id rev-id :file-id fid :revn 2 - :data rev-data}) + :data rev-data + :kind fchanges/revision-kind}) ;; Live revision (changes, no data, no label): untouched. live-id (uuid/next) @@ -77,7 +80,8 @@ {:id live-id :file-id fid :revn 3 - :changes (blob/encode [{:type :dummy}])})] + :changes (blob/encode [{:type :dummy}]) + :kind fchanges/revision-kind})] (run-migration) @@ -168,7 +172,8 @@ :file-id fid :revn 1 :label "v1" - :data stale})] + :data stale + :kind fchanges/snapshot-kind})] (run-migration) diff --git a/backend/test/backend_tests/migrations_partition_file_change_test.clj b/backend/test/backend_tests/migrations_partition_file_change_test.clj new file mode 100644 index 0000000000..d5cc29ad37 --- /dev/null +++ b/backend/test/backend_tests/migrations_partition_file_change_test.clj @@ -0,0 +1,210 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns backend-tests.migrations-partition-file-change-test + "Covers the data copy of 0158-partition-file-change: the rows of the old + table move to the partitioned one with their columns in place, the `kind` + comes from the label (1 snapshot, 2 revision), and the legacy table keeps + the rows." + + (:require + [app.common.time :as ct] + [app.common.uuid :as uuid] + [app.db :as db] + [app.util.blob :as blob] + [backend-tests.helpers :as th] + [clojure.java.io :as io] + [clojure.test :as t])) + +(t/use-fixtures :once th/state-init) +(t/use-fixtures :each th/database-reset) + +(def ^:private sql:old-schema + "CREATE TABLE file (id uuid PRIMARY KEY); + CREATE TABLE profile (id uuid PRIMARY KEY); + + CREATE TABLE file_change ( + file_id uuid NOT NULL, + id uuid NOT NULL DEFAULT gen_random_uuid(), + created_at timestamptz NOT NULL DEFAULT now(), + session_id uuid, + revn bigint NOT NULL DEFAULT 0, + data bytea, + changes bytea, + profile_id uuid, + features text[], + label text, + version integer, + created_by text NOT NULL DEFAULT 'system', + updated_at timestamptz DEFAULT now(), + deleted_at timestamptz, + migrations text[], + locked_by uuid, + data_ref_id uuid, + data_backend text, + CONSTRAINT file_change_pkey PRIMARY KEY (id), + CONSTRAINT file_change_file_id_fkey FOREIGN KEY (file_id) + REFERENCES file(id) DEFERRABLE, + CONSTRAINT file_change_locked_by_fkey FOREIGN KEY (locked_by) + REFERENCES profile(id) ON DELETE SET NULL DEFERRABLE, + CONSTRAINT file_change_profile_id_fkey FOREIGN KEY (profile_id) + REFERENCES profile(id) ON DELETE SET NULL DEFERRABLE); + + CREATE INDEX file_change__data_ref_id__idx + ON file_change (data_ref_id); + CREATE INDEX file_change__deleted_at__idx + ON file_change (deleted_at) WHERE deleted_at IS NOT NULL; + CREATE INDEX file_change__file_id__revn__idx + ON file_change (file_id, revn); + CREATE INDEX file_change__locked_by__idx + ON file_change (locked_by) WHERE locked_by IS NOT NULL; + CREATE INDEX file_change__profile_id__idx + ON file_change (profile_id) WHERE profile_id IS NOT NULL; + CREATE INDEX file_change__system_snapshots__idx + ON file_change (file_id, created_at) + WHERE created_by = 'system' AND deleted_at IS NULL; + CREATE INDEX file_change__user_snapshots__idx + ON file_change (file_id) + WHERE label IS NOT NULL AND created_by = 'user' AND deleted_at IS NULL;") + +(defn- run-migration! + [conn] + (let [sql (slurp (io/resource "app/migrations/sql/0158-partition-file-change.sql"))] + (db/exec! conn [sql]))) + +(t/deftest copies-the-rows-into-the-partitioned-table + ;; The migration runs in its own schema, with the shape of the table before + ;; it: the run below is the only place where its INSERT..SELECT runs with + ;; rows, so a swapped column or a wrong `kind` fails here and not on a real + ;; instance. + (let [schema (str "test_0158_" (subs (str (uuid/next)) 0 8)) + fid (uuid/next) + snap-id (uuid/next) + rev-id (uuid/next) + created-at (ct/in-past "2h") + updated-at (ct/in-past "1h") + deleted-at (ct/in-past "30m") + snap-data (blob/encode {:snapshot 1}) + rev-changes (blob/encode [{:type :add-obj :id (uuid/next)}])] + + (db/tx-run! th/*pool* + (fn [{:keys [::db/conn]}] + ;; the schema is dropped before the commit, and a failure + ;; rolls the whole transaction back: nothing is left behind + (db/exec! conn [(str "CREATE SCHEMA " schema)]) + (db/exec! conn [(str "SET LOCAL search_path TO " schema)]) + (db/exec! conn [sql:old-schema]) + + (db/insert! conn :file {:id fid}) + + (db/insert! conn :file-change + {:id snap-id + :file-id fid + :revn 1 + :data snap-data + :label "v1" + :version 1 + :created-by "user" + :created-at created-at + :updated-at updated-at + :deleted-at deleted-at + ;; a row that still carries the dead columns + :data-ref-id (uuid/next) + :data-backend "legacy-db"}) + + (db/insert! conn :file-change + {:id rev-id + :file-id fid + :revn 2 + :changes rev-changes + :created-at created-at + :updated-at updated-at}) + + (run-migration! conn) + + (let [rows (db/exec! conn ["SELECT id, revn, kind, label, data, changes, + created_at, updated_at, deleted_at, created_by + FROM file_change + ORDER BY revn"]) + snapshot (first rows) + revision (second rows) + legacy (db/exec! conn ["SELECT id, revn, data, changes + FROM file_change_legacy + ORDER BY revn"])] + + (t/testing "both rows are in the partitioned table" + (t/is (= 2 (count rows))) + (t/is (= [1 2] (mapv :revn rows)))) + + (t/testing "the labelled row is a snapshot with its payload" + (t/is (= 1 (:kind snapshot))) + (t/is (= "v1" (:label snapshot))) + (t/is (= "user" (:created-by snapshot))) + (t/is (= (seq snap-data) (seq ^bytes (:data snapshot)))) + (t/is (nil? (:changes snapshot))) + (t/is (= (inst-ms created-at) (inst-ms (:created-at snapshot)))) + (t/is (= (inst-ms updated-at) (inst-ms (:updated-at snapshot)))) + (t/is (= (inst-ms deleted-at) (inst-ms (:deleted-at snapshot))))) + + (t/testing "the unlabelled row is a revision with its changes" + (t/is (= 2 (:kind revision))) + (t/is (nil? (:label revision))) + (t/is (= (seq rev-changes) (seq ^bytes (:changes revision)))) + (t/is (nil? (:data revision))) + (t/is (= (inst-ms created-at) (inst-ms (:created-at revision))))) + + (t/testing "the legacy table keeps the rows for the rollback" + (t/is (= 2 (count legacy))) + (t/is (= [snap-id rev-id] (mapv :id legacy))) + (t/is (= (seq snap-data) + (seq ^bytes (:data (first legacy))))) + (t/is (= (seq rev-changes) + (seq ^bytes (:changes (second legacy))))))) + + (t/testing "the legacy table no longer points at file" + ;; Its foreign key on `file_id` is `NO ACTION`: while the table has + ;; rows, the objects GC cannot delete the files they point to. The + ;; migration drops that one and keeps the two `ON DELETE SET NULL` ones. + (t/is (= ["file_change_locked_by_fkey_legacy" + "file_change_profile_id_fkey_legacy"] + (->> (db/exec! conn ["SELECT con.conname AS name + FROM pg_constraint con + WHERE con.conrelid = 'file_change_legacy'::regclass + AND con.contype = 'f' + ORDER BY con.conname"]) + (mapv :name))))) + + (t/testing "the dead columns are not carried over" + ;; The old row carried `data_ref_id` and `data_backend`; + ;; the new table has no such columns. + (t/is (= [] + (->> (db/exec! conn ["SELECT attname AS name + FROM pg_attribute + WHERE attrelid = 'file_change'::regclass + AND attnum > 0 AND NOT attisdropped + AND attname IN ('data_ref_id', 'data_backend')"]) + (mapv :name))))) + + (db/exec! conn [(str "DROP SCHEMA " schema " CASCADE")]))))) + +(t/deftest drops-the-legacy-table-when-it-is-empty + ;; On a new instance there is nothing to keep: the migration leaves no + ;; legacy table behind. The relation is asserted on its own, so a missing + ;; table can never pass for a clean one. + (let [schema (str "test_0158_empty_" (subs (str (uuid/next)) 0 8))] + (db/tx-run! th/*pool* + (fn [{:keys [::db/conn]}] + (db/exec! conn [(str "CREATE SCHEMA " schema)]) + (db/exec! conn [(str "SET LOCAL search_path TO " schema)]) + (db/exec! conn [sql:old-schema]) + + (run-migration! conn) + + (t/is (-> (db/exec-one! conn ["SELECT to_regclass('file_change_legacy') IS NULL AS empty"]) + :empty) + "the empty legacy table is dropped") + + (db/exec! conn [(str "DROP SCHEMA " schema " CASCADE")]))))) diff --git a/backend/test/backend_tests/objects_gc_test.clj b/backend/test/backend_tests/objects_gc_test.clj index 6696a7f2f0..646650a4ca 100644 --- a/backend/test/backend_tests/objects_gc_test.clj +++ b/backend/test/backend_tests/objects_gc_test.clj @@ -8,8 +8,10 @@ (:require [app.common.logging :as l] [app.common.time :as ct] + [app.common.uuid :as uuid] [app.config :as cf] [app.db :as db] + [app.features.file-changes :as fchanges] [app.rpc :as-alias rpc] [app.storage :as sto] [app.tasks.objects-gc :as ogc] @@ -73,6 +75,69 @@ (t/is (pos? (:processed result))) (t/is (nil? (find-file (:id file)))))) +(t/deftest deletes-the-pending-file-changes + ;; The pending file changes are locked per file, with a constant `file_id`, + ;; so the rows of a file that is pending deletion are gone after the run. + (let [profile (th/create-profile* 1) + file (th/create-file* 1 {:profile-id (:id profile) + :project-id (:default-project-id profile)}) + + _ (th/db-insert! :file-change + {:file-id (:id file) + :kind fchanges/revision-kind + :deleted-at (ct/in-past "10m")}) + + result (th/run-task! :objects-gc {})] + + (t/is (pos? (:processed result))) + (t/is (nil? (th/db-get :file-change {:file-id (:id file)} + {::db/remove-deleted false}))))) + +(t/deftest deletes-the-pending-file-changes-of-several-chunks + ;; More pending rows than the chunk size: the plan walks the cursor and each + ;; chunk locks its own rows, one query per file, with a constant `file_id` + ;; so the planner prunes the partition. + (let [profile (th/create-profile* 1) + file (th/create-file* 1 {:profile-id (:id profile) + :project-id (:default-project-id profile)}) + ids (repeatedly 5 uuid/next) + _ (doseq [id ids] + (th/db-insert! :file-change + {:id id + :file-id (:id file) + :kind fchanges/revision-kind + :deleted-at (ct/in-past "10m")})) + result (th/run-task! :objects-gc {:chunk-size 2})] + + (t/is (pos? (:processed result))) + (t/is (empty? (th/db-query :file-change {:file-id (:id file)} + {::db/remove-deleted false}))))) + +(t/deftest a-chunk-spanning-several-files-deletes-all-of-them + ;; The rows of one chunk can belong to different files. Each group is locked + ;; with its own `file_id`, and every row is still deleted; buffering the + ;; groups instead of locking them would leave them behind. + (let [profile (th/create-profile* 1) + opts {:profile-id (:id profile) + :project-id (:default-project-id profile)} + files [(th/create-file* 1 opts) + (th/create-file* 2 opts)] + + _ (doseq [file files] + (dotimes [_ 2] + (th/db-insert! :file-change + {:file-id (:id file) + :kind fchanges/revision-kind + :deleted-at (ct/in-past "10m")}))) + + ;; one chunk holds the rows of both files + result (th/run-task! :objects-gc {:chunk-size 4})] + + (t/is (pos? (:processed result))) + (doseq [file files] + (t/is (empty? (th/db-query :file-change {:file-id (:id file)} + {::db/remove-deleted false})))))) + (t/deftest profile-cascade-keeps-profile-deletion-date ;; The cascade marks the owned rows with the deletion date of the profile, ;; like the RPC delete-profile flow does, not with the timestamp of the run. diff --git a/backend/test/backend_tests/restore_storage_objects_test.clj b/backend/test/backend_tests/restore_storage_objects_test.clj index 5986170b20..bb2201659d 100644 --- a/backend/test/backend_tests/restore_storage_objects_test.clj +++ b/backend/test/backend_tests/restore_storage_objects_test.clj @@ -9,6 +9,7 @@ [app.common.time :as ct] [app.common.uuid :as uuid] [app.db :as db] + [app.features.file-changes :as fchanges] [app.rpc.commands.files :as files] [app.srepl.main :as srepl] [app.storage :as sto] @@ -130,3 +131,29 @@ (srepl/restore-team! (:id team))) (t/is (nil? (:deleted-at (find-object sobject)))))) + +(t/deftest restoring-a-file-updates-its-file-changes + ;; Restoring a file brings its file changes back to life, and that mutation + ;; has to be observable through `updated_at`, the watermark used by the + ;; online partitioning copy of `file_change`. + (let [profile (th/create-profile* 1) + file (th/create-file* 1 {:profile-id (:id profile) + :project-id (:default-project-id profile)}) + + stale-at #inst "2020-01-01T00:00:00.000Z" + + _ (th/db-insert! :file-change + {:file-id (:id file) + :label "v1" + :kind fchanges/snapshot-kind + :deleted-at (ct/in-past {:days 1}) + :updated-at stale-at})] + + (restore-file! (:id file)) + + (let [change (th/db-get :file-change {:file-id (:id file)} + {::db/remove-deleted false})] + (t/is (nil? (:deleted-at change))) + (t/is (pos? (- (inst-ms (:updated-at change)) + (inst-ms stale-at))) + "updated_at must be bumped by the restore")))) diff --git a/backend/test/backend_tests/rpc_file_snapshot_test.clj b/backend/test/backend_tests/rpc_file_snapshot_test.clj index 91fea25e0b..c90d4c94ac 100644 --- a/backend/test/backend_tests/rpc_file_snapshot_test.clj +++ b/backend/test/backend_tests/rpc_file_snapshot_test.clj @@ -88,6 +88,19 @@ (t/is (= 0 (:revn row))) (t/is (= (:id profile) (:profile-id row)))))) + (t/testing "update snapshot with file-id" + (let [params {::th/type :update-file-snapshot + ::rpc/profile-id (:id profile) + :file-id (:id file) + :id @snapshot-id + :label "label1-updated"} + out (th/command! params)] + ;; (th/print-result! out) + (t/is (nil? (:error out))) + (t/is (true? (:result out))) + (t/is (= "label1-updated" + (:label (th/db-get :file-change {:id @snapshot-id})))))) + (t/testing "restore snapshot" (let [params {::th/type :restore-file-snapshot ::rpc/profile-id (:id profile) @@ -102,9 +115,10 @@ ;; (th/print-result! out) (t/is (nil? (:error out))) (t/is (nil? (:result out))) - ;; Peers learn who restored which version + ;; Peers learn who restored which version, named by the label the + ;; snapshot has now (the previous block renamed it) (t/is (= (:id profile) (:profile-id sent))) - (t/is (= "label1" (:label sent))) + (t/is (= "label1-updated" (:label sent))) (t/is (ct/inst? (:created-at sent)))) (let [[row1 row2 :as rows] @@ -748,3 +762,231 @@ (let [stored (fsnap/get-snapshot th/*system* file-id snap-id)] (t/is (nil? (:migrations stored))) (t/is (= 60 (:version stored))))))) +(t/deftest snapshots-require-file-id + ;; Since version 2.20 the snapshot commands require the `file-id` + ;; parameter, so partition pruning can always be applied on the + ;; `file_change` table. + (let [profile (th/create-profile* 1 {:is-active true}) + team-id (:default-team-id profile) + proj-id (:default-project-id profile) + + file (th/create-file* 1 {:profile-id (:id profile) + :project-id proj-id + :is-shared false}) + + snapshot + (let [params {::th/type :create-file-snapshot + ::rpc/profile-id (:id profile) + :file-id (:id file) + :label "label1"} + out (th/command! params)] + ;; (th/print-result! out) + + (t/is (nil? (:error out))) + (:result out))] + + (doseq [type [:update-file-snapshot :lock-file-snapshot + :unlock-file-snapshot :delete-file-snapshot]] + (let [params (merge {::th/type type + ::rpc/profile-id (:id profile) + :id (:id snapshot)} + (when (= type :update-file-snapshot) + {:label "label2"})) + out (th/command! params)] + + (t/testing (str (name type) " without file-id") + (t/is (th/ex-of-code? (:error out) :params-validation))))))) + +(t/deftest snapshots-reject-a-file-id-that-does-not-match + ;; The snapshot commands filter by `file_id` so that, once `file_change` is + ;; hash-partitioned by it, the lookup prunes to a single partition. Sending + ;; a `file-id` of another file must not find nor touch the snapshot. + (let [profile (th/create-profile* 1 {:is-active true}) + file-a (th/create-file* 1 {:profile-id (:id profile) + :project-id (:default-project-id profile) + :is-shared false}) + file-b (th/create-file* 2 {:profile-id (:id profile) + :project-id (:default-project-id profile) + :is-shared false}) + + snapshot (-> (th/command! {::th/type :create-file-snapshot + ::rpc/profile-id (:id profile) + :file-id (:id file-a) + :label "label1"}) + :result)] + + (doseq [type [:update-file-snapshot :lock-file-snapshot + :unlock-file-snapshot :delete-file-snapshot]] + (let [params (merge {::th/type type + ::rpc/profile-id (:id profile) + :file-id (:id file-b) + :id (:id snapshot)} + (when (= type :update-file-snapshot) + {:label "label2"})) + out (th/command! params)] + + (t/testing (name type) + (t/is (th/ex-info? (:error out))) + (t/is (= :not-found (:type (ex-data (:error out))))) + (t/is (= :snapshot-not-found (:code (ex-data (:error out))))) + (t/is (= "label1" + (:label (th/db-get :file-change {:id (:id snapshot)} + {::db/remove-deleted false}))) + "the snapshot of file-a stays intact")))))) + +(t/deftest file-change-mutations-maintain-updated-at + ;; `updated_at` is the per-row watermark used by the online partitioning + ;; copy of `file_change` (hash partitioning by file_id): every mutation + ;; must bump it, otherwise the copy misses the row. Lock, unlock and the + ;; soft-delete used to leave it stale. + (let [profile (th/create-profile* 1 {:is-active true}) + file (th/create-file* 1 {:profile-id (:id profile) + :project-id (:default-project-id profile) + :is-shared false}) + + snapshot + (let [params {::th/type :create-file-snapshot + ::rpc/profile-id (:id profile) + :file-id (:id file) + :label "label1"} + out (th/command! params)] + ;; (th/print-result! out) + + (t/is (nil? (:error out))) + (:result out)) + + stale-at #inst "2020-01-01T00:00:00.000Z"] + + (doseq [type [:lock-file-snapshot :unlock-file-snapshot + :update-file-snapshot :delete-file-snapshot]] + (t/testing (name type) + ;; make the row look old, so the mutation has to bump it + (th/db-update! :file-change {:updated-at stale-at} + {:id (:id snapshot)} + {::db/remove-deleted false}) + + (let [params (cond-> {::th/type type + ::rpc/profile-id (:id profile) + :file-id (:id file) + :id (:id snapshot)} + (= :update-file-snapshot type) + (assoc :label "renamed")) + out (th/command! params) + row (th/db-get :file-change {:id (:id snapshot)} + {::db/remove-deleted false})] + + (t/is (nil? (:error out))) + (t/is (true? (:result out))) + (t/is (pos? (- (inst-ms (:updated-at row)) (inst-ms stale-at))) + "updated_at must be bumped by the mutation") + (t/is (<= (inst-ms (:updated-at row)) (inst-ms (ct/now))) + "updated_at must not be in the future") + (t/is (<= (- (inst-ms (ct/now)) (inst-ms (:updated-at row))) 60000) + "updated_at must be the moment of the mutation") + (when (= type :delete-file-snapshot) + (t/is (< (inst-ms (:updated-at row)) (inst-ms (:deleted-at row))) + "the soft delete must not stamp updated_at with the future deleted_at"))))) + + (t/testing "the deleted snapshot is marked as deleted" + (t/is (some? (:deleted-at (th/db-get :file-change {:id (:id snapshot)} + {::db/remove-deleted false}))))) + + ;; process the delete-object job enqueued by the snapshot delete + (th/run-pending-jobs))) + +(t/deftest unknown-snapshot-is-reported-as-not-found + ;; The error points at the id that was not found: the snapshots are looked + ;; up by id, not by label. Both commands that resolve a snapshot report it + ;; the same way. + (let [profile (th/create-profile* 1 {:is-active true}) + file (th/create-file* 1 {:profile-id (:id profile) + :project-id (:default-project-id profile) + :is-shared false}) + + params {::rpc/profile-id (:id profile) + :file-id (:id file) + :id (uuid/random)}] + + (doseq [type [:restore-file-snapshot :get-file-snapshot]] + (t/testing (name type) + (let [out (th/command! (assoc params ::th/type type))] + + (t/is (th/ex-of-code? (:error out) :snapshot-not-found)) + (t/is (= "unable to find snapshot with the provided id" + (:hint (ex-data (:error out)))))))))) + +(t/deftest revision-and-snapshot-carry-different-content + ;; A revision is the delta of a save: it carries the changes and nothing + ;; else. A snapshot is the state of the file at a point: it carries the + ;; features and the migrations. + (let [profile (th/create-profile* 1 {:is-active true}) + file (th/create-file* 1 {:profile-id (:id profile) + :project-id (:default-project-id profile) + :is-shared false}) + page-id (uuid/random)] + + (update-file! + :file-id (:id file) + :profile-id (:id profile) + :revn 0 + :changes + [{:type :add-page + :name "test" + :id page-id}]) + + (t/testing "the revision carries the changes and not the features" + (let [rows (th/db-query :file-change {:file-id (:id file)}) + revision (first (filter #(nil? (:label %)) rows))] + + (t/is (some? revision)) + (t/is (some? (:changes revision))) + (t/is (nil? (:features revision))))) + + (t/testing "the snapshot carries the migrations and the features" + (let [out (th/command! {::th/type :create-file-snapshot + ::rpc/profile-id (:id profile) + :file-id (:id file) + :label "label1"})] + + (t/is (nil? (:error out))) + + (let [snapshot (th/db-get :file-change {:id (get-in out [:result :id])})] + (t/is (= "label1" (:label snapshot))) + (t/is (some? (:migrations snapshot))) + (t/is (some? (:features snapshot)))))))) + +(t/deftest file-change-rows-have-a-kind + ;; The kind tells the two kinds of rows of `file_change` apart: a snapshot + ;; (1) and a revision (2). The values are the contract with the database. + (let [profile (th/create-profile* 1 {:is-active true}) + file (th/create-file* 1 {:profile-id (:id profile) + :project-id (:default-project-id profile) + :is-shared false}) + page-id (uuid/random)] + + (update-file! + :file-id (:id file) + :profile-id (:id profile) + :revn 0 + :changes + [{:type :add-page + :name "test" + :id page-id}]) + + (t/testing "the revision is a revision" + (let [rows (th/db-query :file-change {:file-id (:id file)}) + revision (first (filter #(nil? (:label %)) rows))] + + (t/is (some? revision)) + (t/is (= 2 (:kind revision))))) + + (t/testing "the snapshot is a snapshot" + (let [out (th/command! {::th/type :create-file-snapshot + ::rpc/profile-id (:id profile) + :file-id (:id file) + :label "label1"})] + + (t/is (nil? (:error out))) + + (let [snapshot (th/db-get :file-change {:id (get-in out [:result :id])})] + (t/is (= 1 (:kind snapshot)))))))) diff --git a/backend/test/backend_tests/rpc_file_update_repair_test.clj b/backend/test/backend_tests/rpc_file_update_repair_test.clj index f8eba29f53..82201bbb30 100644 --- a/backend/test/backend_tests/rpc_file_update_repair_test.clj +++ b/backend/test/backend_tests/rpc_file_update_repair_test.clj @@ -346,6 +346,10 @@ (t/is (= 2 (count new))) (t/is (= (inc n) (:revn (first new)))) (t/is (= files-update/repair-session-id (:session-id (first new)))) + ;; the repair is a revision like the save: no features, a changes blob + (t/is (= 2 (:kind (first new)))) + (t/is (nil? (:features (first new)))) + (t/is (some? (:changes (first new)))) (t/is (some? (:deleted-at (first new)))) (t/is (= (:created-at (first new)) (:updated-at (first new)))) (t/is (= (+ n 2) (:revn (second new)))) diff --git a/backend/test/backend_tests/rpc_quotes_test.clj b/backend/test/backend_tests/rpc_quotes_test.clj index cd49f85042..8bc7c78fee 100644 --- a/backend/test/backend_tests/rpc_quotes_test.clj +++ b/backend/test/backend_tests/rpc_quotes_test.clj @@ -563,6 +563,7 @@ ;; deleting a version takes it out of the count right away (let [out (th/command! {::th/type :delete-file-snapshot ::rpc/profile-id (:id profile) + :file-id (:id file) :id (:id snapshot)})] (t/is (nil? (:error out))) (t/is (true? (:result out))) diff --git a/backend/test/backend_tests/scripts_decode_file_data_test.clj b/backend/test/backend_tests/scripts_decode_file_data_test.clj index e14618542f..3a6484b4ed 100644 --- a/backend/test/backend_tests/scripts_decode_file_data_test.clj +++ b/backend/test/backend_tests/scripts_decode_file_data_test.clj @@ -10,11 +10,14 @@ [app.common.uuid :as uuid] [app.scripts.decode-file-data :as dfd] [app.util.blob :as blob] + [backend-tests.helpers :as th] [clojure.java.io :as io] [clojure.test :as t]) (:import java.util.Base64)) +(t/use-fixtures :once th/state-init) + (defn- sample-file-data [] (let [p1 (uuid/next) @@ -80,3 +83,11 @@ parsed (json/decode (slurp tmp-out))] (t/is (== 2 (get-in parsed ["stats" "pages"]))) (t/is (= ["Accueil1" "Contact"] (mapv #(get % "name") (get parsed "pages")))))) + +(t/deftest get-snapshot-json-fails-when-the-snapshot-does-not-exist + ;; The snapshot lookup returns nil when there is no match, and a nil + ;; snapshot used to be printed as an empty JSON instead of failing. + (let [file-id (uuid/next) + snapshot-id (uuid/next)] + (t/is (thrown? clojure.lang.ExceptionInfo + (dfd/get-snapshot-json th/*system* file-id snapshot-id))))) diff --git a/frontend/src/app/main/data/workspace/versions.cljs b/frontend/src/app/main/data/workspace/versions.cljs index 30fab0de28..f14ccd36e2 100644 --- a/frontend/src/app/main/data/workspace/versions.cljs +++ b/frontend/src/app/main/data/workspace/versions.cljs @@ -106,7 +106,7 @@ (rx/of (update-versions-state {:editing nil}) (ev/event {::ev/name "rename-version" :file-id file-id})) - (->> (rp/cmd! :update-file-snapshot {:id id :label label}) + (->> (rp/cmd! :update-file-snapshot {:id id :file-id file-id :label label}) (rx/map fetch-versions))))))) (defn- clear-preview-state @@ -149,9 +149,10 @@ (ptk/reify ::delete-version ptk/WatchEvent - (watch [_ _ _] - (->> (rp/cmd! :delete-file-snapshot {:id id}) - (rx/map fetch-versions))))) + (watch [_ state _] + (let [file-id (:current-file-id state)] + (->> (rp/cmd! :delete-file-snapshot {:id id :file-id file-id}) + (rx/map fetch-versions)))))) (defn pin-version [id] @@ -161,8 +162,10 @@ (watch [_ state _] (when-let [version (->> (dm/get-in state [:workspace-versions :data]) (d/seek #(= (:id %) id)))] - (let [params {:id id - :label (ct/format-inst (:created-at version) :localized-date)}] + (let [file-id (:current-file-id state) + params {:id id + :file-id file-id + :label (ct/format-inst (:created-at version) :localized-date)}] (->> (rp/cmd! :update-file-snapshot params) (rx/mapcat (fn [_] @@ -175,18 +178,20 @@ (assert (uuid? id) "expected valid uuid for `id`") (ptk/reify ::lock-version ptk/WatchEvent - (watch [_ _ _] - (->> (rp/cmd! :lock-file-snapshot {:id id}) - (rx/map fetch-versions))))) + (watch [_ state _] + (let [file-id (:current-file-id state)] + (->> (rp/cmd! :lock-file-snapshot {:id id :file-id file-id}) + (rx/map fetch-versions)))))) (defn unlock-version [id] (assert (uuid? id) "expected valid uuid for `id`") (ptk/reify ::unlock-version ptk/WatchEvent - (watch [_ _ _] - (->> (rp/cmd! :unlock-file-snapshot {:id id}) - (rx/map fetch-versions))))) + (watch [_ state _] + (let [file-id (:current-file-id state)] + (->> (rp/cmd! :unlock-file-snapshot {:id id :file-id file-id}) + (rx/map fetch-versions)))))) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; RESTORE VERSION EVENTS diff --git a/frontend/src/app/plugins/file.cljs b/frontend/src/app/plugins/file.cljs index 59d222512b..28c4f9a0a6 100644 --- a/frontend/src/app/plugins/file.cljs +++ b/frontend/src/app/plugins/file.cljs @@ -54,7 +54,7 @@ :else (do (swap! data assoc :label value :created-by "user") - (->> (rp/cmd! :update-file-snapshot {:id (:id @data) :label value}) + (->> (rp/cmd! :update-file-snapshot {:id (:id @data) :file-id file-id :label value}) (rx/take 1) (rx/subs! #(st/emit! (se/event plugin-id "rename-version" :file-id file-id)))))))} @@ -95,7 +95,7 @@ :else (let [version-id (:id @data)] - (->> (rp/cmd! :delete-file-snapshot {:id version-id}) + (->> (rp/cmd! :delete-file-snapshot {:id version-id :file-id file-id}) (rx/map (constantly nil)) (rx/subs! resolve reject))))))) @@ -111,15 +111,22 @@ (u/reject-not-valid reject :pin "Only auto-saved versions can be pinned") :else - (let [params {:id (:id @data) - :label (ct/format-inst (:created-at @data) :localized-date)}] + (let [params {:id (:id @data) + :file-id file-id + :label (ct/format-inst (:created-at @data) :localized-date)}] (->> (rx/zip (rp/cmd! :get-team-users {:file-id file-id}) (rp/cmd! :update-file-snapshot params)) (rx/subs! - (fn [[users data]] + (fn [[users _]] (let [users (d/index-by :id users)] (st/emit! (se/event plugin-id "pin-version" :file-id file-id)) - (resolve (file-version-proxy plugin-id file-id users @data)))) + ;; the command answers `true`: the pinned row is the + ;; current one with the new label, as a user version + (resolve (file-version-proxy plugin-id file-id users + (assoc @data + :label (:label params) + :created-by "user" + :deleted-at nil))))) reject)))))))))) (defn file-proxy? [p] diff --git a/frontend/test/frontend_tests/data/workspace_versions_test.cljs b/frontend/test/frontend_tests/data/workspace_versions_test.cljs index 7753ba4164..0bac593c0b 100644 --- a/frontend/test/frontend_tests/data/workspace_versions_test.cljs +++ b/frontend/test/frontend_tests/data/workspace_versions_test.cljs @@ -220,3 +220,74 @@ (t/is (nil? (get-in result [:workspace-versions :backup])))) (finally (rx/dispose! sub))))) + +;; The snapshot RPC commands require the `file-id` (they use it to prune the +;; partitions of `file_change`), so every workspace version event sends the +;; one of the file that is open. + +(defn- version-event-request + "Run a workspace version event with `file-id` open and `version-id` in the + history, and return the `[command params]` of the RPC request it sends." + [file-id version-id event] + (let [requests (atom []) + state {:current-file-id file-id + :workspace-versions {:data [{:id version-id + :created-at (ct/now) + :created-by "system"}]}}] + (mock/with-mocks* + {rp/cmd! (mock/stub (fn [cmd params] + (swap! requests conj [cmd params]) + (rx/of true)))} + (->> (ptk/watch event state (rx/empty)) + (rx/subs! (fn [_]))) + (first @requests)))) + +(t/deftest ^:async renaming-a-version-sends-the-current-file-id + (let [file-id (uuid/next) + version-id (uuid/next) + [cmd params] (await (version-event-request + file-id version-id + (versions/rename-version version-id "name")))] + (t/is (= :update-file-snapshot cmd)) + (t/is (= version-id (:id params))) + (t/is (= file-id (:file-id params))))) + +(t/deftest ^:async deleting-a-version-sends-the-current-file-id + (let [file-id (uuid/next) + version-id (uuid/next) + [cmd params] (await (version-event-request + file-id version-id + (versions/delete-version version-id)))] + (t/is (= :delete-file-snapshot cmd)) + (t/is (= version-id (:id params))) + (t/is (= file-id (:file-id params))))) + +(t/deftest ^:async locking-a-version-sends-the-current-file-id + (let [file-id (uuid/next) + version-id (uuid/next) + [cmd params] (await (version-event-request + file-id version-id + (versions/lock-version version-id)))] + (t/is (= :lock-file-snapshot cmd)) + (t/is (= version-id (:id params))) + (t/is (= file-id (:file-id params))))) + +(t/deftest ^:async unlocking-a-version-sends-the-current-file-id + (let [file-id (uuid/next) + version-id (uuid/next) + [cmd params] (await (version-event-request + file-id version-id + (versions/unlock-version version-id)))] + (t/is (= :unlock-file-snapshot cmd)) + (t/is (= version-id (:id params))) + (t/is (= file-id (:file-id params))))) + +(t/deftest ^:async pinning-a-version-sends-the-current-file-id + (let [file-id (uuid/next) + version-id (uuid/next) + [cmd params] (await (version-event-request + file-id version-id + (versions/pin-version version-id)))] + (t/is (= :update-file-snapshot cmd)) + (t/is (= version-id (:id params))) + (t/is (= file-id (:file-id params))))) diff --git a/frontend/test/frontend_tests/plugins/file_test.cljs b/frontend/test/frontend_tests/plugins/file_test.cljs index e6f2780035..a71b336943 100644 --- a/frontend/test/frontend_tests/plugins/file_test.cljs +++ b/frontend/test/frontend_tests/plugins/file_test.cljs @@ -6,8 +6,14 @@ (ns frontend-tests.plugins.file-test (:require + [app.common.time :as ct] + [app.common.uuid :as uuid] + [app.main.repo :as rp] [app.plugins.file :as file] - [cljs.test :as t :include-macros true])) + [app.plugins.register :as r] + [beicon.v2.core :as rx] + [cljs.test :as t :include-macros true] + [frontend-tests.helpers.mock :as mock])) (t/deftest file-version-created-at-returns-stored-date (let [created-at (js/Date.) @@ -19,3 +25,109 @@ :label "Version" :created-at created-at})] (t/is (identical? created-at (.-createdAt version))))) + +(defn- make-version + [file-id data] + (file/file-version-proxy (str (uuid/next)) file-id {} data)) + +(defn- record-cmd! + "Mock for `rp/cmd!` that records the calls and answers an observable with + the shape each command expects: a list of team users, and `true` for the + snapshot update (`update-file-snapshot` answers a boolean)." + [calls] + (fn [cmd params] + (swap! calls conj [cmd params]) + (rx/of (if (= :get-team-users cmd) [] true)))) + +;; The snapshot RPC commands require the `file-id` (they use it to prune the +;; partitions of `file_change`); the plugin proxy is the other door to them and +;; sends it on rename, remove and pin. + +(t/deftest ^:async renaming-a-version-sends-the-current-file-id + (let [file-id (uuid/next) + version-id (uuid/next) + calls (atom [])] + + (await + (mock/with-mocks* + {r/check-permission (constantly true) + rp/cmd! (mock/stub (record-cmd! calls))} + + (let [version (make-version file-id + {:id version-id + :label "Version" + :created-at (ct/now) + :created-by "user"})] + (aset version "label" "renamed") + (let [[cmd params] (last @calls)] + (t/is (= :update-file-snapshot cmd)) + (t/is (= version-id (:id params))) + (t/is (= file-id (:file-id params))) + (t/is (= "renamed" (:label params))))))))) + +(t/deftest ^:async removing-a-version-sends-the-current-file-id + (let [file-id (uuid/next) + version-id (uuid/next) + calls (atom [])] + + (await + (mock/with-mocks* + {r/check-permission (constantly true) + rp/cmd! (mock/stub (record-cmd! calls))} + + (let [version (make-version file-id + {:id version-id + :label "Version" + :created-at (ct/now) + :created-by "user"})] + (await (.remove version)) + (let [[cmd params] (last @calls)] + (t/is (= :delete-file-snapshot cmd)) + (t/is (= version-id (:id params))) + (t/is (= file-id (:file-id params))))))))) + +(t/deftest ^:async pinning-a-version-sends-the-current-file-id + (let [file-id (uuid/next) + version-id (uuid/next) + calls (atom [])] + + (await + (mock/with-mocks* + {r/check-permission (constantly true) + rp/cmd! (mock/stub (record-cmd! calls))} + + (let [version (make-version file-id + {:id version-id + :label "Version" + :created-at (ct/now) + :created-by "system"})] + (await (.pin version)) + (t/is (= [[:get-team-users file-id] + [:update-file-snapshot file-id]] + (mapv (fn [[cmd params]] [cmd (:file-id params)]) @calls)))))))) + +(t/deftest ^:async file-version-proxy-pin-resolves-the-pinned-version + ;; `pin` answers with a proxy, not with the boolean the snapshot command + ;; returns: the autosave becomes a user version with a new label. + (let [file-id (uuid/next) + version-id (uuid/next) + created-at (ct/now) + calls (atom [])] + (await + (mock/with-mocks* + {r/check-permission (constantly true) + rp/cmd! (mock/stub (record-cmd! calls))} + (let [version (make-version file-id + {:id version-id + :label "Version" + :created-at created-at + :created-by "system"}) + pinned (await (.pin version))] + (t/is (file/file-version-proxy? pinned)) + (t/is (= [[:get-team-users file-id] + [:update-file-snapshot file-id]] + (mapv (fn [[cmd params]] [cmd (:file-id params)]) @calls))) + (t/is (= (ct/format-inst created-at :localized-date) + (.-label pinned))) + (t/is (false? (.-isAutosave pinned))) + (t/is (identical? created-at (.-createdAt pinned)))))))) diff --git a/operations/2.20.0-file-change-partition.md b/operations/2.20.0-file-change-partition.md new file mode 100644 index 0000000000..b74e589b56 --- /dev/null +++ b/operations/2.20.0-file-change-partition.md @@ -0,0 +1,912 @@ +# Partition `file_change` by `file_id` + +Online procedure to turn `file_change` into a table partitioned by +`HASH (file_id)` with 16 partitions, aligned with the ones of `file_data`. + +- Migration: `backend/src/app/migrations/sql/0158-partition-file-change.sql` +- Issue: + +The result has to be **exactly** the schema of the migration, because on a big +instance the migration is marked as applied and never runs. This procedure +runs the same statements by hand. + +## Sizing your instance + +Every number in this procedure comes from your database: run the queries +below, the procedure does not depend on any absolute size. It is plain SQL +and runs on any supported PostgreSQL (>= 15). + +```sql +-- total size of the database and of file_change (heap + TOAST + indexes) +SELECT pg_size_pretty(pg_database_size(current_database())) AS database; +SELECT pg_size_pretty(pg_total_relation_size('file_change')) AS file_change_total, + pg_size_pretty(pg_table_size('file_change')) AS heap_and_toast, + pg_size_pretty(pg_indexes_size('file_change')) AS indexes; +SELECT count(*) AS rows FROM file_change; + +-- dead tuples waiting for vacuum: their space is not copied +SELECT n_dead_tup FROM pg_stat_user_tables WHERE relname = 'file_change'; +``` + +`pg_table_size` (heap + TOAST) is the upper bound of what the copy reads and +writes. Do not estimate it with `pg_stats.avg_width`: that measures what the +heap stores for a column, which for a TOASTed value is the pointer, not the +value. The estimate reads far below the real size (about ten times less on a +small instance, orders of magnitude on a big one). + +Peak disk during the bulk is about the old table (it does not grow) plus the +new table (at most its `pg_table_size`, less because the dead rows are not +copied) plus its six indexes (about the `pg_indexes_size` of the live rows) +plus the WAL the bulk generates (proportional to what it writes: watch the +free space of the `pg_wal` mount while the first batches run). The disk comes +back when the legacy table is dropped, days later. Time the first batches and +extrapolate: the bulk reads heap + TOAST once sequentially and writes about +the same, so its duration is proportional to `pg_table_size`, not to the heap +alone. + +## What changes + +- `file_change` is partitioned by `HASH (file_id)` with 16 partitions, the + same bounds as `file_data`, so the join between them is resolved partition by + partition. +- The primary key goes from `(id)` to `(file_id, id)`: PostgreSQL requires the + partition key in the primary key, and the application already filters by + `file_id` everywhere. +- The table has 17 columns: `data_ref_id` and `data_backend` are gone (dead + since the file storage refactor), `data` is kept (the `legacy-db` backend + reads old payloads from it) and `kind` appears (1 = snapshot, 2 = revision). +- The old table is kept as `file_change_legacy`, a frozen copy of the + previous layout, and dropped days later. +- Almost all of the size of the table is dead space (see sizing), so the new + table holds at most its heap + TOAST, and less as the dead rows are not + copied. The disk comes back when the legacy table is dropped: the database + shrinks by the size of the old table. + +## Why this mechanism + +`ATTACH` as a `DEFAULT` partition is not possible: the 16 hash siblings cover +the whole hash space, so the constraint of a `DEFAULT` partition could never +hold, and the `ATTACH` fails when it validates every row. A `UNION` +view with triggers, dual writes in the application and logical replication +were discarded too. + +What is left is a copy in phases, driven by the `updated_at` watermark, and a +cutover of minutes: plain SQL, resumable, no new infrastructure. The heap is +small next to the TOAST, so a delta pass without an index scans the heap and +only detoastes the rows that changed. + +--- + +## Phase 0 — Before anything + +### 0.0 The role of the session + +Every session of this procedure starts with `SET ROLE` to the role that owns +`file_change` (normally `penpot`): + +```sql +SET ROLE penpot; -- replace with the owner of file_change (see 0.6) +``` + +The new table, its 16 partitions, its indexes and its TOAST are created with +the role of the session, and `ALTER TABLE ... OWNER TO` does not recurse to the +partitions. Without this, a procedure run as a superuser leaves the table +owned by that superuser and the backend, which connects as `penpot`, gets +`permission denied` after the cutover. + +### 0.1 Mark the migration as applied + +The runner applies at startup every migration that is not in the `migrations` +table. Without this, it would copy the whole table inside the transaction of the +startup. + +```sql +INSERT INTO migrations (module, step) +VALUES ('main', '0158-partition-file-change'); +``` + +The module is `main` and the step has no `.sql` (see +`backend/src/app/util/migrations.clj`). + +### 0.2 Add the `kind` column to the current table + +Until the cutover, `file_change` is the current table, and the new code writes +`kind` on every insert: without the column the backend does not start. + +```sql +ALTER TABLE file_change ADD COLUMN IF NOT EXISTS kind smallint; +``` + +Nullable and without a default on purpose: it is a metadata-only operation +(it does not rewrite a single row of the table), the old code keeps inserting +rows, and the values of the rows that are already there are computed by the +copy from their `label`. + +### 0.3 Deploy the code + +The `updated_at` watermark has to be alive **before** the bulk copy: the new +code updates it on every mutation (locks, soft deletes), the old one does +not. Rows changed between the bulk and a delta pass would be lost. + +```sql +SELECT max(updated_at) FROM file_change; +``` + +### 0.4 Index for the delta passes + +The delta passes filter by `updated_at`. Without an index on that column every +pass scans the whole heap; with it, it is an index range that only +touches the rows that changed. + +```sql +-- CONCURRENTLY: it does not block the writes. It cannot run inside a +-- transaction: run it alone. +CREATE INDEX CONCURRENTLY IF NOT EXISTS file_change__updated_at__idx + ON file_change (updated_at); +``` + +- The build reads the heap once sequentially and writes one narrow entry per + row (check its size with `pg_relation_size` after the build): it does not + block the writes either way. Time the build and extrapolate the bulk from + it (an index on a column without TOAST does not touch the + TOAST). +- If it is interrupted, the index is left `INVALID` and has to be dropped and + built again: + +```sql +SELECT i.indisvalid + FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid + WHERE c.relname = 'file_change__updated_at__idx'; +``` + +- The index goes with the old table: at the cutover it is renamed to + `file_change__updated_at__idx_legacy` and disappears with it. The new table + does not carry it (nothing queries by `updated_at`; the GC uses + `deleted_at`). + +### 0.5 Check the environment + +```sql +-- Space: peak about the old table (it does not grow) + the new table +-- (at most its `pg_table_size`, see sizing) + the WAL the bulk generates +SELECT pg_size_pretty(pg_database_size(current_database())); + +-- The WAL must not run out of room (note the free space of the mount point): +-- df -h + +-- Nothing must reference file_change (0 rows expected): +SELECT conrelid::regclass FROM pg_constraint + WHERE confrelid = 'file_change'::regclass; + +-- There must be no triggers on file_change: +SELECT tgname FROM pg_trigger + WHERE tgrelid = 'file_change'::regclass AND NOT tgisinternal; + +-- Replication slots: the bulk generates WAL proportional to what it writes, +-- watch the retained WAL while the first batches run. +SELECT slot_name, active, pg_size_pretty(pg_wal_lsn_diff( + pg_current_wal_lsn(), restart_lsn)) AS retained_wal + FROM pg_replication_slots; + +-- The legacy blobs of the snapshots were already moved to `file_data` by +-- migration 0157-migrate-legacy-file-data (on the big instances, by hand, +-- with its runbook 2.19.0-migrate-legacy-file-data.md), so there is nothing +-- to copy here: the `data` column travels with the row to the new table. +-- These two numbers are informational guards. The first one has to be 0 +-- after 0157 ran; if it is not, run 0157 (or its runbook) before starting. +SELECT count(*) AS snapshots_without_data + FROM file_change AS fc + WHERE fc.data IS NOT NULL + AND fc.label IS NOT NULL + AND NOT EXISTS (SELECT 1 FROM file_data AS fd + WHERE fd.file_id = fc.file_id + AND fd.id = fc.id + AND fd.type = 'snapshot'); + +-- The revisions with legacy data are not migrated (they are the ephemeral +-- changes, out of scope). This number is informative: their data travels +-- with the row to the new table. +SELECT count(*) AS revisions_with_data + FROM file_change + WHERE data IS NOT NULL + AND label IS NULL; +``` + +- Check that the backup / PITR is working before starting. +- Pick a low activity window for the cutover: a freeze of the writes to + `file_change` of a couple of minutes (the final delta inside the lock + copies only the rows changed since the last delta pass, so it is quick; the + saves of files during the freeze fail and the user retries, and the rest of + the database keeps working). The window has to cover the whole cutover, with + the backend and the workers stopped and started again (see 4.2). + +### 0.6 Owner and grants + +```sql +-- the role that has to own every object at the end (see 0.0) +SELECT tableowner FROM pg_tables + WHERE schemaname = current_schema() AND tablename = 'file_change'; + +-- grants added by the operator (monitoring, reporting). This procedure +-- creates a new table: they do not travel with it, so save them to re-apply +-- them after the cutover. +SELECT grantee, privilege_type + FROM information_schema.role_table_grants + WHERE table_schema = current_schema() AND table_name = 'file_change'; +``` + +--- + +## Phase 1 — Create the partitioned table + +The new indexes carry a `_new` suffix because index names are unique per +schema and the old table still holds the canonical ones until the cutover. +The same goes for the primary key of the new table: its index is named after +the constraint. The constraints that have no index (the `CHECK` and the +foreign keys) are named canonically from the start, because a constraint name +is unique per table, not per schema, so it can live with the one of the old +table; the partitions inherit it when they are created. The partitions are +created with their final name, because their indexes are named after the +partition and not after the table. + +```sql +CREATE TABLE file_change_new ( + file_id uuid NOT NULL, + id uuid NOT NULL DEFAULT gen_random_uuid(), + created_at timestamptz NOT NULL DEFAULT now(), + session_id uuid, + revn bigint NOT NULL DEFAULT 0, + data bytea, + changes bytea, + profile_id uuid, + features text[], + label text, + version integer, + created_by text NOT NULL DEFAULT 'system', + updated_at timestamptz DEFAULT now(), + deleted_at timestamptz, + migrations text[], + locked_by uuid, + kind smallint NOT NULL, + CONSTRAINT file_change_kind_check CHECK (kind IN (1, 2)) +) PARTITION BY HASH (file_id); + +-- the columns keep the storage of PostgreSQL by default (migration +-- 0156-reset-pg-storage-defaults dropped the explicit EXTERNAL overrides) + +COMMENT ON COLUMN file_change_new.kind IS + 'The kind of the row: 1 is a snapshot (it has a label and its data lives in file_data) and 2 is a revision (it has no label and it holds the changes).'; +COMMENT ON COLUMN file_change_new.locked_by IS + 'Profile ID of user who has locked this version. Only the creator can lock/unlock their own versions. Locked versions cannot be deleted by others.'; + +-- 16 partitions, the same bounds as file_data. The name of the table is +-- temporary (the old one holds it until the cutover) but the name of the +-- partitions is not: it is the final one, so their indexes are born with +-- the canonical names (renaming a partition does not rename its indexes) +-- and the result is exactly the schema of the migration. +CREATE TABLE file_change_00 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 0); +CREATE TABLE file_change_01 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 1); +CREATE TABLE file_change_02 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 2); +CREATE TABLE file_change_03 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 3); +CREATE TABLE file_change_04 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 4); +CREATE TABLE file_change_05 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 5); +CREATE TABLE file_change_06 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 6); +CREATE TABLE file_change_07 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 7); +CREATE TABLE file_change_08 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 8); +CREATE TABLE file_change_09 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 9); +CREATE TABLE file_change_10 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 10); +CREATE TABLE file_change_11 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 11); +CREATE TABLE file_change_12 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 12); +CREATE TABLE file_change_13 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 13); +CREATE TABLE file_change_14 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 14); +CREATE TABLE file_change_15 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 15); + +-- The primary key includes the partition key (required on a partitioned table) +ALTER TABLE file_change_new ADD PRIMARY KEY (file_id, id); + +-- Foreign keys (no CREATE TABLE copies them). They take the canonical name +-- already: they have no index, so they do not collide with the old table and +-- the partitions inherit it. +ALTER TABLE file_change_new + ADD CONSTRAINT file_change_file_id_fkey FOREIGN KEY (file_id) + REFERENCES file(id) DEFERRABLE; +ALTER TABLE file_change_new + ADD CONSTRAINT file_change_locked_by_fkey FOREIGN KEY (locked_by) + REFERENCES profile(id) ON DELETE SET NULL DEFERRABLE; +ALTER TABLE file_change_new + ADD CONSTRAINT file_change_profile_id_fkey FOREIGN KEY (profile_id) + REFERENCES profile(id) ON DELETE SET NULL DEFERRABLE; +``` + +```sql +SELECT count(*) AS partitions FROM pg_partition_tree('file_change_new') t + WHERE t.isleaf; +-- 16 +``` + +--- + +## Phase 2 — Bulk copy + +Copies every row (including the soft deleted ones: the GC manages them the +same way in the new table). The `kind` is computed from the `label`: a row +with a label is a snapshot, a row without one is a revision. + +The pointer of the copy lives in the database, not in the terminal: a batch +can be interrupted and resumed without losing anything. + +```sql +CREATE TABLE IF NOT EXISTS file_change_copy_state ( + name text PRIMARY KEY, + last_id uuid, + last_at timestamptz +); + +-- for a procedure that already created the table without the mark +ALTER TABLE file_change_copy_state + ADD COLUMN IF NOT EXISTS last_at timestamptz; + +INSERT INTO file_change_copy_state (name, last_id) +VALUES ('bulk', '00000000-0000-0000-0000-000000000000') +ON CONFLICT (name) DO NOTHING; + +INSERT INTO file_change_copy_state (name) VALUES ('delta') +ON CONFLICT (name) DO NOTHING; +``` + +One batch (idempotent). It **always returns one row**: the number of rows it +copied, `0` when there is nothing left to copy. + +```sql +WITH batch AS MATERIALIZED ( + SELECT id FROM file_change + WHERE id > (SELECT last_id FROM file_change_copy_state WHERE name = 'bulk') + ORDER BY id + LIMIT 10000 +), +copied AS ( + INSERT INTO file_change_new (id, file_id, created_at, session_id, revn, data, changes, + profile_id, features, label, version, created_by, + updated_at, deleted_at, migrations, locked_by, kind) + SELECT id, file_id, created_at, session_id, revn, data, changes, + profile_id, features, label, version, created_by, + updated_at, deleted_at, migrations, locked_by, + CASE WHEN label IS NOT NULL THEN 1 ELSE 2 END + FROM file_change + WHERE id IN (SELECT id FROM batch) + ON CONFLICT (file_id, id) DO UPDATE + SET created_at = EXCLUDED.created_at, + session_id = EXCLUDED.session_id, + revn = EXCLUDED.revn, + data = EXCLUDED.data, + changes = EXCLUDED.changes, + profile_id = EXCLUDED.profile_id, + features = EXCLUDED.features, + label = EXCLUDED.label, + version = EXCLUDED.version, + created_by = EXCLUDED.created_by, + updated_at = EXCLUDED.updated_at, + deleted_at = EXCLUDED.deleted_at, + migrations = EXCLUDED.migrations, + locked_by = EXCLUDED.locked_by, + kind = EXCLUDED.kind + WHERE file_change_new.updated_at IS DISTINCT FROM EXCLUDED.updated_at + RETURNING 1 +), +-- `moved` advances the pointer only when the batch copied something. +-- PostgreSQL executes a data-modifying CTE exactly once even when the final +-- query does not read its output. +moved AS ( + UPDATE file_change_copy_state + SET last_id = (SELECT id FROM batch ORDER BY id DESC LIMIT 1) + WHERE name = 'bulk' + AND EXISTS (SELECT 1 FROM copied) + RETURNING 1 +) +SELECT count(*) AS copied FROM copied; +``` + +Repeat this statement until it returns `0`. The `0` comes from the final +`SELECT`, not from a missing row: a batch that finds nothing to copy still +returns it, so it is the signal to stop. + +- The pointer is a `uuid` (random order): the pagination is stable but it does + not group by physical position. The total read is the heap, in + a scattered order. +- **Estimated duration**: time the first batches and extrapolate. The bulk + reads the heap plus the live TOAST once (see sizing) and writes about the + live data. Batches are rows divided by batch size: count your rows in + sizing and divide. +- Pace the batches yourself (a pause between statements) if the streaming + replica falls behind or the backend sees latency. Every batch detoastes its + share of the TOAST: the cost is I/O, not CPU. +- Watch: replica lag, `pg_stat_activity` (locks between the bulk and + autovacuum), the space of `pg_wal` and of the data dir. + +```sql +SELECT count(*) AS missing + FROM file_change o + WHERE NOT EXISTS (SELECT 1 FROM file_change_new n + WHERE n.file_id = o.file_id + AND n.id = o.id); +``` + +`missing` is zero when no row of the old table is absent from the new one; it +is the check that matters, not the total count. The new table can have **more** +rows than the old one: the GC may have hard deleted a row from the source +after the bulk copied it, and it cleans that row up after the cutover. Rows +written during the bulk are missing until the delta passes pick them up. + +--- + +## Phase 3 — Delta passes + +Every pass copies the rows modified since the **previous pass**. The watermark +lives in `file_change_copy_state.last_at` (the start of the last pass), and the +SQL subtracts one hour of margin, which absorbs the clock drift between the +application (which writes `updated_at`) and the database (which writes +`now()`) and the drift between batches. A row that a pass already copied but +did not change again is not rewritten: the `WHERE` of the `ON CONFLICT` skips +it. + +```sql +-- :bulk_start = the time the bulk started (the margin is applied below). +-- Set it in this session before running the pass: without it the query fails +-- instead of copying nothing and leaving those rows behind for good. +SET penpot.bulk_start = '2026-10-25 08:00:00+02'; + +BEGIN; + +WITH mark AS ( + SELECT COALESCE((SELECT last_at FROM file_change_copy_state WHERE name = 'delta'), + current_setting('penpot.bulk_start')::timestamptz) - INTERVAL '1 hour' AS since +) +INSERT INTO file_change_new (id, file_id, created_at, session_id, revn, data, changes, + profile_id, features, label, version, created_by, + updated_at, deleted_at, migrations, locked_by, kind) +SELECT id, file_id, created_at, session_id, revn, data, changes, + profile_id, features, label, version, created_by, + updated_at, deleted_at, migrations, locked_by, + CASE WHEN label IS NOT NULL THEN 1 ELSE 2 END + FROM file_change, mark + WHERE updated_at > mark.since +ON CONFLICT (file_id, id) DO UPDATE + SET created_at = EXCLUDED.created_at, + session_id = EXCLUDED.session_id, + revn = EXCLUDED.revn, + data = EXCLUDED.data, + changes = EXCLUDED.changes, + profile_id = EXCLUDED.profile_id, + features = EXCLUDED.features, + label = EXCLUDED.label, + version = EXCLUDED.version, + created_by = EXCLUDED.created_by, + updated_at = EXCLUDED.updated_at, + deleted_at = EXCLUDED.deleted_at, + migrations = EXCLUDED.migrations, + locked_by = EXCLUDED.locked_by, + kind = EXCLUDED.kind + WHERE file_change_new.updated_at IS DISTINCT FROM EXCLUDED.updated_at; + +-- The mark is the start of this transaction (`now()`): the rows changed +-- after it are the window of the next pass. Both statements go in the same +-- transaction on purpose; a mark written later would leave a gap. +UPDATE file_change_copy_state SET last_at = now() WHERE name = 'delta'; + +COMMIT; +``` + +- Repeat the transaction every 1-6 h (depending on the activity) until the + pass copies very few rows. With the index of phase 0.4 the pass is an index + range over the window, so it takes seconds on a quiet instance; without it, + it scans the heap. To see how many rows are left in the window: + +```sql +SELECT count(*) FROM file_change + WHERE updated_at > (SELECT last_at FROM file_change_copy_state WHERE name = 'delta'); +``` + +- The objects GC can keep running. Its soft deletes are covered by the + watermark, and its hard deletes leave a row with `deleted_at` in the new + table that the GC itself cleans up later. The hard delete of a **file** is + blocked while the old table (or the new one) has rows for it, because the + foreign key on `file_id` is `NO ACTION`. The GC deletes the pending files in + chunks of 100, and one blocked file aborts the transaction of the whole + chunk: the chunk is logged as `error on chunk deletion` and skipped, its + files wait in the trash, and the next daily run retries the same chunk and + fails again until the cutover. The other chunks keep making progress. **Do + not run the cutover at the same time as a big batch of the GC.** + +--- + +## Phase 4 — Cutover (a freeze of minutes) + +### 4.1 Before the lock + +- One last delta pass. +- Build the secondary indexes of the new table. The bulk and the delta passes + only need the primary key (`ON CONFLICT (file_id, id)`), so building them + here keeps five indexes and their WAL out of every batch. The `deleted_at` + index carries `file_id` (the primary key is `(file_id, id)`): the objects GC + needs the `file_id` to lock by pair, and with the index it reads the rows it + has to delete without touching the table. + +```sql +-- Secondary indexes: the canonical names, with the _new suffix. There is no +-- index for the system snapshots: the old one filtered them by +-- `data IS NOT NULL`, and with the payload living in `file_data` the column +-- no longer tells snapshots apart. The new one could only filter +-- them by `deleted_at IS NULL`, and the system snapshots always carry a +-- deletion date in the future, so it would never match a row. +CREATE INDEX file_change_new__deleted_at__idx + ON file_change_new (deleted_at, file_id, id) WHERE deleted_at IS NOT NULL; +CREATE INDEX file_change_new__file_id__revn__idx + ON file_change_new (file_id, revn); +CREATE INDEX file_change_new__locked_by__idx + ON file_change_new (locked_by) WHERE locked_by IS NOT NULL; +CREATE INDEX file_change_new__profile_id__idx + ON file_change_new (profile_id) WHERE profile_id IS NOT NULL; +CREATE INDEX file_change_new__user_snapshots__idx + ON file_change_new (file_id) + WHERE label IS NOT NULL AND created_by = 'user' AND deleted_at IS NULL; +``` + +- Prepare and rehearse the cutover script in staging. + +### 4.2 The cutover (one transaction, a brief exclusive lock) + +- **Stop the backend and the workers before the transaction, and start them + again after it.** The pool keeps server-side prepared statements, and a + prepared statement holds the OID of the table it was prepared with: + renaming the table does not change that OID. A connection that had already + run `insert into file_change` five times (the default threshold of the + driver) would keep writing to `file_change_legacy` after the cutover, and + the pool recycles its connections every 30 minutes, so the window of + silent writes is long. +- Run the transaction. +- Start the backend and the workers again. + +```sql +SET ROLE penpot; -- the owner of file_change (see 0.0 and 0.6) +SET penpot.bulk_start = '2026-10-25 08:00:00+02'; + +BEGIN; + +LOCK TABLE file_change IN ACCESS EXCLUSIVE MODE; + +-- The final delta: the rows modified since the last pass (its mark, minus +-- the hour of margin). The mark is the one the last pass left; the fallback +-- is the bulk start. +WITH mark AS ( + SELECT COALESCE((SELECT last_at FROM file_change_copy_state WHERE name = 'delta'), + current_setting('penpot.bulk_start')::timestamptz) - INTERVAL '1 hour' AS since +) +INSERT INTO file_change_new (id, file_id, created_at, session_id, revn, data, changes, + profile_id, features, label, version, created_by, + updated_at, deleted_at, migrations, locked_by, kind) +SELECT id, file_id, created_at, session_id, revn, data, changes, + profile_id, features, label, version, created_by, + updated_at, deleted_at, migrations, locked_by, + CASE WHEN label IS NOT NULL THEN 1 ELSE 2 END + FROM file_change, mark + WHERE updated_at > mark.since +ON CONFLICT (file_id, id) DO UPDATE + SET created_at = EXCLUDED.created_at, + session_id = EXCLUDED.session_id, + revn = EXCLUDED.revn, + data = EXCLUDED.data, + changes = EXCLUDED.changes, + profile_id = EXCLUDED.profile_id, + features = EXCLUDED.features, + label = EXCLUDED.label, + version = EXCLUDED.version, + created_by = EXCLUDED.created_by, + updated_at = EXCLUDED.updated_at, + deleted_at = EXCLUDED.deleted_at, + migrations = EXCLUDED.migrations, + locked_by = EXCLUDED.locked_by, + kind = EXCLUDED.kind + WHERE file_change_new.updated_at IS DISTINCT FROM EXCLUDED.updated_at; + +-- Check inside the lock, and abort if it fails: no row of the old table may +-- be missing in the new one. The new table can carry extra rows that the GC +-- hard deleted from the source; it cleans them up after the cutover. +DO $$ +DECLARE missing bigint; +BEGIN + SELECT count(*) INTO missing + FROM file_change o + WHERE NOT EXISTS (SELECT 1 FROM file_change_new n + WHERE n.file_id = o.file_id + AND n.id = o.id); + + IF missing > 0 THEN + RAISE EXCEPTION 'file_change: % rows of the old table are missing in file_change_new; cutover aborted', missing; + END IF; +END $$; + +-- The old table frees the canonical names (the same names the migration +-- 0158 uses, with the _legacy suffix) +ALTER TABLE file_change RENAME CONSTRAINT file_change_pkey + TO file_change_pkey_legacy; +ALTER TABLE file_change RENAME CONSTRAINT file_change_file_id_fkey + TO file_change_file_id_fkey_legacy; +ALTER TABLE file_change RENAME CONSTRAINT file_change_locked_by_fkey + TO file_change_locked_by_fkey_legacy; +ALTER TABLE file_change RENAME CONSTRAINT file_change_profile_id_fkey + TO file_change_profile_id_fkey_legacy; + +ALTER INDEX file_change__data_ref_id__idx + RENAME TO file_change__data_ref_id__idx_legacy; +ALTER INDEX file_change__deleted_at__idx + RENAME TO file_change__deleted_at__idx_legacy; +ALTER INDEX file_change__file_id__revn__idx + RENAME TO file_change__file_id__revn__idx_legacy; +ALTER INDEX file_change__locked_by__idx + RENAME TO file_change__locked_by__idx_legacy; +ALTER INDEX file_change__profile_id__idx + RENAME TO file_change__profile_id__idx_legacy; +ALTER INDEX file_change__system_snapshots__idx + RENAME TO file_change__system_snapshots__idx_legacy; +ALTER INDEX IF EXISTS file_change__user_snapshots__idx + RENAME TO file_change__user_snapshots__idx_legacy; +ALTER INDEX IF EXISTS file_change__updated_at__idx + RENAME TO file_change__updated_at__idx_legacy; + +-- And the new table takes them. Only the primary key needs the rename: the +-- `CHECK` and the foreign keys already carry the canonical name (they have no +-- index, so they could take it from the start), and the partitions inherited +-- it. +ALTER TABLE file_change_new RENAME CONSTRAINT file_change_new_pkey + TO file_change_pkey; + +ALTER INDEX file_change_new__deleted_at__idx + RENAME TO file_change__deleted_at__idx; +ALTER INDEX file_change_new__file_id__revn__idx + RENAME TO file_change__file_id__revn__idx; +ALTER INDEX file_change_new__locked_by__idx + RENAME TO file_change__locked_by__idx; +ALTER INDEX file_change_new__profile_id__idx + RENAME TO file_change__profile_id__idx; +ALTER INDEX file_change_new__user_snapshots__idx + RENAME TO file_change__user_snapshots__idx; + +-- And the tables are swapped. The partitions and their indexes keep the +-- names they were born with (they are named after the partition, not after +-- the table), so there is nothing else to rename. +ALTER TABLE file_change RENAME TO file_change_legacy; +ALTER TABLE file_change_new RENAME TO file_change; + +-- The foreign key of the legacy table goes away: it is `NO ACTION`, so +-- while the legacy table has rows for a file, the objects GC cannot delete +-- that file (it deletes the rows of the file first, and the legacy ones are +-- not its business). The legacy table is frozen: nothing writes to it again. +ALTER TABLE file_change_legacy + DROP CONSTRAINT file_change_file_id_fkey_legacy; + +COMMIT; + +ANALYZE file_change; +``` + +- `file_change_legacy` keeps its data and its indexes (with the `_legacy` + suffix): a frozen copy of the previous layout. Its foreign key on `file_id` + is dropped, so the objects GC can delete the files again. +- The writes to `file_change` during the lock fail: they are seconds-minutes, + the affected saves are retried. +- The objects GC and the four RPC commands keep working: same table name and + same index names after the rename. +- Once the backend is up again, check that the saves go to the partitioned + table and that nothing went to the table that is going away: + +```sql +-- :cutover = the time the backend went up on the new table. Set it in this +-- session: without it the queries fail instead of comparing against a stale +-- date. +SET penpot.cutover = '2026-10-25 08:00:00+02'; + +-- the rows written after the cutover: this one grows with every save +SELECT count(*) FROM file_change + WHERE updated_at > current_setting('penpot.cutover')::timestamptz; + +-- the rows that went to the legacy table: this one has to stay at 0 +SELECT count(*) FROM file_change_legacy + WHERE updated_at > current_setting('penpot.cutover')::timestamptz; +``` + +A number above zero in the second query means that some connection was +still writing with a prepared statement of the old table: those rows are +lost when the legacy table is dropped, so they have to be copied over +before that. + +--- + +## Phase 5 — After the cutover + +1. Check that the access paths prune to one partition (when the planner + chooses an index scan): + +```sql +EXPLAIN SELECT * FROM file_change + WHERE file_id = '00000000-0000-0000-0000-000000000000' + AND id = '00000000-0000-0000-0000-000000000000'; +``` + +It has to show a single partition (`file_change_NN`), not an `Append` of 16. + +2. Check that the schema is the one of the migration `0158`. Every row of this + query has to show `got = expected`: + +```sql +SELECT 'columns' AS what, count(*) AS got, 17 AS expected + FROM pg_attribute + WHERE attrelid = 'file_change'::regclass AND attnum > 0 AND NOT attisdropped +UNION ALL +SELECT 'partitions', count(*), 16 + FROM pg_partition_tree('file_change') t WHERE t.isleaf +UNION ALL +SELECT 'primary key', count(*), 1 + FROM pg_constraint + WHERE conrelid = 'file_change'::regclass AND contype = 'p' +UNION ALL +SELECT 'foreign keys', count(*), 3 + FROM pg_constraint + WHERE conrelid = 'file_change'::regclass AND contype = 'f' +UNION ALL +SELECT 'check constraints', count(*), 1 + FROM pg_constraint + WHERE conrelid = 'file_change'::regclass AND contype = 'c' +UNION ALL +SELECT 'indexes', count(*), 6 + FROM pg_indexes + WHERE tablename = 'file_change' AND schemaname = current_schema() +UNION ALL +SELECT 'partitions out of name', count(*), 0 + FROM pg_class + WHERE relkind = 'r' AND relname ~ '^file_change_new[0-9]' +UNION ALL +SELECT 'child indexes out of name', count(*), 0 + FROM pg_class c + JOIN pg_index i ON i.indexrelid = c.oid + JOIN pg_class p ON p.oid = i.indrelid + WHERE p.relname ~ '^file_change_[0-9]{2}$' + AND c.relname NOT LIKE p.relname || '_%' +UNION ALL +SELECT 'partition constraints out of name', count(*), 0 + FROM pg_constraint c + JOIN pg_class t ON t.oid = c.conrelid + WHERE t.relname ~ '^file_change_[0-9]{2}$' + AND c.contype <> 'n' + AND c.conname LIKE 'file_change_new%'; +``` + + The `NOT NULL` constraints (`contype = 'n'`, tracked in `pg_constraint` from + PostgreSQL 17) are excluded from the last check: their name is generated by + the column and changes with the server version, so they are not part of the + parity. The `CHECK`, the primary key and the foreign keys are. + + And the names, to compare them with the migration: + +```sql +SELECT a.attname AS column + FROM pg_attribute a + WHERE a.attrelid = 'file_change'::regclass AND a.attnum > 0 + AND NOT a.attisdropped + ORDER BY a.attnum; + +SELECT indexname FROM pg_indexes + WHERE tablename = 'file_change' AND schemaname = current_schema() + ORDER BY indexname; + +-- the partitions, with their name +SELECT c.relname AS partition + FROM pg_class c + JOIN pg_partition_tree('file_change') t ON t.relid = c.oid + WHERE t.isleaf + ORDER BY c.relname; + +-- the indexes of the partitions: their name starts with the partition +SELECT c.relname AS child_index + FROM pg_class c + JOIN pg_index i ON i.indexrelid = c.oid + JOIN pg_class p ON p.oid = i.indrelid + WHERE p.relname ~ '^file_change_[0-9]{2}$' + ORDER BY c.relname; + +-- every table, partition and index of the family shares the owner of the +-- table: a session without `SET ROLE` (0.0) shows here +SELECT count(*) AS out_of_owner + FROM pg_class c + WHERE c.relname LIKE 'file_change%' + AND c.relkind IN ('r', 'p', 'i') + AND pg_get_userbyid(c.relowner) <> + (SELECT tableowner FROM pg_tables + WHERE schemaname = current_schema() AND tablename = 'file_change'); +``` + + The owner check has to return `0`. If it does not, the session that + created the table did not use `SET ROLE`: fix it before opening the + service (re-own the table, the 16 partitions and their indexes) and + re-apply the grants saved in 0.6. + +3. The partition-wise join is already on: the pool of the backend sets + `enable_partitionwise_join = on` on every connection (it is in the code, + there is nothing to change in the database role). + +4. Autovacuum: each of the 16 partitions has its own TOAST (check their + sizes with `pg_total_relation_size` per partition); with + `autovacuum_vacuum_scale_factor = 0.2` the threshold is reached earlier + and in steps. Nothing to change, just watch `last_autovacuum` of the + partitions the first week. + +5. The aggregations that do not filter by `file_id` (the team snapshot + quota and the telemetry counts) read all 16 partitions. To know their + cost on your instance, time them before and after with + `EXPLAIN (ANALYZE, BUFFERS)` and compare: + + - The team quota is already fixed in the code: it resolves the `file_id` of + the team first, which lets the planner prune the partitions when it + chooses an index scan (with a sequential scan it reads all of them). + - The telemetry scans the whole heap, the same before and after: the + partitioned run pays a small cost per partition on top. To remove that + scan, the index `file_change__created_at__idx` turns both metrics into an + index scan over the rows of the day, at the cost of one more index on the + biggest table. Decide it before phase 1, the index has to exist in both + tables. + +6. Drop the helper table of the copy and, days later, the old table: + +```sql +DROP TABLE IF EXISTS file_change_copy_state; +DROP TABLE file_change_legacy; -- out of hours: it generates WAL by + -- dropping the pages, let it roll +``` + +That is where the dead space comes back: the database shrinks by about the +size of the old table (see sizing). + +--- + +## Risks + +| Risk | Mitigation | +|---|---| +| Massive WAL during the bulk (proportional to the copy; see sizing) | Throttle between batches; watch `pg_wal`, the replication slots and the replica lag | +| Latency seen by the application during the bulk | Throttle; low traffic hours; the bulk is I/O and does not block writes | +| `updated_at` not reliable in the rows changed before the deploy | The deploy goes before the bulk (0.3) and the delta passes cover from the start of the bulk | +| Lock/unlock/soft delete during the passes | Covered by the watermark; the cutover includes the final delta inside the lock | +| Hard delete of the GC between passes | The row is left with `deleted_at` in the new table; the GC cleans it up later | +| Orphan rows if a `file` is deleted halfway | The foreign key on `file_id` is `NO ACTION`: the GC deletes the rows of the file before the file, so a chunk that fails leaves the file in the trash and the next run deletes it | +| The files in the trash cannot be deleted while the legacy table has rows for them | The foreign key of the legacy table is dropped at the cutover; until then the files wait in the trash | +| Saves failing during the lock of the cutover | Freeze of minutes in a low traffic window; manual retries | +| Planner regression after the cutover | Immediate `ANALYZE` + the pruning `EXPLAIN`; the pool already sets the GUC | +| Clock drift between the application and the database | The delta window starts an hour before the bulk; the margin absorbs the drift | + +## Success metrics + +- After the cutover: `file_change` (partitioned) holds at most the + `pg_table_size` of the old table (see sizing). +- After the `DROP` of the old table: the database shrinks by about the size + that table had. +- The access paths that filter by `file_id` (the lookup of a snapshot, the + quota of a file and the lock of the GC by pair) prune to one partition when + the planner chooses an index scan (`EXPLAIN`). The quota of the team and + the discovery of the GC do not filter by a single file: they read all the + 16 partitions. +- No visible change for the user. diff --git a/operations/README.md b/operations/README.md new file mode 100644 index 0000000000..9590e8e865 --- /dev/null +++ b/operations/README.md @@ -0,0 +1,16 @@ +# Operations + +Procedures that are executed by hand on an instance, not by the application. + +They are not part of the product: they describe how to do a one-off operation +on a running database, with the exact statements to run and what to check +between them. + +Each procedure lives here while the operation is alive. When the operation is +done, the file is either deleted or marked as historical, so the folder does +not grow with procedures that nobody will run again. + +| Procedure | Version | +|---|---| +| [Migrate legacy file payloads to `file_data`](2.19.0-migrate-legacy-file-data.md) | 2.19.0 | +| [Partition `file_change` by `file_id`](2.20.0-file-change-partition.md) | 2.20.0 | From c466c289dee80f78a547389648de7b204430ea8a Mon Sep 17 00:00:00 2001 From: Pablo Alba Date: Tue, 10 Nov 2026 10:16:47 +0100 Subject: [PATCH 03/29] :bug: Fix integrity error copying variants across files --- common/src/app/common/files/variant.cljc | 19 ++ common/src/app/common/logic/libraries.cljc | 22 +- .../app/common/logic/variant_properties.cljc | 18 +- .../test/common_tests/files/variant_test.cljc | 49 +++++ .../logic/variants_paste_test.cljc | 202 ++++++++++++++++++ common/test/common_tests/runner.cljc | 4 + .../app/main/data/workspace/clipboard.cljs | 9 +- 7 files changed, 313 insertions(+), 10 deletions(-) create mode 100644 common/test/common_tests/files/variant_test.cljc create mode 100644 common/test/common_tests/logic/variants_paste_test.cljc diff --git a/common/src/app/common/files/variant.cljc b/common/src/app/common/files/variant.cljc index 989a80a58b..802c90fd1c 100644 --- a/common/src/app/common/files/variant.cljc +++ b/common/src/app/common/files/variant.cljc @@ -28,6 +28,25 @@ (map #(ctcl/get-component data % true)) reverse))) +(defn collect-variant-properties + "Map of component-id → variant-properties for clipboard copy. + Variant container: every related component. Variant main: that + component only. Anything else: empty map." + [data objects shape] + (cond + (ctc/is-variant-container? shape) + (into {} + (map (juxt :id :variant-properties)) + (find-variant-components data objects (:id shape))) + + (ctc/is-variant-main? shape) + (if-let [component (ctcl/get-component data (:component-id shape) true)] + {(:id component) (:variant-properties component)} + {}) + + :else + {})) + (defn extract-properties-names [shape data] (->> shape diff --git a/common/src/app/common/logic/libraries.cljc b/common/src/app/common/logic/libraries.cljc index 940f6aa055..16bf550dcf 100644 --- a/common/src/app/common/logic/libraries.cljc +++ b/common/src/app/common/logic/libraries.cljc @@ -2706,7 +2706,8 @@ page-id nil variant-id - props))] + props + {:apply-changes-local-library? true}))] [root-shape changes])) (defn generate-add-component @@ -3218,6 +3219,25 @@ (and is-component-root? is-component-main?) (regenerate-component new-obj)) + ;; Unknown-component paste into a variant set: align props/names + ;; after regenerate. Reload the shape so we use the new component-id. + ;; Prefit the component name/path to the container so + ;; create-new-properties-from-variant does not treat the component + ;; full name as a former set name and inflate properties (add-name?). + changes (cond-> changes + (and is-component-root? + is-component-main? + (ctk/is-variant-container? parent)) + (as-> $ + (let [shape (get (pcb/get-objects $) (:id new-obj)) + [cpath cname] (cpn/split-group-name (:name parent)) + changes (pcb/update-component + $ + (:component-id shape) + #(assoc % :name cname :path cpath) + {:apply-changes-local-library? true})] + (clvp/generate-make-shapes-variant changes [shape] parent)))) + ;; This is needed for the recursive call to find the new object as parent page' (ctst/add-shape (:id new-obj) new-obj diff --git a/common/src/app/common/logic/variant_properties.cljc b/common/src/app/common/logic/variant_properties.cljc index 3e950a81b8..e7be7842ef 100644 --- a/common/src/app/common/logic/variant_properties.cljc +++ b/common/src/app/common/logic/variant_properties.cljc @@ -247,10 +247,20 @@ :name (:name variant-container)))] (reduce (fn [changes shape] - (let [component (ctcl/get-component data (:component-id shape) true)] - (if (or (zero? num-shapes) ;; do nothing if there are no shapes - (and (= variant-id (:variant-id shape)) ;; or we are only moving the shape inside its parent (it is - (not (:deleted component)))) ;; the same parent and the component isn't deleted) + (let [component (ctcl/get-component data (:component-id shape) true) + already-here? (= variant-id (:variant-id shape)) + prop-names (cfv/extract-properties-names shape data) + base-names (map :name base-props) + props-ok? (= prop-names base-names) + names-ok? (= (cpn/merge-path-item (:path component) (:name component)) + container-name) + variant-ok? (= (:variant-name shape) + (ctv/properties-to-name (:variant-properties component))) + aligned? (and props-ok? names-ok? variant-ok?)] + (if (or (zero? num-shapes) ;; do nothing if there are no shapes + (and already-here? ;; or already a fully aligned child of this container + aligned? + (not (:deleted component)))) changes (let [props (create-new-properties shape total-props) variant-name (ctv/properties-to-name props)] diff --git a/common/test/common_tests/files/variant_test.cljc b/common/test/common_tests/files/variant_test.cljc new file mode 100644 index 0000000000..fda97d565e --- /dev/null +++ b/common/test/common_tests/files/variant_test.cljc @@ -0,0 +1,49 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns common-tests.files.variant-test + (:require + [app.common.files.variant :as cfv] + [app.common.test-helpers.components :as thc] + [app.common.test-helpers.files :as thf] + [app.common.test-helpers.ids-map :as thi] + [app.common.test-helpers.shapes :as ths] + [app.common.test-helpers.variants :as thv] + [clojure.test :as t])) + +(t/use-fixtures :each thi/test-fixture) + +(t/deftest test-collect-variant-properties-for-main + (let [file (-> (thf/sample-file :file1) + (thv/add-variant :v01 :c01 :m01 :c02 :m02)) + page (thf/current-page file) + main (ths/get-shape file :m01) + comp (thc/get-component file :c01) + collected (cfv/collect-variant-properties (:data file) (:objects page) main)] + + (t/is (= {(:id comp) (:variant-properties comp)} collected)))) + +(t/deftest test-collect-variant-properties-for-container + (let [file (-> (thf/sample-file :file1) + (thv/add-variant :v01 :c01 :m01 :c02 :m02)) + page (thf/current-page file) + container (ths/get-shape file :v01) + c01 (thc/get-component file :c01) + c02 (thc/get-component file :c02) + collected (cfv/collect-variant-properties (:data file) (:objects page) container)] + + (t/is (= {(:id c01) (:variant-properties c01) + (:id c02) (:variant-properties c02)} + collected)))) + +(t/deftest test-collect-variant-properties-for-plain-shape + (let [file (-> (thf/sample-file :file1) + (ths/add-sample-shape :rect1 :type :rect)) + page (thf/current-page file) + shape (ths/get-shape file :rect1) + collected (cfv/collect-variant-properties (:data file) (:objects page) shape)] + + (t/is (= {} collected)))) diff --git a/common/test/common_tests/logic/variants_paste_test.cljc b/common/test/common_tests/logic/variants_paste_test.cljc new file mode 100644 index 0000000000..020b8dcd7e --- /dev/null +++ b/common/test/common_tests/logic/variants_paste_test.cljc @@ -0,0 +1,202 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns common-tests.logic.variants-paste-test + (:require + [app.common.data :as d] + [app.common.files.changes-builder :as pcb] + [app.common.files.helpers :as cfh] + [app.common.files.validate :as cfv] + [app.common.geom.point :as gpt] + [app.common.logic.libraries :as cll] + [app.common.logic.variant-properties :as clvp] + [app.common.test-helpers.components :as thc] + [app.common.test-helpers.compositions :as tho] + [app.common.test-helpers.files :as thf] + [app.common.test-helpers.ids-map :as thi] + [app.common.test-helpers.shapes :as ths] + [app.common.test-helpers.variants :as thv] + [app.common.types.component :as ctk] + [app.common.types.variant :as ctv] + [clojure.test :as t])) + +(t/use-fixtures :each thi/test-fixture) + +(defn- paste-variants-into-container + "Paste individual variants from file-a into file-b's variant container. + Omit file-a from `libraries` to hit the unknown-component path; include + it to hit `duplicate-variant`. Optional `variant-props` mirrors clipboard + collection (Task 2)." + [file-a file-b source-labels target-container-label + & {:keys [libraries variant-props] + :or {libraries {(:id file-b) file-b}}}] + (let [page-a (thf/current-page file-a) + page-b (thf/current-page file-b) + container-b (ths/get-shape file-b target-container-label) + parent-id (:id container-b) + + source-ids (mapv thi/id source-labels) + tree-ids (into [] + (mapcat #(cfh/get-children-ids-with-self (:objects page-a) %)) + source-ids) + + pasted (->> tree-ids + (map (d/getf (:objects page-a))) + (map (fn [shape] + (cond-> shape + (contains? (set source-ids) (:id shape)) + (assoc :parent-id parent-id + :frame-id parent-id)))) + (d/index-by :id)) + + all-objects (merge (:objects page-b) pasted)] + (-> (pcb/empty-changes nil) + (pcb/with-page-id (:id page-b)) + (pcb/with-library-data (:data file-b)) + (pcb/with-objects all-objects) + (cll/generate-duplicate-changes all-objects + page-b + (set source-ids) + (gpt/point 0 0) + libraries + (:data file-b) + (:id file-b) + {:variant-props variant-props})))) + +(defn- property-names + [component] + (mapv :name (:variant-properties component))) + +(defn- variant-mains-in + [file container-label] + (let [container (ths/get-shape file container-label)] + (->> (vals (:objects (thf/current-page file))) + (filter ctk/main-instance?) + (filter #(= (:id container) (:variant-id %)))))) + +(defn- other-set-file + [] + (-> (thf/sample-file :file-b) + (thv/add-variant-two-properties :vb :cb1 :mb1 :cb2 :mb2) + (ths/update-shape :vb :name "Other") + (ths/update-shape :mb1 :name "Other") + (ths/update-shape :mb2 :name "Other") + (thc/update-component :cb1 {:name "Other"}) + (thc/update-component :cb2 {:name "Other"}))) + +(t/deftest test-paste-unknown-variant-into-container-validates + ;; #10704: paste individual variants from file A into file B's set + ;; when A is not in libraries must leave a valid file (desired behavior). + (let [file-a (-> (thf/sample-file :file-a) + (thv/add-variant :va :ca1 :ma1 :ca2 :ma2)) + file-b (other-set-file) + changes (paste-variants-into-container file-a file-b [:ma1 :ma2] :vb) + file' (thf/apply-changes file-b changes :validate? false) + errors (cfv/validate-file file' {}) + pasted (->> (variant-mains-in file' :vb) + (remove #(#{(thi/id :mb1) (thi/id :mb2)} (:id %))))] + + (t/is (nil? errors)) + (doseq [shape pasted] + (let [comp (thc/get-component-by-id file' (:component-id shape))] + (t/is (= ["Property 1" "Property 2"] (property-names comp))) + (t/is (= "Other" (:name shape))) + (t/is (= (:variant-name shape) + (ctv/properties-to-name (:variant-properties comp)))))))) + +(t/deftest test-duplicate-variant-in-own-container-bumps-last-value + ;; Same-container duplicate via duplicate-variant: originals keep their + ;; props; the new main gets a bumped last-property value. Does not cover + ;; generate-make-shapes-variant's already-aligned early-exit (new mains + ;; enter with a nil variant-id). + (let [file (-> (thf/sample-file :file1) + (thv/add-variant :v01 :c01 :m01 :c02 :m02)) + c01-props (:variant-properties (thc/get-component file :c01)) + c02-props (:variant-properties (thc/get-component file :c02)) + file' (tho/duplicate-shape file :m01) + mains (variant-mains-in file' :v01) + new-main (->> mains + (remove #(#{(thi/id :m01) (thi/id :m02)} (:id %))) + first) + new-comp (thc/get-component-by-id file' (:component-id new-main))] + + (thf/validate-file! file') + (t/is (= 3 (count mains))) + (t/is (= c01-props (:variant-properties (thc/get-component file' :c01)))) + (t/is (= c02-props (:variant-properties (thc/get-component file' :c02)))) + (t/is (= ["Property 1"] (property-names new-comp))) + (t/is (= (str ctv/value-prefix 3) + (-> new-comp :variant-properties last :value))))) + +(t/deftest test-make-shapes-variant-skips-when-already-aligned + ;; Locks the narrowed early-exit: a fully aligned child must not have its + ;; props or variant-name rewritten. + (let [file (-> (thf/sample-file :file1) + (thv/add-variant :v01 :c01 :m01 :c02 :m02)) + page (thf/current-page file) + container (ths/get-shape file :v01) + shape (ths/get-shape file :m01) + before (:variant-properties (thc/get-component file :c01)) + before-vn (:variant-name shape) + changes (-> (pcb/empty-changes nil) + (pcb/with-page page) + (pcb/with-objects (:objects page)) + (pcb/with-library-data (:data file)) + (clvp/generate-make-shapes-variant [shape] container)) + file' (thf/apply-changes file changes) + shape' (ths/get-shape-by-id file' (:id shape))] + + (t/is (= before (:variant-properties (thc/get-component file' :c01)))) + (t/is (= before-vn (:variant-name shape'))))) + +(t/deftest test-paste-known-variant-into-other-container-aligns + ;; file A present in libraries → duplicate-variant path; props must match + ;; the target set and the file must validate. + (let [file-a (-> (thf/sample-file :file-a) + (thv/add-variant :va :ca1 :ma1 :ca2 :ma2)) + file-b (other-set-file) + libraries {(:id file-a) file-a + (:id file-b) file-b} + changes (paste-variants-into-container file-a file-b [:ma1 :ma2] :vb + :libraries libraries) + file' (thf/apply-changes file-b changes :validate? false) + errors (cfv/validate-file file' {}) + pasted (->> (variant-mains-in file' :vb) + (remove #(#{(thi/id :mb1) (thi/id :mb2)} (:id %))))] + + (t/is (nil? errors)) + (t/is (= 2 (count pasted))) + (doseq [shape pasted] + (let [comp (thc/get-component-by-id file' (:component-id shape))] + (t/is (= ["Property 1" "Property 2"] (property-names comp))) + (t/is (= "Other" (:name shape))) + (t/is (= (:variant-name shape) + (ctv/properties-to-name (:variant-properties comp)))))))) + +(t/deftest test-paste-unknown-preserves-matching-property-values + ;; Clipboard props (Task 2) whose names exist on the target survive regenerate. + (let [file-a (-> (thf/sample-file :file-a) + (thv/add-variant :va :ca1 :ma1 :ca2 :ma2)) + file-b (other-set-file) + props {(:id (thc/get-component file-a :ca1)) + (:variant-properties (thc/get-component file-a :ca1)) + (:id (thc/get-component file-a :ca2)) + (:variant-properties (thc/get-component file-a :ca2))} + changes (paste-variants-into-container file-a file-b [:ma1 :ma2] :vb + :variant-props props) + file' (thf/apply-changes file-b changes :validate? false) + pasted (->> (variant-mains-in file' :vb) + (remove #(#{(thi/id :mb1) (thi/id :mb2)} (:id %))) + (map #(thc/get-component-by-id file' (:component-id %)))) + values (->> pasted + (mapcat :variant-properties) + (filter #(= "Property 1" (:name %))) + (map :value) + set)] + + (thf/validate-file! file') + (t/is (contains? values "Value1")) + (t/is (contains? values "Value2")))) diff --git a/common/test/common_tests/runner.cljc b/common/test/common_tests/runner.cljc index 86044dbf5f..5d24e4ea9f 100644 --- a/common/test/common_tests/runner.cljc +++ b/common/test/common_tests/runner.cljc @@ -30,6 +30,7 @@ [common-tests.files.shapes-builder-test] [common-tests.files.tokens-test] [common-tests.files.validate-test] + [common-tests.files.variant-test] [common-tests.geom-align-test] [common-tests.geom-bounds-layout-nil-test] [common-tests.geom-bounds-map-test] @@ -79,6 +80,7 @@ [common-tests.logic.text-touched-test] [common-tests.logic.token-apply-test] [common-tests.logic.token-test] + [common-tests.logic.variants-paste-test] [common-tests.logic.variants-reparent-test] [common-tests.logic.variants-switch-test] [common-tests.logic.variants-test] @@ -140,6 +142,7 @@ 'common-tests.files.shapes-builder-test 'common-tests.files.tokens-test 'common-tests.files.validate-test + 'common-tests.files.variant-test 'common-tests.geom-align-test 'common-tests.geom-bounds-layout-nil-test 'common-tests.geom-bounds-map-test @@ -189,6 +192,7 @@ 'common-tests.logic.text-touched-test 'common-tests.logic.token-apply-test 'common-tests.logic.token-test + 'common-tests.logic.variants-paste-test 'common-tests.logic.variants-reparent-test 'common-tests.logic.variants-switch-test 'common-tests.logic.variants-test diff --git a/frontend/src/app/main/data/workspace/clipboard.cljs b/frontend/src/app/main/data/workspace/clipboard.cljs index d763e48e2e..7f9bc188bb 100644 --- a/frontend/src/app/main/data/workspace/clipboard.cljs +++ b/frontend/src/app/main/data/workspace/clipboard.cljs @@ -157,10 +157,8 @@ (let [page-id (:current-page-id state) data (dsh/lookup-file-data state) objects (-> (dsh/get-page data page-id) - (get :objects)) - - components (cfv/find-variant-components data objects (:id shape))] - (into {} (map (juxt :id :variant-properties) components)))) + (get :objects))] + (cfv/collect-variant-properties data objects shape))) ;; Collects all the items together and split images into a @@ -176,7 +174,8 @@ (some? images) (update :images into images) - (ctk/is-variant-container? item) + (or (ctk/is-variant-container? item) + (ctk/is-variant-main? item)) (update :variant-properties merge (collect-variants state item)))) (maybe-translate [shape objects parent-frame-id] From f89879cca16ff19a412290006dbaebe3f0d9eb00 Mon Sep 17 00:00:00 2001 From: Alonso Torres Date: Wed, 11 Nov 2026 09:39:04 +0100 Subject: [PATCH 04/29] :bug: Fix plugin component lookups for missing and deleted components (#12755) * :bug: Fix plugin component lookups for missing and deleted components Shape.component() threw "Assert failed: (uuid? id)" on a copy whose library is not available, because the head lookup returns a pair with a nil component and the proxy was built anyway. It now returns null. Shape.componentRoot() built its proxy with the component's file and main page, so a copy on another page or file than its main got a shape that pointed at nothing, and a copy of a missing library threw. It now resolves the root on the shape's own file and page, like componentHead(). LibraryComponent.mainInstance() returned a proxy for the root kept in a deleted component's objects, which is on no page, so every field read empty and parent threw. It now returns null for a deleted component. * :books: Fix the variant example in the MCP instructions The createVariantContainer example called penpot.currentPage.findAllShapes and Shape.isMainComponent, which do not exist, so running it as written failed. Use penpotUtils.findShapes with isComponentMainInstance instead. --- frontend/src/app/plugins/library.cljs | 3 +- frontend/src/app/plugins/shape.cljs | 10 +++-- .../frontend_tests/plugins/library_test.cljs | 40 +++++++++++++++++++ mcp/packages/plugin/src/PenpotUtils.ts | 2 +- .../server/data/initial_instructions.md | 2 +- plugins/CHANGELOG.md | 29 +++++++++----- 6 files changed, 70 insertions(+), 16 deletions(-) diff --git a/frontend/src/app/plugins/library.cljs b/frontend/src/app/plugins/library.cljs index faec7fad16..8fccd73b83 100644 --- a/frontend/src/app/plugins/library.cljs +++ b/frontend/src/app/plugins/library.cljs @@ -1022,7 +1022,8 @@ (let [file (u/locate-file file-id) component (u/locate-library-component file-id id) root (ctf/get-component-root (:data file) component)] - (when (some? root) + ;; A deleted component keeps its root only in its own objects, not on a page + (when (and (some? root) (not (:deleted component))) (shape/shape-proxy plugin-id file-id (:main-instance-page component) (:id root))))) :isVariant diff --git a/frontend/src/app/plugins/shape.cljs b/frontend/src/app/plugins/shape.cljs index a313ae1ee6..3e771b5852 100644 --- a/frontend/src/app/plugins/shape.cljs +++ b/frontend/src/app/plugins/shape.cljs @@ -1535,8 +1535,8 @@ (let [objects (u/locate-objects file-id page-id) shape (u/locate-shape file-id page-id id)] (when (ctn/in-any-component? objects shape) - (let [[root component] (u/locate-component objects shape)] - (shape-proxy plugin-id (:component-file root) (:main-instance-page component) (:id root)))))) + (let [root (ctn/get-instance-root objects shape)] + (shape-proxy plugin-id file-id page-id (:id root)))))) :componentHead (fn [] @@ -1551,8 +1551,10 @@ (let [objects (u/locate-objects file-id page-id) shape (u/locate-shape file-id page-id id)] (when (ctn/in-any-component? objects shape) - (when-let [[head component] (u/locate-head-component objects shape)] - (lib-component-proxy plugin-id (:component-file head) (:id component)))))) + (let [[head component] (u/locate-head-component objects shape)] + ;; The component is nil when its library is not available + (when (some? component) + (lib-component-proxy plugin-id (:component-file head) (:id component))))))) :detach (fn [] diff --git a/frontend/test/frontend_tests/plugins/library_test.cljs b/frontend/test/frontend_tests/plugins/library_test.cljs index 982a62c566..1e798f7a78 100644 --- a/frontend/test/frontend_tests/plugins/library_test.cljs +++ b/frontend/test/frontend_tests/plugins/library_test.cljs @@ -203,6 +203,46 @@ (await (async/settle)) (t/is (empty? @events)))))))) +(t/deftest ^:async main-instance-of-a-deleted-component-is-null + (let [file (file-with-deleted-component) + store (ths/setup-store file {:renderer :svg})] + (await + (mock/with-mocks* + {st/state store + shape/lib-component-proxy library/lib-component-proxy} + (let [copy (shape/shape-proxy plugin-id (:id file) (thi/id :main-page) (thi/id :copy)) + component (.component copy)] + (t/is (some? component)) + (t/is (nil? (.mainInstance component)))))))) + +(t/deftest ^:async component-root-of-a-copy-on-another-page-than-its-main + (let [file (-> (cthf/sample-file :file :page-label :main-page) + (cths/add-sample-shape :main :type :frame) + (cths/add-sample-shape :child :parent-label :main) + (cthc/make-component :component :main) + (cthf/add-sample-page :other-page) + (cthc/instantiate-component :component :copy :children-labels [:copy-child])) + store (ths/setup-store file {:renderer :svg})] + (await + (mock/with-mocks* + {st/state store} + (let [copy-child (shape/shape-proxy plugin-id (:id file) (thi/id :other-page) (thi/id :copy-child)) + root (.componentRoot copy-child)] + (t/is (= (str (thi/id :copy)) (.-id root))) + (t/is (= (str (thi/id :copy)) (.-id (.componentHead copy-child))))))))) + +(t/deftest ^:async component-of-a-copy-whose-library-is-missing-is-null + (let [file (-> (file-with-component) + (cths/update-shape :copy :component-file (uuid/next))) + store (ths/setup-store file {:renderer :svg})] + (await + (mock/with-mocks* + {st/state store + shape/lib-component-proxy library/lib-component-proxy} + (let [copy (shape/shape-proxy plugin-id (:id file) (thi/id :main-page) (thi/id :copy))] + (t/is (nil? (.component copy))) + (t/is (= (str (thi/id :copy)) (.-id (.componentRoot copy))))))))) + (t/deftest library-asset-proxies-expose-library-id (let [file-id (random-uuid) id (random-uuid)] diff --git a/mcp/packages/plugin/src/PenpotUtils.ts b/mcp/packages/plugin/src/PenpotUtils.ts index f856f14cc4..762e75c3dd 100644 --- a/mcp/packages/plugin/src/PenpotUtils.ts +++ b/mcp/packages/plugin/src/PenpotUtils.ts @@ -539,7 +539,7 @@ export class PenpotUtils { * @example * // Three button variants differing in Size * // Given three main components s, m, l (here: the first three main components on the page) - * const [s, m, l] = penpot.currentPage.findAllShapes(sh => sh.isMainComponent()).slice(0, 3) as Board[]; + * const [s, m, l] = PenpotUtils.findShapes(sh => sh.isComponentMainInstance(), penpot.root).slice(0, 3) as Board[]; * const container = PenpotUtils.createVariantContainer([ * { shape: s, properties: { Size: 'Small' } }, * { shape: m, properties: { Size: 'Medium' } }, diff --git a/mcp/packages/server/data/initial_instructions.md b/mcp/packages/server/data/initial_instructions.md index fbca1c5bad..d126811742 100644 --- a/mcp/packages/server/data/initial_instructions.md +++ b/mcp/packages/server/data/initial_instructions.md @@ -352,7 +352,7 @@ Properties are often addressed positionally: `pos` parameter in various methods Use `penpotUtils.createVariantContainer(components)` — it handles the full multi-step workflow in one call: ```js // Given three main components s, m, l (here: the first three main components on the page) -const [s, m, l] = penpot.currentPage.findAllShapes(sh => sh.isMainComponent()).slice(0, 3); +const [s, m, l] = penpotUtils.findShapes(sh => sh.isComponentMainInstance(), penpot.root).slice(0, 3); // Single property: const container = penpotUtils.createVariantContainer([ { shape: s, properties: { Size: 'Small' } }, diff --git a/plugins/CHANGELOG.md b/plugins/CHANGELOG.md index 5a6b8cd350..3ee7edcfc4 100644 --- a/plugins/CHANGELOG.md +++ b/plugins/CHANGELOG.md @@ -1,17 +1,10 @@ -## 1.6.0 (2026-11-10) +## 1.7.0 (Unreleased) ### 🚀 Features - **plugin-types, plugins-runtime:** `penpotMgmt` returns `Team`, `Project` and `ProjectFile` objects. Assigning `name` renames teams, projects and files, `project.pinned` pins a project and `file.shared` shares a file as a team library; `createProject`, `createFile`, `moveTo`, `duplicate`, `remove` and `restore` live on the objects. Added `penpotMgmt.listTeams`, `penpotMgmt.createTeam`, `penpotMgmt.getFile`, `listDeletedProjects` and `listDeletedFiles`, and the `manage:projects`, `manage:teams` and `manage:delete` permissions. Restoration uses the dashboard's Trash operation; projects need recoverable files and empty projects cannot be restored; the API exposes no permanent deletion method. Team deletion is not exposed. - **plugins-runtime, plugin-types**: Added global plugins (`scope: "global"`) that remain active across dashboard and workspace navigation until closed or logged out, with `penpotMgmt` for project and file discovery, file navigation and workspace events. Global scope adds the `allow:global` permission, which users must accept when installing the plugin. -- **plugin-types:** Added `LibraryComponent.restore()` to restore a deleted component and its main instance without changing the current page, selection, zoom, or viewport. Requires `library:write` permission and only applies to the current file's library. -- **plugin-types:** Added `Shape.updateMain()`, `LibraryComponent.duplicate()`, writable component `annotation`, and `Library.deletedComponents`. Declared the existing `LibraryComponent.remove()` method. -- **plugin-types:** Added `LibraryContext.tokensSource` to read or set the current file's token source to a connected library, or restore local tokens. -- **plugin-types:** Added `paddingType` (`'simple' | 'multiple'`) to flex and grid layouts and `marginType` (`'simple' | 'multiple'`) to layout children, exposing whether the four padding/margin sides are mirrored or honoured independently. -- **plugin-types**: Added `waitForLayoutUpdate` to wait until pending layout updates have finished. It rejects when the optional timeout elapses, defaulting to 30 seconds so a wait never hangs. -- **plugin-types**: Added `waitForLayoutUpdate` to the `Shape` interface to wait until the pending layout updates of a shape and its children have finished - **plugin-types:** Added the `contentrollback` event and the `ContentRollback` type. The event fires after the backend rejected some changes and the workspace undid them, with the ids of the files, pages and shapes affected. It fires once the undone data is in the workspace, also when the workspace has to reload the file to undo the changes. -- **plugin-types:** `Stroke` exposes per-side widths for boards and rectangles, also supported by `borderWidth` and `dimension` tokens. Read-only `strokeWidthType` reports `simple` when all four effective widths match, otherwise `multiple`. ### 🩹 Fixes @@ -21,6 +14,25 @@ - **plugins-runtime**: `library.connectLibrary()` now resolves once the library's assets are loaded; it used to resolve with an empty library the first time. `library.connected` and `library.availableLibraries()` ignore files opened earlier in the same tab, which used to show up as connected and be left out of the available libraries. - **plugins-runtime:** Writes now fail with an error while the workspace cannot take them (saves failing, a user who cannot edit, a version preview, a canvas that stopped rendering), whatever the `throwValidationErrors` flag. They used to change the workspace with edits that would never be saved. The error says why. The Inspect tab does not block writes, and undo blocks still open and close. - **plugins-runtime:** Writes that wait for the server (`createShapeFromSvgWithImages`, `uploadMediaData`, `uploadMediaUrl`, `connectLibrary`) check again when the server answers. If the workspace turned read-only meanwhile (a save failed), the promise rejects with the read-only error: no shape is added and no library is linked. An image the server already stored stays in the file's media, unused. +- **plugins-runtime:** `Shape.component()` returns `null` for a copy whose library is not available, instead of throwing `Assert failed: (uuid? id)`. +- **plugins-runtime:** `Shape.componentRoot()` returns the root of the shape's own component tree. It used to return a shape that pointed at nothing when the copy was on another page or file than its main, and threw when the copy's library was not available. +- **plugins-runtime:** `LibraryComponent.mainInstance()` returns `null` for a deleted component, instead of a shape with an empty `id` whose `parent` throws. +- **plugins-runtime**: `TokenFontFamilies.resolvedValue` and the `fontFamilies` member of `TokenTypographyValue` now keep a family whose name has several words (`Hanken Grotesk`) as one entry, instead of splitting it into an array of its words, and keep names with words the resolver would otherwise read as colors or numbers (`Red Hat Display`, `Press Start 2P`) unchanged. + +## 1.6.0 (2026-11-10) + +### 🚀 Features + +- **plugin-types:** Added `LibraryComponent.restore()` to restore a deleted component and its main instance without changing the current page, selection, zoom, or viewport. Requires `library:write` permission and only applies to the current file's library. +- **plugin-types:** Added `Shape.updateMain()`, `LibraryComponent.duplicate()`, writable component `annotation`, and `Library.deletedComponents`. Declared the existing `LibraryComponent.remove()` method. +- **plugin-types:** Added `LibraryContext.tokensSource` to read or set the current file's token source to a connected library, or restore local tokens. +- **plugin-types:** Added `paddingType` (`'simple' | 'multiple'`) to flex and grid layouts and `marginType` (`'simple' | 'multiple'`) to layout children, exposing whether the four padding/margin sides are mirrored or honoured independently. +- **plugin-types**: Added `waitForLayoutUpdate` to wait until pending layout updates have finished. It rejects when the optional timeout elapses, defaulting to 30 seconds so a wait never hangs. +- **plugin-types**: Added `waitForLayoutUpdate` to the `Shape` interface to wait until the pending layout updates of a shape and its children have finished +- **plugin-types:** `Stroke` exposes per-side widths for boards and rectangles, also supported by `borderWidth` and `dimension` tokens. Read-only `strokeWidthType` reports `simple` when all four effective widths match, otherwise `multiple`. + +### 🩹 Fixes + - **plugins-runtime:** `LibraryComponent.remove()` now rejects linked-library and missing component handles, and does nothing for an already deleted component. - **plugins-runtime**: An interaction obtained from `Shape.interactions` now keeps addressing that interaction instead of the position it held when the array was read. Removing every interaction of a shape from a single read removes all of them rather than leaving some behind, and writing through a held interaction after an earlier one is removed no longer lands on a different interaction. - **plugins-runtime**: `Shape.removeInteraction()` now rejects an interaction belonging to a different shape with a validation error, instead of removing whichever interaction sat at the same position on the target shape. @@ -30,7 +42,6 @@ - **plugins-runtime**: Removed the premature deep-hardening of the host plugin context, which froze shared host functions (including `Function.prototype`) before SES override taming, causing `TypeError: Cannot assign to read only property 'toString'` on later host-side function extension. Related to #11001. - **plugins-runtime**: The plugin modal no longer shows two resize grips in Firefox. Firefox now shows only its native grip. Closes #11795. - **plugins-runtime**: Fixed the `fontFamilies` token property mapping so `Shape.applyToken(token, ["fontFamilies"])` resolves to the canonical `:font-family` attribute and applied-token readback exposes the documented `fontFamilies` key instead of the undocumented singular `fontFamily`. Closes #11405. -- **plugins-runtime**: `TokenFontFamilies.resolvedValue` and the `fontFamilies` member of `TokenTypographyValue` now keep a family whose name has several words (`Hanken Grotesk`) as one entry, instead of splitting it into an array of its words, and keep names with words the resolver would otherwise read as colors or numbers (`Red Hat Display`, `Press Start 2P`) unchanged. - **plugins-runtime**: Opening a plugin no longer drops running background plugins (`allowBackground`, such as the MCP plugin) from the runtime registry. They kept running, but their UI messages were no longer delivered to the plugin and they could no longer be unloaded, so MCP tasks timed out after another plugin was opened. - **plugins-runtime**: `Shape.applyToken()`, `Token.applyToShapes()` and `Token.applyToSelected()` now reject an explicit per-side stroke width property (`strokeWidthTop`, `strokeWidthRight`, `strokeWidthBottom`, `strokeWidthLeft`) on shapes other than boards and rectangles, matching the `strokes` setter. The uniform `strokeWidth` property keeps working on every shape. Closes #12319. From 37f4556b0b9bdf0afa5891a75e161e71d0715e84 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?andr=C3=A9s=20gonz=C3=A1lez?= Date: Wed, 11 Nov 2026 09:54:27 +0100 Subject: [PATCH 05/29] :books: Document MCP sign-in with OAuth (#12950) Explain how MCP clients with OAuth support connect to the remote MCP server with a URL that has no key: the user signs in to Penpot, allows access, and the client receives the MCP key. Keep the key-in-URL path for clients and instances without sign-in. Describe the approval page, what allowing access changes, and how regenerating or expired keys affect signed-in clients. Document PENPOT_MCP_AUTH_URI for self-hosted instances. Relates to #12034 AI-assisted-by: claude-opus-5-5 --- docs/mcp/index.md | 61 ++++++++++++++++++++++----- docs/technical-guide/configuration.md | 27 ++++++++++++ 2 files changed, 77 insertions(+), 11 deletions(-) diff --git a/docs/mcp/index.md b/docs/mcp/index.md index 6b55a3f816..d85bccc5bb 100644 --- a/docs/mcp/index.md +++ b/docs/mcp/index.md @@ -65,7 +65,7 @@ There are three key pieces: * **MCP server**: a service that exposes tools to your AI client. It receives requests from the client and forwards them to Penpot. * **MCP plugin in Penpot**: a plugin that runs inside Penpot and connects your open file to the MCP server. It allows the server to access that file and its pages. -* **MCP client**: the tool where you write prompts (Cursor, Claude Code, Copilot-style tools, etc.). It connects to the MCP server using a server URL and an MCP key (or your active Penpot session in the current local setup). +* **MCP client**: the tool where you write prompts (Cursor, Claude Code, Copilot-style tools, etc.). It connects to the MCP server using a server URL and an MCP key. In remote MCP, the client can get the key by asking you to sign in to Penpot, or you can add it to the URL yourself. The local setup uses your active Penpot session instead. ![How the AI client, Penpot MCP server, plugin, and design file connect](/img/mcp/mcp-flow.webp) @@ -73,7 +73,7 @@ There are three key pieces: Some important concepts for users: * **Integrations page**: MCP is configured under **Your account → Integrations → MCP Server**. Here you enable or disable MCP, get the server URL and manage the MCP key. -* **MCP key**: a personal, non-recoverable token that authenticates your AI client with the MCP server. Only one key can exist per user at a time. This is used by the remote MCP setup. +* **MCP key**: a personal, non-recoverable token that authenticates your AI client with the MCP server. Only one key can exist per user at a time. This is used by the remote MCP setup. Clients connected through Penpot sign-in use this same key. * **MCP session**: a connection from a Penpot tab to the MCP server, identified by a short session ID derived from the tab's user session. Several tabs can connect independently, including tabs showing the same file. * **Current page**: the page open in the selected session. Switching browser focus to another tab does not change which session the agent uses. @@ -130,7 +130,29 @@ Because **remote MCP** does not expose local file-system access: If you just want to try Penpot AI workflows quickly through the MCP, follow this path for the **hosted (remote) MCP server**. -### Remote MCP in 5 steps +### Remote MCP in 3 steps + +Use this path if your MCP client can sign in to remote MCP servers (OAuth), for example Claude Code, Codex, or Claude.ai custom connectors. It works in the official SaaS and in self-hosted instances that [enable it](/technical-guide/configuration/#mcp-sign-in). + +1. #### Add the server to your MCP client + Use the server URL without a key: + + ``` + https:///mcp/stream + ``` + + In the official SaaS the domain is `design.penpot.app`. See [Connect your MCP client](#connect-your-mcp-client) for client-specific commands. + +2. #### Sign in and allow access + Your client opens Penpot in the browser. Sign in if needed, check the client name and your account, and choose **Allow access**. Penpot enables MCP and creates your MCP key if you do not have one yet. If you already have a key, Penpot keeps it. Your client stores the key; you never need to copy it. + +3. #### Open a Penpot file and connect MCP + In Penpot, open a design file and choose **MCP → Connect** in the toolbar. Allowing access does not connect a file automatically. + + +### Remote MCP with an MCP key + +Use this path if your client cannot sign in to remote MCP servers, or if your Penpot instance does not offer sign-in. 1. #### Enable MCP in Penpot Go to **Your account → Integrations → MCP Server** and enable the feature. @@ -226,7 +248,7 @@ You can use Penpot MCP server in two main ways: * Hosted for you (no need to run anything on your machine). * Best option for most users, simpler installation and fewer moving parts. * Does **not** have privileged access to your local file system, it can only work with what Penpot exposes (design files, libraries, tokens, etc.). - * The **server URL** is provided in **Your account → Integrations → MCP Server** and looks like: + * The **server URL** is `https:///mcp/stream`. Clients that sign in to Penpot use it as it is. Other clients use the URL shown in **Your account → Integrations → MCP Server**, which adds your key: * `https:///mcp/stream?userToken=YOUR_MCP_KEY` * The domain depends on the Penpot installation. In the official SaaS it will be `design.penpot.app`. * **Local MCP server** @@ -239,10 +261,17 @@ You can use Penpot MCP server in two main ways: ## Connect your MCP client -* **Remote MCP** +* **Remote MCP, signing in to Penpot** + * URL: `https:///mcp/stream`, with no key. + * Add the server to your client, then start the sign-in. Penpot opens in the browser and asks you to allow access. For example: + * Claude Code: run `claude mcp add --transport http penpot https:///mcp/stream`, then run `/mcp`, select `penpot` and choose **Authenticate**. + * Codex: run `codex mcp add penpot --url https:///mcp/stream`, then `codex mcp login penpot`. + * Claude.ai and Claude Desktop: add a custom [connector](http://claude.ai/customize/connectors) with the URL above and choose **Connect**. + * Other clients: follow your client's instructions for remote MCP servers that use OAuth. +* **Remote MCP, with the MCP key in the URL** * URL (copy from your Penpot account overview): `https:///mcp/stream?userToken=YOUR_MCP_KEY` * Configure your MCP client with `npx -y add-mcp -g -n penpot ` or by following your client's instructions for connecting to a remote MCP server. - For Claude Desktop, the server can be added as a custom [connector](http://claude.ai/customize/connectors). + * Use this option for clients that cannot sign in, or if your Penpot instance does not offer sign-in. * **Local MCP** * Configure your MCP client with `npx -y add-mcp -g -n penpot http://localhost:4401/mcp` (adjust the port if you changed `PENPOT_MCP_SERVER_PORT`) or by following your client's instructions for connecting to an HTTP MCP server. @@ -273,6 +302,8 @@ Remote MCP is the easiest way to start using AI agents with Penpot. It's hosted ### Install and activate +If your client signs in to Penpot, you can skip this section: allowing access from the client enables MCP and creates your key. See [Remote MCP in 3 steps](#remote-mcp-in-3-steps). + 1. Open **Your account → Integrations**. 2. In the **MCP Server** section, read the short description to confirm that feature is available for your account. 3. Use the **Status** toggle to enable MCP Server. Penpot remembers this state per user across sessions. You still need to connect each tab explicitly. @@ -286,8 +317,16 @@ Remote MCP is the easiest way to start using AI agents with Penpot. It's hosted ### Connect -For remote mode, use the URL shown in **Your account → Integrations → MCP Server**, which includes your `userToken`. -See the section **Connect your MCP client** for details on how to connect. +There are two ways to connect a client in remote mode: + +* **Sign in to Penpot.** Add `https:///mcp/stream` to your client without a key. When the client starts the sign-in, Penpot opens in the browser. If you are not logged in, Penpot asks you to log in first. The approval page shows the client that is asking for access and the account you are signed in with. Choose **Allow access** to connect, or **Cancel** to stop without changing anything. +* **Use the MCP key.** Use the URL shown in **Your account → Integrations → MCP Server**, which includes your `userToken`. + +Allowing access enables MCP for your account and gives the client your MCP key. Penpot creates a key only if you do not have a valid one; an existing key stays the same, so clients that already use it keep working. The client receives the key directly from Penpot. It never appears in the browser address bar. + +If the approval page says the request expired or was already used, start the connection again from your MCP client. + +See the section [Connect your MCP client](#connect-your-mcp-client) for details on how to connect. ### Setup videos @@ -355,16 +394,16 @@ If no key exists and you enable MCP, a modal guides you through creating one. Th **Regenerate MCP key** -The **Regenerate MCP key** action immediately revokes the current key. A warning explains that any client using the old key will stop working until you update its configuration. After regenerating, Penpot shows the new key and an updated configuration snippet with the new `userToken` that you need to copy into your MCP client configuration. +The **Regenerate MCP key** action immediately revokes the current key. A warning explains that any client using the old key will stop working until you update its configuration. This includes clients connected by signing in to Penpot: sign in again from those clients to give them the new key. After regenerating, Penpot shows the new key and an updated configuration snippet with the new `userToken` that you need to copy into your MCP client configuration. **Expired key state** -If your key is expired, an error block explains that the connection cannot be established until you regenerate the key. Regenerating the key clears the error state. +If your key is expired, an error block explains that the connection cannot be established until you regenerate the key. Regenerating the key clears the error state. Signing in again from a client also replaces an expired key with a new one. Security recommendations to highlight in the Help Center: * Treat your MCP key like a password or access token, do not share it in screenshots or code samples. -* Regenerate the key if you suspect it may have leaked. +* Regenerate the key if you suspect it may have leaked. Clients connected by signing in to Penpot hold the same key, so regenerating it disconnects them too. * Disabling MCP Server disconnects your integrated sessions. Disconnecting one tab stops access through that connection; other connected sessions remain available. *** diff --git a/docs/technical-guide/configuration.md b/docs/technical-guide/configuration.md index 4759855e14..2f6498b02f 100644 --- a/docs/technical-guide/configuration.md +++ b/docs/technical-guide/configuration.md @@ -983,6 +983,33 @@ set, and sends it through the [email configuration](#email-configuration). Each email carries the name and email address of the user who sent it, and uses that address as reply-to. +#### MCP sign-in + +MCP clients with OAuth support can connect with the URL +`https:///mcp/stream`, without a key. The client opens Penpot +in the browser, the user allows access, and Penpot gives the client the user's MCP +key. This is off by default. Turn it on by setting the public URL of the +authorization service on the MCP container: + +```bash +# MCP +PENPOT_MCP_AUTH_URI: https://penpot.example.com/api/mcp/oauth +``` + +- `PENPOT_MCP_AUTH_URI`: Your Penpot public URI followed by `/api/mcp/oauth`. It + must use HTTPS and have no credentials, query or fragment. When unset, clients + must add the key to the URL as `userToken`. + +The MCP server must run in multi-user mode, which the official `penpotapp/mcp` +image does by default. The server does not start if the value is invalid, and it +logs at startup whether sign-in is on. + +The frontend container forwards the sign-in requests to the backend. If your proxy +sends `/api/` straight to the backend instead, send +`/.well-known/oauth-authorization-server/api/mcp/oauth` to the backend too. + +Clients that already add the key to the URL keep working with sign-in on. + ### Internal resolver The frontend container resolves the backend, exporter and MCP service names with the From adda9ad908bc3e7e24f08eebc96d138bf7214a29 Mon Sep 17 00:00:00 2001 From: Alonso Torres Date: Wed, 11 Nov 2026 10:28:11 +0100 Subject: [PATCH 06/29] :sparkles: Restrict global plugins to MCP and localhost (#12949) Plugins with global scope or a management permission (allow:global, manage:projects, manage:teams, manage:delete) keep running outside a file and can drive the management API. That feature is still in progress, so only the bundled MCP plugin and plugins served from localhost, 127.0.0.1 or [::1] get it. The host check ignores letter case and a trailing dot. The check runs at three points: - parse-manifest throws global-not-allowed, so the plugin manager and the dashboard install link refuse remote global plugins with a specific message. Opening an installed plugin fetches its manifest again, so plugins stored before this change are refused too. - start-plugin! and load-plugin! refuse to run such a plugin and warn the user; this also covers bundled plugins without a URL. - has-permission? withholds the global permissions from stored remote plugins, so the management API rejects them. AI-assisted-by: claude-opus-5-5 --- frontend/src/app/main/data/plugins.cljs | 120 ++++++++++-------- frontend/src/app/main/ui/dashboard.cljs | 6 +- .../src/app/main/ui/workspace/plugins.cljs | 14 +- frontend/src/app/plugins/register.cljs | 52 +++++++- .../frontend_tests/plugins/register_test.cljs | 68 ++++++++++ frontend/translations/en.po | 4 + frontend/translations/es.po | 4 + 7 files changed, 205 insertions(+), 63 deletions(-) diff --git a/frontend/src/app/main/data/plugins.cljs b/frontend/src/app/main/data/plugins.cljs index e9b15e0849..f68806fc4a 100644 --- a/frontend/src/app/main/data/plugins.cljs +++ b/frontend/src/app/main/data/plugins.cljs @@ -57,61 +57,75 @@ (update [_ state] (update-in state [:workspace-local :open-plugins] (fnil disj #{}) id)))) +(defn- global-blocked? + "True, after warning the user, when the plugin asks for global scope + or management permissions it is not allowed to have." + [plugin] + (when (and (preg/global-requested? plugin) + (not (preg/global-allowed? plugin))) + (log/warn :hint "Global plugins can only be loaded from localhost" + :plugin-id (:plugin-id plugin) + :host (:host plugin)) + (st/emit! (ntf/warn (tr "workspace.plugins.error.global-not-allowed"))) + true)) + (defn start-plugin! - [{:keys [plugin-id name version description host code permissions allow-background scope]} ^js extensions] - (let [load-plugin (unchecked-get ug/global "ɵloadPlugin")] - (if (fn? load-plugin) - (-> (load-plugin - #js {:pluginId plugin-id - :name name - :version version - :description description - :host host - :code code - :allowBackground (boolean allow-background) - :scope scope - :permissions (apply array permissions)} - nil - extensions) + [{:keys [plugin-id name version description host code permissions allow-background scope] :as plugin} ^js extensions] + (when-not (global-blocked? plugin) + (let [load-plugin (unchecked-get ug/global "ɵloadPlugin")] + (if (fn? load-plugin) + (-> (load-plugin + #js {:pluginId plugin-id + :name name + :version version + :description description + :host host + :code code + :allowBackground (boolean allow-background) + :scope scope + :permissions (apply array permissions)} + nil + extensions) - (p/catch (fn [cause] - (ex/print-throwable cause :prefix "Plugin Error") - (errors/flash :cause cause :type :handled)))) + (p/catch (fn [cause] + (ex/print-throwable cause :prefix "Plugin Error") + (errors/flash :cause cause :type :handled)))) - (log/warn :hint "Plugin runtime not initialized yet" - :plugin-id plugin-id - :action "start-plugin!")))) - -(defn- load-plugin! - [{:keys [plugin-id name version description host code icon permissions scope]}] - (st/emit! (pflag/initialize plugin-id version) - (save-current-plugin plugin-id)) - - (let [load-plugin (unchecked-get ug/global "ɵloadPlugin")] - (if (fn? load-plugin) - (-> (load-plugin - #js {:pluginId plugin-id - :name name - :description description - :version version - :host host - :code code - :icon icon - :scope scope - :permissions (apply array permissions)} - (fn [] - (st/emit! (remove-current-plugin plugin-id)))) - - (p/catch (fn [cause] - (st/emit! (remove-current-plugin plugin-id)) - (ex/print-throwable cause :prefix "Plugin Error") - (errors/flash :cause cause :type :handled)))) - - (do (log/warn :hint "Plugin runtime not initialized yet" :plugin-id plugin-id - :action "load-plugin!") - (st/emit! (remove-current-plugin plugin-id)))))) + :action "start-plugin!"))))) + +(defn- load-plugin! + [{:keys [plugin-id name version description host code icon permissions scope] :as plugin}] + (when-not (global-blocked? plugin) + (st/emit! (pflag/initialize plugin-id version) + (save-current-plugin plugin-id)) + + (let [load-plugin (unchecked-get ug/global "ɵloadPlugin")] + (if (fn? load-plugin) + (-> (load-plugin + #js {:pluginId plugin-id + :name name + :description description + :version version + :host host + :code code + :icon icon + :scope scope + :permissions (apply array permissions)} + (fn [] + (st/emit! (remove-current-plugin plugin-id)))) + + (p/catch (fn [cause] + (st/emit! (remove-current-plugin plugin-id)) + (ex/print-throwable cause :prefix "Plugin Error") + (errors/flash :cause cause :type :handled)))) + + (do + (log/warn :hint "Plugin runtime not initialized yet" + :plugin-id plugin-id + :action "load-plugin!") + (st/emit! (remove-current-plugin plugin-id))))))) (defn open-plugin! [{:keys [url] :as manifest} user-can-edit?] @@ -154,10 +168,12 @@ ;; Manifests match exactly - safe to load :else (load-plugin! manifest)))) - (fn [_err] + (fn [err] ;; Error fetching the manifest - can't verify integrity ;; Show error instead of loading potentially tampered code - (st/emit! (ntf/warn (tr "workspace.plugins.error.unreachable")))))) + (st/emit! (ntf/warn (if (= ::preg/global-not-allowed (:type (ex-data err))) + (tr "workspace.plugins.error.global-not-allowed") + (tr "workspace.plugins.error.unreachable"))))))) ;; Bundled plugins (no URL) - trusted, load directly (load-plugin! manifest))) diff --git a/frontend/src/app/main/ui/dashboard.cljs b/frontend/src/app/main/ui/dashboard.cljs index c83a33e97d..13059c1036 100644 --- a/frontend/src/app/main/ui/dashboard.cljs +++ b/frontend/src/app/main/ui/dashboard.cljs @@ -224,8 +224,10 @@ (st/emit! (ev/event {::ev/name "install-plugin" :name (:name plugin) :url plugin-url})) (open-permissions-dialog plugin)) (st/emit! (notif/error (tr "dashboard.plugins.parse-error"))))) - (fn [_] - (st/emit! (notif/error (tr "dashboard.plugins.bad-url")))))) + (fn [err] + (st/emit! (notif/error (if (= ::preg/global-not-allowed (:type (ex-data err))) + (tr "workspace.plugins.error.global-not-allowed") + (tr "dashboard.plugins.bad-url"))))))) (binding [storage/*sync* true] (swap! storage/session dissoc :plugin-url)))))) diff --git a/frontend/src/app/main/ui/workspace/plugins.cljs b/frontend/src/app/main/ui/workspace/plugins.cljs index cd73c45664..86e237fdc9 100644 --- a/frontend/src/app/main/ui/workspace/plugins.cljs +++ b/frontend/src/app/main/ui/workspace/plugins.cljs @@ -111,12 +111,13 @@ plugin-url* (mf/use-state "") plugin-url (deref plugin-url*) - input-status* (mf/use-state nil) ;; :error-url :error-manifest :success + input-status* (mf/use-state nil) ;; :error-url :error-manifest :error-global :success input-status (deref input-status*) error-url? (= :error-url input-status) error-manifest? (= :error-manifest input-status) - error? (or error-url? error-manifest?) + error-global? (= :error-global input-status) + error? (or error-url? error-manifest? error-global?) permissions (mf/deref refs/permissions) user-can-edit? (get permissions :can-edit) @@ -159,7 +160,10 @@ (fn [err] (.error js/console err) (reset! fetching-manifest? false) - (reset! input-status* :error-url)))))) + (reset! input-status* + (if (= ::preg/global-not-allowed (:type (ex-data err))) + :error-global + :error-url))))))) on-open-plugin (mf/use-fn @@ -236,6 +240,10 @@ [:div {:class (stl/css-case :info true :error error?)} (tr "workspace.plugins.error.manifest")]) + (when error-global? + [:div {:class (stl/css-case :info true :error error?)} + (tr "workspace.plugins.error.global-not-allowed")]) + (when-not (empty? plugins-state) [:> i18n/tr-html* {:class (stl/css :discover) diff --git a/frontend/src/app/plugins/register.cljs b/frontend/src/app/plugins/register.cljs index 3c919eed50..5c7c4b4fe6 100644 --- a/frontend/src/app/plugins/register.cljs +++ b/frontend/src/app/plugins/register.cljs @@ -16,6 +16,7 @@ [app.main.store :as st] [app.util.object :as obj] [beicon.v2.core :as rx] + [cuerdas.core :as str] [promesa.core :as p])) ;; Needs to be here because moving it to `app.main.data.workspace.mcp` will @@ -29,6 +30,33 @@ ;; Keeps the rollbacks not reported yet across reloads "allow:localstorage"}) +;; Permissions only granted to plugins that pass `global-allowed?` +(def global-permissions + #{"allow:global" "manage:projects" "manage:teams" "manage:delete"}) + +(def ^:private local-hosts + #{"localhost" "127.0.0.1" "[::1]" "::1"}) + +(defn- local-url? + "True when the url host is a loopback name, ignoring case and a + trailing dot." + [url] + (contains? local-hosts (some-> url u/uri :host str/lower (str/rtrim ".")))) + +(defn global-requested? + "True when the plugin asks for global scope or a management permission." + [{:keys [scope permissions]}] + (or (= "global" scope) + (boolean (some global-permissions permissions)))) + +(defn global-allowed? + "True when the plugin may run with global scope and use the management + API. The feature is still in progress, so only the bundled MCP plugin + and plugins served from localhost get it." + [{:keys [plugin-id host]}] + (or (= plugin-id mcp-plugin-id) + (local-url? host))) + ;; Promise that resolves when plugins runtime is initialized. ;; Lives here to avoid circular dependency: workspace.mcp -> app.plugins -> app.plugins.api -> workspace (defonce ^:private runtime-ready-promise (p/deferred)) @@ -121,9 +149,18 @@ :code code :icon icon :permissions (into #{} (map str) permissions)})] - (if (sm/validate ctp/schema:registry-entry manifest) - manifest - (.error js/console (clj->js (sm/explain ctp/schema:registry-entry manifest)))))) + (cond + (not (sm/validate ctp/schema:registry-entry manifest)) + (.error js/console (clj->js (sm/explain ctp/schema:registry-entry manifest))) + + (and (global-requested? manifest) + (not (global-allowed? manifest))) + (throw (ex-info "Global plugins can only be loaded from localhost" + {:type ::global-not-allowed + :host origin})) + + :else + manifest))) (defn subscribe-registry! "Subscribes f, called with no arguments on every registry change. @@ -273,13 +310,16 @@ (.error js/console "Failed to remove plugin:" err)))))))) (defn has-permission? - "True when the plugin was granted `permission`." + "True when the plugin was granted `permission`. The permissions in + `global-permissions` also need the plugin to pass `global-allowed?`." [plugin-id permission] (cond (= plugin-id "00000000-0000-0000-0000-000000000000") true (= plugin-id mcp-plugin-id) (contains? mcp-permissions permission) - :else (let [{:keys [permissions]} (dm/get-in @registry [:data plugin-id])] - (contains? permissions permission)))) + :else (let [{:keys [permissions] :as plugin} (dm/get-in @registry [:data plugin-id])] + (and (contains? permissions permission) + (or (not (contains? global-permissions permission)) + (global-allowed? plugin)))))) (defn- read-only-reason "Why plugin writes must fail in `state`, or nil when they can go ahead. diff --git a/frontend/test/frontend_tests/plugins/register_test.cljs b/frontend/test/frontend_tests/plugins/register_test.cljs index a4399ab6af..afa0cfe304 100644 --- a/frontend/test/frontend_tests/plugins/register_test.cljs +++ b/frontend/test/frontend_tests/plugins/register_test.cljs @@ -347,3 +347,71 @@ :permissions #js ["content:read"]})] (t/is (nil? (:scope manifest))) (t/is (= #{"content:read"} (:permissions manifest))))) + +;; --- global plugins restriction --- + +(defn- global-not-allowed-error + "The ex-data type thrown by `(f)`, or nil when it does not throw." + [f] + (try + (f) + nil + (catch :default cause + (:type (ex-data cause))))) + +(t/deftest global-plugins-are-allowed-for-mcp-and-localhost + (t/is (preg/global-allowed? {:plugin-id preg/mcp-plugin-id + :host "https://design.penpot.app/plugins/mcp/"})) + (doseq [host ["http://localhost:4400/" "http://127.0.0.1:4400/" "http://[::1]:4400/" + "http://LOCALHOST:4400/" "http://LocalHost/" "http://localhost.:4400/"]] + (t/is (preg/global-allowed? {:plugin-id "local" :host host}) host)) + (t/is (not (preg/global-allowed? {:plugin-id "remote" :host "https://plugins.example.com/"}))) + (t/is (not (preg/global-allowed? {:plugin-id "remote" :host "https://localhost.example.com/"})))) + +(t/deftest parse-manifest-rejects-remote-global-scope + (t/is (= ::preg/global-not-allowed + (global-not-allowed-error + #(preg/parse-manifest + "https://plugins.example.com/manifest.json" + #js {:name "Remote global plugin" + :code "plugin.js" + :scope "global" + :permissions #js ["content:read"]}))))) + +(t/deftest parse-manifest-rejects-remote-management-permissions + (doseq [permission ["manage:projects" "manage:teams" "manage:delete" "allow:global"]] + (t/is (= ::preg/global-not-allowed + (global-not-allowed-error + #(preg/parse-manifest + "https://plugins.example.com/manifest.json" + #js {:name "Remote management plugin" + :code "plugin.js" + :permissions #js ["content:read" permission]}))) + permission))) + +(t/deftest parse-manifest-keeps-remote-workspace-plugins + (let [manifest (preg/parse-manifest + "https://plugins.example.com/manifest.json" + #js {:name "Remote workspace plugin" + :code "plugin.js" + :permissions #js ["content:read"]})] + (t/is (= #{"content:read"} (:permissions manifest))))) + +(t/deftest ^:async stored-remote-plugins-lose-global-permissions + (await + (mock/with-mocks* + {rp/cmd! (record-cmd-mock (fn [_ _] (rx/of {:ok true})))} + (let [permissions #{"content:read" "allow:global" "manage:projects"}] + (preg/install-plugin! {:plugin-id "reg-global-remote" + :host "https://plugins.example.com/" + :scope "global" + :permissions permissions}) + (preg/install-plugin! {:plugin-id "reg-global-local" + :host "http://localhost:4400/" + :scope "global" + :permissions permissions}) + (t/is (preg/has-permission? "reg-global-remote" "content:read")) + (t/is (not (preg/has-permission? "reg-global-remote" "allow:global"))) + (t/is (not (preg/has-permission? "reg-global-remote" "manage:projects"))) + (t/is (preg/has-permission? "reg-global-local" "allow:global")) + (t/is (preg/has-permission? "reg-global-local" "manage:projects")))))) diff --git a/frontend/translations/en.po b/frontend/translations/en.po index 185602dbb5..c0ef6a81ee 100644 --- a/frontend/translations/en.po +++ b/frontend/translations/en.po @@ -8537,6 +8537,10 @@ msgstr "Discover [more plugins](%s)" msgid "workspace.plugins.empty-plugins" msgstr "No plugins installed yet" +#: src/app/main/data/plugins.cljs, src/app/main/ui/dashboard.cljs, src/app/main/ui/workspace/plugins.cljs +msgid "workspace.plugins.error.global-not-allowed" +msgstr "Plugins that work outside a file can only be installed from localhost for now." + #: src/app/main/ui/workspace/plugins.cljs:237 msgid "workspace.plugins.error.manifest" msgstr "The plugin manifest is incorrect." diff --git a/frontend/translations/es.po b/frontend/translations/es.po index 5871401d67..c8506b8466 100644 --- a/frontend/translations/es.po +++ b/frontend/translations/es.po @@ -8614,6 +8614,10 @@ msgstr "Descubre [más extensiones](%s)" msgid "workspace.plugins.empty-plugins" msgstr "No se encuentran extensiones" +#: src/app/main/data/plugins.cljs, src/app/main/ui/dashboard.cljs, src/app/main/ui/workspace/plugins.cljs +msgid "workspace.plugins.error.global-not-allowed" +msgstr "Por ahora, las extensiones que funcionan fuera de un archivo solo se pueden instalar desde localhost." + #: src/app/main/ui/workspace/plugins.cljs:237 msgid "workspace.plugins.error.manifest" msgstr "El manifiesto de la expansión es incorrecto." From 1599e096daddfd8b17be8a3892e0443e10da1ada Mon Sep 17 00:00:00 2001 From: Alonso Torres Date: Wed, 11 Nov 2026 12:10:34 +0100 Subject: [PATCH 07/29] :bug: Fix MCP session discovery for plugins connected from the dashboard (#12865) * :bug: Fix MCP session discovery for plugins connected from the dashboard With Redis-backed routing, execute_code without a sessionId timed out with SessionDiscoveryError when the plugin ran on the dashboard. The discovery reply schema required a string fileId and fileName, so replies listing a session with no open file were dropped, and discovery waited for its timeout. Share one session schema between the plugin handshake and discovery replies. It accepts a null file and keeps workspaceState. An unreadable reply now fails discovery at once with an error that names the problem and says a retry will not help. Timeout and Redis failures state their reason. Closes #12860 * :bug: Stop blaming server load when MCP discovery fails on shutdown Closing the dispatcher rejected pending discoveries with the default advice, which blamed temporary server load. Shutdown now gives its own advice. The dispatcher tests now cover each discovery failure path: an unreadable reply alongside a valid one rejects with InvalidDiscoveryResponseError and does not dispatch, a late unreadable reply leaves a settled discovery unchanged, and timeout, Redis failure and shutdown each state their reason. --- .../server/src/PenpotSessionSchema.ts | 14 +++ mcp/packages/server/src/PluginBridge.ts | 12 +-- mcp/packages/server/src/RedisBridge.test.ts | 46 +++++++-- mcp/packages/server/src/RedisBridge.ts | 25 +++-- .../MultiInstanceTaskDispatcher.test.ts | 95 +++++++++++++++++-- .../dispatch/MultiInstanceTaskDispatcher.ts | 48 +++++++--- .../server/src/dispatch/TaskDispatcher.ts | 29 +++++- 7 files changed, 224 insertions(+), 45 deletions(-) create mode 100644 mcp/packages/server/src/PenpotSessionSchema.ts diff --git a/mcp/packages/server/src/PenpotSessionSchema.ts b/mcp/packages/server/src/PenpotSessionSchema.ts new file mode 100644 index 0000000000..b70da49330 --- /dev/null +++ b/mcp/packages/server/src/PenpotSessionSchema.ts @@ -0,0 +1,14 @@ +import type { PenpotSession } from "@penpot/mcp-common"; +import { z } from "zod"; + +/** + * Schema of a Penpot session as reported by the plugin and exchanged between MCP instances. + * + * A session without an open file (e.g. a plugin connected from the dashboard) has null `fileId` and `fileName`. + */ +export const penpotSessionSchema = z.object({ + sessionId: z.string().min(1).max(128), + fileId: z.string().min(1).max(128).nullable(), + fileName: z.string().nullable(), + workspaceState: z.enum(["none", "loading", "ready"]).optional(), +}) satisfies z.ZodType; diff --git a/mcp/packages/server/src/PluginBridge.ts b/mcp/packages/server/src/PluginBridge.ts index 340016f44f..8bf426e9a8 100644 --- a/mcp/packages/server/src/PluginBridge.ts +++ b/mcp/packages/server/src/PluginBridge.ts @@ -10,6 +10,7 @@ import { UnclaimedRollbacksMessage, } from "@penpot/mcp-common"; import { createLogger } from "./logger"; +import { penpotSessionSchema } from "./PenpotSessionSchema"; import type { PenpotMcpServer } from "./PenpotMcpServer"; import type { RedisBridge } from "./RedisBridge"; import type { PenpotConnection, PluginLivenessState } from "./PenpotConnection"; @@ -39,12 +40,7 @@ export const HEARTBEAT_STALE_THRESHOLD_MS = 30000; const connectionInitSchema = z.object({ type: z.literal("initialize"), - session: z.object({ - sessionId: z.string().min(1).max(128), - fileId: z.string().min(1).max(128).nullable(), - fileName: z.string().nullable(), - workspaceState: z.enum(["none", "loading", "ready"]).optional(), - }), + session: penpotSessionSchema, }); /** @@ -191,9 +187,7 @@ export class PluginBridge implements TaskDispatchHost { } connection.frozen = false; if (message?.type === "context-update") { - const context = connectionInitSchema.shape.session - .omit({ sessionId: true }) - .parse(message.context); + const context = penpotSessionSchema.omit({ sessionId: true }).parse(message.context); Object.assign(connection.session, context); return; } diff --git a/mcp/packages/server/src/RedisBridge.test.ts b/mcp/packages/server/src/RedisBridge.test.ts index 52080fcc3c..c64650ef1e 100644 --- a/mcp/packages/server/src/RedisBridge.test.ts +++ b/mcp/packages/server/src/RedisBridge.test.ts @@ -7,6 +7,8 @@ import { RedisBridge } from "./RedisBridge"; import { PluginBridge } from "./PluginBridge"; import { PluginTask } from "./PluginTask"; import type { PenpotMcpServer } from "./PenpotMcpServer"; +import type { PenpotSession } from "@penpot/mcp-common"; +import { InvalidDiscoveryResponseError } from "./dispatch/TaskDispatcher"; // Replace only the Redis command boundary; exercise the real transport, // dispatchers, task tracking, and WebSockets above it. @@ -105,8 +107,11 @@ test("Redis discovery receives one reply per subscribed instance and returns pub second.publishDiscoveryResponse(id, { instanceId: "second", sessions: [] }); }); const responders: string[] = []; - const count = await requester.sendDiscoveryRequest("alice", "query", (response) => - responders.push(response.instanceId) + const count = await requester.sendDiscoveryRequest( + "alice", + "query", + (response) => responders.push(response.instanceId), + assert.fail ); assert.equal(count, 2); assert.deepEqual(responders.sort(), ["first", "second"]); @@ -140,12 +145,12 @@ test("a failed discovery subscription is cleaned up and can be retried", async ( owner.subscribeToDiscovery("alice", () => {}), /Subscription failed/ ); - assert.equal(await caller.sendDiscoveryRequest("alice", "after-failure", () => {}), 0); + assert.equal(await caller.sendDiscoveryRequest("alice", "after-failure", () => {}, assert.fail), 0); await caller.unsubscribeFromDiscoveryResponses("after-failure"); await owner.subscribeToDiscovery("alice", ({ id }) => { owner.publishDiscoveryResponse(id, { instanceId: "owner", sessions: [] }); }); - assert.equal(await caller.sendDiscoveryRequest("alice", "retry", () => {}), 1); + assert.equal(await caller.sendDiscoveryRequest("alice", "retry", () => {}, assert.fail), 1); await caller.unsubscribeFromDiscoveryResponses("retry"); }); @@ -172,13 +177,14 @@ function instance(t: TestContext) { const connect = async ( sessionId: string, outcome: "success" | "error" | "silent" = "success", - reply: string = sessionId + reply: string = sessionId, + file: { fileId: string | null; fileName: string | null } = { fileId: "file", fileName: "Design" } ) => { const socket = new WebSocket(`ws://127.0.0.1:${wsPort}?userToken=alice`); sockets.push(socket); await once(socket, "open"); const initialized = once(socket, "message", { signal: AbortSignal.timeout(2000) }); - socket.send(JSON.stringify({ type: "initialize", session: { sessionId, fileId: "file", fileName: "Design" } })); + socket.send(JSON.stringify({ type: "initialize", session: { sessionId, ...file } })); await initialized; socket.on("message", (raw) => { const request = JSON.parse(raw.toString()); @@ -214,6 +220,34 @@ test("full Redis discovery implicitly selects a sole remote WebSocket", async (t assert.equal((await requester.bridge.executePluginTask(new PluginTask("test", {}))).data, "only"); }); +test("full Redis discovery implicitly selects a sole remote WebSocket without an open file", async (t) => { + const owner = instance(t); + const requester = instance(t); + await owner.connect("dashboard", "success", "dashboard", { fileId: null, fileName: null }); + assert.equal((await requester.bridge.executePluginTask(new PluginTask("test", {}))).data, "dashboard"); +}); + +test("an unreadable discovery reply fails discovery at once without suggesting a retry", async (t) => { + const responder = transport(t); + const requester = instance(t); + await responder.subscribeToDiscovery("alice", ({ id }) => { + responder.publishDiscoveryResponse(id, { + instanceId: "other-version", + sessions: [{ sessionId: "tab", fileId: 42 } as unknown as PenpotSession], + }); + }); + const started = Date.now(); + await assert.rejects(requester.bridge.executePluginTask(new PluginTask("test", {})), (error: Error) => { + assert.ok(error instanceof InvalidDiscoveryResponseError); + assert.match(error.message, /cannot be read/); + assert.match(error.message, /fileId/); + assert.match(error.message, /Retrying will not help/); + assert.doesNotMatch(error.message, /temporary server load/); + return true; + }); + assert.ok(Date.now() - started < 1000); +}); + test("forwarded plugin failures retain the regular execution failure route", async (t) => { const owner = instance(t); const requester = instance(t); diff --git a/mcp/packages/server/src/RedisBridge.ts b/mcp/packages/server/src/RedisBridge.ts index b653c70faf..ca3a45364d 100644 --- a/mcp/packages/server/src/RedisBridge.ts +++ b/mcp/packages/server/src/RedisBridge.ts @@ -1,6 +1,7 @@ import Redis from "ioredis"; import { PluginTaskRequest, PluginTaskResponse, PenpotSession } from "@penpot/mcp-common"; import { createLogger } from "./logger"; +import { penpotSessionSchema } from "./PenpotSessionSchema"; import { z } from "zod"; /** Query for the user's sessions held by each subscribed MCP instance. */ @@ -30,13 +31,7 @@ const sessionClaimSchema = z.object({ const discoveryResponseSchema = z.object({ instanceId: z.string().min(1), - sessions: z.array( - z.object({ - sessionId: z.string().min(1).max(128), - fileId: z.string().min(1).max(128), - fileName: z.string(), - }) - ), + sessions: z.array(penpotSessionSchema), }); /** @@ -248,19 +243,29 @@ export class RedisBridge { return `penpot.mcp.${this.tenant}.discovery.res.${this.channelKey(requestId)}`; } - /** Subscribes to replies before publishing; returns the actual recipient count. */ + /** + * Subscribes to replies before publishing; returns the actual recipient count. + * + * A reply that cannot be parsed is passed to `onInvalidResponse` with a readable description of the problem. + */ async sendDiscoveryRequest( userToken: string, requestId: string, - onResponse: (response: SessionDiscoveryResponse) => void + onResponse: (response: SessionDiscoveryResponse) => void, + onInvalidResponse: (error: Error) => void ): Promise { const channel = this.discoveryResponseChannel(requestId); this.handlers.set(channel, (raw) => { + let response: SessionDiscoveryResponse; try { - onResponse(discoveryResponseSchema.parse(JSON.parse(raw))); + response = discoveryResponseSchema.parse(JSON.parse(raw)); } catch (error) { this.logger.error(error, "Invalid discovery response"); + const reason = error instanceof z.ZodError ? z.prettifyError(error) : String(error); + onInvalidResponse(new Error(reason)); + return; } + onResponse(response); }); try { await this.subscriber.subscribe(channel); diff --git a/mcp/packages/server/src/dispatch/MultiInstanceTaskDispatcher.test.ts b/mcp/packages/server/src/dispatch/MultiInstanceTaskDispatcher.test.ts index 4c1b153916..18f3c657bf 100644 --- a/mcp/packages/server/src/dispatch/MultiInstanceTaskDispatcher.test.ts +++ b/mcp/packages/server/src/dispatch/MultiInstanceTaskDispatcher.test.ts @@ -6,14 +6,25 @@ import { PluginTask, type AbstractPluginTask } from "../PluginTask"; import type { RedisBridge, SessionClaim, SessionDiscoveryRequest, SessionDiscoveryResponse } from "../RedisBridge"; import { UserPenpotConnections } from "../UserPenpotConnections"; import { MultiInstanceTaskDispatcher } from "./MultiInstanceTaskDispatcher"; -import { SessionDiscoveryError, type TaskDispatchHost } from "./TaskDispatcher"; +import { InvalidDiscoveryResponseError, SessionDiscoveryError, type TaskDispatchHost } from "./TaskDispatcher"; + +/** Parse failure reported for a discovery reply that the requester cannot read. */ +const INVALID_REPLY_REASON = "sessions[0].fileId: expected string"; + +/** Reply callbacks registered by one discovery request. */ +interface DiscoveryReplyHandlers { + onResponse: (response: SessionDiscoveryResponse) => void; + onInvalidResponse: (error: Error) => void; +} /** In-memory Pub/Sub transport retaining Redis's per-subscriber delivery count. */ class RedisNetwork { readonly tasks = new Map void>>(); readonly discovery = new Map void>>(); readonly taskResponses = new Map) => void>(); - readonly discoveryResponses = new Map void>(); + readonly discoveryResponses = new Map(); + /** reply callbacks of the latest discovery request, kept after its subscription is released. */ + lastDiscoveryReplyHandlers?: DiscoveryReplyHandlers; readonly claims = new Map void>>(); failDiscovery = false; dropClaims = false; @@ -25,6 +36,7 @@ class RedisNetwork { class RedisPeer { replyToDiscovery = true; repeatDiscoveryReplies = false; + sendInvalidDiscoveryReplies = false; constructor(readonly network: RedisNetwork) {} @@ -72,9 +84,16 @@ class RedisPeer { this.network.discovery.get(token)?.delete(this); } - async sendDiscoveryRequest(token: string, id: string, onResponse: (response: SessionDiscoveryResponse) => void) { + async sendDiscoveryRequest( + token: string, + id: string, + onResponse: (response: SessionDiscoveryResponse) => void, + onInvalidResponse: (error: Error) => void + ) { if (this.network.failDiscovery) throw new Error("Redis unavailable for discovery"); - this.network.discoveryResponses.set(id, onResponse); + const handlers = { onResponse, onInvalidResponse }; + this.network.discoveryResponses.set(id, handlers); + this.network.lastDiscoveryReplyHandlers = handlers; await this.network.beforeDiscoveryPublish?.(); const subscribers = this.network.discovery.get(token) ?? new Map(); for (const [peer, handler] of subscribers) { @@ -87,7 +106,12 @@ class RedisPeer { } publishDiscoveryResponse(id: string, response: SessionDiscoveryResponse) { - this.network.discoveryResponses.get(id)?.(response); + const handlers = this.network.discoveryResponses.get(id); + if (this.sendInvalidDiscoveryReplies) { + handlers?.onInvalidResponse(new Error(INVALID_REPLY_REASON)); + } else { + handlers?.onResponse(response); + } } async unsubscribeFromDiscoveryResponses(id: string) { @@ -244,6 +268,7 @@ test("incomplete discovery never dispatches even when it yields exactly one sess slow.redis.replyToDiscovery = false; await assert.rejects(new Instance(network).execute("alice"), (error: Error) => { assert.ok(error instanceof SessionDiscoveryError); + assert.match(error.message, /not every MCP server instance replied within 20ms/); assert.match(error.message, /temporary server load.*retry/); return true; }); @@ -265,7 +290,65 @@ test("duplicate replies from one instance do not substitute for a missing reply" test("discovery transport failures use the retryable discovery error", async () => { const network = new RedisNetwork(); network.failDiscovery = true; - await assert.rejects(new Instance(network).execute("alice"), SessionDiscoveryError); + await assert.rejects(new Instance(network).execute("alice"), (error: Error) => { + assert.ok(error instanceof SessionDiscoveryError); + assert.match(error.message, /the Redis request failed \(Redis unavailable for discovery\)/); + assert.match(error.message, /temporary server load.*retry/); + return true; + }); +}); + +test("an invalid reply fails discovery without dispatching even when another instance replies validly", async () => { + const network = new RedisNetwork(); + const responsive = new Instance(network); + const otherVersion = new Instance(network); + await responsive.connect("alice", "visible"); + await otherVersion.connect("alice", "unreadable"); + otherVersion.redis.sendInvalidDiscoveryReplies = true; + await assert.rejects(new Instance(network).execute("alice"), (error: Error) => { + assert.ok(error instanceof InvalidDiscoveryResponseError); + assert.ok(error.message.includes(INVALID_REPLY_REASON)); + assert.match(error.message, /Retrying will not help/); + return true; + }); + assert.deepEqual(responsive.executions, []); + assert.deepEqual(otherVersion.executions, []); + assert.equal(network.discoveryResponses.size, 0); +}); + +test("an invalid reply after discovery settled leaves its outcome unchanged", async () => { + const network = new RedisNetwork(); + const owner = new Instance(network); + await owner.connect("alice", "only"); + const requester = new Instance(network); + const sessions = requester.dispatcher.discoverSessions("alice"); + const late = network.lastDiscoveryReplyHandlers!; + assert.deepEqual( + (await sessions).map((session) => session.sessionId), + ["only"] + ); + late.onInvalidResponse(new Error(INVALID_REPLY_REASON)); + assert.equal((await requester.execute("alice")).data, "only"); +}); + +test("closing the dispatcher fails pending discoveries without blaming server load", async () => { + const network = new RedisNetwork(); + const silent = new Instance(network); + await silent.connect("alice", "tab"); + silent.redis.replyToDiscovery = false; + const requester = new Instance(network); + const rejected = assert.rejects(requester.execute("alice"), (error: Error) => { + assert.ok(error instanceof SessionDiscoveryError); + assert.match(error.message, /the MCP server is shutting down/); + assert.match(error.message, /You can retry the request/); + assert.doesNotMatch(error.message, /temporary server load/); + return true; + }); + await new Promise((resolve) => setImmediate(resolve)); + await requester.dispatcher.close(); + await rejected; + assert.deepEqual(silent.executions, []); + assert.equal(network.discoveryResponses.size, 0); }); test("disconnecting one tab retains discovery and dispatch for the remaining tab", async () => { diff --git a/mcp/packages/server/src/dispatch/MultiInstanceTaskDispatcher.ts b/mcp/packages/server/src/dispatch/MultiInstanceTaskDispatcher.ts index 10c895e659..891bb12d77 100644 --- a/mcp/packages/server/src/dispatch/MultiInstanceTaskDispatcher.ts +++ b/mcp/packages/server/src/dispatch/MultiInstanceTaskDispatcher.ts @@ -5,7 +5,12 @@ import type { AbstractPluginTask } from "../PluginTask"; import { RemotePluginTask } from "../RemotePluginTask"; import type { RedisBridge, SessionClaim, SessionDiscoveryResponse } from "../RedisBridge"; import { createLogger } from "../logger"; -import { SessionDiscoveryError, TaskDispatcher, type TaskDispatchHost } from "./TaskDispatcher"; +import { + InvalidDiscoveryResponseError, + SessionDiscoveryError, + TaskDispatcher, + type TaskDispatchHost, +} from "./TaskDispatcher"; /** Routes tasks over Redis and discovers live sessions across MCP instances. */ export class MultiInstanceTaskDispatcher extends TaskDispatcher { @@ -44,10 +49,10 @@ export class MultiInstanceTaskDispatcher extends TaskDispatcher { resolve = res; reject = rej; }); - const fail = () => { + const fail = (error: SessionDiscoveryError) => { if (!settled) { settled = true; - reject(new SessionDiscoveryError()); + reject(error); } }; const complete = () => { @@ -56,8 +61,18 @@ export class MultiInstanceTaskDispatcher extends TaskDispatcher { resolve(this.uniqueSessions([...responses.values()].flat())); } }; - const timeout = setTimeout(fail, this.discoveryTimeoutMs); - this.pendingDiscoveries.set(requestId, fail); + const timeout = setTimeout( + () => + fail( + new SessionDiscoveryError( + `not every MCP server instance replied within ${this.discoveryTimeoutMs}ms` + ) + ), + this.discoveryTimeoutMs + ); + this.pendingDiscoveries.set(requestId, () => + fail(new SessionDiscoveryError("the MCP server is shutting down", "You can retry the request.")) + ); const releaseResponses = () => { void this.redisBridge .unsubscribeFromDiscoveryResponses(requestId) @@ -65,18 +80,29 @@ export class MultiInstanceTaskDispatcher extends TaskDispatcher { }; // Replies can arrive before publish returns its recipient count. void this.redisBridge - .sendDiscoveryRequest(token, requestId, (response: SessionDiscoveryResponse) => { - if (settled) return; - responses.set(response.instanceId, response.sessions); - complete(); - }) + .sendDiscoveryRequest( + token, + requestId, + (response: SessionDiscoveryResponse) => { + if (settled) return; + responses.set(response.instanceId, response.sessions); + complete(); + }, + (error) => fail(new InvalidDiscoveryResponseError(error)) + ) .then((count) => { expectedResponses = count; complete(); // A slow subscribe/publish may finish after the discovery timeout. if (settled) releaseResponses(); }) - .catch(fail); + .catch((error) => + fail( + new SessionDiscoveryError( + `the Redis request failed (${error instanceof Error ? error.message : error})` + ) + ) + ); try { return await result; } finally { diff --git a/mcp/packages/server/src/dispatch/TaskDispatcher.ts b/mcp/packages/server/src/dispatch/TaskDispatcher.ts index 062af6b80c..b9ea7b617a 100644 --- a/mcp/packages/server/src/dispatch/TaskDispatcher.ts +++ b/mcp/packages/server/src/dispatch/TaskDispatcher.ts @@ -40,10 +40,33 @@ export abstract class TaskDispatcher { } } -/** Error returned when discovery cannot establish the complete set of sessions. */ +/** + * Error returned when discovery cannot establish the complete set of sessions. + * + * The default message marks the failure as transient; subclasses describe failures that a retry does not fix. + */ export class SessionDiscoveryError extends Error { - constructor() { - super("Penpot session discovery failed. This may be due to temporary server load. You can retry the request."); + /** + * @param reason - what prevented discovery from completing + * @param advice - what the caller can do about it + */ + constructor(reason: string, advice = "This may be due to temporary server load. You can retry the request.") { + super(`Penpot session discovery failed: ${reason}. ${advice}`); this.name = "SessionDiscoveryError"; } } + +/** Error returned when an MCP server instance replies to discovery with a session list that cannot be read. */ +export class InvalidDiscoveryResponseError extends SessionDiscoveryError { + /** + * @param cause - the parse failure of the reply + */ + constructor(cause: Error) { + super( + `an MCP server instance sent a session list that cannot be read (${cause.message})`, + "Retrying will not help; the MCP server instances may be running different versions. " + + "To work around it, ask the user for the session ID shown in the Penpot MCP menu and pass it as sessionId." + ); + this.name = "InvalidDiscoveryResponseError"; + } +} From 0efa550a99bbc5676f795552e5ed2cbee64ba6cb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?andr=C3=A9s=20gonz=C3=A1lez?= Date: Wed, 11 Nov 2026 13:00:06 +0100 Subject: [PATCH 08/29] :sparkles: Improve the changelog flow: dev-only bugs and feature sections (#12981) * :sparkles: Keep bugs that never reached production out of the changelog The team rarely adds `no changelog` to bugs found while building a feature of the same release, so they end up in the changelog and make Penpot look less stable than it is. Add `changelog.py dev-only-bugs`. Bugs whose parent is a Feature or Enhancement of the same milestone are sure: `--apply` labels them `no changelog` on GitHub. Bugs under an epic of the milestone, with the `regression` label, or whose version field names only hourly, develop, staging, PRE or this milestone, are candidates for a human to confirm; the anomaly report lists them as warnings. `gh.py issue` now returns body and parent. The flow memory documents it as step 3. Closes #12747 AI-assisted-by: claude-opus-5-5 * :sparkles: Split features and enhancements in the changelog flow Features were listed together with enhancements under ":sparkles: New features & Enhancements", so they did not stand out. Map the Feature issue type to its own ":tada: Features" section, keep Enhancement under ":sparkles: Enhancements", rename the bug section to ":bug: Bugs" and set the order: boom, rocket, lock, tada, sparkles, bug. The new layout applies from 2.20.0: 2.19.0 and 2.19.1 are already out with the old one, so 2.19.x patches keep it. Closes #12741 AI-assisted-by: claude-opus-5-5 --- .agents/skills/update-changelog/SKILL.md | 5 +- .serena/memories/scripts/changelog.md | 3 +- .serena/memories/scripts/gh.md | 2 +- .../memories/workflow/updating-changelog.md | 52 ++++--- scripts/changelog.py | 131 ++++++++++++++++++ scripts/gh.py | 9 ++ scripts/test_changelog.py | 56 ++++++++ 7 files changed, 235 insertions(+), 23 deletions(-) diff --git a/.agents/skills/update-changelog/SKILL.md b/.agents/skills/update-changelog/SKILL.md index c83ab8d1f7..87c493e7c4 100644 --- a/.agents/skills/update-changelog/SKILL.md +++ b/.agents/skills/update-changelog/SKILL.md @@ -31,7 +31,7 @@ memory; it does not repeat it. layout, entry rules, the `## Breaking change note` comment contract, the advisory (`:lock:`) contract, pre-flight checks, anomaly types, and pitfalls. 2. `mem:scripts/changelog` — `boom`, `security`, `check-merged`, `cross-ref`, - `report`. + `dev-only-bugs`, `report`. 3. `mem:scripts/gh` — `issues`, `issue`, `comments`, `prs`, `advisories`. ## Commands @@ -39,6 +39,8 @@ memory; it does not repeat it. ```bash python3 scripts/gh.py issues "" --exclude "release blocker,no changelog" python3 scripts/gh.py advisories --patched-version "" +python3 scripts/changelog.py dev-only-bugs "" # list bugs that never reached production +python3 scripts/changelog.py dev-only-bugs "" --apply # label the sure ones `no changelog` python3 scripts/changelog.py check-merged python3 scripts/changelog.py boom "" # print the :boom: section python3 scripts/changelog.py boom "" --write # replace it in CHANGES.md @@ -54,4 +56,5 @@ python3 scripts/changelog.py report "" - Only closed, non-excluded milestone issues are included; the issue is the changelog unit and the PR is inline. - `:boom:` and `:rocket:` are label-generated and `:lock:` is advisory-generated: never hand-edit them; edit the label, the `## Breaking change note` comment or the advisory and re-run the command. - Verify PR merge status before writing; only merged PRs are referenced. +- Bugs that never reached production stay out: `dev-only-bugs --apply` labels the sure ones; candidates are the user's call, never label them yourself. - Fill `patched_versions` in the advisories before running `security`; unpublished advisories are valid. diff --git a/.serena/memories/scripts/changelog.md b/.serena/memories/scripts/changelog.md index 7134377fa5..6785692d82 100644 --- a/.serena/memories/scripts/changelog.md +++ b/.serena/memories/scripts/changelog.md @@ -8,7 +8,8 @@ - `boom [--changes CHANGES.md --write]` — generate `### :boom: Breaking changes & Deprecations` from the closed, non-excluded milestone issues labelled `breaking change`, sorted by issue number. Each issue becomes a `#### ` block: content = body of the issue's last `## Breaking change note` comment (whole markdown, headings demoted to `#####` outside code fences; fallback = heading plus refs with a stderr warning), refs = issue link plus merged closing PRs on their own line, with `(by @author)` for `community contribution`. `--write` replaces only that subsection of the version (inserts it after the version header when missing, keeps the heading when there are no entries; idempotent). Print-only by default. Uses `gh.py comments`. - `security [--changes CHANGES.md --write]` — generate `### :lock: Security fixes` from the advisories whose `patched_versions` include the version (via `gh.py advisories --patched-version`), sorted by GHSA id. Entry: `- Fix () (PR: [#N](...))`; description = `Fix` + advisory summary, first word decapitalized unless it starts with an acronym; PRs = merged milestone PRs labelled `pr pt` whose body contains the GHSA id, comma-separated, and no PR is legal. Warns on stderr when a `pr pt` PR references an advisory with no matching `patched_versions`. `--write` replaces only that subsection, placed after `:rocket:` (or `:boom:`, or the version header when neither exists); empty entries remove the subsection; idempotent. Print-only by default. - `cross-ref [--changes CHANGES.md]` — fetch milestone PRs with `--state all` once, report merged PRs missing from the version section plus CLOSED (unmerged) warnings. PR links inside `:lock:` entries count as referenced; advisory URLs never appear here by design. -- `report [--changes CHANGES.md --output CHANGES-ISSUES.md]` — overwrite the anomaly report: A/B milestone mismatches, E `:boom:` entry without the label, F `:rocket:` sync, G `:boom:` sync, plus the C/D `:rocket:` gaps (warnings). `:boom:`/`:rocket:` rule violations are fixed by regenerating or relabelling, not by editing entries in place. +- `dev-only-bugs [--apply]` — list closed milestone bugs that never reached production, without `no changelog`. Sure: parent issue is a `Feature`/`Enhancement` of the same milestone; `--apply` adds `no changelog` to these only. Candidates (never labelled automatically): parent is an epic of the milestone, `regression` label, or the "Affected versions" / "Environment" body field names only hourly, develop, staging, PRE or this milestone (`only_unreleased_envs`; any production hint or other version keeps the bug in). Reads `body` and `parent` from one batched `gh.py issue` lookup. +- `report [--changes CHANGES.md --output CHANGES-ISSUES.md]` — overwrite the anomaly report: A/B milestone mismatches, E `:boom:` entry without the label, F `:rocket:` sync, G `:boom:` sync, plus the C/D `:rocket:` gaps and the `dev-only-bugs` lists (warnings). `:boom:`/`:rocket:` rule violations are fixed by regenerating or relabelling, not by editing entries in place. ## Conventions diff --git a/.serena/memories/scripts/gh.md b/.serena/memories/scripts/gh.md index c2a10d8f6b..6b546e578c 100644 --- a/.serena/memories/scripts/gh.md +++ b/.serena/memories/scripts/gh.md @@ -81,7 +81,7 @@ python3 scripts/gh.py prs --milestone "2.16.0" --state all Fetch issue details by number in batches of 50 (same input styles as `prs`: numbers, `--file`, `--stdin`). No filters. A dead number fails its whole batch (unlike PRs, issues error instead of resolving to null), so the batch falls back to one-by-one lookups and reports misses as `{"number", "error": "not_found"}`. Any other failure aborts. -**Output**: JSON array to stdout; progress to stderr. +**Output**: JSON array to stdout; progress to stderr. Each issue carries the shared issue fields plus `body` and `parent` (`number`, `issue_type`, `milestone`, or null); `changelog.py dev-only-bugs` reads both. ### `comments` diff --git a/.serena/memories/workflow/updating-changelog.md b/.serena/memories/workflow/updating-changelog.md index 66d1220fa7..1c233312b3 100644 --- a/.serena/memories/workflow/updating-changelog.md +++ b/.serena/memories/workflow/updating-changelog.md @@ -24,11 +24,12 @@ Editing `CHANGES.md` happens only on explicit request or as part of the release | Is it a breaking change? | `breaking change` label | | Is it a highlight? | `release highlight` label | -- **Issue Type decides the section**: `Bug` → `:bug:`, `Feature`/`Enhancement` → `:sparkles:`, `Task` is excluded (internal chores). Labels `bug`/`enhancement` and title emoji prefixes are often wrong or missing; do not use them. +- **Issue Type decides the section**: `Feature` → `:tada:`, `Enhancement` → `:sparkles:`, `Bug` → `:bug:`, `Task` is excluded (internal chores). Labels `bug`/`enhancement` and title emoji prefixes are often wrong or missing; do not use them. - **Security fixes come from advisories**: an advisory whose `patched_versions` include the version becomes a `:lock:` entry (section 7). `triage`, `draft` and `published` are all valid states; the advisory may still be unpublished when the changelog is written. A security fix may also carry a regular bug issue; both entries are legal and independent (the issue lives in `:bug:` as any other bug). - **Two labels override the Issue Type**: `breaking change` → an entry only under `:boom:`, never in its regular section; `release highlight` → an extra entry under `:rocket:` on top of its regular entry. - **Only closed issues**. An open issue in the milestone is not generated, even if someone added a provisional entry by hand before the merge. - **Excluded issues**: `no changelog`, `release blocker`, `Task` issue type, `Rejected` project status. +- **Bugs that never reached production** stay out: a bug found and fixed while building a feature of the same, unreleased version never affected users. The team rarely labels them, so the flow finds them with `dev-only-bugs` and labels them `no changelog` (section 8, step 3). - **Excluded PRs**: a PR with `release blocker` or `no issue required` does not bring an entry of its own. - **An issue with no milestone** belongs to another (probably private) project: it is neither changelog candidate nor anomaly. @@ -39,7 +40,7 @@ The newest version goes at the top, right after the `# CHANGELOG` header. Inside ```markdown # CHANGELOG -## 2.19.0 (Unreleased) +## 2.20.0 (Unreleased) ### :boom: Breaking changes & Deprecations @@ -57,21 +58,27 @@ The newest version goes at the top, right after the `# CHANGELOG` header. Inside - Fix () (PR: [#PR](https://github.com/penpot/penpot/pull/PR)) -### :bug: Bugs fixed - -- Fix [#ISSUE](https://github.com/penpot/penpot/issues/ISSUE) (PR: [#PR](https://github.com/penpot/penpot/pull/PR)) - -### :sparkles: New features & Enhancements +### :tada: Features - Add [#ISSUE](https://github.com/penpot/penpot/issues/ISSUE) (PR: [#PR](https://github.com/penpot/penpot/pull/PR)) + +### :sparkles: Enhancements + +- Improve [#ISSUE](https://github.com/penpot/penpot/issues/ISSUE) (PR: [#PR](https://github.com/penpot/penpot/pull/PR)) + +### :bug: Bugs + +- Fix [#ISSUE](https://github.com/penpot/penpot/issues/ISSUE) (PR: [#PR](https://github.com/penpot/penpot/pull/PR)) ``` -- `:boom:` first, then `:rocket:`, `:lock:`, `:bug:`, `:sparkles:`. Other headings exist in old versions (`:heart:` community contributions); do not add them to new versions. +- `:boom:` first, then `:rocket:`, `:lock:`, `:tada:`, `:sparkles:`, `:bug:`. Features come before enhancements so they stand out to readers deciding whether to update. +- **This layout applies from 2.20.0.** Versions up to 2.19.x use the old one: `:bug: Bugs fixed` before `:sparkles: New features & Enhancements`, which mixes features and enhancements. Keep the old layout in those sections, including new 2.19.x patches, so a minor line reads the same. The in-app update dialog only reads `:rocket:`, so the change does not affect it. +- Other headings exist in old versions (`:heart:` community contributions); do not add them to new versions. - Include a section only when it has entries. Historical versions may have an empty `:boom:` or `:rocket:` heading; `boom --write` keeps the `:boom:` heading even when there are no entries. `:lock:` is omitted (and removed on `--write`) when the version has no security advisories. - One blank line between the last entry of a section and the next heading. - No manual line wrapping: one line per bullet or paragraph, however long. -## 4. Regular entries (`:bug:`, `:sparkles:`) +## 4. Regular entries (`:tada:`, `:sparkles:`, `:bug:`) The entry shape is: @@ -152,15 +159,18 @@ Generated from the advisories that declare the version in `patched_versions`; ne 1. **Confirm the version** (a semver string such as `2.19.0`) with the user when it is not obvious. 2. **Fetch the milestone issues**: `python3 scripts/gh.py issues "" --exclude "release blocker,no changelog"` (closed by default; add `--state all` only to inspect open issues). `gh.py` already drops `Task` and `Rejected`; pass `--include-tasks` / `--include-rejected` only to inspect them. To find what is missing from an existing section, add `--compare CHANGES.md`. -3. **Fetch PR details when needed**: `python3 scripts/gh.py prs ` (also `--milestone`). Use it for the PR author, labels and merge state. -4. **Categorize** by Issue Type and the two labels (section 2), and **build the descriptions** (section 4). -5. **Verify PR merge status** before writing: `python3 scripts/changelog.py check-merged `. Only merged PRs go in the changelog; if a closing PR was closed unmerged, look for the PR that superseded it. -6. **Reconcile the existing text** (section 9). -7. **Generate `:boom:`**: `python3 scripts/changelog.py boom ""` prints the section; add `--write` to replace it in `CHANGES.md`. The write replaces only that subsection, inserts it after the version header when missing, keeps the heading when there are no entries, and is idempotent. -8. **Generate `:rocket:`** from the `release highlight` label (section 6). -9. **Generate `:lock:`**: `python3 scripts/changelog.py security ""` prints the section; add `--write` to replace it in `CHANGES.md`. The contract lives in section 7. -10. **Cross-check PRs**: `python3 scripts/changelog.py cross-ref ""`. It lists merged milestone PRs missing from the section and warns about closed (unmerged) PRs. PRs linked from `:lock:` count as referenced; an advisory with no PR is invisible here by design. -11. **Generate the anomaly report**: `python3 scripts/changelog.py report ""`. It overwrites `CHANGES-ISSUES.md` with the current state (section 10). +3. **Keep out bugs that never reached production**: `python3 scripts/changelog.py dev-only-bugs ""` lists them; add `--apply` to label the sure ones `no changelog` on GitHub, then re-fetch step 2. The label puts the decision on the issue, so it survives a regeneration. + - **Sure**: a bug whose parent issue is a `Feature` or `Enhancement` of the same milestone. `--apply` labels these. + - **Candidates**: a bug whose parent is an epic of the same milestone, that carries the `regression` label, or whose "Affected versions" / "Environment" field names only hourly, develop, staging, PRE or a version of this milestone. `regression` also marks regressions users hit in production, and the field says where the bug was seen, not since when it exists, so never label these yourself: show them to the user, who labels the ones they confirm. The anomaly report lists them too (section 10). +4. **Fetch PR details when needed**: `python3 scripts/gh.py prs ` (also `--milestone`). Use it for the PR author, labels and merge state. +5. **Categorize** by Issue Type and the two labels (section 2), and **build the descriptions** (section 4). +6. **Verify PR merge status** before writing: `python3 scripts/changelog.py check-merged `. Only merged PRs go in the changelog; if a closing PR was closed unmerged, look for the PR that superseded it. +7. **Reconcile the existing text** (section 9). +8. **Generate `:boom:`**: `python3 scripts/changelog.py boom ""` prints the section; add `--write` to replace it in `CHANGES.md`. The write replaces only that subsection, inserts it after the version header when missing, keeps the heading when there are no entries, and is idempotent. +9. **Generate `:rocket:`** from the `release highlight` label (section 6). +10. **Generate `:lock:`**: `python3 scripts/changelog.py security ""` prints the section; add `--write` to replace it in `CHANGES.md`. The contract lives in section 7. +11. **Cross-check PRs**: `python3 scripts/changelog.py cross-ref ""`. It lists merged milestone PRs missing from the section and warns about closed (unmerged) PRs. PRs linked from `:lock:` count as referenced; an advisory with no PR is invisible here by design. +12. **Generate the anomaly report**: `python3 scripts/changelog.py report ""`. It overwrites `CHANGES-ISSUES.md` with the current state (section 10). ## 9. Pre-flight: reconcile before writing @@ -171,7 +181,7 @@ Re-fetch the milestone (do not trust cached data) and fix these in `CHANGES.md` 3. **Newly applied exclusion labels** (`no changelog`, `release blocker`) → remove the entry. 4. **Issue no longer closed, deleted, or now `Rejected`** → remove the entry. 5. **Unmerged or moved PR reference** → fix the reference or remove the entry. A PR merged in a different milestone is an anomaly (section 10); do not remove it silently. -6. **Issue Type changed to `Task`** → remove the entry. +6. **Issue Type changed** → to `Task`: remove the entry; to another type: move the entry to that type's section. 7. **Breaking change out of sync** (a `breaking change` issue outside `:boom:`, or a `:boom:` entry without the label) → regenerate the section; a labelled issue has no regular entry. 8. **Highlight out of sync** (`:rocket:` entry without the label, or a labelled issue missing) → remove it or add it. 9. **Missing valid issues** (closed, non-excluded, referenced nowhere) → add them to their section. @@ -188,6 +198,8 @@ Re-fetch the milestone (do not trust cached data) and fix these in `CHANGES.md` - **C — released X.Y.0 version without a `:rocket:` section** (gap; patches never carry highlights). - **D — `:rocket:` entry without issue AND PR references** (gap). +**Bugs that may never have reached production** are warnings too (section 8, step 3): sub-issues of a milestone feature still unlabelled (run `dev-only-bugs --apply`) and candidates for the user to confirm. They never count toward the total. + Anything else is a rule violation, not a report item: fix it in the pre-flight and re-run. ## 11. Pitfalls @@ -202,6 +214,6 @@ Anything else is a rule violation, not a report item: fix it in the pre-flight a ## 12. Where the code is - `scripts/gh.py`: `issues`, `issue`, `comments`, `prs`, `link-issue`, `advisories` (with `--patched-version`). Details: `mem:scripts/gh`. -- `scripts/changelog.py`: `check-merged`, `boom`, `security`, `cross-ref`, `report`. Details: `mem:scripts/changelog`. +- `scripts/changelog.py`: `check-merged`, `boom`, `security`, `cross-ref`, `dev-only-bugs`, `report`. Details: `mem:scripts/changelog`. - Tests: `scripts/test_gh.py`, `scripts/test_changelog.py`. - The note contract for issue authors: `mem:workflow/creating-issues`, section **Breaking changes**. diff --git a/scripts/changelog.py b/scripts/changelog.py index 280ffa1f85..e09cfa1218 100644 --- a/scripts/changelog.py +++ b/scripts/changelog.py @@ -10,6 +10,7 @@ Subcommands: boom Generate the :boom: section from `breaking change` issues security Generate the :lock: section from advisories fixed in a version cross-ref Compare merged milestone PRs against the changelog section + dev-only-bugs List milestone bugs that never reached production report Generate the anomaly report (CHANGES-ISSUES.md) Usage: @@ -20,6 +21,7 @@ Usage: python3 scripts/changelog.py security "2.19.1" python3 scripts/changelog.py security "2.19.1" --write python3 scripts/changelog.py cross-ref "2.16.0" + python3 scripts/changelog.py dev-only-bugs "2.16.0" [--apply] python3 scripts/changelog.py report "2.16.0" python3 scripts/changelog.py report "2.16.0" --changes CHANGES.md --output CHANGES-ISSUES.md @@ -615,6 +617,103 @@ def cmd_cross_ref(args: argparse.Namespace) -> None: print(f" #{pr['number']} {pr['title'][:80]}") +# ───────────────────────────────────────────── +# Subcommand: dev-only-bugs +# ───────────────────────────────────────────── + +# Bugs found and fixed while building a feature of the same, unreleased +# milestone never reached end users and stay out of the changelog. +AUTO_PARENT_TYPES = {"Feature", "Enhancement"} + +VERSION_FIELD_RE = re.compile( + r"^#{2,4}\s*(?:affected versions?|versions?|environment)[^\n]*\n(.*?)(?=^#{2,4}\s|\Z)", + re.IGNORECASE | re.MULTILINE | re.DOTALL, +) +DEV_ENV_RE = re.compile(r"\b(?:hourly|develop|staging|pre|unreleased)\b", re.IGNORECASE) +PROD_ENV_RE = re.compile( + r"\b(?:production|pro|main|saas|design\.penpot\.app|self[- ]?hosted?)\b", + re.IGNORECASE, +) +VERSION_RE = re.compile(r"\b(\d+)\.(\d+)(?:\.\d+)?") + + +def version_field(body: str | None) -> str: + """Return the "Affected versions" / "Environment" text of an issue body.""" + match = VERSION_FIELD_RE.search(body or "") + return " ".join(match.group(1).split()) if match else "" + + +def only_unreleased_envs(text: str, milestone: str) -> bool: + """True when the version text names unreleased environments only: + hourly, develop, staging, PRE or a version of this milestone. Any + production hint or other version number keeps the bug in the changelog.""" + current = tuple(milestone.split(".")[:2]) + versions = [m for m in VERSION_RE.findall(text)] + dev = bool(DEV_ENV_RE.search(text)) or current in versions + prod = bool(PROD_ENV_RE.search(text)) or any(v != current for v in versions) + return dev and not prod + + +def find_dev_only_bugs(milestone: str, issues: list[dict]) -> tuple[list[dict], list[dict]]: + """Split the milestone's changelog bugs that never reached production. + + ``issues`` come from ``gh.py issue`` (they carry ``body`` and + ``parent``). Returns ``(auto, suggested)``: ``auto`` bugs are + sub-issues of a Feature or Enhancement of this milestone and get the + `no changelog` label; ``suggested`` ones need a human to confirm.""" + auto, suggested = [], [] + for issue in issues: + if issue.get("issue_type") != "Bug" or issue.get("state") != "CLOSED": + continue + if {"no changelog", "release blocker"} & set(issue.get("labels", [])): + continue + parent = issue.get("parent") or {} + entry = {"issue": issue["number"], "issue_title": issue.get("title", "")} + if parent.get("milestone") == milestone: + entry["reason"] = f"sub-issue of {parent.get('issue_type')} #{parent['number']} in {milestone}" + (auto if parent.get("issue_type") in AUTO_PARENT_TYPES else suggested).append(entry) + continue + # `regression` says something that worked broke, not where: it + # also marks regressions users hit in production. + if "regression" in issue.get("labels", []): + entry["reason"] = "labelled `regression`" + suggested.append(entry) + continue + text = version_field(issue.get("body")) + if only_unreleased_envs(text, milestone): + entry["reason"] = f'only seen on "{text[:80]}"' + suggested.append(entry) + return auto, suggested + + +def fetch_milestone_bugs(milestone: str) -> list[dict]: + """Closed milestone bugs with body and parent (two batched lookups).""" + issues = json.loads(run_gh_py("issues", milestone)) + numbers = [str(i["number"]) for i in issues if i.get("issue_type") == "Bug"] + return json.loads(run_gh_py("issue", *numbers)) if numbers else [] + + +def cmd_dev_only_bugs(args: argparse.Namespace) -> None: + """List bugs that never reached production; --apply labels the sure ones.""" + auto, suggested = find_dev_only_bugs(args.milestone, fetch_milestone_bugs(args.milestone)) + + print(f"Label `no changelog` ({len(auto)}):") + for e in auto: + print(f" #{e['issue']} {e['issue_title'][:70]} ({e['reason']})") + print(f"Candidates to confirm ({len(suggested)}):") + for e in suggested: + print(f" #{e['issue']} {e['issue_title'][:70]} ({e['reason']})") + + if args.apply: + for e in auto: + subprocess.run( + ["gh", "issue", "edit", str(e["issue"]), "--repo", REPO, + "--add-label", "no changelog"], + check=True, capture_output=True, + ) + print(f"Labelled {len(auto)} issues `no changelog`", file=sys.stderr) + + # ───────────────────────────────────────────── # Subcommand: report # ───────────────────────────────────────────── @@ -872,6 +971,9 @@ def cmd_report(args: argparse.Namespace) -> None: ): anomalies_d.append({"version": ver, "line": line[:100]}) + # Bugs that may never have reached production: warnings, not anomalies. + dev_auto, dev_suggested = find_dev_only_bugs(milestone, fetch_milestone_bugs(milestone)) + def fmt_ms(ms: str | None) -> str: return ms if ms else "_none_" @@ -1033,6 +1135,26 @@ def cmd_report(args: argparse.Namespace) -> None: "✅ No highlight gaps found. All released X.Y.0 versions have properly referenced :rocket: entries.\n\n" ) + if dev_auto or dev_suggested: + f.write("## Bugs that may never have reached production\n\n") + f.write( + "These are warnings, not anomalies. Bugs found while building " + "a feature of this release never affected end users. Add the " + "`no changelog` label to the ones you confirm and re-run.\n\n" + ) + if dev_auto: + f.write( + "### Sub-issues of a feature of this milestone (run `dev-only-bugs --apply`)\n\n" + ) + for e in dev_auto: + f.write(f"- {issue_link_title(e['issue'], e['issue_title'][:80])}: {e['reason']}\n") + f.write("\n") + if dev_suggested: + f.write("### Candidates to confirm\n\n") + for e in dev_suggested: + f.write(f"- {issue_link_title(e['issue'], e['issue_title'][:80])}: {e['reason']}\n") + f.write("\n") + f.write("---\n\n") f.write("## Context\n\n") f.write(f"- Milestone: **{milestone}**\n") @@ -1100,6 +1222,15 @@ def main() -> None: help="Path to the changelog file (default: CHANGES.md)") p_cross.set_defaults(func=cmd_cross_ref) + p_dev = sub.add_parser( + "dev-only-bugs", + help="List milestone bugs that never reached production", + ) + p_dev.add_argument("milestone", help="Milestone title (e.g. '2.16.0')") + p_dev.add_argument("--apply", action="store_true", + help="Label the sub-issues of a milestone feature `no changelog`") + p_dev.set_defaults(func=cmd_dev_only_bugs) + p_report = sub.add_parser( "report", help="Generate the anomaly report (CHANGES-ISSUES.md)" ) diff --git a/scripts/gh.py b/scripts/gh.py index eebd5155fd..30c12e58f2 100755 --- a/scripts/gh.py +++ b/scripts/gh.py @@ -752,8 +752,10 @@ GQL_ISSUE_BY_NUMBER_QUERY_ITEM = """\ number title state + body milestone {{ title }} issueType {{ name }} + parent {{ number issueType {{ name }} milestone {{ title }} }} labels(first: 20) {{ nodes {{ name }} }} assignees(first: 10) {{ nodes {{ login }} }} closedByPullRequestsReferences(first: 5) {{ nodes {{ number }} }} @@ -775,12 +777,19 @@ def issue_node_to_dict(node: dict) -> dict: """Convert a GraphQL issue node to the shared issue dict shape.""" issue_type = node.get("issueType") ms = node.get("milestone") + parent = node.get("parent") return { "number": node["number"], "title": node["title"], "state": node["state"], + "body": node.get("body"), "milestone": ms["title"] if ms else None, "issue_type": issue_type["name"] if issue_type else None, + "parent": { + "number": parent["number"], + "issue_type": (parent.get("issueType") or {}).get("name"), + "milestone": (parent.get("milestone") or {}).get("title"), + } if parent else None, "labels": [lbl["name"] for lbl in node["labels"]["nodes"]], "assignees": node_assignees(node), "closing_prs": [pr["number"] for pr in node["closedByPullRequestsReferences"]["nodes"]], diff --git a/scripts/test_changelog.py b/scripts/test_changelog.py index e27d3de1d2..8eb0311c9f 100644 --- a/scripts/test_changelog.py +++ b/scripts/test_changelog.py @@ -657,5 +657,61 @@ class CmdSecurityTests(unittest.TestCase): self.assertIn("will not be listed", err) +def bug(number, body="", parent=None, labels=()): + return { + "number": number, + "title": f"Bug {number}", + "state": "CLOSED", + "issue_type": "Bug", + "labels": list(labels), + "body": body, + "parent": parent, + } + + +class OnlyUnreleasedEnvsTests(unittest.TestCase): + def check(self, text): + return ch.only_unreleased_envs(text, "2.19.0") + + def test_dev_environments(self): + for text in ["HOURLY", "https://hourly.penpot.dev/", "PRE & hourly", + "staging (current)", "develop (2.19.0 unreleased)", + "2.19.0-RC1 (develop)", "2.19.0"]: + with self.subTest(text=text): + self.assertTrue(self.check(text)) + + def test_production_hints(self): + for text in ["PRO", "pre/pro/hourly", "develop / staging / main", + "2.18.0-RC5", "2.18.1, 2.19.0-RC1", "self-hosted: 2.17.2", + "design.penpot.app and self-host"]: + with self.subTest(text=text): + self.assertFalse(self.check(text)) + + def test_no_signal(self): + for text in ["", "_No response_", "-", "development build"]: + with self.subTest(text=text): + self.assertFalse(self.check(text)) + + +class FindDevOnlyBugsTests(unittest.TestCase): + def test_splits_auto_and_suggested(self): + issues = [ + bug(1, parent={"number": 10, "issue_type": "Feature", "milestone": "2.19.0"}), + bug(2, parent={"number": 11, "issue_type": "Enhancement", "milestone": "2.19.0"}), + bug(3, parent={"number": 12, "issue_type": "EPIC", "milestone": "2.19.0"}), + bug(4, parent={"number": 13, "issue_type": "Feature", "milestone": "2.18.0"}), + bug(5, body="### Affected versions\n\nhourly\n"), + bug(6, body="### Affected versions\n\n2.18.2\n"), + bug(7, parent={"number": 10, "issue_type": "Feature", "milestone": "2.19.0"}, + labels=["no changelog"]), + {**bug(8, parent={"number": 10, "issue_type": "Feature", "milestone": "2.19.0"}), + "issue_type": "Enhancement"}, + bug(9, body="### Affected versions\n\n2.18.2\n", labels=["regression"]), + ] + auto, suggested = ch.find_dev_only_bugs("2.19.0", issues) + self.assertEqual([e["issue"] for e in auto], [1, 2]) + self.assertEqual([e["issue"] for e in suggested], [3, 5, 9]) + + if __name__ == "__main__": unittest.main(verbosity=2) From bcac14a4a85003ebe92a12afb53e858957f49c42 Mon Sep 17 00:00:00 2001 From: Andrey Antukh Date: Wed, 11 Nov 2026 13:02:57 +0100 Subject: [PATCH 09/29] :bug: Skip circular library links on binfile import (#12062) * :bug: Skip circular library links on binfile import File import recreated circular library references that the link-file-to-library command already rejects. The v3 importer now skips manifest relations that would close a cycle, or link a file to itself, and logs each skipped link. Closes #12052. AI-assisted-by: muse-spark-1.3-contributor-free :bug: Address review findings on circular import guard F1 extracts the cycle check into bfc/library-link-status and reuses it in link-file-to-library and v3 import, keeping UI errors intact. F3 orders get-files-rels and manifest relations so the first edge wins deterministically, and documents it. F2 adds a three-node transitive cycle test; F4 asserts file_library_sync has no orphan rows for skipped links and matches rels otherwise. AI-assisted-by: muse-spark-1.3-contributor :bug: Pass team-id in binfile v3 circular library tests The four circular library relation tests (import-skips-circular, import-skips-transitive-cycle, import-skips-self-referencing, import-keeps-acyclic) called v3/import-files! without ::bfc/team-id. resolve-and-link-libraries asserts (uuid? team-id), so with enable-backend-asserts the tests errored instead of running. Add (assoc ::bfc/team-id (:default-team-id profile)) to each test, mirroring the production RPC import path and the earlier fix in a36193e54b for the other binfile v3 import tests. Verified: 1338 tests, 6910 assertions, 0 failures. AI-assisted-by: mistral-large-4 * :bug: Guard the v1 import and restore the RPC check order The version 1 importer recreated the circular library references that the version 3 importer already skips, and a user reaches it through `import-binfile` and the import job. It now asks `bfc/library-link-status` before inserting each relation, so a self link or an edge that would close a cycle is skipped and logged, and it processes the relations in sorted order like v3 does. Closes #12052. The version 2 importer stays out on purpose: only the developer REPL reaches it, and a note in the code records the gap. `link-file-to-library` keeps the shared helper but restores the order of the base commit: the self link guard, then the edition permission and team ownership checks, and only then the cycle check. A caller without permission gets `not-found` again instead of the circular error, and a rejected request no longer runs the recursive library query first. Tests: a v1 import that recreates a cycle, the self link error of the command, and the permission check running before the cycle check. AI-assisted-by: deepseek-v4.1-flash * :bug: Tolerate repeated file entries in the v1 binfile import The v1 exporter unioned the requested files with their libraries without deduplicating, so a package exported from a file inside a library cycle (see #12052) lists that file twice in the metadata. The v1 import created one file per entry, so the second insert hit the primary key of `file` and aborted the whole import with "file already exists". The importer now reads the pair of objects of every entry to keep the stream in sync but imports the first occurrence only, warns about the repeats, and answers with one id per imported file. The v1 export keeps its behavior: it has no caller in the product, and only the tests and the developer REPL use it. Tests: a package with a repeated entry, built with the v1 export from a cyclic library pair, imports the file once and leaves no duplicated rows. AI-assisted-by: deepseek-v4.1-flash --- .serena/memories/backend/core.md | 3 +- backend/src/app/binfile/common.clj | 21 +- backend/src/app/binfile/v1.clj | 157 +++++++----- backend/src/app/binfile/v2.clj | 5 + backend/src/app/binfile/v3.clj | 47 +++- backend/src/app/rpc/commands/files.clj | 9 +- backend/test/backend_tests/binfile_test.clj | 248 ++++++++++++++++++- backend/test/backend_tests/rpc_file_test.clj | 37 +++ 8 files changed, 441 insertions(+), 86 deletions(-) diff --git a/.serena/memories/backend/core.md b/.serena/memories/backend/core.md index 16978ac1ff..77ba723328 100644 --- a/.serena/memories/backend/core.md +++ b/.serena/memories/backend/core.md @@ -67,7 +67,8 @@ Every background job is a job-def: a plain `(defn execute-X [cfg params] ...)` i - Job resources (`app.jobs.storage`) are the objects a job owns through `job.resource_id`: they live in the `job-resource` bucket, are written with the profile of the job as owner (only that profile can read them back from `/assets/by-id`) and are never deduplicated. They carry no expiry of their own, because `expired-at` is stored as `deleted_at` and an object with it is unreadable from birth: the retention of the artifact is the retention of its job row. The writer marks the object as touched right away, so an artifact that no row references (a crash between the upload and the completion) is reclaimed by `storage-gc-touched`, and a row that does reference it freezes it until the jobs GC deletes the row and touches the object. - The binfile jobs (`:export-binfile` and `:import-binfile`) run the core of `bf.v1`/`bf.v3` through `app.binfile.jobs`, which turns the `:progress` taps of the core into `progress` events of the job with `jobs/heartbeat` (the `modified_at` touch at most once per second, progress events at up to ~200ms). Each event is a self-contained milestone: a `:stage` keyword of one vocabulary shared by the two formats (the same work has a different name in v1 and v3) and an optional `:counters` map, with the counter of the activity named after its own stage and the outer scopes in course as context, which the tap carries itself (the reader puts `::outer-counters` in every tap inside them — today only `:files`, extendable to `:teams`/`:projects` for multi-scope exports — so the adapter remembers nothing between taps and a tap that happens outside any scope has no context); the ids and names the taps carry never reach the event. Every tap is reported, so a tap with no units to count still keeps the job alive, and a milestone the throttle drops loses nothing because the next one carries the same state. The adapter checks that the job is still active before the run, and the beat raises an `:interrupt` right there on the run thread when the job is no longer active, so the work stops at its next event point inside the transaction of its handler, which rolls back. A run whose last tap predates the interruption is discarded by the check once it returns. The adapter passes the job id from the handler context. Their job-def components carry pool, storage, metrics and msgbus, because a job with a `profile_id` cannot store an event without a msgbus on its cfg. Both readers answer with the same shape (`:file-ids` and, in version 3, the resolution of the libraries), because the version 1 reader returns a bare set of file ids. - Both binfile handlers revalidate their permission at run time, not only when the job is created, because it can be revoked while the job waits: the export checks the read permission of every file before reading anything and stores the package in `job-resource` as the resource of the job (answering with the completion envelope), and the import checks the edition permission of the destination and runs the core inside a transaction, which is what makes a cancelled import roll back instead of leaving half of a file. The import releases its package (marks it as deleted and drops the reference from the job row in the same transaction) and deletes its temporary copy once the job is terminal, whatever the outcome. Both handlers check their own result against a schema before returning it (the import job-def declares that same schema as its result schema, which is what types the resolution when the row is read). -- The v3 import only materializes a library relation when both ids belong to the package and differ; a relation that points outside the package or a self link is ignored with a warning. Migration `0158-del-forbidden-file-library-rels` deletes the legacy cross-team and self rows from `file_library_rel` and their `file_library_sync` rows, sync rows first. +- The v3 and v1 imports only materialize a library relation when both ids belong to the package and the link would neither be a self link nor close a dependency cycle (the same `bfc/library-link-status` check that `link-file-to-library` uses to raise); relations are processed in sorted `(file_id, library_file_id)` order, which `get-files-rels` also returns, so the first edge wins deterministically, and every edge it ignores is logged with a warning. The v2 import stays out of that guard on purpose: it is only reachable from the developer REPL (`app.srepl.binfile`) and keeps inserting the relations as they come. Migration `0158-del-forbidden-file-library-rels` deletes the legacy cross-team and self rows from `file_library_rel` and their `file_library_sync` rows, sync rows first. +- A version 1 package lists its files in the metadata and the importer creates one file per entry. The v1 export (gone from the product; only tests and the developer REPL use it) unioned the requested files with their libraries without deduplicating, so a package exported from a file inside a library cycle lists that file twice. The importer reads the pair of objects of every entry to keep the stream in sync, imports the first occurrence only, warns about the repeats, and answers with one id per imported file. - The domain cfg of a binfile request lives in `app.binfile.common` (`export-cfg`, `import-cfg`), so the legacy RPC and the jobs freeze the same ids, resolve the same team features and apply the same anti-zip-bomb limits. `resolve-export-type` is the translation the legacy RPC needs for its deprecated boolean flags; the creation command asks the caller for the type instead of guessing a default. - Two retention knobs, easy to confuse: `:jobs-retention` sweeps terminal internal rows (the ones without a profile), while `:jobs-user-ttl` is the default expiry a user-facing job gets when it is created; the jobs GC deletes the row once `expires_at` passes and touches its resource. The heavy work of a user job runs on its own `:binfile` queue (`:worker-binfile-parallelism`, default 1) so it does not block the default runner. - The quote of a family of jobs (`::export-jobs-per-profile`, `::import-jobs-per-profile`) counts the jobs of the profile that are still on their way (`new`/`scheduled`/`running`/`retry`) whose name the registry gives for that family, so a new job of the family is covered without touching the quote. diff --git a/backend/src/app/binfile/common.clj b/backend/src/app/binfile/common.clj index efc6174d4b..43dbc60c2a 100644 --- a/backend/src/app/binfile/common.clj +++ b/backend/src/app/binfile/common.clj @@ -488,10 +488,13 @@ ON fls.file_id = flr.file_id AND fls.library_file_id = flr.library_file_id WHERE flr.file_id = ANY(?) - AND l.deleted_at IS NULL;") + AND l.deleted_at IS NULL + ORDER BY flr.file_id, flr.library_file_id;") (defn get-files-rels - "Given a set of file-id's, return all matching relations with the libraries" + "Given a set of file-id's, return all matching relations with the libraries. + Rows come back in stable `(file-id, library-file-id)` order so export + and import resolve cycles deterministically." [cfg ids] (assert (set? ids) "expected a set of uuids") @@ -534,6 +537,20 @@ (->> (db/exec! conn [sql:get-libraries ids']) (into #{} xf-map-id)))))) +(defn library-link-status + "Check whether linking `file-id` to `library-id` is safe. Returns + `:ok` when the link can be created, `:self-link` when both ids are + the same, and `:circular` when the link would close a dependency + cycle (i.e. `file-id` is already among the transitive libraries of + `library-id`). Shared by the UI command (`link-file-to-library`, + which raises) and the v3 importer (`import-file-relations`, which + skips and logs), so both stay consistent." + [cfg file-id library-id] + (cond + (= file-id library-id) :self-link + (contains? (get-libraries cfg [library-id]) file-id) :circular + :else :ok)) + (defn get-file-object-thumbnails "Return all file object thumbnails for a given file." [cfg file-id] diff --git a/backend/src/app/binfile/v1.clj b/backend/src/app/binfile/v1.clj index 76fc870f5d..9b9bc5af27 100644 --- a/backend/src/app/binfile/v1.clj +++ b/backend/src/app/binfile/v1.clj @@ -482,9 +482,11 @@ (bfm/apply-pending-migrations! cfg) ;; Knowing that the ids of the created files are in index, - ;; just lookup them and return them in the same shape as v3. + ;; just lookup them and return them in the same shape as v3. The + ;; metadata of an old package may repeat a file id, so the answer + ;; carries one id per imported file. (let [files (-> bfc/*state* deref :files)] - {:file-ids (into [] (keep #(get-in @bfc/*state* [:index %])) files)}))))) + {:file-ids (into [] (comp (keep #(get-in @bfc/*state* [:index %])) (distinct)) files)}))))) (defmethod read-import :v1 [options] @@ -510,89 +512,114 @@ (defmethod read-section :v1/files [{:keys [::bfc/input ::bfc/project-id ::bfc/name] :as system}] - (doseq [[idx expected-file-id] (d/enumerate (-> bfc/*state* deref :files))] - (let [file (read-obj! input) - media (read-obj! input) + (let [imported (volatile! #{})] + (doseq [[idx expected-file-id] (d/enumerate (-> bfc/*state* deref :files))] + (let [file (read-obj! input) + media (read-obj! input) + file-id (:id file)] - file-id (:id file) - file-id' (bfc/lookup-index file-id) + (when (not= file-id expected-file-id) + (ex/raise :type :validation + :code :inconsistent-penpot-file + :found-id file-id + :expected-id expected-file-id + :hint "the penpot file seems corrupt, found unexpected uuid (file-id)")) - file (bfc/clean-file-features file) - thumbnails (:thumbnails file)] + (if (contains? @imported file-id) + ;; Old exports could repeat a file id in the metadata (the file was + ;; also one of its own libraries). The entry is read above to keep + ;; the stream in sync, but the file is imported only once. + (l/warn :hint "ignoring repeated file entry" + :file-id (str file-id) + ::l/sync? true) - (when (not= file-id expected-file-id) - (ex/raise :type :validation - :code :inconsistent-penpot-file - :found-id file-id - :expected-id expected-file-id - :hint "the penpot file seems corrupt, found unexpected uuid (file-id)")) + (let [file-id' (bfc/lookup-index file-id) + file (bfc/clean-file-features file) + thumbnails (:thumbnails file)] - (l/dbg :hint "processing file" - :id (str file-id) - :features (:features file) - :version (-> file :data :version) - :media (count media) - :thumbnails (count thumbnails) - ::l/sync? true) + (vswap! imported conj file-id) - (when (seq thumbnails) - (let [thumbnails (remap-thumbnails thumbnails file-id')] - (l/dbg :hint "updated index with thumbnails" - :total (count thumbnails) - ::l/sync? true) - (vswap! bfc/*state* update :thumbnails bfc/into-vec thumbnails))) + (l/dbg :hint "processing file" + :id (str file-id) + :features (:features file) + :version (-> file :data :version) + :media (count media) + :thumbnails (count thumbnails) + ::l/sync? true) - (when (seq media) - ;; Update index with media - (l/dbg :hint "update index with media" :total (count media) ::l/sync? true) - (vswap! bfc/*state* update :index bfc/update-index (map :id media)) + (when (seq thumbnails) + (let [thumbnails (remap-thumbnails thumbnails file-id')] + (l/dbg :hint "updated index with thumbnails" + :total (count thumbnails) + ::l/sync? true) + (vswap! bfc/*state* update :thumbnails bfc/into-vec thumbnails))) - ;; Store file media for later insertion - (l/dbg :hint "update media references" ::l/sync? true) - (vswap! bfc/*state* update :media into (map #(update % :id bfc/lookup-index)) media)) + (when (seq media) + ;; Update index with media + (l/dbg :hint "update index with media" :total (count media) ::l/sync? true) + (vswap! bfc/*state* update :index bfc/update-index (map :id media)) - (let [file (-> file - (assoc :id file-id') - (cond-> (and (= idx 0) (some? name)) - (assoc :name name)) - (assoc :project-id project-id) - (dissoc :thumbnails)) - file (bfc/process-file system file)] + ;; Store file media for later insertion + (l/dbg :hint "update media references" ::l/sync? true) + (vswap! bfc/*state* update :media into (map #(update % :id bfc/lookup-index)) media)) - ;; All features that are enabled and requires explicit migration are - ;; added to the state for a posterior migration step. - (doseq [feature (-> (::bfc/features system) - (set/difference cfeat/no-migration-features) - (set/difference (:features file)))] - (vswap! bfc/*state* update :pending-to-migrate (fnil conj []) [feature file-id'])) + (let [file (-> file + (assoc :id file-id') + (cond-> (and (= idx 0) (some? name)) + (assoc :name name)) + (assoc :project-id project-id) + (dissoc :thumbnails)) + file (bfc/process-file system file)] - (l/dbg :hint "create file" :id (str file-id') ::l/sync? true) - (bfc/save-file! system file ::db/return-keys false) + ;; All features that are enabled and requires explicit migration + ;; are added to the state for a posterior migration step. + (doseq [feature (-> (::bfc/features system) + (set/difference cfeat/no-migration-features) + (set/difference (:features file)))] + (vswap! bfc/*state* update :pending-to-migrate (fnil conj []) [feature file-id'])) - file-id')))) + (l/dbg :hint "create file" :id (str file-id') ::l/sync? true) + (bfc/save-file! system file ::db/return-keys false) + + file-id'))))))) (defmethod read-section :v1/rels - [{:keys [::db/conn ::bfc/input ::bfc/timestamp]}] - (let [rels (read-obj! input) + [{:keys [::db/conn ::bfc/input ::bfc/timestamp] :as cfg}] + (let [rels (->> (read-obj! input) + (sort-by (fn [{:keys [file-id library-file-id]}] + [(str file-id) (str library-file-id)]))) ids (into #{} (-> bfc/*state* deref :files))] - ;; Insert all file relations (doseq [{:keys [library-file-id] :as rel} rels] (let [rel (-> rel (update :file-id bfc/lookup-index) (update :library-file-id bfc/lookup-index))] (if (contains? ids library-file-id) - (do - (l/dbg :hint "create file library link" - :file-id (:file-id rel) - :lib-id (:library-file-id rel) - ::l/sync? true) - (let [rel-params (dissoc rel :synced-at)] - (db/insert! conn :file-library-rel rel-params) - (bfc/upsert-file-library-sync! conn {:file-id (:file-id rel-params) - :library-file-id (:library-file-id rel-params) - :synced-at (or (:synced-at rel) - timestamp)}))) + (case (bfc/library-link-status cfg (:file-id rel) (:library-file-id rel)) + :self-link + (l/warn :hint "ignoring self-referencing library link" + :file-id (:file-id rel) + :lib-id (:library-file-id rel) + ::l/sync? true) + + :circular + (l/warn :hint "ignoring circular library link" + :file-id (:file-id rel) + :lib-id (:library-file-id rel) + ::l/sync? true) + + :ok + (do + (l/dbg :hint "create file library link" + :file-id (:file-id rel) + :lib-id (:library-file-id rel) + ::l/sync? true) + (let [rel-params (dissoc rel :synced-at)] + (db/insert! conn :file-library-rel rel-params) + (bfc/upsert-file-library-sync! conn {:file-id (:file-id rel-params) + :library-file-id (:library-file-id rel-params) + :synced-at (or (:synced-at rel) + timestamp)})))) (l/warn :hint "ignoring file library link" :file-id (:file-id rel) diff --git a/backend/src/app/binfile/v2.clj b/backend/src/app/binfile/v2.clj index b339505ef8..06bb22b7c8 100644 --- a/backend/src/app/binfile/v2.clj +++ b/backend/src/app/binfile/v2.clj @@ -319,6 +319,11 @@ (db/insert! conn :file-tagged-object-thumbnail thumbnail {::db/return-keys false}))) + ;; NOTE: v2 packages are only produced and imported through the + ;; developer REPL (`app.srepl.binfile`), never through the product, so + ;; this path keeps inserting the relations as they come: unlike v1 and + ;; v3 it does not skip self-referencing or circular links (see + ;; `bfc/library-link-status`). (doseq [rel (read-obj cfg :file-rels file-id)] (let [rel (-> rel (update :file-id bfc/lookup-index) diff --git a/backend/src/app/binfile/v3.clj b/backend/src/app/binfile/v3.clj index 9ec984e524..69cdf9fa95 100644 --- a/backend/src/app/binfile/v3.clj +++ b/backend/src/app/binfile/v3.clj @@ -1038,26 +1038,49 @@ file-id'))) (defn- import-file-relations + "Import library relations from the manifest, ignoring relations whose + endpoints are not both part of the package. Of the rest, skip edges that + would close a dependency cycle or link a file to itself (same check as + `link-file-to-library`, see `bfc/library-link-status`). Relations are + processed in sorted manifest order and the first edge wins: an edge + that would close a cycle with previously imported relations is skipped + and logged." [{:keys [::db/conn ::manifest ::bfc/timestamp] :as cfg}] - (let [relations (:relations manifest) + (let [relations (sort-by (fn [[file-id libr-id]] [(str file-id) (str libr-id)]) + (:relations manifest)) file-ids (into #{} (map :id) (:files manifest))] (doseq [[index [file-id libr-id]] (d/enumerate relations)] (tap-progress cfg {:section :relations - :current (inc index) :total (count relations)}) + :current (inc index) + :total (count relations)}) - (if (and (not= file-id libr-id) - (contains? file-ids file-id) + (if (and (contains? file-ids file-id) (contains? file-ids libr-id)) (let [file-id (bfc/lookup-index file-id) libr-id (bfc/lookup-index libr-id)] - (l/dbg :hint "create file library link" - :file-id (str file-id) - :lib-id (str libr-id) - ::l/sync? true) - (let [rel-params {:file-id file-id - :library-file-id libr-id}] - (db/insert! conn :file-library-rel rel-params) - (bfc/upsert-file-library-sync! conn (assoc rel-params :synced-at timestamp)))) + (case (bfc/library-link-status cfg file-id libr-id) + :self-link + (l/warn :hint "ignoring self-referencing library link" + :file-id (str file-id) + :lib-id (str libr-id) + ::l/sync? true) + + :circular + (l/warn :hint "ignoring circular library link" + :file-id (str file-id) + :lib-id (str libr-id) + ::l/sync? true) + + :ok + (do + (l/dbg :hint "create file library link" + :file-id (str file-id) + :lib-id (str libr-id) + ::l/sync? true) + (let [rel-params {:file-id file-id + :library-file-id libr-id}] + (db/insert! conn :file-library-rel rel-params) + (bfc/upsert-file-library-sync! conn (assoc rel-params :synced-at timestamp)))))) (l/warn :hint "ignoring invalid file library relation" :file-id (str file-id) :lib-id (str libr-id)))))) diff --git a/backend/src/app/rpc/commands/files.clj b/backend/src/app/rpc/commands/files.clj index b355bffc3d..4bf2717eee 100644 --- a/backend/src/app/rpc/commands/files.clj +++ b/backend/src/app/rpc/commands/files.clj @@ -1240,11 +1240,10 @@ (check-edition-permissions! conn profile-id library-id) (check-library-team-ownership! conn file-id library-id) - (let [transitive-deps (bfc/get-libraries cfg [library-id])] - (when (contains? transitive-deps file-id) - (ex/raise :type :validation - :code :circular-library-reference - :hint "linking this library would create a circular dependency"))) + (when (= :circular (bfc/library-link-status cfg file-id library-id)) + (ex/raise :type :validation + :code :circular-library-reference + :hint "linking this library would create a circular dependency")) (link-file-to-library conn params) (bfc/get-libraries cfg [library-id])) diff --git a/backend/test/backend_tests/binfile_test.clj b/backend/test/backend_tests/binfile_test.clj index 9e525af2aa..77ec4ec380 100644 --- a/backend/test/backend_tests/binfile_test.clj +++ b/backend/test/backend_tests/binfile_test.clj @@ -3154,7 +3154,8 @@ [[(str (:id file)) (str (:id file))]]) [file-id :as result] (import-bundle profile evil)] (t/is (= 1 (count result))) - (t/is (empty? (th/db-query :file-library-rel {:file-id file-id}))))) + (t/is (empty? (th/db-query :file-library-rel {:file-id file-id}))) + (t/is (empty? (th/db-query :file-library-sync {:file-id file-id}))))) (t/deftest import-v3-keeps-valid-library-relation (let [profile (th/create-profile* 1) @@ -3173,6 +3174,251 @@ (t/is (= 1 (count (th/db-query :file-library-rel {:file-id (:id new-file)}))))))) +(t/deftest import-skips-circular-library-relations + ;; https://github.com/penpot/penpot/issues/12052: two libraries that use + ;; each other (possible before the 2.17.1 guard) export both relations in + ;; the manifest; the import must skip the one closing the cycle instead + ;; of recreating it. The cycle is built with direct inserts because the + ;; UI command `link-file-to-library` already rejects it. Relations are + ;; processed in sorted manifest order and the first edge wins. + (let [profile (th/create-profile* 1) + lib-a (th/create-file* 1 {:profile-id (:id profile) + :project-id (:default-project-id profile) + :is-shared true + :name "Library A"}) + lib-b (th/create-file* 2 {:profile-id (:id profile) + :project-id (:default-project-id profile) + :is-shared true + :name "Library B"})] + (db/insert! th/*system* :file-library-rel + {:file-id (:id lib-a) + :library-file-id (:id lib-b)}) + (db/insert! th/*system* :file-library-rel + {:file-id (:id lib-b) + :library-file-id (:id lib-a)}) + + (let [output (tmp/tempfile :suffix ".zip")] + (v3/export-files! + (-> th/*system* + (assoc ::bfc/file-ids #{(:id lib-a) (:id lib-b)}) + (assoc ::bfc/export-type :include-libraries) + (assoc ::bfc/output (io/output-stream output)))) + + (let [result (import-bundle profile output) + rels (mapcat #(db/query th/*system* :file-library-rel {:file-id %}) + result) + syncs (mapcat #(db/query th/*system* :file-library-sync {:file-id %}) + result)] + (t/is (= 2 (count result))) + ;; only one of the two cycle edges survives the import + (t/is (= 1 (count rels))) + ;; the skipped edge leaves no sync row behind + (t/is (= 1 (count syncs))) + (t/is (= (set (map (juxt :file-id :library-file-id) syncs)) + (set (map (juxt :file-id :library-file-id) rels)))) + ;; no imported file lists itself among its transitive libraries + (doseq [file-id result] + (t/is (not (contains? (bfc/get-libraries th/*system* [file-id]) + file-id)))))))) + +(t/deftest import-skips-transitive-cycle + ;; A three-node cycle (A -> B -> C -> A) only breaks when the check uses + ;; the transitive closure: a direct-edge-only check would keep all three + ;; edges. Built with direct inserts because the UI command rejects it. + (let [profile (th/create-profile* 1) + lib-a (th/create-file* 1 {:profile-id (:id profile) + :project-id (:default-project-id profile) + :is-shared true + :name "Library A"}) + lib-b (th/create-file* 2 {:profile-id (:id profile) + :project-id (:default-project-id profile) + :is-shared true + :name "Library B"}) + lib-c (th/create-file* 3 {:profile-id (:id profile) + :project-id (:default-project-id profile) + :is-shared true + :name "Library C"})] + (db/insert! th/*system* :file-library-rel + {:file-id (:id lib-a) + :library-file-id (:id lib-b)}) + (db/insert! th/*system* :file-library-rel + {:file-id (:id lib-b) + :library-file-id (:id lib-c)}) + (db/insert! th/*system* :file-library-rel + {:file-id (:id lib-c) + :library-file-id (:id lib-a)}) + + (let [output (tmp/tempfile :suffix ".zip")] + (v3/export-files! + (-> th/*system* + (assoc ::bfc/file-ids #{(:id lib-a) (:id lib-b) (:id lib-c)}) + (assoc ::bfc/export-type :include-libraries) + (assoc ::bfc/output (io/output-stream output)))) + + (let [result (import-bundle profile output) + rels (mapcat #(db/query th/*system* :file-library-rel {:file-id %}) + result) + syncs (mapcat #(db/query th/*system* :file-library-sync {:file-id %}) + result)] + (t/is (= 3 (count result))) + ;; only two of the three cycle edges survive the import + (t/is (= 2 (count rels))) + ;; the skipped edge leaves no sync row behind + (t/is (= 2 (count syncs))) + (t/is (= (set (map (juxt :file-id :library-file-id) syncs)) + (set (map (juxt :file-id :library-file-id) rels)))) + ;; no imported file lists itself among its transitive libraries + (doseq [file-id result] + (t/is (not (contains? (bfc/get-libraries th/*system* [file-id]) + file-id)))))))) + +(t/deftest import-keeps-acyclic-library-relations + ;; Control test: a chain without cycles must be imported untouched, so + ;; the new guard only skips edges that would close a cycle. + (let [profile (th/create-profile* 1) + lib-a (th/create-file* 1 {:profile-id (:id profile) + :project-id (:default-project-id profile) + :is-shared true + :name "Library A"}) + lib-b (th/create-file* 2 {:profile-id (:id profile) + :project-id (:default-project-id profile) + :is-shared true + :name "Library B"}) + lib-c (th/create-file* 3 {:profile-id (:id profile) + :project-id (:default-project-id profile) + :is-shared true + :name "Library C"})] + (db/insert! th/*system* :file-library-rel + {:file-id (:id lib-a) + :library-file-id (:id lib-b)}) + (db/insert! th/*system* :file-library-rel + {:file-id (:id lib-b) + :library-file-id (:id lib-c)}) + + (let [output (tmp/tempfile :suffix ".zip")] + (v3/export-files! + (-> th/*system* + (assoc ::bfc/file-ids #{(:id lib-a) (:id lib-b) (:id lib-c)}) + (assoc ::bfc/export-type :include-libraries) + (assoc ::bfc/output (io/output-stream output)))) + + (let [result (import-bundle profile output) + rels (mapcat #(db/query th/*system* :file-library-rel {:file-id %}) + result) + syncs (mapcat #(db/query th/*system* :file-library-sync {:file-id %}) + result)] + (t/is (= 3 (count result))) + (t/is (= 2 (count rels))) + ;; every kept relation has its sync row + (t/is (= 2 (count syncs))) + (t/is (= (set (map (juxt :file-id :library-file-id) syncs)) + (set (map (juxt :file-id :library-file-id) rels)))))))) + +(t/deftest import-v1-skips-circular-library-relations + ;; Issue #12052 also names the version 1 importer: importing a package + ;; whose libraries use each other must not recreate the cycle. The cycle + ;; is built with direct inserts because `link-file-to-library` already + ;; rejects it. The exported file only *uses* one of the libraries, so the + ;; package carries the cycle the same way a real export does. + (let [profile (th/create-profile* 1) + file (prepare-simple-file profile) + lib-a (th/create-file* 2 {:profile-id (:id profile) + :project-id (:default-project-id profile) + :is-shared true + :name "Library A"}) + lib-b (th/create-file* 3 {:profile-id (:id profile) + :project-id (:default-project-id profile) + :is-shared true + :name "Library B"})] + (th/link-file-to-library* {:file-id (:id file) + :library-id (:id lib-a)}) + (db/insert! th/*system* :file-library-rel + {:file-id (:id lib-a) + :library-file-id (:id lib-b)}) + (db/insert! th/*system* :file-library-rel + {:file-id (:id lib-b) + :library-file-id (:id lib-a)}) + + (let [output (tmp/tempfile :suffix ".penpot")] + (v1/export-files! + (-> th/*system* + (assoc ::bfc/file-ids #{(:id file)}) + (assoc ::bfc/include-libraries true)) + (io/output-stream output)) + + (let [result (:file-ids (-> th/*system* + (assoc ::bfc/project-id (:default-project-id profile)) + (assoc ::bfc/profile-id (:id profile)) + (assoc ::bfc/input output) + (v1/import-files!))) + rels (mapcat #(db/query th/*system* :file-library-rel {:file-id %}) + result) + syncs (mapcat #(db/query th/*system* :file-library-sync {:file-id %}) + result)] + (t/is (= 3 (count result))) + ;; the relation of the file to its library, plus one of the two + ;; cycle edges; the edge that would close the cycle is skipped + (t/is (= 2 (count rels))) + ;; the skipped edge leaves no sync row behind + (t/is (= 2 (count syncs))) + (t/is (= (set (map (juxt :file-id :library-file-id) syncs)) + (set (map (juxt :file-id :library-file-id) rels)))) + ;; no imported file lists itself among its transitive libraries + (doseq [file-id result] + (t/is (not (contains? (bfc/get-libraries th/*system* [file-id]) + file-id)))))))) + +(t/deftest import-v1-tolerates-repeated-file-entries + ;; Old v1 exports could list the same file twice in the package metadata: + ;; the exporter unioned the requested files with their libraries without + ;; deduplicating, and a file inside a library cycle is one of its own + ;; libraries. The import must read every entry and import each file once. + (let [profile (th/create-profile* 1) + lib-a (th/create-file* 1 {:profile-id (:id profile) + :project-id (:default-project-id profile) + :is-shared true + :name "Library A"}) + lib-b (th/create-file* 2 {:profile-id (:id profile) + :project-id (:default-project-id profile) + :is-shared true + :name "Library B"})] + (db/insert! th/*system* :file-library-rel + {:file-id (:id lib-a) + :library-file-id (:id lib-b)}) + (db/insert! th/*system* :file-library-rel + {:file-id (:id lib-b) + :library-file-id (:id lib-a)}) + + (let [output (tmp/tempfile :suffix ".penpot")] + ;; exporting only lib-a repeats its id in the metadata, because it is + ;; among its own transitive libraries + (v1/export-files! + (-> th/*system* + (assoc ::bfc/file-ids #{(:id lib-a)}) + (assoc ::bfc/include-libraries true)) + (io/output-stream output)) + + (let [result (:file-ids (-> th/*system* + (assoc ::bfc/project-id (:default-project-id profile)) + (assoc ::bfc/profile-id (:id profile)) + (assoc ::bfc/input output) + (v1/import-files!))) + rels (mapcat #(db/query th/*system* :file-library-rel {:file-id %}) + result) + syncs (mapcat #(db/query th/*system* :file-library-sync {:file-id %}) + result)] + ;; two files: the repeated entry is imported once + (t/is (= 2 (count result))) + ;; one of the two cycle edges survives the import + (t/is (= 1 (count rels))) + (t/is (= 1 (count syncs))) + (t/is (= (set (map (juxt :file-id :library-file-id) syncs)) + (set (map (juxt :file-id :library-file-id) rels)))) + ;; no imported file lists itself among its transitive libraries + (doseq [file-id result] + (t/is (not (contains? (bfc/get-libraries th/*system* [file-id]) + file-id)))))))) + (t/deftest export-multi-file-includes-package-relations (let [profile (th/create-profile* 1) file (prepare-simple-file profile) diff --git a/backend/test/backend_tests/rpc_file_test.clj b/backend/test/backend_tests/rpc_file_test.clj index 593c769546..02eca2175e 100644 --- a/backend/test/backend_tests/rpc_file_test.clj +++ b/backend/test/backend_tests/rpc_file_test.clj @@ -2749,6 +2749,43 @@ (let [edata (-> out :error ex-data)] (t/is (= :circular-library-reference (:code edata))))))) +(t/deftest link-file-to-library-self-reference + (let [profile (th/create-profile* 1) + file (th/create-file* 1 {:profile-id (:id profile) + :project-id (:default-project-id profile)})] + (let [out (th/command! {::th/type :link-file-to-library + ::rpc/profile-id (:id profile) + :file-id (:id file) + :library-id (:id file)})] + (t/is (th/ex-info? (:error out))) + (t/is (th/ex-of-type? (:error out) :validation)) + (t/is (= :invalid-library (:code (ex-data (:error out)))))))) + +(t/deftest permissions-check-precedes-circular-library-check + ;; A caller without edition permission must get `:not-found`, not the + ;; circular error: the permission checks run before the cycle check, so + ;; the dependency structure of files the caller cannot read never leaks. + (let [profile1 (th/create-profile* 1) + profile2 (th/create-profile* 2) + file1 (th/create-file* 1 {:profile-id (:id profile1) + :project-id (:default-project-id profile1) + :is-shared true}) + file2 (th/create-file* 2 {:profile-id (:id profile1) + :project-id (:default-project-id profile1) + :is-shared true}) + file3 (th/create-file* 3 {:profile-id (:id profile1) + :project-id (:default-project-id profile1) + :is-shared true})] + ;; file3 -> file2 -> file1 makes linking file1 -> file3 circular + (th/link-file-to-library* {:file-id (:id file3) :library-id (:id file2)}) + (th/link-file-to-library* {:file-id (:id file2) :library-id (:id file1)}) + (let [out (th/command! {::th/type :link-file-to-library + ::rpc/profile-id (:id profile2) + :file-id (:id file1) + :library-id (:id file3)})] + (t/is (th/ex-info? (:error out))) + (t/is (th/ex-of-type? (:error out) :not-found))))) + (t/deftest get-file-etag-includes-deleted-at (let [profile-id (uuid/random) file1 {:modified-at (ct/now) From 54b1eb5d5f933388ebafcbee6fcc92774371efe8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?andr=C3=A9s=20gonz=C3=A1lez?= Date: Wed, 11 Nov 2026 14:43:11 +0100 Subject: [PATCH 10/29] :sparkles: Add shape count to the apply-tokens audit event (#12970) The apply-tokens event fires once per apply call, so a plugin or MCP script that loops over shapes sends one event per shape while the UI sends one per selection. Add a shape-count prop with the number of shapes the token was applied to, so analytics can compare origins in the same unit. Closes #12900 AI-assisted-by: claude-opus-5-5 --- frontend/src/app/main/data/workspace/tokens/application.cljs | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/frontend/src/app/main/data/workspace/tokens/application.cljs b/frontend/src/app/main/data/workspace/tokens/application.cljs index 3ab4427ce6..2f68e08a85 100644 --- a/frontend/src/app/main/data/workspace/tokens/application.cljs +++ b/frontend/src/app/main/data/workspace/tokens/application.cljs @@ -743,7 +743,8 @@ (-> {::ev/name "apply-tokens" :type type :applied-to attributes - :applied-to-variant any-variant?} + :applied-to-variant any-variant? + :shape-count (count shape-ids)} (merge (meta it))))) (dwu/start-undo-transaction undo-id) (dwsh/update-shapes shape-ids (fn [shape] From 3eebe92606ef635b5dc65cf7e5e6d75fb8972bf7 Mon Sep 17 00:00:00 2001 From: Pablo Alba Date: Wed, 11 Nov 2026 09:47:48 +0100 Subject: [PATCH 11/29] :bug: Keep the swap slot when duplicating a swapped nested copy --- common/src/app/common/logic/libraries.cljc | 6 +- common/src/app/common/types/container.cljc | 2 +- .../logic/comp_remove_swap_slots_test.cljc | 59 +++++++++++++++++++ .../common_tests/types/container_test.cljc | 39 ++++++++++++ 4 files changed, 104 insertions(+), 2 deletions(-) diff --git a/common/src/app/common/logic/libraries.cljc b/common/src/app/common/logic/libraries.cljc index fee7e718c2..acfbe6c2d1 100644 --- a/common/src/app/common/logic/libraries.cljc +++ b/common/src/app/common/logic/libraries.cljc @@ -3165,6 +3165,7 @@ is-component-main? (ctk/main-instance? obj) subinstance-head? (ctk/subinstance-head? obj) instance-root? (ctk/instance-root? obj) + parent-not-in-copy (not (ctk/in-component-copy? parent)) into-component? (and duplicating-component? (ctn/in-any-component? objects parent)) @@ -3174,6 +3175,8 @@ (ctn/get-nesting-level-delta objects obj parent)) new-shape-ref (ctf/advance-shape-ref nil page libraries obj level-delta {:include-deleted? true}) + + regenerate-component (fn [changes shape] (let [[_ changes] (generate-add-component-changes changes shape objects file-id (:id page) variant-props)] @@ -3188,7 +3191,8 @@ (cond-> (and (not instance-root?) subinstance-head? - remove-swap-slot?) + remove-swap-slot? + parent-not-in-copy) (ctk/remove-swap-slot)) (dissoc :shapes diff --git a/common/src/app/common/types/container.cljc b/common/src/app/common/types/container.cljc index 849467d7f4..7c61356cbc 100644 --- a/common/src/app/common/types/container.cljc +++ b/common/src/app/common/types/container.cljc @@ -196,7 +196,7 @@ ;; Calculate how many parent heads share in common the original ;; shape and the new parent. pairs (map vector orig-heads dest-heads) - common-count (count (take-while (fn [a b] (= a b)) pairs))] + common-count (count (take-while (fn [[a b]] (= a b)) pairs))] (- (count orig-heads) common-count))) diff --git a/common/test/common_tests/logic/comp_remove_swap_slots_test.cljc b/common/test/common_tests/logic/comp_remove_swap_slots_test.cljc index da766cd71d..bf62f42c72 100644 --- a/common/test/common_tests/logic/comp_remove_swap_slots_test.cljc +++ b/common/test/common_tests/logic/comp_remove_swap_slots_test.cljc @@ -567,3 +567,62 @@ ;; blue2' has not swap-id after move (t/is (some? blue2')) (t/is (nil? (ctk/get-swap-slot blue2'))))) + +(defn- setup-nested-swap-file + [] + ;; {:frame-icon-a} [:name Frame1] # [Component :icon-a] + ;; {:frame-icon-b} [:name Frame1] # [Component :icon-b] + ;; {:frame-card} [:name Frame1] # [Component :card] + ;; :frame-inner [:name Frame1] + ;; :icon-copy [:name Frame1] @--> :frame-icon-a + ;; :card-copy [:name Frame1] @--> :frame-card + ;; :inner-copy [:name Frame1] @--> :frame-inner + ;; :icon-in-card-copy [:name Frame1] @--> :frame-icon-a + ;; swapped to :icon-b, it becomes + ;; :icon-b-copy [:name Frame1, swap-slot] @--> :frame-icon-b + (-> (thf/sample-file :file1) + (tho/add-frame :frame-icon-a) + (thc/make-component :icon-a :frame-icon-a) + (tho/add-frame :frame-icon-b) + (thc/make-component :icon-b :frame-icon-b) + (tho/add-frame :frame-card) + (thc/make-component :card :frame-card) + (tho/add-frame :frame-inner :parent-label :frame-card) + (thc/instantiate-component :icon-a :icon-copy :parent-label :frame-inner) + (thc/instantiate-component :card :card-copy + :children-labels [:inner-copy :icon-in-card-copy]) + (thc/component-swap :icon-in-card-copy :icon-b :icon-b-copy))) + +(t/deftest test-keep-swap-slot-duplicating-swapped-nested-copy-inside-copy + (let [;; ==== Setup + file (setup-nested-swap-file) + + page (thf/current-page file) + icon-b-copy (ths/get-shape file :icon-b-copy) + + ;; ==== Action + changes (-> (pcb/empty-changes nil) + (cll/generate-duplicate-changes (:objects page) ;; objects + page ;; page + #{(:id icon-b-copy)} ;; ids + (gpt/point 0 0) ;; delta + {(:id file) file} ;; libraries + (:data file) ;; library-data + (:id file)) ;; file-id + (cll/generate-duplicate-changes-update-indices (:objects page) ;; objects + #{(:id icon-b-copy)})) ;; ids + + file' (thf/apply-changes file changes) + + ;; ==== Get + page' (thf/current-page file') + duplicated-icon-b' (find-duplicated-shape icon-b-copy page')] + + ;; ==== Check + + ;; icon-b-copy has swap-slot + (t/is (some? (ctk/get-swap-slot icon-b-copy))) + + ;; duplicated icon-b copy keeps its swap-slot + (t/is (some? duplicated-icon-b')) + (t/is (some? (ctk/get-swap-slot duplicated-icon-b'))))) diff --git a/common/test/common_tests/types/container_test.cljc b/common/test/common_tests/types/container_test.cljc index 9f7a308c8a..f545705d42 100644 --- a/common/test/common_tests/types/container_test.cljc +++ b/common/test/common_tests/types/container_test.cljc @@ -291,3 +291,42 @@ (t/testing "two ordinary outside shapes are both kept" (t/is (= [outside-id outside-2-id] (vec (ctc/ids-for-structure-op objects [outside-id outside-2-id]))))))) + +(t/deftest get-nesting-level-delta + (let [page-id uuid/zero + head-id (uuid/next) + frame-id (uuid/next) + sub-id (uuid/next) + head-2-id (uuid/next) + plain-id (uuid/next) + ;; copy head with a plain frame and a nested copy head inside + head (make-shape head-id + :parent-id page-id + :type :frame + :component-id (uuid/next) + :component-root true + :shape-ref (uuid/next)) + frame (make-shape frame-id :parent-id head-id :type :frame :shape-ref (uuid/next)) + sub (make-shape sub-id + :parent-id frame-id + :type :frame + :component-id (uuid/next) + :shape-ref (uuid/next)) + head-2 (make-shape head-2-id + :parent-id page-id + :type :frame + :component-id (uuid/next) + :component-root true + :shape-ref (uuid/next)) + plain (make-shape plain-id :parent-id page-id) + root (root-shape) + objects (objects-map root head frame sub head-2 plain)] + (t/testing "shape stays under the same copy head" + (t/is (= 0 (ctc/get-nesting-level-delta objects sub frame)))) + (t/testing "shape leaves its copy head to the page" + (t/is (= 1 (ctc/get-nesting-level-delta objects sub root)))) + (t/testing "shape moves to a different copy head" + (t/is (= 1 (ctc/get-nesting-level-delta objects sub head-2)))) + (t/testing "shape outside any copy head goes inside one" + (t/is (= 0 (ctc/get-nesting-level-delta objects plain frame)))))) + From bb672a37232113971d032d23b20f0d5d96872683 Mon Sep 17 00:00:00 2001 From: Pablo Alba Date: Wed, 11 Nov 2026 12:36:23 +0100 Subject: [PATCH 12/29] :bug: Block grid track duplication and paste in component copies --- .../app/main/data/workspace/shape_layout.cljs | 94 +++++----- .../app/main/ui/workspace/context_menu.cljs | 13 +- .../logic/copying_and_duplicating_test.cljs | 162 +++++++++++++++++- 3 files changed, 223 insertions(+), 46 deletions(-) diff --git a/frontend/src/app/main/data/workspace/shape_layout.cljs b/frontend/src/app/main/data/workspace/shape_layout.cljs index 68a34dad95..0a5c7b4cfe 100644 --- a/frontend/src/app/main/data/workspace/shape_layout.cljs +++ b/frontend/src/app/main/data/workspace/shape_layout.cljs @@ -436,6 +436,14 @@ (ptk/data-event :layout/update {:ids ids}) (dwu/commit-undo-transaction undo-id)))))) +(defn grid-in-copy? + "Check if the grid is a component copy or lives inside one. Copies + must keep the structure of their main, so tracks that add shapes are + not allowed there." + [objects grid] + (or (ctc/in-component-copy? grid) + (ctn/has-any-copy-parent? objects grid))) + (defn duplicate-layout-track [ids type index] (assert (#{:row :column} type)) @@ -443,53 +451,53 @@ (ptk/reify ::duplicate-layout-track ptk/WatchEvent (watch [it state _] - (let [file-id (:current-file-id state) - page (dsh/lookup-page state) - objects (:objects page) - libraries (dsh/lookup-libraries state) - library-data (dsh/lookup-file state file-id) - shape-id (first ids) - base-shape (get objects shape-id) + (let [objects (dsh/lookup-page-objects state) + base-shape (get objects (first ids))] + (when-not (grid-in-copy? objects base-shape) + (let [file-id (:current-file-id state) + page (dsh/lookup-page state) + libraries (dsh/lookup-libraries state) + library-data (dsh/lookup-file state file-id) - shapes-by-track - (if (= type :column) - (ctl/shapes-by-column base-shape index false) - (ctl/shapes-by-row base-shape index false)) + shapes-by-track + (if (= type :column) + (ctl/shapes-by-column base-shape index false) + (ctl/shapes-by-row base-shape index false)) - ;; Change to set in order to use auxiliary functions - selected (set shapes-by-track) + ;; Change to set in order to use auxiliary functions + selected (set shapes-by-track) - changes - (-> (pcb/empty-changes it) - (cll/generate-duplicate-changes objects page selected (gpt/point 0 0) libraries library-data file-id) - (cll/generate-duplicate-changes-update-indices objects selected)) + changes + (-> (pcb/empty-changes it) + (cll/generate-duplicate-changes objects page selected (gpt/point 0 0) libraries library-data file-id) + (cll/generate-duplicate-changes-update-indices objects selected)) - ;; Creates a map with shape-id => duplicated-shape-id - ids-map - (->> changes - :redo-changes - (filter #(= (:type %) :add-obj)) - (filter #(selected (:old-id %))) - (map #(vector (:old-id %) (get-in % [:obj :id]))) - (into {})) + ;; Creates a map with shape-id => duplicated-shape-id + ids-map + (->> changes + :redo-changes + (filter #(= (:type %) :add-obj)) + (filter #(selected (:old-id %))) + (map #(vector (:old-id %) (get-in % [:obj :id]))) + (into {})) - changes - (-> changes - (pcb/update-shapes - ids - (fn [shape objects] - ;; The duplication could have altered the grid so we restore the values, we'll calculate the good ones now - (let [shape (merge shape (select-keys base-shape [:layout-grid-cells :layout-grid-columns :layout-grid-rows]))] - (case type - :row (ctl/duplicate-row shape objects index ids-map) - :column (ctl/duplicate-column shape objects index ids-map)))) - {:with-objects? true})) + changes + (-> changes + (pcb/update-shapes + ids + (fn [shape objects] + ;; The duplication could have altered the grid so we restore the values, we'll calculate the good ones now + (let [shape (merge shape (select-keys base-shape [:layout-grid-cells :layout-grid-columns :layout-grid-rows]))] + (case type + :row (ctl/duplicate-row shape objects index ids-map) + :column (ctl/duplicate-column shape objects index ids-map)))) + {:with-objects? true})) - undo-id (js/Symbol)] - (rx/of (dwu/start-undo-transaction undo-id) - (dch/commit-changes changes) - (ptk/data-event :layout/update {:ids ids}) - (dwu/commit-undo-transaction undo-id)))))) + undo-id (js/Symbol)] + (rx/of (dwu/start-undo-transaction undo-id) + (dch/commit-changes changes) + (ptk/data-event :layout/update {:ids ids}) + (dwu/commit-undo-transaction undo-id)))))))) (defn reorder-layout-track [ids type from-index to-index move-content?] @@ -918,7 +926,9 @@ type (:type copied) undo-id (js/Symbol)] - (when (and (seq track-indices) (some? type)) + (when (and (seq track-indices) + (some? type) + (not (grid-in-copy? objects grid))) (let [shapes-by-track-fn (if (= type :row) ctl/shapes-by-row diff --git a/frontend/src/app/main/ui/workspace/context_menu.cljs b/frontend/src/app/main/ui/workspace/context_menu.cljs index 6a11dfd988..817bc040da 100644 --- a/frontend/src/app/main/ui/workspace/context_menu.cljs +++ b/frontend/src/app/main/ui/workspace/context_menu.cljs @@ -873,6 +873,10 @@ {::mf/private true} [{:keys [mdata]}] (let [{:keys [type index grid-id]} mdata + + objects (deref refs/workspace-page-objects) + in-copy? (dwsl/grid-in-copy? objects (get objects grid-id)) + do-delete-track (mf/use-fn (mf/deps grid-id type index) @@ -905,14 +909,14 @@ (if (= type :column) [:* - [:> menu-entry* {:title (tr "workspace.context-menu.grid-track.column.duplicate") :on-click do-duplicate-track}] + [:> menu-entry* {:title (tr "workspace.context-menu.grid-track.column.duplicate") :on-click do-duplicate-track :disabled in-copy?}] [:> menu-entry* {:title (tr "workspace.context-menu.grid-track.column.add-before") :on-click do-add-track-before}] [:> menu-entry* {:title (tr "workspace.context-menu.grid-track.column.add-after") :on-click do-add-track-after}] [:> menu-entry* {:title (tr "workspace.context-menu.grid-track.column.delete") :on-click do-delete-track}] [:> menu-entry* {:title (tr "workspace.context-menu.grid-track.column.delete-shapes") :on-click do-delete-track-shapes}]] [:* - [:> menu-entry* {:title (tr "workspace.context-menu.grid-track.row.duplicate") :on-click do-duplicate-track}] + [:> menu-entry* {:title (tr "workspace.context-menu.grid-track.row.duplicate") :on-click do-duplicate-track :disabled in-copy?}] [:> menu-entry* {:title (tr "workspace.context-menu.grid-track.row.add-before") :on-click do-add-track-before}] [:> menu-entry* {:title (tr "workspace.context-menu.grid-track.row.add-after") :on-click do-add-track-after}] [:> menu-entry* {:title (tr "workspace.context-menu.grid-track.row.delete") :on-click do-delete-track}] @@ -949,6 +953,9 @@ grid-edition (mf/deref grid-edition-ref) has-copied-tracks? (some? (:copied-tracks grid-edition)) + objects (deref refs/workspace-page-objects) + in-copy? (dwsl/grid-in-copy? objects (get objects grid-id)) + do-merge-cells (mf/use-fn (mf/deps grid-id cells) @@ -1002,7 +1009,7 @@ [:> menu-entry* {:title (tr "workspace.context-menu.grid-cells.paste-tracks") :on-click do-paste-tracks - :disabled (not has-copied-tracks?)}]])) + :disabled (or (not has-copied-tracks?) in-copy?)}]])) (def guide-color-presets diff --git a/frontend/test/frontend_tests/logic/copying_and_duplicating_test.cljs b/frontend/test/frontend_tests/logic/copying_and_duplicating_test.cljs index 7939b903df..b096a96417 100644 --- a/frontend/test/frontend_tests/logic/copying_and_duplicating_test.cljs +++ b/frontend/test/frontend_tests/logic/copying_and_duplicating_test.cljs @@ -12,16 +12,19 @@ [app.common.test-helpers.ids-map :as cthi] [app.common.test-helpers.shapes :as cths] [app.common.types.component :as ctk] + [app.common.types.shape.layout :as ctl] [app.common.uuid :as uuid] [app.main.data.workspace :as dw] [app.main.data.workspace.colors :as dc] [app.main.data.workspace.libraries :as dwl] [app.main.data.workspace.pages :as dwp] [app.main.data.workspace.selection :as dws] + [app.main.data.workspace.shape-layout :as dwsl] [cljs.test :as t :include-macros true] [frontend-tests.helpers.mock :as mock] [frontend-tests.helpers.pages :as thp] - [frontend-tests.helpers.state :as ths])) + [frontend-tests.helpers.state :as ths] + [potok.v2.core :as ptk])) (t/use-fixtures :each {:before thp/reset-idmap!}) @@ -653,3 +656,160 @@ "Pasted text shapes have nil position-data")))))) done))) + +;; Grid tracks in component copies + +(defn- add-grid-with-child + "Adds a grid board with one rect child and assigns the child to a cell." + [file grid-label rect-label & {:keys [parent-label params]}] + (let [file (-> file + (ctho/add-frame grid-label + (merge {:name "grid" + :parent-label parent-label + :layout :grid + :layout-grid-dir :row + :layout-grid-columns [] + :layout-grid-rows [] + :layout-grid-cells {}} + params)) + (cths/add-sample-shape rect-label :parent-label grid-label)) + grid (->> (cths/get-shape file grid-label) + (#(ctl/assign-cells % (:objects (cthf/current-page file)))))] + (reduce (fn [file attr] (cths/update-shape file grid-label attr (get grid attr))) + file + [:layout-grid-columns :layout-grid-rows :layout-grid-cells]))) + +(defn- grid-root-file + "Main component whose root board is the grid, and a copy of it." + [] + (-> (cthf/sample-file :file1 :page-label :page-1) + (add-grid-with-child :grid-main :rect-main) + (cthc/make-component :comp :grid-main) + (cthc/instantiate-component :comp :grid-copy :children-labels [:rect-copy]))) + +(defn- grid-nested-file + "Main component with a wrapper board that contains the grid, and a copy." + [] + (-> (cthf/sample-file :file1 :page-label :page-1) + (ctho/add-frame :wrapper-main :name "wrapper") + (add-grid-with-child :grid-main :rect-main :parent-label :wrapper-main) + (cthc/make-component :comp :wrapper-main) + (cthc/instantiate-component :comp :wrapper-copy :children-labels [:grid-copy :rect-copy]))) + +(defn- grid-user-added-in-copy-file + "Copy of a wrapper board, with a plain grid (no :shape-ref) added inside + the copy. Only its parent is a copy." + [] + (let [file (-> (grid-nested-file) + (add-grid-with-child :grid-user :rect-user :parent-label :wrapper-copy))] + (assert (nil? (:shape-ref (cths/get-shape file :grid-user)))) + file)) + +(defn- grid-variant-file + "Variant main whose root board is the grid. The root has :variant-id and + no :shape-ref." + [] + (let [file (-> (cthf/sample-file :file1 :page-label :page-1) + (cths/add-sample-shape :variant :type :frame :is-variant-container true)) + variant-id (cthi/id :variant)] + (-> file + (add-grid-with-child :grid-main :rect-main + :parent-label :variant + :params {:variant-id variant-id :variant-name "Value1"}) + (cthc/make-component :comp :grid-main) + (cthc/update-component :comp {:variant-id variant-id + :variant-properties [{:name "Property 1" :value "Value1"}]})))) + +(defn- copied-tracks-event + [grid-id] + (ptk/reify ::set-copied-tracks + ptk/UpdateEvent + (update [_ state] + (assoc-in state [:workspace-grid-edition grid-id :copied-tracks] + {:track-indices [0] :type :column :grid-id grid-id})))) + +(defn- run-grid-track-events + [file make-events check-fn] + (t/async done + (mock/with-mocks {uuid/next cthi/next-uuid} + (fn [done'] + (let [store (ths/setup-store file) + events (make-events file)] + (ths/run-store + store done' events + (fn [new-state] + (check-fn (cthf/current-page file) + (ths/get-file-from-state new-state)))))) + done))) + +(defn- duplicate-events + [grid-label] + (fn [file] + [(dwsl/duplicate-layout-track [(:id (cths/get-shape file grid-label))] :column 0)])) + +(defn- paste-events + [grid-label] + (fn [file] + (let [grid-id (:id (cths/get-shape file grid-label))] + [(copied-tracks-event grid-id) + (dwsl/paste-grid-tracks grid-id)]))) + +(defn- objects-of + [file] + (:objects (cthf/current-page file))) + +(defn- column-count + [file label] + (count (:layout-grid-columns (cths/get-shape file label)))) + +(defn- check-blocked + [file make-events grid-label] + (run-grid-track-events + file make-events + (fn [page file'] + (t/is (= (:objects page) (objects-of file'))) + (t/is (= (column-count file grid-label) (column-count file' grid-label)))))) + +(defn- check-works + [file make-events grid-label] + (run-grid-track-events + file make-events + (fn [page file'] + (t/is (> (count (objects-of file')) (count (:objects page)))) + (t/is (> (column-count file' grid-label) (column-count file grid-label)))))) + +(t/deftest duplicate-track-blocked-in-copy-root-grid + (check-blocked (grid-root-file) (duplicate-events :grid-copy) :grid-copy)) + +(t/deftest paste-tracks-blocked-in-copy-root-grid + (check-blocked (grid-root-file) (paste-events :grid-copy) :grid-copy)) + +(t/deftest duplicate-track-blocked-in-nested-copy-grid + (check-blocked (grid-nested-file) (duplicate-events :grid-copy) :grid-copy)) + +(t/deftest paste-tracks-blocked-in-nested-copy-grid + (check-blocked (grid-nested-file) (paste-events :grid-copy) :grid-copy)) + +(t/deftest duplicate-track-blocked-in-user-added-grid-inside-copy + (check-blocked (grid-user-added-in-copy-file) (duplicate-events :grid-user) :grid-user)) + +(t/deftest paste-tracks-blocked-in-user-added-grid-inside-copy + (check-blocked (grid-user-added-in-copy-file) (paste-events :grid-user) :grid-user)) + +(t/deftest duplicate-track-works-in-main-grid + (check-works (grid-root-file) (duplicate-events :grid-main) :grid-main)) + +(t/deftest paste-tracks-works-in-main-grid + (check-works (grid-root-file) (paste-events :grid-main) :grid-main)) + +(t/deftest duplicate-track-works-in-main-nested-grid + (check-works (grid-nested-file) (duplicate-events :grid-main) :grid-main)) + +(t/deftest paste-tracks-works-in-main-nested-grid + (check-works (grid-nested-file) (paste-events :grid-main) :grid-main)) + +(t/deftest duplicate-track-works-in-variant-main-grid + (check-works (grid-variant-file) (duplicate-events :grid-main) :grid-main)) + +(t/deftest paste-tracks-works-in-variant-main-grid + (check-works (grid-variant-file) (paste-events :grid-main) :grid-main)) From ca616a39ed824683e1500418d6aa3fa33d783091 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?andr=C3=A9s=20gonz=C3=A1lez?= Date: Wed, 11 Nov 2026 16:02:29 +0100 Subject: [PATCH 13/29] :bug: Send the plugin origin with penpotMgmt dashboard actions (#12971) Dashboard actions made by a plugin or the MCP through penpotMgmt reached the backend audit log without event origin, so they looked like actions made by hand in the dashboard. The backend reads the origin from the x-event-origin header, which rp/cmd! only sends when the params carry ::ev/origin in their meta. Tag the params of every penpotMgmt mutation with the same meta that se/add-event sets (mcp or plugin). The rename, share and team update events now keep that meta on the RPC params, so setters send it too, and set-file-variants-shared takes the caller origin instead of a fixed dashboard one. Closes #12901 AI-assisted-by: claude-opus-5-5 --- frontend/src/app/main/data/dashboard.cljs | 13 +++-- frontend/src/app/plugins/management.cljs | 56 +++++++++++-------- .../plugins/management_test.cljs | 33 +++++++++-- 3 files changed, 69 insertions(+), 33 deletions(-) diff --git a/frontend/src/app/main/data/dashboard.cljs b/frontend/src/app/main/data/dashboard.cljs index ab2ace94da..c46b56806c 100644 --- a/frontend/src/app/main/data/dashboard.cljs +++ b/frontend/src/app/main/data/dashboard.cljs @@ -373,7 +373,7 @@ ptk/WatchEvent (watch [_ _ _] - (let [params {:id id :name name}] + (let [params (with-meta {:id id :name name} (meta params))] (->> (rp/cmd! :rename-project params) (rx/ignore)))))) @@ -435,7 +435,7 @@ ptk/WatchEvent (watch [_ _ _] - (let [params (select-keys params [:id :name])] + (let [params (with-meta (select-keys params [:id :name]) (meta params))] (->> (rp/cmd! :rename-file params) (rx/ignore)))))) @@ -460,15 +460,18 @@ ptk/WatchEvent (watch [_ _ _] - (let [params {:id id :is-shared is-shared}] + (let [mdata (meta params) + origin (::ev/origin mdata "dashboard")] (rx/concat - (->> (rp/cmd! :set-file-shared params) + (->> (rp/cmd! :set-file-shared (with-meta {:id id :is-shared is-shared} mdata)) (rx/ignore)) (when is-shared (->> (rp/cmd! :get-file-summary {:id id}) (rx/map (fn [summary] (when (-> summary :variants :count pos?) - (ev/event {::ev/name "set-file-variants-shared" ::ev/origin "dashboard"}))))))))))) + (ev/event (merge {::ev/name "set-file-variants-shared" + ::ev/origin origin} + (select-keys mdata [::ev/context]))))))))))))) (defn set-file-thumbnail [file-id thumbnail-id] diff --git a/frontend/src/app/plugins/management.cljs b/frontend/src/app/plugins/management.cljs index 91671991f8..fe1662a11f 100644 --- a/frontend/src/app/plugins/management.cljs +++ b/frontend/src/app/plugins/management.cljs @@ -46,6 +46,14 @@ (update-fn state) state)))) +;; Dashboard events are logged by the backend, which reads the origin from +;; the `x-event-origin` header that `rp/cmd!` builds from the params meta. +(defn- with-origin + "Tags RPC `params` with the plugin or MCP origin, as `se/add-event` does + for frontend events." + [plugin-id params] + (se/add-event params plugin-id)) + (defn- refresh-restored-project [team-id project-id] (if (dashboard-of-team? @st/state team-id) @@ -65,8 +73,9 @@ (defn- restore-deleted-files "Uses the dashboard's Trash operation and waits for its final result." - [team-id ids] - (->> (rp/cmd! ::sse/restore-deleted-team-files {:team-id team-id :ids ids}) + [plugin-id team-id ids] + (->> (rp/cmd! ::sse/restore-deleted-team-files + (with-origin plugin-id {:team-id team-id :ids ids})) (rx/filter sse/end-of-stream?) (rx/take 1) (rx/map sse/get-payload) @@ -252,7 +261,7 @@ [plugin-id team-id options] (if-let [name (valid-name (obj/get options "name"))] (request plugin-id "manage:projects" - #(->> (rp/cmd! :create-project {:team-id team-id :name name}) + #(->> (rp/cmd! :create-project (with-origin plugin-id {:team-id team-id :name name})) (rx/map (fn [project] (assoc project :count 0))) (rx/tap (fn [project] (update-dashboard team-id @@ -309,7 +318,7 @@ :else (set-property plugin-id data :name name - (dtm/update-team {:id (:id @data) :name name})))))} + (dtm/update-team (with-origin plugin-id {:id (:id @data) :name name}))))))} :listProjects (fn [] (list-projects plugin-id (:id @data))) @@ -355,7 +364,7 @@ :else (set-property plugin-id data :name name - (dd/rename-project {:id (:id @data) :name name})))))} + (dd/rename-project (with-origin plugin-id {:id (:id @data) :name name}))))))} :pinned {:get #(boolean (:is-pinned @data)) @@ -370,9 +379,10 @@ :else (set-property plugin-id data :is-pinned value - (set-project-pin {:id (:id @data) - :team-id (:team-id @data) - :is-pinned value}))))} + (set-project-pin (with-origin plugin-id + {:id (:id @data) + :team-id (:team-id @data) + :is-pinned value})))))} :listFiles (fn [] @@ -395,7 +405,7 @@ (let [ids (into #{} (comp (filter (fn [file] (= id (:project-id file)))) (map :id)) files)] (if (seq ids) - (restore-deleted-files team-id ids) + (restore-deleted-files plugin-id team-id ids) (rx/throw (js/Error. "Cannot restore a project without recoverable files")))))) (rx/tap (fn [_] (swap! data dissoc :deleted-at))) (rx/mapcat (fn [_] (refresh-restored-project team-id id))))))) @@ -406,7 +416,7 @@ (request plugin-id "content:write" #(let [{:keys [id team-id]} @data features (set/difference (:features @st/state #{}) cfeat/frontend-only-features)] - (->> (rp/cmd! :create-file {:project-id id :name name :features features}) + (->> (rp/cmd! :create-file (with-origin plugin-id {:project-id id :name name :features features})) (rx/map (fn [file] (-> (dissoc file :data) (assoc :team-id team-id)))) (rx/tap (fn [file] (update-dashboard team-id (fn [state] (ptk/update (dd/file-created file) state))))) @@ -420,8 +430,9 @@ (js/Promise.reject (js/Error. "Expected a name with 1 to 250 characters")) (request plugin-id "manage:projects" #(let [{:keys [id team-id]} @data] - (->> (rp/cmd! :duplicate-project (cond-> {:project-id id} - (some? name) (assoc :name (valid-name name)))) + (->> (rp/cmd! :duplicate-project (with-origin plugin-id + (cond-> {:project-id id} + (some? name) (assoc :name (valid-name name))))) (rx/mapcat dd/project-with-file-count) (rx/tap (fn [project] (update-dashboard team-id (fn [state] (ptk/update (dd/project-duplicated project) state))))) @@ -432,7 +443,7 @@ (if-let [team-id (uuid/parse* (obj/get team "id"))] (request plugin-id "manage:teams" #(let [{:keys [id] :as project} @data] - (->> (rp/cmd! :move-project {:project-id id :team-id team-id}) + (->> (rp/cmd! :move-project (with-origin plugin-id {:project-id id :team-id team-id})) (rx/tap (fn [_] (swap! data assoc :team-id team-id) (update-dashboard (:team-id project) (fn [state] (update state :projects dissoc id))))) @@ -443,7 +454,7 @@ (fn [] (request plugin-id "manage:delete" #(let [{:keys [id team-id]} @data] - (->> (rp/cmd! :delete-project {:id id}) + (->> (rp/cmd! :delete-project (with-origin plugin-id {:id id})) (rx/tap (fn [_] (update-dashboard team-id (fn [state] (update state :projects dissoc id))))) (rx/map (constantly nil))))))))) @@ -480,7 +491,7 @@ :else (set-property plugin-id data :name name - (dd/rename-file {:id (:id @data) :name name})))))} + (dd/rename-file (with-origin plugin-id {:id (:id @data) :name name}))))))} :shared {:get #(boolean (:is-shared @data)) @@ -495,7 +506,7 @@ :else (set-property plugin-id data :is-shared value - (dd/set-file-shared {:id (:id @data) :is-shared value}))))} + (dd/set-file-shared (with-origin plugin-id {:id (:id @data) :is-shared value})))))} :open (fn [] @@ -506,7 +517,7 @@ :restore (fn [] (request plugin-id "content:write" - #(->> (restore-deleted-files (:team-id @data) #{(:id @data)}) + #(->> (restore-deleted-files plugin-id (:team-id @data) #{(:id @data)}) (rx/tap (fn [_] (swap! data dissoc :deleted-at :will-be-deleted-at))) (rx/mapcat (fn [_] (refresh-restored-project (:team-id @data) (:project-id @data))))))) @@ -519,7 +530,7 @@ (->> (rp/cmd! :get-project {:id project-id}) (rx/mapcat (fn [target] (if (= team-id (:team-id target)) - (rp/cmd! :move-files {:ids #{id} :project-id project-id}) + (rp/cmd! :move-files (with-origin plugin-id {:ids #{id} :project-id project-id})) (rx/throw (js/Error. "Cannot move a file to a project in another team"))))) (rx/tap (fn [_] (update-dashboard team-id (fn [state] (ptk/update (dd/move-files {:ids #{id} :project-id project-id}) state))) @@ -534,8 +545,9 @@ (js/Promise.reject (js/Error. "Expected a name with 1 to 250 characters")) (request plugin-id "content:write" #(let [{:keys [id team-id]} @data] - (->> (rp/cmd! :duplicate-file (cond-> {:file-id id} - (some? name) (assoc :name (valid-name name)))) + (->> (rp/cmd! :duplicate-file (with-origin plugin-id + (cond-> {:file-id id} + (some? name) (assoc :name (valid-name name))))) (rx/map (fn [file] (-> (dissoc file :data) (assoc :team-id team-id)))) (rx/tap (fn [file] (update-dashboard team-id (fn [state] (ptk/update (dd/file-created file) state))))) @@ -545,7 +557,7 @@ (fn [] (request plugin-id "manage:delete" #(let [{:keys [id project-id team-id]} @data] - (->> (rp/cmd! :delete-file {:id id}) + (->> (rp/cmd! :delete-file (with-origin plugin-id {:id id})) (rx/tap (fn [_] (update-dashboard team-id (fn [state] @@ -576,7 +588,7 @@ (fn [options] (if-let [name (valid-name (obj/get options "name"))] (request plugin-id "manage:teams" - #(->> (rp/cmd! :create-team {:name name :features features/global-enabled-features}) + #(->> (rp/cmd! :create-team (with-origin plugin-id {:name name :features features/global-enabled-features})) (rx/tap (fn [_] (st/emit! (dtm/fetch-teams)))) (rx/map (partial team-proxy plugin-id)))) (js/Promise.reject (js/Error. "Expected a name with 1 to 250 characters")))) diff --git a/frontend/test/frontend_tests/plugins/management_test.cljs b/frontend/test/frontend_tests/plugins/management_test.cljs index 15d04f8f7f..e9637d4d88 100644 --- a/frontend/test/frontend_tests/plugins/management_test.cljs +++ b/frontend/test/frontend_tests/plugins/management_test.cljs @@ -8,6 +8,7 @@ (:require [app.common.uuid :as uuid] [app.main.data.dashboard :as dd] + [app.main.data.event :as ev] [app.main.data.team :as dtm] [app.main.repo :as rp] [app.main.store :as st] @@ -80,6 +81,15 @@ ([command] (swap! requests conj [command]) (respond command nil)) ([command params] (swap! requests conj [command params]) (respond command params))))) +(defn- origins + "Maps each recorded command that is not a read to the origin of its params." + [requests] + (into {} + (keep (fn [[command params]] + (when-not (re-find #"^get-" (name command)) + [command (::ev/origin (meta params))]))) + requests)) + (defn- emit-stub "Mock `st/emit!` applying update functions to `state` and recording events." [state events] @@ -231,18 +241,20 @@ (props team* ["id" "name" "isDefault"]))) (t/is (= [[:create-team "Brand"]] (mapv (fn [[command params]] [command (:name params)]) @requests))) + (t/is (= {:create-team "plugin"} (origins @requests))) (t/is (= 1 (count (filter #(= ::dtm/fetch-teams (ptk/type %)) @events)))) (t/is (= "Expected a name with 1 to 250 characters" (await (message (.createTeam (management/create-context plugin-id) #js {:name ""})))))))))) (t/deftest ^:async setters-update-values-after-the-backend-succeeds - (let [state (atom (throwing dashboard)) - events (atom [])] + (let [state (atom (throwing dashboard)) + events (atom []) + requests (atom [])] (await (mock/with-mocks* {st/state state st/emit! (emit-stub state events) - rp/cmd! (mock/stub (fn [& _] (->> (rx/of {}) (rx/observe-on :async))))} + rp/cmd! (cmd-stub requests (fn [& _] (->> (rx/of {}) (rx/observe-on :async))))} (let [team* (management/team-proxy plugin-id team) project* (management/project-proxy plugin-id project) file* (management/file-proxy plugin-id file)] @@ -260,7 +272,10 @@ (t/is (true? (obj/get project* "pinned"))) (t/is (= "Login" (obj/get file* "name"))) (t/is (true? (obj/get file* "shared"))) - (t/is (= 5 (count @events)))))))) + (t/is (= 5 (count @events))) + (t/is (= {:update-team "plugin" :rename-project "plugin" :update-project-pin "plugin" + :rename-file "plugin" :set-file-shared "plugin"} + (origins @requests)))))))) (t/deftest ^:async stale-object-setter-errors-stay-out-of-the-global-handler (doseq [[constructor property value original] @@ -340,6 +355,7 @@ (t/is (= {"name" "Login" "teamId" (str team-id)} (props file* ["name" "teamId"]))) (t/is (= 2 (obj/get parent "fileCount"))) (t/is (= [:create-project {:team-id team-id :name "Icons"}] (first @requests))) + (t/is (= {:create-project "plugin" :create-file "plugin"} (origins @requests))) (if (= initial dashboard) (do (t/is (contains? (:projects @state) (:id created-project))) (t/is (contains? (:files @state) (:id created-file)))) @@ -383,7 +399,10 @@ [:move-project {:project-id other-id :team-id other-id}] [:delete-file {:id file-id}] [:delete-project {:id other-id}]] - @requests))))))) + @requests)) + (t/is (= {:duplicate-file "plugin" :duplicate-project "plugin" :move-files "plugin" + :move-project "plugin" :delete-file "plugin" :delete-project "plugin"} + (origins @requests)))))))) (t/deftest ^:async methods-reject-invalid-input-and-missing-permissions (let [requests (atom [])] @@ -475,7 +494,8 @@ (doseq [proxy [(management/project-proxy r/mcp-plugin-id project) (management/file-proxy r/mcp-plugin-id file)]] (t/is (nil? (await (.remove proxy))))) - (t/is (= [[:delete-project {:id project-id}] [:delete-file {:id file-id}]] @requests)))))) + (t/is (= [[:delete-project {:id project-id}] [:delete-file {:id file-id}]] @requests)) + (t/is (= {:delete-project "mcp" :delete-file "mcp"} (origins @requests))))))) (t/deftest ^:async deleted-objects-can-be-listed-and-restored (let [state (atom ready) @@ -532,6 +552,7 @@ rp/cmd! (mock/stub (fn [command params] (t/is (= ::sse/restore-deleted-team-files command)) (t/is (= {:team-id team-id :ids #{file-id}} params)) + (t/is (= "plugin" (::ev/origin (meta params)))) (rx/observe-on :async response)))} (let [promise (.then (.restore (management/file-proxy plugin-id file)) #(reset! completed? true))] From 0d05cfaf5e436b9d54617e811481ff491698941d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?andr=C3=A9s=20gonz=C3=A1lez?= Date: Wed, 11 Nov 2026 16:18:15 +0100 Subject: [PATCH 14/29] :sparkles: Add the file to plugin and MCP audit events (#12972) * :sparkles: Add the file and MCP session to plugin audit events Events sent by plugins and the MCP plugin only carried their origin, so the multi-file and dashboard MCP workflows could not be measured, nor duplicated events from several open tabs spotted. The context of se/event and se/add-event now carries the file-id of the file the action ran in (absent on the dashboard) and, for the MCP, the mcp-session-id of the tab connection that ran it. Closes #12902 AI-assisted-by: claude-opus-5-5 * :recycle: Drop the MCP session id from plugin audit events Every audit event already carries the session-id of the tab in its context, and the MCP session id is only a hash of it, so it adds nothing to group events by tab. AI-assisted-by: claude-opus-5-5 * :bug: Use a reified event in the system events test ptk/data-event does not implement IWithMeta, so add-event failed on it. Plugin actions emit reified events, which carry metadata. AI-assisted-by: claude-opus-5-5 --- frontend/src/app/plugins/system_events.cljs | 48 +++++++++-------- .../plugins/system_events_test.cljs | 52 +++++++++++++++++++ frontend/test/frontend_tests/runner.cljs | 2 + 3 files changed, 80 insertions(+), 22 deletions(-) create mode 100644 frontend/test/frontend_tests/plugins/system_events_test.cljs diff --git a/frontend/src/app/plugins/system_events.cljs b/frontend/src/app/plugins/system_events.cljs index 716852d152..99d163e768 100644 --- a/frontend/src/app/plugins/system_events.cljs +++ b/frontend/src/app/plugins/system_events.cljs @@ -10,31 +10,35 @@ [app.main.store :as st] [app.plugins.register :as r])) +(defn- get-origin + [plugin-id] + (if (= plugin-id r/mcp-plugin-id) "mcp" "plugin")) + +(defn- get-context + "Returns the audit context of an action run by a plugin: the file it ran + in (none on the dashboard) and, for plugins other than the MCP, the + plugin name and URL. Events drop the nil values." + [state plugin-id] + (let [file-id (:current-file-id state)] + (if (= plugin-id r/mcp-plugin-id) + {:file-id file-id} + (let [plugin-data (r/get-plugin-data state plugin-id)] + {:file-id file-id + :plugin-name (:name plugin-data) + :plugin-url (:url plugin-data)})))) + ;; Formats an event from the plugin system (defn event [plugin-id name & {:as props}] - (if (= plugin-id r/mcp-plugin-id) - (-> props - (assoc ::ev/name name) - (assoc ::ev/origin "mcp") - (ev/event)) - - (let [plugin-data (r/get-plugin-data @st/state plugin-id)] - (-> props - (assoc ::ev/name name) - (assoc ::ev/origin "plugin") - (assoc ::ev/context - {:plugin-name (:name plugin-data) - :plugin-url (:url plugin-data)}) - (ev/event))))) + (-> props + (assoc ::ev/name name) + (assoc ::ev/origin (get-origin plugin-id)) + (assoc ::ev/context (get-context @st/state plugin-id)) + (ev/event))) (defn add-event [event plugin-id] - (let [plugin-data (r/get-plugin-data @st/state plugin-id)] - (with-meta - event - (if (= plugin-id r/mcp-plugin-id) - {::ev/origin "mcp"} - {::ev/origin "plugin" - ::ev/context {:plugin-name (:name plugin-data) - :plugin-url (:url plugin-data)}})))) + (with-meta + event + {::ev/origin (get-origin plugin-id) + ::ev/context (get-context @st/state plugin-id)})) diff --git a/frontend/test/frontend_tests/plugins/system_events_test.cljs b/frontend/test/frontend_tests/plugins/system_events_test.cljs new file mode 100644 index 0000000000..302a566f9c --- /dev/null +++ b/frontend/test/frontend_tests/plugins/system_events_test.cljs @@ -0,0 +1,52 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns frontend-tests.plugins.system-events-test + (:require + [app.main.data.event :as ev] + [app.main.store :as st] + [app.plugins.register :as r] + [app.plugins.system-events :as se] + [cljs.test :as t :include-macros true] + [potok.v2.core :as ptk])) + +(def ^:private plugin-id "00000000-0000-0000-0000-000000000000") + +(defn- set-state + [file-id] + (set! st/state + (atom {:current-file-id file-id + :profile {:props {:plugins {:data {plugin-id {:name "Plugin" + :url "https://plugin.test"}}}}}}))) + +(t/deftest mcp-event-carries-file + (let [file-id (random-uuid)] + (set-state file-id) + (let [data (deref (se/event r/mcp-plugin-id "create-version" :file-id file-id))] + (t/is (= "mcp" (::ev/origin data))) + (t/is (= {:file-id file-id} (::ev/context data))) + (t/is (= file-id (:file-id data)))))) + +(t/deftest plugin-event-carries-file-and-plugin + (let [file-id (random-uuid)] + (set-state file-id) + (t/is (= {:file-id file-id + :plugin-name "Plugin" + :plugin-url "https://plugin.test"} + (::ev/context (deref (se/event plugin-id "add-flow"))))))) + +(t/deftest mcp-add-event-carries-file + (let [file-id (random-uuid)] + (set-state file-id) + (t/is (= {::ev/origin "mcp" + ::ev/context {:file-id file-id}} + (meta (se/add-event (ptk/reify ::test) r/mcp-plugin-id)))))) + +(t/deftest event-has-no-file-on-the-dashboard + (set-state nil) + (t/is (nil? (-> (deref (se/event r/mcp-plugin-id "create-project")) + ::ev/context + :file-id)))) diff --git a/frontend/test/frontend_tests/runner.cljs b/frontend/test/frontend_tests/runner.cljs index 135797b7f5..b0f699a220 100644 --- a/frontend/test/frontend_tests/runner.cljs +++ b/frontend/test/frontend_tests/runner.cljs @@ -101,6 +101,7 @@ [frontend-tests.plugins.read-only-test] [frontend-tests.plugins.register-test] [frontend-tests.plugins.shape-bugfixes-test] + [frontend-tests.plugins.system-events-test] [frontend-tests.plugins.text-test] [frontend-tests.plugins.tokens-test] [frontend-tests.plugins.user-test] @@ -301,6 +302,7 @@ 'frontend-tests.plugins.read-only-test 'frontend-tests.plugins.register-test 'frontend-tests.plugins.shape-bugfixes-test + 'frontend-tests.plugins.system-events-test 'frontend-tests.plugins.text-test 'frontend-tests.plugins.tokens-test 'frontend-tests.plugins.user-test From 34f0d12b9928436be11e7b9245e80f8f75a30f4f Mon Sep 17 00:00:00 2001 From: Pablo Alba Date: Wed, 11 Nov 2026 17:09:33 +0100 Subject: [PATCH 15/29] :bug: Ignore float noise when syncing moved library components (#12976) --- .../common/decimals-and-coordinates.md | 5 + common/src/app/common/logic/libraries.cljc | 42 ++- common/src/app/common/math.cljc | 13 + common/src/app/common/types/path/segment.cljc | 14 + .../common_tests/logic/comp_sync_test.cljc | 325 ++++++++++++++++++ common/test/common_tests/math_test.cljc | 18 + .../common_tests/types/path_data_test.cljc | 27 ++ 7 files changed, 442 insertions(+), 2 deletions(-) diff --git a/.serena/memories/common/decimals-and-coordinates.md b/.serena/memories/common/decimals-and-coordinates.md index 0a4e8992ad..1cc966f452 100644 --- a/.serena/memories/common/decimals-and-coordinates.md +++ b/.serena/memories/common/decimals-and-coordinates.md @@ -52,3 +52,8 @@ its own drift. Comparing `:selrect.width` from shape A to `:selrect.width` from shape B is comparing two values that "semantically" should be equal but were computed via different operation chains — exact equality will often be false. + +## Library sync comparison + +- `update-attrs` (`app.common.logic.libraries`) compares main vs copy attrs exactly first, then with tolerance for: `drifting-geometry-attrs` (`:x :y :width :height :selrect :points`, via `gsh/close-attrs?`); path and bool `:content` (`segment/content-close?`; bool shapes keep their path in `:content`); and text `:position-data` (`position-data-close?` over `mth/close-map?`: every numeric field of every entry within `mth/close?`, other fields such as fills, fonts and text exact). All other attrs (`:opacity`, radii, `:transform`, `:rotation`, text `:content`) stay exact: tolerance there hides real changes. +- `close-attrs?` pairs lists with `d/zip`, which truncates to the shorter list and treats nil vs vector as equal for `:points`/`:position-data`. Do not use it for variable-length data. That is why `:position-data` has its own helper, and why `attr-equal?` checks the `:points` count before calling it. diff --git a/common/src/app/common/logic/libraries.cljc b/common/src/app/common/logic/libraries.cljc index acfbe6c2d1..1b928d407b 100644 --- a/common/src/app/common/logic/libraries.cljc +++ b/common/src/app/common/logic/libraries.cljc @@ -58,6 +58,44 @@ ;; We don't do automatic update of the `layout-grid-cells` property. (remove #(= :layout-grid-cells %)))) +;; Attrs whose value drifts by float noise when a shape is moved; they +;; are compared with tolerance during sync. The rest compare exactly. +(def ^:private drifting-geometry-attrs + #{:x :y :width :height :selrect :points}) + +(defn- position-data-close? + "Compares text position-data entry by entry: numbers with tolerance, + every other field (fills, fonts, text) exactly." + [a b] + (let [a (vec a) + b (vec b)] + (and (= (count a) (count b)) + (every? true? (map mth/close-map? a b))))) + +(defn- attr-equal? + [attr origin-shape dest-shape] + (let [origin-val (get origin-shape attr) + dest-val (get dest-shape attr)] + (or (= origin-val dest-val) + (cond + ;; close-attrs? zips :points and would ignore a length mismatch + (= :points attr) + (and (= (count origin-val) (count dest-val)) + (gsh/close-attrs? attr origin-val dest-val)) + + (contains? drifting-geometry-attrs attr) + (gsh/close-attrs? attr origin-val dest-val) + + (and (= :content attr) + (or (cfh/path-shape? origin-shape) (cfh/bool-shape? origin-shape))) + (segment/content-close? origin-val dest-val) + + (= :position-data attr) + (position-data-close? origin-val dest-val) + + :else + false)))) + ;; Flex-child layout attrs synced between main and copy (interactions are only kept on swap) (def ^:private flex-child-attrs (disj ctk/swap-keep-attrs :interactions)) @@ -1993,7 +2031,7 @@ reset-pos-data? (and (cfh/text-shape? origin-shape) (= attr :position-data) - (not= (:position-data origin-shape) (:position-data dest-shape)) + (not (attr-equal? :position-data origin-shape dest-shape)) (touched :geometry-group)) ;; On texts, when we want to omit the touched attrs, both text (the actual letters) @@ -2007,7 +2045,7 @@ (touched sync-group)) skip-operations? - (or (= (get origin-shape attr) (get dest-shape attr)) + (or (attr-equal? attr origin-shape dest-shape) (and (touched sync-group) omit-touched? ;; When it is a text-partial-change, we should generate operations diff --git a/common/src/app/common/math.cljc b/common/src/app/common/math.cljc index 41e3047880..c924446c15 100644 --- a/common/src/app/common/math.cljc +++ b/common/src/app/common/math.cljc @@ -208,6 +208,19 @@ ([num1 num2 precision] (<= (abs (- num1 num2)) precision))) +(defn close-map? + "Equality for maps: same keys, numeric values within `close?`, any other + value compared exactly." + [a b] + (and (= (count a) (count b)) + (every? (fn [[k v]] + (and (contains? b k) + (let [w (get b k)] + (if (number? v) + (and (number? w) (close? v w)) + (= v w))))) + a))) + (defn lerp "Calculates a the linear interpolation between two values and a given percent" [v0 v1 t] diff --git a/common/src/app/common/types/path/segment.cljc b/common/src/app/common/types/path/segment.cljc index 561dd4e44b..dbfc1a44bc 100644 --- a/common/src/app/common/types/path/segment.cljc +++ b/common/src/app/common/types/path/segment.cljc @@ -1355,6 +1355,20 @@ (grc/points->rect extremities) (grc/make-rect)))) +(defn- segment-close? + [a b] + (and (= (:command a) (:command b)) + (mth/close-map? (:params a) (:params b)))) + +(defn content-close? + "True when both path contents have the same segments and every param + is within `mth/close?` of its counterpart." + [content-a content-b] + (let [a (vec content-a) + b (vec content-b)] + (and (= (count a) (count b)) + (every? true? (map segment-close? a b))))) + (defn content-center [content] (-> content diff --git a/common/test/common_tests/logic/comp_sync_test.cljc b/common/test/common_tests/logic/comp_sync_test.cljc index 02cd219779..f8ecbe336d 100644 --- a/common/test/common_tests/logic/comp_sync_test.cljc +++ b/common/test/common_tests/logic/comp_sync_test.cljc @@ -19,6 +19,7 @@ [app.common.test-helpers.shapes :as ths] [app.common.types.component :as ctk] [app.common.types.container :as ctn] + [app.common.types.modifiers :as ctm] [app.common.types.shape-tree :as ctst] [clojure.test :as t])) @@ -552,6 +553,330 @@ {(:id file) file} (:id file))) +(defn- changes-attrs + "Set of attrs touched by the :mod-obj changes of a changes map." + [changes] + (->> (:redo-changes changes) + (filter #(= :mod-obj (:type %))) + (mapcat :operations) + (map :attr) + (into #{}))) + +(defn- file-with-path-component + [] + (let [content [{:command :move-to :params {:x 10.3 :y 10.7}} + {:command :line-to :params {:x 24.3 :y 10.7}} + {:command :line-to :params {:x 17.3 :y 24.7}} + {:command :close-path}]] + (-> (thf/sample-file :file1) + (tho/add-frame :main-root {:width 107.3 :height 41.7}) + (ths/add-sample-shape :main-path :type :path :parent-label :main-root :content content) + (thc/make-component :component1 :main-root) + (thc/instantiate-component :component1 :copy-root {:children-labels [:copy-path]})))) + +(t/deftest test-no-sync-changes-when-position-changes-with-float-drift + ;; Regression (#12719): geometry in production carries sub-pixel drift + ;; and non-integer sizes, so move(+delta) followed by reposition(-delta) + ;; does not cancel exactly. A position-only move of the main must still + ;; produce no sync changes, and so no "library updated" notification. + (let [;; ==== Setup + file (-> (thf/sample-file :file1) + (tho/add-simple-component-with-copy :component1 + :main-root + :main-child + :copy-root + :main-root-params {:width 107.3 :height 41.7} + :main-child-params {:width 33.3 :height 17.7} + :copy-root-params {:children-labels [:copy-child]})) + page (thf/current-page file) + main-root (ths/get-shape file :main-root) + main-child (ths/get-shape file :main-child) + copy-child (ths/get-shape file :copy-child) + + ;; Override on the copy, so it carries touched state + changes0 (cls/generate-update-shapes (pcb/empty-changes nil (:id page)) + #{(:id copy-child)} + (fn [shape] + (assoc shape :fills (ths/sample-fills-color :fill-color "#fabada"))) + (:objects page) + {}) + file (thf/apply-changes file changes0) + page (thf/current-page file) + + ;; ==== Action + ;; Move the whole main by 1 px plus sub-pixel noise + delta (gpt/point 1.0000123 0.0000071) + changes1 (cls/generate-update-shapes (pcb/empty-changes nil (:id page)) + #{(:id main-root) (:id main-child)} + (fn [shape] (gsh/move shape delta)) + (:objects page) + {}) + + updated-file (thf/apply-changes file changes1) + + sync-changes (cll/generate-sync-file-changes (pcb/empty-changes) + nil + :components + (:id updated-file) + (thi/id :component1) + (:id updated-file) + {(:id updated-file) updated-file} + (:id updated-file))] + + ;; ==== Check + (t/is (empty? (:redo-changes sync-changes))))) + +(t/deftest test-sync-size-change-of-main-child + ;; The drift tolerance must not swallow real size edits (>> 0.001). + (let [;; ==== Setup + file (-> (thf/sample-file :file1) + (tho/add-simple-component-with-copy :component1 + :main-root + :main-child + :copy-root + :main-root-params {:width 107.3 :height 41.7} + :main-child-params {:width 33.3 :height 17.7} + :copy-root-params {:children-labels [:copy-child]})) + page (thf/current-page file) + + ;; ==== Action + updated-file (thf/apply-changes file + (cls/generate-update-shapes (pcb/empty-changes nil (:id page)) + #{(thi/id :main-child)} + #(gsh/transform-shape % (ctm/change-size % (+ (:width %) 1) nil)) + (:objects page) + {})) + attrs (changes-attrs (sync-file-changes updated-file))] + + ;; ==== Check + (t/is (contains? attrs :width)) + (t/is (contains? attrs :selrect)))) + +(t/deftest test-sync-text-position-data-change-with-same-xy + ;; A change in a text run (here its fill) that keeps x/y must still be + ;; synced: position-data is not a drifting geometry attr. + (let [;; ==== Setup + run (fn [color] + [{:x 0 :y 0 :width 50 :height 20 :text "hello" + :fills [{:fill-color color :fill-opacity 1}]}]) + file (-> (thf/sample-file :file1) + (tho/add-frame-with-text :main-root :main-child "hello") + (thc/make-component :component1 :main-root) + (thc/instantiate-component :component1 :copy-root {:children-labels [:copy-child]})) + set-pd (fn [file color] + (let [page (thf/current-page file) + main-child (ths/get-shape file :main-child)] + (thf/apply-changes + file + (cls/generate-update-shapes (pcb/empty-changes nil (:id page)) + #{(:id main-child)} + #(assoc % :position-data (run color)) + (:objects page) + {})))) + + file1 (set-pd file "#000000") + file2 (thf/apply-changes file1 (sync-file-changes file1)) + file3 (set-pd file2 "#ff0000") + + ;; ==== Action + sync-changes (sync-file-changes file3)] + + ;; ==== Check + (t/is (contains? (changes-attrs sync-changes) :position-data)))) + +(t/deftest test-sync-small-opacity-change + ;; The drift tolerance must not swallow real non-geometry changes. + (let [;; ==== Setup + file (-> (thf/sample-file :file1) + (tho/add-simple-component-with-copy :component1 + :main-root + :main-child + :copy-root + :copy-root-params {:children-labels [:copy-child]})) + set-opacity (fn [file opacity] + (let [page (thf/current-page file) + main-child (ths/get-shape file :main-child)] + (thf/apply-changes + file + (cls/generate-update-shapes (pcb/empty-changes nil (:id page)) + #{(:id main-child)} + #(assoc % :opacity opacity) + (:objects page) + {})))) + + file1 (set-opacity file 1) + file2 (thf/apply-changes file1 (sync-file-changes file1)) + file3 (set-opacity file2 0.9995) + + ;; ==== Action + sync-changes (sync-file-changes file3)] + + ;; ==== Check + (t/is (contains? (changes-attrs sync-changes) :opacity)))) + +(t/deftest test-no-sync-changes-when-path-moves-with-float-drift + ;; Path content stores absolute coordinates, so it drifts like the other + ;; geometry when the main is moved. + (let [;; ==== Setup + file (file-with-path-component) + page (thf/current-page file) + main-root (ths/get-shape file :main-root) + main-path (ths/get-shape file :main-path) + + ;; ==== Action + changes1 (cls/generate-update-shapes (pcb/empty-changes nil (:id page)) + #{(:id main-root) (:id main-path)} + #(gsh/move % (gpt/point 1.0000123 0.0000071)) + (:objects page) + {}) + updated-file (thf/apply-changes file changes1)] + + ;; ==== Check + (t/is (empty? (:redo-changes (sync-file-changes updated-file)))))) + +(t/deftest test-sync-path-moved-inside-main + ;; Moving only the path inside its main is a real change and must sync. + (let [;; ==== Setup + file (file-with-path-component) + page (thf/current-page file) + main-path (ths/get-shape file :main-path) + + ;; ==== Action + changes1 (cls/generate-update-shapes (pcb/empty-changes nil (:id page)) + #{(:id main-path)} + #(gsh/move % (gpt/point 1 0)) + (:objects page) + {}) + updated-file (thf/apply-changes file changes1)] + + ;; ==== Check + (t/is (contains? (changes-attrs (sync-file-changes updated-file)) :content)))) + +(defn- file-with-bool-component + [] + (let [content [{:command :move-to :params {:x 10.3 :y 10.7}} + {:command :line-to :params {:x 24.3 :y 10.7}} + {:command :line-to :params {:x 17.3 :y 24.7}} + {:command :close-path}]] + (-> (thf/sample-file :file1) + (tho/add-frame :main-root {:width 107.3 :height 41.7}) + (ths/add-sample-shape :main-bool :type :bool :bool-type :union :parent-label :main-root :content content) + (ths/add-sample-shape :main-p1 :type :path :parent-label :main-bool :content content) + (ths/add-sample-shape :main-p2 :type :path :parent-label :main-bool :content content) + (thc/make-component :component1 :main-root) + (thc/instantiate-component :component1 :copy-root {:children-labels [:copy-bool :copy-p1 :copy-p2]})))) + +(defn- move-shapes + [file labels delta] + (let [page (thf/current-page file)] + (thf/apply-changes file + (cls/generate-update-shapes (pcb/empty-changes nil (:id page)) + (into #{} (map #(thi/id %)) labels) + #(gsh/move % delta) + (:objects page) + {})))) + +(t/deftest test-no-sync-changes-when-text-moves-with-float-drift + ;; Text position-data moves with the shape and drifts like geometry, so + ;; a position-only move of the main must produce no sync changes. + (let [;; ==== Setup + position-data [{:x 10.3 :y 10.7 :width 33.3 :height 17.7 :text "hello" + :fills [{:fill-color "#000000" :fill-opacity 1}]}] + file (-> (thf/sample-file :file1) + (tho/add-frame-with-text :main-root :main-child "hello") + (thc/make-component :component1 :main-root) + (thc/instantiate-component :component1 :copy-root {:children-labels [:copy-child]})) + page (thf/current-page file) + file (thf/apply-changes file + (cls/generate-update-shapes (pcb/empty-changes nil (:id page)) + #{(thi/id :main-child)} + #(assoc % :position-data position-data) + (:objects page) + {})) + file (thf/apply-changes file (sync-file-changes file)) + + ;; ==== Action + updated-file (move-shapes file [:main-root :main-child] (gpt/point 1.0000123 0.0000071)) + sync-changes (sync-file-changes updated-file)] + + ;; ==== Check + (t/is (= #{} (changes-attrs sync-changes))) + (t/is (empty? (:redo-changes sync-changes))))) + +(t/deftest test-sync-text-moved-inside-main + ;; Moving only the text inside its main is a real change beyond the + ;; tolerance and must sync its position-data. + (let [;; ==== Setup + position-data [{:x 10.3 :y 10.7 :width 33.3 :height 17.7 :text "hello" + :fills [{:fill-color "#000000" :fill-opacity 1}]}] + file (-> (thf/sample-file :file1) + (tho/add-frame-with-text :main-root :main-child "hello") + (thc/make-component :component1 :main-root) + (thc/instantiate-component :component1 :copy-root {:children-labels [:copy-child]})) + page (thf/current-page file) + file (thf/apply-changes file + (cls/generate-update-shapes (pcb/empty-changes nil (:id page)) + #{(thi/id :main-child)} + #(assoc % :position-data position-data) + (:objects page) + {})) + file (thf/apply-changes file (sync-file-changes file)) + + ;; ==== Action + updated-file (move-shapes file [:main-child] (gpt/point 5 0))] + + ;; ==== Check + (t/is (contains? (changes-attrs (sync-file-changes updated-file)) :position-data)))) + +(t/deftest test-sync-text-position-data-entry-count-change + ;; A text that gains a line keeps its first entry untouched but has one + ;; more entry: a length mismatch is a real change and must sync. + (let [;; ==== Setup + entry {:x 10.3 :y 10.7 :width 33.3 :height 17.7 :text "hello" + :fills [{:fill-color "#000000" :fill-opacity 1}]} + set-pd (fn [file position-data] + (let [page (thf/current-page file)] + (thf/apply-changes file + (cls/generate-update-shapes (pcb/empty-changes nil (:id page)) + #{(thi/id :main-child)} + #(assoc % :position-data position-data) + (:objects page) + {})))) + file (-> (thf/sample-file :file1) + (tho/add-frame-with-text :main-root :main-child "hello") + (thc/make-component :component1 :main-root) + (thc/instantiate-component :component1 :copy-root {:children-labels [:copy-child]})) + file (set-pd file [entry]) + file (thf/apply-changes file (sync-file-changes file)) + + ;; ==== Action + updated-file (set-pd file [entry (assoc entry :y 28.7 :text "world")])] + + ;; ==== Check + (t/is (contains? (changes-attrs (sync-file-changes updated-file)) :position-data)))) + +(t/deftest test-no-sync-changes-when-bool-moves-with-float-drift + ;; Bool content stores absolute coordinates, so it drifts like path content. + (let [;; ==== Action + file (file-with-bool-component) + updated-file (move-shapes file + [:main-root :main-bool :main-p1 :main-p2] + (gpt/point 1.0000123 0.0000071)) + sync-changes (sync-file-changes updated-file)] + + ;; ==== Check + (t/is (= #{} (changes-attrs sync-changes))) + (t/is (empty? (:redo-changes sync-changes))))) + +(t/deftest test-sync-bool-content-change-inside-main + ;; Moving only the bool inside its main is a real change and must sync. + (let [;; ==== Action + file (file-with-bool-component) + updated-file (move-shapes file [:main-bool :main-p1 :main-p2] (gpt/point 1 0))] + + ;; ==== Check + (t/is (contains? (changes-attrs (sync-file-changes updated-file)) :content)))) + (t/deftest test-sync-many-updated-shapes ;; The undo changes of a sync used to grow one lazy level per updated ;; shape (#12576). Overflowing the stack needs thousands of shapes, which diff --git a/common/test/common_tests/math_test.cljc b/common/test/common_tests/math_test.cljc index 524963df9b..a8c3188a13 100644 --- a/common/test/common_tests/math_test.cljc +++ b/common/test/common_tests/math_test.cljc @@ -57,3 +57,21 @@ :clj (t/testing "finite? returns false for Double/NaN (CLJ)" (t/is (false? (mth/finite? Double/NaN)))))) + +(t/deftest close-map?-test + (t/testing "numbers within tolerance are close" + (t/is (mth/close-map? {:x 1.0 :y 2.0} {:x 1.0004 :y 2.0}))) + + (t/testing "numbers beyond tolerance are not close" + (t/is (not (mth/close-map? {:x 1.0} {:x 1.01})))) + + (t/testing "non-numeric values compare exactly" + (t/is (mth/close-map? {:text "a"} {:text "a"})) + (t/is (not (mth/close-map? {:text "a"} {:text "b"})))) + + (t/testing "different key sets are not close" + (t/is (not (mth/close-map? {:x 1.0} {:x 1.0 :y 1.0}))) + (t/is (not (mth/close-map? {:x 1.0 :y nil} {:x 1.0 :z nil})))) + + (t/testing "a number is not close to a non-number" + (t/is (not (mth/close-map? {:x 1.0} {:x "1.0"}))))) diff --git a/common/test/common_tests/types/path_data_test.cljc b/common/test/common_tests/types/path_data_test.cljc index 6c1b957691..de9762a2fc 100644 --- a/common/test/common_tests/types/path_data_test.cljc +++ b/common/test/common_tests/types/path_data_test.cljc @@ -2864,3 +2864,30 @@ result (path/splice-content content sub)] (t/is (path.impl/path-data? result)) (t/is (= (into (vec content) sub) (vec result))))) + +(t/deftest content-close? + (let [content [{:command :move-to :params {:x 10.3 :y 10.7}} + {:command :line-to :params {:x 24.3 :y 10.7}} + {:command :close-path}] + shifted (fn [f] (update-in content [1 :params :x] f))] + (t/testing "params within tolerance are close" + (t/is (path.segment/content-close? content (shifted #(+ % 0.0004))))) + + (t/testing "a param beyond tolerance is not close" + (t/is (not (path.segment/content-close? content (shifted #(+ % 0.01)))))) + + (t/testing "a different command is not close" + (t/is (not (path.segment/content-close? content (assoc-in content [1 :command] :move-to))))) + + (t/testing "a different segment count is not close" + (t/is (not (path.segment/content-close? content (butlast content))))) + + (t/testing "a different param count is not close" + (t/is (not (path.segment/content-close? content (assoc-in content [1 :params] {:x 24.3}))))) + + (t/testing "plain vector and PathData are comparable" + (t/is (path.segment/content-close? content (path.impl/path-data content)))) + + (t/testing "nil equals an empty content, but not a non-empty one" + (t/is (path.segment/content-close? nil [])) + (t/is (not (path.segment/content-close? nil content)))))) From e57d358c5335c285b1a3647c792d6f2bdce72b9c Mon Sep 17 00:00:00 2001 From: Andrey Antukh Date: Thu, 12 Nov 2026 07:09:42 +0100 Subject: [PATCH 16/29] :paperclip: Update .gitignore --- .gitignore | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index 5e38fb2361..974b11f495 100644 --- a/.gitignore +++ b/.gitignore @@ -121,7 +121,7 @@ opencode.json /.opencode/reports /.opencode/prompts /.ci-logs -/tmp/ +/tmp /.codex/ /scripts/__pycache__/ /tools/__pycache__ From 12d73d0f637afa3d7fa791869da3728ea9fe797a Mon Sep 17 00:00:00 2001 From: Andrey Antukh Date: Thu, 12 Nov 2026 06:27:40 +0000 Subject: [PATCH 17/29] :bug: Fix flaky assertion in climit waiting-count test The test expected exactly one publication of the :rpc-climit-waiting gauge inside the with-redefs window. The republisher daemon thread, started at component init with no delay and rescheduling every 15 s, publishes the same counter and can land inside that window, so the recorded sequence could be [0 0] and the assertion failed with actual [0 +0] (Backend Tests job in #12979). Assert on the last published value instead of the whole sequence, the contract waiting-last documents and the sibling tests use. AI-assisted-by: inkling-2-small-free --- backend/test/backend_tests/climit_test.clj | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/backend/test/backend_tests/climit_test.clj b/backend/test/backend_tests/climit_test.clj index 76aff9252a..d65b2a500a 100644 --- a/backend/test/backend_tests/climit_test.clj +++ b/backend/test/backend_tests/climit_test.clj @@ -445,9 +445,11 @@ (t/is (thrown? clojure.lang.ExceptionInfo (invoke! cfg (fn [_ _] :never) {})))) - ;; the count came back down instead of staying up for good + ;; the count came back down instead of staying up for good. The + ;; republisher daemon thread can add publications inside the window, + ;; so what is asserted is the last value, not the whole sequence. (t/is (= 0 (waiting-count climit name))) - (t/is (= [0] (waiting-value-of name))) + (t/is (= 0 (waiting-last name))) ;; and the thread dropped the mark: with it left behind, this call would ;; run inline instead of waiting for the permit another thread holds. From 75b55b567f3dcab90597dbac4a229b8912eb8aa0 Mon Sep 17 00:00:00 2001 From: Andrey Antukh Date: Thu, 12 Nov 2026 06:27:44 +0000 Subject: [PATCH 18/29] :books: Document test database and valkey env discipline Agents must respect the PENPOT_TEST_* variables already set in the shell and not override them with values copied from CI logs (the CI endpoints only exist inside the CI docker network) unless the user asks otherwise, such as a temporary database. The shared test database may carry migrations applied by newer versions; when that blocks a test run, dropping and recreating only the database from PENPOT_TEST_DATABASE_URI is explicitly permitted. AI-assisted-by: inkling-2-small-free --- .serena/memories/backend/testing.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.serena/memories/backend/testing.md b/.serena/memories/backend/testing.md index 3cec805699..a3536c63a4 100644 --- a/.serena/memories/backend/testing.md +++ b/.serena/memories/backend/testing.md @@ -3,6 +3,8 @@ JVM `clojure.test` (kaocha runner) under `backend/test/backend_tests/`. - READ `mem:testing` FIRST — it defines the execution discipline (no piping, tee to file, preferred commands) that applies to all JVM test runs. +- Test DB/Redis come from the `PENPOT_TEST_*` env vars already set in the shell (`PENPOT_TEST_DATABASE_URI`, `PENPOT_TEST_REDIS_URI`, ...); per instance they point at `penpot_test` / `penpot_test_wsN` and valkey DB `6+N` (`mem:devenv/core`). Do not override them with values copied from CI logs or other sources unless the user asks (e.g. "use a temporary database"); the CI env (`redis://redis/1`, `postgresql://postgres/penpot_test`) only works inside the CI docker network. +- The test database may have migrations applied from newer versions (it is shared dev infra). Reinitializing it — `dropdb`/`createdb` of the `PENPOT_TEST_*` database only — is explicitly permitted when that blocks a test run; never drop the main `penpot` database or another instance's test database. - All CLI commands must be executed from the `backend/` subdirectory. - Tests are invoked directly via `clojure -M:dev:test` (kaocha) — there is no pnpm wrapper. Kaocha auto-discovers test namespaces, so no runner registration is needed. - `clojure.core/assert` is elided at compile time unless `PENPOT_FLAGS` contains `enable-backend-asserts` (`app.config` sets `*assert*` from it). `scripts/ci` and `tests-backend.yml` export the flag; a bare `clojure -M:dev:test` (or `--focus` run) without it silently skips every precondition assert. From df0092fc601d75fecb4863368a5ab16dba5e12bb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marina=20L=C3=B3pez?= Date: Thu, 12 Nov 2026 08:35:32 +0100 Subject: [PATCH 19/29] :bug: Enforce sso before creating organization teams (#12980) --- frontend/src/app/main/data/team.cljs | 52 +++++++++-- frontend/src/app/main/ui/dashboard.cljs | 3 + .../test/frontend_tests/data/team_test.cljs | 87 +++++++++++++++++++ 3 files changed, 133 insertions(+), 9 deletions(-) diff --git a/frontend/src/app/main/data/team.cljs b/frontend/src/app/main/data/team.cljs index fe2b796ec8..80ab5e0fad 100644 --- a/frontend/src/app/main/data/team.cljs +++ b/frontend/src/app/main/data/team.cljs @@ -13,6 +13,7 @@ [app.common.types.organization :as cto] [app.common.types.team :as ctt] [app.common.uri :as u] + [app.common.uuid :as uuid] [app.config :as cf] [app.main.data.event :as ev] [app.main.data.helpers :as dsh] @@ -24,7 +25,9 @@ [app.main.repo :as rp] [app.main.router :as rt] [app.util.clipboard :as clipboard] + [app.util.dom :as dom] [app.util.i18n :refer [tr]] + [app.util.session-state :as ss] [app.util.storage :as storage] [beicon.v2.core :as rx] [clojure.string :as str] @@ -111,8 +114,8 @@ (f team))))))) (defn check-and-create-team - "Fetches fresh team data from the server to ensure up-to-date organization - permissions, then shows the team-form modal or a no-permission modal." + "Fetches fresh team data, checks organization permissions and SSO, then + shows the team form or the appropriate access error." [team-id] (ptk/reify ::check-and-create-team ptk/WatchEvent @@ -127,13 +130,44 @@ {:organization-perms organization :profile-id profile-id :team-perms (:permissions team)}) - true)] - (rx/of (if can-create? - (modal/show :team-form (if in-organization? - {:organization-id (:id organization) - :organization-name (:name organization)} - {})) - (modal/show :no-permission-modal {:type :create-team})))))))))) + true) + team-form-modal (modal/show :team-form (if in-organization? + {:organization-id (:id organization) + :organization-name (:name organization)} + {}))] + (cond + (not can-create?) + (rx/of (modal/show :no-permission-modal {:type :create-team})) + + (not in-organization?) + (rx/of team-form-modal) + + :else + (let [pending-id (str (uuid/next)) + callback-url (dom/append-query-param (rt/get-current-href) + :pending-action-id pending-id)] + (->> (rp/cmd! :check-nitrate-sso {:team-id team-id + :url callback-url}) + (rx/mapcat + (fn [{:keys [authorized redirect-uri]}] + (cond + authorized + (rx/of team-form-modal) + + redirect-uri + (do + (ss/save-pending-action! pending-id + {:type :create-team-in-organization + :team-id team-id}) + (rx/of (rt/nav-raw :uri (str redirect-uri)))) + + :else + (rx/of (rt/assign-exception + {:type :sso-error + :organization-id (:id organization) + :organization-name (:name organization) + :team-id team-id + :is-dashboard true}))))))))))))))) (defn check-and-delete-team "Fetches fresh team data from the server to ensure up-to-date organization diff --git a/frontend/src/app/main/ui/dashboard.cljs b/frontend/src/app/main/ui/dashboard.cljs index 13059c1036..43f43acbd0 100644 --- a/frontend/src/app/main/ui/dashboard.cljs +++ b/frontend/src/app/main/ui/dashboard.cljs @@ -18,6 +18,7 @@ [app.main.data.plugins :as dp] [app.main.data.profile :as dprof] [app.main.data.project :as dpj] + [app.main.data.team :as dtm] [app.main.refs :as refs] [app.main.router :as rt] [app.main.store :as st] @@ -298,6 +299,8 @@ (st/emit! (dnt/add-team-to-organization {:team-id (:team-id action) :organization-id (:organization-id action) :skip-audit? true})) + :create-team-in-organization + (st/emit! (dtm/check-and-create-team (:team-id action))) nil))))) (mf/defc dashboard* diff --git a/frontend/test/frontend_tests/data/team_test.cljs b/frontend/test/frontend_tests/data/team_test.cljs index 9bf84efc79..87ef76be68 100644 --- a/frontend/test/frontend_tests/data/team_test.cljs +++ b/frontend/test/frontend_tests/data/team_test.cljs @@ -5,9 +5,13 @@ (ns frontend-tests.data.team-test (:require [app.common.uuid :as uuid] + [app.main.data.modal :as modal] [app.main.data.team :as dtm] [app.main.repo :as rp] + [app.main.router :as rt] [app.main.ui.dashboard.team :as dtui] + [app.util.dom :as dom] + [app.util.session-state :as ss] [beicon.v2.core :as rx] [cljs.test :as t :include-macros true] [frontend-tests.helpers.async :as async] @@ -27,6 +31,89 @@ :on-next #(swap! emitted conj %)))) (.then (fn [_] {:team-id team-id :events @emitted}))))) +(t/deftest ^:async check-and-create-team-redirects-to-sso-and-remembers-the-action + (let [team-id "00000000-0000-4000-8000-000000000001" + organization-id "00000000-0000-4000-8000-000000000002" + profile-id "00000000-0000-4000-8000-000000000003" + dashboard-url "https://penpot.example/#/dashboard-recent?team-id=current-team" + callback-url (str dashboard-url "&pending-action-id=pending-id") + team {:id team-id + :organization {:id organization-id + :name "Product" + :owner-id profile-id}} + requests (atom []) + saved-actions (atom []) + emitted (atom [])] + (await + (mock/with-mocks* + {rp/cmd! (mock/stub + (fn [command params] + (swap! requests conj [command params]) + (case command + :get-teams (rx/of [team]) + :check-nitrate-sso + (rx/of {:authorized false + :redirect-uri "https://sso.example.test/login"})))) + rt/get-current-href (mock/stub (constantly dashboard-url)) + dom/append-query-param (mock/stub (fn [_url _key _value] callback-url)) + uuid/next (mock/stub (constantly "pending-id")) + ss/save-pending-action! (mock/stub (fn [& args] + (swap! saved-actions conj args))) + rt/nav-raw (fn [& args] {:nav-raw args}) + modal/show (mock/stub (fn [& args] {:modal-show args}))} + (await (async/observe + (ptk/watch (dtm/check-and-create-team team-id) + {:profile {:id profile-id}} + nil) + :on-next #(swap! emitted conj %))))) + + (t/is (= [[:get-teams nil] + [:check-nitrate-sso {:team-id team-id :url callback-url}]] + @requests)) + (t/is (= [["pending-id" {:type :create-team-in-organization + :team-id team-id}]] + @saved-actions)) + (t/is (= {:nav-raw [:uri "https://sso.example.test/login"]} + (last @emitted))) + (t/is (not-any? #(contains? % :modal-show) @emitted)))) + +(t/deftest ^:async check-and-create-team-shows-form-after-sso-is-authorized + (let [team-id "00000000-0000-4000-8000-000000000001" + organization-id "00000000-0000-4000-8000-000000000002" + profile-id "00000000-0000-4000-8000-000000000003" + dashboard-url "https://penpot.example/#/dashboard-recent?team-id=current-team" + callback-url (str dashboard-url "&pending-action-id=pending-id") + team {:id team-id + :organization {:id organization-id + :name "Product" + :owner-id profile-id}} + requests (atom []) + emitted (atom [])] + (await + (mock/with-mocks* + {rp/cmd! (mock/stub + (fn [command params] + (swap! requests conj [command params]) + (case command + :get-teams (rx/of [team]) + :check-nitrate-sso (rx/of {:authorized true})))) + rt/get-current-href (mock/stub (constantly dashboard-url)) + dom/append-query-param (mock/stub (fn [_url _key _value] callback-url)) + uuid/next (mock/stub (constantly "pending-id")) + modal/show (mock/stub (fn [& args] {:modal-show args}))} + (await (async/observe + (ptk/watch (dtm/check-and-create-team team-id) + {:profile {:id profile-id}} + nil) + :on-next #(swap! emitted conj %))))) + + (t/is (= [:check-nitrate-sso {:team-id team-id :url callback-url}] + (last @requests))) + (t/is (= {:modal-show [:team-form {:organization-id organization-id + :organization-name "Product"}]} + (last @emitted))) + (t/is (= 2 (count @emitted))))) + (t/deftest ^:async fetch-invitations-clears-the-list-on-permission-error ;; The pending invitations are an admin only list: a member that cannot ;; invite gets an insufficient permissions error. That must not become an From 64a51e86fd67bf34bf4ccdc41be812ea4679e448 Mon Sep 17 00:00:00 2001 From: Andrey Antukh Date: Thu, 12 Nov 2026 09:41:49 +0100 Subject: [PATCH 20/29] :bug: Fix files blocked by dangling child references in migrations (#12936) (#12979) The 0033-normalize-main-instance-names migration walks the shape tree following the :shapes vectors. When a parent listed a child id missing from the container :objects map (a dangling reference), the traversal stored the nil child into :objects, creating a nil key that made check-file-data reject the whole file: get-file answered 500 "invalid file data" and the file could not be opened or migrated further. Fix it in two parts: - 0033 returns the container untouched when a listed child does not exist, so the traversal never stores a nil shape. - The append-only migration 0035-fix-dangling-children prunes from every :shapes vector, in pages and components, the ids absent from the owning container :objects map, logging each fix at trace level. It mirrors 0032-fix-children-not-in-parent, which repairs the opposite inconsistency. Both parts are needed: 0033 runs before the repair migration, so without the guard the same file fails again in the same migrate call. Add regression tests for the traversal guard and for the repair migration, and record in mem:clojure/logging the convention that :hints start lowercase. Closes #12936 AI-assisted-by: qwen3.8-flash --- .serena/memories/clojure/logging.md | 7 + .serena/memories/critical-info.md | 1 + common/src/app/common/files/migrations.cljc | 105 ++++++---- .../files_migrations_0033_test.cljc | 138 +++++++++++++ .../files_migrations_0035_test.cljc | 186 ++++++++++++++++++ common/test/common_tests/runner.cljc | 4 + 6 files changed, 406 insertions(+), 35 deletions(-) create mode 100644 .serena/memories/clojure/logging.md create mode 100644 common/test/common_tests/files_migrations_0033_test.cljc create mode 100644 common/test/common_tests/files_migrations_0035_test.cljc diff --git a/.serena/memories/clojure/logging.md b/.serena/memories/clojure/logging.md new file mode 100644 index 0000000000..56fba6a1b3 --- /dev/null +++ b/.serena/memories/clojure/logging.md @@ -0,0 +1,7 @@ +# Clojure Logging Conventions + +Structured logging through `app.common.logging` (`l/trc`/`l/dbg`/`l/info`/`l/wrn`/`l/error`/`l/fatal`): `:hint` carries the human-readable message, every other datum goes in its own keyword field (`:id`, `:container`, `:removed`, ...). + +- `:hint` values start lowercase: `"fix dangling children"`, not `"Fix dangling children"`. Uppercase-first hints exist as legacy (~57 occurrences, the densest in `backend/src`; also `l/trc` in migration `legacy-28` and `log/error` in `app.common.files.repair`) — do not copy them, and do not mass-rewrite them unless the surrounding code needs the change. +- A name that requires uppercase (an env var like `PENPOT_MEDIA_PROCESSING_SERVICE_URI`, an acronym like `MCP` or `LDAP`) stays uppercase inside an otherwise lowercase sentence. +- Repair work done by a file migration logs at `l/trc`, only when it changes something, and with the ids it touched: the house style of `migrations.cljc` (`legacy-28` precedes; the file has no `warn`/`info` lines). The shipped `log4j2.xml` pins the `app` logger to `info` (static file, `monitorInterval="60"`), so those trace lines appear only after the operator raises the level; the durable record of a repaired file is the migration id in its `:migrations`. diff --git a/.serena/memories/critical-info.md b/.serena/memories/critical-info.md index 43f678a643..ac15e674bd 100644 --- a/.serena/memories/critical-info.md +++ b/.serena/memories/critical-info.md @@ -6,6 +6,7 @@ You are working on the GitHub project `penpot/penpot`, a monorepo. - A section's top-level memory is `
/core`. When a section is relevant, read the core memory before focused memories. - Edits/stale refs/duplication cleanup: `mem:memory-maintenance`. - Cross-cutting testing principles, TDD workflow, and anti-patterns: `mem:testing`. +- Structured logging conventions (`:hint` casing and fields, log levels): `mem:clojure/logging` — read before adding or changing any log call. - Verified Clojure language behaviors that contradict common assumptions (e.g. `int?` covers `Long`; `integer?` is the general predicate): `mem:clojure/idioms` — read before assuming stdlib predicate semantics. - Read `mem:clojure/design-rules` (function shape, boundary checks, naming) before writing or modifying Clojure/ClojureScript code — not only when designing a solution or an API. diff --git a/common/src/app/common/files/migrations.cljc b/common/src/app/common/files/migrations.cljc index 6a82e463ef..c2eba53139 100644 --- a/common/src/app/common/files/migrations.cljc +++ b/common/src/app/common/files/migrations.cljc @@ -2246,44 +2246,49 @@ ;; been added to enforce this from now on. ;; Main instances in variants are a special case because they need to match the variant name. (letfn [(update-shape [container shape-id] - (let [shape (ctn/get-shape container shape-id) + (let [shape (ctn/get-shape container shape-id)] + ;; A shape may reference a child that does not exist in the + ;; container (a dangling reference). There is nothing to update + ;; on it, and storing it would add a nil entry to the objects + ;; map, making the whole file invalid. + (if (nil? shape) + container + (let [shape (cond + ;; Variant container must have a normalized name + (ctk/is-variant-container? shape) + (let [name (:name shape) + clean-name (cpn/clean-path name) + clean-name (if (str/blank? clean-name) + "(empty)" ;; this may occur if the name contains only separators + clean-name)] + (if (= name clean-name) + shape + (assoc shape :name clean-name))) - shape (cond - ;; Variant container must have a normalized name - (ctk/is-variant-container? shape) - (let [name (:name shape) - clean-name (cpn/clean-path name) - clean-name (if (str/blank? clean-name) - "(empty)" ;; this may occur if the name contains only separators - clean-name)] - (if (= name clean-name) - shape - (assoc shape :name clean-name))) + ;; Main instance inside a variant must have the same name as the container + (ctk/main-instance? shape) + (if (ctk/is-variant? shape) + (let [variant-container (ctn/get-shape container (:parent-id shape))] + (if (or (= (:name shape) (:name variant-container)) + (nil? variant-container)) + shape + (assoc shape :name (:name variant-container)))) - ;; Main instance inside a variant must have the same name as the container - (ctk/main-instance? shape) - (if (ctk/is-variant? shape) - (let [variant-container (ctn/get-shape container (:parent-id shape))] - (if (or (= (:name shape) (:name variant-container)) - (nil? variant-container)) - shape - (assoc shape :name (:name variant-container)))) + ;; Other main instances must have the same name as the path+name of the component + (let [component (ctkl/get-component data (:component-id shape) true) + expected-name (when component + (cpn/merge-path-item (:path component) (:name component)))] + (if (or (= expected-name (:name shape)) + (nil? expected-name)) + shape + (assoc shape :name expected-name)))) - ;; Other main instances must have the same name as the path+name of the component - (let [component (ctkl/get-component data (:component-id shape) true) - expected-name (when component - (cpn/merge-path-item (:path component) (:name component)))] - (if (or (= expected-name (:name shape)) - (nil? expected-name)) - shape - (assoc shape :name expected-name)))) + ;; Any other shape does not need processing + :else shape)] - ;; Any other shape does not need processing - :else shape)] - - (reduce update-shape - (ctt/set-shape container shape) - (:shapes shape)))) + (reduce update-shape + (ctt/set-shape container shape) + (:shapes shape)))))) (update-container [container] @@ -2335,6 +2340,35 @@ (update :pages-index d/update-vals update-container) (d/update-when :components d/update-vals update-container))))) +(defmethod migrate-data "0035-fix-dangling-children" + [data _] + ;; A shape may list a child id that does not exist in the objects map of + ;; the page or component that owns it (a dangling reference). Such a + ;; reference is invalid: any operation that walks the shape tree through + ;; the children lists trips on it, and the file cannot be migrated nor + ;; opened. Here we remove the dangling ids from the children lists, + ;; keeping every shape that really exists. + (letfn [(fix-shape [container-id objects shape] + (d/update-when shape :shapes + (fn [shapes] + (let [shapes' (filterv #(contains? objects %) shapes)] + (when (not= shapes shapes') + (l/trc :hint "fix dangling children" + :container container-id + :shape-id (:id shape) + :removed (remove #(contains? objects %) shapes))) + shapes')))) + + (fix-objects [container-id objects] + (update-vals objects (partial fix-shape container-id objects))) + + (update-container [container] + (d/update-when container :objects #(fix-objects (:id container) %)))] + + (-> data + (update :pages-index d/update-vals update-container) + (d/update-when :components d/update-vals update-container)))) + (def available-migrations (into (d/ordered-set) ["legacy-2" @@ -2426,4 +2460,5 @@ "0031-migrate-stroke-width-token-attr" "0032-fix-children-not-in-parent" "0033-normalize-main-instance-names" - "0034-fix-path-content-offset"])) + "0034-fix-path-content-offset" + "0035-fix-dangling-children"])) diff --git a/common/test/common_tests/files_migrations_0033_test.cljc b/common/test/common_tests/files_migrations_0033_test.cljc new file mode 100644 index 0000000000..6f56602b77 --- /dev/null +++ b/common/test/common_tests/files_migrations_0033_test.cljc @@ -0,0 +1,138 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns common-tests.files-migrations-0033-test + "Regression coverage for the shape tree traversal of the + `0033-normalize-main-instance-names` migration. + + The migration walks a container from its root down, following the + `:shapes` vectors. A file can hold a `:shapes` entry pointing to a + shape that is not in the `:objects` map (a dangling reference). The + traversal must skip it instead of storing a nil shape into the + `:objects` map, and must keep normalizing the names of the shapes + that do exist." + (:require + [app.common.files.migrations :as cfm] + [app.common.types.shape :as cts] + [app.common.uuid :as uuid] + [clojure.test :as t])) + +(def ^:private migration "0033-normalize-main-instance-names") + +(defn- make-root + [children] + (cts/setup-shape {:id uuid/zero + :type :frame + :name "Root" + :shapes children})) + +(defn- make-main-instance + "A frame declared as the main instance of a component." + [main-id file-id component-id name] + (-> (cts/setup-shape {:id main-id :type :frame :name name}) + (assoc :main-instance true + :component-id component-id + :component-file file-id))) + +(defn- page-data + "File data with a single page owning `objects`." + [page-id objects] + {:pages-index {page-id {:id page-id :objects objects}}}) + +(defn- component-data + "File data with a single component owning `objects`." + [component-id objects] + {:components {component-id {:id component-id :objects objects}}}) + +;; --- The guard: a missing child must not become a nil shape + +(t/deftest missing-child-of-a-page-does-not-create-a-nil-shape + (let [page-id (uuid/next) + child-id (uuid/next) + root (make-root [child-id]) + data (page-data page-id {uuid/zero root}) + migrated (cfm/migrate-data data migration) + objects' (get-in migrated [:pages-index page-id :objects])] + + (t/is (not (contains? objects' nil)) + "no nil key is added to the objects map") + (t/is (= 1 (count objects')) + "the objects map keeps only the shapes that exist") + (t/is (= root (get objects' uuid/zero)) + "the root is left untouched"))) + +(t/deftest missing-child-of-a-component-does-not-create-a-nil-shape + (let [component-id (uuid/next) + child-id (uuid/next) + root (make-root [child-id]) + data (component-data component-id {uuid/zero root}) + migrated (cfm/migrate-data data migration) + objects' (get-in migrated [:components component-id :objects])] + + (t/is (not (contains? objects' nil)) + "no nil key is added to the objects map") + (t/is (= 1 (count objects')) + "the objects map keeps only the shapes that exist") + (t/is (= root (get objects' uuid/zero)) + "the component root is left untouched"))) + +(t/deftest missing-child-deep-in-the-tree-does-not-create-a-nil-shape + (let [page-id (uuid/next) + frame-id (uuid/next) + missing-id (uuid/next) + frame (-> (cts/setup-shape {:id frame-id :type :frame :name "Frame"}) + (assoc :shapes [missing-id])) + data (page-data page-id {uuid/zero (make-root [frame-id]) + frame-id frame}) + migrated (cfm/migrate-data data migration) + objects' (get-in migrated [:pages-index page-id :objects])] + + (t/is (not (contains? objects' nil)) + "no nil key is added to the objects map") + (t/is (= frame (get objects' frame-id)) + "the parent of the missing child is left untouched"))) + +;; --- The behavior: names keep being normalized next to a missing child + +(t/deftest main-instance-of-a-page-is-renamed-next-to-a-missing-child + (let [file-id (uuid/next) + page-id (uuid/next) + component-id (uuid/next) + main-id (uuid/next) + missing-id (uuid/next) + main-shape (make-main-instance main-id file-id component-id "Group/Name") + data (assoc (page-data page-id {uuid/zero (make-root [missing-id main-id]) + main-id main-shape}) + :components {component-id {:id component-id + :name "Name" + :path "Group" + :main-instance-id main-id}}) + migrated (cfm/migrate-data data migration) + objects' (get-in migrated [:pages-index page-id :objects])] + + (t/is (not (contains? objects' nil)) + "no nil key is added to the objects map") + (t/is (= "Group / Name" (get-in objects' [main-id :name])) + "the main instance takes the path and name of its component"))) + +(t/deftest main-instance-of-a-component-is-renamed-next-to-a-missing-child + (let [file-id (uuid/next) + component-id (uuid/next) + main-id (uuid/next) + missing-id (uuid/next) + main-shape (make-main-instance main-id file-id component-id "Group/Name") + data (component-data component-id {uuid/zero (make-root [missing-id main-id]) + main-id main-shape}) + data (assoc-in data [:components component-id :name] "Name") + data (assoc-in data [:components component-id :path] "Group") + data (assoc-in data [:components component-id :main-instance-id] main-id) + migrated (cfm/migrate-data data migration) + objects' (get-in migrated [:components component-id :objects])] + + (t/is (not (contains? objects' nil)) + "no nil key is added to the objects map") + (t/is (= "Group / Name" (get-in objects' [main-id :name])) + "the main instance takes the path and name of its component"))) diff --git a/common/test/common_tests/files_migrations_0035_test.cljc b/common/test/common_tests/files_migrations_0035_test.cljc new file mode 100644 index 0000000000..3680e8daf9 --- /dev/null +++ b/common/test/common_tests/files_migrations_0035_test.cljc @@ -0,0 +1,186 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns common-tests.files-migrations-0035-test + "Coverage for `0035-fix-dangling-children`. + + A shape can list a child id that is not present in the `:objects` map + of the page or component owning it (a dangling reference). Any + traversal of the shape tree trips on it, and the file becomes + unopenable. The migration drops those ids and keeps every shape that + really exists." + (:require + [app.common.files.migrations :as cfm] + [app.common.types.file :as ctf] + [app.common.types.objects-map :as omap] + [app.common.types.shape :as cts] + [app.common.uuid :as uuid] + [clojure.test :as t])) + +(def ^:private migration "0035-fix-dangling-children") + +(defn- make-root + [children] + (cts/setup-shape {:id uuid/zero + :type :frame + :name "Root" + :shapes children})) + +(defn- make-frame + [frame-id child-ids] + (-> (cts/setup-shape {:id frame-id :type :frame :name "Frame"}) + (assoc :shapes child-ids))) + +(defn- page-data + [page-id objects] + {:pages-index {page-id {:id page-id :objects objects}}}) + +(defn- component-data + [component-id objects] + {:components {component-id {:id component-id :objects objects}}}) + +(t/deftest removes-the-dangling-children-of-a-page + (let [page-id (uuid/next) + frame-id (uuid/next) + child-id (uuid/next) + frame (make-frame frame-id [child-id]) + data (page-data page-id {uuid/zero (make-root [frame-id child-id]) + frame-id frame}) + migrated (cfm/migrate-data data migration) + objects' (get-in migrated [:pages-index page-id :objects])] + + (t/is (= [frame-id] (get-in objects' [uuid/zero :shapes])) + "the dangling id is removed from the root children list") + (t/is (= [] (get-in objects' [frame-id :shapes])) + "the dangling id is removed from the frame children list") + (t/is (= 2 (count objects')) + "every shape that exists is kept") + (t/is (some? (get objects' frame-id)) + "the frame itself is kept"))) + +(t/deftest removes-the-dangling-children-of-a-component + (let [component-id (uuid/next) + frame-id (uuid/next) + child-id (uuid/next) + frame (cts/setup-shape {:id frame-id :type :frame :name "Frame"}) + data (component-data component-id {uuid/zero (make-root [frame-id child-id]) + frame-id frame}) + migrated (cfm/migrate-data data migration) + objects' (get-in migrated [:components component-id :objects])] + + (t/is (= [frame-id] (get-in objects' [uuid/zero :shapes])) + "the dangling id is removed from the children list") + (t/is (= 2 (count objects')) + "every shape that exists is kept") + (t/is (= frame (get objects' frame-id)) + "the existing shape is left untouched"))) + +(t/deftest migration-works-with-an-objects-map + ;; The backend stores the page :objects as an ObjectsMap (the lazy + ;; representation enabled by the fdata/objects-map feature). It is an + ;; IPersistentMap that does not implement IFn on the JVM: a migration + ;; must consult it with contains?/get, never by calling it as a + ;; function. + (let [page-id (uuid/next) + frame-id (uuid/next) + child-id (uuid/next) + frame (make-frame frame-id [child-id]) + objects (omap/wrap {uuid/zero (make-root [frame-id child-id]) + frame-id frame}) + data (page-data page-id objects) + migrated (cfm/migrate-data data migration) + objects' (get-in migrated [:pages-index page-id :objects])] + + (t/is (= [frame-id] (get-in objects' [uuid/zero :shapes])) + "the dangling id is removed with the ObjectsMap representation") + (t/is (= [] (get-in objects' [frame-id :shapes])) + "the frame children list is pruned the same way"))) + +(t/deftest keeps-the-order-and-the-shapes-of-a-valid-tree + (let [page-id (uuid/next) + frame-id (uuid/next) + rect-id (uuid/next) + frame (cts/setup-shape {:id frame-id :type :frame :name "Frame"}) + rect (cts/setup-shape {:id rect-id :type :rect :name "Rect"}) + data (page-data page-id {uuid/zero (make-root [frame-id rect-id]) + frame-id frame + rect-id rect}) + migrated (cfm/migrate-data data migration) + objects' (get-in migrated [:pages-index page-id :objects])] + + (t/is (= [frame-id rect-id] (get-in objects' [uuid/zero :shapes])) + "the children list is kept as it was") + (t/is (= data migrated) + "a tree without dangling references is not modified"))) + +(t/deftest leaves-shapes-with-empty-children-alone + (let [page-id (uuid/next) + frame (cts/setup-shape {:id (uuid/next) :type :frame :name "Frame"}) + data (page-data page-id {uuid/zero (make-root []) + (:id frame) frame}) + migrated (cfm/migrate-data data migration)] + + (t/is (= data migrated) + "empty children vectors are left alone"))) + +(t/deftest leaves-shapes-without-children-key-alone + (let [page-id (uuid/next) + rect-id (uuid/next) + rect (cts/setup-shape {:id rect-id :type :rect :name "Rect"}) + data (page-data page-id {uuid/zero (make-root []) + rect-id rect}) + migrated (cfm/migrate-data data migration)] + + (t/is (= data migrated) + "shapes without a children key are left alone") + (t/is (not (contains? (get-in migrated [:pages-index page-id :objects rect-id]) :shapes)) + "no children key is created on the shape"))) + +(t/deftest migration-is-idempotent + (let [page-id (uuid/next) + child-id (uuid/next) + data (page-data page-id {uuid/zero (make-root [child-id])}) + migrated (cfm/migrate-data data migration)] + + (t/is (= migrated (cfm/migrate-data migrated migration)) + "applying the migration twice does not change the result"))) + +(t/deftest file-with-dangling-children-passes-the-whole-migration + (let [file-id (uuid/next) + page-id (uuid/next) + frame-id (uuid/next) + missing-id (uuid/next) + frame (make-frame frame-id [missing-id]) + base (ctf/make-file-data file-id page-id) + data (-> base + (assoc-in [:pages-index page-id :objects uuid/zero] + (make-root [frame-id missing-id])) + (assoc-in [:pages-index page-id :objects frame-id] frame)) + ;; The file reached the state before 0033 ran, so 0033, 0034 and + ;; 0035 are all pending, exactly as in the reported failure. + file {:id file-id + :data data + :migrations (disj cfm/available-migrations + "0033-normalize-main-instance-names" + "0034-fix-path-content-offset" + migration)}] + + (t/is (cfm/need-migration? file) + "the file is scheduled for migration") + + (let [migrated (cfm/migrate-file file {}) + objects' (get-in migrated [:data :pages-index page-id :objects])] + + (t/is (contains? (:migrations migrated) migration) + "the migration is recorded as applied") + (t/is (not (contains? objects' nil)) + "the migrated data has no nil shape entry") + (t/is (= [frame-id] (get-in objects' [uuid/zero :shapes])) + "the dangling reference of the root is removed") + (t/is (= [] (get-in objects' [frame-id :shapes])) + "the dangling reference of the frame is removed") + (t/is (some? (get objects' frame-id)) + "the frame that exists is kept")))) diff --git a/common/test/common_tests/runner.cljc b/common/test/common_tests/runner.cljc index 5d24e4ea9f..c82551cfea 100644 --- a/common/test/common_tests/runner.cljc +++ b/common/test/common_tests/runner.cljc @@ -22,7 +22,9 @@ [common-tests.files-changes-test] [common-tests.files-migrations-0025-test] [common-tests.files-migrations-0026-test] + [common-tests.files-migrations-0033-test] [common-tests.files-migrations-0034-test] + [common-tests.files-migrations-0035-test] [common-tests.files-migrations-test] [common-tests.files.comp-processors-test] [common-tests.files.helpers-test] @@ -134,7 +136,9 @@ 'common-tests.files-changes-test 'common-tests.files-migrations-0025-test 'common-tests.files-migrations-0026-test + 'common-tests.files-migrations-0033-test 'common-tests.files-migrations-0034-test + 'common-tests.files-migrations-0035-test 'common-tests.files-migrations-test 'common-tests.files.comp-processors-test 'common-tests.files.helpers-test From 1eccb3b4bb78b3e1eea31f9c6850d992c52450f1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?andr=C3=A9s=20gonz=C3=A1lez?= Date: Thu, 12 Nov 2026 11:41:11 +0100 Subject: [PATCH 21/29] :sparkles: Track library token usage in analytics events (#12958) Add a from-library property to the apply-tokens event, true when the file takes its tokens from a linked library. Emit a link-file-to-library-tokens event with tokens-count when the linked library has tokens, mirroring link-file-to-library-variants. Both events share one get-library-usage call. Closes #12915 AI-assisted-by: claude-opus-5-5 --- .../app/main/data/workspace/libraries.cljs | 26 ++++++++++++++----- .../data/workspace/tokens/application.cljs | 5 +++- 2 files changed, 23 insertions(+), 8 deletions(-) diff --git a/frontend/src/app/main/data/workspace/libraries.cljs b/frontend/src/app/main/data/workspace/libraries.cljs index 8d2aec1597..7de0e1d932 100644 --- a/frontend/src/app/main/data/workspace/libraries.cljs +++ b/frontend/src/app/main/data/workspace/libraries.cljs @@ -1753,6 +1753,7 @@ (let [libraries (:shared-files state) library (get libraries library-id) variants-count (-> library :library-summary :variants count) + tokens-count (or (-> library :library-summary :tokens-count) 0) loaded-libraries (->> (dsh/lookup-libraries state) (remove (fn [[_ lib]] @@ -1773,14 +1774,25 @@ (rx/of (ntf/error (tr "errors.circular-library-reference"))) (rx/throw cause)))))) (rx/of (ptk/reify ::link-file-to-library-finished)) - (when (pos? variants-count) + (when (or (pos? variants-count) (pos? tokens-count)) (->> (rp/cmd! :get-library-usage {:file-id library-id}) - (rx/map (fn [library-usage] - (ev/event {::ev/name "link-file-to-library-variants" - :file-id file-id - :library-id library-id - :variants-count variants-count - :library-used-in (:used-in library-usage)})))))))))) + (rx/mapcat (fn [library-usage] + (let [used-in (:used-in library-usage)] + (rx/from + (cond-> [] + (pos? variants-count) + (conj (ev/event {::ev/name "link-file-to-library-variants" + :file-id file-id + :library-id library-id + :variants-count variants-count + :library-used-in used-in})) + + (pos? tokens-count) + (conj (ev/event {::ev/name "link-file-to-library-tokens" + :file-id file-id + :library-id library-id + :tokens-count tokens-count + :library-used-in used-in})))))))))))))) (defn cleanup-unlinked-libraries "Remove libraries from state that are no longer linked to the given file. diff --git a/frontend/src/app/main/data/workspace/tokens/application.cljs b/frontend/src/app/main/data/workspace/tokens/application.cljs index 13bfd42689..937204b4a6 100644 --- a/frontend/src/app/main/data/workspace/tokens/application.cljs +++ b/frontend/src/app/main/data/workspace/tokens/application.cljs @@ -733,6 +733,8 @@ (all-attrs-appliable-for-token? attributes (:type token))))))) shape-ids (d/nilv (keys shapes) []) any-variant? (->> shapes vals (some ctk/is-variant?) boolean) + file-data (dsh/lookup-file-data state) + from-library (not (cfo/effective-tokens-source? file-data (:id file-data))) resolved-value (get-in resolved-tokens [(:name token) :resolved-value]) resolved-value (if (contains? cf/flags :tokenscript) @@ -746,7 +748,8 @@ (-> {::ev/name "apply-tokens" :type type :applied-to attributes - :applied-to-variant any-variant?} + :applied-to-variant any-variant? + :from-library from-library} (merge (meta it))))) (dwu/start-undo-transaction undo-id) (dwsh/update-shapes shape-ids (fn [shape] From 2c86db80bd774278f3c2bbed5fb158ffaaaaa556 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andr=C3=A9s=20Moya?= Date: Thu, 12 Nov 2026 11:38:20 +0100 Subject: [PATCH 22/29] :wrench: Unify migration tests --- ...t.cljc => files_migrations_0035_test.cljc} | 179 ++++++++++++++++- .../files_migrations_0035a_test.cljc | 186 ------------------ common/test/common_tests/runner.cljc | 6 +- 3 files changed, 180 insertions(+), 191 deletions(-) rename common/test/common_tests/{files_migrations_0035b_test.cljc => files_migrations_0035_test.cljc} (65%) delete mode 100644 common/test/common_tests/files_migrations_0035a_test.cljc diff --git a/common/test/common_tests/files_migrations_0035b_test.cljc b/common/test/common_tests/files_migrations_0035_test.cljc similarity index 65% rename from common/test/common_tests/files_migrations_0035b_test.cljc rename to common/test/common_tests/files_migrations_0035_test.cljc index 5e2edcc82c..350b669fed 100644 --- a/common/test/common_tests/files_migrations_0035b_test.cljc +++ b/common/test/common_tests/files_migrations_0035_test.cljc @@ -4,16 +4,192 @@ ;; ;; Copyright (c) KALEIDOS SUBSIDIARY SL -(ns common-tests.files-migrations-0035b-test +(ns common-tests.files-migrations-0035-test (:require [app.common.files.migrations :as cfm] [app.common.geom.matrix :as gmt] [app.common.geom.rect :as grc] [app.common.types.file :as ctf] + [app.common.types.objects-map :as omap] [app.common.types.shape :as cts] [app.common.uuid :as uuid] [clojure.test :as t])) +;; ====================================================================== +;; 0035-fix-dangling-children +;; A shape can list a child id that is not present in the `:objects` map +;; of the page or component owning it (a dangling reference). Any +;; traversal of the shape tree trips on it, and the file becomes +;; unopenable. The migration drops those ids and keeps every shape that +;; really exists. +;; ====================================================================== + +(def ^:private migration "0035-fix-dangling-children") + +(defn- make-root + [children] + (cts/setup-shape {:id uuid/zero + :type :frame + :name "Root" + :shapes children})) + +(defn- make-frame + [frame-id child-ids] + (-> (cts/setup-shape {:id frame-id :type :frame :name "Frame"}) + (assoc :shapes child-ids))) + +(defn- page-data + [page-id objects] + {:pages-index {page-id {:id page-id :objects objects}}}) + +(defn- component-data + [component-id objects] + {:components {component-id {:id component-id :objects objects}}}) + +(t/deftest removes-the-dangling-children-of-a-page + (let [page-id (uuid/next) + frame-id (uuid/next) + child-id (uuid/next) + frame (make-frame frame-id [child-id]) + data (page-data page-id {uuid/zero (make-root [frame-id child-id]) + frame-id frame}) + migrated (cfm/migrate-data data migration) + objects' (get-in migrated [:pages-index page-id :objects])] + + (t/is (= [frame-id] (get-in objects' [uuid/zero :shapes])) + "the dangling id is removed from the root children list") + (t/is (= [] (get-in objects' [frame-id :shapes])) + "the dangling id is removed from the frame children list") + (t/is (= 2 (count objects')) + "every shape that exists is kept") + (t/is (some? (get objects' frame-id)) + "the frame itself is kept"))) + +(t/deftest removes-the-dangling-children-of-a-component + (let [component-id (uuid/next) + frame-id (uuid/next) + child-id (uuid/next) + frame (cts/setup-shape {:id frame-id :type :frame :name "Frame"}) + data (component-data component-id {uuid/zero (make-root [frame-id child-id]) + frame-id frame}) + migrated (cfm/migrate-data data migration) + objects' (get-in migrated [:components component-id :objects])] + + (t/is (= [frame-id] (get-in objects' [uuid/zero :shapes])) + "the dangling id is removed from the children list") + (t/is (= 2 (count objects')) + "every shape that exists is kept") + (t/is (= frame (get objects' frame-id)) + "the existing shape is left untouched"))) + +(t/deftest migration-works-with-an-objects-map + ;; The backend stores the page :objects as an ObjectsMap (the lazy + ;; representation enabled by the fdata/objects-map feature). It is an + ;; IPersistentMap that does not implement IFn on the JVM: a migration + ;; must consult it with contains?/get, never by calling it as a + ;; function. + (let [page-id (uuid/next) + frame-id (uuid/next) + child-id (uuid/next) + frame (make-frame frame-id [child-id]) + objects (omap/wrap {uuid/zero (make-root [frame-id child-id]) + frame-id frame}) + data (page-data page-id objects) + migrated (cfm/migrate-data data migration) + objects' (get-in migrated [:pages-index page-id :objects])] + + (t/is (= [frame-id] (get-in objects' [uuid/zero :shapes])) + "the dangling id is removed with the ObjectsMap representation") + (t/is (= [] (get-in objects' [frame-id :shapes])) + "the frame children list is pruned the same way"))) + +(t/deftest keeps-the-order-and-the-shapes-of-a-valid-tree + (let [page-id (uuid/next) + frame-id (uuid/next) + rect-id (uuid/next) + frame (cts/setup-shape {:id frame-id :type :frame :name "Frame"}) + rect (cts/setup-shape {:id rect-id :type :rect :name "Rect"}) + data (page-data page-id {uuid/zero (make-root [frame-id rect-id]) + frame-id frame + rect-id rect}) + migrated (cfm/migrate-data data migration) + objects' (get-in migrated [:pages-index page-id :objects])] + + (t/is (= [frame-id rect-id] (get-in objects' [uuid/zero :shapes])) + "the children list is kept as it was") + (t/is (= data migrated) + "a tree without dangling references is not modified"))) + +(t/deftest leaves-shapes-with-empty-children-alone + (let [page-id (uuid/next) + frame (cts/setup-shape {:id (uuid/next) :type :frame :name "Frame"}) + data (page-data page-id {uuid/zero (make-root []) + (:id frame) frame}) + migrated (cfm/migrate-data data migration)] + + (t/is (= data migrated) + "empty children vectors are left alone"))) + +(t/deftest leaves-shapes-without-children-key-alone + (let [page-id (uuid/next) + rect-id (uuid/next) + rect (cts/setup-shape {:id rect-id :type :rect :name "Rect"}) + data (page-data page-id {uuid/zero (make-root []) + rect-id rect}) + migrated (cfm/migrate-data data migration)] + + (t/is (= data migrated) + "shapes without a children key are left alone") + (t/is (not (contains? (get-in migrated [:pages-index page-id :objects rect-id]) :shapes)) + "no children key is created on the shape"))) + +(t/deftest migration-is-idempotent + (let [page-id (uuid/next) + child-id (uuid/next) + data (page-data page-id {uuid/zero (make-root [child-id])}) + migrated (cfm/migrate-data data migration)] + + (t/is (= migrated (cfm/migrate-data migrated migration)) + "applying the migration twice does not change the result"))) + +(t/deftest file-with-dangling-children-passes-the-whole-migration + (let [file-id (uuid/next) + page-id (uuid/next) + frame-id (uuid/next) + missing-id (uuid/next) + frame (make-frame frame-id [missing-id]) + base (ctf/make-file-data file-id page-id) + data (-> base + (assoc-in [:pages-index page-id :objects uuid/zero] + (make-root [frame-id missing-id])) + (assoc-in [:pages-index page-id :objects frame-id] frame)) + ;; The file reached the state before 0033 ran, so 0033, 0034 and + ;; 0035 are all pending, exactly as in the reported failure. + file {:id file-id + :data data + :migrations (disj cfm/available-migrations + "0033-normalize-main-instance-names" + "0034-fix-path-content-offset" + migration)}] + + (t/is (cfm/need-migration? file) + "the file is scheduled for migration") + + (let [migrated (cfm/migrate-file file {}) + objects' (get-in migrated [:data :pages-index page-id :objects])] + + (t/is (contains? (:migrations migrated) migration) + "the migration is recorded as applied") + (t/is (not (contains? objects' nil)) + "the migrated data has no nil shape entry") + (t/is (= [frame-id] (get-in objects' [uuid/zero :shapes])) + "the dangling reference of the root is removed") + (t/is (= [] (get-in objects' [frame-id :shapes])) + "the dangling reference of the frame is removed") + (t/is (some? (get objects' frame-id)) + "the frame that exists is kept")))) + +;; ====================================================================== ;; 0035-fix-shape-svg-attrs-and-defs ;; Stored svg attribute keys are kebab-case (issue #11947). Old birth and ;; the cleaner era stored camelCase React prop names, which the v3 reader @@ -22,6 +198,7 @@ ;; through the whitelist+kebab transform, svg-raw `:content` attrs are ;; kebab-ized spelling-only. It replaces "0021-fix-shape-svg-attrs" (id ;; withdrawn): files that recorded it migrate again through this one. +;; ====================================================================== (def ^:private migration-id "0035-fix-shape-svg-attrs-and-defs") diff --git a/common/test/common_tests/files_migrations_0035a_test.cljc b/common/test/common_tests/files_migrations_0035a_test.cljc deleted file mode 100644 index 610d30ca37..0000000000 --- a/common/test/common_tests/files_migrations_0035a_test.cljc +++ /dev/null @@ -1,186 +0,0 @@ -;; This Source Code Form is subject to the terms of the Mozilla Public -;; License, v. 2.0. If a copy of the MPL was not distributed with this -;; file, You can obtain one at http://mozilla.org/MPL/2.0/. -;; -;; Copyright (c) KALEIDOS SUBSIDIARY SL - -(ns common-tests.files-migrations-0035a-test - "Coverage for `0035-fix-dangling-children`. - - A shape can list a child id that is not present in the `:objects` map - of the page or component owning it (a dangling reference). Any - traversal of the shape tree trips on it, and the file becomes - unopenable. The migration drops those ids and keeps every shape that - really exists." - (:require - [app.common.files.migrations :as cfm] - [app.common.types.file :as ctf] - [app.common.types.objects-map :as omap] - [app.common.types.shape :as cts] - [app.common.uuid :as uuid] - [clojure.test :as t])) - -(def ^:private migration "0035-fix-dangling-children") - -(defn- make-root - [children] - (cts/setup-shape {:id uuid/zero - :type :frame - :name "Root" - :shapes children})) - -(defn- make-frame - [frame-id child-ids] - (-> (cts/setup-shape {:id frame-id :type :frame :name "Frame"}) - (assoc :shapes child-ids))) - -(defn- page-data - [page-id objects] - {:pages-index {page-id {:id page-id :objects objects}}}) - -(defn- component-data - [component-id objects] - {:components {component-id {:id component-id :objects objects}}}) - -(t/deftest removes-the-dangling-children-of-a-page - (let [page-id (uuid/next) - frame-id (uuid/next) - child-id (uuid/next) - frame (make-frame frame-id [child-id]) - data (page-data page-id {uuid/zero (make-root [frame-id child-id]) - frame-id frame}) - migrated (cfm/migrate-data data migration) - objects' (get-in migrated [:pages-index page-id :objects])] - - (t/is (= [frame-id] (get-in objects' [uuid/zero :shapes])) - "the dangling id is removed from the root children list") - (t/is (= [] (get-in objects' [frame-id :shapes])) - "the dangling id is removed from the frame children list") - (t/is (= 2 (count objects')) - "every shape that exists is kept") - (t/is (some? (get objects' frame-id)) - "the frame itself is kept"))) - -(t/deftest removes-the-dangling-children-of-a-component - (let [component-id (uuid/next) - frame-id (uuid/next) - child-id (uuid/next) - frame (cts/setup-shape {:id frame-id :type :frame :name "Frame"}) - data (component-data component-id {uuid/zero (make-root [frame-id child-id]) - frame-id frame}) - migrated (cfm/migrate-data data migration) - objects' (get-in migrated [:components component-id :objects])] - - (t/is (= [frame-id] (get-in objects' [uuid/zero :shapes])) - "the dangling id is removed from the children list") - (t/is (= 2 (count objects')) - "every shape that exists is kept") - (t/is (= frame (get objects' frame-id)) - "the existing shape is left untouched"))) - -(t/deftest migration-works-with-an-objects-map - ;; The backend stores the page :objects as an ObjectsMap (the lazy - ;; representation enabled by the fdata/objects-map feature). It is an - ;; IPersistentMap that does not implement IFn on the JVM: a migration - ;; must consult it with contains?/get, never by calling it as a - ;; function. - (let [page-id (uuid/next) - frame-id (uuid/next) - child-id (uuid/next) - frame (make-frame frame-id [child-id]) - objects (omap/wrap {uuid/zero (make-root [frame-id child-id]) - frame-id frame}) - data (page-data page-id objects) - migrated (cfm/migrate-data data migration) - objects' (get-in migrated [:pages-index page-id :objects])] - - (t/is (= [frame-id] (get-in objects' [uuid/zero :shapes])) - "the dangling id is removed with the ObjectsMap representation") - (t/is (= [] (get-in objects' [frame-id :shapes])) - "the frame children list is pruned the same way"))) - -(t/deftest keeps-the-order-and-the-shapes-of-a-valid-tree - (let [page-id (uuid/next) - frame-id (uuid/next) - rect-id (uuid/next) - frame (cts/setup-shape {:id frame-id :type :frame :name "Frame"}) - rect (cts/setup-shape {:id rect-id :type :rect :name "Rect"}) - data (page-data page-id {uuid/zero (make-root [frame-id rect-id]) - frame-id frame - rect-id rect}) - migrated (cfm/migrate-data data migration) - objects' (get-in migrated [:pages-index page-id :objects])] - - (t/is (= [frame-id rect-id] (get-in objects' [uuid/zero :shapes])) - "the children list is kept as it was") - (t/is (= data migrated) - "a tree without dangling references is not modified"))) - -(t/deftest leaves-shapes-with-empty-children-alone - (let [page-id (uuid/next) - frame (cts/setup-shape {:id (uuid/next) :type :frame :name "Frame"}) - data (page-data page-id {uuid/zero (make-root []) - (:id frame) frame}) - migrated (cfm/migrate-data data migration)] - - (t/is (= data migrated) - "empty children vectors are left alone"))) - -(t/deftest leaves-shapes-without-children-key-alone - (let [page-id (uuid/next) - rect-id (uuid/next) - rect (cts/setup-shape {:id rect-id :type :rect :name "Rect"}) - data (page-data page-id {uuid/zero (make-root []) - rect-id rect}) - migrated (cfm/migrate-data data migration)] - - (t/is (= data migrated) - "shapes without a children key are left alone") - (t/is (not (contains? (get-in migrated [:pages-index page-id :objects rect-id]) :shapes)) - "no children key is created on the shape"))) - -(t/deftest migration-is-idempotent - (let [page-id (uuid/next) - child-id (uuid/next) - data (page-data page-id {uuid/zero (make-root [child-id])}) - migrated (cfm/migrate-data data migration)] - - (t/is (= migrated (cfm/migrate-data migrated migration)) - "applying the migration twice does not change the result"))) - -(t/deftest file-with-dangling-children-passes-the-whole-migration - (let [file-id (uuid/next) - page-id (uuid/next) - frame-id (uuid/next) - missing-id (uuid/next) - frame (make-frame frame-id [missing-id]) - base (ctf/make-file-data file-id page-id) - data (-> base - (assoc-in [:pages-index page-id :objects uuid/zero] - (make-root [frame-id missing-id])) - (assoc-in [:pages-index page-id :objects frame-id] frame)) - ;; The file reached the state before 0033 ran, so 0033, 0034 and - ;; 0035 are all pending, exactly as in the reported failure. - file {:id file-id - :data data - :migrations (disj cfm/available-migrations - "0033-normalize-main-instance-names" - "0034-fix-path-content-offset" - migration)}] - - (t/is (cfm/need-migration? file) - "the file is scheduled for migration") - - (let [migrated (cfm/migrate-file file {}) - objects' (get-in migrated [:data :pages-index page-id :objects])] - - (t/is (contains? (:migrations migrated) migration) - "the migration is recorded as applied") - (t/is (not (contains? objects' nil)) - "the migrated data has no nil shape entry") - (t/is (= [frame-id] (get-in objects' [uuid/zero :shapes])) - "the dangling reference of the root is removed") - (t/is (= [] (get-in objects' [frame-id :shapes])) - "the dangling reference of the frame is removed") - (t/is (some? (get objects' frame-id)) - "the frame that exists is kept")))) diff --git a/common/test/common_tests/runner.cljc b/common/test/common_tests/runner.cljc index 5066d6954a..df2662df08 100644 --- a/common/test/common_tests/runner.cljc +++ b/common/test/common_tests/runner.cljc @@ -24,8 +24,7 @@ [common-tests.files-migrations-0026-test] [common-tests.files-migrations-0033-test] [common-tests.files-migrations-0034-test] - [common-tests.files-migrations-0035a-test] - [common-tests.files-migrations-0035b-test] + [common-tests.files-migrations-0035-test] [common-tests.files-migrations-test] [common-tests.files.comp-processors-test] [common-tests.files.helpers-test] @@ -141,8 +140,7 @@ 'common-tests.files-migrations-0026-test 'common-tests.files-migrations-0033-test 'common-tests.files-migrations-0034-test - 'common-tests.files-migrations-0035a-test - 'common-tests.files-migrations-0035b-test + 'common-tests.files-migrations-0035-test 'common-tests.files-migrations-test 'common-tests.files.comp-processors-test 'common-tests.files.helpers-test From b5eab6a949662401bd6c15cfc5b04398f8e93213 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marina=20L=C3=B3pez?= Date: Thu, 12 Nov 2026 12:35:13 +0100 Subject: [PATCH 23/29] :lipstick: Update promotional banner (#12944) --- frontend/resources/images/icons/pencil.svg | 3 + .../src/app/main/ui/dashboard/sidebar.cljs | 2 +- .../app/main/ui/dashboard/subscription.cljs | 66 ++++++++--- .../app/main/ui/dashboard/subscription.scss | 111 +++++++++++++++++- .../main/ui/ds/foundations/assets/icon.cljs | 1 + frontend/translations/en.po | 10 +- frontend/translations/es.po | 10 +- 7 files changed, 174 insertions(+), 29 deletions(-) create mode 100644 frontend/resources/images/icons/pencil.svg diff --git a/frontend/resources/images/icons/pencil.svg b/frontend/resources/images/icons/pencil.svg new file mode 100644 index 0000000000..50dfe16628 --- /dev/null +++ b/frontend/resources/images/icons/pencil.svg @@ -0,0 +1,3 @@ + + + diff --git a/frontend/src/app/main/ui/dashboard/sidebar.cljs b/frontend/src/app/main/ui/dashboard/sidebar.cljs index d9514fb927..ca85e0ea26 100644 --- a/frontend/src/app/main/ui/dashboard/sidebar.cljs +++ b/frontend/src/app/main/ui/dashboard/sidebar.cljs @@ -656,7 +656,7 @@ [:* (if (contains? cf/flags :admin-console) [:* - [:> nitrate-sidebar* {:profile profile :teams teams}] + [:> nitrate-sidebar* {:profile profile :teams teams :team team}] [:> nitrate-current-plan* {:profile profile}]] (when (contains? cf/flags :subscriptions) (if (show-subscription-dashboard-banner? profile) diff --git a/frontend/src/app/main/ui/dashboard/subscription.cljs b/frontend/src/app/main/ui/dashboard/subscription.cljs index c93644d90c..fa675fb9e2 100644 --- a/frontend/src/app/main/ui/dashboard/subscription.cljs +++ b/frontend/src/app/main/ui/dashboard/subscription.cljs @@ -118,8 +118,46 @@ :has-dropdown false :is-highlighted false}])))) +(mf/defc nitrate-enterprise-banner* + [{:keys [on-click button-label]}] + (let [collapsed* (mf/use-state false) + collapsed? (deref collapsed*) + toggle-collapse + (mf/use-fn #(swap! collapsed* not))] + + [:div {:class (stl/css-case :nitrate-banner true + :nitrate-enterprise-banner true + :collapsed collapsed?)} + [:button {:class (stl/css :nitrate-enterprise-toggle) + :type "button" + :aria-controls "nitrate-enterprise-content" + :aria-expanded (not collapsed?) + :on-click toggle-collapse} + [:> icon* {:icon-id i/pencil + :size "m" + :class (stl/css :nitrate-enterprise-pencil) + :aria-hidden true}] + [:span {:class (stl/css :nitrate-enterprise-title)} + (tr "subscription.dashboard.banner.unlock-enterprise")] + [:> icon* {:icon-id i/arrow-down + :size "m" + :class (stl/css :nitrate-enterprise-chevron) + :aria-hidden true}]] + [:div {:id "nitrate-enterprise-content" + :class (stl/css :nitrate-enterprise-panel) + :aria-hidden collapsed?} + [:div {:class (stl/css :nitrate-enterprise-panel-inner)} + [:span {:class (stl/css :nitrate-enterprise-description)} + (tr "subscription.dashboard.banner.unlock-features-description")] + [:> button* {:variant "primary" + :type "button" + :class (stl/css :nitrate-bottom-button) + :on-click on-click + :disabled collapsed?} + button-label]]]])) + (mf/defc nitrate-sidebar* - [{:keys [profile teams]}] + [{:keys [profile teams team]}] (let [nitrate? (dnt/is-valid-license? profile) nitrate-license (:subscription profile) manual-license? (:manual nitrate-license) @@ -151,6 +189,12 @@ subscription-type (if nitrate? (:type nitrate-license) (get-subscription-type (-> profile :props :subscription))) + show-unlock-features? + (and (contains? cf/flags :admin-console) + (not nitrate?) + (contains? #{"professional" "unlimited"} subscription-type) + (nil? (dm/get-in team [:organization :id]))) + team-count (count teams) account-age-days (dnt/account-age-days profile) @@ -210,19 +254,13 @@ :class (stl/css :nitrate-bottom-button) :on-click handle-go-to-cc} (tr "nitrate.activation-success.create-organization")]]] - ;; Banner for users without nitrate license - (when (not nitrate?) - [:div {:class (stl/css :nitrate-banner :highlighted)} - [:div {:class (stl/css :nitrate-content)} - [:span {:class (stl/css :nitrate-title)} (tr "subscription.dashboard.banner.unlock-features")]] - [:div {:class (stl/css :nitrate-content)} - [:span {:class (stl/css :nitrate-info)} (tr "subscription.dashboard.banner.unlock-features-description-text")] - [:> button* {:variant "primary" - :type "button" - :class (stl/css :nitrate-bottom-button) - :on-click handle-click} (if (:subscription profile) - (tr "subscription.dashboard.banner.upgrade-nitrate") - (tr "nitrate.form.try-free"))]]])) + ;; Banner for Professional/Unlimited users outside an organization + (when show-unlock-features? + [:> nitrate-enterprise-banner* + {:on-click handle-click + :button-label (if (:subscription profile) + (tr "subscription.dashboard.banner.upgrade-nitrate") + (tr "nitrate.form.try-free"))}])) ;; Banner for users with nitrate license almost expired or expired (when show-subscription-warning? diff --git a/frontend/src/app/main/ui/dashboard/subscription.scss b/frontend/src/app/main/ui/dashboard/subscription.scss index 4699b44d92..7d0bdcb0f0 100644 --- a/frontend/src/app/main/ui/dashboard/subscription.scss +++ b/frontend/src/app/main/ui/dashboard/subscription.scss @@ -208,25 +208,132 @@ } .nitrate-banner { + --nitrate-enterprise-gradient-start: var(--color-accent-success); + --nitrate-enterprise-gradient-end: var(--color-token-border); + + :global(.light) & { + --nitrate-enterprise-gradient-start: var(--color-background-info); + --nitrate-enterprise-gradient-end: var(--color-accent-primary-muted); + } + display: flex; border-radius: var(--sp-s); flex-direction: column; margin: var(--sp-m) var(--sp-m) 0; - background: var(--color-accent-select); + background: linear-gradient( + 110deg, + var(--nitrate-enterprise-gradient-start), + var(--nitrate-enterprise-gradient-end) + ); border: $b-1 solid var(--color-accent-primary-muted); padding: var(--sp-l); } +.nitrate-enterprise-banner { + display: grid; + grid-template-rows: auto minmax(0, 1fr); + background: linear-gradient( + 110deg, + var(--nitrate-enterprise-gradient-start), + var(--nitrate-enterprise-gradient-end) + ); + transition: grid-template-rows 240ms ease; +} + +.nitrate-enterprise-banner.collapsed { + grid-template-rows: auto minmax(0, 0fr); +} + +.nitrate-enterprise-toggle { + appearance: none; + background: transparent; + border: 0; + color: var(--color-foreground-primary); + cursor: pointer; + display: grid; + grid-template-columns: auto minmax(0, 1fr) auto; + align-items: center; + gap: var(--sp-s); + inline-size: 100%; + padding: 0; + text-align: start; + font: inherit; + + &:focus-visible { + outline: var(--color-accent-primary) solid $b-2; + outline-offset: var(--sp-xxs); + border-radius: $br-4; + } +} + +.nitrate-enterprise-title { + @include use-typography("body-large"); + + color: var(--color-foreground-primary); + font-weight: 600; +} + +.nitrate-enterprise-pencil { + --icon-fill-color: currentcolor; + --icon-stroke-color: none; + + color: var(--color-foreground-primary); +} + +.nitrate-enterprise-chevron { + color: var(--color-foreground-secondary); + transition: transform 240ms ease; +} + +.collapsed .nitrate-enterprise-chevron { + transform: rotate(180deg); +} + +.nitrate-enterprise-panel { + min-block-size: 0; + overflow: hidden; + opacity: 1; + transition: opacity 180ms ease; +} + +.collapsed .nitrate-enterprise-panel { + opacity: 0; + pointer-events: none; +} + +.nitrate-enterprise-panel-inner { + display: flex; + flex-direction: column; + min-block-size: 0; + padding-block-start: var(--sp-s); +} + +.nitrate-enterprise-description { + @include use-typography("body-medium"); + + color: var(--color-foreground-primary); + margin-block-end: var(--sp-l); +} + +@media (prefers-reduced-motion: reduce) { + .nitrate-enterprise-banner, + .nitrate-enterprise-chevron, + .nitrate-enterprise-panel { + transition: none; + } +} + .nitrate-title { @include use-typography("body-large"); color: var(--color-foreground-primary); + font-weight: 600; } .nitrate-info { @include use-typography("body-medium"); - color: var(--color-foreground-secondary); + color: var(--color-foreground-primary); margin-block: var(--sp-s) var(--sp-xxl); } diff --git a/frontend/src/app/main/ui/ds/foundations/assets/icon.cljs b/frontend/src/app/main/ui/ds/foundations/assets/icon.cljs index 6d05480deb..126cb7b4bc 100644 --- a/frontend/src/app/main/ui/ds/foundations/assets/icon.cljs +++ b/frontend/src/app/main/ui/ds/foundations/assets/icon.cljs @@ -223,6 +223,7 @@ (def ^:icon-id padding-top "padding-top") (def ^:icon-id padding-top-bottom "padding-top-bottom") (def ^:icon-id path "path") +(def ^:icon-id pencil "pencil") (def ^:icon-id pentool "pentool") (def ^:icon-id percentage "percentage") (def ^:icon-id picker "picker") diff --git a/frontend/translations/en.po b/frontend/translations/en.po index c0ef6a81ee..48ea44fed3 100644 --- a/frontend/translations/en.po +++ b/frontend/translations/en.po @@ -5803,14 +5803,12 @@ msgid "subscription.dashboard.banner.subscription-expired" msgstr "Your Enterprise subscription has expired. (%s)" #: src/app/main/ui/dashboard/subscription.cljs:217 -msgid "subscription.dashboard.banner.unlock-features" -msgstr "Unlock Enterprise features" +msgid "subscription.dashboard.banner.unlock-enterprise" +msgstr "Unlock Enterprise" #: src/app/main/ui/dashboard/subscription.cljs:219 -msgid "subscription.dashboard.banner.unlock-features-description-text" -msgstr "" -"Set fine-grained permissions and stay in control, without compromising the " -"open-source freedom your team loves." +msgid "subscription.dashboard.banner.unlock-features-description" +msgstr "For organizations that need advanced governance, centralized administration, and secure collaboration." #: src/app/main/ui/dashboard/subscription.cljs:224 msgid "subscription.dashboard.banner.upgrade-nitrate" diff --git a/frontend/translations/es.po b/frontend/translations/es.po index c8506b8466..87d6b89e7d 100644 --- a/frontend/translations/es.po +++ b/frontend/translations/es.po @@ -5853,14 +5853,12 @@ msgid "subscription.dashboard.banner.subscription-expired" msgstr "Tu suscripción Enterprise ha caducado. (%s)" #: src/app/main/ui/dashboard/subscription.cljs:217 -msgid "subscription.dashboard.banner.unlock-features" -msgstr "Desbloquea las funciones Enterprise" +msgid "subscription.dashboard.banner.unlock-enterprise" +msgstr "Desbloquea Enterprise" #: src/app/main/ui/dashboard/subscription.cljs:219 -msgid "subscription.dashboard.banner.unlock-features-description-text" -msgstr "" -"Establece permisos detallados y mantén el control, sin renunciar a la " -"libertad del código abierto que tanto valora tu equipo." +msgid "subscription.dashboard.banner.unlock-features-description" +msgstr "Para organizaciones que necesitan una gobernanza avanzada, una administración centralizada y una colaboración segura." #: src/app/main/ui/dashboard/subscription.cljs:224 msgid "subscription.dashboard.banner.upgrade-nitrate" From 9d42ed29cee2e40d408982ee9c9f326259ea6473 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?andr=C3=A9s=20gonz=C3=A1lez?= Date: Thu, 12 Nov 2026 13:04:13 +0100 Subject: [PATCH 24/29] :books: Document how to order variant properties and values (#12996) Users ask how to choose the order of values in the variant property dropdown. The values follow the order of the variants in the Layers panel, and this was not documented. Also document that properties can be reordered by dragging them in the Design tab. AI-assisted-by: claude-opus-5-5 --- docs/user-guide/design-systems/variants.njk | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/user-guide/design-systems/variants.njk b/docs/user-guide/design-systems/variants.njk index 5b41f0fade..5a6a1ff2e1 100644 --- a/docs/user-guide/design-systems/variants.njk +++ b/docs/user-guide/design-systems/variants.njk @@ -86,6 +86,13 @@ desc: Streamline your design workflow with Penpot's Components guide! Learn to c Edit variant property +
Order properties and values
+
    +
  • Properties: In the Design tab, drag a property by its handle to move it up or down.
  • +
  • Values: The values dropdown follows the order of the variants in the Layers panel, from top to bottom. Each value appears where the first variant that uses it sits. To change the order, reorder the variants in the Layers panel. Empty values always go last.
  • +
+

The variant order applies to all properties at once, so you may not be able to set the value order you want for two properties at the same time.

+
Delete properties
  • From the Design tab: Select the main component (not an individual variant) and press the minus button next to the property.
  • From 029283b1f91574359a7549cfd527824fe047d09e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marina=20L=C3=B3pez?= Date: Thu, 12 Nov 2026 14:54:56 +0100 Subject: [PATCH 25/29] :bug: Hide standalone Enterprise plans for team owners (#13017) --- .../app/main/ui/dashboard/subscription.cljs | 44 ++++++++++++------- frontend/test/frontend_tests/runner.cljs | 1 + .../frontend_tests/ui/subscription_test.cljs | 40 +++++++++++++++++ 3 files changed, 70 insertions(+), 15 deletions(-) create mode 100644 frontend/test/frontend_tests/ui/subscription_test.cljs diff --git a/frontend/src/app/main/ui/dashboard/subscription.cljs b/frontend/src/app/main/ui/dashboard/subscription.cljs index fa675fb9e2..b7434d5279 100644 --- a/frontend/src/app/main/ui/dashboard/subscription.cljs +++ b/frontend/src/app/main/ui/dashboard/subscription.cljs @@ -28,6 +28,18 @@ type "professional")) +(defn get-team-plan-type + [team] + (if (some? (dm/get-in team [:organization :id])) + "enterprise" + (get-subscription-type (:subscription team)))) + +(defn show-team-plan? + [is-owner team] + (not (and is-owner + (nil? (dm/get-in team [:organization :id])) + (= "enterprise" (get-subscription-type (:subscription team)))))) + (mf/defc cta-power-up* [{:keys [top-title top-description bottom-description bottom-button bottom-button-href has-dropdown is-highlighted]}] (let [show-data* (mf/use-state false) @@ -310,6 +322,7 @@ [{:keys [is-owner team]}] (let [subscription (:subscription team) subscription-type (get-subscription-type subscription) + team-plan-type (get-team-plan-type team) subscription-is-trial (= "trialing" (:status subscription)) go-to-manage-subscription @@ -324,22 +337,23 @@ href (str "payments/subscriptions/show?returnUrl=" href)] (st/emit! (rt/nav-raw :href href)))))] - [:div {:class (stl/css :team)} - [:div {:class (stl/css :team-label)} - (tr "subscription.dashboard.team-plan")] - [:span {:class (stl/css :team-text)} - (case subscription-type - "professional" (tr "subscription.settings.professional") - "unlimited" (if subscription-is-trial - (tr "subscription.settings.unlimited-trial") - (tr "subscription.settings.unlimited")) + (when (show-team-plan? is-owner team) + [:div {:class (stl/css :team)} + [:div {:class (stl/css :team-label)} + (tr "subscription.dashboard.team-plan")] + [:span {:class (stl/css :team-text)} + (case team-plan-type + "professional" (tr "subscription.settings.professional") + "unlimited" (if subscription-is-trial + (tr "subscription.settings.unlimited-trial") + (tr "subscription.settings.unlimited")) - "enterprise" (tr "subscription.settings.enterprise"))] - (when (and is-owner (not= subscription-type "professional")) - [:button {:class (stl/css :manage-subscription-link) - :on-click go-to-manage-subscription - :data-testid "manage-subscription-link"} - (tr "subscription.settings.manage-your-subscription")])])) + "enterprise" (tr "subscription.settings.enterprise"))] + (when (and is-owner (not= subscription-type "professional")) + [:button {:class (stl/css :manage-subscription-link) + :on-click go-to-manage-subscription + :data-testid "manage-subscription-link"} + (tr "subscription.settings.manage-your-subscription")])]))) (mf/defc menu-team-icon* [{:keys [subscription-type]}] diff --git a/frontend/test/frontend_tests/runner.cljs b/frontend/test/frontend_tests/runner.cljs index b0f699a220..38f5975de3 100644 --- a/frontend/test/frontend_tests/runner.cljs +++ b/frontend/test/frontend_tests/runner.cljs @@ -169,6 +169,7 @@ [frontend-tests.ui.sortable-auto-scroll-test] [frontend-tests.ui.static-test] [frontend-tests.ui.stroke-menu-test] + [frontend-tests.ui.subscription-test] [frontend-tests.ui.text-attrs-multiple-test] [frontend-tests.ui.viewport-top-bar-test] [frontend-tests.ui.workspace-libraries-test] diff --git a/frontend/test/frontend_tests/ui/subscription_test.cljs b/frontend/test/frontend_tests/ui/subscription_test.cljs new file mode 100644 index 0000000000..7517720806 --- /dev/null +++ b/frontend/test/frontend_tests/ui/subscription_test.cljs @@ -0,0 +1,40 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns frontend-tests.ui.subscription-test + (:require + [app.main.ui.dashboard.subscription :as ds] + [cljs.test :as t :include-macros true])) + +(t/deftest team-plan-is-enterprise-while-team-belongs-to-an-organization + (doseq [owner-plan ["professional" "unlimited"]] + (t/is (= "enterprise" + (ds/get-team-plan-type {:organization {:id "org-1"} + :subscription {:type owner-plan + :status "active"}}))))) + +(t/deftest team-plan-returns-to-owner-plan-after-leaving-organization + (doseq [owner-plan ["professional" "unlimited"]] + (t/is (= owner-plan + (ds/get-team-plan-type {:subscription {:type owner-plan + :status "active"}}))))) + +(t/deftest owner-enterprise-plan-is-hidden-for-team-outside-organization + (t/is (false? (ds/show-team-plan? true + {:subscription {:type "enterprise" + :status "active"}})))) + +(t/deftest team-plan-is-visible-for-other-owner-and-organization-cases + (t/is (true? (ds/show-team-plan? true + {:subscription {:type "professional" + :status "active"}}))) + (t/is (true? (ds/show-team-plan? true + {:subscription {:type "enterprise" + :status "active"} + :organization {:id "org-1"}}))) + (t/is (true? (ds/show-team-plan? false + {:subscription {:type "enterprise" + :status "active"}})))) From 5803f2594c10b6164367c9aec119c611d49d9dca Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marina=20L=C3=B3pez?= Date: Thu, 12 Nov 2026 15:11:43 +0100 Subject: [PATCH 26/29] :bug: Show the organization in the new team form (#13023) --- .../check-nitrate-sso-authorized.json | 3 + .../get-teams-organization-switcher.json | 91 +++++++++++++++++++ .../ui/specs/dashboard-teams.spec.js | 63 +++++++++++++ .../src/app/main/ui/dashboard/team_form.cljs | 6 +- .../src/app/main/ui/dashboard/team_form.scss | 3 + frontend/translations/en.po | 4 + frontend/translations/es.po | 4 + 7 files changed, 172 insertions(+), 2 deletions(-) create mode 100644 frontend/playwright/data/dashboard/check-nitrate-sso-authorized.json create mode 100644 frontend/playwright/data/dashboard/get-teams-organization-switcher.json diff --git a/frontend/playwright/data/dashboard/check-nitrate-sso-authorized.json b/frontend/playwright/data/dashboard/check-nitrate-sso-authorized.json new file mode 100644 index 0000000000..4adb4d785d --- /dev/null +++ b/frontend/playwright/data/dashboard/check-nitrate-sso-authorized.json @@ -0,0 +1,3 @@ +{ + "~:authorized": true +} diff --git a/frontend/playwright/data/dashboard/get-teams-organization-switcher.json b/frontend/playwright/data/dashboard/get-teams-organization-switcher.json new file mode 100644 index 0000000000..91b9c2a20e --- /dev/null +++ b/frontend/playwright/data/dashboard/get-teams-organization-switcher.json @@ -0,0 +1,91 @@ +[ + { + "~:features": { + "~#set": [ + "layout/grid", + "styles/v2", + "fdata/pointer-map", + "fdata/objects-map", + "components/v2", + "fdata/shape-data-type" + ] + }, + "~:permissions": { + "~:type": "~:membership", + "~:is-owner": true, + "~:is-admin": true, + "~:can-edit": true + }, + "~:name": "Brand Studio", + "~:modified-at": "~m1713533116375", + "~:id": "~uc7ce0794-0992-8105-8004-38e630f40f6d", + "~:created-at": "~m1713533116375", + "~:is-default": true, + "~:organization": { + "~:id": "~u11111111-1111-4111-8111-111111111111", + "~:name": "Brand Studio", + "~:slug": "brand-studio", + "~:owner-id": "~u22222222-2222-4222-8222-222222222222", + "~:avatar-bg-url": null, + "~:permissions": { + "~:create-teams": "any" + } + } + }, + { + "~:features": { + "~#set": [ + "layout/grid", + "styles/v2", + "fdata/pointer-map", + "fdata/objects-map", + "components/v2", + "fdata/shape-data-type" + ] + }, + "~:permissions": { + "~:type": "~:membership", + "~:is-owner": false, + "~:is-admin": false, + "~:can-edit": true + }, + "~:name": "Product", + "~:modified-at": "~m1713533116375", + "~:id": "~udd33ff88-f4e5-8033-8003-8096cc07bdf3", + "~:created-at": "~m1713533116375", + "~:is-default": true, + "~:organization": { + "~:id": "~u33333333-3333-4333-8333-333333333333", + "~:name": "Product Studio With a Long Organization Name That Must Remain Available", + "~:slug": "product-studio", + "~:owner-id": "~u44444444-4444-4444-8444-444444444444", + "~:avatar-bg-url": null, + "~:permissions": { + "~:create-teams": "any" + } + } + }, + { + "~:features": { + "~#set": [ + "layout/grid", + "styles/v2", + "fdata/pointer-map", + "fdata/objects-map", + "components/v2", + "fdata/shape-data-type" + ] + }, + "~:permissions": { + "~:type": "~:membership", + "~:is-owner": true, + "~:is-admin": true, + "~:can-edit": true + }, + "~:name": "Personal projects", + "~:modified-at": "~m1713533116375", + "~:id": "~ue3c2fd6d-2d3e-47b3-87f6-41b467dff123", + "~:created-at": "~m1713533116375", + "~:is-default": true + } +] diff --git a/frontend/playwright/ui/specs/dashboard-teams.spec.js b/frontend/playwright/ui/specs/dashboard-teams.spec.js index 31153ac4cb..860a6aad18 100644 --- a/frontend/playwright/ui/specs/dashboard-teams.spec.js +++ b/frontend/playwright/ui/specs/dashboard-teams.spec.js @@ -24,3 +24,66 @@ test("BUG 12359 - Selected invitations count is not pluralized", async ({ await page.getByText("test2@mail.com").check(); await expect(page.getByText("2 invitations selected")).toBeVisible(); }); + +async function setupOrganizationTeamSwitcher(page) { + await DashboardPage.mockConfigFlags(page, [ + "disable-onboarding", + "enable-admin-console", + ]); + await DashboardPage.mockRPC( + page, + "get-teams", + "dashboard/get-teams-organization-switcher.json", + ); + await DashboardPage.mockRPC( + page, + "check-nitrate-sso", + "dashboard/check-nitrate-sso-authorized.json", + ); + + const dashboardPage = new DashboardPage(page); + await dashboardPage.goToDashboard(); + return dashboardPage; +} + +test("User sees the destination organization when creating a team", async ({ + page, +}) => { + const organizationName = + "Product Studio With a Long Organization Name That Must Remain Available"; + const dashboardPage = await setupOrganizationTeamSwitcher(page); + + await dashboardPage.teamDropdown.click(); + await page.getByRole("menuitem", { name: organizationName }).click(); + await page.getByRole("menuitem", { name: "Create new team" }).click(); + + const title = page.getByRole("heading", { + name: `Create new team in ${organizationName}`, + exact: true, + }); + await expect(title).toBeVisible(); + await expect(title).toHaveText(`Create new team in ${organizationName}`); + + const titleFits = await title.evaluate( + (element) => element.scrollWidth <= element.clientWidth, + ); + expect(titleFits).toBe(true); + + const modal = title.locator("xpath=../../.."); + const modalBounds = await modal.boundingBox(); + expect(modalBounds.width).toBeLessThanOrEqual(512); +}); + +test("Creating a team under Other teams does not name an organization", async ({ + page, +}) => { + const dashboardPage = await setupOrganizationTeamSwitcher(page); + + await dashboardPage.teamDropdown.click(); + await page.getByRole("menuitem", { name: "Other teams" }).click(); + await page.getByRole("menuitem", { name: "Create new team" }).click(); + + await expect( + page.getByRole("heading", { name: "Create new team", exact: true }), + ).toBeVisible(); +}); diff --git a/frontend/src/app/main/ui/dashboard/team_form.cljs b/frontend/src/app/main/ui/dashboard/team_form.cljs index 08f6dcaad6..c1b45cde76 100644 --- a/frontend/src/app/main/ui/dashboard/team_form.cljs +++ b/frontend/src/app/main/ui/dashboard/team_form.cljs @@ -83,7 +83,7 @@ (mf/defc team-form-modal {::mf/register modal/components ::mf/register-as :team-form} - [{:keys [team organization-id] :as props}] + [{:keys [team organization-id organization-name]}] (let [initial (mf/use-memo (mf/deps team organization-id) (fn [] @@ -117,7 +117,9 @@ [:h2 {:class (stl/css :modal-title)} (tr "labels.rename-team")] [:h2 {:class (stl/css :modal-title)} - (tr "labels.create-team")]) + (if organization-name + (tr "dashboard.create-team-in-organization" organization-name) + (tr "labels.create-team"))]) [:> icon-button* {:variant "ghost" :tooltip-class (stl/css :modal-close-btn) diff --git a/frontend/src/app/main/ui/dashboard/team_form.scss b/frontend/src/app/main/ui/dashboard/team_form.scss index 1047c8f3d8..2cf4583bfd 100644 --- a/frontend/src/app/main/ui/dashboard/team_form.scss +++ b/frontend/src/app/main/ui/dashboard/team_form.scss @@ -26,6 +26,9 @@ @include use-typography("headline-large"); color: var(--color-foreground-primary); + max-inline-size: 100%; + padding-inline-end: var(--sp-xxl); + overflow-wrap: anywhere; } // Tooltip wrapper of the close button: it places it in the corner. diff --git a/frontend/translations/en.po b/frontend/translations/en.po index 48ea44fed3..5f4fc3b975 100644 --- a/frontend/translations/en.po +++ b/frontend/translations/en.po @@ -376,6 +376,10 @@ msgstr "Create new organization" msgid "dashboard.create-new-team" msgstr "Create new team" +#: src/app/main/ui/dashboard/team_form.cljs:122 +msgid "dashboard.create-team-in-organization" +msgstr "Create new team in %s" + #: src/app/main/ui/workspace/main_menu.cljs:712 msgid "dashboard.create-version-menu" msgstr "Pin this version" diff --git a/frontend/translations/es.po b/frontend/translations/es.po index 87d6b89e7d..068dee9cf8 100644 --- a/frontend/translations/es.po +++ b/frontend/translations/es.po @@ -382,6 +382,10 @@ msgstr "Crear nueva organización" msgid "dashboard.create-new-team" msgstr "Crear nuevo equipo" +#: src/app/main/ui/dashboard/team_form.cljs:122 +msgid "dashboard.create-team-in-organization" +msgstr "Crea un nuevo equipo en %s" + #: src/app/main/ui/workspace/main_menu.cljs:712 msgid "dashboard.create-version-menu" msgstr "Guardar esta versión" From dc38eaa843f0c5d0d419bd35f510cca98b4e12e8 Mon Sep 17 00:00:00 2001 From: Andrey Antukh Date: Thu, 12 Nov 2026 15:29:30 +0100 Subject: [PATCH 27/29] :bug: Fix font upload crash and preserve font metadata (#11951) * :bug: Fix font upload crash and preserve variant metadata opentype.js 2.0.0 returns no fontSubfamily name for fonts whose Macintosh name table only holds custom records. That made parse-font-weight/parse-font-style call re-seq on nil and crash the upload. A nil or blank variant is now treated as no match in app.common.media, defaulting to weight 400 and style normal. The variant parsing also accepts oblique as italic, and the new parse-font-family helper derives the family from a filename by stripping the known weight/style tokens, separating a style token glued to a weight token ("Roboto-BoldItalic" -> "Roboto"). first-nonblank-string returns the first nonblank candidate, so a blank preferredFamily/preferredSubfamily no longer shadows a valid fontFamily/fontSubfamily. app.main.data.fonts gains resolve-font-metadata and prepare-font-entry. opentype.js values win per field; any missing value falls back to the filename-derived family, weight and style. The parsed and woff2 branches share the same helper, so metadata is preserved instead of collapsing every face to weight 400, and fonts with missing vertical-metric tables upload with no height warning instead of failing. process-upload reports per-file failures (unknown signature, unparseable file, preparation error) with a single bad-font notification instead of silently dropping them, and never errors for per-file failures. The upload now sends :variant-name to create-font-variant, which accepts it as an optional param (ships in 2.20), so the foundry name survives the upload. Closes #10436 AI-assisted-by: deepseek-v4.1-flash AI-assisted-by: muse-spark-1.3-contributor Signed-off-by: CHOI JUNSOO * :bug: Parse glued font style tokens in the font parsers EvaMarco's review on #11951: parse-font-family and parse-font-weight handled glued tokens like "BoldItalic", but parse-font-style did not. For names such as "opensans-bolditalic-webfont.woff2" the three parsers disagreed: the italic face resolved to style normal and collided with the bold-only file on the (family, weight, style) join key. One binary silently overwrote the other during upload, losing one font face without any warning. A new private normalize-font-name helper is applied by all three parsers. It splits a style token glued to a weight token, now with a lookahead boundary so "LightItalic_v2" splits too, and strips a trailing technical suffix (-VF or _v). parse-font-family, parse-font-weight and parse-font-style now interpret the name consistently. Tests cover the three cases from the review, the bold-only control case, and an end-to-end process-upload check that a bold and a bold-italic WOFF2 survive as two variants. Relates to #10436 and PR #11951. AI-assisted-by: inkling-2-small-free * Remove duplicate frontend test registration --------- Signed-off-by: CHOI JUNSOO --- backend/src/app/rpc/commands/fonts.clj | 4 +- backend/test/backend_tests/rpc_font_test.clj | 45 +++++ common/src/app/common/media.cljc | 84 ++++++-- common/test/common_tests/media_test.cljc | 73 +++++++ frontend/src/app/main/data/fonts.cljs | 182 +++++++++--------- .../test/frontend_tests/data/fonts_test.cljs | 158 +++++++++++++++ frontend/test/frontend_tests/fonts_test.cljs | 42 +++- frontend/test/frontend_tests/runner.cljs | 2 + 8 files changed, 478 insertions(+), 112 deletions(-) create mode 100644 frontend/test/frontend_tests/data/fonts_test.cljs diff --git a/backend/src/app/rpc/commands/fonts.clj b/backend/src/app/rpc/commands/fonts.clj index 46aa009e20..0b5311867a 100644 --- a/backend/src/app/rpc/commands/fonts.clj +++ b/backend/src/app/rpc/commands/fonts.clj @@ -112,6 +112,7 @@ [:font-family types.font/schema:font-family] [:font-weight [::sm/one-of {:format "number"} valid-weight]] [:font-style [::sm/one-of {:format "string"} valid-style]] + [:variant-name {:optional true} [:maybe [::sm/text {:max 250}]]] [:uploads [:map-of ::sm/text ::sm/uuid]]]) (defn- prepare-font-data-from-uploads @@ -139,7 +140,8 @@ chunked-upload API)." {::doc/added "1.18" ::doc/changes [["2.16" "Add :uploads param for chunked upload support"] - ["2.18" "Remove :data param, use :uploads exclusively"]] + ["2.18" "Remove :data param, use :uploads exclusively"] + ["2.20" "Add optional :variant-name param"]] ::climit/id [[:process-font/by-profile ::rpc/profile-id] [:process-font/global]] ::webhooks/event? true diff --git a/backend/test/backend_tests/rpc_font_test.clj b/backend/test/backend_tests/rpc_font_test.clj index 4ef4533f7a..4526676ae7 100644 --- a/backend/test/backend_tests/rpc_font_test.clj +++ b/backend/test/backend_tests/rpc_font_test.clj @@ -403,6 +403,51 @@ (t/is (nil? (:error out))) (assert-font-variant-result params (:result out))))) +(t/deftest create-font-variant-persists-variant-name + "The optional :variant-name is stored and returned by get-font-variants." + (with-mocks [_mock {:target 'app.rpc.quotes/check! :return nil}] + (let [prof (th/create-profile* 1 {:is-active true}) + team-id (:default-team-id prof) + font-id (uuid/custom 10 34) + font-bytes (-> (io/resource "backend_tests/test_files/font-1.ttf") (io/read*)) + session-id (upload-font-chunked! prof font-bytes "font/ttf" (* 4 1024 1024)) + out (th/command! {::th/type :create-font-variant + ::rpc/profile-id (:id prof) + :team-id team-id + :font-id font-id + :font-family "variant-name-font" + :font-weight 600 + :font-style "normal" + :variant-name "SemiBold" + :uploads {"font/ttf" session-id}})] + (t/is (th/success? out)) + (t/is (= "SemiBold" (-> out :result :variant-name))) + + (let [variants (-> (th/command! {::th/type :get-font-variants + ::rpc/profile-id (:id prof) + :team-id team-id}) + :result) + variant (first (filter #(= font-id (:font-id %)) variants))] + (t/is (= "SemiBold" (:variant-name variant))))) + + ;; An explicit nil must be accepted (the param is optional). + (let [prof (th/create-profile* 2 {:is-active true}) + team-id (:default-team-id prof) + font-id (uuid/custom 10 35) + font-bytes (-> (io/resource "backend_tests/test_files/font-1.ttf") (io/read*)) + session-id (upload-font-chunked! prof font-bytes "font/ttf" (* 4 1024 1024)) + out (th/command! {::th/type :create-font-variant + ::rpc/profile-id (:id prof) + :team-id team-id + :font-id font-id + :font-family "variant-name-nil" + :font-weight 400 + :font-style "normal" + :variant-name nil + :uploads {"font/ttf" session-id}})] + (t/is (th/success? out)) + (t/is (nil? (-> out :result :variant-name)))))) + ;; ----------------------------------------------------------------------- ;; Error cases ;; ----------------------------------------------------------------------- diff --git a/common/src/app/common/media.cljc b/common/src/app/common/media.cljc index a82759a7bd..f881684cbe 100644 --- a/common/src/app/common/media.cljc +++ b/common/src/app/common/media.cljc @@ -98,36 +98,90 @@ (let [image-extensions-re #"(\.png)|(\.jpg)|(\.jpeg)|(\.webp)|(\.gif)|(\.svg)$"] (str/replace filename image-extensions-re ""))) +(def ^:private font-variant-tokens + "Weight and style tokens stripped from a font name to derive its family. + Keep the alternatives in sync with `parse-font-weight` and + `parse-font-style`." + ["extra\\s*black" "ultra\\s*black" "extra\\s*bold" "ultra\\s*bold" + "semi\\s*bold" "demi\\s*bold" "extra\\s*light" "ultra\\s*light" + "hairline" "thin" "light" "normal" "regular" "medium" "bold" "black" + "heavy" "solid" "italic" "oblique"]) + +(def ^:private font-variant-tokens-re + ;; The trailing boundary is a lookahead, so two tokens separated by a single + ;; space are both stripped in one pass (the separator is not consumed). + (re-pattern (str "(?i)(^|[-_\\s])(" (str/join "|" font-variant-tokens) ")(?=[-_\\s]|$)"))) + +(def ^:private font-concatenated-style-re + ;; Separate a style token glued to a weight token, e.g. "BoldItalic" -> + ;; "Bold Italic". Only a known weight token followed by a style token is + ;; split, so families such as "OpenSans" are left untouched. The trailing + ;; boundary is a lookahead, so a glued style token followed by a technical + ;; suffix (e.g. "Italic_v2") is split too. + #"(?i)(black|bold|heavy|solid|medium|light|thin|hairline|regular|normal)(italic|oblique)(?=[-_\s]|$)") + +(def ^:private font-technical-suffix-re + ;; Trailing foundry/build suffixes, e.g. "Roboto-Bold-VF" or "Inter_v2". + ;; Requires a separator; stripped before token detection so the parsers + ;; interpret the name the same way. + #"(?i)[-_\s]+(?:vf|v[0-9]+)$") + +(defn- normalize-font-name + "Normalize a font name or variant string before token detection: + split a style token glued to a weight token, then strip a trailing + technical suffix. Applied by `parse-font-family`, `parse-font-weight` + and `parse-font-style` so all three interpret the name the same way." + [s] + (-> s + (str/replace font-concatenated-style-re "$1 $2") + (str/replace font-technical-suffix-re ""))) + +(defn parse-font-family + "Derive a font family name from a base name (a filename without its + extension), stripping the known weight/style tokens." + [base-name] + (let [normalized (normalize-font-name base-name) + stripped (-> normalized + (str/replace font-variant-tokens-re "$1") + (str/replace #"[-_\s]+" " ") + (str/trim))] + (if (str/blank? stripped) base-name stripped))) + (defn parse-font-weight "Parses a font variant/subfamily string into a numeric weight. A nil or non-string variant means the font carries no subfamily metadata, so it defaults to 400 (regular), the same as a blank string." [variant] - (let [variant (if (string? variant) variant "")] + (let [variant (normalize-font-name (if (string? variant) variant ""))] (cond - (re-seq #"(?i)(?:^|[-_\s])(hairline|thin)(?=(?:[-_\s]|$|italic\b))" variant) 100 - (re-seq #"(?i)(?:^|[-_\s])(extra\s*light|ultra\s*light)(?=(?:[-_\s]|$|italic\b))" variant) 200 - (re-seq #"(?i)(?:^|[-_\s])(light)(?=(?:[-_\s]|$|italic\b))" variant) 300 - (re-seq #"(?i)(?:^|[-_\s])(normal|regular)(?=(?:[-_\s]|$|italic\b))" variant) 400 - (re-seq #"(?i)(?:^|[-_\s])(medium)(?=(?:[-_\s]|$|italic\b))" variant) 500 - (re-seq #"(?i)(?:^|[-_\s])(semi\s*bold|demi\s*bold)(?=(?:[-_\s]|$|italic\b))" variant) 600 - (re-seq #"(?i)(?:^|[-_\s])(extra\s*bold|ultra\s*bold)(?=(?:[-_\s]|$|italic\b))" variant) 800 - (re-seq #"(?i)(?:^|[-_\s])(bold)(?=(?:[-_\s]|$|italic\b))" variant) 700 - (re-seq #"(?i)(?:^|[-_\s])(extra\s*black|ultra\s*black)(?=(?:[-_\s]|$|italic\b))" variant) 950 - (re-seq #"(?i)(?:^|[-_\s])(black|heavy|solid)(?=(?:[-_\s]|$|italic\b))" variant) 900 - :else 400))) + (re-seq #"(?i)(?:^|[-_\s])(hairline|thin)(?=(?:[-_\s]|$|italic\b|oblique\b))" variant) 100 + (re-seq #"(?i)(?:^|[-_\s])(extra\s*light|ultra\s*light)(?=(?:[-_\s]|$|italic\b|oblique\b))" variant) 200 + (re-seq #"(?i)(?:^|[-_\s])(light)(?=(?:[-_\s]|$|italic\b|oblique\b))" variant) 300 + (re-seq #"(?i)(?:^|[-_\s])(normal|regular)(?=(?:[-_\s]|$|italic\b|oblique\b))" variant) 400 + (re-seq #"(?i)(?:^|[-_\s])(medium)(?=(?:[-_\s]|$|italic\b|oblique\b))" variant) 500 + (re-seq #"(?i)(?:^|[-_\s])(semi\s*bold|demi\s*bold)(?=(?:[-_\s]|$|italic\b|oblique\b))" variant) 600 + (re-seq #"(?i)(?:^|[-_\s])(extra\s*bold|ultra\s*bold)(?=(?:[-_\s]|$|italic\b|oblique\b))" variant) 800 + (re-seq #"(?i)(?:^|[-_\s])(bold)(?=(?:[-_\s]|$|italic\b|oblique\b))" variant) 700 + (re-seq #"(?i)(?:^|[-_\s])(extra\s*black|ultra\s*black)(?=(?:[-_\s]|$|italic\b|oblique\b))" variant) 950 + (re-seq #"(?i)(?:^|[-_\s])(black|heavy|solid)(?=(?:[-_\s]|$|italic\b|oblique\b))" variant) 900 + :else 400))) (defn parse-font-style "Parses a font variant/subfamily string into \"italic\" or \"normal\". A nil or non-string variant means the font carries no subfamily metadata, so it defaults to \"normal\"." [variant] - (let [variant (if (string? variant) variant "")] - (if (or (re-seq #"(?i)(?:^|[-_\s])(italic)(?:[-_\s]|$)" variant) - (re-seq #"(?i)italic$" variant)) + (let [variant (normalize-font-name (if (string? variant) variant ""))] + (if (or (re-seq #"(?i)(?:^|[-_\s])(italic|oblique)(?:[-_\s]|$)" variant) + (re-seq #"(?i)(?:italic|oblique)$" variant)) "italic" "normal"))) +(defn first-nonblank-string + "Return the first nonblank string from the candidates, or nil." + [& candidates] + (first (filter #(and (string? %) (not (str/blank? %))) candidates))) + (defn font-weight->name [weight] (case (long weight) diff --git a/common/test/common_tests/media_test.cljc b/common/test/common_tests/media_test.cljc index f94525cae7..18ad0fc9fd 100644 --- a/common/test/common_tests/media_test.cljc +++ b/common/test/common_tests/media_test.cljc @@ -37,6 +37,11 @@ (t/is (= 400 (media/parse-font-weight "Thinker"))) (t/is (= 400 (media/parse-font-weight "Mediaeval")))) + (t/testing "matches a weight glued to a style token" + (t/is (= 700 (media/parse-font-weight "Roboto-BoldItalic"))) + (t/is (= 700 (media/parse-font-weight "Roboto-BoldOblique"))) + (t/is (= 600 (media/parse-font-weight "Roboto-SemiBoldItalic")))) + (t/testing "defaults to regular when the variant metadata is missing" (t/is (= 400 (media/parse-font-weight nil))) (t/is (= 400 (media/parse-font-weight ""))) @@ -54,11 +59,79 @@ (t/is (= "normal" (media/parse-font-style "Italica"))) (t/is (= "normal" (media/parse-font-style "Roboto-Regular")))) + (t/testing "treats oblique as italic" + (t/is (= "italic" (media/parse-font-style "Roboto-Oblique"))) + (t/is (= "italic" (media/parse-font-style "Roboto-BoldOblique"))) + (t/is (= "italic" (media/parse-font-style "RobotoOblique")))) + (t/testing "defaults to normal when the variant metadata is missing" (t/is (= "normal" (media/parse-font-style nil))) (t/is (= "normal" (media/parse-font-style ""))) (t/is (= "normal" (media/parse-font-style " "))))) +(t/deftest test-parse-font-family + (t/testing "strips weight and style tokens" + (t/is (= "Roboto" (media/parse-font-family "Roboto-Bold"))) + (t/is (= "Roboto" (media/parse-font-family "Roboto_Bold"))) + (t/is (= "Roboto" (media/parse-font-family "Roboto Bold"))) + (t/is (= "Roboto" (media/parse-font-family "Roboto-Oblique"))) + (t/is (= "OpenSans" (media/parse-font-family "OpenSans-SemiBold"))) + (t/is (= "Lato" (media/parse-font-family "Lato-Light")))) + + (t/testing "separates a style token glued to a weight token" + (t/is (= "Roboto" (media/parse-font-family "Roboto-BoldItalic"))) + (t/is (= "Roboto" (media/parse-font-family "Roboto-SemiBoldItalic"))) + (t/is (= "Roboto" (media/parse-font-family "Roboto-LightOblique"))) + (t/is (= "Roboto" (media/parse-font-family "Roboto-BlackOblique")))) + + (t/testing "does not strip tokens embedded in words" + (t/is (= "Boldini" (media/parse-font-family "Boldini-Bold"))) + (t/is (= "Lighthaus" (media/parse-font-family "Lighthaus"))) + (t/is (= "Blackwood" (media/parse-font-family "Blackwood"))) + (t/is (= "Mediaeval" (media/parse-font-family "Mediaeval")))) + + (t/testing "keeps unknown tokens in the family" + (t/is (= "Satoshi Variable" (media/parse-font-family "Satoshi-Variable")))) + + (t/testing "falls back to the base name when everything is stripped" + (t/is (= "BoldItalic" (media/parse-font-family "BoldItalic"))))) + +(t/deftest test-font-glued-tokens + (t/testing "a glued style token is recognized consistently by all three parsers" + ;; From EvaMarco's review on #11951: without a shared interpretation of + ;; the name, the bold-italic face collided with the bold-only one and + ;; was silently overwritten on upload. + (t/is (= "opensans webfont" (media/parse-font-family "opensans-bolditalic-webfont"))) + (t/is (= 700 (media/parse-font-weight "opensans-bolditalic-webfont"))) + (t/is (= "italic" (media/parse-font-style "opensans-bolditalic-webfont")))) + + (t/testing "a glued style token followed by a technical suffix" + (t/is (= "Roboto" (media/parse-font-family "Roboto-BoldItalic-VF"))) + (t/is (= 700 (media/parse-font-weight "Roboto-BoldItalic-VF"))) + (t/is (= "italic" (media/parse-font-style "Roboto-BoldItalic-VF")))) + + (t/testing "numeric build suffix after a glued style token" + (t/is (= "Roboto" (media/parse-font-family "Roboto-LightItalic_v2"))) + (t/is (= 300 (media/parse-font-weight "Roboto-LightItalic_v2"))) + (t/is (= "italic" (media/parse-font-style "Roboto-LightItalic_v2")))) + + (t/testing "the bold-only control case is unchanged" + (t/is (= "opensans webfont" (media/parse-font-family "opensans-bold-webfont"))) + (t/is (= 700 (media/parse-font-weight "opensans-bold-webfont"))) + (t/is (= "normal" (media/parse-font-style "opensans-bold-webfont"))))) + +(t/deftest test-first-nonblank-string + (t/testing "returns the first nonblank string" + (t/is (= "Bold" (media/first-nonblank-string " " "Bold"))) + (t/is (= "Bold" (media/first-nonblank-string "Bold" "Light"))) + (t/is (= "Bold" (media/first-nonblank-string nil "Bold"))) + (t/is (= "Inter" (media/first-nonblank-string "" "Inter")))) + + (t/testing "returns nil when every candidate is blank or not a string" + (t/is (nil? (media/first-nonblank-string nil " "))) + (t/is (nil? (media/first-nonblank-string))) + (t/is (nil? (media/first-nonblank-string 42 " "))))) + (t/deftest test-strip-image-extension (t/testing "removes extension from supported image files" (t/is (= (media/strip-image-extension "foo.png") "foo")) diff --git a/frontend/src/app/main/data/fonts.cljs b/frontend/src/app/main/data/fonts.cljs index dca3c54263..ec67e752a6 100644 --- a/frontend/src/app/main/data/fonts.cljs +++ b/frontend/src/app/main/data/fonts.cljs @@ -94,7 +94,7 @@ can assemble the chunks and materialise the final font-variant record. Returns an observable that emits the created font-variant." - [{:keys [data team-id font-id font-family font-weight font-style] :as _item}] + [{:keys [data team-id font-id font-family font-weight font-style variant-name] :as _item}] ;; Upload each mtype as a separate chunked session in parallel, collect ;; all [mtype session-id] pairs, then call create-font-variant with :uploads. (->> (rx/from (seq data)) @@ -108,12 +108,100 @@ {}) (rx/mapcat (fn [uploads] (rp/cmd! :create-font-variant - {:team-id team-id - :font-id font-id - :font-family font-family - :font-weight font-weight - :font-style font-style - :uploads uploads}))))) + {:team-id team-id + :font-id font-id + :font-family font-family + :font-weight font-weight + :font-style font-style + :variant-name variant-name + :uploads uploads}))))) + +(defn resolve-font-metadata + "Resolve the upload metadata for a font file. + + `names` is a map carrying the `:family` and `:variant` values read by + opentype.js; each may be nil when the font has no usable name records. + `filename` is the original file name. The opentype.js values win per + field; any missing value is derived from the filename instead." + [names filename] + (let [{:keys [family variant]} names + base-name (str/replace filename #"\.[^.]+$" "") + variant (not-empty (str/trim (or variant ""))) + family (not-empty (str/trim (or family ""))) + variant-source (or variant base-name)] + {:font-family (or family (cm/parse-font-family base-name)) + :font-weight (cm/parse-font-weight variant-source) + :font-style (cm/parse-font-style variant-source) + :variant-name variant})) + +(defn prepare-font-entry + "Prepare a single read font entry for upload. + + `:font` is the object parsed by opentype.js (nil for woff2, which cannot + be parsed); `:name`, `:type` and `:data` come from reading the file. + Returns the entry metadata, the `:content` to upload and, for parsed + fonts, the `:height-warning?` flag. A font with missing vertical-metric + tables uploads with `:height-warning? false` instead of failing." + [{:keys [font type name data] :as _params}] + (if font + ;; Font was parsed with opentype.js (ttf, otf, woff) + (let [;; Select the first nonblank name candidate: a blank + ;; preferredFamily/preferredSubfamily must not shadow a valid + ;; fontFamily/fontSubfamily. + family (cm/first-nonblank-string + (.getEnglishName ^js font "preferredFamily") + (.getEnglishName ^js font "fontFamily")) + variant (cm/first-nonblank-string + (.getEnglishName ^js font "preferredSubfamily") + (.getEnglishName ^js font "fontSubfamily")) + + ;; Vertical metrics determine the baseline in a text and the space between lines of + ;; text. For historical reasons, there are three pairs of ascender/descender + ;; values, known as hhea, OS/2 and uSWin metrics. Depending on the font, operating + ;; system and application a different set will be used to render text on the + ;; screen. On Mac, Safari and Chrome use the hhea values to render text. Firefox + ;; will respect the useTypoMetrics setting and will use the OS/2 if it is set. If + ;; the useTypoMetrics is not set, Firefox will also use metrics from the hhea + ;; table. On Windows, all browsers use the usWin metrics, but respect the + ;; useTypoMetrics setting and if set will use the OS/2 values. + + hhea (some-> ^js font .-tables .-hhea) + os2 (some-> ^js font .-tables .-os2) + + hhea-ascender (some-> ^js hhea .-ascender abs) + hhea-descender (some-> ^js hhea .-descender abs) + + win-ascent (some-> ^js os2 .-usWinAscent abs) + win-descent (some-> ^js os2 .-usWinDescent abs) + + os2-ascent (some-> ^js os2 .-sTypoAscender abs) + os2-descent (some-> ^js os2 .-sTypoDescender abs) + + ;; useTypoMetrics can be read from the 7th bit + f-selection (some-> ^js os2 .-fsSelection (bit-test 7)) + + height-warning? (boolean + (and hhea-ascender hhea-descender win-ascent win-descent + (or (not= hhea-ascender win-ascent) + (not= hhea-descender win-descent) + (and f-selection os2-ascent os2-descent + (or (not= hhea-ascender os2-ascent) + (not= hhea-descender os2-descent)))))) + data (js/Uint8Array. data) + metadata (resolve-font-metadata {:family family :variant variant} name)] + (assoc metadata + :content {:data data + :name name + :type type} + :height-warning? height-warning?)) + ;; Font could not be parsed (woff2), extract metadata from filename + (let [data (js/Uint8Array. data) + metadata (resolve-font-metadata {} name)] + (assoc metadata + :content {:data data + :name name + :type type} + :height-warning? false)))) (defn- parse-font-mtype "Detects a font mime-type from the first four bytes of an ArrayBuffer. @@ -159,84 +247,6 @@ (log/warn :msg (str/fmt "skipping file %s, unsupported format" name)) {:error name})))) -(defn- prepare-font - "Builds a ready-to-upload font entry from a parsed blob. - - Never throws for missing font metadata: a nil subfamily (opentype.js - returns nil when the font carries no English preferred/subfamily - names, as seen with some Fontshare families) defaults to regular - weight/style, and missing vertical-metric tables default to no - height warning." - [{:keys [font type name data] :as _params}] - (let [name (or name "")] - (if (some? font) - ;; Font was parsed with opentype.js (ttf, otf, woff) - (let [family (or (.getEnglishName ^js font "preferredFamily") - (.getEnglishName ^js font "fontFamily") - "") - variant (or (.getEnglishName ^js font "preferredSubfamily") - (.getEnglishName ^js font "fontSubfamily") - "") - - ;; Vertical metrics determine the baseline in a text and the space between lines of - ;; text. For historical reasons, there are three pairs of ascender/descender - ;; values, known as hhea, OS/2 and uSWin metrics. Depending on the font, operating - ;; system and application a different set will be used to render text on the - ;; screen. On Mac, Safari and Chrome use the hhea values to render text. Firefox - ;; will respect the useTypoMetrics setting and will use the OS/2 if it is set. If - ;; the useTypoMetrics is not set, Firefox will also use metrics from the hhea - ;; table. On Windows, all browsers use the usWin metrics, but respect the - ;; useTypoMetrics setting and if set will use the OS/2 values. - - hhea (some-> ^js font .-tables .-hhea) - os2 (some-> ^js font .-tables .-os2) - - hhea-ascender (some-> ^js hhea .-ascender abs) - hhea-descender (some-> ^js hhea .-descender abs) - - win-ascent (some-> ^js os2 .-usWinAscent abs) - win-descent (some-> ^js os2 .-usWinDescent abs) - - os2-ascent (some-> ^js os2 .-sTypoAscender abs) - os2-descent (some-> ^js os2 .-sTypoDescender abs) - - ;; useTypoMetrics can be read from the 7th bit - f-selection (some-> ^js os2 .-fsSelection (bit-test 7)) - - height-warning? (boolean - (and hhea-ascender hhea-descender win-ascent win-descent - (or (not= hhea-ascender win-ascent) - (not= hhea-descender win-descent) - (and f-selection os2-ascent os2-descent - (or (not= hhea-ascender os2-ascent) - (not= hhea-descender os2-descent)))))) - data (js/Uint8Array. data)] - {:content {:data data - :name name - :type type} - :font-family (or family "") - :font-weight (cm/parse-font-weight variant) - :font-style (cm/parse-font-style variant) - :variant-name variant - :height-warning? height-warning?}) - ;; Font could not be parsed (woff2), extract metadata from filename - (let [base-name (str/replace name #"\.[^.]+$" "") - ;; Strip known weight/style tokens and separators to derive family name - ;; Use word boundaries to avoid matching substrings (e.g. "Boldini" should not match "bold") - raw-family-name (-> base-name - (str/replace #"(?i)(^|[-_\s])(extra\s*black|ultra\s*black|extra\s*bold|ultra\s*bold|semi\s*bold|demi\s*bold|extra\s*light|ultra\s*light|hairline|thin|light|normal|regular|medium|bold|black|heavy|solid|italic)([-_\s]|$)" "$1$3") - (str/replace #"[-_\s]+" " ") - (str/trim)) - family-name (if (str/blank? raw-family-name) base-name raw-family-name) - data (js/Uint8Array. data)] - {:content {:data data - :name name - :type type} - :font-family family-name - :font-weight (cm/parse-font-weight base-name) - :font-style (cm/parse-font-style base-name) - :height-warning? false})))) - (defn process-upload "Given a seq of blobs and the team id, creates a ready-to-use fonts map with temporal ID's associated to each font entry. @@ -296,7 +306,7 @@ (if (:error item) (update acc :errors conj (str "'" (:error item) "'")) (try - (update acc :fonts #(join % (prepare-font item))) + (update acc :fonts #(join % (prepare-font-entry item))) (catch :default cause (log/warn :msg "skipping font file" :cause cause :name (:name item)) (update acc :errors conj (str "'" (:name item) "'")))))) diff --git a/frontend/test/frontend_tests/data/fonts_test.cljs b/frontend/test/frontend_tests/data/fonts_test.cljs new file mode 100644 index 0000000000..0bbd806bff --- /dev/null +++ b/frontend/test/frontend_tests/data/fonts_test.cljs @@ -0,0 +1,158 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns frontend-tests.data.fonts-test + "Tests for the upload metadata resolution in app.main.data.fonts." + (:require + [app.common.uuid :as uuid] + [app.main.data.fonts :as df] + [app.main.data.uploads :as uploads] + [app.main.repo :as rp] + [beicon.v2.core :as rx] + [cljs.test :as t :include-macros true] + [frontend-tests.helpers.async :as async] + [frontend-tests.helpers.mock :as mock])) + +(t/deftest resolve-font-metadata-prefers-opentype-values + (t/testing "opentype.js values win over the filename" + (t/is (= {:font-family "Inter" + :font-weight 700 + :font-style "normal" + :variant-name "Bold"} + (df/resolve-font-metadata {:family "Inter" :variant "Bold"} + "Inter.ttf"))))) + +(t/deftest resolve-font-metadata-falls-back-to-filename + (t/testing "derives weight and style from the filename when the variant is missing" + (t/is (= {:font-family "Pretendard" + :font-weight 600 + :font-style "normal" + :variant-name nil} + (df/resolve-font-metadata {:family "Pretendard" :variant nil} + "Pretendard-SemiBold.otf")))) + + (t/testing "derives weight and style from the filename when the variant is blank" + (t/is (= {:font-family "Lato" + :font-weight 300 + :font-style "normal" + :variant-name nil} + (df/resolve-font-metadata {:family "Lato" :variant ""} + "Lato-Light.ttf")))) + + (t/testing "derives the family from the filename when it is missing" + (t/is (= {:font-family "Blender Inter" + :font-weight 700 + :font-style "normal" + :variant-name nil} + (df/resolve-font-metadata {:family nil :variant nil} + "Blender-Inter-Bold.ttf")))) + + (t/testing "derives family, weight and style from the filename when nothing is available" + (t/is (= {:font-family "Pretendard" + :font-weight 600 + :font-style "normal" + :variant-name nil} + (df/resolve-font-metadata {} "Pretendard-SemiBold.woff2")))) + + (t/testing "derives italic from the filename" + (t/is (= {:font-family "Roboto" + :font-weight 400 + :font-style "italic" + :variant-name nil} + (df/resolve-font-metadata {} "Roboto-Italic.ttf")))) + + (t/testing "separates a style token glued to a weight token" + (t/is (= {:font-family "Roboto" + :font-weight 700 + :font-style "italic" + :variant-name nil} + (df/resolve-font-metadata {:family "Roboto" :variant nil} + "Roboto-BoldItalic.ttf")))) + + (t/testing "treats an oblique variant as italic" + (t/is (= {:font-family "Roboto" + :font-weight 700 + :font-style "italic" + :variant-name "BoldOblique"} + (df/resolve-font-metadata {:family "Roboto" :variant "BoldOblique"} + "Roboto.ttf")))) + + (t/testing "handles filenames without extension" + (t/is (= {:font-family "Lato" + :font-weight 300 + :font-style "normal" + :variant-name nil} + (df/resolve-font-metadata {} "Lato-Light"))))) + +(t/deftest ^:async upload-font-variant-sends-variant-name + (await + (mock/with-mocks* + {rp/cmd! mock/rpc-cmd-mock + uploads/upload-blob-chunked + (fn [_blob & _opts] (rx/of {:session-id (uuid/next)}))} + (let [item {:data {"font/ttf" (js/Uint8Array. 4)} + :team-id (uuid/next) + :font-id (uuid/next) + :font-family "Roboto" + :font-weight 700 + :font-style "italic" + :variant-name "BoldOblique"}] + (await (async/->promise (df/upload-font-variant item))) + (let [{:keys [cmd params]} (first @mock/rpc-calls)] + (t/is (= :create-font-variant cmd)) + (t/is (= "BoldOblique" (:variant-name params))))) + (await (async/settle))))) + +(defn- fake-font + "A minimal opentype.js-like font object exposing only what + prepare-font-entry reads." + [names] + #js {:getEnglishName (fn [k] (get names k)) + :tables #js {:hhea #js {:ascender 800 :descender -200} + :os2 #js {:usWinAscent 800 + :usWinDescent 200 + :sTypoAscender 800 + :sTypoDescender -200 + :fsSelection 0}}}) + +(t/deftest prepare-font-entry-falls-back-when-metadata-is-missing + (t/testing "parsed font with a missing subfamily" + (let [result (df/prepare-font-entry + {:font (fake-font {"preferredFamily" "Pretendard"}) + :type "font/ttf" + :name "Pretendard-SemiBold.otf" + :data (js/Uint8Array. 4)})] + (t/is (= "Pretendard" (:font-family result))) + (t/is (= 600 (:font-weight result))) + (t/is (= "normal" (:font-style result))) + (t/is (nil? (:variant-name result))) + (t/is (false? (:height-warning? result))) + (t/is (= {:name "Pretendard-SemiBold.otf" :type "font/ttf"} + (select-keys (:content result) [:name :type]))))) + + (t/testing "unparseable woff2 entry is resolved from the filename" + (let [result (df/prepare-font-entry + {:font nil + :type "font/woff2" + :name "Pretendard-SemiBold.woff2" + :data (js/Uint8Array. 4)})] + (t/is (= "Pretendard" (:font-family result))) + (t/is (= 600 (:font-weight result))) + (t/is (= "normal" (:font-style result))) + (t/is (false? (:height-warning? result))))) + + (t/testing "a blank first candidate does not shadow a valid second one" + (let [result (df/prepare-font-entry + {:font (fake-font {"preferredFamily" " " + "preferredSubfamily" " " + "fontSubfamily" "Bold"}) + :type "font/ttf" + :name "Roboto-Light.ttf" + :data (js/Uint8Array. 4)})] + (t/is (= "Roboto" (:font-family result))) + (t/is (= 700 (:font-weight result))) + (t/is (= "normal" (:font-style result))) + (t/is (= "Bold" (:variant-name result)))))) diff --git a/frontend/test/frontend_tests/fonts_test.cljs b/frontend/test/frontend_tests/fonts_test.cljs index dad9adf043..2be018f0a4 100644 --- a/frontend/test/frontend_tests/fonts_test.cljs +++ b/frontend/test/frontend_tests/fonts_test.cljs @@ -281,28 +281,28 @@ (t/is (nil? (#'df/parse-font-mtype nil))) (t/is (nil? (#'df/parse-font-mtype (bytes->buffer [0 1]))))) -(t/deftest prepare-font-defaults-missing-subfamily +(t/deftest prepare-font-entry-defaults-missing-subfamily (let [font (fake-ot-font {:family "Satoshi" :variant nil :hhea #js {:ascender 1000 :descender 300} :os2 #js {:usWinAscent 1000 :usWinDescent 300 :sTypoAscender 800 :sTypoDescender 200 :fsSelection 0}}) - entry (#'df/prepare-font {:font font - :type "font/otf" - :name "Satoshi-Regular.otf" - :data (bytes->buffer [79 84 84 79 0 0 0 0])})] + entry (df/prepare-font-entry {:font font + :type "font/otf" + :name "Satoshi-Regular.otf" + :data (bytes->buffer [79 84 84 79 0 0 0 0])})] (t/is (= "Satoshi" (:font-family entry))) (t/is (= 400 (:font-weight entry))) (t/is (= "normal" (:font-style entry))) (t/is (false? (:height-warning? entry))))) -(t/deftest prepare-font-tolerates-missing-tables +(t/deftest prepare-font-entry-tolerates-missing-tables (let [font (fake-ot-font {:family "Satoshi" :variant "Regular" :hhea nil :os2 nil}) - entry (#'df/prepare-font {:font font - :type "font/ttf" - :name "Satoshi-Regular.ttf" - :data (bytes->buffer [0 1 0 0 0 0 0 0])})] + entry (df/prepare-font-entry {:font font + :type "font/ttf" + :name "Satoshi-Regular.ttf" + :data (bytes->buffer [0 1 0 0 0 0 0 0])})] (t/is (= "Satoshi" (:font-family entry))) (t/is (false? (:height-warning? entry))))) @@ -351,3 +351,25 @@ (t/is (= 700 (:font-weight entry))) (t/is (= "normal" (:font-style entry)))) (t/is (empty? @emitted)))) + +(t/deftest ^:async process-upload-keeps-bold-and-bolditalic-woff2 + ;; Regression for the review on #11951: the bold-italic file resolved to + ;; style "normal", collided with the bold-only file on the (family, + ;; weight, style) join key and silently overwrote it. + (let [team-id (uuid/next) + blobs [#js {:name "Inter-Bold.woff2"} + #js {:name "Inter-BoldItalic.woff2"}] + emitted (atom []) + result (atom nil)] + (await (mock/with-mocks* + {wa/read-file-as-array-buffer + (mock/stub (fn [_] (rx/of (bytes->buffer [119 79 70 50 0 0 0 0])))) + st/emit! (mock/stub (fn [& evts] (swap! emitted into evts) nil))} + (await (async/observe (df/process-upload blobs team-id) + :on-next #(reset! result %))) + (await (async/settle)))) + (t/is (= 2 (count (vals @result)))) + (t/is (every? #(= "Inter" (:font-family %)) (vals @result))) + (t/is (every? #(= 700 (:font-weight %)) (vals @result))) + (t/is (= #{"normal" "italic"} (set (map :font-style (vals @result))))) + (t/is (empty? @emitted)))) diff --git a/frontend/test/frontend_tests/runner.cljs b/frontend/test/frontend_tests/runner.cljs index 38f5975de3..1be1f38f0a 100644 --- a/frontend/test/frontend_tests/runner.cljs +++ b/frontend/test/frontend_tests/runner.cljs @@ -17,6 +17,7 @@ [frontend-tests.data.exports-assets-test] [frontend-tests.data.exports-files-test] [frontend-tests.data.fix-deleted-fonts-test] + [frontend-tests.data.fonts-test] [frontend-tests.data.imports-test] [frontend-tests.data.jobs-test] [frontend-tests.data.modal-test] @@ -227,6 +228,7 @@ 'frontend-tests.data.store-test 'frontend-tests.data.exports-assets-test 'frontend-tests.data.exports-files-test + 'frontend-tests.data.fonts-test 'frontend-tests.data.imports-test 'frontend-tests.data.jobs-test 'frontend-tests.worker.exports-test From 9bac1d290bdb7bdb7dcd45abea1a4541b369418e Mon Sep 17 00:00:00 2001 From: Andrey Antukh Date: Fri, 13 Nov 2026 06:58:46 +0000 Subject: [PATCH 28/29] :books: Clarify when to run the file_change partition procedure The procedure straddles the upgrade to 2.20.0 and the document said nothing about the order. The preparation (phases 0 and 1) goes before the version update, the bulk and the delta passes after it, and only the cutover needs a scheduled maintenance window. The subtitle claimed the whole thing was online, which is not true. The copy marks no longer travel as a session GUC with a date typed by hand. The delta mark is seeded at the bulk with `now()`, and a CHECK keeps it from ending up empty; the instant of the cutover is recorded in the same table after the commit, so the checks of phase 5 read it instead of comparing against a date the operator retypes in every session. The procedure was run end to end on a scratch database, with the 23 SQL blocks extracted from the document and the application writes simulated between the phases. AI-assisted-by: deepseek-v4.1-flash --- operations/2.20.0-file-change-partition.md | 107 +++++++++++++++------ 1 file changed, 78 insertions(+), 29 deletions(-) diff --git a/operations/2.20.0-file-change-partition.md b/operations/2.20.0-file-change-partition.md index b74e589b56..3a9aabd51b 100644 --- a/operations/2.20.0-file-change-partition.md +++ b/operations/2.20.0-file-change-partition.md @@ -1,7 +1,9 @@ # Partition `file_change` by `file_id` -Online procedure to turn `file_change` into a table partitioned by -`HASH (file_id)` with 16 partitions, aligned with the ones of `file_data`. +Procedure to turn `file_change` into a table partitioned by `HASH (file_id)` +with 16 partitions, aligned with the ones of `file_data`. It runs online +except the cutover (phase 4), which needs a scheduled maintenance window with +the backend and the workers stopped. - Migration: `backend/src/app/migrations/sql/0158-partition-file-change.sql` - Issue: @@ -10,6 +12,42 @@ The result has to be **exactly** the schema of the migration, because on a big instance the migration is marked as applied and never runs. This procedure runs the same statements by hand. +## When to run it, and the stop + +Run it only on the instances where the automatic migration refuses to run: +`file_change` above 10 GiB of total size or above 2 million rows. Below that +the migration does the same work on boot and this procedure has nothing to do. + +The work spans the upgrade to 2.20.0. The order is not a suggestion: the +preparation goes **before** the version update, and phases 2 and 3 need the new +code running (it is the one that moves `updated_at` on every write, the +watermark of the delta passes). + +| Phase | When | Service | +|---|---|---| +| 0 and 1 | Before the upgrade, with 2.19.0 serving | Online | +| — | The upgrade to 2.20.0 | Usual deploy (step 0.3), before phase 2 | +| 2 | After the upgrade, with 2.20.0 serving | Online | +| 3 | From phase 2 to the window, every 1-6 h | Online | +| 4 | **A scheduled maintenance window** | **Backend and workers stopped** | +| 5 | After the cutover | Online | + +Phase 2 is one pass over every row: the long job, hours or days. Phase 3 +repeats a much shorter copy, only the rows that changed since the previous +pass (seconds each), until the day of the window, so the last delta inside the +lock stays short. + +Steps 0.1 and 0.2 cannot wait for the upgrade: a 2.20.0 backend that boots +without them runs the automatic migration inside the startup transaction and, +above the guard, it fails on every boot. + +**The procedure needs one stop, and only one: phase 4, the cutover.** It is a +full stop-start of the backend and the workers, not a rolling restart (4.2 +explains why). The window covers the last delta, the renames and the start +again: seconds to minutes of writes blocked on `file_change`, and the saves +that fall inside fail and the user retries. Everything else runs with the +instance serving traffic. + ## Sizing your instance Every number in this procedure comes from your database: run the queries @@ -366,8 +404,18 @@ INSERT INTO file_change_copy_state (name, last_id) VALUES ('bulk', '00000000-0000-0000-0000-000000000000') ON CONFLICT (name) DO NOTHING; -INSERT INTO file_change_copy_state (name) VALUES ('delta') +-- The mark of the delta passes starts at the bulk: the first pass of phase 3 +-- covers the rows changed since here, and `DO NOTHING` keeps a re-run from +-- resetting a mark that the passes already moved. +INSERT INTO file_change_copy_state (name, last_at) VALUES ('delta', now()) ON CONFLICT (name) DO NOTHING; + +-- The mark is mandatory: a pass without it copies nothing, in silence. The +-- `DROP ... IF EXISTS` keeps this block re-runnable. +ALTER TABLE file_change_copy_state + DROP CONSTRAINT IF EXISTS file_change_copy_state_delta_mark, + ADD CONSTRAINT file_change_copy_state_delta_mark + CHECK (name <> 'delta' OR last_at IS NOT NULL); ``` One batch (idempotent). It **always returns one row**: the number of rows it @@ -458,24 +506,21 @@ written during the bulk are missing until the delta passes pick them up. ## Phase 3 — Delta passes Every pass copies the rows modified since the **previous pass**. The watermark -lives in `file_change_copy_state.last_at` (the start of the last pass), and the -SQL subtracts one hour of margin, which absorbs the clock drift between the -application (which writes `updated_at`) and the database (which writes -`now()`) and the drift between batches. A row that a pass already copied but -did not change again is not rewritten: the `WHERE` of the `ON CONFLICT` skips -it. +lives in `file_change_copy_state.last_at` (the start of the last pass; the +setup block of phase 2 seeds it with the moment of the bulk, so the first pass +covers what changed since then), and the SQL subtracts one hour of margin, +which absorbs the clock drift between the application (which writes +`updated_at`) and the database (which writes `now()`) and the drift between +batches. A row that a pass already copied but did not change again is not +rewritten: the `WHERE` of the `ON CONFLICT` skips it. ```sql --- :bulk_start = the time the bulk started (the margin is applied below). --- Set it in this session before running the pass: without it the query fails --- instead of copying nothing and leaving those rows behind for good. -SET penpot.bulk_start = '2026-10-25 08:00:00+02'; - BEGIN; WITH mark AS ( - SELECT COALESCE((SELECT last_at FROM file_change_copy_state WHERE name = 'delta'), - current_setting('penpot.bulk_start')::timestamptz) - INTERVAL '1 hour' AS since + SELECT last_at - INTERVAL '1 hour' AS since + FROM file_change_copy_state + WHERE name = 'delta' ) INSERT INTO file_change_new (id, file_id, created_at, session_id, revn, data, changes, profile_id, features, label, version, created_by, @@ -579,23 +624,23 @@ CREATE INDEX file_change_new__user_snapshots__idx driver) would keep writing to `file_change_legacy` after the cutover, and the pool recycles its connections every 30 minutes, so the window of silent writes is long. -- Run the transaction. +- Run the transaction, the `ANALYZE`, and the statement that records the + instant of the cutover (all of it in the block below). - Start the backend and the workers again. ```sql SET ROLE penpot; -- the owner of file_change (see 0.0 and 0.6) -SET penpot.bulk_start = '2026-10-25 08:00:00+02'; BEGIN; LOCK TABLE file_change IN ACCESS EXCLUSIVE MODE; -- The final delta: the rows modified since the last pass (its mark, minus --- the hour of margin). The mark is the one the last pass left; the fallback --- is the bulk start. +-- the hour of margin). The mark is the one the last pass left. WITH mark AS ( - SELECT COALESCE((SELECT last_at FROM file_change_copy_state WHERE name = 'delta'), - current_setting('penpot.bulk_start')::timestamptz) - INTERVAL '1 hour' AS since + SELECT last_at - INTERVAL '1 hour' AS since + FROM file_change_copy_state + WHERE name = 'delta' ) INSERT INTO file_change_new (id, file_id, created_at, session_id, revn, data, changes, profile_id, features, label, version, created_by, @@ -703,6 +748,13 @@ ALTER TABLE file_change_legacy COMMIT; ANALYZE file_change; + +-- The instant of the cutover, for the checks of phase 5. It goes after the +-- `COMMIT`: inside the transaction `now()` is the start of it (the moment of +-- the `LOCK`), and the rows the final delta copies could pass for writes +-- happened after the cutover. +INSERT INTO file_change_copy_state (name, last_at) VALUES ('cutover', now()) +ON CONFLICT (name) DO UPDATE SET last_at = excluded.last_at; ``` - `file_change_legacy` keeps its data and its indexes (with the `_legacy` @@ -716,18 +768,15 @@ ANALYZE file_change; table and that nothing went to the table that is going away: ```sql --- :cutover = the time the backend went up on the new table. Set it in this --- session: without it the queries fail instead of comparing against a stale --- date. -SET penpot.cutover = '2026-10-25 08:00:00+02'; - -- the rows written after the cutover: this one grows with every save SELECT count(*) FROM file_change - WHERE updated_at > current_setting('penpot.cutover')::timestamptz; + WHERE updated_at > (SELECT last_at FROM file_change_copy_state + WHERE name = 'cutover'); -- the rows that went to the legacy table: this one has to stay at 0 SELECT count(*) FROM file_change_legacy - WHERE updated_at > current_setting('penpot.cutover')::timestamptz; + WHERE updated_at > (SELECT last_at FROM file_change_copy_state + WHERE name = 'cutover'); ``` A number above zero in the second query means that some connection was From ad710ed143e64d99ff4aaf7fde047330369041bd Mon Sep 17 00:00:00 2001 From: Andrey Antukh Date: Fri, 13 Nov 2026 07:10:24 +0000 Subject: [PATCH 29/29] :books: Document the squash-merge default and the no-squash opt-out Squash merge appeared twice in the guide, as a review-process bullet and in branch naming, with no reason and no way out. Add a Merging subsection: one PR lands as one atomic commit, so the history stays readable and git bisect never stops on an intermediate commit that fails the tests or the linter. A contributor who keeps every commit self-contained may ask for a regular merge in the first line of the PR description. Record the same contract in mem:workflow/creating-prs (merge strategy, and the Merge line above the AI Note line) and in mem:workflow/creating-commits (what a self-contained commit means when a regular merge is requested). AI-assisted-by: deepseek-v4.1-flash --- .serena/memories/workflow/creating-commits.md | 8 ++++++ .serena/memories/workflow/creating-prs.md | 15 +++++++++++ CONTRIBUTING.md | 25 ++++++++++++++++--- 3 files changed, 45 insertions(+), 3 deletions(-) diff --git a/.serena/memories/workflow/creating-commits.md b/.serena/memories/workflow/creating-commits.md index cf08358aaf..d225217867 100644 --- a/.serena/memories/workflow/creating-commits.md +++ b/.serena/memories/workflow/creating-commits.md @@ -56,6 +56,14 @@ that breaks them is wrong, even if the code is right. `:bug:` bug fix · `:sparkles:` enhancement · `:tada:` new feature · `:recycle:` refactor · `:lipstick:` cosmetic · `:ambulance:` critical fix · `:books:` docs · `:construction:` WIP · `:boom:` breaking · `:wrench:` config · `:zap:` perf · `:whale:` docker · `:paperclip:` other · `:arrow_up:` dep upgrade · `:arrow_down:` dep downgrade · `:fire:` removal · `:globe_with_meridians:` translations · `:rocket:` epic/highlight +## Atomic Commits + +A PR squash-merges into one commit by default, so the split inside the branch +does not reach `main` and nothing depends on it. When the PR asks for a +regular merge (`mem:workflow/creating-prs` → Merge Strategy), every commit +must stand alone: it builds, passes the linter, and needs no later commit on +the same branch to make sense. + ## Referencing Issues Use `Closes #NNNN` (not `Fixes #NNNN`) in commit and PR body text for readable context. The explicit issue-to-PR link is assigned with `python3 scripts/gh.py link-issue ` as described in `mem:workflow/creating-prs`; do not rely on the keyword to create that link. diff --git a/.serena/memories/workflow/creating-prs.md b/.serena/memories/workflow/creating-prs.md index 87f10f2121..e8e34181e6 100644 --- a/.serena/memories/workflow/creating-prs.md +++ b/.serena/memories/workflow/creating-prs.md @@ -32,6 +32,19 @@ PR titles follow commit title conventions: See `mem:workflow/creating-commits` for emoji codes. Squash merge uses the PR title as the final commit subject, so title format matters. +## Merge Strategy + +Policy: `CONTRIBUTING.md` → "Merging", under "Pull Requests". + +- Squash merge is the default, so the PR's own commits never reach `main` + and the PR title becomes the landed subject. +- A regular merge is possible on request only. The request is the **first + line** of the PR body, above the `**Note:**` line: + `> **Merge:** do not squash; each commit is self-contained.` +- Ask for it only when every commit on the branch is self-contained (builds + and passes lint on its own). Default to squash unless the user asks for + the opposite. + ## Description Body Include concise sections covering: @@ -60,6 +73,8 @@ PR descriptions follow this structure: ``` The "Note:" line is required at the top. Adjust if this is a manual (non-AI) PR. +When the PR requests a regular merge instead of a squash merge, its +`**Merge:**` line goes above the `**Note:**` line (see Merge Strategy). ## Writing Principles diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 9b23ae0fa9..27ec89998b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -20,6 +20,7 @@ Center](https://help.penpot.app/). - [Title format](#title) - [Description](#description) - [Review process](#review-process) + - [Merging](#merging) - [What we won't accept](#what-we-wont-accept) - [Good first issues](#good-first-issues) - [Commit Guidelines](#commit-guidelines) @@ -111,8 +112,8 @@ Branch names are not enforced, but we recommend the following: - Otherwise, use a short, descriptive name with words separated by hyphens and no slashes (e.g. `fix-ellipse-icon-typo`, `feat-auto-link-libraries`). -Since PRs are squash-merged, the branch name does not survive into the -commit history — what matters is the [PR title](#title). +Since PRs are squash-merged by default, the branch name does not survive +into the commit history — what matters is the [PR title](#title). ### Format @@ -156,9 +157,27 @@ for more concrete information. - Address review feedback by **pushing new commits** — do not force-push during review, as it breaks comment threads. - PRs require at least **one approval** before merge. -- We use **squash-merge** by default. The PR title becomes the final +- We **squash-merge** by default; see [Merging](#merging) for the reason + and for how to ask for a regular merge. The PR title becomes the final commit message, so follow the [title format](#title) above. +### Merging + +PRs are **squash-merged** by default. One PR lands as one commit, which keeps +the change atomic: the history stays readable, and `git bisect` never stops +on an intermediate commit that fails the tests or the linter. + +This is the default, not a hard rule. If you keep your commits atomic and +self-contained — each one builds, passes the linter and makes sense on its +own — you may ask for a regular merge instead. Put the request in the first +line of the PR description, above everything else: + +> **Merge:** do not squash; each commit is self-contained. + +Maintainers honor the request when the commits hold up, and squash-merge +otherwise. After a squash merge the PR title becomes the commit subject on +the main branch; see [Title format](#title). + ### What we won't accept To save time on both sides, please avoid submitting PRs that: