diff --git a/.serena/memories/backend/core.md b/.serena/memories/backend/core.md index 7a70eb4669..16978ac1ff 100644 --- a/.serena/memories/backend/core.md +++ b/.serena/memories/backend/core.md @@ -67,6 +67,7 @@ Every background job is a job-def: a plain `(defn execute-X [cfg params] ...)` i - Job resources (`app.jobs.storage`) are the objects a job owns through `job.resource_id`: they live in the `job-resource` bucket, are written with the profile of the job as owner (only that profile can read them back from `/assets/by-id`) and are never deduplicated. They carry no expiry of their own, because `expired-at` is stored as `deleted_at` and an object with it is unreadable from birth: the retention of the artifact is the retention of its job row. The writer marks the object as touched right away, so an artifact that no row references (a crash between the upload and the completion) is reclaimed by `storage-gc-touched`, and a row that does reference it freezes it until the jobs GC deletes the row and touches the object. - The binfile jobs (`:export-binfile` and `:import-binfile`) run the core of `bf.v1`/`bf.v3` through `app.binfile.jobs`, which turns the `:progress` taps of the core into `progress` events of the job with `jobs/heartbeat` (the `modified_at` touch at most once per second, progress events at up to ~200ms). Each event is a self-contained milestone: a `:stage` keyword of one vocabulary shared by the two formats (the same work has a different name in v1 and v3) and an optional `:counters` map, with the counter of the activity named after its own stage and the outer scopes in course as context, which the tap carries itself (the reader puts `::outer-counters` in every tap inside them — today only `:files`, extendable to `:teams`/`:projects` for multi-scope exports — so the adapter remembers nothing between taps and a tap that happens outside any scope has no context); the ids and names the taps carry never reach the event. Every tap is reported, so a tap with no units to count still keeps the job alive, and a milestone the throttle drops loses nothing because the next one carries the same state. The adapter checks that the job is still active before the run, and the beat raises an `:interrupt` right there on the run thread when the job is no longer active, so the work stops at its next event point inside the transaction of its handler, which rolls back. A run whose last tap predates the interruption is discarded by the check once it returns. The adapter passes the job id from the handler context. Their job-def components carry pool, storage, metrics and msgbus, because a job with a `profile_id` cannot store an event without a msgbus on its cfg. Both readers answer with the same shape (`:file-ids` and, in version 3, the resolution of the libraries), because the version 1 reader returns a bare set of file ids. - Both binfile handlers revalidate their permission at run time, not only when the job is created, because it can be revoked while the job waits: the export checks the read permission of every file before reading anything and stores the package in `job-resource` as the resource of the job (answering with the completion envelope), and the import checks the edition permission of the destination and runs the core inside a transaction, which is what makes a cancelled import roll back instead of leaving half of a file. The import releases its package (marks it as deleted and drops the reference from the job row in the same transaction) and deletes its temporary copy once the job is terminal, whatever the outcome. Both handlers check their own result against a schema before returning it (the import job-def declares that same schema as its result schema, which is what types the resolution when the row is read). +- The v3 import only materializes a library relation when both ids belong to the package and differ; a relation that points outside the package or a self link is ignored with a warning. Migration `0158-del-forbidden-file-library-rels` deletes the legacy cross-team and self rows from `file_library_rel` and their `file_library_sync` rows, sync rows first. - The domain cfg of a binfile request lives in `app.binfile.common` (`export-cfg`, `import-cfg`), so the legacy RPC and the jobs freeze the same ids, resolve the same team features and apply the same anti-zip-bomb limits. `resolve-export-type` is the translation the legacy RPC needs for its deprecated boolean flags; the creation command asks the caller for the type instead of guessing a default. - Two retention knobs, easy to confuse: `:jobs-retention` sweeps terminal internal rows (the ones without a profile), while `:jobs-user-ttl` is the default expiry a user-facing job gets when it is created; the jobs GC deletes the row once `expires_at` passes and touches its resource. The heavy work of a user job runs on its own `:binfile` queue (`:worker-binfile-parallelism`, default 1) so it does not block the default runner. - The quote of a family of jobs (`::export-jobs-per-profile`, `::import-jobs-per-profile`) counts the jobs of the profile that are still on their way (`new`/`scheduled`/`running`/`retry`) whose name the registry gives for that family, so a new job of the family is covered without touching the quote. diff --git a/CHANGES.md b/CHANGES.md index ac091ef363..1fd9718201 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -1,6 +1,22 @@ # CHANGELOG -## 2.19.0 (Unreleased) +## 2.19.0 + +### :boom: Breaking changes & Deprecations + +#### Migrate legacy file payloads to file_data and delete orphan revisions + +File payloads now live in one place: every file state and snapshot is kept in `file_data`, and the orphan `file_change` rows that kept old payloads alive forever are removed. No read path reached those rows and nothing ever collected them, so the upgrade also reclaims the database and TOAST space they held. There is no API or UI change. + +On small and medium instances this happens automatically on boot and there is nothing to do. On large instances (millions of snapshots, hundreds of GB in TOAST) the work does not fit in the startup transaction: run it by hand **before upgrading to 2.19.0** following the [large-instance runbook](https://github.com/penpot/penpot/blob/staging/operations/2.19.0-migrate-legacy-file-data.md). The runbook tells you whether your instance needs it, keeps the instance online while the data moves, and verifies the result before you deploy. + +[#12784](https://github.com/penpot/penpot/issues/12784) (PR: [#12785](https://github.com/penpot/penpot/pull/12785)) + +#### Enable the Admin Console by default + +The Admin Console is now enabled by default and the `enable-admin-console` flag is no longer needed. Self-hosted instances must deploy the `penpot-admin-console` service and set `PENPOT_ADMIN_CONSOLE_URI` in frontend and backend (check the [upgrade guide](https://help.penpot.app/technical-guide/getting-started/docker/#upgrade-to-2.19)). + +[#12807](https://github.com/penpot/penpot/issues/12807) (PR: [#12475](https://github.com/penpot/penpot/pull/12475)) ### :rocket: Epics and highlights @@ -74,6 +90,37 @@ - Fix thumbnails failing on older Chromium browsers and each thumbnail rendering several times [#11878](https://github.com/penpot/penpot/issues/11878) (PR: [#11879](https://github.com/penpot/penpot/pull/11879)) - Fix internal error when opening the libraries modal [#11943](https://github.com/penpot/penpot/issues/11943) (PR: [#11946](https://github.com/penpot/penpot/pull/11946)) - Fix Plugin API `connectLibrary` linking unpublished files as libraries [#12001](https://github.com/penpot/penpot/issues/12001) (PR: [#12002](https://github.com/penpot/penpot/pull/12002)) +- Fix incorrect padding at the top of the tokens and themes section [#9585](https://github.com/penpot/penpot/issues/9585) (PR: [#12422](https://github.com/penpot/penpot/pull/12422)) +- Fix token tree path handling for repeated and nested segments (by @makesomethingshit) [#11588](https://github.com/penpot/penpot/issues/11588) (PR: [#11591](https://github.com/penpot/penpot/pull/11591)) +- Fix comment input duplicating text when confirming a Japanese IME composition with Enter [#11757](https://github.com/penpot/penpot/issues/11757) (PR: [#11768](https://github.com/penpot/penpot/pull/11768)) +- Fix file not saving after deleting a variant [#11758](https://github.com/penpot/penpot/issues/11758) (PR: [#12216](https://github.com/penpot/penpot/pull/12216)) +- Fix component copies not syncing when a shape is moved out of a main component and another change follows at once [#11978](https://github.com/penpot/penpot/issues/11978) (PR: [#11981](https://github.com/penpot/penpot/pull/11981)) +- Fix MCP plugin showing a version mismatch warning on every patch release (by @robertkodra) [#12007](https://github.com/penpot/penpot/issues/12007) (PR: [#12009](https://github.com/penpot/penpot/pull/12009)) +- Fix workspace crash when switching to the path tool while drawing a shape [#12100](https://github.com/penpot/penpot/issues/12100) (PR: [#12101](https://github.com/penpot/penpot/pull/12101)) +- Fix referential integrity error when swapping a component [#12144](https://github.com/penpot/penpot/issues/12144) (PR: [#12333](https://github.com/penpot/penpot/pull/12333)) +- Fix workspace saving text shapes on every page open when a custom font is missing [#12151](https://github.com/penpot/penpot/issues/12151) (PR: [#12164](https://github.com/penpot/penpot/pull/12164)) +- Fix swapped nested components losing their absolute position after a library update [#12176](https://github.com/penpot/penpot/issues/12176) (PR: [#12556](https://github.com/penpot/penpot/pull/12556)) +- Fix theme selector showing the editor message when the tokens source is a library [#12177](https://github.com/penpot/penpot/issues/12177) (PR: [#12203](https://github.com/penpot/penpot/pull/12203)) +- Fix untranslated texts in the This file tab of the libraries modal [#12178](https://github.com/penpot/penpot/issues/12178) (PR: [#12204](https://github.com/penpot/penpot/pull/12204)) +- Fix workspace staying in the saving state and losing unsaved edits when a save request gets no answer [#12222](https://github.com/penpot/penpot/issues/12222) (PR: [#12223](https://github.com/penpot/penpot/pull/12223)) +- Fix scrub cursor showing on numeric inputs with mixed values in the sidebar [#12227](https://github.com/penpot/penpot/issues/12227) (PR: [#12229](https://github.com/penpot/penpot/pull/12229)) +- Fix internal error when copying a swapped nested component [#12289](https://github.com/penpot/penpot/issues/12289) (PR: [#12434](https://github.com/penpot/penpot/pull/12434)) +- Show an error when uploading a custom font fails [#12304](https://github.com/penpot/penpot/issues/12304) (PR: [#12315](https://github.com/penpot/penpot/pull/12315)) +- Fix React warning about the unknown `tooltipClass` prop in layout-item size inputs [#12311](https://github.com/penpot/penpot/issues/12311) (PR: [#12329](https://github.com/penpot/penpot/pull/12329)) +- Fix guides inside a board still being interactive [#12322](https://github.com/penpot/penpot/issues/12322) (PR: [#12460](https://github.com/penpot/penpot/pull/12460)) +- Fix pasted text not being visible in an empty text box until it is resized [#12438](https://github.com/penpot/penpot/issues/12438) (PR: [#12859](https://github.com/penpot/penpot/pull/12859)) +- Fix files failing referential integrity validation staying unsaveable in every session [#12464](https://github.com/penpot/penpot/issues/12464) (PR: [#12524](https://github.com/penpot/penpot/pull/12524), [#12662](https://github.com/penpot/penpot/pull/12662)) +- Fix unicode characters not rendering in text layers with the new render engine [#12469](https://github.com/penpot/penpot/issues/12469) (PR: [#12471](https://github.com/penpot/penpot/pull/12471)) +- Fix text image fills staying blank when media loads after text layout [#12563](https://github.com/penpot/penpot/issues/12563) (PR: [#12564](https://github.com/penpot/penpot/pull/12564)) +- Fix swapped nested icons moving outside the layout after a library update [#12657](https://github.com/penpot/penpot/issues/12657) (PR: [#12716](https://github.com/penpot/penpot/pull/12716)) +- Fix garbled bullets before the element counts in the libraries modal [#12677](https://github.com/penpot/penpot/issues/12677) (PR: [#12678](https://github.com/penpot/penpot/pull/12678)) +- Fix token dropdown of numeric inputs showing the scrub cursor [#12679](https://github.com/penpot/penpot/issues/12679) (PR: [#12680](https://github.com/penpot/penpot/pull/12680)) +- Fix variant property names and values accepting `,` and `=` [#12683](https://github.com/penpot/penpot/issues/12683) (PR: [#12688](https://github.com/penpot/penpot/pull/12688)) +- Fix changing a variant property from another page leaving the main component label out of date [#12692](https://github.com/penpot/penpot/issues/12692) (PR: [#12693](https://github.com/penpot/penpot/pull/12693)) +- Fix `:referential-integrity` errors when moving variant mains in or out of a set [#12696](https://github.com/penpot/penpot/issues/12696) (PR: [#12706](https://github.com/penpot/penpot/pull/12706)) +- Fix `:referential-integrity` errors when creating, combining or renaming variant sets [#12734](https://github.com/penpot/penpot/issues/12734) (PR: [#12779](https://github.com/penpot/penpot/pull/12779)) +- Fix save errors after masking a variant with Ctrl+M [#12748](https://github.com/penpot/penpot/issues/12748) (PR: [#12749](https://github.com/penpot/penpot/pull/12749)) +- Fix icons in component instances not being shown on the canvas [#12752](https://github.com/penpot/penpot/issues/12752) ### :sparkles: New features & Enhancements @@ -89,7 +136,7 @@ - Apply tokens to stroke per side [#9842](https://github.com/penpot/penpot/issues/9842) (PR: [#10913](https://github.com/penpot/penpot/pull/10913)) - Add individual stroke widths per side (top, right, bottom, left) [#9854](https://github.com/penpot/penpot/issues/9854) (PR: [#10648](https://github.com/penpot/penpot/pull/10648)) - Expand the token tree right after importing it (by @AKnassa) [#9819](https://github.com/penpot/penpot/issues/9819) (PR: [#10664](https://github.com/penpot/penpot/pull/10664)) -- Update the selection size badge live during resize, replacing the resize tooltip [#10259](https://github.com/penpot/penpot/issues/10259) +- Update the selection size badge live during resize, replacing the resize tooltip [#10259](https://github.com/penpot/penpot/issues/10259) (PR: [#11884](https://github.com/penpot/penpot/pull/11884)) - Add color variants, rotation correction, viewport flip and hide-when-too-small to the selection size badge [#10264](https://github.com/penpot/penpot/issues/10264) (PR: [#11884](https://github.com/penpot/penpot/pull/11884)) - Support single-file SVG export with the WebGL render [#10546](https://github.com/penpot/penpot/issues/10546) - Persist the hide resolved comments preference and separate the filter options in the comments menu [#10686](https://github.com/penpot/penpot/issues/10686) (PR: [#10694](https://github.com/penpot/penpot/pull/10694)) @@ -113,13 +160,17 @@ - Make "Fix when scrolling" work in the WebGL render [#11684](https://github.com/penpot/penpot/issues/11684) (PR: [#11752](https://github.com/penpot/penpot/pull/11752)) - Complete Catalan translations [#11739](https://github.com/penpot/penpot/issues/11739) (PR: [#11741](https://github.com/penpot/penpot/pull/11741)) - Remove the optional client-provided id from creation RPC endpoints [#11783](https://github.com/penpot/penpot/issues/11783) (PR: [#11784](https://github.com/penpot/penpot/pull/11784)) -- Allow color tokens for the canvas background [#11791](https://github.com/penpot/penpot/issues/11791) (PR: [#11923](https://github.com/penpot/penpot/pull/11923)) +- Allow color tokens for the canvas background [#11791](https://github.com/penpot/penpot/issues/11791) (PR: [#11923](https://github.com/penpot/penpot/pull/11923), [#11950](https://github.com/penpot/penpot/pull/11950)) - Show a "no assets yet" message for empty files in the libraries modal [#11811](https://github.com/penpot/penpot/issues/11811) (PR: [#11818](https://github.com/penpot/penpot/pull/11818)) - Upgrade the MCP SDK to v2 and adopt the new protocol version [#11827](https://github.com/penpot/penpot/issues/11827) (PR: [#11841](https://github.com/penpot/penpot/pull/11841)) - Remove legacy SSE support from the MCP server [#11846](https://github.com/penpot/penpot/issues/11846) (PR: [#11841](https://github.com/penpot/penpot/pull/11841)) - Delete selected path handles with Backspace or Delete [#11858](https://github.com/penpot/penpot/issues/11858) (PR: [#11896](https://github.com/penpot/penpot/pull/11896)) - Use the tooltip component in the token pill info [#11886](https://github.com/penpot/penpot/issues/11886) (PR: [#11900](https://github.com/penpot/penpot/pull/11900)) - Hide the duplicate cursor when alt-dragging a shape that cannot be duplicated (by @0xTHAC0) [#11905](https://github.com/penpot/penpot/issues/11905) (PR: [#11195](https://github.com/penpot/penpot/pull/11195)) +- Document the team auto-add behavior with `disable-email-verification` [#11376](https://github.com/penpot/penpot/issues/11376) (PR: [#11425](https://github.com/penpot/penpot/pull/11425)) +- Make the workspace recover from interrupted saves and show a proper error instead of the internal error page [#11799](https://github.com/penpot/penpot/issues/11799) +- Add component management operations to the Plugin API [#12301](https://github.com/penpot/penpot/issues/12301) (PR: [#12303](https://github.com/penpot/penpot/pull/12303)) +- Record the imported file ids in the import-binfile audit event [#12485](https://github.com/penpot/penpot/issues/12485) (PR: [#12491](https://github.com/penpot/penpot/pull/12491)) ## 2.18.6 diff --git a/backend/src/app/binfile/common.clj b/backend/src/app/binfile/common.clj index 80ccbe22bf..efc6174d4b 100644 --- a/backend/src/app/binfile/common.clj +++ b/backend/src/app/binfile/common.clj @@ -478,6 +478,18 @@ {:team-id team-id :deleted-at nil})) +(def ^:private sql:get-files-rels + "SELECT flr.*, + fls.synced_at + FROM file_library_rel AS flr + JOIN file AS l + ON flr.library_file_id = l.id + LEFT JOIN file_library_sync AS fls + ON fls.file_id = flr.file_id + AND fls.library_file_id = flr.library_file_id + WHERE flr.file_id = ANY(?) + AND l.deleted_at IS NULL;") + (defn get-files-rels "Given a set of file-id's, return all matching relations with the libraries" [cfg ids] @@ -486,18 +498,8 @@ (assert (every? uuid? ids) "expected a set of uuids") (db/run! cfg (fn [{:keys [::db/conn]}] - (let [ids (db/create-array conn "uuid" ids) - sql (str "SELECT flr.*," - " fls.synced_at" - " FROM file_library_rel AS flr" - " JOIN file AS l" - " ON flr.library_file_id = l.id" - " LEFT JOIN file_library_sync AS fls" - " ON fls.file_id = flr.file_id" - " AND fls.library_file_id = flr.library_file_id" - " WHERE flr.file_id = ANY(?)" - " AND l.deleted_at IS NULL;")] - (db/exec! conn [sql ids]))))) + (let [ids (db/create-array conn "uuid" ids)] + (db/exec! conn [sql:get-files-rels ids]))))) (def ^:private sql:upsert-file-library-sync "INSERT INTO file_library_sync (file_id, library_file_id, synced_at) diff --git a/backend/src/app/binfile/v3.clj b/backend/src/app/binfile/v3.clj index 59c344cf02..9ec984e524 100644 --- a/backend/src/app/binfile/v3.clj +++ b/backend/src/app/binfile/v3.clj @@ -1039,15 +1039,17 @@ (defn- import-file-relations [{:keys [::db/conn ::manifest ::bfc/timestamp] :as cfg}] - (let [relations (:relations manifest)] + (let [relations (:relations manifest) + file-ids (into #{} (map :id) (:files manifest))] (doseq [[index [file-id libr-id]] (d/enumerate relations)] (tap-progress cfg {:section :relations :current (inc index) :total (count relations)}) - (let [file-id (bfc/lookup-index file-id) - libr-id (bfc/lookup-index libr-id)] - - (when (and file-id libr-id) + (if (and (not= file-id libr-id) + (contains? file-ids file-id) + (contains? file-ids libr-id)) + (let [file-id (bfc/lookup-index file-id) + libr-id (bfc/lookup-index libr-id)] (l/dbg :hint "create file library link" :file-id (str file-id) :lib-id (str libr-id) @@ -1055,7 +1057,10 @@ (let [rel-params {:file-id file-id :library-file-id libr-id}] (db/insert! conn :file-library-rel rel-params) - (bfc/upsert-file-library-sync! conn (assoc rel-params :synced-at timestamp)))))))) + (bfc/upsert-file-library-sync! conn (assoc rel-params :synced-at timestamp)))) + (l/warn :hint "ignoring invalid file library relation" + :file-id (str file-id) + :lib-id (str libr-id)))))) (defn- import-storage-objects [{:keys [::bfc/input ::entries-index ::bfc/timestamp] :as cfg}] diff --git a/backend/src/app/migrations.clj b/backend/src/app/migrations.clj index 89d161717d..9e1a91b636 100644 --- a/backend/src/app/migrations.clj +++ b/backend/src/app/migrations.clj @@ -531,11 +531,14 @@ {:name "0156-reset-pg-storage-defaults" :fn (mg/resource "app/migrations/sql/0156-reset-pg-storage-defaults.sql")} -< {:name "0157-add-audit-log-pending-table" + {:name "0157-add-audit-log-pending-table" :fn (mg/resource "app/migrations/sql/0157-add-audit-log-pending-table.sql")} {:name "0157-migrate-legacy-file-data" - :fn (mg/resource "app/migrations/sql/0157-migrate-legacy-file-data.sql")}]) + :fn (mg/resource "app/migrations/sql/0157-migrate-legacy-file-data.sql")} + + {:name "0158-del-forbidden-file-library-rels" + :fn (mg/resource "app/migrations/sql/0158-del-forbidden-file-library-rels.sql")}]) (defn apply-migrations! [pool name migrations] diff --git a/backend/src/app/migrations/sql/0158-del-forbidden-file-library-rels.sql b/backend/src/app/migrations/sql/0158-del-forbidden-file-library-rels.sql new file mode 100644 index 0000000000..d7970d34a6 --- /dev/null +++ b/backend/src/app/migrations/sql/0158-del-forbidden-file-library-rels.sql @@ -0,0 +1,38 @@ +-- Remove file-library relations that violate the same-team invariant: +-- relations across teams and self-relations. Leftover from before the +-- binfile export, the view-only bundle, link-file-to-library and the +-- manifest writers enforced same-team libraries. Relations with both +-- endpoints in the same team are kept. +-- +-- The sync rows go first: the second statement reads file_library_rel +-- while it still contains the forbidden rows. + +WITH forbidden AS ( + SELECT flr.file_id, flr.library_file_id + FROM file_library_rel flr + JOIN file f ON f.id = flr.file_id + JOIN project fp ON fp.id = f.project_id + JOIN file l ON l.id = flr.library_file_id + JOIN project lp ON lp.id = l.project_id + WHERE fp.team_id <> lp.team_id + OR flr.file_id = flr.library_file_id +) +DELETE FROM file_library_sync fls +USING forbidden +WHERE fls.file_id = forbidden.file_id + AND fls.library_file_id = forbidden.library_file_id; + +WITH forbidden AS ( + SELECT flr.file_id, flr.library_file_id + FROM file_library_rel flr + JOIN file f ON f.id = flr.file_id + JOIN project fp ON fp.id = f.project_id + JOIN file l ON l.id = flr.library_file_id + JOIN project lp ON lp.id = l.project_id + WHERE fp.team_id <> lp.team_id + OR flr.file_id = flr.library_file_id +) +DELETE FROM file_library_rel flr +USING forbidden +WHERE flr.file_id = forbidden.file_id + AND flr.library_file_id = forbidden.library_file_id; diff --git a/backend/test/backend_tests/binfile_test.clj b/backend/test/backend_tests/binfile_test.clj index 55c0935613..e2fec74708 100644 --- a/backend/test/backend_tests/binfile_test.clj +++ b/backend/test/backend_tests/binfile_test.clj @@ -30,6 +30,7 @@ [app.storage.tmp :as tmp] [backend-tests.helpers :as th] [backend-tests.storage-test :as stt] + [clojure.data.json :as json] [clojure.java.io :as jio] [clojure.test :as t] [cuerdas.core :as str] @@ -37,6 +38,9 @@ [datoteka.io :as io]) (:import java.io.ByteArrayInputStream + java.io.ByteArrayOutputStream + java.io.DataInputStream + java.io.DataOutputStream java.io.OutputStreamWriter java.io.Writer java.util.zip.Deflater @@ -216,6 +220,63 @@ :token-radius token-radius :token-opacity token-opacity})) +(defn- export-file + [file & {:keys [include-libraries]}] + (let [output (tmp/tempfile :suffix ".zip")] + (v3/export-files! + (cond-> (-> th/*system* + (assoc ::bfc/file-ids #{(:id file)}) + (assoc ::bfc/output (io/output-stream output))) + include-libraries + (assoc ::bfc/export-type :include-libraries))) + output)) + +(defn- import-bundle + [profile path] + (-> th/*system* + (assoc ::bfc/project-id (:default-project-id profile)) + (assoc ::bfc/team-id (:default-team-id profile)) + (assoc ::bfc/profile-id (:id profile)) + (assoc ::bfc/input path) + (v3/import-files!) + (:file-ids))) + +(defn- read-manifest + [path] + (with-open [zip (ZipFile. (.toFile ^java.nio.file.Path path))] + (let [entry (.getEntry zip "manifest.json")] + (with-open [input (.getInputStream zip entry)] + (json/read-str (slurp input :encoding "UTF-8")))))) + +(defn- tamper-manifest-relations + [src dst relations] + (with-open [zin (ZipFile. (.toFile ^java.nio.file.Path src)) + zout (ZipOutputStream. (io/output-stream dst))] + (doseq [entry (enumeration-seq (.entries zin))] + (let [name (.getName ^ZipEntry entry)] + (if (= name "manifest.json") + (let [manifest (-> (slurp (.getInputStream zin entry) :encoding "UTF-8") + (json/read-str) + (assoc "relations" relations))] + (.putNextEntry zout (ZipEntry. name)) + (.write zout (.getBytes (json/write-str manifest) "UTF-8")) + (.closeEntry zout)) + (do + (.putNextEntry zout (ZipEntry. name)) + (with-open [input (.getInputStream zin entry)] + (io/copy input zout)) + (.closeEntry zout))))))) + +(defn- prepare-victim-file + [n] + (let [profile (th/create-profile* n) + team (th/create-team* n {:profile-id (:id profile)}) + project (th/create-project* n {:team-id (:id team) + :profile-id (:id profile)}) + file (th/create-file* n {:profile-id (:id profile) + :project-id (:id project)})] + [profile file])) + (def ^:private svg-raw-page-id (uuid/custom 1 1)) (def ^:private svg-raw-root-id (uuid/custom 3 1)) (def ^:private svg-raw-child-id (uuid/custom 3 2)) @@ -2915,3 +2976,181 @@ (t/is (thrown? IllegalArgumentException (bfc/get-file th/*system* (uuid/random) :load-data? false opt true))))) + +;; --- Library relations security tests + +(t/deftest import-v3-ignores-relations-outside-package + (let [profile (th/create-profile* 1) + file (prepare-simple-file profile) + [_ victim] (prepare-victim-file 3) + bundle (export-file file) + evil (tmp/tempfile :suffix ".zip") + _ (tamper-manifest-relations bundle evil + [[(str (:id file)) (str (:id victim))]]) + [file-id :as result] (import-bundle profile evil)] + (t/is (= 1 (count result))) + (t/is (empty? (th/db-query :file-library-rel {:file-id file-id}))) + (t/is (empty? (th/db-query :file-library-sync {:file-id file-id}))))) + +(defn- prepare-other-team + [n] + (let [profile (th/create-profile* n) + team (th/create-team* n {:profile-id (:id profile)}) + project (th/create-project* n {:team-id (:id team) + :profile-id (:id profile)}) + file (th/create-file* n {:profile-id (:id profile) + :project-id (:id project)}) + library (th/create-file* (inc n) {:profile-id (:id profile) + :project-id (:id project)})] + [profile file library])) + +(t/deftest import-v3-ignores-foreign-file-id + (let [profile (th/create-profile* 1) + file (prepare-simple-file profile) + [_ victim] (prepare-victim-file 3) + bundle (export-file file) + evil (tmp/tempfile :suffix ".zip") + _ (tamper-manifest-relations bundle evil + [[(str (:id victim)) (str (:id file))]]) + result (import-bundle profile evil)] + (t/is (= 1 (count result))) + (t/is (empty? (th/db-query :file-library-rel {:file-id (:id victim)}))) + (t/is (empty? (th/db-query :file-library-sync {:file-id (:id victim)}))))) + +(t/deftest import-v3-ignores-unknown-library + (let [profile (th/create-profile* 1) + file (prepare-simple-file profile) + bundle (export-file file) + evil (tmp/tempfile :suffix ".zip") + _ (tamper-manifest-relations bundle evil + [[(str (:id file)) (str (uuid/random))]]) + [file-id :as result] (import-bundle profile evil)] + (t/is (= 1 (count result))) + (t/is (empty? (th/db-query :file-library-rel {:file-id file-id}))))) + +(t/deftest import-v3-ignores-self-library-relation + (let [profile (th/create-profile* 1) + file (prepare-simple-file profile) + bundle (export-file file) + evil (tmp/tempfile :suffix ".zip") + _ (tamper-manifest-relations bundle evil + [[(str (:id file)) (str (:id file))]]) + [file-id :as result] (import-bundle profile evil)] + (t/is (= 1 (count result))) + (t/is (empty? (th/db-query :file-library-rel {:file-id file-id}))))) + +(t/deftest import-v3-keeps-valid-library-relation + (let [profile (th/create-profile* 1) + file (prepare-simple-file profile) + library (th/create-file* 2 {:profile-id (:id profile) + :project-id (:default-project-id profile)})] + (th/link-file-to-library* {:file-id (:id file) + :library-id (:id library)}) + (let [bundle (export-file file :include-libraries true) + result (import-bundle profile bundle) + new-file (->> result + (map #(bfc/get-file th/*system* %)) + (some #(when (= (:name %) (:name file)) %)))] + (t/is (= 2 (count result))) + (t/is (some? new-file)) + (t/is (= 1 (count (th/db-query :file-library-rel + {:file-id (:id new-file)}))))))) + +(t/deftest export-multi-file-includes-package-relations + (let [profile (th/create-profile* 1) + file (prepare-simple-file profile) + library (th/create-file* 2 {:profile-id (:id profile) + :project-id (:default-project-id profile)}) + [_ other-file other-library] (prepare-other-team 3) + _ (th/link-file-to-library* {:file-id (:id file) + :library-id (:id library)}) + _ (th/link-file-to-library* {:file-id (:id other-file) + :library-id (:id other-library)}) + _ (th/db-insert! :file-library-rel + {:file-id (:id file) + :library-file-id (:id other-library)}) + output (tmp/tempfile :suffix ".zip")] + (v3/export-files! + (-> th/*system* + (assoc ::bfc/file-ids #{(:id file) (:id other-file)}) + (assoc ::bfc/export-type :include-libraries) + (assoc ::bfc/output (io/output-stream output)))) + (let [rels (set (map vec (get (read-manifest output) "relations")))] + (t/is (contains? rels [(str (:id file)) (str (:id library))])) + (t/is (contains? rels [(str (:id other-file)) (str (:id other-library))])) + (t/is (contains? rels [(str (:id file)) (str (:id other-library))]))))) + +;; --- Forbidden library relations cleanup (migration candidate) +;; +;; The two statements below are the exact cleanup proposed for a future +;; migration (see tmp/CHECK-LIBRARY-RELATIONS.sql, section 6). The sync +;; rows must go first: the second statement reads file_library_rel +;; while it still contains the forbidden rows. + +(def ^:private sql:delete-forbidden-library-sync + "WITH forbidden AS ( + SELECT flr.file_id, flr.library_file_id + FROM file_library_rel flr + JOIN file f ON f.id = flr.file_id + JOIN project fp ON fp.id = f.project_id + JOIN file l ON l.id = flr.library_file_id + JOIN project lp ON lp.id = l.project_id + WHERE fp.team_id <> lp.team_id + OR flr.file_id = flr.library_file_id + ) + DELETE FROM file_library_sync fls + USING forbidden + WHERE fls.file_id = forbidden.file_id + AND fls.library_file_id = forbidden.library_file_id") + +(def ^:private sql:delete-forbidden-library-rel + "WITH forbidden AS ( + SELECT flr.file_id, flr.library_file_id + FROM file_library_rel flr + JOIN file f ON f.id = flr.file_id + JOIN project fp ON fp.id = f.project_id + JOIN file l ON l.id = flr.library_file_id + JOIN project lp ON lp.id = l.project_id + WHERE fp.team_id <> lp.team_id + OR flr.file_id = flr.library_file_id + ) + DELETE FROM file_library_rel flr + USING forbidden + WHERE flr.file_id = forbidden.file_id + AND flr.library_file_id = forbidden.library_file_id") + +(t/deftest cleanup-deletes-forbidden-library-relations + (let [profile (th/create-profile* 1) + file (prepare-simple-file profile) + library (th/create-file* 2 {:profile-id (:id profile) + :project-id (:default-project-id profile)}) + [_ victim] (prepare-victim-file 3) + _ (th/link-file-to-library* {:file-id (:id file) + :library-id (:id library)}) + _ (th/db-insert! :file-library-rel + {:file-id (:id file) + :library-file-id (:id victim)}) + _ (th/db-insert! :file-library-sync + {:file-id (:id file) + :library-file-id (:id victim)}) + _ (th/db-insert! :file-library-rel + {:file-id (:id file) + :library-file-id (:id file)}) + _ (th/db-insert! :file-library-sync + {:file-id (:id file) + :library-file-id (:id file)})] + (t/is (= 3 (count (th/db-query :file-library-rel {:file-id (:id file)})))) + (th/db-exec! [sql:delete-forbidden-library-sync]) + (th/db-exec! [sql:delete-forbidden-library-rel]) + (t/testing "same-team relation and its sync row survive" + (t/is (= [{:file-id (:id file) :library-file-id (:id library)}] + (map #(select-keys % [:file-id :library-file-id]) + (th/db-query :file-library-rel {:file-id (:id file)})))) + (t/is (= [{:file-id (:id file) :library-file-id (:id library)}] + (map #(select-keys % [:file-id :library-file-id]) + (th/db-query :file-library-sync {:file-id (:id file)}))))) + (t/testing "cross-team and self relations and their sync rows are gone" + (t/is (empty? (th/db-query :file-library-rel {:library-file-id (:id victim)}))) + (t/is (empty? (th/db-query :file-library-sync {:library-file-id (:id victim)}))) + (t/is (empty? (th/db-query :file-library-rel {:library-file-id (:id file)}))) + (t/is (empty? (th/db-query :file-library-sync {:library-file-id (:id file)})))))) diff --git a/docs/contributing-guide/libraries-templates/index.njk b/docs/contributing-guide/libraries-templates/index.njk index 2c477f433f..be4e671da0 100644 --- a/docs/contributing-guide/libraries-templates/index.njk +++ b/docs/contributing-guide/libraries-templates/index.njk @@ -12,5 +12,5 @@ desc: Contribute to Penpot's libraries & templates! Learn how to share your file diff --git a/library/CHANGES.md b/library/CHANGES.md index b1f7acdc70..61112d6081 100644 --- a/library/CHANGES.md +++ b/library/CHANGES.md @@ -1,5 +1,10 @@ # CHANGELOG +## 1.3.0 + +- Update deps and build it with penpot 2.19.0 + + ## 1.2.0 - Update deps and build it with penpot 2.18.3 diff --git a/library/package.json b/library/package.json index 6b1911e2bd..0ddb8634a7 100644 --- a/library/package.json +++ b/library/package.json @@ -1,6 +1,6 @@ { "name": "@penpot/library", - "version": "1.2.0", + "version": "1.3.0", "license": "MPL-2.0", "author": "KALEIDOS SUBSIDIARY SL", "packageManager": "pnpm@12.16.1+sha512.091d1dee7927575661b25dab3c2ae31cafc1c54d6b71ed6a6600cf0c1cb10434a447eae14015f1b23e2e6bedeed79df94a932e2164d2832280dce3fb32f656d1", diff --git a/mcp/package.json b/mcp/package.json index 97012b38b0..d1d1fcc72a 100644 --- a/mcp/package.json +++ b/mcp/package.json @@ -1,6 +1,6 @@ { "name": "@penpot/mcp", - "version": "2.17.0", + "version": "2.19.0", "description": "MCP server for Penpot integration", "license": "MPL-2.0", "bin": {