mirror of
https://github.com/penpot/penpot.git
synced 2026-11-03 16:40:44 +00:00
00dd02b216
`file_change` is the biggest relation of the database: 683 GB in production, of which almost all the TOAST is dead space (the legacy data is already migrated to `file_data`; what is left is the xlog of the revisions, 4.4 kB on average). It grows with every save of a file and every version, and its rows are spread over the whole table, so every query that filters by a file has to walk all of it. The table is now partitioned by `HASH (file_id)` with 16 partitions, the same bounds as `file_data`, so the queries that filter by a file can prune to one partition and the join between the two tables is resolved partition by partition. The primary key has to include the partition key, so it becomes `(file_id, id)`, and every access path of the table filters by `file_id` accordingly: the snapshot commands require it (2.19), the frontend sends it, and the garbage collector locks the pending rows by their `(file_id, id)` pair. The migration creates the partitioned table with the final schema and copies the rows, keeping the old one aside as `file_change_legacy` for the rollback. The copy happens inside the transaction of the migration, so it does not fit a big instance: there the migration is marked as applied and the procedure of `operations/` does it by hand, in phases, driven by the `updated_at` watermark. Along the way: - every mutation of the table moves `updated_at` to the moment of the mutation, so the phased copy does not lose the rows changed while it runs. It is never set to the `deleted_at` of the row, which for the system snapshots is a date in the future. - a new `kind` column tells the snapshots (1) and the revisions (2) of the table apart, and the two paths that insert in it fill it. It has no default on purpose, so an insert that forgets it fails instead of storing the wrong kind silently. It is a reserve: the queries still tell a snapshot from a revision by its label, and no constraint ties the two together. - the revision does not repeat the features of the file anymore: they were 348 bytes written on every save and nobody read them from there. - the legacy file data backend is gone, and with it the `data`, `data_ref_id` and `data_backend` columns, which nobody wrote anymore. - the migration copies the legacy blobs of the snapshots to `file_data` before the column goes away, and stops when a snapshot with legacy data is left without its blob: the backend is better down than losing content. - the foreign key of the legacy table on `file_id` goes away with the copy: it is `NO ACTION`, so while the legacy table has rows, the objects GC cannot delete the files they point to. - the quota of the snapshots of a team resolves the files of the team first and counts their versions with `= ANY`, instead of joining the team with every version of the table. - the pool sets `enable_partitionwise_join` on every connection, so the join between the two partitioned tables is resolved partition by partition. - the snapshot lookup reports an unknown snapshot with its own error instead of the generic "object not found" of the database helper, and the decode script fails instead of printing an empty JSON. - the procedure of `operations/` stops the backend and the workers during the cutover: the pool keeps server-side prepared statements, and a prepared statement holds the OID of the table it was prepared with, so a connection would keep writing to the legacy table after the rename. - the procedure creates the partitions with their final name, so the result is the schema a new installation gets, and its verification checks the names and not only how many there are. - the index of the system snapshots is gone: it filtered them by `deleted_at IS NULL`, and they always carry a deletion date in the future, so it never matched a row. - the configuration guide, the memories and the docstring of the decode script say that the `legacy-db` backend is gone, instead of offering a value that takes the service down. - a test fixes the schema the migration leaves: the 16 partitions with their name, the primary key, the foreign keys, the check of the kind, the six indexes, the absence of the legacy columns, and that an insert without a kind or with an unknown one fails. Closes #12416 AI-assisted-by: glm-5.3-flash AI-assisted-by: minimax-m3.1-flash
366 lines
13 KiB
Clojure
366 lines
13 KiB
Clojure
;; This Source Code Form is subject to the terms of the Mozilla Public
|
|
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
|
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
|
;;
|
|
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
|
|
|
(ns app.plugins.file
|
|
(:require
|
|
[app.common.data :as d]
|
|
[app.common.data.macros :as dm]
|
|
[app.common.files.validate :as cfv]
|
|
[app.common.time :as ct]
|
|
[app.common.uuid :as uuid]
|
|
[app.config :as cf]
|
|
[app.main.data.exports.files :as exports.files]
|
|
[app.main.data.plugins :as dp]
|
|
[app.main.data.workspace :as dw]
|
|
[app.main.data.workspace.versions :as dwv]
|
|
[app.main.repo :as rp]
|
|
[app.main.store :as st]
|
|
[app.main.worker :as mw]
|
|
[app.plugins.format :as format]
|
|
[app.plugins.page :as page]
|
|
[app.plugins.parser :as parser]
|
|
[app.plugins.register :as r]
|
|
[app.plugins.system-events :as se]
|
|
[app.plugins.user :as user]
|
|
[app.plugins.utils :as u]
|
|
[app.util.http :as http]
|
|
[app.util.object :as obj]
|
|
[beicon.v2.core :as rx]))
|
|
|
|
(defn file-version-proxy?
|
|
[proxy]
|
|
(obj/type-of? proxy "FileVersionProxy"))
|
|
|
|
(defn file-version-proxy
|
|
[plugin-id file-id users data]
|
|
(let [data (atom data)]
|
|
(obj/reify {:name "FileVersionProxy"}
|
|
:$plugin {:enumerable false :get (fn [] plugin-id)}
|
|
:$file {:enumerable false :get (fn [] file-id)}
|
|
|
|
:label
|
|
{:get #(:label @data)
|
|
:set
|
|
(fn [value]
|
|
(cond
|
|
(not (r/check-permission plugin-id "content:write"))
|
|
(u/not-valid plugin-id :label "Plugin doesn't have 'content:write' permission")
|
|
|
|
(or (not (string? value)) (empty? value))
|
|
(u/not-valid plugin-id :label value)
|
|
|
|
:else
|
|
(do (swap! data assoc :label value :created-by "user")
|
|
(->> (rp/cmd! :update-file-snapshot {:id (:id @data) :file-id file-id :label value})
|
|
(rx/take 1)
|
|
(rx/subs! #(st/emit! (se/event plugin-id "rename-version" :file-id file-id)))))))}
|
|
|
|
:createdBy
|
|
{:get
|
|
(fn []
|
|
(when (r/check-permission plugin-id "user:read")
|
|
(when-let [user-data (get users (:profile-id @data))]
|
|
(user/user-proxy plugin-id user-data))))}
|
|
|
|
:createdAt
|
|
{:get #(:created-at @data)}
|
|
|
|
:isAutosave
|
|
{:get #(= "system" (:created-by @data))}
|
|
|
|
:restore
|
|
(fn []
|
|
(js/Promise.
|
|
(fn [resolve reject]
|
|
(cond
|
|
(not (r/check-permission plugin-id "content:write"))
|
|
(u/reject-not-valid reject :restore "Plugin doesn't have 'content:write' permission")
|
|
|
|
:else
|
|
(let [version-id (get @data :id)]
|
|
(st/emit!
|
|
(dwv/restore-version-from-plugin file-id version-id resolve reject)
|
|
(se/event plugin-id "restore-version" :file-id file-id)))))))
|
|
|
|
:remove
|
|
(fn []
|
|
(js/Promise.
|
|
(fn [resolve reject]
|
|
(cond
|
|
(not (r/check-permission plugin-id "content:write"))
|
|
(u/reject-not-valid reject :remove "Plugin doesn't have 'content:write' permission")
|
|
|
|
:else
|
|
(let [version-id (:id @data)]
|
|
(->> (rp/cmd! :delete-file-snapshot {:id version-id :file-id file-id})
|
|
(rx/map (constantly nil))
|
|
(rx/subs! resolve reject)))))))
|
|
|
|
:pin
|
|
(fn []
|
|
(js/Promise.
|
|
(fn [resolve reject]
|
|
(cond
|
|
(not (r/check-permission plugin-id "content:write"))
|
|
(u/reject-not-valid reject :pin "Plugin doesn't have 'content:write' permission")
|
|
|
|
(not= "system" (:created-by @data))
|
|
(u/reject-not-valid reject :pin "Only auto-saved versions can be pinned")
|
|
|
|
:else
|
|
(let [params {:id (:id @data)
|
|
:file-id file-id
|
|
:label (ct/format-inst (:created-at @data) :localized-date)}]
|
|
(->> (rx/zip (rp/cmd! :get-team-users {:file-id file-id})
|
|
(rp/cmd! :update-file-snapshot params))
|
|
(rx/subs!
|
|
(fn [[users data]]
|
|
(let [users (d/index-by :id users)]
|
|
(st/emit! (se/event plugin-id "pin-version" :file-id file-id))
|
|
(resolve (file-version-proxy plugin-id file-id users @data))))
|
|
reject))))))))))
|
|
|
|
(defn file-proxy? [p]
|
|
(obj/type-of? p "FileProxy"))
|
|
|
|
(defn file-proxy
|
|
[plugin-id id]
|
|
|
|
(obj/reify {:name "FileProxy"}
|
|
:$plugin {:enumerable false :get (fn [] plugin-id)}
|
|
:$id {:enumerable false :get (fn [] id)}
|
|
|
|
:id
|
|
{:get #(format/format-id id)}
|
|
|
|
:name
|
|
{:get #(-> (u/locate-file id) :name)}
|
|
|
|
:revn
|
|
{:get #(-> (u/locate-file id) :revn)}
|
|
|
|
:pages
|
|
{:this true
|
|
:get #(.getPages ^js %)}
|
|
|
|
:getPages
|
|
(fn []
|
|
(let [file (u/locate-file id)]
|
|
(apply array (sequence (map #(page/page-proxy plugin-id id %)) (dm/get-in file [:data :pages])))))
|
|
|
|
;; Run referential-integrity validation on the file and return the errors
|
|
;; found (an empty array means the file is valid). Each error carries the
|
|
;; validation `code`, a `hint`, and the offending `shapeId`/`pageId`.
|
|
:validate
|
|
(fn []
|
|
(let [file (u/locate-file id)
|
|
libraries (get @st/state :files)]
|
|
(try
|
|
(->> (cfv/validate-file file libraries)
|
|
(map (fn [{:keys [code hint shape-id page-id]}]
|
|
#js {:code (name code)
|
|
:hint hint
|
|
:shapeId (some-> shape-id str)
|
|
:pageId (some-> page-id str)}))
|
|
(apply array))
|
|
(catch :default cause
|
|
#js [#js {:code "validation-error"
|
|
:hint (ex-message cause)
|
|
:shapeId nil
|
|
:pageId nil}]))))
|
|
|
|
;; Plugin data
|
|
:getPluginData
|
|
(fn [key]
|
|
(cond
|
|
(not (string? key))
|
|
(u/not-valid plugin-id :getPluginData-key key)
|
|
|
|
:else
|
|
(let [file (u/locate-file id)]
|
|
(dm/get-in file [:data :plugin-data (keyword "plugin" (str plugin-id)) key]))))
|
|
|
|
:setPluginData
|
|
(fn [key value]
|
|
(cond
|
|
(or (not (string? key)) (empty? key))
|
|
(u/not-valid plugin-id :setPluginData-key key)
|
|
|
|
(not (string? value))
|
|
(u/not-valid plugin-id :setPluginData-value value)
|
|
|
|
(not (r/check-permission plugin-id "content:write"))
|
|
(u/not-valid plugin-id :setPluginData "Plugin doesn't have 'content:write' permission")
|
|
|
|
:else
|
|
(st/emit! (dp/set-plugin-data id :file (keyword "plugin" (str plugin-id)) key value))))
|
|
|
|
:getPluginDataKeys
|
|
(fn []
|
|
(let [file (u/locate-file id)]
|
|
(apply array (keys (dm/get-in file [:data :plugin-data (keyword "plugin" (dm/str plugin-id))])))))
|
|
|
|
:getSharedPluginData
|
|
(fn [namespace key]
|
|
(cond
|
|
(not (string? namespace))
|
|
(u/not-valid plugin-id :getSharedPluginData-namespace namespace)
|
|
|
|
(not (string? key))
|
|
(u/not-valid plugin-id :getSharedPluginData-key key)
|
|
|
|
:else
|
|
(let [file (u/locate-file id)]
|
|
(dm/get-in file [:data :plugin-data (keyword "shared" namespace) key]))))
|
|
|
|
:setSharedPluginData
|
|
(fn [namespace key value]
|
|
(cond
|
|
(or (not (string? namespace)) (empty? namespace))
|
|
(u/not-valid plugin-id :setSharedPluginData-namespace namespace)
|
|
|
|
(or (not (string? key)) (empty? key))
|
|
(u/not-valid plugin-id :setSharedPluginData-key key)
|
|
|
|
(not (string? value))
|
|
(u/not-valid plugin-id :setSharedPluginData-value value)
|
|
|
|
(not (r/check-permission plugin-id "content:write"))
|
|
(u/not-valid plugin-id :setSharedPluginData "Plugin doesn't have 'content:write' permission")
|
|
|
|
:else
|
|
(st/emit! (dp/set-plugin-data id :file (keyword "shared" namespace) key value))))
|
|
|
|
:getSharedPluginDataKeys
|
|
(fn [namespace]
|
|
(cond
|
|
(not (string? namespace))
|
|
(u/not-valid plugin-id :getSharedPluginDataKeys namespace)
|
|
|
|
:else
|
|
(let [file (u/locate-file id)]
|
|
(apply array (keys (dm/get-in file [:data :plugin-data (keyword "shared" namespace)]))))))
|
|
|
|
:createPage
|
|
(fn []
|
|
(cond
|
|
(not (r/check-permission plugin-id "content:write"))
|
|
(u/not-valid plugin-id :createPage "Plugin doesn't have 'content:write' permission")
|
|
|
|
:else
|
|
(let [page-id (uuid/next)]
|
|
(st/emit! (-> (dw/create-page {:page-id page-id :file-id id})
|
|
(se/add-event plugin-id)))
|
|
(page/page-proxy plugin-id id page-id))))
|
|
|
|
:export
|
|
(fn [format type]
|
|
(js/Promise.
|
|
(fn [resolve reject]
|
|
(let [type (or (parser/parse-keyword type) :all)
|
|
;; Backward compatibility: convert old values to new
|
|
type (case type
|
|
:all :include-libraries
|
|
:merge :merge-libraries
|
|
:detach :detach-libraries
|
|
type)]
|
|
(cond
|
|
(and (some? format) (not (contains? #{"penpot" "zip"} format)))
|
|
(u/reject-not-valid reject :format (dm/str "Invalid format: " format))
|
|
|
|
(not (contains? (set exports.files/valid-types) type))
|
|
(u/reject-not-valid reject :format (dm/str "Invalid type: " type))
|
|
|
|
:else
|
|
(let [file (u/locate-file id)
|
|
features (:features @st/state)
|
|
team-id (:current-team-id @st/state)
|
|
format (case format
|
|
"zip" :legacy-zip
|
|
|
|
(if (contains? cf/flags :export-file-v3)
|
|
:binfile-v3
|
|
:binfile-v1))]
|
|
(->> (mw/ask-many!
|
|
{:cmd :export-files
|
|
:format format
|
|
:type type
|
|
:team-id team-id
|
|
:features features
|
|
:files [file]})
|
|
(rx/mapcat
|
|
(fn [msg]
|
|
(.log js/console msg)
|
|
(case (:type msg)
|
|
:error
|
|
(rx/throw (ex-info "cannot export file" {:type :export-file}))
|
|
|
|
:progress
|
|
(rx/empty)
|
|
|
|
:finish
|
|
(http/send! {:method :get
|
|
:uri (:uri msg)
|
|
:mode :no-cors
|
|
:response-type :buffer}))))
|
|
(rx/take 1)
|
|
(rx/map #(js/Uint8Array. (:body %)))
|
|
(rx/tap #(st/emit! (se/event plugin-id "export-binary-files" :format format :type type)))
|
|
(rx/subs! resolve reject))))))))
|
|
:findVersions
|
|
(fn [criteria]
|
|
(let [user (obj/get criteria "createdBy" nil)]
|
|
(js/Promise.
|
|
(fn [resolve reject]
|
|
(cond
|
|
(not (r/check-permission plugin-id "content:read"))
|
|
(u/reject-not-valid reject :findVersions "Plugin doesn't have 'content:read' permission")
|
|
|
|
(and (some? user) (not (user/user-proxy? user)))
|
|
(u/reject-not-valid reject :findVersions-user "Created by user is not a valid user object")
|
|
|
|
:else
|
|
(->> (rx/zip (rp/cmd! :get-team-users {:file-id id})
|
|
(rp/cmd! :get-file-snapshots {:file-id id}))
|
|
(rx/take 1)
|
|
(rx/subs!
|
|
(fn [[users snapshots]]
|
|
(let [users (d/index-by :id users)]
|
|
(->> snapshots
|
|
(filter #(or (not (obj/get user "id"))
|
|
(= (dm/str (:profile-id %))
|
|
(obj/get user "id"))))
|
|
(map #(file-version-proxy plugin-id id users %))
|
|
(sequence)
|
|
(apply array)
|
|
(resolve))))
|
|
reject)))))))
|
|
|
|
:saveVersion
|
|
(fn [label]
|
|
(let [users-promise
|
|
(js/Promise.
|
|
(fn [resolve reject]
|
|
(->> (rp/cmd! :get-team-users {:file-id id})
|
|
(rx/subs! resolve reject))))
|
|
|
|
create-version-promise
|
|
(js/Promise.
|
|
(fn [resolve reject]
|
|
(cond
|
|
(not (r/check-permission plugin-id "content:write"))
|
|
(u/reject-not-valid reject :saveVersion "Plugin doesn't have 'content:write' permission")
|
|
|
|
:else
|
|
(st/emit!
|
|
(dwv/create-version-from-plugins id label resolve reject)
|
|
(se/event plugin-id "create-version" :file-id id)))))]
|
|
(-> (js/Promise.all #js [users-promise create-version-promise])
|
|
(.then
|
|
(fn [[users data]]
|
|
(let [users (d/index-by :id users)]
|
|
(file-version-proxy plugin-id id users data)))))))))
|