github/penpot
1
0
Fork 0
mirror of https://github.com/penpot/penpot.git synced 2026-10-31 15:00:32 +00:00
Code Issues Packages Projects Releases Wiki Activity
penpot/frontend/test/frontend_tests/data/workspace_versions_test.cljs
Andrey Antukh 8b441be02f 💥 Partition file_change by file_id 
`file_change` is the biggest relation of the database: 683 GB in
production, of which almost all the TOAST is dead space (the legacy
data is already migrated to `file_data`; what is left is the xlog of
the revisions, 4.4 kB on average). It grows with every save of a file
and every version, and its rows are spread over the whole table, so
every query that filters by a file has to walk all of it.

The table is now partitioned by `HASH (file_id)` with 16 partitions, the
same bounds as `file_data`, so the queries that filter by a file can
prune to one partition and the join between the two tables is resolved
partition by partition. The primary key has to include the partition
key, so it becomes `(file_id, id)`, and every access path of the table
filters by `file_id` accordingly: the snapshot commands require it
(2.19), the frontend sends it, and the garbage collector locks the
pending rows by their `(file_id, id)` pair.

The migration creates the partitioned table with the final schema and
copies the rows, keeping the old one aside as `file_change_legacy` for
the rollback. The copy happens inside the transaction of the migration,
so it does not fit a big instance: there the migration is marked as
applied and the procedure of `operations/` does it by hand, in phases,
driven by the `updated_at` watermark.

Along the way:

- every mutation of the table moves `updated_at` to the moment of the
  mutation, so the phased copy does not lose the rows changed while it
  runs. It is never set to the `deleted_at` of the row, which for the
  system snapshots is a date in the future.
- a new `kind` column tells the snapshots (1) and the revisions (2) of
  the table apart, and the two paths that insert in it fill it. It has
  no default on purpose, so an insert that forgets it fails instead of
  storing the wrong kind silently. It is a reserve: the queries still
  tell a snapshot from a revision by its label, and no constraint ties
  the two together.
- the revision does not repeat the features of the file anymore: they
  were 348 bytes written on every save and nobody read them from there.
- the legacy file data backend is gone, and with it the `data`,
  `data_ref_id` and `data_backend` columns, which nobody wrote anymore.
- the migration copies the legacy blobs of the snapshots to `file_data`
  before the column goes away, and stops when a snapshot with legacy
  data is left without its blob: the backend is better down than losing
  content.
- the foreign key of the legacy table on `file_id` goes away with the
  copy: it is `NO ACTION`, so while the legacy table has rows, the
  objects GC cannot delete the files they point to.
- the quota of the snapshots of a team resolves the files of the team
  first and counts their versions with `= ANY`, instead of joining the
  team with every version of the table.
- the pool sets `enable_partitionwise_join` on every connection, so the
  join between the two partitioned tables is resolved partition by
  partition.
- the snapshot lookup reports an unknown snapshot with its own error
  instead of the generic "object not found" of the database helper, and
  the decode script fails instead of printing an empty JSON.
- the procedure of `operations/` stops the backend and the workers
  during the cutover: the pool keeps server-side prepared statements,
  and a prepared statement holds the OID of the table it was prepared
  with, so a connection would keep writing to the legacy table after
  the rename.
- the procedure creates the partitions with their final name, so the
  result is the schema a new installation gets, and its verification
  checks the names and not only how many there are.
- the index of the system snapshots is gone: it filtered them by
  `deleted_at IS NULL`, and they always carry a deletion date in the
  future, so it never matched a row.
- the configuration guide, the memories and the docstring of the decode
  script say that the `legacy-db` backend is gone, instead of offering
  a value that takes the service down.
- a test fixes the schema the migration leaves: the 16 partitions with
  their name, the primary key, the foreign keys, the check of the kind,
  the six indexes, the absence of the legacy columns, and that an insert
  without a kind or with an unknown one fails.

Closes #12416

AI-assisted-by: glm-5.3-flash
AI-assisted-by: minimax-m3.1-flash
2026-10-30 10:57:34 +01:00

220 lines
11 KiB
Clojure
Raw Permalink Blame History

;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns frontend-tests.data.workspace-versions-test
(:require
[app.common.time :as ct]
[app.common.uuid :as uuid]
[app.main.data.changes :as dch]
[app.main.data.persistence :as dps]
[app.main.data.workspace :as dw]
[app.main.data.workspace.pages :as dwpg]
[app.main.data.workspace.versions :as versions]
[app.main.refs :as refs]
[app.main.repo :as rp]
[app.util.timers :as tm]
[beicon.v2.core :as rx]
[cljs.test :as t :include-macros true]
[frontend-tests.helpers.mock :as mock]
[potok.v2.core :as ptk]))
(t/deftest version-creation-rejects-save-failure-and-timeout
(doseq [failure [:error :timeout]
operation [:create :plugin-create]]
(t/testing (str operation " after " failure)
(let [file-id (uuid/next)
resolved (atom [])
rejected (atom [])
requests (atom [])
state {:current-file-id file-id}
event (case operation
:create (versions/create-version)
:plugin-create (versions/create-version-from-plugins
file-id "Version"
#(swap! resolved conj %) #(swap! rejected conj %)))]
(with-redefs [refs/persistence (atom {:status (if (= failure :error) :error :saving)
:error {:type :persistence :code :save-permission-denied}})
rp/cmd! (mock/stub (fn [cmd _]
(swap! requests conj cmd)
(rx/of nil)))
rx/timeout (mock/stub (fn [_ fallback source]
(if (= failure :timeout) fallback source)))]
(->> (ptk/watch event state (rx/empty))
(rx/subs! (fn [_]) #(swap! rejected conj %)))
(t/is (empty? @requests) "Do not create a version without saving")
(t/is (empty? @resolved) "Do not resolve the plugin promise on failure")
(t/is (= 1 (count @rejected)))
(t/is (= (if (= failure :error) :save-permission-denied :save-timeout)
(:code (ex-data (first @rejected))))))))))
(t/deftest leaving-the-workspace-clears-the-preview-flags
(let [team-id (uuid/next)
file-id (uuid/next)
other-id (uuid/next)
output (atom [])
state (-> {:current-file-id file-id
:current-team-id team-id
:permissions {:can-edit true}
:files {file-id {:id file-id :revn 0 :vern 0}}
:workspace-global {:read-only? true :preview-id (uuid/next)}}
(as-> $ (ptk/update (dw/finalize-workspace team-id file-id) $))
(assoc :current-file-id other-id)
(assoc-in [:files other-id] {:id other-id :revn 0 :vern 0}))]
(t/is (nil? (get-in state [:workspace-global :preview-id])))
(when-let [result (ptk/watch (dch/commit-changes {:redo-changes [] :undo-changes []})
state (rx/empty))]
(->> result (rx/subs! #(swap! output conj %))))
(t/is (= 1 (count @output))
"Editing must work again on the next file opened in the tab")))
(t/deftest strict-persistence-wait-rejects-a-real-timeout
(t/async done
(mock/with-mocks
{refs/persistence (atom {:status :saving})}
(fn [finish]
(->> (dps/wait-persisted-or-error 0)
(rx/subs! (fn [_]
(t/is false "The stalled save must not succeed")
(finish))
(fn [cause]
(t/is (= :save-timeout (:code (ex-data cause))))
(finish)))))
done)))
(t/deftest strict-persistence-wait-observes-a-later-failure
(let [pstate (atom {:status :saving})
rejected (atom [])
resolved (atom [])]
(with-redefs [refs/persistence pstate]
(let [subscription (->> (dps/wait-persisted-or-error)
(rx/subs! #(swap! resolved conj %) #(swap! rejected conj %)))]
(try
(reset! pstate {:status :error :error {:code :missing-commit}})
(t/is (empty? @resolved))
(t/is (= :missing-commit (:code (ex-data (first @rejected)))))
(finally
(rx/dispose! subscription)))))))
(t/deftest failed-restore-returns-to-the-live-file-before-reporting
;; The plugin promise is rejected on a later tick, so every assertion about
;; the rejection runs once the event loop has drained.
(t/async done
(let [deferred (atom [])]
(doseq [operation [:ui :plugin]
failure [:saving :timeout :restore]
preview? [true false]]
(let [label (str operation " after " failure ", preview=" preview?)
file-id (uuid/next)
version-id (uuid/next)
backup {:id file-id :revn 10}
snapshot {:id file-id :revn 5}
rejected (atom [])
resolved (atom [])
requests (atom [])
initialized (atom [])
at-reject (atom nil)
store (ptk/store {:state (cond-> {:current-file-id file-id
:files {file-id backup}}
preview?
(assoc :files {file-id snapshot}
:workspace-global {:read-only? true :preview-id version-id}
:workspace-versions {:backup backup}))
:on-error #(swap! rejected conj %)})
reject (fn [cause]
(reset! at-reject
{:file (get-in @store [:files file-id])
:preview-id (get-in @store [:workspace-global :preview-id])})
(swap! rejected conj cause))
event (if (= operation :ui)
(#'versions/restore-version version-id)
(versions/restore-version-from-plugin
file-id version-id #(swap! resolved conj %) reject))]
(t/testing label
(with-redefs [refs/persistence (atom {:status (case failure
:saving :error :timeout :saving :saved)})
rp/cmd! (mock/stub (fn [cmd _]
(swap! requests conj cmd)
(rx/throw (ex-info "Restore failed" {:type :internal}))))
rx/timeout (mock/stub (fn [_ fallback source]
(if (= failure :timeout) fallback source)))
dwpg/initialize-page (mock/stub
(fn [_ _]
(fn [state]
(swap! initialized conj (get-in state [:files file-id]))
state)))]
(ptk/emit! store event)
(t/is (empty? @resolved))
(t/is (= (if (= failure :restore) [:restore-file-snapshot] []) @requests)
"Do not restore a version without saving")
(t/is (= backup (get-in @store [:files file-id])))
(t/is (nil? (get-in @store [:workspace-global :read-only?])))
(t/is (nil? (get-in @store [:workspace-global :preview-id])))
(t/is (nil? (get-in @store [:workspace-versions :backup])))
(t/is (= (if preview? [backup] []) @initialized))))
(swap! deferred conj
(fn []
(t/testing label
(t/is (= 1 (count @rejected)))
(when (= operation :plugin)
(t/is (= {:file backup :preview-id nil} @at-reject)
"The live file is back before the plugin promise rejects")))
(rx/dispose! store)))))
(tm/schedule 50 (fn []
(doseq [check @deferred] (check))
(done))))))
(t/deftest successful-restore-clears-preview-only-after-loading
(let [file-id (uuid/next)
state {:current-file-id file-id
:current-team-id (uuid/next)
:current-page-id (uuid/next)
:workspace-global {:read-only? true :preview-id (uuid/next)}
:workspace-versions {:backup {:id file-id}}}
stream (rx/subject)
events (atom [])
cleanup #(filter (ptk/type? ::versions/clear-preview-state) @events)
sub (->> (ptk/watch (#'versions/initialize-version) state stream)
(rx/subs! #(swap! events conj %)))]
(try
(t/is (empty? (cleanup)))
(rx/push! stream (ptk/data-event ::dw/bundle-fetched nil))
(t/is (= 1 (count (cleanup))))
(let [result (ptk/update (first (cleanup)) state)]
(t/is (nil? (get-in result [:workspace-global :read-only?])))
(t/is (nil? (get-in result [:workspace-global :preview-id])))
(t/is (nil? (get-in result [:workspace-versions :backup]))))
(finally
(rx/dispose! sub)))))
(t/deftest version-events-send-the-current-file-id
;; The snapshot RPC commands require the `file-id` (they need it for
;; partition pruning on the `file_change` table); the workspace version
;; events must always send it, taken from the current file.
(let [file-id (uuid/next)
version-id (uuid/next)
state {:current-file-id file-id
:workspace-versions {:data [{:id version-id
:created-at (ct/now)
:created-by "system"}]}}
call! (fn [event]
(let [requests (atom [])]
(with-redefs [rp/cmd! (mock/stub (fn [cmd params]
(swap! requests conj [cmd params])
(rx/of true)))]
(->> (ptk/watch event state (rx/empty))
(rx/subs! (fn [_])))
(first @requests))))]
(doseq [[cmd event] [[:update-file-snapshot (versions/rename-version version-id "name")]
[:delete-file-snapshot (versions/delete-version version-id)]
[:lock-file-snapshot (versions/lock-version version-id)]
[:unlock-file-snapshot (versions/unlock-version version-id)]
[:update-file-snapshot (versions/pin-version version-id)]]]
(t/testing (name cmd)
(let [[sent-cmd params] (call! event)]
(t/is (= cmd sent-cmd))
(t/is (= file-id (:file-id params))))))))
Reference in New Issue View Git Blame Copy Permalink