mirror of
https://github.com/penpot/penpot.git
synced 2026-11-17 07:30:35 +00:00
* ✨ Partition file_change by file_id Partition `file_change` by `HASH (file_id)` in 16 partitions, with the same bounds as `file_data`, so a join between them is resolved partition by partition. The primary key becomes `(file_id, id)`, which a partitioned table needs, and every access filters by `file_id`. The new table carries a `kind` column (1 snapshot, 2 revision) as a reserve: the queries still tell them apart by the label, and `kind` takes over when they are rewritten. The payload of a snapshot stays in `file_data`; the `data` column is kept for reading old `legacy-db` payloads. The four snapshot commands require `file-id`, because it prunes the partition. The objects GC locks and deletes by the pair, in batches. The old table is kept as `file_change_legacy` as the way back. The automatic migration refuses to run above 10 GiB or 2M rows and points to the runbook; big instances partition online with that runbook, which copies by id, follows the `updated_at` watermark in delta passes and checks that no row of the source is missing from the target. Closes #12416 AI-assisted-by: deepseek-v4.1-flash * 🐛 Address the review of the file_change partition The production runbook promised a rollback it did not have: the section was a sketch, and its statements could write to the table about to be dropped. The rollback is removed, and so is every comment that described it. `file_change_legacy` stays as a frozen copy of the previous layout. The cutover check now aborts the transaction when a row of the old table is missing in the new one, instead of leaving the decision to the operator. The five secondary indexes of the new table are built after the last delta pass: the bulk and the delta passes only need the primary key, so the batches no longer maintain them. The copy of 0158 is now replayed in a test against the pre-0158 shape with a labelled and an unlabelled row, plus an empty instance that must leave no legacy table behind. The updated_at assertion of the snapshot mutations is bounded by the test clock, and the repair revision stops writing the `features` of the file, like the normal save. AI-assisted-by: deepseek-v4.1-flash * 📎 Fix linter issues * 🐛 Fix stale helper call in rpc_file_snapshot_test The suite does not load: `th/run-pending-tasks!` does not exist in backend-tests.helpers. The pending jobs runner is `run-pending-jobs` (no arguments), the name every other call site of the suite uses. AI-assisted-by: inkling-2-small-free * 📚 Align the partition change with the 2.20.0 milestone 2.19.0 is already shipped; this work targets 2.20.0. The operations file of the partition procedure carries the 2.20.0 prefix, the references that point at it (the migration, the runbook table and the memory) follow, and the RPC doc/changes tags of the snapshot commands record 2.20 as the release that makes `file-id` required. AI-assisted-by: inkling-2-small-free * 🐛 Fix review findings on the partition rebase The 0157 legacy test was missed by the rebase and inserted file_change rows without the new mandatory `kind` column, so the backend suite fails against the migrated schema. The inserts now carry `:kind` (snapshot-kind for the labelled rows, revision-kind for the rest). The post-cutover verification block of the runbook terminates the first query with a semicolon so the block pastes into psql without a syntax error. The guard message of the migration names the step (0158-partition-file-change) instead of "0158", ambiguous since two migrations share that number. Closes #12416 AI-assisted-by: inkling-2-small-free * 🐛 Fix the stale label in the snapshot restore test The rename block of snapshots-crud runs before the restore block, so the restored version is already named "label1-updated" when the restore publishes its message. The command names the version by the label the row has at that moment, not by the one it had when it was created, so the test was asserting a stale value. Assert the current label, and say why in the comment. AI-assisted-by: deepseek-v4.1-flash * ♻️ Split the version event tests into one test per command The workspace version events and the plugin proxy send the file-id of the open file on every snapshot command, because the command needs it to prune the partition of file_change. Both tests checked that with a doseq and t/testing blocks, so a failure named the whole group and not the command that broke. Give each command its own test. The workspace ones share a helper that runs the event and returns the command and params it sends; the plugin ones keep their own mock of rp/cmd!. AI-assisted-by: deepseek-v4.1-flash
213 lines
8.9 KiB
Clojure
213 lines
8.9 KiB
Clojure
;; This Source Code Form is subject to the terms of the Mozilla Public
|
|
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
|
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
|
;;
|
|
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
|
|
|
(ns backend-tests.file-change-retention-test
|
|
"Retention of `file_change` rows through the file lifecycle.
|
|
|
|
A file has two kinds of row. Durable history: the snapshots, which a restore
|
|
brings back. And ephemeral revisions: the xlog of an edit, born already
|
|
scheduled for deletion one hour later, which nobody needs once the sessions
|
|
that were going to read them are gone. The `kind` column tells them apart.
|
|
|
|
The invariant under test is that a deletion date is assigned once and never
|
|
moved: restoring a file must not unmark a revision, and deleting a file must
|
|
not push a revision's date out to the trash lifetime. When either happens the
|
|
revision becomes invisible to `objects-gc`, which only looks at rows that
|
|
carry a date, and it stays in the table for ever."
|
|
(:require
|
|
[app.common.time :as ct]
|
|
[app.common.uuid :as uuid]
|
|
[app.db :as db]
|
|
[app.features.file-changes :as fchanges]
|
|
[app.rpc :as-alias rpc]
|
|
[app.rpc.commands.files :as files]
|
|
[app.util.blob :as blob]
|
|
[backend-tests.helpers :as th]
|
|
[clojure.test :as t]))
|
|
|
|
(t/use-fixtures :once th/state-init)
|
|
(t/use-fixtures :each th/database-reset)
|
|
|
|
(def ^:private test-now
|
|
"The instant every test pins the clock to. Whole seconds on purpose: a value
|
|
with a sub-second part does not survive the round trip through the database,
|
|
and the test could not compare it for equality."
|
|
(ct/inst "2026-10-01T00:00:00Z"))
|
|
|
|
(def ^:private revision-lifetime
|
|
"The lifetime the `update-file` flow gives a revision when it inserts it."
|
|
(ct/duration {:hours 1}))
|
|
|
|
(defn- create-file
|
|
[i]
|
|
(let [profile (th/create-profile* i {:is-active true})]
|
|
[profile (th/create-file* i {:profile-id (:id profile)
|
|
:project-id (:default-project-id profile)})]))
|
|
|
|
(defn- insert-change
|
|
"Inserts a `file_change` row directly, so the test builds the exact shape it
|
|
needs. The row class is decided by what the caller passes: a revision carries
|
|
only `changes`, a snapshot carries `label`, and a legacy payload row carries
|
|
`data` without a label. Returns the new row id."
|
|
[file-id {:keys [label data changes created-at deleted-at created-by]}]
|
|
(let [created-at (or created-at (ct/now))
|
|
id (uuid/next)]
|
|
(th/db-insert! :file-change
|
|
{:id id
|
|
:file-id file-id
|
|
:created-at created-at
|
|
:updated-at created-at
|
|
:deleted-at deleted-at
|
|
:revn 1
|
|
:version 0
|
|
:created-by (or created-by "system")
|
|
:label label
|
|
:kind (if label fchanges/snapshot-kind fchanges/revision-kind)
|
|
:data data
|
|
:changes changes}
|
|
{::db/return-keys false})
|
|
id))
|
|
|
|
(defn- insert-revision
|
|
"A revision exactly as the `update-file` flow writes it: no label, no data and
|
|
`changes` filled, already scheduled for deletion one hour after creation."
|
|
[file-id created-at]
|
|
(insert-change file-id {:created-at created-at
|
|
:changes (blob/encode {:foo "bar"})
|
|
:deleted-at (ct/plus created-at revision-lifetime)}))
|
|
|
|
(defn- find-change
|
|
[id]
|
|
(th/db-get :file-change {:id id} {::db/remove-deleted false}))
|
|
|
|
(defn- find-file
|
|
[id]
|
|
(th/db-get :file {:id id} {::db/remove-deleted false}))
|
|
|
|
(defn- restore-file
|
|
[file-id]
|
|
(db/tx-run! th/*system*
|
|
(fn [{:keys [::db/conn]}]
|
|
(#'files/restore-files conn [file-id]))))
|
|
|
|
(t/deftest restoring-a-file-leaves-the-revision-schedule-alone
|
|
(binding [ct/*clock* (ct/fixed-clock test-now)]
|
|
(let [[_ file] (create-file 1)
|
|
file-id (:id file)
|
|
trash-at (ct/in-future {:days 7})
|
|
old-at (ct/in-past {:hours 2})
|
|
|
|
revision-id (insert-revision file-id old-at)
|
|
revision-at (:deleted-at (find-change revision-id))
|
|
|
|
snapshot-id (insert-change file-id {:created-at old-at
|
|
:created-by "user"
|
|
:label "a snapshot"
|
|
:deleted-at trash-at})
|
|
|
|
legacy-id (insert-change file-id {:created-at old-at
|
|
:data (blob/encode {:legacy true})
|
|
:deleted-at trash-at})]
|
|
|
|
(restore-file file-id)
|
|
|
|
(t/testing "the revision keeps the schedule it was born with"
|
|
(t/is (= revision-at (:deleted-at (find-change revision-id)))))
|
|
|
|
(t/testing "the snapshot comes back"
|
|
(t/is (nil? (:deleted-at (find-change snapshot-id)))))
|
|
|
|
(t/testing "the legacy payload row keeps its date"
|
|
(t/is (= trash-at (:deleted-at (find-change legacy-id))))))))
|
|
|
|
(t/deftest deleting-a-file-does-not-push-the-revision-schedule-out
|
|
(binding [ct/*clock* (ct/fixed-clock test-now)]
|
|
(let [[_ file] (create-file 1)
|
|
file-id (:id file)
|
|
trash-at (ct/in-future {:days 7})
|
|
old-at (ct/in-past {:hours 2})
|
|
system-at (ct/in-future {:days 2})
|
|
|
|
revision-id (insert-revision file-id old-at)
|
|
revision-at (:deleted-at (find-change revision-id))
|
|
|
|
snapshot-id (insert-change file-id {:created-at old-at
|
|
:created-by "user"
|
|
:label "a snapshot"})
|
|
|
|
system-id (insert-change file-id {:created-at old-at
|
|
:label "internal/snapshot/1"
|
|
:deleted-at system-at})]
|
|
|
|
(th/run-task! :delete-object {:object :file
|
|
:deleted-at trash-at
|
|
:id file-id})
|
|
|
|
(t/testing "the revision keeps the schedule it was born with"
|
|
(t/is (= revision-at (:deleted-at (find-change revision-id)))))
|
|
|
|
(t/testing "the snapshot takes the file's deletion date"
|
|
(t/is (= (:deleted-at (find-file file-id))
|
|
(:deleted-at (find-change snapshot-id)))))
|
|
|
|
(t/testing "the system snapshot keeps the date it already had"
|
|
(t/is (= system-at (:deleted-at (find-change system-id))))))))
|
|
|
|
(t/deftest updating-a-snapshot-cannot-reach-a-revision
|
|
;; The snapshot paths clear the deletion mark of the row they touch, which is
|
|
;; only legitimate for durable history. They reach a row through
|
|
;; `sql:snapshots`, which filters `label IS NOT NULL`, and a revision never
|
|
;; carries one. This test pins that: the row has to keep the date it was born
|
|
;; with, and the RPC has to reject the id. If a future change loosens that
|
|
;; read filter, this goes red instead of the revision losing its date in
|
|
;; silence.
|
|
(binding [ct/*clock* (ct/fixed-clock test-now)]
|
|
(let [[profile file] (create-file 1)
|
|
file-id (:id file)
|
|
revision-id (insert-revision file-id (ct/in-past {:hours 2}))
|
|
revision-at (:deleted-at (find-change revision-id))
|
|
|
|
out (th/command! {::th/type :update-file-snapshot
|
|
::rpc/profile-id (:id profile)
|
|
:file-id file-id
|
|
:id revision-id
|
|
:label "renamed"})]
|
|
|
|
(t/testing "the RPC rejects the id"
|
|
(t/is (th/ex-info? (:error out)))
|
|
(t/is (= :not-found (:type (ex-data (:error out))))))
|
|
|
|
(t/testing "and the revision keeps the date it was born with"
|
|
(t/is (= revision-at (:deleted-at (find-change revision-id))))))))
|
|
|
|
(t/deftest a-deleted-then-restored-file-does-not-leak-its-revisions
|
|
;; The lifecycle that leaked 358,517 rows in production: a file is edited, it
|
|
;; ends up in the trash, it comes back, and the revisions that were scheduled
|
|
;; to die an hour after they were written have to die anyway.
|
|
(binding [ct/*clock* (ct/fixed-clock test-now)]
|
|
(let [[_ file] (create-file 1)
|
|
file-id (:id file)
|
|
|
|
revision-id (insert-revision file-id (ct/in-past {:hours 2}))
|
|
revision-at (:deleted-at (find-change revision-id))]
|
|
|
|
(th/run-task! :delete-object {:object :file
|
|
:deleted-at (ct/in-future {:days 7})
|
|
:id file-id})
|
|
|
|
(t/testing "deleting the file leaves the revision schedule alone"
|
|
(t/is (= revision-at (:deleted-at (find-change revision-id)))))
|
|
|
|
(restore-file file-id)
|
|
|
|
(t/testing "restoring the file leaves it alone too"
|
|
(t/is (= revision-at (:deleted-at (find-change revision-id)))))
|
|
|
|
(th/run-task! :objects-gc {})
|
|
|
|
(t/testing "so the collector takes it"
|
|
(t/is (nil? (find-change revision-id)))))))
|