penpot/backend/test/backend_tests/file_change_retention_test.clj
Andrey Antukh 34c702bc80
✨ Add table partitioning for file-change (#12417)
* ✨ Partition file_change by file_id

Partition `file_change` by `HASH (file_id)` in 16 partitions, with the
same bounds as `file_data`, so a join between them is resolved partition
by partition. The primary key becomes `(file_id, id)`, which a
partitioned table needs, and every access filters by `file_id`.

The new table carries a `kind` column (1 snapshot, 2 revision) as a
reserve: the queries still tell them apart by the label, and `kind`
takes over when they are rewritten. The payload of a snapshot stays in
`file_data`; the `data` column is kept for reading old `legacy-db`
payloads.

The four snapshot commands require `file-id`, because it prunes the
partition. The objects GC locks and deletes by the pair, in batches. The
old table is kept as `file_change_legacy` as the way back.

The automatic migration refuses to run above 10 GiB or 2M rows and
points to the runbook; big instances partition online with that runbook,
which copies by id, follows the `updated_at` watermark in delta passes
and checks that no row of the source is missing from the target.

Closes #12416

AI-assisted-by: deepseek-v4.1-flash

* 🐛 Address the review of the file_change partition

The production runbook promised a rollback it did not have: the section
was a sketch, and its statements could write to the table about to be
dropped. The rollback is removed, and so is every comment that described
it. `file_change_legacy` stays as a frozen copy of the previous layout.

The cutover check now aborts the transaction when a row of the old table
is missing in the new one, instead of leaving the decision to the
operator.

The five secondary indexes of the new table are built after the last
delta pass: the bulk and the delta passes only need the primary key, so
the batches no longer maintain them.

The copy of 0158 is now replayed in a test against the pre-0158 shape
with a labelled and an unlabelled row, plus an empty instance that must
leave no legacy table behind. The updated_at assertion of the snapshot
mutations is bounded by the test clock, and the repair revision stops
writing the `features` of the file, like the normal save.

AI-assisted-by: deepseek-v4.1-flash

* 📎 Fix linter issues

* 🐛 Fix stale helper call in rpc_file_snapshot_test

The suite does not load: `th/run-pending-tasks!` does not exist in
backend-tests.helpers. The pending jobs runner is `run-pending-jobs`
(no arguments), the name every other call site of the suite uses.

AI-assisted-by: inkling-2-small-free

* 📚 Align the partition change with the 2.20.0 milestone

2.19.0 is already shipped; this work targets 2.20.0. The operations
file of the partition procedure carries the 2.20.0 prefix, the
references that point at it (the migration, the runbook table and the
memory) follow, and the RPC doc/changes tags of the snapshot commands
record 2.20 as the release that makes `file-id` required.

AI-assisted-by: inkling-2-small-free

* 🐛 Fix review findings on the partition rebase

The 0157 legacy test was missed by the rebase and inserted file_change
rows without the new mandatory `kind` column, so the backend suite fails
against the migrated schema. The inserts now carry `:kind`
(snapshot-kind for the labelled rows, revision-kind for the rest).

The post-cutover verification block of the runbook terminates the
first query with a semicolon so the block pastes into psql without a
syntax error.

The guard message of the migration names the step
(0158-partition-file-change) instead of "0158", ambiguous since two
migrations share that number.

Closes #12416

AI-assisted-by: inkling-2-small-free

* 🐛 Fix the stale label in the snapshot restore test

The rename block of snapshots-crud runs before the restore block,
so the restored version is already named "label1-updated" when the
restore publishes its message. The command names the version by the
label the row has at that moment, not by the one it had when it was
created, so the test was asserting a stale value.

Assert the current label, and say why in the comment.

AI-assisted-by: deepseek-v4.1-flash

* ♻️ Split the version event tests into one test per command

The workspace version events and the plugin proxy send the file-id
of the open file on every snapshot command, because the command
needs it to prune the partition of file_change. Both tests checked
that with a doseq and t/testing blocks, so a failure named the whole
group and not the command that broke.

Give each command its own test. The workspace ones share a helper
that runs the event and returns the command and params it sends; the
plugin ones keep their own mock of rp/cmd!.

AI-assisted-by: deepseek-v4.1-flash
2026-11-10 23:27:17 +01:00

213 lines
8.9 KiB
Clojure

;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
(ns backend-tests.file-change-retention-test
"Retention of `file_change` rows through the file lifecycle.
A file has two kinds of row. Durable history: the snapshots, which a restore
brings back. And ephemeral revisions: the xlog of an edit, born already
scheduled for deletion one hour later, which nobody needs once the sessions
that were going to read them are gone. The `kind` column tells them apart.
The invariant under test is that a deletion date is assigned once and never
moved: restoring a file must not unmark a revision, and deleting a file must
not push a revision's date out to the trash lifetime. When either happens the
revision becomes invisible to `objects-gc`, which only looks at rows that
carry a date, and it stays in the table for ever."
(:require
[app.common.time :as ct]
[app.common.uuid :as uuid]
[app.db :as db]
[app.features.file-changes :as fchanges]
[app.rpc :as-alias rpc]
[app.rpc.commands.files :as files]
[app.util.blob :as blob]
[backend-tests.helpers :as th]
[clojure.test :as t]))
(t/use-fixtures :once th/state-init)
(t/use-fixtures :each th/database-reset)
(def ^:private test-now
"The instant every test pins the clock to. Whole seconds on purpose: a value
with a sub-second part does not survive the round trip through the database,
and the test could not compare it for equality."
(ct/inst "2026-10-01T00:00:00Z"))
(def ^:private revision-lifetime
"The lifetime the `update-file` flow gives a revision when it inserts it."
(ct/duration {:hours 1}))
(defn- create-file
[i]
(let [profile (th/create-profile* i {:is-active true})]
[profile (th/create-file* i {:profile-id (:id profile)
:project-id (:default-project-id profile)})]))
(defn- insert-change
"Inserts a `file_change` row directly, so the test builds the exact shape it
needs. The row class is decided by what the caller passes: a revision carries
only `changes`, a snapshot carries `label`, and a legacy payload row carries
`data` without a label. Returns the new row id."
[file-id {:keys [label data changes created-at deleted-at created-by]}]
(let [created-at (or created-at (ct/now))
id (uuid/next)]
(th/db-insert! :file-change
{:id id
:file-id file-id
:created-at created-at
:updated-at created-at
:deleted-at deleted-at
:revn 1
:version 0
:created-by (or created-by "system")
:label label
:kind (if label fchanges/snapshot-kind fchanges/revision-kind)
:data data
:changes changes}
{::db/return-keys false})
id))
(defn- insert-revision
"A revision exactly as the `update-file` flow writes it: no label, no data and
`changes` filled, already scheduled for deletion one hour after creation."
[file-id created-at]
(insert-change file-id {:created-at created-at
:changes (blob/encode {:foo "bar"})
:deleted-at (ct/plus created-at revision-lifetime)}))
(defn- find-change
[id]
(th/db-get :file-change {:id id} {::db/remove-deleted false}))
(defn- find-file
[id]
(th/db-get :file {:id id} {::db/remove-deleted false}))
(defn- restore-file
[file-id]
(db/tx-run! th/*system*
(fn [{:keys [::db/conn]}]
(#'files/restore-files conn [file-id]))))
(t/deftest restoring-a-file-leaves-the-revision-schedule-alone
(binding [ct/*clock* (ct/fixed-clock test-now)]
(let [[_ file] (create-file 1)
file-id (:id file)
trash-at (ct/in-future {:days 7})
old-at (ct/in-past {:hours 2})
revision-id (insert-revision file-id old-at)
revision-at (:deleted-at (find-change revision-id))
snapshot-id (insert-change file-id {:created-at old-at
:created-by "user"
:label "a snapshot"
:deleted-at trash-at})
legacy-id (insert-change file-id {:created-at old-at
:data (blob/encode {:legacy true})
:deleted-at trash-at})]
(restore-file file-id)
(t/testing "the revision keeps the schedule it was born with"
(t/is (= revision-at (:deleted-at (find-change revision-id)))))
(t/testing "the snapshot comes back"
(t/is (nil? (:deleted-at (find-change snapshot-id)))))
(t/testing "the legacy payload row keeps its date"
(t/is (= trash-at (:deleted-at (find-change legacy-id))))))))
(t/deftest deleting-a-file-does-not-push-the-revision-schedule-out
(binding [ct/*clock* (ct/fixed-clock test-now)]
(let [[_ file] (create-file 1)
file-id (:id file)
trash-at (ct/in-future {:days 7})
old-at (ct/in-past {:hours 2})
system-at (ct/in-future {:days 2})
revision-id (insert-revision file-id old-at)
revision-at (:deleted-at (find-change revision-id))
snapshot-id (insert-change file-id {:created-at old-at
:created-by "user"
:label "a snapshot"})
system-id (insert-change file-id {:created-at old-at
:label "internal/snapshot/1"
:deleted-at system-at})]
(th/run-task! :delete-object {:object :file
:deleted-at trash-at
:id file-id})
(t/testing "the revision keeps the schedule it was born with"
(t/is (= revision-at (:deleted-at (find-change revision-id)))))
(t/testing "the snapshot takes the file's deletion date"
(t/is (= (:deleted-at (find-file file-id))
(:deleted-at (find-change snapshot-id)))))
(t/testing "the system snapshot keeps the date it already had"
(t/is (= system-at (:deleted-at (find-change system-id))))))))
(t/deftest updating-a-snapshot-cannot-reach-a-revision
;; The snapshot paths clear the deletion mark of the row they touch, which is
;; only legitimate for durable history. They reach a row through
;; `sql:snapshots`, which filters `label IS NOT NULL`, and a revision never
;; carries one. This test pins that: the row has to keep the date it was born
;; with, and the RPC has to reject the id. If a future change loosens that
;; read filter, this goes red instead of the revision losing its date in
;; silence.
(binding [ct/*clock* (ct/fixed-clock test-now)]
(let [[profile file] (create-file 1)
file-id (:id file)
revision-id (insert-revision file-id (ct/in-past {:hours 2}))
revision-at (:deleted-at (find-change revision-id))
out (th/command! {::th/type :update-file-snapshot
::rpc/profile-id (:id profile)
:file-id file-id
:id revision-id
:label "renamed"})]
(t/testing "the RPC rejects the id"
(t/is (th/ex-info? (:error out)))
(t/is (= :not-found (:type (ex-data (:error out))))))
(t/testing "and the revision keeps the date it was born with"
(t/is (= revision-at (:deleted-at (find-change revision-id))))))))
(t/deftest a-deleted-then-restored-file-does-not-leak-its-revisions
;; The lifecycle that leaked 358,517 rows in production: a file is edited, it
;; ends up in the trash, it comes back, and the revisions that were scheduled
;; to die an hour after they were written have to die anyway.
(binding [ct/*clock* (ct/fixed-clock test-now)]
(let [[_ file] (create-file 1)
file-id (:id file)
revision-id (insert-revision file-id (ct/in-past {:hours 2}))
revision-at (:deleted-at (find-change revision-id))]
(th/run-task! :delete-object {:object :file
:deleted-at (ct/in-future {:days 7})
:id file-id})
(t/testing "deleting the file leaves the revision schedule alone"
(t/is (= revision-at (:deleted-at (find-change revision-id)))))
(restore-file file-id)
(t/testing "restoring the file leaves it alone too"
(t/is (= revision-at (:deleted-at (find-change revision-id)))))
(th/run-task! :objects-gc {})
(t/testing "so the collector takes it"
(t/is (nil? (find-change revision-id)))))))