diff --git a/.serena/memories/backend/storage.md b/.serena/memories/backend/storage.md index 2cbe71126f..3e46769b56 100644 --- a/.serena/memories/backend/storage.md +++ b/.serena/memories/backend/storage.md @@ -116,6 +116,17 @@ Since `put-object!` uses backend-specific operations (`impl/resolve-backend` + ` - `fdata/upsert!` touches a storage object from incoming metadata before it stores the new row. - File snapshots use `file_data` for snapshot data and `file_change` for snapshot metadata. +## File Change + +- `file_change` is partitioned by `HASH (file_id)` in 16 partitions, with the same bounds as `file_data`, so a join between them is resolved partition by partition (`enable_partitionwise_join` is set on every pooled connection). +- Its primary key is `(file_id, id)`: a partitioned table needs the partition key in the primary key, so every access filters by `file_id`. +- It has a `kind` column: `1` is a snapshot (it has a label and its data lives in `file_data`) and `2` is a revision (it has no label and it holds the changes). The values are in `app.features.file-changes`. +- `kind` is a deliberate reserve, not a leftover: today no query filters by it, the queries still tell a snapshot from a revision by its label (`label IS NOT NULL`), and `kind` takes over when they are rewritten. No constraint ties `kind` and `label` together on purpose: the day `kind` is the source of truth, that constraint would have to go. +- A revision carries only the delta of a save (the changes). The features of the file are not repeated on it: nobody read them from there. +- A snapshot carries the features and the migrations of the file. +- It keeps the `data` column: the `legacy-db` backend is still supported (migration `0157-migrate-legacy-file-data` moved the existing blobs, but the backend keeps reading and writing that column). It has no `data_ref_id` or `data_backend` columns: nothing read them since the file storage refactor. +- It has no index for the system snapshots. The old one filtered them by `data IS NOT NULL`, and with the payload living in `file_data` the column no longer tells snapshots apart. The new one could only filter them by `deleted_at IS NULL`, but the system snapshots always carry a deletion date in the future, so it would never match a row. The query that would use it filters that date with an "or", which no partial index can serve. + ## Metrics - `bucket` is always the Penpot logical bucket (object metadata), never an S3 bucket. Unknown/absent buckets are labeled `"unknown"`. diff --git a/.serena/memories/clojure/idioms.md b/.serena/memories/clojure/idioms.md index cf85848c3a..82e45d14b3 100644 --- a/.serena/memories/clojure/idioms.md +++ b/.serena/memories/clojure/idioms.md @@ -9,4 +9,5 @@ Behaviors confirmed against the language/stdlib — do not re-derive from assump - Valid: `(defn ^:async f)`, `(defn- ^:async f)`, `(^:async fn [] …)` (the latter is what stock `t/async` generates itself). - `^:async` fns never throw synchronously (rejected promises instead); `try/catch/finally` supported; continuations are microtasks, timers and RxJS schedulers are macrotasks (FIFO). - `!` suffix allowed cases (only exceptions to the no-new-`!` rule in `mem:clojure/design-rules`): mutating public `app.db` API (`insert!`, `update!`, `delete!`, `run!`, `tx-run!`, `exec!`, ...) and fns whose contract is to work over transients (stdlib `conj!`, `assoc!`, `dissoc!`, `disj!`, `persistent!` and custom helpers over them); any further case must be registered here with justification before use. +- The threading macros insert the threaded value at DIFFERENT positions: `->` and `some->` put it FIRST (`(-> x (f a))` → `(f x a)`), `->>` and `some->>` put it LAST (`(->> x (f a))` → `(f a x)`). `some->`/`some->>` also short-circuit on nil. The nesting follows the same split: `->`/`some->` nest to the left (`(-> x (f) (g))` → `(g (f x))`), `->>`/`some->>` to the right (`(->> x (f) (g))` → `(f (g x))`). A pipeline that calls a fn whose first argument is a config/context (`app.features.fdata/resolve-file-data` is `(cfg, file)`) must use `->>`/`some->>`; with `->`/`some->` the arguments are swapped and the call silently returns the wrong value (it cost a broken `get-snapshot` in `app.features.file-snapshots`). - `cuerdas.core/index-of` returns `nil` (not `-1`) when the substring is absent, like `clojure.string/index-of`. `(pos? nil)` **throws on the JVM** but evaluates to `false` in CLJS (JS coercion), so `(if (pos? (str/index-of s "\n")) …)` works in the browser and crashes the backend. Guard with `some?` before `pos?` (this was `app.common.exceptions/first-line`). diff --git a/backend/src/app/db.clj b/backend/src/app/db.clj index d34911ff32..754026e5c5 100644 --- a/backend/src/app/db.clj +++ b/backend/src/app/db.clj @@ -108,7 +108,10 @@ (def initsql (str "SET statement_timeout = 300000;\n" - "SET idle_in_transaction_session_timeout = 300000;")) + "SET idle_in_transaction_session_timeout = 300000;\n" + ;; lets the planner resolve a join between two partitioned tables + ;; pairwise, when both are partitioned with the same bounds + "SET enable_partitionwise_join = on;")) (defn- create-datasource-config [{:keys [::uri] :as cfg}] diff --git a/backend/src/app/features/file_changes.clj b/backend/src/app/features/file_changes.clj new file mode 100644 index 0000000000..d9e9707c99 --- /dev/null +++ b/backend/src/app/features/file_changes.clj @@ -0,0 +1,20 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns app.features.file-changes + "The kinds of the rows of the `file_change` table. + + A snapshot is the state of a file at a point: it has a label and its + data lives in `file_data`. A revision is the delta of a save: it has + no label and it holds the changes.") + +(def snapshot-kind + "The kind of a snapshot row." + 1) + +(def revision-kind + "The kind of a revision row." + 2) diff --git a/backend/src/app/features/file_snapshots.clj b/backend/src/app/features/file_snapshots.clj index 9be4b1b572..24e2162ade 100644 --- a/backend/src/app/features/file_snapshots.clj +++ b/backend/src/app/features/file_snapshots.clj @@ -19,6 +19,7 @@ [app.db :as db] [app.db.sql :as-alias sql] [app.features.fdata :as fdata] + [app.features.file-changes :as fchanges] [app.storage :as sto] [app.util.blob :as blob] [app.worker :as wrk] @@ -66,36 +67,52 @@ LEFT JOIN file_data AS fd ON (fd.file_id = f.id AND fd.id = f.id) WHERE f.id = ?") +(def ^:private sql:snapshot-columns + "c.id, + c.label, + c.revn, + c.created_at, + c.modified_at, + c.deleted_at, + c.profile_id, + c.created_by, + c.locked_by, + c.features, + c.metadata, + c.migrations, + c.version, + c.file_id") + +(def ^:private sql:snapshot-visible + "AND CASE WHEN c.created_by = 'user' + THEN c.deleted_at IS NULL + WHEN c.created_by = 'system' + THEN c.deleted_at IS NULL OR c.deleted_at >= ?::timestamptz + END") + (def ^:private sql:get-snapshot-without-data (str "WITH snapshots AS (" sql:snapshots ")" - "SELECT c.id, - c.label, - c.revn, - c.created_at, - c.modified_at, - c.deleted_at, - c.profile_id, - c.created_by, - c.locked_by, - c.features, - c.metadata, - c.migrations, - c.version, - c.file_id - FROM snapshots AS c - WHERE c.id = ? - AND CASE WHEN c.created_by = 'user' - THEN c.deleted_at IS NULL - WHEN c.created_by = 'system' - THEN c.deleted_at IS NULL OR c.deleted_at >= ?::timestamptz - END")) + "SELECT " sql:snapshot-columns + " FROM snapshots AS c + WHERE c.file_id = ? + AND c.id = ? " sql:snapshot-visible)) (defn get-minimal-snapshot - [cfg snapshot-id] + "Get the minimal snapshot info (without the data blob). The query filters + by `file_id`, which lets the planner prune the partitions of `file_change` + when it chooses an index scan. Raises `:snapshot-not-found` when there is + no matching snapshot." + [cfg file-id snapshot-id] (let [now (ct/now)] - (-> (db/get-with-sql cfg [sql:get-snapshot-without-data snapshot-id now] - {::db/remove-deleted false}) - (decode-snapshot)))) + (or (-> (db/get-with-sql cfg [sql:get-snapshot-without-data file-id snapshot-id now] + {::db/remove-deleted false + ::db/throw-if-not-exists false}) + (decode-snapshot)) + (ex/raise :type :not-found + :code :snapshot-not-found + :hint "unable to find snapshot with the provided id" + :snapshot-id snapshot-id + :file-id file-id)))) (def ^:private sql:get-snapshot (str sql:snapshots @@ -109,14 +126,16 @@ (defn get-snapshot "Get a fully decoded snapshot for read-only preview or restoration. - Returns the snapshot map with decoded :data field." + Returns the snapshot map with decoded :data field, or nil when the file has + no snapshot with that id." [cfg file-id snapshot-id] (let [now (ct/now)] - (->> (db/get-with-sql cfg [sql:get-snapshot file-id snapshot-id now] - {::db/remove-deleted false}) - (decode-snapshot) - (fdata/resolve-file-data cfg) - (fdata/decode-file-data cfg)))) + (some->> (db/get-with-sql cfg [sql:get-snapshot file-id snapshot-id now] + {::db/remove-deleted false + ::db/throw-if-not-exists false}) + (decode-snapshot) + (fdata/resolve-file-data cfg) + (fdata/decode-file-data cfg)))) (def ^:private sql:get-visible-snapshots (str "WITH " @@ -259,6 +278,7 @@ (-> snapshot (update :features into-array) (update :migrations into-array) + (assoc :kind fchanges/snapshot-kind) (assoc :updated-at created-at) (assoc :profile-id profile-id) (assoc :session-id session-id) @@ -338,7 +358,7 @@ (when-not snapshot (ex/raise :type :not-found :code :snapshot-not-found - :hint "unable to find snapshot with the provided label" + :hint "unable to find snapshot with the provided id" :snapshot-id snapshot-id :file-id file-id)) @@ -413,7 +433,8 @@ :file-id file-id :id id}}) (db/update! cfg :file-change - {:deleted-at deleted-at} + {:deleted-at deleted-at + :updated-at (ct/now)} {:id id :file-id file-id} {::db/return-keys false}) true) @@ -422,19 +443,25 @@ (str sql:snapshots " AND c.file_id = ?")) (defn lock-by! - [conn id profile-id] + [conn file-id id profile-id] + (assert (uuid? file-id) "missing file-id") + (assert (uuid? id) "missing id") (-> (db/update! conn :file-change - {:locked-by profile-id} - {:id id} + {:locked-by profile-id + :updated-at (ct/now)} + {:id id :file-id file-id} {::db/return-keys false}) (db/get-update-count) (pos?))) (defn unlock! - [conn id] + [conn file-id id] + (assert (uuid? file-id) "missing file-id") + (assert (uuid? id) "missing id") (-> (db/update! conn :file-change - {:locked-by nil} - {:id id} + {:locked-by nil + :updated-at (ct/now)} + {:id id :file-id file-id} {::db/return-keys false}) (db/get-update-count) (pos?))) diff --git a/backend/src/app/migrations.clj b/backend/src/app/migrations.clj index 42d1fab650..00bcc4579d 100644 --- a/backend/src/app/migrations.clj +++ b/backend/src/app/migrations.clj @@ -529,7 +529,10 @@ :fn (mg/resource "app/migrations/sql/0156-reset-pg-storage-defaults.sql")} {:name "0157-migrate-legacy-file-data" - :fn (mg/resource "app/migrations/sql/0157-migrate-legacy-file-data.sql")}]) + :fn (mg/resource "app/migrations/sql/0157-migrate-legacy-file-data.sql")} + + {:name "0158-partition-file-change" + :fn (mg/resource "app/migrations/sql/0158-partition-file-change.sql")}]) (defn apply-migrations! [pool name migrations] diff --git a/backend/src/app/migrations/sql/0158-partition-file-change.sql b/backend/src/app/migrations/sql/0158-partition-file-change.sql new file mode 100644 index 0000000000..053983a083 --- /dev/null +++ b/backend/src/app/migrations/sql/0158-partition-file-change.sql @@ -0,0 +1,205 @@ +-- Partition `file_change` by `file_id`. +-- +-- The table grows with every file change and its TOAST holds the blobs of +-- the changes, so it is the biggest relation of the database. Partitioning it +-- by `file_id` lets the queries that filter by file prune to a single +-- partition, and lets the join with `file_data` (already partitioned with the +-- same modulus) be resolved partition by partition. +-- +-- The primary key has to include the partition key, so it becomes +-- `(file_id, id)`. Every access by id alone would lose its index, so the +-- access paths of the application filter by `file_id` too. +-- +-- The new table is created with the final schema: the columns that nobody +-- used anymore are gone (`data_ref_id` and `data_backend`, dead since the +-- file storage refactor), the `data` column is kept as-is (the `legacy-db` +-- file data backend is still supported: migration +-- 0157-migrate-legacy-file-data moved the existing blobs, but the backend +-- keeps reading and writing that column), and the new `kind` column tells +-- the snapshots and the revisions apart. +-- +-- The `kind` column is a reserve: the queries still tell a snapshot from a +-- revision by its label (`label IS NOT NULL`), and `kind` takes over when +-- they are rewritten. That is why no constraint ties the two together: the +-- day `kind` is the source of truth, the constraint would have to go. +-- +-- The old table is kept aside as `file_change_legacy`: it is the way back if +-- something goes wrong. It is dropped by hand, days later (or right here, +-- when it is empty: on a new instance there is nothing to go back to). +-- +-- This migration is not meant to run on a big instance: the copy of the +-- rows of the table happens inside the transaction of the migration. On +-- those instances mark the migration as applied and partition the table by +-- hand, with the procedure described in the runbook of the issue. + +-- 1. The legacy blobs of the snapshots were already moved to `file_data` +-- by migration 0157-migrate-legacy-file-data (on the big instances, by +-- hand, with its runbook), so there is nothing to copy here. The `data` +-- column travels with the row in step 4 below: the `legacy-db` backend +-- is still supported. + +-- 2. The current table becomes the legacy one. The names of its indexes and +-- constraints live in the schema, not in the table, so they are renamed +-- with a `_legacy` suffix to free the canonical names. +ALTER TABLE file_change RENAME TO file_change_legacy; + +ALTER TABLE file_change_legacy RENAME CONSTRAINT file_change_pkey + TO file_change_pkey_legacy; +ALTER TABLE file_change_legacy RENAME CONSTRAINT file_change_file_id_fkey + TO file_change_file_id_fkey_legacy; +ALTER TABLE file_change_legacy RENAME CONSTRAINT file_change_locked_by_fkey + TO file_change_locked_by_fkey_legacy; +ALTER TABLE file_change_legacy RENAME CONSTRAINT file_change_profile_id_fkey + TO file_change_profile_id_fkey_legacy; + +ALTER INDEX file_change__data_ref_id__idx + RENAME TO file_change__data_ref_id__idx_legacy; +ALTER INDEX file_change__deleted_at__idx + RENAME TO file_change__deleted_at__idx_legacy; +ALTER INDEX file_change__file_id__revn__idx + RENAME TO file_change__file_id__revn__idx_legacy; +ALTER INDEX file_change__locked_by__idx + RENAME TO file_change__locked_by__idx_legacy; +ALTER INDEX file_change__profile_id__idx + RENAME TO file_change__profile_id__idx_legacy; +ALTER INDEX file_change__system_snapshots__idx + RENAME TO file_change__system_snapshots__idx_legacy; + +-- This one comes from another branch, so an instance may not have it. +ALTER INDEX IF EXISTS file_change__user_snapshots__idx + RENAME TO file_change__user_snapshots__idx_legacy; + +-- 3. The new table, partitioned with the same bounds as `file_data`, so both +-- relations can be joined partition by partition. +CREATE TABLE file_change ( + file_id uuid NOT NULL, + id uuid NOT NULL DEFAULT gen_random_uuid(), + created_at timestamptz NOT NULL DEFAULT now(), + session_id uuid, + revn bigint NOT NULL DEFAULT 0, + data bytea, + changes bytea, + profile_id uuid, + features text[], + label text, + version integer, + created_by text NOT NULL DEFAULT 'system', + updated_at timestamptz DEFAULT now(), + deleted_at timestamptz, + migrations text[], + locked_by uuid, + kind smallint NOT NULL, + CONSTRAINT file_change_kind_check CHECK (kind IN (1, 2)) +) PARTITION BY HASH (file_id); + +-- the columns keep the storage of PostgreSQL by default (migration +-- 0156-reset-pg-storage-defaults dropped the explicit EXTERNAL overrides) + +COMMENT ON COLUMN file_change.kind IS + 'The kind of the row: 1 is a snapshot (it has a label and its data lives in file_data) and 2 is a revision (it has no label and it holds the changes).'; +COMMENT ON COLUMN file_change.locked_by IS + 'Profile ID of user who has locked this version. Only the creator can lock/unlock their own versions. Locked versions cannot be deleted by others.'; + +CREATE TABLE file_change_00 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 0); +CREATE TABLE file_change_01 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 1); +CREATE TABLE file_change_02 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 2); +CREATE TABLE file_change_03 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 3); +CREATE TABLE file_change_04 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 4); +CREATE TABLE file_change_05 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 5); +CREATE TABLE file_change_06 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 6); +CREATE TABLE file_change_07 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 7); +CREATE TABLE file_change_08 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 8); +CREATE TABLE file_change_09 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 9); +CREATE TABLE file_change_10 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 10); +CREATE TABLE file_change_11 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 11); +CREATE TABLE file_change_12 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 12); +CREATE TABLE file_change_13 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 13); +CREATE TABLE file_change_14 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 14); +CREATE TABLE file_change_15 PARTITION OF file_change + FOR VALUES WITH (MODULUS 16, REMAINDER 15); + +-- 4. The rows move to the partitioned table. The column list is explicit +-- because the legacy table has columns that the new one does not have +-- (`data_ref_id` and `data_backend`). +-- The kind comes from the label: a row with a label is a snapshot, a row +-- without one is a revision. +INSERT INTO file_change (id, file_id, created_at, session_id, revn, data, changes, + profile_id, features, label, version, created_by, + updated_at, deleted_at, migrations, locked_by, kind) +SELECT id, file_id, created_at, session_id, revn, data, changes, + profile_id, features, label, version, created_by, + updated_at, deleted_at, migrations, locked_by, + CASE WHEN label IS NOT NULL THEN 1 ELSE 2 END + FROM file_change_legacy; + +-- 5. The primary key includes the partition key now. +ALTER TABLE file_change ADD PRIMARY KEY (file_id, id); + +-- 6. The foreign keys. +ALTER TABLE file_change ADD FOREIGN KEY (file_id) + REFERENCES file(id) DEFERRABLE; +ALTER TABLE file_change ADD FOREIGN KEY (locked_by) + REFERENCES profile(id) ON DELETE SET NULL DEFERRABLE; +ALTER TABLE file_change ADD FOREIGN KEY (profile_id) + REFERENCES profile(id) ON DELETE SET NULL DEFERRABLE; + +-- 7. The indexes, with the canonical names. The one of the deleted rows now +-- carries the `file_id`, so the garbage collector can read the rows it has +-- to delete without touching the table. +-- +-- There is no index for the system snapshots: the old one filtered them by +-- `data IS NOT NULL`, and with the payload living in `file_data` the +-- column no longer tells snapshots apart. The new one could only filter +-- them by `deleted_at IS NULL`, but the system snapshots always carry a +-- deletion date in the future, so it would never match a row (and +-- the query that would use it filters that date with an "or", which no +-- partial index can serve). +CREATE INDEX file_change__deleted_at__idx + ON file_change (deleted_at, file_id, id) WHERE deleted_at IS NOT NULL; +CREATE INDEX file_change__file_id__revn__idx + ON file_change (file_id, revn); +CREATE INDEX file_change__locked_by__idx + ON file_change (locked_by) WHERE locked_by IS NOT NULL; +CREATE INDEX file_change__profile_id__idx + ON file_change (profile_id) WHERE profile_id IS NOT NULL; +CREATE INDEX file_change__user_snapshots__idx + ON file_change (file_id) + WHERE label IS NOT NULL + AND created_by = 'user' + AND deleted_at IS NULL; + +-- 8. The legacy table is kept: it is the way back. It is dropped by hand, +-- days after the cutover: +-- DROP TABLE file_change_legacy; +-- +-- Its foreign key on `file_id` goes away with the copy: it is `NO +-- ACTION`, so while the legacy table has rows, the objects GC cannot +-- delete a file (it deletes the rows of the file first, and the legacy +-- ones are not its business). The legacy table is frozen, and the +-- rollback adds the constraint back with `NOT VALID`. +ALTER TABLE file_change_legacy + DROP CONSTRAINT file_change_file_id_fkey_legacy; + +-- On a new instance there is nothing to go back to, so the empty table +-- is dropped here. +DO $$ +BEGIN + IF NOT EXISTS (SELECT 1 FROM file_change_legacy) THEN + DROP TABLE file_change_legacy; + END IF; +END $$; diff --git a/backend/src/app/rpc/commands/files.clj b/backend/src/app/rpc/commands/files.clj index ad5ebc3e30..3d17e2c3b6 100644 --- a/backend/src/app/rpc/commands/files.clj +++ b/backend/src/app/rpc/commands/files.clj @@ -1354,7 +1354,7 @@ ;; only looks at rows that carry a date, so unmarking one here would keep it ;; in the table for ever. The `data` half is a guard for the rows still ;; pending migration, not a dependency: drop it with the last one. - "UPDATE file_change SET deleted_at = null + "UPDATE file_change SET deleted_at = null, updated_at = now() WHERE file_id = ANY(?::uuid[]) AND (label IS NOT NULL OR data IS NOT NULL)") diff --git a/backend/src/app/rpc/commands/files_snapshot.clj b/backend/src/app/rpc/commands/files_snapshot.clj index cb3e93285f..fa2b26ffe5 100644 --- a/backend/src/app/rpc/commands/files_snapshot.clj +++ b/backend/src/app/rpc/commands/files_snapshot.clj @@ -50,6 +50,7 @@ "Retrieve a file bundle with data from a specific snapshot for read-only preview. Does not modify any database state." {::doc/added "2.16" + ::doc/changes [["2.19" "An unknown snapshot is reported with the :snapshot-not-found error code"]] ::sm/params schema:get-file-snapshot ::sm/result files/schema:file-with-permissions ::db/transaction true} @@ -151,28 +152,32 @@ (def ^:private schema:update-file-snapshot [:map {:title "update-file-snapshot"} + [:file-id ::sm/uuid] [:id ::sm/uuid] [:label ::sm/text]]) (sv/defmethod ::update-file-snapshot {::doc/added "1.20" + ::doc/changes [["2.19" "The :file-id param is now required"]] ::sm/params schema:update-file-snapshot ::db/transaction true} - [{:keys [::db/conn]} {:keys [::rpc/profile-id id label]}] - (let [snapshot (fsnap/get-minimal-snapshot conn id)] + [{:keys [::db/conn]} {:keys [::rpc/profile-id file-id id label]}] + (let [snapshot (fsnap/get-minimal-snapshot conn file-id id)] (files/check-edition-permissions! conn profile-id (:file-id snapshot)) (fsnap/update! conn (assoc snapshot :label label)))) (def ^:private schema:remove-file-snapshot [:map {:title "remove-file-snapshot"} + [:file-id ::sm/uuid] [:id ::sm/uuid]]) (sv/defmethod ::delete-file-snapshot {::doc/added "1.20" + ::doc/changes [["2.19" "The :file-id param is now required"]] ::sm/params schema:remove-file-snapshot ::db/transaction true} - [{:keys [::db/conn]} {:keys [::rpc/profile-id id]}] - (let [snapshot (fsnap/get-minimal-snapshot conn id)] + [{:keys [::db/conn]} {:keys [::rpc/profile-id file-id id]}] + (let [snapshot (fsnap/get-minimal-snapshot conn file-id id)] (files/check-edition-permissions! conn profile-id (:file-id snapshot)) (when (not= (:created-by snapshot) "user") @@ -200,14 +205,16 @@ (def ^:private schema:lock-file-snapshot [:map {:title "lock-file-snapshot"} + [:file-id ::sm/uuid] [:id ::sm/uuid]]) (sv/defmethod ::lock-file-snapshot {::doc/added "1.20" + ::doc/changes [["2.19" "The :file-id param is now required"]] ::sm/params schema:lock-file-snapshot ::db/transaction true} - [{:keys [::db/conn]} {:keys [::rpc/profile-id id]}] - (let [snapshot (fsnap/get-minimal-snapshot conn id)] + [{:keys [::db/conn]} {:keys [::rpc/profile-id file-id id]}] + (let [snapshot (fsnap/get-minimal-snapshot conn file-id id)] (files/check-edition-permissions! conn profile-id (:file-id snapshot)) (when (not= (:created-by snapshot) "user") @@ -235,18 +242,20 @@ :profile-id profile-id :locked-by (:locked-by snapshot))) - (fsnap/lock-by! conn id profile-id))) + (fsnap/lock-by! conn (:file-id snapshot) id profile-id))) (def ^:private schema:unlock-file-snapshot [:map {:title "unlock-file-snapshot"} + [:file-id ::sm/uuid] [:id ::sm/uuid]]) (sv/defmethod ::unlock-file-snapshot {::doc/added "1.20" + ::doc/changes [["2.19" "The :file-id param is now required"]] ::sm/params schema:unlock-file-snapshot ::db/transaction true} - [{:keys [::db/conn]} {:keys [::rpc/profile-id id]}] - (let [snapshot (fsnap/get-minimal-snapshot conn id)] + [{:keys [::db/conn]} {:keys [::rpc/profile-id file-id id]}] + (let [snapshot (fsnap/get-minimal-snapshot conn file-id id)] (files/check-edition-permissions! conn profile-id (:file-id snapshot)) (when (not= (:created-by snapshot) "user") @@ -273,4 +282,4 @@ :snapshot-id id :profile-id profile-id)) - (fsnap/unlock! conn id))) + (fsnap/unlock! conn (:file-id snapshot) id))) diff --git a/backend/src/app/rpc/commands/files_update.clj b/backend/src/app/rpc/commands/files_update.clj index 5c47dbf61d..f1d6cde7ef 100644 --- a/backend/src/app/rpc/commands/files_update.clj +++ b/backend/src/app/rpc/commands/files_update.clj @@ -21,6 +21,7 @@ [app.config :as cf] [app.db :as db] [app.features.fdata :as fdata] + [app.features.file-changes :as fchanges] [app.features.file-snapshots :as fsnap] [app.features.logical-deletion :as ldel] [app.http.errors :as errors] @@ -335,6 +336,7 @@ (when repair (db/insert! conn :file-change {:id (uuid/next) + :kind fchanges/revision-kind :session-id repair-session-id :profile-id profile-id :created-at timestamp @@ -355,8 +357,13 @@ ;; Insert change (xlog) with deleted_at in a future data for ;; make them automatically eleggible for GC once they expires + ;; + ;; The revision carries only the delta of the save: the features of the + ;; file are not repeated here (nobody reads them from a revision, and + ;; they are duplicated on every save). (db/insert! conn :file-change {:id (uuid/next) + :kind fchanges/revision-kind :session-id session-id :profile-id profile-id :created-at change-created-at @@ -365,7 +372,6 @@ :file-id (:id file) :revn (:revn file) :version (:version file) - :features (into-array (:features file)) :changes (blob/encode changes)} {::db/return-keys false}) diff --git a/backend/src/app/rpc/quotes.clj b/backend/src/app/rpc/quotes.clj index 96309afd49..903dc31070 100644 --- a/backend/src/app/rpc/quotes.clj +++ b/backend/src/app/rpc/quotes.clj @@ -422,6 +422,8 @@ (sm/lazy-validator schema:snapshots-per-file)) (def ^:private sql:get-snapshots-per-file + ;; A version is told apart by its label, and the partial index with the same + ;; predicate answers the count without touching the table. "SELECT count(*) AS total FROM file_change AS fc WHERE fc.file_id = ? @@ -453,11 +455,16 @@ (sm/lazy-validator schema:snapshots-per-team)) (def ^:private sql:get-snapshots-per-team + ;; The files of the team are resolved first, so the count filters by + ;; `file_id`: the planner can prune the partitions of `file_change` when it + ;; chooses an index scan (with a sequential scan it reads all of them). A + ;; version is told apart by its label. "SELECT count(*) AS total FROM file_change AS fc - JOIN file AS f ON (f.id = fc.file_id) - JOIN project AS p ON (p.id = f.project_id) - WHERE p.team_id = ? + WHERE fc.file_id = ANY (SELECT f.id + FROM file AS f + JOIN project AS p ON (p.id = f.project_id) + WHERE p.team_id = ?) AND fc.created_by = 'user' AND fc.deleted_at IS NULL AND fc.label IS NOT NULL") diff --git a/backend/src/app/scripts/decode_file_data.clj b/backend/src/app/scripts/decode_file_data.clj index b7d775eafc..a732a34fc1 100644 --- a/backend/src/app/scripts/decode_file_data.clj +++ b/backend/src/app/scripts/decode_file_data.clj @@ -39,6 +39,7 @@ `[.data[\"pages-index\"][] | {id, name}]`, etc." (:require [app.binfile.common :as bfc] + [app.common.exceptions :as ex] [app.common.json :as json] [app.db :as db] [app.features.file-snapshots :as fsnap] @@ -183,16 +184,23 @@ (defn get-snapshot-json "Return a snapshot as JSON (realized). For nREPL. - Takes the same `format` and `output` as `get-file-json`." + Takes the same `format` and `output` as `get-file-json`. It raises when the + file has no snapshot with that id." [system file-id snapshot-id & {:keys [format output] :or {format :full}}] (let [snapshot (db/run! system (fn [cfg] (fsnap/get-snapshot cfg file-id snapshot-id))) - data (case format - :full snapshot - :data (:data snapshot) - :summary (summarize-data (:data snapshot)) - :pages (:pages (summarize-data (:data snapshot))))] + data (if snapshot + (case format + :full snapshot + :data (:data snapshot) + :summary (summarize-data (:data snapshot)) + :pages (:pages (summarize-data (:data snapshot)))) + (ex/raise :type :not-found + :code :snapshot-not-found + :hint "unable to find snapshot with the provided id" + :snapshot-id snapshot-id + :file-id file-id))] (write-output (data->json data) output))) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; diff --git a/backend/src/app/tasks/delete_object.clj b/backend/src/app/tasks/delete_object.clj index 2545fe97e4..0398264f90 100644 --- a/backend/src/app/tasks/delete_object.clj +++ b/backend/src/app/tasks/delete_object.clj @@ -51,7 +51,8 @@ :deleted-at (ct/format-inst deleted-at)) (db/update! conn :file-change - {:deleted-at deleted-at} + {:deleted-at deleted-at + :updated-at (ct/now)} {:id id :file-id file-id} {::db/return-keys false}) @@ -91,7 +92,8 @@ ;; lifetime; the collector would still take it, just much later than it ;; should. (db/update! conn :file-change - {:deleted-at deleted-at} + {:deleted-at deleted-at + :updated-at (ct/now)} {:file-id id :deleted-at nil} {::db/return-keys false}) diff --git a/backend/src/app/tasks/objects_gc.clj b/backend/src/app/tasks/objects_gc.clj index 1d6f842e30..b1ee555772 100644 --- a/backend/src/app/tasks/objects_gc.clj +++ b/backend/src/app/tasks/objects_gc.clj @@ -637,41 +637,49 @@ {:fetch-size 5})))))) (def ^:private sql:get-file-change - "SELECT id FROM file_change + "SELECT id, file_id FROM file_change WHERE deleted_at IS NOT NULL AND deleted_at <= ? ORDER BY deleted_at ASC") (def ^:private sql:lock-file-change "SELECT id, file_id, deleted_at FROM file_change - WHERE id = ANY(?::uuid[]) + WHERE file_id = ? + AND id = ? AND deleted_at IS NOT NULL AND deleted_at <= ? FOR UPDATE SKIP LOCKED") (defn- delete-file-changes-chunk - [cfg ids] + [cfg rows] (try (db/tx-run! cfg (fn [{:keys [::db/conn ::timestamp]}] (db/exec-one! conn ["SET LOCAL rules.deletion_protection TO off"]) - (let [ids (db/create-array conn "uuid" ids)] - (->> (db/exec! conn [sql:lock-file-change ids timestamp]) - (reduce (fn [total {:keys [id file-id deleted-at]}] - (l/trc :obj "file-change" - :id (str id) - :file-id (str file-id) - :deleted-at (ct/format-inst deleted-at)) + (reduce (fn [total {:keys [id file-id]}] + ;; the row is locked by its (file-id, id) pair, so + ;; the lookup uses the primary key and prunes to + ;; the partition that holds the file. + (if-let [{:keys [deleted-at]} + (db/exec-one! conn [sql:lock-file-change + file-id id timestamp])] + (do + (l/trc :obj "file-change" + :id (str id) + :file-id (str file-id) + :deleted-at (ct/format-inst deleted-at)) - (let [affected (-> (db/delete! conn :file-change {:id id}) - (db/get-update-count))] - (+ total affected))) - 0))))) + (let [affected (-> (db/delete! conn :file-change + {:id id :file-id file-id}) + (db/get-update-count))] + (+ total affected))) + total)) + 0 rows))) (catch Throwable cause (l/err :obj "file-change" :hint "error on chunk deletion, skipping it" - :chunk ids + :chunk rows :cause cause) 0))) @@ -681,7 +689,10 @@ ([total chunk] (+ total (delete-file-changes-chunk cfg chunk))))] (db/tx-run! (assoc cfg ::db/read-only true) (fn [{:keys [::db/conn]}] - (transduce (comp (map :id) (partition-all chunk-size)) rf 0 + (transduce (comp (map (fn [row] {:id (:id row) + :file-id (:file-id row)})) + (partition-all chunk-size)) + rf 0 (db/plan conn [sql:get-file-change timestamp] {:fetch-size 5})))))) diff --git a/backend/test/backend_tests/db_test.clj b/backend/test/backend_tests/db_test.clj index 29d3095f32..7345e0b5d5 100644 --- a/backend/test/backend_tests/db_test.clj +++ b/backend/test/backend_tests/db_test.clj @@ -62,3 +62,11 @@ (t/deftest uuid->hash-code-stable-for-zero-uuid (t/is (= 3659997967308761462 (db/uuid->hash-code uuid/zero)))) + +(t/deftest pool-enables-the-partitionwise-join + ;; The pool sets the GUC on every connection, so the planner can resolve a + ;; join between two partitioned tables pairwise, partition by partition. + (let [row (db/exec-one! th/*pool* + ["SELECT current_setting('enable_partitionwise_join') AS partitionwise_join"])] + + (t/is (= "on" (:partitionwise-join row))))) diff --git a/backend/test/backend_tests/fdata_resolve_test.clj b/backend/test/backend_tests/fdata_resolve_test.clj new file mode 100644 index 0000000000..07b598867e --- /dev/null +++ b/backend/test/backend_tests/fdata_resolve_test.clj @@ -0,0 +1,42 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS INC Sucursal en España SL + +(ns backend-tests.fdata-resolve-test + "Resolution of file data by backend (`app.features.fdata/resolve-file-data`). + + A row without a `file_data` twin carries no backend: the read queries mark + it with `coalesce(fd.backend, 'legacy-db')` and hand its blob over as + `:legacy-data`. The `legacy-db` backend is still supported, so that case + has to resolve to the blob instead of nil." + (:require + [app.features.fdata :as fdata] + [clojure.test :as t])) + +(t/deftest legacy-db-resolves-the-legacy-blob + (let [blob (vec (.getBytes "legacy-blob" "UTF-8")) + file {:backend "legacy-db" + :legacy-data (byte-array blob)}] + (let [out (fdata/resolve-file-data nil file)] + (t/is (= blob (vec (:data out)))) + (t/is (not (contains? out :legacy-data)))))) + +(t/deftest missing-backend-defaults-to-legacy-db + ;; The queries leave `:backend` nil when the `LEFT JOIN` with `file_data` + ;; finds no twin; the dispatch default keeps that row on `legacy-db`. + (let [blob (vec (.getBytes "legacy-blob" "UTF-8")) + file {:legacy-data (byte-array blob)}] + (let [out (fdata/resolve-file-data nil file)] + (t/is (= blob (vec (:data out)))) + (t/is (not (contains? out :legacy-data)))))) + +(t/deftest db-backend-drops-the-legacy-key + (let [data (vec (.getBytes "db-blob" "UTF-8")) + file {:backend "db" + :data (byte-array data) + :legacy-data (byte-array (vec (.getBytes "stale" "UTF-8")))}] + (let [out (fdata/resolve-file-data nil file)] + (t/is (= data (vec (:data out)))) + (t/is (not (contains? out :legacy-data)))))) diff --git a/backend/test/backend_tests/file_change_retention_test.clj b/backend/test/backend_tests/file_change_retention_test.clj index 7cede5a164..6c8a378c80 100644 --- a/backend/test/backend_tests/file_change_retention_test.clj +++ b/backend/test/backend_tests/file_change_retention_test.clj @@ -10,7 +10,7 @@ A file has two kinds of row. Durable history: the snapshots, which a restore brings back. And ephemeral revisions: the xlog of an edit, born already scheduled for deletion one hour later, which nobody needs once the sessions - that were going to read them are gone. + that were going to read them are gone. The `kind` column tells them apart. The invariant under test is that a deletion date is assigned once and never moved: restoring a file must not unmark a revision, and deleting a file must @@ -64,6 +64,7 @@ :version 0 :created-by (or created-by "system") :label label + :kind (if label 1 2) :data data :changes changes} {::db/return-keys false}) @@ -170,6 +171,7 @@ out (th/command! {::th/type :update-file-snapshot ::rpc/profile-id (:id profile) + :file-id file-id :id revision-id :label "renamed"})] diff --git a/backend/test/backend_tests/file_change_schema_test.clj b/backend/test/backend_tests/file_change_schema_test.clj new file mode 100644 index 0000000000..c8cade2436 --- /dev/null +++ b/backend/test/backend_tests/file_change_schema_test.clj @@ -0,0 +1,121 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS INC Sucursal en España SL + +(ns backend-tests.file-change-schema-test + (:require + [app.db :as db] + [backend-tests.helpers :as th] + [clojure.test :as t])) + +(t/use-fixtures :once th/state-init) +(t/use-fixtures :each th/database-reset) + +(defn- query + "Runs a query that returns a single value and returns it." + [sql & params] + (-> (db/exec! th/*pool* (into [sql] params)) + first + (vals) + first)) + +(defn- partition-names + [] + (->> (db/exec! th/*pool* ["SELECT c.relname AS name + FROM pg_class c + JOIN pg_partition_tree('file_change') t ON t.relid = c.oid + WHERE t.isleaf + ORDER BY c.relname"]) + (mapv :name))) + +(t/deftest file-change-is-partitioned-in-sixteen-named-partitions + (t/is (= (mapv #(format "file_change_%02d" %) (range 16)) + (partition-names)))) + +(t/deftest file-change-primary-key-is-file-id-and-id + (t/is (= "PRIMARY KEY (file_id, id)" + (query "SELECT pg_get_constraintdef(oid) AS def + FROM pg_constraint + WHERE conrelid = 'file_change'::regclass AND contype = 'p'")))) + +(t/deftest file-change-has-three-foreign-keys-and-one-check + (t/is (= 3 (query "SELECT count(*) AS total + FROM pg_constraint + WHERE conrelid = 'file_change'::regclass AND contype = 'f'"))) + (t/is (= 1 (query "SELECT count(*) AS total + FROM pg_constraint + WHERE conrelid = 'file_change'::regclass AND contype = 'c'"))) + (t/is (re-find #"kind" + (query "SELECT pg_get_constraintdef(oid) AS def + FROM pg_constraint + WHERE conrelid = 'file_change'::regclass AND contype = 'c'")))) + +(t/deftest file-change-has-the-six-canonical-indexes + (t/is (= ["file_change__deleted_at__idx" + "file_change__file_id__revn__idx" + "file_change__locked_by__idx" + "file_change__profile_id__idx" + "file_change__user_snapshots__idx" + "file_change_pkey"] + (->> (db/exec! th/*pool* ["SELECT indexname AS name + FROM pg_indexes + WHERE tablename = 'file_change' + AND schemaname = current_schema() + ORDER BY indexname"]) + (mapv :name))))) + +(t/deftest file-change-keeps-the-data-column-and-a-kind-column + ;; The `data` column is kept: the `legacy-db` backend is still supported + ;; (migration 0157 moved the existing blobs, but the backend keeps reading + ;; and writing that column). The dead `data_ref_id`/`data_backend` columns + ;; stay gone: no code read them since the file storage refactor. + (t/is (= [] + (->> (db/exec! th/*pool* ["SELECT attname AS name + FROM pg_attribute + WHERE attrelid = 'file_change'::regclass + AND attnum > 0 AND NOT attisdropped + AND attname IN ('data_ref_id', 'data_backend')"]) + (mapv :name)))) + (t/is (= ["bytea"] + (->> (db/exec! th/*pool* ["SELECT pg_catalog.format_type(atttypid, atttypmod) AS type + FROM pg_attribute + WHERE attrelid = 'file_change'::regclass + AND attname = 'data'"]) + (mapv :type)))) + (t/is (true? (query "SELECT attnotnull AS notnull + FROM pg_attribute + WHERE attrelid = 'file_change'::regclass + AND attname = 'kind'")))) + +(t/deftest the-legacy-table-has-no-foreign-key-to-file + ;; The legacy table is kept as the way back, but its foreign key on + ;; `file_id` is `NO ACTION`: while it has rows, the objects GC cannot + ;; delete the files they point to. + (t/is (= 0 (query "SELECT count(*) AS total + FROM pg_constraint con + JOIN pg_class rel ON rel.oid = con.conrelid + WHERE rel.relname = 'file_change_legacy' + AND con.contype = 'f' + AND con.confrelid = 'file'::regclass")))) + +(t/deftest inserting-a-row-without-a-kind-fails + ;; The column has no default on purpose: an insert that forgets it has to + ;; fail instead of storing the wrong kind silently. + (let [profile (th/create-profile* 1) + file (th/create-file* 1 {:profile-id (:id profile) + :project-id (:default-project-id profile)})] + (t/is (thrown? java.sql.SQLException + (db/exec! th/*pool* ["INSERT INTO file_change (file_id, id) + VALUES (?, gen_random_uuid())" + (:id file)]))))) + +(t/deftest inserting-a-row-with-an-unknown-kind-fails + (let [profile (th/create-profile* 1) + file (th/create-file* 1 {:profile-id (:id profile) + :project-id (:default-project-id profile)})] + (t/is (thrown? java.sql.SQLException + (db/exec! th/*pool* ["INSERT INTO file_change (file_id, id, kind) + VALUES (?, gen_random_uuid(), 3)" + (:id file)]))))) diff --git a/backend/test/backend_tests/migrations_legacy_file_data_test.clj b/backend/test/backend_tests/migrations_legacy_file_data_test.clj index 70c4c72925..574c02da23 100644 --- a/backend/test/backend_tests/migrations_legacy_file_data_test.clj +++ b/backend/test/backend_tests/migrations_legacy_file_data_test.clj @@ -58,6 +58,7 @@ :revn 1 :data snap-data :label "v1" + :kind 1 :created-at created-at :updated-at updated-at :deleted-at deleted-at}) @@ -69,6 +70,7 @@ {:id rev-id :file-id fid :revn 2 + :kind 2 :data rev-data}) ;; Live revision (changes, no data, no label): untouched. @@ -77,6 +79,7 @@ {:id live-id :file-id fid :revn 3 + :kind 2 :changes (blob/encode [{:type :dummy}])})] (run-migration) diff --git a/backend/test/backend_tests/objects_gc_test.clj b/backend/test/backend_tests/objects_gc_test.clj index 5a3052901f..9f4f880683 100644 --- a/backend/test/backend_tests/objects_gc_test.clj +++ b/backend/test/backend_tests/objects_gc_test.clj @@ -10,6 +10,7 @@ [app.common.time :as ct] [app.config :as cf] [app.db :as db] + [app.features.file-changes :as fchanges] [app.rpc :as-alias rpc] [app.storage :as sto] [app.tasks.objects-gc :as ogc] @@ -73,6 +74,25 @@ (t/is (pos? (:processed result))) (t/is (nil? (find-file (:id file)))))) +(t/deftest deletes-the-pending-file-changes + ;; The pending file changes are locked and deleted by their (file-id, id) + ;; pair, so the rows of a file that is pending deletion are gone after the + ;; run. + (let [profile (th/create-profile* 1) + file (th/create-file* 1 {:profile-id (:id profile) + :project-id (:default-project-id profile)}) + + _ (th/db-insert! :file-change + {:file-id (:id file) + :kind fchanges/revision-kind + :deleted-at (ct/in-past "10m")}) + + result (th/run-task! :objects-gc {})] + + (t/is (pos? (:processed result))) + (t/is (nil? (th/db-get :file-change {:file-id (:id file)} + {::db/remove-deleted false}))))) + (t/deftest profile-cascade-keeps-profile-deletion-date ;; The cascade marks the owned rows with the deletion date of the profile, ;; like the RPC delete-profile flow does, not with the timestamp of the run. diff --git a/backend/test/backend_tests/restore_storage_objects_test.clj b/backend/test/backend_tests/restore_storage_objects_test.clj index 5986170b20..bb2201659d 100644 --- a/backend/test/backend_tests/restore_storage_objects_test.clj +++ b/backend/test/backend_tests/restore_storage_objects_test.clj @@ -9,6 +9,7 @@ [app.common.time :as ct] [app.common.uuid :as uuid] [app.db :as db] + [app.features.file-changes :as fchanges] [app.rpc.commands.files :as files] [app.srepl.main :as srepl] [app.storage :as sto] @@ -130,3 +131,29 @@ (srepl/restore-team! (:id team))) (t/is (nil? (:deleted-at (find-object sobject)))))) + +(t/deftest restoring-a-file-updates-its-file-changes + ;; Restoring a file brings its file changes back to life, and that mutation + ;; has to be observable through `updated_at`, the watermark used by the + ;; online partitioning copy of `file_change`. + (let [profile (th/create-profile* 1) + file (th/create-file* 1 {:profile-id (:id profile) + :project-id (:default-project-id profile)}) + + stale-at #inst "2020-01-01T00:00:00.000Z" + + _ (th/db-insert! :file-change + {:file-id (:id file) + :label "v1" + :kind fchanges/snapshot-kind + :deleted-at (ct/in-past {:days 1}) + :updated-at stale-at})] + + (restore-file! (:id file)) + + (let [change (th/db-get :file-change {:file-id (:id file)} + {::db/remove-deleted false})] + (t/is (nil? (:deleted-at change))) + (t/is (pos? (- (inst-ms (:updated-at change)) + (inst-ms stale-at))) + "updated_at must be bumped by the restore")))) diff --git a/backend/test/backend_tests/rpc_file_snapshot_test.clj b/backend/test/backend_tests/rpc_file_snapshot_test.clj index aed2b5ce39..39b01bb06e 100644 --- a/backend/test/backend_tests/rpc_file_snapshot_test.clj +++ b/backend/test/backend_tests/rpc_file_snapshot_test.clj @@ -79,6 +79,19 @@ (t/is (= 0 (:revn row))) (t/is (= (:id profile) (:profile-id row)))))) + (t/testing "update snapshot with file-id" + (let [params {::th/type :update-file-snapshot + ::rpc/profile-id (:id profile) + :file-id (:id file) + :id @snapshot-id + :label "label1-updated"} + out (th/command! params)] + ;; (th/print-result! out) + (t/is (nil? (:error out))) + (t/is (true? (:result out))) + (t/is (= "label1-updated" + (:label (th/db-get :file-change {:id @snapshot-id})))))) + (t/testing "restore snapshot" (let [params {::th/type :restore-file-snapshot ::rpc/profile-id (:id profile) @@ -220,3 +233,225 @@ ;; (th/print-result! out) (t/is (nil? (:error out))) (t/is (true? (:result out))))))) + +(t/deftest snapshots-require-file-id + ;; Since version 2.19 the snapshot commands require the `file-id` + ;; parameter, so partition pruning can always be applied on the + ;; `file_change` table. + (let [profile (th/create-profile* 1 {:is-active true}) + team-id (:default-team-id profile) + proj-id (:default-project-id profile) + + file (th/create-file* 1 {:profile-id (:id profile) + :project-id proj-id + :is-shared false}) + + snapshot + (let [params {::th/type :create-file-snapshot + ::rpc/profile-id (:id profile) + :file-id (:id file) + :label "label1"} + out (th/command! params)] + ;; (th/print-result! out) + + (t/is (nil? (:error out))) + (:result out))] + + (doseq [type [:update-file-snapshot :lock-file-snapshot + :unlock-file-snapshot :delete-file-snapshot]] + (let [params (merge {::th/type type + ::rpc/profile-id (:id profile) + :id (:id snapshot)} + (when (= type :update-file-snapshot) + {:label "label2"})) + out (th/command! params)] + + (t/testing (str (name type) " without file-id") + (t/is (th/ex-of-code? (:error out) :params-validation))))))) + +(t/deftest snapshots-reject-a-file-id-that-does-not-match + ;; The snapshot commands filter by `file_id` so that, once `file_change` is + ;; hash-partitioned by it, the lookup prunes to a single partition. Sending + ;; a `file-id` of another file must not find nor touch the snapshot. + (let [profile (th/create-profile* 1 {:is-active true}) + file-a (th/create-file* 1 {:profile-id (:id profile) + :project-id (:default-project-id profile) + :is-shared false}) + file-b (th/create-file* 2 {:profile-id (:id profile) + :project-id (:default-project-id profile) + :is-shared false}) + + snapshot (-> (th/command! {::th/type :create-file-snapshot + ::rpc/profile-id (:id profile) + :file-id (:id file-a) + :label "label1"}) + :result)] + + (doseq [type [:update-file-snapshot :lock-file-snapshot + :unlock-file-snapshot :delete-file-snapshot]] + (let [params (merge {::th/type type + ::rpc/profile-id (:id profile) + :file-id (:id file-b) + :id (:id snapshot)} + (when (= type :update-file-snapshot) + {:label "label2"})) + out (th/command! params)] + + (t/testing (name type) + (t/is (th/ex-info? (:error out))) + (t/is (= :not-found (:type (ex-data (:error out))))) + (t/is (= :snapshot-not-found (:code (ex-data (:error out))))) + (t/is (= "label1" + (:label (th/db-get :file-change {:id (:id snapshot)} + {::db/remove-deleted false}))) + "the snapshot of file-a stays intact")))))) + +(t/deftest file-change-mutations-maintain-updated-at + ;; `updated_at` is the per-row watermark used by the online partitioning + ;; copy of `file_change` (hash partitioning by file_id): every mutation + ;; must bump it, otherwise the copy misses the row. Lock, unlock and the + ;; soft-delete used to leave it stale. + (let [profile (th/create-profile* 1 {:is-active true}) + file (th/create-file* 1 {:profile-id (:id profile) + :project-id (:default-project-id profile) + :is-shared false}) + + snapshot + (let [params {::th/type :create-file-snapshot + ::rpc/profile-id (:id profile) + :file-id (:id file) + :label "label1"} + out (th/command! params)] + ;; (th/print-result! out) + + (t/is (nil? (:error out))) + (:result out)) + + stale-at #inst "2020-01-01T00:00:00.000Z"] + + (doseq [type [:lock-file-snapshot :unlock-file-snapshot :delete-file-snapshot]] + (t/testing (name type) + ;; make the row look old, so the mutation has to bump it + (th/db-update! :file-change {:updated-at stale-at} + {:id (:id snapshot)} + {::db/remove-deleted false}) + + (let [params {::th/type type + ::rpc/profile-id (:id profile) + :file-id (:id file) + :id (:id snapshot)} + out (th/command! params)] + + (t/is (nil? (:error out))) + (t/is (true? (:result out))) + (t/is (pos? (- (inst-ms (:updated-at (th/db-get :file-change + {:id (:id snapshot)} + {::db/remove-deleted false}))) + (inst-ms stale-at))) + "updated_at must be bumped by the mutation")))) + + (t/testing "the deleted snapshot is marked as deleted" + (t/is (some? (:deleted-at (th/db-get :file-change {:id (:id snapshot)} + {::db/remove-deleted false}))))) + + ;; process the delete-object task enqueued by the snapshot delete + (th/run-pending-tasks!))) + +(t/deftest unknown-snapshot-is-reported-as-not-found + ;; The error points at the id that was not found: the snapshots are looked + ;; up by id, not by label. Both commands that resolve a snapshot report it + ;; the same way. + (let [profile (th/create-profile* 1 {:is-active true}) + file (th/create-file* 1 {:profile-id (:id profile) + :project-id (:default-project-id profile) + :is-shared false}) + + params {::rpc/profile-id (:id profile) + :file-id (:id file) + :id (uuid/random)}] + + (doseq [type [:restore-file-snapshot :get-file-snapshot]] + (t/testing (name type) + (let [out (th/command! (assoc params ::th/type type))] + + (t/is (th/ex-of-code? (:error out) :snapshot-not-found)) + (t/is (= "unable to find snapshot with the provided id" + (:hint (ex-data (:error out)))))))))) + +(t/deftest revision-and-snapshot-carry-different-content + ;; A revision is the delta of a save: it carries the changes and nothing + ;; else. A snapshot is the state of the file at a point: it carries the + ;; features and the migrations. + (let [profile (th/create-profile* 1 {:is-active true}) + file (th/create-file* 1 {:profile-id (:id profile) + :project-id (:default-project-id profile) + :is-shared false}) + page-id (uuid/random)] + + (update-file! + :file-id (:id file) + :profile-id (:id profile) + :revn 0 + :vern 0 + :changes + [{:type :add-page + :name "test" + :id page-id}]) + + (t/testing "the revision carries the changes and not the features" + (let [rows (th/db-query :file-change {:file-id (:id file)}) + revision (first (filter #(nil? (:label %)) rows))] + + (t/is (some? revision)) + (t/is (some? (:changes revision))) + (t/is (nil? (:features revision))))) + + (t/testing "the snapshot carries the migrations and the features" + (let [out (th/command! {::th/type :create-file-snapshot + ::rpc/profile-id (:id profile) + :file-id (:id file) + :label "label1"})] + + (t/is (nil? (:error out))) + + (let [snapshot (th/db-get :file-change {:id (get-in out [:result :id])})] + (t/is (= "label1" (:label snapshot))) + (t/is (some? (:migrations snapshot))) + (t/is (some? (:features snapshot)))))))) + +(t/deftest file-change-rows-have-a-kind + ;; The kind tells the two kinds of rows of `file_change` apart: a snapshot + ;; (1) and a revision (2). The values are the contract with the database. + (let [profile (th/create-profile* 1 {:is-active true}) + file (th/create-file* 1 {:profile-id (:id profile) + :project-id (:default-project-id profile) + :is-shared false}) + page-id (uuid/random)] + + (update-file! + :file-id (:id file) + :profile-id (:id profile) + :revn 0 + :vern 0 + :changes + [{:type :add-page + :name "test" + :id page-id}]) + + (t/testing "the revision is a revision" + (let [rows (th/db-query :file-change {:file-id (:id file)}) + revision (first (filter #(nil? (:label %)) rows))] + + (t/is (some? revision)) + (t/is (= 2 (:kind revision))))) + + (t/testing "the snapshot is a snapshot" + (let [out (th/command! {::th/type :create-file-snapshot + ::rpc/profile-id (:id profile) + :file-id (:id file) + :label "label1"})] + + (t/is (nil? (:error out))) + + (let [snapshot (th/db-get :file-change {:id (get-in out [:result :id])})] + (t/is (= 1 (:kind snapshot)))))))) diff --git a/backend/test/backend_tests/rpc_quotes_test.clj b/backend/test/backend_tests/rpc_quotes_test.clj index 03e0e05389..cb4bda799f 100644 --- a/backend/test/backend_tests/rpc_quotes_test.clj +++ b/backend/test/backend_tests/rpc_quotes_test.clj @@ -560,6 +560,7 @@ ;; deleting a version takes it out of the count right away (let [out (th/command! {::th/type :delete-file-snapshot ::rpc/profile-id (:id profile) + :file-id (:id file) :id (:id snapshot)})] (t/is (nil? (:error out))) (t/is (true? (:result out))) diff --git a/backend/test/backend_tests/scripts_decode_file_data_test.clj b/backend/test/backend_tests/scripts_decode_file_data_test.clj index e14618542f..3a6484b4ed 100644 --- a/backend/test/backend_tests/scripts_decode_file_data_test.clj +++ b/backend/test/backend_tests/scripts_decode_file_data_test.clj @@ -10,11 +10,14 @@ [app.common.uuid :as uuid] [app.scripts.decode-file-data :as dfd] [app.util.blob :as blob] + [backend-tests.helpers :as th] [clojure.java.io :as io] [clojure.test :as t]) (:import java.util.Base64)) +(t/use-fixtures :once th/state-init) + (defn- sample-file-data [] (let [p1 (uuid/next) @@ -80,3 +83,11 @@ parsed (json/decode (slurp tmp-out))] (t/is (== 2 (get-in parsed ["stats" "pages"]))) (t/is (= ["Accueil1" "Contact"] (mapv #(get % "name") (get parsed "pages")))))) + +(t/deftest get-snapshot-json-fails-when-the-snapshot-does-not-exist + ;; The snapshot lookup returns nil when there is no match, and a nil + ;; snapshot used to be printed as an empty JSON instead of failing. + (let [file-id (uuid/next) + snapshot-id (uuid/next)] + (t/is (thrown? clojure.lang.ExceptionInfo + (dfd/get-snapshot-json th/*system* file-id snapshot-id))))) diff --git a/frontend/src/app/main/data/workspace/versions.cljs b/frontend/src/app/main/data/workspace/versions.cljs index 5e376d6a98..28045405a3 100644 --- a/frontend/src/app/main/data/workspace/versions.cljs +++ b/frontend/src/app/main/data/workspace/versions.cljs @@ -92,7 +92,7 @@ (rx/of (update-versions-state {:editing nil}) (ev/event {::ev/name "rename-version" :file-id file-id})) - (->> (rp/cmd! :update-file-snapshot {:id id :label label}) + (->> (rp/cmd! :update-file-snapshot {:id id :file-id file-id :label label}) (rx/map fetch-versions))))))) (defn- clear-preview-state @@ -135,9 +135,10 @@ (ptk/reify ::delete-version ptk/WatchEvent - (watch [_ _ _] - (->> (rp/cmd! :delete-file-snapshot {:id id}) - (rx/map fetch-versions))))) + (watch [_ state _] + (let [file-id (:current-file-id state)] + (->> (rp/cmd! :delete-file-snapshot {:id id :file-id file-id}) + (rx/map fetch-versions)))))) (defn pin-version [id] @@ -147,8 +148,10 @@ (watch [_ state _] (when-let [version (->> (dm/get-in state [:workspace-versions :data]) (d/seek #(= (:id %) id)))] - (let [params {:id id - :label (ct/format-inst (:created-at version) :localized-date)}] + (let [file-id (:current-file-id state) + params {:id id + :file-id file-id + :label (ct/format-inst (:created-at version) :localized-date)}] (->> (rp/cmd! :update-file-snapshot params) (rx/mapcat (fn [_] @@ -161,18 +164,20 @@ (assert (uuid? id) "expected valid uuid for `id`") (ptk/reify ::lock-version ptk/WatchEvent - (watch [_ _ _] - (->> (rp/cmd! :lock-file-snapshot {:id id}) - (rx/map fetch-versions))))) + (watch [_ state _] + (let [file-id (:current-file-id state)] + (->> (rp/cmd! :lock-file-snapshot {:id id :file-id file-id}) + (rx/map fetch-versions)))))) (defn unlock-version [id] (assert (uuid? id) "expected valid uuid for `id`") (ptk/reify ::unlock-version ptk/WatchEvent - (watch [_ _ _] - (->> (rp/cmd! :unlock-file-snapshot {:id id}) - (rx/map fetch-versions))))) + (watch [_ state _] + (let [file-id (:current-file-id state)] + (->> (rp/cmd! :unlock-file-snapshot {:id id :file-id file-id}) + (rx/map fetch-versions)))))) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; RESTORE VERSION EVENTS diff --git a/frontend/src/app/plugins/file.cljs b/frontend/src/app/plugins/file.cljs index 59d222512b..1124cefd09 100644 --- a/frontend/src/app/plugins/file.cljs +++ b/frontend/src/app/plugins/file.cljs @@ -54,7 +54,7 @@ :else (do (swap! data assoc :label value :created-by "user") - (->> (rp/cmd! :update-file-snapshot {:id (:id @data) :label value}) + (->> (rp/cmd! :update-file-snapshot {:id (:id @data) :file-id file-id :label value}) (rx/take 1) (rx/subs! #(st/emit! (se/event plugin-id "rename-version" :file-id file-id)))))))} @@ -95,7 +95,7 @@ :else (let [version-id (:id @data)] - (->> (rp/cmd! :delete-file-snapshot {:id version-id}) + (->> (rp/cmd! :delete-file-snapshot {:id version-id :file-id file-id}) (rx/map (constantly nil)) (rx/subs! resolve reject))))))) @@ -111,8 +111,9 @@ (u/reject-not-valid reject :pin "Only auto-saved versions can be pinned") :else - (let [params {:id (:id @data) - :label (ct/format-inst (:created-at @data) :localized-date)}] + (let [params {:id (:id @data) + :file-id file-id + :label (ct/format-inst (:created-at @data) :localized-date)}] (->> (rx/zip (rp/cmd! :get-team-users {:file-id file-id}) (rp/cmd! :update-file-snapshot params)) (rx/subs! diff --git a/frontend/test/frontend_tests/data/workspace_versions_test.cljs b/frontend/test/frontend_tests/data/workspace_versions_test.cljs index 5ec7fa87bb..ecf574b917 100644 --- a/frontend/test/frontend_tests/data/workspace_versions_test.cljs +++ b/frontend/test/frontend_tests/data/workspace_versions_test.cljs @@ -6,6 +6,7 @@ (ns frontend-tests.data.workspace-versions-test (:require + [app.common.time :as ct] [app.common.uuid :as uuid] [app.main.data.changes :as dch] [app.main.data.persistence :as dps] @@ -187,3 +188,32 @@ (t/is (nil? (get-in result [:workspace-versions :backup])))) (finally (rx/dispose! sub))))) + +(t/deftest version-events-send-the-current-file-id + ;; The snapshot RPC commands require the `file-id` (they need it for + ;; partition pruning on the `file_change` table); the workspace version + ;; events must always send it, taken from the current file. + (let [file-id (uuid/next) + version-id (uuid/next) + state {:current-file-id file-id + :workspace-versions {:data [{:id version-id + :created-at (ct/now) + :created-by "system"}]}} + call! (fn [event] + (let [requests (atom [])] + (with-redefs [rp/cmd! (mock/stub (fn [cmd params] + (swap! requests conj [cmd params]) + (rx/of true)))] + (->> (ptk/watch event state (rx/empty)) + (rx/subs! (fn [_]))) + (first @requests))))] + + (doseq [[cmd event] [[:update-file-snapshot (versions/rename-version version-id "name")] + [:delete-file-snapshot (versions/delete-version version-id)] + [:lock-file-snapshot (versions/lock-version version-id)] + [:unlock-file-snapshot (versions/unlock-version version-id)] + [:update-file-snapshot (versions/pin-version version-id)]]] + (t/testing (name cmd) + (let [[sent-cmd params] (call! event)] + (t/is (= cmd sent-cmd)) + (t/is (= file-id (:file-id params)))))))) diff --git a/operations/2.19.0-file-change-partition.md b/operations/2.19.0-file-change-partition.md new file mode 100644 index 0000000000..c04c3121c3 --- /dev/null +++ b/operations/2.19.0-file-change-partition.md @@ -0,0 +1,786 @@ +# Partition `file_change` by `file_id` + +Online procedure to turn `file_change` into a table partitioned by +`HASH (file_id)` with 16 partitions, aligned with the ones of `file_data`. + +- Migration: `backend/src/app/migrations/sql/0158-partition-file-change.sql` +- Issue: + +The result has to be **exactly** the schema of the migration, because on a big +instance the migration is marked as applied and never runs. This procedure +runs the same statements by hand. + +## Sizing your instance + +Every number in this procedure comes from your database: run the queries +below, the procedure does not depend on any absolute size. It is plain SQL +and runs on any supported PostgreSQL (>= 15). + +```sql +-- total size of the database and of file_change (heap + TOAST + indexes) +SELECT pg_size_pretty(pg_database_size(current_database())) AS database; +SELECT pg_size_pretty(pg_total_relation_size('file_change')) AS file_change_total, + pg_size_pretty(pg_relation_size('file_change')) AS heap; +SELECT count(*) AS rows FROM file_change; + +-- live data: rows times the average row width from the planner stats. +-- The copy reads the heap once plus this live TOAST and writes about +-- this much; everything above it is dead space the copy leaves behind. +SELECT pg_size_pretty(count(*) * width) AS live_data + FROM file_change, + (SELECT sum(avg_width)::bigint AS width FROM pg_stats + WHERE tablename = 'file_change') s; + +-- dead tuples waiting for vacuum (a rough upper bound of the dead space) +SELECT n_dead_tup FROM pg_stat_user_tables WHERE relname = 'file_change'; +``` + +Peak disk during the bulk is about the old table (it does not grow) plus +the new table (about the live data) plus the WAL the bulk generates +(proportional to what it writes: watch the free space of the `pg_wal` +mount while the first batches run). The disk comes back when the legacy +table is dropped, days later. Time the first batches and extrapolate: +the bulk reads the heap once sequentially, so its duration is +proportional to the heap. + +## What changes + +- `file_change` is partitioned by `HASH (file_id)` with 16 partitions, the + same bounds as `file_data`, so the join between them is resolved partition by + partition. +- The primary key goes from `(id)` to `(file_id, id)`: PostgreSQL requires the + partition key in the primary key, and the application already filters by + `file_id` everywhere. +- The table has 17 columns: `data_ref_id` and `data_backend` are gone (dead + since the file storage refactor), `data` is kept (the `legacy-db` backend + is still supported) and `kind` appears (1 = snapshot, 2 = revision). +- The old table is kept as `file_change_legacy`, the way back if something + goes wrong, and dropped days later. +- Almost all of the size of the table is dead space (see sizing), so the new + table holds about the live data. The disk comes back when the legacy table + is dropped: the database shrinks by the size of the old table. + +## Why this mechanism + +`ATTACH` as a `DEFAULT` partition is not possible: the 16 hash siblings cover +the whole hash space, so the constraint of a `DEFAULT` partition could never +hold, and the `ATTACH` fails when it validates every row. A `UNION` +view with triggers, dual writes in the application and logical replication +were discarded too. + +What is left is a copy in phases, driven by the `updated_at` watermark, and a +cutover of minutes: plain SQL, resumable, no new infrastructure. The heap is +small next to the TOAST, so a delta pass without an index scans the heap and +only detoastes the rows that changed. + +--- + +## Phase 0 — Before anything + +### 0.1 Mark the migration as applied + +The runner applies at startup every migration that is not in the `migrations` +table. Without this, it would copy the whole table inside the transaction of the +startup. + +```sql +INSERT INTO migrations (module, step) +VALUES ('main', '0158-partition-file-change'); +``` + +The module is `main` and the step has no `.sql` (see +`backend/src/app/util/migrations.clj`). + +### 0.2 Add the `kind` column to the current table + +Until the cutover, `file_change` is the current table, and the new code writes +`kind` on every insert: without the column the backend does not start. + +```sql +ALTER TABLE file_change ADD COLUMN IF NOT EXISTS kind smallint; +``` + +Nullable and without a default on purpose: it is a metadata-only operation +(it does not rewrite a single row of the table), the old code keeps inserting +rows, and the values of the rows that are already there are computed by the +copy from their `label`. + +### 0.3 Deploy the code + +The `updated_at` watermark has to be alive **before** the bulk copy: the new +code updates it on every mutation (locks, soft deletes), the old one does +not. Rows changed between the bulk and a delta pass would be lost. + +```sql +SELECT max(updated_at) FROM file_change; +``` + +### 0.4 Index for the delta passes + +The delta passes filter by `updated_at`. Without an index on that column every +pass scans the whole heap; with it, it is an index range that only +touches the rows that changed. + +```sql +-- CONCURRENTLY: it does not block the writes. It cannot run inside a +-- transaction: run it alone. +CREATE INDEX CONCURRENTLY IF NOT EXISTS file_change__updated_at__idx + ON file_change (updated_at); +``` + +- The build reads the heap once sequentially and writes one narrow entry per + row (check its size with `pg_relation_size` after the build): it does not + block the writes either way. Time the build and extrapolate the bulk from + it (an index on a column without TOAST does not touch the + TOAST). +- If it is interrupted, the index is left `INVALID` and has to be dropped and + built again: + +```sql +SELECT i.indisvalid + FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid + WHERE c.relname = 'file_change__updated_at__idx'; +``` + +- The index goes with the old table: at the cutover it is renamed to + `file_change__updated_at__idx_legacy` and disappears with it. The new table + does not carry it (nothing queries by `updated_at`; the GC uses + `deleted_at`). + +### 0.5 Check the environment + +```sql +-- Space: peak about the old table (it does not grow) + the new table +-- (about the live data, see sizing) + the WAL the bulk generates +SELECT pg_size_pretty(pg_database_size(current_database())); + +-- The WAL must not run out of room (note the free space of the mount point): +-- df -h + +-- Nothing must reference file_change (0 rows expected): +SELECT conrelid::regclass FROM pg_constraint + WHERE confrelid = 'file_change'::regclass; + +-- There must be no triggers on file_change: +SELECT tgname FROM pg_trigger + WHERE tgrelid = 'file_change'::regclass AND NOT tgisinternal; + +-- Replication slots: the bulk generates WAL proportional to what it writes, +-- watch the retained WAL while the first batches run. +SELECT slot_name, active, pg_size_pretty(pg_wal_lsn_diff( + pg_current_wal_lsn(), restart_lsn)) AS retained_wal + FROM pg_replication_slots; + +-- The legacy blobs of the snapshots were already moved to `file_data` by +-- migration 0157-migrate-legacy-file-data (on the big instances, by hand, +-- with its runbook 2.19.0-migrate-legacy-file-data.md), so there is nothing +-- to copy here: the `data` column travels with the row to the new table. +-- These two numbers are informational guards. The first one has to be 0 +-- after 0157 ran; if it is not, run 0157 (or its runbook) before starting. +SELECT count(*) AS snapshots_without_data + FROM file_change AS fc + WHERE fc.data IS NOT NULL + AND fc.label IS NOT NULL + AND NOT EXISTS (SELECT 1 FROM file_data AS fd + WHERE fd.file_id = fc.file_id + AND fd.id = fc.id + AND fd.type = 'snapshot'); + +-- The revisions with legacy data are not migrated (they are the ephemeral +-- changes, out of scope). This number is informative: their data travels +-- with the row to the new table. +SELECT count(*) AS revisions_with_data + FROM file_change + WHERE data IS NOT NULL + AND label IS NULL; +``` + +- Check that the backup / PITR is working before starting. +- Pick a low activity window for the cutover: a freeze of the writes to + `file_change` of a couple of minutes (the final delta inside the lock + copies only the rows changed since the bulk, so it is quick; the saves of + files during the freeze fail and the user retries, and the rest of the + database keeps working). The window has to cover the whole cutover, with + the backend and the workers stopped and started again (see 4.2). + +--- + +## Phase 1 — Create the partitioned table + +The new indexes and constraints carry a `_new` suffix because they live with +the ones of the old table until the cutover (the names live in the schema, +not in the table). At the cutover they are swapped for the canonical ones. +The partitions are the exception: they are created with their final name, +because their indexes are named after the partition and not after the +table. + +```sql +CREATE TABLE file_change_new ( + file_id uuid NOT NULL, + id uuid NOT NULL DEFAULT gen_random_uuid(), + created_at timestamptz NOT NULL DEFAULT now(), + session_id uuid, + revn bigint NOT NULL DEFAULT 0, + data bytea, + changes bytea, + profile_id uuid, + features text[], + label text, + version integer, + created_by text NOT NULL DEFAULT 'system', + updated_at timestamptz DEFAULT now(), + deleted_at timestamptz, + migrations text[], + locked_by uuid, + kind smallint NOT NULL, + CONSTRAINT file_change_new_kind_check CHECK (kind IN (1, 2)) +) PARTITION BY HASH (file_id); + +-- the columns keep the storage of PostgreSQL by default (migration +-- 0156-reset-pg-storage-defaults dropped the explicit EXTERNAL overrides) + +COMMENT ON COLUMN file_change_new.kind IS + 'The kind of the row: 1 is a snapshot (it has a label and its data lives in file_data) and 2 is a revision (it has no label and it holds the changes).'; +COMMENT ON COLUMN file_change_new.locked_by IS + 'Profile ID of user who has locked this version. Only the creator can lock/unlock their own versions. Locked versions cannot be deleted by others.'; + +-- 16 partitions, the same bounds as file_data. The name of the table is +-- temporary (the old one holds it until the cutover) but the name of the +-- partitions is not: it is the final one, so their indexes are born with +-- the canonical names (renaming a partition does not rename its indexes) +-- and the result is exactly the schema of the migration. +CREATE TABLE file_change_00 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 0); +CREATE TABLE file_change_01 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 1); +CREATE TABLE file_change_02 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 2); +CREATE TABLE file_change_03 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 3); +CREATE TABLE file_change_04 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 4); +CREATE TABLE file_change_05 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 5); +CREATE TABLE file_change_06 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 6); +CREATE TABLE file_change_07 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 7); +CREATE TABLE file_change_08 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 8); +CREATE TABLE file_change_09 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 9); +CREATE TABLE file_change_10 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 10); +CREATE TABLE file_change_11 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 11); +CREATE TABLE file_change_12 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 12); +CREATE TABLE file_change_13 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 13); +CREATE TABLE file_change_14 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 14); +CREATE TABLE file_change_15 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 15); + +-- The primary key includes the partition key (required on a partitioned table) +ALTER TABLE file_change_new ADD PRIMARY KEY (file_id, id); + +-- Foreign keys (no CREATE TABLE copies them) +ALTER TABLE file_change_new ADD FOREIGN KEY (file_id) + REFERENCES file(id) DEFERRABLE; +ALTER TABLE file_change_new ADD FOREIGN KEY (locked_by) + REFERENCES profile(id) ON DELETE SET NULL DEFERRABLE; +ALTER TABLE file_change_new ADD FOREIGN KEY (profile_id) + REFERENCES profile(id) ON DELETE SET NULL DEFERRABLE; + +-- Secondary indexes: the canonical names, with the _new suffix. There is no +-- index for the system snapshots: the old one filtered them by +-- `data IS NOT NULL`, and with the payload living in `file_data` the column +-- no longer tells snapshots apart. The new one could only filter +-- them by `deleted_at IS NULL`, and the system snapshots always carry a +-- deletion date in the future, so it would never match a row. +CREATE INDEX file_change_new__deleted_at__idx + ON file_change_new (deleted_at, file_id, id) WHERE deleted_at IS NOT NULL; +CREATE INDEX file_change_new__file_id__revn__idx + ON file_change_new (file_id, revn); +CREATE INDEX file_change_new__locked_by__idx + ON file_change_new (locked_by) WHERE locked_by IS NOT NULL; +CREATE INDEX file_change_new__profile_id__idx + ON file_change_new (profile_id) WHERE profile_id IS NOT NULL; +CREATE INDEX file_change_new__user_snapshots__idx + ON file_change_new (file_id) + WHERE label IS NOT NULL AND created_by = 'user' AND deleted_at IS NULL; +``` + +The `deleted_at` index carries `file_id` (the primary key is `(file_id, id)`): +the objects GC needs the `file_id` to lock by pair, and with the index it +reads the rows it has to delete without touching the table. + +```sql +SELECT count(*) AS partitions FROM pg_partition_tree('file_change_new') t + WHERE t.isleaf; +-- 16 +``` + +--- + +## Phase 2 — Bulk copy + +Copies every row (including the soft deleted ones: the GC manages them the +same way in the new table). The `kind` is computed from the `label`: a row +with a label is a snapshot, a row without one is a revision. + +The pointer of the copy lives in the database, not in the terminal: a batch +can be interrupted and resumed without losing anything. + +```sql +CREATE TABLE IF NOT EXISTS file_change_copy_state ( + name text PRIMARY KEY, + last_id uuid +); + +INSERT INTO file_change_copy_state (name, last_id) +VALUES ('bulk', '00000000-0000-0000-0000-000000000000') +ON CONFLICT (name) DO NOTHING; +``` + +One batch (idempotent). It returns how many rows it copied: + +```sql +WITH batch AS MATERIALIZED ( + SELECT id FROM file_change + WHERE id > (SELECT last_id FROM file_change_copy_state WHERE name = 'bulk') + ORDER BY id + LIMIT 10000 +), +copied AS ( + INSERT INTO file_change_new (id, file_id, created_at, session_id, revn, data, changes, + profile_id, features, label, version, created_by, + updated_at, deleted_at, migrations, locked_by, kind) + SELECT id, file_id, created_at, session_id, revn, data, changes, + profile_id, features, label, version, created_by, + updated_at, deleted_at, migrations, locked_by, + CASE WHEN label IS NOT NULL THEN 1 ELSE 2 END + FROM file_change + WHERE id IN (SELECT id FROM batch) + ON CONFLICT (file_id, id) DO UPDATE + SET created_at = EXCLUDED.created_at, + session_id = EXCLUDED.session_id, + revn = EXCLUDED.revn, + data = EXCLUDED.data, + changes = EXCLUDED.changes, + profile_id = EXCLUDED.profile_id, + features = EXCLUDED.features, + label = EXCLUDED.label, + version = EXCLUDED.version, + created_by = EXCLUDED.created_by, + updated_at = EXCLUDED.updated_at, + deleted_at = EXCLUDED.deleted_at, + migrations = EXCLUDED.migrations, + locked_by = EXCLUDED.locked_by, + kind = EXCLUDED.kind + RETURNING 1 +) +UPDATE file_change_copy_state + SET last_id = (SELECT id FROM batch ORDER BY id DESC LIMIT 1) + WHERE name = 'bulk' + AND EXISTS (SELECT 1 FROM copied) +RETURNING (SELECT count(*) FROM copied) AS copied; +``` + +Repeat until it returns `0`: + +```bash +until [ "$(psql -At -f batch-bulk.sql)" = "0" ]; do sleep 0.3; done +``` + +- The pointer is a `uuid` (random order): the pagination is stable but it does + not group by physical position. The total read is the heap, in + a scattered order. +- **Estimated duration**: time the first batches and extrapolate. The bulk + reads the heap plus the live TOAST once (see sizing) and writes about the + live data. Batches are rows divided by batch size: count your rows in + sizing and divide. +- Adjust the throttle (`sleep`) if the streaming replica falls behind or the + backend sees latency. Every batch detoastes its share of the TOAST: the cost + is I/O, not CPU. +- Watch: replica lag, `pg_stat_activity` (locks between the bulk and + autovacuum), the space of `pg_wal` and of the data dir. + +```sql +SELECT (SELECT count(*) FROM file_change) AS old_rows, + (SELECT count(*) FROM file_change_new) AS new_rows; +``` + +They match except for the rows written during the bulk (the delta passes pick +them up). + +--- + +## Phase 3 — Delta passes + +Every pass copies the rows modified since the bulk started. The window is a +constant in the script (not the output of the previous pass): the hour of +margin absorbs the clock drift between the application and the database, and +the drift between batches. + +```sql +-- :bulk_start = the time the bulk started, minus one hour +INSERT INTO file_change_new (id, file_id, created_at, session_id, revn, data, changes, + profile_id, features, label, version, created_by, + updated_at, deleted_at, migrations, locked_by, kind) +SELECT id, file_id, created_at, session_id, revn, data, changes, + profile_id, features, label, version, created_by, + updated_at, deleted_at, migrations, locked_by, + CASE WHEN label IS NOT NULL THEN 1 ELSE 2 END + FROM file_change + WHERE updated_at > TIMESTAMPTZ '2026-10-25 08:00:00+02' -- :bulk_start +ON CONFLICT (file_id, id) DO UPDATE + SET created_at = EXCLUDED.created_at, + session_id = EXCLUDED.session_id, + revn = EXCLUDED.revn, + data = EXCLUDED.data, + changes = EXCLUDED.changes, + profile_id = EXCLUDED.profile_id, + features = EXCLUDED.features, + label = EXCLUDED.label, + version = EXCLUDED.version, + created_by = EXCLUDED.created_by, + updated_at = EXCLUDED.updated_at, + deleted_at = EXCLUDED.deleted_at, + migrations = EXCLUDED.migrations, + locked_by = EXCLUDED.locked_by, + kind = EXCLUDED.kind; +``` + +- Repeat every 1-6 h (depending on the activity) until the pass copies very + few rows. With the index of phase 0.4 the pass is an index range over the + rows changed since the mark, so it takes seconds on a quiet instance; + without it, it scans the heap. To see how many are left: + +```sql +SELECT count(*) FROM file_change WHERE updated_at > TIMESTAMPTZ '2026-10-25 08:00:00+02'; +``` + +- The objects GC can keep running: its soft deletes are covered by the + watermark, and its hard deletes leave a row with `deleted_at` in the new + table that the GC itself cleans up later. The hard delete of a **file** is + blocked while the old table (or the new one) has rows for it, because the + foreign key on `file_id` is `NO ACTION`: the file stays in the trash and + the GC picks it up in the next run. **Do not run the cutover at the same + time as a big batch of the GC.** + +--- + +## Phase 4 — Cutover (a freeze of minutes) + +### 4.1 Before the lock + +- One last delta pass. +- Prepare and rehearse the cutover script in staging. + +### 4.2 The cutover (one transaction, a brief exclusive lock) + +- **Stop the backend and the workers before the transaction, and start them + again after it.** The pool keeps server-side prepared statements, and a + prepared statement holds the OID of the table it was prepared with: + renaming the table does not change that OID. A connection that had already + run `insert into file_change` five times (the default threshold of the + driver) would keep writing to `file_change_legacy` after the cutover, and + the pool recycles its connections every 30 minutes, so the window of + silent writes is long. +- Run the transaction. +- Start the backend and the workers again. + +```sql +BEGIN; + +LOCK TABLE file_change IN ACCESS EXCLUSIVE MODE; + +-- The final delta: the rows modified since the bulk started +INSERT INTO file_change_new (id, file_id, created_at, session_id, revn, data, changes, + profile_id, features, label, version, created_by, + updated_at, deleted_at, migrations, locked_by, kind) +SELECT id, file_id, created_at, session_id, revn, data, changes, + profile_id, features, label, version, created_by, + updated_at, deleted_at, migrations, locked_by, + CASE WHEN label IS NOT NULL THEN 1 ELSE 2 END + FROM file_change + WHERE updated_at > TIMESTAMPTZ '2026-10-25 08:00:00+02' +ON CONFLICT (file_id, id) DO UPDATE + SET updated_at = EXCLUDED.updated_at, + deleted_at = EXCLUDED.deleted_at, + label = EXCLUDED.label, + locked_by = EXCLUDED.locked_by, + created_by = EXCLUDED.created_by, + data = EXCLUDED.data, + changes = EXCLUDED.changes, + kind = EXCLUDED.kind; + +-- Check inside the lock +SELECT (SELECT count(*) FROM file_change) AS old_rows, + (SELECT count(*) FROM file_change_new) AS new_rows; + +-- The old table frees the canonical names (the same names the migration +-- 0158 uses, with the _legacy suffix) +ALTER TABLE file_change RENAME CONSTRAINT file_change_pkey + TO file_change_pkey_legacy; +ALTER TABLE file_change RENAME CONSTRAINT file_change_file_id_fkey + TO file_change_file_id_fkey_legacy; +ALTER TABLE file_change RENAME CONSTRAINT file_change_locked_by_fkey + TO file_change_locked_by_fkey_legacy; +ALTER TABLE file_change RENAME CONSTRAINT file_change_profile_id_fkey + TO file_change_profile_id_fkey_legacy; + +ALTER INDEX file_change__data_ref_id__idx + RENAME TO file_change__data_ref_id__idx_legacy; +ALTER INDEX file_change__deleted_at__idx + RENAME TO file_change__deleted_at__idx_legacy; +ALTER INDEX file_change__file_id__revn__idx + RENAME TO file_change__file_id__revn__idx_legacy; +ALTER INDEX file_change__locked_by__idx + RENAME TO file_change__locked_by__idx_legacy; +ALTER INDEX file_change__profile_id__idx + RENAME TO file_change__profile_id__idx_legacy; +ALTER INDEX file_change__system_snapshots__idx + RENAME TO file_change__system_snapshots__idx_legacy; +ALTER INDEX IF EXISTS file_change__user_snapshots__idx + RENAME TO file_change__user_snapshots__idx_legacy; +ALTER INDEX IF EXISTS file_change__updated_at__idx + RENAME TO file_change__updated_at__idx_legacy; + +-- And the new table takes them +ALTER TABLE file_change_new RENAME CONSTRAINT file_change_new_pkey + TO file_change_pkey; +ALTER TABLE file_change_new RENAME CONSTRAINT file_change_new_file_id_fkey + TO file_change_file_id_fkey; +ALTER TABLE file_change_new RENAME CONSTRAINT file_change_new_locked_by_fkey + TO file_change_locked_by_fkey; +ALTER TABLE file_change_new RENAME CONSTRAINT file_change_new_profile_id_fkey + TO file_change_profile_id_fkey; +ALTER TABLE file_change_new RENAME CONSTRAINT file_change_new_kind_check + TO file_change_kind_check; + +ALTER INDEX file_change_new__deleted_at__idx + RENAME TO file_change__deleted_at__idx; +ALTER INDEX file_change_new__file_id__revn__idx + RENAME TO file_change__file_id__revn__idx; +ALTER INDEX file_change_new__locked_by__idx + RENAME TO file_change__locked_by__idx; +ALTER INDEX file_change_new__profile_id__idx + RENAME TO file_change__profile_id__idx; +ALTER INDEX file_change_new__user_snapshots__idx + RENAME TO file_change__user_snapshots__idx; + +-- And the tables are swapped. The partitions and their indexes keep the +-- names they were born with (they are named after the partition, not after +-- the table), so there is nothing else to rename. +ALTER TABLE file_change RENAME TO file_change_legacy; +ALTER TABLE file_change_new RENAME TO file_change; + +-- The foreign key of the legacy table goes away: it is `NO ACTION`, so +-- while the legacy table has rows for a file, the objects GC cannot delete +-- that file (it deletes the rows of the file first, and the legacy ones are +-- not its business). The legacy table is frozen, and the rollback adds the +-- constraint back. +ALTER TABLE file_change_legacy + DROP CONSTRAINT file_change_file_id_fkey_legacy; + +COMMIT; + +ANALYZE file_change; +``` + +- `file_change_legacy` keeps its data and its indexes (with the `_legacy` + suffix): it stays frozen as the way back. Its foreign key on `file_id` is + dropped, so the objects GC can delete the files again. +- The writes to `file_change` during the lock fail: they are seconds-minutes, + the affected saves are retried. +- The objects GC and the four RPC commands keep working: same table name and + same index names after the rename. +- Once the backend is up again, check that the saves go to the partitioned + table and that nothing went to the table that is going away: + +```sql +-- the rows written after the cutover: this one grows with every save +SELECT count(*) FROM file_change + WHERE updated_at > TIMESTAMPTZ '2026-10-25 08:00:00+02' -- :cutover + +-- the rows that went to the legacy table: this one has to stay at 0 +SELECT count(*) FROM file_change_legacy + WHERE updated_at > TIMESTAMPTZ '2026-10-25 08:00:00+02'; -- :cutover +``` + +A number above zero in the second query means that some connection was +still writing with a prepared statement of the old table: those rows are +lost when the legacy table is dropped, so they have to be copied over +before that. + +### 4.3 Rollback + +- **Before the `COMMIT`**: `ROLLBACK` and that is it. +- **After the cutover**: the new writes go to the partitioned table. To revert, + rename the tables back and copy the delta the other way + (`INSERT INTO file_change_legacy ... WHERE updated_at > :cutover ON CONFLICT + (id) DO UPDATE`, carrying `data` too). The foreign key on `file_id` was dropped at the cutover, + so it goes back with `NOT VALID` (the existing rows are not checked) and it + is validated later: + +```sql +ALTER TABLE file_change_legacy + ADD CONSTRAINT file_change_file_id_fkey + FOREIGN KEY (file_id) REFERENCES file(id) DEFERRABLE NOT VALID; +``` + + Reasonable window: only if there is a serious defect in the first hours. + After that, treat any problem the normal way (the partitioned table is the + good state). + +--- + +## Phase 5 — After the cutover + +1. Check that the access paths prune to one partition (when the planner + chooses an index scan): + +```sql +EXPLAIN SELECT * FROM file_change + WHERE file_id = '00000000-0000-0000-0000-000000000000' + AND id = '00000000-0000-0000-0000-000000000000'; +``` + +It has to show a single partition (`file_change_NN`), not an `Append` of 16. + +2. Check that the schema is the one of the migration `0158`. Every row of this + query has to show `got = expected`: + +```sql +SELECT 'columns' AS what, count(*) AS got, 17 AS expected + FROM pg_attribute + WHERE attrelid = 'file_change'::regclass AND attnum > 0 AND NOT attisdropped +UNION ALL +SELECT 'partitions', count(*), 16 + FROM pg_partition_tree('file_change') t WHERE t.isleaf +UNION ALL +SELECT 'primary key', count(*), 1 + FROM pg_constraint + WHERE conrelid = 'file_change'::regclass AND contype = 'p' +UNION ALL +SELECT 'foreign keys', count(*), 3 + FROM pg_constraint + WHERE conrelid = 'file_change'::regclass AND contype = 'f' +UNION ALL +SELECT 'check constraints', count(*), 1 + FROM pg_constraint + WHERE conrelid = 'file_change'::regclass AND contype = 'c' +UNION ALL +SELECT 'indexes', count(*), 6 + FROM pg_indexes + WHERE tablename = 'file_change' AND schemaname = current_schema() +UNION ALL +SELECT 'partitions out of name', count(*), 0 + FROM pg_class + WHERE relkind = 'r' AND relname ~ '^file_change_new[0-9]' +UNION ALL +SELECT 'child indexes out of name', count(*), 0 + FROM pg_class c + JOIN pg_index i ON i.indexrelid = c.oid + JOIN pg_class p ON p.oid = i.indrelid + WHERE p.relname ~ '^file_change_[0-9]{2}$' + AND c.relname NOT LIKE p.relname || '_%'; +``` + + And the names, to compare them with the migration: + +```sql +SELECT a.attname AS column + FROM pg_attribute a + WHERE a.attrelid = 'file_change'::regclass AND a.attnum > 0 + AND NOT a.attisdropped + ORDER BY a.attnum; + +SELECT indexname FROM pg_indexes + WHERE tablename = 'file_change' AND schemaname = current_schema() + ORDER BY indexname; + +-- the partitions, with their name +SELECT c.relname AS partition + FROM pg_class c + JOIN pg_partition_tree('file_change') t ON t.relid = c.oid + WHERE t.isleaf + ORDER BY c.relname; + +-- the indexes of the partitions: their name starts with the partition +SELECT c.relname AS child_index + FROM pg_class c + JOIN pg_index i ON i.indexrelid = c.oid + JOIN pg_class p ON p.oid = i.indrelid + WHERE p.relname ~ '^file_change_[0-9]{2}$' + ORDER BY c.relname; +``` + +3. The partition-wise join is already on: the pool of the backend sets + `enable_partitionwise_join = on` on every connection (it is in the code, + there is nothing to change in the database role). + +4. Autovacuum: each of the 16 partitions has its own TOAST (check their + sizes with `pg_total_relation_size` per partition); with + `autovacuum_vacuum_scale_factor = 0.2` the threshold is reached earlier + and in steps. Nothing to change, just watch `last_autovacuum` of the + partitions the first week. + +5. The aggregations that do not filter by `file_id` (the team snapshot + quota and the telemetry counts) read all 16 partitions. To know their + cost on your instance, time them before and after with + `EXPLAIN (ANALYZE, BUFFERS)` and compare: + + - The team quota is already fixed in the code: it resolves the `file_id` of + the team first, which lets the planner prune the partitions when it + chooses an index scan (with a sequential scan it reads all of them). + - The telemetry scans the whole heap, the same before and after: the + partitioned run pays a small cost per partition on top. To remove that + scan, the index `file_change__created_at__idx` turns both metrics into an + index scan over the rows of the day, at the cost of one more index on the + biggest table. Decide it before phase 1, the index has to exist in both + tables. + +6. Drop the helper table of the copy and, days later, the old table: + +```sql +DROP TABLE IF EXISTS file_change_copy_state; +DROP TABLE file_change_legacy; -- out of hours: it generates WAL by + -- dropping the pages, let it roll +``` + +That is where the dead space comes back: the database shrinks by about the +size of the old table (see sizing). + +--- + +## Risks + +| Risk | Mitigation | +|---|---| +| Massive WAL during the bulk (proportional to the copy; see sizing) | Throttle between batches; watch `pg_wal`, the replication slots and the replica lag | +| Latency seen by the application during the bulk | Throttle; low traffic hours; the bulk is I/O and does not block writes | +| `updated_at` not reliable in the rows changed before the deploy | The deploy goes before the bulk (0.3) and the delta passes cover from the start of the bulk | +| Lock/unlock/soft delete during the passes | Covered by the watermark; the cutover includes the final delta inside the lock | +| Hard delete of the GC between passes | The row is left with `deleted_at` in the new table; the GC cleans it up later | +| Orphan rows if a `file` is deleted halfway | The foreign key on `file_id` is `NO ACTION`: the GC deletes the rows of the file before the file, so a chunk that fails leaves the file in the trash and the next run deletes it | +| The files in the trash cannot be deleted while the legacy table has rows for them | The foreign key of the legacy table is dropped at the cutover; until then the files wait in the trash | +| Saves failing during the lock of the cutover | Freeze of minutes in a low traffic window; manual retries; the rollback is documented | +| Planner regression after the cutover | Immediate `ANALYZE` + the pruning `EXPLAIN`; the pool already sets the GUC | +| Clock drift between the application and the database | The delta window starts an hour before the bulk; the margin absorbs the drift | + +## Success metrics + +- After the cutover: `file_change` (partitioned) holds about the live data + (see sizing). +- After the `DROP` of the old table: the database shrinks by about the size + that table had. +- The access paths that filter by `file_id` (the lookup of a snapshot, the + quota of a file and the lock of the GC by pair) prune to one partition when + the planner chooses an index scan (`EXPLAIN`). The quota of the team and + the discovery of the GC do not filter by a single file: they read all the + 16 partitions. +- No visible change for the user. diff --git a/operations/README.md b/operations/README.md new file mode 100644 index 0000000000..be1036e7c6 --- /dev/null +++ b/operations/README.md @@ -0,0 +1,16 @@ +# Operations + +Procedures that are executed by hand on an instance, not by the application. + +They are not part of the product: they describe how to do a one-off operation +on a running database, with the exact statements to run and what to check +between them. + +Each procedure lives here while the operation is alive. When the operation is +done, the file is either deleted or marked as historical, so the folder does +not grow with procedures that nobody will run again. + +| Procedure | Version | +|---|---| +| [Migrate legacy file payloads to `file_data`](2.19.0-migrate-legacy-file-data.md) | 2.19.0 | +| [Partition `file_change` by `file_id`](2.19.0-file-change-partition.md) | 2.19.0 |