diff --git a/.serena/memories/backend/subtleties.md b/.serena/memories/backend/subtleties.md index 1b4fa4ddec..2282c6d792 100644 --- a/.serena/memories/backend/subtleties.md +++ b/.serena/memories/backend/subtleties.md @@ -85,8 +85,8 @@ - A `file_data` row is one of three classes, told apart by `type`: `main` is the file's current state and carries `id = file_id`; `snapshot` is a point-in-time copy whose `id` is the one its `file_change` row shares; `fragment` is a piece of the pointer-map, referenced from the file's data. - Their lifecycles differ: `main` and `snapshot` live with the file (dated when it is deleted, undated when it is restored), while a `fragment` carries no date and is collected by `file_gc` once the file stops referencing it. Neither is a session buffer, so do not read a `fragment` as an ephemeral revision. - File snapshots (versions) keep their metadata row in `file_change` and their payload in `file_data` (`type = 'snapshot'`). With the `db` backend `file_change.data` stays NULL: that column is only written by the `legacy-db` backend, plus the rows written before the 2025-07 file storage refactor. -- Tell a snapshot apart by `label IS NOT NULL` (with `created_by` / `deleted_at` as needed), never by `file_change.data IS NOT NULL`: the label holds with every file data backend, and the payload column stopped meaning "this is a snapshot" when the payload moved out. Queries that identify versions must use the label. +- Tell a snapshot apart by `label IS NOT NULL` (with `created_by` / `deleted_at` as needed), never by `file_change.data IS NOT NULL`: the label holds with every file data backend, and the payload column stopped meaning "this is a snapshot" when the payload moved out. Queries that identify versions must use the label. `0157-migrate-legacy-file-data` moved the labelled leftovers to `file_data` and deleted the unlabelled payload rows, so from 2.19 on the label is the only marker there is. - The version quota counts of `app.rpc.quotes` (`sql:get-snapshots-per-file`, `sql:get-snapshots-per-team`) are served by the partial index `file_change__user_snapshots__idx`, whose predicate mirrors their where clause; changing one without the other drops the index only scan and the count walks the whole table again. - Revisions live only in `file_change` and have no `file_data` counterpart. They are born already scheduled for deletion one hour later (`app.rpc.commands.files-update`) and the snapshot paths cannot reach one, because they resolve their row through a `label IS NOT NULL` filter. -- A deletion date is assigned once and never moved: `restore-files` (`app.rpc.commands.files`) unmarks only durable history, the `:file` method of `app.tasks.delete-object` dates only rows that have no date yet, and `objects-gc` collects only rows that carry one. Unmarking a revision leaves it invisible to `objects-gc` for ever, because nothing sets that date again. +- A deletion date is assigned once and never moved: `restore-files` (`app.rpc.commands.files`) unmarks only durable history, which is `label IS NOT NULL` (the unlabelled rows that still carry a legacy payload are removed by `0157-migrate-legacy-file-data`, so restore leaves them dated), the `:file` method of `app.tasks.delete-object` dates only rows that have no date yet, and `objects-gc` collects only rows that carry one. Unmarking a revision leaves it invisible to `objects-gc` for ever, because nothing sets that date again. - Binfile v1/v3 import and the v3 export `get-file` step run inside one long transaction. They raise `idle_in_transaction_session_timeout` to 30 min (`app.binfile.common/transaction-idle-timeout-ms`) instead of disabling it, so a connection left idle while the ZIP is decoded or the file data is realized is bounded; `statement_timeout` stays at 300 s. diff --git a/backend/src/app/migrations.clj b/backend/src/app/migrations.clj index 34751179b7..42d1fab650 100644 --- a/backend/src/app/migrations.clj +++ b/backend/src/app/migrations.clj @@ -526,7 +526,10 @@ :fn (mg/resource "app/migrations/sql/0155-add-idx-to-file-change.sql")} {:name "0156-reset-pg-storage-defaults" - :fn (mg/resource "app/migrations/sql/0156-reset-pg-storage-defaults.sql")}]) + :fn (mg/resource "app/migrations/sql/0156-reset-pg-storage-defaults.sql")} + + {:name "0157-migrate-legacy-file-data" + :fn (mg/resource "app/migrations/sql/0157-migrate-legacy-file-data.sql")}]) (defn apply-migrations! [pool name migrations] diff --git a/backend/src/app/migrations/sql/0157-migrate-legacy-file-data.sql b/backend/src/app/migrations/sql/0157-migrate-legacy-file-data.sql new file mode 100644 index 0000000000..c7cd6d1fe4 --- /dev/null +++ b/backend/src/app/migrations/sql/0157-migrate-legacy-file-data.sql @@ -0,0 +1,55 @@ +-- Migrate legacy payloads to file_data: labelled file_change.data as +-- snapshots, file.data as mains (backend 'db', deleted_at copied so the +-- GC keeps collecting them), then delete orphan file_change rows with +-- data but no label (no read path reaches them). +-- +-- Idempotent: INSERTs ignore conflicts, UPDATEs only null the legacy +-- column where the file_data row exists. Big instances pre-migrate by +-- hand (operations/2.19.0-migrate-legacy-file-data.md) and fake this +-- step instead: +-- +-- INSERT INTO migrations (module, step) +-- VALUES ('main', '0157-migrate-legacy-file-data'); + +INSERT INTO file_data (file_id, id, created_at, modified_at, deleted_at, + type, backend, metadata, data) +SELECT file_id, id, created_at, updated_at, deleted_at, + 'snapshot', 'db', NULL, data + FROM file_change + WHERE data IS NOT NULL + AND label IS NOT NULL +ON CONFLICT (file_id, id) DO NOTHING; + +UPDATE file_change AS fc + SET data = NULL + WHERE fc.data IS NOT NULL + AND fc.label IS NOT NULL + AND EXISTS (SELECT 1 + FROM file_data AS fd + WHERE fd.file_id = fc.file_id + AND fd.id = fc.id + AND fd.type = 'snapshot'); + +INSERT INTO file_data (file_id, id, created_at, modified_at, deleted_at, + type, backend, metadata, data) +SELECT id, id, created_at, modified_at, deleted_at, + 'main', 'db', NULL, data + FROM file + WHERE data IS NOT NULL +ON CONFLICT (file_id, id) DO NOTHING; + +UPDATE file AS f + SET data = NULL + WHERE f.data IS NOT NULL + AND EXISTS (SELECT 1 + FROM file_data AS fd + WHERE fd.file_id = f.id + AND fd.id = f.id + AND fd.type = 'main'); + +-- Orphan revisions: legacy payloads without label. No file_data row +-- corresponds to them (snapshots are told apart by label), so there is +-- nothing to copy; the row itself is garbage. Delete it. +DELETE FROM file_change + WHERE data IS NOT NULL + AND label IS NULL; diff --git a/backend/src/app/rpc/commands/files.clj b/backend/src/app/rpc/commands/files.clj index ad5ebc3e30..d07a9c1280 100644 --- a/backend/src/app/rpc/commands/files.clj +++ b/backend/src/app/rpc/commands/files.clj @@ -1347,16 +1347,16 @@ WHERE file_id = ANY(?::uuid[])") (def ^:private sql:restore-file-changes - ;; Only durable history comes back: a snapshot (carries `label`) or a legacy - ;; payload row (carries `data`, the column the new model replaces with - ;; `file_data`). Revisions have neither and are deliberately left alone: they - ;; are born already scheduled for deletion one hour later, and the collector - ;; only looks at rows that carry a date, so unmarking one here would keep it - ;; in the table for ever. The `data` half is a guard for the rows still - ;; pending migration, not a dependency: drop it with the last one. + ;; Only durable history comes back: a snapshot, the rows that carry `label`. + ;; Everything else is deliberately left alone. Revisions are born already + ;; scheduled for deletion one hour later, and the collector only looks at + ;; rows that carry a date, so unmarking one here would keep it in the table + ;; for ever. Rows that still carry a legacy payload without a label are + ;; garbage that no read path reaches; 0157-migrate-legacy-file-data deletes + ;; them, so they must keep their date and go with the collector. "UPDATE file_change SET deleted_at = null WHERE file_id = ANY(?::uuid[]) - AND (label IS NOT NULL OR data IS NOT NULL)") + AND label IS NOT NULL") (def ^:private sql:restore-file-data "UPDATE file_data SET deleted_at = null diff --git a/backend/src/app/srepl/procs/fdata_storage.clj b/backend/src/app/srepl/procs/fdata_storage.clj deleted file mode 100644 index 5759aa43b9..0000000000 --- a/backend/src/app/srepl/procs/fdata_storage.clj +++ /dev/null @@ -1,141 +0,0 @@ -;; This Source Code Form is subject to the terms of the Mozilla Public -;; License, v. 2.0. If a copy of the MPL was not distributed with this -;; file, You can obtain one at http://mozilla.org/MPL/2.0/. -;; -;; Copyright (c) KALEIDOS SUBSIDIARY SL - -(ns app.srepl.procs.fdata-storage - (:require - [app.common.logging :as l] - [app.db :as db])) - -;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; -;; SNAPSHOTS -;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; - -(def sql:get-unmigrated-snapshots - "SELECT fc.id, fc.file_id - FROM file_change AS fc - WHERE fc.data IS NOT NULL - AND fc.label IS NOT NULL - ORDER BY fc.id ASC") - -(def sql:get-migrated-snapshots - "SELECT f.id, f.file_id - FROM file_data AS f - WHERE f.data IS NOT NULL - AND f.type = 'snapshot' - AND f.id != f.file_id - ORDER BY f.id ASC") - -(defn migrate-snapshot-to-storage - "Migrate the current existing files to store data in new storage - tables." - {:query sql:get-unmigrated-snapshots} - [{:keys [::db/conn]} {:keys [id file-id]} & {:as options}] - (let [{:keys [id file-id data created-at updated-at]} - (db/get* conn :file-change {:id id :file-id file-id} - ::db/for-update true - ::db/remove-deleted false)] - (when data - (l/inf :hint "migrating snapshot" :file-id (str file-id) :id (str id)) - (db/update! conn :file-change - {:data nil} - {:id id :file-id file-id} - {::db/return-keys false}) - (db/insert! conn :file-data - {:backend "db" - :metadata nil - :type "snapshot" - :data data - :created-at created-at - :modified-at updated-at - :file-id file-id - :id id} - {::db/return-keys false})))) - -(defn rollback-snapshot-from-storage - "Migrate back to the file table storage." - {:query sql:get-unmigrated-snapshots} - [{:keys [::db/conn]} {:keys [id file-id]} & {:as opts}] - (when-let [{:keys [id file-id data]} - (db/get* conn :file-data {:id id :file-id file-id :type "snapshot"} - ::db/for-update true - ::db/remove-deleted false)] - (l/inf :hint "rollback snapshot" :file-id (str file-id) :id (str id)) - (db/update! conn :file-change - {:data data} - {:id id :file-id file-id} - {::db/return-keys false}) - (db/delete! conn :file-data - {:id id :file-id file-id :type "snapshot"} - {::db/return-keys false}))) - -;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; -;; FILES -;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; - -(def sql:get-unmigrated-files - "SELECT f.id - FROM file AS f - WHERE f.data IS NOT NULL - ORDER BY f.modified_at ASC") - -(def sql:get-migrated-files - "SELECT f.id, f.file_id - FROM file_data AS f - WHERE f.data IS NOT NULL - AND f.id = f.file_id - ORDER BY f.id ASC") - -(defn migrate-file-to-storage - "Migrate the current existing files to store data in new storage - tables." - {:query sql:get-unmigrated-files} - [{:keys [::db/conn] :as cfg} {:keys [id]} & {:as opts}] - (let [{:keys [id data created-at modified-at]} - (db/get* conn :file {:id id} - ::db/for-update true - ::db/remove-deleted false)] - - (when data - (l/inf :hint "migrating file" :file-id (str id)) - - (db/update! conn :file {:data nil} {:id id} ::db/return-keys false) - (db/insert! conn :file-data - {:backend "db" - :metadata nil - :type "main" - :data data - :created-at created-at - :modified-at modified-at - :file-id id - :id id} - {::db/return-keys false})) - - (let [snapshots-sql - (str "WITH snapshots AS (" sql:get-unmigrated-snapshots ") " - "SELECT s.* FROM snapshots AS s WHERE s.file_id = ?")] - (run! (fn [params] - (migrate-snapshot-to-storage cfg params opts)) - (db/plan cfg [snapshots-sql id]))))) - - -(defn rollback-file-from-storage - "Migrate back to the file table storage." - {:query sql:get-migrated-files} - [{:keys [::db/conn] :as cfg} {:keys [id]} & {:as opts}] - (when-let [{:keys [id data]} - (db/get* conn :file-data {:id id :file-id id :type "main"} - ::db/for-update true - ::db/remove-deleted false)] - (l/inf :hint "rollback file" :file-id (str id)) - (db/update! conn :file {:data data} {:id id} ::db/return-keys false) - (db/delete! conn :file-data {:file-id id :id id :type "main"} ::db/return-keys false) - - (let [snapshots-sql - (str "WITH snapshots AS (" sql:get-migrated-snapshots ") " - "SELECT s.* FROM snapshots AS s WHERE s.file_id = ?")] - (run! (fn [params] - (rollback-snapshot-from-storage cfg params opts)) - (db/plan cfg [snapshots-sql id]))))) diff --git a/backend/test/backend_tests/file_change_retention_test.clj b/backend/test/backend_tests/file_change_retention_test.clj index 7cede5a164..3929e0b129 100644 --- a/backend/test/backend_tests/file_change_retention_test.clj +++ b/backend/test/backend_tests/file_change_retention_test.clj @@ -118,8 +118,8 @@ (t/testing "the snapshot comes back" (t/is (nil? (:deleted-at (find-change snapshot-id))))) - (t/testing "the legacy payload row comes back" - (t/is (nil? (:deleted-at (find-change legacy-id)))))))) + (t/testing "the legacy payload row keeps its date" + (t/is (= trash-at (:deleted-at (find-change legacy-id)))))))) (t/deftest deleting-a-file-does-not-push-the-revision-schedule-out (binding [ct/*clock* (ct/fixed-clock test-now)] diff --git a/backend/test/backend_tests/migrations_legacy_file_data_test.clj b/backend/test/backend_tests/migrations_legacy_file_data_test.clj new file mode 100644 index 0000000000..e2a31ce2bf --- /dev/null +++ b/backend/test/backend_tests/migrations_legacy_file_data_test.clj @@ -0,0 +1,180 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns backend-tests.migrations-legacy-file-data-test + "Covers the 0157 data migration: legacy file/file_change payloads move + to file_data, keeping timestamps and deleted_at, nulling the legacy + column; unlabelled file_change payloads (orphan revisions) are + deleted." + (:require + [app.common.time :as ct] + [app.common.uuid :as uuid] + [app.db :as db] + [app.util.blob :as blob] + [backend-tests.helpers :as th] + [clojure.java.io :as io] + [clojure.test :as t] + [next.jdbc :as jdbc])) + +(t/use-fixtures :once th/state-init) +(t/use-fixtures :each th/database-reset) + +(defn- run-migration + [] + (let [sql (slurp (io/resource "app/migrations/sql/0157-migrate-legacy-file-data.sql"))] + (jdbc/execute! th/*pool* [sql]))) + +(defn- get-file-data + [file-id id] + (th/db-get :file-data {:file-id file-id :id id} {::db/remove-deleted false})) + +(t/deftest migrates-legacy-file-and-snapshot + (let [profile (th/create-profile* 1) + file (th/create-file* 1 {:profile-id (:id profile) + :project-id (:default-project-id profile)}) + fid (:id file) + + ;; Simulate an unmigrated main: move the payload back to + ;; file.data and drop the file_data row. + main-data (:data (get-file-data fid fid)) + _ (t/is (some? main-data) "expected file_data main row from create-file*") + file-row (th/db-get :file {:id fid} {::db/remove-deleted false}) + _ (th/db-delete! :file-data {:file-id fid :id fid}) + _ (th/db-update! :file {:data main-data} {:id fid}) + + ;; Simulate an unmigrated snapshot (labelled) with its own + ;; timestamps and a deletion date. + snap-id (uuid/next) + snap-data (blob/encode {:snapshot 1}) + created-at (ct/in-past "2h") + updated-at (ct/in-past "1h") + deleted-at (ct/in-past "30m") + _ (th/db-insert! :file-change + {:id snap-id + :file-id fid + :revn 1 + :data snap-data + :label "v1" + :created-at created-at + :updated-at updated-at + :deleted-at deleted-at}) + + ;; Orphan revision (data, no label): deleted by the migration. + rev-id (uuid/next) + rev-data (blob/encode {:rev 1}) + _ (th/db-insert! :file-change + {:id rev-id + :file-id fid + :revn 2 + :data rev-data}) + + ;; Live revision (changes, no data, no label): untouched. + live-id (uuid/next) + _ (th/db-insert! :file-change + {:id live-id + :file-id fid + :revn 3 + :changes (blob/encode [{:type :dummy}])})] + + (run-migration) + + (t/testing "main payload moves to file_data as type main" + (let [fd (get-file-data fid fid)] + (t/is (some? fd)) + (t/is (= "main" (:type fd))) + (t/is (= "db" (:backend fd))) + (t/is (nil? (:metadata fd))) + (t/is (= (seq ^bytes main-data) (seq ^bytes (:data fd)))) + (t/is (= (:created-at file-row) (:created-at fd))) + (t/is (= (:modified-at file-row) (:modified-at fd))) + (t/is (= (:deleted-at file-row) (:deleted-at fd))))) + + (t/testing "snapshot payload moves to file_data as type snapshot" + (let [fd (get-file-data fid snap-id)] + (t/is (some? fd)) + (t/is (= "snapshot" (:type fd))) + (t/is (= "db" (:backend fd))) + (t/is (nil? (:metadata fd))) + (t/is (= (seq ^bytes snap-data) (seq ^bytes (:data fd)))) + ;; NOTE: timestamptz stores microsecond precision while the + ;; test instants carry nanoseconds, so compare at millis. + (t/is (= (inst-ms created-at) (inst-ms (:created-at fd)))) + (t/is (= (inst-ms updated-at) (inst-ms (:modified-at fd)))) + (t/is (= (inst-ms deleted-at) (inst-ms (:deleted-at fd)))))) + + (t/testing "legacy columns are nulled" + (t/is (nil? (:data (th/db-get :file {:id fid} {::db/remove-deleted false})))) + (t/is (nil? (:data (th/db-get :file-change {:id snap-id} {::db/remove-deleted false}))))) + + (t/testing "unlabelled payload row is deleted, live revision stays" + (t/is (nil? (th/db-get :file-change {:id rev-id} {::db/remove-deleted false}))) + (let [row (th/db-get :file-change {:id live-id} {::db/remove-deleted false})] + (t/is (some? row)) + (t/is (some? (:changes row))))) + + (t/testing "second run is a no-op" + (run-migration) + (t/is (some? (get-file-data fid fid))) + (t/is (some? (get-file-data fid snap-id))) + (t/is (nil? (:data (th/db-get :file {:id fid} {::db/remove-deleted false})))) + (t/is (nil? (:data (th/db-get :file-change {:id snap-id} {::db/remove-deleted false})))) + (t/is (nil? (th/db-get :file-change {:id rev-id} {::db/remove-deleted false})))))) + +(t/deftest keeps-preexisting-file-data-row + (let [profile (th/create-profile* 1) + file (th/create-file* 1 {:profile-id (:id profile) + :project-id (:default-project-id profile)}) + fid (:id file) + + ;; file_data already holds the current payload; the legacy + ;; column keeps a stale copy that must not overwrite it. + current (:data (get-file-data fid fid)) + stale (blob/encode {:stale true}) + _ (t/is (some? current)) + _ (t/is (not= (seq ^bytes current) (seq ^bytes stale))) + _ (th/db-update! :file {:data stale} {:id fid})] + + (run-migration) + + (t/testing "file_data keeps its data and legacy is nulled" + (let [fd (get-file-data fid fid)] + (t/is (= (seq ^bytes current) (seq ^bytes (:data fd))))) + (t/is (nil? (:data (th/db-get :file {:id fid} {::db/remove-deleted false}))))))) + +(t/deftest keeps-preexisting-snapshot-row + (let [profile (th/create-profile* 1) + file (th/create-file* 1 {:profile-id (:id profile) + :project-id (:default-project-id profile)}) + fid (:id file) + + ;; file_data already holds the snapshot payload; the legacy column + ;; keeps a stale copy that must not overwrite it. + snap-id (uuid/next) + current (blob/encode {:snapshot :current}) + stale (blob/encode {:snapshot :stale}) + _ (t/is (not= (seq ^bytes current) (seq ^bytes stale))) + _ (th/db-insert! :file-data + {:file-id fid + :id snap-id + :type "snapshot" + :backend "db" + :metadata nil + :data current}) + _ (th/db-insert! :file-change + {:id snap-id + :file-id fid + :revn 1 + :label "v1" + :data stale})] + + (run-migration) + + (t/testing "file_data keeps its data and legacy is nulled" + (let [fd (get-file-data fid snap-id)] + (t/is (= (seq ^bytes current) (seq ^bytes (:data fd))))) + (t/is (nil? (:data (th/db-get :file-change {:id snap-id} + {::db/remove-deleted false}))))))) + diff --git a/operations/2.19.0-migrate-legacy-file-data.md b/operations/2.19.0-migrate-legacy-file-data.md new file mode 100644 index 0000000000..df3c82d917 --- /dev/null +++ b/operations/2.19.0-migrate-legacy-file-data.md @@ -0,0 +1,323 @@ +# Migrate legacy file payloads to `file_data` + +Online procedure to move the legacy file payloads (`file.data` and labelled +`file_change.data`) to `file_data` and to delete the orphan `file_change` +rows that carry `data` without a `label`. + +- Migration: `backend/src/app/migrations/sql/0157-migrate-legacy-file-data.sql` +- Issue: + +The result has to be **exactly** the end state of the migration, because on a +big instance the migration is marked as applied and never runs. This procedure +runs the same statements by hand, in small batched transactions. + +Use this only if the automatic migration would take too long on the instance +(millions of snapshots, hundreds of GB in TOAST). Small and medium instances +run the migration automatically on boot and need nothing from here. + +Run it before upgrading to 2.19.0. + +## What changes + +- Every `file_change` row with `data IS NOT NULL AND label IS NOT NULL` + (a snapshot) gets a `file_data` row with `type='snapshot'`, + `backend='db'`, `metadata=NULL`, `created_at` from `created_at`, + `modified_at` from `updated_at`, and `deleted_at` copied as is. Then the + legacy `file_change.data` is set to `NULL`. +- Every `file` row with `data IS NOT NULL` gets a `file_data` row with + `type='main'` (and `file_id = id`), the same `backend`/`metadata` mapping + and the timestamps copied one to one. Then the legacy `file.data` is set + to `NULL`. +- Every `file_change` row with `data IS NOT NULL AND label IS NULL` (an + orphan revision) is deleted outright. No read path reaches these rows: + snapshots resolve through `label IS NOT NULL`, lagged changes read the + `changes` column (not `data`), and nothing references `file_change` rows + from other tables. Rows carrying `deleted_at` would be collected by + `objects-gc` anyway; undated ones would sit in the table and its TOAST + for ever. + +`deleted_at` is copied on purpose: the write path always writes it to +`file_data`, the GC only collects `file_data` rows carrying it, and reads +never filter on `file_data.deleted_at`. Without the copy, migrated rows of +deleted files and snapshots would be orphaned: nobody reads them and nobody +collects them. + +## Why this mechanism + +A single-statement copy inside the transaction of the startup does not fit a +big instance, so the copy runs here in small batched transactions: + +- One statement is one transaction. Never wrap several batches in one + transaction. Run a single stream of batches at a time. +- `FOR UPDATE SKIP LOCKED`: the batches never block the application and are + never blocked by it. A row skipped by a lock stays in the index and the + next execution picks it up. +- Partial temp indexes hold only the pending rows, so every execution finds + its next batch without rescanning what is done, and migrated rows + (`data = NULL`) and deleted orphans drop out of the index on their own. + No cursor is needed. Do not `VACUUM` mid-run to clean them: it would walk + the whole TOAST. Drop them at the end. +- No `ON CONFLICT DO NOTHING` here on purpose: a duplicate key must abort + the batch (atomically: it migrates nothing) so the conflict is + investigated instead of silently skipped. The automatic migration is + lenient so a small instance never fails to boot; here strictness is what + keeps the data auditable. + +--- + +## Phase 0 — Before anything + +### 0.1 Mark the migration as applied + +Do this first: it registers the automatic migration as done, so a backend +restart mid-procedure never runs it on boot (it would copy the whole legacy +payload inside the startup transaction). + +```sql +INSERT INTO migrations (module, step) +VALUES ('main', '0157-migrate-legacy-file-data'); +``` + +The module is `main` and the step has no `.sql` (see +`backend/src/app/util/migrations.clj`). + +### 0.2 Counters + +Repeat the counters any time to follow the progress. Each batch loop ends +when it returns 0; the check in phase 3 confirms nothing is pending. + +```sql +-- Remaining legacy rows. +SELECT (SELECT count(*) FROM file_change + WHERE data IS NOT NULL AND label IS NOT NULL) AS snapshots_remaining, + (SELECT count(*) FROM file_change + WHERE data IS NOT NULL AND label IS NULL) AS orphans_remaining, + (SELECT count(*) FROM file + WHERE data IS NOT NULL) AS files_remaining; +``` + +Optional, reference only: how much is already migrated. Skip it unless +curious: it is a full count over the partitioned table and can take a +while on a big instance. For progress, the remaining counters above and +the per-batch counts are enough. + +```sql +SELECT (SELECT count(*) FROM file_data + WHERE type = 'snapshot' AND backend = 'db') AS snapshots_in_file_data, + (SELECT count(*) FROM file_data + WHERE type = 'main' AND backend = 'db') AS mains_in_file_data; +``` + +## Phase 1 — Temp indexes + +Required by the batch loops, created before them. `CONCURRENTLY` so writes +are not blocked; each statement is its own transaction. + +```sql +CREATE INDEX CONCURRENTLY tmp_file_change_mig ON file_change (id) + WHERE data IS NOT NULL AND label IS NOT NULL; +CREATE INDEX CONCURRENTLY tmp_file_change_orphan_mig ON file_change (id) + WHERE data IS NOT NULL AND label IS NULL; +CREATE INDEX CONCURRENTLY tmp_file_mig ON file (id) + WHERE data IS NOT NULL; +``` + +A failed `CONCURRENTLY` build stays `INVALID` and the planner ignores it +silently, so verify all three indexes exist and are valid: + +```sql +SELECT indexrelid::regclass AS index, indisvalid AS valid + FROM pg_index + WHERE indexrelid::regclass::text LIKE 'tmp_%mig'; +``` + +## Phase 2 — Batch loops + +Repeat each statement until it returns 0. Lower `LIMIT` if a batch hits +`statement_timeout` or memory pressure (`SET statement_timeout = '15min'` +per batch is the usual guard). + +### 2.1 Delete orphans + +Orphans first (or interleaved with the other two; the row sets are +disjoint, and deleting TOAST-heavy rows early frees space for the rest): + +```sql +WITH doomed AS ( + SELECT id + FROM file_change + WHERE data IS NOT NULL + AND label IS NULL + ORDER BY id + LIMIT 10000 + FOR UPDATE SKIP LOCKED +), +del AS ( + DELETE FROM file_change fc + USING doomed + WHERE fc.id = doomed.id + RETURNING fc.id +) +SELECT count(*) AS deleted + FROM del; +``` + +### 2.2 Migrate snapshots + +Repeat until it returns 0 (`~120 MB` per transaction at `LIMIT 10000`): + +```sql +WITH batch AS ( + SELECT id, file_id, data, created_at, updated_at, deleted_at + FROM file_change + WHERE data IS NOT NULL + AND label IS NOT NULL + ORDER BY id + LIMIT 10000 + FOR UPDATE SKIP LOCKED +), +-- NO ON CONFLICT on purpose: duplicates must abort (see above). +-- upd reads from ins (RETURNING), not from batch. +ins AS ( + INSERT INTO file_data (file_id, id, created_at, modified_at, deleted_at, + type, backend, metadata, data) + SELECT file_id, id, created_at, updated_at, deleted_at, + 'snapshot', 'db', NULL, data + FROM batch + RETURNING file_id, id +), +-- Only the origin of what was really inserted is nulled. On conflict +-- the whole transaction aborts, so ins == batch whenever this runs. +upd AS ( + UPDATE file_change fc + SET data = NULL + FROM ins + WHERE ins.file_id = fc.file_id + AND ins.id = fc.id + RETURNING fc.id +) +SELECT count(*) AS migrated + FROM upd; +``` + +### 2.3 Migrate files + +Repeat until it returns 0 (tune `LIMIT` to the average `file.data` size; +start at 5000): + +```sql +WITH batch AS ( + SELECT id, data, created_at, modified_at, deleted_at + FROM file + WHERE data IS NOT NULL + ORDER BY id + LIMIT 5000 + FOR UPDATE SKIP LOCKED +), +ins AS ( + INSERT INTO file_data (file_id, id, created_at, modified_at, deleted_at, + type, backend, metadata, data) + SELECT id, id, created_at, modified_at, deleted_at, + 'main', 'db', NULL, data + FROM batch + RETURNING file_id, id +), +upd AS ( + UPDATE file f + SET data = NULL + FROM ins + WHERE ins.file_id = f.id + AND ins.id = f.id + RETURNING f.id +) +SELECT count(*) AS migrated + FROM upd; +``` + +### 2.4 Duplicate keys + +If a batch aborts on duplicate key, stop and do not blind-retry it: it +means `file_data` already holds that key. Locate the overlap: + +```sql +-- Snapshots with legacy data AND a file_data key already present. +SELECT c.file_id, c.id, c.created_by, c.deleted_at, + pg_size_pretty(length(c.data)) AS legacy_size, + pg_size_pretty(length(fd.data)) AS file_data_size, + fd.modified_at AS file_data_modified_at + FROM file_change c + JOIN file_data fd ON fd.file_id = c.file_id + AND fd.id = c.id + AND fd.type = 'snapshot' + WHERE c.data IS NOT NULL + ORDER BY c.id + LIMIT 20; + +-- Files with legacy data AND a file_data key already present. +SELECT f.id, + pg_size_pretty(length(f.data)) AS legacy_size, + pg_size_pretty(length(fd.data)) AS file_data_size, + fd.modified_at AS file_data_modified_at + FROM file f + JOIN file_data fd ON fd.file_id = f.id + AND fd.id = f.id + AND fd.type = 'main' + WHERE f.data IS NOT NULL + ORDER BY f.id + LIMIT 20; +``` + +Resolve manually, one row at a time, only when `file_data` holds equal or +newer data (compare `modified_at` and sizes above): the legacy copy is +redundant, null the origin, leave `file_data` untouched. + +```sql +-- UPDATE file_change SET data = NULL WHERE file_id = '...' AND id = '...'; +-- UPDATE file SET data = NULL WHERE id = '...'; +``` + +## Phase 3 — Verification and cleanup + +Each loop above already ended returning 0; confirm nothing is pending (if +anything shows up, repeat that loop): + +```sql +-- Remaining legacy rows: all three must read 0. +SELECT (SELECT count(*) FROM file_change + WHERE data IS NOT NULL AND label IS NOT NULL) AS snapshots_remaining, + (SELECT count(*) FROM file_change + WHERE data IS NOT NULL AND label IS NULL) AS orphans_remaining, + (SELECT count(*) FROM file + WHERE data IS NOT NULL) AS files_remaining; +``` + +```sql +-- Spot check: migrated rows exist in file_data with backend 'db'. +SELECT fd.file_id, fd.id, fd.type, fd.backend, + pg_size_pretty(length(fd.data)) AS size_in_file_data + FROM file_data fd + ORDER BY fd.file_id, fd.id + LIMIT 5; +``` + +Drop the temp indexes first, so the vacuum below skips them (each batch +left its dead entries behind), then reclaim the dead tuples and refresh +planner stats. Plain `VACUUM` does not block writes; `VERBOSE` prints +progress, which matters on big tables: + +```sql +DROP INDEX CONCURRENTLY tmp_file_change_mig; +DROP INDEX CONCURRENTLY tmp_file_change_orphan_mig; +DROP INDEX CONCURRENTLY tmp_file_mig; +VACUUM (ANALYZE ON, VERBOSE ON) file_change; +VACUUM (ANALYZE ON, VERBOSE ON) file; +-- file_data is partitioned into 16 tables: the statement walks all of +-- them, so it can take a while. Without it the planner stats of the fresh +-- inserts stay stale until autovacuum catches up. +VACUUM (ANALYZE ON, VERBOSE ON) file_data; +``` + +## Phase 4 — Upgrade + +The migration was marked as applied in phase 0, so the new version skips +it on boot. Deploy normally.