github/penpot
1
0
Fork 0
mirror of https://github.com/penpot/penpot.git synced 2026-11-05 09:30:38 +00:00
Code Issues Packages Projects Releases Wiki Activity

✨ Migrate legacy file payloads to file_data and delete orphans (#12785)

Browse Source 
* ✨ Migrate legacy file payloads to file_data and delete orphans

Copy file.data and labelled file_change.data to file_data on boot
(type main/snapshot, backend db, deleted_at preserved), then delete
file_change rows with data but no label, which no read path reaches.
Large instances pre-migrate with batched statements and fake this
step. Closes #12784.

AI-assisted-by: muse-spark-1.3-contributor

* 📚 Add 2.19.0 operations runbook for legacy file_data migration

Manual pre-migration procedure for big instances, same statements as
migration 0157 in batched transactions, following the convention of
the file-change partition runbook. Closes #12784.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Address review feedback on the legacy file data migration

Fix the issues raised in the review of the migration that moves the
legacy file payloads to file_data.

The migration deletes the unlabelled file_change rows that carry data,
but restore-files still cleared their date, and the retention test
asserted they came back. A row without a date never reaches objects-gc,
which is the leak fixed in #12419. Restore now unmarks only snapshots,
the rows that carry a label, and the retention test checks that the
legacy payload keeps its date instead of coming back.

Cover the conflict path that had no test: a pre-existing snapshot that
collides with a labelled change row, and a second run that must leave
the legacy columns null and the orphan row absent.

The runbook analyses file_data after the inserts, so the planner does
not wait for autovacuum.

Drop app.srepl.procs.fdata-storage: the copy it performed is what the
migration replaces and nothing requires it.

Closes #12784

AI-assisted-by: deepseek-v4.1-flash
This commit is contained in:
Andrey Antukh 2026-11-04 12:25:01 +01:00 committed by GitHub
parent 0e9a9a8950
commit 35baa06606
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
8 changed files with 574 additions and 154 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
.serena/memories/backend/subtleties.md
View File

@ -85,8 +85,8 @@
- A `file_data` row is one of three classes, told apart by `type`: `main` is the file's current state and carries `id = file_id`; `snapshot` is a point-in-time copy whose `id` is the one its `file_change` row shares; `fragment` is a piece of the pointer-map, referenced from the file's data.
- Their lifecycles differ: `main` and `snapshot` live with the file (dated when it is deleted, undated when it is restored), while a `fragment` carries no date and is collected by `file_gc` once the file stops referencing it. Neither is a session buffer, so do not read a `fragment` as an ephemeral revision.
- File snapshots (versions) keep their metadata row in `file_change` and their payload in `file_data` (`type = 'snapshot'`). With the `db` backend `file_change.data` stays NULL: that column is only written by the `legacy-db` backend, plus the rows written before the 2025-07 file storage refactor.
- Tell a snapshot apart by `label IS NOT NULL` (with `created_by` / `deleted_at` as needed), never by `file_change.data IS NOT NULL`: the label holds with every file data backend, and the payload column stopped meaning "this is a snapshot" when the payload moved out. Queries that identify versions must use the label.
- Tell a snapshot apart by `label IS NOT NULL` (with `created_by` / `deleted_at` as needed), never by `file_change.data IS NOT NULL`: the label holds with every file data backend, and the payload column stopped meaning "this is a snapshot" when the payload moved out. Queries that identify versions must use the label. `0157-migrate-legacy-file-data` moved the labelled leftovers to `file_data` and deleted the unlabelled payload rows, so from 2.19 on the label is the only marker there is.
- The version quota counts of `app.rpc.quotes` (`sql:get-snapshots-per-file`, `sql:get-snapshots-per-team`) are served by the partial index `file_change__user_snapshots__idx`, whose predicate mirrors their where clause; changing one without the other drops the index only scan and the count walks the whole table again.
- Revisions live only in `file_change` and have no `file_data` counterpart. They are born already scheduled for deletion one hour later (`app.rpc.commands.files-update`) and the snapshot paths cannot reach one, because they resolve their row through a `label IS NOT NULL` filter.
- A deletion date is assigned once and never moved: `restore-files` (`app.rpc.commands.files`) unmarks only durable history, the `:file` method of `app.tasks.delete-object` dates only rows that have no date yet, and `objects-gc` collects only rows that carry one. Unmarking a revision leaves it invisible to `objects-gc` for ever, because nothing sets that date again.
- A deletion date is assigned once and never moved: `restore-files` (`app.rpc.commands.files`) unmarks only durable history, which is `label IS NOT NULL` (the unlabelled rows that still carry a legacy payload are removed by `0157-migrate-legacy-file-data`, so restore leaves them dated), the `:file` method of `app.tasks.delete-object` dates only rows that have no date yet, and `objects-gc` collects only rows that carry one. Unmarking a revision leaves it invisible to `objects-gc` for ever, because nothing sets that date again.
- Binfile v1/v3 import and the v3 export `get-file` step run inside one long transaction. They raise `idle_in_transaction_session_timeout` to 30 min (`app.binfile.common/transaction-idle-timeout-ms`) instead of disabling it, so a connection left idle while the ZIP is decoded or the file data is realized is bounded; `statement_timeout` stays at 300 s.

5
backend/src/app/migrations.clj
View File

@ -526,7 +526,10 @@
:fn (mg/resource "app/migrations/sql/0155-add-idx-to-file-change.sql")}
{:name "0156-reset-pg-storage-defaults"
:fn (mg/resource "app/migrations/sql/0156-reset-pg-storage-defaults.sql")}])
:fn (mg/resource "app/migrations/sql/0156-reset-pg-storage-defaults.sql")}
{:name "0157-migrate-legacy-file-data"
:fn (mg/resource "app/migrations/sql/0157-migrate-legacy-file-data.sql")}])
(defn apply-migrations!
[pool name migrations]

55
backend/src/app/migrations/sql/0157-migrate-legacy-file-data.sql Normal file
View File

@ -0,0 +1,55 @@
-- Migrate legacy payloads to file_data: labelled file_change.data as
-- snapshots, file.data as mains (backend 'db', deleted_at copied so the
-- GC keeps collecting them), then delete orphan file_change rows with
-- data but no label (no read path reaches them).
--
-- Idempotent: INSERTs ignore conflicts, UPDATEs only null the legacy
-- column where the file_data row exists. Big instances pre-migrate by
-- hand (operations/2.19.0-migrate-legacy-file-data.md) and fake this
-- step instead:
--
-- INSERT INTO migrations (module, step)
-- VALUES ('main', '0157-migrate-legacy-file-data');
INSERT INTO file_data (file_id, id, created_at, modified_at, deleted_at,
type, backend, metadata, data)
SELECT file_id, id, created_at, updated_at, deleted_at,
'snapshot', 'db', NULL, data
FROM file_change
WHERE data IS NOT NULL
AND label IS NOT NULL
ON CONFLICT (file_id, id) DO NOTHING;
UPDATE file_change AS fc
SET data = NULL
WHERE fc.data IS NOT NULL
AND fc.label IS NOT NULL
AND EXISTS (SELECT 1
FROM file_data AS fd
WHERE fd.file_id = fc.file_id
AND fd.id = fc.id
AND fd.type = 'snapshot');
INSERT INTO file_data (file_id, id, created_at, modified_at, deleted_at,
type, backend, metadata, data)
SELECT id, id, created_at, modified_at, deleted_at,
'main', 'db', NULL, data
FROM file
WHERE data IS NOT NULL
ON CONFLICT (file_id, id) DO NOTHING;
UPDATE file AS f
SET data = NULL
WHERE f.data IS NOT NULL
AND EXISTS (SELECT 1
FROM file_data AS fd
WHERE fd.file_id = f.id
AND fd.id = f.id
AND fd.type = 'main');
-- Orphan revisions: legacy payloads without label. No file_data row
-- corresponds to them (snapshots are told apart by label), so there is
-- nothing to copy; the row itself is garbage. Delete it.
DELETE FROM file_change
WHERE data IS NOT NULL
AND label IS NULL;

16
backend/src/app/rpc/commands/files.clj
View File

@ -1347,16 +1347,16 @@
WHERE file_id = ANY(?::uuid[])")
(def ^:private sql:restore-file-changes
;; Only durable history comes back: a snapshot (carries `label`) or a legacy
;; payload row (carries `data`, the column the new model replaces with
;; `file_data`). Revisions have neither and are deliberately left alone: they
;; are born already scheduled for deletion one hour later, and the collector
;; only looks at rows that carry a date, so unmarking one here would keep it
;; in the table for ever. The `data` half is a guard for the rows still
;; pending migration, not a dependency: drop it with the last one.
;; Only durable history comes back: a snapshot, the rows that carry `label`.
;; Everything else is deliberately left alone. Revisions are born already
;; scheduled for deletion one hour later, and the collector only looks at
;; rows that carry a date, so unmarking one here would keep it in the table
;; for ever. Rows that still carry a legacy payload without a label are
;; garbage that no read path reaches; 0157-migrate-legacy-file-data deletes
;; them, so they must keep their date and go with the collector.
"UPDATE file_change SET deleted_at = null
WHERE file_id = ANY(?::uuid[])
AND (label IS NOT NULL OR data IS NOT NULL)")
AND label IS NOT NULL")
(def ^:private sql:restore-file-data
"UPDATE file_data SET deleted_at = null

141
backend/src/app/srepl/procs/fdata_storage.clj
View File

@ -1,141 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.srepl.procs.fdata-storage
(:require
[app.common.logging :as l]
[app.db :as db]))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; SNAPSHOTS
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(def sql:get-unmigrated-snapshots
"SELECT fc.id, fc.file_id
FROM file_change AS fc
WHERE fc.data IS NOT NULL
AND fc.label IS NOT NULL
ORDER BY fc.id ASC")
(def sql:get-migrated-snapshots
"SELECT f.id, f.file_id
FROM file_data AS f
WHERE f.data IS NOT NULL
AND f.type = 'snapshot'
AND f.id != f.file_id
ORDER BY f.id ASC")
(defn migrate-snapshot-to-storage
"Migrate the current existing files to store data in new storage
tables."
{:query sql:get-unmigrated-snapshots}
[{:keys [::db/conn]} {:keys [id file-id]} & {:as options}]
(let [{:keys [id file-id data created-at updated-at]}
(db/get* conn :file-change {:id id :file-id file-id}
::db/for-update true
::db/remove-deleted false)]
(when data
(l/inf :hint "migrating snapshot" :file-id (str file-id) :id (str id))
(db/update! conn :file-change
{:data nil}
{:id id :file-id file-id}
{::db/return-keys false})
(db/insert! conn :file-data
{:backend "db"
:metadata nil
:type "snapshot"
:data data
:created-at created-at
:modified-at updated-at
:file-id file-id
:id id}
{::db/return-keys false}))))
(defn rollback-snapshot-from-storage
"Migrate back to the file table storage."
{:query sql:get-unmigrated-snapshots}
[{:keys [::db/conn]} {:keys [id file-id]} & {:as opts}]
(when-let [{:keys [id file-id data]}
(db/get* conn :file-data {:id id :file-id file-id :type "snapshot"}
::db/for-update true
::db/remove-deleted false)]
(l/inf :hint "rollback snapshot" :file-id (str file-id) :id (str id))
(db/update! conn :file-change
{:data data}
{:id id :file-id file-id}
{::db/return-keys false})
(db/delete! conn :file-data
{:id id :file-id file-id :type "snapshot"}
{::db/return-keys false})))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; FILES
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(def sql:get-unmigrated-files
"SELECT f.id
FROM file AS f
WHERE f.data IS NOT NULL
ORDER BY f.modified_at ASC")
(def sql:get-migrated-files
"SELECT f.id, f.file_id
FROM file_data AS f
WHERE f.data IS NOT NULL
AND f.id = f.file_id
ORDER BY f.id ASC")
(defn migrate-file-to-storage
"Migrate the current existing files to store data in new storage
tables."
{:query sql:get-unmigrated-files}
[{:keys [::db/conn] :as cfg} {:keys [id]} & {:as opts}]
(let [{:keys [id data created-at modified-at]}
(db/get* conn :file {:id id}
::db/for-update true
::db/remove-deleted false)]
(when data
(l/inf :hint "migrating file" :file-id (str id))
(db/update! conn :file {:data nil} {:id id} ::db/return-keys false)
(db/insert! conn :file-data
{:backend "db"
:metadata nil
:type "main"
:data data
:created-at created-at
:modified-at modified-at
:file-id id
:id id}
{::db/return-keys false}))
(let [snapshots-sql
(str "WITH snapshots AS (" sql:get-unmigrated-snapshots ") "
"SELECT s.* FROM snapshots AS s WHERE s.file_id = ?")]
(run! (fn [params]
(migrate-snapshot-to-storage cfg params opts))
(db/plan cfg [snapshots-sql id])))))
(defn rollback-file-from-storage
"Migrate back to the file table storage."
{:query sql:get-migrated-files}
[{:keys [::db/conn] :as cfg} {:keys [id]} & {:as opts}]
(when-let [{:keys [id data]}
(db/get* conn :file-data {:id id :file-id id :type "main"}
::db/for-update true
::db/remove-deleted false)]
(l/inf :hint "rollback file" :file-id (str id))
(db/update! conn :file {:data data} {:id id} ::db/return-keys false)
(db/delete! conn :file-data {:file-id id :id id :type "main"} ::db/return-keys false)
(let [snapshots-sql
(str "WITH snapshots AS (" sql:get-migrated-snapshots ") "
"SELECT s.* FROM snapshots AS s WHERE s.file_id = ?")]
(run! (fn [params]
(rollback-snapshot-from-storage cfg params opts))
(db/plan cfg [snapshots-sql id])))))

4
backend/test/backend_tests/file_change_retention_test.clj
View File

@ -118,8 +118,8 @@
(t/testing "the snapshot comes back"
(t/is (nil? (:deleted-at (find-change snapshot-id)))))
(t/testing "the legacy payload row comes back"
(t/is (nil? (:deleted-at (find-change legacy-id))))))))
(t/testing "the legacy payload row keeps its date"
(t/is (= trash-at (:deleted-at (find-change legacy-id))))))))
(t/deftest deleting-a-file-does-not-push-the-revision-schedule-out
(binding [ct/*clock* (ct/fixed-clock test-now)]

180
backend/test/backend_tests/migrations_legacy_file_data_test.clj Normal file
View File

@ -0,0 +1,180 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns backend-tests.migrations-legacy-file-data-test
"Covers the 0157 data migration: legacy file/file_change payloads move
to file_data, keeping timestamps and deleted_at, nulling the legacy
column; unlabelled file_change payloads (orphan revisions) are
deleted."
(:require
[app.common.time :as ct]
[app.common.uuid :as uuid]
[app.db :as db]
[app.util.blob :as blob]
[backend-tests.helpers :as th]
[clojure.java.io :as io]
[clojure.test :as t]
[next.jdbc :as jdbc]))
(t/use-fixtures :once th/state-init)
(t/use-fixtures :each th/database-reset)
(defn- run-migration
[]
(let [sql (slurp (io/resource "app/migrations/sql/0157-migrate-legacy-file-data.sql"))]
(jdbc/execute! th/*pool* [sql])))
(defn- get-file-data
[file-id id]
(th/db-get :file-data {:file-id file-id :id id} {::db/remove-deleted false}))
(t/deftest migrates-legacy-file-and-snapshot
(let [profile (th/create-profile* 1)
file (th/create-file* 1 {:profile-id (:id profile)
:project-id (:default-project-id profile)})
fid (:id file)
;; Simulate an unmigrated main: move the payload back to
;; file.data and drop the file_data row.
main-data (:data (get-file-data fid fid))
_ (t/is (some? main-data) "expected file_data main row from create-file*")
file-row (th/db-get :file {:id fid} {::db/remove-deleted false})
_ (th/db-delete! :file-data {:file-id fid :id fid})
_ (th/db-update! :file {:data main-data} {:id fid})
;; Simulate an unmigrated snapshot (labelled) with its own
;; timestamps and a deletion date.
snap-id (uuid/next)
snap-data (blob/encode {:snapshot 1})
created-at (ct/in-past "2h")
updated-at (ct/in-past "1h")
deleted-at (ct/in-past "30m")
_ (th/db-insert! :file-change
{:id snap-id
:file-id fid
:revn 1
:data snap-data
:label "v1"
:created-at created-at
:updated-at updated-at
:deleted-at deleted-at})
;; Orphan revision (data, no label): deleted by the migration.
rev-id (uuid/next)
rev-data (blob/encode {:rev 1})
_ (th/db-insert! :file-change
{:id rev-id
:file-id fid
:revn 2
:data rev-data})
;; Live revision (changes, no data, no label): untouched.
live-id (uuid/next)
_ (th/db-insert! :file-change
{:id live-id
:file-id fid
:revn 3
:changes (blob/encode [{:type :dummy}])})]
(run-migration)
(t/testing "main payload moves to file_data as type main"
(let [fd (get-file-data fid fid)]
(t/is (some? fd))
(t/is (= "main" (:type fd)))
(t/is (= "db" (:backend fd)))
(t/is (nil? (:metadata fd)))
(t/is (= (seq ^bytes main-data) (seq ^bytes (:data fd))))
(t/is (= (:created-at file-row) (:created-at fd)))
(t/is (= (:modified-at file-row) (:modified-at fd)))
(t/is (= (:deleted-at file-row) (:deleted-at fd)))))
(t/testing "snapshot payload moves to file_data as type snapshot"
(let [fd (get-file-data fid snap-id)]
(t/is (some? fd))
(t/is (= "snapshot" (:type fd)))
(t/is (= "db" (:backend fd)))
(t/is (nil? (:metadata fd)))
(t/is (= (seq ^bytes snap-data) (seq ^bytes (:data fd))))
;; NOTE: timestamptz stores microsecond precision while the
;; test instants carry nanoseconds, so compare at millis.
(t/is (= (inst-ms created-at) (inst-ms (:created-at fd))))
(t/is (= (inst-ms updated-at) (inst-ms (:modified-at fd))))
(t/is (= (inst-ms deleted-at) (inst-ms (:deleted-at fd))))))
(t/testing "legacy columns are nulled"
(t/is (nil? (:data (th/db-get :file {:id fid} {::db/remove-deleted false}))))
(t/is (nil? (:data (th/db-get :file-change {:id snap-id} {::db/remove-deleted false})))))
(t/testing "unlabelled payload row is deleted, live revision stays"
(t/is (nil? (th/db-get :file-change {:id rev-id} {::db/remove-deleted false})))
(let [row (th/db-get :file-change {:id live-id} {::db/remove-deleted false})]
(t/is (some? row))
(t/is (some? (:changes row)))))
(t/testing "second run is a no-op"
(run-migration)
(t/is (some? (get-file-data fid fid)))
(t/is (some? (get-file-data fid snap-id)))
(t/is (nil? (:data (th/db-get :file {:id fid} {::db/remove-deleted false}))))
(t/is (nil? (:data (th/db-get :file-change {:id snap-id} {::db/remove-deleted false}))))
(t/is (nil? (th/db-get :file-change {:id rev-id} {::db/remove-deleted false}))))))
(t/deftest keeps-preexisting-file-data-row
(let [profile (th/create-profile* 1)
file (th/create-file* 1 {:profile-id (:id profile)
:project-id (:default-project-id profile)})
fid (:id file)
;; file_data already holds the current payload; the legacy
;; column keeps a stale copy that must not overwrite it.
current (:data (get-file-data fid fid))
stale (blob/encode {:stale true})
_ (t/is (some? current))
_ (t/is (not= (seq ^bytes current) (seq ^bytes stale)))
_ (th/db-update! :file {:data stale} {:id fid})]
(run-migration)
(t/testing "file_data keeps its data and legacy is nulled"
(let [fd (get-file-data fid fid)]
(t/is (= (seq ^bytes current) (seq ^bytes (:data fd)))))
(t/is (nil? (:data (th/db-get :file {:id fid} {::db/remove-deleted false})))))))
(t/deftest keeps-preexisting-snapshot-row
(let [profile (th/create-profile* 1)
file (th/create-file* 1 {:profile-id (:id profile)
:project-id (:default-project-id profile)})
fid (:id file)
;; file_data already holds the snapshot payload; the legacy column
;; keeps a stale copy that must not overwrite it.
snap-id (uuid/next)
current (blob/encode {:snapshot :current})
stale (blob/encode {:snapshot :stale})
_ (t/is (not= (seq ^bytes current) (seq ^bytes stale)))
_ (th/db-insert! :file-data
{:file-id fid
:id snap-id
:type "snapshot"
:backend "db"
:metadata nil
:data current})
_ (th/db-insert! :file-change
{:id snap-id
:file-id fid
:revn 1
:label "v1"
:data stale})]
(run-migration)
(t/testing "file_data keeps its data and legacy is nulled"
(let [fd (get-file-data fid snap-id)]
(t/is (= (seq ^bytes current) (seq ^bytes (:data fd)))))
(t/is (nil? (:data (th/db-get :file-change {:id snap-id}
{::db/remove-deleted false})))))))

323
operations/2.19.0-migrate-legacy-file-data.md Normal file
View File

@ -0,0 +1,323 @@
# Migrate legacy file payloads to `file_data`
Online procedure to move the legacy file payloads (`file.data` and labelled
`file_change.data`) to `file_data` and to delete the orphan `file_change`
rows that carry `data` without a `label`.
- Migration: `backend/src/app/migrations/sql/0157-migrate-legacy-file-data.sql`
- Issue: <https://github.com/penpot/penpot/issues/12784>
The result has to be **exactly** the end state of the migration, because on a
big instance the migration is marked as applied and never runs. This procedure
runs the same statements by hand, in small batched transactions.
Use this only if the automatic migration would take too long on the instance
(millions of snapshots, hundreds of GB in TOAST). Small and medium instances
run the migration automatically on boot and need nothing from here.
Run it before upgrading to 2.19.0.
## What changes
- Every `file_change` row with `data IS NOT NULL AND label IS NOT NULL`
(a snapshot) gets a `file_data` row with `type='snapshot'`,
`backend='db'`, `metadata=NULL`, `created_at` from `created_at`,
`modified_at` from `updated_at`, and `deleted_at` copied as is. Then the
legacy `file_change.data` is set to `NULL`.
- Every `file` row with `data IS NOT NULL` gets a `file_data` row with
`type='main'` (and `file_id = id`), the same `backend`/`metadata` mapping
and the timestamps copied one to one. Then the legacy `file.data` is set
to `NULL`.
- Every `file_change` row with `data IS NOT NULL AND label IS NULL` (an
orphan revision) is deleted outright. No read path reaches these rows:
snapshots resolve through `label IS NOT NULL`, lagged changes read the
`changes` column (not `data`), and nothing references `file_change` rows
from other tables. Rows carrying `deleted_at` would be collected by
`objects-gc` anyway; undated ones would sit in the table and its TOAST
for ever.
`deleted_at` is copied on purpose: the write path always writes it to
`file_data`, the GC only collects `file_data` rows carrying it, and reads
never filter on `file_data.deleted_at`. Without the copy, migrated rows of
deleted files and snapshots would be orphaned: nobody reads them and nobody
collects them.
## Why this mechanism
A single-statement copy inside the transaction of the startup does not fit a
big instance, so the copy runs here in small batched transactions:
- One statement is one transaction. Never wrap several batches in one
transaction. Run a single stream of batches at a time.
- `FOR UPDATE SKIP LOCKED`: the batches never block the application and are
never blocked by it. A row skipped by a lock stays in the index and the
next execution picks it up.
- Partial temp indexes hold only the pending rows, so every execution finds
its next batch without rescanning what is done, and migrated rows
(`data = NULL`) and deleted orphans drop out of the index on their own.
No cursor is needed. Do not `VACUUM` mid-run to clean them: it would walk
the whole TOAST. Drop them at the end.
- No `ON CONFLICT DO NOTHING` here on purpose: a duplicate key must abort
the batch (atomically: it migrates nothing) so the conflict is
investigated instead of silently skipped. The automatic migration is
lenient so a small instance never fails to boot; here strictness is what
keeps the data auditable.
---
## Phase 0 — Before anything
### 0.1 Mark the migration as applied
Do this first: it registers the automatic migration as done, so a backend
restart mid-procedure never runs it on boot (it would copy the whole legacy
payload inside the startup transaction).
```sql
INSERT INTO migrations (module, step)
VALUES ('main', '0157-migrate-legacy-file-data');
```
The module is `main` and the step has no `.sql` (see
`backend/src/app/util/migrations.clj`).
### 0.2 Counters
Repeat the counters any time to follow the progress. Each batch loop ends
when it returns 0; the check in phase 3 confirms nothing is pending.
```sql
-- Remaining legacy rows.
SELECT (SELECT count(*) FROM file_change
WHERE data IS NOT NULL AND label IS NOT NULL) AS snapshots_remaining,
(SELECT count(*) FROM file_change
WHERE data IS NOT NULL AND label IS NULL) AS orphans_remaining,
(SELECT count(*) FROM file
WHERE data IS NOT NULL) AS files_remaining;
```
Optional, reference only: how much is already migrated. Skip it unless
curious: it is a full count over the partitioned table and can take a
while on a big instance. For progress, the remaining counters above and
the per-batch counts are enough.
```sql
SELECT (SELECT count(*) FROM file_data
WHERE type = 'snapshot' AND backend = 'db') AS snapshots_in_file_data,
(SELECT count(*) FROM file_data
WHERE type = 'main' AND backend = 'db') AS mains_in_file_data;
```
## Phase 1 — Temp indexes
Required by the batch loops, created before them. `CONCURRENTLY` so writes
are not blocked; each statement is its own transaction.
```sql
CREATE INDEX CONCURRENTLY tmp_file_change_mig ON file_change (id)
WHERE data IS NOT NULL AND label IS NOT NULL;
CREATE INDEX CONCURRENTLY tmp_file_change_orphan_mig ON file_change (id)
WHERE data IS NOT NULL AND label IS NULL;
CREATE INDEX CONCURRENTLY tmp_file_mig ON file (id)
WHERE data IS NOT NULL;
```
A failed `CONCURRENTLY` build stays `INVALID` and the planner ignores it
silently, so verify all three indexes exist and are valid:
```sql
SELECT indexrelid::regclass AS index, indisvalid AS valid
FROM pg_index
WHERE indexrelid::regclass::text LIKE 'tmp_%mig';
```
## Phase 2 — Batch loops
Repeat each statement until it returns 0. Lower `LIMIT` if a batch hits
`statement_timeout` or memory pressure (`SET statement_timeout = '15min'`
per batch is the usual guard).
### 2.1 Delete orphans
Orphans first (or interleaved with the other two; the row sets are
disjoint, and deleting TOAST-heavy rows early frees space for the rest):
```sql
WITH doomed AS (
SELECT id
FROM file_change
WHERE data IS NOT NULL
AND label IS NULL
ORDER BY id
LIMIT 10000
FOR UPDATE SKIP LOCKED
),
del AS (
DELETE FROM file_change fc
USING doomed
WHERE fc.id = doomed.id
RETURNING fc.id
)
SELECT count(*) AS deleted
FROM del;
```
### 2.2 Migrate snapshots
Repeat until it returns 0 (`~120 MB` per transaction at `LIMIT 10000`):
```sql
WITH batch AS (
SELECT id, file_id, data, created_at, updated_at, deleted_at
FROM file_change
WHERE data IS NOT NULL
AND label IS NOT NULL
ORDER BY id
LIMIT 10000
FOR UPDATE SKIP LOCKED
),
-- NO ON CONFLICT on purpose: duplicates must abort (see above).
-- upd reads from ins (RETURNING), not from batch.
ins AS (
INSERT INTO file_data (file_id, id, created_at, modified_at, deleted_at,
type, backend, metadata, data)
SELECT file_id, id, created_at, updated_at, deleted_at,
'snapshot', 'db', NULL, data
FROM batch
RETURNING file_id, id
),
-- Only the origin of what was really inserted is nulled. On conflict
-- the whole transaction aborts, so ins == batch whenever this runs.
upd AS (
UPDATE file_change fc
SET data = NULL
FROM ins
WHERE ins.file_id = fc.file_id
AND ins.id = fc.id
RETURNING fc.id
)
SELECT count(*) AS migrated
FROM upd;
```
### 2.3 Migrate files
Repeat until it returns 0 (tune `LIMIT` to the average `file.data` size;
start at 5000):
```sql
WITH batch AS (
SELECT id, data, created_at, modified_at, deleted_at
FROM file
WHERE data IS NOT NULL
ORDER BY id
LIMIT 5000
FOR UPDATE SKIP LOCKED
),
ins AS (
INSERT INTO file_data (file_id, id, created_at, modified_at, deleted_at,
type, backend, metadata, data)
SELECT id, id, created_at, modified_at, deleted_at,
'main', 'db', NULL, data
FROM batch
RETURNING file_id, id
),
upd AS (
UPDATE file f
SET data = NULL
FROM ins
WHERE ins.file_id = f.id
AND ins.id = f.id
RETURNING f.id
)
SELECT count(*) AS migrated
FROM upd;
```
### 2.4 Duplicate keys
If a batch aborts on duplicate key, stop and do not blind-retry it: it
means `file_data` already holds that key. Locate the overlap:
```sql
-- Snapshots with legacy data AND a file_data key already present.
SELECT c.file_id, c.id, c.created_by, c.deleted_at,
pg_size_pretty(length(c.data)) AS legacy_size,
pg_size_pretty(length(fd.data)) AS file_data_size,
fd.modified_at AS file_data_modified_at
FROM file_change c
JOIN file_data fd ON fd.file_id = c.file_id
AND fd.id = c.id
AND fd.type = 'snapshot'
WHERE c.data IS NOT NULL
ORDER BY c.id
LIMIT 20;
-- Files with legacy data AND a file_data key already present.
SELECT f.id,
pg_size_pretty(length(f.data)) AS legacy_size,
pg_size_pretty(length(fd.data)) AS file_data_size,
fd.modified_at AS file_data_modified_at
FROM file f
JOIN file_data fd ON fd.file_id = f.id
AND fd.id = f.id
AND fd.type = 'main'
WHERE f.data IS NOT NULL
ORDER BY f.id
LIMIT 20;
```
Resolve manually, one row at a time, only when `file_data` holds equal or
newer data (compare `modified_at` and sizes above): the legacy copy is
redundant, null the origin, leave `file_data` untouched.
```sql
-- UPDATE file_change SET data = NULL WHERE file_id = '...' AND id = '...';
-- UPDATE file SET data = NULL WHERE id = '...';
```
## Phase 3 — Verification and cleanup
Each loop above already ended returning 0; confirm nothing is pending (if
anything shows up, repeat that loop):
```sql
-- Remaining legacy rows: all three must read 0.
SELECT (SELECT count(*) FROM file_change
WHERE data IS NOT NULL AND label IS NOT NULL) AS snapshots_remaining,
(SELECT count(*) FROM file_change
WHERE data IS NOT NULL AND label IS NULL) AS orphans_remaining,
(SELECT count(*) FROM file
WHERE data IS NOT NULL) AS files_remaining;
```
```sql
-- Spot check: migrated rows exist in file_data with backend 'db'.
SELECT fd.file_id, fd.id, fd.type, fd.backend,
pg_size_pretty(length(fd.data)) AS size_in_file_data
FROM file_data fd
ORDER BY fd.file_id, fd.id
LIMIT 5;
```
Drop the temp indexes first, so the vacuum below skips them (each batch
left its dead entries behind), then reclaim the dead tuples and refresh
planner stats. Plain `VACUUM` does not block writes; `VERBOSE` prints
progress, which matters on big tables:
```sql
DROP INDEX CONCURRENTLY tmp_file_change_mig;
DROP INDEX CONCURRENTLY tmp_file_change_orphan_mig;
DROP INDEX CONCURRENTLY tmp_file_mig;
VACUUM (ANALYZE ON, VERBOSE ON) file_change;
VACUUM (ANALYZE ON, VERBOSE ON) file;
-- file_data is partitioned into 16 tables: the statement walks all of
-- them, so it can take a while. Without it the planner stats of the fresh
-- inserts stay stale until autovacuum catches up.
VACUUM (ANALYZE ON, VERBOSE ON) file_data;
```
## Phase 4 — Upgrade
The migration was marked as applied in phase 0, so the new version skips
it on boot. Deploy normally.