📚 Add 2.19.0 operations runbook for legacy file_data migration

Manual pre-migration procedure for big instances, same statements as migration 0157 in batched transactions, following the convention of the file-change partition runbook. Closes #12784. AI-assisted-by: muse-spark-1.3-contributor
2026-11-05 01:20:38 +00:00 · 2026-11-03 12:32:21 +00:00 · 2026-11-03 12:32:21 +00:00 · 637843e63f
commit 637843e63f
parent 2fbf5b2a4d
1 changed files with 319 additions and 0 deletions
--- a/operations/2.19.0-migrate-legacy-file-data.md
+++ b/operations/2.19.0-migrate-legacy-file-data.md
@ -0,0 +1,319 @@
+# Migrate legacy file payloads to `file_data`
+
+Online procedure to move the legacy file payloads (`file.data` and labelled
+`file_change.data`) to `file_data` and to delete the orphan `file_change`
+rows that carry `data` without a `label`.
+
+- Migration: `backend/src/app/migrations/sql/0157-migrate-legacy-file-data.sql`
+- Issue: <https://github.com/penpot/penpot/issues/12784>
+
+The result has to be **exactly** the end state of the migration, because on a
+big instance the migration is marked as applied and never runs. This procedure
+runs the same statements by hand, in small batched transactions.
+
+Use this only if the automatic migration would take too long on the instance
+(millions of snapshots, hundreds of GB in TOAST). Small and medium instances
+run the migration automatically on boot and need nothing from here.
+
+Run it before upgrading to 2.19.0.
+
+## What changes
+
+- Every `file_change` row with `data IS NOT NULL AND label IS NOT NULL`
+  (a snapshot) gets a `file_data` row with `type='snapshot'`,
+  `backend='db'`, `metadata=NULL`, `created_at` from `created_at`,
+  `modified_at` from `updated_at`, and `deleted_at` copied as is. Then the
+  legacy `file_change.data` is set to `NULL`.
+- Every `file` row with `data IS NOT NULL` gets a `file_data` row with
+  `type='main'` (and `file_id = id`), the same `backend`/`metadata` mapping
+  and the timestamps copied one to one. Then the legacy `file.data` is set
+  to `NULL`.
+- Every `file_change` row with `data IS NOT NULL AND label IS NULL` (an
+  orphan revision) is deleted outright. No read path reaches these rows:
+  snapshots resolve through `label IS NOT NULL`, lagged changes read the
+  `changes` column (not `data`), and nothing references `file_change` rows
+  from other tables. Rows carrying `deleted_at` would be collected by
+  `objects-gc` anyway; undated ones would sit in the table and its TOAST
+  for ever.
+
+`deleted_at` is copied on purpose: the write path always writes it to
+`file_data`, the GC only collects `file_data` rows carrying it, and reads
+never filter on `file_data.deleted_at`. Without the copy, migrated rows of
+deleted files and snapshots would be orphaned: nobody reads them and nobody
+collects them.
+
+## Why this mechanism
+
+A single-statement copy inside the transaction of the startup does not fit a
+big instance, so the copy runs here in small batched transactions:
+
+- One statement is one transaction. Never wrap several batches in one
+  transaction. Run a single stream of batches at a time.
+- `FOR UPDATE SKIP LOCKED`: the batches never block the application and are
+  never blocked by it. A row skipped by a lock stays in the index and the
+  next execution picks it up.
+- Partial temp indexes hold only the pending rows, so every execution finds
+  its next batch without rescanning what is done, and migrated rows
+  (`data = NULL`) and deleted orphans drop out of the index on their own.
+  No cursor is needed. Do not `VACUUM` mid-run to clean them: it would walk
+  the whole TOAST. Drop them at the end.
+- No `ON CONFLICT DO NOTHING` here on purpose: a duplicate key must abort
+  the batch (atomically: it migrates nothing) so the conflict is
+  investigated instead of silently skipped. The automatic migration is
+  lenient so a small instance never fails to boot; here strictness is what
+  keeps the data auditable.
+
+---
+
+## Phase 0 — Before anything
+
+### 0.1 Mark the migration as applied
+
+Do this first: it registers the automatic migration as done, so a backend
+restart mid-procedure never runs it on boot (it would copy the whole legacy
+payload inside the startup transaction).
+
+```sql
+INSERT INTO migrations (module, step)
+VALUES ('main', '0157-migrate-legacy-file-data');
+```
+
+The module is `main` and the step has no `.sql` (see
+`backend/src/app/util/migrations.clj`).
+
+### 0.2 Counters
+
+Repeat the counters any time to follow the progress. Each batch loop ends
+when it returns 0; the check in phase 3 confirms nothing is pending.
+
+```sql
+-- Remaining legacy rows.
+SELECT (SELECT count(*) FROM file_change
+         WHERE data IS NOT NULL AND label IS NOT NULL) AS snapshots_remaining,
+       (SELECT count(*) FROM file_change
+         WHERE data IS NOT NULL AND label IS NULL)     AS orphans_remaining,
+       (SELECT count(*) FROM file
+         WHERE data IS NOT NULL)                       AS files_remaining;
+```
+
+Optional, reference only: how much is already migrated. Skip it unless
+curious: it is a full count over the partitioned table and can take a
+while on a big instance. For progress, the remaining counters above and
+the per-batch counts are enough.
+
+```sql
+SELECT (SELECT count(*) FROM file_data
+         WHERE type = 'snapshot' AND backend = 'db') AS snapshots_in_file_data,
+       (SELECT count(*) FROM file_data
+         WHERE type = 'main' AND backend = 'db')     AS mains_in_file_data;
+```
+
+## Phase 1 — Temp indexes
+
+Required by the batch loops, created before them. `CONCURRENTLY` so writes
+are not blocked; each statement is its own transaction.
+
+```sql
+CREATE INDEX CONCURRENTLY tmp_file_change_mig ON file_change (id)
+ WHERE data IS NOT NULL AND label IS NOT NULL;
+CREATE INDEX CONCURRENTLY tmp_file_change_orphan_mig ON file_change (id)
+ WHERE data IS NOT NULL AND label IS NULL;
+CREATE INDEX CONCURRENTLY tmp_file_mig ON file (id)
+ WHERE data IS NOT NULL;
+```
+
+A failed `CONCURRENTLY` build stays `INVALID` and the planner ignores it
+silently, so verify all three indexes exist and are valid:
+
+```sql
+SELECT indexrelid::regclass AS index, indisvalid AS valid
+  FROM pg_index
+ WHERE indexrelid::regclass::text LIKE 'tmp_%mig';
+```
+
+## Phase 2 — Batch loops
+
+Repeat each statement until it returns 0. Lower `LIMIT` if a batch hits
+`statement_timeout` or memory pressure (`SET statement_timeout = '15min'`
+per batch is the usual guard).
+
+### 2.1 Delete orphans
+
+Orphans first (or interleaved with the other two; the row sets are
+disjoint, and deleting TOAST-heavy rows early frees space for the rest):
+
+```sql
+WITH doomed AS (
+  SELECT id
+    FROM file_change
+   WHERE data IS NOT NULL
+     AND label IS NULL
+   ORDER BY id
+   LIMIT 10000
+   FOR UPDATE SKIP LOCKED
+),
+del AS (
+  DELETE FROM file_change fc
+   USING doomed
+   WHERE fc.id = doomed.id
+  RETURNING fc.id
+)
+SELECT count(*) AS deleted
+  FROM del;
+```
+
+### 2.2 Migrate snapshots
+
+Repeat until it returns 0 (`~120 MB` per transaction at `LIMIT 10000`):
+
+```sql
+WITH batch AS (
+  SELECT id, file_id, data, created_at, updated_at, deleted_at
+    FROM file_change
+   WHERE data IS NOT NULL
+     AND label IS NOT NULL
+   ORDER BY id
+   LIMIT 10000
+   FOR UPDATE SKIP LOCKED
+),
+-- NO ON CONFLICT on purpose: duplicates must abort (see above).
+-- upd reads from ins (RETURNING), not from batch.
+ins AS (
+  INSERT INTO file_data (file_id, id, created_at, modified_at, deleted_at,
+                         type, backend, metadata, data)
+  SELECT file_id, id, created_at, updated_at, deleted_at,
+         'snapshot', 'db', NULL, data
+    FROM batch
+  RETURNING file_id, id
+),
+-- Only the origin of what was really inserted is nulled. On conflict
+-- the whole transaction aborts, so ins == batch whenever this runs.
+upd AS (
+  UPDATE file_change fc
+     SET data = NULL
+    FROM ins
+   WHERE ins.file_id = fc.file_id
+     AND ins.id = fc.id
+  RETURNING fc.id
+)
+SELECT count(*) AS migrated
+  FROM upd;
+```
+
+### 2.3 Migrate files
+
+Repeat until it returns 0 (tune `LIMIT` to the average `file.data` size;
+start at 5000):
+
+```sql
+WITH batch AS (
+  SELECT id, data, created_at, modified_at, deleted_at
+    FROM file
+   WHERE data IS NOT NULL
+   ORDER BY id
+   LIMIT 5000
+   FOR UPDATE SKIP LOCKED
+),
+ins AS (
+  INSERT INTO file_data (file_id, id, created_at, modified_at, deleted_at,
+                         type, backend, metadata, data)
+  SELECT id, id, created_at, modified_at, deleted_at,
+         'main', 'db', NULL, data
+    FROM batch
+  RETURNING file_id, id
+),
+upd AS (
+  UPDATE file f
+     SET data = NULL
+    FROM ins
+   WHERE ins.file_id = f.id
+     AND ins.id = f.id
+  RETURNING f.id
+)
+SELECT count(*) AS migrated
+  FROM upd;
+```
+
+### 2.4 Duplicate keys
+
+If a batch aborts on duplicate key, stop and do not blind-retry it: it
+means `file_data` already holds that key. Locate the overlap:
+
+```sql
+-- Snapshots with legacy data AND a file_data key already present.
+SELECT c.file_id, c.id, c.created_by, c.deleted_at,
+       pg_size_pretty(length(c.data)) AS legacy_size,
+       pg_size_pretty(length(fd.data)) AS file_data_size,
+       fd.modified_at AS file_data_modified_at
+  FROM file_change c
+  JOIN file_data fd ON fd.file_id = c.file_id
+                   AND fd.id = c.id
+                   AND fd.type = 'snapshot'
+ WHERE c.data IS NOT NULL
+ ORDER BY c.id
+ LIMIT 20;
+
+-- Files with legacy data AND a file_data key already present.
+SELECT f.id,
+       pg_size_pretty(length(f.data)) AS legacy_size,
+       pg_size_pretty(length(fd.data)) AS file_data_size,
+       fd.modified_at AS file_data_modified_at
+  FROM file f
+  JOIN file_data fd ON fd.file_id = f.id
+                    AND fd.id = f.id
+                    AND fd.type = 'main'
+ WHERE f.data IS NOT NULL
+ ORDER BY f.id
+ LIMIT 20;
+```
+
+Resolve manually, one row at a time, only when `file_data` holds equal or
+newer data (compare `modified_at` and sizes above): the legacy copy is
+redundant, null the origin, leave `file_data` untouched.
+
+```sql
+-- UPDATE file_change SET data = NULL WHERE file_id = '...' AND id = '...';
+-- UPDATE file SET data = NULL WHERE id = '...';
+```
+
+## Phase 3 — Verification and cleanup
+
+Each loop above already ended returning 0; confirm nothing is pending (if
+anything shows up, repeat that loop):
+
+```sql
+-- Remaining legacy rows: all three must read 0.
+SELECT (SELECT count(*) FROM file_change
+         WHERE data IS NOT NULL AND label IS NOT NULL) AS snapshots_remaining,
+       (SELECT count(*) FROM file_change
+         WHERE data IS NOT NULL AND label IS NULL)     AS orphans_remaining,
+       (SELECT count(*) FROM file
+         WHERE data IS NOT NULL)                       AS files_remaining;
+```
+
+```sql
+-- Spot check: migrated rows exist in file_data with backend 'db'.
+SELECT fd.file_id, fd.id, fd.type, fd.backend,
+       pg_size_pretty(length(fd.data)) AS size_in_file_data
+  FROM file_data fd
+ ORDER BY fd.file_id, fd.id
+ LIMIT 5;
+```
+
+Drop the temp indexes first, so the vacuum below skips them (each batch
+left its dead entries behind), then reclaim the dead tuples and refresh
+planner stats. Plain `VACUUM` does not block writes; `VERBOSE` prints
+progress, which matters on big tables:
+
+```sql
+DROP INDEX CONCURRENTLY tmp_file_change_mig;
+DROP INDEX CONCURRENTLY tmp_file_change_orphan_mig;
+DROP INDEX CONCURRENTLY tmp_file_mig;
+VACUUM (ANALYZE ON, VERBOSE ON) file_change;
+VACUUM (ANALYZE ON, VERBOSE ON) file;
+```
+
+## Phase 4 — Upgrade
+
+The migration was marked as applied in phase 0, so the new version skips
+it on boot. Deploy normally.