penpot/operations/2.20.0-file-change-partition.md
Andrey Antukh 679e70e48c 🐛 Keep file_change kind nullable so rollback keeps writing
Migration 0158 created file_change with kind NOT NULL, which blocks
any emergency rollback to the previous version: the old code inserts
without kind and every save would fail. kind is still a reserve
(reads use label IS NOT NULL), so it stays nullable until the
rollback window closes; the NOT NULL comes in a later migration.
The runbook keeps parity, the schema and migration tests cover the
nullable column and the rollback insert, and the storage memory
records the decision.
AI-assisted-by: muse-spark-1.3-contributor
2026-11-13 08:05:15 +00:00

41 KiB

Partition file_change by file_id

Procedure to turn file_change into a table partitioned by HASH (file_id) with 16 partitions, aligned with the ones of file_data. It runs online except the cutover (phase 4), which needs a scheduled maintenance window with the backend and the workers stopped.

The result has to be exactly the schema of the migration, because on a big instance the migration is marked as applied and never runs. This procedure runs the same statements by hand.

When to run it, and the stop

Run it only on the instances where the automatic migration refuses to run: file_change above 10 GiB of total size or above 2 million rows. Below that the migration does the same work on boot and this procedure has nothing to do.

The work spans the upgrade to 2.20.0. The order is not a suggestion: the preparation goes before the version update, and phases 2 and 3 need the new code running (it is the one that moves updated_at on every write, the watermark of the delta passes).

Phase When Service
0 and 1 Before the upgrade, with 2.19.0 serving Online
— The upgrade to 2.20.0 Usual deploy (step 0.3), before phase 2
2 After the upgrade, with 2.20.0 serving Online
3 From phase 2 to the window, every 1-6 h Online
4 A scheduled maintenance window Backend and workers stopped
5 After the cutover Online

Phase 2 is one pass over every row: the long job, hours or days. Phase 3 repeats a much shorter copy, only the rows that changed since the previous pass (seconds each), until the day of the window, so the last delta inside the lock stays short.

Steps 0.1 and 0.2 cannot wait for the upgrade: a 2.20.0 backend that boots without them runs the automatic migration inside the startup transaction and, above the guard, it fails on every boot.

The procedure needs one stop, and only one: phase 4, the cutover. It is a full stop-start of the backend and the workers, not a rolling restart (4.2 explains why). The window covers the last delta, the renames and the start again: seconds to minutes of writes blocked on file_change, and the saves that fall inside fail and the user retries. Everything else runs with the instance serving traffic.

Sizing your instance

Every number in this procedure comes from your database: run the queries below, the procedure does not depend on any absolute size. It is plain SQL and runs on any supported PostgreSQL (>= 15).

-- total size of the database and of file_change (heap + TOAST + indexes)
SELECT pg_size_pretty(pg_database_size(current_database())) AS database;
SELECT pg_size_pretty(pg_total_relation_size('file_change')) AS file_change_total,
       pg_size_pretty(pg_table_size('file_change'))         AS heap_and_toast,
       pg_size_pretty(pg_indexes_size('file_change'))       AS indexes;
SELECT count(*) AS rows FROM file_change;

-- dead tuples waiting for vacuum: their space is not copied
SELECT n_dead_tup FROM pg_stat_user_tables WHERE relname = 'file_change';

pg_table_size (heap + TOAST) is the upper bound of what the copy reads and writes. Do not estimate it with pg_stats.avg_width: that measures what the heap stores for a column, which for a TOASTed value is the pointer, not the value. The estimate reads far below the real size (about ten times less on a small instance, orders of magnitude on a big one).

Peak disk during the bulk is about the old table (it does not grow) plus the new table (at most its pg_table_size, less because the dead rows are not copied) plus its six indexes (about the pg_indexes_size of the live rows) plus the WAL the bulk generates (proportional to what it writes: watch the free space of the pg_wal mount while the first batches run). The disk comes back when the legacy table is dropped, days later. Time the first batches and extrapolate: the bulk reads heap + TOAST once sequentially and writes about the same, so its duration is proportional to pg_table_size, not to the heap alone.

What changes

  • file_change is partitioned by HASH (file_id) with 16 partitions, the same bounds as file_data, so the join between them is resolved partition by partition.
  • The primary key goes from (id) to (file_id, id): PostgreSQL requires the partition key in the primary key, and the application already filters by file_id everywhere.
  • The table has 17 columns: data_ref_id and data_backend are gone (dead since the file storage refactor), data is kept (the legacy-db backend reads old payloads from it) and kind appears (1 = snapshot, 2 = revision, nullable until the rollback window closes).
  • The old table is kept as file_change_legacy, a frozen copy of the previous layout, and dropped days later.
  • Almost all of the size of the table is dead space (see sizing), so the new table holds at most its heap + TOAST, and less as the dead rows are not copied. The disk comes back when the legacy table is dropped: the database shrinks by the size of the old table.

Why this mechanism

ATTACH as a DEFAULT partition is not possible: the 16 hash siblings cover the whole hash space, so the constraint of a DEFAULT partition could never hold, and the ATTACH fails when it validates every row. A UNION view with triggers, dual writes in the application and logical replication were discarded too.

What is left is a copy in phases, driven by the updated_at watermark, and a cutover of minutes: plain SQL, resumable, no new infrastructure. The heap is small next to the TOAST, so a delta pass without an index scans the heap and only detoastes the rows that changed.


Phase 0 — Before anything

0.0 The role of the session

Every session of this procedure starts with SET ROLE to the role that owns file_change (normally penpot):

SET ROLE penpot;  -- replace with the owner of file_change (see 0.6)

The new table, its 16 partitions, its indexes and its TOAST are created with the role of the session, and ALTER TABLE ... OWNER TO does not recurse to the partitions. Without this, a procedure run as a superuser leaves the table owned by that superuser and the backend, which connects as penpot, gets permission denied after the cutover.

0.1 Mark the migration as applied

The runner applies at startup every migration that is not in the migrations table. Without this, it would copy the whole table inside the transaction of the startup.

INSERT INTO migrations (module, step)
VALUES ('main', '0158-partition-file-change');

The module is main and the step has no .sql (see backend/src/app/util/migrations.clj).

0.2 Add the kind column to the current table

Until the cutover, file_change is the current table, and the new code writes kind on every insert: without the column the backend does not start.

ALTER TABLE file_change ADD COLUMN IF NOT EXISTS kind smallint;

Nullable and without a default on purpose: it is a metadata-only operation (it does not rewrite a single row of the table), the old code keeps inserting rows, and the values of the rows that are already there are computed by the copy from their label.

0.3 Deploy the code

The updated_at watermark has to be alive before the bulk copy: the new code updates it on every mutation (locks, soft deletes), the old one does not. Rows changed between the bulk and a delta pass would be lost.

SELECT max(updated_at) FROM file_change;

0.4 Index for the delta passes

The delta passes filter by updated_at. Without an index on that column every pass scans the whole heap; with it, it is an index range that only touches the rows that changed.

-- CONCURRENTLY: it does not block the writes. It cannot run inside a
-- transaction: run it alone.
CREATE INDEX CONCURRENTLY IF NOT EXISTS file_change__updated_at__idx
  ON file_change (updated_at);
  • The build reads the heap once sequentially and writes one narrow entry per row (check its size with pg_relation_size after the build): it does not block the writes either way. Time the build and extrapolate the bulk from it (an index on a column without TOAST does not touch the TOAST).
  • If it is interrupted, the index is left INVALID and has to be dropped and built again:
SELECT i.indisvalid
  FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid
 WHERE c.relname = 'file_change__updated_at__idx';
  • The index goes with the old table: at the cutover it is renamed to file_change__updated_at__idx_legacy and disappears with it. The new table does not carry it (nothing queries by updated_at; the GC uses deleted_at).

0.5 Check the environment

-- Space: peak about the old table (it does not grow) + the new table
-- (at most its `pg_table_size`, see sizing) + the WAL the bulk generates
SELECT pg_size_pretty(pg_database_size(current_database()));

-- The WAL must not run out of room (note the free space of the mount point):
--   df -h <pg_wal dir> <data dir>

-- Nothing must reference file_change (0 rows expected):
SELECT conrelid::regclass FROM pg_constraint
 WHERE confrelid = 'file_change'::regclass;

-- There must be no triggers on file_change:
SELECT tgname FROM pg_trigger
 WHERE tgrelid = 'file_change'::regclass AND NOT tgisinternal;

-- Replication slots: the bulk generates WAL proportional to what it writes,
-- watch the retained WAL while the first batches run.
SELECT slot_name, active, pg_size_pretty(pg_wal_lsn_diff(
         pg_current_wal_lsn(), restart_lsn)) AS retained_wal
  FROM pg_replication_slots;

-- The legacy blobs of the snapshots were already moved to `file_data` by
-- migration 0157-migrate-legacy-file-data (on the big instances, by hand,
-- with its runbook 2.19.0-migrate-legacy-file-data.md), so there is nothing
-- to copy here: the `data` column travels with the row to the new table.
-- These two numbers are informational guards. The first one has to be 0
-- after 0157 ran; if it is not, run 0157 (or its runbook) before starting.
SELECT count(*) AS snapshots_without_data
  FROM file_change AS fc
 WHERE fc.data IS NOT NULL
   AND fc.label IS NOT NULL
   AND NOT EXISTS (SELECT 1 FROM file_data AS fd
                    WHERE fd.file_id = fc.file_id
                      AND fd.id = fc.id
                      AND fd.type = 'snapshot');

-- The revisions with legacy data are not migrated (they are the ephemeral
-- changes, out of scope). This number is informative: their data travels
-- with the row to the new table.
SELECT count(*) AS revisions_with_data
  FROM file_change
 WHERE data IS NOT NULL
   AND label IS NULL;
  • Check that the backup / PITR is working before starting.
  • Pick a low activity window for the cutover: a freeze of the writes to file_change of a couple of minutes (the final delta inside the lock copies only the rows changed since the last delta pass, so it is quick; the saves of files during the freeze fail and the user retries, and the rest of the database keeps working). The window has to cover the whole cutover, with the backend and the workers stopped and started again (see 4.2).

0.6 Owner and grants

-- the role that has to own every object at the end (see 0.0)
SELECT tableowner FROM pg_tables
 WHERE schemaname = current_schema() AND tablename = 'file_change';

-- grants added by the operator (monitoring, reporting). This procedure
-- creates a new table: they do not travel with it, so save them to re-apply
-- them after the cutover.
SELECT grantee, privilege_type
  FROM information_schema.role_table_grants
 WHERE table_schema = current_schema() AND table_name = 'file_change';

Phase 1 — Create the partitioned table

The new indexes carry a _new suffix because index names are unique per schema and the old table still holds the canonical ones until the cutover. The same goes for the primary key of the new table: its index is named after the constraint. The constraints that have no index (the CHECK and the foreign keys) are named canonically from the start, because a constraint name is unique per table, not per schema, so it can live with the one of the old table; the partitions inherit it when they are created. The partitions are created with their final name, because their indexes are named after the partition and not after the table.

CREATE TABLE file_change_new (
  file_id       uuid NOT NULL,
  id            uuid NOT NULL DEFAULT gen_random_uuid(),
  created_at    timestamptz NOT NULL DEFAULT now(),
  session_id    uuid,
  revn          bigint NOT NULL DEFAULT 0,
  data          bytea,
  changes       bytea,
  profile_id    uuid,
  features      text[],
  label         text,
  version       integer,
  created_by    text NOT NULL DEFAULT 'system',
  updated_at    timestamptz DEFAULT now(),
  deleted_at    timestamptz,
  migrations    text[],
  locked_by     uuid,
  -- Nullable on purpose, like the migration: the previous version inserts
  -- without `kind`, and a rollback to it has to keep writing. The `NOT NULL`
  -- comes in a later migration, once the rollback window is closed.
  kind          smallint,
  CONSTRAINT file_change_kind_check CHECK (kind IN (1, 2))
) PARTITION BY HASH (file_id);

-- the columns keep the storage of PostgreSQL by default (migration
-- 0156-reset-pg-storage-defaults dropped the explicit EXTERNAL overrides)

COMMENT ON COLUMN file_change_new.kind IS
  'The kind of the row: 1 is a snapshot (it has a label and its data lives in file_data) and 2 is a revision (it has no label and it holds the changes).';
COMMENT ON COLUMN file_change_new.locked_by IS
  'Profile ID of user who has locked this version. Only the creator can lock/unlock their own versions. Locked versions cannot be deleted by others.';

-- 16 partitions, the same bounds as file_data. The name of the table is
-- temporary (the old one holds it until the cutover) but the name of the
-- partitions is not: it is the final one, so their indexes are born with
-- the canonical names (renaming a partition does not rename its indexes)
-- and the result is exactly the schema of the migration.
CREATE TABLE file_change_00 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 0);
CREATE TABLE file_change_01 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 1);
CREATE TABLE file_change_02 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 2);
CREATE TABLE file_change_03 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 3);
CREATE TABLE file_change_04 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 4);
CREATE TABLE file_change_05 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 5);
CREATE TABLE file_change_06 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 6);
CREATE TABLE file_change_07 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 7);
CREATE TABLE file_change_08 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 8);
CREATE TABLE file_change_09 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 9);
CREATE TABLE file_change_10 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 10);
CREATE TABLE file_change_11 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 11);
CREATE TABLE file_change_12 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 12);
CREATE TABLE file_change_13 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 13);
CREATE TABLE file_change_14 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 14);
CREATE TABLE file_change_15 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 15);

-- The primary key includes the partition key (required on a partitioned table)
ALTER TABLE file_change_new ADD PRIMARY KEY (file_id, id);

-- Foreign keys (no CREATE TABLE copies them). They take the canonical name
-- already: they have no index, so they do not collide with the old table and
-- the partitions inherit it.
ALTER TABLE file_change_new
  ADD CONSTRAINT file_change_file_id_fkey FOREIGN KEY (file_id)
  REFERENCES file(id) DEFERRABLE;
ALTER TABLE file_change_new
  ADD CONSTRAINT file_change_locked_by_fkey FOREIGN KEY (locked_by)
  REFERENCES profile(id) ON DELETE SET NULL DEFERRABLE;
ALTER TABLE file_change_new
  ADD CONSTRAINT file_change_profile_id_fkey FOREIGN KEY (profile_id)
  REFERENCES profile(id) ON DELETE SET NULL DEFERRABLE;
SELECT count(*) AS partitions FROM pg_partition_tree('file_change_new') t
 WHERE t.isleaf;
-- 16

Phase 2 — Bulk copy

Copies every row (including the soft deleted ones: the GC manages them the same way in the new table). The kind is computed from the label: a row with a label is a snapshot, a row without one is a revision.

The pointer of the copy lives in the database, not in the terminal: a batch can be interrupted and resumed without losing anything.

CREATE TABLE IF NOT EXISTS file_change_copy_state (
  name    text PRIMARY KEY,
  last_id uuid,
  last_at timestamptz
);

-- for a procedure that already created the table without the mark
ALTER TABLE file_change_copy_state
  ADD COLUMN IF NOT EXISTS last_at timestamptz;

INSERT INTO file_change_copy_state (name, last_id)
VALUES ('bulk', '00000000-0000-0000-0000-000000000000')
ON CONFLICT (name) DO NOTHING;

-- The mark of the delta passes starts at the bulk: the first pass of phase 3
-- covers the rows changed since here, and `DO NOTHING` keeps a re-run from
-- resetting a mark that the passes already moved.
INSERT INTO file_change_copy_state (name, last_at) VALUES ('delta', now())
ON CONFLICT (name) DO NOTHING;

-- The mark is mandatory: a pass without it copies nothing, in silence. The
-- `DROP ... IF EXISTS` keeps this block re-runnable.
ALTER TABLE file_change_copy_state
  DROP CONSTRAINT IF EXISTS file_change_copy_state_delta_mark,
  ADD CONSTRAINT file_change_copy_state_delta_mark
  CHECK (name <> 'delta' OR last_at IS NOT NULL);

One batch (idempotent). It always returns one row: the number of rows it copied, 0 when there is nothing left to copy.

WITH batch AS MATERIALIZED (
  SELECT id FROM file_change
   WHERE id > (SELECT last_id FROM file_change_copy_state WHERE name = 'bulk')
   ORDER BY id
   LIMIT 10000
),
copied AS (
  INSERT INTO file_change_new (id, file_id, created_at, session_id, revn, data, changes,
                               profile_id, features, label, version, created_by,
                               updated_at, deleted_at, migrations, locked_by, kind)
  SELECT id, file_id, created_at, session_id, revn, data, changes,
         profile_id, features, label, version, created_by,
         updated_at, deleted_at, migrations, locked_by,
         CASE WHEN label IS NOT NULL THEN 1 ELSE 2 END
    FROM file_change
   WHERE id IN (SELECT id FROM batch)
  ON CONFLICT (file_id, id) DO UPDATE
     SET created_at = EXCLUDED.created_at,
         session_id = EXCLUDED.session_id,
         revn       = EXCLUDED.revn,
         data       = EXCLUDED.data,
         changes    = EXCLUDED.changes,
         profile_id = EXCLUDED.profile_id,
         features   = EXCLUDED.features,
         label      = EXCLUDED.label,
         version    = EXCLUDED.version,
         created_by = EXCLUDED.created_by,
         updated_at = EXCLUDED.updated_at,
         deleted_at = EXCLUDED.deleted_at,
         migrations = EXCLUDED.migrations,
         locked_by  = EXCLUDED.locked_by,
         kind       = EXCLUDED.kind
   WHERE file_change_new.updated_at IS DISTINCT FROM EXCLUDED.updated_at
  RETURNING 1
),
-- `moved` advances the pointer only when the batch copied something.
-- PostgreSQL executes a data-modifying CTE exactly once even when the final
-- query does not read its output.
moved AS (
  UPDATE file_change_copy_state
     SET last_id = (SELECT id FROM batch ORDER BY id DESC LIMIT 1)
   WHERE name = 'bulk'
     AND EXISTS (SELECT 1 FROM copied)
  RETURNING 1
)
SELECT count(*) AS copied FROM copied;

Repeat this statement until it returns 0. The 0 comes from the final SELECT, not from a missing row: a batch that finds nothing to copy still returns it, so it is the signal to stop.

  • The pointer is a uuid (random order): the pagination is stable but it does not group by physical position. The total read is the heap, in a scattered order.
  • Estimated duration: time the first batches and extrapolate. The bulk reads the heap plus the live TOAST once (see sizing) and writes about the live data. Batches are rows divided by batch size: count your rows in sizing and divide.
  • Pace the batches yourself (a pause between statements) if the streaming replica falls behind or the backend sees latency. Every batch detoastes its share of the TOAST: the cost is I/O, not CPU.
  • Watch: replica lag, pg_stat_activity (locks between the bulk and autovacuum), the space of pg_wal and of the data dir.
SELECT count(*) AS missing
  FROM file_change o
 WHERE NOT EXISTS (SELECT 1 FROM file_change_new n
                    WHERE n.file_id = o.file_id
                      AND n.id = o.id);

missing is zero when no row of the old table is absent from the new one; it is the check that matters, not the total count. The new table can have more rows than the old one: the GC may have hard deleted a row from the source after the bulk copied it, and it cleans that row up after the cutover. Rows written during the bulk are missing until the delta passes pick them up.


Phase 3 — Delta passes

Every pass copies the rows modified since the previous pass. The watermark lives in file_change_copy_state.last_at (the start of the last pass; the setup block of phase 2 seeds it with the moment of the bulk, so the first pass covers what changed since then), and the SQL subtracts one hour of margin, which absorbs the clock drift between the application (which writes updated_at) and the database (which writes now()) and the drift between batches. A row that a pass already copied but did not change again is not rewritten: the WHERE of the ON CONFLICT skips it.

BEGIN;

WITH mark AS (
  SELECT last_at - INTERVAL '1 hour' AS since
    FROM file_change_copy_state
   WHERE name = 'delta'
)
INSERT INTO file_change_new (id, file_id, created_at, session_id, revn, data, changes,
                             profile_id, features, label, version, created_by,
                             updated_at, deleted_at, migrations, locked_by, kind)
SELECT id, file_id, created_at, session_id, revn, data, changes,
       profile_id, features, label, version, created_by,
       updated_at, deleted_at, migrations, locked_by,
       CASE WHEN label IS NOT NULL THEN 1 ELSE 2 END
  FROM file_change, mark
 WHERE updated_at > mark.since
ON CONFLICT (file_id, id) DO UPDATE
   SET created_at = EXCLUDED.created_at,
       session_id = EXCLUDED.session_id,
       revn       = EXCLUDED.revn,
       data       = EXCLUDED.data,
       changes    = EXCLUDED.changes,
       profile_id = EXCLUDED.profile_id,
       features   = EXCLUDED.features,
       label      = EXCLUDED.label,
       version    = EXCLUDED.version,
       created_by = EXCLUDED.created_by,
       updated_at = EXCLUDED.updated_at,
       deleted_at = EXCLUDED.deleted_at,
       migrations = EXCLUDED.migrations,
       locked_by  = EXCLUDED.locked_by,
       kind       = EXCLUDED.kind
 WHERE file_change_new.updated_at IS DISTINCT FROM EXCLUDED.updated_at;

-- The mark is the start of this transaction (`now()`): the rows changed
-- after it are the window of the next pass. Both statements go in the same
-- transaction on purpose; a mark written later would leave a gap.
UPDATE file_change_copy_state SET last_at = now() WHERE name = 'delta';

COMMIT;
  • Repeat the transaction every 1-6 h (depending on the activity) until the pass copies very few rows. With the index of phase 0.4 the pass is an index range over the window, so it takes seconds on a quiet instance; without it, it scans the heap. To see how many rows are left in the window:
SELECT count(*) FROM file_change
 WHERE updated_at > (SELECT last_at FROM file_change_copy_state WHERE name = 'delta');
  • The objects GC can keep running. Its soft deletes are covered by the watermark, and its hard deletes leave a row with deleted_at in the new table that the GC itself cleans up later. The hard delete of a file is blocked while the old table (or the new one) has rows for it, because the foreign key on file_id is NO ACTION. The GC deletes the pending files in chunks of 100, and one blocked file aborts the transaction of the whole chunk: the chunk is logged as error on chunk deletion and skipped, its files wait in the trash, and the next daily run retries the same chunk and fails again until the cutover. The other chunks keep making progress. Do not run the cutover at the same time as a big batch of the GC.

Phase 4 — Cutover (a freeze of minutes)

4.1 Before the lock

  • One last delta pass.
  • Build the secondary indexes of the new table. The bulk and the delta passes only need the primary key (ON CONFLICT (file_id, id)), so building them here keeps five indexes and their WAL out of every batch. The deleted_at index carries file_id (the primary key is (file_id, id)): the objects GC needs the file_id to lock by pair, and with the index it reads the rows it has to delete without touching the table.
-- Secondary indexes: the canonical names, with the _new suffix. There is no
-- index for the system snapshots: the old one filtered them by
-- `data IS NOT NULL`, and with the payload living in `file_data` the column
-- no longer tells snapshots apart. The new one could only filter
-- them by `deleted_at IS NULL`, and the system snapshots always carry a
-- deletion date in the future, so it would never match a row.
CREATE INDEX file_change_new__deleted_at__idx
  ON file_change_new (deleted_at, file_id, id) WHERE deleted_at IS NOT NULL;
CREATE INDEX file_change_new__file_id__revn__idx
  ON file_change_new (file_id, revn);
CREATE INDEX file_change_new__locked_by__idx
  ON file_change_new (locked_by) WHERE locked_by IS NOT NULL;
CREATE INDEX file_change_new__profile_id__idx
  ON file_change_new (profile_id) WHERE profile_id IS NOT NULL;
CREATE INDEX file_change_new__user_snapshots__idx
  ON file_change_new (file_id)
 WHERE label IS NOT NULL AND created_by = 'user' AND deleted_at IS NULL;
  • Prepare and rehearse the cutover script in staging.

4.2 The cutover (one transaction, a brief exclusive lock)

  • Stop the backend and the workers before the transaction, and start them again after it. The pool keeps server-side prepared statements, and a prepared statement holds the OID of the table it was prepared with: renaming the table does not change that OID. A connection that had already run insert into file_change five times (the default threshold of the driver) would keep writing to file_change_legacy after the cutover, and the pool recycles its connections every 30 minutes, so the window of silent writes is long.
  • Run the transaction, the ANALYZE, and the statement that records the instant of the cutover (all of it in the block below).
  • Start the backend and the workers again.
SET ROLE penpot;  -- the owner of file_change (see 0.0 and 0.6)

BEGIN;

LOCK TABLE file_change IN ACCESS EXCLUSIVE MODE;

-- The final delta: the rows modified since the last pass (its mark, minus
-- the hour of margin). The mark is the one the last pass left.
WITH mark AS (
  SELECT last_at - INTERVAL '1 hour' AS since
    FROM file_change_copy_state
   WHERE name = 'delta'
)
INSERT INTO file_change_new (id, file_id, created_at, session_id, revn, data, changes,
                             profile_id, features, label, version, created_by,
                             updated_at, deleted_at, migrations, locked_by, kind)
SELECT id, file_id, created_at, session_id, revn, data, changes,
       profile_id, features, label, version, created_by,
       updated_at, deleted_at, migrations, locked_by,
       CASE WHEN label IS NOT NULL THEN 1 ELSE 2 END
  FROM file_change, mark
 WHERE updated_at > mark.since
ON CONFLICT (file_id, id) DO UPDATE
   SET created_at = EXCLUDED.created_at,
       session_id = EXCLUDED.session_id,
       revn       = EXCLUDED.revn,
       data       = EXCLUDED.data,
       changes    = EXCLUDED.changes,
       profile_id = EXCLUDED.profile_id,
       features   = EXCLUDED.features,
       label      = EXCLUDED.label,
       version    = EXCLUDED.version,
       created_by = EXCLUDED.created_by,
       updated_at = EXCLUDED.updated_at,
       deleted_at = EXCLUDED.deleted_at,
       migrations = EXCLUDED.migrations,
       locked_by  = EXCLUDED.locked_by,
       kind       = EXCLUDED.kind
 WHERE file_change_new.updated_at IS DISTINCT FROM EXCLUDED.updated_at;

-- Check inside the lock, and abort if it fails: no row of the old table may
-- be missing in the new one. The new table can carry extra rows that the GC
-- hard deleted from the source; it cleans them up after the cutover.
DO $$
DECLARE missing bigint;
BEGIN
  SELECT count(*) INTO missing
    FROM file_change o
   WHERE NOT EXISTS (SELECT 1 FROM file_change_new n
                      WHERE n.file_id = o.file_id
                        AND n.id = o.id);

  IF missing > 0 THEN
    RAISE EXCEPTION 'file_change: % rows of the old table are missing in file_change_new; cutover aborted', missing;
  END IF;
END $$;

-- The old table frees the canonical names (the same names the migration
-- 0158 uses, with the _legacy suffix)
ALTER TABLE file_change RENAME CONSTRAINT file_change_pkey
  TO file_change_pkey_legacy;
ALTER TABLE file_change RENAME CONSTRAINT file_change_file_id_fkey
  TO file_change_file_id_fkey_legacy;
ALTER TABLE file_change RENAME CONSTRAINT file_change_locked_by_fkey
  TO file_change_locked_by_fkey_legacy;
ALTER TABLE file_change RENAME CONSTRAINT file_change_profile_id_fkey
  TO file_change_profile_id_fkey_legacy;

ALTER INDEX file_change__data_ref_id__idx
  RENAME TO file_change__data_ref_id__idx_legacy;
ALTER INDEX file_change__deleted_at__idx
  RENAME TO file_change__deleted_at__idx_legacy;
ALTER INDEX file_change__file_id__revn__idx
  RENAME TO file_change__file_id__revn__idx_legacy;
ALTER INDEX file_change__locked_by__idx
  RENAME TO file_change__locked_by__idx_legacy;
ALTER INDEX file_change__profile_id__idx
  RENAME TO file_change__profile_id__idx_legacy;
ALTER INDEX file_change__system_snapshots__idx
  RENAME TO file_change__system_snapshots__idx_legacy;
ALTER INDEX IF EXISTS file_change__user_snapshots__idx
  RENAME TO file_change__user_snapshots__idx_legacy;
ALTER INDEX IF EXISTS file_change__updated_at__idx
  RENAME TO file_change__updated_at__idx_legacy;

-- And the new table takes them. Only the primary key needs the rename: the
-- `CHECK` and the foreign keys already carry the canonical name (they have no
-- index, so they could take it from the start), and the partitions inherited
-- it.
ALTER TABLE file_change_new RENAME CONSTRAINT file_change_new_pkey
  TO file_change_pkey;

ALTER INDEX file_change_new__deleted_at__idx
  RENAME TO file_change__deleted_at__idx;
ALTER INDEX file_change_new__file_id__revn__idx
  RENAME TO file_change__file_id__revn__idx;
ALTER INDEX file_change_new__locked_by__idx
  RENAME TO file_change__locked_by__idx;
ALTER INDEX file_change_new__profile_id__idx
  RENAME TO file_change__profile_id__idx;
ALTER INDEX file_change_new__user_snapshots__idx
  RENAME TO file_change__user_snapshots__idx;

-- And the tables are swapped. The partitions and their indexes keep the
-- names they were born with (they are named after the partition, not after
-- the table), so there is nothing else to rename.
ALTER TABLE file_change     RENAME TO file_change_legacy;
ALTER TABLE file_change_new RENAME TO file_change;

-- The foreign key of the legacy table goes away: it is `NO ACTION`, so
-- while the legacy table has rows for a file, the objects GC cannot delete
-- that file (it deletes the rows of the file first, and the legacy ones are
-- not its business). The legacy table is frozen: nothing writes to it again.
ALTER TABLE file_change_legacy
  DROP CONSTRAINT file_change_file_id_fkey_legacy;

COMMIT;

ANALYZE file_change;

-- The instant of the cutover, for the checks of phase 5. It goes after the
-- `COMMIT`: inside the transaction `now()` is the start of it (the moment of
-- the `LOCK`), and the rows the final delta copies could pass for writes
-- happened after the cutover.
INSERT INTO file_change_copy_state (name, last_at) VALUES ('cutover', now())
ON CONFLICT (name) DO UPDATE SET last_at = excluded.last_at;
  • file_change_legacy keeps its data and its indexes (with the _legacy suffix): a frozen copy of the previous layout. Its foreign key on file_id is dropped, so the objects GC can delete the files again.
  • The writes to file_change during the lock fail: they are seconds-minutes, the affected saves are retried.
  • The objects GC and the four RPC commands keep working: same table name and same index names after the rename.
  • Once the backend is up again, check that the saves go to the partitioned table and that nothing went to the table that is going away:
-- the rows written after the cutover: this one grows with every save
SELECT count(*) FROM file_change
 WHERE updated_at > (SELECT last_at FROM file_change_copy_state
                      WHERE name = 'cutover');

-- the rows that went to the legacy table: this one has to stay at 0
SELECT count(*) FROM file_change_legacy
 WHERE updated_at > (SELECT last_at FROM file_change_copy_state
                      WHERE name = 'cutover');

A number above zero in the second query means that some connection was still writing with a prepared statement of the old table: those rows are lost when the legacy table is dropped, so they have to be copied over before that.


Phase 5 — After the cutover

  1. Check that the access paths prune to one partition (when the planner chooses an index scan):
EXPLAIN SELECT * FROM file_change
 WHERE file_id = '00000000-0000-0000-0000-000000000000'
   AND id     = '00000000-0000-0000-0000-000000000000';

It has to show a single partition (file_change_NN), not an Append of 16.

  1. Check that the schema is the one of the migration 0158. Every row of this query has to show got = expected:
SELECT 'columns' AS what, count(*) AS got, 17 AS expected
  FROM pg_attribute
 WHERE attrelid = 'file_change'::regclass AND attnum > 0 AND NOT attisdropped
UNION ALL
SELECT 'partitions', count(*), 16
  FROM pg_partition_tree('file_change') t WHERE t.isleaf
UNION ALL
SELECT 'primary key', count(*), 1
  FROM pg_constraint
 WHERE conrelid = 'file_change'::regclass AND contype = 'p'
UNION ALL
SELECT 'foreign keys', count(*), 3
  FROM pg_constraint
 WHERE conrelid = 'file_change'::regclass AND contype = 'f'
UNION ALL
SELECT 'check constraints', count(*), 1
  FROM pg_constraint
 WHERE conrelid = 'file_change'::regclass AND contype = 'c'
UNION ALL
SELECT 'indexes', count(*), 6
  FROM pg_indexes
 WHERE tablename = 'file_change' AND schemaname = current_schema()
UNION ALL
SELECT 'partitions out of name', count(*), 0
  FROM pg_class
 WHERE relkind = 'r' AND relname ~ '^file_change_new[0-9]'
UNION ALL
SELECT 'child indexes out of name', count(*), 0
  FROM pg_class c
  JOIN pg_index i ON i.indexrelid = c.oid
  JOIN pg_class p ON p.oid = i.indrelid
 WHERE p.relname ~ '^file_change_[0-9]{2}$'
   AND c.relname NOT LIKE p.relname || '_%'
UNION ALL
SELECT 'partition constraints out of name', count(*), 0
  FROM pg_constraint c
  JOIN pg_class t ON t.oid = c.conrelid
 WHERE t.relname ~ '^file_change_[0-9]{2}$'
   AND c.contype <> 'n'
   AND c.conname LIKE 'file_change_new%';

The NOT NULL constraints (contype = 'n', tracked in pg_constraint from PostgreSQL 17) are excluded from the last check: their name is generated by the column and changes with the server version, so they are not part of the parity. The CHECK, the primary key and the foreign keys are.

And the names, to compare them with the migration:

SELECT a.attname AS column
  FROM pg_attribute a
 WHERE a.attrelid = 'file_change'::regclass AND a.attnum > 0
   AND NOT a.attisdropped
 ORDER BY a.attnum;

SELECT indexname FROM pg_indexes
 WHERE tablename = 'file_change' AND schemaname = current_schema()
 ORDER BY indexname;

-- the partitions, with their name
SELECT c.relname AS partition
  FROM pg_class c
  JOIN pg_partition_tree('file_change') t ON t.relid = c.oid
 WHERE t.isleaf
 ORDER BY c.relname;

-- the indexes of the partitions: their name starts with the partition
SELECT c.relname AS child_index
  FROM pg_class c
  JOIN pg_index i ON i.indexrelid = c.oid
  JOIN pg_class p ON p.oid = i.indrelid
 WHERE p.relname ~ '^file_change_[0-9]{2}$'
 ORDER BY c.relname;

-- every table, partition and index of the family shares the owner of the
-- table: a session without `SET ROLE` (0.0) shows here
SELECT count(*) AS out_of_owner
  FROM pg_class c
 WHERE c.relname LIKE 'file_change%'
   AND c.relkind IN ('r', 'p', 'i')
   AND pg_get_userbyid(c.relowner) <>
       (SELECT tableowner FROM pg_tables
         WHERE schemaname = current_schema() AND tablename = 'file_change');

The owner check has to return 0. If it does not, the session that created the table did not use SET ROLE: fix it before opening the service (re-own the table, the 16 partitions and their indexes) and re-apply the grants saved in 0.6.

  1. The partition-wise join is already on: the pool of the backend sets enable_partitionwise_join = on on every connection (it is in the code, there is nothing to change in the database role).

  2. Autovacuum: each of the 16 partitions has its own TOAST (check their sizes with pg_total_relation_size per partition); with autovacuum_vacuum_scale_factor = 0.2 the threshold is reached earlier and in steps. Nothing to change, just watch last_autovacuum of the partitions the first week.

  3. The aggregations that do not filter by file_id (the team snapshot quota and the telemetry counts) read all 16 partitions. To know their cost on your instance, time them before and after with EXPLAIN (ANALYZE, BUFFERS) and compare:

    • The team quota is already fixed in the code: it resolves the file_id of the team first, which lets the planner prune the partitions when it chooses an index scan (with a sequential scan it reads all of them).
    • The telemetry scans the whole heap, the same before and after: the partitioned run pays a small cost per partition on top. To remove that scan, the index file_change__created_at__idx turns both metrics into an index scan over the rows of the day, at the cost of one more index on the biggest table. Decide it before phase 1, the index has to exist in both tables.
  4. Drop the helper table of the copy and, days later, the old table:

DROP TABLE IF EXISTS file_change_copy_state;
DROP TABLE file_change_legacy;   -- out of hours: it generates WAL by
                                  -- dropping the pages, let it roll

That is where the dead space comes back: the database shrinks by about the size of the old table (see sizing).


Risks

Risk Mitigation
Massive WAL during the bulk (proportional to the copy; see sizing) Throttle between batches; watch pg_wal, the replication slots and the replica lag
Latency seen by the application during the bulk Throttle; low traffic hours; the bulk is I/O and does not block writes
updated_at not reliable in the rows changed before the deploy The deploy goes before the bulk (0.3) and the delta passes cover from the start of the bulk
Lock/unlock/soft delete during the passes Covered by the watermark; the cutover includes the final delta inside the lock
Hard delete of the GC between passes The row is left with deleted_at in the new table; the GC cleans it up later
Orphan rows if a file is deleted halfway The foreign key on file_id is NO ACTION: the GC deletes the rows of the file before the file, so a chunk that fails leaves the file in the trash and the next run deletes it
The files in the trash cannot be deleted while the legacy table has rows for them The foreign key of the legacy table is dropped at the cutover; until then the files wait in the trash
Saves failing during the lock of the cutover Freeze of minutes in a low traffic window; manual retries
Planner regression after the cutover Immediate ANALYZE + the pruning EXPLAIN; the pool already sets the GUC
Clock drift between the application and the database The delta window starts an hour before the bulk; the margin absorbs the drift

Success metrics

  • After the cutover: file_change (partitioned) holds at most the pg_table_size of the old table (see sizing).
  • After the DROP of the old table: the database shrinks by about the size that table had.
  • The access paths that filter by file_id (the lookup of a snapshot, the quota of a file and the lock of the GC by pair) prune to one partition when the planner chooses an index scan (EXPLAIN). The quota of the team and the discovery of the GC do not filter by a single file: they read all the 16 partitions.
  • No visible change for the user.