Migration 0158 created file_change with kind NOT NULL, which blocks any emergency rollback to the previous version: the old code inserts without kind and every save would fail. kind is still a reserve (reads use label IS NOT NULL), so it stays nullable until the rollback window closes; the NOT NULL comes in a later migration. The runbook keeps parity, the schema and migration tests cover the nullable column and the rollback insert, and the storage memory records the decision. AI-assisted-by: muse-spark-1.3-contributor
41 KiB
Partition file_change by file_id
Procedure to turn file_change into a table partitioned by HASH (file_id)
with 16 partitions, aligned with the ones of file_data. It runs online
except the cutover (phase 4), which needs a scheduled maintenance window with
the backend and the workers stopped.
- Migration:
backend/src/app/migrations/sql/0158-partition-file-change.sql - Issue: https://github.com/penpot/penpot/issues/12416
The result has to be exactly the schema of the migration, because on a big instance the migration is marked as applied and never runs. This procedure runs the same statements by hand.
When to run it, and the stop
Run it only on the instances where the automatic migration refuses to run:
file_change above 10 GiB of total size or above 2 million rows. Below that
the migration does the same work on boot and this procedure has nothing to do.
The work spans the upgrade to 2.20.0. The order is not a suggestion: the
preparation goes before the version update, and phases 2 and 3 need the new
code running (it is the one that moves updated_at on every write, the
watermark of the delta passes).
| Phase | When | Service |
|---|---|---|
| 0 and 1 | Before the upgrade, with 2.19.0 serving | Online |
| — | The upgrade to 2.20.0 | Usual deploy (step 0.3), before phase 2 |
| 2 | After the upgrade, with 2.20.0 serving | Online |
| 3 | From phase 2 to the window, every 1-6 h | Online |
| 4 | A scheduled maintenance window | Backend and workers stopped |
| 5 | After the cutover | Online |
Phase 2 is one pass over every row: the long job, hours or days. Phase 3 repeats a much shorter copy, only the rows that changed since the previous pass (seconds each), until the day of the window, so the last delta inside the lock stays short.
Steps 0.1 and 0.2 cannot wait for the upgrade: a 2.20.0 backend that boots without them runs the automatic migration inside the startup transaction and, above the guard, it fails on every boot.
The procedure needs one stop, and only one: phase 4, the cutover. It is a
full stop-start of the backend and the workers, not a rolling restart (4.2
explains why). The window covers the last delta, the renames and the start
again: seconds to minutes of writes blocked on file_change, and the saves
that fall inside fail and the user retries. Everything else runs with the
instance serving traffic.
Sizing your instance
Every number in this procedure comes from your database: run the queries below, the procedure does not depend on any absolute size. It is plain SQL and runs on any supported PostgreSQL (>= 15).
-- total size of the database and of file_change (heap + TOAST + indexes)
SELECT pg_size_pretty(pg_database_size(current_database())) AS database;
SELECT pg_size_pretty(pg_total_relation_size('file_change')) AS file_change_total,
pg_size_pretty(pg_table_size('file_change')) AS heap_and_toast,
pg_size_pretty(pg_indexes_size('file_change')) AS indexes;
SELECT count(*) AS rows FROM file_change;
-- dead tuples waiting for vacuum: their space is not copied
SELECT n_dead_tup FROM pg_stat_user_tables WHERE relname = 'file_change';
pg_table_size (heap + TOAST) is the upper bound of what the copy reads and
writes. Do not estimate it with pg_stats.avg_width: that measures what the
heap stores for a column, which for a TOASTed value is the pointer, not the
value. The estimate reads far below the real size (about ten times less on a
small instance, orders of magnitude on a big one).
Peak disk during the bulk is about the old table (it does not grow) plus the
new table (at most its pg_table_size, less because the dead rows are not
copied) plus its six indexes (about the pg_indexes_size of the live rows)
plus the WAL the bulk generates (proportional to what it writes: watch the
free space of the pg_wal mount while the first batches run). The disk comes
back when the legacy table is dropped, days later. Time the first batches and
extrapolate: the bulk reads heap + TOAST once sequentially and writes about
the same, so its duration is proportional to pg_table_size, not to the heap
alone.
What changes
file_changeis partitioned byHASH (file_id)with 16 partitions, the same bounds asfile_data, so the join between them is resolved partition by partition.- The primary key goes from
(id)to(file_id, id): PostgreSQL requires the partition key in the primary key, and the application already filters byfile_ideverywhere. - The table has 17 columns:
data_ref_idanddata_backendare gone (dead since the file storage refactor),datais kept (thelegacy-dbbackend reads old payloads from it) andkindappears (1 = snapshot, 2 = revision, nullable until the rollback window closes). - The old table is kept as
file_change_legacy, a frozen copy of the previous layout, and dropped days later. - Almost all of the size of the table is dead space (see sizing), so the new table holds at most its heap + TOAST, and less as the dead rows are not copied. The disk comes back when the legacy table is dropped: the database shrinks by the size of the old table.
Why this mechanism
ATTACH as a DEFAULT partition is not possible: the 16 hash siblings cover
the whole hash space, so the constraint of a DEFAULT partition could never
hold, and the ATTACH fails when it validates every row. A UNION
view with triggers, dual writes in the application and logical replication
were discarded too.
What is left is a copy in phases, driven by the updated_at watermark, and a
cutover of minutes: plain SQL, resumable, no new infrastructure. The heap is
small next to the TOAST, so a delta pass without an index scans the heap and
only detoastes the rows that changed.
Phase 0 — Before anything
0.0 The role of the session
Every session of this procedure starts with SET ROLE to the role that owns
file_change (normally penpot):
SET ROLE penpot; -- replace with the owner of file_change (see 0.6)
The new table, its 16 partitions, its indexes and its TOAST are created with
the role of the session, and ALTER TABLE ... OWNER TO does not recurse to the
partitions. Without this, a procedure run as a superuser leaves the table
owned by that superuser and the backend, which connects as penpot, gets
permission denied after the cutover.
0.1 Mark the migration as applied
The runner applies at startup every migration that is not in the migrations
table. Without this, it would copy the whole table inside the transaction of the
startup.
INSERT INTO migrations (module, step)
VALUES ('main', '0158-partition-file-change');
The module is main and the step has no .sql (see
backend/src/app/util/migrations.clj).
0.2 Add the kind column to the current table
Until the cutover, file_change is the current table, and the new code writes
kind on every insert: without the column the backend does not start.
ALTER TABLE file_change ADD COLUMN IF NOT EXISTS kind smallint;
Nullable and without a default on purpose: it is a metadata-only operation
(it does not rewrite a single row of the table), the old code keeps inserting
rows, and the values of the rows that are already there are computed by the
copy from their label.
0.3 Deploy the code
The updated_at watermark has to be alive before the bulk copy: the new
code updates it on every mutation (locks, soft deletes), the old one does
not. Rows changed between the bulk and a delta pass would be lost.
SELECT max(updated_at) FROM file_change;
0.4 Index for the delta passes
The delta passes filter by updated_at. Without an index on that column every
pass scans the whole heap; with it, it is an index range that only
touches the rows that changed.
-- CONCURRENTLY: it does not block the writes. It cannot run inside a
-- transaction: run it alone.
CREATE INDEX CONCURRENTLY IF NOT EXISTS file_change__updated_at__idx
ON file_change (updated_at);
- The build reads the heap once sequentially and writes one narrow entry per
row (check its size with
pg_relation_sizeafter the build): it does not block the writes either way. Time the build and extrapolate the bulk from it (an index on a column without TOAST does not touch the TOAST). - If it is interrupted, the index is left
INVALIDand has to be dropped and built again:
SELECT i.indisvalid
FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid
WHERE c.relname = 'file_change__updated_at__idx';
- The index goes with the old table: at the cutover it is renamed to
file_change__updated_at__idx_legacyand disappears with it. The new table does not carry it (nothing queries byupdated_at; the GC usesdeleted_at).
0.5 Check the environment
-- Space: peak about the old table (it does not grow) + the new table
-- (at most its `pg_table_size`, see sizing) + the WAL the bulk generates
SELECT pg_size_pretty(pg_database_size(current_database()));
-- The WAL must not run out of room (note the free space of the mount point):
-- df -h <pg_wal dir> <data dir>
-- Nothing must reference file_change (0 rows expected):
SELECT conrelid::regclass FROM pg_constraint
WHERE confrelid = 'file_change'::regclass;
-- There must be no triggers on file_change:
SELECT tgname FROM pg_trigger
WHERE tgrelid = 'file_change'::regclass AND NOT tgisinternal;
-- Replication slots: the bulk generates WAL proportional to what it writes,
-- watch the retained WAL while the first batches run.
SELECT slot_name, active, pg_size_pretty(pg_wal_lsn_diff(
pg_current_wal_lsn(), restart_lsn)) AS retained_wal
FROM pg_replication_slots;
-- The legacy blobs of the snapshots were already moved to `file_data` by
-- migration 0157-migrate-legacy-file-data (on the big instances, by hand,
-- with its runbook 2.19.0-migrate-legacy-file-data.md), so there is nothing
-- to copy here: the `data` column travels with the row to the new table.
-- These two numbers are informational guards. The first one has to be 0
-- after 0157 ran; if it is not, run 0157 (or its runbook) before starting.
SELECT count(*) AS snapshots_without_data
FROM file_change AS fc
WHERE fc.data IS NOT NULL
AND fc.label IS NOT NULL
AND NOT EXISTS (SELECT 1 FROM file_data AS fd
WHERE fd.file_id = fc.file_id
AND fd.id = fc.id
AND fd.type = 'snapshot');
-- The revisions with legacy data are not migrated (they are the ephemeral
-- changes, out of scope). This number is informative: their data travels
-- with the row to the new table.
SELECT count(*) AS revisions_with_data
FROM file_change
WHERE data IS NOT NULL
AND label IS NULL;
- Check that the backup / PITR is working before starting.
- Pick a low activity window for the cutover: a freeze of the writes to
file_changeof a couple of minutes (the final delta inside the lock copies only the rows changed since the last delta pass, so it is quick; the saves of files during the freeze fail and the user retries, and the rest of the database keeps working). The window has to cover the whole cutover, with the backend and the workers stopped and started again (see 4.2).
0.6 Owner and grants
-- the role that has to own every object at the end (see 0.0)
SELECT tableowner FROM pg_tables
WHERE schemaname = current_schema() AND tablename = 'file_change';
-- grants added by the operator (monitoring, reporting). This procedure
-- creates a new table: they do not travel with it, so save them to re-apply
-- them after the cutover.
SELECT grantee, privilege_type
FROM information_schema.role_table_grants
WHERE table_schema = current_schema() AND table_name = 'file_change';
Phase 1 — Create the partitioned table
The new indexes carry a _new suffix because index names are unique per
schema and the old table still holds the canonical ones until the cutover.
The same goes for the primary key of the new table: its index is named after
the constraint. The constraints that have no index (the CHECK and the
foreign keys) are named canonically from the start, because a constraint name
is unique per table, not per schema, so it can live with the one of the old
table; the partitions inherit it when they are created. The partitions are
created with their final name, because their indexes are named after the
partition and not after the table.
CREATE TABLE file_change_new (
file_id uuid NOT NULL,
id uuid NOT NULL DEFAULT gen_random_uuid(),
created_at timestamptz NOT NULL DEFAULT now(),
session_id uuid,
revn bigint NOT NULL DEFAULT 0,
data bytea,
changes bytea,
profile_id uuid,
features text[],
label text,
version integer,
created_by text NOT NULL DEFAULT 'system',
updated_at timestamptz DEFAULT now(),
deleted_at timestamptz,
migrations text[],
locked_by uuid,
-- Nullable on purpose, like the migration: the previous version inserts
-- without `kind`, and a rollback to it has to keep writing. The `NOT NULL`
-- comes in a later migration, once the rollback window is closed.
kind smallint,
CONSTRAINT file_change_kind_check CHECK (kind IN (1, 2))
) PARTITION BY HASH (file_id);
-- the columns keep the storage of PostgreSQL by default (migration
-- 0156-reset-pg-storage-defaults dropped the explicit EXTERNAL overrides)
COMMENT ON COLUMN file_change_new.kind IS
'The kind of the row: 1 is a snapshot (it has a label and its data lives in file_data) and 2 is a revision (it has no label and it holds the changes).';
COMMENT ON COLUMN file_change_new.locked_by IS
'Profile ID of user who has locked this version. Only the creator can lock/unlock their own versions. Locked versions cannot be deleted by others.';
-- 16 partitions, the same bounds as file_data. The name of the table is
-- temporary (the old one holds it until the cutover) but the name of the
-- partitions is not: it is the final one, so their indexes are born with
-- the canonical names (renaming a partition does not rename its indexes)
-- and the result is exactly the schema of the migration.
CREATE TABLE file_change_00 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 0);
CREATE TABLE file_change_01 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 1);
CREATE TABLE file_change_02 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 2);
CREATE TABLE file_change_03 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 3);
CREATE TABLE file_change_04 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 4);
CREATE TABLE file_change_05 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 5);
CREATE TABLE file_change_06 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 6);
CREATE TABLE file_change_07 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 7);
CREATE TABLE file_change_08 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 8);
CREATE TABLE file_change_09 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 9);
CREATE TABLE file_change_10 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 10);
CREATE TABLE file_change_11 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 11);
CREATE TABLE file_change_12 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 12);
CREATE TABLE file_change_13 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 13);
CREATE TABLE file_change_14 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 14);
CREATE TABLE file_change_15 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 15);
-- The primary key includes the partition key (required on a partitioned table)
ALTER TABLE file_change_new ADD PRIMARY KEY (file_id, id);
-- Foreign keys (no CREATE TABLE copies them). They take the canonical name
-- already: they have no index, so they do not collide with the old table and
-- the partitions inherit it.
ALTER TABLE file_change_new
ADD CONSTRAINT file_change_file_id_fkey FOREIGN KEY (file_id)
REFERENCES file(id) DEFERRABLE;
ALTER TABLE file_change_new
ADD CONSTRAINT file_change_locked_by_fkey FOREIGN KEY (locked_by)
REFERENCES profile(id) ON DELETE SET NULL DEFERRABLE;
ALTER TABLE file_change_new
ADD CONSTRAINT file_change_profile_id_fkey FOREIGN KEY (profile_id)
REFERENCES profile(id) ON DELETE SET NULL DEFERRABLE;
SELECT count(*) AS partitions FROM pg_partition_tree('file_change_new') t
WHERE t.isleaf;
-- 16
Phase 2 — Bulk copy
Copies every row (including the soft deleted ones: the GC manages them the
same way in the new table). The kind is computed from the label: a row
with a label is a snapshot, a row without one is a revision.
The pointer of the copy lives in the database, not in the terminal: a batch can be interrupted and resumed without losing anything.
CREATE TABLE IF NOT EXISTS file_change_copy_state (
name text PRIMARY KEY,
last_id uuid,
last_at timestamptz
);
-- for a procedure that already created the table without the mark
ALTER TABLE file_change_copy_state
ADD COLUMN IF NOT EXISTS last_at timestamptz;
INSERT INTO file_change_copy_state (name, last_id)
VALUES ('bulk', '00000000-0000-0000-0000-000000000000')
ON CONFLICT (name) DO NOTHING;
-- The mark of the delta passes starts at the bulk: the first pass of phase 3
-- covers the rows changed since here, and `DO NOTHING` keeps a re-run from
-- resetting a mark that the passes already moved.
INSERT INTO file_change_copy_state (name, last_at) VALUES ('delta', now())
ON CONFLICT (name) DO NOTHING;
-- The mark is mandatory: a pass without it copies nothing, in silence. The
-- `DROP ... IF EXISTS` keeps this block re-runnable.
ALTER TABLE file_change_copy_state
DROP CONSTRAINT IF EXISTS file_change_copy_state_delta_mark,
ADD CONSTRAINT file_change_copy_state_delta_mark
CHECK (name <> 'delta' OR last_at IS NOT NULL);
One batch (idempotent). It always returns one row: the number of rows it
copied, 0 when there is nothing left to copy.
WITH batch AS MATERIALIZED (
SELECT id FROM file_change
WHERE id > (SELECT last_id FROM file_change_copy_state WHERE name = 'bulk')
ORDER BY id
LIMIT 10000
),
copied AS (
INSERT INTO file_change_new (id, file_id, created_at, session_id, revn, data, changes,
profile_id, features, label, version, created_by,
updated_at, deleted_at, migrations, locked_by, kind)
SELECT id, file_id, created_at, session_id, revn, data, changes,
profile_id, features, label, version, created_by,
updated_at, deleted_at, migrations, locked_by,
CASE WHEN label IS NOT NULL THEN 1 ELSE 2 END
FROM file_change
WHERE id IN (SELECT id FROM batch)
ON CONFLICT (file_id, id) DO UPDATE
SET created_at = EXCLUDED.created_at,
session_id = EXCLUDED.session_id,
revn = EXCLUDED.revn,
data = EXCLUDED.data,
changes = EXCLUDED.changes,
profile_id = EXCLUDED.profile_id,
features = EXCLUDED.features,
label = EXCLUDED.label,
version = EXCLUDED.version,
created_by = EXCLUDED.created_by,
updated_at = EXCLUDED.updated_at,
deleted_at = EXCLUDED.deleted_at,
migrations = EXCLUDED.migrations,
locked_by = EXCLUDED.locked_by,
kind = EXCLUDED.kind
WHERE file_change_new.updated_at IS DISTINCT FROM EXCLUDED.updated_at
RETURNING 1
),
-- `moved` advances the pointer only when the batch copied something.
-- PostgreSQL executes a data-modifying CTE exactly once even when the final
-- query does not read its output.
moved AS (
UPDATE file_change_copy_state
SET last_id = (SELECT id FROM batch ORDER BY id DESC LIMIT 1)
WHERE name = 'bulk'
AND EXISTS (SELECT 1 FROM copied)
RETURNING 1
)
SELECT count(*) AS copied FROM copied;
Repeat this statement until it returns 0. The 0 comes from the final
SELECT, not from a missing row: a batch that finds nothing to copy still
returns it, so it is the signal to stop.
- The pointer is a
uuid(random order): the pagination is stable but it does not group by physical position. The total read is the heap, in a scattered order. - Estimated duration: time the first batches and extrapolate. The bulk reads the heap plus the live TOAST once (see sizing) and writes about the live data. Batches are rows divided by batch size: count your rows in sizing and divide.
- Pace the batches yourself (a pause between statements) if the streaming replica falls behind or the backend sees latency. Every batch detoastes its share of the TOAST: the cost is I/O, not CPU.
- Watch: replica lag,
pg_stat_activity(locks between the bulk and autovacuum), the space ofpg_waland of the data dir.
SELECT count(*) AS missing
FROM file_change o
WHERE NOT EXISTS (SELECT 1 FROM file_change_new n
WHERE n.file_id = o.file_id
AND n.id = o.id);
missing is zero when no row of the old table is absent from the new one; it
is the check that matters, not the total count. The new table can have more
rows than the old one: the GC may have hard deleted a row from the source
after the bulk copied it, and it cleans that row up after the cutover. Rows
written during the bulk are missing until the delta passes pick them up.
Phase 3 — Delta passes
Every pass copies the rows modified since the previous pass. The watermark
lives in file_change_copy_state.last_at (the start of the last pass; the
setup block of phase 2 seeds it with the moment of the bulk, so the first pass
covers what changed since then), and the SQL subtracts one hour of margin,
which absorbs the clock drift between the application (which writes
updated_at) and the database (which writes now()) and the drift between
batches. A row that a pass already copied but did not change again is not
rewritten: the WHERE of the ON CONFLICT skips it.
BEGIN;
WITH mark AS (
SELECT last_at - INTERVAL '1 hour' AS since
FROM file_change_copy_state
WHERE name = 'delta'
)
INSERT INTO file_change_new (id, file_id, created_at, session_id, revn, data, changes,
profile_id, features, label, version, created_by,
updated_at, deleted_at, migrations, locked_by, kind)
SELECT id, file_id, created_at, session_id, revn, data, changes,
profile_id, features, label, version, created_by,
updated_at, deleted_at, migrations, locked_by,
CASE WHEN label IS NOT NULL THEN 1 ELSE 2 END
FROM file_change, mark
WHERE updated_at > mark.since
ON CONFLICT (file_id, id) DO UPDATE
SET created_at = EXCLUDED.created_at,
session_id = EXCLUDED.session_id,
revn = EXCLUDED.revn,
data = EXCLUDED.data,
changes = EXCLUDED.changes,
profile_id = EXCLUDED.profile_id,
features = EXCLUDED.features,
label = EXCLUDED.label,
version = EXCLUDED.version,
created_by = EXCLUDED.created_by,
updated_at = EXCLUDED.updated_at,
deleted_at = EXCLUDED.deleted_at,
migrations = EXCLUDED.migrations,
locked_by = EXCLUDED.locked_by,
kind = EXCLUDED.kind
WHERE file_change_new.updated_at IS DISTINCT FROM EXCLUDED.updated_at;
-- The mark is the start of this transaction (`now()`): the rows changed
-- after it are the window of the next pass. Both statements go in the same
-- transaction on purpose; a mark written later would leave a gap.
UPDATE file_change_copy_state SET last_at = now() WHERE name = 'delta';
COMMIT;
- Repeat the transaction every 1-6 h (depending on the activity) until the pass copies very few rows. With the index of phase 0.4 the pass is an index range over the window, so it takes seconds on a quiet instance; without it, it scans the heap. To see how many rows are left in the window:
SELECT count(*) FROM file_change
WHERE updated_at > (SELECT last_at FROM file_change_copy_state WHERE name = 'delta');
- The objects GC can keep running. Its soft deletes are covered by the
watermark, and its hard deletes leave a row with
deleted_atin the new table that the GC itself cleans up later. The hard delete of a file is blocked while the old table (or the new one) has rows for it, because the foreign key onfile_idisNO ACTION. The GC deletes the pending files in chunks of 100, and one blocked file aborts the transaction of the whole chunk: the chunk is logged aserror on chunk deletionand skipped, its files wait in the trash, and the next daily run retries the same chunk and fails again until the cutover. The other chunks keep making progress. Do not run the cutover at the same time as a big batch of the GC.
Phase 4 — Cutover (a freeze of minutes)
4.1 Before the lock
- One last delta pass.
- Build the secondary indexes of the new table. The bulk and the delta passes
only need the primary key (
ON CONFLICT (file_id, id)), so building them here keeps five indexes and their WAL out of every batch. Thedeleted_atindex carriesfile_id(the primary key is(file_id, id)): the objects GC needs thefile_idto lock by pair, and with the index it reads the rows it has to delete without touching the table.
-- Secondary indexes: the canonical names, with the _new suffix. There is no
-- index for the system snapshots: the old one filtered them by
-- `data IS NOT NULL`, and with the payload living in `file_data` the column
-- no longer tells snapshots apart. The new one could only filter
-- them by `deleted_at IS NULL`, and the system snapshots always carry a
-- deletion date in the future, so it would never match a row.
CREATE INDEX file_change_new__deleted_at__idx
ON file_change_new (deleted_at, file_id, id) WHERE deleted_at IS NOT NULL;
CREATE INDEX file_change_new__file_id__revn__idx
ON file_change_new (file_id, revn);
CREATE INDEX file_change_new__locked_by__idx
ON file_change_new (locked_by) WHERE locked_by IS NOT NULL;
CREATE INDEX file_change_new__profile_id__idx
ON file_change_new (profile_id) WHERE profile_id IS NOT NULL;
CREATE INDEX file_change_new__user_snapshots__idx
ON file_change_new (file_id)
WHERE label IS NOT NULL AND created_by = 'user' AND deleted_at IS NULL;
- Prepare and rehearse the cutover script in staging.
4.2 The cutover (one transaction, a brief exclusive lock)
- Stop the backend and the workers before the transaction, and start them
again after it. The pool keeps server-side prepared statements, and a
prepared statement holds the OID of the table it was prepared with:
renaming the table does not change that OID. A connection that had already
run
insert into file_changefive times (the default threshold of the driver) would keep writing tofile_change_legacyafter the cutover, and the pool recycles its connections every 30 minutes, so the window of silent writes is long. - Run the transaction, the
ANALYZE, and the statement that records the instant of the cutover (all of it in the block below). - Start the backend and the workers again.
SET ROLE penpot; -- the owner of file_change (see 0.0 and 0.6)
BEGIN;
LOCK TABLE file_change IN ACCESS EXCLUSIVE MODE;
-- The final delta: the rows modified since the last pass (its mark, minus
-- the hour of margin). The mark is the one the last pass left.
WITH mark AS (
SELECT last_at - INTERVAL '1 hour' AS since
FROM file_change_copy_state
WHERE name = 'delta'
)
INSERT INTO file_change_new (id, file_id, created_at, session_id, revn, data, changes,
profile_id, features, label, version, created_by,
updated_at, deleted_at, migrations, locked_by, kind)
SELECT id, file_id, created_at, session_id, revn, data, changes,
profile_id, features, label, version, created_by,
updated_at, deleted_at, migrations, locked_by,
CASE WHEN label IS NOT NULL THEN 1 ELSE 2 END
FROM file_change, mark
WHERE updated_at > mark.since
ON CONFLICT (file_id, id) DO UPDATE
SET created_at = EXCLUDED.created_at,
session_id = EXCLUDED.session_id,
revn = EXCLUDED.revn,
data = EXCLUDED.data,
changes = EXCLUDED.changes,
profile_id = EXCLUDED.profile_id,
features = EXCLUDED.features,
label = EXCLUDED.label,
version = EXCLUDED.version,
created_by = EXCLUDED.created_by,
updated_at = EXCLUDED.updated_at,
deleted_at = EXCLUDED.deleted_at,
migrations = EXCLUDED.migrations,
locked_by = EXCLUDED.locked_by,
kind = EXCLUDED.kind
WHERE file_change_new.updated_at IS DISTINCT FROM EXCLUDED.updated_at;
-- Check inside the lock, and abort if it fails: no row of the old table may
-- be missing in the new one. The new table can carry extra rows that the GC
-- hard deleted from the source; it cleans them up after the cutover.
DO $$
DECLARE missing bigint;
BEGIN
SELECT count(*) INTO missing
FROM file_change o
WHERE NOT EXISTS (SELECT 1 FROM file_change_new n
WHERE n.file_id = o.file_id
AND n.id = o.id);
IF missing > 0 THEN
RAISE EXCEPTION 'file_change: % rows of the old table are missing in file_change_new; cutover aborted', missing;
END IF;
END $$;
-- The old table frees the canonical names (the same names the migration
-- 0158 uses, with the _legacy suffix)
ALTER TABLE file_change RENAME CONSTRAINT file_change_pkey
TO file_change_pkey_legacy;
ALTER TABLE file_change RENAME CONSTRAINT file_change_file_id_fkey
TO file_change_file_id_fkey_legacy;
ALTER TABLE file_change RENAME CONSTRAINT file_change_locked_by_fkey
TO file_change_locked_by_fkey_legacy;
ALTER TABLE file_change RENAME CONSTRAINT file_change_profile_id_fkey
TO file_change_profile_id_fkey_legacy;
ALTER INDEX file_change__data_ref_id__idx
RENAME TO file_change__data_ref_id__idx_legacy;
ALTER INDEX file_change__deleted_at__idx
RENAME TO file_change__deleted_at__idx_legacy;
ALTER INDEX file_change__file_id__revn__idx
RENAME TO file_change__file_id__revn__idx_legacy;
ALTER INDEX file_change__locked_by__idx
RENAME TO file_change__locked_by__idx_legacy;
ALTER INDEX file_change__profile_id__idx
RENAME TO file_change__profile_id__idx_legacy;
ALTER INDEX file_change__system_snapshots__idx
RENAME TO file_change__system_snapshots__idx_legacy;
ALTER INDEX IF EXISTS file_change__user_snapshots__idx
RENAME TO file_change__user_snapshots__idx_legacy;
ALTER INDEX IF EXISTS file_change__updated_at__idx
RENAME TO file_change__updated_at__idx_legacy;
-- And the new table takes them. Only the primary key needs the rename: the
-- `CHECK` and the foreign keys already carry the canonical name (they have no
-- index, so they could take it from the start), and the partitions inherited
-- it.
ALTER TABLE file_change_new RENAME CONSTRAINT file_change_new_pkey
TO file_change_pkey;
ALTER INDEX file_change_new__deleted_at__idx
RENAME TO file_change__deleted_at__idx;
ALTER INDEX file_change_new__file_id__revn__idx
RENAME TO file_change__file_id__revn__idx;
ALTER INDEX file_change_new__locked_by__idx
RENAME TO file_change__locked_by__idx;
ALTER INDEX file_change_new__profile_id__idx
RENAME TO file_change__profile_id__idx;
ALTER INDEX file_change_new__user_snapshots__idx
RENAME TO file_change__user_snapshots__idx;
-- And the tables are swapped. The partitions and their indexes keep the
-- names they were born with (they are named after the partition, not after
-- the table), so there is nothing else to rename.
ALTER TABLE file_change RENAME TO file_change_legacy;
ALTER TABLE file_change_new RENAME TO file_change;
-- The foreign key of the legacy table goes away: it is `NO ACTION`, so
-- while the legacy table has rows for a file, the objects GC cannot delete
-- that file (it deletes the rows of the file first, and the legacy ones are
-- not its business). The legacy table is frozen: nothing writes to it again.
ALTER TABLE file_change_legacy
DROP CONSTRAINT file_change_file_id_fkey_legacy;
COMMIT;
ANALYZE file_change;
-- The instant of the cutover, for the checks of phase 5. It goes after the
-- `COMMIT`: inside the transaction `now()` is the start of it (the moment of
-- the `LOCK`), and the rows the final delta copies could pass for writes
-- happened after the cutover.
INSERT INTO file_change_copy_state (name, last_at) VALUES ('cutover', now())
ON CONFLICT (name) DO UPDATE SET last_at = excluded.last_at;
file_change_legacykeeps its data and its indexes (with the_legacysuffix): a frozen copy of the previous layout. Its foreign key onfile_idis dropped, so the objects GC can delete the files again.- The writes to
file_changeduring the lock fail: they are seconds-minutes, the affected saves are retried. - The objects GC and the four RPC commands keep working: same table name and same index names after the rename.
- Once the backend is up again, check that the saves go to the partitioned table and that nothing went to the table that is going away:
-- the rows written after the cutover: this one grows with every save
SELECT count(*) FROM file_change
WHERE updated_at > (SELECT last_at FROM file_change_copy_state
WHERE name = 'cutover');
-- the rows that went to the legacy table: this one has to stay at 0
SELECT count(*) FROM file_change_legacy
WHERE updated_at > (SELECT last_at FROM file_change_copy_state
WHERE name = 'cutover');
A number above zero in the second query means that some connection was still writing with a prepared statement of the old table: those rows are lost when the legacy table is dropped, so they have to be copied over before that.
Phase 5 — After the cutover
- Check that the access paths prune to one partition (when the planner chooses an index scan):
EXPLAIN SELECT * FROM file_change
WHERE file_id = '00000000-0000-0000-0000-000000000000'
AND id = '00000000-0000-0000-0000-000000000000';
It has to show a single partition (file_change_NN), not an Append of 16.
- Check that the schema is the one of the migration
0158. Every row of this query has to showgot = expected:
SELECT 'columns' AS what, count(*) AS got, 17 AS expected
FROM pg_attribute
WHERE attrelid = 'file_change'::regclass AND attnum > 0 AND NOT attisdropped
UNION ALL
SELECT 'partitions', count(*), 16
FROM pg_partition_tree('file_change') t WHERE t.isleaf
UNION ALL
SELECT 'primary key', count(*), 1
FROM pg_constraint
WHERE conrelid = 'file_change'::regclass AND contype = 'p'
UNION ALL
SELECT 'foreign keys', count(*), 3
FROM pg_constraint
WHERE conrelid = 'file_change'::regclass AND contype = 'f'
UNION ALL
SELECT 'check constraints', count(*), 1
FROM pg_constraint
WHERE conrelid = 'file_change'::regclass AND contype = 'c'
UNION ALL
SELECT 'indexes', count(*), 6
FROM pg_indexes
WHERE tablename = 'file_change' AND schemaname = current_schema()
UNION ALL
SELECT 'partitions out of name', count(*), 0
FROM pg_class
WHERE relkind = 'r' AND relname ~ '^file_change_new[0-9]'
UNION ALL
SELECT 'child indexes out of name', count(*), 0
FROM pg_class c
JOIN pg_index i ON i.indexrelid = c.oid
JOIN pg_class p ON p.oid = i.indrelid
WHERE p.relname ~ '^file_change_[0-9]{2}$'
AND c.relname NOT LIKE p.relname || '_%'
UNION ALL
SELECT 'partition constraints out of name', count(*), 0
FROM pg_constraint c
JOIN pg_class t ON t.oid = c.conrelid
WHERE t.relname ~ '^file_change_[0-9]{2}$'
AND c.contype <> 'n'
AND c.conname LIKE 'file_change_new%';
The NOT NULL constraints (contype = 'n', tracked in pg_constraint from
PostgreSQL 17) are excluded from the last check: their name is generated by
the column and changes with the server version, so they are not part of the
parity. The CHECK, the primary key and the foreign keys are.
And the names, to compare them with the migration:
SELECT a.attname AS column
FROM pg_attribute a
WHERE a.attrelid = 'file_change'::regclass AND a.attnum > 0
AND NOT a.attisdropped
ORDER BY a.attnum;
SELECT indexname FROM pg_indexes
WHERE tablename = 'file_change' AND schemaname = current_schema()
ORDER BY indexname;
-- the partitions, with their name
SELECT c.relname AS partition
FROM pg_class c
JOIN pg_partition_tree('file_change') t ON t.relid = c.oid
WHERE t.isleaf
ORDER BY c.relname;
-- the indexes of the partitions: their name starts with the partition
SELECT c.relname AS child_index
FROM pg_class c
JOIN pg_index i ON i.indexrelid = c.oid
JOIN pg_class p ON p.oid = i.indrelid
WHERE p.relname ~ '^file_change_[0-9]{2}$'
ORDER BY c.relname;
-- every table, partition and index of the family shares the owner of the
-- table: a session without `SET ROLE` (0.0) shows here
SELECT count(*) AS out_of_owner
FROM pg_class c
WHERE c.relname LIKE 'file_change%'
AND c.relkind IN ('r', 'p', 'i')
AND pg_get_userbyid(c.relowner) <>
(SELECT tableowner FROM pg_tables
WHERE schemaname = current_schema() AND tablename = 'file_change');
The owner check has to return 0. If it does not, the session that
created the table did not use SET ROLE: fix it before opening the
service (re-own the table, the 16 partitions and their indexes) and
re-apply the grants saved in 0.6.
-
The partition-wise join is already on: the pool of the backend sets
enable_partitionwise_join = onon every connection (it is in the code, there is nothing to change in the database role). -
Autovacuum: each of the 16 partitions has its own TOAST (check their sizes with
pg_total_relation_sizeper partition); withautovacuum_vacuum_scale_factor = 0.2the threshold is reached earlier and in steps. Nothing to change, just watchlast_autovacuumof the partitions the first week. -
The aggregations that do not filter by
file_id(the team snapshot quota and the telemetry counts) read all 16 partitions. To know their cost on your instance, time them before and after withEXPLAIN (ANALYZE, BUFFERS)and compare:- The team quota is already fixed in the code: it resolves the
file_idof the team first, which lets the planner prune the partitions when it chooses an index scan (with a sequential scan it reads all of them). - The telemetry scans the whole heap, the same before and after: the
partitioned run pays a small cost per partition on top. To remove that
scan, the index
file_change__created_at__idxturns both metrics into an index scan over the rows of the day, at the cost of one more index on the biggest table. Decide it before phase 1, the index has to exist in both tables.
- The team quota is already fixed in the code: it resolves the
-
Drop the helper table of the copy and, days later, the old table:
DROP TABLE IF EXISTS file_change_copy_state;
DROP TABLE file_change_legacy; -- out of hours: it generates WAL by
-- dropping the pages, let it roll
That is where the dead space comes back: the database shrinks by about the size of the old table (see sizing).
Risks
| Risk | Mitigation |
|---|---|
| Massive WAL during the bulk (proportional to the copy; see sizing) | Throttle between batches; watch pg_wal, the replication slots and the replica lag |
| Latency seen by the application during the bulk | Throttle; low traffic hours; the bulk is I/O and does not block writes |
updated_at not reliable in the rows changed before the deploy |
The deploy goes before the bulk (0.3) and the delta passes cover from the start of the bulk |
| Lock/unlock/soft delete during the passes | Covered by the watermark; the cutover includes the final delta inside the lock |
| Hard delete of the GC between passes | The row is left with deleted_at in the new table; the GC cleans it up later |
Orphan rows if a file is deleted halfway |
The foreign key on file_id is NO ACTION: the GC deletes the rows of the file before the file, so a chunk that fails leaves the file in the trash and the next run deletes it |
| The files in the trash cannot be deleted while the legacy table has rows for them | The foreign key of the legacy table is dropped at the cutover; until then the files wait in the trash |
| Saves failing during the lock of the cutover | Freeze of minutes in a low traffic window; manual retries |
| Planner regression after the cutover | Immediate ANALYZE + the pruning EXPLAIN; the pool already sets the GUC |
| Clock drift between the application and the database | The delta window starts an hour before the bulk; the margin absorbs the drift |
Success metrics
- After the cutover:
file_change(partitioned) holds at most thepg_table_sizeof the old table (see sizing). - After the
DROPof the old table: the database shrinks by about the size that table had. - The access paths that filter by
file_id(the lookup of a snapshot, the quota of a file and the lock of the GC by pair) prune to one partition when the planner chooses an index scan (EXPLAIN). The quota of the team and the discovery of the GC do not filter by a single file: they read all the 16 partitions. - No visible change for the user.