* ✨ Ship telemetry shadows with a cursor consumer Send anonymized audit events through a dedicated telemetry consumer, including when Nexus or admin-console also persist full events. Keep the legacy payload and cadence of 10,000 events per chunk and three hours between passes. Nexus skips shadows to avoid archiving duplicates. Move shadow deletion to the audit GC. Delete full rows and shadows along independent cursor paths, and expire shadows after seven days even when they have not been sent. Run the deletion paths in parallel with bounded batches, separate time budgets and independent failure handling. Replace the shared consumer catalog with transport definitions and one generic pass runner. Keep the legacy telemetry report and newsletter in the scheduled task, and update the tests, memories and rollout runbooks. Addresses F1 of the audit-log consumer review. AI-assisted-by: deepseek-v4-flash AI-assisted-by: gpt-6.1-terra * 🐛 Address audit log consumer review findings Enable audit log GC by default and document how to pause it during rollout or before registering a consumer that needs a complete replay. Run GC paths in order on the runner thread. Count each committed batch and preserve partial totals on ordinary failures while propagating interruption. Skip full-row payloads in telemetry queries and decoding. Keep the age limit for local retention only, and document the shared pass contract. AI-assisted-by: gpt-6.2-sol * 🐛 Close audit log consumer review findings Verify telemetry counts processed and skipped events in a mixed batch. Clarify how to pause GC, the scope of the replay race, and what the retention warning counts. Remove private Nexus setup from the public upgrade runbook. AI-assisted-by: gpt-6.2-sol * 📚 Simplify the audit log guide Consolidate the guidance on event trust, privacy and delivery. Keep examples that explain why the design works and link to code and runbooks for details. Clarify replay limits and distinguish age-based deletion from missed delivery. AI-assisted-by: gpt-6.2-sol
67 KiB
2.20.0 — Unified upgrade runbook
Single, self-contained procedure for upgrading an instance to the release that ships:
- Partition
file_changebyHASH (file_id)with 16 partitions, aligned withfile_data:backend/src/app/migrations/sql/0158-partition-file-change.sql, issue https://github.com/penpot/penpot/issues/12416. - The file data storage pointer as a column
file_data.storage_object_idwith a foreign key tostorage_object(id):backend/src/app/migrations/sql/0159-add-file-data-storage-object-id.sql, issue https://github.com/penpot/penpot/issues/12769. - The audit-log cursor consumers and the jobs substrate:
backend/src/app/migrations/sql/0154-add-job-table.sqlandbackend/src/app/migrations/sql/0159-consumer-generalization.sql, issue https://github.com/penpot/penpot/issues/12700.
Migrations apply in this order at boot when they are not registered in the
migrations table: 0154-add-job-table, 0158-partition-file-change,
0159-add-file-data-storage-object-id, then 0159-consumer-generalization.
This document is self-contained and manual. Every block is plain SQL,
paste-able into any client that speaks SQL; there are no shell scripts and no
psql meta-commands. It runs on PostgreSQL 15 or newer. It does not replace
operations/2.20.0-file-change-partition.md,
operations/2.20.0-file-data-storage-object-id.md or
operations/2.20.0-consumer-generalization.md: they stay as the reference for
their individual procedure. Where this document and those disagree, this
document wins for the unified execution; they win for the details of one
area only.
When each phase runs
| Phase | When | Service |
|---|---|---|
| 1 | Before the upgrade; pre-flight and online preparation | 2.19.0 serving |
| 2 | The deploy to 2.20.0 | Full stop for the storage backfill |
| 3 | After the upgrade, 2.20.0 serving; phase 3.3 every 1-6 h for days | Online |
| 4 | A scheduled maintenance window for the file_change partition cutover |
Backend and workers stopped |
| 5 | After the cutover | Online |
This runbook is a complete manual procedure. Follow every phase in order,
regardless of database size; do not skip steps based on how much data a table
contains. Phase 2 is a coordinated full stop of the backend and workers for
the storage backfill during the deploy. Phase 4 is a separate scheduled
maintenance window for the file_change partition cutover after the deploy.
The backfill runs on every instance, even when it updates no rows.
Ordering constraints (do not rearrange)
- The migration steps are pre-registered at the beginning: the 2.20.0 migration runner skips them, and the manual procedure below is the only path that applies their schema changes and backfills.
- Phases 3 and 4 need the new code running: it moves
updated_aton every write, the watermark of the copy passes. The deploy (phase 2) goes before phase 3. - The
file_databackfill always runs in phase 2, while every old instance is stopped and before the new instances start: the old code writes the pointer only tometadata.storage-ref-id, the new code reads only the column. A rolling deploy in between would mix both. - The
0158cutover (phase 4) takes minutes in a low-traffic window. Saves that fall inside fail and the user retries. - The new GC runs by default. Keep
disable-audit-log-gcinPENPOT_FLAGSthroughout the mixed-version rollout. Restore the default in phase 5.6.
Phase 1 — Before the upgrade (2.19.0 serving; online)
1.1 Register every migration step first
Run these inserts before any other step in this runbook. They prevent the 2.20.0 migration runner from starting these jobs automatically. The runbook then owns all schema changes and backfills. Do not deploy 2.20.0 until every manual step and verification in this document is complete.
INSERT INTO migrations (module, step)
VALUES ('main', '0158-partition-file-change')
ON CONFLICT (module, step) DO NOTHING;
INSERT INTO migrations (module, step)
VALUES ('main', '0154-add-job-table')
ON CONFLICT (module, step) DO NOTHING;
INSERT INTO migrations (module, step)
VALUES ('main', '0159-consumer-generalization')
ON CONFLICT (module, step) DO NOTHING;
INSERT INTO migrations (module, step)
VALUES ('main', '0159-add-file-data-storage-object-id')
ON CONFLICT (module, step) DO NOTHING;
These rows are pre-registration for this manual procedure, not permission to deploy early. If the procedure is interrupted, keep 2.20.0 stopped and resume the manual work before deploying.
1.2 Pre-flight sizing
-- total size of the database and of file_change (heap + TOAST + indexes)
SELECT pg_size_pretty(pg_database_size(current_database())) AS database;
SELECT pg_size_pretty(pg_total_relation_size('file_change')) AS file_change_total,
pg_size_pretty(pg_table_size('file_change')) AS heap_and_toast,
pg_size_pretty(pg_indexes_size('file_change')) AS indexes;
SELECT count(*) AS rows FROM file_change;
-- audit_log volume: estimate the time for the out-of-band index builds
SELECT pg_size_pretty(pg_total_relation_size('audit_log')) AS audit_log;
SELECT count(*) FROM audit_log;
-- backfill cost of file_change is proportional to pg_table_size,
-- not to the row count: time the first batches in phase 3 and extrapolate
These measurements help estimate the time and disk space the procedure will
need. They do not change which steps to run: this runbook always applies the
full file_change, audit_log and file_data procedures.
After pre-flight, run subsections 1.3, 1.4 and 1.5 in order while 2.19 serves traffic. Do not stop the web instances. Subsection 1.4 stops the legacy archiver and its GC by flag.
1.3 Partition preparation
1.3.1 Add the kind column to the current table. Until the cutover,
file_change is the current table, and the new code writes kind on every
insert: without the column the backend does not start.
ALTER TABLE file_change ADD COLUMN IF NOT EXISTS kind smallint;
Nullable and without a default on purpose: it is a metadata-only operation
(it does not rewrite a single row), the old code keeps inserting rows, and
the values of the rows that are already there are computed by the copy from
their label.
1.3.2 Index for the delta passes. The delta passes filter by
updated_at. Without an index on that column every pass scans the whole
heap; with it, it is an index range that only touches the rows that changed.
-- CONCURRENTLY: it does not block the writes. It cannot run inside a
-- transaction: run it alone.
CREATE INDEX CONCURRENTLY IF NOT EXISTS file_change__updated_at__idx
ON file_change (updated_at);
- If it is interrupted, the index is left
INVALIDand has to be dropped and built again:
SELECT i.indisvalid
FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid
WHERE c.relname = 'file_change__updated_at__idx';
- The index goes with the old table: at the cutover it is renamed to
file_change__updated_at__idx_legacyand disappears with it. The new table does not carry it (nothing queries byupdated_at; the GC usesdeleted_at).
1.4 Consumer generalization: audit_log and jobs
Session settings: SET statement_timeout = 0; SET lock_timeout = '5s';
Re-issue them in each SQL session.
Step 1 — Stop the legacy archiver and its GC
- Before any 2.20.0 worker starts, set
disable-audit-log-archiveanddisable-audit-log-gcinPENPOT_FLAGSon all workers. Remove any existingenable-audit-log-gc; do not leave both tokens. Removingenablealone does not stop the new GC. There is no:audit-logflag in the new version: writing full rows follows the consumer flags. - If the deployment uses the admin console, keep
:admin-consoleon on every instance, web and worker. Each process decides at startup whether to write full rows; enabling the flag only on workers does not make web instances write them. The:telemetryconsumer (id0) ships the anonymized copies with the pace of the old cron (10 000 rows, a pass every 3 hours) and its first pass replays the shadows the buffer holds. While the old version still runs its cron, a copy can reach the endpoint twice; telemetry is at-least-once. - Roll the workers with these flags. Check each worker's startup log to
confirm
audit-log-gcis not active, and wait for active archive and GC tasks to finish. Keepdisable-audit-log-gcin place until phase 5.6.
Step 2 — Delete what the archiver already archived
DELETE FROM audit_log
WHERE id IN (SELECT id FROM audit_log
WHERE archived_at IS NOT NULL
LIMIT 5000);
Step 3 — Pre-apply the jobs step
BEGIN;
CREATE TABLE job (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
name text NOT NULL,
tenant text NOT NULL,
queue text NOT NULL,
label text,
priority int NOT NULL DEFAULT 100,
scheduled_at timestamptz NOT NULL DEFAULT now(),
retry_num int NOT NULL DEFAULT 0,
max_retries int NOT NULL DEFAULT 3,
status text NOT NULL DEFAULT 'new'
CHECK (status IN ('new', 'scheduled', 'running', 'retry',
'completed', 'failed', 'cancelled',
'aborted')),
created_at timestamptz NOT NULL DEFAULT now(),
modified_at timestamptz NOT NULL DEFAULT now(),
started_at timestamptz,
completed_at timestamptz,
params jsonb NOT NULL DEFAULT '{}',
profile_id uuid NULL REFERENCES profile(id) ON DELETE NO ACTION DEFERRABLE,
error jsonb,
result jsonb,
resource_id uuid NULL REFERENCES storage_object(id) ON DELETE SET NULL DEFERRABLE,
expires_at timestamptz
);
CREATE INDEX job__dispatcher__idx
ON job (tenant, priority DESC, scheduled_at)
WHERE status IN ('new', 'retry');
CREATE INDEX job__orphan__idx
ON job (tenant, modified_at)
WHERE status = 'running';
CREATE INDEX job__profile__idx
ON job (profile_id, created_at DESC)
WHERE profile_id IS NOT NULL;
CREATE INDEX job__resource__idx
ON job (resource_id)
WHERE resource_id IS NOT NULL;
-- Created in step 8 with the `modified_at` definition.
CREATE INDEX job__expires__idx
ON job (expires_at)
WHERE expires_at IS NOT NULL;
CREATE INDEX job__retention__idx
ON job (modified_at)
WHERE status IN ('completed', 'failed', 'cancelled', 'aborted')
AND profile_id IS NULL;
CREATE TABLE job_event (
id bigserial PRIMARY KEY,
job_id uuid NOT NULL REFERENCES job(id) ON DELETE CASCADE DEFERRABLE,
kind text NOT NULL
CHECK (kind IN ('start', 'progress', 'retry', 'end')),
payload jsonb NOT NULL DEFAULT '{}',
created_at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX job_event__job_kind_created_idx
ON job_event (job_id, kind, created_at DESC, id DESC);
CREATE OR REPLACE TRIGGER deletion_protection__tgr
BEFORE DELETE ON job FOR EACH STATEMENT
WHEN ((current_setting('rules.deletion_protection', true) IN ('on', '')) OR
(current_setting('rules.deletion_protection', true) IS NULL))
EXECUTE PROCEDURE raise_deletion_protection();
COMMIT;
If the transaction fails, rerun the whole block.
Step 4 — Audit log: columns, reserved range and defaults
CREATE SEQUENCE IF NOT EXISTS audit_log_seq_seq;
ALTER TABLE audit_log ADD COLUMN IF NOT EXISTS seq bigint;
ALTER TABLE audit_log ADD COLUMN IF NOT EXISTS xid xid8 NOT NULL DEFAULT '0'::xid8;
ALTER SEQUENCE audit_log_seq_seq OWNED BY audit_log.seq;
-- Reserve the low range for the history: new rows start above the backfill.
-- Only on the first application: a sequence that already left 1 is reserved
-- by the operator or in use, and a re-run must not move it.
SELECT setval('audit_log_seq_seq',
(SELECT count(*) + 1000000 FROM audit_log),
false)
WHERE (SELECT last_value FROM audit_log_seq_seq) = 1
AND NOT (SELECT is_called FROM audit_log_seq_seq);
ALTER TABLE audit_log ALTER COLUMN seq SET DEFAULT nextval('audit_log_seq_seq');
ALTER TABLE audit_log ALTER COLUMN xid SET DEFAULT pg_current_xact_id();
Step 5 — Audit log: backfill
Set work_mem high enough for the sort to stay in memory; measure the first
run and adjust if needed.
UPDATE audit_log
SET seq = ordered.rn
FROM (SELECT id, row_number() OVER (ORDER BY created_at, id) AS rn
FROM audit_log
WHERE seq IS NULL) ordered
WHERE audit_log.id = ordered.id;
Step 6 — Audit log: seq and source NOT NULL without a long
ACCESS EXCLUSIVE scan
ALTER TABLE audit_log ADD CONSTRAINT audit_log__seq__not_null
CHECK (seq IS NOT NULL) NOT VALID;
ALTER TABLE audit_log VALIDATE CONSTRAINT audit_log__seq__not_null;
ALTER TABLE audit_log ALTER COLUMN seq SET NOT NULL;
ALTER TABLE audit_log DROP CONSTRAINT audit_log__seq__not_null;
UPDATE audit_log SET source = 'backend' WHERE source IS NULL;
ALTER TABLE audit_log ADD CONSTRAINT audit_log__source__not_null
CHECK (source IS NOT NULL) NOT VALID;
ALTER TABLE audit_log VALIDATE CONSTRAINT audit_log__source__not_null;
ALTER TABLE audit_log ALTER COLUMN source SET NOT NULL;
ALTER TABLE audit_log DROP CONSTRAINT audit_log__source__not_null;
Step 7 — Audit log: cursor index and cursor table
CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS audit_log__xid__seq__idx
ON audit_log (xid, seq);
CREATE TABLE IF NOT EXISTS audit_log_consumer_cursor (
consumer smallint NOT NULL,
xid xid8 NOT NULL,
seq bigint NOT NULL,
updated_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (consumer));
Step 8 — Jobs: exclusive and dedupe columns and indexes
Run in autocommit; CREATE INDEX CONCURRENTLY cannot run inside a transaction.
ALTER TABLE job ADD COLUMN IF NOT EXISTS exclusive boolean NOT NULL DEFAULT false;
ALTER TABLE job ADD COLUMN IF NOT EXISTS dedupe boolean NOT NULL DEFAULT false;
CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS job__live_exclusive__uniq
ON job (tenant, name, queue, label)
WHERE exclusive AND status IN ('new', 'scheduled', 'running', 'retry');
CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS job__new_dedupe__uniq
ON job (tenant, name, queue, label)
WHERE dedupe AND status = 'new';
DROP INDEX IF EXISTS job__scheduled__idx;
CREATE INDEX IF NOT EXISTS job__scheduled__idx
ON job (tenant, modified_at)
WHERE status = 'scheduled';
DROP INDEX IF EXISTS job__name_label__idx;
Step 9 — Gate before deploying
If an index is INVALID, drop and rebuild it before continuing.
SELECT indexname, indexdef FROM pg_indexes
WHERE indexname IN ('job__live_exclusive__uniq',
'job__new_dedupe__uniq',
'job__scheduled__idx',
'audit_log__xid__seq__idx');
SELECT indexrelid::regclass AS index, indisvalid, indisunique
FROM pg_index
WHERE indexrelid IN ('job__live_exclusive__uniq'::regclass,
'job__new_dedupe__uniq'::regclass,
'audit_log__xid__seq__idx'::regclass);
SELECT column_name, is_nullable
FROM information_schema.columns
WHERE table_name = 'audit_log'
AND column_name IN ('seq', 'xid', 'source')
ORDER BY 1;
SELECT to_regclass('public.job') IS NOT NULL AS job,
to_regclass('public.job_event') IS NOT NULL AS job_event,
to_regclass('public.audit_log_consumer_cursor') IS NOT NULL AS cursor;
SELECT indexname FROM pg_indexes
WHERE tablename IN ('job', 'job_event')
AND indexname IN ('job__dispatcher__idx', 'job__orphan__idx',
'job__profile__idx', 'job__resource__idx',
'job__expires__idx', 'job__retention__idx',
'job__scheduled__idx', 'job_event__job_kind_created_idx')
ORDER BY 1;
SELECT column_name, is_nullable
FROM information_schema.columns
WHERE table_name = 'job'
AND column_name IN ('exclusive', 'dedupe')
ORDER BY 1;
SELECT tgname FROM pg_trigger
WHERE tgrelid = to_regclass('public.job')
AND tgname = 'deletion_protection__tgr'
AND NOT tgisinternal;
Pass only if all three tables, eight indexes and the trigger exist; seq,
xid, source, exclusive and dedupe are NOT NULL; and the three unique
indexes are valid and unique. Confirm job__scheduled__idx uses
(tenant, modified_at). Do not deploy if any check fails.
1.5 File data storage object foreign key — online steps
Build the schema of 0159-add-file-data-storage-object-id online, while
2.19 still serves traffic: the storage_object_id column, one index per
partition, the parent index and the foreign key. The backfill is a separate
step in phase 2, inside the coordinated stop. The migration row was
pre-registered in section 1.1, so the boot migration skips this work.
Session for this block: plain SQL; the online ALTERs set their own
SET LOCAL lock_timeout = '3s' inside their transactions.
Step 1 — Column
BEGIN;
SET LOCAL lock_timeout = '3s';
ALTER TABLE file_data
ADD COLUMN IF NOT EXISTS storage_object_id uuid NULL;
COMMIT;
The change is catalog-only (a nullable column without a default), so it takes
ACCESS EXCLUSIVE for an instant. If it fails with lock timeout, wait for
a quieter moment and retry: it did not block anything.
Step 2 — One index per partition, concurrently
CREATE INDEX CONCURRENTLY file_data_00__storage_object_id__idx
ON file_data_00 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_01__storage_object_id__idx
ON file_data_01 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_02__storage_object_id__idx
ON file_data_02 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_03__storage_object_id__idx
ON file_data_03 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_04__storage_object_id__idx
ON file_data_04 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_05__storage_object_id__idx
ON file_data_05 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_06__storage_object_id__idx
ON file_data_06 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_07__storage_object_id__idx
ON file_data_07 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_08__storage_object_id__idx
ON file_data_08 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_09__storage_object_id__idx
ON file_data_09 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_10__storage_object_id__idx
ON file_data_10 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_11__storage_object_id__idx
ON file_data_11 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_12__storage_object_id__idx
ON file_data_12 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_13__storage_object_id__idx
ON file_data_13 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_14__storage_object_id__idx
ON file_data_14 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_15__storage_object_id__idx
ON file_data_15 (storage_object_id);
Run each statement in autocommit: CREATE INDEX CONCURRENTLY cannot run
inside a transaction block. An interrupted build leaves the index
INVALID: drop it and retry.
SELECT c.relname, i.indisvalid
FROM pg_index AS i
JOIN pg_class AS c ON c.oid = i.indexrelid
WHERE c.relname LIKE 'file_data_%storage_object_id%'
ORDER BY c.relname;
Step 3 — Parent index and partition attachment
CREATE INDEX file_data__storage_object_id__idx
ON ONLY file_data (storage_object_id);
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_00__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_01__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_02__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_03__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_04__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_05__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_06__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_07__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_08__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_09__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_10__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_11__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_12__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_13__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_14__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_15__storage_object_id__idx;
The step creates the index unconditionally: if an index with that name already exists, drop it and rebuild it. Verify the result before continuing:
SELECT i.indisvalid, i.indpred IS NULL AS plain, i.indexprs IS NULL AS no_expression
FROM pg_index AS i
WHERE i.indexrelid = 'file_data__storage_object_id__idx'::regclass;
-- t | t | t
SELECT count(*) FROM pg_inherits
WHERE inhparent = 'file_data__storage_object_id__idx'::regclass;
-- 16
Step 4 — Foreign key, online
BEGIN;
SET LOCAL lock_timeout = '3s';
SET LOCAL statement_timeout = '10min'; -- measure first; see below
ALTER TABLE file_data
ADD CONSTRAINT file_data_storage_object_id_fkey
FOREIGN KEY (storage_object_id) REFERENCES storage_object(id)
ON DELETE NO ACTION DEFERRABLE INITIALLY IMMEDIATE;
COMMIT;
The column is NULL until the backfill, but PostgreSQL still scans the table
(or the index built in the previous steps) to prove there are no violations.
The SHARE ROW EXCLUSIVE lock blocks writes for the whole scan, not only
while the lock is acquired: lock_timeout bounds the wait for the lock and
statement_timeout bounds the scan. A timeout aborts the ALTER, releases
the writes and lets you retry in a quieter window; measure the scan in
staging and set the budget accordingly. Retry on lock timeout.
Creating it here, before the backfill, keeps the full validation scan out of the stopped window: the backfill validates each row it writes.
INITIALLY IMMEDIATE is required. The storage GC catches the violation
inside the chunk transaction and retries the chunk; an INITIALLY DEFERRED
check fires at commit, outside that transaction, and the GC would never
move past a blocked chunk.
Step 5 — Temporary index for pending backfill rows (online)
Build this partial index while 2.19 serves traffic, before the stop. It covers only rows that still need migration. Each successful batch removes its rows from the index, so later batches do not scan completed rows. Drop the parent index after the backfill.
Run each CREATE INDEX CONCURRENTLY in autocommit:
CREATE INDEX CONCURRENTLY file_data_00__storage_ref_pending__idx
ON file_data_00 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_01__storage_ref_pending__idx
ON file_data_01 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_02__storage_ref_pending__idx
ON file_data_02 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_03__storage_ref_pending__idx
ON file_data_03 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_04__storage_ref_pending__idx
ON file_data_04 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_05__storage_ref_pending__idx
ON file_data_05 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_06__storage_ref_pending__idx
ON file_data_06 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_07__storage_ref_pending__idx
ON file_data_07 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_08__storage_ref_pending__idx
ON file_data_08 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_09__storage_ref_pending__idx
ON file_data_09 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_10__storage_ref_pending__idx
ON file_data_10 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_11__storage_ref_pending__idx
ON file_data_11 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_12__storage_ref_pending__idx
ON file_data_12 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_13__storage_ref_pending__idx
ON file_data_13 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_14__storage_ref_pending__idx
ON file_data_14 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_15__storage_ref_pending__idx
ON file_data_15 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
Create and attach the matching parent index:
CREATE INDEX file_data__storage_ref_pending__idx ON ONLY file_data (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_00__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_01__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_02__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_03__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_04__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_05__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_06__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_07__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_08__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_09__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_10__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_11__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_12__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_13__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_14__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_15__storage_ref_pending__idx;
SELECT indisvalid, indpred IS NOT NULL AS partial
FROM pg_index
WHERE indexrelid = 'file_data__storage_ref_pending__idx'::regclass;
-- t | t
SELECT count(*) AS attached_partitions
FROM pg_inherits
WHERE inhparent = 'file_data__storage_ref_pending__idx'::regclass;
-- 16
Step 6 — Verify after the backfill
Do not deploy 2.20.0 until every check passes.
-- Column.
SELECT data_type, is_nullable
FROM information_schema.columns
WHERE table_schema = 'public'
AND table_name = 'file_data'
AND column_name = 'storage_object_id';
-- uuid | YES
-- Rows that will be unreadable. Must be 0 after the backfill. The backfill
-- only fills pointers it can resolve, so a non-zero count means pointers
-- that were already corrupt or orphan: inspect them and accept them before
-- continuing. A row pointing at a storage_object with deleted_at set is
-- legitimate and restorable, not unreadable: the backfill fills it on
-- purpose and the restore flow revives the object, so it is not counted.
SELECT count(*) FROM file_data
WHERE backend = 'storage' AND storage_object_id IS NULL;
-- Migrated rows no longer retain the old pointer. Must be 0.
SELECT count(*) FROM file_data
WHERE backend = 'storage' AND storage_object_id IS NOT NULL
AND jsonb_exists(metadata, 'storage-ref-id');
-- Index: valid, plain, with every partition attached.
SELECT i.indisvalid, i.indpred IS NULL AS plain, i.indexprs IS NULL AS no_expression
FROM pg_index AS i
WHERE i.indexrelid = 'file_data__storage_object_id__idx'::regclass;
SELECT count(*) FROM pg_inherits
WHERE inhparent = 'file_data__storage_object_id__idx'::regclass; -- 16
-- Foreign key: deferrable, checked immediately, no action, validated.
SELECT condeferrable, condeferred, confdeltype, convalidated
FROM pg_constraint
WHERE conname = 'file_data_storage_object_id_fkey'
AND conrelid = 'file_data'::regclass;
-- t | f | a | t
Phase 2 — The deploy to 2.20.0
2.1 Full stop and backfill
The backfill runs on every instance, even when no row needs it: an instance
with nothing to backfill updates 0 rows. There is no rolling deploy: the
previous code writes the pointer only to metadata.storage-ref-id and its
storage GC reads the reference from there. Running the backfill while that
code serves traffic would leave the rows it writes without a pointer and
could make the GC delete an object a live row still references.
There is no schema rollback after the deploy: the column is the only reference for the rows 2.20 writes, so a failed upgrade is recovered from the backup.
1. Stop every instance (backend and workers). Keep
:tiered-file-data-storage off until every instance runs 2.20.0.
2. Run the backfill:
WITH candidates AS MATERIALIZED (
SELECT fd.file_id,
fd.id,
CASE WHEN (fd.metadata->>'storage-ref-id') ~*
'^[0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{12}$'
THEN (fd.metadata->>'storage-ref-id')::uuid END AS soid
FROM file_data AS fd
JOIN storage_object AS so
ON so.id = CASE WHEN (fd.metadata->>'storage-ref-id') ~*
'^[0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{12}$'
THEN (fd.metadata->>'storage-ref-id')::uuid END
WHERE fd.backend = 'storage'
AND fd.storage_object_id IS NULL
AND jsonb_exists(fd.metadata, 'storage-ref-id')
AND (fd.metadata->>'storage-ref-id') ~*
'^[0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{12}$'
ORDER BY fd.file_id, fd.id
LIMIT 5000
),
updated AS (
UPDATE file_data AS fd
SET storage_object_id = c.soid,
metadata = fd.metadata - 'storage-ref-id'
FROM candidates AS c
WHERE fd.file_id = c.file_id
AND fd.id = c.id
RETURNING 1
)
SELECT count(*) AS rows_updated FROM updated;
Run the statement repeatedly, committing each batch separately, until
rows_updated is 0. Each batch updates at most 5,000 rows, sets the column and
removes metadata.storage-ref-id atomically. The index built in phase 1.5
keeps the candidate search bounded to rows still pending. The MATERIALIZED
CTE keeps the UUID cast away from values that fail the regex. The join to
storage_object excludes orphan pointers, and the foreign key validates each
written row. Corrupt and orphan pointers remain untouched for investigation;
they must be resolved before the unreadable-row verification can pass.
After all batches finish, remove the temporary index:
DROP INDEX file_data__storage_ref_pending__idx;
The migrated key is not retained as a rollback copy. A rollback to the previous binary after the backfill requires restoring the backup.
3. Verify. Run the Verify block of phase 1.5. All checks must pass.
4. Confirm the pre-registered migration row exists:
SELECT module, step, created_at FROM migrations
WHERE module = 'main' AND step = '0159-add-file-data-storage-object-id';
The row was inserted at the beginning of the runbook. Do not deploy 2.20.0 until the backfill and all verification checks have passed.
5. Start the 2.20.0 instances.
2.2 The watermark goes live
The deploy brings the code that updates updated_at on every mutation
(locks, soft deletes). Phase 3 depends on it. Confirm the watermark moved:
SELECT max(updated_at) FROM file_change;
It has to move on every write from now on. If it does not, do not start phase 3.
Phase 3 — After the upgrade (2.20.0 serving; online)
Phase 3 copies file_change into the new partitioned table in the
background. It spans days: the bulk is one pass over every row (hours to
days), then short delta passes every 1-6 h until the maintenance window.
Everything here runs with the instance serving traffic.
3.1 Create the partitioned table
Create the partitioned table and its 16 partitions:
CREATE TABLE file_change_new (
file_id uuid NOT NULL,
id uuid NOT NULL DEFAULT gen_random_uuid(),
created_at timestamptz NOT NULL DEFAULT now(),
session_id uuid,
revn bigint NOT NULL DEFAULT 0,
data bytea,
changes bytea,
profile_id uuid,
features text[],
label text,
version integer,
created_by text NOT NULL DEFAULT 'system',
updated_at timestamptz DEFAULT now(),
deleted_at timestamptz,
migrations text[],
locked_by uuid,
-- Nullable on purpose, like the migration: the previous version inserts
-- without `kind`, and a rollback to it has to keep writing. The `NOT NULL`
-- comes in a later migration, once the rollback window is closed.
kind smallint,
CONSTRAINT file_change_kind_check CHECK (kind IN (1, 2))
) PARTITION BY HASH (file_id);
-- the columns keep the storage of PostgreSQL by default (migration
-- 0156-reset-pg-storage-defaults dropped the explicit EXTERNAL overrides)
COMMENT ON COLUMN file_change_new.kind IS
'The kind of the row: 1 is a snapshot (it has a label and its data lives in file_data) and 2 is a revision (it has no label and it holds the changes).';
COMMENT ON COLUMN file_change_new.locked_by IS
'Profile ID of user who has locked this version. Only the creator can lock/unlock their own versions. Locked versions cannot be deleted by others.';
-- 16 partitions, the same bounds as file_data. The name of the table is
-- temporary (the old one holds it until the cutover) but the name of the
-- partitions is not: it is the final one, so their indexes are born with
-- the canonical names (renaming a partition does not rename its indexes)
-- and the result is exactly the schema of the migration.
CREATE TABLE file_change_00 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 0);
CREATE TABLE file_change_01 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 1);
CREATE TABLE file_change_02 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 2);
CREATE TABLE file_change_03 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 3);
CREATE TABLE file_change_04 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 4);
CREATE TABLE file_change_05 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 5);
CREATE TABLE file_change_06 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 6);
CREATE TABLE file_change_07 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 7);
CREATE TABLE file_change_08 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 8);
CREATE TABLE file_change_09 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 9);
CREATE TABLE file_change_10 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 10);
CREATE TABLE file_change_11 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 11);
CREATE TABLE file_change_12 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 12);
CREATE TABLE file_change_13 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 13);
CREATE TABLE file_change_14 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 14);
CREATE TABLE file_change_15 PARTITION OF file_change_new
FOR VALUES WITH (MODULUS 16, REMAINDER 15);
-- The primary key includes the partition key (required on a partitioned
-- table)
ALTER TABLE file_change_new ADD PRIMARY KEY (file_id, id);
-- Foreign keys (no CREATE TABLE copies them). They take the canonical
-- name already: they have no index, so they do not collide with the old
-- table and the partitions inherit it.
ALTER TABLE file_change_new
ADD CONSTRAINT file_change_file_id_fkey FOREIGN KEY (file_id)
REFERENCES file(id) DEFERRABLE;
ALTER TABLE file_change_new
ADD CONSTRAINT file_change_locked_by_fkey FOREIGN KEY (locked_by)
REFERENCES profile(id) ON DELETE SET NULL DEFERRABLE;
ALTER TABLE file_change_new
ADD CONSTRAINT file_change_profile_id_fkey FOREIGN KEY (profile_id)
REFERENCES profile(id) ON DELETE SET NULL DEFERRABLE;
SELECT count(*) AS partitions FROM pg_partition_tree('file_change_new') t
WHERE t.isleaf;
-- 16
The new indexes carry a _new suffix because index names are unique per
schema and the old table still holds the canonical ones until the cutover.
The same goes for the primary key of the new table: its index is named
after the constraint. The constraints that have no index (the CHECK and
the foreign keys) are named canonically from the start, because a
constraint name is unique per table, not per schema, so it can live with
the one of the old table; the partitions inherit it when they are created.
The partitions are created with their final name, because their indexes are
named after the partition and not after the table.
3.2 Bulk copy
Copies every row, including the soft deleted ones: the GC manages them the
same way in the new table. kind is computed from label: a row with a
label is a snapshot, a row without one is a revision.
The pointer of the copy lives in the database, not in the terminal: a batch can be interrupted and resumed without losing anything.
CREATE TABLE IF NOT EXISTS file_change_copy_state (
name text PRIMARY KEY,
last_id uuid,
last_at timestamptz
);
-- for a procedure that already created the table without the mark
ALTER TABLE file_change_copy_state
ADD COLUMN IF NOT EXISTS last_at timestamptz;
INSERT INTO file_change_copy_state (name, last_id)
VALUES ('bulk', '00000000-0000-0000-0000-000000000000')
ON CONFLICT (name) DO NOTHING;
-- The mark of the delta passes starts at the bulk: the first pass of phase
-- 3.3 covers the rows changed since here, and `DO NOTHING` keeps a re-run
-- from resetting a mark that the passes already moved.
INSERT INTO file_change_copy_state (name, last_at) VALUES ('delta', now())
ON CONFLICT (name) DO NOTHING;
-- The mark is mandatory: a pass without it copies nothing, in silence. The
-- `DROP ... IF EXISTS` keeps this block re-runnable.
ALTER TABLE file_change_copy_state
ADD CONSTRAINT file_change_copy_state_delta_mark
CHECK (name <> 'delta' OR last_at IS NOT NULL);
Run the create-table block of phase 3.1 first, then this block, then the batch block in a loop until it returns 0.
One batch (idempotent). It always returns one row: the number of rows
it copied, 0 when there is nothing left to copy.
WITH batch AS MATERIALIZED (
SELECT id FROM file_change
WHERE id > (SELECT last_id FROM file_change_copy_state WHERE name = 'bulk')
ORDER BY id
LIMIT 10000
),
copied AS (
INSERT INTO file_change_new (id, file_id, created_at, session_id, revn, data, changes,
profile_id, features, label, version, created_by,
updated_at, deleted_at, migrations, locked_by, kind)
SELECT id, file_id, created_at, session_id, revn, data, changes,
profile_id, features, label, version, created_by,
updated_at, deleted_at, migrations, locked_by,
CASE WHEN label IS NOT NULL THEN 1 ELSE 2 END
FROM file_change
WHERE id IN (SELECT id FROM batch)
ON CONFLICT (file_id, id) DO UPDATE
SET created_at = EXCLUDED.created_at,
session_id = EXCLUDED.session_id,
revn = EXCLUDED.revn,
data = EXCLUDED.data,
changes = EXCLUDED.changes,
profile_id = EXCLUDED.profile_id,
features = EXCLUDED.features,
label = EXCLUDED.label,
version = EXCLUDED.version,
created_by = EXCLUDED.created_by,
updated_at = EXCLUDED.updated_at,
deleted_at = EXCLUDED.deleted_at,
migrations = EXCLUDED.migrations,
locked_by = EXCLUDED.locked_by,
kind = EXCLUDED.kind
WHERE file_change_new.updated_at IS DISTINCT FROM EXCLUDED.updated_at
RETURNING 1
),
-- `moved` advances the pointer only when the batch copied something.
-- PostgreSQL executes a data-modifying CTE exactly once even when the
-- final query does not read its output.
moved AS (
UPDATE file_change_copy_state
SET last_id = (SELECT id FROM batch ORDER BY id DESC LIMIT 1)
WHERE name = 'bulk'
AND EXISTS (SELECT 1 FROM copied)
RETURNING 1
)
SELECT count(*) AS copied FROM copied;
Repeat this statement until it returns 0. The 0 comes from the final
SELECT, not from a missing row: a batch that finds nothing to copy still
returns it, so it is the signal to stop.
- The pointer is a
uuid(random order): the pagination is stable but it does not group by physical position. The total read is the heap, in a scattered order. - Estimated duration: time the first batches and extrapolate. The bulk reads the heap plus the live TOAST once (phase 1) and writes about the live data. Batches are rows divided by batch size.
- Pace the batches yourself (a pause between statements) if the streaming replica falls behind or the backend sees latency. Every batch detoastes its share of the TOAST: the cost is I/O, not CPU.
- Watch: replica lag,
pg_stat_activity(locks between the bulk and autovacuum), the space ofpg_waland of the data dir.
When the bulk is done, check that no row is missing (it is the check that matters, not the total count):
SELECT count(*) AS missing
FROM file_change o
WHERE NOT EXISTS (SELECT 1 FROM file_change_new n
WHERE n.file_id = o.file_id
AND n.id = o.id);
missing is zero when no row of the old table is absent from the new one;
it is the check that matters. The new table can have more rows than the
old one: the GC may have hard deleted a row from the source after the bulk
copied it, and it cleans that row up after the cutover. Rows written during
the bulk are missing until the delta passes pick them up.
3.3 Delta passes
Every pass copies the rows modified since the previous pass. The
watermark lives in file_change_copy_state.last_at (the start of the last
pass; the setup block of phase 3.2 seeds it with the moment of the bulk, so
the first pass covers what changed since then), and the SQL subtracts one
hour of margin, which absorbs the clock drift between the application
(which writes updated_at) and the database (which writes now()) and the
drift between batches. A row that a pass already copied but did not change
again is not rewritten: the WHERE of the ON CONFLICT skips it.
BEGIN;
WITH mark AS (
SELECT last_at - INTERVAL '1 hour' AS since
FROM file_change_copy_state
WHERE name = 'delta'
)
INSERT INTO file_change_new (id, file_id, created_at, session_id, revn, data, changes,
profile_id, features, label, version, created_by,
updated_at, deleted_at, migrations, locked_by, kind)
SELECT id, file_id, created_at, session_id, revn, data, changes,
profile_id, features, label, version, created_by,
updated_at, deleted_at, migrations, locked_by,
CASE WHEN label IS NOT NULL THEN 1 ELSE 2 END
FROM file_change, mark
WHERE updated_at > mark.since
ON CONFLICT (file_id, id) DO UPDATE
SET created_at = EXCLUDED.created_at,
session_id = EXCLUDED.session_id,
revn = EXCLUDED.revn,
data = EXCLUDED.data,
changes = EXCLUDED.changes,
profile_id = EXCLUDED.profile_id,
features = EXCLUDED.features,
label = EXCLUDED.label,
version = EXCLUDED.version,
created_by = EXCLUDED.created_by,
updated_at = EXCLUDED.updated_at,
deleted_at = EXCLUDED.deleted_at,
migrations = EXCLUDED.migrations,
locked_by = EXCLUDED.locked_by,
kind = EXCLUDED.kind
WHERE file_change_new.updated_at IS DISTINCT FROM EXCLUDED.updated_at;
-- The mark is the start of this transaction (`now()`): the rows changed
-- after it are the window of the next pass. Both statements go in the
-- same transaction on purpose; a mark written later would leave a gap.
UPDATE file_change_copy_state SET last_at = now() WHERE name = 'delta';
COMMIT;
- Repeat the transaction every 1-6 h (depending on the activity) until the
pass copies very few rows. With the
updated_atindex of phase 1.3 the pass is an index range over the window, so it takes seconds on a quiet instance; without it, it scans the heap. To see how many rows are left in the window:
SELECT count(*) FROM file_change
WHERE updated_at > (SELECT last_at FROM file_change_copy_state WHERE name = 'delta');
- The objects GC can keep running. Its soft deletes are covered by the
watermark, and its hard deletes leave a row with
deleted_atin the new table that the GC itself cleans up later. The hard delete of a file is blocked while the old table (or the new one) has rows for it, because the foreign key onfile_idisNO ACTION. The GC deletes the pending files in chunks of 100, and one blocked file aborts the transaction of the whole chunk: the chunk is logged aserror on chunk deletionand skipped, its files wait in the trash, and the next daily run retries the same chunk and fails again until the cutover. The other chunks keep making progress. Do not run the cutover at the same time as a big batch of the GC.
Phase 4 — The maintenance window (after the deploy)
4.1 Before the lock (online, still 2.20.0 serving)
- One last delta pass (the block of phase 3.3).
- Build the secondary indexes of the new table. The bulk and the delta
passes only need the primary key (
ON CONFLICT (file_id, id)), so building them here keeps five indexes and their WAL out of every batch. Thedeleted_atindex carriesfile_id(the primary key is(file_id, id)): the objects GC needs thefile_idto lock by pair, and with the index it reads the rows it has to delete without touching the table.
-- Secondary indexes: the canonical names, with the _new suffix. There is no
-- index for the system snapshots: the old one filtered them by
-- `data IS NOT NULL`, and with the payload living in `file_data` the column
-- no longer tells snapshots apart. The new one could only filter them by
-- `deleted_at IS NULL`, and the system snapshots always carry a deletion
-- date in the future, so it would never match a row.
CREATE INDEX file_change_new__deleted_at__idx
ON file_change_new (deleted_at, file_id, id) WHERE deleted_at IS NOT NULL;
CREATE INDEX file_change_new__file_id__revn__idx
ON file_change_new (file_id, revn);
CREATE INDEX file_change_new__locked_by__idx
ON file_change_new (locked_by) WHERE locked_by IS NOT NULL;
CREATE INDEX file_change_new__profile_id__idx
ON file_change_new (profile_id) WHERE profile_id IS NOT NULL;
CREATE INDEX file_change_new__user_snapshots__idx
ON file_change_new (file_id)
WHERE label IS NOT NULL AND created_by = 'user' AND deleted_at IS NULL;
- Prepare and rehearse the cutover block in staging.
- Pick a low activity window: a freeze of the writes to
file_changeof a couple of minutes (the final delta inside the lock copies only the rows changed since the last delta pass, so it is quick; the saves of files during the freeze fail and the user retries, and the rest of the database keeps working). The window has to cover the whole cutover, with the backend and the workers stopped and started again (see 4.2).
4.2 The cutover (one transaction, a brief exclusive lock)
- Stop the backend and the workers before the transaction, and start
them again after it. The pool keeps server-side prepared statements,
and a prepared statement holds the OID of the table it was prepared
with: renaming the table does not change that OID. A connection that had
already run
insert into file_changefive times (the default threshold of the driver) would keep writing tofile_change_legacyafter the cutover, and the pool recycles its connections every 30 minutes, so the window of silent writes is long. - Run the transaction, the
ANALYZE, and the statement that records the instant of the cutover (all of it in the block below). - Start the backend and the workers again.
BEGIN;
LOCK TABLE file_change IN ACCESS EXCLUSIVE MODE;
-- The final delta: the rows modified since the last pass (its mark, minus
-- the hour of margin). The mark is the one the last pass left.
WITH mark AS (
SELECT last_at - INTERVAL '1 hour' AS since
FROM file_change_copy_state
WHERE name = 'delta'
)
INSERT INTO file_change_new (id, file_id, created_at, session_id, revn, data, changes,
profile_id, features, label, version, created_by,
updated_at, deleted_at, migrations, locked_by, kind)
SELECT id, file_id, created_at, session_id, revn, data, changes,
profile_id, features, label, version, created_by,
updated_at, deleted_at, migrations, locked_by,
CASE WHEN label IS NOT NULL THEN 1 ELSE 2 END
FROM file_change, mark
WHERE updated_at > mark.since
ON CONFLICT (file_id, id) DO UPDATE
SET created_at = EXCLUDED.created_at,
session_id = EXCLUDED.session_id,
revn = EXCLUDED.revn,
data = EXCLUDED.data,
changes = EXCLUDED.changes,
profile_id = EXCLUDED.profile_id,
features = EXCLUDED.features,
label = EXCLUDED.label,
version = EXCLUDED.version,
created_by = EXCLUDED.created_by,
updated_at = EXCLUDED.updated_at,
deleted_at = EXCLUDED.deleted_at,
migrations = EXCLUDED.migrations,
locked_by = EXCLUDED.locked_by,
kind = EXCLUDED.kind
WHERE file_change_new.updated_at IS DISTINCT FROM EXCLUDED.updated_at;
-- Check inside the lock, and abort if it fails: no row of the old table
-- may be missing in the new one. The new table can carry extra rows that
-- the GC hard deleted from the source; it cleans them up after the
-- cutover.
DO $$
DECLARE missing bigint;
BEGIN
SELECT count(*) INTO missing
FROM file_change o
WHERE NOT EXISTS (SELECT 1 FROM file_change_new n
WHERE n.file_id = o.file_id
AND n.id = o.id);
IF missing > 0 THEN
RAISE EXCEPTION 'file_change: % rows of the old table are missing in file_change_new; cutover aborted', missing;
END IF;
END $$;
-- The old table frees the canonical names (the same names the migration
-- 0158 uses, with the _legacy suffix)
ALTER TABLE file_change RENAME CONSTRAINT file_change_pkey
TO file_change_pkey_legacy;
ALTER TABLE file_change RENAME CONSTRAINT file_change_file_id_fkey
TO file_change_file_id_fkey_legacy;
ALTER TABLE file_change RENAME CONSTRAINT file_change_locked_by_fkey
TO file_change_locked_by_fkey_legacy;
ALTER TABLE file_change RENAME CONSTRAINT file_change_profile_id_fkey
TO file_change_profile_id_fkey_legacy;
ALTER INDEX file_change__data_ref_id__idx
RENAME TO file_change__data_ref_id__idx_legacy;
ALTER INDEX file_change__deleted_at__idx
RENAME TO file_change__deleted_at__idx_legacy;
ALTER INDEX file_change__file_id__revn__idx
RENAME TO file_change__file_id__revn__idx_legacy;
ALTER INDEX file_change__locked_by__idx
RENAME TO file_change__locked_by__idx_legacy;
ALTER INDEX file_change__profile_id__idx
RENAME TO file_change__profile_id__idx_legacy;
ALTER INDEX file_change__system_snapshots__idx
RENAME TO file_change__system_snapshots__idx_legacy;
ALTER INDEX IF EXISTS file_change__user_snapshots__idx
RENAME TO file_change__user_snapshots__idx_legacy;
ALTER INDEX IF EXISTS file_change__updated_at__idx
RENAME TO file_change__updated_at__idx_legacy;
-- And the new table takes them. Only the primary key needs the rename: the
-- `CHECK` and the foreign keys already carry the canonical name (they have
-- no index, so they could take it from the start), and the partitions
-- inherited it.
ALTER TABLE file_change_new RENAME CONSTRAINT file_change_new_pkey
TO file_change_pkey;
ALTER INDEX file_change_new__deleted_at__idx
RENAME TO file_change__deleted_at__idx;
ALTER INDEX file_change_new__file_id__revn__idx
RENAME TO file_change__file_id__revn__idx;
ALTER INDEX file_change_new__locked_by__idx
RENAME TO file_change__locked_by__idx;
ALTER INDEX file_change_new__profile_id__idx
RENAME TO file_change__profile_id__idx;
ALTER INDEX file_change_new__user_snapshots__idx
RENAME TO file_change__user_snapshots__idx;
-- And the tables are swapped. The partitions and their indexes keep the
-- names they were born with (they are named after the partition, not after
-- the table), so there is nothing else to rename.
ALTER TABLE file_change RENAME TO file_change_legacy;
ALTER TABLE file_change_new RENAME TO file_change;
-- The foreign key of the legacy table goes away: it is `NO ACTION`, so
-- while the legacy table has rows for a file, the objects GC cannot delete
-- that file (it deletes the rows of the file first, and the legacy ones are
-- not its business). The legacy table is frozen: nothing writes to it again.
ALTER TABLE file_change_legacy
DROP CONSTRAINT file_change_file_id_fkey_legacy;
COMMIT;
ANALYZE file_change;
-- The instant of the cutover, for the checks of phase 5. It goes after
-- the `COMMIT`: inside the transaction `now()` is the start of it (the
-- moment of the `LOCK`), and the rows the final delta copies could pass
-- for writes happened after the cutover.
INSERT INTO file_change_copy_state (name, last_at) VALUES ('cutover', now())
ON CONFLICT (name) DO UPDATE SET last_at = excluded.last_at;
file_change_legacykeeps its data and its indexes (with the_legacysuffix): a frozen copy of the previous layout. Its foreign key onfile_idis dropped, so the objects GC can delete the files again.- The writes to
file_changeduring the lock fail: they are seconds-minutes, the affected saves are retried. - The objects GC and the four RPC commands keep working: same table name and same index names after the rename.
- Once the backend is up again, check that the saves go to the partitioned table and that nothing went to the table that is going away:
-- the rows written after the cutover: this one grows with every save
SELECT count(*) FROM file_change
WHERE updated_at > (SELECT last_at FROM file_change_copy_state
WHERE name = 'cutover');
-- the rows that went to the legacy table: this one has to stay at 0
SELECT count(*) FROM file_change_legacy
WHERE updated_at > (SELECT last_at FROM file_change_copy_state
WHERE name = 'cutover');
A number above zero in the second query means that some connection was still writing with a prepared statement of the old table: those rows are lost when the legacy table is dropped, so they have to be copied over before that.
4.3 The storage backfill is not part of this window
The backfill always runs in the phase 2 stop, before the new instances start. After the deploy, any row that still carries the pointer is unreadable by the new code, so this is not a step to defer to the phase 4 window.
Phase 5 — After the cutover (online)
5.1 Partition pruning and access paths
Check that the access paths prune to one partition (when the planner chooses an index scan):
EXPLAIN SELECT * FROM file_change
WHERE file_id = '00000000-0000-0000-0000-000000000000'
AND id = '00000000-0000-0000-0000-000000000000';
It has to show a single partition (file_change_NN), not an Append of
16.
5.2 Schema parity with the migration, and ownership
Every row of this query has to show got = expected:
SELECT 'columns' AS what, count(*) AS got, 17 AS expected
FROM pg_attribute
WHERE attrelid = 'file_change'::regclass AND attnum > 0 AND NOT attisdropped
UNION ALL
SELECT 'partitions', count(*), 16
FROM pg_partition_tree('file_change') t WHERE t.isleaf
UNION ALL
SELECT 'primary key', count(*), 1
FROM pg_constraint
WHERE conrelid = 'file_change'::regclass AND contype = 'p'
UNION ALL
SELECT 'foreign keys', count(*), 3
FROM pg_constraint
WHERE conrelid = 'file_change'::regclass AND contype = 'f'
UNION ALL
SELECT 'check constraints', count(*), 1
FROM pg_constraint
WHERE conrelid = 'file_change'::regclass AND contype = 'c'
UNION ALL
SELECT 'indexes', count(*), 6
FROM pg_indexes
WHERE tablename = 'file_change' AND schemaname = current_schema()
UNION ALL
SELECT 'partitions out of name', count(*), 0
FROM pg_class
WHERE relkind = 'r' AND relname ~ '^file_change_new[0-9]'
UNION ALL
SELECT 'child indexes out of name', count(*), 0
FROM pg_class c
JOIN pg_index i ON i.indexrelid = c.oid
JOIN pg_class p ON p.oid = i.indrelid
WHERE p.relname ~ '^file_change_[0-9]{2}$'
AND c.relname NOT LIKE p.relname || '_%'
UNION ALL
SELECT 'partition constraints out of name', count(*), 0
FROM pg_constraint c
JOIN pg_class t ON t.oid = c.conrelid
WHERE t.relname ~ '^file_change_[0-9]{2}$'
AND c.contype <> 'n'
AND c.conname LIKE 'file_change_new%';
The NOT NULL constraints are excluded from the last check (contype <> 'n'): their names are not stable across server versions, so they are not
part of the parity. The CHECK, the primary key and the foreign keys are.
And the names, to compare them with the migration:
SELECT a.attname AS column
FROM pg_attribute a
WHERE a.attrelid = 'file_change'::regclass AND a.attnum > 0
AND NOT a.attisdropped
ORDER BY a.attnum;
SELECT indexname FROM pg_indexes
WHERE tablename = 'file_change' AND schemaname = current_schema()
ORDER BY indexname;
-- the partitions, with their name
SELECT c.relname AS partition
FROM pg_class c
JOIN pg_partition_tree('file_change') t ON t.relid = c.oid
WHERE t.isleaf
ORDER BY c.relname;
-- the indexes of the partitions: their name starts with the partition
SELECT c.relname AS child_index
FROM pg_class c
JOIN pg_index i ON i.indexrelid = c.oid
JOIN pg_class p ON p.oid = i.indrelid
WHERE p.relname ~ '^file_change_[0-9]{2}$'
ORDER BY c.relname;
-- every table, partition and index of the family shares the owner of the
-- table
SELECT count(*) AS out_of_owner
FROM pg_class c
WHERE c.relname LIKE 'file_change%'
AND c.relkind IN ('r', 'p', 'i')
AND pg_get_userbyid(c.relowner) <>
(SELECT tableowner FROM pg_tables
WHERE schemaname = current_schema() AND tablename = 'file_change');
The owner check has to return 0. If it does not, fix the ownership
before opening the service (re-own the table, the 16 partitions and
their indexes) and re-apply any grants the old table had.
5.3 Partition-wise join and autovacuum
- The partition-wise join is already on: the pool of the backend sets
enable_partitionwise_join = onon every connection (it is in the code, there is nothing to change in the database role). - Autovacuum: each of the 16 partitions has its own TOAST (check their
sizes with
pg_total_relation_sizeper partition); withautovacuum_vacuum_scale_factor = 0.2the threshold is reached earlier and in steps. Nothing to change, just watchlast_autovacuumof the partitions the first week. - The aggregations that do not filter by
file_id(the team snapshot quota and the telemetry counts) read all 16 partitions. To know their cost on your instance, time them before and after withEXPLAIN (ANALYZE, BUFFERS)and compare. The team quota is already fixed in the code: it resolves thefile_idof the team first, which lets the planner prune the partitions when it chooses an index scan (with a sequential scan it reads all of them).
5.4 Drop the helper table and, days later, the old table
DROP TABLE IF EXISTS file_change_copy_state;
DROP TABLE file_change_legacy; -- out of hours: it generates WAL by
-- dropping the pages, let it roll
That is where the dead space comes back: the database shrinks by about the size of the old table (phase 1). Copy the rows from the legacy table first if phase 4 showed any (see the check after the cutover); they are lost when it is dropped.
5.5 Post-deploy storage checks
SELECT conname, condeferrable, condeferred, confdeltype, convalidated
FROM pg_constraint
WHERE conname = 'file_data_storage_object_id_fkey'
AND conrelid = 'file_data'::regclass;
SELECT count(*) FROM file_data
WHERE backend = 'storage' AND storage_object_id IS NULL;
-- Run the foreign-key check; ROLLBACK keeps the object.
BEGIN;
EXPLAIN (ANALYZE, BUFFERS)
DELETE FROM storage_object WHERE id = '<uuid of a deleted object with no references>';
ROLLBACK;
5.6 After every instance runs 2.20.0: turn on the new GC
Once every instance runs this version and the new consumers have replayed
the unarchived buffer, remove disable-audit-log-gc and roll the workers to
restore the default GC schedule. The GC deletes the rows the consumers have
passed and applies the 7-day retention of the telemetry shadows. The GC has
two cursor paths: the full rows up to the oldest cursor of the consumers
that read them, and the shadows up to the cursor 0; a slow telemetry pass
does not retain the full rows. Remove the old archiver flag setting; the new
version does not use it.
The seven-day window limits local retention, not sending. The telemetry consumer can send older shadows if it reads them before the GC deletes them.
If a consumer registers while a cursor-based GC batch is in progress, it
can miss up to one batch per concurrent GC run that already read the previous
cursor limit. The GC reads the cursors again for each batch but does not lock
out registration. The 7-day shadow retention deletes
independently of cursors. For a complete replay of the buffer, set
disable-audit-log-gc on all workers sharing the database and remove any
enable-audit-log-gc token. Restart the workers, verify the effective flag
in each startup log, and wait for all queued or running GC jobs to finish
before enabling the new consumer. Do not invoke the
GC manually during the pause. Confirm the consumer has registered and
replayed the buffer before removing disable-audit-log-gc and rolling the
workers again. This cannot recover rows the GC already deleted.
To retire the telemetry consumer, turn :telemetry off on every instance,
let the GC run once with its cursor frozen and then delete the cursor
(DELETE FROM audit_log_consumer_cursor WHERE consumer = 0;). The statement
deletes no shadow by itself: the copies the consumer had not read wait for the
retention window. Turning the flag back on registers the cursor at (0, 0)
again and replays whatever the buffer still holds.
Rollback
- Before the phase 2 deploy: phase 1 may have applied out-of-band schema changes, but the new code has not started. Keep 2.19 serving; assess and reverse the completed phase 1 changes before removing the pre-registration rows or switching to the automatic migration path.
- Partition cutover:
file_change_legacyis a frozen copy of the previous layout, kept for days. Rolling back before dropping it means pointing the code at the legacy layout, which is not supported by the release; treat theDROP TABLE file_change_legacyas the point of no return and schedule it deliberately. - Consumer generalization: the migration keeps the legacy
archived_atmark and its indexes for one release; the two versions can coexist during the rollout. Rolling the binaries back is possible whiledisable-audit-log-gcis set and the consumers are idle. Keep that setting until you are done, and call it a rollback only within the rollout.
Risks
| Risk | Mitigation |
|---|---|
| Massive WAL during the bulk copy (proportional to the copy) | Throttle between batches; watch pg_wal, the replication slots and the replica lag |
| Latency seen by the application during the bulk | Throttle; low traffic hours; the bulk is I/O and does not block writes |
updated_at not reliable in rows changed before the deploy |
The deploy goes before the bulk (phase 2.2); the delta passes cover from the start of the bulk |
| Lock/unlock/soft delete during the passes | Covered by the watermark; the cutover includes the final delta inside the lock |
| Hard delete of the GC between passes | The row is left with deleted_at in the new table; the GC cleans it up later |
Orphan rows if a file is deleted halfway |
The foreign key on file_id is NO ACTION: the GC deletes the rows of the file before the file, so a chunk that fails leaves the file in the trash and the next run deletes it |
| The files in the trash cannot be deleted while the legacy table has rows for them | The foreign key of the legacy table is dropped at the cutover; until then the files wait in the trash |
| Saves failing during the lock of the cutover | Freeze of minutes in a low traffic window; manual retries |
| Planner regression after the cutover | Immediate ANALYZE + the pruning EXPLAIN; the pool already sets the GUC |
| Clock drift between the application and the database | The delta window starts an hour before the bulk; the margin absorbs the drift |
CONCURRENTLY build interrupted, index left INVALID |
Drop it and rebuild it; IF NOT EXISTS matches by name and would accept the broken index: the gates (phase 1.4 step 9 and phase 1.5 verify) are mandatory |
| Running the backfill while 2.19 serves | Do not: phase 2 stops every instance before the backfill |
Success metrics
- After the cutover:
file_change(partitioned) holds at most thepg_table_sizeof the old table (phase 1). - After the
DROPof the old table: the database shrinks by about the size that table had. - The access paths that filter by
file_id(the lookup of a snapshot, the quota of a file and the lock of the GC by pair) prune to one partition when the planner chooses an index scan (EXPLAIN). The quota of the team and the discovery of the GC do not filter by a single file: they read all the 16 partitions. - No visible change for the user.