penpot/operations/2.20.0-unified.md
Andrey Antukh 90a06e982b
♻️ Rework the audit-log consumers and the jobs substrate (#12701)
* ♻️ Rework audit-log consumers and jobs substrate

Fold the implementation, follow-up fixes, migrations and tests into one

code commit.

AI-assisted-by: gpt-6-luna

* 📚 Consolidate 2.20.0 upgrade runbooks

Keep the operational upgrade procedure and its supporting runbooks in a

separate documentation commit.

AI-assisted-by: gpt-6-luna

* ♻️ Remove the audit-log flag and derive persistence at boot

Persistence now follows the consumer flags instead of a separate switch.
app.config computes cf/audit-persist? once at boot from :nexus and
:admin-console, and every write site reads that value, so request paths
no longer check flags per event. With no consumer enabled no full rows
are written; telemetry shadows and webhook deliveries keep their own
flags. The frontend wire signal stays :audit-log as a derived value.

Update the 2.20 runbooks, memories and the devenv flags to match, and
migrate the tests that used to toggle :audit-log directly.

AI-assisted-by: hy4-flash-free

* 🐛 Accept error-reporting batches and precompute sink flags

push-audit-events now accepts crash batches with :error-reporting alone, so
the frontend signal from get-environment-data matches a real destination.
The reporters receive unhandled-exception events without storing any row.

All per-event sink checks read booleans computed once at boot: webhooks,
telemetry and error reporting join audit-persist? in app.config. The test
fixture and with-audit-config rebind all four values together, and the
tests that toggled those flags directly were migrated.

Address F2 of the review of the previous commit. F1 stays deferred to a
separate plan.

AI-assisted-by: hy4-flash-free

* 💄 Match the layout cljfmt expects in the flags set

The first comment of the varia set sat on its own line after the opening
delimiter, and cljfmt joins such a comment to the delimiter line, so the
Common Tests lint step failed on common/src/app/common/flags.cljc. Keep
the comment on the opening line, the same shape the login and email sets
already use.

Note in mem:scripts/ci that the backend lint lints ../common/src but does
not format-check it, so common has to be passed to scripts/ci as well.

AI-assisted-by: deepseek-v4-flash
2026-11-20 11:12:00 +01:00

65 KiB

2.20.0 — Unified upgrade runbook

Single, self-contained procedure for upgrading an instance to the release that ships:

Migrations apply in this order at boot when they are not registered in the migrations table: 0154-add-job-table, 0158-partition-file-change, 0159-add-file-data-storage-object-id, then 0159-consumer-generalization.

This document is self-contained and manual. Every block is plain SQL, paste-able into any client that speaks SQL; there are no shell scripts and no psql meta-commands. It runs on PostgreSQL 15 or newer. It does not replace operations/2.20.0-file-change-partition.md, operations/2.20.0-file-data-storage-object-id.md or operations/2.20.0-consumer-generalization.md: they stay as the reference for their individual procedure. Where this document and those disagree, this document wins for the unified execution; they win for the details of one area only.

When each phase runs

Phase When Service
1 Before the upgrade; pre-flight and online preparation 2.19.0 serving
2 The deploy to 2.20.0 Full stop for the storage backfill
3 After the upgrade, 2.20.0 serving; phase 3.3 every 1-6 h for days Online
4 A scheduled maintenance window for the file_change partition cutover Backend and workers stopped
5 After the cutover Online

This runbook is a complete manual procedure. Follow every phase in order, regardless of database size; do not skip steps based on how much data a table contains. Phase 2 is a coordinated full stop of the backend and workers for the storage backfill during the deploy. Phase 4 is a separate scheduled maintenance window for the file_change partition cutover after the deploy. The backfill runs on every instance, even when it updates no rows.

Ordering constraints (do not rearrange)

  • The migration steps are pre-registered at the beginning: the 2.20.0 migration runner skips them, and the manual procedure below is the only path that applies their schema changes and backfills.
  • Phases 3 and 4 need the new code running: it moves updated_at on every write, the watermark of the copy passes. The deploy (phase 2) goes before phase 3.
  • The file_data backfill always runs in phase 2, while every old instance is stopped and before the new instances start: the old code writes the pointer only to metadata.storage-ref-id, the new code reads only the column. A rolling deploy in between would mix both.
  • The 0158 cutover (phase 4) takes minutes in a low-traffic window. Saves that fall inside fail and the user retries.
  • After every instance runs 2.20.0, turn on :audit-log-gc (phase 5.6). Until then the old and new versions coexist and the new GC must stay off.

Phase 1 — Before the upgrade (2.19.0 serving; online)

1.1 Register every migration step first

Run these inserts before any other step in this runbook. They prevent the 2.20.0 migration runner from starting these jobs automatically. The runbook then owns all schema changes and backfills. Do not deploy 2.20.0 until every manual step and verification in this document is complete.

INSERT INTO migrations (module, step)
VALUES ('main', '0158-partition-file-change')
ON CONFLICT (module, step) DO NOTHING;

INSERT INTO migrations (module, step)
VALUES ('main', '0154-add-job-table')
ON CONFLICT (module, step) DO NOTHING;

INSERT INTO migrations (module, step)
VALUES ('main', '0159-consumer-generalization')
ON CONFLICT (module, step) DO NOTHING;

INSERT INTO migrations (module, step)
VALUES ('main', '0159-add-file-data-storage-object-id')
ON CONFLICT (module, step) DO NOTHING;

These rows are pre-registration for this manual procedure, not permission to deploy early. If the procedure is interrupted, keep 2.20.0 stopped and resume the manual work before deploying.

1.2 Pre-flight sizing

-- total size of the database and of file_change (heap + TOAST + indexes)
SELECT pg_size_pretty(pg_database_size(current_database())) AS database;
SELECT pg_size_pretty(pg_total_relation_size('file_change')) AS file_change_total,
       pg_size_pretty(pg_table_size('file_change'))         AS heap_and_toast,
       pg_size_pretty(pg_indexes_size('file_change'))       AS indexes;
SELECT count(*) AS rows FROM file_change;

-- audit_log volume: estimate the time for the out-of-band index builds
SELECT pg_size_pretty(pg_total_relation_size('audit_log')) AS audit_log;
SELECT count(*) FROM audit_log;

-- backfill cost of file_change is proportional to pg_table_size,
-- not to the row count: time the first batches in phase 3 and extrapolate

These measurements help estimate the time and disk space the procedure will need. They do not change which steps to run: this runbook always applies the full file_change, audit_log and file_data procedures.

After pre-flight, run subsections 1.3, 1.4 and 1.5 in order while 2.19 serves traffic. Do not stop the web instances. Subsection 1.4 stops the legacy archiver and its GC by flag.

1.3 Partition preparation

1.3.1 Add the kind column to the current table. Until the cutover, file_change is the current table, and the new code writes kind on every insert: without the column the backend does not start.

ALTER TABLE file_change ADD COLUMN IF NOT EXISTS kind smallint;

Nullable and without a default on purpose: it is a metadata-only operation (it does not rewrite a single row), the old code keeps inserting rows, and the values of the rows that are already there are computed by the copy from their label.

1.3.2 Index for the delta passes. The delta passes filter by updated_at. Without an index on that column every pass scans the whole heap; with it, it is an index range that only touches the rows that changed.

-- CONCURRENTLY: it does not block the writes. It cannot run inside a
-- transaction: run it alone.
CREATE INDEX CONCURRENTLY IF NOT EXISTS file_change__updated_at__idx
  ON file_change (updated_at);
  • If it is interrupted, the index is left INVALID and has to be dropped and built again:
SELECT i.indisvalid
  FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid
 WHERE c.relname = 'file_change__updated_at__idx';
  • The index goes with the old table: at the cutover it is renamed to file_change__updated_at__idx_legacy and disappears with it. The new table does not carry it (nothing queries by updated_at; the GC uses deleted_at).

1.4 Consumer generalization: audit_log and jobs

Session settings: SET statement_timeout = 0; SET lock_timeout = '5s'; Re-issue them in each SQL session.

Step 1 — Stop the legacy archiver and its GC

  1. Turn off :audit-log-archive and :audit-log-gc. There is no :audit-log flag in the new version: writing full rows follows the consumer flags.
  2. Enable :nexus only if :nexus-uri is configured. Keep :admin-console on if used. At least one of them must be on on every instance, web and worker: each process computes its own persistence value at boot, so a web instance without a consumer flag writes no rows even if a worker drains them. With all consumers off there is no backlog to replay later.
  3. Set :nexus-uri to the base URL, for example https://nexus.example/proxy. The new consumer appends /api/audit; do not copy the old full :audit-log-archive-uri into :nexus-uri.
  4. Roll the workers with these flags and wait for active archive and GC tasks to finish.

Step 2 — Delete what the archiver already archived

DELETE FROM audit_log
 WHERE id IN (SELECT id FROM audit_log
               WHERE archived_at IS NOT NULL
               LIMIT 5000);

Step 3 — Pre-apply the jobs step

BEGIN;
CREATE TABLE job (
    id           uuid PRIMARY KEY DEFAULT gen_random_uuid(),
    name         text NOT NULL,
    tenant       text NOT NULL,
    queue        text NOT NULL,
    label        text,
    priority     int NOT NULL DEFAULT 100,
    scheduled_at timestamptz NOT NULL DEFAULT now(),
    retry_num    int NOT NULL DEFAULT 0,
    max_retries  int NOT NULL DEFAULT 3,
    status       text NOT NULL DEFAULT 'new'
                 CHECK (status IN ('new', 'scheduled', 'running', 'retry',
                                   'completed', 'failed', 'cancelled',
                                   'aborted')),
    created_at   timestamptz NOT NULL DEFAULT now(),
    modified_at  timestamptz NOT NULL DEFAULT now(),
    started_at   timestamptz,
    completed_at timestamptz,
    params       jsonb NOT NULL DEFAULT '{}',

    profile_id   uuid NULL REFERENCES profile(id) ON DELETE NO ACTION DEFERRABLE,
    error        jsonb,
    result       jsonb,
    resource_id  uuid NULL REFERENCES storage_object(id) ON DELETE SET NULL DEFERRABLE,
    expires_at   timestamptz
);

CREATE INDEX job__dispatcher__idx
    ON job (tenant, priority DESC, scheduled_at)
    WHERE status IN ('new', 'retry');

CREATE INDEX job__orphan__idx
    ON job (tenant, modified_at)
    WHERE status = 'running';

CREATE INDEX job__profile__idx
    ON job (profile_id, created_at DESC)
    WHERE profile_id IS NOT NULL;

CREATE INDEX job__resource__idx
    ON job (resource_id)
    WHERE resource_id IS NOT NULL;

-- Created in step 8 with the `modified_at` definition.

CREATE INDEX job__expires__idx
    ON job (expires_at)
    WHERE expires_at IS NOT NULL;

CREATE INDEX job__retention__idx
    ON job (modified_at)
    WHERE status IN ('completed', 'failed', 'cancelled', 'aborted')
      AND profile_id IS NULL;

CREATE TABLE job_event (
    id         bigserial PRIMARY KEY,
    job_id     uuid NOT NULL REFERENCES job(id) ON DELETE CASCADE DEFERRABLE,
    kind       text NOT NULL
               CHECK (kind IN ('start', 'progress', 'retry', 'end')),
    payload    jsonb NOT NULL DEFAULT '{}',
    created_at timestamptz NOT NULL DEFAULT now()
);

CREATE INDEX job_event__job_kind_created_idx
    ON job_event (job_id, kind, created_at DESC, id DESC);

CREATE OR REPLACE TRIGGER deletion_protection__tgr
BEFORE DELETE ON job FOR EACH STATEMENT
  WHEN ((current_setting('rules.deletion_protection', true) IN ('on', '')) OR
        (current_setting('rules.deletion_protection', true) IS NULL))
  EXECUTE PROCEDURE raise_deletion_protection();
COMMIT;

If the transaction fails, rerun the whole block.

Step 4 — Audit log: columns, reserved range and defaults

CREATE SEQUENCE IF NOT EXISTS audit_log_seq_seq;
ALTER TABLE audit_log ADD COLUMN IF NOT EXISTS seq bigint;
ALTER TABLE audit_log ADD COLUMN IF NOT EXISTS xid xid8 NOT NULL DEFAULT '0'::xid8;
ALTER SEQUENCE audit_log_seq_seq OWNED BY audit_log.seq;

-- Reserve the low range for the history: new rows start above the backfill.
-- Only on the first application: a sequence that already left 1 is reserved
-- by the operator or in use, and a re-run must not move it.
SELECT setval('audit_log_seq_seq',
              (SELECT count(*) + 1000000 FROM audit_log),
              false)
 WHERE (SELECT last_value FROM audit_log_seq_seq) = 1
   AND NOT (SELECT is_called FROM audit_log_seq_seq);

ALTER TABLE audit_log ALTER COLUMN seq SET DEFAULT nextval('audit_log_seq_seq');
ALTER TABLE audit_log ALTER COLUMN xid SET DEFAULT pg_current_xact_id();

Step 5 — Audit log: backfill

Set work_mem high enough for the sort to stay in memory; measure the first run and adjust if needed.

UPDATE audit_log
   SET seq = ordered.rn
  FROM (SELECT id, row_number() OVER (ORDER BY created_at, id) AS rn
          FROM audit_log
         WHERE seq IS NULL) ordered
 WHERE audit_log.id = ordered.id;

Step 6 — Audit log: seq and source NOT NULL without a long ACCESS EXCLUSIVE scan

ALTER TABLE audit_log ADD CONSTRAINT audit_log__seq__not_null
    CHECK (seq IS NOT NULL) NOT VALID;
ALTER TABLE audit_log VALIDATE CONSTRAINT audit_log__seq__not_null;
ALTER TABLE audit_log ALTER COLUMN seq SET NOT NULL;
ALTER TABLE audit_log DROP CONSTRAINT audit_log__seq__not_null;

UPDATE audit_log SET source = 'backend' WHERE source IS NULL;

ALTER TABLE audit_log ADD CONSTRAINT audit_log__source__not_null
    CHECK (source IS NOT NULL) NOT VALID;
ALTER TABLE audit_log VALIDATE CONSTRAINT audit_log__source__not_null;
ALTER TABLE audit_log ALTER COLUMN source SET NOT NULL;
ALTER TABLE audit_log DROP CONSTRAINT audit_log__source__not_null;

Step 7 — Audit log: cursor index and cursor table

CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS audit_log__xid__seq__idx
    ON audit_log (xid, seq);

CREATE TABLE IF NOT EXISTS audit_log_consumer_cursor (
  consumer   smallint    NOT NULL,
  xid        xid8        NOT NULL,
  seq        bigint      NOT NULL,
  updated_at timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (consumer));

Step 8 — Jobs: exclusive and dedupe columns and indexes

Run in autocommit; CREATE INDEX CONCURRENTLY cannot run inside a transaction.

ALTER TABLE job ADD COLUMN IF NOT EXISTS exclusive boolean NOT NULL DEFAULT false;
ALTER TABLE job ADD COLUMN IF NOT EXISTS dedupe    boolean NOT NULL DEFAULT false;

CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS job__live_exclusive__uniq
    ON job (tenant, name, queue, label)
    WHERE exclusive AND status IN ('new', 'scheduled', 'running', 'retry');

CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS job__new_dedupe__uniq
    ON job (tenant, name, queue, label)
    WHERE dedupe AND status = 'new';

DROP INDEX IF EXISTS job__scheduled__idx;
CREATE INDEX IF NOT EXISTS job__scheduled__idx
    ON job (tenant, modified_at)
 WHERE status = 'scheduled';
DROP INDEX IF EXISTS job__name_label__idx;

Step 9 — Gate before deploying

If an index is INVALID, drop and rebuild it before continuing.

SELECT indexname, indexdef FROM pg_indexes
 WHERE indexname IN ('job__live_exclusive__uniq',
                     'job__new_dedupe__uniq',
                     'job__scheduled__idx',
                     'audit_log__xid__seq__idx');

SELECT indexrelid::regclass AS index, indisvalid, indisunique
  FROM pg_index
 WHERE indexrelid IN ('job__live_exclusive__uniq'::regclass,
                      'job__new_dedupe__uniq'::regclass,
                      'audit_log__xid__seq__idx'::regclass);

SELECT column_name, is_nullable
  FROM information_schema.columns
 WHERE table_name = 'audit_log'
   AND column_name IN ('seq', 'xid', 'source')
 ORDER BY 1;

SELECT to_regclass('public.job') IS NOT NULL AS job,
       to_regclass('public.job_event') IS NOT NULL AS job_event,
       to_regclass('public.audit_log_consumer_cursor') IS NOT NULL AS cursor;

SELECT indexname FROM pg_indexes
 WHERE tablename IN ('job', 'job_event')
   AND indexname IN ('job__dispatcher__idx', 'job__orphan__idx',
                     'job__profile__idx', 'job__resource__idx',
                     'job__expires__idx', 'job__retention__idx',
                     'job__scheduled__idx', 'job_event__job_kind_created_idx')
 ORDER BY 1;

SELECT column_name, is_nullable
  FROM information_schema.columns
  WHERE table_name = 'job'
   AND column_name IN ('exclusive', 'dedupe')
  ORDER BY 1;

SELECT tgname FROM pg_trigger
 WHERE tgrelid = to_regclass('public.job')
   AND tgname = 'deletion_protection__tgr'
   AND NOT tgisinternal;

Pass only if all three tables, eight indexes and the trigger exist; seq, xid, source, exclusive and dedupe are NOT NULL; and the three unique indexes are valid and unique. Confirm job__scheduled__idx uses (tenant, modified_at). Do not deploy if any check fails.

1.5 File data storage object foreign key — online steps

Build the schema of 0159-add-file-data-storage-object-id online, while 2.19 still serves traffic: the storage_object_id column, one index per partition, the parent index and the foreign key. The backfill is a separate step in phase 2, inside the coordinated stop. The migration row was pre-registered in section 1.1, so the boot migration skips this work.

Session for this block: plain SQL; the online ALTERs set their own SET LOCAL lock_timeout = '3s' inside their transactions.

Step 1 — Column

BEGIN;
SET LOCAL lock_timeout = '3s';
ALTER TABLE file_data
  ADD COLUMN IF NOT EXISTS storage_object_id uuid NULL;
COMMIT;

The change is catalog-only (a nullable column without a default), so it takes ACCESS EXCLUSIVE for an instant. If it fails with lock timeout, wait for a quieter moment and retry: it did not block anything.

Step 2 — One index per partition, concurrently

CREATE INDEX CONCURRENTLY file_data_00__storage_object_id__idx
    ON file_data_00 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_01__storage_object_id__idx
    ON file_data_01 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_02__storage_object_id__idx
    ON file_data_02 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_03__storage_object_id__idx
    ON file_data_03 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_04__storage_object_id__idx
    ON file_data_04 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_05__storage_object_id__idx
    ON file_data_05 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_06__storage_object_id__idx
    ON file_data_06 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_07__storage_object_id__idx
    ON file_data_07 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_08__storage_object_id__idx
    ON file_data_08 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_09__storage_object_id__idx
    ON file_data_09 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_10__storage_object_id__idx
    ON file_data_10 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_11__storage_object_id__idx
    ON file_data_11 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_12__storage_object_id__idx
    ON file_data_12 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_13__storage_object_id__idx
    ON file_data_13 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_14__storage_object_id__idx
    ON file_data_14 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_15__storage_object_id__idx
    ON file_data_15 (storage_object_id);

Run each statement in autocommit: CREATE INDEX CONCURRENTLY cannot run inside a transaction block. An interrupted build leaves the index INVALID: drop it and retry.

SELECT c.relname, i.indisvalid
  FROM pg_index AS i
  JOIN pg_class AS c ON c.oid = i.indexrelid
 WHERE c.relname LIKE 'file_data_%storage_object_id%'
 ORDER BY c.relname;

Step 3 — Parent index and partition attachment

CREATE INDEX file_data__storage_object_id__idx
    ON ONLY file_data (storage_object_id);

ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_00__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_01__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_02__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_03__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_04__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_05__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_06__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_07__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_08__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_09__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_10__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_11__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_12__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_13__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_14__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_15__storage_object_id__idx;

The step creates the index unconditionally: if an index with that name already exists, drop it and rebuild it. Verify the result before continuing:

SELECT i.indisvalid, i.indpred IS NULL AS plain, i.indexprs IS NULL AS no_expression
  FROM pg_index AS i
 WHERE i.indexrelid = 'file_data__storage_object_id__idx'::regclass;
-- t | t | t

SELECT count(*) FROM pg_inherits
 WHERE inhparent = 'file_data__storage_object_id__idx'::regclass;
-- 16

Step 4 — Foreign key, online

BEGIN;
SET LOCAL lock_timeout = '3s';
SET LOCAL statement_timeout = '10min';   -- measure first; see below
ALTER TABLE file_data
  ADD CONSTRAINT file_data_storage_object_id_fkey
  FOREIGN KEY (storage_object_id) REFERENCES storage_object(id)
  ON DELETE NO ACTION DEFERRABLE INITIALLY IMMEDIATE;
COMMIT;

The column is NULL until the backfill, but PostgreSQL still scans the table (or the index built in the previous steps) to prove there are no violations. The SHARE ROW EXCLUSIVE lock blocks writes for the whole scan, not only while the lock is acquired: lock_timeout bounds the wait for the lock and statement_timeout bounds the scan. A timeout aborts the ALTER, releases the writes and lets you retry in a quieter window; measure the scan in staging and set the budget accordingly. Retry on lock timeout.

Creating it here, before the backfill, keeps the full validation scan out of the stopped window: the backfill validates each row it writes.

INITIALLY IMMEDIATE is required. The storage GC catches the violation inside the chunk transaction and retries the chunk; an INITIALLY DEFERRED check fires at commit, outside that transaction, and the GC would never move past a blocked chunk.

Step 5 — Temporary index for pending backfill rows (online)

Build this partial index while 2.19 serves traffic, before the stop. It covers only rows that still need migration. Each successful batch removes its rows from the index, so later batches do not scan completed rows. Drop the parent index after the backfill.

Run each CREATE INDEX CONCURRENTLY in autocommit:

CREATE INDEX CONCURRENTLY file_data_00__storage_ref_pending__idx
    ON file_data_00 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_01__storage_ref_pending__idx
    ON file_data_01 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_02__storage_ref_pending__idx
    ON file_data_02 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_03__storage_ref_pending__idx
    ON file_data_03 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_04__storage_ref_pending__idx
    ON file_data_04 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_05__storage_ref_pending__idx
    ON file_data_05 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_06__storage_ref_pending__idx
    ON file_data_06 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_07__storage_ref_pending__idx
    ON file_data_07 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_08__storage_ref_pending__idx
    ON file_data_08 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_09__storage_ref_pending__idx
    ON file_data_09 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_10__storage_ref_pending__idx
    ON file_data_10 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_11__storage_ref_pending__idx
    ON file_data_11 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_12__storage_ref_pending__idx
    ON file_data_12 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_13__storage_ref_pending__idx
    ON file_data_13 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_14__storage_ref_pending__idx
    ON file_data_14 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_15__storage_ref_pending__idx
    ON file_data_15 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');

Create and attach the matching parent index:

CREATE INDEX file_data__storage_ref_pending__idx ON ONLY file_data (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');

ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_00__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_01__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_02__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_03__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_04__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_05__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_06__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_07__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_08__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_09__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_10__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_11__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_12__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_13__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_14__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_15__storage_ref_pending__idx;

SELECT indisvalid, indpred IS NOT NULL AS partial
  FROM pg_index
 WHERE indexrelid = 'file_data__storage_ref_pending__idx'::regclass;
-- t | t

SELECT count(*) AS attached_partitions
  FROM pg_inherits
 WHERE inhparent = 'file_data__storage_ref_pending__idx'::regclass;
-- 16

Step 6 — Verify after the backfill

Do not deploy 2.20.0 until every check passes.

-- Column.
SELECT data_type, is_nullable
  FROM information_schema.columns
 WHERE table_schema = 'public'
   AND table_name = 'file_data'
   AND column_name = 'storage_object_id';
-- uuid | YES

-- Rows that will be unreadable. Must be 0 after the backfill. The backfill
-- only fills pointers it can resolve, so a non-zero count means pointers
-- that were already corrupt or orphan: inspect them and accept them before
-- continuing. A row pointing at a storage_object with deleted_at set is
-- legitimate and restorable, not unreadable: the backfill fills it on
-- purpose and the restore flow revives the object, so it is not counted.
SELECT count(*) FROM file_data
 WHERE backend = 'storage' AND storage_object_id IS NULL;

-- Migrated rows no longer retain the old pointer. Must be 0.
SELECT count(*) FROM file_data
 WHERE backend = 'storage' AND storage_object_id IS NOT NULL
   AND jsonb_exists(metadata, 'storage-ref-id');

-- Index: valid, plain, with every partition attached.
SELECT i.indisvalid, i.indpred IS NULL AS plain, i.indexprs IS NULL AS no_expression
  FROM pg_index AS i
 WHERE i.indexrelid = 'file_data__storage_object_id__idx'::regclass;
SELECT count(*) FROM pg_inherits
 WHERE inhparent = 'file_data__storage_object_id__idx'::regclass;   -- 16

-- Foreign key: deferrable, checked immediately, no action, validated.
SELECT condeferrable, condeferred, confdeltype, convalidated
  FROM pg_constraint
 WHERE conname = 'file_data_storage_object_id_fkey'
   AND conrelid = 'file_data'::regclass;
-- t | f | a | t

Phase 2 — The deploy to 2.20.0

2.1 Full stop and backfill

The backfill runs on every instance, even when no row needs it: an instance with nothing to backfill updates 0 rows. There is no rolling deploy: the previous code writes the pointer only to metadata.storage-ref-id and its storage GC reads the reference from there. Running the backfill while that code serves traffic would leave the rows it writes without a pointer and could make the GC delete an object a live row still references.

There is no schema rollback after the deploy: the column is the only reference for the rows 2.20 writes, so a failed upgrade is recovered from the backup.

1. Stop every instance (backend and workers). Keep :tiered-file-data-storage off until every instance runs 2.20.0.

2. Run the backfill:

WITH candidates AS MATERIALIZED (
  SELECT fd.file_id,
         fd.id,
         CASE WHEN (fd.metadata->>'storage-ref-id') ~*
                   '^[0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{12}$'
              THEN (fd.metadata->>'storage-ref-id')::uuid END AS soid
    FROM file_data AS fd
    JOIN storage_object AS so
      ON so.id = CASE WHEN (fd.metadata->>'storage-ref-id') ~*
                          '^[0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{12}$'
                     THEN (fd.metadata->>'storage-ref-id')::uuid END
   WHERE fd.backend = 'storage'
     AND fd.storage_object_id IS NULL
     AND jsonb_exists(fd.metadata, 'storage-ref-id')
     AND (fd.metadata->>'storage-ref-id') ~*
         '^[0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{12}$'
   ORDER BY fd.file_id, fd.id
   LIMIT 5000
 ),
 updated AS (
  UPDATE file_data AS fd
     SET storage_object_id = c.soid,
         metadata = fd.metadata - 'storage-ref-id'
    FROM candidates AS c
   WHERE fd.file_id = c.file_id
     AND fd.id = c.id
  RETURNING 1
)
SELECT count(*) AS rows_updated FROM updated;

Run the statement repeatedly, committing each batch separately, until rows_updated is 0. Each batch updates at most 5,000 rows, sets the column and removes metadata.storage-ref-id atomically. The index built in phase 1.5 keeps the candidate search bounded to rows still pending. The MATERIALIZED CTE keeps the UUID cast away from values that fail the regex. The join to storage_object excludes orphan pointers, and the foreign key validates each written row. Corrupt and orphan pointers remain untouched for investigation; they must be resolved before the unreadable-row verification can pass.

After all batches finish, remove the temporary index:

DROP INDEX file_data__storage_ref_pending__idx;

The migrated key is not retained as a rollback copy. A rollback to the previous binary after the backfill requires restoring the backup.

3. Verify. Run the Verify block of phase 1.5. All checks must pass.

4. Confirm the pre-registered migration row exists:

SELECT module, step, created_at FROM migrations
 WHERE module = 'main' AND step = '0159-add-file-data-storage-object-id';

The row was inserted at the beginning of the runbook. Do not deploy 2.20.0 until the backfill and all verification checks have passed.

5. Start the 2.20.0 instances.

2.2 The watermark goes live

The deploy brings the code that updates updated_at on every mutation (locks, soft deletes). Phase 3 depends on it. Confirm the watermark moved:

SELECT max(updated_at) FROM file_change;

It has to move on every write from now on. If it does not, do not start phase 3.


Phase 3 — After the upgrade (2.20.0 serving; online)

Phase 3 copies file_change into the new partitioned table in the background. It spans days: the bulk is one pass over every row (hours to days), then short delta passes every 1-6 h until the maintenance window. Everything here runs with the instance serving traffic.

3.1 Create the partitioned table

Create the partitioned table and its 16 partitions:

CREATE TABLE file_change_new (
  file_id       uuid NOT NULL,
  id            uuid NOT NULL DEFAULT gen_random_uuid(),
  created_at    timestamptz NOT NULL DEFAULT now(),
  session_id    uuid,
  revn          bigint NOT NULL DEFAULT 0,
  data          bytea,
  changes       bytea,
  profile_id    uuid,
  features      text[],
  label         text,
  version       integer,
  created_by    text NOT NULL DEFAULT 'system',
  updated_at    timestamptz DEFAULT now(),
  deleted_at    timestamptz,
  migrations    text[],
  locked_by     uuid,
  -- Nullable on purpose, like the migration: the previous version inserts
  -- without `kind`, and a rollback to it has to keep writing. The `NOT NULL`
  -- comes in a later migration, once the rollback window is closed.
  kind          smallint,
  CONSTRAINT file_change_kind_check CHECK (kind IN (1, 2))
) PARTITION BY HASH (file_id);

-- the columns keep the storage of PostgreSQL by default (migration
-- 0156-reset-pg-storage-defaults dropped the explicit EXTERNAL overrides)

COMMENT ON COLUMN file_change_new.kind IS
  'The kind of the row: 1 is a snapshot (it has a label and its data lives in file_data) and 2 is a revision (it has no label and it holds the changes).';
COMMENT ON COLUMN file_change_new.locked_by IS
  'Profile ID of user who has locked this version. Only the creator can lock/unlock their own versions. Locked versions cannot be deleted by others.';

-- 16 partitions, the same bounds as file_data. The name of the table is
-- temporary (the old one holds it until the cutover) but the name of the
-- partitions is not: it is the final one, so their indexes are born with
-- the canonical names (renaming a partition does not rename its indexes)
-- and the result is exactly the schema of the migration.
CREATE TABLE file_change_00 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 0);
CREATE TABLE file_change_01 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 1);
CREATE TABLE file_change_02 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 2);
CREATE TABLE file_change_03 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 3);
CREATE TABLE file_change_04 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 4);
CREATE TABLE file_change_05 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 5);
CREATE TABLE file_change_06 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 6);
CREATE TABLE file_change_07 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 7);
CREATE TABLE file_change_08 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 8);
CREATE TABLE file_change_09 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 9);
CREATE TABLE file_change_10 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 10);
CREATE TABLE file_change_11 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 11);
CREATE TABLE file_change_12 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 12);
CREATE TABLE file_change_13 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 13);
CREATE TABLE file_change_14 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 14);
CREATE TABLE file_change_15 PARTITION OF file_change_new
  FOR VALUES WITH (MODULUS 16, REMAINDER 15);

-- The primary key includes the partition key (required on a partitioned
-- table)
ALTER TABLE file_change_new ADD PRIMARY KEY (file_id, id);

-- Foreign keys (no CREATE TABLE copies them). They take the canonical
-- name already: they have no index, so they do not collide with the old
-- table and the partitions inherit it.
ALTER TABLE file_change_new
  ADD CONSTRAINT file_change_file_id_fkey FOREIGN KEY (file_id)
  REFERENCES file(id) DEFERRABLE;
ALTER TABLE file_change_new
  ADD CONSTRAINT file_change_locked_by_fkey FOREIGN KEY (locked_by)
  REFERENCES profile(id) ON DELETE SET NULL DEFERRABLE;
ALTER TABLE file_change_new
  ADD CONSTRAINT file_change_profile_id_fkey FOREIGN KEY (profile_id)
  REFERENCES profile(id) ON DELETE SET NULL DEFERRABLE;
SELECT count(*) AS partitions FROM pg_partition_tree('file_change_new') t
 WHERE t.isleaf;
-- 16

The new indexes carry a _new suffix because index names are unique per schema and the old table still holds the canonical ones until the cutover. The same goes for the primary key of the new table: its index is named after the constraint. The constraints that have no index (the CHECK and the foreign keys) are named canonically from the start, because a constraint name is unique per table, not per schema, so it can live with the one of the old table; the partitions inherit it when they are created. The partitions are created with their final name, because their indexes are named after the partition and not after the table.

3.2 Bulk copy

Copies every row, including the soft deleted ones: the GC manages them the same way in the new table. kind is computed from label: a row with a label is a snapshot, a row without one is a revision.

The pointer of the copy lives in the database, not in the terminal: a batch can be interrupted and resumed without losing anything.

CREATE TABLE IF NOT EXISTS file_change_copy_state (
  name    text PRIMARY KEY,
  last_id uuid,
  last_at timestamptz
);

-- for a procedure that already created the table without the mark
ALTER TABLE file_change_copy_state
  ADD COLUMN IF NOT EXISTS last_at timestamptz;

INSERT INTO file_change_copy_state (name, last_id)
VALUES ('bulk', '00000000-0000-0000-0000-000000000000')
ON CONFLICT (name) DO NOTHING;

-- The mark of the delta passes starts at the bulk: the first pass of phase
-- 3.3 covers the rows changed since here, and `DO NOTHING` keeps a re-run
-- from resetting a mark that the passes already moved.
INSERT INTO file_change_copy_state (name, last_at) VALUES ('delta', now())
ON CONFLICT (name) DO NOTHING;

-- The mark is mandatory: a pass without it copies nothing, in silence. The
-- `DROP ... IF EXISTS` keeps this block re-runnable.
ALTER TABLE file_change_copy_state
  ADD CONSTRAINT file_change_copy_state_delta_mark
  CHECK (name <> 'delta' OR last_at IS NOT NULL);

Run the create-table block of phase 3.1 first, then this block, then the batch block in a loop until it returns 0.

One batch (idempotent). It always returns one row: the number of rows it copied, 0 when there is nothing left to copy.

WITH batch AS MATERIALIZED (
  SELECT id FROM file_change
   WHERE id > (SELECT last_id FROM file_change_copy_state WHERE name = 'bulk')
   ORDER BY id
   LIMIT 10000
),
copied AS (
  INSERT INTO file_change_new (id, file_id, created_at, session_id, revn, data, changes,
                               profile_id, features, label, version, created_by,
                               updated_at, deleted_at, migrations, locked_by, kind)
  SELECT id, file_id, created_at, session_id, revn, data, changes,
         profile_id, features, label, version, created_by,
         updated_at, deleted_at, migrations, locked_by,
         CASE WHEN label IS NOT NULL THEN 1 ELSE 2 END
    FROM file_change
   WHERE id IN (SELECT id FROM batch)
  ON CONFLICT (file_id, id) DO UPDATE
     SET created_at = EXCLUDED.created_at,
         session_id = EXCLUDED.session_id,
         revn       = EXCLUDED.revn,
         data       = EXCLUDED.data,
         changes    = EXCLUDED.changes,
         profile_id = EXCLUDED.profile_id,
         features   = EXCLUDED.features,
         label      = EXCLUDED.label,
         version    = EXCLUDED.version,
         created_by = EXCLUDED.created_by,
         updated_at = EXCLUDED.updated_at,
         deleted_at = EXCLUDED.deleted_at,
         migrations = EXCLUDED.migrations,
         locked_by  = EXCLUDED.locked_by,
         kind       = EXCLUDED.kind
   WHERE file_change_new.updated_at IS DISTINCT FROM EXCLUDED.updated_at
  RETURNING 1
),
-- `moved` advances the pointer only when the batch copied something.
-- PostgreSQL executes a data-modifying CTE exactly once even when the
-- final query does not read its output.
moved AS (
  UPDATE file_change_copy_state
     SET last_id = (SELECT id FROM batch ORDER BY id DESC LIMIT 1)
   WHERE name = 'bulk'
     AND EXISTS (SELECT 1 FROM copied)
  RETURNING 1
)
SELECT count(*) AS copied FROM copied;

Repeat this statement until it returns 0. The 0 comes from the final SELECT, not from a missing row: a batch that finds nothing to copy still returns it, so it is the signal to stop.

  • The pointer is a uuid (random order): the pagination is stable but it does not group by physical position. The total read is the heap, in a scattered order.
  • Estimated duration: time the first batches and extrapolate. The bulk reads the heap plus the live TOAST once (phase 1) and writes about the live data. Batches are rows divided by batch size.
  • Pace the batches yourself (a pause between statements) if the streaming replica falls behind or the backend sees latency. Every batch detoastes its share of the TOAST: the cost is I/O, not CPU.
  • Watch: replica lag, pg_stat_activity (locks between the bulk and autovacuum), the space of pg_wal and of the data dir.

When the bulk is done, check that no row is missing (it is the check that matters, not the total count):

SELECT count(*) AS missing
  FROM file_change o
 WHERE NOT EXISTS (SELECT 1 FROM file_change_new n
                    WHERE n.file_id = o.file_id
                      AND n.id = o.id);

missing is zero when no row of the old table is absent from the new one; it is the check that matters. The new table can have more rows than the old one: the GC may have hard deleted a row from the source after the bulk copied it, and it cleans that row up after the cutover. Rows written during the bulk are missing until the delta passes pick them up.

3.3 Delta passes

Every pass copies the rows modified since the previous pass. The watermark lives in file_change_copy_state.last_at (the start of the last pass; the setup block of phase 3.2 seeds it with the moment of the bulk, so the first pass covers what changed since then), and the SQL subtracts one hour of margin, which absorbs the clock drift between the application (which writes updated_at) and the database (which writes now()) and the drift between batches. A row that a pass already copied but did not change again is not rewritten: the WHERE of the ON CONFLICT skips it.

BEGIN;

WITH mark AS (
  SELECT last_at - INTERVAL '1 hour' AS since
    FROM file_change_copy_state
   WHERE name = 'delta'
)
INSERT INTO file_change_new (id, file_id, created_at, session_id, revn, data, changes,
                             profile_id, features, label, version, created_by,
                             updated_at, deleted_at, migrations, locked_by, kind)
SELECT id, file_id, created_at, session_id, revn, data, changes,
       profile_id, features, label, version, created_by,
       updated_at, deleted_at, migrations, locked_by,
       CASE WHEN label IS NOT NULL THEN 1 ELSE 2 END
  FROM file_change, mark
 WHERE updated_at > mark.since
ON CONFLICT (file_id, id) DO UPDATE
   SET created_at = EXCLUDED.created_at,
       session_id = EXCLUDED.session_id,
       revn       = EXCLUDED.revn,
       data       = EXCLUDED.data,
       changes    = EXCLUDED.changes,
       profile_id = EXCLUDED.profile_id,
       features   = EXCLUDED.features,
       label      = EXCLUDED.label,
       version    = EXCLUDED.version,
       created_by = EXCLUDED.created_by,
       updated_at = EXCLUDED.updated_at,
       deleted_at = EXCLUDED.deleted_at,
       migrations = EXCLUDED.migrations,
       locked_by  = EXCLUDED.locked_by,
       kind       = EXCLUDED.kind
 WHERE file_change_new.updated_at IS DISTINCT FROM EXCLUDED.updated_at;

-- The mark is the start of this transaction (`now()`): the rows changed
-- after it are the window of the next pass. Both statements go in the
-- same transaction on purpose; a mark written later would leave a gap.
UPDATE file_change_copy_state SET last_at = now() WHERE name = 'delta';

COMMIT;
  • Repeat the transaction every 1-6 h (depending on the activity) until the pass copies very few rows. With the updated_at index of phase 1.3 the pass is an index range over the window, so it takes seconds on a quiet instance; without it, it scans the heap. To see how many rows are left in the window:
SELECT count(*) FROM file_change
 WHERE updated_at > (SELECT last_at FROM file_change_copy_state WHERE name = 'delta');
  • The objects GC can keep running. Its soft deletes are covered by the watermark, and its hard deletes leave a row with deleted_at in the new table that the GC itself cleans up later. The hard delete of a file is blocked while the old table (or the new one) has rows for it, because the foreign key on file_id is NO ACTION. The GC deletes the pending files in chunks of 100, and one blocked file aborts the transaction of the whole chunk: the chunk is logged as error on chunk deletion and skipped, its files wait in the trash, and the next daily run retries the same chunk and fails again until the cutover. The other chunks keep making progress. Do not run the cutover at the same time as a big batch of the GC.

Phase 4 — The maintenance window (after the deploy)

4.1 Before the lock (online, still 2.20.0 serving)

  • One last delta pass (the block of phase 3.3).
  • Build the secondary indexes of the new table. The bulk and the delta passes only need the primary key (ON CONFLICT (file_id, id)), so building them here keeps five indexes and their WAL out of every batch. The deleted_at index carries file_id (the primary key is (file_id, id)): the objects GC needs the file_id to lock by pair, and with the index it reads the rows it has to delete without touching the table.
-- Secondary indexes: the canonical names, with the _new suffix. There is no
-- index for the system snapshots: the old one filtered them by
-- `data IS NOT NULL`, and with the payload living in `file_data` the column
-- no longer tells snapshots apart. The new one could only filter them by
-- `deleted_at IS NULL`, and the system snapshots always carry a deletion
-- date in the future, so it would never match a row.
CREATE INDEX file_change_new__deleted_at__idx
  ON file_change_new (deleted_at, file_id, id) WHERE deleted_at IS NOT NULL;
CREATE INDEX file_change_new__file_id__revn__idx
  ON file_change_new (file_id, revn);
CREATE INDEX file_change_new__locked_by__idx
  ON file_change_new (locked_by) WHERE locked_by IS NOT NULL;
CREATE INDEX file_change_new__profile_id__idx
  ON file_change_new (profile_id) WHERE profile_id IS NOT NULL;
CREATE INDEX file_change_new__user_snapshots__idx
  ON file_change_new (file_id)
 WHERE label IS NOT NULL AND created_by = 'user' AND deleted_at IS NULL;
  • Prepare and rehearse the cutover block in staging.
  • Pick a low activity window: a freeze of the writes to file_change of a couple of minutes (the final delta inside the lock copies only the rows changed since the last delta pass, so it is quick; the saves of files during the freeze fail and the user retries, and the rest of the database keeps working). The window has to cover the whole cutover, with the backend and the workers stopped and started again (see 4.2).

4.2 The cutover (one transaction, a brief exclusive lock)

  • Stop the backend and the workers before the transaction, and start them again after it. The pool keeps server-side prepared statements, and a prepared statement holds the OID of the table it was prepared with: renaming the table does not change that OID. A connection that had already run insert into file_change five times (the default threshold of the driver) would keep writing to file_change_legacy after the cutover, and the pool recycles its connections every 30 minutes, so the window of silent writes is long.
  • Run the transaction, the ANALYZE, and the statement that records the instant of the cutover (all of it in the block below).
  • Start the backend and the workers again.
BEGIN;

LOCK TABLE file_change IN ACCESS EXCLUSIVE MODE;

-- The final delta: the rows modified since the last pass (its mark, minus
-- the hour of margin). The mark is the one the last pass left.
WITH mark AS (
  SELECT last_at - INTERVAL '1 hour' AS since
    FROM file_change_copy_state
   WHERE name = 'delta'
)
INSERT INTO file_change_new (id, file_id, created_at, session_id, revn, data, changes,
                             profile_id, features, label, version, created_by,
                             updated_at, deleted_at, migrations, locked_by, kind)
SELECT id, file_id, created_at, session_id, revn, data, changes,
       profile_id, features, label, version, created_by,
       updated_at, deleted_at, migrations, locked_by,
       CASE WHEN label IS NOT NULL THEN 1 ELSE 2 END
  FROM file_change, mark
 WHERE updated_at > mark.since
ON CONFLICT (file_id, id) DO UPDATE
   SET created_at = EXCLUDED.created_at,
       session_id = EXCLUDED.session_id,
       revn       = EXCLUDED.revn,
       data       = EXCLUDED.data,
       changes    = EXCLUDED.changes,
       profile_id = EXCLUDED.profile_id,
       features   = EXCLUDED.features,
       label      = EXCLUDED.label,
       version    = EXCLUDED.version,
       created_by = EXCLUDED.created_by,
       updated_at = EXCLUDED.updated_at,
       deleted_at = EXCLUDED.deleted_at,
       migrations = EXCLUDED.migrations,
       locked_by  = EXCLUDED.locked_by,
       kind       = EXCLUDED.kind
 WHERE file_change_new.updated_at IS DISTINCT FROM EXCLUDED.updated_at;

-- Check inside the lock, and abort if it fails: no row of the old table
-- may be missing in the new one. The new table can carry extra rows that
-- the GC hard deleted from the source; it cleans them up after the
-- cutover.
DO $$
DECLARE missing bigint;
BEGIN
  SELECT count(*) INTO missing
    FROM file_change o
   WHERE NOT EXISTS (SELECT 1 FROM file_change_new n
                      WHERE n.file_id = o.file_id
                        AND n.id = o.id);

  IF missing > 0 THEN
    RAISE EXCEPTION 'file_change: % rows of the old table are missing in file_change_new; cutover aborted', missing;
  END IF;
END $$;

-- The old table frees the canonical names (the same names the migration
-- 0158 uses, with the _legacy suffix)
ALTER TABLE file_change RENAME CONSTRAINT file_change_pkey
  TO file_change_pkey_legacy;
ALTER TABLE file_change RENAME CONSTRAINT file_change_file_id_fkey
  TO file_change_file_id_fkey_legacy;
ALTER TABLE file_change RENAME CONSTRAINT file_change_locked_by_fkey
  TO file_change_locked_by_fkey_legacy;
ALTER TABLE file_change RENAME CONSTRAINT file_change_profile_id_fkey
  TO file_change_profile_id_fkey_legacy;

ALTER INDEX file_change__data_ref_id__idx
  RENAME TO file_change__data_ref_id__idx_legacy;
ALTER INDEX file_change__deleted_at__idx
  RENAME TO file_change__deleted_at__idx_legacy;
ALTER INDEX file_change__file_id__revn__idx
  RENAME TO file_change__file_id__revn__idx_legacy;
ALTER INDEX file_change__locked_by__idx
  RENAME TO file_change__locked_by__idx_legacy;
ALTER INDEX file_change__profile_id__idx
  RENAME TO file_change__profile_id__idx_legacy;
ALTER INDEX file_change__system_snapshots__idx
  RENAME TO file_change__system_snapshots__idx_legacy;
ALTER INDEX IF EXISTS file_change__user_snapshots__idx
  RENAME TO file_change__user_snapshots__idx_legacy;
ALTER INDEX IF EXISTS file_change__updated_at__idx
  RENAME TO file_change__updated_at__idx_legacy;

-- And the new table takes them. Only the primary key needs the rename: the
-- `CHECK` and the foreign keys already carry the canonical name (they have
-- no index, so they could take it from the start), and the partitions
-- inherited it.
ALTER TABLE file_change_new RENAME CONSTRAINT file_change_new_pkey
  TO file_change_pkey;

ALTER INDEX file_change_new__deleted_at__idx
  RENAME TO file_change__deleted_at__idx;
ALTER INDEX file_change_new__file_id__revn__idx
  RENAME TO file_change__file_id__revn__idx;
ALTER INDEX file_change_new__locked_by__idx
  RENAME TO file_change__locked_by__idx;
ALTER INDEX file_change_new__profile_id__idx
  RENAME TO file_change__profile_id__idx;
ALTER INDEX file_change_new__user_snapshots__idx
  RENAME TO file_change__user_snapshots__idx;

-- And the tables are swapped. The partitions and their indexes keep the
-- names they were born with (they are named after the partition, not after
-- the table), so there is nothing else to rename.
ALTER TABLE file_change     RENAME TO file_change_legacy;
ALTER TABLE file_change_new RENAME TO file_change;

-- The foreign key of the legacy table goes away: it is `NO ACTION`, so
-- while the legacy table has rows for a file, the objects GC cannot delete
-- that file (it deletes the rows of the file first, and the legacy ones are
-- not its business). The legacy table is frozen: nothing writes to it again.
ALTER TABLE file_change_legacy
  DROP CONSTRAINT file_change_file_id_fkey_legacy;

COMMIT;

ANALYZE file_change;

-- The instant of the cutover, for the checks of phase 5. It goes after
-- the `COMMIT`: inside the transaction `now()` is the start of it (the
-- moment of the `LOCK`), and the rows the final delta copies could pass
-- for writes happened after the cutover.
INSERT INTO file_change_copy_state (name, last_at) VALUES ('cutover', now())
ON CONFLICT (name) DO UPDATE SET last_at = excluded.last_at;
  • file_change_legacy keeps its data and its indexes (with the _legacy suffix): a frozen copy of the previous layout. Its foreign key on file_id is dropped, so the objects GC can delete the files again.
  • The writes to file_change during the lock fail: they are seconds-minutes, the affected saves are retried.
  • The objects GC and the four RPC commands keep working: same table name and same index names after the rename.
  • Once the backend is up again, check that the saves go to the partitioned table and that nothing went to the table that is going away:
-- the rows written after the cutover: this one grows with every save
SELECT count(*) FROM file_change
 WHERE updated_at > (SELECT last_at FROM file_change_copy_state
                      WHERE name = 'cutover');

-- the rows that went to the legacy table: this one has to stay at 0
SELECT count(*) FROM file_change_legacy
 WHERE updated_at > (SELECT last_at FROM file_change_copy_state
                      WHERE name = 'cutover');

A number above zero in the second query means that some connection was still writing with a prepared statement of the old table: those rows are lost when the legacy table is dropped, so they have to be copied over before that.

4.3 The storage backfill is not part of this window

The backfill always runs in the phase 2 stop, before the new instances start. After the deploy, any row that still carries the pointer is unreadable by the new code, so this is not a step to defer to the phase 4 window.


Phase 5 — After the cutover (online)

5.1 Partition pruning and access paths

Check that the access paths prune to one partition (when the planner chooses an index scan):

EXPLAIN SELECT * FROM file_change
 WHERE file_id = '00000000-0000-0000-0000-000000000000'
   AND id     = '00000000-0000-0000-0000-000000000000';

It has to show a single partition (file_change_NN), not an Append of 16.

5.2 Schema parity with the migration, and ownership

Every row of this query has to show got = expected:

SELECT 'columns' AS what, count(*) AS got, 17 AS expected
  FROM pg_attribute
 WHERE attrelid = 'file_change'::regclass AND attnum > 0 AND NOT attisdropped
UNION ALL
SELECT 'partitions', count(*), 16
  FROM pg_partition_tree('file_change') t WHERE t.isleaf
UNION ALL
SELECT 'primary key', count(*), 1
  FROM pg_constraint
 WHERE conrelid = 'file_change'::regclass AND contype = 'p'
UNION ALL
SELECT 'foreign keys', count(*), 3
  FROM pg_constraint
 WHERE conrelid = 'file_change'::regclass AND contype = 'f'
UNION ALL
SELECT 'check constraints', count(*), 1
  FROM pg_constraint
 WHERE conrelid = 'file_change'::regclass AND contype = 'c'
UNION ALL
SELECT 'indexes', count(*), 6
  FROM pg_indexes
 WHERE tablename = 'file_change' AND schemaname = current_schema()
UNION ALL
SELECT 'partitions out of name', count(*), 0
  FROM pg_class
 WHERE relkind = 'r' AND relname ~ '^file_change_new[0-9]'
UNION ALL
SELECT 'child indexes out of name', count(*), 0
  FROM pg_class c
  JOIN pg_index i ON i.indexrelid = c.oid
  JOIN pg_class p ON p.oid = i.indrelid
 WHERE p.relname ~ '^file_change_[0-9]{2}$'
   AND c.relname NOT LIKE p.relname || '_%'
UNION ALL
SELECT 'partition constraints out of name', count(*), 0
  FROM pg_constraint c
  JOIN pg_class t ON t.oid = c.conrelid
 WHERE t.relname ~ '^file_change_[0-9]{2}$'
   AND c.contype <> 'n'
   AND c.conname LIKE 'file_change_new%';

The NOT NULL constraints are excluded from the last check (contype <> 'n'): their names are not stable across server versions, so they are not part of the parity. The CHECK, the primary key and the foreign keys are.

And the names, to compare them with the migration:

SELECT a.attname AS column
  FROM pg_attribute a
 WHERE a.attrelid = 'file_change'::regclass AND a.attnum > 0
   AND NOT a.attisdropped
 ORDER BY a.attnum;

SELECT indexname FROM pg_indexes
 WHERE tablename = 'file_change' AND schemaname = current_schema()
 ORDER BY indexname;

-- the partitions, with their name
SELECT c.relname AS partition
  FROM pg_class c
  JOIN pg_partition_tree('file_change') t ON t.relid = c.oid
 WHERE t.isleaf
 ORDER BY c.relname;

-- the indexes of the partitions: their name starts with the partition
SELECT c.relname AS child_index
  FROM pg_class c
  JOIN pg_index i ON i.indexrelid = c.oid
  JOIN pg_class p ON p.oid = i.indrelid
 WHERE p.relname ~ '^file_change_[0-9]{2}$'
 ORDER BY c.relname;

-- every table, partition and index of the family shares the owner of the
-- table
SELECT count(*) AS out_of_owner
  FROM pg_class c
 WHERE c.relname LIKE 'file_change%'
   AND c.relkind IN ('r', 'p', 'i')
   AND pg_get_userbyid(c.relowner) <>
       (SELECT tableowner FROM pg_tables
         WHERE schemaname = current_schema() AND tablename = 'file_change');

The owner check has to return 0. If it does not, fix the ownership before opening the service (re-own the table, the 16 partitions and their indexes) and re-apply any grants the old table had.

5.3 Partition-wise join and autovacuum

  • The partition-wise join is already on: the pool of the backend sets enable_partitionwise_join = on on every connection (it is in the code, there is nothing to change in the database role).
  • Autovacuum: each of the 16 partitions has its own TOAST (check their sizes with pg_total_relation_size per partition); with autovacuum_vacuum_scale_factor = 0.2 the threshold is reached earlier and in steps. Nothing to change, just watch last_autovacuum of the partitions the first week.
  • The aggregations that do not filter by file_id (the team snapshot quota and the telemetry counts) read all 16 partitions. To know their cost on your instance, time them before and after with EXPLAIN (ANALYZE, BUFFERS) and compare. The team quota is already fixed in the code: it resolves the file_id of the team first, which lets the planner prune the partitions when it chooses an index scan (with a sequential scan it reads all of them).

5.4 Drop the helper table and, days later, the old table

DROP TABLE IF EXISTS file_change_copy_state;
DROP TABLE file_change_legacy;   -- out of hours: it generates WAL by
                                 -- dropping the pages, let it roll

That is where the dead space comes back: the database shrinks by about the size of the old table (phase 1). Copy the rows from the legacy table first if phase 4 showed any (see the check after the cutover); they are lost when it is dropped.

5.5 Post-deploy storage checks

SELECT conname, condeferrable, condeferred, confdeltype, convalidated
  FROM pg_constraint
 WHERE conname = 'file_data_storage_object_id_fkey'
   AND conrelid = 'file_data'::regclass;

SELECT count(*) FROM file_data
 WHERE backend = 'storage' AND storage_object_id IS NULL;

-- Run the foreign-key check; ROLLBACK keeps the object.
BEGIN;
EXPLAIN (ANALYZE, BUFFERS)
  DELETE FROM storage_object WHERE id = '<uuid of a deleted object with no references>';
ROLLBACK;

5.6 After every instance runs 2.20.0: turn on the new GC

Once every instance runs this version, the new consumers have replayed the unarchived buffer. Turn on :audit-log-gc (rolling restart of the new workers) so the new GC deletes the rows the consumers have passed. Remove :audit-log-archive; the new version does not use it.


Rollback

  • Before the phase 2 deploy: phase 1 may have applied out-of-band schema changes, but the new code has not started. Keep 2.19 serving; assess and reverse the completed phase 1 changes before removing the pre-registration rows or switching to the automatic migration path.
  • Partition cutover: file_change_legacy is a frozen copy of the previous layout, kept for days. Rolling back before dropping it means pointing the code at the legacy layout, which is not supported by the release; treat the DROP TABLE file_change_legacy as the point of no return and schedule it deliberately.
  • Consumer generalization: the migration keeps the legacy archived_at mark and its indexes for one release; the two versions can coexist during the rollout. Rolling the binaries back is possible while :audit-log-gc is off and the consumers are idle; keep the flag off until you are done, and call it a rollback only within the rollout.

Risks

Risk Mitigation
Massive WAL during the bulk copy (proportional to the copy) Throttle between batches; watch pg_wal, the replication slots and the replica lag
Latency seen by the application during the bulk Throttle; low traffic hours; the bulk is I/O and does not block writes
updated_at not reliable in rows changed before the deploy The deploy goes before the bulk (phase 2.2); the delta passes cover from the start of the bulk
Lock/unlock/soft delete during the passes Covered by the watermark; the cutover includes the final delta inside the lock
Hard delete of the GC between passes The row is left with deleted_at in the new table; the GC cleans it up later
Orphan rows if a file is deleted halfway The foreign key on file_id is NO ACTION: the GC deletes the rows of the file before the file, so a chunk that fails leaves the file in the trash and the next run deletes it
The files in the trash cannot be deleted while the legacy table has rows for them The foreign key of the legacy table is dropped at the cutover; until then the files wait in the trash
Saves failing during the lock of the cutover Freeze of minutes in a low traffic window; manual retries
Planner regression after the cutover Immediate ANALYZE + the pruning EXPLAIN; the pool already sets the GUC
Clock drift between the application and the database The delta window starts an hour before the bulk; the margin absorbs the drift
CONCURRENTLY build interrupted, index left INVALID Drop it and rebuild it; IF NOT EXISTS matches by name and would accept the broken index: the gates (phase 1.4 step 9 and phase 1.5 verify) are mandatory
Running the backfill while 2.19 serves Do not: phase 2 stops every instance before the backfill

Success metrics

  • After the cutover: file_change (partitioned) holds at most the pg_table_size of the old table (phase 1).
  • After the DROP of the old table: the database shrinks by about the size that table had.
  • The access paths that filter by file_id (the lookup of a snapshot, the quota of a file and the lock of the GC by pair) prune to one partition when the planner chooses an index scan (EXPLAIN). The quota of the team and the discovery of the GC do not filter by a single file: they read all the 16 partitions.
  • No visible change for the user.