penpot/operations/2.20.0-file-data-storage-object-id.md
Andrey Antukh 90a06e982b
♻️ Rework the audit-log consumers and the jobs substrate (#12701)
* ♻️ Rework audit-log consumers and jobs substrate

Fold the implementation, follow-up fixes, migrations and tests into one

code commit.

AI-assisted-by: gpt-6-luna

* 📚 Consolidate 2.20.0 upgrade runbooks

Keep the operational upgrade procedure and its supporting runbooks in a

separate documentation commit.

AI-assisted-by: gpt-6-luna

* ♻️ Remove the audit-log flag and derive persistence at boot

Persistence now follows the consumer flags instead of a separate switch.
app.config computes cf/audit-persist? once at boot from :nexus and
:admin-console, and every write site reads that value, so request paths
no longer check flags per event. With no consumer enabled no full rows
are written; telemetry shadows and webhook deliveries keep their own
flags. The frontend wire signal stays :audit-log as a derived value.

Update the 2.20 runbooks, memories and the devenv flags to match, and
migrate the tests that used to toggle :audit-log directly.

AI-assisted-by: hy4-flash-free

* 🐛 Accept error-reporting batches and precompute sink flags

push-audit-events now accepts crash batches with :error-reporting alone, so
the frontend signal from get-environment-data matches a real destination.
The reporters receive unhandled-exception events without storing any row.

All per-event sink checks read booleans computed once at boot: webhooks,
telemetry and error reporting join audit-persist? in app.config. The test
fixture and with-audit-config rebind all four values together, and the
tests that toggled those flags directly were migrated.

Address F2 of the review of the previous commit. F1 stays deferred to a
separate plan.

AI-assisted-by: hy4-flash-free

* 💄 Match the layout cljfmt expects in the flags set

The first comment of the varia set sat on its own line after the opening
delimiter, and cljfmt joins such a comment to the delimiter line, so the
Common Tests lint step failed on common/src/app/common/flags.cljc. Keep
the comment on the opening line, the same shape the login and email sets
already use.

Note in mem:scripts/ci that the backend lint lints ../common/src but does
not format-check it, so common has to be passed to scripts/ci as well.

AI-assisted-by: deepseek-v4-flash
2026-11-20 11:12:00 +01:00

21 KiB

2.20.0 — File data storage object foreign key

Runbook for the release that moves the file data storage pointer out of file_data.metadata->>'storage-ref-id' and into a column, file_data.storage_object_id, with a foreign key to storage_object(id).

When to run this

The migration 0159-add-file-data-storage-object-id does the whole job on a fresh database and on small instances: it adds the column, backfills the pointer, creates the index and the foreign key.

A large instance builds everything online and keeps the stopped window as short as possible:

  • the column, the index and the foreign key are built online, without stopping the service;
  • the backfill runs in the stopped window only if the metadata pointer exists; when the storage backend is off, the pre-flight finds no pointer, tiered offload is off and the queue is empty, there is nothing to backfill and the new binaries roll out like any other deploy;
  • the step is then marked as applied in the migrations table, so the migration body does not run at startup.

The coordinated restart is required when the instance uses the storage backend (PENPOT_FILE_DATA_BACKEND=storage), has :tiered-file-data-storage on or pending offload-file-data tasks, or still has rows that carry the metadata pointer. 2.19 writes storage rows through the tiered offload even with the default db backend: turn the flag off and wait for the queue before the pre-flight. The previous code writes and reads metadata.storage-ref-id, the new code reads the column, and a rolling deploy would mix both. With the default db backend, no tiered offload, no pending offloads and no row carrying the pointer (see the pre-flight query), the previous and the new code read the same file_data.data column, so a rolling deploy is safe.

This procedure is one-way. It has no schema rollback: after an attempt, restore the backup. The column is the only reference for the rows 2.20 writes.

Do not run the backfill while the previous version is serving when PENPOT_FILE_DATA_BACKEND is configured to storage, or when the table still has rows with the metadata pointer. The previous code writes the pointer only to metadata.storage-ref-id, and its storage GC reads the reference from there. Removing the key while that code runs leaves the rows it writes without a pointer (unreadable after the deploy), and it can make the GC delete an object that a live row still references. When the pre-flight query returns 0, tiered offload is off and the queue is empty, there is nothing to backfill and this risk does not apply.

0. Pre-flight

SELECT count(*) FROM pg_inherits
 WHERE inhparent = 'file_data'::regclass;   -- 16

-- Before the next two queries, turn off `:tiered-file-data-storage` in every
-- 2.19 worker and wait for the pending offloads: that task writes storage rows
-- with the default db backend.
--
-- Whether the storage backend was ever used. When this is 0 there is no
-- pointer in the metadata and step 6 is skipped; it does not prove the backend
-- is off.
SELECT count(*) FROM file_data
 WHERE backend = 'storage' AND jsonb_exists(metadata, 'storage-ref-id');

-- offloads queued or running on the 2.19 task table. Must be 0.
SELECT count(*) AS pending_offloads
  FROM task
 WHERE name = 'offload-file-data'
   AND status NOT IN ('completed', 'failed');

-- Every online ALTER uses a short lock timeout and is retried, so a long
-- transaction cannot make it queue and block the table behind it.
SELECT pid, state, now() - xact_start AS age, left(query, 60)
  FROM pg_stat_activity
 WHERE xact_start IS NOT NULL
   AND now() - xact_start > interval '1 minute';

1. Column

BEGIN;
SET LOCAL lock_timeout = '3s';
ALTER TABLE file_data
  ADD COLUMN storage_object_id uuid NULL;
COMMIT;

Skip this step if the column already exists. The change is catalog-only (a nullable column without a default), so it takes ACCESS EXCLUSIVE for an instant. If it fails with lock timeout, wait for a quieter moment and retry: it did not block anything.

2. One index per partition, concurrently

CREATE INDEX CONCURRENTLY file_data_00__storage_object_id__idx
    ON file_data_00 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_01__storage_object_id__idx
    ON file_data_01 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_02__storage_object_id__idx
    ON file_data_02 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_03__storage_object_id__idx
    ON file_data_03 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_04__storage_object_id__idx
    ON file_data_04 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_05__storage_object_id__idx
    ON file_data_05 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_06__storage_object_id__idx
    ON file_data_06 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_07__storage_object_id__idx
    ON file_data_07 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_08__storage_object_id__idx
    ON file_data_08 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_09__storage_object_id__idx
    ON file_data_09 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_10__storage_object_id__idx
    ON file_data_10 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_11__storage_object_id__idx
    ON file_data_11 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_12__storage_object_id__idx
    ON file_data_12 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_13__storage_object_id__idx
    ON file_data_13 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_14__storage_object_id__idx
    ON file_data_14 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_15__storage_object_id__idx
    ON file_data_15 (storage_object_id);

Run each statement in autocommit: CREATE INDEX CONCURRENTLY cannot run inside a transaction block. An interrupted build leaves the index INVALID: drop it and retry.

SELECT c.relname, i.indisvalid
  FROM pg_index AS i
  JOIN pg_class AS c ON c.oid = i.indexrelid
 WHERE c.relname LIKE 'file_data_%storage_object_id%'
 ORDER BY c.relname;

3. Parent index and partition attachment

CREATE INDEX file_data__storage_object_id__idx
    ON ONLY file_data (storage_object_id);

ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_00__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_01__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_02__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_03__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_04__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_05__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_06__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_07__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_08__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_09__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_10__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_11__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_12__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_13__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_14__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_15__storage_object_id__idx;

The step creates the index unconditionally: if an index with that name already exists, drop it and rebuild it. Verify the result before continuing:

SELECT i.indisvalid, i.indpred IS NULL AS plain, i.indexprs IS NULL AS no_expression
  FROM pg_index AS i
 WHERE i.indexrelid = 'file_data__storage_object_id__idx'::regclass;
-- t | t | t

SELECT count(*) FROM pg_inherits
 WHERE inhparent = 'file_data__storage_object_id__idx'::regclass;
-- 16

4. Foreign key, online

BEGIN;
SET LOCAL lock_timeout = '3s';
SET LOCAL statement_timeout = '10min';   -- measure first; see below
ALTER TABLE file_data
  ADD CONSTRAINT file_data_storage_object_id_fkey
  FOREIGN KEY (storage_object_id) REFERENCES storage_object(id)
  ON DELETE NO ACTION DEFERRABLE INITIALLY IMMEDIATE;
COMMIT;

The column is NULL until step 6, but PostgreSQL still scans the table (or the index built in the previous steps) to prove there are no violations. The SHARE ROW EXCLUSIVE lock blocks writes for the whole scan, not only while the lock is acquired: lock_timeout bounds the wait for the lock and statement_timeout bounds the scan. A timeout aborts the ALTER, releases the writes and lets you retry in a quieter window; measure the scan in staging and set the budget accordingly. Retry on lock timeout. If the constraint already exists, do not add it again: verify its shape with the query in step 7. Drop it first only if the shape is wrong.

Creating it here, before the backfill, keeps the full validation scan out of the stopped window: the backfill validates each row it writes.

INITIALLY IMMEDIATE is required. The storage GC catches the violation inside the chunk transaction and retries the chunk; an INITIALLY DEFERRED check fires at commit, outside that transaction, and the GC would never move past a blocked chunk.

Temporary index for pending backfill rows — online

Build this partial index online, before the stop. It covers only rows that still need migration. Each successful batch removes its rows from the index, so later batches do not scan completed rows. Drop the parent index after the backfill.

Run each CREATE INDEX CONCURRENTLY in autocommit:

CREATE INDEX CONCURRENTLY file_data_00__storage_ref_pending__idx
    ON file_data_00 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_01__storage_ref_pending__idx
    ON file_data_01 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_02__storage_ref_pending__idx
    ON file_data_02 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_03__storage_ref_pending__idx
    ON file_data_03 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_04__storage_ref_pending__idx
    ON file_data_04 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_05__storage_ref_pending__idx
    ON file_data_05 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_06__storage_ref_pending__idx
    ON file_data_06 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_07__storage_ref_pending__idx
    ON file_data_07 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_08__storage_ref_pending__idx
    ON file_data_08 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_09__storage_ref_pending__idx
    ON file_data_09 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_10__storage_ref_pending__idx
    ON file_data_10 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_11__storage_ref_pending__idx
    ON file_data_11 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_12__storage_ref_pending__idx
    ON file_data_12 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_13__storage_ref_pending__idx
    ON file_data_13 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_14__storage_ref_pending__idx
    ON file_data_14 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_15__storage_ref_pending__idx
    ON file_data_15 (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');

Create and attach the matching parent index:

CREATE INDEX file_data__storage_ref_pending__idx ON ONLY file_data (file_id, id)
    WHERE backend = 'storage'
      AND storage_object_id IS NULL
      AND jsonb_exists(metadata, 'storage-ref-id');

ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_00__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_01__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_02__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_03__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_04__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_05__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_06__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_07__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_08__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_09__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_10__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_11__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_12__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_13__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_14__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
  ATTACH PARTITION file_data_15__storage_ref_pending__idx;

SELECT indisvalid, indpred IS NOT NULL AS partial
  FROM pg_index
 WHERE indexrelid = 'file_data__storage_ref_pending__idx'::regclass;
-- t | t

SELECT count(*) AS attached_partitions
  FROM pg_inherits
 WHERE inhparent = 'file_data__storage_ref_pending__idx'::regclass;
-- 16

5. Stop every instance

Stop every instance before the next step when the coordinated restart applies (see "When to run this"): the storage backend is on, the tiered offload is on or has pending offload-file-data tasks, or the pre-flight found rows with the metadata pointer. No rolling deploy.

When there is no storage writer (no PENPOT_FILE_DATA_BACKEND=storage, no :tiered-file-data-storage, no pending offload-file-data task) and the pre-flight count is 0, there is nothing to backfill: run steps 1-4 online, run the verification (step 7) and mark the step as applied (step 8) without stopping the service.

6. Backfill, only if the metadata pointer exists

Skip this step only when there is no storage writer (no PENPOT_FILE_DATA_BACKEND=storage, no :tiered-file-data-storage, no pending offload-file-data task) and the pre-flight count is 0: no row carries the pointer, and the new code reads all file data from the data column. If a writer was present, stop it and repeat the count before deciding; a writer can create the first row after the pre-flight.

WITH candidates AS MATERIALIZED (
  SELECT fd.file_id,
         fd.id,
         CASE WHEN (fd.metadata->>'storage-ref-id') ~*
                   '^[0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{12}$'
              THEN (fd.metadata->>'storage-ref-id')::uuid END AS soid
    FROM file_data AS fd
    JOIN storage_object AS so
      ON so.id = CASE WHEN (fd.metadata->>'storage-ref-id') ~*
                          '^[0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{12}$'
                     THEN (fd.metadata->>'storage-ref-id')::uuid END
   WHERE fd.backend = 'storage'
     AND fd.storage_object_id IS NULL
     AND jsonb_exists(fd.metadata, 'storage-ref-id')
     AND (fd.metadata->>'storage-ref-id') ~*
         '^[0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{12}$'
   ORDER BY fd.file_id, fd.id
   LIMIT 5000
 ),
 updated AS (
  UPDATE file_data AS fd
     SET storage_object_id = c.soid,
         metadata = fd.metadata - 'storage-ref-id'
    FROM candidates AS c
   WHERE fd.file_id = c.file_id
     AND fd.id = c.id
  RETURNING 1
)
SELECT count(*) AS rows_updated FROM updated;

Run the statement repeatedly, committing each batch separately, until rows_updated is 0. Each batch updates at most 5,000 rows and removes the metadata key in the same update that sets the column. The partial index keeps later candidate searches focused on rows still pending. The MATERIALIZED CTE keeps the UUID cast away from values that fail the regex. The join to storage_object excludes orphan pointers, and the foreign key validates each written row. Corrupt and orphan pointers remain untouched for investigation; they must be resolved before the unreadable-row verification can pass.

After all batches finish, remove the temporary index:

DROP INDEX file_data__storage_ref_pending__idx;

The migrated key is not retained as a rollback copy. A rollback to the previous binary after the backfill requires restoring the backup.

7. Verify

Do not mark the migration as applied until every check passes.

-- Column.
SELECT data_type, is_nullable
  FROM information_schema.columns
 WHERE table_schema = 'public'
   AND table_name = 'file_data'
   AND column_name = 'storage_object_id';
-- uuid | YES

-- Rows that will be unreadable. Must be 0. The backfill only fills pointers it
-- can resolve, so a non-zero count means pointers that were already corrupt or
-- orphan: inspect them and accept them before continuing. With the backfill
-- skipped this must still be 0, or the storage backend was enabled after all.
-- A row pointing at a storage_object with deleted_at set is legitimate and
-- restorable, not unreadable: the backfill fills it on purpose and the
-- restore flow revives the object, so it is not counted here.
SELECT count(*) FROM file_data
 WHERE backend = 'storage' AND storage_object_id IS NULL;

-- Migrated rows no longer retain the old pointer. Must be 0.
SELECT count(*) FROM file_data
 WHERE backend = 'storage' AND storage_object_id IS NOT NULL
   AND jsonb_exists(metadata, 'storage-ref-id');

-- Index: valid, plain, with every partition attached.
SELECT i.indisvalid, i.indpred IS NULL AS plain, i.indexprs IS NULL AS no_expression
  FROM pg_index AS i
 WHERE i.indexrelid = 'file_data__storage_object_id__idx'::regclass;
SELECT count(*) FROM pg_inherits
 WHERE inhparent = 'file_data__storage_object_id__idx'::regclass;   -- 16

-- Foreign key: deferrable, checked immediately, no action, validated.
SELECT condeferrable, condeferred, confdeltype, convalidated
  FROM pg_constraint
 WHERE conname = 'file_data_storage_object_id_fkey'
   AND conrelid = 'file_data'::regclass;
-- t | f | a | t

8. Mark the migration as applied

Only after step 7 passes:

INSERT INTO migrations (module, step)
VALUES ('main', '0159-add-file-data-storage-object-id')
ON CONFLICT (module, step) DO NOTHING;

Check it:

SELECT module, step, created_at FROM migrations
 WHERE module = 'main' AND step = '0159-add-file-data-storage-object-id';

The row is the only thing that skips the migration at startup. Without it, the migration runs at the next start and repeats the backfill and the foreign key.

9. Start the new instances

Start every instance you stopped in step 5. When there was no storage writer and the pre-flight count was 0, nothing was stopped, and the rolling deploy simply finishes.

Post-deploy checks

SELECT conname, condeferrable, condeferred, confdeltype, convalidated
  FROM pg_constraint
 WHERE conname = 'file_data_storage_object_id_fkey';

SELECT count(*) FROM file_data
 WHERE backend = 'storage' AND storage_object_id IS NULL;

-- The delete check must use the index, not a sequential scan.
BEGIN;
EXPLAIN (ANALYZE, BUFFERS)
  DELETE FROM storage_object WHERE id = '<uuid of a deleted object with no references>';
ROLLBACK;