* ♻️ Rework audit-log consumers and jobs substrate Fold the implementation, follow-up fixes, migrations and tests into one code commit. AI-assisted-by: gpt-6-luna * 📚 Consolidate 2.20.0 upgrade runbooks Keep the operational upgrade procedure and its supporting runbooks in a separate documentation commit. AI-assisted-by: gpt-6-luna * ♻️ Remove the audit-log flag and derive persistence at boot Persistence now follows the consumer flags instead of a separate switch. app.config computes cf/audit-persist? once at boot from :nexus and :admin-console, and every write site reads that value, so request paths no longer check flags per event. With no consumer enabled no full rows are written; telemetry shadows and webhook deliveries keep their own flags. The frontend wire signal stays :audit-log as a derived value. Update the 2.20 runbooks, memories and the devenv flags to match, and migrate the tests that used to toggle :audit-log directly. AI-assisted-by: hy4-flash-free * 🐛 Accept error-reporting batches and precompute sink flags push-audit-events now accepts crash batches with :error-reporting alone, so the frontend signal from get-environment-data matches a real destination. The reporters receive unhandled-exception events without storing any row. All per-event sink checks read booleans computed once at boot: webhooks, telemetry and error reporting join audit-persist? in app.config. The test fixture and with-audit-config rebind all four values together, and the tests that toggled those flags directly were migrated. Address F2 of the review of the previous commit. F1 stays deferred to a separate plan. AI-assisted-by: hy4-flash-free * 💄 Match the layout cljfmt expects in the flags set The first comment of the varia set sat on its own line after the opening delimiter, and cljfmt joins such a comment to the delimiter line, so the Common Tests lint step failed on common/src/app/common/flags.cljc. Keep the comment on the opening line, the same shape the login and email sets already use. Note in mem:scripts/ci that the backend lint lints ../common/src but does not format-check it, so common has to be passed to scripts/ci as well. AI-assisted-by: deepseek-v4-flash
21 KiB
2.20.0 — File data storage object foreign key
Runbook for the release that moves the file data storage pointer out of
file_data.metadata->>'storage-ref-id' and into a column,
file_data.storage_object_id, with a foreign key to storage_object(id).
When to run this
The migration 0159-add-file-data-storage-object-id does the whole job on a
fresh database and on small instances: it adds the column, backfills the
pointer, creates the index and the foreign key.
A large instance builds everything online and keeps the stopped window as short as possible:
- the column, the index and the foreign key are built online, without stopping the service;
- the backfill runs in the stopped window only if the metadata pointer exists; when the storage backend is off, the pre-flight finds no pointer, tiered offload is off and the queue is empty, there is nothing to backfill and the new binaries roll out like any other deploy;
- the step is then marked as applied in the
migrationstable, so the migration body does not run at startup.
The coordinated restart is required when the instance uses the storage backend
(PENPOT_FILE_DATA_BACKEND=storage), has :tiered-file-data-storage on or
pending offload-file-data tasks, or still has rows that carry the metadata
pointer. 2.19 writes storage rows through the tiered offload even with the
default db backend: turn the flag off and wait for the queue before the
pre-flight. The previous code writes and reads metadata.storage-ref-id, the
new code reads the column, and a rolling deploy would mix both. With the
default db backend, no tiered offload, no pending offloads and no row
carrying the pointer (see the pre-flight query), the previous and the new code
read the same file_data.data column, so a rolling deploy is safe.
This procedure is one-way. It has no schema rollback: after an attempt, restore the backup. The column is the only reference for the rows 2.20 writes.
Do not run the backfill while the previous version is serving when
PENPOT_FILE_DATA_BACKEND is configured to storage, or when the table still
has rows with the metadata pointer. The previous code writes the pointer only to
metadata.storage-ref-id, and its storage GC reads the reference from there.
Removing the key while that code runs leaves the rows it writes without a
pointer (unreadable after the deploy), and it can make the GC delete an object
that a live row still references. When the pre-flight query returns 0, tiered
offload is off and the queue is empty, there is nothing to backfill and this
risk does not apply.
0. Pre-flight
SELECT count(*) FROM pg_inherits
WHERE inhparent = 'file_data'::regclass; -- 16
-- Before the next two queries, turn off `:tiered-file-data-storage` in every
-- 2.19 worker and wait for the pending offloads: that task writes storage rows
-- with the default db backend.
--
-- Whether the storage backend was ever used. When this is 0 there is no
-- pointer in the metadata and step 6 is skipped; it does not prove the backend
-- is off.
SELECT count(*) FROM file_data
WHERE backend = 'storage' AND jsonb_exists(metadata, 'storage-ref-id');
-- offloads queued or running on the 2.19 task table. Must be 0.
SELECT count(*) AS pending_offloads
FROM task
WHERE name = 'offload-file-data'
AND status NOT IN ('completed', 'failed');
-- Every online ALTER uses a short lock timeout and is retried, so a long
-- transaction cannot make it queue and block the table behind it.
SELECT pid, state, now() - xact_start AS age, left(query, 60)
FROM pg_stat_activity
WHERE xact_start IS NOT NULL
AND now() - xact_start > interval '1 minute';
1. Column
BEGIN;
SET LOCAL lock_timeout = '3s';
ALTER TABLE file_data
ADD COLUMN storage_object_id uuid NULL;
COMMIT;
Skip this step if the column already exists. The change is catalog-only (a
nullable column without a default), so it takes ACCESS EXCLUSIVE for an
instant. If it fails with lock timeout, wait for a quieter moment and retry:
it did not block anything.
2. One index per partition, concurrently
CREATE INDEX CONCURRENTLY file_data_00__storage_object_id__idx
ON file_data_00 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_01__storage_object_id__idx
ON file_data_01 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_02__storage_object_id__idx
ON file_data_02 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_03__storage_object_id__idx
ON file_data_03 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_04__storage_object_id__idx
ON file_data_04 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_05__storage_object_id__idx
ON file_data_05 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_06__storage_object_id__idx
ON file_data_06 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_07__storage_object_id__idx
ON file_data_07 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_08__storage_object_id__idx
ON file_data_08 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_09__storage_object_id__idx
ON file_data_09 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_10__storage_object_id__idx
ON file_data_10 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_11__storage_object_id__idx
ON file_data_11 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_12__storage_object_id__idx
ON file_data_12 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_13__storage_object_id__idx
ON file_data_13 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_14__storage_object_id__idx
ON file_data_14 (storage_object_id);
CREATE INDEX CONCURRENTLY file_data_15__storage_object_id__idx
ON file_data_15 (storage_object_id);
Run each statement in autocommit: CREATE INDEX CONCURRENTLY cannot run inside
a transaction block. An interrupted build leaves the index INVALID: drop it
and retry.
SELECT c.relname, i.indisvalid
FROM pg_index AS i
JOIN pg_class AS c ON c.oid = i.indexrelid
WHERE c.relname LIKE 'file_data_%storage_object_id%'
ORDER BY c.relname;
3. Parent index and partition attachment
CREATE INDEX file_data__storage_object_id__idx
ON ONLY file_data (storage_object_id);
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_00__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_01__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_02__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_03__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_04__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_05__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_06__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_07__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_08__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_09__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_10__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_11__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_12__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_13__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_14__storage_object_id__idx;
ALTER INDEX file_data__storage_object_id__idx
ATTACH PARTITION file_data_15__storage_object_id__idx;
The step creates the index unconditionally: if an index with that name already exists, drop it and rebuild it. Verify the result before continuing:
SELECT i.indisvalid, i.indpred IS NULL AS plain, i.indexprs IS NULL AS no_expression
FROM pg_index AS i
WHERE i.indexrelid = 'file_data__storage_object_id__idx'::regclass;
-- t | t | t
SELECT count(*) FROM pg_inherits
WHERE inhparent = 'file_data__storage_object_id__idx'::regclass;
-- 16
4. Foreign key, online
BEGIN;
SET LOCAL lock_timeout = '3s';
SET LOCAL statement_timeout = '10min'; -- measure first; see below
ALTER TABLE file_data
ADD CONSTRAINT file_data_storage_object_id_fkey
FOREIGN KEY (storage_object_id) REFERENCES storage_object(id)
ON DELETE NO ACTION DEFERRABLE INITIALLY IMMEDIATE;
COMMIT;
The column is NULL until step 6, but PostgreSQL still scans the table (or the
index built in the previous steps) to prove there are no violations. The
SHARE ROW EXCLUSIVE lock blocks writes for the whole scan, not only while
the lock is acquired: lock_timeout bounds the wait for the lock and
statement_timeout bounds the scan. A timeout aborts the ALTER, releases
the writes and lets you retry in a quieter window; measure the scan in staging
and set the budget accordingly. Retry on lock timeout. If the constraint
already exists, do not add it again: verify its shape with the query in step 7.
Drop it first only if the shape is wrong.
Creating it here, before the backfill, keeps the full validation scan out of the stopped window: the backfill validates each row it writes.
INITIALLY IMMEDIATE is required. The storage GC catches the violation inside
the chunk transaction and retries the chunk; an INITIALLY DEFERRED check
fires at commit, outside that transaction, and the GC would never move past a
blocked chunk.
Temporary index for pending backfill rows — online
Build this partial index online, before the stop. It covers only rows that still need migration. Each successful batch removes its rows from the index, so later batches do not scan completed rows. Drop the parent index after the backfill.
Run each CREATE INDEX CONCURRENTLY in autocommit:
CREATE INDEX CONCURRENTLY file_data_00__storage_ref_pending__idx
ON file_data_00 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_01__storage_ref_pending__idx
ON file_data_01 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_02__storage_ref_pending__idx
ON file_data_02 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_03__storage_ref_pending__idx
ON file_data_03 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_04__storage_ref_pending__idx
ON file_data_04 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_05__storage_ref_pending__idx
ON file_data_05 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_06__storage_ref_pending__idx
ON file_data_06 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_07__storage_ref_pending__idx
ON file_data_07 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_08__storage_ref_pending__idx
ON file_data_08 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_09__storage_ref_pending__idx
ON file_data_09 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_10__storage_ref_pending__idx
ON file_data_10 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_11__storage_ref_pending__idx
ON file_data_11 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_12__storage_ref_pending__idx
ON file_data_12 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_13__storage_ref_pending__idx
ON file_data_13 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_14__storage_ref_pending__idx
ON file_data_14 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
CREATE INDEX CONCURRENTLY file_data_15__storage_ref_pending__idx
ON file_data_15 (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
Create and attach the matching parent index:
CREATE INDEX file_data__storage_ref_pending__idx ON ONLY file_data (file_id, id)
WHERE backend = 'storage'
AND storage_object_id IS NULL
AND jsonb_exists(metadata, 'storage-ref-id');
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_00__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_01__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_02__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_03__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_04__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_05__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_06__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_07__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_08__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_09__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_10__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_11__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_12__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_13__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_14__storage_ref_pending__idx;
ALTER INDEX file_data__storage_ref_pending__idx
ATTACH PARTITION file_data_15__storage_ref_pending__idx;
SELECT indisvalid, indpred IS NOT NULL AS partial
FROM pg_index
WHERE indexrelid = 'file_data__storage_ref_pending__idx'::regclass;
-- t | t
SELECT count(*) AS attached_partitions
FROM pg_inherits
WHERE inhparent = 'file_data__storage_ref_pending__idx'::regclass;
-- 16
5. Stop every instance
Stop every instance before the next step when the coordinated restart applies
(see "When to run this"): the storage backend is on, the tiered offload is on
or has pending offload-file-data tasks, or the pre-flight found rows with the
metadata pointer. No rolling deploy.
When there is no storage writer (no PENPOT_FILE_DATA_BACKEND=storage, no
:tiered-file-data-storage, no pending offload-file-data task) and the
pre-flight count is 0, there is nothing to backfill: run steps 1-4 online, run
the verification (step 7) and mark the step as applied (step 8) without
stopping the service.
6. Backfill, only if the metadata pointer exists
Skip this step only when there is no storage writer (no
PENPOT_FILE_DATA_BACKEND=storage, no :tiered-file-data-storage, no pending
offload-file-data task) and the pre-flight count is 0: no row carries the
pointer, and the new code reads all file data from the data column. If a
writer was present, stop it and repeat the count before deciding; a writer can
create the first row after the pre-flight.
WITH candidates AS MATERIALIZED (
SELECT fd.file_id,
fd.id,
CASE WHEN (fd.metadata->>'storage-ref-id') ~*
'^[0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{12}$'
THEN (fd.metadata->>'storage-ref-id')::uuid END AS soid
FROM file_data AS fd
JOIN storage_object AS so
ON so.id = CASE WHEN (fd.metadata->>'storage-ref-id') ~*
'^[0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{12}$'
THEN (fd.metadata->>'storage-ref-id')::uuid END
WHERE fd.backend = 'storage'
AND fd.storage_object_id IS NULL
AND jsonb_exists(fd.metadata, 'storage-ref-id')
AND (fd.metadata->>'storage-ref-id') ~*
'^[0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{12}$'
ORDER BY fd.file_id, fd.id
LIMIT 5000
),
updated AS (
UPDATE file_data AS fd
SET storage_object_id = c.soid,
metadata = fd.metadata - 'storage-ref-id'
FROM candidates AS c
WHERE fd.file_id = c.file_id
AND fd.id = c.id
RETURNING 1
)
SELECT count(*) AS rows_updated FROM updated;
Run the statement repeatedly, committing each batch separately, until
rows_updated is 0. Each batch updates at most 5,000 rows and removes the metadata
key in the same update that sets the column. The partial index keeps later
candidate searches focused on rows still pending. The MATERIALIZED CTE keeps
the UUID cast away from values that fail the regex. The join to
storage_object excludes orphan pointers, and the foreign key validates each
written row. Corrupt and orphan pointers remain untouched for investigation;
they must be resolved before the unreadable-row verification can pass.
After all batches finish, remove the temporary index:
DROP INDEX file_data__storage_ref_pending__idx;
The migrated key is not retained as a rollback copy. A rollback to the previous binary after the backfill requires restoring the backup.
7. Verify
Do not mark the migration as applied until every check passes.
-- Column.
SELECT data_type, is_nullable
FROM information_schema.columns
WHERE table_schema = 'public'
AND table_name = 'file_data'
AND column_name = 'storage_object_id';
-- uuid | YES
-- Rows that will be unreadable. Must be 0. The backfill only fills pointers it
-- can resolve, so a non-zero count means pointers that were already corrupt or
-- orphan: inspect them and accept them before continuing. With the backfill
-- skipped this must still be 0, or the storage backend was enabled after all.
-- A row pointing at a storage_object with deleted_at set is legitimate and
-- restorable, not unreadable: the backfill fills it on purpose and the
-- restore flow revives the object, so it is not counted here.
SELECT count(*) FROM file_data
WHERE backend = 'storage' AND storage_object_id IS NULL;
-- Migrated rows no longer retain the old pointer. Must be 0.
SELECT count(*) FROM file_data
WHERE backend = 'storage' AND storage_object_id IS NOT NULL
AND jsonb_exists(metadata, 'storage-ref-id');
-- Index: valid, plain, with every partition attached.
SELECT i.indisvalid, i.indpred IS NULL AS plain, i.indexprs IS NULL AS no_expression
FROM pg_index AS i
WHERE i.indexrelid = 'file_data__storage_object_id__idx'::regclass;
SELECT count(*) FROM pg_inherits
WHERE inhparent = 'file_data__storage_object_id__idx'::regclass; -- 16
-- Foreign key: deferrable, checked immediately, no action, validated.
SELECT condeferrable, condeferred, confdeltype, convalidated
FROM pg_constraint
WHERE conname = 'file_data_storage_object_id_fkey'
AND conrelid = 'file_data'::regclass;
-- t | f | a | t
8. Mark the migration as applied
Only after step 7 passes:
INSERT INTO migrations (module, step)
VALUES ('main', '0159-add-file-data-storage-object-id')
ON CONFLICT (module, step) DO NOTHING;
Check it:
SELECT module, step, created_at FROM migrations
WHERE module = 'main' AND step = '0159-add-file-data-storage-object-id';
The row is the only thing that skips the migration at startup. Without it, the migration runs at the next start and repeats the backfill and the foreign key.
9. Start the new instances
Start every instance you stopped in step 5. When there was no storage writer and the pre-flight count was 0, nothing was stopped, and the rolling deploy simply finishes.
Post-deploy checks
SELECT conname, condeferrable, condeferred, confdeltype, convalidated
FROM pg_constraint
WHERE conname = 'file_data_storage_object_id_fkey';
SELECT count(*) FROM file_data
WHERE backend = 'storage' AND storage_object_id IS NULL;
-- The delete check must use the index, not a sequential scan.
BEGIN;
EXPLAIN (ANALYZE, BUFFERS)
DELETE FROM storage_object WHERE id = '<uuid of a deleted object with no references>';
ROLLBACK;