penpot/operations/2.20.0-file-data-storage-object-id.md
Andrey Antukh e23ca02aa2
✨ Reference the file data storage object with a foreign key (#12769)
* ✨ Reference the file data storage object with a foreign key

The file data pointer lived in `file_data.metadata.storage-ref-id`, a JSON
key. The storage GC parsed it to find live references, and nothing stopped it
from physically deleting an object a live row still pointed at: a failure in
that scan lost the blob.

Migration 0159 moves the pointer to a nullable `storage_object_id` column on
`file_data`, backfills it from the metadata key, removes the key from the
rows it backfills, indexes the column and rebuilds the foreign key to
`storage_object(id)` as `NO ACTION DEFERRABLE INITIALLY IMMEDIATE`.
`INITIALLY IMMEDIATE` is deliberate: the GC catches the violation inside the
chunk transaction, while a deferred check would fire at commit, outside it,
and the sweep would never move past the blocked chunk.

The column is the only source of truth. The `storage-ref-id` entry is gone
from the file metadata schema, and the single point that serializes the
metadata drops it, so no write can resurrect it. `upsert!` does not query
the row: the pointer of the row being replaced travels in its params, and
the object is touched before the row is written. Fragments and snapshots
pass nil because a modified pointer always gets a new id; the offload passes
its own row; `insert-file!` drops it, since an INSERT never replaces a
persisted object.

`update-file!` takes the file map read from the database plus the fields to
change and merges them, so the pointer comes from the read map and no caller
can lose it by building a partial map. A map without `:data` raises
`:missing-file-data`: it is not `assert`, which is compiled out in the
deployed binary. The params cannot override the pointer or retarget the
write with `:id`/`:file-id`.

`bfc/file->response` strips the pointer from every response (`get-file`,
`create-file`, `duplicate-file`, the view-only bundle, the team access
request, the comments read, the library summary, the snapshot read and the
debug download). A `file-data` object is authentication-only, so a leaked id
would let any logged-in profile fetch the blob from `/assets/by-id`.
`get-file-fragment` resolves through the file data backend, so a fragment on
cold storage no longer hands nil to `blob/decode`, and answers only the keys
of its result schema.

The storage GC reference scan, the restore union and `objects-gc` read the
column. The offload reads `SELECT fd.*`, so its rows go through a
`decode-row` that decodes jsonb metadata and drops a JSON null, which the row
schema rejected; a row without a readable object raises
`:missing-storage-object` with the file and row ids. A chunk blocked by the
foreign key (SQLState 23503) is a real anomaly, since the reference scan
should have frozen the object: it stays an error with its own hint and the
chunk is retried on the next run until the reference is fixed.

The release is a coordinated restart, not a rolling deploy, and not
rollback-safe for the rows it writes: the previous code writes and scans the
metadata pointer, so a key it writes misses the backfill and removing it
loses the reference. `operations/2.20.0-file-data-storage-object-id.md`
builds the column, the index and the foreign key online and runs the
backfill with every instance stopped, then marks the step as applied; a
fresh or small instance does the whole job in the migration. The runbook
keeps the validation scan out of the stopped window and adds a lock timeout
and retry to the online statements.

Tests cover the foreign key shape, the backfill and its corrupt and orphan
guards, the rebuild of a missing index and of a wrong, non-deferrable or
deferred constraint, the deletion refused by a live reference, the metadata
cleanup, the offload row normalizations, the touched objects of the unshare
and the restore, the response stripping on every call site and the chunk
blocked by the foreign key. The backend memories and the developer guide
document the pointer, the GC behavior and the two migration paths.

Closes #12768

AI-assisted-by: deepseek-v4.1-flash
AI-assisted-by: inkling-2-small-free

* 🐛 Hold the file row lock across the migration write

`get-file` applies the pending migrations in memory and, when it finds
any, persists the result. That write read the row without a lock and
`update-file!` is an unconditional UPDATE, so a save that committed in
between was overwritten: the saved content, `revn` and `modified-at`
went back, the next save from that client failed with `:revn-conflict`
and the object the save left became an orphan the storage GC cannot
revisit once it is frozen.

The read now runs in a transaction of its own (`get-file` calls
`get-file*` inside `db/tx-run!` and accepts a cfg map only), so the
`file` row lock covers the re-check and the write. `get-file*` reads
the row once (`sql:get-file` plus `resolve-applied-migrations`) and
completes it with `attach-file-data`; only when a migration is pending
it reads the file again under `FOR UPDATE`, re-checks `need-migration?`
on that fresh map and persists it (`migrate-file`). The common case
stays at one row read and one migrations resolution, the migration case
keeps a single decoded copy in the heap, and a row that changed in
between is not overwritten. A migration found on the unlocked read is
applied in memory only, for the next read to persist.

`migrate-file` returns the reattached map when it persists, with the
`:storage-object-id` of the row the write left, and the realized map
when it does not; `:realize?` is what returns the read fully plain.
`fdata/upsert!` answers `{:written ... :storage-object-id ...}` instead
of a boolean, `update-file!` propagates it and the migrating read puts
it back in the map it returns, so a caller that persists again touches
the current object instead of the replaced one. `insert-file!` keeps
nil: an INSERT never replaces a persisted object.

The write also compares the stored pointer with the caller's value
(`IS NOT DISTINCT FROM`) and raises `:stale-file-data-pointer` when it
affects no row. Every writer holds the `file` row lock, so a pointer
that no longer matches is a protocol violation, not a race to retry.
The expected pointer is a separate argument because the storage branch
replaces `:storage-object-id` before the upsert, and the check lives in
`upsert-in-database`: the `legacy-db` branches do not use that SQL and
their count can legitimately be zero.

The save, snapshot and restore-backup reads take `FOR UPDATE` instead
of `FOR SHARE`: they inherit the lock, and the migration branch asks
for the exclusive one in the same transaction, so a shared lock that
upgrades deadlocks two concurrent readers. A data read rejects
`:lock-for-share?` for the same reason; only row-only reads accept it.
`sql:get-file-without-data` is a plain select now, not a CTE, which
makes locking legal without loading the data. Realization has a single
owner: `attach-file-data` only resolves and decodes, and `get-file*`
realizes once per path. The applied migrations no longer travel in the
`get-file-with-summary` response, cache hit and cache miss alike.

`embed-assets` reads each library without `:realize?`, so its
`:components` is an unloaded pointer map; `absorb-assets` walks it with
`vals`, `PointerMap.seq` calls `load!` and that raised with no loader
bound, so every export with `:merge-libraries` of a library with
components failed. The absorb runs under the library's own loader now.

The in-place binfile import (`import-file-and-overwrite*`,
`::overwrite`, `::bfc/file-id`) had no product caller and cannot carry
the previous `storage_object_id`, so overwriting a storage-backed row
would leave its object unreclaimed. `import-files!` rejects a cfg with
`::bfc/file-id` (`:not-supported`) and the helpers only that path used
go with it (`invalidate-thumbnails`, `*reference-file*`, the
`::overwrite` id branch of `index-object` and of `save-file!`, the
`on-conflict-do-nothing?` flag of the media insert).

Tests: a new `file-migration-read-test` pins the single read, the
locked re-read, the concurrent save interleaving, the double migration
and the read-only path; the upsert tests cover the return shape and the
stale pointer; the SQL recorder they share lives in
`backend-tests.helpers`, and the save, snapshot, restore and summary
sequences follow the new reads. The backend memories document the read
contract and the pointer.

AI-assisted-by: deepseek-v4.1-flash

* 🐛 Harden the get-file guard and skip its read-only transaction

The cfg guard of `get-file` was an `assert`, which `app.config` elides
when `:backend-asserts` is off, so a deployed binary accepted a bare
pool or connection while the docstring promised it raises. It throws
`IllegalArgumentException` now, the class the option guards of the same
namespace already use, and the existing test expects it instead of
`AssertionError`.

A read with `:read-only?` applies the pending migrations in memory and
writes nothing, so the transaction the migration write needs only costs
two round trips there. `get-file` runs it with `db/run!` instead: with
a cfg that carries a connection it still joins the caller's
transaction, and only the reads that would have opened their own skip
it. A lock option needs a read-write transaction from the caller,
because PostgreSQL rejects row locks in a read-only one; the docstring
states that.

Tests: the guard test expects the new class; a cfg marked
`::db/read-only` reads a pending migration without persisting it (the
config flag that fills that key is legacy, but the key is set by hand);
and a read-only read is pinned as not opening a transaction while the
same call without the option does. The backend memory documents the
split and the guard.

AI-assisted-by: deepseek-v4.1-flash
2026-11-14 19:27:18 +01:00

10 KiB

2.20.0 — File data storage object foreign key

Runbook for the release that moves the file data storage pointer out of file_data.metadata->>'storage-ref-id' and into a column, file_data.storage_object_id, with a foreign key to storage_object(id).

When to run this

The migration 0159-add-file-data-storage-object-id does the whole job on a fresh database and on small instances: it adds the column, backfills the pointer, creates the index and the foreign key.

A large instance builds everything online and keeps the stopped window as short as possible:

  • the column, the index and the foreign key are built online, without stopping the service;
  • the backfill runs in the stopped window only if the metadata pointer exists; when the storage backend was never enabled there is nothing to backfill and the new binaries roll out like any other deploy;
  • the step is then marked as applied in the migrations table, so the migration body does not run at startup.

The coordinated restart is required only when the instance uses the storage backend (PENPOT_FILE_DATA_BACKEND=storage) or still has rows that carry the metadata pointer. The previous code writes and reads metadata.storage-ref-id, the new code reads the column, and a rolling deploy would mix both. With the default db backend and no row carrying the pointer (see the pre-flight query), the previous and the new code read the same file_data.data column, so a rolling deploy is safe.

Do not run the backfill while the previous version is serving when PENPOT_FILE_DATA_BACKEND is configured to storage, or when the table still has rows with the metadata pointer. The previous code writes the pointer only to metadata.storage-ref-id, and its storage GC reads the reference from there. Removing the key while that code runs leaves the rows it writes without a pointer (unreadable after the deploy), and it can make the GC delete an object that a live row still references. When the pre-flight query returns 0 there is nothing to backfill and this risk does not apply.

0. Pre-flight

SELECT version();   -- 16.x: `NOT VALID` foreign keys on a partitioned table need 18

SELECT count(*) FROM pg_inherits
 WHERE inhparent = 'file_data'::regclass;   -- 16

-- Whether the storage backend was ever enabled. When this is 0 there is no
-- pointer in the metadata and step 6 is skipped.
SELECT count(*) FROM file_data
 WHERE backend = 'storage' AND jsonb_exists(metadata, 'storage-ref-id');

-- Every online ALTER uses a short lock timeout and is retried, so a long
-- transaction cannot make it queue and block the table behind it.
SELECT pid, state, now() - xact_start AS age, left(query, 60)
  FROM pg_stat_activity
 WHERE xact_start IS NOT NULL
   AND now() - xact_start > interval '1 minute';

1. Column

BEGIN;
SET LOCAL lock_timeout = '3s';
ALTER TABLE file_data
  ADD COLUMN storage_object_id uuid NULL;
COMMIT;

Skip this step if the column already exists. The change is catalog-only (a nullable column without a default), so it takes ACCESS EXCLUSIVE for an instant. If it fails with lock timeout, wait for a quieter moment and retry: it did not block anything.

2. One index per partition, concurrently

CREATE INDEX CONCURRENTLY file_data_00__storage_object_id__idx
    ON file_data_00 (storage_object_id);
-- ... up to file_data_15

Run each statement in autocommit: CREATE INDEX CONCURRENTLY cannot run inside a transaction block. An interrupted build leaves the index INVALID: drop it and retry.

SELECT c.relname, i.indisvalid
  FROM pg_index AS i
  JOIN pg_class AS c ON c.oid = i.indexrelid
 WHERE c.relname LIKE 'file_data_%storage_object_id%'
 ORDER BY c.relname;

3. Parent index and partition attachment

CREATE INDEX file_data__storage_object_id__idx
    ON ONLY file_data (storage_object_id);

ALTER INDEX file_data__storage_object_id__idx
  ATTACH PARTITION file_data_00__storage_object_id__idx;
-- ... up to file_data_15

The step creates the index unconditionally: if an index with that name already exists, drop it and rebuild it. Verify the result before continuing:

SELECT i.indisvalid, i.indpred IS NULL AS plain, i.indexprs IS NULL AS no_expression
  FROM pg_index AS i
 WHERE i.indexrelid = 'file_data__storage_object_id__idx'::regclass;
-- t | t | t

SELECT count(*) FROM pg_inherits
 WHERE inhparent = 'file_data__storage_object_id__idx'::regclass;
-- 16

4. Foreign key, online

BEGIN;
SET LOCAL lock_timeout = '3s';
ALTER TABLE file_data
  ADD CONSTRAINT file_data_storage_object_id_fkey
  FOREIGN KEY (storage_object_id) REFERENCES storage_object(id)
  ON DELETE NO ACTION DEFERRABLE INITIALLY IMMEDIATE;
COMMIT;

The column is NULL until step 6, so the validation finds nothing to check and the SHARE ROW EXCLUSIVE lock (it blocks writes, not reads) is brief. Retry on lock timeout. If the constraint already exists, do not add it again: verify its shape with the query in step 7. Drop it first only if the shape is wrong.

Creating it here, before the backfill, keeps the full validation scan out of the stopped window: the backfill validates each row it writes.

INITIALLY IMMEDIATE is required. The storage GC catches the violation inside the chunk transaction and retries the chunk; an INITIALLY DEFERRED check fires at commit, outside that transaction, and the GC would never move past a blocked chunk.

5. Stop every instance

Coordinated restart when the pre-flight count is not 0: stop all of them before the next step. No rolling deploy. When the count is 0 there is nothing to backfill: run steps 1-4 online, run the verification (step 7) and mark the step as applied (step 8) without stopping the service.

6. Backfill, only if the metadata pointer exists

Skip this step when the pre-flight count is 0: the storage backend was never enabled, no row carries the pointer, and the new code reads all file data from the data column.

WITH candidates AS MATERIALIZED (
  SELECT file_id,
         id,
         (metadata->>'storage-ref-id')::uuid AS soid
    FROM file_data
   WHERE backend = 'storage'
     AND storage_object_id IS NULL
     AND (metadata->>'storage-ref-id') ~*
         '^[0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{12}$'
)
UPDATE file_data AS fd
   SET storage_object_id = c.soid,
       metadata = fd.metadata - 'storage-ref-id'
  FROM candidates AS c
 WHERE fd.file_id = c.file_id
   AND fd.id = c.id
   AND EXISTS (SELECT 1 FROM storage_object AS so WHERE so.id = c.soid);

The MATERIALIZED CTE keeps the uuid cast away from values that did not pass the regex: without it PostgreSQL can inline the subquery and cast a corrupt value, aborting the statement. The foreign key already exists, so every written row is validated here instead of in a separate full scan.

7. Verify

Do not mark the migration as applied until every check passes.

-- Column.
SELECT data_type, is_nullable
  FROM information_schema.columns
 WHERE table_schema = 'public'
   AND table_name = 'file_data'
   AND column_name = 'storage_object_id';
-- uuid | YES

-- Rows that will be unreadable. Must be 0. The backfill only fills pointers it
-- can resolve, so a non-zero count means pointers that were already corrupt or
-- orphan: inspect them and accept them before continuing. With the backfill
-- skipped this must still be 0, or the storage backend was enabled after all.
-- A row pointing at a storage_object with deleted_at set is legitimate and
-- restorable, not unreadable: the backfill fills it on purpose and the
-- restore flow revives the object, so it is not counted here.
SELECT count(*) FROM file_data
 WHERE backend = 'storage' AND storage_object_id IS NULL;

-- Index: valid, plain, with every partition attached.
SELECT i.indisvalid, i.indpred IS NULL AS plain, i.indexprs IS NULL AS no_expression
  FROM pg_index AS i
 WHERE i.indexrelid = 'file_data__storage_object_id__idx'::regclass;
SELECT count(*) FROM pg_inherits
 WHERE inhparent = 'file_data__storage_object_id__idx'::regclass;   -- 16

-- Foreign key: deferrable, checked immediately, no action, validated.
SELECT condeferrable, condeferred, confdeltype, convalidated
  FROM pg_constraint
 WHERE conname = 'file_data_storage_object_id_fkey'
   AND conrelid = 'file_data'::regclass;
-- t | f | a | t

8. Mark the migration as applied

Only after step 7 passes:

INSERT INTO migrations (module, step)
VALUES ('main', '0159-add-file-data-storage-object-id')
ON CONFLICT (module, step) DO NOTHING;

Check it:

SELECT module, step, created_at FROM migrations
 WHERE module = 'main' AND step = '0159-add-file-data-storage-object-id';

The row is the only thing that skips the migration at startup. Without it, the migration runs at the next start and repeats the backfill and the foreign key.

9. Start the new instances

Start every instance you stopped in step 5. When the pre-flight count was 0 nothing was stopped, and the rolling deploy simply finishes.

Post-deploy checks

SELECT conname, condeferrable, condeferred, confdeltype, convalidated
  FROM pg_constraint
 WHERE conname = 'file_data_storage_object_id_fkey';

SELECT count(*) FROM file_data
 WHERE backend = 'storage' AND storage_object_id IS NULL;

-- The delete check must use the index, not a sequential scan.
BEGIN;
EXPLAIN (ANALYZE, BUFFERS)
  DELETE FROM storage_object WHERE id = '<uuid of a deleted object with no references>';
ROLLBACK;

Rollback

Before step 8:

ALTER TABLE file_data
  DROP CONSTRAINT IF EXISTS file_data_storage_object_id_fkey;

-- A partitioned index cannot be dropped with CONCURRENTLY; the plain drop
-- takes a brief ACCESS EXCLUSIVE.
DROP INDEX file_data__storage_object_id__idx;

ALTER TABLE file_data DROP COLUMN storage_object_id;

After step 8 the backfill has already removed the metadata key that the previous code reads, so a rollback to the previous binaries is not supported when the instance uses the storage backend: the files offloaded by the new code would be unreadable, and the recovery is a data restore, not a binary switch. With the default db backend and no storage rows, the previous binaries read the same data column and the rollback is a plain binary switch.