mirror of
https://github.com/penpot/penpot.git
synced 2026-11-06 18:10:39 +00:00
d67a00c1d5
* ✨ Add Malli schema for storage metadata with dual decode Phase 1 of the storage_object.metadata migration: reads accept both Transit and plain JSON (sniffed by the marker) and always return the normalized shape; writes validate against a closed per-bucket Malli schema and still serialize as Transit unless the new :storage-metadata-as-json config flag is set. The 0155 migration normalizes existing rows inside Transit (reference to bucket, default bucket, drop of chunk leftovers) and is idempotent; large instances should fake it and run the batched script instead. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Address review findings on storage metadata Phase 1 Collapse the dead :reference leg of the gc-touched bucket fallback (the decode always sets :bucket on non-nil metadata, so it is only reachable with a NULL column) and fix its comment. Pin the write flag off in the transit-assuming metadata tests so the suite proves the same with the flag set, and add coverage for the flag rollback contract, JSON hash survival, NULL metadata in gc-touched, and the 0155 normalization statements. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Backfill NULLs, canonical buckets, comment fix Backfill NULL metadata columns in 0155 via coalesce (the key-missing rule already matches them), derive valid-buckets from the Malli schema dispatch entries so the list lives in one place, and correct the lookup-bucket fallback comment to NULL columns. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Defer corrupt metadata rows in storage gc-touched Decode touched rows individually so one non-map metadata value no longer aborts the whole chunk: corrupt rows are logged and deferred exactly one day in the same transaction, keeping their metadata intact for a later repair, while healthy rows process normally. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Address storage metadata phase 1 review findings Address the review findings on the storage metadata phase 1 branch: - Fix put-and-delete-object: it stored the object with ::sto/expired-at, so the row was already deleted and del-object! returned false. Add delete-expired-object-returns-false to keep the expired-delete case covered. - Cache the Malli decoder and encoder per process. Building them compiles the closed multi-dispatch schema, and decode-metadata runs on every read path (get-object, dedup probes, GC batches). - Catch Exception instead of Throwable in try-decode-row so JVM Errors are not deferred as corrupt metadata. - Add penpot_storage_gc_poison_total, emitted from storage-gc-touched; wire ::mtx/metrics into its handler. - Anchor the encoding sniff to the start of the document so a plain JSON value that begins with a Transit-looking prefix is not read as Transit. - Cover every bucket on both encodings, a JSON roundtrip through the jsonb column, nil metadata, the canonical bucket set and a poison-only GC chunk. - Rename private check-metadata! to check-metadata. AI-assisted-by: deepseek-v4.1-flash * ♻️ Simplify the storage metadata schema to a single map Replace the per-bucket :multi dispatch with a single closed map: the bucket is validated with ::sm/one-of over metadata-buckets (now a plain set) and the remaining keys are typed optional fields. Per-bucket enforcement shrinks to a one-line :fn guard requiring :file-id and :id for file-data, whose ids the GC reads to resolve references. - Drop the dead (sm/register! ::metadata ...): nothing references the schema by keyword. - Define tempfile-bucket and upload-session-bucket in the schema and alias them from app.storage, removing duplicated literals. - Keep content-type required and the map closed, so an unknown bucket or key still fails fast on write. AI-assisted-by: deepseek-v4.1-flash * ♻️ Drop input coercion from encode-metadata encode-metadata no longer runs the json-transformer decoder before validation. On the write path its only effect was coercing string UUIDs to UUID, and every producer already passes native UUIDs (the RPC profile-id, uuid/random, or binfile ids decoded as ::sm/uuid). Reads keep decoding, so stored Transit or JSON values still come back as native types. - Replace encode-accepts-string-uuids with encode-rejects-string-uuids, pinning the stricter contract. - Pass native UUIDs in encode-writes-plain-json-with-flag. AI-assisted-by: deepseek-v4.1-flash * 📚 Document each statement in the storage metadata migration Move the per-statement rules out of the header and add a comment to each UPDATE explaining what it does: drop chunk leftovers, promote the legacy "~:reference" to "~:bucket", drop residual "~:reference", and backfill the default bucket. The header keeps the scope, the encoding note, the `->` vs `?` note and the large-instance warning. AI-assisted-by: deepseek-v4.1-flash * 📚 Unwrap wrapped lines in the backend storage memory One line per bullet or paragraph, as mem:memory-maintenance requires. Only formatting; no content change. AI-assisted-by: deepseek-v4.1-flash * ♻️ Defer storage GC poison rows in their own transaction process-chunk! no longer takes poison-ids; it only processes the healthy chunk. The deferral moves to defer-poison! and process-touched! runs it in its own transaction, separate from the freeze/delete work. The loop still drains while there is chunk or poison, so a batch made only of poison rows does not leave healthy rows behind the LIMIT 10 waiting for the next run. Add a regression test: ten poison rows plus one healthy row with a later touched_at are all handled in the same run. AI-assisted-by: deepseek-v4.1-flash * ♻️ Declare per-bucket metadata requirements in one map Replace the file-data-specific predicate with bucket-requirements, a map from bucket to the extra keys it must carry. metadata-buckets is derived from its keys and a single generic :fn enforces presence, so a new bucket and its contract are one entry. organization now requires :organization-id; file-data keeps requiring :file-id and :id. Update the http-assets test helper to set organization-id for its organization objects. AI-assisted-by: deepseek-v4.1-flash * ♻️ Drop the ! suffix from storage GC helpers Rename the internal helpers in app.storage.gc-touched (process-chunk, defer-poison, mark-freeze-in-bulk, ...) to drop the trailing !. AI-assisted-by: deepseek-v4.1-flash * ♻️ Drop the ! suffix from storage GC deleted helpers Rename the internal helpers in app.storage.gc-deleted (clean-deleted, delete-sobjects, delete-give-up, ...) to drop the trailing !. AI-assisted-by: deepseek-v4.1-flash * 🐛 Fix dedup lookup for JSON-encoded storage metadata get-database-object-by-hash only matched the Transit keys, so once the :storage-metadata-as-json flag wrote plain JSON rows the dedup stopped finding them and duplicated blobs. Match both encodings with a UNION ALL of two indexable branches. - Add migration 0156 with the plain-key dedup index; the legacy 0068 index stays until Transit support is removed. - Cover it with a JSON dedup test and a Transit -> JSON cross test. AI-assisted-by: deepseek-v4.1-flash
53 lines
2.5 KiB
SQL
53 lines
2.5 KiB
SQL
--- Normalize storage_object.metadata inside Transit (Phase 1).
|
|
---
|
|
--- Brings every row to a canonical shape WITHOUT changing the encoding
|
|
--- family (still Transit JSON, "~:" keys), so old and new code keep
|
|
--- reading while it runs. Each statement below documents its own rule.
|
|
---
|
|
--- NOTE: key-existence is tested with `-> ... IS [NOT] NULL` instead of
|
|
--- the `?` operator because the migration runner prepares statements
|
|
--- and a bare `?` is parsed as a bind placeholder.
|
|
---
|
|
--- Idempotent: re-running changes zero rows.
|
|
--- NOTE for large instances (e.g. our production, ~33M rows): DO NOT run
|
|
--- this inline. Mark the migration as executed (fake) and run the batched
|
|
--- equivalent by hand: .agents/plans/2026-09-11-storage-normalize-phase1.sql
|
|
|
|
-- Drop leftovers from the old chunked-upload metadata: no code reads
|
|
-- "~:upload-id" / "~:chunk-index" anymore (#11651 moved chunks to
|
|
-- upload_session_chunk and objects-gc purges the orphan rows), and the
|
|
-- Phase 1 readers already ignore them on decode.
|
|
UPDATE storage_object
|
|
SET metadata = metadata - '~:upload-id' - '~:chunk-index'
|
|
WHERE (metadata -> '~:upload-id') IS NOT NULL
|
|
OR (metadata -> '~:chunk-index') IS NOT NULL;
|
|
|
|
-- Rows that carry only the legacy "~:reference": promote it to "~:bucket"
|
|
-- with the same value. The value comes as a Transit keyword ("~:xxx"), so
|
|
-- the "~:" prefix is stripped; plain strings are kept as they are.
|
|
UPDATE storage_object
|
|
SET metadata = (metadata - '~:reference')
|
|
|| jsonb_build_object('~:bucket',
|
|
CASE WHEN metadata->>'~:reference' LIKE '~:%'
|
|
THEN substr(metadata->>'~:reference', 3)
|
|
ELSE metadata->>'~:reference'
|
|
END)
|
|
WHERE (metadata -> '~:reference') IS NOT NULL
|
|
AND (metadata -> '~:bucket') IS NULL;
|
|
|
|
-- Rows that carry both keys: "~:bucket" wins and the residual
|
|
-- "~:reference" is dropped.
|
|
UPDATE storage_object
|
|
SET metadata = metadata - '~:reference'
|
|
WHERE (metadata -> '~:reference') IS NOT NULL
|
|
AND (metadata -> '~:bucket') IS NOT NULL;
|
|
|
|
-- Rows with neither key get the historical default bucket
|
|
-- "file-media-object" (the value `lookup-bucket` used to fall back to).
|
|
-- NULL columns match this rule too (`->` on NULL is NULL) and are
|
|
-- backfilled by coalesce.
|
|
UPDATE storage_object
|
|
SET metadata = coalesce(metadata, '{}') || '{"~:bucket": "file-media-object"}'
|
|
WHERE (metadata -> '~:bucket') IS NULL
|
|
AND (metadata -> '~:reference') IS NULL;
|