mirror of
https://github.com/penpot/penpot.git
synced 2026-11-20 17:11:15 +00:00
* ✨ Add Malli schema for storage metadata with dual decode Phase 1 of the storage_object.metadata migration: reads accept both Transit and plain JSON (sniffed by the marker) and always return the normalized shape; writes validate against a closed per-bucket Malli schema and still serialize as Transit unless the new :storage-metadata-as-json config flag is set. The 0155 migration normalizes existing rows inside Transit (reference to bucket, default bucket, drop of chunk leftovers) and is idempotent; large instances should fake it and run the batched script instead. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Address review findings on storage metadata Phase 1 Collapse the dead :reference leg of the gc-touched bucket fallback (the decode always sets :bucket on non-nil metadata, so it is only reachable with a NULL column) and fix its comment. Pin the write flag off in the transit-assuming metadata tests so the suite proves the same with the flag set, and add coverage for the flag rollback contract, JSON hash survival, NULL metadata in gc-touched, and the 0155 normalization statements. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Backfill NULLs, canonical buckets, comment fix Backfill NULL metadata columns in 0155 via coalesce (the key-missing rule already matches them), derive valid-buckets from the Malli schema dispatch entries so the list lives in one place, and correct the lookup-bucket fallback comment to NULL columns. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Defer corrupt metadata rows in storage gc-touched Decode touched rows individually so one non-map metadata value no longer aborts the whole chunk: corrupt rows are logged and deferred exactly one day in the same transaction, keeping their metadata intact for a later repair, while healthy rows process normally. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Address storage metadata phase 1 review findings Address the review findings on the storage metadata phase 1 branch: - Fix put-and-delete-object: it stored the object with ::sto/expired-at, so the row was already deleted and del-object! returned false. Add delete-expired-object-returns-false to keep the expired-delete case covered. - Cache the Malli decoder and encoder per process. Building them compiles the closed multi-dispatch schema, and decode-metadata runs on every read path (get-object, dedup probes, GC batches). - Catch Exception instead of Throwable in try-decode-row so JVM Errors are not deferred as corrupt metadata. - Add penpot_storage_gc_poison_total, emitted from storage-gc-touched; wire ::mtx/metrics into its handler. - Anchor the encoding sniff to the start of the document so a plain JSON value that begins with a Transit-looking prefix is not read as Transit. - Cover every bucket on both encodings, a JSON roundtrip through the jsonb column, nil metadata, the canonical bucket set and a poison-only GC chunk. - Rename private check-metadata! to check-metadata. AI-assisted-by: deepseek-v4.1-flash * ♻️ Simplify the storage metadata schema to a single map Replace the per-bucket :multi dispatch with a single closed map: the bucket is validated with ::sm/one-of over metadata-buckets (now a plain set) and the remaining keys are typed optional fields. Per-bucket enforcement shrinks to a one-line :fn guard requiring :file-id and :id for file-data, whose ids the GC reads to resolve references. - Drop the dead (sm/register! ::metadata ...): nothing references the schema by keyword. - Define tempfile-bucket and upload-session-bucket in the schema and alias them from app.storage, removing duplicated literals. - Keep content-type required and the map closed, so an unknown bucket or key still fails fast on write. AI-assisted-by: deepseek-v4.1-flash * ♻️ Drop input coercion from encode-metadata encode-metadata no longer runs the json-transformer decoder before validation. On the write path its only effect was coercing string UUIDs to UUID, and every producer already passes native UUIDs (the RPC profile-id, uuid/random, or binfile ids decoded as ::sm/uuid). Reads keep decoding, so stored Transit or JSON values still come back as native types. - Replace encode-accepts-string-uuids with encode-rejects-string-uuids, pinning the stricter contract. - Pass native UUIDs in encode-writes-plain-json-with-flag. AI-assisted-by: deepseek-v4.1-flash * 📚 Document each statement in the storage metadata migration Move the per-statement rules out of the header and add a comment to each UPDATE explaining what it does: drop chunk leftovers, promote the legacy "~:reference" to "~:bucket", drop residual "~:reference", and backfill the default bucket. The header keeps the scope, the encoding note, the `->` vs `?` note and the large-instance warning. AI-assisted-by: deepseek-v4.1-flash * 📚 Unwrap wrapped lines in the backend storage memory One line per bullet or paragraph, as mem:memory-maintenance requires. Only formatting; no content change. AI-assisted-by: deepseek-v4.1-flash * ♻️ Defer storage GC poison rows in their own transaction process-chunk! no longer takes poison-ids; it only processes the healthy chunk. The deferral moves to defer-poison! and process-touched! runs it in its own transaction, separate from the freeze/delete work. The loop still drains while there is chunk or poison, so a batch made only of poison rows does not leave healthy rows behind the LIMIT 10 waiting for the next run. Add a regression test: ten poison rows plus one healthy row with a later touched_at are all handled in the same run. AI-assisted-by: deepseek-v4.1-flash * ♻️ Declare per-bucket metadata requirements in one map Replace the file-data-specific predicate with bucket-requirements, a map from bucket to the extra keys it must carry. metadata-buckets is derived from its keys and a single generic :fn enforces presence, so a new bucket and its contract are one entry. organization now requires :organization-id; file-data keeps requiring :file-id and :id. Update the http-assets test helper to set organization-id for its organization objects. AI-assisted-by: deepseek-v4.1-flash * ♻️ Drop the ! suffix from storage GC helpers Rename the internal helpers in app.storage.gc-touched (process-chunk, defer-poison, mark-freeze-in-bulk, ...) to drop the trailing !. AI-assisted-by: deepseek-v4.1-flash * ♻️ Drop the ! suffix from storage GC deleted helpers Rename the internal helpers in app.storage.gc-deleted (clean-deleted, delete-sobjects, delete-give-up, ...) to drop the trailing !. AI-assisted-by: deepseek-v4.1-flash * 🐛 Fix dedup lookup for JSON-encoded storage metadata get-database-object-by-hash only matched the Transit keys, so once the :storage-metadata-as-json flag wrote plain JSON rows the dedup stopped finding them and duplicated blobs. Match both encodings with a UNION ALL of two indexable branches. - Add migration 0156 with the plain-key dedup index; the legacy 0068 index stays until Transit support is removed. - Cover it with a JSON dedup test and a Transit -> JSON cross test. AI-assisted-by: deepseek-v4.1-flash
247 lines
6.6 KiB
Clojure
247 lines
6.6 KiB
Clojure
;; This Source Code Form is subject to the terms of the Mozilla Public
|
|
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
|
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
|
;;
|
|
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
|
|
|
(ns app.storage.impl
|
|
"Storage backends abstraction layer."
|
|
(:require
|
|
[app.common.data.macros :as dm]
|
|
[app.common.exceptions :as ex]
|
|
[app.storage :as-alias sto]
|
|
[app.storage.schema :as stsch]
|
|
[buddy.core.codecs :as bc]
|
|
[buddy.core.hash :as bh]
|
|
[clojure.java.io :as jio]
|
|
[datoteka.io :as io])
|
|
(:import
|
|
java.nio.ByteBuffer
|
|
java.nio.file.Files
|
|
java.nio.file.Path
|
|
java.util.UUID))
|
|
|
|
(defn decode-row
|
|
"Decode the storage-object row fields"
|
|
[{:keys [metadata] :as row}]
|
|
(cond-> row
|
|
(some? metadata)
|
|
(assoc :metadata (stsch/decode-metadata metadata))))
|
|
|
|
;; --- API Definition
|
|
|
|
(defmulti put-object (fn [cfg _ _] (::sto/type cfg)))
|
|
|
|
(defmethod put-object :default
|
|
[cfg _ _]
|
|
(ex/raise :type :internal
|
|
:code :invalid-storage-backend
|
|
:context cfg))
|
|
|
|
(defmulti get-object-data (fn [cfg _] (::sto/type cfg)))
|
|
|
|
(defmethod get-object-data :default
|
|
[cfg _]
|
|
(ex/raise :type :internal
|
|
:code :invalid-storage-backend
|
|
:context cfg))
|
|
|
|
(defmulti get-object-bytes (fn [cfg _] (::sto/type cfg)))
|
|
|
|
(defmethod get-object-bytes :default
|
|
[cfg _]
|
|
(ex/raise :type :internal
|
|
:code :invalid-storage-backend
|
|
:context cfg))
|
|
|
|
(defmulti get-object-url (fn [cfg _ _] (::sto/type cfg)))
|
|
|
|
(defmethod get-object-url :default
|
|
[cfg _ _]
|
|
(ex/raise :type :internal
|
|
:code :invalid-storage-backend
|
|
:context cfg))
|
|
|
|
|
|
(defmulti del-object (fn [cfg _] (::sto/type cfg)))
|
|
|
|
(defmethod del-object :default
|
|
[cfg _]
|
|
(ex/raise :type :internal
|
|
:code :invalid-storage-backend
|
|
:context cfg))
|
|
|
|
(defmulti del-objects-in-bulk
|
|
"Delete multiple objects in bulk. Returns #{fail-ids} — the set of ids
|
|
whose blob deletion failed. Empty set = all succeeded."
|
|
(fn [cfg _] (::sto/type cfg)))
|
|
|
|
(defmethod del-objects-in-bulk :default
|
|
[cfg _]
|
|
(ex/raise :type :internal
|
|
:code :invalid-storage-backend
|
|
:context cfg))
|
|
|
|
(defmulti exists-object? (fn [cfg _] (::sto/type cfg)))
|
|
|
|
(defmethod exists-object? :default
|
|
[cfg _]
|
|
(ex/raise :type :internal
|
|
:code :invalid-storage-backend
|
|
:context cfg))
|
|
|
|
;; --- HELPERS
|
|
|
|
(defn uuid->hex
|
|
[^UUID v]
|
|
(let [buffer (ByteBuffer/allocate 16)]
|
|
(.putLong buffer (.getMostSignificantBits v))
|
|
(.putLong buffer (.getLeastSignificantBits v))
|
|
(bc/bytes->hex (.array buffer))))
|
|
|
|
(defn id->path
|
|
[id]
|
|
(let [tokens (->> (uuid->hex id)
|
|
(re-seq #"[\w\d]{2}"))
|
|
prefix (take 2 tokens)
|
|
suffix (drop 2 tokens)]
|
|
(str (apply str (interpose "/" prefix))
|
|
"/"
|
|
(apply str suffix))))
|
|
|
|
(defn coerce-id
|
|
[id]
|
|
(cond
|
|
(string? id) (parse-uuid id)
|
|
(uuid? id) id
|
|
:else (ex/raise :type :internal
|
|
:code :invalid-id-type
|
|
:hint "id should be string or uuid")))
|
|
|
|
(defprotocol IContentObject
|
|
(get-size [_] "get object size"))
|
|
|
|
(defprotocol IContentHash
|
|
(get-hash [_] "get precalculated hash"))
|
|
|
|
(defn- path->content
|
|
[^Path path ^long size]
|
|
(reify
|
|
IContentObject
|
|
(get-size [_] size)
|
|
|
|
jio/IOFactory
|
|
(make-reader [this opts]
|
|
(jio/make-reader this opts))
|
|
(make-writer [_ _]
|
|
(throw (UnsupportedOperationException. "not implemented")))
|
|
(make-input-stream [_ _]
|
|
(-> (io/input-stream path)
|
|
(io/bounded-input-stream size)))
|
|
(make-output-stream [_ _]
|
|
(throw (UnsupportedOperationException. "not implemented")))))
|
|
|
|
(defn- bytes->content
|
|
[^bytes data ^long size]
|
|
(reify
|
|
IContentObject
|
|
(get-size [_] size)
|
|
|
|
jio/IOFactory
|
|
(make-reader [this opts]
|
|
(jio/make-reader this opts))
|
|
(make-writer [_ _]
|
|
(throw (UnsupportedOperationException. "not implemented")))
|
|
(make-input-stream [_ _]
|
|
(-> (io/bytes-input-stream data)
|
|
(io/bounded-input-stream size)))
|
|
(make-output-stream [_ _]
|
|
(throw (UnsupportedOperationException. "not implemented")))))
|
|
|
|
(defn content
|
|
([data] (content data nil))
|
|
([data size]
|
|
(cond
|
|
(instance? java.nio.file.Path data)
|
|
(path->content data (or size (Files/size data)))
|
|
|
|
(instance? java.io.File data)
|
|
(content (.toPath ^java.io.File data) size)
|
|
|
|
(instance? String data)
|
|
(let [data (.getBytes ^String data "UTF-8")]
|
|
(bytes->content data (alength ^bytes data)))
|
|
|
|
(bytes? data)
|
|
(bytes->content data (or size (alength ^bytes data)))
|
|
|
|
;; (instance? InputStream data)
|
|
;; (do
|
|
;; (when-not size
|
|
;; (throw (UnsupportedOperationException. "size should be provided on InputStream")))
|
|
;; (make-content data size))
|
|
|
|
:else
|
|
(throw (IllegalArgumentException. "invalid argument type")))))
|
|
|
|
(defn wrap-with-hash
|
|
[content ^String hash]
|
|
(when-not (satisfies? IContentObject content)
|
|
(throw (UnsupportedOperationException. "`content` should be an instance of IContentObject")))
|
|
|
|
(when-not (satisfies? jio/IOFactory content)
|
|
(throw (UnsupportedOperationException. "`content` should be an instance of IOFactory")))
|
|
|
|
(reify
|
|
IContentObject
|
|
(get-size [_] (get-size content))
|
|
|
|
IContentHash
|
|
(get-hash [_] hash)
|
|
|
|
jio/IOFactory
|
|
(make-reader [_ opts]
|
|
(jio/make-reader content opts))
|
|
(make-writer [_ opts]
|
|
(jio/make-writer content opts))
|
|
(make-input-stream [_ opts]
|
|
(jio/make-input-stream content opts))
|
|
(make-output-stream [_ opts]
|
|
(jio/make-output-stream content opts))))
|
|
|
|
(defn calculate-hash
|
|
[resource]
|
|
(let [result (dm/with-open [input (io/input-stream resource)]
|
|
(-> (bh/blake2b-256 input)
|
|
(bc/bytes->hex)))]
|
|
(str "blake2b:" result)))
|
|
|
|
(defn resolve-backend
|
|
[storage backend-id]
|
|
(let [backend (get-in storage [::sto/backends backend-id])]
|
|
(when-not backend
|
|
(ex/raise :type :internal
|
|
:code :backend-not-configured
|
|
:hint (dm/fmt "backend '%' not configured" backend-id)))
|
|
(assoc backend ::sto/id backend-id)))
|
|
|
|
(defrecord StorageObject [id size created-at expired-at touched-at backend])
|
|
|
|
(ns-unmap *ns* '->StorageObject)
|
|
(ns-unmap *ns* 'map->StorageObject)
|
|
|
|
(defn storage-object
|
|
([id size created-at expired-at touched-at backend]
|
|
(StorageObject. id size created-at expired-at touched-at backend))
|
|
([id size created-at expired-at touched-at backend mdata]
|
|
(StorageObject. id size created-at expired-at touched-at backend mdata nil)))
|
|
|
|
(defn object?
|
|
[v]
|
|
(instance? StorageObject v))
|
|
|
|
(defn content?
|
|
[v]
|
|
(satisfies? IContentObject v))
|
|
|