✨ Make legacy-db file data read-only (#13102)

legacy-db is no longer a write backend: fdata/upsert! rejects it
(:legacy-backend-read-only), handle-persistence stops writing
file.data and file_change.data, and the update-params schema of
fdata no longer accepts it. Reading old payloads keeps working, and
the file-data-backend default stays db.

The configuration guide, the backend config (a startup warning reads
PENPOT_FILE_DATA_BACKEND=legacy-db as db instead of failing every
save) and the storage memories say so, with the upgrade note for
deployments that still set the variable.

AI-assisted-by: deepseek-v4.1-flash
AI-assisted-by: inkling-2-small-free
This commit is contained in:
Andrey Antukh 2026-11-17 09:24:23 +01:00 committed by GitHub
parent a050a21fae
commit 114ce6ec03
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
7 changed files with 127 additions and 88 deletions

View File

@ -118,8 +118,8 @@ Since `put-object!` uses backend-specific operations (`impl/resolve-backend` + `
## File Data
- `file-data-backend` accepts `legacy-db`, `db`, or `storage`.
- `legacy-db` stores main data in `file.data` and snapshots in `file_change.data`.
- `file-data-backend` accepts `legacy-db`, `db`, or `storage`; `legacy-db` is read-only and the default is `db`. The configuration reads `legacy-db` as `db` at startup, with a warning (`normalize-file-data-backend`), so a deployment that kept the variable saves instead of failing on every write.
- `legacy-db` reads main data from `file.data` and snapshots from `file_change.data`; it no longer writes them (`fdata/upsert!` rejects it).
- `db` stores encoded data in `file_data.data`.
- `storage` stores encoded data in storage subsystem with `file-data` bucket, keeps `data` nil in `file_data` table, and points at the object with `file_data.storage_object_id`.
- `file_data.storage_object_id` is a `NO ACTION DEFERRABLE INITIALLY IMMEDIATE` foreign key to `storage_object(id)`: the database refuses to physically delete a referenced object. The `IMMEDIATE` is load-bearing: the RI violation fires inside the `storage-gc-deleted` chunk transaction, so the chunk rolls back, is logged at error level with a generic hint plus the `constraint` and `table` the database reports (any `NO ACTION` foreign key on `storage_object` can block the delete, not only `file_data`'s) and is retried on the next run; with `INITIALLY DEFERRED` the check would fire at commit, outside the catch, and the sweep would never move past the blocked chunk.
@ -143,13 +143,13 @@ Since `put-object!` uses backend-specific operations (`impl/resolve-backend` + `
- `file_change` is partitioned by `HASH (file_id)` in 16 partitions, with the same bounds as `file_data`, so a join between them is resolved partition by partition (`enable_partitionwise_join` is set on every pooled connection).
- The automatic migration (`0158-partition-file-change`) refuses to run when `file_change` is above 10 GiB total size or 2M rows (`pg_total_relation_size` + `pg_class.reltuples`, catalog reads). It analyzes the table first, because `reltuples` is -1 without statistics. It raises with the path of `operations/2.20.0-file-change-partition.md`, because the copy happens inside the migration transaction and a big instance would restart in a loop. Below the guard it raises `statement_timeout` to 30 minutes for the copy and the index builds, and analyzes the new partitions at the end, after the copy and the indexes, so the planner does not start with empty statistics.
- Its primary key is `(file_id, id)`: a partitioned table needs the partition key in the primary key, so every access filters by `file_id`. From 2.20 the four snapshot commands (`update-file-snapshot`, `delete-file-snapshot`, `lock-file-snapshot`, `unlock-file-snapshot`) require the `file-id` param, because they use it to prune the partition; without it they answer `:params-validation`.
- It has a `kind` column: `1` is a snapshot (it has a label and its data lives in `file_data`) and `2` is a revision (it has no label and it holds the changes). The values are in `app.features.file-changes`.
- It has a `kind` column: `1` is a snapshot (it has a label and its data lives in `file_data`; the read-only `legacy-db` backend keeps old payloads in `data`) and `2` is a revision (it has no label and it holds the changes). The values are in `app.features.file-changes`.
- `kind` is a deliberate reserve, not a leftover: today no query filters by it, the queries still tell a snapshot from a revision by its label (`label IS NOT NULL`), and `kind` takes over when they are rewritten. No constraint ties `kind` and `label` together on purpose: the day `kind` is the source of truth, that constraint would have to go. It stays nullable on purpose: the previous version inserts without it, and a rollback to it has to keep writing; the `NOT NULL` comes in a later migration.
- Every write of the application to a row of `file_change` moves `updated_at` to the moment of the write (`lock-by!`, `unlock!`, the snapshot rename and delete, `delete-object`, the restore of a file, and the `legacy-db` snapshot write of `fdata`). The actions of the foreign keys (`ON DELETE SET NULL` on `locked_by` and `profile_id`) change the row without touching it; during the partition both tables carry the same keys, so both receive them. The runbook of the partition uses it as the watermark of its delta passes: a write of the application that leaves it behind is lost at the cutover.
- Every write of the application to a row of `file_change` moves `updated_at` to the moment of the write (`lock-by!`, `unlock!`, the snapshot rename and delete, `delete-object`, the restore of a file). The actions of the foreign keys (`ON DELETE SET NULL` on `locked_by` and `profile_id`) change the row without touching it; during the partition both tables carry the same keys, so both receive them. The runbook of the partition uses it as the watermark of its delta passes: a write of the application that leaves it behind is lost at the cutover.
- `file_change_legacy` is a frozen copy of the previous layout of the partition: it keeps the rows and the indexes of the old table. A new instance (empty table) does not keep it. On an upgraded instance the table stays until the operator drops it by hand days later (`DROP TABLE file_change_legacy;`, see `operations/2.20.0-file-change-partition.md`); nothing drops it automatically.
- A revision carries only the delta of a save (the changes). The features of the file are not repeated on it: nobody read them from there.
- A snapshot carries the features and the migrations of the file.
- It keeps the `data` column: the `legacy-db` backend is still supported (migration `0157-migrate-legacy-file-data` moved the existing blobs, but the backend keeps reading and writing that column). It has no `data_ref_id` or `data_backend` columns: nothing read them since the file storage refactor.
- It keeps the `data` column for reading only: migration `0157-migrate-legacy-file-data` moved the existing blobs to `file_data`, and `legacy-db` remains as a read-only backend for the payloads still in `data`. `fdata/upsert!` rejects `legacy-db` and the `file-data-backend` default is `db`. It has no `data_ref_id` or `data_backend` columns: nothing read them since the file storage refactor.
- It has no index for the system snapshots. The old one filtered them by `data IS NOT NULL`, and with the payload living in `file_data` the column no longer tells snapshots apart. The new one could only filter them by `deleted_at IS NULL`, but the system snapshots always carry a deletion date in the future, so it would never match a row. The query that would use it filters that date with an "or", which no partial index can serve.
- The `objects-gc` purge groups each chunk by `file_id` and locks it with `file_id = ?` and `id = ANY(?::uuid[])`: a tuple `IN ((file_id, id), ...)` gives the planner no constant partition key, so it scans all 16 partitions in every chunk. Each group is one `FOR UPDATE SKIP LOCKED` query inside the chunk transaction.

View File

@ -91,7 +91,7 @@
## File data persistence
- File data backends are `legacy-db`, `db`, and `storage`. The storage backend keeps encoded file data in storage bucket `file-data`; the DB row stores nil data and points at the object with the `storage_object_id` column (FK `NO ACTION DEFERRABLE`).
- File data backends for writing are `db` and `storage`; `legacy-db` is a read-only alias for old payloads. The storage backend keeps encoded file data in storage bucket `file-data`; the DB row stores nil data and points at the object with the `storage_object_id` column (FK `NO ACTION DEFERRABLE`).
- `fdata/upsert!` touches the storage object the persisted row pointed at before replacing it; the pointer travels in the params (`file->file-data-params` from the read file map, `nil` for a new row, the row for the offload) and the params of `update-file!` cannot override it. `binfile.common/update-file!` merges the read file map with the fields to change, so the pointer survives; `insert-file!` drops it because an INSERT never replaces a persisted object. The column is the only source of truth and the metadata no longer carries `storage-ref-id` (the migration removes it from the backfilled rows and the write path never stores it); the release is a coordinated restart when the storage backend is in use (the previous code cannot read the new rows), while the default `db` backend reads the same data column and tolerates a rolling deploy. Responses built from a file map go through `bfc/file->response`, which strips the pointer and a residual `storage-ref-id` from the metadata.
- Pointer-map fragments are persisted separately as type `fragment`, and only modified pointer maps are written. A pointer keeps its id while nothing changes; content (`assoc`/`without`) and metadata (`with-meta`) changes produce a fresh id, so a fragment row is never rewritten after it is written. Re-assoc with an identical value returns the same map in array and hash maps alike, so a save keeps the fragment id of every page it does not change.
- `enable-objects-map` skips unloaded pointer pages: their fragment was already wrapped when it was written, and loading it only to wrap it again costs one page read per untouched page on every save. A page with a plain fragment is wrapped when it is first loaded or edited; a data migration that enables `fdata/objects-map` must rewrite the data itself.
@ -105,7 +105,7 @@
- `fdata/realize` combines pointer realization and object-map realization. Use it before operations that need complete in-memory file data instead of pointer placeholders.
- A `file_data` row is one of three classes, told apart by `type`: `main` is the file's current state and carries `id = file_id`; `snapshot` is a point-in-time copy whose `id` is the one its `file_change` row shares; `fragment` is a piece of the pointer-map, referenced from the file's data.
- Their lifecycles differ: `main` and `snapshot` live with the file (dated when it is deleted, undated when it is restored), while a `fragment` carries no date and is collected by `file_gc` once the file stops referencing it. Neither is a session buffer, so do not read a `fragment` as an ephemeral revision.
- File snapshots (versions) keep their metadata row in `file_change` and their payload in `file_data` (`type = 'snapshot'`). With the `db` backend `file_change.data` stays NULL: that column is only written by the `legacy-db` backend, plus the rows written before the 2025-07 file storage refactor.
- File snapshots (versions) keep their metadata row in `file_change` and their payload in `file_data` (`type = 'snapshot'`). With the `db` backend `file_change.data` stays NULL: that column is only read by the `legacy-db` backend (it is no longer written), plus the rows written before the 2025-07 file storage refactor.
- Tell a snapshot apart by `label IS NOT NULL` (with `created_by` / `deleted_at` as needed), never by `file_change.data IS NOT NULL`: the label holds with every file data backend, and the payload column stopped meaning "this is a snapshot" when the payload moved out. Queries that identify versions must use the label. `0157-migrate-legacy-file-data` moved the labelled leftovers to `file_data` and deleted the unlabelled payload rows, so from 2.19 on the label is the only marker there is.
- The version quota counts of `app.rpc.quotes` (`sql:get-snapshots-per-file`, `sql:get-snapshots-per-team`) are served by the partial index `file_change__user_snapshots__idx`, whose predicate mirrors their where clause; changing one without the other drops the index only scan and the count walks the whole table again.
- Revisions live only in `file_change` and have no `file_data` counterpart. They are born already scheduled for deletion one hour later (`app.rpc.commands.files-update`) and the snapshot paths cannot reach one, because they resolve their row through a `label IS NOT NULL` filter.

View File

@ -10,6 +10,7 @@
[app.common.data :as d]
[app.common.exceptions :as ex]
[app.common.flags :as flags]
[app.common.logging :as l]
[app.common.schema :as sm]
[app.common.time :as ct]
[app.common.uri :as u]
@ -408,6 +409,22 @@
(def explain-config
(sm/explainer schema:config))
(defn- normalize-file-data-backend
"The file data moved to `file_data` (migration `0157-migrate-legacy-file-data`), so
`file.data` and `file_change.data` are read-only leftovers and the `legacy-db`
back end is read-only: a write against it fails on every save. Accepting it would
start a healthy instance that cannot save a single file, so it reads as `db`,
which serves the legacy rows too. Warn instead of aborting: the operator may
still be halfway through the migration."
[config]
(if (= "legacy-db" (c/get config :file-data-backend))
(do
(l/warn :hint "file-data-backend=legacy-db is read-only, using db"
:file-data-backend "legacy-db"
:replacement "db")
(assoc config :file-data-backend "db"))
config))
(defn read-config
"Reads the configuration from environment variables and decodes all
known values."
@ -415,7 +432,8 @@
(->> (read-env prefix)
(merge default)
(normalize-tenant)
(decode-config)))
(decode-config)
(normalize-file-data-backend)))
(def version
(v/parse (or (some-> (io/resource "version.txt")

View File

@ -182,7 +182,7 @@
result))
(defn- handle-persistence
[cfg {:keys [type backend id file-id data] :as params}]
[cfg {:keys [backend id file-id data] :as params}]
(let [expected (:storage-object-id params)]
(cond
(= backend "storage")
@ -212,43 +212,6 @@
{:result (upsert-in-database cfg (assoc params :storage-object-id nil) expected)
:storage-object-id nil}
(= backend "legacy-db")
{:result
(cond
(= type "main")
(do
(db/delete! cfg :file-data
{:id id :file-id file-id :type "main"}
{::db/return-keys false})
(db/update! cfg :file
{:data data}
{:id file-id}
{::db/return-keys false}))
(= type "snapshot")
(do
(db/delete! cfg :file-data
{:id id :file-id file-id :type "snapshot"}
{::db/return-keys false})
;; every mutation of `file_change` moves `updated_at`: the partition
;; runbook uses it as the watermark of the delta passes
(db/update! cfg :file-change
{:data data
:updated-at (ct/now)}
{:file-id file-id :id id}
{::db/return-keys false}))
(= type "fragment")
(upsert-in-database cfg
(-> (dissoc params :metadata)
(assoc :backend "db")
(assoc :storage-object-id nil))
expected)
:else
(throw (RuntimeException. "not implemented")))
:storage-object-id nil}
:else
(throw (IllegalArgumentException.
(str "backend '" backend "' not supported"))))))
@ -264,7 +227,13 @@
(defn- default-backend
[backend]
(or backend (cf/get :file-data-backend)))
(let [backend (or backend (cf/get :file-data-backend))]
(when (= "legacy-db" backend)
(ex/raise :type :validation
:code :legacy-backend-read-only
:hint "the legacy-db file data backend is read-only; set the file data backend to db or storage"
:backend backend))
backend))
(defn decode-metadata
[metadata]
@ -277,7 +246,7 @@
[:id ::sm/uuid]
[:type [:enum "main" "snapshot" "fragment"]]
[:file-id ::sm/uuid]
[:backend {:optional true} [:enum "db" "legacy-db" "storage"]]
[:backend {:optional true} [:enum "db" "storage"]]
[:metadata {:optional true} ctf/schema:file-metadata]
;; The storage object the persisted row pointed at before this write; it is
;; the object to touch when the row replaces it. Required so a new caller

View File

@ -6,9 +6,11 @@
(ns backend-tests.config-test
(:require
[app.common.logging :as l]
[app.common.schema :as sm]
[app.common.time :as ct]
[app.config :as cf]
[clojure.core :as c]
[clojure.test :as t]))
(t/deftest tenant-is-normalized-to-lowercase
@ -86,3 +88,51 @@
(t/testing "and falls back to a week"
(with-redefs [cf/config (dissoc cf/config :jobs-user-ttl)]
(t/is (= (ct/duration {:days 7}) (cf/get-jobs-user-ttl))))))
;; `legacy-db` is read-only: the file data moved to `file_data`, and
;; `file.data` / `file_change.data` are read-only leftovers. An instance
;; that keeps it starts healthy and then fails on every save. It is read
;; as `db` instead, which also serves the legacy rows.
(defn- read-config
"`read-config` with the environment replaced by `env`."
[env & {:keys [default] :or {default {}}}]
(with-redefs [cf/read-env (constantly env)]
(cf/read-config :default default)))
(defn- read-config-with-warning
"Reads the configuration and collects the warnings it emits. Logging is
emitted asynchronously, so the record is awaited (bounded)."
[env]
(let [records (atom [])]
(add-watch l/log-record ::capture
(fn [_ _ _ record] (swap! records conj record)))
(try
(let [config (read-config env)]
[(loop [remaining 100]
(let [warns (filter #(= :warn (::l/level %)) @records)]
(cond
(seq warns) warns
(zero? remaining) warns
:else (do (Thread/sleep 10) (recur (dec remaining))))))
config])
(finally
(remove-watch l/log-record ::capture)))))
(t/deftest read-config-replaces-the-read-only-legacy-db
(let [[warns config] (read-config-with-warning {:file-data-backend "legacy-db"})]
(t/testing "the configuration writes through db"
(t/is (= "db" (c/get config :file-data-backend))))
(t/testing "the operator is warned"
(t/is (contains? (into #{} (map #(c/get (::l/props %) :hint)) warns)
"file-data-backend=legacy-db is read-only, using db")))))
(t/deftest read-config-keeps-the-writable-back-ends
(doseq [backend ["db" "storage"]]
(t/testing backend
(t/is (= backend (c/get (read-config {:file-data-backend backend})
:file-data-backend))))))
(t/deftest read-config-takes-the-default-back-end
(t/is (= "db" (c/get (read-config {} :default cf/default)
:file-data-backend))))

View File

@ -5,55 +5,40 @@
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns backend-tests.fdata-upsert-test
"Covers the invariants of `fdata/upsert!` that the partition runbook relies
on."
"Covers the write contract of `fdata/upsert!` after the `legacy-db` backend
became read-only."
(:require
[app.common.time :as ct]
[app.common.uuid :as uuid]
[app.config :as cf]
[app.db :as db]
[app.features.fdata :as fdata]
[app.features.file-changes :as fchanges]
[backend-tests.helpers :as th]
[clojure.test :as t]))
(t/use-fixtures :once th/state-init)
(t/use-fixtures :each th/database-reset)
(t/deftest legacy-db-snapshot-moves-the-updated-at-watermark
;; The runbook of the partition picks the rows changed since the bulk copy
;; with `updated_at`, so every mutation of `file_change` has to move it. The
;; `legacy-db` snapshot path writes the payload into the row, and it used to
;; leave the watermark behind.
(let [profile (th/create-profile* 1)
file (th/create-file* 1 {:profile-id (:id profile)
:project-id (:default-project-id profile)})
id (uuid/next)
past (ct/in-past "2h")
payload (byte-array [1 2 3])]
(defn- upsert-cause
[params]
(try
(fdata/upsert! th/*system* params)
nil
(catch Exception cause
cause)))
(th/db-insert! :file-change
{:id id
:file-id (:id file)
:revn 1
:label "v1"
:kind fchanges/snapshot-kind
:updated-at past})
(t/is (true? (:written (fdata/upsert! th/*system*
{:id id
:file-id (:id file)
:type "snapshot"
:backend "legacy-db"
:storage-object-id nil
:data payload}))))
(let [row (th/db-get :file-change {:id id} {::db/remove-deleted false})]
(t/testing "the payload lands in the legacy column"
(t/is (= (seq payload) (seq ^bytes (:data row)))))
(t/testing "the watermark moves to the moment of the mutation"
(t/is (pos? (compare (inst-ms (:updated-at row)) (inst-ms past))))))))
(t/deftest legacy-db-param-is-rejected
;; `legacy-db` is not a write backend anymore: the schema refuses it before
;; touching the database.
(let [cause (upsert-cause {:id (uuid/next)
:file-id (uuid/next)
:type "snapshot"
:backend "legacy-db"
:storage-object-id nil
:data (byte-array [1 2 3])})]
(t/is (some? cause))
(t/is (= :assertion (:type (ex-data cause))))
(t/is (= :data-validation (:code (ex-data cause))))))
(t/deftest upsert-reports-the-pointer-it-created
;; The write reports the object it created, so a caller that persists again
@ -124,3 +109,19 @@
(t/is (= (:backend before) (:backend after)))
(t/is (= (:storage-object-id before) (:storage-object-id after)))
(t/is (= (vec (:data before)) (vec (:data after))))))))
(t/deftest legacy-db-config-is-rejected-as-read-only
;; The configuration value is kept for reading old data, but a write with
;; it as the default backend fails with a clear message.
(with-redefs [cf/config (assoc cf/config :file-data-backend "legacy-db")]
(let [cause (upsert-cause {:id (uuid/next)
:file-id (uuid/next)
:type "snapshot"
:storage-object-id nil
:data (byte-array [1 2 3])})]
(t/is (some? cause))
(t/is (= :validation (:type (ex-data cause))))
(t/is (= :legacy-backend-read-only (:code (ex-data cause)))))))
(t/deftest file-data-backend-defaults-to-db
(t/is (= "db" (cf/get :file-data-backend))))

View File

@ -837,12 +837,13 @@ These settings are equally useful if you have a Minio storage system.
__Since version 2.11.0__
You can change the default file data storage backend with `PENPOT_FILE_DATA_BACKEND` environment variable. Possible values are:
You can change the file data storage backend with `PENPOT_FILE_DATA_BACKEND` environment variable. Possible values are:
- `legacy-db`: the current default backend, continues storing the file data of files and snapshots in the same location as previous versions of Penpot (< 2.11.0), this is a conservative default behaviour and will be changed to `db` in next versions.
- `db`: stores the file data on an specific table (the future default backend).
- `db`: the default. Stores the file data on an specific table.
- `storage`: stores the file data using the objects storage system (S3 or FS, depending on which one is configured)
The `legacy-db` value is still accepted as a **read-only** alias for the payloads written by versions before 2.11.0: the backend reads them, but it no longer writes file data there. **If you upgrade a deployment with `PENPOT_FILE_DATA_BACKEND=legacy-db`, remove the variable (or set it to `db`) before the upgrade**: the backend reads `legacy-db` as `db` and logs a warning, so saves keep working, but remove the variable anyway so the deployment does not claim a read-only backend.
This also comes with an additional feature that allows offload the "inactive" files on file storage backend and leaves the database only for the active files. To enable it, you should use the `enable-tiered-file-data-storage` flag and `db` as file data storage backend.
```bash