Merge remote-tracking branch 'origin/staging' into develop

This commit is contained in:
Andrey Antukh 2026-11-13 09:49:27 +01:00
commit 8715450f53
8 changed files with 257 additions and 54 deletions

View File

@ -137,7 +137,7 @@ Since `put-object!` uses backend-specific operations (`impl/resolve-backend` + `
- The automatic migration (`0158-partition-file-change`) refuses to run when `file_change` is above 10 GiB total size or 2M rows (`pg_total_relation_size` + `pg_class.reltuples`, catalog reads). It analyzes the table first, because `reltuples` is -1 without statistics. It raises with the path of `operations/2.20.0-file-change-partition.md`, because the copy happens inside the migration transaction and a big instance would restart in a loop. Below the guard it raises `statement_timeout` to 30 minutes for the copy and the index builds, and analyzes the new partitions at the end, after the copy and the indexes, so the planner does not start with empty statistics.
- Its primary key is `(file_id, id)`: a partitioned table needs the partition key in the primary key, so every access filters by `file_id`. From 2.20 the four snapshot commands (`update-file-snapshot`, `delete-file-snapshot`, `lock-file-snapshot`, `unlock-file-snapshot`) require the `file-id` param, because they use it to prune the partition; without it they answer `:params-validation`.
- It has a `kind` column: `1` is a snapshot (it has a label and its data lives in `file_data`) and `2` is a revision (it has no label and it holds the changes). The values are in `app.features.file-changes`.
- `kind` is a deliberate reserve, not a leftover: today no query filters by it, the queries still tell a snapshot from a revision by its label (`label IS NOT NULL`), and `kind` takes over when they are rewritten. No constraint ties `kind` and `label` together on purpose: the day `kind` is the source of truth, that constraint would have to go.
- `kind` is a deliberate reserve, not a leftover: today no query filters by it, the queries still tell a snapshot from a revision by its label (`label IS NOT NULL`), and `kind` takes over when they are rewritten. No constraint ties `kind` and `label` together on purpose: the day `kind` is the source of truth, that constraint would have to go. It stays nullable on purpose: the previous version inserts without it, and a rollback to it has to keep writing; the `NOT NULL` comes in a later migration.
- Every write of the application to a row of `file_change` moves `updated_at` to the moment of the write (`lock-by!`, `unlock!`, the snapshot rename and delete, `delete-object`, the restore of a file, and the `legacy-db` snapshot write of `fdata`). The actions of the foreign keys (`ON DELETE SET NULL` on `locked_by` and `profile_id`) change the row without touching it; during the partition both tables carry the same keys, so both receive them. The runbook of the partition uses it as the watermark of its delta passes: a write of the application that leaves it behind is lost at the cutover.
- `file_change_legacy` is a frozen copy of the previous layout of the partition: it keeps the rows and the indexes of the old table. A new instance (empty table) does not keep it. On an upgraded instance the table stays until the operator drops it by hand days later (`DROP TABLE file_change_legacy;`, see `operations/2.20.0-file-change-partition.md`); nothing drops it automatically.
- A revision carries only the delta of a save (the changes). The features of the file are not repeated on it: nobody read them from there.

View File

@ -22,6 +22,10 @@
-- they are rewritten. That is why no constraint ties the two together: the
-- day `kind` is the source of truth, the constraint would have to go.
--
-- It stays nullable on purpose: the previous version inserts without it,
-- and a rollback to that version has to keep writing. The `NOT NULL` comes
-- in a later migration, once the rollback window is closed.
--
-- The old table is kept aside as `file_change_legacy`: a frozen copy of the
-- previous layout. It is dropped by hand, days later (or right here, when it
-- is empty: on a new instance there is nothing to keep).
@ -127,7 +131,10 @@ CREATE TABLE file_change (
deleted_at timestamptz,
migrations text[],
locked_by uuid,
kind smallint NOT NULL,
-- Nullable on purpose: the previous version inserts without `kind`, and
-- a rollback to it has to keep writing. The `NOT NULL` comes in a later
-- migration, once the rollback window is closed.
kind smallint,
CONSTRAINT file_change_kind_check CHECK (kind IN (1, 2))
) PARTITION BY HASH (file_id);

View File

@ -84,21 +84,26 @@
WHERE attrelid = 'file_change'::regclass
AND attname = 'data'"])
(mapv :type))))
(t/is (true? (query "SELECT attnotnull AS notnull
;; Nullable on purpose: a rollback to the previous version inserts
;; without `kind` and has to keep writing (migration 0158).
(t/is (false? (query "SELECT attnotnull AS notnull
FROM pg_attribute
WHERE attrelid = 'file_change'::regclass
AND attname = 'kind'"))))
(t/deftest inserting-a-row-without-a-kind-fails
;; The column has no default on purpose: an insert that forgets it has to
;; fail instead of storing the wrong kind silently.
(t/deftest inserting-a-row-without-a-kind-succeeds
;; The column is nullable on purpose: the previous version inserts without
;; it, and a rollback to that version has to keep writing. The backfill of
;; a later NOT NULL migration turns these NULLs into 1/2.
(let [profile (th/create-profile* 1)
file (th/create-file* 1 {:profile-id (:id profile)
:project-id (:default-project-id profile)})]
(t/is (thrown? java.sql.SQLException
(db/exec! th/*pool* ["INSERT INTO file_change (file_id, id)
VALUES (?, gen_random_uuid())"
(:id file)])))))
(db/exec! th/*pool* ["INSERT INTO file_change (file_id, id)
VALUES (?, gen_random_uuid())"
(:id file)])
(t/is (nil? (query "SELECT kind AS k
FROM file_change
WHERE file_id = ?" (:id file))))))
(t/deftest inserting-a-row-with-an-unknown-kind-fails
(let [profile (th/create-profile* 1)

View File

@ -7,8 +7,8 @@
(ns backend-tests.migrations-partition-file-change-test
"Covers the data copy of 0158-partition-file-change: the rows of the old
table move to the partitioned one with their columns in place, the `kind`
comes from the label (1 snapshot, 2 revision), and the legacy table keeps
the rows."
comes from the label (1 snapshot, 2 revision), it stays nullable so a
rollback keeps writing, and the legacy table keeps the rows."
(:require
[app.common.time :as ct]
@ -188,8 +188,23 @@
AND attname IN ('data_ref_id', 'data_backend')"])
(mapv :name)))))
(t/testing "kind stays nullable so a rollback keeps writing"
;; The previous version inserts without `kind`: the column
;; must accept NULL, like the runbook's staging column.
(t/is (false? (:notnull (db/exec-one! conn ["SELECT attnotnull AS notnull
FROM pg_attribute
WHERE attrelid = 'file_change'::regclass
AND attname = 'kind'"]))))
(let [rb-id (uuid/next)]
(db/insert! conn :file-change {:id rb-id :file-id fid :revn 99})
(t/is (nil? (:kind (db/exec-one! conn ["SELECT kind
FROM file_change
WHERE file_id = ? AND id = ?"
fid rb-id]))))))
(db/exec! conn [(str "DROP SCHEMA " schema " CASCADE")])))))
(t/deftest drops-the-legacy-table-when-it-is-empty
;; On a new instance there is nothing to keep: the migration leaves no
;; legacy table behind. The relation is asserted on its own, so a missing

View File

@ -659,7 +659,8 @@
(merge (meta it))))
(dwu/start-undo-transaction undo-id)
(dch/commit-changes changes)
(dwtp/propagate-workspace-tokens) ;; Make the new instance get the token values from the current file, not from the component's library
;; Make the new instance get the token values from the current file, not from the component's library
(dwtp/propagate-workspace-tokens {:page-id (:id page) :root-ids [(:id new-shape)]})
(ptk/data-event :layout/update {:ids [(:id new-shape)]})
(dws/select-shapes (d/ordered-set (:id new-shape)))
(when start-move?
@ -1172,7 +1173,7 @@
(dch/commit-changes changes)
;; Make the new instance get the token values from the current file,
;; not from the component's library
(dwtp/propagate-workspace-tokens)
(dwtp/propagate-workspace-tokens {:page-id (:id page) :root-ids [(:id new-shape)]})
(when (and (features/active-feature? state "render-wasm/v1")
(seq new-text-ids))
(dwwt/resize-wasm-text-all new-text-ids {:skip-component-sync? true})))

View File

@ -97,8 +97,11 @@
(->> (map (fn [[value attrs]] [attrs {value #{object-id}}]) attrs-values-map)
(into {})))
(defn- collect-shapes-update-info [resolved-tokens objects]
(loop [items (seq objects)
(defn- collect-shapes-update-info
"Collect the token updates of `shapes`, a seq of [id shape] entries of
`objects`."
[resolved-tokens objects shapes]
(loop [items (seq shapes)
frame-ids #{}
text-ids []
tokens {}]
@ -150,27 +153,49 @@
(pcb/with-page page)
(pcb/mod-page {:background color})))))))))
(defn- scoped-shapes
"Entries of `objects` for the shapes in `root-ids` and all their descendants."
[objects root-ids]
(into []
(comp (mapcat #(cfh/get-children-ids-with-self objects %))
(distinct)
(keep (fn [id]
(when-let [shape (get objects id)]
[id shape]))))
root-ids))
(defn propagate-tokens
"Propagate tokens values to all shapes and page backgrounds where they
are applied"
[state resolved-tokens]
are applied. A non-nil `scope` of `:page-id` and `:root-ids` limits it
to those shapes and their descendants in that page."
[state resolved-tokens {scope-page-id :page-id root-ids :root-ids :as scope}]
(let [file-id (get state :current-file-id)
current-page-id (get state :current-page-id)
fdata (dsh/lookup-file-data state file-id)
scoped? (some? scope)
tpoint (ct/tpoint-ms)]
(l/inf :status "START" :hint "propagate-tokens")
(->> (rx/concat
(rx/of current-page-id)
(->> (rx/from (:pages fdata))
(rx/filter (fn [id] (not= id current-page-id)))))
(l/inf :status "START" :hint "propagate-tokens" :scoped scoped?)
(->> (if scoped?
(rx/of scope-page-id)
(rx/concat
(rx/of current-page-id)
(->> (rx/from (:pages fdata))
(rx/filter (fn [id] (not= id current-page-id))))))
(rx/mapcat
(fn [page-id]
(let [page
(dsh/get-page fdata page-id)
objects
(:objects page)
[attrs frame-ids text-ids]
(collect-shapes-update-info resolved-tokens (:objects page))
(collect-shapes-update-info resolved-tokens
objects
(if scoped?
(scoped-shapes objects root-ids)
objects))
actions
(actionize-shapes-update-info page-id attrs)
@ -188,7 +213,7 @@
(when (seq observable) (apply rx/merge observable))
(when (seq normal) (rx/concat-all (rx/of normal)))
(when (:background-token page)
(when (and (not scoped?) (:background-token page))
(rx/of (update-canvas-background page-id resolved-tokens)))
(->> (rx/from frame-ids)
@ -207,31 +232,37 @@
(l/inf :status "END" :hint "propagate-tokens" :elapsed elapsed)))))))
(defn propagate-workspace-tokens
[]
(ptk/reify ::propagate-workspace-tokens
ptk/WatchEvent
(watch [_ state _]
(let [tokens-lib (dsh/lookup-tokens-lib state)
tokens-status (dsh/lookup-tokens-status state)]
(when (and tokens-lib tokens-status)
(when-let [tokens-tree (cfo/get-tokens-in-active-sets tokens-status tokens-lib)]
(->> (if (contains? cf/flags :tokenscript)
(rx/of (ts/resolve-tokens tokens-tree))
(sd/resolve-tokens tokens-tree))
(rx/mapcat
(fn [sd-tokens]
(let [undo-id (js/Symbol)]
(rx/concat
(rx/of (dwu/start-undo-transaction undo-id :timeout false))
"Apply the resolved values of the active tokens to the shapes that use
them. Without `scope` it covers every page of the file. A `scope` of
`:page-id` and `:root-ids` limits it to those shapes and their
descendants."
([]
(propagate-workspace-tokens nil))
([scope]
(ptk/reify ::propagate-workspace-tokens
ptk/WatchEvent
(watch [_ state _]
(let [tokens-lib (dsh/lookup-tokens-lib state)
tokens-status (dsh/lookup-tokens-status state)]
(when (and tokens-lib tokens-status)
(when-let [tokens-tree (cfo/get-tokens-in-active-sets tokens-status tokens-lib)]
(->> (if (contains? cf/flags :tokenscript)
(rx/of (ts/resolve-tokens tokens-tree))
(sd/resolve-tokens tokens-tree))
(rx/mapcat
(fn [sd-tokens]
(let [undo-id (js/Symbol)]
(rx/concat
(rx/of (dwu/start-undo-transaction undo-id :timeout false))
;; FIXME: now the tokens propagations is done by accumulating the update-shapes
;; into a single commit-changes. This is not really the best way, the token application
;; should be done with a changes_builder and sending only one `commit-changes` instead
;; of creating lots of `update-shapes`.
(rx/of (dwsh/update-shapes-buffer-start))
(->> (propagate-tokens state sd-tokens)
(rx/catch #(rx/concat
(rx/of (dwsh/update-shapes-buffer-stop))
(rx/throw %))))
(rx/of (dwsh/update-shapes-buffer-stop))
(rx/of (dwu/commit-undo-transaction undo-id)))))))))))))
;; FIXME: now the tokens propagations is done by accumulating the update-shapes
;; into a single commit-changes. This is not really the best way, the token application
;; should be done with a changes_builder and sending only one `commit-changes` instead
;; of creating lots of `update-shapes`.
(rx/of (dwsh/update-shapes-buffer-start))
(->> (propagate-tokens state sd-tokens scope)
(rx/catch #(rx/concat
(rx/of (dwsh/update-shapes-buffer-stop))
(rx/throw %))))
(rx/of (dwsh/update-shapes-buffer-stop))
(rx/of (dwu/commit-undo-transaction undo-id))))))))))))))

View File

@ -9,6 +9,7 @@
[app.common.math :as mth]
[app.common.test-helpers.components :as cthc]
[app.common.test-helpers.compositions :as ctho]
[app.common.test-helpers.files :as cthf]
[app.common.test-helpers.ids-map :as cthi]
[app.common.test-helpers.shapes :as cths]
[app.common.test-helpers.tokens :as ctht]
@ -36,6 +37,7 @@
(defn- setup-base-file
[]
(-> (ctht/sample-file-with-tokens
:page-label :page1
:lib-fn #(-> %
(ctob/add-set (ctob/make-token-set :id (cthi/new-id! :test-token-set)
:name "test-token-set"))
@ -439,3 +441,141 @@
(tohs/run-store-async
store step2 events identity))))
(defn- setup-file-with-stale-values
"Adds :frame2 (with a :rect1 child) and :frame3 in :page1, and :frame4
in :page2, whose stored radius differs from the resolved token value."
[]
(-> (setup-base-file)
(ctho/add-frame-with-child :frame2 :rect1)
(ctho/add-frame :frame3)
(ctht/apply-token-to-shape :frame2 "test-token-2" [:r1 :r2 :r3 :r4] [:r1 :r2 :r3 :r4] 10)
(ctht/apply-token-to-shape :rect1 "test-token-3" [:r1 :r2 :r3 :r4] [:r1 :r2 :r3 :r4] 10)
(ctht/apply-token-to-shape :frame3 "test-token-2" [:r1 :r2 :r3 :r4] [:r1 :r2 :r3 :r4] 10)
(cthf/add-sample-page :page2)
(ctho/add-frame :frame4)
(ctht/apply-token-to-shape :frame4 "test-token-2" [:r1 :r2 :r3 :r4] [:r1 :r2 :r3 :r4] 10)
(cthf/switch-to-page :page1)))
(t/deftest scoped-propagation-updates-only-the-scoped-tree
(t/async
done
(let [;; ==== Setup
file (setup-file-with-stale-values)
store (ths/setup-store file)
;; ==== Action
events [(dwtp/propagate-workspace-tokens
{:page-id (cthf/current-page-id file)
:root-ids [(cthi/id :frame2)]})]]
(tohs/run-store-async
store done events
(fn [new-state]
(let [;; ==== Get
file' (ths/get-file-from-state new-state)
frame2' (cths/get-shape file' :frame2)
rect1' (cths/get-shape file' :rect1)
frame3' (cths/get-shape file' :frame3)
frame4' (cths/get-shape file' :frame4 :page-label :page2)]
;; ==== Check
(t/is (= (:r1 frame2') 50))
(t/is (= (:r1 rect1') 75))
(t/is (= (:r1 frame3') 10))
(t/is (= (:r1 frame4') 10))))))))
(t/deftest unscoped-propagation-updates-every-shape
(t/async
done
(let [;; ==== Setup
file (setup-file-with-stale-values)
store (ths/setup-store file)
;; ==== Action
events [(dwtp/propagate-workspace-tokens)]]
(tohs/run-store-async
store done events
(fn [new-state]
(let [;; ==== Get
file' (ths/get-file-from-state new-state)
frame2' (cths/get-shape file' :frame2)
rect1' (cths/get-shape file' :rect1)
frame3' (cths/get-shape file' :frame3)
frame4' (cths/get-shape file' :frame4 :page-label :page2)]
;; ==== Check
(t/is (= (:r1 frame2') 50))
(t/is (= (:r1 rect1') 75))
(t/is (= (:r1 frame3') 50))
(t/is (= (:r1 frame4') 50))))))))
(defn- setup-file-with-stale-components
"Adds :component1 (from :frame1) and :component2 (from :frame2), and a
plain :frame3, all of them with a stored radius that differs from the
resolved token value."
[]
(-> (setup-base-file)
(ctht/apply-token-to-shape :frame1 "test-token-1" [:r1 :r2 :r3 :r4] [:r1 :r2 :r3 :r4] 10)
(cthc/make-component :component1 :frame1)
(ctho/add-frame :frame2)
(ctht/apply-token-to-shape :frame2 "test-token-2" [:r1 :r2 :r3 :r4] [:r1 :r2 :r3 :r4] 10)
(cthc/make-component :component2 :frame2)
(ctho/add-frame :frame3)
(ctht/apply-token-to-shape :frame3 "test-token-3" [:r1 :r2 :r3 :r4] [:r1 :r2 :r3 :r4] 10)))
(t/deftest instantiate-component-applies-token-values
(t/async
done
(let [;; ==== Setup
file (setup-file-with-stale-components)
store (ths/setup-store file)
;; ==== Action
events [(dwl/instantiate-component (:id file)
(cthi/id :component1)
(geom/point 0 0))]]
(tohs/run-store-async
store done events
(fn [new-state]
(let [;; ==== Get
copy' (dsh/lookup-shape new-state (first (dsh/lookup-selected new-state)))
file' (ths/get-file-from-state new-state)
frame3' (cths/get-shape file' :frame3)]
;; ==== Check
(t/is (= (get-in copy' [:applied-tokens :r1]) "test-token-1"))
(t/is (= (:r1 copy') 25))
(t/is (= (:r4 copy') 25))
(t/is (= (:r1 frame3') 10))))))))
(t/deftest component-swap-applies-token-values
(t/async
done
(let [;; ==== Setup
file (-> (setup-file-with-stale-components)
(cthc/instantiate-component :component1 :c-frame1))
store (ths/setup-store file)
;; ==== Action
events [(dwl/component-swap (cths/get-shape file :c-frame1)
(:id file)
(cthi/id :component2)
true)]]
(tohs/run-store-async
store done events
(fn [new-state]
(let [;; ==== Get
copy' (dsh/lookup-shape new-state (first (dsh/lookup-selected new-state)))
file' (ths/get-file-from-state new-state)
frame3' (cths/get-shape file' :frame3)]
;; ==== Check
(t/is (= (:component-id copy') (cthi/id :component2)))
(t/is (= (get-in copy' [:applied-tokens :r1]) "test-token-2"))
(t/is (= (:r1 copy') 50))
(t/is (= (:r4 copy') 50))
(t/is (= (:r1 frame3') 10))))))))

View File

@ -92,7 +92,8 @@ alone.
`file_id` everywhere.
- The table has 17 columns: `data_ref_id` and `data_backend` are gone (dead
since the file storage refactor), `data` is kept (the `legacy-db` backend
reads old payloads from it) and `kind` appears (1 = snapshot, 2 = revision).
reads old payloads from it) and `kind` appears (1 = snapshot, 2 = revision,
nullable until the rollback window closes).
- The old table is kept as `file_change_legacy`, a frozen copy of the
previous layout, and dropped days later.
- Almost all of the size of the table is dead space (see sizing), so the new
@ -305,7 +306,10 @@ CREATE TABLE file_change_new (
deleted_at timestamptz,
migrations text[],
locked_by uuid,
kind smallint NOT NULL,
-- Nullable on purpose, like the migration: the previous version inserts
-- without `kind`, and a rollback to it has to keep writing. The `NOT NULL`
-- comes in a later migration, once the rollback window is closed.
kind smallint,
CONSTRAINT file_change_kind_check CHECK (kind IN (1, 2))
) PARTITION BY HASH (file_id);