🐛 Fix shared files 500 on shapes with invalid child ids (#12528)

* 🐛 Fix shared files 500 on shapes with invalid child ids

Old files can list in :shapes ids that are not uuids or that point to
no shape. Migration 0033 walked them and hit the uuid assert in
get-shape, which throws an AssertionError. The shared files list only
caught Exception, so that Error broke the whole list with a 500.

Make migration 0033 skip those ids, and catch Throwable when loading
each shared file, so a broken file is skipped and logged.

Closes #12343

AI-assisted-by: claude-opus-5-5

* 🐛 Repair invalid child ids in migration 0033 v2

Skipping invalid child ids in migration 0033 was not enough: the ids
stayed in :shapes, so the file still failed schema validation and
could not load. Files that already applied the migration would also
never run a changed version of it.

The migration now repairs each container before the naming walk:
string ids are parsed with uuid/coerce, nil and ids that point to no
shape are dropped, the duplicate that 0032 adds when a child is
listed as a string is removed, and a :shapes list becomes a vector.
Only the shapes that change are re-assoc'ed, so a healthy ObjectsMap
is not marked as modified and its page is not rewritten.

It ships as 0033-normalize-main-instance-names-v2 at the same
position in available-migrations, so files that recorded the old id
run it again. That position also makes it parse strings before
0035-fix-dangling-children drops the ids it cannot find.

The shared files listing restores the interrupt flag when the cause
of a skipped file is an InterruptedException.

Tests cover children outside the root walk, legacy components, the
objects-map wrapper of the unrealized save path, a list turned into a
vector, a shape without a :shapes key, a file that applied the old
id, and the end-to-end load of a broken shared file.

Relates to #12343

AI-assisted-by: claude-opus-5-5
AI-assisted-by: inkling-2-small-free
This commit is contained in:
Eva Marco 2026-11-18 13:30:40 +01:00 committed by GitHub
parent dad4479b9c
commit fff44ec758
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
7 changed files with 260 additions and 42 deletions

View File

@ -35,3 +35,6 @@
- Closed schemas (`schema:stroke-attrs` and others) reject a key that was dropped from the declaration, so removing a declared attr needs a migration in the same change, for data already written with it.
- Rebase collision on a migration number: upstream keeps the id. Renumber the branch migration to the next free one, keep it last in `available-migrations`, and rename everything that spells the number — its test file and namespace (`files_migrations_00NN_test.cljc`), the test names, and comments in production code or memories that name it.
- Withdrawing a migration id: remove both the `defmethod` and the id from `available-migrations`; the replacement migration subsumes it and files that recorded the old id re-migrate through the replacement (persisted `:migrations` keep the old id via union — do not rewrite them, including playwright fixture JSONs). Idempotent/pure-cleanup migrations only; never withdraw one whose transform must not run again unless the replacement is safe for those files. Precedents: "0021-fix-shape-svg-attrs" -> "0035-fix-shape-svg-attrs-and-defs"; "legacy-36" -> "0036-clear-nil-keys-in-objects".
- A shipped migration whose behavior must change AND reach every file (most files already recorded its id, so `migrate` skips it): rename it to `<id>-v2` and swap the old id for the new one in `available-migrations` at the SAME position. Precedents: `0002-normalize-bool-content-v2`, `0003-convert-path-content-v2`, `0008-fix-library-colors-v4`, and `0035-fix-shape-svg-attrs-and-defs` withdrawing `0021-fix-shape-svg-attrs`. Files keep the old id in `:migrations`, which is harmless because `migrate` diffs against the available set. Cost: one full pass over every file on its next read (the `get-file` ETag hashes `available-migrations`, so clients refetch).
- The alternative is patch the old id plus add a new migration id for the files that already applied it. Then the position in the list decides which repair runs first: a coerce-then-drop repair must come before a drop-only one, or the strings are gone before they can be parsed.
- `0033-normalize-main-instance-names-v2` repairs the container before its naming walk: child ids stored as strings are parsed with `uuid/coerce`, ids missing from `:objects` are dropped, duplicates are removed (the one `0032` adds when a child is stored as a string). The walk only re-assocs a shape when the normalization changed it (`identical?` on the normalized shape), so a healthy container keeps its `ObjectsMap` instance: no `modified` flag, no page rewrite. `0035-fix-dangling-children` prunes whatever is left.

View File

@ -628,11 +628,14 @@
(fn [{:keys [id library-file-ids]}]
;; One shared file that fails to load (for example, a migration
;; error on old data) must not break the whole list; skip it
;; and report it with its id.
;; and report it with its id. Old data can also fail with a JVM
;; Error (an assert or a deep recursion), so catch Throwable.
(try
(let [file (get-file-with-summary cfg id)]
(assoc file :library-file-ids (db/decode-pgarray library-file-ids #{})))
(catch Exception cause
(catch Throwable cause
(when (instance? InterruptedException cause)
(.interrupt (Thread/currentThread)))
(l/error :hint "unable to load shared file"
:file-id (str id)
:team-id (str team-id)

View File

@ -9,6 +9,7 @@
[app.binfile.common :as bfc]
[app.common.features :as cfeat]
[app.common.files.changes-builder :as pcb]
[app.common.files.migrations :as fmg]
[app.common.files.tokens :as cfo]
[app.common.pprint :as pp]
[app.common.thumbnails :as thc]
@ -3355,17 +3356,49 @@
file2 (th/create-file* 2 {:profile-id (:id profile)
:project-id (:default-project-id profile)
:is-shared true})
get-file (deref #'files/get-file-with-summary)
broken (fn [cfg id]
(if (= id (:id file1))
(throw (ex-info "broken file" {:type :assertion}))
(get-file cfg id)))
out (with-redefs-fn {#'files/get-file-with-summary broken}
#(th/command! {::th/type :get-team-shared-files
::rpc/profile-id (:id profile)
:team-id (:default-team-id profile)}))]
get-file (deref #'files/get-file-with-summary)]
;; Old file data can fail with a JVM Error (an assert or a deep
;; recursion), not only with an exception.
(doseq [cause [(ex-info "broken file" {:type :assertion})
(AssertionError. "point instance expected")
(StackOverflowError.)]]
(let [broken (fn [cfg id]
(if (= id (:id file1))
(throw cause)
(get-file cfg id)))
out (with-redefs-fn {#'files/get-file-with-summary broken}
#(th/command! {::th/type :get-team-shared-files
::rpc/profile-id (:id profile)
:team-id (:default-team-id profile)}))]
(t/is (th/success? out) (str "skips file failing with " (class cause)))
(t/is (= #{(:id file2)} (into #{} (map :id) (:result out))))))))
(t/deftest get-team-shared-files-repairs-invalid-children-ids
;; A stored shared file that lists nil and a dangling id in :shapes and
;; has not run 0033 yet: the migrations repair it on load, so it is
;; listed instead of skipped.
(let [profile (th/create-profile* 1)
file (th/create-file* 1 {:profile-id (:id profile)
:project-id (:default-project-id profile)
:is-shared true})
page-id (first (get-in file [:data :pages]))
root [:data :pages-index page-id :objects uuid/zero :shapes]
broken (-> (bfc/get-file th/*system* (:id file) :realize? true)
(update-in root conj nil (str (uuid/next)))
(assoc :migrations (disj fmg/available-migrations
"0033-normalize-main-instance-names-v2")))
_ (db/tx-run! th/*system*
(fn [cfg]
(bfc/update-file! cfg broken ::bfc/reset-migrations? true)))
out (th/command! {::th/type :get-team-shared-files
::rpc/profile-id (:id profile)
:team-id (:default-team-id profile)})]
(t/is (th/success? out))
(t/is (= #{(:id file2)} (into #{} (map :id) (:result out))))))
(t/is (= #{(:id file)} (into #{} (map :id) (:result out))))
(t/is (= [] (get-in (bfc/get-file th/*system* (:id file) :realize? true) root))
"the repaired data is stored")))
(t/deftest get-team-shared-files-does-not-expose-the-storage-pointer
(let [profile (th/create-profile* 1 {:is-active true})

View File

@ -2212,7 +2212,34 @@
(update :pages-index d/update-vals update-container)
(d/update-when :components d/update-vals update-container))))
(defmethod migrate-data "0033-normalize-main-instance-names"
(defn- repair-shape-children
[objects shape]
(if-let [shapes (:shapes shape)]
(let [shapes' (into []
(comp (keep uuid/coerce)
(filter #(contains? objects %))
(distinct))
shapes)]
(if (and (vector? shapes) (= shapes shapes'))
shape
(assoc shape :shapes shapes')))
shape))
(defn- repair-container-children
[container]
(d/update-when container :objects
(fn [objects]
;; Only assoc the shapes that change, so an ObjectsMap is
;; not marked as modified when nothing is repaired.
(reduce-kv (fn [result id shape]
(let [shape' (repair-shape-children objects shape)]
(if (identical? shape shape')
result
(assoc result id shape'))))
objects
objects))))
(defmethod migrate-data "0033-normalize-main-instance-names-v2"
[data _]
;; In penpot shape names always have "/" characters normalized with one space between and after,
;; to form paths. But this is not enforced and may be places that still admit names like
@ -2221,15 +2248,13 @@
;; Here we normalize the names of all main instances, and a new file integrity validation has
;; been added to enforce this from now on.
;; Main instances in variants are a special case because they need to match the variant name.
;; Old files may also list in :shapes child ids stored as strings, nil, or ids that point to no
;; shape. They fail the shape schema and the uuid assert of get-shape, so the container is
;; repaired first (see `repair-container-children`). That repair is the reason for the `-v2`:
;; files that already applied the original migration need to run it again to get it.
(letfn [(update-shape [container shape-id]
(let [shape (ctn/get-shape container shape-id)]
;; A shape may reference a child that does not exist in the
;; container (a dangling reference). There is nothing to update
;; on it, and storing it would add a nil entry to the objects
;; map, making the whole file invalid.
(if (nil? shape)
container
(let [shape (cond
(let [shape (ctn/get-shape container shape-id)
shape' (cond
;; Variant container must have a normalized name
(ctk/is-variant-container? shape)
(let [name (:name shape)
@ -2262,13 +2287,19 @@
;; Any other shape does not need processing
:else shape)]
(reduce update-shape
(ctt/set-shape container shape)
(:shapes shape))))))
;; Only assoc the shape when the normalization changed it, so an
;; ObjectsMap is not marked as modified, and the page is not
;; rewritten, when there is nothing to normalize.
(reduce update-shape
(if (identical? shape shape')
container
(ctt/set-shape container shape'))
(:shapes shape'))))
(update-container
[container]
(let [root (ctn/get-container-root container)]
(let [container (repair-container-children container)
root (ctn/get-container-root container)]
(if root
(update-shape container (:id root))
container)))]
@ -2502,7 +2533,7 @@
"0030-remove-stroke-per-side-attr"
"0031-migrate-stroke-width-token-attr"
"0032-fix-children-not-in-parent"
"0033-normalize-main-instance-names"
"0033-normalize-main-instance-names-v2"
"0034-fix-path-content-offset"
"0035-fix-dangling-children"
"0036-clear-nil-keys-in-objects"

View File

@ -6,21 +6,21 @@
(ns common-tests.files-migrations-0033-test
"Regression coverage for the shape tree traversal of the
`0033-normalize-main-instance-names` migration.
`0033-normalize-main-instance-names-v2` migration.
The migration walks a container from its root down, following the
`:shapes` vectors. A file can hold a `:shapes` entry pointing to a
shape that is not in the `:objects` map (a dangling reference). The
traversal must skip it instead of storing a nil shape into the
`:objects` map, and must keep normalizing the names of the shapes
that do exist."
repair that runs before the walk drops it instead of storing a nil
shape into the `:objects` map, and the walk keeps normalizing the
names of the shapes that do exist."
(:require
[app.common.files.migrations :as cfm]
[app.common.types.shape :as cts]
[app.common.uuid :as uuid]
[clojure.test :as t]))
(def ^:private migration "0033-normalize-main-instance-names")
(def ^:private migration "0033-normalize-main-instance-names-v2")
(defn- make-root
[children]
@ -47,7 +47,7 @@
[component-id objects]
{:components {component-id {:id component-id :objects objects}}})
;; --- The guard: a missing child must not become a nil shape
;; --- A missing child is dropped, and never becomes a nil shape
(t/deftest missing-child-of-a-page-does-not-create-a-nil-shape
(let [page-id (uuid/next)
@ -61,8 +61,8 @@
"no nil key is added to the objects map")
(t/is (= 1 (count objects'))
"the objects map keeps only the shapes that exist")
(t/is (= root (get objects' uuid/zero))
"the root is left untouched")))
(t/is (= (assoc root :shapes []) (get objects' uuid/zero))
"the dangling child is dropped, the rest of the root is left untouched")))
(t/deftest missing-child-of-a-component-does-not-create-a-nil-shape
(let [component-id (uuid/next)
@ -76,8 +76,8 @@
"no nil key is added to the objects map")
(t/is (= 1 (count objects'))
"the objects map keeps only the shapes that exist")
(t/is (= root (get objects' uuid/zero))
"the component root is left untouched")))
(t/is (= (assoc root :shapes []) (get objects' uuid/zero))
"the dangling child is dropped, the rest of the component root is left untouched")))
(t/deftest missing-child-deep-in-the-tree-does-not-create-a-nil-shape
(let [page-id (uuid/next)
@ -92,8 +92,8 @@
(t/is (not (contains? objects' nil))
"no nil key is added to the objects map")
(t/is (= frame (get objects' frame-id))
"the parent of the missing child is left untouched")))
(t/is (= (assoc frame :shapes []) (get objects' frame-id))
"the missing child is dropped, the rest of its parent is left untouched")))
;; --- The behavior: names keep being normalized next to a missing child

View File

@ -166,7 +166,7 @@
file {:id file-id
:data data
:migrations (disj cfm/available-migrations
"0033-normalize-main-instance-names"
"0033-normalize-main-instance-names-v2"
"0034-fix-path-content-offset"
migration)}]

View File

@ -17,6 +17,7 @@
[app.common.test-helpers.shapes :as ths]
[app.common.test-helpers.variants :as thv]
[app.common.types.file :as ctf]
[app.common.types.objects-map :as omap]
[app.common.types.path :as path]
[app.common.types.shape :as cts]
[app.common.types.token :as ctt]
@ -715,7 +716,7 @@
"path points repaired before schema validation")))
(t/deftest migration-0033-normalize-main-instance-diff-from-component
(let [migration-id "0033-normalize-main-instance-names"
(let [migration-id "0033-normalize-main-instance-names-v2"
file (-> (thf/sample-file :file1)
(tho/add-simple-component :component1 :root1 :child1
:component-params {:path "Group / Subgroup"
@ -746,7 +747,7 @@
(t/is (= "Other Group/Other Subgroup / Other Name" (:name shape1')))))
(t/deftest migration-0033-normalize-variant-instance-diff-from-container
(let [migration-id "0033-normalize-main-instance-names"
(let [migration-id "0033-normalize-main-instance-names-v2"
file (-> (thf/sample-file :file1)
(thv/add-variant :variant1 :component1 :root1 :component2 :root2)
(ths/update-shape :variant1 :name "Group / Subgroup / Component")
@ -778,7 +779,7 @@
(t/is (= "Group / Subgroup/Component" (:name copy2')))))
(t/deftest migration-0033-normalize-variant-name-not-normalized
(let [migration-id "0033-normalize-main-instance-names"
(let [migration-id "0033-normalize-main-instance-names-v2"
file (-> (thf/sample-file :file1)
(thv/add-variant :variant1 :component1 :root1 :component2 :root2)
(ths/update-shape :variant1 :name "Group/Subgroup / Component")
@ -808,3 +809,150 @@
(t/is (= "Component" (:name component1')))
(t/is (= "Component" (:name component2')))
(t/is (= "Group / Subgroup/Component" (:name copy2')))))
(t/deftest migration-0033-fixes-children-ids
;; Old files may list in :shapes child ids stored as strings, nil or
;; ids that point to no shape; strings become uuids, the rest goes.
(let [migration-id "0033-normalize-main-instance-names-v2"
file (-> (thf/sample-file :file1)
(tho/add-frame-with-child :frame1 :child1)
(tho/add-simple-component :component1 :root1 :child2))
child1-id (thi/id :child1)
child2-id (thi/id :child2)
file (-> file
(ths/update-shape :frame1 :shapes
[nil "not-a-uuid" (str child1-id)
(uuid/next) child1-id])
(ths/update-shape :root1 :shapes [(str child2-id)]))
data' (cfm/migrate-data (ctf/file-data file) migration-id)
file' (assoc file :data data')]
(t/is (= [child1-id] (:shapes (ths/get-shape file' :frame1))))
(t/is (= [child2-id] (:shapes (ths/get-shape file' :root1))))
(t/is (= data' (cfm/migrate-data data' migration-id))
"migration is idempotent")))
(t/deftest migration-0033-fixes-children-ids-outside-the-root-walk
;; A shape that the page root does not reach is not visited by the
;; walk of 0033, so its children must be repaired before it.
(let [migration-id "0033-normalize-main-instance-names-v2"
file (-> (thf/sample-file :file1)
(tho/add-frame-with-child :frame1 :child1)
(tho/add-frame :frame2))
frame2-id (thi/id :frame2)
file (-> file
(ths/update-shape :frame1 :shapes [])
(ths/update-shape :child1 :parent-id (uuid/next))
(ths/update-shape :child1 :shapes [nil (str frame2-id)]))
data' (cfm/migrate-data (ctf/file-data file) migration-id)
file' (assoc file :data data')]
(t/is (= [frame2-id] (:shapes (ths/get-shape file' :child1))))))
(t/deftest migration-0033-fixes-children-ids-in-legacy-components
;; Legacy components keep their own :objects, which the data schema
;; does not validate shape by shape.
(let [migration-id "0033-normalize-main-instance-names-v2"
component-id (uuid/next)
root-id (uuid/next)
child-id (uuid/next)
data (-> (ctf/file-data (thf/sample-file :file1))
(assoc-in [:components component-id]
{:id component-id
:name "Component"
:objects {root-id {:id root-id :shapes [nil (str child-id) (uuid/next)]}
child-id {:id child-id}}}))
data' (cfm/migrate-data data migration-id)]
(t/is (= [child-id] (get-in data' [:components component-id :objects root-id :shapes])))))
(t/deftest migration-0033-turns-shapes-lists-into-vectors
(let [migration-id "0033-normalize-main-instance-names-v2"
file (-> (thf/sample-file :file1)
(tho/add-frame-with-child :frame1 :child1))
child1-id (thi/id :child1)
file (ths/update-shape file :frame1 :shapes (list child1-id))
data' (cfm/migrate-data (ctf/file-data file) migration-id)
shapes (:shapes (ths/get-shape (assoc file :data data') :frame1))]
(t/is (vector? shapes))
(t/is (= [child1-id] shapes))))
(t/deftest migration-0033-leaves-valid-objects-untouched
;; Nothing to repair: the objects map must be the same instance, so an
;; ObjectsMap is not marked as modified and its page is not rewritten.
(let [migration-id "0033-normalize-main-instance-names-v2"
file (-> (thf/sample-file :file1)
(tho/add-frame-with-child :frame1 :child1))
page-id (thf/current-page-id file)
data (ctf/file-data file)
data' (cfm/migrate-data data migration-id)]
(t/is (identical? (get-in data [:pages-index page-id :objects])
(get-in data' [:pages-index page-id :objects]))))
;; Same property through the objects-map wrapper the backend uses on the
;; unrealized save path, whose `assoc` marks the page as modified.
(let [migration-id "0033-normalize-main-instance-names-v2"
file (-> (thf/sample-file :file1)
(tho/add-frame-with-child :frame1 :child1))
page-id (thf/current-page-id file)
objects (omap/wrap (get-in (ctf/file-data file) [:pages-index page-id :objects]))
data (assoc-in (ctf/file-data file) [:pages-index page-id :objects] objects)
data' (cfm/migrate-data data migration-id)]
(t/is (omap/objects-map? objects) "wrapped in an ObjectsMap")
(t/is (identical? objects (get-in data' [:pages-index page-id :objects])))))
(t/deftest migration-0033-v2-runs-on-files-that-applied-the-original-id
;; Files that applied the original migration before it repaired children
;; never run it again, so the id moved to -v2. A file still carrying the
;; old id must run the v2, and keeping the old id does no harm.
(let [file (-> (thf/sample-file :file1)
(tho/add-frame-with-child :frame1 :child1))
child1-id (thi/id :child1)
file (-> file
(ths/update-shape :frame1 :shapes [(str child1-id) nil])
(assoc :migrations
(conj (disj cfm/available-migrations
"0033-normalize-main-instance-names-v2")
"0033-normalize-main-instance-names")))
file' (cfm/migrate-file file {})]
(t/is (not (cfm/need-migration? file')) "the v2 migration is recorded")
(t/is (contains? (:migrations file') "0033-normalize-main-instance-names")
"the old id is kept")
(t/is (= [child1-id] (:shapes (ths/get-shape file' :frame1))))))
(t/deftest migrations-leave-shapes-without-the-shapes-key-untouched
;; Only containers have a :shapes key; other shapes come out of the
;; repair unchanged, without a :shapes key being added.
(let [migration-id "0033-normalize-main-instance-names-v2"
file (-> (thf/sample-file :file1)
(tho/add-frame-with-child :frame1 :child1))
child (ths/get-shape file :child1)
data' (cfm/migrate-data (ctf/file-data file) migration-id)
child' (ths/get-shape (assoc file :data data') :child1)]
(t/is (not (contains? child :shapes)) "the fixture shape has no :shapes key")
(t/is (not (contains? child' :shapes)) (str migration-id))
(t/is (= child child') migration-id)))
(t/deftest migration-0033-fixes-children-ids-through-file-migration
;; 0032 adds the uuid of a child listed as a string, so the child is
;; listed twice before 0033; the file must still pass validation. The v2
;; keeps the position of the original migration, so it parses the string
;; before 0035-fix-dangling-children drops the ids it cannot find.
(let [file (-> (thf/sample-file :file1)
(tho/add-frame-with-child :frame1 :child1))
child1-id (thi/id :child1)
file (-> file
(ths/update-shape :frame1 :shapes [(str child1-id)])
(assoc :migrations
(disj cfm/available-migrations
"0032-fix-children-not-in-parent"
"0033-normalize-main-instance-names-v2")))
file' (cfm/migrate-file file {})]
(t/is (not (cfm/need-migration? file')) "new migrations recorded")
(t/is (= [child1-id] (:shapes (ths/get-shape file' :frame1))))))