get-nesting-level-delta counted the copy heads that a shape and its new
parent share with a two-argument predicate over a seq of pairs, so it
threw an arity error whenever both had copy heads. Its callers,
duplicate and the advance of copied objects, only pass parents outside
every copy today, so nobody reached it. The predicate now takes the
pair and compares the heads by id.
The advance of copied objects passed uuid/zero where the function
expects a shape; it worked only because the id of a uuid is nil. It now
passes the root shape.
AI-assisted-by: claude-opus-5-5
In an inverse sync, when a copy child paired with a main child further
along the main's children, compare-children asked whether the copy
child was the main of the main child, with the arguments of
ctk/is-main-of? reversed. The answer was always no, so the pair took
the swapped callback, which does nothing in an inverse sync, and only
the move ran: the overrides of that copy child never reached the main.
The UI does not let a copy reorder its children, so it only showed
when the order differed for another reason, such as the undo of a
sync reorder.
The inverse branch now asks synced-pair? with the main child first,
as the direct branch does.
Benchmark (lower quantile, ms; baseline / now / ratio):
deepest-edit-through-three-levels 35.2 / 35.5 / 1.01
edit-through-a-chain-of-dependents 68.9 / 66.1 / 0.96
library-update-into-copies 83.1 / 77.0 / 0.93
main-edit-into-10-copies 1.8 / 1.6 / 0.92
main-edit-into-100-copies 15.4 / 16.2 / 1.05
main-edit-into-1000-copies 153.1 / 156.4 / 1.02
variant-switch-into-copies 32.1 / 28.3 / 0.88
AI-assisted-by: claude-opus-5-5
When a component nested in others is deleted (inner in mid in outer,
mid deleted), a change to inner still reaches mid's stored shapes,
mid's copies, outer and outer's copies, and restoring mid brings back
an up-to-date main. Keep this behavior: outer is a live component, and
its users cannot see a deleted mid between it and inner.
Extend the deleted-component propagation test with a copy of the
deleted component and its restore, and reword the contract, which said
copies of a deleted component do not sync.
AI-assisted-by: claude-opus-5-5
When a copy head took the place of a main shape and its component was
the one of that place, it stopped being a swap, but its children kept
referring to the main of its own component instead of to the children
of that place. Validation did not see it, and the next sync deleted
those children and added them again, losing their overrides, or the
head did not follow later changes to that place. It happened in:
- a variant switch that carries a swapped child into a variant whose
child there has the same component;
- update main of a copy that swapped a nested copy back to the
component a deeper main puts in that place, after the main in between
swapped it for another one; the clone in the main also kept
:swap-group;
- the repair of :ref-not-in-near-main, which also never cleared
:swap-group.
The swap in a copy, the switch, update main and the repair now share
ctf/take-position: the head refers to the main shape of its position,
is swapped while its component differs from that shape's, and else its
descendants refer to the descendants of that shape.
The switch also read the swapped child from the objects of the swap,
where it was already gone with its old parent. It now reads it once it
is moved into the new copy, so its placement turns its children too,
and the undo takes it back to the root as the move to the root left it.
The swap finds the new shapes in its own objects, not in its changes.
The component memory says how a head takes a position.
Benchmark (lower quantile, ms; baseline / now / ratio):
deepest-edit-through-three-levels 35.2 / 37.1 / 1.05
edit-through-a-chain-of-dependents 68.9 / 70.5 / 1.02
library-update-into-copies 83.1 / 77.3 / 0.93
main-edit-into-10-copies 1.8 / 1.7 / 0.94
main-edit-into-100-copies 15.4 / 16.0 / 1.04
main-edit-into-1000-copies 153.1 / 160.6 / 1.05
variant-switch-into-copies 32.1 / 28.7 / 0.89
AI-assisted-by: claude-opus-5-5
The file builder behind @penpot/library pinned the features and
migrations it writes to the ones Penpot 2.7 accepted, so its files
could open there. It now writes the current defaults: the features a
new file gets and every available migration. Generated files were
already in the current format (paths, for one, are stored as path
data), so they now declare fdata/path-data and components/swap-group
as they should, and the import runs no migration on them.
Files built this way need the Penpot version the library is built with,
or a later one.
AI-assisted-by: claude-opus-5-5
A swapped copy head followed the position, constraints and layout item
attributes of the main child it replaces, but not its rotation or
flips. When the main rotated or flipped that child, the copies that
kept it turned and the swapped ones did not. Sync, and a variant switch
that carries a swap, now give the head and its subtree the flips and
then the rotation of that child, around the head's own center, unless
the user changed the head's geometry. The swap itself uses the same
step.
Benchmark (lower quantile, ms; baseline / now / ratio):
deepest-edit-through-three-levels 35.2 / 34.1 / 0.97
edit-through-a-chain-of-dependents 68.9 / 75.5 / 1.10
library-update-into-copies 83.1 / 81.3 / 0.98
main-edit-into-10-copies 1.8 / 1.6 / 0.88
main-edit-into-100-copies 15.4 / 15.6 / 1.01
main-edit-into-1000-copies 153.1 / 153.2 / 1.00
variant-switch-into-copies 32.1 / 31.4 / 0.98
AI-assisted-by: claude-opus-5-5
A swap placed the new copy at the position of the shape it replaces,
but not with its rotation or flips. When that shape was rotated or
flipped with its parent, the new copy came out straight. The new copy
and its subtree now take the flips and then the rotation of the shape
it replaces, around their own center, as geometry that comes from the
parent, not as an override. A variant switch goes through the same
swap.
Benchmark (lower quantile, ms; baseline / now / ratio):
deepest-edit-through-three-levels 35.2 / 38.3 / 1.09
edit-through-a-chain-of-dependents 68.9 / 70.7 / 1.03
library-update-into-copies 83.1 / 82.2 / 0.99
main-edit-into-10-copies 1.8 / 1.7 / 0.95
main-edit-into-100-copies 15.4 / 16.7 / 1.08
main-edit-into-1000-copies 153.1 / 167.4 / 1.09
variant-switch-into-copies 32.1 / 29.6 / 0.92
AI-assisted-by: claude-opus-5-5
A variant switch with keep-touched moves a swapped child out of the old
copy to the page root, then into the new copy, which takes the same id
as the old one. The undo of the second move took the child back to
that id, so it landed in the new copy and went away with it when the
undo deleted the new copy. The undo now takes it back to the root,
and from there into the old copy.
AI-assisted-by: claude-opus-5-5
A variant switch with keep-touched carries a swapped child into the
new variant, but left it where it was in the old one. The next sync
then moved it to the place of the child it replaces, at a moment
unrelated to the switch. The switch now gives it that placement at
once, through the same step sync uses: position, constraints and
layout item attributes the user did not change.
Benchmark, against the same code without this change under the same
load (lower quantile, ms; without / with):
deepest-edit-through-three-levels 36.0 / 36.0
edit-through-a-chain-of-dependents 65.2 / 70.2
library-update-into-copies 74.3 / 75.9
main-edit-into-10-copies 1.6 / 1.5
main-edit-into-100-copies 16.3 / 15.7
main-edit-into-1000-copies 162.6 / 165.8
variant-switch-into-copies 27.2 / 27.8
AI-assisted-by: claude-opus-5-5
The migration 0033 stores every swap of a file as an override group.
A swap made in a copy is paired with the child of the near main that
holds the same position, as sync paired it; any other slot is dropped.
All pairings are found on the data as it was, so swaps nested in swaps
pair against the content of the outer one, and libraries pair the same
whether they are migrated or not. A head that refers to its own slot
only inherited it, and drops it even when its near main cannot be
reached. Any other slot that matches no child, or whose near main is in
a library that is not loaded, keeps the slot as its ref, so validation
reports it. An import, which migrates its files before it links their
libraries, leaves those slots and converts them once the libraries are
linked. A migrated head that sits elsewhere than the child it replaces
would place it gets its geometry marked as touched, so the first sync
does not move it. Migrated and new files carry the feature
components/swap-group, so a client that still writes slots cannot open
them.
Validation checks the new form: :misplaced-slot reports any slot,
:misplaced-swap-group the group anywhere but on a nested copy head in a
copy, :ref-not-in-near-main a nested copy head whose ref is not a child
of the near main of its parent, and :duplicate-ref two children of a
copy that refer to the same main shape. Each has a repair.
Nothing but the migrations reads slots any more: the slot branches go
from sync, swap, switch, reset, detach, paste, duplicate and move. The
helpers that read and write a slot on a shape move to files/swap-slots,
and the ones that follow refs to files/comp-processors. The graph
projection drops the FillsSwapSlot edge, the components debugger shows
the component a swapped head takes its content from, and the plugin API
documents componentRefShape for a swapped copy. The swap tests are
named after what they check.
Benchmark (lower quantile, ms; baseline / now / ratio):
deepest-edit-through-three-levels 35.2 / 34.6 / 0.98
edit-through-a-chain-of-dependents 68.9 / 63.5 / 0.92
library-update-into-copies 83.1 / 76.4 / 0.92
main-edit-into-10-copies 1.8 / 1.6 / 0.87
main-edit-into-100-copies 15.4 / 15.1 / 0.98
main-edit-into-1000-copies 153.1 / 148.8 / 0.97
variant-switch-into-copies 32.1 / 26.4 / 0.82
AI-assisted-by: claude-opus-5-5
A swapped copy sub-head now keeps the shape-ref of the main child it
replaces (its position), takes its content from its own component, and
carries the touched group :swap-group. It used to point at the root of
the new component and store a swap slot naming the replaced child.
Every path reads and writes that form:
- sync pairs a swapped head with the child it replaces and syncs it
from its own component; it takes from the replaced child only its
placement: position, constraints and layout item attributes;
- reset undoes a swap in place, keeping the head's id and the other
overrides of the copy;
- detach, paste and duplicate make a head that leaves every copy level
a plain copy of its own component, and advance a deeper one by
position; the children of a swapped head never advance;
- a variant switch keeps a swap stored this way, and counts only the
copy's own swap, not one made in a main of the chain;
- update main replaces the main child with a clone of the swapped one,
and from a swapped head updates its own component;
- a swap inside a copy keeps the old shape-ref and adds :swap-group;
swapping back to the component of that position gives a plain nested
copy whose children refer to the position's children; a swap inside
a main is a plain nested copy.
Files that hold swap slots keep working until a migration converts
them. The component memories describe the model.
This fixes what came from the slot: reset of a swapped child cleared
the other overrides of the copy (F12) and gave copies of a main new
ids (F66); update main left a swap mark (F16); a second variant switch
lost the overrides (F29); grid cells did not pair a swapped child
(F44); a swapped child did not follow its position (F65). F49 keeps
failing, in another form.
The tests that asserted slots now assert the override group; the tests
of the slot code build the old form with :as-slot?.
Benchmark (lower quantile, ms; baseline / now / ratio):
deepest-edit-through-three-levels 35.2 / 35.4 / 1.01
edit-through-a-chain-of-dependents 68.9 / 68.7 / 1.00
library-update-into-copies 83.1 / 77.5 / 0.93
main-edit-into-10-copies 1.8 / 1.6 / 0.92
main-edit-into-100-copies 15.4 / 15.4 / 1.00
main-edit-into-1000-copies 153.1 / 151.1 / 0.99
variant-switch-into-copies 32.1 / 32.4 / 1.01
AI-assisted-by: claude-opus-5-5
No components v1 file reaches the code any more: opening one fails the
feature check, and imports reject it or give it the feature. What only
served v1 goes:
- validate-file and validate-file-affected no longer skip a file
without components/v2;
- the root of a deleted component is found by its main instance id
only, in the type helper and the component sprite;
- generate-sync-library, which synced shapes inside live components,
and the live components in containers-seq, so asset usages walk the
pages only (deleted components keep their objects and still sync);
- the unused :components-v2 file option, from the schema, the file
data and the test fixtures; stored files that carry it still
validate;
- the team migration-in-progress check and its frontend alert, which
leaves ephimeral features out of the team comparison;
- the components v2 introduction dialog, its styles and icons, and the
:v2-info-shown profile prop;
- the graphic asset events and builders and the reads of graphics in
the selection (the :add-media, :mod-media and :del-media changes stay,
so stored changes still apply), and the uncalled render-page and
render-components helpers.
AI-assisted-by: claude-opus-5-5
Add a battery of tests for text overrides in copies.
- A short notation for text content in tests (thtx): paragraphs of
runs with short style names, read back with equal runs joined,
edits in the notation, set-text and set-content through the change
path, and the override rule for texts (override-units), which says
which overrides a copy must hold against its main.
- Sync matrix: every whole-text copy edit against every main edit,
plus the cards of the worked examples, the "Hello world" table, a
token on a copy, typographies and a library color sync. Where the
code differs from decisions (k) to (o), the result is pinned
against F72 to F80.
- Switch matrix: copy edits against variants that differ in letters,
size, color, a new line or a bolded word; F76, F77, F79 and F81
pinned.
- Composable case BA: EditText also edits fills, font family and a
whole text; the copy and the main change different styles, then
reset or update main, in both runners; F72 pinned.
- Editor output: the content the WASM and v1 editors commit when a
copy text is typed into, retyped, split with Enter and has a word
bolded, recorded in the workspace and replayed on a copy. Typing
sets only the text groups and a style change no text override;
F79 pinned on both editors' content.
A review found tests that would pass with what they test broken.
- Runners: both judge one variant at a time, fail a variant with no
assertion and a known-failure mark that applies nowhere, and give
the asserter the state the operations produced. The pure runner
validates the final file and rejects operations of unknown kind.
The frontend interpreter fails a step when the store reports an
event error (Potok only logs it), reports asserter errors against
their case, and stops the watchers that install! starts.
- Nodes drive the real code: a main root rename goes through
end-rename-shape, and the clipboard's advance of nested copies is
one common function, cll/advance-copied-objects, that copy-selected
and both test paths call.
- Cases N, AH, AI, AK, AD, AE, K, C, H, S, AY, I, Y, AX and F check
what they claim; O, R and T get real sizes.
- Stricter common unit tests found F60 (a repair detaches healthy
copies), F61 (a switch drops a fixed-sizing size), F62 (the ref
chain never stops at its target) and F63 (latent); all pinned.
- Frontend library tests: the RPC double records on subscribe, the
dropped commits check that nothing syncs, and the link and share
tests check the events sent.
- Plugin suite: OpSwap and OpDetach throw when the API call did
nothing, and case AB checks that the reset head is an instance of
the original component.
- The composable suite memory records the new rules.
The composable interpreter treated a step as done after a 60 ms gap
with no commits, and treated its 2 s timeout as done too. Slow sync or
reflow work could finish after the step, so a case could assert on a
half-synced file.
Each event step now waits, as the plugin waitForLayoutUpdate does,
until every pending work tracked by the reflow registry has drained:
the watcher's sync wait, the sync it starts, and layout and text
reflow. A step that does not drain within 2 s fails the test and names
the work still pending, and the next variant starts with nothing
pending.
The tests cover an edit with a deliberately slow sync, undo and redo
steps that must wait for the watcher's touch-component, translation
and derived commits, a sync that never finishes, and a restart of the
edit watchers, which must keep the undo entries of an edit the same.
Also in this commit:
- Keep page-less changes from adding a nil page. Applying a commit locally
rebuilds the object indices of every page its changes name. Changes that touch
no page, such as :mod-component, name none, and their nil page id was rebuilt
too: the file data in the store gained a page under a nil key holding
{:objects nil}. Every component touch after an edit to a main added it, and
undo never removed it.
- Check undo and redo of every step in the composable cases. The composable
cases tested undo only where a case asked for it, so a change generator with a
wrong inverse, or a step that splits into several undo entries, went
unnoticed.
- Add operations and cases to the composable component suites. The composable
runners could edit, add, remove and move children, swap and switch, but not
run the other component operations.
- Test component validation, repair and library asset sync.
- Add a component sync benchmark with a recorded baseline. The coming refactors
and fixes change sync code, and a slowdown must be seen and explained in the
commit that causes it.
- Tidy the composable suite and make its cases stricter.
- Test the component sync paths no test reached. A coverage run found sync paths
that no test ran.
- Test variant events, token sync and switch overrides. The variant events had
almost no frontend tests, and several switch, paste, token and restore paths
never ran.
- Test the file and container helpers and the library events. The component
helpers of types/file and types/container and the library events of the
workspace had the lowest coverage left.
The composable test engine, nodes, setups and cases lived in the
frontend test tree, so only the frontend interpreter could run them.
Move them to CLJC test helpers under app.common.test-helpers.composable
and add a pure runner (comp.runner) that does what the workspace does
around each operation, without the frontend store:
- after a user operation it syncs the changed components until none is
left, as the frontend watcher does with one pass per sync commit;
- the operation and its syncs form one undo group, and the undo node
reverts the latest one;
- it validates the file once the sync or the undo has settled.
Each node declares its kind: assembly, user, undo or frontend-only.
User nodes record the production changes they apply, and the runner
throws when a node breaks that. The swap and switch nodes build their
changes with the production clv/generate-component-swap-in-place.
Rotate and ChangeHeight need the frontend placement step, so the pure
runner rejects them.
The cases are zero-arg functions in comp.cases, shared by the frontend
interpreter and common-tests.logic.composable-sync-test. The latter
runs cases B to M and the new case O, a switch of an overridden copy,
on the JVM and in JS. Case N and the rotation case stay frontend-only.
The variant switch event picked its target variant, and the component
swap event worked out where the new head goes, inside the frontend
events. Code that runs without the frontend store, such as the pure
composable runner, could not do the same switch or swap. The swap and
the switch also checked for nesting loops with two different
functions.
Move them to app.common.logic.variants:
- find-switch-target picks the nearest variant with the requested
value.
- generate-component-swap-in-place builds the swap changes keeping the
old head's index, grid cell and swap-keep-attrs, plus the
keep-touched step when asked.
- swap-nesting-loop? checks that a swap or switch does not nest a
component inside itself. The swap menu, the plugin swapComponent and
the variants-switch event all use it.
The frontend events behave as before, with two exceptions. Switching
to a value no other variant has does nothing, where it threw in
min-key. A swap also updates the layout of the shapes the keep-touched
step adds.
When components whose name matches the variant container are moved into
it, removing the container name leaves an empty value. Combining several
components named the same produced "Value 1" for the first variant and
empty values for the rest, so variants shared the same combination.
Assign the first unused "Value N" to the last property when all computed
values are blank, tracking values given within the same batch.
Combine moves the remaining mains into the container one at a time, in
the combine order, so the generated values follow that order instead of
the layer order.
Closes#12878
* ⚡ Send objects-map pages to the worker without a second encode
When switching between pages on a large file e.g Material Design Icons, there's
some UI freeze from :index/initialize encoding object map shapes to send to the
worker. An objects map already stores the encoded JSON for each shape, and
`encode-str` was wrapping those strings in a second layer of JSON on the main
thread.
We can just transmit the already encoded JSON in a js/Map alongisde the
payload, similar to :blob, so structured clone just copies the strings verbaitm
without re-encoding. To do this this PR adds two functions to ObjectsMap that
converts to and from a js/Map.
The "Main components" page of the Material Design Icons has ~30k objects, so
this reduces the freeze when switching pages from 306ms -> 189ms (1.63x) on a
release build.
AI-assisted-by: claude-sonnet-5-5
* Split property tests out into new deftest
---------
Signed-off-by: Luke Lau <luke_lau@icloud.com>
Co-authored-by: Andrey Antukh <niwi@niwi.nz>
* ✨ Cache the transit reader when decoding strings
On the cljs side, we create a new transit reader on every call to decode-str.
This turns out to have a fair bit overhead in both transit-cljs and transit-js,
since it has to merge the handlers, call clj->js, allocate a new ReadCache,
etc.
By reusing it the measured decode time goes from ~58us -> ~41us, so about 1.4x
faster. This particularly helps for some operations like whenever a component
is edited and every object needs to be decoded, see
https://github.com/penpot/penpot/issues/9344, although they could probably be
made more efficient in other ways.
The reader usually clears its read cache after a successful read, but whenever
it throws it hangs around. So in this case we need to reset the cache
ourselves, otherwise the next read will be from a clobbered cache.
The tests were written by Claude Sonnet 5.5, and should cover the clobbering
behaviour above.
Signed-off-by: Luke Lau <luke@igalia.com>
* Keep one transit reader per type and rebuild it on add-handlers!
---------
Signed-off-by: Luke Lau <luke@igalia.com>
Co-authored-by: Andrey Antukh <niwi@niwi.nz>
* ⚡ Compute text fonts once per shape when uploading to wasm
* ⚡ Write the shapes batch in place and coerce fills once
* 🐛 Free the upload buffer when a batch record write fails
* 🐛 Normalize stored SVG attribute keys to kebab-case
The v3 reader already emits kebab-case keys, but birth stored the
camelCase React prop names, so import and export never converged:
fill rules painted wrongly and filter defs lost their blur.
Birth now normalizes every stored attribute key to kebab-case
through the `svg-props` whitelist (`attrs->kebab-props`), applied on
all element paths of the svg importer, on the file builder and on
defs. svg-raw `:content` is kebab-ized in spelling only and keeps
unknown keys, because the sidebar reads them. Non-spec attrs Penpot
must keep live in `penpot-extra-attrs`, seeded with `:stroke-style`.
Render boundaries translate back to camelCase for React and for the
wasm props, and the shape schema declares the closed kebab contract
derived from that same whitelist, so birth and validation cannot
drift.
Migration `0035-fix-shape-svg-attrs-and-defs` converts old files
once with the birth transforms, and carries back the svg-viewbox
rect normalization the withdrawn `0021` had. That old id leaves
`available-migrations`, so files recording it migrate again through
this one, and the binfile cleaner drops its own svg repair.
Review fixes included here: `setup-opacity` and the blend mode read
the stored keys, gradient stops accept `:style` as a CSS string or a
map, `inherit-attributes` reads `:style` so a child's inline style
beats the inherited group attr per CSS, and the svg-attrs menu shows
spec spelling (`viewBox`, `class`) through `stored-attr-display-name`.
Tests cover the migration, the closed svg-attr schemas and the
import/export round trip; the svg-content schema test pins its
sample seed, which dealt no svg-raw shape on unlucky runs.
Closes#11947
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Keep legacy svg node tags and transforms valid in migration 0035
The closed svg node schemas require keyword tags and map attrs, but
stored files hold def and svg-raw content tags as strings, and some
node `:attrs` as nil. Migration 0035 left them as they were, so the
file data check that runs after migrating rejected those files and
they could not be opened. `normalize-node-tree` now turns string tags
into keywords and nil attrs into an empty map, at birth and in 0035.
With keyword tags, the svg defs renderer applies the shape transform
to user-space defs. Legacy `:svg-transform` values are plain maps,
which multiply into NaN matrices. `gmt/coerce` turns them into
matrices, in 0035 and in the binfile cleaner, because every v3 import
decodes the transform as a plain map again. Legacy masks and gradients
now render as designed in the svg renderer.
The binfile test helper calls `export-files!` with the current config
keys.
AI-assisted-by: claude-opus-5-5
---------
A component whose :main-instance-id points to a missing shape fails
validation on its copies with :ref-shape-not-found. A repair scoped to
the copy's page never sees the main's own error, so it detached the
copy, even when the main was still on the component's page.
The repair of :ref-shape-not-found now looks for the component's main
on its page first and points :main-instance-id back to it, which keeps
the copy linked. A copy is still detached when no main is found.
update-file checked the schema of the whole file on every save, which
loads and decodes every page even when the changes touch one. On files
with several pages that was most of the time of a save.
validate-file-schema-affected! checks the file data with the pages index
reduced to the pages the changes modify; the data outside pages is still
checked whole, and a change type it does not know makes it check the
whole file. update-file uses it for the strict and the soft schema check.
A save of 10 commits on a file of 10 pages of 2k shapes goes from ~1.1 s
to ~240 ms; a single page file costs the same.
A save whose changes failed validation was rejected as a whole: autosave
stopped, the edits made after it piled up unsent, and a reload lost
them all, the valid ones included.
The frontend now sends each commit of a save as its own group with its
commit id (:changes-with-metadata). When applying and validating all of
them at once fails, the backend searches by bisection for the first
commit that fails, keeps the commits before it and answers with
:rejected (that commit and every later one). Only the kept changes are
stored, snapshotted and broadcast, so other users never see the rejected
ones. Flat :changes and groups without ids stay all-or-nothing; database
and network errors still fail the whole request. A rejection without an
error type is reported as an internal error.
The frontend rolls the file back to what the server holds: it undoes
every pending commit of that file, applies again the remote changes that
arrived meanwhile, and redoes the kept commits. Undoing only the
rejected commits would write stale values over a peer's later change to
the same attribute. The rejected and later commits leave the queue, the
buffer and the undo history, and a warning that hides after ten seconds
tells the user. The pure bookkeeping lives in
app.main.data.persistence.commits.
The pending notice of a commit the rollback dropped no longer leaves the
save status at pending, which made "Pin this version" hang and fail.
Debug tools drive these cases by hand: debug.commitInvalidChange() makes a
save fail validation, holdSaves() delays the answers, failNextSave() and
failNextRollback() fail them on purpose (backend flag debug-save-faults),
and persistenceState() shows the queue.
* ⚡ Keep only the shape data the worker reads
The worker kept a full copy of the current page for its selection and
snap indexes. It now keeps only the attributes the indexes and the
change replay read, and re-indexes only the shapes that changed.
On a 52k-shape page the worker heap goes from 698 MB to 286 MB.
Closes#12740
AI-assisted-by: claude-opus-5-5
* ♻️ Share the worker re-trim with its replay test
The worker and its replay test now share `retrim-objects`.
AI-assisted-by: claude-opus-5-5
* ⚡ Avoid decoding the opened page twice for its frame index
The frame index was built over a with-meta copy, so pages got decoded again.
Read it from the page objects themselves and fix the with-meta identity check.
* ⚡ Intern keywords when decoding transit on the frontend
Each shape is decoded on its own, so every shape had its own key keywords.
Cache them in the read handler so all decoded data shares one instance.
* ⚡ Use an array map as the wasm shape proxy delegate
A hash map with ~35 keys costs ~3 KB of nodes per shape on large pages.
An array map holds the same entries in one array, at a small lookup cost.
Some files have component copies whose path drawing (:content) sits away
from its box (:selrect and :points). The renderer draws the content, while
the sidebar, layouts and syncs use the box, so these icons render outside
their boards and are clipped or drawn over other shapes. Moving the shape
keeps the gap, and sync never repairs it because both groups are touched.
Add the migration 0034-fix-path-content-offset. For every path in pages
and in component objects, deleted components included, it derives the box
from the content as update-geometry does and, when the sizes agree but the
position does not, moves the content onto the stored box. The box, the
transform and the touched groups stay as they are; paths whose drawing has
another size are left untouched.
Closes#12752
* ✨ Stabilize fragmented file data across snapshots and GC
Snapshots, automatic and user, now store pointer-based rows that share
fragments with the live file instead of duplicating page content.
`get-file-snapshot` resolves those pointers on the server so version
previews keep rendering plain data, and `restore!` reuses the shared
rows. Export still materializes (the package carries no fragments) and
import regenerates them.
`file-gc` treats the main file and every unpurged snapshot as liveness
roots, reads the file unrealized, and reattaches the original pointer
wherever the cleaned value is unchanged, so a pass over an untouched
file writes no fragments and a pass over an edited file keeps the ids
of the pages it did not touch. Snapshot creation and the restore
backup take the file row `FOR SHARE`, which serializes them with the
pass and closes the race that let a new snapshot reference fragments
the pass was about to mark.
All backend pointer bindings go through the `fdata/with-pointers`,
`with-tracked`, `with-tracked-pointers` and `with-untracked` macros.
The loader reads fragments a previous pass marked until `objects-gc`
purges them, and the frontend fails loudly with retry instead of
dropping an unresolved fragment.
Review fixes in this branch: the automatic snapshot attaches before
pending migrations, `realize-pointers` is the single verb,
`reattach-unchanged` takes the identity fast path, the restore guard
takes its revn from the backup read, and `PointerMap.withMeta`
generates a fresh id when metadata changes.
Tests cover both snapshot writers (red without the lock), snapshot
sharing and pin release through the real delete and purge paths,
reattach against orphan marking, preview after offload, the pointer
binding contracts and the with-meta id change.
Closes#12410
AI-assisted-by: muse-spark-1.3-contributor
AI-assisted-by: deepseek-v4.1-flash
♻️ Pin no-churn behavior for untouched pages
Pin the behavior the churn verification measured: a save keeps the
fragment id of every page it does not touch, with a page of eleven keys
(PersistentHashMap), a 46-shape objects map and a nine-page index (hash
map). Also pin the mixed case: a data-level objects-map flip rewrites
the old pointer pages once and the next save keeps the new ids.
The memory records the identity rule behind it: re-assoc with the same
value returns the same map in array and hash maps alike.
Closes#12410
AI-assisted-by: deepseek-v4.1-flash
⚡ Stop loading untouched pages when enabling objects-map
`enable-objects-map` wrapped every page on every save, and reading a
page's objects forced its PointerMap to load the fragment. A save of
one page in a seven-page file therefore made seven fragment reads.
Skip unloaded pointers: their fragment was already wrapped by the
encode-file that wrote it, so loading it only to wrap it again is a
no-op. The save now loads only the pages it edits. A page with a plain
fragment (a data-level feature flip) keeps its plain fragment until it
is first loaded or edited; a migration that enables the feature must
rewrite the data itself.
Cover the change with load-count tests for save and restore, update the
flip test to the new contract, and drop the stale restore comment.
Closes#12410
AI-assisted-by: deepseek-v4.1-flash
⚡ Keep page fragment ids through pending migrations
Applying a pending migration on read realized the file, ran the
migration and persisted it again; the persist wrapped every plain page
in a new PointerMap, so every page got a new fragment id even when the
migration did not change it. That leaves the fragment cache cold and
rewrites every fragment on the first read after a deploy.
Capture the original pointers before realizing, reattach the unchanged
positions after migrating (same value comparison file-gc already uses),
persist the reattached file and return the realized one: the read
contract is plain data. Read-only connections skip the persist and the
reattach. A migration that adds objects-map skips the reattach so the
pages get wrapped.
Move the three helpers to fdata so file-gc and the migration path share
them, and cover the path with a real 0031 migration test: the unchanged
page keeps its id, the changed page gets a new one, the caller gets
realized data and a second read does not migrate again.
Closes#12410
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Realize library pointers with each library's own loader
Fragment rows are keyed by file id, and every caller binds the
fragment loader of the file it works on. get-resolved-file-libraries
returned libraries with their pages still as pointers, so the first
read of a library page under fdata/pointer-map asked the consuming
file for the fragment and failed with :fragment-not-found. That
broke migrations and file validation for every file using a
pointer-mapped library.
The weak-map loader now realizes each library's pointers with the
library's own id. The test fails without the change.
AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
* 🐛 Restore the pointer-map require the repair path still needs
The branch dropped `app.util.pointer-map` from the namespace while the
auto-repair path (added to develop before this branch) still rebinds
`pmap/*tracked*` around validation and repair in
`repair-preexisting-damage` and `process-changes-and-validate`. The
namespace did not compile, so the whole backend failed to load.
Those toggles do not fit the fdata macros: `with-untracked` binds nil
and `with-tracked` creates a fresh atom, while the repair path has to
turn tracking off and restore the existing atom (and read it before the
repair starts). Restore the require and keep the bindings explicit.
Also pin the feature-off path with a test: with `fdata/pointer-map` off
(the default), a save creates no fragments, file-gc writes nothing and
the snapshot preview still returns plain data. Drop a stray `\n` char
literal left at the end of one deftest.
Update the memories with the exception to the macro rule, the snapshot
FOR SHARE locking and the pre-migration automatic snapshot.
Closes#12410
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Validate debug files with their own pointer loader
The `/dbg/actions` file validation passed the file id where
`get-resolved-file-libraries` expects the file map, so the library map
was seeded with a nil key. It also validated a file read without
realizing it, so the first page lookup of a pointer-mapped file raised
"load is not supported when *load-fn* is not bind" and the endpoint
could not validate exactly the files the feature touches.
Read the file realized (validation walks every page and component
anyway) and build the library map from the file. The library loader
already resolves each library's pointers with its own id, so the
realized file and its libraries validate as plain data. Covered by a
handler-level test that fails before the change.
AI-assisted-by: deepseek-v4.1-flash
* ♻️ Bind the snapshot loader through the fdata helper
`reduce-snapshots` hand-rolled the same `binding` that
`fdata/with-pointers` builds (it also loaded the function's own
pointer loader before entering the reduction). Use the macro so the
rule holds there too and the namespace no longer needs
`app.util.pointer-map`; the loader is identical (same cfg and file-id)
and snapshot walking behaves the same.
AI-assisted-by: deepseek-v4.1-flash
* 📚 Note the pointer-binding exceptions and snapshot ordering
Record what the feature review settled:
- Two direct `binding`s stay on purpose: the fine-grained `*tracked*`
toggles of `app.rpc.commands.files-update` (validation/repair turn
tracking off and restore the existing atom) and
`binfile.common/encode-file` (fresh tracked set inheriting the
current one). Atom-scoped macros would only be worth it if the set
grows.
- The automatic snapshot is taken before the pending migration, and
the pre-migration window is narrow: the `get-file` RPC migrates on
read and its ETag hashes the available migrations, so a post-deploy
read cannot be a 304 from the pre-deploy state. The old-format
snapshot only appears when the save is the first request to touch
the file after the deploy, and `get-file-snapshot` previews it as
stored.
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Apply pending migrations when previewing a version
Automatic snapshots are attached before the save applies pending file
migrations, so a version can store the old format. The preview RPC
handed that data to the UI as stored, and the viewport renders it with
the current code.
Realize the snapshot with the live file id (fragments are keyed by
file, not by snapshot), normalize its migrations to a set (the decoder
keeps the pgarray as a vector, and `set/difference` would compare
indexes) and apply the pending migrations with the pure
`fmg/migrate-file` and the live libraries. Nothing is persisted: the
stored version keeps its data and restoring it re-migrates on the next
read. The response now carries the snapshot's migrations.
Use `:features` the way `get-file` does for `fdata/objects-map`: a
client that does not declare it gets plain objects. The historical
feature set is not checked against the enabled one on purpose: a
version may predate a feature change.
Cover the preview of a snapshot with a pending migration and the
objects realization; both pin that the stored row is untouched. Update
the file-data memory with the preview contract and the vector/set
trap.
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Decode snapshot migrations as an ordered set
`file-snapshots/decode-snapshot` handed the `file_change.migrations`
pgarray as a vector. `set/difference` falls back to index lookups when
the first collection is shorter than the second, so the pending set was
right only by accident: a stored subset happens to work, while a
stored set larger than `available-migrations` (a snapshot taken by a
newer build) marks every available migration as pending and re-applies
it to already migrated data.
Decode it as a `d/ordered-set`, the type the rest of the code and
`schema:file` expect. This also covers `restore!` and the snapshot
list, and it lets `get-file-snapshot` drop the `(into #{} ...)`
normalization it was carrying. Re-wrap the union in `fmg/migrate` so
the invariant holds even when a caller hands a plain set.
A snapshot with NULL `migrations` predates SQL migration 0140 and
keeps the format version of its time. Pin that legacy path: the
preview regenerates the applied set from that version, migrates in
memory and leaves the stored row untouched. Update the file-data
memory with the invariant and the legacy path.
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Fix flaky waiting-gauge assertion in climit test
The throwable-while-waiting test asserted the exact list of
waiting-gauge publications, but the component sampler republishes
the same count when its first tick lands inside the recorded
window under load. Assert only the last value, which is the
contract the helper documents.
AI-assisted-by: muse-spark-1.3
---------
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
Co-authored-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
* 🐛 Fix stack overflow when applying a library update
Each shape updated by a component sync added its reg-objects undo change
with (update :undo-changes concat [...]). That wrapped the whole undo list
in one more lazy concat per shape, with the old list at the head. After a
few thousand shapes, walking the list overflowed the stack, which matches
the nested LazySeq frames reported when a library update is applied.
add-update-attr-changes now conjs the reg-objects before the mod-obj, so
undo restores each shape and then recalculates its parents, as redo does.
The trailing reg-objects also recalculated groups after undoing shapes
that the sync added or removed. To keep that, add-shape-to-instance,
add-shape-to-main and remove-shape now add their own undo reg-objects,
conjed before their other undo changes so it runs after them.
Tests cover many calls to add-update-attr-changes, a sync and undo of
many shapes, and undoing a sync that moves, adds or removes a child of a
group in the copy.
Closes#12576
AI-assisted-by: claude-opus-5-5
* 🐛 Test undo of an inverse sync that adds a shape
The previous commit added an undo :reg-objects to add-shape-to-main, so
undoing an "update main component" recalculates the main's groups after
removing the added shapes. No test covered it.
The new test adds a shape at the start of a group in the copy and changes
a sibling's fill, runs the inverse sync and undoes it. Without the undo
reg, the main group keeps the bounds of the removed shape.
AI-assisted-by: claude-opus-5-5
* 🐛 Keep swapped and moved children in place when a component syncs
A sync resets an untouched shape of a copy to the main's geometry and the
layout then moves it back. Children the sync keeps, because they are
swapped or the user moved them, did not take the first move but did take
the second, so each sync pushed them away from their parent by the gap
between the copy and the main (e.g. a swapped icon in a copy that fills a
wider layout).
The sync now moves those children by the offset it applies to their
parent. Flex-child attributes still sync as before, so swapped children
keep their absolute position.
Closes#12657
* 🐛 Place swapped children where the swap puts them on sync and switch
A swapped child that the user never moved now goes to the top-left of the
shape it replaced, as a swap does: on a sync, the place of the main's slot
occupant; on a variant switch, the place of the new variant's occupant.
A swapped child the user moved keeps its offset inside its parent.
This stops the variant switch from leaving the swapped child at its old
position while the new variant is created at the main's size, and puts
back children that drifted before the sync fix the next time their copy
syncs.
* 🐛 Make kept children follow their parent's constraints on component sync
When a sync resets the size of a copy's shape, for example a copy root
that fills a wider layout, the layout then grows it back and moves its
children by their constraints. Children the sync keeps (moved by the
user, or swapped) only followed the parent's top-left, so with a right,
center or scale constraint they drifted on every sync, and a swapped
child with a scale constraint grew on every sync.
The sync now passes the parent's geometry before and after down the
tree, and kept children follow that change through their constraints.
Untouched swapped children also undo the change through their
constraints before they go to the swap place, so they keep their size.
The default constraint modifier returns empty modifiers instead of an
empty vector, which add-modifiers could not take.
* 🐛 Make swapped children follow their parent's constraints on switch
A variant switch only moved a swapped child by the parent's position
change. When the new variant's root had a different size, a user-moved
child with a right constraint drifted, and a scale-constrained one kept
its grown size.
The switch now applies the parent change from the old parent to the new
one through the child's constraints. An untouched swapped child then
takes the place of the shape it replaces, as a swap does.
* 🐛 Sync nested swapped copies before the copies that hold them
A swapped child is a copy of its own component, so one sync handles it
twice: once on its own, and once when the copy that holds it places it.
Each step read the shape from the data as it was before the sync, so
the step that ran last won, and the order came from the order of shape
ids. When the swapped child came last, it wrote its old size and
position back over the placement, leaving its x and selrect apart, and
a swapped child with a scale constraint grew on every sync.
The sync now visits the most deeply nested shapes first, and each step
reads the shapes as the earlier steps left them, so the placement
always comes last and builds on the child's own sync.
* 🐛 Keep moved children with a scale constraint at their size on switch
A variant switch carries the size of a child the user moved as an
override. In a copy that fills a wider layout, a child with a scale
constraint already has the size the layout gave it, so the switch
carried that size onto the new variant, created at the main's size, and
the layout then scaled it again: the child grew on every switch.
Before comparing, the switch now resizes the previous child the way its
constraints follow the change from its old parent to its new one. A
child that only grew with the layout then matches the main and is not
carried as an override.
* 🐛 Keep the content of swapped children at its size on sync and switch
When a sync or a variant switch undid the parent change on a swapped
child through its constraints, it applied the child's resize to every
shape inside it. The layout then grows the child back through its own
constraint, but its content follows its own constraints (left and top
by default), so it stayed shrunk: the shape inside a swapped icon with
a scale constraint lost most of its width on every switch.
The change of the swapped child now goes down its subtree the way a
layout update does, each shape through its own constraints.
* 🐛 Accept mixed and empty values in the layout size constraints menu
With several shapes or a group selected, the sidebar merges the layout
item attributes and tokens of the shapes: nil when a shape lacks one,
:multiple when they differ. The props schema of layout-size-constraints*
only accepted the values of a single shape, so selecting such shapes
with fill sizing raised "invalid props".
The schema now accepts nil and :multiple, as the numeric inputs those
values go to do.
* ♻️ Share the swapped child follow-parent step between sync and switch
The component sync and the variant switch both make a swapped child and
its children follow the parent change through their constraints and then
move them to the swap place. Both now use generate-follow-parent, so the
two paths stay the same. The helper that writes geometry without marking
it touched is renamed to update-geometry-untouched.
Adds a test for a swapped child without absolute position in a slot
without layout, and fixes the test comments on why constraints are set
again after a swap.