📎 Log targets and final shapes in both engines of the replay

With `debug`, the browser replay logs each target and the shapes every
program ends with whether the WASM engine is on or off, so a WASM failure
can be compared with the same program on the plain engine.

Harness only; the benchmark is not run.

Also in this commit:

- Apply the generated typography only where the file holds it. A text style edit
  picks the generated typography for one of its values, but only worlds with
  text styles add that typography to the file. In the others the model applied
  the edit's styles while the frontend applied a missing typography, which sets
  no style, so the two diverged. The edit names the typography only when the
  file holds it.
- Replay programs in the browser as the frontend tests do. The browser replay
  mocks the backend and the timers as the frontend generated tests do, turns the
  WASM engine off for the plain engine even when the page's configuration turns
  it on, and in debug logs the content a text edit commits and each text's
  characters and font weights.
- Name the boards a deleted component's main was held by. The canonical design
  data names shapes and components by their place in the tree, so the model and
  the frontend compare although each allocates its own ids. A deleted component
  keeps the ids of the board that held its main, as a variant's component keeps
  its container's once the variant is undone; with that board gone the ids
  stayed raw and differed between the two runs. They are named after the
  component when no live shape is.
- Report where the model and the frontend first differ. A backend mismatch in
  the frontend replay printed the whole diff of the two canonical designs, which
  lists float noise the comparison accepts next to the difference it rejects.
  `oracle/inequivalence` walks the two designs as `equivalent?` compares them
  and returns the path of the first rejected value with both sides; the mismatch
  message leads with it.
- Keep text sizes and take inline programs in the browser replay. With the WASM
  engine on, the replay keeps each text's stored size in place of the engine's
  text resizes: the model keeps stored sizes, and the generated texts are too
  narrow to measure, so a new copy's measured text differed from its main's
  unmeasured one. The page also takes the programs inline in an `edn` query
  parameter, which a shrink loop drives.

AI-assisted-by: claude-opus-5-5
This commit is contained in:
alonso.torres 2026-11-14 16:45:55 +01:00
parent ac31d67366
commit abc64f0243
4 changed files with 112 additions and 28 deletions

View File

@ -877,9 +877,12 @@
(thtx/restyle text :size (nth [14 18 24] (mod (or (nth tuple 3 nil) 0) 3))))]
(n/edit-text target :content text))
(:text-style :library-text-style)
(let [typography? (and (= 8 (mod (or (nth tuple 2 nil) 0) (count text-style-edits)))
(odd? (or (nth tuple 3 nil) 0)))
file-id (or (:library-id selected) (:id (tm/file situation)))
(let [file-id (or (:library-id selected) (:id (tm/file situation)))
;; a typography applies only when the file holds it
typography? (and (= 8 (mod (or (nth tuple 2 nil) 0) (count text-style-edits)))
(odd? (or (nth tuple 3 nil) 0))
(contains? (get-in (cf/all-files situation) [file-id :data :typographies])
(thi/id :generated-typography)))
content (cond-> (edit-text-style (:shape selected) (nth tuple 2 nil) (nth tuple 3 nil))
(and typography? (:library-id selected))
(thtx/restyle :typography-ref-file file-id))]

View File

@ -308,7 +308,14 @@
:let [path (conj path :component index)]]
(swap! names assoc component-id path)
(when (:objects component)
(name-tree names (:objects component) (:main-instance-id component) (conj path :main)))))]
(name-tree names (:objects component) (:main-instance-id component) (conj path :main))
;; a deleted component keeps the ids of the board that held
;; its main, which may be gone, as a variant container is
;; once the variant is undone
(let [main (get (:objects component) (:main-instance-id component))]
(doseq [id [(:variant-id component) (:variant-id main) (:frame-id main) (:parent-id main)]
:when (and (some? id) (not (contains? @names id)))]
(swap! names assoc id (conj path :retained-container)))))))]
(doseq [[file-index [_ file]] (map-indexed vector files)]
(name-file ids file [:file file-index]))
(when (:accepted-libraries situation)
@ -360,6 +367,37 @@
(every? #(equivalent? (get expected %) (get actual %)) (keys expected))))
(uc/same-state? expected actual 0.01)))
(defn inequivalence
"The path of the first place where `expected` and `actual` are not
`equivalent?`, with the two values there, or nil."
([expected actual] (inequivalence expected actual []))
([expected actual path]
(cond
(equivalent? expected actual)
nil
(and (map? expected) (map? actual)
(= (set (keys expected)) (set (keys actual)))
(not (and (contains? expected :parent-id) (contains? actual :parent-id))))
(or (some #(inequivalence (get expected %) (get actual %) (conj path %)) (keys expected))
{:path path :expected expected :actual actual})
(and (map? expected) (map? actual) (= (set (keys expected)) (set (keys actual))))
(let [paired-expected (paired-matrix-effects expected actual)
paired-actual (paired-matrix-effects actual expected)]
(or (some (fn [k]
(when-not (uc/same-state? (get paired-expected k) (get paired-actual k) 0.01)
{:path (conj path k) :expected (get paired-expected k) :actual (get paired-actual k)}))
(keys paired-expected))
{:path path :expected paired-expected :actual paired-actual}))
(and (map? expected) (map? actual))
{:path path
:keys [(remove (set (keys actual)) (keys expected)) (remove (set (keys expected)) (keys actual))]}
:else
{:path path :expected expected :actual actual})))
(defn- constraint-derived-attributes
"The dimensions a constraint ties to the parent. Under a plain parent the
parent's resize history is unknown, and such a resize is not an override."

View File

@ -93,6 +93,7 @@
actual (oracle/canonical next-live parity)]
(t/is (true? (oracle/equivalent? expected actual))
(str "Backend mismatch at " tuple "; replay " (pr-str program)
"; first " (pr-str (oracle/inequivalence expected actual))
"; diff " (pr-str (thn/diff expected actual)))))
(t/is (empty? (uc/failures next-live)) (pr-str (uc/failures next-live)))
(recur next-state next-live (rest tuples))))

View File

@ -11,19 +11,26 @@
program replays as the frontend generated tests replay it, and the
results go to the console and to `globalThis.wasmReplay`.
Query parameters: `programs` names the file, `start` the first index,
`engine=plain` replays with the WASM engine off, and `debug` logs the
targets, the WASM layouts and the shapes of a failing program."
Query parameters: `programs` names the file or `edn` holds the programs,
`start` the first index, `engine=plain` replays with the WASM engine off,
and `debug` logs the targets, the WASM layouts and the shapes each
program ends with."
(:require
[app.common.test-helpers.composable.comp.nodes :as n]
[app.common.test-helpers.composable.core :as tm]
[app.common.types.text :as txt]
[app.main.data.helpers :as dsh]
[app.main.data.workspace.wasm-text :as dwwt]
[app.main.repo :as rp]
[app.main.store :as st]
[app.render-wasm.api :as wasm.api]
[beicon.v2.core :as rx]
[cljs.reader :as reader]
[cljs.test :as t]
[frontend-tests.composable-tests.generated-test :as gt]
[frontend-tests.composable-tests.interpreter :as ftm]
[frontend-tests.helpers.mock :as mock]
[potok.v2.core :as ptk]
[promesa.core :as p]))
(defn- query-param
@ -64,19 +71,42 @@
(add-watch st/state ::wasm-objects load-objects!)
situation)
(defn- svg-install!
"Install `situation` as `ftm/install!` does, with the WASM engine off even
when the page's configuration turns it on."
[situation]
(plain-install! situation)
(st/emit! #(update % :features disj "render-wasm/v1"))
situation)
(defn- kept-text-size
"An event in place of the WASM engine's text resizes. The engine measures
the texts it lays out, but the model keeps a text's stored size, and the
generated texts are too narrow to measure: a measured copy text would
differ from its unmeasured main. It takes the arities of the events it
stands for, which callers invoke directly."
([id] (kept-text-size id nil))
([_ _] (ptk/reify ::kept-text-size)))
(def ^:private plain-op->events ftm/op->events)
(defn- svg-text-events
"The events of `op` as `ftm/op->events` gives them, but a text edit runs
with the WASM engine off: the WASM text editor commits through its own
editing session, which the headless page does not have, so the edit goes
through the SVG editor's commit and the engine lays out what follows."
(defn- replay-events
"The events of `op` as `ftm/op->events` gives them, logging its target in
debug. With the WASM engine on, a text edit runs with the engine off: the
WASM text editor commits through its own editing session, which the
headless page does not have, so the edit goes through the SVG editor's
commit and the engine lays out what follows."
[op situation]
(when (and (query-param "debug") (some? (:target op)))
(js/console.log "WASM-REPLAY target" (pr-str (type op))
(pr-str (try (tm/target-shape-id situation (:target op)) (catch :default _ nil)))))
(when (and (query-param "debug") (instance? n/EditText op))
(let [shape (tm/shape-by-id situation (tm/target-shape-id situation (:target op)))]
(js/console.log "WASM-REPLAY edit" (pr-str (:edit op)) (pr-str (:value op))
(pr-str (some->> (:content (n/text-edit-commit shape (:edit op) (:value op)))
txt/node-seq (keep :font-weight) distinct)))))
(let [events (plain-op->events op situation)]
(if (instance? n/EditText op)
(if (and (instance? n/EditText op) (not (plain-engine?)))
(-> [#(update % :features disj "render-wasm/v1")]
(into events)
(conj #(update % :features (fnil conj #{}) "render-wasm/v1")))
@ -105,8 +135,10 @@
(plain-step! live op))
(defn- log-shapes!
"Log the shapes of the current page when the page names the `debug` query
parameter: name, type, parent, rectangle, transform and touched groups."
"Log the shapes of the current page at the end of a program when the page
names the `debug` query parameter: name, type, id, parent, rectangle,
transform, touched groups, shape ref, layout, and a text's characters and
font weights."
[]
(when (query-param "debug")
(let [objects (dsh/lookup-page-objects @st/state)]
@ -116,20 +148,26 @@
(:name (get objects (:parent-id shape)))
(some-> (:selrect shape) (select-keys [:x :y :width :height]))
(:transform shape) (:touched shape)
(:shape-ref shape) (:layout shape)]))))))
(:shape-ref shape) (:layout shape)
(some-> (:content shape) txt/content->text)
(some->> (:content shape) txt/node-seq (keep :font-weight) distinct)]))))))
(defn- ^:async replay
"Replay `program`, returning its failures and errors."
"Replay `program` with the backend and timers mocked, as the frontend
generated tests replay it, returning its failures and errors."
[program]
(t/set-env! (t/empty-env))
((:before ftm/fixture))
(try
(await (gt/replay-program program))
(catch :default cause
(js/console.log "WASM-REPLAY error" (ex-message cause) (some-> (.-stack cause) (subs 0 1500)))
(t/do-report {:type :error :message (ex-message cause) :actual cause})
(log-shapes!))
(await (mock/with-mocks* {rp/cmd! mock/rpc-cmd-mock rx/timer mock/timer-mock}
(try
(await (gt/replay-program program))
(catch :default cause
(js/console.log "WASM-REPLAY error" (ex-message cause)
(some-> (.-stack cause) (subs 0 1500)))
(t/do-report {:type :error :message (ex-message cause) :actual cause})))))
(finally
(log-shapes!)
(remove-watch st/state ::wasm-objects)
((:after ftm/fixture))))
(let [counters (:report-counters (t/get-current-env))]
@ -159,10 +197,14 @@
[]
(enable-console-print!)
(set! ftm/run-generated-step logged-step!)
(when-not (plain-engine?)
(set! ftm/install! wasm-install!)
(set! ftm/op->events svg-text-events)
(set! wasm.api/propagate-modifiers counted-propagate))
(set! ftm/op->events replay-events)
(if (plain-engine?)
(set! ftm/install! svg-install!)
(do (set! ftm/install! wasm-install!)
(set! wasm.api/propagate-modifiers counted-propagate)
(set! dwwt/resize-wasm-text kept-text-size)
(set! dwwt/resize-wasm-text-debounce kept-text-size)
(set! dwwt/resize-wasm-text-all kept-text-size)))
(.addEventListener js/globalThis "unhandledrejection"
#(js/console.log "WASM-REPLAY unhandled" (str (.-reason %))))
(p/let [_ @wasm.api/module
@ -171,8 +213,8 @@
_ (set! (.-height canvas) 600)
_ (.appendChild js/document.body canvas)
ready (wasm.api/init-canvas-context canvas)
resp (js/fetch (programs-uri))
text (.text resp)]
text (or (query-param "edn")
(p/let [resp (js/fetch (programs-uri))] (.text resp)))]
(js/console.log "WASM-REPLAY engine" (if ready "ready" "not ready")
(if (plain-engine?) "(off)" "(on)"))
(run-programs (reader/read-string text))))