github/penpot
1
0
Fork 0
mirror of https://github.com/penpot/penpot.git synced 2026-11-05 17:40:39 +00:00
Code Issues Packages Projects Releases Wiki Activity

✨ Save raw benchmark phases and reports

Browse Source 
Record raw phases and report statistics per execution.

AI-assisted-by: gpt-6.1-sol
This commit is contained in:
Miguel de Benito Delgado 2026-11-04 09:56:26 +00:00
parent 0bd3fcd679
commit b8dc029da1
14 changed files with 916 additions and 1337 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

157
frontend/performance/benches/render_wasm/README.md
View File

@ -93,9 +93,9 @@ The suite has three parts:
fingerprinting, upload-order derivation, and upload. It then runs the optional
preparation callback and drains to `Full` unless the case declares
`:preparation {:render :none}`. One page owns one preparation.
- **Attempt**: one execution of a case against a preparation. An attempt
records one outcome, one set of metrics, and a raw trace. A failed attempt
still consumes its allocated slot.
- **Attempt**: one execution of a case against a preparation. It records an
`:attempt-id`, one outcome, main and optional reset phase blocks, and failure
or partial evidence. A failed attempt still consumes its allocated slot.
- **Warmup**: an attempt that runs the case before measured attempts to warm
caches and stabilize the page. The run records warmups but excludes them from
statistics.
@ -105,9 +105,9 @@ The suite has three parts:
processes it starts, fixes warmup and repetition counts, records failures in
chronological order, and performs no hidden retries.
- **Run record**: the saved evidence for one run: plan, build and environment
provenance, preparations, attempts, metrics, traces, failures, remaining
slots, and termination. The runner stores it as a data file validated by a
schema.
provenance, quantity descriptions, preparations, attempts with their phase
histories, failures, remaining slots, and termination. The runner stores it as
a data file validated by a schema.
- **Bridge**: the object that runs in the browser, receives plain requests from
Node, owns the page's renderer context, runs preparations and attempts, and
returns plain results.
@ -120,13 +120,14 @@ The suite has three parts:
- **Measurement**: a numeric result for a quantity such as elapsed duration.
The suite measures module start, graphics start, upload, first render, and
camera movement separately.
- **Metric**: the name and definition of a measured quantity. Saved run metrics
declare a unit, semantics, basis, and zero policy. Semantics identifies what
the value measures. The basis states whether a value covers an attempt or
averages its inner calls. The zero policy states whether zero supplies a valid
value for analysis.
- **Observation**: one valid value from a measured attempt for statistical
analysis. A batch average supplies one observation for its attempt.
- **Metric**: the name of a measured quantity. A command event records its raw
numeric values in a vector. A saved run supplies a `:unit` and an optional
`:description` for each metric type. It stores no statistic or reduction.
- **Metric key**: `[phase scope metric-type]`. Scope is the command that recorded
the metric or `:global` for every command with that type in the phase.
- **Observation**: one eligible phase execution's sum for one metric key. Inner
calls, slices, and loop repetitions stay within that execution and do not add
observations. Warmups and failed attempts supply none.
- **Slice**: a trace entry for one renderer call: timestamp, flags, frame type,
and duration. Slices describe the work within an attempt.
- **Trace**: details of the work, including individual renderer calls,
@ -189,7 +190,7 @@ The suite has three parts:
- **Report**: the statistics and tables computed from one or more run records.
Reports run in Node or the browser and do not load the renderer.
- **Comparison**: an offline analysis of two saved run records. It pairs cases,
checks compatibility, and reports changes for each metric with intervals.
checks compatibility, and reports changes for each metric key with intervals.
- **A/B comparison**: a comparison of two runs that differ by one renderer
feature. The feature is off in one run and on in the other.
- **Permission**: In principle, only run records with equal builds and equal
@ -263,8 +264,12 @@ sequenceDiagram
B->>W: create the WebGL2 context
B->>B: build, validate, fingerprint the snapshot
B->>W: upload shapes in tree order
B->>W: render until Full
B-->>N: preparation evidence and raw phase blocks
B->>B: run preparation callback
B->>W: render Full unless disabled
B-->>N: preparation evidence and raw phase block
N->>N: persist the run record
N->>B: run main and reset
B-->>N: raw main and reset phase blocks
N->>N: persist the run record
```
@ -789,7 +794,9 @@ reading. Promise callbacks can run later as the browser schedules queued work.
Measured command events keep each metric's raw values in a vector. Repeated
commands produce separate events, including when they name the same metric.
Raw values preserve zero, negative, and non-finite measurements for analysis.
Raw values preserve zero, negative, and non-finite measurements in the saved
record. Reports sum one metric type within one phase execution before computing
statistics; saved records contain no aggregates.
### Renderer completion
@ -854,25 +861,29 @@ A run record contains:
timeout policy;
- metadata: Git revision and dirty flag, build facts, environment, and diagnostics;
- the resolved analysis settings;
- saved quantity descriptions;
- preparations in chronological order;
- attempts in chronological order, each with its case, preparation, outcome,
metrics, trace, and failure evidence;
- attempts in chronological order, each with its case, preparation,
`:attempt-id`, warmup flag, outcome, main/reset phase blocks, and failure or
partial evidence;
- remaining slots for each case;
- the termination reason.
A preparation records its case, fingerprint, effective graphics, and setup
diagnostics. An attempt records one outcome and its metric entries. Entries can
describe invalid values or missing measurements. Errors retain available
traces of render calls and waits after a failure.
A preparation records its case, warmup flag, setup diagnostics, phase history,
and any failure. A completed preparation also has its fingerprint and effective
graphics. An attempt records one outcome and its main and optional reset phase
blocks. A phase block retains ordered command events, raw metric vectors, and
optional completion diagnostics. Errors retain available phase evidence and
traces after a failure.
### Failures and termination
A failed attempt still consumes its allocated slot. A fresh preparation failure
records a failed attempt and consumes the slot. A warm preparation failure or a
warm attempt failure releases the prepared page and context and leaves that
case's remaining slots unattempted. A browser or page failure closes the browser
it started, abandons the current case, and starts a new browser for the next
case when possible. The run fixes its attempt count before execution and uses
records a failed attempt and consumes the slot. A reuse case's preparation
failure or attempt failure releases the prepared page and context and leaves
that case's remaining slots unattempted. A browser or page failure closes the
browser it started, abandons the current case, and starts a new browser for the
next case when possible. The run fixes its attempt count before execution and uses
each allocated slot at most once.
A run terminates in one of four states:
@ -890,22 +901,44 @@ that ended in failure.
### Summaries
A summary reports, for each case and metric:
A summary reports one entry per case and metric key
`[phase scope metric-type]`. The scope is the command that recorded the metric,
or `:global` for every command with that type in the phase. For each key, the
report sums the raw values of one metric type within each eligible phase
execution and computes statistics across those execution sums. It labels the
reduction `:statistic :sum`.
For each metric key, the report includes:
- the observed mean, median, minimum, maximum, sample standard deviation, and
median absolute deviation;
- a bootstrap confidence interval for the median. The interval comes from
repeated resampling of the observed attempts;
- observed p95 and p99 values, the 95th and 99th percentile durations, when the
sample is large enough;
- counts in the valid, invalid, absent, failed, unattempted, and warmup
categories;
- the definitions and units of the metric.
repeated resampling of the observed execution sums;
- observed p95 and p99 values when the sample is large enough;
- counts for the valid, invalid, absent, failed, unattempted, and warmup
categories, plus command invocation and raw-value counts;
- the saved quantity unit and optional description, and any validity
diagnostics.
Statistics use valid values from completed measured attempts. The record also
retains warmups and failures. A completed attempt can supply a valid value for
one metric and an invalid value for another. A pooled estimate combines values
across attempts and requires matching metric definitions.
Statistical `n` is the valid count. Invocations count command events and
raw-values count all recorded numbers across eligible executions; neither is the
number of observations. Warmups and failed attempts are excluded. A failed
attempt excludes both its main and reset blocks. A completed preparation remains
eligible after a later main or reset failure. A fresh warmup preparation is
excluded; one reuse preparation supplies one observation, not one per attempt.
A finite zero is valid. Negative and non-finite raw values remain in the saved
record but invalidate the dependent execution sum and aggregate; independent
metric keys remain usable. Unknown or malformed quantity metadata appears as a
diagnostic and leaves that key without usable statistics. A phase can complete
with no events, but a completed run requires at least one measurement across
each case's phases.
The case report also counts actual preparation, main, and reset executions.
Completed and failed counts use each execution's own outcome, and a separate
count records warmup executions. These phase counts do not filter by attempt
eligibility. A completed main block from an attempt whose reset failed still
appears in the main phase counts but supplies no observation.
The default analysis settings are:
@ -922,10 +955,10 @@ The run records draw count, seed, confidence, and minimum counts, and a report
can override them at analysis time. An estimate that cannot be computed keeps a
reason and, when applicable, the minimum count it needs.
The intervals are exploratory and conditional on the observed attempts.
Attempts within one run share evolving caches and can have dependent durations.
Repeated whole runs provide evidence about reproducibility across those cache
conditions.
The intervals are exploratory and conditional on available executions.
Incomplete and timed-out cases can select which observations survive. Attempts
within one run share evolving caches and can have dependent durations. Repeated
whole runs provide evidence about reproducibility across those cache conditions.
### Comparisons
@ -943,15 +976,21 @@ By default, a comparison requires equal:
- effective graphics settings and runtime environment;
- build mode, effective features, instrumentation settings, and recorded build
environment;
- warmup count, timeout policy, and metric definitions.
- warmup count, timeout policy, and saved quantity units.
Comparison pairs metric keys `[phase scope metric-type]`. Quantity description
text does not affect compatibility; changed prose is not a difference. Unknown
or malformed quantity metadata appears in the per-side report diagnostics and
leaves that key without usable statistics.
Git revision, dirty state, run ids, timestamps, and saved analysis defaults
provide provenance for interpreting the paired cases.
The comparison reports absolute and percentage median changes, with a bootstrap
interval for each change. A zero baseline suppresses a percentage change. The
comparison leaves a metric with differing definitions out unless the
all-differences permission allows raw values.
The comparison reports absolute and percentage median changes between execution
sums, with a bootstrap interval for each change. A zero baseline suppresses a
percentage change. The comparison leaves a metric key with differing quantity
units or incomplete quantity evidence out unless the all-differences
permission allows its valid raw values.
### Comparison permissions
@ -961,16 +1000,16 @@ mismatch with the rule that allowed it.
- `--allow-build-differences` acknowledges recorded build settings, effective
features, instrumentation, and the recorded build environment. Runtime
environment, graphics, operation settings, and metric definitions stay
environment, graphics, operation settings, and saved quantity units stay
strict.
- `--allow-scene-differences` acknowledges scene identity, scene seed, scene
parameters, and differences in fingerprint content. Execution settings stay
strict. Missing or varying fingerprint evidence still needs the
all-differences permission.
- `--allow-all-differences` acknowledges every compatibility difference,
including mixed metric definitions. The report compares valid raw values in
their recorded units. Record validation and measurement validity still govern
the analysis.
including mixed quantity units. The report compares valid raw execution sums
in their recorded units. Record validation and measurement validity still
govern the analysis.
Build and scene permissions combine by union. All-differences includes both.
A report shows the selected permissions, every recorded difference, and the
@ -1093,7 +1132,7 @@ and makes the invocation exit 1.
Other build differences require `--allow-build-differences`. Scene differences
require `--allow-scene-differences`. Operation, view,
context, preparation, completion, batch size, graphics, runtime environment, and
metric-definition differences stay strict unless `--allow-all-differences` is
saved quantity-unit differences stay strict unless `--allow-all-differences` is
selected. If two configurations report the same effective feature set, the
comparison includes their edge and labels it with that fact. Effective feature
sets describe the code settings that accompany the observed timings.
@ -1346,11 +1385,11 @@ frontend/performance/benches/render_wasm/
├── snapshot.cljs snapshot validation and upload order
├── fingerprint.cljs digest of the starting snapshot
├── identity.cljc build identity and bundle compatibility checks
├── measurement.cljs metric normalization
├── measurement.cljs projection of phase evidence into attempts
├── random.cljs seeded random generators
├── result.cljs run, comparison and feature-comparison records
├── result.cljs raw run, comparison and feature-comparison records
├── README.md this document
└── REPORTING.md records, metrics, and statistics
└── REPORTING.md raw records, metrics, and statistics
frontend/test/performance_tests/render_wasm/
├── runner.cljs test entry point, namespace registration, focus selection
@ -1366,9 +1405,9 @@ render-wasm/ Rust renderer and build
## Further reading
- [`REPORTING.md`](REPORTING.md) describes the record schemas, metric entries,
and statistical calculations in detail. This guide is the source for the
command line and the A/B workflow.
- [`REPORTING.md`](REPORTING.md) describes the record schemas, raw command
histories, quantity metadata, and statistical calculations in detail. This
guide is the source for the command line and the A/B workflow.
- [`render-wasm/README.md`](../../../../render-wasm/README.md) and the renderer
source describe the Rust side.
- [`frontend/src/app/render_wasm/`](../../../src/app/render_wasm/) contains the

72
frontend/performance/benches/render_wasm/measurement.cljs
View File

@ -5,67 +5,23 @@
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns benches.render-wasm.measurement
"Measurement normalization runs after renderer timing ends.
Explicit entries declare which values supply observations. Raw traces retain
zero spans, non-finite values and diagnostic settings. Render-call totals sum
existing final/cached call durations and supply one observation per attempt."
(:require
[benches.render-wasm.result :as result]))
(defn- duration
"Declares one total duration with its timing semantics and zero policy."
[value semantics zero-policy]
(result/metric {:value value :unit :ms :semantics semantics
:basis {:kind :attempt-total} :zero-policy zero-policy}))
(defn- add-duration
"Adds a known duration only when its raw field exists, preserving explicit nil."
[metrics measurement field id semantics zero-policy]
(if (contains? measurement field)
(assoc metrics id (duration (get measurement field) semantics zero-policy))
metrics))
(defn render-call-total
"Sums existing call durations after timing, without interpreting GPU or CPU work.
No calls or missing durations yield nil; recorded zero durations remain zero.
NaN and infinities propagate as raw values."
[measurement]
(let [slices (concat (:slices measurement) (:cached-slices measurement))]
(when (and (seq slices) (every? #(number? (:duration-ms %)) slices))
(reduce + 0 (map :duration-ms slices)))))
(defn metrics
"Normalizes standard durations and any authored explicit metric entries.
Fresh Full completion has the sole metric ID :first-render-ms. Warm setup
stays in preparation diagnostics. Requested waits and counts stay raw."
[measurement fresh? setup]
(let [entries (into {} (map (fn [[id entry]] [id (result/metric entry)])) (:metrics measurement))
entries (if fresh?
(-> entries
(add-duration setup :module-init-ms :module-init-ms :module-instantiation :positive)
(add-duration setup :graphics-init-ms :graphics-init-ms :graphics-initialization :positive)
(add-duration setup :upload-ms :upload-ms :shared-encoding-and-upload :positive)
(add-duration measurement :full-ms :first-render-ms :prepared-scene-to-full :positive)
(add-duration measurement :viewport-ready-ms :viewport-ready-ms :prepared-scene-to-viewport-ready :positive))
(-> entries
(add-duration measurement :interact-ms :interact-ms :camera-session-to-full :positive)
(add-duration measurement :active-ms :active-ms :camera-active-span :positive)
(add-duration measurement :settling-actual-ms :settling-actual-ms :actual-settling-span :allow)
(add-duration measurement :set-view-end-ms :set-view-end-ms :synchronous-set-view-end-call :allow)
(add-duration measurement :time-to-viewport-ready-ms :time-to-viewport-ready-ms :finalization-to-viewport-ready :positive)
(add-duration measurement :time-to-full-ms :time-to-full-ms :finalization-to-full :positive)
(add-duration measurement :last-input-to-full-ms :last-input-to-full-ms :last-input-to-full :positive)))]
(if (or (contains? measurement :slices) (contains? measurement :cached-slices))
(assoc entries :renderer-call-total-ms
(duration (render-call-total measurement) :synchronous-render-call-elapsed :allow))
entries)))
"Projects browser phase evidence into saved attempts.")
(defn attempt
"Projects bridge evidence into a raw attempt; the run assigns its ordinal.
Failed evidence remains raw and supplies no statistical observations."
"Returns an attempt with raw phase histories and available failure evidence.
Args:
case-id: saved case identifier.
preparation-id: preparation identifier, or nil for an early failure.
warmup?: whether this execution occupies a warmup slot.
evidence: browser response with status, phases and optional failure details.
Returns:
An attempt ready for chronological recording."
[case-id preparation-id warmup? evidence]
(cond-> {:case-id case-id :preparation-id preparation-id :warmup? warmup?
:outcome (if (= "ok" (:status evidence)) :completed :failed)
:metrics (or (:metrics evidence) {}) :trace (or (:trace evidence) {})}
(not= "ok" (:status evidence)) (assoc :failure (select-keys evidence [:status :phase :message :cause]))
:phases (filterv #(#{:main :reset} (:phase %)) (:phases evidence))}
(not= "ok" (:status evidence))
(assoc :failure (select-keys evidence [:status :phase :message :cause]))
(:partial evidence) (assoc :partial (:partial evidence))))

34
frontend/performance/benches/render_wasm/node/runner/run.cljs
View File

@ -58,7 +58,7 @@
"Projects a bridge failure response into diagnostics evidence."
[evidence fallback-phase]
(if (map? evidence)
(cond-> (select-keys evidence [:status :phase :message :cause :partial])
(cond-> (select-keys evidence [:status :phase :message :cause :partial :phases])
(nil? (:status evidence)) (assoc :status "failed")
(nil? (:phase evidence)) (assoc :phase (name fallback-phase)))
{:status "failed"
@ -194,23 +194,31 @@
(str "/renderer/" file "?build=" (js/encodeURIComponent build-id)))
(defn- ^:async prepare-case
[state case page deadline]
[state case page deadline warmup?]
(let [opts (:opts state)
build-id (:id (:selected-build state))
preparation-id (str (random-uuid))
response (attempt
(await ((get-in state [:services :prepare])
page
(transit/encode-str
{:case case
:preparation-id (str (random-uuid))
:preparation-id preparation-id
:timeout-ms (:timeout-ms opts)
:module-url (renderer-url build-id "render-wasm.js")
:wasm-url (renderer-url build-id "render-wasm.wasm")})
{:deadline deadline :signal (:signal state) :phase :preparation})))]
(cond
(failure? response) {:failure (cause-evidence response :preparation)}
(= "ok" (:status response)) {:preparation (:preparation response)}
:else {:failure (failure-evidence response :preparation)})))
(if (and (not (failure? response)) (= "ok" (:status response)))
{:preparation (assoc (:preparation response) :warmup? warmup?)}
(let [failure (if (failure? response)
(cause-evidence response :preparation)
(failure-evidence response :preparation))]
(record-preparation state {:preparation-id preparation-id :case-id (:id case)
:warmup? warmup?
:phases (or (:phases failure) [])
:diagnostics (or (get-in failure [:partial :setup]) {})
:failure (dissoc failure :phases)})
{:failure failure :preparation-id preparation-id}))))
(defn- ^:async run-attempt
"Returns the bridge evidence of one attempt, or the failure."
@ -363,10 +371,10 @@
(defn- ^:async run-fresh-slot
[state case page warmup? final?]
(let [deadline (process/deadline (:timeout-ms (:opts state)))
prepared (await (prepare-case state case page deadline))]
prepared (await (prepare-case state case page deadline warmup?))]
(if-let [evidence (:failure prepared)]
(do
(record-attempt state (measurement/attempt (:id case) nil warmup? evidence))
(record-attempt state (measurement/attempt (:id case) (:preparation-id prepared) warmup? evidence))
(await (finish-slot state case page
(if (interrupted? state) :interrupted :failed) final?)))
(let [preparation (:preparation prepared)]
@ -435,7 +443,7 @@
(defn- ^:async run-reuse-page
[state case page]
(let [deadline (process/deadline (:timeout-ms (:opts state)))
prepared (await (prepare-case state case page deadline))]
prepared (await (prepare-case state case page deadline false))]
(if-let [evidence (:failure prepared)]
(await (abandon-preparation state case evidence))
(await (run-reuse-prepared state case page (:preparation prepared))))))
@ -563,7 +571,11 @@
(defn- finish-run
[state]
(let [termination (get-termination @(:run-ref state) (interruption (:signal state))
@(:harness-failed? state))]
@(:harness-failed? state))
unmeasured (result/get-unmeasured-cases @(:run-ref state))
termination (if (and (= :completed (:reason termination)) (seq unmeasured))
{:reason :no-measurements :case-ids unmeasured}
termination)]
(when (:screenshots (:opts state))
(swap! (:run-ref state) update-in [:metadata :diagnostics :screenshots]
complete-screenshots (:cases state)))

23
frontend/performance/benches/render_wasm/quantities.cljs Normal file
View File

@ -0,0 +1,23 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns benches.render-wasm.quantities
"Portable descriptions of raw measured quantities.")
(def schema:description
[:map {:closed true}
[:unit keyword?]
[:description {:optional true} string?]])
(def descriptions
{:command/elapsed
{:unit :ms :description "Command core start through its completion endpoint."}
:renderer/call-elapsed
{:unit :ms :description "Elapsed time of each synchronous renderer call."}
:renderer/cache-call-elapsed
{:unit :ms :description "Elapsed time of each synchronous cached renderer call."}
:camera/finalization-to-full
{:unit :ms :description "Camera finalization through the renderer return at Full."}})

21
frontend/performance/benches/render_wasm/report/compare.cljs
View File

@ -67,7 +67,8 @@
[field baseline candidate]
(concat
(when-let [d (difference field baseline candidate)] [d])
(when (or (not= 1 (count baseline)) (not= 1 (count candidate)))
(when (or (not= 1 (count baseline)) (not= 1 (count candidate))
(contains? baseline nil) (contains? candidate nil))
[{:field [field :incomplete-or-varying-evidence]
:baseline baseline :candidate candidate :waived? false}])))
@ -114,8 +115,8 @@
(evidence-differences :effective-graphics (preparation-facts baseline b :effective-graphics)
(preparation-facts candidate c :effective-graphics))
(mapcat (fn [metric-id]
(let [bdefs (summary/definitions baseline b metric-id)
cdefs (summary/definitions candidate c metric-id)]
(let [bdefs (map #(dissoc % :description) (summary/definitions baseline b metric-id))
cdefs (map #(dissoc % :description) (summary/definitions candidate c metric-id))]
(evidence-differences [:metric metric-id] (set bdefs) (set cdefs))))
shared)))))
@ -168,14 +169,8 @@
(defn- side-report
"Reports one side with saved definitions, setup evidence and sample accounting."
[run id shared analysis allow-all-differences]
{:case-id id
:definition (some #(when (= id (:id %)) %) (get-in run [:plan :cases]))
:preparations (preparations run id)
:remaining (get-in run [:remaining id])
:failed-attempts (summary/get-failed-attempts run id)
:metrics (into {} (map (fn [metric-id]
[metric-id (summary/summarize-metric run id metric-id analysis allow-all-differences)])) shared)})
[run id shared analysis]
(summary/case-report run id shared analysis))
(defn- metric-changes
"Keeps diagnostic changes unavailable for differing or varying definitions.
@ -209,8 +204,8 @@
{:type ::incompatible :baseline b :candidate c :mismatches mismatches})))
{:explicit? explicit?
:mismatches mismatches
:baseline (side-report baseline b shared analysis allow-all-differences)
:candidate (side-report candidate c shared analysis allow-all-differences)
:baseline (side-report baseline b shared analysis)
:candidate (side-report candidate c shared analysis)
:metrics (into {}
(map (fn [id]
[id (metric-changes baseline candidate b c id mismatches

26
frontend/performance/benches/render_wasm/report/format.cljs
View File

@ -8,7 +8,7 @@
"Text formatting for run summaries and saved comparisons.
Text leads with observed median, bootstrap interval and distinct counts.
Full statistics and acknowledged differences retain units and observation bases."
Full statistics and acknowledged differences include units and phase identity."
(:require
[benches.render-wasm.result :as result]
[clojure.string :as str]))
@ -32,16 +32,25 @@
(defn- counts->text
"Prints valid, invalid, absent, failed, unattempted and excluded warmup counts."
[{:keys [valid invalid absent failed unattempted warmup]}]
[{:keys [valid invalid absent failed unattempted warmup invocations raw-values]}]
(str "valid=" valid " invalid=" invalid " absent=" absent " failed=" failed
" unattempted=" unattempted " warmup=" warmup))
" unattempted=" unattempted " warmup=" warmup
" invocations=" invocations " raw-values=" raw-values " n=" valid))
(defn metric->text
"Formats all statistics for one metric, including batch basis and support limits."
[id {:keys [definitions counts statistics]}]
(str (pr-str id) " median " (estimate->text (or (:median statistics) statistics))
"Formats statistics across phase execution sums with their units and counts.
Args:
id: vector of phase, command or global scope, and metric type.
report: quantity descriptions, accounting counts, statistics and diagnostics.
Returns:
Text with observed estimates, support limits and raw-value counts."
[id {:keys [definitions counts statistics diagnostics]}]
(str (pr-str id) " sum per execution; median " (estimate->text (or (:median statistics) statistics))
" CI " (estimate->text (or (:median-ci statistics) statistics)) " | " (counts->text counts)
"\n definitions " (pr-str definitions)
(when (seq diagnostics) (str "\n diagnostics " (pr-str diagnostics)))
(if-let [reason (:reason statistics)]
(str "\n unavailable (" (name reason) ")")
(str "\n mean " (estimate->text (:mean statistics))
@ -52,9 +61,12 @@
(defn- case-status->text
"Displays pending slots and failed attempts, including excluded warmups."
[{:keys [remaining failed-attempts]}]
[{:keys [remaining failed-attempts preparations phases]}]
(str " remaining " (pr-str remaining)
"\n phases " (pr-str phases)
" failed-warmups=" (count (filter :warmup? failed-attempts))
(when-let [failures (seq (keep :failure preparations))]
(str "\n preparation failures " (pr-str failures)))
(when (seq failed-attempts)
(str "\n failed attempts " (pr-str failed-attempts)))))

283
frontend/performance/benches/render_wasm/report/summarize.cljs
View File

@ -5,21 +5,17 @@
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns benches.render-wasm.report.summarize
"Reports use saved definitions and explicit metrics from schema:run records.
Warmups and failed attempts supply no observations. A completed attempt can
supply valid, invalid or absent metrics independently. Unattempted slots remain
separate. Batches and render slices never multiply observation counts."
"Offline analysis of raw command histories, grouped within phase executions."
(:require
[benches.render-wasm.report.analysis :as analysis]
[benches.render-wasm.result :as result]))
[app.common.schema :as sm]
[benches.render-wasm.quantities :as quantities]
[benches.render-wasm.report.analysis :as analysis]))
(def source-coverage
"Limits of compatibility checks on stored inputs and settings."
"Recorded inputs and settings checked; runtime-helper identity is unverified. Git SHA and dirty state record provenance.")
(def interpretation
"Scope of intervals and of renderer completion measurements."
"Intervals are exploratory and conditional on observed runs. Caches can evolve between attempts; repeat whole runs to investigate findings. Full denotes renderer submission completion. Render-call totals measure synchronous elapsed calls, without exclusive CPU or GPU-completion claims.")
"Intervals are exploratory and conditional on available executions. Incomplete and timed-out cases can select which observations survive. Caches can correlate attempts; repeat whole runs to investigate findings. Full denotes renderer submission completion. Sums combine only one metric type within one phase execution. Finite clock resolution can produce valid zero durations.")
(defn case-attempts
"Returns chronological attempts for an exact saved case identifier."
@ -31,94 +27,211 @@
[run case-id]
(filterv #(= :failed (:outcome %)) (case-attempts run case-id)))
(defn metric-ids
"Discovers dynamic metric identifiers in first-appearance order, including failures."
(defn executions
"Returns actual phase executions with their eligibility and saved identities.
Args:
run: validated saved run.
case-id: exact saved case identifier.
phase: preparation, main or reset.
Returns:
Chronological phase histories with phase and preparation/attempt outcomes."
[run case-id phase]
(let [attempts (case-attempts run case-id)]
(if (= :preparation phase)
(let [definition (some #(when (= case-id (:id %)) %) (get-in run [:plan :cases]))]
(mapv (fn [prep]
(let [block (first (:phases prep))]
{:preparation-id (:preparation-id prep)
:events (:events block)
:warmup? (and (not= :reuse (:context definition)) (:warmup? prep))
:outcome (:outcome block)
:preparation-outcome (if (:failure prep) :failed :completed)}))
(filter #(and (= case-id (:case-id %)) (seq (:phases %))) (:preparations run))))
(into []
(keep (fn [attempt]
(when-let [block (some #(when (= phase (:phase %)) %) (:phases attempt))]
(merge (select-keys attempt [:attempt-id :preparation-id :warmup?])
{:outcome (:outcome block) :attempt-outcome (:outcome attempt)
:events (:events block)}))))
attempts))))
(defn phase-counts
"Counts actual phase executions, including empty histories and excluded work.
Args:
run: validated saved run.
case-id: exact saved case identifier.
Returns:
Execution, completion, failure and warmup counts for each phase."
[run case-id]
(vec (distinct (mapcat (comp keys :metrics) (case-attempts run case-id)))))
(into {}
(map (fn [phase]
(let [blocks (executions run case-id phase)]
[phase {:executions (count blocks)
:completed (count (filter #(= :completed (:outcome %)) blocks))
:failed (count (filter #(= :failed (:outcome %)) blocks))
:warmup (count (filter :warmup? blocks))}])))
[:preparation :main :reset]))
(defn metric-ids
"Returns [phase command-or-global metric-type] keys in first appearance order.
Args:
run: validated saved run.
case-id: exact saved case identifier.
Returns:
Distinct report keys, including quantities recorded during failed work."
[run case-id]
(vec (distinct
(for [phase [:preparation :main :reset]
block (executions run case-id phase)
event (:events block)
metric (keys (:metrics event))
scope [(:command event) :global]]
[phase scope metric]))))
(defn definitions
"Returns every recorded definition for a metric, so drift remains visible."
[run case-id metric-id]
(->> (case-attempts run case-id)
(keep #(get-in % [:metrics metric-id]))
(map result/metric-definition)
(distinct)
(vec)))
"Returns valid saved quantity descriptions for a report key.
(defn observation-definitions
"Returns definitions only for valid, completed observations outside warmups."
[run case-id metric-id]
(->> (case-attempts run case-id)
(remove :warmup?)
(filter #(= :completed (:outcome %)))
(keep #(get-in % [:metrics metric-id]))
(remove :invalid-reason)
(map result/metric-definition)
(distinct)
(vec)))
Args:
run: validated saved run.
_case-id: exact saved case identifier.
id: vector of phase, command or global scope, and metric type.
Returns:
A vector containing the description, or an empty vector for invalid metadata."
[run _case-id [_phase _scope metric]]
(let [description (get-in run [:quantities metric])]
(if (sm/validate quantities/schema:description description) [description] [])))
(defn- invalid-reason
[value]
(cond
(not (js/Number.isFinite value)) :non-finite
(neg? value) :negative
:else nil))
(defn- execution-value
[block scope metric]
(let [events (->> (:events block)
(map-indexed vector)
(filter (fn [[_ event]]
(and (or (= :global scope) (= scope (:command event)))
(contains? (:metrics event) metric)))))
values (vec (mapcat #(get-in (second %) [:metrics metric]) events))
invalid (vec (for [[event-index event] events
[value-index value] (map-indexed vector (get-in event [:metrics metric]))
:let [reason (invalid-reason value)]
:when reason]
(merge (select-keys block [:attempt-id :preparation-id])
{:event-index event-index :value-index value-index
:value value :reason reason})))
total (when (and (seq values) (empty? invalid)) (reduce + 0 values))]
{:invocations (count events) :raw-values (count values)
:diagnostics (cond-> invalid
(and (some? total) (not (js/Number.isFinite total)))
(conj {:reason :non-finite-sum}))
:value total}))
(defn observations
"Returns valid values and distinct counts for a saved case/metric.
:unattempted counts remaining measured repetitions; case :remaining also
retains pending warmups. :warmup counts executed warmup attempts.
Failed measured attempts count as failed even with partial metrics.
Failed warmups count as warmup attempts; case :failed-attempts retains
their failures separately from measured observation counts."
[run case-id metric-id]
(reduce
(fn [out {:keys [warmup? outcome metrics]}]
(let [entry (get metrics metric-id)
kind (cond
warmup? :warmup
(= :failed outcome) :failed
(not (contains? metrics metric-id)) :absent
(:invalid-reason entry) :invalid
:else :valid)]
(cond-> (update-in out [:counts kind] inc)
(= kind :valid) (update :values conj (:value entry)))))
{:values []
:counts {:valid 0 :invalid 0 :absent 0 :failed 0
:unattempted (get-in run [:remaining case-id :repetitions])
:warmup 0}}
(case-attempts run case-id)))
"Sums one metric type per eligible phase execution.
Invocation and raw-value counts cover eligible executions, including invalid
values. Statistical n counts valid execution sums. Failed attempts exclude
both main and reset. Successful preparation survives later attempt failures.
Args:
run: validated saved run.
case-id: exact saved case identifier.
id: vector of phase, command or global scope, and metric type.
Returns:
Valid execution sums, accounting counts and invalidity diagnostics."
[run case-id [phase scope metric :as id]]
(let [description (get-in run [:quantities metric])
diagnostic (when (empty? (definitions run case-id id))
{:reason (if (contains? (:quantities run) metric)
:malformed-quantity :unknown-quantity)
:metric metric :description description})]
(reduce
(fn [out {:keys [warmup? outcome attempt-outcome preparation-outcome] :as block}]
(let [{:keys [value invocations raw-values diagnostics]} (execution-value block scope metric)
kind (cond
warmup? :warmup
(or (= :failed outcome) (= :failed attempt-outcome)
(= :failed preparation-outcome)) :failed
(zero? invocations) :absent
(or diagnostic (seq diagnostics)) :invalid
:else :valid)
eligible? (not (#{:warmup :failed} kind))]
(cond-> (update-in out [:counts kind] inc)
eligible? (update-in [:counts :invocations] + invocations)
eligible? (update-in [:counts :raw-values] + raw-values)
eligible? (update :diagnostics into diagnostics)
(= kind :valid) (update :values conj value))))
{:values [] :diagnostics (if diagnostic [diagnostic] [])
:counts {:valid 0 :invalid 0 :absent 0 :failed 0 :warmup 0
:invocations 0 :raw-values 0
:unattempted (if (= :preparation phase) 0
(get-in run [:remaining case-id :repetitions]))}}
(executions run case-id phase))))
(defn summarize-metric
"Computes statistics for one dynamic metric with its definitions and counts.
The optional allow-all-differences flag permits mixed-definition statistics;
scoped comparison permissions never change this summary policy."
([run case-id metric-id analysis] (summarize-metric run case-id metric-id analysis false))
([run case-id metric-id analysis allow-all-differences]
(let [{:keys [values counts]} (observations run case-id metric-id)
definitions (definitions run case-id metric-id)
mixed? (> (count (observation-definitions run case-id metric-id)) 1)]
{:definitions definitions
:counts counts
:mixed-definitions? mixed?
:statistics (if (and mixed? (not allow-all-differences))
{:reason :inconsistent-metric-definitions :count (count values)}
(analysis/statistics values analysis))})))
"Computes statistics across execution sums for one phase, scope and metric type.
Args:
run: validated saved run.
case-id: exact saved case identifier.
id: vector of phase, command or global scope, and metric type.
analysis: resolved statistical settings.
Returns:
Quantity descriptions, counts, diagnostics and statistics across executions."
[run case-id id analysis]
(let [{:keys [values counts diagnostics]} (observations run case-id id)]
{:definitions (definitions run case-id id)
:statistic :sum :counts counts :diagnostics diagnostics
:statistics (analysis/statistics values analysis)}))
(defn case-report
"Reports saved case facts and phase quantities using one analysis method.
Args:
run: validated saved run.
id: exact saved case identifier.
ids: selected report keys.
analysis: resolved statistical settings.
Returns:
Case definition, raw preparation/failure evidence, counts and metric reports."
[run id ids analysis]
{:case-id id
:definition (some #(when (= id (:id %)) %) (get-in run [:plan :cases]))
:preparations (filterv #(= id (:case-id %)) (:preparations run))
:phases (phase-counts run id)
:remaining (get-in run [:remaining id])
:failed-attempts (get-failed-attempts run id)
:metrics (into {} (map (fn [metric-id]
[metric-id (summarize-metric run id metric-id analysis)])) ids)})
(defn summarize
"Builds a portable report from a constructed or decoded schema:run record.
The run must already satisfy schema:run. Reporting uses one
resolved method.
Overrides affect reporting without changing the stored record."
"Reports a validated saved run using saved cases and quantity descriptions.
Args:
run: validated saved run.
overrides: optional statistical settings overriding the saved defaults.
Returns:
A report with provenance, termination and separate analyses for each case."
([run] (summarize run nil))
([run overrides]
(let [analysis (analysis/resolve-settings (merge (:analysis run) overrides))]
{:run-id (:run-id run)
:metadata (:metadata run)
:analysis analysis
{:run-id (:run-id run) :metadata (:metadata run) :analysis analysis
:termination (:termination run)
:source-coverage source-coverage
:interpretation interpretation
:cases (mapv (fn [{:keys [id] :as definition}]
{:case-id id
:definition definition
:preparations (filterv #(= id (:case-id %)) (:preparations run))
:remaining (get-in run [:remaining id])
:failed-attempts (get-failed-attempts run id)
:metrics (into {} (map (fn [metric-id]
[metric-id (summarize-metric run id metric-id analysis)]))
(metric-ids run id))})
:source-coverage source-coverage :interpretation interpretation
:cases (mapv (fn [{:keys [id]}]
(case-report run id (metric-ids run id) analysis))
(get-in run [:plan :cases]))})))

206
frontend/performance/benches/render_wasm/result.cljs
View File

@ -11,33 +11,24 @@
[app.common.transit :as transit]
[benches.render-wasm.codec :as codec]
[benches.render-wasm.features.plan :as feature-plan]
[benches.render-wasm.quantities :as quantities]
[benches.render-wasm.report.analysis :as analysis]))
(def schema:basis
"An attempt total or an average over a declared number of inner calls."
[:or
[:map {:closed true} [:kind [:= :attempt-total]]]
[:map {:closed true} [:kind [:= :per-call-average]] [:calls [:int {:min 1}]]]])
(def schema:metric-definition
"The declared meaning and observation basis of a metric.
Authors define units and semantics; the schema accepts any vocabulary.
Comparison checks their equality without interpreting or converting them."
(def schema:event
"One measured command invocation with its ordered raw values."
[:map {:closed true}
[:unit :any]
[:semantics :any]
[:basis schema:basis]
[:zero-policy [:enum :positive :allow]]])
[:command qualified-keyword?]
[:metrics [:map-of qualified-keyword? [:vector {:min 1} number?]]]
[:trace {:optional true} :any]])
(def schema:metric
"An explicit observation, including raw invalid values and their reason."
(def schema:phase
"Raw command history for one phase execution, including partial failures."
[:map {:closed true}
[:value :any]
[:unit :any]
[:semantics :any]
[:basis schema:basis]
[:zero-policy [:enum :positive :allow]]
[:invalid-reason [:maybe keyword?]]])
[:phase [:enum :preparation :main :reset]]
[:preparation-id some?]
[:outcome [:enum :completed :failed]]
[:events [:vector schema:event]]
[:diagnostics {:optional true} [:vector :any]]])
(def schema:fingerprint
"The content digest of the actual validated starting snapshot."
@ -46,24 +37,26 @@
[:digest [:re #"^[0-9a-f]{64}$"]]])
(def schema:preparation
"Untimed setup evidence for one owned scene."
"Setup diagnostics and callback measurements for one owned scene."
[:map {:closed true}
[:preparation-id some?]
[:case-id some?]
[:fingerprint schema:fingerprint]
[:effective-graphics map?]
[:diagnostics map?]])
[:warmup? boolean?]
[:fingerprint {:optional true} schema:fingerprint]
[:effective-graphics {:optional true} map?]
[:diagnostics map?]
[:phases [:vector schema:phase]]
[:failure {:optional true} map?]])
(def schema:attempt
"One chronological execution; failure consumes a slot and may retain evidence."
[:map {:closed true}
[:ordinal [:int {:min 0}]]
[:attempt-id [:int {:min 0}]]
[:case-id some?]
[:preparation-id [:maybe some?]]
[:warmup? boolean?]
[:outcome [:enum :completed :failed]]
[:metrics [:map-of :any schema:metric]]
[:trace map?]
[:phases [:vector schema:phase]]
[:failure {:optional true} map?]
[:partial {:optional true} map?]])
@ -98,6 +91,7 @@
[:timeout-policy [:map [:attempt-ms [:int {:min 1}]]]]]]
[:metadata schema:metadata]
[:analysis analysis/schema:settings]
[:quantities [:map-of qualified-keyword? :any]]
[:preparations [:vector schema:preparation]]
[:attempts [:vector schema:attempt]]
[:remaining [:map-of :any [:map {:closed true}
@ -132,13 +126,15 @@
[:map {:closed true} [:reason keyword?] [:count [:int {:min 0}]]]])
(def schema:metric-report
"Every metric definition, distinct counts and computed estimates."
"A sum within each phase execution and statistics across eligible executions."
[:map {:closed true}
[:definitions [:vector schema:metric-definition]]
[:definitions [:vector quantities/schema:description]]
[:statistic [:= :sum]]
[:counts [:map {:closed true}
[:valid [:int {:min 0}]] [:invalid [:int {:min 0}]] [:absent [:int {:min 0}]]
[:failed [:int {:min 0}]] [:unattempted [:int {:min 0}]] [:warmup [:int {:min 0}]]]]
[:mixed-definitions? boolean?]
[:failed [:int {:min 0}]] [:unattempted [:int {:min 0}]] [:warmup [:int {:min 0}]]
[:invocations [:int {:min 0}]] [:raw-values [:int {:min 0}]]]]
[:diagnostics [:vector map?]]
[:statistics schema:statistics]])
(def schema:comparison-side
@ -146,6 +142,9 @@
[:map {:closed true}
[:case-id some?] [:definition [:map [:id some?]]]
[:preparations [:vector schema:preparation]]
[:phases [:map-of :keyword [:map {:closed true}
[:executions nat-int?] [:completed nat-int?]
[:failed nat-int?] [:warmup nat-int?]]]]
[:remaining [:map {:closed true} [:warmups [:int {:min 0}]] [:repetitions [:int {:min 0}]]]]
[:failed-attempts [:vector schema:attempt]]
[:metrics [:map-of :any schema:metric-report]]])
@ -260,33 +259,6 @@
:base base-value
:additional additional-value}))))
(defn invalid-reason
"Classifies one raw measurement without changing its value.
Nil denotes an explicit missing value; an absent metric has no entry.
Aggregate durations require positivity unless their entry permits zero."
[{:keys [value zero-policy]}]
(cond
(nil? value) :missing-value
(not (number? value)) :not-numeric
(not (js/Number.isFinite value)) :non-finite
(neg? value) :negative
(and (zero? value) (= zero-policy :positive)) :below-resolution
:else nil))
(defn metric
"Validates an explicit metric definition and attaches its invalidity reason.
Authored invalidity reasons survive when the numeric value otherwise qualifies."
[entry]
(let [entry (assoc entry :invalid-reason (or (invalid-reason entry)
(:invalid-reason entry)))]
(require-valid! (sm/validate schema:metric entry) "invalid metric entry" {:metric entry})
entry))
(defn metric-definition
"Returns unit, semantics, basis and zero policy without a measurement value."
[entry]
(select-keys entry [:unit :semantics :basis :zero-policy]))
(defn- initial-remaining
"Creates each saved case's fixed budget of unattempted slots."
[{:keys [cases warmups repetitions]}]
@ -304,6 +276,23 @@
"measured attempt precedes planned warmups" {:case-id case-id})
(update-in remaining path dec)))
(defn get-unmeasured-cases
"Returns saved case ids whose phase histories contain no measured commands.
Args:
run: constructed or decoded run record.
Returns:
A vector of case identifiers in plan order."
[run]
(into []
(keep (fn [{:keys [id]}]
(when-not (some (comp seq :events)
(mapcat :phases (filter #(= id (:case-id %))
(concat (:preparations run) (:attempts run)))))
id)))
(get-in run [:plan :cases])))
(defn- check-completion
"Requires all planned attempts to succeed before a run can be completed."
[run termination]
@ -311,7 +300,55 @@
(require-valid! (every? #(every? zero? (vals %)) (vals (:remaining run)))
"completed run still has unattempted slots" {})
(require-valid! (not-any? #(= :failed (:outcome %)) (:attempts run))
"completed run has failed attempts" {})))
"completed run has failed attempts" {})
(require-valid! (not-any? :failure (:preparations run))
"completed run has failed preparations" {})
(require-valid! (empty? (get-unmeasured-cases run))
"completed case has no measurements" {:case-ids (get-unmeasured-cases run)})))
(defn- check-phases
[record allowed]
(let [phases (:phases record)]
(require-valid! (= (mapv :phase phases)
(filterv (set (map :phase phases)) allowed))
"phase order or identity is invalid" {})
(doseq [phase phases]
(require-valid! (= (:preparation-id record) (:preparation-id phase))
"phase references a different preparation" {})
(doseq [event (:events phase)]
(require-valid! (= 1 (count (get-in event [:metrics :command/elapsed])))
"command needs one elapsed value" {:command (:command event)})))
(when-not (or (:failure record) (= :failed (:outcome record)))
(require-valid! (every? #(= :completed (:outcome %)) phases)
"successful execution has a failed phase" {}))))
(defn- check-preparation
[preparation]
(require-valid! (sm/validate schema:preparation preparation) "invalid preparation" {})
(check-phases preparation [:preparation])
(when-not (:failure preparation)
(require-valid! (and (:fingerprint preparation) (:effective-graphics preparation)
(= [:preparation] (mapv :phase (:phases preparation))))
"completed preparation needs scene, graphics and phase evidence" {})))
(defn- check-attempt
[attempt preparation]
(require-valid! (sm/validate schema:attempt attempt) "invalid attempt" {})
(check-phases attempt [:main :reset])
(when (:preparation-id attempt)
(require-valid! (= (:case-id attempt) (:case-id preparation))
"attempt references an unknown or different preparation" {}))
(when (= :completed (:outcome attempt))
(require-valid! (and preparation (not (:failure preparation))
(not (:failure attempt))
(= :main (get-in attempt [:phases 0 :phase])))
"completed attempt needs successful preparation and main" {}))
(when (seq (:phases attempt))
(require-valid! (= :main (get-in attempt [:phases 0 :phase]))
"attempt phases must start with main" {})
(require-valid! (not (:failure preparation)) "attempt phases need successful preparation" {}))
(when (= :failed (get-in attempt [:phases 0 :outcome]))
(require-valid! (= 1 (count (:phases attempt))) "reset follows failed main" {})))
(defn check-run
"Validates schema:run and chronological accounting at a saved-data boundary."
@ -326,17 +363,11 @@
(require-valid! (= (count prep-ids) (count (set prep-ids))) "duplicate preparation" {})
(require-valid! (every? #(contains? (set case-ids) %) (map :case-id preps)) "unknown preparation case" {})
(require-valid! (= left (:remaining run)) "remaining slots disagree with attempts" {})
(doseq [[ordinal attempt] (map-indexed vector (:attempts run))]
(require-valid! (= ordinal (:ordinal attempt)) "attempts are not chronological" {})
(when-some [id (:preparation-id attempt)]
(require-valid! (= (:case-id attempt) (:case-id (get prep-map id)))
"attempt references an unknown or different preparation" {:preparation-id id}))
(require-valid! (or (= :failed (:outcome attempt)) (some? (:preparation-id attempt)))
"completed attempt needs a preparation" {})
(doseq [[id entry] (:metrics attempt)]
(require-valid! (= (:invalid-reason entry)
(or (invalid-reason entry) (:invalid-reason entry)))
"metric validity disagrees with raw value" {:metric-id id})))
(doseq [prep preps]
(check-preparation prep))
(doseq [[attempt-id attempt] (map-indexed vector (:attempts run))]
(require-valid! (= attempt-id (:attempt-id attempt)) "attempts are not chronological" {})
(check-attempt attempt (get prep-map (:preparation-id attempt))))
(check-completion run (:termination run)))
run)
@ -345,11 +376,12 @@
The plan defaults to 3 warmups, 10 repetitions and a 30000 ms attempt timeout.
For each case, all warmup slots precede measured slots. Failed attempts consume
their slot too; attempts retain chronological order across cases."
[{:keys [run-id started-at plan metadata analysis]}]
[{:keys [run-id started-at plan metadata analysis quantities]}]
(let [plan (merge {:warmups 3 :repetitions 10 :timeout-policy {:attempt-ms 30000}} plan)]
(check-run {:type :run-record
:run-id run-id :started-at started-at :ended-at nil
:plan plan :metadata metadata :analysis (analysis/resolve-settings analysis)
:quantities (merge quantities/descriptions quantities)
:preparations [] :attempts [] :remaining (initial-remaining plan)
:termination {:reason :running}})))
@ -363,7 +395,7 @@
The run must come from create-run or decode; earlier evidence is trusted."
[run preparation]
(check-open! run)
(require-valid! (sm/validate schema:preparation preparation) "invalid preparation" {})
(check-preparation preparation)
(require-valid! (contains? (:remaining run) (:case-id preparation)) "unknown preparation case" {})
(require-valid! (not-any? #(= (:preparation-id preparation) (:preparation-id %))
(:preparations run))
@ -372,20 +404,15 @@
(defn record-attempt
"Checks and appends one new execution, consuming its scheduled slot.
The run must come from create-run or decode, and metric entries from metric.
The run must come from create-run or decode.
Missing preparation is allowed for failures before scene setup completes.
Earlier attempts are trusted; encode and decode check the complete record."
[run attempt]
(check-open! run)
(let [attempt (-> (merge {:preparation-id nil :metrics {} :trace {}} attempt)
(assoc :ordinal (count (:attempts run))))]
(require-valid! (sm/validate schema:attempt attempt) "invalid attempt" {})
(when-some [id (:preparation-id attempt)]
(let [preparation (some #(when (= id (:preparation-id %)) %) (:preparations run))]
(require-valid! (= (:case-id attempt) (:case-id preparation))
"attempt references an unknown or different preparation" {:preparation-id id})))
(require-valid! (or (= :failed (:outcome attempt)) (some? (:preparation-id attempt)))
"completed attempt needs a preparation" {})
(let [attempt (-> (merge {:preparation-id nil :phases []} attempt)
(assoc :attempt-id (count (:attempts run))))]
(check-attempt attempt (some #(when (= (:preparation-id attempt) (:preparation-id %)) %)
(:preparations run)))
(-> run
(update :attempts conj attempt)
(update :remaining consume-slot attempt))))
@ -407,6 +434,7 @@
warmups and adds `additional`'s measured repetitions. Warmup attempts
and preparations used only by them are excluded from `additional`.
The merged record uses the base run's quantity descriptions.
The caller supplies the merged run id and may supply its end time."
[base additional {:keys [run-id ended-at]}]
(let [base (check-run base)
@ -433,6 +461,13 @@
[[:metadata :environment] "runtime environment"]
[[:metadata :scored?] "scored flag"]]]
(require-compatible! base additional path label))
(doseq [run [base additional]]
(require-valid! (every? #(sm/validate quantities/schema:description %)
(vals (:quantities run)))
"cannot merge malformed quantity descriptions" {:run-id (:run-id run)}))
(when-not (= (update-vals (:quantities base) #(dissoc % :description))
(update-vals (:quantities additional) #(dissoc % :description)))
(incompatible! "cannot merge runs with different quantity units" {}))
(let [base-attempts (:attempts base)
additional-measured (filterv #(not (:warmup? %)) (:attempts additional))
base-preparations (:preparations base)
@ -448,7 +483,7 @@
(let [offset (count base-attempts)
attempts (into (vec base-attempts)
(map-indexed (fn [index attempt]
(assoc attempt :ordinal (+ offset index)))
(assoc attempt :attempt-id (+ offset index)))
additional-measured))
plan (assoc (:plan base)
:repetitions (+ (get-in base [:plan :repetitions])
@ -476,6 +511,7 @@
:plan plan
:metadata metadata
:analysis (:analysis base)
:quantities (:quantities base)
:preparations (into (vec base-preparations) additional-preparations)
:attempts attempts
:remaining remaining

46
frontend/test/performance_tests/render_wasm/cli_integration.cjs
View File

@ -458,36 +458,35 @@ function runFixtureText({
}) {
const k = transit.keyword;
const map = (pairs) => transit.map(pairs);
const metric = map([
k("value"),
value,
k("unit"),
k("ms"),
k("semantics"),
k("elapsed"),
k("basis"),
map([k("kind"), k("attempt-total")]),
k("zero-policy"),
k("positive"),
k("invalid-reason"),
null,
const event = map([
k("command"), k("fixture/work"),
k("metrics"), map([k("command/elapsed"), [value]]),
]);
const preparation = map([
k("preparation-id"),
`${runId}-preparation`,
k("case-id"),
k(caseId),
k("warmup?"),
false,
k("fingerprint"),
map([k("algorithm"), k("sha-256"), k("digest"), fingerprint]),
k("effective-graphics"),
map([k("webgl2"), true]),
k("diagnostics"),
map([]),
k("phases"),
[map([
k("phase"), k("preparation"),
k("preparation-id"), `${runId}-preparation`,
k("outcome"), k("completed"),
k("events"), [],
])],
]);
const makeAttempt = (ordinal, warmup) =>
const makeAttempt = (attemptId, warmup) =>
map([
k("ordinal"),
ordinal,
k("attempt-id"),
attemptId,
k("case-id"),
k(caseId),
k("preparation-id"),
@ -496,10 +495,13 @@ function runFixtureText({
warmup,
k("outcome"),
k("completed"),
k("metrics"),
map([k("elapsed"), metric]),
k("trace"),
map([]),
k("phases"),
[map([
k("phase"), k("main"),
k("preparation-id"), `${runId}-preparation`,
k("outcome"), k("completed"),
k("events"), [event],
])],
]);
const attempts = [
...Array.from({ length: warmups }, (_, index) => makeAttempt(index, true)),
@ -583,6 +585,8 @@ function runFixtureText({
k("min-p99"),
500,
]),
k("quantities"),
map([k("command/elapsed"), map([k("unit"), k("ms")])]),
k("preparations"),
[preparation],
k("attempts"),
@ -1762,7 +1766,7 @@ test(
assert.equal(plan.get(k("warmups")), 1);
assert.equal(plan.get(k("repetitions")), 5);
assert.deepEqual(
attempts.map((attempt) => attempt.get(k("ordinal"))),
attempts.map((attempt) => attempt.get(k("attempt-id"))),
[0, 1, 2, 3, 4, 5],
);
assert.deepEqual(

18
frontend/test/performance_tests/render_wasm/compare_test.cljs
View File

@ -28,11 +28,7 @@
(defn- metric
[value]
(result/metric {:value value
:unit :ms
:semantics :elapsed
:basis {:kind :attempt-total}
:zero-policy :positive}))
{:command :fixture/work :metrics {:command/elapsed [value]}})
(defn- case-definition
[case-id scene]
@ -68,19 +64,21 @@
run (result/record-preparation
run
{:preparation-id preparation-id
:case-id case-id
:case-id case-id :warmup? false
:fingerprint {:algorithm :sha-256
:digest fingerprint}
:effective-graphics {:webgl2 true}
:diagnostics {}})
:diagnostics {}
:phases [{:phase :preparation :preparation-id preparation-id
:outcome :completed :events []}]})
run (result/record-attempt
run
{:case-id case-id
:preparation-id preparation-id
:warmup? false
:outcome :completed
:metrics {:elapsed (metric value)}
:trace {}})]
:phases [{:phase :main :preparation-id preparation-id
:outcome :completed :events [(metric value)]}]})]
(result/finish-run run "2026-10-26T00:01:00.000Z" {:reason :completed})))
(defn- write-run
@ -111,7 +109,7 @@
:allow-all-differences false}
(:permissions comparison)))
(t/is (empty? (get-in comparison [:pairs 0 :mismatches])))
(t/is (= 2 (get-in comparison [:pairs 0 :metrics :elapsed :absolute :value])))
(t/is (= 2 (get-in comparison [:pairs 0 :metrics [:main :fixture/work :command/elapsed] :absolute :value])))
(t/is (str/includes? text "Baseline"))
(t/is (str/includes? text "Candidate"))
(t/is (str/includes? text "case/load"))

24
frontend/test/performance_tests/render_wasm/feature_comparison_test.cljs
View File

@ -447,11 +447,7 @@
(defn- metric-entry
[value]
(result/metric {:value value
:unit :ms
:semantics :duration
:basis {:kind :attempt-total}
:zero-policy :positive}))
{:command :fixture/work :metrics {:command/elapsed [value]}})
(defn- fingerprint
[character]
@ -460,11 +456,12 @@
(defn- preparation
[case-id digest]
{:preparation-id (str (random-uuid))
:case-id case-id
:fingerprint (fingerprint digest)
:effective-graphics {:backend :fixture}
:diagnostics {}})
(let [id (str (random-uuid))]
{:preparation-id id :case-id case-id :warmup? false
:fingerprint (fingerprint digest)
:effective-graphics {:backend :fixture}
:diagnostics {}
:phases [{:phase :preparation :preparation-id id :outcome :completed :events []}]}))
(defn- fake-run-record
[selected-build run-home cases options {:keys [kind digest metric-value]
@ -494,8 +491,7 @@
:preparation-id nil
:warmup? false
:outcome :failed
:metrics {}
:trace {}
:phases []
:failure {:message "fixture failure"}})
(let [prep (preparation case-id digest)]
(-> run
@ -505,8 +501,8 @@
:preparation-id (:preparation-id prep)
:warmup? false
:outcome :completed
:metrics {:fixture/total-ms (metric-entry metric-value)}
:trace {}}))))))
:phases [{:phase :main :preparation-id (:preparation-id prep)
:outcome :completed :events [(metric-entry metric-value)]}]}))))))
run
cases))
termination (case kind

36
frontend/test/performance_tests/render_wasm/merge_test.cljs
View File

@ -6,16 +6,13 @@
(ns performance-tests.render-wasm.merge-test
(:require
[benches.render-wasm.report.summarize :as summary]
[benches.render-wasm.result :as result]
[cljs.test :as t :include-macros true]))
(defn- metric
[value]
(result/metric {:value value
:unit :ms
:semantics :elapsed
:basis {:kind :attempt-total}
:zero-policy :positive}))
{:command :fixture/work :metrics {:command/elapsed [value]}})
(defn- case-definition
[case-id]
@ -33,11 +30,12 @@
(defn- preparation
[preparation-id case-id]
{:preparation-id preparation-id
:case-id case-id
:case-id case-id :warmup? false
:fingerprint {:algorithm :sha-256
:digest (apply str (repeat 64 "a"))}
:effective-graphics {:webgl2 true}
:diagnostics {}})
:diagnostics {}
:phases [{:phase :preparation :preparation-id preparation-id :outcome :completed :events []}]})
(defn- attempt
[case-id preparation-id warmup?]
@ -45,8 +43,8 @@
:preparation-id preparation-id
:warmup? warmup?
:outcome :completed
:metrics {:elapsed (metric 1.0)}
:trace {}})
:phases [{:phase :main :preparation-id preparation-id
:outcome :completed :events [(metric 1.0)]}]})
(defn- make-run
[{:keys [run-id cases warmups repetitions git-sha build environment analysis
@ -110,7 +108,7 @@
(t/is (= 1 (get-in merged [:plan :warmups])))
(t/is (= 5 (get-in merged [:plan :repetitions])))
(t/is (= 12 (count (:attempts merged))))
(t/is (= (range 12) (map :ordinal (:attempts merged))))
(t/is (= (range 12) (map :attempt-id (:attempts merged))))
(t/is (= [true false false true false false false false false false false false]
(map :warmup? (:attempts merged))))
(t/is (every? #(every? zero? (vals %)) (vals (:remaining merged))))
@ -200,3 +198,21 @@
(t/is (thrown-with-msg? js/Error #"duplicate preparation ids"
(result/merge-runs base additional
{:run-id "merged"})))))
(t/deftest quantity-prose-does-not-change-merge-compatibility
(let [base (make-run {:run-id "base" :cases [:case/load] :warmups 0 :repetitions 1})
additional (-> (make-run {:run-id "additional" :cases [:case/load] :warmups 0 :repetitions 1})
(assoc-in [:quantities :command/elapsed :description] "Elapsed command work."))
merged (result/merge-runs base additional {:run-id "merged"})]
(t/is (= 2 (get-in merged [:plan :repetitions])))
(t/is (= merged (result/decode (result/encode merged))))
(let [malformed (assoc-in additional [:quantities :command/elapsed :description] 42)
report (summary/summarize malformed)]
(t/is (= :malformed-quantity
(get-in report [:cases 0 :metrics [:main :fixture/work :command/elapsed] :diagnostics 0 :reason])))
(t/is (thrown-with-msg? js/Error #"malformed quantity descriptions"
(result/merge-runs base malformed {:run-id "invalid"}))))
(t/is (thrown-with-msg? js/Error #"different quantity units"
(result/merge-runs base
(assoc-in additional [:quantities :command/elapsed :unit] :seconds)
{:run-id "invalid"})))))

1274
frontend/test/performance_tests/render_wasm/report_test.cljs
View File

File diff suppressed because it is too large Load Diff

33
frontend/test/performance_tests/render_wasm/run_test.cljs
View File

@ -87,8 +87,9 @@
(defn- fake-browser
[{:keys [launch-failure new-page-failure close-page-failure
prepare-failure run-failure screenshot-failure
interruption controller effective-graphics metrics]
:or {effective-graphics (constantly {}) metrics {}}}]
interruption controller effective-graphics events]
:or {effective-graphics (constantly {})
events [{:command :fixture/work :metrics {:command/elapsed [1]}}]}}]
(let [state (atom {:launches 0 :pages 0 :close-pages 0 :closes 0
:host nil :runs {} :preparations {}})]
{:state state
@ -142,7 +143,9 @@
:fingerprint {:algorithm :sha-256
:digest (apply str (repeat 64 "a"))}
:effective-graphics (effective-graphics case-id (:id page))
:diagnostics {}}]
:diagnostics {}
:phases [{:phase :preparation :preparation-id preparation-id
:outcome :completed :events []}]}]
(swap! state assoc-in [:preparations preparation-id] case-id)
{:status "ok" :preparation preparation}))))
:run
@ -165,7 +168,9 @@
:response
{:status "failed" :phase "interact" :message "attempt rejected"}
{:status "ok" :metrics metrics :trace {}})))
{:status "ok"
:phases [{:phase :main :preparation-id preparation-id :outcome :completed
:events events}]})))
:screenshot
(fn [page file _]
(check-page page)
@ -200,6 +205,15 @@
[run case-id]
(filterv #(= case-id (:case-id %)) (:attempts run)))
(t/deftest ^:async unmeasured-successful-work-saves-a-terminal-failure
(let [fake (fake-browser {:events []})
response (await (execute fake [(collected :case/unmeasured :reuse)]
(options fake (path/join @temporary-root "unmeasured"))))
saved (result/decode (fs/readFileSync (get-in response [:home :record-path]) "utf8"))]
(t/is (= 1 (:exit-code response)))
(t/is (= {:reason :no-measurements :case-ids [:case/unmeasured]} (:termination saved)))
(t/is (= :completed (get-in saved [:attempts 0 :outcome])))))
(defn- preparations-for
[run case-id]
(filterv #(= case-id (:case-id %)) (:preparations run)))
@ -343,9 +357,10 @@
run (:run result)
tries (attempts-for run :case/fresh)]
(t/is (= [:failed :completed] (mapv :outcome tries)))
(t/is (nil? (:preparation-id (first tries))))
(t/is (= (:preparation-id (first (preparations-for run :case/fresh)))
(:preparation-id (first tries))))
(t/is (= "scene-build" (get-in (first tries) [:failure :phase])))
(t/is (= 1 (count (preparations-for run :case/fresh))))
(t/is (= 2 (count (preparations-for run :case/fresh))))
(t/is (= :ok (get-in run [:metadata :diagnostics :screenshots "case/fresh" :status])))
(t/is (every? zero? (vals (get-in run [:remaining :case/fresh]))))
(t/is (= :failed (get-in run [:termination :reason])))
@ -886,11 +901,7 @@
(t/deftest ^:async feature-comparison-completes-with-managed-output
(let [root @temporary-root
cases [(collected :case/a :fresh)]
fake (fake-browser {:metrics {:first-render-ms
(result/metric {:value 1 :unit :ms
:semantics :prepared-scene-to-full
:basis {:kind :attempt-total}
:zero-policy :positive})}})
fake (fake-browser {})
prepared {:plan (feature-plan/plan ["a"] {:dry-run true})
:cases cases
:output-dir nil