github/penpot
1
0
Fork 0
mirror of https://github.com/penpot/penpot.git synced 2026-11-05 09:30:38 +00:00
Code Issues Packages Projects Releases Wiki Activity

📚 Improve readme, some docstrings, simplify

Browse Source
This commit is contained in:
Miguel de Benito Delgado 2026-11-03 11:58:38 +01:00
parent cbfc3208dc
commit 08098e2397
27 changed files with 894 additions and 348 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

584
frontend/performance/benches/render_wasm/README.md
View File

@ -1,30 +1,26 @@
# WASM renderer benchmarks
The renderer benchmark suite measures the Penpot WASM renderer in Chromium. It
provides means to declare parameterized scenes, then render them on a browser
process, recording how long each of several phases take. It can measure "cold"
starts, "warm" interactions like pan and zoom, and compare runs with different
renderer builds.
| 💡 Jump to [How to run](#how-to-run) below for usage instructions. |
| ------------------------------------------------------------------ |
The renderer benchmark suite measures the Penpot WebAssembly (WASM) renderer in
Chromium. Scenes describe shapes; cases describe the work to perform on them.
The suite records setup, rendering, pan and zoom durations and compares runs
with different renderer builds. [How to run](#how-to-run) gives the commands.
The suite has three parts:
1. A Node runner controls the experiment. It prepares or reuses a build, selects
test cases, serves the browser executor, launches the browser, schedules
"attempts" (since cases can fail), saves results, and prints reports.
2. The browser executes the experiment. It owns the WASM module, graphics
context, scene snapshot, "upload" (to the renderer), interactions, and
timing.
3. The reports tool analyses saved records. It runs in Node or the browser
without loading the renderer.
cases, serves the browser code, launches Chromium, schedules attempts, saves
results, and prints reports. An attempt is one execution of a case.
2. The browser creates the renderer and graphics context, constructs the scene
snapshot, uploads its shapes, performs the work, and records durations.
3. The reporting code computes statistics from saved records in Node or the
browser without loading the renderer.
## Contents
- [Main concepts](#main-concepts)
- [Architecture](#architecture)
- [Workloads](#workloads)
- [Renderer commands](#renderer-commands)
- [Measurement](#measurement)
- [Results and reports](#results-and-reports)
- [A/B feature experiments](#ab-feature-experiments)
@ -64,10 +60,14 @@ The suite has three parts:
- **Case**: a scene plus its execution settings: parameters, base view, context
policy, completion rule, optional operation settings, and a run function. A
case is the unit that is measured.
- **Operation**: the work a case performs during one attempt, written as a
function that runs in the browser. The function receives a runtime context
with the scene, module, view, clock, frame scheduler, and renderer hooks.
Standard operations load, pan, and zoom.
- **Operation**: the function that performs a case's work during one attempt.
It receives a runtime context with the case data, scene snapshot, renderer
module, view, and hooks. Standard operations load, pan, and zoom.
- **View**: the camera's zoom `:scale` and offsets `:x` and `:y`.
Optional `:viewport` settings give the canvas width, height, and device pixel
ratio, the number of drawing-buffer pixels per CSS pixel along each dimension.
- **Hook**: a function supplied through `:hooks` for clock readings,
scheduling, view updates, rendering, or checking whether work may continue.
- **Case descriptor**: What gets sent to the browser. It contains the case id,
scene, derived seed, parameters, view, context policy, completion rule, batch
size, preparation settings, and operation settings. Any functions are removed.
@ -80,16 +80,16 @@ The suite has three parts:
### Execution
- **Build**: one compiled renderer plus the matching catalogue and browser
executor. A build id ties the three artefacts together, and the runner rejects
executor. A build id ties the three outputs together, and the runner rejects
a mismatch.
- **Retained build**: a ready build that the suite stores under the managed
build directory and later runs can reuse without recompiling.
build directory for reuse in later runs.
- **Fresh case**: a case with `:context :fresh`. Each attempt gets a new page,
module, graphics context, and upload.
- **Warm case**: a case with `:context :reuse`. One page, module, graphics
context, snapshot, and upload serve all attempts of the case.
- **Preparation**: the untimed setup for one measured context. It covers module
instantiation, WebGL context creation, scene build, snapshot validation,
- **Preparation**: setup for one graphics context. It covers module
instantiation, WebGL2 context creation, scene build, snapshot validation,
fingerprinting, upload-order derivation, and upload. A warm preparation also
restores the starting view and drains to `Full`. One page owns one
preparation.
@ -114,13 +114,39 @@ The suite has three parts:
- **Runner**: the Node command line interface that builds or reuses a build,
selects cases, serves the build, launches Chromium, runs attempts, saves
records, and prints reports.
- **Measurement**: a timed phase. The suite times module start, graphics start,
upload, first render, and warm interaction separately. Each measurement
records its value, unit, and basis. The basis says whether the value covers a
whole attempt or averages a number of inner calls.
- **Slice**: one renderer call record inside an attempt: timestamp, flags,
frame type, and call duration. Slices are raw evidence. They are not
independent observations.
- **Browser executor**: the compiled page and JavaScript that perform
preparation and case operations. Its bridge exposes the operations that Node
can request.
- **Measurement**: a numeric result for a quantity such as elapsed duration.
The suite measures module start, graphics start, upload, first render, and
camera movement separately.
- **Metric**: the name and definition of a measured quantity. Saved run metrics
declare a unit, semantics, basis, and zero policy. Semantics identifies what
the value measures. The basis states whether a value covers an attempt or
averages its inner calls. The zero policy states whether zero supplies a valid
value for analysis.
- **Observation**: one valid value from a measured attempt for statistical
analysis. A batch average supplies one observation for its attempt.
- **Slice**: a trace entry for one renderer call: timestamp, flags, frame type,
and duration. Slices describe the work within an attempt.
- **Trace**: details of the work, including individual renderer calls,
cached previews, requested waits, and observed durations.
- **Clock reading**: a time in milliseconds from the clock's origin. Subtracting
two readings from that clock gives an elapsed duration.
- **Endpoint**: the clock reading at the end of the measured work. The renderer
endpoint is the reading immediately after a renderer call returns.
- **Render drain**: repeated renderer calls until one returns `Full`. Partial
completion schedules another call on the next animation frame.
- **Cached preview**: one render from cached tiles after a view update.
- **Camera session**: camera start, view updates with cached previews, a settling
wait, camera end, and the final render drain.
- **Active interval**: the part of a camera session from camera start through
the last cached preview and its guard check, or an explicit pause that follows
the preview.
- **Settling wait**: the requested pause after the active interval and before
ending the interaction. The observed wait includes scheduling delays.
- **Camera finalization**: the interval from immediately before ending the
interaction through the renderer call that returns `Full`.
- **Frame type**: the completion signal returned by one renderer call. `None`
means no frame; `Partial` means visible work remains; `Full` means the
renderer finished the current request; `ViewportReady` means the visible
@ -130,6 +156,34 @@ The suite has three parts:
- **Atlas**: a GPU texture that stores many tiles or a document-space preview.
Atlases let warm pan and zoom present cached pixels before new tiles finish.
### Command execution
- **Renderer command**: a function that accepts a command context or its Promise
and returns a Promise of the next context. The base function uses the declared
name, such as `pan`. The measured function adds `*`, such as `pan*`, and records
an event for its work.
- **Command context**: the map with keys `:case`, `:scene`, `:view`, and `:hooks`.
It carries the case descriptor, prepared scene snapshot, current view, and
hooks.
- **Command body**: the work declared by `defcommand`. A synchronous body
returns the next context. An asynchronous body completes through callbacks
named by its declaration.
- **Composite command**: a command that combines calls to other commands.
A declaration with `:measured-variant false` defines its base function and
collects events from the measured commands it calls.
- **Command execution**: one call to `commands/execute`, which runs a function
from one command context to the next and records measured calls in completion
order.
- **Event**: the record for one measured command call. It contains the command
name, vectors of raw metric values, and an optional trace.
- **Guard**: the `:check` hook. A return authorizes work; an error rejects it.
The renderer owner is the execution authorized to use the renderer. The guard
can report cancellation, deadline expiry, context loss, or replacement of
that owner.
- **Completion callback**: `complete` or `fail` in an asynchronous command body.
The first call determines the outcome: `complete` supplies the next context
and optional measurements; `fail` supplies a cause.
### Results
- **Report**: the statistics and tables computed from one or more run records.
@ -151,10 +205,10 @@ The suite has three parts:
- **Feature configuration**: the set of Cargo features enabled for one renderer
build.
- **Feature comparison**: all on and off combinations of the requested features.
`N` requested features produce `2^N` configurations.
$N$ requested features produce $2^N$ configurations.
- **Edge**: two configurations that differ in exactly one feature, compared from
the configuration with the feature off to the configuration with the feature
on. A comparison with `N` features has `N * 2^(N-1)` edges.
on. A comparison with $N$ features has $N 2^{N-1}$ edges.
- **Edge comparison**: the comparison record for one edge, stored with the
feature comparison.
- **Feature-comparison record**: the saved plan, configurations, runs, edges,
@ -168,20 +222,20 @@ The Node runner owns everything outside the page:
- it prepares or loads a build;
- it reads the compiled catalogue and selects cases;
- it serves the selected build from an owned loopback server;
- it serves the selected build from a server on `127.0.0.1`;
- it launches Chromium and creates pages;
- it schedules warmups and measured attempts;
- it records preparations, attempts, failures, and termination;
- it writes the run record after every material change;
- it asks the shared report code for the terminal summary.
- it writes the run record as work progresses;
- it prints a final report.
The browser bridge owns everything inside the page:
- it validates a request before claiming the page;
- it validates a request before assigning the page to that work;
- it imports the renderer module and creates the canvas and WebGL2 context;
- it builds, validates, fingerprints, orders, and uploads the scene;
- it restores the starting view outside timing for warm cases;
- it runs the case operation and times the requested phases;
- it runs the case operation and records the requested measurements;
- it returns plain data and disposes the context when asked.
The shared reporting code owns analysis:
@ -214,38 +268,37 @@ sequenceDiagram
N->>N: persist the run record
```
The node runner saves the initial plan before it starts a browser, and updates
The Node runner saves the initial plan before it starts a browser, and updates
the run record after each preparation and attempt, and at termination. Reports
are build from these saved records later.
use these saved records.
### Builds and catalogues
A build has three artefacts that must agree:
A build has three matching outputs:
- the renderer module;
- the case catalogue compiled for Node;
- the browser executor and its page.
Preparation compiles the renderer in release mode with Cargo, then compiles the
catalogue and browser executor with the same build _id_, tying the three artefacts
together. The runner checks the id before case collection and when a page starts,
and rejects a mismatch.
Build preparation compiles the renderer in release mode with Cargo, then compiles
the catalogue and browser executor with the same build id. The runner checks
the id before case collection and when a page starts, and rejects a mismatch.
An ordinary measurement build uses the renderer's default cargo feature set. A
An ordinary measurement build uses the renderer's default Cargo feature set. A
feature experiment may enable selected functional features. Both kinds of
measurement build should exclude diagnostic features such as `stats`, `profile`,
`profile-macros`, and `profile-raf`. The build checks that we are in release
mode, the optimization level, and disabled profiling marks.
The build record also captures a fixed list of environment values that Cargo
receives, such as compiler, target, and Emscripten settings. Note that it only
records the environment supplied to Cargo via a whitelist. It's not a full
environment dump, and it does not certify a hermetic build.
The build record captures a fixed list of environment values supplied to Cargo,
including compiler, target, and Emscripten settings. These values provide
provenance for interpreting a run. Reproducing a build also requires its other
environment settings and dependencies.
Builds are saved under `target/performance/render-wasm/builds/<id>/`. The
manifest records the build id, status, functional settings, Git provenance, and
build facts. An experiment run can select a retained build with `--build <id>`,
but a run without `--build` always prepares a new one.
and `run` prepares a new build when it omits `--build`.
`build` and a run that prepares its build accept `--features`,
`--[no-]default-features`, and `--build-timeout-ms`. A run with `--build` reuses
@ -266,10 +319,11 @@ process it starts. Cases run serially. A fresh case gets a new page and context
for each preparation. A warm case keeps one page and context for all of its
attempts.
The runner bounds every phase with a deadline. When a page stalls past its
deadline, Node stops the browser process and records the failure. The browser
bridge also checks for context loss, expired deadlines, and stale work from an
earlier page before and after asynchronous work.
The runner applies deadlines to startup, preparation, and attempts. When a page
stalls past its deadline, Node stops the browser process and records the failure.
The browser bridge also checks for context loss, expired deadlines, and work associated
with a superseded renderer owner before and after asynchronous work. A browser
thread occupied by synchronous work relies on the Node deadline to stop it.
Clean-up closes the page, then the browser, then the server. The runner deletes
a build selected with `--remove` only after the final run record is saved.
@ -299,23 +353,25 @@ produce the same snapshot.
(rect))))
```
Scene scopes do not nest. Ordinary `let`, `doseq`, and functions defined outside
the scope work inside the body. Each shape constructor accepts an optional label
Each scene has one construction scope. Starting another scene inside it raises
an error. Ordinary `let`, `doseq`, and functions defined outside the scope work
inside the body. Each shape constructor accepts an optional label
or a label plus attributes. Attributes override generated defaults by key.
Labels must be keywords or vectors and must be unique within the snapshot.
Containers follow the same pattern:
- `frame` keeps its authored `:x`, `:y`, `:width`, and `:height`.
- `frame` keeps the supplied `:x`, `:y`, `:width`, and `:height`.
- `group` derives bounds from its children. A masked group uses its first child
as the mask.
- `bool` derives content and bounds from its children with the standard path
engine. It requires a boolean operation type and cannot contain frames.
engine. It requires a boolean operation type. Frame children raise a
construction error.
### Declaring a case
A case id uses the scene namespace. The `defcase` form derives the scene from
that namespace and attaches the run function.
For a scene named `:rects`, case ids start with `:rects/`. The `defcase` form
derives the scene from that keyword namespace and attaches the run function.
```clojure
(defcase :rects/pan
@ -344,20 +400,20 @@ The case declares:
- `:batch-size` and `:preparation`: execution settings recorded in the
descriptor and checked by comparisons. A batch size groups inner calls inside
one attempt; a repetition is a separate attempt. Standard load, pan, and zoom
cases run one call per attempt.
cases run one call per attempt.
### Case collection and seeds
Node collects cases from the compiled catalogue. Collection loads every
declaration, validates every scene and case, derives the scene seed, projects
the case to plain data, and checks that the descriptor survives a serialization
round trip. The process does not build shapes, start a browser, or initialize
the renderer.
declaration, validates every scene and case, derives the scene seed, selects
the case's data fields, and checks that the descriptor survives a serialization
round trip. Shape construction, browser startup, and renderer initialization
take place during execution.
The scene seed depends only on the master seed, the scene id, and the canonical
parameters. It does not depend on the case id, collection order, operation
body, attempt count, or feature configuration. This keeps load, pan, and zoom
cases on the same scene content and keeps A/B configurations comparable.
parameters. These inputs give load, pan, and zoom cases the same starting
content across case bodies, collection orders, attempt counts, and feature
configurations. They also give A/B configurations matching scene content.
A filter selects cases by a substring of the case id. An empty selection is an
error. Declarations outside the filter are still validated.
@ -369,8 +425,8 @@ their object keys, consistent child and parent links, a tree that reaches every
shape from the root, and references that point at present shapes. Validation
runs before any timer starts.
The browser derives upload order from the snapshot and does not store it in the
snapshot. The root comes first, then descendants in the order of each shape's
The browser derives upload order from the snapshot. The root comes first, then
descendants in the order of each shape's
child list. The browser computes the order and the fingerprint before the upload
clock starts. The upload uses the same batch serializer as the editor, with
structure, fill, and stroke data batched and path and SVG attributes handled per
@ -389,16 +445,214 @@ A load case renders a fresh scene and times module start, graphics start,
upload, and the first render to `Full`. A pan or zoom case prepares the scene
once, restores the starting view outside timing for each attempt, then times a
camera session. The standard camera sessions use 20 view updates and a 100 ms
settle wait. Pan and zoom parameters vary by scene.
settling wait. Pan and zoom parameters vary by scene.
## Renderer commands
A renderer command accepts a command context or its Promise and returns a
Promise of the next context. The command context has four fields:
| Field | Meaning |
| --- | --- |
| `:case` | Case descriptor, including the case id. |
| `:scene` | Prepared starting snapshot with concrete shapes and references. |
| `:view` | Current zoom scale, camera offsets, and optional viewport settings. |
| `:hooks` | Functions for clocks, scheduling, rendering, view changes, and the execution guard. |
The `:scene` field describes the starting snapshot. Commands return view changes
through `:view` and apply renderer work through hooks. Every command context
supplies `:now`, `:check`, `:frame`, `:sleep`, `:render`, `:render-sync`,
`:render-from-cache`, `:set-view`, `:set-view-start`, and `:set-view-end` as
functions. The `:now` hook returns a clock reading in milliseconds.
The `:check` guard authorizes work by returning and reports cancellation,
deadline expiry, context loss, or a superseded owner
by throwing an error. The owner is the execution authorized to use the renderer.
Returned contexts retain the execution's `:check` guard; replacing it rejects
completion.
### Declaring a command
`defcommand` defines two functions from one body. The base function runs the
work; the function with a `*` suffix also records a measurement event. Measured
functions require an active command execution. Both functions resolve their
incoming context before running the body. A synchronous body
returns the next command context.
```clojure
(ns example.renderer-commands
(:require
[benches.render-wasm.runtime.camera :as camera]
[benches.render-wasm.runtime.commands :as commands :refer [defcommand]]
[benches.render-wasm.runtime.protocol :as protocol]))
(defcommand render-sync
"Renders the prepared scene synchronously."
[rtx]
((get-in rtx [:hooks :render-sync]))
rtx)
```
`:pre` conditions validate arguments before work and timing in every build.
Input resolution, context validation, and argument validation precede the start
clock reading. Context validation checks the complete set of hooks. Output
validation and event recording follow the endpoint.
The body duration is
\[
\Delta t = t_{\mathrm{end}} - t_{\mathrm{start}},
\]
where both clock readings use milliseconds from the same origin. A synchronous
command captures the endpoint immediately after its body returns. An
asynchronous command supplies an endpoint or captures it when `complete` runs.
### Asynchronous completion
The option `:async [complete fail]` supplies two completion callbacks to the
body. Their names belong to the declaration; the public command arguments stay
the same. The first callback call determines the outcome:
- `(complete next-rtx)` succeeds with the next command context and captures the
endpoint at that call.
- `(complete next-rtx evidence)` succeeds with an explicit endpoint and raw
measurements when `evidence` supplies them.
- `(fail cause)` rejects the command with its cause and available partial trace.
Successful completion requires a call to `complete`. A returned Promise forwards
its rejection to `fail`. The execution records diagnostics for repeated
completion calls while it is active. Its guard checks authorization at
completion, and the measured command records its event after output validation.
The optional completion map has these fields:
| Field | Meaning |
| --- | --- |
| `:end-ms` | Clock reading at the end of the work, using the same clock origin as the start. |
| `:metrics` | Namespaced metric keywords mapped to nonempty vectors of raw numbers. |
| `:trace` | Details of the work, such as traces of render calls and requested waits. |
The command supplies `:command/elapsed`, its body duration in milliseconds.
`:metrics` supplies additional quantities. Command events preserve zero,
negative, `NaN`, and infinite numeric values as raw measurements.
Camera completion illustrates the callback interface:
```clojure
(defcommand pan
"Moves the view and completes when rendering reaches Full."
{:async [complete fail]}
[rtx {:keys [dx dy steps settle-ms]}]
(let [target (-> (:view rtx)
(update :x + dx)
(update :y + dy))]
(-> (protocol/start-camera! rtx {:settle-ms settle-ms})
(protocol/animate-view! {:to target :steps steps})
(protocol/finish-camera!)
(.then (fn [trace]
(complete (assoc rtx :view target)
{:end-ms (:end-ms trace)
:metrics {:renderer/call-elapsed
(mapv :duration-ms (:slices trace))}
:trace trace})))
(.catch fail))))
```
The final renderer call can return before the Promise callback runs. The
`:end-ms` field carries that renderer endpoint through the scheduling delay.
For example, renderer work lasting 7 ms followed by a 500 ms callback delay
still supplies a renderer duration of 7 ms. A clock reading in the later
callback would measure 507 ms.
The provided `camera/pan` and `camera/pan*` commands accept explicit `:dx`, `:dy`,
`:steps`, and `:settle-ms`. They submit interpolated views on animation frames,
draw cached previews, wait for settling, and finish rendering to `Full`. They
resolve to the context with the final view. The measured function records:
| Metric | Values in one event, in milliseconds |
| --- | --- |
| `:command/elapsed` | One duration from the start of the body through the renderer endpoint. |
| `:renderer/call-elapsed` | One duration per renderer call in the final drain. |
| `:renderer/cache-call-elapsed` | One duration per cached render call. |
| `:camera/finalization-to-full` | One duration from before ending the interaction through the renderer endpoint. |
Its trace also contains individual render calls, view-update durations, waits,
and the final `:end-ms`. Each vector preserves the call order within its quantity.
### Composition and execution
A composite combines commands. `:measured-variant false` defines its base
function and selects measurements through the calls in its body:
```clojure
(defcommand selected-pans
{:measured-variant false}
[rtx]
(-> rtx
(camera/pan* {:dx 10 :dy 0 :steps 2 :settle-ms 5})
(camera/pan {:dx 0 :dy 5 :steps 2 :settle-ms 5})
(camera/pan* {:dx -10 :dy 0 :steps 2 :settle-ms 5})))
```
All three calls update the view and renderer. The first and third record
events. Command functions resolve incoming Promises, so threading passes each
updated context to the next command.
A command execution runs a function from one command context to the next,
authorizes its work, and collects events in completion order. Given a prepared
command context `rtx`:
```clojure
(commands/execute rtx selected-pans)
```
The returned Promise resolves to an outcome. A successful outcome contains
`:outcome :completed`, the next `:context`, and ordered `:events`. A failed
outcome contains `:outcome :failed`, `:error`, and the events recorded before
failure. Existing Errors retain their identity and partial trace. Other thrown
values become Errors with the original value in `:thrown`. An active execution
can also record repeated completion calls in `:diagnostics`.
An event has this shape:
```clojure
{:command :benches.render-wasm.runtime.camera/pan
:metrics {:command/elapsed [74]
:renderer/call-elapsed [7 7]
:renderer/cache-call-elapsed [2 2]
:camera/finalization-to-full [33]}
:trace {:end-ms 74}}
```
One event represents one measured command call. Its metric vectors describe
the individual values that call produced. The command execution owns this
history; its guard authorizes work for that execution. Pass the resulting
context to `commands/execute` to start another command execution.
### Runtime code
[`runtime/commands.clj`](runtime/commands.clj) declares commands;
[`runtime/commands.cljs`](runtime/commands.cljs) executes them.
[`runtime/camera.cljs`](runtime/camera.cljs) defines rendering and camera
operations. [`runtime/protocol.cljs`](runtime/protocol.cljs) defines render
drains, camera sessions, hook contracts, and trace fields.
Case declarations such as [`scenes/rects.cljs`](scenes/rects.cljs) attach operation
bodies through [`declarations.cljc`](declarations.cljc). Node loads these
declarations and functions during catalogue collection. Renderer operations run
through the supplied hooks when the functions execute in the browser.
[`browser/bridge.cljs`](browser/bridge.cljs) constructs the runtime context for
case operations and prepares graphics contexts, snapshots, and uploads. Reused
cases restore their starting view outside interaction timing before each attempt.
## Measurement
### Preparation and measurement boundaries
The suite keeps slow setup out of timed code. This ties the numbers to renderer
work instead of test code.
Each duration has a defined start and endpoint. Setup and diagnostics run
around the interval whose work they prepare or describe.
Outside timers:
Work around those intervals includes:
- scene build, snapshot validation, and fingerprinting;
- upload order derivation;
@ -407,7 +661,7 @@ Outside timers:
- module script import;
- camera restore and the drain that prepares a warm case.
Inside timers:
The timed work includes:
- module instantiation;
- graphics context creation and initial size synchronization;
@ -445,15 +699,15 @@ runs a camera session:
1. start the camera interaction;
2. for each view update, wait for the next animation frame, apply the view, and
present cached pixels;
3. wait for the settle interval;
3. wait for the requested settling time;
4. end the camera interaction;
5. render until `Full`.
A pure pan can reuse cached textures while it updates which tiles are visible.
A zoom rebuilds and invalidates tile textures. The suite keeps the 100 ms settle
interval and records the requested and actual waits separately from renderer
work. The final drain includes the end of the interaction; the run also records
that call's own duration.
A zoom rebuilds and invalidates tile textures. Standard cases request a 100 ms
settling wait and record the requested and observed durations separately from
renderer calls. Camera finalization includes the call that ends the interaction;
the run also records that call's duration.
### Slices and metrics
@ -472,34 +726,39 @@ A fresh attempt reports one drain:
A warm attempt reports the camera session and the final drain:
- `cached-slices`: one record per gesture frame with `timestamp`,
`duration-ms` for the cached-preview call, and `view-and-preview-ms` for the
`duration-ms` for the cached preview call, and `view-and-preview-ms` for the
view update plus preview call;
- `slices`: the final drain, with the same shape as a fresh drain;
- `interact-ms`: the whole camera session, from before the interaction starts
through the final `Full`;
- `active-ms`: from before the interaction starts to after the last cached
preview, or after an authored pause that follows it;
preview and its guard check, or after an explicit pause that follows it;
- `settling-requested-ms` and `settling-actual-ms`: the declared and observed
settle spans;
- `set-view-end-ms`: the call that ends the interaction, on its own clock;
- `time-to-viewport-ready-ms` and `time-to-full-ms`: the final drain
boundaries;
- `last-input-to-full-ms`: from before the last view update to after the final
drain resolves.
settling waits;
- `set-view-end-ms`: elapsed time during the call that ends the interaction;
- `time-to-viewport-ready-ms` and `time-to-full-ms`: durations from camera
finalization through the first `ViewportReady` or `Full`, and through `Full`,
respectively;
- `last-input-to-full-ms`: from before the last view update to the endpoint of
the renderer call that returns `Full`.
Both drains and camera sessions return `end-ms`, the clock reading immediately
after the final renderer call returns `Full`. Completion durations end at that
reading. Promise callbacks can run later as the browser schedules queued work.
Both kinds of attempt report `renderer-call-total-ms`, the sum of the renderer
call durations, computed after timing.
A metric names its unit, semantics, basis, and zero policy. A batch average
declares its call count, but one attempt still supplies one observation. Raw
slices keep zero durations and non-finite values; the metric layer marks values
that cannot supply an observation.
A saved run metric declares its unit, semantics, basis, and zero policy. A batch
average records its call count; one attempt supplies one observation. Raw slices
keep zero durations and non-finite values. The metric entry classifies the value
for analysis and records a reason when it is invalid.
### What completion means
### Renderer completion
The renderer finishes in tile budgets. One renderer call returns a frame type,
and the caller continues while the renderer reports `Partial` or
`ViewportReady`. The run completes when a call returns `Full`. `None` or an
`ViewportReady`. The drain completes when a call returns `Full`. `None` or an
unknown value fails the attempt.
| Value | Frame type | Meaning |
@ -509,15 +768,14 @@ unknown value fails the attempt.
| 2 | `Full` | the renderer finished the current request |
| 3 | `ViewportReady` | the visible viewport was submitted; work outside it may remain |
`Full` is a renderer submission boundary. It does not mean that the GPU has
finished or that pixels are visible. Skia and WebGL submit commands without
waiting for completion, and the suite has no signal for GPU completion or
displayed pixels. A call duration measures synchronous work inside the renderer
call plus any driver stall that happens on the calling thread.
`Full` marks renderer submission of the completed request. Skia and WebGL queue
GPU work, and GPU execution and display follow their own schedules. A call
duration measures synchronous work on the calling thread, including any driver
stall. These clock readings describe work through the renderer return.
For scheduled calls, the browser passes the timestamp of the animation frame.
For immediate calls, it reads the clock. Each slice records its timestamp so
that call sequences can be reconstructed.
For immediate calls, it reads the clock. Each slice records its timestamp to
describe the call sequence.
### Graphics environment
@ -547,8 +805,7 @@ the GPU policy and environment signal under diagnostics. Default runs allow
SwiftShader fallback and warn when they record software
rendering. Comparisons check effective graphics settings and resolved launch
arguments. Comparing hardware and software runs requires
`--allow-all-differences`: whether that comparison needs a narrower permission
is still an open question.
`--allow-all-differences`.
## Results and reports
@ -567,8 +824,9 @@ A run record contains:
- the termination reason.
A preparation records its case, fingerprint, effective graphics, and setup
diagnostics. An attempt records one outcome which can have some invalid or
absent metrics due to failures. We try to retain trace evidence after a failure.
diagnostics. An attempt records one outcome and its metric entries. Entries can
describe invalid values or missing measurements. Errors retain available
traces of render calls and waits after a failure.
### Failures and termination
@ -577,8 +835,8 @@ records a failed attempt and consumes the slot. A warm preparation failure or a
warm attempt failure releases the prepared page and context and leaves that
case's remaining slots unattempted. A browser or page failure closes the browser
it started, abandons the current case, and starts a new browser for the next
case when possible. The runner performs no hidden retries and no adaptive
stopping.
case when possible. The run fixes its attempt count before execution and uses
each allocated slot at most once.
A run terminates in one of four states:
@ -589,8 +847,9 @@ A run terminates in one of four states:
failure stopped the case.
- **interrupted**: `SIGINT` or `SIGTERM` stopped the run.
The runner persists partial records before it exits. Exit status is
authoritative; a saved record alone does not prove success.
The runner saves partial records before it exits. Use exit status to determine
whether the command completed successfully; saved records also describe work
that ended in failure.
### Summaries
@ -606,10 +865,10 @@ A summary reports, for each case and metric:
categories;
- the definitions and units of the metric.
Warmups and wholly failed attempts supply no observations. A completed attempt
can supply a valid value for one metric and an invalid value for another. The
report suppresses pooled estimates when valid observations use different metric
definitions.
Statistics use valid values from completed measured attempts. The record also
retains warmups and failures. A completed attempt can supply a valid value for
one metric and an invalid value for another. A pooled estimate combines values
across attempts and requires matching metric definitions.
The default analysis settings are:
@ -627,16 +886,16 @@ can override them at analysis time. An estimate that cannot be computed keeps a
reason and, when applicable, the minimum count it needs.
The intervals are exploratory and conditional on the observed attempts.
Attempts within one run share evolving caches, so they are not independent
samples from a stable process. Repeated whole runs are the way to confirm a
finding.
Attempts within one run share evolving caches and can have dependent durations.
Repeated whole runs provide evidence about reproducibility across those cache
conditions.
### Comparisons
A comparison pairs saved cases by identical case id unless `--cases A,B`
selects one explicit baseline, candidate pair. The CLI checks the selector
syntax before it loads either record, then resolves each selector by saved case
ID or exact name.
id or exact name.
By default, a comparison requires equal:
@ -649,8 +908,8 @@ By default, a comparison requires equal:
environment;
- warmup count, timeout policy, and metric definitions.
Git revision, dirty state, run ids, timestamps, and each run's saved analysis
defaults are provenance. They do not block pairing.
Git revision, dirty state, run ids, timestamps, and saved analysis defaults
provide provenance for interpreting the paired cases.
The comparison reports absolute and percentage median changes, with a bootstrap
interval for each change. A zero baseline suppresses a percentage change. The
@ -659,8 +918,8 @@ all-differences permission allows raw values.
### Comparison permissions
The comparison records compatibility differences and never hides them. A
permission acknowledges a class of differences and labels each acknowledged
The comparison records every compatibility difference. A permission acknowledges
a class of differences and labels each acknowledged
mismatch with the rule that allowed it.
- `--allow-build-differences` acknowledges recorded build settings, effective
@ -672,13 +931,14 @@ mismatch with the rule that allowed it.
strict. Missing or varying fingerprint evidence still needs the
all-differences permission.
- `--allow-all-differences` acknowledges every compatibility difference,
including mixed metric definitions. The report compares valid raw values
without converting units. Malformed records and unusable measurements remain
unusable.
including mixed metric definitions. The report compares valid raw values in
their recorded units. Record validation and measurement validity still govern
the analysis.
Build and scene permissions combine by union. All-differences includes both.
A report shows the selected permissions, every recorded difference, and the
rule that acknowledged it. Timing values never determine the exit status.
rule that acknowledged it. Exit status reports execution and compatibility
checks; the estimates describe the observed timing differences.
## A/B feature experiments
@ -686,7 +946,8 @@ An A/B experiment answers one question: how do the selected cases behave when a
renderer feature is off versus on, with everything else held constant?
The `ab` command takes the requested features, plans the configurations, builds
every configuration, runs every configuration, and compares every "edge".
every configuration, runs every configuration, and compares every edge: a pair
of configurations that differ by one feature.
### Planning
@ -703,9 +964,9 @@ A feature configuration is a set of enabled Cargo features. The planner:
6. emits every pair of configurations that differ in exactly one feature, from
the configuration with that feature off to the configuration with it on.
The command requires at least one requested feature. For `N` features, the
comparison has `2^N` configurations and `N * 2^(N-1)` edges. The planner
compares each edge once, not once per execution order. For features
The command requires at least one requested feature. For $N$ features, the
comparison has $2^N$ configurations and $N 2^{N-1}$ edges. The planner
compares each edge once, in the direction from off to on. For features
`[a, b]`, the configurations are `00, 01, 11, 10` and the edges are:
```text
@ -745,15 +1006,15 @@ pnpm run render-bench -- ab \
The feature names must exist in the renderer manifest. Placeholders appear here
because the manifest decides which features are available.
A configuration that records attempts but fails does not stop the comparison.
Its failures and remaining slots are preserved. A build failure, a catalogue
After a configuration records attempts, scheduling continues with its recorded
failures and remaining slots. A build failure, a catalogue
identity failure, or a run with zero attempts stops scheduling later
configurations; those configurations are recorded as unattempted with a reason.
`SIGINT` and `SIGTERM` save the available state and return 130 or 143.
### Feature-comparison output
The default home is
The default output directory is
`target/performance/render-wasm/feature-comparisons/<ab-id>/`. It contains:
```text
@ -788,24 +1049,23 @@ match that evidence, the difference remains unacknowledged.
The comparison record also carries the edge context: the source and target
configuration slugs and the expected feature facts. An edge is analysed when
both endpoint records exist, even if one run recorded failed attempts. A missing
endpoint or an analysis error remains an explicit edge outcome. An edge that
endpoint or an analysis error produces an explicit edge outcome. An edge that
retains an unacknowledged difference keeps its saved report, is marked failed,
and makes the invocation exit 1.
Other build differences remain subject to `--allow-build-differences`. Scene
differences remain subject to `--allow-scene-differences`. Operation, view,
Other build differences require `--allow-build-differences`. Scene differences
require `--allow-scene-differences`. Operation, view,
context, preparation, completion, batch size, graphics, runtime environment, and
metric-definition differences stay strict unless `--allow-all-differences` is
selected. If the two configurations report the same effective feature set, the
edge remains in the comparison and is labelled with that fact. The suite does
not
infer a performance effect from the requested toggle alone.
selected. If two configurations report the same effective feature set, the
comparison includes their edge and labels it with that fact. Effective feature
sets describe the code settings that accompany the observed timings.
The command returns 0 when every configuration completes, every edge analysis
finishes, and every difference is acknowledged. It returns 1 for execution,
persistence, or analysis failures, including unacknowledged differences. Edge
reports show raw times and the usual statistics. They describe the observed
difference; they do not prove that the feature caused it.
reports show raw times and statistics for the observed differences. Establishing
a causal effect requires experimental control and repeated measurements.
## How to run
@ -859,12 +1119,12 @@ Require hardware rendering for rectangle load, pan, and zoom in a GPU container:
pnpm run render-bench -- run --filter rects --require-gpu
```
For a functional smoke check, add `--warmups 0 --repetitions 1`. A successful
For a functional check, add `--warmups 0 --repetitions 1`. A successful
run renders the scenes through `Full` and saves the vendor and renderer strings
with `:software false` in each preparation's `:effective-graphics`.
An explicit `--output-dir` must not exist; the runner refuses to overwrite it.
The same rule applies to an A/B feature-comparison home.
Choose a new path for `--output-dir`. The runner rejects an existing output
directory. A/B feature comparisons apply the same rule.
Compare two saved runs offline:
@ -939,21 +1199,22 @@ Each CLI command produces an outcome with an exit code, errors, log path, and
optional report text. The runner prints the final output once and returns the
numeric status.
Progress and diagnostics go to stderr. The default shows build phases, case
Progress and diagnostics go to stderr. The default shows build stages, case
headings, and attempt progress. `-q` / `--quiet` shows results, warnings, and
errors. `-v` / `--verbose` also shows debug details and raw build output. Quiet
and verbose are mutually exclusive. These options apply to every command.
`--color` forces terminal colours; `--no-color` disables them. Automatic colour
uses a terminal stream when `NO_COLOR` is empty or absent and `TERM` is not
`dumb`. Results and file logs receive no logger colours. Child output is read as
lines with normalized line endings. Verbose terminal output prefixes and dims
these lines; file logs store them without those decorations. Terminal errors
include a bounded log tail when available.
`dumb`. Colour choices apply to terminal diagnostics. Result text and file logs
use plain text. The runner reads child output as lines and normalizes their
line endings. Verbose terminal output prefixes and dims these lines; file logs
store the lines as text. Terminal errors include a limited extract from the end
of the log when available.
Files receive every log level, including debug output hidden from the console.
Each build writes `build.log`; each A/B comparison writes
`feature-comparison.log`. The default run home is
`feature-comparison.log`. The default run directory is
`target/performance/render-wasm/runs/<run-id>/`, which contains `run.log`,
`run.transit`, and optional screenshots.
@ -963,9 +1224,9 @@ validated merged record for `merge`, every configuration and edge for `ab`, and
a clean purge for `purge`. Exit 1 means a failed preparation, a failed attempt,
an incomplete run, an invalid selection, incompatible merge inputs, an
unacknowledged comparison difference, or a harness failure. Exits 130 and 143
mean interruption by `SIGINT` and `SIGTERM` for `build`, `run` and `ab`. An
interrupted command prints no error for the aborted work; it prints a failed
write of its own record or log.
mean interruption by `SIGINT` and `SIGTERM` for `build`, `run` and `ab`.
Interruption uses those statuses to report aborted work. Errors writing the
command's record or log also produce diagnostics.
`--warmups` defaults to 3, `--repetitions` to 10, `--timeout-ms` to 30000, and
`--startup-timeout-ms` to 30000. `build`, a run that prepares its build, and an
@ -1010,14 +1271,15 @@ pass and 1 when they fail.
## Scope and limitations
The suite measures renderer phases inside Chromium exclusively, which means that
the numbers we generate end at the renderer submission boundary. In particular,
we **do not** measure memory, power, frame rate, frame drops, visual
correctness, network, or editor input-to-paint.
The suite records durations for setup, render calls, and camera movement inside
Chromium.
Rendering durations end when the renderer returns after submitting work.
Memory, power, frame rate, frame drops, visual correctness, network latency,
and editor input-to-paint each require measurements for those quantities.
In addition, a fresh context starts a new page, module, and WebGL context. The
browser process persists across cases, and the module script import runs before
the module clock, so the measurement is not cold.
A fresh case starts a new page, module, and WebGL2 graphics context. The browser
process persists across successful cases, and module script import precedes
module timing. These startup conditions define measurements for fresh cases.
Attempts share evolving caches, so bootstrap intervals describe the observations
within a run. Repeat whole runs to assess reproducibility, and judge whether the
@ -1030,7 +1292,7 @@ frontend/performance/benches/render_wasm/
├── browser/ page entry and browser bridge
├── node/ CLI, build preparation, server, browser host, run, artefacts
├── report/ summaries, comparisons, statistics, formatting
├── runtime/ render drain and camera session protocol
├── runtime/ renderer commands, render drains, and camera sessions
├── scenes/ scene recipes and their cases
├── builder.cljc scene construction
├── declarations.cljc scene and case registry, schemas, seed derivation

2
frontend/performance/benches/render_wasm/node/artifacts.cljs
View File

@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns benches.render-wasm.node.artifacts
"Managed renderer builds and output homes for benchmark runs and comparisons.

78
frontend/performance/benches/render_wasm/node/build.cljs
View File

@ -5,13 +5,17 @@
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns benches.render-wasm.node.build
"Prepares retained release artifacts under a single preparation deadline.
"These functions build the renderer and package it with the code that runs
benchmarks in Node and the browser.
The caller owns exclusive preparation for this checkout. The renderer's
lock protects Cargo and copying; CLJS compilation follows after it releases.
Preparation uses installed dependencies. Cargo may fetch missing crates and
the custom Skia binaries on a cold build. Every attempt retains its manifest
and compiler log, including failed attempts."
Each build gets its own directory under
frontend/target/performance/render-wasm/builds/. It contains the renderer
compiled to WebAssembly, a list of benchmarks that Node can load, and the
page and JavaScript that run them in a browser.
Run `prepare` from Node in a checkout with the project's build tools and
dependencies installed. Stop other builds and any processes that rebuild the
renderer on file changes until it finishes."
(:require
["fs" :as fs]
["path" :as path]
@ -27,7 +31,16 @@
[clojure.string :as str]))
(defn ^:async git-provenance
"Captures revision and dirty state through the same bounded child boundary."
"Read the Git commit ID and whether the checkout containing `root` has local
changes, including staged changes and untracked files.
`bounds` supplies :deadline, an absolute time in milliseconds since the Unix
epoch, and an optional :signal AbortSignal to cancel the work.
Return a promise of a map containing :sha, the commit ID, and :dirty, which
is true when the checkout has local changes. If Git information is
unavailable, log a warning and return nil for both values. A timeout or
cancellation rejects the promise."
[root bounds]
(try
(let [options (merge bounds {:command "git" :cwd root})
@ -42,6 +55,9 @@
{:sha nil :dirty nil})))))
(defn- check-profile-marks
"Require :cljs-profile-marks to be false in the map describing the compiled
code. This flag indicates whether extra calls record timings for profiling;
those calls can affect benchmark measurements. Return the map or throw."
[compiled]
(when-not (false? (:cljs-profile-marks compiled))
(throw (ex-info "compiled benchmark enables profiling marks or omits its setting"
@ -49,8 +65,17 @@
compiled)
(defn- ^:async validate-browser
"Imports the release bridge in Node without preparing a renderer.
The bridge's identity method returns the compiled identity over Transit."
"Checks that the JavaScript compiled for the browser belongs to the requested
build and has profiling disabled.
build: supplies the expected :id and :browser-dir, the directory containing
browser.js.
root: the frontend directory.
bounds: supplies :deadline, an absolute time in milliseconds since the Unix
epoch, and an optional :signal AbortSignal for cancellation.
Returns a promise of the code's build information, including :build-id and
:cljs-profile-marks. Rejects the promise if loading or checking the code fails."
[build root bounds]
(let [module-url (.-href (url/pathToFileURL (path/join (:browser-dir build) "browser.js")))
script "const {default: bridge} = await import(process.env.RENDER_BENCH_MODULE_URL); process.stdout.write(bridge.buildIdentity());"
@ -61,14 +86,35 @@
(check-profile-marks (identity/check (:id build) (transit/decode-str (:stdout result))))))
(defn ^:async prepare
"Returns a ready build reference after compilation and identity validation.
"Build the renderer and the programs that run benchmarks, and save the files
for later benchmark runs.
Options: frontend :root, :selection or :features, :build-timeout-ms
(300000), AbortSignal :signal, and optional :on-created callback. The callback
receives the new build reference once the output directory, log and initial
manifest exist. Storage errors have no recorded ID. Later errors carry
:build-id and :phase. An error also carries :manifest-error and is fatal when
the failed manifest cannot be saved. A retry always creates a new build."
Pass an options map with :root set to the frontend directory. If omitted,
the function locates that directory from the Node program's location.
Cargo features choose which optional parts of the renderer to compile. Use
:selection {:default-features? true :features []} to enable Cargo's defaults.
Put additional feature names from render-wasm/Cargo.toml in the vector, and
set :default-features? to false to disable the declared defaults. The
:features option accepts the same map when :selection is absent. Omit both
options to use Cargo's defaults.
:build-timeout-ms sets the time allowed for building, in milliseconds
(300000 by default). An optional `:signal AbortSignal` cancels the work.
An optional :on-created function receives a map with the build's ID and
paths after its directory, log and initial metadata file exist. This callback
runs before compilation. Await the promise from `prepare` before using the
compiled files.
The promise resolves to a map with :id, the build's unique identifier;
:dir, its output directory; :log-path, its compiler log; and :manifest,
metadata describing the build and its settings. :cases-path points to the
JavaScript module that lists the benchmarks for Node. :browser-dir and
:renderer-dir contain the compiled programs, and :index-path points to the
HTML page.
A failure rejects the promise and keeps any files already written for
inspection. Error data includes :build-id and :log-path when available."
[{:keys [root features selection build-timeout-ms signal on-created]
:or {build-timeout-ms options/default-build-timeout-ms}}]
(when-not (and (js/Number.isSafeInteger build-timeout-ms) (pos? build-timeout-ms))

41
frontend/performance/benches/render_wasm/node/case_catalog.cljs
View File

@ -2,13 +2,20 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns benches.render-wasm.node.case-catalog
"Node-library entry points compiled with the browser's scene declarations.
"Node entry points of a compiled renderer benchmark build.
Export collect-cases as collectCases and build-identity as buildIdentity.
Each build's cases.cjs must keep its own JavaScript scope and registry."
The `:bench-render-wasm-catalog` build in `frontend/shadow-cljs.edn` names
this namespace as its compilation entry through `:exports`, which exposes
`collect-cases` to Node as `collectCases` and `build-identity` as
`buildIdentity`. Both functions use Transit-encoded strings at the Node
boundary.
Requiring this namespace loads the benchmark scene declarations. Each build's
`cases.cjs` artifact has its own scene registry and compiled identity; load
the artifact of the build being queried."
(:require
[app.common.transit :as transit]
[benches.render-wasm.cases :as cases]
@ -16,11 +23,37 @@
[benches.render-wasm.identity :as identity]))
(defn collect-cases
"Collects the benchmark cases selected from this build.
Args:
encoded-selection: A Transit-encoded map. The required :master-seed is an
integer from 0 through 4294967295 used to derive the seed of each case.
The optional :filter is a substring; a blank value selects all cases,
otherwise only case ids that contain it are collected.
Returns:
A Transit-encoded map with :build-id, the identity compiled into this build,
and :cases, a non-empty vector of validated case descriptors.
Each descriptor has the case's :id and :scene, a :scene-description, its
:params and :scene-seed, and the execution settings used by the runner.
Throws:
When the selection cannot be decoded, :master-seed is missing or not in
0 through 4294967295, no cases are selected, or a registered scene or case
fails validation. Case collection failures carry a :type in their
exception data."
[encoded-selection]
(codec/encode-str
{:build-id identity/build-id
:cases (cases/collect-cases (transit/decode-str encoded-selection))}))
(defn build-identity
"Returns the identity compiled into this build.
Returns:
A Transit-encoded string. Decoding it yields a map with :build-id, the
identifier compiled into this artifact, defaulting to `unprepared`, and
:cljs-profile-marks, a boolean that records whether renderer profiling
marks were enabled at compile time."
[]
(codec/encode-str (identity/data)))

2
frontend/performance/benches/render_wasm/node/catalog.cljs
View File

@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns benches.render-wasm.node.catalog
"Loads case descriptors from a selected build, without the runner's registry."

2
frontend/performance/benches/render_wasm/node/cli.cljs
View File

@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns benches.render-wasm.node.cli
"Command-specific argument resolution and help for renderer benchmarks."

2
frontend/performance/benches/render_wasm/node/cli/ab.cljs
View File

@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns benches.render-wasm.node.cli.ab
"Options for comparing configurations of Cargo features."

2
frontend/performance/benches/render_wasm/node/cli/build.cljs
View File

@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns benches.render-wasm.node.cli.build
"Options for preparing a benchmark build."

2
frontend/performance/benches/render_wasm/node/cli/compare.cljs
View File

@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns benches.render-wasm.node.cli.compare
"Options for comparing saved benchmark records."

2
frontend/performance/benches/render_wasm/node/cli/console.cljs
View File

@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns benches.render-wasm.node.cli.console
"Console settings carried by resolved benchmark requests."

2
frontend/performance/benches/render_wasm/node/cli/help.cljs
View File

@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns benches.render-wasm.node.cli.help
"Options for benchmark CLI help."

2
frontend/performance/benches/render_wasm/node/cli/merge.cljs
View File

@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns benches.render-wasm.node.cli.merge
"Options for merging saved benchmark records."

2
frontend/performance/benches/render_wasm/node/cli/purge.cljs
View File

@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns benches.render-wasm.node.cli.purge
"Options for purging managed benchmark artifacts."

2
frontend/performance/benches/render_wasm/node/cli/registry.cljs
View File

@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns benches.render-wasm.node.cli.registry
"Ordered command definitions for parsing, request schemas and help."

2
frontend/performance/benches/render_wasm/node/cli/run.cljs
View File

@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns benches.render-wasm.node.cli.run
"Options for a benchmark run and its selected build source."

2
frontend/performance/benches/render_wasm/node/cli/shared.cljs
View File

@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns benches.render-wasm.node.cli.shared
"Value constraints and option groups shared by benchmark commands."

7
frontend/performance/benches/render_wasm/node/commands.cljs
View File

@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns benches.render-wasm.node.commands
"Deferred execution entry for build, run and feature comparison commands."
@ -13,7 +13,6 @@
[benches.render-wasm.node.build :as build]
[benches.render-wasm.node.execution :as execution]
[benches.render-wasm.node.feature-comparison :as feature-comparison]
[benches.render-wasm.node.feature-plan :as feature-plan]
[benches.render-wasm.node.outcome :as outcome]
[benches.render-wasm.node.run :as run]))
@ -42,9 +41,9 @@
(defn- ^:async ab-command
[{:keys [get-root signal services]} options]
(let [root (get-root)
prepared (await (feature-plan/prepare root options signal))]
prepared (await (feature-comparison/prepare root options signal))]
(if (:dry-run options)
(outcome/succeeded {:report (feature-plan/format-plan prepared)})
(outcome/succeeded {:report (feature-comparison/format-plan prepared)})
(let [{:keys [home exit-code] :as result}
(await (feature-comparison/execute root prepared signal {:services services}))]
(cond-> result

118
frontend/performance/benches/render_wasm/node/execution.cljs
View File

@ -2,10 +2,21 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns benches.render-wasm.node.execution
"Checked command execution, final publication and exit status."
"Validate benchmark command requests, run their handlers and print results.
Both the CLI entry and the separately compiled renderer command entry use
this lifecycle. Callers supply a complete request and the handlers available
at that entry. Handlers own the command's files, processes and cleanup; this
module supplies cancellation, prints final reports and errors, and returns
a numeric status for the entry to use as its exit code.
`fail` reports parsing, loading or decoding failures that occur before a
request reaches `execute`. Signal helpers also support work outside the
checked command lifecycle. Execution and failure reporting share console
settings; do not overlap calls that require different settings or streams."
(:require
[app.common.schema :as sm]
[benches.render-wasm.errors :as errors :refer [attempt failure?]]
@ -15,8 +26,16 @@
[benches.render-wasm.node.request :as request]))
(defn on-signals
"Installs SIGINT/SIGTERM callbacks receiving the signal name. Returns an
idempotent remover. The caller can abort its work and let output drain."
"Listen for process interrupts so the caller can stop work and finish cleanup.
Args:
callback: Function called with the string `SIGINT` or `SIGTERM` each time the
Node process receives that signal while the listeners are installed.
Returns:
An idempotent function that removes only the listeners installed by this
call. Call it when the work settles. This helper does not abort work,
wait for cleanup or choose an exit status."
[callback]
(let [interrupt (fn [] (callback "SIGINT"))
terminate (fn [] (callback "SIGTERM"))]
@ -27,9 +46,25 @@
(.removeListener js/process "SIGTERM" terminate))))
(defn ^:async with-interruption
"Awaits `(f signal)` and returns its outcome. SIGINT/SIGTERM abort the signal
and select status 130/143. Interrupted outcomes retain fatal errors and partial
evidence. Thrown or rejected failures become outcomes."
"Run work with cooperative cancellation and preserve its result on interruption.
The first SIGINT or SIGTERM aborts the supplied signal. Its reason is an
exception with `:signal` set to that signal's name. Later signals do not
replace the reason or force termination. The work must observe cancellation
and settle after its own cleanup; this wrapper imposes no timeout and cannot
preempt synchronous code. Listeners are removed after the work settles.
Args:
f: Function called once with an AbortSignal. Return an outcome map, or a
promise of one, containing `:exit-code`, an `:errors` vector and
`:log-path` (nil when there is no log). Additional fields may retain partial
results, file paths or a final `:report` string.
Returns:
A promise of the outcome. Thrown or rejected failures become outcomes with
exit code 1. After interruption, set the exit code to 130 for SIGINT or 143
for SIGTERM and retain only errors recognized by `errors/fatal?`. Other
outcome fields are preserved. This function does not print the outcome."
[f]
(let [controller (js/AbortController.)
signal (.-signal controller)
@ -92,6 +127,7 @@
(subs message 0 (min 1024 (count message))) "\n")))))
(defn- finish
"Print the optional report and errors without retrying failed report writes."
[result local]
(binding [log/*log-path* (:log-path result)]
(let [finished (if-some [report (:report result)]
@ -105,8 +141,24 @@
(:exit-code finished)))))
(defn fail
"Prints a failure and returns its numeric status. Optional resolved console
settings and local streams configure its output."
"Report a failure that occurred before checked command execution could begin.
Use this for argument parsing, module loading or request decoding failures.
Do not report a failure again after `execute` has returned its numeric status.
Args:
cause: Error or thrown value to report. Error data may supply a log path,
usage text, validation details or comparison differences for diagnosis.
console: Trusted console settings with `:level` (`:info`, `:warn`, `:debug`)
and `:color` (`:auto`, `:always`, `:never`). An empty map uses defaults.
Do not take these settings from a request that failed validation.
local: Optional `:stdout` and `:stderr` writable streams with a Node-style
`write` method. Omitted streams use the process streams.
Returns:
The number 1 after attempting to write the error to stderr and any log
named by its error data. Reporting failures get one bounded stderr fallback
attempt. The one-argument form uses default console settings and streams."
([cause]
(fail cause {} {}))
([cause console local]
@ -114,10 +166,50 @@
(finish (outcome/failed (errors/as-failure cause) {}) local)))
(defn ^:async execute
"Checks a complete received request, awaits its supported handler and returns
a numeric exit status after final publication. Handlers receive local signal,
root access and services plus resolved options. Local settings may supply
streams, a root thunk and services."
"Validate a command request, await its handler and print its final outcome.
Check the request without adding defaults or coercing values. Invalid
requests use default console settings; valid requests choose their console
settings before handler lookup and command policy checks. Unsupported
commands and policy failures are reported without invoking a handler or
looking up the frontend root.
For `run` and non-dry-run `ab` commands, `:require-gpu` requires the environment
variable `PENPOT_GPU` to equal the string `1` before invoking the handler.
This checks an environment prerequisite, not actual GPU availability.
Args:
received: Complete request map with a string `:command`, resolved
`:console` settings and command-specific `:options`. Obtain it from
`cli/parse-resolve` or decode it from the CLI's Transit request. Missing
required fields and unknown fields fail validation.
handlers: Map of supported command-name strings to functions called as
`(handler context options)`. Return an outcome or a promise of one, with
`:exit-code`, an `:errors` vector and `:log-path` (nil when there is no log).
An optional `:report` string supplies all final stdout text. Other
result fields do not generate output. Handlers own progress logging
and must await their work and cleanup before returning.
local: Optional map containing `:stdout` and `:stderr` writable streams,
a zero-argument `:get-root` function returning the frontend directory,
and a `:services` map of handler-specific overrides. The context contains
an AbortSignal at `:signal`, plus `:get-root` and `:services` when supplied.
Root lookup and service calls remain the handler's responsibility. These
functions, streams and services are local, not part of the request.
Returns:
A promise of a numeric exit status after the handler settles and final
output writes have been attempted. Print `:report` once, when present,
followed by errors on stderr, using the outcome's log path for file output.
No report means no final stdout text. Output writes do not wait for streams
to drain. The two-argument form uses process streams and supplies no root
function or services.
Validation, policy and handler failures produce status 1. Interruption
follows `with-interruption`: await cleanup, preserve partial result fields
and fatal errors, and return 130 for SIGINT or 143 for SIGTERM. A report or
error-reporting failure changes other statuses to 1, with one bounded
stderr fallback if error reporting fails. Interrupted statuses take
precedence. Failed report writes are not retried."
([received handlers]
(execute received handlers {}))
([received handlers local]

245
frontend/performance/benches/render_wasm/node/feature_comparison.cljs
View File

@ -5,25 +5,40 @@
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns benches.render-wasm.node.feature-comparison
"Serial feature A/B orchestration and persistence of comparisons."
"Plan and run benchmarks for combinations of optional Cargo renderer features.
`prepare` reads Cargo metadata and collects the selected benchmark cases.
Call `format-plan` to inspect that data, or await `execute` to build and run
each configuration in sequence. Each configuration disables Cargo defaults.
An edge is a comparison between two configurations that differ by one
requested feature, from disabled to enabled.
Execution saves progress, run records, logs and edge reports for inspection
after completion, failure or cancellation. Builds use the checkout's renderer
and benchmark sources. Keep other renderer builds and watchers stopped until
execution finishes."
(:require
["fs" :as fs]
["node:timers/promises" :as timers]
["path" :as path]
[benches.render-wasm.errors :as errors :refer [attempt failure?]]
[benches.render-wasm.node.artifacts :as artifacts]
[benches.render-wasm.node.build :as build]
[benches.render-wasm.node.catalog :as catalog]
[benches.render-wasm.node.files :as files]
[benches.render-wasm.node.log :as log]
[benches.render-wasm.node.outcome :as outcome]
[benches.render-wasm.node.process :as process]
[benches.render-wasm.node.run :as run]
[benches.render-wasm.node.runs :as runs]
[benches.render-wasm.node.time :as time]
[benches.render-wasm.report.compare :as compare]
[benches.render-wasm.result :as result]
[clojure.set :as set]))
["fs" :as fs]
["node:timers/promises" :as timers]
["path" :as path]
[benches.render-wasm.cases :as cases]
[benches.render-wasm.errors :as errors :refer [attempt failure?]]
[benches.render-wasm.feature-plan :as feature-plan]
[benches.render-wasm.node.artifacts :as artifacts]
[benches.render-wasm.node.build :as build]
[benches.render-wasm.node.catalog :as catalog]
[benches.render-wasm.node.files :as files]
[benches.render-wasm.node.log :as log]
[benches.render-wasm.node.options :as options]
[benches.render-wasm.node.outcome :as outcome]
[benches.render-wasm.node.process :as process]
[benches.render-wasm.node.run :as run]
[benches.render-wasm.node.runs :as runs]
[benches.render-wasm.node.time :as time]
[benches.render-wasm.report.compare :as compare]
[benches.render-wasm.result :as result]
[clojure.set :as set]
[clojure.string :as str]))
(def ^:private warnings
["Timing values do not determine success."
@ -36,11 +51,26 @@
:execute-run run/execute})
(defn create-home
"Creates the feature-comparison home directory
"Creates a directory and an empty log for a new feature comparison.
The default lives under the managed feature-comparisons directory.
An explicit directory resolves from the caller's cwd and must not exist.
Paths inside the managed builds tree are rejected."
Args:
root: The frontend directory.
comparison-id: Contains 1 to 128 ASCII letters, digits, underscores or
hyphens, starting with a letter or digit.
output-dir: With nil `output-dir`, uses
`target/performance/render-wasm/feature-comparisons/<comparison-id>` under
root. An explicit path resolves from the Node process's working directory.
The chosen directory must not exist and must not be inside the managed
builds tree, including through symlink ancestors.
Returns:
A map containing `:comparison-id`, `:dir`, `:log-path` and `:record-path`.
The log is `feature-comparison.log`; `:record-path` names the future
`feature-comparison.transit` file, which this function does not write.
Throws:
Invalid identifiers, rejected paths and filesystem failures throw an error;
path and filesystem errors include `:path` when available."
[root comparison-id output-dir]
(artifacts/check-home-id comparison-id "invalid feature comparison id")
(let [default-dir (path/join (artifacts/managed-root root)
@ -53,7 +83,21 @@
:comparison-id comparison-id)))
(defn write-record
"Validates schema:feature-comparison and atomically replaces the saved file."
"Validates and saves a feature-comparison record.
Encodes `record` as Transit after schema validation, then atomically replaces
that file so readers see either the previous complete record or the new one.
Args:
home: Supplies `:record-path` in an existing directory.
record: The feature-comparison record.
Returns:
The record.
Throws:
Validation, write and rename failures throw without replacing the previous
record."
[home record]
(files/atomic-write (:record-path home) (result/encode record))
record)
@ -95,6 +139,7 @@
(some #(when (= slug (:slug %)) %) (:configurations record)))
(defn- persist
"Save updated progress before exposing it as the current and last saved record."
[state update]
(let [next (assoc (update @(:record-ref state)) :updated-at (time/now))
saved (try
@ -503,9 +548,13 @@
:unacknowledged (unacknowledged-count comparison)})))
(defn- record-edge-outcome
"Records the result of analyzing an edge and returns the first error seen so
far. Throws a fatal failure after the edge and the remaining edges are
recorded."
"Records the result of analyzing an edge.
Returns:
The first error seen so far.
Throws:
A fatal failure after the edge and the remaining edges are recorded."
[state edge result error]
(cond
(and (failure? result) (errors/fatal? result))
@ -614,14 +663,49 @@
result)))))
(defn ^:async execute
"Executes a prepared feature comparison and owns builds, runs and reports.
"Build and benchmark every prepared feature configuration, then compare runs.
- prepared: node.feature-plan/prepare output.
- settings: accepts an optional :services map with :prepare-build, :collect-catalog,
:create-run-home and :execute-run overrides.
The caller must handle `:dry-run` by calling `format-plan` instead; this function
always executes.
Returns the comparison record, home, log path, exit code and the failures
in :errors. The saved record reflects the last successful atomic write."
Create a fresh comparison directory and save its initial record before the
first build. Build and run configurations serially, checking each compiled
catalog against the prepared cases. Runs live in `configs/<slug>` under the
comparison directory; edge reports live in `comparisons/<from>__<to>.transit`.
Build artifacts remain in the managed builds directory.
Timing differences alone do not cause failure. Unacknowledged differences
in build, scene or other comparison metadata do cause an edge to fail.
The prepared options `:allow-build-differences` and `:allow-scene-differences`
acknowledge their respective categories; `:allow-all-differences` acknowledges
every compatibility mismatch. Reports retain acknowledged differences.
Build or catalog failures stop later configurations. A failed run may allow
later configurations when it retains attempts and usable evidence; missing
evidence, GPU failure or zero attempts stop them. Cancellation skips remaining
work. Files already written remain for diagnosis.
Args:
root: The frontend directory.
prepared: The result of `prepare`.
signal: An optional AbortSignal that cancels execution.
settings: May contain `:services` overrides with the same contracts as the
defaults: `:prepare-build` uses `node.build/prepare`, `:collect-catalog`
uses `node.catalog/collect`, `:create-run-home` uses `node.runs/create-home`,
and `:execute-run` uses `node.run/execute`. Omitted services use
`default-services`.
Returns:
Await the returned promise for a map with `:comparison`, the last
successfully saved progress record; `:home`, the comparison ID and paths;
`:log-path`; `:errors`, retained error objects; and `:exit-code`. The exit
code is 0 only when all configurations and edges complete, 1 on failure,
or 130/143 on interruption.
If storage fails, the returned `:comparison` reflects the last successful
atomic write, or nil if no record was saved; `:home` is nil if directory
creation failed. Inspect the saved statuses and `:errors` together, since
`:errors` is not an exhaustive list of failed configurations or edges."
[root prepared signal settings]
(let [services (merge default-services (:services settings))
comparison-id (str (random-uuid))
@ -631,3 +715,100 @@
(await
(execute-comparison
(create-comparison-state root prepared signal services home comparison-id))))))
(defn- get-root-features
"Read feature definitions for cwd/Cargo.toml from Cargo metadata JSON."
[cwd text]
(let [metadata (js->clj (js/JSON.parse text))
manifest (fs/realpathSync (path/join cwd "Cargo.toml"))
package (first (filter #(= manifest (fs/realpathSync
(path/resolve cwd (get % "manifest_path"))))
(get metadata "packages")))]
(when-not package
(throw (ex-info "Cargo metadata does not contain the renderer manifest"
{:type ::missing-package :manifest-path manifest})))
(get package "features")))
(defn ^:async prepare
"Collect and validate a feature comparison plan without building or running it.
Run `cargo metadata --offline --locked --no-deps` in `render-wasm` with the
inherited environment. Cargo and its offline metadata must be available.
`:build-timeout-ms` bounds that subprocess in milliseconds, not the entire
planning operation.
Require empty declared Cargo defaults and known requested feature names;
Cargo remains responsible for resolving feature activation during builds.
Planning creates no comparison output and starts no build, server or browser,
including for a dry-run.
Args:
root: The frontend directory, with render-wasm as its sibling.
opts: Resolved CLI options, including defaults. `:features` supplies the
requested Cargo feature names, `:seed` supplies the master scene seed,
and `:filter` selects case IDs by substring. `:max-features` limits the
requested feature count. Execution requires `:confirm` for more than
eight configurations or a blank filter; `:dry-run` bypasses that
confirmation requirement only.
An optional `:output-dir` resolves from the Node process's working
directory and must not exist. Check it without creating or reserving it.
`execute` checks it again and also rejects paths inside the managed
builds tree.
signal: An optional AbortSignal that cancels the process.
Returns:
Await the promise for a map with `:plan`, `:cases`, `:output-dir` and
`:options`. The `:plan` map contains `:features` in sorted order,
`:configurations` with a `:slug` and Cargo `:selection`, and `:edges` with
`:from` and `:to` slugs and a `:feature`. Slugs are bit strings in sorted
feature order: `0` means disabled, `1` enabled. The `:cases` vector holds
case descriptions collected once. `:output-dir` is the absolute explicit
path or nil. `:options` contains the supplied options with normalized
features and output path. Pass this map to `format-plan` or `execute`.
Throws:
Invalid options or paths, Cargo failures or cancellation, unsupported
manifest features, and case collection failures reject the promise.
An empty case selection is an error."
[root opts signal]
(let [plan (feature-plan/plan (:features opts) opts)
output-dir (artifacts/check-output-dir (:output-dir opts))
cwd (path/resolve root ".." "render-wasm")
metadata (await (process/run
{:command "cargo"
:args ["metadata" "--offline" "--locked" "--no-deps" "--format-version" "1"]
:cwd cwd
:deadline (process/deadline (:build-timeout-ms opts options/default-build-timeout-ms))
:signal signal}))]
(feature-plan/check-manifest (:features plan) (get-root-features cwd (:stdout metadata)))
{:plan plan
:cases (cases/collect-cases {:master-seed (:seed opts) :filter (:filter opts)})
:output-dir output-dir
:options (assoc opts :features (:features plan) :output-dir output-dir)}))
(defn format-plan
"Return a text summary of the map produced by `prepare`.
This function neither collects cases again nor reads the filesystem,
starts processes or executes benchmarks.
Returns:
Feature names, configurations, comparison edges, case count, sampling
counts and configured timeouts."
[{:keys [plan cases options]}]
(str "Feature comparison plan\n"
"Features: " (str/join ", " (:features plan)) "\n"
"Configurations (Cargo defaults disabled):\n"
(str/join "\n" (map #(str " " (:slug %) ": "
(str/join ", " (get-in % [:selection :features])))
(:configurations plan))) "\n"
"Edges:\n"
(str/join "\n" (map #(str " " (:from %) " -> " (:to %) ": " (:feature %))
(:edges plan))) "\n"
"Cases: " (count cases) "\n"
"Attempts per case: " (:warmups options) " warmups, " (:repetitions options) " measured\n"
"Timeouts (ms): attempt " (:timeout-ms options)
", build " (:build-timeout-ms options) ", startup " (:startup-timeout-ms options) "\n"))

66
frontend/performance/benches/render_wasm/node/feature_plan.cljs
View File

@ -1,66 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns benches.render-wasm.node.feature-plan
"Plan comparisons based on cargo metadata and cases collected."
(:require
["fs" :as fs]
["path" :as path]
[benches.render-wasm.cases :as cases]
[benches.render-wasm.feature-plan :as feature-plan]
[benches.render-wasm.node.artifacts :as artifacts]
[benches.render-wasm.node.options :as options]
[benches.render-wasm.node.process :as process]
[clojure.string :as str]))
(defn- get-root-features
[cwd text]
(let [metadata (js->clj (js/JSON.parse text))
manifest (fs/realpathSync (path/join cwd "Cargo.toml"))
package (first (filter #(= manifest (fs/realpathSync
(path/resolve cwd (get % "manifest_path"))))
(get metadata "packages")))]
(when-not package
(throw (ex-info "Cargo metadata does not contain the renderer manifest"
{:type ::missing-package :manifest-path manifest})))
(get package "features")))
(defn ^:async prepare
"Accepts the frontend root and resolved CLI options. Reads Cargo metadata in
render-wasm with the inherited environment, checks an optional fresh output
path, and collects selected case data once."
[root opts signal]
(let [plan (feature-plan/plan (:features opts) opts)
output-dir (artifacts/check-output-dir (:output-dir opts))
cwd (path/resolve root ".." "render-wasm")
metadata (await (process/run
{:command "cargo"
:args ["metadata" "--offline" "--locked" "--no-deps" "--format-version" "1"]
:cwd cwd
:deadline (process/deadline (:build-timeout-ms opts options/default-build-timeout-ms))
:signal signal}))]
(feature-plan/check-manifest (:features plan) (get-root-features cwd (:stdout metadata)))
{:plan plan
:cases (cases/collect-cases {:master-seed (:seed opts) :filter (:filter opts)})
:output-dir output-dir
:options (assoc opts :features (:features plan) :output-dir output-dir)}))
(defn format-plan
"Formats prepared data without collecting again or reading the filesystem."
[{:keys [plan cases options]}]
(str "Feature comparison plan\n"
"Features: " (str/join ", " (:features plan)) "\n"
"Configurations (Cargo defaults disabled):\n"
(str/join "\n" (map #(str " " (:slug %) ": "
(str/join ", " (get-in % [:selection :features])))
(:configurations plan))) "\n"
"Edges:\n"
(str/join "\n" (map #(str " " (:from %) " -> " (:to %) ": " (:feature %))
(:edges plan))) "\n"
"Cases: " (count cases) "\n"
"Attempts per case: " (:warmups options) " warmups, " (:repetitions options) " measured\n"
"Timeouts (ms): attempt " (:timeout-ms options)
", build " (:build-timeout-ms options) ", startup " (:startup-timeout-ms options) "\n"))

2
frontend/performance/benches/render_wasm/node/request.cljs
View File

@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns benches.render-wasm.node.request
"Resolved benchmark requests, sender defaults and receiver validation."

2
frontend/performance/benches/render_wasm/node/runner.cljs
View File

@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns benches.render-wasm.node.runner
"Command-line entry for renderer benchmarks and saved results."

2
frontend/performance/benches/render_wasm/node/server.cljs
View File

@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns benches.render-wasm.node.server
"An owned loopback HTTP server for one immutable renderer build."

2
frontend/test/performance_tests/render_wasm/artifacts_test.cljs
View File

@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns performance-tests.render-wasm.artifacts-test
(:require

65
frontend/test/performance_tests/render_wasm/feature_comparison_test.cljs
View File

@ -6,26 +6,25 @@
(ns performance-tests.render-wasm.feature-comparison-test
(:require
["fs" :as fs]
["node:timers/promises" :as timers]
["os" :as os]
["path" :as path]
[benches.render-wasm.cases :as cases]
[benches.render-wasm.declarations :as decl]
[benches.render-wasm.errors :as errors]
[benches.render-wasm.feature-plan :as feature-plan]
[benches.render-wasm.node.build :as build]
[benches.render-wasm.node.execution :as execution]
[benches.render-wasm.node.feature-comparison :as feature-comparison]
[benches.render-wasm.node.feature-plan :as node-plan]
[benches.render-wasm.node.files :as files]
[benches.render-wasm.node.log :as log]
[benches.render-wasm.node.process :as process]
[benches.render-wasm.node.request :as request]
[benches.render-wasm.result :as result]
[cljs.test :as t :include-macros true]
[clojure.string :as str]
[frontend-tests.helpers.mock :as mock]))
["fs" :as fs]
["node:timers/promises" :as timers]
["os" :as os]
["path" :as path]
[benches.render-wasm.cases :as cases]
[benches.render-wasm.declarations :as decl]
[benches.render-wasm.errors :as errors]
[benches.render-wasm.feature-plan :as feature-plan]
[benches.render-wasm.node.build :as build]
[benches.render-wasm.node.execution :as execution]
[benches.render-wasm.node.feature-comparison :as feature-comparison]
[benches.render-wasm.node.files :as files]
[benches.render-wasm.node.log :as log]
[benches.render-wasm.node.process :as process]
[benches.render-wasm.node.request :as request]
[benches.render-wasm.result :as result]
[cljs.test :as t :include-macros true]
[clojure.string :as str]
[frontend-tests.helpers.mock :as mock]))
(def ^:private temporary-root (atom nil))
@ -153,7 +152,7 @@
(let [output (path/join @temporary-root "new" "comparison")
opts (assoc (planning-options) :output-dir output)
before (vec (fs/readdirSync (renderer-root)))
ready (await (node-plan/prepare (frontend-root) opts nil))]
ready (await (feature-comparison/prepare (frontend-root) opts nil))]
(t/is (= ["a" "b"] (get-in ready [:plan :features])))
(t/is (= (cases/collect-cases {:master-seed 42 :filter "rects/load"}) (:cases ready)))
(t/is (= (assoc opts :features ["a" "b"])
@ -162,7 +161,7 @@
(t/is (not (fs/existsSync (path/join @temporary-root "new"))))
(t/is (= before (vec (fs/readdirSync (renderer-root)))))
(t/is (empty? (fs/readdirSync (frontend-root))))
(let [text (node-plan/format-plan ready)]
(let [text (feature-comparison/format-plan ready)]
(doseq [part ["a, b" "00" "01" "11" "10" "00 -> 10: a" "Cases: 1"
"0 warmups, 1 measured" "attempt 2000, build 10000, startup 3000"]]
(t/is (str/includes? text part))))))
@ -179,7 +178,7 @@
(t/is (= (renderer-root) (:cwd request)))
(t/is (not (contains? request :env)))
(js/Promise.resolve (metadata-output {"default" [] "a" [] "b" ["a"]})))}
(let [ready (await (node-plan/prepare (frontend-root) (planning-options) nil))]
(let [ready (await (feature-comparison/prepare (frontend-root) (planning-options) nil))]
(t/is (= ["a" "b"] (get-in ready [:plan :features])))))))
(t/deftest ^:async collected-custom-case-data-and-order-survive-planning
@ -196,8 +195,8 @@
:preparation {:reset :outside-timing :data [1 2]}
:run! (fn [_] (throw (js/Error. "planning must not run an operation")))}))
(try
(let [ready (await (node-plan/prepare (frontend-root)
(assoc (planning-options) :filter "feature-plan-fixture/") nil))
(let [ready (await (feature-comparison/prepare (frontend-root)
(assoc (planning-options) :filter "feature-plan-fixture/") nil))
data (:cases ready)]
(t/is (= [:feature-plan-fixture/second :feature-plan-fixture/first] (mapv :id data)))
(t/is (= (cases/collect-cases {:master-seed 42 :filter "feature-plan-fixture/"}) data))
@ -218,10 +217,10 @@
(await (mock/with-mocks*
{process/run (fn [_] (js/Promise.resolve (metadata-output definitions)))}
(t/is (= message (await (error-message
(node-plan/prepare (frontend-root) (planning-options) nil))))))))
(feature-comparison/prepare (frontend-root) (planning-options) nil))))))))
(t/is (= "no benchmark cases match the selection"
(await (error-message (node-plan/prepare (frontend-root)
(assoc (planning-options) :filter "missing-case") nil))))))
(await (error-message (feature-comparison/prepare (frontend-root)
(assoc (planning-options) :filter "missing-case") nil))))))
(t/deftest ^:async invalid-input-and-existing-output-fail-before-cargo
(write-cargo-fixture)
@ -229,18 +228,18 @@
(mock/with-mocks*
{process/run (fn [_] (throw (js/Error. "unexpected Cargo invocation")))}
(t/is (= "output directory already exists"
(await (error-message (node-plan/prepare (frontend-root)
(assoc (planning-options) :output-dir (frontend-root)) nil)))))
(await (error-message (feature-comparison/prepare (frontend-root)
(assoc (planning-options) :output-dir (frontend-root)) nil)))))
(t/is (= "ab requires at least one requested feature"
(await (error-message (node-plan/prepare (frontend-root)
(assoc (planning-options) :features []) nil))))))))
(await (error-message (feature-comparison/prepare (frontend-root)
(assoc (planning-options) :features []) nil))))))))
(t/deftest ^:async cancellation-reaches-the-process-boundary
(write-cargo-fixture)
(let [controller (js/AbortController.)]
(.abort controller "SIGINT")
(t/is (str/includes? (await (error-message
(node-plan/prepare (frontend-root) (planning-options) (.-signal controller))))
(feature-comparison/prepare (frontend-root) (planning-options) (.-signal controller))))
"cancelled"))))
(defn- planned-record

2
frontend/test/performance_tests/render_wasm/request_test.cljs
View File

@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns performance-tests.render-wasm.request-test
(:require

2
frontend/test/performance_tests/render_wasm/server_test.cljs
View File

@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns performance-tests.render-wasm.server-test
(:require