The renderer benchmark suite measures the Penpot WebAssembly (WASM) renderer in Chromium. Scenes describe shapes; cases describe the work to perform on them. The suite records setup, rendering, pan and zoom durations and compares runs with different renderer builds. How to run gives the commands.

The suite has three parts:

A Node runner controls the experiment. It prepares or reuses a build, selects cases, serves the browser code, launches Chromium, schedules attempts, saves results, and prints reports. An attempt is one execution of a case.
The browser creates the renderer and graphics context, constructs the scene snapshot, uploads its shapes, performs the work, and records durations.
The reporting code computes statistics from saved records in Node or the browser without loading the renderer.

Main concepts
Architecture
Workloads
Renderer commands
Measurement
Results and reports
A/B feature experiments
How to run
Unit tests
Scope and limitations
Repository map
Further reading

Main concepts

Workload

Renderer: the Rust Skia renderer compiled to WebAssembly. It owns shape storage, tiles, caches, and the WebGL2 graphics state for one canvas.
Module: one instantiated copy of the renderer. The browser creates a module per prepared page and releases it with the page.
Graphics context: the WebGL2 context and renderer state bound to one canvas. A benchmark case can use either a fresh or a reused context.
Scene: a recipe that produces a starting set of shapes. A scene has an id, a closed parameter schema, and a builder function. This function is given a random seed and returns a snapshot. Scenes can be procedural, or captured. The former build shapes from code, parameters, and a random seed, whereas the latter render a page exported from Penpot.
Scene snapshot: the concrete shapes a run renders. This is only static data, camera movement and other interactions belong to a case.
Master seed: the initial seed for the run, supplied on the command line. It defaults to 42 and accepts a 32-bit unsigned integer.
Seed: a 32-bit integer derived from the master seed, scene id, and canonical parameters. The seed makes a scene reproducible. Load, pan, and zoom cases for the same scene and parameters share one seed.
Scene builder: the construction API used inside a scene. It provides basic shapes (rect, path, circle, frame, group, and bool) constructors, and a (hopefully) self-explanatory API.
Reference: a label that a scene attaches to a shape during construction. A case operation uses references to find shapes it needs.
Case: a scene plus its execution settings: parameters, base view, context policy, completion rule, optional operation settings, and a run function. A case is the unit that is measured.
Operation: the function that performs a case's work during one attempt. It receives a runtime context with the case data, scene snapshot, renderer module, view, and hooks. Standard operations load, pan, and zoom.
View: the camera's zoom :scale and offsets :x and :y. Optional :viewport settings give the canvas width, height, and device pixel ratio, the number of drawing-buffer pixels per CSS pixel along each dimension.
Hook: a function supplied through :hooks for clock readings, scheduling, view updates, rendering, or checking whether work may continue.
Case descriptor: What gets sent to the browser. It contains the case id, scene, derived seed, parameters, view, context policy, completion rule, batch size, preparation settings, and operation settings. Any functions are removed.
Catalogue: the compiled list of case descriptors and scene declarations for one build. Node reads the catalogue to select cases. The browser executor contains the same declarations and looks up the run function by case id.
Fingerprint: SHA-256 digest of the validated starting snapshot. A comparison checks it to establish equal starting scene content.

Execution

Build: one compiled renderer plus the matching catalogue and browser executor. A build id ties the three outputs together, and the runner rejects a mismatch.
Retained build: a ready build that the suite stores under the managed build directory for reuse in later runs.
Fresh case: a case with :context :fresh. Each attempt gets a new page, module, graphics context, and upload.
Warm case: a case with :context :reuse. One page, module, graphics context, snapshot, and upload serve all attempts of the case.
Preparation: setup for one graphics context. It covers module instantiation, WebGL2 context creation, scene build, snapshot validation, fingerprinting, upload-order derivation, and upload. A warm preparation also restores the starting view and drains to Full. One page owns one preparation.
Attempt: one execution of a case against a preparation. An attempt records one outcome, one set of metrics, and a raw trace. A failed attempt still consumes its allocated slot.
Warmup: an attempt that runs the case before measured attempts to warm caches and stabilize the page. The run records warmups but excludes them from statistics.
Repetition: a measured attempt. A case has a fixed number of repetitions chosen before the run.
Run: a set of attempts over selected cases. A run owns the browser processes it starts, fixes warmup and repetition counts, records failures in chronological order, and performs no hidden retries.
Run record: the saved evidence for one run: plan, build and environment provenance, preparations, attempts, metrics, traces, failures, remaining slots, and termination. The runner stores it as a data file validated by a schema.
Bridge: the object that runs in the browser, receives plain requests from Node, owns the page's renderer context, runs preparations and attempts, and returns plain results.
Runner: the Node command line interface that builds or reuses a build, selects cases, serves the build, launches Chromium, runs attempts, saves records, and prints reports.
Browser executor: the compiled page and JavaScript that perform preparation and case operations. Its bridge exposes the operations that Node can request.
Measurement: a numeric result for a quantity such as elapsed duration. The suite measures module start, graphics start, upload, first render, and camera movement separately.
Metric: the name and definition of a measured quantity. Saved run metrics declare a unit, semantics, basis, and zero policy. Semantics identifies what the value measures. The basis states whether a value covers an attempt or averages its inner calls. The zero policy states whether zero supplies a valid value for analysis.
Observation: one valid value from a measured attempt for statistical analysis. A batch average supplies one observation for its attempt.
Slice: a trace entry for one renderer call: timestamp, flags, frame type, and duration. Slices describe the work within an attempt.
Trace: details of the work, including individual renderer calls, cached previews, requested waits, and observed durations.
Clock reading: a time in milliseconds from the clock's origin. Subtracting two readings from that clock gives an elapsed duration.
Endpoint: the clock reading at the end of the measured work. The renderer endpoint is the reading immediately after a renderer call returns.
Render drain: repeated renderer calls until one returns Full. Partial completion schedules another call on the next animation frame.
Cached preview: one render from cached tiles after a view update.
Camera session: camera start, view updates with cached previews, a settling wait, camera end, and the final render drain.
Active interval: the part of a camera session from camera start through the last cached preview and its guard check, or an explicit pause that follows the preview.
Settling wait: the requested pause after the active interval and before ending the interaction. The observed wait includes scheduling delays.
Camera finalization: the interval from immediately before ending the interaction through the renderer call that returns Full.
Frame type: the completion signal returned by one renderer call. None means no frame; Partial means visible work remains; Full means the renderer finished the current request; ViewportReady means the visible viewport was submitted while work outside the viewport may remain.
Tile: a unit of renderer work with a fixed size. A logical tile covers 512 document units. The renderer caches rendered tiles in a texture atlas.
Atlas: a GPU texture that stores many tiles or a document-space preview. Atlases let warm pan and zoom present cached pixels before new tiles finish.

Command execution

Renderer command: a function that accepts a command context or its Promise and returns a Promise of the next context. The base function uses the declared name, such as pan. The measured function adds *, such as pan*, and records an event for its work.
Command context: the map with keys :case, :scene, :view, and :hooks. It carries the case descriptor, prepared scene snapshot, current view, and hooks.
Command body: the work declared by defcommand. A synchronous body returns the next context. An asynchronous body completes through callbacks named by its declaration.
Composite command: a command that combines calls to other commands. A declaration with :measured-variant false defines its base function and collects events from the measured commands it calls.
Command execution: one call to commands/execute, which runs a function from one command context to the next and records measured calls in completion order.
Event: the record for one measured command call. It contains the command name, vectors of raw metric values, and an optional trace.
Guard: the :check hook. A return authorizes work; an error rejects it. The renderer owner is the execution authorized to use the renderer. The guard can report cancellation, deadline expiry, context loss, or replacement of that owner.
Completion callback: complete or fail in an asynchronous command body. The first call determines the outcome: complete supplies the next context and optional measurements; fail supplies a cause.

Results

Report: the statistics and tables computed from one or more run records. Reports run in Node or the browser and do not load the renderer.
Comparison: an offline analysis of two saved run records. It pairs cases, checks compatibility, and reports changes for each metric with intervals.
A/B comparison: a comparison of two runs that differ by one renderer feature. The feature is off in one run and on in the other.
Permission: In principle, only run records with equal builds and equal scenes are comparable. Because one sometimes needs to override this strict requirement, permissions allow build environment or scene differences in a comparison.
Termination: the final state of a run: completed, failed, incomplete, or interrupted.

Feature experiments

Feature: a switch at compile time in the renderer's Cargo manifest.
Feature configuration: the set of Cargo features enabled for one renderer build.
Feature comparison: all on and off combinations of the requested features. N requested features produce 2^N configurations.
Edge: two configurations that differ in exactly one feature, compared from the configuration with the feature off to the configuration with the feature on. A comparison with N features has N 2^{N-1} edges.
Edge comparison: the comparison record for one edge, stored with the feature comparison.
Feature-comparison record: the saved plan, configurations, runs, edges, failures, and termination for one A/B experiment.

Architecture

Components

The Node runner owns everything outside the page:

it prepares or loads a build;
it reads the compiled catalogue and selects cases;
it serves the selected build from a server on 127.0.0.1;
it launches Chromium and creates pages;
it schedules warmups and measured attempts;
it records preparations, attempts, failures, and termination;
it writes the run record as work progresses;
it prints a final report.

The browser bridge owns everything inside the page:

it validates a request before assigning the page to that work;
it imports the renderer module and creates the canvas and WebGL2 context;
it builds, validates, fingerprints, orders, and uploads the scene;
it restores the starting view outside timing for warm cases;
it runs the case operation and records the requested measurements;
it returns plain data and disposes the context when asked.

The shared reporting code owns analysis:

it normalizes run records into summaries;
it computes statistics and intervals for each metric;
it pairs saved cases and checks compatibility;
it formats run summaries, comparisons, and feature-comparison edge reports;
it imports no browser, server, or renderer code.

End-to-end path

sequenceDiagram
    participant N as Node runner
    participant S as Owned server
    participant B as Browser bridge
    participant W as WASM renderer

    N->>N: select cases from the catalogue
    N->>S: serve the selected build
    N->>N: launch Chromium
    N->>B: open a page
    B->>W: instantiate the module
    B->>W: create the WebGL2 context
    B->>B: build, validate, fingerprint the snapshot
    B->>W: upload shapes in tree order
    B->>W: render until Full
    B-->>N: metrics, slices, and preparation evidence
    N->>N: persist the run record

The Node runner saves the initial plan before it starts a browser, and updates the run record after each preparation and attempt, and at termination. Reports use these saved records.

Builds and catalogues

A build has three matching outputs:

the renderer module;
the case catalogue compiled for Node;
the browser executor and its page.

Build preparation compiles the renderer in release mode with Cargo, then compiles the catalogue and browser executor with the same build id. The runner checks the id before case collection and when a page starts, and rejects a mismatch.

An ordinary measurement build uses the renderer's default Cargo feature set. A feature experiment may enable selected functional features. Both kinds of measurement build should exclude diagnostic features such as stats, profile, profile-macros, and profile-raf. The build checks that we are in release mode, the optimization level, and disabled profiling marks.

The build record captures a fixed list of environment values supplied to Cargo, including compiler, target, and Emscripten settings. These values provide provenance for interpreting a run. Reproducing a build also requires its other environment settings and dependencies.

Builds are saved under target/performance/render-wasm/builds/<id>/. The manifest records the build id, status, functional settings, Git provenance, and build facts. An experiment run can select a retained build with --build <id>, and run prepares a new build when it omits --build.

build and a run that prepares its build accept --features, --[no-]default-features, and --build-timeout-ms. A run with --build reuses the retained build and rejects those preparation options.

Run one build preparation at a time in a checkout and keep renderer watchers stopped until it finishes. The renderer build lock covers Cargo compilation and copying the renderer files. The build script releases that lock before ClojureScript compilation.

Browser host

The runner serves the selected build directory from 127.0.0.1 on an allocated port.

The runner launches headless Chromium with Playwright and owns the browser process it starts. Cases run serially. A fresh case gets a new page and context for each preparation. A warm case keeps one page and context for all of its attempts.

The runner applies deadlines to startup, preparation, and attempts. When a page stalls past its deadline, Node stops the browser process and records the failure. The browser bridge also checks for context loss, expired deadlines, and work associated with a superseded renderer owner before and after asynchronous work. A browser thread occupied by synchronous work relies on the Node deadline to stop it.

Clean-up closes the page, then the browser, then the server. The runner deletes a build selected with --remove only after the final run record is saved.

Workloads

Declaring a scene

A scene declaration names an id, a description, a closed parameter schema, and a builder function. The builder runs inside a seeded scope. Random values come from one generator owned by that scope, so the same recipe, parameters, and seed produce the same snapshot.

(defscene :rects
  {:description "Seeded rectangles with translucent fills and centered strokes"
   :params-schema schema:params}
  [params]
  (scene {:seed (:seed params)
          :root {:x 0 :y 0 :width (:width params) :height (:height params)}
          :defaults {:rect {:x (gen-int 0 (:width params))
                            :y (gen-int 0 (:height params))
                            :width (gen-int 20 100)
                            :height (gen-int 20 100)
                            :fills (gen-vector (gen-fill))}}}
    (doseq [_ (range (:count params))]
      (rect))))

Each scene has one construction scope. Starting another scene inside it raises an error. Ordinary let, doseq, and functions defined outside the scope work inside the body. Each shape constructor accepts an optional label or a label plus attributes. Attributes override generated defaults by key. Labels must be keywords or vectors and must be unique within the snapshot.

Containers follow the same pattern:

frame keeps the supplied :x, :y, :width, and :height.
group derives bounds from its children. A masked group uses its first child as the mask.
bool derives content and bounds from its children with the standard path engine. It requires a boolean operation type. Frame children raise a construction error.

Declaring a case

For a scene named :rects, case ids start with :rects/. The defcase form derives the scene from that keyword namespace and attaches the run function.

(defcase :rects/pan
  {:params default-params
   :view {:scale 1 :x 0 :y 0}
   :context :reuse
   :completion :render-full
   :operation {:steps 20 :dx 200 :dy 0 :settle-ms 100}}
  [rtx]
  (camera/pan! rtx))

The case declares:

:params: scene parameters. Collection validates them against the scene's closed schema.
:view: the starting camera, with an optional viewport override. The default viewport is 1920 by 1080 at device pixel ratio 2.
:context: :fresh creates a page, module, and context per attempt. :reuse keeps one page, module, context, snapshot, and upload for all attempts of the case.
:completion: the declared completion rule, :call, :await, or :render-full. The standard load body drains to Full; the standard pan and zoom bodies finish a camera session with a final Full drain.
:operation: settings read by the operation body.
:batch-size and :preparation: execution settings recorded in the descriptor and checked by comparisons. A batch size groups inner calls inside one attempt; a repetition is a separate attempt. Standard load, pan, and zoom cases run one call per attempt.

Case collection and seeds

Node collects cases from the compiled catalogue. Collection loads every declaration, validates every scene and case, derives the scene seed, selects the case's data fields, and checks that the descriptor survives a serialization round trip. Shape construction, browser startup, and renderer initialization take place during execution.

The scene seed depends only on the master seed, the scene id, and the canonical parameters. These inputs give load, pan, and zoom cases the same starting content across case bodies, collection orders, attempt counts, and feature configurations. They also give A/B configurations matching scene content.

A filter selects cases by a substring of the case id. An empty selection is an error. Declarations outside the filter are still validated.

Snapshots and upload order

The snapshot contract requires the canonical root frame, shape ids that match their object keys, consistent child and parent links, a tree that reaches every shape from the root, and references that point at present shapes. Validation runs before any timer starts.

The browser derives upload order from the snapshot. The root comes first, then descendants in the order of each shape's child list. The browser computes the order and the fingerprint before the upload clock starts. The upload uses the same batch serializer as the editor, with structure, fill, and stroke data batched and path and SVG attributes handled per shape.

Some example scenes

Scene	Content	Cases
`rects`	1000 rectangles with random size, translucent fills, strokes, and radii	load, pan, zoom
`paths`	1000 curved paths with fills and strokes	load, pan, zoom
`plus`	1000 plus signs made from two open line subpaths	load, pan, zoom
`masks`	24 masked groups, each with a circle mask and 8 striped rectangles	load, pan, zoom

A load case renders a fresh scene and times module start, graphics start, upload, and the first render to Full. A pan or zoom case prepares the scene once, restores the starting view outside timing for each attempt, then times a camera session. The standard camera sessions use 20 view updates and a 100 ms settling wait. Pan and zoom parameters vary by scene.

Renderer commands

A renderer command accepts a command context or its Promise and returns a Promise of the next context. The command context has four fields:

Field	Meaning
`:case`	Case descriptor, including the case id.
`:scene`	Prepared starting snapshot with concrete shapes and references.
`:view`	Current zoom scale, camera offsets, and optional viewport settings.
`:hooks`	Functions for clocks, scheduling, rendering, view changes, and the execution guard.

The :scene field describes the starting snapshot. Commands return view changes through :view and apply renderer work through hooks. Every command context supplies :now, :check, :frame, :sleep, :render, :render-sync, :render-from-cache, :set-view, :set-view-start, and :set-view-end as functions. The :now hook returns a clock reading in milliseconds. The :check guard authorizes work by returning and reports cancellation, deadline expiry, context loss, or a superseded owner by throwing an error. The owner is the execution authorized to use the renderer. Returned contexts retain the execution's :check guard; replacing it rejects completion.

Declaring a command

defcommand defines two functions from one body. The base function runs the work; the function with a * suffix also records a measurement event. Measured functions require an active command execution. Both functions resolve their incoming context before running the body. A synchronous body returns the next command context.

(ns example.renderer-commands
  (:require
   [benches.render-wasm.runtime.camera :as camera]
   [benches.render-wasm.runtime.commands :as commands :refer [defcommand]]
   [benches.render-wasm.runtime.protocol :as protocol]))

(defcommand render-sync
  "Renders the prepared scene synchronously."
  [rtx]
  ((get-in rtx [:hooks :render-sync]))
  rtx)

:pre conditions validate arguments before work and timing in every build. Input resolution, context validation, and argument validation precede the start clock reading. Context validation checks the complete set of hooks. Output validation and event recording follow the endpoint.

The body duration is


\Delta t = t_{\mathrm{end}} - t_{\mathrm{start}},

where both clock readings use milliseconds from the same origin. A synchronous command captures the endpoint immediately after its body returns. An asynchronous command supplies an endpoint or captures it when complete runs.

Asynchronous completion

The option :async [complete fail] supplies two completion callbacks to the body. Their names belong to the declaration; the public command arguments stay the same. The first callback call determines the outcome:

(complete next-rtx) succeeds with the next command context and captures the endpoint at that call.
(complete next-rtx evidence) succeeds with an explicit endpoint and raw measurements when evidence supplies them.
(fail cause) rejects the command with its cause and available partial trace.

Successful completion requires a call to complete. A returned Promise forwards its rejection to fail. The execution records diagnostics for repeated completion calls while it is active. Its guard checks authorization at completion, and the measured command records its event after output validation.

The optional completion map has these fields:

Field	Meaning
`:end-ms`	Clock reading at the end of the work, using the same clock origin as the start.
`:metrics`	Namespaced metric keywords mapped to nonempty vectors of raw numbers.
`:trace`	Details of the work, such as traces of render calls and requested waits.

The command supplies :command/elapsed, its body duration in milliseconds. :metrics supplies additional quantities. Command events preserve zero, negative, NaN, and infinite numeric values as raw measurements.

Camera completion illustrates the callback interface:

(defcommand pan
  "Moves the view and completes when rendering reaches Full."
  {:async [complete fail]}
  [rtx {:keys [dx dy steps settle-ms]}]
  (let [target (-> (:view rtx)
                   (update :x + dx)
                   (update :y + dy))]
    (-> (protocol/start-camera! rtx {:settle-ms settle-ms})
        (protocol/animate-view! {:to target :steps steps})
        (protocol/finish-camera!)
        (.then (fn [trace]
                 (complete (assoc rtx :view target)
                           {:end-ms (:end-ms trace)
                            :metrics {:renderer/call-elapsed
                                      (mapv :duration-ms (:slices trace))}
                            :trace trace})))
        (.catch fail))))

The final renderer call can return before the Promise callback runs. The :end-ms field carries that renderer endpoint through the scheduling delay. For example, renderer work lasting 7 ms followed by a 500 ms callback delay still supplies a renderer duration of 7 ms. A clock reading in the later callback would measure 507 ms.

The provided camera/pan and camera/pan* commands accept explicit :dx, :dy, :steps, and :settle-ms. They submit interpolated views on animation frames, draw cached previews, wait for settling, and finish rendering to Full. They resolve to the context with the final view. The measured function records:

Metric	Values in one event, in milliseconds
`:command/elapsed`	One duration from the start of the body through the renderer endpoint.
`:renderer/call-elapsed`	One duration per renderer call in the final drain.
`:renderer/cache-call-elapsed`	One duration per cached render call.
`:camera/finalization-to-full`	One duration from before ending the interaction through the renderer endpoint.

Its trace also contains individual render calls, view-update durations, waits, and the final :end-ms. Each vector preserves the call order within its quantity.

Composition and execution

A composite combines commands. :measured-variant false defines its base function and selects measurements through the calls in its body:

(defcommand selected-pans
  {:measured-variant false}
  [rtx]
  (-> rtx
      (camera/pan* {:dx 10 :dy 0 :steps 2 :settle-ms 5})
      (camera/pan {:dx 0 :dy 5 :steps 2 :settle-ms 5})
      (camera/pan* {:dx -10 :dy 0 :steps 2 :settle-ms 5})))

All three calls update the view and renderer. The first and third record events. Command functions resolve incoming Promises, so threading passes each updated context to the next command.

A command execution runs a function from one command context to the next, authorizes its work, and collects events in completion order. Given a prepared command context rtx:

(commands/execute rtx selected-pans)

The returned Promise resolves to an outcome. A successful outcome contains :outcome :completed, the next :context, and ordered :events. A failed outcome contains :outcome :failed, :error, and the events recorded before failure. Existing Errors retain their identity and partial trace. Other thrown values become Errors with the original value in :thrown. An active execution can also record repeated completion calls in :diagnostics.

An event has this shape:

{:command :benches.render-wasm.runtime.camera/pan
 :metrics {:command/elapsed [74]
           :renderer/call-elapsed [7 7]
           :renderer/cache-call-elapsed [2 2]
           :camera/finalization-to-full [33]}
 :trace {:end-ms 74}}

One event represents one measured command call. Its metric vectors describe the individual values that call produced. The command execution owns this history; its guard authorizes work for that execution. Pass the resulting context to commands/execute to start another command execution.

Runtime code

runtime/commands.clj declares commands; runtime/commands.cljs executes them. runtime/camera.cljs defines rendering and camera operations. runtime/protocol.cljs defines render drains, camera sessions, hook contracts, and trace fields.

Case declarations such as scenes/rects.cljs attach operation bodies through declarations.cljc. Node loads these declarations and functions during catalogue collection. Renderer operations run through the supplied hooks when the functions execute in the browser. browser/bridge.cljs constructs the runtime context for case operations and prepares graphics contexts, snapshots, and uploads. Reused cases restore their starting view outside interaction timing before each attempt.

Measurement

Preparation and measurement boundaries

Each duration has a defined start and endpoint. Setup and diagnostics run around the interval whose work they prepare or describe.

Work around those intervals includes:

scene build, snapshot validation, and fingerprinting;
upload order derivation;
request and result serialization;
validation, logging, and screenshots;
module script import;
camera restore and the drain that prepares a warm case.

The timed work includes:

module instantiation;
graphics context creation and initial size synchronization;
shape batch upload, including the loading lifecycle and first view setup;
the render drain to Full;
for warm cases, the camera session: view updates, cached previews, settling, ending the interaction, and the final drain.

Lifecycle of a fresh load

flowchart TD
    A[Import module script - untimed] --> B[Time module start]
    B --> C[Time graphics start]
    C --> D[Build and validate snapshot - untimed]
    D --> E[Fingerprint and derive upload order - untimed]
    E --> F[Time upload and first view setup]
    F --> G[Time render drain to Full]
    G --> H[Record result]
    H --> I[Dispose page and context]

The module script import is untimed because the browser loads it once per page. Module instantiation starts a new renderer state. Graphics start creates the WebGL2 context and synchronizes the drawing buffer. Upload covers the loading lifecycle, batch serialization, and tile preparation. The first render clock starts after upload ends.

Warm interactions

A warm case owns one prepared page. Before each attempt, the browser restores the case's starting view and drains to Full outside timing. The operation then runs a camera session:

start the camera interaction;
for each view update, wait for the next animation frame, apply the view, and present cached pixels;
wait for the requested settling time;
end the camera interaction;
render until Full.

A pure pan can reuse cached textures while it updates which tiles are visible. A zoom rebuilds and invalidates tile textures. Standard cases request a 100 ms settling wait and record the requested and observed durations separately from renderer calls. Camera finalization includes the call that ends the interaction; the run also records that call's duration.

Slices and metrics

The browser returns raw evidence for each attempt. A fresh attempt also reports module-init-ms, graphics-init-ms, and upload-ms from preparation.

A fresh attempt reports one drain:

slices: one record per renderer call with timestamp, flags, frame-type, and duration-ms;
viewport-ready-ms: the first ViewportReady or Full completion from the drain origin;
full-ms: the Full completion from the drain origin;
first-render-ms: the single first-render metric for a fresh case.

A warm attempt reports the camera session and the final drain:

cached-slices: one record per gesture frame with timestamp, duration-ms for the cached preview call, and view-and-preview-ms for the view update plus preview call;
slices: the final drain, with the same shape as a fresh drain;
interact-ms: the whole camera session, from before the interaction starts through the final Full;
active-ms: from before the interaction starts to after the last cached preview and its guard check, or after an explicit pause that follows it;
settling-requested-ms and settling-actual-ms: the declared and observed settling waits;
set-view-end-ms: elapsed time during the call that ends the interaction;
time-to-viewport-ready-ms and time-to-full-ms: durations from camera finalization through the first ViewportReady or Full, and through Full, respectively;
last-input-to-full-ms: from before the last view update to the endpoint of the renderer call that returns Full.

Both drains and camera sessions return end-ms, the clock reading immediately after the final renderer call returns Full. Completion durations end at that reading. Promise callbacks can run later as the browser schedules queued work.

Both kinds of attempt report renderer-call-total-ms, the sum of the renderer call durations, computed after timing.

A saved run metric declares its unit, semantics, basis, and zero policy. A batch average records its call count; one attempt supplies one observation. Raw slices keep zero durations and non-finite values. The metric entry classifies the value for analysis and records a reason when it is invalid.

Renderer completion

The renderer finishes in tile budgets. One renderer call returns a frame type, and the caller continues while the renderer reports Partial or ViewportReady. The drain completes when a call returns Full. None or an unknown value fails the attempt.

Value	Frame type	Meaning
0	`None`	no frame
1	`Partial`	visible work remains
2	`Full`	the renderer finished the current request
3	`ViewportReady`	the visible viewport was submitted; work outside it may remain

Full marks renderer submission of the completed request. Skia and WebGL queue GPU work, and GPU execution and display follow their own schedules. A call duration measures synchronous work on the calling thread, including any driver stall. These clock readings describe work through the renderer return.

For scheduled calls, the browser passes the timestamp of the animation frame. For immediate calls, it reads the clock. Each slice records its timestamp to describe the call sequence.

Graphics environment

The benchmark uses WebGL2 with antialias=false, depth=true, stencil=true, alpha=true, and preserveDrawingBuffer=true. The default viewport is 1920 by 1080 at device pixel ratio 2. The preparation records the effective CSS size, drawing-buffer size, device pixel ratio, context attributes, renderer and vendor strings, and a software rendering warning.

Benchmarks run inside the Linux development container. Start it with ./manage.sh run-devenv --gpu to enable hardware access. This sets PENPOT_GPU=1; the runner then selects ANGLE Vulkan, and pnpm run render-bench starts Xvfb to supply the display that hardware rendering needs in the container.

--require-gpu makes run and ab require hardware WebGL2. The command first checks for PENPOT_GPU=1, before builds, catalogue checks, or output allocation. After each preparation returns, the runner checks its existing :effective-graphics metadata before executing attempts. It requires WebGL2, usable vendor and renderer identities, and :software false. A rejected preparation aborts the run with exit 1 and stops later A/B configurations. The record retains the preparation and graphics failure evidence, and leaves pending attempt slots unattempted. The failure names the observed renderer and the commands needed to enable GPU access. ab --dry-run skips these GPU checks.

Each preparation records the graphics identity of its own context. The run stores the GPU policy and environment signal under diagnostics. Default runs allow SwiftShader fallback and warn when they record software rendering. Comparisons check effective graphics settings and resolved launch arguments. Comparing hardware and software runs requires --allow-all-differences.

Results and reports

Run record

A run record contains:

the plan: selected case descriptors, warmup count, repetition count, and timeout policy;
metadata: Git revision and dirty flag, build facts, environment, and diagnostics;
the resolved analysis settings;
preparations in chronological order;
attempts in chronological order, each with its case, preparation, outcome, metrics, trace, and failure evidence;
remaining slots for each case;
the termination reason.

A preparation records its case, fingerprint, effective graphics, and setup diagnostics. An attempt records one outcome and its metric entries. Entries can describe invalid values or missing measurements. Errors retain available traces of render calls and waits after a failure.

Failures and termination

A failed attempt still consumes its allocated slot. A fresh preparation failure records a failed attempt and consumes the slot. A warm preparation failure or a warm attempt failure releases the prepared page and context and leaves that case's remaining slots unattempted. A browser or page failure closes the browser it started, abandons the current case, and starts a new browser for the next case when possible. The run fixes its attempt count before execution and uses each allocated slot at most once.

A run terminates in one of four states:

completed: every allocated slot was attempted and every attempt completed.
failed: every allocated slot was attempted and at least one attempt failed.
incomplete: slots remain because a preparation, browser, or storage failure stopped the case.
interrupted: SIGINT or SIGTERM stopped the run.

The runner saves partial records before it exits. Use exit status to determine whether the command completed successfully; saved records also describe work that ended in failure.

Summaries

A summary reports, for each case and metric:

the observed mean, median, minimum, maximum, sample standard deviation, and median absolute deviation;
a bootstrap confidence interval for the median. The interval comes from repeated resampling of the observed attempts;
observed p95 and p99 values, the 95th and 99th percentile durations, when the sample is large enough;
counts in the valid, invalid, absent, failed, unattempted, and warmup categories;
the definitions and units of the metric.

Statistics use valid values from completed measured attempts. The record also retains warmups and failures. A completed attempt can supply a valid value for one metric and an invalid value for another. A pooled estimate combines values across attempts and requires matching metric definitions.

The default analysis settings are:

Setting	Default
Draws	2000
Seed	`0x70656e70`
Confidence	0.95
Minimum CI	10
Minimum p95	100
Minimum p99	500

The run records draw count, seed, confidence, and minimum counts, and a report can override them at analysis time. An estimate that cannot be computed keeps a reason and, when applicable, the minimum count it needs.

The intervals are exploratory and conditional on the observed attempts. Attempts within one run share evolving caches and can have dependent durations. Repeated whole runs provide evidence about reproducibility across those cache conditions.

Comparisons

A comparison pairs saved cases by identical case id unless --cases A,B selects one explicit baseline, candidate pair. The CLI checks the selector syntax before it loads either record, then resolves each selector by saved case id or exact name.

By default, a comparison requires equal:

scene identity, scene seed, and scene parameters;
starting-snapshot fingerprints;
view, context policy, completion rule, batch size, preparation settings, and operation settings;
effective graphics settings and runtime environment;
build mode, effective features, instrumentation settings, and recorded build environment;
warmup count, timeout policy, and metric definitions.

Git revision, dirty state, run ids, timestamps, and saved analysis defaults provide provenance for interpreting the paired cases.

The comparison reports absolute and percentage median changes, with a bootstrap interval for each change. A zero baseline suppresses a percentage change. The comparison leaves a metric with differing definitions out unless the all-differences permission allows raw values.

Comparison permissions

The comparison records every compatibility difference. A permission acknowledges a class of differences and labels each acknowledged mismatch with the rule that allowed it.

--allow-build-differences acknowledges recorded build settings, effective features, instrumentation, and the recorded build environment. Runtime environment, graphics, operation settings, and metric definitions stay strict.
--allow-scene-differences acknowledges scene identity, scene seed, scene parameters, and differences in fingerprint content. Execution settings stay strict. Missing or varying fingerprint evidence still needs the all-differences permission.
--allow-all-differences acknowledges every compatibility difference, including mixed metric definitions. The report compares valid raw values in their recorded units. Record validation and measurement validity still govern the analysis.

Build and scene permissions combine by union. All-differences includes both. A report shows the selected permissions, every recorded difference, and the rule that acknowledged it. Exit status reports execution and compatibility checks; the estimates describe the observed timing differences.

A/B feature experiments

An A/B experiment answers one question: how do the selected cases behave when a renderer feature is off versus on, with everything else held constant?

The ab command takes the requested features, plans the configurations, builds every configuration, runs every configuration, and compares every edge: a pair of configurations that differ by one feature.

Planning

A feature configuration is a set of enabled Cargo features. The planner:

normalizes the requested names by trimming, deduplicating, and sorting them;
validates them against the renderer manifest with Cargo metadata;
requires an empty Cargo default feature set, so each configuration states its features explicitly;
rejects diagnostic features used for profiling or statistics;
orders the configurations in reflected Gray order, so adjacent configurations differ by one feature, with the first requested feature as the most significant bit;
emits every pair of configurations that differ in exactly one feature, from the configuration with that feature off to the configuration with it on.

The command requires at least one requested feature. For N features, the comparison has 2^N configurations and N 2^{N-1} edges. The planner compares each edge once, in the direction from off to on. For features [a, b], the configurations are 00, 01, 11, 10 and the edges are:

00 -> 10 : a
00 -> 01 : b
01 -> 11 : a
10 -> 11 : b

--max-features accepts 1 through 6 and defaults to 3. When --dry-run is absent, the command requires --confirm for more than 8 configurations or a filter that is absent, empty, or contains only whitespace. --dry-run checks the requested features using Cargo metadata, selects cases from their declarations, and prints the features, configurations, edges, selected case count, and time limits. It creates no comparison directory, compiles no build, and launches no browser.

Running a feature comparison

Every configuration gets its own build, catalogue, browser executor, server, browser, and run. Each configuration runs and cleans up before the next one starts. The configurations share the selected cases and their seeds, parameters, views, operations, budgets, preparation settings, and batch sizes. Collection checks each configuration's catalogue identity and requires the same selected case descriptors in the same order.

pnpm run build:renderer-benchmarks:runner

pnpm run render-bench -- ab \
  --features <feature-a>,<feature-b> \
  --filter rects/load \
  --warmups 3 --repetitions 10

The feature names must exist in the renderer manifest. Placeholders appear here because the manifest decides which features are available.

After a configuration records attempts, scheduling continues with its recorded failures and remaining slots. A build failure, a catalogue identity failure, or a run with zero attempts stops scheduling later configurations; those configurations are recorded as unattempted with a reason. SIGINT and SIGTERM save the available state and return 130 or 143.

Feature-comparison output

The default output directory is target/performance/render-wasm/feature-comparisons/<ab-id>/. It contains:

feature-comparison.transit             plan, progress, edges, termination
feature-comparison.log                 comparison progress and diagnostics
configs/<slug>/run.transit             the configuration's run record
configs/<slug>/run.log                 the configuration's log
configs/<slug>/*.png                   optional screenshots
comparisons/<from>__<to>.transit       one edge comparison

The feature-comparison record stores the normalized options, comparison permissions, selected cases, configuration order, requested and effective features, build and run references, edges, failures, and interpretation warnings. The runner validates the record with a schema and writes it atomically. It writes the initial plan before the first build and updates the record after each configuration, after each edge, and at termination. The command prints Comparison <id>, Output <absolute path>, and Status <status> on stdout at termination. Progress and failures go to stderr. Each configuration has its own build and run log.

Edge comparisons

An edge comparison uses the ordinary comparison path. It automatically acknowledges one difference: the effective feature change that matches the planner's evidence. The evidence records the requested feature, both effective feature sets, and the added and removed features. It also requires the two configurations' requested feature selections to differ only by the requested toggle and to record the same default-feature policy. If the saved sets do not match that evidence, the difference remains unacknowledged.

The comparison record also carries the edge context: the source and target configuration slugs and the expected feature facts. An edge is analysed when both endpoint records exist, even if one run recorded failed attempts. A missing endpoint or an analysis error produces an explicit edge outcome. An edge that retains an unacknowledged difference keeps its saved report, is marked failed, and makes the invocation exit 1.

Other build differences require --allow-build-differences. Scene differences require --allow-scene-differences. Operation, view, context, preparation, completion, batch size, graphics, runtime environment, and metric-definition differences stay strict unless --allow-all-differences is selected. If two configurations report the same effective feature set, the comparison includes their edge and labels it with that fact. Effective feature sets describe the code settings that accompany the observed timings.

The command returns 0 when every configuration completes, every edge analysis finishes, and every difference is acknowledged. It returns 1 for execution, persistence, or analysis failures, including unacknowledged differences. Edge reports show raw times and statistics for the observed differences. Establishing a causal effect requires experimental control and repeated measurements.

How to run

Run these commands from frontend/ inside the development container with dependencies and Playwright Chromium installed. Keep renderer watchers stopped until preparation finishes.

The literal -- after render-bench is required to forward the command and its arguments through the shell wrapper.

List the commands, or show one command's options and defaults:

pnpm run render-bench -- --help
pnpm run render-bench -- run --help

Command options select declared cases and control the build source, sampling, output and hardware policy. Case declarations own workload bodies, scene parameters, view and operation settings, and case lifecycle.

Build the runner and its command module:

pnpm run build:renderer-benchmarks:runner

Run one case with one measured attempt:

pnpm run render-bench -- run \
  --filter paths/pan --seed 42 --warmups 0 --repetitions 1 --screenshots

Prepare a build once and reuse it:

pnpm run render-bench -- build
pnpm run render-bench -- run --build <build-id>

Run a selection with the default sampling:

pnpm run render-bench -- run --filter rects

Require hardware rendering for rectangle load, pan, and zoom in a GPU container:

pnpm run render-bench -- run --filter rects --require-gpu

For a functional check, add --warmups 0 --repetitions 1. A successful run renders the scenes through Full and saves the vendor and renderer strings with :software false in each preparation's :effective-graphics.

Choose a new path for --output-dir. The runner rejects an existing output directory. A/B feature comparisons apply the same rule.

Compare two saved runs offline:

pnpm run render-bench -- compare \
  target/performance/render-wasm/runs/<baseline>/run.transit \
  target/performance/render-wasm/runs/<candidate>/run.transit

Allow a difference in build configuration, or compare two given scenes, irrespective of their differences:

pnpm run render-bench -- compare \
  target/performance/render-wasm/runs/<baseline>/run.transit \
  target/performance/render-wasm/runs/<candidate>/run.transit \
  --allow-build-differences

pnpm run render-bench -- compare \
  target/performance/render-wasm/runs/<baseline>/run.transit \
  target/performance/render-wasm/runs/<candidate>/run.transit \
  --cases rects/load,paths/load --allow-scene-differences

Merge an additional saved run into a base run:

pnpm run render-bench -- merge \
  target/performance/render-wasm/runs/<base>/run.transit \
  target/performance/render-wasm/runs/<additional>/run.transit \
  --output-dir target/performance/render-wasm/runs/<merged>

The merged record gets a new run id and a new run directory. Both input records stay unchanged. Both inputs must be completed and compatible: same cases, timeout policy, analysis, build facts, runtime environment, and Git provenance. The additional run must use the same warmup count. merge appends only the additional measured attempts and uses the base run's warmups. The additional warmup attempts stay in the additional record. The command starts no build, server, or browser.

Inspect and clean managed outputs:

pnpm run render-bench -- purge builds
pnpm run render-bench -- purge runs
pnpm run render-bench -- purge comparisons
pnpm run render-bench -- purge all

purge requires exactly one target: builds, runs, comparisons, or all. It removes known build, standalone-run, or feature-comparison outputs and keeps directories and unrelated files. It follows directory symlinks and can remove matching output files from their targets, including targets outside the managed directories. all removes every category. A missing, unknown or multiple target is an argument error.

The commands write results to stdout:

build: Build <id> and Output <absolute path> after build preparation ends.
run: Run <id> and Output <absolute path> after the run ends. When a report is available, a Report heading and the formatted summary follow.
ab: Comparison <id>, Output <absolute path>, and final Status <status>.
compare: the comparison report.
merge: Run <id> and Output <absolute path>.
--help and COMMAND --help: usage text with that command's options and defaults. purge produces no output on success.

Each CLI command produces an outcome with an exit code, errors, log path, and optional report text. The runner prints the final output once and returns the numeric status.

Progress and diagnostics go to stderr. The default shows build stages, case headings, and attempt progress. -q / --quiet shows results, warnings, and errors. -v / --verbose also shows debug details and raw build output. Quiet and verbose are mutually exclusive. These options apply to every command.

--color forces terminal colours; --no-color disables them. Automatic colour uses a terminal stream when NO_COLOR is empty or absent and TERM is not dumb. Colour choices apply to terminal diagnostics. Result text and file logs use plain text. The runner reads child output as lines and normalizes their line endings. Verbose terminal output prefixes and dims these lines; file logs store the lines as text. Terminal errors include a limited extract from the end of the log when available.

Files receive every log level, including debug output hidden from the console. Each build writes build.log; each A/B comparison writes feature-comparison.log. The default run directory is target/performance/render-wasm/runs/<run-id>/, which contains run.log, run.transit, and optional screenshots.

Exit 0 means the command completed successfully: a ready build for build, every allocated attempt for run, a valid comparison for compare, a validated merged record for merge, every configuration and edge for ab, and a clean purge for purge. Exit 1 means a failed preparation, a failed attempt, an incomplete run, an invalid selection, incompatible merge inputs, an unacknowledged comparison difference, or a harness failure. Exits 130 and 143 mean interruption by SIGINT and SIGTERM for build, run and ab. Interruption uses those statuses to report aborted work. Errors writing the command's record or log also produce diagnostics.

--warmups defaults to 3, --repetitions to 10, --timeout-ms to 30000, and --startup-timeout-ms to 30000. build, a run that prepares its build, and an A/B experiment use a default --build-timeout-ms of 300000; a run with --build uses the retained build's recorded provenance. The master seed defaults to 42 and accepts a 32-bit unsigned integer. A filter matches a substring of the case id.

Unit tests

Run these commands from frontend/ inside the development container with dependencies and Playwright Chromium installed. The benchmark tests use the performance-tests build target and the runner in test/performance_tests/render_wasm/runner.cljs.

Build and run the full suite:

pnpm run test:renderer-benchmarks

Build once, then run the compiled tests:

pnpm run build:renderer-benchmarks:tests
node target/performance-tests/test.js

Focus on one namespace or one test var:

node target/performance-tests/test.js \
  --focus performance-tests.render-wasm.contracts-test

node target/performance-tests/test.js \
  --focus performance-tests.render-wasm.contracts-test/defscene-registers-metadata-and-build

Rebuild the test bundle after changing source or tests. --focus accepts one namespace or one namespace/test-var. The runner exits 0 when the selected tests pass and 1 when they fail.

Scope and limitations

The suite records durations for setup, render calls, and camera movement inside Chromium. Rendering durations end when the renderer returns after submitting work. Memory, power, frame rate, frame drops, visual correctness, network latency, and editor input-to-paint each require measurements for those quantities.

A fresh case starts a new page, module, and WebGL2 graphics context. The browser process persists across successful cases, and module script import precedes module timing. These startup conditions define measurements for fresh cases.

Attempts share evolving caches, so bootstrap intervals describe the observations within a run. Repeat whole runs to assess reproducibility, and judge whether the observed change matters for the workload.

Repository map

frontend/performance/benches/render_wasm/
├── browser/          page entry and browser bridge
├── features/         feature selection, configurations and comparison edges
├── node/
│   ├── catalog/      compiled catalog entry and retained catalog loader
│   ├── cli/          CLI entry, resolution, requests and execution lifecycle
│   │   └── commands/ per-command schemas, help and resolution
│   ├── report/       saved-record comparison and merge
│   ├── runner/       build preparation, server, browser host and run orchestration
│   └── storage/      artifacts and saved runs
├── report/           summaries, comparisons, statistics, formatting
├── runtime/          renderer commands, render drains, and camera sessions
├── scenes/           scene recipes and their cases
├── builder.cljc      scene construction
├── declarations.cljc scene and case registry, schemas, seed derivation
├── cases.cljs        case discovery and collection
├── codec.cljs        Transit round-trip validation
├── errors.cljc       failure values and diagnostics
├── snapshot.cljs     snapshot validation and upload order
├── fingerprint.cljs  digest of the starting snapshot
├── identity.cljc     build identity and bundle compatibility checks
├── measurement.cljs  metric normalization
├── random.cljs       seeded random generators
├── result.cljs       run, comparison and feature-comparison records
├── README.md         this document
└── REPORTING.md      records, metrics, and statistics

frontend/test/performance_tests/render_wasm/
├── runner.cljs         test entry point, namespace registration, focus selection
├── *_test.cljs         tests for benchmark modules
├── test_helpers.cljs   shared scene-test helpers
└── cli_integration.cjs tests for the CLI through Node

frontend/package.json          build and test commands
frontend/shadow-cljs.edn       benchmark and performance-tests build targets
frontend/src/app/render_wasm/  production renderer wrapper
render-wasm/                   Rust renderer and build

README.md

WASM renderer benchmarks

Contents