diff --git a/frontend/performance/benches/render_wasm/README.md b/frontend/performance/benches/render_wasm/README.md index 76e02fd3c8..4ddf5dda9c 100644 --- a/frontend/performance/benches/render_wasm/README.md +++ b/frontend/performance/benches/render_wasm/README.md @@ -1,30 +1,26 @@ # WASM renderer benchmarks -The renderer benchmark suite measures the Penpot WASM renderer in Chromium. It -provides means to declare parameterized scenes, then render them on a browser -process, recording how long each of several phases take. It can measure "cold" -starts, "warm" interactions like pan and zoom, and compare runs with different -renderer builds. - -| 💡 Jump to [How to run](#how-to-run) below for usage instructions. | -| ------------------------------------------------------------------ | +The renderer benchmark suite measures the Penpot WebAssembly (WASM) renderer in +Chromium. Scenes describe shapes; cases describe the work to perform on them. +The suite records setup, rendering, pan and zoom durations and compares runs +with different renderer builds. [How to run](#how-to-run) gives the commands. The suite has three parts: 1. A Node runner controls the experiment. It prepares or reuses a build, selects - test cases, serves the browser executor, launches the browser, schedules - "attempts" (since cases can fail), saves results, and prints reports. -2. The browser executes the experiment. It owns the WASM module, graphics - context, scene snapshot, "upload" (to the renderer), interactions, and - timing. -3. The reports tool analyses saved records. It runs in Node or the browser - without loading the renderer. + cases, serves the browser code, launches Chromium, schedules attempts, saves + results, and prints reports. An attempt is one execution of a case. +2. The browser creates the renderer and graphics context, constructs the scene + snapshot, uploads its shapes, performs the work, and records durations. +3. The reporting code computes statistics from saved records in Node or the + browser without loading the renderer. ## Contents - [Main concepts](#main-concepts) - [Architecture](#architecture) - [Workloads](#workloads) +- [Renderer commands](#renderer-commands) - [Measurement](#measurement) - [Results and reports](#results-and-reports) - [A/B feature experiments](#ab-feature-experiments) @@ -64,10 +60,14 @@ The suite has three parts: - **Case**: a scene plus its execution settings: parameters, base view, context policy, completion rule, optional operation settings, and a run function. A case is the unit that is measured. -- **Operation**: the work a case performs during one attempt, written as a - function that runs in the browser. The function receives a runtime context - with the scene, module, view, clock, frame scheduler, and renderer hooks. - Standard operations load, pan, and zoom. +- **Operation**: the function that performs a case's work during one attempt. + It receives a runtime context with the case data, scene snapshot, renderer + module, view, and hooks. Standard operations load, pan, and zoom. +- **View**: the camera's zoom `:scale` and offsets `:x` and `:y`. + Optional `:viewport` settings give the canvas width, height, and device pixel + ratio, the number of drawing-buffer pixels per CSS pixel along each dimension. +- **Hook**: a function supplied through `:hooks` for clock readings, + scheduling, view updates, rendering, or checking whether work may continue. - **Case descriptor**: What gets sent to the browser. It contains the case id, scene, derived seed, parameters, view, context policy, completion rule, batch size, preparation settings, and operation settings. Any functions are removed. @@ -80,16 +80,16 @@ The suite has three parts: ### Execution - **Build**: one compiled renderer plus the matching catalogue and browser - executor. A build id ties the three artefacts together, and the runner rejects + executor. A build id ties the three outputs together, and the runner rejects a mismatch. - **Retained build**: a ready build that the suite stores under the managed - build directory and later runs can reuse without recompiling. + build directory for reuse in later runs. - **Fresh case**: a case with `:context :fresh`. Each attempt gets a new page, module, graphics context, and upload. - **Warm case**: a case with `:context :reuse`. One page, module, graphics context, snapshot, and upload serve all attempts of the case. -- **Preparation**: the untimed setup for one measured context. It covers module - instantiation, WebGL context creation, scene build, snapshot validation, +- **Preparation**: setup for one graphics context. It covers module + instantiation, WebGL2 context creation, scene build, snapshot validation, fingerprinting, upload-order derivation, and upload. A warm preparation also restores the starting view and drains to `Full`. One page owns one preparation. @@ -114,13 +114,39 @@ The suite has three parts: - **Runner**: the Node command line interface that builds or reuses a build, selects cases, serves the build, launches Chromium, runs attempts, saves records, and prints reports. -- **Measurement**: a timed phase. The suite times module start, graphics start, - upload, first render, and warm interaction separately. Each measurement - records its value, unit, and basis. The basis says whether the value covers a - whole attempt or averages a number of inner calls. -- **Slice**: one renderer call record inside an attempt: timestamp, flags, - frame type, and call duration. Slices are raw evidence. They are not - independent observations. +- **Browser executor**: the compiled page and JavaScript that perform + preparation and case operations. Its bridge exposes the operations that Node + can request. +- **Measurement**: a numeric result for a quantity such as elapsed duration. + The suite measures module start, graphics start, upload, first render, and + camera movement separately. +- **Metric**: the name and definition of a measured quantity. Saved run metrics + declare a unit, semantics, basis, and zero policy. Semantics identifies what + the value measures. The basis states whether a value covers an attempt or + averages its inner calls. The zero policy states whether zero supplies a valid + value for analysis. +- **Observation**: one valid value from a measured attempt for statistical + analysis. A batch average supplies one observation for its attempt. +- **Slice**: a trace entry for one renderer call: timestamp, flags, frame type, + and duration. Slices describe the work within an attempt. +- **Trace**: details of the work, including individual renderer calls, + cached previews, requested waits, and observed durations. +- **Clock reading**: a time in milliseconds from the clock's origin. Subtracting + two readings from that clock gives an elapsed duration. +- **Endpoint**: the clock reading at the end of the measured work. The renderer + endpoint is the reading immediately after a renderer call returns. +- **Render drain**: repeated renderer calls until one returns `Full`. Partial + completion schedules another call on the next animation frame. +- **Cached preview**: one render from cached tiles after a view update. +- **Camera session**: camera start, view updates with cached previews, a settling + wait, camera end, and the final render drain. +- **Active interval**: the part of a camera session from camera start through + the last cached preview and its guard check, or an explicit pause that follows + the preview. +- **Settling wait**: the requested pause after the active interval and before + ending the interaction. The observed wait includes scheduling delays. +- **Camera finalization**: the interval from immediately before ending the + interaction through the renderer call that returns `Full`. - **Frame type**: the completion signal returned by one renderer call. `None` means no frame; `Partial` means visible work remains; `Full` means the renderer finished the current request; `ViewportReady` means the visible @@ -130,6 +156,34 @@ The suite has three parts: - **Atlas**: a GPU texture that stores many tiles or a document-space preview. Atlases let warm pan and zoom present cached pixels before new tiles finish. +### Command execution + +- **Renderer command**: a function that accepts a command context or its Promise + and returns a Promise of the next context. The base function uses the declared + name, such as `pan`. The measured function adds `*`, such as `pan*`, and records + an event for its work. +- **Command context**: the map with keys `:case`, `:scene`, `:view`, and `:hooks`. + It carries the case descriptor, prepared scene snapshot, current view, and + hooks. +- **Command body**: the work declared by `defcommand`. A synchronous body + returns the next context. An asynchronous body completes through callbacks + named by its declaration. +- **Composite command**: a command that combines calls to other commands. + A declaration with `:measured-variant false` defines its base function and + collects events from the measured commands it calls. +- **Command execution**: one call to `commands/execute`, which runs a function + from one command context to the next and records measured calls in completion + order. +- **Event**: the record for one measured command call. It contains the command + name, vectors of raw metric values, and an optional trace. +- **Guard**: the `:check` hook. A return authorizes work; an error rejects it. + The renderer owner is the execution authorized to use the renderer. The guard + can report cancellation, deadline expiry, context loss, or replacement of + that owner. +- **Completion callback**: `complete` or `fail` in an asynchronous command body. + The first call determines the outcome: `complete` supplies the next context + and optional measurements; `fail` supplies a cause. + ### Results - **Report**: the statistics and tables computed from one or more run records. @@ -151,10 +205,10 @@ The suite has three parts: - **Feature configuration**: the set of Cargo features enabled for one renderer build. - **Feature comparison**: all on and off combinations of the requested features. - `N` requested features produce `2^N` configurations. + $N$ requested features produce $2^N$ configurations. - **Edge**: two configurations that differ in exactly one feature, compared from the configuration with the feature off to the configuration with the feature - on. A comparison with `N` features has `N * 2^(N-1)` edges. + on. A comparison with $N$ features has $N 2^{N-1}$ edges. - **Edge comparison**: the comparison record for one edge, stored with the feature comparison. - **Feature-comparison record**: the saved plan, configurations, runs, edges, @@ -168,20 +222,20 @@ The Node runner owns everything outside the page: - it prepares or loads a build; - it reads the compiled catalogue and selects cases; -- it serves the selected build from an owned loopback server; +- it serves the selected build from a server on `127.0.0.1`; - it launches Chromium and creates pages; - it schedules warmups and measured attempts; - it records preparations, attempts, failures, and termination; -- it writes the run record after every material change; -- it asks the shared report code for the terminal summary. +- it writes the run record as work progresses; +- it prints a final report. The browser bridge owns everything inside the page: -- it validates a request before claiming the page; +- it validates a request before assigning the page to that work; - it imports the renderer module and creates the canvas and WebGL2 context; - it builds, validates, fingerprints, orders, and uploads the scene; - it restores the starting view outside timing for warm cases; -- it runs the case operation and times the requested phases; +- it runs the case operation and records the requested measurements; - it returns plain data and disposes the context when asked. The shared reporting code owns analysis: @@ -214,38 +268,37 @@ sequenceDiagram N->>N: persist the run record ``` -The node runner saves the initial plan before it starts a browser, and updates +The Node runner saves the initial plan before it starts a browser, and updates the run record after each preparation and attempt, and at termination. Reports -are build from these saved records later. +use these saved records. ### Builds and catalogues -A build has three artefacts that must agree: +A build has three matching outputs: - the renderer module; - the case catalogue compiled for Node; - the browser executor and its page. -Preparation compiles the renderer in release mode with Cargo, then compiles the -catalogue and browser executor with the same build _id_, tying the three artefacts -together. The runner checks the id before case collection and when a page starts, -and rejects a mismatch. +Build preparation compiles the renderer in release mode with Cargo, then compiles +the catalogue and browser executor with the same build id. The runner checks +the id before case collection and when a page starts, and rejects a mismatch. -An ordinary measurement build uses the renderer's default cargo feature set. A +An ordinary measurement build uses the renderer's default Cargo feature set. A feature experiment may enable selected functional features. Both kinds of measurement build should exclude diagnostic features such as `stats`, `profile`, `profile-macros`, and `profile-raf`. The build checks that we are in release mode, the optimization level, and disabled profiling marks. -The build record also captures a fixed list of environment values that Cargo -receives, such as compiler, target, and Emscripten settings. Note that it only -records the environment supplied to Cargo via a whitelist. It's not a full -environment dump, and it does not certify a hermetic build. +The build record captures a fixed list of environment values supplied to Cargo, +including compiler, target, and Emscripten settings. These values provide +provenance for interpreting a run. Reproducing a build also requires its other +environment settings and dependencies. Builds are saved under `target/performance/render-wasm/builds//`. The manifest records the build id, status, functional settings, Git provenance, and build facts. An experiment run can select a retained build with `--build `, -but a run without `--build` always prepares a new one. +and `run` prepares a new build when it omits `--build`. `build` and a run that prepares its build accept `--features`, `--[no-]default-features`, and `--build-timeout-ms`. A run with `--build` reuses @@ -266,10 +319,11 @@ process it starts. Cases run serially. A fresh case gets a new page and context for each preparation. A warm case keeps one page and context for all of its attempts. -The runner bounds every phase with a deadline. When a page stalls past its -deadline, Node stops the browser process and records the failure. The browser -bridge also checks for context loss, expired deadlines, and stale work from an -earlier page before and after asynchronous work. +The runner applies deadlines to startup, preparation, and attempts. When a page +stalls past its deadline, Node stops the browser process and records the failure. +The browser bridge also checks for context loss, expired deadlines, and work associated +with a superseded renderer owner before and after asynchronous work. A browser +thread occupied by synchronous work relies on the Node deadline to stop it. Clean-up closes the page, then the browser, then the server. The runner deletes a build selected with `--remove` only after the final run record is saved. @@ -299,23 +353,25 @@ produce the same snapshot. (rect)))) ``` -Scene scopes do not nest. Ordinary `let`, `doseq`, and functions defined outside -the scope work inside the body. Each shape constructor accepts an optional label +Each scene has one construction scope. Starting another scene inside it raises +an error. Ordinary `let`, `doseq`, and functions defined outside the scope work +inside the body. Each shape constructor accepts an optional label or a label plus attributes. Attributes override generated defaults by key. Labels must be keywords or vectors and must be unique within the snapshot. Containers follow the same pattern: -- `frame` keeps its authored `:x`, `:y`, `:width`, and `:height`. +- `frame` keeps the supplied `:x`, `:y`, `:width`, and `:height`. - `group` derives bounds from its children. A masked group uses its first child as the mask. - `bool` derives content and bounds from its children with the standard path - engine. It requires a boolean operation type and cannot contain frames. + engine. It requires a boolean operation type. Frame children raise a + construction error. ### Declaring a case -A case id uses the scene namespace. The `defcase` form derives the scene from -that namespace and attaches the run function. +For a scene named `:rects`, case ids start with `:rects/`. The `defcase` form +derives the scene from that keyword namespace and attaches the run function. ```clojure (defcase :rects/pan @@ -344,20 +400,20 @@ The case declares: - `:batch-size` and `:preparation`: execution settings recorded in the descriptor and checked by comparisons. A batch size groups inner calls inside one attempt; a repetition is a separate attempt. Standard load, pan, and zoom - cases run one call per attempt. +cases run one call per attempt. ### Case collection and seeds Node collects cases from the compiled catalogue. Collection loads every -declaration, validates every scene and case, derives the scene seed, projects -the case to plain data, and checks that the descriptor survives a serialization -round trip. The process does not build shapes, start a browser, or initialize -the renderer. +declaration, validates every scene and case, derives the scene seed, selects +the case's data fields, and checks that the descriptor survives a serialization +round trip. Shape construction, browser startup, and renderer initialization +take place during execution. The scene seed depends only on the master seed, the scene id, and the canonical -parameters. It does not depend on the case id, collection order, operation -body, attempt count, or feature configuration. This keeps load, pan, and zoom -cases on the same scene content and keeps A/B configurations comparable. +parameters. These inputs give load, pan, and zoom cases the same starting +content across case bodies, collection orders, attempt counts, and feature +configurations. They also give A/B configurations matching scene content. A filter selects cases by a substring of the case id. An empty selection is an error. Declarations outside the filter are still validated. @@ -369,8 +425,8 @@ their object keys, consistent child and parent links, a tree that reaches every shape from the root, and references that point at present shapes. Validation runs before any timer starts. -The browser derives upload order from the snapshot and does not store it in the -snapshot. The root comes first, then descendants in the order of each shape's +The browser derives upload order from the snapshot. The root comes first, then +descendants in the order of each shape's child list. The browser computes the order and the fingerprint before the upload clock starts. The upload uses the same batch serializer as the editor, with structure, fill, and stroke data batched and path and SVG attributes handled per @@ -389,16 +445,214 @@ A load case renders a fresh scene and times module start, graphics start, upload, and the first render to `Full`. A pan or zoom case prepares the scene once, restores the starting view outside timing for each attempt, then times a camera session. The standard camera sessions use 20 view updates and a 100 ms -settle wait. Pan and zoom parameters vary by scene. +settling wait. Pan and zoom parameters vary by scene. + +## Renderer commands + +A renderer command accepts a command context or its Promise and returns a +Promise of the next context. The command context has four fields: + +| Field | Meaning | +| --- | --- | +| `:case` | Case descriptor, including the case id. | +| `:scene` | Prepared starting snapshot with concrete shapes and references. | +| `:view` | Current zoom scale, camera offsets, and optional viewport settings. | +| `:hooks` | Functions for clocks, scheduling, rendering, view changes, and the execution guard. | + +The `:scene` field describes the starting snapshot. Commands return view changes +through `:view` and apply renderer work through hooks. Every command context +supplies `:now`, `:check`, `:frame`, `:sleep`, `:render`, `:render-sync`, +`:render-from-cache`, `:set-view`, `:set-view-start`, and `:set-view-end` as +functions. The `:now` hook returns a clock reading in milliseconds. +The `:check` guard authorizes work by returning and reports cancellation, +deadline expiry, context loss, or a superseded owner +by throwing an error. The owner is the execution authorized to use the renderer. +Returned contexts retain the execution's `:check` guard; replacing it rejects +completion. + +### Declaring a command + +`defcommand` defines two functions from one body. The base function runs the +work; the function with a `*` suffix also records a measurement event. Measured +functions require an active command execution. Both functions resolve their +incoming context before running the body. A synchronous body +returns the next command context. + +```clojure +(ns example.renderer-commands + (:require + [benches.render-wasm.runtime.camera :as camera] + [benches.render-wasm.runtime.commands :as commands :refer [defcommand]] + [benches.render-wasm.runtime.protocol :as protocol])) + +(defcommand render-sync + "Renders the prepared scene synchronously." + [rtx] + ((get-in rtx [:hooks :render-sync])) + rtx) +``` + +`:pre` conditions validate arguments before work and timing in every build. +Input resolution, context validation, and argument validation precede the start +clock reading. Context validation checks the complete set of hooks. Output +validation and event recording follow the endpoint. + +The body duration is + +\[ +\Delta t = t_{\mathrm{end}} - t_{\mathrm{start}}, +\] + +where both clock readings use milliseconds from the same origin. A synchronous +command captures the endpoint immediately after its body returns. An +asynchronous command supplies an endpoint or captures it when `complete` runs. + +### Asynchronous completion + +The option `:async [complete fail]` supplies two completion callbacks to the +body. Their names belong to the declaration; the public command arguments stay +the same. The first callback call determines the outcome: + +- `(complete next-rtx)` succeeds with the next command context and captures the + endpoint at that call. +- `(complete next-rtx evidence)` succeeds with an explicit endpoint and raw + measurements when `evidence` supplies them. +- `(fail cause)` rejects the command with its cause and available partial trace. + +Successful completion requires a call to `complete`. A returned Promise forwards +its rejection to `fail`. The execution records diagnostics for repeated +completion calls while it is active. Its guard checks authorization at +completion, and the measured command records its event after output validation. + +The optional completion map has these fields: + +| Field | Meaning | +| --- | --- | +| `:end-ms` | Clock reading at the end of the work, using the same clock origin as the start. | +| `:metrics` | Namespaced metric keywords mapped to nonempty vectors of raw numbers. | +| `:trace` | Details of the work, such as traces of render calls and requested waits. | + +The command supplies `:command/elapsed`, its body duration in milliseconds. +`:metrics` supplies additional quantities. Command events preserve zero, +negative, `NaN`, and infinite numeric values as raw measurements. + +Camera completion illustrates the callback interface: + +```clojure +(defcommand pan + "Moves the view and completes when rendering reaches Full." + {:async [complete fail]} + [rtx {:keys [dx dy steps settle-ms]}] + (let [target (-> (:view rtx) + (update :x + dx) + (update :y + dy))] + (-> (protocol/start-camera! rtx {:settle-ms settle-ms}) + (protocol/animate-view! {:to target :steps steps}) + (protocol/finish-camera!) + (.then (fn [trace] + (complete (assoc rtx :view target) + {:end-ms (:end-ms trace) + :metrics {:renderer/call-elapsed + (mapv :duration-ms (:slices trace))} + :trace trace}))) + (.catch fail)))) +``` + +The final renderer call can return before the Promise callback runs. The +`:end-ms` field carries that renderer endpoint through the scheduling delay. +For example, renderer work lasting 7 ms followed by a 500 ms callback delay +still supplies a renderer duration of 7 ms. A clock reading in the later +callback would measure 507 ms. + +The provided `camera/pan` and `camera/pan*` commands accept explicit `:dx`, `:dy`, +`:steps`, and `:settle-ms`. They submit interpolated views on animation frames, +draw cached previews, wait for settling, and finish rendering to `Full`. They +resolve to the context with the final view. The measured function records: + +| Metric | Values in one event, in milliseconds | +| --- | --- | +| `:command/elapsed` | One duration from the start of the body through the renderer endpoint. | +| `:renderer/call-elapsed` | One duration per renderer call in the final drain. | +| `:renderer/cache-call-elapsed` | One duration per cached render call. | +| `:camera/finalization-to-full` | One duration from before ending the interaction through the renderer endpoint. | + +Its trace also contains individual render calls, view-update durations, waits, +and the final `:end-ms`. Each vector preserves the call order within its quantity. + +### Composition and execution + +A composite combines commands. `:measured-variant false` defines its base +function and selects measurements through the calls in its body: + +```clojure +(defcommand selected-pans + {:measured-variant false} + [rtx] + (-> rtx + (camera/pan* {:dx 10 :dy 0 :steps 2 :settle-ms 5}) + (camera/pan {:dx 0 :dy 5 :steps 2 :settle-ms 5}) + (camera/pan* {:dx -10 :dy 0 :steps 2 :settle-ms 5}))) +``` + +All three calls update the view and renderer. The first and third record +events. Command functions resolve incoming Promises, so threading passes each +updated context to the next command. + +A command execution runs a function from one command context to the next, +authorizes its work, and collects events in completion order. Given a prepared +command context `rtx`: + +```clojure +(commands/execute rtx selected-pans) +``` + +The returned Promise resolves to an outcome. A successful outcome contains +`:outcome :completed`, the next `:context`, and ordered `:events`. A failed +outcome contains `:outcome :failed`, `:error`, and the events recorded before +failure. Existing Errors retain their identity and partial trace. Other thrown +values become Errors with the original value in `:thrown`. An active execution +can also record repeated completion calls in `:diagnostics`. + +An event has this shape: + +```clojure +{:command :benches.render-wasm.runtime.camera/pan + :metrics {:command/elapsed [74] + :renderer/call-elapsed [7 7] + :renderer/cache-call-elapsed [2 2] + :camera/finalization-to-full [33]} + :trace {:end-ms 74}} +``` + +One event represents one measured command call. Its metric vectors describe +the individual values that call produced. The command execution owns this +history; its guard authorizes work for that execution. Pass the resulting +context to `commands/execute` to start another command execution. + +### Runtime code + +[`runtime/commands.clj`](runtime/commands.clj) declares commands; +[`runtime/commands.cljs`](runtime/commands.cljs) executes them. +[`runtime/camera.cljs`](runtime/camera.cljs) defines rendering and camera +operations. [`runtime/protocol.cljs`](runtime/protocol.cljs) defines render +drains, camera sessions, hook contracts, and trace fields. + +Case declarations such as [`scenes/rects.cljs`](scenes/rects.cljs) attach operation +bodies through [`declarations.cljc`](declarations.cljc). Node loads these +declarations and functions during catalogue collection. Renderer operations run +through the supplied hooks when the functions execute in the browser. +[`browser/bridge.cljs`](browser/bridge.cljs) constructs the runtime context for +case operations and prepares graphics contexts, snapshots, and uploads. Reused +cases restore their starting view outside interaction timing before each attempt. ## Measurement ### Preparation and measurement boundaries -The suite keeps slow setup out of timed code. This ties the numbers to renderer -work instead of test code. +Each duration has a defined start and endpoint. Setup and diagnostics run +around the interval whose work they prepare or describe. -Outside timers: +Work around those intervals includes: - scene build, snapshot validation, and fingerprinting; - upload order derivation; @@ -407,7 +661,7 @@ Outside timers: - module script import; - camera restore and the drain that prepares a warm case. -Inside timers: +The timed work includes: - module instantiation; - graphics context creation and initial size synchronization; @@ -445,15 +699,15 @@ runs a camera session: 1. start the camera interaction; 2. for each view update, wait for the next animation frame, apply the view, and present cached pixels; -3. wait for the settle interval; +3. wait for the requested settling time; 4. end the camera interaction; 5. render until `Full`. A pure pan can reuse cached textures while it updates which tiles are visible. -A zoom rebuilds and invalidates tile textures. The suite keeps the 100 ms settle -interval and records the requested and actual waits separately from renderer -work. The final drain includes the end of the interaction; the run also records -that call's own duration. +A zoom rebuilds and invalidates tile textures. Standard cases request a 100 ms +settling wait and record the requested and observed durations separately from +renderer calls. Camera finalization includes the call that ends the interaction; +the run also records that call's duration. ### Slices and metrics @@ -472,34 +726,39 @@ A fresh attempt reports one drain: A warm attempt reports the camera session and the final drain: - `cached-slices`: one record per gesture frame with `timestamp`, - `duration-ms` for the cached-preview call, and `view-and-preview-ms` for the + `duration-ms` for the cached preview call, and `view-and-preview-ms` for the view update plus preview call; - `slices`: the final drain, with the same shape as a fresh drain; - `interact-ms`: the whole camera session, from before the interaction starts through the final `Full`; - `active-ms`: from before the interaction starts to after the last cached - preview, or after an authored pause that follows it; + preview and its guard check, or after an explicit pause that follows it; - `settling-requested-ms` and `settling-actual-ms`: the declared and observed - settle spans; -- `set-view-end-ms`: the call that ends the interaction, on its own clock; -- `time-to-viewport-ready-ms` and `time-to-full-ms`: the final drain - boundaries; -- `last-input-to-full-ms`: from before the last view update to after the final - drain resolves. + settling waits; +- `set-view-end-ms`: elapsed time during the call that ends the interaction; +- `time-to-viewport-ready-ms` and `time-to-full-ms`: durations from camera + finalization through the first `ViewportReady` or `Full`, and through `Full`, + respectively; +- `last-input-to-full-ms`: from before the last view update to the endpoint of + the renderer call that returns `Full`. + +Both drains and camera sessions return `end-ms`, the clock reading immediately +after the final renderer call returns `Full`. Completion durations end at that +reading. Promise callbacks can run later as the browser schedules queued work. Both kinds of attempt report `renderer-call-total-ms`, the sum of the renderer call durations, computed after timing. -A metric names its unit, semantics, basis, and zero policy. A batch average -declares its call count, but one attempt still supplies one observation. Raw -slices keep zero durations and non-finite values; the metric layer marks values -that cannot supply an observation. +A saved run metric declares its unit, semantics, basis, and zero policy. A batch +average records its call count; one attempt supplies one observation. Raw slices +keep zero durations and non-finite values. The metric entry classifies the value +for analysis and records a reason when it is invalid. -### What completion means +### Renderer completion The renderer finishes in tile budgets. One renderer call returns a frame type, and the caller continues while the renderer reports `Partial` or -`ViewportReady`. The run completes when a call returns `Full`. `None` or an +`ViewportReady`. The drain completes when a call returns `Full`. `None` or an unknown value fails the attempt. | Value | Frame type | Meaning | @@ -509,15 +768,14 @@ unknown value fails the attempt. | 2 | `Full` | the renderer finished the current request | | 3 | `ViewportReady` | the visible viewport was submitted; work outside it may remain | -`Full` is a renderer submission boundary. It does not mean that the GPU has -finished or that pixels are visible. Skia and WebGL submit commands without -waiting for completion, and the suite has no signal for GPU completion or -displayed pixels. A call duration measures synchronous work inside the renderer -call plus any driver stall that happens on the calling thread. +`Full` marks renderer submission of the completed request. Skia and WebGL queue +GPU work, and GPU execution and display follow their own schedules. A call +duration measures synchronous work on the calling thread, including any driver +stall. These clock readings describe work through the renderer return. For scheduled calls, the browser passes the timestamp of the animation frame. -For immediate calls, it reads the clock. Each slice records its timestamp so -that call sequences can be reconstructed. +For immediate calls, it reads the clock. Each slice records its timestamp to +describe the call sequence. ### Graphics environment @@ -547,8 +805,7 @@ the GPU policy and environment signal under diagnostics. Default runs allow SwiftShader fallback and warn when they record software rendering. Comparisons check effective graphics settings and resolved launch arguments. Comparing hardware and software runs requires -`--allow-all-differences`: whether that comparison needs a narrower permission -is still an open question. +`--allow-all-differences`. ## Results and reports @@ -567,8 +824,9 @@ A run record contains: - the termination reason. A preparation records its case, fingerprint, effective graphics, and setup -diagnostics. An attempt records one outcome which can have some invalid or -absent metrics due to failures. We try to retain trace evidence after a failure. +diagnostics. An attempt records one outcome and its metric entries. Entries can +describe invalid values or missing measurements. Errors retain available +traces of render calls and waits after a failure. ### Failures and termination @@ -577,8 +835,8 @@ records a failed attempt and consumes the slot. A warm preparation failure or a warm attempt failure releases the prepared page and context and leaves that case's remaining slots unattempted. A browser or page failure closes the browser it started, abandons the current case, and starts a new browser for the next -case when possible. The runner performs no hidden retries and no adaptive -stopping. +case when possible. The run fixes its attempt count before execution and uses +each allocated slot at most once. A run terminates in one of four states: @@ -589,8 +847,9 @@ A run terminates in one of four states: failure stopped the case. - **interrupted**: `SIGINT` or `SIGTERM` stopped the run. -The runner persists partial records before it exits. Exit status is -authoritative; a saved record alone does not prove success. +The runner saves partial records before it exits. Use exit status to determine +whether the command completed successfully; saved records also describe work +that ended in failure. ### Summaries @@ -606,10 +865,10 @@ A summary reports, for each case and metric: categories; - the definitions and units of the metric. -Warmups and wholly failed attempts supply no observations. A completed attempt -can supply a valid value for one metric and an invalid value for another. The -report suppresses pooled estimates when valid observations use different metric -definitions. +Statistics use valid values from completed measured attempts. The record also +retains warmups and failures. A completed attempt can supply a valid value for +one metric and an invalid value for another. A pooled estimate combines values +across attempts and requires matching metric definitions. The default analysis settings are: @@ -627,16 +886,16 @@ can override them at analysis time. An estimate that cannot be computed keeps a reason and, when applicable, the minimum count it needs. The intervals are exploratory and conditional on the observed attempts. -Attempts within one run share evolving caches, so they are not independent -samples from a stable process. Repeated whole runs are the way to confirm a -finding. +Attempts within one run share evolving caches and can have dependent durations. +Repeated whole runs provide evidence about reproducibility across those cache +conditions. ### Comparisons A comparison pairs saved cases by identical case id unless `--cases A,B` selects one explicit baseline, candidate pair. The CLI checks the selector syntax before it loads either record, then resolves each selector by saved case -ID or exact name. +id or exact name. By default, a comparison requires equal: @@ -649,8 +908,8 @@ By default, a comparison requires equal: environment; - warmup count, timeout policy, and metric definitions. -Git revision, dirty state, run ids, timestamps, and each run's saved analysis -defaults are provenance. They do not block pairing. +Git revision, dirty state, run ids, timestamps, and saved analysis defaults +provide provenance for interpreting the paired cases. The comparison reports absolute and percentage median changes, with a bootstrap interval for each change. A zero baseline suppresses a percentage change. The @@ -659,8 +918,8 @@ all-differences permission allows raw values. ### Comparison permissions -The comparison records compatibility differences and never hides them. A -permission acknowledges a class of differences and labels each acknowledged +The comparison records every compatibility difference. A permission acknowledges +a class of differences and labels each acknowledged mismatch with the rule that allowed it. - `--allow-build-differences` acknowledges recorded build settings, effective @@ -672,13 +931,14 @@ mismatch with the rule that allowed it. strict. Missing or varying fingerprint evidence still needs the all-differences permission. - `--allow-all-differences` acknowledges every compatibility difference, - including mixed metric definitions. The report compares valid raw values - without converting units. Malformed records and unusable measurements remain - unusable. + including mixed metric definitions. The report compares valid raw values in + their recorded units. Record validation and measurement validity still govern + the analysis. Build and scene permissions combine by union. All-differences includes both. A report shows the selected permissions, every recorded difference, and the -rule that acknowledged it. Timing values never determine the exit status. +rule that acknowledged it. Exit status reports execution and compatibility +checks; the estimates describe the observed timing differences. ## A/B feature experiments @@ -686,7 +946,8 @@ An A/B experiment answers one question: how do the selected cases behave when a renderer feature is off versus on, with everything else held constant? The `ab` command takes the requested features, plans the configurations, builds -every configuration, runs every configuration, and compares every "edge". +every configuration, runs every configuration, and compares every edge: a pair +of configurations that differ by one feature. ### Planning @@ -703,9 +964,9 @@ A feature configuration is a set of enabled Cargo features. The planner: 6. emits every pair of configurations that differ in exactly one feature, from the configuration with that feature off to the configuration with it on. -The command requires at least one requested feature. For `N` features, the -comparison has `2^N` configurations and `N * 2^(N-1)` edges. The planner -compares each edge once, not once per execution order. For features +The command requires at least one requested feature. For $N$ features, the +comparison has $2^N$ configurations and $N 2^{N-1}$ edges. The planner +compares each edge once, in the direction from off to on. For features `[a, b]`, the configurations are `00, 01, 11, 10` and the edges are: ```text @@ -745,15 +1006,15 @@ pnpm run render-bench -- ab \ The feature names must exist in the renderer manifest. Placeholders appear here because the manifest decides which features are available. -A configuration that records attempts but fails does not stop the comparison. -Its failures and remaining slots are preserved. A build failure, a catalogue +After a configuration records attempts, scheduling continues with its recorded +failures and remaining slots. A build failure, a catalogue identity failure, or a run with zero attempts stops scheduling later configurations; those configurations are recorded as unattempted with a reason. `SIGINT` and `SIGTERM` save the available state and return 130 or 143. ### Feature-comparison output -The default home is +The default output directory is `target/performance/render-wasm/feature-comparisons//`. It contains: ```text @@ -788,24 +1049,23 @@ match that evidence, the difference remains unacknowledged. The comparison record also carries the edge context: the source and target configuration slugs and the expected feature facts. An edge is analysed when both endpoint records exist, even if one run recorded failed attempts. A missing -endpoint or an analysis error remains an explicit edge outcome. An edge that +endpoint or an analysis error produces an explicit edge outcome. An edge that retains an unacknowledged difference keeps its saved report, is marked failed, and makes the invocation exit 1. -Other build differences remain subject to `--allow-build-differences`. Scene -differences remain subject to `--allow-scene-differences`. Operation, view, +Other build differences require `--allow-build-differences`. Scene differences +require `--allow-scene-differences`. Operation, view, context, preparation, completion, batch size, graphics, runtime environment, and metric-definition differences stay strict unless `--allow-all-differences` is -selected. If the two configurations report the same effective feature set, the -edge remains in the comparison and is labelled with that fact. The suite does -not -infer a performance effect from the requested toggle alone. +selected. If two configurations report the same effective feature set, the +comparison includes their edge and labels it with that fact. Effective feature +sets describe the code settings that accompany the observed timings. The command returns 0 when every configuration completes, every edge analysis finishes, and every difference is acknowledged. It returns 1 for execution, persistence, or analysis failures, including unacknowledged differences. Edge -reports show raw times and the usual statistics. They describe the observed -difference; they do not prove that the feature caused it. +reports show raw times and statistics for the observed differences. Establishing +a causal effect requires experimental control and repeated measurements. ## How to run @@ -859,12 +1119,12 @@ Require hardware rendering for rectangle load, pan, and zoom in a GPU container: pnpm run render-bench -- run --filter rects --require-gpu ``` -For a functional smoke check, add `--warmups 0 --repetitions 1`. A successful +For a functional check, add `--warmups 0 --repetitions 1`. A successful run renders the scenes through `Full` and saves the vendor and renderer strings with `:software false` in each preparation's `:effective-graphics`. -An explicit `--output-dir` must not exist; the runner refuses to overwrite it. -The same rule applies to an A/B feature-comparison home. +Choose a new path for `--output-dir`. The runner rejects an existing output +directory. A/B feature comparisons apply the same rule. Compare two saved runs offline: @@ -939,21 +1199,22 @@ Each CLI command produces an outcome with an exit code, errors, log path, and optional report text. The runner prints the final output once and returns the numeric status. -Progress and diagnostics go to stderr. The default shows build phases, case +Progress and diagnostics go to stderr. The default shows build stages, case headings, and attempt progress. `-q` / `--quiet` shows results, warnings, and errors. `-v` / `--verbose` also shows debug details and raw build output. Quiet and verbose are mutually exclusive. These options apply to every command. `--color` forces terminal colours; `--no-color` disables them. Automatic colour uses a terminal stream when `NO_COLOR` is empty or absent and `TERM` is not -`dumb`. Results and file logs receive no logger colours. Child output is read as -lines with normalized line endings. Verbose terminal output prefixes and dims -these lines; file logs store them without those decorations. Terminal errors -include a bounded log tail when available. +`dumb`. Colour choices apply to terminal diagnostics. Result text and file logs +use plain text. The runner reads child output as lines and normalizes their +line endings. Verbose terminal output prefixes and dims these lines; file logs +store the lines as text. Terminal errors include a limited extract from the end +of the log when available. Files receive every log level, including debug output hidden from the console. Each build writes `build.log`; each A/B comparison writes -`feature-comparison.log`. The default run home is +`feature-comparison.log`. The default run directory is `target/performance/render-wasm/runs//`, which contains `run.log`, `run.transit`, and optional screenshots. @@ -963,9 +1224,9 @@ validated merged record for `merge`, every configuration and edge for `ab`, and a clean purge for `purge`. Exit 1 means a failed preparation, a failed attempt, an incomplete run, an invalid selection, incompatible merge inputs, an unacknowledged comparison difference, or a harness failure. Exits 130 and 143 -mean interruption by `SIGINT` and `SIGTERM` for `build`, `run` and `ab`. An -interrupted command prints no error for the aborted work; it prints a failed -write of its own record or log. +mean interruption by `SIGINT` and `SIGTERM` for `build`, `run` and `ab`. +Interruption uses those statuses to report aborted work. Errors writing the +command's record or log also produce diagnostics. `--warmups` defaults to 3, `--repetitions` to 10, `--timeout-ms` to 30000, and `--startup-timeout-ms` to 30000. `build`, a run that prepares its build, and an @@ -1010,14 +1271,15 @@ pass and 1 when they fail. ## Scope and limitations -The suite measures renderer phases inside Chromium exclusively, which means that -the numbers we generate end at the renderer submission boundary. In particular, -we **do not** measure memory, power, frame rate, frame drops, visual -correctness, network, or editor input-to-paint. +The suite records durations for setup, render calls, and camera movement inside +Chromium. +Rendering durations end when the renderer returns after submitting work. +Memory, power, frame rate, frame drops, visual correctness, network latency, +and editor input-to-paint each require measurements for those quantities. -In addition, a fresh context starts a new page, module, and WebGL context. The -browser process persists across cases, and the module script import runs before -the module clock, so the measurement is not cold. +A fresh case starts a new page, module, and WebGL2 graphics context. The browser +process persists across successful cases, and module script import precedes +module timing. These startup conditions define measurements for fresh cases. Attempts share evolving caches, so bootstrap intervals describe the observations within a run. Repeat whole runs to assess reproducibility, and judge whether the @@ -1030,7 +1292,7 @@ frontend/performance/benches/render_wasm/ ├── browser/ page entry and browser bridge ├── node/ CLI, build preparation, server, browser host, run, artefacts ├── report/ summaries, comparisons, statistics, formatting -├── runtime/ render drain and camera session protocol +├── runtime/ renderer commands, render drains, and camera sessions ├── scenes/ scene recipes and their cases ├── builder.cljc scene construction ├── declarations.cljc scene and case registry, schemas, seed derivation diff --git a/frontend/performance/benches/render_wasm/node/artifacts.cljs b/frontend/performance/benches/render_wasm/node/artifacts.cljs index c294336675..e0fb539bba 100644 --- a/frontend/performance/benches/render_wasm/node/artifacts.cljs +++ b/frontend/performance/benches/render_wasm/node/artifacts.cljs @@ -2,7 +2,7 @@ ;; License, v. 2.0. If a copy of the MPL was not distributed with this ;; file, You can obtain one at http://mozilla.org/MPL/2.0/. ;; -;; Copyright (c) KALEIDOS INC +;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns benches.render-wasm.node.artifacts "Managed renderer builds and output homes for benchmark runs and comparisons. diff --git a/frontend/performance/benches/render_wasm/node/build.cljs b/frontend/performance/benches/render_wasm/node/build.cljs index 5ed95102df..a52f2fc0f5 100644 --- a/frontend/performance/benches/render_wasm/node/build.cljs +++ b/frontend/performance/benches/render_wasm/node/build.cljs @@ -5,13 +5,17 @@ ;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns benches.render-wasm.node.build - "Prepares retained release artifacts under a single preparation deadline. + "These functions build the renderer and package it with the code that runs + benchmarks in Node and the browser. - The caller owns exclusive preparation for this checkout. The renderer's - lock protects Cargo and copying; CLJS compilation follows after it releases. - Preparation uses installed dependencies. Cargo may fetch missing crates and - the custom Skia binaries on a cold build. Every attempt retains its manifest - and compiler log, including failed attempts." + Each build gets its own directory under + frontend/target/performance/render-wasm/builds/. It contains the renderer + compiled to WebAssembly, a list of benchmarks that Node can load, and the + page and JavaScript that run them in a browser. + + Run `prepare` from Node in a checkout with the project's build tools and + dependencies installed. Stop other builds and any processes that rebuild the + renderer on file changes until it finishes." (:require ["fs" :as fs] ["path" :as path] @@ -27,7 +31,16 @@ [clojure.string :as str])) (defn ^:async git-provenance - "Captures revision and dirty state through the same bounded child boundary." + "Read the Git commit ID and whether the checkout containing `root` has local + changes, including staged changes and untracked files. + + `bounds` supplies :deadline, an absolute time in milliseconds since the Unix + epoch, and an optional :signal AbortSignal to cancel the work. + + Return a promise of a map containing :sha, the commit ID, and :dirty, which + is true when the checkout has local changes. If Git information is + unavailable, log a warning and return nil for both values. A timeout or + cancellation rejects the promise." [root bounds] (try (let [options (merge bounds {:command "git" :cwd root}) @@ -42,6 +55,9 @@ {:sha nil :dirty nil}))))) (defn- check-profile-marks + "Require :cljs-profile-marks to be false in the map describing the compiled + code. This flag indicates whether extra calls record timings for profiling; + those calls can affect benchmark measurements. Return the map or throw." [compiled] (when-not (false? (:cljs-profile-marks compiled)) (throw (ex-info "compiled benchmark enables profiling marks or omits its setting" @@ -49,8 +65,17 @@ compiled) (defn- ^:async validate-browser - "Imports the release bridge in Node without preparing a renderer. - The bridge's identity method returns the compiled identity over Transit." + "Checks that the JavaScript compiled for the browser belongs to the requested + build and has profiling disabled. + + build: supplies the expected :id and :browser-dir, the directory containing + browser.js. + root: the frontend directory. + bounds: supplies :deadline, an absolute time in milliseconds since the Unix + epoch, and an optional :signal AbortSignal for cancellation. + + Returns a promise of the code's build information, including :build-id and + :cljs-profile-marks. Rejects the promise if loading or checking the code fails." [build root bounds] (let [module-url (.-href (url/pathToFileURL (path/join (:browser-dir build) "browser.js"))) script "const {default: bridge} = await import(process.env.RENDER_BENCH_MODULE_URL); process.stdout.write(bridge.buildIdentity());" @@ -61,14 +86,35 @@ (check-profile-marks (identity/check (:id build) (transit/decode-str (:stdout result)))))) (defn ^:async prepare - "Returns a ready build reference after compilation and identity validation. + "Build the renderer and the programs that run benchmarks, and save the files + for later benchmark runs. - Options: frontend :root, :selection or :features, :build-timeout-ms - (300000), AbortSignal :signal, and optional :on-created callback. The callback - receives the new build reference once the output directory, log and initial - manifest exist. Storage errors have no recorded ID. Later errors carry - :build-id and :phase. An error also carries :manifest-error and is fatal when - the failed manifest cannot be saved. A retry always creates a new build." + Pass an options map with :root set to the frontend directory. If omitted, + the function locates that directory from the Node program's location. + + Cargo features choose which optional parts of the renderer to compile. Use + :selection {:default-features? true :features []} to enable Cargo's defaults. + Put additional feature names from render-wasm/Cargo.toml in the vector, and + set :default-features? to false to disable the declared defaults. The + :features option accepts the same map when :selection is absent. Omit both + options to use Cargo's defaults. + + :build-timeout-ms sets the time allowed for building, in milliseconds + (300000 by default). An optional `:signal AbortSignal` cancels the work. + An optional :on-created function receives a map with the build's ID and + paths after its directory, log and initial metadata file exist. This callback + runs before compilation. Await the promise from `prepare` before using the + compiled files. + + The promise resolves to a map with :id, the build's unique identifier; + :dir, its output directory; :log-path, its compiler log; and :manifest, + metadata describing the build and its settings. :cases-path points to the + JavaScript module that lists the benchmarks for Node. :browser-dir and + :renderer-dir contain the compiled programs, and :index-path points to the + HTML page. + + A failure rejects the promise and keeps any files already written for + inspection. Error data includes :build-id and :log-path when available." [{:keys [root features selection build-timeout-ms signal on-created] :or {build-timeout-ms options/default-build-timeout-ms}}] (when-not (and (js/Number.isSafeInteger build-timeout-ms) (pos? build-timeout-ms)) diff --git a/frontend/performance/benches/render_wasm/node/case_catalog.cljs b/frontend/performance/benches/render_wasm/node/case_catalog.cljs index a8b1eb2f02..16ace92583 100644 --- a/frontend/performance/benches/render_wasm/node/case_catalog.cljs +++ b/frontend/performance/benches/render_wasm/node/case_catalog.cljs @@ -2,13 +2,20 @@ ;; License, v. 2.0. If a copy of the MPL was not distributed with this ;; file, You can obtain one at http://mozilla.org/MPL/2.0/. ;; -;; Copyright (c) KALEIDOS INC +;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns benches.render-wasm.node.case-catalog - "Node-library entry points compiled with the browser's scene declarations. + "Node entry points of a compiled renderer benchmark build. - Export collect-cases as collectCases and build-identity as buildIdentity. - Each build's cases.cjs must keep its own JavaScript scope and registry." + The `:bench-render-wasm-catalog` build in `frontend/shadow-cljs.edn` names + this namespace as its compilation entry through `:exports`, which exposes + `collect-cases` to Node as `collectCases` and `build-identity` as + `buildIdentity`. Both functions use Transit-encoded strings at the Node + boundary. + + Requiring this namespace loads the benchmark scene declarations. Each build's + `cases.cjs` artifact has its own scene registry and compiled identity; load + the artifact of the build being queried." (:require [app.common.transit :as transit] [benches.render-wasm.cases :as cases] @@ -16,11 +23,37 @@ [benches.render-wasm.identity :as identity])) (defn collect-cases + "Collects the benchmark cases selected from this build. + + Args: + encoded-selection: A Transit-encoded map. The required :master-seed is an + integer from 0 through 4294967295 used to derive the seed of each case. + The optional :filter is a substring; a blank value selects all cases, + otherwise only case ids that contain it are collected. + + Returns: + A Transit-encoded map with :build-id, the identity compiled into this build, + and :cases, a non-empty vector of validated case descriptors. + Each descriptor has the case's :id and :scene, a :scene-description, its + :params and :scene-seed, and the execution settings used by the runner. + + Throws: + When the selection cannot be decoded, :master-seed is missing or not in + 0 through 4294967295, no cases are selected, or a registered scene or case + fails validation. Case collection failures carry a :type in their + exception data." [encoded-selection] (codec/encode-str {:build-id identity/build-id :cases (cases/collect-cases (transit/decode-str encoded-selection))})) (defn build-identity + "Returns the identity compiled into this build. + + Returns: + A Transit-encoded string. Decoding it yields a map with :build-id, the + identifier compiled into this artifact, defaulting to `unprepared`, and + :cljs-profile-marks, a boolean that records whether renderer profiling + marks were enabled at compile time." [] (codec/encode-str (identity/data))) diff --git a/frontend/performance/benches/render_wasm/node/catalog.cljs b/frontend/performance/benches/render_wasm/node/catalog.cljs index 6ff5026552..34a2a60b4c 100644 --- a/frontend/performance/benches/render_wasm/node/catalog.cljs +++ b/frontend/performance/benches/render_wasm/node/catalog.cljs @@ -2,7 +2,7 @@ ;; License, v. 2.0. If a copy of the MPL was not distributed with this ;; file, You can obtain one at http://mozilla.org/MPL/2.0/. ;; -;; Copyright (c) KALEIDOS INC +;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns benches.render-wasm.node.catalog "Loads case descriptors from a selected build, without the runner's registry." diff --git a/frontend/performance/benches/render_wasm/node/cli.cljs b/frontend/performance/benches/render_wasm/node/cli.cljs index e15191fc69..497d00e19d 100644 --- a/frontend/performance/benches/render_wasm/node/cli.cljs +++ b/frontend/performance/benches/render_wasm/node/cli.cljs @@ -2,7 +2,7 @@ ;; License, v. 2.0. If a copy of the MPL was not distributed with this ;; file, You can obtain one at http://mozilla.org/MPL/2.0/. ;; -;; Copyright (c) KALEIDOS INC +;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns benches.render-wasm.node.cli "Command-specific argument resolution and help for renderer benchmarks." diff --git a/frontend/performance/benches/render_wasm/node/cli/ab.cljs b/frontend/performance/benches/render_wasm/node/cli/ab.cljs index abc90ad406..f1b9db0251 100644 --- a/frontend/performance/benches/render_wasm/node/cli/ab.cljs +++ b/frontend/performance/benches/render_wasm/node/cli/ab.cljs @@ -2,7 +2,7 @@ ;; License, v. 2.0. If a copy of the MPL was not distributed with this ;; file, You can obtain one at http://mozilla.org/MPL/2.0/. ;; -;; Copyright (c) KALEIDOS INC +;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns benches.render-wasm.node.cli.ab "Options for comparing configurations of Cargo features." diff --git a/frontend/performance/benches/render_wasm/node/cli/build.cljs b/frontend/performance/benches/render_wasm/node/cli/build.cljs index b7bbf72305..86bb6a9e20 100644 --- a/frontend/performance/benches/render_wasm/node/cli/build.cljs +++ b/frontend/performance/benches/render_wasm/node/cli/build.cljs @@ -2,7 +2,7 @@ ;; License, v. 2.0. If a copy of the MPL was not distributed with this ;; file, You can obtain one at http://mozilla.org/MPL/2.0/. ;; -;; Copyright (c) KALEIDOS INC +;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns benches.render-wasm.node.cli.build "Options for preparing a benchmark build." diff --git a/frontend/performance/benches/render_wasm/node/cli/compare.cljs b/frontend/performance/benches/render_wasm/node/cli/compare.cljs index a409f2132d..1dcbdda56e 100644 --- a/frontend/performance/benches/render_wasm/node/cli/compare.cljs +++ b/frontend/performance/benches/render_wasm/node/cli/compare.cljs @@ -2,7 +2,7 @@ ;; License, v. 2.0. If a copy of the MPL was not distributed with this ;; file, You can obtain one at http://mozilla.org/MPL/2.0/. ;; -;; Copyright (c) KALEIDOS INC +;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns benches.render-wasm.node.cli.compare "Options for comparing saved benchmark records." diff --git a/frontend/performance/benches/render_wasm/node/cli/console.cljs b/frontend/performance/benches/render_wasm/node/cli/console.cljs index c5ef86b793..9a70f2b539 100644 --- a/frontend/performance/benches/render_wasm/node/cli/console.cljs +++ b/frontend/performance/benches/render_wasm/node/cli/console.cljs @@ -2,7 +2,7 @@ ;; License, v. 2.0. If a copy of the MPL was not distributed with this ;; file, You can obtain one at http://mozilla.org/MPL/2.0/. ;; -;; Copyright (c) KALEIDOS INC +;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns benches.render-wasm.node.cli.console "Console settings carried by resolved benchmark requests." diff --git a/frontend/performance/benches/render_wasm/node/cli/help.cljs b/frontend/performance/benches/render_wasm/node/cli/help.cljs index bb7ebe802c..7ab3f06c11 100644 --- a/frontend/performance/benches/render_wasm/node/cli/help.cljs +++ b/frontend/performance/benches/render_wasm/node/cli/help.cljs @@ -2,7 +2,7 @@ ;; License, v. 2.0. If a copy of the MPL was not distributed with this ;; file, You can obtain one at http://mozilla.org/MPL/2.0/. ;; -;; Copyright (c) KALEIDOS INC +;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns benches.render-wasm.node.cli.help "Options for benchmark CLI help." diff --git a/frontend/performance/benches/render_wasm/node/cli/merge.cljs b/frontend/performance/benches/render_wasm/node/cli/merge.cljs index 0ba0fa4ea2..77f8935adb 100644 --- a/frontend/performance/benches/render_wasm/node/cli/merge.cljs +++ b/frontend/performance/benches/render_wasm/node/cli/merge.cljs @@ -2,7 +2,7 @@ ;; License, v. 2.0. If a copy of the MPL was not distributed with this ;; file, You can obtain one at http://mozilla.org/MPL/2.0/. ;; -;; Copyright (c) KALEIDOS INC +;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns benches.render-wasm.node.cli.merge "Options for merging saved benchmark records." diff --git a/frontend/performance/benches/render_wasm/node/cli/purge.cljs b/frontend/performance/benches/render_wasm/node/cli/purge.cljs index 0d777a3f9b..eda1674596 100644 --- a/frontend/performance/benches/render_wasm/node/cli/purge.cljs +++ b/frontend/performance/benches/render_wasm/node/cli/purge.cljs @@ -2,7 +2,7 @@ ;; License, v. 2.0. If a copy of the MPL was not distributed with this ;; file, You can obtain one at http://mozilla.org/MPL/2.0/. ;; -;; Copyright (c) KALEIDOS INC +;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns benches.render-wasm.node.cli.purge "Options for purging managed benchmark artifacts." diff --git a/frontend/performance/benches/render_wasm/node/cli/registry.cljs b/frontend/performance/benches/render_wasm/node/cli/registry.cljs index 6dd2d62a6b..5d73f4a006 100644 --- a/frontend/performance/benches/render_wasm/node/cli/registry.cljs +++ b/frontend/performance/benches/render_wasm/node/cli/registry.cljs @@ -2,7 +2,7 @@ ;; License, v. 2.0. If a copy of the MPL was not distributed with this ;; file, You can obtain one at http://mozilla.org/MPL/2.0/. ;; -;; Copyright (c) KALEIDOS INC +;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns benches.render-wasm.node.cli.registry "Ordered command definitions for parsing, request schemas and help." diff --git a/frontend/performance/benches/render_wasm/node/cli/run.cljs b/frontend/performance/benches/render_wasm/node/cli/run.cljs index cc6a65a50c..eec5fb1baf 100644 --- a/frontend/performance/benches/render_wasm/node/cli/run.cljs +++ b/frontend/performance/benches/render_wasm/node/cli/run.cljs @@ -2,7 +2,7 @@ ;; License, v. 2.0. If a copy of the MPL was not distributed with this ;; file, You can obtain one at http://mozilla.org/MPL/2.0/. ;; -;; Copyright (c) KALEIDOS INC +;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns benches.render-wasm.node.cli.run "Options for a benchmark run and its selected build source." diff --git a/frontend/performance/benches/render_wasm/node/cli/shared.cljs b/frontend/performance/benches/render_wasm/node/cli/shared.cljs index 3401af6232..8ea71ba04c 100644 --- a/frontend/performance/benches/render_wasm/node/cli/shared.cljs +++ b/frontend/performance/benches/render_wasm/node/cli/shared.cljs @@ -2,7 +2,7 @@ ;; License, v. 2.0. If a copy of the MPL was not distributed with this ;; file, You can obtain one at http://mozilla.org/MPL/2.0/. ;; -;; Copyright (c) KALEIDOS INC +;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns benches.render-wasm.node.cli.shared "Value constraints and option groups shared by benchmark commands." diff --git a/frontend/performance/benches/render_wasm/node/commands.cljs b/frontend/performance/benches/render_wasm/node/commands.cljs index 09a336b273..13532bb033 100644 --- a/frontend/performance/benches/render_wasm/node/commands.cljs +++ b/frontend/performance/benches/render_wasm/node/commands.cljs @@ -2,7 +2,7 @@ ;; License, v. 2.0. If a copy of the MPL was not distributed with this ;; file, You can obtain one at http://mozilla.org/MPL/2.0/. ;; -;; Copyright (c) KALEIDOS INC +;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns benches.render-wasm.node.commands "Deferred execution entry for build, run and feature comparison commands." @@ -13,7 +13,6 @@ [benches.render-wasm.node.build :as build] [benches.render-wasm.node.execution :as execution] [benches.render-wasm.node.feature-comparison :as feature-comparison] - [benches.render-wasm.node.feature-plan :as feature-plan] [benches.render-wasm.node.outcome :as outcome] [benches.render-wasm.node.run :as run])) @@ -42,9 +41,9 @@ (defn- ^:async ab-command [{:keys [get-root signal services]} options] (let [root (get-root) - prepared (await (feature-plan/prepare root options signal))] + prepared (await (feature-comparison/prepare root options signal))] (if (:dry-run options) - (outcome/succeeded {:report (feature-plan/format-plan prepared)}) + (outcome/succeeded {:report (feature-comparison/format-plan prepared)}) (let [{:keys [home exit-code] :as result} (await (feature-comparison/execute root prepared signal {:services services}))] (cond-> result diff --git a/frontend/performance/benches/render_wasm/node/execution.cljs b/frontend/performance/benches/render_wasm/node/execution.cljs index 0fed79be09..ed439bece7 100644 --- a/frontend/performance/benches/render_wasm/node/execution.cljs +++ b/frontend/performance/benches/render_wasm/node/execution.cljs @@ -2,10 +2,21 @@ ;; License, v. 2.0. If a copy of the MPL was not distributed with this ;; file, You can obtain one at http://mozilla.org/MPL/2.0/. ;; -;; Copyright (c) KALEIDOS INC +;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns benches.render-wasm.node.execution - "Checked command execution, final publication and exit status." + "Validate benchmark command requests, run their handlers and print results. + + Both the CLI entry and the separately compiled renderer command entry use + this lifecycle. Callers supply a complete request and the handlers available + at that entry. Handlers own the command's files, processes and cleanup; this + module supplies cancellation, prints final reports and errors, and returns + a numeric status for the entry to use as its exit code. + + `fail` reports parsing, loading or decoding failures that occur before a + request reaches `execute`. Signal helpers also support work outside the + checked command lifecycle. Execution and failure reporting share console + settings; do not overlap calls that require different settings or streams." (:require [app.common.schema :as sm] [benches.render-wasm.errors :as errors :refer [attempt failure?]] @@ -15,8 +26,16 @@ [benches.render-wasm.node.request :as request])) (defn on-signals - "Installs SIGINT/SIGTERM callbacks receiving the signal name. Returns an - idempotent remover. The caller can abort its work and let output drain." + "Listen for process interrupts so the caller can stop work and finish cleanup. + + Args: + callback: Function called with the string `SIGINT` or `SIGTERM` each time the + Node process receives that signal while the listeners are installed. + + Returns: + An idempotent function that removes only the listeners installed by this + call. Call it when the work settles. This helper does not abort work, + wait for cleanup or choose an exit status." [callback] (let [interrupt (fn [] (callback "SIGINT")) terminate (fn [] (callback "SIGTERM"))] @@ -27,9 +46,25 @@ (.removeListener js/process "SIGTERM" terminate)))) (defn ^:async with-interruption - "Awaits `(f signal)` and returns its outcome. SIGINT/SIGTERM abort the signal - and select status 130/143. Interrupted outcomes retain fatal errors and partial - evidence. Thrown or rejected failures become outcomes." + "Run work with cooperative cancellation and preserve its result on interruption. + + The first SIGINT or SIGTERM aborts the supplied signal. Its reason is an + exception with `:signal` set to that signal's name. Later signals do not + replace the reason or force termination. The work must observe cancellation + and settle after its own cleanup; this wrapper imposes no timeout and cannot + preempt synchronous code. Listeners are removed after the work settles. + + Args: + f: Function called once with an AbortSignal. Return an outcome map, or a + promise of one, containing `:exit-code`, an `:errors` vector and + `:log-path` (nil when there is no log). Additional fields may retain partial + results, file paths or a final `:report` string. + + Returns: + A promise of the outcome. Thrown or rejected failures become outcomes with + exit code 1. After interruption, set the exit code to 130 for SIGINT or 143 + for SIGTERM and retain only errors recognized by `errors/fatal?`. Other + outcome fields are preserved. This function does not print the outcome." [f] (let [controller (js/AbortController.) signal (.-signal controller) @@ -92,6 +127,7 @@ (subs message 0 (min 1024 (count message))) "\n"))))) (defn- finish + "Print the optional report and errors without retrying failed report writes." [result local] (binding [log/*log-path* (:log-path result)] (let [finished (if-some [report (:report result)] @@ -105,8 +141,24 @@ (:exit-code finished))))) (defn fail - "Prints a failure and returns its numeric status. Optional resolved console - settings and local streams configure its output." + "Report a failure that occurred before checked command execution could begin. + + Use this for argument parsing, module loading or request decoding failures. + Do not report a failure again after `execute` has returned its numeric status. + + Args: + cause: Error or thrown value to report. Error data may supply a log path, + usage text, validation details or comparison differences for diagnosis. + console: Trusted console settings with `:level` (`:info`, `:warn`, `:debug`) + and `:color` (`:auto`, `:always`, `:never`). An empty map uses defaults. + Do not take these settings from a request that failed validation. + local: Optional `:stdout` and `:stderr` writable streams with a Node-style + `write` method. Omitted streams use the process streams. + + Returns: + The number 1 after attempting to write the error to stderr and any log + named by its error data. Reporting failures get one bounded stderr fallback + attempt. The one-argument form uses default console settings and streams." ([cause] (fail cause {} {})) ([cause console local] @@ -114,10 +166,50 @@ (finish (outcome/failed (errors/as-failure cause) {}) local))) (defn ^:async execute - "Checks a complete received request, awaits its supported handler and returns - a numeric exit status after final publication. Handlers receive local signal, - root access and services plus resolved options. Local settings may supply - streams, a root thunk and services." + "Validate a command request, await its handler and print its final outcome. + + Check the request without adding defaults or coercing values. Invalid + requests use default console settings; valid requests choose their console + settings before handler lookup and command policy checks. Unsupported + commands and policy failures are reported without invoking a handler or + looking up the frontend root. + + For `run` and non-dry-run `ab` commands, `:require-gpu` requires the environment + variable `PENPOT_GPU` to equal the string `1` before invoking the handler. + This checks an environment prerequisite, not actual GPU availability. + + Args: + received: Complete request map with a string `:command`, resolved + `:console` settings and command-specific `:options`. Obtain it from + `cli/parse-resolve` or decode it from the CLI's Transit request. Missing + required fields and unknown fields fail validation. + handlers: Map of supported command-name strings to functions called as + `(handler context options)`. Return an outcome or a promise of one, with + `:exit-code`, an `:errors` vector and `:log-path` (nil when there is no log). + An optional `:report` string supplies all final stdout text. Other + result fields do not generate output. Handlers own progress logging + and must await their work and cleanup before returning. + local: Optional map containing `:stdout` and `:stderr` writable streams, + a zero-argument `:get-root` function returning the frontend directory, + and a `:services` map of handler-specific overrides. The context contains + an AbortSignal at `:signal`, plus `:get-root` and `:services` when supplied. + Root lookup and service calls remain the handler's responsibility. These + functions, streams and services are local, not part of the request. + + Returns: + A promise of a numeric exit status after the handler settles and final + output writes have been attempted. Print `:report` once, when present, + followed by errors on stderr, using the outcome's log path for file output. + No report means no final stdout text. Output writes do not wait for streams + to drain. The two-argument form uses process streams and supplies no root + function or services. + + Validation, policy and handler failures produce status 1. Interruption + follows `with-interruption`: await cleanup, preserve partial result fields + and fatal errors, and return 130 for SIGINT or 143 for SIGTERM. A report or + error-reporting failure changes other statuses to 1, with one bounded + stderr fallback if error reporting fails. Interrupted statuses take + precedence. Failed report writes are not retried." ([received handlers] (execute received handlers {})) ([received handlers local] diff --git a/frontend/performance/benches/render_wasm/node/feature_comparison.cljs b/frontend/performance/benches/render_wasm/node/feature_comparison.cljs index b12d6b8940..21dfe2e5ad 100644 --- a/frontend/performance/benches/render_wasm/node/feature_comparison.cljs +++ b/frontend/performance/benches/render_wasm/node/feature_comparison.cljs @@ -5,25 +5,40 @@ ;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns benches.render-wasm.node.feature-comparison - "Serial feature A/B orchestration and persistence of comparisons." + "Plan and run benchmarks for combinations of optional Cargo renderer features. + + `prepare` reads Cargo metadata and collects the selected benchmark cases. + Call `format-plan` to inspect that data, or await `execute` to build and run + each configuration in sequence. Each configuration disables Cargo defaults. + An edge is a comparison between two configurations that differ by one + requested feature, from disabled to enabled. + + Execution saves progress, run records, logs and edge reports for inspection + after completion, failure or cancellation. Builds use the checkout's renderer + and benchmark sources. Keep other renderer builds and watchers stopped until + execution finishes." (:require - ["fs" :as fs] - ["node:timers/promises" :as timers] - ["path" :as path] - [benches.render-wasm.errors :as errors :refer [attempt failure?]] - [benches.render-wasm.node.artifacts :as artifacts] - [benches.render-wasm.node.build :as build] - [benches.render-wasm.node.catalog :as catalog] - [benches.render-wasm.node.files :as files] - [benches.render-wasm.node.log :as log] - [benches.render-wasm.node.outcome :as outcome] - [benches.render-wasm.node.process :as process] - [benches.render-wasm.node.run :as run] - [benches.render-wasm.node.runs :as runs] - [benches.render-wasm.node.time :as time] - [benches.render-wasm.report.compare :as compare] - [benches.render-wasm.result :as result] - [clojure.set :as set])) + ["fs" :as fs] + ["node:timers/promises" :as timers] + ["path" :as path] + [benches.render-wasm.cases :as cases] + [benches.render-wasm.errors :as errors :refer [attempt failure?]] + [benches.render-wasm.feature-plan :as feature-plan] + [benches.render-wasm.node.artifacts :as artifacts] + [benches.render-wasm.node.build :as build] + [benches.render-wasm.node.catalog :as catalog] + [benches.render-wasm.node.files :as files] + [benches.render-wasm.node.log :as log] + [benches.render-wasm.node.options :as options] + [benches.render-wasm.node.outcome :as outcome] + [benches.render-wasm.node.process :as process] + [benches.render-wasm.node.run :as run] + [benches.render-wasm.node.runs :as runs] + [benches.render-wasm.node.time :as time] + [benches.render-wasm.report.compare :as compare] + [benches.render-wasm.result :as result] + [clojure.set :as set] + [clojure.string :as str])) (def ^:private warnings ["Timing values do not determine success." @@ -36,11 +51,26 @@ :execute-run run/execute}) (defn create-home - "Creates the feature-comparison home directory + "Creates a directory and an empty log for a new feature comparison. - The default lives under the managed feature-comparisons directory. - An explicit directory resolves from the caller's cwd and must not exist. - Paths inside the managed builds tree are rejected." + Args: + root: The frontend directory. + comparison-id: Contains 1 to 128 ASCII letters, digits, underscores or + hyphens, starting with a letter or digit. + output-dir: With nil `output-dir`, uses + `target/performance/render-wasm/feature-comparisons/` under + root. An explicit path resolves from the Node process's working directory. + The chosen directory must not exist and must not be inside the managed + builds tree, including through symlink ancestors. + + Returns: + A map containing `:comparison-id`, `:dir`, `:log-path` and `:record-path`. + The log is `feature-comparison.log`; `:record-path` names the future + `feature-comparison.transit` file, which this function does not write. + + Throws: + Invalid identifiers, rejected paths and filesystem failures throw an error; + path and filesystem errors include `:path` when available." [root comparison-id output-dir] (artifacts/check-home-id comparison-id "invalid feature comparison id") (let [default-dir (path/join (artifacts/managed-root root) @@ -53,7 +83,21 @@ :comparison-id comparison-id))) (defn write-record - "Validates schema:feature-comparison and atomically replaces the saved file." + "Validates and saves a feature-comparison record. + + Encodes `record` as Transit after schema validation, then atomically replaces + that file so readers see either the previous complete record or the new one. + + Args: + home: Supplies `:record-path` in an existing directory. + record: The feature-comparison record. + + Returns: + The record. + + Throws: + Validation, write and rename failures throw without replacing the previous + record." [home record] (files/atomic-write (:record-path home) (result/encode record)) record) @@ -95,6 +139,7 @@ (some #(when (= slug (:slug %)) %) (:configurations record))) (defn- persist + "Save updated progress before exposing it as the current and last saved record." [state update] (let [next (assoc (update @(:record-ref state)) :updated-at (time/now)) saved (try @@ -503,9 +548,13 @@ :unacknowledged (unacknowledged-count comparison)}))) (defn- record-edge-outcome - "Records the result of analyzing an edge and returns the first error seen so - far. Throws a fatal failure after the edge and the remaining edges are - recorded." + "Records the result of analyzing an edge. + + Returns: + The first error seen so far. + + Throws: + A fatal failure after the edge and the remaining edges are recorded." [state edge result error] (cond (and (failure? result) (errors/fatal? result)) @@ -614,14 +663,49 @@ result))))) (defn ^:async execute - "Executes a prepared feature comparison and owns builds, runs and reports. + "Build and benchmark every prepared feature configuration, then compare runs. - - prepared: node.feature-plan/prepare output. - - settings: accepts an optional :services map with :prepare-build, :collect-catalog, - :create-run-home and :execute-run overrides. + The caller must handle `:dry-run` by calling `format-plan` instead; this function + always executes. - Returns the comparison record, home, log path, exit code and the failures - in :errors. The saved record reflects the last successful atomic write." + Create a fresh comparison directory and save its initial record before the + first build. Build and run configurations serially, checking each compiled + catalog against the prepared cases. Runs live in `configs/` under the + comparison directory; edge reports live in `comparisons/__.transit`. + Build artifacts remain in the managed builds directory. + + Timing differences alone do not cause failure. Unacknowledged differences + in build, scene or other comparison metadata do cause an edge to fail. + The prepared options `:allow-build-differences` and `:allow-scene-differences` + acknowledge their respective categories; `:allow-all-differences` acknowledges + every compatibility mismatch. Reports retain acknowledged differences. + + Build or catalog failures stop later configurations. A failed run may allow + later configurations when it retains attempts and usable evidence; missing + evidence, GPU failure or zero attempts stop them. Cancellation skips remaining + work. Files already written remain for diagnosis. + + Args: + root: The frontend directory. + prepared: The result of `prepare`. + signal: An optional AbortSignal that cancels execution. + settings: May contain `:services` overrides with the same contracts as the + defaults: `:prepare-build` uses `node.build/prepare`, `:collect-catalog` + uses `node.catalog/collect`, `:create-run-home` uses `node.runs/create-home`, + and `:execute-run` uses `node.run/execute`. Omitted services use + `default-services`. + + Returns: + Await the returned promise for a map with `:comparison`, the last + successfully saved progress record; `:home`, the comparison ID and paths; + `:log-path`; `:errors`, retained error objects; and `:exit-code`. The exit + code is 0 only when all configurations and edges complete, 1 on failure, + or 130/143 on interruption. + + If storage fails, the returned `:comparison` reflects the last successful + atomic write, or nil if no record was saved; `:home` is nil if directory + creation failed. Inspect the saved statuses and `:errors` together, since + `:errors` is not an exhaustive list of failed configurations or edges." [root prepared signal settings] (let [services (merge default-services (:services settings)) comparison-id (str (random-uuid)) @@ -631,3 +715,100 @@ (await (execute-comparison (create-comparison-state root prepared signal services home comparison-id)))))) + + + +(defn- get-root-features + "Read feature definitions for cwd/Cargo.toml from Cargo metadata JSON." + [cwd text] + (let [metadata (js->clj (js/JSON.parse text)) + manifest (fs/realpathSync (path/join cwd "Cargo.toml")) + package (first (filter #(= manifest (fs/realpathSync + (path/resolve cwd (get % "manifest_path")))) + (get metadata "packages")))] + (when-not package + (throw (ex-info "Cargo metadata does not contain the renderer manifest" + {:type ::missing-package :manifest-path manifest}))) + (get package "features"))) + +(defn ^:async prepare + "Collect and validate a feature comparison plan without building or running it. + + Run `cargo metadata --offline --locked --no-deps` in `render-wasm` with the + inherited environment. Cargo and its offline metadata must be available. + `:build-timeout-ms` bounds that subprocess in milliseconds, not the entire + planning operation. + Require empty declared Cargo defaults and known requested feature names; + Cargo remains responsible for resolving feature activation during builds. + + Planning creates no comparison output and starts no build, server or browser, + including for a dry-run. + + Args: + root: The frontend directory, with render-wasm as its sibling. + opts: Resolved CLI options, including defaults. `:features` supplies the + requested Cargo feature names, `:seed` supplies the master scene seed, + and `:filter` selects case IDs by substring. `:max-features` limits the + requested feature count. Execution requires `:confirm` for more than + eight configurations or a blank filter; `:dry-run` bypasses that + confirmation requirement only. + + An optional `:output-dir` resolves from the Node process's working + directory and must not exist. Check it without creating or reserving it. + `execute` checks it again and also rejects paths inside the managed + builds tree. + signal: An optional AbortSignal that cancels the process. + + Returns: + Await the promise for a map with `:plan`, `:cases`, `:output-dir` and + `:options`. The `:plan` map contains `:features` in sorted order, + `:configurations` with a `:slug` and Cargo `:selection`, and `:edges` with + `:from` and `:to` slugs and a `:feature`. Slugs are bit strings in sorted + feature order: `0` means disabled, `1` enabled. The `:cases` vector holds + case descriptions collected once. `:output-dir` is the absolute explicit + path or nil. `:options` contains the supplied options with normalized + features and output path. Pass this map to `format-plan` or `execute`. + + Throws: + Invalid options or paths, Cargo failures or cancellation, unsupported + manifest features, and case collection failures reject the promise. + An empty case selection is an error." + [root opts signal] + (let [plan (feature-plan/plan (:features opts) opts) + output-dir (artifacts/check-output-dir (:output-dir opts)) + cwd (path/resolve root ".." "render-wasm") + metadata (await (process/run + {:command "cargo" + :args ["metadata" "--offline" "--locked" "--no-deps" "--format-version" "1"] + :cwd cwd + :deadline (process/deadline (:build-timeout-ms opts options/default-build-timeout-ms)) + :signal signal}))] + (feature-plan/check-manifest (:features plan) (get-root-features cwd (:stdout metadata))) + {:plan plan + :cases (cases/collect-cases {:master-seed (:seed opts) :filter (:filter opts)}) + :output-dir output-dir + :options (assoc opts :features (:features plan) :output-dir output-dir)})) + +(defn format-plan + "Return a text summary of the map produced by `prepare`. + + This function neither collects cases again nor reads the filesystem, + starts processes or executes benchmarks. + + Returns: + Feature names, configurations, comparison edges, case count, sampling + counts and configured timeouts." + [{:keys [plan cases options]}] + (str "Feature comparison plan\n" + "Features: " (str/join ", " (:features plan)) "\n" + "Configurations (Cargo defaults disabled):\n" + (str/join "\n" (map #(str " " (:slug %) ": " + (str/join ", " (get-in % [:selection :features]))) + (:configurations plan))) "\n" + "Edges:\n" + (str/join "\n" (map #(str " " (:from %) " -> " (:to %) ": " (:feature %)) + (:edges plan))) "\n" + "Cases: " (count cases) "\n" + "Attempts per case: " (:warmups options) " warmups, " (:repetitions options) " measured\n" + "Timeouts (ms): attempt " (:timeout-ms options) + ", build " (:build-timeout-ms options) ", startup " (:startup-timeout-ms options) "\n")) diff --git a/frontend/performance/benches/render_wasm/node/feature_plan.cljs b/frontend/performance/benches/render_wasm/node/feature_plan.cljs deleted file mode 100644 index 6ed9dc271c..0000000000 --- a/frontend/performance/benches/render_wasm/node/feature_plan.cljs +++ /dev/null @@ -1,66 +0,0 @@ -;; This Source Code Form is subject to the terms of the Mozilla Public -;; License, v. 2.0. If a copy of the MPL was not distributed with this -;; file, You can obtain one at http://mozilla.org/MPL/2.0/. -;; -;; Copyright (c) KALEIDOS SUBSIDIARY SL - -(ns benches.render-wasm.node.feature-plan - "Plan comparisons based on cargo metadata and cases collected." - (:require - ["fs" :as fs] - ["path" :as path] - [benches.render-wasm.cases :as cases] - [benches.render-wasm.feature-plan :as feature-plan] - [benches.render-wasm.node.artifacts :as artifacts] - [benches.render-wasm.node.options :as options] - [benches.render-wasm.node.process :as process] - [clojure.string :as str])) - -(defn- get-root-features - [cwd text] - (let [metadata (js->clj (js/JSON.parse text)) - manifest (fs/realpathSync (path/join cwd "Cargo.toml")) - package (first (filter #(= manifest (fs/realpathSync - (path/resolve cwd (get % "manifest_path")))) - (get metadata "packages")))] - (when-not package - (throw (ex-info "Cargo metadata does not contain the renderer manifest" - {:type ::missing-package :manifest-path manifest}))) - (get package "features"))) - -(defn ^:async prepare - "Accepts the frontend root and resolved CLI options. Reads Cargo metadata in - render-wasm with the inherited environment, checks an optional fresh output - path, and collects selected case data once." - [root opts signal] - (let [plan (feature-plan/plan (:features opts) opts) - output-dir (artifacts/check-output-dir (:output-dir opts)) - cwd (path/resolve root ".." "render-wasm") - metadata (await (process/run - {:command "cargo" - :args ["metadata" "--offline" "--locked" "--no-deps" "--format-version" "1"] - :cwd cwd - :deadline (process/deadline (:build-timeout-ms opts options/default-build-timeout-ms)) - :signal signal}))] - (feature-plan/check-manifest (:features plan) (get-root-features cwd (:stdout metadata))) - {:plan plan - :cases (cases/collect-cases {:master-seed (:seed opts) :filter (:filter opts)}) - :output-dir output-dir - :options (assoc opts :features (:features plan) :output-dir output-dir)})) - -(defn format-plan - "Formats prepared data without collecting again or reading the filesystem." - [{:keys [plan cases options]}] - (str "Feature comparison plan\n" - "Features: " (str/join ", " (:features plan)) "\n" - "Configurations (Cargo defaults disabled):\n" - (str/join "\n" (map #(str " " (:slug %) ": " - (str/join ", " (get-in % [:selection :features]))) - (:configurations plan))) "\n" - "Edges:\n" - (str/join "\n" (map #(str " " (:from %) " -> " (:to %) ": " (:feature %)) - (:edges plan))) "\n" - "Cases: " (count cases) "\n" - "Attempts per case: " (:warmups options) " warmups, " (:repetitions options) " measured\n" - "Timeouts (ms): attempt " (:timeout-ms options) - ", build " (:build-timeout-ms options) ", startup " (:startup-timeout-ms options) "\n")) diff --git a/frontend/performance/benches/render_wasm/node/request.cljs b/frontend/performance/benches/render_wasm/node/request.cljs index d7e62adad6..7324b32941 100644 --- a/frontend/performance/benches/render_wasm/node/request.cljs +++ b/frontend/performance/benches/render_wasm/node/request.cljs @@ -2,7 +2,7 @@ ;; License, v. 2.0. If a copy of the MPL was not distributed with this ;; file, You can obtain one at http://mozilla.org/MPL/2.0/. ;; -;; Copyright (c) KALEIDOS INC +;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns benches.render-wasm.node.request "Resolved benchmark requests, sender defaults and receiver validation." diff --git a/frontend/performance/benches/render_wasm/node/runner.cljs b/frontend/performance/benches/render_wasm/node/runner.cljs index 218fd643ec..3d659fff2b 100644 --- a/frontend/performance/benches/render_wasm/node/runner.cljs +++ b/frontend/performance/benches/render_wasm/node/runner.cljs @@ -2,7 +2,7 @@ ;; License, v. 2.0. If a copy of the MPL was not distributed with this ;; file, You can obtain one at http://mozilla.org/MPL/2.0/. ;; -;; Copyright (c) KALEIDOS INC +;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns benches.render-wasm.node.runner "Command-line entry for renderer benchmarks and saved results." diff --git a/frontend/performance/benches/render_wasm/node/server.cljs b/frontend/performance/benches/render_wasm/node/server.cljs index 0ab9b32424..3d049330a5 100644 --- a/frontend/performance/benches/render_wasm/node/server.cljs +++ b/frontend/performance/benches/render_wasm/node/server.cljs @@ -2,7 +2,7 @@ ;; License, v. 2.0. If a copy of the MPL was not distributed with this ;; file, You can obtain one at http://mozilla.org/MPL/2.0/. ;; -;; Copyright (c) KALEIDOS INC +;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns benches.render-wasm.node.server "An owned loopback HTTP server for one immutable renderer build." diff --git a/frontend/test/performance_tests/render_wasm/artifacts_test.cljs b/frontend/test/performance_tests/render_wasm/artifacts_test.cljs index c7b29cfc21..64d478f8d7 100644 --- a/frontend/test/performance_tests/render_wasm/artifacts_test.cljs +++ b/frontend/test/performance_tests/render_wasm/artifacts_test.cljs @@ -2,7 +2,7 @@ ;; License, v. 2.0. If a copy of the MPL was not distributed with this ;; file, You can obtain one at http://mozilla.org/MPL/2.0/. ;; -;; Copyright (c) KALEIDOS INC +;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns performance-tests.render-wasm.artifacts-test (:require diff --git a/frontend/test/performance_tests/render_wasm/feature_comparison_test.cljs b/frontend/test/performance_tests/render_wasm/feature_comparison_test.cljs index 9b56250d32..1338d58360 100644 --- a/frontend/test/performance_tests/render_wasm/feature_comparison_test.cljs +++ b/frontend/test/performance_tests/render_wasm/feature_comparison_test.cljs @@ -6,26 +6,25 @@ (ns performance-tests.render-wasm.feature-comparison-test (:require - ["fs" :as fs] - ["node:timers/promises" :as timers] - ["os" :as os] - ["path" :as path] - [benches.render-wasm.cases :as cases] - [benches.render-wasm.declarations :as decl] - [benches.render-wasm.errors :as errors] - [benches.render-wasm.feature-plan :as feature-plan] - [benches.render-wasm.node.build :as build] - [benches.render-wasm.node.execution :as execution] - [benches.render-wasm.node.feature-comparison :as feature-comparison] - [benches.render-wasm.node.feature-plan :as node-plan] - [benches.render-wasm.node.files :as files] - [benches.render-wasm.node.log :as log] - [benches.render-wasm.node.process :as process] - [benches.render-wasm.node.request :as request] - [benches.render-wasm.result :as result] - [cljs.test :as t :include-macros true] - [clojure.string :as str] - [frontend-tests.helpers.mock :as mock])) + ["fs" :as fs] + ["node:timers/promises" :as timers] + ["os" :as os] + ["path" :as path] + [benches.render-wasm.cases :as cases] + [benches.render-wasm.declarations :as decl] + [benches.render-wasm.errors :as errors] + [benches.render-wasm.feature-plan :as feature-plan] + [benches.render-wasm.node.build :as build] + [benches.render-wasm.node.execution :as execution] + [benches.render-wasm.node.feature-comparison :as feature-comparison] + [benches.render-wasm.node.files :as files] + [benches.render-wasm.node.log :as log] + [benches.render-wasm.node.process :as process] + [benches.render-wasm.node.request :as request] + [benches.render-wasm.result :as result] + [cljs.test :as t :include-macros true] + [clojure.string :as str] + [frontend-tests.helpers.mock :as mock])) (def ^:private temporary-root (atom nil)) @@ -153,7 +152,7 @@ (let [output (path/join @temporary-root "new" "comparison") opts (assoc (planning-options) :output-dir output) before (vec (fs/readdirSync (renderer-root))) - ready (await (node-plan/prepare (frontend-root) opts nil))] + ready (await (feature-comparison/prepare (frontend-root) opts nil))] (t/is (= ["a" "b"] (get-in ready [:plan :features]))) (t/is (= (cases/collect-cases {:master-seed 42 :filter "rects/load"}) (:cases ready))) (t/is (= (assoc opts :features ["a" "b"]) @@ -162,7 +161,7 @@ (t/is (not (fs/existsSync (path/join @temporary-root "new")))) (t/is (= before (vec (fs/readdirSync (renderer-root))))) (t/is (empty? (fs/readdirSync (frontend-root)))) - (let [text (node-plan/format-plan ready)] + (let [text (feature-comparison/format-plan ready)] (doseq [part ["a, b" "00" "01" "11" "10" "00 -> 10: a" "Cases: 1" "0 warmups, 1 measured" "attempt 2000, build 10000, startup 3000"]] (t/is (str/includes? text part)))))) @@ -179,7 +178,7 @@ (t/is (= (renderer-root) (:cwd request))) (t/is (not (contains? request :env))) (js/Promise.resolve (metadata-output {"default" [] "a" [] "b" ["a"]})))} - (let [ready (await (node-plan/prepare (frontend-root) (planning-options) nil))] + (let [ready (await (feature-comparison/prepare (frontend-root) (planning-options) nil))] (t/is (= ["a" "b"] (get-in ready [:plan :features]))))))) (t/deftest ^:async collected-custom-case-data-and-order-survive-planning @@ -196,8 +195,8 @@ :preparation {:reset :outside-timing :data [1 2]} :run! (fn [_] (throw (js/Error. "planning must not run an operation")))})) (try - (let [ready (await (node-plan/prepare (frontend-root) - (assoc (planning-options) :filter "feature-plan-fixture/") nil)) + (let [ready (await (feature-comparison/prepare (frontend-root) + (assoc (planning-options) :filter "feature-plan-fixture/") nil)) data (:cases ready)] (t/is (= [:feature-plan-fixture/second :feature-plan-fixture/first] (mapv :id data))) (t/is (= (cases/collect-cases {:master-seed 42 :filter "feature-plan-fixture/"}) data)) @@ -218,10 +217,10 @@ (await (mock/with-mocks* {process/run (fn [_] (js/Promise.resolve (metadata-output definitions)))} (t/is (= message (await (error-message - (node-plan/prepare (frontend-root) (planning-options) nil)))))))) + (feature-comparison/prepare (frontend-root) (planning-options) nil)))))))) (t/is (= "no benchmark cases match the selection" - (await (error-message (node-plan/prepare (frontend-root) - (assoc (planning-options) :filter "missing-case") nil)))))) + (await (error-message (feature-comparison/prepare (frontend-root) + (assoc (planning-options) :filter "missing-case") nil)))))) (t/deftest ^:async invalid-input-and-existing-output-fail-before-cargo (write-cargo-fixture) @@ -229,18 +228,18 @@ (mock/with-mocks* {process/run (fn [_] (throw (js/Error. "unexpected Cargo invocation")))} (t/is (= "output directory already exists" - (await (error-message (node-plan/prepare (frontend-root) - (assoc (planning-options) :output-dir (frontend-root)) nil))))) + (await (error-message (feature-comparison/prepare (frontend-root) + (assoc (planning-options) :output-dir (frontend-root)) nil))))) (t/is (= "ab requires at least one requested feature" - (await (error-message (node-plan/prepare (frontend-root) - (assoc (planning-options) :features []) nil)))))))) + (await (error-message (feature-comparison/prepare (frontend-root) + (assoc (planning-options) :features []) nil)))))))) (t/deftest ^:async cancellation-reaches-the-process-boundary (write-cargo-fixture) (let [controller (js/AbortController.)] (.abort controller "SIGINT") (t/is (str/includes? (await (error-message - (node-plan/prepare (frontend-root) (planning-options) (.-signal controller)))) + (feature-comparison/prepare (frontend-root) (planning-options) (.-signal controller)))) "cancelled")))) (defn- planned-record diff --git a/frontend/test/performance_tests/render_wasm/request_test.cljs b/frontend/test/performance_tests/render_wasm/request_test.cljs index 98f8a864d2..f76dfba914 100644 --- a/frontend/test/performance_tests/render_wasm/request_test.cljs +++ b/frontend/test/performance_tests/render_wasm/request_test.cljs @@ -2,7 +2,7 @@ ;; License, v. 2.0. If a copy of the MPL was not distributed with this ;; file, You can obtain one at http://mozilla.org/MPL/2.0/. ;; -;; Copyright (c) KALEIDOS INC +;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns performance-tests.render-wasm.request-test (:require diff --git a/frontend/test/performance_tests/render_wasm/server_test.cljs b/frontend/test/performance_tests/render_wasm/server_test.cljs index 1308a49f6a..bc9f87a7c2 100644 --- a/frontend/test/performance_tests/render_wasm/server_test.cljs +++ b/frontend/test/performance_tests/render_wasm/server_test.cljs @@ -2,7 +2,7 @@ ;; License, v. 2.0. If a copy of the MPL was not distributed with this ;; file, You can obtain one at http://mozilla.org/MPL/2.0/. ;; -;; Copyright (c) KALEIDOS INC +;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns performance-tests.render-wasm.server-test (:require