github/penpot
1
0
Fork 0
mirror of https://github.com/penpot/penpot.git synced 2026-11-05 09:30:38 +00:00
Code Issues Packages Projects Releases Wiki Activity

✨ Complete benchmarking cli redesign

Browse Source 
AI-assisted-by: gpt-6.1-sol, deepseek-flash
This commit is contained in:
Miguel de Benito Delgado 2026-11-02 23:45:07 +01:00
parent 69980c5100
commit cbfc3208dc
20 changed files with 415 additions and 1012 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

95
frontend/performance/benches/render_wasm/README.md
View File

@ -247,11 +247,14 @@ manifest records the build id, status, functional settings, Git provenance, and
build facts. An experiment run can select a retained build with `--build <id>`,
but a run without `--build` always prepares a new one.
A manual build can select Cargo features with `--features`, but `--build` cannot
be combined with feature selection.
`build` and a run that prepares its build accept `--features`,
`--[no-]default-features`, and `--build-timeout-ms`. A run with `--build` reuses
the retained build and rejects those preparation options.
We use the same locks as the main build system to ensure that two preparations
do not run concurrently in the same checkout.
Run one build preparation at a time in a checkout and keep renderer watchers
stopped until it finishes. The renderer build lock covers Cargo compilation
and copying the renderer files. The build script releases that lock before
ClojureScript compilation.
### Browser host
@ -631,7 +634,9 @@ finding.
### Comparisons
A comparison pairs saved cases by identical case id unless `--cases A,B`
fixes the selection to single baseline = A, candidate = B.
selects one explicit baseline, candidate pair. The CLI checks the selector
syntax before it loads either record, then resolves each selector by saved case
ID or exact name.
By default, a comparison requires equal:
@ -710,11 +715,13 @@ compares each edge once, not once per execution order. For features
10 -> 11 : b
```
`--max-features` accepts 1 through 6 and defaults to 3. The command requires
`--confirm` when the comparison has more than 8 configurations or when the
filter selects every case. `--dry-run` prints the features, configurations,
edges, selected case count, and budgets without creating a comparison home,
compiling, or launching a browser.
`--max-features` accepts 1 through 6 and defaults to 3. When `--dry-run`
is absent, the command requires `--confirm` for more than 8 configurations
or a filter that is absent, empty, or contains only whitespace. `--dry-run`
checks the requested features using Cargo metadata, selects cases from their
declarations, and prints the features, configurations, edges, selected case
count, and time limits. It creates no comparison directory, compiles no build,
and launches no browser.
### Running a feature comparison
@ -764,9 +771,9 @@ features, build and run references, edges, failures, and interpretation
warnings. The runner validates the record with a schema and writes it
atomically. It writes the initial plan before the first build and updates the
record after each configuration, after each edge, and at termination. The
command prints `Comparison <id>` and `Output <absolute path>` on stdout before
the first build, then `Status <status>` at termination. Progress and failures
go to stderr. Each configuration has its own build and run log.
command prints `Comparison <id>`, `Output <absolute path>`, and `Status <status>`
on stdout at termination. Progress and failures go to stderr. Each configuration
has its own build and run log.
### Edge comparisons
@ -809,6 +816,17 @@ until preparation finishes.
The literal `--` after `render-bench` is required to forward the command and its
arguments through the shell wrapper.
List the commands, or show one command's options and defaults:
```bash
pnpm run render-bench -- --help
pnpm run render-bench -- run --help
```
Command options select declared cases and control the build source, sampling,
output and hardware policy. Case declarations own workload bodies, scene
parameters, view and operation settings, and case lifecycle.
Build the runner and its command module:
```bash
@ -880,35 +898,46 @@ pnpm run render-bench -- merge \
--output-dir target/performance/render-wasm/runs/<merged>
```
The merged record gets a new run id and a fresh run home. Both inputs must be
completed and compatible: same cases, timeout policy, analysis, build facts,
runtime environment, and Git provenance. The additional run must use the same
warmup count. `merge` appends only the additional measured attempts; its warmup
attempts remain in the additional record. The command starts no build, server,
or browser.
The merged record gets a new run id and a new run directory. Both input records
stay unchanged. Both inputs must be completed and compatible: same cases,
timeout policy, analysis, build facts, runtime environment, and Git provenance.
The additional run must use the same warmup count. `merge` appends only the
additional measured attempts and uses the base run's warmups. The additional
warmup attempts stay in the additional record. The command starts no build,
server, or browser.
Inspect and clean managed outputs:
```bash
pnpm run render-bench -- purge --only-builds
pnpm run render-bench -- purge --only-runs
pnpm run render-bench -- purge --only-comparisons
pnpm run render-bench -- purge builds
pnpm run render-bench -- purge runs
pnpm run render-bench -- purge comparisons
pnpm run render-bench -- purge all
```
`purge` removes known build, standalone-run, or feature-comparison outputs. It
keeps the managed directories and unrelated files. The three `--only-*` options
are mutually exclusive; no option selects every category.
`purge` requires exactly one target: `builds`, `runs`, `comparisons`, or `all`.
It removes known build, standalone-run, or feature-comparison outputs and keeps
directories and unrelated files. It follows directory symlinks and can remove
matching output files from their targets, including targets outside the managed
directories. `all` removes every category. A missing, unknown or multiple target
is an argument error.
The commands write results to stdout:
- `build`: `Build <id>` and `Output <absolute path>` at allocation.
- `run`: `Run <id>` and `Output <absolute path>` at allocation, then a `Report`
heading and the formatted summary.
- `build`: `Build <id>` and `Output <absolute path>` after build preparation
ends.
- `run`: `Run <id>` and `Output <absolute path>` after the run ends. When a
report is available, a `Report` heading and the formatted summary follow.
- `ab`: `Comparison <id>`, `Output <absolute path>`, and final
`Status <status>`.
- `compare`: the comparison report.
- `merge`: `Run <id>` and `Output <absolute path>`.
- Help: usage text. `purge` produces no output on success.
- `--help` and `COMMAND --help`: usage text with that command's options and
defaults. `purge` produces no output on success.
Each CLI command produces an outcome with an exit code, errors, log path, and
optional report text. The runner prints the final output once and returns the
numeric status.
Progress and diagnostics go to stderr. The default shows build phases, case
headings, and attempt progress. `-q` / `--quiet` shows results, warnings, and
@ -938,9 +967,11 @@ mean interruption by `SIGINT` and `SIGTERM` for `build`, `run` and `ab`. An
interrupted command prints no error for the aborted work; it prints a failed
write of its own record or log.
`--warmups` defaults to 3, `--repetitions` to 10, `--timeout-ms` to 30000,
`--build-timeout-ms` to 300000, and `--startup-timeout-ms` to 30000. The master
seed defaults to 42 and accepts a 32-bit unsigned integer. A filter matches a
`--warmups` defaults to 3, `--repetitions` to 10, `--timeout-ms` to 30000, and
`--startup-timeout-ms` to 30000. `build`, a run that prepares its build, and an
A/B experiment use a default `--build-timeout-ms` of 300000; a run with
`--build` uses the retained build's recorded provenance. The master seed
defaults to 42 and accepts a 32-bit unsigned integer. A filter matches a
substring of the case id.
## Unit tests

6
frontend/performance/benches/render_wasm/node/build.cljs
View File

@ -65,10 +65,10 @@
Options: frontend :root, :selection or :features, :build-timeout-ms
(300000), AbortSignal :signal, and optional :on-created callback. The callback
runs once after directory, log and initial manifest creation. CLI callers
print the ID there. Storage errors have no recorded ID. Later errors carry
receives the new build reference once the output directory, log and initial
manifest exist. Storage errors have no recorded ID. Later errors carry
:build-id and :phase. An error also carries :manifest-error and is fatal when
the failed manifest cannot be saved. A retry always creates a new attempt."
the failed manifest cannot be saved. A retry always creates a new build."
[{:keys [root features selection build-timeout-ms signal on-created]
:or {build-timeout-ms options/default-build-timeout-ms}}]
(when-not (and (js/Number.isSafeInteger build-timeout-ms) (pos? build-timeout-ms))

133
frontend/performance/benches/render_wasm/node/commands.cljs
View File

@ -2,106 +2,67 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
;; Copyright (c) KALEIDOS INC
(ns benches.render-wasm.node.commands
"Execution commands loaded after CLI validation."
"Deferred execution entry for build, run and feature comparison commands."
(:require
[app.common.schema :as sm]
[app.common.transit :as transit]
[benches.render-wasm.errors :as errors :refer [attempt failure?]]
[benches.render-wasm.errors :refer [attempt failure?]]
[benches.render-wasm.node.artifacts :as artifacts]
[benches.render-wasm.node.build :as build]
[benches.render-wasm.node.execution :as execution]
[benches.render-wasm.node.feature-comparison :as feature-comparison]
[benches.render-wasm.node.feature-plan :as feature-plan]
[benches.render-wasm.node.log :as log]
[benches.render-wasm.node.options :as options]
[benches.render-wasm.node.outcome :as outcome]
[benches.render-wasm.node.run :as run]))
(defn- run-command
[root options]
(execution/with-interruption
#(run/run root (select-keys options (sm/keys run/schema:run-options)) %)))
(defn- ^:async build-command
[{:keys [get-root signal]} options]
(let [created (atom nil)
result (attempt :catch-fatal
(await (build/prepare
{:root (get-root)
:selection {:default-features? (:default-features options)
:features (:features options)}
:build-timeout-ms (:build-timeout-ms options)
:signal signal :on-created #(reset! created %)})))
result (if (failure? result)
(outcome/failed result {:build @created})
(outcome/succeeded {:build result :log-path (:log-path result)}))]
(cond-> result
@created (assoc :report (str "Build " (:id @created) "\nOutput " (:dir @created))))))
(defn- build-command
[root options]
(execution/with-interruption
(^:async fn [signal]
(await (build/prepare {:root root
:features {:default-features? (:default-features options)
:features (or (:features options) [])}
:build-timeout-ms (:build-timeout-ms options)
:signal signal
:on-created (fn [build]
(log/result (str "Build " (:id build)))
(log/result (str "Output " (:dir build))))}))
{:exit-code 0})))
(defn- ^:async run-command
[{:keys [get-root signal services]} options]
(let [{:keys [home report] :as result} (await (run/run (get-root) options signal services))]
(assoc result :report (str "Run " (:run-id home) "\nOutput " (:dir home)
(when report (str "\nReport\n" report))))))
(defn- ab-command
[root options]
(execution/with-interruption
(^:async fn [signal]
(let [prepared (await (feature-plan/prepare root options signal))]
(if (:dry-run options)
(do
(log/result (feature-plan/format-plan prepared))
{:exit-code 0})
(await (feature-comparison/execute root prepared signal {})))))))
(defn- ^:async ab-command
[{:keys [get-root signal services]} options]
(let [root (get-root)
prepared (await (feature-plan/prepare root options signal))]
(if (:dry-run options)
(outcome/succeeded {:report (feature-plan/format-plan prepared)})
(let [{:keys [home exit-code] :as result}
(await (feature-comparison/execute root prepared signal {:services services}))]
(cond-> result
home (assoc :report (str "Comparison " (:comparison-id home)
"\nOutput " (:dir home)
"\nStatus " (cond
(.-aborted signal) "interrupted"
(zero? exit-code) "completed"
:else "failed"))))))))
(defn- ^:async run-request
"Runs the validated command and returns its outcome."
[request]
(let [{:keys [command options]} (transit/decode-str request)
_ (when (and (:require-gpu options)
(or (= command "run")
(and (= command "ab") (not (:dry-run options))))
(not= "1" (.-PENPOT_GPU (.-env js/process))))
(throw (ex-info "--require-gpu needs PENPOT_GPU=1. Start the container with ./manage.sh run-devenv --gpu and use pnpm run render-bench."
{:type ::gpu-required})))
root (artifacts/frontend-root)]
(log/configure {:level (options/console-level options)
:color (options/color-mode options)})
(case command
"purge"
(do
(artifacts/purge root (cond (:only-builds options) :builds
(:only-runs options) :runs
(:only-comparisons options) :comparisons
:else :all))
{:exit-code 0})
"build"
(await (build-command root options))
"run"
(await (run-command root options))
"ab"
(await (ab-command root options)))))
(defn- log-failures
"Prints each failure of `result`. Only the first prints a log tail."
[result]
(binding [log/*log-path* (:log-path result)]
(doseq [[index cause] (map-indexed vector (:errors result))]
(let [data (ex-data cause)]
(log/error {:message (errors/message-of cause)
:fields (filterv (comp some? second)
[[:type (:type data)] [:phase (:phase data)]
[:build (:build-id data)] [:reason (:reason data)]
[:manifest-error (:manifest-error data)]])
:tail-log (:log-path data)
:tail? (zero? index)})))))
(def ^:private handlers
{"build" build-command "run" run-command "ab" ab-command})
(defn ^:async execute
"Accepts a validated Transit command from the CLI and returns its exit code.
Each compiled module owns its CLJS values; only the string crosses modules.
Execution errors are printed here before returning a numeric exit code."
[request]
(log/configure {})
(let [result (attempt :catch-fatal (await (run-request request)))
finished (if (failure? result) (outcome/failed result {}) result)]
(log-failures finished)
(:exit-code finished)))
"Accepts a Transit string containing a complete request and returns its
numeric exit status after final publication."
[text]
(let [request (attempt :catch-fatal (transit/decode-str text))]
(if (failure? request)
(execution/fail request)
(await (execution/execute request handlers {:get-root artifacts/frontend-root})))))

8
frontend/performance/benches/render_wasm/node/compare.cljs
View File

@ -23,13 +23,13 @@
It is also possible to restrict comparison to specific scene ids.
Options:
- `:cases` is one A,B selector, where A is a baseline an B the candidate.
- `:case-pair` contains the baseline and candidate selectors, or nil.
- The three `:allow-*-differences` options select build, scene or all
compatibility permissions.
Returns the validated `comparison` with its selected permissions, retained
differences and formatted text."
[baseline-path candidate-path {:keys [cases allow-build-differences
[baseline-path candidate-path {:keys [case-pair allow-build-differences
allow-scene-differences allow-all-differences]}]
(let [baseline (runs/get-record baseline-path)
candidate (runs/get-record candidate-path)
@ -39,8 +39,8 @@
(cond-> {:allow-build-differences (boolean allow-build-differences)
:allow-scene-differences (boolean allow-scene-differences)
:allow-all-differences (boolean allow-all-differences)}
cases
case-pair
(assoc :pairs (compare/selector-pair baseline candidate
cases)))))]
case-pair)))))]
{:comparison comparison
:text (format/format-comparison comparison)}))

71
frontend/performance/benches/render_wasm/node/feature_comparison.cljs
View File

@ -7,24 +7,23 @@
(ns benches.render-wasm.node.feature-comparison
"Serial feature A/B orchestration and persistence of comparisons."
(:require
["fs" :as fs]
["node:timers/promises" :as timers]
["path" :as path]
[app.common.schema :as sm]
[benches.render-wasm.errors :as errors :refer [attempt failure?]]
[benches.render-wasm.node.artifacts :as artifacts]
[benches.render-wasm.node.build :as build]
[benches.render-wasm.node.catalog :as catalog]
[benches.render-wasm.node.files :as files]
[benches.render-wasm.node.log :as log]
[benches.render-wasm.node.outcome :as outcome]
[benches.render-wasm.node.process :as process]
[benches.render-wasm.node.run :as run]
[benches.render-wasm.node.runs :as runs]
[benches.render-wasm.node.time :as time]
[benches.render-wasm.report.compare :as compare]
[benches.render-wasm.result :as result]
[clojure.set :as set]))
["fs" :as fs]
["node:timers/promises" :as timers]
["path" :as path]
[benches.render-wasm.errors :as errors :refer [attempt failure?]]
[benches.render-wasm.node.artifacts :as artifacts]
[benches.render-wasm.node.build :as build]
[benches.render-wasm.node.catalog :as catalog]
[benches.render-wasm.node.files :as files]
[benches.render-wasm.node.log :as log]
[benches.render-wasm.node.outcome :as outcome]
[benches.render-wasm.node.process :as process]
[benches.render-wasm.node.run :as run]
[benches.render-wasm.node.runs :as runs]
[benches.render-wasm.node.time :as time]
[benches.render-wasm.report.compare :as compare]
[benches.render-wasm.result :as result]
[clojure.set :as set]))
(def ^:private warnings
["Timing values do not determine success."
@ -49,8 +48,9 @@
dir (artifacts/check-output-dir (or output-dir default-dir))]
(when (artifacts/inside-builds? root dir)
(artifacts/invalid-home "feature comparison cannot use a directory inside builds" {:path dir}))
(artifacts/create-home dir "feature-comparison"
"cannot create feature comparison home")))
(assoc (artifacts/create-home dir "feature-comparison"
"cannot create feature comparison home")
:comparison-id comparison-id)))
(defn write-record
"Validates schema:feature-comparison and atomically replaces the saved file."
@ -316,7 +316,7 @@
(attempt
(await ((:execute-run (:services state))
(:root state) (:run-home flow) (:build flow) (:cases state)
(select-keys (:options state) (sm/keys run/schema:run-options))
(run/get-execution-options (:options state))
(:signal state)))))
(defn- get-evidence-error
@ -595,20 +595,10 @@
(defn- initialize-comparison
[state]
(let [initial @(:record-ref state)
home (:home state)]
(log/result (str "Comparison " (:comparison-id initial)))
(log/result (str "Output " (:dir home)))
(let [saved (write-record home initial)]
(reset! (:record-ref state) saved)
(reset! (:saved-ref state) saved))))
(defn- publish-status
[result]
(outcome/publish result
#(log/result (str "Status " (case (:exit-code result)
0 "completed"
(130 143) "interrupted"
"failed")))))
home (:home state)
saved (write-record home initial)]
(reset! (:record-ref state) saved)
(reset! (:saved-ref state) saved)))
(defn- ^:async execute-comparison
[state]
@ -617,12 +607,11 @@
(let [result (attempt :catch-fatal
(initialize-comparison state)
(await (run-comparisons state (:configurations @(:record-ref state)))))]
(publish-status
(if (failure? result)
(outcome/failed result {:comparison @(:saved-ref state)
:home home
:log-path (:log-path home)})
result))))))
(if (failure? result)
(outcome/failed result {:comparison @(:saved-ref state)
:home home
:log-path (:log-path home)})
result)))))
(defn ^:async execute
"Executes a prepared feature comparison and owns builds, runs and reports.

9
frontend/performance/benches/render_wasm/node/feature_plan.cljs
View File

@ -33,14 +33,7 @@
render-wasm with the inherited environment, checks an optional fresh output
path, and collects selected case data once."
[root opts signal]
(let [opts (merge {:max-features feature-plan/default-max-features
:confirm false :dry-run false :seed 42
:allow-build-differences false
:allow-scene-differences false
:allow-all-differences false}
opts
{:default-features false})
plan (feature-plan/plan (:features opts) opts)
(let [plan (feature-plan/plan (:features opts) opts)
output-dir (artifacts/check-output-dir (:output-dir opts))
cwd (path/resolve root ".." "render-wasm")
metadata (await (process/run

3
frontend/performance/benches/render_wasm/node/merge.cljs
View File

@ -8,7 +8,6 @@
"Offline merge of two saved schema:run records."
(:require
["fs" :as fs]
[benches.render-wasm.node.log :as log]
[benches.render-wasm.node.runs :as runs]
[benches.render-wasm.result :as result]))
@ -26,6 +25,4 @@
(runs/write-record! home merged)
(fs/appendFileSync (:log-path home)
(str "merge " (:run-id base) " + " (:run-id additional) "\n"))
(log/result (str "Run " run-id))
(log/result (str "Output " (:dir home)))
{:run merged :home home}))

5
frontend/performance/benches/render_wasm/node/outcome.cljs
View File

@ -19,6 +19,11 @@
[signal-name]
(case signal-name "SIGTERM" 143 130))
(defn succeeded
"Returns a successful outcome with command result data."
[data]
(merge {:exit-code 0 :errors [] :log-path nil} data))
(defn failed
"Returns an outcome with exit code 1 for `cause`. The log path comes from
the ex-data of `cause`. `data` adds more keys."

110
frontend/performance/benches/render_wasm/node/run.cljs
View File

@ -23,7 +23,6 @@
["module" :as node-module]
["os" :as os]
["path" :as path]
[app.common.schema :as sm]
[app.common.transit :as transit]
[benches.render-wasm.declarations :as declarations]
[benches.render-wasm.errors :as errors :refer [attempt failure?]]
@ -43,30 +42,11 @@
[benches.render-wasm.result :as result]
[clojure.string :as str]))
(def schema:run-options
"Resolved options for run preparation and execution."
(let [timeout-ms [:int {:min 1 :max 2147483647}]]
[:map {:closed true}
[:seed [:int {:min 0 :max 4294967295}]]
[:warmups [:int {:min 0 :max js/Number.MAX_SAFE_INTEGER}]]
[:repetitions [:int {:min 1 :max js/Number.MAX_SAFE_INTEGER}]]
[:timeout-ms timeout-ms]
[:build-timeout-ms timeout-ms]
[:startup-timeout-ms timeout-ms]
[:filter {:optional true} [:maybe :string]]
[:build-id {:optional true} [:string {:min 1}]]
[:output-dir {:optional true} [:maybe [:string {:min 1}]]]
[:default-features {:optional true} :boolean]
[:features {:optional true} [:vector [:string {:min 1}]]]
[:screenshots {:optional true} :boolean]
[:remove {:optional true} :boolean]
[:require-gpu {:optional true} :boolean]
[:browser {:optional true} [:map-of :keyword fn?]]]))
(def ^:private check-options
(sm/check-fn schema:run-options
:hint "invalid run options"
:type ::invalid-options))
(defn get-execution-options
"Returns resolved settings for executing a prepared run."
[options]
(select-keys options [:seed :filter :warmups :repetitions :timeout-ms
:startup-timeout-ms :screenshots :require-gpu :remove]))
(defn- cause-evidence
"Plain failure evidence for a rejected browser, server or persistence action."
@ -650,20 +630,20 @@
(assoc :environment (get-in selected-build [:manifest :environment])))
:environment (environment root selected-build)
:scored? true
:diagnostics {:seed (:seed opts)
:filter (:filter opts)
:warmups (:warmups opts)
:repetitions (:repetitions opts)
:timeout-ms (:timeout-ms opts)
:build-timeout-ms (:build-timeout-ms opts)
:startup-timeout-ms (:startup-timeout-ms opts)
:screenshots? (boolean (:screenshots opts))
:require-gpu? (boolean (:require-gpu opts))
:penpot-gpu (or (.-PENPOT_GPU (.-env js/process)) nil)
:runner-git {:sha nil :dirty nil}
:runtime-helper-identity :unverified
:screenshots {}
:failures []}})
:diagnostics (cond-> {:seed (:seed opts)
:filter (:filter opts)
:warmups (:warmups opts)
:repetitions (:repetitions opts)
:timeout-ms (:timeout-ms opts)
:startup-timeout-ms (:startup-timeout-ms opts)
:screenshots? (boolean (:screenshots opts))
:require-gpu? (boolean (:require-gpu opts))
:penpot-gpu (or (.-PENPOT_GPU (.-env js/process)) nil)
:runner-git {:sha nil :dirty nil}
:runtime-helper-identity :unverified
:screenshots {}
:failures []}
(:build-timeout-ms opts) (assoc :build-timeout-ms (:build-timeout-ms opts)))})
(defn- create-run-record
[root run-id selected-build cases opts]
@ -772,7 +752,7 @@
- `home`: the reserved output home.
- `selected-build`: the prepared build.
- `cases`: the selected cases.
- `options`: resolved options satisfying `schema:run-options`.
- `options`: resolved execution settings, including optional local :browser services.
- `signal`: an optional abort signal.
A browser startup failure leaves a valid partial record.
@ -782,7 +762,6 @@
write, including a failure followed by a successful write. An interruption
is not a failure."
[root home selected-build cases options signal]
(check-options options)
(binding [log/*log-path* (:log-path home)]
(let [state (create-run-state root home selected-build cases options signal)]
(persist state)
@ -792,14 +771,13 @@
(defn- ^:async get-selected-build
[root opts signal]
(if-let [id (:build-id opts)]
(artifacts/load-build root id)
(await (build/prepare
{:root root
:features {:default-features? (:default-features opts true)
:features (or (:features opts) [])}
:build-timeout-ms (:build-timeout-ms opts)
:signal signal}))))
(let [{:keys [kind id default-features features build-timeout-ms]} (:build opts)]
(case kind
:existing (artifacts/load-build root id)
:prepare (await (build/prepare
{:root root
:selection {:default-features? default-features :features features}
:build-timeout-ms build-timeout-ms :signal signal})))))
(defn- get-cases
[selected-build opts]
@ -811,35 +789,29 @@
:log-path (:log-path selected-build))
cause)))))
(defn- publish-report
[result]
(outcome/publish result
#(when-let [report (:report result)]
(log/result (str "Report\n" report)))))
(defn ^:async run
"Prepares or loads a build, collects matching cases and executes one run.
- `root`: the frontend checkout root.
- `options`: resolved options satisfying `schema:run-options`.
- `options`: resolved run options with a tagged :build source.
- `signal`: an optional abort signal.
- `services`: local service overrides, with browser functions under :browser.
Validates options before allocating output. Returns the execution result.
A build or catalog failure throws with the run log path and build facts."
[root options signal]
(let [opts (check-options options)
home (runs/create-home {:root root :run-id (str (random-uuid))
:output-dir (:output-dir opts) :remove (:remove opts)})]
Returns an outcome with the reserved home, available evidence and failures."
[root options signal services]
(let [home (runs/create-home {:root root :run-id (str (random-uuid))
:output-dir (:output-dir options) :remove (:remove options)})
opts (cond-> (assoc (get-execution-options options) :browser (:browser services))
(= :prepare (get-in options [:build :kind]))
(assoc :build-timeout-ms (get-in options [:build :build-timeout-ms])))]
(binding [log/*log-path* (:log-path home)]
(try
(log/result (str "Run " (:run-id home)))
(log/result (str "Output " (:dir home)))
(let [selected-build (await (get-selected-build root opts signal))
(let [selected-build (await (get-selected-build root options signal))
cases (get-cases selected-build opts)]
(log/info (str " build " (:id selected-build)))
(log/info (str " selected " (count cases) " cases"))
(publish-report (await (execute root home selected-build cases opts signal))))
(await (execute root home selected-build cases opts signal)))
(catch :default cause
(throw (-> cause
errors/as-failure
(errors/with-data {:log-path (:log-path home)}))))))))
(outcome/failed (-> cause errors/as-failure
(errors/with-data {:log-path (:log-path home)}))
{:home home :run nil :report nil}))))))

357
frontend/performance/benches/render_wasm/node/runner.cljs
View File

@ -2,336 +2,59 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
;; Copyright (c) KALEIDOS INC
(ns benches.render-wasm.node.runner
"CLI parsing, offline comparison and deferred execution commands."
"Command-line entry for renderer benchmarks and saved results."
(:require
["path" :as path]
[app.common.transit :as transit]
[benches.render-wasm.errors :as errors :refer [attempt failure?]]
[benches.render-wasm.feature-plan :as feature-plan]
[benches.render-wasm.errors :refer [attempt failure?]]
[benches.render-wasm.node.artifacts :as artifacts]
[benches.render-wasm.node.cli :as cli]
[benches.render-wasm.node.compare :as compare]
[benches.render-wasm.node.log :as log]
[benches.render-wasm.node.merge :as run-merge]
[benches.render-wasm.node.options :as options]
[clojure.string :as str]
[clojure.tools.cli :refer [parse-opts]]))
[benches.render-wasm.node.execution :as execution]
[benches.render-wasm.node.merge :as merge]
[benches.render-wasm.node.outcome :as outcome]))
(def ^:private max-timeout-ms 2147483647)
(defn- help-command
[_ {:keys [topic]}]
(outcome/succeeded {:report (cli/get-help topic)}))
(def ^:private positive-timeout
[(fn [n] (and (js/Number.isSafeInteger n)
(pos? n)
(<= n max-timeout-ms)))
"Must be a positive integer up to 2147483647"])
(defn- compare-command
[_ {:keys [baseline-path candidate-path] :as options}]
(let [result (compare/execute baseline-path candidate-path options)]
(outcome/succeeded {:comparison (:comparison result) :report (:text result)})))
(def ^:private nonnegative-integer
[(fn [n] (and (js/Number.isSafeInteger n) (not (neg? n))))
"Must be a nonnegative integer"])
(defn- merge-command
[{:keys [get-root]} {:keys [base-path additional-path] :as options}]
(let [{:keys [run home]} (merge/execute (get-root) base-path additional-path options)]
(outcome/succeeded {:run run :home home :log-path (:log-path home)
:report (str "Run " (:run-id run) "\nOutput " (:dir home))})))
(def ^:private positive-integer
[(fn [n] (and (js/Number.isSafeInteger n) (pos? n)))
"Must be a positive integer"])
(defn- purge-command
[{:keys [get-root]} {:keys [target]}]
(artifacts/purge (get-root) target)
(outcome/succeeded {}))
(def ^:private uint32
[(fn [n] (and (js/Number.isSafeInteger n) (<= 0 n 4294967295)))
"Must be a uint32 integer"])
(def ^:private nonempty
[(fn [s] (not (str/blank? s))) "Must be a nonempty value"])
(defn- parse-number
"Parses a numeric CLI value. Blank input stays invalid instead of becoming zero."
[text]
(if (str/blank? text) js/NaN (js/Number text)))
(def cli-options
[["-h" "--help" "Print this help"]
["-q" "--quiet" "Only warnings, errors and results"]
["-v" "--verbose" "Show debug details and raw build output"]
[nil "--[no-]color" "Colorize terminal output; default auto" :default nil]
[nil "--[no-]default-features" "Enable Cargo defaults" :default true]
[nil "--features FEATURES" "Comma-separated functional Cargo features"
:parse-fn options/parse-feature-list]
[nil "--max-features N" "ab feature limit (1..6)"
:default feature-plan/default-max-features
:parse-fn parse-number
:validate [#(and (js/Number.isSafeInteger %) (<= 1 % 6)) "must be an integer from 1 to 6"]]
[nil "--confirm" "Confirm ab with >8 configurations or an empty filter" :default false]
[nil "--dry-run" "Validate and print the ab plan without executing" :default false]
[nil "--build ID" "Reuse a ready build" :id :build-id
:validate nonempty]
[nil "--filter SUBSTR" "Select cases whose id contains SUBSTR"]
[nil "--seed N" "Master scene seed" :default 42
:parse-fn parse-number :validate uint32]
[nil "--warmups N" "Warmup attempts per case" :default 3
:parse-fn parse-number :validate nonnegative-integer]
[nil "--repetitions N" "Measured attempts per case" :default 10
:parse-fn parse-number :validate positive-integer]
[nil "--timeout-ms MS" "Attempt deadline" :default 30000
:parse-fn parse-number :validate positive-timeout]
[nil "--build-timeout-ms MS" "Complete preparation deadline"
:default options/default-build-timeout-ms
:parse-fn parse-number :validate positive-timeout]
[nil "--startup-timeout-ms MS" "Host/browser readiness deadline"
:default options/default-startup-timeout-ms
:parse-fn parse-number :validate positive-timeout]
[nil "--output-dir DIR" "Explicit run home; must not exist"
:validate nonempty]
[nil "--remove" "Remove the used build after shutdown"]
[nil "--screenshots" "Capture one screenshot per case"]
[nil "--require-gpu" "Fail when GPU rendering is unavailable"]
[nil "--only-builds" "Purge build attempts"]
[nil "--only-runs" "Purge managed runs"]
[nil "--only-comparisons" "Purge managed feature comparisons"]
[nil "--cases A,B" "Select one baseline,candidate pair"]
[nil "--allow-build-differences" "Allow recorded build differences"]
[nil "--allow-scene-differences" "Allow scene inputs and fingerprint content differences"]
[nil "--allow-all-differences" "Allow all comparison compatibility differences"]])
(defn usage
[summary]
(str "Usage: runner build [options]\n"
" runner purge [--only-builds | --only-runs | --only-comparisons]\n"
" runner run [options]\n"
" runner ab --features A,B [--filter SUBSTR] [--max-features N]\n"
" [--confirm] [--dry-run] [run and comparison options]\n"
" runner compare BASELINE CANDIDATE [--cases A,B]\n"
" [--allow-build-differences] [--allow-scene-differences]\n"
" [--allow-all-differences]\n"
" runner merge BASE ADDITIONAL [--output-dir DIR]\n\n"
"build prints Build <id> and Output <absolute path> on stdout. Exit 0 means\n"
"ready; errors retain the build.transit and build.log for that ID.\n"
"run prints Run <id>, Output <absolute path> and Report on stdout.\n"
"Progress and diagnostics go to stderr. -q/--quiet hides progress;\n"
"-v/--verbose includes raw build output. --[no-]color overrides terminal\n"
"color detection. File logs receive all levels without logger colors.\n"
"Builds and managed runs live under target/performance/render-wasm.\n"
"run and ab accept --require-gpu. Start the container with\n"
"./manage.sh run-devenv --gpu and use pnpm run render-bench -- run ...\n"
"to require hardware WebGL2 in preparation metadata. ab --dry-run skips it.\n"
"compare reads saved schema:run records and starts no build, host or browser.\n"
"Build and scene permissions combine; operation and runtime checks stay strict.\n"
"All-differences permits every compatibility difference. Reports retain them.\n"
"merge reads two saved schema:run records and writes a merged record in a fresh\n"
"run home. It appends the additional run's measured attempts, keeps the base\n"
"warmups, and requires compatible completed runs. It starts no build or browser.\n\n"
"ab uses every on/off configuration with Cargo defaults disabled; declared\n"
"defaults must be empty. --max-features defaults to 3 (range 1..6).\n"
"Use --confirm above eight configurations or with an empty filter.\n"
"--dry-run checks metadata, cases and output freshness without confirmation,\n"
"output creation, compilation or browser startup. ab accepts run budgets,\n"
"timeouts, seed, filter, output and screenshots, plus comparison permissions.\n"
"It rejects --build, --remove, --cases and --[no-]default-features.\n"
"ab prints Comparison <id>, Output <path> and Status <status> on stdout;\n"
"feature-comparison.log records comparison progress. Diagnostics and\n"
"failures go to stderr. --dry-run prints only the plan without creating\n"
"the comparison home. Managed output lives under\n"
"feature-comparisons/<ab-id>/feature-comparison.transit.\n\n"
"Preparation and runs require an idle checkout with dependencies installed.\n"
"Keep renderer watchers stopped until preparation ends. purge assumes no\n"
"experiment is using the managed paths.\n\n"
summary))
(def ^:private value-flags
"Long options that consume the next argument when no value follows an equals sign."
(into #{}
(keep (fn [[_ long-option]]
(when (and long-option (re-find #"\s" long-option))
(first (str/split long-option #"\s+")))))
cli-options))
(defn- supplied-flags
[args]
(loop [tokens (seq args)
flags #{}
value? false
operand? false]
(if-let [token (first tokens)]
(cond
value?
(recur (next tokens) flags false operand?)
(and (not operand?) (= "--" token))
(recur (next tokens) flags false true)
operand?
(recur (next tokens) flags false true)
(str/starts-with? token "--")
(let [[flag inline-value] (str/split token #"=" 2)]
(recur (next tokens)
(conj flags flag)
(and (contains? value-flags flag) (nil? inline-value))
false))
:else
(recur (next tokens) flags false false))
flags)))
(def ^:private build-flags
#{"--default-features" "--no-default-features" "--features" "--build-timeout-ms"})
(def ^:private run-flags
#{"--build" "--filter" "--seed" "--warmups" "--repetitions" "--timeout-ms"
"--startup-timeout-ms" "--output-dir" "--remove" "--screenshots" "--require-gpu"})
(def ^:private purge-flags
#{"--only-builds" "--only-runs" "--only-comparisons"})
(def ^:private ab-flags
#{"--max-features" "--confirm" "--dry-run"})
(def ^:private compare-flags
#{"--cases" "--allow-build-differences" "--allow-scene-differences"
"--allow-all-differences"})
(defn- invalid-args
([message]
(invalid-args message nil))
([message cause]
(throw (ex-info message {:type ::invalid-args} cause))))
(defn- some-flags?
[flags group]
(boolean (some flags group)))
(defn- check-command-flags!
[command flags]
(when (and (not= command "ab") (some-flags? flags ab-flags))
(invalid-args "ab planning options require ab"))
(let [build? (some-flags? flags build-flags)
run? (some-flags? flags run-flags)
purge? (some-flags? flags purge-flags)
compare? (some-flags? flags compare-flags)]
(case command
"build"
(do
(when purge? (invalid-args "purge category flags require purge"))
(when run? (invalid-args "run options require run"))
(when compare? (invalid-args "compare options require compare")))
"purge"
(do
(when build? (invalid-args "build options require build"))
(when run? (invalid-args "run options require run"))
(when compare? (invalid-args "compare options require compare")))
"run"
(do
(when purge? (invalid-args "purge category flags require purge"))
(when compare? (invalid-args "compare options require compare")))
"ab"
(do
(when purge? (invalid-args "purge category flags require purge"))
(when (some-flags? flags ["--build" "--remove" "--cases"
"--default-features" "--no-default-features"])
(invalid-args "ab rejects --build, --remove, --cases and --[no-]default-features")))
"compare"
(do
(when purge? (invalid-args "purge category flags require purge"))
(when build? (invalid-args "build options require build"))
(when run? (invalid-args "run options require run")))
"merge"
(do
(when purge? (invalid-args "purge category flags require purge"))
(when build? (invalid-args "build options require build"))
(when compare? (invalid-args "compare options require compare"))
(when (some-flags? flags (disj run-flags "--output-dir"))
(invalid-args "run options require run"))))))
(defn- check-arguments!
[command arguments]
(let [extra (rest arguments)]
(case command
"compare"
(when-not (= 2 (count extra))
(invalid-args "compare requires BASELINE and CANDIDATE paths"))
"merge"
(when-not (= 2 (count extra))
(invalid-args "merge requires BASE and ADDITIONAL paths"))
(when (seq extra)
(invalid-args (str command " takes no positional arguments"))))))
(defn parse-args
"Parses and validates one runner command without loading a catalog, build,
server, browser or renderer. Options are resolved before any run home."
[args]
(let [args (cond-> args (= "--" (first args)) rest)
{:keys [arguments options errors summary]} (parse-opts args cli-options)
command (first arguments)
flags (supplied-flags args)]
(when (seq errors)
(invalid-args (str/join "\n" errors)))
(when (and (:quiet options) (:verbose options))
(invalid-args "--quiet and --verbose are mutually exclusive"))
(when-not (:help options)
(when-not (and (= 1 (count (take 1 arguments)))
(contains? #{"build" "purge" "run" "ab" "compare" "merge"} command))
(invalid-args "expected build, purge, run, ab, compare or merge; try --help"))
(check-command-flags! command flags)
(check-arguments! command arguments)
(when (and (= command "purge")
(> (count (filter options [:only-builds :only-runs :only-comparisons])) 1))
(invalid-args "--only-builds, --only-runs and --only-comparisons are mutually exclusive"))
(when (and (= command "run")
(:build-id options)
(some-flags? flags ["--default-features" "--no-default-features" "--features"]))
(invalid-args "--build cannot be combined with feature-selection flags"))
(when (= command "ab")
(try
(feature-plan/plan (:features options []) options)
(catch :default cause
(invalid-args (errors/message-of cause) cause)))))
{:command command
:arguments arguments
:options (cond-> options (= command "ab") (assoc :default-features false))
:summary summary}))
(def ^:private handlers
{"help" help-command "compare" compare-command "merge" merge-command "purge" purge-command})
(defn- ^:async run-cli
[args]
(let [{:keys [command arguments options summary] :as request} (parse-args args)]
(log/configure {:level (options/console-level options)
:color (options/color-mode options)})
(cond
(:help options)
(log/result (usage summary))
(= command "compare")
(log/result (:text (compare/execute (second arguments) (nth arguments 2)
(select-keys options [:cases :allow-build-differences
:allow-scene-differences
:allow-all-differences]))))
(= command "merge")
(run-merge/execute (artifacts/frontend-root)
(second arguments)
(nth arguments 2)
(select-keys options [:output-dir]))
:else
(let [^js commands (js/require (path/join js/__dirname "commands.cjs"))]
(set! (.-exitCode js/process)
(await (.execute commands
(transit/encode-str (select-keys request [:command :options])))))))))
[argv]
(let [request (attempt :catch-fatal (cli/parse-resolve argv))]
(if (failure? request)
(execution/fail request)
(if (contains? handlers (:command request))
(await (execution/execute request handlers {:get-root artifacts/frontend-root}))
(let [status (attempt :catch-fatal
(let [commands (js/require (path/join js/__dirname "commands.cjs"))]
(await (.execute commands (transit/encode-str request)))))]
(if (failure? status)
(execution/fail status (:console request) {})
status))))))
(defn ^:async -main
[& args]
(log/configure {})
(let [failed (attempt :catch-fatal (await (run-cli args)))]
(when (failure? failed)
(let [data (ex-data failed)]
(log/error {:message (str (errors/message-of failed) "\nTry --help for usage.")
:fields (into (filterv (comp some? second)
[[:type (:type data)] [:baseline (:baseline data)]
[:candidate (:candidate data)]])
(map #(vector :difference (:field %)) (:mismatches data)))
:tail? false}))
(set! (.-exitCode js/process) 1))))
"Runs argv and sets the process exit status after publication and cleanup."
[& argv]
(set! (.-exitCode js/process) (await (run-cli argv))))

10
frontend/performance/benches/render_wasm/report/compare.cljs
View File

@ -31,13 +31,9 @@
{:type ::unknown-case :selector selector :available (mapv result/case-name ids)})))))
(defn selector-pair
"Resolves a single A,B comparison selector into an extensible pair list.
A belongs to baseline; B belongs to candidate. Force is a separate option."
[baseline candidate selector]
(let [names (str/split selector #"," -1)]
(when-not (and (= 2 (count names)) (every? seq names))
(throw (ex-info "--cases requires one exact A,B pair" {:type ::invalid-selector})))
[[(resolve-case baseline (first names)) (resolve-case candidate (second names))]]))
"Resolves baseline and candidate selectors into an extensible pair list."
[baseline candidate [baseline-selector candidate-selector]]
[[(resolve-case baseline baseline-selector) (resolve-case candidate candidate-selector)]])
(defn- difference
"Returns one compatibility difference when recorded values disagree."

182
frontend/test/performance_tests/render_wasm/cli_integration.cjs
View File

@ -154,6 +154,125 @@ function readManifest(file) {
return transit.reader("json").read(fs.readFileSync(file, "utf8"));
}
function requestText(command, options, color = "never") {
const k = transit.keyword;
return transit.writer("json").write(transit.map([
k("command"), command,
k("console"), transit.map([k("level"), k("info"), k("color"), k(color)]),
k("options"), transit.map(Object.entries(options).flatMap(([key, value]) => [k(key), value])),
]));
}
test("deferred transport carries a complete request and numeric status", testOptions, async (t) => {
const fixture = createFrontend(t);
const capture = path.join(fixture.managed, "request.transit");
fs.writeFileSync(path.join(fixture.managed, "commands.cjs"), `
module.exports.execute = (text) => {
if (typeof text !== "string") throw new Error("expected Transit text");
require("node:fs").writeFileSync(${JSON.stringify(capture)}, text);
return Promise.resolve(37);
};
`);
const result = await runCli(fixture, ["run", "--build", "ready", "--warmups", "0", "--quiet", "--no-color"]);
assert.equal(result.exitCode, 37, result.stderr);
const k = transit.keyword;
const request = readManifest(capture);
assert.equal(request.get(k("command")), "run");
assert.equal(request.get(k("console")).get(k("level")).toString(), ":warn");
assert.equal(request.get(k("console")).get(k("color")).toString(), ":never");
const options = request.get(k("options"));
assert.equal(options.get(k("build")).get(k("kind")).toString(), ":existing");
assert.equal(options.get(k("build")).get(k("id")), "ready");
assert.equal(options.get(k("warmups")), 0);
assert.equal(options.get(k("repetitions")), 10);
assert.equal(options.get(k("startup-timeout-ms")), 30000);
});
test("decoded requests reject incomplete, unsupported and policy-invalid commands before acquisition", testOptions, async (t) => {
const fixture = createFrontend(t);
fs.unlinkSync(path.resolve(fixture.managed, "../../../deps.edn"));
const entry = path.join(fixture.managed, "receive.cjs");
fs.writeFileSync(entry, `
require("./commands.cjs").execute(process.env.COMMAND_REQUEST)
.then(code => { process.exitCode = code; });
`);
const before = tree(fixture.managed);
const measurement = { seed: 42, filter: null, warmups: 3, repetitions: 10,
"timeout-ms": 30000, "startup-timeout-ms": 30000, "output-dir": null,
screenshots: false, "require-gpu": false };
const ab = { ...measurement, features: ["a"], "max-features": 3, confirm: false,
"dry-run": false, "build-timeout-ms": 300000, "allow-build-differences": false,
"allow-scene-differences": false, "allow-all-differences": false };
const k = transit.keyword;
const build = { "default-features": true, features: [], "build-timeout-ms": 300000 };
for (const [text, diagnostic] of [
["invalid Transit", /JSON/],
[requestText("build", {}), /default-features|build-timeout-ms/],
[requestText("purge", { target: k("all") }, "always"), /unsupported-command/],
[requestText("build", { ...build, features: ["profile"] }), /diagnostic-features/],
[requestText("run", { ...measurement, remove: false,
build: transit.map([k("kind"), k("prepare"), ...Object.entries({ ...build, features: ["stats"] }).flatMap(([key, value]) => [k(key), value])]) }), /diagnostic-features/],
[requestText("ab", ab), /confirmation-required/],
[requestText("ab", { ...ab, features: ["a", "b"], "max-features": 1, "dry-run": true }), /too-many-features/],
]) {
fixture.env.COMMAND_REQUEST = text;
const result = await runCli({ ...fixture, runner: entry }, []);
assert.equal(result.exitCode, 1, result.stderr);
assert.equal(result.stdout, "");
assert.match(result.stderr, diagnostic);
assert.equal((result.stderr.match(/✖/g) ?? []).length, 1);
assert.deepEqual(tree(fixture.managed), before);
}
});
test("final publication failures select a failing status in local and deferred commands", testOptions, async (t) => {
for (const command of [["--help"], ["build"]]) {
const fixture = createFrontend(t);
if (command[0] === "build") installPreparedCompiler(fixture, false);
const guard = path.join(fixture.managed, "writer.cjs");
fs.writeFileSync(guard, `process.stdout.write = () => { throw new Error("publication unavailable"); };`);
fixture.nodeArgs = ["--require", guard];
const result = await runCli(fixture, command);
assert.equal(result.exitCode, 1, result.stderr);
assert.match(result.stderr, /publication unavailable/);
assert.equal((result.stderr.match(/✖/g) ?? []).length, 1);
if (command[0] === "build") {
const [id] = fs.readdirSync(fixture.builds);
assert.equal(readManifest(path.join(fixture.builds, id, "build.transit")).get(transit.keyword("status")).toString(), ":ready");
}
}
});
test("merge and purge execute with unavailable process, HTTP and browser dependencies", testOptions, async (t) => {
const fixture = createFrontend(t);
const guard = path.join(fixture.managed, "offline.cjs");
fs.writeFileSync(guard, `
const Module = require("node:module");
const load = Module._load;
Module._load = function (id, ...args) {
if (["playwright", "http", "node:http", "child_process", "node:child_process"].includes(id)) {
throw new Error("Unavailable dependency: " + id);
}
return load.call(this, id, ...args);
};
`);
fixture.nodeArgs = ["--require", guard];
const base = path.join(fixture.managed, "base.transit");
const additional = path.join(fixture.managed, "additional.transit");
for (const [file, runId, value] of [[base, "base", 10], [additional, "additional", 12]]) {
fs.writeFileSync(file, runFixtureText({ runId, caseId: "case/load", value, gitSha: "same" }));
}
const merged = await runCli(fixture, ["merge", base, additional]);
assert.equal(merged.exitCode, 0, merged.stderr);
assert.match(merged.stdout, /^Run .+\nOutput .+\n$/);
seedBuilds(fixture);
const purged = await runCli(fixture, ["purge", "builds"]);
assert.equal(purged.exitCode, 0, purged.stderr);
for (const id of fs.readdirSync(fixture.builds)) {
assert.deepEqual(fs.readdirSync(path.join(fixture.builds, id)), ["keep.txt"]);
}
});
function manifestText(id, status, type) {
const k = transit.keyword;
return transit
@ -727,7 +846,7 @@ test(
fs.readFileSync(logPath, "utf8"),
/compiler stderr before failure/,
);
await runCli(fixture, ["purge", "--only-comparisons"]);
await runCli(fixture, ["purge", "comparisons"]);
assert.equal(fs.existsSync(logPath), false);
},
);
@ -782,7 +901,13 @@ test(
command === "ab"
? ["ab", "--features", "a", "--filter", "rects/load"]
: ["run"];
const child = spawn(process.execPath, [fixture.runner, ...args], {
const failPublication = command === "run" && signal === "SIGINT";
if (failPublication) {
const writer = path.join(fixture.managed, "writer.cjs");
fs.writeFileSync(writer, `process.stdout.write = () => { throw new Error("publication unavailable"); };`);
fixture.nodeArgs = ["--require", writer];
}
const child = spawn(process.execPath, [...(fixture.nodeArgs ?? []), fixture.runner, ...args], {
cwd: os.tmpdir(),
env: { ...process.env, ...fixture.env },
stdio: ["ignore", "pipe", "pipe"],
@ -806,6 +931,7 @@ test(
assert.ok(fs.existsSync(marker), stderr);
child.kill(signal);
assert.deepEqual(await exited, [code, null], stderr);
if (failPublication) assert.match(stderr, /publication unavailable/);
}
}
},
@ -906,7 +1032,7 @@ test("purge deletes only the selected artifacts", testOptions, async (t) => {
const runs = tree(fixture.runs);
const comparisons = tree(fixture.comparisons);
assert.deepEqual(await runCli(fixture, ["purge", "--only-builds"]), {
assert.deepEqual(await runCli(fixture, ["purge", "builds"]), {
exitCode: 0,
stdout: "",
stderr: "",
@ -927,7 +1053,7 @@ test("purge deletes only the selected artifacts", testOptions, async (t) => {
seedBuilds(fixture);
const builds = tree(fixture.builds);
assert.deepEqual(await runCli(fixture, ["purge", "--only-runs"]), {
assert.deepEqual(await runCli(fixture, ["purge", "runs"]), {
exitCode: 0,
stdout: "",
stderr: "",
@ -946,7 +1072,7 @@ test("purge deletes only the selected artifacts", testOptions, async (t) => {
const buildsBeforeComparisonPurge = tree(fixture.builds);
const runsBeforeComparisonPurge = tree(fixture.runs);
assert.deepEqual(await runCli(fixture, ["purge", "--only-comparisons"]), {
assert.deepEqual(await runCli(fixture, ["purge", "comparisons"]), {
exitCode: 0,
stdout: "",
stderr: "",
@ -982,7 +1108,7 @@ test("purge deletes only the selected artifacts", testOptions, async (t) => {
seedRuns(fixture);
seedComparisons(fixture);
assert.deepEqual(await runCli(fixture, ["purge"]), {
assert.deepEqual(await runCli(fixture, ["purge", "all"]), {
exitCode: 0,
stdout: "",
stderr: "",
@ -1118,18 +1244,16 @@ test(
const original = tree(fixture.managed);
for (const args of [
["purge", "--only-builds", "--only-runs"],
["purge", "--only-builds", "--only-comparisons"],
["purge", "--only-runs", "--only-comparisons"],
["purge", "--only-ready-builds"],
["purge", "--only-builds", "ready"],
["purge"],
["purge", "unknown"],
["purge", "builds", "runs"],
]) {
const result = await runCli(fixture, args);
assert.equal(result.exitCode, 1, args.join(" "));
assert.equal(result.stdout, "");
assert.match(
result.stderr,
/type\s+:benches.render-wasm.node.runner\/invalid-args/,
/Usage: runner purge/,
);
assert.deepEqual(tree(fixture.managed), original, args.join(" "));
}
@ -1216,19 +1340,14 @@ fs.writeFileSync(${JSON.stringify(marker)}, "ran");
);
const original = tree(fixture.managed);
const result = await runCli(fixture, [
"run",
"--build",
"retained-id",
"--features",
"functional",
]);
assert.equal(result.exitCode, 1);
assert.equal(result.stdout, "");
assert.match(result.stderr, /cannot be combined/);
assert.equal(fs.existsSync(marker), false);
assert.deepEqual(tree(fixture.managed), original);
for (const flags of [["--features", "functional"], ["--build-timeout-ms", "1000"]]) {
const result = await runCli(fixture, ["run", "--build", "retained-id", ...flags]);
assert.equal(result.exitCode, 1);
assert.equal(result.stdout, "");
assert.match(result.stderr, /cannot be combined/);
assert.equal(fs.existsSync(marker), false);
assert.deepEqual(tree(fixture.managed), original);
}
},
);
@ -1280,7 +1399,7 @@ test(
const abHelp = await runCli(fixture, ["ab", "--help"]);
assert.equal(abHelp.exitCode, 0, abHelp.stderr);
assert.match(abHelp.stdout, /runner ab --features/);
assert.match(abHelp.stdout, /--only-comparisons/);
assert.match(abHelp.stdout, /--max-features/);
const mergeHelp = await runCli(fixture, ["merge", "--help"]);
assert.equal(mergeHelp.exitCode, 0, mergeHelp.stderr);
assert.match(mergeHelp.stdout, /runner merge BASE ADDITIONAL/);
@ -1327,14 +1446,15 @@ test(
assert.match(result.stdout, /runner run/);
assert.match(result.stdout, /runner compare/);
assert.match(result.stdout, /runner merge/);
assert.match(result.stdout, /--cases/);
assert.match(result.stdout, /--allow-build-differences/);
assert.match(result.stdout, /--allow-scene-differences/);
assert.match(result.stdout, /--allow-all-differences/);
const compareHelp = await runCli(fixture, ["compare", "--help"]);
assert.equal(compareHelp.exitCode, 0, compareHelp.stderr);
assert.match(compareHelp.stdout, /--cases/);
assert.match(compareHelp.stdout, /--allow-build-differences/);
assert.match(compareHelp.stdout, /--allow-scene-differences/);
assert.match(compareHelp.stdout, /--allow-all-differences/);
assert.match(result.stdout, /-q, --quiet/);
assert.match(result.stdout, /-v, --verbose/);
assert.match(result.stdout, /--\[no-\]color/);
assert.doesNotMatch(result.stdout, /--force/);
assert.deepEqual(tree(fixture.managed), original);
},
);

2
frontend/test/performance_tests/render_wasm/cli_test.cljs
View File

@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns performance-tests.render-wasm.cli-test
(:require

10
frontend/test/performance_tests/render_wasm/compare_test.cljs
View File

@ -125,7 +125,7 @@
candidate (write-run (path/join @temporary-root "candidate.transit")
(make-run "candidate" :b/pan 12))
{:keys [text]} (compare/execute baseline candidate
{:cases "a/pan,b/pan"})]
{:case-pair ["a/pan" "b/pan"]})]
(t/is (str/includes? text "Pair a/pan,b/pan"))))
(t/deftest cases-requires-scene-permission-for-different-scenes
@ -134,7 +134,7 @@
candidate (write-run (path/join @temporary-root "candidate.transit")
(make-run "candidate" :b/load 12 :scene :other
:fingerprint (apply str (repeat 64 "b"))))
selector {:cases "a/load,b/load"}]
selector {:case-pair ["a/load" "b/load"]}]
(t/is (= ::report-compare/incompatible
(:type (ex-data (failure #(compare/execute baseline candidate selector))))))
(t/is (= ::report-compare/incompatible
@ -254,7 +254,7 @@
(t/is (str/includes? text "ACKNOWLEDGED DIFFERENCE"))
(t/is (str/includes? text ":rule :all")))))
(t/deftest invalid-selectors-and-unreadable-records-fail-at-the-boundary
(t/deftest unreadable-records-fail-at-the-boundary
(let [baseline (write-run (path/join @temporary-root "baseline.transit")
(make-run "baseline" :case/load 10))
candidate (write-run (path/join @temporary-root "candidate.transit")
@ -264,10 +264,6 @@
(fs/writeFileSync file (result/encode record))
(t/is (thrown-with-msg? js/Error #"expected a saved run record"
(compare/execute baseline file {}))))
(doseq [selector ["case/load" "case/load,other/load,third/load" ""]]
(t/is (= ::report-compare/invalid-selector
(:type (ex-data (failure #(compare/execute baseline candidate
{:cases selector})))))))
(fs/writeFileSync (path/join @temporary-root "broken.transit") "not transit")
(t/is (thrown? js/Error (compare/execute baseline
(path/join @temporary-root "broken.transit")

15
frontend/test/performance_tests/render_wasm/execution_test.cljs
View File

@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns performance-tests.render-wasm.execution-test
(:require
@ -30,7 +30,7 @@
#js [promise
(js/Promise.
(fn [_ reject]
(vreset! timer (js/setTimeout #(reject (js/Error. "Signal test stalled")) 5000))))])
(vreset! timer (js/setTimeout #(reject (js/Error. "Test stalled")) 5000))))])
(.finally #(js/clearTimeout @timer)))))
(t/deftest ^:async real-signals-reach-the-callback-and-remover-restores-listeners
@ -38,7 +38,7 @@
before-term (.listenerCount js/process "SIGTERM")
received (atom [])
next-signal (volatile! nil)
remove (signals/on-signals
remove (execution/on-signals
(fn [signal]
(swap! received conj signal)
(@next-signal signal)))
@ -83,15 +83,6 @@
[]
[(.listenerCount js/process "SIGINT") (.listenerCount js/process "SIGTERM")])
(defn- within
[promise]
(let [timer (volatile! nil)]
(-> (js/Promise.race
#js [promise
(js/Promise.
(fn [_ reject]
(vreset! timer (js/setTimeout #(reject (js/Error. "Execution test stalled")) 5000))))])
(.finally #(js/clearTimeout @timer)))))
(t/deftest ^:async sync-and-async-handlers-publish-once-and-release-listeners
(doseq [async [false true]]

13
frontend/test/performance_tests/render_wasm/feature_comparison_test.cljs
View File

@ -21,6 +21,7 @@
[benches.render-wasm.node.files :as files]
[benches.render-wasm.node.log :as log]
[benches.render-wasm.node.process :as process]
[benches.render-wasm.node.request :as request]
[benches.render-wasm.result :as result]
[cljs.test :as t :include-macros true]
[clojure.string :as str]
@ -50,9 +51,11 @@
(defn- planning-options
[]
{:features ["b" "a"] :filter "rects/load" :seed 42 :dry-run true
:warmups 0 :repetitions 1 :timeout-ms 2000
:build-timeout-ms 10000 :startup-timeout-ms 3000})
(:options (request/apply-defaults
{:command "ab"
:options {:features ["b" "a"] :filter "rects/load" :seed 42 :dry-run true
:warmups 0 :repetitions 1 :timeout-ms 2000
:build-timeout-ms 10000 :startup-timeout-ms 3000}})))
(defn- metadata-output
[definitions]
@ -153,8 +156,7 @@
ready (await (node-plan/prepare (frontend-root) opts nil))]
(t/is (= ["a" "b"] (get-in ready [:plan :features])))
(t/is (= (cases/collect-cases {:master-seed 42 :filter "rects/load"}) (:cases ready)))
(t/is (= (assoc opts :features ["a" "b"] :default-features false :max-features 3 :confirm false
:allow-build-differences false :allow-scene-differences false :allow-all-differences false)
(t/is (= (assoc opts :features ["a" "b"])
(:options ready)))
(t/is (= output (:output-dir ready)))
(t/is (not (fs/existsSync (path/join @temporary-root "new"))))
@ -1051,7 +1053,6 @@
:execute-run (execute-service {})})})))
record (:comparison result)]
(t/is (= 1 (:exit-code result)))
(t/is (str/ends-with? @stdout "Status failed\n"))
(t/is (seq (:errors result)))
(t/is (= :running (:status record)))
(t/is (= :building (get-in record [:configurations 0 :status])))

5
frontend/test/performance_tests/render_wasm/report_test.cljs
View File

@ -543,7 +543,7 @@
:prep (assoc-in preparation [:fingerprint :digest] (apply str (repeat 64 "b")))})
result/encode
result/decode)
pairs (compare/selector-pair base other "saved/pan,different/zoom")
pairs (compare/selector-pair base other ["saved/pan" "different/zoom"])
rejected (error-data #(compare/compare-runs base other {:pairs pairs}))
comparison (-> (compare/compare-runs base other {:pairs pairs :allow-all-differences true})
result/encode result/decode)
@ -580,8 +580,7 @@
(t/is (every? :waived? mismatches))
(t/is (str/includes? text "ACKNOWLEDGED DIFFERENCE"))
(t/is (str/includes? text ":field :scored?")))
(t/is (= ["saved/pan"] (:available (error-data #(compare/resolve-case base "pan")))))
(t/is (= ::compare/invalid-selector (:type (error-data #(compare/selector-pair base other "saved/pan")))))))
(t/is (= ["saved/pan"] (:available (error-data #(compare/resolve-case base "pan")))))))
(t/deftest explicit-pairs-allow-unlike-names-with-matching-execution-and-fingerprints
(let [base (-> (run-with [1]) result/encode result/decode)

55
frontend/test/performance_tests/render_wasm/run_test.cljs
View File

@ -15,6 +15,7 @@
[benches.render-wasm.node.artifacts :as artifacts]
[benches.render-wasm.node.feature-comparison :as feature-comparison]
[benches.render-wasm.node.log :as log]
[benches.render-wasm.node.request :as request]
[benches.render-wasm.node.run :as run]
[benches.render-wasm.node.runs :as runs]
[benches.render-wasm.result :as result]
@ -180,7 +181,6 @@
:warmups 0
:repetitions 1
:timeout-ms 5000
:build-timeout-ms 60000
:startup-timeout-ms 5000
:screenshots false
:remove false
@ -208,46 +208,7 @@
[result]
(result/decode (fs/readFileSync (get-in result [:home :record-path]) "utf8")))
(t/deftest ^:async run-rejects-invalid-options-before-allocating-output
(doseq [invalid [{:seed 4294967296}
{:warmups -1}
{:repetitions 1.5}
{:timeout-ms js/Infinity}
{:build-timeout-ms 2147483648}
{:startup-timeout-ms 0}
{:screenshots "false"}
{:remove nil}
{:require-gpu 1}
{:repetitons 1}]]
(let [out (output-dir)
opts (merge (dissoc (options nil out) :browser)
{:build-id "unavailable"} invalid)
cause (try
(await (run/run @temporary-root opts nil))
nil
(catch :default cause cause))]
(t/testing (pr-str invalid)
(t/is (= ::run/invalid-options (:type (ex-data cause))))
(t/is (not (fs/existsSync out)))))))
(t/deftest ^:async execute-rejects-invalid-options-before-writing-evidence
(let [root @temporary-root
fake (fake-browser {})
home (runs/create-home {:root root :run-id "invalid-options"
:output-dir (output-dir)})
files (vec (fs/readdirSync (:dir home)))
log-text (fs/readFileSync (:log-path home) "utf8")
opts (assoc (options fake (:dir home)) :screenshots "false")
cause (try
(await (run/execute root home (selected-build (build-dir) "test-build")
[(collected :case/a :fresh)] opts nil))
nil
(catch :default cause cause))]
(t/is (= ::run/invalid-options (:type (ex-data cause))))
(t/is (= files (vec (fs/readdirSync (:dir home)))))
(t/is (= log-text (fs/readFileSync (:log-path home) "utf8")))))
(t/deftest ^:async retained-run-prints-results-and-keeps-all-levels-in-its-log
(t/deftest ^:async retained-run-keeps-all-progress-levels-in-its-log
(doseq [level [:info :warn :debug]]
(let [root @temporary-root
cases [(collected :case/a :fresh) (collected :case/b :reuse)]
@ -272,19 +233,21 @@
(log/configure {:level level :color :never
:stdout #js {:write #(swap! stdout str %)}
:stderr #js {:write #(swap! stderr str %)}})
(let [response (await (run/run root (assoc (options fake out) :build-id (:id build)
:warmups 1 :repetitions 2) nil))
(let [resolved (:options (request/apply-defaults
{:command "run"
:options (assoc (dissoc (options fake out) :browser)
:build {:kind :existing :id (:id build)}
:warmups 1 :repetitions 2)}))
response (await (run/run root resolved nil {:browser (:service fake)}))
saved (saved-record response)
text (fs/readFileSync (get-in response [:home :log-path]) "utf8")]
(t/is (= 0 (:exit-code response)))
(t/is (= (str "Run " (:run-id saved) "\nOutput " out "\nReport\n"
(str/replace (:report response) #"\n$" "") "\n") @stdout))
(t/is (= (:id build) (get-in saved [:metadata :build :provenance :build-id])))
(t/is (str/includes? text "▸ Case case/a (fresh)\n"))
(doseq [case-text (rest (str/split text #"▸ Case "))]
(t/is (str/includes? case-text " attempt 1/3 completed (warmup)\n"))
(t/is (str/includes? case-text " attempt 2/3 completed\n"))
(t/is (str/includes? case-text " attempt 3/3 completed\n")))
(t/is (str/includes? text (:report response)))
(t/is (str/includes? text "Browser FakeBrowser/1"))
(t/is (= (not= :warn level) (str/includes? @stderr "▸ Case case/a")))
(t/is (= (= :debug level) (str/includes? @stderr "Browser FakeBrowser/1")))))))

8
frontend/test/performance_tests/render_wasm/runner.cljs
View File

@ -34,11 +34,9 @@
[performance-tests.render-wasm.report-test]
[performance-tests.render-wasm.request-test]
[performance-tests.render-wasm.run-test]
[performance-tests.render-wasm.runner-test]
[performance-tests.render-wasm.runs-test]
[performance-tests.render-wasm.scene-snapshot-test]
[performance-tests.render-wasm.server-test]
[performance-tests.render-wasm.signals-test]))
[performance-tests.render-wasm.server-test]))
(enable-console-print!)
@ -76,11 +74,9 @@
'performance-tests.render-wasm.report-test
'performance-tests.render-wasm.request-test
'performance-tests.render-wasm.run-test
'performance-tests.render-wasm.runner-test
'performance-tests.render-wasm.runs-test
'performance-tests.render-wasm.scene-snapshot-test
'performance-tests.render-wasm.server-test
'performance-tests.render-wasm.signals-test])
'performance-tests.render-wasm.server-test])
(assert (every? find-ns-obj test-namespaces)
"test-namespaces contains a namespace that isn't required in runner.cljs")

330
frontend/test/performance_tests/render_wasm/runner_test.cljs
View File

@ -1,330 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns performance-tests.render-wasm.runner-test
(:require
[benches.render-wasm.node.options :as options]
[benches.render-wasm.node.runner :as runner]
[cljs.test :as t :include-macros true]
[clojure.string :as str]))
(t/deftest output-options-apply-to-every-command
(doseq [args [["build"] ["run"] ["purge"] ["compare" "base" "candidate"]
["merge" "base" "additional"]
["ab" "--features" "a" "--filter" "rects/load"]]]
(t/is (= :info (options/console-level (:options (runner/parse-args args)))))
(t/is (= :auto (options/color-mode (:options (runner/parse-args args)))))
(t/is (= :warn (options/console-level (:options (runner/parse-args (conj args "-q"))))))
(t/is (= :debug (options/console-level (:options (runner/parse-args (conj args "-v"))))))
(t/is (= :always (options/color-mode (:options (runner/parse-args (conj args "--color"))))))
(t/is (= :never (options/color-mode (:options (runner/parse-args (conj args "--no-color")))))))
(t/is (thrown-with-msg? js/Error #"mutually exclusive"
(runner/parse-args ["run" "-q" "-v"]))))
(t/deftest build-parses-defaults-and-explicit-functional-feature-selection
(let [{:keys [command options]} (runner/parse-args ["build"])]
(t/is (= "build" command))
(t/is (true? (:default-features options)))
(t/is (= options/default-build-timeout-ms (:build-timeout-ms options))))
(let [{:keys [command options]} (runner/parse-args
["--" "build" "--no-default-features"
"--features" "functional_b,functional-a,functional_b"
"--build-timeout-ms" "1250"])]
(t/is (= "build" command))
(t/is (= {:default-features false
:features ["functional-a" "functional_b"]
:build-timeout-ms 1250}
(select-keys options [:default-features :features :build-timeout-ms])))))
(t/deftest diagnostic-and-malformed-features-fail-at-the-parser-boundary
(doseq [feature ["stats" "profile" "profile-macros" "profile-raf"]]
(t/is (thrown-with-msg? js/Error #"diagnostic Cargo features"
(runner/parse-args ["build" "--features" (str "functional," feature)]))))
(doseq [feature ["" "functional,,other" "functional/other"]]
(t/is (thrown-with-msg? js/Error #"invalid Cargo feature selection"
(runner/parse-args ["build" "--features" feature]))))
(t/is (thrown? js/Error (runner/parse-args ["build" "--features"]))))
(t/deftest preparation-budget-requires-a-complete-positive-safe-integer-number
;; Number accepts whitespace and exponent notation when the result is an
;; integer; these are valid CLI budgets, not partial numeric parses.
(doseq [[text expected] [["1" 1] [" 1000 " 1000] ["1e3" 1000]]]
(t/is (= expected (get-in (runner/parse-args ["build" "--build-timeout-ms" text])
[:options :build-timeout-ms]))))
(doseq [text ["" "0" "-1" "1.5" "1000ms" "NaN" "Infinity" "9007199254740992"]]
(t/is (thrown-with-msg? js/Error #"Must be a positive integer"
(runner/parse-args ["build" "--build-timeout-ms" text]))))
(t/is (thrown? js/Error (runner/parse-args ["build" "--build-timeout-ms"]))))
(t/deftest purge-categories-and-build-flags-cannot-be-mixed
(doseq [[args selected] [[["purge"] {}]
[["purge" "--only-builds"] {:only-builds true}]
[["purge" "--only-runs"] {:only-runs true}]]]
(let [{:keys [command options]} (runner/parse-args args)]
(t/is (= "purge" command))
(t/is (= selected (select-keys options [:only-builds :only-runs])))))
(t/is (thrown-with-msg? js/Error #"mutually exclusive"
(runner/parse-args ["purge" "--only-builds" "--only-runs"])))
(doseq [flag ["--only-builds" "--only-runs"]]
(t/is (thrown-with-msg? js/Error #"purge category flags require purge"
(runner/parse-args ["build" flag]))))
(doseq [args [["purge" "--default-features"]
["purge" "--no-default-features"]
["purge" "--features" "functional"]
["purge" "--build-timeout-ms" "1000"]
["purge" "--features=functional"]
["purge" "--build-timeout-ms=1000"]]]
(t/is (thrown-with-msg? js/Error #"build options require build"
(runner/parse-args args)))))
(t/deftest help-needs-no-command-and-other-command-lines-must-be-valid
(doseq [args [["--help"] ["-h"] ["build" "--help"] ["run" "--help"]
["compare" "--help"] ["merge" "--help"]]]
(let [{:keys [options summary]} (runner/parse-args args)
help (runner/usage summary)]
(t/is (true? (:help options)))
(t/is (str/includes? help "runner build"))
(t/is (str/includes? help "runner purge"))
(t/is (str/includes? help "runner run"))
(t/is (str/includes? help "runner compare"))
(t/is (str/includes? help "runner merge"))
(t/is (str/includes? help "--build-timeout-ms"))
(t/is (str/includes? help "--cases"))
(t/is (str/includes? help "--allow-build-differences"))
(t/is (str/includes? help "--allow-scene-differences"))
(t/is (str/includes? help "--allow-all-differences"))
(t/is (not (str/includes? help "--force")))))
(t/is (thrown-with-msg? js/Error #"expected build, purge, run, ab, compare or merge"
(runner/parse-args [])))
(doseq [args [["unknown"] ["build" "purge"] ["build" "--unknown"]
["run" "extra"] ["compare" "only-one"]]]
(t/is (thrown? js/Error (runner/parse-args args)))))
(t/deftest run-resolves-default-and-explicit-options-before-resources
(let [{:keys [command options arguments]} (runner/parse-args ["run"])]
(t/is (= "run" command))
(t/is (= ["run"] arguments))
(t/is (= {:seed 42 :warmups 3 :repetitions 10 :timeout-ms 30000
:build-timeout-ms 300000 :startup-timeout-ms 30000}
(select-keys options [:seed :warmups :repetitions :timeout-ms
:build-timeout-ms :startup-timeout-ms])))
(t/is (nil? (:build-id options)))
(t/is (nil? (:filter options)))
(t/is (not (:remove options)))
(t/is (not (:screenshots options))))
(let [{:keys [options]} (runner/parse-args
["run" "--seed" "7" "--filter" "rects"
"--warmups" "0" "--repetitions" "2"
"--timeout-ms" "100" "--build-timeout-ms" "200"
"--startup-timeout-ms" "300" "--output-dir" "out"
"--remove" "--screenshots"])]
(t/is (= {:seed 7 :filter "rects" :warmups 0 :repetitions 2
:timeout-ms 100 :build-timeout-ms 200 :startup-timeout-ms 300
:output-dir "out" :remove true :screenshots true}
(select-keys options [:seed :filter :warmups :repetitions
:timeout-ms :build-timeout-ms
:startup-timeout-ms :output-dir
:remove :screenshots])))))
(t/deftest retained-runs-reject-feature-selection-and-build-runs-accept-it
(t/is (= "retained-id" (get-in (runner/parse-args ["run" "--build" "retained-id"])
[:options :build-id])))
(doseq [args [["run" "--build" "retained-id" "--no-default-features"]
["run" "--no-default-features" "--build" "retained-id"]
["run" "--build" "retained-id" "--features" "functional"]
["run" "--build" "retained-id" "--default-features"]]]
(t/is (thrown-with-msg? js/Error #"cannot be combined"
(runner/parse-args args))))
(let [{:keys [options]} (runner/parse-args
["run" "--no-default-features" "--features" "functional"])]
(t/is (false? (:default-features options)))
(t/is (= ["functional"] (:features options)))))
(t/deftest compare-takes-two-file-paths-and-its-own-options
(let [{:keys [command arguments options]}
(runner/parse-args ["compare" "baseline.transit" "candidate.transit"])]
(t/is (= "compare" command))
(t/is (= ["compare" "baseline.transit" "candidate.transit"] arguments))
(t/is (nil? (:cases options)))
(t/is (not-any? options [:allow-build-differences :allow-scene-differences
:allow-all-differences])))
(doseq [[flag option] [["--allow-build-differences" :allow-build-differences]
["--allow-scene-differences" :allow-scene-differences]
["--allow-all-differences" :allow-all-differences]]
selectors [[] ["--cases" "rects/pan,paths/pan"]]]
(let [{:keys [options]}
(runner/parse-args (into ["compare" "a.transit" "b.transit" flag] selectors))]
(t/is (= (when (seq selectors) "rects/pan,paths/pan") (:cases options)))
(t/is (true? (get options option)))
(t/is (not-any? options (remove #{option} [:allow-build-differences
:allow-scene-differences
:allow-all-differences])))))
(let [{:keys [options]}
(runner/parse-args ["compare" "a.transit" "b.transit"
"--allow-build-differences" "--allow-scene-differences"])]
(t/is (true? (:allow-build-differences options)))
(t/is (true? (:allow-scene-differences options)))
(t/is (not (:allow-all-differences options)))))
(t/deftest merge-takes-two-paths-and-only-output-dir
(let [{:keys [command arguments options]}
(runner/parse-args ["merge" "base.transit" "additional.transit"
"--output-dir" "merged-home"])]
(t/is (= "merge" command))
(t/is (= ["merge" "base.transit" "additional.transit"] arguments))
(t/is (= "merged-home" (:output-dir options))))
(doseq [args [["merge"] ["merge" "only-base"]
["merge" "base" "additional" "extra"]]]
(t/is (thrown? js/Error (runner/parse-args args))))
(doseq [flags [["--build" "id"]
["--filter" "rects"]
["--seed" "1"]
["--require-gpu"]
["--features" "functional"]
["--cases" "a,b"]
["--allow-build-differences"]
["--only-builds"]
["--dry-run"]]]
(t/is (thrown? js/Error (runner/parse-args (into ["merge" "base" "additional"] flags))))))
(t/deftest force-is-rejected-without-an-alias
(doseq [args [["compare" "a.transit" "b.transit" "--force"]
["compare" "a.transit" "b.transit" "--force" "--allow-all-differences"]
["compare" "a.transit" "b.transit" "--cases" "a,b" "--force"]]]
(t/is (thrown? js/Error (runner/parse-args args)))))
(t/deftest commands-reject-flags-owned-by-other-commands
(doseq [[args message]
[[["run" "--only-builds"] #"purge category flags require purge"]
[["run" "--cases" "a,b"] #"compare options require compare"]
[["compare" "a" "b" "--seed" "1"] #"run options require run"]
[["compare" "a" "b" "--features" "functional"] #"build options require build"]
[["build" "--build" "retained"] #"run options require run"]
[["build" "--only-runs"] #"purge category flags require purge"]
[["purge" "--seed" "1"] #"run options require run"]
[["purge" "--cases" "a,b"] #"compare options require compare"]
[["merge" "a" "b" "--seed" "1"] #"run options require run"]
[["merge" "a" "b" "--features" "functional"] #"build options require build"]
[["merge" "a" "b" "--cases" "a,b"] #"compare options require compare"]
[["merge" "a" "b" "--only-builds"] #"purge category flags require purge"]
[["merge" "a" "b" "--dry-run"] #"ab planning options require ab"]]]
(t/is (thrown-with-msg? js/Error message (runner/parse-args args))))
(doseq [command ["build" "purge" "run"]
flag ["--allow-build-differences" "--allow-scene-differences"
"--allow-all-differences"]]
(t/is (thrown-with-msg? js/Error #"compare options require compare"
(runner/parse-args [command flag])))))
(t/deftest numeric-and-path-options-reject-invalid-bounds
(doseq [args [["run" "--seed" "4294967296"]
["run" "--seed" "-1"]
["run" "--warmups" "-1"]
["run" "--warmups" "1.5"]
["run" "--repetitions" "0"]
["run" "--timeout-ms" "0"]
["run" "--timeout-ms" "2147483648"]
["run" "--startup-timeout-ms" "0"]
["run" "--build" ""]
["run" "--output-dir" ""]]]
(t/is (thrown? js/Error (runner/parse-args args))))
(t/is (= 4294967295 (:seed (:options (runner/parse-args ["run" "--seed" "4294967295"])))))
(t/is (= 0 (:warmups (:options (runner/parse-args ["run" "--warmups" "0"]))))))
(t/deftest option-looking-values-and-filenames-are-not-command-flags
(let [parsed (runner/parse-args ["compare" "--" "--seed" "candidate.transit"])]
(t/is (= ["compare" "--seed" "candidate.transit"] (:arguments parsed))))
(let [parsed (runner/parse-args ["run" "--filter" "--allow-all-differences"])]
(t/is (= "--allow-all-differences" (:filter (:options parsed))))))
(t/deftest numeric-options-reject-blank-values
(doseq [args [["run" "--seed" ""]
["run" "--warmups" " "]
["run" "--repetitions" ""]
["run" "--timeout-ms" " "]]]
(t/is (thrown? js/Error (runner/parse-args args)))))
(t/deftest gpu-policy-is-available-for-runs-and-feature-comparisons
(doseq [args [["run" "--require-gpu"]
["ab" "--features" "a" "--filter" "rects/load" "--require-gpu"]
["ab" "--features" "a" "--require-gpu" "--dry-run"]]]
(t/is (true? (get-in (runner/parse-args args) [:options :require-gpu]))))
(doseq [args [["build" "--require-gpu"]
["purge" "--require-gpu"]
["compare" "a" "b" "--require-gpu"]]]
(t/is (thrown-with-msg? js/Error #"run options require run" (runner/parse-args args))))
(t/is (re-find #"--require-gpu" (runner/usage ""))))
(t/deftest ab-accepts-run-budgets-and-comparison-permissions
(let [{:keys [command options]}
(runner/parse-args ["ab" "--features" " b,a,b " "--filter" "rects/load"
"--seed" "7" "--warmups" "0" "--repetitions" "2"
"--timeout-ms" "1000" "--build-timeout-ms" "2000"
"--startup-timeout-ms" "3000" "--output-dir" "new-home"
"--screenshots" "--allow-build-differences"
"--allow-scene-differences" "--allow-all-differences"])]
(t/is (= "ab" command))
(t/is (= {:features ["a" "b"] :filter "rects/load" :seed 7
:warmups 0 :repetitions 2 :timeout-ms 1000 :build-timeout-ms 2000
:startup-timeout-ms 3000 :output-dir "new-home" :screenshots true
:allow-build-differences true :allow-scene-differences true :allow-all-differences true
:max-features 3 :confirm false :dry-run false :default-features false :color nil}
options))))
(t/deftest ab-rejects-missing-features-and-incompatible-flags-before-execution
(t/is (thrown-with-msg? js/Error #"at least one" (runner/parse-args ["ab" "--dry-run"])))
(doseq [args [["--build" "id"] ["--remove"] ["--cases" "a,b"]
["--default-features"] ["--no-default-features"]]]
(t/is (thrown-with-msg? js/Error #"ab rejects"
(runner/parse-args (into ["ab" "--features" "a" "--dry-run"] args)))))
(doseq [features ["" "a," "profile" "a/b"]]
(t/is (thrown? js/Error (runner/parse-args ["ab" "--features" features "--dry-run"]))))
(t/is (thrown-with-msg? js/Error #"purge category flags require purge"
(runner/parse-args ["ab" "--features" "a" "--only-comparisons"])))
(t/is (thrown? js/Error (runner/parse-args ["ab" "extra" "--features" "a" "--dry-run"]))))
(t/deftest ab-planning-errors-keep-their-cause
(let [error (try
(runner/parse-args ["ab" "--dry-run"])
nil
(catch :default cause cause))]
(t/is (= :benches.render-wasm.node.runner/invalid-args
(:type (ex-data error))))
(t/is (some? (ex-cause error)))
(t/is (str/includes? (ex-message (ex-cause error)) "at least one"))))
(t/deftest ab-limits-and-confirmation-apply-at-the-parser-boundary
(t/is (thrown-with-msg? js/Error #"requires --confirm" (runner/parse-args ["ab" "--features" "a"])))
(t/is (:confirm (:options (runner/parse-args ["ab" "--features" "a" "--confirm"]))))
(t/is (:dry-run (:options (runner/parse-args ["ab" "--features" "a" "--dry-run"]))))
(t/is (thrown-with-msg? js/Error #"exceed max-features"
(runner/parse-args ["ab" "--features" "a,b,c,d" "--dry-run"])))
(t/is (thrown-with-msg? js/Error #"requires --confirm"
(runner/parse-args ["ab" "--features" "a,b,c,d" "--max-features" "4" "--filter" "rects"])))
(doseq [flag ["--confirm" "--dry-run"]]
(t/is (= 4 (:max-features (:options (runner/parse-args ["ab" "--features" "a,b,c,d" "--max-features" "4" flag]))))))
(doseq [limit ["" " " "0" "7" "-1" "1.5" "NaN" "Infinity" "3x"]]
(t/is (thrown? js/Error (runner/parse-args ["ab" "--features" "a" "--max-features" limit "--dry-run"])))))
(t/deftest planning-flags-are-only-for-ab-and-help-explains-the-contract
(doseq [command [["build"] ["run"] ["purge"] ["compare" "before" "after"]]
flags [["--dry-run"] ["--confirm"] ["--max-features" "2"]]]
(t/is (thrown-with-msg? js/Error #"ab planning options require ab"
(runner/parse-args (into command flags)))))
(let [{:keys [summary]} (runner/parse-args ["ab" "--help"])
help (runner/usage summary)]
(doseq [part ["runner ab --features" "--only-comparisons" "--max-features"
"--confirm" "--dry-run" "Cargo defaults disabled"
"feature-comparisons/<ab-id>/feature-comparison.transit"]]
(t/is (str/includes? help part)))))
(t/deftest comparison-purge-is-exclusive-and-only-for-purge
(t/is (true? (get-in (runner/parse-args ["purge" "--only-comparisons"]) [:options :only-comparisons])))
(doseq [other ["--only-builds" "--only-runs"]]
(t/is (thrown-with-msg? js/Error #"mutually exclusive"
(runner/parse-args ["purge" "--only-comparisons" other]))))
(doseq [command [["build"] ["run"] ["compare" "before" "after"]]]
(t/is (thrown-with-msg? js/Error #"purge category flags require purge"
(runner/parse-args (conj command "--only-comparisons"))))))