diff --git a/frontend/performance/benches/render_wasm/README.md b/frontend/performance/benches/render_wasm/README.md index 2b1feb0a6a..76e02fd3c8 100644 --- a/frontend/performance/benches/render_wasm/README.md +++ b/frontend/performance/benches/render_wasm/README.md @@ -247,11 +247,14 @@ manifest records the build id, status, functional settings, Git provenance, and build facts. An experiment run can select a retained build with `--build `, but a run without `--build` always prepares a new one. -A manual build can select Cargo features with `--features`, but `--build` cannot -be combined with feature selection. +`build` and a run that prepares its build accept `--features`, +`--[no-]default-features`, and `--build-timeout-ms`. A run with `--build` reuses +the retained build and rejects those preparation options. -We use the same locks as the main build system to ensure that two preparations -do not run concurrently in the same checkout. +Run one build preparation at a time in a checkout and keep renderer watchers +stopped until it finishes. The renderer build lock covers Cargo compilation +and copying the renderer files. The build script releases that lock before +ClojureScript compilation. ### Browser host @@ -631,7 +634,9 @@ finding. ### Comparisons A comparison pairs saved cases by identical case id unless `--cases A,B` -fixes the selection to single baseline = A, candidate = B. +selects one explicit baseline, candidate pair. The CLI checks the selector +syntax before it loads either record, then resolves each selector by saved case +ID or exact name. By default, a comparison requires equal: @@ -710,11 +715,13 @@ compares each edge once, not once per execution order. For features 10 -> 11 : b ``` -`--max-features` accepts 1 through 6 and defaults to 3. The command requires -`--confirm` when the comparison has more than 8 configurations or when the -filter selects every case. `--dry-run` prints the features, configurations, -edges, selected case count, and budgets without creating a comparison home, -compiling, or launching a browser. +`--max-features` accepts 1 through 6 and defaults to 3. When `--dry-run` +is absent, the command requires `--confirm` for more than 8 configurations +or a filter that is absent, empty, or contains only whitespace. `--dry-run` +checks the requested features using Cargo metadata, selects cases from their +declarations, and prints the features, configurations, edges, selected case +count, and time limits. It creates no comparison directory, compiles no build, +and launches no browser. ### Running a feature comparison @@ -764,9 +771,9 @@ features, build and run references, edges, failures, and interpretation warnings. The runner validates the record with a schema and writes it atomically. It writes the initial plan before the first build and updates the record after each configuration, after each edge, and at termination. The -command prints `Comparison ` and `Output ` on stdout before -the first build, then `Status ` at termination. Progress and failures -go to stderr. Each configuration has its own build and run log. +command prints `Comparison `, `Output `, and `Status ` +on stdout at termination. Progress and failures go to stderr. Each configuration +has its own build and run log. ### Edge comparisons @@ -809,6 +816,17 @@ until preparation finishes. The literal `--` after `render-bench` is required to forward the command and its arguments through the shell wrapper. +List the commands, or show one command's options and defaults: + +```bash +pnpm run render-bench -- --help +pnpm run render-bench -- run --help +``` + +Command options select declared cases and control the build source, sampling, +output and hardware policy. Case declarations own workload bodies, scene +parameters, view and operation settings, and case lifecycle. + Build the runner and its command module: ```bash @@ -880,35 +898,46 @@ pnpm run render-bench -- merge \ --output-dir target/performance/render-wasm/runs/ ``` -The merged record gets a new run id and a fresh run home. Both inputs must be -completed and compatible: same cases, timeout policy, analysis, build facts, -runtime environment, and Git provenance. The additional run must use the same -warmup count. `merge` appends only the additional measured attempts; its warmup -attempts remain in the additional record. The command starts no build, server, -or browser. +The merged record gets a new run id and a new run directory. Both input records +stay unchanged. Both inputs must be completed and compatible: same cases, +timeout policy, analysis, build facts, runtime environment, and Git provenance. +The additional run must use the same warmup count. `merge` appends only the +additional measured attempts and uses the base run's warmups. The additional +warmup attempts stay in the additional record. The command starts no build, +server, or browser. Inspect and clean managed outputs: ```bash -pnpm run render-bench -- purge --only-builds -pnpm run render-bench -- purge --only-runs -pnpm run render-bench -- purge --only-comparisons +pnpm run render-bench -- purge builds +pnpm run render-bench -- purge runs +pnpm run render-bench -- purge comparisons +pnpm run render-bench -- purge all ``` -`purge` removes known build, standalone-run, or feature-comparison outputs. It -keeps the managed directories and unrelated files. The three `--only-*` options -are mutually exclusive; no option selects every category. +`purge` requires exactly one target: `builds`, `runs`, `comparisons`, or `all`. +It removes known build, standalone-run, or feature-comparison outputs and keeps +directories and unrelated files. It follows directory symlinks and can remove +matching output files from their targets, including targets outside the managed +directories. `all` removes every category. A missing, unknown or multiple target +is an argument error. The commands write results to stdout: -- `build`: `Build ` and `Output ` at allocation. -- `run`: `Run ` and `Output ` at allocation, then a `Report` - heading and the formatted summary. +- `build`: `Build ` and `Output ` after build preparation + ends. +- `run`: `Run ` and `Output ` after the run ends. When a + report is available, a `Report` heading and the formatted summary follow. - `ab`: `Comparison `, `Output `, and final `Status `. - `compare`: the comparison report. - `merge`: `Run ` and `Output `. -- Help: usage text. `purge` produces no output on success. +- `--help` and `COMMAND --help`: usage text with that command's options and + defaults. `purge` produces no output on success. + +Each CLI command produces an outcome with an exit code, errors, log path, and +optional report text. The runner prints the final output once and returns the +numeric status. Progress and diagnostics go to stderr. The default shows build phases, case headings, and attempt progress. `-q` / `--quiet` shows results, warnings, and @@ -938,9 +967,11 @@ mean interruption by `SIGINT` and `SIGTERM` for `build`, `run` and `ab`. An interrupted command prints no error for the aborted work; it prints a failed write of its own record or log. -`--warmups` defaults to 3, `--repetitions` to 10, `--timeout-ms` to 30000, -`--build-timeout-ms` to 300000, and `--startup-timeout-ms` to 30000. The master -seed defaults to 42 and accepts a 32-bit unsigned integer. A filter matches a +`--warmups` defaults to 3, `--repetitions` to 10, `--timeout-ms` to 30000, and +`--startup-timeout-ms` to 30000. `build`, a run that prepares its build, and an +A/B experiment use a default `--build-timeout-ms` of 300000; a run with +`--build` uses the retained build's recorded provenance. The master seed +defaults to 42 and accepts a 32-bit unsigned integer. A filter matches a substring of the case id. ## Unit tests diff --git a/frontend/performance/benches/render_wasm/node/build.cljs b/frontend/performance/benches/render_wasm/node/build.cljs index 67fae639a7..5ed95102df 100644 --- a/frontend/performance/benches/render_wasm/node/build.cljs +++ b/frontend/performance/benches/render_wasm/node/build.cljs @@ -65,10 +65,10 @@ Options: frontend :root, :selection or :features, :build-timeout-ms (300000), AbortSignal :signal, and optional :on-created callback. The callback - runs once after directory, log and initial manifest creation. CLI callers - print the ID there. Storage errors have no recorded ID. Later errors carry + receives the new build reference once the output directory, log and initial + manifest exist. Storage errors have no recorded ID. Later errors carry :build-id and :phase. An error also carries :manifest-error and is fatal when - the failed manifest cannot be saved. A retry always creates a new attempt." + the failed manifest cannot be saved. A retry always creates a new build." [{:keys [root features selection build-timeout-ms signal on-created] :or {build-timeout-ms options/default-build-timeout-ms}}] (when-not (and (js/Number.isSafeInteger build-timeout-ms) (pos? build-timeout-ms)) diff --git a/frontend/performance/benches/render_wasm/node/commands.cljs b/frontend/performance/benches/render_wasm/node/commands.cljs index e7700d3f2e..09a336b273 100644 --- a/frontend/performance/benches/render_wasm/node/commands.cljs +++ b/frontend/performance/benches/render_wasm/node/commands.cljs @@ -2,106 +2,67 @@ ;; License, v. 2.0. If a copy of the MPL was not distributed with this ;; file, You can obtain one at http://mozilla.org/MPL/2.0/. ;; -;; Copyright (c) KALEIDOS SUBSIDIARY SL +;; Copyright (c) KALEIDOS INC (ns benches.render-wasm.node.commands - "Execution commands loaded after CLI validation." + "Deferred execution entry for build, run and feature comparison commands." (:require - [app.common.schema :as sm] [app.common.transit :as transit] - [benches.render-wasm.errors :as errors :refer [attempt failure?]] + [benches.render-wasm.errors :refer [attempt failure?]] [benches.render-wasm.node.artifacts :as artifacts] [benches.render-wasm.node.build :as build] [benches.render-wasm.node.execution :as execution] [benches.render-wasm.node.feature-comparison :as feature-comparison] [benches.render-wasm.node.feature-plan :as feature-plan] - [benches.render-wasm.node.log :as log] - [benches.render-wasm.node.options :as options] [benches.render-wasm.node.outcome :as outcome] [benches.render-wasm.node.run :as run])) -(defn- run-command - [root options] - (execution/with-interruption - #(run/run root (select-keys options (sm/keys run/schema:run-options)) %))) +(defn- ^:async build-command + [{:keys [get-root signal]} options] + (let [created (atom nil) + result (attempt :catch-fatal + (await (build/prepare + {:root (get-root) + :selection {:default-features? (:default-features options) + :features (:features options)} + :build-timeout-ms (:build-timeout-ms options) + :signal signal :on-created #(reset! created %)}))) + result (if (failure? result) + (outcome/failed result {:build @created}) + (outcome/succeeded {:build result :log-path (:log-path result)}))] + (cond-> result + @created (assoc :report (str "Build " (:id @created) "\nOutput " (:dir @created)))))) -(defn- build-command - [root options] - (execution/with-interruption - (^:async fn [signal] - (await (build/prepare {:root root - :features {:default-features? (:default-features options) - :features (or (:features options) [])} - :build-timeout-ms (:build-timeout-ms options) - :signal signal - :on-created (fn [build] - (log/result (str "Build " (:id build))) - (log/result (str "Output " (:dir build))))})) - {:exit-code 0}))) +(defn- ^:async run-command + [{:keys [get-root signal services]} options] + (let [{:keys [home report] :as result} (await (run/run (get-root) options signal services))] + (assoc result :report (str "Run " (:run-id home) "\nOutput " (:dir home) + (when report (str "\nReport\n" report)))))) -(defn- ab-command - [root options] - (execution/with-interruption - (^:async fn [signal] - (let [prepared (await (feature-plan/prepare root options signal))] - (if (:dry-run options) - (do - (log/result (feature-plan/format-plan prepared)) - {:exit-code 0}) - (await (feature-comparison/execute root prepared signal {}))))))) +(defn- ^:async ab-command + [{:keys [get-root signal services]} options] + (let [root (get-root) + prepared (await (feature-plan/prepare root options signal))] + (if (:dry-run options) + (outcome/succeeded {:report (feature-plan/format-plan prepared)}) + (let [{:keys [home exit-code] :as result} + (await (feature-comparison/execute root prepared signal {:services services}))] + (cond-> result + home (assoc :report (str "Comparison " (:comparison-id home) + "\nOutput " (:dir home) + "\nStatus " (cond + (.-aborted signal) "interrupted" + (zero? exit-code) "completed" + :else "failed")))))))) -(defn- ^:async run-request - "Runs the validated command and returns its outcome." - [request] - (let [{:keys [command options]} (transit/decode-str request) - _ (when (and (:require-gpu options) - (or (= command "run") - (and (= command "ab") (not (:dry-run options)))) - (not= "1" (.-PENPOT_GPU (.-env js/process)))) - (throw (ex-info "--require-gpu needs PENPOT_GPU=1. Start the container with ./manage.sh run-devenv --gpu and use pnpm run render-bench." - {:type ::gpu-required}))) - root (artifacts/frontend-root)] - (log/configure {:level (options/console-level options) - :color (options/color-mode options)}) - (case command - "purge" - (do - (artifacts/purge root (cond (:only-builds options) :builds - (:only-runs options) :runs - (:only-comparisons options) :comparisons - :else :all)) - {:exit-code 0}) - - "build" - (await (build-command root options)) - - "run" - (await (run-command root options)) - - "ab" - (await (ab-command root options))))) - -(defn- log-failures - "Prints each failure of `result`. Only the first prints a log tail." - [result] - (binding [log/*log-path* (:log-path result)] - (doseq [[index cause] (map-indexed vector (:errors result))] - (let [data (ex-data cause)] - (log/error {:message (errors/message-of cause) - :fields (filterv (comp some? second) - [[:type (:type data)] [:phase (:phase data)] - [:build (:build-id data)] [:reason (:reason data)] - [:manifest-error (:manifest-error data)]]) - :tail-log (:log-path data) - :tail? (zero? index)}))))) +(def ^:private handlers + {"build" build-command "run" run-command "ab" ab-command}) (defn ^:async execute - "Accepts a validated Transit command from the CLI and returns its exit code. - Each compiled module owns its CLJS values; only the string crosses modules. - Execution errors are printed here before returning a numeric exit code." - [request] - (log/configure {}) - (let [result (attempt :catch-fatal (await (run-request request))) - finished (if (failure? result) (outcome/failed result {}) result)] - (log-failures finished) - (:exit-code finished))) + "Accepts a Transit string containing a complete request and returns its + numeric exit status after final publication." + [text] + (let [request (attempt :catch-fatal (transit/decode-str text))] + (if (failure? request) + (execution/fail request) + (await (execution/execute request handlers {:get-root artifacts/frontend-root}))))) diff --git a/frontend/performance/benches/render_wasm/node/compare.cljs b/frontend/performance/benches/render_wasm/node/compare.cljs index 3452e81a2b..be9337e57d 100644 --- a/frontend/performance/benches/render_wasm/node/compare.cljs +++ b/frontend/performance/benches/render_wasm/node/compare.cljs @@ -23,13 +23,13 @@ It is also possible to restrict comparison to specific scene ids. Options: - - `:cases` is one A,B selector, where A is a baseline an B the candidate. + - `:case-pair` contains the baseline and candidate selectors, or nil. - The three `:allow-*-differences` options select build, scene or all compatibility permissions. Returns the validated `comparison` with its selected permissions, retained differences and formatted text." - [baseline-path candidate-path {:keys [cases allow-build-differences + [baseline-path candidate-path {:keys [case-pair allow-build-differences allow-scene-differences allow-all-differences]}] (let [baseline (runs/get-record baseline-path) candidate (runs/get-record candidate-path) @@ -39,8 +39,8 @@ (cond-> {:allow-build-differences (boolean allow-build-differences) :allow-scene-differences (boolean allow-scene-differences) :allow-all-differences (boolean allow-all-differences)} - cases + case-pair (assoc :pairs (compare/selector-pair baseline candidate - cases)))))] + case-pair)))))] {:comparison comparison :text (format/format-comparison comparison)})) diff --git a/frontend/performance/benches/render_wasm/node/feature_comparison.cljs b/frontend/performance/benches/render_wasm/node/feature_comparison.cljs index f980c0193d..b12d6b8940 100644 --- a/frontend/performance/benches/render_wasm/node/feature_comparison.cljs +++ b/frontend/performance/benches/render_wasm/node/feature_comparison.cljs @@ -7,24 +7,23 @@ (ns benches.render-wasm.node.feature-comparison "Serial feature A/B orchestration and persistence of comparisons." (:require - ["fs" :as fs] - ["node:timers/promises" :as timers] - ["path" :as path] - [app.common.schema :as sm] - [benches.render-wasm.errors :as errors :refer [attempt failure?]] - [benches.render-wasm.node.artifacts :as artifacts] - [benches.render-wasm.node.build :as build] - [benches.render-wasm.node.catalog :as catalog] - [benches.render-wasm.node.files :as files] - [benches.render-wasm.node.log :as log] - [benches.render-wasm.node.outcome :as outcome] - [benches.render-wasm.node.process :as process] - [benches.render-wasm.node.run :as run] - [benches.render-wasm.node.runs :as runs] - [benches.render-wasm.node.time :as time] - [benches.render-wasm.report.compare :as compare] - [benches.render-wasm.result :as result] - [clojure.set :as set])) + ["fs" :as fs] + ["node:timers/promises" :as timers] + ["path" :as path] + [benches.render-wasm.errors :as errors :refer [attempt failure?]] + [benches.render-wasm.node.artifacts :as artifacts] + [benches.render-wasm.node.build :as build] + [benches.render-wasm.node.catalog :as catalog] + [benches.render-wasm.node.files :as files] + [benches.render-wasm.node.log :as log] + [benches.render-wasm.node.outcome :as outcome] + [benches.render-wasm.node.process :as process] + [benches.render-wasm.node.run :as run] + [benches.render-wasm.node.runs :as runs] + [benches.render-wasm.node.time :as time] + [benches.render-wasm.report.compare :as compare] + [benches.render-wasm.result :as result] + [clojure.set :as set])) (def ^:private warnings ["Timing values do not determine success." @@ -49,8 +48,9 @@ dir (artifacts/check-output-dir (or output-dir default-dir))] (when (artifacts/inside-builds? root dir) (artifacts/invalid-home "feature comparison cannot use a directory inside builds" {:path dir})) - (artifacts/create-home dir "feature-comparison" - "cannot create feature comparison home"))) + (assoc (artifacts/create-home dir "feature-comparison" + "cannot create feature comparison home") + :comparison-id comparison-id))) (defn write-record "Validates schema:feature-comparison and atomically replaces the saved file." @@ -316,7 +316,7 @@ (attempt (await ((:execute-run (:services state)) (:root state) (:run-home flow) (:build flow) (:cases state) - (select-keys (:options state) (sm/keys run/schema:run-options)) + (run/get-execution-options (:options state)) (:signal state))))) (defn- get-evidence-error @@ -595,20 +595,10 @@ (defn- initialize-comparison [state] (let [initial @(:record-ref state) - home (:home state)] - (log/result (str "Comparison " (:comparison-id initial))) - (log/result (str "Output " (:dir home))) - (let [saved (write-record home initial)] - (reset! (:record-ref state) saved) - (reset! (:saved-ref state) saved)))) - -(defn- publish-status - [result] - (outcome/publish result - #(log/result (str "Status " (case (:exit-code result) - 0 "completed" - (130 143) "interrupted" - "failed"))))) + home (:home state) + saved (write-record home initial)] + (reset! (:record-ref state) saved) + (reset! (:saved-ref state) saved))) (defn- ^:async execute-comparison [state] @@ -617,12 +607,11 @@ (let [result (attempt :catch-fatal (initialize-comparison state) (await (run-comparisons state (:configurations @(:record-ref state)))))] - (publish-status - (if (failure? result) - (outcome/failed result {:comparison @(:saved-ref state) - :home home - :log-path (:log-path home)}) - result)))))) + (if (failure? result) + (outcome/failed result {:comparison @(:saved-ref state) + :home home + :log-path (:log-path home)}) + result))))) (defn ^:async execute "Executes a prepared feature comparison and owns builds, runs and reports. diff --git a/frontend/performance/benches/render_wasm/node/feature_plan.cljs b/frontend/performance/benches/render_wasm/node/feature_plan.cljs index 35e9b45f76..6ed9dc271c 100644 --- a/frontend/performance/benches/render_wasm/node/feature_plan.cljs +++ b/frontend/performance/benches/render_wasm/node/feature_plan.cljs @@ -33,14 +33,7 @@ render-wasm with the inherited environment, checks an optional fresh output path, and collects selected case data once." [root opts signal] - (let [opts (merge {:max-features feature-plan/default-max-features - :confirm false :dry-run false :seed 42 - :allow-build-differences false - :allow-scene-differences false - :allow-all-differences false} - opts - {:default-features false}) - plan (feature-plan/plan (:features opts) opts) + (let [plan (feature-plan/plan (:features opts) opts) output-dir (artifacts/check-output-dir (:output-dir opts)) cwd (path/resolve root ".." "render-wasm") metadata (await (process/run diff --git a/frontend/performance/benches/render_wasm/node/merge.cljs b/frontend/performance/benches/render_wasm/node/merge.cljs index f5cffc015f..e1d9854737 100644 --- a/frontend/performance/benches/render_wasm/node/merge.cljs +++ b/frontend/performance/benches/render_wasm/node/merge.cljs @@ -8,7 +8,6 @@ "Offline merge of two saved schema:run records." (:require ["fs" :as fs] - [benches.render-wasm.node.log :as log] [benches.render-wasm.node.runs :as runs] [benches.render-wasm.result :as result])) @@ -26,6 +25,4 @@ (runs/write-record! home merged) (fs/appendFileSync (:log-path home) (str "merge " (:run-id base) " + " (:run-id additional) "\n")) - (log/result (str "Run " run-id)) - (log/result (str "Output " (:dir home))) {:run merged :home home})) diff --git a/frontend/performance/benches/render_wasm/node/outcome.cljs b/frontend/performance/benches/render_wasm/node/outcome.cljs index 3af7b37a8b..c0af9d1341 100644 --- a/frontend/performance/benches/render_wasm/node/outcome.cljs +++ b/frontend/performance/benches/render_wasm/node/outcome.cljs @@ -19,6 +19,11 @@ [signal-name] (case signal-name "SIGTERM" 143 130)) +(defn succeeded + "Returns a successful outcome with command result data." + [data] + (merge {:exit-code 0 :errors [] :log-path nil} data)) + (defn failed "Returns an outcome with exit code 1 for `cause`. The log path comes from the ex-data of `cause`. `data` adds more keys." diff --git a/frontend/performance/benches/render_wasm/node/run.cljs b/frontend/performance/benches/render_wasm/node/run.cljs index ec37205b26..b9985059d2 100644 --- a/frontend/performance/benches/render_wasm/node/run.cljs +++ b/frontend/performance/benches/render_wasm/node/run.cljs @@ -23,7 +23,6 @@ ["module" :as node-module] ["os" :as os] ["path" :as path] - [app.common.schema :as sm] [app.common.transit :as transit] [benches.render-wasm.declarations :as declarations] [benches.render-wasm.errors :as errors :refer [attempt failure?]] @@ -43,30 +42,11 @@ [benches.render-wasm.result :as result] [clojure.string :as str])) -(def schema:run-options - "Resolved options for run preparation and execution." - (let [timeout-ms [:int {:min 1 :max 2147483647}]] - [:map {:closed true} - [:seed [:int {:min 0 :max 4294967295}]] - [:warmups [:int {:min 0 :max js/Number.MAX_SAFE_INTEGER}]] - [:repetitions [:int {:min 1 :max js/Number.MAX_SAFE_INTEGER}]] - [:timeout-ms timeout-ms] - [:build-timeout-ms timeout-ms] - [:startup-timeout-ms timeout-ms] - [:filter {:optional true} [:maybe :string]] - [:build-id {:optional true} [:string {:min 1}]] - [:output-dir {:optional true} [:maybe [:string {:min 1}]]] - [:default-features {:optional true} :boolean] - [:features {:optional true} [:vector [:string {:min 1}]]] - [:screenshots {:optional true} :boolean] - [:remove {:optional true} :boolean] - [:require-gpu {:optional true} :boolean] - [:browser {:optional true} [:map-of :keyword fn?]]])) - -(def ^:private check-options - (sm/check-fn schema:run-options - :hint "invalid run options" - :type ::invalid-options)) +(defn get-execution-options + "Returns resolved settings for executing a prepared run." + [options] + (select-keys options [:seed :filter :warmups :repetitions :timeout-ms + :startup-timeout-ms :screenshots :require-gpu :remove])) (defn- cause-evidence "Plain failure evidence for a rejected browser, server or persistence action." @@ -650,20 +630,20 @@ (assoc :environment (get-in selected-build [:manifest :environment]))) :environment (environment root selected-build) :scored? true - :diagnostics {:seed (:seed opts) - :filter (:filter opts) - :warmups (:warmups opts) - :repetitions (:repetitions opts) - :timeout-ms (:timeout-ms opts) - :build-timeout-ms (:build-timeout-ms opts) - :startup-timeout-ms (:startup-timeout-ms opts) - :screenshots? (boolean (:screenshots opts)) - :require-gpu? (boolean (:require-gpu opts)) - :penpot-gpu (or (.-PENPOT_GPU (.-env js/process)) nil) - :runner-git {:sha nil :dirty nil} - :runtime-helper-identity :unverified - :screenshots {} - :failures []}}) + :diagnostics (cond-> {:seed (:seed opts) + :filter (:filter opts) + :warmups (:warmups opts) + :repetitions (:repetitions opts) + :timeout-ms (:timeout-ms opts) + :startup-timeout-ms (:startup-timeout-ms opts) + :screenshots? (boolean (:screenshots opts)) + :require-gpu? (boolean (:require-gpu opts)) + :penpot-gpu (or (.-PENPOT_GPU (.-env js/process)) nil) + :runner-git {:sha nil :dirty nil} + :runtime-helper-identity :unverified + :screenshots {} + :failures []} + (:build-timeout-ms opts) (assoc :build-timeout-ms (:build-timeout-ms opts)))}) (defn- create-run-record [root run-id selected-build cases opts] @@ -772,7 +752,7 @@ - `home`: the reserved output home. - `selected-build`: the prepared build. - `cases`: the selected cases. - - `options`: resolved options satisfying `schema:run-options`. + - `options`: resolved execution settings, including optional local :browser services. - `signal`: an optional abort signal. A browser startup failure leaves a valid partial record. @@ -782,7 +762,6 @@ write, including a failure followed by a successful write. An interruption is not a failure." [root home selected-build cases options signal] - (check-options options) (binding [log/*log-path* (:log-path home)] (let [state (create-run-state root home selected-build cases options signal)] (persist state) @@ -792,14 +771,13 @@ (defn- ^:async get-selected-build [root opts signal] - (if-let [id (:build-id opts)] - (artifacts/load-build root id) - (await (build/prepare - {:root root - :features {:default-features? (:default-features opts true) - :features (or (:features opts) [])} - :build-timeout-ms (:build-timeout-ms opts) - :signal signal})))) + (let [{:keys [kind id default-features features build-timeout-ms]} (:build opts)] + (case kind + :existing (artifacts/load-build root id) + :prepare (await (build/prepare + {:root root + :selection {:default-features? default-features :features features} + :build-timeout-ms build-timeout-ms :signal signal}))))) (defn- get-cases [selected-build opts] @@ -811,35 +789,29 @@ :log-path (:log-path selected-build)) cause))))) -(defn- publish-report - [result] - (outcome/publish result - #(when-let [report (:report result)] - (log/result (str "Report\n" report))))) - (defn ^:async run "Prepares or loads a build, collects matching cases and executes one run. - `root`: the frontend checkout root. - - `options`: resolved options satisfying `schema:run-options`. + - `options`: resolved run options with a tagged :build source. - `signal`: an optional abort signal. + - `services`: local service overrides, with browser functions under :browser. - Validates options before allocating output. Returns the execution result. - A build or catalog failure throws with the run log path and build facts." - [root options signal] - (let [opts (check-options options) - home (runs/create-home {:root root :run-id (str (random-uuid)) - :output-dir (:output-dir opts) :remove (:remove opts)})] + Returns an outcome with the reserved home, available evidence and failures." + [root options signal services] + (let [home (runs/create-home {:root root :run-id (str (random-uuid)) + :output-dir (:output-dir options) :remove (:remove options)}) + opts (cond-> (assoc (get-execution-options options) :browser (:browser services)) + (= :prepare (get-in options [:build :kind])) + (assoc :build-timeout-ms (get-in options [:build :build-timeout-ms])))] (binding [log/*log-path* (:log-path home)] (try - (log/result (str "Run " (:run-id home))) - (log/result (str "Output " (:dir home))) - (let [selected-build (await (get-selected-build root opts signal)) + (let [selected-build (await (get-selected-build root options signal)) cases (get-cases selected-build opts)] (log/info (str " build " (:id selected-build))) (log/info (str " selected " (count cases) " cases")) - (publish-report (await (execute root home selected-build cases opts signal)))) + (await (execute root home selected-build cases opts signal))) (catch :default cause - (throw (-> cause - errors/as-failure - (errors/with-data {:log-path (:log-path home)})))))))) + (outcome/failed (-> cause errors/as-failure + (errors/with-data {:log-path (:log-path home)})) + {:home home :run nil :report nil})))))) diff --git a/frontend/performance/benches/render_wasm/node/runner.cljs b/frontend/performance/benches/render_wasm/node/runner.cljs index 81e7f3ac08..218fd643ec 100644 --- a/frontend/performance/benches/render_wasm/node/runner.cljs +++ b/frontend/performance/benches/render_wasm/node/runner.cljs @@ -2,336 +2,59 @@ ;; License, v. 2.0. If a copy of the MPL was not distributed with this ;; file, You can obtain one at http://mozilla.org/MPL/2.0/. ;; -;; Copyright (c) KALEIDOS SUBSIDIARY SL +;; Copyright (c) KALEIDOS INC (ns benches.render-wasm.node.runner - "CLI parsing, offline comparison and deferred execution commands." + "Command-line entry for renderer benchmarks and saved results." (:require ["path" :as path] [app.common.transit :as transit] - [benches.render-wasm.errors :as errors :refer [attempt failure?]] - [benches.render-wasm.feature-plan :as feature-plan] + [benches.render-wasm.errors :refer [attempt failure?]] [benches.render-wasm.node.artifacts :as artifacts] + [benches.render-wasm.node.cli :as cli] [benches.render-wasm.node.compare :as compare] - [benches.render-wasm.node.log :as log] - [benches.render-wasm.node.merge :as run-merge] - [benches.render-wasm.node.options :as options] - [clojure.string :as str] - [clojure.tools.cli :refer [parse-opts]])) + [benches.render-wasm.node.execution :as execution] + [benches.render-wasm.node.merge :as merge] + [benches.render-wasm.node.outcome :as outcome])) -(def ^:private max-timeout-ms 2147483647) +(defn- help-command + [_ {:keys [topic]}] + (outcome/succeeded {:report (cli/get-help topic)})) -(def ^:private positive-timeout - [(fn [n] (and (js/Number.isSafeInteger n) - (pos? n) - (<= n max-timeout-ms))) - "Must be a positive integer up to 2147483647"]) +(defn- compare-command + [_ {:keys [baseline-path candidate-path] :as options}] + (let [result (compare/execute baseline-path candidate-path options)] + (outcome/succeeded {:comparison (:comparison result) :report (:text result)}))) -(def ^:private nonnegative-integer - [(fn [n] (and (js/Number.isSafeInteger n) (not (neg? n)))) - "Must be a nonnegative integer"]) +(defn- merge-command + [{:keys [get-root]} {:keys [base-path additional-path] :as options}] + (let [{:keys [run home]} (merge/execute (get-root) base-path additional-path options)] + (outcome/succeeded {:run run :home home :log-path (:log-path home) + :report (str "Run " (:run-id run) "\nOutput " (:dir home))}))) -(def ^:private positive-integer - [(fn [n] (and (js/Number.isSafeInteger n) (pos? n))) - "Must be a positive integer"]) +(defn- purge-command + [{:keys [get-root]} {:keys [target]}] + (artifacts/purge (get-root) target) + (outcome/succeeded {})) -(def ^:private uint32 - [(fn [n] (and (js/Number.isSafeInteger n) (<= 0 n 4294967295))) - "Must be a uint32 integer"]) - -(def ^:private nonempty - [(fn [s] (not (str/blank? s))) "Must be a nonempty value"]) - -(defn- parse-number - "Parses a numeric CLI value. Blank input stays invalid instead of becoming zero." - [text] - (if (str/blank? text) js/NaN (js/Number text))) - -(def cli-options - [["-h" "--help" "Print this help"] - ["-q" "--quiet" "Only warnings, errors and results"] - ["-v" "--verbose" "Show debug details and raw build output"] - [nil "--[no-]color" "Colorize terminal output; default auto" :default nil] - [nil "--[no-]default-features" "Enable Cargo defaults" :default true] - [nil "--features FEATURES" "Comma-separated functional Cargo features" - :parse-fn options/parse-feature-list] - [nil "--max-features N" "ab feature limit (1..6)" - :default feature-plan/default-max-features - :parse-fn parse-number - :validate [#(and (js/Number.isSafeInteger %) (<= 1 % 6)) "must be an integer from 1 to 6"]] - [nil "--confirm" "Confirm ab with >8 configurations or an empty filter" :default false] - [nil "--dry-run" "Validate and print the ab plan without executing" :default false] - [nil "--build ID" "Reuse a ready build" :id :build-id - :validate nonempty] - [nil "--filter SUBSTR" "Select cases whose id contains SUBSTR"] - [nil "--seed N" "Master scene seed" :default 42 - :parse-fn parse-number :validate uint32] - [nil "--warmups N" "Warmup attempts per case" :default 3 - :parse-fn parse-number :validate nonnegative-integer] - [nil "--repetitions N" "Measured attempts per case" :default 10 - :parse-fn parse-number :validate positive-integer] - [nil "--timeout-ms MS" "Attempt deadline" :default 30000 - :parse-fn parse-number :validate positive-timeout] - [nil "--build-timeout-ms MS" "Complete preparation deadline" - :default options/default-build-timeout-ms - :parse-fn parse-number :validate positive-timeout] - [nil "--startup-timeout-ms MS" "Host/browser readiness deadline" - :default options/default-startup-timeout-ms - :parse-fn parse-number :validate positive-timeout] - [nil "--output-dir DIR" "Explicit run home; must not exist" - :validate nonempty] - [nil "--remove" "Remove the used build after shutdown"] - [nil "--screenshots" "Capture one screenshot per case"] - [nil "--require-gpu" "Fail when GPU rendering is unavailable"] - [nil "--only-builds" "Purge build attempts"] - [nil "--only-runs" "Purge managed runs"] - [nil "--only-comparisons" "Purge managed feature comparisons"] - [nil "--cases A,B" "Select one baseline,candidate pair"] - [nil "--allow-build-differences" "Allow recorded build differences"] - [nil "--allow-scene-differences" "Allow scene inputs and fingerprint content differences"] - [nil "--allow-all-differences" "Allow all comparison compatibility differences"]]) - -(defn usage - [summary] - (str "Usage: runner build [options]\n" - " runner purge [--only-builds | --only-runs | --only-comparisons]\n" - " runner run [options]\n" - " runner ab --features A,B [--filter SUBSTR] [--max-features N]\n" - " [--confirm] [--dry-run] [run and comparison options]\n" - " runner compare BASELINE CANDIDATE [--cases A,B]\n" - " [--allow-build-differences] [--allow-scene-differences]\n" - " [--allow-all-differences]\n" - " runner merge BASE ADDITIONAL [--output-dir DIR]\n\n" - "build prints Build and Output on stdout. Exit 0 means\n" - "ready; errors retain the build.transit and build.log for that ID.\n" - "run prints Run , Output and Report on stdout.\n" - "Progress and diagnostics go to stderr. -q/--quiet hides progress;\n" - "-v/--verbose includes raw build output. --[no-]color overrides terminal\n" - "color detection. File logs receive all levels without logger colors.\n" - "Builds and managed runs live under target/performance/render-wasm.\n" - "run and ab accept --require-gpu. Start the container with\n" - "./manage.sh run-devenv --gpu and use pnpm run render-bench -- run ...\n" - "to require hardware WebGL2 in preparation metadata. ab --dry-run skips it.\n" - "compare reads saved schema:run records and starts no build, host or browser.\n" - "Build and scene permissions combine; operation and runtime checks stay strict.\n" - "All-differences permits every compatibility difference. Reports retain them.\n" - "merge reads two saved schema:run records and writes a merged record in a fresh\n" - "run home. It appends the additional run's measured attempts, keeps the base\n" - "warmups, and requires compatible completed runs. It starts no build or browser.\n\n" - "ab uses every on/off configuration with Cargo defaults disabled; declared\n" - "defaults must be empty. --max-features defaults to 3 (range 1..6).\n" - "Use --confirm above eight configurations or with an empty filter.\n" - "--dry-run checks metadata, cases and output freshness without confirmation,\n" - "output creation, compilation or browser startup. ab accepts run budgets,\n" - "timeouts, seed, filter, output and screenshots, plus comparison permissions.\n" - "It rejects --build, --remove, --cases and --[no-]default-features.\n" - "ab prints Comparison , Output and Status on stdout;\n" - "feature-comparison.log records comparison progress. Diagnostics and\n" - "failures go to stderr. --dry-run prints only the plan without creating\n" - "the comparison home. Managed output lives under\n" - "feature-comparisons//feature-comparison.transit.\n\n" - "Preparation and runs require an idle checkout with dependencies installed.\n" - "Keep renderer watchers stopped until preparation ends. purge assumes no\n" - "experiment is using the managed paths.\n\n" - summary)) - -(def ^:private value-flags - "Long options that consume the next argument when no value follows an equals sign." - (into #{} - (keep (fn [[_ long-option]] - (when (and long-option (re-find #"\s" long-option)) - (first (str/split long-option #"\s+"))))) - cli-options)) - -(defn- supplied-flags - [args] - (loop [tokens (seq args) - flags #{} - value? false - operand? false] - (if-let [token (first tokens)] - (cond - value? - (recur (next tokens) flags false operand?) - - (and (not operand?) (= "--" token)) - (recur (next tokens) flags false true) - - operand? - (recur (next tokens) flags false true) - - (str/starts-with? token "--") - (let [[flag inline-value] (str/split token #"=" 2)] - (recur (next tokens) - (conj flags flag) - (and (contains? value-flags flag) (nil? inline-value)) - false)) - - :else - (recur (next tokens) flags false false)) - flags))) - -(def ^:private build-flags - #{"--default-features" "--no-default-features" "--features" "--build-timeout-ms"}) - -(def ^:private run-flags - #{"--build" "--filter" "--seed" "--warmups" "--repetitions" "--timeout-ms" - "--startup-timeout-ms" "--output-dir" "--remove" "--screenshots" "--require-gpu"}) - -(def ^:private purge-flags - #{"--only-builds" "--only-runs" "--only-comparisons"}) - -(def ^:private ab-flags - #{"--max-features" "--confirm" "--dry-run"}) - -(def ^:private compare-flags - #{"--cases" "--allow-build-differences" "--allow-scene-differences" - "--allow-all-differences"}) - -(defn- invalid-args - ([message] - (invalid-args message nil)) - ([message cause] - (throw (ex-info message {:type ::invalid-args} cause)))) - -(defn- some-flags? - [flags group] - (boolean (some flags group))) - -(defn- check-command-flags! - [command flags] - (when (and (not= command "ab") (some-flags? flags ab-flags)) - (invalid-args "ab planning options require ab")) - (let [build? (some-flags? flags build-flags) - run? (some-flags? flags run-flags) - purge? (some-flags? flags purge-flags) - compare? (some-flags? flags compare-flags)] - (case command - "build" - (do - (when purge? (invalid-args "purge category flags require purge")) - (when run? (invalid-args "run options require run")) - (when compare? (invalid-args "compare options require compare"))) - - "purge" - (do - (when build? (invalid-args "build options require build")) - (when run? (invalid-args "run options require run")) - (when compare? (invalid-args "compare options require compare"))) - - "run" - (do - (when purge? (invalid-args "purge category flags require purge")) - (when compare? (invalid-args "compare options require compare"))) - - "ab" - (do - (when purge? (invalid-args "purge category flags require purge")) - (when (some-flags? flags ["--build" "--remove" "--cases" - "--default-features" "--no-default-features"]) - (invalid-args "ab rejects --build, --remove, --cases and --[no-]default-features"))) - - "compare" - (do - (when purge? (invalid-args "purge category flags require purge")) - (when build? (invalid-args "build options require build")) - (when run? (invalid-args "run options require run"))) - - "merge" - (do - (when purge? (invalid-args "purge category flags require purge")) - (when build? (invalid-args "build options require build")) - (when compare? (invalid-args "compare options require compare")) - (when (some-flags? flags (disj run-flags "--output-dir")) - (invalid-args "run options require run")))))) - -(defn- check-arguments! - [command arguments] - (let [extra (rest arguments)] - (case command - "compare" - (when-not (= 2 (count extra)) - (invalid-args "compare requires BASELINE and CANDIDATE paths")) - - "merge" - (when-not (= 2 (count extra)) - (invalid-args "merge requires BASE and ADDITIONAL paths")) - - (when (seq extra) - (invalid-args (str command " takes no positional arguments")))))) - -(defn parse-args - "Parses and validates one runner command without loading a catalog, build, - server, browser or renderer. Options are resolved before any run home." - [args] - (let [args (cond-> args (= "--" (first args)) rest) - {:keys [arguments options errors summary]} (parse-opts args cli-options) - command (first arguments) - flags (supplied-flags args)] - (when (seq errors) - (invalid-args (str/join "\n" errors))) - (when (and (:quiet options) (:verbose options)) - (invalid-args "--quiet and --verbose are mutually exclusive")) - (when-not (:help options) - (when-not (and (= 1 (count (take 1 arguments))) - (contains? #{"build" "purge" "run" "ab" "compare" "merge"} command)) - (invalid-args "expected build, purge, run, ab, compare or merge; try --help")) - (check-command-flags! command flags) - (check-arguments! command arguments) - (when (and (= command "purge") - (> (count (filter options [:only-builds :only-runs :only-comparisons])) 1)) - (invalid-args "--only-builds, --only-runs and --only-comparisons are mutually exclusive")) - (when (and (= command "run") - (:build-id options) - (some-flags? flags ["--default-features" "--no-default-features" "--features"])) - (invalid-args "--build cannot be combined with feature-selection flags")) - (when (= command "ab") - (try - (feature-plan/plan (:features options []) options) - (catch :default cause - (invalid-args (errors/message-of cause) cause))))) - {:command command - :arguments arguments - :options (cond-> options (= command "ab") (assoc :default-features false)) - :summary summary})) +(def ^:private handlers + {"help" help-command "compare" compare-command "merge" merge-command "purge" purge-command}) (defn- ^:async run-cli - [args] - (let [{:keys [command arguments options summary] :as request} (parse-args args)] - (log/configure {:level (options/console-level options) - :color (options/color-mode options)}) - (cond - (:help options) - (log/result (usage summary)) - - (= command "compare") - (log/result (:text (compare/execute (second arguments) (nth arguments 2) - (select-keys options [:cases :allow-build-differences - :allow-scene-differences - :allow-all-differences])))) - - (= command "merge") - (run-merge/execute (artifacts/frontend-root) - (second arguments) - (nth arguments 2) - (select-keys options [:output-dir])) - - :else - (let [^js commands (js/require (path/join js/__dirname "commands.cjs"))] - (set! (.-exitCode js/process) - (await (.execute commands - (transit/encode-str (select-keys request [:command :options]))))))))) + [argv] + (let [request (attempt :catch-fatal (cli/parse-resolve argv))] + (if (failure? request) + (execution/fail request) + (if (contains? handlers (:command request)) + (await (execution/execute request handlers {:get-root artifacts/frontend-root})) + (let [status (attempt :catch-fatal + (let [commands (js/require (path/join js/__dirname "commands.cjs"))] + (await (.execute commands (transit/encode-str request)))))] + (if (failure? status) + (execution/fail status (:console request) {}) + status)))))) (defn ^:async -main - [& args] - (log/configure {}) - (let [failed (attempt :catch-fatal (await (run-cli args)))] - (when (failure? failed) - (let [data (ex-data failed)] - (log/error {:message (str (errors/message-of failed) "\nTry --help for usage.") - :fields (into (filterv (comp some? second) - [[:type (:type data)] [:baseline (:baseline data)] - [:candidate (:candidate data)]]) - (map #(vector :difference (:field %)) (:mismatches data))) - :tail? false})) - (set! (.-exitCode js/process) 1)))) + "Runs argv and sets the process exit status after publication and cleanup." + [& argv] + (set! (.-exitCode js/process) (await (run-cli argv)))) diff --git a/frontend/performance/benches/render_wasm/report/compare.cljs b/frontend/performance/benches/render_wasm/report/compare.cljs index 996a69ac41..85070ca0cf 100644 --- a/frontend/performance/benches/render_wasm/report/compare.cljs +++ b/frontend/performance/benches/render_wasm/report/compare.cljs @@ -31,13 +31,9 @@ {:type ::unknown-case :selector selector :available (mapv result/case-name ids)}))))) (defn selector-pair - "Resolves a single A,B comparison selector into an extensible pair list. - A belongs to baseline; B belongs to candidate. Force is a separate option." - [baseline candidate selector] - (let [names (str/split selector #"," -1)] - (when-not (and (= 2 (count names)) (every? seq names)) - (throw (ex-info "--cases requires one exact A,B pair" {:type ::invalid-selector}))) - [[(resolve-case baseline (first names)) (resolve-case candidate (second names))]])) + "Resolves baseline and candidate selectors into an extensible pair list." + [baseline candidate [baseline-selector candidate-selector]] + [[(resolve-case baseline baseline-selector) (resolve-case candidate candidate-selector)]]) (defn- difference "Returns one compatibility difference when recorded values disagree." diff --git a/frontend/test/performance_tests/render_wasm/cli_integration.cjs b/frontend/test/performance_tests/render_wasm/cli_integration.cjs index 1f0f844efc..b8f858e6b3 100644 --- a/frontend/test/performance_tests/render_wasm/cli_integration.cjs +++ b/frontend/test/performance_tests/render_wasm/cli_integration.cjs @@ -154,6 +154,125 @@ function readManifest(file) { return transit.reader("json").read(fs.readFileSync(file, "utf8")); } +function requestText(command, options, color = "never") { + const k = transit.keyword; + return transit.writer("json").write(transit.map([ + k("command"), command, + k("console"), transit.map([k("level"), k("info"), k("color"), k(color)]), + k("options"), transit.map(Object.entries(options).flatMap(([key, value]) => [k(key), value])), + ])); +} + +test("deferred transport carries a complete request and numeric status", testOptions, async (t) => { + const fixture = createFrontend(t); + const capture = path.join(fixture.managed, "request.transit"); + fs.writeFileSync(path.join(fixture.managed, "commands.cjs"), ` + module.exports.execute = (text) => { + if (typeof text !== "string") throw new Error("expected Transit text"); + require("node:fs").writeFileSync(${JSON.stringify(capture)}, text); + return Promise.resolve(37); + }; + `); + const result = await runCli(fixture, ["run", "--build", "ready", "--warmups", "0", "--quiet", "--no-color"]); + assert.equal(result.exitCode, 37, result.stderr); + const k = transit.keyword; + const request = readManifest(capture); + assert.equal(request.get(k("command")), "run"); + assert.equal(request.get(k("console")).get(k("level")).toString(), ":warn"); + assert.equal(request.get(k("console")).get(k("color")).toString(), ":never"); + const options = request.get(k("options")); + assert.equal(options.get(k("build")).get(k("kind")).toString(), ":existing"); + assert.equal(options.get(k("build")).get(k("id")), "ready"); + assert.equal(options.get(k("warmups")), 0); + assert.equal(options.get(k("repetitions")), 10); + assert.equal(options.get(k("startup-timeout-ms")), 30000); +}); + +test("decoded requests reject incomplete, unsupported and policy-invalid commands before acquisition", testOptions, async (t) => { + const fixture = createFrontend(t); + fs.unlinkSync(path.resolve(fixture.managed, "../../../deps.edn")); + const entry = path.join(fixture.managed, "receive.cjs"); + fs.writeFileSync(entry, ` + require("./commands.cjs").execute(process.env.COMMAND_REQUEST) + .then(code => { process.exitCode = code; }); + `); + const before = tree(fixture.managed); + const measurement = { seed: 42, filter: null, warmups: 3, repetitions: 10, + "timeout-ms": 30000, "startup-timeout-ms": 30000, "output-dir": null, + screenshots: false, "require-gpu": false }; + const ab = { ...measurement, features: ["a"], "max-features": 3, confirm: false, + "dry-run": false, "build-timeout-ms": 300000, "allow-build-differences": false, + "allow-scene-differences": false, "allow-all-differences": false }; + const k = transit.keyword; + const build = { "default-features": true, features: [], "build-timeout-ms": 300000 }; + for (const [text, diagnostic] of [ + ["invalid Transit", /JSON/], + [requestText("build", {}), /default-features|build-timeout-ms/], + [requestText("purge", { target: k("all") }, "always"), /unsupported-command/], + [requestText("build", { ...build, features: ["profile"] }), /diagnostic-features/], + [requestText("run", { ...measurement, remove: false, + build: transit.map([k("kind"), k("prepare"), ...Object.entries({ ...build, features: ["stats"] }).flatMap(([key, value]) => [k(key), value])]) }), /diagnostic-features/], + [requestText("ab", ab), /confirmation-required/], + [requestText("ab", { ...ab, features: ["a", "b"], "max-features": 1, "dry-run": true }), /too-many-features/], + ]) { + fixture.env.COMMAND_REQUEST = text; + const result = await runCli({ ...fixture, runner: entry }, []); + assert.equal(result.exitCode, 1, result.stderr); + assert.equal(result.stdout, ""); + assert.match(result.stderr, diagnostic); + assert.equal((result.stderr.match(/✖/g) ?? []).length, 1); + assert.deepEqual(tree(fixture.managed), before); + } +}); + +test("final publication failures select a failing status in local and deferred commands", testOptions, async (t) => { + for (const command of [["--help"], ["build"]]) { + const fixture = createFrontend(t); + if (command[0] === "build") installPreparedCompiler(fixture, false); + const guard = path.join(fixture.managed, "writer.cjs"); + fs.writeFileSync(guard, `process.stdout.write = () => { throw new Error("publication unavailable"); };`); + fixture.nodeArgs = ["--require", guard]; + const result = await runCli(fixture, command); + assert.equal(result.exitCode, 1, result.stderr); + assert.match(result.stderr, /publication unavailable/); + assert.equal((result.stderr.match(/✖/g) ?? []).length, 1); + if (command[0] === "build") { + const [id] = fs.readdirSync(fixture.builds); + assert.equal(readManifest(path.join(fixture.builds, id, "build.transit")).get(transit.keyword("status")).toString(), ":ready"); + } + } +}); + +test("merge and purge execute with unavailable process, HTTP and browser dependencies", testOptions, async (t) => { + const fixture = createFrontend(t); + const guard = path.join(fixture.managed, "offline.cjs"); + fs.writeFileSync(guard, ` + const Module = require("node:module"); + const load = Module._load; + Module._load = function (id, ...args) { + if (["playwright", "http", "node:http", "child_process", "node:child_process"].includes(id)) { + throw new Error("Unavailable dependency: " + id); + } + return load.call(this, id, ...args); + }; + `); + fixture.nodeArgs = ["--require", guard]; + const base = path.join(fixture.managed, "base.transit"); + const additional = path.join(fixture.managed, "additional.transit"); + for (const [file, runId, value] of [[base, "base", 10], [additional, "additional", 12]]) { + fs.writeFileSync(file, runFixtureText({ runId, caseId: "case/load", value, gitSha: "same" })); + } + const merged = await runCli(fixture, ["merge", base, additional]); + assert.equal(merged.exitCode, 0, merged.stderr); + assert.match(merged.stdout, /^Run .+\nOutput .+\n$/); + seedBuilds(fixture); + const purged = await runCli(fixture, ["purge", "builds"]); + assert.equal(purged.exitCode, 0, purged.stderr); + for (const id of fs.readdirSync(fixture.builds)) { + assert.deepEqual(fs.readdirSync(path.join(fixture.builds, id)), ["keep.txt"]); + } +}); + function manifestText(id, status, type) { const k = transit.keyword; return transit @@ -727,7 +846,7 @@ test( fs.readFileSync(logPath, "utf8"), /compiler stderr before failure/, ); - await runCli(fixture, ["purge", "--only-comparisons"]); + await runCli(fixture, ["purge", "comparisons"]); assert.equal(fs.existsSync(logPath), false); }, ); @@ -782,7 +901,13 @@ test( command === "ab" ? ["ab", "--features", "a", "--filter", "rects/load"] : ["run"]; - const child = spawn(process.execPath, [fixture.runner, ...args], { + const failPublication = command === "run" && signal === "SIGINT"; + if (failPublication) { + const writer = path.join(fixture.managed, "writer.cjs"); + fs.writeFileSync(writer, `process.stdout.write = () => { throw new Error("publication unavailable"); };`); + fixture.nodeArgs = ["--require", writer]; + } + const child = spawn(process.execPath, [...(fixture.nodeArgs ?? []), fixture.runner, ...args], { cwd: os.tmpdir(), env: { ...process.env, ...fixture.env }, stdio: ["ignore", "pipe", "pipe"], @@ -806,6 +931,7 @@ test( assert.ok(fs.existsSync(marker), stderr); child.kill(signal); assert.deepEqual(await exited, [code, null], stderr); + if (failPublication) assert.match(stderr, /publication unavailable/); } } }, @@ -906,7 +1032,7 @@ test("purge deletes only the selected artifacts", testOptions, async (t) => { const runs = tree(fixture.runs); const comparisons = tree(fixture.comparisons); - assert.deepEqual(await runCli(fixture, ["purge", "--only-builds"]), { + assert.deepEqual(await runCli(fixture, ["purge", "builds"]), { exitCode: 0, stdout: "", stderr: "", @@ -927,7 +1053,7 @@ test("purge deletes only the selected artifacts", testOptions, async (t) => { seedBuilds(fixture); const builds = tree(fixture.builds); - assert.deepEqual(await runCli(fixture, ["purge", "--only-runs"]), { + assert.deepEqual(await runCli(fixture, ["purge", "runs"]), { exitCode: 0, stdout: "", stderr: "", @@ -946,7 +1072,7 @@ test("purge deletes only the selected artifacts", testOptions, async (t) => { const buildsBeforeComparisonPurge = tree(fixture.builds); const runsBeforeComparisonPurge = tree(fixture.runs); - assert.deepEqual(await runCli(fixture, ["purge", "--only-comparisons"]), { + assert.deepEqual(await runCli(fixture, ["purge", "comparisons"]), { exitCode: 0, stdout: "", stderr: "", @@ -982,7 +1108,7 @@ test("purge deletes only the selected artifacts", testOptions, async (t) => { seedRuns(fixture); seedComparisons(fixture); - assert.deepEqual(await runCli(fixture, ["purge"]), { + assert.deepEqual(await runCli(fixture, ["purge", "all"]), { exitCode: 0, stdout: "", stderr: "", @@ -1118,18 +1244,16 @@ test( const original = tree(fixture.managed); for (const args of [ - ["purge", "--only-builds", "--only-runs"], - ["purge", "--only-builds", "--only-comparisons"], - ["purge", "--only-runs", "--only-comparisons"], - ["purge", "--only-ready-builds"], - ["purge", "--only-builds", "ready"], + ["purge"], + ["purge", "unknown"], + ["purge", "builds", "runs"], ]) { const result = await runCli(fixture, args); assert.equal(result.exitCode, 1, args.join(" ")); assert.equal(result.stdout, ""); assert.match( result.stderr, - /type\s+:benches.render-wasm.node.runner\/invalid-args/, + /Usage: runner purge/, ); assert.deepEqual(tree(fixture.managed), original, args.join(" ")); } @@ -1216,19 +1340,14 @@ fs.writeFileSync(${JSON.stringify(marker)}, "ran"); ); const original = tree(fixture.managed); - const result = await runCli(fixture, [ - "run", - "--build", - "retained-id", - "--features", - "functional", - ]); - - assert.equal(result.exitCode, 1); - assert.equal(result.stdout, ""); - assert.match(result.stderr, /cannot be combined/); - assert.equal(fs.existsSync(marker), false); - assert.deepEqual(tree(fixture.managed), original); + for (const flags of [["--features", "functional"], ["--build-timeout-ms", "1000"]]) { + const result = await runCli(fixture, ["run", "--build", "retained-id", ...flags]); + assert.equal(result.exitCode, 1); + assert.equal(result.stdout, ""); + assert.match(result.stderr, /cannot be combined/); + assert.equal(fs.existsSync(marker), false); + assert.deepEqual(tree(fixture.managed), original); + } }, ); @@ -1280,7 +1399,7 @@ test( const abHelp = await runCli(fixture, ["ab", "--help"]); assert.equal(abHelp.exitCode, 0, abHelp.stderr); assert.match(abHelp.stdout, /runner ab --features/); - assert.match(abHelp.stdout, /--only-comparisons/); + assert.match(abHelp.stdout, /--max-features/); const mergeHelp = await runCli(fixture, ["merge", "--help"]); assert.equal(mergeHelp.exitCode, 0, mergeHelp.stderr); assert.match(mergeHelp.stdout, /runner merge BASE ADDITIONAL/); @@ -1327,14 +1446,15 @@ test( assert.match(result.stdout, /runner run/); assert.match(result.stdout, /runner compare/); assert.match(result.stdout, /runner merge/); - assert.match(result.stdout, /--cases/); - assert.match(result.stdout, /--allow-build-differences/); - assert.match(result.stdout, /--allow-scene-differences/); - assert.match(result.stdout, /--allow-all-differences/); + const compareHelp = await runCli(fixture, ["compare", "--help"]); + assert.equal(compareHelp.exitCode, 0, compareHelp.stderr); + assert.match(compareHelp.stdout, /--cases/); + assert.match(compareHelp.stdout, /--allow-build-differences/); + assert.match(compareHelp.stdout, /--allow-scene-differences/); + assert.match(compareHelp.stdout, /--allow-all-differences/); assert.match(result.stdout, /-q, --quiet/); assert.match(result.stdout, /-v, --verbose/); assert.match(result.stdout, /--\[no-\]color/); - assert.doesNotMatch(result.stdout, /--force/); assert.deepEqual(tree(fixture.managed), original); }, ); diff --git a/frontend/test/performance_tests/render_wasm/cli_test.cljs b/frontend/test/performance_tests/render_wasm/cli_test.cljs index 312cb824be..55de4daa9d 100644 --- a/frontend/test/performance_tests/render_wasm/cli_test.cljs +++ b/frontend/test/performance_tests/render_wasm/cli_test.cljs @@ -2,7 +2,7 @@ ;; License, v. 2.0. If a copy of the MPL was not distributed with this ;; file, You can obtain one at http://mozilla.org/MPL/2.0/. ;; -;; Copyright (c) KALEIDOS INC +;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns performance-tests.render-wasm.cli-test (:require diff --git a/frontend/test/performance_tests/render_wasm/compare_test.cljs b/frontend/test/performance_tests/render_wasm/compare_test.cljs index 4fbdd734a1..23e8b64455 100644 --- a/frontend/test/performance_tests/render_wasm/compare_test.cljs +++ b/frontend/test/performance_tests/render_wasm/compare_test.cljs @@ -125,7 +125,7 @@ candidate (write-run (path/join @temporary-root "candidate.transit") (make-run "candidate" :b/pan 12)) {:keys [text]} (compare/execute baseline candidate - {:cases "a/pan,b/pan"})] + {:case-pair ["a/pan" "b/pan"]})] (t/is (str/includes? text "Pair a/pan,b/pan")))) (t/deftest cases-requires-scene-permission-for-different-scenes @@ -134,7 +134,7 @@ candidate (write-run (path/join @temporary-root "candidate.transit") (make-run "candidate" :b/load 12 :scene :other :fingerprint (apply str (repeat 64 "b")))) - selector {:cases "a/load,b/load"}] + selector {:case-pair ["a/load" "b/load"]}] (t/is (= ::report-compare/incompatible (:type (ex-data (failure #(compare/execute baseline candidate selector)))))) (t/is (= ::report-compare/incompatible @@ -254,7 +254,7 @@ (t/is (str/includes? text "ACKNOWLEDGED DIFFERENCE")) (t/is (str/includes? text ":rule :all"))))) -(t/deftest invalid-selectors-and-unreadable-records-fail-at-the-boundary +(t/deftest unreadable-records-fail-at-the-boundary (let [baseline (write-run (path/join @temporary-root "baseline.transit") (make-run "baseline" :case/load 10)) candidate (write-run (path/join @temporary-root "candidate.transit") @@ -264,10 +264,6 @@ (fs/writeFileSync file (result/encode record)) (t/is (thrown-with-msg? js/Error #"expected a saved run record" (compare/execute baseline file {})))) - (doseq [selector ["case/load" "case/load,other/load,third/load" ""]] - (t/is (= ::report-compare/invalid-selector - (:type (ex-data (failure #(compare/execute baseline candidate - {:cases selector}))))))) (fs/writeFileSync (path/join @temporary-root "broken.transit") "not transit") (t/is (thrown? js/Error (compare/execute baseline (path/join @temporary-root "broken.transit") diff --git a/frontend/test/performance_tests/render_wasm/execution_test.cljs b/frontend/test/performance_tests/render_wasm/execution_test.cljs index ccb005340a..f5ec6976c7 100644 --- a/frontend/test/performance_tests/render_wasm/execution_test.cljs +++ b/frontend/test/performance_tests/render_wasm/execution_test.cljs @@ -2,7 +2,7 @@ ;; License, v. 2.0. If a copy of the MPL was not distributed with this ;; file, You can obtain one at http://mozilla.org/MPL/2.0/. ;; -;; Copyright (c) KALEIDOS INC +;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns performance-tests.render-wasm.execution-test (:require @@ -30,7 +30,7 @@ #js [promise (js/Promise. (fn [_ reject] - (vreset! timer (js/setTimeout #(reject (js/Error. "Signal test stalled")) 5000))))]) + (vreset! timer (js/setTimeout #(reject (js/Error. "Test stalled")) 5000))))]) (.finally #(js/clearTimeout @timer))))) (t/deftest ^:async real-signals-reach-the-callback-and-remover-restores-listeners @@ -38,7 +38,7 @@ before-term (.listenerCount js/process "SIGTERM") received (atom []) next-signal (volatile! nil) - remove (signals/on-signals + remove (execution/on-signals (fn [signal] (swap! received conj signal) (@next-signal signal))) @@ -83,15 +83,6 @@ [] [(.listenerCount js/process "SIGINT") (.listenerCount js/process "SIGTERM")]) -(defn- within - [promise] - (let [timer (volatile! nil)] - (-> (js/Promise.race - #js [promise - (js/Promise. - (fn [_ reject] - (vreset! timer (js/setTimeout #(reject (js/Error. "Execution test stalled")) 5000))))]) - (.finally #(js/clearTimeout @timer))))) (t/deftest ^:async sync-and-async-handlers-publish-once-and-release-listeners (doseq [async [false true]] diff --git a/frontend/test/performance_tests/render_wasm/feature_comparison_test.cljs b/frontend/test/performance_tests/render_wasm/feature_comparison_test.cljs index 4335001f2e..9b56250d32 100644 --- a/frontend/test/performance_tests/render_wasm/feature_comparison_test.cljs +++ b/frontend/test/performance_tests/render_wasm/feature_comparison_test.cljs @@ -21,6 +21,7 @@ [benches.render-wasm.node.files :as files] [benches.render-wasm.node.log :as log] [benches.render-wasm.node.process :as process] + [benches.render-wasm.node.request :as request] [benches.render-wasm.result :as result] [cljs.test :as t :include-macros true] [clojure.string :as str] @@ -50,9 +51,11 @@ (defn- planning-options [] - {:features ["b" "a"] :filter "rects/load" :seed 42 :dry-run true - :warmups 0 :repetitions 1 :timeout-ms 2000 - :build-timeout-ms 10000 :startup-timeout-ms 3000}) + (:options (request/apply-defaults + {:command "ab" + :options {:features ["b" "a"] :filter "rects/load" :seed 42 :dry-run true + :warmups 0 :repetitions 1 :timeout-ms 2000 + :build-timeout-ms 10000 :startup-timeout-ms 3000}}))) (defn- metadata-output [definitions] @@ -153,8 +156,7 @@ ready (await (node-plan/prepare (frontend-root) opts nil))] (t/is (= ["a" "b"] (get-in ready [:plan :features]))) (t/is (= (cases/collect-cases {:master-seed 42 :filter "rects/load"}) (:cases ready))) - (t/is (= (assoc opts :features ["a" "b"] :default-features false :max-features 3 :confirm false - :allow-build-differences false :allow-scene-differences false :allow-all-differences false) + (t/is (= (assoc opts :features ["a" "b"]) (:options ready))) (t/is (= output (:output-dir ready))) (t/is (not (fs/existsSync (path/join @temporary-root "new")))) @@ -1051,7 +1053,6 @@ :execute-run (execute-service {})})}))) record (:comparison result)] (t/is (= 1 (:exit-code result))) - (t/is (str/ends-with? @stdout "Status failed\n")) (t/is (seq (:errors result))) (t/is (= :running (:status record))) (t/is (= :building (get-in record [:configurations 0 :status]))) diff --git a/frontend/test/performance_tests/render_wasm/report_test.cljs b/frontend/test/performance_tests/render_wasm/report_test.cljs index 0e5d937f77..8d8ce5e4c9 100644 --- a/frontend/test/performance_tests/render_wasm/report_test.cljs +++ b/frontend/test/performance_tests/render_wasm/report_test.cljs @@ -543,7 +543,7 @@ :prep (assoc-in preparation [:fingerprint :digest] (apply str (repeat 64 "b")))}) result/encode result/decode) - pairs (compare/selector-pair base other "saved/pan,different/zoom") + pairs (compare/selector-pair base other ["saved/pan" "different/zoom"]) rejected (error-data #(compare/compare-runs base other {:pairs pairs})) comparison (-> (compare/compare-runs base other {:pairs pairs :allow-all-differences true}) result/encode result/decode) @@ -580,8 +580,7 @@ (t/is (every? :waived? mismatches)) (t/is (str/includes? text "ACKNOWLEDGED DIFFERENCE")) (t/is (str/includes? text ":field :scored?"))) - (t/is (= ["saved/pan"] (:available (error-data #(compare/resolve-case base "pan"))))) - (t/is (= ::compare/invalid-selector (:type (error-data #(compare/selector-pair base other "saved/pan"))))))) + (t/is (= ["saved/pan"] (:available (error-data #(compare/resolve-case base "pan"))))))) (t/deftest explicit-pairs-allow-unlike-names-with-matching-execution-and-fingerprints (let [base (-> (run-with [1]) result/encode result/decode) diff --git a/frontend/test/performance_tests/render_wasm/run_test.cljs b/frontend/test/performance_tests/render_wasm/run_test.cljs index bd776fcb56..ceebc197d0 100644 --- a/frontend/test/performance_tests/render_wasm/run_test.cljs +++ b/frontend/test/performance_tests/render_wasm/run_test.cljs @@ -15,6 +15,7 @@ [benches.render-wasm.node.artifacts :as artifacts] [benches.render-wasm.node.feature-comparison :as feature-comparison] [benches.render-wasm.node.log :as log] + [benches.render-wasm.node.request :as request] [benches.render-wasm.node.run :as run] [benches.render-wasm.node.runs :as runs] [benches.render-wasm.result :as result] @@ -180,7 +181,6 @@ :warmups 0 :repetitions 1 :timeout-ms 5000 - :build-timeout-ms 60000 :startup-timeout-ms 5000 :screenshots false :remove false @@ -208,46 +208,7 @@ [result] (result/decode (fs/readFileSync (get-in result [:home :record-path]) "utf8"))) -(t/deftest ^:async run-rejects-invalid-options-before-allocating-output - (doseq [invalid [{:seed 4294967296} - {:warmups -1} - {:repetitions 1.5} - {:timeout-ms js/Infinity} - {:build-timeout-ms 2147483648} - {:startup-timeout-ms 0} - {:screenshots "false"} - {:remove nil} - {:require-gpu 1} - {:repetitons 1}]] - (let [out (output-dir) - opts (merge (dissoc (options nil out) :browser) - {:build-id "unavailable"} invalid) - cause (try - (await (run/run @temporary-root opts nil)) - nil - (catch :default cause cause))] - (t/testing (pr-str invalid) - (t/is (= ::run/invalid-options (:type (ex-data cause)))) - (t/is (not (fs/existsSync out))))))) - -(t/deftest ^:async execute-rejects-invalid-options-before-writing-evidence - (let [root @temporary-root - fake (fake-browser {}) - home (runs/create-home {:root root :run-id "invalid-options" - :output-dir (output-dir)}) - files (vec (fs/readdirSync (:dir home))) - log-text (fs/readFileSync (:log-path home) "utf8") - opts (assoc (options fake (:dir home)) :screenshots "false") - cause (try - (await (run/execute root home (selected-build (build-dir) "test-build") - [(collected :case/a :fresh)] opts nil)) - nil - (catch :default cause cause))] - (t/is (= ::run/invalid-options (:type (ex-data cause)))) - (t/is (= files (vec (fs/readdirSync (:dir home))))) - (t/is (= log-text (fs/readFileSync (:log-path home) "utf8"))))) - -(t/deftest ^:async retained-run-prints-results-and-keeps-all-levels-in-its-log +(t/deftest ^:async retained-run-keeps-all-progress-levels-in-its-log (doseq [level [:info :warn :debug]] (let [root @temporary-root cases [(collected :case/a :fresh) (collected :case/b :reuse)] @@ -272,19 +233,21 @@ (log/configure {:level level :color :never :stdout #js {:write #(swap! stdout str %)} :stderr #js {:write #(swap! stderr str %)}}) - (let [response (await (run/run root (assoc (options fake out) :build-id (:id build) - :warmups 1 :repetitions 2) nil)) + (let [resolved (:options (request/apply-defaults + {:command "run" + :options (assoc (dissoc (options fake out) :browser) + :build {:kind :existing :id (:id build)} + :warmups 1 :repetitions 2)})) + response (await (run/run root resolved nil {:browser (:service fake)})) saved (saved-record response) text (fs/readFileSync (get-in response [:home :log-path]) "utf8")] (t/is (= 0 (:exit-code response))) - (t/is (= (str "Run " (:run-id saved) "\nOutput " out "\nReport\n" - (str/replace (:report response) #"\n$" "") "\n") @stdout)) + (t/is (= (:id build) (get-in saved [:metadata :build :provenance :build-id]))) (t/is (str/includes? text "▸ Case case/a (fresh)\n")) (doseq [case-text (rest (str/split text #"▸ Case "))] (t/is (str/includes? case-text " attempt 1/3 completed (warmup)\n")) (t/is (str/includes? case-text " attempt 2/3 completed\n")) (t/is (str/includes? case-text " attempt 3/3 completed\n"))) - (t/is (str/includes? text (:report response))) (t/is (str/includes? text "Browser FakeBrowser/1")) (t/is (= (not= :warn level) (str/includes? @stderr "▸ Case case/a"))) (t/is (= (= :debug level) (str/includes? @stderr "Browser FakeBrowser/1"))))))) diff --git a/frontend/test/performance_tests/render_wasm/runner.cljs b/frontend/test/performance_tests/render_wasm/runner.cljs index c7754f2152..d8ecaba150 100644 --- a/frontend/test/performance_tests/render_wasm/runner.cljs +++ b/frontend/test/performance_tests/render_wasm/runner.cljs @@ -34,11 +34,9 @@ [performance-tests.render-wasm.report-test] [performance-tests.render-wasm.request-test] [performance-tests.render-wasm.run-test] - [performance-tests.render-wasm.runner-test] [performance-tests.render-wasm.runs-test] [performance-tests.render-wasm.scene-snapshot-test] - [performance-tests.render-wasm.server-test] - [performance-tests.render-wasm.signals-test])) + [performance-tests.render-wasm.server-test])) (enable-console-print!) @@ -76,11 +74,9 @@ 'performance-tests.render-wasm.report-test 'performance-tests.render-wasm.request-test 'performance-tests.render-wasm.run-test - 'performance-tests.render-wasm.runner-test 'performance-tests.render-wasm.runs-test 'performance-tests.render-wasm.scene-snapshot-test - 'performance-tests.render-wasm.server-test - 'performance-tests.render-wasm.signals-test]) + 'performance-tests.render-wasm.server-test]) (assert (every? find-ns-obj test-namespaces) "test-namespaces contains a namespace that isn't required in runner.cljs") diff --git a/frontend/test/performance_tests/render_wasm/runner_test.cljs b/frontend/test/performance_tests/render_wasm/runner_test.cljs deleted file mode 100644 index 82d5d98a00..0000000000 --- a/frontend/test/performance_tests/render_wasm/runner_test.cljs +++ /dev/null @@ -1,330 +0,0 @@ -;; This Source Code Form is subject to the terms of the Mozilla Public -;; License, v. 2.0. If a copy of the MPL was not distributed with this -;; file, You can obtain one at http://mozilla.org/MPL/2.0/. -;; -;; Copyright (c) KALEIDOS SUBSIDIARY SL - -(ns performance-tests.render-wasm.runner-test - (:require - [benches.render-wasm.node.options :as options] - [benches.render-wasm.node.runner :as runner] - [cljs.test :as t :include-macros true] - [clojure.string :as str])) - -(t/deftest output-options-apply-to-every-command - (doseq [args [["build"] ["run"] ["purge"] ["compare" "base" "candidate"] - ["merge" "base" "additional"] - ["ab" "--features" "a" "--filter" "rects/load"]]] - (t/is (= :info (options/console-level (:options (runner/parse-args args))))) - (t/is (= :auto (options/color-mode (:options (runner/parse-args args))))) - (t/is (= :warn (options/console-level (:options (runner/parse-args (conj args "-q")))))) - (t/is (= :debug (options/console-level (:options (runner/parse-args (conj args "-v")))))) - (t/is (= :always (options/color-mode (:options (runner/parse-args (conj args "--color")))))) - (t/is (= :never (options/color-mode (:options (runner/parse-args (conj args "--no-color"))))))) - (t/is (thrown-with-msg? js/Error #"mutually exclusive" - (runner/parse-args ["run" "-q" "-v"])))) - -(t/deftest build-parses-defaults-and-explicit-functional-feature-selection - (let [{:keys [command options]} (runner/parse-args ["build"])] - (t/is (= "build" command)) - (t/is (true? (:default-features options))) - (t/is (= options/default-build-timeout-ms (:build-timeout-ms options)))) - (let [{:keys [command options]} (runner/parse-args - ["--" "build" "--no-default-features" - "--features" "functional_b,functional-a,functional_b" - "--build-timeout-ms" "1250"])] - (t/is (= "build" command)) - (t/is (= {:default-features false - :features ["functional-a" "functional_b"] - :build-timeout-ms 1250} - (select-keys options [:default-features :features :build-timeout-ms]))))) - -(t/deftest diagnostic-and-malformed-features-fail-at-the-parser-boundary - (doseq [feature ["stats" "profile" "profile-macros" "profile-raf"]] - (t/is (thrown-with-msg? js/Error #"diagnostic Cargo features" - (runner/parse-args ["build" "--features" (str "functional," feature)])))) - (doseq [feature ["" "functional,,other" "functional/other"]] - (t/is (thrown-with-msg? js/Error #"invalid Cargo feature selection" - (runner/parse-args ["build" "--features" feature])))) - (t/is (thrown? js/Error (runner/parse-args ["build" "--features"])))) - -(t/deftest preparation-budget-requires-a-complete-positive-safe-integer-number - ;; Number accepts whitespace and exponent notation when the result is an - ;; integer; these are valid CLI budgets, not partial numeric parses. - (doseq [[text expected] [["1" 1] [" 1000 " 1000] ["1e3" 1000]]] - (t/is (= expected (get-in (runner/parse-args ["build" "--build-timeout-ms" text]) - [:options :build-timeout-ms])))) - (doseq [text ["" "0" "-1" "1.5" "1000ms" "NaN" "Infinity" "9007199254740992"]] - (t/is (thrown-with-msg? js/Error #"Must be a positive integer" - (runner/parse-args ["build" "--build-timeout-ms" text])))) - (t/is (thrown? js/Error (runner/parse-args ["build" "--build-timeout-ms"])))) - -(t/deftest purge-categories-and-build-flags-cannot-be-mixed - (doseq [[args selected] [[["purge"] {}] - [["purge" "--only-builds"] {:only-builds true}] - [["purge" "--only-runs"] {:only-runs true}]]] - (let [{:keys [command options]} (runner/parse-args args)] - (t/is (= "purge" command)) - (t/is (= selected (select-keys options [:only-builds :only-runs]))))) - (t/is (thrown-with-msg? js/Error #"mutually exclusive" - (runner/parse-args ["purge" "--only-builds" "--only-runs"]))) - (doseq [flag ["--only-builds" "--only-runs"]] - (t/is (thrown-with-msg? js/Error #"purge category flags require purge" - (runner/parse-args ["build" flag])))) - (doseq [args [["purge" "--default-features"] - ["purge" "--no-default-features"] - ["purge" "--features" "functional"] - ["purge" "--build-timeout-ms" "1000"] - ["purge" "--features=functional"] - ["purge" "--build-timeout-ms=1000"]]] - (t/is (thrown-with-msg? js/Error #"build options require build" - (runner/parse-args args))))) - -(t/deftest help-needs-no-command-and-other-command-lines-must-be-valid - (doseq [args [["--help"] ["-h"] ["build" "--help"] ["run" "--help"] - ["compare" "--help"] ["merge" "--help"]]] - (let [{:keys [options summary]} (runner/parse-args args) - help (runner/usage summary)] - (t/is (true? (:help options))) - (t/is (str/includes? help "runner build")) - (t/is (str/includes? help "runner purge")) - (t/is (str/includes? help "runner run")) - (t/is (str/includes? help "runner compare")) - (t/is (str/includes? help "runner merge")) - (t/is (str/includes? help "--build-timeout-ms")) - (t/is (str/includes? help "--cases")) - (t/is (str/includes? help "--allow-build-differences")) - (t/is (str/includes? help "--allow-scene-differences")) - (t/is (str/includes? help "--allow-all-differences")) - (t/is (not (str/includes? help "--force"))))) - (t/is (thrown-with-msg? js/Error #"expected build, purge, run, ab, compare or merge" - (runner/parse-args []))) - (doseq [args [["unknown"] ["build" "purge"] ["build" "--unknown"] - ["run" "extra"] ["compare" "only-one"]]] - (t/is (thrown? js/Error (runner/parse-args args))))) - -(t/deftest run-resolves-default-and-explicit-options-before-resources - (let [{:keys [command options arguments]} (runner/parse-args ["run"])] - (t/is (= "run" command)) - (t/is (= ["run"] arguments)) - (t/is (= {:seed 42 :warmups 3 :repetitions 10 :timeout-ms 30000 - :build-timeout-ms 300000 :startup-timeout-ms 30000} - (select-keys options [:seed :warmups :repetitions :timeout-ms - :build-timeout-ms :startup-timeout-ms]))) - (t/is (nil? (:build-id options))) - (t/is (nil? (:filter options))) - (t/is (not (:remove options))) - (t/is (not (:screenshots options)))) - (let [{:keys [options]} (runner/parse-args - ["run" "--seed" "7" "--filter" "rects" - "--warmups" "0" "--repetitions" "2" - "--timeout-ms" "100" "--build-timeout-ms" "200" - "--startup-timeout-ms" "300" "--output-dir" "out" - "--remove" "--screenshots"])] - (t/is (= {:seed 7 :filter "rects" :warmups 0 :repetitions 2 - :timeout-ms 100 :build-timeout-ms 200 :startup-timeout-ms 300 - :output-dir "out" :remove true :screenshots true} - (select-keys options [:seed :filter :warmups :repetitions - :timeout-ms :build-timeout-ms - :startup-timeout-ms :output-dir - :remove :screenshots]))))) - -(t/deftest retained-runs-reject-feature-selection-and-build-runs-accept-it - (t/is (= "retained-id" (get-in (runner/parse-args ["run" "--build" "retained-id"]) - [:options :build-id]))) - (doseq [args [["run" "--build" "retained-id" "--no-default-features"] - ["run" "--no-default-features" "--build" "retained-id"] - ["run" "--build" "retained-id" "--features" "functional"] - ["run" "--build" "retained-id" "--default-features"]]] - (t/is (thrown-with-msg? js/Error #"cannot be combined" - (runner/parse-args args)))) - (let [{:keys [options]} (runner/parse-args - ["run" "--no-default-features" "--features" "functional"])] - (t/is (false? (:default-features options))) - (t/is (= ["functional"] (:features options))))) - -(t/deftest compare-takes-two-file-paths-and-its-own-options - (let [{:keys [command arguments options]} - (runner/parse-args ["compare" "baseline.transit" "candidate.transit"])] - (t/is (= "compare" command)) - (t/is (= ["compare" "baseline.transit" "candidate.transit"] arguments)) - (t/is (nil? (:cases options))) - (t/is (not-any? options [:allow-build-differences :allow-scene-differences - :allow-all-differences]))) - (doseq [[flag option] [["--allow-build-differences" :allow-build-differences] - ["--allow-scene-differences" :allow-scene-differences] - ["--allow-all-differences" :allow-all-differences]] - selectors [[] ["--cases" "rects/pan,paths/pan"]]] - (let [{:keys [options]} - (runner/parse-args (into ["compare" "a.transit" "b.transit" flag] selectors))] - (t/is (= (when (seq selectors) "rects/pan,paths/pan") (:cases options))) - (t/is (true? (get options option))) - (t/is (not-any? options (remove #{option} [:allow-build-differences - :allow-scene-differences - :allow-all-differences]))))) - (let [{:keys [options]} - (runner/parse-args ["compare" "a.transit" "b.transit" - "--allow-build-differences" "--allow-scene-differences"])] - (t/is (true? (:allow-build-differences options))) - (t/is (true? (:allow-scene-differences options))) - (t/is (not (:allow-all-differences options))))) - -(t/deftest merge-takes-two-paths-and-only-output-dir - (let [{:keys [command arguments options]} - (runner/parse-args ["merge" "base.transit" "additional.transit" - "--output-dir" "merged-home"])] - (t/is (= "merge" command)) - (t/is (= ["merge" "base.transit" "additional.transit"] arguments)) - (t/is (= "merged-home" (:output-dir options)))) - (doseq [args [["merge"] ["merge" "only-base"] - ["merge" "base" "additional" "extra"]]] - (t/is (thrown? js/Error (runner/parse-args args)))) - (doseq [flags [["--build" "id"] - ["--filter" "rects"] - ["--seed" "1"] - ["--require-gpu"] - ["--features" "functional"] - ["--cases" "a,b"] - ["--allow-build-differences"] - ["--only-builds"] - ["--dry-run"]]] - (t/is (thrown? js/Error (runner/parse-args (into ["merge" "base" "additional"] flags)))))) - -(t/deftest force-is-rejected-without-an-alias - (doseq [args [["compare" "a.transit" "b.transit" "--force"] - ["compare" "a.transit" "b.transit" "--force" "--allow-all-differences"] - ["compare" "a.transit" "b.transit" "--cases" "a,b" "--force"]]] - (t/is (thrown? js/Error (runner/parse-args args))))) - -(t/deftest commands-reject-flags-owned-by-other-commands - (doseq [[args message] - [[["run" "--only-builds"] #"purge category flags require purge"] - [["run" "--cases" "a,b"] #"compare options require compare"] - [["compare" "a" "b" "--seed" "1"] #"run options require run"] - [["compare" "a" "b" "--features" "functional"] #"build options require build"] - [["build" "--build" "retained"] #"run options require run"] - [["build" "--only-runs"] #"purge category flags require purge"] - [["purge" "--seed" "1"] #"run options require run"] - [["purge" "--cases" "a,b"] #"compare options require compare"] - [["merge" "a" "b" "--seed" "1"] #"run options require run"] - [["merge" "a" "b" "--features" "functional"] #"build options require build"] - [["merge" "a" "b" "--cases" "a,b"] #"compare options require compare"] - [["merge" "a" "b" "--only-builds"] #"purge category flags require purge"] - [["merge" "a" "b" "--dry-run"] #"ab planning options require ab"]]] - (t/is (thrown-with-msg? js/Error message (runner/parse-args args)))) - (doseq [command ["build" "purge" "run"] - flag ["--allow-build-differences" "--allow-scene-differences" - "--allow-all-differences"]] - (t/is (thrown-with-msg? js/Error #"compare options require compare" - (runner/parse-args [command flag]))))) - -(t/deftest numeric-and-path-options-reject-invalid-bounds - (doseq [args [["run" "--seed" "4294967296"] - ["run" "--seed" "-1"] - ["run" "--warmups" "-1"] - ["run" "--warmups" "1.5"] - ["run" "--repetitions" "0"] - ["run" "--timeout-ms" "0"] - ["run" "--timeout-ms" "2147483648"] - ["run" "--startup-timeout-ms" "0"] - ["run" "--build" ""] - ["run" "--output-dir" ""]]] - (t/is (thrown? js/Error (runner/parse-args args)))) - (t/is (= 4294967295 (:seed (:options (runner/parse-args ["run" "--seed" "4294967295"]))))) - (t/is (= 0 (:warmups (:options (runner/parse-args ["run" "--warmups" "0"])))))) - -(t/deftest option-looking-values-and-filenames-are-not-command-flags - (let [parsed (runner/parse-args ["compare" "--" "--seed" "candidate.transit"])] - (t/is (= ["compare" "--seed" "candidate.transit"] (:arguments parsed)))) - (let [parsed (runner/parse-args ["run" "--filter" "--allow-all-differences"])] - (t/is (= "--allow-all-differences" (:filter (:options parsed)))))) - -(t/deftest numeric-options-reject-blank-values - (doseq [args [["run" "--seed" ""] - ["run" "--warmups" " "] - ["run" "--repetitions" ""] - ["run" "--timeout-ms" " "]]] - (t/is (thrown? js/Error (runner/parse-args args))))) - -(t/deftest gpu-policy-is-available-for-runs-and-feature-comparisons - (doseq [args [["run" "--require-gpu"] - ["ab" "--features" "a" "--filter" "rects/load" "--require-gpu"] - ["ab" "--features" "a" "--require-gpu" "--dry-run"]]] - (t/is (true? (get-in (runner/parse-args args) [:options :require-gpu])))) - (doseq [args [["build" "--require-gpu"] - ["purge" "--require-gpu"] - ["compare" "a" "b" "--require-gpu"]]] - (t/is (thrown-with-msg? js/Error #"run options require run" (runner/parse-args args)))) - (t/is (re-find #"--require-gpu" (runner/usage "")))) - -(t/deftest ab-accepts-run-budgets-and-comparison-permissions - (let [{:keys [command options]} - (runner/parse-args ["ab" "--features" " b,a,b " "--filter" "rects/load" - "--seed" "7" "--warmups" "0" "--repetitions" "2" - "--timeout-ms" "1000" "--build-timeout-ms" "2000" - "--startup-timeout-ms" "3000" "--output-dir" "new-home" - "--screenshots" "--allow-build-differences" - "--allow-scene-differences" "--allow-all-differences"])] - (t/is (= "ab" command)) - (t/is (= {:features ["a" "b"] :filter "rects/load" :seed 7 - :warmups 0 :repetitions 2 :timeout-ms 1000 :build-timeout-ms 2000 - :startup-timeout-ms 3000 :output-dir "new-home" :screenshots true - :allow-build-differences true :allow-scene-differences true :allow-all-differences true - :max-features 3 :confirm false :dry-run false :default-features false :color nil} - options)))) - -(t/deftest ab-rejects-missing-features-and-incompatible-flags-before-execution - (t/is (thrown-with-msg? js/Error #"at least one" (runner/parse-args ["ab" "--dry-run"]))) - (doseq [args [["--build" "id"] ["--remove"] ["--cases" "a,b"] - ["--default-features"] ["--no-default-features"]]] - (t/is (thrown-with-msg? js/Error #"ab rejects" - (runner/parse-args (into ["ab" "--features" "a" "--dry-run"] args))))) - (doseq [features ["" "a," "profile" "a/b"]] - (t/is (thrown? js/Error (runner/parse-args ["ab" "--features" features "--dry-run"])))) - (t/is (thrown-with-msg? js/Error #"purge category flags require purge" - (runner/parse-args ["ab" "--features" "a" "--only-comparisons"]))) - (t/is (thrown? js/Error (runner/parse-args ["ab" "extra" "--features" "a" "--dry-run"])))) - -(t/deftest ab-planning-errors-keep-their-cause - (let [error (try - (runner/parse-args ["ab" "--dry-run"]) - nil - (catch :default cause cause))] - (t/is (= :benches.render-wasm.node.runner/invalid-args - (:type (ex-data error)))) - (t/is (some? (ex-cause error))) - (t/is (str/includes? (ex-message (ex-cause error)) "at least one")))) - -(t/deftest ab-limits-and-confirmation-apply-at-the-parser-boundary - (t/is (thrown-with-msg? js/Error #"requires --confirm" (runner/parse-args ["ab" "--features" "a"]))) - (t/is (:confirm (:options (runner/parse-args ["ab" "--features" "a" "--confirm"])))) - (t/is (:dry-run (:options (runner/parse-args ["ab" "--features" "a" "--dry-run"])))) - (t/is (thrown-with-msg? js/Error #"exceed max-features" - (runner/parse-args ["ab" "--features" "a,b,c,d" "--dry-run"]))) - (t/is (thrown-with-msg? js/Error #"requires --confirm" - (runner/parse-args ["ab" "--features" "a,b,c,d" "--max-features" "4" "--filter" "rects"]))) - (doseq [flag ["--confirm" "--dry-run"]] - (t/is (= 4 (:max-features (:options (runner/parse-args ["ab" "--features" "a,b,c,d" "--max-features" "4" flag])))))) - (doseq [limit ["" " " "0" "7" "-1" "1.5" "NaN" "Infinity" "3x"]] - (t/is (thrown? js/Error (runner/parse-args ["ab" "--features" "a" "--max-features" limit "--dry-run"]))))) - -(t/deftest planning-flags-are-only-for-ab-and-help-explains-the-contract - (doseq [command [["build"] ["run"] ["purge"] ["compare" "before" "after"]] - flags [["--dry-run"] ["--confirm"] ["--max-features" "2"]]] - (t/is (thrown-with-msg? js/Error #"ab planning options require ab" - (runner/parse-args (into command flags))))) - (let [{:keys [summary]} (runner/parse-args ["ab" "--help"]) - help (runner/usage summary)] - (doseq [part ["runner ab --features" "--only-comparisons" "--max-features" - "--confirm" "--dry-run" "Cargo defaults disabled" - "feature-comparisons//feature-comparison.transit"]] - (t/is (str/includes? help part))))) - -(t/deftest comparison-purge-is-exclusive-and-only-for-purge - (t/is (true? (get-in (runner/parse-args ["purge" "--only-comparisons"]) [:options :only-comparisons]))) - (doseq [other ["--only-builds" "--only-runs"]] - (t/is (thrown-with-msg? js/Error #"mutually exclusive" - (runner/parse-args ["purge" "--only-comparisons" other])))) - (doseq [command [["build"] ["run"] ["compare" "before" "after"]]] - (t/is (thrown-with-msg? js/Error #"purge category flags require purge" - (runner/parse-args (conj command "--only-comparisons"))))))