diff --git a/frontend/performance/benches/render_wasm/README.md b/frontend/performance/benches/render_wasm/README.md index 590bdca0fd..712a8aff36 100644 --- a/frontend/performance/benches/render_wasm/README.md +++ b/frontend/performance/benches/render_wasm/README.md @@ -93,9 +93,9 @@ The suite has three parts: fingerprinting, upload-order derivation, and upload. It then runs the optional preparation callback and drains to `Full` unless the case declares `:preparation {:render :none}`. One page owns one preparation. -- **Attempt**: one execution of a case against a preparation. An attempt - records one outcome, one set of metrics, and a raw trace. A failed attempt - still consumes its allocated slot. +- **Attempt**: one execution of a case against a preparation. It records an + `:attempt-id`, one outcome, main and optional reset phase blocks, and failure + or partial evidence. A failed attempt still consumes its allocated slot. - **Warmup**: an attempt that runs the case before measured attempts to warm caches and stabilize the page. The run records warmups but excludes them from statistics. @@ -105,9 +105,9 @@ The suite has three parts: processes it starts, fixes warmup and repetition counts, records failures in chronological order, and performs no hidden retries. - **Run record**: the saved evidence for one run: plan, build and environment - provenance, preparations, attempts, metrics, traces, failures, remaining - slots, and termination. The runner stores it as a data file validated by a - schema. + provenance, quantity descriptions, preparations, attempts with their phase + histories, failures, remaining slots, and termination. The runner stores it as + a data file validated by a schema. - **Bridge**: the object that runs in the browser, receives plain requests from Node, owns the page's renderer context, runs preparations and attempts, and returns plain results. @@ -120,13 +120,14 @@ The suite has three parts: - **Measurement**: a numeric result for a quantity such as elapsed duration. The suite measures module start, graphics start, upload, first render, and camera movement separately. -- **Metric**: the name and definition of a measured quantity. Saved run metrics - declare a unit, semantics, basis, and zero policy. Semantics identifies what - the value measures. The basis states whether a value covers an attempt or - averages its inner calls. The zero policy states whether zero supplies a valid - value for analysis. -- **Observation**: one valid value from a measured attempt for statistical - analysis. A batch average supplies one observation for its attempt. +- **Metric**: the name of a measured quantity. A command event records its raw + numeric values in a vector. A saved run supplies a `:unit` and an optional + `:description` for each metric type. It stores no statistic or reduction. +- **Metric key**: `[phase scope metric-type]`. Scope is the command that recorded + the metric or `:global` for every command with that type in the phase. +- **Observation**: one eligible phase execution's sum for one metric key. Inner + calls, slices, and loop repetitions stay within that execution and do not add + observations. Warmups and failed attempts supply none. - **Slice**: a trace entry for one renderer call: timestamp, flags, frame type, and duration. Slices describe the work within an attempt. - **Trace**: details of the work, including individual renderer calls, @@ -189,7 +190,7 @@ The suite has three parts: - **Report**: the statistics and tables computed from one or more run records. Reports run in Node or the browser and do not load the renderer. - **Comparison**: an offline analysis of two saved run records. It pairs cases, - checks compatibility, and reports changes for each metric with intervals. + checks compatibility, and reports changes for each metric key with intervals. - **A/B comparison**: a comparison of two runs that differ by one renderer feature. The feature is off in one run and on in the other. - **Permission**: In principle, only run records with equal builds and equal @@ -263,8 +264,12 @@ sequenceDiagram B->>W: create the WebGL2 context B->>B: build, validate, fingerprint the snapshot B->>W: upload shapes in tree order - B->>W: render until Full - B-->>N: preparation evidence and raw phase blocks + B->>B: run preparation callback + B->>W: render Full unless disabled + B-->>N: preparation evidence and raw phase block + N->>N: persist the run record + N->>B: run main and reset + B-->>N: raw main and reset phase blocks N->>N: persist the run record ``` @@ -789,7 +794,9 @@ reading. Promise callbacks can run later as the browser schedules queued work. Measured command events keep each metric's raw values in a vector. Repeated commands produce separate events, including when they name the same metric. -Raw values preserve zero, negative, and non-finite measurements for analysis. +Raw values preserve zero, negative, and non-finite measurements in the saved +record. Reports sum one metric type within one phase execution before computing +statistics; saved records contain no aggregates. ### Renderer completion @@ -854,25 +861,29 @@ A run record contains: timeout policy; - metadata: Git revision and dirty flag, build facts, environment, and diagnostics; - the resolved analysis settings; +- saved quantity descriptions; - preparations in chronological order; -- attempts in chronological order, each with its case, preparation, outcome, - metrics, trace, and failure evidence; +- attempts in chronological order, each with its case, preparation, + `:attempt-id`, warmup flag, outcome, main/reset phase blocks, and failure or + partial evidence; - remaining slots for each case; - the termination reason. -A preparation records its case, fingerprint, effective graphics, and setup -diagnostics. An attempt records one outcome and its metric entries. Entries can -describe invalid values or missing measurements. Errors retain available -traces of render calls and waits after a failure. +A preparation records its case, warmup flag, setup diagnostics, phase history, +and any failure. A completed preparation also has its fingerprint and effective +graphics. An attempt records one outcome and its main and optional reset phase +blocks. A phase block retains ordered command events, raw metric vectors, and +optional completion diagnostics. Errors retain available phase evidence and +traces after a failure. ### Failures and termination A failed attempt still consumes its allocated slot. A fresh preparation failure -records a failed attempt and consumes the slot. A warm preparation failure or a -warm attempt failure releases the prepared page and context and leaves that -case's remaining slots unattempted. A browser or page failure closes the browser -it started, abandons the current case, and starts a new browser for the next -case when possible. The run fixes its attempt count before execution and uses +records a failed attempt and consumes the slot. A reuse case's preparation +failure or attempt failure releases the prepared page and context and leaves +that case's remaining slots unattempted. A browser or page failure closes the +browser it started, abandons the current case, and starts a new browser for the +next case when possible. The run fixes its attempt count before execution and uses each allocated slot at most once. A run terminates in one of four states: @@ -890,22 +901,44 @@ that ended in failure. ### Summaries -A summary reports, for each case and metric: +A summary reports one entry per case and metric key +`[phase scope metric-type]`. The scope is the command that recorded the metric, +or `:global` for every command with that type in the phase. For each key, the +report sums the raw values of one metric type within each eligible phase +execution and computes statistics across those execution sums. It labels the +reduction `:statistic :sum`. + +For each metric key, the report includes: - the observed mean, median, minimum, maximum, sample standard deviation, and median absolute deviation; - a bootstrap confidence interval for the median. The interval comes from - repeated resampling of the observed attempts; -- observed p95 and p99 values, the 95th and 99th percentile durations, when the - sample is large enough; -- counts in the valid, invalid, absent, failed, unattempted, and warmup - categories; -- the definitions and units of the metric. + repeated resampling of the observed execution sums; +- observed p95 and p99 values when the sample is large enough; +- counts for the valid, invalid, absent, failed, unattempted, and warmup + categories, plus command invocation and raw-value counts; +- the saved quantity unit and optional description, and any validity + diagnostics. -Statistics use valid values from completed measured attempts. The record also -retains warmups and failures. A completed attempt can supply a valid value for -one metric and an invalid value for another. A pooled estimate combines values -across attempts and requires matching metric definitions. +Statistical `n` is the valid count. Invocations count command events and +raw-values count all recorded numbers across eligible executions; neither is the +number of observations. Warmups and failed attempts are excluded. A failed +attempt excludes both its main and reset blocks. A completed preparation remains +eligible after a later main or reset failure. A fresh warmup preparation is +excluded; one reuse preparation supplies one observation, not one per attempt. + +A finite zero is valid. Negative and non-finite raw values remain in the saved +record but invalidate the dependent execution sum and aggregate; independent +metric keys remain usable. Unknown or malformed quantity metadata appears as a +diagnostic and leaves that key without usable statistics. A phase can complete +with no events, but a completed run requires at least one measurement across +each case's phases. + +The case report also counts actual preparation, main, and reset executions. +Completed and failed counts use each execution's own outcome, and a separate +count records warmup executions. These phase counts do not filter by attempt +eligibility. A completed main block from an attempt whose reset failed still +appears in the main phase counts but supplies no observation. The default analysis settings are: @@ -922,10 +955,10 @@ The run records draw count, seed, confidence, and minimum counts, and a report can override them at analysis time. An estimate that cannot be computed keeps a reason and, when applicable, the minimum count it needs. -The intervals are exploratory and conditional on the observed attempts. -Attempts within one run share evolving caches and can have dependent durations. -Repeated whole runs provide evidence about reproducibility across those cache -conditions. +The intervals are exploratory and conditional on available executions. +Incomplete and timed-out cases can select which observations survive. Attempts +within one run share evolving caches and can have dependent durations. Repeated +whole runs provide evidence about reproducibility across those cache conditions. ### Comparisons @@ -943,15 +976,21 @@ By default, a comparison requires equal: - effective graphics settings and runtime environment; - build mode, effective features, instrumentation settings, and recorded build environment; -- warmup count, timeout policy, and metric definitions. +- warmup count, timeout policy, and saved quantity units. + +Comparison pairs metric keys `[phase scope metric-type]`. Quantity description +text does not affect compatibility; changed prose is not a difference. Unknown +or malformed quantity metadata appears in the per-side report diagnostics and +leaves that key without usable statistics. Git revision, dirty state, run ids, timestamps, and saved analysis defaults provide provenance for interpreting the paired cases. -The comparison reports absolute and percentage median changes, with a bootstrap -interval for each change. A zero baseline suppresses a percentage change. The -comparison leaves a metric with differing definitions out unless the -all-differences permission allows raw values. +The comparison reports absolute and percentage median changes between execution +sums, with a bootstrap interval for each change. A zero baseline suppresses a +percentage change. The comparison leaves a metric key with differing quantity +units or incomplete quantity evidence out unless the all-differences +permission allows its valid raw values. ### Comparison permissions @@ -961,16 +1000,16 @@ mismatch with the rule that allowed it. - `--allow-build-differences` acknowledges recorded build settings, effective features, instrumentation, and the recorded build environment. Runtime - environment, graphics, operation settings, and metric definitions stay + environment, graphics, operation settings, and saved quantity units stay strict. - `--allow-scene-differences` acknowledges scene identity, scene seed, scene parameters, and differences in fingerprint content. Execution settings stay strict. Missing or varying fingerprint evidence still needs the all-differences permission. - `--allow-all-differences` acknowledges every compatibility difference, - including mixed metric definitions. The report compares valid raw values in - their recorded units. Record validation and measurement validity still govern - the analysis. + including mixed quantity units. The report compares valid raw execution sums + in their recorded units. Record validation and measurement validity still + govern the analysis. Build and scene permissions combine by union. All-differences includes both. A report shows the selected permissions, every recorded difference, and the @@ -1093,7 +1132,7 @@ and makes the invocation exit 1. Other build differences require `--allow-build-differences`. Scene differences require `--allow-scene-differences`. Operation, view, context, preparation, completion, batch size, graphics, runtime environment, and -metric-definition differences stay strict unless `--allow-all-differences` is +saved quantity-unit differences stay strict unless `--allow-all-differences` is selected. If two configurations report the same effective feature set, the comparison includes their edge and labels it with that fact. Effective feature sets describe the code settings that accompany the observed timings. @@ -1346,11 +1385,11 @@ frontend/performance/benches/render_wasm/ ├── snapshot.cljs snapshot validation and upload order ├── fingerprint.cljs digest of the starting snapshot ├── identity.cljc build identity and bundle compatibility checks -├── measurement.cljs metric normalization +├── measurement.cljs projection of phase evidence into attempts ├── random.cljs seeded random generators -├── result.cljs run, comparison and feature-comparison records +├── result.cljs raw run, comparison and feature-comparison records ├── README.md this document -└── REPORTING.md records, metrics, and statistics +└── REPORTING.md raw records, metrics, and statistics frontend/test/performance_tests/render_wasm/ ├── runner.cljs test entry point, namespace registration, focus selection @@ -1366,9 +1405,9 @@ render-wasm/ Rust renderer and build ## Further reading -- [`REPORTING.md`](REPORTING.md) describes the record schemas, metric entries, - and statistical calculations in detail. This guide is the source for the - command line and the A/B workflow. +- [`REPORTING.md`](REPORTING.md) describes the record schemas, raw command + histories, quantity metadata, and statistical calculations in detail. This + guide is the source for the command line and the A/B workflow. - [`render-wasm/README.md`](../../../../render-wasm/README.md) and the renderer source describe the Rust side. - [`frontend/src/app/render_wasm/`](../../../src/app/render_wasm/) contains the diff --git a/frontend/performance/benches/render_wasm/measurement.cljs b/frontend/performance/benches/render_wasm/measurement.cljs index 02e92f9030..1eda48be87 100644 --- a/frontend/performance/benches/render_wasm/measurement.cljs +++ b/frontend/performance/benches/render_wasm/measurement.cljs @@ -5,67 +5,23 @@ ;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns benches.render-wasm.measurement - "Measurement normalization runs after renderer timing ends. - Explicit entries declare which values supply observations. Raw traces retain - zero spans, non-finite values and diagnostic settings. Render-call totals sum - existing final/cached call durations and supply one observation per attempt." - (:require - [benches.render-wasm.result :as result])) - -(defn- duration - "Declares one total duration with its timing semantics and zero policy." - [value semantics zero-policy] - (result/metric {:value value :unit :ms :semantics semantics - :basis {:kind :attempt-total} :zero-policy zero-policy})) - -(defn- add-duration - "Adds a known duration only when its raw field exists, preserving explicit nil." - [metrics measurement field id semantics zero-policy] - (if (contains? measurement field) - (assoc metrics id (duration (get measurement field) semantics zero-policy)) - metrics)) - -(defn render-call-total - "Sums existing call durations after timing, without interpreting GPU or CPU work. - No calls or missing durations yield nil; recorded zero durations remain zero. - NaN and infinities propagate as raw values." - [measurement] - (let [slices (concat (:slices measurement) (:cached-slices measurement))] - (when (and (seq slices) (every? #(number? (:duration-ms %)) slices)) - (reduce + 0 (map :duration-ms slices))))) - -(defn metrics - "Normalizes standard durations and any authored explicit metric entries. - Fresh Full completion has the sole metric ID :first-render-ms. Warm setup - stays in preparation diagnostics. Requested waits and counts stay raw." - [measurement fresh? setup] - (let [entries (into {} (map (fn [[id entry]] [id (result/metric entry)])) (:metrics measurement)) - entries (if fresh? - (-> entries - (add-duration setup :module-init-ms :module-init-ms :module-instantiation :positive) - (add-duration setup :graphics-init-ms :graphics-init-ms :graphics-initialization :positive) - (add-duration setup :upload-ms :upload-ms :shared-encoding-and-upload :positive) - (add-duration measurement :full-ms :first-render-ms :prepared-scene-to-full :positive) - (add-duration measurement :viewport-ready-ms :viewport-ready-ms :prepared-scene-to-viewport-ready :positive)) - (-> entries - (add-duration measurement :interact-ms :interact-ms :camera-session-to-full :positive) - (add-duration measurement :active-ms :active-ms :camera-active-span :positive) - (add-duration measurement :settling-actual-ms :settling-actual-ms :actual-settling-span :allow) - (add-duration measurement :set-view-end-ms :set-view-end-ms :synchronous-set-view-end-call :allow) - (add-duration measurement :time-to-viewport-ready-ms :time-to-viewport-ready-ms :finalization-to-viewport-ready :positive) - (add-duration measurement :time-to-full-ms :time-to-full-ms :finalization-to-full :positive) - (add-duration measurement :last-input-to-full-ms :last-input-to-full-ms :last-input-to-full :positive)))] - (if (or (contains? measurement :slices) (contains? measurement :cached-slices)) - (assoc entries :renderer-call-total-ms - (duration (render-call-total measurement) :synchronous-render-call-elapsed :allow)) - entries))) + "Projects browser phase evidence into saved attempts.") (defn attempt - "Projects bridge evidence into a raw attempt; the run assigns its ordinal. - Failed evidence remains raw and supplies no statistical observations." + "Returns an attempt with raw phase histories and available failure evidence. + + Args: + case-id: saved case identifier. + preparation-id: preparation identifier, or nil for an early failure. + warmup?: whether this execution occupies a warmup slot. + evidence: browser response with status, phases and optional failure details. + + Returns: + An attempt ready for chronological recording." [case-id preparation-id warmup? evidence] (cond-> {:case-id case-id :preparation-id preparation-id :warmup? warmup? :outcome (if (= "ok" (:status evidence)) :completed :failed) - :metrics (or (:metrics evidence) {}) :trace (or (:trace evidence) {})} - (not= "ok" (:status evidence)) (assoc :failure (select-keys evidence [:status :phase :message :cause])) + :phases (filterv #(#{:main :reset} (:phase %)) (:phases evidence))} + (not= "ok" (:status evidence)) + (assoc :failure (select-keys evidence [:status :phase :message :cause])) (:partial evidence) (assoc :partial (:partial evidence)))) diff --git a/frontend/performance/benches/render_wasm/node/runner/run.cljs b/frontend/performance/benches/render_wasm/node/runner/run.cljs index ecc4af0471..f82d557764 100644 --- a/frontend/performance/benches/render_wasm/node/runner/run.cljs +++ b/frontend/performance/benches/render_wasm/node/runner/run.cljs @@ -58,7 +58,7 @@ "Projects a bridge failure response into diagnostics evidence." [evidence fallback-phase] (if (map? evidence) - (cond-> (select-keys evidence [:status :phase :message :cause :partial]) + (cond-> (select-keys evidence [:status :phase :message :cause :partial :phases]) (nil? (:status evidence)) (assoc :status "failed") (nil? (:phase evidence)) (assoc :phase (name fallback-phase))) {:status "failed" @@ -194,23 +194,31 @@ (str "/renderer/" file "?build=" (js/encodeURIComponent build-id))) (defn- ^:async prepare-case - [state case page deadline] + [state case page deadline warmup?] (let [opts (:opts state) build-id (:id (:selected-build state)) + preparation-id (str (random-uuid)) response (attempt (await ((get-in state [:services :prepare]) page (transit/encode-str {:case case - :preparation-id (str (random-uuid)) + :preparation-id preparation-id :timeout-ms (:timeout-ms opts) :module-url (renderer-url build-id "render-wasm.js") :wasm-url (renderer-url build-id "render-wasm.wasm")}) {:deadline deadline :signal (:signal state) :phase :preparation})))] - (cond - (failure? response) {:failure (cause-evidence response :preparation)} - (= "ok" (:status response)) {:preparation (:preparation response)} - :else {:failure (failure-evidence response :preparation)}))) + (if (and (not (failure? response)) (= "ok" (:status response))) + {:preparation (assoc (:preparation response) :warmup? warmup?)} + (let [failure (if (failure? response) + (cause-evidence response :preparation) + (failure-evidence response :preparation))] + (record-preparation state {:preparation-id preparation-id :case-id (:id case) + :warmup? warmup? + :phases (or (:phases failure) []) + :diagnostics (or (get-in failure [:partial :setup]) {}) + :failure (dissoc failure :phases)}) + {:failure failure :preparation-id preparation-id})))) (defn- ^:async run-attempt "Returns the bridge evidence of one attempt, or the failure." @@ -363,10 +371,10 @@ (defn- ^:async run-fresh-slot [state case page warmup? final?] (let [deadline (process/deadline (:timeout-ms (:opts state))) - prepared (await (prepare-case state case page deadline))] + prepared (await (prepare-case state case page deadline warmup?))] (if-let [evidence (:failure prepared)] (do - (record-attempt state (measurement/attempt (:id case) nil warmup? evidence)) + (record-attempt state (measurement/attempt (:id case) (:preparation-id prepared) warmup? evidence)) (await (finish-slot state case page (if (interrupted? state) :interrupted :failed) final?))) (let [preparation (:preparation prepared)] @@ -435,7 +443,7 @@ (defn- ^:async run-reuse-page [state case page] (let [deadline (process/deadline (:timeout-ms (:opts state))) - prepared (await (prepare-case state case page deadline))] + prepared (await (prepare-case state case page deadline false))] (if-let [evidence (:failure prepared)] (await (abandon-preparation state case evidence)) (await (run-reuse-prepared state case page (:preparation prepared)))))) @@ -563,7 +571,11 @@ (defn- finish-run [state] (let [termination (get-termination @(:run-ref state) (interruption (:signal state)) - @(:harness-failed? state))] + @(:harness-failed? state)) + unmeasured (result/get-unmeasured-cases @(:run-ref state)) + termination (if (and (= :completed (:reason termination)) (seq unmeasured)) + {:reason :no-measurements :case-ids unmeasured} + termination)] (when (:screenshots (:opts state)) (swap! (:run-ref state) update-in [:metadata :diagnostics :screenshots] complete-screenshots (:cases state))) diff --git a/frontend/performance/benches/render_wasm/quantities.cljs b/frontend/performance/benches/render_wasm/quantities.cljs new file mode 100644 index 0000000000..79fe16c1f4 --- /dev/null +++ b/frontend/performance/benches/render_wasm/quantities.cljs @@ -0,0 +1,23 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns benches.render-wasm.quantities + "Portable descriptions of raw measured quantities.") + +(def schema:description + [:map {:closed true} + [:unit keyword?] + [:description {:optional true} string?]]) + +(def descriptions + {:command/elapsed + {:unit :ms :description "Command core start through its completion endpoint."} + :renderer/call-elapsed + {:unit :ms :description "Elapsed time of each synchronous renderer call."} + :renderer/cache-call-elapsed + {:unit :ms :description "Elapsed time of each synchronous cached renderer call."} + :camera/finalization-to-full + {:unit :ms :description "Camera finalization through the renderer return at Full."}}) diff --git a/frontend/performance/benches/render_wasm/report/compare.cljs b/frontend/performance/benches/render_wasm/report/compare.cljs index 85070ca0cf..f6696ba5a7 100644 --- a/frontend/performance/benches/render_wasm/report/compare.cljs +++ b/frontend/performance/benches/render_wasm/report/compare.cljs @@ -67,7 +67,8 @@ [field baseline candidate] (concat (when-let [d (difference field baseline candidate)] [d]) - (when (or (not= 1 (count baseline)) (not= 1 (count candidate))) + (when (or (not= 1 (count baseline)) (not= 1 (count candidate)) + (contains? baseline nil) (contains? candidate nil)) [{:field [field :incomplete-or-varying-evidence] :baseline baseline :candidate candidate :waived? false}]))) @@ -114,8 +115,8 @@ (evidence-differences :effective-graphics (preparation-facts baseline b :effective-graphics) (preparation-facts candidate c :effective-graphics)) (mapcat (fn [metric-id] - (let [bdefs (summary/definitions baseline b metric-id) - cdefs (summary/definitions candidate c metric-id)] + (let [bdefs (map #(dissoc % :description) (summary/definitions baseline b metric-id)) + cdefs (map #(dissoc % :description) (summary/definitions candidate c metric-id))] (evidence-differences [:metric metric-id] (set bdefs) (set cdefs)))) shared))))) @@ -168,14 +169,8 @@ (defn- side-report "Reports one side with saved definitions, setup evidence and sample accounting." - [run id shared analysis allow-all-differences] - {:case-id id - :definition (some #(when (= id (:id %)) %) (get-in run [:plan :cases])) - :preparations (preparations run id) - :remaining (get-in run [:remaining id]) - :failed-attempts (summary/get-failed-attempts run id) - :metrics (into {} (map (fn [metric-id] - [metric-id (summary/summarize-metric run id metric-id analysis allow-all-differences)])) shared)}) + [run id shared analysis] + (summary/case-report run id shared analysis)) (defn- metric-changes "Keeps diagnostic changes unavailable for differing or varying definitions. @@ -209,8 +204,8 @@ {:type ::incompatible :baseline b :candidate c :mismatches mismatches}))) {:explicit? explicit? :mismatches mismatches - :baseline (side-report baseline b shared analysis allow-all-differences) - :candidate (side-report candidate c shared analysis allow-all-differences) + :baseline (side-report baseline b shared analysis) + :candidate (side-report candidate c shared analysis) :metrics (into {} (map (fn [id] [id (metric-changes baseline candidate b c id mismatches diff --git a/frontend/performance/benches/render_wasm/report/format.cljs b/frontend/performance/benches/render_wasm/report/format.cljs index 438d4c305c..e383fb2ea4 100644 --- a/frontend/performance/benches/render_wasm/report/format.cljs +++ b/frontend/performance/benches/render_wasm/report/format.cljs @@ -8,7 +8,7 @@ "Text formatting for run summaries and saved comparisons. Text leads with observed median, bootstrap interval and distinct counts. - Full statistics and acknowledged differences retain units and observation bases." + Full statistics and acknowledged differences include units and phase identity." (:require [benches.render-wasm.result :as result] [clojure.string :as str])) @@ -32,16 +32,25 @@ (defn- counts->text "Prints valid, invalid, absent, failed, unattempted and excluded warmup counts." - [{:keys [valid invalid absent failed unattempted warmup]}] + [{:keys [valid invalid absent failed unattempted warmup invocations raw-values]}] (str "valid=" valid " invalid=" invalid " absent=" absent " failed=" failed - " unattempted=" unattempted " warmup=" warmup)) + " unattempted=" unattempted " warmup=" warmup + " invocations=" invocations " raw-values=" raw-values " n=" valid)) (defn metric->text - "Formats all statistics for one metric, including batch basis and support limits." - [id {:keys [definitions counts statistics]}] - (str (pr-str id) " median " (estimate->text (or (:median statistics) statistics)) + "Formats statistics across phase execution sums with their units and counts. + + Args: + id: vector of phase, command or global scope, and metric type. + report: quantity descriptions, accounting counts, statistics and diagnostics. + + Returns: + Text with observed estimates, support limits and raw-value counts." + [id {:keys [definitions counts statistics diagnostics]}] + (str (pr-str id) " sum per execution; median " (estimate->text (or (:median statistics) statistics)) " CI " (estimate->text (or (:median-ci statistics) statistics)) " | " (counts->text counts) "\n definitions " (pr-str definitions) + (when (seq diagnostics) (str "\n diagnostics " (pr-str diagnostics))) (if-let [reason (:reason statistics)] (str "\n unavailable (" (name reason) ")") (str "\n mean " (estimate->text (:mean statistics)) @@ -52,9 +61,12 @@ (defn- case-status->text "Displays pending slots and failed attempts, including excluded warmups." - [{:keys [remaining failed-attempts]}] + [{:keys [remaining failed-attempts preparations phases]}] (str " remaining " (pr-str remaining) + "\n phases " (pr-str phases) " failed-warmups=" (count (filter :warmup? failed-attempts)) + (when-let [failures (seq (keep :failure preparations))] + (str "\n preparation failures " (pr-str failures))) (when (seq failed-attempts) (str "\n failed attempts " (pr-str failed-attempts))))) diff --git a/frontend/performance/benches/render_wasm/report/summarize.cljs b/frontend/performance/benches/render_wasm/report/summarize.cljs index 44721d475e..f5f5612a3b 100644 --- a/frontend/performance/benches/render_wasm/report/summarize.cljs +++ b/frontend/performance/benches/render_wasm/report/summarize.cljs @@ -5,21 +5,17 @@ ;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns benches.render-wasm.report.summarize - "Reports use saved definitions and explicit metrics from schema:run records. - Warmups and failed attempts supply no observations. A completed attempt can - supply valid, invalid or absent metrics independently. Unattempted slots remain - separate. Batches and render slices never multiply observation counts." + "Offline analysis of raw command histories, grouped within phase executions." (:require - [benches.render-wasm.report.analysis :as analysis] - [benches.render-wasm.result :as result])) + [app.common.schema :as sm] + [benches.render-wasm.quantities :as quantities] + [benches.render-wasm.report.analysis :as analysis])) (def source-coverage - "Limits of compatibility checks on stored inputs and settings." "Recorded inputs and settings checked; runtime-helper identity is unverified. Git SHA and dirty state record provenance.") (def interpretation - "Scope of intervals and of renderer completion measurements." - "Intervals are exploratory and conditional on observed runs. Caches can evolve between attempts; repeat whole runs to investigate findings. Full denotes renderer submission completion. Render-call totals measure synchronous elapsed calls, without exclusive CPU or GPU-completion claims.") + "Intervals are exploratory and conditional on available executions. Incomplete and timed-out cases can select which observations survive. Caches can correlate attempts; repeat whole runs to investigate findings. Full denotes renderer submission completion. Sums combine only one metric type within one phase execution. Finite clock resolution can produce valid zero durations.") (defn case-attempts "Returns chronological attempts for an exact saved case identifier." @@ -31,94 +27,211 @@ [run case-id] (filterv #(= :failed (:outcome %)) (case-attempts run case-id))) -(defn metric-ids - "Discovers dynamic metric identifiers in first-appearance order, including failures." +(defn executions + "Returns actual phase executions with their eligibility and saved identities. + + Args: + run: validated saved run. + case-id: exact saved case identifier. + phase: preparation, main or reset. + + Returns: + Chronological phase histories with phase and preparation/attempt outcomes." + [run case-id phase] + (let [attempts (case-attempts run case-id)] + (if (= :preparation phase) + (let [definition (some #(when (= case-id (:id %)) %) (get-in run [:plan :cases]))] + (mapv (fn [prep] + (let [block (first (:phases prep))] + {:preparation-id (:preparation-id prep) + :events (:events block) + :warmup? (and (not= :reuse (:context definition)) (:warmup? prep)) + :outcome (:outcome block) + :preparation-outcome (if (:failure prep) :failed :completed)})) + (filter #(and (= case-id (:case-id %)) (seq (:phases %))) (:preparations run)))) + (into [] + (keep (fn [attempt] + (when-let [block (some #(when (= phase (:phase %)) %) (:phases attempt))] + (merge (select-keys attempt [:attempt-id :preparation-id :warmup?]) + {:outcome (:outcome block) :attempt-outcome (:outcome attempt) + :events (:events block)})))) + attempts)))) + +(defn phase-counts + "Counts actual phase executions, including empty histories and excluded work. + + Args: + run: validated saved run. + case-id: exact saved case identifier. + + Returns: + Execution, completion, failure and warmup counts for each phase." [run case-id] - (vec (distinct (mapcat (comp keys :metrics) (case-attempts run case-id))))) + (into {} + (map (fn [phase] + (let [blocks (executions run case-id phase)] + [phase {:executions (count blocks) + :completed (count (filter #(= :completed (:outcome %)) blocks)) + :failed (count (filter #(= :failed (:outcome %)) blocks)) + :warmup (count (filter :warmup? blocks))}]))) + [:preparation :main :reset])) + +(defn metric-ids + "Returns [phase command-or-global metric-type] keys in first appearance order. + + Args: + run: validated saved run. + case-id: exact saved case identifier. + + Returns: + Distinct report keys, including quantities recorded during failed work." + [run case-id] + (vec (distinct + (for [phase [:preparation :main :reset] + block (executions run case-id phase) + event (:events block) + metric (keys (:metrics event)) + scope [(:command event) :global]] + [phase scope metric])))) (defn definitions - "Returns every recorded definition for a metric, so drift remains visible." - [run case-id metric-id] - (->> (case-attempts run case-id) - (keep #(get-in % [:metrics metric-id])) - (map result/metric-definition) - (distinct) - (vec))) + "Returns valid saved quantity descriptions for a report key. -(defn observation-definitions - "Returns definitions only for valid, completed observations outside warmups." - [run case-id metric-id] - (->> (case-attempts run case-id) - (remove :warmup?) - (filter #(= :completed (:outcome %))) - (keep #(get-in % [:metrics metric-id])) - (remove :invalid-reason) - (map result/metric-definition) - (distinct) - (vec))) + Args: + run: validated saved run. + _case-id: exact saved case identifier. + id: vector of phase, command or global scope, and metric type. + + Returns: + A vector containing the description, or an empty vector for invalid metadata." + [run _case-id [_phase _scope metric]] + (let [description (get-in run [:quantities metric])] + (if (sm/validate quantities/schema:description description) [description] []))) + +(defn- invalid-reason + [value] + (cond + (not (js/Number.isFinite value)) :non-finite + (neg? value) :negative + :else nil)) + +(defn- execution-value + [block scope metric] + (let [events (->> (:events block) + (map-indexed vector) + (filter (fn [[_ event]] + (and (or (= :global scope) (= scope (:command event))) + (contains? (:metrics event) metric))))) + values (vec (mapcat #(get-in (second %) [:metrics metric]) events)) + invalid (vec (for [[event-index event] events + [value-index value] (map-indexed vector (get-in event [:metrics metric])) + :let [reason (invalid-reason value)] + :when reason] + (merge (select-keys block [:attempt-id :preparation-id]) + {:event-index event-index :value-index value-index + :value value :reason reason}))) + total (when (and (seq values) (empty? invalid)) (reduce + 0 values))] + {:invocations (count events) :raw-values (count values) + :diagnostics (cond-> invalid + (and (some? total) (not (js/Number.isFinite total))) + (conj {:reason :non-finite-sum})) + :value total})) (defn observations - "Returns valid values and distinct counts for a saved case/metric. - :unattempted counts remaining measured repetitions; case :remaining also - retains pending warmups. :warmup counts executed warmup attempts. - Failed measured attempts count as failed even with partial metrics. - Failed warmups count as warmup attempts; case :failed-attempts retains - their failures separately from measured observation counts." - [run case-id metric-id] - (reduce - (fn [out {:keys [warmup? outcome metrics]}] - (let [entry (get metrics metric-id) - kind (cond - warmup? :warmup - (= :failed outcome) :failed - (not (contains? metrics metric-id)) :absent - (:invalid-reason entry) :invalid - :else :valid)] - (cond-> (update-in out [:counts kind] inc) - (= kind :valid) (update :values conj (:value entry))))) - {:values [] - :counts {:valid 0 :invalid 0 :absent 0 :failed 0 - :unattempted (get-in run [:remaining case-id :repetitions]) - :warmup 0}} - (case-attempts run case-id))) + "Sums one metric type per eligible phase execution. + Invocation and raw-value counts cover eligible executions, including invalid + values. Statistical n counts valid execution sums. Failed attempts exclude + both main and reset. Successful preparation survives later attempt failures. + + Args: + run: validated saved run. + case-id: exact saved case identifier. + id: vector of phase, command or global scope, and metric type. + + Returns: + Valid execution sums, accounting counts and invalidity diagnostics." + [run case-id [phase scope metric :as id]] + (let [description (get-in run [:quantities metric]) + diagnostic (when (empty? (definitions run case-id id)) + {:reason (if (contains? (:quantities run) metric) + :malformed-quantity :unknown-quantity) + :metric metric :description description})] + (reduce + (fn [out {:keys [warmup? outcome attempt-outcome preparation-outcome] :as block}] + (let [{:keys [value invocations raw-values diagnostics]} (execution-value block scope metric) + kind (cond + warmup? :warmup + (or (= :failed outcome) (= :failed attempt-outcome) + (= :failed preparation-outcome)) :failed + (zero? invocations) :absent + (or diagnostic (seq diagnostics)) :invalid + :else :valid) + eligible? (not (#{:warmup :failed} kind))] + (cond-> (update-in out [:counts kind] inc) + eligible? (update-in [:counts :invocations] + invocations) + eligible? (update-in [:counts :raw-values] + raw-values) + eligible? (update :diagnostics into diagnostics) + (= kind :valid) (update :values conj value)))) + {:values [] :diagnostics (if diagnostic [diagnostic] []) + :counts {:valid 0 :invalid 0 :absent 0 :failed 0 :warmup 0 + :invocations 0 :raw-values 0 + :unattempted (if (= :preparation phase) 0 + (get-in run [:remaining case-id :repetitions]))}} + (executions run case-id phase)))) (defn summarize-metric - "Computes statistics for one dynamic metric with its definitions and counts. - The optional allow-all-differences flag permits mixed-definition statistics; - scoped comparison permissions never change this summary policy." - ([run case-id metric-id analysis] (summarize-metric run case-id metric-id analysis false)) - ([run case-id metric-id analysis allow-all-differences] - (let [{:keys [values counts]} (observations run case-id metric-id) - definitions (definitions run case-id metric-id) - mixed? (> (count (observation-definitions run case-id metric-id)) 1)] - {:definitions definitions - :counts counts - :mixed-definitions? mixed? - :statistics (if (and mixed? (not allow-all-differences)) - {:reason :inconsistent-metric-definitions :count (count values)} - (analysis/statistics values analysis))}))) + "Computes statistics across execution sums for one phase, scope and metric type. + + Args: + run: validated saved run. + case-id: exact saved case identifier. + id: vector of phase, command or global scope, and metric type. + analysis: resolved statistical settings. + + Returns: + Quantity descriptions, counts, diagnostics and statistics across executions." + [run case-id id analysis] + (let [{:keys [values counts diagnostics]} (observations run case-id id)] + {:definitions (definitions run case-id id) + :statistic :sum :counts counts :diagnostics diagnostics + :statistics (analysis/statistics values analysis)})) + +(defn case-report + "Reports saved case facts and phase quantities using one analysis method. + + Args: + run: validated saved run. + id: exact saved case identifier. + ids: selected report keys. + analysis: resolved statistical settings. + + Returns: + Case definition, raw preparation/failure evidence, counts and metric reports." + [run id ids analysis] + {:case-id id + :definition (some #(when (= id (:id %)) %) (get-in run [:plan :cases])) + :preparations (filterv #(= id (:case-id %)) (:preparations run)) + :phases (phase-counts run id) + :remaining (get-in run [:remaining id]) + :failed-attempts (get-failed-attempts run id) + :metrics (into {} (map (fn [metric-id] + [metric-id (summarize-metric run id metric-id analysis)])) ids)}) (defn summarize - "Builds a portable report from a constructed or decoded schema:run record. - The run must already satisfy schema:run. Reporting uses one - resolved method. - Overrides affect reporting without changing the stored record." + "Reports a validated saved run using saved cases and quantity descriptions. + + Args: + run: validated saved run. + overrides: optional statistical settings overriding the saved defaults. + + Returns: + A report with provenance, termination and separate analyses for each case." ([run] (summarize run nil)) ([run overrides] (let [analysis (analysis/resolve-settings (merge (:analysis run) overrides))] - {:run-id (:run-id run) - :metadata (:metadata run) - :analysis analysis + {:run-id (:run-id run) :metadata (:metadata run) :analysis analysis :termination (:termination run) - :source-coverage source-coverage - :interpretation interpretation - :cases (mapv (fn [{:keys [id] :as definition}] - {:case-id id - :definition definition - :preparations (filterv #(= id (:case-id %)) (:preparations run)) - :remaining (get-in run [:remaining id]) - :failed-attempts (get-failed-attempts run id) - :metrics (into {} (map (fn [metric-id] - [metric-id (summarize-metric run id metric-id analysis)])) - (metric-ids run id))}) + :source-coverage source-coverage :interpretation interpretation + :cases (mapv (fn [{:keys [id]}] + (case-report run id (metric-ids run id) analysis)) (get-in run [:plan :cases]))}))) diff --git a/frontend/performance/benches/render_wasm/result.cljs b/frontend/performance/benches/render_wasm/result.cljs index 49d7e5b512..1e5b118197 100644 --- a/frontend/performance/benches/render_wasm/result.cljs +++ b/frontend/performance/benches/render_wasm/result.cljs @@ -11,33 +11,24 @@ [app.common.transit :as transit] [benches.render-wasm.codec :as codec] [benches.render-wasm.features.plan :as feature-plan] + [benches.render-wasm.quantities :as quantities] [benches.render-wasm.report.analysis :as analysis])) -(def schema:basis - "An attempt total or an average over a declared number of inner calls." - [:or - [:map {:closed true} [:kind [:= :attempt-total]]] - [:map {:closed true} [:kind [:= :per-call-average]] [:calls [:int {:min 1}]]]]) - -(def schema:metric-definition - "The declared meaning and observation basis of a metric. - Authors define units and semantics; the schema accepts any vocabulary. - Comparison checks their equality without interpreting or converting them." +(def schema:event + "One measured command invocation with its ordered raw values." [:map {:closed true} - [:unit :any] - [:semantics :any] - [:basis schema:basis] - [:zero-policy [:enum :positive :allow]]]) + [:command qualified-keyword?] + [:metrics [:map-of qualified-keyword? [:vector {:min 1} number?]]] + [:trace {:optional true} :any]]) -(def schema:metric - "An explicit observation, including raw invalid values and their reason." +(def schema:phase + "Raw command history for one phase execution, including partial failures." [:map {:closed true} - [:value :any] - [:unit :any] - [:semantics :any] - [:basis schema:basis] - [:zero-policy [:enum :positive :allow]] - [:invalid-reason [:maybe keyword?]]]) + [:phase [:enum :preparation :main :reset]] + [:preparation-id some?] + [:outcome [:enum :completed :failed]] + [:events [:vector schema:event]] + [:diagnostics {:optional true} [:vector :any]]]) (def schema:fingerprint "The content digest of the actual validated starting snapshot." @@ -46,24 +37,26 @@ [:digest [:re #"^[0-9a-f]{64}$"]]]) (def schema:preparation - "Untimed setup evidence for one owned scene." + "Setup diagnostics and callback measurements for one owned scene." [:map {:closed true} [:preparation-id some?] [:case-id some?] - [:fingerprint schema:fingerprint] - [:effective-graphics map?] - [:diagnostics map?]]) + [:warmup? boolean?] + [:fingerprint {:optional true} schema:fingerprint] + [:effective-graphics {:optional true} map?] + [:diagnostics map?] + [:phases [:vector schema:phase]] + [:failure {:optional true} map?]]) (def schema:attempt "One chronological execution; failure consumes a slot and may retain evidence." [:map {:closed true} - [:ordinal [:int {:min 0}]] + [:attempt-id [:int {:min 0}]] [:case-id some?] [:preparation-id [:maybe some?]] [:warmup? boolean?] [:outcome [:enum :completed :failed]] - [:metrics [:map-of :any schema:metric]] - [:trace map?] + [:phases [:vector schema:phase]] [:failure {:optional true} map?] [:partial {:optional true} map?]]) @@ -98,6 +91,7 @@ [:timeout-policy [:map [:attempt-ms [:int {:min 1}]]]]]] [:metadata schema:metadata] [:analysis analysis/schema:settings] + [:quantities [:map-of qualified-keyword? :any]] [:preparations [:vector schema:preparation]] [:attempts [:vector schema:attempt]] [:remaining [:map-of :any [:map {:closed true} @@ -132,13 +126,15 @@ [:map {:closed true} [:reason keyword?] [:count [:int {:min 0}]]]]) (def schema:metric-report - "Every metric definition, distinct counts and computed estimates." + "A sum within each phase execution and statistics across eligible executions." [:map {:closed true} - [:definitions [:vector schema:metric-definition]] + [:definitions [:vector quantities/schema:description]] + [:statistic [:= :sum]] [:counts [:map {:closed true} [:valid [:int {:min 0}]] [:invalid [:int {:min 0}]] [:absent [:int {:min 0}]] - [:failed [:int {:min 0}]] [:unattempted [:int {:min 0}]] [:warmup [:int {:min 0}]]]] - [:mixed-definitions? boolean?] + [:failed [:int {:min 0}]] [:unattempted [:int {:min 0}]] [:warmup [:int {:min 0}]] + [:invocations [:int {:min 0}]] [:raw-values [:int {:min 0}]]]] + [:diagnostics [:vector map?]] [:statistics schema:statistics]]) (def schema:comparison-side @@ -146,6 +142,9 @@ [:map {:closed true} [:case-id some?] [:definition [:map [:id some?]]] [:preparations [:vector schema:preparation]] + [:phases [:map-of :keyword [:map {:closed true} + [:executions nat-int?] [:completed nat-int?] + [:failed nat-int?] [:warmup nat-int?]]]] [:remaining [:map {:closed true} [:warmups [:int {:min 0}]] [:repetitions [:int {:min 0}]]]] [:failed-attempts [:vector schema:attempt]] [:metrics [:map-of :any schema:metric-report]]]) @@ -260,33 +259,6 @@ :base base-value :additional additional-value})))) -(defn invalid-reason - "Classifies one raw measurement without changing its value. - Nil denotes an explicit missing value; an absent metric has no entry. - Aggregate durations require positivity unless their entry permits zero." - [{:keys [value zero-policy]}] - (cond - (nil? value) :missing-value - (not (number? value)) :not-numeric - (not (js/Number.isFinite value)) :non-finite - (neg? value) :negative - (and (zero? value) (= zero-policy :positive)) :below-resolution - :else nil)) - -(defn metric - "Validates an explicit metric definition and attaches its invalidity reason. - Authored invalidity reasons survive when the numeric value otherwise qualifies." - [entry] - (let [entry (assoc entry :invalid-reason (or (invalid-reason entry) - (:invalid-reason entry)))] - (require-valid! (sm/validate schema:metric entry) "invalid metric entry" {:metric entry}) - entry)) - -(defn metric-definition - "Returns unit, semantics, basis and zero policy without a measurement value." - [entry] - (select-keys entry [:unit :semantics :basis :zero-policy])) - (defn- initial-remaining "Creates each saved case's fixed budget of unattempted slots." [{:keys [cases warmups repetitions]}] @@ -304,6 +276,23 @@ "measured attempt precedes planned warmups" {:case-id case-id}) (update-in remaining path dec))) +(defn get-unmeasured-cases + "Returns saved case ids whose phase histories contain no measured commands. + + Args: + run: constructed or decoded run record. + + Returns: + A vector of case identifiers in plan order." + [run] + (into [] + (keep (fn [{:keys [id]}] + (when-not (some (comp seq :events) + (mapcat :phases (filter #(= id (:case-id %)) + (concat (:preparations run) (:attempts run))))) + id))) + (get-in run [:plan :cases]))) + (defn- check-completion "Requires all planned attempts to succeed before a run can be completed." [run termination] @@ -311,7 +300,55 @@ (require-valid! (every? #(every? zero? (vals %)) (vals (:remaining run))) "completed run still has unattempted slots" {}) (require-valid! (not-any? #(= :failed (:outcome %)) (:attempts run)) - "completed run has failed attempts" {}))) + "completed run has failed attempts" {}) + (require-valid! (not-any? :failure (:preparations run)) + "completed run has failed preparations" {}) + (require-valid! (empty? (get-unmeasured-cases run)) + "completed case has no measurements" {:case-ids (get-unmeasured-cases run)}))) + +(defn- check-phases + [record allowed] + (let [phases (:phases record)] + (require-valid! (= (mapv :phase phases) + (filterv (set (map :phase phases)) allowed)) + "phase order or identity is invalid" {}) + (doseq [phase phases] + (require-valid! (= (:preparation-id record) (:preparation-id phase)) + "phase references a different preparation" {}) + (doseq [event (:events phase)] + (require-valid! (= 1 (count (get-in event [:metrics :command/elapsed]))) + "command needs one elapsed value" {:command (:command event)}))) + (when-not (or (:failure record) (= :failed (:outcome record))) + (require-valid! (every? #(= :completed (:outcome %)) phases) + "successful execution has a failed phase" {})))) + +(defn- check-preparation + [preparation] + (require-valid! (sm/validate schema:preparation preparation) "invalid preparation" {}) + (check-phases preparation [:preparation]) + (when-not (:failure preparation) + (require-valid! (and (:fingerprint preparation) (:effective-graphics preparation) + (= [:preparation] (mapv :phase (:phases preparation)))) + "completed preparation needs scene, graphics and phase evidence" {}))) + +(defn- check-attempt + [attempt preparation] + (require-valid! (sm/validate schema:attempt attempt) "invalid attempt" {}) + (check-phases attempt [:main :reset]) + (when (:preparation-id attempt) + (require-valid! (= (:case-id attempt) (:case-id preparation)) + "attempt references an unknown or different preparation" {})) + (when (= :completed (:outcome attempt)) + (require-valid! (and preparation (not (:failure preparation)) + (not (:failure attempt)) + (= :main (get-in attempt [:phases 0 :phase]))) + "completed attempt needs successful preparation and main" {})) + (when (seq (:phases attempt)) + (require-valid! (= :main (get-in attempt [:phases 0 :phase])) + "attempt phases must start with main" {}) + (require-valid! (not (:failure preparation)) "attempt phases need successful preparation" {})) + (when (= :failed (get-in attempt [:phases 0 :outcome])) + (require-valid! (= 1 (count (:phases attempt))) "reset follows failed main" {}))) (defn check-run "Validates schema:run and chronological accounting at a saved-data boundary." @@ -326,17 +363,11 @@ (require-valid! (= (count prep-ids) (count (set prep-ids))) "duplicate preparation" {}) (require-valid! (every? #(contains? (set case-ids) %) (map :case-id preps)) "unknown preparation case" {}) (require-valid! (= left (:remaining run)) "remaining slots disagree with attempts" {}) - (doseq [[ordinal attempt] (map-indexed vector (:attempts run))] - (require-valid! (= ordinal (:ordinal attempt)) "attempts are not chronological" {}) - (when-some [id (:preparation-id attempt)] - (require-valid! (= (:case-id attempt) (:case-id (get prep-map id))) - "attempt references an unknown or different preparation" {:preparation-id id})) - (require-valid! (or (= :failed (:outcome attempt)) (some? (:preparation-id attempt))) - "completed attempt needs a preparation" {}) - (doseq [[id entry] (:metrics attempt)] - (require-valid! (= (:invalid-reason entry) - (or (invalid-reason entry) (:invalid-reason entry))) - "metric validity disagrees with raw value" {:metric-id id}))) + (doseq [prep preps] + (check-preparation prep)) + (doseq [[attempt-id attempt] (map-indexed vector (:attempts run))] + (require-valid! (= attempt-id (:attempt-id attempt)) "attempts are not chronological" {}) + (check-attempt attempt (get prep-map (:preparation-id attempt)))) (check-completion run (:termination run))) run) @@ -345,11 +376,12 @@ The plan defaults to 3 warmups, 10 repetitions and a 30000 ms attempt timeout. For each case, all warmup slots precede measured slots. Failed attempts consume their slot too; attempts retain chronological order across cases." - [{:keys [run-id started-at plan metadata analysis]}] + [{:keys [run-id started-at plan metadata analysis quantities]}] (let [plan (merge {:warmups 3 :repetitions 10 :timeout-policy {:attempt-ms 30000}} plan)] (check-run {:type :run-record :run-id run-id :started-at started-at :ended-at nil :plan plan :metadata metadata :analysis (analysis/resolve-settings analysis) + :quantities (merge quantities/descriptions quantities) :preparations [] :attempts [] :remaining (initial-remaining plan) :termination {:reason :running}}))) @@ -363,7 +395,7 @@ The run must come from create-run or decode; earlier evidence is trusted." [run preparation] (check-open! run) - (require-valid! (sm/validate schema:preparation preparation) "invalid preparation" {}) + (check-preparation preparation) (require-valid! (contains? (:remaining run) (:case-id preparation)) "unknown preparation case" {}) (require-valid! (not-any? #(= (:preparation-id preparation) (:preparation-id %)) (:preparations run)) @@ -372,20 +404,15 @@ (defn record-attempt "Checks and appends one new execution, consuming its scheduled slot. - The run must come from create-run or decode, and metric entries from metric. + The run must come from create-run or decode. Missing preparation is allowed for failures before scene setup completes. Earlier attempts are trusted; encode and decode check the complete record." [run attempt] (check-open! run) - (let [attempt (-> (merge {:preparation-id nil :metrics {} :trace {}} attempt) - (assoc :ordinal (count (:attempts run))))] - (require-valid! (sm/validate schema:attempt attempt) "invalid attempt" {}) - (when-some [id (:preparation-id attempt)] - (let [preparation (some #(when (= id (:preparation-id %)) %) (:preparations run))] - (require-valid! (= (:case-id attempt) (:case-id preparation)) - "attempt references an unknown or different preparation" {:preparation-id id}))) - (require-valid! (or (= :failed (:outcome attempt)) (some? (:preparation-id attempt))) - "completed attempt needs a preparation" {}) + (let [attempt (-> (merge {:preparation-id nil :phases []} attempt) + (assoc :attempt-id (count (:attempts run))))] + (check-attempt attempt (some #(when (= (:preparation-id attempt) (:preparation-id %)) %) + (:preparations run))) (-> run (update :attempts conj attempt) (update :remaining consume-slot attempt)))) @@ -407,6 +434,7 @@ warmups and adds `additional`'s measured repetitions. Warmup attempts and preparations used only by them are excluded from `additional`. + The merged record uses the base run's quantity descriptions. The caller supplies the merged run id and may supply its end time." [base additional {:keys [run-id ended-at]}] (let [base (check-run base) @@ -433,6 +461,13 @@ [[:metadata :environment] "runtime environment"] [[:metadata :scored?] "scored flag"]]] (require-compatible! base additional path label)) + (doseq [run [base additional]] + (require-valid! (every? #(sm/validate quantities/schema:description %) + (vals (:quantities run))) + "cannot merge malformed quantity descriptions" {:run-id (:run-id run)})) + (when-not (= (update-vals (:quantities base) #(dissoc % :description)) + (update-vals (:quantities additional) #(dissoc % :description))) + (incompatible! "cannot merge runs with different quantity units" {})) (let [base-attempts (:attempts base) additional-measured (filterv #(not (:warmup? %)) (:attempts additional)) base-preparations (:preparations base) @@ -448,7 +483,7 @@ (let [offset (count base-attempts) attempts (into (vec base-attempts) (map-indexed (fn [index attempt] - (assoc attempt :ordinal (+ offset index))) + (assoc attempt :attempt-id (+ offset index))) additional-measured)) plan (assoc (:plan base) :repetitions (+ (get-in base [:plan :repetitions]) @@ -476,6 +511,7 @@ :plan plan :metadata metadata :analysis (:analysis base) + :quantities (:quantities base) :preparations (into (vec base-preparations) additional-preparations) :attempts attempts :remaining remaining diff --git a/frontend/test/performance_tests/render_wasm/cli_integration.cjs b/frontend/test/performance_tests/render_wasm/cli_integration.cjs index df0d9ff4f8..82e47e70e6 100644 --- a/frontend/test/performance_tests/render_wasm/cli_integration.cjs +++ b/frontend/test/performance_tests/render_wasm/cli_integration.cjs @@ -458,36 +458,35 @@ function runFixtureText({ }) { const k = transit.keyword; const map = (pairs) => transit.map(pairs); - const metric = map([ - k("value"), - value, - k("unit"), - k("ms"), - k("semantics"), - k("elapsed"), - k("basis"), - map([k("kind"), k("attempt-total")]), - k("zero-policy"), - k("positive"), - k("invalid-reason"), - null, + const event = map([ + k("command"), k("fixture/work"), + k("metrics"), map([k("command/elapsed"), [value]]), ]); const preparation = map([ k("preparation-id"), `${runId}-preparation`, k("case-id"), k(caseId), + k("warmup?"), + false, k("fingerprint"), map([k("algorithm"), k("sha-256"), k("digest"), fingerprint]), k("effective-graphics"), map([k("webgl2"), true]), k("diagnostics"), map([]), + k("phases"), + [map([ + k("phase"), k("preparation"), + k("preparation-id"), `${runId}-preparation`, + k("outcome"), k("completed"), + k("events"), [], + ])], ]); - const makeAttempt = (ordinal, warmup) => + const makeAttempt = (attemptId, warmup) => map([ - k("ordinal"), - ordinal, + k("attempt-id"), + attemptId, k("case-id"), k(caseId), k("preparation-id"), @@ -496,10 +495,13 @@ function runFixtureText({ warmup, k("outcome"), k("completed"), - k("metrics"), - map([k("elapsed"), metric]), - k("trace"), - map([]), + k("phases"), + [map([ + k("phase"), k("main"), + k("preparation-id"), `${runId}-preparation`, + k("outcome"), k("completed"), + k("events"), [event], + ])], ]); const attempts = [ ...Array.from({ length: warmups }, (_, index) => makeAttempt(index, true)), @@ -583,6 +585,8 @@ function runFixtureText({ k("min-p99"), 500, ]), + k("quantities"), + map([k("command/elapsed"), map([k("unit"), k("ms")])]), k("preparations"), [preparation], k("attempts"), @@ -1762,7 +1766,7 @@ test( assert.equal(plan.get(k("warmups")), 1); assert.equal(plan.get(k("repetitions")), 5); assert.deepEqual( - attempts.map((attempt) => attempt.get(k("ordinal"))), + attempts.map((attempt) => attempt.get(k("attempt-id"))), [0, 1, 2, 3, 4, 5], ); assert.deepEqual( diff --git a/frontend/test/performance_tests/render_wasm/compare_test.cljs b/frontend/test/performance_tests/render_wasm/compare_test.cljs index f67148a2f1..78af5e74e6 100644 --- a/frontend/test/performance_tests/render_wasm/compare_test.cljs +++ b/frontend/test/performance_tests/render_wasm/compare_test.cljs @@ -28,11 +28,7 @@ (defn- metric [value] - (result/metric {:value value - :unit :ms - :semantics :elapsed - :basis {:kind :attempt-total} - :zero-policy :positive})) + {:command :fixture/work :metrics {:command/elapsed [value]}}) (defn- case-definition [case-id scene] @@ -68,19 +64,21 @@ run (result/record-preparation run {:preparation-id preparation-id - :case-id case-id + :case-id case-id :warmup? false :fingerprint {:algorithm :sha-256 :digest fingerprint} :effective-graphics {:webgl2 true} - :diagnostics {}}) + :diagnostics {} + :phases [{:phase :preparation :preparation-id preparation-id + :outcome :completed :events []}]}) run (result/record-attempt run {:case-id case-id :preparation-id preparation-id :warmup? false :outcome :completed - :metrics {:elapsed (metric value)} - :trace {}})] + :phases [{:phase :main :preparation-id preparation-id + :outcome :completed :events [(metric value)]}]})] (result/finish-run run "2026-10-26T00:01:00.000Z" {:reason :completed}))) (defn- write-run @@ -111,7 +109,7 @@ :allow-all-differences false} (:permissions comparison))) (t/is (empty? (get-in comparison [:pairs 0 :mismatches]))) - (t/is (= 2 (get-in comparison [:pairs 0 :metrics :elapsed :absolute :value]))) + (t/is (= 2 (get-in comparison [:pairs 0 :metrics [:main :fixture/work :command/elapsed] :absolute :value]))) (t/is (str/includes? text "Baseline")) (t/is (str/includes? text "Candidate")) (t/is (str/includes? text "case/load")) diff --git a/frontend/test/performance_tests/render_wasm/feature_comparison_test.cljs b/frontend/test/performance_tests/render_wasm/feature_comparison_test.cljs index 806642110a..33f57856d5 100644 --- a/frontend/test/performance_tests/render_wasm/feature_comparison_test.cljs +++ b/frontend/test/performance_tests/render_wasm/feature_comparison_test.cljs @@ -447,11 +447,7 @@ (defn- metric-entry [value] - (result/metric {:value value - :unit :ms - :semantics :duration - :basis {:kind :attempt-total} - :zero-policy :positive})) + {:command :fixture/work :metrics {:command/elapsed [value]}}) (defn- fingerprint [character] @@ -460,11 +456,12 @@ (defn- preparation [case-id digest] - {:preparation-id (str (random-uuid)) - :case-id case-id - :fingerprint (fingerprint digest) - :effective-graphics {:backend :fixture} - :diagnostics {}}) + (let [id (str (random-uuid))] + {:preparation-id id :case-id case-id :warmup? false + :fingerprint (fingerprint digest) + :effective-graphics {:backend :fixture} + :diagnostics {} + :phases [{:phase :preparation :preparation-id id :outcome :completed :events []}]})) (defn- fake-run-record [selected-build run-home cases options {:keys [kind digest metric-value] @@ -494,8 +491,7 @@ :preparation-id nil :warmup? false :outcome :failed - :metrics {} - :trace {} + :phases [] :failure {:message "fixture failure"}}) (let [prep (preparation case-id digest)] (-> run @@ -505,8 +501,8 @@ :preparation-id (:preparation-id prep) :warmup? false :outcome :completed - :metrics {:fixture/total-ms (metric-entry metric-value)} - :trace {}})))))) + :phases [{:phase :main :preparation-id (:preparation-id prep) + :outcome :completed :events [(metric-entry metric-value)]}]})))))) run cases)) termination (case kind diff --git a/frontend/test/performance_tests/render_wasm/merge_test.cljs b/frontend/test/performance_tests/render_wasm/merge_test.cljs index 3e1f97ad8a..87d912c30b 100644 --- a/frontend/test/performance_tests/render_wasm/merge_test.cljs +++ b/frontend/test/performance_tests/render_wasm/merge_test.cljs @@ -6,16 +6,13 @@ (ns performance-tests.render-wasm.merge-test (:require + [benches.render-wasm.report.summarize :as summary] [benches.render-wasm.result :as result] [cljs.test :as t :include-macros true])) (defn- metric [value] - (result/metric {:value value - :unit :ms - :semantics :elapsed - :basis {:kind :attempt-total} - :zero-policy :positive})) + {:command :fixture/work :metrics {:command/elapsed [value]}}) (defn- case-definition [case-id] @@ -33,11 +30,12 @@ (defn- preparation [preparation-id case-id] {:preparation-id preparation-id - :case-id case-id + :case-id case-id :warmup? false :fingerprint {:algorithm :sha-256 :digest (apply str (repeat 64 "a"))} :effective-graphics {:webgl2 true} - :diagnostics {}}) + :diagnostics {} + :phases [{:phase :preparation :preparation-id preparation-id :outcome :completed :events []}]}) (defn- attempt [case-id preparation-id warmup?] @@ -45,8 +43,8 @@ :preparation-id preparation-id :warmup? warmup? :outcome :completed - :metrics {:elapsed (metric 1.0)} - :trace {}}) + :phases [{:phase :main :preparation-id preparation-id + :outcome :completed :events [(metric 1.0)]}]}) (defn- make-run [{:keys [run-id cases warmups repetitions git-sha build environment analysis @@ -110,7 +108,7 @@ (t/is (= 1 (get-in merged [:plan :warmups]))) (t/is (= 5 (get-in merged [:plan :repetitions]))) (t/is (= 12 (count (:attempts merged)))) - (t/is (= (range 12) (map :ordinal (:attempts merged)))) + (t/is (= (range 12) (map :attempt-id (:attempts merged)))) (t/is (= [true false false true false false false false false false false false] (map :warmup? (:attempts merged)))) (t/is (every? #(every? zero? (vals %)) (vals (:remaining merged)))) @@ -200,3 +198,21 @@ (t/is (thrown-with-msg? js/Error #"duplicate preparation ids" (result/merge-runs base additional {:run-id "merged"}))))) + +(t/deftest quantity-prose-does-not-change-merge-compatibility + (let [base (make-run {:run-id "base" :cases [:case/load] :warmups 0 :repetitions 1}) + additional (-> (make-run {:run-id "additional" :cases [:case/load] :warmups 0 :repetitions 1}) + (assoc-in [:quantities :command/elapsed :description] "Elapsed command work.")) + merged (result/merge-runs base additional {:run-id "merged"})] + (t/is (= 2 (get-in merged [:plan :repetitions]))) + (t/is (= merged (result/decode (result/encode merged)))) + (let [malformed (assoc-in additional [:quantities :command/elapsed :description] 42) + report (summary/summarize malformed)] + (t/is (= :malformed-quantity + (get-in report [:cases 0 :metrics [:main :fixture/work :command/elapsed] :diagnostics 0 :reason]))) + (t/is (thrown-with-msg? js/Error #"malformed quantity descriptions" + (result/merge-runs base malformed {:run-id "invalid"})))) + (t/is (thrown-with-msg? js/Error #"different quantity units" + (result/merge-runs base + (assoc-in additional [:quantities :command/elapsed :unit] :seconds) + {:run-id "invalid"}))))) diff --git a/frontend/test/performance_tests/render_wasm/report_test.cljs b/frontend/test/performance_tests/render_wasm/report_test.cljs index 8d8ce5e4c9..dc5887a5c2 100644 --- a/frontend/test/performance_tests/render_wasm/report_test.cljs +++ b/frontend/test/performance_tests/render_wasm/report_test.cljs @@ -5,8 +5,6 @@ ;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns performance-tests.render-wasm.report-test - "Pure tests for raw records, exploratory statistics and saved-run comparisons. - Fixtures use saved case definitions without importing a registry or renderer." (:require [app.common.transit :as transit] [benches.render-wasm.measurement :as measurement] @@ -18,73 +16,62 @@ [cljs.test :as t :include-macros true] [clojure.string :as str])) -(def metric-id [:custom/latency "arbitrary metric"]) - (def saved-case - "A saved definition that requires no current registration." - {:id :saved/pan :scene :saved :scene-seed 42 - :params {:count 10} :view {:scale 1 :x 0 :y 0} - :operation {:steps 3} :context :reuse :completion :render-full - :batch-size 1 :preparation {}}) + {:id :saved/pan :scene :saved :scene-seed 42 :params {:count 10} + :view {:scale 1 :x 0 :y 0} :context :reuse :preparation {}}) (def metadata - "Functional build facts are separate from ephemeral build provenance." {:git {:sha "abc" :dirty false} - :build {:functional {:mode :release :features #{:default} :env {"PROFILE" "0"}} - :provenance {:run-marker "first"}} - :environment {:browser "Chromium 1" :node "v22" :playwright "1" :os "Linux" :cpu "CPU"} - :scored? true}) + :build {:functional {:mode :release :features []}} + :environment {:browser "Chromium" :node "v24"} :scored? true}) -(def preparation - "One content identity and its effective graphics." - {:preparation-id :prep/one :case-id :saved/pan +(defn- event + [command values] + {:command command :metrics {:command/elapsed values}}) + +(defn- block + [phase prep events] + {:phase phase :preparation-id prep :outcome :completed :events events}) + +(defn- preparation + [id events] + {:preparation-id id :case-id :saved/pan :warmup? false :fingerprint {:algorithm :sha-256 :digest (apply str (repeat 64 "a"))} - :effective-graphics {:vendor "vendor" :renderer "hardware" :software false :dpr 2} - :diagnostics {:upload-ms 123}}) + :effective-graphics {:renderer "hardware"} + :diagnostics {:upload-ms 123} + :phases [(block :preparation id events)]}) -(defn- entry - "Declares a total duration; overrides supply explicit alternative definitions." - ([value] (entry value {})) - ([value overrides] - (result/metric (merge {:value value :unit :ms :semantics :duration - :basis {:kind :attempt-total} :zero-policy :positive} overrides)))) +(defn- new-run + [warmups repetitions context] + (result/create-run {:run-id "run" :started-at "start" + :plan {:cases [(assoc saved-case :context context)] + :warmups warmups :repetitions repetitions} + :metadata metadata :analysis {:draws 50}})) -(defn- error-data - "Returns boundary error data, allowing tests to assert the refusal reason." +(defn- add-attempt + [run prep warmup? outcome phases] + (result/record-attempt + run (measurement/attempt :saved/pan prep warmup? + (cond-> {:status (if (= :completed outcome) "ok" "failed") + :phases phases} + (= :failed outcome) (assoc :message "renderer failed"))))) + +(defn- sample-run + [values] + (reduce (fn [run value] + (add-attempt run :prep false :completed + [(block :main :prep [(event :test/pan [value])])])) + (result/record-preparation (new-run 0 (count values) :reuse) + (preparation :prep [])) values)) + +(defn- error-type [f] - (try (f) nil (catch :default cause (ex-data cause)))) + (try (f) nil (catch :default cause (:type (ex-data cause))))) -(defn- run-with - "Creates a raw fixture with warmups and a fixed number of measured slots. - :absent and :failed retain separate outcomes. Optional overrides change facts." - ([values] (run-with values {})) - ([values {:keys [definition prep facts analysis warmups repetitions] - :or {definition saved-case prep preparation facts metadata warmups 0}}] - (let [id (:id definition) - prep (assoc prep :case-id id) - run (-> (result/create-run {:run-id "run" :started-at "start" - :plan {:cases [definition] :warmups warmups - :repetitions (or repetitions (count values))} - :metadata facts :analysis analysis}) - (result/record-preparation prep)) - add (fn [run value warmup?] - (result/record-attempt - run {:case-id id :preparation-id (:preparation-id prep) :warmup? warmup? - :outcome (if (= value :failed) :failed :completed) - :metrics (if (#{:absent :failed} value) {} {metric-id (entry value)}) - :trace {}}))] - (reduce #(add %1 %2 false) - (reduce #(add %1 %2 true) run (repeat warmups 1000)) values)))) - -(defn- run-with-features - "Adds independent effective-feature evidence and a saved requested selection." - [values features selection] - (run-with values {:facts (-> metadata - (assoc-in [:build :functional :features] features) - (assoc-in [:build :provenance :feature-selection] selection))})) +(def main-pan [:main :test/pan :command/elapsed]) +(def main-global [:main :global :command/elapsed]) (defn- approximately? - "Compares computed numbers with a tolerance for floating point arithmetic." [a b] (< (js/Math.abs (- a b)) 1e-9)) @@ -105,20 +92,18 @@ (t/deftest numeric-arguments-throw-while-unavailable-results-retain-reasons (doseq [probability [-0.1 1.1 js/NaN js/Infinity js/-Infinity nil false "0.5"]] (t/is (= ::method/invalid-probability - (:type (error-data #(method/quantile [1 2] probability)))))) + (error-type #(method/quantile [1 2] probability))))) (doseq [values [[js/NaN] [js/Infinity] [js/-Infinity] [nil] ["1"] 42 {} #{1 2}]] - (t/is (= ::method/invalid-observations - (:type (error-data #(method/quantile values 0.5)))))) + (t/is (= ::method/invalid-observations (error-type #(method/quantile values 0.5))))) + (doseq [override [{:draws 0} {:seed -1} {:seed 4294967296} {:confidence 1} + {:confidence js/NaN} {:min-ci 0} {:unknown 1}]] + (t/is (= ::method/invalid-settings (error-type #(method/resolve-settings override))))) (t/is (nil? (method/quantile [] 0.5))) (t/is (nil? (method/quantile nil 0.5))) - (t/is (= ::method/invalid-settings - (:type (error-data #(method/resolve-settings {:draws 0}))))) - (t/is (= {:reason :no-valid-observations} (:median (method/statistics [] method/defaults)))) - (t/is (= {:reason :insufficient-observations :minimum 10} - (:median-ci (method/statistics [1] method/defaults))))) + (t/is (= {:reason :no-valid-observations} (:median (method/statistics [] method/defaults))))) (t/deftest calculated-numbers-and-overflow-have-independent-result-maps - (let [stats (method/statistics [4] method/defaults) + (let [stats (method/statistics [4] method/defaults) changes (method/changes [1e-308] [1e308] (method/resolve-settings {:draws 5 :min-ci 1}))] (t/is (= {:value 4} (:median stats))) (t/is (= {:value 0} (:mad stats))) @@ -140,20 +125,10 @@ (t/is (= :insufficient-observations (get-in (method/statistics (vec (repeat 499 7)) settings) [:p99 :reason]))) (t/is (= {:value 7} (:p99 (method/statistics (vec (repeat 500 7)) settings)))))) -(t/deftest analysis-settings-are-validated-and-recorded - (doseq [override [{:draws 0} {:seed -1} {:seed 4294967296} {:confidence 1} - {:confidence js/NaN} {:min-ci 0} {:unknown 1}]] - (t/is (= ::method/invalid-settings (:type (error-data #(method/resolve-settings override)))))) - (t/is (= 0 (:seed (method/resolve-settings {:seed 0})))) - (t/is (= method/defaults (:analysis (run-with [1]))))) - (t/deftest deterministic-bootstrap-and-independent-difference-vectors - ;; Bounds also come from an independent Python Mulberry32 implementation. (let [settings (method/resolve-settings {:draws 100 :seed 123 :confidence 0.8}) - base (vec (range 1 11)) - stats (method/statistics base settings) - changes (method/changes base (mapv #(+ 5 %) base) settings)] - (t/is (= stats (method/statistics base settings))) + stats (method/statistics (vec (range 1 11)) settings) + changes (method/changes (vec (range 1 11)) (vec (range 6 16)) settings)] (t/is (= 5.5 (get-in stats [:median :value]))) (t/is (= {:lower 3.5 :upper 7} (:median-ci stats))) (t/is (= 5 (get-in changes [:absolute :value]))) @@ -161,7 +136,7 @@ (t/is (= 8 (get-in changes [:absolute-ci :upper]))) (t/is (approximately? 31.201923076923077 (get-in changes [:percentage-ci :lower]))) (t/is (approximately? 215.35714285714295 (get-in changes [:percentage-ci :upper]))) - (t/is (not= stats (method/statistics base (assoc settings :seed 0)))))) + (t/is (= stats (method/statistics (vec (range 1 11)) settings))))) (t/deftest zero-baseline-percentages-and-undefined-draws (let [settings (method/resolve-settings {:draws 100 :min-ci 1 :seed 0})] @@ -170,896 +145,289 @@ (t/is (= :zero-bootstrap-baseline (get-in (method/changes [0 1] [2 3] settings) [:percentage-ci :reason]))) (t/is (= :no-valid-observations (get-in (method/changes [] [1] settings) [:absolute :reason]))))) -(t/deftest partial-chronology-and-distinct-counts - (let [run (run-with [1 :failed] {:repetitions 3 :warmups 1}) - ended (result/finish-run run "end" {:reason :interrupted}) - observations (summary/observations ended :saved/pan metric-id)] - (t/is (= [0 1 2] (mapv :ordinal (:attempts ended)))) - (t/is (= {:warmups 0 :repetitions 1} (get-in ended [:remaining :saved/pan]))) - (t/is (= {:valid 1 :invalid 0 :absent 0 :failed 1 :unattempted 1 :warmup 1} (:counts observations))) - (t/is (= [1] (:values observations))) - (t/is (= ::result/invalid-record (:type (error-data #(result/finish-run run "end" {:reason :completed}))))) - (t/is (= ::result/invalid-record (:type (error-data #(result/record-attempt ended {}))))))) +(t/deftest repeated-commands-keep-invocations-values-and-statistical-n-distinct + (let [phases [(block :main :prep + [(assoc-in (event :test/pan [2]) [:metrics :renderer/call-elapsed] [1 2]) + (assoc-in (event :test/zoom [4]) [:metrics :renderer/call-elapsed] [4]) + (assoc-in (event :test/pan [3]) [:metrics :renderer/call-elapsed] [3])])] + run (-> (new-run 1 2 :reuse) + (result/record-preparation (preparation :prep [])) + (add-attempt :prep true :completed phases) + (add-attempt :prep false :completed phases) + (add-attempt :prep false :completed phases) + (result/finish-run "end" {:reason :completed}) + result/encode result/decode) + report (first (:cases (summary/summarize run))) + pan (get-in report [:metrics [:main :test/pan :renderer/call-elapsed]]) + global (get-in report [:metrics [:main :global :renderer/call-elapsed]])] + (t/is (= 3 (count (get-in run [:attempts 1 :phases 0 :events])))) + (t/is (= {:valid 2 :invalid 0 :absent 0 :failed 0 :warmup 1 :unattempted 0 + :invocations 4 :raw-values 6} (:counts pan))) + (t/is (= 2 (get-in pan [:statistics :count]))) + (t/is (= 6 (get-in pan [:statistics :median :value]))) + (t/is (= 10 (get-in global [:statistics :median :value]))) + (t/is (= 6 (get-in global [:counts :invocations]))) + (t/is (= 8 (get-in global [:counts :raw-values]))) + (t/is (str/includes? (format/format-run (summary/summarize run)) "invocations=4 raw-values=6 n=2")))) -(t/deftest completed-runs-reject-failed-warmups-and-measured-attempts - (doseq [failed-warmup? [true false]] - (let [run (-> (result/create-run {:run-id "failed" :started-at "start" - :plan {:cases [saved-case] :warmups 1 :repetitions 1} - :metadata metadata}) - (result/record-preparation preparation) - (result/record-attempt {:case-id :saved/pan :preparation-id :prep/one - :warmup? true - :outcome (if failed-warmup? :failed :completed) - :metrics {metric-id (entry 1000)} :trace {}}) - (result/record-attempt {:case-id :saved/pan :preparation-id :prep/one - :warmup? false - :outcome (if failed-warmup? :completed :failed) - :metrics {metric-id (entry 2)} :trace {}})) - forged (assoc run :ended-at "end" :termination {:reason :completed}) - failed (result/finish-run run "end" {:reason :failed})] - (t/is (= {:warmups 0 :repetitions 0} (get-in run [:remaining :saved/pan]))) - (t/is (= ::result/invalid-record - (:type (error-data #(result/finish-run run "end" {:reason :completed}))))) - (t/is (= ::result/invalid-record (:type (error-data #(result/check-run forged))))) - (t/is (= ::result/invalid-record - (:type (error-data #(result/decode (transit/encode-str forged)))))) - (t/is (= failed (-> failed result/encode result/decode))))) - (let [completed (result/finish-run (run-with [2] {:warmups 1}) "end" {:reason :completed})] - (t/is (= completed (-> completed result/encode result/decode))))) +(t/deftest phase-eligibility-keeps-preparation-after-reset-failure + (let [run (-> (new-run 1 3 :reuse) + (result/record-preparation (preparation :prep [(event :test/setup [8])])) + (add-attempt :prep true :completed [(block :main :prep []) (block :reset :prep [])]) + (add-attempt :prep false :completed + [(block :main :prep [(event :test/pan [2])]) + (block :reset :prep [(event :test/restore [3])])]) + (add-attempt :prep false :failed + [(block :main :prep [(event :test/pan [100])]) + (assoc (block :reset :prep [(event :test/restore [200])]) :outcome :failed)]) + (result/finish-run "end" {:reason :failed}) + result/encode result/decode) + report (first (:cases (summary/summarize run))) + prep (get-in report [:metrics [:preparation :test/setup :command/elapsed]])] + (t/is (= [0 1 2] (mapv :attempt-id (:attempts run)))) + (t/is (= {:warmups 0 :repetitions 1} (:remaining report))) + (t/is (= 8 (get-in prep [:statistics :median :value]))) + (t/is (= 1 (get-in prep [:statistics :count]))) + (t/is (= :insufficient-observations (get-in prep [:statistics :median-ci :reason]))) + (t/is (= 0 (get-in prep [:counts :warmup]))) + (t/is (= [2] (:values (summary/observations run :saved/pan main-pan)))) + (t/is (= [3] (:values (summary/observations run :saved/pan [:reset :test/restore :command/elapsed])))) + (t/is (= {:executions 3 :completed 3 :failed 0 :warmup 1} (get-in report [:phases :main]))) + (t/is (= {:executions 3 :completed 2 :failed 1 :warmup 1} (get-in report [:phases :reset]))))) -(t/deftest reports-show-failed-warmups-without-changing-measured-counts - (let [failed {:case-id :saved/pan :preparation-id :prep/one :warmup? true - :outcome :failed :metrics {metric-id (entry 1000)} - :trace {:render-slices [3]} - :failure {:stage :warmup :message "warmup render failed"} - :partial {:phase :render-full}} - base (-> (run-with [1 2] {:warmups 1}) - (result/finish-run "end" {:reason :completed}) - result/encode result/decode) - other (-> (result/create-run {:run-id "candidate" :started-at "start" - :plan {:cases [saved-case] :warmups 1 :repetitions 2} - :metadata metadata}) - (result/record-preparation preparation) - (result/record-attempt failed) - (result/record-attempt {:case-id :saved/pan :preparation-id :prep/one - :warmup? false :outcome :completed - :metrics {metric-id (entry 2)} :trace {}}) - (result/record-attempt {:case-id :saved/pan :preparation-id :prep/one - :warmup? false :outcome :completed - :metrics {metric-id (entry 4)} :trace {}}) - (result/finish-run "end" {:reason :failed - :failure {:message "warmup render failed"}}) - result/encode result/decode) - report (summary/summarize other) - comparison (-> (compare/compare-runs base other) result/encode result/decode) - counts {:valid 2 :invalid 0 :absent 0 :failed 0 :unattempted 0 :warmup 1} - evidence [(assoc failed :ordinal 0)] - text (format/format-comparison comparison)] - (t/is (= evidence (get-in report [:cases 0 :failed-attempts]))) - (t/is (= evidence (get-in comparison [:pairs 0 :candidate :failed-attempts]))) - (t/is (= counts (get-in report [:cases 0 :metrics metric-id :counts]))) - (t/is (= counts (get-in comparison [:pairs 0 :candidate :metrics metric-id :counts]))) - (t/is (= 3 (get-in comparison [:pairs 0 :candidate :metrics metric-id :statistics :median :value]))) - (t/is (str/includes? text "Baseline termination {:reason :completed}")) - (t/is (str/includes? text "Candidate termination {:reason :failed")) - (doseq [output [text (format/format-run report)]] - (t/is (str/includes? output "failed-warmups=1")) - (t/is (str/includes? output "warmup render failed")) - (t/is (str/includes? output ":phase :render-full")) - (t/is (str/includes? output ":render-slices [3]"))))) +(t/deftest fresh-preparations-follow-their-own-slot-eligibility + (let [run (-> (new-run 1 2 :fresh) + (result/record-preparation (assoc (preparation :warm [(event :test/setup [900])]) :warmup? true)) + (add-attempt :warm true :completed [(block :main :warm [])]) + (result/record-preparation (preparation :first [(event :test/setup [5])])) + (add-attempt :first false :completed [(block :main :first [])]) + (result/record-preparation (preparation :second [(event :test/setup [7])])) + (add-attempt :second false :failed [(assoc (block :main :second []) :outcome :failed)])) + obs (summary/observations run :saved/pan [:preparation :test/setup :command/elapsed])] + (t/is (= [5 7] (:values obs))) + (t/is (= 1 (get-in obs [:counts :warmup]))) + (t/is (= 2 (get-in obs [:counts :valid]))))) -(t/deftest comparisons-display-running-interrupted-and-preparation-failure-status - (let [base (-> (run-with [1] {:repetitions 2}) result/encode result/decode)] - (doseq [termination [{:reason :running} - {:reason :interrupted :signal "SIGINT"} - {:reason :preparation-error :failure {:message "cannot prepare renderer"}}]] - (let [other (if (= :running (:reason termination)) - base - (result/finish-run base "end" termination)) - comparison (-> (compare/compare-runs other other) result/encode result/decode) - text (format/format-comparison comparison)] - (t/is (str/includes? text (str "Baseline termination " (pr-str termination)))) - (t/is (str/includes? text (str "Candidate termination " (pr-str termination)))) - (t/is (str/includes? text "remaining {:warmups 0, :repetitions 1}")))) - (let [empty (result/create-run {:run-id "empty" :started-at "start" - :plan {:cases [saved-case] :warmups 0 :repetitions 2} - :metadata metadata}) - stopped (-> empty - (result/finish-run "end" {:reason :preparation-error - :failure {:message "cannot prepare renderer"}}) - result/encode result/decode) - report (summary/summarize stopped) - text (format/format-run report)] - (t/is (str/includes? text ":reason :preparation-error")) - (t/is (str/includes? text "cannot prepare renderer")) - (t/is (str/includes? text "remaining {:warmups 0, :repetitions 2}")) - (t/is (empty? (get-in report [:cases 0 :metrics])))))) +(t/deftest fresh-warmup-preparation-is-excluded-before-main-delivery + (let [run (-> (new-run 1 1 :fresh) + (result/record-preparation + (assoc (preparation :prep [(event :test/setup [9])]) :warmup? true)) + result/encode result/decode) + report (get-in (summary/summarize run) + [:cases 0 :metrics [:preparation :test/setup :command/elapsed]])] + (t/is (= 1 (get-in report [:counts :warmup]))) + (t/is (= :no-valid-observations (get-in report [:statistics :median :reason]))))) -(t/deftest invalid-missing-and-failed-evidence - (let [run (run-with [1 js/NaN js/Infinity js/-Infinity 0 nil :absent :failed]) - obs (summary/observations run :saved/pan metric-id)] - (t/is (= {:valid 1 :invalid 5 :absent 1 :failed 1 :unattempted 0 :warmup 0} (:counts obs))) - (t/is (= [1] (:values obs))) - (t/is (= :below-resolution (:invalid-reason (entry 0)))) - (t/is (nil? (:invalid-reason (entry 0 {:zero-policy :allow})))) - (t/is (= :negative (:invalid-reason (entry -1 {:zero-policy :allow})))) - (t/is (= :not-numeric (:invalid-reason (entry "bad")))) - (t/is (= :missing-value (:invalid-reason (entry nil)))))) +(t/deftest failed-preparation-excludes-a-completed-callback-from-observations + (let [run (-> (new-run 0 1 :reuse) + (result/record-preparation + (assoc (preparation :prep [(event :test/setup [9])]) + :failure {:message "initial Full failed"}))) + report (first (:cases (summary/summarize run))) + metric (get-in report [:metrics [:preparation :test/setup :command/elapsed]])] + (t/is (= {:executions 1 :completed 1 :failed 0 :warmup 0} (get-in report [:phases :preparation]))) + (t/is (= 1 (get-in metric [:counts :failed]))) + (t/is (= :no-valid-observations (get-in metric [:statistics :median :reason]))))) -(t/deftest raw-transit-roundtrip-preserves-dynamic-identities-and-special-values - (let [run (-> (run-with [js/NaN js/Infinity js/-Infinity nil] {:repetitions 5}) - (result/record-attempt {:case-id :saved/pan :warmup? false :outcome :failed - :metrics {metric-id (entry 900)} - :trace {:slices [{:duration-ms 0}]} - :failure {:cause [{:message "failure"}]} - :partial {:raw js/NaN}})) - decoded (result/decode (result/encode run))] - (t/is (js/Number.isNaN (get-in decoded [:attempts 0 :metrics metric-id :value]))) - (t/is (= js/Infinity (get-in decoded [:attempts 1 :metrics metric-id :value]))) - (t/is (= js/-Infinity (get-in decoded [:attempts 2 :metrics metric-id :value]))) - (t/is (nil? (get-in decoded [:attempts 3 :metrics metric-id :value]))) - (t/is (= {:slices [{:duration-ms 0}]} (get-in decoded [:attempts 4 :trace]))) - (t/is (js/Number.isNaN (get-in decoded [:attempts 4 :partial :raw]))) - (t/is (= {:cause [{:message "failure"}]} (get-in decoded [:attempts 4 :failure]))) - (t/is (= 1 (get-in (summary/observations decoded :saved/pan metric-id) [:counts :failed]))))) +(t/deftest invalid-members-propagate-only-to-dependent-quantities + (let [bad (assoc-in (event :test/pan [0]) [:metrics :renderer/call-elapsed] [1 js/NaN]) + good (assoc-in (event :test/zoom [0]) [:metrics :renderer/call-elapsed] [2]) + run (-> (new-run 0 1 :reuse) + (result/record-preparation (preparation :prep [])) + (add-attempt :prep false :completed [(block :main :prep [bad good])])) + report (first (:cases (summary/summarize run)))] + (t/is (= [0] (:values (summary/observations run :saved/pan main-global)))) + (t/is (= [2] (:values (summary/observations run :saved/pan [:main :test/zoom :renderer/call-elapsed])))) + (doseq [scope [:test/pan :global]] + (let [metric (get-in report [:metrics [:main scope :renderer/call-elapsed]])] + (t/is (= 1 (get-in metric [:counts :invalid]))) + (t/is (= :non-finite (get-in metric [:diagnostics 0 :reason]))) + (t/is (= :no-valid-observations (get-in metric [:statistics :median :reason]))))))) -(t/deftest malformed-records-and-poisoned-accounting-refuse - (let [run (run-with [1])] - (t/is (= ::result/unsupported-format - (:type (error-data #(result/decode (transit/encode-str (assoc run :type :unknown))))))) - (doseq [bad [(dissoc run :plan) - (assoc run :attempts {}) - (assoc-in run [:analysis :draws] 0) - (assoc-in run [:preparations 0 :fingerprint :algorithm] :unknown) - (assoc-in run [:preparations 0 :fingerprint :digest] "not a digest") - (assoc-in run [:attempts 0 :ordinal] 3) - (assoc-in run [:attempts 0 :preparation-id] :unknown) - (assoc-in run [:remaining :saved/pan :repetitions] 2) - (assoc-in run [:attempts 0 :metrics metric-id :value] js/NaN)]] - (t/is (= ::result/invalid-record - (:type (error-data #(result/decode (transit/encode-str bad))))))))) +(t/deftest raw-special-values-and-partial-failures-survive-transit + (let [values [0 -1 js/NaN js/Infinity js/-Infinity] + initial (-> (new-run 0 6 :reuse) + (result/record-preparation (preparation :prep []))) + run (-> (reduce (fn [run value] + (add-attempt run :prep false :completed + [(block :main :prep [(event :test/pan [value])])])) + initial values) + (result/record-attempt + (measurement/attempt :saved/pan :prep false + {:status "failed" :message "render failed" + :cause [{:message "WASM trap"}] + :partial {:clock js/NaN} + :phases [(assoc (block :main :prep [(event :test/pan [99])]) + :outcome :failed)]})) + (result/finish-run "end" {:reason :failed}) + result/encode result/decode) + raw (mapv #(get-in % [:phases 0 :events 0 :metrics :command/elapsed 0]) (:attempts run)) + obs (summary/observations run :saved/pan main-pan)] + (t/is (= [0 -1] (subvec raw 0 2))) + (t/is (js/Number.isNaN (nth raw 2))) + (t/is (= [js/Infinity js/-Infinity] (subvec raw 3 5))) + (t/is (= [{:message "WASM trap"}] (get-in run [:attempts 5 :failure :cause]))) + (t/is (js/Number.isNaN (get-in run [:attempts 5 :partial :clock]))) + (t/is (= 99 (nth raw 5))) + (t/is (= [0] (:values obs))) + (t/is (= 1 (get-in obs [:counts :failed]))) + (t/is (= 4 (get-in obs [:counts :invalid]))))) -(t/deftest recording-keeps-preparation-links-and-failure-accounting - (let [run (run-with [] {:repetitions 2}) - attempt (measurement/attempt :saved/pan :prep/one false - {:status "ok" :metrics {metric-id (entry 4)}})] - (doseq [bad [(assoc preparation :case-id :unknown) - preparation]] - (t/is (= ::result/invalid-record - (:type (error-data #(result/record-preparation run bad)))))) - (doseq [bad [(assoc attempt :preparation-id :unknown) - (assoc attempt :preparation-id nil) - (assoc attempt :case-id :unknown)]] - (t/is (= ::result/invalid-record - (:type (error-data #(result/record-attempt run bad)))))) - (let [failed (assoc attempt :preparation-id nil :outcome :failed) - saved (-> run - (result/record-attempt failed) - (result/record-attempt attempt) - (result/finish-run "end" {:reason :failed}) - result/encode - result/decode) - report (summary/summarize saved)] - (t/is (= {:warmups 0 :repetitions 0} (get-in saved [:remaining :saved/pan]))) - (t/is (= {:valid 1 :invalid 0 :absent 0 :failed 1 :unattempted 0 :warmup 0} - (get-in report [:cases 0 :metrics metric-id :counts]))) - (t/is (= 4 (get-in report [:cases 0 :metrics metric-id :statistics :median :value]))) - (t/is (= ::result/invalid-record - (:type (error-data #(result/record-preparation saved (assoc preparation :preparation-id :prep/two))))))))) - -(t/deftest standard-metrics-and-batches-use-one-observation - (let [raw {:full-ms 12 :viewport-ready-ms 10 - :slices [{:duration-ms 0} {:duration-ms 2}] - :cached-slices [{:duration-ms 3}] :settling-requested-ms 100} - fresh (measurement/metrics raw true {:upload-ms 99}) - warm (measurement/metrics raw false {:upload-ms 99}) - batch (entry 2 {:basis {:kind :per-call-average :calls 200}}) - run (run-with [2]) - run (assoc-in run [:attempts 0 :metrics metric-id] batch)] - (t/is (= 5 (get-in fresh [:renderer-call-total-ms :value]))) - (t/is (= 12 (get-in fresh [:first-render-ms :value])) "upload stays outside first render") - (t/is (not (contains? fresh :full-ms))) - (t/is (not (contains? warm :upload-ms))) - (t/is (not (contains? warm :settling-requested-ms))) - (t/is (= 1 (get-in (summary/summarize run) [:cases 0 :metrics metric-id :statistics :count]))) - (t/is (str/includes? (format/format-run (summary/summarize run)) ":per-call-average")) - (t/is (js/Number.isNaN (measurement/render-call-total {:slices [{:duration-ms js/NaN}]}))))) - -(t/deftest render-call-evidence-distinguishes-missing-from-zero - (let [raw [{} - {:slices []} - {:cached-slices []} - {:slices [] :cached-slices []} - {:slices [{:duration-ms 0}] :cached-slices []} - {:slices [] :cached-slices [{:duration-ms 0}]} - {:slices [{:frame-type "full"}]}] - run (reduce (fn [run trace] - (result/record-attempt - run (measurement/attempt - :saved/pan :prep/one false - {:status "ok" :metrics (measurement/metrics trace false {}) - :trace trace}))) - (run-with [] {:repetitions (count raw)}) raw) - saved (-> (result/finish-run run "end" {:reason :completed}) - result/encode result/decode) - metric (get-in (summary/summarize saved) [:cases 0 :metrics :renderer-call-total-ms])] - (t/is (= raw (mapv :trace (:attempts saved)))) - (t/is (= {:valid 2 :invalid 4 :absent 1 :failed 0 :unattempted 0 :warmup 0} - (:counts metric))) - (t/is (= 2 (get-in metric [:statistics :count]))) - (t/is (= 0 (get-in metric [:statistics :median :value]))) - (doseq [ordinal [1 2 3 6]] - (t/is (= :missing-value - (get-in saved [:attempts ordinal :metrics :renderer-call-total-ms :invalid-reason])))))) - -(t/deftest report-numbers-keep-integers-readable-and-small-values-precise - (doseq [[value text] [[10000000 "10000000"] - [0.125 "0.125000"] - [0.000000125 "1.25000e-7"]]] - (t/is (str/includes? (format/format-run (summary/summarize (run-with [value]))) - (str " median " text " CI "))))) - -(t/deftest warmup-order-and-interrupted-warmup-accounting - (let [run (-> (result/create-run {:run-id "warmup" :started-at "start" - :plan {:cases [saved-case] :warmups 3 :repetitions 2} - :metadata metadata}) - (result/record-preparation preparation)) - attempt (measurement/attempt :saved/pan :prep/one true - {:status "ok" :metrics {metric-id (entry 1)}}) - partial (-> run - (result/record-attempt (assoc attempt :outcome :failed)) - (result/record-attempt attempt)) - ended (result/finish-run partial "end" {:reason :interrupted}) - saved (-> ended result/encode result/decode) - case-summary (first (:cases (summary/summarize saved))) - ready (result/record-attempt partial attempt) - completed (result/record-attempt ready (assoc attempt :warmup? false))] +(t/deftest empty-blocks-support-preparation-only-and-reset-only-cases + (doseq [phase [:preparation :reset]] + (let [run (-> (new-run 0 1 :reuse) + (result/record-preparation + (preparation :prep (if (= :preparation phase) [(event :test/work [2])] []))) + (add-attempt :prep false :completed + [(block :main :prep []) + (block :reset :prep (if (= :reset phase) [(event :test/work [2])] []))]) + (result/finish-run "end" {:reason :completed}) + result/encode result/decode)] + (t/is (= [2] (:values (summary/observations run :saved/pan [phase :global :command/elapsed])))))) + (let [empty-run (-> (new-run 0 1 :reuse) + (result/record-preparation (preparation :prep [])) + (add-attempt :prep false :completed [(block :main :prep [])]))] (t/is (= ::result/invalid-record - (:type (error-data #(result/record-attempt partial (assoc attempt :warmup? false)))))) - (t/is (= {:warmups 1 :repetitions 2} (:remaining case-summary))) - (t/is (= {:valid 0 :invalid 0 :absent 0 :failed 0 :unattempted 2 :warmup 2} - (get-in case-summary [:metrics metric-id :counts]))) - (t/is (= {:warmups 0 :repetitions 1} (get-in completed [:remaining :saved/pan]))) - (t/is (= ::result/invalid-record - (:type (error-data #(result/record-attempt ready attempt))))) - (let [reordered (assoc completed :attempts - (mapv (fn [ordinal attempt] (assoc attempt :ordinal ordinal)) - (range) (reverse (:attempts completed))))] - (t/is (= ::result/invalid-record - (:type (error-data #(result/decode (transit/encode-str reordered))))))))) + (error-type #(result/finish-run empty-run "end" {:reason :completed}))))) + (let [failed (-> (new-run 0 1 :reuse) + (result/record-preparation {:preparation-id :prep :case-id :saved/pan :warmup? false + :diagnostics {} :phases [] + :failure {:message "setup failed"}}) + (result/finish-run "end" {:reason :preparation-error}))] + (t/is (str/includes? (format/format-run (summary/summarize (result/decode (result/encode failed)))) + "setup failed")))) -(t/deftest records-roundtrip-summarize-and-compare-without-registry - (let [run (run-with (vec (range 1 11))) - transported (-> run result/encode result/decode) - comparison (-> (compare/compare-runs transported transported) +(t/deftest offline-comparison-uses-one-analysis-and-saved-quantity-units + (let [base (-> (sample-run [1 2 3]) result/encode result/decode) + candidate (-> (sample-run [3 4 5]) + (assoc-in [:analysis :seed] 99) + (assoc-in [:quantities :command/elapsed :description] "Reworded description") + result/encode result/decode) + comparison (-> (compare/compare-runs base candidate {:analysis {:draws 20}}) result/encode result/decode)] - (t/is (= (summary/summarize run) (summary/summarize transported))) - (t/is (= 0 (get-in comparison [:pairs 0 :metrics metric-id :absolute :value]))) + (t/is (= 2 (get-in comparison [:pairs 0 :metrics main-pan :absolute :value]))) + (t/is (= 20 (get-in comparison [:analysis :draws]))) (t/is (empty? (get-in comparison [:pairs 0 :mismatches]))) - (t/is (str/includes? (format/format-run (summary/summarize transported)) "runtime-helper identity is unverified")) - (t/is (= 99 (get-in (summary/summarize run {:draws 99}) [:analysis :draws]))) - (t/is (= method/defaults (:analysis run))))) + (t/is (str/includes? (format/format-comparison comparison) "median change 2")) + (let [changed (assoc-in candidate [:quantities :command/elapsed :unit] :seconds)] + (t/is (= ::compare/incompatible (error-type #(compare/compare-runs base changed)))) + (t/is (= 2 (get-in (compare/compare-runs base changed {:allow-all-differences true}) + [:pairs 0 :metrics main-pan :absolute :value])))))) -(t/deftest all-dynamic-metric-identifiers-remain-distinct - (let [ids [:dynamic/id "duration" 'custom/time [:operation 7] nil false] - run (assoc-in (run-with [1]) [:attempts 0 :metrics] - (into {} (map (fn [id] [id (entry 1)])) ids)) - decoded (result/decode (result/encode run)) - comparison (compare/compare-runs decoded decoded)] - (t/is (= (set ids) (set (keys (get-in (summary/summarize decoded) [:cases 0 :metrics]))))) - (t/is (= (set ids) (set (keys (get-in comparison [:pairs 0 :metrics]))))))) +(t/deftest unknown-and-malformed-quantities-produce-diagnostics + (doseq [[description reason] [[nil :unknown-quantity] [{:unit 2} :malformed-quantity]]] + (let [run (cond-> (sample-run [1]) + (nil? description) (update :quantities dissoc :command/elapsed) + (some? description) (assoc-in [:quantities :command/elapsed] description)) + report (get-in (summary/summarize (result/decode (result/encode run))) + [:cases 0 :metrics main-pan])] + (t/is (= reason (get-in report [:diagnostics 0 :reason]))) + (t/is (= :no-valid-observations (get-in report [:statistics :median :reason])))))) -(t/deftest startup-and-incomplete-evidence-retain-unattempted-slots - (let [empty (result/create-run {:run-id "empty" :started-at "start" - :plan {:cases [saved-case] :warmups 0 :repetitions 3} - :metadata metadata}) - stopped (result/finish-run empty "end" {:reason :startup-error :failure {:message "cannot launch"}})] - (t/is (empty? (:attempts stopped))) - (t/is (= {:warmups 0 :repetitions 3} (get-in stopped [:remaining :saved/pan]))) - (t/is (= stopped (result/decode (result/encode stopped)))))) +(defn- feature-run + [features selection] + (-> (sample-run [1]) + (assoc-in [:metadata :build :functional :features] features) + (assoc-in [:metadata :build :provenance :feature-selection] selection))) -(t/deftest completed-attempts-exclude-invalid-metrics-individually - (let [run (assoc-in (run-with [1]) [:attempts 0 :metrics :bad] (entry js/NaN)) - report (summary/summarize run)] - (t/is (= 1 (get-in report [:cases 0 :metrics metric-id :counts :valid]))) - (t/is (= 1 (get-in report [:cases 0 :metrics :bad :counts :invalid]))) - (t/is (= :no-valid-observations (get-in report [:cases 0 :metrics :bad :statistics :median :reason]))))) - -(t/deftest definition-drift-is-visible-and-all-differences-compares-raw-values - (let [run (assoc-in (run-with [1 2]) [:attempts 1 :metrics metric-id :basis] - {:kind :per-call-average :calls 2}) - report (summary/summarize run) - comparison (compare/compare-runs run run {:allow-all-differences true})] - (t/is (= :inconsistent-metric-definitions (get-in report [:cases 0 :metrics metric-id :statistics :reason]))) - (t/is (= ::compare/incompatible (:type (error-data #(compare/compare-runs run run))))) - (t/is (= 0 (get-in comparison [:pairs 0 :metrics metric-id :absolute :value]))) - (t/is (= 1.5 (get-in comparison [:pairs 0 :baseline :metrics metric-id :statistics :median :value]))) - (t/is (true? (get-in comparison [:pairs 0 :baseline :metrics metric-id :mixed-definitions?]))))) - -(t/deftest failed-invalid-and-warmup-definitions-do-not-suppress-valid-statistics - (let [run (-> (run-with [1 :failed js/NaN] {:warmups 1}) - (assoc-in [:attempts 0 :metrics metric-id] (entry 100 {:unit :seconds})) - (assoc-in [:attempts 2 :metrics metric-id] (entry 100 {:unit :seconds})) - (assoc-in [:attempts 3 :metrics metric-id] (entry js/NaN {:unit :seconds}))) - report (summary/summarize run)] - (t/is (= 1 (get-in report [:cases 0 :metrics metric-id :statistics :median :value]))) - (t/is (= {:valid 1 :invalid 1 :absent 0 :failed 1 :unattempted 0 :warmup 1} - (get-in report [:cases 0 :metrics metric-id :counts]))))) - -(t/deftest matching-cases-ignore-provenance-and-repetition-counts - (let [baseline (run-with [1 2 3]) - candidate (run-with [2 3 4 5] {:definition (assoc saved-case :scene-description "Reworded description") - :analysis {:seed 99} - :facts (-> metadata - (assoc :git {:sha "other" :dirty true}) - (assoc-in [:build :provenance :run-marker] "second"))}) - comparison (compare/compare-runs baseline (assoc candidate :run-id "other" :started-at "later"))] - (t/is (= method/defaults (:analysis comparison))) - (t/is (empty? (get-in comparison [:pairs 0 :mismatches]))) - (t/is (= 1.5 (get-in comparison [:pairs 0 :metrics metric-id :absolute :value]))) - (t/is (= comparison (-> comparison result/encode result/decode))))) - -(t/deftest strict-and-all-differences-compatibility - (let [base (run-with [1 2])] - (doseq [[field changed] - [[:fingerprint (run-with [3] {:prep (assoc-in preparation [:fingerprint :digest] (apply str (repeat 64 "b")))})] - [:execution (run-with [3] {:definition (assoc saved-case :operation {:steps 30})})] - [:scored? (run-with [3] {:facts (assoc metadata :scored? false)})] - [:execution (run-with [3] {:definition (assoc saved-case :batch-size 20)})] - [:execution (run-with [3] {:definition (assoc saved-case :completion :call)})] - [:execution (run-with [3] {:definition (assoc saved-case :completion :await)})] - [:execution (run-with [3] {:definition (assoc saved-case :preparation {:settle-ms 25})})] - [:environment (run-with [3] {:facts (assoc-in metadata [:environment :browser] "other")})] - [:effective-features (run-with [3] {:facts (assoc-in metadata [:build :functional :features] #{:other})})] - [:functional-build (run-with [3] {:facts (assoc-in metadata [:build :functional :mode] :debug)})] - [:scene (run-with [3] {:definition (assoc saved-case :scene :other)})] - [:scene (run-with [3] {:definition (assoc saved-case :scene-seed 43)})] - [:scene (run-with [3] {:definition (assoc saved-case :params {:count 20})})] - [:warmups (run-with [3] {:warmups 1})] - [:effective-graphics (run-with [3] {:prep (assoc preparation :effective-graphics {:renderer "software" :software true})})] - [:timeout-policy (assoc-in (run-with [3]) [:plan :timeout-policy :attempt-ms] 100)]]] - (let [changed (-> changed result/encode result/decode) - rejected (error-data #(compare/compare-runs base changed)) - selected (error-data #(compare/compare-runs base changed {:pairs [[:saved/pan :saved/pan]]})) - comparison (compare/compare-runs base changed {:allow-all-differences true}) - mismatches (get-in comparison [:pairs 0 :mismatches]) - text (format/format-comparison comparison)] - (t/is (= ::compare/incompatible (:type rejected))) - (t/is (= rejected selected)) - (t/is (= {:allow-build-differences false :allow-scene-differences false :allow-all-differences true} - (:permissions comparison))) - (t/is (= [field] (mapv :field mismatches))) - (t/is (= (mapv #(assoc % :waived? true :rule :all) (:mismatches rejected)) mismatches)) - (t/is (every? #(and (false? (:waived? %)) (not (contains? % :rule))) (:mismatches rejected))) - (t/is (str/includes? text "Permissions")) - (t/is (str/includes? text "ACKNOWLEDGED DIFFERENCE")) - (t/is (str/includes? text (str ":field " (pr-str field)))))))) - -(t/deftest explicit-pairs-select-cases-without-waiving-compatibility - (let [base (run-with [1]) - other (-> (run-with [2] {:definition (assoc saved-case :id :different/zoom :scene :different - :operation {:steps 8} :batch-size 20 - :completion :call :preparation {:settle-ms 25}) - :prep (assoc-in preparation [:fingerprint :digest] (apply str (repeat 64 "b")))}) - result/encode - result/decode) - pairs (compare/selector-pair base other ["saved/pan" "different/zoom"]) - rejected (error-data #(compare/compare-runs base other {:pairs pairs})) - comparison (-> (compare/compare-runs base other {:pairs pairs :allow-all-differences true}) - result/encode result/decode) - execution (first (filter #(= :execution (:field %)) (get-in comparison [:pairs 0 :mismatches]))) - execution-keys [:operation :batch-size :completion :preparation]] - (t/is (= [[:saved/pan :different/zoom]] pairs)) - (t/is (= 1 (get-in comparison [:pairs 0 :metrics metric-id :absolute :value]))) - (t/is (= ::compare/incompatible (:type rejected))) - (t/is (= #{:scene :execution :fingerprint} (set (map :field (:mismatches rejected))))) - (t/is (= (mapv #(assoc % :waived? true :rule :all) (:mismatches rejected)) - (get-in comparison [:pairs 0 :mismatches]))) - (t/is (every? :waived? (get-in comparison [:pairs 0 :mismatches]))) - (t/is (str/includes? (format/format-comparison comparison) "ACKNOWLEDGED DIFFERENCE")) - (t/is (str/includes? (format/format-comparison comparison) ":field :execution")) - (t/is (str/includes? (format/format-comparison comparison) ":field :fingerprint")) - (t/is (= {:operation {:steps 3} - :batch-size 1 :completion :render-full :preparation {}} - (select-keys (:baseline execution) execution-keys))) - (t/is (= {:operation {:steps 8} - :batch-size 20 :completion :call :preparation {:settle-ms 25}} - (select-keys (:candidate execution) execution-keys))) - (t/is (= ::compare/incompatible - (:type (error-data #(compare/compare-runs base (assoc-in other [:metadata :environment :cpu] "other") - {:pairs pairs}))))) - (let [unscored (assoc-in other [:metadata :scored?] false) - rejected (error-data #(compare/compare-runs base unscored {:pairs pairs})) - comparison (compare/compare-runs base unscored {:pairs pairs :allow-all-differences true}) - mismatches (get-in comparison [:pairs 0 :mismatches]) - text (format/format-comparison comparison)] - (t/is (= ::compare/incompatible (:type rejected))) - (t/is (true? (get-in comparison [:permissions :allow-all-differences]))) - (t/is (= (mapv #(assoc % :waived? true :rule :all) (:mismatches rejected)) mismatches)) - (t/is (= #{:scene :execution :fingerprint :scored?} (set (map :field mismatches)))) - (t/is (every? :waived? mismatches)) - (t/is (str/includes? text "ACKNOWLEDGED DIFFERENCE")) - (t/is (str/includes? text ":field :scored?"))) - (t/is (= ["saved/pan"] (:available (error-data #(compare/resolve-case base "pan"))))))) - -(t/deftest explicit-pairs-allow-unlike-names-with-matching-execution-and-fingerprints - (let [base (-> (run-with [1]) result/encode result/decode) - other (-> (run-with [2] {:definition (assoc saved-case :id :different/zoom)}) - result/encode result/decode) - comparison (-> (compare/compare-runs base other {:pairs [[:saved/pan :different/zoom]]}) - result/encode result/decode)] - (t/is (= {:allow-build-differences false :allow-scene-differences false :allow-all-differences false} - (:permissions comparison))) - (t/is (true? (get-in comparison [:pairs 0 :explicit?]))) - (t/is (empty? (get-in comparison [:pairs 0 :mismatches]))) - (t/is (= 1 (get-in comparison [:pairs 0 :metrics metric-id :absolute :value]))))) - -(t/deftest all-differences-preserves-metric-definitions-and-excludes-failed-values - (let [base (run-with [1 :failed]) - changed (assoc-in (run-with [2]) [:attempts 0 :metrics metric-id] - (entry 2 {:unit :seconds :basis {:kind :per-call-average :calls 50}})) - comparison (compare/compare-runs base changed {:allow-all-differences true})] - (t/is (= 1 (get-in comparison [:pairs 0 :metrics metric-id :absolute :value]))) - (t/is (= :seconds (get-in comparison [:pairs 0 :candidate :metrics metric-id :definitions 0 :unit]))) - (t/is (= 1 (get-in comparison [:pairs 0 :baseline :metrics metric-id :counts :failed]))) - (t/is (= ::compare/no-shared-metrics - (:type (error-data #(compare/compare-runs base - (assoc-in changed [:attempts 0 :metrics] {:other (entry 2)}) - {:allow-all-differences true}))))))) - -(t/deftest comparison-boundary-validates-nested-evidence - (let [run (run-with [1 2]) - comparison (compare/compare-runs run run) - changes-path [:pairs 0 :metrics metric-id] - report-path [:pairs 0 :baseline :metrics metric-id]] - (doseq [changed [(assoc-in comparison (conj changes-path :absolute :value) js/NaN) - (assoc-in comparison (conj changes-path :absolute-ci) {:lower 2 :upper 1}) - (assoc-in comparison (conj report-path :counts :valid) -1) - (assoc-in comparison (conj report-path :definitions 0 :basis :calls) 0) - (assoc-in comparison [:baseline :metadata :scored?] :unknown) - (dissoc comparison :pairs) - (assoc-in comparison [:analysis :confidence] 1)]] - (t/is (= ::result/invalid-record (:type (error-data #(result/decode (transit/encode-str changed))))))))) - -(t/deftest all-differences-keeps-invalid-measurements-unusable - (let [baseline (run-with [js/NaN js/Infinity :absent :failed] {:repetitions 5}) - candidate (run-with [2] {:facts (assoc-in metadata [:environment :cpu] "other")}) - comparison (compare/compare-runs baseline candidate {:allow-all-differences true})] - (t/is (= {:valid 0 :invalid 2 :absent 1 :failed 1 :unattempted 1 :warmup 0} - (get-in comparison [:pairs 0 :baseline :metrics metric-id :counts]))) - (t/is (= :no-valid-observations (get-in comparison [:pairs 0 :metrics metric-id :absolute :reason]))) - (t/is (= :no-valid-observations (get-in comparison [:pairs 0 :metrics metric-id :percentage-ci :reason]))) - (t/is (= comparison (result/decode (result/encode comparison)))))) - -(t/deftest build-permission-acknowledges-only-recorded-build-differences - (let [baseline (run-with [1])] - (doseq [[key value field] [[:mode :debug :functional-build] - [:features ["other"] :effective-features] - [:default-features? false :functional-build] - [:instrumentation {:stats true} :functional-build] - [:env {"PROFILE" "1"} :functional-build]]] - (let [candidate (run-with [2] {:facts (assoc-in metadata [:build :functional key] value)}) - comparison (compare/compare-runs baseline candidate {:allow-build-differences true}) - mismatches (get-in comparison [:pairs 0 :mismatches])] - (t/is (= ::compare/incompatible (:type (error-data #(compare/compare-runs baseline candidate))))) - (t/is (= ::compare/incompatible - (:type (error-data #(compare/compare-runs baseline candidate {:allow-scene-differences true}))))) - (t/is (= [field] (mapv :field mismatches))) - (t/is (every? #(and (:waived? %) (= :build (:rule %))) mismatches)) - (t/is (= {:allow-build-differences true :allow-scene-differences false :allow-all-differences false} - (:permissions comparison))) - (t/is (= comparison (-> comparison result/encode result/decode))))))) - -(t/deftest run-build-environment-validates-strings-and-preserves-absence - (let [missing (-> (run-with [1]) result/encode result/decode)] - (t/is (not (contains? (get-in missing [:metadata :build]) :environment))) - (doseq [environment [{} {"CFLAGS" "" "RUSTFLAGS" "-C\u001fopt-level=3"}]] - (let [run (run-with [1] {:facts (assoc-in metadata [:build :environment] environment)})] - (t/is (= environment (get-in (-> run result/encode result/decode) [:metadata :build :environment]))))) - (doseq [environment [nil [] {:CFLAGS ""} {"CFLAGS" nil} {"CFLAGS" 0}]] - (t/is (= ::result/invalid-record - (:type (error-data #(result/decode - (transit/encode-str - (assoc-in missing [:metadata :build :environment] environment)))))))))) - -(t/deftest build-environment-differences-require-build-or-all-permission - (doseq [[before after] [[::missing {}] [{} ::missing] - [{} {"CFLAGS" ""}] - [{"CFLAGS" "-O2"} {"CFLAGS" "-O3"}]]] - (let [baseline (run-with [1] {:facts (cond-> metadata - (not= ::missing before) (assoc-in [:build :environment] before))}) - candidate (run-with [2] {:facts (cond-> metadata - (not= ::missing after) (assoc-in [:build :environment] after))})] - (t/is (empty? (get-in (compare/compare-runs baseline baseline) [:pairs 0 :mismatches]))) - (doseq [permissions [{} {:allow-scene-differences true}]] - (t/is (= ::compare/incompatible - (:type (error-data #(compare/compare-runs baseline candidate permissions)))))) - (doseq [[permissions rule] [[{:allow-build-differences true} :build] - [{:allow-all-differences true} :all]]] - (let [comparison (compare/compare-runs baseline candidate permissions)] - (t/is (= [{:field :build-environment - :baseline (when-not (= ::missing before) before) - :candidate (when-not (= ::missing after) after) - :waived? true :rule rule}] - (get-in comparison [:pairs 0 :mismatches]))) - (t/is (= comparison (-> comparison result/encode result/decode)))))))) - -(t/deftest feature-toggle-does-not-acknowledge-build-environment-changes - (let [baseline (assoc-in (run-with-features [1] [] {:default-features? false :features []}) - [:metadata :build :environment] {"CFLAGS" "-O2"}) - candidate (assoc-in (run-with-features [2] ["toggle"] {:default-features? false :features ["toggle"]}) - [:metadata :build :environment] {"CFLAGS" "-O3"}) - expected {:feature "toggle" :baseline-features [] :candidate-features ["toggle"] - :added ["toggle"] :removed []} - diagnostic (compare/compare-runs baseline candidate {:expected-feature-change expected - :on-unallowed-difference :record})] - (t/is (= ::compare/incompatible - (:type (error-data #(compare/compare-runs baseline candidate {:expected-feature-change expected}))))) - (t/is (= [[:effective-features true :feature-toggle] [:build-environment false nil]] - (mapv (juxt :field :waived? :rule) (get-in diagnostic [:pairs 0 :mismatches])))) - (let [allowed (compare/compare-runs baseline candidate {:expected-feature-change expected - :allow-build-differences true})] - (t/is (= [[:effective-features true :feature-toggle] [:build-environment true :build]] - (mapv (juxt :field :waived? :rule) (get-in allowed [:pairs 0 :mismatches]))))))) - -(t/deftest scene-permission-acknowledges-scene-inputs-and-fingerprint-content - (let [baseline (run-with [1])] - (doseq [[field options] [[:scene {:definition (assoc saved-case :scene :other)}] - [:scene {:definition (assoc saved-case :scene-seed 43)}] - [:scene {:definition (assoc saved-case :params {:count 2000})}] - [:fingerprint {:prep (assoc-in preparation [:fingerprint :digest] - (apply str (repeat 64 "b")))}]]] - (let [candidate (run-with [2] options) - comparison (compare/compare-runs baseline candidate {:allow-scene-differences true}) - mismatches (get-in comparison [:pairs 0 :mismatches])] - (t/is (= ::compare/incompatible (:type (error-data #(compare/compare-runs baseline candidate))))) - (t/is (= ::compare/incompatible - (:type (error-data #(compare/compare-runs baseline candidate {:allow-build-differences true}))))) - (t/is (= [field] (mapv :field mismatches))) - (t/is (every? #(and (:waived? %) (= :scene (:rule %))) mismatches)) - (t/is (= 1 (get-in comparison [:pairs 0 :metrics metric-id :absolute :value]))) - (t/is (= comparison (-> comparison result/encode result/decode))))))) - -(t/deftest build-and-scene-permissions-combine-by-union - (let [baseline (run-with [1]) - candidate (run-with [2] {:definition (assoc saved-case :id :other/load :params {:count 2000}) - :facts (assoc-in metadata [:build :functional :mode] :debug) - :prep (assoc-in preparation [:fingerprint :digest] (apply str (repeat 64 "b")))}) - pairs [[:saved/pan :other/load]] - options {:pairs pairs :allow-build-differences true :allow-scene-differences true} - comparison (compare/compare-runs baseline candidate options) - mismatches (get-in comparison [:pairs 0 :mismatches])] - (doseq [permissions [{} {:allow-build-differences true} {:allow-scene-differences true}]] - (t/is (= ::compare/incompatible - (:type (error-data #(compare/compare-runs baseline candidate (assoc permissions :pairs pairs))))))) - (t/is (= #{[:scene :scene] [:functional-build :build] [:fingerprint :scene]} - (set (map (juxt :field :rule) mismatches)))) - (t/is (every? :waived? mismatches)) - (t/is (= {:allow-build-differences true :allow-scene-differences true :allow-all-differences false} - (:permissions comparison))) - (t/is (= comparison (-> comparison result/encode result/decode))))) - -(t/deftest scoped-permissions-retain-execution-runtime-and-measurement-checks - (let [baseline (run-with [1]) - scoped {:allow-build-differences true :allow-scene-differences true}] - (doseq [[field candidate] - [[:execution (run-with [2] {:definition (assoc saved-case :operation {:steps 30})})] - [:execution (run-with [2] {:definition (assoc saved-case :operation {:steps 3 :args {:distance 20}})})] - [:execution (run-with [2] {:definition (assoc saved-case :view {:scale 2 :x 0 :y 0})})] - [:execution (run-with [2] {:definition (assoc saved-case :context :fresh)})] - [:execution (run-with [2] {:definition (assoc saved-case :completion :call)})] - [:execution (run-with [2] {:definition (assoc saved-case :batch-size 10)})] - [:execution (run-with [2] {:definition (assoc saved-case :preparation {:settle-ms 25})})] - [:environment (run-with [2] {:facts (assoc-in metadata [:environment :cpu] "other")})] - [:effective-graphics (run-with [2] {:prep (assoc-in preparation [:effective-graphics :dpr] 1)})] - [:warmups (run-with [2] {:warmups 1})] - [:timeout-policy (assoc-in (run-with [2]) [:plan :timeout-policy :attempt-ms] 100)] - [:scored? (run-with [2] {:facts (assoc metadata :scored? false)})] - [[:metric metric-id] (assoc-in (run-with [2]) [:attempts 0 :metrics metric-id :unit] :seconds)]]] - (let [candidate (-> candidate result/encode result/decode) - rejected (error-data #(compare/compare-runs baseline candidate scoped)) - comparison (compare/compare-runs baseline candidate (assoc scoped :allow-all-differences true))] - (t/is (= ::compare/incompatible (:type rejected))) - (t/is (= [field] (mapv :field (:mismatches rejected)))) - (t/is (every? #(and (false? (:waived? %)) (not (contains? % :rule))) (:mismatches rejected))) - (t/is (= (mapv #(assoc % :waived? true :rule :all) (:mismatches rejected)) - (get-in comparison [:pairs 0 :mismatches]))))))) - -(t/deftest scene-permission-never-waives-incomplete-or-varying-evidence - (let [baseline (run-with [1]) - varying (result/record-preparation - (run-with [2]) - (-> preparation - (assoc :preparation-id :prep/two) - (assoc-in [:fingerprint :digest] (apply str (repeat 64 "b"))))) - missing (-> (run-with [:failed]) - (assoc :preparations []) - (assoc-in [:attempts 0 :preparation-id] nil) - (assoc-in [:attempts 0 :metrics metric-id] (entry 2))) - scoped {:allow-scene-differences true :allow-build-differences true}] - (doseq [candidate [varying missing]] - (let [candidate (-> candidate result/encode result/decode) - rejected (error-data #(compare/compare-runs baseline candidate scoped)) - diagnostic (compare/compare-runs baseline candidate (assoc scoped :on-unallowed-difference :record)) - mismatches (get-in diagnostic [:pairs 0 :mismatches]) - evidence (first (filter #(= [:fingerprint :incomplete-or-varying-evidence] (:field %)) mismatches))] - (t/is (= ::compare/incompatible (:type rejected))) - (t/is (= (:mismatches rejected) mismatches)) - (t/is (false? (:waived? evidence))) - (t/is (not (contains? evidence :rule))) - (t/is (= :scene (:rule (first (filter #(= :fingerprint (:field %)) mismatches))))) - (t/is (every? :waived? - (get-in (compare/compare-runs baseline candidate {:allow-all-differences true}) - [:pairs 0 :mismatches]))) - (t/is (= diagnostic (-> diagnostic result/encode result/decode))))))) - -(t/deftest diagnostic-record-retains-unacknowledged-differences-and-selected-rules - (let [baseline (run-with [1]) - candidate (run-with [2] {:definition (assoc saved-case :params {:count 2000} :operation {:steps 30}) - :facts (assoc-in metadata [:build :functional :mode] :debug)}) - options {:allow-build-differences true :on-unallowed-difference :record} - diagnostic (compare/compare-runs baseline candidate options) - mismatches (get-in diagnostic [:pairs 0 :mismatches]) - text (format/format-comparison diagnostic)] - (t/is (= (:mismatches (error-data #(compare/compare-runs baseline candidate (dissoc options :on-unallowed-difference)))) - mismatches)) - (t/is (= [[:scene false nil] [:execution false nil] [:functional-build true :build]] - (mapv (juxt :field :waived? :rule) mismatches))) - (t/is (= 1 (get-in diagnostic [:pairs 0 :metrics metric-id :absolute :value]))) - (t/is (str/includes? text "Permissions {:allow-build-differences true")) - (t/is (str/includes? text "ACKNOWLEDGED DIFFERENCE {:field :functional-build")) - (t/is (str/includes? text "UNACKNOWLEDGED DIFFERENCE {:field :scene")) - (t/is (str/includes? text ":rule :build")) - (t/is (= diagnostic (-> diagnostic result/encode result/decode))))) - -(t/deftest diagnostic-record-keeps-strict-definition-statistics - (let [baseline (run-with [1 2]) - mixed (assoc-in (run-with [3 4]) [:attempts 1 :metrics metric-id :basis] - {:kind :per-call-average :calls 2}) - differing (assoc-in (run-with [3 4]) [:attempts 0 :metrics metric-id :unit] :seconds) - single-definition (-> (run-with [3 4]) - (assoc-in [:attempts 0 :metrics metric-id :unit] :seconds) - (assoc-in [:attempts 1 :metrics metric-id :unit] :seconds))] - (doseq [candidate [mixed differing single-definition]] - (let [diagnostic (compare/compare-runs baseline candidate {:allow-build-differences true - :allow-scene-differences true - :on-unallowed-difference :record}) - allowed (compare/compare-runs baseline candidate {:allow-all-differences true}) - changes (get-in diagnostic [:pairs 0 :metrics metric-id])] - (t/is (every? #(= {:reason :inconsistent-metric-definitions} %) (vals changes))) - (t/is (= 2 (get-in diagnostic [:pairs 0 :candidate :metrics metric-id :counts :valid]))) - (t/is (every? #(false? (:waived? %)) (get-in diagnostic [:pairs 0 :mismatches]))) - (t/is (= 2 (get-in allowed [:pairs 0 :metrics metric-id :absolute :value]))) - (t/is (= 3.5 (get-in allowed [:pairs 0 :candidate :metrics metric-id :statistics :median :value]))) - (if (= candidate single-definition) - (t/is (= 3.5 (get-in diagnostic [:pairs 0 :candidate :metrics metric-id :statistics :median :value]))) - (t/is (= :inconsistent-metric-definitions - (get-in diagnostic [:pairs 0 :candidate :metrics metric-id :statistics :reason])))) - (t/is (= diagnostic (-> diagnostic result/encode result/decode))))))) - -(t/deftest feature-toggle-acknowledges-only-exact-effective-feature-evidence - (let [baseline (run-with-features [1] ["base" "old"] {:default-features? false :features ["base"]}) - candidate (run-with-features [2] ["base" "toggle" "transitive"] - {:default-features? false :features ["base" "toggle"]}) - expected {:feature "toggle" :baseline-features ["base" "old"] - :candidate-features ["base" "toggle" "transitive"] - :added ["toggle" "transitive"] :removed ["old"]} - edge (assoc expected :from "0" :to "1") - comparison (compare/compare-runs baseline candidate {:expected-feature-change expected :edge edge}) - mismatch (get-in comparison [:pairs 0 :mismatches 0])] - (t/is (= :effective-features (:field mismatch))) - (t/is (true? (:waived? mismatch))) - (t/is (= :feature-toggle (:rule mismatch))) - (t/is (= edge (:edge comparison))) - (t/is (= {:allow-build-differences false :allow-scene-differences false :allow-all-differences false} - (:permissions comparison))) - (t/is (= comparison (-> comparison result/encode result/decode))) - (doseq [invalid [nil - (assoc expected :feature "base") - (assoc expected :feature :toggle) +(t/deftest automatic-feature-permission-checks-effective-and-requested-evidence + (let [baseline (feature-run ["base" "old"] {:default-features? false :features ["base"]}) + candidate (feature-run ["base" "toggle" "transitive"] + {:default-features? false :features ["base" "toggle"]}) + expected {:feature "toggle" :baseline-features ["base" "old"] + :candidate-features ["base" "toggle" "transitive"] + :added ["toggle" "transitive"] :removed ["old"]} + edge (assoc expected :from "0" :to "1") + report (compare/compare-runs baseline candidate {:expected-feature-change expected :edge edge})] + (t/is (= :feature-toggle (get-in report [:pairs 0 :mismatches 0 :rule]))) + (t/is (= report (result/decode (result/encode report)))) + (doseq [invalid [nil (assoc expected :feature "base") (assoc expected :feature :toggle) (assoc expected :baseline-features ["base"]) (assoc expected :candidate-features ["base" "toggle"]) - (assoc expected :added ["toggle"]) - (assoc expected :removed []) + (assoc expected :added ["toggle"]) (assoc expected :removed []) (assoc expected :added #{"toggle" "transitive"}) (dissoc expected :baseline-features)]] - (let [options {:expected-feature-change invalid} - rejected (error-data #(compare/compare-runs baseline candidate options)) - diagnostic (compare/compare-runs baseline candidate (assoc options :on-unallowed-difference :record))] - (t/is (= ::compare/incompatible (:type rejected))) - (t/is (false? (get-in diagnostic [:pairs 0 :mismatches 0 :waived?]))) - (t/is (not (contains? (get-in diagnostic [:pairs 0 :mismatches 0]) :rule))))) - (let [changed (-> candidate - (assoc-in [:metadata :build :functional :mode] :debug) - (assoc-in [:plan :cases 0 :params] {:count 2000}) - (assoc-in [:plan :cases 0 :operation] {:steps 30})) - diagnostic (compare/compare-runs baseline changed {:expected-feature-change expected - :on-unallowed-difference :record})] - (t/is (= [[:scene false nil] [:execution false nil] - [:effective-features true :feature-toggle] [:functional-build false nil]] - (mapv (juxt :field :waived? :rule) (get-in diagnostic [:pairs 0 :mismatches]))))))) - -(t/deftest edge-context-roundtrips-and-labels-identical-effective-sets - (let [run (run-with-features [1] ["base" "alias"] {:default-features? false :features ["base"]}) - candidate (run-with-features [2] ["alias" "base"] {:default-features? false :features ["base" "toggle"]}) - expected {:feature "toggle" :baseline-features ["base" "alias"] - :candidate-features ["alias" "base"] :added [] :removed []} - edge (assoc expected :from "0" :to "1") - comparison (compare/compare-runs run candidate {:expected-feature-change expected :edge edge})] - (t/is (empty? (get-in comparison [:pairs 0 :mismatches]))) - (t/is (= edge (:edge comparison))) - (t/is (= comparison (-> comparison result/encode result/decode))) - (t/is (str/includes? (format/format-comparison comparison) "Edge 0 -> 1 feature toggle")) - (t/is (str/includes? (format/format-comparison comparison) "effective feature sets identical")) - (let [contradictory (assoc edge :candidate-features ["base" "alias" "toggle"] :added ["toggle"])] - (doseq [opts [{:edge contradictory} - {:edge contradictory - :expected-feature-change (dissoc contradictory :from :to)}]] - (t/is (= ::compare/invalid-edge - (:type (error-data #(compare/compare-runs run candidate opts))))))))) - -(t/deftest comparison-schema-validates-permissions-rules-and-edge-context - (let [baseline (run-with [1]) - candidate (run-with [2] {:facts (assoc-in metadata [:build :functional :mode] :debug)}) - comparison (compare/compare-runs baseline candidate {:allow-build-differences true}) - edge {:from "0" :to "1" :feature "toggle" :baseline-features [] - :candidate-features ["toggle"] :added ["toggle"] :removed []}] - (doseq [changed [(dissoc comparison :permissions) - (assoc-in comparison [:permissions :allow-build-differences] :yes) - (update comparison :permissions dissoc :allow-scene-differences) - (assoc comparison :forced? true) - (assoc-in comparison [:pairs 0 :mismatches 0 :rule] :other) - (assoc-in comparison [:pairs 0 :mismatches 0 :rule] nil) - (assoc-in comparison [:pairs 0 :mismatches 0 :waived?] false) - (update-in comparison [:pairs 0 :mismatches 0] dissoc :rule) - (assoc comparison :edge (dissoc edge :removed)) - (assoc comparison :edge (assoc edge :feature :toggle)) - (assoc comparison :edge (assoc edge :added #{"toggle"}))]] - (t/is (= ::result/invalid-record (:type (error-data #(result/decode (transit/encode-str changed))))))) - (doseq [rule [:feature-toggle :build :scene :all]] - (let [record (-> comparison (assoc :edge edge) (assoc-in [:pairs 0 :mismatches 0 :rule] rule))] - (t/is (= record (-> record result/encode result/decode))) - (t/is (= ::result/invalid-record - (:type (error-data #(result/decode (transit/encode-str - (update-in record [:pairs 0 :mismatches 0] dissoc :rule))))))))))) - -(t/deftest every-permission-keeps-invalid-measurements-and-no-shared-metrics-unusable - (let [baseline (run-with [js/NaN js/Infinity 0 -1 :absent :failed] {:warmups 1 :repetitions 7}) - candidate (run-with [2] {:warmups 1})] - (doseq [permissions [{} - {:allow-build-differences true} - {:allow-scene-differences true} - {:allow-build-differences true :allow-scene-differences true} - {:allow-all-differences true} - {:on-unallowed-difference :record}]] - (let [comparison (compare/compare-runs baseline candidate permissions)] - (t/is (= {:valid 0 :invalid 4 :absent 1 :failed 1 :unattempted 1 :warmup 1} - (get-in comparison [:pairs 0 :baseline :metrics metric-id :counts]))) - (t/is (= :no-valid-observations (get-in comparison [:pairs 0 :metrics metric-id :absolute :reason]))) - (t/is (= :no-valid-observations (get-in comparison [:pairs 0 :metrics metric-id :percentage-ci :reason]))) - (t/is (= comparison (-> comparison result/encode result/decode))) - (t/is (= ::compare/no-shared-metrics - (:type (error-data #(compare/compare-runs baseline - (update candidate :attempts - (fn [attempts] - (mapv (fn [attempt] - (assoc attempt :metrics {:other (entry 2)})) attempts))) - permissions))))))))) - -(t/deftest missing-nil-and-empty-effective-feature-evidence-stay-distinct-through-transit - (let [empty-run (run-with-features [1] [] {:default-features? false :features []}) - missing (-> empty-run - (update-in [:metadata :build :functional] dissoc :features) - result/encode result/decode) - nil-run (-> empty-run - (assoc-in [:metadata :build :functional :features] nil) - result/encode result/decode)] - (t/is (empty? (get-in (compare/compare-runs empty-run empty-run) [:pairs 0 :mismatches]))) - (doseq [[baseline candidate bfeatures cfeatures] - [[missing empty-run ::compare/missing-features #{}] - [nil-run empty-run nil #{}] - [missing nil-run ::compare/missing-features nil] - [empty-run missing #{} ::compare/missing-features] - [empty-run nil-run #{} nil]]] - (let [rejected (error-data #(compare/compare-runs baseline candidate)) - diagnostic (compare/compare-runs baseline candidate {:on-unallowed-difference :record}) - mismatch {:field :effective-features :baseline bfeatures :candidate cfeatures :waived? false}] - (t/is (= ::compare/incompatible (:type rejected))) - (t/is (= [mismatch] (:mismatches rejected))) - (t/is (= [mismatch] (get-in diagnostic [:pairs 0 :mismatches]))) - (t/is (= diagnostic (-> diagnostic result/encode result/decode))))))) - -(t/deftest automatic-feature-acknowledgment-requires-saved-effective-vectors-of-strings - (let [baseline (run-with-features [1] [] {:default-features? false :features []}) - candidate (run-with-features [2] ["toggle"] {:default-features? false :features ["toggle"]}) - expected {:feature "toggle" :baseline-features [] :candidate-features ["toggle"] - :added ["toggle"] :removed []}] - (t/is (= :feature-toggle - (get-in (compare/compare-runs baseline candidate {:expected-feature-change expected}) - [:pairs 0 :mismatches 0 :rule]))) - (doseq [[b c] [[(update-in baseline [:metadata :build :functional] dissoc :features) candidate] - [(assoc-in baseline [:metadata :build :functional :features] nil) candidate] - [(assoc-in baseline [:metadata :build :functional :features] #{}) candidate] - [baseline (assoc-in candidate [:metadata :build :functional :features] #{"toggle"})] - [baseline (assoc-in candidate [:metadata :build :functional :features] [:toggle])]]] - (let [b (-> b result/encode result/decode) - c (-> c result/encode result/decode) - options {:expected-feature-change expected} - rejected (error-data #(compare/compare-runs b c options)) - diagnostic (compare/compare-runs b c (assoc options :on-unallowed-difference :record))] - (t/is (= ::compare/incompatible (:type rejected))) - (t/is (= [:effective-features] (mapv :field (get-in diagnostic [:pairs 0 :mismatches])))) - (t/is (false? (get-in diagnostic [:pairs 0 :mismatches 0 :waived?]))) - (t/is (not (contains? (get-in diagnostic [:pairs 0 :mismatches 0]) :rule))) - (t/is (= diagnostic (-> diagnostic result/encode result/decode))))))) - -(t/deftest automatic-feature-acknowledgment-checks-saved-requested-selections - (let [b-request {:default-features? false :features ["base"]} - c-request {:default-features? false :features ["base" "toggle"]} - expected {:feature "toggle" :baseline-features ["base" "old"] - :candidate-features ["base" "toggle" "transitive"] - :added ["toggle" "transitive"] :removed ["old"]}] - (doseq [default-features? [false true]] - (let [baseline (run-with-features [1] (:baseline-features expected) - (assoc b-request :default-features? default-features?)) - candidate (run-with-features [2] (:candidate-features expected) - {:default-features? default-features? :features ["toggle" "base"]}) - comparison (compare/compare-runs (-> baseline result/encode result/decode) - (-> candidate result/encode result/decode) - {:expected-feature-change expected})] - (t/is (= [:feature-toggle] (mapv :rule (get-in comparison [:pairs 0 :mismatches])))) - (t/is (every? :waived? (get-in comparison [:pairs 0 :mismatches]))))) - (doseq [[b c] [[::missing c-request] - [b-request ::missing] - [nil c-request] - [b-request nil] - [{} c-request] - [b-request {}] - [[] c-request] - [(dissoc b-request :default-features?) c-request] - [b-request (dissoc c-request :default-features?)] - [(assoc b-request :default-features? :false) c-request] - [b-request (assoc c-request :default-features? true)] - [(dissoc b-request :features) c-request] - [b-request (dissoc c-request :features)] - [(assoc b-request :features #{"base"}) c-request] - [b-request (assoc c-request :features ["base" :toggle])] - [c-request c-request] - [b-request b-request] - [b-request (assoc c-request :features ["base" "extra" "toggle"])] - [(assoc b-request :features ["base" "old"]) c-request]]] - (let [baseline (cond-> (run-with-features [1] (:baseline-features expected) b) - (= ::missing b) (update-in [:metadata :build :provenance] dissoc :feature-selection)) - candidate (cond-> (run-with-features [2] (:candidate-features expected) c) - (= ::missing c) (update-in [:metadata :build :provenance] dissoc :feature-selection)) - baseline (-> baseline result/encode result/decode) - candidate (-> candidate result/encode result/decode) - options {:expected-feature-change expected} - rejected (error-data #(compare/compare-runs baseline candidate options)) - diagnostic (compare/compare-runs baseline candidate (assoc options :on-unallowed-difference :record))] - (t/is (= ::compare/incompatible (:type rejected))) - (t/is (= [:effective-features] (mapv :field (get-in diagnostic [:pairs 0 :mismatches])))) - (t/is (false? (get-in diagnostic [:pairs 0 :mismatches 0 :waived?]))) - (t/is (not (contains? (get-in diagnostic [:pairs 0 :mismatches 0]) :rule))) - (t/is (= diagnostic (-> diagnostic result/encode result/decode))))))) - -(t/deftest edge-option-retains-context-and-rejects-conflicting-expected-facts - (let [baseline (run-with-features [1] [] {:default-features? false :features []}) - candidate (run-with-features [2] ["toggle" "transitive"] {:default-features? false :features ["toggle"]}) - expected {:feature "toggle" :baseline-features [] :candidate-features ["toggle" "transitive"] - :added ["toggle" "transitive"] :removed []} - edge (assoc expected :from "0" :to "1") - comparison (compare/compare-runs baseline candidate {:expected-feature-change expected :edge edge})] - (t/is (= edge (:edge comparison))) - (t/is (= comparison (-> comparison result/encode result/decode))) - (t/is (str/includes? (format/format-comparison comparison) "Edge 0 -> 1 feature toggle")) - (t/is (not (str/includes? (format/format-comparison comparison) "effective feature sets identical"))) - (doseq [changed [(assoc edge :feature "other") - (assoc edge :baseline-features ["base"]) - (assoc edge :candidate-features ["toggle"]) - (assoc edge :added ["toggle"]) - (assoc edge :removed ["base"])]] + (t/is (= ::compare/incompatible + (error-type #(compare/compare-runs baseline candidate {:expected-feature-change invalid}))))) + (doseq [selection [nil {} {:default-features? true :features ["base" "toggle"]} + {:default-features? false :features ["toggle"]} + {:default-features? false :features #{"base" "toggle"}}]] + (t/is (= ::compare/incompatible + (error-type #(compare/compare-runs + baseline (assoc-in candidate [:metadata :build :provenance :feature-selection] selection) + {:expected-feature-change expected}))))) + (doseq [bad-edge [(dissoc edge :removed) (assoc edge :added []) (assoc edge :feature :toggle)]] (t/is (= ::compare/invalid-edge - (:type (error-data #(compare/compare-runs baseline candidate - {:expected-feature-change expected :edge changed - :on-unallowed-difference :record})))))))) + (error-type #(compare/compare-runs baseline candidate {:edge bad-edge}))))) + (t/is (= ::compare/invalid-edge + (error-type #(compare/compare-runs baseline candidate + {:edge edge :expected-feature-change (assoc expected :added [])})))))) + +(t/deftest missing-nil-and-empty-effective-features-keep-distinct-meaning + (doseq [before [::missing nil []] after [::missing nil []]] + (let [with-features (fn [features] + (cond-> (sample-run [1]) + (= ::missing features) (update-in [:metadata :build :functional] dissoc :features) + (not= ::missing features) (assoc-in [:metadata :build :functional :features] features))) + baseline (-> (with-features before) result/encode result/decode) + candidate (-> (with-features after) result/encode result/decode) + compared (compare/compare-runs baseline candidate {:allow-build-differences true})] + (t/is (= (if (= before after) [] [:effective-features]) + (mapv :field (get-in compared [:pairs 0 :mismatches]))))))) + +(t/deftest comparison-boundaries-validate-permissions-rules-and-intervals + (let [base (sample-run [1]) + candidate (assoc-in (sample-run [2]) [:metadata :build :functional :mode] :debug) + compared (compare/compare-runs base candidate {:allow-build-differences true}) + path [:pairs 0 :metrics main-pan]] + (doseq [bad [(dissoc compared :permissions) + (assoc-in compared [:permissions :allow-build-differences] :yes) + (update compared :permissions dissoc :allow-scene-differences) + (assoc-in compared [:pairs 0 :mismatches 0 :rule] :other) + (assoc-in compared [:pairs 0 :mismatches 0 :waived?] false) + (update-in compared [:pairs 0 :mismatches 0] dissoc :rule) + (assoc-in compared (conj path :absolute :value) js/NaN) + (assoc-in compared (conj path :absolute-ci) {:lower 2 :upper 1}) + (assoc-in compared [:pairs 0 :baseline :metrics main-pan :counts :valid] -1)]] + (t/is (= ::result/invalid-record (error-type #(result/decode (transit/encode-str bad)))))))) + +(t/deftest scene-permission-requires-complete-consistent-preparation-evidence + (let [base (sample-run [1]) + changed (result/record-preparation base + (assoc-in (preparation :other []) [:fingerprint :digest] + (apply str (repeat 64 "b")))) + report (compare/compare-runs base changed {:allow-all-differences true})] + (t/is (= ::compare/incompatible + (error-type #(compare/compare-runs base changed {:allow-scene-differences true})))) + (t/is (some #(= [:fingerprint :incomplete-or-varying-evidence] (:field %)) + (get-in report [:pairs 0 :mismatches]))))) + +(t/deftest matching-missing-preparation-facts-produce-incomplete-evidence + (let [run (-> (new-run 0 1 :reuse) + (result/record-preparation + (-> (preparation :prep [(event :test/setup [9])]) + (dissoc :fingerprint :effective-graphics) + (assoc :failure {:message "setup failed"}))))] + (doseq [options [{} {:allow-scene-differences true}]] + (t/is (= ::compare/incompatible (error-type #(compare/compare-runs run run options))))) + (let [report (compare/compare-runs run run {:allow-all-differences true})] + (t/is (= #{[:fingerprint :incomplete-or-varying-evidence] + [:effective-graphics :incomplete-or-varying-evidence]} + (set (map :field (get-in report [:pairs 0 :mismatches])))))))) + +(t/deftest saved-boundaries-refuse-corrupt-identities-and-phase-history + (let [run (sample-run [1])] + (doseq [bad [(assoc-in run [:attempts 0 :attempt-id] 3) + (assoc-in run [:attempts 0 :phases] []) + (assoc-in run [:preparations 0 :phases] []) + (assoc-in run [:preparations 0 :failure] {:message "setup failed"}) + (update-in run [:attempts 0] assoc :outcome :failed + :phases [(assoc (block :main :prep []) :outcome :failed) + (block :reset :prep [])]) + (assoc-in run [:attempts 0 :preparation-id] :unknown) + (assoc-in run [:attempts 0 :phases 0 :preparation-id] :unknown) + (assoc-in run [:attempts 0 :phases 0 :events 0 :metrics :command/elapsed] []) + (assoc-in run [:attempts 0 :phases 0 :events 0 :metrics :command/elapsed] [1 2]) + (assoc-in run [:remaining :saved/pan :repetitions] 2)]] + (t/is (= ::result/invalid-record + (error-type #(result/decode (transit/encode-str bad)))))))) diff --git a/frontend/test/performance_tests/render_wasm/run_test.cljs b/frontend/test/performance_tests/render_wasm/run_test.cljs index 6d11986432..5d4f59ec7f 100644 --- a/frontend/test/performance_tests/render_wasm/run_test.cljs +++ b/frontend/test/performance_tests/render_wasm/run_test.cljs @@ -87,8 +87,9 @@ (defn- fake-browser [{:keys [launch-failure new-page-failure close-page-failure prepare-failure run-failure screenshot-failure - interruption controller effective-graphics metrics] - :or {effective-graphics (constantly {}) metrics {}}}] + interruption controller effective-graphics events] + :or {effective-graphics (constantly {}) + events [{:command :fixture/work :metrics {:command/elapsed [1]}}]}}] (let [state (atom {:launches 0 :pages 0 :close-pages 0 :closes 0 :host nil :runs {} :preparations {}})] {:state state @@ -142,7 +143,9 @@ :fingerprint {:algorithm :sha-256 :digest (apply str (repeat 64 "a"))} :effective-graphics (effective-graphics case-id (:id page)) - :diagnostics {}}] + :diagnostics {} + :phases [{:phase :preparation :preparation-id preparation-id + :outcome :completed :events []}]}] (swap! state assoc-in [:preparations preparation-id] case-id) {:status "ok" :preparation preparation})))) :run @@ -165,7 +168,9 @@ :response {:status "failed" :phase "interact" :message "attempt rejected"} - {:status "ok" :metrics metrics :trace {}}))) + {:status "ok" + :phases [{:phase :main :preparation-id preparation-id :outcome :completed + :events events}]}))) :screenshot (fn [page file _] (check-page page) @@ -200,6 +205,15 @@ [run case-id] (filterv #(= case-id (:case-id %)) (:attempts run))) +(t/deftest ^:async unmeasured-successful-work-saves-a-terminal-failure + (let [fake (fake-browser {:events []}) + response (await (execute fake [(collected :case/unmeasured :reuse)] + (options fake (path/join @temporary-root "unmeasured")))) + saved (result/decode (fs/readFileSync (get-in response [:home :record-path]) "utf8"))] + (t/is (= 1 (:exit-code response))) + (t/is (= {:reason :no-measurements :case-ids [:case/unmeasured]} (:termination saved))) + (t/is (= :completed (get-in saved [:attempts 0 :outcome]))))) + (defn- preparations-for [run case-id] (filterv #(= case-id (:case-id %)) (:preparations run))) @@ -343,9 +357,10 @@ run (:run result) tries (attempts-for run :case/fresh)] (t/is (= [:failed :completed] (mapv :outcome tries))) - (t/is (nil? (:preparation-id (first tries)))) + (t/is (= (:preparation-id (first (preparations-for run :case/fresh))) + (:preparation-id (first tries)))) (t/is (= "scene-build" (get-in (first tries) [:failure :phase]))) - (t/is (= 1 (count (preparations-for run :case/fresh)))) + (t/is (= 2 (count (preparations-for run :case/fresh)))) (t/is (= :ok (get-in run [:metadata :diagnostics :screenshots "case/fresh" :status]))) (t/is (every? zero? (vals (get-in run [:remaining :case/fresh])))) (t/is (= :failed (get-in run [:termination :reason]))) @@ -886,11 +901,7 @@ (t/deftest ^:async feature-comparison-completes-with-managed-output (let [root @temporary-root cases [(collected :case/a :fresh)] - fake (fake-browser {:metrics {:first-render-ms - (result/metric {:value 1 :unit :ms - :semantics :prepared-scene-to-full - :basis {:kind :attempt-total} - :zero-policy :positive})}}) + fake (fake-browser {}) prepared {:plan (feature-plan/plan ["a"] {:dry-run true}) :cases cases :output-dir nil