From 6fa70dc47db5491041020da234b012f3683a4d8d Mon Sep 17 00:00:00 2001 From: Miguel de Benito Delgado Date: Wed, 4 Nov 2026 11:17:48 +0000 Subject: [PATCH] :sparkles: Enforce benchmark case deadlines Stop failed cases and preserve their execution evidence. AI-assisted-by: gpt-6.1-sol --- .../performance/benches/render_wasm/README.md | 96 +++-- .../benches/render_wasm/browser/bridge.cljs | 26 +- .../benches/render_wasm/measurement.cljs | 30 +- .../benches/render_wasm/node/cli/shared.cljs | 2 +- .../render_wasm/node/runner/browser.cljs | 41 ++- .../node/runner/feature_comparison.cljs | 2 +- .../benches/render_wasm/node/runner/run.cljs | 329 +++++++++++------- .../benches/render_wasm/report/format.cljs | 3 +- .../benches/render_wasm/report/summarize.cljs | 1 + .../benches/render_wasm/result.cljs | 74 +++- .../render_wasm/cli_integration.cjs | 4 +- .../render_wasm/cli_test.cljs | 2 + .../render_wasm/compare_test.cljs | 6 +- .../render_wasm/feature_comparison_test.cljs | 9 +- .../render_wasm/lifecycle_test.cljs | 23 ++ .../render_wasm/merge_test.cljs | 7 +- .../render_wasm/report_test.cljs | 23 ++ .../render_wasm/run_test.cljs | 327 +++++++++++++++-- .../render_wasm/runs_test.cljs | 2 +- 19 files changed, 774 insertions(+), 233 deletions(-) diff --git a/frontend/performance/benches/render_wasm/README.md b/frontend/performance/benches/render_wasm/README.md index 712a8aff36..844fef3e01 100644 --- a/frontend/performance/benches/render_wasm/README.md +++ b/frontend/performance/benches/render_wasm/README.md @@ -106,8 +106,8 @@ The suite has three parts: chronological order, and performs no hidden retries. - **Run record**: the saved evidence for one run: plan, build and environment provenance, quantity descriptions, preparations, attempts with their phase - histories, failures, remaining slots, and termination. The runner stores it as - a data file validated by a schema. + histories, failures, terminal case outcomes, remaining slots, and termination. + The runner stores it as a data file validated by a schema. - **Bridge**: the object that runs in the browser, receives plain requests from Node, owns the page's renderer context, runs preparations and attempts, and returns plain results. @@ -324,11 +324,22 @@ process it starts. Cases run serially. A fresh case gets a new page and context for each preparation. A warm case keeps one page and context for all of its attempts. -The runner applies deadlines to startup, preparation, and attempts. When a page -stalls past its deadline, Node stops the browser process and records the failure. -The browser bridge also checks for context loss, expired deadlines, and work associated -with a superseded renderer owner before and after asynchronous work. A browser -thread occupied by synchronous work relies on the Node deadline to stop it. +Before cases run, `--startup-timeout-ms` bounds Git provenance, server startup, +and the initial browser launch. Each case then gets one absolute `--timeout-ms` +deadline. The deadline covers the case's page setup, preparation, warmups, +repetitions, reset, and screenshot diagnostics. It does not restart for a phase, +attempt, or browser replacement. Work on a reused preparation keeps its original +case deadline; a request timeout can only tighten that deadline. A browser +launched after a failed case uses the next case's remaining deadline. + +When a page stalls past its deadline, Node stops the browser process and records +the failure. The browser bridge also checks for context loss, expired deadlines, +and work associated with a superseded renderer owner before and after +asynchronous work. A browser thread occupied by synchronous work relies on the +Node deadline to stop it. The runner persists each delivered preparation and +attempt. It does not stream browser-local records and cannot recover in-flight +data when a blocked browser cannot return; records delivered before the block +remain saved. Clean-up closes the page, then the browser, then the server. The runner deletes a build selected with `--remove` only after the final run record is saved. @@ -668,7 +679,9 @@ preparation block's selected events. Reset runs after successful main. Its returned context supplies the next reused attempt. A phase failure releases its owned context and returns the failed block with earlier events and available traces. Completed preparation retains -its own evidence when main or reset fails. +its own evidence when main or reset fails. Any preparation, main, or reset +failure stops the case; the runner attempts no later slot. It disposes the +case's browser and starts the next case in a new browser when the run continues. ### Runtime code @@ -858,7 +871,7 @@ arguments. Comparing hardware and software runs requires A run record contains: - the plan: selected case descriptors, warmup count, repetition count, and - timeout policy; + timeout policy (`:case-ms`); - metadata: Git revision and dirty flag, build facts, environment, and diagnostics; - the resolved analysis settings; - saved quantity descriptions; @@ -866,6 +879,7 @@ A run record contains: - attempts in chronological order, each with its case, preparation, `:attempt-id`, warmup flag, outcome, main/reset phase blocks, and failure or partial evidence; +- terminal outcomes in `:case-outcomes`; - remaining slots for each case; - the termination reason. @@ -876,23 +890,44 @@ blocks. A phase block retains ordered command events, raw metric vectors, and optional completion diagnostics. Errors retain available phase evidence and traces after a failure. +Each case has a terminal entry in `:case-outcomes` with `:outcome` set to +`:completed`, `:failed`, `:timed-out`, or `:interrupted` and an optional +`:failure`. Recording a terminal outcome closes the case: later preparation or +attempt evidence for that case is rejected. + ### Failures and termination A failed attempt still consumes its allocated slot. A fresh preparation failure -records a failed attempt and consumes the slot. A reuse case's preparation -failure or attempt failure releases the prepared page and context and leaves -that case's remaining slots unattempted. A browser or page failure closes the -browser it started, abandons the current case, and starts a new browser for the -next case when possible. The run fixes its attempt count before execution and uses -each allocated slot at most once. +records a failed attempt and consumes the current slot. A reuse preparation +failure records no main or reset execution and consumes no main slot. Any +preparation, main, or reset failure stops the case for fresh and reuse alike, +so later slots stay unattempted. The runner disposes the failed case's browser +and starts the next case in a new browser when the run continues. A browser or +page failure follows the same case-stop rule. A failure to clean up the browser +for a failed case aborts later cases. The run fixes its attempt count before +execution and uses each allocated slot at most once. + +A malformed delivered response follows the same slot rules. An attempt response +consumes the slot it invoked, keeps the valid phase prefix, and saves the raw +response as partial evidence. A malformed preparation response fails its +preparation; a fresh case still consumes the current slot. + +A screenshot failure is diagnostic when it is an ordinary error. A deadline +expiry or interruption during the screenshot stops the case, while earlier +completed attempts remain saved. + +A case that finishes all its slots without a measured command gets a failed +case outcome whose failure has `:reason :no-measurements`. The run then +terminates as failed or incomplete according to the remaining slots. A run terminates in one of four states: -- **completed**: every allocated slot was attempted and every attempt completed. -- **failed**: every allocated slot was attempted and at least one attempt - failed. -- **incomplete**: slots remain because a preparation, browser, or storage - failure stopped the case. +- **completed**: every allocated slot was attempted, every case outcome is + `:completed`, and every case has at least one measurement. +- **failed**: every allocated slot was attempted and at least one attempt failed + or one case outcome is not `:completed`. +- **incomplete**: slots remain because a case stopped early or a startup, + browser, or storage failure prevented execution. - **interrupted**: `SIGINT` or `SIGTERM` stopped the run. The runner saves partial records before it exits. Use exit status to determine @@ -931,8 +966,10 @@ A finite zero is valid. Negative and non-finite raw values remain in the saved record but invalidate the dependent execution sum and aggregate; independent metric keys remain usable. Unknown or malformed quantity metadata appears as a diagnostic and leaves that key without usable statistics. A phase can complete -with no events, but a completed run requires at least one measurement across -each case's phases. +with no events. Preparation-only and reset-only measured cases are valid. A +case that finishes all its slots without any measured command gets a failed +case outcome with `:reason :no-measurements`; the run is failed when every slot +was attempted and incomplete when slots remain. The case report also counts actual preparation, main, and reset executions. Completed and failed counts use each execution's own outcome, and a separate @@ -976,7 +1013,7 @@ By default, a comparison requires equal: - effective graphics settings and runtime environment; - build mode, effective features, instrumentation settings, and recorded build environment; -- warmup count, timeout policy, and saved quantity units. +- warmup count, timeout policy (`:case-ms`), and saved quantity units. Comparison pairs metric keys `[phase scope metric-type]`. Quantity description text does not affect compatibility; changed prose is not a difference. Unknown @@ -1304,12 +1341,13 @@ mean interruption by `SIGINT` and `SIGTERM` for `build`, `run` and `ab`. Interruption uses those statuses to report aborted work. Errors writing the command's record or log also produce diagnostics. -`--warmups` defaults to 3, `--repetitions` to 10, `--timeout-ms` to 30000, and -`--startup-timeout-ms` to 30000. `build`, a run that prepares its build, and an -A/B experiment use a default `--build-timeout-ms` of 300000; a run with -`--build` uses the retained build's recorded provenance. The master seed -defaults to 42 and accepts a 32-bit unsigned integer. A filter matches a -substring of the case id. +`--warmups` defaults to 3, `--repetitions` to 10, `--timeout-ms` (the case +deadline) to 30000, and `--startup-timeout-ms` (host and initial browser +readiness) to 30000. `build`, a run that prepares its build, and an A/B +experiment use a default `--build-timeout-ms` of 300000; a run with `--build` +uses the retained build's recorded provenance. The master seed defaults to 42 +and accepts a 32-bit unsigned integer. A filter matches a substring of the case +id. ## Unit tests diff --git a/frontend/performance/benches/render_wasm/browser/bridge.cljs b/frontend/performance/benches/render_wasm/browser/bridge.cljs index 764958d870..3c356d3204 100644 --- a/frontend/performance/benches/render_wasm/browser/bridge.cljs +++ b/frontend/performance/benches/render_wasm/browser/bridge.cljs @@ -104,7 +104,7 @@ [:module-url {:optional true} string?] [:wasm-url {:optional true} string?]]) -(def ^:const ^:private attempt-ms-budget +(def ^:const ^:private case-ms-budget "Default elapsed-time deadline for setup or one attempt, checked by `browser-hooks`. Requests can supply a timeout. The outer process timeout covers blocked pages." @@ -128,15 +128,15 @@ :frame #(js/Promise. (fn [resolve] (.requestAnimationFrame js/window resolve))) :sleep #(js/Promise. (fn [resolve] (js/setTimeout resolve %))) - ;; checks that this attempt still owns the page, WebGL context hasn't been + ;; checks that this execution still owns the page, WebGL context hasn't been ;; lost, and the deadline hasn't passed :check (fn [] (when @wasm/context-lost? (throw (ex-info "WebGL context lost" {:phase "context-lost"}))) (guard-current! epoch "protocol") - (when (> (now) deadline-ms) - (throw (ex-info "attempt deadline exceeded" + (when (>= (now) deadline-ms) + (throw (ex-info "case deadline exceeded" {:phase "deadline"})))) ;; WASM calls :render (fn [timestamp flags] @@ -976,7 +976,7 @@ Validation runs before the owner changes. A valid request releases the prior owner, starts a new owner epoch, builds browser hooks with a deadline and prepares the scene. `:preparation-id` defaults to the new epoch and - `:timeout-ms` defaults to `attempt-ms-budget`. + `:timeout-ms` defaults to `case-ms-budget`. Args: request: Transit string matching `schema:load-scene-args`. @@ -994,8 +994,10 @@ (js/Promise.resolve (errors/fail-data (errors/evidence "invalid-args" parsed))) (let [{:keys [params case-desc]} parsed {:keys [epoch] :as owner} (begin-load! case-desc) - timeout-ms (or (:timeout-ms params) attempt-ms-budget) - owner (assoc owner :hooks (browser-hooks epoch (+ (now) timeout-ms)))] + timeout-ms (or (:timeout-ms params) case-ms-budget) + deadline-ms (+ (now) timeout-ms) + owner (assoc owner :deadline-ms deadline-ms + :hooks (browser-hooks epoch deadline-ms))] (prepare-owned-scene (assoc parsed :owner owner :timeout-ms timeout-ms @@ -1009,8 +1011,8 @@ Args: preparation-id: identifier of the prepared scene to use. - timeout-ms: optional deadline for this attempt; defaults to the - preparation's timeout. + timeout-ms: optional remaining budget in milliseconds, bounded by the + preparation's case deadline. Returns: A Promise of status ok with `:preparation` evidence, top-level `:phases` @@ -1030,8 +1032,10 @@ (js/Promise.resolve (errors/fail-data {:phase "invalid-args" :message "preparation unavailable, busy or already used"})) (let [epoch (:epoch owner) - timeout-ms (or timeout-ms (:timeout-ms prepared)) - owner (assoc owner :hooks (browser-hooks epoch (+ (now) timeout-ms)))] + deadline-ms (if timeout-ms + (min (:deadline-ms owner) (+ (now) timeout-ms)) + (:deadline-ms owner)) + owner (assoc owner :hooks (browser-hooks epoch deadline-ms))] (swap! prepared* assoc :busy? true :used? true) (-> (run-case owner case-desc local-case graphics context) (.then (fn [evidence] diff --git a/frontend/performance/benches/render_wasm/measurement.cljs b/frontend/performance/benches/render_wasm/measurement.cljs index 1eda48be87..b9fc292d33 100644 --- a/frontend/performance/benches/render_wasm/measurement.cljs +++ b/frontend/performance/benches/render_wasm/measurement.cljs @@ -5,7 +5,33 @@ ;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns benches.render-wasm.measurement - "Projects browser phase evidence into saved attempts.") + "Projects browser phase evidence into saved attempts." + (:require + [app.common.schema :as sm] + [benches.render-wasm.result :as result])) + +(defn partial-phases + "Returns the valid prefix of delivered phase history. + + Args: + phases: raw blocks from a browser response. + preparation-id: expected owner of the blocks. + order: allowed phase names in execution order. + + Returns: + Valid blocks up to the first malformed block or failed phase." + [phases preparation-id order] + (loop [phases (when (sequential? phases) (seq phases)) order order out []] + (if-let [phase (first phases)] + (if (and (sm/validate result/schema:phase phase) + (= preparation-id (:preparation-id phase)) + (= (first order) (:phase phase)) + (every? #(= 1 (count (get-in % [:metrics :command/elapsed]))) (:events phase))) + (let [out (conj out phase)] + (if (= :failed (:outcome phase)) out + (recur (next phases) (next order) out))) + out) + out))) (defn attempt "Returns an attempt with raw phase histories and available failure evidence. @@ -21,7 +47,7 @@ [case-id preparation-id warmup? evidence] (cond-> {:case-id case-id :preparation-id preparation-id :warmup? warmup? :outcome (if (= "ok" (:status evidence)) :completed :failed) - :phases (filterv #(#{:main :reset} (:phase %)) (:phases evidence))} + :phases (or (:phases evidence) [])} (not= "ok" (:status evidence)) (assoc :failure (select-keys evidence [:status :phase :message :cause])) (:partial evidence) (assoc :partial (:partial evidence)))) diff --git a/frontend/performance/benches/render_wasm/node/cli/shared.cljs b/frontend/performance/benches/render_wasm/node/cli/shared.cljs index 0f8d2210db..671d9036e9 100644 --- a/frontend/performance/benches/render_wasm/node/cli/shared.cljs +++ b/frontend/performance/benches/render_wasm/node/cli/shared.cljs @@ -80,7 +80,7 @@ "must be a nonnegative JavaScript-safe integer") (number-option "--repetitions N" "Measured attempts per case" schema:positive-count "must be a positive JavaScript-safe integer") - (number-option "--timeout-ms MS" "Attempt deadline" schema:timeout-ms + (number-option "--timeout-ms MS" "Case deadline including setup, attempts and diagnostics" schema:timeout-ms "must be a positive integer within Node's timer range") (number-option "--startup-timeout-ms MS" "Host/browser readiness deadline" schema:timeout-ms "must be a positive integer within Node's timer range") diff --git a/frontend/performance/benches/render_wasm/node/runner/browser.cljs b/frontend/performance/benches/render_wasm/node/runner/browser.cljs index 8efd5ba558..218fc0daba 100644 --- a/frontend/performance/benches/render_wasm/node/runner/browser.cljs +++ b/frontend/performance/benches/render_wasm/node/runner/browser.cljs @@ -88,6 +88,14 @@ (catch :default cause (failed cause)))))))))) +(defn- deadline-failure + [cause deadline] + (if (contains? (ex-data cause) ::deadline-exceeded) + cause + (errors/with-context cause (errors/message-of cause) + {::deadline-exceeded (or (= ::timeout (:type (ex-data cause))) + (zero? (process/remaining deadline)))}))) + (defn evaluate "Calls a fixed bridge method, optionally with a Transit string. JSON quoting embeds the argument; descriptors and results remain Transit." @@ -258,17 +266,18 @@ bounds))] (assoc owner :browser browser :version (.version browser) :endpoint address)) (catch :default cause - (try - (await (close owner)) - (catch :default cleanup-cause - (throw (ex-info "browser shutdown failed after startup error" - {:type ::startup-cleanup - :endpoint @(:address endpoint) :user-data-dir user-data-dir - :startup-failure (errors/describe-cause cause)} - cleanup-cause)))) - (throw (ex-info (errors/message-of cause) - (assoc (ex-data cause) :endpoint @(:address endpoint) :user-data-dir user-data-dir) - cause))))))) + (let [cause (deadline-failure cause deadline)] + (try + (await (close owner)) + (catch :default cleanup-cause + (throw (ex-info "browser shutdown failed after startup error" + {:type ::startup-cleanup + :endpoint @(:address endpoint) :user-data-dir user-data-dir + :startup-failure (errors/describe-cause cause)} + cleanup-cause)))) + (throw (ex-info (errors/message-of cause) + (assoc (ex-data cause) :endpoint @(:address endpoint) :user-data-dir user-data-dir) + cause)))))))) (defn ^:async new-page "Creates a context and page in a browser process we own, navigates to host, @@ -301,11 +310,13 @@ (transit/decode-str (await (within #(evaluate page "buildIdentity") bounds)))) page-own (catch :default cause - (ex/ignoring (await (close-page page-own))) - (throw cause)))) + (let [cause (deadline-failure cause deadline)] + (ex/ignoring (await (close-page page-own))) + (throw cause))))) (catch :default cause - (ex/ignoring (await (close-page {:context context}))) - (throw cause))))) + (let [cause (deadline-failure cause deadline)] + (ex/ignoring (await (close-page {:context context}))) + (throw cause)))))) (defn connected? "True while the owned CDP connection is live." diff --git a/frontend/performance/benches/render_wasm/node/runner/feature_comparison.cljs b/frontend/performance/benches/render_wasm/node/runner/feature_comparison.cljs index 201f22d4ef..a8138e60c3 100644 --- a/frontend/performance/benches/render_wasm/node/runner/feature_comparison.cljs +++ b/frontend/performance/benches/render_wasm/node/runner/feature_comparison.cljs @@ -810,5 +810,5 @@ (:edges plan))) "\n" "Cases: " (count cases) "\n" "Attempts per case: " (:warmups options) " warmups, " (:repetitions options) " measured\n" - "Timeouts (ms): attempt " (:timeout-ms options) + "Timeouts (ms): case " (:timeout-ms options) ", build " (:build-timeout-ms options) ", startup " (:startup-timeout-ms options) "\n")) diff --git a/frontend/performance/benches/render_wasm/node/runner/run.cljs b/frontend/performance/benches/render_wasm/node/runner/run.cljs index f82d557764..116ff66247 100644 --- a/frontend/performance/benches/render_wasm/node/runner/run.cljs +++ b/frontend/performance/benches/render_wasm/node/runner/run.cljs @@ -51,14 +51,16 @@ (defn- cause-evidence "Plain failure evidence for a rejected browser, server or persistence action." [cause fallback-phase] - (errors/fail-data - (errors/evidence (or (:phase (ex-data cause)) fallback-phase) cause))) + (cond-> (errors/fail-data + (errors/evidence (or (:phase (ex-data cause)) fallback-phase) cause)) + (contains? (ex-data cause) ::browser/deadline-exceeded) + (assoc :deadline-exceeded (::browser/deadline-exceeded (ex-data cause))))) (defn- failure-evidence "Projects a bridge failure response into diagnostics evidence." [evidence fallback-phase] (if (map? evidence) - (cond-> (select-keys evidence [:status :phase :message :cause :partial :phases]) + (cond-> (select-keys evidence [:status :phase :message :cause :partial :phases :reason :deadline-exceeded]) (nil? (:status evidence)) (assoc :status "failed") (nil? (:phase evidence)) (assoc :phase (name fallback-phase))) {:status "failed" @@ -74,6 +76,20 @@ [state] (some? (interruption (:signal state)))) +(defn- check-deadline + [deadline] + (when (zero? (process/remaining deadline)) + (throw (ex-info "case deadline exceeded" {:type ::browser/timeout :phase :deadline})))) + +(defn- failure-outcome + [state deadline evidence] + (cond + (interrupted? state) :interrupted + (if (contains? evidence :deadline-exceeded) + (:deadline-exceeded evidence) + (or (zero? (process/remaining deadline)) (= "deadline" (:phase evidence)))) :timed-out + :else :failed)) + (defn- keep-sink-failure "Remembers the first failed write to the run record or log. The failure stays in the outcome even when a later write succeeds." @@ -126,7 +142,7 @@ (defn- record-failure [state case-id stage evidence] - (let [failure (cond-> (assoc (select-keys evidence [:status :phase :message :cause :partial]) + (let [failure (cond-> (assoc (select-keys evidence [:status :phase :message :cause :partial :reason]) :stage stage) (some? case-id) (assoc :case-id case-id))] (reset! (:harness-failed? state) true) @@ -162,11 +178,10 @@ (record-screenshot-skip state (:id case) reason)))) (defn- ^:async screenshot-case - [state case page] - (when (:screenshots (:opts state)) + [state case page deadline] + (if (:screenshots (:opts state)) (let [{:keys [file path]} (runs/screenshot-path (:home state) (:id case)) - opts (:opts state) - bounds {:deadline (process/deadline (:startup-timeout-ms opts)) + bounds {:deadline deadline :signal (:signal state) :phase :screenshot} shot (attempt (await ((get-in state [:services :screenshot]) page path bounds)))] (if (failure? shot) @@ -177,10 +192,16 @@ :cause (errors/describe-cause shot)}) (write-log state log/warn (str "Screenshot failed for " (result/case-name (:id case)) ": " - (errors/message-of shot) "\n"))) + (errors/message-of shot) "\n")) + (let [outcome (failure-outcome state deadline (cause-evidence shot :screenshot))] + (when (= :timed-out outcome) + (record-failure state (:id case) :case-deadline (cause-evidence shot :screenshot))) + (if (= :failed outcome) :ok outcome))) (do (record-screenshot-status state (:id case) {:file file :status :ok}) - (write-log state log/debug (str "Screenshot " file))))))) + (write-log state log/debug (str "Screenshot " file)) + :ok))) + :ok)) (defn- case-slots "Returns each scheduled slot in order: all warmups, then measured slots. @@ -195,8 +216,8 @@ (defn- ^:async prepare-case [state case page deadline warmup?] - (let [opts (:opts state) - build-id (:id (:selected-build state)) + (check-deadline deadline) + (let [build-id (:id (:selected-build state)) preparation-id (str (random-uuid)) response (attempt (await ((get-in state [:services :prepare]) @@ -204,18 +225,30 @@ (transit/encode-str {:case case :preparation-id preparation-id - :timeout-ms (:timeout-ms opts) + :timeout-ms (max 1 (process/remaining deadline)) :module-url (renderer-url build-id "render-wasm.js") :wasm-url (renderer-url build-id "render-wasm.wasm")}) - {:deadline deadline :signal (:signal state) :phase :preparation})))] - (if (and (not (failure? response)) (= "ok" (:status response))) - {:preparation (assoc (:preparation response) :warmup? warmup?)} - (let [failure (if (failure? response) - (cause-evidence response :preparation) - (failure-evidence response :preparation))] + {:deadline deadline :signal (:signal state) :phase :preparation}))) + accepted (when (and (not (failure? response)) (= "ok" (:status response))) + (attempt + (let [preparation (assoc (:preparation response) :warmup? warmup?)] + (when-not (and (= preparation-id (:preparation-id preparation)) + (= (:id case) (:case-id preparation)) + (not (:failure preparation))) + (throw (ex-info "preparation success response is inconsistent" {}))) + (record-preparation state preparation) + preparation)))] + (if (and accepted (not (failure? accepted))) + {:preparation accepted} + (let [failure (cond + (failure? accepted) (assoc (cause-evidence accepted :invalid-output) + :partial {:response response}) + (failure? response) (cause-evidence response :preparation) + :else (failure-evidence response :preparation)) + phases (if (failure? accepted) (get-in response [:preparation :phases]) (:phases failure))] (record-preparation state {:preparation-id preparation-id :case-id (:id case) :warmup? warmup? - :phases (or (:phases failure) []) + :phases (measurement/partial-phases phases preparation-id [:preparation]) :diagnostics (or (get-in failure [:partial :setup]) {}) :failure (dissoc failure :phases)}) {:failure failure :preparation-id preparation-id})))) @@ -234,17 +267,23 @@ (defn- ^:async recycle-browser [state] (when-let [owner @(:owner-ref state)] - (reset! (:owner-ref state) nil) (let [closed (attempt (await ((get-in state [:services :close]) owner)))] - (when (failure? closed) - (write-log state log/warn (str "Browser cleanup failed: " (errors/message-of closed))))))) + (if (failure? closed) + (do + (record-failure state nil :browser-close (cause-evidence closed :browser-close)) + (reset! (:abort-run? state) true)) + (do + (reset! (:owner-ref state) nil) + (reset! (:page-ref state) nil)))))) (defn- ^:async close-current-page - [state] + [state case] (let [owner @(:owner-ref state) connected? (get-in state [:services :connected?])] (if (and owner connected? (not (connected? owner))) (do + (record-failure state (:id case) :browser-disconnected + {:phase "browser-disconnected" :message "browser disconnected"}) (reset! (:page-ref state) nil) (await (recycle-browser state)) false) @@ -254,20 +293,18 @@ (let [closed (attempt (await ((get-in state [:services :close-page]) page)))] (if (failure? closed) (do - (record-failure state nil :browser-close (cause-evidence closed :browser-close)) + (record-failure state (:id case) :browser-close (cause-evidence closed :browser-close)) (await (recycle-browser state)) false) true))) true)))) (defn- ^:async ensure-browser - [state] + [state deadline] (or @(:owner-ref state) (let [opts (:opts state) - options (cond-> {:startup-timeout-ms (:startup-timeout-ms opts) - :signal (:signal state)} - (some? @(:launch-deadline state)) - (assoc :deadline @(:launch-deadline state))) + options {:startup-timeout-ms (:startup-timeout-ms opts) + :signal (:signal state) :deadline deadline} owner (await ((get-in state [:services :launch]) options))] (reset! (:owner-ref state) owner) (swap! (:run-ref state) assoc-in [:metadata :environment :browser] (:version owner)) @@ -301,11 +338,12 @@ :ok))) (defn- ^:async open-case-page - [state case owner] + [state case owner deadline] (let [opened (attempt (await ((get-in state [:services :new-page]) owner @(:host-ref state) {:viewport (declarations/resolve-viewport (:view case)) + :deadline deadline :startup-timeout-ms (:startup-timeout-ms (:opts state)) :signal (:signal state)})))] (cond @@ -318,141 +356,150 @@ {:outcome :interrupted} :else - (do - (record-failure state (:id case) :browser-page (cause-evidence opened :browser-page)) + (let [evidence (cause-evidence opened :browser-page) + outcome (failure-outcome state deadline evidence)] + (record-failure state (:id case) :browser-page evidence) (await (recycle-browser state)) - {:outcome :abandon})))) + {:outcome outcome})))) (defn- ^:async open-page - [state case] + [state case deadline] + (check-deadline deadline) (if (interrupted? state) {:outcome :interrupted} - (let [owner (attempt (await (ensure-browser state)))] + (let [owner (attempt (await (ensure-browser state deadline)))] (cond (not (failure? owner)) - (await (open-case-page state case owner)) + (await (open-case-page state case owner deadline)) (interrupted? state) {:outcome :interrupted} :else - (do - (record-failure state (:id case) :browser-launch - (cause-evidence owner :browser-launch)) - (reset! (:abort-run? state) true) - {:outcome :abort}))))) - -(defn- accept-preparation - [state preparation] - (record-preparation state preparation) - (check-gpu state preparation)) + (let [evidence (cause-evidence owner :browser-launch) + outcome (failure-outcome state deadline evidence)] + (record-failure state (:id case) :browser-launch evidence) + (if (and (= :timed-out outcome) + (not= ::browser/startup-cleanup (:type (ex-data owner)))) + {:outcome outcome} + (do + (reset! (:abort-run? state) true) + {:outcome :abort}))))))) (defn- ^:async execute-slot [state case page preparation-id warmup? deadline] (let [response (await (run-attempt state page preparation-id deadline)) - measured (measurement/attempt (:id case) preparation-id warmup? - (if (failure? response) - (cause-evidence response :attempt) - response))] - (record-attempt state measured) + evidence (if (failure? response) (cause-evidence response :attempt) response) + accepted (attempt + (let [measured (measurement/attempt (:id case) preparation-id warmup? evidence)] + (record-attempt state measured) + measured)) + measured (if (failure? accepted) + (let [invalid (assoc (cause-evidence accepted :invalid-output) + :phases (measurement/partial-phases (:phases evidence) preparation-id [:main :reset]) + :partial {:response evidence}) + failed (measurement/attempt (:id case) preparation-id warmup? invalid)] + (record-attempt state failed) + failed) + accepted)] (cond (and (failure? response) (interrupted? state)) :interrupted (= :completed (:outcome measured)) :ok - :else :failed))) + :else (failure-outcome state deadline (if (failure? response) + (cause-evidence response :attempt) + response))))) (defn- ^:async finish-slot - [state case page outcome final?] - (when final? - (if (= :ok outcome) - (await (screenshot-case state case page)) - (record-screenshot-skip state (:id case) :no-completed-measured-attempt))) - outcome) + [state case page outcome final? deadline] + (if (and final? (= :ok outcome)) + (await (screenshot-case state case page deadline)) + (do + (when (not= :ok outcome) + (record-screenshot-skip state (:id case) :no-completed-measured-attempt)) + outcome))) (defn- ^:async run-fresh-slot - [state case page warmup? final?] - (let [deadline (process/deadline (:timeout-ms (:opts state))) - prepared (await (prepare-case state case page deadline warmup?))] + [state case page warmup? final? deadline] + (let [prepared (await (prepare-case state case page deadline warmup?))] (if-let [evidence (:failure prepared)] (do - (record-attempt state (measurement/attempt (:id case) (:preparation-id prepared) warmup? evidence)) + (record-attempt state (measurement/attempt (:id case) (:preparation-id prepared) warmup? + (dissoc evidence :phases))) (await (finish-slot state case page - (if (interrupted? state) :interrupted :failed) final?))) + (failure-outcome state deadline evidence) final? deadline))) (let [preparation (:preparation prepared)] - (if (= :abort (accept-preparation state preparation)) + (if (= :abort (check-gpu state preparation)) :abort (let [outcome (await (execute-slot state case page (:preparation-id preparation) warmup? deadline))] - (await (finish-slot state case page outcome final?)))))))) + (await (finish-slot state case page outcome final? deadline)))))))) (defn- ^:async run-fresh-page - [state case warmup? final?] - (let [{:keys [page outcome]} (await (open-page state case))] + [state case warmup? final? deadline] + (let [{:keys [page outcome]} (await (open-page state case deadline))] (if page - (let [outcome (await (run-fresh-slot state case page warmup? final?)) - closed? (await (close-current-page state))] - (cond - (#{:abort :interrupted} outcome) outcome - (not closed?) :abandon - :else outcome)) + (let [outcome (await (run-fresh-slot state case page warmup? final? deadline))] + (if (= :ok outcome) + (if (await (close-current-page state case)) outcome :failed) + outcome)) (or outcome :abandon)))) (defn- ^:async run-fresh-slots - [state case slots] + [state case slots deadline] (if-let [[warmup? & pending] (seq slots)] - (let [outcome (await (run-fresh-page state case warmup? (empty? pending)))] - (if (#{:ok :failed} outcome) - (run-fresh-slots state case pending) + (let [outcome (await (run-fresh-page state case warmup? (empty? pending) deadline))] + (if (= :ok outcome) + (run-fresh-slots state case pending deadline) outcome)) :done)) (defn- ^:async run-fresh-case - [state case] - (let [outcome (await (run-fresh-slots state case (case-slots (:opts state))))] + [state case deadline] + (let [outcome (await (run-fresh-slots state case (case-slots (:opts state)) deadline))] (ensure-screenshot-status state case :case-abandoned) outcome)) (defn- ^:async run-reuse-slots - [state case page preparation-id slots] + [state case page preparation-id slots deadline] (if-let [[warmup? & pending] (seq slots)] - (let [deadline (process/deadline (:timeout-ms (:opts state))) - outcome (await (execute-slot state case page preparation-id warmup? deadline))] - (await (finish-slot state case page outcome (empty? pending))) + (let [_ (check-deadline deadline) + executed (await (execute-slot state case page preparation-id warmup? deadline)) + outcome (await (finish-slot state case page executed (empty? pending) deadline))] (if (= :ok outcome) - (run-reuse-slots state case page preparation-id pending) + (run-reuse-slots state case page preparation-id pending deadline) outcome)) :done)) (defn- ^:async abandon-preparation - [state case evidence] + [state case evidence deadline] (record-failure state (:id case) :preparation evidence) (ensure-screenshot-status state case :no-completed-measured-attempt) - (await (close-current-page state)) - (if (interrupted? state) :interrupted :abandon)) + (failure-outcome state deadline evidence)) (defn- ^:async run-reuse-prepared - [state case page preparation] - (let [accepted (accept-preparation state preparation) + [state case page preparation deadline] + (let [accepted (check-gpu state preparation) outcome (if (= :abort accepted) :abort (await (run-reuse-slots state case page (:preparation-id preparation) - (case-slots (:opts state))))) - closed? (await (close-current-page state))] + (case-slots (:opts state)) deadline)))] (ensure-screenshot-status state case :case-abandoned) - (if closed? outcome :abandon))) + (if (= :done outcome) + (if (await (close-current-page state case)) outcome :failed) + outcome))) (defn- ^:async run-reuse-page - [state case page] - (let [deadline (process/deadline (:timeout-ms (:opts state))) - prepared (await (prepare-case state case page deadline false))] + [state case page deadline] + (let [prepared (await (prepare-case state case page deadline false))] (if-let [evidence (:failure prepared)] - (await (abandon-preparation state case evidence)) - (await (run-reuse-prepared state case page (:preparation prepared)))))) + (await (abandon-preparation state case evidence deadline)) + (await (run-reuse-prepared state case page (:preparation prepared) deadline))))) (defn- ^:async run-reuse-case - [state case] - (let [{:keys [page outcome]} (await (open-page state case))] + [state case deadline] + (let [{:keys [page outcome]} (await (open-page state case deadline))] (cond - page (await (run-reuse-page state case page)) + page (await (run-reuse-page state case page deadline)) (= :abort outcome) :abort :else @@ -471,9 +518,8 @@ false) (defn- ^:async launch-initial - [state] - (let [owner (attempt (await (ensure-browser state)))] - (reset! (:launch-deadline state) nil) + [state deadline] + (let [owner (attempt (await (ensure-browser state deadline)))] (if (failure? owner) (fail-startup state :browser-launch owner) true))) @@ -508,15 +554,57 @@ [state] (and (not @(:abort-run? state)) (not (interrupted? state)))) +(defn- get-case-failure + [run case-id] + (or (some :failure (reverse (filter #(= case-id (:case-id %)) (:attempts run)))) + (some :failure (reverse (filter #(= case-id (:case-id %)) (:preparations run)))) + (some #(when (= case-id (:case-id %)) %) + (reverse (get-in run [:metadata :diagnostics :failures]))))) + +(defn- ^:async run-case + [state case deadline] + (let [executed (attempt + (if (= :fresh (:context case)) + (await (run-fresh-case state case deadline)) + (await (run-reuse-case state case deadline)))) + outcome (if (failure? executed) + (let [evidence (cause-evidence executed :case)] + (record-failure state (:id case) :case evidence) + (failure-outcome state deadline evidence)) + executed) + unmeasured? (and (= :done outcome) + (not (result/has-measurements? @(:run-ref state) (:id case)))) + outcome (if unmeasured? + (do + (record-failure state (:id case) :case + {:phase "case" :reason :no-measurements + :message "case has no measurements"}) + :failed) + outcome) + saved-outcome (cond + (= :done outcome) :completed + (= :interrupted outcome) :interrupted + (= :timed-out outcome) :timed-out + :else :failed) + failure (when (not= :completed saved-outcome) + (get-case-failure @(:run-ref state) (:id case)))] + (swap! (:run-ref state) result/record-case-outcome (:id case) + (cond-> {:outcome saved-outcome} + failure (assoc :failure failure))) + (ensure-screenshot-status state case :case-abandoned) + (persist state) + (when (not= :done outcome) + (await (recycle-browser state))) + outcome)) + (defn- ^:async run-cases [state cases] (when (and (seq cases) (session-active? state)) - (let [case (first cases)] + (let [case (first cases) + deadline (process/deadline (:timeout-ms (:opts state)))] (write-log state log/info (str "▸ Case " (result/case-name (:id case)) " (" (name (:context case)) ")")) - (let [outcome (if (= :fresh (:context case)) - (await (run-fresh-case state case)) - (await (run-reuse-case state case)))] + (let [outcome (await (run-case state case deadline))] (if (#{:abort :interrupted} outcome) (do (reset! (:abort-run? state) true) @@ -525,13 +613,12 @@ (defn- ^:async run-session [state] - (let [deadline (process/deadline (:startup-timeout-ms (:opts state)))] - (reset! (:launch-deadline state) deadline) - (let [provenance-ready? (await (record-provenance state deadline)) - server-ready? (and provenance-ready? (await (start-server state deadline))) - browser-ready? (and server-ready? (session-active? state) (await (launch-initial state)))] - (when (and browser-ready? (session-active? state)) - (await (run-cases state (:cases state))))))) + (let [deadline (process/deadline (:startup-timeout-ms (:opts state))) + provenance-ready? (await (record-provenance state deadline)) + server-ready? (and provenance-ready? (await (start-server state deadline))) + browser-ready? (and server-ready? (session-active? state) (await (launch-initial state deadline)))] + (when (and browser-ready? (session-active? state)) + (await (run-cases state (:cases state)))))) (defn- get-termination [run interrupted harness-failed?] @@ -539,7 +626,8 @@ (and (zero? (:warmups slots)) (zero? (:repetitions slots)))) (:remaining run)) - failed? (some #(= :failed (:outcome %)) (:attempts run)) + failed? (or (some #(= :failed (:outcome %)) (:attempts run)) + (some #(not= :completed (:outcome %)) (vals (:case-outcomes run)))) reason (cond interrupted :interrupted all-attempted? (if failed? :failed :completed) @@ -571,11 +659,7 @@ (defn- finish-run [state] (let [termination (get-termination @(:run-ref state) (interruption (:signal state)) - @(:harness-failed? state)) - unmeasured (result/get-unmeasured-cases @(:run-ref state)) - termination (if (and (= :completed (:reason termination)) (seq unmeasured)) - {:reason :no-measurements :case-ids unmeasured} - termination)] + @(:harness-failed? state))] (when (:screenshots (:opts state)) (swap! (:run-ref state) update-in [:metadata :diagnostics :screenshots] complete-screenshots (:cases state))) @@ -665,7 +749,7 @@ :plan {:cases cases :warmups (:warmups opts) :repetitions (:repetitions opts) - :timeout-policy {:attempt-ms (:timeout-ms opts)}} + :timeout-policy {:case-ms (:timeout-ms opts)}} :metadata (metadata root selected-build opts)})) (defn- create-run-state @@ -681,7 +765,6 @@ :owner-ref (atom nil) :page-ref (atom nil) :host-ref (atom nil) - :launch-deadline (atom nil) :abort-run? (atom false) :harness-failed? (atom false) :services (merge browser/service (:browser opts))}) diff --git a/frontend/performance/benches/render_wasm/report/format.cljs b/frontend/performance/benches/render_wasm/report/format.cljs index e383fb2ea4..dcb82bd870 100644 --- a/frontend/performance/benches/render_wasm/report/format.cljs +++ b/frontend/performance/benches/render_wasm/report/format.cljs @@ -61,8 +61,9 @@ (defn- case-status->text "Displays pending slots and failed attempts, including excluded warmups." - [{:keys [remaining failed-attempts preparations phases]}] + [{:keys [remaining failed-attempts preparations phases outcome]}] (str " remaining " (pr-str remaining) + (when outcome (str "\n outcome " (pr-str outcome))) "\n phases " (pr-str phases) " failed-warmups=" (count (filter :warmup? failed-attempts)) (when-let [failures (seq (keep :failure preparations))] diff --git a/frontend/performance/benches/render_wasm/report/summarize.cljs b/frontend/performance/benches/render_wasm/report/summarize.cljs index f5f5612a3b..e6c1e8e0c9 100644 --- a/frontend/performance/benches/render_wasm/report/summarize.cljs +++ b/frontend/performance/benches/render_wasm/report/summarize.cljs @@ -210,6 +210,7 @@ [run id ids analysis] {:case-id id :definition (some #(when (= id (:id %)) %) (get-in run [:plan :cases])) + :outcome (get-in run [:case-outcomes id]) :preparations (filterv #(= id (:case-id %)) (:preparations run)) :phases (phase-counts run id) :remaining (get-in run [:remaining id]) diff --git a/frontend/performance/benches/render_wasm/result.cljs b/frontend/performance/benches/render_wasm/result.cljs index 1e5b118197..09c68c4c24 100644 --- a/frontend/performance/benches/render_wasm/result.cljs +++ b/frontend/performance/benches/render_wasm/result.cljs @@ -76,6 +76,11 @@ "A run's current state or final reason, with optional diagnostic details." [:map [:reason keyword?]]) +(def schema:case-outcome + [:map {:closed true} + [:outcome [:enum :completed :failed :timed-out :interrupted]] + [:failure {:optional true} map?]]) + (def schema:run "schema:run stores saved case definitions, independent of any registry. Build :functional holds compatibility settings; :provenance holds identities." @@ -88,12 +93,13 @@ [:cases [:vector [:map [:id some?]]]] [:warmups [:int {:min 0}]] [:repetitions [:int {:min 0}]] - [:timeout-policy [:map [:attempt-ms [:int {:min 1}]]]]]] + [:timeout-policy [:map {:closed true} [:case-ms [:int {:min 1}]]]]]] [:metadata schema:metadata] [:analysis analysis/schema:settings] [:quantities [:map-of qualified-keyword? :any]] [:preparations [:vector schema:preparation]] [:attempts [:vector schema:attempt]] + [:case-outcomes [:map-of :any schema:case-outcome]] [:remaining [:map-of :any [:map {:closed true} [:warmups [:int {:min 0}]] [:repetitions [:int {:min 0}]]]]] @@ -141,6 +147,7 @@ "One saved case's definition, preparations and shared metric reports." [:map {:closed true} [:case-id some?] [:definition [:map [:id some?]]] + [:outcome [:maybe schema:case-outcome]] [:preparations [:vector schema:preparation]] [:phases [:map-of :keyword [:map {:closed true} [:executions nat-int?] [:completed nat-int?] @@ -276,6 +283,21 @@ "measured attempt precedes planned warmups" {:case-id case-id}) (update-in remaining path dec))) +(defn has-measurements? + "Reports whether a case has measured commands in its saved phase histories. + + Args: + run: constructed or decoded run record. + case-id: exact saved case identifier. + + Returns: + True when any preparation, main or reset block contains a command event." + [run case-id] + (boolean + (some (comp seq :events) + (mapcat :phases (filter #(= case-id (:case-id %)) + (concat (:preparations run) (:attempts run))))))) + (defn get-unmeasured-cases "Returns saved case ids whose phase histories contain no measured commands. @@ -287,9 +309,7 @@ [run] (into [] (keep (fn [{:keys [id]}] - (when-not (some (comp seq :events) - (mapcat :phases (filter #(= id (:case-id %)) - (concat (:preparations run) (:attempts run))))) + (when-not (has-measurements? run id) id))) (get-in run [:plan :cases]))) @@ -303,6 +323,10 @@ "completed run has failed attempts" {}) (require-valid! (not-any? :failure (:preparations run)) "completed run has failed preparations" {}) + (require-valid! (every? #(= :completed (:outcome %)) (vals (:case-outcomes run))) + "completed run has failed cases" {}) + (require-valid! (= (set (map :id (get-in run [:plan :cases]))) (set (keys (:case-outcomes run)))) + "completed run needs every case outcome" {}) (require-valid! (empty? (get-unmeasured-cases run)) "completed case has no measurements" {:case-ids (get-unmeasured-cases run)}))) @@ -350,6 +374,18 @@ (when (= :failed (get-in attempt [:phases 0 :outcome])) (require-valid! (= 1 (count (:phases attempt))) "reset follows failed main" {}))) +(defn- check-case-outcome + [run case-id outcome] + (require-valid! (contains? (:remaining run) case-id) "unknown case outcome" {}) + (require-valid! (sm/validate schema:case-outcome outcome) "invalid case outcome" {}) + (when (= :completed (:outcome outcome)) + (require-valid! (and (not (:failure outcome)) + (every? zero? (vals (get-in run [:remaining case-id]))) + (not-any? #(and (= case-id (:case-id %)) (= :failed (:outcome %))) (:attempts run)) + (not-any? #(and (= case-id (:case-id %)) (:failure %)) (:preparations run)) + (has-measurements? run case-id)) + "completed case has pending, failed or unmeasured work" {:case-id case-id}))) + (defn check-run "Validates schema:run and chronological accounting at a saved-data boundary." [run] @@ -363,6 +399,8 @@ (require-valid! (= (count prep-ids) (count (set prep-ids))) "duplicate preparation" {}) (require-valid! (every? #(contains? (set case-ids) %) (map :case-id preps)) "unknown preparation case" {}) (require-valid! (= left (:remaining run)) "remaining slots disagree with attempts" {}) + (doseq [[case-id outcome] (:case-outcomes run)] + (check-case-outcome run case-id outcome)) (doseq [prep preps] (check-preparation prep)) (doseq [[attempt-id attempt] (map-indexed vector (:attempts run))] @@ -373,16 +411,16 @@ (defn create-run "Creates an empty raw run from externally supplied identities and facts. - The plan defaults to 3 warmups, 10 repetitions and a 30000 ms attempt timeout. + The plan defaults to 3 warmups, 10 repetitions and a 30000 ms case timeout. For each case, all warmup slots precede measured slots. Failed attempts consume their slot too; attempts retain chronological order across cases." [{:keys [run-id started-at plan metadata analysis quantities]}] - (let [plan (merge {:warmups 3 :repetitions 10 :timeout-policy {:attempt-ms 30000}} plan)] + (let [plan (merge {:warmups 3 :repetitions 10 :timeout-policy {:case-ms 30000}} plan)] (check-run {:type :run-record :run-id run-id :started-at started-at :ended-at nil :plan plan :metadata metadata :analysis (analysis/resolve-settings analysis) :quantities (merge quantities/descriptions quantities) - :preparations [] :attempts [] :remaining (initial-remaining plan) + :preparations [] :attempts [] :case-outcomes {} :remaining (initial-remaining plan) :termination {:reason :running}}))) (defn- check-open! @@ -395,6 +433,7 @@ The run must come from create-run or decode; earlier evidence is trusted." [run preparation] (check-open! run) + (require-valid! (not (contains? (:case-outcomes run) (:case-id preparation))) "case already terminated" {}) (check-preparation preparation) (require-valid! (contains? (:remaining run) (:case-id preparation)) "unknown preparation case" {}) (require-valid! (not-any? #(= (:preparation-id preparation) (:preparation-id %)) @@ -409,6 +448,7 @@ Earlier attempts are trusted; encode and decode check the complete record." [run attempt] (check-open! run) + (require-valid! (not (contains? (:case-outcomes run) (:case-id attempt))) "case already terminated" {}) (let [attempt (-> (merge {:preparation-id nil :phases []} attempt) (assoc :attempt-id (count (:attempts run))))] (check-attempt attempt (some #(when (= (:preparation-id attempt) (:preparation-id %)) %) @@ -426,6 +466,25 @@ (check-completion run termination) (assoc run :ended-at ended-at :termination termination)) +(defn record-case-outcome + "Records the terminal outcome of one case. + + Args: + run: constructed or decoded run that accepts new evidence. + case-id: exact saved case identifier. + outcome: completion, failure, timeout or interruption with available cause. + + Returns: + The run with its case outcome. + + Throws: + ExceptionInfo for unknown cases, duplicate outcomes or malformed evidence." + [run case-id outcome] + (check-open! run) + (require-valid! (not (contains? (:case-outcomes run) case-id)) "duplicate case outcome" {}) + (check-case-outcome run case-id outcome) + (assoc-in run [:case-outcomes case-id] outcome)) + (defn merge-runs "Returns a new validated run that appends `additional`'s measured attempts to `base`. @@ -514,6 +573,7 @@ :quantities (:quantities base) :preparations (into (vec base-preparations) additional-preparations) :attempts attempts + :case-outcomes (merge (:case-outcomes base) (:case-outcomes additional)) :remaining remaining :termination {:reason :completed}}))))) diff --git a/frontend/test/performance_tests/render_wasm/cli_integration.cjs b/frontend/test/performance_tests/render_wasm/cli_integration.cjs index 82e47e70e6..a411e58f0c 100644 --- a/frontend/test/performance_tests/render_wasm/cli_integration.cjs +++ b/frontend/test/performance_tests/render_wasm/cli_integration.cjs @@ -550,7 +550,7 @@ function runFixtureText({ k("repetitions"), repetitions, k("timeout-policy"), - map([k("attempt-ms"), 30000]), + map([k("case-ms"), 30000]), ]), k("metadata"), map([ @@ -591,6 +591,8 @@ function runFixtureText({ [preparation], k("attempts"), attempts, + k("case-outcomes"), + map([k(caseId), map([k("outcome"), k("completed")])]), k("remaining"), map([k(caseId), map([k("warmups"), 0, k("repetitions"), 0])]), k("termination"), diff --git a/frontend/test/performance_tests/render_wasm/cli_test.cljs b/frontend/test/performance_tests/render_wasm/cli_test.cljs index e930163144..c001463051 100644 --- a/frontend/test/performance_tests/render_wasm/cli_test.cljs +++ b/frontend/test/performance_tests/render_wasm/cli_test.cljs @@ -267,6 +267,8 @@ ["run" #"--warmups N\s+3"] ["run" #"--seed N\s+42"] ["run" #"--repetitions N\s+10"] + ["run" #"--timeout-ms MS\s+30000\s+Case deadline"] + ["ab" #"--timeout-ms MS\s+30000\s+Case deadline"] ["run" #"--features FEATURES\s+\[\]"] ["run" #"--\[no-\]default-features\s+true"] ["run" #"--build-timeout-ms MS\s+300000"] diff --git a/frontend/test/performance_tests/render_wasm/compare_test.cljs b/frontend/test/performance_tests/render_wasm/compare_test.cljs index 78af5e74e6..4281bad296 100644 --- a/frontend/test/performance_tests/render_wasm/compare_test.cljs +++ b/frontend/test/performance_tests/render_wasm/compare_test.cljs @@ -56,7 +56,7 @@ :plan {:cases [(merge (case-definition case-id scene) case-settings)] :warmups 0 :repetitions 1 - :timeout-policy {:attempt-ms 30000}} + :timeout-policy {:case-ms 30000}} :metadata {:git {:sha run-id :dirty false} :build {:functional functional} :environment environment @@ -79,7 +79,9 @@ :outcome :completed :phases [{:phase :main :preparation-id preparation-id :outcome :completed :events [(metric value)]}]})] - (result/finish-run run "2026-10-26T00:01:00.000Z" {:reason :completed}))) + (-> run + (result/record-case-outcome case-id {:outcome :completed}) + (result/finish-run "2026-10-26T00:01:00.000Z" {:reason :completed})))) (defn- write-run [file run] diff --git a/frontend/test/performance_tests/render_wasm/feature_comparison_test.cljs b/frontend/test/performance_tests/render_wasm/feature_comparison_test.cljs index 33f57856d5..ea4356d89e 100644 --- a/frontend/test/performance_tests/render_wasm/feature_comparison_test.cljs +++ b/frontend/test/performance_tests/render_wasm/feature_comparison_test.cljs @@ -163,7 +163,7 @@ (t/is (empty? (fs/readdirSync (frontend-root)))) (let [text (feature-comparison/format-plan ready)] (doseq [part ["a, b" "00" "01" "11" "10" "00 -> 10: a" "Cases: 1" - "0 warmups, 1 measured" "attempt 2000, build 10000, startup 3000"]] + "0 warmups, 1 measured" "case 2000, build 10000, startup 3000"]] (t/is (str/includes? text part)))))) (t/deftest ^:async metadata-selects-the-package-by-manifest-not-array-position @@ -472,7 +472,7 @@ :plan {:cases cases :warmups 0 :repetitions 1 - :timeout-policy {:attempt-ms (:timeout-ms options)}} + :timeout-policy {:case-ms (:timeout-ms options)}} :metadata {:git {:sha nil :dirty nil} :build {:functional (get-in selected-build [:manifest :functional]) :provenance (get-in selected-build [:manifest :provenance])} @@ -509,7 +509,10 @@ :failed {:reason :failed :failure {:message "fixture failure"}} :zero-attempts {:reason :incomplete} :interrupted {:reason :interrupted :signal "SIGINT"} - {:reason :completed})] + {:reason :completed}) + run (if (= :completed (:reason termination)) + (reduce #(result/record-case-outcome %1 (:id %2) {:outcome :completed}) run cases) + run)] (result/finish-run run (.toISOString (js/Date.)) termination))) (defn- execute-service diff --git a/frontend/test/performance_tests/render_wasm/lifecycle_test.cljs b/frontend/test/performance_tests/render_wasm/lifecycle_test.cljs index 3a30248e7a..cb2ab2e3fd 100644 --- a/frontend/test/performance_tests/render_wasm/lifecycle_test.cljs +++ b/frontend/test/performance_tests/render_wasm/lifecycle_test.cljs @@ -623,6 +623,29 @@ (finally (cleanup))))))) +(t/deftest ^:async reused-preparation-keeps-its-original-case-deadline + (await + (with-entry-env + (^:async fn [cleanup] + (try + (unchecked-set js/globalThis "__benchNow" 0) + (register-phase-case {:context :reuse :preparation {:render :none}} camera/render-sync*) + (let [request (-> (scene-args 7 {}) transit/decode-str + (assoc :timeout-ms 100) transit/encode-str) + prepared (transit/decode-str (await (browser/prepare-scene-request request))) + id (get-in prepared [:preparation :preparation-id])] + (unchecked-set js/globalThis "__benchNow" 60) + (let [first (transit/decode-str (await (browser/run-prepared + (transit/encode-str {:preparation-id id :timeout-ms 10000}))))] + (t/is (= "ok" (:status first)))) + (unchecked-set js/globalThis "__benchNow" 101) + (let [expired (transit/decode-str (await (browser/run-prepared + (transit/encode-str {:preparation-id id}))))] + (t/is (= "failed" (:status expired))) + (t/is (= "deadline" (:phase expired))))) + (finally + (cleanup))))))) + (t/deftest ^:async fresh-contexts-prepare-and-reset-once (await (with-entry-env diff --git a/frontend/test/performance_tests/render_wasm/merge_test.cljs b/frontend/test/performance_tests/render_wasm/merge_test.cljs index 87d912c30b..7d3dd74442 100644 --- a/frontend/test/performance_tests/render_wasm/merge_test.cljs +++ b/frontend/test/performance_tests/render_wasm/merge_test.cljs @@ -62,7 +62,7 @@ :plan {:cases (mapv case-definition cases) :warmups warmups :repetitions repetitions - :timeout-policy {:attempt-ms timeout-ms}} + :timeout-policy {:case-ms timeout-ms}} :metadata {:git {:sha git-sha :dirty false} :build {:functional build} :environment environment @@ -87,7 +87,10 @@ (result/record-attempt run (attempt case-id measured-id false))) run (range repetitions)))))) - run cases)] + run cases) + run (if (= :completed (:reason termination)) + (reduce #(result/record-case-outcome %1 %2 {:outcome :completed}) run cases) + run)] (result/finish-run run ended-at termination))) (t/deftest merges-measured-attempts-and-updates-the-plan diff --git a/frontend/test/performance_tests/render_wasm/report_test.cljs b/frontend/test/performance_tests/render_wasm/report_test.cljs index dc5887a5c2..6ca2469676 100644 --- a/frontend/test/performance_tests/render_wasm/report_test.cljs +++ b/frontend/test/performance_tests/render_wasm/report_test.cljs @@ -155,6 +155,7 @@ (add-attempt :prep true :completed phases) (add-attempt :prep false :completed phases) (add-attempt :prep false :completed phases) + (result/record-case-outcome :saved/pan {:outcome :completed}) (result/finish-run "end" {:reason :completed}) result/encode result/decode) report (first (:cases (summary/summarize run))) @@ -281,6 +282,7 @@ (add-attempt :prep false :completed [(block :main :prep []) (block :reset :prep (if (= :reset phase) [(event :test/work [2])] []))]) + (result/record-case-outcome :saved/pan {:outcome :completed}) (result/finish-run "end" {:reason :completed}) result/encode result/decode)] (t/is (= [2] (:values (summary/observations run :saved/pan [phase :global :command/elapsed])))))) @@ -431,3 +433,24 @@ (assoc-in run [:remaining :saved/pan :repetitions] 2)]] (t/is (= ::result/invalid-record (error-type #(result/decode (transit/encode-str bad)))))))) + +(t/deftest case-outcomes-validate-completion-and-close-the-recording-boundary + (let [run (sample-run [1]) + completed (result/record-case-outcome run :saved/pan {:outcome :completed}) + ended (result/finish-run completed "end" {:reason :completed})] + (t/is (= ended (result/decode (result/encode ended)))) + (doseq [bad [(assoc ended :case-outcomes {}) + (-> run + (assoc-in [:plan :repetitions] 2) + (assoc-in [:remaining :saved/pan :repetitions] 1) + (assoc-in [:case-outcomes :saved/pan] {:outcome :completed})) + (assoc-in ended [:case-outcomes :unknown] {:outcome :completed})]] + (t/is (= ::result/invalid-record (error-type #(result/decode (transit/encode-str bad)))))) + (t/is (= ::result/invalid-record + (error-type #(result/record-preparation completed (preparation :other []))))) + (let [stopped (-> run + (assoc-in [:plan :repetitions] 2) + (assoc-in [:remaining :saved/pan :repetitions] 1) + (result/record-case-outcome :saved/pan {:outcome :failed}))] + (t/is (thrown-with-msg? js/Error #"case already terminated" + (add-attempt stopped :prep false :completed [(block :main :prep [])])))))) diff --git a/frontend/test/performance_tests/render_wasm/run_test.cljs b/frontend/test/performance_tests/render_wasm/run_test.cljs index 5d4f59ec7f..4033eda45a 100644 --- a/frontend/test/performance_tests/render_wasm/run_test.cljs +++ b/frontend/test/performance_tests/render_wasm/run_test.cljs @@ -14,6 +14,7 @@ [benches.render-wasm.features.plan :as feature-plan] [benches.render-wasm.node.cli.request :as request] [benches.render-wasm.node.log :as log] + [benches.render-wasm.node.runner.browser :as browser] [benches.render-wasm.node.runner.feature-comparison :as feature-comparison] [benches.render-wasm.node.runner.run :as run] [benches.render-wasm.node.storage.artifacts :as artifacts] @@ -87,7 +88,7 @@ (defn- fake-browser [{:keys [launch-failure new-page-failure close-page-failure prepare-failure run-failure screenshot-failure - interruption controller effective-graphics events] + interruption controller effective-graphics events close-failure] :or {effective-graphics (constantly {}) events [{:command :fixture/work :metrics {:command/elapsed [1]}}]}}] (let [state (atom {:launches 0 :pages 0 :close-pages 0 :closes 0 @@ -100,9 +101,9 @@ (swap! state assoc :launches n) (when (and launch-failure (launch-failure n)) (throw (ex-info "browser launch failed" {:phase :browser-startup}))) - {:version "FakeBrowser/1" :browser #js {}})) + {:id n :version "FakeBrowser/1" :browser #js {}})) :new-page - (fn [_ host _] + (fn [owner host _] (when-not (and (string? (:base-url host)) (some? (get-in host [:build :id]))) (throw (ex-info "new-page requires the opened host" @@ -112,7 +113,7 @@ (swap! state assoc :pages n) (when (and new-page-failure (new-page-failure n)) (throw (ex-info "browser page failed" {:phase :browser-page-startup}))) - {:page {:id n} :context {}})) + {:page {:id n :browser-id (:id owner)} :context {}})) :close-page (fn [_] (let [n (next-count state :close-pages)] @@ -121,7 +122,9 @@ (throw (ex-info "page close failed" {:phase :browser-page-close}))))) :close (fn [_] - (swap! state update :closes inc)) + (let [n (:closes (swap! state update :closes inc))] + (when (and close-failure (close-failure n)) + (throw (ex-info "browser shutdown failed" {:phase :browser-close}))))) :connected? (fn [_] true) :prepare @@ -143,7 +146,7 @@ :fingerprint {:algorithm :sha-256 :digest (apply str (repeat 64 "a"))} :effective-graphics (effective-graphics case-id (:id page)) - :diagnostics {} + :diagnostics {:browser-id (:browser-id page)} :phases [{:phase :preparation :preparation-id preparation-id :outcome :completed :events []}]}] (swap! state assoc-in [:preparations preparation-id] case-id) @@ -211,7 +214,8 @@ (options fake (path/join @temporary-root "unmeasured")))) saved (result/decode (fs/readFileSync (get-in response [:home :record-path]) "utf8"))] (t/is (= 1 (:exit-code response))) - (t/is (= {:reason :no-measurements :case-ids [:case/unmeasured]} (:termination saved))) + (t/is (= :failed (get-in saved [:termination :reason]))) + (t/is (= :no-measurements (get-in saved [:case-outcomes :case/unmeasured :failure :reason]))) (t/is (= :completed (get-in saved [:attempts 0 :outcome]))))) (defn- preparations-for @@ -310,7 +314,7 @@ (t/is (= 2 (:launches state))) (t/is (= 2 (:closes state)))))) -(t/deftest ^:async failed-attempts-consume-slots-and-the-runner-continues +(t/deftest ^:async failed-attempts-stop-the-case-and-keep-pending-slots (doseq [failure [:throw :response]] (let [fake (fake-browser {:run-failure (fn [case-id n] (when (and (= case-id :case/fresh) (= n 1)) @@ -319,12 +323,12 @@ result (await (execute fake [(collected :case/fresh :fresh)] opts)) run (:run result) tries (attempts-for run :case/fresh)] - (t/is (= 2 (count tries))) - (t/is (= [:failed :completed] (mapv :outcome tries))) - (t/is (every? zero? (vals (get-in run [:remaining :case/fresh])))) - (t/is (= :failed (get-in run [:termination :reason]))) + (t/is (= [:failed] (mapv :outcome tries))) + (t/is (= {:warmups 0 :repetitions 1} (get-in run [:remaining :case/fresh]))) + (t/is (= :failed (get-in run [:case-outcomes :case/fresh :outcome]))) + (t/is (= :incomplete (get-in run [:termination :reason]))) (t/is (= 1 (:exit-code result))) - (t/is (= 2 (count (preparations-for run :case/fresh))))))) + (t/is (= 1 (count (preparations-for run :case/fresh))))))) (t/deftest ^:async reuse-preparation-failure-leaves-slots-and-continues-to-the-next-case (doseq [kind [:throw :response]] @@ -346,7 +350,7 @@ (t/is (= :case/reuse (:case-id failure))) (t/is (= "scene-build" (:phase failure))))))) -(t/deftest ^:async fresh-preparation-failure-consumes-one-slot-and-retries +(t/deftest ^:async fresh-preparation-failure-consumes-its-slot-and-stops-the-case (doseq [kind [:throw :response]] (let [preparations (atom 0) fake (fake-browser {:prepare-failure #(when (and (= % :case/fresh) @@ -356,14 +360,14 @@ (assoc (options fake (output-dir)) :repetitions 2 :screenshots true))) run (:run result) tries (attempts-for run :case/fresh)] - (t/is (= [:failed :completed] (mapv :outcome tries))) + (t/is (= [:failed] (mapv :outcome tries))) (t/is (= (:preparation-id (first (preparations-for run :case/fresh))) (:preparation-id (first tries)))) (t/is (= "scene-build" (get-in (first tries) [:failure :phase]))) - (t/is (= 2 (count (preparations-for run :case/fresh)))) - (t/is (= :ok (get-in run [:metadata :diagnostics :screenshots "case/fresh" :status]))) - (t/is (every? zero? (vals (get-in run [:remaining :case/fresh])))) - (t/is (= :failed (get-in run [:termination :reason]))) + (t/is (= 1 (count (preparations-for run :case/fresh)))) + (t/is (= :skipped (get-in run [:metadata :diagnostics :screenshots "case/fresh" :status]))) + (t/is (= {:warmups 0 :repetitions 1} (get-in run [:remaining :case/fresh]))) + (t/is (= :incomplete (get-in run [:termination :reason]))) (t/is (= 1 (:exit-code result)))))) (t/deftest ^:async reuse-attempt-failure-leaves-pending-slots-and-continues-to-the-next-case @@ -377,7 +381,7 @@ (t/is (= [:failed] (mapv :outcome (attempts-for run :case/reuse)))) (t/is (= {:warmups 0 :repetitions 2} (get-in run [:remaining :case/reuse]))) (t/is (= [:completed :completed :completed] (mapv :outcome (attempts-for run :case/fresh)))) - (t/is (= {:status :skipped :reason :case-abandoned} + (t/is (= {:status :skipped :reason :no-completed-measured-attempt} (get-in run [:metadata :diagnostics :screenshots "case/reuse"]))) (t/is (= :incomplete (get-in run [:termination :reason]))) (t/is (= 1 (:exit-code result)))))) @@ -399,10 +403,7 @@ (t/is (= 1 (:exit-code result)))))) (t/deftest ^:async browser-close-failure-abandons-the-case-and-relaunches-for-the-next - (let [fake (fake-browser {:run-failure (fn [case-id n] - (when (and (= case-id :case/a) (= n 1)) - :throw)) - :close-page-failure #(= % 1)}) + (let [fake (fake-browser {:close-page-failure #(= % 1)}) opts (assoc (options fake (output-dir)) :repetitions 2) result (await (execute fake [(collected :case/a :fresh) (collected :case/b :fresh)] opts)) @@ -410,7 +411,8 @@ state @(:state fake)] (t/is (= 2 (:launches state))) (t/is (= 1 (count (attempts-for run :case/a)))) - (t/is (= :failed (:outcome (first (attempts-for run :case/a))))) + (t/is (= :completed (:outcome (first (attempts-for run :case/a))))) + (t/is (= :failed (get-in run [:case-outcomes :case/a :outcome]))) (t/is (= 1 (get-in run [:remaining :case/a :repetitions]))) (t/is (= 2 (count (attempts-for run :case/b)))) (t/is (every? #(= :completed (:outcome %)) (attempts-for run :case/b))) @@ -423,7 +425,7 @@ result (await (execute fake [(collected :case/a :fresh)] (options fake (output-dir)))) run (:run result)] - (t/is (= :completed (get-in run [:termination :reason]))) + (t/is (= :failed (get-in run [:termination :reason]))) (t/is (every? zero? (vals (get-in run [:remaining :case/a])))) (t/is (= :browser-close (:stage (last (get-in run [:metadata :diagnostics :failures]))))) @@ -481,7 +483,6 @@ (t/is (str/includes? (:message failure) (pr-str (:renderer graphics)))) (t/is (str/includes? (:message failure) "./manage.sh run-devenv --gpu")) (t/is (str/includes? (:message failure) "pnpm run render-bench")) - (t/is (= 1 (:close-pages state))) (t/is (= 1 (:closes state)))))) (t/deftest ^:async graphics-rejection-preserves-earlier-attempts-and-stops-the-run @@ -502,7 +503,6 @@ (t/is (= [hardware-graphics software] (mapv :effective-graphics (:preparations run)))) (t/is (= {:case/a {:warmups 0 :repetitions (if (= context :fresh) 1 0)} :case/b {:warmups 0 :repetitions 2}} (:remaining run))) - (t/is (= 2 (:close-pages @(:state fake)))) (t/is (= 1 (:closes @(:state fake))))))) (t/deftest ^:async sigint-and-sigterm-persist-partial-interrupted-records @@ -560,7 +560,265 @@ (t/is (= (if (and (= method :prepare) (= context :reuse)) [:preparation] []) (mapv :stage (get-in run [:metadata :diagnostics :failures]))))))) -(t/deftest ^:async fresh-preparation-and-rendering-share-the-attempt-time-budget +(t/deftest ^:async phase-and-output-failures-retain-evidence-and-stop-the-case + (doseq [context [:fresh :reuse] + phase [:preparation :main :reset :invalid-preparation :contradictory-preparation :invalid-main :invalid-reset]] + (let [base (fake-browser {:prepare-failure (when (= :preparation phase) + #(when (= :case/a %) :response))}) + services (:service base) + fake (assoc base :service + (assoc services + :prepare (fn [page request bounds] + (let [response ((:prepare services) page request bounds)] + (if (= :case/a (get-in (transit/decode-str request) [:case :id])) + (case phase + :invalid-preparation (update response :preparation dissoc :fingerprint) + :contradictory-preparation + (-> response + (assoc-in [:preparation :failure] {:message "callback failed"}) + (assoc-in [:preparation :phases 0 :outcome] :failed)) + response) + response))) + :run (fn [page request bounds] + (let [id (:preparation-id (transit/decode-str request)) + case-id (get-in @(:state base) [:preparations id]) + block {:phase :main :preparation-id id :outcome :completed + :events [{:command :fixture/work :metrics {:command/elapsed [1]}}]}] + (if (= :case/a case-id) + (case phase + :main {:status "failed" :phase "main" :message "main failed" + :phases [(assoc block :outcome :failed)]} + :reset {:status "failed" :phase "reset" :message "reset failed" + :phases [block (assoc block :phase :reset :outcome :failed)]} + :invalid-main {:status "ok" :phases []} + :invalid-reset {:status "ok" :phases [block {:phase :reset}]}) + ((:run services) page request bounds)))))) + response (await (execute fake [(collected :case/a context) (collected :case/b context)] + (assoc (options fake (output-dir)) :repetitions 2))) + saved (saved-record response) + prep-failed? (#{:preparation :invalid-preparation :contradictory-preparation} phase) + attempts (attempts-for saved :case/a)] + (t/is (= :failed (get-in saved [:case-outcomes :case/a :outcome]))) + (t/is (= (if (and prep-failed? (= :reuse context)) [] [:failed]) (mapv :outcome attempts))) + (t/is (= {:warmups 0 :repetitions (if (and prep-failed? (= :reuse context)) 2 1)} + (get-in saved [:remaining :case/a]))) + (t/is (= [:completed :completed] (mapv :outcome (attempts-for saved :case/b)))) + (when (= :contradictory-preparation phase) + (t/is (= "invalid-output" (get-in (first (preparations-for saved :case/a)) [:failure :phase])))) + (when (= :invalid-reset phase) + (t/is (= [:main] (mapv :phase (:phases (first attempts))))) + (t/is (= {:phase :reset} (get-in (first attempts) [:partial :response :phases 1])))) + (when (#{:invalid-main :invalid-reset} phase) + (t/is (= "invalid-output" (get-in (first attempts) [:failure :phase]))))))) + +(t/deftest ^:async runner-terminates-a-blocked-page-and-completes-the-next-case + (let [dir (build-dir) + owners (atom []) + prepared (atom {}) + attempts (atom {}) + identity-text (.stringify js/JSON (transit/encode-str {:build-id "test-build"})) + services (assoc browser/service + :launch (^:async fn [options] + (when-let [previous (last @owners)] + (t/is (false? (.isConnected ^js (:browser previous)))) + (t/is (not (fs/existsSync (:user-data-dir previous))))) + (let [owner (await (browser/launch options))] + (swap! owners conj owner) + owner)) + :prepare (fn [_ request _] + (let [{:keys [preparation-id] definition :case} (transit/decode-str request)] + (swap! prepared assoc preparation-id (:id definition)) + {:status "ok" + :preparation {:preparation-id preparation-id :case-id (:id definition) + :fingerprint {:algorithm :sha-256 :digest (apply str (repeat 64 "a"))} + :effective-graphics {} :diagnostics {} + :phases [{:phase :preparation :preparation-id preparation-id + :outcome :completed :events []}]}})) + :run (^:async fn [page request bounds] + (let [id (:preparation-id (transit/decode-str request)) + case-id (get @prepared id) + n (get (swap! attempts update case-id (fnil inc 0)) case-id)] + (await (browser/within + #(.evaluate ^js page + (if (and (= :case/a case-id) (= n 2)) + "while (true) {}" "1")) + bounds)) + {:status "ok" + :phases [{:phase :main :preparation-id id :outcome :completed + :events [{:command :fixture/work :metrics {:command/elapsed [1]}}]}]})))] + (fs/writeFileSync (path/join dir "index.html") + (str "")) + (try + (let [opts (assoc (options nil (output-dir)) :browser services :repetitions 2 + :timeout-ms 3000 :startup-timeout-ms 30000) + response (await (run/execute (artifacts/frontend-root) + (runs/create-home {:root (artifacts/frontend-root) :run-id "blocked-case" + :output-dir (:output-dir opts)}) + (selected-build dir "test-build") + [(collected :case/a :reuse) (collected :case/b :reuse)] opts nil)) + saved (saved-record response)] + (t/is (= [:completed :failed] (mapv :outcome (attempts-for saved :case/a)))) + (t/is (= :timed-out (get-in saved [:case-outcomes :case/a :outcome]))) + (t/is (= [:completed :completed] (mapv :outcome (attempts-for saved :case/b)))) + (t/is (= :completed (get-in saved [:case-outcomes :case/b :outcome]))) + (t/is (= 2 (count @owners))) + (t/is (not= (:user-data-dir (first @owners)) (:user-data-dir (second @owners)))) + (doseq [owner @owners] + (t/is (false? (.isConnected ^js (:browser owner)))) + (t/is (not (fs/existsSync (:user-data-dir owner)))))) + (finally + (doseq [owner @owners] + (await (browser/close owner))))))) + +(t/deftest ^:async opening-and-screenshot-work-share-the-case-deadline + (doseq [stage [:page :screenshot]] + (let [system-now (.-now js/Date) + clock (atom (system-now)) + base (fake-browser {}) + spend (fn [page bounds] + (when (= 1 (:id page)) + (swap! clock + 300) + (when (>= @clock (:deadline bounds)) + (throw (ex-info "browser work timed out" {:phase (name stage)}))))) + fake (if (= :page stage) + (assoc-in base [:service :new-page] + (fn [owner host bounds] + (let [opened ((get-in base [:service :new-page]) owner host bounds)] + (spend (:page opened) bounds) + opened))) + (assoc-in base [:service :screenshot] + (fn [page file bounds] + (spend page bounds) + ((get-in base [:service :screenshot]) page file bounds))))] + (try + (set! (.-now js/Date) (fn [] @clock)) + (let [response (await (execute fake [(collected :case/a :reuse) (collected :case/b :reuse)] + (assoc (options fake (output-dir)) :timeout-ms 250 :screenshots true))) + saved (saved-record response)] + (t/is (= :timed-out (get-in saved [:case-outcomes :case/a :outcome]))) + (t/is (= :completed (get-in saved [:case-outcomes :case/b :outcome]))) + (t/is (= {:warmups 0 :repetitions (if (= :page stage) 1 0)} + (get-in saved [:remaining :case/a]))) + (when (= :screenshot stage) + (t/is (= [:completed] (mapv :outcome (attempts-for saved :case/a)))))) + (finally + (set! (.-now js/Date) system-now)))))) + +(t/deftest ^:async failed-browser-disposal-stops-later-cases + (let [fake (fake-browser {:run-failure (fn [_ _] :response) :close-failure #(= 1 %)}) + response (await (execute fake [(collected :case/a :reuse) (collected :case/b :reuse)] + (assoc (options fake (output-dir)) :repetitions 2))) + saved (saved-record response)] + (t/is (= [:case/a] (mapv :case-id (:attempts saved)))) + (t/is (= {:warmups 0 :repetitions 2} (get-in saved [:remaining :case/b]))) + (t/is (= :browser-close (get-in saved [:termination :failure :stage]))))) + +(t/deftest ^:async failed-startup-cleanup-aborts-even-after-the-case-deadline + (let [system-now (.-now js/Date) + clock (atom (system-now)) + base (fake-browser {:run-failure (fn [_ _] :response)}) + fake (assoc-in base [:service :launch] + (fn [bounds] + (let [owner ((get-in base [:service :launch]) bounds)] + (when (= 2 (:id owner)) + (reset! clock (inc (:deadline bounds))) + (throw (ex-info "browser shutdown failed after startup error" + {:type ::browser/startup-cleanup}))) + owner)))] + (try + (set! (.-now js/Date) (fn [] @clock)) + (let [response (await (execute fake [(collected :case/a :reuse) (collected :case/b :reuse) + (collected :case/c :reuse)] + (assoc (options fake (output-dir)) :timeout-ms 250))) + saved (saved-record response)] + (t/is (= :failed (get-in saved [:case-outcomes :case/b :outcome]))) + (t/is (= {:warmups 0 :repetitions 1} (get-in saved [:remaining :case/c]))) + (t/is (= :browser-launch (get-in saved [:termination :failure :stage])))) + (finally + (set! (.-now js/Date) system-now))))) + +(t/deftest ^:async cleanup-time-does-not-change-the-case-failure + (let [system-now (.-now js/Date) + clock (atom (system-now)) + base (fake-browser {:new-page-failure #(= 1 %)}) + fake (assoc-in base [:service :close] + (fn [owner] + (swap! clock + 1000) + ((get-in base [:service :close]) owner)))] + (try + (set! (.-now js/Date) (fn [] @clock)) + (let [response (await (execute fake [(collected :case/a :reuse) (collected :case/b :reuse)] + (assoc (options fake (output-dir)) :timeout-ms 500))) + saved (saved-record response)] + (t/is (= :failed (get-in saved [:case-outcomes :case/a :outcome]))) + (t/is (= :completed (get-in saved [:case-outcomes :case/b :outcome])))) + (finally + (set! (.-now js/Date) system-now))))) + +(t/deftest ^:async service-cleanup-preserves-the-originating-failure-classification + (let [system-now (.-now js/Date) + clock (atom (system-now)) + base (fake-browser {:new-page-failure #(= 1 %)}) + fake (assoc-in base [:service :new-page] + (fn [owner host bounds] + (try + ((get-in base [:service :new-page]) owner host bounds) + (catch :default cause + (let [annotated (errors/with-context cause "page startup failed" + {::browser/deadline-exceeded false})] + (swap! clock + 1000) + (throw annotated))))))] + (try + (set! (.-now js/Date) (fn [] @clock)) + (let [response (await (execute fake [(collected :case/a :reuse) (collected :case/b :reuse)] + (assoc (options fake (output-dir)) :timeout-ms 250))) + saved (saved-record response)] + (t/is (= :failed (get-in saved [:case-outcomes :case/a :outcome]))) + (t/is (= :completed (get-in saved [:case-outcomes :case/b :outcome])))) + (finally + (set! (.-now js/Date) system-now))))) + +(t/deftest ^:async case-expiry-keeps-delivered-attempts-and-continues-in-a-new-browser + (doseq [context [:fresh :reuse]] + (let [system-now (.-now js/Date) + clock (atom (system-now)) + base (fake-browser {}) + active-case (atom nil) + spend (fn [duration bounds] + (when (= :case/a @active-case) + (swap! clock + duration) + (when (>= @clock (:deadline bounds)) + (throw (ex-info "browser work timed out" {:phase (:phase bounds)}))))) + services (:service base) + fake (assoc base :service + (assoc services + :prepare (fn [page request bounds] + (reset! active-case (get-in (transit/decode-str request) [:case :id])) + (spend 25 bounds) + ((:prepare services) page request bounds)) + :run (fn [page request bounds] + (spend 110 bounds) + ((:run services) page request bounds))))] + (try + (set! (.-now js/Date) (fn [] @clock)) + (let [response (await (execute fake [(collected :case/a context) (collected :case/b context)] + (assoc (options fake (output-dir)) + :warmups 1 :repetitions 3 :timeout-ms 350))) + saved (saved-record response) + first-prep (first (preparations-for saved :case/a)) + next-prep (first (preparations-for saved :case/b))] + (t/is (= [:completed :completed :failed] (mapv :outcome (attempts-for saved :case/a)))) + (t/is (= {:warmups 0 :repetitions 1} (get-in saved [:remaining :case/a]))) + (t/is (= :timed-out (get-in saved [:case-outcomes :case/a :outcome]))) + (t/is (= [:completed :completed :completed :completed] + (mapv :outcome (attempts-for saved :case/b)))) + (t/is (not= (get-in first-prep [:diagnostics :browser-id]) + (get-in next-prep [:diagnostics :browser-id])))) + (finally + (set! (.-now js/Date) system-now)))))) + +(t/deftest ^:async preparation-and-rendering-share-the-case-budget (doseq [context [:fresh :reuse]] (let [system-now (.-now js/Date) clock (atom (system-now)) @@ -582,8 +840,9 @@ (let [result (await (execute fake [(collected :case/a context)] (assoc (options fake (output-dir)) :timeout-ms 250))) saved (saved-record result)] - (t/is (= (if (= context :fresh) :failed :completed) (:outcome (first (:attempts saved))))) - (t/is (= (if (= context :fresh) 1 0) (:exit-code result)))) + (t/is (= :failed (:outcome (first (:attempts saved))))) + (t/is (= :timed-out (get-in saved [:case-outcomes :case/a :outcome]))) + (t/is (= 1 (:exit-code result)))) (finally (set! (.-now js/Date) system-now)))))) @@ -797,7 +1056,7 @@ result (await (execute fake [(collected :case/a :fresh)] opts)) run (:run result) diag (get-in run [:metadata :diagnostics])] - (t/is (= {:attempt-ms 1234} (get-in run [:plan :timeout-policy]))) + (t/is (= {:case-ms 1234} (get-in run [:plan :timeout-policy]))) (t/is (= {:sha "build-revision" :dirty false} (get-in run [:metadata :git]))) (t/is (= {:mode :release :default-features? true :features [] :cljs-profile-marks false} (get-in run [:metadata :build :functional]))) @@ -944,7 +1203,7 @@ :execute-run (fn [root home selected cases opts signal] (let [fake (fake-browser - {:close-page-failure (when close-fails #(= % 2)) + {:close-failure (when close-fails (constantly true)) :effective-graphics (fn [_ page-id] (if (and hardware-first (= page-id 1)) @@ -959,7 +1218,7 @@ (t/is (= 1 (:exit-code response))) (t/is (= (if hardware-first 1 0) (count (:attempts run)))) (t/is (= software (get-in failure [:partial :graphics]))) - (t/is (= (if close-fails :browser-close :gpu) (get-in run [:termination :failure :stage]))) + (t/is (= (if close-fails :cleanup :gpu) (get-in run [:termination :failure :stage]))) (t/is (= :failed (get-in saved [:configurations 0 :status]))) (t/is (= :gpu (get-in saved [:configurations 0 :failure :stage]))) (t/is (= :unattempted (get-in saved [:configurations 1 :status])))))) diff --git a/frontend/test/performance_tests/render_wasm/runs_test.cljs b/frontend/test/performance_tests/render_wasm/runs_test.cljs index 2199715ebf..ffcb2cf64e 100644 --- a/frontend/test/performance_tests/render_wasm/runs_test.cljs +++ b/frontend/test/performance_tests/render_wasm/runs_test.cljs @@ -37,7 +37,7 @@ :plan {:cases [{:id :rects/load}] :warmups 0 :repetitions 1 - :timeout-policy {:attempt-ms 30000}} + :timeout-policy {:case-ms 30000}} :metadata {:git {:sha "abc" :dirty false} :build {:functional {:mode :release}} :environment {:node "v24.0.0"}