github/penpot
1
0
Fork 0
mirror of https://github.com/penpot/penpot.git synced 2026-11-05 17:40:39 +00:00
Code Issues Packages Projects Releases Wiki Activity

✨ Enforce benchmark case deadlines

Browse Source 
Stop failed cases and preserve their execution evidence.

AI-assisted-by: gpt-6.1-sol
This commit is contained in:
Miguel de Benito Delgado 2026-11-04 11:17:48 +00:00
parent b8dc029da1
commit 6fa70dc47d
19 changed files with 774 additions and 233 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

96
frontend/performance/benches/render_wasm/README.md
View File

@ -106,8 +106,8 @@ The suite has three parts:
chronological order, and performs no hidden retries.
- **Run record**: the saved evidence for one run: plan, build and environment
provenance, quantity descriptions, preparations, attempts with their phase
histories, failures, remaining slots, and termination. The runner stores it as
a data file validated by a schema.
histories, failures, terminal case outcomes, remaining slots, and termination.
The runner stores it as a data file validated by a schema.
- **Bridge**: the object that runs in the browser, receives plain requests from
Node, owns the page's renderer context, runs preparations and attempts, and
returns plain results.
@ -324,11 +324,22 @@ process it starts. Cases run serially. A fresh case gets a new page and context
for each preparation. A warm case keeps one page and context for all of its
attempts.
The runner applies deadlines to startup, preparation, and attempts. When a page
stalls past its deadline, Node stops the browser process and records the failure.
The browser bridge also checks for context loss, expired deadlines, and work associated
with a superseded renderer owner before and after asynchronous work. A browser
thread occupied by synchronous work relies on the Node deadline to stop it.
Before cases run, `--startup-timeout-ms` bounds Git provenance, server startup,
and the initial browser launch. Each case then gets one absolute `--timeout-ms`
deadline. The deadline covers the case's page setup, preparation, warmups,
repetitions, reset, and screenshot diagnostics. It does not restart for a phase,
attempt, or browser replacement. Work on a reused preparation keeps its original
case deadline; a request timeout can only tighten that deadline. A browser
launched after a failed case uses the next case's remaining deadline.
When a page stalls past its deadline, Node stops the browser process and records
the failure. The browser bridge also checks for context loss, expired deadlines,
and work associated with a superseded renderer owner before and after
asynchronous work. A browser thread occupied by synchronous work relies on the
Node deadline to stop it. The runner persists each delivered preparation and
attempt. It does not stream browser-local records and cannot recover in-flight
data when a blocked browser cannot return; records delivered before the block
remain saved.
Clean-up closes the page, then the browser, then the server. The runner deletes
a build selected with `--remove` only after the final run record is saved.
@ -668,7 +679,9 @@ preparation block's selected events.
Reset runs after successful main. Its returned context supplies the next reused
attempt. A phase failure releases its owned context and returns the failed
block with earlier events and available traces. Completed preparation retains
its own evidence when main or reset fails.
its own evidence when main or reset fails. Any preparation, main, or reset
failure stops the case; the runner attempts no later slot. It disposes the
case's browser and starts the next case in a new browser when the run continues.
### Runtime code
@ -858,7 +871,7 @@ arguments. Comparing hardware and software runs requires
A run record contains:
- the plan: selected case descriptors, warmup count, repetition count, and
timeout policy;
timeout policy (`:case-ms`);
- metadata: Git revision and dirty flag, build facts, environment, and diagnostics;
- the resolved analysis settings;
- saved quantity descriptions;
@ -866,6 +879,7 @@ A run record contains:
- attempts in chronological order, each with its case, preparation,
`:attempt-id`, warmup flag, outcome, main/reset phase blocks, and failure or
partial evidence;
- terminal outcomes in `:case-outcomes`;
- remaining slots for each case;
- the termination reason.
@ -876,23 +890,44 @@ blocks. A phase block retains ordered command events, raw metric vectors, and
optional completion diagnostics. Errors retain available phase evidence and
traces after a failure.
Each case has a terminal entry in `:case-outcomes` with `:outcome` set to
`:completed`, `:failed`, `:timed-out`, or `:interrupted` and an optional
`:failure`. Recording a terminal outcome closes the case: later preparation or
attempt evidence for that case is rejected.
### Failures and termination
A failed attempt still consumes its allocated slot. A fresh preparation failure
records a failed attempt and consumes the slot. A reuse case's preparation
failure or attempt failure releases the prepared page and context and leaves
that case's remaining slots unattempted. A browser or page failure closes the
browser it started, abandons the current case, and starts a new browser for the
next case when possible. The run fixes its attempt count before execution and uses
each allocated slot at most once.
records a failed attempt and consumes the current slot. A reuse preparation
failure records no main or reset execution and consumes no main slot. Any
preparation, main, or reset failure stops the case for fresh and reuse alike,
so later slots stay unattempted. The runner disposes the failed case's browser
and starts the next case in a new browser when the run continues. A browser or
page failure follows the same case-stop rule. A failure to clean up the browser
for a failed case aborts later cases. The run fixes its attempt count before
execution and uses each allocated slot at most once.
A malformed delivered response follows the same slot rules. An attempt response
consumes the slot it invoked, keeps the valid phase prefix, and saves the raw
response as partial evidence. A malformed preparation response fails its
preparation; a fresh case still consumes the current slot.
A screenshot failure is diagnostic when it is an ordinary error. A deadline
expiry or interruption during the screenshot stops the case, while earlier
completed attempts remain saved.
A case that finishes all its slots without a measured command gets a failed
case outcome whose failure has `:reason :no-measurements`. The run then
terminates as failed or incomplete according to the remaining slots.
A run terminates in one of four states:
- **completed**: every allocated slot was attempted and every attempt completed.
- **failed**: every allocated slot was attempted and at least one attempt
failed.
- **incomplete**: slots remain because a preparation, browser, or storage
failure stopped the case.
- **completed**: every allocated slot was attempted, every case outcome is
`:completed`, and every case has at least one measurement.
- **failed**: every allocated slot was attempted and at least one attempt failed
or one case outcome is not `:completed`.
- **incomplete**: slots remain because a case stopped early or a startup,
browser, or storage failure prevented execution.
- **interrupted**: `SIGINT` or `SIGTERM` stopped the run.
The runner saves partial records before it exits. Use exit status to determine
@ -931,8 +966,10 @@ A finite zero is valid. Negative and non-finite raw values remain in the saved
record but invalidate the dependent execution sum and aggregate; independent
metric keys remain usable. Unknown or malformed quantity metadata appears as a
diagnostic and leaves that key without usable statistics. A phase can complete
with no events, but a completed run requires at least one measurement across
each case's phases.
with no events. Preparation-only and reset-only measured cases are valid. A
case that finishes all its slots without any measured command gets a failed
case outcome with `:reason :no-measurements`; the run is failed when every slot
was attempted and incomplete when slots remain.
The case report also counts actual preparation, main, and reset executions.
Completed and failed counts use each execution's own outcome, and a separate
@ -976,7 +1013,7 @@ By default, a comparison requires equal:
- effective graphics settings and runtime environment;
- build mode, effective features, instrumentation settings, and recorded build
environment;
- warmup count, timeout policy, and saved quantity units.
- warmup count, timeout policy (`:case-ms`), and saved quantity units.
Comparison pairs metric keys `[phase scope metric-type]`. Quantity description
text does not affect compatibility; changed prose is not a difference. Unknown
@ -1304,12 +1341,13 @@ mean interruption by `SIGINT` and `SIGTERM` for `build`, `run` and `ab`.
Interruption uses those statuses to report aborted work. Errors writing the
command's record or log also produce diagnostics.
`--warmups` defaults to 3, `--repetitions` to 10, `--timeout-ms` to 30000, and
`--startup-timeout-ms` to 30000. `build`, a run that prepares its build, and an
A/B experiment use a default `--build-timeout-ms` of 300000; a run with
`--build` uses the retained build's recorded provenance. The master seed
defaults to 42 and accepts a 32-bit unsigned integer. A filter matches a
substring of the case id.
`--warmups` defaults to 3, `--repetitions` to 10, `--timeout-ms` (the case
deadline) to 30000, and `--startup-timeout-ms` (host and initial browser
readiness) to 30000. `build`, a run that prepares its build, and an A/B
experiment use a default `--build-timeout-ms` of 300000; a run with `--build`
uses the retained build's recorded provenance. The master seed defaults to 42
and accepts a 32-bit unsigned integer. A filter matches a substring of the case
id.
## Unit tests

26
frontend/performance/benches/render_wasm/browser/bridge.cljs
View File

@ -104,7 +104,7 @@
[:module-url {:optional true} string?]
[:wasm-url {:optional true} string?]])
(def ^:const ^:private attempt-ms-budget
(def ^:const ^:private case-ms-budget
"Default elapsed-time deadline for setup or one attempt, checked by
`browser-hooks`. Requests can supply a timeout. The outer process timeout
covers blocked pages."
@ -128,15 +128,15 @@
:frame #(js/Promise.
(fn [resolve] (.requestAnimationFrame js/window resolve)))
:sleep #(js/Promise. (fn [resolve] (js/setTimeout resolve %)))
;; checks that this attempt still owns the page, WebGL context hasn't been
;; checks that this execution still owns the page, WebGL context hasn't been
;; lost, and the deadline hasn't passed
:check (fn []
(when @wasm/context-lost?
(throw (ex-info "WebGL context lost"
{:phase "context-lost"})))
(guard-current! epoch "protocol")
(when (> (now) deadline-ms)
(throw (ex-info "attempt deadline exceeded"
(when (>= (now) deadline-ms)
(throw (ex-info "case deadline exceeded"
{:phase "deadline"}))))
;; WASM calls
:render (fn [timestamp flags]
@ -976,7 +976,7 @@
Validation runs before the owner changes. A valid request releases the prior
owner, starts a new owner epoch, builds browser hooks with a deadline and
prepares the scene. `:preparation-id` defaults to the new epoch and
`:timeout-ms` defaults to `attempt-ms-budget`.
`:timeout-ms` defaults to `case-ms-budget`.
Args:
request: Transit string matching `schema:load-scene-args`.
@ -994,8 +994,10 @@
(js/Promise.resolve (errors/fail-data (errors/evidence "invalid-args" parsed)))
(let [{:keys [params case-desc]} parsed
{:keys [epoch] :as owner} (begin-load! case-desc)
timeout-ms (or (:timeout-ms params) attempt-ms-budget)
owner (assoc owner :hooks (browser-hooks epoch (+ (now) timeout-ms)))]
timeout-ms (or (:timeout-ms params) case-ms-budget)
deadline-ms (+ (now) timeout-ms)
owner (assoc owner :deadline-ms deadline-ms
:hooks (browser-hooks epoch deadline-ms))]
(prepare-owned-scene (assoc parsed
:owner owner
:timeout-ms timeout-ms
@ -1009,8 +1011,8 @@
Args:
preparation-id: identifier of the prepared scene to use.
timeout-ms: optional deadline for this attempt; defaults to the
preparation's timeout.
timeout-ms: optional remaining budget in milliseconds, bounded by the
preparation's case deadline.
Returns:
A Promise of status ok with `:preparation` evidence, top-level `:phases`
@ -1030,8 +1032,10 @@
(js/Promise.resolve (errors/fail-data {:phase "invalid-args"
:message "preparation unavailable, busy or already used"}))
(let [epoch (:epoch owner)
timeout-ms (or timeout-ms (:timeout-ms prepared))
owner (assoc owner :hooks (browser-hooks epoch (+ (now) timeout-ms)))]
deadline-ms (if timeout-ms
(min (:deadline-ms owner) (+ (now) timeout-ms))
(:deadline-ms owner))
owner (assoc owner :hooks (browser-hooks epoch deadline-ms))]
(swap! prepared* assoc :busy? true :used? true)
(-> (run-case owner case-desc local-case graphics context)
(.then (fn [evidence]

30
frontend/performance/benches/render_wasm/measurement.cljs
View File

@ -5,7 +5,33 @@
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns benches.render-wasm.measurement
"Projects browser phase evidence into saved attempts.")
"Projects browser phase evidence into saved attempts."
(:require
[app.common.schema :as sm]
[benches.render-wasm.result :as result]))
(defn partial-phases
"Returns the valid prefix of delivered phase history.
Args:
phases: raw blocks from a browser response.
preparation-id: expected owner of the blocks.
order: allowed phase names in execution order.
Returns:
Valid blocks up to the first malformed block or failed phase."
[phases preparation-id order]
(loop [phases (when (sequential? phases) (seq phases)) order order out []]
(if-let [phase (first phases)]
(if (and (sm/validate result/schema:phase phase)
(= preparation-id (:preparation-id phase))
(= (first order) (:phase phase))
(every? #(= 1 (count (get-in % [:metrics :command/elapsed]))) (:events phase)))
(let [out (conj out phase)]
(if (= :failed (:outcome phase)) out
(recur (next phases) (next order) out)))
out)
out)))
(defn attempt
"Returns an attempt with raw phase histories and available failure evidence.
@ -21,7 +47,7 @@
[case-id preparation-id warmup? evidence]
(cond-> {:case-id case-id :preparation-id preparation-id :warmup? warmup?
:outcome (if (= "ok" (:status evidence)) :completed :failed)
:phases (filterv #(#{:main :reset} (:phase %)) (:phases evidence))}
:phases (or (:phases evidence) [])}
(not= "ok" (:status evidence))
(assoc :failure (select-keys evidence [:status :phase :message :cause]))
(:partial evidence) (assoc :partial (:partial evidence))))

2
frontend/performance/benches/render_wasm/node/cli/shared.cljs
View File

@ -80,7 +80,7 @@
"must be a nonnegative JavaScript-safe integer")
(number-option "--repetitions N" "Measured attempts per case" schema:positive-count
"must be a positive JavaScript-safe integer")
(number-option "--timeout-ms MS" "Attempt deadline" schema:timeout-ms
(number-option "--timeout-ms MS" "Case deadline including setup, attempts and diagnostics" schema:timeout-ms
"must be a positive integer within Node's timer range")
(number-option "--startup-timeout-ms MS" "Host/browser readiness deadline" schema:timeout-ms
"must be a positive integer within Node's timer range")

41
frontend/performance/benches/render_wasm/node/runner/browser.cljs
View File

@ -88,6 +88,14 @@
(catch :default cause
(failed cause))))))))))
(defn- deadline-failure
[cause deadline]
(if (contains? (ex-data cause) ::deadline-exceeded)
cause
(errors/with-context cause (errors/message-of cause)
{::deadline-exceeded (or (= ::timeout (:type (ex-data cause)))
(zero? (process/remaining deadline)))})))
(defn evaluate
"Calls a fixed bridge method, optionally with a Transit string.
JSON quoting embeds the argument; descriptors and results remain Transit."
@ -258,17 +266,18 @@
bounds))]
(assoc owner :browser browser :version (.version browser) :endpoint address))
(catch :default cause
(try
(await (close owner))
(catch :default cleanup-cause
(throw (ex-info "browser shutdown failed after startup error"
{:type ::startup-cleanup
:endpoint @(:address endpoint) :user-data-dir user-data-dir
:startup-failure (errors/describe-cause cause)}
cleanup-cause))))
(throw (ex-info (errors/message-of cause)
(assoc (ex-data cause) :endpoint @(:address endpoint) :user-data-dir user-data-dir)
cause)))))))
(let [cause (deadline-failure cause deadline)]
(try
(await (close owner))
(catch :default cleanup-cause
(throw (ex-info "browser shutdown failed after startup error"
{:type ::startup-cleanup
:endpoint @(:address endpoint) :user-data-dir user-data-dir
:startup-failure (errors/describe-cause cause)}
cleanup-cause))))
(throw (ex-info (errors/message-of cause)
(assoc (ex-data cause) :endpoint @(:address endpoint) :user-data-dir user-data-dir)
cause))))))))
(defn ^:async new-page
"Creates a context and page in a browser process we own, navigates to host,
@ -301,11 +310,13 @@
(transit/decode-str (await (within #(evaluate page "buildIdentity") bounds))))
page-own
(catch :default cause
(ex/ignoring (await (close-page page-own)))
(throw cause))))
(let [cause (deadline-failure cause deadline)]
(ex/ignoring (await (close-page page-own)))
(throw cause)))))
(catch :default cause
(ex/ignoring (await (close-page {:context context})))
(throw cause)))))
(let [cause (deadline-failure cause deadline)]
(ex/ignoring (await (close-page {:context context})))
(throw cause))))))
(defn connected?
"True while the owned CDP connection is live."

2
frontend/performance/benches/render_wasm/node/runner/feature_comparison.cljs
View File

@ -810,5 +810,5 @@
(:edges plan))) "\n"
"Cases: " (count cases) "\n"
"Attempts per case: " (:warmups options) " warmups, " (:repetitions options) " measured\n"
"Timeouts (ms): attempt " (:timeout-ms options)
"Timeouts (ms): case " (:timeout-ms options)
", build " (:build-timeout-ms options) ", startup " (:startup-timeout-ms options) "\n"))

329
frontend/performance/benches/render_wasm/node/runner/run.cljs
View File

@ -51,14 +51,16 @@
(defn- cause-evidence
"Plain failure evidence for a rejected browser, server or persistence action."
[cause fallback-phase]
(errors/fail-data
(errors/evidence (or (:phase (ex-data cause)) fallback-phase) cause)))
(cond-> (errors/fail-data
(errors/evidence (or (:phase (ex-data cause)) fallback-phase) cause))
(contains? (ex-data cause) ::browser/deadline-exceeded)
(assoc :deadline-exceeded (::browser/deadline-exceeded (ex-data cause)))))
(defn- failure-evidence
"Projects a bridge failure response into diagnostics evidence."
[evidence fallback-phase]
(if (map? evidence)
(cond-> (select-keys evidence [:status :phase :message :cause :partial :phases])
(cond-> (select-keys evidence [:status :phase :message :cause :partial :phases :reason :deadline-exceeded])
(nil? (:status evidence)) (assoc :status "failed")
(nil? (:phase evidence)) (assoc :phase (name fallback-phase)))
{:status "failed"
@ -74,6 +76,20 @@
[state]
(some? (interruption (:signal state))))
(defn- check-deadline
[deadline]
(when (zero? (process/remaining deadline))
(throw (ex-info "case deadline exceeded" {:type ::browser/timeout :phase :deadline}))))
(defn- failure-outcome
[state deadline evidence]
(cond
(interrupted? state) :interrupted
(if (contains? evidence :deadline-exceeded)
(:deadline-exceeded evidence)
(or (zero? (process/remaining deadline)) (= "deadline" (:phase evidence)))) :timed-out
:else :failed))
(defn- keep-sink-failure
"Remembers the first failed write to the run record or log. The failure stays
in the outcome even when a later write succeeds."
@ -126,7 +142,7 @@
(defn- record-failure
[state case-id stage evidence]
(let [failure (cond-> (assoc (select-keys evidence [:status :phase :message :cause :partial])
(let [failure (cond-> (assoc (select-keys evidence [:status :phase :message :cause :partial :reason])
:stage stage)
(some? case-id) (assoc :case-id case-id))]
(reset! (:harness-failed? state) true)
@ -162,11 +178,10 @@
(record-screenshot-skip state (:id case) reason))))
(defn- ^:async screenshot-case
[state case page]
(when (:screenshots (:opts state))
[state case page deadline]
(if (:screenshots (:opts state))
(let [{:keys [file path]} (runs/screenshot-path (:home state) (:id case))
opts (:opts state)
bounds {:deadline (process/deadline (:startup-timeout-ms opts))
bounds {:deadline deadline
:signal (:signal state) :phase :screenshot}
shot (attempt (await ((get-in state [:services :screenshot]) page path bounds)))]
(if (failure? shot)
@ -177,10 +192,16 @@
:cause (errors/describe-cause shot)})
(write-log state log/warn (str "Screenshot failed for "
(result/case-name (:id case)) ": "
(errors/message-of shot) "\n")))
(errors/message-of shot) "\n"))
(let [outcome (failure-outcome state deadline (cause-evidence shot :screenshot))]
(when (= :timed-out outcome)
(record-failure state (:id case) :case-deadline (cause-evidence shot :screenshot)))
(if (= :failed outcome) :ok outcome)))
(do
(record-screenshot-status state (:id case) {:file file :status :ok})
(write-log state log/debug (str "Screenshot " file)))))))
(write-log state log/debug (str "Screenshot " file))
:ok)))
:ok))
(defn- case-slots
"Returns each scheduled slot in order: all warmups, then measured slots.
@ -195,8 +216,8 @@
(defn- ^:async prepare-case
[state case page deadline warmup?]
(let [opts (:opts state)
build-id (:id (:selected-build state))
(check-deadline deadline)
(let [build-id (:id (:selected-build state))
preparation-id (str (random-uuid))
response (attempt
(await ((get-in state [:services :prepare])
@ -204,18 +225,30 @@
(transit/encode-str
{:case case
:preparation-id preparation-id
:timeout-ms (:timeout-ms opts)
:timeout-ms (max 1 (process/remaining deadline))
:module-url (renderer-url build-id "render-wasm.js")
:wasm-url (renderer-url build-id "render-wasm.wasm")})
{:deadline deadline :signal (:signal state) :phase :preparation})))]
(if (and (not (failure? response)) (= "ok" (:status response)))
{:preparation (assoc (:preparation response) :warmup? warmup?)}
(let [failure (if (failure? response)
(cause-evidence response :preparation)
(failure-evidence response :preparation))]
{:deadline deadline :signal (:signal state) :phase :preparation})))
accepted (when (and (not (failure? response)) (= "ok" (:status response)))
(attempt
(let [preparation (assoc (:preparation response) :warmup? warmup?)]
(when-not (and (= preparation-id (:preparation-id preparation))
(= (:id case) (:case-id preparation))
(not (:failure preparation)))
(throw (ex-info "preparation success response is inconsistent" {})))
(record-preparation state preparation)
preparation)))]
(if (and accepted (not (failure? accepted)))
{:preparation accepted}
(let [failure (cond
(failure? accepted) (assoc (cause-evidence accepted :invalid-output)
:partial {:response response})
(failure? response) (cause-evidence response :preparation)
:else (failure-evidence response :preparation))
phases (if (failure? accepted) (get-in response [:preparation :phases]) (:phases failure))]
(record-preparation state {:preparation-id preparation-id :case-id (:id case)
:warmup? warmup?
:phases (or (:phases failure) [])
:phases (measurement/partial-phases phases preparation-id [:preparation])
:diagnostics (or (get-in failure [:partial :setup]) {})
:failure (dissoc failure :phases)})
{:failure failure :preparation-id preparation-id}))))
@ -234,17 +267,23 @@
(defn- ^:async recycle-browser
[state]
(when-let [owner @(:owner-ref state)]
(reset! (:owner-ref state) nil)
(let [closed (attempt (await ((get-in state [:services :close]) owner)))]
(when (failure? closed)
(write-log state log/warn (str "Browser cleanup failed: " (errors/message-of closed)))))))
(if (failure? closed)
(do
(record-failure state nil :browser-close (cause-evidence closed :browser-close))
(reset! (:abort-run? state) true))
(do
(reset! (:owner-ref state) nil)
(reset! (:page-ref state) nil))))))
(defn- ^:async close-current-page
[state]
[state case]
(let [owner @(:owner-ref state)
connected? (get-in state [:services :connected?])]
(if (and owner connected? (not (connected? owner)))
(do
(record-failure state (:id case) :browser-disconnected
{:phase "browser-disconnected" :message "browser disconnected"})
(reset! (:page-ref state) nil)
(await (recycle-browser state))
false)
@ -254,20 +293,18 @@
(let [closed (attempt (await ((get-in state [:services :close-page]) page)))]
(if (failure? closed)
(do
(record-failure state nil :browser-close (cause-evidence closed :browser-close))
(record-failure state (:id case) :browser-close (cause-evidence closed :browser-close))
(await (recycle-browser state))
false)
true)))
true))))
(defn- ^:async ensure-browser
[state]
[state deadline]
(or @(:owner-ref state)
(let [opts (:opts state)
options (cond-> {:startup-timeout-ms (:startup-timeout-ms opts)
:signal (:signal state)}
(some? @(:launch-deadline state))
(assoc :deadline @(:launch-deadline state)))
options {:startup-timeout-ms (:startup-timeout-ms opts)
:signal (:signal state) :deadline deadline}
owner (await ((get-in state [:services :launch]) options))]
(reset! (:owner-ref state) owner)
(swap! (:run-ref state) assoc-in [:metadata :environment :browser] (:version owner))
@ -301,11 +338,12 @@
:ok)))
(defn- ^:async open-case-page
[state case owner]
[state case owner deadline]
(let [opened (attempt
(await ((get-in state [:services :new-page])
owner @(:host-ref state)
{:viewport (declarations/resolve-viewport (:view case))
:deadline deadline
:startup-timeout-ms (:startup-timeout-ms (:opts state))
:signal (:signal state)})))]
(cond
@ -318,141 +356,150 @@
{:outcome :interrupted}
:else
(do
(record-failure state (:id case) :browser-page (cause-evidence opened :browser-page))
(let [evidence (cause-evidence opened :browser-page)
outcome (failure-outcome state deadline evidence)]
(record-failure state (:id case) :browser-page evidence)
(await (recycle-browser state))
{:outcome :abandon}))))
{:outcome outcome}))))
(defn- ^:async open-page
[state case]
[state case deadline]
(check-deadline deadline)
(if (interrupted? state)
{:outcome :interrupted}
(let [owner (attempt (await (ensure-browser state)))]
(let [owner (attempt (await (ensure-browser state deadline)))]
(cond
(not (failure? owner))
(await (open-case-page state case owner))
(await (open-case-page state case owner deadline))
(interrupted? state)
{:outcome :interrupted}
:else
(do
(record-failure state (:id case) :browser-launch
(cause-evidence owner :browser-launch))
(reset! (:abort-run? state) true)
{:outcome :abort})))))
(defn- accept-preparation
[state preparation]
(record-preparation state preparation)
(check-gpu state preparation))
(let [evidence (cause-evidence owner :browser-launch)
outcome (failure-outcome state deadline evidence)]
(record-failure state (:id case) :browser-launch evidence)
(if (and (= :timed-out outcome)
(not= ::browser/startup-cleanup (:type (ex-data owner))))
{:outcome outcome}
(do
(reset! (:abort-run? state) true)
{:outcome :abort})))))))
(defn- ^:async execute-slot
[state case page preparation-id warmup? deadline]
(let [response (await (run-attempt state page preparation-id deadline))
measured (measurement/attempt (:id case) preparation-id warmup?
(if (failure? response)
(cause-evidence response :attempt)
response))]
(record-attempt state measured)
evidence (if (failure? response) (cause-evidence response :attempt) response)
accepted (attempt
(let [measured (measurement/attempt (:id case) preparation-id warmup? evidence)]
(record-attempt state measured)
measured))
measured (if (failure? accepted)
(let [invalid (assoc (cause-evidence accepted :invalid-output)
:phases (measurement/partial-phases (:phases evidence) preparation-id [:main :reset])
:partial {:response evidence})
failed (measurement/attempt (:id case) preparation-id warmup? invalid)]
(record-attempt state failed)
failed)
accepted)]
(cond
(and (failure? response) (interrupted? state)) :interrupted
(= :completed (:outcome measured)) :ok
:else :failed)))
:else (failure-outcome state deadline (if (failure? response)
(cause-evidence response :attempt)
response)))))
(defn- ^:async finish-slot
[state case page outcome final?]
(when final?
(if (= :ok outcome)
(await (screenshot-case state case page))
(record-screenshot-skip state (:id case) :no-completed-measured-attempt)))
outcome)
[state case page outcome final? deadline]
(if (and final? (= :ok outcome))
(await (screenshot-case state case page deadline))
(do
(when (not= :ok outcome)
(record-screenshot-skip state (:id case) :no-completed-measured-attempt))
outcome)))
(defn- ^:async run-fresh-slot
[state case page warmup? final?]
(let [deadline (process/deadline (:timeout-ms (:opts state)))
prepared (await (prepare-case state case page deadline warmup?))]
[state case page warmup? final? deadline]
(let [prepared (await (prepare-case state case page deadline warmup?))]
(if-let [evidence (:failure prepared)]
(do
(record-attempt state (measurement/attempt (:id case) (:preparation-id prepared) warmup? evidence))
(record-attempt state (measurement/attempt (:id case) (:preparation-id prepared) warmup?
(dissoc evidence :phases)))
(await (finish-slot state case page
(if (interrupted? state) :interrupted :failed) final?)))
(failure-outcome state deadline evidence) final? deadline)))
(let [preparation (:preparation prepared)]
(if (= :abort (accept-preparation state preparation))
(if (= :abort (check-gpu state preparation))
:abort
(let [outcome (await (execute-slot state case page (:preparation-id preparation)
warmup? deadline))]
(await (finish-slot state case page outcome final?))))))))
(await (finish-slot state case page outcome final? deadline))))))))
(defn- ^:async run-fresh-page
[state case warmup? final?]
(let [{:keys [page outcome]} (await (open-page state case))]
[state case warmup? final? deadline]
(let [{:keys [page outcome]} (await (open-page state case deadline))]
(if page
(let [outcome (await (run-fresh-slot state case page warmup? final?))
closed? (await (close-current-page state))]
(cond
(#{:abort :interrupted} outcome) outcome
(not closed?) :abandon
:else outcome))
(let [outcome (await (run-fresh-slot state case page warmup? final? deadline))]
(if (= :ok outcome)
(if (await (close-current-page state case)) outcome :failed)
outcome))
(or outcome :abandon))))
(defn- ^:async run-fresh-slots
[state case slots]
[state case slots deadline]
(if-let [[warmup? & pending] (seq slots)]
(let [outcome (await (run-fresh-page state case warmup? (empty? pending)))]
(if (#{:ok :failed} outcome)
(run-fresh-slots state case pending)
(let [outcome (await (run-fresh-page state case warmup? (empty? pending) deadline))]
(if (= :ok outcome)
(run-fresh-slots state case pending deadline)
outcome))
:done))
(defn- ^:async run-fresh-case
[state case]
(let [outcome (await (run-fresh-slots state case (case-slots (:opts state))))]
[state case deadline]
(let [outcome (await (run-fresh-slots state case (case-slots (:opts state)) deadline))]
(ensure-screenshot-status state case :case-abandoned)
outcome))
(defn- ^:async run-reuse-slots
[state case page preparation-id slots]
[state case page preparation-id slots deadline]
(if-let [[warmup? & pending] (seq slots)]
(let [deadline (process/deadline (:timeout-ms (:opts state)))
outcome (await (execute-slot state case page preparation-id warmup? deadline))]
(await (finish-slot state case page outcome (empty? pending)))
(let [_ (check-deadline deadline)
executed (await (execute-slot state case page preparation-id warmup? deadline))
outcome (await (finish-slot state case page executed (empty? pending) deadline))]
(if (= :ok outcome)
(run-reuse-slots state case page preparation-id pending)
(run-reuse-slots state case page preparation-id pending deadline)
outcome))
:done))
(defn- ^:async abandon-preparation
[state case evidence]
[state case evidence deadline]
(record-failure state (:id case) :preparation evidence)
(ensure-screenshot-status state case :no-completed-measured-attempt)
(await (close-current-page state))
(if (interrupted? state) :interrupted :abandon))
(failure-outcome state deadline evidence))
(defn- ^:async run-reuse-prepared
[state case page preparation]
(let [accepted (accept-preparation state preparation)
[state case page preparation deadline]
(let [accepted (check-gpu state preparation)
outcome (if (= :abort accepted)
:abort
(await (run-reuse-slots state case page (:preparation-id preparation)
(case-slots (:opts state)))))
closed? (await (close-current-page state))]
(case-slots (:opts state)) deadline)))]
(ensure-screenshot-status state case :case-abandoned)
(if closed? outcome :abandon)))
(if (= :done outcome)
(if (await (close-current-page state case)) outcome :failed)
outcome)))
(defn- ^:async run-reuse-page
[state case page]
(let [deadline (process/deadline (:timeout-ms (:opts state)))
prepared (await (prepare-case state case page deadline false))]
[state case page deadline]
(let [prepared (await (prepare-case state case page deadline false))]
(if-let [evidence (:failure prepared)]
(await (abandon-preparation state case evidence))
(await (run-reuse-prepared state case page (:preparation prepared))))))
(await (abandon-preparation state case evidence deadline))
(await (run-reuse-prepared state case page (:preparation prepared) deadline)))))
(defn- ^:async run-reuse-case
[state case]
(let [{:keys [page outcome]} (await (open-page state case))]
[state case deadline]
(let [{:keys [page outcome]} (await (open-page state case deadline))]
(cond
page (await (run-reuse-page state case page))
page (await (run-reuse-page state case page deadline))
(= :abort outcome) :abort
:else
@ -471,9 +518,8 @@
false)
(defn- ^:async launch-initial
[state]
(let [owner (attempt (await (ensure-browser state)))]
(reset! (:launch-deadline state) nil)
[state deadline]
(let [owner (attempt (await (ensure-browser state deadline)))]
(if (failure? owner)
(fail-startup state :browser-launch owner)
true)))
@ -508,15 +554,57 @@
[state]
(and (not @(:abort-run? state)) (not (interrupted? state))))
(defn- get-case-failure
[run case-id]
(or (some :failure (reverse (filter #(= case-id (:case-id %)) (:attempts run))))
(some :failure (reverse (filter #(= case-id (:case-id %)) (:preparations run))))
(some #(when (= case-id (:case-id %)) %)
(reverse (get-in run [:metadata :diagnostics :failures])))))
(defn- ^:async run-case
[state case deadline]
(let [executed (attempt
(if (= :fresh (:context case))
(await (run-fresh-case state case deadline))
(await (run-reuse-case state case deadline))))
outcome (if (failure? executed)
(let [evidence (cause-evidence executed :case)]
(record-failure state (:id case) :case evidence)
(failure-outcome state deadline evidence))
executed)
unmeasured? (and (= :done outcome)
(not (result/has-measurements? @(:run-ref state) (:id case))))
outcome (if unmeasured?
(do
(record-failure state (:id case) :case
{:phase "case" :reason :no-measurements
:message "case has no measurements"})
:failed)
outcome)
saved-outcome (cond
(= :done outcome) :completed
(= :interrupted outcome) :interrupted
(= :timed-out outcome) :timed-out
:else :failed)
failure (when (not= :completed saved-outcome)
(get-case-failure @(:run-ref state) (:id case)))]
(swap! (:run-ref state) result/record-case-outcome (:id case)
(cond-> {:outcome saved-outcome}
failure (assoc :failure failure)))
(ensure-screenshot-status state case :case-abandoned)
(persist state)
(when (not= :done outcome)
(await (recycle-browser state)))
outcome))
(defn- ^:async run-cases
[state cases]
(when (and (seq cases) (session-active? state))
(let [case (first cases)]
(let [case (first cases)
deadline (process/deadline (:timeout-ms (:opts state)))]
(write-log state log/info (str "▸ Case " (result/case-name (:id case))
" (" (name (:context case)) ")"))
(let [outcome (if (= :fresh (:context case))
(await (run-fresh-case state case))
(await (run-reuse-case state case)))]
(let [outcome (await (run-case state case deadline))]
(if (#{:abort :interrupted} outcome)
(do
(reset! (:abort-run? state) true)
@ -525,13 +613,12 @@
(defn- ^:async run-session
[state]
(let [deadline (process/deadline (:startup-timeout-ms (:opts state)))]
(reset! (:launch-deadline state) deadline)
(let [provenance-ready? (await (record-provenance state deadline))
server-ready? (and provenance-ready? (await (start-server state deadline)))
browser-ready? (and server-ready? (session-active? state) (await (launch-initial state)))]
(when (and browser-ready? (session-active? state))
(await (run-cases state (:cases state)))))))
(let [deadline (process/deadline (:startup-timeout-ms (:opts state)))
provenance-ready? (await (record-provenance state deadline))
server-ready? (and provenance-ready? (await (start-server state deadline)))
browser-ready? (and server-ready? (session-active? state) (await (launch-initial state deadline)))]
(when (and browser-ready? (session-active? state))
(await (run-cases state (:cases state))))))
(defn- get-termination
[run interrupted harness-failed?]
@ -539,7 +626,8 @@
(and (zero? (:warmups slots))
(zero? (:repetitions slots))))
(:remaining run))
failed? (some #(= :failed (:outcome %)) (:attempts run))
failed? (or (some #(= :failed (:outcome %)) (:attempts run))
(some #(not= :completed (:outcome %)) (vals (:case-outcomes run))))
reason (cond
interrupted :interrupted
all-attempted? (if failed? :failed :completed)
@ -571,11 +659,7 @@
(defn- finish-run
[state]
(let [termination (get-termination @(:run-ref state) (interruption (:signal state))
@(:harness-failed? state))
unmeasured (result/get-unmeasured-cases @(:run-ref state))
termination (if (and (= :completed (:reason termination)) (seq unmeasured))
{:reason :no-measurements :case-ids unmeasured}
termination)]
@(:harness-failed? state))]
(when (:screenshots (:opts state))
(swap! (:run-ref state) update-in [:metadata :diagnostics :screenshots]
complete-screenshots (:cases state)))
@ -665,7 +749,7 @@
:plan {:cases cases
:warmups (:warmups opts)
:repetitions (:repetitions opts)
:timeout-policy {:attempt-ms (:timeout-ms opts)}}
:timeout-policy {:case-ms (:timeout-ms opts)}}
:metadata (metadata root selected-build opts)}))
(defn- create-run-state
@ -681,7 +765,6 @@
:owner-ref (atom nil)
:page-ref (atom nil)
:host-ref (atom nil)
:launch-deadline (atom nil)
:abort-run? (atom false)
:harness-failed? (atom false)
:services (merge browser/service (:browser opts))})

3
frontend/performance/benches/render_wasm/report/format.cljs
View File

@ -61,8 +61,9 @@
(defn- case-status->text
"Displays pending slots and failed attempts, including excluded warmups."
[{:keys [remaining failed-attempts preparations phases]}]
[{:keys [remaining failed-attempts preparations phases outcome]}]
(str " remaining " (pr-str remaining)
(when outcome (str "\n outcome " (pr-str outcome)))
"\n phases " (pr-str phases)
" failed-warmups=" (count (filter :warmup? failed-attempts))
(when-let [failures (seq (keep :failure preparations))]

1
frontend/performance/benches/render_wasm/report/summarize.cljs
View File

@ -210,6 +210,7 @@
[run id ids analysis]
{:case-id id
:definition (some #(when (= id (:id %)) %) (get-in run [:plan :cases]))
:outcome (get-in run [:case-outcomes id])
:preparations (filterv #(= id (:case-id %)) (:preparations run))
:phases (phase-counts run id)
:remaining (get-in run [:remaining id])

74
frontend/performance/benches/render_wasm/result.cljs
View File

@ -76,6 +76,11 @@
"A run's current state or final reason, with optional diagnostic details."
[:map [:reason keyword?]])
(def schema:case-outcome
[:map {:closed true}
[:outcome [:enum :completed :failed :timed-out :interrupted]]
[:failure {:optional true} map?]])
(def schema:run
"schema:run stores saved case definitions, independent of any registry.
Build :functional holds compatibility settings; :provenance holds identities."
@ -88,12 +93,13 @@
[:cases [:vector [:map [:id some?]]]]
[:warmups [:int {:min 0}]]
[:repetitions [:int {:min 0}]]
[:timeout-policy [:map [:attempt-ms [:int {:min 1}]]]]]]
[:timeout-policy [:map {:closed true} [:case-ms [:int {:min 1}]]]]]]
[:metadata schema:metadata]
[:analysis analysis/schema:settings]
[:quantities [:map-of qualified-keyword? :any]]
[:preparations [:vector schema:preparation]]
[:attempts [:vector schema:attempt]]
[:case-outcomes [:map-of :any schema:case-outcome]]
[:remaining [:map-of :any [:map {:closed true}
[:warmups [:int {:min 0}]]
[:repetitions [:int {:min 0}]]]]]
@ -141,6 +147,7 @@
"One saved case's definition, preparations and shared metric reports."
[:map {:closed true}
[:case-id some?] [:definition [:map [:id some?]]]
[:outcome [:maybe schema:case-outcome]]
[:preparations [:vector schema:preparation]]
[:phases [:map-of :keyword [:map {:closed true}
[:executions nat-int?] [:completed nat-int?]
@ -276,6 +283,21 @@
"measured attempt precedes planned warmups" {:case-id case-id})
(update-in remaining path dec)))
(defn has-measurements?
"Reports whether a case has measured commands in its saved phase histories.
Args:
run: constructed or decoded run record.
case-id: exact saved case identifier.
Returns:
True when any preparation, main or reset block contains a command event."
[run case-id]
(boolean
(some (comp seq :events)
(mapcat :phases (filter #(= case-id (:case-id %))
(concat (:preparations run) (:attempts run)))))))
(defn get-unmeasured-cases
"Returns saved case ids whose phase histories contain no measured commands.
@ -287,9 +309,7 @@
[run]
(into []
(keep (fn [{:keys [id]}]
(when-not (some (comp seq :events)
(mapcat :phases (filter #(= id (:case-id %))
(concat (:preparations run) (:attempts run)))))
(when-not (has-measurements? run id)
id)))
(get-in run [:plan :cases])))
@ -303,6 +323,10 @@
"completed run has failed attempts" {})
(require-valid! (not-any? :failure (:preparations run))
"completed run has failed preparations" {})
(require-valid! (every? #(= :completed (:outcome %)) (vals (:case-outcomes run)))
"completed run has failed cases" {})
(require-valid! (= (set (map :id (get-in run [:plan :cases]))) (set (keys (:case-outcomes run))))
"completed run needs every case outcome" {})
(require-valid! (empty? (get-unmeasured-cases run))
"completed case has no measurements" {:case-ids (get-unmeasured-cases run)})))
@ -350,6 +374,18 @@
(when (= :failed (get-in attempt [:phases 0 :outcome]))
(require-valid! (= 1 (count (:phases attempt))) "reset follows failed main" {})))
(defn- check-case-outcome
[run case-id outcome]
(require-valid! (contains? (:remaining run) case-id) "unknown case outcome" {})
(require-valid! (sm/validate schema:case-outcome outcome) "invalid case outcome" {})
(when (= :completed (:outcome outcome))
(require-valid! (and (not (:failure outcome))
(every? zero? (vals (get-in run [:remaining case-id])))
(not-any? #(and (= case-id (:case-id %)) (= :failed (:outcome %))) (:attempts run))
(not-any? #(and (= case-id (:case-id %)) (:failure %)) (:preparations run))
(has-measurements? run case-id))
"completed case has pending, failed or unmeasured work" {:case-id case-id})))
(defn check-run
"Validates schema:run and chronological accounting at a saved-data boundary."
[run]
@ -363,6 +399,8 @@
(require-valid! (= (count prep-ids) (count (set prep-ids))) "duplicate preparation" {})
(require-valid! (every? #(contains? (set case-ids) %) (map :case-id preps)) "unknown preparation case" {})
(require-valid! (= left (:remaining run)) "remaining slots disagree with attempts" {})
(doseq [[case-id outcome] (:case-outcomes run)]
(check-case-outcome run case-id outcome))
(doseq [prep preps]
(check-preparation prep))
(doseq [[attempt-id attempt] (map-indexed vector (:attempts run))]
@ -373,16 +411,16 @@
(defn create-run
"Creates an empty raw run from externally supplied identities and facts.
The plan defaults to 3 warmups, 10 repetitions and a 30000 ms attempt timeout.
The plan defaults to 3 warmups, 10 repetitions and a 30000 ms case timeout.
For each case, all warmup slots precede measured slots. Failed attempts consume
their slot too; attempts retain chronological order across cases."
[{:keys [run-id started-at plan metadata analysis quantities]}]
(let [plan (merge {:warmups 3 :repetitions 10 :timeout-policy {:attempt-ms 30000}} plan)]
(let [plan (merge {:warmups 3 :repetitions 10 :timeout-policy {:case-ms 30000}} plan)]
(check-run {:type :run-record
:run-id run-id :started-at started-at :ended-at nil
:plan plan :metadata metadata :analysis (analysis/resolve-settings analysis)
:quantities (merge quantities/descriptions quantities)
:preparations [] :attempts [] :remaining (initial-remaining plan)
:preparations [] :attempts [] :case-outcomes {} :remaining (initial-remaining plan)
:termination {:reason :running}})))
(defn- check-open!
@ -395,6 +433,7 @@
The run must come from create-run or decode; earlier evidence is trusted."
[run preparation]
(check-open! run)
(require-valid! (not (contains? (:case-outcomes run) (:case-id preparation))) "case already terminated" {})
(check-preparation preparation)
(require-valid! (contains? (:remaining run) (:case-id preparation)) "unknown preparation case" {})
(require-valid! (not-any? #(= (:preparation-id preparation) (:preparation-id %))
@ -409,6 +448,7 @@
Earlier attempts are trusted; encode and decode check the complete record."
[run attempt]
(check-open! run)
(require-valid! (not (contains? (:case-outcomes run) (:case-id attempt))) "case already terminated" {})
(let [attempt (-> (merge {:preparation-id nil :phases []} attempt)
(assoc :attempt-id (count (:attempts run))))]
(check-attempt attempt (some #(when (= (:preparation-id attempt) (:preparation-id %)) %)
@ -426,6 +466,25 @@
(check-completion run termination)
(assoc run :ended-at ended-at :termination termination))
(defn record-case-outcome
"Records the terminal outcome of one case.
Args:
run: constructed or decoded run that accepts new evidence.
case-id: exact saved case identifier.
outcome: completion, failure, timeout or interruption with available cause.
Returns:
The run with its case outcome.
Throws:
ExceptionInfo for unknown cases, duplicate outcomes or malformed evidence."
[run case-id outcome]
(check-open! run)
(require-valid! (not (contains? (:case-outcomes run) case-id)) "duplicate case outcome" {})
(check-case-outcome run case-id outcome)
(assoc-in run [:case-outcomes case-id] outcome))
(defn merge-runs
"Returns a new validated run that appends `additional`'s measured attempts to
`base`.
@ -514,6 +573,7 @@
:quantities (:quantities base)
:preparations (into (vec base-preparations) additional-preparations)
:attempts attempts
:case-outcomes (merge (:case-outcomes base) (:case-outcomes additional))
:remaining remaining
:termination {:reason :completed}})))))

4
frontend/test/performance_tests/render_wasm/cli_integration.cjs
View File

@ -550,7 +550,7 @@ function runFixtureText({
k("repetitions"),
repetitions,
k("timeout-policy"),
map([k("attempt-ms"), 30000]),
map([k("case-ms"), 30000]),
]),
k("metadata"),
map([
@ -591,6 +591,8 @@ function runFixtureText({
[preparation],
k("attempts"),
attempts,
k("case-outcomes"),
map([k(caseId), map([k("outcome"), k("completed")])]),
k("remaining"),
map([k(caseId), map([k("warmups"), 0, k("repetitions"), 0])]),
k("termination"),

2
frontend/test/performance_tests/render_wasm/cli_test.cljs
View File

@ -267,6 +267,8 @@
["run" #"--warmups N\s+3"]
["run" #"--seed N\s+42"]
["run" #"--repetitions N\s+10"]
["run" #"--timeout-ms MS\s+30000\s+Case deadline"]
["ab" #"--timeout-ms MS\s+30000\s+Case deadline"]
["run" #"--features FEATURES\s+\[\]"]
["run" #"--\[no-\]default-features\s+true"]
["run" #"--build-timeout-ms MS\s+300000"]

6
frontend/test/performance_tests/render_wasm/compare_test.cljs
View File

@ -56,7 +56,7 @@
:plan {:cases [(merge (case-definition case-id scene) case-settings)]
:warmups 0
:repetitions 1
:timeout-policy {:attempt-ms 30000}}
:timeout-policy {:case-ms 30000}}
:metadata {:git {:sha run-id :dirty false}
:build {:functional functional}
:environment environment
@ -79,7 +79,9 @@
:outcome :completed
:phases [{:phase :main :preparation-id preparation-id
:outcome :completed :events [(metric value)]}]})]
(result/finish-run run "2026-10-26T00:01:00.000Z" {:reason :completed})))
(-> run
(result/record-case-outcome case-id {:outcome :completed})
(result/finish-run "2026-10-26T00:01:00.000Z" {:reason :completed}))))
(defn- write-run
[file run]

9
frontend/test/performance_tests/render_wasm/feature_comparison_test.cljs
View File

@ -163,7 +163,7 @@
(t/is (empty? (fs/readdirSync (frontend-root))))
(let [text (feature-comparison/format-plan ready)]
(doseq [part ["a, b" "00" "01" "11" "10" "00 -> 10: a" "Cases: 1"
"0 warmups, 1 measured" "attempt 2000, build 10000, startup 3000"]]
"0 warmups, 1 measured" "case 2000, build 10000, startup 3000"]]
(t/is (str/includes? text part))))))
(t/deftest ^:async metadata-selects-the-package-by-manifest-not-array-position
@ -472,7 +472,7 @@
:plan {:cases cases
:warmups 0
:repetitions 1
:timeout-policy {:attempt-ms (:timeout-ms options)}}
:timeout-policy {:case-ms (:timeout-ms options)}}
:metadata {:git {:sha nil :dirty nil}
:build {:functional (get-in selected-build [:manifest :functional])
:provenance (get-in selected-build [:manifest :provenance])}
@ -509,7 +509,10 @@
:failed {:reason :failed :failure {:message "fixture failure"}}
:zero-attempts {:reason :incomplete}
:interrupted {:reason :interrupted :signal "SIGINT"}
{:reason :completed})]
{:reason :completed})
run (if (= :completed (:reason termination))
(reduce #(result/record-case-outcome %1 (:id %2) {:outcome :completed}) run cases)
run)]
(result/finish-run run (.toISOString (js/Date.)) termination)))
(defn- execute-service

23
frontend/test/performance_tests/render_wasm/lifecycle_test.cljs
View File

@ -623,6 +623,29 @@
(finally
(cleanup)))))))
(t/deftest ^:async reused-preparation-keeps-its-original-case-deadline
(await
(with-entry-env
(^:async fn [cleanup]
(try
(unchecked-set js/globalThis "__benchNow" 0)
(register-phase-case {:context :reuse :preparation {:render :none}} camera/render-sync*)
(let [request (-> (scene-args 7 {}) transit/decode-str
(assoc :timeout-ms 100) transit/encode-str)
prepared (transit/decode-str (await (browser/prepare-scene-request request)))
id (get-in prepared [:preparation :preparation-id])]
(unchecked-set js/globalThis "__benchNow" 60)
(let [first (transit/decode-str (await (browser/run-prepared
(transit/encode-str {:preparation-id id :timeout-ms 10000}))))]
(t/is (= "ok" (:status first))))
(unchecked-set js/globalThis "__benchNow" 101)
(let [expired (transit/decode-str (await (browser/run-prepared
(transit/encode-str {:preparation-id id}))))]
(t/is (= "failed" (:status expired)))
(t/is (= "deadline" (:phase expired)))))
(finally
(cleanup)))))))
(t/deftest ^:async fresh-contexts-prepare-and-reset-once
(await
(with-entry-env

7
frontend/test/performance_tests/render_wasm/merge_test.cljs
View File

@ -62,7 +62,7 @@
:plan {:cases (mapv case-definition cases)
:warmups warmups
:repetitions repetitions
:timeout-policy {:attempt-ms timeout-ms}}
:timeout-policy {:case-ms timeout-ms}}
:metadata {:git {:sha git-sha :dirty false}
:build {:functional build}
:environment environment
@ -87,7 +87,10 @@
(result/record-attempt
run (attempt case-id measured-id false)))
run (range repetitions))))))
run cases)]
run cases)
run (if (= :completed (:reason termination))
(reduce #(result/record-case-outcome %1 %2 {:outcome :completed}) run cases)
run)]
(result/finish-run run ended-at termination)))
(t/deftest merges-measured-attempts-and-updates-the-plan

23
frontend/test/performance_tests/render_wasm/report_test.cljs
View File

@ -155,6 +155,7 @@
(add-attempt :prep true :completed phases)
(add-attempt :prep false :completed phases)
(add-attempt :prep false :completed phases)
(result/record-case-outcome :saved/pan {:outcome :completed})
(result/finish-run "end" {:reason :completed})
result/encode result/decode)
report (first (:cases (summary/summarize run)))
@ -281,6 +282,7 @@
(add-attempt :prep false :completed
[(block :main :prep [])
(block :reset :prep (if (= :reset phase) [(event :test/work [2])] []))])
(result/record-case-outcome :saved/pan {:outcome :completed})
(result/finish-run "end" {:reason :completed})
result/encode result/decode)]
(t/is (= [2] (:values (summary/observations run :saved/pan [phase :global :command/elapsed]))))))
@ -431,3 +433,24 @@
(assoc-in run [:remaining :saved/pan :repetitions] 2)]]
(t/is (= ::result/invalid-record
(error-type #(result/decode (transit/encode-str bad))))))))
(t/deftest case-outcomes-validate-completion-and-close-the-recording-boundary
(let [run (sample-run [1])
completed (result/record-case-outcome run :saved/pan {:outcome :completed})
ended (result/finish-run completed "end" {:reason :completed})]
(t/is (= ended (result/decode (result/encode ended))))
(doseq [bad [(assoc ended :case-outcomes {})
(-> run
(assoc-in [:plan :repetitions] 2)
(assoc-in [:remaining :saved/pan :repetitions] 1)
(assoc-in [:case-outcomes :saved/pan] {:outcome :completed}))
(assoc-in ended [:case-outcomes :unknown] {:outcome :completed})]]
(t/is (= ::result/invalid-record (error-type #(result/decode (transit/encode-str bad))))))
(t/is (= ::result/invalid-record
(error-type #(result/record-preparation completed (preparation :other [])))))
(let [stopped (-> run
(assoc-in [:plan :repetitions] 2)
(assoc-in [:remaining :saved/pan :repetitions] 1)
(result/record-case-outcome :saved/pan {:outcome :failed}))]
(t/is (thrown-with-msg? js/Error #"case already terminated"
(add-attempt stopped :prep false :completed [(block :main :prep [])]))))))

327
frontend/test/performance_tests/render_wasm/run_test.cljs
View File

@ -14,6 +14,7 @@
[benches.render-wasm.features.plan :as feature-plan]
[benches.render-wasm.node.cli.request :as request]
[benches.render-wasm.node.log :as log]
[benches.render-wasm.node.runner.browser :as browser]
[benches.render-wasm.node.runner.feature-comparison :as feature-comparison]
[benches.render-wasm.node.runner.run :as run]
[benches.render-wasm.node.storage.artifacts :as artifacts]
@ -87,7 +88,7 @@
(defn- fake-browser
[{:keys [launch-failure new-page-failure close-page-failure
prepare-failure run-failure screenshot-failure
interruption controller effective-graphics events]
interruption controller effective-graphics events close-failure]
:or {effective-graphics (constantly {})
events [{:command :fixture/work :metrics {:command/elapsed [1]}}]}}]
(let [state (atom {:launches 0 :pages 0 :close-pages 0 :closes 0
@ -100,9 +101,9 @@
(swap! state assoc :launches n)
(when (and launch-failure (launch-failure n))
(throw (ex-info "browser launch failed" {:phase :browser-startup})))
{:version "FakeBrowser/1" :browser #js {}}))
{:id n :version "FakeBrowser/1" :browser #js {}}))
:new-page
(fn [_ host _]
(fn [owner host _]
(when-not (and (string? (:base-url host))
(some? (get-in host [:build :id])))
(throw (ex-info "new-page requires the opened host"
@ -112,7 +113,7 @@
(swap! state assoc :pages n)
(when (and new-page-failure (new-page-failure n))
(throw (ex-info "browser page failed" {:phase :browser-page-startup})))
{:page {:id n} :context {}}))
{:page {:id n :browser-id (:id owner)} :context {}}))
:close-page
(fn [_]
(let [n (next-count state :close-pages)]
@ -121,7 +122,9 @@
(throw (ex-info "page close failed" {:phase :browser-page-close})))))
:close
(fn [_]
(swap! state update :closes inc))
(let [n (:closes (swap! state update :closes inc))]
(when (and close-failure (close-failure n))
(throw (ex-info "browser shutdown failed" {:phase :browser-close})))))
:connected?
(fn [_] true)
:prepare
@ -143,7 +146,7 @@
:fingerprint {:algorithm :sha-256
:digest (apply str (repeat 64 "a"))}
:effective-graphics (effective-graphics case-id (:id page))
:diagnostics {}
:diagnostics {:browser-id (:browser-id page)}
:phases [{:phase :preparation :preparation-id preparation-id
:outcome :completed :events []}]}]
(swap! state assoc-in [:preparations preparation-id] case-id)
@ -211,7 +214,8 @@
(options fake (path/join @temporary-root "unmeasured"))))
saved (result/decode (fs/readFileSync (get-in response [:home :record-path]) "utf8"))]
(t/is (= 1 (:exit-code response)))
(t/is (= {:reason :no-measurements :case-ids [:case/unmeasured]} (:termination saved)))
(t/is (= :failed (get-in saved [:termination :reason])))
(t/is (= :no-measurements (get-in saved [:case-outcomes :case/unmeasured :failure :reason])))
(t/is (= :completed (get-in saved [:attempts 0 :outcome])))))
(defn- preparations-for
@ -310,7 +314,7 @@
(t/is (= 2 (:launches state)))
(t/is (= 2 (:closes state))))))
(t/deftest ^:async failed-attempts-consume-slots-and-the-runner-continues
(t/deftest ^:async failed-attempts-stop-the-case-and-keep-pending-slots
(doseq [failure [:throw :response]]
(let [fake (fake-browser {:run-failure (fn [case-id n]
(when (and (= case-id :case/fresh) (= n 1))
@ -319,12 +323,12 @@
result (await (execute fake [(collected :case/fresh :fresh)] opts))
run (:run result)
tries (attempts-for run :case/fresh)]
(t/is (= 2 (count tries)))
(t/is (= [:failed :completed] (mapv :outcome tries)))
(t/is (every? zero? (vals (get-in run [:remaining :case/fresh]))))
(t/is (= :failed (get-in run [:termination :reason])))
(t/is (= [:failed] (mapv :outcome tries)))
(t/is (= {:warmups 0 :repetitions 1} (get-in run [:remaining :case/fresh])))
(t/is (= :failed (get-in run [:case-outcomes :case/fresh :outcome])))
(t/is (= :incomplete (get-in run [:termination :reason])))
(t/is (= 1 (:exit-code result)))
(t/is (= 2 (count (preparations-for run :case/fresh)))))))
(t/is (= 1 (count (preparations-for run :case/fresh)))))))
(t/deftest ^:async reuse-preparation-failure-leaves-slots-and-continues-to-the-next-case
(doseq [kind [:throw :response]]
@ -346,7 +350,7 @@
(t/is (= :case/reuse (:case-id failure)))
(t/is (= "scene-build" (:phase failure)))))))
(t/deftest ^:async fresh-preparation-failure-consumes-one-slot-and-retries
(t/deftest ^:async fresh-preparation-failure-consumes-its-slot-and-stops-the-case
(doseq [kind [:throw :response]]
(let [preparations (atom 0)
fake (fake-browser {:prepare-failure #(when (and (= % :case/fresh)
@ -356,14 +360,14 @@
(assoc (options fake (output-dir)) :repetitions 2 :screenshots true)))
run (:run result)
tries (attempts-for run :case/fresh)]
(t/is (= [:failed :completed] (mapv :outcome tries)))
(t/is (= [:failed] (mapv :outcome tries)))
(t/is (= (:preparation-id (first (preparations-for run :case/fresh)))
(:preparation-id (first tries))))
(t/is (= "scene-build" (get-in (first tries) [:failure :phase])))
(t/is (= 2 (count (preparations-for run :case/fresh))))
(t/is (= :ok (get-in run [:metadata :diagnostics :screenshots "case/fresh" :status])))
(t/is (every? zero? (vals (get-in run [:remaining :case/fresh]))))
(t/is (= :failed (get-in run [:termination :reason])))
(t/is (= 1 (count (preparations-for run :case/fresh))))
(t/is (= :skipped (get-in run [:metadata :diagnostics :screenshots "case/fresh" :status])))
(t/is (= {:warmups 0 :repetitions 1} (get-in run [:remaining :case/fresh])))
(t/is (= :incomplete (get-in run [:termination :reason])))
(t/is (= 1 (:exit-code result))))))
(t/deftest ^:async reuse-attempt-failure-leaves-pending-slots-and-continues-to-the-next-case
@ -377,7 +381,7 @@
(t/is (= [:failed] (mapv :outcome (attempts-for run :case/reuse))))
(t/is (= {:warmups 0 :repetitions 2} (get-in run [:remaining :case/reuse])))
(t/is (= [:completed :completed :completed] (mapv :outcome (attempts-for run :case/fresh))))
(t/is (= {:status :skipped :reason :case-abandoned}
(t/is (= {:status :skipped :reason :no-completed-measured-attempt}
(get-in run [:metadata :diagnostics :screenshots "case/reuse"])))
(t/is (= :incomplete (get-in run [:termination :reason])))
(t/is (= 1 (:exit-code result))))))
@ -399,10 +403,7 @@
(t/is (= 1 (:exit-code result))))))
(t/deftest ^:async browser-close-failure-abandons-the-case-and-relaunches-for-the-next
(let [fake (fake-browser {:run-failure (fn [case-id n]
(when (and (= case-id :case/a) (= n 1))
:throw))
:close-page-failure #(= % 1)})
(let [fake (fake-browser {:close-page-failure #(= % 1)})
opts (assoc (options fake (output-dir)) :repetitions 2)
result (await (execute fake [(collected :case/a :fresh)
(collected :case/b :fresh)] opts))
@ -410,7 +411,8 @@
state @(:state fake)]
(t/is (= 2 (:launches state)))
(t/is (= 1 (count (attempts-for run :case/a))))
(t/is (= :failed (:outcome (first (attempts-for run :case/a)))))
(t/is (= :completed (:outcome (first (attempts-for run :case/a)))))
(t/is (= :failed (get-in run [:case-outcomes :case/a :outcome])))
(t/is (= 1 (get-in run [:remaining :case/a :repetitions])))
(t/is (= 2 (count (attempts-for run :case/b))))
(t/is (every? #(= :completed (:outcome %)) (attempts-for run :case/b)))
@ -423,7 +425,7 @@
result (await (execute fake [(collected :case/a :fresh)]
(options fake (output-dir))))
run (:run result)]
(t/is (= :completed (get-in run [:termination :reason])))
(t/is (= :failed (get-in run [:termination :reason])))
(t/is (every? zero? (vals (get-in run [:remaining :case/a]))))
(t/is (= :browser-close
(:stage (last (get-in run [:metadata :diagnostics :failures])))))
@ -481,7 +483,6 @@
(t/is (str/includes? (:message failure) (pr-str (:renderer graphics))))
(t/is (str/includes? (:message failure) "./manage.sh run-devenv --gpu"))
(t/is (str/includes? (:message failure) "pnpm run render-bench"))
(t/is (= 1 (:close-pages state)))
(t/is (= 1 (:closes state))))))
(t/deftest ^:async graphics-rejection-preserves-earlier-attempts-and-stops-the-run
@ -502,7 +503,6 @@
(t/is (= [hardware-graphics software] (mapv :effective-graphics (:preparations run))))
(t/is (= {:case/a {:warmups 0 :repetitions (if (= context :fresh) 1 0)}
:case/b {:warmups 0 :repetitions 2}} (:remaining run)))
(t/is (= 2 (:close-pages @(:state fake))))
(t/is (= 1 (:closes @(:state fake)))))))
(t/deftest ^:async sigint-and-sigterm-persist-partial-interrupted-records
@ -560,7 +560,265 @@
(t/is (= (if (and (= method :prepare) (= context :reuse)) [:preparation] [])
(mapv :stage (get-in run [:metadata :diagnostics :failures])))))))
(t/deftest ^:async fresh-preparation-and-rendering-share-the-attempt-time-budget
(t/deftest ^:async phase-and-output-failures-retain-evidence-and-stop-the-case
(doseq [context [:fresh :reuse]
phase [:preparation :main :reset :invalid-preparation :contradictory-preparation :invalid-main :invalid-reset]]
(let [base (fake-browser {:prepare-failure (when (= :preparation phase)
#(when (= :case/a %) :response))})
services (:service base)
fake (assoc base :service
(assoc services
:prepare (fn [page request bounds]
(let [response ((:prepare services) page request bounds)]
(if (= :case/a (get-in (transit/decode-str request) [:case :id]))
(case phase
:invalid-preparation (update response :preparation dissoc :fingerprint)
:contradictory-preparation
(-> response
(assoc-in [:preparation :failure] {:message "callback failed"})
(assoc-in [:preparation :phases 0 :outcome] :failed))
response)
response)))
:run (fn [page request bounds]
(let [id (:preparation-id (transit/decode-str request))
case-id (get-in @(:state base) [:preparations id])
block {:phase :main :preparation-id id :outcome :completed
:events [{:command :fixture/work :metrics {:command/elapsed [1]}}]}]
(if (= :case/a case-id)
(case phase
:main {:status "failed" :phase "main" :message "main failed"
:phases [(assoc block :outcome :failed)]}
:reset {:status "failed" :phase "reset" :message "reset failed"
:phases [block (assoc block :phase :reset :outcome :failed)]}
:invalid-main {:status "ok" :phases []}
:invalid-reset {:status "ok" :phases [block {:phase :reset}]})
((:run services) page request bounds))))))
response (await (execute fake [(collected :case/a context) (collected :case/b context)]
(assoc (options fake (output-dir)) :repetitions 2)))
saved (saved-record response)
prep-failed? (#{:preparation :invalid-preparation :contradictory-preparation} phase)
attempts (attempts-for saved :case/a)]
(t/is (= :failed (get-in saved [:case-outcomes :case/a :outcome])))
(t/is (= (if (and prep-failed? (= :reuse context)) [] [:failed]) (mapv :outcome attempts)))
(t/is (= {:warmups 0 :repetitions (if (and prep-failed? (= :reuse context)) 2 1)}
(get-in saved [:remaining :case/a])))
(t/is (= [:completed :completed] (mapv :outcome (attempts-for saved :case/b))))
(when (= :contradictory-preparation phase)
(t/is (= "invalid-output" (get-in (first (preparations-for saved :case/a)) [:failure :phase]))))
(when (= :invalid-reset phase)
(t/is (= [:main] (mapv :phase (:phases (first attempts)))))
(t/is (= {:phase :reset} (get-in (first attempts) [:partial :response :phases 1]))))
(when (#{:invalid-main :invalid-reset} phase)
(t/is (= "invalid-output" (get-in (first attempts) [:failure :phase])))))))
(t/deftest ^:async runner-terminates-a-blocked-page-and-completes-the-next-case
(let [dir (build-dir)
owners (atom [])
prepared (atom {})
attempts (atom {})
identity-text (.stringify js/JSON (transit/encode-str {:build-id "test-build"}))
services (assoc browser/service
:launch (^:async fn [options]
(when-let [previous (last @owners)]
(t/is (false? (.isConnected ^js (:browser previous))))
(t/is (not (fs/existsSync (:user-data-dir previous)))))
(let [owner (await (browser/launch options))]
(swap! owners conj owner)
owner))
:prepare (fn [_ request _]
(let [{:keys [preparation-id] definition :case} (transit/decode-str request)]
(swap! prepared assoc preparation-id (:id definition))
{:status "ok"
:preparation {:preparation-id preparation-id :case-id (:id definition)
:fingerprint {:algorithm :sha-256 :digest (apply str (repeat 64 "a"))}
:effective-graphics {} :diagnostics {}
:phases [{:phase :preparation :preparation-id preparation-id
:outcome :completed :events []}]}}))
:run (^:async fn [page request bounds]
(let [id (:preparation-id (transit/decode-str request))
case-id (get @prepared id)
n (get (swap! attempts update case-id (fnil inc 0)) case-id)]
(await (browser/within
#(.evaluate ^js page
(if (and (= :case/a case-id) (= n 2))
"while (true) {}" "1"))
bounds))
{:status "ok"
:phases [{:phase :main :preparation-id id :outcome :completed
:events [{:command :fixture/work :metrics {:command/elapsed [1]}}]}]})))]
(fs/writeFileSync (path/join dir "index.html")
(str "<!doctype html><script>window.__benchReady=true;"
"window.__benchBridge={buildIdentity:()=>" identity-text "};</script>"))
(try
(let [opts (assoc (options nil (output-dir)) :browser services :repetitions 2
:timeout-ms 3000 :startup-timeout-ms 30000)
response (await (run/execute (artifacts/frontend-root)
(runs/create-home {:root (artifacts/frontend-root) :run-id "blocked-case"
:output-dir (:output-dir opts)})
(selected-build dir "test-build")
[(collected :case/a :reuse) (collected :case/b :reuse)] opts nil))
saved (saved-record response)]
(t/is (= [:completed :failed] (mapv :outcome (attempts-for saved :case/a))))
(t/is (= :timed-out (get-in saved [:case-outcomes :case/a :outcome])))
(t/is (= [:completed :completed] (mapv :outcome (attempts-for saved :case/b))))
(t/is (= :completed (get-in saved [:case-outcomes :case/b :outcome])))
(t/is (= 2 (count @owners)))
(t/is (not= (:user-data-dir (first @owners)) (:user-data-dir (second @owners))))
(doseq [owner @owners]
(t/is (false? (.isConnected ^js (:browser owner))))
(t/is (not (fs/existsSync (:user-data-dir owner))))))
(finally
(doseq [owner @owners]
(await (browser/close owner)))))))
(t/deftest ^:async opening-and-screenshot-work-share-the-case-deadline
(doseq [stage [:page :screenshot]]
(let [system-now (.-now js/Date)
clock (atom (system-now))
base (fake-browser {})
spend (fn [page bounds]
(when (= 1 (:id page))
(swap! clock + 300)
(when (>= @clock (:deadline bounds))
(throw (ex-info "browser work timed out" {:phase (name stage)})))))
fake (if (= :page stage)
(assoc-in base [:service :new-page]
(fn [owner host bounds]
(let [opened ((get-in base [:service :new-page]) owner host bounds)]
(spend (:page opened) bounds)
opened)))
(assoc-in base [:service :screenshot]
(fn [page file bounds]
(spend page bounds)
((get-in base [:service :screenshot]) page file bounds))))]
(try
(set! (.-now js/Date) (fn [] @clock))
(let [response (await (execute fake [(collected :case/a :reuse) (collected :case/b :reuse)]
(assoc (options fake (output-dir)) :timeout-ms 250 :screenshots true)))
saved (saved-record response)]
(t/is (= :timed-out (get-in saved [:case-outcomes :case/a :outcome])))
(t/is (= :completed (get-in saved [:case-outcomes :case/b :outcome])))
(t/is (= {:warmups 0 :repetitions (if (= :page stage) 1 0)}
(get-in saved [:remaining :case/a])))
(when (= :screenshot stage)
(t/is (= [:completed] (mapv :outcome (attempts-for saved :case/a))))))
(finally
(set! (.-now js/Date) system-now))))))
(t/deftest ^:async failed-browser-disposal-stops-later-cases
(let [fake (fake-browser {:run-failure (fn [_ _] :response) :close-failure #(= 1 %)})
response (await (execute fake [(collected :case/a :reuse) (collected :case/b :reuse)]
(assoc (options fake (output-dir)) :repetitions 2)))
saved (saved-record response)]
(t/is (= [:case/a] (mapv :case-id (:attempts saved))))
(t/is (= {:warmups 0 :repetitions 2} (get-in saved [:remaining :case/b])))
(t/is (= :browser-close (get-in saved [:termination :failure :stage])))))
(t/deftest ^:async failed-startup-cleanup-aborts-even-after-the-case-deadline
(let [system-now (.-now js/Date)
clock (atom (system-now))
base (fake-browser {:run-failure (fn [_ _] :response)})
fake (assoc-in base [:service :launch]
(fn [bounds]
(let [owner ((get-in base [:service :launch]) bounds)]
(when (= 2 (:id owner))
(reset! clock (inc (:deadline bounds)))
(throw (ex-info "browser shutdown failed after startup error"
{:type ::browser/startup-cleanup})))
owner)))]
(try
(set! (.-now js/Date) (fn [] @clock))
(let [response (await (execute fake [(collected :case/a :reuse) (collected :case/b :reuse)
(collected :case/c :reuse)]
(assoc (options fake (output-dir)) :timeout-ms 250)))
saved (saved-record response)]
(t/is (= :failed (get-in saved [:case-outcomes :case/b :outcome])))
(t/is (= {:warmups 0 :repetitions 1} (get-in saved [:remaining :case/c])))
(t/is (= :browser-launch (get-in saved [:termination :failure :stage]))))
(finally
(set! (.-now js/Date) system-now)))))
(t/deftest ^:async cleanup-time-does-not-change-the-case-failure
(let [system-now (.-now js/Date)
clock (atom (system-now))
base (fake-browser {:new-page-failure #(= 1 %)})
fake (assoc-in base [:service :close]
(fn [owner]
(swap! clock + 1000)
((get-in base [:service :close]) owner)))]
(try
(set! (.-now js/Date) (fn [] @clock))
(let [response (await (execute fake [(collected :case/a :reuse) (collected :case/b :reuse)]
(assoc (options fake (output-dir)) :timeout-ms 500)))
saved (saved-record response)]
(t/is (= :failed (get-in saved [:case-outcomes :case/a :outcome])))
(t/is (= :completed (get-in saved [:case-outcomes :case/b :outcome]))))
(finally
(set! (.-now js/Date) system-now)))))
(t/deftest ^:async service-cleanup-preserves-the-originating-failure-classification
(let [system-now (.-now js/Date)
clock (atom (system-now))
base (fake-browser {:new-page-failure #(= 1 %)})
fake (assoc-in base [:service :new-page]
(fn [owner host bounds]
(try
((get-in base [:service :new-page]) owner host bounds)
(catch :default cause
(let [annotated (errors/with-context cause "page startup failed"
{::browser/deadline-exceeded false})]
(swap! clock + 1000)
(throw annotated))))))]
(try
(set! (.-now js/Date) (fn [] @clock))
(let [response (await (execute fake [(collected :case/a :reuse) (collected :case/b :reuse)]
(assoc (options fake (output-dir)) :timeout-ms 250)))
saved (saved-record response)]
(t/is (= :failed (get-in saved [:case-outcomes :case/a :outcome])))
(t/is (= :completed (get-in saved [:case-outcomes :case/b :outcome]))))
(finally
(set! (.-now js/Date) system-now)))))
(t/deftest ^:async case-expiry-keeps-delivered-attempts-and-continues-in-a-new-browser
(doseq [context [:fresh :reuse]]
(let [system-now (.-now js/Date)
clock (atom (system-now))
base (fake-browser {})
active-case (atom nil)
spend (fn [duration bounds]
(when (= :case/a @active-case)
(swap! clock + duration)
(when (>= @clock (:deadline bounds))
(throw (ex-info "browser work timed out" {:phase (:phase bounds)})))))
services (:service base)
fake (assoc base :service
(assoc services
:prepare (fn [page request bounds]
(reset! active-case (get-in (transit/decode-str request) [:case :id]))
(spend 25 bounds)
((:prepare services) page request bounds))
:run (fn [page request bounds]
(spend 110 bounds)
((:run services) page request bounds))))]
(try
(set! (.-now js/Date) (fn [] @clock))
(let [response (await (execute fake [(collected :case/a context) (collected :case/b context)]
(assoc (options fake (output-dir))
:warmups 1 :repetitions 3 :timeout-ms 350)))
saved (saved-record response)
first-prep (first (preparations-for saved :case/a))
next-prep (first (preparations-for saved :case/b))]
(t/is (= [:completed :completed :failed] (mapv :outcome (attempts-for saved :case/a))))
(t/is (= {:warmups 0 :repetitions 1} (get-in saved [:remaining :case/a])))
(t/is (= :timed-out (get-in saved [:case-outcomes :case/a :outcome])))
(t/is (= [:completed :completed :completed :completed]
(mapv :outcome (attempts-for saved :case/b))))
(t/is (not= (get-in first-prep [:diagnostics :browser-id])
(get-in next-prep [:diagnostics :browser-id]))))
(finally
(set! (.-now js/Date) system-now))))))
(t/deftest ^:async preparation-and-rendering-share-the-case-budget
(doseq [context [:fresh :reuse]]
(let [system-now (.-now js/Date)
clock (atom (system-now))
@ -582,8 +840,9 @@
(let [result (await (execute fake [(collected :case/a context)]
(assoc (options fake (output-dir)) :timeout-ms 250)))
saved (saved-record result)]
(t/is (= (if (= context :fresh) :failed :completed) (:outcome (first (:attempts saved)))))
(t/is (= (if (= context :fresh) 1 0) (:exit-code result))))
(t/is (= :failed (:outcome (first (:attempts saved)))))
(t/is (= :timed-out (get-in saved [:case-outcomes :case/a :outcome])))
(t/is (= 1 (:exit-code result))))
(finally
(set! (.-now js/Date) system-now))))))
@ -797,7 +1056,7 @@
result (await (execute fake [(collected :case/a :fresh)] opts))
run (:run result)
diag (get-in run [:metadata :diagnostics])]
(t/is (= {:attempt-ms 1234} (get-in run [:plan :timeout-policy])))
(t/is (= {:case-ms 1234} (get-in run [:plan :timeout-policy])))
(t/is (= {:sha "build-revision" :dirty false} (get-in run [:metadata :git])))
(t/is (= {:mode :release :default-features? true :features [] :cljs-profile-marks false}
(get-in run [:metadata :build :functional])))
@ -944,7 +1203,7 @@
:execute-run
(fn [root home selected cases opts signal]
(let [fake (fake-browser
{:close-page-failure (when close-fails #(= % 2))
{:close-failure (when close-fails (constantly true))
:effective-graphics
(fn [_ page-id]
(if (and hardware-first (= page-id 1))
@ -959,7 +1218,7 @@
(t/is (= 1 (:exit-code response)))
(t/is (= (if hardware-first 1 0) (count (:attempts run))))
(t/is (= software (get-in failure [:partial :graphics])))
(t/is (= (if close-fails :browser-close :gpu) (get-in run [:termination :failure :stage])))
(t/is (= (if close-fails :cleanup :gpu) (get-in run [:termination :failure :stage])))
(t/is (= :failed (get-in saved [:configurations 0 :status])))
(t/is (= :gpu (get-in saved [:configurations 0 :failure :stage])))
(t/is (= :unattempted (get-in saved [:configurations 1 :status]))))))

2
frontend/test/performance_tests/render_wasm/runs_test.cljs
View File

@ -37,7 +37,7 @@
:plan {:cases [{:id :rects/load}]
:warmups 0
:repetitions 1
:timeout-policy {:attempt-ms 30000}}
:timeout-policy {:case-ms 30000}}
:metadata {:git {:sha "abc" :dirty false}
:build {:functional {:mode :release}}
:environment {:node "v24.0.0"}