mirror of
https://github.com/penpot/penpot.git
synced 2026-10-31 23:10:31 +00:00
d57cdb4588
* ✨ Add async test helpers and the with-mocks* macro Backport of the test scaffolding that the persistence retry work needs, taken from staging. Purely additive: no existing helper changes behavior, so the suite result is unchanged (740 tests, 0 failures). - frontend-tests.helpers.async: ->promise, await-response, settle, observe and wait-for. wait-for checks immediately and then polls one macrotask at a time, bounded, so it fails instead of hanging. - mock/with-mocks*: installs mocks with set! and evaluates its body in a generated ^:async fn, so the test reads sequentially and there is no done to thread. Evaluates to a promise. - mock/asap and mock/run-mocked: run-mocked awaits the body and restores the originals always, in that order, reporting a rejection or a non-promise return as an :error. The matching change that made with-mocks defer its body via asap is NOT backported: it would force all 58 existing uses across 14 files into done-chained contexts, which is a separate refactor. AI-assisted-by: space-bunny-free * ✨ Add an error report governor with compact payloads One browser session could emit an unbounded number of identical error reports, and a dropped connection was reported as an application defect with the full payload. Both are fixed here, in one place, because the persistence retry work that follows depends on the new submit-report contract. - A report governor in app.main.errors. Each report carries a fingerprint (report-name|type|code|hint|first stack frame); the first occurrence is always emitted, repeats inside a 2 minute window are counted, and the next emitted report carries them as :occurrences. The cache is bounded by evicting the oldest entry, so memory stays fixed. - flash reserves the report before generating it, so a suppressed occurrence never pays the report-building cost. Its whole body now runs inside one ts/schedule callback, a reporting or notification failure is logged instead of propagating, and it returns a total promise so tests can await it. - generate-report accepts :format :compact|:full. The compact payload keeps the context header plus type, code and uri, and skips the stack, the ex-data dump (it can carry request headers) and the last-events list. - environment-error-types is derived from rp/retryable-types, so the transport classification has a single source of truth. Environment failures are audit-only handled-exceptions and never reach the exception page or the alerts. :offline gets its own handler instead of falling through to :default. :occurrences travels inside the audit event :props, which the backend schema accepts as a free-form map, so no backend change is needed. The dedup governor test file is new: 7 tests over the fingerprint, the pure reservation decision, and the flash pipeline. Relates to #11726, #11743 AI-assisted-by: space-bunny-free * ♻️ Answer commit refusals with an empty stream commit-changes used to return nil when it refused to build a commit, which forced every caller that wanted to observe the outcome to special-case nil. It now returns an empty stream, so a refusal and an empty result look the same and callers can subscribe uniformly. Only the refusal branch changes: the accepted branch already answered with (rx/of commit). Potok treats a nil watch result as "nothing", so both spellings were already correct at the store level. The 57 call sites that wrap this event in rx/of are unaffected. They hold the ::commit-changes event itself, not the value its watch returns, so they keep processing exactly as before. AI-assisted-by: space-bunny-free * ✨ Retry transient saves with backoff and a reconnect notice Any failed update-file landed on the same path: the queue halted in :error, the retained changes were flushed away, and the user was told to reload. A dropped connection and a revoked permission are not the same thing, so they now take different paths. transient-error? splits them, checking :type and :cause-type because persistence-failed wraps the original cause. Transient failures enter a :retrying episode: the head commit stays queued and resends with backoff at 2s, 8s and 20s, then falls through to today's terminal path. Everything else is terminal and keeps the exact :error path. - Resends rotate the ::request-id stamp only once the old request has left active-requests, carry the same :commit-id, and never double send while one request is in flight. - Retry timers carry an episode token; a timer whose episode was superseded stays silent instead of sending or failing. - One tagged reconnect notice per episode, hidden by tag on save and on terminal failure. Recovery is silent. - The browser online event re-enters the runner, but only for a live :retrying episode, guarded by exists? js/window. - wait-persisted-or-error waits through :retrying instead of cutting the wait short. The episode keeps its :run-id, so a new local edit only joins the queue. Re-entering the runner on every edit would bypass the backoff and spend one attempt per batched commit. The retry test namespace is new: 13 tests over the classification, the backoff schedule, stamp rotation, stale tokens and the notice lifecycle. Relates to #11724 AI-assisted-by: space-bunny-free * 🐛 Fix stalled saves that never answer and edits lost on leave Two holes the retry work did not close. An update-file request that never gets an answer kept its attempt stamp forever, so the queue could not move and the user saw a saving state that never resolved. The request now has a 7 minute timeout that fails it as a transient network error, which hands it to the retry path instead of blocking. The timeout is longer than the 5 minute stall threshold on purpose: the stalled report must fire before the retry, not instead of it. The leave-page warning also had gaps. It now tracks unsaved edits directly, through sync-unload-guard, which holds :persistence in st/ongoing-tasks while edits sit in the buffer or the queue. Edits kept in the queue after a failed save keep the guard on too, so closing the tab warns instead of dropping them. store.cljs guards its ongoing-tasks watcher with exists? js/window, because the new tests exercise it without a window. Four new tests: three for the guard across buffering, an in-flight save and a retained edit, and one for a hung request that must resend and save. Closes #12223 AI-assisted-by: space-bunny-free * 🐛 Skip the save when a missing font cannot be remapped fix-deleted-fonts emitted text and typography changes even when no replacement font existed, so opening a page that uses a deleted custom font rewrote identical content and cleared position-data on every visit. Every visit became a save with nothing to save. Both loops now compare the remapped value with the original and emit nothing when they match. The remapped path is unchanged. Two new tests cover the no-replacement case and the remapped case. Closes #12151 AI-assisted-by: space-bunny-free * ✨ Add a debug helper that simulates an intermediary response A save that never reaches the backend is hard to reproduce: you need a proxy, a captive portal or a CDN in the way. This wraps fetch and answers the named API method with a status and a body that carries no Penpot error code, leaving every other request alone. Without a duration the simulation is spent after answering one call. With one it answers every matching call for that long, which is how to watch the save retry, warn, keep retrying and finally recover. debug.simulateHttpError('update-file') debug.simulateHttpError('get-comment-threads', 403) debug.simulateHttpError('update-file', 524, 120) debug.clearHttpErrorSimulation() Dev-only and self-contained: no production path reads it. AI-assisted-by: space-bunny-free * 🌐 Add the save retry and connection error strings Three strings the new persistence and error paths need: - errors.save-retrying, shown once per retry episode while the queue resends a queued save after a connection loss. - workspace.header.retrying, the save indicator state for an episode in progress. - errors.connection-error, shown by the network and offline handlers, which no longer fall through to the generic message. Added to en.po only. The translations script also normalizes every other locale and drops 439 strings that were already stale before this work; that belongs in its own maintenance pass, not in a backport. AI-assisted-by: space-bunny-free * 🐛 Report save failures as unhandled when they are our defect flash-persistence reported every failure it did not delegate as handled-exception, including the ones that are plainly our fault. A validation failure means the client sent something the server refused, yet it landed in the audit log with a name that skips the internal reports and the alerts. Nobody was paged. The report type now follows the cause. :validation, :server-error and :internal report as unhandled-exception, so they alert. The rest are unchanged. Two exclusions are deliberate: - Environment failures stay handled. They reach this point after the retry budget is spent, and the user's connection is not our bug. - :assertion stays handled, because its own handler already reports it that way and the two paths should not disagree. The cause type is read from :cause-type first, since persistence-failed wraps the original under :type :persistence. Reading :type alone would match the wrapper every time and change nothing. The user still gets the same sticky toast and report link; only the audit event name moves. The test that asserted a validation failure is "still notified" now asserts which event name it gets, across the bare and wrapped shapes, the environment types that must stay handled, and an unclassified one. AI-assisted-by: space-bunny-free * 🐛 Unblock persistence when opening a file after a terminal failure A terminal save failure set the state to :error and left the queue alone. recover-persistence only resumed for :save-permission-denied, so for every other terminal code both of its branches were false and it emitted nothing. Leaving and reopening the same file therefore changed nothing: the editor stayed halted until a reload, which rebuilds the whole store and loses the queue anyway. The path had two jobs under one name. initialize-persistence fires when the workspace view mounts and tears down the previous subscription; recover-persistence fires when a file is opened or the role changes. They are now separate: initialization picks up a queue left behind by a previous session, and deliberately does not lift a halt, so undeliverable edits are released when the user acts rather than when the view mounts. Recovery lifts the halt by releasing what can never be saved. release-failed-file drops every queued commit of the file that failed, then resume-persistence clears the error and settles an empty queue to :saved. Retaining them until that point is deliberate: it is what keeps wait-persisted-or-error rejecting, which is what stops workspace/versions.cljs from building a snapshot out of unsaved edits. Releasing at failure time would make waiters resolve instead. Permission loss is treated as permanent too, which the previous exception got wrong. The backend raises :not-found for a missing edit permission, so a resend does not succeed — it routes the user to the exception page. The old branch made that reachable: it resumed the save whenever the cached can-edit flag said yes, and the server disagreed. The only case where the flag can return to true is a team switch, where the queue is gone anyway, so nothing real is lost by releasing. Because the queue is shared, dropping the failed file is also what lets the other files' pending edits drain. One file that cannot be saved no longer blocks everything else. Tests: the failure retains the queue and notifies once, recovery releases it without replaying, a failed file does not block a second one, permission loss is released rather than resumed, and initialization leaves a halt alone. AI-assisted-by: space-bunny-free * ✨ Offer a reload on the save failure toast A terminal save failure stops autosave for the whole session, but the toast asked the user to reload without offering the reload. It listed five implied actions, none of them reachable from the banner, and it never said that the changes could not be saved. flash now takes a :links vector of actions, appended after the download link that :report-link? produces, so a caller can add recovery actions without reaching into the report handling. flash-persistence passes a reload built from labels.reload-page and location.reload, the pattern already used elsewhere in the app for a full restart. The message now says what happened, what is lost, and what to do about it: autosave has stopped for this session, the changes since the last successful save cannot be saved, and either reloading or leaving the file and opening it again restarts it. Naming both matters, because the toast links are easy to miss. Saying the edits are lost is not optional: recovery releases them, so without it the user finds out when they come back to the file. Translated in en.po and es.po only. The translations script would also normalize all 40 locales and drop entries that were already stale, which is a maintenance pass of its own. The reload callback cannot be invoked under the test runner, since js/location does not exist there, so it is covered by its label and by being callable. Whether the reload works is a manual check. AI-assisted-by: space-bunny-free * 🐛 Classify the responses the backend never wrote A reply written by a proxy instead of the backend carries no Penpot error code, so handle-response raised it as :internal. A 504 on a save skipped the retry cycle and stopped autosave at once, and because :internal reports as unhandled, it raised an alert. In a standard deployment this is how a slow save ends: nginx answers 504 after 300s, before the client's 7 minute timeout fires. The status now decides the type: - 504 and the Cloudflare 520-524 family raise :gateway-error, which joins retryable-types. Saves retry with backoff like 502 and 503, GETs retry, and a save that spends its retries reports as handled, because the report rules derive from the same set. - 429 raises :rate-limit, and any other error status :unexpected-response. Neither retries; both show a toast instead of the error page. - :internal is kept for a response below 400 that does not decode. Bodies we cannot read are cut to 500 characters, since proxy error pages run to tens of kilobytes. Tests: classification and retries per status, the transport handlers keep the page, a 504 and a 524 built by the real handle-response go through the save path and land on a retry, and the new types stay handled once the retries are spent. AI-assisted-by: claude-opus-5-5 * ✨ Keep retrying failed saves on a slow cycle Once the three backoff retries were spent, a save that failed on the connection stopped autosave like any terminal failure, and opening a file again dropped its edits. A connection that comes back after a minute lost the work all the same. A transport failure that outlasts the backoff now arms a slow cycle: one silent resend every 30s, until a save lands or 24h have passed since the failure started. The cycle stops when persistence restarts, the queue resumes another way, or the workspace closes. A save that keeps failing slowly is not resent for a day. A gateway that answers 504 after its own timeout may leave the backend applying the save, so each resend can cost the whole save again, and the failing file holds back every other file's queued edits. An attempt that fails after 90s or more counts as a slow failure; at three, the slow cycle stops sending the commit and the save fails as before. Fast failures, such as being offline, never count. - The first failure shows a toast that says the changes are kept and keep being sent, with no reload link, since a reload loses them. A save that lands hides it. - Opening a file sends a queue still inside its retry window once more instead of releasing it. Failures outside the window, and every non-transport failure, are still released. - A freshly loaded file gets back the queued edits of its version, so reopening it shows the changes that are still waiting to be saved. An edit that no longer applies to the loaded data leaves the file as the server sent it, so the file still opens; the edits stay queued. - A commit whose earlier request is no longer in flight is simply sent again under the same commit id: changes are idempotent, so there is no unknown-outcome guard any more. A request still in flight still blocks a parallel send. flash takes a :tag so the save failure toast can be hidden by tag. Tests: the spent backoff arms the slow cycle, a 504 arms it, a terminal failure does not, the cycle stops when the window closes or at the slow failure limit, recovery resends instead of releasing, reopening restores only the loaded file's edits and survives one that no longer applies, and the retrying toast offers no reload. * 🐛 Turn off the leave-page warning when failed edits are released The leave-page warning only followed the queue on appends and discards. Opening a file again releases the edits of a file that failed for good, but that release changed the queue without either event, so the warning stayed on with nothing left to save. Leaving the page asked about unsaved changes until the next save went through. The warning now also follows the release. Tests: releasing the last failed edit turns the warning off. * 🐛 Stop retrying failed saves outside the workspace Leaving the workspace was meant to stop the slow retry cycle, but only a cycle already armed stopped. A backoff timer kept firing after the user left, and a backoff spent outside the workspace armed a slow cycle that kept sending for up to a day from the dashboard. Those sends check the edit permission of whatever team is open, so a switch to a team where the user only views failed the queued edits for good, and the next file they opened dropped them. Leaving the workspace now suspends retries. A pending backoff timer goes stale, and while suspended a transport failure schedules no backoff, a spent backoff arms no slow cycle and the online event sends nothing. The edits stay queued and a request in flight still lands. Opening a file lifts the suspension and resends a retrying save whose backoff was cut short; a role change does not lift it. Tests: leaving stops a pending backoff, a failure after leaving schedules no retry and ignores the online event, a backoff spent outside the workspace arms no slow cycle, and opening a file saves the held edit in each case. --------- Co-authored-by: alonso.torres <alonso.torres@kaleidos.net>
885 lines
44 KiB
Clojure
885 lines
44 KiB
Clojure
;; This Source Code Form is subject to the terms of the Mozilla Public
|
|
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
|
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
|
;;
|
|
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
|
|
|
(ns frontend-tests.data.persistence-retry-test
|
|
"Retry-episode tests for the file persistence save pipeline
|
|
(`app.main.data.persistence`).
|
|
|
|
Covers the transient-failure retry contract: transient failures resend
|
|
the head commit with backoff, the budget bounds the episode, guards
|
|
(in-flight, stale token, online resume) prevent double-sends, and hung
|
|
resends across governor windows still earn their own stall report."
|
|
(:require
|
|
[app.common.time :as ct]
|
|
[app.common.uuid :as uuid]
|
|
[app.main.data.changes :as dch]
|
|
[app.main.data.event :as ev]
|
|
[app.main.data.persistence :as dps]
|
|
[app.main.errors :as errors]
|
|
[app.main.repo :as rp]
|
|
[app.main.router :as rt]
|
|
[app.main.store :as st]
|
|
[app.util.i18n :as i18n]
|
|
[beicon.v2.core :as rx]
|
|
[cljs.test :as t :include-macros true]
|
|
[frontend-tests.helpers.async :as async]
|
|
[frontend-tests.helpers.mock :as mock]
|
|
[potok.v2.core :as ptk]))
|
|
|
|
(defn- local-commit
|
|
"Builds a synthetic local commit event for `file-id`: a page edit as redo,
|
|
no undo, from the `:local` source."
|
|
[file-id]
|
|
(ptk/data-event ::dch/commit
|
|
{:id (uuid/next)
|
|
:file-id file-id
|
|
:file-revn 0
|
|
:file-vern 0
|
|
:source :local
|
|
:features #{}
|
|
:redo-changes [{:type :mod-page :id (uuid/next) :name "Edited"}]
|
|
:undo-changes []}))
|
|
|
|
(defn- with-persistence
|
|
"Async fixture for the persistence tests: mocks the transport, flash and
|
|
the leave-page task set (`:tasks`), and runs `f` with persistence
|
|
initialized.
|
|
|
|
Evaluates to a promise resolving once `f` settles and teardown completes;
|
|
`await` it from an `^:async` test.
|
|
|
|
By default the transport responds through the `:response` subject. Pass a
|
|
`respond` function (`(fn [cmd params] observable)`) to drive the transport
|
|
response directly instead. Delivery through the subject is asynchronous
|
|
(`observe-on :async`): await each effect — via `wait-for` — before
|
|
asserting it."
|
|
[f & [respond]]
|
|
(let [file-id (uuid/next)
|
|
response (rx/subject)
|
|
failures (atom [])
|
|
requests (atom [])
|
|
tasks (atom #{})
|
|
store (ptk/store {:state {:current-file-id file-id
|
|
:permissions {:can-edit true}
|
|
:files {file-id {:id file-id :revn 0}}}
|
|
:on-error #(t/is false (str %))})]
|
|
(mock/with-mocks*
|
|
{rp/cmd! (mock/stub (fn [cmd params]
|
|
(swap! requests conj [cmd params])
|
|
(if respond
|
|
(respond cmd params)
|
|
(->> response (rx/take 1) (rx/observe-on :async)))))
|
|
errors/flash (fn [& {:keys [cause]}]
|
|
(swap! failures conj cause))
|
|
st/ongoing-tasks tasks}
|
|
(try
|
|
(ptk/emit! store (dps/initialize-persistence))
|
|
(await (f {:file-id file-id :response response :failures failures
|
|
:requests requests :store store :tasks tasks}))
|
|
(finally
|
|
(rx/dispose! store)
|
|
(rx/end! response))))))
|
|
|
|
(defn- instant-backoff
|
|
"Timer double: the backoff delays fire at once and the slow cycle never
|
|
does, so a failure that outlasts the backoff settles in `:error`."
|
|
[ms]
|
|
(if (= ms dps/slow-retry-delay-ms)
|
|
(rx/subject)
|
|
(rx/of :tick)))
|
|
|
|
(defn- check-failed-save-response
|
|
"Feeds `result` as the transport response and asserts the commit stays
|
|
queued as a failed save instead of being treated as persisted. Retry
|
|
timers fire instantly (stubbed `rx/timer`, slow cycle held), so a
|
|
permanently bad answer exhausts the 3-retry budget and lands terminal: `:error` carrying
|
|
`:invalid-save-response`, queue intact, exactly 4 sends."
|
|
[result]
|
|
(with-persistence
|
|
(^:async fn [{:keys [file-id requests store]}]
|
|
(await
|
|
(mock/with-mocks*
|
|
{rx/timer (mock/stub instant-backoff)}
|
|
(ptk/emit! store (local-commit file-id) ::dps/force-persist)
|
|
(await (async/wait-for #(and (= :error (get-in @store [:persistence :status]))
|
|
(= :invalid-save-response
|
|
(get-in @store [:persistence :error :code]))
|
|
(= 1 (count (get-in @store [:persistence :queue]))))
|
|
"invalid response fails the save"))
|
|
(t/is (= :error (get-in @store [:persistence :status])))
|
|
(t/is (= :invalid-save-response (get-in @store [:persistence :error :code])))
|
|
(t/is (= 1 (count (get-in @store [:persistence :queue]))))
|
|
(t/is (= 4 (count @requests)) "initial send plus 3 retries"))))
|
|
(fn [_ _] result)))
|
|
|
|
;; Variant: an empty answer.
|
|
(t/deftest ^:async empty-save-response-preserves-the-queue-as-failed
|
|
(await (check-failed-save-response (rx/empty))))
|
|
|
|
;; Variant: a nil answer.
|
|
(t/deftest ^:async nil-save-response-preserves-the-queue-as-failed
|
|
(await (check-failed-save-response (rx/of nil))))
|
|
|
|
;; Variant: a negative revision answer.
|
|
(t/deftest ^:async invalid-revision-save-response-preserves-the-queue-as-failed
|
|
(await (check-failed-save-response (rx/of {:revn -1}))))
|
|
|
|
;; Retry tests.
|
|
;;
|
|
;; Production contract under test: a transient save failure keeps the head
|
|
;; commit queued under `:retrying` and resends it with backoff (2s / 8s /
|
|
;; 20s, then terminal). Retry timers are stubbed — instant when the test
|
|
;; drives completion, manually fired when it scripts the race — and every
|
|
;; stub records its delays so the schedule itself is asserted.
|
|
;;
|
|
;; Same async pattern as above: triggers stay bare, every assert block is
|
|
;; preceded by `wait-for` on its leading signal.
|
|
|
|
;; Scenario: the first send fails transiently, the retry succeeds. Both
|
|
;; sends carry the same `:commit-id`; the save lands with an empty queue
|
|
;; and the episode metadata is cleared. Proves: one transient failure
|
|
;; retries the same commit instead of erroring.
|
|
(t/deftest ^:async transient-failure-retries-and-saves
|
|
(let [calls (atom 0)]
|
|
(await
|
|
(with-persistence
|
|
(^:async fn [{:keys [file-id requests store]}]
|
|
(await
|
|
(mock/with-mocks*
|
|
{rx/timer (mock/stub instant-backoff)}
|
|
(ptk/emit! store (local-commit file-id) ::dps/force-persist)
|
|
(await (async/wait-for #(and (= :saved (get-in @store [:persistence :status]))
|
|
(empty? (get-in @store [:persistence :queue])))
|
|
"retry saves the file"))
|
|
(t/is (= :saved (get-in @store [:persistence :status])))
|
|
(t/is (empty? (get-in @store [:persistence :queue])))
|
|
(t/is (= 2 (count @requests)) "failed send plus one retry")
|
|
(t/is (apply = (map (comp :commit-id second) @requests))
|
|
"both sends carry the same commit id")
|
|
(t/is (nil? (get-in @store [:persistence :attempts]))
|
|
"the episode metadata is cleared on success"))))
|
|
(fn [_ _]
|
|
(if (= 1 (swap! calls inc))
|
|
(rx/throw (ex-info "offline" {:type :offline}))
|
|
(rx/of {:revn 1})))))))
|
|
|
|
(defn- check-gateway-response-retries
|
|
"Answers the first send with a proxy error page for `status`, built by the
|
|
real `rp/handle-response`, and the retry with a valid revision. The save
|
|
must take the retry path and land, never halt in `:error`."
|
|
[status]
|
|
(let [calls (atom 0)]
|
|
(with-persistence
|
|
(^:async fn [{:keys [file-id failures requests store]}]
|
|
(await
|
|
(mock/with-mocks*
|
|
{rx/timer (mock/stub instant-backoff)}
|
|
(ptk/emit! store (local-commit file-id) ::dps/force-persist)
|
|
(await (async/wait-for #(and (= :saved (get-in @store [:persistence :status]))
|
|
(empty? (get-in @store [:persistence :queue])))
|
|
(str "a " status " is retried and the save lands")))
|
|
(t/is (= 2 (count @requests)) "the proxy error plus one retry")
|
|
(t/is (empty? @failures) "a retried proxy error never reaches the failure toast"))))
|
|
(fn [_ _]
|
|
(if (= 1 (swap! calls inc))
|
|
(rp/handle-response {:status status
|
|
:headers {"content-type" "text/html"}
|
|
:body "<html><body>Gateway Timeout</body></html>"
|
|
:uri "https://penpot.example.com/api/main/methods/update-file"})
|
|
(rx/of {:revn 1}))))))
|
|
|
|
;; Variant: the proxy in front of the backend timed out.
|
|
(t/deftest ^:async gateway-timeout-save-is-retried
|
|
(await (check-gateway-response-retries 504)))
|
|
|
|
;; Variant: Cloudflare gave up waiting for the origin.
|
|
(t/deftest ^:async cloudflare-origin-timeout-save-is-retried
|
|
(await (check-gateway-response-retries 524)))
|
|
|
|
;; Scenario: every send fails transiently. The 2s / 8s / 20s retries fire
|
|
;; and then the failure falls through to the exact terminal path: `:error`
|
|
;; carrying the cause, queue intact, one flash. Proves: the budget bounds
|
|
;; the episode (4 sends) and exhaustion is today's terminal behavior.
|
|
(t/deftest ^:async retry-exhaustion-goes-terminal
|
|
(let [delays (atom [])]
|
|
(await
|
|
(with-persistence
|
|
(^:async fn [{:keys [file-id failures requests store]}]
|
|
(await
|
|
(mock/with-mocks*
|
|
{rx/timer (mock/stub (fn [ms] (swap! delays conj ms) (instant-backoff ms)))}
|
|
(ptk/emit! store (local-commit file-id) ::dps/force-persist)
|
|
(await (async/wait-for #(and (= :error (get-in @store [:persistence :status]))
|
|
(= 4 (count @requests)))
|
|
"retries exhaust into terminal error"))
|
|
(t/is (= :error (get-in @store [:persistence :status])))
|
|
(t/is (= :save-failed (get-in @store [:persistence :error :code])))
|
|
(t/is (= 1 (count (get-in @store [:persistence :queue]))))
|
|
(t/is (= [2000 8000 20000 dps/slow-retry-delay-ms] @delays)
|
|
"the backoff schedule fires in order, then the slow cycle is armed")
|
|
(t/is (= 1 (count @failures)) "exhaustion flashes exactly once"))))
|
|
(fn [_ _] (rx/throw (ex-info "offline" {:type :offline})))))))
|
|
|
|
;; Scenario: a retry timer fires while the replacement request is still in
|
|
;; flight. The timer is driven by hand: fail the first send, let the
|
|
;; online signal resend the head (hanging), then fire the pending timer.
|
|
;; Proves: the firing is skipped instead of double-sending the same commit.
|
|
(t/deftest ^:async retry-skips-resend-while-previous-request-is-in-flight
|
|
(let [calls (atom 0)
|
|
delays (atom [])
|
|
timer-s (rx/subject)]
|
|
(await
|
|
(with-persistence
|
|
(^:async fn [{:keys [file-id requests store]}]
|
|
(await
|
|
(mock/with-mocks*
|
|
{rx/timer (mock/stub (fn [ms] (swap! delays conj ms) timer-s))}
|
|
;; Phase 1 — first send fails transiently; the retry pends on
|
|
;; the hand-fired timer.
|
|
(ptk/emit! store (local-commit file-id) ::dps/force-persist)
|
|
(await (async/wait-for #(= :retrying (get-in @store [:persistence :status]))
|
|
"transient failure retries"))
|
|
(t/is (= 1 (count @requests)))
|
|
;; Phase 2 — the online signal resends the head under the live
|
|
;; episode (status stays :retrying), hanging in flight.
|
|
(ptk/emit! store (#'dps/resume-on-online))
|
|
(await (async/wait-for #(= 2 (count @requests)) "online resends"))
|
|
(t/is (= :retrying (get-in @store [:persistence :status])))
|
|
(t/is (= 1 (get-in @store [:persistence :attempts])))
|
|
;; Phase 3 — the pending timer fires into the in-flight request:
|
|
;; skipped, never a third send.
|
|
(rx/push! timer-s :tick)
|
|
(await (async/settle))
|
|
(t/is (= 2 (count @requests)) "no double-send while in flight")
|
|
(t/is (= :retrying (get-in @store [:persistence :status])))
|
|
(t/is (= 1 (count (get-in @store [:persistence :queue]))))
|
|
(rx/end! timer-s))))
|
|
(fn [_ _]
|
|
(if (= 1 (swap! calls inc))
|
|
(rx/throw (ex-info "offline" {:type :offline}))
|
|
(rx/subject)))))))
|
|
|
|
;; Scenario: a persist-commit arrives with a superseded episode token after
|
|
;; a transient failure. Without the token guard it would send outside the
|
|
;; backoff schedule; with it, nothing happens. Proves: stale retry timers
|
|
;; stay silent.
|
|
(t/deftest ^:async stale-retry-token-stays-silent
|
|
(await
|
|
(with-persistence
|
|
(^:async fn [{:keys [file-id requests store]}]
|
|
(await
|
|
(mock/with-mocks*
|
|
{rx/timer (mock/stub (fn [_] (rx/subject)))}
|
|
(ptk/emit! store (local-commit file-id) ::dps/force-persist)
|
|
(await (async/wait-for #(= :retrying (get-in @store [:persistence :status]))
|
|
"transient failure retries"))
|
|
(t/is (= 1 (count @requests)))
|
|
(ptk/emit! store (#'dps/persist-commit
|
|
(peek (get-in @store [:persistence :queue]))
|
|
{:token (uuid/next)}))
|
|
(await (async/settle))
|
|
(t/is (= 1 (count @requests)) "stale token sends nothing")
|
|
(t/is (= :retrying (get-in @store [:persistence :status])))
|
|
(t/is (nil? (get-in @store [:persistence :error]))))))
|
|
(fn [_ _] (rx/throw (ex-info "offline" {:type :offline}))))))
|
|
|
|
;; Scenario: a new edit lands while the episode waits on its backoff timer.
|
|
;; The edit only joins the queue: nothing is sent and no attempt is spent
|
|
;; until the timer fires. The retry then saves the head and the runner
|
|
;; sends the new edit after it. Proves: edits during an episode never
|
|
;; bypass the backoff nor consume the retry budget.
|
|
(t/deftest ^:async new-edit-during-retry-waits-for-the-backoff
|
|
(let [calls (atom 0)
|
|
delays (atom [])
|
|
timer-s (rx/subject)]
|
|
(await
|
|
(with-persistence
|
|
(^:async fn [{:keys [file-id requests store]}]
|
|
(await
|
|
(mock/with-mocks*
|
|
{rx/timer (mock/stub (fn [ms] (swap! delays conj ms) timer-s))}
|
|
;; Phase 1 — first send fails transiently; the retry pends on
|
|
;; the hand-fired timer.
|
|
(ptk/emit! store (local-commit file-id) ::dps/force-persist)
|
|
(await (async/wait-for #(= :retrying (get-in @store [:persistence :status]))
|
|
"transient failure retries"))
|
|
(t/is (= 1 (count @requests)))
|
|
;; Phase 2 — a new edit joins the queue without resending.
|
|
(ptk/emit! store (local-commit file-id) ::dps/force-persist)
|
|
(await (async/wait-for #(= 2 (count (get-in @store [:persistence :queue])))
|
|
"second edit is queued"))
|
|
(await (async/settle))
|
|
(t/is (= 1 (count @requests)) "the edit does not resend the head")
|
|
(t/is (= 1 (get-in @store [:persistence :attempts])) "no attempt is spent")
|
|
(t/is (= [2000] @delays) "the pending backoff is kept")
|
|
(t/is (= :retrying (get-in @store [:persistence :status])))
|
|
;; Phase 3 — the timer fires: the head saves, then the new edit.
|
|
(rx/push! timer-s :tick)
|
|
(await (async/wait-for #(and (= :saved (get-in @store [:persistence :status]))
|
|
(empty? (get-in @store [:persistence :queue])))
|
|
"both edits save"))
|
|
(t/is (= 3 (count @requests)) "failed send, retry, then the new edit")
|
|
(let [[first-id retry-id edit-id] (map (comp :commit-id second) @requests)]
|
|
(t/is (= first-id retry-id) "the retry resends the head")
|
|
(t/is (not= retry-id edit-id) "the new edit is sent after the head"))
|
|
(rx/end! timer-s))))
|
|
(fn [_ _]
|
|
(if (= 1 (swap! calls inc))
|
|
(rx/throw (ex-info "offline" {:type :offline}))
|
|
(rx/of {:revn @calls})))))))
|
|
|
|
;; Scenario: a transient failure raises the reconnect notice; the retry
|
|
;; then succeeds. The timer is driven by hand so the test observes the
|
|
;; episode mid-flight: one visible notice (the store holds a single toast,
|
|
;; so episodes never stack), gone once the save lands. Proves: exactly one
|
|
;; notice per episode, hidden on recovery.
|
|
(t/deftest ^:async retry-shows-a-single-reconnect-notice-until-saved
|
|
(let [calls (atom 0)
|
|
timer-s (rx/subject)]
|
|
(await
|
|
(with-persistence
|
|
(^:async fn [{:keys [file-id store]}]
|
|
(await
|
|
(mock/with-mocks*
|
|
{rx/timer (mock/stub (fn [_] timer-s))
|
|
i18n/tr (mock/stub #(str "translated:" %))}
|
|
(ptk/emit! store (local-commit file-id) ::dps/force-persist)
|
|
(await (async/wait-for #(= 1 (get-in @store [:persistence :attempts]))
|
|
"first attempt fails into retrying"))
|
|
(let [notice (get @store :notification)]
|
|
(t/is (map? notice) "a single notice is visible, never stacked"))
|
|
(t/is (= :persistence-reconnecting (get-in @store [:notification :tag])))
|
|
(t/is (= "translated:errors.save-retrying"
|
|
(get-in @store [:notification :content])))
|
|
(rx/push! timer-s :tick)
|
|
(await (async/wait-for #(and (= :saved (get-in @store [:persistence :status]))
|
|
(nil? (get @store :notification)))
|
|
"save hides the notice"))
|
|
(t/is (= :saved (get-in @store [:persistence :status])))
|
|
(t/is (nil? (get @store :notification)) "recovery is silent")
|
|
(rx/end! timer-s))))
|
|
(fn [_ _]
|
|
(if (= 1 (swap! calls inc))
|
|
(rx/throw (ex-info "offline" {:type :offline}))
|
|
(rx/of {:revn 1})))))))
|
|
|
|
;; Scenario: every send fails transiently with instant timers. Exhaustion
|
|
;; takes the terminal path, which hides the reconnect notice explicitly
|
|
;; before flashing. Proves: no stale notice survives a terminal failure.
|
|
(t/deftest ^:async retry-exhaustion-hides-the-reconnect-notice
|
|
(await
|
|
(with-persistence
|
|
(^:async fn [{:keys [file-id store]}]
|
|
(await
|
|
(mock/with-mocks*
|
|
{rx/timer (mock/stub instant-backoff)}
|
|
(ptk/emit! store (local-commit file-id) ::dps/force-persist)
|
|
(await (async/wait-for #(= :error (get-in @store [:persistence :status]))
|
|
"retries exhaust into terminal error"))
|
|
(t/is (= :error (get-in @store [:persistence :status])))
|
|
(t/is (nil? (get @store :notification))
|
|
"the terminal path hides the reconnect notice"))))
|
|
(fn [_ _] (rx/throw (ex-info "offline" {:type :offline}))))))
|
|
|
|
;; Scenario: a retrying episode waits on its backoff timer when the browser
|
|
;; reports connectivity back. The pending timer never fires, so only the
|
|
;; online signal can resume: the head resends under the live episode and,
|
|
;; once answered, the file saves. A second online signal with an empty
|
|
;; queue sends nothing. Proves: reconnect resumes promptly without
|
|
;; terminal failures staying terminal.
|
|
(t/deftest ^:async online-event-resumes-a-retrying-episode
|
|
(let [calls (atom 0)
|
|
timer-s (rx/subject)
|
|
second-s (atom nil)]
|
|
(await
|
|
(with-persistence
|
|
(^:async fn [{:keys [file-id requests store]}]
|
|
(await
|
|
(mock/with-mocks*
|
|
{rx/timer (mock/stub (fn [_] timer-s))}
|
|
(ptk/emit! store (local-commit file-id) ::dps/force-persist)
|
|
(await (async/wait-for #(= :retrying (get-in @store [:persistence :status]))
|
|
"transient failure retries"))
|
|
(t/is (= 1 (count @requests)))
|
|
(ptk/emit! store (#'dps/resume-on-online))
|
|
(await (async/wait-for #(= 2 (count @requests)) "online resends"))
|
|
(t/is (= 2 (count @requests)))
|
|
(t/is (= :retrying (get-in @store [:persistence :status])))
|
|
(t/is (= 1 (get-in @store [:persistence :attempts])))
|
|
(rx/push! @second-s {:revn 1})
|
|
(await (async/wait-for #(and (= :saved (get-in @store [:persistence :status]))
|
|
(empty? (get-in @store [:persistence :queue])))
|
|
"resumed save lands"))
|
|
(t/is (= :saved (get-in @store [:persistence :status])))
|
|
(ptk/emit! store (#'dps/resume-on-online))
|
|
(await (async/settle))
|
|
(t/is (= 2 (count @requests)) "online with an empty queue sends nothing")
|
|
(rx/end! timer-s))))
|
|
(fn [_ _]
|
|
(if (= 1 (swap! calls inc))
|
|
(rx/throw (ex-info "offline" {:type :offline}))
|
|
(let [s (rx/subject)] (reset! second-s s) s)))))))
|
|
|
|
(defn- audit-events
|
|
"The audit events (report emissions) out of everything collected through
|
|
the `st/emit!` double. Discriminates by `ptk/type`, which is total (never
|
|
throws)."
|
|
[events]
|
|
(filter #(= ::ev/event (ptk/type %)) events))
|
|
|
|
;; Scenario: the `online` event arrives while a retry request is already in
|
|
;; flight. The first send fails transiently, the retry timer resends
|
|
;; (hanging), then connectivity reports back mid-flight. Proves: the online
|
|
;; entry point honors the in-flight guard instead of double-sending — the
|
|
;; same guard as the retry-timer path, through `resume-on-online` ->
|
|
;; `run-persistence-task`.
|
|
(t/deftest ^:async online-event-does-not-resend-an-in-flight-request
|
|
(let [calls (atom 0)
|
|
timer-s (rx/subject)]
|
|
(await
|
|
(with-persistence
|
|
(^:async fn [{:keys [file-id requests store]}]
|
|
(await
|
|
(mock/with-mocks*
|
|
{rx/timer (mock/stub (fn [_] timer-s))}
|
|
;; Phase 1 — first send fails transiently; the retry pends on
|
|
;; the hand-fired timer.
|
|
(ptk/emit! store (local-commit file-id) ::dps/force-persist)
|
|
(await (async/wait-for #(= :retrying (get-in @store [:persistence :status]))
|
|
"transient failure retries"))
|
|
(t/is (= 1 (count @requests)))
|
|
;; Phase 2 — the retry timer fires and resends the head,
|
|
;; hanging in flight.
|
|
(rx/push! timer-s :tick)
|
|
(await (async/wait-for #(= 2 (count @requests)) "retry resends"))
|
|
(t/is (= :retrying (get-in @store [:persistence :status])))
|
|
;; Phase 3 — online arrives mid-flight: silent, never a third send.
|
|
(ptk/emit! store (#'dps/resume-on-online))
|
|
(await (async/settle))
|
|
(t/is (= 2 (count @requests)) "online sends nothing while in flight")
|
|
(t/is (= :retrying (get-in @store [:persistence :status])))
|
|
(rx/end! timer-s))))
|
|
(fn [_ _]
|
|
(if (= 1 (swap! calls inc))
|
|
(rx/throw (ex-info "offline" {:type :offline}))
|
|
(rx/subject)))))))
|
|
|
|
;; Scenario: the same file stalls twice, more than a governor window apart
|
|
;; (scripted clock). Each stall emits its own report: the second is a new
|
|
;; episode, not a suppressed duplicate of the first. Proves: identical
|
|
;; stall causes coalesce only inside the window.
|
|
;;
|
|
;; Note: unlike `with-watchdog`, the real `submit-report` runs here (only
|
|
;; `generate-report` stays doubled), so the test exercises the governor,
|
|
;; the thunk contract and the emit path end to end.
|
|
(t/deftest ^:async repeated-stalls-across-windows-emit-each-report
|
|
(let [clock (atom 0)
|
|
ticks (rx/subject)
|
|
response (rx/subject)
|
|
requests (atom [])
|
|
causes (atom [])
|
|
events (atom [])
|
|
file-id (uuid/next)
|
|
store (ptk/store {:state {:current-file-id file-id
|
|
:permissions {:can-edit true}
|
|
:files {file-id {:id file-id :revn 0}}}
|
|
:on-error #(t/is false (str %))})]
|
|
(errors/reset-report-governor!)
|
|
(await
|
|
(mock/with-mocks*
|
|
{ct/now (mock/stub #(ct/inst @clock))
|
|
rx/interval (mock/stub (fn [_] ticks))
|
|
rp/cmd! (mock/stub (fn [cmd params]
|
|
(swap! requests conj {:cmd cmd :params params})
|
|
(->> response (rx/take 1) (rx/observe-on :async))))
|
|
st/state store
|
|
errors/generate-report (fn [cause & _]
|
|
(swap! causes conj cause)
|
|
"report")
|
|
st/emit! (mock/stub (fn [& emitted] (swap! events into emitted)))
|
|
rt/get-current-href (constantly "https://penpot.example.com/#/workspace")
|
|
st/ongoing-tasks (atom #{})}
|
|
(try
|
|
(ptk/emit! store (dps/initialize-persistence))
|
|
;; Phase 1 — the first stall reports once.
|
|
(ptk/emit! store (local-commit file-id) ::dps/force-persist)
|
|
(reset! clock 300001)
|
|
(rx/push! ticks :tick)
|
|
(await (async/wait-for #(= 1 (count (audit-events @events)))
|
|
"first stall reports"))
|
|
(t/is (= 1 (count @requests)))
|
|
;; Phase 2 — the save lands, resetting the stall clock; a new
|
|
;; edit stalls again, over a governor window later.
|
|
(rx/push! response {:revn 1})
|
|
(await (async/wait-for #(= :saved (get-in @store [:persistence :status]))
|
|
"first save lands"))
|
|
(ptk/emit! store (local-commit file-id) ::dps/force-persist)
|
|
(await (async/wait-for #(= 2 (count @requests)) "second edit is sent"))
|
|
(reset! clock (+ 300001 errors/report-window-ms 300001))
|
|
(rx/push! ticks :tick)
|
|
(await (async/wait-for #(= 2 (count (audit-events @events)))
|
|
"second stall reports"))
|
|
(let [audits (audit-events @events)]
|
|
(t/is (= 2 (count audits)))
|
|
(t/is (= 2 (count @causes)) "each granted stall builds its report")
|
|
(t/is (every? #(= 1 (:occurrences (deref %))) audits)
|
|
"each stall is a fresh emission, never a coalesced repeat"))
|
|
(finally
|
|
(rx/dispose! store)
|
|
(rx/end! ticks)
|
|
(rx/end! response)))))))
|
|
|
|
;; Scenario: the first send never answers. The request timeout fails it as
|
|
;; a transient error, and the retry resends the same commit and saves. The
|
|
;; timeout is shortened by wrapping the real operator. Proves: a hung save
|
|
;; does not block the queue.
|
|
(t/deftest ^:async hung-save-request-times-out-and-retries
|
|
(let [calls (atom 0)
|
|
timeouts (atom [])
|
|
timeout rx/timeout]
|
|
(await
|
|
(with-persistence
|
|
(^:async fn [{:keys [file-id requests store]}]
|
|
(await
|
|
(mock/with-mocks*
|
|
{rx/timer (mock/stub instant-backoff)
|
|
rx/timeout (mock/stub (fn [ms with ob]
|
|
(swap! timeouts conj ms)
|
|
(timeout 0 with ob)))}
|
|
(ptk/emit! store (local-commit file-id) ::dps/force-persist)
|
|
(await (async/wait-for #(and (= :saved (get-in @store [:persistence :status]))
|
|
(empty? (get-in @store [:persistence :queue])))
|
|
"the retry saves the file"))
|
|
(t/is (= 2 (count @requests)) "hung send plus one retry")
|
|
(t/is (apply = (map (comp :commit-id second) @requests))
|
|
"both sends carry the same commit id")
|
|
(t/is (= [420000 420000] @timeouts) "every send gets the 7 minute timeout"))))
|
|
(fn [_ _]
|
|
(if (= 1 (swap! calls inc))
|
|
(rx/subject)
|
|
(rx/of {:revn 1})))))))
|
|
|
|
;; ---------------------------------------------------------------------------
|
|
;; slow cycle
|
|
;; ---------------------------------------------------------------------------
|
|
|
|
(defn- slow-cycle-timer
|
|
"Timer double: the backoff delays fire at once, and every slow cycle waits
|
|
on a subject collected in `cycles`, so the test fires them one by one."
|
|
[cycles]
|
|
(fn [ms]
|
|
(if (= ms dps/slow-retry-delay-ms)
|
|
(let [cycle-s (rx/subject)]
|
|
(swap! cycles conj cycle-s)
|
|
cycle-s)
|
|
(rx/of :tick))))
|
|
|
|
;; Scenario: the backoff is spent while the network is down. The queue keeps
|
|
;; trying on the slow cycle, one send per cycle, without warning the user
|
|
;; again, and saves once the network is back. Proves: a session outlasting
|
|
;; the backoff still recovers on its own.
|
|
(t/deftest ^:async a-spent-backoff-keeps-trying-on-a-slow-cycle
|
|
(let [calls (atom 0)
|
|
cycles (atom [])]
|
|
(await
|
|
(with-persistence
|
|
(^:async fn [{:keys [file-id failures requests store]}]
|
|
(await
|
|
(mock/with-mocks*
|
|
{rx/timer (mock/stub (slow-cycle-timer cycles))}
|
|
(ptk/emit! store (local-commit file-id) ::dps/force-persist)
|
|
(await (async/wait-for #(and (= :error (get-in @store [:persistence :status]))
|
|
(= 1 (count @cycles)))
|
|
"the spent backoff arms the slow cycle"))
|
|
(t/is (= 4 (count @requests)))
|
|
(t/is (= 1 (count @failures)))
|
|
|
|
(rx/push! (last @cycles) :tick)
|
|
(await (async/wait-for #(= 2 (count @cycles))
|
|
"a failed slow attempt arms the next cycle"))
|
|
(t/is (= 5 (count @requests)) "a slow cycle sends once")
|
|
(t/is (= :error (get-in @store [:persistence :status])))
|
|
(t/is (= 1 (count @failures)) "the user is not warned again")
|
|
|
|
;; The warning the first failure left on screen.
|
|
(ptk/emit! store #(assoc % :notification {:tag errors/persistence-failed-tag
|
|
:level :error}))
|
|
(rx/push! (last @cycles) :tick)
|
|
(await (async/wait-for #(and (= :saved (get-in @store [:persistence :status]))
|
|
(empty? (get-in @store [:persistence :queue])))
|
|
"the slow attempt saves"))
|
|
(t/is (= 6 (count @requests)))
|
|
(t/is (apply = (map (comp :commit-id second) @requests))
|
|
"every attempt carries the same commit id")
|
|
(t/is (nil? (get-in @store [:persistence :recovering])))
|
|
(t/is (nil? (get-in @store [:persistence :failing-since])))
|
|
(t/is (nil? (get @store :notification))
|
|
"the failure warning goes away once the save lands"))))
|
|
(fn [_ _]
|
|
(if (< (swap! calls inc) 6)
|
|
(rx/throw (ex-info "offline" {:type :offline}))
|
|
(rx/of {:revn 1})))))))
|
|
|
|
;; Scenario: a gateway error outlasts the backoff, then the proxy recovers.
|
|
;; Proves: each gateway status takes the slow cycle and saves the held edit.
|
|
(t/deftest ^:async a-gateway-error-arms-the-slow-cycle
|
|
(doseq [status [504 520 521 522 523 524]]
|
|
(let [calls (atom 0)
|
|
cycles (atom [])]
|
|
(await
|
|
(with-persistence
|
|
(^:async fn [{:keys [file-id requests store]}]
|
|
(await
|
|
(mock/with-mocks*
|
|
{rx/timer (mock/stub (slow-cycle-timer cycles))}
|
|
(ptk/emit! store (local-commit file-id) ::dps/force-persist)
|
|
(await (async/wait-for #(= 1 (count @cycles))
|
|
(str status " arms the slow cycle")))
|
|
(t/is (= 4 (count @requests)))
|
|
(t/is (= :gateway-error (get-in @store [:persistence :error :cause-type])))
|
|
(t/is (= 1 (count (get-in @store [:persistence :queue]))))
|
|
|
|
(rx/push! (last @cycles) :tick)
|
|
(await (async/wait-for #(and (= :saved (get-in @store [:persistence :status]))
|
|
(empty? (get-in @store [:persistence :queue])))
|
|
(str status " recovers on the slow cycle")))
|
|
(t/is (= 5 (count @requests)))
|
|
(t/is (apply = (map (comp :commit-id second) @requests))
|
|
"the slow cycle saves the held commit"))))
|
|
(fn [_ _]
|
|
(if (<= (swap! calls inc) 4)
|
|
(rp/handle-response {:status status :body "<html>Gateway error</html>" :uri "/api"})
|
|
(rx/of {:revn 1}))))))))
|
|
|
|
;; Scenario: a save fails for a reason no attempt can get past. Proves: only
|
|
;; transport failures arm the slow cycle.
|
|
(t/deftest ^:async a-terminal-failure-does-not-arm-the-slow-cycle
|
|
(let [cycles (atom [])]
|
|
(await
|
|
(with-persistence
|
|
(^:async fn [{:keys [file-id requests store]}]
|
|
(await
|
|
(mock/with-mocks*
|
|
{rx/timer (mock/stub (slow-cycle-timer cycles))}
|
|
(ptk/emit! store (local-commit file-id) ::dps/force-persist)
|
|
(await (async/wait-for #(= :error (get-in @store [:persistence :status]))
|
|
"the save fails"))
|
|
(await (async/settle))
|
|
(t/is (= 1 (count @requests)))
|
|
(t/is (empty? @cycles)))))
|
|
(fn [_ _] (rx/throw (ex-info "Validation failed" {:type :validation})))))))
|
|
|
|
;; Scenario: a gateway answers 504 only after its own timeout, while the
|
|
;; backend may still be applying the save. Proves: slow failures below the
|
|
;; limit still arm the slow cycle; at the limit the save stops being sent,
|
|
;; and opening a file releases it instead of sending it again.
|
|
(t/deftest ^:async slow-failures-stop-the-slow-cycle-at-the-limit
|
|
(doseq [[slow-count armed?] [[(dec dps/max-slow-failures) true]
|
|
[dps/max-slow-failures false]]]
|
|
(let [clock (atom 0)
|
|
calls (atom 0)
|
|
cycles (atom [])]
|
|
(await
|
|
(with-persistence
|
|
(^:async fn [{:keys [file-id requests store]}]
|
|
(await
|
|
(mock/with-mocks*
|
|
{rx/timer (mock/stub (slow-cycle-timer cycles))
|
|
ct/now (mock/stub #(ct/inst @clock))}
|
|
(ptk/emit! store (local-commit file-id) ::dps/force-persist)
|
|
(await (async/wait-for #(= :error (get-in @store [:persistence :status]))
|
|
(str slow-count " slow failures spend the backoff")))
|
|
(await (async/settle))
|
|
(t/is (= 4 (count @requests)))
|
|
(t/is (= slow-count (get-in @store [:persistence :slow-failures])))
|
|
(t/is (= (if armed? 1 0) (count @cycles))
|
|
(str slow-count " slow failures " (if armed? "arm" "do not arm") " the slow cycle"))
|
|
|
|
(when-not armed?
|
|
(ptk/emit! store (ptk/data-event :app.main.data.common/change-team-role))
|
|
(await (async/wait-for #(empty? (get-in @store [:persistence :queue]))
|
|
"recovery releases the held edit"))
|
|
(t/is (= 4 (count @requests)) "the released edit is not sent again")))))
|
|
(fn [_ _]
|
|
(when (<= (swap! calls inc) slow-count)
|
|
(swap! clock + dps/slow-failure-ms))
|
|
(rp/handle-response {:status 504 :body "<html>Gateway timeout</html>" :uri "/api"})))))))
|
|
|
|
;; Scenario: the slow cycle is still failing when the retry window closes.
|
|
;; Proves: the cycle gives up, keeping the edits queued.
|
|
(t/deftest ^:async the-slow-cycle-stops-when-the-retry-window-closes
|
|
(let [clock (atom 0)
|
|
cycles (atom [])]
|
|
(await
|
|
(with-persistence
|
|
(^:async fn [{:keys [file-id requests store]}]
|
|
(await
|
|
(mock/with-mocks*
|
|
{rx/timer (mock/stub (slow-cycle-timer cycles))
|
|
ct/now (mock/stub #(ct/inst @clock))}
|
|
(ptk/emit! store (local-commit file-id) ::dps/force-persist)
|
|
(await (async/wait-for #(= 1 (count @cycles)) "the slow cycle is armed"))
|
|
|
|
(reset! clock (+ dps/retry-give-up-ms 1000))
|
|
(rx/push! (last @cycles) :tick)
|
|
(await (async/wait-for #(= 5 (count @requests)) "the last slow attempt"))
|
|
(await (async/wait-for #(= :error (get-in @store [:persistence :status]))
|
|
"the attempt fails"))
|
|
(await (async/settle))
|
|
(t/is (= 1 (count @cycles)) "no further cycle is armed")
|
|
(t/is (= 1 (count (get-in @store [:persistence :queue])))))))
|
|
(fn [_ _] (rx/throw (ex-info "offline" {:type :offline})))))))
|
|
|
|
;; Scenario: the backoff is spent and the user leaves the workspace, which
|
|
;; stops the slow cycle, then opens a file again. Recovery sends the queue
|
|
;; once more instead of releasing it, and it saves. Proves: edits held by a
|
|
;; transport failure are never dropped by recovery.
|
|
(t/deftest ^:async recovery-resends-a-transport-failure-instead-of-releasing-it
|
|
(let [calls (atom 0)
|
|
cycles (atom [])]
|
|
(await
|
|
(with-persistence
|
|
(^:async fn [{:keys [file-id requests store]}]
|
|
(await
|
|
(mock/with-mocks*
|
|
{rx/timer (mock/stub (slow-cycle-timer cycles))}
|
|
(ptk/emit! store (local-commit file-id) ::dps/force-persist)
|
|
(await (async/wait-for #(= 1 (count @cycles)) "the slow cycle is armed"))
|
|
(ptk/emit! store (ptk/data-event :app.main.data.workspace/finalize-workspace))
|
|
(ptk/emit! store (ptk/data-event :app.main.data.common/change-team-role))
|
|
(await (async/wait-for #(and (= :saved (get-in @store [:persistence :status]))
|
|
(empty? (get-in @store [:persistence :queue])))
|
|
"recovery saves the held edit"))
|
|
(t/is (= 5 (count @requests)))
|
|
(t/is (apply = (map (comp :commit-id second) @requests))
|
|
"recovery sends the same commit, not a released queue"))))
|
|
(fn [_ _]
|
|
(if (< (swap! calls inc) 5)
|
|
(rx/throw (ex-info "offline" {:type :offline}))
|
|
(rx/of {:revn 1})))))))
|
|
|
|
;; Leaving the workspace tests.
|
|
|
|
;; Scenario: a save fails on a transport error and the user leaves the
|
|
;; workspace before the backoff fires. The pending retry stays silent, and
|
|
;; opening a file sends the held edit again. Proves: the backoff does not
|
|
;; run outside the workspace and resumes once a file is opened.
|
|
(t/deftest ^:async leaving-the-workspace-stops-the-backoff
|
|
(let [calls (atom 0)
|
|
timers (atom [])]
|
|
(await
|
|
(with-persistence
|
|
(^:async fn [{:keys [file-id requests store]}]
|
|
(await
|
|
(mock/with-mocks*
|
|
{rx/timer (mock/stub (fn [_]
|
|
(let [timer-s (rx/subject)]
|
|
(swap! timers conj timer-s)
|
|
timer-s)))}
|
|
(ptk/emit! store (local-commit file-id) ::dps/force-persist)
|
|
(await (async/wait-for #(= 1 (count @timers)) "the failure schedules a retry"))
|
|
|
|
(ptk/emit! store (ptk/data-event :app.main.data.workspace/finalize-workspace))
|
|
(await (async/wait-for #(get-in @store [:persistence :suspended])
|
|
"leaving the workspace suspends retries"))
|
|
(rx/push! (first @timers) :tick)
|
|
(await (async/settle))
|
|
(t/is (= 1 (count @requests)) "the pending retry stays silent")
|
|
(t/is (= :retrying (get-in @store [:persistence :status])))
|
|
(t/is (= 1 (count (get-in @store [:persistence :queue]))))
|
|
|
|
(ptk/emit! store (ptk/data-event :app.main.data.workspace/workspace-initialized))
|
|
(await (async/wait-for #(and (= :saved (get-in @store [:persistence :status]))
|
|
(empty? (get-in @store [:persistence :queue])))
|
|
"opening a file saves the held edit"))
|
|
(t/is (= 2 (count @requests)))
|
|
(t/is (apply = (map (comp :commit-id second) @requests))
|
|
"the same commit is sent again")
|
|
(t/is (nil? (get-in @store [:persistence :suspended]))))))
|
|
(fn [_ _]
|
|
(if (= 1 (swap! calls inc))
|
|
(rx/throw (ex-info "offline" {:type :offline}))
|
|
(rx/of {:revn 1})))))))
|
|
|
|
;; Scenario: the user leaves the workspace while a save is in flight, and the
|
|
;; save then fails on a transport error. No backoff is scheduled, the online
|
|
;; event does not resend, and opening a file does. Proves: a failure that
|
|
;; lands outside the workspace waits for the user instead of retrying there.
|
|
(t/deftest ^:async a-failure-after-leaving-the-workspace-arms-no-retry
|
|
(let [calls (atom 0)
|
|
timers (atom [])
|
|
first-s (rx/subject)]
|
|
(await
|
|
(with-persistence
|
|
(^:async fn [{:keys [file-id requests store]}]
|
|
(await
|
|
(mock/with-mocks*
|
|
{rx/timer (mock/stub (fn [ms] (swap! timers conj ms) (rx/subject)))}
|
|
(ptk/emit! store (local-commit file-id) ::dps/force-persist)
|
|
(await (async/wait-for #(= 1 (count @requests)) "the edit is sent"))
|
|
|
|
(ptk/emit! store (ptk/data-event :app.main.data.workspace/finalize-workspace))
|
|
(await (async/wait-for #(get-in @store [:persistence :suspended])
|
|
"leaving the workspace suspends retries"))
|
|
(rx/error! first-s (ex-info "offline" {:type :offline}))
|
|
(await (async/wait-for #(= :retrying (get-in @store [:persistence :status]))
|
|
"the failure keeps the edit queued"))
|
|
(await (async/settle))
|
|
(t/is (empty? @timers) "no backoff is scheduled")
|
|
|
|
(ptk/emit! store (#'dps/resume-on-online))
|
|
(await (async/settle))
|
|
(t/is (= 1 (count @requests)) "the online event does not resend")
|
|
|
|
(ptk/emit! store (ptk/data-event :app.main.data.workspace/workspace-initialized))
|
|
(await (async/wait-for #(and (= :saved (get-in @store [:persistence :status]))
|
|
(empty? (get-in @store [:persistence :queue])))
|
|
"opening a file saves the held edit"))
|
|
(t/is (= 2 (count @requests))))))
|
|
(fn [_ _]
|
|
(if (= 1 (swap! calls inc))
|
|
first-s
|
|
(rx/of {:revn 1})))))))
|
|
|
|
;; Scenario: the last backoff attempt is in flight when the user leaves the
|
|
;; workspace, and it fails. The queue halts without arming the slow cycle,
|
|
;; and opening a file sends it once more. Proves: the slow cycle never
|
|
;; starts outside the workspace.
|
|
(t/deftest ^:async a-spent-backoff-outside-the-workspace-arms-no-slow-cycle
|
|
(let [calls (atom 0)
|
|
cycles (atom [])
|
|
last-s (rx/subject)]
|
|
(await
|
|
(with-persistence
|
|
(^:async fn [{:keys [file-id requests store]}]
|
|
(await
|
|
(mock/with-mocks*
|
|
{rx/timer (mock/stub (slow-cycle-timer cycles))}
|
|
(ptk/emit! store (local-commit file-id) ::dps/force-persist)
|
|
(await (async/wait-for #(= 4 (count @requests)) "the last retry is sent"))
|
|
|
|
(ptk/emit! store (ptk/data-event :app.main.data.workspace/finalize-workspace))
|
|
(await (async/wait-for #(get-in @store [:persistence :suspended])
|
|
"leaving the workspace suspends retries"))
|
|
(rx/error! last-s (ex-info "offline" {:type :offline}))
|
|
(await (async/wait-for #(= :error (get-in @store [:persistence :status]))
|
|
"the spent backoff halts the queue"))
|
|
(await (async/settle))
|
|
(t/is (empty? @cycles) "no slow cycle is armed")
|
|
(t/is (= 1 (count (get-in @store [:persistence :queue]))))
|
|
|
|
(ptk/emit! store (ptk/data-event :app.main.data.workspace/workspace-initialized))
|
|
(await (async/wait-for #(and (= :saved (get-in @store [:persistence :status]))
|
|
(empty? (get-in @store [:persistence :queue])))
|
|
"opening a file saves the held edit"))
|
|
(t/is (= 5 (count @requests))))))
|
|
(fn [_ _]
|
|
(let [n (swap! calls inc)]
|
|
(cond
|
|
(< n 4) (rx/throw (ex-info "offline" {:type :offline}))
|
|
(= n 4) last-s
|
|
:else (rx/of {:revn 1}))))))))
|