diff --git a/.serena/memories/frontend/workspace-state-persistence-subtleties.md b/.serena/memories/frontend/workspace-state-persistence-subtleties.md index 57094e1994..2ca4855b58 100644 --- a/.serena/memories/frontend/workspace-state-persistence-subtleties.md +++ b/.serena/memories/frontend/workspace-state-persistence-subtleties.md @@ -26,7 +26,7 @@ - A terminal failure halts the queue on purpose: the edits stay queued so `wait-persisted-or-error` keeps rejecting, and `workspace/versions.cljs` relies on that to avoid snapshotting a file whose edits were never saved. `recover-persistence` lifts the halt when a file is opened again or the role changes. A transport failure still inside its retry window (`slow-retry-pending?`) is resent once (`resume-persistence true`), never released; any other halt emits `release-failed-file` (drops every queued commit of the file that failed) then `resume-persistence`. Opening a file also replays the queued edits of its version onto the loaded data (`pending-file-changes` → `restore-pending-changes` in `bundle-fetched`); a replay that throws keeps the server data and the queue. That is what unblocks the editor without a reload; because the queue is shared, dropping the failed file is also what lets other files' pending edits drain. `release-failed-file` runs at recovery time, never at failure time — clearing the error eagerly would make waiters resolve. - The split between `initialize-persistence-state` and `recover-persistence` is deliberate: initialization deliberately does NOT lift a halt, so undeliverable edits are released when the user acts, not when the view mounts. - **Every non-transport terminal save failure is permanent.** There is no "stale but valid" category: permission lost, file gone, and server-rejected changes all fail again on a resend. Losing edit permission is not temporary — the backend raises `:not-found` for it (`check-edition-permissions!` → `perms/make-check-fn`), and `:not-found` routes to the exception page. So `:save-permission-denied` (a client-side `can-edit` check) must never be resumed: replaying the save to discover that routes the user to the exception page. Release it instead. -- Save failures split transient vs terminal (`transient-error?`: the repo retryable types `:network`/`:offline`/`:bad-gateway`/`:service-unavailable`/`:gateway-error` plus `:invalid-save-response`; everything else is terminal). Terminal keeps the `:error` halt + `flash-persistence` path; transient enters a `:retrying` episode: the head commit stays queued and resends with backoff 2s/8s/20s (3 retries, then the `:error` path). A transient failure that outlasts the backoff arms `slow-retry-cycle`: one silent resend (`:recovering`) every `slow-retry-delay-ms` (30s) until a save lands, `retry-give-up-ms` (24h since `:failing-since`) passes, or the head commit collects `max-slow-failures` (3) attempts that failed after `slow-failure-ms` (90s) or more (`:slow-failures`, counted in `persistence-transient-failure` from the attempt's elapsed time): a gateway timeout may leave the backend applying the save, so a save that keeps failing slowly must not be replayed for a day, while fast failures (offline) never count; the cycle stops on `::resume-persistence`, `::initialize-persistence` and `finalize-workspace`. Leaving the workspace also emits `suspend-retries` (`:suspended` in `:persistence`, drops `:retry-token` so a pending backoff timer goes stale): no backoff, slow cycle or `online` resend runs outside the workspace; only `workspace-initialized` (not a role change) lifts it via `recover-persistence` (`opened-file?`), which also resends a `:retrying` episode whose backoff was cut short. Its toast is `errors.save-still-retrying` without a reload link; a landing save hides the failure toast (`errors/persistence-failed-tag`). Replaying a save is safe because changes are idempotent, so there is no unknown-outcome guard: a commit whose earlier request left `active-requests` is simply sent again under the same `:commit-id`; only a request still in flight blocks a resend (`:in-flight` stays silent). Tests hold the slow cycle by stubbing `rx/timer` to never fire for `slow-retry-delay-ms`, or instant timers loop forever. Retry timers carry the episode token; a superseded token stays silent. Status stays `:retrying` through re-entries (`next-status` refuses `:pending`/`:saving` from it); waiters (`wait-persisted-or-error`) wait through it and reject only on `:error`. +- Save failures split transient vs terminal (`transient-error?`: the repo retryable types `:network`/`:offline`/`:bad-gateway`/`:service-unavailable`/`:gateway-error`, plus `:invalid-save-response`, plus save-only `busy-server-error?`: `:rate-limit` (429 from rlimit/climit — a save parked on another save's file lock holds the profile's single `:update-file/by-profile` permit, so the user's saves to other files 429 after 60 s) and backend-written 504 `:server-error` with `:statement-timeout`/`:idle-in-transaction-timeout` (Postgres cancels the lock wait at 300 s, before nginx; rolled back, so resend is safe). Kept out of `rp/retryable-types` on purpose so GETs do not retry them; everything else is terminal). Terminal keeps the `:error` halt + `flash-persistence` path; transient enters a `:retrying` episode: the head commit stays queued and resends with backoff 2s/8s/20s (3 retries, then the `:error` path). A transient failure that outlasts the backoff arms `slow-retry-cycle`: one silent resend (`:recovering`) every `slow-retry-delay-ms` (30s) until a save lands, `retry-give-up-ms` (24h since `:failing-since`) passes, or the head commit collects `max-slow-failures` (3) attempts that failed after `slow-failure-ms` (90s) or more (`:slow-failures`, counted in `persistence-transient-failure` from the attempt's elapsed time): a gateway timeout may leave the backend applying the save, so a save that keeps failing slowly must not be replayed for a day, while fast failures (offline) never count; the cycle stops on `::resume-persistence`, `::initialize-persistence` and `finalize-workspace`. Leaving the workspace also emits `suspend-retries` (`:suspended` in `:persistence`, drops `:retry-token` so a pending backoff timer goes stale): no backoff, slow cycle or `online` resend runs outside the workspace; only `workspace-initialized` (not a role change) lifts it via `recover-persistence` (`opened-file?`), which also resends a `:retrying` episode whose backoff was cut short. Its toast is `errors.save-still-retrying` without a reload link; a landing save hides the failure toast (`errors/persistence-failed-tag`). Replaying a save is safe because changes are idempotent, so there is no unknown-outcome guard: a commit whose earlier request left `active-requests` is simply sent again under the same `:commit-id`; only a request still in flight blocks a resend (`:in-flight` stays silent). Tests hold the slow cycle by stubbing `rx/timer` to never fire for `slow-retry-delay-ms`, or instant timers loop forever. Retry timers carry the episode token; a superseded token stays silent. Status stays `:retrying` through re-entries (`next-status` refuses `:pending`/`:saving` from it); waiters (`wait-persisted-or-error`) wait through it and reject only on `:error`. - One reconnect notice per episode: sticky toast tagged `:persistence-reconnecting` (single-toast store, re-show replaces), hidden by tag on drain (`:saved`) and on terminal failure; recovery is silent. Header indicator has a `:retrying` state (`workspace.header.retrying`). - Resume triggers: backoff timer and the browser `online` event (guarded by `exists? js/window`; re-enters the runner only for a live `:retrying` episode). New local edits during `:retrying` only join the queue: the episode keeps its `:run-id`, so `append-commit` does not re-enter, and the live runner (still waiting on the head's `commit-persisted`) sends them after the head. Re-entering on edits would bypass the backoff and spend an attempt per batched commit. - Tests instant-trigger retries by stubbing `rx/timer` (recording delays to assert the schedule); dynamic bindings do not survive `await` continuations, so no dynamic var for delays. diff --git a/frontend/src/app/main/data/persistence.cljs b/frontend/src/app/main/data/persistence.cljs index 76031e7c85..0a857da12c 100644 --- a/frontend/src/app/main/data/persistence.cljs +++ b/frontend/src/app/main/data/persistence.cljs @@ -138,17 +138,31 @@ (#{:error :saved} status) (dissoc :run-id :last-progress-at :stall-reported)))))))) +(def ^:private busy-server-codes + "Server error codes for a database timeout that rolled the save back." + #{:statement-timeout :idle-in-transaction-timeout}) + +(defn- busy-server-error? + "True when the backend was too busy to apply the save: it refused it + (429) or a database timeout rolled it back. Resending it is safe." + [{:keys [type cause-type code]}] + (let [type (or cause-type type)] + (or (= :rate-limit type) + (and (= :server-error type) + (contains? busy-server-codes code))))) + (defn transient-error? "True when a save failure is worth retrying with backoff: a transient transport failure (from the shared `repo/retryable-types` set, which says - nothing about the edits themselves) or an unusable save response. - Everything else (auth, validation, state) is terminal and keeps the - `:error` path. Checked against both `:type` and `:cause-type` because + nothing about the edits themselves), a busy backend, or an unusable save + response. Everything else (auth, validation, state) is terminal and keeps + the `:error` path. Checked against both `:type` and `:cause-type` because `persistence-failed` wraps the original cause under `:type :persistence`." - [{:keys [type cause-type code]}] + [{:keys [type cause-type code] :as error}] (boolean (or (contains? rp/retryable-types type) (contains? rp/retryable-types cause-type) + (busy-server-error? error) (= :invalid-save-response code)))) (def ^:private retry-delays-ms diff --git a/frontend/test/frontend_tests/data/persistence_retry_test.cljs b/frontend/test/frontend_tests/data/persistence_retry_test.cljs index 6bbfbedf47..1eba1bd1b5 100644 --- a/frontend/test/frontend_tests/data/persistence_retry_test.cljs +++ b/frontend/test/frontend_tests/data/persistence_retry_test.cljs @@ -199,6 +199,68 @@ (t/deftest ^:async cloudflare-origin-timeout-save-is-retried (await (check-gateway-response-retries 524))) +(defn- check-busy-server-response-retries + "Answers the first send with `response`, a refusal the backend wrote + because it was busy, built by the real `rp/handle-response`, and the retry + with a valid revision. The save was never applied, so it must take the + retry path and land, never halt in `:error`." + [response] + (let [calls (atom 0)] + (with-persistence + (^:async fn [{:keys [file-id failures requests store]}] + (await + (mock/with-mocks* + {rx/timer (mock/stub instant-backoff)} + (ptk/emit! store (local-commit file-id) ::dps/force-persist) + (await (async/wait-for #(and (= :saved (get-in @store [:persistence :status])) + (empty? (get-in @store [:persistence :queue]))) + (str "a busy " (:status response) " is retried and the save lands"))) + (t/is (= 2 (count @requests)) "the busy answer plus one retry") + (t/is (empty? @failures) "a retried busy answer never reaches the failure toast")))) + (fn [_ _] + (if (= 1 (swap! calls inc)) + (rp/handle-response (assoc response :uri "https://penpot.example.com/api/main/methods/update-file")) + (rx/of {:revn 1})))))) + +;; Variant: the save waited too long for a concurrency slot (no body). +(t/deftest ^:async concurrency-limit-save-is-retried + (await (check-busy-server-response-retries {:status 429 :body nil}))) + +;; Variant: the database cancelled the save while it waited for the file lock. +(t/deftest ^:async statement-timeout-save-is-retried + (await (check-busy-server-response-retries + {:status 504 + :body {:type :server-error + :code :statement-timeout + :hint "The operation took too long and was cancelled"}}))) + +;; Variant: the database ended the save's transaction after it sat idle. +(t/deftest ^:async idle-transaction-timeout-save-is-retried + (await (check-busy-server-response-retries + {:status 504 + :body {:type :server-error + :code :idle-in-transaction-timeout + :hint "The transaction was idle too long and was cancelled"}}))) + +;; Scenario: the backend fails the save for any other reason. Proves: only +;; the busy-server codes retry; a server defect stops saving at once. +(t/deftest ^:async other-server-errors-stay-terminal + (await + (with-persistence + (^:async fn [{:keys [file-id requests store]}] + (await + (mock/with-mocks* + {rx/timer (mock/stub instant-backoff)} + (ptk/emit! store (local-commit file-id) ::dps/force-persist) + (await (async/wait-for #(= :error (get-in @store [:persistence :status])) + "the save fails")) + (t/is (= :server-error (get-in @store [:persistence :error :cause-type]))) + (t/is (= 1 (count @requests)) "no retry")))) + (fn [_ _] + (rp/handle-response {:status 500 + :body {:type :server-error :code :unhandled} + :uri "https://penpot.example.com/api/main/methods/update-file"}))))) + ;; Scenario: every send fails transiently. The 2s / 8s / 20s retries fire ;; and then the failure falls through to the exact terminal path: `:error` ;; carrying the cause, queue intact, one flash. Proves: the budget bounds diff --git a/frontend/test/frontend_tests/data/persistence_test.cljs b/frontend/test/frontend_tests/data/persistence_test.cljs index 9735771305..f6555fa165 100644 --- a/frontend/test/frontend_tests/data/persistence_test.cljs +++ b/frontend/test/frontend_tests/data/persistence_test.cljs @@ -693,6 +693,19 @@ (t/is (not (dps/transient-error? {:type :persistence :code :save-permission-denied}))) (t/is (not (dps/transient-error? {})))) +;; A busy backend refused or rolled back the save, so it is safe to resend: +;; a rate or concurrency limit, or a database timeout. Any other server +;; error stays terminal. +(t/deftest busy-server-classification + (t/is (dps/transient-error? {:type :rate-limit})) + (t/is (dps/transient-error? {:type :persistence :cause-type :rate-limit :code :rate-limited})) + (t/is (dps/transient-error? {:type :server-error :code :statement-timeout})) + (t/is (dps/transient-error? {:type :persistence :cause-type :server-error :code :statement-timeout})) + (t/is (dps/transient-error? {:type :persistence :cause-type :server-error :code :idle-in-transaction-timeout})) + (t/is (not (dps/transient-error? {:type :server-error}))) + (t/is (not (dps/transient-error? {:type :persistence :cause-type :server-error :code :unhandled}))) + (t/is (not (dps/transient-error? {:type :persistence :cause-type :validation :code :statement-timeout})))) + ;; The transient set is owned by `repo/retryable-types`: every transport ;; type retryable at the HTTP layer must also be transient for the save ;; pipeline, in both the direct and the wrapped (`persistence-failed`) @@ -988,3 +1001,4 @@ (await (async/wait-for #(not (contains? @tasks :persistence)) "the release turns the guard off")) (t/is (= :saved (get-in @store [:persistence :status]))))))) +