mirror of
https://github.com/penpot/penpot.git
synced 2026-11-10 03:50:33 +00:00
ba78ccb8c9
Take the file row with FOR UPDATE SKIP LOCKED when the offload job starts and skip the run when the row is locked or gone, so the job never runs while the file is being edited. Every live hot-storage row of the file moves to cold storage inside one all-or-nothing transaction, and the handler answers a summary map with the offloaded rows, bytes and per-type stats. Add bounded metrics for the job: rows and bytes counters labeled by row type on successful runs only, and a timing histogram with the ok, skip and error outcomes. The job definition now requires metrics, the recorders live in app.jobs.metrics, and the existing ephemeral job timing histogram pins its buckets. Cover the job with a new test namespace: the happy path, the locked row, missing and deleted files, the rows and bytes summary with its no-op rerun, the whole-file abort on storage failure, the metric definitions and emissions, and the submit-time params validation. An end-to-end suite reads, snapshots, writes, exports, imports and deletes offloaded files; concurrency tests pin the stale-revn skip, the blocked-edit retry and the reclaimable orphans of a partial failure. The file-gc scheduler test pins one job per eligible file, and the scheduler private helper drops its bang. Document the lock invariant, the deliberate no-pacing decision and the metric label sets in the backend storage memory and the configuration guide. Related to #12398. AI-assisted-by: muse-spark-1.3-contributor
44 lines
2.0 KiB
Clojure
44 lines
2.0 KiB
Clojure
;; This Source Code Form is subject to the terms of the Mozilla Public
|
|
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
|
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
|
;;
|
|
;; Copyright (c) KALEIDOS INC
|
|
|
|
(ns backend-tests.file-gc-scheduler-test
|
|
(:require
|
|
[app.common.time :as ct]
|
|
[app.jobs :as jobs]
|
|
[backend-tests.helpers :as th]
|
|
[clojure.test :as t]))
|
|
|
|
(t/use-fixtures :once th/state-init)
|
|
(t/use-fixtures :each th/database-reset)
|
|
|
|
(t/deftest min-age-validation
|
|
(let [cfg th/*system*]
|
|
(t/testing "int millis, text and absence validate"
|
|
(t/is (uuid? (jobs/submit cfg {::jobs/name :file-gc-scheduler
|
|
::jobs/params {:min-age 3600000}})))
|
|
(t/is (uuid? (jobs/submit cfg {::jobs/name :file-gc-scheduler
|
|
::jobs/params {:min-age "1h"}})))
|
|
(t/is (uuid? (jobs/submit cfg {::jobs/name :file-gc-scheduler
|
|
::jobs/params {}}))))
|
|
(t/testing "garbage fails at submit instead of burning retries"
|
|
(t/is (thrown-with-msg? Exception #"check error"
|
|
(jobs/submit cfg {::jobs/name :file-gc-scheduler
|
|
::jobs/params {:min-age {:bogus true}}}))))))
|
|
|
|
(t/deftest scheduler-submits-one-job-per-eligible-file
|
|
(let [profile (th/create-profile* 1)]
|
|
(doseq [i [1 2 3]]
|
|
(let [file (th/create-file* i {:profile-id (:id profile)
|
|
:project-id (:default-project-id profile)})]
|
|
;; Make the files deterministically eligible: pending trim and
|
|
;; modified long enough ago.
|
|
(th/db-update! :file {:has-media-trimmed false
|
|
:modified-at (ct/in-past "2h")}
|
|
{:id (:id file)})))
|
|
(let [result (th/run-task! :file-gc-scheduler {:min-age "1h"})]
|
|
(t/is (= 3 (:processed result)))
|
|
(t/is (= 3 (count (th/db-query :job {:name "file-gc"})))))))
|