mirror of
https://github.com/penpot/penpot.git
synced 2026-10-29 22:10:34 +00:00
42f6c952e1
The media-storage-bytes-per-team quota ran a full SUM over storage_object on every upload, taking seconds on large teams. A new team_storage_usage table (migration 0158) caches the counter: a third audit-log consumer (storage-usage) selectively queues the events that change media storage, marks their teams dirty, and a scheduler turns the marks into per-team recompute jobs on a dedicated :storage queue. The quota reads the counter, always answers one row, and fails open when the row is missing; the three commands that write media bytes pay it. The dirty-team stream runs in one read-only transaction and submits each 50-team chunk through its own writable one (db/own-transaction), so a killed run keeps what it already enqueued and the next tick resumes the rest. The recompute only clears marks older than its read, and the query also returns the rows nobody refreshed for a week, so a lost mark heals at a bounded rate instead of never. Producers resolve team ids at emission with one batched lookup, so the drain never joins production tables: file-gc carries its team, the delete cascade reports flat and nested ids, moves carry explicit source/destination teams (mixed origins rejected), restores carry theirs, and purges carry only their entity key. Selective queueing is gated per consumer flag on both ingest paths, and telemetry shadows never queue the admin-console consumer. An audit event that cannot be stored now fails loudly instead of leaving a consumer stale. The delete cascade owns library relations: the RPC only marks the file deleted, and the job captures dependents, removes the relations and absorbs them under a savepoint, only for survivors. Jobs queued by the previous release still absorb through a deprecated :file-ids snapshot. Closes #12462 AI-assisted-by: muse-spark-1.3-contributor
83 lines
3.3 KiB
Clojure
83 lines
3.3 KiB
Clojure
;; This Source Code Form is subject to the terms of the Mozilla Public
|
|
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
|
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
|
;;
|
|
;; Copyright (c) KALEIDOS INC
|
|
|
|
(ns backend-tests.worker-test
|
|
(:require
|
|
[app.common.schema :as sm]
|
|
[app.config :as cf]
|
|
[app.db :as db]
|
|
[app.jobs :as jobs]
|
|
[app.main :as main]
|
|
[app.metrics :as-alias mtx]
|
|
[app.msgbus :as mbus]
|
|
[app.worker :as wrk]
|
|
[app.worker.cron :as wcron]
|
|
[app.worker.dispatcher :as wdisp]
|
|
[app.worker.runner :as wrkr]
|
|
[clojure.test :as t]))
|
|
|
|
;; The dispatcher's push and the runner's pop agree on the Redis list
|
|
;; only through this helper, so the test helpers use it too. That
|
|
;; keeps producer and consumer in sync by construction, but it also
|
|
;; means no other test would catch a malformed shape. This test pins
|
|
;; the physical format instead.
|
|
|
|
(t/deftest queue-key-pins-the-physical-redis-shape
|
|
(t/testing "tenant and queue compose the documented key"
|
|
(t/is (= "penpot.worker.queue:acme:default"
|
|
(wrk/queue-key "acme" :default))))
|
|
(t/testing "a string queue gives the same key as a keyword one"
|
|
(t/is (= "penpot.worker.queue:acme:default"
|
|
(wrk/queue-key "acme" "default"))))
|
|
(t/testing "hyphens travel through untouched"
|
|
(t/is (= "penpot.worker.queue:my-tenant:daily-cleanup"
|
|
(wrk/queue-key "my-tenant" :daily-cleanup)))))
|
|
|
|
;; Booting the backend asserts these integrant keys, and assert-key
|
|
;; compiles the schema first: a ref with no registered schema crashes
|
|
;; the boot with :malli.core/invalid-schema instead of asserting. The
|
|
;; component schemas lean on the shared registry (db pool, metrics,
|
|
;; redis client, msgbus, job defs), so the registry dependencies must
|
|
;; be loaded for them to compile, like the full system does.
|
|
(t/deftest worker-component-schemas-resolve-every-ref
|
|
(t/testing "cron, dispatcher and runner schemas compile"
|
|
(t/is (sm/schema? (sm/schema ::mbus/msgbus)))
|
|
(doseq [s [@#'wcron/schema:params
|
|
@#'wdisp/schema:dispatcher
|
|
@#'wrkr/schema:params]]
|
|
(t/is (sm/schema? (sm/schema s))))))
|
|
|
|
(t/deftest the-storage-runner-is-wired
|
|
(let [cfg (get main/worker-config [:app.main/storage ::wrk/runner])]
|
|
|
|
(t/is (some? cfg))
|
|
|
|
(t/testing "the slow per-team SUM does not share the default runner"
|
|
(t/is (= :storage (::wrk/queue cfg)))
|
|
(t/is (= (cf/get :worker-storage-parallelism 2) (::wrk/parallelism cfg))))
|
|
|
|
(t/testing "and the runner resolves jobs, metrics and the pool"
|
|
(t/is (contains? cfg ::jobs/defs))
|
|
(t/is (contains? cfg ::mtx/metrics))
|
|
(t/is (contains? cfg ::db/pool)))))
|
|
|
|
(t/deftest the-binfile-runner-is-wired
|
|
(let [cfg (get main/worker-config [:app.main/binfile ::wrk/runner])]
|
|
|
|
(t/is (some? cfg))
|
|
|
|
(t/testing "the heavy work of a user job does not share the default runner"
|
|
(t/is (= :binfile (::wrk/queue cfg)))
|
|
(t/is (= (cf/get :worker-binfile-parallelism 1) (::wrk/parallelism cfg))))
|
|
|
|
(t/testing "a job of a profile cannot store its events without a msgbus"
|
|
(t/is (contains? cfg ::mbus/msgbus)))
|
|
|
|
(t/testing "and the runner resolves jobs, metrics and the pool"
|
|
(t/is (contains? cfg ::jobs/defs))
|
|
(t/is (contains? cfg ::mtx/metrics))
|
|
(t/is (contains? cfg ::db/pool)))))
|