mirror of
https://github.com/penpot/penpot.git
synced 2026-09-24 12:56:15 +00:00
452f38cf5d
* ✨ Add storage operation metrics for S3 and buckets Expose Prometheus metrics for the object storage subsystem. The S3 backend now attaches an AWS SDK MetricPublisher that counts API calls, retries and latency per operation and target. The storage layer counts logical operations and deduplication outcomes per Penpot bucket, and the assets handlers count served requests per route. Closes #11676 AI-assisted-by: muse-spark-1.3-contributor * ✨ Fix storage metrics labels, errors and test gaps Address the review findings on the storage metrics commit. Label reads with the object's own backend, count failed asset serving as errors without swallowing them, and cover the failed S3 call, S3 asset path and permission-denied branches with tests. Also share the label helper and reuse the metrics test helper. Closes #11676 AI-assisted-by: muse-spark-1.3-contributor * ✨ Harden storage metrics and fill test gaps Address the second-round review findings on storage metrics. Unknown backends now fail explicitly and count as errors, exists stays paired with its dedup outcome, and the thumbnail, missing storage, expired reads, unknown buckets and write failure paths are covered by tests. Label coercion goes through the shared metrics helper. Closes #11676 AI-assisted-by: muse-spark-1.3-contributor * ✨ Harden storage metrics accuracy and coverage Address the full-branch review findings on storage metrics. Touch and delete emit only on changed rows, reads emit after the backend fetch, unknown backends fail explicitly, and tempfile mismatches count as unauthorized. Publisher nil policy, pairing rules and attempt semantics are documented and covered by tests. Closes #11676 AI-assisted-by: muse-spark-1.3-contributor * ✨ Address full-branch review findings on storage metrics Touch and delete resolve labels from the row, reads stay paired, failures are covered by tests, and logging, ranges and docs are tightened. Includes the label helper unit tests and the retries wording clarification. Closes #11676 AI-assisted-by: muse-spark-1.3-contributor * ⚡ Label touch and del metrics from UPDATE RETURNING The storage metrics change resolved metric labels for touch-object! and del-object! with an extra SELECT per id-based call. Since app.main instruments storage unconditionally, every GC collector and binfile import paid that extra round trip: deleting a team with 10k media objects doubled the storage_object statements exactly on the paths that already process the most rows. touch-object! and del-object! now take only the object id (UUID) and read the labels from the updated row itself via RETURNING id, backend, metadata: one statement, no pre-read, and labels that always match the row actually mutated. del-object! additionally guards on deleted_at IS NULL, so a repeated delete returns false and emits no metric. Also from the review of the full branch: extract the duplicated serve/emit/rethrow block in app.http.assets into one helper; give penpot_storage_s3_timing explicit histogram buckets up to 60s (the default cap at 7.5s hid the slow S3 calls the metric exists for); drop the unused ::target-id config key from the S3 backend and hardcode the :default target label until per-bucket routing lands. AI-assisted-by: glm-5.3-flash * ✨ Harden storage metric recording and definitions The metric definition schema is now closed and declares every key the collectors read: buckets, quantiles, max-age and reg. A typo such as a misspelled ::mdef/buckets used to compile and silently fall back to the default histogram buckets; it now fails at startup. The asset result-label fallback coerced an absent status to 500, so a future serve path without a status would have counted successes as errors. The mapping is now explicit and documented: served below 400, unauthorized for 401/403, not-found for 404, and error for everything else, including an absent status. The never-fail try/catch around metric recording existed four times with drift. One app.metrics/run-safe! helper replaces them: it no-ops on a nil metrics instance and logs the first failure per hint at warn level, then at debug, so a broken setup surfaces once without flooding the log. The S3 publisher keeps its outer try/catch: it is the SDK MetricPublisher contract boundary. AI-assisted-by: glm-5.3-flash * ✨ Make metrics mandatory and run! safe by default Recording a metric must never change the behavior of the operation being measured, so `run!` now catches recording failures itself: the first failure per metric id logs at warn, later ones at debug. This replaces the `run-safe!` helper, whose four copies had drifted, and applies the guarantee to every emit site instead of only storage. The metrics instance precondition is a plain assert, and the collector lookup stays outside the recording guard, so a missing instance fails hard even when asserts are disabled. Metrics is therefore no longer optional: the storage, s3-backend and db-pool schemas require `::mtx/metrics`, and the assets handler cfg always carries it. `wrap-publisher` no longer returns nil for a nil instance, and the db pool wires the prometheus tracker unconditionally. AI-assisted-by: deepseek-v4.1-flash
65 lines
2.4 KiB
Clojure
65 lines
2.4 KiB
Clojure
;; This Source Code Form is subject to the terms of the Mozilla Public
|
|
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
|
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
|
;;
|
|
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
|
|
|
(ns backend-tests.db-test
|
|
(:require
|
|
[app.common.uuid :as uuid]
|
|
[app.db :as db]
|
|
[backend-tests.helpers :as th]
|
|
[clojure.test :as t]
|
|
[integrant.core :as ig])
|
|
(:import
|
|
com.zaxxer.hikari.HikariConfig
|
|
com.zaxxer.hikari.HikariDataSource
|
|
java.sql.Connection))
|
|
|
|
(t/use-fixtures :once th/state-init)
|
|
|
|
(t/deftest pool-requires-metrics
|
|
;; Metrics is required by the pool schema: a pool wired without it
|
|
;; fails at the boundary (AssertionError when asserts are enabled) or
|
|
;; when the prometheus tracker is wired, instead of silently skipping
|
|
;; the instrumentation.
|
|
(t/is (thrown? Throwable
|
|
(ig/init-key :app.db/pool
|
|
{::db/uri (java.net.URI. "postgresql://localhost/test")}))))
|
|
|
|
(t/deftest pool-stats-returns-expected-keys
|
|
(let [stats (db/pool-stats th/*pool*)]
|
|
(t/testing "all expected keys are present"
|
|
(t/is (contains? stats :active-connections))
|
|
(t/is (contains? stats :idle-connections))
|
|
(t/is (contains? stats :threads-awaiting-connection))
|
|
(t/is (contains? stats :total-connections))
|
|
(t/is (contains? stats :maximum-pool-size))
|
|
(t/is (contains? stats :minimum-idle)))
|
|
|
|
(t/testing "values are non-negative integers"
|
|
(t/is (>= (:active-connections stats) 0))
|
|
(t/is (>= (:idle-connections stats) 0))
|
|
(t/is (>= (:threads-awaiting-connection stats) 0))
|
|
(t/is (>= (:total-connections stats) 0))
|
|
(t/is (>= (:maximum-pool-size stats) 0))
|
|
(t/is (>= (:minimum-idle stats) 0)))
|
|
|
|
(t/testing "total connections equals active + idle"
|
|
(t/is (= (:total-connections stats)
|
|
(+ (:active-connections stats)
|
|
(:idle-connections stats)))))
|
|
|
|
(t/testing "maximum pool size is reasonable"
|
|
(t/is (pos? (:maximum-pool-size stats))))))
|
|
|
|
(t/deftest uuid->hash-code-is-deterministic
|
|
(t/is (= (db/uuid->hash-code uuid/zero)
|
|
(db/uuid->hash-code uuid/zero))))
|
|
|
|
(t/deftest uuid->hash-code-returns-long
|
|
(t/is (instance? Long (db/uuid->hash-code uuid/zero))))
|
|
|
|
(t/deftest uuid->hash-code-stable-for-zero-uuid
|
|
(t/is (= 3659997967308761462 (db/uuid->hash-code uuid/zero))))
|