mirror of
https://github.com/penpot/penpot.git
synced 2026-11-18 16:12:24 +00:00
✨ Add pgjdbc prepared statement cache settings (#13099)
* ✨ Add pgjdbc prepared statement cache settings Expose pgjdbc's server-side prepared statement cache as three optional backend variables: PENPOT_DATABASE_PREPARE_THRESHOLD, PENPOT_DATABASE_PREPARED_STATEMENT_CACHE_QUERIES and PENPOT_DATABASE_PREPARED_STATEMENT_CACHE_SIZE (bytes, converted to whole MiB for the driver, rounding up). When unset, the driver defaults apply (5 executions, 256 entries, 5 MiB), so the behavior is unchanged. The values reach Hikari as dataSourceProperties strings, because pgjdbc only reads String values; flushCacheOnDdl stays at its default. Closes #13092 AI-assisted-by: inkling-2-small-free * ♻️ Sync the pool options schema with the code The integrant pool options schema validated a ::connect-timeout option that nothing reads, while the keys the code consumes (::connection-timeout, ::idle-timeout, ::max-lifetime) were missing from it. The schema now declares the three, matching the code and the defaults map. The removed key was not referenced anywhere else. Follow-up to #13092. AI-assisted-by: inkling-2-small-free * ♻️ Rename the prepared statement cache threshold setting Keep the three knobs under one name in the configuration: PENPOT_DATABASE_PREPARED_STATEMENT_CACHE_THRESHOLD PENPOT_DATABASE_PREPARED_STATEMENT_CACHE_QUERIES PENPOT_DATABASE_PREPARED_STATEMENT_CACHE_SIZE The old PENPOT_DATABASE_PREPARE_THRESHOLD read as a verb and was the only one of the three that did not name the prepared statement cache. Fix the documentation of the values: 0 in either cache value disables the cache, not only the size, and -1 prepares a statement from its first execution and forces binary transfer, which is what pgjdbc does. Cover the -1 threshold end to end: it already decoded and validated, but nothing proved that it reaches the driver. The tests also pin 0, so a falsy check cannot silently drop it. Closes #13092 AI-assisted-by: deepseek-v4.1-flash * ♻️ Fix the prepared statement cache contracts The threshold schema accepted any integer, but the driver clamps every negative to -1 and turns on binary transfer, so a typo like -10 changed the wire protocol instead of failing at boot. Both schemas now declare {:min -1}: -1 and the non-negatives stay valid. The tests compared our own dataSourceProperties literals with our own literals, which cannot catch the failure the feature fears (a property the driver ignores). They now read the threshold back from a real connection: 5 with no options, 1 and 0 when configured. The size ceiling is covered one byte over a MiB, and queries=0 too. The docs, the app.db comment and the memory described the threshold 0 as disabling the cache. It does not: 0 stops the server-side prepared statements while the driver's cache of parsed queries stays in use, bounded by the other two values. The same texts lost the pg_prepared_statements pointer (the view only lists the session that runs the query) together with the flushCacheOnDdl and PgBouncer sentences, and the section now carries its version marker. Closes #13092 AI-assisted-by: deepseek-v4.1-flash * 💄 Format the prepared statement cache pool test The continuation arguments of the two with-connection calls were aligned under the options map instead of two spaces past the form, which cljfmt rejects. Local verification missed it because `./scripts/ci backend --test` runs the tests only: lint and format are separate tasks, so the pre-push check is the whole `./scripts/ci backend` or an explicit --lint/--fmt. AI-assisted-by: deepseek-v4.1-flash * ♻️ Address the review of the prepared statement cache The pool startup line shows the three values, so the configured cache is visible without reading the environment back; a nil means the driver default. bytes->mib-ceil uses Math/ceilDiv: the same rounding in one expression, and the size schema already rejects negatives. The cache-limit tests count a closed statement in pg_prepared_statements instead of comparing the property literals. A closed statement stays prepared in the server only while the driver's cache holds it, which is what tells an accepted property from an effective one; the positive case keeps the cache enabled with a sub-MiB size. The config test drops the groups that only exercised the decoder and keeps the ones that pin the accepted range. The manual and the memory note that a query parameter of the same name in PENPOT_DATABASE_URI wins over the variable: pgjdbc ignores the Properties value when the URL carries it. AI-assisted-by: deepseek-v4.1-flash AI-assisted-by: claude-opus-5-5
This commit is contained in:
parent
e0832335c7
commit
e6eb8bcd5a
@ -51,6 +51,7 @@
|
||||
|
||||
- `app.config/config` and `flags` are dynamic `defonce` vars populated from `PENPOT_*` env vars through the shared schema string transformer. Tests and tooling can bind them.
|
||||
- Duration config values are parsed as ISO-8601 with the `PT` prefix, so a day is written in hours (`:jobs-retention` and `:jobs-user-ttl` default to a week, and an operator writes `168h`): `7d` does not parse, and the value fails validation at boot instead of being silently ignored.
|
||||
- `app.main` wires `PENPOT_DATABASE_*` into `::db/pool`, and `app.db/create-datasource-config` forwards pgjdbc's own knobs as `dataSourceProperties`, which `Properties.getProperty` silently drops unless the value is a String (hence the `str`). A query parameter of the same name in `PENPOT_DATABASE_URI` (for example `?prepareThreshold=0`) wins over the property: pgjdbc ignores the Properties value when the URL carries it, so the URI is the place to look when a variable seems to do nothing. The prepared statement cache is three optional `PENPOT_DATABASE_PREPARED_STATEMENT_CACHE_*` keys: the threshold is the executions before a statement is server-prepared and accepts `-1` or non-negatives (`{:min -1}`; the driver clamps any other negative to `-1`, so rejecting it only turns a typo into a boot error), its `0` stops server-side prepared statements while the driver's client-side cache of parsed queries stays in use, `..._QUERIES` and `..._SIZE` reject negatives and `0` in either disables that cache, and the size is bytes rounded up to the whole MiB the driver accepts. Do not describe the threshold `0` as disabling the cache: it only stops the server-side prepares.
|
||||
- `parse-flags` automatically adds `:disable-secure-session-cookies` when `public-uri` is plain HTTP and not localhost. This changes cookie defaults without an explicit env flag.
|
||||
- The backend sets Clojure `*assert*` globally from the `:backend-asserts` feature flag. Assertion-dependent checks can therefore differ by runtime flags.
|
||||
- Request body parsing is mostly POST-oriented and supports Transit JSON plus plain JSON. Plain JSON request keys are kebab-decoded before being merged into `:params`. A body the parser cannot read is a client error, classified at the parse site and never by exception class (`clojure.data.json` signals its syntax errors with a bare `java.lang.Exception`): 400 `:validation`/`:malformed-json`, one `warn` line (`malformed request body`) carrying path, peer address and content-type instead of a stack trace, and `penpot_http_malformed_requests_total`. `RequestTooBigException` keeps its 413.
|
||||
|
||||
@ -241,6 +241,12 @@
|
||||
[:database-readonly {:optional true} ::sm/boolean]
|
||||
[:database-min-pool-size {:optional true} ::sm/int]
|
||||
[:database-max-pool-size {:optional true} ::sm/int]
|
||||
;; pgjdbc prepared statement cache (see app.db). Size is in bytes and is
|
||||
;; converted to the whole MiB the driver accepts. Absent keys mean the
|
||||
;; driver defaults; see docs/technical-guide/configuration.md.
|
||||
[:database-prepared-statement-cache-threshold {:optional true} [::sm/int {:min -1}]]
|
||||
[:database-prepared-statement-cache-queries {:optional true} [::sm/int {:min 0}]]
|
||||
[:database-prepared-statement-cache-size {:optional true} [::sm/int {:min 0}]]
|
||||
|
||||
[:quotes-teams-per-profile {:optional true} ::sm/int]
|
||||
[:quotes-access-tokens-per-profile {:optional true} ::sm/int]
|
||||
|
||||
@ -57,7 +57,9 @@
|
||||
|
||||
(def ^:private schema:pool-options
|
||||
[:map {:title "pool-options"}
|
||||
[::connect-timeout {:optional true} ::sm/int]
|
||||
[::connection-timeout {:optional true} ::sm/int]
|
||||
[::idle-timeout {:optional true} ::sm/int]
|
||||
[::max-lifetime {:optional true} ::sm/int]
|
||||
[::max-size {:optional true} ::sm/int]
|
||||
[::min-size {:optional true} ::sm/int]
|
||||
[::name {:optional true} :keyword]
|
||||
@ -66,6 +68,17 @@
|
||||
[::username {:optional true} :string]
|
||||
[::validation-timeout {:optional true} ::sm/int]
|
||||
[::read-only {:optional true} ::sm/boolean]
|
||||
;; pgjdbc prepared statement cache. Absent keys mean the driver
|
||||
;; defaults; the size is in bytes and is converted to whole MiB for
|
||||
;; the driver (see create-datasource-config). The threshold is the
|
||||
;; executions before a statement is server-prepared: 0 stops
|
||||
;; server-side prepared statements (the cache of parsed queries stays
|
||||
;; in use) and -1 prepares from the first execution. Any other
|
||||
;; negative is a typo the driver would clamp to -1, so the schema
|
||||
;; rejects it.
|
||||
[::prepared-statement-cache-threshold {:optional true} [::sm/int {:min -1}]]
|
||||
[::prepared-statement-cache-queries {:optional true} [::sm/int {:min 0}]]
|
||||
[::prepared-statement-cache-size {:optional true} [::sm/int {:min 0}]]
|
||||
[::mtx/metrics ::mtx/metrics]])
|
||||
|
||||
(def defaults
|
||||
@ -94,7 +107,10 @@
|
||||
:credentials (and (contains? cfg ::username)
|
||||
(contains? cfg ::password))
|
||||
:min-size (::min-size cfg)
|
||||
:max-size (::max-size cfg))
|
||||
:max-size (::max-size cfg)
|
||||
:prepared-statement-cache-threshold (::prepared-statement-cache-threshold cfg)
|
||||
:prepared-statement-cache-queries (::prepared-statement-cache-queries cfg)
|
||||
:prepared-statement-cache-size (::prepared-statement-cache-size cfg))
|
||||
(create-pool cfg))))
|
||||
|
||||
(defmethod ig/halt-key! ::pool
|
||||
@ -113,6 +129,17 @@
|
||||
;; pairwise, when both are partitioned with the same bounds
|
||||
"SET enable_partitionwise_join = on;"))
|
||||
|
||||
;; pgjdbc reads dataSourceProperties through Properties.getProperty, which
|
||||
;; drops non-String values silently, so everything is passed as a string.
|
||||
;; preparedStatementCacheSizeMiB only accepts a whole-MiB integer: the
|
||||
;; configured size (bytes) is converted up, so any positive value keeps the
|
||||
;; cache enabled and 0 disables the cache. A query parameter of the same name
|
||||
;; in the URI wins over these properties: pgjdbc ignores the Properties value
|
||||
;; when the URL carries it.
|
||||
(defn- bytes->mib-ceil
|
||||
[bytes]
|
||||
(Math/ceilDiv (long bytes) (* 1024 1024)))
|
||||
|
||||
(defn- create-datasource-config
|
||||
[{:keys [::uri] :as cfg}]
|
||||
|
||||
@ -132,6 +159,13 @@
|
||||
(.setConnectionInitSql initsql)
|
||||
(.setInitializationFailTimeout -1))
|
||||
|
||||
(some->> (::prepared-statement-cache-threshold cfg) (str)
|
||||
(.addDataSourceProperty config "prepareThreshold"))
|
||||
(some->> (::prepared-statement-cache-queries cfg) (str)
|
||||
(.addDataSourceProperty config "preparedStatementCacheQueries"))
|
||||
(some->> (::prepared-statement-cache-size cfg) (bytes->mib-ceil) (str)
|
||||
(.addDataSourceProperty config "preparedStatementCacheSizeMiB"))
|
||||
|
||||
(->> (mtx/get-registry (::mtx/metrics cfg))
|
||||
(PrometheusMetricsTrackerFactory.)
|
||||
(.setMetricsTrackerFactory config))
|
||||
|
||||
@ -425,6 +425,9 @@
|
||||
::db/read-only (cf/get :database-readonly false)
|
||||
::db/min-size (cf/get :database-min-pool-size)
|
||||
::db/max-size (cf/get :database-max-pool-size)
|
||||
::db/prepared-statement-cache-threshold (cf/get :database-prepared-statement-cache-threshold)
|
||||
::db/prepared-statement-cache-queries (cf/get :database-prepared-statement-cache-queries)
|
||||
::db/prepared-statement-cache-size (cf/get :database-prepared-statement-cache-size)
|
||||
::mtx/metrics (ig/ref ::mtx/metrics)}
|
||||
|
||||
;; Default netty IO pool (shared between several services)
|
||||
|
||||
@ -76,6 +76,23 @@
|
||||
:worker-default-parallelism 4)]
|
||||
(t/is (= 4 (cf/get :worker-default-parallelism 1))))))
|
||||
|
||||
(t/deftest prepared-statement-cache-keys
|
||||
(t/testing "negative sizes are rejected; the threshold takes -1 or non-negatives"
|
||||
(t/is (not (cf/validate-config (assoc cf/config :database-prepared-statement-cache-queries -1))))
|
||||
(t/is (not (cf/validate-config (assoc cf/config :database-prepared-statement-cache-size -1))))
|
||||
(t/is (not (cf/validate-config (assoc cf/config :database-prepared-statement-cache-threshold -2))))
|
||||
(t/is (cf/validate-config (assoc cf/config :database-prepared-statement-cache-threshold -1))))
|
||||
(t/testing "a -1 threshold survives the environment decoding and validation"
|
||||
(let [cfg (merge cf/config
|
||||
(cf/decode-config {:database-prepared-statement-cache-threshold "-1"}))]
|
||||
(t/is (= -1 (:database-prepared-statement-cache-threshold cfg)))
|
||||
(t/is (cf/validate-config cfg))))
|
||||
(t/testing "a decoded -2 threshold fails validation"
|
||||
(let [cfg (merge cf/config
|
||||
(cf/decode-config {:database-prepared-statement-cache-threshold "-2"}))]
|
||||
(t/is (= -2 (:database-prepared-statement-cache-threshold cfg)))
|
||||
(t/is (not (cf/validate-config cfg))))))
|
||||
|
||||
(t/deftest jobs-user-ttl
|
||||
(t/testing "the env value is decoded as a duration"
|
||||
(let [decoded (cf/decode-config {:jobs-user-ttl "72h"})]
|
||||
|
||||
@ -36,6 +36,113 @@
|
||||
(do (Thread/sleep 10)
|
||||
(recur (dec attempt)))))))
|
||||
|
||||
(t/deftest pool-passes-prepared-statement-cache-properties
|
||||
;; The driver reads dataSourceProperties through Properties.getProperty,
|
||||
;; which keeps only String values, so the wiring must set them as strings.
|
||||
;; Comparing the literals alone cannot catch a property the driver ignores,
|
||||
;; so the threshold cases also read it back from a real connection.
|
||||
(let [metrics (ig/init-key :app.metrics/metrics {:default {}})
|
||||
init-pool (fn [opts]
|
||||
(ig/init-key :app.db/pool
|
||||
(merge {:app.db/uri (java.net.URI. "postgresql://localhost/test")
|
||||
:app.metrics/metrics metrics
|
||||
:app.db/name :test-prepared-statement-cache}
|
||||
opts)))
|
||||
with-connection (fn [opts f]
|
||||
(let [ds (init-pool (merge {:app.db/uri (:database-uri th/config)
|
||||
:app.db/username (:database-username th/config)
|
||||
:app.db/password (:database-password th/config)}
|
||||
opts))]
|
||||
(try
|
||||
(with-open [conn (.getConnection ^HikariDataSource ds)]
|
||||
(f conn))
|
||||
(finally
|
||||
(.close ^HikariDataSource ds)))))
|
||||
prepare-threshold (fn [conn]
|
||||
(.getPrepareThreshold
|
||||
^org.postgresql.PGConnection
|
||||
(.unwrap conn org.postgresql.PGConnection)))]
|
||||
(t/testing "absent keys stay absent"
|
||||
(let [ds (init-pool {})]
|
||||
(t/is (empty? (.getDataSourceProperties ^HikariDataSource ds)))
|
||||
(.close ^HikariDataSource ds)))
|
||||
(t/testing "the driver defaults apply when no option is set"
|
||||
(with-connection {} (fn [conn] (t/is (= 5 (prepare-threshold conn))))))
|
||||
(t/testing "configured keys reach the driver as strings"
|
||||
(let [ds (init-pool {:app.db/prepared-statement-cache-threshold 1
|
||||
:app.db/prepared-statement-cache-queries 512
|
||||
:app.db/prepared-statement-cache-size 10485760})
|
||||
props (.getDataSourceProperties ^HikariDataSource ds)]
|
||||
(t/is (= "1" (.getProperty props "prepareThreshold")))
|
||||
(t/is (= "512" (.getProperty props "preparedStatementCacheQueries")))
|
||||
(t/is (= "10" (.getProperty props "preparedStatementCacheSizeMiB")))
|
||||
(.close ^HikariDataSource ds)))
|
||||
(t/testing "a configured threshold reaches the driver"
|
||||
(with-connection {:app.db/prepared-statement-cache-threshold 1}
|
||||
(fn [conn] (t/is (= 1 (prepare-threshold conn)))))
|
||||
(with-connection {:app.db/prepared-statement-cache-threshold 0}
|
||||
(fn [conn] (t/is (= 0 (prepare-threshold conn))))))
|
||||
(t/testing "a -1 threshold reaches the driver (forceBinary)"
|
||||
(let [ds (init-pool {:app.db/prepared-statement-cache-threshold -1})]
|
||||
(t/is (= "-1" (.getProperty (.getDataSourceProperties ^HikariDataSource ds)
|
||||
"prepareThreshold")))
|
||||
(.close ^HikariDataSource ds)))
|
||||
(t/testing "a zero threshold reaches the driver (server-side statements off)"
|
||||
(let [ds (init-pool {:app.db/prepared-statement-cache-threshold 0})]
|
||||
(t/is (= "0" (.getProperty (.getDataSourceProperties ^HikariDataSource ds)
|
||||
"prepareThreshold")))
|
||||
(.close ^HikariDataSource ds)))
|
||||
(t/testing "a positive sub-MiB size rounds up so the cache stays enabled"
|
||||
(let [ds (init-pool {:app.db/prepared-statement-cache-size 1})]
|
||||
(t/is (= "1" (.getProperty (.getDataSourceProperties ^HikariDataSource ds)
|
||||
"preparedStatementCacheSizeMiB")))
|
||||
(.close ^HikariDataSource ds)))
|
||||
(t/testing "the MiB ceiling rounds a byte over the boundary up"
|
||||
(let [ds (init-pool {:app.db/prepared-statement-cache-size 1048577})]
|
||||
(t/is (= "2" (.getProperty (.getDataSourceProperties ^HikariDataSource ds)
|
||||
"preparedStatementCacheSizeMiB")))
|
||||
(.close ^HikariDataSource ds)))
|
||||
(t/testing "the cache limits hold a closed statement in the server"
|
||||
;; A closed statement stays prepared in the server only while the
|
||||
;; driver's cache holds it, so counting it in pg_prepared_statements is
|
||||
;; what tells an accepted property from an effective one.
|
||||
(let [cached? (fn [opts]
|
||||
(with-connection
|
||||
(merge {:app.db/prepared-statement-cache-threshold 1} opts)
|
||||
(fn [conn]
|
||||
(dotimes [_ 2]
|
||||
(with-open [stmt (.prepareStatement ^Connection conn "SELECT ?::int AS cache_probe")]
|
||||
(.setInt stmt 1 1)
|
||||
(with-open [rs (.executeQuery stmt)] (.next rs))))
|
||||
(-> (db/exec-one! conn ["SELECT count(*) AS n FROM pg_prepared_statements
|
||||
WHERE statement LIKE '%AS cache_probe'"])
|
||||
:n
|
||||
pos?))))]
|
||||
(t/testing "a positive sub-MiB size keeps the cache enabled"
|
||||
(t/is (cached? {:app.db/prepared-statement-cache-size 1})))
|
||||
(t/testing "zero size disables the cache"
|
||||
(t/is (not (cached? {:app.db/prepared-statement-cache-size 0}))))
|
||||
(t/testing "zero queries disables the cache"
|
||||
(t/is (not (cached? {:app.db/prepared-statement-cache-queries 0}))))))))
|
||||
|
||||
(t/deftest pool-rejects-a-non-minus-one-negative-threshold
|
||||
;; The pool boundary mirrors app.config: -1 prepares from the first
|
||||
;; execution, 0 stops server-side prepared statements, and any other
|
||||
;; negative is a typo the driver would silently clamp to -1.
|
||||
(let [options {:app.db/uri (:database-uri th/config)
|
||||
:app.db/username (:database-username th/config)
|
||||
:app.db/password (:database-password th/config)
|
||||
:app.metrics/metrics (ig/init-key :app.metrics/metrics {:default {}})}]
|
||||
(t/testing "a -2 threshold fails the pool schema"
|
||||
(t/is (thrown? clojure.lang.ExceptionInfo
|
||||
(ig/assert-key :app.db/pool
|
||||
(assoc options :app.db/prepared-statement-cache-threshold -2)))))
|
||||
(t/testing "-1 and 0 pass the pool schema"
|
||||
(t/is (nil? (ig/assert-key :app.db/pool
|
||||
(assoc options :app.db/prepared-statement-cache-threshold -1))))
|
||||
(t/is (nil? (ig/assert-key :app.db/pool
|
||||
(assoc options :app.db/prepared-statement-cache-threshold 0)))))))
|
||||
|
||||
(t/deftest pool-requires-metrics
|
||||
;; Metrics is required by the pool schema: a pool wired without it
|
||||
;; fails at the boundary (AssertionError when asserts are enabled) or
|
||||
|
||||
@ -780,6 +780,33 @@ If the PostgreSQL server is shared with other applications, make sure
|
||||
`PENPOT_DATABASE_MAX_POOL_SIZE` plus the connections used by everything else stays below
|
||||
the server's `max_connections` (100 by default).
|
||||
|
||||
#### Prepared statement cache
|
||||
|
||||
__Since version 2.20.0__
|
||||
|
||||
The driver keeps two things per connection: a cache of parsed queries and, once a query
|
||||
runs enough times, a statement prepared in the server whose plan the server can reuse.
|
||||
Both are on by default; you only need these settings to tune them or to turn them off:
|
||||
|
||||
```bash
|
||||
# Backend
|
||||
PENPOT_DATABASE_PREPARED_STATEMENT_CACHE_THRESHOLD: 5
|
||||
PENPOT_DATABASE_PREPARED_STATEMENT_CACHE_QUERIES: 256
|
||||
PENPOT_DATABASE_PREPARED_STATEMENT_CACHE_SIZE: 5242880
|
||||
```
|
||||
|
||||
The values above are the driver defaults. `PENPOT_DATABASE_PREPARED_STATEMENT_CACHE_THRESHOLD`
|
||||
is how many times a query runs on a connection before the driver prepares it server-side:
|
||||
`-1` prepares it from the first execution and forces binary transfer, and `0` stops the
|
||||
driver from preparing statements in the server. The cache of parsed queries stays in use
|
||||
with the threshold at `0`: the connection caches up to
|
||||
`PENPOT_DATABASE_PREPARED_STATEMENT_CACHE_QUERIES` queries, using at most
|
||||
`PENPOT_DATABASE_PREPARED_STATEMENT_CACHE_SIZE` bytes per connection (the size is rounded
|
||||
up to whole MiB for the driver), and setting either of those two to `0` disables the
|
||||
cache. With 60 connections, the ceiling is 60 × the size. Leave a variable unset and the
|
||||
driver default applies. A parameter of the same name in `PENPOT_DATABASE_URI`, such as
|
||||
`?prepareThreshold=0`, takes precedence over these variables.
|
||||
|
||||
### Storage
|
||||
|
||||
Storage refers to storing the user uploaded different objects in Penpot (assets, file data,...).
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user