Andrey Antukh e6eb8bcd5a
✨ Add pgjdbc prepared statement cache settings (#13099)
* ✨ Add pgjdbc prepared statement cache settings

Expose pgjdbc's server-side prepared statement cache as three
optional backend variables: PENPOT_DATABASE_PREPARE_THRESHOLD,
PENPOT_DATABASE_PREPARED_STATEMENT_CACHE_QUERIES and
PENPOT_DATABASE_PREPARED_STATEMENT_CACHE_SIZE (bytes, converted
to whole MiB for the driver, rounding up). When unset, the
driver defaults apply (5 executions, 256 entries, 5 MiB), so the
behavior is unchanged. The values reach Hikari as
dataSourceProperties strings, because pgjdbc only reads String
values; flushCacheOnDdl stays at its default.

Closes #13092

AI-assisted-by: inkling-2-small-free

* ♻️ Sync the pool options schema with the code

The integrant pool options schema validated a ::connect-timeout
option that nothing reads, while the keys the code consumes
(::connection-timeout, ::idle-timeout, ::max-lifetime) were
missing from it. The schema now declares the three, matching the
code and the defaults map. The removed key was not referenced
anywhere else.

Follow-up to #13092.

AI-assisted-by: inkling-2-small-free

* ♻️ Rename the prepared statement cache threshold setting

Keep the three knobs under one name in the configuration:

  PENPOT_DATABASE_PREPARED_STATEMENT_CACHE_THRESHOLD
  PENPOT_DATABASE_PREPARED_STATEMENT_CACHE_QUERIES
  PENPOT_DATABASE_PREPARED_STATEMENT_CACHE_SIZE

The old PENPOT_DATABASE_PREPARE_THRESHOLD read as a verb and was the only
one of the three that did not name the prepared statement cache.

Fix the documentation of the values: 0 in either cache value disables the
cache, not only the size, and -1 prepares a statement from its first
execution and forces binary transfer, which is what pgjdbc does.

Cover the -1 threshold end to end: it already decoded and validated, but
nothing proved that it reaches the driver. The tests also pin 0, so a
falsy check cannot silently drop it.

Closes #13092

AI-assisted-by: deepseek-v4.1-flash

* ♻️ Fix the prepared statement cache contracts

The threshold schema accepted any integer, but the driver clamps every
negative to -1 and turns on binary transfer, so a typo like -10 changed
the wire protocol instead of failing at boot. Both schemas now declare
{:min -1}: -1 and the non-negatives stay valid.

The tests compared our own dataSourceProperties literals with our own
literals, which cannot catch the failure the feature fears (a property
the driver ignores). They now read the threshold back from a real
connection: 5 with no options, 1 and 0 when configured. The size ceiling
is covered one byte over a MiB, and queries=0 too.

The docs, the app.db comment and the memory described the threshold 0 as
disabling the cache. It does not: 0 stops the server-side prepared
statements while the driver's cache of parsed queries stays in use,
bounded by the other two values. The same texts lost the
pg_prepared_statements pointer (the view only lists the session that runs
the query) together with the flushCacheOnDdl and PgBouncer sentences, and
the section now carries its version marker.

Closes #13092

AI-assisted-by: deepseek-v4.1-flash

* 💄 Format the prepared statement cache pool test

The continuation arguments of the two with-connection calls were aligned
under the options map instead of two spaces past the form, which cljfmt
rejects.

Local verification missed it because `./scripts/ci backend --test` runs
the tests only: lint and format are separate tasks, so the pre-push check
is the whole `./scripts/ci backend` or an explicit --lint/--fmt.

AI-assisted-by: deepseek-v4.1-flash

* ♻️ Address the review of the prepared statement cache

The pool startup line shows the three values, so the configured cache is
visible without reading the environment back; a nil means the driver
default.

bytes->mib-ceil uses Math/ceilDiv: the same rounding in one expression,
and the size schema already rejects negatives.

The cache-limit tests count a closed statement in pg_prepared_statements
instead of comparing the property literals. A closed statement stays
prepared in the server only while the driver's cache holds it, which is
what tells an accepted property from an effective one; the positive case
keeps the cache enabled with a sub-MiB size. The config test drops the
groups that only exercised the decoder and keeps the ones that pin the
accepted range.

The manual and the memory note that a query parameter of the same name in
PENPOT_DATABASE_URI wins over the variable: pgjdbc ignores the Properties
value when the URL carries it.

AI-assisted-by: deepseek-v4.1-flash
AI-assisted-by: claude-opus-5-5
2026-11-17 14:06:04 +01:00

198 lines
10 KiB
Clojure

;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns backend-tests.db-test
(:require
[app.common.uuid :as uuid]
[app.db :as db]
[backend-tests.helpers :as th]
[clojure.test :as t]
[integrant.core :as ig])
(:import
com.zaxxer.hikari.HikariConfig
com.zaxxer.hikari.HikariDataSource
java.sql.Connection))
(t/use-fixtures :once th/state-init)
(defn- get-consistent-pool-stats
"Read pool stats until total equals active + idle, or attempts run out.
Hikari exposes each counter through a separate MXBean call, so a borrow
or return between the reads can leave a single snapshot inconsistent
(total off by one). Retrying until the pool is quiescent keeps the test
deterministic without weakening the invariant."
[pool]
(loop [attempt 20]
(let [stats (db/pool-stats pool)
total (:total-connections stats)
active (:active-connections stats)
idle (:idle-connections stats)]
(if (or (= total (+ active idle))
(zero? attempt))
stats
(do (Thread/sleep 10)
(recur (dec attempt)))))))
(t/deftest pool-passes-prepared-statement-cache-properties
;; The driver reads dataSourceProperties through Properties.getProperty,
;; which keeps only String values, so the wiring must set them as strings.
;; Comparing the literals alone cannot catch a property the driver ignores,
;; so the threshold cases also read it back from a real connection.
(let [metrics (ig/init-key :app.metrics/metrics {:default {}})
init-pool (fn [opts]
(ig/init-key :app.db/pool
(merge {:app.db/uri (java.net.URI. "postgresql://localhost/test")
:app.metrics/metrics metrics
:app.db/name :test-prepared-statement-cache}
opts)))
with-connection (fn [opts f]
(let [ds (init-pool (merge {:app.db/uri (:database-uri th/config)
:app.db/username (:database-username th/config)
:app.db/password (:database-password th/config)}
opts))]
(try
(with-open [conn (.getConnection ^HikariDataSource ds)]
(f conn))
(finally
(.close ^HikariDataSource ds)))))
prepare-threshold (fn [conn]
(.getPrepareThreshold
^org.postgresql.PGConnection
(.unwrap conn org.postgresql.PGConnection)))]
(t/testing "absent keys stay absent"
(let [ds (init-pool {})]
(t/is (empty? (.getDataSourceProperties ^HikariDataSource ds)))
(.close ^HikariDataSource ds)))
(t/testing "the driver defaults apply when no option is set"
(with-connection {} (fn [conn] (t/is (= 5 (prepare-threshold conn))))))
(t/testing "configured keys reach the driver as strings"
(let [ds (init-pool {:app.db/prepared-statement-cache-threshold 1
:app.db/prepared-statement-cache-queries 512
:app.db/prepared-statement-cache-size 10485760})
props (.getDataSourceProperties ^HikariDataSource ds)]
(t/is (= "1" (.getProperty props "prepareThreshold")))
(t/is (= "512" (.getProperty props "preparedStatementCacheQueries")))
(t/is (= "10" (.getProperty props "preparedStatementCacheSizeMiB")))
(.close ^HikariDataSource ds)))
(t/testing "a configured threshold reaches the driver"
(with-connection {:app.db/prepared-statement-cache-threshold 1}
(fn [conn] (t/is (= 1 (prepare-threshold conn)))))
(with-connection {:app.db/prepared-statement-cache-threshold 0}
(fn [conn] (t/is (= 0 (prepare-threshold conn))))))
(t/testing "a -1 threshold reaches the driver (forceBinary)"
(let [ds (init-pool {:app.db/prepared-statement-cache-threshold -1})]
(t/is (= "-1" (.getProperty (.getDataSourceProperties ^HikariDataSource ds)
"prepareThreshold")))
(.close ^HikariDataSource ds)))
(t/testing "a zero threshold reaches the driver (server-side statements off)"
(let [ds (init-pool {:app.db/prepared-statement-cache-threshold 0})]
(t/is (= "0" (.getProperty (.getDataSourceProperties ^HikariDataSource ds)
"prepareThreshold")))
(.close ^HikariDataSource ds)))
(t/testing "a positive sub-MiB size rounds up so the cache stays enabled"
(let [ds (init-pool {:app.db/prepared-statement-cache-size 1})]
(t/is (= "1" (.getProperty (.getDataSourceProperties ^HikariDataSource ds)
"preparedStatementCacheSizeMiB")))
(.close ^HikariDataSource ds)))
(t/testing "the MiB ceiling rounds a byte over the boundary up"
(let [ds (init-pool {:app.db/prepared-statement-cache-size 1048577})]
(t/is (= "2" (.getProperty (.getDataSourceProperties ^HikariDataSource ds)
"preparedStatementCacheSizeMiB")))
(.close ^HikariDataSource ds)))
(t/testing "the cache limits hold a closed statement in the server"
;; A closed statement stays prepared in the server only while the
;; driver's cache holds it, so counting it in pg_prepared_statements is
;; what tells an accepted property from an effective one.
(let [cached? (fn [opts]
(with-connection
(merge {:app.db/prepared-statement-cache-threshold 1} opts)
(fn [conn]
(dotimes [_ 2]
(with-open [stmt (.prepareStatement ^Connection conn "SELECT ?::int AS cache_probe")]
(.setInt stmt 1 1)
(with-open [rs (.executeQuery stmt)] (.next rs))))
(-> (db/exec-one! conn ["SELECT count(*) AS n FROM pg_prepared_statements
WHERE statement LIKE '%AS cache_probe'"])
:n
pos?))))]
(t/testing "a positive sub-MiB size keeps the cache enabled"
(t/is (cached? {:app.db/prepared-statement-cache-size 1})))
(t/testing "zero size disables the cache"
(t/is (not (cached? {:app.db/prepared-statement-cache-size 0}))))
(t/testing "zero queries disables the cache"
(t/is (not (cached? {:app.db/prepared-statement-cache-queries 0}))))))))
(t/deftest pool-rejects-a-non-minus-one-negative-threshold
;; The pool boundary mirrors app.config: -1 prepares from the first
;; execution, 0 stops server-side prepared statements, and any other
;; negative is a typo the driver would silently clamp to -1.
(let [options {:app.db/uri (:database-uri th/config)
:app.db/username (:database-username th/config)
:app.db/password (:database-password th/config)
:app.metrics/metrics (ig/init-key :app.metrics/metrics {:default {}})}]
(t/testing "a -2 threshold fails the pool schema"
(t/is (thrown? clojure.lang.ExceptionInfo
(ig/assert-key :app.db/pool
(assoc options :app.db/prepared-statement-cache-threshold -2)))))
(t/testing "-1 and 0 pass the pool schema"
(t/is (nil? (ig/assert-key :app.db/pool
(assoc options :app.db/prepared-statement-cache-threshold -1))))
(t/is (nil? (ig/assert-key :app.db/pool
(assoc options :app.db/prepared-statement-cache-threshold 0)))))))
(t/deftest pool-requires-metrics
;; Metrics is required by the pool schema: a pool wired without it
;; fails at the boundary (AssertionError when asserts are enabled) or
;; when the prometheus tracker is wired, instead of silently skipping
;; the instrumentation.
(t/is (thrown? Throwable
(ig/init-key :app.db/pool
{::db/uri (java.net.URI. "postgresql://localhost/test")}))))
(t/deftest pool-stats-returns-expected-keys
(let [stats (get-consistent-pool-stats th/*pool*)]
(t/testing "all expected keys are present"
(t/is (contains? stats :active-connections))
(t/is (contains? stats :idle-connections))
(t/is (contains? stats :threads-awaiting-connection))
(t/is (contains? stats :total-connections))
(t/is (contains? stats :maximum-pool-size))
(t/is (contains? stats :minimum-idle)))
(t/testing "values are non-negative integers"
(t/is (>= (:active-connections stats) 0))
(t/is (>= (:idle-connections stats) 0))
(t/is (>= (:threads-awaiting-connection stats) 0))
(t/is (>= (:total-connections stats) 0))
(t/is (>= (:maximum-pool-size stats) 0))
(t/is (>= (:minimum-idle stats) 0)))
(t/testing "total connections equals active + idle"
(t/is (= (:total-connections stats)
(+ (:active-connections stats)
(:idle-connections stats)))))
(t/testing "maximum pool size is reasonable"
(t/is (pos? (:maximum-pool-size stats))))))
(t/deftest uuid->hash-code-is-deterministic
(t/is (= (db/uuid->hash-code uuid/zero)
(db/uuid->hash-code uuid/zero))))
(t/deftest uuid->hash-code-returns-long
(t/is (instance? Long (db/uuid->hash-code uuid/zero))))
(t/deftest uuid->hash-code-stable-for-zero-uuid
(t/is (= 3659997967308761462 (db/uuid->hash-code uuid/zero))))
(t/deftest pool-enables-the-partitionwise-join
;; The pool sets the GUC on every connection, so the planner can resolve a
;; join between two partitioned tables pairwise, partition by partition.
(let [row (db/exec-one! th/*pool*
["SELECT current_setting('enable_partitionwise_join') AS partitionwise_join"])]
(t/is (= "on" (:partitionwise-join row)))))