Andrey Antukh e23ca02aa2
✨ Reference the file data storage object with a foreign key (#12769)
* ✨ Reference the file data storage object with a foreign key

The file data pointer lived in `file_data.metadata.storage-ref-id`, a JSON
key. The storage GC parsed it to find live references, and nothing stopped it
from physically deleting an object a live row still pointed at: a failure in
that scan lost the blob.

Migration 0159 moves the pointer to a nullable `storage_object_id` column on
`file_data`, backfills it from the metadata key, removes the key from the
rows it backfills, indexes the column and rebuilds the foreign key to
`storage_object(id)` as `NO ACTION DEFERRABLE INITIALLY IMMEDIATE`.
`INITIALLY IMMEDIATE` is deliberate: the GC catches the violation inside the
chunk transaction, while a deferred check would fire at commit, outside it,
and the sweep would never move past the blocked chunk.

The column is the only source of truth. The `storage-ref-id` entry is gone
from the file metadata schema, and the single point that serializes the
metadata drops it, so no write can resurrect it. `upsert!` does not query
the row: the pointer of the row being replaced travels in its params, and
the object is touched before the row is written. Fragments and snapshots
pass nil because a modified pointer always gets a new id; the offload passes
its own row; `insert-file!` drops it, since an INSERT never replaces a
persisted object.

`update-file!` takes the file map read from the database plus the fields to
change and merges them, so the pointer comes from the read map and no caller
can lose it by building a partial map. A map without `:data` raises
`:missing-file-data`: it is not `assert`, which is compiled out in the
deployed binary. The params cannot override the pointer or retarget the
write with `:id`/`:file-id`.

`bfc/file->response` strips the pointer from every response (`get-file`,
`create-file`, `duplicate-file`, the view-only bundle, the team access
request, the comments read, the library summary, the snapshot read and the
debug download). A `file-data` object is authentication-only, so a leaked id
would let any logged-in profile fetch the blob from `/assets/by-id`.
`get-file-fragment` resolves through the file data backend, so a fragment on
cold storage no longer hands nil to `blob/decode`, and answers only the keys
of its result schema.

The storage GC reference scan, the restore union and `objects-gc` read the
column. The offload reads `SELECT fd.*`, so its rows go through a
`decode-row` that decodes jsonb metadata and drops a JSON null, which the row
schema rejected; a row without a readable object raises
`:missing-storage-object` with the file and row ids. A chunk blocked by the
foreign key (SQLState 23503) is a real anomaly, since the reference scan
should have frozen the object: it stays an error with its own hint and the
chunk is retried on the next run until the reference is fixed.

The release is a coordinated restart, not a rolling deploy, and not
rollback-safe for the rows it writes: the previous code writes and scans the
metadata pointer, so a key it writes misses the backfill and removing it
loses the reference. `operations/2.20.0-file-data-storage-object-id.md`
builds the column, the index and the foreign key online and runs the
backfill with every instance stopped, then marks the step as applied; a
fresh or small instance does the whole job in the migration. The runbook
keeps the validation scan out of the stopped window and adds a lock timeout
and retry to the online statements.

Tests cover the foreign key shape, the backfill and its corrupt and orphan
guards, the rebuild of a missing index and of a wrong, non-deferrable or
deferred constraint, the deletion refused by a live reference, the metadata
cleanup, the offload row normalizations, the touched objects of the unshare
and the restore, the response stripping on every call site and the chunk
blocked by the foreign key. The backend memories and the developer guide
document the pointer, the GC behavior and the two migration paths.

Closes #12768

AI-assisted-by: deepseek-v4.1-flash
AI-assisted-by: inkling-2-small-free

* 🐛 Hold the file row lock across the migration write

`get-file` applies the pending migrations in memory and, when it finds
any, persists the result. That write read the row without a lock and
`update-file!` is an unconditional UPDATE, so a save that committed in
between was overwritten: the saved content, `revn` and `modified-at`
went back, the next save from that client failed with `:revn-conflict`
and the object the save left became an orphan the storage GC cannot
revisit once it is frozen.

The read now runs in a transaction of its own (`get-file` calls
`get-file*` inside `db/tx-run!` and accepts a cfg map only), so the
`file` row lock covers the re-check and the write. `get-file*` reads
the row once (`sql:get-file` plus `resolve-applied-migrations`) and
completes it with `attach-file-data`; only when a migration is pending
it reads the file again under `FOR UPDATE`, re-checks `need-migration?`
on that fresh map and persists it (`migrate-file`). The common case
stays at one row read and one migrations resolution, the migration case
keeps a single decoded copy in the heap, and a row that changed in
between is not overwritten. A migration found on the unlocked read is
applied in memory only, for the next read to persist.

`migrate-file` returns the reattached map when it persists, with the
`:storage-object-id` of the row the write left, and the realized map
when it does not; `:realize?` is what returns the read fully plain.
`fdata/upsert!` answers `{:written ... :storage-object-id ...}` instead
of a boolean, `update-file!` propagates it and the migrating read puts
it back in the map it returns, so a caller that persists again touches
the current object instead of the replaced one. `insert-file!` keeps
nil: an INSERT never replaces a persisted object.

The write also compares the stored pointer with the caller's value
(`IS NOT DISTINCT FROM`) and raises `:stale-file-data-pointer` when it
affects no row. Every writer holds the `file` row lock, so a pointer
that no longer matches is a protocol violation, not a race to retry.
The expected pointer is a separate argument because the storage branch
replaces `:storage-object-id` before the upsert, and the check lives in
`upsert-in-database`: the `legacy-db` branches do not use that SQL and
their count can legitimately be zero.

The save, snapshot and restore-backup reads take `FOR UPDATE` instead
of `FOR SHARE`: they inherit the lock, and the migration branch asks
for the exclusive one in the same transaction, so a shared lock that
upgrades deadlocks two concurrent readers. A data read rejects
`:lock-for-share?` for the same reason; only row-only reads accept it.
`sql:get-file-without-data` is a plain select now, not a CTE, which
makes locking legal without loading the data. Realization has a single
owner: `attach-file-data` only resolves and decodes, and `get-file*`
realizes once per path. The applied migrations no longer travel in the
`get-file-with-summary` response, cache hit and cache miss alike.

`embed-assets` reads each library without `:realize?`, so its
`:components` is an unloaded pointer map; `absorb-assets` walks it with
`vals`, `PointerMap.seq` calls `load!` and that raised with no loader
bound, so every export with `:merge-libraries` of a library with
components failed. The absorb runs under the library's own loader now.

The in-place binfile import (`import-file-and-overwrite*`,
`::overwrite`, `::bfc/file-id`) had no product caller and cannot carry
the previous `storage_object_id`, so overwriting a storage-backed row
would leave its object unreclaimed. `import-files!` rejects a cfg with
`::bfc/file-id` (`:not-supported`) and the helpers only that path used
go with it (`invalidate-thumbnails`, `*reference-file*`, the
`::overwrite` id branch of `index-object` and of `save-file!`, the
`on-conflict-do-nothing?` flag of the media insert).

Tests: a new `file-migration-read-test` pins the single read, the
locked re-read, the concurrent save interleaving, the double migration
and the read-only path; the upsert tests cover the return shape and the
stale pointer; the SQL recorder they share lives in
`backend-tests.helpers`, and the save, snapshot, restore and summary
sequences follow the new reads. The backend memories document the read
contract and the pointer.

AI-assisted-by: deepseek-v4.1-flash

* 🐛 Harden the get-file guard and skip its read-only transaction

The cfg guard of `get-file` was an `assert`, which `app.config` elides
when `:backend-asserts` is off, so a deployed binary accepted a bare
pool or connection while the docstring promised it raises. It throws
`IllegalArgumentException` now, the class the option guards of the same
namespace already use, and the existing test expects it instead of
`AssertionError`.

A read with `:read-only?` applies the pending migrations in memory and
writes nothing, so the transaction the migration write needs only costs
two round trips there. `get-file` runs it with `db/run!` instead: with
a cfg that carries a connection it still joins the caller's
transaction, and only the reads that would have opened their own skip
it. A lock option needs a read-write transaction from the caller,
because PostgreSQL rejects row locks in a read-only one; the docstring
states that.

Tests: the guard test expects the new class; a cfg marked
`::db/read-only` reads a pending migration without persisting it (the
config flag that fills that key is legacy, but the key is set by hand);
and a read-only read is pinned as not opening a transaction while the
same call without the option does. The backend memory documents the
split and the guard.

AI-assisted-by: deepseek-v4.1-flash
2026-11-14 19:27:18 +01:00

940 lines
33 KiB
Clojure

;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns backend-tests.helpers
(:require
[app.auth]
[app.common.data :as d]
[app.common.data.macros :as dm]
[app.common.exceptions :as ex]
[app.common.features :as cfeat]
[app.common.flags :as flags]
[app.common.pprint :as pp]
[app.common.schema :as sm]
[app.common.spec :as us]
[app.common.time :as ct]
[app.common.transit :as tr]
[app.common.types.shape :as cts]
[app.common.uuid :as uuid]
[app.config :as cf]
[app.db :as db]
[app.jobs :as jobs]
[app.main :as main]
[app.media]
[app.media :as-alias mtx]
[app.metrics :as-alias metrics]
[app.migrations]
[app.msgbus :as-alias mbus]
[app.rpc :as-alias rpc]
[app.rpc.climit :as-alias climit]
[app.rpc.commands.auth :as cmd.auth]
[app.rpc.commands.files :as files]
[app.rpc.commands.files-create :as files.create]
[app.rpc.commands.files-update :as files.update]
[app.rpc.commands.projects :as projects]
[app.rpc.commands.teams :as teams]
[app.rpc.helpers :as rph]
[app.util.blob :as blob]
[app.util.services :as sv]
[app.worker.runner]
[clojure.java.io :as io]
[clojure.spec.alpha :as s]
[clojure.test :as t]
[cuerdas.core :as str]
[datoteka.fs :as fs]
[environ.core :refer [env]]
[expound.alpha :as expound]
[integrant.core :as ig]
[mockery.core :as mk]
[promesa.core :as p]
[promesa.exec :as px]
[ring.core.protocols :as rcp]
[yetti.request :as yreq]
[yetti.response :as yres])
(:import
java.io.PipedInputStream
java.io.PipedOutputStream
java.util.UUID
org.postgresql.ds.PGSimpleDataSource))
(def ^:dynamic *system* nil)
(def ^:dynamic *pool* nil)
;; Fallback values used when no PENPOT_TEST_* env vars are set (e.g. CI sets
;; them explicitly, see .github/workflows/tests-backend.yml). Inside the
;; devenv each wsN container receives per-instance values via
;; manage.sh (PENPOT_TEST_DATABASE_URI=postgresql://postgres/penpot_test_wsN,
;; PENPOT_TEST_REDIS_URI=redis://valkey/<6+N>), so parallel test runs on
;; different workspaces never share a database. cf/read-config picks the env
;; vars up through the "penpot-test" prefix.
(def default
{:database-uri "postgresql://postgres/penpot_test"
:redis-uri "redis://valkey/1"
:auto-file-snapshot-every 1
:file-data-backend "db"})
(def config
(cf/read-config :prefix "penpot-test"
:default (merge cf/default default)))
(def default-flags
[:enable-secure-session-cookies
:enable-email-verification
:enable-smtp
:enable-quotes
:enable-rpc-climit
:enable-auto-file-snapshot
:disable-file-validation
:disable-admin-console])
(defn init-config
([next]
(init-config nil next))
([extra-flags next]
(let [flags (into default-flags extra-flags)]
(with-redefs [app.config/flags (flags/parse flags/default flags)
app.config/config config
app.loggers.audit/submit (constantly nil)
app.auth/derive-password identity
app.auth/verify-password (fn [a b] {:valid (= a b)})
app.common.features/get-enabled-features
(fn [& _] app.common.features/supported-features)]
(cf/validate! :exit-on-error? false)
(fs/create-dir "/tmp/penpot")
(next)))))
(defn- remove-job-runners
"Drops every `[<profile> :app.worker/runner]` from the system.
The tests drive the runner loop by hand (`wdisp/run-batch` plus
`run-worker-loop` over a system they build themselves), so a live runner
of any queue races them: it takes the job the test just dispatched and
completes it behind the test's back. Matching the key by shape, instead
of listing the known queues by hand, keeps a new queue from silently
bringing that race back (which is what happened to the `binfile` runner
this key list was missing)."
[system]
(apply dissoc system
(filter #(and (vector? %)
(= :app.worker/runner (second %)))
(keys system))))
(defn init-system
[next]
(let [templates [{:id "test"
:name "test"
:file-uri "test"
:thumbnail-uri "test"
:path (-> "backend_tests/test_files/template.penpot" io/resource fs/path)}]
system (-> (merge main/system-config main/worker-config)
(assoc-in [:app.redis/client :app.redis/uri] (:redis-uri config))
(assoc-in [::db/pool ::db/uri] (:database-uri config))
(assoc-in [::db/pool ::db/username] (:database-username config))
(assoc-in [::db/pool ::db/password] (:database-password config))
(assoc-in [:app.rpc/methods :app.setup/templates] templates)
(assoc-in [:app.rpc/methods :app.setup/templates] templates)
(update :app.rpc/rlimit assoc
:app.loggers.mattermost/reporter nil
:app.loggers.database/reporter nil)
(update :app.rpc/methods assoc
:app.setup/templates templates
:app.loggers.mattermost/reporter nil
:app.loggers.database/reporter nil)
(dissoc :app.srepl/server
:app.http/server
:app.http/route
:app.setup/templates
:app.http.oauth/handler
:app.notifications/handler
:app.loggers.mattermost/reporter
:app.loggers.database/reporter
:app.loggers.metrics/reporter
:app.worker/cron
:app.worker/dispatcher)
(remove-job-runners))
_ (ig/load-namespaces system)
system (-> (ig/expand system) (ig/init))]
(try
(binding [*system* system
*pool* (:app.db/pool system)]
(next))
(finally
(ig/halt! system)))))
(def state-init
(t/compose-fixtures init-config init-system))
(defn database-reset
[next]
(let [sql (str "SELECT table_name "
" FROM information_schema.tables "
" WHERE table_schema = 'public' "
" AND table_name != 'migrations';")]
(db/transact! *pool* (fn [conn]
(db/exec-one! conn ["SET CONSTRAINTS ALL DEFERRED"])
(db/exec-one! conn ["SET LOCAL rules.deletion_protection TO off"])
(let [result (->> (db/exec! conn [sql])
(map :table-name))]
(doseq [table result]
(db/exec! conn [(str "delete from " table ";")])))))
(next)))
(defn clean-storage
[next]
(let [path (fs/path "/tmp/penpot")]
(when (fs/exists? path)
(fs/delete (fs/path "/tmp/penpot")))
(fs/create-dir "/tmp/penpot")
(next)))
(defn serial
[& funcs]
(fn [next]
(loop [f (first funcs)
fs (rest funcs)]
(when f
(let [prm (promise)]
(f #(deliver prm true))
(deref prm)
(recur (first fs)
(rest fs)))))
(next)))
(defn mk-uuid
[prefix & args]
(UUID/nameUUIDFromBytes (-> (apply str prefix args)
(.getBytes "UTF-8"))))
;; --- FACTORIES
(defn create-profile*
([i] (create-profile* *system* i {}))
([i params] (create-profile* *system* i params))
([system i params]
(let [params (merge {:id (mk-uuid "profile" i)
:fullname (str "Profile " i)
:email (str "profile" i ".test@nodomain.com")
:password "Test123!"
:is-demo false}
params)]
(db/run! system
(fn [cfg]
(->> params
(cmd.auth/create-profile cfg)
(cmd.auth/create-profile-rels cfg)))))))
(defn create-project*
([i params] (create-project* *system* i params))
([system i {:keys [profile-id team-id] :as params}]
(assert (uuid? profile-id))
(assert (uuid? team-id))
(let [timestamp (ct/now)]
(db/run! system
(fn [cfg]
(->> (merge {:id (mk-uuid "project" i)
:name (str "project" i)}
params
{::rpc/request-at timestamp})
(#'projects/create-project cfg)))))))
(defn create-file*
([i params]
(create-file* *system* i params))
([system i {:keys [profile-id project-id] :as params}]
(dm/assert! "expected uuid" (uuid? profile-id))
(dm/assert! "expected uuid" (uuid? project-id))
(db/tx-run! system
(fn [system]
(let [features (cfeat/get-enabled-features cf/flags)]
(files.create/create-file system
(merge {:id (mk-uuid "file" i)
:name (str "file" i)
:features features}
params)))))))
(defn create-job*
"A job row in the state a worker finds it: running, in the `binfile` queue,
owned by `profile-id` and holding `resource-id` when it consumes one.
`job-name` is the kind of the job (a keyword like the `::jobs/name` a caller
would pass, or the text the row holds) and it is what resolves the job-def
of the row when its result is read or checked."
[job-name {:keys [profile-id resource-id]}]
(let [id (uuid/next)]
(db/insert! *pool* :job {:id id
:name (name job-name)
:tenant (cf/get :tenant)
:queue "binfile"
:params (db/json {})
:priority 100
:max-retries 0
:retry-num 0
:status "running"
:profile-id profile-id
:resource-id resource-id
:scheduled-at (ct/now)
:created-at (ct/now)
:modified-at (ct/now)})
id))
(defn mark-file-deleted*
([params]
(mark-file-deleted* *system* params))
([system {:keys [id] :as params}]
(db/tx-run! system
(fn [cfg]
(#'files/mark-file-deleted cfg {} id)))))
(defn create-team*
([i params] (create-team* *system* i params))
([system i {:keys [profile-id] :as params}]
(us/assert uuid? profile-id)
(dm/with-open [conn (db/open system)]
(let [id (mk-uuid "team" i)
features (cfeat/get-enabled-features cf/flags)]
(teams/create-team {::db/conn conn} {:id id
:profile-id profile-id
:features features
:name (str "team" i)})))))
(defn create-file-media-object*
([params] (create-file-media-object* *system* params))
([system {:keys [name width height mtype file-id is-local media-id]
:or {name "sample" width 100 height 100 mtype "image/svg+xml" is-local true}}]
(dm/with-open [conn (db/open system)]
(db/insert! conn :file-media-object
{:id (uuid/next)
:file-id file-id
:is-local is-local
:name name
:media-id media-id
:width width
:height height
:mtype mtype}))))
(defn link-file-to-library*
([params] (link-file-to-library* *system* params))
([system {:keys [file-id library-id] :as params}]
(dm/with-open [conn (db/open system)]
(#'files/link-file-to-library conn {:file-id file-id :library-id library-id}))))
(defn create-complaint-for
[system {:keys [id created-at type]}]
(dm/with-open [conn (db/open system)]
(db/insert! conn :profile-complaint-report
{:profile-id id
:created-at (or created-at (ct/now))
:type (name type)
:content (db/tjson {})})))
(defn create-global-complaint-for
[system {:keys [email type created-at]}]
(dm/with-open [conn (db/open system)]
(db/insert! conn :global-complaint-report
{:email email
:type (name type)
:created-at (or created-at (ct/now))
:content (db/tjson {})})))
(defn create-team-role*
([params] (create-team-role* *system* params))
([system {:keys [team-id profile-id role] :or {role :owner}}]
(dm/with-open [conn (db/open system)]
(#'teams/create-team-role {::db/conn conn}
{:team-id team-id
:profile-id profile-id
:role role}))))
(defn create-project-role*
([params] (create-project-role* *system* params))
([system {:keys [project-id profile-id role] :or {role :owner}}]
(dm/with-open [conn (db/open system)]
(#'teams/create-project-role conn profile-id project-id role))))
(defn create-file-role*
([params] (create-file-role* *system* params))
([system {:keys [file-id profile-id role] :or {role :owner}}]
(dm/with-open [conn (db/open system)]
(files.create/create-file-role! conn {:file-id file-id
:profile-id profile-id
:role role}))))
(defn update-file*
([params] (update-file* *system* params))
([system {:keys [file-id changes session-id profile-id revn]
:or {session-id (uuid/next) revn 0}}]
(-> system
(assoc ::files.update/timestamp (ct/now))
(db/tx-run! (fn [system]
(let [file (files.update/get-file system file-id)]
(#'files.update/update-file* system
{:id file-id
:revn revn
:vern 0
:file file
:features (:features file)
:changes changes
:session-id session-id
:profile-id profile-id})))))))
(defn add-rect-change
"A change that adds a 10x10 rect `id` to the root of `page-id`."
[page-id id]
{:type :add-obj
:page-id page-id
:id id
:parent-id uuid/zero
:frame-id uuid/zero
:obj (cts/setup-shape {:id id
:type :rect
:name "rect"
:x 0 :y 0 :width 10 :height 10
:parent-id uuid/zero
:frame-id uuid/zero})})
(declare command!)
(defn update-file! [& {:keys [profile-id file-id changes revn] :or {revn 0}}]
(let [features (cfeat/get-enabled-features cf/flags)
params {::type :update-file
::rpc/profile-id profile-id
:id file-id
:session-id (uuid/random)
:revn revn
:vern 0
:features features
:changes changes}
out (command! params)]
(t/is (nil? (:error out)))
(:result out)))
(defn create-webhook*
([params] (create-webhook* *system* params))
([system {:keys [team-id id uri mtype is-active]
:or {is-active true
mtype "application/json"
uri "http://example.com/webhook"}}]
(db/run! system (fn [{:keys [::db/conn]}]
(db/insert! conn :webhook
{:id (or id (uuid/next))
:team-id team-id
:uri uri
:is-active is-active
:mtype mtype})))))
;; --- RPC HELPERS
(defn handle-error
[^Throwable err]
(if (instance? java.util.concurrent.ExecutionException err)
(handle-error (.getCause err))
err))
(defmacro try-on!
[expr]
`(try
(let [result# ~expr
result# (cond-> result# (rph/wrapped? result#) deref)]
{:error nil
:result result#})
(catch Exception e#
{:error (handle-error e#)
:result nil})))
(defrecord DummyRequest [headers cookies method body-stream
remote-addr server-name server-port
scheme protocol path query ssl-client-cert]
yreq/IRequestCookies
(get-cookie [_ name]
{:value (get cookies name)})
yreq/IRequest
(get-header [_ name]
(get headers name))
(method [_] method)
(body [_] body-stream)
(path [_] path)
(query [_] query)
(server-port [_] server-port)
(server-name [_] server-name)
(remote-addr [_] remote-addr)
(ssl-client-cert [_] ssl-client-cert)
(scheme [_] scheme)
(protocol [_] protocol))
(defn make-dummy-request
"Constructs a DummyRequest from options. Called with keyword
arguments, a single options map, or no arguments; every key is
optional and missing values fall back to sensible defaults. New
fields added to DummyRequest won't break existing call sites
as long as this constructor keeps its `:or` defaults in sync.
Recognized keys:
:headers — map of header name → value
:cookies — map of cookie name → value
:method — HTTP method keyword (default :get)
:body-stream — InputStream for the body (used directly)
:body-bytes — bytes or string for the body; wrapped in a
ByteArrayInputStream if :body-stream is not
given
:remote-addr — string (default \"127.0.0.1\")
:server-name — string (default \"test\")
:server-port — long (default 0)
:scheme — keyword (default :http)
:protocol — string (default \"HTTP/1.1\")
:path — string (default \"/test\")
:query — string or nil (default nil)
:ssl-client-cert — X509Certificate or nil (default nil)"
[& {:keys [headers cookies method body-stream body-bytes
remote-addr server-name server-port scheme protocol
path query ssl-client-cert]
:or {headers {} cookies {} method :get
body-stream nil
remote-addr "127.0.0.1" server-name "test" server-port 0
scheme :http protocol "HTTP/1.1" path "/test" query nil
ssl-client-cert nil}}]
(let [body-stream (or body-stream
(when body-bytes
(java.io.ByteArrayInputStream.
(if (string? body-bytes)
(.getBytes ^String body-bytes "UTF-8")
body-bytes))))]
(->DummyRequest headers cookies method body-stream
remote-addr server-name server-port
scheme protocol path query ssl-client-cert)))
(defn- prepare-rpc-params
[data]
(let [params (reduce-kv (fn [params k v]
(if (qualified-keyword? k)
(assoc params k v)
params))
{}
(dissoc data ::type))
;; Honor a caller-supplied request map from the data metadata so
;; tests can inject headers, remote address and auth context (e.g.
;; :app.http/auth-key-id). Non-map requests (reify IRequest stubs)
;; fall back to the dummy request, which carries the body params
;; the validation wrapper reads.
supplied (-> data meta :app.http/request)
request (if (map? supplied)
(assoc supplied :params (merge (:params supplied)
(d/without-qualified data)))
(assoc (make-dummy-request)
:params (d/without-qualified data)))]
(-> params
(assoc :app.rpc/request-at (ct/now))
;; Mirror of the production flag in `app.rpc/make-rpc-handler`:
;; upload maps are only trusted over multipart. Tests declare it
;; with a content-type header on the supplied request map.
(assoc ::rpc/is-multipart
(str/starts-with?
(str (yreq/get-header request "content-type"))
"multipart/"))
(with-meta {:app.http/request request}))))
(defn command!
[{:keys [::type] :as data}]
(let [[mdata method-fn] (get-in *system* [:app.rpc/methods type])]
(when-not method-fn
(ex/raise :type :assertion
:code :rpc-method-not-found
:hint (str/ffmt "rpc method '%' not found" (name type))))
(let [params (prepare-rpc-params data)]
(try-on! (method-fn params)))))
(defn command-raw!
"Like `command!`, but returns the raw result without unwrapping it, so a test
can read the result metadata (for example the audit props a command sets on a
nil result). The HTTP layer unwraps it the same way in production."
[{:keys [::type] :as data}]
(let [[_ method-fn] (get-in *system* [:app.rpc/methods type])]
(when-not method-fn
(ex/raise :type :assertion
:code :rpc-method-not-found
:hint (str/ffmt "rpc method '%' not found" (name type))))
(let [params (prepare-rpc-params data)]
(method-fn params))))
(defn management-command!
"Invokes a management RPC method.
Resolves the shared-key id from an explicit `::rpc/auth-key-id`, from
`:app.http/auth-key-id` on the data or on the supplied request, and falls
back to the first id the method declares. The fallback lets the existing
tests exercise the allowed path without repeating the key; a test that
checks a rejection declares the key explicitly, including `nil`."
[{:keys [::type] :as data}]
(let [[mdata method-fn] (get-in *system* [:app.rpc/management-methods type])]
(when-not method-fn
(ex/raise :type :assertion
:code :rpc-method-not-found
:hint (str/ffmt "management rpc method '%' not found" (name type))))
(let [request (-> data meta :app.http/request)
key-id (cond
(contains? data ::rpc/auth-key-id)
(::rpc/auth-key-id data)
(contains? data :app.http/auth-key-id)
(:app.http/auth-key-id data)
(and (map? request)
(contains? request :app.http/auth-key-id))
(:app.http/auth-key-id request)
:else
(first (::rpc/allowed-shared-keys mdata)))
data (if (or (contains? data ::rpc/auth-key-id)
(nil? key-id))
data
(assoc data ::rpc/auth-key-id key-id))
params (prepare-rpc-params data)]
(try-on! (method-fn params)))))
(def multipart-headers
{"content-type" "multipart/form-data; boundary=test"})
(defn with-multipart
"Attach a multipart content-type to RPC test params, declaring that the
upload maps they carry arrived over multipart (as the production
multipart parser would deliver them). Preserves any request metadata
already present on the data."
[data]
(let [request (-> data meta :app.http/request)]
(vary-meta data assoc :app.http/request
(assoc (if (map? request) request {})
:headers (merge multipart-headers
(when (map? request)
(:headers request)))))))
(defn multipart-command!
"Like `command!` but declares the request as multipart, for calls that
carry upload maps (`:file`, `:content`, `:media`)."
[data]
(command! (with-multipart data)))
(defn multipart-management-command!
"Like `management-command!` but declares the request as multipart."
[data]
(management-command! (with-multipart data)))
(defn run-task!
"Execute a job handler directly (in-process, no row): the handler gets a
nil context, because there is no job to describe."
([name]
(run-task! name {}))
([name params]
(jobs/invoke (-> *system*
(assoc ::jobs/name name)
(assoc ::jobs/params params)))))
(def sql:pending-jobs
"select * from job
where status = 'new'
order by priority desc, scheduled_at")
(defn run-pending-jobs
"Execute the pending (status='new') `job` rows in-process (simulating
the dispatcher + runner for the tests). Each row gets its context and its
job-id, so heartbeats and progress reach it exactly like in the runner.
Does not touch the row status; only the handler side effects matter."
[]
(db/tx-run! *system*
(fn [{:keys [::db/conn]}]
(let [jobs-rows (db/exec! conn [sql:pending-jobs])]
(doseq [row jobs-rows]
(jobs/invoke (-> *system*
;; the row stores a string; invoke
;; takes the canonical keyword
(assoc ::jobs/name (keyword (:name row)))
(assoc ::jobs/params (:params row))
(assoc ::jobs/context (jobs/make-context row))
(assoc ::jobs/job-id (:id row)))))))))
(defn metrics-stub
"A metrics instance with no collectors, where every `run!` is a no-op."
[]
(reify metrics/IMetrics
(get-handler [_] nil)
(get-collector [_ _] nil)
(get-registry [_] nil)))
(defn init-climit
"Build a concurrency limiter component from an inline config map. The
optional second argument is the metrics instance to publish to (a
`metrics-stub` by default, where every `run!` is a no-op).
The root limiters are added with permits that no test reaches: every
request goes through them, and a test that does not list them would log a
warning per request."
([config]
(init-climit config (metrics-stub)))
([config metrics]
(let [file (doto (java.io.File/createTempFile "climit" ".edn")
(.deleteOnExit))]
(spit file (pr-str (merge {:root/global {:permits 1000 :timeout 3000}
:root/by-profile {:permits 1000 :timeout 3000}}
config)))
(ig/init-key ::rpc/climit
{::climit/enabled true
::climit/config (.getAbsolutePath file)
::metrics/metrics metrics}))))
(defmacro with-climit
"Build a concurrency limiter from an inline config map, run the body and
halt it, so the republisher thread it starts never outlives the test. An
optional third binding element is the metrics instance to publish to."
[[sym config & [metrics]] & body]
`(let [~sym (init-climit ~config ~@(when metrics [metrics]))]
(try
~@body
(finally
(ig/halt-key! ::rpc/climit ~sym)))))
;; --- UTILS
(defn print-error!
[error]
(let [data (ex-data error)]
(cond
(= :spec-validation (:code data))
(println
(us/pretty-explain data))
(= :params-validation (:code data))
(println
(sm/humanize-explain (::sm/explain data)))
(= :data-validation (:code data))
(println
(sm/humanize-explain (::sm/explain data)))
(= :service-error (:type data))
(print-error! (.getCause ^Throwable error))
:else
(.printStackTrace ^Throwable error))))
(defn print-result!
[{:keys [error result]}]
(if error
(do
(println "====> START ERROR")
(print-error! error)
(println "====> END ERROR"))
(do
(println "====> START RESPONSE")
(pp/pprint result)
(println "====> END RESPONSE"))))
(defn exception?
[v]
(instance? Throwable v))
(defn ex-info?
[v]
(ex/error? v))
(defn ex-type
[e]
(:type (ex-data e)))
(defn ex-code
[e]
(:code (ex-data e)))
(defn ex-of-type?
[e type]
(let [data (ex-data e)]
(= type (:type data))))
(defn ex-of-code?
[e code]
(let [data (ex-data e)]
(= code (:code data))))
(defn ex-with-code?
[e code]
(let [data (ex-data e)]
(= code (:code data))))
(defn success?
[{:keys [result error]}]
(nil? error))
(defn tempfile
[source]
(let [rsc (io/resource source)
tmp (fs/create-tempfile :dir "/tmp/penpot" :prefix "test-")]
(io/copy (io/file rsc)
(io/file tmp))
tmp))
(defmacro with-temp-file
"Evaluate `body` with `sym` bound to the path of a temporary file that
holds `content`. The file is created under /tmp/penpot, one of the two
places the upload validation accepts (/tmp/undertow is the other one),
and it is deleted when the body ends, also when it throws."
[[sym content] & body]
`(let [path# (fs/create-tempfile :dir "/tmp/penpot" :prefix "penpot-test-")]
(try
(spit (str path#) ~content :encoding "UTF-8")
(let [~sym path#]
~@body)
(finally
(when (fs/exists? path#)
(fs/delete path#))))))
(defn pause
[]
(let [^java.io.Console cnsl (System/console)]
(println "[waiting RETURN]")
(.readLine cnsl)
nil))
(defn db-exec!
[sql]
(db/exec! *pool* sql))
(defn db-exec-one!
[sql]
(db/exec-one! *pool* sql))
(defn db-delete!
[& params]
(apply db/delete! *pool* params))
(defn db-update!
[& params]
(apply db/update! *pool* params))
(defn db-insert!
[& params]
(apply db/insert! *pool* params))
(defn db-delete!
[& params]
(apply db/delete! *pool* params))
(defn db-force-delete
"Deletes rows whose table is guarded by `raise_deletion_protection`,
inside a transaction that disables the guard. Test-only escape hatch:
production delete paths disable protection in their own transaction
(see `app.jobs.gc`)."
[& params]
(db/tx-run! *pool*
(fn [{:keys [::db/conn]}]
(db/exec-one! conn ["SET LOCAL rules.deletion_protection TO off"])
(apply db/delete! conn params))))
(defn db-query
[& params]
(apply db/query *pool* params))
(defn db-get
[& params]
(apply db/get* *pool* params))
(defn read-recording-sql
"Run `f` and return [result sqls], where `sqls` are the SQL statements of
every `db/get-with-sql` call. Those are exactly the file reads of
`binfile.common/get-file`: the fragment loads go through `db/get*`."
[f]
(let [sqls (atom [])]
(with-redefs [db/get-with-sql
(let [orig db/get-with-sql]
(fn [ds sql & opts]
(swap! sqls conj (first sql))
(apply orig ds sql opts)))]
[(f) @sqls])))
(defn sleep
[ms-or-duration]
(Thread/sleep (inst-ms (ct/duration ms-or-duration))))
(defn config-get-mock
[data]
(fn
([key]
(get data key (get cf/config key)))
([key default]
(get data key (get cf/config key default)))))
(defn reset-mock!
[m]
(swap! m (fn [m]
(-> m
(assoc :called? false)
(assoc :call-count 0)
(assoc :return-list [])
(assoc :call-args nil)
(assoc :call-args-list [])))))
(defn- slurp'
[input & opts]
(let [sw (java.io.StringWriter.)]
(with-open [^java.io.Reader r (java.io.InputStreamReader. input "UTF-8")]
(io/copy r sw)
(.toString sw))))
(defn parse-sse
[content]
(let [state
(reduce (fn [{:keys [events data event id] :as state} line]
(cond
;; empty line → dispatch event if we have data
(str/blank? line)
(if (seq data)
(-> state
(update :events conj {:event (or event "message")
:data (-> (str/join "\n" data))})
(assoc :data [] :event nil))
state)
;; comment line (starts with :)
(str/starts-with? line ":")
state
:else
(let [[field raw-value] (str/split line #":" 2)
value (some-> raw-value (str/replace #"^ " ""))]
(case field
"data" (update state :data conj (or value ""))
"event" (assoc state :event value)
;; ignore retry and unknown fields
state))))
{:events [] :data [] :event nil}
(str/split content #"\r?\n"))
;; handle unterminated last event (no trailing blank line)
state (if (seq (:data state))
(update state :events conj
{:event (or (:event state) "message")
:data (str/join "\n" (:data state))})
state)]
(:events state)))
(defn consume-sse
[callback]
(let [{:keys [::yres/status ::yres/body ::yres/headers] :as response} (callback {})
output (PipedOutputStream.)
input (PipedInputStream. output)]
(try
(px/exec! :virtual #(rcp/write-body-to-stream body nil output))
(into []
(map (fn [{:keys [event data]}]
(d/vec2 (keyword event)
(tr/decode-str data))))
(parse-sse (slurp' input)))
(finally
(.close input)))))