115 Commits

Author SHA1 Message Date
Usama Sadiq
861bb5bb37 🐛 Fix database pool min-size falling back to max-size (#13052)
* 🐛 Fix database pool min-size falling back to max-size

Since 2.16 the pool minimum defaults to the maximum (60), so an idle
backend opens and holds 60 connections at startup. Restore the pre-2.16
default of 0 so the pool grows with load and idle connections expire.

Closes #12622

Signed-off-by: Usama Sadiq <usama7274@gmail.com>

* 📚 Document database connection pool settings

Describe PENPOT_DATABASE_MIN_POOL_SIZE and PENPOT_DATABASE_MAX_POOL_SIZE,
their defaults, and sizing on a PostgreSQL shared with other apps.

Signed-off-by: Usama Sadiq <usama7274@gmail.com>

---------

Signed-off-by: Usama Sadiq <usama7274@gmail.com>
2026-11-16 08:37:49 +01:00
Andrey Antukh
452f38cf5d
✨ Add Prometheus metrics for storage operations (#11700)
* ✨ Add storage operation metrics for S3 and buckets

Expose Prometheus metrics for the object storage subsystem.

The S3 backend now attaches an AWS SDK MetricPublisher that counts
API calls, retries and latency per operation and target. The storage
layer counts logical operations and deduplication outcomes per Penpot
bucket, and the assets handlers count served requests per route.

Closes #11676

AI-assisted-by: muse-spark-1.3-contributor

* ✨ Fix storage metrics labels, errors and test gaps

Address the review findings on the storage metrics commit.

Label reads with the object's own backend, count failed asset
serving as errors without swallowing them, and cover the failed
S3 call, S3 asset path and permission-denied branches with tests.
Also share the label helper and reuse the metrics test helper.

Closes #11676

AI-assisted-by: muse-spark-1.3-contributor

* ✨ Harden storage metrics and fill test gaps

Address the second-round review findings on storage metrics.

Unknown backends now fail explicitly and count as errors, exists
stays paired with its dedup outcome, and the thumbnail, missing
storage, expired reads, unknown buckets and write failure paths
are covered by tests. Label coercion goes through the shared
metrics helper.

Closes #11676

AI-assisted-by: muse-spark-1.3-contributor

* ✨ Harden storage metrics accuracy and coverage

Address the full-branch review findings on storage metrics.

Touch and delete emit only on changed rows, reads emit after the
backend fetch, unknown backends fail explicitly, and tempfile
mismatches count as unauthorized. Publisher nil policy, pairing
rules and attempt semantics are documented and covered by tests.

Closes #11676

AI-assisted-by: muse-spark-1.3-contributor

* ✨ Address full-branch review findings on storage metrics

Touch and delete resolve labels from the row, reads stay paired,
failures are covered by tests, and logging, ranges and docs are
tightened. Includes the label helper unit tests and the retries
wording clarification.

Closes #11676

AI-assisted-by: muse-spark-1.3-contributor

* ⚡ Label touch and del metrics from UPDATE RETURNING

The storage metrics change resolved metric labels for touch-object!
and del-object! with an extra SELECT per id-based call. Since app.main
instruments storage unconditionally, every GC collector and binfile
import paid that extra round trip: deleting a team with 10k media
objects doubled the storage_object statements exactly on the paths
that already process the most rows.

touch-object! and del-object! now take only the object id (UUID) and
read the labels from the updated row itself via RETURNING id, backend,
metadata: one statement, no pre-read, and labels that always match the
row actually mutated. del-object! additionally guards on deleted_at
IS NULL, so a repeated delete returns false and emits no metric.

Also from the review of the full branch: extract the duplicated
serve/emit/rethrow block in app.http.assets into one helper; give
penpot_storage_s3_timing explicit histogram buckets up to 60s (the
default cap at 7.5s hid the slow S3 calls the metric exists for);
drop the unused ::target-id config key from the S3 backend and
hardcode the :default target label until per-bucket routing lands.

AI-assisted-by: glm-5.3-flash

* ✨ Harden storage metric recording and definitions

The metric definition schema is now closed and declares every key
the collectors read: buckets, quantiles, max-age and reg. A typo
such as a misspelled ::mdef/buckets used to compile and silently
fall back to the default histogram buckets; it now fails at
startup.

The asset result-label fallback coerced an absent status to 500,
so a future serve path without a status would have counted
successes as errors. The mapping is now explicit and documented:
served below 400, unauthorized for 401/403, not-found for 404,
and error for everything else, including an absent status.

The never-fail try/catch around metric recording existed four
times with drift. One app.metrics/run-safe! helper replaces them:
it no-ops on a nil metrics instance and logs the first failure
per hint at warn level, then at debug, so a broken setup surfaces
once without flooding the log. The S3 publisher keeps its outer
try/catch: it is the SDK MetricPublisher contract boundary.

AI-assisted-by: glm-5.3-flash

* ✨ Make metrics mandatory and run! safe by default

Recording a metric must never change the behavior of the operation
being measured, so `run!` now catches recording failures itself: the
first failure per metric id logs at warn, later ones at debug. This
replaces the `run-safe!` helper, whose four copies had drifted, and
applies the guarantee to every emit site instead of only storage.

The metrics instance precondition is a plain assert, and the collector
lookup stays outside the recording guard, so a missing instance fails
hard even when asserts are disabled. Metrics is therefore no longer
optional: the storage, s3-backend and db-pool schemas require
`::mtx/metrics`, and the assets handler cfg always carries it.

`wrap-publisher` no longer returns nil for a nil instance, and the db
pool wires the prometheus tracker unconditionally.

AI-assisted-by: deepseek-v4.1-flash
2026-09-23 15:25:21 +02:00
Andrey Antukh
f5aad7b1ae Merge remote-tracking branch 'origin/staging' into develop 2026-09-01 12:45:17 +02:00
Andrey Antukh
a1079cf788
⬆️ Update JVM, pnpm and node dependencies (#11404)
* ⬆️ Update pnpm and its deps

* ⬆️ Update JVM dependencies in backend and common

Update several JVM dependencies across backend and common:

- passay 1.6.6 -> 2.0.0 (package reorg, ctor-based rules)
- siphash 2.0.0 -> 3.0.0 (SipHasher* renamed to SipHash*)
- lettuce-core, guava, sqlite-jdbc, jsoup, lz4-java, markdown-clj,
  awssdk s3/sts, selmer, jackson-core/databind, shadow-cljs

Adapt passay validation to the new API (moved packages, constructor
configuration) and siphash to the renamed classes. Add tests for
password validation and UUID advisory-lock hashing.

AI-assisted-by: deepseek-v4-flash

* ⬆️ Update node on docker images

* 📎 Minor fixes related to pnpm12 compatibility
2026-09-01 12:01:35 +02:00
Andrey Antukh
81c3b3cd56
📎 Update copyright name on file header (#11346) 2026-08-25 11:55:10 +02:00
Andrey Antukh
8e9df0c515 ✨ Add insert-multi chunked variant to app.db 2026-07-09 09:25:59 +02:00
Yamila Moreno
ddba2ffa75
📎 Update Kaleidos Copyright (#9929) 2026-05-29 11:24:58 +02:00
Andrey Antukh
405a73e8ba
✨ Add climit impl and config for file snapshot methods (#9722)
* ✨ Add dedicated concurrency limit for restore-file-snapshot

This adds a dedicated climit configuration for the restore-file-snapshot
RPC method with :permits 1 per profile (plus queue of 2 and 60s timeout)
and a global limit of 3. Previously the method only used the generic
root/by-profile and root/global limits, allowing up to 7 concurrent
restore operations per profile which caused database row lock contention
on FOR UPDATE and connection pool exhaustion.

* ✨ Skip locking on restore! to avoid blocking other operations

Changes the row lock acquisition in restore! from a blocking FOR UPDATE
to FOR UPDATE SKIP LOCKED. If the file row is already locked by another
concurrent operation (e.g., another restore or an update-file), the query
returns no rows and the caller fails fast with a clear conflict error
instead of blocking indefinitely holding a database connection.

* ✨ Add queue and timeout limits to root/by-profile concurrency limit

Previously root/by-profile had no queue limit (unbounded Integer/MAX_VALUE)
and no timeout, allowing requests to pile up indefinitely behind a profile
whose permits were exhausted by long-running operations. This could lead
to memory pressure and cascading failures. Now limited to 30 queued
requests with a 30-second timeout so excess requests fail fast.

* ✨ Move backup snapshot creation outside restore transaction

The backup snapshot (fsnap/create!) is now created in its own short-lived
connection before the actual restore transaction begins. This ensures the
backup is persisted independently of the restore outcome and reduces the
restore transaction window.

The restore itself runs inside a db/tx-run! block with an optimistic
locking check: it reads the file with FOR UPDATE and compares its revn
against the value captured at backup time. If the file was edited
concurrently, the restore aborts with a conflict error to prevent data
loss.

Co-dependent with the SKIP LOCKED change in restore! — the FOR UPDATE
acquired here is in the same transaction as restore!, so the SKIP LOCKED
inside restore! correctly sees the row as unlocked (same transaction).

* ♻️ Remove unused private function get-minimal-file

The local get-minimal-file function in file_snapshots.clj is no longer
used since restore! switched to direct exec-one! with FOR UPDATE SKIP
LOCKED. The sql:get-minimal-file SQL constant is still used directly.

* ✨ Add minor improvements on db connection management

* ♻️ Refactor create-file-snapshot to use explicit transaction management

Remove automatic transaction wrapping (`::db/transaction true`) and
pass `cfg` through the call chain instead of destructured `conn`.
Wrap `fsnap/create!` in an explicit `db/tx-run!` for clearer
transaction boundaries.

Signed-off-by: Andrey Antukh <niwi@niwi.nz>

* ✨ Add dedicated concurrency limit for create-file-snapshot

This adds a dedicated climit configuration for the create-file-snapshot
RPC method with :permits 1 per profile (plus queue of 2 and 60s timeout)
and a global limit of 3. Previously the method only used the generic
root/by-profile and root/global limits, allowing up to 10 concurrent
snapshot creation operations per profile which could cause database
contention and connection pool exhaustion.

Signed-off-by: Andrey Antukh <niwi@niwi.nz>

---------

Signed-off-by: Andrey Antukh <niwi@niwi.nz>
2026-05-19 14:30:44 +02:00
Andrey Antukh
d380efdb0c
⬆️ Update devenv dependencies (#9142)
* ⬆️ Update devenv dependencies

* ✨ Fix formatting issues

* 📎 Fix linter issues
2026-04-24 14:07:51 +02:00
Andrey Antukh
9b3f68ad14 ✨ Remove unnecesary report on duplicate email error validation 2025-11-04 20:34:25 +01:00
Alejandro Alonso
2b7bd8fa5c 🐛 Fix deleted files are accesible 2025-10-13 12:24:05 +02:00
Andrey Antukh
5717708b56 ♻️ Refactor file storage
Make it more scallable and make it easily extensible
2025-10-13 12:24:05 +02:00
Andrey Antukh
283eb0419c ♻️ Refactor time related namespaces
Mainly removes the custom app.util.time namespace
from frontend and backend and normalize all to use
the app.common.time namespace
2025-08-01 11:20:01 +02:00
Andrey Antukh
4bdba6894d ✨ Add get-with-sql helper to db module 2025-07-30 12:06:41 +02:00
Andrey Antukh
2aa2525d0e ✨ Add db conn dynamic binding for srepl helpers 2025-05-06 13:39:17 +02:00
Andrey Antukh
3074fc9ab5
♻️ Remove deprecated with-atomic and refactor tx-run! (#5915)
* ♻️ Remove deprecated with-atomic and refactor tx-run!

* ✨ Do not hold open connection for the whole clone-template operation
2025-02-24 11:15:44 +01:00
Andrey Antukh
15f7fac8f4 ✨ Add more reasonable default fetch size for db/plan
Making it more general purpose and not only for streaming
2025-01-27 12:53:11 +01:00
Andrey Antukh
da0704081f 📎 Normalize default opts for db/plan function 2025-01-20 23:23:31 +01:00
Andrey Antukh
066b1235a6 🐛 Pass correct default options on db/plan fn 2025-01-20 23:23:31 +01:00
Andrey Antukh
fde0f3c182 🐛 Pass correct default options on db/plan fn 2025-01-20 16:34:05 +01:00
Andrey Antukh
88fb5e7ab5 ♻️ Update integrant to latest version
This upgrade also includes complete elimination of use spec
from the backend codebase, completing the long running migration
to fully use malli for validation and decoding.
2024-11-13 19:09:19 +01:00
Andrey Antukh
7df68bb8bd ✨ Add :params prop to :not-found exception 2024-08-12 11:55:19 +02:00
Andrey Antukh
0e92bcc0de 🎉 Add file-data offload mechanism 2024-08-09 14:28:18 +02:00
Andrey Antukh
a9e7ed57d9 ✨ Use proper exceptions on internal db functions 2024-02-01 17:24:41 +01:00
Andrey Antukh
f73ce6572c ✨ Improve rollback handlong on db ns 2024-01-25 16:03:29 +01:00
Andrey Antukh
d4d3f9ca81 🎉 Add the ability to export import entire team
For now only available as srepl helper
2024-01-16 17:17:30 +01:00
Andrey Antukh
746d898245 ✨ Improve the db api efficiency
Mainly setup proper defaults and reduce unnecesary allocations
on every db api call.
2024-01-04 12:41:16 +01:00
Andrey Antukh
addb392ecc ✨ Add safety mechanism for direct object deletion
The main objective is prevent deletion of objects that can leave
unreachable orphan objects which we are unable to correctly track.

Additionally, this commit includes:

1. Properly implement safe cascade deletion of all participating
   tables on soft deletion in the objects-gc task;

2. Make the file thumbnail related tables also participate in the
   touch/refcount mechanism applyign to the same safety checks;

3. Add helper for db query lazy iteration using PostgreSQL support
   for server side cursors;

4. Fix efficiency issues on gc related task using server side
   cursors instead of custom chunked iteration for processing data.

   The problem resided when a large chunk of rows that has identical
   value on the deleted_at column and the chunk size is small (the
   default); when the custom chunked iteration only reads a first N
   items and skip the rest of the set to the next run.

   This has caused many objects to remain pending to be eliminated,
   taking up space for longer than expected. The server side cursor
   based iteration does not has this problem and iterates correctly
   over all objects.

5. Fix refcount issues on font variant deletion RPC methods
2024-01-03 10:56:57 +01:00
Andrey Antukh
417366d998 ✨ Reorganize fdata/pointer-map feature helpers
Mainly move all pointer-map related helpers from app.rpc.commands.files
to the the app.features.fdata namespace and normalizes codestile around
feature handling on all affected code.

This commit also comes with several features related bugifxes on the
components-v2 migration code:

- properly migrate legacy feature names on apply components-v2 migration
- start using new fdata feature related functions
- prevent generation of a ephimeral pointer on each graphic migration
  operation; on large files this caused a very noticiable overhead of
  creating a big number of completly unused pointer maps
- do persistence after validation and not before
2023-12-13 14:20:07 +01:00
Andrey Antukh
0a77bae8a7 ✨ Improve options handling on db module 2023-12-13 11:56:20 +01:00
Andrey Antukh
6bff6d24b9 🐛 Fix issue on db/get-connectable impl 2023-12-13 11:56:20 +01:00
Andrey Antukh
87615ce221 💄 Fix format issues on backend module 2023-11-29 12:55:58 +01:00
Andrey Antukh
47e877d6c3 ✨ Add minor internal db api improvements 2023-11-08 10:44:44 +01:00
Andrey Antukh
6f93b41920 🎉 Add features assignation for teams 2023-11-07 12:48:31 +01:00
Andrey Antukh
c7f9774524 ✨ Add more flexible call flow for db interacting methods 2023-09-04 12:04:15 +02:00
Andrey Antukh
d1128a6b1e 🎉 Add helpers for take file snapshots 2023-08-03 17:51:34 +02:00
Andrey Antukh
804addfa66 📎 Add srepl helper for process files 2023-08-03 11:49:14 +02:00
Andrey Antukh
27fb4c7ed9 ✨ Improve with-atomic macro to accept cfg 2023-07-24 13:27:27 +02:00
Andrey Antukh
4e7f32aa88 ✨ Improve retry mechanism and macros 2023-05-17 15:47:21 +02:00
Andrey Antukh
90d48c1d30 ✨ Add the ability to check read-only state of connection
on the db ns helper; previously it only worked with datasource
instances
2023-03-31 12:48:24 +02:00
Andrey Antukh
ab3b9cba45 ♻️ Refactor storage and assets related modules
- improve internal error handling
- add more specs and more asserts
2023-02-07 18:16:55 +01:00
Andrey Antukh
3cf4a3facc ✨ Improve db/pool initialization and reusability 2023-02-07 18:16:55 +01:00
Andrey Antukh
1325e46192 ✨ Improve internal state validation on db module 2023-02-02 14:20:13 +01:00
Andrey Antukh
db689d151e ♻️ Refactor profile and session handling
- makes the profile access more efficient (replace in-app joins to a
  simple select query on profile table
- add partial support for access-tokens (still missing some RPC methods)
- move router definitions to specific modules and simplify the main http
  module definitions to simple includes
- simplifiy authentication code related to access-tokens and sessions
- normalize db parameters with proper namespaced props
- more work on convert all modules initialization to use proper specs
  with fully-qualified keyword config props
2023-01-18 10:51:58 +01:00
Andrey Antukh
73a3e0c0ae 🎉 Add usage quotes 2022-12-31 11:22:36 +01:00
Andrey Antukh
d768711caa ✨ Improve null handling on more db helpers 2022-12-13 16:17:31 +01:00
Andrey Antukh
d584ae5a0f ✨ Improve json encode/decode api 2022-12-13 16:17:31 +01:00
Andrey Antukh
39b9daa3a7 🎉 Add webhooks rpc API 2022-12-05 15:20:29 +01:00
Andrey Antukh
0600b2abe4 ♻️ Make the worker abstraction more scalable
Start using redis for dispatcher to worker communication
and add the ability to start multiple threads to worker
for increase the concurrency.
2022-11-28 12:21:44 +01:00
Andrey Antukh
c027de2592 ✨ Make nil safe some decode helpers on db ns 2022-11-08 13:02:14 +01:00