3727 Commits

Author SHA1 Message Date
alonso.torres
6754c883ab 🐛 Restore merged imports and multipart test setup
Keep the string helper required by the variant rename predicate after
removing the old swap-slot helpers. Sort the merged test imports.

Send import-audit fixtures through the existing multipart RPC helper so
they exercise auditing through the required upload path. Start the edit
watcher in the isolated component test so it forwards recorded transitions.

AI-assisted-by: GPT-6
2026-10-28 16:09:55 +01:00
alonso.torres
11d7a630dc 🐛 Restore migration and test compatibility after rebase
Keep the container root helper used by the main-instance name migration.
Adapt imported-file ids and component swap tests to the current APIs, and
check copied swaps through their position refs and swap groups.

AI-assisted-by: gpt-6
2026-10-28 15:53:11 +01:00
alonso.torres
ee448fd5a1 🐛 Absorb the libraries a move to another team leaves behind
Moving files or a project to another team unlinked every library that
ended up in another team than the file using it, but absorbed nothing:
the copies kept pointing at a library their file could no longer
reach. A move now does what deleting the library does (PENPOT-1540):
it absorbs each such library into the files using it, then unlinks
them; as on deletion, a failed absorption is logged and does not stop
the move. That covers a moved library and the files left behind using
it, a moved file and the libraries it used, and a moved project.

The links checked are those that start or end at a moved file. They
were those of the files in the source and destination projects, so a
file in another project of the old team kept its link to a moved
library across teams.

rpc-management-test: a moved library is absorbed into a file in its
project and into one in a project the move does not touch; a moved
file absorbs its library; a moved project absorbs the library it left
behind.

AI-assisted-by: claude-opus-5-5
2026-10-28 15:53:07 +01:00
alonso.torres
6ce88bd0c1 🐛 Absorb a deleted library into the files that used it
Deleting a shared library removes its links to the files using it at
once, so they stop listing it, and queues the delete-object job, which
absorbs the library into the files it finds through those links. By
then there were none, so nothing was absorbed: the files lost the
library and their copies kept pointing at the deleted file
(PENPOT-1092). Unpublishing absorbs before it unlinks, so it was not
affected.

The delete reads the files using the library before unlinking them
and gives them to the job (:absorb-file-ids), which absorbs into them;
without them, as from the project and team deletions, the job still
finds them through the links.

rpc-file-test/delete-shared-library-absorbs-it-into-the-files-using-it
deletes the library through the RPC and runs the job it queued.

AI-assisted-by: claude-opus-5-5
2026-10-28 15:53:07 +01:00
alonso.torres
9c701da24a 🐛 Store the fragments a library absorption changes
Absorbing a library into a file that uses it (unpublishing or deleting
the library) saved the file without its features. Saving stores the
changed pointer-map fragments only when the features name
fdata/pointer-map, so with that feature the file pointed at a fragment
of components that was never written, and loading it failed with
:fragment-not-found. The file's features now go with the save.

rpc-file-test/unpublish-library-absorbs-it-into-the-files-using-it
unpublishes a library with a copy in another file, then loads that
file. The existing absorption test turns fdata/pointer-map off, so it
missed this.

AI-assisted-by: claude-opus-5-5
2026-10-28 15:53:07 +01:00
alonso.torres
f0d24314d7 ✨ Migrate copy texts to the per-style text overrides
Files older than the feature components/text-overrides mark a copy text
with :text-content-attribute (any style) or :text-content-structure
(letters and styles). Migration 0034 compares each such copy with its
near main and gives it the groups its old flags allowed where it
differs; an old flag where nothing differs sets nothing. A copy whose
near main cannot be reached keeps every group its flags allowed, so it
loses no override. The migration adds the feature, so a client on the
old code cannot write the old groups into a migrated file.

Binfile import leaves the copies whose near main is in a library not
linked yet, as it does with swap slots, and converts them once the
libraries are linked: the deferred pass of the import now runs both
conversions (convert-deferred!, *defer-unreachable*).

Validation reports the old groups (:legacy-text-touched), and its
repair recomputes them against the near main; a text override group
without :content-group is still :invalid-text-touched. A shape with no
letters in it and its main no longer counts as differing in its styles.

Benchmark (lower quantile, ms; baseline / now / ratio):
  deepest-edit-through-three-levels    35.2 /  34.8 / 0.99
  edit-through-a-chain-of-dependents   68.9 /  65.0 / 0.94
  library-update-into-copies           83.1 /  86.9 / 1.05
  main-edit-into-10-copies              1.8 /   1.5 / 0.86
  main-edit-into-100-copies            15.4 /  15.0 / 0.97
  main-edit-into-1000-copies          153.1 / 150.6 / 0.98
  variant-switch-into-copies           32.1 /  26.8 / 0.84

AI-assisted-by: claude-opus-5-5
2026-10-28 15:53:06 +01:00
alonso.torres
03312bfef4 ✨ Migrate swap slots to override groups and remove them
The migration 0033 stores every swap of a file as an override group.
A swap made in a copy is paired with the child of the near main that
holds the same position, as sync paired it; any other slot is dropped.
All pairings are found on the data as it was, so swaps nested in swaps
pair against the content of the outer one, and libraries pair the same
whether they are migrated or not. A head that refers to its own slot
only inherited it, and drops it even when its near main cannot be
reached. Any other slot that matches no child, or whose near main is in
a library that is not loaded, keeps the slot as its ref, so validation
reports it. An import, which migrates its files before it links their
libraries, leaves those slots and converts them once the libraries are
linked. A migrated head that sits elsewhere than the child it replaces
would place it gets its geometry marked as touched, so the first sync
does not move it. Migrated and new files carry the feature
components/swap-group, so a client that still writes slots cannot open
them.

Validation checks the new form: :misplaced-slot reports any slot,
:misplaced-swap-group the group anywhere but on a nested copy head in a
copy, :ref-not-in-near-main a nested copy head whose ref is not a child
of the near main of its parent, and :duplicate-ref two children of a
copy that refer to the same main shape. Each has a repair.

Nothing but the migrations reads slots any more: the slot branches go
from sync, swap, switch, reset, detach, paste, duplicate and move. The
helpers that read and write a slot on a shape move to files/swap-slots,
and the ones that follow refs to files/comp-processors. The graph
projection drops the FillsSwapSlot edge, the components debugger shows
the component a swapped head takes its content from, and the plugin API
documents componentRefShape for a swapped copy. The swap tests are
named after what they check.

Benchmark (lower quantile, ms; baseline / now / ratio):
  deepest-edit-through-three-levels    35.2 /  34.6 / 0.98
  edit-through-a-chain-of-dependents   68.9 /  63.5 / 0.92
  library-update-into-copies           83.1 /  76.4 / 0.92
  main-edit-into-10-copies              1.8 /   1.6 / 0.87
  main-edit-into-100-copies            15.4 /  15.1 / 0.98
  main-edit-into-1000-copies          153.1 / 148.8 / 0.97
  variant-switch-into-copies           32.1 /  26.4 / 0.82

AI-assisted-by: claude-opus-5-5
2026-10-28 15:53:05 +01:00
alonso.torres
07ab13d6bc 🔥 Remove the components v1 leftovers
No components v1 file reaches the code any more: opening one fails the
feature check, and imports reject it or give it the feature. What only
served v1 goes:

- validate-file and validate-file-affected no longer skip a file
  without components/v2;
- the root of a deleted component is found by its main instance id
  only, in the type helper and the component sprite;
- generate-sync-library, which synced shapes inside live components,
  and the live components in containers-seq, so asset usages walk the
  pages only (deleted components keep their objects and still sync);
- the unused :components-v2 file option, from the schema, the file
  data and the test fixtures; stored files that carry it still
  validate;
- the team migration-in-progress check and its frontend alert, which
  leaves ephimeral features out of the team comparison;
- the components v2 introduction dialog, its styles and icons, and the
  :v2-info-shown profile prop;
- the graphic asset events and builders and the reads of graphics in
  the selection (the :add-media, :mod-media and :del-media changes stay,
  so stored changes still apply), and the uncalled render-page and
  render-components helpers.

AI-assisted-by: claude-opus-5-5
2026-10-28 15:53:05 +01:00
alonso.torres
86c0c76a86 ✨ Reject imported components v1 files that have components
Both binfile importers stored a file without the components/v2 feature
as a v2 file, though its component data was never migrated. Such a file
is now rejected with :file-in-components-v1, the code that opening a v1
file already raises, when it has components or shapes linked to them
(:component-id, :shape-ref or :main-instance). A file without component
data takes the feature and imports as before.

Since every imported file now has the feature before the pending
migrations are registered, the no-op components/v2 migration goes.

The import dialog explains the rejection: the entry says that the file
uses an old type of components that Penpot no longer supports, with its
own text, as errors.deprecated talks about opening a file.

AI-assisted-by: claude-opus-5-5
2026-10-28 15:53:05 +01:00
Alejandro Alonso
4dc684dd9c Merge remote-tracking branch 'origin/staging' into develop 2026-10-28 11:50:56 +01:00
Alejandro Alonso
506923e2e9 Merge remote-tracking branch 'origin/main' into staging 2026-10-28 11:50:36 +01:00
María Valderrama
ff9506c551
✨ Add member profile support for organization attributes (#12652) 2026-10-28 11:02:42 +01:00
Alejandro Alonso
a8e8a54a2b Merge remote-tracking branch 'origin/staging' into develop 2026-10-28 10:35:31 +01:00
Alejandro Alonso
00dec9143d
🎉 Add SREPL helpers to update emails and clear bounce reports (#12668)
Support currently fixes wrong profile emails and false bounce marks with
raw SQL. Add update-profile-email! (rejects collisions) and
clear-email-complaint-reports! (deletes global and profile reports and
unmutes the matching profile) so these ops run through the REPL.

Closes #12663
2026-10-28 10:25:43 +01:00
Pablo Alba
e4c1b154e3
✨ Add profile name and email to nitrate audit log (#12636)
* ✨ Add profile name and email to nitrate audit log

* ✨ Mr changes
2026-10-27 09:30:28 +01:00
Eva Marco
01e0b32850
🐛 Remap library refs inside deleted components on absorb (#12232)
* 🐛 Remap library refs inside deleted components on absorb

Deleted components keep their shapes in :objects, to be restored and
to resolve their remaining copies. When a library was absorbed into a
file (export with libraries merged, unsharing or deleting the
library), absorb-assets only looked for usages in pages and in live
components. The shapes inside deleted components kept pointing to the
library, so they no longer matched their copies in the pages: the file
failed validation with :component-id-mismatch and could not be saved.

Also look for usages in deleted components, so their library
components, colors and typographies are remapped to the local file.

Relates to #10806

AI-assisted-by: claude-opus-5-5

* 🐛 Persist pointer-map fragments when absorbing a library

absorb-library-by-file saved the file without its :features. Without
"fdata/pointer-map" there, encode-file does not persist the changed
fragments, so in files with pointer maps the pages and components
that absorb-assets remapped were never written. After unpublishing or
deleting a library, its consumers failed to load with "fragment not
found".

Pass :features to update-file!. Exporting with shared libraries does
not go through this path, so it was not affected.

AI-assisted-by: claude-opus-5-5
2026-10-26 17:56:31 +01:00
Eva Marco
86e01d56a2
🐛 Update canvas and keep local sets when switching tokens source (#12442)
Setting a library as tokens source did not update shapes that use tokens,
because set-tokens-source never ran the token propagation that every other
token operation runs. Run it after the commit.

Going back to the local tokens, or unlinking the library that was the
source, left no local set active: the file has a single tokens status, and
switching to a library replaced it with the library's one. Keep the status
of the file's own tokens lib aside in a new optional :own-tokens-status
attribute while an external library is the source, and restore it when the
file is its own source again. Exporting without libraries and absorbing a
deleted source library restore it as well.

Closes #12430

AI-assisted-by: claude-opus-5-5
2026-10-26 15:23:05 +01:00
andrés gonzález
3f8aee7933
✨ Record imported file ids in the import-binfile audit event (#12491)
The RPC layer emitted the event as soon as the handler returned the
SSE response, before the import ran, so the event could not say which
files the import created. Emit it from inside the SSE handler once the
import ends, with the ids and count of the created files and whether
the import completed or failed.

Closes #12485

AI-assisted-by: claude-opus-5-5
2026-10-26 12:38:02 +01:00
Pablo Alba
3cd073fc6a
✨ Add auto-repair for integrity error files (#12524)
* ✨ Add auto-repair for integrity error files

* ✨ MR changes
2026-10-26 11:04:06 +01:00
Andrey Antukh
74add05226 Merge remote-tracking branch 'origin/staging' into develop 2026-10-23 13:40:44 +02:00
Andrey Antukh
c04af5ddc5 Merge remote-tracking branch 'origin/main' into staging 2026-10-23 13:11:52 +02:00
Andrey Antukh
76a83387d9
⬆️ Update JVM, Node, pnpm and devenv dependencies (#12600)
Move the backend JDK to its latest maintenance build within Java 26 and
refresh the dependency versions pinned by hand in the development and
production images.

- Backend: Zulu JDK 26.0.2 to 26.0.2.1. The jlink module list keeps
  jdk.management, so the prometheus process metrics stay exported.
- Node.js 24.22.1 in .nvmrc, the development image tarball and the two
  plugins deploy workflows that pin it.
- pnpm 12.15.0 in the four Dockerfiles, the 34 packageManager fields and
  the eleven lockfiles.
- Development tools: clj-kondo 2026.10.20, Babashka 1.13.226, Pixi
  0.83.0, GitHub CLI 2.103.0, AWS CLI 2.38.5 and uv 0.13.5.
- Infra services: RustFS 1.0.3, Mailpit v1.32.0, and Valkey pinned to
  8.1.10 instead of the floating 8.1 tag.

Fix SERENA_UPDATE_VERSION in the compose file. A value there makes the
start script reinstall that Serena release on every start, overriding
the one baked into the image; it was replacing 1.7.0 with 1.6.1.

The hardened base image updates are left to Renovate.

Closes #12599

AI-assisted-by: deepseek-v4.1-flash

Signed-off-by: Andrey Antukh <niwi@niwi.nz>
2026-10-23 12:36:11 +02:00
Andrey Antukh
ba78ccb8c9
✨ Harden file data offload job with lock, summary and metrics (#12403)
Take the file row with FOR UPDATE SKIP LOCKED when the offload job
starts and skip the run when the row is locked or gone, so the job
never runs while the file is being edited. Every live hot-storage
row of the file moves to cold storage inside one all-or-nothing
transaction, and the handler answers a summary map with the
offloaded rows, bytes and per-type stats.

Add bounded metrics for the job: rows and bytes counters labeled by
row type on successful runs only, and a timing histogram with the
ok, skip and error outcomes. The job definition now requires
metrics, the recorders live in app.jobs.metrics, and the existing
ephemeral job timing histogram pins its buckets.

Cover the job with a new test namespace: the happy path, the locked
row, missing and deleted files, the rows and bytes summary with its
no-op rerun, the whole-file abort on storage failure, the metric
definitions and emissions, and the submit-time params validation.
An end-to-end suite reads, snapshots, writes, exports, imports and
deletes offloaded files; concurrency tests pin the stale-revn skip,
the blocked-edit retry and the reclaimable orphans of a partial
failure. The file-gc scheduler test pins one job per eligible file,
and the scheduler private helper drops its bang.

Document the lock invariant, the deliberate no-pacing decision and
the metric label sets in the backend storage memory and the
configuration guide.

Related to #12398.

AI-assisted-by: muse-spark-1.3-contributor
2026-10-22 15:31:16 +02:00
Andrey Antukh
f8f1a668f7
♻️ Use skip-if-active for cron no-overlap in backend worker (#12412)
The cron tick kept its own active-job count beside the shared
submit option. Delegate the no-overlap check to
::jobs/skip-if-active so both paths share one queue-qualified
check served by the partial index.

A tick now skips only when a live job sits on the :cron queue,
and a skip records both the cron counter and the generic skipped
counter.

Closes #12411

AI-assisted-by: muse-spark-1.3-contributor
2026-10-22 15:29:37 +02:00
Alejandro Alonso
347a52d514
🐛 Keep Path out of update-team-photo audit props (#12530)
Nitrate encodes audit chunks as JSON and cannot serialize java.nio.file.Path;
mirror update-profile-photo and replace the upload map with safe props.
2026-10-22 11:07:07 +02:00
Elena Torró
05e1981acc
✨ Support running several exporter replicas (#12487)
* ✨ Support running several exporter replicas

* 📚 Document one exporter per machine instead of compose replicas

* 📚 Trim exporter replica docs to the essentials

* ✨ Cancel an exporter's jobs when it shuts down

* 📚 Replace exporter replica docs with a short beta note
2026-10-21 15:30:46 +02:00
Eva Marco
d8862c759b
🐛 Fix shared files 500 on path content without coordinates (#12426)
* 🐛 Fix legacy path migrations on content without coordinates

PRE still returned 500 on get-team-shared-files after #12357, now with
"expected rect instance" in legacy-6. When a path has no points, legacy-6
rebuilds its rect from the content, but content whose points are not
numbers (NaN or infinite coordinates, segments without params, only a
close path) gives no rect either, and rect->points rejects nil.

Fall back to the empty rect in legacy-6. A path with segments missing
coordinates then failed later, in 0003-convert-path-content-v2, since the
binary path data can not store them; drop those segments there.

Relates to #12343

AI-assisted-by: claude-opus-5-5

* 🐛 Skip shared files that fail to load in get-team-shared-files

get-team-shared-files loads and migrates every shared file of the team,
so one file whose migration fails made the whole request return 500 and
the "Add library" modal could not open.

Catch the error per file, log it with the file and team ids, and leave
that file out of the list. The other libraries still show, and the error
report names the broken file.

Closes #12343

AI-assisted-by: claude-opus-5-5
2026-10-21 12:31:03 +02:00
Andrey Antukh
fcf9c27635 Merge remote-tracking branch 'origin/staging' into develop 2026-10-21 10:14:55 +02:00
Andrey Antukh
028547e19b Merge remote-tracking branch 'origin/main' into staging 2026-10-21 10:09:59 +02:00
Andrey Antukh
021492a577 🐛 Make migration 0156 compatible with PostgreSQL 15
`SET STORAGE DEFAULT` needs PostgreSQL 16, but the official docker
images pin `postgres:15`, so migration 0156 failed on those
deployments. The migration now uses `SET STORAGE EXTENDED`, the
storage mode of the type of every column it touches (text, text[],
jsonb and bytea), which leaves the catalog in the same state on any
supported version.

The documented PostgreSQL floor goes back to >=15, the reason for
raising it was that syntax.

Verified on the test database: the 62 columns reset and no table
column keeps the legacy EXTERNAL storage.

The backend memory now records that migrations must stay within
PostgreSQL 15 syntax.

AI-assisted-by: minimax-m3.1-flash-free
2026-10-20 20:36:24 +00:00
Andrey Antukh
114928ea5f 🐛 Fix the key reported when a redis DEL times out
`app.redis/del` logged `:key key`, but `key` is not bound in that
`let`: it resolved to `clojure.core/key`, so the line carried
`#object[clojure.core$key ...]` instead of the keys that failed to be
deleted. Log `:keys keys`, which is always the normalized vector, so a
single key and a collection report the same shape.

The tests pin the line: the keys, the hint, the error level, the logger
and the timeout as cause, plus the `nil` the caller gets.

The prod-infra memory now notes that client-side redis timeouts mean the
JVM is paused, not that the server is slow.

AI-assisted-by: deepseek-v4.1-flash
2026-10-20 15:33:37 +00:00
Pablo Alba
a5dac03073
✨ Add organization SSO and invitation audit events for nitrate (#12461) 2026-10-20 14:08:40 +02:00
Andrey Antukh
be2e117f67
♻️ Rename audit-log archivers to consumers and use base Nexus URI (#12459)
* ♻️ Rename audit-log archivers to consumers

The pending-queue owners are no longer only external archives: a
local, non-archiving drain is about to join them, so "archiver" no
longer fits. Rename the column, identifiers, jobs, metrics and docs
to "consumer" in one release; migration 0157 never reached
production, so a single RENAME COLUMN is enough, with no backfill.

AI-assisted-by: muse-spark-1.3-contributor

* ♻️ Use base Nexus URI with appended ingest path

Mirror the admin-console setup: PENPOT_NEXUS_URI holds the base
URI and the consumer appends api/audit itself, instead of a full
endpoint URI. Keeps the explicit :uri job param as override and the
:task-not-configured guard when no base is set.

AI-assisted-by: muse-spark-1.3-contributor
2026-10-19 23:25:44 +02:00
Andrey Antukh
ccf99c7dcf
♻️ Archive audit log from a pending queue with two jobs (#12373)
* ♻️ Archive audit log from a pending queue with two jobs

Replace the mark-and-sweep audit archive with a pending queue
(audit_log_pending, migration 0157): every row is born pending
for the archivers enabled at insert time, in the same
transaction, so a rollback takes both. Two jobs share one
namespace, one queue each: :audit-log-archive-nexus and
:audit-log-archive-nitrate, each with its own closed params
schema, deps and every-5m cron entry gated by its send flag.

Each job drains its queue oldest-first in 128-row chunks (one
transaction per chunk, same total order, no SKIP LOCKED so
arrival stays chronological) and deletes its pendings on
completion; whoever deletes the last pending deletes the row.
Housekeeping lives in both jobs: reap of disabled pendings
plus sweep of orphans, both idempotent. A refused chunk raises
:archive-chunk-refused and aborts the job; the cron retries.
No separate GC task.

Metrics with bounded labels only: archived/failed/skipped per
destination, discarded per reason (reaped pendings, swept rows:
the silent R3 loss, now dashboard-visible). Timing comes from
the jobs substrate; queue depth is read off the replica DB.
No gauge, no custom histogram. Destination hosts log once at
job-def init; per-run logs carry counts plus frozen elapsed
(log props evaluate lazily in the logger thread).

Pre-existing rows are swept unsent (no backfill, accepted
loss); rows born with no archiver on share that fate (R3).
Memory mem:audit-log documents the design.

AI-assisted-by: muse-spark-1.3-contributor

* ♻️ Address review comments on audit log archive jobs

Rename the second archive destination from nitrate to admin-console
(jobs, metric and log labels, error code), reusing the :admin-console
flag. Unify the pending-queue insert into a single helper used by
both producers. Stagger the archive crons (Nexus at second 0, admin
console at second 30). Add per-call dbg logs to the admin-console
send. Update the audit-log memory and the affected tests.

Closes #12372

AI-assisted-by: muse-spark-1.3-contributor
2026-10-19 22:35:58 +02:00
Andrey Antukh
fffc3ee222 Merge remote-tracking branch 'origin/staging' into develop 2026-10-19 22:18:27 +02:00
Andrey Antukh
fdd66978fd Merge branch 'main' into staging 2026-10-19 20:10:20 +02:00
Andrey Antukh
2d335e9e83 🔧 Reset explicit EXTERNAL storage to PG defaults
Drop the legacy TOAST STORAGE EXTERNAL overrides set when PostgreSQL
ran on ZFS with transparent compression. Migration 0156 restores the
type default (SET STORAGE DEFAULT) on 62 columns across 21 tables;
existing TOAST data keeps its strategy until rewritten.

Raise the documented PostgreSQL floor to >=16, required by the new
SET STORAGE DEFAULT syntax.

AI-assisted-by: muse-spark-1.3-contributor
2026-10-19 15:37:13 +00:00
Andrey Antukh
61a66f31ab 🚑 Disable media storage quota check on upload
Comment out the media-storage-bytes-per-team check in
upload-file-media-object. The SUM over storage_object takes
3-12s per upload on prod (3.5TB DB, ~34 slow runs/hour)
and blocks uploads. Keep the SQL and quote definition
for re-enable with a cheaper implementation.

Update the upload quota test to document the disabled
state until the new implementation lands.

AI-assisted-by: muse-spark-1.3-contributor
2026-10-19 15:22:10 +00:00
Andrey Antukh
5abf66dbf0 🐛 Set a finite idle timeout on binfile transactions
Binfile v1/v3 import and the v3 export get-file step hold a single
transaction while the ZIP is decoded and the file data is realized.
Disabling idle_in_transaction_session_timeout left a stuck connection
holding locks (and a pool slot) forever.

Raise it to 30 minutes instead. The longest import/export observed in
production is under 8 minutes, so the limit still bounds a stuck
session. statement_timeout stays at 300 s.

Add set-transaction-idle-timeout and configure-import-transaction in
app.binfile.common, use them from both paths, and cover the helper
with a test.

AI-assisted-by: deepseek-v4.1-flash
2026-10-19 16:31:15 +02:00
Andrey Antukh
688c0fc398
🐛 Keep file_change revisions on their own deletion schedule (#12419)
* 🐛 Keep file_change revisions on their own deletion schedule

Restoring a file from the trash cleared the deletion mark on every
file_change row of that file, revisions included. A revision is born
already scheduled for deletion one hour later, and objects-gc only
looks at rows that carry a date, so unmarking one left it in the
table for ever: production had 358,517 of them across 3,614 files.
The file deletion cascade had the mirror problem: it rewrote the date
of rows that already had one, pushing a revision out to the trash
lifetime.

Restrict the restore to durable history and date only the rows that
have no date yet, so a revision keeps the schedule it was born with.

A pinning test covers the snapshot update path, which cannot reach a
revision because it resolves its row through a read filter. The
backend memories now carry the model of the three file_data classes
and this invariant, in place of the two superseded stubs.

Closes #12418

AI-assisted-by: deepseek-v4.1-flash

* 📚 Add autovacuum and Postgres exporter notes to Grafana memory

Record what this Grafana instance can and cannot show about autovacuum
so the next session does not have to re-derive it:

- The Postgres exporter scrapes the primary as well as the
  read-only replica, so the primary's pg_stat_activity and
  pg_settings_* are in Prometheus even though the SQL
  datasource cannot reach them.
- It publishes no pg_stat_progress_* collector, and
  pg_stat_activity_count has no backend_type label: autovacuum
  workers are the usename/application_name-less bucket with
  state=active. The launcher shows up as the AutoVacuumMain
  wait event, and it is absent on the replica.
- A metric the exporter does not publish needs a custom query
  block in its YAML queries file and a restart of the exporter.
- The Postgres server ships its own log to Loki (the job is
  postgresql, primary only), whose streams carry only filename,
  instance, job and level. The completed vacuum lines live
  there: a per-table history, not a view of what runs now.
- The PostgreSQL section now ends with the vacuum panels,
  with the Vacuum TXID age table last; an in-flight per-table
  view would need a pg_stat_progress_vacuum collector in the
  exporter itself.

AI-assisted-by: deepseek-v4.1-flash
2026-10-19 14:00:13 +02:00
Andrey Antukh
84d92cfd5a
✨ Add tail buckets and update-file validation timings (#12409)
* ✨ Extend backend histogram buckets beyond 7.5s

Give every backend timing histogram a tail past the shared 7500ms cap
so slow RPC calls, CLIMIT waits and background tasks report real
quantiles instead of clipping at the top bucket.

Request and wait metrics grow to 60s, background tasks to 300s and
Redis eval to 10s; the sub-7.5s buckets stay untouched to keep the
history comparable.

Closes #12408

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Fix flaky assertion in climit republish test

Read the last published waiting value instead of the exact
publication list. The sampler thread started with the component
runs its first tick asynchronously, and when it lands inside the
recording window it publishes the same count twice, which made
the full-suite CI run fail while focused runs stayed green.

AI-assisted-by: muse-spark-1.3-contributor

* ✨ Add file_data blob to JSON decode helper

Allow decoding file_data blobs (app.util.blob) to JSON
for investigations with jq, offline from a blob file or
from nREPL with the backend running.

AI-assisted-by: muse-spark-1.3-contributor

* ✨ Add validation step timings to update-file logs and metrics

Measure the six phases inside process-changes-and-validate
(resolve-libs, process-changes, soft and hard affected and
schema validations) with tpoint and try/finally, so timings
are recorded even when a hard validation throws.

Publish each phase to a new histogram
penpot_rpc_update_file_validation_timing with a fixed step
label using the request buckets (up to 60000 ms), and extend
the update-file debug log with the same *-ms fields plus
file, project and team ids. The deferred log only runs on
success, so failures get an inline error log with partial
timings.

AI-assisted-by: muse-spark-1.3-contributor

* 💄 Translate decode-file-data comments to English

Translate the Spanish docstrings, section headers and test
messages of the file_data decode helper to English, keeping
code untouched. The repo convention is English in code and
comments; the helper was first written in Spanish.

AI-assisted-by: muse-spark-1.3-contributor
2026-10-19 13:01:35 +02:00
Andrey Antukh
5262aab3ff
🐛 Count saved versions in the version quota (#12405)
* 🐛 Count saved versions in the version quota

The snapshots-per-file and snapshots-per-team quotes counted the
saved versions by file_change.data, a column that only the legacy-db
file data backend writes: versions keep their payload in file_data,
so the count returned 0 and a configured quota never applied.

Count by label instead, which tells a version apart with every file
data backend, and add a partial index on file_change with the same
predicate so both counts are answered by an index only scan instead
of walking every row of every file of the team (10.3 seconds in
production for a team with 829 versions).

Closes #12404

AI-assisted-by: deepseek-v4.1-flash

* ♻️ Drop the ! suffix from the snapshot quota test helpers

The check helpers assert the outcome of a command instead of mutating
anything, so they do not take the ! that this codebase reserves for
functions whose contract changes state or runs a command (the
th/command! call inside them already carries it).

AI-assisted-by: deepseek-v4.1-flash
2026-10-19 11:15:51 +02:00
Rafael X.
addc5bc31e
📚 Fix typo in incomplete user info hint (#12413)
Signed-off-by: Rafael X. <chocoladisco@gmail.com>
2026-10-18 22:29:18 +02:00
Andrey Antukh
49aeeb053b 🐛 Use multipart helper in jobs upload-chunk test
The multipart-only upload guard rejects the :content map sent by
upload-chunked! over plain command!. Send it through
multipart-command! like the other upload tests; forged-path tests
stay on plain command!.

AI-assisted-by: muse-spark-1.3-contributor
2026-10-17 21:53:29 +00:00
Andrey Antukh
b633093336 Merge remote-tracking branch 'origin/staging' into develop 2026-10-17 16:28:59 +02:00
Andrey Antukh
70b317a976
🐛 Harden upload path handling and validation error bodies (#12067)
* 🚑 Reject forged upload paths outside multipart

Transit bodies can smuggle upload maps whose path points at any
server file; the schema only checks the shape and handlers open
the path directly.

app.rpc now injects a server-side multipart flag that bodies cannot
forge, and methods declare their upload keys with the new
multipart-only-params metadata, enforced centrally after schema
validation with a params-validation error. Covers all nine upload
methods, including chunked and management flows.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Sanitize raw explain in generic validation error bodies

The generic :validation branch returned ex-data untouched, so errors
carrying the raw malli explain (live schema objects, e.g. storage
invalid-storage-metadata) broke JSON encoding mid-stream and clients
got truncated bodies. Apply the same clean treatment the known codes
already use: drop the raw explain and attach its humanized text.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Add e2e path traversal tests for binfile and media upload

Add two e2e suites that hand-write transit bodies with a forged
["~#path", ...] upload path (no transit library needed) and expect
400 validation/params-validation. They cover import-binfile :file,
upload-file-media-object :content and upload-chunk :content, plus a
plain-JSON string-path control case. Before the multipart-path fix
they failed (500 io-exception, invalid-image, 200 stored chunk);
after it they pass.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Use multipart helper in binfile SVG test setup

The SVG import setup uploaded through plain command!, so the new
multipart-only guard rejected it with params-validation. Switch
the setup to multipart-command! like the other upload tests.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Use multipart helper in remaining upload tests

The multipart-only upload guard rejects plain RPC calls carrying
upload maps. Five tests with real files still used command! and
failed with params-validation. Switch them to multipart-command!
while keeping the forged-content tests on plain command!.

AI-assisted-by: muse-spark-1.3-contributor
2026-10-17 16:01:04 +02:00
Andrey Antukh
15606a3f85 🐛 Answer 400 on a request body the parser cannot read
A body declared as JSON that clojure.data.json cannot read ended as a 500
:unexpected with the whole stack trace in the log: wrap-parse-request
classified the failure by exception class, and the library signals its
syntax errors with a bare java.lang.Exception.

Classify it where it happens instead: read-body-params reads the body and
raises :validation :malformed-json (400) for anything it cannot parse,
keeps RequestTooBigException at 413 and re-raises a structured error from
deeper untouched. The body is read outside the handler try, so the error
goes out as a throw like the other specific-exception branches.

The failure is now one log line (path, peer address, content-type, parser
message) without the stack trace, and it is counted: the middleware takes
the metrics instance as a middleware argument, so the backend exports
penpot_http_malformed_requests_total and nexus declares its own
nexus_http_malformed_requests_total.

Tests: parse-request-malformed-json-body reproduces the production body
and asserts the 400 and the counter; the existing parse-request tests now
pass a metrics stub.

AI-assisted-by: deepseek-v4.1-flash
2026-10-17 15:54:52 +02:00
Andrey Antukh
8c5113b815 Merge remote-tracking branch 'origin/staging' into develop 2026-10-17 14:44:44 +02:00
Andrey Antukh
d1aa36b814 ⬆️ Update pnpm to 12.14.0 and Node to v24.22.0
Bump the pnpm pin to 12.14.0 in the devenv and the production images
and stamp the same version into every package.json, realigning the
drift between the devenv (12.9.1) and the rest (12.6.0).

Bump Node to v24.22.0, the newest patch of the Krypton LTS line, in
.nvmrc, the devenv and image Dockerfiles and the plugins-deploy
workflows.

The pnpm arch checksums are those of the GitHub release tarballs, not
of the npm package. Lockfile changes are metadata only.

AI-assisted-by: deepseek-v4.1-flash
2026-10-17 14:00:43 +02:00
Andrey Antukh
da1791bf00 📎 Disable temporary the storage gc job from cron 2026-10-16 15:57:32 +02:00