Keep the string helper required by the variant rename predicate after
removing the old swap-slot helpers. Sort the merged test imports.
Send import-audit fixtures through the existing multipart RPC helper so
they exercise auditing through the required upload path. Start the edit
watcher in the isolated component test so it forwards recorded transitions.
AI-assisted-by: GPT-6
Keep the container root helper used by the main-instance name migration.
Adapt imported-file ids and component swap tests to the current APIs, and
check copied swaps through their position refs and swap groups.
AI-assisted-by: gpt-6
Moving files or a project to another team unlinked every library that
ended up in another team than the file using it, but absorbed nothing:
the copies kept pointing at a library their file could no longer
reach. A move now does what deleting the library does (PENPOT-1540):
it absorbs each such library into the files using it, then unlinks
them; as on deletion, a failed absorption is logged and does not stop
the move. That covers a moved library and the files left behind using
it, a moved file and the libraries it used, and a moved project.
The links checked are those that start or end at a moved file. They
were those of the files in the source and destination projects, so a
file in another project of the old team kept its link to a moved
library across teams.
rpc-management-test: a moved library is absorbed into a file in its
project and into one in a project the move does not touch; a moved
file absorbs its library; a moved project absorbs the library it left
behind.
AI-assisted-by: claude-opus-5-5
Deleting a shared library removes its links to the files using it at
once, so they stop listing it, and queues the delete-object job, which
absorbs the library into the files it finds through those links. By
then there were none, so nothing was absorbed: the files lost the
library and their copies kept pointing at the deleted file
(PENPOT-1092). Unpublishing absorbs before it unlinks, so it was not
affected.
The delete reads the files using the library before unlinking them
and gives them to the job (:absorb-file-ids), which absorbs into them;
without them, as from the project and team deletions, the job still
finds them through the links.
rpc-file-test/delete-shared-library-absorbs-it-into-the-files-using-it
deletes the library through the RPC and runs the job it queued.
AI-assisted-by: claude-opus-5-5
Absorbing a library into a file that uses it (unpublishing or deleting
the library) saved the file without its features. Saving stores the
changed pointer-map fragments only when the features name
fdata/pointer-map, so with that feature the file pointed at a fragment
of components that was never written, and loading it failed with
:fragment-not-found. The file's features now go with the save.
rpc-file-test/unpublish-library-absorbs-it-into-the-files-using-it
unpublishes a library with a copy in another file, then loads that
file. The existing absorption test turns fdata/pointer-map off, so it
missed this.
AI-assisted-by: claude-opus-5-5
Files older than the feature components/text-overrides mark a copy text
with :text-content-attribute (any style) or :text-content-structure
(letters and styles). Migration 0034 compares each such copy with its
near main and gives it the groups its old flags allowed where it
differs; an old flag where nothing differs sets nothing. A copy whose
near main cannot be reached keeps every group its flags allowed, so it
loses no override. The migration adds the feature, so a client on the
old code cannot write the old groups into a migrated file.
Binfile import leaves the copies whose near main is in a library not
linked yet, as it does with swap slots, and converts them once the
libraries are linked: the deferred pass of the import now runs both
conversions (convert-deferred!, *defer-unreachable*).
Validation reports the old groups (:legacy-text-touched), and its
repair recomputes them against the near main; a text override group
without :content-group is still :invalid-text-touched. A shape with no
letters in it and its main no longer counts as differing in its styles.
Benchmark (lower quantile, ms; baseline / now / ratio):
deepest-edit-through-three-levels 35.2 / 34.8 / 0.99
edit-through-a-chain-of-dependents 68.9 / 65.0 / 0.94
library-update-into-copies 83.1 / 86.9 / 1.05
main-edit-into-10-copies 1.8 / 1.5 / 0.86
main-edit-into-100-copies 15.4 / 15.0 / 0.97
main-edit-into-1000-copies 153.1 / 150.6 / 0.98
variant-switch-into-copies 32.1 / 26.8 / 0.84
AI-assisted-by: claude-opus-5-5
The migration 0033 stores every swap of a file as an override group.
A swap made in a copy is paired with the child of the near main that
holds the same position, as sync paired it; any other slot is dropped.
All pairings are found on the data as it was, so swaps nested in swaps
pair against the content of the outer one, and libraries pair the same
whether they are migrated or not. A head that refers to its own slot
only inherited it, and drops it even when its near main cannot be
reached. Any other slot that matches no child, or whose near main is in
a library that is not loaded, keeps the slot as its ref, so validation
reports it. An import, which migrates its files before it links their
libraries, leaves those slots and converts them once the libraries are
linked. A migrated head that sits elsewhere than the child it replaces
would place it gets its geometry marked as touched, so the first sync
does not move it. Migrated and new files carry the feature
components/swap-group, so a client that still writes slots cannot open
them.
Validation checks the new form: :misplaced-slot reports any slot,
:misplaced-swap-group the group anywhere but on a nested copy head in a
copy, :ref-not-in-near-main a nested copy head whose ref is not a child
of the near main of its parent, and :duplicate-ref two children of a
copy that refer to the same main shape. Each has a repair.
Nothing but the migrations reads slots any more: the slot branches go
from sync, swap, switch, reset, detach, paste, duplicate and move. The
helpers that read and write a slot on a shape move to files/swap-slots,
and the ones that follow refs to files/comp-processors. The graph
projection drops the FillsSwapSlot edge, the components debugger shows
the component a swapped head takes its content from, and the plugin API
documents componentRefShape for a swapped copy. The swap tests are
named after what they check.
Benchmark (lower quantile, ms; baseline / now / ratio):
deepest-edit-through-three-levels 35.2 / 34.6 / 0.98
edit-through-a-chain-of-dependents 68.9 / 63.5 / 0.92
library-update-into-copies 83.1 / 76.4 / 0.92
main-edit-into-10-copies 1.8 / 1.6 / 0.87
main-edit-into-100-copies 15.4 / 15.1 / 0.98
main-edit-into-1000-copies 153.1 / 148.8 / 0.97
variant-switch-into-copies 32.1 / 26.4 / 0.82
AI-assisted-by: claude-opus-5-5
No components v1 file reaches the code any more: opening one fails the
feature check, and imports reject it or give it the feature. What only
served v1 goes:
- validate-file and validate-file-affected no longer skip a file
without components/v2;
- the root of a deleted component is found by its main instance id
only, in the type helper and the component sprite;
- generate-sync-library, which synced shapes inside live components,
and the live components in containers-seq, so asset usages walk the
pages only (deleted components keep their objects and still sync);
- the unused :components-v2 file option, from the schema, the file
data and the test fixtures; stored files that carry it still
validate;
- the team migration-in-progress check and its frontend alert, which
leaves ephimeral features out of the team comparison;
- the components v2 introduction dialog, its styles and icons, and the
:v2-info-shown profile prop;
- the graphic asset events and builders and the reads of graphics in
the selection (the :add-media, :mod-media and :del-media changes stay,
so stored changes still apply), and the uncalled render-page and
render-components helpers.
AI-assisted-by: claude-opus-5-5
Both binfile importers stored a file without the components/v2 feature
as a v2 file, though its component data was never migrated. Such a file
is now rejected with :file-in-components-v1, the code that opening a v1
file already raises, when it has components or shapes linked to them
(:component-id, :shape-ref or :main-instance). A file without component
data takes the feature and imports as before.
Since every imported file now has the feature before the pending
migrations are registered, the no-op components/v2 migration goes.
The import dialog explains the rejection: the entry says that the file
uses an old type of components that Penpot no longer supports, with its
own text, as errors.deprecated talks about opening a file.
AI-assisted-by: claude-opus-5-5
Support currently fixes wrong profile emails and false bounce marks with
raw SQL. Add update-profile-email! (rejects collisions) and
clear-email-complaint-reports! (deletes global and profile reports and
unmutes the matching profile) so these ops run through the REPL.
Closes#12663
* 🐛 Remap library refs inside deleted components on absorb
Deleted components keep their shapes in :objects, to be restored and
to resolve their remaining copies. When a library was absorbed into a
file (export with libraries merged, unsharing or deleting the
library), absorb-assets only looked for usages in pages and in live
components. The shapes inside deleted components kept pointing to the
library, so they no longer matched their copies in the pages: the file
failed validation with :component-id-mismatch and could not be saved.
Also look for usages in deleted components, so their library
components, colors and typographies are remapped to the local file.
Relates to #10806
AI-assisted-by: claude-opus-5-5
* 🐛 Persist pointer-map fragments when absorbing a library
absorb-library-by-file saved the file without its :features. Without
"fdata/pointer-map" there, encode-file does not persist the changed
fragments, so in files with pointer maps the pages and components
that absorb-assets remapped were never written. After unpublishing or
deleting a library, its consumers failed to load with "fragment not
found".
Pass :features to update-file!. Exporting with shared libraries does
not go through this path, so it was not affected.
AI-assisted-by: claude-opus-5-5
The RPC layer emitted the event as soon as the handler returned the
SSE response, before the import ran, so the event could not say which
files the import created. Emit it from inside the SSE handler once the
import ends, with the ids and count of the created files and whether
the import completed or failed.
Closes#12485
AI-assisted-by: claude-opus-5-5
Take the file row with FOR UPDATE SKIP LOCKED when the offload job
starts and skip the run when the row is locked or gone, so the job
never runs while the file is being edited. Every live hot-storage
row of the file moves to cold storage inside one all-or-nothing
transaction, and the handler answers a summary map with the
offloaded rows, bytes and per-type stats.
Add bounded metrics for the job: rows and bytes counters labeled by
row type on successful runs only, and a timing histogram with the
ok, skip and error outcomes. The job definition now requires
metrics, the recorders live in app.jobs.metrics, and the existing
ephemeral job timing histogram pins its buckets.
Cover the job with a new test namespace: the happy path, the locked
row, missing and deleted files, the rows and bytes summary with its
no-op rerun, the whole-file abort on storage failure, the metric
definitions and emissions, and the submit-time params validation.
An end-to-end suite reads, snapshots, writes, exports, imports and
deletes offloaded files; concurrency tests pin the stale-revn skip,
the blocked-edit retry and the reclaimable orphans of a partial
failure. The file-gc scheduler test pins one job per eligible file,
and the scheduler private helper drops its bang.
Document the lock invariant, the deliberate no-pacing decision and
the metric label sets in the backend storage memory and the
configuration guide.
Related to #12398.
AI-assisted-by: muse-spark-1.3-contributor
The cron tick kept its own active-job count beside the shared
submit option. Delegate the no-overlap check to
::jobs/skip-if-active so both paths share one queue-qualified
check served by the partial index.
A tick now skips only when a live job sits on the :cron queue,
and a skip records both the cron counter and the generic skipped
counter.
Closes#12411
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Fix legacy path migrations on content without coordinates
PRE still returned 500 on get-team-shared-files after #12357, now with
"expected rect instance" in legacy-6. When a path has no points, legacy-6
rebuilds its rect from the content, but content whose points are not
numbers (NaN or infinite coordinates, segments without params, only a
close path) gives no rect either, and rect->points rejects nil.
Fall back to the empty rect in legacy-6. A path with segments missing
coordinates then failed later, in 0003-convert-path-content-v2, since the
binary path data can not store them; drop those segments there.
Relates to #12343
AI-assisted-by: claude-opus-5-5
* 🐛 Skip shared files that fail to load in get-team-shared-files
get-team-shared-files loads and migrates every shared file of the team,
so one file whose migration fails made the whole request return 500 and
the "Add library" modal could not open.
Catch the error per file, log it with the file and team ids, and leave
that file out of the list. The other libraries still show, and the error
report names the broken file.
Closes#12343
AI-assisted-by: claude-opus-5-5
`app.redis/del` logged `:key key`, but `key` is not bound in that
`let`: it resolved to `clojure.core/key`, so the line carried
`#object[clojure.core$key ...]` instead of the keys that failed to be
deleted. Log `:keys keys`, which is always the normalized vector, so a
single key and a collection report the same shape.
The tests pin the line: the keys, the hint, the error level, the logger
and the timeout as cause, plus the `nil` the caller gets.
The prod-infra memory now notes that client-side redis timeouts mean the
JVM is paused, not that the server is slow.
AI-assisted-by: deepseek-v4.1-flash
* ♻️ Rename audit-log archivers to consumers
The pending-queue owners are no longer only external archives: a
local, non-archiving drain is about to join them, so "archiver" no
longer fits. Rename the column, identifiers, jobs, metrics and docs
to "consumer" in one release; migration 0157 never reached
production, so a single RENAME COLUMN is enough, with no backfill.
AI-assisted-by: muse-spark-1.3-contributor
* ♻️ Use base Nexus URI with appended ingest path
Mirror the admin-console setup: PENPOT_NEXUS_URI holds the base
URI and the consumer appends api/audit itself, instead of a full
endpoint URI. Keeps the explicit :uri job param as override and the
:task-not-configured guard when no base is set.
AI-assisted-by: muse-spark-1.3-contributor
* ♻️ Archive audit log from a pending queue with two jobs
Replace the mark-and-sweep audit archive with a pending queue
(audit_log_pending, migration 0157): every row is born pending
for the archivers enabled at insert time, in the same
transaction, so a rollback takes both. Two jobs share one
namespace, one queue each: :audit-log-archive-nexus and
:audit-log-archive-nitrate, each with its own closed params
schema, deps and every-5m cron entry gated by its send flag.
Each job drains its queue oldest-first in 128-row chunks (one
transaction per chunk, same total order, no SKIP LOCKED so
arrival stays chronological) and deletes its pendings on
completion; whoever deletes the last pending deletes the row.
Housekeeping lives in both jobs: reap of disabled pendings
plus sweep of orphans, both idempotent. A refused chunk raises
:archive-chunk-refused and aborts the job; the cron retries.
No separate GC task.
Metrics with bounded labels only: archived/failed/skipped per
destination, discarded per reason (reaped pendings, swept rows:
the silent R3 loss, now dashboard-visible). Timing comes from
the jobs substrate; queue depth is read off the replica DB.
No gauge, no custom histogram. Destination hosts log once at
job-def init; per-run logs carry counts plus frozen elapsed
(log props evaluate lazily in the logger thread).
Pre-existing rows are swept unsent (no backfill, accepted
loss); rows born with no archiver on share that fate (R3).
Memory mem:audit-log documents the design.
AI-assisted-by: muse-spark-1.3-contributor
* ♻️ Address review comments on audit log archive jobs
Rename the second archive destination from nitrate to admin-console
(jobs, metric and log labels, error code), reusing the :admin-console
flag. Unify the pending-queue insert into a single helper used by
both producers. Stagger the archive crons (Nexus at second 0, admin
console at second 30). Add per-call dbg logs to the admin-console
send. Update the audit-log memory and the affected tests.
Closes#12372
AI-assisted-by: muse-spark-1.3-contributor
Comment out the media-storage-bytes-per-team check in
upload-file-media-object. The SUM over storage_object takes
3-12s per upload on prod (3.5TB DB, ~34 slow runs/hour)
and blocks uploads. Keep the SQL and quote definition
for re-enable with a cheaper implementation.
Update the upload quota test to document the disabled
state until the new implementation lands.
AI-assisted-by: muse-spark-1.3-contributor
Binfile v1/v3 import and the v3 export get-file step hold a single
transaction while the ZIP is decoded and the file data is realized.
Disabling idle_in_transaction_session_timeout left a stuck connection
holding locks (and a pool slot) forever.
Raise it to 30 minutes instead. The longest import/export observed in
production is under 8 minutes, so the limit still bounds a stuck
session. statement_timeout stays at 300 s.
Add set-transaction-idle-timeout and configure-import-transaction in
app.binfile.common, use them from both paths, and cover the helper
with a test.
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Keep file_change revisions on their own deletion schedule
Restoring a file from the trash cleared the deletion mark on every
file_change row of that file, revisions included. A revision is born
already scheduled for deletion one hour later, and objects-gc only
looks at rows that carry a date, so unmarking one left it in the
table for ever: production had 358,517 of them across 3,614 files.
The file deletion cascade had the mirror problem: it rewrote the date
of rows that already had one, pushing a revision out to the trash
lifetime.
Restrict the restore to durable history and date only the rows that
have no date yet, so a revision keeps the schedule it was born with.
A pinning test covers the snapshot update path, which cannot reach a
revision because it resolves its row through a read filter. The
backend memories now carry the model of the three file_data classes
and this invariant, in place of the two superseded stubs.
Closes#12418
AI-assisted-by: deepseek-v4.1-flash
* 📚 Add autovacuum and Postgres exporter notes to Grafana memory
Record what this Grafana instance can and cannot show about autovacuum
so the next session does not have to re-derive it:
- The Postgres exporter scrapes the primary as well as the
read-only replica, so the primary's pg_stat_activity and
pg_settings_* are in Prometheus even though the SQL
datasource cannot reach them.
- It publishes no pg_stat_progress_* collector, and
pg_stat_activity_count has no backend_type label: autovacuum
workers are the usename/application_name-less bucket with
state=active. The launcher shows up as the AutoVacuumMain
wait event, and it is absent on the replica.
- A metric the exporter does not publish needs a custom query
block in its YAML queries file and a restart of the exporter.
- The Postgres server ships its own log to Loki (the job is
postgresql, primary only), whose streams carry only filename,
instance, job and level. The completed vacuum lines live
there: a per-table history, not a view of what runs now.
- The PostgreSQL section now ends with the vacuum panels,
with the Vacuum TXID age table last; an in-flight per-table
view would need a pg_stat_progress_vacuum collector in the
exporter itself.
AI-assisted-by: deepseek-v4.1-flash
* ✨ Extend backend histogram buckets beyond 7.5s
Give every backend timing histogram a tail past the shared 7500ms cap
so slow RPC calls, CLIMIT waits and background tasks report real
quantiles instead of clipping at the top bucket.
Request and wait metrics grow to 60s, background tasks to 300s and
Redis eval to 10s; the sub-7.5s buckets stay untouched to keep the
history comparable.
Closes#12408
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Fix flaky assertion in climit republish test
Read the last published waiting value instead of the exact
publication list. The sampler thread started with the component
runs its first tick asynchronously, and when it lands inside the
recording window it publishes the same count twice, which made
the full-suite CI run fail while focused runs stayed green.
AI-assisted-by: muse-spark-1.3-contributor
* ✨ Add file_data blob to JSON decode helper
Allow decoding file_data blobs (app.util.blob) to JSON
for investigations with jq, offline from a blob file or
from nREPL with the backend running.
AI-assisted-by: muse-spark-1.3-contributor
* ✨ Add validation step timings to update-file logs and metrics
Measure the six phases inside process-changes-and-validate
(resolve-libs, process-changes, soft and hard affected and
schema validations) with tpoint and try/finally, so timings
are recorded even when a hard validation throws.
Publish each phase to a new histogram
penpot_rpc_update_file_validation_timing with a fixed step
label using the request buckets (up to 60000 ms), and extend
the update-file debug log with the same *-ms fields plus
file, project and team ids. The deferred log only runs on
success, so failures get an inline error log with partial
timings.
AI-assisted-by: muse-spark-1.3-contributor
* 💄 Translate decode-file-data comments to English
Translate the Spanish docstrings, section headers and test
messages of the file_data decode helper to English, keeping
code untouched. The repo convention is English in code and
comments; the helper was first written in Spanish.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Count saved versions in the version quota
The snapshots-per-file and snapshots-per-team quotes counted the
saved versions by file_change.data, a column that only the legacy-db
file data backend writes: versions keep their payload in file_data,
so the count returned 0 and a configured quota never applied.
Count by label instead, which tells a version apart with every file
data backend, and add a partial index on file_change with the same
predicate so both counts are answered by an index only scan instead
of walking every row of every file of the team (10.3 seconds in
production for a team with 829 versions).
Closes#12404
AI-assisted-by: deepseek-v4.1-flash
* ♻️ Drop the ! suffix from the snapshot quota test helpers
The check helpers assert the outcome of a command instead of mutating
anything, so they do not take the ! that this codebase reserves for
functions whose contract changes state or runs a command (the
th/command! call inside them already carries it).
AI-assisted-by: deepseek-v4.1-flash
The multipart-only upload guard rejects the :content map sent by
upload-chunked! over plain command!. Send it through
multipart-command! like the other upload tests; forged-path tests
stay on plain command!.
AI-assisted-by: muse-spark-1.3-contributor
* 🚑 Reject forged upload paths outside multipart
Transit bodies can smuggle upload maps whose path points at any
server file; the schema only checks the shape and handlers open
the path directly.
app.rpc now injects a server-side multipart flag that bodies cannot
forge, and methods declare their upload keys with the new
multipart-only-params metadata, enforced centrally after schema
validation with a params-validation error. Covers all nine upload
methods, including chunked and management flows.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Sanitize raw explain in generic validation error bodies
The generic :validation branch returned ex-data untouched, so errors
carrying the raw malli explain (live schema objects, e.g. storage
invalid-storage-metadata) broke JSON encoding mid-stream and clients
got truncated bodies. Apply the same clean treatment the known codes
already use: drop the raw explain and attach its humanized text.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Add e2e path traversal tests for binfile and media upload
Add two e2e suites that hand-write transit bodies with a forged
["~#path", ...] upload path (no transit library needed) and expect
400 validation/params-validation. They cover import-binfile :file,
upload-file-media-object :content and upload-chunk :content, plus a
plain-JSON string-path control case. Before the multipart-path fix
they failed (500 io-exception, invalid-image, 200 stored chunk);
after it they pass.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Use multipart helper in binfile SVG test setup
The SVG import setup uploaded through plain command!, so the new
multipart-only guard rejected it with params-validation. Switch
the setup to multipart-command! like the other upload tests.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Use multipart helper in remaining upload tests
The multipart-only upload guard rejects plain RPC calls carrying
upload maps. Five tests with real files still used command! and
failed with params-validation. Switch them to multipart-command!
while keeping the forged-content tests on plain command!.
AI-assisted-by: muse-spark-1.3-contributor
A body declared as JSON that clojure.data.json cannot read ended as a 500
:unexpected with the whole stack trace in the log: wrap-parse-request
classified the failure by exception class, and the library signals its
syntax errors with a bare java.lang.Exception.
Classify it where it happens instead: read-body-params reads the body and
raises :validation :malformed-json (400) for anything it cannot parse,
keeps RequestTooBigException at 413 and re-raises a structured error from
deeper untouched. The body is read outside the handler try, so the error
goes out as a throw like the other specific-exception branches.
The failure is now one log line (path, peer address, content-type, parser
message) without the stack trace, and it is counted: the middleware takes
the metrics instance as a middleware argument, so the backend exports
penpot_http_malformed_requests_total and nexus declares its own
nexus_http_malformed_requests_total.
Tests: parse-request-malformed-json-body reproduces the production body
and asserts the 400 and the counter; the existing parse-request tests now
pass a metrics stub.
AI-assisted-by: deepseek-v4.1-flash
The storage-gc-deleted task used to stream all the pending rows with
db/plan inside one long read-only transaction, blocking the table
vacuum for the whole run and generating bloat on long deletions.
Now every chunk of 25 objects is one short read-write transaction:
the SELECT reads and locks the next objects after a (deleted_at, id)
keyset cursor with FOR UPDATE SKIP LOCKED, deletes the database rows
grouped by backend and removes their backend content. Concurrent runs
never take the same chunk, and the double SELECT of the old lock-ids
step is gone.
A chunk that fails aborts its transaction: nothing of the chunk is
deleted, its locks are released, and the run skips it; the cursor
still advances to the last row read, so a failed chunk is not visited
again in the same run.
AI-assisted-by: glm-5.3-flash
* ✨ Count the application logs as a metric, by logger and level
The error panels had no metric to read: they counted log lines in Loki,
because the backend published nothing about the errors of the application
(the only thing alive in Prometheus was the http listener error count,
which measures another thing). Now there is
`penpot_log_events_total{logger, level}`.
The counter comes from a new component, `app.loggers.metrics/reporter`,
which hooks a watch on the log record like the database and mattermost
reporters do, but without a thread or a channel: the increment is in
memory, so it runs in the watcher itself. Only `warn`, `error` and `fatal`
are counted, and only the records that go through `app.common.logging`:
the lines a library writes straight to log4j2 are not counted (one in a
thousand today).
The body of the watcher is wrapped in a `try/catch` that swallows the
error on purpose: the watcher fires from the `swap!` of the log record,
inside the logging path, so an exception there would reach whoever was
logging and would take the logging executor down with it. A test pins that
a failing publication does not propagate.
The suite removes the reporter (it would count the logs of the tests and
the recording of their `mtx/run!`), so a test of its own pins the wiring.
The memories carry the counter, its two limits and why the `try/catch` is
not noise; the grafana one also brings the notes of this afternoon about
the service account, the dashboard search and Cloudflare, which were
waiting for a commit.
AI-assisted-by: deepseek-v4.1-flash
* 📚 Address the review findings on the log-events metric
The review of the counter found that the reason written for the
`try/catch` was not the real one, so the memory and the comments say now
what happens if it is removed: with `::l/sync?` the exception reaches
whoever was logging, and on the async path it dies in the uncaught
handler of the logging executor -- outside the application's logging --
where the pool replaces the worker, so logging continues and that line is
lost. They also say that a publication failing every time is silent: the
counter stays flat and nothing is written about it.
The rest are small things the review asked for: the test of the `fatal`
level (it was in the counted set and nothing verified it), the HELP of the
metric naming the three levels, the require of the test without the unused
alias that clj-kondo flagged, and the note that the `level` is lowercase
here and uppercase in Loki.
The grafana memory loses what was said twice: the two pairs of bullets
about reading the API (the service account 403s and the search that lies)
are merged, the CLIMIT notes are one bullet instead of two, and the four
bullets about editing dashboards are grouped in their own section. Its
host name is gone as well, which the first line of that file forbids.
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Decode job results with the schema of the job-def
A job result is stored as plain JSON, so a uuid comes back as the text
it is on disk and a uuid used as a map key comes back as a keyword. The
import resolution is a map keyed by file id, and the wizard lost that
identity: choosing a library in the link step created no link at all,
with no error and with the modal closing as if it had worked.
A job-def can now declare a result schema and its precompiled decoder,
the pair the params already have, and the substrate applies them when the
row is read. A def that declares neither answers with the JSON the row
holds, as before; a row whose job-def is gone is read untouched instead
of raising.
Relates to #12339
AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 🐛 Type the import result so its resolution keeps uuid keys
The resolution of a version 3 import is a map keyed by the new file id,
and the result column is plain JSON: without a schema of its own every
key came back as a keyword while the :id of the entry came back as text,
so the dashboard wizard could not match the file it was linking and
created no library link at all.
Declare the shape of the resolution (files, done and pending entries and
their candidates) and give the job-def its result schema and decoder.
The check before the result is stored keeps validating the same schema.
Closes#12339
AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 🐛 Take the imported file id from the entry, not the map key
The wizard looked a file up in the selection with the key of the
resolution map while it had indexed the selection by the :id of the
entry. The key is keywordized on its way through the job result, so the
lookup missed and the confirm button created no library link at all,
with no error and with the modal closing as if it had worked.
Take the id from the entry, the source the rest of the wizard already
uses, and move the decision to a pure function covered by tests of both
shapes of the resolution.
Relates to #12339
AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 🐛 Test that an import result keeps its ids as uuids
The regression this branch fixes is only visible through the read path
of the substrate: the resolution of an import is a map keyed by the new
file id, and a test that works with the value in memory never sees the
keys coming back as keywords from the plain JSON column.
Import a package whose library needs a replacement chosen by the user,
read the result with `jobs/get-job` as the client does, and check the
keys, the file ids and the ids of the candidates. Without the result
decoder of the import job-def the test answers with text ids.
Relates to #12339
AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 📚 Document the result schema of a job-def
The memories and the docstring of the substrate said a job result has no
schema of its own and comes back as the JSON the row holds. A job-def can
now declare its result schema and decoder, and the substrate applies them
when the row is read.
Keep the rule next to it: a result map keyed by uuid needs its schema
declared, or every key comes back as a keyword, and the link-later step
of the import wizard takes the file id from the entry and never from the
key of the resolution map.
Relates to #12339
AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 📚 Fix the documented contract of a job result
The docstring of get-job said a result has no schema of its own and comes
back as the JSON the row holds, which is what this branch changes: the
result is decoded with the schema its job-def declares, and a result that
does not conform never reaches the column because the write validates it.
Say the same in the management command that stores the result of an
external worker, the other writer of the column.
Relates to #12339
AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 🐛 Check the result of a job against its schema before storing it
The result of a job crosses a plain JSON column, so a uuid comes back as
text and a uuid used as a map key as a keyword. The job-def now declares
the schema of its result and the substrate compiles the decoder from it,
which is what the read path uses to get the types back.
The same schema is the contract of the write path: complete refuses a
result that does not conform, with the malli explanation, so nothing
invalid reaches the column, and the answer is the same for the runner and
for the external worker that completes a job through the management RPC.
A job that is no longer completable is skipped without checking, and the
read path decodes only: a row written by another version stays readable.
Relates to #12339
AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 🐛 Check the result of an import inside its transaction
The check ran after the transaction of the import had committed, so a
result the schema does not accept failed the job with the files already
created in the destination project: a half imported project and an error.
The core decides whether the whole import is kept or rolled back, so the
check belongs inside its transaction, which is what turns a contract
violation into a rollback.
The substrate checks the result again when the job is completed, which is
the only net for a job that does not validate its own result.
Relates to #12339
AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 🐛 Cover the edges of an import result read by the client
The shape of a result with no libraries to resolve, and the one of a
version 1 import, is the most common in production and neither was covered
by the tests that read a result the way the client does.
Cover both through jobs/get-job, and the case of a registry that no longer
knows the kind of the row: it answers with the JSON of the column,
keywordized keys and uuid as text, instead of raising. Say in the two tests
that depend on the decoder what implementation without it makes them fail.
Relates to #12339
AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 📚 Fix the prose and the fixtures of the result contract
The result schema of a job-def is what the read decodes with and what the
write checks: say that in the memories, with the rule that any uuid of a
result needs it, whether a value or a map key.
The motive of the wizard helper was a fact this branch makes false: what a
key of the resolution map holds depends on the result schema of the job-def
that wrote it, so the rule is to read the id from the entry. Same
correction in its memory and in the test that pins the shape.
The candidates of a pending entry match by slugified name, not by name, and
every test row now states the kind of its job, because the name is what
resolves its job-def when the result is read or checked.
Relates to #12339
AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* ♻️ Keep the wizard link helper private and cover its shape
The helper that decides which links the wizard creates belongs to its
namespace, as its neighbours do: reach it from the test with @#', the
pattern the frontend tests already use for a private helper.
Cover the shapes it had no test for: a file with two libraries pending
resolution, a file whose libraries were auto-linked, and a selection where
only one of two entries was resolved.
Relates to #12339
AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* ♻️ Share one job-row helper across the job tests
Three test namespaces had their own copy of the same insert, differing only
in the kind of the job and in whether the row holds the resource it
consumes. Move it to the test helpers as create-job*, with the kind of the
job as its first argument: it is what resolves the job-def of the row when
its result is read or checked, so every test states it.
The bang of the name goes away with it: the design rules keep it for the
mutating API of app.db, and this is a fixture of the tests. The three
namespaces lose the requires that only their removed copy used.
Relates to #12339
AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
---------
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* ✨ Handle tokens libraries when exporting .penpot files
Exporting a file that takes its tokens from an external library left
the :tokens-source of the file pointing at a library that is not in the
export, and the tokens lib of that library was not exported.
With :detach-libraries, unset the :tokens-source and the tokens status
in detach-external-references. The file is exported without any link to
the library, and the shapes keep their applied tokens.
With :merge-libraries, add cfo/absorb-tokens-lib, which copies the tokens
lib of a library into a file and unsets its :tokens-source, so the file
becomes its own tokens source. The tokens status of the file is kept; if
it has none, the one of the library is copied. If the library has no
tokens lib, only the source is unset. embed-assets calls it for the
library that is the tokens source of the file, and ignores the other
linked libraries.
Add unit tests for absorb-tokens-lib, and binfile tests that export a
file with tokens from an external library with both modes, checking the
exported files, the tokens lib, the tokens source and the applied
tokens.
Also reformat the export-type docstring in binfile v3 and remove a
leftover debug prn in the export dialog.
AI-assisted-by: claude-sonnet-5-5
* 🐛 Preserve active token sets when importing linked libraries
Cache each decoded tokens library for the duration of an import so files
and their tokens source resolve active sets against the same generated IDs.
Add a round-trip regression test for exports that include linked libraries.
* 🐛 Preserve tokens when deleting or unpublishing libraries
Absorb the tokens library into files that use it as their tokens source.
Keep referencing file IDs in the deletion task before removing relations.
Embed tokens from deleted, unlinked sources during export, and clear stale
source and status data when the library no longer exists. Add regression
tests for deletion, unpublishing, and exports with unavailable libraries.
---------
Co-authored-by: alonso.torres <alonso.torres@kaleidos.net>
Route upload-session chunk liveness through the mark-time reference
scan instead of the delete-time cleanup in gc-deleted.
gc-touched now checks upload_session_chunk before marking a chunk,
so referenced chunks freeze and only orphans are marked. The freeze
path re-verifies references under row locks, which also closes the
stale-freeze race for every scanned bucket. gc-deleted drops its
upload-session special case: deferred rows keep their mappings until
the final give-up, and a live reference aborts the chunk loud.
Closes#12302
AI-assisted-by: muse-spark-1.3-contributor
* ✨ Enrich backend audit events with team and entity ids
Backend audit events now carry the data a consumer needs to rebuild
state without joining production tables:
- prepare-rpc-event adds :team-id when the event carries a project or
file id; commands that already loaded the team set it themselves.
- add-profile-to-team! emits add-team-member on every real insert.
- create-project and create-file emit the new row id.
- clone-template, import-binfile and duplicate-project emit one event
per created entity.
- both binfile versions return {:file-ids [...]} from import-files!.
Closes#12219
AI-assisted-by: deepseek-v4.1-flash
* ✨ Emit delete and hard-delete audit events with entity data
The event stream must let Nexus rebuild local state without joining
production tables. This completes the enrichment with the delete paths:
- delete-team keeps returning nil but now carries the deleted row
(id, name, timestamps) in the audit props, like delete-project.
- create-team adds created-at to its audit props.
- delete-profile carries id, email, fullname and deleted-at; its
props used to be empty because every param is namespaced.
- objects-gc emits hard-delete-team/project/file per purged batch,
with the deleted ids, only when the audit log is on.
- delete-team-member needs no change: the delete is physical and the
params already carry team-id and member-id.
The nitrate push-audit-events tests now drop the membership events
emitted by their own profile setup before counting.
Closes#12219
AI-assisted-by: muse-spark-1.3-free
* ✨ Announce cascade deletes and missing ids in audit events
The :delete-object worker task marked teams, projects and files without
leaving any event, so an event-sourced projection kept them alive for
weeks. It now emits one cascade-deleted event per execution with every
affected id, in the same transaction, only with :audit-log on. Rows
purged by objects-gc and snapshot cascades stay silent on purpose.
Also: create-team-with-invitations carries the new team id, and
update-team-member-role names the demoted member on owner transfers.
Closes#12219
AI-assisted-by: muse-spark-1.3-free
* 🐛 Identify the SREPL profile delete in audit events
The SREPL delete-profile! emitted no profile-id and empty props, so the
row could not be attributed by any consumer. It now carries the profile
id in both fields. delete-team! named the wrong cause in its context.
Closes#12219
AI-assisted-by: muse-spark-1.3-free
* 🐛 Identify the SREPL profile restore in audit events
The SREPL restore-profile! emitted no profile-id, so the consumer
keyed on the zero profile instead of the restored one. It now carries
the profile id like a normal profile event, and no consumer special
case is needed.
Closes#12219
AI-assisted-by: muse-spark-1.3-free
* 🐛 Stop attributing SREPL events to the subject profile
profile-id names the actor, and SREPL calls have none: setting the
subject id blamed the user for an admin action and would have leaked
those rows into PostHog as user activity. The subject id stays in
props, where the consumer resolves it, and the event profile stays
zero.
Closes#12219
AI-assisted-by: muse-spark-1.3-free
* ✨ Expose file analysis data for the nexus pipeline
The nexus file-security pipeline needs file content and scheduling
state from the backend. This adds a read-only management method,
`get-file-for-analysis`, returning the semantic projection of one
file (pages, frames, texts, links, interactions; ids as strings;
capped with a truncated flag), plus identity, revn, timestamps and
share state. No prior projection code existed, so the method ships
its own pure `file->projection`.
Audit events now carry what the `:local` consumer schedules from:
revn and modified-at on update-file, name and timestamps on every
create-file emitter, deleted-at on delete-file, and new
create/delete-share-link events with file and link ids.
AI-assisted-by: muse-spark-1.3-contributor
* ♻️ Rename deletion audit events to action names
Event names are actions, not past tense: `cascade-deleted` becomes
`delete-object-with-cascade` with the same shape and transaction.
The three `hard-delete-*` events merge into `hard-delete-objects`:
one event per purged chunk carrying only that chunk's entity key
(`:team-ids`, `:project-ids` or `:file-ids`), because each GC
section purges in its own chunk transactions and a single event
could never share them all. Every submit stays inside its delete
transaction, `triggered-by` stays annotated, and silent paths stay
silent.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Read worker parallelism knobs with plain config keys
The runner wiring looked up `:app.main/worker-*-parallelism`
(namespaced) while the schema and the environment provide plain
keys, so `PENPOT_WORKER_DEFAULT_PARALLELISM` and
`PENPOT_WORKER_WEBHOOK_PARALLELISM` never applied and workers
always ran with the fallback. Read the keys the config holds.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Fix binfile audit test with a real file row
`submit-file-created-event` reads the light columns back from the
row, so the mocked importer must return an existing file id: a
phantom id emits no event and the test saw zero `create-file`
events. Create the file first and use its id. The audit-log memory
now tells both bulk-event helpers apart.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Fix audit event fidelity from review findings
The review of the audit enrichment found data-fidelity gaps; fix
them while the branch is fresh.
Membership events now belong to the actor: the flows where the
owner adds somebody else pass an explicit actor id, the role is
derived from the flags (the table has no role column), and the
event carries the request context like its siblings.
Also: team-id resolution skips read events, duplicate-project
emits the project name and timestamps, delete-profile is
idempotent again, the GC tests assert physical purge, and
get-file-for-analysis leaves an audit trace.
AI-assisted-by: muse-spark-1.3-contributor
* ♻️ Nest cascade ids under :cascade in delete event
The cascade event carried every affected id flat next to the
object identity; nest them under `:cascade` so the payload reads
as header plus affected set. The event is an action like its
siblings, not a command. Drop the `:audit-log` flag re-check:
`submit` already fans out per flag.
AI-assisted-by: muse-spark-1.3-contributor
* 📎 Update copyright name on file header
The company name changed; a single header in this branch still
carried the old one. Align it with the current form.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Fix nil event-id on webhook trigger audit rows
process-event generated the row id inside append-audit-entry, so the
:process-webhook-event task never received it and the trigger row
stored a nil :event-id. Assign the id up-front and share it between
the stored row and the worker params; the telemetry shadow row keeps
its own id to avoid a PK collision. Add regression tests covering
the handoff and the end-to-end trigger row.
AI-assisted-by: muse-spark-1.3-contributor
After the unified jobs migration, cron submitted empty params that
failed validation because the job schema still required Integrant
system keys. Narrow the schema to optional REPL overrides, wire
:app.nitrate/client onto the archive job-def, and drop the unused
::jobs/defs ref from nitrate that created an Integrant cycle.
AI-assisted-by: composer
The merge of main into staging brought the closed storage metadata
schema, which requires file-id and id on file-data objects. The
restore test built one without those keys, so it failed with an
invalid metadata error. Its helper now takes extra metadata and
the test passes the ids of the file and of its file-data row, as
production does.
The republish test asserted the exact list of waiting-gauge
publications, but the component's sampler starts its first tick right
away and under load lands inside the recorded window, publishing the
same count twice. The test now builds the counters alone, so the
republish call is the only publisher.
AI-assisted-by: deepseek-v4.1-flash
