An :add-page change can carry a whole page under :page instead of an
id and a name. extract-affected-ids reads the page to check from the
change's top-level :id, so that form left the new page unchecked and
a referential error in it was stored.
Collect those page ids after the changes are applied and run the
affected check on them in soft mode: log the file id, the page ids
and the error codes, and never refuse the save. This is the first
step of the two-step rollout. The follow-up moves the page id into
extract-affected-ids and makes the check hard. The counters of
#12665 decide when to flip it.
Relates to #12666
AI-assisted-by: claude-opus-5-5, deepseek-v4.1-flash
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
Move the backend JDK to its latest maintenance build within Java 26 and
refresh the dependency versions pinned by hand in the development and
production images.
- Backend: Zulu JDK 26.0.2 to 26.0.2.1. The jlink module list keeps
jdk.management, so the prometheus process metrics stay exported.
- Node.js 24.22.1 in .nvmrc, the development image tarball and the two
plugins deploy workflows that pin it.
- pnpm 12.15.0 in the four Dockerfiles, the 34 packageManager fields and
the eleven lockfiles.
- Development tools: clj-kondo 2026.10.20, Babashka 1.13.226, Pixi
0.83.0, GitHub CLI 2.103.0, AWS CLI 2.38.5 and uv 0.13.5.
- Infra services: RustFS 1.0.3, Mailpit v1.32.0, and Valkey pinned to
8.1.10 instead of the floating 8.1 tag.
Fix SERENA_UPDATE_VERSION in the compose file. A value there makes the
start script reinstall that Serena release on every start, overriding
the one baked into the image; it was replacing 1.7.0 with 1.6.1.
The hardened base image updates are left to Renovate.
Closes#12599
AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
Take the file row with FOR UPDATE SKIP LOCKED when the offload job
starts and skip the run when the row is locked or gone, so the job
never runs while the file is being edited. Every live hot-storage
row of the file moves to cold storage inside one all-or-nothing
transaction, and the handler answers a summary map with the
offloaded rows, bytes and per-type stats.
Add bounded metrics for the job: rows and bytes counters labeled by
row type on successful runs only, and a timing histogram with the
ok, skip and error outcomes. The job definition now requires
metrics, the recorders live in app.jobs.metrics, and the existing
ephemeral job timing histogram pins its buckets.
Cover the job with a new test namespace: the happy path, the locked
row, missing and deleted files, the rows and bytes summary with its
no-op rerun, the whole-file abort on storage failure, the metric
definitions and emissions, and the submit-time params validation.
An end-to-end suite reads, snapshots, writes, exports, imports and
deletes offloaded files; concurrency tests pin the stale-revn skip,
the blocked-edit retry and the reclaimable orphans of a partial
failure. The file-gc scheduler test pins one job per eligible file,
and the scheduler private helper drops its bang.
Document the lock invariant, the deliberate no-pacing decision and
the metric label sets in the backend storage memory and the
configuration guide.
Related to #12398.
AI-assisted-by: muse-spark-1.3-contributor
The cron tick kept its own active-job count beside the shared
submit option. Delegate the no-overlap check to
::jobs/skip-if-active so both paths share one queue-qualified
check served by the partial index.
A tick now skips only when a live job sits on the :cron queue,
and a skip records both the cron counter and the generic skipped
counter.
Closes#12411
AI-assisted-by: muse-spark-1.3-contributor
* ✨ Support running several exporter replicas
* 📚 Document one exporter per machine instead of compose replicas
* 📚 Trim exporter replica docs to the essentials
* ✨ Cancel an exporter's jobs when it shuts down
* 📚 Replace exporter replica docs with a short beta note
* 🐛 Fix legacy path migrations on content without coordinates
PRE still returned 500 on get-team-shared-files after #12357, now with
"expected rect instance" in legacy-6. When a path has no points, legacy-6
rebuilds its rect from the content, but content whose points are not
numbers (NaN or infinite coordinates, segments without params, only a
close path) gives no rect either, and rect->points rejects nil.
Fall back to the empty rect in legacy-6. A path with segments missing
coordinates then failed later, in 0003-convert-path-content-v2, since the
binary path data can not store them; drop those segments there.
Relates to #12343
AI-assisted-by: claude-opus-5-5
* 🐛 Skip shared files that fail to load in get-team-shared-files
get-team-shared-files loads and migrates every shared file of the team,
so one file whose migration fails made the whole request return 500 and
the "Add library" modal could not open.
Catch the error per file, log it with the file and team ids, and leave
that file out of the list. The other libraries still show, and the error
report names the broken file.
Closes#12343
AI-assisted-by: claude-opus-5-5
`SET STORAGE DEFAULT` needs PostgreSQL 16, but the official docker
images pin `postgres:15`, so migration 0156 failed on those
deployments. The migration now uses `SET STORAGE EXTENDED`, the
storage mode of the type of every column it touches (text, text[],
jsonb and bytea), which leaves the catalog in the same state on any
supported version.
The documented PostgreSQL floor goes back to >=15, the reason for
raising it was that syntax.
Verified on the test database: the 62 columns reset and no table
column keeps the legacy EXTERNAL storage.
The backend memory now records that migrations must stay within
PostgreSQL 15 syntax.
AI-assisted-by: minimax-m3.1-flash-free
`app.redis/del` logged `:key key`, but `key` is not bound in that
`let`: it resolved to `clojure.core/key`, so the line carried
`#object[clojure.core$key ...]` instead of the keys that failed to be
deleted. Log `:keys keys`, which is always the normalized vector, so a
single key and a collection report the same shape.
The tests pin the line: the keys, the hint, the error level, the logger
and the timeout as cause, plus the `nil` the caller gets.
The prod-infra memory now notes that client-side redis timeouts mean the
JVM is paused, not that the server is slow.
AI-assisted-by: deepseek-v4.1-flash
* ♻️ Rename audit-log archivers to consumers
The pending-queue owners are no longer only external archives: a
local, non-archiving drain is about to join them, so "archiver" no
longer fits. Rename the column, identifiers, jobs, metrics and docs
to "consumer" in one release; migration 0157 never reached
production, so a single RENAME COLUMN is enough, with no backfill.
AI-assisted-by: muse-spark-1.3-contributor
* ♻️ Use base Nexus URI with appended ingest path
Mirror the admin-console setup: PENPOT_NEXUS_URI holds the base
URI and the consumer appends api/audit itself, instead of a full
endpoint URI. Keeps the explicit :uri job param as override and the
:task-not-configured guard when no base is set.
AI-assisted-by: muse-spark-1.3-contributor
* ♻️ Archive audit log from a pending queue with two jobs
Replace the mark-and-sweep audit archive with a pending queue
(audit_log_pending, migration 0157): every row is born pending
for the archivers enabled at insert time, in the same
transaction, so a rollback takes both. Two jobs share one
namespace, one queue each: :audit-log-archive-nexus and
:audit-log-archive-nitrate, each with its own closed params
schema, deps and every-5m cron entry gated by its send flag.
Each job drains its queue oldest-first in 128-row chunks (one
transaction per chunk, same total order, no SKIP LOCKED so
arrival stays chronological) and deletes its pendings on
completion; whoever deletes the last pending deletes the row.
Housekeeping lives in both jobs: reap of disabled pendings
plus sweep of orphans, both idempotent. A refused chunk raises
:archive-chunk-refused and aborts the job; the cron retries.
No separate GC task.
Metrics with bounded labels only: archived/failed/skipped per
destination, discarded per reason (reaped pendings, swept rows:
the silent R3 loss, now dashboard-visible). Timing comes from
the jobs substrate; queue depth is read off the replica DB.
No gauge, no custom histogram. Destination hosts log once at
job-def init; per-run logs carry counts plus frozen elapsed
(log props evaluate lazily in the logger thread).
Pre-existing rows are swept unsent (no backfill, accepted
loss); rows born with no archiver on share that fate (R3).
Memory mem:audit-log documents the design.
AI-assisted-by: muse-spark-1.3-contributor
* ♻️ Address review comments on audit log archive jobs
Rename the second archive destination from nitrate to admin-console
(jobs, metric and log labels, error code), reusing the :admin-console
flag. Unify the pending-queue insert into a single helper used by
both producers. Stagger the archive crons (Nexus at second 0, admin
console at second 30). Add per-call dbg logs to the admin-console
send. Update the audit-log memory and the affected tests.
Closes#12372
AI-assisted-by: muse-spark-1.3-contributor
Drop the legacy TOAST STORAGE EXTERNAL overrides set when PostgreSQL
ran on ZFS with transparent compression. Migration 0156 restores the
type default (SET STORAGE DEFAULT) on 62 columns across 21 tables;
existing TOAST data keeps its strategy until rewritten.
Raise the documented PostgreSQL floor to >=16, required by the new
SET STORAGE DEFAULT syntax.
AI-assisted-by: muse-spark-1.3-contributor
Comment out the media-storage-bytes-per-team check in
upload-file-media-object. The SUM over storage_object takes
3-12s per upload on prod (3.5TB DB, ~34 slow runs/hour)
and blocks uploads. Keep the SQL and quote definition
for re-enable with a cheaper implementation.
Update the upload quota test to document the disabled
state until the new implementation lands.
AI-assisted-by: muse-spark-1.3-contributor
Binfile v1/v3 import and the v3 export get-file step hold a single
transaction while the ZIP is decoded and the file data is realized.
Disabling idle_in_transaction_session_timeout left a stuck connection
holding locks (and a pool slot) forever.
Raise it to 30 minutes instead. The longest import/export observed in
production is under 8 minutes, so the limit still bounds a stuck
session. statement_timeout stays at 300 s.
Add set-transaction-idle-timeout and configure-import-transaction in
app.binfile.common, use them from both paths, and cover the helper
with a test.
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Keep file_change revisions on their own deletion schedule
Restoring a file from the trash cleared the deletion mark on every
file_change row of that file, revisions included. A revision is born
already scheduled for deletion one hour later, and objects-gc only
looks at rows that carry a date, so unmarking one left it in the
table for ever: production had 358,517 of them across 3,614 files.
The file deletion cascade had the mirror problem: it rewrote the date
of rows that already had one, pushing a revision out to the trash
lifetime.
Restrict the restore to durable history and date only the rows that
have no date yet, so a revision keeps the schedule it was born with.
A pinning test covers the snapshot update path, which cannot reach a
revision because it resolves its row through a read filter. The
backend memories now carry the model of the three file_data classes
and this invariant, in place of the two superseded stubs.
Closes#12418
AI-assisted-by: deepseek-v4.1-flash
* 📚 Add autovacuum and Postgres exporter notes to Grafana memory
Record what this Grafana instance can and cannot show about autovacuum
so the next session does not have to re-derive it:
- The Postgres exporter scrapes the primary as well as the
read-only replica, so the primary's pg_stat_activity and
pg_settings_* are in Prometheus even though the SQL
datasource cannot reach them.
- It publishes no pg_stat_progress_* collector, and
pg_stat_activity_count has no backend_type label: autovacuum
workers are the usename/application_name-less bucket with
state=active. The launcher shows up as the AutoVacuumMain
wait event, and it is absent on the replica.
- A metric the exporter does not publish needs a custom query
block in its YAML queries file and a restart of the exporter.
- The Postgres server ships its own log to Loki (the job is
postgresql, primary only), whose streams carry only filename,
instance, job and level. The completed vacuum lines live
there: a per-table history, not a view of what runs now.
- The PostgreSQL section now ends with the vacuum panels,
with the Vacuum TXID age table last; an in-flight per-table
view would need a pg_stat_progress_vacuum collector in the
exporter itself.
AI-assisted-by: deepseek-v4.1-flash
* ✨ Extend backend histogram buckets beyond 7.5s
Give every backend timing histogram a tail past the shared 7500ms cap
so slow RPC calls, CLIMIT waits and background tasks report real
quantiles instead of clipping at the top bucket.
Request and wait metrics grow to 60s, background tasks to 300s and
Redis eval to 10s; the sub-7.5s buckets stay untouched to keep the
history comparable.
Closes#12408
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Fix flaky assertion in climit republish test
Read the last published waiting value instead of the exact
publication list. The sampler thread started with the component
runs its first tick asynchronously, and when it lands inside the
recording window it publishes the same count twice, which made
the full-suite CI run fail while focused runs stayed green.
AI-assisted-by: muse-spark-1.3-contributor
* ✨ Add file_data blob to JSON decode helper
Allow decoding file_data blobs (app.util.blob) to JSON
for investigations with jq, offline from a blob file or
from nREPL with the backend running.
AI-assisted-by: muse-spark-1.3-contributor
* ✨ Add validation step timings to update-file logs and metrics
Measure the six phases inside process-changes-and-validate
(resolve-libs, process-changes, soft and hard affected and
schema validations) with tpoint and try/finally, so timings
are recorded even when a hard validation throws.
Publish each phase to a new histogram
penpot_rpc_update_file_validation_timing with a fixed step
label using the request buckets (up to 60000 ms), and extend
the update-file debug log with the same *-ms fields plus
file, project and team ids. The deferred log only runs on
success, so failures get an inline error log with partial
timings.
AI-assisted-by: muse-spark-1.3-contributor
* 💄 Translate decode-file-data comments to English
Translate the Spanish docstrings, section headers and test
messages of the file_data decode helper to English, keeping
code untouched. The repo convention is English in code and
comments; the helper was first written in Spanish.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Count saved versions in the version quota
The snapshots-per-file and snapshots-per-team quotes counted the
saved versions by file_change.data, a column that only the legacy-db
file data backend writes: versions keep their payload in file_data,
so the count returned 0 and a configured quota never applied.
Count by label instead, which tells a version apart with every file
data backend, and add a partial index on file_change with the same
predicate so both counts are answered by an index only scan instead
of walking every row of every file of the team (10.3 seconds in
production for a team with 829 versions).
Closes#12404
AI-assisted-by: deepseek-v4.1-flash
* ♻️ Drop the ! suffix from the snapshot quota test helpers
The check helpers assert the outcome of a command instead of mutating
anything, so they do not take the ! that this codebase reserves for
functions whose contract changes state or runs a command (the
th/command! call inside them already carries it).
AI-assisted-by: deepseek-v4.1-flash
The multipart-only upload guard rejects the :content map sent by
upload-chunked! over plain command!. Send it through
multipart-command! like the other upload tests; forged-path tests
stay on plain command!.
AI-assisted-by: muse-spark-1.3-contributor
* 🚑 Reject forged upload paths outside multipart
Transit bodies can smuggle upload maps whose path points at any
server file; the schema only checks the shape and handlers open
the path directly.
app.rpc now injects a server-side multipart flag that bodies cannot
forge, and methods declare their upload keys with the new
multipart-only-params metadata, enforced centrally after schema
validation with a params-validation error. Covers all nine upload
methods, including chunked and management flows.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Sanitize raw explain in generic validation error bodies
The generic :validation branch returned ex-data untouched, so errors
carrying the raw malli explain (live schema objects, e.g. storage
invalid-storage-metadata) broke JSON encoding mid-stream and clients
got truncated bodies. Apply the same clean treatment the known codes
already use: drop the raw explain and attach its humanized text.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Add e2e path traversal tests for binfile and media upload
Add two e2e suites that hand-write transit bodies with a forged
["~#path", ...] upload path (no transit library needed) and expect
400 validation/params-validation. They cover import-binfile :file,
upload-file-media-object :content and upload-chunk :content, plus a
plain-JSON string-path control case. Before the multipart-path fix
they failed (500 io-exception, invalid-image, 200 stored chunk);
after it they pass.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Use multipart helper in binfile SVG test setup
The SVG import setup uploaded through plain command!, so the new
multipart-only guard rejected it with params-validation. Switch
the setup to multipart-command! like the other upload tests.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Use multipart helper in remaining upload tests
The multipart-only upload guard rejects plain RPC calls carrying
upload maps. Five tests with real files still used command! and
failed with params-validation. Switch them to multipart-command!
while keeping the forged-content tests on plain command!.
AI-assisted-by: muse-spark-1.3-contributor
A body declared as JSON that clojure.data.json cannot read ended as a 500
:unexpected with the whole stack trace in the log: wrap-parse-request
classified the failure by exception class, and the library signals its
syntax errors with a bare java.lang.Exception.
Classify it where it happens instead: read-body-params reads the body and
raises :validation :malformed-json (400) for anything it cannot parse,
keeps RequestTooBigException at 413 and re-raises a structured error from
deeper untouched. The body is read outside the handler try, so the error
goes out as a throw like the other specific-exception branches.
The failure is now one log line (path, peer address, content-type, parser
message) without the stack trace, and it is counted: the middleware takes
the metrics instance as a middleware argument, so the backend exports
penpot_http_malformed_requests_total and nexus declares its own
nexus_http_malformed_requests_total.
Tests: parse-request-malformed-json-body reproduces the production body
and asserts the 400 and the counter; the existing parse-request tests now
pass a metrics stub.
AI-assisted-by: deepseek-v4.1-flash
Bump the pnpm pin to 12.14.0 in the devenv and the production images
and stamp the same version into every package.json, realigning the
drift between the devenv (12.9.1) and the rest (12.6.0).
Bump Node to v24.22.0, the newest patch of the Krypton LTS line, in
.nvmrc, the devenv and image Dockerfiles and the plugins-deploy
workflows.
The pnpm arch checksums are those of the GitHub release tarballs, not
of the npm package. Lockfile changes are metadata only.
AI-assisted-by: deepseek-v4.1-flash
The storage-gc-deleted task used to stream all the pending rows with
db/plan inside one long read-only transaction, blocking the table
vacuum for the whole run and generating bloat on long deletions.
Now every chunk of 25 objects is one short read-write transaction:
the SELECT reads and locks the next objects after a (deleted_at, id)
keyset cursor with FOR UPDATE SKIP LOCKED, deletes the database rows
grouped by backend and removes their backend content. Concurrent runs
never take the same chunk, and the double SELECT of the old lock-ids
step is gone.
A chunk that fails aborts its transaction: nothing of the chunk is
deleted, its locks are released, and the run skips it; the cursor
still advances to the last row read, so a failed chunk is not visited
again in the same run.
AI-assisted-by: glm-5.3-flash
* ✨ Count the application logs as a metric, by logger and level
The error panels had no metric to read: they counted log lines in Loki,
because the backend published nothing about the errors of the application
(the only thing alive in Prometheus was the http listener error count,
which measures another thing). Now there is
`penpot_log_events_total{logger, level}`.
The counter comes from a new component, `app.loggers.metrics/reporter`,
which hooks a watch on the log record like the database and mattermost
reporters do, but without a thread or a channel: the increment is in
memory, so it runs in the watcher itself. Only `warn`, `error` and `fatal`
are counted, and only the records that go through `app.common.logging`:
the lines a library writes straight to log4j2 are not counted (one in a
thousand today).
The body of the watcher is wrapped in a `try/catch` that swallows the
error on purpose: the watcher fires from the `swap!` of the log record,
inside the logging path, so an exception there would reach whoever was
logging and would take the logging executor down with it. A test pins that
a failing publication does not propagate.
The suite removes the reporter (it would count the logs of the tests and
the recording of their `mtx/run!`), so a test of its own pins the wiring.
The memories carry the counter, its two limits and why the `try/catch` is
not noise; the grafana one also brings the notes of this afternoon about
the service account, the dashboard search and Cloudflare, which were
waiting for a commit.
AI-assisted-by: deepseek-v4.1-flash
* 📚 Address the review findings on the log-events metric
The review of the counter found that the reason written for the
`try/catch` was not the real one, so the memory and the comments say now
what happens if it is removed: with `::l/sync?` the exception reaches
whoever was logging, and on the async path it dies in the uncaught
handler of the logging executor -- outside the application's logging --
where the pool replaces the worker, so logging continues and that line is
lost. They also say that a publication failing every time is silent: the
counter stays flat and nothing is written about it.
The rest are small things the review asked for: the test of the `fatal`
level (it was in the counted set and nothing verified it), the HELP of the
metric naming the three levels, the require of the test without the unused
alias that clj-kondo flagged, and the note that the `level` is lowercase
here and uppercase in Loki.
The grafana memory loses what was said twice: the two pairs of bullets
about reading the API (the service account 403s and the search that lies)
are merged, the CLIMIT notes are one bullet instead of two, and the four
bullets about editing dashboards are grouped in their own section. Its
host name is gone as well, which the first line of that file forbids.
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Decode job results with the schema of the job-def
A job result is stored as plain JSON, so a uuid comes back as the text
it is on disk and a uuid used as a map key comes back as a keyword. The
import resolution is a map keyed by file id, and the wizard lost that
identity: choosing a library in the link step created no link at all,
with no error and with the modal closing as if it had worked.
A job-def can now declare a result schema and its precompiled decoder,
the pair the params already have, and the substrate applies them when the
row is read. A def that declares neither answers with the JSON the row
holds, as before; a row whose job-def is gone is read untouched instead
of raising.
Relates to #12339
AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 🐛 Type the import result so its resolution keeps uuid keys
The resolution of a version 3 import is a map keyed by the new file id,
and the result column is plain JSON: without a schema of its own every
key came back as a keyword while the :id of the entry came back as text,
so the dashboard wizard could not match the file it was linking and
created no library link at all.
Declare the shape of the resolution (files, done and pending entries and
their candidates) and give the job-def its result schema and decoder.
The check before the result is stored keeps validating the same schema.
Closes#12339
AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 🐛 Take the imported file id from the entry, not the map key
The wizard looked a file up in the selection with the key of the
resolution map while it had indexed the selection by the :id of the
entry. The key is keywordized on its way through the job result, so the
lookup missed and the confirm button created no library link at all,
with no error and with the modal closing as if it had worked.
Take the id from the entry, the source the rest of the wizard already
uses, and move the decision to a pure function covered by tests of both
shapes of the resolution.
Relates to #12339
AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 🐛 Test that an import result keeps its ids as uuids
The regression this branch fixes is only visible through the read path
of the substrate: the resolution of an import is a map keyed by the new
file id, and a test that works with the value in memory never sees the
keys coming back as keywords from the plain JSON column.
Import a package whose library needs a replacement chosen by the user,
read the result with `jobs/get-job` as the client does, and check the
keys, the file ids and the ids of the candidates. Without the result
decoder of the import job-def the test answers with text ids.
Relates to #12339
AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 📚 Document the result schema of a job-def
The memories and the docstring of the substrate said a job result has no
schema of its own and comes back as the JSON the row holds. A job-def can
now declare its result schema and decoder, and the substrate applies them
when the row is read.
Keep the rule next to it: a result map keyed by uuid needs its schema
declared, or every key comes back as a keyword, and the link-later step
of the import wizard takes the file id from the entry and never from the
key of the resolution map.
Relates to #12339
AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 📚 Fix the documented contract of a job result
The docstring of get-job said a result has no schema of its own and comes
back as the JSON the row holds, which is what this branch changes: the
result is decoded with the schema its job-def declares, and a result that
does not conform never reaches the column because the write validates it.
Say the same in the management command that stores the result of an
external worker, the other writer of the column.
Relates to #12339
AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 🐛 Check the result of a job against its schema before storing it
The result of a job crosses a plain JSON column, so a uuid comes back as
text and a uuid used as a map key as a keyword. The job-def now declares
the schema of its result and the substrate compiles the decoder from it,
which is what the read path uses to get the types back.
The same schema is the contract of the write path: complete refuses a
result that does not conform, with the malli explanation, so nothing
invalid reaches the column, and the answer is the same for the runner and
for the external worker that completes a job through the management RPC.
A job that is no longer completable is skipped without checking, and the
read path decodes only: a row written by another version stays readable.
Relates to #12339
AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 🐛 Check the result of an import inside its transaction
The check ran after the transaction of the import had committed, so a
result the schema does not accept failed the job with the files already
created in the destination project: a half imported project and an error.
The core decides whether the whole import is kept or rolled back, so the
check belongs inside its transaction, which is what turns a contract
violation into a rollback.
The substrate checks the result again when the job is completed, which is
the only net for a job that does not validate its own result.
Relates to #12339
AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 🐛 Cover the edges of an import result read by the client
The shape of a result with no libraries to resolve, and the one of a
version 1 import, is the most common in production and neither was covered
by the tests that read a result the way the client does.
Cover both through jobs/get-job, and the case of a registry that no longer
knows the kind of the row: it answers with the JSON of the column,
keywordized keys and uuid as text, instead of raising. Say in the two tests
that depend on the decoder what implementation without it makes them fail.
Relates to #12339
AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 📚 Fix the prose and the fixtures of the result contract
The result schema of a job-def is what the read decodes with and what the
write checks: say that in the memories, with the rule that any uuid of a
result needs it, whether a value or a map key.
The motive of the wizard helper was a fact this branch makes false: what a
key of the resolution map holds depends on the result schema of the job-def
that wrote it, so the rule is to read the id from the entry. Same
correction in its memory and in the test that pins the shape.
The candidates of a pending entry match by slugified name, not by name, and
every test row now states the kind of its job, because the name is what
resolves its job-def when the result is read or checked.
Relates to #12339
AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* ♻️ Keep the wizard link helper private and cover its shape
The helper that decides which links the wizard creates belongs to its
namespace, as its neighbours do: reach it from the test with @#', the
pattern the frontend tests already use for a private helper.
Cover the shapes it had no test for: a file with two libraries pending
resolution, a file whose libraries were auto-linked, and a selection where
only one of two entries was resolved.
Relates to #12339
AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* ♻️ Share one job-row helper across the job tests
Three test namespaces had their own copy of the same insert, differing only
in the kind of the job and in whether the row holds the resource it
consumes. Move it to the test helpers as create-job*, with the kind of the
job as its first argument: it is what resolves the job-def of the row when
its result is read or checked, so every test states it.
The bang of the name goes away with it: the design rules keep it for the
mutating API of app.db, and this is a fixture of the tests. The three
namespaces lose the requires that only their removed copy used.
Relates to #12339
AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
---------
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* ✨ Handle tokens libraries when exporting .penpot files
Exporting a file that takes its tokens from an external library left
the :tokens-source of the file pointing at a library that is not in the
export, and the tokens lib of that library was not exported.
With :detach-libraries, unset the :tokens-source and the tokens status
in detach-external-references. The file is exported without any link to
the library, and the shapes keep their applied tokens.
With :merge-libraries, add cfo/absorb-tokens-lib, which copies the tokens
lib of a library into a file and unsets its :tokens-source, so the file
becomes its own tokens source. The tokens status of the file is kept; if
it has none, the one of the library is copied. If the library has no
tokens lib, only the source is unset. embed-assets calls it for the
library that is the tokens source of the file, and ignores the other
linked libraries.
Add unit tests for absorb-tokens-lib, and binfile tests that export a
file with tokens from an external library with both modes, checking the
exported files, the tokens lib, the tokens source and the applied
tokens.
Also reformat the export-type docstring in binfile v3 and remove a
leftover debug prn in the export dialog.
AI-assisted-by: claude-sonnet-5-5
* 🐛 Preserve active token sets when importing linked libraries
Cache each decoded tokens library for the duration of an import so files
and their tokens source resolve active sets against the same generated IDs.
Add a round-trip regression test for exports that include linked libraries.
* 🐛 Preserve tokens when deleting or unpublishing libraries
Absorb the tokens library into files that use it as their tokens source.
Keep referencing file IDs in the deletion task before removing relations.
Embed tokens from deleted, unlinked sources during export, and clear stale
source and status data when the library no longer exists. Add regression
tests for deletion, unpublishing, and exports with unavailable libraries.
---------
Co-authored-by: alonso.torres <alonso.torres@kaleidos.net>
Route upload-session chunk liveness through the mark-time reference
scan instead of the delete-time cleanup in gc-deleted.
gc-touched now checks upload_session_chunk before marking a chunk,
so referenced chunks freeze and only orphans are marked. The freeze
path re-verifies references under row locks, which also closes the
stale-freeze race for every scanned bucket. gc-deleted drops its
upload-session special case: deferred rows keep their mappings until
the final give-up, and a live reference aborts the chunk loud.
Closes#12302
AI-assisted-by: muse-spark-1.3-contributor
* ✨ Enrich backend audit events with team and entity ids
Backend audit events now carry the data a consumer needs to rebuild
state without joining production tables:
- prepare-rpc-event adds :team-id when the event carries a project or
file id; commands that already loaded the team set it themselves.
- add-profile-to-team! emits add-team-member on every real insert.
- create-project and create-file emit the new row id.
- clone-template, import-binfile and duplicate-project emit one event
per created entity.
- both binfile versions return {:file-ids [...]} from import-files!.
Closes#12219
AI-assisted-by: deepseek-v4.1-flash
* ✨ Emit delete and hard-delete audit events with entity data
The event stream must let Nexus rebuild local state without joining
production tables. This completes the enrichment with the delete paths:
- delete-team keeps returning nil but now carries the deleted row
(id, name, timestamps) in the audit props, like delete-project.
- create-team adds created-at to its audit props.
- delete-profile carries id, email, fullname and deleted-at; its
props used to be empty because every param is namespaced.
- objects-gc emits hard-delete-team/project/file per purged batch,
with the deleted ids, only when the audit log is on.
- delete-team-member needs no change: the delete is physical and the
params already carry team-id and member-id.
The nitrate push-audit-events tests now drop the membership events
emitted by their own profile setup before counting.
Closes#12219
AI-assisted-by: muse-spark-1.3-free
* ✨ Announce cascade deletes and missing ids in audit events
The :delete-object worker task marked teams, projects and files without
leaving any event, so an event-sourced projection kept them alive for
weeks. It now emits one cascade-deleted event per execution with every
affected id, in the same transaction, only with :audit-log on. Rows
purged by objects-gc and snapshot cascades stay silent on purpose.
Also: create-team-with-invitations carries the new team id, and
update-team-member-role names the demoted member on owner transfers.
Closes#12219
AI-assisted-by: muse-spark-1.3-free
* 🐛 Identify the SREPL profile delete in audit events
The SREPL delete-profile! emitted no profile-id and empty props, so the
row could not be attributed by any consumer. It now carries the profile
id in both fields. delete-team! named the wrong cause in its context.
Closes#12219
AI-assisted-by: muse-spark-1.3-free
* 🐛 Identify the SREPL profile restore in audit events
The SREPL restore-profile! emitted no profile-id, so the consumer
keyed on the zero profile instead of the restored one. It now carries
the profile id like a normal profile event, and no consumer special
case is needed.
Closes#12219
AI-assisted-by: muse-spark-1.3-free
* 🐛 Stop attributing SREPL events to the subject profile
profile-id names the actor, and SREPL calls have none: setting the
subject id blamed the user for an admin action and would have leaked
those rows into PostHog as user activity. The subject id stays in
props, where the consumer resolves it, and the event profile stays
zero.
Closes#12219
AI-assisted-by: muse-spark-1.3-free
* ✨ Expose file analysis data for the nexus pipeline
The nexus file-security pipeline needs file content and scheduling
state from the backend. This adds a read-only management method,
`get-file-for-analysis`, returning the semantic projection of one
file (pages, frames, texts, links, interactions; ids as strings;
capped with a truncated flag), plus identity, revn, timestamps and
share state. No prior projection code existed, so the method ships
its own pure `file->projection`.
Audit events now carry what the `:local` consumer schedules from:
revn and modified-at on update-file, name and timestamps on every
create-file emitter, deleted-at on delete-file, and new
create/delete-share-link events with file and link ids.
AI-assisted-by: muse-spark-1.3-contributor
* ♻️ Rename deletion audit events to action names
Event names are actions, not past tense: `cascade-deleted` becomes
`delete-object-with-cascade` with the same shape and transaction.
The three `hard-delete-*` events merge into `hard-delete-objects`:
one event per purged chunk carrying only that chunk's entity key
(`:team-ids`, `:project-ids` or `:file-ids`), because each GC
section purges in its own chunk transactions and a single event
could never share them all. Every submit stays inside its delete
transaction, `triggered-by` stays annotated, and silent paths stay
silent.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Read worker parallelism knobs with plain config keys
The runner wiring looked up `:app.main/worker-*-parallelism`
(namespaced) while the schema and the environment provide plain
keys, so `PENPOT_WORKER_DEFAULT_PARALLELISM` and
`PENPOT_WORKER_WEBHOOK_PARALLELISM` never applied and workers
always ran with the fallback. Read the keys the config holds.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Fix binfile audit test with a real file row
`submit-file-created-event` reads the light columns back from the
row, so the mocked importer must return an existing file id: a
phantom id emits no event and the test saw zero `create-file`
events. Create the file first and use its id. The audit-log memory
now tells both bulk-event helpers apart.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Fix audit event fidelity from review findings
The review of the audit enrichment found data-fidelity gaps; fix
them while the branch is fresh.
Membership events now belong to the actor: the flows where the
owner adds somebody else pass an explicit actor id, the role is
derived from the flags (the table has no role column), and the
event carries the request context like its siblings.
Also: team-id resolution skips read events, duplicate-project
emits the project name and timestamps, delete-profile is
idempotent again, the GC tests assert physical purge, and
get-file-for-analysis leaves an audit trace.
AI-assisted-by: muse-spark-1.3-contributor
* ♻️ Nest cascade ids under :cascade in delete event
The cascade event carried every affected id flat next to the
object identity; nest them under `:cascade` so the payload reads
as header plus affected set. The event is an action like its
siblings, not a command. Drop the `:audit-log` flag re-check:
`submit` already fans out per flag.
AI-assisted-by: muse-spark-1.3-contributor
* 📎 Update copyright name on file header
The company name changed; a single header in this branch still
carried the old one. Align it with the current form.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Fix nil event-id on webhook trigger audit rows
process-event generated the row id inside append-audit-entry, so the
:process-webhook-event task never received it and the trigger row
stored a nil :event-id. Assign the id up-front and share it between
the stored row and the worker params; the telemetry shadow row keeps
its own id to avoid a PK collision. Add regression tests covering
the handoff and the end-to-end trigger row.
AI-assisted-by: muse-spark-1.3-contributor
After the unified jobs migration, cron submitted empty params that
failed validation because the job schema still required Integrant
system keys. Narrow the schema to optional REPL overrides, wire
:app.nitrate/client onto the archive job-def, and drop the unused
::jobs/defs ref from nitrate that created an Integrant cycle.
AI-assisted-by: composer
The merge of main into staging brought the closed storage metadata
schema, which requires file-id and id on file-data objects. The
restore test built one without those keys, so it failed with an
invalid metadata error. Its helper now takes extra metadata and
the test passes the ids of the file and of its file-data row, as
production does.
The republish test asserted the exact list of waiting-gauge
publications, but the component's sampler starts its first tick right
away and under load lands inside the recorded window, publishing the
same count twice. The test now builds the counters alone, so the
republish call is the only publisher.
AI-assisted-by: deepseek-v4.1-flash
The queue gauge was written by whichever limiter of a name ran last, so a
per-profile pile-up never showed: a healthy profile writing 0 overwrote the
15 of the profile that was queueing (4 of 1441 samples in production), and
a limiter whose requests never come back read 0 while 72 threads stayed
parked.
A queue is an entry and an exit, so the new gauge is accumulated: one
counter per limiter name and instance, up when a call starts waiting for a
permit, down when it stops, so a call that never leaves stays visible. It
is always written with `:val`, and a daemon sampler republishes every count
every 15s so a name whose traffic died does not go stale in the scrape;
halting the component stops that thread.
Rejections had no metric at all: penpot_rpc_climit_rejected_total counts
them by name and reason (queue-full or timeout), and the log lines now
carry queue-max, permits, code and reason, so an operator can tell whether
the limiter did its job or was saturated.
The gauges that could not tell the truth are gone: queue and permits (same
last-write-wins problem, and waiting plus the wait histogram replace them)
and http_worker_busy_threads (reads 0 on an instance with 72 blocked
threads; pool_size is the signal). No other dashboard or alert rule reads
them.
AI-assisted-by: deepseek-v4.1-flash
Add a per-process availability flag (app.health) that /readyz
reports as 503 OFFLINE while offline, so the load balancer
drains the instance and in-flight requests finish untouched.
Operator surface is app.srepl.main/set-offline and
app.srepl.main/set-online; a restart comes back online.
Covered by backend-tests.health-test; HA docs updated.
AI-assisted-by: muse-spark-1.3-contributor
`file_object_thumbnail` was replaced by `file_tagged_object_thumbnail`
and nothing refers to it anymore: no query in the backend touches it,
no table has a foreign key pointing at it, no view depends on it, and
it is empty in production. It also lacked `deleted_at`, which is why
restoring files had to leave it out. Only legacy naming survives in two
helpers (`get-file-object-thumbnails` and
`delete-file-object-thumbnails!`, which operate on
`file_tagged_object_thumbnail`).
The storage bucket named "file-object-thumbnail", the one that
`file_tagged_object_thumbnail` uses, is a different thing and stays.
AI-assisted-by: deepseek-v4.1-flash
Restoring a file only un-marked the database rows (`file`,
`file_media_object`, `file_change`, `file_data`, `file_thumbnail`,
`file_tagged_object_thumbnail`), not the storage objects those rows
reference. An object that was already marked for deletion stayed
marked while a live row pointed at it, and those foreign keys are `NO
ACTION`: the storage GC cannot delete it and skips it forever (and
before the GC learned to skip, it cost the whole nightly run).
Restoring a team had the same gap with its font variants.
Both restores now unmark the storage objects of the rows they bring
back: the media and the thumbnail of the file media objects, the media
of the file and object thumbnails, the `storage-ref-id` of the file
data, and the four files of a font variant. Objects referenced by other
rows are left alone.
AI-assisted-by: deepseek-v4.1-flash
objects-gc deleted every kind of object inside one transaction per proc,
looping until empty, so a single row that a foreign key still references
aborted the whole run: the task died on its first file every night, after
the earlier steps had already spent around twenty minutes. The per-file
savepoint only saved the files proc and left the same shape everywhere
else.
Now every proc streams its pending keys with db/plan inside a read-only
transaction of its own and deletes them chunk by chunk, each chunk in a
transaction of its own with FOR UPDATE SKIP LOCKED. A chunk that fails is
reported with l/err and skipped, so the run reaches the end and the next
run retries the skipped rows.
AI-assisted-by: muse-spark-1.3-contributor
