951 Commits

Author SHA1 Message Date
Andrey Antukh
be2e117f67
♻️ Rename audit-log archivers to consumers and use base Nexus URI (#12459)
* ♻️ Rename audit-log archivers to consumers

The pending-queue owners are no longer only external archives: a
local, non-archiving drain is about to join them, so "archiver" no
longer fits. Rename the column, identifiers, jobs, metrics and docs
to "consumer" in one release; migration 0157 never reached
production, so a single RENAME COLUMN is enough, with no backfill.

AI-assisted-by: muse-spark-1.3-contributor

* ♻️ Use base Nexus URI with appended ingest path

Mirror the admin-console setup: PENPOT_NEXUS_URI holds the base
URI and the consumer appends api/audit itself, instead of a full
endpoint URI. Keeps the explicit :uri job param as override and the
:task-not-configured guard when no base is set.

AI-assisted-by: muse-spark-1.3-contributor
2026-10-19 23:25:44 +02:00
Andrey Antukh
ccf99c7dcf
♻️ Archive audit log from a pending queue with two jobs (#12373)
* ♻️ Archive audit log from a pending queue with two jobs

Replace the mark-and-sweep audit archive with a pending queue
(audit_log_pending, migration 0157): every row is born pending
for the archivers enabled at insert time, in the same
transaction, so a rollback takes both. Two jobs share one
namespace, one queue each: :audit-log-archive-nexus and
:audit-log-archive-nitrate, each with its own closed params
schema, deps and every-5m cron entry gated by its send flag.

Each job drains its queue oldest-first in 128-row chunks (one
transaction per chunk, same total order, no SKIP LOCKED so
arrival stays chronological) and deletes its pendings on
completion; whoever deletes the last pending deletes the row.
Housekeeping lives in both jobs: reap of disabled pendings
plus sweep of orphans, both idempotent. A refused chunk raises
:archive-chunk-refused and aborts the job; the cron retries.
No separate GC task.

Metrics with bounded labels only: archived/failed/skipped per
destination, discarded per reason (reaped pendings, swept rows:
the silent R3 loss, now dashboard-visible). Timing comes from
the jobs substrate; queue depth is read off the replica DB.
No gauge, no custom histogram. Destination hosts log once at
job-def init; per-run logs carry counts plus frozen elapsed
(log props evaluate lazily in the logger thread).

Pre-existing rows are swept unsent (no backfill, accepted
loss); rows born with no archiver on share that fate (R3).
Memory mem:audit-log documents the design.

AI-assisted-by: muse-spark-1.3-contributor

* ♻️ Address review comments on audit log archive jobs

Rename the second archive destination from nitrate to admin-console
(jobs, metric and log labels, error code), reusing the :admin-console
flag. Unify the pending-queue insert into a single helper used by
both producers. Stagger the archive crons (Nexus at second 0, admin
console at second 30). Add per-call dbg logs to the admin-console
send. Update the audit-log memory and the affected tests.

Closes #12372

AI-assisted-by: muse-spark-1.3-contributor
2026-10-19 22:35:58 +02:00
Andrey Antukh
fffc3ee222 Merge remote-tracking branch 'origin/staging' into develop 2026-10-19 22:18:27 +02:00
Andrey Antukh
fdd66978fd Merge branch 'main' into staging 2026-10-19 20:10:20 +02:00
Andrey Antukh
61a66f31ab 🚑 Disable media storage quota check on upload
Comment out the media-storage-bytes-per-team check in
upload-file-media-object. The SUM over storage_object takes
3-12s per upload on prod (3.5TB DB, ~34 slow runs/hour)
and blocks uploads. Keep the SQL and quote definition
for re-enable with a cheaper implementation.

Update the upload quota test to document the disabled
state until the new implementation lands.

AI-assisted-by: muse-spark-1.3-contributor
2026-10-19 15:22:10 +00:00
Andrey Antukh
688c0fc398
🐛 Keep file_change revisions on their own deletion schedule (#12419)
* 🐛 Keep file_change revisions on their own deletion schedule

Restoring a file from the trash cleared the deletion mark on every
file_change row of that file, revisions included. A revision is born
already scheduled for deletion one hour later, and objects-gc only
looks at rows that carry a date, so unmarking one left it in the
table for ever: production had 358,517 of them across 3,614 files.
The file deletion cascade had the mirror problem: it rewrote the date
of rows that already had one, pushing a revision out to the trash
lifetime.

Restrict the restore to durable history and date only the rows that
have no date yet, so a revision keeps the schedule it was born with.

A pinning test covers the snapshot update path, which cannot reach a
revision because it resolves its row through a read filter. The
backend memories now carry the model of the three file_data classes
and this invariant, in place of the two superseded stubs.

Closes #12418

AI-assisted-by: deepseek-v4.1-flash

* 📚 Add autovacuum and Postgres exporter notes to Grafana memory

Record what this Grafana instance can and cannot show about autovacuum
so the next session does not have to re-derive it:

- The Postgres exporter scrapes the primary as well as the
  read-only replica, so the primary's pg_stat_activity and
  pg_settings_* are in Prometheus even though the SQL
  datasource cannot reach them.
- It publishes no pg_stat_progress_* collector, and
  pg_stat_activity_count has no backend_type label: autovacuum
  workers are the usename/application_name-less bucket with
  state=active. The launcher shows up as the AutoVacuumMain
  wait event, and it is absent on the replica.
- A metric the exporter does not publish needs a custom query
  block in its YAML queries file and a restart of the exporter.
- The Postgres server ships its own log to Loki (the job is
  postgresql, primary only), whose streams carry only filename,
  instance, job and level. The completed vacuum lines live
  there: a per-table history, not a view of what runs now.
- The PostgreSQL section now ends with the vacuum panels,
  with the Vacuum TXID age table last; an in-flight per-table
  view would need a pg_stat_progress_vacuum collector in the
  exporter itself.

AI-assisted-by: deepseek-v4.1-flash
2026-10-19 14:00:13 +02:00
Andrey Antukh
84d92cfd5a
✨ Add tail buckets and update-file validation timings (#12409)
* ✨ Extend backend histogram buckets beyond 7.5s

Give every backend timing histogram a tail past the shared 7500ms cap
so slow RPC calls, CLIMIT waits and background tasks report real
quantiles instead of clipping at the top bucket.

Request and wait metrics grow to 60s, background tasks to 300s and
Redis eval to 10s; the sub-7.5s buckets stay untouched to keep the
history comparable.

Closes #12408

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Fix flaky assertion in climit republish test

Read the last published waiting value instead of the exact
publication list. The sampler thread started with the component
runs its first tick asynchronously, and when it lands inside the
recording window it publishes the same count twice, which made
the full-suite CI run fail while focused runs stayed green.

AI-assisted-by: muse-spark-1.3-contributor

* ✨ Add file_data blob to JSON decode helper

Allow decoding file_data blobs (app.util.blob) to JSON
for investigations with jq, offline from a blob file or
from nREPL with the backend running.

AI-assisted-by: muse-spark-1.3-contributor

* ✨ Add validation step timings to update-file logs and metrics

Measure the six phases inside process-changes-and-validate
(resolve-libs, process-changes, soft and hard affected and
schema validations) with tpoint and try/finally, so timings
are recorded even when a hard validation throws.

Publish each phase to a new histogram
penpot_rpc_update_file_validation_timing with a fixed step
label using the request buckets (up to 60000 ms), and extend
the update-file debug log with the same *-ms fields plus
file, project and team ids. The deferred log only runs on
success, so failures get an inline error log with partial
timings.

AI-assisted-by: muse-spark-1.3-contributor

* 💄 Translate decode-file-data comments to English

Translate the Spanish docstrings, section headers and test
messages of the file_data decode helper to English, keeping
code untouched. The repo convention is English in code and
comments; the helper was first written in Spanish.

AI-assisted-by: muse-spark-1.3-contributor
2026-10-19 13:01:35 +02:00
Andrey Antukh
b633093336 Merge remote-tracking branch 'origin/staging' into develop 2026-10-17 16:28:59 +02:00
Andrey Antukh
70b317a976
🐛 Harden upload path handling and validation error bodies (#12067)
* 🚑 Reject forged upload paths outside multipart

Transit bodies can smuggle upload maps whose path points at any
server file; the schema only checks the shape and handlers open
the path directly.

app.rpc now injects a server-side multipart flag that bodies cannot
forge, and methods declare their upload keys with the new
multipart-only-params metadata, enforced centrally after schema
validation with a params-validation error. Covers all nine upload
methods, including chunked and management flows.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Sanitize raw explain in generic validation error bodies

The generic :validation branch returned ex-data untouched, so errors
carrying the raw malli explain (live schema objects, e.g. storage
invalid-storage-metadata) broke JSON encoding mid-stream and clients
got truncated bodies. Apply the same clean treatment the known codes
already use: drop the raw explain and attach its humanized text.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Add e2e path traversal tests for binfile and media upload

Add two e2e suites that hand-write transit bodies with a forged
["~#path", ...] upload path (no transit library needed) and expect
400 validation/params-validation. They cover import-binfile :file,
upload-file-media-object :content and upload-chunk :content, plus a
plain-JSON string-path control case. Before the multipart-path fix
they failed (500 io-exception, invalid-image, 200 stored chunk);
after it they pass.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Use multipart helper in binfile SVG test setup

The SVG import setup uploaded through plain command!, so the new
multipart-only guard rejected it with params-validation. Switch
the setup to multipart-command! like the other upload tests.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Use multipart helper in remaining upload tests

The multipart-only upload guard rejects plain RPC calls carrying
upload maps. Five tests with real files still used command! and
failed with params-validation. Switch them to multipart-command!
while keeping the forged-content tests on plain command!.

AI-assisted-by: muse-spark-1.3-contributor
2026-10-17 16:01:04 +02:00
Andrey Antukh
dfc55a7192 Merge remote-tracking branch 'origin/staging' into develop 2026-10-16 12:48:49 +02:00
Andrey Antukh
57fa280209
🐛 Validate and decode a job result with its declared schema (#12350)
* 🐛 Decode job results with the schema of the job-def

A job result is stored as plain JSON, so a uuid comes back as the text
it is on disk and a uuid used as a map key comes back as a keyword. The
import resolution is a map keyed by file id, and the wizard lost that
identity: choosing a library in the link step created no link at all,
with no error and with the modal closing as if it had worked.

A job-def can now declare a result schema and its precompiled decoder,
the pair the params already have, and the substrate applies them when the
row is read. A def that declares neither answers with the JSON the row
holds, as before; a row whose job-def is gone is read untouched instead
of raising.

Relates to #12339

AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>

* 🐛 Type the import result so its resolution keeps uuid keys

The resolution of a version 3 import is a map keyed by the new file id,
and the result column is plain JSON: without a schema of its own every
key came back as a keyword while the :id of the entry came back as text,
so the dashboard wizard could not match the file it was linking and
created no library link at all.

Declare the shape of the resolution (files, done and pending entries and
their candidates) and give the job-def its result schema and decoder.
The check before the result is stored keeps validating the same schema.

Closes #12339

AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>

* 🐛 Take the imported file id from the entry, not the map key

The wizard looked a file up in the selection with the key of the
resolution map while it had indexed the selection by the :id of the
entry. The key is keywordized on its way through the job result, so the
lookup missed and the confirm button created no library link at all,
with no error and with the modal closing as if it had worked.

Take the id from the entry, the source the rest of the wizard already
uses, and move the decision to a pure function covered by tests of both
shapes of the resolution.

Relates to #12339

AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>

* 🐛 Test that an import result keeps its ids as uuids

The regression this branch fixes is only visible through the read path
of the substrate: the resolution of an import is a map keyed by the new
file id, and a test that works with the value in memory never sees the
keys coming back as keywords from the plain JSON column.

Import a package whose library needs a replacement chosen by the user,
read the result with `jobs/get-job` as the client does, and check the
keys, the file ids and the ids of the candidates. Without the result
decoder of the import job-def the test answers with text ids.

Relates to #12339

AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>

* 📚 Document the result schema of a job-def

The memories and the docstring of the substrate said a job result has no
schema of its own and comes back as the JSON the row holds. A job-def can
now declare its result schema and decoder, and the substrate applies them
when the row is read.

Keep the rule next to it: a result map keyed by uuid needs its schema
declared, or every key comes back as a keyword, and the link-later step
of the import wizard takes the file id from the entry and never from the
key of the resolution map.

Relates to #12339

AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>

* 📚 Fix the documented contract of a job result

The docstring of get-job said a result has no schema of its own and comes
back as the JSON the row holds, which is what this branch changes: the
result is decoded with the schema its job-def declares, and a result that
does not conform never reaches the column because the write validates it.

Say the same in the management command that stores the result of an
external worker, the other writer of the column.

Relates to #12339

AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>

* 🐛 Check the result of a job against its schema before storing it

The result of a job crosses a plain JSON column, so a uuid comes back as
text and a uuid used as a map key as a keyword. The job-def now declares
the schema of its result and the substrate compiles the decoder from it,
which is what the read path uses to get the types back.

The same schema is the contract of the write path: complete refuses a
result that does not conform, with the malli explanation, so nothing
invalid reaches the column, and the answer is the same for the runner and
for the external worker that completes a job through the management RPC.
A job that is no longer completable is skipped without checking, and the
read path decodes only: a row written by another version stays readable.

Relates to #12339

AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>

* 🐛 Check the result of an import inside its transaction

The check ran after the transaction of the import had committed, so a
result the schema does not accept failed the job with the files already
created in the destination project: a half imported project and an error.
The core decides whether the whole import is kept or rolled back, so the
check belongs inside its transaction, which is what turns a contract
violation into a rollback.

The substrate checks the result again when the job is completed, which is
the only net for a job that does not validate its own result.

Relates to #12339

AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>

* 🐛 Cover the edges of an import result read by the client

The shape of a result with no libraries to resolve, and the one of a
version 1 import, is the most common in production and neither was covered
by the tests that read a result the way the client does.

Cover both through jobs/get-job, and the case of a registry that no longer
knows the kind of the row: it answers with the JSON of the column,
keywordized keys and uuid as text, instead of raising. Say in the two tests
that depend on the decoder what implementation without it makes them fail.

Relates to #12339

AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>

* 📚 Fix the prose and the fixtures of the result contract

The result schema of a job-def is what the read decodes with and what the
write checks: say that in the memories, with the rule that any uuid of a
result needs it, whether a value or a map key.

The motive of the wizard helper was a fact this branch makes false: what a
key of the resolution map holds depends on the result schema of the job-def
that wrote it, so the rule is to read the id from the entry. Same
correction in its memory and in the test that pins the shape.

The candidates of a pending entry match by slugified name, not by name, and
every test row now states the kind of its job, because the name is what
resolves its job-def when the result is read or checked.

Relates to #12339

AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>

* ♻️ Keep the wizard link helper private and cover its shape

The helper that decides which links the wizard creates belongs to its
namespace, as its neighbours do: reach it from the test with @#', the
pattern the frontend tests already use for a private helper.

Cover the shapes it had no test for: a file with two libraries pending
resolution, a file whose libraries were auto-linked, and a selection where
only one of two entries was resolved.

Relates to #12339

AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>

* ♻️ Share one job-row helper across the job tests

Three test namespaces had their own copy of the same insert, differing only
in the kind of the job and in whether the row holds the resource it
consumes. Move it to the test helpers as create-job*, with the kind of the
job as its first argument: it is what resolves the job-def of the row when
its result is read or checked, so every test states it.

The bang of the name goes away with it: the design rules keep it for the
mutating API of app.db, and this is a fixture of the tests. The three
namespaces lose the requires that only their removed copy used.

Relates to #12339

AI-assisted-by: deepseek-v4.1-flash
Signed-off-by: Andrey Antukh <niwi@niwi.nz>

---------

Signed-off-by: Andrey Antukh <niwi@niwi.nz>
2026-10-15 19:18:04 +02:00
Andrés Moya
feb73f0fd8
✨ Handle tokens libraries when exporting .penpot files (#12077)
* ✨ Handle tokens libraries when exporting .penpot files

Exporting a file that takes its tokens from an external library left
the :tokens-source of the file pointing at a library that is not in the
export, and the tokens lib of that library was not exported.

With :detach-libraries, unset the :tokens-source and the tokens status
in detach-external-references. The file is exported without any link to
the library, and the shapes keep their applied tokens.

With :merge-libraries, add cfo/absorb-tokens-lib, which copies the tokens
lib of a library into a file and unsets its :tokens-source, so the file
becomes its own tokens source. The tokens status of the file is kept; if
it has none, the one of the library is copied. If the library has no
tokens lib, only the source is unset. embed-assets calls it for the
library that is the tokens source of the file, and ignores the other
linked libraries.

Add unit tests for absorb-tokens-lib, and binfile tests that export a
file with tokens from an external library with both modes, checking the
exported files, the tokens lib, the tokens source and the applied
tokens.

Also reformat the export-type docstring in binfile v3 and remove a
leftover debug prn in the export dialog.

AI-assisted-by: claude-sonnet-5-5

* 🐛 Preserve active token sets when importing linked libraries

Cache each decoded tokens library for the duration of an import so files
and their tokens source resolve active sets against the same generated IDs.
Add a round-trip regression test for exports that include linked libraries.

* 🐛 Preserve tokens when deleting or unpublishing libraries

Absorb the tokens library into files that use it as their tokens source.
Keep referencing file IDs in the deletion task before removing relations.

Embed tokens from deleted, unlinked sources during export, and clear stale
source and status data when the library no longer exists. Add regression
tests for deletion, unpublishing, and exports with unavailable libraries.

---------

Co-authored-by: alonso.torres <alonso.torres@kaleidos.net>
2026-10-15 16:53:30 +02:00
Andrey Antukh
a53481c3b7
✨ Enrich backend audit events with team and entity ids (#12233)
* ✨ Enrich backend audit events with team and entity ids

Backend audit events now carry the data a consumer needs to rebuild
state without joining production tables:

- prepare-rpc-event adds :team-id when the event carries a project or
  file id; commands that already loaded the team set it themselves.
- add-profile-to-team! emits add-team-member on every real insert.
- create-project and create-file emit the new row id.
- clone-template, import-binfile and duplicate-project emit one event
  per created entity.
- both binfile versions return {:file-ids [...]} from import-files!.

Closes #12219

AI-assisted-by: deepseek-v4.1-flash

* ✨ Emit delete and hard-delete audit events with entity data

The event stream must let Nexus rebuild local state without joining
production tables. This completes the enrichment with the delete paths:

- delete-team keeps returning nil but now carries the deleted row
  (id, name, timestamps) in the audit props, like delete-project.
- create-team adds created-at to its audit props.
- delete-profile carries id, email, fullname and deleted-at; its
  props used to be empty because every param is namespaced.
- objects-gc emits hard-delete-team/project/file per purged batch,
  with the deleted ids, only when the audit log is on.
- delete-team-member needs no change: the delete is physical and the
  params already carry team-id and member-id.

The nitrate push-audit-events tests now drop the membership events
emitted by their own profile setup before counting.

Closes #12219

AI-assisted-by: muse-spark-1.3-free

* ✨ Announce cascade deletes and missing ids in audit events

The :delete-object worker task marked teams, projects and files without
leaving any event, so an event-sourced projection kept them alive for
weeks. It now emits one cascade-deleted event per execution with every
affected id, in the same transaction, only with :audit-log on. Rows
purged by objects-gc and snapshot cascades stay silent on purpose.

Also: create-team-with-invitations carries the new team id, and
update-team-member-role names the demoted member on owner transfers.

Closes #12219

AI-assisted-by: muse-spark-1.3-free

* 🐛 Identify the SREPL profile delete in audit events

The SREPL delete-profile! emitted no profile-id and empty props, so the
row could not be attributed by any consumer. It now carries the profile
id in both fields. delete-team! named the wrong cause in its context.

Closes #12219

AI-assisted-by: muse-spark-1.3-free

* 🐛 Identify the SREPL profile restore in audit events

The SREPL restore-profile! emitted no profile-id, so the consumer
keyed on the zero profile instead of the restored one. It now carries
the profile id like a normal profile event, and no consumer special
case is needed.

Closes #12219

AI-assisted-by: muse-spark-1.3-free

* 🐛 Stop attributing SREPL events to the subject profile

profile-id names the actor, and SREPL calls have none: setting the
subject id blamed the user for an admin action and would have leaked
those rows into PostHog as user activity. The subject id stays in
props, where the consumer resolves it, and the event profile stays
zero.

Closes #12219

AI-assisted-by: muse-spark-1.3-free

* ✨ Expose file analysis data for the nexus pipeline

The nexus file-security pipeline needs file content and scheduling
state from the backend. This adds a read-only management method,
`get-file-for-analysis`, returning the semantic projection of one
file (pages, frames, texts, links, interactions; ids as strings;
capped with a truncated flag), plus identity, revn, timestamps and
share state. No prior projection code existed, so the method ships
its own pure `file->projection`.

Audit events now carry what the `:local` consumer schedules from:
revn and modified-at on update-file, name and timestamps on every
create-file emitter, deleted-at on delete-file, and new
create/delete-share-link events with file and link ids.

AI-assisted-by: muse-spark-1.3-contributor

* ♻️ Rename deletion audit events to action names

Event names are actions, not past tense: `cascade-deleted` becomes
`delete-object-with-cascade` with the same shape and transaction.
The three `hard-delete-*` events merge into `hard-delete-objects`:
one event per purged chunk carrying only that chunk's entity key
(`:team-ids`, `:project-ids` or `:file-ids`), because each GC
section purges in its own chunk transactions and a single event
could never share them all. Every submit stays inside its delete
transaction, `triggered-by` stays annotated, and silent paths stay
silent.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Read worker parallelism knobs with plain config keys

The runner wiring looked up `:app.main/worker-*-parallelism`
(namespaced) while the schema and the environment provide plain
keys, so `PENPOT_WORKER_DEFAULT_PARALLELISM` and
`PENPOT_WORKER_WEBHOOK_PARALLELISM` never applied and workers
always ran with the fallback. Read the keys the config holds.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Fix binfile audit test with a real file row

`submit-file-created-event` reads the light columns back from the
row, so the mocked importer must return an existing file id: a
phantom id emits no event and the test saw zero `create-file`
events. Create the file first and use its id. The audit-log memory
now tells both bulk-event helpers apart.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Fix audit event fidelity from review findings

The review of the audit enrichment found data-fidelity gaps; fix
them while the branch is fresh.

Membership events now belong to the actor: the flows where the
owner adds somebody else pass an explicit actor id, the role is
derived from the flags (the table has no role column), and the
event carries the request context like its siblings.

Also: team-id resolution skips read events, duplicate-project
emits the project name and timestamps, delete-profile is
idempotent again, the GC tests assert physical purge, and
get-file-for-analysis leaves an audit trace.

AI-assisted-by: muse-spark-1.3-contributor

* ♻️ Nest cascade ids under :cascade in delete event

The cascade event carried every affected id flat next to the
object identity; nest them under `:cascade` so the payload reads
as header plus affected set. The event is an action like its
siblings, not a command. Drop the `:audit-log` flag re-check:
`submit` already fans out per flag.

AI-assisted-by: muse-spark-1.3-contributor

* 📎 Update copyright name on file header

The company name changed; a single header in this branch still
carried the old one. Align it with the current form.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Fix nil event-id on webhook trigger audit rows

process-event generated the row id inside append-audit-entry, so the
:process-webhook-event task never received it and the trigger row
stored a nil :event-id. Assign the id up-front and share it between
the stored row and the worker params; the telemetry shadow row keeps
its own id to avoid a PK collision. Add regression tests covering
the handoff and the end-to-end trigger row.

AI-assisted-by: muse-spark-1.3-contributor
2026-10-15 13:01:18 +02:00
Andrey Antukh
0c0336da42 Merge remote-tracking branch 'origin/staging' into develop 2026-10-13 21:04:58 +02:00
Andrey Antukh
eafa483a18 Merge remote-tracking branch 'origin/main' into staging 2026-10-13 14:56:45 +02:00
Andrey Antukh
575b68d2c9 🐛 Unmark the storage objects that a restore brings back
Restoring a file only un-marked the database rows (`file`,
`file_media_object`, `file_change`, `file_data`, `file_thumbnail`,
`file_tagged_object_thumbnail`), not the storage objects those rows
reference. An object that was already marked for deletion stayed
marked while a live row pointed at it, and those foreign keys are `NO
ACTION`: the storage GC cannot delete it and skips it forever (and
before the GC learned to skip, it cost the whole nightly run).
Restoring a team had the same gap with its font variants.

Both restores now unmark the storage objects of the rows they bring
back: the media and the thumbnail of the file media objects, the media
of the file and object thumbnails, the `storage-ref-id` of the file
data, and the four files of a font variant. Objects referenced by other
rows are left alone.

AI-assisted-by: deepseek-v4.1-flash
2026-10-13 14:33:32 +02:00
Andrey Antukh
18705f2d00 🐛 Make the audit events concurrency limit per profile
The command declared its limit as a plain keyword next to
`::climit/key-fn`, metadata that `climit/wrap` does not read: it
takes the key only from the positional `key-fn` of the vector form.
The limit was a single shared one, so when one call got stuck it
held the only permit and every audit batch of that instance was
rejected with 429 from then on.

The test builds the limiter from the command metadata and checks
that two profiles end up with different limiters.

AI-assisted-by: deepseek-v4.1-flash
2026-10-13 14:33:32 +02:00
Filip Sajdak
10955f17e2
✨ Add get-file-tokens RPC to read a file's design tokens (#12172)
Clients that only need a file's design tokens, such as a CI job that
syncs them to a git repository after an update-file webhook, had to
download the whole file with get-file.

get-file-tokens returns the file's tokens as a DTCG document, the same
format the token export produces, or nil when the file has none. Active
themes and sets come from the file's tokens status, so they reflect
what is currently active.

The document is returned as a plain map. A bare TokensLib is not a
collection, so the response middleware would hand it to ring unencoded.

Relates to #7091

Signed-off-by: Filip Sajdak <filip.sajdak@siili.com>
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
Co-authored-by: Andrey Antukh <niwi@niwi.nz>
2026-10-09 20:08:14 +02:00
Filip Sajdak
d7ccc0f3a6
✨ Add get-file-plugin-data RPC to read file plugin data (#10171)
The Plugin API can write arbitrary key/value plugin data onto the file
root, but it could only be read back by downloading and decoding the
whole file. Add a `get-file-plugin-data` read command that returns the
file-level plugin data map, optionally narrowed to a single namespace,
so external tooling can fetch metadata such as issue or commit
references cheaply.

Read-only: reuses `bfc/get-file` and `check-read-permissions!`, and
adds no data-model or change-type changes. Object/page level reads are
intentionally left out of this first command.

Relates to #10169

Signed-off-by: Filip Sajdak <filip.sajdak@siili.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Andrey Antukh <niwi@niwi.nz>
2026-10-09 18:15:07 +02:00
Pablo Alba
76e9c132ef
✨ Send more enriched events to nitrate log (#12210)
* ✨ Send more enriched events to nitrate log

* ✨ Fix mr
2026-10-09 10:58:05 +02:00
Andrey Antukh
9abcfebf60
✨ Run binfile import and export as durable jobs (#12056)
* ✨ Run binfile import and export as durable jobs

Import and export of `.penpot` files become durable jobs: two commands
create them and answer at once with a job id the caller follows, while
a worker of the `binfile` queue does the work. Each job keeps its own
ledger (owner, params frozen at creation, result, progress history and
an expiry), and the legacy commands and their SSE keep working.

Why: both operations ran inside the request. A large import held it
open for minutes, progress only existed while the caller kept the
stream open, and a client that went away left work nobody recorded,
retried or could cancel. The unified `job` substrate already dispatched,
leased, retried and logged events, but it had no user-facing ledger.

How:
- The substrate gains the pieces a user-facing job needs: optional
  job-def metadata (family, resource-role), an expiry on submit, the
  owner in the runner context, and job resources in their own bucket,
  written with the profile of the job and never deduplicated.
- One generic command per family: the caller names the job and sends its
  business params, the registry checks the family, the job-def decodes
  and validates, and permission and quota are checked first.
- Progress taps of the core become milestones of the job: a shared stage
  vocabulary with self-contained counters, throttled heartbeats, and an
  interrupt that stops the run inside its own transaction on cancel.
- Cancel reaches pending and running jobs from both modals, and closing
  a modal mid-run cancels the job it was following.
- The export re-checks the read permission at run time and answers with
  the package descriptor; the import re-checks the edition permission,
  runs the core in a transaction so a cancellation rolls it back, and
  releases the consumed package and its temporary copy.

AI-assisted-by: deepseek-v4.1-flash

* ♻️ Handle externally-cancelled binfile jobs end to end

The `cancelled` outcome of a job nobody followed no longer hangs its
screens: the data layers emit an explicit terminal message for it.

- The export emits `{:cancelled true}` per file and the modal paints a
  neutral cancelled row instead of success; a message without an
  artifact never reaches the download branch.
- The import closes every entry with a dedicated cancelled message
  through the existing error rendering, so the wizard completes.
- Both flows are pinned by data-layer tests, including the red state
  they replace.

Also in this change: the progress-contract comment of the management
API, early business-params validation in create-import-job (before
assembling the upload), a `release-resource` helper without fabricated
job ids, a note on the heartbeat write cost, and a memory cleanup.

AI-assisted-by: muse-spark-1.3-contributor

* ♻️ Pre-translate the cancelled import message

The cancelled entry of the import wizard carried its message as a key
for the UI to translate, which needed a rehash marker comment. Like
its neighboring errors, it now translates eagerly in the data layer:
the literal call keeps rehash clean with no marker, and the tests
assert through the same call instead of the raw key.

AI-assisted-by: muse-spark-1.3-contributor

* ♻️ Move cancelled strings under generic labels keys

The dashboard-specific cancelled key never shipped: before use it moves
under `labels.*` next to the generic strings both flows already use,
with Spanish translations following the established wording.

AI-assisted-by: muse-spark-1.3-contributor

* ✨ Add minor improvements

* ♻️ Rename export job command to create-export-binfile-job

Export now has one command per job type, the shape the :export-assets
command needs: create-export-binfile-job freezes exactly the
:export-binfile job, so :name no longer travels in the body and only
:params does. Import keeps the generic envelope.

submit-job reads the queue from the job-def metadata ::jobs/queue-name
(falling back to :binfile), read by the command and never by the
substrate, so a job-def can route its work to another queue without a
change in the command.

The rename reaches the frontend and the tests: the binfile export is
asked for under its own name, and the exporter service keeps
create-export-job.

AI-assisted-by: deepseek-v4.1-flash

* ✨ Polish the import and export status rows

The status row of a file already carries what the file is doing, so the
generic "Uploading file…"/"Downloading file…" lines are gone: the
import wizard drops its own (the export one went with the jobs
refactor), along with the now-unused style.

In the export, every file is queued up front, even the ones whose job
is created only after the previous one is over, and a file stays queued
until its first milestone: a worker picking the job up no longer ends
the queue. The queued text is a step smaller than the milestone.

AI-assisted-by: deepseek-v4.1-flash

* 🐛 Show the cancelled export and fix the import error

The cancelled export was unreachable: mark-file-cancelled sets :loading
false and the label lived inside the loading branch, so a cancelled
file only painted a neutral notification. The terminal state now
carries the label, and the export flags lose the `?` suffix
(predicates keep it; a data prop is not a question).

The import wizard rendered (:error entry) through tr, but every
producer writes user-facing text (a hint or a translated label), never
a key: it renders the text as-is now, and the dynamic-key ignores and
the stale comment are gone.

The total-text-limit comment no longer references the 200MiB it
replaced; the value is 500MiB in develop already.

AI-assisted-by: deepseek-v4.1-flash

* ♻️ Reflect the real job params in the creation commands

The creation commands declared their params as an open empty map, so the
RPC contract and the generated docs said nothing about them. Each command
now imports the schema from the job namespace
(`export-binfile/schema:params`, `import-binfile/schema:create-params`),
and the RPC validates the business params before the command runs. The
job-def still owns the params and re-validates the assembled ones at run
time. The commands no longer decode the params again: the RPC did it.

Import joins export as one command per job type: `create-import-job`
becomes `create-import-binfile-job`, its body no longer carries `:name`
and the job-def is resolved by name like export's. Its creation schema
keeps the version optional, because the command reads it from the package
header when the caller does not say it, and fills the manifest metadata
kept for audit.

A version out of range is now refused by the schema (`:params-validation`)
instead of the command's own `:unsupported-version` check, which is gone.

AI-assisted-by: deepseek-v4.1-flash

* 🐛 Stop the test system from running job runners

The test system is main/system-config plus main/worker-config with a
few components dissoc'ed. The runners of default, webhook and cron were
listed by hand; the binfile runner added with the durable binfile jobs
was not, so a real penpot/job-runner/binfile/0 thread ran during the
whole suite.

That runner races the tests that drive the loop by hand (run-batch plus
run-worker-loop): it takes the job the test just dispatched and
completes it, and the test then reads a state it did not produce. That
is the reported flake of binfile-jobs-integration-test, whose middle
assertion waits for a runner of another queue to not take the job. It
only reproduced in CI: in the devenv PENPOT_TENANT makes the tenant
baked into main/worker-config differ from the one the test builds, so
the two used different queue keys locally.

Drop every [<profile> :app.worker/runner] by key shape instead of
listing the known queues, so a new queue cannot bring the race back.

AI-assisted-by: deepseek-v4.1-flash

* ✨ Upload the package one chunk at a time and as a percentage

The upload milestone counts chunks, not units of work, so the row wrote
"Uploading the package 3/9" over a file the user had uploaded once: it
read as if nine packages were on their way. Write the stage as a
percentage of the file instead ("Uploading file… (33%)"), and make the
English text of the stage say file, not package.

The chunks also go one at a time now (concat-all instead of merge-all
with two in flight), so the percentage moves in a straight line instead
of jumping. upload-blob-chunked is the generic uploader, so media and
font uploads become sequential too.

AI-assisted-by: deepseek-v4.1-flash

* 🌐 Complete the Spanish catalogue

es.po was 25 keys short of en.po: the 15 that the durable binfile jobs
introduced (14 jobs.progress.stage.* and labels.queued) plus 10 gaps
that predate them (errors.connection-error, errors.save-retrying, the
find shortcuts, workspace.header.retrying,
workspace.tokens.stroke-width and the mcp session labels).

sync -l es then sorts the entries, syncs the #: comments from en and
drops the 7 dashboard.import.progress.* keys en no longer has. The
msgid sets of en and es are now identical (0 missing, 0 extra).

AI-assisted-by: deepseek-v4.1-flash

* ♻️ Move the job status keys into the jobs namespace

The three status words the durable import and export rows render were
filed under labels.*, next to the generic labels bucket, while their
progress stages live under jobs.progress.stage.*: the strings of one
feature sat in two places.

Rename them so the whole job vocabulary is one namespace:

  labels.queued           -> jobs.queued
  labels.export-cancelled -> jobs.export-cancelled
  labels.import-cancelled -> jobs.import-cancelled

No other locale has these keys yet, so the rename costs no translation.
The convention (a key names the feature that renders it; labels.* holds
the generic reusable words) is written down in the frontend/translations
memory.

AI-assisted-by: deepseek-v4.1-flash

* 🔥 Remove the translation keys nothing references

Six keys of en.po (and of es.po, which now mirrors it) have no (tr ...)
call site left in frontend/src or common/src, and no literal reference
either:

  labels.uploading-file    a duplicate of jobs.progress.stage.upload
                           since the upload stage says "Uploading file…"
  labels.downloading-file  dead since the durable jobs replaced the old
                           import and export progress rows
  inspect.attributes.stroke.alignment.center
  inspect.attributes.stroke.alignment.inner
  inspect.attributes.stroke.alignment.outer
  workspace.toolbar.mcp-connect-here

The first two are leftovers of the durable binfile jobs, the other four
predate them. Both catalogues keep an identical msgid set afterwards.

AI-assisted-by: deepseek-v4.1-flash
2026-10-07 11:52:48 +02:00
Andrey Antukh
746184caf9 Merge remote-tracking branch 'origin/staging' into develop 2026-10-06 15:58:01 +02:00
Andrey Antukh
51ae83202e Merge remote-tracking branch 'origin/main' into staging 2026-10-06 15:56:55 +02:00
Andrey Antukh
d3ca9a0b83
✨ Add unified jobs substrate replacing legacy one-shot task system (#11542)
* 🎉 Move backend jobs to the unified job table substrate

Replace the generic 'task' table with a unified 'job' table as the
durable substrate for all backend jobs, and migrate every producer
and consumer to it. The legacy 'task' table stops receiving writes
but stays dormant (still cleaned by tasks-gc); upload-session
chunking and the other post-branch upstream changes are preserved.

New 0154 migration creates the 'job' table: dispatch and lifecycle
columns (name, queue, priority, scheduled_at, retries, status,
timestamps, props) plus optional user-facing ledger columns
(profile_id, target, progress, error, result, resource_id,
expires_at). No modified_at trigger: the application updates it on
every write for lease and orphan detection. Partial indexes cover
the dispatcher claim query, orphan scanning, and per-profile
listing.

New app.jobs public API: submit!/invoke!, heartbeats and progress
reporting, a precompiled job-def registry (name, schema, handler,
decoder, validator), an ephemeral request/response mode, and a
management API for external workers. The dispatcher claims
new/retry rows (FOR UPDATE SKIP LOCKED) with a plain JSON redis
payload and marks lease-expired running rows as orphaned; the
runner executes per-queue with the same retry conventions.

New jobs GC (daily): deletes expired rows and retains internal
terminal rows per :jobs-retention, touching referenced storage
objects so storage-gc-touched reclaims them. Job resources live in
a dedicated job-resource bucket with a reclaim branch in
storage-gc-touched.

All production call-sites (sendmail, delete-object, demo-purge,
file-gc, offload, webhooks, quotes, nitrate bulk delete,
snapshots, telemetry) and the ten cron tasks now submit through
app.jobs; cron is a pure scheduler and the legacy worker registry,
task-table inserts, and per-namespace handlers are removed.

Includes migrations, dispatcher, runner, jobs, request/response,
jobs GC, management API, and cron test suites.

AI-assisted-by: glm-5.3-flash
AI-assisted-by: muse-spark-1.3-contributor
AI-assisted-by: kimi-k3
AI-assisted-by: mimo-v2.5

* ✨ Address review comments on jobs substrate

Resolves the four open review threads on the jobs PR. Orphaned jobs
are now marked aborted, a system-side terminal state with no retry:
the dispatcher sweep stores the end event with outcome aborted in a
single bulk insert in the same transaction, records the terminal
outcome next to the orphaned counter, and reports an error log;
orphans keep alerting through the log and the counter, with no
user notification.

The aborted state is terminal everywhere: the status CHECK, the
sweep, the lifecycle writers, the jobs GC retention, the backlog
gauges and the user status mapping, with tests covering each
transition. The claim comment, the 2.20 doc version and the legacy
task table notes are corrected, and objects-gc heartbeats per chunk.

AI-assisted-by: muse-spark-1.3-contributor
2026-10-06 07:33:12 +02:00
Andrey Antukh
ab58922f2c
🚑 Revoke file access when a member leaves the team (#12097)
Delete the team-scoped file, project and pin relations on leave-team
and delete-team-member, require a live team membership in the file,
project, search and listing permission queries, and clean
pre-existing orphan rows with a migration. Former members now get
not-found on file actions instead of keeping read and rename access
(GHSA-v9r9-h77c-55m2).

AI-assisted-by: muse-spark-1.3-contributor
2026-10-05 16:47:07 +02:00
Andrey Antukh
341e5d1027 Merge remote-tracking branch 'origin/staging' into develop 2026-10-05 11:59:05 +02:00
Elena Torró
aed4e12c63
⚡ Avoid decoding all pages when building file thumbnail data (#12054) 2026-10-05 11:34:10 +02:00
Andrey Antukh
ea3d18c0a5
✨ Audit-log docs, error-reporting flag and frontend initiator (#11997)
* ✨ Gate error reporters behind :error-reporting flag

Error reporters used to start on every boot with no way to
switch them off. They now only start when the new
:error-reporting flag is on (off by default, enable with
PENPOT_FLAGS=enable-error-reporting). The database reporter
needs the flag alone; mattermost needs the flag plus its
webhook url. Shutdown with the flag off is safe, and the
database loop now honors its runtime switch.

AI-assisted-by: muse-spark-1.3-contributor

* 📚 Add audit-log reference memory and update dev tooling

Adds the top-level audit-log reference memory with all event
flows and turns backend/audit-log into a redirect stub.
Also includes the staged dev-tooling updates (playwright
deps, mdts args, gitignore).

AI-assisted-by: muse-spark-1.3-contributor

* ✨ Assign server-side initiator to frontend audit events

Frontend audit events now carry a server-assigned initiator
derived from the x-client header (penpot-frontend goes to
app, penpot-admin-console goes to admin-console, legacy
penpot-nitrate stays on admin-console, all else goes to
app). Client-sent values are always overwritten, and the
key survives on telemetry shadow rows.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Keep initiator on frontend telemetry context

The strips-pii-keys test still asserted the old contract
(initiator stripped). The initiator only names the sending
app, so it is not personal data and must survive the
telemetry filter like on the backend channel.

AI-assisted-by: muse-spark-1.3-contributor

* 📚 Document initiator on PostHog and browser origins

Complete the audit-log memory for the latest changes: the
initiator property forwarded to PostHog on both channels
and the eventOrigin rule for admin-console browser events.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Fix unscaped argument in script

---------

Co-authored-by: alonso.torres <alonso.torres@kaleidos.net>
2026-10-02 11:08:13 +02:00
Andrey Antukh
b5fbc4fd8c
✨ Add size limits to profile props and plugin registry (#11596)
* ✨ Add size limits to profile props and plugin registry

Bound the total serialized size of profile settings to 2 MiB
(:profile-props-max-size), checked on the merged result before
persisting, with a controlled :props-too-large error. Profiles
that already exceed the limit can still shrink but cannot grow.

Cap plugin registry entries in the shared schema (code 1 MiB, 50
plugins max, bounded name/host/description/icon) and restore rate
limiting on the plugin RPCs (profile-mutations bucket, one write
at a time per profile). The plugin manager now asks for
confirmation before removal and ignores repeated clicks while a
persist request is in flight.

Closes #11592

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Enforce plugin count cap, byte sizes and removal guard

Enforce the declared 50-plugin cap in add-profile-plugin with a
specific :too-many-plugins error (updates of existing entries
still pass); the cap lives in a shared max-plugins constant.

Measure profile props size in UTF-8 bytes instead of chars so
multibyte content cannot slip past the limit.

Cover install/remove persist logic with mocked-RPC frontend tests
(release semantics, in-flight dedupe, validation vs rollback
split) and add the missing boundary tests in common.

Expose the in-flight persist set from the plugin registry and
disable the remove button of entries being saved.

Closes #11592

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Fix rollback loops and restore paths in plugin registry

Restore the previous plugin version instead of dropping the entry
when a validation error rejects an update of an installed plugin.

Make compensating writes one-shot with terminal callbacks so a
persistent failure cannot ping-pong between install and remove.
Restores keep the original list position; the unused public
plugin-persisting? predicate is removed.

Pin count-before-size precedence with a dedicated test and fix
translation source refs to their canonical lines.

Closes #11592

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Guard notifications write and fix restore ordering

Route update-profile-notifications through check-props-size! so
oversized profiles cannot grow through that path; document the
exempt system writers. Remove the duplicated stale entries in
en.po, keeping the canonical translation refs.

Restore rejected plugin updates at their original list position
instead of leaving the optimistic move in place.

Closes #11592

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Skip no-op plugin removal and clarify size comments

Return early from remove-profile-plugin when the id is absent:
no wasted write, no size check, and no manufactured :plugins key
that could spuriously fail on oversized profiles.

Clarify that per-field string caps count chars while the byte
budget is enforced by profile-props-max-size.

Closes #11592

AI-assisted-by: muse-spark-1.3-contributor

* 📎 Fix formatting in rlimit.edn for profile operations

Signed-off-by: Andrey Antukh <niwi@niwi.nz>

* 📎 Fix formatting of import-binfile/global entry

Signed-off-by: Andrey Antukh <niwi@niwi.nz>

* ♻️ Simplify props size check and tighten plugin entry caps

Measure props with transit bytes directly instead of the
PGobject string roundtrip.

Rename check-props-size! to check-props-size: single hard limit
on the merged props, no growth comparison, and return props so
writers thread the check into the update.

Move the 2 MiB default into default-props-max-size on the
profile namespace, still overridable with the optional
:profile-props-max-size config entry.

Tighten registry-entry :code and :icon to 500 chars: they hold
manifest paths, not content.

Closes #11592

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Fix compatibility problems

---------

Signed-off-by: Andrey Antukh <niwi@niwi.nz>
Co-authored-by: alonso.torres <alonso.torres@kaleidos.net>
2026-10-01 14:31:30 +02:00
Andrey Antukh
94d6f5a25c Merge remote-tracking branch 'origin/main' into staging 2026-09-29 23:07:26 +02:00
Andrey Antukh
3b886995ba 🐛 Emit the accept-organization-invitation audit event once
The event was written twice per accepted organization invitation. The
backend submitted it, and the browser then re-submitted a copy of the
props that the backend had already put in the response under
`:organization-invitation-audit` (`handle-token :team-invitation` in
`verify-token.cljs`). Both rows carried the same name with different prop
vocabularies, and the browser copy only existed when the browser finished
the flow.

Emit the event from the backend only. It now also carries the three props
that lived in the browser copy: the organization member count before the
add, the add source, and whether the invitee also joined a team. The
origin moves to the event context as `:event-origin`. The response no
longer includes `:organization-invitation-audit`, so the browser stops
emitting the event and `verify-token.cljs` drops its
`app.main.data.event` require.

The `accept-*` events of this command now share one prop vocabulary:
`:profile-id` for the accepting profile, `:invited-by` for the inviter
and `:profile-email` for the email, replacing the mix of
`:user-id`/`:user-who-send-invitation` and `:email`.

Audit consumers of `accept-organization-invitation` now see one row per
acceptance instead of two, and must read the new prop names.

AI-assisted-by: space-bunny-free
2026-09-29 22:40:03 +02:00
Andrey Antukh
dc0ea3a69c
🐛 Attribute audit events to the caller, not the response (#11952)
* 🐛 Attribute audit events to the caller, not the response

prepare-rpc-event took the event profile-id from the result map
whenever it carried one, before falling back to the caller. Any
command returning a response with a :profile-id key silently
credited the action to somebody else.

get-error-report returns the report with its decoded content
merged in, and that content holds the profile that owned the
report, so privileged reads were logged against the users whose
crashes were being inspected. verify-token on a team invitation
returns the inviter's profile-id, so accepting an invitation was
logged against the inviter.

Resolution is now ::audit/profile-id metadata, then
::rpc/profile-id, then the zero uuid; the response is never
consulted. The two verify-token branches that relied on it now
declare the profile in the result metadata. Every other command
either already declared it or returns no :profile-id; all 30
registered command namespaces were checked.

The tests that pinned the old behavior are replaced by ones
covering the new contract.

AI-assisted-by: space-bunny-free

* 🐛 Coerce the audit profile-id override to a uuid

The only sanctioned way for a command to override the profile of an
audit event is the ::audit/profile-id metadata, and the value is set
by hand in a dozen commands, some of them reading it from token
claims or other sources we do not type.

schema:event requires a uuid and submit* swallows the validation
error, so a string did not fail loudly: the row was dropped silently.
Values that cannot become a uuid are now discarded with a warning
and the event falls back to the caller, which is always a valid uuid.
A uuid, the common case, exits on the first check.

AI-assisted-by: space-bunny-free
2026-09-29 09:47:07 +02:00
Andrey Antukh
72dad5d67d
✨ Add renderer option on create-demo-profile command (#11894)
* ✨ Add renderer option on create-demo-profile command

Allow create-demo-profile to take an optional renderer (svg or
wasm) and store it on profile props. When omitted, no renderer is
written so the Penpot default still applies.

Closes #11893

AI-assisted-by: Muse Spark 1.3 Free

* 📚 Fix renderer doc version on create-demo-profile

Correct the ::doc/changes version for the new renderer param from
2.20 to 2.18.1.

AI-assisted-by: Muse Spark 1.3 Free
2026-09-28 18:21:07 +02:00
Andrey Antukh
eec06a5987
🐛 Allow registration with disabled public registration (#11912)
* 🐛 Allow invitation-based registration when disable-registration is set (#5178)

Per documentation, disable-registration 'disables registration
(still enabled for invitations only)'. Two bugs prevented this:

1. verify_token.clj: when processing an invitation token for a
   non-logged-in user with no member-id, the redirect included
   registration-disabled? in its condition, sending invited users
   to the login page instead of the register page.

2. auth.clj validate-register-attempt!: the registration-disabled
   check fired unconditionally before the invitation-token check,
   rejecting the actual register RPC even with a valid invitation.

Fix: in verify_token.clj remove registration-disabled? from the
redirect condition for new-user invitations. In auth.clj restructure
the check as an if/else: with an invitation token, validate the token
and allow registration; without one, enforce the flag as before.

* 🐛 Allow registration with disabled public registration

Allow valid team invitations to create new profiles when public
registration is disabled, while keeping password login and invitation
validation required.

Add backend regression coverage for flag combinations and verify-token
redirects, frontend route coverage, and configuration documentation.
Closes #5178

AI-assisted-by: space-bunny-free

* 🐛 Revalidate active invitation during registration

Require a live, unexpired team invitation before using the
registration exception, and recheck it before creating a profile.
Reuse the same lookup in invitation token verification.

Add regression tests for canceled and expired invitations, the
registration race, and explicit redirect contracts. Update docs
and backend auth guidance.

AI-assisted-by: Space Bunny Free

* 🐛 Lock and normalize invitation registration checks

Lock active invitation rows during transactional registration and
acceptance so cancellations cannot race with profile or membership
creation.

Normalize invitation emails before comparisons and database lookups.
Add concurrency, email casing, and final flag regression tests.

AI-assisted-by: Space Bunny Free

---------

Co-authored-by: Sumit Ridhal <sridhal@redhat.com>
2026-09-25 14:13:12 +02:00
Marina López
0255bed6c4
✨ Add deployment info to events (#11867)
* ✨ Add deployment info to events

* ✨ Add get-environment-data RPC method

Add a single public RPC method returning the deployment type and
the enabled environment flags. It replaces get-deployment on the
management API and get-enabled-flags on the main API.

get-enabled-flags stays as a deprecated alias returning only the
flags, so existing callers keep working until it is removed.

The frontend event initialization now reads the flags from the new
method. The exposed flags stay limited to audit-log and telemetry
to avoid leaking internal backend flags.

AI-assisted-by: deepseek-v4.1-flash

---------

Co-authored-by: Andrey Antukh <niwi@niwi.nz>
2026-09-25 11:22:43 +02:00
María Valderrama
f35e12f716
🐛 Fix organization/team switcher issues from UX review (#11899)
* 🐛 Fix organization/team switcher issues from UX review

* 🐛 Let members leave an organization without SSO credentials

* 🐛 Fix style from previewed organization in the team switcher

* 🐛 Fix leave-organization modal test stubs
2026-09-25 09:54:58 +02:00
Marina López
dfc1848ffc
♻️ Build organization invitation audit event in frontend (#11825)
* ♻️ Build organization invitation audit event in frontend

* ♻️ Align invitation token profile-id with created-by

The invitation token carried the minter in :profile-id while the
invitation row tracks the creator in :created-by. Both mean the
inviter, so re-sends or re-requested links made them disagree and
forced a second response key, :user-who-send-invitation.

Mint :profile-id from :created-by in both token creators, backfill
it from the row on accept (covers stale in-flight tokens), and drop
the duplicate response key. The frontend maps :profile-id to the
unchanged :user-who-send-invitation audit prop.

AI-assisted-by: Muse Spark 1.3 Free

* ♻️ Reuse token ids in invitation accept response

Backfill :member-id with the accepting profile and drop the
:user-id duplicate from the verify-token response, mirroring the
:profile-id/:user-who-send-invitation cleanup. The frontend maps
:member-id to the unchanged :user-id audit prop.

AI-assisted-by: Muse Spark 1.3 Free

---------

Co-authored-by: Andrey Antukh <niwi@niwi.nz>
2026-09-24 14:13:52 +02:00
Andrey Antukh
643695b7fa
✨ Send notification email on password change (#11393)
* ✨ Send notification email on password change

Add a password-changed email template and send it after every
successful password change or first password set operation. This
gives users a visible signal when a new authentication factor is
established on their account.

Closes #11392

AI-assisted-by: longcat-2.0

* 🐛 Address code review feedback on password notification

- Send password-changed notification from recover-profile too
  (forgot-password reset path was missing the email).
- Strengthen test assertions to verify email factory, recipient,
  and name via :call-args-list instead of just call-count.
- Add negative test: no email sent when old-password is wrong.
- Wrap pre-existing update-profile-password test with send! mock
  to keep its scope focused.

Ref: PR #11393

AI-assisted-by: longcat-2.0
2026-09-23 13:43:18 +02:00
Andrey Antukh
1c7a73ec16
✨ Add account lockout after failed login attempts (#11402)
* ✨ Add account lockout after failed login attempts

Implement per-account brute-force protection using a Redis-backed
failed-login counter. After 5 failed attempts within 15 minutes, the
account is temporarily locked out and all login attempts (including
with the correct password) are rejected with a 429 response.

Closes #11397

AI-assisted-by: longcat-2.0

* 🐛 Bind LDAP session to directory-verified profile

The account-lockout change added a shortcut that preferred the
profile matching the typed email over the one returned by the LDAP
directory. These can differ with aliases, UPNs, or multi-valued mail
attributes, letting a user with valid LDAP credentials bind a session
to another Penpot account.

Keep the typed-email profile only for lockout checks. After LDAP
succeeds, resolve the session profile from the directory identity as
before and clear failed attempts on that profile.

AI-assisted-by: deepseek-v4.1-flash
2026-09-23 12:31:50 +02:00
María Valderrama
3a5481c577
✨ Error page for instances without admin-console configuration (#11812)
* ✨ Error page for instances without admin-console configuration (#11649)

* ✨ Error page for instances without admin-console configuration

* 📎 Code review

* 🐛 Fix link nitrate-not-configured
2026-09-23 10:36:40 +02:00
Andrey Antukh
2f679eaa0e
💥 Remove client-provided id from creation RPC commands (#11784)
The seven creation commands no longer accept an optional client
id: create-file, create-project, create-team,
create-team-with-invitations, upload-file-media-object,
create-file-media-object-from-url and assemble-file-media-object.
The server always generates the identifier; a sent id is ignored.

Malli maps are open and the RPC layer never strips unknown params,
so the handlers that would still honor an id (create-file,
create-project) now drop it explicitly. Internal callers that pass
remapped ids (project duplicate, binfile import) keep working.

Closes #11783

AI-assisted-by: muse-spark-1.3-contributor
2026-09-22 15:51:50 +02:00
Andrey Antukh
38212e757c
🐛 Bound decompressed size of JSON entries on binfile v3 import (#11607)
* 🐛 Bound decompressed size of JSON entries on binfile v3 import

Every JSON/text zip entry (manifest, files, pages, shapes, colors, components, typographies, tokens, plugin-data) was decompressed without any size limit, letting a small .penpot archive exhaust the backend heap (GHSA-qcw7-v626-g6cf). Only binary storage blobs were guarded. Reuse the existing size-limiting-stream guard on the text path: 20 MiB cap per entry, 200 MiB cumulative budget per import job, plus a cheap declared-size pre-check. Both limits are configurable and wired through the binfile, management and debug entry points. Adds zip-bomb regression tests for the file entry, the synchronous manifest read and the cumulative budget. Closes #11606

AI-assisted-by: muse-spark-1.3-contributor

* ♻️ Uniform binfile import limits behind init-limits

Move the binfile import limits to a single source of truth in app.binfile.common (default-* vars) and drop the duplicated entries from config/default; env overrides keep working through the schema. Resolve all limits once per job with init-limits (::max-size, ::total-max, ::current-size, ::max-object-size, ::max-zip-entries) instead of rebuilding the map per zip entry. Thread cfg as the first arg through the v3 readers, collapse read-plain-entry into read-entry, and give size-limiting-stream a single explicit-counter arity. v1 keeps using the compiled default (mechanical rename only). No behavior change. AI-assisted-by: muse-spark-1.3-contributor

* ♻️ Rename binfile limits to text-entry/binary-entry terms

Use text-entry/binary-entry vocabulary consistently across config keys, bfc input keys, default-* vars and the limits resolved by setup-limits (::max-text-entry-size, ::max-text-total-size, ::current-text-size, ::max-binary-entry-size). Rename init-limits to setup-limits. No behavior change. AI-assisted-by: muse-spark-1.3-contributor

* ♻️ Rename cumulative text counter and document binary limit

Rename ::current-text-size to ::accumulated-total-text-size for clarity and expand the default-max-binary-entry-size comment to match the other limit vars. No behavior change. AI-assisted-by: muse-spark-1.3-contributor

* ♻️ Harden binfile guards and prove budget accumulation

Add a regression test that only passes when text bytes accumulate across entries (budget between largest entry and summed total; verified red against a per-entry atom). Include the entry name in streaming-guard errors, count skipped bytes against the budget with a direct unit test, and forward all four limit keys in get-manifest. No behavior change. AI-assisted-by: muse-spark-1.3-contributor

* ♻️ Thread cfg through get-manifest

get-manifest now takes the caller cfg and resolves limits with setup-limits like the import job itself, instead of building a single-use mini-cfg from cf/get. No behavior change. AI-assisted-by: muse-spark-1.3-contributor
2026-09-22 15:16:59 +02:00
Andrey Antukh
96c44c4c30
🐛 Trim linked-library data in view-only bundle for share links (#11627)
* 🐛 Trim linked-library data in view-only bundle for share links

The anonymous get-view-only-bundle RPC merged each linked library whole,
exposing library pages the share link never granted. For share-link
permissions, each library is now reduced to the narrow data keys with
its own pages dropped and only the components referenced by the allowed
pages kept (nested references followed); membership bundles are
unchanged.

Closes #11617

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Resolve nested library components via main instance in bundle trim

Stored components carry no objects, so the transitive walk missed nested
components. Follow references through the main-instance subtree instead,
share the narrow data keys between the primary and library scopes, and
cover component filtering with a real-instance RPC test.

Review follow-ups F1-F3 for #11617

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Address review findings on view-only library trim

Hoist invariant refs out of the fixpoint, make the cross-library
main-instance fallback deterministic, pin the trimmed envelope,
and add RPC tests for disallowed-page isolation and
cross-library nesting.

Follow-ups to #11617

AI-assisted-by: muse-spark-1.3-contributor

* ♻️ Index libraries by id with d/index-by

Replace the manual into/juxt index with the shared helper.
No behavior change.

AI-assisted-by: muse-spark-1.3-contributor
2026-09-22 14:43:04 +02:00
Andrey Antukh
2255266d45
✨ Enable closed schemas for RPC methods (#11136)
* ✨ Enable closed schemas for RPC methods

* 🐛 Fix duplicate make-dummy-request test helper definition

The branch added a variadic DummyRequest/make-dummy-request pair but
left the pre-existing single-arg definition in place. Because it was
loaded last, zero-arg (make-dummy-request) calls added by
prepare-rpc-params and rpc-nitrate-test threw ArityException, which
broke 384 tests and caused 14 downstream assertion failures.

Remove the stale duplicate so the variadic definition is the only
one, and drop the now-unused yrq alias and duplicate yres alias.

AI-assisted-by: deepseek-v4.1-flash

* ✨ Add focused tests for make-dummy-request helper

Pin the call contract of make-dummy-request, which the suite uses
in three styles: no arguments, a single options map, and keyword
arguments. The helper's redefinition shadowing in 8ca95adb98 was
only caught by a full-suite run with hundreds of unrelated errors;
these tests fail locally in a focused --focus run.

Cover the zero-arg defaults, map and keyword overrides, the
:body-bytes -> ByteArrayInputStream wrapping, :body-stream
precedence, and cookie readback. Also clarify the docstring to
list all supported call styles.

AI-assisted-by: deepseek-v4.1-flash

* 🚑 Prevent RPC client params from overriding auth context

Strip qualified keys from decoded request params before merging
them with the server-built auth context, so transit bodies can
no longer override ::profile-id, ::auth-type or ::token-perms.
Adds a regression test proving the override and the fix.

AI-assisted-by: muse-spark-1.3-contributor

* 📚 Merge backend subtleties memories under generic name

Rename rpc-db-worker-subtleties to subtleties and fold in
http-storage-filedata-subtleties, so the name no longer
enumerates topics. Update all mem: references accordingly.

AI-assisted-by: muse-spark-1.3-contributor

* ✨ Add realistic tests for RPC auth override

Cover the transit wire vector and the real wrapped :get-profile
method with two database profiles, proving a session cannot read
another profile by smuggling :app.rpc/profile-id in the body.

AI-assisted-by: muse-spark-1.3-contributor

* ✨ Add e2e test for RPC auth context override

Parametrize rpcPost with contentType, accept and query so e2e
can send hand-written transit bodies without new dependencies.
The new test proves a transit-smuggled :app.rpc/profile-id no
longer overrides the session in get-profile. Also fix the demo
email assertion in auth-flow to the current uuid format.

AI-assisted-by: muse-spark-1.3-contributor
2026-09-22 14:03:02 +02:00
Andrey Antukh
117c8db0bb Merge remote-tracking branch 'origin/staging' into develop 2026-09-22 10:24:30 +02:00
Andrey Antukh
e05747b546
✨ Restrict optional RPC ids to user-provided UUIDs (#11777)
* ✨ Restrict optional RPC ids to user-provided UUIDs

Add ::sm/user-provided-uuid, backed by a version and variant
aware regex that only accepts v4, v7 and v8 instances. Use it
for the optional :id of the creation RPC commands so reserved
versions such as v3 are rejected at validation time. Reads
such as get-team keep the lax ::sm/uuid. Cover the predicate
and the schema on both JVM and JS runtimes.

AI-assisted-by: muse-spark-1.3-contributor

* ✨ Cover id version restriction at the RPC boundary

Add backend regression tests proving the seven creation commands
reject reserved-version ids (v3) with :params-validation and
accept v4 ids (plus v7/v8 on create-team) through the real
decode and validate path. Also drop two duplicated assertions
and document the version and variant of every fixture UUID in
user-provided-test.

AI-assisted-by: muse-spark-1.3-contributor
2026-09-22 09:45:50 +02:00
Andrés Moya
3cd9bfa9de
✨ Add sync with design tokens in external libraries (#10293)
* ✨ Auto link tokens when adding external libraries (provisional)

* 🔧 Refactor tokens-lib initialization

* 🔧 Add separated TokenStatus to store status apart of TokensLib

* 🔧 Make all status operations use the new data structure

* 🔧 Normalize status helper functions and access token sets by id

* 🔧 Rename :tokens-file to :tokens-source

* 🎉 Allow the user to choose the tokens-source of a file

* 🎉 Make tokens library readonly when it's in an external file

* 🎉 Show tokens in library summaries

* 🎉 Show source info in sidebar

* 🔧 Fix integration tests

* 🐛 Propagate changes of token values in external library

* 🎉 Layout updates

* 🔧 Refactor tokens source calculations

* 🔧 Add harder checks for nil or empty values in everything

* 🐛 Fix some integration tests

* 🔧 Add integration tests for tokens in external libs

* 🔧 Validate and repair missing tokens status

* 🎉 Make ui changes optional with config flag

* 🐛 Propagate tokens after synchronizing components in ext library

* 🐛 Propagate tokens after creating new instances

* 🐛 Propagate tokens after synchronizing tokens in ext library

* 🐛 Add a tokens source icon to libraries section (#11439)

* 🐛 Add a tokens source icon to libraries section

* 🐛 Fix ellipsis on library names

* ♻️ Remove code under flag on legacy component

* 🐛 Fix token theme name on inspect tab

* 🎉 Add changes notification (#11476)

* 🎉 Add changes notification

* ♻️ Change fn names

* 🐛 Fix tokens source label truncation and missing translations (#11533)

* 🐛 Fix tokens source label truncation and missing translations

The tokens source file name always showed, even for the current file,
and long names wrapped onto a second line instead of truncating
because the header used flex-wrap and overflow-wrap: break-word
instead of single-line ellipsis.

Show the source row unconditionally (it now displays "This file" when
the source is the current file, matching the connected-library case),
truncate the file name to one line with an ellipsis, and only attach a
tooltip with the full name when the text is actually truncated.

Replace the hardcoded UI strings with translated ones and add their
English and Spanish entries.

AI-assisted-by: claude-sonnet-5

* 🐛 Remove redundant effect dependency in tokens source

file-name-truncated? was listed as a dependency of the with-effect
that checks and observes label truncation, even though it isn't
read inside the effect body. Since the effect itself flips that
state via check-file-name-truncated, including it as a dependency
caused the ResizeObserver to be needlessly disconnected and
reconnected on every truncation change.

AI-assisted-by: claude-sonnet-5

* 🐛 Fix small visual error

* 🐛 Fix problem with plugins

* 🐛 Fix playwright tests

---------

Co-authored-by: Eva Marco <evamarcod@gmail.com>
Co-authored-by: Eva Marco <eva.marco@kaleidos.net>
Co-authored-by: alonso.torres <alonso.torres@kaleidos.net>
2026-09-21 15:37:45 +02:00
Andrey Antukh
df383be6b2 Merge remote-tracking branch 'origin/staging' into develop 2026-09-15 19:59:54 +02:00
makesomethingshit
4a5c6fce7f
⚡ Fetch only caller share-link in view-only bundle (#11657)
* ⚡ Fetch only caller share-link in view-only bundle

Share-link callers now resolve a single row with a composite
(id, file-id) predicate instead of loading all sibling rows
and filtering in memory. Membership path keeps full query.

Related #11633

AI-assisted-by: muse-spark-1.3-contributor

Signed-off-by: makesomethingshit <junsoo1172@gmail.com>

* ⚡ Add DB-access regression test for share-link bundle

The share-link path must resolve the caller row with a composite
(id, file-id) single-row lookup and never run the full
file-id query. Keep cross-file replay coverage.

Related #11633

AI-assisted-by: muse-spark-1.3-contributor
Signed-off-by: makesomethingshit <junsoo1172@gmail.com>

---------

Signed-off-by: makesomethingshit <junsoo1172@gmail.com>
Co-authored-by: Andrey Antukh <niwi@niwi.nz>
2026-09-15 17:40:52 +02:00
Andrey Antukh
ad7e035b63
✨ Add indirection for upload-chunks storage via new table (#11651)
* ✨ Add upload_session_chunk indirection for chunked uploads

Chunks now live in the upload_session_chunk table with non-deleting
foreign keys to storage_object and upload_session, instead of tempfile
objects with session metadata. Reads go through a JOIN, so chunk state
never scans storage_object.

Uploads validate the live session, reject duplicate indexes, and store
objects in the new upload-session bucket without extra metadata.
Assemble removes mappings and marks the session consumed; objects-gc
procedurally purges consumed and stalled sessions, touching referenced
objects first. Touched-gc and deleted-gc handle the new bucket, and
upload-session-gc is removed.

Closes #11644

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Fix quota, give-up and coverage for session chunks

Exclude consumed sessions from the sessions-per-profile quota so
finished uploads free their slot at once. Remove chunk mappings before
the gc-deleted give-up delete to respect the NO ACTION keys. Catch
java.sql.SQLException for duplicate chunks. Cover the profile-owned
session purge and the UNIQUE race backstop with tests.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Align chunked upload tests with upload_session_chunk

Drop the duplicate-index tests written against metadata-backed
chunks; the UNIQUE mapping makes those cases unrepresentable and
the new tests cover them. Rewrite the rejected-duplicate tests to
expect :validation/:chunk-already-exists and assert against the
upload_session_chunk table, and scope the chunk-too-large "nothing
stored" check to the mapping table.

AI-assisted-by: muse-spark-1.3-contributor

* ♻️ Use NO ACTION DEFERRABLE session FKs in single migration

Fold the profile FK change into 0154 so the feature ships one
migration. All three upload session FKs use ON DELETE NO ACTION
DEFERRABLE: identical to RESTRICT in normal operation, but deferrable
for tooling that relies on SET CONSTRAINTS ALL DEFERRED. Extend the
RESTRICT test to the direct profile delete.

AI-assisted-by: muse-spark-1.3-contributor

* ♻️ Reserve chunk slot before writing blob in upload-chunk

Make object_id nullable and insert the mapping with NULL inside the
session-locking transaction, then write the blob outside it and link
it with a conditional update. A failed write removes the mapping and
reraises; a mid-flight death leaves a NULL row and the client starts
a new session.

AI-assisted-by: muse-spark-1.3-contributor

* 🔥 Remove redundant session_id index on upload_session_chunk

The UNIQUE(session_id, chunk_index) btree already serves
session_id-only lookups and the session FK check through its
leftmost column, so the standalone index only taxed the
per-chunk INSERT path. Verified with EXPLAIN on an equivalent
table shape.

AI-assisted-by: muse-spark-1.3-contributor

* ⚡ Merge chunk touch and delete into single RETURNING query

Replace the SELECT-then-DELETE round-trip in
delete-upload-sessions! with DELETE ... RETURNING object_id,
touching each returned object. Same semantics, one less query
per purged session. Follows the RETURNING pattern already used
in file-gc.

AI-assisted-by: muse-spark-1.3-contributor

* ♻️ Let objects-gc own chunk mapping deletion

Assemble-chunks now only marks the session as consumed; the
chunk mappings stay until objects-gc purges them (touching the
chunk objects first), leaving a single procedural deletion
path for consumed, stalled and profile-purge sessions.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Fix font-deletion GC expectations for chunk objects

Update final storage-gc-touched counts to include the two
chunk objects touched by objects-gc when purging consumed
upload sessions (8/5/5 instead of 6/3/3).

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Release chunk reservation when the link UPDATE fails

Review feedback on #11651: the link UPDATE in upload-chunk could
leave a NULL reservation behind, blocking retries of the same
index with :chunk-already-exists. Remove the reservation when
the link fails so the client can retry in the same session;
the orphaned blob stays touched for touched-gc. Also realign
the process-bucket! cond branches in gc-touched.

Tests: chunked-upload-link-failure-releases-slot and
chunked-upload-null-reservation-blocks-retry.

AI-assisted-by: muse-spark-1.3-contributor
2026-09-15 17:26:43 +02:00