82 Commits

Author SHA1 Message Date
Andrey Antukh
93d36d04d8
✨ Stabilize fragmented file data across snapshots and GC (#12420)
* ✨ Stabilize fragmented file data across snapshots and GC

Snapshots, automatic and user, now store pointer-based rows that share
fragments with the live file instead of duplicating page content.
`get-file-snapshot` resolves those pointers on the server so version
previews keep rendering plain data, and `restore!` reuses the shared
rows. Export still materializes (the package carries no fragments) and
import regenerates them.

`file-gc` treats the main file and every unpurged snapshot as liveness
roots, reads the file unrealized, and reattaches the original pointer
wherever the cleaned value is unchanged, so a pass over an untouched
file writes no fragments and a pass over an edited file keeps the ids
of the pages it did not touch. Snapshot creation and the restore
backup take the file row `FOR SHARE`, which serializes them with the
pass and closes the race that let a new snapshot reference fragments
the pass was about to mark.

All backend pointer bindings go through the `fdata/with-pointers`,
`with-tracked`, `with-tracked-pointers` and `with-untracked` macros.
The loader reads fragments a previous pass marked until `objects-gc`
purges them, and the frontend fails loudly with retry instead of
dropping an unresolved fragment.

Review fixes in this branch: the automatic snapshot attaches before
pending migrations, `realize-pointers` is the single verb,
`reattach-unchanged` takes the identity fast path, the restore guard
takes its revn from the backup read, and `PointerMap.withMeta`
generates a fresh id when metadata changes.

Tests cover both snapshot writers (red without the lock), snapshot
sharing and pin release through the real delete and purge paths,
reattach against orphan marking, preview after offload, the pointer
binding contracts and the with-meta id change.

Closes #12410

AI-assisted-by: muse-spark-1.3-contributor
AI-assisted-by: deepseek-v4.1-flash

♻️ Pin no-churn behavior for untouched pages

Pin the behavior the churn verification measured: a save keeps the
fragment id of every page it does not touch, with a page of eleven keys
(PersistentHashMap), a 46-shape objects map and a nine-page index (hash
map). Also pin the mixed case: a data-level objects-map flip rewrites
the old pointer pages once and the next save keeps the new ids.

The memory records the identity rule behind it: re-assoc with the same
value returns the same map in array and hash maps alike.

Closes #12410

AI-assisted-by: deepseek-v4.1-flash

⚡ Stop loading untouched pages when enabling objects-map

`enable-objects-map` wrapped every page on every save, and reading a
page's objects forced its PointerMap to load the fragment. A save of
one page in a seven-page file therefore made seven fragment reads.

Skip unloaded pointers: their fragment was already wrapped by the
encode-file that wrote it, so loading it only to wrap it again is a
no-op. The save now loads only the pages it edits. A page with a plain
fragment (a data-level feature flip) keeps its plain fragment until it
is first loaded or edited; a migration that enables the feature must
rewrite the data itself.

Cover the change with load-count tests for save and restore, update the
flip test to the new contract, and drop the stale restore comment.

Closes #12410

AI-assisted-by: deepseek-v4.1-flash

⚡ Keep page fragment ids through pending migrations

Applying a pending migration on read realized the file, ran the
migration and persisted it again; the persist wrapped every plain page
in a new PointerMap, so every page got a new fragment id even when the
migration did not change it. That leaves the fragment cache cold and
rewrites every fragment on the first read after a deploy.

Capture the original pointers before realizing, reattach the unchanged
positions after migrating (same value comparison file-gc already uses),
persist the reattached file and return the realized one: the read
contract is plain data. Read-only connections skip the persist and the
reattach. A migration that adds objects-map skips the reattach so the
pages get wrapped.

Move the three helpers to fdata so file-gc and the migration path share
them, and cover the path with a real 0031 migration test: the unchanged
page keeps its id, the changed page gets a new one, the caller gets
realized data and a second read does not migrate again.

Closes #12410

AI-assisted-by: deepseek-v4.1-flash

* 🐛 Realize library pointers with each library's own loader

Fragment rows are keyed by file id, and every caller binds the
fragment loader of the file it works on. get-resolved-file-libraries
returned libraries with their pages still as pointers, so the first
read of a library page under fdata/pointer-map asked the consuming
file for the fragment and failed with :fragment-not-found. That
broke migrations and file validation for every file using a
pointer-mapped library.

The weak-map loader now realizes each library's pointers with the
library's own id. The test fails without the change.

AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>

* 🐛 Restore the pointer-map require the repair path still needs

The branch dropped `app.util.pointer-map` from the namespace while the
auto-repair path (added to develop before this branch) still rebinds
`pmap/*tracked*` around validation and repair in
`repair-preexisting-damage` and `process-changes-and-validate`. The
namespace did not compile, so the whole backend failed to load.

Those toggles do not fit the fdata macros: `with-untracked` binds nil
and `with-tracked` creates a fresh atom, while the repair path has to
turn tracking off and restore the existing atom (and read it before the
repair starts). Restore the require and keep the bindings explicit.

Also pin the feature-off path with a test: with `fdata/pointer-map` off
(the default), a save creates no fragments, file-gc writes nothing and
the snapshot preview still returns plain data. Drop a stray `\n` char
literal left at the end of one deftest.

Update the memories with the exception to the macro rule, the snapshot
FOR SHARE locking and the pre-migration automatic snapshot.

Closes #12410

AI-assisted-by: deepseek-v4.1-flash

* 🐛 Validate debug files with their own pointer loader

The `/dbg/actions` file validation passed the file id where
`get-resolved-file-libraries` expects the file map, so the library map
was seeded with a nil key. It also validated a file read without
realizing it, so the first page lookup of a pointer-mapped file raised
"load is not supported when *load-fn* is not bind" and the endpoint
could not validate exactly the files the feature touches.

Read the file realized (validation walks every page and component
anyway) and build the library map from the file. The library loader
already resolves each library's pointers with its own id, so the
realized file and its libraries validate as plain data. Covered by a
handler-level test that fails before the change.

AI-assisted-by: deepseek-v4.1-flash

* ♻️ Bind the snapshot loader through the fdata helper

`reduce-snapshots` hand-rolled the same `binding` that
`fdata/with-pointers` builds (it also loaded the function's own
pointer loader before entering the reduction). Use the macro so the
rule holds there too and the namespace no longer needs
`app.util.pointer-map`; the loader is identical (same cfg and file-id)
and snapshot walking behaves the same.

AI-assisted-by: deepseek-v4.1-flash

* 📚 Note the pointer-binding exceptions and snapshot ordering

Record what the feature review settled:

- Two direct `binding`s stay on purpose: the fine-grained `*tracked*`
  toggles of `app.rpc.commands.files-update` (validation/repair turn
  tracking off and restore the existing atom) and
  `binfile.common/encode-file` (fresh tracked set inheriting the
  current one). Atom-scoped macros would only be worth it if the set
  grows.
- The automatic snapshot is taken before the pending migration, and
  the pre-migration window is narrow: the `get-file` RPC migrates on
  read and its ETag hashes the available migrations, so a post-deploy
  read cannot be a 304 from the pre-deploy state. The old-format
  snapshot only appears when the save is the first request to touch
  the file after the deploy, and `get-file-snapshot` previews it as
  stored.

AI-assisted-by: deepseek-v4.1-flash

* 🐛 Apply pending migrations when previewing a version

Automatic snapshots are attached before the save applies pending file
migrations, so a version can store the old format. The preview RPC
handed that data to the UI as stored, and the viewport renders it with
the current code.

Realize the snapshot with the live file id (fragments are keyed by
file, not by snapshot), normalize its migrations to a set (the decoder
keeps the pgarray as a vector, and `set/difference` would compare
indexes) and apply the pending migrations with the pure
`fmg/migrate-file` and the live libraries. Nothing is persisted: the
stored version keeps its data and restoring it re-migrates on the next
read. The response now carries the snapshot's migrations.

Use `:features` the way `get-file` does for `fdata/objects-map`: a
client that does not declare it gets plain objects. The historical
feature set is not checked against the enabled one on purpose: a
version may predate a feature change.

Cover the preview of a snapshot with a pending migration and the
objects realization; both pin that the stored row is untouched. Update
the file-data memory with the preview contract and the vector/set
trap.

AI-assisted-by: deepseek-v4.1-flash

* 🐛 Decode snapshot migrations as an ordered set

`file-snapshots/decode-snapshot` handed the `file_change.migrations`
pgarray as a vector. `set/difference` falls back to index lookups when
the first collection is shorter than the second, so the pending set was
right only by accident: a stored subset happens to work, while a
stored set larger than `available-migrations` (a snapshot taken by a
newer build) marks every available migration as pending and re-applies
it to already migrated data.

Decode it as a `d/ordered-set`, the type the rest of the code and
`schema:file` expect. This also covers `restore!` and the snapshot
list, and it lets `get-file-snapshot` drop the `(into #{} ...)`
normalization it was carrying. Re-wrap the union in `fmg/migrate` so
the invariant holds even when a caller hands a plain set.

A snapshot with NULL `migrations` predates SQL migration 0140 and
keeps the format version of its time. Pin that legacy path: the
preview regenerates the applied set from that version, migrates in
memory and leaves the stored row untouched. Update the file-data
memory with the invariant and the legacy path.

AI-assisted-by: deepseek-v4.1-flash

* 🐛 Fix flaky waiting-gauge assertion in climit test

The throwable-while-waiting test asserted the exact list of
waiting-gauge publications, but the component sampler republishes
the same count when its first tick lands inside the recorded
window under load. Assert only the last value, which is the
contract the helper documents.

AI-assisted-by: muse-spark-1.3

---------

Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
Co-authored-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
2026-11-02 15:02:55 +01:00
Alejandro Alonso
a8e8a54a2b Merge remote-tracking branch 'origin/staging' into develop 2026-10-28 10:35:31 +01:00
Eva Marco
01e0b32850
🐛 Remap library refs inside deleted components on absorb (#12232)
* 🐛 Remap library refs inside deleted components on absorb

Deleted components keep their shapes in :objects, to be restored and
to resolve their remaining copies. When a library was absorbed into a
file (export with libraries merged, unsharing or deleting the
library), absorb-assets only looked for usages in pages and in live
components. The shapes inside deleted components kept pointing to the
library, so they no longer matched their copies in the pages: the file
failed validation with :component-id-mismatch and could not be saved.

Also look for usages in deleted components, so their library
components, colors and typographies are remapped to the local file.

Relates to #10806

AI-assisted-by: claude-opus-5-5

* 🐛 Persist pointer-map fragments when absorbing a library

absorb-library-by-file saved the file without its :features. Without
"fdata/pointer-map" there, encode-file does not persist the changed
fragments, so in files with pointer maps the pages and components
that absorb-assets remapped were never written. After unpublishing or
deleting a library, its consumers failed to load with "fragment not
found".

Pass :features to update-file!. Exporting with shared libraries does
not go through this path, so it was not affected.

AI-assisted-by: claude-opus-5-5
2026-10-26 17:56:31 +01:00
Andrey Antukh
74add05226 Merge remote-tracking branch 'origin/staging' into develop 2026-10-23 13:40:44 +02:00
Eva Marco
d8862c759b
🐛 Fix shared files 500 on path content without coordinates (#12426)
* 🐛 Fix legacy path migrations on content without coordinates

PRE still returned 500 on get-team-shared-files after #12357, now with
"expected rect instance" in legacy-6. When a path has no points, legacy-6
rebuilds its rect from the content, but content whose points are not
numbers (NaN or infinite coordinates, segments without params, only a
close path) gives no rect either, and rect->points rejects nil.

Fall back to the empty rect in legacy-6. A path with segments missing
coordinates then failed later, in 0003-convert-path-content-v2, since the
binary path data can not store them; drop those segments there.

Relates to #12343

AI-assisted-by: claude-opus-5-5

* 🐛 Skip shared files that fail to load in get-team-shared-files

get-team-shared-files loads and migrates every shared file of the team,
so one file whose migration fails made the whole request return 500 and
the "Add library" modal could not open.

Catch the error per file, log it with the file and team ids, and leave
that file out of the list. The other libraries still show, and the error
report names the broken file.

Closes #12343

AI-assisted-by: claude-opus-5-5
2026-10-21 12:31:03 +02:00
Andrey Antukh
b633093336 Merge remote-tracking branch 'origin/staging' into develop 2026-10-17 16:28:59 +02:00
Andrey Antukh
70b317a976
🐛 Harden upload path handling and validation error bodies (#12067)
* 🚑 Reject forged upload paths outside multipart

Transit bodies can smuggle upload maps whose path points at any
server file; the schema only checks the shape and handlers open
the path directly.

app.rpc now injects a server-side multipart flag that bodies cannot
forge, and methods declare their upload keys with the new
multipart-only-params metadata, enforced centrally after schema
validation with a params-validation error. Covers all nine upload
methods, including chunked and management flows.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Sanitize raw explain in generic validation error bodies

The generic :validation branch returned ex-data untouched, so errors
carrying the raw malli explain (live schema objects, e.g. storage
invalid-storage-metadata) broke JSON encoding mid-stream and clients
got truncated bodies. Apply the same clean treatment the known codes
already use: drop the raw explain and attach its humanized text.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Add e2e path traversal tests for binfile and media upload

Add two e2e suites that hand-write transit bodies with a forged
["~#path", ...] upload path (no transit library needed) and expect
400 validation/params-validation. They cover import-binfile :file,
upload-file-media-object :content and upload-chunk :content, plus a
plain-JSON string-path control case. Before the multipart-path fix
they failed (500 io-exception, invalid-image, 200 stored chunk);
after it they pass.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Use multipart helper in binfile SVG test setup

The SVG import setup uploaded through plain command!, so the new
multipart-only guard rejected it with params-validation. Switch
the setup to multipart-command! like the other upload tests.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Use multipart helper in remaining upload tests

The multipart-only upload guard rejects plain RPC calls carrying
upload maps. Five tests with real files still used command! and
failed with params-validation. Switch them to multipart-command!
while keeping the forged-content tests on plain command!.

AI-assisted-by: muse-spark-1.3-contributor
2026-10-17 16:01:04 +02:00
Andrey Antukh
dfc55a7192 Merge remote-tracking branch 'origin/staging' into develop 2026-10-16 12:48:49 +02:00
Andrés Moya
feb73f0fd8
✨ Handle tokens libraries when exporting .penpot files (#12077)
* ✨ Handle tokens libraries when exporting .penpot files

Exporting a file that takes its tokens from an external library left
the :tokens-source of the file pointing at a library that is not in the
export, and the tokens lib of that library was not exported.

With :detach-libraries, unset the :tokens-source and the tokens status
in detach-external-references. The file is exported without any link to
the library, and the shapes keep their applied tokens.

With :merge-libraries, add cfo/absorb-tokens-lib, which copies the tokens
lib of a library into a file and unsets its :tokens-source, so the file
becomes its own tokens source. The tokens status of the file is kept; if
it has none, the one of the library is copied. If the library has no
tokens lib, only the source is unset. embed-assets calls it for the
library that is the tokens source of the file, and ignores the other
linked libraries.

Add unit tests for absorb-tokens-lib, and binfile tests that export a
file with tokens from an external library with both modes, checking the
exported files, the tokens lib, the tokens source and the applied
tokens.

Also reformat the export-type docstring in binfile v3 and remove a
leftover debug prn in the export dialog.

AI-assisted-by: claude-sonnet-5-5

* 🐛 Preserve active token sets when importing linked libraries

Cache each decoded tokens library for the duration of an import so files
and their tokens source resolve active sets against the same generated IDs.
Add a round-trip regression test for exports that include linked libraries.

* 🐛 Preserve tokens when deleting or unpublishing libraries

Absorb the tokens library into files that use it as their tokens source.
Keep referencing file IDs in the deletion task before removing relations.

Embed tokens from deleted, unlinked sources during export, and clear stale
source and status data when the library no longer exists. Add regression
tests for deletion, unpublishing, and exports with unavailable libraries.

---------

Co-authored-by: alonso.torres <alonso.torres@kaleidos.net>
2026-10-15 16:53:30 +02:00
Andrey Antukh
a53481c3b7
✨ Enrich backend audit events with team and entity ids (#12233)
* ✨ Enrich backend audit events with team and entity ids

Backend audit events now carry the data a consumer needs to rebuild
state without joining production tables:

- prepare-rpc-event adds :team-id when the event carries a project or
  file id; commands that already loaded the team set it themselves.
- add-profile-to-team! emits add-team-member on every real insert.
- create-project and create-file emit the new row id.
- clone-template, import-binfile and duplicate-project emit one event
  per created entity.
- both binfile versions return {:file-ids [...]} from import-files!.

Closes #12219

AI-assisted-by: deepseek-v4.1-flash

* ✨ Emit delete and hard-delete audit events with entity data

The event stream must let Nexus rebuild local state without joining
production tables. This completes the enrichment with the delete paths:

- delete-team keeps returning nil but now carries the deleted row
  (id, name, timestamps) in the audit props, like delete-project.
- create-team adds created-at to its audit props.
- delete-profile carries id, email, fullname and deleted-at; its
  props used to be empty because every param is namespaced.
- objects-gc emits hard-delete-team/project/file per purged batch,
  with the deleted ids, only when the audit log is on.
- delete-team-member needs no change: the delete is physical and the
  params already carry team-id and member-id.

The nitrate push-audit-events tests now drop the membership events
emitted by their own profile setup before counting.

Closes #12219

AI-assisted-by: muse-spark-1.3-free

* ✨ Announce cascade deletes and missing ids in audit events

The :delete-object worker task marked teams, projects and files without
leaving any event, so an event-sourced projection kept them alive for
weeks. It now emits one cascade-deleted event per execution with every
affected id, in the same transaction, only with :audit-log on. Rows
purged by objects-gc and snapshot cascades stay silent on purpose.

Also: create-team-with-invitations carries the new team id, and
update-team-member-role names the demoted member on owner transfers.

Closes #12219

AI-assisted-by: muse-spark-1.3-free

* 🐛 Identify the SREPL profile delete in audit events

The SREPL delete-profile! emitted no profile-id and empty props, so the
row could not be attributed by any consumer. It now carries the profile
id in both fields. delete-team! named the wrong cause in its context.

Closes #12219

AI-assisted-by: muse-spark-1.3-free

* 🐛 Identify the SREPL profile restore in audit events

The SREPL restore-profile! emitted no profile-id, so the consumer
keyed on the zero profile instead of the restored one. It now carries
the profile id like a normal profile event, and no consumer special
case is needed.

Closes #12219

AI-assisted-by: muse-spark-1.3-free

* 🐛 Stop attributing SREPL events to the subject profile

profile-id names the actor, and SREPL calls have none: setting the
subject id blamed the user for an admin action and would have leaked
those rows into PostHog as user activity. The subject id stays in
props, where the consumer resolves it, and the event profile stays
zero.

Closes #12219

AI-assisted-by: muse-spark-1.3-free

* ✨ Expose file analysis data for the nexus pipeline

The nexus file-security pipeline needs file content and scheduling
state from the backend. This adds a read-only management method,
`get-file-for-analysis`, returning the semantic projection of one
file (pages, frames, texts, links, interactions; ids as strings;
capped with a truncated flag), plus identity, revn, timestamps and
share state. No prior projection code existed, so the method ships
its own pure `file->projection`.

Audit events now carry what the `:local` consumer schedules from:
revn and modified-at on update-file, name and timestamps on every
create-file emitter, deleted-at on delete-file, and new
create/delete-share-link events with file and link ids.

AI-assisted-by: muse-spark-1.3-contributor

* ♻️ Rename deletion audit events to action names

Event names are actions, not past tense: `cascade-deleted` becomes
`delete-object-with-cascade` with the same shape and transaction.
The three `hard-delete-*` events merge into `hard-delete-objects`:
one event per purged chunk carrying only that chunk's entity key
(`:team-ids`, `:project-ids` or `:file-ids`), because each GC
section purges in its own chunk transactions and a single event
could never share them all. Every submit stays inside its delete
transaction, `triggered-by` stays annotated, and silent paths stay
silent.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Read worker parallelism knobs with plain config keys

The runner wiring looked up `:app.main/worker-*-parallelism`
(namespaced) while the schema and the environment provide plain
keys, so `PENPOT_WORKER_DEFAULT_PARALLELISM` and
`PENPOT_WORKER_WEBHOOK_PARALLELISM` never applied and workers
always ran with the fallback. Read the keys the config holds.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Fix binfile audit test with a real file row

`submit-file-created-event` reads the light columns back from the
row, so the mocked importer must return an existing file id: a
phantom id emits no event and the test saw zero `create-file`
events. Create the file first and use its id. The audit-log memory
now tells both bulk-event helpers apart.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Fix audit event fidelity from review findings

The review of the audit enrichment found data-fidelity gaps; fix
them while the branch is fresh.

Membership events now belong to the actor: the flows where the
owner adds somebody else pass an explicit actor id, the role is
derived from the flags (the table has no role column), and the
event carries the request context like its siblings.

Also: team-id resolution skips read events, duplicate-project
emits the project name and timestamps, delete-profile is
idempotent again, the GC tests assert physical purge, and
get-file-for-analysis leaves an audit trace.

AI-assisted-by: muse-spark-1.3-contributor

* ♻️ Nest cascade ids under :cascade in delete event

The cascade event carried every affected id flat next to the
object identity; nest them under `:cascade` so the payload reads
as header plus affected set. The event is an action like its
siblings, not a command. Drop the `:audit-log` flag re-check:
`submit` already fans out per flag.

AI-assisted-by: muse-spark-1.3-contributor

* 📎 Update copyright name on file header

The company name changed; a single header in this branch still
carried the old one. Align it with the current form.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Fix nil event-id on webhook trigger audit rows

process-event generated the row id inside append-audit-entry, so the
:process-webhook-event task never received it and the trigger row
stored a nil :event-id. Assign the id up-front and share it between
the stored row and the worker params; the telemetry shadow row keeps
its own id to avoid a PK collision. Add regression tests covering
the handoff and the end-to-end trigger row.

AI-assisted-by: muse-spark-1.3-contributor
2026-10-15 13:01:18 +02:00
Filip Sajdak
10955f17e2
✨ Add get-file-tokens RPC to read a file's design tokens (#12172)
Clients that only need a file's design tokens, such as a CI job that
syncs them to a git repository after an update-file webhook, had to
download the whole file with get-file.

get-file-tokens returns the file's tokens as a DTCG document, the same
format the token export produces, or nil when the file has none. Active
themes and sets come from the file's tokens status, so they reflect
what is currently active.

The document is returned as a plain map. A bare TokensLib is not a
collection, so the response middleware would hand it to ring unencoded.

Relates to #7091

Signed-off-by: Filip Sajdak <filip.sajdak@siili.com>
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
Co-authored-by: Andrey Antukh <niwi@niwi.nz>
2026-10-09 20:08:14 +02:00
Filip Sajdak
d7ccc0f3a6
✨ Add get-file-plugin-data RPC to read file plugin data (#10171)
The Plugin API can write arbitrary key/value plugin data onto the file
root, but it could only be read back by downloading and decoding the
whole file. Add a `get-file-plugin-data` read command that returns the
file-level plugin data map, optionally narrowed to a single namespace,
so external tooling can fetch metadata such as issue or commit
references cheaply.

Read-only: reuses `bfc/get-file` and `check-read-permissions!`, and
adds no data-model or change-type changes. Object/page level reads are
intentionally left out of this first command.

Relates to #10169

Signed-off-by: Filip Sajdak <filip.sajdak@siili.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Andrey Antukh <niwi@niwi.nz>
2026-10-09 18:15:07 +02:00
Pablo Alba
76e9c132ef
✨ Send more enriched events to nitrate log (#12210)
* ✨ Send more enriched events to nitrate log

* ✨ Fix mr
2026-10-09 10:58:05 +02:00
Andrey Antukh
d3ca9a0b83
✨ Add unified jobs substrate replacing legacy one-shot task system (#11542)
* 🎉 Move backend jobs to the unified job table substrate

Replace the generic 'task' table with a unified 'job' table as the
durable substrate for all backend jobs, and migrate every producer
and consumer to it. The legacy 'task' table stops receiving writes
but stays dormant (still cleaned by tasks-gc); upload-session
chunking and the other post-branch upstream changes are preserved.

New 0154 migration creates the 'job' table: dispatch and lifecycle
columns (name, queue, priority, scheduled_at, retries, status,
timestamps, props) plus optional user-facing ledger columns
(profile_id, target, progress, error, result, resource_id,
expires_at). No modified_at trigger: the application updates it on
every write for lease and orphan detection. Partial indexes cover
the dispatcher claim query, orphan scanning, and per-profile
listing.

New app.jobs public API: submit!/invoke!, heartbeats and progress
reporting, a precompiled job-def registry (name, schema, handler,
decoder, validator), an ephemeral request/response mode, and a
management API for external workers. The dispatcher claims
new/retry rows (FOR UPDATE SKIP LOCKED) with a plain JSON redis
payload and marks lease-expired running rows as orphaned; the
runner executes per-queue with the same retry conventions.

New jobs GC (daily): deletes expired rows and retains internal
terminal rows per :jobs-retention, touching referenced storage
objects so storage-gc-touched reclaims them. Job resources live in
a dedicated job-resource bucket with a reclaim branch in
storage-gc-touched.

All production call-sites (sendmail, delete-object, demo-purge,
file-gc, offload, webhooks, quotes, nitrate bulk delete,
snapshots, telemetry) and the ten cron tasks now submit through
app.jobs; cron is a pure scheduler and the legacy worker registry,
task-table inserts, and per-namespace handlers are removed.

Includes migrations, dispatcher, runner, jobs, request/response,
jobs GC, management API, and cron test suites.

AI-assisted-by: glm-5.3-flash
AI-assisted-by: muse-spark-1.3-contributor
AI-assisted-by: kimi-k3
AI-assisted-by: mimo-v2.5

* ✨ Address review comments on jobs substrate

Resolves the four open review threads on the jobs PR. Orphaned jobs
are now marked aborted, a system-side terminal state with no retry:
the dispatcher sweep stores the end event with outcome aborted in a
single bulk insert in the same transaction, records the terminal
outcome next to the orphaned counter, and reports an error log;
orphans keep alerting through the log and the counter, with no
user notification.

The aborted state is terminal everywhere: the status CHECK, the
sweep, the lifecycle writers, the jobs GC retention, the backlog
gauges and the user status mapping, with tests covering each
transition. The claim comment, the 2.20 doc version and the legacy
task table notes are corrected, and objects-gc heartbeats per chunk.

AI-assisted-by: muse-spark-1.3-contributor
2026-10-06 07:33:12 +02:00
Elena Torró
aed4e12c63
⚡ Avoid decoding all pages when building file thumbnail data (#12054) 2026-10-05 11:34:10 +02:00
Yuito Akatsuki (Tani Yutaka)
aeeca59b1a 🐛 Fix shared library deletion with storage-backed file data
Inject the storage component into the delete-object task handler and
validate it on initialization, so library absorption no longer fails with
"expected valid storage to be provided" for storage-backed file data.

Also strengthen the regression test: it now creates a real component in
the shared library and an instance in the referencing file, and verifies
the component is absorbed after deletion instead of only checking the
revision bump. The test disables the fdata/pointer-map feature to stay
focused on the storage fix.

Signed-off-by: Yuito Akatsuki (Tani Yutaka) <yuito@yuito-it.jp>
AI-assisted-by: deepseek-v4.1-flash
2026-09-24 18:17:07 +02:00
Andrey Antukh
2f679eaa0e
💥 Remove client-provided id from creation RPC commands (#11784)
The seven creation commands no longer accept an optional client
id: create-file, create-project, create-team,
create-team-with-invitations, upload-file-media-object,
create-file-media-object-from-url and assemble-file-media-object.
The server always generates the identifier; a sent id is ignored.

Malli maps are open and the RPC layer never strips unknown params,
so the handlers that would still honor an id (create-file,
create-project) now drop it explicitly. Internal callers that pass
remapped ids (project duplicate, binfile import) keep working.

Closes #11783

AI-assisted-by: muse-spark-1.3-contributor
2026-09-22 15:51:50 +02:00
Andrey Antukh
117c8db0bb Merge remote-tracking branch 'origin/staging' into develop 2026-09-22 10:24:30 +02:00
Andrey Antukh
e05747b546
✨ Restrict optional RPC ids to user-provided UUIDs (#11777)
* ✨ Restrict optional RPC ids to user-provided UUIDs

Add ::sm/user-provided-uuid, backed by a version and variant
aware regex that only accepts v4, v7 and v8 instances. Use it
for the optional :id of the creation RPC commands so reserved
versions such as v3 are rejected at validation time. Reads
such as get-team keep the lax ::sm/uuid. Cover the predicate
and the schema on both JVM and JS runtimes.

AI-assisted-by: muse-spark-1.3-contributor

* ✨ Cover id version restriction at the RPC boundary

Add backend regression tests proving the seven creation commands
reject reserved-version ids (v3) with :params-validation and
accept v4 ids (plus v7/v8 on create-team) through the real
decode and validate path. Also drop two duplicated assertions
and document the version and variant of every fixture UUID in
user-provided-test.

AI-assisted-by: muse-spark-1.3-contributor
2026-09-22 09:45:50 +02:00
Andrey Antukh
81c3b3cd56
📎 Update copyright name on file header (#11346) 2026-08-25 11:55:10 +02:00
Andrey Antukh
52573be074
🐛 Enforce share-link page scope in get-page and get-file-fragment RPC commands (#11284)
The get-page RPC command did not validate that the requested page-id was within the share-link's authorized :pages set, allowing share-link holders to read out-of-scope pages. The get-file-fragment command had the same issue.

This fix adds page scope validation to get-page, rejecting requests for pages not in the share-link's :pages set with a :not-found error. For get-file-fragment, share-link access is denied entirely as fragments lack direct page-id mapping.

The fix aligns these commands with the existing correct behavior in get-view-only-bundle, which already filters pages based on share-link scope.

Closes #11281

AI-assisted-by: qwen3.7-plus
2026-08-25 09:37:42 +02:00
Andrey Antukh
209aea8365
🐛 Add proper ownership check on managing/deleting shared link on a file (#11290)
* 🐛 Add ownership check to share-link deletion

The delete-share-link RPC command only verified file-level edit
permission but did not check if the caller owned the share-link.
This allowed any file editor to delete share-links created by
other users, disrupting collaborative workflows.

The fix adds an ownership check that allows deletion only by:
- The share-link creator (owner-id matches profile-id)
- File admins (is-admin permission)
- File owners (is-owner permission)

Implemented using TDD:
- RED: Test demonstrates IDOR vulnerability (editor can delete)
- GREEN: Ownership check prevents unauthorized deletion
- All existing tests continue to pass

Closes #11289

AI-assisted-by: qwen3.7-plus

* 🐛 Add test coverage for share-link deletion escape hatches

Address code review feedback for PR #11290:

- Add test for editor deleting their own share-link
- Add test for admin deleting editor's share-link
- Add test for owner deleting editor's share-link
- Remove redundant :is-owner check (already included in :is-admin)
- Add clarifying comment about :is-admin including :is-owner

Closes #11289

AI-assisted-by: qwen3.7-plus
2026-08-19 18:26:35 +02:00
Andrey Antukh
399b00b86d
🐛 Add permission checks to WebSocket subscription handlers (#11054)
* 🐛 Add permission checks to WebSocket subscription handlers

Check file and team read permissions before allowing WebSocket
subscriptions to prevent resource enumeration via presence
notifications.

AI-assisted-by: mimo-v2.5-pro

* 🐛 Fix random backend test failure
2026-08-07 11:24:24 +02:00
Andrey Antukh
5906312dff
🐛 Normalize error response on duplicate file ID (#11050)
Capture unique constraint violation in insert-file! and return
generic :not-found error instead of propagating raw PostgreSQL
exception, preventing file existence oracle.

AI-assisted-by: mimo-v2.5-pro
2026-08-05 17:52:14 +02:00
Andrey Antukh
fb07273897
🐛 Validate library belongs to same team in link/unlink/sync handlers (#11016)
Add check-library-team-ownership! helper that verifies both the file
and library share the same team before creating or modifying library
relations. This prevents cross-team library injection where a user
with edit permissions on files in different teams could link them
across team boundaries.

Applied to link-file-to-library, unlink-file-from-library, and
update-file-library-sync-status handlers.

AI-assisted-by: mimo-v2.5
2026-08-05 17:40:04 +02:00
Andrey Antukh
6c2b61e1ad
🐛 Fix several issues in RPC command handlers (#10670)
* 🐛 Fix several issues in RPC command handlers

- Reject circular library references in link-file-to-library
- Add explicit team permission check in search-files
- Constrain search-term max length to 250 chars
- Include :deleted-at in file ETag for COND caching
- Move storage I/O outside DB transaction in create-file-thumbnail

AI-assisted-by: deepseek-v4-pro

* 📎 Check perms before circular link checks

* 🐛 Handle circular library reference error

Catch :circular-library-reference error from backend when linking
files to libraries. Show user-friendly toast notification instead of
propagating unhandled error. Add English and Spanish translations.

AI-assisted-by: qwen3.7-plus
2026-07-28 10:59:16 +02:00
Andrey Antukh
0ac092d177
🐛 Make set-file-shared idempotent to fix race condition with optimistic updates (#10093) 2026-06-10 10:58:50 +02:00
Andrey Antukh
c5de4c27b0 Merge remote-tracking branch 'origin/main' into staging 2026-06-01 12:57:39 +02:00
Yamila Moreno
ddba2ffa75
📎 Update Kaleidos Copyright (#9929) 2026-05-29 11:24:58 +02:00
Pablo Alba
fffafdab93
🐛 Fix library updates reappear after file is reloaded (#9563)
* 🐛 Fix library updates reappear after file is reloaded

Summary
Migrate synced_at timestamps to a standalone file_library_sync table to ensure sync state is tracked for both direct and transitive libraries.

Problem
Transitive libraries (libraries imported by other libraries) are not stored as direct rows in file_library_rel. Because the system previously coupled synced_at directly to the file_library_rel schema, transitive libraries lacked a persistent location for their sync timestamps. This caused sync states to be lost or incorrectly reported for nested dependencies.

Changes
Schema Migration: Created file_library_sync and migrated existing synced_at values from file_library_rel.

Decoupling: Removed tight Foreign Key coupling to allow sync rows to exist independently of specific relationship records.

Persistent Writes: Added upsert-file-library-sync! helper. Updated all import, duplication, and RPC write paths (v1/v2/v3 importers, link-file-library) to ensure every write persists a sync row.

Unified Reads: Updated both direct and recursive/transitive library queries to fetch synced_at from the new table.

Testing: Added regression tests to verify that sync rows are correctly created/updated even when a transitive relation is absent in file_library_rel.

Impact
This fix ensures that the system accurately records and retrieves sync states for the entire library dependency tree, resolving the bug where nested libraries appeared out of sync.

* ✨ MR review
2026-05-13 11:29:05 +02:00
Edwin Rivera
2579527e64
🎉 Add get-file-stats RPC command (#9074)
* 🎉 Add get-file-stats RPC command

Introduce a new lightweight RPC query that returns aggregate statistics
for a single file: page count, shape counts by type, component/color/
typography counts, and inbound and outbound library reference counts.
Mirrors the existing get-file-summary permission and decoding pattern.

Useful for plugin authors enforcing per-file budgets, the
@penpot/library npm SDK, and future admin dashboards. Purely additive
— no migrations, no UI, no breaking changes.

Signed-off-by: edwin-rivera-dev <bytelogic772@gmail.com>

* 🐛 Bind *load-fn* around file data walk in get-file-stats

The binding previously wrapped only  — a plain key
lookup that does not realize any pointers — so by the time
 walked  and accessed  on
each page,  was unbound and every PointerMap
dereference threw , failing the three new  tests.

Move  inside the  form so the walk runs
with  available, matching the existing pattern used in
.

Signed-off-by: Edwin Rivera <bytelogic772@gmail.com>

---------

Signed-off-by: edwin-rivera-dev <bytelogic772@gmail.com>
Signed-off-by: Edwin Rivera <bytelogic772@gmail.com>
2026-04-22 12:49:39 +02:00
Andrey Antukh
97d234a566 ✨ Add 2h min-age threshold to storage/gc_touched task
Skip storage objects touched less than 2 hours ago, matching the pattern
used by upload-session-gc. Update all affected tests to advance the clock
past the threshold using ct/*clock* bindings.
2026-04-22 08:48:04 +02:00
Andrey Antukh
974beca12d ✨ Add 2h min-age threshold to storage/gc_touched task
Skip storage objects touched less than 2 hours ago, matching the pattern
used by upload-session-gc. Update all affected tests to advance the clock
past the threshold using ct/*clock* bindings.
2026-04-16 19:43:57 +02:00
Andrey Antukh
77955d7f91 ✨ Add several redundant checks for library-id on file rpc methods 2026-03-09 10:01:29 +01:00
Andrey Antukh
0ab126748f
💄 Add format rule for code comments (#8211)
* 💄 Add format rule for code comments

* ⬆️ Update linter and formatter on devenv
2026-01-27 15:07:18 +01:00
Andrey Antukh
bcaf76d055 ✨ Make the virtual clock by profile and not global
And make it affect only RPC/HTTP requests and not worker
tasks.
2026-01-09 12:58:44 +01:00
Andrey Antukh
7b5817f407
♻️ Make several adjustments to the dashboard deleted page (#7999)
* ♻️ Make several sustantial adjustments to the dashboard deleted page

* 📎 Add PR feedback changes
2025-12-30 09:52:29 +01:00
Alejandro Alonso
4f54469629
Merge pull request #7747 from penpot/niwinz-develop-storage-changes
✨ Make the binfile exportation process more reliable
2025-11-20 07:58:57 +01:00
Andrey Antukh
51e54a6bad
🐛 Fix incorrect project restoration on restoring file (#7778) 2025-11-19 18:24:24 +01:00
Andrey Antukh
94719eebf8 ♻️ Make storage and other objects deletion task vclock aware
This simplifes the mental model on how it works and simplifies testing
of the related code.

This also normalizes storage object deletion in the same way as the
rest of objects in penpot (now future deletion date on storage object
also means storage object to be deleted).
2025-11-19 17:28:55 +01:00
Andrey Antukh
83763b46ce ✨ Add RPC methods for manage deleted files
This includes: get already deletedf files, restore deleted files
and permanently delete files marked for deletion.
2025-10-31 16:07:23 +01:00
Andrey Antukh
76e610dd06 🔥 Remove duplicated functions from file tests namespace 2025-10-31 16:07:23 +01:00
Alejandro Alonso
2b7bd8fa5c 🐛 Fix deleted files are accesible 2025-10-13 12:24:05 +02:00
Andrey Antukh
5717708b56 ♻️ Refactor file storage
Make it more scallable and make it easily extensible
2025-10-13 12:24:05 +02:00
Andrey Antukh
283eb0419c ♻️ Refactor time related namespaces
Mainly removes the custom app.util.time namespace
from frontend and backend and normalize all to use
the app.common.time namespace
2025-08-01 11:20:01 +02:00
Alejandro Alonso
055ee27be0 🐛 Add migrations to fix colors 2025-06-04 14:01:48 +02:00
Andrey Antukh
a706907b26 ♻️ Normalize logical deletion to future dates
Instead of managing the ...
2025-05-20 22:25:57 +02:00
Andrey Antukh
29d23577d2 🎉 Add .penpot (binfile-v3) support for library 2025-05-20 13:06:07 +02:00
Andrey Antukh
541ed6282f ✨ Normalize file retrieving and persistence on binfile, file-gc and helpers 2025-01-27 12:38:10 +01:00
Andrey Antukh
471699960f
🐛 Update media references after instantiation of a component (#5652)
🐛 Update media references after instantiation of a component
2025-01-27 11:58:13 +01:00