save-loads-only-edited-page counted fragment loads only. Assert as well
that the one load is the edited page's fragment, that the untouched
pages keep their fragment ids, that the rewritten page is stored as an
objects map, and that the file reloads equal to the edit applied to the
file before the save.
Under the old enable-objects-map, which wrapped every page, the test
fails with three loads instead of one. Replaying 1,000 recorded saves
of a 49-page file on a pointer-map copy found no other load the save
path adds beyond the edited page and the components map.
AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
The copies summary now follows the same cascade the post-merge sync
does: when a copy that follows a changed component sits inside another
component's main, that outer component's copies are listed too, marked
with the component they follow through. The summary is computed only
when a caller asks for it (the compare view and the conflict
responses), so the ahead and behind counts skip it.
AI-assisted-by: claude-opus-5-5
The merge summary now says, for each component whose main the incoming
side changed, which copies will follow it after the integration and
which keep an override, per sync group and token slot. Each copy entry
lists what the copy rule settled, with the side and the reason, and
what the user still decides. A copy the rule settles entirely stays in
the summary as an informational entry instead of disappearing, and
applies nothing. A conflict on a component's main counts the copies
its resolution also decides.
AI-assisted-by: claude-opus-5-5
A copy's applied tokens were one attribute outside every sync group,
so when both sides applied different tokens to a component, every
untouched copy showed a conflict of its own.
The copy rule now covers applied tokens one slot at a time, as
Penpot's own component sync does: a slot follows the component unless
the sync group of the attribute it drives is overridden. A token
applied to a copy by hand on one side wins, and only a slot both sides
applied differently by hand stays a conflict. The side the user picks
for that conflict decides only the open slots.
AI-assisted-by: claude-opus-5-5
A copy whose override the base holds can be reset on one side while
the other side leaves it alone. The tests check that the reset reaches
the other file in both directions: the copy loses its override, takes
the component's value, and follows a later component edit. The merge
direction failed before copies followed their component.
AI-assisted-by: claude-opus-5-5
A merge or an update from main compared a component's main and each
copy as separate shapes. A copy added on the other side kept the old
look, and when both sides changed the component the dialog showed one
conflict per copy, so a copy could end up disagreeing with its
component without being marked as overridden.
Copies now follow the merged component, as they do when the component
is edited. Per copy attribute: if no side overrode it, it raises no
conflict and follows the component; if one side overrode it, that
side wins; only overrides on both sides that differ stay a conflict.
After the merge or update applies, the copies of every local component
whose main changed are synced, nested copies included, and the sync
changes are validated and recorded with the rest. A clean merge of a
copy no longer marks it as overridden on the target.
AI-assisted-by: claude-opus-5-5
A branch reads the images it inherited through main's media rows. When
materialize turned it into an ordinary file, the file kept pointing at
those rows and released the base that kept main's file-gc away from
them, so an image main later dropped, or main's deletion, broke the
materialized file.
Materialize now copies every media row the document uses that another
file owns into the branch file under a fresh id, and re-points the
document at the copies before it persists it.
AI-assisted-by: claude-opus-5-5
A branch derives its document from main's base snapshot, and the derive
never re-pointed main's ids: :data :id stayed main's, and so did every
inherited reference to the file itself (component instances, library
colour and typography refs). The branch workspace then took its own
components for an external library it never loads and read its own
tokens through main: validation reported :component-main-external and
:component-not-found on a fresh branch, the local component sync
skipped inherited copies, an update from main refused once the branch
held its own copy of a base component, and materialize detached the
main instance of every inherited component.
The derive now sets the branch's id on the document before the replay,
so a change or migration comparing a reference with the document's id
sees the branch, and re-routes main's id to the branch's after it, so
op log rows written before this fix are covered too. The base keeps
main's ids because it shares main's fragments; comparisons already map
the branch's id back to main's.
relink-refs rebuilds a collection only when something inside it
changed (median 803 ms instead of 1506 ms on a 90,612-shape file), and
it now relinks the :ref-file of shadow and grid colours, the key
migration 0015 gave them, which it had missed since.
AI-assisted-by: claude-opus-5-5
The merge into iop-vc-preview kept the "Archive this branch" option of
the later merge dialog. Juan's decision in penpot#12596 holds instead: a
merged branch is always deleted, never archived. The merge and conflict
dialogs lose the option, merge-file-branch no longer takes keep-branch,
merged branches leave the branch list and count again, and the tests
for kept merged branches go, as in the pull request. The merge comment
stays. The warning about left-out changes no longer suggests archiving.
AI-assisted-by: claude-opus-5-5
Takes penpot#12596: branch access by team role alone (no file role on
a branch, the owner role on materialise, branches move with their
source file, migrations 0807 and 0808), the merge record with its
comment (migration 0806) and the reviews dialog, the new git icons, and
the workspace tidy-up.
Where Juan's later redesign of the compare, conflict and merge dialogs
(c68daed816, e4e237346e) already changed the same code, the redesign
wins: the old dialogs stay in branch_review/, the redrawn compare
icon stays, and the merge keeps its "Archive this branch" option. So
merge-file-branch accepts both `comment` and `keep-branch`, a kept
merged branch is listed and counted again, and the keep-branch tests
the PR removed are restored. The merge dialog asks for the comment
beside the archive option, and the conflicts dialog sends it on.
AI-assisted-by: claude-opus-5-5
The save in file-gc-keeps-the-fragments-a-base-names took an automatic
snapshot of pre-edit main, which names the same fragments as the branch
base, so the test passed even with bases left out of liveness. It now
drops every other snapshot before the pass.
AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
The page order lives in the document root, not in a page fragment, so
a reorder on main moves no fragment, and a compare bounded to the moved
pages compared the order of no page at all. moved-pages now leaves the
whole-file comparison in place when main orders the pages it shares
with the base differently from the base.
AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
A page whose fragment id main shares with the branch's base holds the
same document on both sides, because a fragment row never changes after
it is written. The derive now hands back the fragment ids of the base
pages, and a comparison loads only the pages of main whose id differs:
the base's realized page stands in for every other page. The direction
that reports main's changes (the compare's main->branch and the reverse
pass of the diff counts) is bounded to those pages with :only-pages.
A base that was migrated, or that holds no pointers, gives no ids and
keeps the whole-file comparison. On a copy of the scaling corpus rung L7
(69 pages, 47,514 objects) the main->branch compare drops from 5,690 ms
to 1,965 ms and the branch->main compare from 3,254 ms to 1,981 ms; what
remains is the branch derive, which still realizes the whole base.
The compare's size gate now runs after the derive, because main's
document is assembled from the base. The depth gate still runs before
it, and the base passed the size gate when it was written.
AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
A branch base was a fully realized copy of main. With fdata/pointer-map,
creation now stores main's fragment pointers: the base costs one pointer
per page and shares main's fragment rows.
Fragment rows are keyed by (file_id, id), so the base readers (the branch
derive and get-base-data) realize the base with the source file id
through realize-branch-base. The branch file's own loader finds none of
main's fragments.
Every writer of a snapshot that names main's fragments reads main's row
FOR SHARE, so it serializes with file-gc, which takes the row FOR UPDATE
and marks every fragment no committed snapshot names. Creation, the
pre-merge snapshot, and the base repositioned by an update from main
follow that rule. The merge and the update put main's stored pointer
back on every page they left unchanged, so the merge rewrites only the
pages it changed and a repositioned base shares main's fragments.
The hand-written pmap bindings move to fdata/with-tracked-pointers.
AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
Brings in the fragmented file data stabilization across snapshots and
GC (penpot#12410) and the per-library pointer loader fix. Four files
conflicted.
file_gc.clj `process-file!` keeps the branch's `:is-branch` skip and
its warning; ordinary files run the new body under `with-pointers`
(snapshot liveness, reattach of unchanged fragments, fragment
cleaning).
files_snapshot.clj `restore-file-snapshot` takes the new backup
transaction, which reads the file FOR SHARE and gives the guard revn.
`fsnap/check-restorable!` now runs on that read, before the backup
snapshot is created, so a branch file is still refused without a
committed backup.
binfile_test.clj and rpc_file_snapshot_test.clj keep the tests both
sides appended.
AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
Merges develop at f6f9df9a1a. Twelve files conflicted; each keeps
both intents.
files_update.clj: develop's per-stage validation timings wrap the
branch save path. `update-file` keeps the branch's early timer and
passes it to `update-file*`, inside develop's try/catch timing log.
`process-changes-and-validate` times the changes application and
still returns the file, the fixed-up change vector and the media
index. The audit props carry develop's revn and modified-at plus the
branch props. The timing test destructures the new return shape.
v3.clj keeps the branch-file overwrite guard and takes develop's
`configure-import-transaction`. migrations.clj puts develop's 0155
to 0157 before the branch's 0800 to 0805, so the branch's stay the
tail of the vector.
errors.cljs checks the merged and archived branch refusals before
develop's retrying case. left_header.cljs keeps the branch wrapper,
the review badge and the branch bar on develop's markup: the status
dot without icons and the DS library icon on the shared badge.
palette.scss and assets.scss keep the branch clearance terms on
develop's logical properties. The test runners, requires and
deftests are unions.
AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
Fragment rows are keyed by file id, and every caller binds the
fragment loader of the file it works on. get-resolved-file-libraries
returned libraries with their pages still as pointers, so the first
read of a library page under fdata/pointer-map asked the consuming
file for the fragment and failed with :fragment-not-found. That
broke migrations and file validation for every file using a
pointer-mapped library.
The weak-map loader now realizes each library's pointers with the
library's own id. The test fails without the change.
AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
Snapshots, automatic and user, now store pointer-based rows that share
fragments with the live file instead of duplicating page content.
`get-file-snapshot` resolves those pointers on the server so version
previews keep rendering plain data, and `restore!` reuses the shared
rows. Export still materializes (the package carries no fragments) and
import regenerates them.
`file-gc` treats the main file and every unpurged snapshot as liveness
roots, reads the file unrealized, and reattaches the original pointer
wherever the cleaned value is unchanged, so a pass over an untouched
file writes no fragments and a pass over an edited file keeps the ids
of the pages it did not touch. Snapshot creation and the restore
backup take the file row `FOR SHARE`, which serializes them with the
pass and closes the race that let a new snapshot reference fragments
the pass was about to mark.
All backend pointer bindings go through the `fdata/with-pointers`,
`with-tracked`, `with-tracked-pointers` and `with-untracked` macros.
The loader reads fragments a previous pass marked until `objects-gc`
purges them, and the frontend fails loudly with retry instead of
dropping an unresolved fragment.
Review fixes in this branch: the automatic snapshot attaches before
pending migrations, `realize-pointers` is the single verb,
`reattach-unchanged` takes the identity fast path, the restore guard
takes its revn from the backup read, and `PointerMap.withMeta`
generates a fresh id when metadata changes.
Tests cover both snapshot writers (red without the lock), snapshot
sharing and pin release through the real delete and purge paths,
reattach against orphan marking, preview after offload, the pointer
binding contracts and the with-meta id change.
Closes#12410
AI-assisted-by: muse-spark-1.3-contributor
AI-assisted-by: deepseek-v4.1-flash
♻️ Pin no-churn behavior for untouched pages
Pin the behavior the churn verification measured: a save keeps the
fragment id of every page it does not touch, with a page of eleven keys
(PersistentHashMap), a 46-shape objects map and a nine-page index (hash
map). Also pin the mixed case: a data-level objects-map flip rewrites
the old pointer pages once and the next save keeps the new ids.
The memory records the identity rule behind it: re-assoc with the same
value returns the same map in array and hash maps alike.
Closes#12410
AI-assisted-by: deepseek-v4.1-flash
⚡ Stop loading untouched pages when enabling objects-map
`enable-objects-map` wrapped every page on every save, and reading a
page's objects forced its PointerMap to load the fragment. A save of
one page in a seven-page file therefore made seven fragment reads.
Skip unloaded pointers: their fragment was already wrapped by the
encode-file that wrote it, so loading it only to wrap it again is a
no-op. The save now loads only the pages it edits. A page with a plain
fragment (a data-level feature flip) keeps its plain fragment until it
is first loaded or edited; a migration that enables the feature must
rewrite the data itself.
Cover the change with load-count tests for save and restore, update the
flip test to the new contract, and drop the stale restore comment.
Closes#12410
AI-assisted-by: deepseek-v4.1-flash
⚡ Keep page fragment ids through pending migrations
Applying a pending migration on read realized the file, ran the
migration and persisted it again; the persist wrapped every plain page
in a new PointerMap, so every page got a new fragment id even when the
migration did not change it. That leaves the fragment cache cold and
rewrites every fragment on the first read after a deploy.
Capture the original pointers before realizing, reattach the unchanged
positions after migrating (same value comparison file-gc already uses),
persist the reattached file and return the realized one: the read
contract is plain data. Read-only connections skip the persist and the
reattach. A migration that adds objects-map skips the reattach so the
pages get wrapped.
Move the three helpers to fdata so file-gc and the migration path share
them, and cover the path with a real 0031 migration test: the unchanged
page keeps its id, the changed page gets a new one, the caller gets
realized data and a second read does not migrate again.
Closes#12410
AI-assisted-by: deepseek-v4.1-flash
A branch must be open to every member of the team of its source file,
by role: owners, admins and editors open and edit it, viewers read it.
Two things got in the way.
Branch creation granted its creator the owner role on the branch file
(a file_profile_rel row), so a team editor became owner and admin of
their own branch. create-file-branch grants no file role any more: a
branch takes its permissions from the team and project roles alone.
materialize-file-branch, which turns a branch into an ordinary file,
now grants the owner role to the profile that runs it, as creating a
file does. Migration 0807 deletes the file roles of existing branches;
materialised files are no longer branches and keep theirs.
move-files moved a file but left its branches in the old project. The
permissions of a branch come from the project it lives in, so after a
move to another team its members got not-found and the workspace asked
them to request access. move-files now moves the branches of every
moved file with it, deleted branch rows included since their files
live on until the GC, and the broken library links cleanup already
covers them. It refuses to move a branch file on its own
(:cant-move-a-branch), which would part it from its source. Migration
0808 puts every branch back in the project of its source file and
drops the links of those branches to libraries of their old team.
Assigning a reviewer to a design review grants no access: the email
and the dashboard notification only let the reviewer know.
Tests cover a branch created with no file role, the creator keeping
editor rights, another editor saving to the branch, a viewer reading
it, the owner role granted on materialise, a lone branch refused by
move-files and an editor of the new team opening a moved branch.
* 🐛 Refuse saves to a branch kept after its merge
A branch kept after its merge stays an ordinary file with status
"merged", so every save passed the edition check and appended ops to
an op log that no command integrates any more. Refuse such saves with
a :branch-merged validation error instead of stranding them. An
archived branch still takes saves: restoring it brings them back.
AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
* ✨ Tell merged and archived branches apart
The workspace treated "merged" and "archived" as one state: an
archived branch read "Merged branch" and lost every action, and a
merged row offered Restore, which the backend always refused.
Each status now has its own presentation. A kept merged branch keeps
"Merged branch" and View in main, and opens read-only: the workspace
permissions drop :can-edit while it is the open file, the way a
viewer gets a read-only workspace. An archived branch says so in the
banner and offers Compare and Restore. The header notes each status,
the switcher lists merged and archived branches in two collapsible
groups, the all-branches dialog gets a Merged tab, and the branch info
dialog offers Restore and Compare for an archived branch.
AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
* ✨ Refuse saves to an archived branch
An archived branch is now read-only, like a merged one: the user
restores it, or turns it into an ordinary file, to keep editing.
persist-branch-file! refuses its saves with a :branch-archived
validation error, distinct from :branch-merged so the client can say
which way out applies. A restored branch takes saves again, and so
does an archived branch turned into an ordinary file.
AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
* ✨ Open archived branches read-only with their ways out
An archived branch now opens read-only, like a merged one. Its banner,
its branch info dialog and its row in the all-branches dialog offer
the two ways back to editing, Restore and Turn into an ordinary file,
next to Compare. The header and switcher note reads "Archived, read
only".
A save the server refuses because the branch is merged or archived
now shows a warning that names the way out, instead of the generic
autosave error with a report link.
AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
---------
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
* ✨ Add a branch count query for the workspace header
The header shows its branch row only when the file has branches, and
it learnt that from get-file-branches, which compares every diverged
branch against main. get-file-branch-count answers the same question
with one indexed count: archived and merged branches included, counted
from main when the file is itself a branch, with no comparison and no
climit, like the branches_count subquery of the dashboard listings.
The listing keeps its climit, and :get-file-branches/global now sets
the 60 s timeout that the header of climit.edn promises, so a waiter
behind six cold listings gets an error instead of waiting without
bound. Its rationale comment names the callers it serves now.
AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
* ✨ List branches only when the switcher opens
Opening a file ran the counted branch listing for every user, because
the header decided whether to show its branch row from that listing.
A cold listing runs a three-way comparison per diverged branch, and
one save to main makes every branch of the file a cache miss again.
The header now loads the branch count and the branch context with the
file. The listing loads when the branch switcher or the all-branches
dialog opens, and both show "Loading branches…" until it arrives
instead of a list from an earlier opening.
A failed listing used to read as an empty list, which on main hid the
branch row and sent the shortcut to the create dialog. It now ends in
an error state: the switcher and the dialog say "Couldn't load
branches" with a Retry action, and the row stays, since its visibility
comes from the count. A successful listing refreshes the count, so
deleting the last branch from the list still hides the row.
Creating a pull request from the branch banner looked the branch up
in the listing to decide whether to save pending edits first. Without
the listing loaded at open it would skip the save, so it now reads the
branch context of the open file.
AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
* ✨ Show the branch row when the branch count fails
A failed get-file-branch-count left no count, so on main the header
hid its branch row and the shortcut opened the create dialog on a file
that may have branches. The failure is now kept as an error state
under :workspace-branch-count, and has-branches? reads it as true.
The row then shows the Main chip and, where the status note goes,
"Couldn't load branches" in the warning colour. Opening the switcher
loads the listing through the existing loading and Retry path, and a
successful listing settles the count, which replaces the note.
AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
* 📚 Name the callers of the branch listing limit again
The merge of iop-vc-preview took its climit.edn, whose comment on
:get-file-branches/global still described a panel open. Restore the
comment this branch wrote: the listing runs when the switcher, the
all-branches dialog or the dashboard popover opens, and a file open
only counts branches.
AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
* 🐛 Mock the branch count in the library copy tests
The header now shows its branch row from get-file-branch-count, not
from the listing, so the library copy tests mock the count as well.
AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
---------
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
A branch whose only change was file-level plugin data or the base font
size merged with a success response, and the change was lost. The merge
engine compared colours, typographies, components, media, pages, and
tokens only, so such a branch produced an empty summary and the merge
took the empty-changes shortcut.
The engine now classifies two more kinds three-way: file plugin data,
flattened to namespace and key as page plugin data already is, and the
file options, where an unset base font size reads as the default. Both
kinds count towards ahead, behind, and conflicts, appear in the compare
dialog, and translate into set-plugin-data and set-base-font-size
changes for merge and update from main. Both sides changing the same
key to different values is a modify-modify conflict.
AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
* 🐛 Give the branch concurrency limits a timeout
Every global entry for branch and pull-request commands lacked a
:timeout, so a saturated limiter parked its waiters forever and
climit-test/example-config-bounds-every-wait failed on seven ids.
Each entry now waits at most 60 s, the value of the other slow
operations. The size gates in files_branch.clj refuse files whose
operation would exceed 30 s, so the wait covers one in-flight
operation with as much again in margin.
AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
* 🐛 Make the quota racer a member of the file team
File access requires membership of the file's team since the
permission check started joining team_profile_rel. The racer in
create-checks-the-branch-quota-inside-the-lock held only a file role,
so its create failed with :object-not-found and the test never raced.
The racer now joins the file's team as an editor, so both creates
again compete for the last branch slot.
AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
---------
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
Take the file row with FOR UPDATE SKIP LOCKED when the offload job
starts and skip the run when the row is locked or gone, so the job
never runs while the file is being edited. Every live hot-storage
row of the file moves to cold storage inside one all-or-nothing
transaction, and the handler answers a summary map with the
offloaded rows, bytes and per-type stats.
Add bounded metrics for the job: rows and bytes counters labeled by
row type on successful runs only, and a timing histogram with the
ok, skip and error outcomes. The job definition now requires
metrics, the recorders live in app.jobs.metrics, and the existing
ephemeral job timing histogram pins its buckets.
Cover the job with a new test namespace: the happy path, the locked
row, missing and deleted files, the rows and bytes summary with its
no-op rerun, the whole-file abort on storage failure, the metric
definitions and emissions, and the submit-time params validation.
An end-to-end suite reads, snapshots, writes, exports, imports and
deletes offloaded files; concurrency tests pin the stale-revn skip,
the blocked-edit retry and the reclaimable orphans of a partial
failure. The file-gc scheduler test pins one job per eligible file,
and the scheduler private helper drops its bang.
Document the lock invariant, the deliberate no-pacing decision and
the metric label sets in the backend storage memory and the
configuration guide.
Related to #12398.
AI-assisted-by: muse-spark-1.3-contributor
The cron tick kept its own active-job count beside the shared
submit option. Delegate the no-overlap check to
::jobs/skip-if-active so both paths share one queue-qualified
check served by the partial index.
A tick now skips only when a live job sits on the :cron queue,
and a skip records both the cron counter and the generic skipped
counter.
Closes#12411
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Fix legacy path migrations on content without coordinates
PRE still returned 500 on get-team-shared-files after #12357, now with
"expected rect instance" in legacy-6. When a path has no points, legacy-6
rebuilds its rect from the content, but content whose points are not
numbers (NaN or infinite coordinates, segments without params, only a
close path) gives no rect either, and rect->points rejects nil.
Fall back to the empty rect in legacy-6. A path with segments missing
coordinates then failed later, in 0003-convert-path-content-v2, since the
binary path data can not store them; drop those segments there.
Relates to #12343
AI-assisted-by: claude-opus-5-5
* 🐛 Skip shared files that fail to load in get-team-shared-files
get-team-shared-files loads and migrates every shared file of the team,
so one file whose migration fails made the whole request return 500 and
the "Add library" modal could not open.
Catch the error per file, log it with the file and team ids, and leave
that file out of the list. The other libraries still show, and the error
report names the broken file.
Closes#12343
AI-assisted-by: claude-opus-5-5
`app.redis/del` logged `:key key`, but `key` is not bound in that
`let`: it resolved to `clojure.core/key`, so the line carried
`#object[clojure.core$key ...]` instead of the keys that failed to be
deleted. Log `:keys keys`, which is always the normalized vector, so a
single key and a collection report the same shape.
The tests pin the line: the keys, the hint, the error level, the logger
and the timeout as cause, plus the `nil` the caller gets.
The prod-infra memory now notes that client-side redis timeouts mean the
JVM is paused, not that the server is slow.
AI-assisted-by: deepseek-v4.1-flash
* ♻️ Rename audit-log archivers to consumers
The pending-queue owners are no longer only external archives: a
local, non-archiving drain is about to join them, so "archiver" no
longer fits. Rename the column, identifiers, jobs, metrics and docs
to "consumer" in one release; migration 0157 never reached
production, so a single RENAME COLUMN is enough, with no backfill.
AI-assisted-by: muse-spark-1.3-contributor
* ♻️ Use base Nexus URI with appended ingest path
Mirror the admin-console setup: PENPOT_NEXUS_URI holds the base
URI and the consumer appends api/audit itself, instead of a full
endpoint URI. Keeps the explicit :uri job param as override and the
:task-not-configured guard when no base is set.
AI-assisted-by: muse-spark-1.3-contributor
* ♻️ Archive audit log from a pending queue with two jobs
Replace the mark-and-sweep audit archive with a pending queue
(audit_log_pending, migration 0157): every row is born pending
for the archivers enabled at insert time, in the same
transaction, so a rollback takes both. Two jobs share one
namespace, one queue each: :audit-log-archive-nexus and
:audit-log-archive-nitrate, each with its own closed params
schema, deps and every-5m cron entry gated by its send flag.
Each job drains its queue oldest-first in 128-row chunks (one
transaction per chunk, same total order, no SKIP LOCKED so
arrival stays chronological) and deletes its pendings on
completion; whoever deletes the last pending deletes the row.
Housekeeping lives in both jobs: reap of disabled pendings
plus sweep of orphans, both idempotent. A refused chunk raises
:archive-chunk-refused and aborts the job; the cron retries.
No separate GC task.
Metrics with bounded labels only: archived/failed/skipped per
destination, discarded per reason (reaped pendings, swept rows:
the silent R3 loss, now dashboard-visible). Timing comes from
the jobs substrate; queue depth is read off the replica DB.
No gauge, no custom histogram. Destination hosts log once at
job-def init; per-run logs carry counts plus frozen elapsed
(log props evaluate lazily in the logger thread).
Pre-existing rows are swept unsent (no backfill, accepted
loss); rows born with no archiver on share that fate (R3).
Memory mem:audit-log documents the design.
AI-assisted-by: muse-spark-1.3-contributor
* ♻️ Address review comments on audit log archive jobs
Rename the second archive destination from nitrate to admin-console
(jobs, metric and log labels, error code), reusing the :admin-console
flag. Unify the pending-queue insert into a single helper used by
both producers. Stagger the archive crons (Nexus at second 0, admin
console at second 30). Add per-call dbg logs to the admin-console
send. Update the audit-log memory and the affected tests.
Closes#12372
AI-assisted-by: muse-spark-1.3-contributor
Comment out the media-storage-bytes-per-team check in
upload-file-media-object. The SUM over storage_object takes
3-12s per upload on prod (3.5TB DB, ~34 slow runs/hour)
and blocks uploads. Keep the SQL and quote definition
for re-enable with a cheaper implementation.
Update the upload quota test to document the disabled
state until the new implementation lands.
AI-assisted-by: muse-spark-1.3-contributor
Binfile v1/v3 import and the v3 export get-file step hold a single
transaction while the ZIP is decoded and the file data is realized.
Disabling idle_in_transaction_session_timeout left a stuck connection
holding locks (and a pool slot) forever.
Raise it to 30 minutes instead. The longest import/export observed in
production is under 8 minutes, so the limit still bounds a stuck
session. statement_timeout stays at 300 s.
Add set-transaction-idle-timeout and configure-import-transaction in
app.binfile.common, use them from both paths, and cover the helper
with a test.
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Keep file_change revisions on their own deletion schedule
Restoring a file from the trash cleared the deletion mark on every
file_change row of that file, revisions included. A revision is born
already scheduled for deletion one hour later, and objects-gc only
looks at rows that carry a date, so unmarking one left it in the
table for ever: production had 358,517 of them across 3,614 files.
The file deletion cascade had the mirror problem: it rewrote the date
of rows that already had one, pushing a revision out to the trash
lifetime.
Restrict the restore to durable history and date only the rows that
have no date yet, so a revision keeps the schedule it was born with.
A pinning test covers the snapshot update path, which cannot reach a
revision because it resolves its row through a read filter. The
backend memories now carry the model of the three file_data classes
and this invariant, in place of the two superseded stubs.
Closes#12418
AI-assisted-by: deepseek-v4.1-flash
* 📚 Add autovacuum and Postgres exporter notes to Grafana memory
Record what this Grafana instance can and cannot show about autovacuum
so the next session does not have to re-derive it:
- The Postgres exporter scrapes the primary as well as the
read-only replica, so the primary's pg_stat_activity and
pg_settings_* are in Prometheus even though the SQL
datasource cannot reach them.
- It publishes no pg_stat_progress_* collector, and
pg_stat_activity_count has no backend_type label: autovacuum
workers are the usename/application_name-less bucket with
state=active. The launcher shows up as the AutoVacuumMain
wait event, and it is absent on the replica.
- A metric the exporter does not publish needs a custom query
block in its YAML queries file and a restart of the exporter.
- The Postgres server ships its own log to Loki (the job is
postgresql, primary only), whose streams carry only filename,
instance, job and level. The completed vacuum lines live
there: a per-table history, not a view of what runs now.
- The PostgreSQL section now ends with the vacuum panels,
with the Vacuum TXID age table last; an in-flight per-table
view would need a pg_stat_progress_vacuum collector in the
exporter itself.
AI-assisted-by: deepseek-v4.1-flash
* ✨ Extend backend histogram buckets beyond 7.5s
Give every backend timing histogram a tail past the shared 7500ms cap
so slow RPC calls, CLIMIT waits and background tasks report real
quantiles instead of clipping at the top bucket.
Request and wait metrics grow to 60s, background tasks to 300s and
Redis eval to 10s; the sub-7.5s buckets stay untouched to keep the
history comparable.
Closes#12408
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Fix flaky assertion in climit republish test
Read the last published waiting value instead of the exact
publication list. The sampler thread started with the component
runs its first tick asynchronously, and when it lands inside the
recording window it publishes the same count twice, which made
the full-suite CI run fail while focused runs stayed green.
AI-assisted-by: muse-spark-1.3-contributor
* ✨ Add file_data blob to JSON decode helper
Allow decoding file_data blobs (app.util.blob) to JSON
for investigations with jq, offline from a blob file or
from nREPL with the backend running.
AI-assisted-by: muse-spark-1.3-contributor
* ✨ Add validation step timings to update-file logs and metrics
Measure the six phases inside process-changes-and-validate
(resolve-libs, process-changes, soft and hard affected and
schema validations) with tpoint and try/finally, so timings
are recorded even when a hard validation throws.
Publish each phase to a new histogram
penpot_rpc_update_file_validation_timing with a fixed step
label using the request buckets (up to 60000 ms), and extend
the update-file debug log with the same *-ms fields plus
file, project and team ids. The deferred log only runs on
success, so failures get an inline error log with partial
timings.
AI-assisted-by: muse-spark-1.3-contributor
* 💄 Translate decode-file-data comments to English
Translate the Spanish docstrings, section headers and test
messages of the file_data decode helper to English, keeping
code untouched. The repo convention is English in code and
comments; the helper was first written in Spanish.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Count saved versions in the version quota
The snapshots-per-file and snapshots-per-team quotes counted the
saved versions by file_change.data, a column that only the legacy-db
file data backend writes: versions keep their payload in file_data,
so the count returned 0 and a configured quota never applied.
Count by label instead, which tells a version apart with every file
data backend, and add a partial index on file_change with the same
predicate so both counts are answered by an index only scan instead
of walking every row of every file of the team (10.3 seconds in
production for a team with 829 versions).
Closes#12404
AI-assisted-by: deepseek-v4.1-flash
* ♻️ Drop the ! suffix from the snapshot quota test helpers
The check helpers assert the outcome of a command instead of mutating
anything, so they do not take the ! that this codebase reserves for
functions whose contract changes state or runs a command (the
th/command! call inside them already carries it).
AI-assisted-by: deepseek-v4.1-flash
The multipart-only upload guard rejects the :content map sent by
upload-chunked! over plain command!. Send it through
multipart-command! like the other upload tests; forged-path tests
stay on plain command!.
AI-assisted-by: muse-spark-1.3-contributor
* 🚑 Reject forged upload paths outside multipart
Transit bodies can smuggle upload maps whose path points at any
server file; the schema only checks the shape and handlers open
the path directly.
app.rpc now injects a server-side multipart flag that bodies cannot
forge, and methods declare their upload keys with the new
multipart-only-params metadata, enforced centrally after schema
validation with a params-validation error. Covers all nine upload
methods, including chunked and management flows.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Sanitize raw explain in generic validation error bodies
The generic :validation branch returned ex-data untouched, so errors
carrying the raw malli explain (live schema objects, e.g. storage
invalid-storage-metadata) broke JSON encoding mid-stream and clients
got truncated bodies. Apply the same clean treatment the known codes
already use: drop the raw explain and attach its humanized text.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Add e2e path traversal tests for binfile and media upload
Add two e2e suites that hand-write transit bodies with a forged
["~#path", ...] upload path (no transit library needed) and expect
400 validation/params-validation. They cover import-binfile :file,
upload-file-media-object :content and upload-chunk :content, plus a
plain-JSON string-path control case. Before the multipart-path fix
they failed (500 io-exception, invalid-image, 200 stored chunk);
after it they pass.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Use multipart helper in binfile SVG test setup
The SVG import setup uploaded through plain command!, so the new
multipart-only guard rejected it with params-validation. Switch
the setup to multipart-command! like the other upload tests.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Use multipart helper in remaining upload tests
The multipart-only upload guard rejects plain RPC calls carrying
upload maps. Five tests with real files still used command! and
failed with params-validation. Switch them to multipart-command!
while keeping the forged-content tests on plain command!.
AI-assisted-by: muse-spark-1.3-contributor
A body declared as JSON that clojure.data.json cannot read ended as a 500
:unexpected with the whole stack trace in the log: wrap-parse-request
classified the failure by exception class, and the library signals its
syntax errors with a bare java.lang.Exception.
Classify it where it happens instead: read-body-params reads the body and
raises :validation :malformed-json (400) for anything it cannot parse,
keeps RequestTooBigException at 413 and re-raises a structured error from
deeper untouched. The body is read outside the handler try, so the error
goes out as a throw like the other specific-exception branches.
The failure is now one log line (path, peer address, content-type, parser
message) without the stack trace, and it is counted: the middleware takes
the metrics instance as a middleware argument, so the backend exports
penpot_http_malformed_requests_total and nexus declares its own
nexus_http_malformed_requests_total.
Tests: parse-request-malformed-json-body reproduces the production body
and asserts the 400 and the counter; the existing parse-request tests now
pass a metrics stub.
AI-assisted-by: deepseek-v4.1-flash
The storage-gc-deleted task used to stream all the pending rows with
db/plan inside one long read-only transaction, blocking the table
vacuum for the whole run and generating bloat on long deletions.
Now every chunk of 25 objects is one short read-write transaction:
the SELECT reads and locks the next objects after a (deleted_at, id)
keyset cursor with FOR UPDATE SKIP LOCKED, deletes the database rows
grouped by backend and removes their backend content. Concurrent runs
never take the same chunk, and the double SELECT of the old lock-ids
step is gone.
A chunk that fails aborts its transaction: nothing of the chunk is
deleted, its locks are released, and the run skips it; the cursor
still advances to the last row read, so a failed chunk is not visited
again in the same run.
AI-assisted-by: glm-5.3-flash
