The three test namespaces still carried graph-overlay in their
names and files. They are backend-tests.relation-overlay-test,
-sync-test, and -console-test now, so the branch carries no old
holder name outside dated records and the storage literal.
AI-assisted-by: glm-5.3-flash
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
The holder vocabulary names the persisted relation write an
export, so the batch tier takes app.graph.export, its file entry
point becomes export-file!, and the two on-connection helpers
follow inside the namespace. print-ingest! and the srepl
ingest-file-to-graph! keep their names, with their references
repointed; their own renames can ride a reviewer's ask.
AI-assisted-by: glm-5.3-flash
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
The datascript dependency's comment in frontend/deps.edn still
named the graph-overlay and the retired derived-index worker, and
the Ladybug-mirror set's docstring still named the graph-worker
branch. Both now use the holder names.
AI-assisted-by: glm-5.3-flash
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
The holder vocabulary names a graph by scope and a holder by
location, scope, and form, so the overlay namespace takes the
holder's name: app.graph.relation-overlay, with its console,
queries, and sync children. Aliases keep their short forms.
schema-version keeps its "penpot-graph-overlay-2" literal: it is
a storage version, and renaming it would rekey stored artefacts
without a schema change.
AI-assisted-by: glm-5.3-flash
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
The cascade's verify gate lints backend, common, and frontend/src/bench,
and clj-kondo exits non-zero on a warning, so the redundant let in
apply-set-token-set held every rebuilt candidate of this branch back.
The two lets become one, with no change in behaviour. An unused pid
binding in graph-overlay-test gets its underscore.
cljfmt fix over the three files ./scripts/ci reported as formatted
incorrectly, so the backend and frontend fmt checks pass.
clj-kondo over the verify scope: 0 errors, 0 warnings. The overlay
suites: 56 tests, 788 assertions, 0 failures.
AI-assisted-by: claude-opus-5-5
develop added ingest-on-connection-requires-allocator to
graph_sync_parity_test.clj, beside the round trip it shares no fixture
with. This branch retires that file with app.graph.sync, while
app.graph.ingest stays as the batch-export tier, so the test moves to a
namespace of its own rather than leaving with the suite.
AI-assisted-by: claude-opus-5-5
develop replaced :set-active-token-themes with :set-tokens-status, which
writes the active theme and set ids into :tokens-status, and added
:set-tokens-source, which points a file at an external tokens library.
the-classification-table-names-every-change-type fails by name on both,
which is what it is for.
Both are no-ops for the overlay. The builder indexes token sets and
tokens from the file's own :tokens-lib and stores no active status, and
it reads that lib whatever :tokens-source says, so neither change can
reach a datom a rebuild would write.
AI-assisted-by: claude-opus-5-5
The move mirror re-pointed a parent edge for every shape a :mov-objects
change named, so the fold followed a move the document refused. Two
measured consequences: moving a shape into its own subtree left the fold
disagreeing with a rebuild, and moving a shape under its own descendant
closed a cycle in the fold's parent edges, after which the fold had not
returned eight seconds later with app.graph.overlay.sync/subtree-eids on
the stack.
The gate is all-or-nothing for the whole change, which is the half a
per-shape mirror cannot see: process-change :mov-objects proceeds only
when the change names a shape, the parent exists, and every named shape
passes its own is-valid-move?, so one invalid shape in a batch of ten
moves none of them. The mirror now asks the same question of the running
state, in the change's own order.
It asks rather than answers, because a predicate written again in the
applier would be a second definition of which moves are legal, and the
two would drift the first time Penpot changes a nesting rule. So
calculate-invalid-targets and the predicate move out of the change
method into valid-move? at the namespace level, taking the objects view
and the two flags, with the method's own call site delegating to them and
no behaviour changed. The objects view is what the answer takes rather
than a container: the decision walks the moved shape's subtree down
:shapes and the destination's chain up :parent-id, reading :component-id
and :shape-ref off every shape it meets, and the overlay holds all of
those, which is why the same predicate answers for an index.
subtree-eids gains the guard the cycle case needs, so a cyclic index
fails loudly instead of spinning: the walk tracks the shapes it has
entered and returns.
Six new tests, and each mutation of the repair fails by name rather than
in an unrelated test: dropping the gate fails seven assertions across the
three gate tests, weakening every? to some fails the three that pin the
change-level reading, asking the predicate about the wrong destination
fails the same seven, and removing the walk guard fails the two that pin
the cycle.
backend-tests.graph-overlay-sync-test: 29 tests, 228 assertions, 0
failures.
Entry: pp:graph:decide-a-move-the-way-the-document-does open -> done
AI-assisted-by: deepseek-flash
Child order lives in the document as each container's :shapes vector, and
the projection recorded it only as the global Euler intervals, so the
index had no attribute a consumer could read to learn that a shape is the
third child of its parent rather than the fourth. :shape/order is that
attribute now: a plain 0-based integer, dense within one parent, declared
in the schema beside :shape/id and written by the builder per shape from
the parent's vector, exactly as the intervals are derived from it. The
intervals keep their purpose, which is containment, and the ordinal does
not answer a containment query.
With an order in the index, :reorder-children stops being a skip.
process-change carries :parent-id and :shapes, so the new arm writes the
container's ordinals from that vector and retracts the ordinal of a child
the vector no longer holds. The classification table's row moves to
:covered, and the tier supports twenty-five of the thirty-nine change
types rather than twenty-four.
The applier's other gaps close beside it. The raw colour and typography
reference ids ride in the index beside the resolved refs, so an asset
added after the shape that named it repairs every shape that was waiting
for it. A token name written onto a taken name displaces the holder as
one retraction rather than refusing. Every declared shape attribute names
its writer, so an attribute cannot be added to the schema and to no
writer. And a batch that skipped or dropped a change marks the session
stale, so both read gates refuse rather than answer from an index that
disagrees with the document.
Measured on a 5,101-object page: 20,411 datoms before one insertion and
20,412 after, of which one is the new shape's ordinal and none touches a
shape already present, so an append rewrites no sibling while a deletion
rewrites the ordinals after it (9,997 of them for the hundredth child)
and a move pays the interval renumbering it already paid.
backend-tests.graph-overlay-sync-test: 23 tests, 203 assertions, 0
failures.
Entry: pp:graph:sibling-order-needs-an-authored-attribute open -> done
Entry: pp:graph:mirror-the-reorder-children-change open -> done
Entry: pp:graph:store-the-raw-asset-ref open -> done
Entry: pp:graph:mirror-the-token-name-displacements open -> done
Entry: pp:graph:pin-the-attribute-vocabulary open -> done
Entry: pp:graph:does-a-set-token-carry-its-id open -> answered
Entry: pp:graph:stale-index-gate open -> done
Entry: pp:graph:close-applier-coverage-gap open -> done
AI-assisted-by: deepseek-flash
Review found two defects in the bulk build.
A reference may name a tempid the transaction never defines, because
ref-resolver answers with whatever ctf/find-ref-shape names and that
shape can sit in a container the builder skipped. The flattener resolved
such a target to nil, which puts a nil-valued reference in the index.
d/db-with refuses the whole transaction there, with :transact/syntax and
"Tempids used only as value", so the bulk path refuses it too and names
the defect: :projection/dangling-reference. A unit test pins both
refusals, and pins that the same transaction works on both paths once the
target is defined. The flattener is now tx->datoms, taking a
transaction, so the case is testable without a document that reaches it.
The timings in the docstring were the prototype's, taken on a JVM that
had served the session. Re-taken thirty seconds after boot, with the
first pass reported apart: a build is 800 ms cold and 471 to 513 ms warm
against 2,357 ms and 2,147 to 2,181 ms, so 4.2 times rather than 5.3.
The walk alone is 314 to 325 ms and steady from the first pass, which
puts the flatten and the load together at about 185 ms of a warm build.
Entity ids are an artefact of the interning order, and the docstring now
says so: deleting one leaf shape shifts every later id. Two builds of one
document are comparable by id and two builds of two documents are not,
which is a constraint on whatever compares them.
backend suite unchanged at 161 pre-existing failures, 826 tests and 5499
assertions against 824 and 5491 before.
The transact path spends its time resolving one tempid per entity, not
inserting datoms: the 2026-08-21 retrofit added 48 per cent more datoms
for no extra time. datascript's init-db skips that resolution and builds
the B-trees from sorted arrays, and build-tx already mints deterministic
string tempids from document ids, so assigning final entity ids is one
interning pass over the transaction.
On the 21,218-shape design system, measured back to back in one warm
JVM: 410 ms against 2,150 ms for the same 217,598 datoms. The graphs are
the same graph. Entity ids differ and carry no meaning, and the datom
sets are identical once every id is read back through the tempid that
minted it, which the new test asserts along with the stats and the edge
counts.
This is sound only while every build transform reads the document rather
than the half-built index, because init-db has no index to read. The
existing transforms all do: euler-numbering walks :objects, ref-resolver
goes through ctf/find-ref-shape, and the asset maps come from :data.
backend suite unchanged at 161 pre-existing failures (media and storage,
unrelated), 825 tests and 5495 assertions against 824 and 5491 before.
The experiment entry records the rule as (uses-token ?s ?tok ?attr),
with the applying property a rule variable rather than a per-shape
attribute walk; the two-place form was this port's own invention.
shapes-using-token now reads the property from the rule's third place.
AI-assisted-by: claude-fable-5
Four architectural results of the 2026-08-20 study become structure
(docs.local/graph/20260820-report-graph-overlay-datascript.md):
The relation vocabulary is a rule set, not raw clauses: child-of,
descendant-of, instance-of, refers-to, fills-swap-slot and uses-token
are named once in app.graph.overlay.queries, and every consumer query
and backported beadpot assertion traverses relations through them, so
an encoding can change under a rule without a consumer moving.
The subtle helpers run in the builder, never in a query:
ctf/find-ref-shape is called once per copy shape at build time and its
answer is the plain :shape/refers-to reference. The sync path
re-resolves the affected set when resolution inputs change: the shape
on :shape-ref, the subtree on head attributes and reparents, every
instance subtree of a component whose record or copy moved.
Containment carries a second form: one global Euler-tour numbering,
one DFS per container from a single global counter, so "inside subtree
X" is two interval comparisons. Per-container counters are the bug the
suite pins against: ranges would overlap numerically across pages.
The sync path renumbers a container from beyond the global maximum;
interval containment, not the numbers, is the invariant.
Token applications fold the property into the attribute name:
{:token/fill "layerTwo.background"} over the closed vocabulary of
app.common.types.token/all-keys, behind the uses-token rule so the
choice stays reversible. The relation-entity encoding and its two
extra datoms per application are gone.
31 tests, 650 assertions, 0 failures across the three suites.
AI-assisted-by: claude-fable-5
app.graph.sync existed to keep a live Ladybug session in step with the
editor's change vectors; with sessions on the overlay it has no caller.
Its parity suite and the binder gate move with it: the overlay sync
parity test replays the same change vocabulary against
app.graph.overlay.sync, and the console gate test guards the Datalog
surface where validate-on-connection! guarded Cypher. The validator
itself stays in app.graph.ladybug with the batch tier.
AI-assisted-by: claude-fable-5
An index of identity, type, and topology in datascript, replacing the
per-file Ladybug database for reverse and cross-container questions.
Entity identity is the (container, shape) pair, never the shape id
alone: uuid/zero keys every page's root frame, duplicated pages share
non-root ids, and deleted components keep their own copy of the
main-instance subtree under the same ids.
app.graph.overlay projects a file's data into the index: pages and
component containers, shapes, components, and the asset node kinds the
Ladybug projection never gained (colors, typographies, token sets,
tokens), with their link transforms from beadpot's ledger (UsesColor,
UsesTypography, UsesToken) as reference attributes and relation
entities. Color references are stored per source (fill, stroke, text
content) so the sync path can rebuild each contribution from a single
:set operation.
app.graph.overlay.queries holds the shared rule set and the standing
question set. The four Cypher edge kinds are rules over indexed
attributes, so they cannot drift from what they restate. The helpers in
app.common.types.component and friends stay the oracle: instance-head?
is the [?s :shape/component-id _] clause, never a reimplementation, and
the local upward walk stays with ctn/get-parent-heads when the caller
holds the document, because the pointer chain beats the rule walk by
three orders of magnitude.
app.graph.overlay.sync applies Penpot change vectors to the index,
pure db-to-db, mirroring app.common.files.changes/process-change per
clause, including load-component-objects on soft component deletion.
app.graph.overlay.console parses, gates, and formats free-form Datalog:
datascript resolves namespace-qualified symbols in call position via
resolve, so every call symbol must be a built-in, an aggregate, or a
shared rule.
The suites pin: overlay answers identical to the helpers on constructed
fixtures (177 assertions, beadpot's graph tests backported with each
claim verified against a helper), synced index equal to a rebuilt one
after every prefix of a 29-change replay covering all 12 supported
change types, and the console gate.
Measured on the Pencil design system (49 pages, 21 256 shapes, 730
components): build 1.63 s, 157 893 datoms, about 7.5 MB retained,
answers identical to the helpers across 724 components, 1 000 sampled
parent-head chains, and 5 549 frame subtrees; sync p95 0.28 ms per
commit. The Ladybug session tier measured 33 to 43 s and 144 MB for a
20k-shape file the projection can ingest; the projection still rejects
every real file at the Path node.
AI-assisted-by: claude-fable-5
* 🎉 Move backend jobs to the unified job table substrate
Replace the generic 'task' table with a unified 'job' table as the
durable substrate for all backend jobs, and migrate every producer
and consumer to it. The legacy 'task' table stops receiving writes
but stays dormant (still cleaned by tasks-gc); upload-session
chunking and the other post-branch upstream changes are preserved.
New 0154 migration creates the 'job' table: dispatch and lifecycle
columns (name, queue, priority, scheduled_at, retries, status,
timestamps, props) plus optional user-facing ledger columns
(profile_id, target, progress, error, result, resource_id,
expires_at). No modified_at trigger: the application updates it on
every write for lease and orphan detection. Partial indexes cover
the dispatcher claim query, orphan scanning, and per-profile
listing.
New app.jobs public API: submit!/invoke!, heartbeats and progress
reporting, a precompiled job-def registry (name, schema, handler,
decoder, validator), an ephemeral request/response mode, and a
management API for external workers. The dispatcher claims
new/retry rows (FOR UPDATE SKIP LOCKED) with a plain JSON redis
payload and marks lease-expired running rows as orphaned; the
runner executes per-queue with the same retry conventions.
New jobs GC (daily): deletes expired rows and retains internal
terminal rows per :jobs-retention, touching referenced storage
objects so storage-gc-touched reclaims them. Job resources live in
a dedicated job-resource bucket with a reclaim branch in
storage-gc-touched.
All production call-sites (sendmail, delete-object, demo-purge,
file-gc, offload, webhooks, quotes, nitrate bulk delete,
snapshots, telemetry) and the ten cron tasks now submit through
app.jobs; cron is a pure scheduler and the legacy worker registry,
task-table inserts, and per-namespace handlers are removed.
Includes migrations, dispatcher, runner, jobs, request/response,
jobs GC, management API, and cron test suites.
AI-assisted-by: glm-5.3-flash
AI-assisted-by: muse-spark-1.3-contributor
AI-assisted-by: kimi-k3
AI-assisted-by: mimo-v2.5
* ✨ Address review comments on jobs substrate
Resolves the four open review threads on the jobs PR. Orphaned jobs
are now marked aborted, a system-side terminal state with no retry:
the dispatcher sweep stores the end event with outcome aborted in a
single bulk insert in the same transaction, records the terminal
outcome next to the orphaned counter, and reports an error log;
orphans keep alerting through the log and the counter, with no
user notification.
The aborted state is terminal everywhere: the status CHECK, the
sweep, the lifecycle writers, the jobs GC retention, the backlog
gauges and the user status mapping, with tests covering each
transition. The claim comment, the 2.20 doc version and the legacy
task table notes are corrected, and objects-gc heartbeats per chunk.
AI-assisted-by: muse-spark-1.3-contributor
* ✨ Gate error reporters behind :error-reporting flag
Error reporters used to start on every boot with no way to
switch them off. They now only start when the new
:error-reporting flag is on (off by default, enable with
PENPOT_FLAGS=enable-error-reporting). The database reporter
needs the flag alone; mattermost needs the flag plus its
webhook url. Shutdown with the flag off is safe, and the
database loop now honors its runtime switch.
AI-assisted-by: muse-spark-1.3-contributor
* 📚 Add audit-log reference memory and update dev tooling
Adds the top-level audit-log reference memory with all event
flows and turns backend/audit-log into a redirect stub.
Also includes the staged dev-tooling updates (playwright
deps, mdts args, gitignore).
AI-assisted-by: muse-spark-1.3-contributor
* ✨ Assign server-side initiator to frontend audit events
Frontend audit events now carry a server-assigned initiator
derived from the x-client header (penpot-frontend goes to
app, penpot-admin-console goes to admin-console, legacy
penpot-nitrate stays on admin-console, all else goes to
app). Client-sent values are always overwritten, and the
key survives on telemetry shadow rows.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Keep initiator on frontend telemetry context
The strips-pii-keys test still asserted the old contract
(initiator stripped). The initiator only names the sending
app, so it is not personal data and must survive the
telemetry filter like on the backend channel.
AI-assisted-by: muse-spark-1.3-contributor
* 📚 Document initiator on PostHog and browser origins
Complete the audit-log memory for the latest changes: the
initiator property forwarded to PostHog on both channels
and the eventOrigin rule for admin-console browser events.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Fix unscaped argument in script
---------
Co-authored-by: alonso.torres <alonso.torres@kaleidos.net>
* ✨ Isolate backend-test database per devenv instance
Each wsN now gets its own backend-test database
(penpot_test_wsN) and Valkey DB (6+N) on the shared
infra, so parallel test runs no longer wipe each
other. The main database stays shared on purpose.
manage.sh creates the test database on bring-up and
passes PENPOT_TEST_* into the main container. Test
helpers already read those vars, so no runtime code
changes. scripts/psql and db-schema gain --ws.
AI-assisted-by: muse-spark-1.3-contributor-free
* 🐛 Fix per-instance test database creation on bring-up
psql -c does not reliably mix SQL with psql-only
commands, so the CREATE DATABASE ... WHERE NOT
EXISTS ... \gexec one-liner never ran. Use two
plain SQL round-trips instead (check pg_database,
then CREATE DATABASE) and print what happens, so a
future failure shows up in the bring-up output.
AI-assisted-by: muse-spark-1.3-contributor-free
Run the audit archive cron when :nexus or :admin-console is on. Ship
allowlisted events to Admin Console first (with row ids for
idempotency), then the full chunk to Nexus when :nexus is set. Mark
archived_at for the whole chunk on success, including nitrate-only
mode. Rename the gate flag from :audit-log-archive to :nexus.
AI-assisted-by: Composer
Co-authored-by: Cursor <cursoragent@cursor.com>
* ✨ Add size limits to profile props and plugin registry
Bound the total serialized size of profile settings to 2 MiB
(:profile-props-max-size), checked on the merged result before
persisting, with a controlled :props-too-large error. Profiles
that already exceed the limit can still shrink but cannot grow.
Cap plugin registry entries in the shared schema (code 1 MiB, 50
plugins max, bounded name/host/description/icon) and restore rate
limiting on the plugin RPCs (profile-mutations bucket, one write
at a time per profile). The plugin manager now asks for
confirmation before removal and ignores repeated clicks while a
persist request is in flight.
Closes#11592
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Enforce plugin count cap, byte sizes and removal guard
Enforce the declared 50-plugin cap in add-profile-plugin with a
specific :too-many-plugins error (updates of existing entries
still pass); the cap lives in a shared max-plugins constant.
Measure profile props size in UTF-8 bytes instead of chars so
multibyte content cannot slip past the limit.
Cover install/remove persist logic with mocked-RPC frontend tests
(release semantics, in-flight dedupe, validation vs rollback
split) and add the missing boundary tests in common.
Expose the in-flight persist set from the plugin registry and
disable the remove button of entries being saved.
Closes#11592
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Fix rollback loops and restore paths in plugin registry
Restore the previous plugin version instead of dropping the entry
when a validation error rejects an update of an installed plugin.
Make compensating writes one-shot with terminal callbacks so a
persistent failure cannot ping-pong between install and remove.
Restores keep the original list position; the unused public
plugin-persisting? predicate is removed.
Pin count-before-size precedence with a dedicated test and fix
translation source refs to their canonical lines.
Closes#11592
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Guard notifications write and fix restore ordering
Route update-profile-notifications through check-props-size! so
oversized profiles cannot grow through that path; document the
exempt system writers. Remove the duplicated stale entries in
en.po, keeping the canonical translation refs.
Restore rejected plugin updates at their original list position
instead of leaving the optimistic move in place.
Closes#11592
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Skip no-op plugin removal and clarify size comments
Return early from remove-profile-plugin when the id is absent:
no wasted write, no size check, and no manufactured :plugins key
that could spuriously fail on oversized profiles.
Clarify that per-field string caps count chars while the byte
budget is enforced by profile-props-max-size.
Closes#11592
AI-assisted-by: muse-spark-1.3-contributor
* 📎 Fix formatting in rlimit.edn for profile operations
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 📎 Fix formatting of import-binfile/global entry
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* ♻️ Simplify props size check and tighten plugin entry caps
Measure props with transit bytes directly instead of the
PGobject string roundtrip.
Rename check-props-size! to check-props-size: single hard limit
on the merged props, no growth comparison, and return props so
writers thread the check into the update.
Move the 2 MiB default into default-props-max-size on the
profile namespace, still overridable with the optional
:profile-props-max-size config entry.
Tighten registry-entry :code and :icon to 500 chars: they hold
manifest paths, not content.
Closes#11592
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Fix compatibility problems
---------
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
Co-authored-by: alonso.torres <alonso.torres@kaleidos.net>
* ♻️ Scope organization notifications to specific WebSocket topics
Publish team/org notifications to team-id and organization-id topics
instead of broadcasting to all connections via uuid/zero. Dashboard and
workspace now subscribe to their current team and organization on
initialization, receiving only relevant events.
Backend: added subscribe-organization/unsubscribe-organization WebSocket
handlers and updated :close to clean up organization subscriptions.
Modified notify-team-change, notify-organization-deletion, and
notify-organization-change-sso to publish to specific topics.
Frontend: dashboard and workspace now subscribe to team-id and
organization-id (when applicable) on initialization, with nil-guard
in topic filters.
Closes#11455
AI-assisted-by: longcat-2.0
* 🔧 Remove unused session-id binding in unsubscribe-organization
Clj-kondo lint fix.
AI-assisted-by: longcat-2.0
* 🐛 Fix permission check on team ws connection
---------
Co-authored-by: alonso.torres <alonso.torres@kaleidos.net>
* ✨ Enforce idle and absolute session expiration
Sessions now expire on two server-side conditions: an idle window
(PENPOT_AUTH_TOKEN_COOKIE_MAX_AGE, default 7d) and an absolute cap
from creation (PENPOT_AUTH_TOKEN_COOKIE_MAX_AGE_ABSOLUTE, default
30d, enforced by the token :exp claim). The daily session-gc task
deletes rows that exceed either window, so idle sessions can no
longer be replayed and active sessions are not deleted at the idle
window.
Also remove the legacy v1 HTTP sessions: the http_session table and
the string-id / :ver 0 token code paths are gone. Any v1 cookie now
requires a fresh login.
Document the session expiration configuration in the technical guide
and add a backend memory describing the token, renewal and GC model.
Closes#11646
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Address session-expiration review findings F1-F4
Fix the unreadable test (a stray paren broke whole-suite
discovery), enforce idle expiration on every request in
wrap-authz, fail boot fast when the absolute cap sits below
the idle window, and align config defaults with the memory
rule while fixing its migration number and stale reference.
Closes#11646
AI-assisted-by: muse-spark-1.3-contributor
* ✨ Add Malli schema for storage metadata with dual decode
Phase 1 of the storage_object.metadata migration: reads accept both
Transit and plain JSON (sniffed by the marker) and always return the
normalized shape; writes validate against a closed per-bucket Malli
schema and still serialize as Transit unless the new
:storage-metadata-as-json config flag is set.
The 0155 migration normalizes existing rows inside Transit (reference
to bucket, default bucket, drop of chunk leftovers) and is
idempotent; large instances should fake it and run the batched
script instead.
AI-assisted-by: muse-spark-1.3-contributor
* ♻️ Address review findings on storage metadata Phase 1
Collapse the dead :reference leg of the gc-touched bucket fallback
(the decode always sets :bucket on non-nil metadata, so it is only
reachable with a NULL column) and fix its comment.
Pin the write flag off in the transit-assuming metadata tests so the
suite proves the same with the flag set, and add coverage for the
flag rollback contract, JSON hash survival, NULL metadata in
gc-touched, and the 0155 normalization statements.
AI-assisted-by: muse-spark-1.3-contributor
* ♻️ Backfill NULLs, canonical buckets, comment fix
Backfill NULL metadata columns in 0155 via coalesce (the key-missing
rule already matches them), derive valid-buckets from the Malli schema
dispatch entries so the list lives in one place, and correct the
lookup-bucket fallback comment to NULL columns.
AI-assisted-by: muse-spark-1.3-contributor
* ♻️ Defer corrupt metadata rows in storage gc-touched
Decode touched rows individually so one non-map metadata value no
longer aborts the whole chunk: corrupt rows are logged and deferred
exactly one day in the same transaction, keeping their metadata
intact for a later repair, while healthy rows process normally.
AI-assisted-by: muse-spark-1.3-contributor
* ♻️ Address storage metadata phase 1 review findings
Address the review findings on the storage metadata phase 1 branch:
- Fix put-and-delete-object: it stored the object with
::sto/expired-at, so the row was already deleted and del-object!
returned false. Add delete-expired-object-returns-false to keep
the expired-delete case covered.
- Cache the Malli decoder and encoder per process. Building them
compiles the closed multi-dispatch schema, and decode-metadata
runs on every read path (get-object, dedup probes, GC batches).
- Catch Exception instead of Throwable in try-decode-row so JVM
Errors are not deferred as corrupt metadata.
- Add penpot_storage_gc_poison_total, emitted from
storage-gc-touched; wire ::mtx/metrics into its handler.
- Anchor the encoding sniff to the start of the document so a
plain JSON value that begins with a Transit-looking prefix is
not read as Transit.
- Cover every bucket on both encodings, a JSON roundtrip through
the jsonb column, nil metadata, the canonical bucket set and a
poison-only GC chunk.
- Rename private check-metadata! to check-metadata.
AI-assisted-by: deepseek-v4.1-flash
* ♻️ Simplify the storage metadata schema to a single map
Replace the per-bucket :multi dispatch with a single closed map: the
bucket is validated with ::sm/one-of over metadata-buckets (now a plain
set) and the remaining keys are typed optional fields. Per-bucket
enforcement shrinks to a one-line :fn guard requiring :file-id and :id
for file-data, whose ids the GC reads to resolve references.
- Drop the dead (sm/register! ::metadata ...): nothing references the
schema by keyword.
- Define tempfile-bucket and upload-session-bucket in the schema and
alias them from app.storage, removing duplicated literals.
- Keep content-type required and the map closed, so an unknown bucket
or key still fails fast on write.
AI-assisted-by: deepseek-v4.1-flash
* ♻️ Drop input coercion from encode-metadata
encode-metadata no longer runs the json-transformer decoder before
validation. On the write path its only effect was coercing string
UUIDs to UUID, and every producer already passes native UUIDs (the
RPC profile-id, uuid/random, or binfile ids decoded as ::sm/uuid).
Reads keep decoding, so stored Transit or JSON values still come
back as native types.
- Replace encode-accepts-string-uuids with
encode-rejects-string-uuids, pinning the stricter contract.
- Pass native UUIDs in encode-writes-plain-json-with-flag.
AI-assisted-by: deepseek-v4.1-flash
* 📚 Document each statement in the storage metadata migration
Move the per-statement rules out of the header and add a comment to each
UPDATE explaining what it does: drop chunk leftovers, promote the legacy
"~:reference" to "~:bucket", drop residual "~:reference", and backfill the
default bucket. The header keeps the scope, the encoding note, the `->`
vs `?` note and the large-instance warning.
AI-assisted-by: deepseek-v4.1-flash
* 📚 Unwrap wrapped lines in the backend storage memory
One line per bullet or paragraph, as mem:memory-maintenance requires.
Only formatting; no content change.
AI-assisted-by: deepseek-v4.1-flash
* ♻️ Defer storage GC poison rows in their own transaction
process-chunk! no longer takes poison-ids; it only processes the healthy
chunk. The deferral moves to defer-poison! and process-touched! runs it in
its own transaction, separate from the freeze/delete work. The loop still
drains while there is chunk or poison, so a batch made only of poison rows
does not leave healthy rows behind the LIMIT 10 waiting for the next run.
Add a regression test: ten poison rows plus one healthy row with a later
touched_at are all handled in the same run.
AI-assisted-by: deepseek-v4.1-flash
* ♻️ Declare per-bucket metadata requirements in one map
Replace the file-data-specific predicate with bucket-requirements, a map
from bucket to the extra keys it must carry. metadata-buckets is derived
from its keys and a single generic :fn enforces presence, so a new bucket
and its contract are one entry. organization now requires
:organization-id; file-data keeps requiring :file-id and :id.
Update the http-assets test helper to set organization-id for its
organization objects.
AI-assisted-by: deepseek-v4.1-flash
* ♻️ Drop the ! suffix from storage GC helpers
Rename the internal helpers in app.storage.gc-touched (process-chunk,
defer-poison, mark-freeze-in-bulk, ...) to drop the trailing !.
AI-assisted-by: deepseek-v4.1-flash
* ♻️ Drop the ! suffix from storage GC deleted helpers
Rename the internal helpers in app.storage.gc-deleted (clean-deleted,
delete-sobjects, delete-give-up, ...) to drop the trailing !.
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Fix dedup lookup for JSON-encoded storage metadata
get-database-object-by-hash only matched the Transit keys, so once the
:storage-metadata-as-json flag wrote plain JSON rows the dedup stopped
finding them and duplicated blobs. Match both encodings with a UNION ALL
of two indexable branches.
- Add migration 0156 with the plain-key dedup index; the legacy 0068
index stays until Transit support is removed.
- Cover it with a JSON dedup test and a Transit -> JSON cross test.
AI-assisted-by: deepseek-v4.1-flash
The import-svg-attrs-asset helper called (first result) on the
map returned by import-files!, yielding a clojure.lang.MapEntry
instead of a file UUID. PostgreSQL rejected the MapEntry as a
query parameter with "Can't infer the SQL type".
Extract the id via (first (:file-ids result)) as the neighboring
test already does.
AI-assisted-by: longcat-2.5-preview-free
The event was written twice per accepted organization invitation. The
backend submitted it, and the browser then re-submitted a copy of the
props that the backend had already put in the response under
`:organization-invitation-audit` (`handle-token :team-invitation` in
`verify-token.cljs`). Both rows carried the same name with different prop
vocabularies, and the browser copy only existed when the browser finished
the flow.
Emit the event from the backend only. It now also carries the three props
that lived in the browser copy: the organization member count before the
add, the add source, and whether the invitee also joined a team. The
origin moves to the event context as `:event-origin`. The response no
longer includes `:organization-invitation-audit`, so the browser stops
emitting the event and `verify-token.cljs` drops its
`app.main.data.event` require.
The `accept-*` events of this command now share one prop vocabulary:
`:profile-id` for the accepting profile, `:invited-by` for the inviter
and `:profile-email` for the email, replacing the mix of
`:user-id`/`:user-who-send-invitation` and `:email`.
Audit consumers of `accept-organization-invitation` now see one row per
acceptance instead of two, and must read the new prop names.
AI-assisted-by: space-bunny-free
The json reader of the binfile v3 import rewrites every key of every
entry to kebab-case, nested maps included. Shape `:svg-attrs` is the one
map whose keys are camelCase react prop names, because the svg import
path runs them through `attrs->props`, so an attribute exported as
`fillRule` came back as `:fill-rule` and was stored that way.
The renderer looks the attribute up by its camelCase name, does not find
it and falls back to the default fill rule, so a shape exported with
`fillRule: evenodd` was painted without its hole, and the attributes
panel showed `fill-rule`.
`clean-shape-post-decode` already runs on every shape right after the
schema decode, for page shapes and component objects alike, so the
repair goes there: run `:svg-attrs` back through `attrs->props`, the
same transform that built the keys. It is idempotent, so shapes that
arrive correct are left untouched.
The new tests import a real export that carries the attribute, for
page shapes and component shapes.
Closes#11954
AI-assisted-by: space-bunny-free
* 🐛 Attribute audit events to the caller, not the response
prepare-rpc-event took the event profile-id from the result map
whenever it carried one, before falling back to the caller. Any
command returning a response with a :profile-id key silently
credited the action to somebody else.
get-error-report returns the report with its decoded content
merged in, and that content holds the profile that owned the
report, so privileged reads were logged against the users whose
crashes were being inspected. verify-token on a team invitation
returns the inviter's profile-id, so accepting an invitation was
logged against the inviter.
Resolution is now ::audit/profile-id metadata, then
::rpc/profile-id, then the zero uuid; the response is never
consulted. The two verify-token branches that relied on it now
declare the profile in the result metadata. Every other command
either already declared it or returns no :profile-id; all 30
registered command namespaces were checked.
The tests that pinned the old behavior are replaced by ones
covering the new contract.
AI-assisted-by: space-bunny-free
* 🐛 Coerce the audit profile-id override to a uuid
The only sanctioned way for a command to override the profile of an
audit event is the ::audit/profile-id metadata, and the value is set
by hand in a dozen commands, some of them reading it from token
claims or other sources we do not type.
schema:event requires a uuid and submit* swallows the validation
error, so a string did not fail loudly: the row was dropped silently.
Values that cannot become a uuid are now discarded with a warning
and the event falls back to the caller, which is always a valid uuid.
A uuid, the common case, exits on the first check.
AI-assisted-by: space-bunny-free
* 🐛 Migrate openUIApi schema to Zod v4 function syntax
Zod 4 removed z.function().args(), which broke the
plugins-runtime build with implicit-any errors on every
openUIApi parameter and knock-on possibly-null errors on
the modal in plugin-manager.
Declare the inputs with z.function({ input: [...] }) so the
parameter and return types infer again; behavior is unchanged.
Add a regression spec covering delegation, optional args and
rejection of invalid theme and title values.
AI-assisted-by: muse-spark-1.3-contributor
* 📚 Fix deprecated markdown-it-anchor permalink option in docs
Migrate docs Eleventy config to the markdown-it-anchor v10 API.
Replace the deprecated boolean permalink option with
linkInsideHeader, keeping the same symbol and class.
Bump markdown-it-anchor to v10 and related docs deps.
AI-assisted-by: muse-spark-1.3-contributor
* ⬆️ Update deps
* ⬆️ Update base docker images
* 📎 Fix mcp tests
* ✨ Add renderer option on create-demo-profile command
Allow create-demo-profile to take an optional renderer (svg or
wasm) and store it on profile props. When omitted, no renderer is
written so the Penpot default still applies.
Closes#11893
AI-assisted-by: Muse Spark 1.3 Free
* 📚 Fix renderer doc version on create-demo-profile
Correct the ::doc/changes version for the new renderer param from
2.20 to 2.18.1.
AI-assisted-by: Muse Spark 1.3 Free
* 🐛 Allow invitation-based registration when disable-registration is set (#5178)
Per documentation, disable-registration 'disables registration
(still enabled for invitations only)'. Two bugs prevented this:
1. verify_token.clj: when processing an invitation token for a
non-logged-in user with no member-id, the redirect included
registration-disabled? in its condition, sending invited users
to the login page instead of the register page.
2. auth.clj validate-register-attempt!: the registration-disabled
check fired unconditionally before the invitation-token check,
rejecting the actual register RPC even with a valid invitation.
Fix: in verify_token.clj remove registration-disabled? from the
redirect condition for new-user invitations. In auth.clj restructure
the check as an if/else: with an invitation token, validate the token
and allow registration; without one, enforce the flag as before.
* 🐛 Allow registration with disabled public registration
Allow valid team invitations to create new profiles when public
registration is disabled, while keeping password login and invitation
validation required.
Add backend regression coverage for flag combinations and verify-token
redirects, frontend route coverage, and configuration documentation.
Closes#5178
AI-assisted-by: space-bunny-free
* 🐛 Revalidate active invitation during registration
Require a live, unexpired team invitation before using the
registration exception, and recheck it before creating a profile.
Reuse the same lookup in invitation token verification.
Add regression tests for canceled and expired invitations, the
registration race, and explicit redirect contracts. Update docs
and backend auth guidance.
AI-assisted-by: Space Bunny Free
* 🐛 Lock and normalize invitation registration checks
Lock active invitation rows during transactional registration and
acceptance so cancellations cannot race with profile or membership
creation.
Normalize invitation emails before comparisons and database lookups.
Add concurrency, email casing, and final flag regression tests.
AI-assisted-by: Space Bunny Free
---------
Co-authored-by: Sumit Ridhal <sridhal@redhat.com>
* ✨ Add deployment info to events
* ✨ Add get-environment-data RPC method
Add a single public RPC method returning the deployment type and
the enabled environment flags. It replaces get-deployment on the
management API and get-enabled-flags on the main API.
get-enabled-flags stays as a deprecated alias returning only the
flags, so existing callers keep working until it is removed.
The frontend event initialization now reads the flags from the new
method. The exposed flags stay limited to audit-log and telemetry
to avoid leaking internal backend flags.
AI-assisted-by: deepseek-v4.1-flash
---------
Co-authored-by: Andrey Antukh <niwi@niwi.nz>
* 🐛 Fix organization/team switcher issues from UX review
* 🐛 Let members leave an organization without SSO credentials
* 🐛 Fix style from previewed organization in the team switcher
* 🐛 Fix leave-organization modal test stubs
Inject the storage component into the delete-object task handler and
validate it on initialization, so library absorption no longer fails with
"expected valid storage to be provided" for storage-backed file data.
Also strengthen the regression test: it now creates a real component in
the shared library and an instance in the referencing file, and verifies
the component is absorbed after deletion instead of only checking the
revision bump. The test disables the fdata/pointer-map feature to stay
focused on the storage fix.
Signed-off-by: Yuito Akatsuki (Tani Yutaka) <yuito@yuito-it.jp>
AI-assisted-by: deepseek-v4.1-flash
* ♻️ Build organization invitation audit event in frontend
* ♻️ Align invitation token profile-id with created-by
The invitation token carried the minter in :profile-id while the
invitation row tracks the creator in :created-by. Both mean the
inviter, so re-sends or re-requested links made them disagree and
forced a second response key, :user-who-send-invitation.
Mint :profile-id from :created-by in both token creators, backfill
it from the row on accept (covers stale in-flight tokens), and drop
the duplicate response key. The frontend maps :profile-id to the
unchanged :user-who-send-invitation audit prop.
AI-assisted-by: Muse Spark 1.3 Free
* ♻️ Reuse token ids in invitation accept response
Backfill :member-id with the accepting profile and drop the
:user-id duplicate from the verify-token response, mirroring the
:profile-id/:user-who-send-invitation cleanup. The frontend maps
:member-id to the unchanged :user-id audit prop.
AI-assisted-by: Muse Spark 1.3 Free
---------
Co-authored-by: Andrey Antukh <niwi@niwi.nz>
* 🐛 Restore rate-limit headers and add Retry-After
The account-lockout change replaced the header-forwarding 429 handler
with a body-only one, so existing RPC rate-limit responses lost their
x-rate-limit-remaining and x-rate-limit-reset headers. Account lockout
never sent Retry-After either.
Make handle-error :rate-limit preserve ::http/headers and add a
retry-after header when the exception carries a non-nil :ttl in
seconds, keeping the current JSON body. Add focused tests for both the
lockout and the RPC limiter paths.
Document activation, defaults, password/LDAP scope, Redis fail-open
behavior, and the lockout risk, and record the final HTTP contract in
the backend auth memory.
Refs #11397
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Add Retry-After to RPC 429 and expose headers in CORS
Address review follow-ups on the account-lockout 429 contract:
- The RPC limiter now sets retry-after in its 429 headers (seconds
until the longest rejecting limit resets), so it matches the
account-lockout response and the HTTP standard.
- CORS exposes retry-after, x-rate-limit-remaining, and
x-rate-limit-reset so browser clients can read them.
- Use backticks for Retry-After and account-locked in the docs for
consistency with nearby sections.
Refs #11397
AI-assisted-by: deepseek-v4.1-flash
The push-audit-events initiator test injects a request with
:app.http/auth-key-id via :app.http/request metadata. prepare-rpc-params
always replaced that metadata with a fresh DummyRequest, so the key id
was lost and the initiator fell back to "app".
Honor a caller-supplied request map, merging body params into its
:params, and keep the dummy request only for non-map IRequest stubs
(reify requests used by the audit tests, which cannot be assoc'd).
Add focused regression tests for the three cases and document the
helper contract in the backend testing memory.
AI-assisted-by: deepseek-v4.1-flash
* 🐳 Add upstream diagnostics to nginx access log
Enrich every access-log line with the internal journey of the request:
the status the backend answered (us), the time spent connecting to it
(uct), the time spent waiting for its answer (urt) and the internal
address that served the request (ua).
A plain 502 line used to say nothing about where the request died. With
this format, the tail of the line classifies the failure: connection
rejected, backend accepted and hung (uct + urt under 1s), or backend
stuck until read timeout. This was the missing witness in the Sep 20
incident, where nginx received connection resets with zero timeouts and
zero rejections.
Applied both to the production image template and the devenv config.
With proxy_pass on variables there is no upstream keepalive, so uct
measures one real TCP connection per request.
Parsing the new fields (us, uct, urt, ua) on the log shipper is left to
ops, so they can be filtered in Loki.
AI-assisted-by: glm-5.3-flash
* 🐳 Add stub_status endpoint for nginx metrics
Add a dedicated localhost-only server (listen 127.0.0.1:8082) exposing
/stub_status next to every other location of the public server. Ops can
run the official nginx-prometheus-exporter as a sidecar against
http://127.0.0.1:8082/stub_status and get nginx_connections_active,
accepted vs handled, reading/writing/waiting and request rates in
Prometheus.
Binding it to localhost and its own server keeps it unreachable from
outside the host and out of the public surface, and access_log off
avoids polluting Loki with one line per Prometheus scrape. The base
image already ships stub_status compiled in, so no image rebuild is
needed.
Applied both to the production image template and the devenv config.
AI-assisted-by: glm-5.3-flash
* ✨ Expose http server gate metrics (worker and connector)
The backend already measured dispatch latency but nothing reported the
state of the "house door": the xnio worker queue and threads, and the
monitor-level listener counters. This was the exact blind spot of the
Sep 20 incident, where the server kept answering health checks while it
accepted connections and dropped them without response.
Add a periodic metrics sampler that lives and dies with the http
server (single daemon thread, 15s interval, each sample guarded so an
unexpected error does not cancel subsequent runs) and publishes:
- worker (xnio MXBean gauges): penpot_http_worker_queue_size,
busy_threads, pool_size and max_pool_size. Negative samples are
discarded: the MXBean transiently reports -1 on the busy thread
count (verified live), and a stale negative would read as zero.
- listener (Undertow connector statistics, enabled via the new
:server/statistics yetti option): penpot_http_connector_active*
_connections gauge and requests_total / errors_total counters.
Undertow exposes absolute totals, so the sampler keeps a watermark
atom and publishes deltas, skipping (and moving forward past) a
counter reset.
The connector-level part depends on yetti v11.11, which now accepts
a :server/statistics server option (patch authored and released
upstream; before it, ListenerInfo#getConnectorStatistics always
returned nil).
New tests cover the samplers with fake MXBean/collector statistics
against real prometheus collectors, including the negative-sample
filter, the delta/watermark logic and the sampler lifecycle.
AI-assisted-by: glm-5.3-flash
* 🐛 Include jdk.management in the backend runtime JRE
The production image builds a trimmed JRE with jlink and omitted
jdk.management. Without that module the OS MXBean is
sun.management.BaseOperatingSystemImpl, which has no
getProcessCpuTime, getOpenFileDescriptorCount nor
getMaxFileDescriptorCount. The prometheus client StandardExports
reads those getters reflectively and collect() swallows the
NoSuchMethodException, so process_open_fds, process_max_fds and
process_cpu_seconds_total silently disappeared from /metrics while
the other process_* families kept flowing.
Verified against Prometheus: the app job only ever exposed
process_start_time_seconds, process_virtual_memory_bytes and
process_resident_memory_bytes; the fd and cpu families were absent.
Reproduced locally by running the backend metrics registry on a JRE
built with the same jlink module list (false/false/false) and on one
with jdk.management added (true/true/true).
Add the module to --add-modules and pin the metric contract with
backend-tests.metrics-test.
AI-assisted-by: deepseek-v4.1-flash
* ♻️ Build the http metrics sampler on promesa.exec
Replace the hand-rolled ScheduledThreadPoolExecutor and ThreadFactory
with promesa.exec primitives: px/scheduled-executor with a daemon
thread factory, and a px/schedule chain that reschedules the next
sample when the current one finishes.
Beyond fitting the existing periodic-task pattern (worker/cron,
rpc/rlimit), the chained schedule makes the docstring promise real:
with scheduleAtFixedRate an exception escaping the runnable cancelled
the following executions, while the reschedule now happens in a
finally block.
The sampler shutdown uses px/shutdown-now (shutdown! is deprecated in
promesa 12.0.0) to cancel the pending sample, keeping the previous
halt semantics.
The lifecycle test moves to the promesa predicates and a new test
covers the error-resilience promise: the first sample runs, throws,
and the next one is still scheduled.
AI-assisted-by: deepseek-v4.1-flash
* ♻️ Tighten the http metrics samplers
The samplers are leaf functions: they receive what they need and
publish it. Drop the internal nil guards (if there is no metrics
instance or no mxbean there is nothing to call them for) and move the
checks to the boundary, where the optional data is resolved:
sample-http-metrics now short-circuits with some-> and when-let.
Write the four worker gauges as four static operations instead of a
vector of pairs walked by doseq: the set is fixed, so the collection
only adds an allocation and hides each operation.
Drop the ! suffix from the sample-*-metrics family: ! marks a function
whose contract is to mutate state, while these report, and the mutation
happens in the mtx/run! they call. The constant true return, which only
existed so the removed guard tests could assert it, goes away too.
Tests follow the move: the internal-guard tests are replaced by one
boundary test (a nil server publishes nothing).
AI-assisted-by: deepseek-v4.1-flash
* 📚 Add the function design rules memory
Document the rules that came out of the http metrics sampler review:
preconditions are checked at the boundary instead of re-checked in the
core, optional-by-design data is guarded where the optionality is born,
a fixed set of operations is written statically, ! marks mutation and
not reporting, and production code is not shaped for tests.
Also state in the memory maintenance guide that memories must not use
manual line wrapping.
Linked from critical-info so it is read when designing a solution or
an API, not only when touching the samplers.
AI-assisted-by: deepseek-v4.1-flash
* 📚 Unwrap the critical-info memory lines
The memory maintenance guide forbids manual line wrapping, so rewrite
critical-info with one line per bullet and paragraph. A stray `*` at
the start of one continuation line is dropped.
AI-assisted-by: deepseek-v4.1-flash
* ♻️ Drop the redundant guard in the http server halt
create-metrics-sampler always returns the scheduler, so the sampler is
always present when integrant calls halt-key!; the nil check was dead
code, same as the yt/stop! call next to it.
AI-assisted-by: deepseek-v4.1-flash
* ✨ Add srepl helper to delete profiles by email
Add `delete-profiles-by-email!` to app.srepl.main. It accepts a
single email, a comma separated list of emails or a coll of emails,
resolves each profile, logs it to audit and enqueues the
delete-object task. The deleted-at is backdated with the configured
deletion-delay so profiles and their owned teams are purged on the
next gc pass.
Extract the per-email deletion logic into a private fn and reuse it
from `delete-profiles-in-bulk!`. Add tests for the new
`parse-emails` helper.
AI-assisted-by: glm-5.3-flash
* ✨ Add storage operation metrics for S3 and buckets
Expose Prometheus metrics for the object storage subsystem.
The S3 backend now attaches an AWS SDK MetricPublisher that counts
API calls, retries and latency per operation and target. The storage
layer counts logical operations and deduplication outcomes per Penpot
bucket, and the assets handlers count served requests per route.
Closes#11676
AI-assisted-by: muse-spark-1.3-contributor
* ✨ Fix storage metrics labels, errors and test gaps
Address the review findings on the storage metrics commit.
Label reads with the object's own backend, count failed asset
serving as errors without swallowing them, and cover the failed
S3 call, S3 asset path and permission-denied branches with tests.
Also share the label helper and reuse the metrics test helper.
Closes#11676
AI-assisted-by: muse-spark-1.3-contributor
* ✨ Harden storage metrics and fill test gaps
Address the second-round review findings on storage metrics.
Unknown backends now fail explicitly and count as errors, exists
stays paired with its dedup outcome, and the thumbnail, missing
storage, expired reads, unknown buckets and write failure paths
are covered by tests. Label coercion goes through the shared
metrics helper.
Closes#11676
AI-assisted-by: muse-spark-1.3-contributor
* ✨ Harden storage metrics accuracy and coverage
Address the full-branch review findings on storage metrics.
Touch and delete emit only on changed rows, reads emit after the
backend fetch, unknown backends fail explicitly, and tempfile
mismatches count as unauthorized. Publisher nil policy, pairing
rules and attempt semantics are documented and covered by tests.
Closes#11676
AI-assisted-by: muse-spark-1.3-contributor
* ✨ Address full-branch review findings on storage metrics
Touch and delete resolve labels from the row, reads stay paired,
failures are covered by tests, and logging, ranges and docs are
tightened. Includes the label helper unit tests and the retries
wording clarification.
Closes#11676
AI-assisted-by: muse-spark-1.3-contributor
* ⚡ Label touch and del metrics from UPDATE RETURNING
The storage metrics change resolved metric labels for touch-object!
and del-object! with an extra SELECT per id-based call. Since app.main
instruments storage unconditionally, every GC collector and binfile
import paid that extra round trip: deleting a team with 10k media
objects doubled the storage_object statements exactly on the paths
that already process the most rows.
touch-object! and del-object! now take only the object id (UUID) and
read the labels from the updated row itself via RETURNING id, backend,
metadata: one statement, no pre-read, and labels that always match the
row actually mutated. del-object! additionally guards on deleted_at
IS NULL, so a repeated delete returns false and emits no metric.
Also from the review of the full branch: extract the duplicated
serve/emit/rethrow block in app.http.assets into one helper; give
penpot_storage_s3_timing explicit histogram buckets up to 60s (the
default cap at 7.5s hid the slow S3 calls the metric exists for);
drop the unused ::target-id config key from the S3 backend and
hardcode the :default target label until per-bucket routing lands.
AI-assisted-by: glm-5.3-flash
* ✨ Harden storage metric recording and definitions
The metric definition schema is now closed and declares every key
the collectors read: buckets, quantiles, max-age and reg. A typo
such as a misspelled ::mdef/buckets used to compile and silently
fall back to the default histogram buckets; it now fails at
startup.
The asset result-label fallback coerced an absent status to 500,
so a future serve path without a status would have counted
successes as errors. The mapping is now explicit and documented:
served below 400, unauthorized for 401/403, not-found for 404,
and error for everything else, including an absent status.
The never-fail try/catch around metric recording existed four
times with drift. One app.metrics/run-safe! helper replaces them:
it no-ops on a nil metrics instance and logs the first failure
per hint at warn level, then at debug, so a broken setup surfaces
once without flooding the log. The S3 publisher keeps its outer
try/catch: it is the SDK MetricPublisher contract boundary.
AI-assisted-by: glm-5.3-flash
* ✨ Make metrics mandatory and run! safe by default
Recording a metric must never change the behavior of the operation
being measured, so `run!` now catches recording failures itself: the
first failure per metric id logs at warn, later ones at debug. This
replaces the `run-safe!` helper, whose four copies had drifted, and
applies the guarantee to every emit site instead of only storage.
The metrics instance precondition is a plain assert, and the collector
lookup stays outside the recording guard, so a missing instance fails
hard even when asserts are disabled. Metrics is therefore no longer
optional: the storage, s3-backend and db-pool schemas require
`::mtx/metrics`, and the assets handler cfg always carries it.
`wrap-publisher` no longer returns nil for a nil instance, and the db
pool wires the prometheus tracker unconditionally.
AI-assisted-by: deepseek-v4.1-flash
