`apply-add-page` sent the new Page node through `nodes/validate-node`,
which checks a map against the registry schema and returns it unchanged.
Every other node on both write paths goes through
`nodes/project-attrs`, which also selects the projected keys and is the
single place a column-level rule can live. A rule added there reached a
rebuilt page and not a synced one.
AI-assisted-by: mixed models
Cold projection and incremental sync are two implementations of one
mapping and nothing checked that they agree. They did not.
`backend-tests.graph-sync-parity-test` projects a file into one
`:memory:` database, applies a change list to that database and the same
list to the file data, projects the result into a second database, and
diffs the two down to the row and the column. It found four
disagreements, each fixed here.
**Sibling order was inverted.** A container's stored `:shapes` list runs
bottom to top and `IsChildOf.position` numbers children in Penpot
z-order, so appending to the list means taking position 0 and pushing
every sibling up. Sync instead handed each new child the next free
number, so any container edited live carried its children in the
opposite order to a rebuild, and a delete left a gap where a rebuild
renumbers densely. `insert-position` and `renumber-siblings` put the two
paths on the same rule for `:add-obj`, `:mov-objects` and `:del-obj`,
including a block move and `:after-shape`.
**A moved shape kept its old parent.** `:mov-objects` moved the edge and
left the shape's own `parent_id` and `frame_id` columns pointing at the
container it came from. Both now follow, and `frame_id` follows through
the whole subtree the shape carries, as
`app.common.files.changes` does for `:mov-objects`. A top-level shape's
column holds `uuid/zero`, the page's root frame, while its edge points
at the Page.
**A container's `shapes` column went stale.** Nothing maintained it
after an add, a move or a delete. It is now rebuilt from the sibling
order on every change that touches a container.
**Pages came out backwards.** `projection-data` reversed `:pages` before
numbering them, which is right for child shapes and wrong for pages:
`:pages` is the tab order and has no second ordering to undo. `Page.index`
and the page's `IsChildOf.position` are now that order.
One defect the test does not reach, fixed on the way past:
`index-add-shape!` accepted `:component-ctx` and dropped it, so a shape
added under an instance head added in the same session inherited no
`component-id`.
AI-assisted-by: mixed models
`debug/query-session!` ran whatever it was handed against the session
connection. A session graph is a projection of a file, rebuilt from that
file by Reload, so a mutation from the console produces a graph no
rebuild reproduces and no query result explains.
Bind the statement against the live schema first. A statement that does
not bind reports the binder's own message and executes nothing, which
also turns a misspelt table or property into an immediate error instead
of an empty result. A statement that binds runs only when the engine's
own read/write analysis calls it read-only.
The console's query box is labelled read-only. Load, Reload, Unload and
live sync are unaffected: they are separate handlers and do not go
through this path.
AI-assisted-by: mixed models
The graph namespaces explained themselves by citing a separate project
whose Python pipeline reads the graphs this backend writes. A reader of
this repository does not have that project and should not need it, and a
docstring that justifies a choice by pointing elsewhere cannot be checked
here.
Every claim survives; only the framing changes. Column names and types
are Penpot's own decision, recorded with the reason for each divergence
from the snake_case default. The transform registry describes the edges
it materializes. The denormalizations in `app.graph.project.document`
are justified by the walk already holding both answers.
Three corrections fall out of the rewrite:
- `app.graph.schema.contract` claimed a test, `graph_contract_test`,
that walks a checked-in schema manifest and fails on any divergence.
No such test exists. The paragraph is gone.
- `app.graph.project.document` pointed at
`app.graph.meta/projection-transforms`, which does not exist.
- `app.graph.project.transforms/registry` claimed its entries were "in
application order" while `apply-transforms!` reduced over the literal
vector. The three registered transforms read disjoint columns, so the
order is not load-bearing. The docstring now says so, and the one real
ordering constraint is stated where it applies: `link-swap-slots!`
strips `swap-slot-*` entries from `touched`, so anything reading
`touched` has to run before it.
`contract/pending-beadpot-columns` becomes `contract/unprojected-keys`.
It is referenced nowhere else.
AI-assisted-by: mixed models
`node-batch` named every top-level Arrow field with backticks, so that a
column whose name is a reserved word (`Page.index`, `Document.options`)
survived the DDL Ladybug generates for a staged table. The engine now
quotes those identifiers itself, and it does not collapse a doubled
backtick, so a pre-quoted name reaches the parser as ``index`` and
`createArrowTable` fails outright:
Parser exception: mismatched input '``' expecting PRIMARY
Name the fields with `column-name`. The `COPY` projection is Cypher
rather than DDL and keeps its own backticks through
`cypher-property-key`, and STRUCT member names keep theirs too: those
come out of `LogicalType::toString()`, which the DDL builder does not
touch, so an unquoted member called `column` still fails to parse.
Measured with `probes/arrow/probe25.clj` against lbug 0.19.1: a plain
top-level reserved word loads and reads back, a pre-quoted one fails to
parse, a plain STRUCT member fails to parse, and a pre-quoted one loads
and reads back.
Also re-dates the engine facts in the `app.graph.arrow` docstring to the
version they were checked against, drops the SIGSEGV note from
`->param-value` now that `Connection.execute` rejects an unwrapped
parameter, and removes two references to the CSV loader.
AI-assisted-by: mixed models
`set-document-revision-statement` emitted `SET d.revn`, but the column is
`revision`: the beadpot contract renames `:revn` and the DDL has followed
it since. The statement is the last one in every sync batch, so each
batch raised after its mutations had already committed, and the session's
in-memory index stayed frozen at its load-time revision.
Name the column through `nodes/cypher-property-key` rather than spelling
it, so the DDL and the statement cannot disagree again.
Found by the binder gate in the next commit, on its first run.
AI-assisted-by: mixed models
`app.graph.ladybug` could only run Cypher as text. Every value the sync
path writes is therefore concatenated into the statement, and nothing can
ask the engine whether a statement is even valid without running it.
Add the four functions that close both gaps. `prepare-on-connection!`
parses and binds without executing. `execute-prepared!` binds a parameter
map and runs it. `exec-prepared-on-connection!` prepares every statement
in a batch before executing any of them, so a parse or bind failure
aborts before the first mutation. `validate-on-connection!` returns
`{:ok? :error :read-only?}` instead of raising, which is what a gate
wants.
`->param-value` is the only `Value` constructor on the write path. It is
unconditional: on lbug 0.18.2 an unwrapped parameter does not raise, it
SIGSEGVs the JVM inside `lbug_value_clone`. Parameters are scalars only,
because the JNI `Value` constructor takes no list or map, so `MAP`,
`STRUCT` and `T[]` columns stay literal-rendered and the `:else` branch
raises rather than crashing.
Two departures from the design, both closing a JNI-handle leak on the
error path: `prepare-on-connection!` closes the failed
`PreparedStatement` before raising, and `execute-prepared!` closes every
`Value` it built, including the ones built before a later parameter was
rejected.
`as-statement` accepts a bare string, so the sync builders can convert to
bound parameters one family at a time rather than in one commit.
AI-assisted-by: mixed models
app.graph.arrow stages rows as Arrow VectorSchemaRoots and COPYs from them. No file is written at any point and no value is rendered as text for the engine to re-parse, so the defect class that produced three of this branch's four backend defects cannot recur.
app.graph.bulk is deleted whole. csv-representable?, defer-to-cypher?, multiline?, fixup-statements, ladybug-literal, ladybug-list-element, ladybug-list-cell and staging-dir go with it, along with the post-COPY Cypher pass that emitted one SET per row.
Measured before deciding: the fixup pass was ~77% execution, 16-22% parse and 6-7% round-trip, and prepared statements could not have recovered any of it — every fixup row carries a MAP column and Ladybug binds scalars only. So this replaces rather than optimizes. Marginal ingest 4.0 -> 1.21 ms/shape; ~25 s extrapolated at 20k shapes against the ~2 min the CSV path projected. Size unchanged.
Four engine facts the implementation rests on, each verified against 0.18.2 with a standalone probe:
- An Arrow table is not a COPY source identifier but is a MATCH-able node label.
- A MAP vector's entries child must be a non-nullable struct, and MapVector.getWriter promotes it to a sparse union, so map vectors are built from an explicit Field and filled child-first.
- Ladybug names a staged table's columns and struct fields from the Arrow field names and quotes none of them, so anything needing quotes must arrive quoted — hence cypher-property-key, not column-name, names the Arrow fields.
- createArrowRelTable cannot resolve endpoints against a UUID-keyed node table under any encoding, so edges stage as a node table and the COPY subquery joins them.
values/coerce is reused unchanged, so the Arrow and Cypher writers cannot disagree about a value's shape; nodes/column-map-key-fn is extracted so they cannot disagree about a MAP's key spelling either.
Verified with pytest --graph-origin=penpot-only unchanged at 225/38/1 and --graph-origin=penpot unchanged at 258 passed / 2 pre-existing failures, both baselines re-established against a reverted backend rather than assumed; with bp graph diff between a CSV-built and an Arrow-built graph reporting "Graphs agree"; and with an adversarial round-trip carrying a quote, a backslash, a newline, a CRLF and a tab through STRING, STRING[] elements and MAP values.
The diff was necessary, not belt-and-braces: both parity suites passed an earlier revision of this change that was writing EDN into every JSON column, because beadpot's assertions never parse those columns. It also showed Arrow correcting a CSV defect — an empty Component.path was being stored as NULL, because Ladybug's CSV reader cannot distinguish an empty field from an absent one.
Split out of "🐛 Declare the shape attributes stored files carry",
which is now #11125 and carries only its `common/` half. This commit is
the graph's own side of that change, and it stays on this branch.
`app.graph.schema.contract` pins `svg_viewbox` to `DOUBLE[4]` and
`svg_transform` to `DOUBLE[6]`. The shape schema types both `:map` on
purpose, because legacy files hold them as plain maps rather than as
`::grc/rect` and `::gmt/matrix` records, and a tighter *schema* would
reject those files. A tighter *column* costs nothing, since
`app.graph.schema.values/coerce` reads either form.
`app.graph.schema.nodes` declares four file-level attributes as
projection `:extra` rather than in `ctf/schema:file`: `:options`,
`:backend`, `:comment-thread-seqn`, and `:ignore-sync-until`. Declaring
them in the file schema breaks saving, measured at 185 failures, because
`app.binfile.common/update-file!` derives its UPDATE column list from a
file map's keys and the `file` table has no `backend` column, that value
being synthesized on read. An `:extra` is local to the graph and cannot
reach a write.
`app.graph.project.document` lifts `:options` out of `:data` before the
blob is dropped, so a consumer reads file-level configuration without
opening the blob.
AI-assisted-by: mixed models
Ladybug is schema-first and strongly typed: a property key gets its type
at table-creation time and there is no widening later. That makes the
Malli to Ladybug mapping the whole of the graph's typing, and it was
leaving a lot on the table: a transform stored as `STRING`, a rect as
`JSON`, a set of feature flags as a single `STRING`. A column typed
`DOUBLE[4]` is four numbers a consumer reads as a tensor row; the same
value as JSON is text somebody has to parse and trust.
`app.graph.schema.types` now maps, in order: scalars; Penpot value types
whose layout is fixed even though Malli only sees a map or a string
(`::gmt/matrix` to `DOUBLE[6]`, `::gpt/point` to `DOUBLE[2]`,
`::grc/rect` to `DOUBLE[4]`, `::clr/hex-color` to `UINT32`); then
structure, with collections to `T[]`, `:map-of` to `MAP(k, v)`, and a
closed map of scalars to a `STRUCT`. JSON is the fallback of last
resort, for schemas that genuinely admit more than one shape.
Two defects fell out. `::sm/set` was unmapped, so `features` and
`migrations` were single strings rather than `STRING[]`, and
`::sm/one-of`, how Penpot spells a closed set of keywords, was unmapped
too, so `blend-mode`, `grow-type`, the constraints and every `layout-*`
were mistyped.
A tight column is only worth having if the writer fills it in that
shape, so `app.graph.schema.values` shapes a value for its type: a
matrix record into six doubles, a hex colour into a packed integer, a
map into a struct's fields. Both writers go through it, so the bulk load
and the incremental sync cannot disagree. What that required:
- STRUCT field names must be backticked in the DDL *and* in every
literal, because a grid cell has a field named `column`. The catalog
reports them bare.
- A struct literal's type is its field list, so every declared field
must appear, and an absent one needs `cast(NULL, '<type>')`. A bare
NULL is typed STRING and changes the struct's type.
- `STRUCT(…)[]` starts with `STRUCT(` but is a list, so the list check
comes first.
- Nested lists cannot be rendered with `str`: Clojure's `[1 2]` is
space-separated and Ladybug reads it as a one-element array.
Three more corrections in the same area:
- `project-attrs` used truthiness where it meant `some?`, so `opacity 0`
and `blocked false` projected as absent.
- Set-valued columns are written sorted. A set has no order, so the
column varied between builds of the same file, which is precisely what
stops two builds being diffable.
- An empty collection is written as `[]` rather than skipped. A shape
with no fills has none; NULL would say "unknown".
Renamed the `kuzu-*` helpers to `ladybug-*`: Kùzu is deprecated and
Ladybug substitutes it, so a name bearing the engine should bear this
one. The one remaining mention cites the upstream issue Ladybug
inherits.
AI-assisted-by: mixed models
Three parity failures against beadpot's suite, all one cause: the bulk loader
put compound and multi-line values into CSV, where Ladybug parses a field's
*contents* as a literal with no escape mechanism at all. Verified against
0.18: a comma inside a list element ends the element, quotes are kept as part
of the value rather than delimiting it, and the parallel reader rejects
quoted newlines outright.
So a value now goes through CSV only if it cannot be misread there — UUIDs,
numbers, booleans, single-line strings, and lists of those. Everything else
(MAP, STRUCT, STRING[]/JSON[], any string containing a newline) is written
after the COPY by one Cypher statement per row, where `app.graph.ladybug`
escapes properly. Parquet removes the distinction entirely and is still the
right destination (masterplan P0 T1); this is what CSV can honestly do.
Consequences beyond the encoding:
- `touched` entries reached the graph as `:swap-slot-…`, keywords stringified
with their colon, so `LinkSwapSlots` matched nothing. Keywords now render
through `name`.
- Shape names lost their newlines to a flattening step that existed only to
keep the CSV writer happy. They are preserved.
- `applied_tokens` keys are rendered camelCase, the form Penpot's own JSON
encoder produces and the one beadpot's `AppliedTokenKey` holds — a MAP
column's keys are values, not schema, so they are not snake_cased.
- `link-component-instances!` keys on `component-file`, not `component-id`
alone. The projection denormalizes `component-id` down the shape tree, after
which it no longer tells an instance head from a shape inside one, and the
transform linked every descendant frame; `ctk/instance-of?` requires both
keys anyway. IsInstanceOf on the variants fixture: 78 -> 60, matching
beadpot exactly.
`app.graph.schema.nodes/format-column-value` is now the single place that
knows a column's type and its contract details, used by the bulk loader and
the incremental sync alike so the two cannot disagree about a value's shape.
A projected graph is a cache of one file at one revision, built by one
schema, and nothing in it said so. `GraphMeta` records the file, the
revision, the schema version and the producer, and is written last, so
its presence also marks the build complete and its contents say whether
a cached database is still worth opening.
- `graph/meta.clj`: the `GraphMeta` table and its writer.
- `graph/schema/contract.clj`: one place that maps a Penpot key to its
graph column. The rule is snake_case of the key; every exception, be
it a rename, a drop or a type override, is recorded there with its
reason, so a divergence is a diff to review rather than a silent
rename.
- `graph/project/document.clj`: `page-id` and the inherited
`component-id` are written during the tree walk, which already knows
both, rather than by a post-ingest statement. `graph/sync.clj` does
the same on the incremental path, so a live-synced graph matches a
rebuild.
- `graph/project/transforms.clj`: a registry, so adding a derived-link
pass is one entry. Adds `RefersTo` (from `shape-ref`) and
`FillsSwapSlot` (from `swap-slot-*` entries in `touched`, then
stripped as `ctk/normal-touched-groups` does).
- `graph/debug.clj`, `graph/stats.clj`: enumerate relationship tables
from the catalog instead of naming them, so the console's graph view
and the ingest counts pick up new edge types without being told.
- `graph/debug.clj`, `http/debug.clj`: `graph-export` gains
`source=session`, which snapshots the live in-memory console graph
through EXPORT/IMPORT DATABASE. Live sync moves that graph away from a
fresh projection, and taking it away to query elsewhere is the point
of asking for it.
AI-assisted-by: mixed models
`app.graph.ladybug` imports `com.ladybugdb.*` at namespace load. Two
namespaces reach the subsystem and both required it at the top level:
`app.http.debug`, which registers the `/dbg` routes, and
`app.srepl.main`, which loads with the REPL server. Every backend built
from this branch therefore linked the Ladybug native library into the
JVM at boot, whether or not a graph was ever used.
Add a `:graph` flag to `varia`, deliberately absent from `default` so
that a released Penpot ships with the subsystem off. Both require sites
now resolve `app.graph.*` at call time, so with the flag off no
`com.ladybugdb` class is loaded. The nine `/dbg` graph routes are
registered only when the flag is on, and 404 otherwise. The `/dbg` admin
gate is untouched: the flag decides which routes exist, not who may
reach them. When the flag is on, route init requires the subsystem
eagerly, so a missing or unusable native library fails the boot rather
than the first console request.
No tracked file turns the flag on. `backend/scripts/_env` leaves it out,
so a devenv boots with the subsystem off exactly as a released build
does, and `docker/images/docker-compose.yaml`, the self-hosting
distribution, is untouched. Whoever works on the graph turns it on for
one checkout through the gitignored `backend/scripts/_env.local`, which
every backend and exporter dev script sources right after `_env`.
Verified with `-verbose:class` over a boot's namespace load plus
`ig/init-key ::routes`: 9 `com.ladybugdb` classes before this change
with no flag set, 0 after it with the flag off, 9 with `enable-graph`.
develop renamed app.main/system to app.system/system and dropped the
app.main require while this branch was away. Rebasing replays the old
call, so clj-kondo reports an unresolved namespace and the ns will not
load.
graph-data gains bm-bytes (CALL bm_info() -> [mem_limit mem_usage], nil-safe, under the session lock); the session panel shows it as MiB behind the node/edge counts — real resident memory replacing the removed estimate.
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
The default query is now multi-line with // comments that explain the filter_* column convention in place (Kuzu accepts comments and blank lines mid-statement; verified against an in-memory database through the console query path). The query fieldset is retitled 'LadybugDB Cypher' with the Cypher word linking to https://docs.ladybugdb.com/cypher/.
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
Graph view gains an on-canvas G6 toolbar (auto-fit, expand, restore - the fullscreen icons drive the existing in-page expand), an 'animate' checkbox that disables animation unconditionally when off (persisted, adaptive <=100-node rule applies only when on), and a ResizeObserver on the canvas so the panel follows window/flex resizes without touching the user's viewport. Preset tree positions are now only injected for the built-in tree layout, removing the tree-then-layout flash on animated re-renders under G6 layouts. Refetches skip the repaint when the display projection (nodes, edges, truncated) is byte-identical, so attribute-only change bursts no longer repaint.
Console: default query returns s/t name+label over all edges plus filter_src_id/filter_tgt_id columns; filter_* columns are hidden from the results table (client and server render) but still feed the 'Show result in graph view' id harvest, keeping the table legible while the graph filter stays available. The query text persists in localStorage across page reloads (restored only over the default, never over a server-rendered query). Legend shows colored Unicode glyphs matching node shapes instead of squares with textual annotations. Load/Unload buttons share one row (HTML5 form attribute), and the loaded file name links to the Penpot workspace via the legacy /#/workspace/<project-id>/<file-id> route resolved client-side from the files-tree payload.
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
Graph view moves to its own sticky right column (overrides .widget max-width). New /dbg/actions/graph-files endpoint lists teams -> projects -> files for the profile; the console renders it as a collapsible tree where clicking a file loads it. Maximize button fullscreens the graph panel and resizes G6 on fullscreenchange.
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
COPY failed on any file with container shapes: list-typed DDL columns (shapes UUID[], points STRING[], strokes JSON[], ...) were JSON-encoded in staging CSVs, which Ladybug's list parser rejects. Write Kuzu list literals instead, typed per column. Also: value->clj no longer crashes on LIST/STRUCT values (binding lacks value_get_value support; fall back to string), and the debug session Connection is now guarded by a per-session lock — it was shared unsynchronized between the msgbus sync loop and HTTP query/export handlers, and one lost DETACH DELETE was observed under concurrent refetch load.
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
POC per work/g6/plan.md. New /dbg/actions/graph-data exports the in-memory Ladybug session as plain JSON (per-table node queries + multi-table IsChildOf match, row cap 100k with truncation flag). Console page renders it with AntV G6 v5 (jsDelivr CDN, antv-dagre BT layout, color+glyph per node table, validated palette) and refetches debounced on live :file-change messages.
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
`create-font-variant` destructures `uploads` and never reads it: the
handler passes the whole `params` map to `prepare-font-data-from-uploads`.
`clj-kondo` reports it as an unused binding and exits 2, which fails the
Lint step of the Backend workflow, and the Lint step runs before the
tests, so no branch based on `develop` can run the backend suite at all.
AI-assisted-by: mixed models
Replace the inline organization map in schema:create-organization-invitation with cto/schema:organization-with-avatar, eliminating schema duplication and fixing mismatched validation rules for :logo and :sso-active fields.
AI-assisted-by: mimo-v2.5
Prevent cross-team font injection by checking that when a font-id
already has variants, they belong to the same team. This closes a
BOLA gap where a user with team edit permissions could create a
font variant referencing a font-id from another team.
AI-assisted-by: mimo-v2.5-pro
* 🐛 Add backend password validation with complexity rules and dictionary check
Enforce minimum 8-character password length, require at least 1 lowercase
letter, 1 uppercase letter, 1 digit, and 1 special character, and reject
common passwords using Passay library with a 10k-entry wordlist from
SecLists during registration and password change flows.
AI-assisted-by: mimo-v2.5-pro
* ✨ Improve user feedback
When the password is invalid, the user now gets extra indications to make it stronger, so it can be valid.
* 🐛 Fix remove unneeded common password check
The dictionary check is only relevant for passwords that meet all other requirements, but all 10,000 common passwords would fail the character requirements, so this check is not needed
---------
Co-authored-by: Luis de Dios <luis.dedios@kaleidos.net>
Remove :skip-ssrf-check? true from prepare-organization-sso-provider so
SSRF protection is active when validating organization SSO configs.
The endpoint is already protected by shared-key authentication
(admin-console), but enabling SSRF protection prevents potential misuse
of internal network resources if the shared key were ever compromised
(defense-in-depth).
Add test prepare-organization-sso-provider-does-not-skip-ssrf-check to
verify the SSRF check is not skipped.
AI-assisted-by: qwen3.7-plus
Add sanitize-svg function that removes dangerous elements and attributes:
- script tags
- foreignObject elements
- Event handler attributes (onload, onmouseover, etc.)
- javascript: URLs from href/xlink:href attributes
Apply sanitization in process-main-image before storing SVG files.
AI-assisted-by: mimo-v2.5-pro
Add normalize-string helper in app.common.data that trims whitespace
and returns empty string for nil input. Apply to profile, team, and
project string fields (fullname, lang, theme, name) before storage.
AI-assisted-by: qwen3.7-plus
Capture unique constraint violation in insert-file! and return
generic :not-found error instead of propagating raw PostgreSQL
exception, preventing file existence oracle.
AI-assisted-by: mimo-v2.5-pro
Add authorization check to generic-handler in assets.clj so that
/assets/by-file-media-id/:id and its /thumbnail variant verify the
requesting profile has read access to the parent file. Return 404
(not 403) when access is denied to avoid confirming existence.
Also switch get-file-media-object from db/get to db/get* so that
non-existent media objects return nil instead of raising.
AI-assisted-by: mimo-v2.5-pro
