mirror of
https://github.com/penpot/penpot.git
synced 2026-08-29 16:18:48 +00:00
61 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Álvaro Tejero-Cantero
|
38004e6bb2
|
✨ Add the graph subsystem and graph visualization console to the backend (#11101)
* 🎉 Basic lbug connection for ingestion * ✨ Add Penpot-to-Ladybug graph ingest vertical slice * ✨ Use embedded Ladybug Java API instead of CLI * ♻️ Share Ladybug connection across ingest and stats * ✨ Validate graph ingest projections with Malli * ✨ Project nested shapes recursively into the graph * ⚡ Load graph ingest via Ladybug COPY bulk import * 🐛 Fix graph COPY ingest for multiline text names * ✨ Add Ladybug graph export to debug UI * ✨ Add debug graph console for in-memory Cypher queries * ✨ Add live file-change feed to debug graph console * ✨ Incrementally sync debug graph from Penpot file changes * ✨ Handle mov-objects in debug graph sync * 🐛 Fix batch delete sync and keep graph console feed alive * ♻️ Derive graph node schema from Malli registry * ✨ Add G6 graph view to debug graph console POC per work/g6/plan.md. New /dbg/actions/graph-data exports the in-memory Ladybug session as plain JSON (per-table node queries + multi-table IsChildOf match, row cap 100k with truncation flag). Console page renders it with AntV G6 v5 (jsDelivr CDN, antv-dagre BT layout, color+glyph per node table, validated palette) and refetches debounced on live :file-change messages. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 🐛 Fix list-column CSV ingest and serialize graph session access COPY failed on any file with container shapes: list-typed DDL columns (shapes UUID[], points STRING[], strokes JSON[], ...) were JSON-encoded in staging CSVs, which Ladybug's list parser rejects. Write Kuzu list literals instead, typed per column. Also: value->clj no longer crashes on LIST/STRUCT values (binding lacks value_get_value support; fall back to string), and the debug session Connection is now guarded by a per-session lock — it was shared unsynchronized between the msgbus sync loop and HTTP query/export handlers, and one lost DETACH DELETE was observed under concurrent refetch load. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Split graph console in two columns; add file tree and fullscreen Graph view moves to its own sticky right column (overrides .widget max-width). New /dbg/actions/graph-files endpoint lists teams -> projects -> files for the profile; the console renders it as a collapsible tree where clicking a file loads it. Maximize button fullscreens the graph panel and resizes G6 on fullscreenchange. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ♻️ Replace fullscreen with in-page expand for graph view Fullscreen API took over the whole output and broke window-manager splits (and is denied in some environments). The Expand button now toggles a fixed-position overlay covering the page while keeping browser chrome; Esc restores. Column positioning moved from inline style to the stylesheet so the expanded class can override it. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Fold containers as collapsible combos in graph view Non-empty containers (Page, Frame, Group, Boolean, SVGRaw) render as nested G6 rect combos holding their own node plus direct children; Document stays a plain node. Double-click folds/expands (collapse-expand behavior); collapsed combos show a member count and re-route child edges. Fold state is read back from getComboData and re-marked on every refetch, so it survives live redraws. Layout gains sortByCombo to keep same-rank nodes grouped by box. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ⚡ Fix graph view freeze on large files; add fold toggle and root rule Root cause of the tab freeze on ~1700-node files was G6's default entrance animation: measured 1700 nodes at >2 min animated vs 1.5 s with animation: false. Secondary cost was antv-dagre (~7 s at that size); since IsChildOf is a tree, an O(n) tidy layout (depth = rank, post-order leaf slots, parents centered) computed client-side replaces it and renders the same file in ~1.4 s. A guard skips auto-render above 4000 nodes with an explicit Render-anyway button, so opening the console with a huge session loaded stays responsive. Folding is now switchable ('fold containers' checkbox, persisted in localStorage) and generalized: any node with children folds except the IsChildOf root of the loaded graph, so Documents (and later Projects/Teams) fold automatically once they gain a parent node. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Add layout dropdown to graph view Adds a layout <select> next to the fold toggle, populated from the LAYOUTS map in the template: 'tree' (the O(n) preset layout, default) plus 13 G6 layouts (antv-dagre, dagre, circular, concentric, radial, grid, force, d3-force, force-atlas2, fruchterman, mds, combo-combined, random), all smoke-tested against combo data on this UMD build. Layout and fold toggle are independent; switching layouts recreates the graph instance (cheap with animation off); both choices persist in localStorage. antv-dagre stays available for when non-tree edges arrive. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Add query-result subgraph, ws auto-reconnect, adaptive animation The Cypher result pane now offers 'Show result in graph view': any UUID found in any result cell selects the matching nodes in the cached export and the view renders the induced subgraph (edges kept when both endpoints match); 'Show full graph' resets. No graph reconstruction from the query result is needed. The notifications websocket reconnects automatically (3 s retry) and resubscribes + refetches on reopen, so backend restarts no longer permanently kill the live feed; a lost session now reports 'no graph session (backend restarted?) - reload a file' instead of a bare 404. Animation is size-adaptive: graphs (or filtered subgraphs) up to 100 nodes render animated for didactics, larger ones stay animation-free; crossing the threshold recreates the instance like a layout switch. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Add graph toolbar, animate toggle, filter columns, repaint skip Graph view gains an on-canvas G6 toolbar (auto-fit, expand, restore - the fullscreen icons drive the existing in-page expand), an 'animate' checkbox that disables animation unconditionally when off (persisted, adaptive <=100-node rule applies only when on), and a ResizeObserver on the canvas so the panel follows window/flex resizes without touching the user's viewport. Preset tree positions are now only injected for the built-in tree layout, removing the tree-then-layout flash on animated re-renders under G6 layouts. Refetches skip the repaint when the display projection (nodes, edges, truncated) is byte-identical, so attribute-only change bursts no longer repaint. Console: default query returns s/t name+label over all edges plus filter_src_id/filter_tgt_id columns; filter_* columns are hidden from the results table (client and server render) but still feed the 'Show result in graph view' id harvest, keeping the table legible while the graph filter stays available. The query text persists in localStorage across page reloads (restored only over the default, never over a server-rendered query). Legend shows colored Unicode glyphs matching node shapes instead of squares with textual annotations. Load/Unload buttons share one row (HTML5 form attribute), and the loaded file name links to the Penpot workspace via the legacy /#/workspace/<project-id>/<file-id> route resolved client-side from the files-tree payload. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 🐛 Fix runaway graph panel growth and blank canvas; drop Expand button Root cause of 'graph flashes on load then disappears' plus unbounded horizontal growth of the graph panel: fieldsets default to min-inline-size: min-content, so #graph-view-panel sized to its content, and the new ResizeObserver->setSize path closed a feedback loop (setSize -> slightly wider G6 canvas -> wider fieldset -> wider .dashboard flex column -> observer fires) that grew the page ~10px per frame and wiped the painted canvas on every step. Fix severs the feedback path: #graph-view-panel gets min-inline-size: 0, #graph-canvas gets overflow: hidden, and the page section gets flex: 1 1 0 with min-width: 0 so column widths are viewport-driven, never content-driven. This also fixes the original narrow-window scrollbars defect for real. The observer stays (guarded by a current-size comparison) because G6's autoResize is inert on this UMD build (verified: window resizes left the canvas size untouched); the inert autoResize flag is dropped. Legend items now join with spaces so the nowrap spans can wrap between entries. Also removes the header Expand button - the toolbar's expand/exit icons cover it, Esc still restores. Verified against the running devenv with a logged-in profile and variants_simple loaded: graph renders and persists, widths stable over multiple seconds at 1400px and 1000px viewports with no horizontal overflow, canvas follows both window shrink and grow, toolbar expand gives a full-page canvas and Esc restores. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Make the default query self-explanatory; link the Cypher docs The default query is now multi-line with // comments that explain the filter_* column convention in place (Kuzu accepts comments and blank lines mid-statement; verified against an in-memory database through the console query path). The query fieldset is retitled 'LadybugDB Cypher' with the Cypher word linking to https://docs.ladybugdb.com/cypher/. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 📎 Fix linter issues * ✨ Add Component nodes and IsInstanceOf edges * 🐛 Fix memory leak * ✨ Style Component nodes and IsInstanceOf edges in graph console Slice-3 export sends edges with a rel field. Derive tree ranking, combo derivation and fold-ability from IsChildOf only; draw other rels as overlay edges with per-rel styles (EDGE_STYLES: IsInstanceOf violet dashed, matching the new Component diamond in NODE_STYLES). Legend now lists only displayed node tables and rels, re-rendered per redraw; help text trimmed to essentials. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Add graph diff marks with step fade to graph console Each display-changing refetch is a step: added nodes/edges get a green halo, removed ones stay as ghosts with a dashed crimson halo (nodes, fading opacity) or thicker crimson stroke (edges), re-entering layout and combos through their ghost IsChildOf edges. Marks fade linearly and drop after N steps; N is the new "fade" number input (localStorage, 0 = off). Dash + fade carry the added/removed distinction under red-green CVD (#40c057/#c2255c, deutan dE 17.4); diff is vs the previous display step, not arbitrary revisions. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 💄 Reserve chroma for changes in graph console diff mode Monochrome entity scheme: all node tables share one slate hue, lightness separates within-glyph siblings (validated, worst pair dE 17.5), SVGRaw becomes the hollow hexagon, both rels go grey with dash as the only separator. Diff marks now own all color: thick green/crimson stroke ring (dashed for removals) plus a larger, subtler halo; the legend gains +/- entries while marks are live. Two additions to guide the eye: a brief DOM-overlay pulse on age-0 elements (independent of the G6 animation gate) and a "fold unchanged" toggle that collapses every combo not on an ancestor path of a changed element. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 🐛 Expand folded combos that gain changed elements setData merges datum props by id on a live G6 instance, so omitting style.collapsed retained a previous true: with "fold unchanged" on, a change inside a folded combo pulsed but never expanded it. Write the boolean explicitly both ways. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Label edge rels with compact unicode symbols Dash variants alone cannot carry the growing rel roster: EDGE_STYLES entries gain a sym rendered as a small mid-edge label with a white backing (IsInstanceOf = "∈"; IsChildOf stays unlabeled as the background structure), and the legend shows the symbol. Convention from the abacus viewer EDGE_SYM dict. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Add node inspector panel to graph console Clicking a node fetches its full attribute row (MATCH (n:`Table` {id: uuid(...)}) RETURN n.*) through the query endpoint and renders non-null attrs into a panel under the canvas (count of empty attrs noted). Panel over tooltip: projected tables carry ~80 columns, and the panel persists for reading without obstructing the graph. Table/id are validated before Cypher interpolation; the listener is re-attached on every instance recreation. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 💄 Graph console QoL round "Show result in graph view" moves into an actions bar above the results table; results scroll inside a 45vh container (client and server render paths); the Loaded-session fieldset gains a live "Graph size" line that stays fresh through skipped repaints; IsInstanceOf mid-edge label becomes the spelled-out rel name (∈ read as membership, not derivation) with the legend falling back to the dash-arrow for long syms. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Sync Component library changes into the Ladybug graph * 💄 Polish graph console session panel and edge labels Loaded-session fieldset: graph size gains a resident-memory estimate (fit to graph_sizes.md: ~1.1 MiB floor + ~5.4 KiB/node) with per-table counts on hover, replacing the load-time Projection stats; loaded-at compacts to local HH:MM with the full instant on hover. Edge rel labels drop to 7 px and lose the dashed stroke — the text label alone carries rel identity. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Add PNG export and hover tooltips to graph console Toolbar gains an export item: graph.toDataURL({mode: "overall"}) downloads the whole laid-out graph as graph-<revn>.png — page-chrome-free captures, also the fast path for agents debugging the console. A hover tooltip (table, label, id) backs the reduced/absent labels on dense layouts. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 💄 Prune graph console layout roster and tune overlap Remove grid, random, force, fruchterman, force-atlas2 (nothing over the kept set) and mds (stress layout degenerates to spokes on tree distances, no collision term to tune). Parameterize the keepers against node overlap — concentric/radial get preventOverlap+nodeSize, d3-force a collide radius — and shrink node labels to 7 px on those layouts (DENSE_LABEL_LAYOUTS), verified against variants_simple (72 nodes). Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Highlight clicked node neighborhood in graph console click-select behavior with degree 1: the clicked element keeps a black ring, direct neighbors stay full-strength, everything else dims to 0.2 opacity (inactive state); clicking empty canvas clears. Works on edges too (selects both endpoints) and composes with the node inspector on the same click. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Add overview mode: fold containers at or beyond a depth "fold >= depth" number input (root = 0, empty = off, localStorage): every combo whose container sits at that IsChildOf depth or deeper collapses, giving a top-of-file overview (e.g. 2 folds the containers hanging from a Page). Composes with fold-unchanged — depth folds first, changed ancestor paths are then drilled open. Derived fold state overrides manual folds while active. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 🐛 Restore fold-containers as the combo master gate Since fold-unchanged and depth folding arrived, withCombos ORed them in, so unchecking "fold containers" could no longer remove the combo boxes. The checkbox is the gate again; the derived fold rules are dormant without it. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 💄 Console UI polish round Merge the load form and files tree into one "Load graph from Penpot" box (tree first, uuid + Load/Unload in a row); Loaded session carries HH:MM in its legend; the Live changes box stays hidden until the first change arrives; query fieldset reads "Query graph (LadybugDB Cypher)" with the link covering both terms. Drop the hover tooltips (distracting, useless zoomed out) and the resident-size estimate (per-table counts stay on hover); every toggle gets a "When set/checked ..." title. Depth fold: 0 now expands every container (no more hunting for max depth). Node inspector: two-column flow, structured or long values folded behind the file-tree disclosure triangle. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Report actual graph memory from the buffer manager graph-data gains bm-bytes (CALL bm_info() -> [mem_limit mem_usage], nil-safe, under the session lock); the session panel shows it as MiB behind the node/edge counts — real resident memory replacing the removed estimate. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 💄 Console control-bar and session-panel rework Left column narrowed 440->330 px (uuid input flexes). Control bar reordered: layout first, then animate and fade (narrow inputs), then the fold set; "fold containers" renamed "foldable containers" (on = foldable, not folded). Load becomes Reload once a session exists (same operation as the removed Full-reload button — load-session! on the current id; tooltip explains the fallback role) with Unload beside it. Session panel: revisions on one line ("ingested at N · graph now M", hover explains the difference), duplicate uuid after the file name dropped. Tried and rejected: fishbone (no positions on graph data) and compact-box (G6 tree layouts walk parent->child, IsChildOf points child->parent). Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Legend entries toggle node-table visibility Clicking a legend entry hides/shows that table across the view (struck-through while hidden, kept listed for re-enabling; pure client-side id filter through filteredGraphData, edges drop with their endpoints, ghosts respect it). Also: setting fold >= depth above 0 now switches foldable containers on — a positive depth was silently inert without combos. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * ✨ Enable the edge-bundling plugin Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 💄 Session breadcrumb, changelog colors, spacing File line becomes team › project › file (clickable) with the resident-memory figure beside it (moved up from the graph-size line; breadcrumb resolves from the files-tree payload, so files outside the profiles teams show plain). add-obj/del-obj in Live changes wear the canvas diff colors. Paragraph margins tightened above Feed; left column 330→350 px. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 🐛 Guard renders against heavy graphs; add ?safe escape hatch A heavy file could freeze the tab on load-and-render despite the animation gate: the render guard counted nodes only, and the edge-bundling plugin is iteration-heavy in edges. Guard now also trips on edges (8000), edge bundling only activates at <= 300 edges, and /dbg/graph?safe disables auto-render entirely (counts + "Render anyway"), so a page that hung can always be re-entered with the session intact. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 🔥 Remove the edge-bundling plugin Bundled edges render unsmooth and ugly on this build; the gating constant goes with it. Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 💄 One row per operation in the Live changes table Columns revn | op | id: the revn repeats across a batch, the op wears the canvas diff colors (shape/attrs detail on hover), and the id column shows the uuid last group with the full uuid on hover, or N/A for ops without a subject id (e.g. mov-objects). Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 🐛 Use app.system/system in the graph ingest helper develop renamed app.main/system to app.system/system and dropped the app.main require while this branch was away. Rebasing replays the old call, so clj-kondo reports an unresolved namespace and the ns will not load. * ✨ Put the graph subsystem behind a flag, off by default (#11075) `app.graph.ladybug` imports `com.ladybugdb.*` at namespace load. Two namespaces reach the subsystem and both required it at the top level: `app.http.debug`, which registers the `/dbg` routes, and `app.srepl.main`, which loads with the REPL server. Every backend built from this branch therefore linked the Ladybug native library into the JVM at boot, whether or not a graph was ever used. Add a `:graph` flag to `varia`, deliberately absent from `default` so that a released Penpot ships with the subsystem off. Both require sites now resolve `app.graph.*` at call time, so with the flag off no `com.ladybugdb` class is loaded. The nine `/dbg` graph routes are registered only when the flag is on, and 404 otherwise. The `/dbg` admin gate is untouched: the flag decides which routes exist, not who may reach them. When the flag is on, route init requires the subsystem eagerly, so a missing or unusable native library fails the boot rather than the first console request. No tracked file turns the flag on. `backend/scripts/_env` leaves it out, so a devenv boots with the subsystem off exactly as a released build does, and `docker/images/docker-compose.yaml`, the self-hosting distribution, is untouched. Whoever works on the graph turns it on for one checkout through the gitignored `backend/scripts/_env.local`, which every backend and exporter dev script sources right after `_env`. Verified with `-verbose:class` over a boot's namespace load plus `ig/init-key ::routes`: 9 `com.ladybugdb` classes before this change with no flag set, 0 after it with the flag off, 9 with `enable-graph`. * ⬆️ Take Ladybug 0.19.1 `com.ladybugdb/lbug` moves from 0.18.0 to 0.19.1, the current release on Maven Central. The engine fixes a SIGSEGV on an unwrapped parameter and moves parameter coercion out of JNI, so shipping 0.18.0 would land a native library into `develop` with a known crash already fixed upstream. Nothing else changes. This branch has no `app.graph.arrow`, so the top-level Arrow field-name backticking that 0.19.x retires does not exist here and there is no workaround to remove alongside the bump. AI-assisted-by: mixed models * 📎 Pin the graph console's G6 bundle to an exact version The console loaded `@antv/g6@5` from jsDelivr, a floating major range, so the JavaScript served into the page could change without a Penpot release. Pin it to 5.1.1, the version the range resolves to today. Where the dependency finally belongs is an open question for review: vendored into `backend/resources`, declared in `frontend/package.json` if the console moves out of `/dbg`, or left on the CDN. Pinning removes the floating-code problem without pre-empting that decision. AI-assisted-by: mixed models * ✨ Add graph provenance, column naming and two transforms A projected graph is a cache of one file at one revision, built by one schema, and nothing in it said so. `GraphMeta` records the file, the revision, the schema version and the producer, and is written last, so its presence also marks the build complete and its contents say whether a cached database is still worth opening. - `graph/meta.clj`: the `GraphMeta` table and its writer. - `graph/schema/contract.clj`: one place that maps a Penpot key to its graph column. The rule is snake_case of the key; every exception, be it a rename, a drop or a type override, is recorded there with its reason, so a divergence is a diff to review rather than a silent rename. - `graph/project/document.clj`: `page-id` and the inherited `component-id` are written during the tree walk, which already knows both, rather than by a post-ingest statement. `graph/sync.clj` does the same on the incremental path, so a live-synced graph matches a rebuild. - `graph/project/transforms.clj`: a registry, so adding a derived-link pass is one entry. Adds `RefersTo` (from `shape-ref`) and `FillsSwapSlot` (from `swap-slot-*` entries in `touched`, then stripped as `ctk/normal-touched-groups` does). - `graph/debug.clj`, `graph/stats.clj`: enumerate relationship tables from the catalog instead of naming them, so the console's graph view and the ingest counts pick up new edge types without being told. - `graph/debug.clj`, `http/debug.clj`: `graph-export` gains `source=session`, which snapshots the live in-memory console graph through EXPORT/IMPORT DATABASE. Live sync moves that graph away from a fresh projection, and taking it away to query elsewhere is the point of asking for it. AI-assisted-by: mixed models * 🐛 Write graph values Ladybug's CSV reader cannot carry through Cypher Three parity failures against beadpot's suite, all one cause: the bulk loader put compound and multi-line values into CSV, where Ladybug parses a field's *contents* as a literal with no escape mechanism at all. Verified against 0.18: a comma inside a list element ends the element, quotes are kept as part of the value rather than delimiting it, and the parallel reader rejects quoted newlines outright. So a value now goes through CSV only if it cannot be misread there — UUIDs, numbers, booleans, single-line strings, and lists of those. Everything else (MAP, STRUCT, STRING[]/JSON[], any string containing a newline) is written after the COPY by one Cypher statement per row, where `app.graph.ladybug` escapes properly. Parquet removes the distinction entirely and is still the right destination (masterplan P0 T1); this is what CSV can honestly do. Consequences beyond the encoding: - `touched` entries reached the graph as `:swap-slot-…`, keywords stringified with their colon, so `LinkSwapSlots` matched nothing. Keywords now render through `name`. - Shape names lost their newlines to a flattening step that existed only to keep the CSV writer happy. They are preserved. - `applied_tokens` keys are rendered camelCase, the form Penpot's own JSON encoder produces and the one beadpot's `AppliedTokenKey` holds — a MAP column's keys are values, not schema, so they are not snake_cased. - `link-component-instances!` keys on `component-file`, not `component-id` alone. The projection denormalizes `component-id` down the shape tree, after which it no longer tells an instance head from a shape inside one, and the transform linked every descendant frame; `ctk/instance-of?` requires both keys anyway. IsInstanceOf on the variants fixture: 78 -> 60, matching beadpot exactly. `app.graph.schema.nodes/format-column-value` is now the single place that knows a column's type and its contract details, used by the bulk loader and the incremental sync alike so the two cannot disagree about a value's shape. * ✨ Type graph columns as tightly as Ladybug allows Ladybug is schema-first and strongly typed: a property key gets its type at table-creation time and there is no widening later. That makes the Malli to Ladybug mapping the whole of the graph's typing, and it was leaving a lot on the table: a transform stored as `STRING`, a rect as `JSON`, a set of feature flags as a single `STRING`. A column typed `DOUBLE[4]` is four numbers a consumer reads as a tensor row; the same value as JSON is text somebody has to parse and trust. `app.graph.schema.types` now maps, in order: scalars; Penpot value types whose layout is fixed even though Malli only sees a map or a string (`::gmt/matrix` to `DOUBLE[6]`, `::gpt/point` to `DOUBLE[2]`, `::grc/rect` to `DOUBLE[4]`, `::clr/hex-color` to `UINT32`); then structure, with collections to `T[]`, `:map-of` to `MAP(k, v)`, and a closed map of scalars to a `STRUCT`. JSON is the fallback of last resort, for schemas that genuinely admit more than one shape. Two defects fell out. `::sm/set` was unmapped, so `features` and `migrations` were single strings rather than `STRING[]`, and `::sm/one-of`, how Penpot spells a closed set of keywords, was unmapped too, so `blend-mode`, `grow-type`, the constraints and every `layout-*` were mistyped. A tight column is only worth having if the writer fills it in that shape, so `app.graph.schema.values` shapes a value for its type: a matrix record into six doubles, a hex colour into a packed integer, a map into a struct's fields. Both writers go through it, so the bulk load and the incremental sync cannot disagree. What that required: - STRUCT field names must be backticked in the DDL *and* in every literal, because a grid cell has a field named `column`. The catalog reports them bare. - A struct literal's type is its field list, so every declared field must appear, and an absent one needs `cast(NULL, '<type>')`. A bare NULL is typed STRING and changes the struct's type. - `STRUCT(…)[]` starts with `STRUCT(` but is a list, so the list check comes first. - Nested lists cannot be rendered with `str`: Clojure's `[1 2]` is space-separated and Ladybug reads it as a one-element array. Three more corrections in the same area: - `project-attrs` used truthiness where it meant `some?`, so `opacity 0` and `blocked false` projected as absent. - Set-valued columns are written sorted. A set has no order, so the column varied between builds of the same file, which is precisely what stops two builds being diffable. - An empty collection is written as `[]` rather than skipped. A shape with no fills has none; NULL would say "unknown". Renamed the `kuzu-*` helpers to `ladybug-*`: Kùzu is deprecated and Ladybug substitutes it, so a name bearing the engine should bear this one. The one remaining mention cites the upstream issue Ladybug inherits. AI-assisted-by: mixed models * ✨ Add the file-level graph columns and tighten the svg ones Split out of "🐛 Declare the shape attributes stored files carry", which is now #11125 and carries only its `common/` half. This commit is the graph's own side of that change, and it stays on this branch. `app.graph.schema.contract` pins `svg_viewbox` to `DOUBLE[4]` and `svg_transform` to `DOUBLE[6]`. The shape schema types both `:map` on purpose, because legacy files hold them as plain maps rather than as `::grc/rect` and `::gmt/matrix` records, and a tighter *schema* would reject those files. A tighter *column* costs nothing, since `app.graph.schema.values/coerce` reads either form. `app.graph.schema.nodes` declares four file-level attributes as projection `:extra` rather than in `ctf/schema:file`: `:options`, `:backend`, `:comment-thread-seqn`, and `:ignore-sync-until`. Declaring them in the file schema breaks saving, measured at 185 failures, because `app.binfile.common/update-file!` derives its UPDATE column list from a file map's keys and the `file` table has no `backend` column, that value being synthesized on read. An `:extra` is local to the graph and cannot reach a write. `app.graph.project.document` lifts `:options` out of `:data` before the blob is dropped, so a consumer reads file-level configuration without opening the blob. AI-assisted-by: mixed models * ✨ Add the Arrow prerequisites for in-memory bulk load lbug pulls arrow-memory-core and arrow-vector but no allocation-manager implementation, so RootAllocator cannot be constructed; arrow-memory-netty 18.2.0 matches the arrow-vector lbug already brings and pulls only netty-buffer, netty-common, jackson and slf4j-api, all of which the backend already has. --add-opens=java.base/java.nio=ALL-UNNAMED is the second half: without it MemoryUtil's static initializer dies with an InaccessibleObjectException that surfaces as an unhelpful NoClassDefFoundError from anything touching RootAllocator. It has to be present at JVM start, hence all three places. Note app.main/restart will not pick it up — it restarts integrant inside the same JVM, so the process must be restarted. Worth a reviewer's attention: this is a JVM-wide flag added for one subsystem. It is the standard Arrow requirement and grants nothing beyond reflective access to java.nio, but it strengthens the case for putting the whole graph subsystem behind a feature flag. * ✨ Bulk load through in-memory Arrow; delete the CSV loader app.graph.arrow stages rows as Arrow VectorSchemaRoots and COPYs from them. No file is written at any point and no value is rendered as text for the engine to re-parse, so the defect class that produced three of this branch's four backend defects cannot recur. app.graph.bulk is deleted whole. csv-representable?, defer-to-cypher?, multiline?, fixup-statements, ladybug-literal, ladybug-list-element, ladybug-list-cell and staging-dir go with it, along with the post-COPY Cypher pass that emitted one SET per row. Measured before deciding: the fixup pass was ~77% execution, 16-22% parse and 6-7% round-trip, and prepared statements could not have recovered any of it — every fixup row carries a MAP column and Ladybug binds scalars only. So this replaces rather than optimizes. Marginal ingest 4.0 -> 1.21 ms/shape; ~25 s extrapolated at 20k shapes against the ~2 min the CSV path projected. Size unchanged. Four engine facts the implementation rests on, each verified against 0.18.2 with a standalone probe: - An Arrow table is not a COPY source identifier but is a MATCH-able node label. - A MAP vector's entries child must be a non-nullable struct, and MapVector.getWriter promotes it to a sparse union, so map vectors are built from an explicit Field and filled child-first. - Ladybug names a staged table's columns and struct fields from the Arrow field names and quotes none of them, so anything needing quotes must arrive quoted — hence cypher-property-key, not column-name, names the Arrow fields. - createArrowRelTable cannot resolve endpoints against a UUID-keyed node table under any encoding, so edges stage as a node table and the COPY subquery joins them. values/coerce is reused unchanged, so the Arrow and Cypher writers cannot disagree about a value's shape; nodes/column-map-key-fn is extracted so they cannot disagree about a MAP's key spelling either. Verified with pytest --graph-origin=penpot-only unchanged at 225/38/1 and --graph-origin=penpot unchanged at 258 passed / 2 pre-existing failures, both baselines re-established against a reverted backend rather than assumed; with bp graph diff between a CSV-built and an Arrow-built graph reporting "Graphs agree"; and with an adversarial round-trip carrying a quote, a backslash, a newline, a CRLF and a tab through STRING, STRING[] elements and MAP values. The diff was necessary, not belt-and-braces: both parity suites passed an earlier revision of this change that was writing EDN into every JSON column, because beadpot's assertions never parse those columns. It also showed Arrow correcting a CSV defect — an empty Component.path was being stored as NULL, because Ladybug's CSV reader cannot distinguish an empty field from an absent one. * ✨ Add a prepared-statement surface to the graph connection `app.graph.ladybug` could only run Cypher as text. Every value the sync path writes is therefore concatenated into the statement, and nothing can ask the engine whether a statement is even valid without running it. Add the four functions that close both gaps. `prepare-on-connection!` parses and binds without executing. `execute-prepared!` binds a parameter map and runs it. `exec-prepared-on-connection!` prepares every statement in a batch before executing any of them, so a parse or bind failure aborts before the first mutation. `validate-on-connection!` returns `{:ok? :error :read-only?}` instead of raising, which is what a gate wants. `->param-value` is the only `Value` constructor on the write path. It is unconditional: on lbug 0.18.2 an unwrapped parameter does not raise, it SIGSEGVs the JVM inside `lbug_value_clone`. Parameters are scalars only, because the JNI `Value` constructor takes no list or map, so `MAP`, `STRUCT` and `T[]` columns stay literal-rendered and the `:else` branch raises rather than crashing. Two departures from the design, both closing a JNI-handle leak on the error path: `prepare-on-connection!` closes the failed `PreparedStatement` before raising, and `execute-prepared!` closes every `Value` it built, including the ones built before a later parameter was rejected. `as-statement` accepts a bare string, so the sync builders can convert to bound parameters one family at a time rather than in one commit. AI-assisted-by: mixed models * 🐛 Write the document revision to the column that exists `set-document-revision-statement` emitted `SET d.revn`, but the column is `revision`: the beadpot contract renames `:revn` and the DDL has followed it since. The statement is the last one in every sync batch, so each batch raised after its mutations had already committed, and the session's in-memory index stayed frozen at its load-time revision. Name the column through `nodes/cypher-property-key` rather than spelling it, so the DDL and the statement cannot disagree again. Found by the binder gate in the next commit, on its first run. AI-assisted-by: mixed models * ✨ Gate every sync statement template through the binder Nothing checked that the eleven Cypher templates `app.graph.sync` emits still bind against the DDL the schema registry generates. A renamed column, a dropped table or a reserved word emitted unquoted surfaced only when a live session ran the statement, and by then the batch's earlier mutations had committed. `backend-tests.graph-binder-gate-test` opens a `:memory:` database, creates the live schema on it, and *prepares* one instance of each template without executing any of them. 14 tests, 51 assertions: the eleven templates, label coverage over all twelve registered node tables, and two assertions on the gate itself, that a `RETURN` reads as read-only and a `SET` does not, and that an unbindable statement is reported rather than thrown. It was not green on HEAD: it caught `set-document-revision-statement` writing a column that no longer exists, fixed in the previous commit. Red on both injected templates tried. No `:jvm-opts` change: CI's `-M:dev:test` already carries the native access flags the engine needs. AI-assisted-by: mixed models * 🐛 Let the engine quote the Arrow field names it interpolates `node-batch` named every top-level Arrow field with backticks, so that a column whose name is a reserved word (`Page.index`, `Document.options`) survived the DDL Ladybug generates for a staged table. The engine now quotes those identifiers itself, and it does not collapse a doubled backtick, so a pre-quoted name reaches the parser as ``index`` and `createArrowTable` fails outright: Parser exception: mismatched input '``' expecting PRIMARY Name the fields with `column-name`. The `COPY` projection is Cypher rather than DDL and keeps its own backticks through `cypher-property-key`, and STRUCT member names keep theirs too: those come out of `LogicalType::toString()`, which the DDL builder does not touch, so an unquoted member called `column` still fails to parse. Measured with `probes/arrow/probe25.clj` against lbug 0.19.1: a plain top-level reserved word loads and reads back, a pre-quoted one fails to parse, a plain STRUCT member fails to parse, and a pre-quoted one loads and reads back. Also re-dates the engine facts in the `app.graph.arrow` docstring to the version they were checked against, drops the SIGSEGV note from `->param-value` now that `Connection.execute` rejects an unwrapped parameter, and removes two references to the CSV loader. AI-assisted-by: mixed models * 📚 State what the graph schema does, not what it mirrors The graph namespaces explained themselves by citing a separate project whose Python pipeline reads the graphs this backend writes. A reader of this repository does not have that project and should not need it, and a docstring that justifies a choice by pointing elsewhere cannot be checked here. Every claim survives; only the framing changes. Column names and types are Penpot's own decision, recorded with the reason for each divergence from the snake_case default. The transform registry describes the edges it materializes. The denormalizations in `app.graph.project.document` are justified by the walk already holding both answers. Three corrections fall out of the rewrite: - `app.graph.schema.contract` claimed a test, `graph_contract_test`, that walks a checked-in schema manifest and fails on any divergence. No such test exists. The paragraph is gone. - `app.graph.project.document` pointed at `app.graph.meta/projection-transforms`, which does not exist. - `app.graph.project.transforms/registry` claimed its entries were "in application order" while `apply-transforms!` reduced over the literal vector. The three registered transforms read disjoint columns, so the order is not load-bearing. The docstring now says so, and the one real ordering constraint is stated where it applies: `link-swap-slots!` strips `swap-slot-*` entries from `touched`, so anything reading `touched` has to run before it. `contract/pending-beadpot-columns` becomes `contract/unprojected-keys`. It is referenced nowhere else. AI-assisted-by: mixed models * ✨ Refuse a mutating query from the graph console `debug/query-session!` ran whatever it was handed against the session connection. A session graph is a projection of a file, rebuilt from that file by Reload, so a mutation from the console produces a graph no rebuild reproduces and no query result explains. Bind the statement against the live schema first. A statement that does not bind reports the binder's own message and executes nothing, which also turns a misspelt table or property into an immediate error instead of an empty result. A statement that binds runs only when the engine's own read/write analysis calls it read-only. The console's query box is labelled read-only. Load, Reload, Unload and live sync are unaffected: they are separate handlers and do not go through this path. AI-assisted-by: mixed models * 🐛 Keep a synced graph equal to a rebuilt one Cold projection and incremental sync are two implementations of one mapping and nothing checked that they agree. They did not. `backend-tests.graph-sync-parity-test` projects a file into one `:memory:` database, applies a change list to that database and the same list to the file data, projects the result into a second database, and diffs the two down to the row and the column. It found four disagreements, each fixed here. **Sibling order was inverted.** A container's stored `:shapes` list runs bottom to top and `IsChildOf.position` numbers children in Penpot z-order, so appending to the list means taking position 0 and pushing every sibling up. Sync instead handed each new child the next free number, so any container edited live carried its children in the opposite order to a rebuild, and a delete left a gap where a rebuild renumbers densely. `insert-position` and `renumber-siblings` put the two paths on the same rule for `:add-obj`, `:mov-objects` and `:del-obj`, including a block move and `:after-shape`. **A moved shape kept its old parent.** `:mov-objects` moved the edge and left the shape's own `parent_id` and `frame_id` columns pointing at the container it came from. Both now follow, and `frame_id` follows through the whole subtree the shape carries, as `app.common.files.changes` does for `:mov-objects`. A top-level shape's column holds `uuid/zero`, the page's root frame, while its edge points at the Page. **A container's `shapes` column went stale.** Nothing maintained it after an add, a move or a delete. It is now rebuilt from the sibling order on every change that touches a container. **Pages came out backwards.** `projection-data` reversed `:pages` before numbering them, which is right for child shapes and wrong for pages: `:pages` is the tab order and has no second ordering to undo. `Page.index` and the page's `IsChildOf.position` are now that order. One defect the test does not reach, fixed on the way past: `index-add-shape!` accepted `:component-ctx` and dropped it, so a shape added under an instance head added in the same session inherited no `component-id`. AI-assisted-by: mixed models * 🐛 Build a synced page node the way the projection does `apply-add-page` sent the new Page node through `nodes/validate-node`, which checks a map against the registry schema and returns it unchanged. Every other node on both write paths goes through `nodes/project-attrs`, which also selects the projected keys and is the single place a column-level rule can live. A rule added there reached a rebuilt page and not a synced one. AI-assisted-by: mixed models * 🐛 Let the graph view's query filter follow the graph "Show result in graph view" froze the set of node ids the query returned and filtered every later repaint against it. Live sync creates ids the set has never seen, so a shape created while a filter was on could not appear in the view at any point, and clicking "Show full graph" was the only way to see it. A node the query would no longer match stayed. Keep the query beside the ids and re-run it whenever the graph repaints, which is only when the projection actually changed. A failed re-run keeps the ids in hand and says so on the status line rather than passing a stale view off as current. `idsInResult` and `presentIds` are extracted from the two places that scraped UUIDs out of a result. Verified in the devenv: with a filter showing 108 of 276 nodes, a `:file-change` adding a Frame published on the session's msgbus topic took the view to 109 of 277, with the new node carrying its added mark, and no interaction. AI-assisted-by: mixed models * ♻️ Rename app.graph.project to app.graph.projection `project` is a Penpot noun: a team holds projects and a project holds files, and the graph will carry a `Project` node table. A namespace called `app.graph.project.document` therefore reads as "the graph of a Penpot project" and means the opposite. `projection` is the word the rest of the subsystem already uses for the operation: `projection-data`, `load-projection!`, `:projection` in the ingest report, and `app.graph.schema.projection`. Pure rename. Both namespaces and every alias move; nothing else changes. AI-assisted-by: mixed models * 📎 Apply the project formatter to the graph namespaces `cljfmt check src/ test/` is a step of the Backend workflow and these two files did not pass it: an import block sorted the way a human reads it rather than the way the formatter sorts it, and a `cond` in `format-typed-value` indented one column short. Formatter output only. No semantic change. AI-assisted-by: mixed models * 📚 Document graph experiment architecture Add Serena memory coverage for the embedded Ladybug graph subsystem.\nDocument projection, incremental sync, console data flow, tests, and operational risks.\n\nAI-assisted-by: gpt-5.6-luna --------- Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> Co-authored-by: Alejandro Alonso <alejandroalonsofernandez@gmail.com> Co-authored-by: Andrey Antukh <niwi@niwi.nz> |
||
|
Andrey Antukh
|
0e388442a1
|
✨ Add storage object status lifecycle and verified dedup (#11345)
* ♻️ Simplify storage GC delays and add skip-delay task params The touched GC no longer applies an extra deletion-delay when marking storage objects as deleted. By the time a storage object is touched, its referencing domain row has already passed its own deletion delay, and the reference scan is the only safety check needed. Touched objects are now marked with deleted_at = now, so the deleted GC removes them on the next run. For the tempfile bucket, upload chunks now set touched-at in the future (1h, aligned with the upload-session-gc TTL) instead of relying on a special-case deletion delay. Task handlers now read their task props: - storage-gc-touched accepts :skip-delay to process all touched objects immediately, bypassing the min-age threshold. - objects-gc accepts :chunk-size and :skip-delay to process recently deleted rows without waiting for the deletion delay. This allows running the deletion cascade immediately from the REPL via run-task! with the skip-delay option. AI-assisted-by: deepseek-v4-flash * ✨ Add storage object status lifecycle, verified dedup, and deletion retry tracking Storage object lifecycle hardening: - Add status column ('valid' | 'pending') as write-ahead marker for object creation. put-object! inserts in 'pending' state, writes blob, then promotes to 'valid'. Failed writes remove the pending row. - Add :storage-pending-gc task to reclaim orphaned pending rows (e.g. after crash between blob write and promotion). - Verify blob existence on every dedup hit via exists-object? (fs stat / s3 headObject). Missing blobs mark the row as deleted and create fresh object. - Add deletion_attempts column (migration 0154) to track physical blob deletion attempts. Restructure gc_deleted to use chunked processing with per-chunk transactions (short lock duration). Failed deletions are deferred to tomorrow (deleted_at = NOW() + 1 day) to prevent infinite loops. After 7 attempts, give up and accept orphan. - Change del-objects-in-bulk contract to return #{fail-ids} for precise per-id tracking (fs and s3 backends updated). - Use tmp/tempfile for fs atomic writes with cleanup queue registration (crashed-JVM temp files swept ~60min later). Document ATOMIC_MOVE POSIX-only assumption. - Add linear backoff to s3 exists-object? retries (100ms/200ms/300ms). - Wrap compensating delete in put-object! catch block to prevent masking original error when connection is aborted. - Fix assert messages in pending_gc.clj and gc_deleted.clj (pool assertion said 'expected valid storage' instead of 'db pool'). - Add pending-objects-excluded-from-gc-deleted test. Use unique path in put-object-write-failure-leaves-no-row test to avoid collisions. AI-assisted-by: qwen3.7-plus * 🐛 Fix review comments on gc-deleted and storage - Fix process-chunk! returning nil causing (+ acc nil) crash - Add FOR UPDATE SKIP LOCKED to sql:get-deleted-chunk to prevent infinite loop when another worker holds locks - Pass :cause to log messages in gc_deleted.clj and s3.clj - Fix extra space in log hint string - Remove unused ::blob-missing? reference from storage memory - Rename test to match actual behavior (leaves pending row) - Add test for gc-deleted giving up after max attempts AI-assisted-by: qwen3.7-plus |
||
|
Andrey Antukh
|
87c51090b1 | Merge remote-tracking branch 'origin/staging' into develop | ||
|
Andrey Antukh
|
6e173a02fb |
📚 Split backend testing memory and link from testing skill
Extract the backend Testing section from backend/core into a dedicated backend/testing memory, following the pattern of common, frontend, and exporter. Update the testing skill and root testing memory to point at the new location, and add exporter/testing to the skill's required reading list. AI-assisted-by: deepseek-v4-flash |
||
|
Alejandro Alonso
|
7419bc7007
|
🐛 Evict multi-scale tile cache on shape edits (#11337)
those textures across zoom for progressive previews, and invalidate by old∪new document coverage so rotate/move edits do not leave stale fragments on zoom-out. |
||
|
Alejandro Alonso
|
57c0e81616
|
⚡ Present viewport before interest and clamp paint to atlas (#11313)
Present visible tiles via ViewportReady so zoom settle turns sharp without waiting on the interest ring, and paint at atlas slot size so DPR 2 does not rasterize 1024 only to downscale into 512 slots. |
||
|
Alejandro Alonso
|
689d506788 |
⚡ Render eligible frame drop shadows via direct geometry path
Add a direct container-geometry path for eligible frames: inline blur when the kernel fits the tile margin, otherwise a cached filter-surface pass reused across tiles via DropShadowFilterCache on both the direct and slow render_shape paths. Move frame shadow logic into shadows.rs. Fix nested/clipped frame shadows by deferring parent clip to composite time, apply negative spread via inset, and allow rotated/transformed frames on the direct path. Skip descendant extrect walks for clipped frames when only nested drop shadows matter, and skip child silhouettes when the container fill already covers shadow descendants. |
||
|
Alejandro Alonso
|
2dcf1a8a0a | Merge remote-tracking branch 'origin/staging' into develop | ||
|
Andrey Antukh
|
aa3bc1ae98 |
🐛 Fix linear gradients in SVG text exports (#11272)
* 🐛 Use gradient type instead of export type in SVG renderer data->gradient-def was comparing the render `type` parameter (:svg, :png, :pdf) against "linear" to decide between linearGradient and radialGradient elements. Since the export type is never "linear", the comparison always fell through to radialGradient, causing all linear gradients to be exported as radial in SVG output. Read the gradient type from the data map instead: (get-in data ["gradient" "type"]) Closes #5972 * 🐛 Add SVG gradient export regression test Extract SVG gradient definition generation from the renderer so it can be tested directly. Add exporter test build wiring and cover both linear and radial gradient output. AI-assisted-by: gpt-5.6-luna * ✨ Standardize exporter testing workflow Align exporter scripts with the frontend testing pattern. Add a dedicated GitHub Actions workflow and document the canonical exporter commands in Serena memories. AI-assisted-by: gpt-5.6-luna * ✨ Add focused exporter test execution Mirror frontend test-runner behavior for focused namespaces and test vars. Support --focus, --log-level, and --help, and document the commands. AI-assisted-by: gpt-5.6-luna * 🐛 Replace shell exec with execFile in exporter Replace child_process.exec with execFile to eliminate shell interpretation. Add hex color validation in exporter and frontend to reject malformed input before command construction. This fixes GHSA-4f36-m4hj-cv86 (CVSS 9.9 Critical), an authenticated OS command injection vulnerability where malicious fill-color values could execute arbitrary commands in the exporter container. Defense in depth: - Layer 1: execFile passes arguments directly without shell parsing - Layer 2: Exporter validates colors with strict hex regex - Layer 3: Frontend filters invalid colors before DOM emission All three independent reporters' attack vectors are addressed: - Quote breakout (lyhtheori) - Command substitution (B1gN0Se) - Path traversal (KimiSecurityTeam) AI-assisted-by: qwen3.7-plus * 🐛 Use existing hex-color-string? and fix test path mismatch Address code review feedback: - Replace duplicated hex-color-rx and valid-hex-color? with existing hex-color-string? from app.common.types.color - Fix RCE test to use marker path in payload instead of hardcoded /tmp/pwned AI-assisted-by: qwen3.7-plus --------- Co-authored-by: Sumit Ridhal <sridhal@redhat.com> |
||
|
Andrey Antukh
|
60d87a6342
|
🐛 Fix linear gradients in SVG text exports (#11272)
* 🐛 Use gradient type instead of export type in SVG renderer data->gradient-def was comparing the render `type` parameter (:svg, :png, :pdf) against "linear" to decide between linearGradient and radialGradient elements. Since the export type is never "linear", the comparison always fell through to radialGradient, causing all linear gradients to be exported as radial in SVG output. Read the gradient type from the data map instead: (get-in data ["gradient" "type"]) Closes #5972 * 🐛 Add SVG gradient export regression test Extract SVG gradient definition generation from the renderer so it can be tested directly. Add exporter test build wiring and cover both linear and radial gradient output. AI-assisted-by: gpt-5.6-luna * ✨ Standardize exporter testing workflow Align exporter scripts with the frontend testing pattern. Add a dedicated GitHub Actions workflow and document the canonical exporter commands in Serena memories. AI-assisted-by: gpt-5.6-luna * ✨ Add focused exporter test execution Mirror frontend test-runner behavior for focused namespaces and test vars. Support --focus, --log-level, and --help, and document the commands. AI-assisted-by: gpt-5.6-luna * 🐛 Replace shell exec with execFile in exporter Replace child_process.exec with execFile to eliminate shell interpretation. Add hex color validation in exporter and frontend to reject malformed input before command construction. This fixes GHSA-4f36-m4hj-cv86 (CVSS 9.9 Critical), an authenticated OS command injection vulnerability where malicious fill-color values could execute arbitrary commands in the exporter container. Defense in depth: - Layer 1: execFile passes arguments directly without shell parsing - Layer 2: Exporter validates colors with strict hex regex - Layer 3: Frontend filters invalid colors before DOM emission All three independent reporters' attack vectors are addressed: - Quote breakout (lyhtheori) - Command substitution (B1gN0Se) - Path traversal (KimiSecurityTeam) AI-assisted-by: qwen3.7-plus * 🐛 Use existing hex-color-string? and fix test path mismatch Address code review feedback: - Replace duplicated hex-color-rx and valid-hex-color? with existing hex-color-string? from app.common.types.color - Fix RCE test to use marker path in payload instead of hardcoded /tmp/pwned AI-assisted-by: qwen3.7-plus --------- Co-authored-by: Sumit Ridhal <sridhal@redhat.com> |
||
|
Andrey Antukh
|
4d90fe9126 | ✨ Add advisories access helper to gh tool | ||
|
Andrey Antukh
|
4339d8d244 | 📎 Update serena documentation about backend storage | ||
|
Alejandro Alonso
|
cb57fd9dfa
|
⚡ Skip save_layer for plain image fills (#11230)
Avoid an offscreen buffer per Fill::Image during tile walks: only use save_layer when a shape image filter is present; axis-aligned rects and frames without corner radii also skip the redundant container clip. |
||
|
Andrey Antukh
|
aeedb96260
|
✨ Add media-processor service for image and font processing (#10767)
* ✨ Add media-processor service for image and font processing Externalizes ImageMagick and FontForge subprocess invocations into a separate Node.js HTTP service (media-processor/). Backend dispatches via feature flag :use-remote-media-processing. Key changes: - media-processor module (TypeScript, Express 5, Sharp, FontForge/woff) - POST /api/image/info, /api/image/thumbnail, /api/font/generate - Resource limits: 128MP rejection, prlimit (512MB + 30s CPU) - Streaming multipart via SequenceInputStream - app.media split into validation (leaf), local (shell impls), remote (HTTP) - Schema enforcement: :upload and :input schemas in validation namespace - Configurable timeout (PENPOT_MEDIA_PROCESSING_SERVICE_TIMEOUT) - 78 tests across 4 files (image, font, middleware, config) - FontForge path escaping for command injection prevention - Parallel font variant conversions with Promise.all AI-assisted-by: mimo-v2.5-pro * 🐳 Revert docker-compose changes from media-processor commit Remove docker-compose.yaml modifications that were part of the media-processor service commit. The media-processor service definition, flags, and environment variables are reverted to their previous state. AI-assisted-by: qwen3.7-plus * ⬆️ Update dependencies * 🐛 Fix PR review issues in media-processor - Font path bug: sfntToWoff and woff2ToSfnt now copy input to temp dir when input is a file path, ensuring output lands in expected location - Error preservation: execCommand preserves killed/signal/code properties from child process errors for OOM detection - Content-Length: service-multipart-request calculates and includes Content-Length header for streaming multipart requests AI-assisted-by: qwen3.7-plus * 🐛 Fix code review issues in media-processor - Rename PENPOT_MEDIA_PROCESSOR_SECRET_KEY to PENPOT_MEDIA_PROCESSOR_SHARED_KEY in devenv to match backend config key - Fix timeout middleware to destroy request AFTER response finishes, preventing truncated 504 responses - Fix quality=0 parsing to preserve explicit zero (was silently overridden to 85) - Replace require('fs') with proper ES module import in upload-storage.ts - Refactor font conversion temp-dir boilerplate into withTempInput helper - Document FontForge escaping limitations (single quotes only) - Fix misleading comment in image.ts about sharp metadata decoding AI-assisted-by: qwen3.7-plus * 🐛 Fix code review issues in media-processor (round 2) - Fix queue middleware to skip next() when response already ended, preventing orphaned work after timeout - Fix hybrid storage to use disk when Content-Length is absent (chunked transfer), preventing unbounded memory allocation - Add source image format validation in generateThumbnail to reject unsupported formats (TIFF, BMP, etc.) with 400 instead of 500 - Remove dead code in convertFont for unreachable woff→woff path - Remove unused isEnabled() method from LokiLogTransport - Fix sfntToWoff to use correct extension (.ttf/.otf) based on source type - Extract queue middleware to separate file for testability - Add comprehensive tests for queue middleware and upload storage AI-assisted-by: qwen3.7-plus * 🐛 Fix code review issues in media-processor (round 3) - Fix disk-backed upload cleanup after successful requests by adding cleanup middleware that removes temp files on response finish/close - Wrap sharp metadata/decoding errors as 400 validation errors instead of 500 internal errors - Only apply flatten() for JPEG output to preserve alpha channel in PNG and WebP outputs AI-assisted-by: qwen3.7-plus * ✨ Add comprehensive tests for media-processor Phase 1 - Cleanup verification: - Add cleanup middleware unit tests (6 tests) - Add HTTP upload cleanup integration tests (5 tests) Phase 2 - Error handling & alpha preservation: - Add sharp error wrapping tests (4 tests) - Add HTTP malformed image tests (2 tests) - Add alpha preservation tests (3 tests) Phase 3 - Edge cases: - Add upload storage edge case tests (3 tests) - Add queue middleware edge case tests (4 tests) Phase 4 - Backend mock verification: - Fix backend mocks to include :mtype field in image info responses - Verify all error codes match actual service behavior Total: 27 new tests added (160 tests passing) AI-assisted-by: qwen3.7-plus * 🐛 Fix code review issues in media-processor (round 4) - Add Zod validation constraints for config values (int, positive, min) - Fix auth middleware to compare Buffer byte lengths instead of string lengths - Validate requested output dimensions in generateThumbnail (crop mode) - Change queue middleware to release slot via callback in finally block - Add comprehensive tests for all fixes AI-assisted-by: qwen3.7-plus * 🐛 Close HTTP response streams in backend media remote - Wrap stream consumption in try/finally with .close() calls - Add tests to verify stream closure for info, font-convert, and thumbnail AI-assisted-by: qwen3.7-plus * 🐛 Fix queue slot leak on upload failures Make releaseQueue idempotent and attach fallback listener to release slot when response finishes. This covers Multer errors that bypass the route handler's finally block, preventing permanent queue stall. AI-assisted-by: qwen3.7-plus * 🐛 Cancel processing on timeout Create AbortController in timeout middleware and abort signal when timeout fires. Pass signal to Sharp and FontForge to cancel ongoing processing and release resources when request is cancelled. AI-assisted-by: qwen3.7-plus * 🐛 Fix code review issues in media-processor (round 6) - Error handler: check headersSent before writing response to prevent ERR_HTTP_HEADERS_SENT when timeout already sent 504 - Timeout config: increase default requestTimeout from 60s to 180s to match font processing timeout (120s) and backend request timeout - Image processing: check abort signal before starting Sharp operations to cancel processing when timeout fires - Queue lifecycle: remove res.on('close', release) fallback to hold queue slot until processing completes, preventing concurrency limit violation when client disconnects AI-assisted-by: qwen3.7-plus * 🐛 Close HTTP response stream in download-image Wrap response body in with-open to ensure stream is closed after writing to temp file, preventing HTTP connection leaks on repeated URL imports. AI-assisted-by: qwen3.7-plus * 🐛 Close HTTP response stream on validation errors in download-image Move with-open to wrap the entire validation and processing block, ensuring the response body stream is closed even when validation fails (non-2xx status, missing size, invalid media type). This prevents HTTP connection leaks on repeated failed downloads. Add test to verify stream closure on validation errors. AI-assisted-by: qwen3.7-plus * 🐛 Pass abort signal to Sharp toBuffer for timeout cancellation Wrap Sharp's toBuffer() with Promise.race to check abort signal during processing. This ensures large thumbnails stop processing when the request times out, preventing wasted CPU/memory and queue capacity. Add test to verify abort during toBuffer operation. AI-assisted-by: qwen3.7-plus * 🐛 Hold queue slot until Sharp completes and handle client disconnect - Remove Promise.race from generateThumbnail — Sharp processing now completes fully before queue slot is released, preventing concurrency limit violations under timeout conditions - Remove res.on("finish", release) fallback from queue middleware — error handler now explicitly calls releaseQueue in all error paths - Add res.on("close") handler in timeout middleware to abort signal when client disconnects, ensuring processing stops early - Add tests for client disconnect handling and queue slot lifecycle AI-assisted-by: qwen3.7-plus * 🐛 Address round 9 review findings - Document Sharp 0.35.3 cancellation limitation in image.ts - Add integration test for timeout cleanup with large images - Fix font tools (sfntToWoff, woffToSfnt, woff2ToSfnt) to throw ProcessingError on resource limit kills instead of returning null - Validate font signatures for same-format conversions to prevent arbitrary files from being persisted as valid fonts - Fix concurrent mkdtemp race in upload-storage by using shared initialization promise AI-assisted-by: qwen3.7-plus * 🐛 Address round 10 review findings - Add tmpdir assertion in font.ts to prevent path injection - Preserve original error in queue middleware catch handler - Change auth middleware response type from "internal" to "authorization" - Add cleanup flag to prevent double cleanup in cleanup middleware - Move quality clamping into parseQuality function for consistency - Add integration tests for quality parameter clamping at route level - Update existing tests to match new auth response type AI-assisted-by: qwen3.7-plus * 🐛 Address round 11 review findings - Extract releaseSlot helper in error-handler to reduce duplication - Remove redundant try/catch in font.ts withTempDir cleanup - Improve font path validation error message for clarity - Move path validation before try/catch to prevent swallowing - Add debug logging for cleanup failures in cleanup middleware - Inline TransportTargetSpec type alias in logger.ts - Extract logging middleware to separate file for consistency - Remove duplicate MIME validation in image thumbnail route - Add test for font path validation (outside tmpdir rejection) - Add tests for error handler queue release across all branches AI-assisted-by: qwen3.7-plus * 🐛 Remove Content-Length header from multipart requests The JDK's HttpClient rejects Content-Length as a restricted header, causing IllegalArgumentException when sending multipart requests to the media-processor. Remove the explicit Content-Length header and let the JDK use chunked transfer encoding. The media-processor will use disk storage for all multipart requests (safe default behavior). Remove unused size computations (file-size, header-bytes, footer-bytes, total-size) that were only used for Content-Length. Update test to verify Content-Length is not present in request headers. AI-assisted-by: qwen3.7-plus * 🐛 Fix pino ESM bundling for media-processor Mark pino and its transports (pino-pretty, pino-loki) as external to avoid bundling issues with worker thread modules that reference __dirname (not available in ES modules). AI-assisted-by: qwen3.7-plus |
||
|
Andrey Antukh
|
23ea2bbad6 | 📎 Update creating-commits serena workflow file | ||
|
Andrey Antukh
|
b507a6b667 |
♻️ Convert commiter agent to create-commit skill
Replace the commiter subagent with a create-commit skill, consistent with the create-pr and create-issue skill patterns. - Remove .opencode/agents/commiter.md - Add .opencode/skills/create-commit/SKILL.md - Update implement-plan.md to use the skill instead of subagent delegation - Document commit body line wrapping at 72 chars in creating-commits memory and skill AI-assisted-by: mimo-v2.5 |
||
|
Andrey Antukh
|
49119e0339
|
♻️ Rename nitrate config to admin-console (#10929)
* ♻️ Rename nitrate config to admin-console Rename user-facing configuration from 'nitrate' to 'admin-console': - Feature flags: :nitrate -> :admin-console, :nitrate-bulk-create-profiles -> :admin-console-bulk-create-profiles - Config keys: :nitrate-shared-key -> :admin-console-shared-key, :nitrate-backend-uri -> :admin-console-uri - Shared-keys map entry: :nitrate -> :admin-console (setup.clj + main.clj) - Env vars: PENPOT_NITRATE_SHARED_KEY -> PENPOT_ADMIN_CONSOLE_SHARED_KEY, PENPOT_NITRATE_BACKEND_URI removed (consolidated into PENPOT_ADMIN_CONSOLE_URI) - Docker/nginx: PENPOT_NITRATE_URI -> PENPOT_ADMIN_CONSOLE_URI Code namespaces, file paths, CSS classes, and i18n keys stay as-is. AI-assisted-by: mimo-v2.5-pro * ♻️ Rename initialize-user-in-nitrate-organization to initialize-user-in-organization Part of the nitrate -> admin-console rename series. The function and all 9 references across 6 files have been renamed. * ♻️ Rename :nitrate-bulk-create-profiles-not-allowed to :bulk-create-profiles-not-allowed * ♻️ Inline nitrate-permissions into app.common.types.organization - Delete app.common.types.nitrate-permissions and its test - Move permission rules (allowed?, can-send-invitations?, etc.) into organization.cljc - Harmonize all consumers to use alias cto for app.common.types.organization - Update test runner and create organization_test.cljc |
||
|
Andrey Antukh
|
32520e66e5 |
✨ Allow stopping ws0 independently of other workspaces
Remove the guard in stop-devenv that refused to stop ws0 while any ws1+ instance was running. Each workspace is now fully independent and can be started/stopped in any order. Shared infra shuts down only when no instances remain running. Updated docs (devenv.md, agentic-devenv.md) and devenv memory to reflect the new behavior. AI-assisted-by: mimo-v2.5-pro |
||
|
Andrey Antukh
|
73519778ae | 📎 Update serena memories | ||
|
Andrey Antukh
|
e741313add | Merge remote-tracking branch 'origin/staging' into develop | ||
|
Andrey Antukh
|
764b62906b
|
🐛 Handle unrecognized JSON escape sequences as malformed-json (#10808)
* 📎 Update serena documentation about creating-prs workflow * 🐛 Handle unrecognized JSON escape sequences as malformed-json When clojure.data.json's read-escaped-char encounters an unrecognized escape sequence (e.g. a backslash followed by '}', or other case fall-throughs in the parser) in a JSON request body, it throws a bare IllegalArgumentException. Previously this fell through to the generic RuntimeException branch in wrap-parse-request's handle-error, which unwrapped and recurred without matching, eventually reaching the internal-error handler and producing HTTP 500 + an error report — even though the root cause was malformed client input, not a server bug. The fix converts any IllegalArgumentException raised in the JSON parse path into a `:validation`/`:malformed-json` error by raising a new ex-info (which is caught by the top-level error handler in `app.http/router-handler`). The result is an HTTP 400 response with a descriptive hint, and no error report is generated. This addresses ~10% of all error reports received. The new IAE branch is placed before the RuntimeException branch in the cond (since IllegalArgumentException IS-A RuntimeException) and uses the throw-style (ex/raise) to match the existing RequestTooBigException / EOFException branches. A comment above the handle-error cond documents why raising is intentional and is caught by the top-level app.http error handler, not by the per-route wrap-errors middleware. Test suite changes: - Extend the existing `DummyRequest` defrecord in `http_middleware_test.clj` from 2 fields to 12 fields, implementing every IRequest method, and add a private `make-dummy-request` constructor that accepts an options map with every key optional and sensible `:or` defaults. Future fields added to DummyRequest won't break existing call sites as long as the `:or` defaults are kept in sync. - Remove the now-redundant `JsonRequest` defrecord and migrate all 11 `->DummyRequest` call sites to `make-dummy-request`. - Add 6 new deftest cases: - parse-request-illegal-argument-exception: malformed JSON body (containing `\}`) is converted to `:malformed-json`. - parse-request-request-too-big-exception: RequestTooBigException is converted to `:request-body-too-large`. - parse-request-eof-exception: java.io.EOFException is converted to `:malformed-json`. - parse-request-runtime-exception-with-cause: a wrapped RuntimeException recurses on ex-cause and dispatches to the matching specific branch. - parse-request-runtime-exception-without-cause: a bare RuntimeException falls through to errors/handle, returning 500 with :type :server-error :code :unexpected. - parse-request-non-runtime-throwable: java.io.IOException (a non-RuntimeException Throwable) is handled by the dedicated handle-exception method, returning 500 with :code :io-exception. Together, the new tests cover all 6 branches of wrap-parse-request's handle-error cond. Refs #10804. AI-assisted-by: minimax-m3 |
||
|
Alonso Torres
|
014ec34249
|
🐛 Fix system crash on grid element deletion of a component (#10956)
* 🐛 Add regression test: main-side component edits break copy swap slots Reproduces the :missing-slot referential-integrity failure ("Shape has been swapped, should have swap slot") that crashes files with component copies. Root cause: reordering or deleting a nested sub-head IN THE MAIN of a component, while copies exist, does not propagate swap slots to the copies. find-near-match matches a copy's sub-heads to the main's children by POSITION, so once the main's order changes the copies' shape-refs no longer match their position and, lacking a swap slot, fail referential-integrity validation. - Copy-side edits are handled correctly (characterization tests, pass today). - The two main-side tests fail today with :missing-slot and go green once the sync assigns swap slots to copies on a main reorder/delete. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * 🐛 Fix integrity crashes from copy/main child-order divergence Copy sub-heads were matched to their main's children purely by position (find-near-match), while several code paths reorder or mutilate one side only. Any of them made file validation fail with :missing-slot ("Shape has been swapped, should have swap slot"), crashing the workspace on the next validated commit, or persisting a corrupt file whose later edits crash. Reproduced live: deleting a sub-head inside a copy (which only hides it) and then reflowing the copy's grid moved the hidden (cell-less) child to the front of :shapes, knocking every sibling out of its positional slot. Four fixes, one per divergence path: - validate/check-required-swap-slot: a sub-head whose shape-ref is still a child of the near main parent is a REORDER (the component sync realigns it), not a swap; a swap slot is required only when the ref points outside the near main parent (a real swap). This matches how the sync engine itself pairs children (by shape-ref, not by position). comp-processors/fix-missing-swap-slots (migration 0019) is aligned: adding slots to merely-reordered sub-heads would freeze them out of normal synchronization. - changes/:reorder-children now refuses to alter the child structure of component copies unless allow-altering-copies is set, mirroring the is-valid-move? rule of :mov-objects; that structure is owned by the component sync engine. Grid reflows emitted this change type with no guard. pcb/reorder-grid-children also skips copy grids producer-side. - layout/reorder-grid-children keeps children that participate in no cell (hidden or absolute positioned) at their original index instead of lumping them at the front: moving them gratuitously changed their z-order and, in copies, broke the positional matching. Note :shapes stays reversed relative to the sorted cell order for in-cell children. - logic/generate-delete-shapes: deleting shapes from inside a main (without deleting the main root, whose copies keep working against the deleted component) now also deletes the copy shapes that reference them, transitively (copies of copies) and across all pages of the file, so no dangling shape-refs remain. Skipped for allow-altering-copies flows (component swap replaces the shape and the sync reconciles copies via swap slots). Cross-page removals build redo/undo changes against that page's objects directly, since the changes-builder mounts only the current page; their undo mov-objects carry allow-altering-copies so restoring inside a copy is not rejected by the new guard. The regression tests assert the fixed semantics: main-side reorders and deletes keep copies valid, the previously crashing full chain (unvalidated main reorder + later copy edit) stays healthy, :reorder-children cannot scramble copies, and reorder-grid-children keeps cell-less children in place. The namespace is now also registered in the JS test runner. AI-assisted-by: Claude Opus 4.8 (1M context) * ✨ Extend composable slot cases to the grid-reflow crash sweep Case D (CopySubheadDeletePreservesSlots) now sweeps which copy sub-head is deleted (first or last) and whether the copy root is resized afterwards, forcing a grid reflow: the reflow used to move the hidden (cell-less) child to the front of the copy's children, shifting every sibling out of its positional slot. Deleting the FIRST sub-head masked the bug (moving it to the front is a no-op), which is why the case passed before this sweep. The foundation layout is grid accordingly. Case E (MainReorderKeepsCopySlots) no longer hangs the app now that a main-side reorder leaves a valid file, so its warning docstring is replaced: it runs as a routine test (verified headless, passing) and is safe in a "run all". SlotIntegrity's doc is updated to the new validator semantics (a slot is required only for real swaps, not reorders); the positional alignment it asserts remains the correct, stronger steady-state invariant for these cases. The lockfile change materializes the playwright devDependency already declared in package.json. AI-assisted-by: Claude Opus 4.8 (1M context) * 📚 Record copy/main order-divergence invariants in memories Swap-slot semantics (membership, not positional; slots only for real swaps), the copy-structure guards on :mov-objects/:reorder-children, the grid reorder stability for cell-less children, and the main-side delete propagation in generate-delete-shapes. AI-assisted-by: Claude Opus 4.8 (1M context) * 🐛 Add adjustements to the code --------- Co-authored-by: Michael Panchenko <michael.panchenko@oraios-ai.de> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Andrey Antukh
|
0ed151d2ca | Merge remote-tracking branch 'origin/staging' into develop | ||
|
Andrey Antukh
|
adda7e6645 | 📚 Update testing serena memories | ||
|
Andrey Antukh
|
2f8d71e143 | Merge remote-tracking branch 'origin/staging' into develop | ||
|
Andrey Antukh
|
b54c1f316a | ✨ Add minor improvements for error report script | ||
|
Dr. Dominik Jain
|
caacd482af
|
✨ Add composable test framework plugin (applied to component tests, runs in CI) (#10679)
* ✨ Add plugin with composable test framework and component tests The plugin provides a framework for writing composable tests against the Plugin API, and applies it to systematic end-to-end testing of component semantics. The framework's core ideas: a test is written once as a composition of operations over a starting configuration; choice points among the operations (optional steps, alternatives) expand the composition into a full sweep of test variants, so a single case definition yields broad combinatorial coverage; and the operations drive the real Plugin API with real change propagation, testing the full production implementation. The initial application is a suite of component test cases covering synchronization, overrides, swap slots and variants — the TypeScript/e2e continuation of the ClojureScript composable test suite (frontend_tests.composable_tests). Several cases originate from reproducing real defects (e.g. #10109 and the swap-slot corruptions). Tests run from an interactive panel in Penpot: cases are listed with plain-language descriptions, tests can be run selectively, results stream in live, and every checkbox carries a stable DOM id — so the panel can equally be driven programmatically (the basis for running the suite in CI), as documented in the plugin's README. Lives at plugins/apps/composable-test-suite as a regular member of the plugins workspace (init script, start:plugin:composable-test-suite, shared dev port 4202, covered by build:plugins via the new ./apps/*-test-suite filter). Related to #10584. AI-assisted-by: claude-fable-5 * ✨ Run the composable test suite headlessly in CI Adds a headless run mode for the composable test suite, following the plugin-api-test-suite's CI architecture, and a workflow that runs it as a per-PR gate. An in-sandbox entry (src/ci/headless.ts) runs the suite without the panel UI — the framework's runner was UI-free by construction, so no refactoring was needed — and streams each result through console markers, addressed by the same composite identifiers the panel uses (e.g. MainEditSyncs-2), with durations and, on failure, the error and the applied-steps transcript. It is built as a single self-executing bundle and evaluated directly inside a real Penpot plugin sandbox by the driver (ci/run-ci.ts), so no plugin dev server or port is involved. The driver needs no backend and no login: it serves the prebuilt frontend bundle via the frontend e2e static server and intercepts every backend RPC with Playwright fixtures. The mocked backend is not a limitation for this suite — everything it asserts is frontend store logic executed in memory — which the full run confirms: all 48 tests behave identically to the interactive panel, including variants and swap slots, with the single (currently expected) failure of MainEditSyncs-2 reproducing bug #10109 under the mock. TEST_FILTER selects tests by identifier substring; CI_TIMEOUT_MS bounds the run. The mock harness mirrors the frontend e2e harness (see the provenance note in the driver). Related to #10584. AI-assisted-by: claude-fable-5 * 📚 Restructure the composable-tests memory around both suites Present the composable component tests top-down: the shared framework principles upfront, then the two implementations — the ClojureScript suite in the frontend test tree and the TypeScript suite in the plugin, which tests fully end-to-end with a slightly more elaborate set of abstractions — and the plugin's headless CI run, pointing to the plugin's README for operational details. Also records this session's additions (geometry operations, case N, the CI harness). AI-assisted-by: claude-fable-5 * 📎 Refine the PR-description conventions in the creating-prs memory Encourage digestible descriptions: bullet items over prose (grouped by area with bold lead-ins for larger PRs) and no manual line wraps, since the rendered markdown adapts to the viewport. Also drop the outdated 'MCP' from the standard Note line. AI-assisted-by: claude-fable-5 * 🔧 Set Prettier endOfLine to auto in plugins workspace Prettier defaults to endOfLine "lf", which is incompatible with checkouts on Windows that use core.autocrlf=true * 🐛 Fix problems with suite --------- Co-authored-by: alonso.torres <alonso.torres@kaleidos.net> |
||
|
Andrey Antukh
|
c467d98c9e | Merge remote-tracking branch 'origin/staging' into develop | ||
|
Andrey Antukh
|
f7c312021b |
✨ Improve error-reports CLI with streaming, time-range, and stats
Server changes: - Switch list ordering from DESC to ASC (oldest first) - Flip cursor direction to > for forward pagination - Add 'until' param for server-side upper-bound filtering CLI changes: - Add --from/--to flags mapping to server's since/until - Streaming output for --all and --format ndjson - Add --format ndjson option (one JSON object per line) - Add --normalize-hints flag to strip dynamic values - Add --output flag to write list results to file - Add 'stats' subcommand with aggregations (signature, host, tenant, version, source, kind, hour) reading from API, file, stdin - stats input supports JSON, JSON array, and NDJSON formats Test changes: - Fix pagination assertions for ASC ordering AI-assisted-by: mimo-v2.5-pro |
||
|
Andrey Antukh
|
40ab48ea01 | 🐛 Fix inconsistencies on serenea memories | ||
|
Andrey Antukh
|
2344ba22a6 |
🎉 Add error reports API and CLI tool
Implement RPC methods for querying server error reports with pagination and filtering. Add CLI tool (tools/error-reports.mjs) for convenient access with table and JSON output formats. Extract profile-id from audit events and logging context for better error categorization. Build improved HREF using request path when available. AI-assisted-by: qwen3.7-plus |
||
|
Andrey Antukh
|
f9439d2942 | Merge remote-tracking branch 'origin/staging' into develop | ||
|
Andrey Antukh
|
e4cddd8536 | 📚 Update serena memories | ||
|
Dr. Dominik Jain
|
d4e87ec59d
|
⬆️ Update Serena to 1.6.1 in agentic devenv (#10770)
Update project files accordingly |
||
|
Andrey Antukh
|
1c917951b6 | Merge remote-tracking branch 'origin/staging' into develop | ||
|
Andrey Antukh
|
f3bf24b4f6 |
♻️ Consolidate dev tooling into scripts/ and reorganize docs
Move all development tools from tools/ to scripts/ for consistency. Rename lint/fmt/check-fmt to lint-clj/fmt-clj/check-fmt-clj to clarify they target Clojure specifically. Remove unused scripts (attach-opencode, start-opencode, start-opencode-server) and the backport-commit skill. Update all internal references across .serena/, AGENTS.md, and CONTRIBUTING.md to point to the new script locations. Simplify CONTRIBUTING.md by delegating module-specific fmt/lint instructions to the respective serena memories. AI-assisted-by: deepseek-v4-flash |
||
|
Andrey Antukh
|
5acc3aac99 | Merge remote-tracking branch 'origin/staging' into develop | ||
|
Andrey Antukh
|
a96001894c |
📚 Add explicit AI-assisted-by format rules to commit memory
AI-assisted-by: mimo-v2.5 |
||
|
Andrey Antukh
|
a4347451d0 | Merge remote-tracking branch 'origin/staging' into develop | ||
|
Andrey Antukh
|
779983d38e |
📎 Standardize test scripts and add execution discipline docs
- Remove conditional build from test scripts (frontend, common) - Remove test:jvm from common package.json (JVM tests via clojure directly) - Remove test from backend package.json (JVM tests via clojure directly) - Unify common/scripts/test-quiet.js with frontend's BUILD_STEPS pattern - Add execution discipline section to mem:testing (no piping, tee to file) - Add READ mem:testing FIRST directives to module testing docs AI-assisted-by: deepseek-v4-flash |
||
|
Andrey Antukh
|
c50ec233ae | Merge remote-tracking branch 'origin/staging' into develop | ||
|
Andrey Antukh
|
85dbf14344 | 📎 Add better planner skill and improve testing doc | ||
|
Dr. Dominik Jain
|
2c15dcdb84
|
✨ Add systematic component tests via a composable test model (#10529)
* ✨ Add systematic component tests via a composable test model Introduce a framework for systematically testing Penpot component behaviour (synchronisation/propagation, swaps, variant switches, nesting), plus a first suite of cases built on it. A test is expressed as a COMPOSITION OF OPERATIONS over a "situation" (an in-memory file value plus named role bindings). Operations are reified as data and composed by two combinators — `in-sequence` (threads the situation) and `one-of`/`optional` (alternatives, enumerated into concrete variants). So one written case stands for a whole matrix of variants, and coverage grows by composition rather than by copying tests. Operations drive the REAL production change pipeline, and event-operations dispatch the REAL workspace events and await settlement, so the production watcher's automatic propagation is what is exercised — the tests reflect genuine app behaviour, not a reimplementation. Structure (frontend/test/frontend_tests/composable_tests/): - core — the domain-agnostic engine: situation, the operation and enumeration protocols, the combinators, and the runners. - comp/nodes — the component operations (create/instantiate/reset, nesting, swap, the variant ops, child add/remove/move, change, undo, library sync). - comp/setups — component-shaped starting configurations. - interpreter — runs a case against the real frontend store: sync-ops apply directly, event-ops dispatch real events and await settlement (absorbing sync-file's delayed status RPC, which would otherwise leak an error into subsequent tests). - comp/sync-test — the cases (B-F, H, I, K, L, M). This is test-only code with a single consumer — the frontend test suite (the layer that runs the real app) — so it lives entirely under the frontend test tree as .cljs, not under app/common. The framework and its cases are documented in the project memory frontend/composable-component-tests, added alongside. Co-authored-by: Claude <noreply@anthropic.com> * 🐛 Guard WASM mock teardown against an empty snapshot `teardown-wasm-mocks!` unconditionally restored from the `originals` atom. When run without a matching setup (double teardown, or `with-wasm-mocks*` misused around an async test body), the snapshot is empty and every WASM API function was `set!` to nil — permanently, for the remainder of the test run. Any later code calling one of them (e.g. a leaked debounced resize-wasm-text event firing during a subsequent test namespace) then crashed with "initialized? is not a function". Make the restore a no-op when there is nothing to restore. Co-authored-by: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Andrey Antukh
|
1f4b85209e | 📚 Simplify the ia asistance note on creating-prs serena workflow | ||
|
Andrey Antukh
|
ad4dae5f28 | 📚 Add testing principles to serena doc | ||
|
Andrey Antukh
|
64b0bff7bd | 📚 Update serena creating-prs workflow documentation | ||
|
Andrey Antukh
|
455c7f5cae | 📚 Update serena doc related to creating issues | ||
|
Andrey Antukh
|
b3d1a7aa8b | 📚 Add better dev tools doc to serena | ||
|
Andrey Antukh
|
bcf77767b2 | 📖 Update ai agents documentation | ||
|
Andrey Antukh
|
8b734d9844 | 📎 Add postgresql client tool wrapper for devenv |