github/penpot
1
0
Fork 0
mirror of https://github.com/penpot/penpot.git synced 2026-10-30 06:20:32 +00:00
Code Issues Packages Projects Releases Wiki Activity

♻️ Swap the graph console from Cypher sessions to the overlay

Browse Source 
app.graph.debug sessions now hold a datascript overlay in an atom:
readers never lock, the msgbus sync loop is the only writer, and the
Ladybug engine no longer appears on the session path. The console
query box speaks Datalog; the default query lists component instances
by page, with filter_* columns carrying entity ids for the graph-view
result filter exactly as before. The node inspector asks
[:find ?a ?v :where [<eid> ?a ?v]] and renders datom pairs. Node ids
in the graph-view export are datascript eids, which keeps two shapes
sharing one uuid in different containers distinct; the Cypher export
merged them.

export-session-database! is retired with the session tier: an overlay
has no database file. The reproducible .lbug artifact stays on
source=file through app.graph.ingest, which keeps the batch-export
tier and the beadpot parity route alive.

AI-assisted-by: claude-fable-5
This commit is contained in:
Álvaro Tejero Cantero 2026-08-21 02:27:32 +02:00
parent 31fb7c64a6
commit d5324e9e02
No known key found for this signature in database
3 changed files with 276 additions and 347 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

107
backend/resources/app/templates/graph-console.tmpl
View File

@ -23,8 +23,8 @@ Graph Console
<fieldset>
<legend>Load graph from Penpot</legend>
<desc>
Click file or paste UUID to load Penpot file into an in-memory
Ladybug database. Loading a new file replaces the previous one.
Click file or paste UUID to load a Penpot file into an in-memory
datascript overlay. Loading a new file replaces the previous one.
</desc>
<div id="graph-files-tree" style="font-size: 13px; margin-bottom: 6px;">Loading&hellip;</div>
<form id="graph-load-form" method="post" action="/dbg/actions/graph-load">
@ -34,7 +34,7 @@ Graph Console
value="{% if session %}{{session.file-id}}{% endif %}" />
{% if session %}
<input type="submit" value="Reload"
title="Re-ingests the file in the box from scratch — the recovery fallback when live sync drifted or skipped changes. Paste a different UUID to switch files." />
title="Rebuilds the session for the file in the box from scratch — the recovery fallback when live sync drifted or skipped changes. Paste a different UUID to switch files." />
<input type="submit" value="Unload" form="graph-unload-form"
title="Drop the in-memory session and free its memory" />
{% else %}
@ -55,10 +55,10 @@ Graph Console
File: <span id="graph-file-crumbs"></span><b><a id="graph-penpot-link"
data-file-id="{{session.file-id}}"
target="_blank">{{session.name}}</a></b>
<span id="graph-bm" style="color: #666;"
title="resident memory of the session's in-memory DB (buffer manager)"></span><br />
<span title="The file's revision when it was ingested vs the revision of the last live change applied to the graph. They start equal; a graph value behind the workspace means missed changes — use Reload.">
Revisions: ingested at <b>{{session.revn}}</b> · graph now
<span id="graph-datoms" style="color: #666;"
title="indexed datoms in the session's in-memory overlay"></span><br />
<span title="The file's revision when the session was loaded vs the revision of the last live change applied to the graph. They start equal; a graph value behind the workspace means missed changes — use Reload.">
Revisions: loaded at <b>{{session.revn}}</b> · graph now
<b id="graph-sync-revn">{% if session.graph-revn %}{{session.graph-revn}}{% else %}{{session.revn}}{% endif %}</b>
</span><br />
Graph size: <b id="graph-size">…</b><br />
@ -88,8 +88,8 @@ Graph Console
</fieldset>
<fieldset>
<legend>Query graph, read-only (<a href="https://docs.ladybugdb.com/cypher/"
target="_blank">LadybugDB Cypher</a>)</legend>
<legend>Query graph, read-only (<a href="https://github.com/tonsky/datascript"
target="_blank">datascript Datalog</a>)</legend>
<form id="graph-query-form" method="post" action="/dbg/actions/graph-query">
<div class="row">
<textarea name="query" rows="8" style="width:100%; font-family: monospace;"
@ -171,7 +171,7 @@ Graph Console
</label>
</legend>
<desc>
Live view of the in-memory Ladybug graph (AntV G6). Double-click
Live view of the in-memory overlay graph (AntV G6). Double-click
folds containers when folding is on.
</desc>
<div id="graph-legend" style="font-size: 12px; margin: 4px 0;"></div>
@ -691,9 +691,9 @@ Graph Console
function toG6Data(data, collapsedIds, withCombos, presetPositions, diffFold) {
const nodes = data.nodes || [];
const edges = (data.edges || [])
.slice()
.sort(function (a, b) { return (a.position || 0) - (b.position || 0); });
// The datascript export carries no edge position; hierarchy comes
// from IsChildOf, so the export's edge order is fine as-is.
const edges = data.edges || [];
// Hierarchy (tree ranking, combo derivation, fold-ability) comes from
// IsChildOf only — still a tree by construction. Other rels
// (IsInstanceOf, …) are overlay edges drawn between positioned nodes.
@ -779,7 +779,7 @@ Graph Console
});
const g6edges = edges.map(function (e) {
return { source: e.source, target: e.target,
data: { position: e.position, rel: e.rel || "IsChildOf" } };
data: { rel: e.rel || "IsChildOf" } };
});
return { nodes: g6nodes, edges: g6edges, combos: combos };
}
@ -1040,8 +1040,9 @@ Graph Console
}
// Query filter: induced subgraph of the already-exported graph, selected
// by node ids found in the last Cypher result. No reconstruction from the
// query result is needed — ids are enough to slice the cached export.
// by node ids found in the last query result. No reconstruction from the
// query result is needed — the filter_* column values are eids, enough to
// slice the cached export.
function filteredGraphData() {
if (!graphFilterIds && !hiddenTables.size) return lastGraphData;
const keep = {};
@ -1058,14 +1059,21 @@ Graph Console
revn: lastGraphData.revn, truncated: lastGraphData.truncated };
}
const UUID_RE = /[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/gi;
// Node ids in results are datascript eids: decimal strings that match the
// export's node ids. Only filter_* columns carry them (convention; see the
// default query and `renderQueryOutput`).
function idsInResult(result) {
const ids = new Set();
const filterIdx = [];
(result.columns || []).forEach(function (col, i) {
if (/^filter_/.test(col)) filterIdx.push(i);
});
(result.rows || []).forEach(function (row) {
row.forEach(function (cell) {
const found = String(cell).match(UUID_RE);
if (found) found.forEach(function (m) { ids.add(m.toLowerCase()); });
filterIdx.forEach(function (i) {
const cell = row[i];
if (cell == null || cell === "") return;
const s = String(cell);
if (/^\d+$/.test(s)) ids.add(s);
});
});
return ids;
@ -1159,13 +1167,11 @@ Graph Console
// Session-fieldset size line stays live even through skipped
// repaints (unlike the status line under the canvas); hover
// shows per-table counts.
// bm-bytes = the session DB's buffer-manager usage (CALL
// bm_info()), i.e. actual resident memory; shown next to the
// file name. Floor-dominated: the wide slice-3 schema costs
// ~115 MiB before any data.
const bmEl = document.getElementById("graph-bm");
if (bmEl && data["bm-bytes"]) {
bmEl.textContent = "(" + (data["bm-bytes"] / 1048576).toFixed(1) + " MiB)";
// datoms = the overlay's indexed datom count; shown next to the
// file name.
const datomsEl = document.getElementById("graph-datoms");
if (datomsEl && data.datoms != null) {
datomsEl.textContent = "datoms: " + data.datoms;
}
const sizeEl = document.getElementById("graph-size");
if (sizeEl) {
@ -1199,10 +1205,10 @@ Graph Console
}
// --- node inspector --------------------------------------------------
// Click a node → its full attribute row from the graph DB, rendered in
// a panel under the canvas. Panel over tooltip: the projected tables
// carry ~80 columns — too much for a tooltip, and the panel persists
// for side-by-side reading without obstructing the graph.
// Click a node → its datoms from the overlay, rendered in a panel under
// the canvas. Panel over tooltip: one datom row per indexed attribute,
// and the panel persists for side-by-side reading without obstructing
// the graph.
function runConsoleQuery(query) {
const formData = new FormData();
formData.append("query", query);
@ -1218,17 +1224,17 @@ Graph Console
function showNodeInspector(id) {
const panel = document.getElementById("graph-node-inspector");
if (!panel || !g6graph) return;
const eid = String(id);
// The id is interpolated into a Datalog data pattern — accept only
// what our own export produces (datascript eids, decimal strings).
if (!/^\d+$/.test(eid)) return;
let datum = null;
try { datum = g6graph.getNodeData(id); } catch (_err) { return; }
const table = datum && datum.data && datum.data.table;
try { datum = g6graph.getNodeData(eid); } catch (_err) { return; }
const table = (datum && datum.data && datum.data.table) || "";
const label = (datum && datum.data && datum.data.label) || "";
// Both values are interpolated into Cypher — accept only what our own
// export produces (bare table names, UUID ids).
if (!table || !/^[A-Za-z]+$/.test(table)) return;
if (!/^[0-9a-f-]{36}$/i.test(id)) return;
panel.style.display = "block";
panel.textContent = "loading " + id + "…";
runConsoleQuery("MATCH (n:`" + table + "` {id: uuid('" + id + "')}) RETURN n.*;")
panel.textContent = "loading " + eid + "…";
runConsoleQuery("[:find ?a ?v :where [" + eid + " ?a ?v]]")
.then(function (data) {
if (data.error) {
panel.textContent = "inspector error: " + data.error;
@ -1239,33 +1245,32 @@ Graph Console
let html = '<div style="display: flex; justify-content: space-between;">'
+ "<b>" + escapeHtml(table) + " " + escapeHtml(label) + "</b>"
+ '<button type="button" id="graph-inspector-close">close</button></div>'
+ '<div style="color: #666;">' + escapeHtml(id) + "</div>";
+ '<div style="color: #666;">' + escapeHtml(eid) + "</div>";
if (!rows.length) {
html += "<div>not in the graph DB (removed?)</div>";
} else {
const columns = result.columns || [];
let hidden = 0;
// Two-column flow; JSON-like values fold behind the same
// One datom row per pair; JSON-like values fold behind the same
// disclosure triangle the file tree uses.
html += '<div style="column-count: 2; column-gap: 16px; margin-top: 4px;">';
columns.forEach(function (col, i) {
const val = rows[0][i];
// Ladybug's value->clj fallback renders SQL nulls as "NULL".
rows.forEach(function (row) {
const key = row[0];
const val = row[1];
// The console's format-cell renders nil as "NULL".
if (val === null || val === undefined || val === "" || val === "NULL") {
hidden += 1;
return;
}
const key = escapeHtml(col.replace(/^n\./, ""));
const sval = String(val);
// Fold structured values ({…}) and walls of text (long arrays
// like migrations) behind a disclosure triangle.
if (sval.indexOf("{") !== -1 || sval.length > 120) {
html += '<details style="break-inside: avoid;">'
+ '<summary style="cursor: pointer;"><code>' + key + '</code></summary>'
+ '<summary style="cursor: pointer;"><code>' + escapeHtml(key) + '</code></summary>'
+ '<pre style="white-space: pre-wrap; word-break: break-all; margin: 2px 0 4px 1.2em;">'
+ escapeHtml(sval) + "</pre></details>";
} else {
html += '<div style="break-inside: avoid;"><code>' + key
html += '<div style="break-inside: avoid;"><code>' + escapeHtml(key)
+ '</code>: <code>' + escapeHtml(sval) + "</code></div>";
}
});
@ -1532,8 +1537,8 @@ Graph Console
html += "</tbody></table></div></fieldset>";
output.innerHTML = html;
// Offer to view the result as an induced subgraph: any UUID appearing
// in any result cell selects that node in the loaded graph. The query
// Offer to view the result as an induced subgraph: filter_* columns
// carry eids, which select those nodes in the loaded graph. The query
// goes with the ids, so the view can re-run it and follow the graph.
const ids = idsInResult(result);
if (ids.size) {

459
backend/src/app/graph/debug.clj
View File

@ -5,24 +5,30 @@
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.debug
"In-memory Ladybug sessions for the debug graph console."
"In-memory overlay sessions for the debug graph console.
A session is a datascript overlay of one file
(`app.graph.overlay`), maintained from the msgbus file-change feed by
`app.graph.overlay.sync` and queried in Datalog through
`app.graph.overlay.console`. The Ladybug engine no longer appears on
this path: it keeps the batch-export tier (`app.graph.ingest`), where
columnar output is the point. The overlay is immutable data in an atom,
so readers never lock and the sync loop is the only writer."
(:require
[app.binfile.common :as bfc]
[app.common.exceptions :as ex]
[app.common.logging :as l]
[app.common.time :as ct]
[app.graph.ingest :as graph.ingest]
[app.graph.ladybug :as ladybug]
[app.graph.schema.nodes :as nodes]
[app.graph.sync :as graph.sync]
[app.db :as db]
[app.graph.overlay :as overlay]
[app.graph.overlay.console :as console]
[app.graph.overlay.queries :as queries]
[app.graph.overlay.sync :as overlay.sync]
[app.msgbus :as mbus]
[clojure.java.io :as io]
[app.srepl.helpers :as h]
[clojure.string :as str]
[promesa.exec.csp :as sp])
(:import
com.ladybugdb.Connection
com.ladybugdb.Database
org.apache.arrow.memory.BufferAllocator
org.apache.arrow.memory.RootAllocator))
[datascript.core :as d]
[promesa.exec.csp :as sp]))
(set! *warn-on-reflection* true)
@ -30,15 +36,20 @@
"Default console query, written to be self-explanatory in the textarea.
The `filter_*` columns carry node ids for the graph-view result filter;
the results table hides them (see `hide-filter-columns` and the
template's `renderQueryOutput`)."
(str "MATCH (s)-[r]->(t)\n"
"// WHERE some condition\n"
"RETURN label(s) AS src, s.name,\n"
" label(r) AS rel,\n"
" t.name, label(t) AS tgt,\n"
"\n"
"// filter_* columns omitted from table; these needed for graph view\n"
"s.id AS filter_src_id, t.id AS filter_tgt_id;"))
template's `renderQueryOutput`). `$` is bound to the overlay and `%` to
the shared rule set of `app.graph.overlay.queries`."
(str "[:find ?component ?instance ?page\n"
" ?filter_src_id ?filter_tgt_id\n"
" :in $ %\n"
" :where\n"
" (is-instance-of ?s ?c)\n"
" [?c :component/name ?component]\n"
" [?s :shape/name ?instance]\n"
" [?s :shape/container ?pc]\n"
" [?pc :container/name ?page]\n"
" ;; filter_* columns omitted from the table; they drive the graph view\n"
" [(identity ?s) ?filter_src_id]\n"
" [(identity ?c) ?filter_tgt_id]]"))
(defonce ^:private sessions
(atom {}))
@ -48,30 +59,11 @@
(str profile-id))
(defn- destroy-session!
"Close a session's Ladybug resources.
The Arrow allocator outlives the connection: Ladybug keeps the staged
buffers until its tables are dropped, no later than connection close, so
the allocator closes last — see `app.graph.arrow/with-allocator!`."
[{:keys [conn db allocator sync-ch msgbus]}]
[{:keys [sync-ch msgbus]}]
(when sync-ch
(sp/close! sync-ch)
(when msgbus
(mbus/purge! msgbus [sync-ch])))
(when conn
(ex/ignoring (.close ^Connection conn)))
(when db
(ex/ignoring (.close ^Database db)))
(when allocator
(ex/ignoring (.close ^BufferAllocator allocator))))
(defn- slim-ingest-meta
"Drop full projection rows from session meta.
`build-index` needs `:nodes`/`:edges` once; keeping them in the session
duplicates the entire graph on the JVM heap for every Load."
[meta]
(update meta :projection #(select-keys % [:stats])))
(mbus/purge! msgbus [sync-ch]))))
(defn- format-cell
[value]
@ -90,32 +82,27 @@
:row-count (count rows)})
(defn- apply-file-change!
[conn profile-id {:keys [changes revn file-id]}]
[profile-id {:keys [changes revn file-id]}]
(try
(some-> (get @sessions (session-key profile-id))
(as-> current
(when (= file-id (:file-id current))
(let [lock (:lock current)
result (locking lock
(graph.sync/apply-changes!
conn (:index current) changes revn))
sync-at (ct/now)]
(swap! sessions assoc-in [(session-key profile-id) :index]
(:index result))
(swap! sessions update-in [(session-key profile-id) :meta]
(fn [meta]
(cond-> (-> meta
(update :sync dissoc :error)
(assoc-in [:sync :last-at] sync-at)
(assoc-in [:sync :last-applied] (:applied result))
(assoc-in [:sync :last-skipped] (:skipped result)))
(seq (:applied result))
(assoc :revn (:revn result)))))
(when (seq (:skipped result))
(l/dbg :hint "graph sync skipped changes"
:file-id (str file-id)
:revn revn
:skipped (:skipped result)))))))
(when-let [current (get @sessions (session-key profile-id))]
(when (= file-id (:file-id current))
(let [result (overlay.sync/apply-changes @(:db-atom current) changes)
sync-at (ct/now)]
(reset! (:db-atom current) (:db result))
(swap! sessions update-in [(session-key profile-id) :meta]
(fn [meta]
(cond-> (-> meta
(update :sync dissoc :error)
(assoc-in [:sync :last-at] sync-at)
(assoc-in [:sync :last-applied] (:applied result))
(assoc-in [:sync :last-skipped] (:skipped result)))
(seq (:applied result))
(assoc :graph-revn (long revn)))))
(when (seq (:skipped result))
(l/dbg :hint "graph sync skipped changes"
:file-id (str file-id)
:revn revn
:skipped (:skipped result))))))
(catch Throwable cause
(l/wrn :hint "graph sync failed"
:file-id (str file-id)
@ -124,17 +111,16 @@
(ex-message cause)))))
(defn- start-sync-loop!
[{:keys [conn profile-id file-id] :as session}]
[{:keys [profile-id file-id] :as session}]
(if-let [msgbus (:msgbus session)]
(let [sync-ch (sp/chan :buf (sp/dropping-buffer 64))]
(mbus/sub! msgbus :topic file-id :chan sync-ch)
;; Recur ONLY while the channel is open. A bare `(recur)` after
;; `take!` returns nil would spin forever and pin this Connection
;; (and its Ladybug Database native memory) across every Load.
;; `take!` returns nil would spin forever across every Load.
(sp/go-loop []
(when-let [message (sp/take! sync-ch)]
(when (= :file-change (:type message))
(apply-file-change! conn profile-id message))
(apply-file-change! profile-id message))
(recur)))
(assoc session :sync-ch sync-ch))
session))
@ -142,11 +128,11 @@
(defn session-info
"Return a public view of the current session for `profile-id`, if any."
[profile-id]
(when-let [{:keys [file-id meta loaded-at index]} (get @sessions (session-key profile-id))]
(when-let [{:keys [file-id meta loaded-at]} (get @sessions (session-key profile-id))]
{:file-id file-id
:name (:name meta)
:revn (:revn meta)
:graph-revn (:revn index)
:graph-revn (:graph-revn meta)
:schema-version (:schema-version meta)
:projection (:projection meta)
:sync (:sync meta)
@ -156,223 +142,180 @@
"Return incremental sync status for the active session."
[profile-id]
(when-let [session (get @sessions (session-key profile-id))]
(let [{:keys [file-id meta index loaded-at]} session]
(let [{:keys [file-id meta loaded-at]} session]
{:file-id file-id
:revn (:revn meta)
:graph-revn (:revn index)
:graph-revn (:graph-revn meta)
:sync (:sync meta)
:loaded-at (ct/format-inst loaded-at :iso)})))
(defn unload-session!
"Close and discard the in-memory graph for `profile-id`."
"Discard the in-memory overlay for `profile-id`."
[profile-id]
(when-let [session (get @sessions (session-key profile-id))]
(destroy-session! session))
(swap! sessions dissoc (session-key profile-id)))
(defn load-session!
"Ingest `file-id` into a new in-memory Ladybug database for `profile-id`.
(defn- fetch-file!
[system file-id]
(let [file-id (h/parse-uuid file-id)
file (db/run! system #(bfc/get-file % file-id :realize? true))]
(when-not file
(ex/raise :type :not-found
:code :file-not-found
:hint "file not found"
:file-id (str file-id)))
(when-not (:data file)
(ex/raise :type :internal
:code :file-without-data
:hint "file has no data blob"
:file-id (str file-id)))
[file-id file]))
The session owns its Arrow allocator for its lifetime: the allocator stays
open until the connection and database close, so it is created here, passed
to ingest, and closed by `destroy-session!` — never inside ingest."
(defn load-session!
"Build the overlay of `file-id` for `profile-id`."
[cfg profile-id file-id]
(unload-session! profile-id)
(let [^BufferAllocator allocator (RootAllocator.)
^Database db (Database.)
^Connection conn (Connection. db)
msgbus (::mbus/msgbus cfg)]
(.setQueryTimeout conn 0)
(ladybug/ensure-extensions! conn)
(try
(let [meta (graph.ingest/ingest-on-connection! cfg conn file-id
:db-path ":memory:"
:skip-stats? true
:skip-validation? true
:arrow-alloc allocator)
index (graph.sync/build-index file-id (:revn meta) (:projection meta))
;; Discard projection rows after indexing — they are only needed
;; to seed the sync index and would otherwise leak heap on each Load.
meta (slim-ingest-meta meta)
session
;; :lock serializes access to the shared Connection between the
;; msgbus sync loop (writes) and HTTP handlers (reads); the Java
;; binding gives no thread-safety guarantee for one Connection.
(-> {:db db
:conn conn
:allocator allocator
:lock (Object.)
:file-id file-id
:meta meta
:index index
:msgbus msgbus
:profile-id profile-id
:loaded-at (ct/now)}
start-sync-loop!)]
(swap! sessions assoc (session-key profile-id) session)
meta)
(catch Throwable cause
(destroy-session! {:conn conn :db db :allocator allocator :msgbus msgbus})
(throw cause)))))
(let [msgbus (::mbus/msgbus cfg)
[file-id file] (fetch-file! cfg file-id)
started (System/nanoTime)
db (overlay/build (:data file) file)
build-ms (/ (- (System/nanoTime) started) 1e6)
stats (assoc (queries/stats db)
:edges (queries/edge-counts db)
:build-ms (long build-ms))
meta {:name (:name file)
:revn (:revn file)
:graph-revn (:revn file)
:schema-version overlay/schema-version
:projection {:stats stats}}
session (-> {:db-atom (atom db)
:file-id file-id
:meta meta
:msgbus msgbus
:profile-id profile-id
:loaded-at (ct/now)}
start-sync-loop!)]
(swap! sessions assoc (session-key profile-id) session)
meta))
(defn session-db
"The current overlay value for `profile-id`, or nil."
[profile-id]
(some-> (get @sessions (session-key profile-id)) :db-atom deref))
(defn query-session!
"Run a read-only `statement` against the in-memory graph for `profile-id`.
"Run a Datalog `statement` against the overlay for `profile-id`.
The statement is bound against the live schema before it runs, so a query
naming a table or a property that does not exist reports the binder's own
message and executes nothing. The engine's read/write analysis then decides
whether it may run at all: the console is an inspection surface, and a
session graph is rebuilt from the file by Reload, so a mutation from here
would produce a graph no rebuild reproduces."
Read-only by construction — `d/q` cannot transact — and gated against
function smuggling by `app.graph.overlay.console/check-query!`."
[profile-id statement]
(when (str/blank? statement)
(when (or (nil? statement) (= "" statement))
(ex/raise :type :validation
:code :missing-query
:hint "cypher query is required"))
(if-let [{:keys [conn lock]} (get @sessions (session-key profile-id))]
(locking lock
(let [{:keys [ok? error read-only?]} (ladybug/validate-on-connection! conn statement)]
(when-not ok?
(ex/raise :type :validation
:code :graph-query-invalid
:hint error))
(when-not read-only?
(ex/raise :type :validation
:code :graph-query-not-read-only
:hint "the graph console runs read-only queries"))
(-> (ladybug/query-on-connection! conn statement)
format-query-result)))
:hint "datalog query is required"))
(if-let [db (session-db profile-id)]
(-> (console/run-query db statement)
format-query-result)
(ex/raise :type :not-found
:code :graph-session-not-loaded
:hint "load a file graph before running queries")))
(def ^:private export-max-rows
"Row cap for graph-view export queries; far above expected per-file node
and edge counts. `:truncated` in the export signals when it was hit."
100000)
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; graph-view export
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defn- export-nodes
[conn]
(reduce
(fn [acc {:keys [table]}]
(let [stmt (str "MATCH (n:" (nodes/match-label table)
") RETURN n.id AS id, n.name AS name;")
{:keys [rows truncated?]}
(ladybug/query-on-connection! conn stmt :max-rows export-max-rows)]
(-> acc
(update :nodes into
(map (fn [[id label]]
{:id (str id) :label (str label) :table table}))
rows)
(update :truncated? #(or % truncated?)))))
{:nodes [] :truncated? false}
nodes/node-types))
(defn- entity-node
"Node payload for the G6 view. Node ids are datascript eids, which keeps
two shapes sharing one uuid in different containers distinct — the
Cypher export merged them."
[db eid]
(let [e (d/entity db eid)]
(cond
(:document/id e)
{:id (str eid) :label (or (:document/name e) (str (:document/id e))) :table "Document"}
(defn rel-tables
"Every relationship table in the open database, with whether it carries a
`position` property.
(:component/id e)
{:id (str eid) :label (or (:component/name e) (str (:component/id e))) :table "Component"}
Read from the catalog rather than listed here, so a newly ported transform's
rel table appears in the graph view without the console being told about it."
[conn]
(for [[table] (:rows (ladybug/query-on-connection!
conn "CALL show_tables() WHERE type = 'REL' RETURN name;"
:max-rows 1000))
:let [props (->> (ladybug/query-on-connection!
conn (str "CALL table_info('" table "') RETURN *;")
:max-rows 1000)
:rows
(into #{} (map (comp str second))))]]
{:table table :position? (contains? props "position")}))
(:container/id e)
{:id (str eid) :label (or (:container/name e) (str (:container/id e))) :table "Page"}
(defn- export-edges
[conn]
(reduce
(fn [acc {:keys [table position?]}]
(let [stmt (str "MATCH (a)-[r:`" table "`]->(b) "
"RETURN a.id AS source, b.id AS target, "
(if position? "r.position" "NULL") " AS position, "
"'" table "' AS rel;")
{:keys [rows truncated?]}
(ladybug/query-on-connection! conn stmt :max-rows export-max-rows)]
(-> acc
(update :edges into
(map (fn [[source target position rel]]
(cond-> {:source (str source)
:target (str target)
:rel (str rel)}
(some? position) (assoc :position position))))
rows)
(update :truncated? #(or % truncated?)))))
{:edges [] :truncated? false}
(rel-tables conn)))
(:shape/id e)
{:id (str eid)
:label (or (:shape/name e) (str (:shape/id e)))
:table (get overlay/shape-type->table (:shape/type e) "Shape")}
(defn- bm-usage-bytes
"Buffer-manager memory in use by this session's in-memory database
(`CALL bm_info()` → [mem_limit mem_usage]); nil if the call fails."
[conn]
(ex/ignoring
(-> (ladybug/query-on-connection! conn "CALL bm_info() RETURN *;" :max-rows 1)
:rows first second)))
(:color/id e)
{:id (str eid) :label (or (:color/name e) (str (:color/id e))) :table "Color"}
(:typography/id e)
{:id (str eid) :label (or (:typography/name e) (str (:typography/id e))) :table "Typography"}
(:token-set/id e)
{:id (str eid) :label (or (:token-set/name e) (str (:token-set/id e))) :table "TokenSet"}
(:token/id e)
{:id (str eid) :label (:token/name e) :table "Token"})))
(defn- ref-edges
[db attr rel]
(map (fn [dtm] {:source (str (:e dtm)) :target (str (:v dtm)) :rel rel})
(d/datoms db :avet attr)))
(defn- rule-edges
[db rule rel]
(map (fn [[s t]] {:source (str s) :target (str t) :rel rel})
(d/q (into [] (concat '[:find ?s ?t :in $ % :where] [(list rule '?s '?t)]))
db queries/rules)))
(defn- token-use-edges
[db]
(keep (fn [dtm]
(let [e (d/entity db (:e dtm))]
(when-let [token (:token-use/token e)]
{:source (str (:db/id (:token-use/shape e)))
:target (str (:db/id token))
:rel "UsesToken"})))
(d/datoms db :avet :token-use/shape)))
(defn export-graph-data!
"Export the node/edge inventory of the in-memory graph for `profile-id`
as plain data for the debug graph view. Returns nil when no session is
loaded. Queries the Ladybug database (not the sync index) so the view
reflects actual DB state, including drift."
"Export the node/edge inventory of the overlay for `profile-id` as plain
data for the debug graph view. Returns nil when no session is loaded."
[profile-id]
(when-let [{:keys [conn lock file-id index]} (get @sessions (session-key profile-id))]
(locking lock
(let [{:keys [nodes] nodes-truncated? :truncated?} (export-nodes conn)
{:keys [edges] edges-truncated? :truncated?} (export-edges conn)]
{:file-id (str file-id)
:revn (:revn index)
:truncated (boolean (or nodes-truncated? edges-truncated?))
:bm-bytes (bm-usage-bytes conn)
:nodes nodes
:edges edges}))))
(defn- delete-tree!
[^java.io.File file]
(when (.exists file)
(doseq [f (reverse (file-seq file))]
(.delete ^java.io.File f))))
(defn export-session-database!
"Materialize the in-memory session graph of `profile-id` as a `.lbug` file.
The console's graph is in-memory and live-synced, so it can differ from a
fresh projection of the same file — which is exactly when someone wants to
take it away and query it elsewhere. There is no \"save this database\"
primitive, so the transfer goes through Ladybug's `EXPORT DATABASE` (Parquet
per table) into a fresh on-disk database via `IMPORT DATABASE`.
Note the round trip drops table comments. Nothing in the graph is addressed
by a table comment: every table is resolved by name, so the loss costs
nothing.
Returns the path of the written database, or nil when no session is loaded.
The caller owns the file and must delete it once streamed."
[profile-id]
(when-let [{:keys [conn lock file-id]} (get @sessions (session-key profile-id))]
(let [stamp (System/nanoTime)
staging (io/file (System/getProperty "java.io.tmpdir")
(str "penpot-graph-session-" file-id "-" stamp))
db-path (str (io/file (System/getProperty "java.io.tmpdir")
(str file-id "-session-" stamp ".lbug")))]
(try
(locking lock
(ladybug/exec-on-connection!
conn [(str "EXPORT DATABASE '" (.getAbsolutePath staging)
"' (format='parquet');")]))
(ladybug/with-connection! db-path
(fn [target]
(ladybug/exec-on-connection!
target [(str "IMPORT DATABASE '" (.getAbsolutePath staging) "';")
"CHECKPOINT;"])))
db-path
(finally
(delete-tree! staging))))))
(when-let [{:keys [db-atom file-id meta]} (get @sessions (session-key profile-id))]
(let [db @db-atom
eids (into (sorted-set)
(mapcat #(map :e (d/datoms db :avet %)))
[:document/id :container/id :component/id :shape/id
:color/id :typography/id :token-set/id :token/id])
nodes (into [] (keep #(entity-node db %)) eids)
edges (-> []
(into (ref-edges db :shape/parent "IsChildOf"))
(into (ref-edges db :container/document "IsChildOf"))
(into (ref-edges db :component/document "IsChildOf"))
(into (ref-edges db :color/document "IsChildOf"))
(into (ref-edges db :typography/document "IsChildOf"))
(into (ref-edges db :token-set/document "IsChildOf"))
(into (ref-edges db :token/set "IsChildOf"))
(into (rule-edges db 'is-instance-of "IsInstanceOf"))
(into (rule-edges db 'refers-to "RefersTo"))
(into (rule-edges db 'fills-swap-slot "FillsSwapSlot"))
(into (ref-edges db :shape/fill-color "UsesColor"))
(into (ref-edges db :shape/stroke-color "UsesColor"))
(into (ref-edges db :shape/text-color "UsesColor"))
(into (ref-edges db :shape/uses-typography "UsesTypography"))
(into (token-use-edges db)))
;; component containers appear in both the component and the
;; container eid sweeps; the sorted set already deduplicates,
;; and `entity-node` classifies them as Component.
edges (into [] (distinct) edges)]
{:file-id (str file-id)
:revn (:graph-revn meta)
:truncated false
:datoms (count db)
:nodes nodes
:edges edges})))
(defn- hide-filter-columns
"Drop `filter_*` columns from a query result before HTML table render;

57
backend/src/app/http/debug.clj
View File

@ -395,10 +395,12 @@
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; `app.graph.*` resolves at call time, never at the top of this namespace.
;; `app.graph.ladybug` imports `com.ladybugdb.*`, so requiring it links the
;; Ladybug native library into the JVM, and this namespace loads on every
;; backend boot. The routes below are registered only under the `:graph` flag,
;; so with the flag off nothing resolves and no native code loads.
;; `app.graph.debug` (the console session tier) is a pure datascript overlay
;; and no longer touches Ladybug, but `app.graph.ingest` still projects
;; through `app.graph.ladybug`, which imports `com.ladybugdb.*` and links the
;; native library into the JVM; this namespace loads on every backend boot.
;; The routes below are registered only under the `:graph` flag, so with the
;; flag off nothing resolves and no native code loads.
(defn- graph-export-file
"Path of a freshly projected graph for `file-id`."
@ -413,59 +415,38 @@
:db-path db-path))
db-path))
(defn- graph-export-session
"Path of a snapshot of the caller's live in-memory graph for `file-id`."
[profile-id file-id]
(let [session-info (requiring-resolve 'app.graph.debug/session-info)
export-session-database! (requiring-resolve 'app.graph.debug/export-session-database!)
info (session-info profile-id)]
(when-not info
(ex/raise :type :not-found
:code :graph-session-not-loaded
:hint "no in-memory graph is loaded; load one first, or use source=file"))
(when-not (= file-id (:file-id info))
(ex/raise :type :validation
:code :graph-session-file-mismatch
:hint "the loaded session holds a different file"
:requested (str file-id)
:loaded (str (:file-id info))))
(export-session-database! profile-id)))
(defn graph-export-handler
"Stream a Ladybug `.lbug` database for a file.
"Stream a Ladybug `.lbug` database for a file, projected afresh from the
database by `app.graph.ingest/ingest-file!` — the reproducible artifact.
`source=file` (default) projects the file afresh from the database — the
reproducible artifact. `source=session` snapshots the caller's live
in-memory console graph instead, which live-sync may have moved away from a
fresh projection; taking that away to query it elsewhere is the whole point
of asking for it. Synchronous on each request."
[cfg {:keys [params] :as request}]
`source=session` is retired: a console session is a datascript overlay
with no database file to stream, so it raises
`:graph-session-export-retired`. Synchronous on each request."
[cfg {:keys [params]}]
(let [file-id (some-> params :file-id parse-uuid)
source (or (some-> params :source str/lower) "file")]
(when-not file-id
(ex/raise :type :validation
:code :missing-arguments
:hint "missing file-id"))
(when (= "session" source)
(ex/raise :type :validation
:code :graph-session-export-retired
:hint "the overlay session has no database file; use source=file for a fresh projection"))
(when-not (contains? #{"file" "session"} source)
(ex/raise :type :validation
:code :invalid-arguments
:hint "source must be 'file' or 'session'"
:source source))
(let [session? (= "session" source)
db-path (if session?
(graph-export-session (::session/profile-id request) file-id)
(graph-export-file cfg file-id))]
(let [db-path (graph-export-file cfg file-id)]
{::yres/status 200
;; A session export is a temp file this request owns; deleting it on
;; close would race the streaming body, so it is left for the OS temp
;; sweep. A file export is the canonical per-file database and is meant
;; The file export is the canonical per-file database and is meant
;; to persist.
::yres/body (io/input-stream db-path)
::yres/headers {"content-type" "application/octet-stream"
"content-disposition"
(str "attachment; filename=" file-id
(when session? "-session") ".lbug")}})))
(str "attachment; filename=" file-id ".lbug")}})))
(defn- graph-console-response
[data]