diff --git a/backend/src/app/graph/arrow.clj b/backend/src/app/graph/arrow.clj index baee44b48a..b698e3ab10 100644 --- a/backend/src/app/graph/arrow.clj +++ b/backend/src/app/graph/arrow.clj @@ -9,11 +9,9 @@ Rows are built as Arrow `VectorSchemaRoot`s in the JVM's off-heap memory, handed to Ladybug as a virtual table, and `COPY`d into the real one. No file - is written and no text is parsed, which is the whole point: the CSV path had - to serialize every value into a literal Ladybug re-parsed, and its inner - parser has no escaping at all, so a third of the old loader existed to detect - and route around that (`app.graph.bulk`, since deleted). Arrow carries MAP, - STRUCT, fixed-size arrays and multi-line strings natively. + is written and no value is rendered as text for the engine to re-parse, so + nothing in this path needs escaping. Arrow carries MAP, STRUCT, fixed-size + arrays and multi-line strings natively. The type language is Ladybug's, read recursively by `app.graph.schema.values`; this namespace adds the matching Arrow `Field` and a writer for each shape. @@ -21,19 +19,20 @@ into a packed integer — exactly as it does for the Cypher path, so the two writers cannot disagree. - Engine facts this file depends on, each verified against lbug 0.18.2: + Engine facts this file depends on, each verified against lbug 0.19.1: - An Arrow table is **not** a `COPY` source identifier, but it *is* a MATCH-able node label: `COPY T FROM (MATCH (n:stg) RETURN n.a AS a, …)`. - A MAP vector's `entries` child struct must be non-nullable, and `MapVector/getWriter` silently promotes it to a sparse union — so map vectors are built from an explicit `Field` and filled child-first. - - Ladybug names a staged struct's fields from the Arrow child names and - quotes none of them, so a reserved field name (`column`) must arrive - already backticked. - - `createArrowRelTable` cannot resolve endpoints against a UUID-keyed node - table under any endpoint encoding, so edges are staged as a node table and - joined by the `COPY` subquery instead." + - Ladybug quotes the column and table names it interpolates into the staged + table's DDL, and does not quote a STRUCT member name. So a top-level field + arrives plain and a struct member whose name is a reserved word (`column`) + arrives backticked. + - `createArrowRelTable` resolves a UUID-keyed endpoint only from a + `FixedSizeBinary(16)` column carrying the `arrow.uuid` extension, so edges + are staged as a node table and joined by the `COPY` subquery instead." (:require [app.common.json :as json] [app.graph.ladybug :as ladybug] @@ -240,20 +239,19 @@ (defn- node-batch "One `VectorSchemaRoot` holding every projected row of `table`. - Fields are named with their **backticks** (`cypher-property-key`, not - `column-name`): Ladybug derives the staged table's DDL from the Arrow field - names and quotes none of them, so a column whose name is a reserved word — - `Page.index`, `Document.options` — fails `createArrowTable` outright unless it - arrives already quoted. Same rule as for struct field names, and the `COPY` - projection below spells them the same way." + Fields carry the plain column name. Ladybug quotes every identifier it + interpolates into the staged table's DDL, so a name that is a reserved word + (`Page.index`, `Document.options`) arrives unquoted and a name arriving + pre-quoted comes out doubly backticked and fails to parse. The `COPY` + projection below is Cypher, not DDL, so it quotes the same names itself." ^VectorSchemaRoot [^BufferAllocator allocator table rows] (let [columns (nodes/column-keys table) - fields (mapv (fn [k] (column-field (nodes/cypher-property-key table k) + fields (mapv (fn [k] (column-field (nodes/column-name table k) (nodes/column-ladybug-type table k))) columns) root (VectorSchemaRoot/create (Schema. ^List fields) allocator)] (doseq [k columns] - (fill-vector! root (nodes/cypher-property-key table k) + (fill-vector! root (nodes/column-name table k) (nodes/column-ladybug-type table k) rows #(get % k) (nodes/column-map-key-fn table k))) (.setRowCount root (count rows)) diff --git a/backend/src/app/graph/ladybug.clj b/backend/src/app/graph/ladybug.clj index 15fa276520..ddf8936759 100644 --- a/backend/src/app/graph/ladybug.clj +++ b/backend/src/app/graph/ladybug.clj @@ -103,7 +103,7 @@ :else (format-string (str v)))) (defn map-type? - "Is `ladybug-type` a MAP column? Those cannot be bulk-loaded from CSV." + "Is `ladybug-type` a MAP column?" [ladybug-type] (and (string? ladybug-type) (str/starts-with? ladybug-type "MAP(") @@ -362,11 +362,11 @@ (defn- ->param-value "Clojure scalar → `Value` for prepared-statement binding. - Every parameter goes through here, unconditionally: an unwrapped value does - not raise, it SIGSEGVs the JVM in `lbug_value_clone`. Lists and maps are not - supported by the JNI `Value` constructor at all, so `MAP`, `STRUCT` and - `T[]` columns stay literal-rendered (`format-typed-value`) — the `:else` - raise below means a caller tried to bind one." + This is the only `Value` constructor on the write path, so every parameter + is wrapped here. Parameters are scalars: the `Value` constructor takes no + list or map, so `MAP`, `STRUCT` and `T[]` columns stay literal-rendered + (`format-typed-value`) and the `:else` raise below means a caller tried to + bind one." ^Value [v] (cond (nil? v) (Value/createNull) ; no explicit type needed diff --git a/backend/src/app/graph/schema/nodes.clj b/backend/src/app/graph/schema/nodes.clj index 9f863ee310..e45c32c6b9 100644 --- a/backend/src/app/graph/schema/nodes.clj +++ b/backend/src/app/graph/schema/nodes.clj @@ -9,7 +9,7 @@ Each registry entry declares Penpot Malli sources plus projection options (`:drop`, optional `:extra`). Derived artifacts — Ladybug - DDL, CSV columns, validation, type dispatch — all flow from that. + DDL, Arrow fields, validation, type dispatch — all flow from that. Column *names* and *types* are not ours to choose: they are the contract downstream consumers read, and beadpot owns it. Every @@ -170,8 +170,8 @@ (defn column-keys "Projected column keys for `table`, in registry order. - Keys the beadpot contract drops on this table are omitted, so column - order, the CSV header and the DDL cannot disagree about what exists." + Keys the beadpot contract drops on this table are omitted, so the column + order, the Arrow batch and the DDL cannot disagree about what exists." [table] (into [] (comp (map first)