🐛 Let the engine quote the Arrow field names it interpolates

`node-batch` named every top-level Arrow field with backticks, so that a
column whose name is a reserved word (`Page.index`, `Document.options`)
survived the DDL Ladybug generates for a staged table. The engine now
quotes those identifiers itself, and it does not collapse a doubled
backtick, so a pre-quoted name reaches the parser as ``index`` and
`createArrowTable` fails outright:

    Parser exception: mismatched input '``' expecting PRIMARY

Name the fields with `column-name`. The `COPY` projection is Cypher
rather than DDL and keeps its own backticks through
`cypher-property-key`, and STRUCT member names keep theirs too: those
come out of `LogicalType::toString()`, which the DDL builder does not
touch, so an unquoted member called `column` still fails to parse.

Measured with `probes/arrow/probe25.clj` against lbug 0.19.1: a plain
top-level reserved word loads and reads back, a pre-quoted one fails to
parse, a plain STRUCT member fails to parse, and a pre-quoted one loads
and reads back.

Also re-dates the engine facts in the `app.graph.arrow` docstring to the
version they were checked against, drops the SIGSEGV note from
`->param-value` now that `Connection.execute` rejects an unwrapped
parameter, and removes two references to the CSV loader.

AI-assisted-by: mixed models
This commit is contained in:
Álvaro Tejero Cantero 2026-08-07 00:58:28 +02:00
parent 9126c2090d
commit ead3c54e5b
No known key found for this signature in database
3 changed files with 27 additions and 29 deletions

View File

@ -9,11 +9,9 @@
Rows are built as Arrow `VectorSchemaRoot`s in the JVM's off-heap memory,
handed to Ladybug as a virtual table, and `COPY`d into the real one. No file
is written and no text is parsed, which is the whole point: the CSV path had
to serialize every value into a literal Ladybug re-parsed, and its inner
parser has no escaping at all, so a third of the old loader existed to detect
and route around that (`app.graph.bulk`, since deleted). Arrow carries MAP,
STRUCT, fixed-size arrays and multi-line strings natively.
is written and no value is rendered as text for the engine to re-parse, so
nothing in this path needs escaping. Arrow carries MAP, STRUCT, fixed-size
arrays and multi-line strings natively.
The type language is Ladybug's, read recursively by `app.graph.schema.values`;
this namespace adds the matching Arrow `Field` and a writer for each shape.
@ -21,19 +19,20 @@
into a packed integer exactly as it does for the Cypher path, so the two
writers cannot disagree.
Engine facts this file depends on, each verified against lbug 0.18.2:
Engine facts this file depends on, each verified against lbug 0.19.1:
- An Arrow table is **not** a `COPY` source identifier, but it *is* a
MATCH-able node label: `COPY T FROM (MATCH (n:stg) RETURN n.a AS a, )`.
- A MAP vector's `entries` child struct must be non-nullable, and
`MapVector/getWriter` silently promotes it to a sparse union so map
vectors are built from an explicit `Field` and filled child-first.
- Ladybug names a staged struct's fields from the Arrow child names and
quotes none of them, so a reserved field name (`column`) must arrive
already backticked.
- `createArrowRelTable` cannot resolve endpoints against a UUID-keyed node
table under any endpoint encoding, so edges are staged as a node table and
joined by the `COPY` subquery instead."
- Ladybug quotes the column and table names it interpolates into the staged
table's DDL, and does not quote a STRUCT member name. So a top-level field
arrives plain and a struct member whose name is a reserved word (`column`)
arrives backticked.
- `createArrowRelTable` resolves a UUID-keyed endpoint only from a
`FixedSizeBinary(16)` column carrying the `arrow.uuid` extension, so edges
are staged as a node table and joined by the `COPY` subquery instead."
(:require
[app.common.json :as json]
[app.graph.ladybug :as ladybug]
@ -240,20 +239,19 @@
(defn- node-batch
"One `VectorSchemaRoot` holding every projected row of `table`.
Fields are named with their **backticks** (`cypher-property-key`, not
`column-name`): Ladybug derives the staged table's DDL from the Arrow field
names and quotes none of them, so a column whose name is a reserved word
`Page.index`, `Document.options` fails `createArrowTable` outright unless it
arrives already quoted. Same rule as for struct field names, and the `COPY`
projection below spells them the same way."
Fields carry the plain column name. Ladybug quotes every identifier it
interpolates into the staged table's DDL, so a name that is a reserved word
(`Page.index`, `Document.options`) arrives unquoted and a name arriving
pre-quoted comes out doubly backticked and fails to parse. The `COPY`
projection below is Cypher, not DDL, so it quotes the same names itself."
^VectorSchemaRoot [^BufferAllocator allocator table rows]
(let [columns (nodes/column-keys table)
fields (mapv (fn [k] (column-field (nodes/cypher-property-key table k)
fields (mapv (fn [k] (column-field (nodes/column-name table k)
(nodes/column-ladybug-type table k)))
columns)
root (VectorSchemaRoot/create (Schema. ^List fields) allocator)]
(doseq [k columns]
(fill-vector! root (nodes/cypher-property-key table k)
(fill-vector! root (nodes/column-name table k)
(nodes/column-ladybug-type table k)
rows #(get % k) (nodes/column-map-key-fn table k)))
(.setRowCount root (count rows))

View File

@ -103,7 +103,7 @@
:else (format-string (str v))))
(defn map-type?
"Is `ladybug-type` a MAP column? Those cannot be bulk-loaded from CSV."
"Is `ladybug-type` a MAP column?"
[ladybug-type]
(and (string? ladybug-type)
(str/starts-with? ladybug-type "MAP(")
@ -362,11 +362,11 @@
(defn- ->param-value
"Clojure scalar `Value` for prepared-statement binding.
Every parameter goes through here, unconditionally: an unwrapped value does
not raise, it SIGSEGVs the JVM in `lbug_value_clone`. Lists and maps are not
supported by the JNI `Value` constructor at all, so `MAP`, `STRUCT` and
`T[]` columns stay literal-rendered (`format-typed-value`) the `:else`
raise below means a caller tried to bind one."
This is the only `Value` constructor on the write path, so every parameter
is wrapped here. Parameters are scalars: the `Value` constructor takes no
list or map, so `MAP`, `STRUCT` and `T[]` columns stay literal-rendered
(`format-typed-value`) and the `:else` raise below means a caller tried to
bind one."
^Value [v]
(cond
(nil? v) (Value/createNull) ; no explicit type needed

View File

@ -9,7 +9,7 @@
Each registry entry declares Penpot Malli sources plus projection
options (`:drop`, optional `:extra`). Derived artifacts Ladybug
DDL, CSV columns, validation, type dispatch all flow from that.
DDL, Arrow fields, validation, type dispatch all flow from that.
Column *names* and *types* are not ours to choose: they are the
contract downstream consumers read, and beadpot owns it. Every
@ -170,8 +170,8 @@
(defn column-keys
"Projected column keys for `table`, in registry order.
Keys the beadpot contract drops on this table are omitted, so column
order, the CSV header and the DDL cannot disagree about what exists."
Keys the beadpot contract drops on this table are omitted, so the column
order, the Arrow batch and the DDL cannot disagree about what exists."
[table]
(into []
(comp (map first)