⚡ Build the overlay bottom-up from datoms instead of transacting

The transact path spends its time resolving one tempid per entity, not
inserting datoms: the 2026-08-21 retrofit added 48 per cent more datoms
for no extra time. datascript's init-db skips that resolution and builds
the B-trees from sorted arrays, and build-tx already mints deterministic
string tempids from document ids, so assigning final entity ids is one
interning pass over the transaction.

On the 21,218-shape design system, measured back to back in one warm
JVM: 410 ms against 2,150 ms for the same 217,598 datoms. The graphs are
the same graph. Entity ids differ and carry no meaning, and the datom
sets are identical once every id is read back through the tempid that
minted it, which the new test asserts along with the stats and the edge
counts.

This is sound only while every build transform reads the document rather
than the half-built index, because init-db has no index to read. The
existing transforms all do: euler-numbering walks :objects, ref-resolver
goes through ctf/find-ref-shape, and the asset maps come from :data.

backend suite unchanged at 161 pre-existing failures (media and storage,
unrelated), 825 tests and 5495 assertions against 824 and 5491 before.

backend/src/app/graph/overlay.clj

@@ -471,9 +471,70 @@
                   (:pages data)))]
     tx))
 
+(def ^:private card-many-attrs
+  "The attributes `schema` declares `:db.cardinality/many`, so that the
+  flattener knows which values are collections rather than scalars."
+  (into #{}
+        (keep (fn [[attr decl]]
+                (when (= :db.cardinality/many (:db/cardinality decl)) attr)))
+        schema))
+
+(def ^:private ref-attrs
+  "The attributes `schema` declares `:db.type/ref`, so that the flattener
+  knows which values are tempids to resolve."
+  (into #{}
+        (keep (fn [[attr decl]]
+                (when (= :db.type/ref (:db/valueType decl)) attr)))
+        schema))
+
+(defn build-datoms
+  "The projection of `build-tx`, flattened into datoms whose entity ids are
+  already assigned, which is what `datascript.core/init-db` takes.
+
+  Interning the tempid strings in first-appearance order is the whole of
+  the id assignment, and it is sound only because every entity is emitted
+  once and takes its identity from the document rather than from an upsert.
+  A build transform that read the index instead would break that property
+  and force the build back onto the transact path, which is what the rule
+  in this project's entry set exists to prevent."
+  [data file]
+  (let [tx   (build-tx data file)
+        eids (reduce (fn [acc entity]
+                       (let [tempid (:db/id entity)]
+                         (if (contains? acc tempid)
+                           acc
+                           (assoc acc tempid (inc (count acc))))))
+                     {}
+                     tx)]
+    (persistent!
+     (reduce (fn [acc entity]
+               (let [eid (get eids (:db/id entity))]
+                 (reduce-kv
+                  (fn [acc attr value]
+                    (if (= :db/id attr)
+                      acc
+                      (let [ref? (contains? ref-attrs attr)
+                            emit (fn [acc value]
+                                   (conj! acc (d/datom eid attr
+                                                       (if ref? (get eids value) value))))]
+                        (if (contains? card-many-attrs attr)
+                          (reduce emit acc value)
+                          (emit acc value)))))
+                  acc
+                  entity)))
+             (transient [])
+             tx))))
+
 (defn build
   "Project file `data` into a fresh overlay. Pure: returns a datascript
-  database value."
+  database value.
+
+  The index is built bottom-up from datoms rather than transacted, which
+  on a 21,218-shape file is 410 ms against 2,150 ms for the same 217,598
+  datoms, measured back to back in one JVM. The two paths produce the same
+  graph: entity ids differ, and the datom set is identical once the ids are
+  read back through their tempids, which `backend-tests.graph-overlay-test`
+  asserts."
   ([data] (build data nil))
   ([data file]
-   (d/db-with (d/empty-db schema) (build-tx data file))))
+   (d/init-db (build-datoms data file) schema)))

backend/test/backend_tests/graph_overlay_test.clj

@@ -870,3 +870,51 @@
               :refers-to       6
               :fills-swap-slot 1}
              (queries/ladybug-parity-counts db)))))
+
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;; the build path
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+
+(t/deftest the-bulk-build-and-a-transaction-produce-one-graph
+  ;; `overlay/build` assigns entity ids itself and loads the index bottom-up,
+  ;; where transacting the same entity maps lets datascript assign them. The
+  ;; two must index the same graph, and a transform that read the half-built
+  ;; index instead of the document is the way that stops being true: the
+  ;; bulk path has no half-built index to read, so such a transform would
+  ;; silently see nothing here and see something on the transact path.
+  ;;
+  ;; Entity ids differ between the paths and carry no meaning, so the
+  ;; comparison reads every id back through the tempid that minted it.
+  (let [{:keys [file]} (semantic-fixture)
+        tx          (overlay/build-tx (:data file) file)
+        transacted  (d/with (d/empty-db overlay/schema) tx)
+        by-tempid   (fn [tempids]
+                      (into {} (map (fn [[tempid eid]] [eid tempid]))
+                            (dissoc tempids :db/current-tx)))
+        ref-attr?   (into #{}
+                          (keep (fn [[attr decl]]
+                                  (when (= :db.type/ref (:db/valueType decl)) attr)))
+                          overlay/schema)
+        canonical   (fn [db eid->tempid]
+                      (into #{}
+                            (map (fn [datom]
+                                   [(get eid->tempid (:e datom) (:e datom))
+                                    (:a datom)
+                                    (if (ref-attr? (:a datom))
+                                      (get eid->tempid (:v datom) (:v datom))
+                                      (:v datom))]))
+                            (d/datoms db :eavt)))
+        bulk        (overlay/build (:data file) file)
+        bulk-ids    (reduce (fn [acc entity]
+                              (let [tempid (:db/id entity)]
+                                (if (contains? acc tempid)
+                                  acc
+                                  (assoc acc tempid (inc (count acc))))))
+                            {}
+                            tx)]
+    (t/is (= (count (d/datoms (:db-after transacted) :eavt))
+             (count (d/datoms bulk :eavt))))
+    (t/is (= (canonical (:db-after transacted) (by-tempid (:tempids transacted)))
+             (canonical bulk (by-tempid (into {} (map (fn [[t e]] [t e])) bulk-ids)))))
+    (t/is (= (queries/stats (:db-after transacted)) (queries/stats bulk)))
+    (t/is (= (queries/edge-counts (:db-after transacted)) (queries/edge-counts bulk)))))