🐛 Name datascript's externs in the builds that rename properties

datascript.query labels a datom's slots with the string names "e", "a",
"v" and "tx", and reads one back with a bracket access on the datom
object. Closure renames the field and leaves the string, so the read
succeeds and returns whichever field inherited the old name. The index
reads the same slots as real fields, so storage keeps answering
correctly and the corruption stays silent.

Datascript ships datascript/externs.js to pin those five names and
declares it in its deps.cljs. shadow-cljs collects that declaration
into a :deps-externs entry it never populates, and reads it only behind
an unset :cljsjs-externs flag, so the declaration is inert and the
:worker release names the file itself. It costs 170 B on the probe
build's 436 197 B.

bench.debug becomes the gate rather than a reproducer: six readings,
one per datom slot plus a join and an index control, each paired with
the answer it must give, exiting non-zero on a wrong one. The new
ds-probe build compiles it with the worker's release options. Without
the externs entry that build reads nil for every transaction, and a
vanilla shadow project loses the value slot as well, which is why the
gate binds each slot on its own.

The ab-bench pin stays at :simple, and its comment now names the
reason that still holds: one compiler configuration across both
engines, matching the 2026-08-21 V8 table.

Answers mirror:ds-advanced-corruption.

AI-assisted-by: opus-5
This commit is contained in:
Álvaro Tejero Cantero 2026-08-31 23:51:18 +02:00
parent 6f537eb82e
commit 75c96c32cc
No known key found for this signature in database
2 changed files with 103 additions and 7 deletions

View File

@ -122,6 +122,12 @@
:source-map true
:elide-asserts true
:anon-fn-naming-policy :off
;; datascript's query layer reads a datom's slots by string property
;; name, so property renaming silently returns the wrong slot: see
;; bench.debug, the gate that proves it, and mirror:ds-advanced-corruption.
;; The library ships these externs and declares them in its deps.cljs,
;; which shadow-cljs does not read, so the build names them here.
:externs ["datascript/externs.js"]
:source-map-detail-level :all}}}
;; FIXME: maybe rename to :components ? (there are nothing storybook
@ -160,6 +166,10 @@
{:test {:init-fn frontend-tests.runner/init
:prepend-js ";if (typeof globalThis.navigator?.userAgent === 'undefined') { globalThis.navigator = {userAgent: ''}; };"}}}
;; The A/B bench pins :simple so the two engines are measured under one
;; compiler configuration; the 2026-08-21 datahike report's V8 table was
;; read that way. Lifting the pin needs :externs ["datascript/externs.js"]
;; here, for the reason bench.debug states, and a fresh table.
:ab-bench
{:target :node-script
:output-to "target/ab-bench.js"
@ -171,6 +181,32 @@
:output-wrapper false
:warnings {:fn-deprecated false}}}
;; The gate for mirror:ds-advanced-corruption: bench.debug reads one
;; database six ways and exits non-zero on a wrong answer. Its release
;; options mirror :worker's, because the worker is the build the answer is
;; about. To see the corruption, delete the :externs entry below and
;; rebuild: `--config-merge` reaches a build's own keys and never a
;; :release profile's, and externs from several places are concatenated, so
;; no command line can take this one away.
:ds-probe
{:target :node-script
:output-to "target/ds-probe.js"
:output-dir "target/ds-probe/"
:main bench.debug/-main
:compiler-options
{:output-feature-set :es2020
:output-wrapper false
:warnings {:fn-deprecated false}}
:release
{:compiler-options
{:fn-invoke-direct true
:optimizations #shadow/env ["PENPOT_BUILD_OPTIMIZATIONS" :as :keyword :default :advanced]
:elide-asserts true
:anon-fn-naming-policy :off
:externs ["datascript/externs.js"]}}}
:bench
{:target :node-script
:output-to "target/bench.js"

View File

@ -1,10 +1,70 @@
(ns bench.debug
"The `:advanced` gate for the engine that carries the worker overlay.
`datascript.query/lookup-pattern-db` labels a datom's slots with the
string property names \"e\", \"a\", \"v\" and \"tx\", and the query
engine reads them back with a raw property access
(`datascript.arrays/aget`). Closure's `:advanced` property renaming
moves those four fields and lets an unrelated one answer to the old
name, so the read succeeds and returns the wrong slot: a full-scan
query hands back a protocol-mask integer where a string belongs, and a
bound-value query matches nothing and returns the empty set. The index
reads the same datoms through real field access
(`datascript.db/val-at-datom`) and stays correct, which is what makes
the corruption silent.
Datascript ships the externs that pin those property names,
`datascript/externs.js`, and declares them in its `deps.cljs`.
Shadow-cljs collects that declaration into `:deps-externs` and never
populates or reads it, so every build that ships the engine names the
externs file in its own `:compiler-options`.
Run this against the release toolchain, never against dev, because dev
renames nothing and answers correctly either way:
clojure -M:dev:shadow-cljs release ds-probe && node target/ds-probe.js
Exits non-zero and names every reading that came back wrong."
(:require [datascript.core :as d]))
(defn- db
"One entity, two datoms, `:x/name` indexed so the `:avet` read is legal."
[]
(d/db-with (d/empty-db {:x/name {:db/index true}})
[{:db/id -1 :x/id 7 :x/name "a"}]))
(defn- readings
"Six readings paired with the answer each must give. Four of them bind
one datom slot apiece, because which slot a build corrupts is a fact
about that build's whole property set and not about datascript. The
fifth reads a slot across a join, which is the shape a real query has.
The last one reads the same datoms through the index, so it stays
correct in a corrupted build and tells a reader storage was never the
problem; it is also the oracle for the transaction id, which no
literal can name."
[db]
(let [datom (first (d/datoms db :eavt 1 :x/name))]
[[:query-value (d/q '[:find ?v :where [?e :x/name ?v]] db) #{["a"]}]
[:query-entity (d/q '[:find ?e :where [?e :x/name "a"]] db) #{[1]}]
[:query-attribute (d/q '[:find ?a :where [1 ?a "a"]] db) #{[:x/name]}]
[:query-tx (d/q '[:find ?tx :where [?e :x/name _ ?tx]] db) #{[(:tx datom)]}]
[:query-join (d/q '[:find ?e :in $ ?n :where [?e :x/name ?n]] db "a") #{[1]}]
[:index-control (mapv (juxt :e :a :v) (d/datoms db :avet :x/name "a"))
[[1 :x/name "a"]]]]))
(defn -main [& _]
(let [db (d/db-with (d/empty-db {:x/id {:db/index true}})
[{:db/id -1 :x/id 7 :x/name "a"}])]
(println :scan-q (pr-str (d/q '[:find ?e ?v :where [?e :x/name ?v]] db)))
(println :bound-q (pr-str (d/q '[:find ?e :in $ ?n :where [?e :x/name ?n]] db "a")))
(println :avet-part (pr-str (vec (d/datoms db :avet :x/name "a"))))
(println :eavt-part (pr-str (vec (d/datoms db :eavt 1 :x/name))))
(println :seek (pr-str (first (d/seek-datoms db :avet :x/name))))))
(let [rs (readings (db))
wrong (into [] (comp (remove (fn [[_ answer expected]] (= expected answer)))
(map first))
rs)]
(doseq [[label answer expected] rs]
(println label (pr-str answer)
(if (= expected answer)
"ok"
(str "WRONG, expected " (pr-str expected)))))
(if (seq wrong)
(do (println "FAIL" (pr-str wrong)
"— this build renamed the properties the query layer reads by name;"
"add :externs [\"datascript/externs.js\"] to its :compiler-options")
(js/process.exit 1))
(println "PASS — the query layer survives this build's optimizations"))))