penpot/backend/src/app/scripts/decode_file_data.clj
Andrey Antukh 34c702bc80
✨ Add table partitioning for file-change (#12417)
* ✨ Partition file_change by file_id

Partition `file_change` by `HASH (file_id)` in 16 partitions, with the
same bounds as `file_data`, so a join between them is resolved partition
by partition. The primary key becomes `(file_id, id)`, which a
partitioned table needs, and every access filters by `file_id`.

The new table carries a `kind` column (1 snapshot, 2 revision) as a
reserve: the queries still tell them apart by the label, and `kind`
takes over when they are rewritten. The payload of a snapshot stays in
`file_data`; the `data` column is kept for reading old `legacy-db`
payloads.

The four snapshot commands require `file-id`, because it prunes the
partition. The objects GC locks and deletes by the pair, in batches. The
old table is kept as `file_change_legacy` as the way back.

The automatic migration refuses to run above 10 GiB or 2M rows and
points to the runbook; big instances partition online with that runbook,
which copies by id, follows the `updated_at` watermark in delta passes
and checks that no row of the source is missing from the target.

Closes #12416

AI-assisted-by: deepseek-v4.1-flash

* 🐛 Address the review of the file_change partition

The production runbook promised a rollback it did not have: the section
was a sketch, and its statements could write to the table about to be
dropped. The rollback is removed, and so is every comment that described
it. `file_change_legacy` stays as a frozen copy of the previous layout.

The cutover check now aborts the transaction when a row of the old table
is missing in the new one, instead of leaving the decision to the
operator.

The five secondary indexes of the new table are built after the last
delta pass: the bulk and the delta passes only need the primary key, so
the batches no longer maintain them.

The copy of 0158 is now replayed in a test against the pre-0158 shape
with a labelled and an unlabelled row, plus an empty instance that must
leave no legacy table behind. The updated_at assertion of the snapshot
mutations is bounded by the test clock, and the repair revision stops
writing the `features` of the file, like the normal save.

AI-assisted-by: deepseek-v4.1-flash

* 📎 Fix linter issues

* 🐛 Fix stale helper call in rpc_file_snapshot_test

The suite does not load: `th/run-pending-tasks!` does not exist in
backend-tests.helpers. The pending jobs runner is `run-pending-jobs`
(no arguments), the name every other call site of the suite uses.

AI-assisted-by: inkling-2-small-free

* 📚 Align the partition change with the 2.20.0 milestone

2.19.0 is already shipped; this work targets 2.20.0. The operations
file of the partition procedure carries the 2.20.0 prefix, the
references that point at it (the migration, the runbook table and the
memory) follow, and the RPC doc/changes tags of the snapshot commands
record 2.20 as the release that makes `file-id` required.

AI-assisted-by: inkling-2-small-free

* 🐛 Fix review findings on the partition rebase

The 0157 legacy test was missed by the rebase and inserted file_change
rows without the new mandatory `kind` column, so the backend suite fails
against the migrated schema. The inserts now carry `:kind`
(snapshot-kind for the labelled rows, revision-kind for the rest).

The post-cutover verification block of the runbook terminates the
first query with a semicolon so the block pastes into psql without a
syntax error.

The guard message of the migration names the step
(0158-partition-file-change) instead of "0158", ambiguous since two
migrations share that number.

Closes #12416

AI-assisted-by: inkling-2-small-free

* 🐛 Fix the stale label in the snapshot restore test

The rename block of snapshots-crud runs before the restore block,
so the restored version is already named "label1-updated" when the
restore publishes its message. The command names the version by the
label the row has at that moment, not by the one it had when it was
created, so the test was asserting a stale value.

Assert the current label, and say why in the comment.

AI-assisted-by: deepseek-v4.1-flash

* ♻️ Split the version event tests into one test per command

The workspace version events and the plugin proxy send the file-id
of the open file on every snapshot command, because the command
needs it to prune the partition of file_change. Both tests checked
that with a doseq and t/testing blocks, so a failure named the whole
group and not the command that broke.

Give each command its own test. The workspace ones share a helper
that runs the event and returns the command and params it sends; the
plugin ones keep their own mock of rp/cmd!.

AI-assisted-by: deepseek-v4.1-flash
2026-11-10 23:27:17 +01:00

309 lines
10 KiB
Clojure

;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
(ns app.scripts.decode-file-data
"Decode a file's `file_data` binary blob to `jq`-readable JSON.
The content of `file_data.data` (and `file.data` on the `legacy-db`
backend, or the `file-data` object on storage) is not text: it is a blob
encoded with `app.util.blob` (versions 1, 3, 4 and 5; transit+json or
fressian plus lz4/zstd). That is why it cannot be inspected with SQL
directly.
Two uses:
1. Offline (no DB, just the blob): dump the `bytea` to a file with
`psql` (`\\copy ...` or `encode(data, 'base64')`) and pass it with `--input`.
Needs no system and no `file_data` read permissions.
backend/scripts/decode-file-data --input /tmp/blob.bin --format summary | jq .
backend/scripts/decode-file-data --input /tmp/blob.b64 --from-base64 --format pages | jq .
2. Inside the penpot context (nREPL with the backend running, where
`sys/system` already exists): call `get-file-json` or `get-snapshot-json`.
They resolve the backend (`legacy-db`, `db`, `storage`), realize the
pointer-maps and objects-maps and return JSON.
(require '[app.scripts.decode-file-data :as dfd] :reload)
(spit \"/tmp/maquette.json\"
(dfd/get-file-json sys/system #uuid \"9e056cfe-f83f-81f8-8008-cc885732f00d\"
:format :full))
(spit \"/tmp/snap-518.json\"
(dfd/get-snapshot-json sys/system #uuid \"9e056cfe-f83f-81f8-8008-cc885732f00d\"
#uuid \"<snapshot-id>\" :format :summary))
Then filter with `jq`: `.pages`, `.data[\"pages-index\"] | keys`,
`[.data[\"pages-index\"][] | {id, name}]`, etc."
(:require
[app.binfile.common :as bfc]
[app.common.exceptions :as ex]
[app.common.json :as json]
[app.db :as db]
[app.features.file-snapshots :as fsnap]
[app.util.blob :as blob]
[app.util.objects-map :as omap-legacy]
[app.util.pointer-map :as pmap]
[clojure.java.io :as io]
[clojure.string :as str])
(:import
java.util.Base64))
(set! *warn-on-reflection* true)
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; JSON sanitization
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defn- sanitize-key
[k]
(cond
(keyword? k) (subs (str k) 1)
(symbol? k) (str k)
(uuid? k) (str k)
:else (str k)))
(declare sanitize)
(defn- sanitize-map
[m]
(persistent!
(reduce-kv (fn [acc k v]
(assoc! acc (sanitize-key k) (sanitize v)))
(transient {})
m)))
(defn sanitize
"Convert a blob-decoded value to plain JSON-ready data.
- Unrealized pointers become `{\"$pointer\": \"<uuid>\"}`.
- `objects-map` instances are materialized to a plain map.
- Bytes become `{\"$bytes\": \"<base64>\"}`.
- UUIDs, instants, keywords and symbols become strings."
[x]
(cond
(nil? x) nil
(string? x) x
(boolean? x) x
(number? x)
(if (ratio? x) (double x) x)
(keyword? x) (subs (str x) 1)
(symbol? x) (str x)
(uuid? x) (str x)
(inst? x) (str x)
(bytes? x)
{"$bytes" (.encodeToString (Base64/getEncoder) ^bytes x)}
(pmap/pointer-map? x)
{"$pointer" (str (pmap/get-id x))}
(omap-legacy/objects-map? x)
(sanitize (into {} x))
(map? x)
(sanitize-map x)
(set? x)
(mapv sanitize x)
(coll? x)
(mapv sanitize x)
:else
(str x)))
(defn data->json
"Encode sanitizable data to a JSON string."
[data]
(json/encode (sanitize data)))
(defn blob->json
"Decode a `file_data` blob (`app.util.blob`) and return it as JSON."
[^bytes blob-bytes]
(data->json (blob/decode blob-bytes)))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; jq-oriented investigation summary
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defn summarize-data
"Small, stable summary for `jq`: pages (in `:pages` order) with object
counts, plus component, color and typography counters. Takes the already
decoded `:data` (not the JSON)."
[data]
(let [pages (:pages data)
pages-index (:pages-index data)
page-entry (fn [pid]
{:id (str pid)
:name (get-in pages-index [pid :name])
:objects (count (get-in pages-index [pid :objects]))})]
{:pages (mapv page-entry (or pages []))
:stats {:pages (count (or pages []))
:components (count (:components data))
:colors (count (:colors data))
:typographies (count (:typographies data))}}))
(defn file-data-summary-json
[data]
(data->json (summarize-data data)))
(defn file-data-pages-json
[data]
(data->json (:pages (summarize-data data))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; DB access (for use from nREPL with the backend running)
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defn- write-output
[^String payload output]
(if output
(do (spit output payload) output)
(print payload))
payload)
(defn get-file-json
"Return a file as JSON (realized). For nREPL.
`format` can be `:full` (whole file), `:data` (`:data` only),
`:summary` or `:pages`. With `:output` it also writes it to a file."
[system file-id & {:keys [format output] :or {format :full}}]
(let [file (db/run! system bfc/get-file file-id :realize? true)
data (case format
:full file
:data (:data file)
:summary (summarize-data (:data file))
:pages (:pages (summarize-data (:data file))))]
(write-output (data->json data) output)))
(defn get-snapshot-json
"Return a snapshot as JSON (realized). For nREPL.
Takes the same `format` and `output` as `get-file-json`. It raises when the
file has no snapshot with that id."
[system file-id snapshot-id & {:keys [format output] :or {format :full}}]
(let [snapshot (db/run! system
(fn [cfg]
(fsnap/get-snapshot cfg file-id snapshot-id)))
data (if snapshot
(case format
:full snapshot
:data (:data snapshot)
:summary (summarize-data (:data snapshot))
:pages (:pages (summarize-data (:data snapshot))))
(ex/raise :type :not-found
:code :snapshot-not-found
:hint "unable to find snapshot with the provided id"
:snapshot-id snapshot-id
:file-id file-id))]
(write-output (data->json data) output)))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Offline CLI: decode a file holding the blob
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(def ^:private cli-usage
(str "Usage: decode-file-data --input PATH [--output PATH] [--format full|data|summary|pages]\n"
" [--from-base64|--from-hex]\n"
"\n"
"Decode a file_data blob (app.util.blob) to JSON for inspection with jq.\n"
"\n"
"Examples:\n"
" psql -c \"\\\\copy (select data from file_data where file_id='...' and id='...') to '/tmp/blob.bin' with binary\"\n"
" decode-file-data --input /tmp/blob.bin --format summary | jq .\n"
" decode-file-data --input /tmp/blob.b64 --from-base64 --format pages | jq .\n"))
(defn- parse-args
[args]
(loop [args args
opts {:format "full"}]
(if (empty? args)
opts
(let [[a b & rest-args] args]
(cond
(contains? #{"-h" "--help"} a)
(assoc opts :help true)
(= a "--input")
(recur rest-args (assoc opts :input b))
(= a "--output")
(recur rest-args (assoc opts :output b))
(= a "--format")
(recur rest-args (assoc opts :format b))
(= a "--from-base64")
(recur (cons b rest-args) (assoc opts :from :base64))
(= a "--from-hex")
(recur (cons b rest-args) (assoc opts :from :hex))
:else
(assoc opts :error (str "unknown argument: " a)))))))
(defn- hex->bytes
[^String s]
(let [s (str/trim s)
s (cond
(str/starts-with? s "\\x") (subs s 2)
(str/starts-with? s "0x") (subs s 2)
:else s)
s (str/replace s #"\s+" "")
n (/ (count s) 2)
out (byte-array n)]
(dotimes [i n]
(aset-byte out i (unchecked-byte (Integer/parseInt (subs s (* i 2) (+ (* i 2) 2)) 16))))
out))
(defn- read-input-bytes
[{:keys [input from]}]
(cond
(nil? input)
(throw (ex-info "missing --input" {}))
(= from :base64)
(.decode (Base64/getDecoder) ^String (str/trim (slurp (io/file input))))
(= from :hex)
(hex->bytes (slurp (io/file input)))
:else
(java.nio.file.Files/readAllBytes (.toPath (io/file input)))))
(defn -main
[& args]
(let [{:keys [help error input format output] :as opts} (parse-args (vec args))]
(cond
help
(do (print cli-usage) nil)
error
(do (binding [*out* *err*] (println error) (println cli-usage)) (System/exit 2))
(nil? input)
(do (binding [*out* *err*] (println "missing --input") (println cli-usage)) (System/exit 2))
(not (contains? #{"full" "data" "summary" "pages"} (or format "full")))
(do (binding [*out* *err*] (println (str "unknown format: " format))) (System/exit 2))
:else
(let [blob-bytes (read-input-bytes opts)
decoded (blob/decode blob-bytes)
payload (case format
"full" decoded
"data" (:data decoded (:data decoded decoded))
"summary" (summarize-data (:data decoded decoded))
"pages" (:pages (summarize-data (:data decoded decoded))))
;; NOTE: the file_data blob already IS the file's :data; but if
;; someone dumps the whole serialized row, tolerate both.
out (data->json payload)]
(if output
(spit output out)
(println out))))))