2 Commits

Author	SHA1	Message	Date
Andrey Antukh	dd4e768707	✨ Partition file_change by file_id ... `file_change` is the biggest relation of the database: 683 GB in production, of which almost all the TOAST is dead space (the legacy data is already migrated to `file_data`; what is left is the xlog of the revisions, 4.4 kB on average). It grows with every save of a file and every version, and its rows are spread over the whole table, so every query that filters by a file has to walk all of it. The table is now partitioned by `HASH (file_id)` with 16 partitions, the same bounds as `file_data`, so the queries that filter by a file can prune to one partition and the join between the two tables is resolved partition by partition. The primary key has to include the partition key, so it becomes `(file_id, id)`, and every access path of the table filters by `file_id` accordingly: the snapshot commands require it (2.19), the frontend sends it, and the garbage collector locks the pending rows by their `(file_id, id)` pair. The migration creates the partitioned table with the final schema and copies the rows, keeping the old one aside as `file_change_legacy` for the rollback. The copy happens inside the transaction of the migration, so it does not fit a big instance: there the migration is marked as applied and the procedure of `operations/` does it by hand, in phases, driven by the `updated_at` watermark. Along the way: - every mutation of the table moves `updated_at` to the moment of the mutation, so the phased copy does not lose the rows changed while it runs. It is never set to the `deleted_at` of the row, which for the system snapshots is a date in the future. - a new `kind` column tells the snapshots (1) and the revisions (2) of the table apart, and the paths that insert in it fill it, including the repair path. It has no default on purpose, so an insert that forgets it fails instead of storing the wrong kind silently. It is a reserve: the queries still tell a snapshot from a revision by its label, and no constraint ties the two together. - the revision does not repeat the features of the file anymore: they were 348 bytes written on every save and nobody read them from there. - the `data_ref_id` and `data_backend` columns are gone, dead since the file storage refactor. The `data` column is kept: the `legacy-db` file data backend is still supported (migration 0157 moved the existing blobs, but the backend keeps reading and writing it). - the legacy blobs of the snapshots were already moved to `file_data` by migration 0157 (on the big instances, by hand, with its runbook), so there is nothing to copy here: the `data` column travels with the row. - the foreign key of the legacy table on `file_id` goes away with the copy: it is `NO ACTION`, so while the legacy table has rows, the objects GC cannot delete the files they point to. - the quota of the snapshots of a team resolves the files of the team first and counts their versions with `= ANY`, instead of joining the team with every version of the table. - the pool sets `enable_partitionwise_join` on every connection, so the join between the two partitioned tables is resolved partition by partition. - the snapshot lookup reports an unknown snapshot with its own error instead of the generic "object not found" of the database helper, and the decode script fails instead of printing an empty JSON. - the procedure of `operations/` stops the backend and the workers during the cutover: the pool keeps server-side prepared statements, and a prepared statement holds the OID of the table it was prepared with, so a connection would keep writing to the legacy table after the rename. - the procedure creates the partitions with their final name, so the result is the schema a new installation gets, and its verification checks the names and not only how many there are. - the index of the system snapshots is gone: the old one filtered them by `data IS NOT NULL`, and with the payload living in `file_data` the column no longer tells snapshots apart. - the configuration guide, the memories and the docstring of the decode script keep offering the `legacy-db` backend, which is still supported. - the tests fix the schema the migration leaves: the 16 partitions with their name, the primary key, the foreign keys, the check of the kind, the six indexes, the kept `data` column, and that an insert without a kind or with an unknown one fails. New tests cover the kind of both kinds of row and the resolution of the `legacy-db` blob. Closes #12416 AI-assisted-by: glm-5.3-flash AI-assisted-by: minimax-m3.1-flash AI-assisted-by: muse-spark-1.3-contributor	2026-11-03 14:21:01 +01:00
Andrey Antukh	637843e63f	📚 Add 2.19.0 operations runbook for legacy file_data migration ... Manual pre-migration procedure for big instances, same statements as migration 0157 in batched transactions, following the convention of the file-change partition runbook. Closes #12784. AI-assisted-by: muse-spark-1.3-contributor	2026-11-03 13:33:22 +01:00