📚 Clarify when to run the file_change partition procedure

The procedure straddles the upgrade to 2.20.0 and the document said
nothing about the order. The preparation (phases 0 and 1) goes before
the version update, the bulk and the delta passes after it, and only
the cutover needs a scheduled maintenance window. The subtitle claimed
the whole thing was online, which is not true.

The copy marks no longer travel as a session GUC with a date typed by
hand. The delta mark is seeded at the bulk with `now()`, and a CHECK
keeps it from ending up empty; the instant of the cutover is recorded
in the same table after the commit, so the checks of phase 5 read it
instead of comparing against a date the operator retypes in every
session.

The procedure was run end to end on a scratch database, with the 23
SQL blocks extracted from the document and the application writes
simulated between the phases.

AI-assisted-by: deepseek-v4.1-flash
This commit is contained in:
Andrey Antukh 2026-11-13 06:58:46 +00:00
parent ec1e2c9e1a
commit 9bac1d290b

View File

@ -1,7 +1,9 @@
# Partition `file_change` by `file_id`
Online procedure to turn `file_change` into a table partitioned by
`HASH (file_id)` with 16 partitions, aligned with the ones of `file_data`.
Procedure to turn `file_change` into a table partitioned by `HASH (file_id)`
with 16 partitions, aligned with the ones of `file_data`. It runs online
except the cutover (phase 4), which needs a scheduled maintenance window with
the backend and the workers stopped.
- Migration: `backend/src/app/migrations/sql/0158-partition-file-change.sql`
- Issue: <https://github.com/penpot/penpot/issues/12416>
@ -10,6 +12,42 @@ The result has to be **exactly** the schema of the migration, because on a big
instance the migration is marked as applied and never runs. This procedure
runs the same statements by hand.
## When to run it, and the stop
Run it only on the instances where the automatic migration refuses to run:
`file_change` above 10 GiB of total size or above 2 million rows. Below that
the migration does the same work on boot and this procedure has nothing to do.
The work spans the upgrade to 2.20.0. The order is not a suggestion: the
preparation goes **before** the version update, and phases 2 and 3 need the new
code running (it is the one that moves `updated_at` on every write, the
watermark of the delta passes).
| Phase | When | Service |
|---|---|---|
| 0 and 1 | Before the upgrade, with 2.19.0 serving | Online |
| — | The upgrade to 2.20.0 | Usual deploy (step 0.3), before phase 2 |
| 2 | After the upgrade, with 2.20.0 serving | Online |
| 3 | From phase 2 to the window, every 1-6 h | Online |
| 4 | **A scheduled maintenance window** | **Backend and workers stopped** |
| 5 | After the cutover | Online |
Phase 2 is one pass over every row: the long job, hours or days. Phase 3
repeats a much shorter copy, only the rows that changed since the previous
pass (seconds each), until the day of the window, so the last delta inside the
lock stays short.
Steps 0.1 and 0.2 cannot wait for the upgrade: a 2.20.0 backend that boots
without them runs the automatic migration inside the startup transaction and,
above the guard, it fails on every boot.
**The procedure needs one stop, and only one: phase 4, the cutover.** It is a
full stop-start of the backend and the workers, not a rolling restart (4.2
explains why). The window covers the last delta, the renames and the start
again: seconds to minutes of writes blocked on `file_change`, and the saves
that fall inside fail and the user retries. Everything else runs with the
instance serving traffic.
## Sizing your instance
Every number in this procedure comes from your database: run the queries
@ -366,8 +404,18 @@ INSERT INTO file_change_copy_state (name, last_id)
VALUES ('bulk', '00000000-0000-0000-0000-000000000000')
ON CONFLICT (name) DO NOTHING;
INSERT INTO file_change_copy_state (name) VALUES ('delta')
-- The mark of the delta passes starts at the bulk: the first pass of phase 3
-- covers the rows changed since here, and `DO NOTHING` keeps a re-run from
-- resetting a mark that the passes already moved.
INSERT INTO file_change_copy_state (name, last_at) VALUES ('delta', now())
ON CONFLICT (name) DO NOTHING;
-- The mark is mandatory: a pass without it copies nothing, in silence. The
-- `DROP ... IF EXISTS` keeps this block re-runnable.
ALTER TABLE file_change_copy_state
DROP CONSTRAINT IF EXISTS file_change_copy_state_delta_mark,
ADD CONSTRAINT file_change_copy_state_delta_mark
CHECK (name <> 'delta' OR last_at IS NOT NULL);
```
One batch (idempotent). It **always returns one row**: the number of rows it
@ -458,24 +506,21 @@ written during the bulk are missing until the delta passes pick them up.
## Phase 3 — Delta passes
Every pass copies the rows modified since the **previous pass**. The watermark
lives in `file_change_copy_state.last_at` (the start of the last pass), and the
SQL subtracts one hour of margin, which absorbs the clock drift between the
application (which writes `updated_at`) and the database (which writes
`now()`) and the drift between batches. A row that a pass already copied but
did not change again is not rewritten: the `WHERE` of the `ON CONFLICT` skips
it.
lives in `file_change_copy_state.last_at` (the start of the last pass; the
setup block of phase 2 seeds it with the moment of the bulk, so the first pass
covers what changed since then), and the SQL subtracts one hour of margin,
which absorbs the clock drift between the application (which writes
`updated_at`) and the database (which writes `now()`) and the drift between
batches. A row that a pass already copied but did not change again is not
rewritten: the `WHERE` of the `ON CONFLICT` skips it.
```sql
-- :bulk_start = the time the bulk started (the margin is applied below).
-- Set it in this session before running the pass: without it the query fails
-- instead of copying nothing and leaving those rows behind for good.
SET penpot.bulk_start = '2026-10-25 08:00:00+02';
BEGIN;
WITH mark AS (
SELECT COALESCE((SELECT last_at FROM file_change_copy_state WHERE name = 'delta'),
current_setting('penpot.bulk_start')::timestamptz) - INTERVAL '1 hour' AS since
SELECT last_at - INTERVAL '1 hour' AS since
FROM file_change_copy_state
WHERE name = 'delta'
)
INSERT INTO file_change_new (id, file_id, created_at, session_id, revn, data, changes,
profile_id, features, label, version, created_by,
@ -579,23 +624,23 @@ CREATE INDEX file_change_new__user_snapshots__idx
driver) would keep writing to `file_change_legacy` after the cutover, and
the pool recycles its connections every 30 minutes, so the window of
silent writes is long.
- Run the transaction.
- Run the transaction, the `ANALYZE`, and the statement that records the
instant of the cutover (all of it in the block below).
- Start the backend and the workers again.
```sql
SET ROLE penpot; -- the owner of file_change (see 0.0 and 0.6)
SET penpot.bulk_start = '2026-10-25 08:00:00+02';
BEGIN;
LOCK TABLE file_change IN ACCESS EXCLUSIVE MODE;
-- The final delta: the rows modified since the last pass (its mark, minus
-- the hour of margin). The mark is the one the last pass left; the fallback
-- is the bulk start.
-- the hour of margin). The mark is the one the last pass left.
WITH mark AS (
SELECT COALESCE((SELECT last_at FROM file_change_copy_state WHERE name = 'delta'),
current_setting('penpot.bulk_start')::timestamptz) - INTERVAL '1 hour' AS since
SELECT last_at - INTERVAL '1 hour' AS since
FROM file_change_copy_state
WHERE name = 'delta'
)
INSERT INTO file_change_new (id, file_id, created_at, session_id, revn, data, changes,
profile_id, features, label, version, created_by,
@ -703,6 +748,13 @@ ALTER TABLE file_change_legacy
COMMIT;
ANALYZE file_change;
-- The instant of the cutover, for the checks of phase 5. It goes after the
-- `COMMIT`: inside the transaction `now()` is the start of it (the moment of
-- the `LOCK`), and the rows the final delta copies could pass for writes
-- happened after the cutover.
INSERT INTO file_change_copy_state (name, last_at) VALUES ('cutover', now())
ON CONFLICT (name) DO UPDATE SET last_at = excluded.last_at;
```
- `file_change_legacy` keeps its data and its indexes (with the `_legacy`
@ -716,18 +768,15 @@ ANALYZE file_change;
table and that nothing went to the table that is going away:
```sql
-- :cutover = the time the backend went up on the new table. Set it in this
-- session: without it the queries fail instead of comparing against a stale
-- date.
SET penpot.cutover = '2026-10-25 08:00:00+02';
-- the rows written after the cutover: this one grows with every save
SELECT count(*) FROM file_change
WHERE updated_at > current_setting('penpot.cutover')::timestamptz;
WHERE updated_at > (SELECT last_at FROM file_change_copy_state
WHERE name = 'cutover');
-- the rows that went to the legacy table: this one has to stay at 0
SELECT count(*) FROM file_change_legacy
WHERE updated_at > current_setting('penpot.cutover')::timestamptz;
WHERE updated_at > (SELECT last_at FROM file_change_copy_state
WHERE name = 'cutover');
```
A number above zero in the second query means that some connection was