diff --git a/operations/2.20.0-file-change-partition.md b/operations/2.20.0-file-change-partition.md index b74e589b56..3a9aabd51b 100644 --- a/operations/2.20.0-file-change-partition.md +++ b/operations/2.20.0-file-change-partition.md @@ -1,7 +1,9 @@ # Partition `file_change` by `file_id` -Online procedure to turn `file_change` into a table partitioned by -`HASH (file_id)` with 16 partitions, aligned with the ones of `file_data`. +Procedure to turn `file_change` into a table partitioned by `HASH (file_id)` +with 16 partitions, aligned with the ones of `file_data`. It runs online +except the cutover (phase 4), which needs a scheduled maintenance window with +the backend and the workers stopped. - Migration: `backend/src/app/migrations/sql/0158-partition-file-change.sql` - Issue: @@ -10,6 +12,42 @@ The result has to be **exactly** the schema of the migration, because on a big instance the migration is marked as applied and never runs. This procedure runs the same statements by hand. +## When to run it, and the stop + +Run it only on the instances where the automatic migration refuses to run: +`file_change` above 10 GiB of total size or above 2 million rows. Below that +the migration does the same work on boot and this procedure has nothing to do. + +The work spans the upgrade to 2.20.0. The order is not a suggestion: the +preparation goes **before** the version update, and phases 2 and 3 need the new +code running (it is the one that moves `updated_at` on every write, the +watermark of the delta passes). + +| Phase | When | Service | +|---|---|---| +| 0 and 1 | Before the upgrade, with 2.19.0 serving | Online | +| — | The upgrade to 2.20.0 | Usual deploy (step 0.3), before phase 2 | +| 2 | After the upgrade, with 2.20.0 serving | Online | +| 3 | From phase 2 to the window, every 1-6 h | Online | +| 4 | **A scheduled maintenance window** | **Backend and workers stopped** | +| 5 | After the cutover | Online | + +Phase 2 is one pass over every row: the long job, hours or days. Phase 3 +repeats a much shorter copy, only the rows that changed since the previous +pass (seconds each), until the day of the window, so the last delta inside the +lock stays short. + +Steps 0.1 and 0.2 cannot wait for the upgrade: a 2.20.0 backend that boots +without them runs the automatic migration inside the startup transaction and, +above the guard, it fails on every boot. + +**The procedure needs one stop, and only one: phase 4, the cutover.** It is a +full stop-start of the backend and the workers, not a rolling restart (4.2 +explains why). The window covers the last delta, the renames and the start +again: seconds to minutes of writes blocked on `file_change`, and the saves +that fall inside fail and the user retries. Everything else runs with the +instance serving traffic. + ## Sizing your instance Every number in this procedure comes from your database: run the queries @@ -366,8 +404,18 @@ INSERT INTO file_change_copy_state (name, last_id) VALUES ('bulk', '00000000-0000-0000-0000-000000000000') ON CONFLICT (name) DO NOTHING; -INSERT INTO file_change_copy_state (name) VALUES ('delta') +-- The mark of the delta passes starts at the bulk: the first pass of phase 3 +-- covers the rows changed since here, and `DO NOTHING` keeps a re-run from +-- resetting a mark that the passes already moved. +INSERT INTO file_change_copy_state (name, last_at) VALUES ('delta', now()) ON CONFLICT (name) DO NOTHING; + +-- The mark is mandatory: a pass without it copies nothing, in silence. The +-- `DROP ... IF EXISTS` keeps this block re-runnable. +ALTER TABLE file_change_copy_state + DROP CONSTRAINT IF EXISTS file_change_copy_state_delta_mark, + ADD CONSTRAINT file_change_copy_state_delta_mark + CHECK (name <> 'delta' OR last_at IS NOT NULL); ``` One batch (idempotent). It **always returns one row**: the number of rows it @@ -458,24 +506,21 @@ written during the bulk are missing until the delta passes pick them up. ## Phase 3 — Delta passes Every pass copies the rows modified since the **previous pass**. The watermark -lives in `file_change_copy_state.last_at` (the start of the last pass), and the -SQL subtracts one hour of margin, which absorbs the clock drift between the -application (which writes `updated_at`) and the database (which writes -`now()`) and the drift between batches. A row that a pass already copied but -did not change again is not rewritten: the `WHERE` of the `ON CONFLICT` skips -it. +lives in `file_change_copy_state.last_at` (the start of the last pass; the +setup block of phase 2 seeds it with the moment of the bulk, so the first pass +covers what changed since then), and the SQL subtracts one hour of margin, +which absorbs the clock drift between the application (which writes +`updated_at`) and the database (which writes `now()`) and the drift between +batches. A row that a pass already copied but did not change again is not +rewritten: the `WHERE` of the `ON CONFLICT` skips it. ```sql --- :bulk_start = the time the bulk started (the margin is applied below). --- Set it in this session before running the pass: without it the query fails --- instead of copying nothing and leaving those rows behind for good. -SET penpot.bulk_start = '2026-10-25 08:00:00+02'; - BEGIN; WITH mark AS ( - SELECT COALESCE((SELECT last_at FROM file_change_copy_state WHERE name = 'delta'), - current_setting('penpot.bulk_start')::timestamptz) - INTERVAL '1 hour' AS since + SELECT last_at - INTERVAL '1 hour' AS since + FROM file_change_copy_state + WHERE name = 'delta' ) INSERT INTO file_change_new (id, file_id, created_at, session_id, revn, data, changes, profile_id, features, label, version, created_by, @@ -579,23 +624,23 @@ CREATE INDEX file_change_new__user_snapshots__idx driver) would keep writing to `file_change_legacy` after the cutover, and the pool recycles its connections every 30 minutes, so the window of silent writes is long. -- Run the transaction. +- Run the transaction, the `ANALYZE`, and the statement that records the + instant of the cutover (all of it in the block below). - Start the backend and the workers again. ```sql SET ROLE penpot; -- the owner of file_change (see 0.0 and 0.6) -SET penpot.bulk_start = '2026-10-25 08:00:00+02'; BEGIN; LOCK TABLE file_change IN ACCESS EXCLUSIVE MODE; -- The final delta: the rows modified since the last pass (its mark, minus --- the hour of margin). The mark is the one the last pass left; the fallback --- is the bulk start. +-- the hour of margin). The mark is the one the last pass left. WITH mark AS ( - SELECT COALESCE((SELECT last_at FROM file_change_copy_state WHERE name = 'delta'), - current_setting('penpot.bulk_start')::timestamptz) - INTERVAL '1 hour' AS since + SELECT last_at - INTERVAL '1 hour' AS since + FROM file_change_copy_state + WHERE name = 'delta' ) INSERT INTO file_change_new (id, file_id, created_at, session_id, revn, data, changes, profile_id, features, label, version, created_by, @@ -703,6 +748,13 @@ ALTER TABLE file_change_legacy COMMIT; ANALYZE file_change; + +-- The instant of the cutover, for the checks of phase 5. It goes after the +-- `COMMIT`: inside the transaction `now()` is the start of it (the moment of +-- the `LOCK`), and the rows the final delta copies could pass for writes +-- happened after the cutover. +INSERT INTO file_change_copy_state (name, last_at) VALUES ('cutover', now()) +ON CONFLICT (name) DO UPDATE SET last_at = excluded.last_at; ``` - `file_change_legacy` keeps its data and its indexes (with the `_legacy` @@ -716,18 +768,15 @@ ANALYZE file_change; table and that nothing went to the table that is going away: ```sql --- :cutover = the time the backend went up on the new table. Set it in this --- session: without it the queries fail instead of comparing against a stale --- date. -SET penpot.cutover = '2026-10-25 08:00:00+02'; - -- the rows written after the cutover: this one grows with every save SELECT count(*) FROM file_change - WHERE updated_at > current_setting('penpot.cutover')::timestamptz; + WHERE updated_at > (SELECT last_at FROM file_change_copy_state + WHERE name = 'cutover'); -- the rows that went to the legacy table: this one has to stay at 0 SELECT count(*) FROM file_change_legacy - WHERE updated_at > current_setting('penpot.cutover')::timestamptz; + WHERE updated_at > (SELECT last_at FROM file_change_copy_state + WHERE name = 'cutover'); ``` A number above zero in the second query means that some connection was