mirror of
https://github.com/penpot/penpot.git
synced 2026-11-16 15:11:17 +00:00
📚 Clarify when to run the file_change partition procedure
The procedure straddles the upgrade to 2.20.0 and the document said nothing about the order. The preparation (phases 0 and 1) goes before the version update, the bulk and the delta passes after it, and only the cutover needs a scheduled maintenance window. The subtitle claimed the whole thing was online, which is not true. The copy marks no longer travel as a session GUC with a date typed by hand. The delta mark is seeded at the bulk with `now()`, and a CHECK keeps it from ending up empty; the instant of the cutover is recorded in the same table after the commit, so the checks of phase 5 read it instead of comparing against a date the operator retypes in every session. The procedure was run end to end on a scratch database, with the 23 SQL blocks extracted from the document and the application writes simulated between the phases. AI-assisted-by: deepseek-v4.1-flash
This commit is contained in:
parent
ec1e2c9e1a
commit
9bac1d290b
@ -1,7 +1,9 @@
|
||||
# Partition `file_change` by `file_id`
|
||||
|
||||
Online procedure to turn `file_change` into a table partitioned by
|
||||
`HASH (file_id)` with 16 partitions, aligned with the ones of `file_data`.
|
||||
Procedure to turn `file_change` into a table partitioned by `HASH (file_id)`
|
||||
with 16 partitions, aligned with the ones of `file_data`. It runs online
|
||||
except the cutover (phase 4), which needs a scheduled maintenance window with
|
||||
the backend and the workers stopped.
|
||||
|
||||
- Migration: `backend/src/app/migrations/sql/0158-partition-file-change.sql`
|
||||
- Issue: <https://github.com/penpot/penpot/issues/12416>
|
||||
@ -10,6 +12,42 @@ The result has to be **exactly** the schema of the migration, because on a big
|
||||
instance the migration is marked as applied and never runs. This procedure
|
||||
runs the same statements by hand.
|
||||
|
||||
## When to run it, and the stop
|
||||
|
||||
Run it only on the instances where the automatic migration refuses to run:
|
||||
`file_change` above 10 GiB of total size or above 2 million rows. Below that
|
||||
the migration does the same work on boot and this procedure has nothing to do.
|
||||
|
||||
The work spans the upgrade to 2.20.0. The order is not a suggestion: the
|
||||
preparation goes **before** the version update, and phases 2 and 3 need the new
|
||||
code running (it is the one that moves `updated_at` on every write, the
|
||||
watermark of the delta passes).
|
||||
|
||||
| Phase | When | Service |
|
||||
|---|---|---|
|
||||
| 0 and 1 | Before the upgrade, with 2.19.0 serving | Online |
|
||||
| — | The upgrade to 2.20.0 | Usual deploy (step 0.3), before phase 2 |
|
||||
| 2 | After the upgrade, with 2.20.0 serving | Online |
|
||||
| 3 | From phase 2 to the window, every 1-6 h | Online |
|
||||
| 4 | **A scheduled maintenance window** | **Backend and workers stopped** |
|
||||
| 5 | After the cutover | Online |
|
||||
|
||||
Phase 2 is one pass over every row: the long job, hours or days. Phase 3
|
||||
repeats a much shorter copy, only the rows that changed since the previous
|
||||
pass (seconds each), until the day of the window, so the last delta inside the
|
||||
lock stays short.
|
||||
|
||||
Steps 0.1 and 0.2 cannot wait for the upgrade: a 2.20.0 backend that boots
|
||||
without them runs the automatic migration inside the startup transaction and,
|
||||
above the guard, it fails on every boot.
|
||||
|
||||
**The procedure needs one stop, and only one: phase 4, the cutover.** It is a
|
||||
full stop-start of the backend and the workers, not a rolling restart (4.2
|
||||
explains why). The window covers the last delta, the renames and the start
|
||||
again: seconds to minutes of writes blocked on `file_change`, and the saves
|
||||
that fall inside fail and the user retries. Everything else runs with the
|
||||
instance serving traffic.
|
||||
|
||||
## Sizing your instance
|
||||
|
||||
Every number in this procedure comes from your database: run the queries
|
||||
@ -366,8 +404,18 @@ INSERT INTO file_change_copy_state (name, last_id)
|
||||
VALUES ('bulk', '00000000-0000-0000-0000-000000000000')
|
||||
ON CONFLICT (name) DO NOTHING;
|
||||
|
||||
INSERT INTO file_change_copy_state (name) VALUES ('delta')
|
||||
-- The mark of the delta passes starts at the bulk: the first pass of phase 3
|
||||
-- covers the rows changed since here, and `DO NOTHING` keeps a re-run from
|
||||
-- resetting a mark that the passes already moved.
|
||||
INSERT INTO file_change_copy_state (name, last_at) VALUES ('delta', now())
|
||||
ON CONFLICT (name) DO NOTHING;
|
||||
|
||||
-- The mark is mandatory: a pass without it copies nothing, in silence. The
|
||||
-- `DROP ... IF EXISTS` keeps this block re-runnable.
|
||||
ALTER TABLE file_change_copy_state
|
||||
DROP CONSTRAINT IF EXISTS file_change_copy_state_delta_mark,
|
||||
ADD CONSTRAINT file_change_copy_state_delta_mark
|
||||
CHECK (name <> 'delta' OR last_at IS NOT NULL);
|
||||
```
|
||||
|
||||
One batch (idempotent). It **always returns one row**: the number of rows it
|
||||
@ -458,24 +506,21 @@ written during the bulk are missing until the delta passes pick them up.
|
||||
## Phase 3 — Delta passes
|
||||
|
||||
Every pass copies the rows modified since the **previous pass**. The watermark
|
||||
lives in `file_change_copy_state.last_at` (the start of the last pass), and the
|
||||
SQL subtracts one hour of margin, which absorbs the clock drift between the
|
||||
application (which writes `updated_at`) and the database (which writes
|
||||
`now()`) and the drift between batches. A row that a pass already copied but
|
||||
did not change again is not rewritten: the `WHERE` of the `ON CONFLICT` skips
|
||||
it.
|
||||
lives in `file_change_copy_state.last_at` (the start of the last pass; the
|
||||
setup block of phase 2 seeds it with the moment of the bulk, so the first pass
|
||||
covers what changed since then), and the SQL subtracts one hour of margin,
|
||||
which absorbs the clock drift between the application (which writes
|
||||
`updated_at`) and the database (which writes `now()`) and the drift between
|
||||
batches. A row that a pass already copied but did not change again is not
|
||||
rewritten: the `WHERE` of the `ON CONFLICT` skips it.
|
||||
|
||||
```sql
|
||||
-- :bulk_start = the time the bulk started (the margin is applied below).
|
||||
-- Set it in this session before running the pass: without it the query fails
|
||||
-- instead of copying nothing and leaving those rows behind for good.
|
||||
SET penpot.bulk_start = '2026-10-25 08:00:00+02';
|
||||
|
||||
BEGIN;
|
||||
|
||||
WITH mark AS (
|
||||
SELECT COALESCE((SELECT last_at FROM file_change_copy_state WHERE name = 'delta'),
|
||||
current_setting('penpot.bulk_start')::timestamptz) - INTERVAL '1 hour' AS since
|
||||
SELECT last_at - INTERVAL '1 hour' AS since
|
||||
FROM file_change_copy_state
|
||||
WHERE name = 'delta'
|
||||
)
|
||||
INSERT INTO file_change_new (id, file_id, created_at, session_id, revn, data, changes,
|
||||
profile_id, features, label, version, created_by,
|
||||
@ -579,23 +624,23 @@ CREATE INDEX file_change_new__user_snapshots__idx
|
||||
driver) would keep writing to `file_change_legacy` after the cutover, and
|
||||
the pool recycles its connections every 30 minutes, so the window of
|
||||
silent writes is long.
|
||||
- Run the transaction.
|
||||
- Run the transaction, the `ANALYZE`, and the statement that records the
|
||||
instant of the cutover (all of it in the block below).
|
||||
- Start the backend and the workers again.
|
||||
|
||||
```sql
|
||||
SET ROLE penpot; -- the owner of file_change (see 0.0 and 0.6)
|
||||
SET penpot.bulk_start = '2026-10-25 08:00:00+02';
|
||||
|
||||
BEGIN;
|
||||
|
||||
LOCK TABLE file_change IN ACCESS EXCLUSIVE MODE;
|
||||
|
||||
-- The final delta: the rows modified since the last pass (its mark, minus
|
||||
-- the hour of margin). The mark is the one the last pass left; the fallback
|
||||
-- is the bulk start.
|
||||
-- the hour of margin). The mark is the one the last pass left.
|
||||
WITH mark AS (
|
||||
SELECT COALESCE((SELECT last_at FROM file_change_copy_state WHERE name = 'delta'),
|
||||
current_setting('penpot.bulk_start')::timestamptz) - INTERVAL '1 hour' AS since
|
||||
SELECT last_at - INTERVAL '1 hour' AS since
|
||||
FROM file_change_copy_state
|
||||
WHERE name = 'delta'
|
||||
)
|
||||
INSERT INTO file_change_new (id, file_id, created_at, session_id, revn, data, changes,
|
||||
profile_id, features, label, version, created_by,
|
||||
@ -703,6 +748,13 @@ ALTER TABLE file_change_legacy
|
||||
COMMIT;
|
||||
|
||||
ANALYZE file_change;
|
||||
|
||||
-- The instant of the cutover, for the checks of phase 5. It goes after the
|
||||
-- `COMMIT`: inside the transaction `now()` is the start of it (the moment of
|
||||
-- the `LOCK`), and the rows the final delta copies could pass for writes
|
||||
-- happened after the cutover.
|
||||
INSERT INTO file_change_copy_state (name, last_at) VALUES ('cutover', now())
|
||||
ON CONFLICT (name) DO UPDATE SET last_at = excluded.last_at;
|
||||
```
|
||||
|
||||
- `file_change_legacy` keeps its data and its indexes (with the `_legacy`
|
||||
@ -716,18 +768,15 @@ ANALYZE file_change;
|
||||
table and that nothing went to the table that is going away:
|
||||
|
||||
```sql
|
||||
-- :cutover = the time the backend went up on the new table. Set it in this
|
||||
-- session: without it the queries fail instead of comparing against a stale
|
||||
-- date.
|
||||
SET penpot.cutover = '2026-10-25 08:00:00+02';
|
||||
|
||||
-- the rows written after the cutover: this one grows with every save
|
||||
SELECT count(*) FROM file_change
|
||||
WHERE updated_at > current_setting('penpot.cutover')::timestamptz;
|
||||
WHERE updated_at > (SELECT last_at FROM file_change_copy_state
|
||||
WHERE name = 'cutover');
|
||||
|
||||
-- the rows that went to the legacy table: this one has to stay at 0
|
||||
SELECT count(*) FROM file_change_legacy
|
||||
WHERE updated_at > current_setting('penpot.cutover')::timestamptz;
|
||||
WHERE updated_at > (SELECT last_at FROM file_change_copy_state
|
||||
WHERE name = 'cutover');
|
||||
```
|
||||
|
||||
A number above zero in the second query means that some connection was
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user