penpot/operations/2.20.0-consumer-generalization.md
Andrey Antukh 90a06e982b
♻️ Rework the audit-log consumers and the jobs substrate (#12701)
* ♻️ Rework audit-log consumers and jobs substrate

Fold the implementation, follow-up fixes, migrations and tests into one

code commit.

AI-assisted-by: gpt-6-luna

* 📚 Consolidate 2.20.0 upgrade runbooks

Keep the operational upgrade procedure and its supporting runbooks in a

separate documentation commit.

AI-assisted-by: gpt-6-luna

* ♻️ Remove the audit-log flag and derive persistence at boot

Persistence now follows the consumer flags instead of a separate switch.
app.config computes cf/audit-persist? once at boot from :nexus and
:admin-console, and every write site reads that value, so request paths
no longer check flags per event. With no consumer enabled no full rows
are written; telemetry shadows and webhook deliveries keep their own
flags. The frontend wire signal stays :audit-log as a derived value.

Update the 2.20 runbooks, memories and the devenv flags to match, and
migrate the tests that used to toggle :audit-log directly.

AI-assisted-by: hy4-flash-free

* 🐛 Accept error-reporting batches and precompute sink flags

push-audit-events now accepts crash batches with :error-reporting alone, so
the frontend signal from get-environment-data matches a real destination.
The reporters receive unhandled-exception events without storing any row.

All per-event sink checks read booleans computed once at boot: webhooks,
telemetry and error reporting join audit-persist? in app.config. The test
fixture and with-audit-config rebind all four values together, and the
tests that toggled those flags directly were migrated.

Address F2 of the review of the previous commit. F1 stays deferred to a
separate plan.

AI-assisted-by: hy4-flash-free

* 💄 Match the layout cljfmt expects in the flags set

The first comment of the varia set sat on its own line after the opening
delimiter, and cljfmt joins such a comment to the delimiter line, so the
Common Tests lint step failed on common/src/app/common/flags.cljc. Keep
the comment on the opening line, the same shape the login and email sets
already use.

Note in mem:scripts/ci that the backend lint lints ../common/src but does
not format-check it, so common has to be passed to scripts/ci as well.

AI-assisted-by: deepseek-v4-flash
2026-11-20 11:12:00 +01:00

20 KiB

Consumer generalization: consumer handover and out-of-band migration

Procedure to run before upgrading to the release that ships the audit-log cursor consumers and the jobs substrate. Steps 1-2 (the consumer handover) apply to every instance that ran the legacy archiver or its GC; the out-of-band DDL is for a large instance.

When you need this

Run the consumer handover before upgrading to the release that ships the audit-log cursor consumers and the jobs substrate, while the current version is still serving traffic. The web instances keep writing for the whole procedure (step 4 relies on it); only the legacy archiver and its GC stop (step 1).

Steps 1 and 2 are mandatory whenever the legacy archiver or its GC was enabled in 2.19, even when audit_log is small: the new consumers start at (0, 0), so the rows the old archiver already archived would be replayed and the old GC could delete rows the new consumers have not read. Only the out-of-band DDL is optional by size.

You need the out-of-band DDL when audit_log is large. The migration runner executes every pending step in a single transaction: the plain index builds hold a SHARE lock for the whole build and every ALTER keeps its ACCESS EXCLUSIVE until that one commit, so on a large table the upgrade stalls writes for minutes. Pre-applying the same statements out of band and in this order (outside a transaction block, except the jobs step, which runs as one explicit BEGIN/COMMIT), plus registering the steps first (step 0), makes the runner skip the group, so the deploy takes no such locks. On a small table the migration runs fast enough on its own: skip the DDL steps, not the handover. This procedure covers 0159-consumer-generalization and the 0154-add-job-table step it depends on.

Pre-applying does not make the migration free by itself: the guards (IF NOT EXISTS, seq IS NULL) skip the work, not the locks. Registering the step is what makes the runner skip the group.

The application keeps writing to audit_log for the whole procedure. Do not stop the web instances. The DDL takes a brief ACCESS EXCLUSIVE (metadata-only) that lock_timeout bounds; the index builds use CONCURRENTLY; the backfill takes row locks only. Stopping the archiver and its GC is fine; stopping the writes is not.

While the new version runs, the consumers read only rows below pg_snapshot_xmin(pg_current_snapshot()), the oldest transaction of the whole database: a long write transaction (a binfile import runs in one for up to 30 minutes) pauses archiving and the GC until it commits, and a slow destination does too, because a chunk sends inside its transaction and holds the cut back until it ends. Nothing is lost and the lag gauge shows it; it is not a stuck consumer.

Session settings

Run every block below in a session with these two settings. They are plain SQL (no psql meta-commands), so any client accepts them, and they last for the session: if you paste a block into a new session, run them first.

SET statement_timeout = 0;
SET lock_timeout = '5s';

statement_timeout = 0 keeps a client's default from killing the backfill or the CONCURRENTLY builds. lock_timeout = '5s' makes an ALTER that waits behind a long transaction fail and be retried, instead of queueing in front of every later query on the table. Both are safe to repeat.

0. Register the steps first

INSERT INTO migrations (module, step)
VALUES ('main', '0154-add-job-table'),
       ('main', '0159-consumer-generalization')
ON CONFLICT (module, step) DO NOTHING;

These rows are what make the out-of-band run worth it. Without them, the runner finds the pre-applied schema and runs the statements again: the guards skip the work, not the ACCESS EXCLUSIVE of each ALTER nor the backfill scan.

Registering first is the safe order for the locks: from this moment the runner never touches the group, whatever deploy lands while you work. The other side of the deal is that the steps below become the only thing that applies the schema, so finish them before you deploy the new version.

The insert is unconditional: a database that already has the table still gets its row here.

1. Stop the legacy archiver and its GC

Turn off the old archiver and its garbage collection with their flags: :audit-log-archive off and :audit-log-gc off. The new version uses :audit-log-gc for its own GC, so the same flag keeps both off during the rollout. The new version has no :audit-log flag: writing full rows follows the consumer flags. Keep on the destinations this instance already uses: :nexus only if it has a Nexus URI after the conversion below (it is an internal destination), and :admin-console only if the instance uses it. At least one of those flags must be on on every instance, web and worker: each process computes its own persistence value at boot, so a web instance without a consumer flag writes no rows even if a worker drains them. Do not turn on a flag whose URI you do not have: a Nexus flag without :nexus-uri fails the boot. 2.19 ignores :nexus (the name is new), but it does read :admin-console for its Nitrate client, so keeping that flag on does not enable the new audit consumer there. Roll the old workers with the old flags off (the web instances keep writing through the 2.19 code; the other jobs keep running) and wait for the active archive and GC tasks to finish. With all consumers off the new version writes no rows, so there is no backlog to replay later. The old and new versions do not serialize each other: a concurrent old archiver can mark and send rows the new producer wrote, and the old GC can delete rows the new consumers have not read yet.

Nexus URI. 2.19 used :audit-log-archive-uri as the full HTTP endpoint (for example https://nexus.example/proxy/api/audit). The new consumer reads :nexus-uri as the base and appends api/audit: configure https://nexus.example/proxy and check that :nexus-uri + api/audit is the URL Nexus expects before the deploy. Copying the old full URL into :nexus-uri duplicates the path.

2. Delete what the archiver already archived

Delete the rows the old archiver had already archived, so the replay does not send them to Nexus again. They are the rows the old GC would delete anyway. Delete in batches (repeat until the statement removes no rows):

DELETE FROM audit_log
 WHERE id IN (SELECT id FROM audit_log
               WHERE archived_at IS NOT NULL
               LIMIT 5000);

Do not drop archived_at or the indexes that served it (audit_log__created_at__idx, audit_log__archived_at__idx, audit_log_id_archived_at_idx). The old archiver may still be running on an instance that has not been replaced, and a later release removes them. The consumers order by (xid, seq) (step 7); telemetry keeps audit_log__source__created_at__idx plus a sort.

3. Pre-apply the jobs step

2.19 does not have the job table: 0154-add-job-table creates it and is new in this release. Step 0 already registered the step, so the runner will never apply it. Check whether the table exists:

SELECT to_regclass('public.job') AS job;

If it answers a row, nothing to do here: the schema and the registration row are both in place. If it answers null, run this block as-is (the whole step, copied from backend/src/app/migrations/sql/0154-add-job-table.sql so you can paste it whole):

BEGIN;
CREATE TABLE job (
    id           uuid PRIMARY KEY DEFAULT gen_random_uuid(),
    name         text NOT NULL,
    tenant       text NOT NULL,
    queue        text NOT NULL,
    label        text,
    priority     int NOT NULL DEFAULT 100,
    scheduled_at timestamptz NOT NULL DEFAULT now(),
    retry_num    int NOT NULL DEFAULT 0,
    max_retries  int NOT NULL DEFAULT 3,
    status       text NOT NULL DEFAULT 'new'
                 CHECK (status IN ('new', 'scheduled', 'running', 'retry',
                                   'completed', 'failed', 'cancelled',
                                   'aborted')),
    created_at   timestamptz NOT NULL DEFAULT now(),
    modified_at  timestamptz NOT NULL DEFAULT now(),
    started_at   timestamptz,
    completed_at timestamptz,
    params       jsonb NOT NULL DEFAULT '{}',

    -- optional ledger columns (user-facing jobs)
    profile_id   uuid NULL REFERENCES profile(id) ON DELETE NO ACTION DEFERRABLE,
    error        jsonb,
    result       jsonb,
    resource_id  uuid NULL REFERENCES storage_object(id) ON DELETE SET NULL DEFERRABLE,
    expires_at   timestamptz
);

-- Dispatcher claim index: the worker selects due `new`/`retry` rows of
-- its own tenant, orders them by priority and scheduled time, and locks
-- the batch with SKIP LOCKED. The partial predicate keeps terminal rows
-- out and `status` needs no key column for that reason; `tenant` leads
-- because it is an equality the index can answer, and the leading
-- columns then carry the ORDER BY unchanged. `queue` does not lead: the
-- dispatcher claims every queue of its tenant and groups them when
-- pushing, so filtering on one queue would be wrong.
CREATE INDEX job__dispatcher__idx
    ON job (tenant, priority DESC, scheduled_at)
    WHERE status IN ('new', 'retry');

-- Dispatcher orphan sweep: the worker finds the `running` rows of its
-- own tenant whose modified_at is older than the lease and marks them
-- `aborted` (system-side terminal, never retried).
CREATE INDEX job__orphan__idx
    ON job (tenant, modified_at)
    WHERE status = 'running';

-- User-facing job ledger index: it supports filtering by profile and
-- ordering a user's jobs by newest first. No current production query
-- uses it yet; it is kept for the user-facing ledger path.
CREATE INDEX job__profile__idx
    ON job (profile_id, created_at DESC)
    WHERE profile_id IS NOT NULL;

-- Storage GC reference check: it answers whether any job still points
-- to a candidate storage object. Rows without a resource are excluded.
CREATE INDEX job__resource__idx
    ON job (resource_id)
    WHERE resource_id IS NOT NULL;

-- The lost-sweep index of `scheduled` jobs is created by
-- `0159-consumer-generalization`, on `modified_at` (the hand-off time),
-- which is the column the sweep filters by: building it here on
-- `scheduled_at` would only throw it away in the same transaction.

-- Jobs GC expiration: it scans jobs past expires_at while excluding
-- running and retry jobs from the expiration delete.
CREATE INDEX job__expires__idx
    ON job (expires_at)
    WHERE expires_at IS NOT NULL;

-- Jobs GC retention: it finds old internal terminal jobs with no
-- profile, which are eligible for removal after the retention delay.
CREATE INDEX job__retention__idx
    ON job (modified_at)
    WHERE status IN ('completed', 'failed', 'cancelled', 'aborted')
      AND profile_id IS NULL;

-- Append-only job event log: `start`, `progress`, `retry` and `end` rows in
-- insertion order. This is the only source of truth for progress; there is no
-- Redis key and no mutable progress column. The foreign key cascades so
-- deleting a job removes its whole history, and it is DEFERRABLE like every
-- other new reference of the substrate.
CREATE TABLE job_event (
    id         bigserial PRIMARY KEY,
    job_id     uuid NOT NULL REFERENCES job(id) ON DELETE CASCADE DEFERRABLE,
    kind       text NOT NULL
               CHECK (kind IN ('start', 'progress', 'retry', 'end')),
    payload    jsonb NOT NULL DEFAULT '{}',
    created_at timestamptz NOT NULL DEFAULT now()
);

-- Event history read: all events of a job, and the last progress report
-- (`WHERE job_id = ? AND kind = 'progress' ORDER BY created_at DESC LIMIT 1`).
CREATE INDEX job_event__job_kind_created_idx
    ON job_event (job_id, kind, created_at DESC, id DESC);

-- Deletion protection, like the domain tables (`team`, `profile`, the
-- file thumbnail tables). The only intended deleter of `job` rows is the
-- jobs GC, which also marks the referenced storage objects as touched in
-- the same transaction; a stray `DELETE FROM job` would drop the row
-- (and cascade `job_event`) without touching the object, orphaning it
-- forever. The jobs GC disables the guard inside its transaction with
-- `SET LOCAL rules.deletion_protection TO off`, the same escape hatch used
-- by `app.tasks.objects-gc`; `app.jobs/submit` never deletes.
CREATE OR REPLACE TRIGGER deletion_protection__tgr
BEFORE DELETE ON job FOR EACH STATEMENT
  WHEN ((current_setting('rules.deletion_protection', true) IN ('on', '')) OR
        (current_setting('rules.deletion_protection', true) IS NULL))
  EXECUTE PROCEDURE raise_deletion_protection();
COMMIT;

The tables are new and empty and the old version never writes them, so every statement is instant. A failure rolls the block back: re-run it whole instead of completing objects by hand. The block is the whole step, not just the tables: job with its six indexes (dispatcher, orphan, profile, resource, expires, retention), job_event with its history index, and the deletion_protection__tgr trigger. Step 9 checks every one of them.

The old version never touches this table, but the new version needs it to boot.

4. Audit log: columns, reserved range and defaults

Columns first (metadata-only), the reserved low range and the defaults the old web instances need immediately: they keep inserting while the DDL and the backfill run, and the database fills the new columns for them. The old code ignores them.

CREATE SEQUENCE IF NOT EXISTS audit_log_seq_seq;
ALTER TABLE audit_log ADD COLUMN IF NOT EXISTS seq bigint;
ALTER TABLE audit_log ADD COLUMN IF NOT EXISTS xid xid8 NOT NULL DEFAULT '0'::xid8;
ALTER SEQUENCE audit_log_seq_seq OWNED BY audit_log.seq;

-- Reserve the low range for the history: new rows start above the backfill.
-- Only on the first application: a sequence that already left 1 is reserved
-- by the operator or in use, and a re-run must not move it.
SELECT setval('audit_log_seq_seq',
              (SELECT count(*) + 1000000 FROM audit_log),
              false)
 WHERE (SELECT last_value FROM audit_log_seq_seq) = 1
   AND NOT (SELECT is_called FROM audit_log_seq_seq);

ALTER TABLE audit_log ALTER COLUMN seq SET DEFAULT nextval('audit_log_seq_seq');
ALTER TABLE audit_log ALTER COLUMN xid SET DEFAULT pg_current_xact_id();

5. Audit log: backfill

One statement, row locks only (never a table lock), and it filters seq IS NULL, so it can be interrupted and re-run. The archiver is stopped, but the web instances keep inserting; their new rows get seq from the default, so the backfill only touches the history. It sorts by (created_at, id): the (created_at, id) index this branch used belonged to the pending queue and never shipped. On a large table, raise work_mem for the session before running it so the sort stays in memory:

UPDATE audit_log
   SET seq = ordered.rn
  FROM (SELECT id, row_number() OVER (ORDER BY created_at, id) AS rn
          FROM audit_log
         WHERE seq IS NULL) ordered
 WHERE audit_log.id = ordered.id;

6. Audit log: seq and source NOT NULL without a long ACCESS EXCLUSIVE scan

No row can be NULL in seq here: the default was set in step 4. source is nullable since 0064; fill what is left first.

ALTER TABLE audit_log ADD CONSTRAINT audit_log__seq__not_null
    CHECK (seq IS NOT NULL) NOT VALID;
ALTER TABLE audit_log VALIDATE CONSTRAINT audit_log__seq__not_null;
ALTER TABLE audit_log ALTER COLUMN seq SET NOT NULL;
ALTER TABLE audit_log DROP CONSTRAINT audit_log__seq__not_null;

UPDATE audit_log SET source = 'backend' WHERE source IS NULL;

ALTER TABLE audit_log ADD CONSTRAINT audit_log__source__not_null
    CHECK (source IS NOT NULL) NOT VALID;
ALTER TABLE audit_log VALIDATE CONSTRAINT audit_log__source__not_null;
ALTER TABLE audit_log ALTER COLUMN source SET NOT NULL;
ALTER TABLE audit_log DROP CONSTRAINT audit_log__source__not_null;

7. Audit log: cursor index and cursor table

The index with CONCURRENTLY, and the cursor table, which starts empty:

CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS audit_log__xid__seq__idx
    ON audit_log (xid, seq);

CREATE TABLE IF NOT EXISTS audit_log_consumer_cursor (
  consumer   smallint    NOT NULL,
  xid        xid8        NOT NULL,
  seq        bigint      NOT NULL,
  updated_at timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (consumer));

8. Jobs: exclusive and dedupe columns and indexes

The two columns and the two unique indexes, the indexes with CONCURRENTLY:

ALTER TABLE job ADD COLUMN IF NOT EXISTS exclusive boolean NOT NULL DEFAULT false;
ALTER TABLE job ADD COLUMN IF NOT EXISTS dedupe    boolean NOT NULL DEFAULT false;

CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS job__live_exclusive__uniq
    ON job (tenant, name, queue, label)
    WHERE exclusive AND status IN ('new', 'scheduled', 'running', 'retry');

CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS job__new_dedupe__uniq
    ON job (tenant, name, queue, label)
    WHERE dedupe AND status = 'new';

The lost sweep index changes shape: it filters by modified_at, not by scheduled_at. job is not in production yet, so a plain rebuild is enough:

DROP INDEX IF EXISTS job__scheduled__idx;
CREATE INDEX IF NOT EXISTS job__scheduled__idx
    ON job (tenant, modified_at)
 WHERE status = 'scheduled';

If an early 0154 built the cron probe index, drop it too (the submit resolves the overlap at insert time now):

DROP INDEX IF EXISTS job__name_label__idx;

9. Gate before deploying

An interrupted CONCURRENTLY build leaves the index INVALID, and IF NOT EXISTS matches by name, so the migration would accept it and the dedupe/exclusivity would stop enforcing silently. The checks:

SELECT indexname, indexdef FROM pg_indexes
 WHERE indexname IN ('job__live_exclusive__uniq',
                     'job__new_dedupe__uniq',
                     'job__scheduled__idx',
                     'audit_log__xid__seq__idx');

SELECT indexrelid::regclass AS index, indisvalid, indisunique
  FROM pg_index
 WHERE indexrelid IN ('job__live_exclusive__uniq'::regclass,
                      'job__new_dedupe__uniq'::regclass,
                      'audit_log__xid__seq__idx'::regclass);

SELECT column_name, is_nullable
  FROM information_schema.columns
 WHERE table_name = 'audit_log'
   AND column_name IN ('seq', 'xid', 'source')
 ORDER BY 1;

Any index with indisvalid = false must be dropped and rebuilt before the deploy, and the three columns of audit_log must answer NO.

The job side is a gate too: both job tables, the cursor table, the indexes of the check and the trigger must exist before the deploy. A half-applied 0154 or a skipped step 8 would otherwise only surface as a boot failure.

SELECT to_regclass('public.job') IS NOT NULL AS job,
       to_regclass('public.job_event') IS NOT NULL AS job_event,
       to_regclass('public.audit_log_consumer_cursor') IS NOT NULL AS cursor;

SELECT indexname FROM pg_indexes
 WHERE tablename IN ('job', 'job_event')
   AND indexname IN ('job__dispatcher__idx', 'job__orphan__idx',
                     'job__profile__idx', 'job__resource__idx',
                     'job__expires__idx', 'job__retention__idx',
                     'job__scheduled__idx', 'job_event__job_kind_created_idx')
 ORDER BY 1;

SELECT column_name, is_nullable
  FROM information_schema.columns
  WHERE table_name = 'job'
   AND column_name IN ('exclusive', 'dedupe')
  ORDER BY 1;

SELECT tgname FROM pg_trigger
 WHERE tgrelid = to_regclass('public.job')
   AND tgname = 'deletion_protection__tgr'
   AND NOT tgisinternal;

The three tables must answer true, the eight indexes must be listed, the two flags must answer NO, and the trigger must appear. The job__scheduled__idx definition of the first check must read (tenant, modified_at), not (tenant, scheduled_at): step 8 drops and rebuilds it, and CREATE INDEX IF NOT EXISTS matches by name. This is a gate: do not deploy if it fails. The catalog checks are instant; the CHECK NOT VALID / VALIDATE / SET NOT NULL dance already proved there is no NULL, so a count(*) would only scan the table again.

10. After the rollout

Once every instance runs this version, the new consumers have replayed the unarchived buffer. Turn on :audit-log-gc (rolling restart of the new workers) so the new GC deletes the rows the consumers have passed. Remove :audit-log-archive; the new version does not use it.