* ♻️ Rework audit-log consumers and jobs substrate Fold the implementation, follow-up fixes, migrations and tests into one code commit. AI-assisted-by: gpt-6-luna * 📚 Consolidate 2.20.0 upgrade runbooks Keep the operational upgrade procedure and its supporting runbooks in a separate documentation commit. AI-assisted-by: gpt-6-luna * ♻️ Remove the audit-log flag and derive persistence at boot Persistence now follows the consumer flags instead of a separate switch. app.config computes cf/audit-persist? once at boot from :nexus and :admin-console, and every write site reads that value, so request paths no longer check flags per event. With no consumer enabled no full rows are written; telemetry shadows and webhook deliveries keep their own flags. The frontend wire signal stays :audit-log as a derived value. Update the 2.20 runbooks, memories and the devenv flags to match, and migrate the tests that used to toggle :audit-log directly. AI-assisted-by: hy4-flash-free * 🐛 Accept error-reporting batches and precompute sink flags push-audit-events now accepts crash batches with :error-reporting alone, so the frontend signal from get-environment-data matches a real destination. The reporters receive unhandled-exception events without storing any row. All per-event sink checks read booleans computed once at boot: webhooks, telemetry and error reporting join audit-persist? in app.config. The test fixture and with-audit-config rebind all four values together, and the tests that toggled those flags directly were migrated. Address F2 of the review of the previous commit. F1 stays deferred to a separate plan. AI-assisted-by: hy4-flash-free * 💄 Match the layout cljfmt expects in the flags set The first comment of the varia set sat on its own line after the opening delimiter, and cljfmt joins such a comment to the delimiter line, so the Common Tests lint step failed on common/src/app/common/flags.cljc. Keep the comment on the opening line, the same shape the login and email sets already use. Note in mem:scripts/ci that the backend lint lints ../common/src but does not format-check it, so common has to be passed to scripts/ci as well. AI-assisted-by: deepseek-v4-flash
20 KiB
Consumer generalization: consumer handover and out-of-band migration
Procedure to run before upgrading to the release that ships the audit-log cursor consumers and the jobs substrate. Steps 1-2 (the consumer handover) apply to every instance that ran the legacy archiver or its GC; the out-of-band DDL is for a large instance.
- Migration:
backend/src/app/migrations/sql/0159-consumer-generalization.sql - Issue: https://github.com/penpot/penpot/issues/12700
When you need this
Run the consumer handover before upgrading to the release that ships the audit-log cursor consumers and the jobs substrate, while the current version is still serving traffic. The web instances keep writing for the whole procedure (step 4 relies on it); only the legacy archiver and its GC stop (step 1).
Steps 1 and 2 are mandatory whenever the legacy archiver or its GC was
enabled in 2.19, even when audit_log is small: the new consumers start at
(0, 0), so the rows the old archiver already archived would be replayed and
the old GC could delete rows the new consumers have not read. Only the
out-of-band DDL is optional by size.
You need the out-of-band DDL when audit_log is large. The migration runner
executes every pending step in a single transaction: the plain index builds
hold a SHARE lock for the whole build and every ALTER keeps its ACCESS EXCLUSIVE until that one commit, so on a large table the upgrade stalls
writes for minutes. Pre-applying the same statements out of band and in this
order (outside a transaction block, except the jobs step, which runs as one
explicit BEGIN/COMMIT), plus registering the steps first (step 0), makes
the runner skip the group, so the deploy takes no such locks. On a small table
the migration runs fast enough on its own: skip the DDL steps, not the
handover. This procedure covers 0159-consumer-generalization and the
0154-add-job-table step it depends on.
Pre-applying does not make the migration free by itself: the guards
(IF NOT EXISTS, seq IS NULL) skip the work, not the locks. Registering the
step is what makes the runner skip the group.
The application keeps writing to audit_log for the whole procedure. Do
not stop the web instances. The DDL takes a brief ACCESS EXCLUSIVE
(metadata-only) that lock_timeout bounds; the index builds use
CONCURRENTLY; the backfill takes row locks only. Stopping the archiver and its
GC is fine; stopping the writes is not.
While the new version runs, the consumers read only rows below
pg_snapshot_xmin(pg_current_snapshot()), the oldest transaction of the whole
database: a long write transaction (a binfile import runs in one for up to 30
minutes) pauses archiving and the GC until it commits, and a slow destination
does too, because a chunk sends inside its transaction and holds the cut back
until it ends. Nothing is lost and the lag gauge shows it; it is not a stuck
consumer.
Session settings
Run every block below in a session with these two settings. They are plain SQL
(no psql meta-commands), so any client accepts them, and they last for the
session: if you paste a block into a new session, run them first.
SET statement_timeout = 0;
SET lock_timeout = '5s';
statement_timeout = 0 keeps a client's default from killing the backfill or
the CONCURRENTLY builds. lock_timeout = '5s' makes an ALTER that waits
behind a long transaction fail and be retried, instead of queueing in front of
every later query on the table. Both are safe to repeat.
0. Register the steps first
INSERT INTO migrations (module, step)
VALUES ('main', '0154-add-job-table'),
('main', '0159-consumer-generalization')
ON CONFLICT (module, step) DO NOTHING;
These rows are what make the out-of-band run worth it. Without them, the
runner finds the pre-applied schema and runs the statements again: the guards
skip the work, not the ACCESS EXCLUSIVE of each ALTER nor the backfill
scan.
Registering first is the safe order for the locks: from this moment the runner never touches the group, whatever deploy lands while you work. The other side of the deal is that the steps below become the only thing that applies the schema, so finish them before you deploy the new version.
The insert is unconditional: a database that already has the table still gets its row here.
1. Stop the legacy archiver and its GC
Turn off the old archiver and its garbage collection with their flags:
:audit-log-archive off and :audit-log-gc off. The new version uses
:audit-log-gc for its own GC, so the same flag keeps both off during the
rollout. The new version has no :audit-log flag: writing full rows follows
the consumer flags. Keep on the destinations this instance already uses:
:nexus only if it has a Nexus URI after the conversion below (it is an
internal destination), and :admin-console only if the instance uses it. At
least one of those flags must be on on every instance, web and worker:
each process computes its own persistence value at boot, so a web instance
without a consumer flag writes no rows even if a worker drains them. Do not
turn on a flag whose URI you do not have: a Nexus flag without :nexus-uri
fails the boot. 2.19 ignores :nexus (the name is new), but it does read
:admin-console for its Nitrate client, so keeping that flag on does not
enable the new audit consumer there. Roll the old workers with the old flags
off (the web instances keep writing through the 2.19 code; the other jobs
keep running) and wait for the active archive and GC tasks to finish. With
all consumers off the new version writes no rows, so there is no backlog to
replay later. The old and new versions do not serialize each other: a
concurrent old archiver can mark and send rows the new producer wrote, and
the old GC can delete rows the new consumers have not read yet.
Nexus URI. 2.19 used :audit-log-archive-uri as the full HTTP endpoint
(for example https://nexus.example/proxy/api/audit). The new consumer reads
:nexus-uri as the base and appends api/audit: configure
https://nexus.example/proxy and check that :nexus-uri + api/audit is
the URL Nexus expects before the deploy. Copying the old full URL into
:nexus-uri duplicates the path.
2. Delete what the archiver already archived
Delete the rows the old archiver had already archived, so the replay does not send them to Nexus again. They are the rows the old GC would delete anyway. Delete in batches (repeat until the statement removes no rows):
DELETE FROM audit_log
WHERE id IN (SELECT id FROM audit_log
WHERE archived_at IS NOT NULL
LIMIT 5000);
Do not drop archived_at or the indexes that served it
(audit_log__created_at__idx, audit_log__archived_at__idx,
audit_log_id_archived_at_idx). The old archiver may still be running on an
instance that has not been replaced, and a later release removes them. The
consumers order by (xid, seq) (step 7); telemetry keeps
audit_log__source__created_at__idx plus a sort.
3. Pre-apply the jobs step
2.19 does not have the job table: 0154-add-job-table creates it and is
new in this release. Step 0 already registered the step, so the runner will
never apply it. Check whether the table exists:
SELECT to_regclass('public.job') AS job;
If it answers a row, nothing to do here: the schema and the registration row
are both in place. If it answers null, run this block as-is (the whole step, copied from
backend/src/app/migrations/sql/0154-add-job-table.sql
so you can paste it whole):
BEGIN;
CREATE TABLE job (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
name text NOT NULL,
tenant text NOT NULL,
queue text NOT NULL,
label text,
priority int NOT NULL DEFAULT 100,
scheduled_at timestamptz NOT NULL DEFAULT now(),
retry_num int NOT NULL DEFAULT 0,
max_retries int NOT NULL DEFAULT 3,
status text NOT NULL DEFAULT 'new'
CHECK (status IN ('new', 'scheduled', 'running', 'retry',
'completed', 'failed', 'cancelled',
'aborted')),
created_at timestamptz NOT NULL DEFAULT now(),
modified_at timestamptz NOT NULL DEFAULT now(),
started_at timestamptz,
completed_at timestamptz,
params jsonb NOT NULL DEFAULT '{}',
-- optional ledger columns (user-facing jobs)
profile_id uuid NULL REFERENCES profile(id) ON DELETE NO ACTION DEFERRABLE,
error jsonb,
result jsonb,
resource_id uuid NULL REFERENCES storage_object(id) ON DELETE SET NULL DEFERRABLE,
expires_at timestamptz
);
-- Dispatcher claim index: the worker selects due `new`/`retry` rows of
-- its own tenant, orders them by priority and scheduled time, and locks
-- the batch with SKIP LOCKED. The partial predicate keeps terminal rows
-- out and `status` needs no key column for that reason; `tenant` leads
-- because it is an equality the index can answer, and the leading
-- columns then carry the ORDER BY unchanged. `queue` does not lead: the
-- dispatcher claims every queue of its tenant and groups them when
-- pushing, so filtering on one queue would be wrong.
CREATE INDEX job__dispatcher__idx
ON job (tenant, priority DESC, scheduled_at)
WHERE status IN ('new', 'retry');
-- Dispatcher orphan sweep: the worker finds the `running` rows of its
-- own tenant whose modified_at is older than the lease and marks them
-- `aborted` (system-side terminal, never retried).
CREATE INDEX job__orphan__idx
ON job (tenant, modified_at)
WHERE status = 'running';
-- User-facing job ledger index: it supports filtering by profile and
-- ordering a user's jobs by newest first. No current production query
-- uses it yet; it is kept for the user-facing ledger path.
CREATE INDEX job__profile__idx
ON job (profile_id, created_at DESC)
WHERE profile_id IS NOT NULL;
-- Storage GC reference check: it answers whether any job still points
-- to a candidate storage object. Rows without a resource are excluded.
CREATE INDEX job__resource__idx
ON job (resource_id)
WHERE resource_id IS NOT NULL;
-- The lost-sweep index of `scheduled` jobs is created by
-- `0159-consumer-generalization`, on `modified_at` (the hand-off time),
-- which is the column the sweep filters by: building it here on
-- `scheduled_at` would only throw it away in the same transaction.
-- Jobs GC expiration: it scans jobs past expires_at while excluding
-- running and retry jobs from the expiration delete.
CREATE INDEX job__expires__idx
ON job (expires_at)
WHERE expires_at IS NOT NULL;
-- Jobs GC retention: it finds old internal terminal jobs with no
-- profile, which are eligible for removal after the retention delay.
CREATE INDEX job__retention__idx
ON job (modified_at)
WHERE status IN ('completed', 'failed', 'cancelled', 'aborted')
AND profile_id IS NULL;
-- Append-only job event log: `start`, `progress`, `retry` and `end` rows in
-- insertion order. This is the only source of truth for progress; there is no
-- Redis key and no mutable progress column. The foreign key cascades so
-- deleting a job removes its whole history, and it is DEFERRABLE like every
-- other new reference of the substrate.
CREATE TABLE job_event (
id bigserial PRIMARY KEY,
job_id uuid NOT NULL REFERENCES job(id) ON DELETE CASCADE DEFERRABLE,
kind text NOT NULL
CHECK (kind IN ('start', 'progress', 'retry', 'end')),
payload jsonb NOT NULL DEFAULT '{}',
created_at timestamptz NOT NULL DEFAULT now()
);
-- Event history read: all events of a job, and the last progress report
-- (`WHERE job_id = ? AND kind = 'progress' ORDER BY created_at DESC LIMIT 1`).
CREATE INDEX job_event__job_kind_created_idx
ON job_event (job_id, kind, created_at DESC, id DESC);
-- Deletion protection, like the domain tables (`team`, `profile`, the
-- file thumbnail tables). The only intended deleter of `job` rows is the
-- jobs GC, which also marks the referenced storage objects as touched in
-- the same transaction; a stray `DELETE FROM job` would drop the row
-- (and cascade `job_event`) without touching the object, orphaning it
-- forever. The jobs GC disables the guard inside its transaction with
-- `SET LOCAL rules.deletion_protection TO off`, the same escape hatch used
-- by `app.tasks.objects-gc`; `app.jobs/submit` never deletes.
CREATE OR REPLACE TRIGGER deletion_protection__tgr
BEFORE DELETE ON job FOR EACH STATEMENT
WHEN ((current_setting('rules.deletion_protection', true) IN ('on', '')) OR
(current_setting('rules.deletion_protection', true) IS NULL))
EXECUTE PROCEDURE raise_deletion_protection();
COMMIT;
The tables are new and empty and the old version never writes them, so every
statement is instant. A failure rolls the block back: re-run it whole
instead of completing objects by hand. The block is the whole step, not just
the tables: job with its six indexes (dispatcher, orphan, profile,
resource, expires, retention), job_event with its history index, and the
deletion_protection__tgr trigger. Step 9 checks every one of them.
The old version never touches this table, but the new version needs it to boot.
4. Audit log: columns, reserved range and defaults
Columns first (metadata-only), the reserved low range and the defaults the old web instances need immediately: they keep inserting while the DDL and the backfill run, and the database fills the new columns for them. The old code ignores them.
CREATE SEQUENCE IF NOT EXISTS audit_log_seq_seq;
ALTER TABLE audit_log ADD COLUMN IF NOT EXISTS seq bigint;
ALTER TABLE audit_log ADD COLUMN IF NOT EXISTS xid xid8 NOT NULL DEFAULT '0'::xid8;
ALTER SEQUENCE audit_log_seq_seq OWNED BY audit_log.seq;
-- Reserve the low range for the history: new rows start above the backfill.
-- Only on the first application: a sequence that already left 1 is reserved
-- by the operator or in use, and a re-run must not move it.
SELECT setval('audit_log_seq_seq',
(SELECT count(*) + 1000000 FROM audit_log),
false)
WHERE (SELECT last_value FROM audit_log_seq_seq) = 1
AND NOT (SELECT is_called FROM audit_log_seq_seq);
ALTER TABLE audit_log ALTER COLUMN seq SET DEFAULT nextval('audit_log_seq_seq');
ALTER TABLE audit_log ALTER COLUMN xid SET DEFAULT pg_current_xact_id();
5. Audit log: backfill
One statement, row locks only (never a table lock), and it filters
seq IS NULL, so it can be interrupted and re-run. The
archiver is stopped, but the web instances keep inserting; their new rows get
seq from the default, so the backfill only touches the history. It sorts by
(created_at, id): the (created_at, id) index this branch used belonged to
the pending queue and never shipped. On a large table, raise work_mem for the
session before running it so the sort stays in memory:
UPDATE audit_log
SET seq = ordered.rn
FROM (SELECT id, row_number() OVER (ORDER BY created_at, id) AS rn
FROM audit_log
WHERE seq IS NULL) ordered
WHERE audit_log.id = ordered.id;
6. Audit log: seq and source NOT NULL without a long ACCESS EXCLUSIVE scan
No row can be NULL in seq here: the default was set in step 4. source is
nullable since 0064; fill what is left first.
ALTER TABLE audit_log ADD CONSTRAINT audit_log__seq__not_null
CHECK (seq IS NOT NULL) NOT VALID;
ALTER TABLE audit_log VALIDATE CONSTRAINT audit_log__seq__not_null;
ALTER TABLE audit_log ALTER COLUMN seq SET NOT NULL;
ALTER TABLE audit_log DROP CONSTRAINT audit_log__seq__not_null;
UPDATE audit_log SET source = 'backend' WHERE source IS NULL;
ALTER TABLE audit_log ADD CONSTRAINT audit_log__source__not_null
CHECK (source IS NOT NULL) NOT VALID;
ALTER TABLE audit_log VALIDATE CONSTRAINT audit_log__source__not_null;
ALTER TABLE audit_log ALTER COLUMN source SET NOT NULL;
ALTER TABLE audit_log DROP CONSTRAINT audit_log__source__not_null;
7. Audit log: cursor index and cursor table
The index with CONCURRENTLY, and the cursor table, which starts empty:
CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS audit_log__xid__seq__idx
ON audit_log (xid, seq);
CREATE TABLE IF NOT EXISTS audit_log_consumer_cursor (
consumer smallint NOT NULL,
xid xid8 NOT NULL,
seq bigint NOT NULL,
updated_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (consumer));
8. Jobs: exclusive and dedupe columns and indexes
The two columns and the two unique indexes, the indexes with CONCURRENTLY:
ALTER TABLE job ADD COLUMN IF NOT EXISTS exclusive boolean NOT NULL DEFAULT false;
ALTER TABLE job ADD COLUMN IF NOT EXISTS dedupe boolean NOT NULL DEFAULT false;
CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS job__live_exclusive__uniq
ON job (tenant, name, queue, label)
WHERE exclusive AND status IN ('new', 'scheduled', 'running', 'retry');
CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS job__new_dedupe__uniq
ON job (tenant, name, queue, label)
WHERE dedupe AND status = 'new';
The lost sweep index changes shape: it filters by modified_at, not by
scheduled_at. job is not in production yet, so a plain rebuild is enough:
DROP INDEX IF EXISTS job__scheduled__idx;
CREATE INDEX IF NOT EXISTS job__scheduled__idx
ON job (tenant, modified_at)
WHERE status = 'scheduled';
If an early 0154 built the cron probe index, drop it too (the submit resolves
the overlap at insert time now):
DROP INDEX IF EXISTS job__name_label__idx;
9. Gate before deploying
An interrupted CONCURRENTLY build leaves the index INVALID, and
IF NOT EXISTS matches by name, so the migration would accept it and the
dedupe/exclusivity would stop enforcing silently. The checks:
SELECT indexname, indexdef FROM pg_indexes
WHERE indexname IN ('job__live_exclusive__uniq',
'job__new_dedupe__uniq',
'job__scheduled__idx',
'audit_log__xid__seq__idx');
SELECT indexrelid::regclass AS index, indisvalid, indisunique
FROM pg_index
WHERE indexrelid IN ('job__live_exclusive__uniq'::regclass,
'job__new_dedupe__uniq'::regclass,
'audit_log__xid__seq__idx'::regclass);
SELECT column_name, is_nullable
FROM information_schema.columns
WHERE table_name = 'audit_log'
AND column_name IN ('seq', 'xid', 'source')
ORDER BY 1;
Any index with indisvalid = false must be dropped and rebuilt before the
deploy, and the three columns of audit_log must answer NO.
The job side is a gate too: both job tables, the cursor table, the indexes of
the check and the trigger must exist before the deploy. A half-applied 0154
or a skipped step 8 would otherwise only surface as a boot failure.
SELECT to_regclass('public.job') IS NOT NULL AS job,
to_regclass('public.job_event') IS NOT NULL AS job_event,
to_regclass('public.audit_log_consumer_cursor') IS NOT NULL AS cursor;
SELECT indexname FROM pg_indexes
WHERE tablename IN ('job', 'job_event')
AND indexname IN ('job__dispatcher__idx', 'job__orphan__idx',
'job__profile__idx', 'job__resource__idx',
'job__expires__idx', 'job__retention__idx',
'job__scheduled__idx', 'job_event__job_kind_created_idx')
ORDER BY 1;
SELECT column_name, is_nullable
FROM information_schema.columns
WHERE table_name = 'job'
AND column_name IN ('exclusive', 'dedupe')
ORDER BY 1;
SELECT tgname FROM pg_trigger
WHERE tgrelid = to_regclass('public.job')
AND tgname = 'deletion_protection__tgr'
AND NOT tgisinternal;
The three tables must answer true, the eight indexes must be listed, the two
flags must answer NO, and the trigger must appear. The
job__scheduled__idx definition of the first check must read
(tenant, modified_at), not (tenant, scheduled_at): step 8 drops and
rebuilds it, and CREATE INDEX IF NOT EXISTS matches by name. This is a gate:
do not deploy if it fails. The catalog checks are instant; the
CHECK NOT VALID / VALIDATE / SET NOT NULL dance already proved there is
no NULL, so a count(*) would only scan the table again.
10. After the rollout
Once every instance runs this version, the new consumers have replayed the
unarchived buffer. Turn on :audit-log-gc (rolling restart of the new
workers) so the new GC deletes the rows the consumers have passed. Remove
:audit-log-archive; the new version does not use it.