penpot/operations/2.20.0-consumer-generalization.md
Andrey Antukh e0194ce60a
✨ Deliver telemetry audit events independently (#13230)
* ✨ Ship telemetry shadows with a cursor consumer

Send anonymized audit events through a dedicated telemetry consumer,
including when Nexus or admin-console also persist full events. Keep the
legacy payload and cadence of 10,000 events per chunk and three hours
between passes. Nexus skips shadows to avoid archiving duplicates.

Move shadow deletion to the audit GC. Delete full rows and shadows along
independent cursor paths, and expire shadows after seven days even when
they have not been sent. Run the deletion paths in parallel with bounded
batches, separate time budgets and independent failure handling.

Replace the shared consumer catalog with transport definitions and one
generic pass runner. Keep the legacy telemetry report and newsletter in
the scheduled task, and update the tests, memories and rollout runbooks.

Addresses F1 of the audit-log consumer review.

AI-assisted-by: deepseek-v4-flash
AI-assisted-by: gpt-6.1-terra

* 🐛 Address audit log consumer review findings

Enable audit log GC by default and document how to pause it during rollout
or before registering a consumer that needs a complete replay.

Run GC paths in order on the runner thread. Count each committed batch and
preserve partial totals on ordinary failures while propagating interruption.

Skip full-row payloads in telemetry queries and decoding. Keep the age
limit for local retention only, and document the shared pass contract.

AI-assisted-by: gpt-6.2-sol

* 🐛 Close audit log consumer review findings

Verify telemetry counts processed and skipped events in a mixed batch.
Clarify how to pause GC, the scope of the replay race, and what the
retention warning counts. Remove private Nexus setup from the public
upgrade runbook.

AI-assisted-by: gpt-6.2-sol

* 📚 Simplify the audit log guide

Consolidate the guidance on event trust, privacy and delivery. Keep examples
that explain why the design works and link to code and runbooks for details.
Clarify replay limits and distinguish age-based deletion from missed delivery.

AI-assisted-by: gpt-6.2-sol
2026-11-20 16:28:42 +01:00

23 KiB

Consumer generalization: consumer handover and out-of-band migration

Procedure to run before upgrading to the release that ships the audit-log cursor consumers and the jobs substrate. Steps 1-2 (the consumer handover) apply to every instance that ran the legacy archiver or its GC; the out-of-band DDL is for a large instance.

When you need this

Run the consumer handover before upgrading to the release that ships the audit-log cursor consumers and the jobs substrate, while the current version is still serving traffic. The web instances keep writing for the whole procedure (step 4 relies on it); only the legacy archiver and its GC stop (step 1).

Steps 1 and 2 are mandatory whenever the legacy archiver or its GC was enabled in 2.19, even when audit_log is small: the new consumers start at (0, 0), so the rows the old archiver already archived would be replayed and the old GC could delete rows the new consumers have not read. Only the out-of-band DDL is optional by size.

You need the out-of-band DDL when audit_log is large. The migration runner executes every pending step in a single transaction: the plain index builds hold a SHARE lock for the whole build and every ALTER keeps its ACCESS EXCLUSIVE until that one commit, so on a large table the upgrade stalls writes for minutes. Pre-applying the same statements out of band and in this order (outside a transaction block, except the jobs step, which runs as one explicit BEGIN/COMMIT), plus registering the steps first (step 0), makes the runner skip the group, so the deploy takes no such locks. On a small table the migration runs fast enough on its own: skip the DDL steps, not the handover. This procedure covers 0159-consumer-generalization and the 0154-add-job-table step it depends on.

Pre-applying does not make the migration free by itself: the guards (IF NOT EXISTS, seq IS NULL) skip the work, not the locks. Registering the step is what makes the runner skip the group.

The application keeps writing to audit_log for the whole procedure. Do not stop the web instances. The DDL takes a brief ACCESS EXCLUSIVE (metadata-only) that lock_timeout bounds; the index builds use CONCURRENTLY; the backfill takes row locks only. Stopping the archiver and its GC is fine; stopping the writes is not.

While the new version runs, the consumers read only rows below pg_snapshot_xmin(pg_current_snapshot()), the oldest transaction of the whole database: a long write transaction (a binfile import runs in one for up to 30 minutes) pauses archiving and the GC until it commits, and a slow destination does too, because a chunk sends inside its transaction and holds the cut back until it ends. Nothing is lost and the lag gauge shows it; it is not a stuck consumer.

Session settings

Run every block below in a session with these two settings. They are plain SQL (no psql meta-commands), so any client accepts them, and they last for the session: if you paste a block into a new session, run them first.

SET statement_timeout = 0;
SET lock_timeout = '5s';

statement_timeout = 0 keeps a client's default from killing the backfill or the CONCURRENTLY builds. lock_timeout = '5s' makes an ALTER that waits behind a long transaction fail and be retried, instead of queueing in front of every later query on the table. Both are safe to repeat.

0. Register the steps first

INSERT INTO migrations (module, step)
VALUES ('main', '0154-add-job-table'),
       ('main', '0159-consumer-generalization')
ON CONFLICT (module, step) DO NOTHING;

These rows are what make the out-of-band run worth it. Without them, the runner finds the pre-applied schema and runs the statements again: the guards skip the work, not the ACCESS EXCLUSIVE of each ALTER nor the backfill scan.

Registering first is the safe order for the locks: from this moment the runner never touches the group, whatever deploy lands while you work. The other side of the deal is that the steps below become the only thing that applies the schema, so finish them before you deploy the new version.

The insert is unconditional: a database that already has the table still gets its row here.

1. Stop the legacy archiver and its GC

Before any 2.20.0 worker starts, set disable-audit-log-archive and disable-audit-log-gc in PENPOT_FLAGS on all workers. Remove any existing enable-audit-log-gc; do not leave both tokens. After restarting, check each worker's startup log and confirm audit-log-gc is not active. Keep disable-audit-log-gc throughout the mixed-version rollout: the new GC is on by default, so removing enable-audit-log-gc alone does not stop it. The new version has no :audit-log flag: writing full rows follows the consumer flags. Keep on the destinations this instance already uses: :nexus only if it has a Nexus URI after the conversion below (it is an internal destination), and :admin-console only if the instance uses it. At least one of those flags must be on on every instance, web and worker: each process computes its own persistence value at boot, so a web instance without a consumer flag writes no rows even if a worker drains them. Do not turn on a flag whose URI you do not have: a Nexus flag without :nexus-uri fails the boot. 2.19 ignores :nexus (the name is new), but it does read :admin-console for its Nitrate client, so keeping that flag on does not enable the new audit consumer there. Roll the old workers with the old flags off (the web instances keep writing through the 2.19 code; the other jobs keep running) and wait for the active archive and GC tasks to finish. With all consumers off the new version writes no rows, so there is no backlog to replay later. The old and new versions do not serialize each other: a concurrent old archiver can mark and send rows the new producer wrote, and the old GC can delete rows the new consumers have not read yet.

Nexus URI. 2.19 used :audit-log-archive-uri as the full HTTP endpoint (for example https://nexus.example/proxy/api/audit). The new consumer reads :nexus-uri as the base and appends api/audit: configure https://nexus.example/proxy and check that :nexus-uri + api/audit is the URL Nexus expects before the deploy. Copying the old full URL into :nexus-uri duplicates the path.

2. Delete what the archiver already archived

Delete the rows the old archiver had already archived, so the replay does not send them to Nexus again. They are the rows the old GC would delete anyway. Delete in batches (repeat until the statement removes no rows):

DELETE FROM audit_log
 WHERE id IN (SELECT id FROM audit_log
               WHERE archived_at IS NOT NULL
               LIMIT 5000);

Do not drop archived_at or the indexes that served it (audit_log__created_at__idx, audit_log__archived_at__idx, audit_log_id_archived_at_idx). The old archiver may still be running on an instance that has not been replaced, and a later release removes them. The consumers order by (xid, seq) (step 7), so audit_log__source__created_at__idx is not used and waits for the same cleanup.

3. Pre-apply the jobs step

2.19 does not have the job table: 0154-add-job-table creates it and is new in this release. Step 0 already registered the step, so the runner will never apply it. Check whether the table exists:

SELECT to_regclass('public.job') AS job;

If it answers a row, nothing to do here: the schema and the registration row are both in place. If it answers null, run this block as-is (the whole step, copied from backend/src/app/migrations/sql/0154-add-job-table.sql so you can paste it whole):

BEGIN;
CREATE TABLE job (
    id           uuid PRIMARY KEY DEFAULT gen_random_uuid(),
    name         text NOT NULL,
    tenant       text NOT NULL,
    queue        text NOT NULL,
    label        text,
    priority     int NOT NULL DEFAULT 100,
    scheduled_at timestamptz NOT NULL DEFAULT now(),
    retry_num    int NOT NULL DEFAULT 0,
    max_retries  int NOT NULL DEFAULT 3,
    status       text NOT NULL DEFAULT 'new'
                 CHECK (status IN ('new', 'scheduled', 'running', 'retry',
                                   'completed', 'failed', 'cancelled',
                                   'aborted')),
    created_at   timestamptz NOT NULL DEFAULT now(),
    modified_at  timestamptz NOT NULL DEFAULT now(),
    started_at   timestamptz,
    completed_at timestamptz,
    params       jsonb NOT NULL DEFAULT '{}',

    -- optional ledger columns (user-facing jobs)
    profile_id   uuid NULL REFERENCES profile(id) ON DELETE NO ACTION DEFERRABLE,
    error        jsonb,
    result       jsonb,
    resource_id  uuid NULL REFERENCES storage_object(id) ON DELETE SET NULL DEFERRABLE,
    expires_at   timestamptz
);

-- Dispatcher claim index: the worker selects due `new`/`retry` rows of
-- its own tenant, orders them by priority and scheduled time, and locks
-- the batch with SKIP LOCKED. The partial predicate keeps terminal rows
-- out and `status` needs no key column for that reason; `tenant` leads
-- because it is an equality the index can answer, and the leading
-- columns then carry the ORDER BY unchanged. `queue` does not lead: the
-- dispatcher claims every queue of its tenant and groups them when
-- pushing, so filtering on one queue would be wrong.
CREATE INDEX job__dispatcher__idx
    ON job (tenant, priority DESC, scheduled_at)
    WHERE status IN ('new', 'retry');

-- Dispatcher orphan sweep: the worker finds the `running` rows of its
-- own tenant whose modified_at is older than the lease and marks them
-- `aborted` (system-side terminal, never retried).
CREATE INDEX job__orphan__idx
    ON job (tenant, modified_at)
    WHERE status = 'running';

-- User-facing job ledger index: it supports filtering by profile and
-- ordering a user's jobs by newest first. No current production query
-- uses it yet; it is kept for the user-facing ledger path.
CREATE INDEX job__profile__idx
    ON job (profile_id, created_at DESC)
    WHERE profile_id IS NOT NULL;

-- Storage GC reference check: it answers whether any job still points
-- to a candidate storage object. Rows without a resource are excluded.
CREATE INDEX job__resource__idx
    ON job (resource_id)
    WHERE resource_id IS NOT NULL;

-- The lost-sweep index of `scheduled` jobs is created by
-- `0159-consumer-generalization`, on `modified_at` (the hand-off time),
-- which is the column the sweep filters by: building it here on
-- `scheduled_at` would only throw it away in the same transaction.

-- Jobs GC expiration: it scans jobs past expires_at while excluding
-- running and retry jobs from the expiration delete.
CREATE INDEX job__expires__idx
    ON job (expires_at)
    WHERE expires_at IS NOT NULL;

-- Jobs GC retention: it finds old internal terminal jobs with no
-- profile, which are eligible for removal after the retention delay.
CREATE INDEX job__retention__idx
    ON job (modified_at)
    WHERE status IN ('completed', 'failed', 'cancelled', 'aborted')
      AND profile_id IS NULL;

-- Append-only job event log: `start`, `progress`, `retry` and `end` rows in
-- insertion order. This is the only source of truth for progress; there is no
-- Redis key and no mutable progress column. The foreign key cascades so
-- deleting a job removes its whole history, and it is DEFERRABLE like every
-- other new reference of the substrate.
CREATE TABLE job_event (
    id         bigserial PRIMARY KEY,
    job_id     uuid NOT NULL REFERENCES job(id) ON DELETE CASCADE DEFERRABLE,
    kind       text NOT NULL
               CHECK (kind IN ('start', 'progress', 'retry', 'end')),
    payload    jsonb NOT NULL DEFAULT '{}',
    created_at timestamptz NOT NULL DEFAULT now()
);

-- Event history read: all events of a job, and the last progress report
-- (`WHERE job_id = ? AND kind = 'progress' ORDER BY created_at DESC LIMIT 1`).
CREATE INDEX job_event__job_kind_created_idx
    ON job_event (job_id, kind, created_at DESC, id DESC);

-- Deletion protection, like the domain tables (`team`, `profile`, the
-- file thumbnail tables). The only intended deleter of `job` rows is the
-- jobs GC, which also marks the referenced storage objects as touched in
-- the same transaction; a stray `DELETE FROM job` would drop the row
-- (and cascade `job_event`) without touching the object, orphaning it
-- forever. The jobs GC disables the guard inside its transaction with
-- `SET LOCAL rules.deletion_protection TO off`, the same escape hatch used
-- by `app.tasks.objects-gc`; `app.jobs/submit` never deletes.
CREATE OR REPLACE TRIGGER deletion_protection__tgr
BEFORE DELETE ON job FOR EACH STATEMENT
  WHEN ((current_setting('rules.deletion_protection', true) IN ('on', '')) OR
        (current_setting('rules.deletion_protection', true) IS NULL))
  EXECUTE PROCEDURE raise_deletion_protection();
COMMIT;

The tables are new and empty and the old version never writes them, so every statement is instant. A failure rolls the block back: re-run it whole instead of completing objects by hand. The block is the whole step, not just the tables: job with its six indexes (dispatcher, orphan, profile, resource, expires, retention), job_event with its history index, and the deletion_protection__tgr trigger. Step 9 checks every one of them.

The old version never touches this table, but the new version needs it to boot.

4. Audit log: columns, reserved range and defaults

Columns first (metadata-only), the reserved low range and the defaults the old web instances need immediately: they keep inserting while the DDL and the backfill run, and the database fills the new columns for them. The old code ignores them.

CREATE SEQUENCE IF NOT EXISTS audit_log_seq_seq;
ALTER TABLE audit_log ADD COLUMN IF NOT EXISTS seq bigint;
ALTER TABLE audit_log ADD COLUMN IF NOT EXISTS xid xid8 NOT NULL DEFAULT '0'::xid8;
ALTER SEQUENCE audit_log_seq_seq OWNED BY audit_log.seq;

-- Reserve the low range for the history: new rows start above the backfill.
-- Only on the first application: a sequence that already left 1 is reserved
-- by the operator or in use, and a re-run must not move it.
SELECT setval('audit_log_seq_seq',
              (SELECT count(*) + 1000000 FROM audit_log),
              false)
 WHERE (SELECT last_value FROM audit_log_seq_seq) = 1
   AND NOT (SELECT is_called FROM audit_log_seq_seq);

ALTER TABLE audit_log ALTER COLUMN seq SET DEFAULT nextval('audit_log_seq_seq');
ALTER TABLE audit_log ALTER COLUMN xid SET DEFAULT pg_current_xact_id();

5. Audit log: backfill

One statement, row locks only (never a table lock), and it filters seq IS NULL, so it can be interrupted and re-run. The archiver is stopped, but the web instances keep inserting; their new rows get seq from the default, so the backfill only touches the history. It sorts by (created_at, id): the (created_at, id) index this branch used belonged to the pending queue and never shipped. On a large table, raise work_mem for the session before running it so the sort stays in memory:

UPDATE audit_log
   SET seq = ordered.rn
  FROM (SELECT id, row_number() OVER (ORDER BY created_at, id) AS rn
          FROM audit_log
         WHERE seq IS NULL) ordered
 WHERE audit_log.id = ordered.id;

6. Audit log: seq and source NOT NULL without a long ACCESS EXCLUSIVE scan

No row can be NULL in seq here: the default was set in step 4. source is nullable since 0064; fill what is left first.

ALTER TABLE audit_log ADD CONSTRAINT audit_log__seq__not_null
    CHECK (seq IS NOT NULL) NOT VALID;
ALTER TABLE audit_log VALIDATE CONSTRAINT audit_log__seq__not_null;
ALTER TABLE audit_log ALTER COLUMN seq SET NOT NULL;
ALTER TABLE audit_log DROP CONSTRAINT audit_log__seq__not_null;

UPDATE audit_log SET source = 'backend' WHERE source IS NULL;

ALTER TABLE audit_log ADD CONSTRAINT audit_log__source__not_null
    CHECK (source IS NOT NULL) NOT VALID;
ALTER TABLE audit_log VALIDATE CONSTRAINT audit_log__source__not_null;
ALTER TABLE audit_log ALTER COLUMN source SET NOT NULL;
ALTER TABLE audit_log DROP CONSTRAINT audit_log__source__not_null;

7. Audit log: cursor index and cursor table

The index with CONCURRENTLY, and the cursor table, which starts empty:

CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS audit_log__xid__seq__idx
    ON audit_log (xid, seq);

CREATE TABLE IF NOT EXISTS audit_log_consumer_cursor (
  consumer   smallint    NOT NULL,
  xid        xid8        NOT NULL,
  seq        bigint      NOT NULL,
  updated_at timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (consumer));

8. Jobs: exclusive and dedupe columns and indexes

The two columns and the two unique indexes, the indexes with CONCURRENTLY:

ALTER TABLE job ADD COLUMN IF NOT EXISTS exclusive boolean NOT NULL DEFAULT false;
ALTER TABLE job ADD COLUMN IF NOT EXISTS dedupe    boolean NOT NULL DEFAULT false;

CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS job__live_exclusive__uniq
    ON job (tenant, name, queue, label)
    WHERE exclusive AND status IN ('new', 'scheduled', 'running', 'retry');

CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS job__new_dedupe__uniq
    ON job (tenant, name, queue, label)
    WHERE dedupe AND status = 'new';

The lost sweep index changes shape: it filters by modified_at, not by scheduled_at. job is not in production yet, so a plain rebuild is enough:

DROP INDEX IF EXISTS job__scheduled__idx;
CREATE INDEX IF NOT EXISTS job__scheduled__idx
    ON job (tenant, modified_at)
 WHERE status = 'scheduled';

If an early 0154 built the cron probe index, drop it too (the submit resolves the overlap at insert time now):

DROP INDEX IF EXISTS job__name_label__idx;

9. Gate before deploying

An interrupted CONCURRENTLY build leaves the index INVALID, and IF NOT EXISTS matches by name, so the migration would accept it and the dedupe/exclusivity would stop enforcing silently. The checks:

SELECT indexname, indexdef FROM pg_indexes
 WHERE indexname IN ('job__live_exclusive__uniq',
                     'job__new_dedupe__uniq',
                     'job__scheduled__idx',
                     'audit_log__xid__seq__idx');

SELECT indexrelid::regclass AS index, indisvalid, indisunique
  FROM pg_index
 WHERE indexrelid IN ('job__live_exclusive__uniq'::regclass,
                      'job__new_dedupe__uniq'::regclass,
                      'audit_log__xid__seq__idx'::regclass);

SELECT column_name, is_nullable
  FROM information_schema.columns
 WHERE table_name = 'audit_log'
   AND column_name IN ('seq', 'xid', 'source')
 ORDER BY 1;

Any index with indisvalid = false must be dropped and rebuilt before the deploy, and the three columns of audit_log must answer NO.

The job side is a gate too: both job tables, the cursor table, the indexes of the check and the trigger must exist before the deploy. A half-applied 0154 or a skipped step 8 would otherwise only surface as a boot failure.

SELECT to_regclass('public.job') IS NOT NULL AS job,
       to_regclass('public.job_event') IS NOT NULL AS job_event,
       to_regclass('public.audit_log_consumer_cursor') IS NOT NULL AS cursor;

SELECT indexname FROM pg_indexes
 WHERE tablename IN ('job', 'job_event')
   AND indexname IN ('job__dispatcher__idx', 'job__orphan__idx',
                     'job__profile__idx', 'job__resource__idx',
                     'job__expires__idx', 'job__retention__idx',
                     'job__scheduled__idx', 'job_event__job_kind_created_idx')
 ORDER BY 1;

SELECT column_name, is_nullable
  FROM information_schema.columns
  WHERE table_name = 'job'
   AND column_name IN ('exclusive', 'dedupe')
  ORDER BY 1;

SELECT tgname FROM pg_trigger
 WHERE tgrelid = to_regclass('public.job')
   AND tgname = 'deletion_protection__tgr'
   AND NOT tgisinternal;

The three tables must answer true, the eight indexes must be listed, the two flags must answer NO, and the trigger must appear. The job__scheduled__idx definition of the first check must read (tenant, modified_at), not (tenant, scheduled_at): step 8 drops and rebuilds it, and CREATE INDEX IF NOT EXISTS matches by name. This is a gate: do not deploy if it fails. The catalog checks are instant; the CHECK NOT VALID / VALIDATE / SET NOT NULL dance already proved there is no NULL, so a count(*) would only scan the table again.

10. After the rollout

Once every instance runs this version and the new consumers have replayed the unarchived buffer, remove disable-audit-log-gc and roll the workers to restore the default GC schedule. The GC deletes rows the consumers have passed and applies the 7-day retention of telemetry shadows. Remove the old archiver flag setting; the new version does not use it.

Registering a consumer after the rollout

The GC reads the cursors again for each batch, but it does not lock out registration. If a consumer registers while a cursor-based delete batch is in progress, it can miss up to one batch per concurrent GC run that already read the previous cursor limit. The 7-day shadow retention deletes independently of cursors.

For a complete replay of the buffer, keep only disable-audit-log-gc on all workers sharing the database. Restart them, verify the effective flag in each startup log, and wait for all queued or running GC jobs to finish before enabling the new consumer. Do not invoke the GC manually during the pause. Confirm the consumer has registered and replayed the buffer before removing disable-audit-log-gc and rolling the workers again. This cannot recover rows the GC already deleted.

The telemetry shadows

The anonymized copies (source LIKE 'telemetry:%') have their own consumer since this version: id 0, flag :telemetry. It ships them with the pace of the old cron (10 000 rows per request, a pass every 3 hours) and the GC deletes them once its cursor has passed them; no task deletes them any more.

  • On the first boot of a worker with :telemetry on, the consumer registers its cursor at (0, 0) and its first pass ships every shadow the buffer holds. While the old version still runs its own cron, the same copy can reach the endpoint twice during the overlap window; telemetry is at-least-once and nothing deduplicates it.

  • The GC has two cursor paths: the full rows up to the oldest cursor of the consumers that read them, and the shadows up to the cursor 0. A slow telemetry pass (three hours) does not retain the events Nexus already archived, and turning telemetry off does not stop the deletion of the full rows.

  • Shadows older than 7 days are deleted by the GC whatever the cursor says, with the endpoint down included. That window only runs with :audit-log-gc on, and it can drop copies that were never sent. It limits local retention, not sending: the consumer can send an older shadow if it reads it before the GC deletes it.

  • To retire the telemetry consumer, turn :telemetry off on every instance first, let the GC run once with the cursor frozen (it deletes the copies the consumer had already passed), and then delete the cursor:

    DELETE FROM audit_log_consumer_cursor WHERE consumer = 0;
    

    The statement deletes no shadow by itself: the copies the consumer had not read wait for the retention window (or a manual delete). Turning :telemetry back on registers the cursor at (0, 0) again, so the first pass replays whatever the buffer still holds and skips the full rows.