From 90a06e982b84c126326143587ee9039e46f38174 Mon Sep 17 00:00:00 2001 From: Andrey Antukh Date: Fri, 20 Nov 2026 11:12:00 +0100 Subject: [PATCH] :recycle: Rework the audit-log consumers and the jobs substrate (#12701) * :recycle: Rework audit-log consumers and jobs substrate Fold the implementation, follow-up fixes, migrations and tests into one code commit. AI-assisted-by: gpt-6-luna * :books: Consolidate 2.20.0 upgrade runbooks Keep the operational upgrade procedure and its supporting runbooks in a separate documentation commit. AI-assisted-by: gpt-6-luna * :recycle: Remove the audit-log flag and derive persistence at boot Persistence now follows the consumer flags instead of a separate switch. app.config computes cf/audit-persist? once at boot from :nexus and :admin-console, and every write site reads that value, so request paths no longer check flags per event. With no consumer enabled no full rows are written; telemetry shadows and webhook deliveries keep their own flags. The frontend wire signal stays :audit-log as a derived value. Update the 2.20 runbooks, memories and the devenv flags to match, and migrate the tests that used to toggle :audit-log directly. AI-assisted-by: hy4-flash-free * :bug: Accept error-reporting batches and precompute sink flags push-audit-events now accepts crash batches with :error-reporting alone, so the frontend signal from get-environment-data matches a real destination. The reporters receive unhandled-exception events without storing any row. All per-event sink checks read booleans computed once at boot: webhooks, telemetry and error reporting join audit-persist? in app.config. The test fixture and with-audit-config rebind all four values together, and the tests that toggled those flags directly were migrated. Address F2 of the review of the previous commit. F1 stays deferred to a separate plan. AI-assisted-by: hy4-flash-free * :lipstick: Match the layout cljfmt expects in the flags set The first comment of the varia set sat on its own line after the opening delimiter, and cljfmt joins such a comment to the delimiter line, so the Common Tests lint step failed on common/src/app/common/flags.cljc. Keep the comment on the opening line, the same shape the login and email sets already use. Note in mem:scripts/ci that the backend lint lints ../common/src but does not format-check it, so common has to be passed to scripts/ci as well. AI-assisted-by: deepseek-v4-flash --- .serena/memories/audit-log.md | 95 +- .serena/memories/backend/core.md | 16 +- .serena/memories/backend/storage.md | 2 +- .serena/memories/backend/subtleties.md | 23 +- .serena/memories/backend/testing.md | 24 +- .serena/memories/clojure/core.md | 2 +- .serena/memories/devenv/core.md | 4 +- .serena/memories/memory-maintenance.md | 2 +- .serena/memories/prod-infra/core.md | 29 +- .serena/memories/scripts/ci.md | 4 + AGENTS.md | 7 + backend/scripts/_env | 1 - backend/src/app/config.clj | 17 +- backend/src/app/db.clj | 59 + backend/src/app/jobs.clj | 359 ++-- backend/src/app/jobs/metrics.clj | 170 +- backend/src/app/loggers/audit.clj | 138 +- .../src/app/loggers/audit/consumer_task.clj | 689 ------- backend/src/app/loggers/audit/consumers.clj | 531 ++++++ .../loggers/audit/consumers/admin_console.clj | 318 ++++ .../src/app/loggers/audit/consumers/nexus.clj | 148 ++ backend/src/app/loggers/audit/gc.clj | 160 ++ backend/src/app/main.clj | 236 ++- backend/src/app/migrations.clj | 8 +- .../app/migrations/sql/0154-add-job-table.sql | 28 +- .../sql/0157-add-audit-log-pending-table.sql | 39 - .../0159-add-file-data-storage-object-id.sql | 12 +- .../sql/0159-consumer-generalization.sql | 216 +++ backend/src/app/nitrate.clj | 20 +- backend/src/app/redis.clj | 9 + backend/src/app/rpc.clj | 13 +- backend/src/app/rpc/commands/audit.clj | 48 +- backend/src/app/rpc/commands/files.clj | 29 +- backend/src/app/rpc/commands/files_update.clj | 49 +- backend/src/app/rpc/commands/management.clj | 63 +- backend/src/app/rpc/commands/profile.clj | 19 +- backend/src/app/rpc/commands/teams.clj | 2 +- backend/src/app/srepl/main.clj | 105 +- backend/src/app/tasks/delete_object.clj | 117 +- backend/src/app/tasks/file_gc.clj | 16 +- backend/src/app/tasks/objects_gc.clj | 2 +- backend/src/app/tasks/telemetry.clj | 45 +- backend/src/app/worker/cron.clj | 13 +- backend/src/app/worker/dispatcher.clj | 215 ++- backend/src/app/worker/runner.clj | 11 +- backend/test/backend_tests/auth_oidc_test.clj | 6 +- backend/test/backend_tests/config_test.clj | 16 + backend/test/backend_tests/db_test.clj | 90 + .../test/backend_tests/delete_object_test.clj | 21 +- backend/test/backend_tests/helpers.clj | 41 +- .../test/backend_tests/jobs_metrics_test.clj | 252 +-- backend/test/backend_tests/jobs_test.clj | 464 +++-- .../loggers_audit_consumer_task_test.clj | 837 --------- .../loggers_audit_consumers_test.clj | 1479 +++++++++++++++ .../test/backend_tests/migrations_test.clj | 253 ++- .../nitrate_ingest_audit_log_test.clj | 24 + .../test/backend_tests/objects_gc_test.clj | 16 +- backend/test/backend_tests/rpc_audit_test.clj | 178 +- .../test/backend_tests/rpc_binfile_test.clj | 2 +- backend/test/backend_tests/rpc_file_test.clj | 226 ++- .../backend_tests/rpc_file_update_test.clj | 54 + .../rpc_management_nitrate_test.clj | 10 +- .../backend_tests/rpc_management_test.clj | 69 +- backend/test/backend_tests/rpc_team_test.clj | 14 +- .../test/backend_tests/srepl_main_test.clj | 121 +- .../backend_tests/tasks_telemetry_test.clj | 157 +- .../backend_tests/worker_dispatcher_test.clj | 245 ++- .../test/backend_tests/worker_runner_test.clj | 17 +- backend/test/backend_tests/worker_test.clj | 49 +- common/scripts/repl | 2 +- common/src/app/common/flags.cljc | 4 +- docs/technical-guide/developer/devenv.md | 11 +- operations/2.20.0-consumer-generalization.md | 471 +++++ operations/2.20.0-file-change-partition.md | 10 +- .../2.20.0-file-data-storage-object-id.md | 383 +++- operations/2.20.0-unified.md | 1613 +++++++++++++++++ operations/README.md | 1 + 77 files changed, 8290 insertions(+), 2959 deletions(-) delete mode 100644 backend/src/app/loggers/audit/consumer_task.clj create mode 100644 backend/src/app/loggers/audit/consumers.clj create mode 100644 backend/src/app/loggers/audit/consumers/admin_console.clj create mode 100644 backend/src/app/loggers/audit/consumers/nexus.clj create mode 100644 backend/src/app/loggers/audit/gc.clj delete mode 100644 backend/src/app/migrations/sql/0157-add-audit-log-pending-table.sql create mode 100644 backend/src/app/migrations/sql/0159-consumer-generalization.sql delete mode 100644 backend/test/backend_tests/loggers_audit_consumer_task_test.clj create mode 100644 backend/test/backend_tests/loggers_audit_consumers_test.clj create mode 100644 operations/2.20.0-consumer-generalization.md create mode 100644 operations/2.20.0-unified.md diff --git a/.serena/memories/audit-log.md b/.serena/memories/audit-log.md index dbbe8eef44..0d36fa806d 100644 --- a/.serena/memories/audit-log.md +++ b/.serena/memories/audit-log.md @@ -12,14 +12,15 @@ The system has: - **Three producers** (they create events): the Penpot backend (RPC layer), the Penpot frontend (browser app), and the Nitrate admin-console, which sends both server-side and client-side events. - **One central buffer**: the Postgres `audit_log` table in Penpot. It is temporary: it holds events for minutes/hours until they are archived, then they are deleted. -- **Four consumers** (they read events): webhooks, error reporters, telemetry, and the external archives in Nexus and in the admin console. +- **Consumers** (they read events): webhooks, error reporters, telemetry, the external archives in Nexus and in the admin console. - **Two long-term archives**: **Nexus**, a separate service that stores events for the long term, and the **admin console** (Nitrate), which ingests its own allowlisted events through a second archive job (see section 12). ``` Penpot frontend ─ POST push-audit-events (public) ─────────▶┐ Nitrate admin-console (browser) ─ POST push-audit-events ──▶│ -Nitrate admin-console (server) ─ POST push-audit-events ──▶│ ─▶ audit_log table (buffer) ──┬─ cron 0s every 5m ──▶ Nexus (archive) -Penpot backend (RPC wrap-audit) ─ submit ───────────────────┘ │ └─ cron 30s every 5m ─▶ admin console (archive) +Nitrate admin-console (server) ─ POST push-audit-events ──▶│ ─▶ audit_log table (buffer) ──┬─ nexus consumer ───▶ Nexus +Penpot backend (RPC wrap-audit) ─ submit ───────────────────┘ │ ├─ admin-console ───▶ admin console + │ └─ gc ───────────▶ delete drained rows │ ├──▶ webhooks │ ├──▶ error reporters └──▶ telemetry (shadow rows) @@ -54,19 +55,23 @@ Main columns (migration `backend/src/app/migrations/sql/0054-add-audit-log-table - `created_at` timestamptz NOT NULL default `now()`: **server time**, the source of truth. - `tracked_at` timestamptz default `now()`: what the client claimed; fixed on ingest. - `profile_id` uuid NOT NULL: who did it. -- `source` text: `backend` / `frontend` for full rows, `telemetry:backend` / `telemetry:frontend` for anonymized copies (see section 10). +- `source` text, NOT NULL (from `0159`, which backfills what is left to `backend`): `backend` / `frontend` for full rows, `telemetry:backend` / `telemetry:frontend` for anonymized copies (see section 10). - `ip_addr` inet: real IP (or `0.0.0.0` on telemetry rows). - `props` / `context` jsonb holding transit-encoded maps: action data and metadata (versions, browser, request-id...). -- No `archived_at`: the `audit_log_pending` queue (migration `0157`, consumer `1` = Nexus, `2` = admin console, ids never reused) tracks what is left. Every record is born pending only for the consumers enabled at insert time (never a fixed both): cheaper on single-consumer hosts and true to the design doc. Consequence, accepted on purpose: rows born with no send flag on get no pendings, so enabling archiving later does not pick them up — the sweep deletes them unarchived. +- `seq` bigint (`bigserial`): the insertion order, from a sequence (no WAL cost). +- `xid` xid8: the transaction that wrote the row (`pg_current_xact_id()` as the column default). Together with `seq` it is the position a consumer reads from; `xid8` is 64 bits and does not wrap (the row `xmin` does). +- The legacy `archived_at` and the indexes that served the old archiver stay until a later cleanup; the new code ignores them. The drain reads no pending queue: each consumer keeps its position in `audit_log_consumer_cursor` (migration `0159-consumer-generalization`, consumer `1` = Nexus, `2` = admin console; ids are never reused, and `3` is reserved for the team-storage consumer that ships in a follow-up). The pending queue never shipped: this release does not create `audit_log_pending`; the migration drops the table if a pre-release database has it. The producer only inserts the row; it does not know the consumers. A consumer with no row registers its cursor at `(0, 0)` when its component starts (`ensure-cursor`), before its first pass, so the first deploy **replays** the buffer (the legacy archiver and its GC are stopped and the already-archived rows are deleted before deploying), instead of skipping the unarchived backlog; enabling a consumer later starts from whatever the buffer holds; with no consumer at all nothing is written, so there is no backlog to replay. `ensure-cursor` starts the cursor below any real position; the cut is applied per chunk when reading, so the first pass drains the buffer that exists and never steps over a row whose transaction was still open. -Indexes (each consumer has its own, so a slow consumer never blocks the others): +Position and cut: a consumer reads rows with `(xid, seq) > cursor` and `xid <= pg_snapshot_xmin(pg_current_snapshot())` (the oldest transaction still in flight), in `(xid, seq)` order. The cut is exact: a row of an open transaction is invisible until it commits, and the cursor never steps over it (a `seq`-only cursor lost those rows). A transaction that inserts many rows keeps them together: `seq` orders them inside the same `xid`. The cursor points at the last row the consumer processed, so the GC deletes `<= cursor`. The cut is conservative beyond `audit_log`: `pg_snapshot_xmin` is the oldest transaction of the whole database, so any long write transaction (a binfile import runs in one for up to 30 minutes) pauses every consumer and the GC until it commits, and a slow destination pauses them the same way because the chunk sends inside its transaction and the `FOR UPDATE` on the cursor row assigns an xid; nothing is lost or skipped, and `penpot_audit_consumer_lag_events` shows it. -- `audit_log_pending` PK `(consumer, audit_log_id)`: each consumer owns its queue at constant cost, even with millions pending. -- `audit_log (created_at, id)` (migration `0157`): the chronological drain order (`ORDER BY created_at ASC, id ASC`; same total order for both consumers, so concurrent drains cannot deadlock). -- `audit_log_pending (audit_log_id)`: serves the "is anything still pending?" check of the final delete. -- `(source, created_at)`: used by the telemetry scan. +Indexes: -Life of a row: `INSERT` (row plus one pending per enabled consumer, same transaction; a rollback takes both) → (every 5 min, Nexus at second 0 and admin console at second 30) each archive job sends its own 128-row batches oldest-first and deletes its pendings in the same transaction → whoever deletes the last pending deletes the row. A batch the destination refuses rolls back and stays for the next tick. Pendings of consumers that are off are reaped at the start of each job run (covers later deactivations), and orphan rows are swept at the end of each run, so there is no separate GC task. The FK is plain `DEFERRABLE` (never `INITIALLY DEFERRED`): every deleter removes pendings before rows, including the telemetry task's own deletes. +- unique `audit_log (xid, seq)` (`audit_log__xid__seq__idx`, migration `0159-consumer-generalization`): serves the range and the order of the position scan (both predicates are `Index Cond`; the chunk also projects columns, so it is not an Index Only Scan). +- `audit_log_consumer_cursor` PK `(consumer)`: one row per consumer, locked `FOR UPDATE` during the chunk. +- `(source, created_at)` (`audit_log__source__created_at__idx`): narrows the telemetry scan by source; telemetry then sorts by `created_at`. The `(created_at, id)` index belonged to the pending queue and never shipped, so this release does not build it (the same path 2.18 had). +- The legacy `archived_at` and the indexes that served it (`audit_log__created_at__idx`, `audit_log__archived_at__idx`, `audit_log_id_archived_at_idx`) stay: `0159` does not drop them. They are dead after the rollout and a later release removes them; the column/index drops are catalog operations, not a rewrite. + +Life of a row: `INSERT` (one row, no queue) → each consumer's pass reads its own 128-row chunks from its cursor up to the frontier it took at its start, sends them and advances the cursor in the same transaction (a refusal rolls both back) → the GC job (`:audit-log-gc`) deletes the rows up to and including the oldest cursor in `audit_log_consumer_cursor`, in batches, skipping telemetry shadows (the telemetry task owns them: it deletes them after sending and purges them after 7 days). A consumer that is off keeps its cursor row and holds the GC at that position; when enabled again it rejoins from its position. To stop waiting for it, its cursor row must be deleted (retirement; no procedure ships yet). A consumer registers its cursor at `(0, 0)` when its component starts (`ensure-cursor`), before its first pass, below any real position, so the first deploy replays the unarchived backlog and enabling a consumer later starts from whatever the buffer holds; with no consumer at all nothing is written, so there is no backlog to replay. The GC treats a missing row as position 0 and deletes nothing for that run; with no cursor at all it deletes nothing (the `:audit-log-gc` flag is the switch). ## 4. Anatomy of an event @@ -79,7 +84,6 @@ Every event ends up with this shape (schema `schema:event` in `backend/src/app/l - `props`: action data (team/project/file ids, visited route...). **Never secrets** (see `clean-props`). - `context`: metadata about how/when it was captured (backend/frontend version, user-agent, locale, browser, `request-id`, `initiator`, `event-origin`...). See section 8 for the full field guide, including `initiator` (which app started it: `"app"` means Penpot itself). - `source`: `backend`, `frontend`, `telemetry:backend`, `telemetry:frontend`. Nitrate admin-console server events land as `backend`; its browser events land as `frontend` (see section 7). -- `source`: `backend`, `frontend`, `telemetry:backend`, `telemetry:frontend`. - `tracked-at` / `created-at`: claimed time and server time. - Memory-only (never stored): `::webhooks/event?`, `::webhooks/batch-key`, `::webhooks/batch-timeout` for the webhook fan-out. @@ -107,7 +111,7 @@ Frontend `navigate` example: ## 5. Backend events: the happy path -Most backend events need **no manual code**. The `wrap-audit` wrapper in `backend/src/app/rpc.clj` runs after every RPC command when `:webhooks`, `:audit-log`, or `:telemetry` is on (unless the command sets `::audit/skip`). It builds the event with `prepare-rpc-event` and passes it to `audit/submit`. +Most backend events need **no manual code**. The `wrap-audit` wrapper in `backend/src/app/rpc.clj` runs after every RPC command when `:webhooks`, the boot-computed persistence value, or `:telemetry` is on (unless the command sets `::audit/skip`). It builds the event with `prepare-rpc-event` and passes it to `audit/submit`. Each command can customize through result metadata (`rph/with-meta`): @@ -118,34 +122,36 @@ Each command can customize through result metadata (`rph/with-meta`): Critical `profile-id` INVARIANT: the event belongs to **whoever made the request**. It resolves as `::audit/profile-id` metadata → `::rpc/profile-id` → `uuid/zero`. It is **never** taken from the response: some results carry someone else's `:profile-id` (for example `get-error-report` returns the report with its content merged; `verify-token` on an invitation returns the inviter), and using it would blame the wrong person. Commands with `::rpc/auth false` (`login-*`, `register-profile`, `create-demo-profile`, `verify-token`, `prepare-register-profile`) have no caller to fall back on, so they **must** set `::audit/profile-id` in the metadata or the event lands on `uuid/zero`. -The `::audit/profile-id` override goes through `coerce-profile-id`: a string is parsed, anything that cannot become a uuid is dropped (with a warning) and the event falls back to the caller. Do not pass the value raw: `schema:event` needs a uuid and `submit*` swallows the validation error (it only logs, the RPC still succeeds), so a non-uuid override **loses the row silently**. Hand-built events with `event-from-rpc-params` + `submit` (clone template, accept team invitation, `management/nitrate` push-audit-events) do **not** go through that coercion: they must already carry a uuid. +The `::audit/profile-id` override goes through `coerce-profile-id`: a string is parsed, anything that cannot become a uuid is dropped (with a warning) and the event falls back to the caller. Do not pass the value raw: `schema:event` needs a uuid and `submit*` **propagates** the validation error, and `wrap-audit` runs after the command's transaction commits (moving it inside is project P8 in `TODO.md`), so a non-uuid override would fail the response over an already committed action instead of only losing the row. Hand-built events with `event-from-rpc-params` + `submit` (clone template, accept team invitation, `management/nitrate` push-audit-events) do **not** go through that coercion: they must already carry a uuid. ### Team id and entity ids (projection data) The event stream is meant to let a consumer (Nexus) rebuild what it needs **without joining production tables**, so events carry the ids a projection keys on: - **Team id enrichment.** After `clean-props`, `prepare-rpc-event` calls `enrich-team-id`: when props have no `:team-id` but carry a `:project-id` or `:file-id`, it resolves the team with a local query and adds `:team-id`. It never resolves from `:id` (read commands put the entity id there, and that would add a query to every read), and it skips read events entirely (`get-*` names plus `has-file-libraries`: their events carry no consumer state). Commands that already loaded the team add `:team-id` themselves in `::audit/props` (`delete-file`, comments, media upload, `delete/update-webhook`, `clone-template`, ...), so the resolver does not run. -- **Membership.** `add-profile-to-team!` emits `add-team-member` (props `:team-id`, `:member-id`, `:role`, `:is-owner`, `:is-admin`, `:can-edit`) whenever it actually inserts a row, only with `:audit-log` on, inside the caller's transaction. It covers the flows that add a member with no event of their own (`create-team` owner, invitee added directly when `:email-verification` is off, approved join-request, duplicated team). With `on-conflict-do-nothing?` a conflict returns nil and emits nothing, so an existing membership is reported once. The event belongs to the actor: the two flows where the owner adds somebody else (`create-invitation`, `add-member-to-team`) pass `:actor-id` through a separate `event-opts` arg (never in `params`, which is inserted as-is); everywhere else actor and member coincide. `:role` is derived in the emitter from the flags (inverse of `permissions-for-role`: owner > admin > editor > viewer) because the table has no role column; request context travels in `event-opts` as `:rpc-params`. +- **Membership.** `add-profile-to-team!` emits `add-team-member` (props `:team-id`, `:member-id`, `:role`, `:is-owner`, `:is-admin`, `:can-edit`) whenever it actually inserts a row, only when full rows are persisted, inside the caller's transaction. It covers the flows that add a member with no event of their own (`create-team` owner, invitee added directly when `:email-verification` is off, approved join-request, duplicated team). With `on-conflict-do-nothing?` a conflict returns nil and emits nothing, so an existing membership is reported once. The event belongs to the actor: the two flows where the owner adds somebody else (`create-invitation`, `add-member-to-team`) pass `:actor-id` through a separate `event-opts` arg (never in `params`, which is inserted as-is); everywhere else actor and member coincide. `:role` is derived in the emitter from the flags (inverse of `permissions-for-role`: owner > admin > editor > viewer) because the table has no role column; request context travels in `event-opts` as `:rpc-params`. - **Created entity ids.** `create-project` and `create-file` add the new row's `:id` (and `:project-id`/`:team-id`) to `::audit/props`, so a projection can key it. -- **Bulk operations.** `clone-template`, `import-binfile` and `duplicate-project` create entities through internal calls that emit nothing on their own; they emit a `create-file` (and `create-project`) per entity (only with `:audit-log`). The import paths go through `audit/submit-file-created-event`, which reads the light columns (`id`, `name`, `project-id`, timestamps, `revn`) back from the row — the file must really exist, a phantom id emits nothing. `duplicate-project` already holds the rows and uses `audit/submit-create-event` directly. Both binfile versions return the same shape from `import-files!` (`{:file-ids [...]}`). +- **Change ids.** `update-file` carries the id of the change row (xlog) it created in `:file-change-id`: the event says that a file was saved, but what changed lives in that row. A consumer that needs it must read it with `::db/remove-deleted false` (the row is born with its `deleted_at` in the future). A save writes two change rows (the change itself, without a label, and the snapshot of the previous state, with one) and the event points at the change one. A save that ends fully rejected (`:rejected` and no kept prefix) writes no row and the prop is absent: an id that resolves to nothing cannot be told apart from one the GC already purged, so do not expect it always. The event also carries `:outcome`: `saved` (every change written), `partial` (the accepted prefix written and the rest reported in the response, `:file-change-id` present) or `rejected` (nothing written, `:file-change-id` absent, `:modified-at` set to the row's persisted value instead of the request time). A consumer must look at `:outcome` before treating `update-file` as a state change. +- **Bulk operations.** `clone-template`, `import-binfile` and `duplicate-project` create entities through internal calls that emit nothing on their own; they emit a `create-file` (and `create-project`) per entity (only when full rows are persisted). The import paths go through `audit/submit-file-created-event`, which reads the light columns (`id`, `name`, `project-id`, timestamps, `revn`) back from the row — the file must really exist, a phantom id emits nothing. `duplicate-project` already holds the rows and uses `audit/submit-create-event` directly. Both binfile versions return the same shape from `import-files!` (`{:file-ids [...]}`). - **Deletes.** `delete-team` still returns nil but carries the deleted row (`:id`, `:name`, `:created-at`, `:modified-at`, `:deleted-at`) in `::audit/props`, like `delete-project`. `delete-profile` carries `:id`, `:email`, `:fullname`, `:created-at`, `:modified-at` and `:deleted-at` (without this its props are empty: every param is namespaced and `clean-props` drops them). `create-team` adds `:created-at` and `:modified-at`. `delete-team-member` needs nothing: the delete is physical and the params already carry `:team-id`/`:member-id`. -- **Hard deletes.** `objects-gc` emits one `hard-delete-objects` event per purged chunk, carrying only that chunk's entity key (`:team-ids`, `:project-ids` or `:file-ids`; one chunk transaction purges a single entity, so one event can never span sections) with context `{:triggered-by "objects-gc"}`, only with `:audit-log` on. No other GC path emits (profiles, fonts, thumbnails); those purges stay silent. -- **SREPL deletes.** `delete-profile!` carries the subject id in props (it used to emit nothing usable); the event profile stays zero because SREPL has no actor. `delete-team!` props already carried the id. Both travel as actions with a `triggered-by: srepl` context. -- **SREPL restores.** `restore-profile!` carries the full row (with id) in props with a zero event profile; `restore-team!` and `restore-project!` already carried their ids. The consumer resolves the subject from props by a uniform rule, with no SREPL special case. -- **Cascades.** The `:delete-object` worker task emits one `delete-object-with-cascade` event per execution, type `action`, with `:object`/`:id`/`:deleted-at` plus the affected ids nested under `:cascade` (`:team-ids`, `:project-ids`, `:file-ids`), in the same transaction. The handler submits unconditionally (`submit` fans out per flag, so no `:audit-log` re-check); the `audit_log` row itself is still only written with `:audit-log` on. Direct calls (objects-gc) and snapshot cascades stay silent on purpose. +- **Moves.** `move-files` and `move-project` carry `:source-team-id` and `:dest-team-id`. A `move-files` that mixes files from more than one source team is refused (`:cant-move-from-multiple-teams`), so the single origin in the props is always correct. +- **Hard deletes.** `objects-gc` emits one `hard-delete-objects` event per purged chunk, carrying only that chunk's entity key (`:team-ids`, `:project-ids` or `:file-ids`; one chunk transaction purges a single entity, so one event can never span sections), with context `{:triggered-by "objects-gc"}`, only when full rows are persisted. No extra team-id resolution: the event carries the key of the purge, nothing else. No other GC path emits (profiles, fonts, thumbnails); those purges stay silent. `file-gc` is the exception: it emits `file-gc` (`action`, `triggered-by: cron-task`) with the cleaned `:file-id` and its `:team-id`, so a consumer can key on it without joining production tables. +- **SREPL deletes.** `delete-profile!` carries the subject id in props (it used to emit nothing usable); the event profile stays zero because SREPL has no actor. `delete-team!` props already carried the id, and `delete-file!` adds its `:team-id` so a consumer can key on it. All travel as actions with a `triggered-by: srepl` context. +- **SREPL restores.** `restore-profile!` carries the full row (with id) in props with a zero event profile, plus `:team-ids` with the teams its cascade took (only those: a team the owner deleted earlier stays deleted); `restore-team!` carries its `:team-id` and `restore-project!` its id. The consumer resolves the subject from props by a uniform rule, with no SREPL special case. +- **Cascades.** The `:delete-object` worker task emits one `delete-object-with-cascade` event per execution, type `action`, with `:object`/`:id`/`:deleted-at`, flat `:team-ids` resolved at emission plus the affected ids nested under `:cascade` (`:team-ids`, `:project-ids`, `:file-ids`), in the same transaction. The handler submits unconditionally (`submit` fans out per flag, so no persistence re-check); the `audit_log` row itself is still only written when full rows are persisted. Direct calls (objects-gc) and snapshot cascades stay silent on purpose. - **Ownership.** `create-team-with-invitations` carries the new team `:id` like `create-team`; `update-team-member-role` names the demoted member in `:prev-owner-id` on owner transfers. Entry points: -- `submit`: the normal one (fills defaults, validates, runs inside `tx-run!`, never breaks the RPC). -- `insert`: low level, for CLI helpers and the webhook subsystem; direct write, no webhook/telemetry fan-out, silent unless `:audit-log` is on. +- `submit`: the normal one (fills defaults, validates, runs inside `tx-run!`). A failure **propagates** (a lost event is a stale consumer, and hiding it made that invisible), and what it costs depends on where the caller runs: `wrap-audit` runs **after** the command's transaction commits, so a failure there cannot undo the action and the client gets an error over already committed work (moving the event inside the transaction is project P8 in `TODO.md`); a `submit` made inside the command's own transaction (the membership emitter, the `:delete-object` handler) throws through it and rolls the action back with the event. +- `insert`: low level, for CLI helpers and the webhook subsystem; direct write, no webhook/telemetry fan-out, silent unless full rows are persisted. - Boot emits `trigger/instance-start` from `setup/props` so every restart stays visible. Cases already covered by tests, do not break: `accept-organization-invitation` tells its two flows apart **with props, not with origin**: `:organization-member-add-source` is `"direct-organization-invitation"` or `"team-invitation"`, and `:belongs-to-team-on-add` repeats it as a boolean. Do not add a third prop. The `context` `:event-origin` belongs to the browser (`make-data-event` sets it and it is on the frontend allowlist; `safe-backend-context-keys` has no `:event-origin`, so a backend event has nowhere to put one). ## 6. Penpot frontend events: what is collected, with what guarantees -The browser **never writes to the table**; it POSTs transit batches to `push-audit-events` (`backend/src/app/rpc/commands/audit.clj`), which stamps server `id`, session `profile-id`, request IP, server `created-at`, and server-side `:initiator` (from the `x-client` header, see section 8; anything the client sends as `initiator` is overwritten), and distrusts the client clock (future or >1h-late `tracked-at` is reset, the original kept in `context.original-tracked-at`). Without `:audit-log`/`:telemetry`, or on a read-only pool, the endpoint is a no-op. +The browser **never writes to the table**; it POSTs transit batches to `push-audit-events` (`backend/src/app/rpc/commands/audit.clj`), which stamps server `id`, session `profile-id`, request IP, server `created-at`, and server-side `:initiator` (from the `x-client` header, see section 8; anything the client sends as `initiator` is overwritten), and distrusts the client clock (future or >1h-late `tracked-at` is reset, the original kept in `context.original-tracked-at`). Without persistence, `:telemetry` or `:error-reporting`, or on a read-only pool, the endpoint is a no-op. With `:error-reporting` alone the batch is accepted only to feed the reporters: no full row and no telemetry shadow is written. Valid types (`schema:frontend-event`): `"action"`, `"identify"`, `"trigger"`. Names must match `#"[\d\w-]{1,50}"` (max 250). `props` is a free map with no schema: **tests are the only guard of the contract** (see section 13). @@ -153,7 +159,7 @@ What they are for: usage metrics (navigation, editing), workspace file stats, Ni How the browser collects (`frontend/src/app/main/data/event.cljs`), most to least important: -1. **On-demand start**: `initialize` asks the backend with `get-environment-data` (old `get-enabled-flags` is deprecated) and only starts on `:audit-log` or `:telemetry`. On RPC failure it still starts in telemetry mode (the backend will reject if needed). +1. **On-demand start**: `initialize` asks the backend with `get-environment-data` (old `get-enabled-flags` is deprecated) and only starts on the derived `:audit-log` wire signal or `:telemetry` (see section 13). On RPC failure it still starts in telemetry mode (the backend will reject if needed). 2. **Potok translation**: `make-proto-event` (events with the `Event` protocol) and `make-data-event` (data-events with `::name`) turn the action into `{type, name, props, context}`. `simplify-props` flattens complex values (maps → `:placeholder/map`, and so on) and drops namespaced keys and nils. Context gains `:event-origin`, `:event-namespace`, `:event-symbol`. 3. **Capped buffer**: queue of max 1024 events (`append-to-buffer` drops beyond that); every 2s debounce (or on logout, or `::force-persist`) a chunk of max 100 is taken, filtered to the current profile, and sent fire-and-forget to `api/main/methods/push-audit-events` as transit. On confirm, the chunk leaves the buffer (`::chunk-persisted`). 4. **Performance noise**: `PerformanceObserver` (`event`/`longtask`) plus blocking measurement with `scheduler.postTask` create `trigger` events (`performance-blocking-event`, `user-input`, `long-task`) with 1s debounce and 1s threshold. Info only, not audit. @@ -273,19 +279,19 @@ Nexus dispatches on `:type`: `"telemetry-events"` batches are coerced and insert - **No personal data is expected.** Profile ids are random UUIDs with no PII, so the telemetry consumer needs no login guard; grouping is by `instance`, not by person (see posthog-telemetry below). - **The endpoint always answers 200.** Failures are swallowed and only counted (`nexus_telemetry_errors_total`), so a bad batch never breaks the sending instance; successes are counted (`nexus_telemetry_events_total`, `nexus_telemetry_reports_total`). A body the parser cannot read never reaches the handler: it answers 400 and counts in `nexus_http_malformed_requests_total`. Old instances may send a fressian+zstd base64 blob instead of a plain vector; Nexus decodes both (50 MiB cap against decompression bombs). -## 12. Archive to Nexus and to the admin console +## 12. The cursor consumers: Nexus and the admin console ### How it works today -Every 5 min each archive job drains its own queue (`:audit-log-consumer-nexus` on flag `:nexus`, cron `0 */5 * * * ?`; `:audit-log-consumer-admin-console` on flag `:admin-console`, cron `30 */5 * * * ?`, 30s offset so the two drains never start on the same second), in 128-row chunks taken oldest-first (`ORDER BY created_at ASC, id ASC`) with `FOR UPDATE OF audit_log` (`SET LOCAL lock_timeout = '30s'`) inside the same transaction that sends and completes them, with 100 ms pauses between chunks. Batches arrive at Nexus in chronological order (it numbers by arrival for order-sensitive consumers). Nexus receives transit `{:events [...]}` with `x-shared-key: "nexus "` (`:nexus-shared-key`, or derived from the instance secret) and `origin` = public-uri; flag on but no URI raises `:task-not-configured`. The admin console receives only its allowlisted event names (enriched with team names, props emails, and optional top-level `:profile-email` / `:profile-name` from the actor `:profile-id`); anything filtered out counts as done for the admin-console queue. A chunk whose destination refuses it (Nexus non-204 raises `:consumer-chunk-refused`) or throws (transport failure, admin-console error) rolls back and stays for the next tick, and the job aborts as failed: it archives what it can, and the cron retries the rest. Committed chunks stay sent. The send fns return counts or throw carrying the attempted count as `:events`; `run-single-consumer` records the `failed` metric from it, outside every chunk transaction (a local DB failure carries no `:events` and stays uncounted — it is not a destination failure). No `warn` before the throw: the runner reports the aborted job. Each job touches only its own queue, so a fallen Nexus never starves the admin console (nor vice versa). +Each consumer is a **loop in its own thread**, not a job (nexus on flag `:nexus`, admin console on `:admin-console`; `:audit-log-gc` stays a cron job on flag `:audit-log-gc`). The pass first takes the consumer's session advisory lock (`pg_try_advisory_lock(730000, consumer id)`, the two-key form on a pooled connection held until the pass ends: a lock space of its own, so it can never collide with the one-key lock `update-file` takes from the file id), so one backend drains that consumer at a time once every instance runs this version (a rolling deploy still has the old job-based instance, which does not know the tick; the two versions do not serialize each other, see below). The pass runs entirely on that connection (frontier, chunk transactions and lag gauge), so the pool must admit at least one connection per enabled consumer; a pass that finds it taken answers `::consumer/busy` (`:app.loggers.audit.consumers/busy`) and the loop sleeps `sleep-ms` (5 s) without touching the data. A pass whose consumer flag is off answers `::consumer/disabled` before borrowing a connection: the loop is gated at boot, but a direct caller (REPL, helper) must not send or advance a cursor for a disabled consumer, and that pass records no metric. Holding the lock it reads the **frontier** of the pass (`xid <= pg_snapshot_xmin(pg_current_snapshot())`, ordered `xid DESC, seq DESC`, first row) and drains chunks only up to it, so the pass always ends and every chunk but the last is 128 rows; a row committed while the pass runs waits for the next one. The position is the pair `(xid, seq)` stored in `audit_log_consumer_cursor`; each chunk locks that row `FOR UPDATE` (`SET LOCAL lock_timeout = '30s'`), reads 128 rows with `(xid, seq) > cursor AND (xid, seq) <= frontier AND xid <= pg_snapshot_xmin(pg_current_snapshot())` in `(xid, seq)` order, sends them, and advances the cursor in the same transaction. The row lock only serializes passes of the same consumer (the cursor row takes part in no FK); no `audit_log` row is locked, so the transports never wait for each other. On the first deploy each consumer has no cursor and registers it at `(0, 0)` at boot (`ensure-cursor`, before its first pass; after stopping the legacy archiver and its GC and deleting the rows it had already archived), so the backlog is replayed instead of dropped. Batches arrive at Nexus in insertion order (it numbers by arrival for order-sensitive consumers). Nexus receives transit `{:events [...]}` with `x-shared-key: "nexus "` (`:nexus-shared-key`, or derived from the instance secret) and `origin` = public-uri; flag on but no URI raises `:task-not-configured`. The admin console receives only its allowlisted event names (enriched with team names, props emails, and optional top-level `:profile-email` / `:profile-name` from the actor `:profile-id`), excluding telemetry shadows; the rows outside its filter count as `skipped` and the cursor passes over them (a selective consumer that finds nothing advances to the end of the chunk, so it never blocks the GC). A chunk whose destination refuses it (Nexus non-204 raises `:consumer-chunk-refused`) or throws (transport failure, admin-console error) rolls back the cursor advance and stays for the next pass: the pass aborts with the chunks it already committed, and the loop logs the failure and retries after its pause. Committed chunks stay sent. The send fns return counts or throw carrying the attempted count as `:events`; `drain-consumer` records the `failed` metric from it, outside every chunk transaction (a local DB failure carries no `:events` and stays uncounted — it is not a destination failure). No `warn` before the throw: the loop reports the aborted pass. Each consumer has its own cursor, so a fallen Nexus never starves the admin console (nor vice versa). -A chunk is **one POST** and the task advances only on 204, so a single oversized event wedges the whole queue: a crash report whose message embeds a whole payload (a serialized file, for example) can push the request past the proxy body limit and the rejection blocks every later event until it is pruned by hand. Frontend crash reports are therefore size-bounded at the source (`mem:frontend/routing-app-shell-subtleties`): `app.common.exceptions/get-hint` and the CLJS `format-throwable` clamp exception messages (`max-message-length`). +The legacy archiver (the one this release replaces) sent **one POST** for the whole chunk and advanced only on 204, so a single oversized event wedged the whole queue: a crash report whose message embeds a whole payload (a serialized file, for example) can push the request past the proxy body limit and the rejection blocks every later event until it is pruned by hand. Frontend crash reports are therefore size-bounded at the source (`mem:frontend/routing-app-shell-subtleties`): `app.common.exceptions/get-hint` and the CLJS `format-throwable` clamp exception messages (`max-message-length`). On the other side, Nexus checks `origin` against its allowlist (403 if not allowed), coerces the events, and inserts them into its own `audit_log` (partitioned by `received_at`, with an incrementing sequence number and `source`). Watch the rename: Penpot's `created-at` is stored as **`received-at`** in Nexus; Nexus's `created-at` is when it was created there. `tracked-at` keeps the origin time. Telemetry batches arriving at `telemetry.penpot.app` land in the same table through the telemetry endpoint above. -Concurrent jobs are safe: both lock rows in `created_at ASC, id ASC` order (one total order, deliberately blocking — no `SKIP LOCKED`, so archive arrival stays chronological), so the second waits for the first to commit and then sees whether it was last. Only acknowledged work deletes pendings; an admin-console failure never blocks the Nexus send of the same rows (each job owns its queue). +Concurrent passes are safe: the tick lock makes a second backend skip its pass instead of overlapping, and the events are sent in cursor order, like an event log. During a rolling deploy the old archiver and GC are stopped (flags `:audit-log-archive` off, `:audit-log-gc` off) while the web instances keep writing; the new consumers start at boot and replay the unarchived buffer. The old and the new version do not serialize their reads (the old version does not know the cursor table, and the new one takes no lock the old one respects because it only reads rows), but with the old GC off no row is deleted under the new cursors, and the archived rows are deleted before the replay. The new GC stays off (`:audit-log-gc`) until every instance runs the new version (see the migration NOTE). No consumer waits for another one, because each has its own cursor. The GC reads the cursors without locking them and deletes only up to the oldest cursor in the table, inclusive (`FOR UPDATE SKIP LOCKED` in its batch select), so it never waits for a consumer and a consumer never sees a row deleted under its cursor; a consumer registers at boot, so its row exists before its first pass, and with no cursor at all the GC deletes nothing. A disabled consumer keeps its cursor and holds the GC at that position. With no cursor at all the GC deletes nothing: the `:audit-log-gc` flag is the deployment switch, and the old drain-everything branch was removed. **Nobody writes full rows when no consumer is on**: persistence follows the consumer flags, so the buffer stays empty and enabling a consumer later starts from an empty buffer. The operator must set the consumer flags on every instance, web and worker, because each process computes its own value at boot. The buffer itself has no tenant column: it is one log per database, with the cursor rows and the tick lock shared by every instance, so any worker instance drains any consumer. Adding a tenant would touch the producer, the cursor key, the GC, the telemetry task and the webhooks, and there is no historical value to backfill; it is a plan of its own, not part of this change. The drain does not touch `audit_log_pending`: released versions never created it, and the migration drops the table on a pre-release database so a leftover pending row cannot block the new GC with its foreign key. Its path has no FK check. -Observability (`backend/src/app/loggers/audit/consumer_task.clj`, definitions in `backend/src/app/main.clj:default-metrics`): one `dbg` summary per job run (`:consumer` label, sent/skipped/chunks, reaped disabled pendings, swept orphans, `:elapsed` as `"1.5s"` via `ct/format-duration`, frozen in a strict binding first because log props evaluate lazily in the logger thread), one `dbg` per admin-console send call (`ready` with totals, `sent`/`failed` with counts), plus `dbg` when the queue is empty or the job's send flag is off. Queue depth is Grafana's business: it queries the database directly, so the run performs no `GROUP BY` for the logs. No `warn` before a throw: the runner reports the aborted job (refusal details travel in the error data). Metrics, all recorded with bounded labels only: `penpot_audit_consumer_events_total{ destination = nexus|admin-console, outcome = archived|failed|skipped }` (`skipped` = filtered by the admin-console allowlist) and `penpot_audit_consumer_discarded_total{ reason = reaped|swept }` (entries discarded without archiving: `reaped` counts pendings dropped for disabled consumers, `swept` counts rows deleted while nobody waited for them — the silent R3 loss, now dashboard-visible instead of log-only). Timing needs no custom histogram: the jobs substrate already records execution time per job (`jobs-execution-timing`, split by job name), so each archive job is timed as itself. No queue-depth gauge: pending depth is read off the replica database directly. The bulk frontend ingest counts `penpot_audit_ingest_events_total{ kind = full|shadow, outcome = inserted|failed }` (both faces share one transaction, so a failure drops the batch on both). Archive success counts are recorded by `run-single-consumer` on the drain totals: a chunk that returns normally committed its transaction (a rollback always throws instead), so they cannot inflate. Logs never carry event data (`props`, `context`, `profile-id`, `ip-addr`, `team-name`, emails): only counts, statuses and the uri host. +Observability (the transports in `backend/src/app/loggers/audit/consumers/{nexus,admin_console}.clj`, the shared drain in `backend/src/app/loggers/audit/consumers.clj`, the GC in `backend/src/app/loggers/audit/gc.clj`, definitions in `backend/src/app/main.clj:default-metrics`): one `dbg` summary per pass (`:consumer` label, processed/skipped/chunks, `:elapsed` as `"1.5s"` via `ct/format-duration`, frozen in a strict binding first because log props evaluate lazily in the logger thread), one `dbg` per admin-console ingest call (`ready` with totals, `processed`/`failed` with counts), plus `dbg` when there is nothing to read or the component did not start (flag off). No `warn` before a throw: the loop logs the aborted pass and keeps going (refusal details travel in the error data). Metrics, all recorded with bounded labels only: `penpot_audit_consumer_events_total{ destination = nexus|admin-console, outcome = processed|failed|skipped }` (all consumers report the successful rows as `processed`; `skipped` = filtered by the admin-console allowlist or a telemetry shadow) and `penpot_audit_consumer_discarded_total{ reason = gc }` (rows the GC removed once every enabled consumer had passed them). `penpot_audit_consumer_lag_events{ destination }` is a gauge with the confirmed events after a consumer's cursor, capped at 10000 (past that it reads 10000, so a long backlog is not an O(backlog) count on every pass): it rises when the stream falls behind (a slow destination, or the burst after a big transaction commits). While a transaction is open its rows are invisible to everybody, so they cannot be counted as lag. Timing needs no custom histogram and no longer comes from the jobs substrate: a pass has no job row. `penpot_audit_consumer_passes_total{ destination, outcome }` counts them with `ok` (drained or empty), `busy` (another instance holds the tick) and `failed` (the pass threw; a local DB failure appears only here and in the error log), so a `busy` rate is normal with several instances but a sustained busy with `ok` at zero on every instance means a retained tick; a disabled pass (flag off, direct caller) records nothing. The signal is that counter plus the lag gauge. The bulk frontend ingest counts `penpot_audit_ingest_events_total{ kind = full|shadow, outcome = inserted|failed }` (both faces share one transaction, so a failure drops the batch on both). Archive success counts are recorded by `drain-consumer` on the drain totals: a chunk that returns normally committed its transaction (a rollback always throws instead), so they cannot inflate, and a pass that aborts on chunk N still records the `processed`/`skipped` of chunks 1..N-1 (the totals live in a volatile updated after each commit). Logs never carry event data (`props`, `context`, `profile-id`, `ip-addr`, `team-name`, emails): only counts, statuses and the uri host. ### How Nexus consumes the data (and forwards to PostHog) @@ -299,7 +305,7 @@ So the full journey is: Penpot buffer → Nexus `audit_log` (via either archive ### Admin console: the second consumer -Penpot → admin-console archiving lives in its own job (`:audit-log-consumer-admin-console`, consumer `2`, flag `:admin-console`, params closed empty): it drains its own pending queue and ingests allowlisted events through `nitrate/call :ingest-audit-log`, enriched as described above. What also exists (do not confuse with archiving): +Penpot → admin-console archiving lives in its own consumer (id `2`, flag `:admin-console`, a loop in a thread): it reads from its own cursor and ingests the allowlisted events through `nitrate/call :ingest-audit-log`, enriched as described above. The allowlist lives in the consumer now (`backend/src/app/loggers/audit/consumers/admin_console.clj`): the producer does not queue for anybody, so rows outside it are normally `skipped`. What also exists (do not confuse with archiving): - **Nitrate → Penpot**: `POST /api/management/methods/push-audit-events` (`backend/src/app/rpc/management/nitrate.clj`) lets the trusted Nitrate backend inject events into the Penpot audit log (no RPC auth, `::audit/skip`, request context + `submit`; only the `admin-console` shared-key id can call it). Section 7a is the main user. - **Admin-console browser → collector**: usage events from the admin-console UI traveling through the normal public `push-audit-events` (section 7b). @@ -311,13 +317,31 @@ Flags (`common/src/app/common/flags.cljc`, varia; enabled with `PENPOT_FLAGS=ena | Flag | What it does | |---|---| -| `:audit-log` | Stores full rows; turns on `wrap-audit` and frontend ingest | -| `:nexus` | Send flag of the Nexus archive job `:audit-log-consumer-nexus` (cron `0 */5 * * * ?`; needs `:nexus-uri` (ingest path `api/audit` appended by the job)) | -| `:admin-console` | Send flag of the admin-console archive job `:audit-log-consumer-admin-console` (cron `30 */5 * * * ?`, 30s after Nexus); also the module flag for the admin-console integration | +| `:audit-log-gc` | Runs the buffer GC; the old 2.18 GC and the new one share it. Keep it off while the old and new versions coexist and turn it on after the rollout. With no consumer enabled the GC deletes nothing | +| `:nexus` | Send flag of the Nexus consumer; it appends the ingest path to `:nexus-uri`, and the component raises `:task-not-configured` at boot when the flag is on and no URI is configured | +| `:admin-console` | Send flag of the admin-console consumer; also the module flag for the admin-console integration | | `:audit-log-logger` | Structured `app.audit` log per event | -| `:error-reporting` | Starts the error reporters (off by default); database reporter on flag alone, Mattermost on flag plus webhook URL | +| `:error-reporting` | Starts the error reporters (off by default); database reporter on flag alone, Mattermost on flag plus webhook URL; also opens `push-audit-events` for crash batches without any other sink | | `:telemetry` | Stores shadow rows and turns on shipping (auto with `:telemetry-enabled`) | +`:audit-log` is **not a flag any more**. `app.config` computes +`cf/audit-persist?` once at boot from the consumer flags (`:nexus` or +`:admin-console`): when neither is on, no full rows are written anywhere, and +telemetry shadows are the only rows the producer inserts. The other event-path +sinks get the same treatment: `cf/webhooks-enabled?`, `cf/telemetry-enabled?` +and `cf/error-reporting-enabled?` are computed at boot from their flags and +read by `wrap-audit`, `process-event`, the frontend ingest and the telemetry +task. Component startup (`ig/init-key` of the reporters) still reads the flags +directly; it is not a per-event path. All write sites read `cf/audit-persist?`: +`process-event`, `submit-create-event`, `insert`, the frontend ingest, +`wrap-audit`, and the membership/GC emitters. `get-environment-data` +still puts `:audit-log` on the wire as a derived signal (persistence, +`:telemetry`, or `:error-reporting`), so `event.cljs` and the RPC schema do not +change. Telemetry keeps shipping shadows and reports on its own flag; with +persistence off the telemetry task also runs its legacy collection mode. The +`webhook`/`trigger` row from `webhooks.clj` disappears when no consumer is on, +but the webhook deliveries themselves still work. + Config (`backend/src/app/config.clj`): `:nexus-uri`, `:telemetry-uri` (default `https://telemetry.penpot.app/`), `:nexus-shared-key` / instance secret, `:public-uri` (sent as `origin` to Nexus). ## 14. Practical guide: add or change events @@ -326,6 +350,7 @@ Config (`backend/src/app/config.clj`): `:nexus-uri`, `:telemetry-uri` (default ` - **New Penpot frontend event**: create the data-event or implement `Event`/`PerformanceEvent` in `app.main.data.event`, with `::name`/`::type`/`::origin`. Honor `skip-audit?` if the action can resume. Remember: with no props schema, **your asserts are the contract**. - **New admin-console event**: server action → add the backend-log call next to the mutation, best-effort so it never breaks the action; browser action → add a new event helper plus its test. - **Command that creates an entity or already knows its team**: put the new row's `:id` in `::audit/props` (`create-project`/`create-file`) and set `:team-id` when the team is loaded, so the consumer never has to join production tables. For entities created inside an internal call (`clone-template`, `import-binfile`, `duplicate-project`), emit one event per entity: `audit/submit-file-created-event` when only the id is known (it reads the row back), `audit/submit-create-event` when the row is already in hand. +- **New audit-log consumer**: add its id to `app.loggers.audit.consumers` (never reuse one), do not seed its cursor in the migration: `ensure-cursor` creates it at the beginning of the buffer on the first pass, so the consumer replays what the buffer holds, add a `::consumer` component (the thread that runs its passes, gated by its flag) and a namespace with its `process-chunk`. The producer does not change. It must be idempotent: the drain is at-least-once, so the same event can reach it twice. - **Never** put secrets in `props`; `clean-props` is a last defense, not an excuse. - **Do not duplicate events**: if the backend already emits an event for an action, do not add a frontend (or admin-console browser) event for the same action — the backend one wins because it is reliable. A second event for the same action is only acceptable with a strong reason (for example the backend cannot see it at all), and then it must be clearly justified. - **Follow the naming policy**: use kebab-case names (`"create-organization"`, not `"createOrganization"`), and before inventing prop keys, look at similar existing events and reuse their key names (`organization-id`, `team-id`, `file-id`, `event-origin`...). Same concept, same key spelling — that is what keeps the data queryable later. @@ -337,6 +362,6 @@ Config (`backend/src/app/config.clj`): `:nexus-uri`, `:telemetry-uri` (default ` - Penpot frontend producer: `frontend/src/app/main/data/event.cljs` (collector, buffer, send), `frontend/src/app/main/repo.cljs` + `frontend/src/app/util/http.cljs` (headers), `frontend/src/app/main/data/nitrate.cljs` (Nitrate membership events). - Admin-console producer (private repo, no paths cited here): server-side organization/subscription events through the management `push-audit-events`, browser-side UI events through the public `push-audit-events`. - Consumers: `backend/src/app/loggers/webhooks.clj`, `backend/src/app/loggers/database.clj`, `backend/src/app/loggers/mattermost.clj`, `backend/src/app/tasks/telemetry.clj`. -- Archive (two jobs sharing one namespace + pending queue, no GC): `backend/src/app/loggers/audit/consumer_task.clj` (`:audit-log-consumer-nexus` / `:audit-log-consumer-admin-console` job-defs, shared drain/reap/sweep helpers), producer hook in `backend/src/app/loggers/audit.clj` (`append-audit-entry`, single `queue-pendings` helper) plus the bulk ingest in `backend/src/app/rpc/commands/audit.clj` (`handle-events`), metrics/cron wiring in `backend/src/app/main.clj` (one cron entry per job, Nexus at second 0 and admin console at second 30, each gated by its own send flag), single migration `0157` (queue, drop of `archived_at`, covering `(created_at, id)` index for the chronological scan; no backfill — pre-existing rows are swept unsent, an accepted loss). +- Archive (two cursor consumers + GC): `backend/src/app/loggers/audit/consumers.clj` (the shared mechanics: consumer ids and catalog, the tick lock and loop, the cursor lock/read/advance and the frontier, the lag gauge and the metrics) plus one namespace per transport: `consumers/nexus.clj` (the shared keys/HTTP client component) and `consumers/admin_console.clj` (with the enrichment and its allowlist); each transport declares its loop as `::consumer`, so the wiring in `backend/src/app/main.clj` (the `worker-config`) points at `:app.loggers.audit.consumers.{nexus,admin-console}/consumer`, and only worker instances start them. The GC lives in `backend/src/app/loggers/audit/gc.clj` (`:audit-log-gc`, `::job-def`, still a cron job). The producer only inserts (`append-audit-entry` in `backend/src/app/loggers/audit.clj`, `insert-rows!` in `backend/src/app/rpc/commands/audit.clj`): no queue, no allowlist. The schema ships in one migration, `0159-consumer-generalization` (the runner applies every pending step in one transaction, so the whole feature is a single step): the `seq`/`xid` columns, the unique `(xid, seq)` index and the cursor table, empty on purpose: each consumer registers its row at `(0, 0)` at boot through `ensure-cursor`, before its first pass, and the job exclusivity/dedupe columns and indexes. The pending queue never shipped: the migration does not create `audit_log_pending` and drops the table if a pre-release database has it. The out-of-band runbook lives in `operations/2.20.0-consumer-generalization.md` (the migration comment points to it): it registers the steps first (step 0, `INSERT INTO migrations`, so no deploy lands while the runner could still take the locks, and the steps below become the only path that applies the schema), then stops the legacy archiver and its GC, deletes the archived rows, keeps the legacy mark and its indexes for a later cleanup, pre-applies `0154-add-job-table` (2.19 has no `job` table) and the additive `0159` schema with the `setval(count + 1000000)` reserve, the `CONCURRENTLY` indexes, the `CHECK NOT VALID` dance and a gate that also checks the job tables, its eight indexes, the trigger and the two flags (`indisvalid` included); the web writes never stop because the old producer names neither `seq` nor `xid`, so the database fills them, and the new GC is turned on only after the rollout. Both steps are registered in step 0, so the runner never applies them: everything they create must be applied by hand, and the gate is what proves it before the deploy. The two consumers start a thread gated by their flag; the GC keeps its 5-min cron entry. - Nexus (private repo, no paths cited here): archive ingest, ordered consumers (local profile mirror, PostHog SaaS events, PostHog telemetry), telemetry endpoint, analytics views. - Tests: `backend_tests/rpc-audit-test.clj`, `rpc-team-test/accept-organization-invitation-audit-event`. diff --git a/.serena/memories/backend/core.md b/.serena/memories/backend/core.md index 9be0eb103f..941c103403 100644 --- a/.serena/memories/backend/core.md +++ b/.serena/memories/backend/core.md @@ -42,7 +42,7 @@ Backend RPC command areas without focused memories include access tokens (auth r - Next.js/JDBC caveat: where-maps don't support value vectors (no `IN`); use `status = ANY(?)` with `db/create-array` inside `db/tx-run!`. - Use `db/run!` for multiple operations on one connection. - Use `db/tx-run!` for transactions. -- `job`-substrate specifics (unified jobs): the only writers of a job status are `claim`, `retry-job`, `complete`, `fail` and `cancel` in `app.jobs`; each one is a conditional update plus its `job_event` row in the same transaction, so history can never contradict the row. A writer that affects no row writes no event and no metric. They never join the caller's transaction, because a lifecycle write reports on work that already happened: a rolled back caller cannot undo a transition, and an independent connection cannot see the caller's uncommitted job row, so a job must be committed before it can be claimed. GC deletes return resource ids via `RETURNING` inside one transaction. +- `job`-substrate specifics (unified jobs): the lifecycle writers in `app.jobs` are `claim`, `retry-job`, `complete`, `fail` and `cancel`, and the dispatcher owns the transition to `scheduled` and the two abort sweeps (`lost` and `orphan`), and writes their events itself; each one is a conditional update plus its `job_event` row in the same transaction, so history can never contradict the row. A writer that affects no row writes no event and no metric. They never join the caller's transaction, because a lifecycle write reports on work that already happened: a rolled back caller cannot undo a transition, and an independent connection cannot see the caller's uncommitted job row, so a job must be committed before it can be claimed. GC deletes return resource ids via `RETURNING` inside one transaction. - Every public function of `app.jobs` takes a cfg map. A bare pool or connection is a caller bug and is not coerced: the substrate has no conversion of its own, and the ones in `app.db` are the db layer's business. - Every new foreign key of the substrate is `DEFERRABLE`, `job_event.job_id` included. - Some tables carry a SQL deletion guard (`raise_deletion_protection` + a `deletion_protection__tgr` statement trigger): `profile`, `team`, `team_font_variant`, `file_media_object`, `file_thumbnail`, `file_tagged_object_thumbnail` and `job`. A `DELETE` raises unless the transaction runs `SET LOCAL rules.deletion_protection TO off` (the guard is on when the GUC is unset, empty or `on`); see `app.tasks.objects-gc`, `app.jobs.gc` and `th/db-force-delete`. @@ -81,14 +81,14 @@ For worker dispatch, cron, retry semantics (`ex/raise :type ::wrk/retry` with `: ## Jobs metrics -- `app.jobs.metrics` owns the jobs metric names, bounded labels and the periodic backlog sampler. -- Metrics use only `name`, `queue`, `outcome`, `reason`, `stage` and `kind` labels. Job IDs, profile IDs, params, reply bodies and exception text must never be labels. Every label goes as-is (lowercased where case varies): values come from fixed call sites, so a new value in Prometheus means new code, not noise. Only a missing value falls back (`other`, or `failed` for outcomes). +- `app.jobs.metrics` owns the jobs metric names and bounded labels. It has no gauge and no sampler: to see the current state, the operator runs one SQL query (`SELECT status, count(*) FROM job GROUP BY status`); the namespace runs no query and starts no thread. +- Metrics use `name`, `queue`, `outcome`, `reason`, `stage`, `kind`, `action`, `type` and `result` labels, each on its own family. `queue` survives only on `penpot_jobs_dispatcher_batch_size`: of the measurements that used to carry it, every other one is now keyed by the job name, because the job-def owns its routing (`::jobs/queue-name`) and the name already determines the queue; the GC, cron, request, event and file-data-offload families keep their own dimensions. Job IDs, profile IDs, params, reply bodies and exception text must never be labels. Every label goes as-is (lowercased where case varies): values come from fixed call sites, so a new value in Prometheus means new code, not noise. Only a missing value falls back, and each label has its own default: `other` (name, queue, GC kind, offload type), `failed` (the total and batch outcomes, offload result), `error` (cron and request outcomes), `backoff` (retry reason), `execution` (dispatcher stage), `deleted` (GC action) and `none` (cron reason). The submit outcome has no fallback on purpose: its three values come from fixed call sites. - The canonical job name is a keyword (`app.jobs/schema:job-name`): `submit` and `invoke` reject strings. The row stores text, so `jobs/get-job-def` still coerces (it is the runner's boundary), cron entries carry the job under `:job` with `d/name` only at the SQL/log boundary, and test helpers keyword-ize row names before invoking. -- The durable lifecycle metrics are submitted, dispatched, completed, retries, orphaned and rescheduled counters; queue-wait, execution and total-time histograms; dispatcher and backlog state; GC, cron and ephemeral request metrics. +- The durable lifecycle metrics are one submit counter with `outcome` (`created`, `skipped`, `refreshed`), dispatched and retries counters; claim-wait, execution and total-time histograms; dispatcher batch timing and size; GC, cron and ephemeral request metrics. `penpot_jobs_dispatched_total` and the claim-wait histogram are keyed by the job name. - `app.metrics/run!` is safe for recording failures, but metrics components still require a valid metrics instance. Durable submit call sites carry `::mtx/metrics` through RPC or job-def configuration. The jobs substrate takes the **cfg**, not the instance, and resolves it with `app.metrics/instance`, which raises `:missing-metrics` when absent: `app.jobs`, `app.jobs.metrics`, the runner, the dispatcher, cron and the jobs GC all follow that. `app.http`, `app.rpc.climit` and `app.storage.s3.metrics` still take the instance and were left out of scope on purpose, so do not read the codebase as uniform. The six job writers resolve at the top of their body, before writing, because their metric lands in a `db/after-commit` callback that swallows exceptions. The GC, the dispatcher and cron resolve inside that callback instead, so their guarantee rests on the component schema instead. A job that calls `heartbeat` from a component cfg must carry `::mtx/metrics` on it, and the event helper records the counter from the cfg the transaction hands it. -- SQL-backed job metrics are registered with `app.db/after-commit` from INSIDE the transaction that changed the row, so a rollback cannot inflate submitted, terminal, retry, event or dispatcher counters. Every lifecycle writer owns that transaction (`app.jobs/own-transaction` drops `::db/conn`), and `db/transact!` gives a transaction that joins no other its own callback context, so the write and its metric commit or vanish together. `submit` is the exception and joins the caller on purpose: creating a job is part of the caller's business work. +- SQL-backed job metrics are registered with `app.db/after-commit` from INSIDE the transaction that changed the row, so a rollback cannot inflate submit, terminal, retry, event or dispatcher counters. Every lifecycle writer owns that transaction (`app.db/own-transaction` drops `::db/conn` and `::db/read-only`, so a write can own its transaction even inside a read-only caller), and `db/transact!` gives a transaction that joins no other its own callback context, so the write and its metric commit or vanish together. `submit` is the exception and joins the caller on purpose: creating a job is part of the caller's business work. - `penpot_jobs_events_total` counts stored job events with no labels: the interesting split is `kind`, and grouping a growing append-only table to build a metric is not worth the query on the write path. -- The backlog sampler is `::jobs-metrics/sampler`; it runs every 30 seconds on worker-enabled, writable systems, groups by status and publishes the oldest pending age, and does not start on a read-only database. +- The terminal counter per outcome is the `_count` of `penpot_jobs_total_ms` (`completed`, `failed`, `cancelled`, `aborted`), so there is no separate terminal or orphan counter: `_count{outcome="aborted"}` counts the system-side aborts: an expired orphan lease, or a `scheduled` job whose queue has nothing waiting after the grace or that passed the 24-hour cap. There is no backlog gauge and no pending-age gauge: the backlog is a SQL query, and `created_at` is not renewed by a debounce refresh, so an age gauge would lie. - `penpot_tasks_timing` remains exported for compatibility while the jobs-specific histograms are adopted. @@ -127,8 +127,8 @@ Fixtures can populate local data for manual testing/perf work. From the backend * **Type Hinting:** Use explicit JVM type hints (e.g. `^String`, `^long`) in performance-critical paths to avoid reflection overhead. * **Batch inserts:** Use `db/insert-many!` for bulk row inserts — generates a single SQL with multiple parameter tuples. Avoid on very large datasets (SQL length / parameter count limits). -* **Server-side cursors:** Use `db/plan` (fetch-size 1000, forward-only, read-only) for large result sets. Never fetch large collections into memory at once. `db/plan` is an `IReduceInit`: `reduce`/`transduce` over it (`sequence` does not work on it), and with `partition-all` the transducer's completing arity is what flushes the last partial chunk. A plan row is a **view over the open result set**, valid only inside the reducing function: read the columns you need right there — buffering rows to read them later (a `partition-all` buffer, a lazy seq) silently yields nil columns. `db/cursor` (manual `DECLARE`/`FETCH`) is a leftover with no callers left. -* **Transaction discipline:** Use `tx-run!` for writes (opens a transaction), `run!` for reads (single connection, no transaction). Set `:read-only` on `tx-run!` when applicable to let PostgreSQL optimize. +* **Server-side cursors:** Use `db/plan` (fetch-size 1000, forward-only, read-only) for large result sets, always inside `tx-run!` (prefer `::db/read-only true`): the pool runs with auto-commit on, and the PostgreSQL JDBC driver only honors `fetch-size` (real streaming cursor) with auto-commit off, so a `plan` outside a transaction silently materializes the whole `ResultSet` in memory. Never fetch large collections into memory at once. `db/plan` is an `IReduceInit`: `reduce`/`transduce` over it (`sequence` does not work on it), and with `partition-all` the transducer's completing arity is what flushes the last partial chunk; the `reduce` holds one connection (and, inside `tx-run!`, the whole transaction) open until it finishes, so keep chunk transactions separate from the plan one when each chunk must commit on its own (the objects GC does). A plan row is a **view over the open result set**, valid only inside the reducing function: read the columns you need right there — buffering rows to read them later (a `partition-all` buffer, a lazy seq) silently yields nil columns. `db/cursor` (manual `DECLARE`/`FETCH`) is a leftover with no callers left. +* **Transaction discipline:** Use `tx-run!` for writes (opens a transaction), `run!` for reads (single connection, no transaction) — except `db/plan`, which always needs `tx-run!` (see above). Set `:read-only` on `tx-run!` when applicable to let PostgreSQL optimize. ## Lint and Format diff --git a/.serena/memories/backend/storage.md b/.serena/memories/backend/storage.md index 4a9b98ea31..baab076365 100644 --- a/.serena/memories/backend/storage.md +++ b/.serena/memories/backend/storage.md @@ -123,7 +123,7 @@ Since `put-object!` uses backend-specific operations (`impl/resolve-backend` + ` - `db` stores encoded data in `file_data.data`. - `storage` stores encoded data in storage subsystem with `file-data` bucket, keeps `data` nil in `file_data` table, and points at the object with `file_data.storage_object_id`. - `file_data.storage_object_id` is a `NO ACTION DEFERRABLE INITIALLY IMMEDIATE` foreign key to `storage_object(id)`: the database refuses to physically delete a referenced object. The `IMMEDIATE` is load-bearing: the RI violation fires inside the `storage-gc-deleted` chunk transaction, so the chunk rolls back, is logged at error level with a generic hint plus the `constraint` and `table` the database reports (any `NO ACTION` foreign key on `storage_object` can block the delete, not only `file_data`'s) and is retried on the next run; with `INITIALLY DEFERRED` the check would fire at commit, outside the catch, and the sweep would never move past the blocked chunk. -- The column is the only source of truth. The release is a coordinated restart (no rolling deploy) when the instance uses the storage backend (`PENPOT_FILE_DATA_BACKEND=storage`) or still has rows carrying `metadata.storage-ref-id`; with the default `db` backend and no such rows the previous and the new code read the same `file_data.data` column, so a rolling deploy is safe. `metadata.storage-ref-id` is no longer written or read, and the `file-data` metadata schema does not declare it. The migration backfills the rows it can resolve (valid uuid with an existing `storage_object`) and removes the key from those rows; a corrupt or orphan pointer keeps the key, which nothing reads. A large instance runs `operations/2.20.0-file-data-storage-object-id.md`: column, index and FK online, then the backfill (only if a row still carries `metadata.storage-ref-id`) with every instance stopped, verifies, and marks the migration as applied (`INSERT INTO migrations`). The backfill must never run while the previous code serves when the storage backend is in use (or rows with the pointer remain), because that code writes and scans the metadata pointer. A rollback to the previous code cannot read the rows written by this release when the storage backend is in use. +- The column is the only source of truth. The release is a coordinated restart (no rolling deploy) when the instance uses the storage backend (`PENPOT_FILE_DATA_BACKEND=storage`), has `:tiered-file-data-storage` on or pending `offload-file-data` tasks (that task writes storage rows with the default `db` backend), or still has rows carrying `metadata.storage-ref-id`; with the default `db` backend, the tiered offload off, the queue empty and no such rows the previous and the new code read the same `file_data.data` column, so a rolling deploy is safe. `metadata.storage-ref-id` is no longer written or read, and the `file-data` metadata schema does not declare it. The automatic migration and the large-instance runbook both backfill only valid UUIDs with an existing `storage_object`, set `storage_object_id` and remove the old key in the same update. Corrupt and orphan pointers are not modified and must be resolved before the unreadable-row check passes. The migration runs as one statement; the large-instance runbook uses batches of 5,000 and a temporary partial index on each partition for rows with the metadata key and a NULL column pointer. A successful batch removes its rows from that index; drop the parent index after the backfill. The backfill runs with every instance stopped because old code writes and scans the metadata pointer. The key is not retained as a rollback copy: rollback to the previous binary after backfill requires restoring the backup. The unified runbook `operations/2.20.0-unified.md` pre-registers all migration steps at the start, then runs every manual procedure before deploying 2.20.0. - `fdata/upsert!` touches the `storage_object_id` it receives in the params (the pointer the persisted row had) before replacing it, so the replaced blob is reclaimed. Callers pass the column from the file map (main/restore), `nil` for a new row (fragments, snapshots, new files) or the row itself (offload); there is no query in the upsert path, which is why every caller reads the file under the row lock first (the migration-on-read path included, see `mem:backend/subtleties`). It returns `{:written :storage-object-id }`, the pointer of the object the write created (nil for `db`/`legacy-db`), and `binfile.common/update-file!` propagates it so a migrating read returns the map with the pointer of the row it persisted instead of the pre-write one. The write also compares the expected pointer with `IS NOT DISTINCT FROM` (the caller's value, not the new one) and raises `:stale-file-data-pointer` when the row changed under it, so a stale map fails instead of leaking the replaced blob; the check lives in `upsert-in-database` because the `legacy-db` branches do not use the upsert SQL. - `binfile.common/update-file!` takes the file map read from the database plus the fields to change (`[cfg file & params]`), merges them and builds the `file_data` params from the merged map, so the pointer travels in the read map and a caller cannot lose it by building a partial map; the params cannot override `:storage-object-id`. `insert-file!` always drops it (an INSERT never replaces a persisted pointer; `duplicate-file` hands the source map there). `bfc/get-file` keeps the pointer for the write paths, and every response goes through `bfc/file->response`, which strips it and any residual `storage-ref-id` left in the metadata of the rows the migration does not backfill: each `file-data` object is authentication-only and any logged-in profile could otherwise fetch it from `/assets/by-id`. - The in-place binfile import was removed: `import-files!` raises `:not-supported` when the cfg carries `::bfc/file-id`, so no caller can overwrite a storage-backed row without carrying its previous pointer. diff --git a/.serena/memories/backend/subtleties.md b/.serena/memories/backend/subtleties.md index 4fdfbf3f81..09b38b010b 100644 --- a/.serena/memories/backend/subtleties.md +++ b/.serena/memories/backend/subtleties.md @@ -30,21 +30,23 @@ - Whether a call is nested is decided by the CONNECTION, not by an ambient flag: `db/transact!` binds `*after-commit-conn*` around the body, and a transactable that is not identical to that connection is an independent unit of work that owns its own after-commit context. A `tx-run!` that passes a pool while a transaction is open therefore commits on its own and drains its own callbacks, instead of leaving them in a context that may roll back. - `after-commit` callbacks run on commit only. A transaction marked `::db/rollback` (used by `srepl` and `app.tasks/offload-file-data`) commits nothing, so it runs none of them. - `db/run!` opens/reuses one connection but does not create a transaction. +- `db/own-transaction` returns a cfg that owns its next transaction: it drops `::db/conn` (so `tx-run!` opens its own) and `::db/read-only` (so a caller running read-only, like the storage scheduler streaming a plan cursor, cannot make the nested write read-only too). Lifecycle writes use it so a rollback of the surrounding work can undo neither the write nor its metric. - `db/tjson` is Transit JSON for jsonb storage; `db/json` is plain JSON. Job params (`job.params`) are plain JSON, decoded with the job-def decoder (`app.jobs/decode-params`); only the legacy `task` props were Transit (`decode-transit-pgobject`). Job results (`job.result`) are plain JSON too: they are read with the result decoder of the job-def (`app.jobs/decode-result`), and a result map keyed by uuid must declare its schema (`[:map-of ::sm/uuid ...]`, the import resolution does) or every key comes back as a keyword and every uuid as the text it is on disk. -- Advisory transaction locks accept UUIDs or ints. UUID locks are hashed using a zero-UUID seeded siphash. +- Advisory locks (`app.db`) come in two independent axes. **Scope**: the transaction ones (`xact-lock!`, `xact-try-lock!`) vanish with the transaction, and the session ones (`session-try-lock`, `session-unlock`, `session-unlock-all`) live until unlocked or the session ends — a pooled connection returned with one keeps it. **Form**: the one-key form takes a uuid or an int (uuids hashed with a zero-UUID seeded siphash) and the two-key form takes a `classid`/`objid` pair; PostgreSQL keys them by `objsubid` (1 vs 2), so the two spaces never conflict even with the same numbers. `update-file` and the setup take the one-key form; the audit consumers take the two-key one with `classid` `730000` and the consumer id as `objid`, which is why a consumer tick can never block a file update, and they clear whatever a borrowed connection carries before taking it. ## Workers and cron -- `job` carries `tenant` (the instance the job belongs to) and a bare `queue`. Several instances share one database and the tenant is what keeps an instance from claiming another environment's jobs, so every query that acts on one instance's work filters on `tenant`: dispatch claim, orphan sweep, lost-job reschedule, cron no-overlap count, submit dedupe. The `job__dispatcher__idx` leads with `tenant` (equality) then `(priority DESC, scheduled_at)`; `queue` does not lead because the dispatcher claims every queue of its tenant. The `:` prefix survives only in the Redis key, composed once by `app.worker/queue-key`. -- Submit dedupe only removes not-yet-due `new` jobs of the same tenant with the same name/queue/label; it does not dedupe due, scheduled, retry, running, or completed work. The `::jobs/skip-if-active` submit option goes further: it skips the insert (returning `nil`, recording `penpot_jobs_skipped_total`) when a live job (`new`/`scheduled`/`running`/`retry`) already exists for the same tenant/name/queue/label, requires an explicit non-blank label, and carries the same best-effort cross-backend race caveat as dedupe. -- Jobs metrics rule (one event, one family): counters exist only for events without latency; when an event has a histogram, its `_count` is the counter (no twin counter). `dispatched` (pushes) vs `queue-wait` (claims) are deliberately different events — do not merge them. -- The dispatcher selects `new`/`retry` jobs with `FOR UPDATE SKIP LOCKED`, marks them `scheduled`, and publishes the JSON payload `[id scheduled-at]` to the `penpot.worker.queue::` Redis list. The runner skips Redis messages whose scheduled timestamp no longer matches DB state. -- Lost `scheduled` jobs are rescheduled after 5 minutes; `running` jobs untouched longer than `:jobs-lease` (default 30 min) are marked `aborted` with `app.jobs/orphan-error` (`type :internal`, `code :orphan`). `aborted` is a system-side terminal state like `cancelled` (never retried, swept by retention like other terminals, reported with an error log); long jobs must `heartbeat` per chunk so a healthy backlog is never mistaken for a dead process. The `:progress` report is optional and only feeds the durable log. A beat whose write finds the job no longer active raises an `:interrupt` (`code :job-interrupted`) instead of counting zero: it is the internal stop-the-work signal, not an error, so neither the listener nor the runner logs it as one. -- An orphan gets its `end` event (`outcome "aborted"`) from the sweep itself: one bulk `INSERT ... SELECT` in the same transaction as the bulk `UPDATE`, never one insert per orphan. No msgbus notification goes out for it; orphans alert through the error log and the orphaned counter. +- `job` carries `tenant` (the instance the job belongs to) and a bare `queue`. Several instances share one database, and the tenant is what keeps an instance from claiming another environment's jobs: every query that acts on one instance's work filters on it (dispatch claim, lost sweep, orphan sweep, cron submit), and the tenant is part of the key of the dedupe and exclusivity indexes of `submit` (the cron no-overlap check is one of those inserts). +- Two submit options resolve their key in the database instead of reading before writing, each through its own partial unique index over (tenant, name, queue, label). `::jobs/skip-if-active` inserts the row as **exclusive** and `job__live_exclusive__uniq` admits a single live row (`new`/`scheduled`/`running`/`retry`). `::jobs/dedupe` inserts it as **deduped** and `job__new_dedupe__uniq` admits a single not-started row (`new`) whatever its `scheduled_at`; a conflicting insert affects no row, the submit returns `nil` and records `penpot_jobs_submit_total{outcome="skipped"}`. The pending row is never deleted nor modified: it runs when it is due, with its own params. `::jobs/debounce` uses the same key and index but resolves the conflict with `ON CONFLICT ... DO UPDATE`, refreshing `scheduled_at` (to `now + delay`), `params` and `modified_at` of the pending row and answering with the id it already had: that is the sliding window of the webhook batches, where the last payload is the one delivered. A refresh is not a new job: the submit records `outcome="refreshed"`, apart from the `created` ones, and the id it reports is the id the statement wrote. Dedupe and debounce are mutually exclusive. Both require an explicit non-blank label and only cover the jobs that asked for the option: a live job that was not submitted with it (a webhook delivery, a notification) does not block a flagged submit, and two plain jobs can share a label. The `exclusive` and `dedupe` columns are what keep that distinction; the label stays a label. +- A single insert carries a single `ON CONFLICT` clause, so `skip-if-active` cannot be combined with `dedupe`/`debounce`: an exclusive job already admits one live row, PostgreSQL admits one conflict target per statement, and the submit raises `:skip-if-active-cannot-be-combined`. +- Jobs metrics rule (one event, one family): counters exist only for events without latency; when an event has a histogram, its `_count` is the counter (no twin counter). `dispatched` (pushes) vs `claim_wait` (claims) are deliberately different events — do not merge them. `claim_wait` measures `now - scheduled_at` at the claim, so it is how late the claim happened, including the dispatcher gap, not time inside a queue. `dispatcher_batch_ms` times one dispatcher pass and pairs with `dispatcher_batch_size`; `gc_pass_ms` times one GC pass (`expired` or `retained`). +- The dispatcher claims `new`/`retry` jobs atomically, marks them `scheduled` and hands them to their queue; the runner skips a message whose scheduled timestamp no longer matches the database state (a rescheduled or reclaimed job). +- A job handed to a queue (`scheduled`) is marked `aborted` with `app.jobs/lost-error` (`code :lost`) when its queue has nothing waiting (`LLEN`) and it has been `scheduled` longer than the grace (5 min; it also covers the pop-to-claim gap of a message popped right after the hand-off, but not one that waited longer in the queue and is popped while the sweep runs: the round trip can lose that race), or when it has been `scheduled` past the cap (24 h, for a message lost behind a backlog or a queue without a consumer; a daily cron can miss one run). Leftover rows are never re-dispatched, so nothing returns to `new` and the dispatcher cannot violate `job__new_dedupe__uniq` (the thresholds are `lost-grace` and `lost-cap`, and `lost-batch-size` bounds each read in `app.worker.dispatcher`: past-cap rows are aborted without an `LLEN` and grace rows with one, so a younger lost row behind a large legitimate backlog may wait for the cap instead of the grace). `running` jobs untouched past `:jobs-lease` are marked `aborted` with `app.jobs/orphan-error` (`type :internal`, `code :orphan`). `aborted` is a system-side terminal state like `cancelled` (never retried, swept by retention like other terminals, reported with a log line: `warn` for lost, `error` for orphan); long jobs must `heartbeat` per chunk so a healthy backlog is never mistaken for a dead process. The `:progress` report is optional and only feeds the durable log. A beat whose write finds the job no longer active raises an `:interrupt` (`code :job-interrupted`) instead of counting zero: it is the internal stop-the-work signal, not an error, so neither the listener nor the runner logs it as one. +- Both abort sweeps (lost and orphan) write the `end` event (`outcome "aborted"`) in the same transaction as the status change, never one insert per row. No msgbus notification goes out for them; the aborted rows alert through the log and `penpot_jobs_total_ms_count{outcome="aborted"}`, which counts both causes (there is no separate counter). - `job_event` is append-only and deleted by cascade with the job. `progress` is a row there, not a column: no Redis and no mutable column to keep in sync. -- `job` is deletion-protected at the SQL level (`deletion_protection__tgr`): a stray `DELETE FROM job` raises instead of dropping the row and its `job_event` cascade without touching the referenced `storage_object`. The only intended deleters, the jobs GC and the submit dedupe, disable the guard with `SET LOCAL rules.deletion_protection TO off` inside their transaction; tests use `th/db-force-delete`. +- `job` is deletion-protected at the SQL level: a stray `DELETE FROM job` raises instead of dropping the row and its `job_event` cascade without touching the referenced `storage_object`. The intended deleter is the jobs GC, which disables the guard inside its transaction; tests use `th/db-force-delete`. `app.jobs/submit` never deletes: its dedupe is a conflict of the insert. - A missing job-def raises (`:no-job-definition`) instead of completing. Throwing with `ex-data :type ::retry` still controls retry behavior; `:strategy ::noop` retries without incrementing retry count. -- Cron entries claim their `scheduled_task` row with `FOR UPDATE SKIP LOCKED`, disable statement/idle-in-transaction timeouts locally, submit one `job` row per entry when no active instance exists (no-overlap), and reschedule themselves in `finally` unless interrupted. Worker, dispatcher, and cron components do not start when the DB pool is read-only. +- Cron entries submit one exclusive job per entry when no live instance exists, so two backends cannot overlap the same entry; the database's partial unique index decides it, not a count. They reschedule themselves unless interrupted, and the worker, dispatcher and cron components do not start on a read-only database. - Integrant params schemas name injected components as bare `::ns/key` entries (pool, metrics, msgbus, job defs), and a bare entry is a malli registry lookup, not `any?`: the owning namespace must `sm/register!` it at load, or the backend crashes at boot with `:malli.core/invalid-schema` instead of asserting. Unit tests never build those keys, so `backend-tests.worker-test` compiles the three worker schemas to catch it. ## Config and HTTP/session middleware @@ -83,6 +85,7 @@ ## Storage and media - The `media-storage-bytes-per-team` quota check on `upload-file-media-object` is TEMPORARILY DISABLED (2026-10-19, kept commented in `app.rpc.commands.media` with its `sql:get-team-id-for-file` lookup still active for the audit props): the quote performs a full `SUM` over `storage_object` per upload and was too slow in production. The `upload-sessions-per-profile` quota is unaffected and stays active. Re-enable only with a cheaper approximation. +- `file-gc` trims emit `file-gc` with the cleaned `:file-id` and its `:team-id`; `delete-object-with-cascade` carries flat `:team-ids` next to the nested `:cascade` map. A consumer can key on those props without joining production tables. - Storage abstraction, backend configuration, logical buckets, object lifecycle, deduplication, access rules, and garbage collection: `mem:backend/storage`. - SVG validation strips DOCTYPE and uses secure SAX parsing. Basic SVG info falls back to 100x100 dimensions when width/height/viewBox are missing. - The SVG sanitizer (`app.media.svg`) is a **blocklist**, not an allowlist: it drops dangerous elements and attributes by name and escapes every text and attribute value when it writes the tree back, so a payload hidden in a value cannot become live markup again. The allowlist rework is [#12190](https://github.com/penpot/penpot/issues/12190) and lands in media-processor, not here. `sanitize-svg` also tracks the namespace bindings through the recursion and drops any element that is not in the SVG namespace, so a document whose root is not an `svg` element in the SVG namespace is rejected and a child that rebinds `xmlns` to XHTML (`iframe`/`srcdoc`/`meta`) is dropped with its subtree; it compares the scheme of an URL the way the browser does (dropping tab/CR/LF/spaces), so `java script:` does not slip through the `javascript:` filter. The paths that store an uploaded SVG as its final object go through `app.media.svg/sanitize-svg-upload` (media upload, the URL download, both thumbnail writers and the organization logo); the three binfile imports go through `app.binfile.common/sanitize-imported-svg`, which calls `sanitize-svg` directly. The profile picture is not an SVG: `import-profile-picture` ignores an `image/svg+xml` download, because the avatar is always rendered as an image. The caller checks the size before calling it: the helper reads the whole file, and the size limit applies to the upload, not to the sanitized copy, which can be up to six times bigger. @@ -109,7 +112,7 @@ - File snapshots (versions) keep their metadata row in `file_change` and their payload in `file_data` (`type = 'snapshot'`). With the `db` backend `file_change.data` stays NULL: that column is only read by the `legacy-db` backend (it is no longer written), plus the rows written before the 2025-07 file storage refactor. - Tell a snapshot apart by `label IS NOT NULL` (with `created_by` / `deleted_at` as needed), never by `file_change.data IS NOT NULL`: the label holds with every file data backend, and the payload column stopped meaning "this is a snapshot" when the payload moved out. Queries that identify versions must use the label. `0157-migrate-legacy-file-data` moved the labelled leftovers to `file_data` and deleted the unlabelled payload rows, so from 2.19 on the label is the only marker there is. - The version quota counts of `app.rpc.quotes` (`sql:get-snapshots-per-file`, `sql:get-snapshots-per-team`) are served by the partial index `file_change__user_snapshots__idx`, whose predicate mirrors their where clause; changing one without the other drops the index only scan and the count walks the whole table again. -- Revisions live only in `file_change` and have no `file_data` counterpart. They are born already scheduled for deletion one hour later (`app.rpc.commands.files-update`) and the snapshot paths cannot reach one, because they resolve their row through a `label IS NOT NULL` filter. +- Revisions live only in `file_change` and have no `file_data` counterpart. They are born already scheduled for deletion one hour later (`app.rpc.commands.files-update`) and the snapshot paths cannot reach one, because they resolve their row through a `label IS NOT NULL` filter. The revision row created by a save travels to the audit event in `:file-change-id` (a consumer reads it with `::db/remove-deleted false`, because the row is born with its `deleted_at` in the future); a save writes two rows, the revision (no `label`) and the snapshot of the previous state (with one). - A deletion date is assigned once and never moved: `restore-files` (`app.rpc.commands.files`) unmarks only durable history, which is `label IS NOT NULL` (the unlabelled rows that still carry a legacy payload are removed by `0157-migrate-legacy-file-data`, so restore leaves them dated), the `:file` method of `app.tasks.delete-object` dates only rows that have no date yet, and `objects-gc` collects only rows that carry one. Unmarking a revision leaves it invisible to `objects-gc` for ever, because nothing sets that date again. - Binfile v1/v3 import and the v3 export `get-file` step run inside one long transaction. They raise `idle_in_transaction_session_timeout` to 30 min (`app.binfile.common/transaction-idle-timeout-ms`) instead of disabling it, so a connection left idle while the ZIP is decoded or the file data is realized is bounded; `statement_timeout` stays at 300 s. - `update-file` with `:changes-with-metadata` groups that all carry `:commit-id` is partial: fast path applies and validates everything once; on a change failure it bisects the commits for the longest prefix that applies and validates (each attempt applies a prefix from the original data and runs referential integrity + file schema; ~log2 K attempts, none for a single commit, the first attempt's error reused), keeping that prefix (a commit that repairs an earlier one can keep both) and answering 200 with `:rejected {:commit-ids :failed-commit-id :error}`, plus `:preexisting-damage true` when the data the first rejected commit touches already fails validation without it (`preexisting-damage?`: one more referential + schema check of its pages/components on the base file, the accepted prefix or the original; the frontend then says the page data is damaged). Only accepted changes reach `file_change`, persistence and the msgbus; a fully rejected request writes and broadcasts nothing (revn unchanged). Flat `:changes` or groups without ids stay all-or-nothing. Before each attempt (`apply-prefix`, which returns `{:file :tracked :refs}` or the cause) `pmap/*tracked*` and the `cpc/*state*` atom are reset to their pre-attempt values (at the end `apply-commits` sets those of the kept prefix): a pointer map already `modified?` keeps its id when changed again, so the failed version would be persisted. DB/IO/interrupt errors and `:not-found` fragments (`change-failure?`) fail the whole request, never reject a commit. A rejection whose ex-info has no `:type` reports `:internal :unexpected-error`. `app.http.errors` answers 503 (`:server-error` + code) for transient SQLSTATEs (classes 08/53, 57P01–3, 40001, 40P01), `SQLTransientException`, `InterruptedException` and only the `IOException`s that pass on their own (`transient-io-exception?`: `SocketException`, `InterruptedIOException` incl. socket timeouts, `UnknownHostException`, `HttpTimeoutException`), which the client's save retries (and GETs retry by themselves); any other `IOException` (EOF from a corrupt blob, Jackson parse errors, missing files) stays 500 `:io-exception`, as do other SQL errors. With the `:debug-save-faults` flag, a `:set-plugin-data` file change in namespace `:penpot-debug` keyed `save-fault:` raises that fault in `apply-changes` (used by `debug.failNextSave`/`commitInvalidChange`). With a schema check running (hard schema validation without skip-validate, or soft) and 2+ commits, the touched pages' objects are decoded on the original file first (`decode-objects`; referential validation alone reads few objects, so no pre-decode then): objects-map caches decoded values per instance and a derived map copies the cache, so otherwise every attempt re-decodes the page (~260 ms at 10k shapes). Validation step timings (`with-validation-timing`, upstream `penpot_rpc_update_file_validation_timing` + the `update-file` debug log) come from the first attempt only (retries run with a cfg without metrics and timings); the whole bisection is one `retry` step sample, and the rejection log carries `:attempts`/`:elapsed`. diff --git a/.serena/memories/backend/testing.md b/.serena/memories/backend/testing.md index 3ed98a539e..bba466f198 100644 --- a/.serena/memories/backend/testing.md +++ b/.serena/memories/backend/testing.md @@ -10,27 +10,15 @@ JVM `clojure.test` (kaocha runner) under `backend/test/backend_tests/`. - **Preferred way to run them: `scripts/ci --test`**, which runs lint, tests and format per module and writes a log per task under `.ci-logs/`. It exports `PENPOT_FLAGS` with `enable-backend-asserts` appended, so use it rather than calling `clojure -M:dev:test` by hand. - **Why the flag matters:** `clojure.core/assert` is elided at compile time when `*assert*` is false, and `app.config` only sets that var from the flags when it loads. The test fixture redefines the flags after that, so they never reach it: without the flag the suite runs with every assertion compiled out, and a test that depends on an assert only means something with it set. `scripts/ci` and `tests-backend.yml` export the flag (GitHub Actions sets it too). When running the suite directly — including a `--focus` run — the command is `PENPOT_FLAGS="enable-backend-asserts" clojure -M:dev:test`. - Coverage: if code is added or modified in `src/`, corresponding tests in `test/backend_tests/` must be added or updated. +- **Log lines are not tested.** A log record is for the operator, not a contract: do not add a test that captures it (`add-watch` on `app.common.logging/log-record`, or `with-redefs` on `enabled?`/`emit-log`) to assert its message, hint or props, and do not keep such a test alive when the line changes. Whatever must be guaranteed belongs in a returned value, an event or a metric, and the test asserts that; a log line may change its wording at any time. - Isolated run: `clojure -M:dev:test --focus backend-tests.my-ns-test` for a specific test namespace, or `clojure -M:dev:test --focus backend-tests.my-ns-test/my-test-var` for a specific test var. - Multiple focus symbols are supported by REPEATING the flag (kaocha accumulates them, last-wins does not apply): `clojure -M:dev:test --focus backend-tests.ns-one --focus backend-tests.ns-two` — preferred over a shell loop of isolated runs. - Regression run: `clojure -M:dev:test` to ensure no regressions in related functional areas. - Line coverage: append the `:coverage` alias (`clojure -M:dev:test:coverage`, kaocha-cloverage). It instruments every `app.*` namespace under `src/` at load, before any test runs, so even a `--focus` run instruments them all; options live under `:cloverage/opts` in `tests.edn`, output in `target/coverage/` (lcov, HTML, and EMMA `coverage.xml` whose form counts the PR report shows as statements; cloverage measures no functions or branches) (`lcov.info`, HTML). About 2x slower and noisy with reflection warnings (instrumented forms lose type hints): never use it for normal runs. `tests-backend.yml` runs it and renders the step summary with `scripts/coverage-summary.mjs`. - If you need to filter output, tee to a temp file first: `clojure -M:dev:test 2>&1 | tee /tmp/penpot-test-output.txt`. - RPC test helpers `command!`/`management-command!` split the data map: qualified keys become server params, unqualified keys become request body params. To inject request-level context (headers, `:app.http/auth-key-id`, ip), pass a map under `:app.http/request` metadata; non-map `IRequest` stubs fall back to a dummy request. -- The test system is `main/system-config` + `main/worker-config` with the - components that must not run in the background dropped: HTTP, the srepl, - cron, the dispatcher and **every** `[ :app.worker/runner]` - (`helpers/remove-job-runners`). A live runner of any queue races the tests - that drive the loop by hand (`wdisp/run-batch` + `@#'wrkr/run-worker-loop`): - it takes the job the test just dispatched and completes it behind the test's - back, so the test's own assertions read a state it did not produce. Do not - re-add a runner by editing that list by hand; the removal matches by key - shape on purpose. -- `system-config` and `worker-config` are `def`s: their `(cf/get ...)` calls - are evaluated when the namespace loads, against the real environment, while - test code calling `cf/get` at run time sees the `penpot-test`-prefixed - config. The two can disagree — in the devenv `:tenant` is `devenv-ws0` for - the defs and `default` inside a test — so a component built from - `worker-config` can talk to a different Redis key prefix than a cfg the test - builds itself. The symptom is a component that silently does nothing - locally and races in CI (which sets no `PENPOT_TENANT`). -- `backend-tests.helpers` has `metrics-stub` (a metrics instance with no collectors, where every `run!` is a no-op), `init-climit` (a concurrency limiter built from an inline config map, with an optional second argument for the metrics instance to publish to) and `with-climit`, the macro that builds one, runs the body and halts it. `init-climit` starts the republisher daemon thread, so a test that builds a limiter without halting it leaks that thread for the rest of the JVM run: use `with-climit`. Log lines are capturable with `add-watch` on `app.common.logging/log-record` (it holds the last record; the watcher sees every one, but only of the levels the logger has enabled: a line the app writes at TRACE never gets there). The logger swaps the record on an executor thread, after the call that logged it returns: keep the watch installed until the record arrives (poll for it) and only then remove it, or the record lands on nobody under load. +- The test system is `main/system-config` + `main/worker-config` with the components that must not run in the background dropped: HTTP, the srepl, cron, the dispatcher, the two audit-log consumer loops (`:app.loggers.audit.consumers.nexus/consumer`, `:app.loggers.audit.consumers.admin-console/consumer`) and **every** `[ :app.worker/runner]` (`helpers/remove-job-runners`). A live runner of any queue races the tests that drive the loop by hand (`wdisp/run-batch` + `@#'wrkr/run-worker-loop`): it takes the job the test just dispatched and completes it behind the test's back, so the test's own assertions read a state it did not produce. A live consumer loop races the tests the same way: it drains `audit_log` in the background. Do not re-add any of them by editing that list by hand; the runner removal matches by key shape on purpose. +- `th/run-task!` invokes a job by name; the audit-log consumers are no longer jobs, so a test builds the pass cfg (`::db/pool`, `::mtx/metrics` and the transport deps) and calls the transport's `run-pass` directly (`consumer.nexus/run-pass`, `consumer.admin-console/run-pass`). +- `app.loggers.audit/submit` propagates its failure (`submit*` no longer swallows it), so a DB-free unit test that reaches a code path emitting an audit event must stub it (`with-redefs [app.loggers.audit/submit (constantly nil)]`, as the `helpers.clj` fixture does globally) or pass a cfg carrying `::db/pool`. Otherwise the throw detours the flow under test into its error path. +- `system-config` and `worker-config` are `def`s: their `(cf/get ...)` calls are evaluated when the namespace loads, against the real environment, while test code calling `cf/get` at run time sees the `penpot-test`-prefixed config. The two can disagree — in the devenv `:tenant` is `devenv-ws0` for the defs and `default` inside a test — so a component built from `worker-config` can talk to a different Redis key prefix than a cfg the test builds itself. The symptom is a component that silently does nothing locally and races in CI (which sets no `PENPOT_TENANT`). +- `backend-tests.helpers` has `metrics-stub` (a metrics instance with no collectors, where every `run!` is a no-op), `init-climit` (a concurrency limiter built from an inline config map, with an optional second argument for the metrics instance to publish to) and `with-climit`, the macro that builds one, runs the body and halts it. `init-climit` starts the republisher daemon thread, so a test that builds a limiter without halting it leaks that thread for the rest of the JVM run: use `with-climit`. diff --git a/.serena/memories/clojure/core.md b/.serena/memories/clojure/core.md index 876b003a49..0e1c636311 100644 --- a/.serena/memories/clojure/core.md +++ b/.serena/memories/clojure/core.md @@ -2,9 +2,9 @@ How to shape a function in this codebase. Each rule came from a review round; do not re-derive from taste. +- **HARD RULE: no new `!` names.** Do not introduce a function or macro ending in `!`; side effects alone do not justify the suffix. Allowed exceptions, and only these: mutating public fns in `app.db` (`insert!`, `update!`, `delete!`, `run!`, `tx-run!`, ...) and fns whose contract is to work over transients (stdlib `conj!`/`assoc!`/`dissoc!`/`persistent!` and custom helpers over them). Add a new exception to `mem:clojure/idioms` with its justification **before** using it. Existing `!` elsewhere (e.g. `emit!`, `send!`, `submit!`, `validate-file!`) stays until its refactor, so a family stays consistent meanwhile; that is a reason to keep an old name, never to add a new one. Reporting helpers (log, sample-and-publish) never take `!`: the mutation happens in the `!` API they call. - **Preconditions belong to the boundary, not to the core.** A function documents what it needs (hints, docstring) and assumes it. Absence checks live where the value enters the system: the caller that resolves optional objects, `ig/init-key`, the request handler — including `nil` that is optional-by-design (`when-let [cs (some-> server ...)]` at the entry point, as with `ConnectorStatistics` when the server option is off). A `(when (and (some? a) (some? b)) ...)` inside a leaf function means the guard is in the wrong place — if you cannot get `a` or `b`, you should not be calling it. - **A static set of operations is written statically.** With a fixed, small set (four metrics) write the four calls. Do not build a collection and iterate it (`doseq` over a literal vector): it allocates, adds indirection and hides each operation. Use a collection when the set is dynamic (config, registry, input). -- **Default to no `!`; existing `!` is legacy being removed gradually.** Do not introduce new `!` names except the allowed cases documented in `mem:clojure/idioms` (add and justify the special case there first). Allowed: mutating public fns in `app.db` (`insert!`, `update!`, `delete!`, `run!`, `tx-run!`, ...) and fns whose contract is to work over transients (stdlib `conj!`/`assoc!`/`dissoc!`/`persistent!` and custom helpers over them). Existing `!` elsewhere (e.g. `emit!`, `send!`, `submit!`, `validate-file!`) stays until its refactor; keep a family consistent in the meantime. Reporting helpers (log, sample-and-publish) never take `!`: the mutation happens in the `!` API they call. - **Functions that obtain data start with `get-` or `fetch-`.** A reader must see at a glance that the function reads (`get-file`, `get-snapshot`, `get-file-json`); a bare noun hides it (`file-json` says nothing about reading). New helpers that fetch by id follow this, never a `current-` prefix: the id parameter already says which one it is. - **`?` marks a predicate, not a data prop.** A var or function that answers a question takes `?` (`export-error?`, `(defn editable? …)`); a map key that names a property does not (`:export-error`, never `:export-error?`), even when its value is a boolean. Existing `?` keys are legacy to remove gradually; do not add a new one. - **Do not shape production code for tests.** No return values added just to assert them; assert on the observable effect. When preconditions move, move the tests with them. diff --git a/.serena/memories/devenv/core.md b/.serena/memories/devenv/core.md index a6645ed04f..6949a0b004 100644 --- a/.serena/memories/devenv/core.md +++ b/.serena/memories/devenv/core.md @@ -15,7 +15,7 @@ Compose-based dev environment under `docker/devenv/`, driven by `manage.sh`. Par - For ws1+, `instance-env-overrides` computes the per-instance overrides (container/volume names, host ports offset `10000·N`, `PENPOT_PUBLIC_URI`, `PENPOT_REDIS_URI`, `PENPOT_TEST_DATABASE_URI`, `PENPOT_TEST_REDIS_URI`) and `instance-compose` injects them as env vars at compose time — never written to disk, recomputed each call so they can't drift. ws0 uses `defaults.env` as-is. - Per-instance test databases are created by `ensure-instance-test-database` (called from `start-instance`): `CREATE DATABASE` guarded by a `pg_database` existence check, so it is idempotent. Needed because `postgresql_init.sql` only runs on first volume creation. The backend applies migrations itself on first use, so an empty database is enough. - `backend/scripts/_env`: backend-internal only — secret keys, `PENPOT_FLAGS` (with `enable-backend-worker` gated on `PENPOT_BACKEND_WORKER`), `JAVA_OPTS`, `setup_s3_bucket()`. Never duplicates `defaults.env`. -- Frontend admin-console is off by default in devenv: `docker-compose.main.yml` sets `PENPOT_DEVENV=true`, which makes `app.config` (`goog-define devenv-mode`) prepend `disable-admin-console` ahead of the user `config.js` flags; flags parse in order and user flags win, so `enable-admin-console` in `frontend/resources/public/js/config.js` re-enables it. Outside devenv (prod builds) the default is unset and the admin-console stays on, matching `flags/default`. +- Frontend admin-console is off by default in devenv: `docker-compose.main.yml` sets `PENPOT_DEVENV=true`, which makes `app.config` (`goog-define devenv-mode`) prepend `disable-admin-console` ahead of the user `config.js` flags; flags parse in order and user flags win, so `enable-admin-console` in `frontend/resources/public/js/config.js` re-enables it. Outside devenv (prod builds) the default is unset and the admin-console stays on, matching `flags/default`. The backend `_env` also adds `disable-admin-console` and keeps `enable-nexus` commented out, so both cursor consumers are off in devenv: `cf/audit-persist?` is false and no full audit rows are written (telemetry is disabled there too). Uncomment `enable-nexus` in `_env` when integration testing the consumer. - Compose files use pure `${VAR}` substitution; missing var = compose fails. ## Invariants @@ -30,7 +30,7 @@ Compose-based dev environment under `docker/devenv/`, driven by `manage.sh`. Par ## Worker policy -Backend workers run only on ws0. `_env` gates `enable-backend-worker` on `PENPOT_BACKEND_WORKER`; ws1+ inject it as false. Workers are pure fire-and-forget: `jobs/submit` inserts a row into the shared Postgres `job` table and returns (the legacy `task` table stays dormant); RPC handlers never wait on completion and workers never publish to msgbus. The reason for "ws0 only" is avoiding multi-instance worker races (cron dedup is best-effort across instances, `jobs/submit` `dedupe` is racy across submitters); details in `mem:prod-infra/core`. +Backend workers run only on ws0. `_env` gates `enable-backend-worker` on `PENPOT_BACKEND_WORKER`; ws1+ inject it as false. Workers are fire-and-forget from the RPC's point of view: `jobs/submit` inserts a row into the shared Postgres `job` table and returns (the legacy `task` table stays dormant) and RPC handlers never wait on completion. A job with an owner profile does publish `:job-event` on that profile's msgbus topic after the commit, on the msgbus of the workspace that executes it; with per-workspace valkey, a job enqueued from ws1 and run on ws0 notifies ws0's bus, not ws1's. The reason for "ws0 only" is avoiding multi-instance worker races (cron dedup is best-effort across instances; `jobs/submit` resolves its dedupe and exclusivity in the database, so the submit itself does not race); details in `mem:prod-infra/core`. Each workspace is independent and can be started/stopped in any order. Shared infra (Postgres, RustFS, etc.) is shut down only when no instances remain running. diff --git a/.serena/memories/memory-maintenance.md b/.serena/memories/memory-maintenance.md index 67a480a770..0f7f26aee5 100644 --- a/.serena/memories/memory-maintenance.md +++ b/.serena/memories/memory-maintenance.md @@ -21,7 +21,7 @@ Dense agent notes, not prose docs. Prefer invariants, terse bullets. Avoid obvious context, rationale, and examples unless they prevent likely mistakes. Keep guidance durable and generalizable, not task-local. -No manual line wrapping: one line per bullet or paragraph, however long. Memories render adapting to the viewport; hard-wrapped lines degrade rendering and diffs. +**No manual line wrapping**, in new and edited memories alike: one line per bullet or paragraph, however long. Memories render adapting to the viewport; hard-wrapped lines degrade rendering and diffs. When you touch a memory that is hard-wrapped, unwrap the whole file in the same change (join each paragraph and each bullet's continuation lines with a single space), so the wrapping does not survive merely because nobody rewrote the file. ## Add/update threshold diff --git a/.serena/memories/prod-infra/core.md b/.serena/memories/prod-infra/core.md index 09a58d9784..586aa149d1 100644 --- a/.serena/memories/prod-infra/core.md +++ b/.serena/memories/prod-infra/core.md @@ -4,36 +4,37 @@ Backend (`app.config`, `PENPOT_*` env vars) is parameterized; deployments choose ## Services -- **PostgreSQL**: durable store. Profiles, teams, files, sessions, audit, `storage_object` metadata, the `job` work queue (plus the dormant legacy `task` table), `scheduled_task` cron registry, migrations. File-data also lives here when the file-data backend is `legacy-db`/`db`. One shared DB across all backends. -- **Redis (Valkey-compatible)**: per-backend message bus and cache. Concrete uses: msgbus Pub/Sub for collaborative-editing broadcasts and team/profile-org notifications fired by RPC handlers (`app.rpc.notifications`, `files_update`, `teams`, `websocket`); file-summary cache gated by `enable-redis-cache`; rate-limit counters; and the dispatcher→runner work hand-off list `penpot.worker.queue::`. `PENPOT_REDIS_URI`. A burst of client-side `RedisCommandTimeoutException` (2s on `DEL`/`SET`, 10s on `BLPOP`) with the exporter healthy is the JVM paused or starved, not Redis: the timeout fires because nothing reads the reply. +- **PostgreSQL**: durable store: profiles, teams, files, sessions, audit, the jobs work queue, the cron registry and migrations. File data also lives here depending on the file-data backend. One shared DB across all backends. +- **Redis (Valkey-compatible)**: per-backend message bus and cache: collaborative-editing broadcasts and team/profile notifications, the file-summary cache, rate-limit counters and the dispatcher→runner hand-off. `PENPOT_REDIS_URI`. A burst of client-side `RedisCommandTimeoutException` with the exporter healthy is the JVM paused or starved, not Redis: the timeout fires because nothing reads the reply. - **Object storage**: backends `:s3` and `:fs`. S3 in prod; devenv uses RustFS. Holds uploaded media, file-data when the file-data backend is `storage`, exports. Backend-side details (resolve, dedup, bucket set, object lifecycle, and file-data backends): `mem:backend/storage`. - **SMTP mailer**: invitations, password resets, email verification (sent via the `:sendmail` worker job). - **LDAP** (optional auth provider): helpers in `app.auth.*`, gated by `enable-login-with-ldap`. ## Task queue and worker model -Async jobs are enqueued via `jobs/submit` (`app.jobs`), which inserts a row into the shared Postgres `job` table tagged with `queue = ":"`. Submission is **fire-and-forget** — RPC handlers never poll and never wait. The completion signal is the `job` row's `status` / `completed_at` columns, which the creation commands read back to answer with the state of the job they just created. A job submitted with `::jobs/profile-id` also publishes a `:job-event` message on that profile's msgbus topic after each transaction commits, and the binfile commands (`create-export-binfile-job`, `create-import-binfile-job`) are the callers that pass that owner, so the progress of their jobs does reach the msgbus. Soft-delete RPCs return immediately after marking the top-level row, leaving the cascade and reaping to workers. +Async jobs are enqueued via `jobs/submit` (`app.jobs`), which inserts a row into the shared Postgres `job` table tagged with its tenant and queue. Submission is **fire-and-forget**: RPC handlers never poll and never wait. The completion signal is the job row's status, which the creation commands read back to answer with the state of the job they just created. A job submitted with an owner profile publishes its progress on that profile's msgbus topic after each transaction commits; the binfile commands are the callers that pass the owner. Soft-delete RPCs return immediately after marking the top-level row, leaving the cascade and reaping to workers. -Workers run on backends with `enable-backend-worker` in `PENPOT_FLAGS`. Each worker-enabled backend has a `dispatcher` (polls `job` with `FOR UPDATE SKIP LOCKED`, marks status='scheduled', RPUSHes the JSON `[id scheduled-at]` payloads into **its own** Redis list) and one or more `runner`s per queue (BLPOP from that same local list, execute, update the Postgres row). The Redis hand-off list is purely intra-backend — cross-backend coordination happens at the Postgres row level. The queues are `default`, `webhooks`, `cron` and `binfile`: the last one carries the heavy work of a user job (`:worker-binfile-parallelism`, default 1), so a long export or import does not block the default runner. +Workers run on backends with `enable-backend-worker` in `PENPOT_FLAGS`. Each worker-enabled backend has a dispatcher that claims jobs and pushes them to the shared Redis list `penpot.worker.queue::` (`app.worker/queue-key`), and one or more runners per queue that pop from the same list and update the Postgres row. The tenant segment keeps two tenants sharing one Redis apart; a job handed off by one backend can be popped by any backend running that tenant's queue, and the Postgres row is the only claim. The queues are `default`, `webhooks`, `cron` and `binfile`: `binfile` carries the heavy work of a user job, so a long export or import does not block the default runner that serves the delete cascades. ## Cross-backend safety -Postgres row locking is the only correctness primitive: `job` claims via `FOR UPDATE SKIP LOCKED`, cron firing via `FOR UPDATE SKIP LOCKED` on the `scheduled_task` row, plus job-handler-internal locks (e.g. `file_gc_scheduler` locks candidate file rows). This makes the work-claim path safe across any number of worker-enabled backends. +Postgres row locking is the only correctness primitive: job claims, cron firing and the job-handler-internal locks (for example, the file-gc scheduler locks its candidate rows) all rely on it. This makes the work-claim path safe across any number of worker-enabled backends. Two known race patterns survive multi-backend operation: -- **Cron dedup is best-effort.** The lock on `scheduled_task` is released when the task body finishes. If two backends' cron timers fire for the same scheduled instant with a gap larger than the task body's runtime, both execute it. Penpot's cron entries are idempotent (`session-gc`, `objects-gc`, `storage-gc-*`, `storage-pending-gc`, `jobs-gc`, `file-gc-scheduler`); the exceptions are `:telemetry` (would double-report) and `:audit-log-consumer-nexus` / `:audit-log-consumer-admin-console` (depend on archive target idempotency). -- **`jobs/submit ::dedupe true`** does a non-atomic `DELETE` then `INSERT`. Concurrent cross-backend submits can both bypass the `DELETE` (each sees the other's uncommitted insert as absent) and end up with duplicate `'new'` rows. Each row claims and runs once independently, so the underlying work is fine; the "at most one pending" guarantee weakens. +- **Cron dedup is best-effort.** The lock on `scheduled_task` is released when the task body finishes. If two backends' cron timers fire for the same scheduled instant with a gap larger than the task body's runtime, both execute it. Penpot's cron entries are idempotent; the exception is `:telemetry`, which would double-report. The audit consumers are not served by cron any more: each one is a loop in a thread and takes a session advisory lock for the whole pass, so a double-fire of a cron entry cannot overlap them. +- **`jobs/submit ::dedupe` / `::debounce`** are not affected by it: the row is inserted as deduped and a partial unique index (`status = 'new'`) admits a single not-started row per tenant/name/queue/label. The insert resolves the conflict with `ON CONFLICT DO NOTHING` (skip) or `DO UPDATE` (refresh the schedule and the params of the pending row), so there is no delete before the insert and no cross-backend duplicate. Neither gives "at most one execution": a row that already left `new` (dispatched, running, retrying) does not block the next submit. +- **`jobs/submit ::skip-if-active true`** works the same way one level up: the row is inserted as exclusive and a partial unique index admits a single live row (`new`/`scheduled`/`running`/`retry`) per tenant/name/queue/label, resolved with `ON CONFLICT DO NOTHING`. What it does not give is "at most one execution": the row can go terminal while its work is still running (a retry, an `aborted` orphan), so the work itself must be claimed on the data it touches. Penpot in production lives with both: horizontal-scale deployments accept "exactly-once" as "essentially-once for idempotent operations." Devenv parallel instances handle it by running workers only on ws0 (see `mem:devenv/core`). ## Jobs observability -Worker-enabled, writable backends publish unified jobs metrics through the normal Prometheus registry. Counters describe submit, dispatch, terminal outcomes, retries, orphan recovery, rescheduling and total stored job events (unlabelled); histograms separate queue wait, handler execution and total job age. SQL-backed events are recorded after the outer PostgreSQL transaction commits. The `::jobs-metrics/sampler` component publishes backlog by status and the oldest pending-job age every 30 seconds. All labels are bounded operational values; job IDs, profile IDs, params and error text are excluded. Read-only backends do not start the sampler or the workers. +Worker-enabled, writable backends publish unified jobs metrics through the normal Prometheus registry. One submit counter splits by `outcome` (`created`, `skipped`, `refreshed`), and the dispatch and retry counters are keyed by job name; terminal outcomes are the `_count` of the total-time histogram (`completed`, `failed`, `cancelled`, `aborted`), which counts the system-side aborts (an expired orphan lease or a job never claimed after being scheduled), so there is no separate terminal or orphan counter. Histograms separate claim wait (how late the claim happened), handler execution and total job age, plus dispatcher batch timing and size. `queue` survives only on the batch size: a batch mixes job names, and every other measurement is keyed by a name that already determines its queue. SQL-backed events are recorded after the outer PostgreSQL transaction commits. There is no gauge and no sampler: the backlog and the age of the oldest pending job are one SQL query when an operator asks. All labels are bounded operational values; job IDs, profile IDs, params and error text are excluded. Read-only backends do not start the workers. ## Draining one instance -Each backend has a per-process availability flag (`app.health`, a global atom). While offline, `/readyz` answers `503 OFFLINE` without querying the database, so the load balancer removes that instance from rotation while requests in flight still finish. Operator surface: `app.srepl.main/set-offline` / `set-online`; a restart comes back online. In-flight worker jobs are safe: jobs claimed as `scheduled` return to `new` after 5 minutes via `sql:reschedule-lost` and another instance picks them up. +Each backend has a per-process availability flag (`app.health`, a global atom). While offline, `/readyz` answers `503 OFFLINE` without querying the database, so the load balancer removes that instance from rotation while requests in flight still finish. Operator surface: `app.srepl.main/set-offline` / `set-online`; a restart comes back online. Nothing stays stuck after an instance leaves: a job handed to a queue (`scheduled`) that no worker claims within the lost window is aborted with `:lost`, and a `running` job whose lease expires is aborted with `:orphan`. The in-flight work is abandoned, not resumed, and the sweep never retries it. ## Touching production data @@ -43,12 +44,14 @@ Production runs the deployed toolbox, not the devenv one: - **Start from the ids, not from an inventory.** The ids come from the error or the logs. Unbounded queries over tables with tens of millions of rows (`storage_object`, `file_media_object`) do not finish in a reasonable time; bounded to an id list they are index lookups. - **One query per reference column.** Two columns in one `IN (col_a, col_b)` lose the index and scan the whole table; split them. - The read-only replica cuts long queries: see `mem:prod-infra/grafana`. -- A migration that builds an index runs inside the migration transaction, so it cannot use `CONCURRENTLY`, and a plain `CREATE INDEX` locks the table against writes for the whole build. On a large table build the index out of band, before deploying, with `CREATE INDEX CONCURRENTLY` on its own (never inside a transaction block). A concurrent build interrupted midway leaves the index invalid: drop it and retry (check `pg_index` and `pg_index.indisvalid`). +- A migration that builds an index runs inside the migration transaction, so it cannot use `CONCURRENTLY`, and a plain `CREATE INDEX` locks the table against writes for the whole build. On a large table (file saves write to `file_change` on every save) build the index out of band, before deploying, with `CREATE INDEX CONCURRENTLY` on its own (never inside a transaction block); the shipped migration uses `IF NOT EXISTS`, finds it and skips the build (it still takes the `ALTER`s' locks, so the operator registers the step in `migrations`; the consumer-generalization runbook registers it first of all, before the DDL). A concurrent build interrupted midway leaves the index invalid: drop it and retry (check `pg_index` and `pg_index.indisvalid`). The runner applies **every pending step in one transaction**, so pre-applying one migration of a group does not avoid the locks of the others. +- The 2.20 consumer generalization is the concrete case, and its runbook is the contract for a large instance: it ships as `0159-consumer-generalization` (plus `0154-add-job-table`, which a 2.19 database does not have); the runbook lives in `operations/2.20.0-consumer-generalization.md` (the migration comment points to it). It opens by registering the steps (`INSERT INTO migrations`, `module = 'main'`, `step = '0154-add-job-table'` and `'0159-consumer-generalization'`) so the runner skips the group from the first minute, whatever deploy lands in between, and the steps below become the only path that applies the schema; without that row the guards skip the work, not the `ACCESS EXCLUSIVE` locks nor the backfill scan. The steps: the `IF NOT EXISTS` ALTERs (metadata-only), the `CONCURRENTLY` unique indexes (`job__live_exclusive__uniq`, `job__new_dedupe__uniq`, `audit_log__xid__seq__idx`), the `seq` backfill (a single ordered `UPDATE`, no table lock, re-runnable) with the sequence reserved above it (`setval(count + 1000000)`, so new rows sort above the history), the `CHECK NOT VALID` + `VALIDATE` + `SET NOT NULL` for `audit_log.seq`, and the `indisvalid` gate before deploying. Before that, the runbook stops the legacy archiver **and its GC** with the flags `:audit-log-archive` off and `:audit-log-gc` off, keeps at least one consumer flag (`:nexus` or `:admin-console`) on **on every instance**, and deletes the already-archived rows. There is no `:audit-log` flag any more: each process computes its persistence value at boot from the consumer flags, so a web instance without one writes no rows even when a worker drains them. With all consumers off there is no backlog to replay. The two versions do not serialize each other: a concurrent old archiver/GC could archive and delete rows the new consumers have not read yet. The legacy mark and its indexes stay; a later release drops them with catalog operations. The new GC is gated by `:audit-log-gc` (the old 2.18 flag): keep it off while the old and new versions coexist and turn it on after the rollout; with no consumer enabled it deletes nothing. The pending queue never shipped: the migration does not create `audit_log_pending` and drops the table outright if a pre-release database has it, so leftover rows there cannot block the new GC. +- The 2.20.0 release ships three out-of-band migration procedures; the unified, self-contained runbook for the whole upgrade (phases from pre-flight to the maintenance window) is `operations/2.20.0-unified.md`, indexed in `operations/README.md`. The individual 2.20.0 runbooks stay as the reference per area. - A partitioned parent cannot take `CREATE INDEX CONCURRENTLY` (`cannot create index on partitioned table ... concurrently`). Build one index per partition with `CONCURRENTLY`, then `CREATE INDEX ... ON ONLY` on the parent and `ALTER INDEX ... ATTACH PARTITION` for each child; the parent index only becomes valid when every partition is attached. -- Write the migration as plain, idempotent SQL: `ADD COLUMN IF NOT EXISTS`, `CREATE INDEX IF NOT EXISTS`, and `DROP CONSTRAINT IF EXISTS` followed by `ADD CONSTRAINT`. Do not encode shape checks as `pg_catalog` introspection: they are easy to get wrong (a missing `NOT condeferred` accepts an `INITIALLY DEFERRED` foreign key) and they duplicate what the runbook verifies by hand. `IF NOT EXISTS` matches the name, not the definition, so the out-of-band procedure verifies the index shape and completeness. -- The runbook is the contract for a large instance: it builds and verifies the out-of-band work and then marks the migration step as applied (`INSERT INTO migrations (module, step) ...`), so the body does not run at startup. Verify the column, the pending rows, the index (valid, no predicate or expression, every partition attached) and the constraint shape before inserting the row; the marker is the only thing that skips the migration. +- Write the migration as plain, idempotent SQL: `ADD COLUMN IF NOT EXISTS`, `CREATE INDEX IF NOT EXISTS`, and `DROP CONSTRAINT IF EXISTS` followed by `ADD CONSTRAINT`. Do not encode shape checks as `pg_catalog` introspection: they are easy to get wrong (a missing `NOT condeferred` accepts an `INITIALLY DEFERRED` foreign key) and they duplicate what the runbook verifies by hand. `IF NOT EXISTS` matches the name, not the definition, so the out-of-band procedure verifies the index shape and completeness; an `INVALID` index from an interrupted build also satisfies `IF NOT EXISTS` and the dedupe/exclusivity stops enforcing silently, so the `indisvalid` gate is mandatory. +- The runbook is the contract for a large instance: it pre-registers the migration steps (`INSERT INTO migrations (module, step) ...`) at the start, then builds and verifies the out-of-band work before deployment. The rows make the runner skip the migrations, so the manual runbook must finish every schema change and backfill before 2.20.0 starts. The unified 2.20.0 runbook registers all steps together at the beginning. - A reference that a GC task resolves must be `DEFERRABLE INITIALLY IMMEDIATE`, never `INITIALLY DEFERRED`: the immediate check fires inside the transaction that catches the violation and retries the chunk; the deferred one fires at commit, outside the catch. A validated foreign key belongs out of band: `ADD FOREIGN KEY` runs as an `ALTER TABLE` and it scans the whole referencing table; `NOT VALID` on a partitioned table only exists since PostgreSQL 18. -- A backfill that removes a key an old reader uses (for example `metadata.storage-ref-id`, which the previous storage GC scans) must run with the old code stopped. Otherwise the old reader loses the reference and can delete a live object, and the rows it writes in the meantime miss the backfill. +- A backfill that removes a key an old reader uses (for example `metadata.storage-ref-id`, which the previous storage GC scans) must run with the old code stopped. Otherwise the old reader loses the reference and can delete a live object, and the rows it writes in the meantime miss the backfill. The 2.20 file-data migration does this atomically per row; the large-instance runbook batches updates, clears the key with the column write, uses a temporary partial index for pending rows, and requires restoring the backup to roll back to the old binary. - `ADD COLUMN` takes a brief `ACCESS EXCLUSIVE` even when `IF NOT EXISTS` skips it, and `ADD FOREIGN KEY` takes `ACCESS EXCLUSIVE` on the referencing table and `SHARE ROW EXCLUSIVE` on the referenced one. A migration accepts those locks; a deployment that cannot take them runs the runbook and marks the step as applied. ## See also diff --git a/.serena/memories/scripts/ci.md b/.serena/memories/scripts/ci.md index be509b7a22..94c058a906 100644 --- a/.serena/memories/scripts/ci.md +++ b/.serena/memories/scripts/ci.md @@ -17,6 +17,10 @@ assertion is compiled out. GitHub Actions sets the same flag. finishing (AGENTS.md: run the applicable lint and format checks). - When `common/` changed: validate its consumers too (frontend, backend, exporter; see the dependency graph in `mem:critical-info`). +- Pass every module you touched, `common` included: the backend `lint:clj` + lints `../common/src` with clj-kondo but does not run `cljfmt` over it, so + a `common/` formatting error only shows in the `common:fmt` task (its own + `check-fmt:clj`), which is what CI runs. - To fix formatting across a module (`--fix`) or repair delimiters (`--paren-repair`) before linting. diff --git a/AGENTS.md b/AGENTS.md index b492067467..b0d0bbfc04 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -15,6 +15,13 @@ - **Never pipe test output directly to filters** (`| head`, `| tail`, `| grep`, etc.). Always redirect to a file first: `command > /tmp/output.txt 2>&1`, then read/grep the file. This prevents hiding test failures. See `mem:testing` for details. +- **Never introduce a new function or macro name ending in `!`.** Side + effects alone do not justify it: only the exceptions documented in + `mem:clojure/design-rules` (public mutators of `app.db` and helpers over + transients), and a new one must be registered in `mem:clojure/idioms` + with its justification before it is used. Existing names stay until their + refactor; a new one fails review. See `mem:clojure/design-rules` before + naming anything. - **Read `AGENTS.local.md` after this file whenever it exists.** It carries the developer's steering for this project and overrides this file on prose style and workflow, but not on project policy, where instructions here prevail. diff --git a/backend/scripts/_env b/backend/scripts/_env index e91ae20161..470eda4dc7 100644 --- a/backend/scripts/_env +++ b/backend/scripts/_env @@ -42,7 +42,6 @@ export PENPOT_FLAGS="\ enable-backend-asserts \ disable-feature-fdata-pointer-map \ enable-feature-fdata-objects-map \ - enable-audit-log \ enable-transit-readable-response \ disable-remote-media-processing \ enable-demo-users \ diff --git a/backend/src/app/config.clj b/backend/src/app/config.clj index 206336ca97..57bbf5d22c 100644 --- a/backend/src/app/config.clj +++ b/backend/src/app/config.clj @@ -392,7 +392,18 @@ ;; When telemetry-enabled config is true, add :telemetry flag (true? (c/get config :telemetry-enabled)) (conj :enable-telemetry))] - (flags/parse flags/default extra-flags (:flags config)))) + ;; :audit-log is no longer configurable: persistence follows the + ;; consumer flags, so a stale enable-audit-log in the environment is + ;; dropped instead of lingering in the set as a misleading no-op. + (-> (flags/parse flags/default extra-flags (:flags config)) + (disj :audit-log)))) + +(defn persist-audit-events? + "True when at least one audit-log consumer is enabled. The result is + computed once at boot into `audit-persist?`; event paths read that value + instead of checking the flags on every event." + [flags] + (boolean (some #{:nexus :admin-console} flags))) (defn read-env [prefix] @@ -449,6 +460,10 @@ (defonce ^:dynamic config (read-config :default default)) (defonce ^:dynamic flags (parse-flags config)) +(defonce ^:dynamic audit-persist? (persist-audit-events? flags)) +(defonce ^:dynamic webhooks-enabled? (contains? flags :webhooks)) +(defonce ^:dynamic telemetry-enabled? (contains? flags :telemetry)) +(defonce ^:dynamic error-reporting-enabled? (contains? flags :error-reporting)) (defn validate! "Validate the currently loaded configuration data." diff --git a/backend/src/app/db.clj b/backend/src/app/db.clj index 8bdeae67a4..71d9b85811 100644 --- a/backend/src/app/db.clj +++ b/backend/src/app/db.clj @@ -683,6 +683,19 @@ (finally (reset! context []))))))) +(defn own-transaction + "A cfg for a write that must own its transaction: drops `::db/conn` + so the next `tx-run!` opens its own transaction on its own + connection, and `::db/read-only` so that transaction is writable + even when the caller runs read-only (as the storage scheduler does + for its plan cursor). It never joins the transaction of whoever + called it; because that transaction joins no other, it owns its + after-commit context too, so the write and its metric commit or + vanish together and a rollback of the surrounding work can undo + neither." + [cfg] + (dissoc cfg ::conn ::read-only)) + (defn tx-run! "Run a function in a transaction." [system f & params] @@ -820,6 +833,52 @@ row (exec-one! conn ["select pg_try_advisory_xact_lock(?::bigint) as lock" n])] (:lock row))) +(defn- check-lock-key + "A two-key advisory lock takes int4 keys; anything else (a uuid, a value + out of range) would reach the database as an opaque cast error." + [n] + (if (and (int? n) (<= Integer/MIN_VALUE (long n) Integer/MAX_VALUE)) + n + (throw (IllegalArgumentException. "int4 key required")))) + +(defn session-try-lock + "Take a session-level advisory lock of two keys without waiting. + + `classid` is the namespace of the lock and `objid` the key inside it. + This two-key form lives in its own lock space, apart from the one-key + `pg_advisory_lock(bigint)` of `xact-lock!` (the file locks and the setup + one): PostgreSQL keys a lock by `objsubid` (1 for one key, 2 for two), so + the same numbers in the two forms do not conflict. That is how a + subsystem takes a namespace of its own without touching the others. + + Unlike the `xact-*` variants the lock survives the transaction: it lives + until `session-unlock` releases it, until the connection is really closed + or until the session ends. A pooled connection returned with a lock held + keeps it, so the caller must release it; `session-unlock-all` clears + whatever a borrowed connection carries." + [conn classid objid] + (let [row (exec-one! conn ["select pg_try_advisory_lock(?::int, ?::int) as lock" + (check-lock-key classid) + (check-lock-key objid)])] + (:lock row))) + +(defn session-unlock + "Release one two-key session-level advisory lock held by this connection. + Answers false when this connection did not hold that lock." + [conn classid objid] + (let [row (exec-one! conn ["select pg_advisory_unlock(?::int, ?::int) as lock" + (check-lock-key classid) + (check-lock-key objid)])] + (:lock row))) + +(defn session-unlock-all + "Release every session-level advisory lock held by this connection. + Used before taking one on a borrowed connection, so a lock some earlier + borrower failed to release cannot outlive it." + [conn] + (exec-one! conn ["select pg_advisory_unlock_all()"]) + true) + (defn sql-exception? [cause] (instance? java.sql.SQLException cause)) diff --git a/backend/src/app/jobs.clj b/backend/src/app/jobs.clj index 271414ebcc..cec6e8740e 100644 --- a/backend/src/app/jobs.clj +++ b/backend/src/app/jobs.clj @@ -281,37 +281,52 @@ ;; SUBMIT API ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; Two columns mark the jobs whose key (tenant, name, queue, label) is +;; resolved by the database instead of by the caller: `exclusive` admits a +;; single live row (`job__live_exclusive__uniq`) and `dedupe` admits a +;; single not-started one (`job__new_dedupe__uniq`). Both are resolved by +;; the insert: PostgreSQL admits a single `ON CONFLICT` clause per +;; statement, so a submit is one of these four and never two at once. +;; +;; The conflict target repeats the index predicate on purpose: that is +;; how PostgreSQL infers a partial unique index. (def ^:private sql:insert-new-job "insert into job (id, name, tenant, params, queue, label, priority, max_retries, profile_id, resource_id, expires_at, - created_at, modified_at, scheduled_at) - values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + created_at, modified_at, scheduled_at, exclusive, dedupe) + values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)") + +;; A single live row per key: `::jobs/skip-if-active` affects no row and +;; the submit reports the job as skipped. +(def ^:private sql:conflict-exclusive + "on conflict (tenant, name, queue, label) + where exclusive + and status in ('new', 'scheduled', 'running', 'retry') + do nothing returning id") -;; `tenant` is part of the match, not an extra: without it this DELETE -;; reaches the not-started jobs of every other instance sharing the -;; database. -(def ^:private sql:remove-not-started-jobs - "DELETE FROM job - WHERE name=? - AND tenant=? - AND queue=? - AND label=? - AND status = 'new' - AND scheduled_at > ?") +;; A single not-started row per key: `::jobs/dedupe` leaves it alone and +;; `::jobs/debounce` refreshes it in place. The refreshed row keeps its +;; id, its events and its `created_at`; only the schedule and the params +;; travel from the new submission. +(def ^:private sql:conflict-dedupe + "on conflict (tenant, name, queue, label) + where dedupe and status = 'new' + do nothing + returning id") -;; The cron no-overlap check and the skip-if-active submit option share -;; this shape: both look for jobs of one tenant with the same name, queue -;; and label that have not reached a terminal state yet. The -;; `job__name_label__idx` partial index serves both queries. -(def ^:private sql:count-active-jobs - "SELECT count(*) AS n - FROM job - WHERE name = ? - AND tenant = ? - AND queue = ? - AND label = ? - AND status IN ('new', 'scheduled', 'running', 'retry')") +(def ^:private sql:conflict-debounce + "on conflict (tenant, name, queue, label) + where dedupe and status = 'new' + do update set params = excluded.params, + scheduled_at = excluded.scheduled_at, + modified_at = excluded.modified_at + returning id") + +;; A plain submit has no conflict target but still answers with the id of +;; the row it wrote: every insert returns it, whichever clause it carries. +(def ^:private sql:returning-id + "returning id") (def ^:private schema:options [:map {:title "submit-options"} @@ -322,7 +337,15 @@ [::queue {:optional true} [:or ::sm/text :keyword]] [::priority {:optional true} ::sm/int] [::max-retries {:optional true} ::sm/int] + ;; Skip the insert when a not-started job already covers the same + ;; (tenant, name, queue, label), whatever its `scheduled_at`. Requires + ;; an explicit label, like the other two. [::dedupe {:optional true} ::sm/boolean] + ;; Same key as `dedupe`, but instead of skipping, move the pending job + ;; to `now + delay` and store the new params. That is the sliding + ;; window of a batch: the receiver gets the last payload, not the + ;; first. Mutually exclusive with `dedupe`. + [::debounce {:optional true} ::sm/boolean] ;; Skip the insert when a live job already covers the same ;; (tenant, name, queue, label). Requires an explicit label: the ;; default empty one would block every flagged submission at once. @@ -352,6 +375,40 @@ (sm/check (::schema job-def) params)) params) +(defn- check-option-label + "Raise unless the option that keys the submit of a job has an explicit + label: the label is the key of the dedupe and of the exclusion, and the + default empty one would block every flagged submission at once." + [option label job-name] + (when (or (not (string? label)) (str/blank? label)) + (ex/raise :type :validation + :code (keyword (str option "-requires-label")) + :hint (str option " requires an explicit non-blank label") + :name job-name))) + +(defn- check-keyed-options + "Validate the options that need an explicit label and the ones that + contradict each other. Runs before the schema check so the failure names + the option, not the shape: the schema would reject a blank label with a + generic \"check error\" either way." + [{::keys [name label dedupe debounce skip-if-active]}] + (when (and dedupe debounce) + (ex/raise :type :validation + :code :dedupe-and-debounce-are-exclusive + :hint "dedupe and debounce cannot be used together" + :name (d/name name))) + (when (and skip-if-active (or dedupe debounce)) + (ex/raise :type :validation + :code :skip-if-active-cannot-be-combined + :hint "skip-if-active cannot be combined with dedupe or debounce" + :name (d/name name))) + (when dedupe + (check-option-label "dedupe" label (d/name name))) + (when debounce + (check-option-label "debounce" label (d/name name))) + (when skip-if-active + (check-option-label "skip-if-active" label (d/name name)))) + (defn submit "Submit a durable job: validates the params with the job-def validator, encodes them as plain JSON and inserts a row into the `job` table. @@ -361,36 +418,59 @@ `::expires-at`) are validated and stored as columns; nothing is inferred from `params`. - With `::skip-if-active`, the insert is skipped (and `nil` returned) - when a live job (`new`, `scheduled`, `running` or `retry`) already - exists for the same tenant, name, queue and label. The check and the - insert share one transaction. A skip deletes nothing and inserts - nothing; it records a skipped metric instead of a submitted one, so - dashboards can tell withheld work apart from new work. The option - needs an explicit non-blank `::label`, otherwise it raises. + With `::skip-if-active`, the row is inserted as exclusive: the + database admits a single live row (a `new`, `scheduled`, `running` or + `retry` one) per tenant, name, queue and label, and the insert is + skipped (and `nil` returned) when there is already one. The decision + is the unique index `job__live_exclusive__uniq`, so it holds across + backends and does not depend on a check racing an insert. A skip + inserts nothing and records the submit with `outcome=\"skipped\"` + instead of `created`, so dashboards can tell withheld work apart from + new work. The option needs an explicit non-blank `::label`, otherwise + it raises: the label is the key of the exclusion. - NOTE: the dedupe DELETE and the INSERT run atomically: joined to - the caller's transaction when the cfg provides `::db/conn`, wrapped - in their own transaction otherwise. Concurrent cross-backend - submissions can, in rare race conditions, produce duplicated 'new' - rows (accepted risk, see prod-infra documentation). The same caveat - covers the skip-if-active check: a race between backends can slip - two inserts past the count." + The exclusion is between the jobs that asked for it: a live job that + was not submitted as exclusive (a webhook delivery, a notification) + does not block an exclusive submit, and two such jobs can share a + label. That is why the flag is a column and not a rule about the + label: the label stays a label. + + With `::dedupe`, the row is inserted as deduplicated: the database + admits a single not-started (`new`) row per tenant, name, queue and + label, whatever its `scheduled_at`, and the insert is skipped (and + `nil` returned) when there is already one. The pending job is not + touched: it runs when it is due, with its own params. The decision is + the unique index `job__new_dedupe__uniq`, so it holds across backends. + `::debounce` uses the same key and the same index but, instead of + skipping, refreshes the pending row: it moves `scheduled_at` to + `now + delay`, stores the new params and returns the id the row already + had. That is the sliding window of a batch job, where the last event is + the one worth delivering. A refresh is not a new job, so the submit + records it with `outcome=\"refreshed\"`, apart from the ones that created + a row. Both options need an explicit non-blank `::label` and are + mutually exclusive. + + Only the jobs submitted with `::dedupe` or `::debounce` are part of + that key, so a plain job never blocks a deduplicated one, and two plain + jobs can share a label. + + An insert carries a single `ON CONFLICT` clause, so `::skip-if-active` + cannot be combined with `::dedupe` or `::debounce`: they are different + keys, an exclusive job already admits a single live row, and the + combination raises. + + The submit runs joined to the caller's transaction when the cfg + provides `::db/conn`, wrapped in its own otherwise, so a rollback takes + the row and the metric with it. There is no read before the write: the + unique index of the option decides." [cfg {:keys [::params ::name ::delay ::queue ::priority ::max-retries - ::dedupe ::skip-if-active ::label + ::dedupe ::debounce ::skip-if-active ::label ::profile-id ::resource-id ::expires-at] :or {delay 0 queue :default priority 100 max-retries 3 label ""} :as options}] (assert (mtx/instance cfg) "missing ::mtx/metrics on the jobs cfg") - ;; Before the schema check so the failure names the option, not the - ;; shape: the schema would reject a blank label with a generic - ;; "check error" either way. - (when (and skip-if-active (or (not (string? label)) (str/blank? label))) - (ex/raise :type :validation - :code :skip-if-active-requires-label - :hint "skip-if-active requires an explicit non-blank label" - :name (d/name name))) + (check-keyed-options options) (check-options options) (let [job-def (get-job-def (get-defs cfg) name) params (validate-params job-def params) @@ -413,62 +493,62 @@ tenant (cf/get :tenant) job-name (d/name name) queue (d/name queue) - ;; Dedupe is best-effort: we delete not-started jobs of this - ;; tenant with the same name/queue/label, then insert. A race - ;; between backends could create duplicates, but this is - ;; acceptable: cross-backend races are rare, jobs are idempotent, - ;; and dedupe is best-effort. - ;; Skip-if-active is best-effort like dedupe: the count and the - ;; insert share this transaction, but a race between backends - ;; can still slip two inserts past the count. - active? (fn [conn] - (when skip-if-active - (pos? (or (:n (db/exec-one! conn [sql:count-active-jobs - job-name tenant queue label])) - 0)))) - insert! (fn [conn] - (let [deleted (when dedupe - ;; the DELETE below is intentional: - ;; disable the `job` deletion-protection - ;; guard inside this transaction - (db/exec-one! conn ["SET LOCAL rules.deletion_protection TO off"]) - (-> (db/exec-one! conn [sql:remove-not-started-jobs - job-name tenant queue label now]) - (db/get-update-count)))] - (l/trc :hint "submit" - :name job-name - :job-id (str id) - :tenant tenant - :queue queue - :label label - :dedupe (boolean dedupe) - :delay (ct/format-duration delay) - :replace (or deleted 0)) - (db/exec-one! conn [sql:insert-new-job id job-name tenant payload queue - label priority max-retries - profile-id resource-id expires-at - now now scheduled-at])))] - ;; Both statements always run inside db/tx-run!: joined to the - ;; caller's transaction when the cfg provides a connection, - ;; wrapped in their own otherwise (a failed INSERT can never - ;; orphan a committed DELETE, even on an autocommit caller conn). - ;; A skip returns nil: no DELETE, no INSERT, no submitted metric. + ;; A single ON CONFLICT clause per insert: the option picks it, + ;; and the unique index of that option decides. The insert is the + ;; only statement, so there is nothing to delete and nothing to + ;; roll back on its own. + conflict (cond + skip-if-active sql:conflict-exclusive + debounce sql:conflict-debounce + dedupe sql:conflict-dedupe + :else nil) + sql (str/join " " [sql:insert-new-job + (or conflict sql:returning-id)])] + ;; The insert runs inside db/tx-run!: joined to the caller's + ;; transaction when the cfg provides a connection, wrapped in its own + ;; otherwise. The statement is what decides: a conflicting one affects + ;; no row (the skip) or refreshes the pending row (the debounce) and + ;; answers with its id, so the row it names is always a real one. (db/tx-run! cfg (fn [{:keys [::db/conn]}] - (if (active? conn) - (do (db/after-commit - #(metrics/record-skipped cfg job-name queue)) - (l/trc :hint "submit skipped, active job exists" - :name job-name - :tenant tenant - :queue queue - :label label - :skipped true)) - (let [result (insert! conn)] + (if-let [result + (db/exec-one! conn [sql id job-name tenant payload + queue label priority max-retries + profile-id resource-id expires-at + now now scheduled-at + (boolean skip-if-active) + (boolean (or dedupe debounce))])] + (let [created? (= id (:id result))] (db/after-commit - #(metrics/record-submitted cfg job-name queue)) - result - id)))))) + #(if created? + (metrics/record-created cfg job-name) + (metrics/record-refreshed cfg job-name))) + (l/trc :hint "submit" + :outcome (if created? "created" "refreshed") + :name job-name + :job-id (str (:id result)) + :tenant tenant + :queue queue + :label label + :dedupe (boolean dedupe) + :debounce (boolean debounce) + :exclusive (boolean skip-if-active) + :delay (ct/format-duration delay)) + (:id result)) + (do + (db/after-commit + #(metrics/record-skipped cfg job-name)) + (l/trc :hint "submit" + :outcome "skipped" + :name job-name + :tenant tenant + :queue queue + :label label + :dedupe (boolean dedupe) + :debounce (boolean debounce) + :exclusive (boolean skip-if-active) + :delay (ct/format-duration delay)) + nil)))))) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; JOB API @@ -526,9 +606,9 @@ (def ^:private known-outcomes ;; mirrors the `job_event` end outcomes. `aborted` is only ever written - ;; by the dispatcher orphan sweep (bulk insert in the same transaction, - ;; see `app.worker.dispatcher/mark-orphan-jobs`): no `app.jobs` writer - ;; emits it. + ;; by the dispatcher sweeps, orphan and lost (one bulk insert per sweep + ;; in the same transaction of its status change, see + ;; `app.worker.dispatcher`): no `app.jobs` writer emits it. #{"completed" "failed" "cancelled" "aborted"}) (def ^:private known-retry-reasons @@ -716,21 +796,6 @@ :created-at created-at}))) {:event-id id :created-at created-at :kind kind :payload payload}))) -(defn- own-transaction - "A cfg for a job write that must own its transaction. - - A lifecycle write reports on work that already happened elsewhere, so - it never joins the transaction of whoever called it: the `::db/conn` is - dropped and `db/tx-run!` opens its own transaction on its own - connection. Because that transaction joins no other, it owns its - after-commit context too, so the write and its metric commit or vanish - together and a rollback of the surrounding work can undo neither. - - `submit` is the deliberate exception: creating a job is part of the - business unit of work of whoever asks for it." - [cfg] - (dissoc cfg ::db/conn)) - ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; HEARTBEAT / PROGRESS ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; @@ -825,10 +890,10 @@ here, so a progress event can never land on an already terminal job. Returns 1 when the event was stored. - Like every lifecycle write it owns its transaction (see `own-transaction`), + Like every lifecycle write it owns its transaction (see `db/own-transaction`), so a beat survives the rollback of the work around it." [cfg job-id progress] - (db/tx-run! (own-transaction cfg) + (db/tx-run! (db/own-transaction cfg) (fn [{:keys [::db/conn] :as tx-cfg}] (if (db/exec-one! conn [sql:lock-active-job job-id]) (do @@ -942,7 +1007,7 @@ ;; Every metric is registered with `db/after-commit` from inside the ;; transaction, next to the event, so it lands on the outermost commit ;; whatever that is: a rolled back transition is never counted. Each -;; writer owns that transaction (see `own-transaction`), so the commit +;; writer owns that transaction (see `db/own-transaction`), so the commit ;; in question is always its own. ;; ;; That callback swallows exceptions, which is why every writer below @@ -969,18 +1034,19 @@ "Record the total time of a terminal outcome. Runs after the commit that made the job terminal, so a rolled back transition is never counted. The histogram `_count` is the terminal total per outcome: - no separate counter. Takes the cfg and resolves the metrics - instance from it, so the callback is not tied to an instance read - at registration time." + no separate counter, and `outcome=\"aborted\"` counts the system-side + aborts (an orphan lease or a job lost on its way to the queue). + The row always carries `created_at` (`NOT NULL` in the table, and every + caller reads it), so there is no guard: a caller that forgets it fails + in `db/after-commit`, which logs it. Takes the cfg and resolves the + metrics instance from it, so the callback is not tied to an instance + read at registration time." [cfg job outcome] - (when job - (when (ct/inst? (:created-at job)) - (metrics/record-total - cfg - (:name job) - (:queue job) - outcome - (- (inst-ms (ct/now)) (inst-ms (:created-at job))))))) + (metrics/record-total + cfg + (:name job) + outcome + (- (inst-ms (ct/now)) (inst-ms (:created-at job))))) (def schema:job-error "Structured error of a job that did not complete. `type` and `code` are @@ -1007,6 +1073,16 @@ :code :orphan :hint "job execution lease expired"})) +(def lost-error + "Error of a job that was handed to its queue and no worker claimed it + within the dispatcher window. The queue message is gone (`scheduled` + means pushed), so the job is marked `aborted` (system-side terminal, + never retried) with its `end` event in the same transaction instead of + going back to `new`." + (check-job-error {:type :internal + :code :lost + :hint "job was not claimed after being scheduled"})) + (def ^:private decode-job-error-value (sm/decoder schema:job-error sm/json-transformer)) @@ -1036,15 +1112,15 @@ Only transitions a pending row (new/scheduled/retry) to `running`, and requires an exact `scheduled-at` match with the value advertised in the - queue payload, so a stale payload (row rescheduled or claimed in the - meantime) affects 0 rows and must be skipped. Returns the number of - affected rows. + queue payload, so a duplicate or stale payload (the row was claimed in + the meantime, or is already aborted) affects 0 rows and must be + skipped. Returns the number of affected rows. The event carries `attempt` = `retry_num + 1`: the number of the attempt this run consumes from the retry budget." [cfg job-id scheduled-at] (assert (mtx/instance cfg) "missing ::mtx/metrics on the jobs cfg") - (db/tx-run! (own-transaction cfg) + (db/tx-run! (db/own-transaction cfg) (fn [{:keys [::db/conn] :as tx-cfg}] (if-let [{:keys [retry-num]} (db/exec-one! conn [sql:claim-job job-id scheduled-at])] @@ -1076,7 +1152,7 @@ (assert (mtx/instance cfg) "missing ::mtx/metrics on the jobs cfg") (let [job (get-job cfg job-id) encoded (db/json (check-job-error error))] - (db/tx-run! (own-transaction cfg) + (db/tx-run! (db/own-transaction cfg) (fn [{:keys [::db/conn] :as tx-cfg}] (if-let [{:keys [retry-num]} (db/exec-one! conn [sql:retry-job (ct/now) scheduled-at @@ -1088,7 +1164,6 @@ (db/after-commit #(metrics/record-retry tx-cfg (:name job) - (:queue job) reason)) 1) 0))))) @@ -1151,7 +1226,7 @@ [cfg & {:keys [job-id result resource-id]}] (assert (mtx/instance cfg) "missing ::mtx/metrics on the jobs cfg") (check-resource-id resource-id) - (let [n (db/tx-run! (own-transaction cfg) + (let [n (db/tx-run! (db/own-transaction cfg) (fn [{:keys [::db/conn] :as tx-cfg}] (let [job (some-> (db/exec-one! conn [sql:lock-job job-id]) decode-row) @@ -1206,7 +1281,7 @@ (assert (mtx/instance cfg) "missing ::mtx/metrics on the jobs cfg") (let [job (get-job cfg job-id) encoded (db/json (check-job-error error)) - n (db/tx-run! (own-transaction cfg) + n (db/tx-run! (db/own-transaction cfg) (fn [{:keys [::db/conn] :as tx-cfg}] (if (db/exec-one! conn [sql:fail-job (ct/now) encoded job-id]) (do @@ -1239,7 +1314,7 @@ [cfg job-id] (assert (mtx/instance cfg) "missing ::mtx/metrics on the jobs cfg") (let [job (get-job cfg job-id) - n (db/tx-run! (own-transaction cfg) + n (db/tx-run! (db/own-transaction cfg) (fn [{:keys [::db/conn] :as tx-cfg}] (if (db/exec-one! conn [sql:cancel-job (ct/now) job-id]) (do diff --git a/backend/src/app/jobs/metrics.clj b/backend/src/app/jobs/metrics.clj index ab1941b1b7..f5a7742f18 100644 --- a/backend/src/app/jobs/metrics.clj +++ b/backend/src/app/jobs/metrics.clj @@ -5,27 +5,22 @@ ;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns app.jobs.metrics - "Metric names, labels and the jobs backlog sampler. + "Metric names and labels used by the jobs substrate. Labels are bounded by construction, not by allowlist: every value recorded here comes from a fixed call site (a job name from the - registry, a queue from submit, an outcome from the lifecycle - writers), never from params, IDs or exception text. A new value - showing up in Prometheus means new code, which is exactly what you - want to see: folding it into some other bucket would only hide it. + registry, a queue from the dispatcher batch, an outcome from submit or + the lifecycle writers), never from params, IDs or exception text. A new + value showing up in Prometheus means new code, which is exactly what + you want to see: folding it into some other bucket would only hide it. This namespace is intentionally free of job handlers. Callers pass only values already known at their boundary; IDs, props and exception text never become metric labels." (:require [app.common.data :as d] - [app.common.logging :as l] - [app.common.schema :as sm] - [app.db :as db] [app.metrics :as mtx] - [cuerdas.core :as str] - [integrant.core :as ig] - [promesa.exec :as px])) + [cuerdas.core :as str])) (set! *warn-on-reflection* true) @@ -59,29 +54,40 @@ (defn- record-count [metrics id labels amount] (mtx/run! metrics :id id :labels labels :inc amount)) -(defn record-submitted - [cfg name queue] - (record-count (mtx/instance cfg) :jobs-submitted - [(name-label name) (queue-label queue)] +(defn- record-submit + "One measurement with three outcomes: the submit created a row, skipped + it because a live job covered the key (`::jobs/skip-if-active`, any status + from `new` to `retry`), or refreshed the not-started row that covered it + (`::jobs/debounce`). The outcome comes from a fixed call site, so it has + no fallback, unlike `outcome-label`, which would say `failed` for a + submission that cannot fail." + [cfg outcome name] + (record-count (mtx/instance cfg) :jobs-submit + [(name-label name) (d/name outcome)] 1)) +(defn record-created + [cfg name] + (record-submit cfg :created name)) + (defn record-skipped - [cfg name queue] - (record-count (mtx/instance cfg) :jobs-skipped - [(name-label name) (queue-label queue)] - 1)) + [cfg name] + (record-submit cfg :skipped name)) + +(defn record-refreshed + [cfg name] + (record-submit cfg :refreshed name)) (defn record-dispatched - [cfg name queue amount] + [cfg name amount] (record-count (mtx/instance cfg) :jobs-dispatched - [(name-label name) (queue-label queue)] + [(name-label name)] amount)) (defn record-retry - [cfg name queue reason] + [cfg name reason] (record-count (mtx/instance cfg) :jobs-retries - [(name-label name) (queue-label queue) - (retry-reason-label reason)] + [(name-label name) (retry-reason-label reason)] 1)) (defn record-event @@ -91,53 +97,41 @@ [cfg] (record-count (mtx/instance cfg) :jobs-events-total [] 1)) -(defn record-orphan - [cfg queue] - (record-count (mtx/instance cfg) :jobs-orphaned - [(queue-label queue)] - 1)) - -(defn record-rescheduled - [cfg queue] - (record-count (mtx/instance cfg) :jobs-rescheduled - [(queue-label queue)] - 1)) - -(defn record-queue-wait - [cfg name queue millis] - (record-value (mtx/instance cfg) :jobs-queue-wait-timing - [(name-label name) (queue-label queue)] +(defn record-claim-wait + "How late a claim happened: `now - scheduled_at` at the moment the + runner takes the job, including the gap the dispatcher added. It is not + time spent inside a queue, hence the name." + [cfg name millis] + (record-value (mtx/instance cfg) :jobs-claim-wait-timing + [(name-label name)] (max 0 (long millis)))) (defn record-execution - [cfg name queue millis] + [cfg name millis] (record-value (mtx/instance cfg) :jobs-execution-timing - [(name-label name) (queue-label queue)] + [(name-label name)] (max 0 (long millis)))) (defn record-legacy-execution "Histogram `penpot_tasks_timing`, from the pre-jobs `task` queue. - It only carries the job name label, with no queue to tell jobs of the - same name apart. Use `record-execution`, which adds the queue label, - for anything new. Kept exported only so the dashboards and the alerts - that already read it keep working: like every other jobs metric, - the label is the raw job name." + It carries the job name label, like every other jobs metric. Kept + exported only so the dashboards and the alerts that already read it + keep working: the label is the raw job name." [cfg name millis] (record-value (mtx/instance cfg) :tasks-timing [(mtx/label name "other")] (max 0 (long millis)))) (defn record-total - [cfg name queue outcome millis] + [cfg name outcome millis] (record-value (mtx/instance cfg) :jobs-total-timing - [(name-label name) (queue-label queue) - (outcome-label outcome)] + [(name-label name) (outcome-label outcome)] (max 0 (long millis)))) (defn record-dispatcher-batch [cfg stage outcome millis] - (record-value (mtx/instance cfg) :jobs-dispatcher-timing + (record-value (mtx/instance cfg) :jobs-dispatcher-batch-timing [(stage-label stage) (outcome-label outcome)] (max 0 (long millis)))) @@ -155,8 +149,9 @@ amount)) (defn record-gc-duration + "One pass of the jobs garbage collector, `expired` or `retained`." [cfg kind millis] - (record-value (mtx/instance cfg) :jobs-gc-timing + (record-value (mtx/instance cfg) :jobs-gc-pass-timing [(gc-kind-label kind)] (max 0 (long millis)))) @@ -215,74 +210,3 @@ (record-value (mtx/instance cfg) :file-data-offload-timing [(offload-result-label result)] (max 0 (long millis)))) - -(def ^:private backlog-statuses - ["new" "scheduled" "running" "retry" "completed" "failed" "cancelled" "aborted"]) - -(def ^:private sql:backlog - "SELECT status, count(*) AS n - FROM job - GROUP BY status") - -(def ^:private sql:oldest-pending - "SELECT coalesce(extract(epoch FROM (now() - min(created_at))), 0) AS age - FROM job - WHERE status IN ('new', 'scheduled', 'retry')") - -(defn sample-backlog - "Update the current job backlog gauges. The query is intentionally small - and runs periodically rather than from the worker hot path." - [{:keys [::db/pool] :as cfg}] - (let [rows (db/exec! pool [sql:backlog]) - by-id (into {} (map (juxt :status #(or (:n %) 0))) rows) - age (-> (db/exec-one! pool [sql:oldest-pending]) - :age - (max 0) - (double)) - metrics (mtx/instance cfg)] - (doseq [status backlog-statuses] - (record-value metrics :jobs-backlog [status] (get by-id status 0))) - (record-value metrics :jobs-oldest-pending-age [] age) - age)) - -(defn- sample-jobs-metrics [cfg] - (try - (sample-backlog cfg) - (catch Throwable cause - (l/warn :hint "unable to sample jobs metrics" :cause cause)))) - -(def ^:private sample-interval-ms 30000) - -(defn create-sampler - "Create the daemon scheduler used by the jobs metrics component." - [cfg] - (let [scheduler (px/scheduled-executor - :parallelism 1 - :factory (px/thread-factory :prefix "penpot/jobs-metrics/" - :daemon true)) - sample (fn sample [] - (try - (sample-jobs-metrics cfg) - (finally - (px/schedule scheduler sample-interval-ms sample))))] - (px/schedule scheduler 0 sample) - scheduler)) - -(def ^:private schema:sampler - [:map - ::db/pool - ::mtx/metrics]) - -(defmethod ig/assert-key ::sampler - [_ cfg] - (assert (sm/check schema:sampler cfg))) - -(defmethod ig/init-key ::sampler - [_ {:keys [::db/pool] :as cfg}] - (if (db/read-only? pool) - (l/warn :hint "not started (db is read-only)") - (create-sampler cfg))) - -(defmethod ig/halt-key! ::sampler - [_ sampler] - (some-> sampler px/shutdown-now)) diff --git a/backend/src/app/loggers/audit.clj b/backend/src/app/loggers/audit.clj index 0d9a0b560f..1d9472baaf 100644 --- a/backend/src/app/loggers/audit.clj +++ b/backend/src/app/loggers/audit.clj @@ -21,8 +21,6 @@ [app.http :as-alias http] [app.http.access-token :as-alias actoken] [app.jobs :as jobs] - [app.loggers.audit.consumer-task :as consumer] - [app.loggers.audit.tasks :as-alias tasks] [app.loggers.webhooks :as-alias webhooks] [app.rpc :as-alias rpc] [app.setup :as-alias setup] @@ -88,6 +86,17 @@ :sync-failed :snapshot-complete}) +(def ^:private telemetry-sources + #{"telemetry:backend" "telemetry:frontend"}) + +(defn telemetry-shadow-source? + "True for the anonymized shadow sources of the telemetry consumer. A + selective consumer must skip them: the full row carries the event + already, and the shadow keeps the event name, so processing it would + ingest the same event twice (once full, once anonymized)." + [source] + (contains? telemetry-sources (str source))) + (def ^:private safe-backend-context-keys #{:version :initiator @@ -282,13 +291,13 @@ (update :props db/tjson) (update :context db/tjson) (update :ip-addr db/inet)) - params (select-keys params event-keys) - row (db/insert! cfg :audit-log params)] - ;; The consumer queue follows the row in the same transaction: one - ;; pending per enabled consumer. A rollback takes both on its way - ;; out, so a row never outlives its queue. - (consumer/queue-pendings cfg [(:id row)]) - row)) + params (select-keys params event-keys)] + ;; The producer knows nothing about the consumers: one row, and each + ;; consumer reads it from its own cursor. The row and the cursor + ;; position commit independently, so a consumer can process a row + ;; only after it is visible here (the snapshot cut in the consumer + ;; guarantees it). + (db/insert! cfg :audit-log params))) (def ^:private xf:filter-telemetry-props "Transducer that keeps only map entries whose values are UUIDs, @@ -318,11 +327,11 @@ :props (json/encode (:props event) :key-fn json/write-camel-key) :context (json/encode (:context event) :key-fn json/write-camel-key))) - (when (contains? cf/flags :audit-log) + (when cf/audit-persist? (append-audit-entry cfg event)) - (when (contains? cf/flags :telemetry) - ;; NOTE: when both audit-log and telemetry are enabled, events are stored + (when cf/telemetry-enabled? + ;; NOTE: when both persistence and telemetry are enabled, events are stored ;; twice: once with full details (above) and once stripped of props and ;; ip-addr, tagged with source="telemetry" so the telemetry task can ;; collect and ship them. The profile-id is preserved (UUIDs are already @@ -341,7 +350,7 @@ (assoc :ip-addr "0.0.0.0"))] (append-audit-entry cfg event))) - (when (and (contains? cf/flags :webhooks) + (when (and cf/webhooks-enabled? (::webhooks/event? event)) (let [batch-key (::webhooks/batch-key event) batch-timeout (::webhooks/batch-timeout event) @@ -350,14 +359,17 @@ (ifn? batch-key) (dm/str label ":" (batch-key (::rpc/params event))) (string? batch-key) (dm/str label ":" batch-key) :else label) - dedupe? (boolean (and batch-key batch-timeout))] + debounce? (boolean (and batch-key batch-timeout))] (jobs/submit cfg {::jobs/name :process-webhook-event ::jobs/queue :webhooks ::jobs/max-retries 0 ::jobs/delay (or batch-timeout 0) - ::jobs/dedupe dedupe? + ;; A batch slides its window with every event and + ;; delivers the last payload, so the receiver is + ;; told once about the whole burst. + ::jobs/debounce debounce? ::jobs/label label ;; The event travels as an opaque transit blob: ;; transit preserves the instant/UUID/set types @@ -373,13 +385,13 @@ event)) (defn submit* - "A public API, lower-level than submit, assumes all required fields are filled" + "A public API, lower-level than submit, assumes all required fields are + filled. A failure propagates: an event that cannot be stored never + reaches any consumer, and nothing records that it was lost. Swallowing + it here made that invisible." [cfg event] - (try - (let [event (check-event event)] - (db/tx-run! cfg process-event event)) - (catch Throwable cause - (l/error :hint "unexpected error processing event" :cause cause)))) + (let [event (check-event event)] + (db/tx-run! cfg process-event event))) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; PUBLIC API @@ -454,12 +466,14 @@ (defn- coerce-profile-id "Normalize a hand-written `::audit/profile-id` override to a uuid. - `schema:event` requires a uuid and `submit*` swallows the validation - error, so a value that is not a uuid loses the event instead of - failing loudly. Commands read the override from places that are not - typed by us (token claims, stringly-typed drivers), so a string has - to be accepted. Anything that cannot become a uuid is discarded, and - the event falls back to the caller, which is always a valid uuid." + `schema:event` requires a uuid and `submit*` propagates the validation + error; `wrap-audit` runs after the command commits (moving it inside the + transaction is project P8 in `TODO.md`), so an invalid override would fail + the response over an already committed action. Commands read the + override from places that are not typed by us (token claims, + stringly-typed drivers), so a string has to be accepted. Anything that + cannot become a uuid is discarded, and the event falls back to the + caller, which is always a valid uuid." [v] (let [coerced (cond ;; Fast path: the override comes straight from a `profile` @@ -478,15 +492,57 @@ coerced)) -(def ^:private sql:team-id-by-project - "SELECT id FROM team - WHERE id = (SELECT team_id FROM project WHERE id = ?)") +(def ^:private sql:team-ids-by-projects + "SELECT team_id + FROM project + WHERE id = ANY(?::uuid[])") -(def ^:private sql:team-id-by-file - "SELECT t.id FROM team AS t - JOIN project AS p ON (p.team_id = t.id) - JOIN file AS f ON (f.project_id = p.id) - WHERE f.id = ?") +(def ^:private sql:team-ids-by-files + "SELECT p.team_id AS team_id + FROM file AS f + JOIN project AS p ON (p.id = f.project_id) + WHERE f.id = ANY(?::uuid[])") + +(def ^:private xf:team-ids + "Transducer: pull the team id out of a team/project/file resolution + row. Allocated once so hot paths (every audited RPC) do not rebuild + it per call." + (map :team-id)) + +(defn- resolve-team-ids + "Resolve one bag of project or file ids to team ids with one ANY(?) + query on a plain connection. The result holds one tiny row per id, + so neither a cursor nor a transaction is needed." + [cfg kind ids] + ;; Fetch inside the connection, reduce outside it: the rows are + ;; already materialized, so the pool connection is held as briefly + ;; as possible. + (->> (db/run! cfg + (fn [{:keys [::db/conn]}] + (db/exec! conn [(case kind + :project sql:team-ids-by-projects + :file sql:team-ids-by-files) + (db/create-array conn "uuid" ids)]))) + (into #{} xf:team-ids))) + +(defn get-team-ids + "Resolve a mixed bag of team, project and file ids to team ids with + two batched lookups. Producers of cascade-style events call it so + consumers never join production tables. The pure part runs without + touching the database; each lookup resolves its connection lazily, + only when it has ids (reusing the caller transaction). Every id + reaching this function is a UUID already (RPC params and frontend + ingest travel as transit, jobs read DB rows): anything else is a + bug at the source, never coerced here." + [cfg {:keys [team-ids project-ids file-ids]}] + (let [direct (into #{} team-ids)] + (-> direct + (into (if (seq project-ids) + (resolve-team-ids cfg :project project-ids) + #{})) + (into (if (seq file-ids) + (resolve-team-ids cfg :file file-ids) + #{}))))) (def ^:private read-event-names "Read commands excluded from team-id resolution on top of the `get-*` @@ -513,9 +569,9 @@ (let [project-id (:project-id props) file-id (:file-id props) team-id (or (when (some? project-id) - (:id (db/exec-one! cfg [sql:team-id-by-project project-id]))) + (first (get-team-ids cfg {:project-ids [project-id]}))) (when (some? file-id) - (:id (db/exec-one! cfg [sql:team-id-by-file file-id]))))] + (first (get-team-ids cfg {:file-ids [file-id]}))))] (cond-> props (some? team-id) (assoc :team-id team-id))))) @@ -622,7 +678,7 @@ event is built by hand and does not pass through `prepare-rpc-event`, so its props are passed as-is minus nils." [cfg params event-name props] - (when (contains? cf/flags :audit-log) + (when cf/audit-persist? (submit cfg (-> (event-from-rpc-params params) (assoc :name event-name) @@ -647,9 +703,9 @@ command line helpers because this skips all webhooks and telemetry logic." [cfg event] - (when (contains? cf/flags :audit-log) - ;; Transactional (not a bare connection): the row and its - ;; pendings commit or vanish together (see append-audit-entry). + (when cf/audit-persist? + ;; Transactional (not a bare connection): the row commits or + ;; vanishes whole (see append-audit-entry). (let [tnow (ct/now) event (-> event (assoc :created-at tnow) diff --git a/backend/src/app/loggers/audit/consumer_task.clj b/backend/src/app/loggers/audit/consumer_task.clj deleted file mode 100644 index 9d1146bf22..0000000000 --- a/backend/src/app/loggers/audit/consumer_task.clj +++ /dev/null @@ -1,689 +0,0 @@ -;; This Source Code Form is subject to the terms of the Mozilla Public -;; License, v. 2.0. If a copy of the MPL was not distributed with this -;; file, You can obtain one at http://mozilla.org/MPL/2.0/. -;; -;; Copyright (c) KALEIDOS SUBSIDIARY SL - -(ns app.loggers.audit.consumer-task - (:require - [app.common.exceptions :as ex] - [app.common.logging :as l] - [app.common.schema :as sm] - [app.common.time :as ct] - [app.common.transit :as t] - [app.common.uri :as u] - [app.common.uuid :as uuid] - [app.config :as cf] - [app.db :as db] - [app.db.sql :as sql] - [app.http.client :as http] - [app.jobs :as jobs] - [app.metrics :as mtx] - [app.nitrate :as nitrate] - [app.setup :as-alias setup] - [integrant.core :as ig] - [promesa.exec :as px])) - -;; This task drains the audit log buffer into the long-term archives. -;; -;; Two separate jobs share this namespace and the queue helpers, one -;; per destination: `:audit-log-consumer-nexus` and -;; `:audit-log-consumer-admin-console`. Each job drains only its own queue -;; and deletes its pendings on completion; whoever deletes the last -;; pending also deletes the record ("last one out turns off the -;; light"), so no separate GC task is needed. Housekeeping (reaping -;; disabled pendings, sweeping orphans) lives in both jobs on purpose: -;; only the enabled job runs, so it must clean up for the rest. - -;; Consumer ids are fixed constants, never reuse one: retired -;; pendings are reaped by id, so a reused id would inherit the -;; previous owner's queue. -(def consumer-nexus 1) -(def consumer-admin-console 2) - -(def ^:private consumers - [{:id consumer-nexus :flag :nexus :label "nexus"} - {:id consumer-admin-console :flag :admin-console :label "admin-console"}]) - -(def ^:private xf:enabled-consumers - "Static transducer: keep the consumers whose send flag is on. - Allocated once so hot paths (every audited RPC) do not rebuild - it per call; the flag lookup itself still reads the live flags." - (comp (filter #(contains? cf/flags (:flag %))) - (map :id))) - -(defn enabled-consumers - "Consumers allowed to work right now, from the send flags. Public: - producers queue every record for exactly these consumers." - [] - (into [] xf:enabled-consumers consumers)) - -(defn queue-pendings - "Queue records for every enabled consumer, nothing when there is - nothing to queue or no consumer is on. One multi-row statement for - all ids instead of one per consumer: this runs inside every audited - RPC and the bulk frontend ingest. Runs inside the row transaction: - a rollback takes both." - [cfg ids] - (let [consumers (enabled-consumers)] - (when (and (seq ids) (seq consumers)) - (db/insert-many! cfg :audit-log-pending - [:audit-log-id :consumer] - (for [id ids - consumer consumers] - [id consumer]) - {::sql/on-conflict-do-nothing true})))) - -(def ^:private consumer-labels - (into {} (map (juxt :id :label)) consumers)) - -(defn- consumer-label - [consumer] - (get consumer-labels consumer)) - -(defn- decode-row - [{:keys [props ip-addr context] :as row}] - (cond-> row - (db/pgobject? props) - (assoc :props (db/decode-transit-pgobject props)) - - (db/pgobject? context) - (assoc :context (db/decode-transit-pgobject context)) - - (db/pgobject? ip-addr "inet") - (assoc :ip-addr (db/decode-inet ip-addr)))) - -(def ^:private event-keys - [:type - :name - :source - :created-at - :tracked-at - :profile-id - :ip-addr - :props - :context]) - -(defn- row->event - [row] - (select-keys row event-keys)) - -(defn- record-count - [metrics id labels amount] - (mtx/run! metrics :id id :labels labels :inc amount)) - -(defn- truncate-body - [body limit] - (when (some? body) - (let [s (str body)] - (if (> (count s) limit) - (subs s 0 limit) - s)))) - -;; NOTE: deliberately no SKIP LOCKED here. Batches must reach the -;; destinations in chronological order (Nexus numbers by arrival), -;; so every consumer takes the same total order and a second -;; overlapping run waits for the first to commit, then sees whether -;; it was last. The brief blocking is accepted; skipping rows could -;; reorder archive arrival. -(def ^:private sql:pending-chunk - "SELECT a.* - FROM audit_log_pending p - JOIN audit_log a ON a.id = p.audit_log_id - WHERE p.consumer = ?::smallint - ORDER BY a.created_at ASC, a.id ASC - LIMIT 128 - FOR UPDATE OF a") - -(def ^:private sql:delete-pendings - "DELETE FROM audit_log_pending - WHERE consumer = ?::smallint - AND audit_log_id = ANY(?)") - -(def ^:private sql:delete-drained - "DELETE FROM audit_log a - WHERE a.id = ANY(?) - AND NOT EXISTS ( - SELECT 1 FROM audit_log_pending p - WHERE p.audit_log_id = a.id - )") - -(defn- get-pending-rows - [{:keys [::db/conn] :as cfg} consumer] - (->> (db/exec! conn [sql:pending-chunk consumer]) - (not-empty))) - -(defn- complete-batch - "Delete our pendings for the batch, then the records nobody is - waiting for anymore. Two statements on purpose: inside a single - statement the NOT EXISTS would still see the pending just deleted - and never delete a record." - [{:keys [::db/conn]} consumer rows] - (let [ids (db/create-array conn "uuid" (map :id rows))] - (db/exec! conn [sql:delete-pendings consumer ids]) - (db/exec! conn [sql:delete-drained ids]))) - -(defn- generate-nexus-uri - "Joins relative path segments to the Nexus backend URI, same shape - as the admin-console helper in `app.nitrate` (there the ingest path - is `api/audit-log`; here it is `api/audit`). - Segments must not start with `/`. Returns nil when no base URI is - configured, so the caller keeps the `:task-not-configured` guard." - [& segments] - (when-let [base (cf/get :nexus-uri)] - (apply cf/join-uri base segments))) - -(defn- send-to-nexus - "POST events to Nexus. Returns the number of events sent; throws on - transport failure or non-204 refusal, carrying the attempted count - as :events in the error data so the caller records the failure. - No metrics or logs here: the runner reports the aborted job." - [{:keys [::uri] :as cfg} events] - (let [n (count events) - skey (-> cfg ::setup/shared-keys :nexus) - body (t/encode {:events events}) - params {:uri uri - :timeout 12000 - :method :post - :headers {"content-type" "application/transit+json" - "origin" (str (cf/get :public-uri)) - "x-shared-key" (str "nexus " skey)} - :body body} - resp (try - (http/req cfg params {:skip-ssrf-check? true}) - (catch Throwable cause - (throw (ex/raise :type :internal - :code :nexus-send-failed - :hint "unable to send events to nexus" - :cause cause - :events n))))] - (if (= (:status resp) 204) - n - (throw (ex/raise :type :internal - :code :consumer-chunk-refused - :hint "nexus refused archive chunk" - :resp-status (:status resp) - :resp-body (truncate-body (:body resp) 500) - :events n))))) - -(def ^:private event-names-for-admin-console - #{"accept-organization-invitation" - "accept-organization-invitation-from" - "accept-team-invitation" - "accept-team-invitation-from" - "add-member-to-organization" - "add-team-to-organization" - "assign-organization-attribute" - "auditlog-exported" - "auditlog-guide-downloaded" - "cancel-organization-invitation" - "create-export-binfile-job" - "create-file" - "create-organization" - "create-organization-attribute" - "create-organization-invitation" - "create-project" - "create-team" - "create-team-access-request" - "create-team-invitation" - "create-team-invitations" - "create-webhook" - "delete-file" - "delete-font" - "delete-organization" - "delete-organization-attribute" - "delete-organization-member" - "delete-project" - "delete-team" - "delete-team-invitation" - "delete-team-member" - "leave-team" - "move-project" - "move-team-to-organization" - "organization-sso-auth-failed" - "organization-sso-auth-started" - "organization-sso-auth-succeeded" - "organization-sso-token-issued" - "organization-sso-token-revoked" - "permanently-delete-team-files" - "remove-organization-team" - "remove-team-from-organization" - "rename-organization" - "rename-project" - "restore-deleted-team-files" - "sso-config-deleted" - "sso-config-tested" - "success-activate-organization-sso" - "success-apply-organization-sso-changes" - "unassign-organization-attribute" - "update-organization-attribute" - "update-organization-invitation" - "update-organization-permissions" - "update-team-invitation" - "update-team-invitation-role" - "update-team-member-role" - "update-team-photo" - "verify-token"}) - -;; Props id key -> email prop filled from profile.email (admin-console path only). -;; Some producers store ids as UUID strings (e.g. :user-who-send-invitation). -(def ^:private id->email-prop - {:profile-id :profile-email - :user-id :user-email - :member-id :member-email - :user-who-send-invitation :user-who-send-invitation-email}) - -(defn- collect-team-ids-needing-name - [events] - (into #{} - (comp (map :props) - (keep (fn [props] - (when-let [team-id (and (not (contains? props :team-name)) - (uuid/coerce (:team-id props)))] - team-id)))) - events)) - -(defn- collect-profile-ids-needing-email - [events] - (into #{} - (mapcat (fn [{:keys [props]}] - (keep (fn [[id-key email-key]] - (when-not (contains? props email-key) - (uuid/coerce (get props id-key)))) - id->email-prop))) - events)) - -(defn- collect-actor-ids-needing-profile - "Actor ids (event :profile-id) that still need top-level :profile-email - and/or :profile-name. Skips uuid/zero and non-uuids." - [events] - (into #{} - (keep (fn [event] - (when (or (not (contains? event :profile-email)) - (not (contains? event :profile-name))) - (when-let [id (uuid/coerce (:profile-id event))] - (when-not (uuid/zero? id) - id))))) - events)) - -(defn- load-team-names - [conn ids] - (if (seq ids) - (let [arr (db/create-array conn "uuid" ids) - rows (db/exec! conn ["SELECT id, name FROM team WHERE id = ANY(?)" arr])] - (into {} (map (juxt :id :name) rows))) - {})) - -(defn- load-profiles - "Batch-load profile email and fullname (including soft-deleted)." - [conn ids] - (if (seq ids) - (let [arr (db/create-array conn "uuid" ids) - rows (db/exec! conn ["SELECT id, email, fullname FROM profile WHERE id = ANY(?)" arr])] - (into {} (map (juxt :id identity) rows))) - {})) - -(defn- maybe-assoc-team-name - [event team-names] - (let [props (:props event)] - (if-let [team-name (and (not (contains? props :team-name)) - (when-let [team-id (uuid/coerce (:team-id props))] - (get team-names team-id)))] - (assoc-in event [:props :team-name] team-name) - event))) - -(defn- maybe-assoc-emails - [event profiles] - (update event :props - (fn [props] - (reduce-kv (fn [props id-key email-key] - (if-let [email (and (not (contains? props email-key)) - (when-let [id (uuid/coerce (get props id-key))] - (:email (get profiles id))))] - (assoc props email-key email) - props)) - props - id->email-prop)))) - -(defn- maybe-assoc-actor-profile - "Fill-only top-level :profile-email / :profile-name from the event - actor (:profile-id). Omits keys when the id is missing, zero, the - profile row is absent, or the corresponding column is empty - (fullname defaults to \"\" in the schema)." - [event profiles] - (let [id (uuid/coerce (:profile-id event))] - (if (or (nil? id) (uuid/zero? id)) - event - (if-let [profile (get profiles id)] - (cond-> event - (and (not (contains? event :profile-email)) - (not-empty (:email profile))) - (assoc :profile-email (:email profile)) - - (and (not (contains? event :profile-name)) - (not-empty (:fullname profile))) - (assoc :profile-name (:fullname profile))) - event)))) - -(defn- enrich-events-for-admin-console - "Enrich allowlisted events before the admin-console ingest: - 1. Fill missing :team-name from team id (including soft-deleted). - 2. Fill missing emails from matching id keys in props (including - soft-deleted profiles): :profile-email, :user-email, :member-email, - :user-who-send-invitation-email. - 3. Fill missing top-level :profile-email / :profile-name from the - event actor (:profile-id), including soft-deleted profiles; omit - when there is no usable actor id or the value is empty." - [{:keys [::db/conn]} events] - (let [team-ids (collect-team-ids-needing-name events) - profile-ids (into (collect-profile-ids-needing-email events) - (collect-actor-ids-needing-profile events)) - team-names (load-team-names conn team-ids) - profiles (load-profiles conn profile-ids)] - (mapv (fn [event] - (-> event - (maybe-assoc-team-name team-names) - (maybe-assoc-emails profiles) - (maybe-assoc-actor-profile profiles))) - events))) - -(defn- send-to-admin-console - "Ingest rows into the admin console. Returns {:sent :skipped} - (filtered-out rows count as processed for the queue); throws - carrying the attempted count as :events, like send-to-nexus. Enrich - failures propagate untouched: a local DB/decode failure is ours, not - the destination's, and must not count as a send failure." - [cfg rows] - (let [events (filterv #(contains? event-names-for-admin-console (:name %)) rows) - total (count rows) - skipped (- total (count events)) - uri-host (some-> (cf/get :admin-console-uri) u/uri u/get-domain)] - (l/dbg :hint "consumer events chunk ready" - :consumer "admin-console" - :uri-host uri-host - :total total - :skipped skipped) - (if (empty? events) - {:sent 0 :skipped skipped} - ;; Enrich first, outside the try: a local DB/decode failure is - ;; ours, not the destination's, and must not count as - ;; admin-console/failed. - (let [enriched (enrich-events-for-admin-console cfg events) - n (count events)] - (try - (nitrate/call cfg :ingest-audit-log {:events enriched}) - (l/dbg :hint "consumer events chunk sent" - :consumer "admin-console" - :uri-host uri-host - :sent n - :skipped skipped) - {:sent n :skipped skipped} - (catch Throwable cause - (l/dbg :hint "consumer events chunk failed" - :consumer "admin-console" - :uri-host uri-host - :events n - :code :admin-console-send-failed) - (throw (ex/raise :type :internal - :code :admin-console-send-failed - :hint "unable to archive events to the admin console" - :cause cause - :events n)))))))) - -(defn- consumer-nexus-chunk - "Send one pending chunk to Nexus and complete it. Returns {:sent n} - or nil when the queue is empty; throws on refusal. Takes a - transactional cfg: the caller opens one transaction per chunk. No - metrics here: a normal return means the transaction committed, so - the caller records the success." - [cfg] - (db/exec! cfg ["SET LOCAL lock_timeout = '30s'"]) - (when-let [rows (get-pending-rows cfg consumer-nexus)] - (let [events (mapv row->event (mapv decode-row rows)) - ;; send-to-nexus returns the count or throws: a refusal - ;; aborts the job, the rows stay for the next tick. - n (send-to-nexus cfg events)] - - (l/trc :hint "consumer events chunk" - :consumer "nexus" - :events n) - - (complete-batch cfg consumer-nexus rows) - {:sent n}))) - -(defn- consumer-admin-console-chunk - "Send one pending chunk to the admin console and complete it. Returns - {:sent :skipped} or nil when the queue is empty; throws on failure. - Takes a transactional cfg, no metrics here: same contract as - consumer-nexus-chunk." - [cfg] - (db/exec! cfg ["SET LOCAL lock_timeout = '30s'"]) - (when-let [rows (get-pending-rows cfg consumer-admin-console)] - (let [decoded (mapv decode-row rows) - _ (l/trc :hint "consumer events chunk" - :consumer "admin-console" - :events (count decoded)) - admin-console-res (send-to-admin-console cfg decoded)] - ;; Filtered-out rows count as processed for the - ;; admin-console queue: their pending is done, the - ;; record stays until every consumer is done. - (complete-batch cfg consumer-admin-console rows) - admin-console-res))) - -(defn- drain-consumer - "Drain one consumer queue chunk by chunk, one transaction per chunk. - Returns totals; a chunk the destination refuses (or that throws) - aborts the drain with a throw, and the committed chunks so far - stay sent." - [cfg send-chunk] - (loop [sent 0 - skipped 0 - chunks 0] - ;; One transaction per chunk: fetch, send and complete commit (or - ;; roll back) together, and the next chunk sees a fresh snapshot. - (if-let [res (db/tx-run! cfg send-chunk)] - (do - (jobs/heartbeat cfg) - (px/sleep 100) - (recur (+ sent ^long (:sent res)) - (+ skipped ^long (:skipped res 0)) - (inc chunks))) - {:sent sent :skipped skipped :chunks chunks}))) - -(defn- sql:reap-pendings - [consumer] - (str "DELETE FROM audit_log_pending" - " WHERE consumer = " consumer - " AND audit_log_id IN (SELECT audit_log_id FROM audit_log_pending" - " WHERE consumer = " consumer - " ORDER BY audit_log_id LIMIT 5000)")) - -(def ^:private sql:sweep-orphans - "DELETE FROM audit_log a - WHERE a.id IN (SELECT a2.id FROM audit_log a2 - WHERE NOT EXISTS (SELECT 1 FROM audit_log_pending p - WHERE p.audit_log_id = a2.id) - ORDER BY a2.id LIMIT 5000)") - -(defn- delete-batched - "Run a batched DELETE until it removes nothing. Returns the total." - [cfg sql] - (loop [total 0] - (let [n (db/get-update-count (db/exec-one! cfg [sql]))] - (if (pos? n) - (do - (px/sleep 100) - (recur (+ total ^long n))) - total)))) - -(defn- reap-disabled - "Drop the pendings of consumers that are off: a disabled destination - will never drain them, and without this they would pin their records - forever. This also absorbs any later deactivation, with no manual step. - Dropped pendings count as discarded: their rows may still be archived - by an enabled consumer, or swept unsent when nothing waits for them." - [cfg] - (let [total (->> consumers - (remove #(contains? cf/flags (:flag %))) - (map #(delete-batched cfg (sql:reap-pendings (:id %)))) - (reduce + 0))] - (when (pos? total) - (record-count (mtx/instance cfg) :audit-consumer-discarded ["reaped"] total)) - total)) - -(defn- sweep-orphans - "Delete records nobody waits for anymore: rows born while no - consumer was on, rows left pending-less by the reap, and leftovers - archived by the old task. Batched and normally a no-op. Deleted rows - count as discarded: on a live system they were never archived." - [cfg] - (let [total (delete-batched cfg sql:sweep-orphans)] - (when (pos? total) - (record-count (mtx/instance cfg) :audit-consumer-discarded ["swept"] total)) - total)) - -(declare execute-audit-log-consumer-nexus) -(declare execute-audit-log-consumer-admin-console) - -(def schema:audit-log-consumer-nexus-params - "Overrides for the nexus job repl invocation defaults. - Cron submits {}; system deps (pool, shared-keys, http) live on the - job-def cfg closed over by the handler, not in job params. - Closed on purpose: a stale key (like the old :enabled) must fail - validation loudly instead of skipping silently." - [:map {:closed true} - [:uri {:optional true} ::sm/uri]]) - -(def schema:audit-log-consumer-admin-console-params - "Params of the admin-console job: none. Cron submits {}; system deps - (pool, admin-console client) live on the job-def cfg closed over by - the handler, not in job params. Closed on purpose, same as the nexus - schema." - [:map {:closed true}]) - -(defmethod ig/assert-key ::nexus-job-def - [_ cfg] - (assert (db/pool? (::db/pool cfg)) "expected a valid database pool") - (assert (http/client? (::http/client cfg)) "expected a valid http client") - (assert (mtx/metrics? (::mtx/metrics cfg)) "expected a valid metrics instance")) - -(defmethod ig/init-key ::nexus-job-def - [_ cfg] - ;; Effective destination, once at boot: per-run logs carry counts, - ;; the runner says the job started, this says where it sends. - (l/dbg :hint "audit log consumer job initialized" - :consumer "nexus" - :uri-host (some-> (cf/get :nexus-uri) u/uri u/get-domain)) - {::jobs/name :audit-log-consumer-nexus - ::jobs/schema schema:audit-log-consumer-nexus-params - ::jobs/handler - (fn [_context params] - (execute-audit-log-consumer-nexus cfg params)) - ::jobs/decoder (sm/decoder schema:audit-log-consumer-nexus-params sm/json-transformer) - ::jobs/validator (sm/validator schema:audit-log-consumer-nexus-params)}) - -(defmethod ig/assert-key ::admin-console-job-def - [_ cfg] - ;; No assert on the admin-console client: it inits to nil without the - ;; :admin-console flag, and the job skips on its flag anyway. - (assert (db/pool? (::db/pool cfg)) "expected a valid database pool") - (assert (mtx/metrics? (::mtx/metrics cfg)) "expected a valid metrics instance")) - -(defmethod ig/init-key ::admin-console-job-def - [_ cfg] - ;; Same as the nexus job-def: destination host once at boot. The - ;; ingest uri is resolved by nitrate/call at send time (it raises - ;; when missing); here it is display-only. - (l/dbg :hint "audit log consumer job initialized" - :consumer "admin-console" - :uri-host (some-> (cf/get :admin-console-uri) u/uri u/get-domain)) - {::jobs/name :audit-log-consumer-admin-console - ::jobs/schema schema:audit-log-consumer-admin-console-params - ::jobs/handler - (fn [_context params] - (execute-audit-log-consumer-admin-console cfg params)) - ::jobs/decoder (sm/decoder schema:audit-log-consumer-admin-console-params sm/json-transformer) - ::jobs/validator (sm/validator schema:audit-log-consumer-admin-console-params)}) - -(defn- run-single-consumer - "Drain one consumer queue, then housekeeping (reap disabled - pendings, sweep orphans) and the per-run summary. Housekeeping lives - in both jobs: only the enabled job runs, so it must reap the queues - of the consumers that are off; both halves are idempotent deletes. - A refused chunk (or a throw) aborts the job: the committed chunks - stay sent, the rest waits for the next tick, and the cron retries. - Every effect is rows and logs (drain metrics land here, outside - every chunk transaction)." - [cfg consumer send-chunk tpoint] - (let [label (consumer-label consumer) - metrics (mtx/instance cfg) - reaped (reap-disabled cfg) - ;; All drain metrics land here, outside every chunk - ;; transaction. Success is recorded on the totals: a normal - ;; chunk return means its transaction committed (a rollback - ;; always throws, taking the failure path instead). The send - ;; fns carry the attempted count as :events; anything else (a - ;; local DB failure) is not a destination failure and stays - ;; uncounted. The throw keeps rolling: a failed chunk aborts - ;; the job, the cron retries. - drained (try - (drain-consumer cfg send-chunk) - (catch Throwable cause - (let [n (:events (ex-data cause))] - (when (and (integer? n) (pos? n)) - (record-count metrics :audit-consumer-events [label "failed"] n))) - (throw cause))) - _ (let [{:keys [sent skipped]} drained] - (when (pos? sent) - (record-count metrics :audit-consumer-events [label "archived"] sent)) - (when (pos? skipped) - (record-count metrics :audit-consumer-events [label "skipped"] skipped))) - swept (sweep-orphans cfg) - sent (:sent drained) - skipped (:skipped drained) - chunks (:chunks drained) - ;; Strict binding on purpose: log props evaluate lazily in - ;; the logger thread, so the elapsed must be frozen here. - elapsed (ct/format-duration (tpoint))] - (if (and (zero? sent) - (zero? skipped) - (zero? reaped) - (zero? swept)) - (l/dbg :hint "no pending audit events" - :consumer label) - (l/dbg :hint "events archived" - :consumer label - :elapsed elapsed - :sent sent - :skipped skipped - :chunks chunks - :reaped reaped - :swept swept)))) - -(defn execute-audit-log-consumer-nexus - "Nexus job handler: drain the nexus queue in chunks, then clean up - (heartbeat per iteration: the sent chunk batches can be long)." - [cfg params] - ;; NOTE: this let allows overwrite default configured values from - ;; the repl, when manually invoking the task. - (if-not (contains? cf/flags :nexus) - (l/dbg :hint "audit log consumer (nexus) skipped, flag off") - (let [uri (or (:uri params) (generate-nexus-uri "api/audit")) - ;; Normalize to an uri object; params may carry a plain string - ;; on direct invocations (validation only applies on submit!). - uri (u/uri uri) - cfg (assoc cfg ::uri uri)] - (when-not uri - (ex/raise :type :internal - :code :task-not-configured - :hint "consumer task not configured, missing uri")) - (run-single-consumer cfg consumer-nexus consumer-nexus-chunk (ct/tpoint))))) - -(defn execute-audit-log-consumer-admin-console - "Admin-console job handler: drain the admin-console queue in chunks, - then clean up (heartbeat per iteration: the sent chunk batches can - be long)." - [cfg _params] - (if-not (contains? cf/flags :admin-console) - (l/dbg :hint "audit log consumer (admin-console) skipped, flag off") - (run-single-consumer cfg consumer-admin-console consumer-admin-console-chunk (ct/tpoint)))) diff --git a/backend/src/app/loggers/audit/consumers.clj b/backend/src/app/loggers/audit/consumers.clj new file mode 100644 index 0000000000..9dbc464171 --- /dev/null +++ b/backend/src/app/loggers/audit/consumers.clj @@ -0,0 +1,531 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns app.loggers.audit.consumers + "The mechanics shared by the consumers of the audit log buffer. + + Two consumers read the events of the buffer, each one from its own + position (a cursor) over `audit_log`: + + - the nexus consumer sends them to Nexus. + - the admin-console consumer sends the allowlisted ones to the admin + console. + + Each transport lives in its own namespace (`...consumers.nexus`, + `...consumers.admin-console`); this one + holds what they share: the consumer ids, the catalog, the cursor + (lock, read, advance), the tick mutex, the loop and the metrics. The + garbage collection of the buffer lives in `app.loggers.audit.gc`. + + A consumer is a loop in a thread, not a job: the pass takes the tick + lock, drains up to the frontier it found at its start and sleeps. The + position is the pair `(xid, seq)`: the transaction that wrote the row + and its insertion order. A consumer reads only rows below + `pg_snapshot_xmin(pg_current_snapshot())` (the oldest transaction still + in flight), so it never steps over a row whose transaction is still + open. The producer does not know about consumers: it only inserts the + row, and the GC deletes it once every enabled consumer has passed it." + (:require + [app.common.exceptions :as ex] + [app.common.logging :as l] + [app.common.time :as ct] + [app.config :as cf] + [app.db :as db] + [app.metrics :as mtx] + [promesa.exec :as px])) + +;; Consumer ids are fixed constants, never reuse one: a retired id keeps +;; meaning its old position in `audit_log_consumer_cursor`. +(def nexus 1) +(def admin-console 2) + +(def ^:private consumers + [{:id nexus :flag :nexus :label "nexus"} + {:id admin-console :flag :admin-console :label "admin-console"}]) + +(def ^:private consumers-by-id + (into {} (map (juxt :id identity)) consumers)) + +(defn enabled? + "True when the flag of a consumer is on. The transport components use it + to decide at boot whether to start the loop; the GC uses the cursor table + to decide whose cursor it waits for." + [consumer-id] + (contains? cf/flags (get-in consumers-by-id [consumer-id :flag]))) + +(defn- consumer-label + [consumer] + (get-in consumers-by-id [consumer :label])) + +(defn decode-row + "Decode the transit-encoded columns of an audit-log row. The + transports read the rows of their chunk, so this is shared." + [{:keys [props ip-addr context] :as row}] + (cond-> row + (db/pgobject? props) + (assoc :props (db/decode-transit-pgobject props)) + + (db/pgobject? context) + (assoc :context (db/decode-transit-pgobject context)) + + (db/pgobject? ip-addr "inet") + (assoc :ip-addr (db/decode-inet ip-addr)))) + +(def ^:private event-keys + [:type + :name + :source + :created-at + :tracked-at + :profile-id + :ip-addr + :props + :context]) + +(defn row->event + "The view of an audit-log row that travels to an archive: the decoded + event alone, without the row id or the cursor columns." + [row] + (select-keys row event-keys)) + +(defn record-count + "Record a counter of the consumers metrics." + [metrics id labels amount] + (mtx/run! metrics :id id :labels labels :inc amount)) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; CURSOR +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(def ^:private chunk-size + "Rows per chunk. Kept in a var so tests can shrink it." + 128) + +(def ^:private sql:lock-cursor + "Lock the position of one consumer for the duration of the chunk. + The lock is on the cursor row, which takes part in no foreign key: + two passes of the same consumer serialize here, and no row lock on + `audit_log` is needed (the GC never locks the cursor). The tick mutex + is the advisory lock of `with-tick-lock`; this row lock is the safety + net for a pass that bypasses the loop (a REPL against a running + consumer, for instance). + + `xid::text` is the JDBC bridge, not a sorting choice: `xid8` is an + unsigned 64-bit transaction counter with no Java type and no standard + driver mapping, so it crosses as text and goes back to SQL as + `?::xid8`. The alias is `xid_text` on purpose: an output column named + `xid` would make `ORDER BY xid` sort the text." + "SELECT xid::text AS xid_text, seq + FROM audit_log_consumer_cursor + WHERE consumer = ?::smallint + FOR UPDATE") + +(def ^:private sql:get-cursor + "SELECT xid::text AS xid_text, seq + FROM audit_log_consumer_cursor + WHERE consumer = ?::smallint") + +(def ^:private sql:ensure-cursor + "A consumer with no cursor starts at the beginning of the buffer: it + processes what the buffer holds, not only what arrives from now on. + (0, 0) sits below any real position, so the first chunk includes the + oldest row; the table starts empty because the migration cannot know + which consumers are enabled. ON CONFLICT keeps an existing position + untouched." + "INSERT INTO audit_log_consumer_cursor (consumer, xid, seq) + VALUES (?::smallint, '0'::xid8, 0) + ON CONFLICT (consumer) DO NOTHING") + +(def ^:private registry-lock-classid + "The two-key advisory lock of the cursor registry. It lives in its own + classid, apart from the consumer tick (730000) and from the one-key form + the file locks use." + 730001) + +(def ^:private registry-lock-objid 0) + +(defn lock-registry + "Take the registry lock for the current transaction. `ensure-cursor` and + each GC batch call it before touching the cursor table, so a cursor + registered between two batches is seen by the next one." + [conn] + (db/exec-one! conn ["select pg_advisory_xact_lock(?::int, ?::int) as lock" + registry-lock-classid registry-lock-objid]) + true) + +(defn ensure-cursor + "Create the cursor of a consumer when it does not exist yet: at the + beginning of the buffer, so the first pass drains what is there. The + cursor means 'last row processed' and the chunk query is a strict + `(xid, seq) > cursor`, so the starting position must sit *before* the + oldest row: (0, 0) is below any real position and needs no extra read. + + Serialized with the GC by `lock-registry`: a batch that reads the oldest + cursor and deletes cannot interleave with this insert, so the rows that + exist when the consumer registers survive its first pass. The GC waits + only for the cursors that exist; a consumer with no row has not run yet. + The normal drain path finds the row and does not reach this; when a drain + finds it missing, the lock is held until that chunk transaction commits." + [cfg consumer] + (db/tx-run! + cfg + (fn [cfg] + (lock-registry (::db/conn cfg)) + (db/exec-one! cfg [sql:ensure-cursor consumer])))) + +(def ^:private sql:chunk + "The next chunk of events for one consumer. + + The tuple comparison walks the `(xid, seq)` index from the cursor; the + snapshot cut leaves out every row whose transaction is still in flight, + and the watermark bounds the pass to what was visible when it started, + so the pass always ends. The order is the insertion order that commits + respect: a transaction with a lower `xid` committed (or aborted) before + a higher one can be visible, and `seq` unties the rows of a single + transaction. + + No `SKIP LOCKED` and no row lock: the tick mutex and the cursor lock + already serialize the passes of this consumer, and the rows are only + read." + "SELECT a.id, + a.name, + a.type, + a.source, + a.created_at, + a.tracked_at, + a.profile_id, + a.ip_addr, + a.props, + a.context, + a.seq, + a.xid::text AS xid_text + FROM audit_log a + WHERE (a.xid, a.seq) > (?::xid8, ?::bigint) + AND (a.xid, a.seq) <= (?::xid8, ?::bigint) + AND a.xid <= pg_snapshot_xmin(pg_current_snapshot()) + ORDER BY a.xid ASC, a.seq ASC + LIMIT ?") + +(def ^:private sql:last-visible + "The frontier of the pass: the newest row whose transaction is already + committed, with the same visibility rule the chunk uses. The pass drains + up to it, so it always ends and every chunk but the last is full. + + The `ORDER BY` names the table column: `xid` alone would resolve to the + text projection (`xid_text`) and sort '999' above '1000'." + "SELECT xid::text AS xid_text, seq + FROM audit_log + WHERE xid <= pg_snapshot_xmin(pg_current_snapshot()) + ORDER BY audit_log.xid DESC, seq DESC + LIMIT 1") + +(def ^:private sql:advance-cursor + "UPDATE audit_log_consumer_cursor + SET xid = ?::xid8, seq = ?::bigint, updated_at = now() + WHERE consumer = ?::smallint") + +(defn- lock-cursor + [cfg consumer] + (db/exec-one! cfg [sql:lock-cursor consumer])) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; TICK +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(def ^:private tick-lock-classid + "Namespace of the two-key advisory locks of the consumers. The two-key + form lives in its own lock space, disjoint from the one-key + `pg_advisory_lock(bigint)` that `update-file` and the setup take, so a + consumer tick can never block a file update (nor the other way around) + even if the numbers coincide." + 730000) + +(defn- tick-lock-key + "The two-key advisory lock of one consumer: the namespace of the audit + log and the consumer id." + [consumer-id] + [tick-lock-classid consumer-id]) + +(defn- with-tick-lock + "Run `f` with the session advisory lock of the consumer held, passing it + the connection that holds the lock; answer ::busy without running it + when another instance is draining. + + The lock lives on a connection borrowed from the pool and is released + before that connection goes back, because a pooled connection returned + with a session lock keeps it. A lock an earlier pass failed to release + is cleared when the connection is borrowed again, so one bad unlock + cannot stop the consumer for good. A failed unlock is logged and + swallowed: it must not replace the error of the pass that is unwinding." + [{:keys [::db/pool]} consumer-id f] + (with-open [conn (db/open pool)] + (db/session-unlock-all conn) + (let [[classid objid] (tick-lock-key consumer-id)] + (if (db/session-try-lock conn classid objid) + (try + (f conn) + (finally + (try + (db/session-unlock conn classid objid) + (catch Throwable cause + (l/wrn :hint "unable to release the audit-consumer lock" + :consumer (consumer-label consumer-id) + :cause cause))))) + ::busy)))) + +(defn- get-chunk-rows + [cfg cursor watermark] + (->> (db/exec! cfg [sql:chunk + (:xid-text cursor) (:seq cursor) + (:xid-text watermark) (:seq watermark) + chunk-size]) + (not-empty))) + +(defn- advance-cursor + [{:keys [::db/conn]} consumer row] + (db/exec-one! conn [sql:advance-cursor (:xid-text row) (:seq row) consumer])) + +(defn- drain-chunks + "Drain one consumer cursor chunk by chunk, one transaction per chunk, up + to the watermark of the pass. The processor receives the rows of each + chunk and returns {:processed n :skipped n}; the cursor advances past + every row of the chunk in the same transaction, so a refusal (or a + throw) rolls the advance back and the chunk is retried on the next pass. + The send runs inside that transaction too: the `FOR UPDATE` on the + cursor row assigns an xid, so a slow destination holds + `pg_snapshot_xmin` back for every consumer until the chunk ends. + `totals` is updated after each committed chunk, so a pass that aborts + later still reports what it already archived. Answers the totals." + [cfg conn consumer process-chunk watermark totals] + (loop [] + ;; One transaction per chunk on the connection that holds the tick: + ;; fetch, process and advance commit (or roll back) together, and the + ;; next chunk sees a fresh snapshot. A pass needs no second connection. + (if-let [res (db/tx-run! (-> cfg (dissoc ::db/pool) (assoc ::db/conn conn)) + (fn [cfg] + (db/exec! cfg ["SET LOCAL lock_timeout = '30s'"]) + (let [consumer-id (:id consumer) + cursor (or (lock-cursor cfg consumer-id) + (do (ensure-cursor cfg consumer-id) + (lock-cursor cfg consumer-id)))] + (when-let [rows (get-chunk-rows cfg cursor watermark)] + (let [res (process-chunk cfg rows)] + (advance-cursor cfg consumer-id (peek rows)) + res)))))] + (do + ;; Committed: its rows count even if a later chunk fails. + (vswap! totals (fn [{:keys [processed skipped chunks]}] + {:processed (+ (long processed) (long (or (:processed res) 0))) + :skipped (+ (long skipped) (long (or (:skipped res) 0))) + :chunks (inc (long chunks))})) + (px/sleep 100) + (recur)) + @totals))) + +(def ^:private sql:lag + "Events confirmed in the buffer that this consumer has not processed + yet, capped: past 10000 the gauge reads 10000, so a long backlog does + not turn every pass into an O(backlog) count. It rises when the stream + falls behind (a slow destination, or a big transaction that just + committed); while a transaction is open its rows are invisible to + everybody, so they cannot be counted here." + "SELECT count(*) AS lag + FROM (SELECT 1 + FROM audit_log a + WHERE (a.xid, a.seq) > (?::xid8, ?::bigint) + LIMIT 10000) s") + +(defn- record-lag + [conn metrics consumer] + (when-let [cursor (db/exec-one! conn [sql:get-cursor (:id consumer)])] + (let [lag (:lag (db/exec-one! conn [sql:lag (:xid-text cursor) (:seq cursor)]))] + (mtx/run! metrics + :id :audit-consumer-lag + :labels [(consumer-label (:id consumer))] + :val lag)))) + +(defn- record-totals + "Record the event counters of the chunks a pass committed. Called on the + happy path and from the catch, so a pass that aborts on chunk N still + counts chunks 1..N-1." + [metrics label {:keys [processed skipped]}] + (when (pos? processed) + (record-count metrics :audit-consumer-events [label "processed"] processed)) + (when (pos? skipped) + (record-count metrics :audit-consumer-events [label "skipped"] skipped))) + +(defn drain-consumer + "Run one pass of a consumer: take the tick lock, drain its cursor chunk by + chunk (one transaction per chunk) up to the frontier of the pass, and + record the summary. Answers ::busy without doing anything when another + instance is draining this consumer, and ::disabled without touching the + data when the consumer's flag is off: the loop is gated at boot, but a + direct caller (REPL, tests, a helper) must not advance a cursor or send + events for a consumer the deployment has disabled. + + The processor receives the rows of each chunk and returns + {:processed n :skipped n}; the cursor advances past every row of the + chunk, so a selective consumer still advances over the events it does + not take (they are counted as skipped, not lost: the next chunk + continues from there). + + The frontier is the newest visible row when the pass starts, so the pass + always ends and a row committed while it runs waits for the next one. A + refused chunk (or a throw) aborts the pass: the committed chunks stay + processed, the rest waits for the next pass. The whole pass runs on the + connection that holds the tick (frontier, chunk transactions and lag), so + the pool needs one connection per enabled consumer. The metrics land + here, outside every chunk transaction: `penpot_audit_consumer_passes_total` + counts `ok` (drained or empty), `busy` (another instance holds the tick) + and `failed` (anything the pass throws: the frontier read, a chunk or the + lag query), so a failure cannot leave the family flat." + [cfg consumer-id process-chunk] + (cond + (not (contains? consumers-by-id consumer-id)) + (ex/raise :type :internal + :code :unknown-audit-consumer + :hint "unknown audit-log consumer" + :consumer consumer-id) + + (not (enabled? consumer-id)) + ::disabled + + :else + (let [consumer (get consumers-by-id consumer-id) + label (consumer-label consumer-id) + metrics (mtx/instance cfg) + totals (volatile! {:processed 0 :skipped 0 :chunks 0}) + result (with-tick-lock cfg consumer-id + (fn [conn] + (try + (if-let [watermark (db/exec-one! conn [sql:last-visible])] + (let [tpoint (ct/tpoint) + drained (drain-chunks cfg conn consumer process-chunk watermark totals) + _ (record-lag conn metrics consumer) + _ (record-totals metrics label drained) + _ (record-count metrics :audit-consumer-passes [label "ok"] 1) + processed (:processed drained) + skipped (:skipped drained) + chunks (:chunks drained) + ;; Strict binding on purpose: log props evaluate lazily in + ;; the logger thread, so the elapsed must be frozen here. + elapsed (ct/format-duration (tpoint))] + (if (and (zero? processed) + (zero? skipped)) + (l/dbg :hint "no new audit events" + :consumer label) + (l/dbg :hint "audit events processed" + :consumer label + :elapsed elapsed + :processed processed + :skipped skipped + :chunks chunks))) + ;; Nothing was visible when the pass started: no chunk query. + (do + (record-lag conn metrics consumer) + (record-count metrics :audit-consumer-passes [label "ok"] 1) + (l/dbg :hint "no new audit events" + :consumer label) + nil)) + (catch Throwable cause + ;; The counter covers the whole pass: a failed frontier + ;; read or lag query counts as a failed pass too, not + ;; only a refused chunk. + (record-totals metrics label @totals) + (record-count metrics :audit-consumer-passes [label "failed"] 1) + ;; The failed event count comes from the error data: the + ;; send fns carry the attempted events as :events. A local + ;; DB failure carries none and is not counted as a + ;; destination failure. + (let [n (:events (ex-data cause))] + (when (and (integer? n) (pos? n)) + (record-count metrics :audit-consumer-events [label "failed"] n))) + (throw cause)))))] + (if (= ::busy result) + (do + (record-count metrics :audit-consumer-passes [label "busy"] 1) + ::busy) + result)))) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; LOOP +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(def ^:private sleep-ms + "Pause after every pass. Long enough to keep the database quiet while the + buffer is empty, short enough that an event waits seconds, not minutes." + 5000) + +(defn start-loop + "Start the daemon thread of one consumer: a pass, a pause, forever. + + `run-pass` gets the cfg and may throw: the loop logs the failure and + keeps going, because a consumer that stops on the first refused chunk + stops for good. Answers the thread, to be interrupted on halt." + [cfg consumer-id run-pass] + (let [label (consumer-label consumer-id)] + (px/thread + {:name (str "penpot/audit-consumer/" label)} + (l/inf :hint "started" :consumer label) + (try + (loop [] + (when (px/interrupted?) + (throw (InterruptedException. "interrupted"))) + + (try + (run-pass cfg) + (catch InterruptedException cause + (throw cause)) + (catch Throwable cause + (l/err :hint "consumer pass failed, retrying after the pause" + :consumer label + :cause cause))) + + (px/sleep sleep-ms) + (recur)) + + (catch InterruptedException _ + (l/dbg :hint "interrupted" :consumer label)) + (catch Throwable cause + (l/err :hint "unexpected exception" :consumer label :cause cause)) + (finally + (l/inf :hint "terminated" :consumer label)))))) + + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; COMPONENT +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(defn init-component + "Start the loop of a consumer when its flag is on and the database is + writable; nil otherwise. `prepare` runs only on the start path and + answers the cfg the pass will use (Nexus resolves its uri there). + + The cursor is registered after `prepare` succeeds, so a consumer whose + configuration is rejected leaves no row behind." + [cfg consumer-id prepare run-pass] + (cond + (not (enabled? consumer-id)) + (l/dbg :hint "not started (flag off)" :consumer (consumer-label consumer-id)) + + (db/read-only? (::db/pool cfg)) + (l/wrn :hint "not started (db is read-only)" :consumer (consumer-label consumer-id)) + + :else + (let [cfg (prepare cfg)] + ;; Register the consumer before the loop reads anything: the GC waits + ;; for its cursor, so the row must exist even if the first pass has + ;; nothing visible to drain yet. + (ensure-cursor cfg consumer-id) + (start-loop cfg consumer-id run-pass)))) + +(defn halt-component + "Stop the loop of a consumer and wait for it. Tolerates the nil an init + that skipped the start answers." + [thread] + (when thread + (px/interrupt! thread) + (.join ^Thread thread 5000))) diff --git a/backend/src/app/loggers/audit/consumers/admin_console.clj b/backend/src/app/loggers/audit/consumers/admin_console.clj new file mode 100644 index 0000000000..83471e623b --- /dev/null +++ b/backend/src/app/loggers/audit/consumers/admin_console.clj @@ -0,0 +1,318 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns app.loggers.audit.consumers.admin-console + "The admin console transport of the audit log consumer. + + It reads the stream from its own cursor and ingests the allowlisted + events, enriched with the name of their team and the emails of the + profiles they mention (the admin console shows people, not ids). The + events outside the allowlist, and the telemetry shadows, count as + skipped for the cursor: they are passed over and never reach the + destination. + + The allowlist lives here: it is this consumer's contract, not the + producer's (the producer only inserts rows). The component starts the + consumer loop at boot when its flag is on; the mechanics (the tick + mutex, the cursor, the frontier of the pass, the GC) live in + `app.loggers.audit.consumers`." + (:require + [app.common.exceptions :as ex] + [app.common.logging :as l] + [app.common.uri :as u] + [app.common.uuid :as uuid] + [app.config :as cf] + [app.db :as db] + [app.loggers.audit :as audit] + [app.loggers.audit.consumers :as consumers] + [app.metrics :as mtx] + [app.nitrate :as nitrate] + [integrant.core :as ig])) + +(def event-names + "The event names the admin console archives. Each consumer owns its + filter now: the producer only inserts the row, and this consumer + decides on read which events concern it." + #{"accept-organization-invitation" + "accept-organization-invitation-from" + "accept-team-invitation" + "accept-team-invitation-from" + "add-member-to-organization" + "add-team-to-organization" + "assign-organization-attribute" + "auditlog-exported" + "auditlog-guide-downloaded" + "cancel-organization-invitation" + "create-export-binfile-job" + "create-file" + "create-organization" + "create-organization-attribute" + "create-organization-invitation" + "create-project" + "create-team" + "create-team-access-request" + "create-team-invitation" + "create-team-invitations" + "create-webhook" + "delete-file" + "delete-font" + "delete-organization" + "delete-organization-attribute" + "delete-organization-member" + "delete-project" + "delete-team" + "delete-team-invitation" + "delete-team-member" + "leave-team" + "move-project" + "move-team-to-organization" + "organization-sso-auth-failed" + "organization-sso-auth-started" + "organization-sso-auth-succeeded" + "organization-sso-token-issued" + "organization-sso-token-revoked" + "permanently-delete-team-files" + "remove-organization-team" + "remove-team-from-organization" + "rename-organization" + "rename-project" + "restore-deleted-team-files" + "sso-config-deleted" + "sso-config-tested" + "success-activate-organization-sso" + "success-apply-organization-sso-changes" + "unassign-organization-attribute" + "update-organization-attribute" + "update-organization-invitation" + "update-organization-permissions" + "update-team-invitation" + "update-team-invitation-role" + "update-team-member-role" + "update-team-photo" + "verify-token"}) + +;; Props id key -> email prop filled from profile.email (admin-console path only). +;; Some producers store ids as UUID strings (e.g. :user-who-send-invitation). +(def ^:private id->email-prop + {:profile-id :profile-email + :user-id :user-email + :member-id :member-email + :user-who-send-invitation :user-who-send-invitation-email}) + +(defn- collect-team-ids-needing-name + [events] + (into #{} + (comp (map :props) + (keep (fn [props] + (when-let [team-id (and (not (contains? props :team-name)) + (uuid/coerce (:team-id props)))] + team-id)))) + events)) + +(defn- collect-profile-ids-needing-email + [events] + (into #{} + (mapcat (fn [{:keys [props]}] + (keep (fn [[id-key email-key]] + (when-not (contains? props email-key) + (uuid/coerce (get props id-key)))) + id->email-prop))) + events)) + +(defn- collect-actor-ids-needing-profile + "Actor ids (event :profile-id) that still need top-level :profile-email + and/or :profile-name. Skips uuid/zero and non-uuids." + [events] + (into #{} + (keep (fn [event] + (when (or (not (contains? event :profile-email)) + (not (contains? event :profile-name))) + (when-let [id (uuid/coerce (:profile-id event))] + (when-not (uuid/zero? id) + id))))) + events)) + +(defn- load-team-names + [conn ids] + (if (seq ids) + (let [arr (db/create-array conn "uuid" ids) + rows (db/exec! conn ["SELECT id, name FROM team WHERE id = ANY(?)" arr])] + (into {} (map (juxt :id :name) rows))) + {})) + +(defn- load-profiles + "Batch-load profile email and fullname (including soft-deleted)." + [conn ids] + (if (seq ids) + (let [arr (db/create-array conn "uuid" ids) + rows (db/exec! conn ["SELECT id, email, fullname FROM profile WHERE id = ANY(?)" arr])] + (into {} (map (juxt :id identity) rows))) + {})) + +(defn- maybe-assoc-team-name + [event team-names] + (let [props (:props event)] + (if-let [team-name (and (not (contains? props :team-name)) + (when-let [team-id (uuid/coerce (:team-id props))] + (get team-names team-id)))] + (assoc-in event [:props :team-name] team-name) + event))) + +(defn- maybe-assoc-emails + [event profiles] + (update event :props + (fn [props] + (reduce-kv (fn [props id-key email-key] + (if-let [email (and (not (contains? props email-key)) + (when-let [id (uuid/coerce (get props id-key))] + (:email (get profiles id))))] + (assoc props email-key email) + props)) + props + id->email-prop)))) + +(defn- maybe-assoc-actor-profile + "Fill-only top-level :profile-email / :profile-name from the event + actor (:profile-id). Omits keys when the id is missing, zero, the + profile row is absent, or the corresponding column is empty + (fullname defaults to \"\" in the schema)." + [event profiles] + (let [id (uuid/coerce (:profile-id event))] + (if (or (nil? id) (uuid/zero? id)) + event + (if-let [profile (get profiles id)] + (cond-> event + (and (not (contains? event :profile-email)) + (not-empty (:email profile))) + (assoc :profile-email (:email profile)) + + (and (not (contains? event :profile-name)) + (not-empty (:fullname profile))) + (assoc :profile-name (:fullname profile))) + event)))) + +(defn- enrich-events + "Enrich allowlisted events before the admin-console ingest: + 1. Fill missing :team-name from team id (including soft-deleted). + 2. Fill missing emails from matching id keys in props (including + soft-deleted profiles): :profile-email, :user-email, :member-email, + :user-who-send-invitation-email. + 3. Fill missing top-level :profile-email / :profile-name from the + event actor (:profile-id), including soft-deleted profiles; omit + when there is no usable actor id or the value is empty." + [{:keys [::db/conn]} events] + (let [team-ids (collect-team-ids-needing-name events) + profile-ids (into (collect-profile-ids-needing-email events) + (collect-actor-ids-needing-profile events)) + team-names (load-team-names conn team-ids) + profiles (load-profiles conn profile-ids)] + (mapv (fn [event] + (-> event + (maybe-assoc-team-name team-names) + (maybe-assoc-emails profiles) + (maybe-assoc-actor-profile profiles))) + events))) + +(defn- send-events + "Ingest rows into the admin console. Returns {:processed :skipped} + (filtered-out rows count as skipped); throws carrying the attempted + count as :events. Enrich failures propagate untouched: a local + DB/decode failure is ours, not the destination's, and must not count as + a send failure." + [cfg rows] + (let [events (filterv #(and (contains? event-names (:name %)) + (not (audit/telemetry-shadow-source? (:source %)))) + rows) + total (count rows) + skipped (- total (count events)) + uri-host (::uri-host cfg)] + (l/dbg :hint "consumer events chunk ready" + :consumer "admin-console" + :uri-host uri-host + :total total + :skipped skipped) + (if (empty? events) + {:processed 0 :skipped skipped} + ;; Enrich first, outside the try: a local DB/decode failure is + ;; ours, not the destination's, and must not count as + ;; admin-console/failed. + (let [enriched (enrich-events cfg events) + n (count events)] + (try + (nitrate/call cfg :ingest-audit-log {:events enriched}) + (l/dbg :hint "consumer events chunk ingested" + :consumer "admin-console" + :uri-host uri-host + :processed n + :skipped skipped) + {:processed n :skipped skipped} + (catch Throwable cause + (l/dbg :hint "consumer events chunk failed" + :consumer "admin-console" + :uri-host uri-host + :events n + :code :admin-console-send-failed) + (throw (ex/raise :type :internal + :code :admin-console-send-failed + :hint "unable to archive events to the admin console" + :cause cause + :events n)))))))) + +(defn- process-chunk + "Send one chunk to the admin console. Returns {:processed :skipped}: + the row outside the allowlist (and the telemetry shadows) count as + skipped. Throws on failure, same contract as the nexus one." + [cfg rows] + (let [decoded (mapv consumers/decode-row rows) + _ (l/trc :hint "consumer events chunk" + :consumer "admin-console" + :events (count decoded))] + (send-events cfg decoded))) + +(defn run-pass + "One pass of the admin-console consumer: drain the stream from its cursor + up to the frontier of the pass. The loop of `::consumer` calls it once + per pass; direct callers (REPL, tests) get the same behavior." + [cfg] + (consumers/drain-consumer cfg + consumers/admin-console + process-chunk)) + +(defmethod ig/assert-key ::consumer + [_ cfg] + ;; No assert on the admin-console client: it inits to nil without the + ;; :admin-console flag, and the component skips on its flag anyway. + (assert (db/pool? (::db/pool cfg)) "expected a valid database pool") + (assert (mtx/metrics? (::mtx/metrics cfg)) "expected a valid metrics instance")) + +(defn- prepare + "Resolve the destination once at boot. An incomplete configuration + raises here and the worker does not start: a consumer whose ingest can + never reach its destination would fail every pass without advancing + its cursor, and while enabled it also holds the audit-log GC. The host + travels in the cfg so the per-chunk logs do not resolve it again." + [cfg] + (let [uri (some-> (cf/get :admin-console-uri) u/uri)] + (when (nil? uri) + (ex/raise :type :internal + :code :task-not-configured + :hint "admin-console consumer not configured, missing :admin-console-uri")) + (let [uri-host (u/get-domain uri)] + (l/dbg :hint "audit log consumer initialized" + :consumer "admin-console" + :uri-host uri-host) + (assoc cfg ::uri-host uri-host)))) + +(defmethod ig/init-key ::consumer + [_ cfg] + (consumers/init-component cfg + consumers/admin-console + prepare + run-pass)) + +(defmethod ig/halt-key! ::consumer + [_ thread] + (consumers/halt-component thread)) diff --git a/backend/src/app/loggers/audit/consumers/nexus.clj b/backend/src/app/loggers/audit/consumers/nexus.clj new file mode 100644 index 0000000000..4c73541df6 --- /dev/null +++ b/backend/src/app/loggers/audit/consumers/nexus.clj @@ -0,0 +1,148 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns app.loggers.audit.consumers.nexus + "The Nexus transport of the audit log consumer. + + It reads the whole stream from the Nexus cursor (the oldest events + first) and POSTs them to Nexus in batches. A batch Nexus refuses rolls + back and stays for the next pass: the cursor does not advance, the pass + aborts and the loop retries after its pause. + + The component starts the consumer loop at boot when its flag is on. The + mechanics (the tick mutex, the cursor, the frontier of the pass, the GC) + live in `app.loggers.audit.consumers`." + (:require + [app.common.exceptions :as ex] + [app.common.logging :as l] + [app.common.transit :as t] + [app.common.uri :as u] + [app.config :as cf] + [app.db :as db] + [app.http.client :as http] + [app.loggers.audit.consumers :as consumers] + [app.metrics :as mtx] + [app.setup :as-alias setup] + [integrant.core :as ig])) + +(defn- generate-uri + "Joins relative path segments to the Nexus backend URI, same shape + as the admin-console helper in `app.nitrate` (there the ingest path + is `api/audit-log`; here it is `api/audit`). + Segments must not start with `/`. Returns nil when no base URI is + configured, so the caller keeps the `:task-not-configured` guard." + [& segments] + (when-let [base (cf/get :nexus-uri)] + (apply cf/join-uri base segments))) + +(defn- resolve-uri + "The Nexus endpoint of this consumer: the override on the cfg, or the one + built from `:nexus-uri`. Raises when neither is available, at boot and on + every pass, so a consumer without a destination fails loudly instead of + skipping silently. Failing the boot is deliberate: a worker that starts + with its archive destination silently disabled is worse than one that + refuses to start." + [cfg] + (or (some-> (::uri cfg) u/uri) + (generate-uri "api/audit") + (ex/raise :type :internal + :code :task-not-configured + :hint "consumer task not configured, missing uri"))) + +(defn- truncate-body + [body limit] + (when (some? body) + (let [s (str body)] + (if (> (count s) limit) + (subs s 0 limit) + s)))) + +(defn- send-events + "POST events to Nexus. Returns the number of events sent; throws on + transport failure or non-204 refusal, carrying the attempted count + as :events in the error data so the caller records the failure. + No metrics or logs here: the loop reports the aborted pass." + [{:keys [::uri] :as cfg} events] + (let [n (count events) + skey (-> cfg ::setup/shared-keys :nexus) + body (t/encode {:events events}) + params {:uri uri + :timeout 12000 + :method :post + :headers {"content-type" "application/transit+json" + "origin" (str (cf/get :public-uri)) + "x-shared-key" (str "nexus " skey)} + :body body} + resp (try + (http/req cfg params {:skip-ssrf-check? true}) + (catch Throwable cause + (throw (ex/raise :type :internal + :code :nexus-send-failed + :hint "unable to send events to nexus" + :cause cause + :events n))))] + (if (= (:status resp) 204) + n + (throw (ex/raise :type :internal + :code :consumer-chunk-refused + :hint "nexus refused archive chunk" + :resp-status (:status resp) + :resp-body (truncate-body (:body resp) 500) + :events n))))) + +(defn- process-chunk + "Send one chunk of the stream to Nexus. Returns {:processed n}; throws + on refusal, so the chunk rolls back and the cursor stays where it was. + No metrics here: a normal return means the transaction committed, so + the caller records the success." + [cfg rows] + (let [events (mapv (comp consumers/row->event consumers/decode-row) rows) + ;; send-events returns the count or throws: a refusal + ;; aborts the pass, the rows stay for the next one. + n (send-events cfg events)] + + (l/trc :hint "consumer events chunk" + :consumer "nexus" + :events n) + + {:processed n :skipped 0})) + +(defn run-pass + "One pass of the Nexus consumer: drain the stream from its cursor up to + the frontier of the pass. The loop of `::consumer` calls it once per + pass; direct callers (REPL, tests) get the same behavior." + [cfg] + (consumers/drain-consumer (assoc cfg ::uri (resolve-uri cfg)) + consumers/nexus + process-chunk)) + +(defmethod ig/assert-key ::consumer + [_ cfg] + (assert (db/pool? (::db/pool cfg)) "expected a valid database pool") + (assert (http/client? (::http/client cfg)) "expected a valid http client") + (assert (mtx/metrics? (::mtx/metrics cfg)) "expected a valid metrics instance")) + +(defn- prepare + "Resolve the destination once at boot: per-pass logs carry counts, this + says where the consumer sends. An incomplete configuration raises here + (see `resolve-uri`) and the worker does not start: intended." + [cfg] + (let [uri (resolve-uri cfg)] + (l/dbg :hint "audit log consumer initialized" + :consumer "nexus" + :uri-host (u/get-domain uri)) + (assoc cfg ::uri uri))) + +(defmethod ig/init-key ::consumer + [_ cfg] + (consumers/init-component cfg + consumers/nexus + prepare + run-pass)) + +(defmethod ig/halt-key! ::consumer + [_ thread] + (consumers/halt-component thread)) diff --git a/backend/src/app/loggers/audit/gc.clj b/backend/src/app/loggers/audit/gc.clj new file mode 100644 index 0000000000..d8edb10592 --- /dev/null +++ b/backend/src/app/loggers/audit/gc.clj @@ -0,0 +1,160 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns app.loggers.audit.gc + "The garbage collection of the audit log buffer. + + The producer only inserts rows and each consumer keeps its position, + so somebody has to delete the rows every registered consumer has + already passed. This job does it in batches below the oldest cursor, + never touching the telemetry shadows (the telemetry task owns them). + + The consumer framework (ids, flags and cursor helpers) lives in + `app.loggers.audit.consumers`." + (:require + [app.common.logging :as l] + [app.common.schema :as sm] + [app.db :as db] + [app.jobs :as jobs] + [app.loggers.audit.consumers :as consumers] + [app.metrics :as mtx] + [integrant.core :as ig] + [promesa.exec :as px])) + +(def ^:private batch-size + "Rows per delete: a big buffer is cleaned in bounded statements, each + one its own snapshot, so the job heartbeats between batches." + 5000) + +(defn- get-oldest-cursor + "The oldest position among every cursor in the table, or nil when there + is none. + + The set is not derived from the flags of the process that runs the job: + the cursor table is shared by every instance, so a consumer disabled on + this worker still holds the position it reached and keeps its unread + rows alive. A consumer registers its (0, 0) cursor at boot, before its + first pass, so the first row it has to read is never deleted under it. + With no cursor at all the GC deletes nothing: it cannot know whether a + consumer is about to start." + [cfg] + (when-let [row (first (db/exec! cfg + ["SELECT xid::text AS xid_text, seq + FROM audit_log_consumer_cursor + ORDER BY xid ASC, seq ASC + LIMIT 1"]))] + (select-keys row [:xid-text :seq]))) + +(def ^:private sql:delete-batch + "Delete one batch of rows up to and including a position, telemetry + shadows excluded: the cursor points at the last row a consumer + processed, so that row is done too. The telemetry task owns the + shadows (it deletes them after sending and purges the ones older than + its retention window), so deleting them here would lose events before + they are shipped. + + The inner select walks the `(xid, seq)` index and skips locked rows: + a producer never holds an old row, and two GC runs must not wait for + each other." + "DELETE FROM audit_log a + WHERE a.id IN (SELECT a2.id + FROM audit_log a2 + WHERE (a2.xid, a2.seq) <= (?::xid8, ?::bigint) + AND a2.source NOT LIKE 'telemetry:%' + ORDER BY a2.xid ASC, a2.seq ASC + LIMIT ? + FOR UPDATE SKIP LOCKED)") + +(defn- delete-one-batch + "Read the oldest cursor and delete one batch under the registry lock, in + one transaction. The read is inside the batch, so a cursor registered + between two batches is seen before the next delete. Answers + `{:deleted n :found? bool}`." + [cfg] + (db/tx-run! + cfg + (fn [cfg] + (consumers/lock-registry (::db/conn cfg)) + (if-let [cursor (get-oldest-cursor cfg)] + {:deleted (or (db/get-update-count + (db/exec-one! cfg [sql:delete-batch + (:xid-text cursor) (:seq cursor) + batch-size])) + 0) + :found? true} + {:deleted 0 :found? false})))) + +(defn- delete-batched + "Run a batched DELETE until it removes nothing, re-reading the oldest + cursor on every batch. Answers `{:total n :found? bool}`." + [cfg] + (loop [total 0 found? false] + (let [{:keys [deleted] :as batch} (delete-one-batch cfg)] + (if (pos? (long deleted)) + (do + (jobs/heartbeat cfg) + (px/sleep 100) + (recur (+ total (long deleted)) (or found? (:found? batch)))) + {:total total :found? (or found? (:found? batch))})))) + +(defn delete-consumed + "Delete the rows every registered consumer has already passed. The + position is the oldest cursor, inclusive (the cursor points at the last + row the consumer processed). The set is the cursor table, not the flags + of this worker: a consumer that is off still holds the buffer (its + cursor freezes and rejoins from wherever it is when it is enabled + again), so a temporary disable never deletes events it has not read. + + A consumer registers its (0, 0) cursor when its component starts, so a + consumer that is enabled but has not drained yet still holds the buffer + at position 0. With no cursor at all nothing is deleted: the + `:audit-log-gc` flag is the deployment switch, and the GC waits for the + consumers to register. Deleting the buffer behind the back of a + consumer that may start later loses events; the disk risk is the + operator's, not a reason to drain. + + Returns the number of discarded rows; they count in the `discarded` + metric with reason `gc`." + [cfg] + (let [{:keys [total found?]} (delete-batched cfg)] + (if-not found? + (do + (l/wrn :hint "audit log GC is on with no consumer cursor; the buffer will not be cleaned") + 0) + (do + (when (pos? total) + (consumers/record-count (mtx/instance cfg) :audit-consumer-discarded ["gc"] total)) + total)))) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; JOB +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(def schema:params + "Params of the GC job: none. Cron submits {}; system deps (pool, + metrics) live on the job-def cfg closed over by the handler." + [:map {:closed true}]) + +(defmethod ig/assert-key ::job-def + [_ cfg] + (assert (db/pool? (::db/pool cfg)) "expected a valid database pool") + (assert (mtx/metrics? (::mtx/metrics cfg)) "expected a valid metrics instance")) + +(defmethod ig/init-key ::job-def + [_ cfg] + (l/dbg :hint "audit log GC job initialized") + {::jobs/name :audit-log-gc + ::jobs/schema schema:params + ::jobs/handler + (fn [_context _params] + ;; The no-cursor branch deletes nothing on purpose: deleting would lose + ;; events for a consumer that may start later. Without the warning the + ;; operator only sees `discarded 0` at `dbg` and the buffer grows. + (let [total (delete-consumed cfg)] + (l/dbg :hint "audit log buffer cleaned" :discarded total) + total)) + ::jobs/decoder (sm/decoder schema:params sm/json-transformer) + ::jobs/validator (sm/validator schema:params)}) diff --git a/backend/src/app/main.clj b/backend/src/app/main.clj index 79033f9442..80ecbdd4d8 100644 --- a/backend/src/app/main.clj +++ b/backend/src/app/main.clj @@ -27,7 +27,13 @@ [app.http.websocket :as http.ws] [app.jobs :as-alias jobs] [app.jobs.gc :as-alias jobs.gc] - [app.jobs.metrics :as jobs-metrics] + ;; The consumer transports live in their own namespaces; requiring them + ;; registers their `ig/init-key`/`ig/halt-key!` methods before the + ;; worker components start (`ig/load-namespaces` also loads them from + ;; the config keys). + [app.loggers.audit.consumers.admin-console] + [app.loggers.audit.consumers.nexus] + [app.loggers.audit.gc] [app.loggers.webhooks :as-alias webhooks] [app.metrics :as mtx] [app.metrics.definition :as-alias mdef] @@ -121,12 +127,11 @@ :tasks-timing {::mdef/name "penpot_tasks_timing" - ;; Deprecated: superseded by `penpot_jobs_execution_ms`, which adds a - ;; queue label. Kept only for existing dashboards and alerts. The help - ;; text is what someone reading /metrics sees, so it is where the - ;; redirect belongs: the name and the labels stay untouched, so no - ;; panel or alert breaks. - ::mdef/help "Deprecated: use penpot_jobs_execution_ms, which adds a queue label." + ;; Deprecated: superseded by `penpot_jobs_execution_ms`. Kept only for + ;; existing dashboards and alerts. The help text is what someone reading + ;; /metrics sees, so it is where the redirect belongs: the name and the + ;; labels stay untouched, so no panel or alert breaks. + ::mdef/help "Deprecated: use penpot_jobs_execution_ms. Kept for existing dashboards and alerts." ::mdef/labels ["name"] ::mdef/buckets mtx/default-tasks-histogram-buckets ::mdef/type :histogram} @@ -262,28 +267,22 @@ ::mdef/help "Total number of handler errors in the http listener." ::mdef/type :counter} - :jobs-submitted - {::mdef/name "penpot_jobs_submitted_total" - ::mdef/help "Total number of durable jobs submitted." - ::mdef/labels ["name" "queue"] - ::mdef/type :counter} - - :jobs-skipped - {::mdef/name "penpot_jobs_skipped_total" - ::mdef/help "Total number of durable job submissions skipped by skip-if-active." - ::mdef/labels ["name" "queue"] + :jobs-submit + {::mdef/name "penpot_jobs_submit_total" + ::mdef/help "Total number of durable job submissions by outcome: created, skipped or refreshed." + ::mdef/labels ["name" "outcome"] ::mdef/type :counter} :jobs-dispatched {::mdef/name "penpot_jobs_dispatched_total" - ::mdef/help "Total number of jobs pushed to a worker queue. Counts pushes, not claims: compare with the _count of penpot_jobs_queue_wait_ms." - ::mdef/labels ["name" "queue"] + ::mdef/help "Total number of jobs pushed to a worker queue. Counts pushes, not claims: compare with the _count of penpot_jobs_claim_wait_ms." + ::mdef/labels ["name"] ::mdef/type :counter} :jobs-retries {::mdef/name "penpot_jobs_retries_total" ::mdef/help "Total number of durable jobs rescheduled for retry." - ::mdef/labels ["name" "queue" "reason"] + ::mdef/labels ["name" "reason"] ::mdef/type :counter} :jobs-events-total @@ -291,41 +290,29 @@ ::mdef/help "Total number of job events stored." ::mdef/type :counter} - :jobs-orphaned - {::mdef/name "penpot_jobs_orphaned_total" - ::mdef/help "Total number of running jobs marked as orphaned." - ::mdef/labels ["queue"] - ::mdef/type :counter} - - :jobs-rescheduled - {::mdef/name "penpot_jobs_rescheduled_total" - ::mdef/help "Total number of scheduled jobs recovered after transit loss." - ::mdef/labels ["queue"] - ::mdef/type :counter} - - :jobs-queue-wait-timing - {::mdef/name "penpot_jobs_queue_wait_ms" + :jobs-claim-wait-timing + {::mdef/name "penpot_jobs_claim_wait_ms" ::mdef/help "Time from job scheduling to worker claim (milliseconds). Its _count counts claims, not pushes: compare with penpot_jobs_dispatched_total." - ::mdef/labels ["name" "queue"] + ::mdef/labels ["name"] ::mdef/type :histogram ::mdef/buckets [1 10 50 100 500 1000 5000 10000 30000 60000 300000 600000]} :jobs-execution-timing {::mdef/name "penpot_jobs_execution_ms" ::mdef/help "Time spent executing a job handler (milliseconds)." - ::mdef/labels ["name" "queue"] + ::mdef/labels ["name"] ::mdef/type :histogram ::mdef/buckets [1 10 50 100 500 1000 5000 10000 30000 60000 300000 600000]} :jobs-total-timing {::mdef/name "penpot_jobs_total_ms" ::mdef/help "Time from job creation to a terminal state (milliseconds). Its _count is the terminal total per outcome." - ::mdef/labels ["name" "queue" "outcome"] + ::mdef/labels ["name" "outcome"] ::mdef/type :histogram ::mdef/buckets [10 50 100 500 1000 5000 10000 30000 60000 300000 600000 3600000]} - :jobs-dispatcher-timing - {::mdef/name "penpot_jobs_dispatcher_ms" + :jobs-dispatcher-batch-timing + {::mdef/name "penpot_jobs_dispatcher_batch_ms" ::mdef/help "Dispatcher batch processing time (milliseconds)." ::mdef/labels ["stage" "outcome"] ::mdef/type :histogram @@ -338,25 +325,14 @@ ::mdef/type :histogram ::mdef/buckets [1 5 10 25 50 100 250 500 1000]} - :jobs-backlog - {::mdef/name "penpot_jobs_backlog" - ::mdef/help "Current number of jobs grouped by status." - ::mdef/labels ["status"] - ::mdef/type :gauge} - - :jobs-oldest-pending-age - {::mdef/name "penpot_jobs_oldest_pending_age_seconds" - ::mdef/help "Age in seconds of the oldest pending job." - ::mdef/type :gauge} - :jobs-gc-rows {::mdef/name "penpot_jobs_gc_rows_total" ::mdef/help "Rows removed or resources touched by the jobs garbage collector." ::mdef/labels ["kind" "action"] ::mdef/type :counter} - :jobs-gc-timing - {::mdef/name "penpot_jobs_gc_ms" + :jobs-gc-pass-timing + {::mdef/name "penpot_jobs_gc_pass_ms" ::mdef/help "Time spent in each jobs garbage collection pass (milliseconds)." ::mdef/labels ["kind"] ::mdef/type :histogram @@ -382,7 +358,13 @@ :audit-consumer-events {::mdef/name "penpot_audit_consumer_events_total" - ::mdef/help "Audit log events drained by the consumer task, by destination and outcome." + ::mdef/help "Audit log events drained by the audit-log consumers, by destination and outcome." + ::mdef/labels ["destination" "outcome"] + ::mdef/type :counter} + + :audit-consumer-passes + {::mdef/name "penpot_audit_consumer_passes_total" + ::mdef/help "Audit log consumer passes, by destination and outcome." ::mdef/labels ["destination" "outcome"] ::mdef/type :counter} @@ -392,6 +374,12 @@ ::mdef/labels ["reason"] ::mdef/type :counter} + :audit-consumer-lag + {::mdef/name "penpot_audit_consumer_lag_events" + ::mdef/help "Audit log events confirmed in the buffer that a consumer has not processed yet (capped at 10000)." + ::mdef/labels ["destination"] + ::mdef/type :gauge} + :audit-ingest-events {::mdef/name "penpot_audit_ingest_events_total" ::mdef/help "Audit log frontend events ingested, by kind and outcome." @@ -630,8 +618,7 @@ :app.nitrate/client {::http.client/client (ig/ref ::http.client/client) - ;; No ::jobs/defs: nitrate never submits jobs, and consumer-task's - ;; job-def needs this client — that would cycle through jobs/defs. + ;; No ::jobs/defs: nitrate never submits jobs. ::setup/shared-keys (ig/ref ::setup/shared-keys)} :app.rpc/management-methods @@ -659,26 +646,25 @@ ::setup/shared-keys (ig/ref ::setup/shared-keys)} ::jobs/defs - {:sendmail (ig/ref ::email/job-def) - :send-templated-mail (ig/ref ::email/templated-job-def) - :delete-object (ig/ref :app.tasks.delete-object/job-def) - :demo-purge (ig/ref :app.tasks.demo-purge/job-def) - :run-webhook (ig/ref ::webhooks/run-webhook-job-def) - :process-webhook-event (ig/ref ::webhooks/process-webhook-event-job-def) - :file-gc (ig/ref :app.tasks.file-gc/job-def) - :offload-file-data (ig/ref :app.tasks.offload-file-data/job-def) - :objects-gc (ig/ref :app.tasks.objects-gc/job-def) - :export-binfile (ig/ref :app.tasks.export-binfile/job-def) - :import-binfile (ig/ref :app.tasks.import-binfile/job-def) - :storage-gc-deleted (ig/ref ::sto.gc-deleted/job-def) - :storage-gc-touched (ig/ref ::sto.gc-touched/job-def) - :storage-pending-gc (ig/ref ::sto.pending-gc/job-def) - :jobs-gc (ig/ref :app.jobs.gc/job-def) - :telemetry (ig/ref :app.tasks.telemetry/job-def) - :session-gc (ig/ref ::session/job-def) - :file-gc-scheduler (ig/ref :app.tasks.file-gc-scheduler/job-def) - :audit-log-consumer-nexus (ig/ref :app.loggers.audit.consumer-task/nexus-job-def) - :audit-log-consumer-admin-console (ig/ref :app.loggers.audit.consumer-task/admin-console-job-def)} + {:sendmail (ig/ref ::email/job-def) + :send-templated-mail (ig/ref ::email/templated-job-def) + :delete-object (ig/ref :app.tasks.delete-object/job-def) + :demo-purge (ig/ref :app.tasks.demo-purge/job-def) + :run-webhook (ig/ref ::webhooks/run-webhook-job-def) + :process-webhook-event (ig/ref ::webhooks/process-webhook-event-job-def) + :file-gc (ig/ref :app.tasks.file-gc/job-def) + :offload-file-data (ig/ref :app.tasks.offload-file-data/job-def) + :objects-gc (ig/ref :app.tasks.objects-gc/job-def) + :export-binfile (ig/ref :app.tasks.export-binfile/job-def) + :import-binfile (ig/ref :app.tasks.import-binfile/job-def) + :storage-gc-deleted (ig/ref ::sto.gc-deleted/job-def) + :storage-gc-touched (ig/ref ::sto.gc-touched/job-def) + :storage-pending-gc (ig/ref ::sto.pending-gc/job-def) + :jobs-gc (ig/ref :app.jobs.gc/job-def) + :telemetry (ig/ref :app.tasks.telemetry/job-def) + :session-gc (ig/ref ::session/job-def) + :file-gc-scheduler (ig/ref :app.tasks.file-gc-scheduler/job-def) + :audit-log-gc (ig/ref :app.loggers.audit.gc/job-def)} ::jobs/request-pool {::rds/client (ig/ref ::rds/client) @@ -771,16 +757,9 @@ {::db/pool (ig/ref ::db/pool) ::mtx/metrics (ig/ref ::mtx/metrics)} - :app.loggers.audit.consumer-task/nexus-job-def - {::db/pool (ig/ref ::db/pool) - ::http.client/client (ig/ref ::http.client/client) - ::setup/shared-keys (ig/ref ::setup/shared-keys) - ::mtx/metrics (ig/ref ::mtx/metrics)} - - :app.loggers.audit.consumer-task/admin-console-job-def - {::db/pool (ig/ref ::db/pool) - :app.nitrate/client (ig/ref :app.nitrate/client) - ::mtx/metrics (ig/ref ::mtx/metrics)} + :app.loggers.audit.gc/job-def + {::db/pool (ig/ref ::db/pool) + ::mtx/metrics (ig/ref ::mtx/metrics)} ::email/blacklist {} @@ -871,43 +850,46 @@ {::sto.fs/directory (or (cf/get :storage-assets-fs-directory) (cf/get :objects-storage-fs-directory))}}) +(defn worker-cron-entries + "Cron entries of the worker. The audit-log GC is a separate switch: the + old and the new version share `:audit-log-gc`, so keeping it off during + the rollout stops both until the consumers are the only readers." + [] + (into [] (remove nil?) + [{:cron #penpot/cron "0 0 0 * * ?" ;; daily + :job :session-gc} + + {:cron #penpot/cron "0 0 0 * * ?" ;; daily + :job :objects-gc} + + {:cron #penpot/cron "0 0 0 * * ?" ;; daily + :job :storage-gc-deleted} + + {:cron #penpot/cron "0 0 0 * * ?" ;; daily + :job :storage-gc-touched} + + {:cron #penpot/cron "0 0 0 * * ?" ;; daily + :job :storage-pending-gc} + + {:cron #penpot/cron "0 0 0 * * ?" ;; daily + :job :jobs-gc} + + {:cron #penpot/cron "0 0 2 * * ?" ;; daily + :job :file-gc-scheduler} + + {:cron #penpot/cron "0 30 */3,23 * * ?" + :job :telemetry} + + (when (contains? cf/flags :audit-log-gc) + {:cron #penpot/cron "45 */5 * * * ?" ;; every 5m + :job :audit-log-gc})])) + (def worker-config {::wrk/cron {::jobs/defs (ig/ref ::jobs/defs) ::db/pool (ig/ref ::db/pool) ::mtx/metrics (ig/ref ::mtx/metrics) - ::wrk/entries - [{:cron #penpot/cron "0 0 0 * * ?" ;; daily - :job :session-gc} - - {:cron #penpot/cron "0 0 0 * * ?" ;; daily - :job :objects-gc} - - {:cron #penpot/cron "0 0 0 * * ?" ;; daily - :job :storage-gc-deleted} - - {:cron #penpot/cron "0 0 0 * * ?" ;; daily - :job :storage-gc-touched} - - {:cron #penpot/cron "0 0 0 * * ?" ;; daily - :job :storage-pending-gc} - - {:cron #penpot/cron "0 0 0 * * ?" ;; daily - :job :jobs-gc} - - {:cron #penpot/cron "0 0 2 * * ?" ;; daily - :job :file-gc-scheduler} - - {:cron #penpot/cron "0 30 */3,23 * * ?" - :job :telemetry} - - (when (contains? cf/flags :nexus) - {:cron #penpot/cron "0 */5 * * * ?" ;; every 5m - :job :audit-log-consumer-nexus}) - - (when (contains? cf/flags :admin-console) - {:cron #penpot/cron "30 */5 * * * ?" ;; every 5m, offset 30s from nexus - :job :audit-log-consumer-admin-console})]} + ::wrk/entries (worker-cron-entries)} ::wrk/dispatcher {::rds/client (ig/ref ::rds/client) @@ -915,10 +897,6 @@ ::db/pool (ig/ref ::db/pool) ::wrk/tenant (cf/get :tenant)} - ::jobs-metrics/sampler - {::db/pool (ig/ref ::db/pool) - ::mtx/metrics (ig/ref ::mtx/metrics)} - [::default ::wrk/runner] {::wrk/parallelism (cf/get :worker-default-parallelism 1) ::wrk/queue :default @@ -958,7 +936,21 @@ ::jobs/defs (ig/ref ::jobs/defs) ::mtx/metrics (ig/ref ::mtx/metrics) ::db/pool (ig/ref ::db/pool) - ::mbus/msgbus (ig/ref ::mbus/msgbus)}}) + ::mbus/msgbus (ig/ref ::mbus/msgbus)} + + ;; The audit-log consumers are loops, not jobs: each one starts a + ;; thread that takes its tick lock, drains up to the frontier of the + ;; pass and sleeps. Only worker instances run them. + :app.loggers.audit.consumers.nexus/consumer + {::db/pool (ig/ref ::db/pool) + ::http.client/client (ig/ref ::http.client/client) + ::setup/shared-keys (ig/ref ::setup/shared-keys) + ::mtx/metrics (ig/ref ::mtx/metrics)} + + :app.loggers.audit.consumers.admin-console/consumer + {::db/pool (ig/ref ::db/pool) + :app.nitrate/client (ig/ref :app.nitrate/client) + ::mtx/metrics (ig/ref ::mtx/metrics)}}) (defn start diff --git a/backend/src/app/migrations.clj b/backend/src/app/migrations.clj index 76bf5fb9d0..90e6d94369 100644 --- a/backend/src/app/migrations.clj +++ b/backend/src/app/migrations.clj @@ -531,9 +531,6 @@ {:name "0156-reset-pg-storage-defaults" :fn (mg/resource "app/migrations/sql/0156-reset-pg-storage-defaults.sql")} - {:name "0157-add-audit-log-pending-table" - :fn (mg/resource "app/migrations/sql/0157-add-audit-log-pending-table.sql")} - {:name "0157-migrate-legacy-file-data" :fn (mg/resource "app/migrations/sql/0157-migrate-legacy-file-data.sql")} @@ -544,7 +541,10 @@ :fn (mg/resource "app/migrations/sql/0158-partition-file-change.sql")} {:name "0159-add-file-data-storage-object-id" - :fn (mg/resource "app/migrations/sql/0159-add-file-data-storage-object-id.sql")}]) + :fn (mg/resource "app/migrations/sql/0159-add-file-data-storage-object-id.sql")} + + {:name "0159-consumer-generalization" + :fn (mg/resource "app/migrations/sql/0159-consumer-generalization.sql")}]) (defn apply-migrations! [pool name migrations] diff --git a/backend/src/app/migrations/sql/0154-add-job-table.sql b/backend/src/app/migrations/sql/0154-add-job-table.sql index 4c2c7c6bae..502cd08fec 100644 --- a/backend/src/app/migrations/sql/0154-add-job-table.sql +++ b/backend/src/app/migrations/sql/0154-add-job-table.sql @@ -19,8 +19,10 @@ -- database and the tenant is what keeps them from claiming each other's work: -- as an equality the filter is something an index can serve, where a -- `:` prefix could only be matched with a LIKE over text. Every --- query that acts on one instance's work (dispatch, orphan sweep, reschedule, --- cron no-overlap check, submit dedupe) filters on `tenant` for that reason. +-- query that acts on one instance's work (dispatch, lost sweep, orphan sweep, +-- cron no-overlap check) filters on `tenant` for that reason, and the tenant +-- leads the keys of the submit dedupe and exclusivity indexes (see the +-- consumer-generalization migration). -- The prefix still exists, but only in the Redis key the queue's payloads -- travel through, composed in one place by `app.worker/queue-key`. -- @@ -84,24 +86,16 @@ CREATE INDEX job__profile__idx ON job (profile_id, created_at DESC) WHERE profile_id IS NOT NULL; --- Cron no-overlap check and submit dedupe: both look for jobs of one --- tenant with the same name and queue and label that have not started --- yet, so one index serves the two queries. -CREATE INDEX job__name_label__idx - ON job (tenant, name, queue, label) - WHERE status IN ('new', 'scheduled', 'running', 'retry'); - -- Storage GC reference check: it answers whether any job still points -- to a candidate storage object. Rows without a resource are excluded. CREATE INDEX job__resource__idx ON job (resource_id) WHERE resource_id IS NOT NULL; --- Dispatcher recovery: it finds the `scheduled` jobs of its own tenant --- that were pushed to Redis but not claimed within the recovery window. -CREATE INDEX job__scheduled__idx - ON job (tenant, scheduled_at) - WHERE status = 'scheduled'; +-- The lost-sweep index of `scheduled` jobs is created by +-- `0159-consumer-generalization`, on `modified_at` (the hand-off time), +-- which is the column the sweep filters by: building it here on +-- `scheduled_at` would only throw it away in the same transaction. -- Jobs GC expiration: it scans jobs past expires_at while excluding -- running and retry jobs from the expiration delete. @@ -140,9 +134,9 @@ CREATE INDEX job_event__job_kind_created_idx -- jobs GC, which also marks the referenced storage objects as touched in -- the same transaction; a stray `DELETE FROM job` would drop the row -- (and cascade `job_event`) without touching the object, orphaning it --- forever. The GC and the submit dedupe disable the guard inside their --- transaction with `SET LOCAL rules.deletion_protection TO off`, the --- same escape hatch used by `app.tasks.objects-gc`. +-- forever. The jobs GC disables the guard inside its transaction with +-- `SET LOCAL rules.deletion_protection TO off`, the same escape hatch used +-- by `app.tasks.objects-gc`; `app.jobs/submit` never deletes. CREATE OR REPLACE TRIGGER deletion_protection__tgr BEFORE DELETE ON job FOR EACH STATEMENT WHEN ((current_setting('rules.deletion_protection', true) IN ('on', '')) OR diff --git a/backend/src/app/migrations/sql/0157-add-audit-log-pending-table.sql b/backend/src/app/migrations/sql/0157-add-audit-log-pending-table.sql deleted file mode 100644 index ce797dd12c..0000000000 --- a/backend/src/app/migrations/sql/0157-add-audit-log-pending-table.sql +++ /dev/null @@ -1,39 +0,0 @@ --- Work queue for the audit log consumers: one row per (record, --- consumer) pair still pending. Inserting a record creates its --- pendings; each consumer deletes its own on completion, and whoever --- deletes the last pending also deletes the record. Consumer ids are --- fixed in code and never reused: 1 = nexus, 2 = admin console. - -CREATE TABLE audit_log_pending ( - -- DEFERRABLE so the test-suite reset (which deletes table by table - -- with SET CONSTRAINTS ALL DEFERRED) keeps working; the guard still - -- fires when a record with pendings is deleted. Plain DEFERRABLE - -- (initially immediate) is enough: every deleter removes pendings - -- before rows, so the check passes at each statement. - audit_log_id uuid NOT NULL REFERENCES audit_log(id) DEFERRABLE, - consumer smallint NOT NULL CHECK (consumer BETWEEN 1 AND 20), - PRIMARY KEY (consumer, audit_log_id)); - --- Lets the final delete ask "is anything still pending for this --- record?" without scanning the whole queue. -CREATE INDEX ON audit_log_pending (audit_log_id); - --- NOTE: no backfill on purpose. Rows predating this queue (sent or --- not) get no pendings, so the orphan sweep deletes them unsent. --- Losing the unarchived ones at upgrade time is accepted: this --- table is a minutes/hours buffer, not a store. - --- The pending queue replaces the archived_at mark: whoever deletes --- the last pending deletes the record, so the flag column and the --- indexes that served the old mark-and-sweep scans go away. -DROP INDEX IF EXISTS audit_log__created_at__idx; -DROP INDEX IF EXISTS audit_log__archived_at__idx; -DROP INDEX IF EXISTS audit_log_id_archived_at_idx; -ALTER TABLE audit_log DROP COLUMN IF EXISTS archived_at; - --- Serves the consumers' chronological scan (ORDER BY created_at, id): --- one total order for every consumer, so concurrent drains cannot --- deadlock on row locks. Covers the whole ORDER BY (bulk ingest --- shares request-at constantly, so ties are the norm). -CREATE INDEX IF NOT EXISTS audit_log__created_at__idx - ON audit_log (created_at ASC, id); diff --git a/backend/src/app/migrations/sql/0159-add-file-data-storage-object-id.sql b/backend/src/app/migrations/sql/0159-add-file-data-storage-object-id.sql index 3a92172edf..0b58bdbce8 100644 --- a/backend/src/app/migrations/sql/0159-add-file-data-storage-object-id.sql +++ b/backend/src/app/migrations/sql/0159-add-file-data-storage-object-id.sql @@ -27,13 +27,14 @@ ALTER TABLE file_data ADD COLUMN IF NOT EXISTS storage_object_id uuid; --- The CTE is MATERIALIZED so the uuid cast only sees rows that passed the --- regex. A plain subquery can be inlined, and a corrupt value would reach the --- cast and abort the migration. +-- The cast is guarded as well as materialized: corrupt values cannot reach it +-- even if PostgreSQL reorders expressions in the WHERE clause. WITH candidates AS MATERIALIZED ( SELECT file_id, id, - (metadata->>'storage-ref-id')::uuid AS soid + CASE WHEN (metadata->>'storage-ref-id') ~* + '^[0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{12}$' + THEN (metadata->>'storage-ref-id')::uuid END AS soid FROM file_data WHERE backend = 'storage' AND storage_object_id IS NULL @@ -48,6 +49,9 @@ UPDATE file_data AS fd AND fd.id = c.id AND EXISTS (SELECT 1 FROM storage_object AS so WHERE so.id = c.soid); +-- The old key is removed from every row migrated by this statement. A rollback +-- to the previous code after the backfill requires restoring the backup. + -- The deletion check of the foreign key looks a row up by this column for -- every deleted object; without the index it scans all partitions. CREATE INDEX IF NOT EXISTS file_data__storage_object_id__idx diff --git a/backend/src/app/migrations/sql/0159-consumer-generalization.sql b/backend/src/app/migrations/sql/0159-consumer-generalization.sql new file mode 100644 index 0000000000..486da36679 --- /dev/null +++ b/backend/src/app/migrations/sql/0159-consumer-generalization.sql @@ -0,0 +1,216 @@ +-- Consumer generalization: the schema of the whole feature in one step. +-- +-- This single migration replaces the steps this branch used to ship +-- separately (exclusive jobs, the audit cursor and dedupe jobs). The +-- migration runner executes every pending step inside ONE transaction, so +-- several files bought no isolation: their locks piled up until the same +-- commit, and pre-applying one of them out of band did not avoid the +-- others. +-- +-- The pending-queue step that sat between them never shipped and is gone: +-- released versions never created `audit_log_pending`, so this migration +-- creates neither the table nor the `(created_at, id)` index it added. The +-- table itself is dropped below, for a pre-release database that happens +-- to have it. +-- +-- The legacy mark (`archived_at`) and the indexes that served it stay in +-- place on purpose: the old archiver may still be running during the +-- rollout, and a later release removes them. This migration does not build +-- the `(created_at, id)` index either; the consumers order by `(xid, seq)` +-- and telemetry keeps its `(source, created_at)` index plus a sort. +-- +-- The migration is idempotent: every statement is IF NOT EXISTS / IF +-- EXISTS, so it is safe to re-run. Applying it out of band does not make +-- it free, though: the guards turn the statements into no-ops, not into +-- lock-free ones (an ALTER still takes ACCESS EXCLUSIVE and the backfill +-- still scans `audit_log`), so the operator registers the step in +-- `migrations` once the schema is pre-applied (step 0 of the runbook, +-- `operations/2.20.0-consumer-generalization.md`). +-- +-- The statements take the locks in the same order the audit-log producer +-- uses (audit_log, then audit_log_pending, then job), so a deploy that +-- still has the previous version running cannot deadlock against it: the +-- producer inserts the audit row, then its pending (when that version has +-- it), then submits the webhook job, and this migration walks the same +-- order. What it takes last (`job`) is what the producer takes last. + +-- --------------------------------------------------------------------- +-- Audit log cursors: each consumer keeps a position over `audit_log` +-- instead of a pending queue, and the row is deleted when every enabled +-- consumer has passed it. +-- +-- `seq` is the insertion order (a sequence, so it costs no extra WAL); +-- `xid` is the transaction that wrote the row. The pair gives a total +-- order that real commits respect: the consumer reads only up to +-- pg_snapshot_xmin(), the oldest live transaction, so it never steps +-- over a row whose transaction is still open. `xid8` (64 bits) does not +-- wrap, unlike the row `xmin` (32 bits); it exists since PostgreSQL 13. +-- --------------------------------------------------------------------- + +CREATE SEQUENCE IF NOT EXISTS audit_log_seq_seq; + +ALTER TABLE audit_log + ADD COLUMN IF NOT EXISTS seq bigint, + -- Fast default (no table rewrite): the rows that already exist get 0 + -- and sort before every real xid8. They are the history, and the first + -- deploy replays them (the consumers create their cursor at (0, 0) on + -- their first pass). + ADD COLUMN IF NOT EXISTS xid xid8 NOT NULL DEFAULT '0'::xid8; + +ALTER SEQUENCE audit_log_seq_seq OWNED BY audit_log.seq; + +-- Reserve the low range for the history: the new rows start above the +-- backfill, so any `ORDER BY seq` stays chronological and the backfill +-- can never collide with a value produced by nextval. The margin covers +-- the rows the previous version may insert between this count and the +-- SET DEFAULT (a few statements away). +-- +-- Only the first application sets it: a sequence that was never called +-- (`last_value = 1` with `is_called = false`) is intact; any other state +-- (reserved by the operator, in use, or already reserved by a previous +-- run) must be left alone, because moving it back would reuse values above +-- the history and break the reserve. +SELECT setval('audit_log_seq_seq', + (SELECT count(*) + 1000000 FROM audit_log), + false) + WHERE (SELECT last_value FROM audit_log_seq_seq) = 1 + AND NOT (SELECT is_called FROM audit_log_seq_seq); + +-- From here no new row can be NULL: the previous version keeps inserting +-- during the rollout and must leave the columns filled. +ALTER TABLE audit_log + ALTER COLUMN seq SET DEFAULT nextval('audit_log_seq_seq'), + ALTER COLUMN xid SET DEFAULT pg_current_xact_id(); + +-- Backfill the history with 1..N, in the order the previous version sent +-- it (`created_at, id`), so the replay keeps the archive order. A single +-- statement: it takes row locks, never a table lock, so the previous +-- version keeps inserting and the statement can be interrupted and +-- re-run (it filters `seq IS NULL`). The sort is on purpose: the +-- `(created_at, id)` index belonged to the pending queue and never +-- shipped, so there is no index to serve this order. +UPDATE audit_log + SET seq = ordered.rn + FROM (SELECT id, row_number() OVER (ORDER BY created_at, id) AS rn + FROM audit_log + WHERE seq IS NULL) ordered + WHERE audit_log.id = ordered.id; + +ALTER TABLE audit_log + ALTER COLUMN seq SET NOT NULL; + +-- `source` is nullable since 0064 (its backfill only covered the rows of +-- that moment). The producer has written it ever since, but the GC excludes +-- telemetry shadows with `source NOT LIKE 'telemetry:%'`, and with a NULL +-- source that predicate is unknown: neither the GC nor the telemetry task +-- would ever delete such a row. Fill what is left and close the column. +UPDATE audit_log SET source = 'backend' WHERE source IS NULL; + +ALTER TABLE audit_log + ALTER COLUMN source SET NOT NULL; + +-- Covers both predicates of the scan: the cursor comparison and the +-- snapshot cut are Index Cond, so the index serves the range and the +-- (xid, seq) order of the scan. +CREATE UNIQUE INDEX IF NOT EXISTS audit_log__xid__seq__idx + ON audit_log (xid, seq); + +-- One row per consumer: its position, updated in the same transaction +-- that sends the chunk. Consumer ids live in app.loggers.audit.consumers +-- and are never reused: 1 = nexus, 2 = admin console. +-- +-- The table starts empty on purpose: the migration does not know which +-- consumers are enabled. `ensure-cursor` creates the row at (0, 0) on a +-- consumer's first pass, so the first deploy **replays** the unarchived +-- buffer instead of dropping it (the old archiver is disabled and the +-- already-archived rows are deleted before deploying, see the NOTE). +CREATE TABLE IF NOT EXISTS audit_log_consumer_cursor ( + consumer smallint NOT NULL, + xid xid8 NOT NULL, + seq bigint NOT NULL, + updated_at timestamptz NOT NULL DEFAULT now(), + PRIMARY KEY (consumer)); + +-- --------------------------------------------------------------------- +-- The old pending queue never shipped: this release's producer does not +-- fill it, and no released version ever created the table. Only a +-- pre-release database that ran the pending-queue step has it, and that +-- one may keep rows left behind. Drop the table: after the rollout nobody +-- drains the queue, so a pending row would make the new GC fail on every +-- pass when it deletes the row it references. A released database does +-- not have it, so this is a no-op there. +-- --------------------------------------------------------------------- + +DROP TABLE IF EXISTS audit_log_pending; + +-- --------------------------------------------------------------------- +-- Jobs: at most one live exclusive job and at most one not-started +-- dedupe job per (tenant, name, queue, label). +-- +-- `exclusive` marks the row of a `::jobs/skip-if-active` submit, and the +-- index admits a single live row (new, scheduled, running or retry) for +-- its key. `dedupe` marks a `::jobs/dedupe`/`::jobs/debounce` submit, and +-- its index admits a single `new` row. The submit no longer counts or +-- deletes before inserting: the partial unique index decides and the +-- insert resolves the conflict with `ON CONFLICT`, so the exclusion holds +-- across backends. +-- +-- Both columns default to false, so the rows that never asked for the +-- option stay out of the indexes (the label stays a label) and the +-- indexes start empty: no cleanup of duplicates is needed. +-- --------------------------------------------------------------------- + +ALTER TABLE job + ADD COLUMN IF NOT EXISTS exclusive boolean NOT NULL DEFAULT false; + +ALTER TABLE job + ADD COLUMN IF NOT EXISTS dedupe boolean NOT NULL DEFAULT false; + +CREATE UNIQUE INDEX IF NOT EXISTS job__live_exclusive__uniq + ON job (tenant, name, queue, label) + WHERE exclusive + AND status IN ('new', 'scheduled', 'running', 'retry'); + +CREATE UNIQUE INDEX IF NOT EXISTS job__new_dedupe__uniq + ON job (tenant, name, queue, label) + WHERE dedupe + AND status = 'new'; + +-- The plain index over (tenant, name, queue, label) served the old cron +-- no-overlap probe. Nothing reads it now: the cron resolves the overlap at +-- insert time with a `::jobs/skip-if-active` submit (that is, +-- `job__live_exclusive__uniq`), which is one index instead of a probe plus +-- a race. Bases built from an early 0154 drop it here. +DROP INDEX IF EXISTS job__name_label__idx; + +-- The lost sweep looks for `scheduled` rows by their hand-off time +-- (`modified_at`), not by their due time: the old index served the removed +-- reschedule and only narrows by tenant now. The `job` table is not in +-- production yet, so the rebuild does not need CONCURRENTLY. +DROP INDEX IF EXISTS job__scheduled__idx; + +CREATE INDEX IF NOT EXISTS job__scheduled__idx + ON job (tenant, modified_at) + WHERE status = 'scheduled'; + +-- --------------------------------------------------------------------- +-- NOTE: on an instance with a large `job` or `audit_log` table, apply this +-- migration out of band before deploying. The runner executes all pending +-- steps in a single transaction, so the plain indexes below hold a SHARE +-- lock on the table for the whole build and every ALTER keeps its ACCESS +-- EXCLUSIVE until that one commit; on tables that keep taking writes (every +-- file save inserts a job and an audit row) that is minutes of stalled +-- writes. +-- +-- The complete procedure (lock timeouts, the `CHECK NOT VALID` dance, the +-- `indisvalid` gate, registering the step and the optional post-rollout +-- drop for a pre-release database) lives in +-- `operations/2.20.0-consumer-generalization.md`. Pre-applying does not make +-- the migration free by itself: the guards skip the work, not the locks, so +-- registering the step is what makes the runner skip the group. +-- +-- The new consumers read only rows below +-- `pg_snapshot_xmin(pg_current_snapshot())`, the oldest transaction of the +-- whole database: a long write transaction (a binfile import runs in one for +-- up to 30 minutes) pauses archiving and the GC until it commits. Nothing is +-- lost and the lag gauge shows it; it is not a stuck consumer. diff --git a/backend/src/app/nitrate.clj b/backend/src/app/nitrate.clj index 6fb33b6ebf..1a9bc9de95 100644 --- a/backend/src/app/nitrate.clj +++ b/backend/src/app/nitrate.clj @@ -113,8 +113,22 @@ :status status :hint (str "nitrate HTTP " status " at " uri)) nil)) - (= status 204) ;; 204 doesn't return any body + ;; A 2xx with no expected body is success: 204 by contract, and any + ;; other 2xx for the methods that never return a body. Building a + ;; coercer from a nil schema throws before the try below can catch it. + (and (<= 200 status 299) + (or (= status 204) (nil? schema))) nil + ;; A redirect is a configuration error (a proxy normalising the url, + ;; a trailing slash, an auth hop), never a success: raise a typed + ;; error so the caller does not read it as an empty or confirmed + ;; response. This sits outside `with-retries` on purpose. + (<= 300 status 399) + (do + (l/error :hint "nitrate request was redirected" :uri uri :status status) + (ex/raise :type :nitrate-http-error + :status status + :hint (str "nitrate redirected, HTTP " status " at " uri))) :else ;; For success status codes, validate the response (let [coercer-http (sm/coercer schema :type :validation @@ -508,8 +522,8 @@ (defn- ingest-audit-log-api "POST a batch of audit events to Admin Console `/api/audit-log`. - Expects HTTP 204; returns nil on success. 4xx raises when - `:throw-on-error?` is set." + Any 2xx is success and returns nil (the endpoint answers 204 by + contract). 4xx raises when `:throw-on-error?` is set." [cfg {:keys [events] :as params}] (let [request-params {:events (mapv ->audit-event events)} params (assoc params diff --git a/backend/src/app/redis.clj b/backend/src/app/redis.clj index 146e8f54fe..6389511b70 100644 --- a/backend/src/app/redis.clj +++ b/backend/src/app/redis.clj @@ -71,6 +71,7 @@ "Public API of default redis connection" (-publish [_ topic message]) (-rpush [_ key payload]) + (-llen [_ key]) (-blpop [_ timeout keys]) (-eval [_ script]) (-get [_ key]) @@ -195,6 +196,9 @@ (catch RedisCommandInterruptedException cause (throw (InterruptedException. (ex-message cause)))))) + (-llen [_ key] + (.llen cmd ^String key)) + (-blpop [_ keys timeout] (try (let [keys (into-array String keys)] @@ -445,6 +449,11 @@ (assert (every? string? keys) "only string keys allowed") (-blpop conn keys timeout))) +(defn llen + [conn key] + (assert (string? key) "key must be string instance") + (-llen conn key)) + (defn rpush [conn key elements] (assert (string? key) "key must be string instance") diff --git a/backend/src/app/rpc.clj b/backend/src/app/rpc.clj index f9c9d7eabf..84e5769230 100644 --- a/backend/src/app/rpc.clj +++ b/backend/src/app/rpc.clj @@ -212,6 +212,13 @@ :else (f cfg params)))))) +;; Only the methods that declare `::db/transaction true` get a transaction; +;; every wrapper applied after this point in `wrap` runs outside it. That is +;; the case of `wrap-audit`: the event is written after the command commits, +;; so a failure there cannot roll the action back and the client gets an +;; error over committed work. Making the transaction the default for every +;; command, so the audit event can move inside it, is project P8 in TODO.md +;; and is out of scope here. (defn- wrap-db-transaction [_ f mdata] (if (::db/transaction mdata) @@ -221,9 +228,9 @@ (defn- wrap-audit [_ f mdata] - (if (or (contains? cf/flags :webhooks) - (contains? cf/flags :audit-log) - (contains? cf/flags :telemetry)) + (if (or cf/webhooks-enabled? + cf/audit-persist? + cf/telemetry-enabled?) (if-not (::audit/skip mdata) (fn [cfg params] (let [result (f cfg params)] diff --git a/backend/src/app/rpc/commands/audit.clj b/backend/src/app/rpc/commands/audit.clj index 4d5f26a402..a0ca727f19 100644 --- a/backend/src/app/rpc/commands/audit.clj +++ b/backend/src/app/rpc/commands/audit.clj @@ -16,7 +16,6 @@ [app.db :as db] [app.http :as-alias http] [app.loggers.audit :as audit] - [app.loggers.audit.consumer-task :as consumer] [app.loggers.database :as loggers.db] [app.loggers.mattermost :as loggers.mm] [app.metrics :as mtx] @@ -25,8 +24,7 @@ [app.rpc.doc :as-alias doc] [app.rpc.helpers :as rph] [app.util.inet :as inet] - [app.util.services :as sv] - [clojure.set :as set])) + [app.util.services :as sv])) (def ^:private event-columns [:id @@ -125,10 +123,10 @@ (defn- insert-rows! [tx rows] (when (seq rows) - (db/insert-many! tx :audit-log event-columns rows) - ;; Same transaction as the rows: every ingested record is born - ;; pending for every enabled consumer (see consumer-task). - (consumer/queue-pendings tx (map first rows)))) + ;; The producer only inserts: each consumer reads the row from its + ;; own cursor, so there is no queue to fill here (and no consumer + ;; contract to keep in sync with this namespace). + (db/insert-many! tx :audit-log event-columns rows))) (defn- handle-events [{:keys [::db/pool] :as cfg} params] @@ -141,15 +139,15 @@ (run! (partial loggers.db/emit cfg) events) (run! (partial loggers.mm/emit cfg) events)) - ;; Rows and their pendings commit together in one transaction: a - ;; row never outlives its queue. Both faces are best-effort, so a - ;; failure drops the whole batch (counted below as failed); the + ;; The rows of both faces commit together (the consumers read them + ;; from their own cursors afterwards). Both faces are best-effort, + ;; so a failure drops the whole batch (counted below as failed); the ;; endpoint still answers without breaking the request. (let [metrics (mtx/instance cfg) - full-rows (if (contains? cf/flags :audit-log) + full-rows (if cf/audit-persist? (into [] xf:map-event-row events) []) - shadow-rows (if (contains? cf/flags :telemetry) + shadow-rows (if cf/telemetry-enabled? (into [] xf:map-telemetry-event-row events) [])] (try @@ -196,10 +194,14 @@ ::doc/skip true ::doc/added "1.17"} [{:keys [::db/pool] :as cfg} params] - (let [telemetry? (contains? cf/flags :telemetry) - audit-log? (contains? cf/flags :audit-log) + (let [telemetry? cf/telemetry-enabled? + ;; :error-reporting also needs the batch: the crash events reach + ;; the reporters here, before any row is written. The wire signal + ;; in get-environment-data includes the same case. enabled? (and (not (db/read-only? pool)) - (or audit-log? telemetry?))] + (or cf/audit-persist? + telemetry? + cf/error-reporting-enabled?))] (when enabled? (try (handle-events cfg params) @@ -213,15 +215,19 @@ ;; GET-ENVIRONMENT-DATA ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; -(def ^:private enabled-flags - "Subset of the backend flags that are safe to expose publicly and that - drive frontend event collection." - #{:audit-log :telemetry}) - (defn- environment-data + "The audit-log signal on the wire means the backend stores full rows. It is + derived from the boot-computed persistence value, not from a configurable + flag, and also covers error reporting so frontend crash events keep flowing." [] {:deployment (if (cf/get :is-saas) "saas" "selfhost") - :flags (set/intersection cf/flags enabled-flags)}) + :flags (cond-> #{} + (or cf/audit-persist? + cf/error-reporting-enabled?) + (conj :audit-log) + + cf/telemetry-enabled? + (conj :telemetry))}) (def ^:private schema:get-environment-data-result [:map diff --git a/backend/src/app/rpc/commands/files.clj b/backend/src/app/rpc/commands/files.clj index b49a0951f2..ba2428f576 100644 --- a/backend/src/app/rpc/commands/files.clj +++ b/backend/src/app/rpc/commands/files.clj @@ -1033,7 +1033,10 @@ :modified-at (ct/now) :has-media-trimmed false)))) -(defn- get-referenced-files +(defn get-referenced-files + "Ids of the live files using the given file as a library, oldest + first. Public so the delete cascade can capture the dependents + before removing the relations." [cfg library-id] (->> (db/exec! cfg [sql:get-referenced-files library-id]) (into [] bfc/xf-map-id))) @@ -1147,23 +1150,17 @@ file (db/update! conn :file {:deleted-at (ct/in-future delay)} {:id file-id} - {::db/return-keys [:id :name :is-shared :deleted-at - :project-id :created-at :modified-at]}) - ;; Files using the library, captured before their relations are removed - file-ids (when (:is-shared file) - (get-referenced-files conn file-id))] - - ;; Remove all possible relations for that file - (db/delete! conn :file-library-rel - {:library-file-id file-id}) - + {::db/return-keys [:name :deleted-at + :project-id :created-at :modified-at]})] + ;; Only the mark happens here, so the file disappears at once. The + ;; cascade job captures the files using this one as a library, + ;; removes the relations and absorbs them: the relations must still + ;; exist when the job runs. (jobs/submit cfg {::jobs/name :delete-object - ::jobs/params (cond-> {:object :file - :deleted-at (:deleted-at file) - :id file-id} - (some? file-ids) - (assoc :file-ids file-ids))}) + ::jobs/params {:object :file + :deleted-at (:deleted-at file) + :id file-id}}) file)) (def ^:private diff --git a/backend/src/app/rpc/commands/files_update.clj b/backend/src/app/rpc/commands/files_update.clj index c47b018c0f..8dcae7ab7d 100644 --- a/backend/src/app/rpc/commands/files_update.clj +++ b/backend/src/app/rpc/commands/files_update.clj @@ -340,6 +340,12 @@ deleted-at (ct/plus timestamp (ct/duration {:hours 1})) + ;; The id of the change row (xlog) goes to the audit event + ;; too: a consumer that reads the event needs to know what the + ;; save changed, and that row is the only place with it. + file-change-id + (uuid/next) + repair (::repair file) @@ -397,7 +403,7 @@ ;; file are not repeated here (nobody reads them from a revision, and ;; they are duplicated on every save). (db/insert! conn :file-change - {:id (uuid/next) + {:id file-change-id :kind fchanges/revision-kind :session-id session-id :profile-id profile-id @@ -424,18 +430,35 @@ (some? rejected) (assoc :rejected rejected)) {::audit/replace-props - {:id (:id file) - :name (:name file) - :features (:features file) - :project-id (:project-id file) - :team-id (:team-id file) - ;; NOTE: `file` carries the post-update revn here (`revn` - ;; above is the pre-update one kept for the response), and - ;; `modified-at` mirrors `persist-file!`: same `::timestamp` - ;; fallback, so both agree to the millisecond when the - ;; request timestamp exists. - :revn (if saved? (:revn file) revn) - :modified-at (or timestamp (ct/now))}})))) + (cond-> {:id (:id file) + :name (:name file) + :features (:features file) + :project-id (:project-id file) + :team-id (:team-id file) + ;; NOTE: `file` carries the post-update revn here (`revn` + ;; above is the pre-update one kept for the response). + :revn (if saved? (:revn file) revn) + ;; How the save ended: every change written, the accepted + ;; prefix written and the rest rejected, or nothing + ;; written. A consumer must not read `update-file` as a + ;; state change without looking at this. + :outcome (cond + (not saved?) "rejected" + (some? rejected) "partial" + :else "saved") + ;; A rejected save does not call `persist-file!`, so the + ;; row keeps its previous instant: mirror the persisted + ;; value instead of claiming a modification. + :modified-at (if saved? + (or timestamp (ct/now)) + (:modified-at file))} + + ;; The id of the change row inserted above: what the save + ;; changed lives in that row, not in the event. A fully rejected + ;; save writes no row, and an id that resolves to nothing cannot + ;; be told apart from one the GC already purged. + saved? + (assoc :file-change-id file-change-id))})))) (defn get-file "Get not-decoded file, only decodes the features set. diff --git a/backend/src/app/rpc/commands/management.clj b/backend/src/app/rpc/commands/management.clj index 79cf51781b..75ee867bec 100644 --- a/backend/src/app/rpc/commands/management.clj +++ b/backend/src/app/rpc/commands/management.clj @@ -25,6 +25,7 @@ [app.rpc.commands.projects :as proj] [app.rpc.commands.teams :as teams] [app.rpc.doc :as-alias doc] + [app.rpc.helpers :as rph] [app.setup :as-alias setup] [app.setup.templates :as tmpl] [app.storage.tmp :as tmp] @@ -273,7 +274,10 @@ ;; --- COMMAND: Move file (def sql:get-files - "select id, features, project_id from file where id = ANY(?)") + "select f.id, f.features, f.project_id, p.team_id + from file as f + join project as p on (p.id = f.project_id) + where f.id = ANY(?)") (def sql:move-files "update file set project_id = ? where id = ANY(?)") @@ -301,7 +305,15 @@ (map files/decode-row)) source (into #{} (map :project-id) files) pids (->> (conj source project-id) - (db/create-array conn "uuid"))] + (db/create-array conn "uuid")) + ;; A move carries a single origin team in its props: refuse + ;; mixed-origin requests up front instead of guessing. + source-teams (into #{} (keep :team-id) files)] + + (when (> (count source-teams) 1) + (ex/raise :type :validation + :code :cant-move-from-multiple-teams + :hint "Unable to move files from different teams at once")) (when (contains? source project-id) (ex/raise :type :validation @@ -315,7 +327,9 @@ (doseq [project-id source] (proj/check-edition-permissions! conn profile-id project-id)) - ;; Check the team compatibility + ;; Check the team compatibility, then move: the destination team + ;; is already loaded here, so the event props below need no extra + ;; query. (let [orig-team (teams/get-team conn :profile-id profile-id :project-id (first source)) dest-team (teams/get-team conn :profile-id profile-id :project-id project-id)] (cfeat/check-teams-compatibility! orig-team dest-team) @@ -323,26 +337,30 @@ ;; Check if all pending to move files are compaib (let [features (cfeat/get-team-enabled-features cf/flags dest-team)] (doseq [file files] - (cfeat/check-file-features! features (:features file))))) + (cfeat/check-file-features! features (:features file)))) - ;; move all files to the project - (db/exec-one! conn [sql:move-files project-id fids]) + ;; move all files to the project + (db/exec-one! conn [sql:move-files project-id fids]) - ;; delete possible broken relations on moved files - (db/exec-one! conn [sql:delete-broken-relations pids]) + ;; delete possible broken relations on moved files + (db/exec-one! conn [sql:delete-broken-relations pids]) - ;; Update the modification date of the all affected projects - ;; ensuring that the destination project is the most recent one. - (loop [project-ids (into (list project-id) source) - modified-at (ct/now)] - (when-let [project-id (first project-ids)] - (db/update! conn :project - {:modified-at modified-at} - {:id project-id}) - (recur (rest project-ids) - (ct/plus modified-at 10)))) + ;; Update the modification date of the all affected projects + ;; ensuring that the destination project is the most recent one. + (loop [project-ids (into (list project-id) source) + modified-at (ct/now)] + (when-let [project-id (first project-ids)] + (db/update! conn :project + {:modified-at modified-at} + {:id project-id}) + (recur (rest project-ids) + (ct/plus modified-at 10)))) - nil)) + ;; Both teams travel explicitly in the event props so a consumer + ;; never joins production tables. + (rph/with-meta (rph/wrap) + {::audit/props {:source-team-id (first source-teams) + :dest-team-id (:id dest-team)}})))) (def ^:private schema:move-files @@ -353,6 +371,7 @@ (sv/defmethod ::move-files "Move a set of files from one project to other." {::doc/added "1.16" + ::doc/changes [["2.20" "Files from more than one source team are rejected with :cant-move-from-multiple-teams"]] ::webhooks/event? true ::sm/params schema:move-files} [cfg {:keys [::rpc/profile-id] :as params}] @@ -411,7 +430,11 @@ ;; delete possible broken relations on moved files (db/exec-one! conn [sql:delete-broken-relations pids]) - nil)) + ;; Both teams travel explicitly in the event props so a consumer + ;; never joins production tables. + (rph/with-meta (rph/wrap) + {::audit/props {:source-team-id (:team-id project) + :dest-team-id team-id}}))) (def ^:private schema:move-project diff --git a/backend/src/app/rpc/commands/profile.clj b/backend/src/app/rpc/commands/profile.clj index 2a9ee61118..864ad52b8f 100644 --- a/backend/src/app/rpc/commands/profile.clj +++ b/backend/src/app/rpc/commands/profile.clj @@ -645,6 +645,11 @@ ;; --- HELPERS +;; Owned teams with their participant count. The predicate over +;; `t.deleted_at` is injected: the read path asks for live teams, and the +;; restore path asks for the ones the delete cascade stamped with the +;; profile's own instant. Both predicates are constants of this namespace, +;; never operator input. (def sql:owned-teams "WITH owner_teams AS ( SELECT tpr.team_id AS id @@ -652,7 +657,7 @@ JOIN team AS t ON (t.id = tpr.team_id) WHERE tpr.is_owner IS TRUE AND tpr.profile_id = ? - AND t.deleted_at IS NULL + AND %s ) SELECT tpr.team_id AS id, count(tpr.profile_id) - 1 AS participants @@ -661,8 +666,16 @@ GROUP BY 1") (defn get-owned-teams - [conn profile-id] - (db/exec! conn [sql:owned-teams profile-id])) + "Teams the profile owns. `:deleted-at` returns the ones soft-deleted at + that exact instant: the delete cascade stamps the children with the + parent's date, so the restore path uses it to bring back only what the + cascade took. Without it the answer holds live teams only." + [conn profile-id & {:keys [deleted-at]}] + (if (some? deleted-at) + (db/exec! conn [(format sql:owned-teams "t.deleted_at = ?::timestamptz") + profile-id deleted-at]) + (db/exec! conn [(format sql:owned-teams "t.deleted_at IS NULL") + profile-id]))) (def ^:private sql:profile-existence "select exists (select * from profile diff --git a/backend/src/app/rpc/commands/teams.clj b/backend/src/app/rpc/commands/teams.clj index 4ab171e08d..e9d56b9a7e 100644 --- a/backend/src/app/rpc/commands/teams.clj +++ b/backend/src/app/rpc/commands/teams.clj @@ -654,7 +654,7 @@ action owner, defaulting to the member) and `:rpc-params` (request context, which sibling events get from `prepare-rpc-event`)." [cfg {:keys [profile-id team-id] :as params} {:keys [actor-id rpc-params]}] - (when (contains? cf/flags :audit-log) + (when cf/audit-persist? (audit/submit cfg (-> (audit/event-from-rpc-params (or rpc-params {})) (assoc :name "add-team-member" diff --git a/backend/src/app/srepl/main.clj b/backend/src/app/srepl/main.clj index a0bab89055..337f4ef3aa 100644 --- a/backend/src/app/srepl/main.clj +++ b/backend/src/app/srepl/main.clj @@ -767,12 +767,16 @@ "Mark a project for deletion" [file-id] (let [file-id (h/parse-uuid file-id) - tnow (ct/now)] + tnow (ct/now) + ;; The event carries its team, so a consumer can key on it + ;; without joining production tables. + team-id (first (audit/get-team-ids sys/system {:file-ids [file-id]}))] (audit/insert sys/system {:name "delete-file" :type "action" - :props {:id file-id} + :props (cond-> {:id file-id} + (some? team-id) (assoc :team-id team-id)) :context {:triggered-by "srepl" :cause "explicit call to delete-file!"} :tracked-at tnow}) @@ -792,13 +796,17 @@ (when-let [file (db/get* system :file {:id file-id} {::db/remove-deleted false - ::sql/columns [:id :name]})] - (audit/insert system - {:name "restore-file" - :type "action" - :props file - :context {:triggered-by "srepl" - :cause "explicit call to restore-file!"}}) + ::sql/columns [:id :name :project-id]})] + (let [team-id (first (audit/get-team-ids + system + {:project-ids [(:project-id file)]}))] + (audit/insert system + {:name "restore-file" + :type "action" + :props (cond-> file + (some? team-id) (assoc :team-id team-id)) + :context {:triggered-by "srepl" + :cause "explicit call to restore-file!"}})) (#'files/restore-files conn [file-id])) :restored)))) @@ -920,7 +928,7 @@ (audit/insert system {:name "restore-team" :type "action" - :props team + :props (assoc team :team-id (:id team)) :context {:triggered-by "srepl" :cause "explicit call to restore-team!"}}) @@ -948,7 +956,10 @@ :deleted)) (defn restore-profile! - "Mark a team and all related objects as not deleted" + "Mark a profile and the teams its deletion cascade soft-deleted as not + deleted. Answers `:already-active` without touching anything when the + profile is not deleted: running the cascade over live teams would revive + font variants and files deleted on their own." [profile-id] (let [profile-id (h/parse-uuid profile-id)] (db/tx-run! sys/system @@ -957,22 +968,34 @@ {:id profile-id} {::db/remove-deleted false}) (profile/decode-row))] - (audit/insert system - {:name "restore-profile" - :type "action" - :props (audit/profile->props profile) - :context {:triggered-by "srepl" - :cause "explicit call to restore-profile!"}}) + (if (nil? (:deleted-at profile)) + :already-active + (do + (db/update! system :profile + {:deleted-at nil} + {:id profile-id} + {::db/return-keys false}) - (db/update! system :profile - {:deleted-at nil} - {:id profile-id} - {::db/return-keys false}) + ;; Owned teams are still soft-deleted here, so resolve + ;; them through the membership table, and only the ones + ;; the cascade stamped with the profile's own instant: a + ;; team the owner deleted earlier stays deleted. + (let [team-ids (mapv :id (profile/get-owned-teams system (:id profile) + :deleted-at (:deleted-at profile)))] + (doseq [id team-ids] + (restore-team* system id)) - (doseq [{:keys [id]} (profile/get-owned-teams system profile-id)] - (restore-team* system id)) + ;; The event carries the restored teams explicitly, + ;; so a consumer never joins production tables. + (audit/insert system + {:name "restore-profile" + :type "action" + :props (assoc (audit/profile->props profile) + :team-ids team-ids) + :context {:triggered-by "srepl" + :cause "explicit call to restore-profile!"}})) - :restored))))) + :restored))))))) (defn delete-profiles-in-bulk! [system path] @@ -1007,13 +1030,15 @@ the whole table, and enqueues the job instead of running it inline, so it can be used against production in bounded batches. The job is idempotent (it only marks the children with the row's own date), `label` is the row id and - `dedupe` drops not-yet-due duplicates. + `dedupe` skips the row when its job is still not started. Options: `:limit` (max rows read), `:ids` (restrict the run to these row ids) and `:dry-run?` (count without enqueueing). `table` and `object` are internal constants, never operator input. - Returns `{:found n :submitted m}`. + Returns `{:found n :submitted m}`. `m` counts the rows whose job was + really enqueued: a row whose `dedupe` key still holds a not-started job + is counted in `n` but not in `m`. Note: the submissions are given the outer `system` (with the pool) on purpose; handing them the read-only transaction cfg would write on it." @@ -1028,20 +1053,22 @@ (fn [{:keys [::db/conn]}] (let [query (cond-> [sql] (seq ids) (conj (db/create-array conn "uuid" ids))) - found (reduce (fn [total {:keys [id deleted-at]}] - (when-not dry-run? - (jobs/submit system - {::jobs/name :delete-object - ::jobs/params {:object object - :deleted-at deleted-at - :id id} - ::jobs/label (str id) - ::jobs/dedupe true})) - (inc total)) - 0 - (db/plan conn query))] + [found submitted] + (reduce (fn [[found submitted] {:keys [id deleted-at]}] + (if (and (not dry-run?) + (jobs/submit system + {::jobs/name :delete-object + ::jobs/params {:object object + :deleted-at deleted-at + :id id} + ::jobs/label (str id) + ::jobs/dedupe true})) + [(inc found) (inc submitted)] + [(inc found) submitted])) + [0 0] + (db/plan conn query))] {:found found - :submitted (if dry-run? 0 found)}))))) + :submitted submitted}))))) (defn process-deleted-profiles-cascade "Replay the deletion cascade of every soft-deleted profile." diff --git a/backend/src/app/tasks/delete_object.clj b/backend/src/app/tasks/delete_object.clj index b45e0d8ca2..6fc2b24fc1 100644 --- a/backend/src/app/tasks/delete_object.clj +++ b/backend/src/app/tasks/delete_object.clj @@ -7,6 +7,7 @@ (ns app.tasks.delete-object "A generic task for object deletion cascade handling" (:require + [app.common.data :as d] [app.common.logging :as l] [app.common.schema :as sm] [app.common.time :as ct] @@ -21,6 +22,14 @@ (def ^:dynamic *team-deletion* false) +(def ^:dynamic *deleting-file-ids* + "Ids of the files the running cascade is going to delete. Bound by + the `:project` method with its own files so `:file` only absorbs the + dependents that survive: absorbing into files that die in the same + run is wasted work. Team deletions need no such set: they bind + `*team-deletion*` instead, which skips absorb entirely." + nil) + (def ^:dynamic *cascade* "Accumulates the ids marked deleted by the current cascade execution, grouped by entity. Bound by the task entry point; nil on direct calls @@ -79,14 +88,45 @@ {::db/return-keys false}) (when (and (:is-shared file) + ;; Team deletions skip absorb entirely: every file dies + ;; in the same cascade, so no survivor can absorb. (not *team-deletion*)) - ;; NOTE: we don't prevent file deletion on absorb operation failure - (try - (db/tx-run! cfg files/absorb-library id :file-ids file-ids) - (catch Throwable cause - (l/warn :hint "error on absorbing library" - :file-id id - :cause cause)))) + ;; NOTE: we don't prevent file deletion on absorb operation failure. + ;; Only live dependents outside this cascade matter: if the whole + ;; closure dies together there is nothing to absorb and the + ;; relations die with their files (both FKs are ON DELETE CASCADE). + ;; DEPRECATED `:file-ids`: jobs queued by the previous release + ;; already deleted the relations in the RPC, so the live lookup + ;; finds nothing. Trust their snapshot for one release; new jobs + ;; never carry the key. + (let [doomed (or *deleting-file-ids* #{}) + dependents (if (seq file-ids) + (into [] (remove doomed) file-ids) + (into [] (remove doomed) + (files/get-referenced-files conn id)))] + (when (seq dependents) + ;; The relations are dropped unconditionally, before the absorb + ;; runs: they are dead weight once the library is being + ;; deleted, regardless of whether the absorb succeeds. A + ;; failure here fails the job instead of being swallowed, + ;; like every other write in the cascade. + (db/delete! conn :file-library-rel {:library-file-id id}) + ;; Runs in the caller transaction: a nested transaction here + ;; would join it anyway, so the call goes directly. The + ;; savepoint contains an absorb failure: it rolls back the + ;; whole absorb and the cascade goes on. + ;; Nothing retries the absorb itself: the job succeeds and the + ;; dependents keep the data they had before the absorb ran. + (let [sp (db/savepoint conn "absorb-library")] + (try + (files/absorb-library cfg id :file-ids dependents) + (catch Throwable cause + (db/rollback! conn sp) + (l/warn :hint "error on absorbing library" + :file-id id + :cause cause)) + (finally + (db/release! conn sp))))))) ;; Mark file change to be deleted, but only the rows that do not carry a ;; date yet. A revision is born already scheduled for deletion one hour @@ -134,12 +174,16 @@ {:id id} {::db/return-keys false}) - (doseq [file (db/query conn :file - {:project-id id} - {::db/columns [:id :deleted-at]})] - (delete-object cfg (assoc file - :object :file - :deleted-at deleted-at))) + (let [files (db/query conn :file + {:project-id id} + {::db/columns [:id :deleted-at]})] + ;; Every file below dies in this run: absorbing a library into + ;; them is wasted work, so the files only absorb survivors. + (binding [*deleting-file-ids* (into #{} d/xf:map-id files)] + (doseq [file files] + (delete-object cfg (assoc file + :object :file + :deleted-at deleted-at))))) (record :projects id)) @@ -193,29 +237,38 @@ [:deleted-at ::ct/inst] [:id ::sm/uuid] [:file-id {:optional true} ::sm/uuid] + ;; DEPRECATED: only jobs queued before the cascade owned the + ;; relations carry it (their RPC already deleted them). Remove once + ;; no such rows can be left. [:file-ids {:optional true} [:vector ::sm/uuid]]]) (defn execute-delete-object "Plain job handler: run the delete-object multimethod on the provided - params inside a single transaction, tracking the cascade and reporting - it as an audit event when anything was marked." + params, tracking the cascade and reporting it as an audit event when + anything was marked. Runs inside the caller transaction (the job-def + handler opens it with `tx-run!`)." [cfg params] - (db/tx-run! cfg - (fn [cfg] - (when (top-exists? cfg params) - (binding [*cascade* (atom {:teams [] :projects [] :files []})] - (delete-object cfg params) - (let [{:keys [teams projects files]} @*cascade*] - (when (or (seq teams) (seq projects) (seq files)) - (audit/submit cfg {:name "delete-object-with-cascade" - :type "action" - :props {:object (:object params) - :id (:id params) - :deleted-at (:deleted-at params) - :cascade {:team-ids teams - :project-ids projects - :file-ids files}} - :context {:triggered-by "delete-object"}})))))))) + (when (top-exists? cfg params) + (binding [*cascade* (atom {:teams [] :projects [] :files []})] + (delete-object cfg params) + (let [{:keys [teams projects files]} @*cascade*] + (when (or (seq teams) (seq projects) (seq files)) + ;; Flat team ids, so a consumer can key on them without + ;; joining production tables. + (let [team-ids (audit/get-team-ids cfg + {:team-ids teams + :project-ids projects + :file-ids files})] + (audit/submit cfg {:name "delete-object-with-cascade" + :type "action" + :props {:object (:object params) + :id (:id params) + :deleted-at (:deleted-at params) + :team-ids (vec team-ids) + :cascade {:team-ids teams + :project-ids projects + :file-ids files}} + :context {:triggered-by "delete-object"}}))))))) (defmethod ig/assert-key ::job-def [_ params] @@ -228,6 +281,6 @@ ::jobs/schema schema:delete-object-params ::jobs/handler (fn [_context params] - (execute-delete-object cfg params)) + (db/tx-run! cfg execute-delete-object params)) ::jobs/decoder (sm/decoder schema:delete-object-params sm/json-transformer) ::jobs/validator (sm/validator schema:delete-object-params)}) diff --git a/backend/src/app/tasks/file_gc.clj b/backend/src/app/tasks/file_gc.clj index c8b2e41870..1cfc71b9a3 100644 --- a/backend/src/app/tasks/file_gc.clj +++ b/backend/src/app/tasks/file_gc.clj @@ -26,6 +26,7 @@ [app.features.fdata :as feat.fdata] [app.features.file-snapshots :as fsnap] [app.jobs :as jobs] + [app.loggers.audit :as audit] [app.storage :as sto] [integrant.core :as ig])) @@ -72,10 +73,21 @@ (into #{} (map :id)))] (jobs/heartbeat cfg) - (doseq [id unused-media] + (doseq [media-id unused-media] (l/trc :obj "media-object" :file-id (str id) - :id (str id))) + :id (str media-id))) + + ;; A trimmed file is a destructive action that no RPC reports: emit + ;; it so the audit trail records it. The team id travels in props so + ;; a consumer never joins production tables. + (when (and (seq unused-media) cf/audit-persist?) + (let [team-id (first (audit/get-team-ids cfg {:file-ids [id]}))] + (audit/submit cfg {:name "file-gc" + :type "action" + :props (cond-> {:file-id id} + (some? team-id) (assoc :team-id team-id)) + :context {:triggered-by "cron-task"}}))) file)) diff --git a/backend/src/app/tasks/objects_gc.clj b/backend/src/app/tasks/objects_gc.clj index 8f4ff44a98..20cfe5b1a6 100644 --- a/backend/src/app/tasks/objects_gc.clj +++ b/backend/src/app/tasks/objects_gc.clj @@ -44,7 +44,7 @@ their transactions. Runs inside the caller transaction, next to the deletes: if the purge aborts, the event aborts with it." [cfg ids-key ids] - (when (contains? cf/flags :audit-log) + (when cf/audit-persist? (audit/submit cfg {:name "hard-delete-objects" :type "command" diff --git a/backend/src/app/tasks/telemetry.clj b/backend/src/app/tasks/telemetry.clj index 05cd056d92..73cb6a4c8b 100644 --- a/backend/src/app/tasks/telemetry.clj +++ b/backend/src/app/tasks/telemetry.clj @@ -221,29 +221,18 @@ (def ^:private sql:aged-telemetry "source LIKE 'telemetry:%' AND created_at < now() - interval '7 days'") -(def ^:private sql:gc-pendings - (str "DELETE FROM audit_log_pending WHERE audit_log_id IN" - " (SELECT id FROM audit_log WHERE " sql:aged-telemetry ")")) - (def ^:private sql:gc-events (str "DELETE FROM audit_log WHERE " sql:aged-telemetry)) (defn- gc-events "Delete telemetry-mode events older than `telemetry-retention-days` - so that the buffer stays bounded." + so that the buffer stays bounded. The telemetry shadows are outside + the consumer GC on purpose: this task owns them." [{:keys [::db/conn]}] - ;; One transaction for both deletes: a crash between them would - ;; leave pending-less rows for the consumer sweep to delete unsent. - ;; Pendings first: the queue holds a row per consumer for every - ;; record, and the FK would reject deleting a row that still has - ;; pendings. - (db/tx-run! conn - (fn [{:keys [::db/conn]}] - (db/exec-one! conn [sql:gc-pendings]) - (let [result (db/exec-one! conn [sql:gc-events])] - (when (pos? (:next.jdbc/update-count result)) - (l/warn :hint "purged stale telemetry events" - :count (:next.jdbc/update-count result))))))) + (let [result (db/exec-one! conn [sql:gc-events])] + (when (pos? (:next.jdbc/update-count result)) + (l/warn :hint "purged stale telemetry events" + :count (:next.jdbc/update-count result))))) (def ^:private sql:fetch-telemetry-events "SELECT id, name, type, source, tracked_at, profile_id, props, context @@ -287,17 +276,13 @@ false)))) (defn- delete-sent-events - "Delete rows by their ids after a successful send. One transaction - for both deletes: a crash between them would leave pending-less - rows for the consumer sweep to delete unsent." + "Delete rows by their ids after a successful send. The telemetry + shadows are outside the consumer GC, so this task is their only + deleter: it can remove a row a consumer has not read yet (the cursor + simply skips over the missing position)." [conn ids] - (db/tx-run! conn - (fn [tx] - (let [tx-conn (::db/conn tx) - arr (db/create-array tx-conn "uuid" ids)] - ;; Pendings first: same FK reason as in `gc-events`. - (db/exec-one! tx-conn ["DELETE FROM audit_log_pending WHERE audit_log_id = ANY(?)" arr]) - (db/exec-one! tx-conn ["DELETE FROM audit_log WHERE id = ANY(?)" arr]))))) + (let [arr (db/create-array conn "uuid" ids)] + (db/exec-one! conn ["DELETE FROM audit_log WHERE id = ANY(?)" arr]))) (defn- collect-and-send-audit-events "Collect anonymous telemetry-mode audit events and ship them to the @@ -350,7 +335,7 @@ [cfg params] (let [send? (get params :send? true) enabled? (or (get params :enabled? false) - (contains? cf/flags :telemetry)) + cf/telemetry-enabled?) ;; Deferred so the query runs only when a report is ;; actually going to be sent. subs (delay (get-subscriptions cfg))] @@ -375,8 +360,8 @@ :cause cause))) ;; Ship any anonymous audit-log events accumulated in - ;; telemetry mode (only when audit-log feature is off). - (when-not (contains? cf/flags :audit-log) + ;; telemetry mode (only when the buffer has no consumer). + (when-not cf/audit-persist? (try (db/run! cfg collect-and-send-audit-events) (catch Exception cause diff --git a/backend/src/app/worker/cron.clj b/backend/src/app/worker/cron.clj index 0105d9d57e..6135701c21 100644 --- a/backend/src/app/worker/cron.clj +++ b/backend/src/app/worker/cron.clj @@ -91,13 +91,12 @@ {::db/return-keys false}) ;; The no-overlap check lives in the submit itself - ;; (`::jobs/skip-if-active`): the check and the - ;; insert share this transaction, so a job could - ;; still transition from running to completed - ;; between the check and the insert. This is - ;; acceptable: the next tick will submit if - ;; needed. The FOR UPDATE SKIP LOCKED on the - ;; scheduled_task row prevents race conditions + ;; (`::jobs/skip-if-active`): the partial unique + ;; index decides on the insert, so there is no + ;; check racing it. The only residual race is a + ;; live job going terminal right before the insert, + ;; which the next tick covers. The FOR UPDATE SKIP + ;; LOCKED on the scheduled_task row prevents races ;; between nodes. (if-let [job-id (submit-cron-job cfg entry)] (let [elapsed (ct/format-duration (tpoint))] diff --git a/backend/src/app/worker/dispatcher.clj b/backend/src/app/worker/dispatcher.clj index b084966714..5e51c5c941 100644 --- a/backend/src/app/worker/dispatcher.clj +++ b/backend/src/app/worker/dispatcher.clj @@ -59,23 +59,61 @@ "UPDATE job SET status = 'scheduled', modified_at = ? WHERE id = ANY(?)") -;; Both sweeps are bulk UPDATEs that run before every dispatch batch, so -;; they are the two queries where a missing tenant filter does the most -;; damage: without it any instance with a worker requeues another -;; instance's scheduled rows and, worse, aborts its running jobs once -;; their lease expires. -(def ^:private sql:reschedule-lost +;; The sweeps are bulk UPDATEs that run before every dispatch batch, so +;; they are the queries where a missing tenant filter does the most +;; damage: without it any instance with a worker aborts another +;; instance's jobs. +(def ^:private lost-batch-size + "Rows per lost-sweep read. The dispatcher loops without pausing while it + finds work, so an unbounded read would turn a big `scheduled` backlog + into a repeated scan. Aborting a row takes it out of the candidate set, + so the next pass continues where this one stopped." + 1000) + +(def ^:private sql:lost-past-cap + "The `scheduled` rows of one tenant that have waited past the cap, oldest + first. They are lost behind a backlog or in a queue with no consumer + whatever the queue holds, so the sweep aborts them without an `LLEN`. + `modified_at` is the hand-off time (the push), so it measures time in the + queue, not the due time." + "SELECT id, queue, modified_at + FROM job + WHERE status = 'scheduled' + AND tenant = ? + AND modified_at < ?::timestamptz + ORDER BY modified_at ASC + LIMIT ?") + +(def ^:private sql:lost-in-grace + "The `scheduled` rows of one tenant inside the grace window, oldest first. + These are the only candidates that need the per-queue `LLEN`: they are + lost when nothing is waiting for them." + "SELECT id, queue, modified_at + FROM job + WHERE status = 'scheduled' + AND tenant = ? + AND modified_at >= ?::timestamptz + AND modified_at < ?::timestamptz + ORDER BY modified_at ASC + LIMIT ?") + +(def ^:private sql:abort-lost + "Abort the lost rows by id. The status, the tenant and the hand-off time + are repeated in the outer WHERE so a claim, a heartbeat or a competing + dispatcher that lands between the candidate read and this UPDATE affects + no row." "UPDATE job - SET status='new', scheduled_at=?::timestamptz - FROM (SELECT t.id - FROM job AS t - WHERE status = 'scheduled' - AND t.tenant = ? - AND t.scheduled_at < ?::timestamptz - '5 min'::interval) AS subquery - WHERE job.id=subquery.id -RETURNING job.id, job.queue") + SET status='aborted', modified_at=?::timestamptz, error=?::jsonb + WHERE id = ANY(?) + AND status = 'scheduled' + AND tenant = ? + AND modified_at < ?::timestamptz +RETURNING job.id, job.queue, job.name, job.created_at") (def ^:private sql:mark-orphan + "Abort the running rows past the lease. The predicate is repeated in the + outer WHERE so a heartbeat that renews the row between the subquery and + the update affects no row." "UPDATE job SET status='aborted', modified_at=?::timestamptz, error=?::jsonb FROM (SELECT t.id @@ -84,18 +122,38 @@ RETURNING job.id, job.queue") AND t.tenant = ? AND t.modified_at < ?::timestamptz) AS subquery WHERE job.id=subquery.id + AND job.status = 'running' + AND job.modified_at < ?::timestamptz RETURNING job.id, job.queue, job.name, job.created_at") +(def ^:private lost-grace + "How long a `scheduled` row may wait before its queue is inspected. It + also covers the pop-to-claim gap of a message popped right after the + hand-off. A message that waited longer in the queue and is popped while + the sweep runs is not covered: the window is the pop-to-claim round trip + and the abort can win it (see `abort-lost-jobs`)." + (ct/duration "5m")) + +(def ^:private lost-cap + "Beyond this age a `scheduled` row is aborted even if its queue still has + messages: the job is lost behind a backlog, or its queue has no consumer. + A daily cron lost in a backed-up queue can miss one run, because its live + row blocks the next submit until the cap." + (ct/duration "24h")) + ;; the error travels as text and PostgreSQL casts it: a PGobject parameter ;; on a `::jsonb` placeholder is not what the driver expects here. Same -;; for the orphan `end` event payload below. +;; for the `end` event payload below. (def ^:private orphan-error-json (json/encode jobs/orphan-error)) +(def ^:private lost-error-json + (json/encode jobs/lost-error)) + (def ^:private aborted-outcome-json (json/encode {:outcome "aborted"})) -(def ^:private sql:insert-orphan-events +(def ^:private sql:insert-aborted-events "INSERT INTO job_event (job_id, kind, payload) SELECT unnest(?::uuid[]), 'end', ?::jsonb") @@ -103,42 +161,92 @@ RETURNING job.id, job.queue, job.name, job.created_at") [{:keys [id scheduled-at]}] (json/encode [(str id) (ct/format-inst scheduled-at)])) -(defn- reschedule-lost-jobs - [{:keys [::db/conn ::timestamp ::wrk/tenant] :as cfg}] - (let [rows (db/exec! conn [sql:reschedule-lost timestamp tenant timestamp] - {:return-keys true})] - (db/after-commit - (fn [] - (doseq [{:keys [id queue]} rows] - (metrics/record-rescheduled cfg queue) - (l/wrn :hint "reschedule" - :id (str id) - :tenant tenant - :queue queue)))))) +(defn- record-aborted + "The durable side of one abort sweep: the `end` events in the same + transaction (one bulk insert, never one per row) and the terminal metric + after the commit. No msgbus notification: an aborted job alerts through + the error log and `penpot_jobs_total_ms{outcome=\"aborted\"}`." + [{:keys [::db/conn ::timestamp] :as cfg} rows] + (when (seq rows) + (db/exec-one! conn [sql:insert-aborted-events + (db/create-array conn "uuid" (map :id rows)) + aborted-outcome-json])) + (db/after-commit + (fn [] + (doseq [{:keys [name created-at]} rows] + (metrics/record-total cfg name :aborted + (- (inst-ms timestamp) (inst-ms created-at))))))) + +(defn- get-waiting + "Messages waiting in each of the given queues, one LLEN per queue." + [{:keys [::rds/conn ::wrk/tenant]} queues] + (into {} + (map (fn [queue] + [queue (rds/llen conn (wrk/queue-key tenant queue))])) + queues)) + +(defn- abort-lost-jobs + "Abort the rows that were handed to a queue and never claimed. A row is + lost when it has been `scheduled` past the cap (behind a backlog, or in a + queue without a consumer), or when its queue has nothing waiting and it + has been `scheduled` longer than the grace (a worker that popped it died + before claiming). Lost rows are never re-dispatched: nothing returns to + `new`, which keeps the dedupe index out of this path. + + The two branches read a bounded page of candidates, oldest first: the + dispatcher loops without pausing while it finds work, so an unbounded + read would scan a big `scheduled` backlog on every iteration. Aborting + removes a row from the next read. Only the grace rows pay the `LLEN`; a + row past the cap is lost whatever its queue holds, so a younger lost row + behind a large legitimate backlog may wait for the cap instead of the + grace. + + The grace is measured from the push, so a worker that pops a message + older than the grace and stalls before claiming it can still lose that + job: the sweep cannot tell a slow worker from a dead one." + [cfg] + (let [{:keys [::db/conn ::timestamp ::wrk/tenant]} cfg + grace (ct/minus timestamp lost-grace) + cap (ct/minus timestamp lost-cap) + capped (db/exec! conn [sql:lost-past-cap tenant cap lost-batch-size]) + in-grace (db/exec! conn [sql:lost-in-grace tenant cap grace lost-batch-size]) + waiting (get-waiting cfg (into #{} (map :queue) in-grace)) + lost (into (vec capped) + (filter (fn [{:keys [queue]}] + (zero? (get waiting queue)))) + in-grace)] + (when (seq lost) + (let [aborted (db/exec! conn [sql:abort-lost + timestamp + lost-error-json + (db/create-array conn "uuid" (map :id lost)) + tenant + grace] + {:return-keys true})] + (record-aborted cfg aborted) + (db/after-commit + (fn [] + (doseq [{:keys [id queue name]} aborted] + (l/wrn :hint "marked job as aborted (never claimed after being scheduled)" + :name name + :id (str id) + :tenant tenant + :queue queue)))))))) (defn- mark-orphan-jobs + "Abort the running jobs whose process stopped reporting: their lease + expired, so nothing alive can finish them." [{:keys [::db/conn ::timestamp ::lease ::wrk/tenant] :as cfg}] (let [cutoff (ct/minus timestamp (or lease (cf/get-jobs-lease))) rows (db/exec! conn [sql:mark-orphan timestamp orphan-error-json - tenant cutoff] + tenant cutoff cutoff] {:return-keys true})] - ;; The sweep is the only status change outside `app.jobs`, so it owns - ;; its `end` events too: one bulk insert in the same transaction (never - ;; one insert per orphan) keeps the history consistent with the rows. - ;; No msgbus notification: orphans alert through the error log and the - ;; orphaned counter, like before. - (when (seq rows) - (db/exec-one! conn [sql:insert-orphan-events - (db/create-array conn "uuid" (map :id rows)) - aborted-outcome-json])) + (record-aborted cfg rows) (db/after-commit (fn [] - (doseq [{:keys [id queue name created-at]} rows] - (metrics/record-orphan cfg queue) - (when (ct/inst? created-at) - (metrics/record-total cfg name queue :aborted - (- (inst-ms timestamp) (inst-ms created-at)))) + (doseq [{:keys [id queue name]} rows] (l/err :hint "marked job as aborted (orphan lease expired)" + :name name :id (str id) :tenant tenant :queue queue)))))) @@ -159,8 +267,9 @@ RETURNING job.id, job.queue, job.name, job.created_at") (defn- push-jobs [{:keys [::rds/conn ::wrk/tenant] :as cfg} [queue jobs]] ;; Mark first, push last: a crash between mark and push leaves a - ;; `scheduled` row that reschedule-lost-jobs re-queues, while the - ;; reverse order leaves a duplicate payload nothing deduplicates. + ;; `scheduled` row that the lost sweep aborts after its window, while the + ;; reverse order leaves a payload for a row the dispatcher will push + ;; again (two deliveries of the same job). (mark-as-scheduled cfg jobs) (let [items (mapv encode-payload jobs) key (wrk/queue-key tenant queue)] @@ -168,7 +277,7 @@ RETURNING job.id, job.queue, job.name, job.created_at") (rds/rpush conn key items) (metrics/record-dispatcher-size cfg queue (count jobs)) (doseq [{:keys [id name queue]} jobs] - (metrics/record-dispatched cfg name queue 1) + (metrics/record-dispatched cfg name 1) (l/trc :hist "schedule" :tenant tenant :queue queue @@ -178,9 +287,10 @@ RETURNING job.id, job.queue, job.name, job.created_at") [cfg] (let [cfg (assoc cfg ::timestamp (ct/now)) tpoint (ct/tpoint)] - ;; Reschedule lost in transit jobs (can happen when - ;; redis server is restarted just after job is pushed) - (reschedule-lost-jobs cfg) + ;; The jobs that were handed to a queue and never claimed are + ;; terminal: a lost message is not re-dispatched, so no job ever goes + ;; back to `new` and the dedupe index stays out of this path. + (abort-lost-jobs cfg) ;; Mark as aborted all jobs that are still marked as running but ;; their last modification (heartbeat or progress) is older than @@ -207,9 +317,10 @@ RETURNING job.id, job.queue, job.name, job.created_at") (px/sleep (or (::timeout cfg) (ct/duration "10s")))) (defn run-batch - "Execute a single dispatch batch: reschedule lost jobs, mark orphans - (lease-based) and claim pending jobs into their Redis queues. Exposed - as a function for testability; the dispatcher thread loops on it." + "Execute a single dispatch batch: abort the lost scheduled jobs, abort + the orphans (lease-based) and claim pending jobs into their Redis + queues. Exposed as a function for testability; the dispatcher thread + loops on it." [cfg] (let [tpoint (ct/tpoint) tenant (::wrk/tenant cfg)] diff --git a/backend/src/app/worker/runner.clj b/backend/src/app/worker/runner.clj index 2202722c0d..efe00f9d00 100644 --- a/backend/src/app/worker/runner.clj +++ b/backend/src/app/worker/runner.clj @@ -37,9 +37,8 @@ terminal job produces 0 affected rows and is skipped without touching its state (first-terminal-wins companion). Also predicates the payload - scheduled_at: a row rescheduled after the payload was pushed - (dispatcher re-pushes on reschedule) is never claimed with the stale - payload." + scheduled_at: a duplicate or stale payload is never claimed with a + timestamp that no longer matches the row." [cfg job-id scheduled-at] (jobs/claim cfg job-id scheduled-at)) @@ -111,10 +110,9 @@ (do (let [job-def (jobs/get-job-def defs (:name job)) - _ (jobs-metrics/record-queue-wait + _ (jobs-metrics/record-claim-wait cfg (::jobs/name job-def) - queue (- (inst-ms (ct/now)) (inst-ms (:scheduled-at job)))) params (try (->> (:params job) @@ -140,7 +138,6 @@ (jobs-metrics/record-execution cfg (:name job) - queue (inst-ms (tpoint))) (jobs-metrics/record-legacy-execution cfg @@ -236,7 +233,7 @@ (not= (inst-ms scheduled-at) (inst-ms (:scheduled-at job))) - (l/wrn :hint "skipping job, rescheduled" + (l/wrn :hint "skipping job, stale payload" :job-id (str job-id) :runner-id id :tenant tenant diff --git a/backend/test/backend_tests/auth_oidc_test.clj b/backend/test/backend_tests/auth_oidc_test.clj index 8b624a6fbb..3e0d924afa 100644 --- a/backend/test/backend_tests/auth_oidc_test.clj +++ b/backend/test/backend_tests/auth_oidc_test.clj @@ -821,7 +821,8 @@ (binding [cf/config {:public-uri "https://penpot.example.com"}] (with-redefs [app.nitrate/call (constantly {:active true}) app.auth.oidc/prepare-organization-sso-provider (constantly {:type "oidc"}) - app.auth.oidc/get-info (constantly {})] + app.auth.oidc/get-info (constantly {}) + app.loggers.audit/submit (constantly nil)] (let [result (#'oidc/callback-handler cfg request)] (t/is (= "https://penpot.example.com/" (redirect-location result)))))))) @@ -838,7 +839,8 @@ app.auth.oidc/prepare-organization-sso-provider (constantly {:type "oidc"}) app.auth.oidc/get-info (fn [& _] (ex/raise :type :internal - :code :unable-to-retrieve-user-info))] + :code :unable-to-retrieve-user-info)) + app.loggers.audit/submit (constantly nil)] (let [result (#'oidc/callback-handler cfg request) loc (redirect-location result)] (t/is (not (.contains loc "evil.example"))) diff --git a/backend/test/backend_tests/config_test.clj b/backend/test/backend_tests/config_test.clj index 5e0e2418b8..7ac854eaee 100644 --- a/backend/test/backend_tests/config_test.clj +++ b/backend/test/backend_tests/config_test.clj @@ -153,3 +153,19 @@ (t/deftest read-config-takes-the-default-back-end (t/is (= "db" (c/get (read-config {} :default cf/default) :file-data-backend)))) + +(t/deftest audit-persistence-follows-consumer-flags + (t/testing "a consumer flag enables persistence" + (t/is (cf/persist-audit-events? #{:nexus})) + (t/is (cf/persist-audit-events? #{:admin-console}))) + (t/testing "other sinks alone do not enable persistence" + (doseq [flags [#{} #{:webhooks} #{:telemetry} #{:error-reporting} + #{:audit-log-logger}]] + (t/is (not (cf/persist-audit-events? flags)) (str flags)))) + (t/testing "the boot value is derived from the loaded flags" + (t/is (= (cf/persist-audit-events? cf/flags) cf/audit-persist?))) + (t/testing "a stale enable-audit-log in the environment is dropped" + (let [flags (#'cf/parse-flags {:flags #{"enable-audit-log" "enable-nexus"}})] + (t/is (not (contains? flags :audit-log))) + (t/is (contains? flags :nexus)) + (t/is (cf/persist-audit-events? flags))))) diff --git a/backend/test/backend_tests/db_test.clj b/backend/test/backend_tests/db_test.clj index 71de5b7b24..f00812a732 100644 --- a/backend/test/backend_tests/db_test.clj +++ b/backend/test/backend_tests/db_test.clj @@ -195,3 +195,93 @@ ["SELECT current_setting('enable_partitionwise_join') AS partitionwise_join"])] (t/is (= "on" (:partitionwise-join row))))) + +(t/deftest session-locks-are-held-per-connection + (let [classid 730000 + objid 1] + (with-open [a (db/open th/*pool*) + b (db/open th/*pool*)] + (t/testing "the first connection takes it, the second cannot" + (t/is (true? (db/session-try-lock a classid objid))) + (t/is (false? (db/session-try-lock b classid objid)))) + (t/testing "releasing it lets the other take it" + (t/is (true? (db/session-unlock a classid objid))) + (t/is (true? (db/session-try-lock b classid objid))) + (t/is (true? (db/session-unlock b classid objid))))))) + +(t/deftest session-locks-keep-the-two-key-space-apart + ;; PostgreSQL documents the split: a bigint key is shown with its + ;; high-order half in classid, its low-order half in objid and + ;; objsubid 1, while a two-int key keeps both numbers and objsubid 2. + ;; The two forms are therefore different locks even with the same + ;; numbers, which is what the consumers rely on to never block a file + ;; update. The test proves each step instead of assuming it. + (let [classid 730000 + objid 2 + bigint-key (+ (* (long classid) 4294967296) objid)] + (with-open [a (db/open th/*pool*) + b (db/open th/*pool*) + c (db/open th/*pool*)] + (t/testing "the one-key form really holds that identity" + (t/is (true? (:lock (db/exec-one! a ["select pg_try_advisory_lock(?::bigint) as lock" + bigint-key])))) + (t/is (false? (:lock (db/exec-one! b ["select pg_try_advisory_lock(?::bigint) as lock" + bigint-key]))))) + + (t/testing "PostgreSQL displays that bigint as the (classid, objid) pair" + (let [{:keys [classid objid objsubid]} + (db/exec-one! a ["select classid, objid, objsubid from pg_locks + where pid = pg_backend_pid() + and locktype = 'advisory' + and objsubid = 1 + and granted"])] + (t/is (= [730000 2 1] [classid objid objsubid])) + ;; and the displayed halves reassemble to the same bigint + (t/is (= bigint-key (+ (* (long classid) 4294967296) objid))))) + + (t/testing "the same numbers in the two-key form are another lock" + (t/is (true? (db/session-try-lock c classid objid))) + (t/is (true? (db/session-unlock c classid objid)))) + + (t/is (true? (:lock (db/exec-one! a ["select pg_advisory_unlock(?::bigint) as lock" + bigint-key]))))))) + +(t/deftest session-locks-reject-a-key-that-is-not-int4 + (with-open [a (db/open th/*pool*)] + (t/is (thrown? IllegalArgumentException + (db/session-try-lock a uuid/zero 1))) + (t/is (thrown? IllegalArgumentException + (db/session-unlock a 730000 uuid/zero))) + (t/is (thrown? IllegalArgumentException + (db/session-try-lock a 730000 (inc Integer/MAX_VALUE)))))) + +(t/deftest session-unlock-all-releases-every-lock-of-the-connection + (let [classid 730000] + (with-open [a (db/open th/*pool*) + b (db/open th/*pool*)] + (t/is (true? (db/session-try-lock a classid 1))) + (t/is (true? (db/session-try-lock a classid 2))) + (t/is (false? (db/session-try-lock b classid 1))) + (db/session-unlock-all a) + (t/is (true? (db/session-try-lock b classid 1))) + (t/is (true? (db/session-try-lock b classid 2))) + (db/session-unlock-all b)))) + +(t/deftest session-unlock-answers-false-for-a-lock-not-held + (with-open [a (db/open th/*pool*)] + (t/is (false? (db/session-unlock a 730000 3))))) + +(t/deftest own-transaction-drops-the-connection-and-the-read-only-flag + ;; The contract has two halves: the write must not join the caller's + ;; connection, and it must be writable even when the caller is + ;; read-only (`::db/read-only` dropped). + (let [caller-conn (Object.) + cfg {::db/pool th/*pool* + ::db/conn caller-conn + ::db/read-only true}] + (t/testing "the connection and the read-only flag are gone" + (t/is (= {::db/pool th/*pool*} (db/own-transaction cfg)))) + (t/testing "so the write opens its own transaction on another connection" + (db/tx-run! (db/own-transaction cfg) + (fn [{:keys [::db/conn]}] + (t/is (not (identical? conn caller-conn)))))))) diff --git a/backend/test/backend_tests/delete_object_test.clj b/backend/test/backend_tests/delete_object_test.clj index 878bea305d..5f7b2fe7a3 100644 --- a/backend/test/backend_tests/delete_object_test.clj +++ b/backend/test/backend_tests/delete_object_test.clj @@ -32,7 +32,7 @@ ;; carries every affected id so a projection can follow without reading ;; production tables. (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] - (with-redefs [cf/flags (conj cf/flags :audit-log)] + (th/with-audit-config (conj cf/flags :nexus) (let [profile (th/create-profile* 1 {:is-active true}) team (th/create-team* 91 {:profile-id (:id profile)}) project (th/create-project* 91 {:profile-id (:id profile) @@ -56,9 +56,13 @@ (t/is (= "team" (name (:object props)))) (t/is (= (:id team) (:id props))) (t/is (= deleted-at (:deleted-at props))) - (t/is (not (contains? props :team-ids))) - (t/is (not (contains? props :project-ids))) - (t/is (not (contains? props :file-ids))) + ;; The shape is the contract: only the team ids go flat, the + ;; rest stays under `:cascade`. + (t/is (= #{:object :id :deleted-at :team-ids :cascade} + (set (keys props)))) + ;; Flat team ids for team-keyed consumers, next to the + ;; per-entity cascade breakdown. + (t/is (= #{(:id team)} (set (:team-ids props)))) (t/is (= #{(:id team)} (set (get-in props [:cascade :team-ids])))) (t/is (= #{(:default-project-id team) (:id project)} (set (get-in props [:cascade :project-ids])))) @@ -66,7 +70,7 @@ (t/deftest delete-object-file-cascade-emits-event (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] - (with-redefs [cf/flags (conj cf/flags :audit-log)] + (th/with-audit-config (conj cf/flags :nexus) (let [profile (th/create-profile* 1 {:is-active true}) team (th/create-team* 92 {:profile-id (:id profile)}) project (th/create-project* 92 {:profile-id (:id profile) @@ -85,12 +89,13 @@ (t/is (= (:id file) (:id props))) (t/is (= [] (get-in props [:cascade :team-ids]))) (t/is (= [] (get-in props [:cascade :project-ids]))) - (t/is (= [(:id file)] (get-in props [:cascade :file-ids]))))))))) + (t/is (= [(:id file)] (get-in props [:cascade :file-ids]))) + (t/is (= #{(:id team)} (set (:team-ids props)))))))))) (t/deftest delete-object-missing-row-emits-nothing ;; A cascade over an unknown id marks nothing, so there is nothing to report. (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] - (with-redefs [cf/flags (conj cf/flags :audit-log)] + (th/with-audit-config (conj cf/flags :nexus) (th/run-task! :delete-object {:object :team :id (uuid/next) :deleted-at (ct/now)}) @@ -99,7 +104,7 @@ (t/deftest delete-object-snapshot-emits-nothing ;; Snapshots have no projection table; their cascade stays silent. (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] - (with-redefs [cf/flags (conj cf/flags :audit-log)] + (th/with-audit-config (conj cf/flags :nexus) (th/run-task! :delete-object {:object :snapshot :id (uuid/next) :file-id (uuid/next) diff --git a/backend/test/backend_tests/helpers.clj b/backend/test/backend_tests/helpers.clj index 1c7a76973d..165cd593c5 100644 --- a/backend/test/backend_tests/helpers.clj +++ b/backend/test/backend_tests/helpers.clj @@ -63,6 +63,23 @@ (def ^:dynamic *system* nil) (def ^:dynamic *pool* nil) +(defmacro with-audit-config + "Rebind the flags and the values derived from them at boot. An optional + binding vector rebinds other vars in the same scope." + [flags & body] + (let [extra (if (and (seq body) (vector? (first body))) + (first body) + []) + body (if (seq extra) (rest body) body)] + `(let [flags# ~flags] + (with-redefs [cf/flags flags# + cf/audit-persist? (cf/persist-audit-events? flags#) + cf/webhooks-enabled? (contains? flags# :webhooks) + cf/telemetry-enabled? (contains? flags# :telemetry) + cf/error-reporting-enabled? (contains? flags# :error-reporting) + ~@extra] + ~@body)))) + ;; Fallback values used when no PENPOT_TEST_* env vars are set (e.g. CI sets ;; them explicitly, see .github/workflows/tests-backend.yml). Inside the ;; devenv each wsN container receives per-instance values via @@ -94,8 +111,12 @@ ([next] (init-config nil next)) ([extra-flags next] - (let [flags (into default-flags extra-flags)] - (with-redefs [app.config/flags (flags/parse flags/default flags) + (let [flags (flags/parse flags/default (into default-flags extra-flags))] + (with-redefs [app.config/flags flags + app.config/audit-persist? (cf/persist-audit-events? flags) + app.config/webhooks-enabled? (contains? flags :webhooks) + app.config/telemetry-enabled? (contains? flags :telemetry) + app.config/error-reporting-enabled? (contains? flags :error-reporting) app.config/config config app.loggers.audit/submit (constantly nil) app.auth/derive-password identity @@ -153,7 +174,9 @@ :app.loggers.database/reporter :app.loggers.metrics/reporter :app.worker/cron - :app.worker/dispatcher) + :app.worker/dispatcher + :app.loggers.audit.consumers.nexus/consumer + :app.loggers.audit.consumers.admin-console/consumer) (remove-job-runners)) _ (ig/load-namespaces system) system (-> (ig/expand system) (ig/init))] @@ -658,6 +681,18 @@ (assoc ::jobs/context (jobs/make-context row)) (assoc ::jobs/job-id (:id row))))))))) +(defn wait-for + "Poll `pred` until it holds, up to `timeout` ms (5 s by default). Answers + true when it held, false on timeout." + ([pred] (wait-for pred 5000)) + ([pred timeout] + (let [deadline (+ (System/currentTimeMillis) timeout)] + (loop [] + (cond + (pred) true + (< (System/currentTimeMillis) deadline) (do (Thread/sleep 10) (recur)) + :else false))))) + (defn metrics-stub "A metrics instance with no collectors, where every `run!` is a no-op." [] diff --git a/backend/test/backend_tests/jobs_metrics_test.clj b/backend/test/backend_tests/jobs_metrics_test.clj index 7b2523c320..69a196af68 100644 --- a/backend/test/backend_tests/jobs_metrics_test.clj +++ b/backend/test/backend_tests/jobs_metrics_test.clj @@ -6,9 +6,6 @@ (ns backend-tests.jobs-metrics-test (:require - [app.common.time :as ct] - [app.common.uuid :as uuid] - [app.config :as cf] [app.db :as db] [app.jobs :as jobs] [app.jobs.metrics :as jobs-metrics] @@ -24,29 +21,22 @@ io.prometheus.client.Collector$MetricFamilySamples$Sample io.prometheus.client.CollectorRegistry io.prometheus.client.Counter - io.prometheus.client.Counter$Child - io.prometheus.client.Gauge - io.prometheus.client.Gauge$Child)) + io.prometheus.client.Counter$Child)) (t/use-fixtures :once th/state-init) (t/use-fixtures :each th/database-reset) (def ^:private metric-ids - [:jobs-submitted - :jobs-skipped + [:jobs-submit :jobs-dispatched :jobs-retries - :jobs-orphaned - :jobs-rescheduled - :jobs-queue-wait-timing + :jobs-claim-wait-timing :jobs-execution-timing :jobs-total-timing - :jobs-dispatcher-timing + :jobs-dispatcher-batch-timing :jobs-dispatcher-batch-size - :jobs-backlog - :jobs-oldest-pending-age :jobs-gc-rows - :jobs-gc-timing + :jobs-gc-pass-timing :jobs-cron-total :jobs-request-timing :jobs-events-total]) @@ -90,29 +80,34 @@ ;; assertions use, and (= 0 0.0) is false in Clojure (reduce + 0.0)))) -(defn- gauge-value [metrics id labels] - (let [collector (mtx/get-collector metrics id) - instance (::mdef/instance collector) - child (.labels ^Gauge instance (into-array String labels))] - (.get ^Gauge$Child child))) +(defn- echo-defs + "The minimal job-def the tests submit and drive through the lifecycle." + [] + {:echo {::jobs/name :echo + ::jobs/schema [:map] + ::jobs/handler (fn [_context params] params) + ::jobs/decoder identity + ::jobs/validator (constantly true)}}) (t/deftest default-metrics-cover-the-jobs-observability-contract (doseq [id metric-ids] (let [definition (get main/default-metrics id)] (t/is (some? definition) id) (t/is (re-find #"^penpot_jobs_" (::mdef/name definition)) id))) - (t/is (= ["name" "queue" "outcome"] + (t/is (= ["name" "outcome"] + (::mdef/labels (:jobs-submit main/default-metrics)))) + (t/is (= ["name" "outcome"] (::mdef/labels (:jobs-total-timing main/default-metrics)))) - (t/is (= ["status"] - (::mdef/labels (:jobs-backlog main/default-metrics))))) + (t/is (= ["name"] + (::mdef/labels (:jobs-claim-wait-timing main/default-metrics)))) + (t/testing "one submit family and one terminal counter, with no twin ids" + (t/is (not (contains? main/default-metrics :jobs-submitted))) + (t/is (not (contains? main/default-metrics :jobs-skipped))) + (t/is (not (contains? main/default-metrics :jobs-orphaned))))) (t/deftest submit-metric-is-not-recorded-when-the-outer-transaction-rolls-back (let [metrics (make-metrics) - defs {:echo {::jobs/name :echo - ::jobs/schema [:map] - ::jobs/handler (fn [_context params] params) - ::jobs/decoder identity - ::jobs/validator (constantly true)}} + defs (echo-defs) cfg {::jobs/defs defs ::db/pool th/*pool* ::mtx/metrics metrics}] @@ -123,68 +118,114 @@ {::jobs/name :echo ::jobs/params {}}) (throw (ex-info "rollback" {})))))) - (t/is (= 0.0 (counter-value metrics :jobs-submitted ["echo" "default"]))))) + (t/is (= 0.0 (counter-value metrics :jobs-submit ["echo" "created"]))))) (t/deftest submit-and-terminal-writers-record-production-events (let [metrics (make-metrics) - defs {:echo {::jobs/name :echo - ::jobs/schema [:map] - ::jobs/handler (fn [_context params] params) - ::jobs/decoder identity - ::jobs/validator (constantly true)}} + defs (echo-defs) cfg {::jobs/defs defs ::db/pool th/*pool* ::mtx/metrics metrics} job-id (jobs/submit cfg {::jobs/name :echo ::jobs/params {}})] (jobs/claim cfg job-id (:scheduled-at (jobs/get-job cfg job-id))) (jobs/complete cfg :job-id job-id :result {:ok true}) - (t/is (= 1.0 (counter-value metrics :jobs-submitted ["echo" "default"]))) - (t/is (= 1.0 (histogram-count metrics :jobs-total-timing ["echo" "default" "completed"]))))) + (t/is (= 1.0 (counter-value metrics :jobs-submit ["echo" "created"]))) + (t/is (= 1.0 (histogram-count metrics :jobs-total-timing ["echo" "completed"]))))) (t/deftest skip-if-active-records-skipped-metric (let [metrics (make-metrics) - defs {:echo {::jobs/name :echo - ::jobs/schema [:map] - ::jobs/handler (fn [_context params] params) - ::jobs/decoder identity - ::jobs/validator (constantly true)}} + defs (echo-defs) cfg {::jobs/defs defs ::db/pool th/*pool* ::mtx/metrics metrics} - _ (jobs/submit cfg {::jobs/name :echo - ::jobs/params {} - ::jobs/label "skip-metric"}) - skipped (jobs/submit cfg {::jobs/name :echo - ::jobs/params {} - ::jobs/label "skip-metric" + _ (jobs/submit cfg {::jobs/name :echo + ::jobs/params {} + ::jobs/label "skip-metric" + ::jobs/skip-if-active true}) + skipped (jobs/submit cfg {::jobs/name :echo + ::jobs/params {} + ::jobs/label "skip-metric" ::jobs/skip-if-active true})] (t/is (nil? skipped)) - (t/is (= 1.0 (counter-value metrics :jobs-skipped ["echo" "default"]))) - (t/is (= 1.0 (counter-value metrics :jobs-submitted ["echo" "default"])) - "the withheld submit records no submission"))) + (t/is (= 1.0 (counter-value metrics :jobs-submit ["echo" "skipped"]))) + (t/is (= 1.0 (counter-value metrics :jobs-submit ["echo" "created"])) + "the withheld submit records no creation"))) + +(t/deftest dedupe-skip-is-counted-apart-from-a-creation + (let [metrics (make-metrics) + cfg {::jobs/defs (echo-defs) + ::db/pool th/*pool* + ::mtx/metrics metrics} + options {::jobs/name :echo + ::jobs/params {} + ::jobs/label "dedupe-metric"} + created (jobs/submit cfg (assoc options ::jobs/dedupe true)) + skipped (jobs/submit cfg (assoc options ::jobs/dedupe true))] + (t/is (some? created)) + (t/is (nil? skipped)) + (t/is (= 1.0 (counter-value metrics :jobs-submit ["echo" "created"]))) + (t/is (= 1.0 (counter-value metrics :jobs-submit ["echo" "skipped"]))) + (t/is (= 0.0 (counter-value metrics :jobs-submit ["echo" "refreshed"]))))) + +(t/deftest debounce-refresh-is-counted-apart-from-a-creation + (let [metrics (make-metrics) + cfg {::jobs/defs (echo-defs) + ::db/pool th/*pool* + ::mtx/metrics metrics} + options {::jobs/name :echo + ::jobs/params {} + ::jobs/label "debounce-metric" + ::jobs/debounce true} + created (jobs/submit cfg options) + refreshed (jobs/submit cfg options)] + (t/is (some? created)) + (t/is (= created refreshed) + "a refresh answers with the id the row already had") + (t/is (= 1.0 (counter-value metrics :jobs-submit ["echo" "created"]))) + (t/is (= 1.0 (counter-value metrics :jobs-submit ["echo" "refreshed"]))) + (t/is (= 0.0 (counter-value metrics :jobs-submit ["echo" "skipped"]))))) + +(t/deftest the-terminal-histogram-count-is-the-counter-of-every-outcome + (let [metrics (make-metrics) + cfg {::jobs/defs (echo-defs) + ::db/pool th/*pool* + ::mtx/metrics metrics} + claim! (fn [id] + (jobs/claim cfg id (:scheduled-at (jobs/get-job cfg id)))) + completed (jobs/submit cfg {::jobs/name :echo ::jobs/params {}}) + failed (jobs/submit cfg {::jobs/name :echo ::jobs/params {}}) + cancelled (jobs/submit cfg {::jobs/name :echo ::jobs/params {}})] + (claim! completed) + (jobs/complete cfg :job-id completed :result {:ok true}) + (claim! failed) + (jobs/fail cfg failed {:type :internal :code :boom :hint "boom"}) + (claim! cancelled) + (jobs/cancel cfg cancelled) + + (t/testing "one observation per outcome, with no twin counter" + (t/is (= 1.0 (histogram-count metrics :jobs-total-timing ["echo" "completed"]))) + (t/is (= 1.0 (histogram-count metrics :jobs-total-timing ["echo" "failed"]))) + (t/is (= 1.0 (histogram-count metrics :jobs-total-timing ["echo" "cancelled"]))) + (t/is (= 0.0 (histogram-count metrics :jobs-total-timing ["echo" "aborted"])))))) (t/deftest lifecycle-helpers-use-labels-as-is (let [metrics (make-metrics) cfg (metrics-cfg metrics)] - (jobs-metrics/record-submitted cfg "delete-object" "default") - (jobs-metrics/record-skipped cfg "delete-object" "default") - (jobs-metrics/record-dispatched cfg "delete-object" "default" 2) - (jobs-metrics/record-total cfg "delete-object" "default" :completed 42) - (jobs-metrics/record-retry cfg "delete-object" "default" :backoff) - (jobs-metrics/record-orphan cfg "webhooks") - (jobs-metrics/record-rescheduled cfg "custom") + (jobs-metrics/record-created cfg "delete-object") + (jobs-metrics/record-skipped cfg "delete-object") + (jobs-metrics/record-dispatched cfg "delete-object" 2) + (jobs-metrics/record-total cfg "delete-object" :completed 42) + (jobs-metrics/record-retry cfg "delete-object" :backoff) (jobs-metrics/record-gc-rows cfg :expired :deleted 3) (jobs-metrics/record-cron cfg :submitted :none) (jobs-metrics/record-request cfg :replied 12) (jobs-metrics/record-legacy-execution cfg "delete-object" 42) - (t/is (= 1.0 (counter-value metrics :jobs-submitted ["delete-object" "default"]))) - (t/is (= 1.0 (counter-value metrics :jobs-skipped ["delete-object" "default"]))) - (t/is (= 2.0 (counter-value metrics :jobs-dispatched ["delete-object" "default"]))) - (t/is (= 1.0 (histogram-count metrics :jobs-total-timing ["delete-object" "default" "completed"]))) - (t/is (= 1.0 (counter-value metrics :jobs-retries ["delete-object" "default" "backoff"]))) - (t/is (= 1.0 (counter-value metrics :jobs-orphaned ["webhooks"]))) - (t/is (= 1.0 (counter-value metrics :jobs-rescheduled ["custom"]))) + (t/is (= 1.0 (counter-value metrics :jobs-submit ["delete-object" "created"]))) + (t/is (= 1.0 (counter-value metrics :jobs-submit ["delete-object" "skipped"]))) + (t/is (= 2.0 (counter-value metrics :jobs-dispatched ["delete-object"]))) + (t/is (= 1.0 (histogram-count metrics :jobs-total-timing ["delete-object" "completed"]))) + (t/is (= 1.0 (counter-value metrics :jobs-retries ["delete-object" "backoff"]))) (t/is (= 3.0 (counter-value metrics :jobs-gc-rows ["expired" "deleted"]))) (t/is (= 1.0 (counter-value metrics :jobs-cron-total ["submitted" "none"]))) (t/is (= 1.0 (histogram-count metrics :jobs-request-timing ["replied"]))) @@ -197,15 +238,15 @@ (t/is (= 0.0 (histogram-count metrics :tasks-timing ["other"])))))) (t/deftest labels-go-to-metrics-as-is - (t/testing "queues are lowercased, with other only when missing" + (t/testing "the queue of a batch is lowercased, other only when missing" (t/is (= "unexpected" (jobs-metrics/queue-label "unexpected"))) (t/is (= "default" (jobs-metrics/queue-label "default"))) (t/is (= "default" (jobs-metrics/queue-label :default))) (t/is (= "other" (jobs-metrics/queue-label nil)))) (let [metrics (make-metrics) cfg (metrics-cfg metrics)] - (jobs-metrics/record-total cfg "echo" "unexpected" :unexpected 42) - (t/is (= 1.0 (histogram-count metrics :jobs-total-timing ["echo" "unexpected" "unexpected"]))) + (jobs-metrics/record-total cfg "echo" :unexpected 42) + (t/is (= 1.0 (histogram-count metrics :jobs-total-timing ["echo" "unexpected"]))) (t/testing "a new value is recorded, never folded or dropped" (jobs-metrics/record-gc-rows cfg :unknown :deleted 1) (jobs-metrics/record-cron cfg :brand-new :whatever) @@ -214,57 +255,17 @@ (t/is (= 1.0 (counter-value metrics :jobs-cron-total ["brand-new" "whatever"]))) (t/is (= 1.0 (histogram-count metrics :jobs-request-timing ["brand-new"])))))) -(t/deftest sampler-is-wired-and-does-not-start-on-read-only - (t/is (contains? main/worker-config :app.jobs.metrics/sampler)) - (let [metrics (make-metrics) - cfg {::db/pool th/*pool* - ::mtx/metrics metrics}] - (with-redefs [db/read-only? (constantly true)] - (t/is (nil? (ig/init-key :app.jobs.metrics/sampler cfg)))) - (let [sampler (ig/init-key :app.jobs.metrics/sampler cfg)] - (try - (t/is (some? sampler)) - (finally - (ig/halt-key! :app.jobs.metrics/sampler sampler)))))) - (t/deftest unknown-names-go-to-metrics-as-is (let [metrics (make-metrics) cfg (metrics-cfg metrics)] - (jobs-metrics/record-submitted cfg "unknown-job" "default") - (t/is (= 1.0 (counter-value metrics :jobs-submitted ["unknown-job" "default"]))))) - -(t/deftest backlog-sampler-updates-status-and-age-gauges - (let [metrics (make-metrics) - cfg {::db/pool th/*pool* - ::mtx/metrics metrics} - now (ct/now)] - (th/db-insert! :job {:id (uuid/next) - :name "echo" - :tenant (cf/get :tenant) - :queue "default" - :params (db/json {}) - :priority 100 - :max-retries 3 - :retry-num 0 - :status "new" - :scheduled-at now - :created-at (ct/in-past {:minutes 2}) - :modified-at now}) - (jobs-metrics/sample-backlog cfg) - (t/is (= 1.0 (gauge-value metrics :jobs-backlog ["new"]))) - (t/is (= 0.0 (gauge-value metrics :jobs-backlog ["completed"]))) - (t/is (= 0.0 (gauge-value metrics :jobs-backlog ["aborted"]))) - (t/is (>= (gauge-value metrics :jobs-oldest-pending-age []) 120.0)))) + (jobs-metrics/record-created cfg "unknown-job") + (t/is (= 1.0 (counter-value metrics :jobs-submit ["unknown-job" "created"]))))) (t/deftest event-counter-has-no-labels-and-follows-the-events (let [metrics (make-metrics) cfg {::db/pool th/*pool* ::mtx/metrics metrics} - defs {:echo {::jobs/name :echo - ::jobs/schema [:map] - ::jobs/handler (fn [_context params] params) - ::jobs/decoder identity - ::jobs/validator (constantly true)}} + defs (echo-defs) cfg2 (assoc cfg ::jobs/defs defs) job-id (jobs/submit cfg2 {::jobs/name :echo ::jobs/params {}})] (jobs/claim cfg2 job-id (:scheduled-at (jobs/get-job cfg2 job-id))) @@ -278,11 +279,7 @@ (t/deftest event-counter-follows-a-lifecycle-write-that-outlives-the-caller (let [metrics (make-metrics) - defs {:echo {::jobs/name :echo - ::jobs/schema [:map] - ::jobs/handler (fn [_context params] params) - ::jobs/decoder identity - ::jobs/validator (constantly true)}} + defs (echo-defs) cfg {::jobs/defs defs ::db/pool th/*pool* ::mtx/metrics metrics} @@ -302,15 +299,11 @@ (t/is (= 2 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job_event"]))))) (t/testing "the terminal counter followed its own commit, not the caller one" - (t/is (= 1.0 (histogram-count metrics :jobs-total-timing ["echo" "default" "completed"])))))) + (t/is (= 1.0 (histogram-count metrics :jobs-total-timing ["echo" "completed"])))))) (t/deftest progress-counter-follows-the-report-that-outlives-the-caller (let [metrics (make-metrics) - defs {:echo {::jobs/name :echo - ::jobs/schema [:map] - ::jobs/handler (fn [_context params] params) - ::jobs/decoder identity - ::jobs/validator (constantly true)}} + defs (echo-defs) cfg {::jobs/defs defs ::db/pool th/*pool* ::mtx/metrics metrics} @@ -329,7 +322,7 @@ job-id])))) (t/is (= (inc before) (counter-value metrics :jobs-events-total []))))))) -(t/deftest the-binfile-queue-is-labelled-like-any-other +(t/deftest the-submit-carries-no-queue-and-the-batch-size-still-does (let [metrics (make-metrics) defs {:export-binfile {::jobs/name :export-binfile ::jobs/schema [:map] @@ -344,6 +337,15 @@ ::jobs/params {} ::jobs/queue :binfile}) - (t/testing "the queue of the job is its label, never other" - (t/is (= 1.0 (counter-value metrics :jobs-submitted ["export-binfile" "binfile"]))) - (t/is (= 0.0 (counter-value metrics :jobs-submitted ["export-binfile" "other"])))))) + (t/testing "the submit is keyed by the job name and what it did" + (t/is (= 1.0 (counter-value metrics :jobs-submit ["export-binfile" "created"]))) + (t/is (= ["name" "outcome"] + (::mdef/labels (:jobs-submit main/default-metrics))))) + + (t/testing "a batch mixes job names, so it keeps the queue" + (jobs-metrics/record-dispatcher-size cfg "binfile" 3) + (t/is (= 1.0 (histogram-count metrics + :jobs-dispatcher-batch-size + ["binfile"]))) + (t/is (= ["queue"] + (::mdef/labels (:jobs-dispatcher-batch-size main/default-metrics))))))) diff --git a/backend/test/backend_tests/jobs_test.clj b/backend/test/backend_tests/jobs_test.clj index 15c7e6d83b..556a47a139 100644 --- a/backend/test/backend_tests/jobs_test.clj +++ b/backend/test/backend_tests/jobs_test.clj @@ -32,6 +32,20 @@ [f] (try (f) nil (catch Throwable cause cause))) +(defn- race + "Run every thunk on a thread of its own and return what each returned, + in the order they were given. They all wait on the same gate, so they + contend for whatever they share instead of running one after another." + [& thunks] + (let [gate (promise) + results (mapv (fn [f] + (future + @gate + (f))) + thunks)] + (deliver gate true) + (mapv deref results))) + ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; JOB-DEF (plain handler + precompiled init-key) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; @@ -108,14 +122,6 @@ :id (uuid/next) :file-id (uuid/next)}) -(defn- insert-job-statement? - "True when a `db/exec-one!` call is the job INSERT, whatever the - argument shape (a `[sql & params]` vector or a bare SQL string)." - [args] - (let [stmt (second args) - sql (if (string? stmt) stmt (first stmt))] - (and (string? sql) (str/starts-with? sql "insert into job")))) - (def ^:private test-error {:type :internal :code :boom @@ -433,70 +439,74 @@ (t/is (uuid? (:file-id decoded))) (t/is (ct/inst? (:deleted-at decoded))))))))) -(t/deftest submit-dedupe-replaces-not-due-new-rows +;; --- DEDUPE / DEBOUNCE + +(t/deftest submit-dedupe-skips-a-pending-job (let [cfg (make-cfg (get-job-defs)) - params (make-params) label "dedupe-label" - id1 (jobs/submit cfg {::jobs/name :echo - ::jobs/params params - ::jobs/label label - ::jobs/dedupe true - ::jobs/delay 10000}) - id2 (jobs/submit cfg {::jobs/name :echo - ::jobs/params params - ::jobs/label label - ::jobs/dedupe true - ::jobs/delay 10000})] + opts {::jobs/name :echo + ::jobs/label label + ::jobs/dedupe true + ::jobs/delay 10000} + id1 (jobs/submit cfg (assoc opts ::jobs/params (make-params))) + row1 (jobs/get-job cfg id1) + id2 (jobs/submit cfg (assoc opts + ::jobs/params (assoc (make-params) + :text "second")))] (t/is (uuid? id1)) - (t/is (uuid? id2)) - (t/is (not= id1 id2)) + (t/testing "the pending job is not touched and nothing is inserted" + (t/is (nil? id2)) + (t/is (= 1 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job + WHERE label = ?" label])))) + (let [row2 (jobs/get-job cfg id1)] + (t/is (= (:scheduled-at row1) (:scheduled-at row2)) + "the schedule of the pending job does not move") + (t/is (= "hello" (:text (:params row2))) + "and it keeps its own params"))))) - (t/testing "the first new row (not due yet) is removed" - (t/is (nil? (jobs/get-job cfg id1))) +(t/deftest submit-dedupe-checks-any-schedule + (let [cfg (make-cfg (get-job-defs)) + params (make-params)] - (t/testing "and only the new row remains" + (t/testing "a due `new` row still counts as pending" + (let [label "dedupe-due-label" + id1 (jobs/submit cfg {::jobs/name :echo + ::jobs/params params + ::jobs/label label + ::jobs/dedupe true + ::jobs/delay 10000}) + _ (th/db-update! :job {:scheduled-at (ct/minus (ct/now) + (ct/duration {:seconds 5}))} + {:id id1}) + id2 (jobs/submit cfg {::jobs/name :echo + ::jobs/params params + ::jobs/label label + ::jobs/dedupe true})] + (t/is (nil? id2)) (t/is (= 1 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job - WHERE label = ?" label])))))))) + WHERE label = ?" label])))))) -(t/deftest submit-dedupe-keeps-due-and-running-rows - (let [cfg (make-cfg (get-job-defs)) - params (make-params) - label "dedupe-active-label" - id1 (jobs/submit cfg {::jobs/name :echo - ::jobs/params params - ::jobs/label label}) - _ (th/db-update! :job {:status "running"} {:id id1}) - id2 (jobs/submit cfg {::jobs/name :echo - ::jobs/params params - ::jobs/label label - ::jobs/dedupe true})] - - (t/is (some? (jobs/get-job cfg id1))) - (t/is (= 2 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job - WHERE label = ?" label]))))) - - (let [cfg (make-cfg (get-job-defs)) - params (make-params) - label "dedupe-due-label" - id1 (jobs/submit cfg {::jobs/name :echo - ::jobs/params params - ::jobs/label label}) - _ (th/db-update! :job {:scheduled-at (ct/minus (ct/now) - (ct/duration {:seconds 5}))} - {:id id1}) - id2 (jobs/submit cfg {::jobs/name :echo - ::jobs/params params - ::jobs/label label - ::jobs/dedupe true})] - - (t/is (some? (jobs/get-job cfg id1))) - (t/is (= 2 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job - WHERE label = ?" label])))))) + (t/testing "a started row does not block the next one" + (doseq [status ["scheduled" "running" "retry"]] + (let [label (str "dedupe-started-" status) + id1 (jobs/submit cfg {::jobs/name :echo + ::jobs/params params + ::jobs/label label + ::jobs/dedupe true}) + _ (th/db-update! :job {:status status} {:id id1}) + id2 (jobs/submit cfg {::jobs/name :echo + ::jobs/params params + ::jobs/label label + ::jobs/dedupe true})] + (t/is (uuid? id2)) + (t/is (= 2 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job + WHERE label = ?" label]))))))))) (t/deftest submit-dedupe-leaves-another-tenants-rows-alone - ;; The DELETE matches name, queue and label. Without the tenant in the - ;; match it would reach into another instance's not-started jobs. + ;; The tenant is part of the key of the index. Without it this submit + ;; would see the not-started job of another instance sharing the + ;; database and skip its own work. (let [cfg (make-cfg (get-job-defs)) params (make-params) label "dedupe-shared-label" @@ -511,22 +521,149 @@ :priority 100 :max-retries 3 :retry-num 0 + :dedupe true :scheduled-at (ct/plus (ct/now) (ct/duration {:seconds 10})) :created-at (ct/now) :modified-at (ct/now)}) - (jobs/submit cfg {::jobs/name :echo - ::jobs/params params - ::jobs/queue :default - ::jobs/label label - ::jobs/dedupe true - ::jobs/delay 10000}) - (t/is (some? (jobs/get-job cfg foreign-id)) - "another tenant's matching job must survive our dedupe") - (t/testing "and only our own row was written" + (let [id (jobs/submit cfg {::jobs/name :echo + ::jobs/params params + ::jobs/queue :default + ::jobs/label label + ::jobs/dedupe true + ::jobs/delay 10000})] + (t/is (uuid? id)) + (t/is (some? (jobs/get-job cfg foreign-id)) + "another tenant's matching job must survive our submit") + (t/testing "and only our own row was written" + (t/is (= 1 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job + WHERE label = ? + AND tenant = ?" + label (cf/get :tenant)])))))))) + +(t/deftest submit-dedupe-does-not-block-on-plain-jobs + ;; Only the jobs submitted with the option are part of its key, so the + ;; label of an ordinary job stays a label. + (let [cfg (make-cfg (get-job-defs)) + params (make-params) + label "dedupe-over-plain"] + (t/is (uuid? (jobs/submit cfg {::jobs/name :echo + ::jobs/params params + ::jobs/label label}))) + (t/is (uuid? (jobs/submit cfg {::jobs/name :echo + ::jobs/params params + ::jobs/label label + ::jobs/dedupe true}))) + (t/is (= 2 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job + WHERE label = ?" label])))))) + +(t/deftest two-concurrent-dedupe-submits-leave-one-row + ;; The exclusion is the index, not a read before the write: two + ;; backends submitting the same key at the same time cannot both win. + (let [cfg (make-cfg (get-job-defs)) + params (make-params) + label "dedupe-race" + submit #(jobs/submit cfg {::jobs/name :echo + ::jobs/params params + ::jobs/label label + ::jobs/dedupe true})] + (let [[a b] (race submit submit)] + (t/is (= 1 (count (filter uuid? [a b]))) + "exactly one of the two submits got a job") (t/is (= 1 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job - WHERE label = ? - AND tenant = ?" - label (cf/get :tenant)]))))))) + WHERE label = ?" label]))))))) + +(t/deftest submit-debounce-refreshes-the-pending-job + (let [cfg (make-cfg (get-job-defs)) + label "debounce-label" + opts {::jobs/name :echo + ::jobs/label label + ::jobs/debounce true} + id1 (jobs/submit cfg (assoc opts + ::jobs/params (make-params) + ::jobs/delay 10000)) + row1 (jobs/get-job cfg id1) + id2 (jobs/submit cfg (assoc opts + ::jobs/params (assoc (make-params) + :text "second") + ::jobs/delay 60000)) + row2 (jobs/get-job cfg id1)] + + (t/is (= id1 id2) + "a refresh answers with the id the row already had") + (t/is (= 1 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job + WHERE label = ?" label])))) + (t/testing "the pending job keeps its id and its creation date" + (t/is (some? row2)) + (t/is (= (:id row1) (:id row2))) + (t/is (= (:created-at row1) (:created-at row2)))) + (t/testing "and takes the new schedule and the new params" + (t/is (pos? (compare (:scheduled-at row2) (:scheduled-at row1)))) + (t/is (= "second" (:text (:params row2))))))) + +(t/deftest submit-debounce-inserts-when-nothing-is-pending + (let [cfg (make-cfg (get-job-defs)) + params (make-params) + label "debounce-idle-label" + id (jobs/submit cfg {::jobs/name :echo + ::jobs/params params + ::jobs/label label + ::jobs/debounce true})] + (t/is (uuid? id)) + (t/is (= 1 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job + WHERE label = ?" label])))))) + +(t/deftest submit-debounce-inserts-past-started-rows + (doseq [status ["scheduled" "running" "retry"]] + (t/testing (str "started status " status) + (let [cfg (make-cfg (get-job-defs)) + params (make-params) + label (str "debounce-started-" status) + id1 (jobs/submit cfg {::jobs/name :echo + ::jobs/params params + ::jobs/label label + ::jobs/debounce true}) + _ (th/db-update! :job {:status status} {:id id1}) + id2 (jobs/submit cfg {::jobs/name :echo + ::jobs/params params + ::jobs/label label + ::jobs/debounce true})] + (t/is (uuid? id2)) + (t/is (not= id1 id2)) + (t/is (= 2 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job + WHERE label = ?" label])))))))) + +(t/deftest submit-keyed-options-need-a-label-and-are-exclusive + (let [cfg (make-cfg (get-job-defs)) + params (make-params)] + (t/testing "dedupe" + (t/is (thrown-with-msg? Exception #"dedupe requires" + (jobs/submit cfg {::jobs/name :echo + ::jobs/params params + ::jobs/dedupe true}))) + (t/is (thrown-with-msg? Exception #"dedupe requires" + (jobs/submit cfg {::jobs/name :echo + ::jobs/params params + ::jobs/label " " + ::jobs/dedupe true})))) + (t/testing "debounce" + (t/is (thrown-with-msg? Exception #"debounce requires" + (jobs/submit cfg {::jobs/name :echo + ::jobs/params params + ::jobs/debounce true})))) + (t/testing "and they contradict each other" + (t/is (thrown-with-msg? Exception #"cannot be used together" + (jobs/submit cfg {::jobs/name :echo + ::jobs/params params + ::jobs/label "both-label" + ::jobs/dedupe true + ::jobs/debounce true})))) + (t/testing "and neither fits an exclusive submit" + (t/is (thrown-with-msg? Exception #"cannot be combined" + (jobs/submit cfg {::jobs/name :echo + ::jobs/params params + ::jobs/label "both-label" + ::jobs/dedupe true + ::jobs/skip-if-active true})))))) (t/deftest submit-skip-if-active-skips-live-jobs (doseq [status ["new" "scheduled" "running" "retry"]] @@ -534,13 +671,14 @@ (let [cfg (make-cfg (get-job-defs)) params (make-params) label (str "skip-active-" status) - id1 (jobs/submit cfg {::jobs/name :echo - ::jobs/params params - ::jobs/label label}) + id1 (jobs/submit cfg {::jobs/name :echo + ::jobs/params params + ::jobs/label label + ::jobs/skip-if-active true}) _ (th/db-update! :job {:status status} {:id id1}) - id2 (jobs/submit cfg {::jobs/name :echo - ::jobs/params params - ::jobs/label label + id2 (jobs/submit cfg {::jobs/name :echo + ::jobs/params params + ::jobs/label label ::jobs/skip-if-active true})] (t/is (nil? id2)) (t/testing "no row deleted, none inserted" @@ -554,18 +692,109 @@ (let [cfg (make-cfg (get-job-defs)) params (make-params) label (str "skip-terminal-" status) - id1 (jobs/submit cfg {::jobs/name :echo - ::jobs/params params - ::jobs/label label}) + id1 (jobs/submit cfg {::jobs/name :echo + ::jobs/params params + ::jobs/label label + ::jobs/skip-if-active true}) _ (th/db-update! :job {:status status} {:id id1}) - id2 (jobs/submit cfg {::jobs/name :echo - ::jobs/params params - ::jobs/label label + id2 (jobs/submit cfg {::jobs/name :echo + ::jobs/params params + ::jobs/label label ::jobs/skip-if-active true})] (t/is (uuid? id2)) (t/is (= 2 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job WHERE label = ?" label])))))))) +(t/deftest submit-skip-if-active-does-not-block-on-plain-jobs + ;; The exclusion is between the jobs that asked for it: a live job + ;; that was not submitted as exclusive (a webhook delivery, a + ;; notification) does not block an exclusive submit. The label stays + ;; a label, and two plain jobs can share it. + (let [cfg (make-cfg (get-job-defs)) + params (make-params) + label "skip-over-plain" + plain (jobs/submit cfg {::jobs/name :echo + ::jobs/params params + ::jobs/label label}) + excl (jobs/submit cfg {::jobs/name :echo + ::jobs/params params + ::jobs/label label + ::jobs/skip-if-active true})] + (t/is (uuid? plain)) + (t/is (uuid? excl)) + (t/is (= 2 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job + WHERE label = ?" label])))) + (t/testing "and two plain jobs share the label too" + (let [other (jobs/submit cfg {::jobs/name :echo + ::jobs/params params + ::jobs/label label})] + (t/is (uuid? other)) + (t/is (= 3 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job + WHERE label = ?" label])))))))) + +(t/deftest exclusive-jobs-are-scoped-by-their-key + ;; The key of the exclusion is (tenant, name, queue, label): a job + ;; that differs in any of them is a different job and lives with the + ;; first one. + (let [cfg (make-cfg (get-job-defs)) + params (make-params) + base {::jobs/name :echo + ::jobs/params params + ::jobs/skip-if-active true}] + (t/is (uuid? (jobs/submit cfg (assoc base ::jobs/label "scope-a")))) + (t/is (uuid? (jobs/submit cfg (assoc base ::jobs/label "scope-a" + ::jobs/queue :storage)))) + (t/is (uuid? (jobs/submit cfg (assoc base ::jobs/label "scope-a" + ::jobs/name :typed)))) + (t/is (nil? (jobs/submit cfg (assoc base ::jobs/label "scope-a")))))) + +(t/deftest the-job-table-refuses-two-live-exclusive-rows + ;; The exclusion does not live in the submit: the unique index of the + ;; live exclusive jobs refuses the second row, whoever writes it. + (let [cfg (make-cfg (get-job-defs)) + params (make-params) + label "db-enforced" + id1 (jobs/submit cfg {::jobs/name :echo + ::jobs/params params + ::jobs/label label + ::jobs/skip-if-active true}) + row {:name "echo" + :tenant (cf/get :tenant) + :queue "default" + :label label + :params (db/json params) + :priority 100 + :max-retries 3 + :retry-num 0 + :exclusive true + :scheduled-at (ct/now) + :created-at (ct/now) + :modified-at (ct/now)}] + (t/is (uuid? id1)) + (t/is (thrown? Exception + (th/db-insert! :job (assoc row :id (uuid/next))))) + (t/testing "and the row that was there is untouched" + (t/is (some? (jobs/get-job cfg id1))) + (t/is (= 1 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job + WHERE label = ?" label]))))))) + +(t/deftest two-concurrent-exclusive-submits-leave-one-row + ;; The check and the insert are the same statement, so two backends + ;; submitting the same exclusive key at the same time cannot both + ;; win: the second one is rejected by the index. + (let [cfg (make-cfg (get-job-defs)) + params (make-params) + label "exclusive-race" + submit #(jobs/submit cfg {::jobs/name :echo + ::jobs/params params + ::jobs/label label + ::jobs/skip-if-active true})] + (let [[a b] (race submit submit)] + (t/is (= 1 (count (filter uuid? [a b]))) + "exactly one of the two submits got a job") + (t/is (= 1 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job + WHERE label = ?" label]))))))) + (t/deftest submit-skip-if-active-requires-an-explicit-label (let [cfg (make-cfg (get-job-defs)) params (make-params)] @@ -591,55 +820,6 @@ (t/is (= 1 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job WHERE label = ?" label])))))) -(t/deftest submit-dedupe-rolls-back-on-insert-failure - (let [cfg (make-cfg (get-job-defs)) - params (make-params) - opts {::jobs/name :echo - ::jobs/params params - ::jobs/dedupe true - ::jobs/label "atomic-label"} - kept (jobs/submit cfg opts) - orig @#'db/exec-one!] - ;; fault the INSERT: the DELETE before it must roll back too - (alter-var-root #'db/exec-one! - (constantly (fn [& args] - (when (insert-job-statement? args) - (throw (ex-info "boom" {}))) - (apply orig args)))) - (try - (t/is (thrown? Exception (jobs/submit cfg opts))) - (t/testing "the original row survives, no duplicate left behind" - (t/is (some? (jobs/get-job cfg kept))) - (t/is (= 1 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job WHERE label = ?" "atomic-label"]))))) - (finally - (alter-var-root #'db/exec-one! (constantly orig)))))) - -(t/deftest submit-dedupe-atomic-on-autocommit-conn - (let [cfg (make-cfg (get-job-defs)) - params (make-params) - opts {::jobs/name :echo - ::jobs/params params - ::jobs/dedupe true - ::jobs/label "atomic-autocommit-label"} - kept (jobs/submit cfg opts) - orig @#'db/exec-one!] - ;; a raw connection outside any transaction: DELETE+INSERT must - ;; still share one transaction opened on it - (with-open [conn (db/open th/*pool*)] - (alter-var-root #'db/exec-one! - (constantly (fn [& args] - (when (insert-job-statement? args) - (throw (ex-info "boom" {}))) - (apply orig args)))) - (try - (t/is (thrown? Exception - (jobs/submit (assoc cfg ::db/conn conn) opts))) - (t/testing "the original row survives, no duplicate left behind" - (t/is (some? (jobs/get-job cfg kept))) - (t/is (= 1 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job WHERE label = ?" "atomic-autocommit-label"]))))) - (finally - (alter-var-root #'db/exec-one! (constantly orig))))))) - (t/deftest submit-requires-metrics-on-bare-connectable (let [defs (get-job-defs) prev @@#'jobs/defs-registry] @@ -1683,20 +1863,6 @@ ;; on. Both are only meaningful when two callers arrive at once, so the ;; tests below put real threads in front of them. -(defn- race - "Run every thunk on a thread of its own and return what each returned, - in the order they were given. They all wait on the same gate, so they - contend for whatever they share instead of running one after another." - [& thunks] - (let [gate (promise) - results (mapv (fn [f] - (future - @gate - (f))) - thunks)] - (deliver gate true) - (mapv deref results))) - (t/deftest the-first-terminal-transition-wins (let [cfg (make-cfg (get-job-defs)) job-id (mk-running cfg)] diff --git a/backend/test/backend_tests/loggers_audit_consumer_task_test.clj b/backend/test/backend_tests/loggers_audit_consumer_task_test.clj deleted file mode 100644 index e538ae31d6..0000000000 --- a/backend/test/backend_tests/loggers_audit_consumer_task_test.clj +++ /dev/null @@ -1,837 +0,0 @@ -;; This Source Code Form is subject to the terms of the Mozilla Public -;; License, v. 2.0. If a copy of the MPL was not distributed with this -;; file, You can obtain one at http://mozilla.org/MPL/2.0/. -;; -;; Copyright (c) KALEIDOS SUBSIDIARY SL - -(ns backend-tests.loggers-audit-consumer-task-test - (:require - [app.common.json :as json] - [app.common.schema :as sm] - [app.common.time :as ct] - [app.common.transit :as transit] - [app.common.uri :as u] - [app.common.uuid :as uuid] - [app.config :as cf] - [app.db :as db] - [app.http.client :as-alias http] - [app.jobs :as jobs] - [app.loggers.audit :as audit] - [app.loggers.audit.consumer-task :as consumer] - [app.main :as main] - [app.metrics :as mtx] - [app.metrics.definition :as-alias mdef] - [app.setup :as-alias setup] - [backend-tests.helpers :as th] - [clojure.test :as t] - [cuerdas.core :as str] - [integrant.core :as ig] - [mockery.core :refer [with-mocks]]) - (:import - io.prometheus.client.Counter - io.prometheus.client.Counter$Child)) - -(t/use-fixtures :once th/state-init) -(t/use-fixtures :each th/database-reset) - -(def ^:private archive-uri "http://nexus.example/archive") - -(defn- insert-pendings! - "Queue the row with the real producer helper: pendings for the - enabled consumers, like append-audit-entry does." - [row-id] - (consumer/queue-pendings (:app.db/pool th/*system*) [row-id])) - -(defn- pending-count - [] - (:pending (first (th/db-exec! ["SELECT count(*) AS pending FROM audit_log_pending"])))) - -(defn- insert-audit-row! - ([profile-id name] - (insert-audit-row! profile-id name {:team-id (uuid/next)})) - ([profile-id name props] - (let [row (th/db-insert! :audit-log - {:id (uuid/next) - :name name - :type "action" - :source "backend" - :profile-id profile-id - :ip-addr (db/inet "127.0.0.1") - :props (db/tjson props) - :context (db/tjson {}) - :tracked-at (ct/now) - :created-at (ct/now)})] - (insert-pendings! (:id row)) - row))) - -(defn- admin-console-event - [admin-console-mock] - (let [[_ _ params] (:call-args @admin-console-mock)] - (first (:events params)))) - -(defn- admin-console-event-props - [admin-console-mock] - (:props (admin-console-event admin-console-mock))) - -(defn- run-both! - "Run both consumer jobs, Nexus first (the historical single-run - order). The admin-console job takes no params (closed empty schema)." - [params] - (th/run-task! :audit-log-consumer-nexus params) - (th/run-task! :audit-log-consumer-admin-console {})) - -(t/deftest consumer-jobs-drain-their-own-queues - ;; Two separate jobs, one queue each: the nexus job sends without - ;; touching the admin console, and only the admin-console job - ;; deletes the last pending and the row. Create profile before - ;; enabling :admin-console — the system admin-console client is nil - ;; in tests, and team bootstrap would call it. - (let [prof (th/create-profile* 1 {:is-active true})] - (with-redefs [cf/flags #{:nexus :admin-console}] - (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil} - http-mock {:target 'app.http.client/req :return {:status 204}}] - (insert-audit-row! (:id prof) "create-project") - (insert-audit-row! (:id prof) "create-file") - (th/run-task! :audit-log-consumer-nexus {:uri archive-uri}) - (t/is (false? (:called? @admin-console-mock))) - (t/is (:called? @http-mock)) - (t/is (= 1 (:call-count @http-mock))) - ;; Nexus drained its own queue; both rows wait on the admin console. - (t/is (= 2 (count (th/db-exec! ["select * from audit_log"])))) - (t/is (= 2 (pending-count))) - (th/run-task! :audit-log-consumer-admin-console {}) - (t/is (= 1 (:call-count @admin-console-mock))) - (let [[_ method params] (:call-args @admin-console-mock)] - (t/is (= :ingest-audit-log method)) - (t/is (= 2 (count (:events params)))) - (t/is (= #{"create-project" "create-file"} - (into #{} (map :name) (:events params)))) - (t/is (every? uuid? (map :id (:events params))))) - (let [rows (th/db-exec! ["select * from audit_log"])] - (t/is (= 0 (count rows)))))))) - -(t/deftest consumer-events-filters-admin-console-event-names - (let [prof (th/create-profile* 1 {:is-active true})] - (with-redefs [cf/flags #{:nexus :admin-console}] - (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil} - http-mock {:target 'app.http.client/req :return {:status 204}}] - (insert-audit-row! (:id prof) "create-project") - (insert-audit-row! (:id prof) "create-organization-attribute") - (insert-audit-row! (:id prof) "delete-organization-attribute") - (insert-audit-row! (:id prof) "update-organization-attribute") - (insert-audit-row! (:id prof) "auditlog-exported") - (insert-audit-row! (:id prof) "unrelated-event") - (run-both! {:uri archive-uri}) - (t/is (:called? @admin-console-mock)) - (t/is (:called? @http-mock)) - (let [[_ _ params] (:call-args @admin-console-mock)] - (t/is (= #{"create-project" - "create-organization-attribute" - "delete-organization-attribute" - "update-organization-attribute" - "auditlog-exported"} - (into #{} (map :name) (:events params))))) - (let [rows (th/db-exec! ["select * from audit_log"])] - (t/is (= 0 (count rows)))))))) - -(t/deftest consumer-events-skips-admin-console-when-no-matching-names - (let [prof (th/create-profile* 1 {:is-active true})] - (with-redefs [cf/flags #{:nexus :admin-console}] - (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil} - http-mock {:target 'app.http.client/req :return {:status 204}}] - (insert-audit-row! (:id prof) "unrelated-event") - (run-both! {:uri archive-uri}) - (t/is (false? (:called? @admin-console-mock))) - (t/is (:called? @http-mock)) - (let [rows (th/db-exec! ["select * from audit_log"])] - (t/is (= 0 (count rows)))))))) - -(t/deftest consumer-events-admin-console-only-marks-without-uri - (let [prof (th/create-profile* 1 {:is-active true})] - (with-redefs [cf/flags #{:admin-console}] - (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil} - http-mock {:target 'app.http.client/req :return {:status 204}}] - (insert-audit-row! (:id prof) "create-project") - (th/run-task! :audit-log-consumer-admin-console {}) - (t/is (:called? @admin-console-mock)) - (t/is (false? (:called? @http-mock))) - (let [rows (th/db-exec! ["select * from audit_log"])] - (t/is (= 0 (count rows)))))))) - -(t/deftest consumer-events-admin-console-only-marks-non-allowlisted - (let [prof (th/create-profile* 1 {:is-active true})] - (with-redefs [cf/flags #{:admin-console}] - (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil} - http-mock {:target 'app.http.client/req :return {:status 204}}] - (insert-audit-row! (:id prof) "unrelated-event") - (th/run-task! :audit-log-consumer-admin-console {}) - (t/is (false? (:called? @admin-console-mock))) - (t/is (false? (:called? @http-mock))) - (let [rows (th/db-exec! ["select * from audit_log"])] - (t/is (= 0 (count rows)))))))) - -(t/deftest consumer-events-admin-console-error-still-drains-nexus - ;; Mirror of the nexus-throw test: an admin-console failure aborts - ;; only the admin-console job; Nexus already drained its own queue - ;; in its job. The row stays on the admin-console pending. - (let [prof (th/create-profile* 1 {:is-active true})] - (with-redefs [cf/flags #{:nexus :admin-console}] - (with-mocks [admin-console-mock {:target 'app.nitrate/call - :throw (ex-info "nitrate down" {})} - http-mock {:target 'app.http.client/req :return {:status 204}}] - (insert-audit-row! (:id prof) "create-project") - (th/run-task! :audit-log-consumer-nexus {:uri archive-uri}) - (t/is (thrown? clojure.lang.ExceptionInfo - (th/run-task! :audit-log-consumer-admin-console {}))) - ;; Nexus already drained its queue; the admin-console failure no - ;; longer blocks it, but the row stays on the admin-console pending. - (t/is (:called? @admin-console-mock)) - (t/is (:called? @http-mock)) - (let [rows (th/db-exec! ["select * from audit_log"])] - (t/is (= 1 (count rows)))) - (t/is (= 1 (pending-count))))))) - -(t/deftest consumer-events-uses-nexus-without-admin-console - (let [prof (th/create-profile* 1 {:is-active true})] - (with-redefs [cf/flags #{:nexus}] - (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}} - admin-console-mock {:target 'app.nitrate/call :return nil}] - (insert-audit-row! (:id prof) "create-project") - (th/run-task! :audit-log-consumer-nexus {:uri archive-uri}) - (t/is (false? (:called? @admin-console-mock))) - (t/is (:called? @http-mock)) - (let [rows (th/db-exec! ["select * from audit_log"])] - (t/is (= 0 (count rows)))))))) - -(t/deftest consumer-events-skips-mark-when-nexus-fails - (let [prof (th/create-profile* 1 {:is-active true})] - (with-redefs [cf/flags #{:nexus}] - (with-mocks [http-mock {:target 'app.http.client/req :return {:status 500}}] - (insert-audit-row! (:id prof) "create-project") - (t/is (thrown? clojure.lang.ExceptionInfo - (th/run-task! :audit-log-consumer-nexus {:uri archive-uri}))) - (t/is (:called? @http-mock)) - (let [rows (th/db-exec! ["select * from audit_log"])] - (t/is (= 1 (count rows)))))))) - -(t/deftest consumer-events-both-flags-skips-mark-when-nexus-fails - (let [prof (th/create-profile* 1 {:is-active true})] - (with-redefs [cf/flags #{:nexus :admin-console}] - (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil} - http-mock {:target 'app.http.client/req :return {:status 500}}] - (insert-audit-row! (:id prof) "create-project") - (t/is (thrown? clojure.lang.ExceptionInfo - (th/run-task! :audit-log-consumer-nexus {:uri archive-uri}))) - (th/run-task! :audit-log-consumer-admin-console {}) - (t/is (:called? @admin-console-mock)) - (t/is (:called? @http-mock)) - (let [rows (th/db-exec! ["select * from audit_log"])] - (t/is (= 1 (count rows)))))))) - -(t/deftest consumer-events-encodes-db-rows-for-admin-console - (let [prof (th/create-profile* 1 {:is-active true})] - (with-redefs [cf/flags #{:nexus :admin-console} - cf/config (assoc cf/config - :admin-console-uri "http://ac.example/admin-console/" - :admin-console-shared-key "ac-key" - :nexus-shared-key "nexus-key")] - (let [admin-console-bodies (atom []) - shared-keys {:admin-console "ac-key" :nexus "nexus-key"} - client (ig/init-key :app.nitrate/client - {::http/client (Object.) - ::setup/shared-keys shared-keys}) - ;; Each consumer job is a job-def: the handler closes over the - ;; cfg it was given at init and takes a nil context, because - ;; an in-process invocation has no job row to describe. - nexus-job-def (ig/init-key :app.loggers.audit.consumer-task/nexus-job-def - {::db/pool (:app.db/pool th/*system*) - ::setup/shared-keys shared-keys - ::http/client (:app.http.client/client th/*system*) - ::mtx/metrics (th/metrics-stub) - :app.nitrate/client client}) - admin-console-job-def (ig/init-key :app.loggers.audit.consumer-task/admin-console-job-def - {::db/pool (:app.db/pool th/*system*) - ::mtx/metrics (th/metrics-stub) - :app.nitrate/client client})] - (with-mocks [http-mock {:target 'app.http.client/req - :return - (fn [_ req & _] - (when (str/includes? (str (:uri req)) "api/audit-log") - (swap! admin-console-bodies conj (:body req))) - {:status 204})}] - (let [team-id (uuid/next) - row (th/db-insert! :audit-log - {:id (uuid/next) - :name "create-project" - :type "action" - :source "backend" - :profile-id (:id prof) - :ip-addr (db/inet "127.0.0.1") - :props (db/tjson {:team-id team-id}) - :context (db/tjson {}) - :tracked-at (ct/now) - :created-at (ct/now)})] - (insert-pendings! (:id row)) - ;; Admin console first so its body is captured before Nexus - ;; deletes the last pending and the row. - ((::jobs/handler admin-console-job-def) nil {}) - ((::jobs/handler nexus-job-def) nil {:uri archive-uri}) - (t/is (= 1 (count @admin-console-bodies))) - (let [body (json/decode (first @admin-console-bodies) :key-fn json/read-kebab-key) - event (first (:events body))] - (t/is (= 1 (count (:events body)))) - (t/is (= (str (:id row)) (str (:id event)))) - (t/is (= "create-project" (:name event))) - (t/is (= "127.0.0.1" (:ip-addr event))) - (t/is (string? (:created-at event))) - (t/is (string? (:tracked-at event))) - (t/is (= (str team-id) (str (get-in event [:props :team-id]))))) - (let [rows (th/db-exec! ["select * from audit_log"])] - (t/is (= 0 (count rows)))))))))) - -(t/deftest consumer-nexus-default-uri-appends-api-audit - ;; Without an explicit :uri param, the job posts to the Nexus base - ;; URI plus the ingest path, mirroring generate-nitrate-uri. - (let [prof (th/create-profile* 1 {:is-active true}) - uris (atom [])] - (with-redefs [cf/flags #{:nexus} - cf/config (assoc cf/config :nexus-uri (u/uri "http://nexus.example"))] - (with-mocks [http-mock {:target 'app.http.client/req - :return (fn [_ params & _] - (swap! uris conj (str (:uri params))) - {:status 204})}] - (insert-audit-row! (:id prof) "create-project") - (th/run-task! :audit-log-consumer-nexus {}) - (t/is (= ["http://nexus.example/api/audit"] @uris)) - (t/is (= 0 (count (th/db-exec! ["select * from audit_log"])))))))) - -(t/deftest consumer-nexus-job-requires-uri-when-nexus-flag-set - (with-redefs [cf/flags #{:nexus} - cf/config (dissoc cf/config :nexus-uri)] - (let [data (ex-data (try - (th/run-task! :audit-log-consumer-nexus {}) - (catch Throwable cause - cause)))] - (t/is (= :task-not-configured (:code data)))))) - -(t/deftest consumer-events-enriches-team-name-and-emails - (let [actor (th/create-profile* 1 {:is-active true}) - member (th/create-profile* 2 {:is-active true}) - team (th/create-team* 1 {:profile-id (:id actor)})] - (with-redefs [cf/flags #{:admin-console}] - (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}] - (insert-audit-row! (:id actor) "create-project" - {:team-id (:id team) - :profile-id (:id actor) - :user-id (:id member) - :member-id (:id member) - ;; Producers store this as a string UUID. - :user-who-send-invitation (str (:id actor))}) - (th/run-task! :audit-log-consumer-admin-console {}) - (t/is (:called? @admin-console-mock)) - (let [event (admin-console-event admin-console-mock) - props (:props event)] - (t/is (= (:name team) (:team-name props))) - (t/is (= (:email actor) (:profile-email props))) - (t/is (= (:email member) (:user-email props))) - (t/is (= (:email member) (:member-email props))) - (t/is (= (:email actor) (:user-who-send-invitation-email props))) - ;; Top-level actor fields (event :profile-id), distinct from props. - (t/is (= (:email actor) (:profile-email event))) - (t/is (= (:fullname actor) (:profile-name event)))) - (let [rows (th/db-exec! ["select * from audit_log"])] - (t/is (= 0 (count rows)))))))) - -(t/deftest consumer-events-enriches-soft-deleted-team-and-profile - (let [actor (th/create-profile* 1 {:is-active true}) - other (th/create-profile* 2 {:is-active true}) - team (th/create-team* 1 {:profile-id (:id actor)})] - (th/db-update! :team {:deleted-at (ct/now)} {:id (:id team)}) - (th/db-update! :profile {:deleted-at (ct/now)} {:id (:id other)}) - (with-redefs [cf/flags #{:admin-console}] - (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}] - (insert-audit-row! (:id actor) "create-project" - {:team-id (:id team) - :profile-id (:id other)}) - (th/run-task! :audit-log-consumer-admin-console {}) - (let [event (admin-console-event admin-console-mock) - props (:props event)] - (t/is (= (:name team) (:team-name props))) - (t/is (= (:email other) (:profile-email props))) - (t/is (= (:email actor) (:profile-email event))) - (t/is (= (:fullname actor) (:profile-name event)))) - (let [rows (th/db-exec! ["select * from audit_log"])] - (t/is (= 0 (count rows)))))))) - -(t/deftest consumer-events-enriches-soft-deleted-actor-profile - (let [actor (th/create-profile* 1 {:is-active true}) - team (th/create-team* 1 {:profile-id (:id actor)})] - (th/db-update! :profile {:deleted-at (ct/now)} {:id (:id actor)}) - (with-redefs [cf/flags #{:admin-console}] - (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}] - (insert-audit-row! (:id actor) "create-project" - {:team-id (:id team)}) - (th/run-task! :audit-log-consumer-admin-console {}) - (let [event (admin-console-event admin-console-mock)] - (t/is (= (:email actor) (:profile-email event))) - (t/is (= (:fullname actor) (:profile-name event)))))))) - -(t/deftest consumer-events-omits-actor-profile-when-no-usable-id - (let [actor (th/create-profile* 1 {:is-active true}) - team (th/create-team* 1 {:profile-id (:id actor)}) - missing (uuid/next)] - (with-redefs [cf/flags #{:admin-console}] - (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}] - (insert-audit-row! uuid/zero "create-project" {:team-id (:id team)}) - (insert-audit-row! missing "create-team" {:team-id (:id team)}) - (th/run-task! :audit-log-consumer-admin-console {}) - (t/is (:called? @admin-console-mock)) - (let [[_ _ params] (:call-args @admin-console-mock) - events (:events params) - by-name (into {} (map (juxt :name identity) events)) - zero-evt (get by-name "create-project") - missing-evt (get by-name "create-team")] - (t/is (not (contains? zero-evt :profile-email))) - (t/is (not (contains? zero-evt :profile-name))) - (t/is (not (contains? missing-evt :profile-email))) - (t/is (not (contains? missing-evt :profile-name)))))))) - -(t/deftest consumer-events-does-not-overwrite-existing-enrich-props - (let [actor (th/create-profile* 1 {:is-active true}) - team (th/create-team* 1 {:profile-id (:id actor)})] - (with-redefs [cf/flags #{:admin-console}] - (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}] - (insert-audit-row! (:id actor) "create-project" - {:team-id (:id team) - :team-name "already-set-name" - :profile-id (:id actor) - :profile-email "already-set@example.com" - :user-id (:id actor) - :user-email "already-user@example.com" - :member-id (:id actor) - :member-email "already-member@example.com" - :user-who-send-invitation (str (:id actor)) - :user-who-send-invitation-email "already-inviter@example.com"}) - (th/run-task! :audit-log-consumer-admin-console {}) - (let [props (admin-console-event-props admin-console-mock)] - (t/is (= "already-set-name" (:team-name props))) - (t/is (= "already-set@example.com" (:profile-email props))) - (t/is (= "already-user@example.com" (:user-email props))) - (t/is (= "already-member@example.com" (:member-email props))) - (t/is (= "already-inviter@example.com" - (:user-who-send-invitation-email props)))))))) - -(t/deftest consumer-events-does-not-overwrite-existing-actor-profile-fields - ;; Top-level :profile-email / :profile-name are not stored on the row; - ;; exercise fill-only via the enrich helper directly. - (let [actor (th/create-profile* 1 {:is-active true}) - event {:profile-id (:id actor) - :profile-email "preset@example.com" - :profile-name "Preset Name" - :props {}}] - (db/tx-run! th/*system* - (fn [cfg] - (let [enriched (first (#'consumer/enrich-events-for-admin-console - cfg [event]))] - (t/is (= "preset@example.com" (:profile-email enriched))) - (t/is (= "Preset Name" (:profile-name enriched)))))))) - -(t/deftest consumer-events-fills-missing-actor-profile-field-independently - ;; Each top-level key fills on its own; guards an or→and regression in - ;; collect-actor-ids-needing-profile. - (let [actor (th/create-profile* 1 {:is-active true})] - (db/tx-run! th/*system* - (fn [cfg] - (let [email-only (first (#'consumer/enrich-events-for-admin-console - cfg [{:profile-id (:id actor) - :profile-email "preset@example.com" - :props {}}])) - name-only (first (#'consumer/enrich-events-for-admin-console - cfg [{:profile-id (:id actor) - :profile-name "Preset Name" - :props {}}]))] - (t/is (= "preset@example.com" (:profile-email email-only))) - (t/is (= (:fullname actor) (:profile-name email-only))) - (t/is (= (:email actor) (:profile-email name-only))) - (t/is (= "Preset Name" (:profile-name name-only)))))))) - -(t/deftest consumer-events-omits-empty-actor-profile-name - ;; profile.fullname is NOT NULL DEFAULT ''; empty must not become - ;; :profile-name "". - (let [actor (th/create-profile* 1 {:is-active true}) - team (th/create-team* 1 {:profile-id (:id actor)})] - (th/db-update! :profile {:fullname ""} {:id (:id actor)}) - (with-redefs [cf/flags #{:admin-console}] - (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}] - (insert-audit-row! (:id actor) "create-project" {:team-id (:id team)}) - (th/run-task! :audit-log-consumer-admin-console {}) - (let [event (admin-console-event admin-console-mock)] - (t/is (= (:email actor) (:profile-email event))) - (t/is (not (contains? event :profile-name)))))))) - -(t/deftest consumer-events-enrich-skips-missing-and-non-uuid-ids - (let [actor (th/create-profile* 1 {:is-active true})] - (with-redefs [cf/flags #{:admin-console}] - (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}] - (insert-audit-row! (:id actor) "create-project" - {:team-id (uuid/next) - :profile-id (uuid/next) - :user-id "not-a-uuid" - :member-id 42}) - (th/run-task! :audit-log-consumer-admin-console {}) - (t/is (:called? @admin-console-mock)) - (let [props (admin-console-event-props admin-console-mock)] - (t/is (nil? (:team-name props))) - (t/is (nil? (:profile-email props))) - (t/is (nil? (:user-email props))) - (t/is (nil? (:member-email props))) - (t/is (= "not-a-uuid" (:user-id props))) - (t/is (= 42 (:member-id props)))) - (let [rows (th/db-exec! ["select * from audit_log"])] - (t/is (= 0 (count rows)))))))) - -(t/deftest consumer-events-forwards-export-job-not-legacy-export - (let [actor (th/create-profile* 1 {:is-active true}) - team (th/create-team* 1 {:profile-id (:id actor)})] - (with-redefs [cf/flags #{:admin-console}] - (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}] - (insert-audit-row! (:id actor) "create-export-binfile-job" - {:team-id (:id team) - :job-id (uuid/next) - :files 1 - :export-type "detach-libraries"}) - (insert-audit-row! (:id actor) "export-binfile" - {:file-id (uuid/next)}) - (th/run-task! :audit-log-consumer-admin-console {}) - (let [[_ _ params] (:call-args @admin-console-mock) - events (:events params) - props (:props (first events))] - (t/is (= ["create-export-binfile-job"] (mapv :name events))) - (t/is (= (:id team) (:team-id props))) - (t/is (= (:name team) (:team-name props)))) - (let [rows (th/db-exec! ["select * from audit_log"])] - (t/is (= 0 (count rows)))))))) - -(t/deftest consumer-params-accept-empty-cron-props - ;; Cron always submits (or props {}) — system deps live on the - ;; job-def cfg, not in params. Requiring them here blocks every tick. - ;; Both schemas are closed: a stale manual invocation must fail - ;; validation loudly instead of skipping silently. - (let [nexus-valid? (sm/validator consumer/schema:audit-log-consumer-nexus-params) - admin-console-valid? (sm/validator consumer/schema:audit-log-consumer-admin-console-params)] - (t/is (true? (nexus-valid? {}))) - (t/is (false? (nexus-valid? {:enabled true}))) - (t/is (true? (nexus-valid? {:uri (u/uri archive-uri)}))) - (t/is (true? (admin-console-valid? {}))) - (t/is (false? (admin-console-valid? {:enabled true}))) - (t/is (false? (admin-console-valid? {:uri (u/uri archive-uri)}))))) - -;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; -;; METRICS SMOKE TEST -;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; - -;; The job-def cfg of these tests carries an isolated metrics registry -;; (pattern from `jobs-metrics-test`), so assertions read exact values -;; instead of the shared test-system registry. `th/run-task!` would use -;; the latter and its counts would leak across tests. -(def ^:private consumer-metric-ids - [:audit-consumer-events :audit-consumer-discarded]) - -(defn- make-consumer-metrics - [] - (ig/init-key :app.metrics/metrics - {:default (select-keys main/default-metrics consumer-metric-ids)})) - -(defn- invoke-nexus! - "Invoke the nexus consumer handler in-process with an isolated - metrics registry. The handler takes a nil context because there is - no job row to describe (same as `th/run-task!`)." - [metrics params] - (let [job-def (ig/init-key :app.loggers.audit.consumer-task/nexus-job-def - {::db/pool (:app.db/pool th/*system*) - ::setup/shared-keys {:admin-console "ac-key" :nexus "nexus-key"} - ::http/client (:app.http.client/client th/*system*) - ::mtx/metrics metrics})] - ((::jobs/handler job-def) nil params))) - -(defn- invoke-admin-console! - "Invoke the admin-console consumer handler in-process with an - isolated metrics registry. The admin-console client is the (nil in - tests) system one: tests mock `app.nitrate/call`, so it is never - dereferenced." - [metrics] - (let [job-def (ig/init-key :app.loggers.audit.consumer-task/admin-console-job-def - {::db/pool (:app.db/pool th/*system*) - :app.nitrate/client (:app.nitrate/client th/*system*) - ::mtx/metrics metrics})] - ((::jobs/handler job-def) nil {}))) - -(defn- invoke-both! - "Invoke both consumer handlers sharing one isolated metrics - registry, Nexus first (the historical single-run order)." - [metrics params] - (invoke-nexus! metrics params) - (invoke-admin-console! metrics)) - -(defn- counter-value - [metrics id labels] - (let [collector (mtx/get-collector metrics id) - instance (::mdef/instance collector) - child (.labels ^Counter instance (into-array String labels))] - (.get ^Counter$Child child))) - -(t/deftest consumer-records-metrics-on-success - ;; Smoke test: the run records its domain metrics into the wired - ;; registry (isolated per test, pattern from `jobs-metrics-test`). - ;; Logging itself is not asserted: `dbg` summaries run below the - ;; test log level and failure `warn`s only repeat the outcome the - ;; metrics already prove. - (let [prof (th/create-profile* 1 {:is-active true}) - metrics (make-consumer-metrics)] - (with-redefs [cf/flags #{:nexus :admin-console}] - (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil} - http-mock {:target 'app.http.client/req :return {:status 204}}] - (insert-audit-row! (:id prof) "create-project") - (insert-audit-row! (:id prof) "unrelated-event") - (invoke-both! metrics {:uri archive-uri}) - (t/is (= 2.0 (counter-value metrics :audit-consumer-events ["nexus" "archived"]))) - (t/is (= 1.0 (counter-value metrics :audit-consumer-events ["admin-console" "archived"]))) - (t/is (= 1.0 (counter-value metrics :audit-consumer-events ["admin-console" "skipped"]))) - (t/is (= 0 (pending-count))) - (let [rows (th/db-exec! ["select * from audit_log"])] - (t/is (= 0 (count rows)))))))) - -;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; -;; PENDING-QUEUE TESTS -;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; - -(defn- full-event - "A complete event for submit*, which (unlike submit) assumes every - required field is filled." - [profile-id] - {:name "create-file" :type "action" :source "backend" - :profile-id profile-id :ip-addr "127.0.0.1" - :props {} :context {} - :created-at (ct/now) :tracked-at (ct/now)}) - -(t/deftest consumer-submit-creates-pendings-for-enabled-consumers - ;; The producer hook: a row is born pending for the enabled - ;; consumers, in the same transaction as the row itself. Goes - ;; through submit* because the suite stubs submit itself (see - ;; helpers/init-config). - (let [prof (th/create-profile* 1 {:is-active true})] - (with-redefs [cf/flags #{:audit-log :nexus :admin-console}] - (audit/submit* {::db/pool (:app.db/pool th/*system*)} - (full-event (:id prof))) - (t/is (= 1 (count (th/db-exec! ["select * from audit_log"])))) - (t/is (= 2 (pending-count)))))) - -(t/deftest consumer-submit-single-flag-creates-one-pending - (let [prof (th/create-profile* 1 {:is-active true})] - (with-redefs [cf/flags #{:audit-log :nexus}] - (audit/submit* {::db/pool (:app.db/pool th/*system*)} - (full-event (:id prof))) - (t/is (= 1 (count (th/db-exec! ["select * from audit_log"])))) - (t/is (= 1 (pending-count)))))) - -(t/deftest consumer-submit-rollback-keeps-no-pendings - ;; A rollback takes the row and its queue together: a row never - ;; outlives its pendings, and no phantom pending pins nothing. - (let [prof (th/create-profile* 1 {:is-active true})] - (with-redefs [cf/flags #{:audit-log :nexus :admin-console}] - (db/tx-run! {::db/pool (:app.db/pool th/*system*) - ::db/rollback true} - (fn [cfg] - (audit/submit* cfg (full-event (:id prof))) - (t/is (= 1 (count (db/exec! cfg ["select * from audit_log"])))) - (t/is (= 2 (count (db/exec! cfg ["select * from audit_log_pending"])))))) - (t/is (= 0 (count (th/db-exec! ["select * from audit_log"])))) - (t/is (= 0 (pending-count)))))) - -(t/deftest consumer-keeps-row-until-last-consumer-done - ;; Nexus refuses the chunk: the nexus job aborts (failed count - ;; recorded, isolated registry) while the admin-console job still - ;; drains its own queue. The row and the nexus pending stay; once - ;; nexus accepts, everything is gone. - (let [prof (th/create-profile* 1 {:is-active true}) - metrics (make-consumer-metrics)] - (with-redefs [cf/flags #{:nexus :admin-console}] - (insert-audit-row! (:id prof) "create-project") - (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil} - http-mock {:target 'app.http.client/req :return {:status 500}}] - (let [err (try - (invoke-nexus! metrics {:uri archive-uri}) - nil - (catch clojure.lang.ExceptionInfo cause cause))] - (t/is (= :consumer-chunk-refused (:code (ex-data err))))) - (invoke-admin-console! metrics) - (t/is (:called? @admin-console-mock)) - (t/is (= 1.0 (counter-value metrics :audit-consumer-events ["nexus" "failed"]))) - (t/is (= 1 (count (th/db-exec! ["select * from audit_log"])))) - (t/is (= 1 (pending-count)))) - (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil} - http-mock {:target 'app.http.client/req :return {:status 204}}] - (run-both! {:uri archive-uri}) - (t/is (= 0 (count (th/db-exec! ["select * from audit_log"])))) - (t/is (= 0 (pending-count))))))) - -(t/deftest consumer-sweeps-orphan-rows - ;; A row with no pendings (old archived leftover, or born while no - ;; consumer was on) is deleted without any send. - (let [prof (th/create-profile* 1 {:is-active true})] - (with-redefs [cf/flags #{:nexus :admin-console}] - (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil} - http-mock {:target 'app.http.client/req :return {:status 204}}] - ;; Raw insert, no pendings on purpose. - (th/db-insert! :audit-log - {:id (uuid/next) - :name "create-project" - :type "action" - :source "backend" - :profile-id (:id prof) - :ip-addr (db/inet "127.0.0.1") - :props (db/tjson {}) - :context (db/tjson {}) - :tracked-at (ct/now) - :created-at (ct/now)}) - (run-both! {:uri archive-uri}) - (t/is (false? (:called? @admin-console-mock))) - (t/is (false? (:called? @http-mock))) - (t/is (= 0 (count (th/db-exec! ["select * from audit_log"])))))))) - -(t/deftest consumer-nexus-transport-failure-records-failed - ;; A transport exception (timeout, DNS, refused) counts the events - ;; as failed and aborts the nexus job: the chunk rolls back and - ;; the rows stay for the next tick. - (let [prof (th/create-profile* 1 {:is-active true}) - metrics (make-consumer-metrics)] - (with-redefs [cf/flags #{:nexus}] - (with-mocks [http-mock {:target 'app.http.client/req - :throw (ex-info "timeout" {})}] - (insert-audit-row! (:id prof) "create-project") - (t/is (thrown? clojure.lang.ExceptionInfo - (invoke-nexus! metrics {:uri archive-uri}))) - (t/is (= 1.0 (counter-value metrics :audit-consumer-events ["nexus" "failed"]))) - (t/is (= 1 (count (th/db-exec! ["select * from audit_log"])))) - (t/is (= 1 (pending-count))))))) - -(t/deftest consumer-sends-chunks-in-chronological-order - ;; Nexus numbers by arrival for order-sensitive consumers, so - ;; multi-chunk drains must arrive oldest-first no matter how the - ;; queue is scanned. - (let [prof (th/create-profile* 1 {:is-active true}) - base (ct/now) - bodies (atom [])] - (with-redefs [cf/flags #{:nexus}] - (with-mocks [http-mock {:target 'app.http.client/req - :return (fn [_ params & _] - (swap! bodies conj (:body params)) - {:status 204})}] - (doseq [i (range 130)] - (let [row (insert-audit-row! (:id prof) (str "event-" i))] - (th/db-update! :audit-log - {:created-at (.plusSeconds ^java.time.Instant base (long i))} - {:id (:id row)}))) - (th/run-task! :audit-log-consumer-nexus {:uri archive-uri}) - (let [received (mapcat #(:events (transit/decode %)) @bodies) - times (map :created-at received)] - (t/is (= 130 (count received))) - (t/is (= (sort times) times)) - (t/is (= 0 (count (th/db-exec! ["select * from audit_log"]))))))))) - -(t/deftest consumer-reaps-disabled-consumer-pendings - ;; R3 reap by its real path: an admin-console pending born while - ;; admin-console was on is dropped once it is off; nexus still - ;; archives its queue and the row is gone with nothing stuck. The - ;; dropped pending counts as discarded (isolated registry). - (let [prof (th/create-profile* 1 {:is-active true}) - metrics (make-consumer-metrics)] - (with-redefs [cf/flags #{:nexus}] - (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}} - admin-console-mock {:target 'app.nitrate/call :return nil}] - (let [row (insert-audit-row! (:id prof) "create-project")] - ;; Admin console was on at insert time: both pendings on purpose. - (th/db-exec! ["INSERT INTO audit_log_pending (audit_log_id, consumer) - VALUES (?::uuid, 2) - ON CONFLICT DO NOTHING" - (:id row)]) - (t/is (= 2 (pending-count))) - (invoke-nexus! metrics {:uri archive-uri}) - (t/is (:called? @http-mock)) - (t/is (false? (:called? @admin-console-mock))) - (t/is (= 1.0 (counter-value metrics :audit-consumer-discarded ["reaped"]))) - (t/is (= 0 (count (th/db-exec! ["select * from audit_log"])))) - (t/is (= 0 (pending-count)))))))) - -(t/deftest consumer-sweeps-rows-born-with-no-consumer-on - ;; Accepted R3 consequence by its real path (not a hand-made - ;; orphan): ingested with no send flag on, the row is swept - ;; unarchived once the task runs, with zero sends. - (let [prof (th/create-profile* 1 {:is-active true}) - metrics (make-consumer-metrics)] - (with-redefs [cf/flags #{:audit-log}] - (audit/submit* {::db/pool (:app.db/pool th/*system*)} - (full-event (:id prof))) - (t/is (= 1 (count (th/db-exec! ["select * from audit_log"])))) - (t/is (= 0 (pending-count)))) - (with-redefs [cf/flags #{:audit-log :nexus}] - (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}] - (invoke-nexus! metrics {:uri archive-uri}) - (t/is (false? (:called? @http-mock))) - (t/is (= 0 (count (th/db-exec! ["select * from audit_log"])))) - (t/is (= 0 (pending-count))) - (t/is (= 1.0 (counter-value metrics :audit-consumer-discarded ["swept"]))) - (t/is (= 0.0 (counter-value metrics :audit-consumer-events ["nexus" "archived"]))))))) - -(t/deftest consumer-nexus-throw-still-drains-admin-console - ;; Consumer independence, now structural: a Nexus transport throw - ;; aborts only the nexus job, and the admin-console job still drains - ;; its own queue. Nexus keeps its pending (so the row stays); the - ;; admin console completes its own queue. - (let [prof (th/create-profile* 1 {:is-active true}) - metrics (make-consumer-metrics)] - (with-redefs [cf/flags #{:nexus :admin-console}] - (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil} - http-mock {:target 'app.http.client/req - :throw (ex-info "timeout" {})}] - (insert-audit-row! (:id prof) "create-project") - (t/is (thrown? clojure.lang.ExceptionInfo - (invoke-nexus! metrics {:uri archive-uri}))) - (invoke-admin-console! metrics) - (t/is (:called? @admin-console-mock)) - (t/is (= 1.0 (counter-value metrics :audit-consumer-events ["nexus" "failed"]))) - (t/is (= 1 (count (th/db-exec! ["select * from audit_log"])))) - (t/is (= 1 (pending-count))))))) - -(t/deftest consumer-reap-then-sweeps-without-sends - ;; End to end of the reap path: a row born with only an - ;; admin-console pending is reaped once :admin-console goes off, - ;; then swept with zero sends on either destination. - (let [prof (th/create-profile* 1 {:is-active true})] - (with-redefs [cf/flags #{:admin-console}] - (insert-audit-row! (:id prof) "create-project") - (t/is (= 1 (pending-count)))) - (with-redefs [cf/flags #{:nexus}] - (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}} - admin-console-mock {:target 'app.nitrate/call :return nil}] - (th/run-task! :audit-log-consumer-nexus {:uri archive-uri}) - (t/is (false? (:called? @http-mock))) - (t/is (false? (:called? @admin-console-mock))) - (t/is (= 0 (count (th/db-exec! ["select * from audit_log"])))) - (t/is (= 0 (pending-count))))))) - -(t/deftest consumer-fully-filtered-admin-console-chunk-counts-skipped - ;; An admin-console chunk with every event outside the allowlist - ;; never calls ingest, but still counts its rows as skipped and - ;; completes its queue. - (let [prof (th/create-profile* 1 {:is-active true}) - metrics (make-consumer-metrics)] - (with-redefs [cf/flags #{:nexus :admin-console}] - (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil} - http-mock {:target 'app.http.client/req :return {:status 204}}] - (insert-audit-row! (:id prof) "unrelated-event") - (invoke-both! metrics {:uri archive-uri}) - (t/is (false? (:called? @admin-console-mock))) - (t/is (= 1.0 (counter-value metrics :audit-consumer-events ["admin-console" "skipped"]))) - (t/is (= 0 (count (th/db-exec! ["select * from audit_log"])))) - (t/is (= 0 (pending-count))))))) diff --git a/backend/test/backend_tests/loggers_audit_consumers_test.clj b/backend/test/backend_tests/loggers_audit_consumers_test.clj new file mode 100644 index 0000000000..6c5a8f7541 --- /dev/null +++ b/backend/test/backend_tests/loggers_audit_consumers_test.clj @@ -0,0 +1,1479 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns backend-tests.loggers-audit-consumers-test + "The consumers of the audit log buffer: their cursors, the snapshot + cut that keeps an open transaction from being skipped, the transports + and the GC of the buffer." + (:require + [app.common.json :as json] + [app.common.time :as ct] + [app.common.transit :as transit] + [app.common.uri :as u] + [app.common.uuid :as uuid] + [app.config :as cf] + [app.db :as db] + [app.http.client :as-alias http] + [app.jobs :as jobs] + [app.loggers.audit :as audit] + [app.loggers.audit.consumers :as consumer] + [app.loggers.audit.consumers.admin-console :as consumer.admin-console] + [app.loggers.audit.consumers.nexus :as consumer.nexus] + [app.loggers.audit.gc :as gc] + [app.main :as main] + [app.metrics :as mtx] + [app.metrics.definition :as-alias mdef] + [app.rpc :as-alias rpc] + [app.rpc.commands.audit :as cmd-audit] + [app.setup :as-alias setup] + [backend-tests.helpers :as th] + [clojure.test :as t] + [cuerdas.core :as str] + [integrant.core :as ig] + [mockery.core :refer [with-mocks]] + [promesa.exec :as px]) + (:import + io.prometheus.client.Counter + io.prometheus.client.Counter$Child + io.prometheus.client.Gauge + io.prometheus.client.Gauge$Child)) + +(t/use-fixtures :once th/state-init) + +(def ^:private archive-uri "http://nexus.example/archive") + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; FIXTURES AND HELPERS +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(defn- seed-cursors + "Both cursors exist at (0, 0) before every test: the database reset + deletes them, so seeding keeps the starting state deterministic, lets + the GC delete without waiting for a first pass, and leaves the + 'no cursor' case to the tests that create it on purpose." + [next] + (doseq [consumer-id [consumer/nexus + consumer/admin-console]] + (th/db-exec! ["INSERT INTO audit_log_consumer_cursor (consumer, xid, seq) + VALUES (?::smallint, '0'::xid8, 0)" + consumer-id])) + (next)) + +(t/use-fixtures :each th/database-reset seed-cursors) + +(defn- cursor-of + [consumer-id] + (th/db-exec-one! ["SELECT xid::text AS xid, seq + FROM audit_log_consumer_cursor + WHERE consumer = ?::smallint" + consumer-id])) + +(defn- unread-count + "Rows after the cursor of a consumer, whatever their visibility." + [consumer-id] + (let [{:keys [xid seq]} (cursor-of consumer-id)] + (:cnt (th/db-exec-one! + ["SELECT count(*) AS cnt + FROM audit_log a + WHERE (a.xid, a.seq) > (?::xid8, ?::bigint)" + xid seq])))) + +(defn- insert-audit-row + ([profile-id name] + (insert-audit-row profile-id name {:team-id (uuid/next)})) + ([profile-id name props] + (insert-audit-row profile-id name props "backend")) + ([profile-id name props source] + (th/db-insert! :audit-log + {:id (uuid/next) + :name name + :type "action" + :source source + :profile-id profile-id + :ip-addr (db/inet "127.0.0.1") + :props (db/tjson props) + :context (db/tjson {}) + :tracked-at (ct/now) + :created-at (ct/now)}))) + +(defn- nexus-cfg + ([uri] (nexus-cfg uri (:app.metrics/metrics th/*system*))) + ([uri metrics] + (cond-> {::db/pool (:app.db/pool th/*system*) + ::setup/shared-keys {:admin-console "ac-key" :nexus "nexus-key"} + ::http/client (:app.http.client/client th/*system*) + ::mtx/metrics metrics} + uri (assoc ::consumer.nexus/uri uri)))) + +(defn- run-nexus + ([] (run-nexus {})) + ([params] + (consumer.nexus/run-pass + (nexus-cfg (:uri params))))) + +(defn- admin-console-cfg + [metrics] + {::db/pool (:app.db/pool th/*system*) + :app.nitrate/client (:app.nitrate/client th/*system*) + ::consumer.admin-console/uri-host "admin-console.example" + ::mtx/metrics metrics}) + +(defn- run-admin-console + [] + (consumer.admin-console/run-pass + (admin-console-cfg (:app.metrics/metrics th/*system*)))) + +(defn- run-gc + [] + (th/run-task! :audit-log-gc {})) + +(defn- admin-console-event + [admin-console-mock] + (let [[_ _ params] (:call-args @admin-console-mock)] + (first (:events params)))) + +(defn- admin-console-event-props + [admin-console-mock] + (:props (admin-console-event admin-console-mock))) + +(defn- insert-rows-in-one-transaction + "Insert `n` rows from a raw connection, in a transaction the caller + commits or rolls back. Returns the connection." + [conn prefix n] + (db/exec! conn ["BEGIN"]) + (db/exec! conn ["INSERT INTO audit_log (name, type, profile_id, created_at, tracked_at, + source, ip_addr, props, context) + SELECT ? || g, 'action', ?, now(), now(), + 'backend', '127.0.0.1', ?, ? + FROM generate_series(1, ?) AS g" + prefix uuid/zero (db/tjson {}) (db/tjson {}) n]) + conn) + +(defn- xid-pair-across-a-decimal-boundary + "Two xid values below the snapshot cut whose numeric order is the + opposite of their text order (1000 sorts before 999 as text), so a + text sort cannot produce the right answer by accident." + [] + (let [xmin (Long/parseLong + (:xmin (th/db-exec-one! + ["SELECT pg_snapshot_xmin(pg_current_snapshot())::text AS xmin"]))) + p (long (Math/pow 10 (Math/floor (Math/log10 (double xmin)))))] + (if (< p xmin) + [(dec p) p] + [(dec (quot p 10)) (quot p 10)]))) + +(defn- insert-audit-row-at + "Insert one audit row at an explicit `(xid, seq)` position: the tests + that pin the numeric order of the cursor place rows across a decimal + boundary, as a real stream does when the transaction counter grows." + [xid seq] + (let [id (uuid/next)] + (th/db-exec! ["INSERT INTO audit_log (id, name, type, source, profile_id, + ip_addr, props, context, tracked_at, + created_at, xid, seq) + VALUES (?, 'create-project', 'action', 'backend', ?, + '127.0.0.1', ?, ?, now(), now(), ?::xid8, ?::bigint)" + id uuid/zero (db/tjson {}) (db/tjson {}) (str xid) seq]) + id)) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; NEXUS: THE WHOLE STREAM FROM ITS CURSOR +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(t/deftest consumer-nexus-drains-from-its-cursor + (let [prof (th/create-profile* 1 {:is-active true})] + (with-redefs [cf/flags #{:nexus}] + (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}] + (insert-audit-row (:id prof) "create-project") + (insert-audit-row (:id prof) "create-file") + (t/is (= 2 (unread-count consumer/nexus))) + (run-nexus {:uri archive-uri}) + (t/is (:called? @http-mock)) + (t/is (= 1 (:call-count @http-mock))) + (let [events (-> @http-mock :call-args second :body transit/decode :events)] + (t/is (= ["create-project" "create-file"] (mapv :name events)))) + ;; The cursor advanced; the rows stay until the GC removes them. + (t/is (= 0 (unread-count consumer/nexus))) + (t/is (= 2 (count (th/db-exec! ["SELECT * FROM audit_log"])))) + ;; Nobody else moved: each consumer has its own position. + (t/is (= 2 (unread-count consumer/admin-console))))))) + +(t/deftest consumer-nexus-sends-in-insertion-order + (let [prof (th/create-profile* 2 {:is-active true}) + names (mapv #(str "event-" %) (range 130))] + (with-redefs [cf/flags #{:nexus}] + (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}] + (doseq [name names] + (insert-audit-row (:id prof) name)) + (run-nexus {:uri archive-uri}) + (let [bodies (map #(transit/decode (:body (second %))) (:call-args-list @http-mock)) + events (mapcat :events bodies)] + (t/is (= 2 (count bodies))) + (t/is (= names (mapv :name events))) + (t/is (= 0 (unread-count consumer/nexus)))))))) + +(t/deftest consumer-nexus-chunks-by-the-configured-chunk-size + ;; `chunk-size` is a var so a test can shrink it: the loop must honour + ;; it and still send every event, in order, however many chunks it takes. + (let [prof (th/create-profile* 4 {:is-active true}) + names (mapv #(str "small-" %) (range 1 8))] + (with-redefs [cf/flags #{:nexus} + consumer/chunk-size 3] + (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}] + (doseq [name names] + (insert-audit-row (:id prof) name)) + (run-nexus {:uri archive-uri}) + (let [bodies (map #(transit/decode (:body (second %))) (:call-args-list @http-mock)) + events (mapcat :events bodies)] + (t/is (= [3 3 1] (mapv #(count (:events %)) bodies))) + (t/is (= names (mapv :name events)))) + (t/is (= 0 (unread-count consumer/nexus))))))) + +(t/deftest consumer-nexus-default-uri-appends-api-audit + (let [prof (th/create-profile* 3 {:is-active true}) + uris (atom [])] + (with-redefs [cf/flags #{:nexus} + cf/config (assoc cf/config :nexus-uri (u/uri "http://nexus.example"))] + (with-mocks [http-mock {:target 'app.http.client/req + :return (fn [_ params & _] + (swap! uris conj (str (:uri params))) + {:status 204})}] + (insert-audit-row (:id prof) "create-project") + (run-nexus {}) + (t/is (= ["http://nexus.example/api/audit"] @uris)))))) + +(t/deftest consumer-nexus-component-requires-uri-when-flag-set + (with-redefs [cf/flags #{:nexus} + cf/config (dissoc cf/config :nexus-uri)] + (let [data (ex-data (try + (ig/init-key :app.loggers.audit.consumers.nexus/consumer + (nexus-cfg nil)) + (catch Throwable cause + cause)))] + (t/is (= :task-not-configured (:code data)))))) + +(t/deftest consumer-nexus-component-does-not-start-with-the-flag-off + (with-redefs [cf/flags #{}] + (t/is (nil? (ig/init-key :app.loggers.audit.consumers.nexus/consumer + (nexus-cfg archive-uri)))))) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; ADMIN CONSOLE: ITS ALLOWLIST, ITS OWN FILTER +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(t/deftest consumer-admin-console-component-requires-uri-when-flag-set + ;; Without a destination the consumer can never advance its cursor, and + ;; the GC waits for it: fail at boot instead of growing the buffer. + (with-redefs [cf/flags #{:admin-console} + cf/config (dissoc cf/config :admin-console-uri)] + (let [data (ex-data (try + (ig/init-key :app.loggers.audit.consumers.admin-console/consumer + (admin-console-cfg nil)) + (catch Throwable cause + cause)))] + (t/is (= :task-not-configured (:code data)))))) + +(t/deftest consumer-admin-console-filters-allowlist-and-shadows + (let [prof (th/create-profile* 10 {:is-active true})] + (with-redefs [cf/flags #{:admin-console}] + (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil} + http-mock {:target 'app.http.client/req :return {:status 204}}] + (insert-audit-row (:id prof) "create-project") + (insert-audit-row (:id prof) "unrelated-event") + (insert-audit-row (:id prof) "create-file" {:team-id (uuid/next)} "telemetry:backend") + (insert-audit-row (:id prof) "create-file" {:team-id (uuid/next)} "backend") + (run-admin-console) + (t/is (:called? @admin-console-mock)) + (t/is (false? (:called? @http-mock))) + (let [[_ _ params] (:call-args @admin-console-mock)] + (t/is (= ["create-project" "create-file"] + (mapv :name (:events params))))) + ;; Every row was passed over: the skipped ones and the ones sent. + (t/is (= 0 (unread-count consumer/admin-console))))))) + +(t/deftest consumer-admin-console-forwards-organization-attribute-events + ;; The allowlist dropped these two names during the rebase (upstream + ;; #12861 and #12932 added them): an event outside it counts as skipped + ;; and the GC deletes the row, so the admin console never sees the + ;; attribute change. + (let [prof (th/create-profile* 13 {:is-active true})] + (with-redefs [cf/flags #{:admin-console}] + (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}] + (insert-audit-row (:id prof) "create-organization-attribute") + (insert-audit-row (:id prof) "delete-organization-attribute") + (insert-audit-row (:id prof) "update-organization-attribute") + (insert-audit-row (:id prof) "unrelated-event") + (run-admin-console) + (t/is (:called? @admin-console-mock)) + (let [[_ _ params] (:call-args @admin-console-mock)] + (t/is (= ["create-organization-attribute" + "delete-organization-attribute" + "update-organization-attribute"] + (mapv :name (:events params))))) + (t/is (= 0 (unread-count consumer/admin-console))))))) + +(t/deftest consumer-admin-console-advances-over-non-matching-rows + ;; A selective consumer that finds nothing must still move: otherwise + ;; the GC would wait for it forever and the buffer would grow. + (let [prof (th/create-profile* 11 {:is-active true})] + (with-redefs [cf/flags #{:admin-console}] + (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}] + (dotimes [i 200] + (insert-audit-row (:id prof) (str "unrelated-" i))) + (run-admin-console) + (t/is (false? (:called? @admin-console-mock))) + (t/is (= 0 (unread-count consumer/admin-console))))))) + +(t/deftest consumer-admin-console-error-still-lets-nexus-drain + (let [prof (th/create-profile* 12 {:is-active true})] + (with-redefs [cf/flags #{:nexus :admin-console}] + (with-mocks [admin-console-mock {:target 'app.nitrate/call + :throw (ex-info "nitrate down" {})} + http-mock {:target 'app.http.client/req :return {:status 204}}] + (insert-audit-row (:id prof) "create-project") + (run-nexus {:uri archive-uri}) + (t/is (thrown? clojure.lang.ExceptionInfo + (run-admin-console))) + (t/is (= 0 (unread-count consumer/nexus))) + (t/is (= 1 (unread-count consumer/admin-console))))))) + +(t/deftest consumer-admin-console-component-does-not-start-with-the-flag-off + (with-redefs [cf/flags #{}] + (t/is (nil? (ig/init-key :app.loggers.audit.consumers.admin-console/consumer + (admin-console-cfg (:app.metrics/metrics th/*system*))))))) + +(t/deftest consumer-admin-console-encodes-db-rows-for-admin-console + (let [prof (th/create-profile* 13 {:is-active true})] + (with-redefs [cf/flags #{:admin-console} + cf/config (assoc cf/config + :admin-console-uri "http://ac.example/admin-console/" + :admin-console-shared-key "ac-key" + :nexus-shared-key "nexus-key")] + (let [admin-console-bodies (atom []) + shared-keys {:admin-console "ac-key" :nexus "nexus-key"} + client (ig/init-key :app.nitrate/client + {::http/client (Object.) + ::setup/shared-keys shared-keys}) + consumer-cfg {::db/pool (:app.db/pool th/*system*) + ::mtx/metrics (th/metrics-stub) + :app.nitrate/client client}] + (with-mocks [http-mock {:target 'app.http.client/req + :return + (fn [_ req & _] + (when (str/includes? (str (:uri req)) "api/audit-log") + (swap! admin-console-bodies conj (:body req))) + {:status 204})}] + (let [team-id (uuid/next) + row (insert-audit-row (:id prof) "create-project" + {:team-id team-id})] + (consumer.admin-console/run-pass consumer-cfg) + (t/is (= 1 (count @admin-console-bodies))) + (let [body (json/decode (first @admin-console-bodies) :key-fn json/read-kebab-key) + event (first (:events body))] + (t/is (= 1 (count (:events body)))) + (t/is (= (str (:id row)) (str (:id event)))) + (t/is (= "create-project" (:name event))) + (t/is (= "127.0.0.1" (:ip-addr event))) + (t/is (string? (:created-at event))) + (t/is (string? (:tracked-at event))) + (t/is (= (str team-id) (str (get-in event [:props :team-id]))))) + (t/is (= 0 (unread-count consumer/admin-console))))))))) + +(t/deftest consumer-admin-console-accepts-a-2xx-response + ;; The ingest endpoint answers 204 by contract, but a proxy may normalize + ;; it to another 2xx. Any 2xx is success: the chunk commits and the cursor + ;; advances instead of freezing on a :malli.core/invalid-schema. + (let [prof (th/create-profile* 75 {:is-active true})] + (with-redefs [cf/flags #{:admin-console} + cf/config (assoc cf/config + :admin-console-uri "http://ac.example/admin-console/" + :admin-console-shared-key "ac-key" + :nexus-shared-key "nexus-key")] + (let [shared-keys {:admin-console "ac-key" :nexus "nexus-key"} + client (ig/init-key :app.nitrate/client + {::http/client (Object.) + ::setup/shared-keys shared-keys}) + consumer-cfg {::db/pool (:app.db/pool th/*system*) + ::mtx/metrics (th/metrics-stub) + :app.nitrate/client client}] + (with-mocks [http-mock {:target 'app.http.client/req + :return {:status 200 :body "{}"}}] + (insert-audit-row (:id prof) "create-project") + (consumer.admin-console/run-pass consumer-cfg) + (t/is (= 1 (:call-count @http-mock))) + (t/is (= 0 (unread-count consumer/admin-console)))))))) + +(t/deftest consumer-admin-console-enriches-team-name-and-emails + (let [actor (th/create-profile* 14 {:is-active true}) + member (th/create-profile* 15 {:is-active true}) + team (th/create-team* 14 {:profile-id (:id actor)})] + (with-redefs [cf/flags #{:admin-console}] + (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}] + (insert-audit-row (:id actor) "create-project" + {:team-id (:id team) + :profile-id (:id actor) + :user-id (:id member) + :member-id (:id member) + ;; Producers store this as a string UUID. + :user-who-send-invitation (str (:id actor))}) + (run-admin-console) + (let [event (admin-console-event admin-console-mock) + props (:props event)] + (t/is (= (:name team) (:team-name props))) + (t/is (= (:email actor) (:profile-email props))) + (t/is (= (:email member) (:user-email props))) + (t/is (= (:email member) (:member-email props))) + (t/is (= (:email actor) (:user-who-send-invitation-email props))) + ;; Top-level actor fields (event :profile-id), distinct from props. + (t/is (= (:email actor) (:profile-email event))) + (t/is (= (:fullname actor) (:profile-name event)))))))) + +(t/deftest consumer-admin-console-enriches-soft-deleted-team-and-profile + (let [actor (th/create-profile* 16 {:is-active true}) + other (th/create-profile* 17 {:is-active true}) + team (th/create-team* 16 {:profile-id (:id actor)})] + (th/db-update! :team {:deleted-at (ct/now)} {:id (:id team)}) + (th/db-update! :profile {:deleted-at (ct/now)} {:id (:id other)}) + (with-redefs [cf/flags #{:admin-console}] + (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}] + (insert-audit-row (:id actor) "create-project" + {:team-id (:id team) + :profile-id (:id other)}) + (run-admin-console) + (let [event (admin-console-event admin-console-mock) + props (:props event)] + (t/is (= (:name team) (:team-name props))) + (t/is (= (:email other) (:profile-email props))) + (t/is (= (:email actor) (:profile-email event))) + (t/is (= (:fullname actor) (:profile-name event)))))))) + +(t/deftest consumer-admin-console-enriches-soft-deleted-actor-profile + (let [actor (th/create-profile* 70 {:is-active true}) + team (th/create-team* 70 {:profile-id (:id actor)})] + (th/db-update! :profile {:deleted-at (ct/now)} {:id (:id actor)}) + (with-redefs [cf/flags #{:admin-console}] + (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}] + (insert-audit-row (:id actor) "create-project" + {:team-id (:id team)}) + (run-admin-console) + (let [event (admin-console-event admin-console-mock)] + (t/is (= (:email actor) (:profile-email event))) + (t/is (= (:fullname actor) (:profile-name event)))))))) + +(t/deftest consumer-admin-console-omits-actor-profile-when-no-usable-id + (let [actor (th/create-profile* 71 {:is-active true}) + team (th/create-team* 71 {:profile-id (:id actor)}) + missing (uuid/next)] + (with-redefs [cf/flags #{:admin-console}] + (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}] + (insert-audit-row uuid/zero "create-project" {:team-id (:id team)}) + (insert-audit-row missing "create-team" {:team-id (:id team)}) + (run-admin-console) + (t/is (:called? @admin-console-mock)) + (let [[_ _ params] (:call-args @admin-console-mock) + events (:events params) + by-name (into {} (map (juxt :name identity) events)) + zero-evt (get by-name "create-project") + missing-evt (get by-name "create-team")] + (t/is (not (contains? zero-evt :profile-email))) + (t/is (not (contains? zero-evt :profile-name))) + (t/is (not (contains? missing-evt :profile-email))) + (t/is (not (contains? missing-evt :profile-name)))))))) + +(t/deftest consumer-admin-console-does-not-overwrite-existing-enrich-props + (let [actor (th/create-profile* 18 {:is-active true}) + team (th/create-team* 18 {:profile-id (:id actor)})] + (with-redefs [cf/flags #{:admin-console}] + (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}] + (insert-audit-row (:id actor) "create-project" + {:team-id (:id team) + :team-name "already-set-name" + :profile-id (:id actor) + :profile-email "already-set@example.com" + :user-id (:id actor) + :user-email "already-user@example.com" + :member-id (:id actor) + :member-email "already-member@example.com" + :user-who-send-invitation (str (:id actor)) + :user-who-send-invitation-email "already-inviter@example.com"}) + (run-admin-console) + (let [props (admin-console-event-props admin-console-mock)] + (t/is (= "already-set-name" (:team-name props))) + (t/is (= "already-set@example.com" (:profile-email props))) + (t/is (= "already-user@example.com" (:user-email props))) + (t/is (= "already-member@example.com" (:member-email props))) + (t/is (= "already-inviter@example.com" + (:user-who-send-invitation-email props)))))))) + +(t/deftest consumer-admin-console-does-not-overwrite-existing-actor-profile-fields + ;; Top-level :profile-email / :profile-name are not stored on the row; + ;; exercise fill-only via the enrich helper directly. + (let [actor (th/create-profile* 72 {:is-active true}) + event {:profile-id (:id actor) + :profile-email "preset@example.com" + :profile-name "Preset Name" + :props {}}] + (db/tx-run! th/*system* + (fn [cfg] + (let [enriched (first (#'consumer.admin-console/enrich-events + cfg [event]))] + (t/is (= "preset@example.com" (:profile-email enriched))) + (t/is (= "Preset Name" (:profile-name enriched)))))))) + +(t/deftest consumer-admin-console-fills-missing-actor-profile-field-independently + ;; Each top-level key fills on its own; guards an or→and regression in + ;; collect-actor-ids-needing-profile. + (let [actor (th/create-profile* 73 {:is-active true})] + (db/tx-run! th/*system* + (fn [cfg] + (let [email-only (first (#'consumer.admin-console/enrich-events + cfg [{:profile-id (:id actor) + :profile-email "preset@example.com" + :props {}}])) + name-only (first (#'consumer.admin-console/enrich-events + cfg [{:profile-id (:id actor) + :profile-name "Preset Name" + :props {}}]))] + (t/is (= "preset@example.com" (:profile-email email-only))) + (t/is (= (:fullname actor) (:profile-name email-only))) + (t/is (= (:email actor) (:profile-email name-only))) + (t/is (= "Preset Name" (:profile-name name-only)))))))) + +(t/deftest consumer-admin-console-omits-empty-actor-profile-name + ;; profile.fullname is NOT NULL DEFAULT ''; empty must not become + ;; :profile-name "". + (let [actor (th/create-profile* 74 {:is-active true}) + team (th/create-team* 74 {:profile-id (:id actor)})] + (th/db-update! :profile {:fullname ""} {:id (:id actor)}) + (with-redefs [cf/flags #{:admin-console}] + (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}] + (insert-audit-row (:id actor) "create-project" {:team-id (:id team)}) + (run-admin-console) + (let [event (admin-console-event admin-console-mock)] + (t/is (= (:email actor) (:profile-email event))) + (t/is (not (contains? event :profile-name)))))))) + +(t/deftest consumer-admin-console-enrich-skips-missing-and-non-uuid-ids + (let [actor (th/create-profile* 19 {:is-active true})] + (with-redefs [cf/flags #{:admin-console}] + (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}] + (insert-audit-row (:id actor) "create-project" + {:team-id (uuid/next) + :profile-id (uuid/next) + :user-id "not-a-uuid" + :member-id 42}) + (run-admin-console) + (let [props (admin-console-event-props admin-console-mock)] + (t/is (nil? (:team-name props))) + (t/is (nil? (:profile-email props))) + (t/is (nil? (:user-email props))) + (t/is (nil? (:member-email props))) + (t/is (= "not-a-uuid" (:user-id props))) + (t/is (= 42 (:member-id props)))))))) + +(t/deftest consumer-admin-console-forwards-export-job-not-legacy-export + (let [actor (th/create-profile* 20 {:is-active true}) + team (th/create-team* 20 {:profile-id (:id actor)})] + (with-redefs [cf/flags #{:admin-console}] + (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}] + (insert-audit-row (:id actor) "create-export-binfile-job" + {:team-id (:id team) + :job-id (uuid/next) + :files 1 + :export-type "detach-libraries"}) + (insert-audit-row (:id actor) "export-binfile" + {:file-id (uuid/next)}) + (run-admin-console) + (let [[_ _ params] (:call-args @admin-console-mock) + events (:events params) + props (:props (first events))] + (t/is (= ["create-export-binfile-job"] (mapv :name events))) + (t/is (= (:id team) (:team-id props))) + (t/is (= (:name team) (:team-name props)))))))) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; THE SNAPSHOT CUT: A ROW OF AN OPEN TRANSACTION IS NEVER SKIPPED +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(t/deftest cursor-does-not-pass-an-open-transaction + (let [prof (th/create-profile* 30 {:is-active true})] + (with-redefs [cf/flags #{:nexus}] + (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}] + (with-open [conn (db/open th/*system*)] + ;; A inserts and keeps the transaction open. + (db/exec! conn ["BEGIN"]) + (db/exec! conn ["INSERT INTO audit_log (name, type, profile_id, created_at, tracked_at, + source, ip_addr, props, context) + VALUES ('tx-A', 'action', ?, now(), now(), + 'backend', '127.0.0.1', ?, ?)" + (:id prof) (db/tjson {}) (db/tjson {})]) + ;; B inserts and commits after A, with a higher xid. + (insert-audit-row (:id prof) "tx-B") + (run-nexus {:uri archive-uri}) + ;; A is the oldest live transaction: the cut leaves out B too. + (t/is (false? (:called? @http-mock))) + (t/is (= 0 (:seq (cursor-of consumer/nexus)))) + (db/exec! conn ["COMMIT"])) + ;; A is committed: both rows are visible, in xid order. + (run-nexus {:uri archive-uri}) + (let [events (-> @http-mock :call-args second :body transit/decode :events)] + (t/is (= ["tx-A" "tx-B"] (mapv :name events)))) + (t/is (= 0 (unread-count consumer/nexus))))))) + +(t/deftest cursor-processes-a-large-transaction-in-chunks-without-loss + (let [prof (th/create-profile* 31 {:is-active true}) + names (mapv #(str "lote-" %) (range 1 301))] + (with-redefs [cf/flags #{:nexus}] + (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}] + (with-open [conn (db/open th/*system*)] + (insert-rows-in-one-transaction conn "lote-" 300) + ;; While the transaction is open nothing is visible. + (run-nexus {:uri archive-uri}) + (t/is (false? (:called? @http-mock))) + (t/is (= 0 (:seq (cursor-of consumer/nexus)))) + (db/exec! conn ["COMMIT"])) + ;; After the commit the 300 rows are read in 128-row chunks, + ;; and the seq within the transaction keeps their order. + (run-nexus {:uri archive-uri}) + (let [bodies (map #(transit/decode (:body (second %))) + (:call-args-list @http-mock)) + events (mapcat :events bodies)] + (t/is (= [128 128 44] (mapv #(count (:events %)) bodies))) + (t/is (= names (mapv :name events)))) + (t/is (= 0 (unread-count consumer/nexus))))))) + +(t/deftest cursor-lock-serializes-two-runs-of-the-same-consumer + (let [prof (th/create-profile* 32 {:is-active true})] + (with-redefs [cf/flags #{:nexus}] + (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}] + (insert-audit-row (:id prof) "create-project") + (with-open [conn (db/open th/*system*)] + (db/exec! conn ["BEGIN"]) + (db/exec! conn ["SELECT consumer FROM audit_log_consumer_cursor + WHERE consumer = ?::smallint FOR UPDATE" + consumer/nexus]) + (let [system th/*system* + job (future (binding [th/*system* system] + (run-nexus {:uri archive-uri})))] + (Thread/sleep 500) + ;; The run is blocked on the cursor: it never sent. + (t/is (false? (:called? @http-mock))) + (db/exec! conn ["COMMIT"]) + (t/is (not= :timeout (deref job 10000 :timeout)))) + (t/is (:called? @http-mock)) + (t/is (= 0 (unread-count consumer/nexus)))))))) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; THE TICK: ONE PASS ACROSS INSTANCES, WITH A FRONTIER +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(defn- sent-event-names + "The event names of every batch the http mock received, in order." + [http-mock] + (->> (:call-args-list @http-mock) + (mapcat (comp :events transit/decode :body second)) + (mapv :name))) + +(defn- hold-tick + "Take the tick lock of a consumer on a connection, as another instance + would, answering the [classid objid] pair the caller releases with." + [conn consumer-id] + (let [[classid objid] (#'consumer/tick-lock-key consumer-id)] + (assert (true? (db/session-try-lock conn classid objid))) + [classid objid])) + +(defn- release-tick + [conn [classid objid]] + (db/session-unlock conn classid objid)) + +(t/deftest consumer-tick-skips-when-another-instance-holds-it + (let [prof (th/create-profile* 33 {:is-active true})] + (with-redefs [cf/flags #{:nexus}] + (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}] + (insert-audit-row (:id prof) "create-project") + (with-open [conn (db/open th/*system*)] + (let [tick (hold-tick conn consumer/nexus)] + ;; Another instance is draining: this pass does nothing. + (t/is (= ::consumer/busy (run-nexus {:uri archive-uri}))) + (t/is (false? (:called? @http-mock))) + (t/is (= 0 (:seq (cursor-of consumer/nexus)))) + (t/is (true? (release-tick conn tick))))) + ;; With the tick free, the same pass drains. + (run-nexus {:uri archive-uri}) + (t/is (:called? @http-mock)) + (t/is (= 0 (unread-count consumer/nexus))))))) + +(t/deftest consumer-tick-busy-records-only-the-pass-counter + (let [prof (th/create-profile* 34 {:is-active true}) + recorded (atom [])] + (with-redefs [cf/flags #{:nexus} + consumer/record-count (fn [_ id labels amount] + (swap! recorded conj [id labels amount]))] + (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}] + (insert-audit-row (:id prof) "create-project") + (with-open [conn (db/open th/*system*)] + (let [tick (hold-tick conn consumer/nexus)] + (t/is (= ::consumer/busy (run-nexus {:uri archive-uri}))) + (release-tick conn tick))) + (t/is (= [[:audit-consumer-passes ["nexus" "busy"] 1]] @recorded)) + (t/is (false? (:called? @http-mock))))))) + +(t/deftest consumer-tick-never-collides-with-a-file-lock + ;; `update-file` takes a one-key advisory lock from the hash of the file + ;; id; the consumers take a two-key one, a different lock space, so a + ;; file lock with the same numbers cannot block a pass. + (let [prof (th/create-profile* 36 {:is-active true}) + [classid objid] (#'consumer/tick-lock-key consumer/nexus) + bigint-key (+ (* (long classid) 4294967296) objid)] + (with-redefs [cf/flags #{:nexus}] + (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}] + (insert-audit-row (:id prof) "create-project") + (with-open [conn (db/open th/*system*)] + (t/is (true? (:lock (db/exec-one! conn + ["select pg_try_advisory_lock(?::bigint) as lock" + bigint-key])))) + (t/is (not= ::consumer/busy (run-nexus {:uri archive-uri}))) + (t/is (:called? @http-mock)) + (db/exec-one! conn ["select pg_advisory_unlock(?::bigint)" bigint-key])))))) + +(defn- make-pool + "A dedicated pool, to pin how many connections a pass needs." + ([max-size] (make-pool max-size {})) + ([max-size extra] + (ig/init-key :app.db/pool + (merge {::db/uri (:database-uri th/config) + ::db/username (:database-username th/config) + ::db/password (:database-password th/config) + ::db/max-size max-size + ::db/name :test-consumers + ::mtx/metrics (ig/init-key :app.metrics/metrics {:default {}})} + extra)))) + +(defn- run-nexus-on + "Run one nexus pass merging `overrides` over the test cfg." + [overrides] + (consumer.nexus/run-pass + (merge (nexus-cfg archive-uri) overrides))) + +(t/deftest consumer-pass-needs-one-connection + ;; The frontier, every chunk transaction and the lag gauge run on the + ;; connection that holds the tick: a pool of one is enough. + (let [prof (th/create-profile* 37 {:is-active true}) + pool (make-pool 1)] + (try + (with-redefs [cf/flags #{:nexus}] + (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}] + (with-open [conn (db/open th/*system*)] + (insert-rows-in-one-transaction conn "lote-" 300) + (db/exec! conn ["COMMIT"])) + (run-nexus-on {::db/pool pool}) + (let [bodies (map #(transit/decode (:body (second %))) + (:call-args-list @http-mock))] + (t/is (= [128 128 44] (mapv #(count (:events %)) bodies)))) + (t/is (= 0 (unread-count consumer/nexus))))) + (finally + (ig/halt-key! :app.db/pool pool))))) + +(t/deftest consumer-pass-clears-a-stale-tick-on-the-borrowed-connection + ;; A pass that could not release the tick leaves it on its pooled + ;; connection; the next pass borrows that connection, clears whatever it + ;; carries and drains. + (let [prof (th/create-profile* 38 {:is-active true}) + pool (make-pool 1)] + (try + (with-redefs [cf/flags #{:nexus}] + (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}] + (insert-audit-row (:id prof) "create-project") + (with-open [conn (db/open pool)] + (let [[classid objid] (#'consumer/tick-lock-key consumer/nexus)] + (t/is (true? (db/session-try-lock conn classid objid))))) + (run-nexus-on {::db/pool pool}) + (t/is (:called? @http-mock)) + (t/is (= 0 (unread-count consumer/nexus))))) + (finally + (ig/halt-key! :app.db/pool pool))))) + +(t/deftest consumer-without-a-cursor-starts-at-the-beginning + ;; The migration does not seed cursors: a consumer with no row creates + ;; one at (0, 0), so its first pass processes the buffer that is already + ;; there instead of skipping the history. + (let [prof (th/create-profile* 45 {:is-active true})] + (with-redefs [cf/flags #{:nexus}] + (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}] + (insert-audit-row (:id prof) "create-project") + (insert-audit-row (:id prof) "create-file") + (th/db-exec! ["DELETE FROM audit_log_consumer_cursor + WHERE consumer = ?::smallint" + consumer/nexus]) + (run-nexus {:uri archive-uri}) + (t/is (:called? @http-mock)) + (t/is (some? (cursor-of consumer/nexus))) + (t/is (= 0 (unread-count consumer/nexus))))))) + +(t/deftest consumer-pass-with-its-flag-off-sends-nothing + ;; `run-pass` is public and its docstring invites direct calls. The loop + ;; is gated at boot, so the pass checks the flag too: with it off nothing + ;; is sent and the cursor is neither created nor advanced. + (let [prof (th/create-profile* 46 {:is-active true})] + (with-redefs [cf/flags #{}] + (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}] + (insert-audit-row (:id prof) "create-project") + (th/db-exec! ["DELETE FROM audit_log_consumer_cursor + WHERE consumer = ?::smallint" + consumer/nexus]) + (t/is (= ::consumer/disabled (run-nexus {:uri archive-uri}))) + (t/is (false? (:called? @http-mock))) + (t/is (nil? (cursor-of consumer/nexus))))))) + +(t/deftest drain-consumer-rejects-an-unknown-id + ;; The flag lookup answers false for an id outside the registry, so the + ;; unknown-consumer raise must happen first: a typo must fail, not silently + ;; disable the pass. + (let [data (try + (consumer/drain-consumer (nexus-cfg nil) 999 (fn [_ _] nil)) + nil + (catch clojure.lang.ExceptionInfo e (ex-data e)))] + (t/is (= :internal (:type data))) + (t/is (= :unknown-audit-consumer (:code data))) + (t/is (= 999 (:consumer data))))) + +(t/deftest admin-console-pass-with-its-flag-off-sends-nothing + ;; `nitrate/call` returns nil when `:admin-console` is off, so without the + ;; gate the pass would count the nil as success and advance the cursor + ;; over events that never reached the destination. + (let [prof (th/create-profile* 47 {:is-active true})] + (th/with-audit-config #{:nexus} + (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}] + (insert-audit-row (:id prof) "create-project") + (th/db-exec! ["DELETE FROM audit_log_consumer_cursor + WHERE consumer = ?::smallint" + consumer/admin-console]) + (t/is (= ::consumer/disabled (run-admin-console))) + (t/is (false? (:called? @http-mock))) + (t/is (nil? (cursor-of consumer/admin-console))))))) + +(t/deftest consumer-pass-stops-at-the-watermark + ;; A pass drains up to the newest row that existed when it started: a + ;; row committed while the pass runs is left for the next one, so the + ;; pass ends instead of chasing a tail of tiny chunks for ever. + (let [prof (th/create-profile* 35 {:is-active true}) + arrived (atom 0)] + (with-redefs [cf/flags #{:nexus}] + (with-mocks [http-mock {:target 'app.http.client/req + :return (fn [_ _ & _] + (when (= 1 (swap! arrived inc)) + (insert-audit-row (:id prof) "mid-pass")) + {:status 204})}] + (insert-audit-row (:id prof) "before-1") + (insert-audit-row (:id prof) "before-2") + (run-nexus {:uri archive-uri}) + (t/is (= 1 (:call-count @http-mock))) + (t/is (= ["before-1" "before-2"] (sent-event-names http-mock))) + ;; The row of the pass in course waits for the next one. + (t/is (= 1 (unread-count consumer/nexus))) + (run-nexus {:uri archive-uri}) + (t/is (= 2 (:call-count @http-mock))) + (t/is (= ["before-1" "before-2" "mid-pass"] (sent-event-names http-mock))) + (t/is (= 0 (unread-count consumer/nexus))))))) + +(t/deftest consumer-loop-runs-passes-until-interrupted + (let [passes (atom 0)] + (with-redefs-fn {#'consumer/sleep-ms 10} + (fn [] + (let [thread (consumer/start-loop + {} + consumer/nexus + (fn [_] (swap! passes inc)))] + (try + (loop [i 0] + (when (and (< @passes 3) (< i 300)) + (Thread/sleep 5) + (recur (inc i)))) + (t/is (<= 3 @passes)) + (finally + (px/interrupt! thread))) + (.join ^Thread thread 5000) + (t/is (false? (.isAlive ^Thread thread)))))))) + +(t/deftest consumer-loop-survives-a-failing-pass + (let [passes (atom 0)] + (with-redefs-fn {#'consumer/sleep-ms 10} + (fn [] + (let [thread (consumer/start-loop + {} + consumer/nexus + (fn [_] + (swap! passes inc) + (throw (ex-info "boom" {}))))] + (try + (loop [i 0] + (when (and (< @passes 3) (< i 300)) + (Thread/sleep 5) + (recur (inc i)))) + (t/is (<= 3 @passes)) + (finally + (px/interrupt! thread))) + (.join ^Thread thread 5000) + (t/is (false? (.isAlive ^Thread thread)))))))) + +(t/deftest consumer-component-starts-a-loop-that-halt-stops + ;; The component is the only door to the loop: it starts a pass, keeps + ;; running it and `halt-component` stops the thread. The pass is a fake + ;; on purpose: a real drain here would keep the shared database busy in + ;; the background while the rest of the namespace runs, which is what + ;; made this suite alternate. The drain itself is covered by the + ;; `run-pass` tests. + (let [passes (atom 0) + cfg {::db/pool (:app.db/pool th/*system*)}] + (with-redefs [cf/flags #{:nexus} + consumer/sleep-ms 10] + (let [thread (consumer/init-component cfg consumer/nexus identity + (fn [_] (swap! passes inc)))] + (try + (t/is (th/wait-for #(<= 3 @passes))) + (finally + (consumer/halt-component thread))) + (t/is (false? (.isAlive ^Thread thread))))))) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; METRICS +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +;; The pass cfg of these tests carries an isolated metrics registry +;; (pattern from `jobs-metrics-test`), so assertions read exact values +;; instead of the shared test-system registry. `th/run-task!` would use +;; the latter and its counts would leak across tests. +(def ^:private consumer-metric-ids + [:audit-consumer-events :audit-consumer-discarded :audit-consumer-lag + :audit-consumer-passes]) + +(defn- make-consumer-metrics + [] + (ig/init-key :app.metrics/metrics + {:default (select-keys main/default-metrics consumer-metric-ids)})) + +(defn- invoke-nexus + "Run one nexus pass with an isolated metrics registry, so assertions + read exact values instead of the shared test-system registry." + [metrics params] + (consumer.nexus/run-pass + (nexus-cfg (:uri params) metrics))) + +(defn- invoke-admin-console + "Run one admin-console pass with an isolated metrics registry." + [metrics] + (consumer.admin-console/run-pass + (admin-console-cfg metrics))) + +(defn- invoke-both + "Invoke both consumer handlers sharing one isolated metrics + registry, Nexus first (the historical single-run order)." + [metrics params] + (invoke-nexus metrics params) + (invoke-admin-console metrics)) + +(defn- counter-value + [metrics id labels] + (let [collector (mtx/get-collector metrics id) + instance (::mdef/instance collector) + child (.labels ^Counter instance (into-array String labels))] + (.get ^Counter$Child child))) + +(defn- gauge-value + [metrics id labels] + (let [collector (mtx/get-collector metrics id) + instance (::mdef/instance collector) + child (.labels ^Gauge instance (into-array String labels))] + (.get ^Gauge$Child child))) + +(t/deftest consumer-admin-console-rejects-a-3xx-response + ;; A redirect is a configuration error, not a success: the chunk rolls + ;; back, the cursor does not advance and the pass counts failed. The test + ;; pins the typed error: without the 3xx branch the pass also fails, but + ;; with :malli.core/invalid-schema. + (let [prof (th/create-profile* 76 {:is-active true}) + metrics (make-consumer-metrics)] + (with-redefs [cf/flags #{:admin-console} + cf/config (assoc cf/config + :admin-console-uri "http://ac.example/admin-console/" + :admin-console-shared-key "ac-key" + :nexus-shared-key "nexus-key")] + (let [shared-keys {:admin-console "ac-key" :nexus "nexus-key"} + client (ig/init-key :app.nitrate/client + {::http/client (Object.) + ::setup/shared-keys shared-keys}) + consumer-cfg {::db/pool (:app.db/pool th/*system*) + ::mtx/metrics metrics + :app.nitrate/client client}] + (with-mocks [http-mock {:target 'app.http.client/req + :return {:status 302 + :headers {"location" "http://ac.example/admin-console/api/audit-log"} + :body ""}}] + (insert-audit-row (:id prof) "create-project") + (let [err (try + (consumer.admin-console/run-pass consumer-cfg) + nil + (catch clojure.lang.ExceptionInfo e e))] + (t/is (= :admin-console-send-failed (:code (ex-data err)))) + (t/is (= :nitrate-http-error (:type (ex-data (ex-cause err))))) + (t/is (= 302 (:status (ex-data (ex-cause err)))))) + (t/is (= 1 (unread-count consumer/admin-console))) + (t/is (= 1.0 (counter-value metrics :audit-consumer-passes ["admin-console" "failed"])))))))) + +(t/deftest consumer-empty-pass-records-lag-zero + (let [metrics (make-consumer-metrics)] + (with-redefs [cf/flags #{:nexus}] + (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}] + ;; No rows: the pass finds no frontier and sends nothing. + (invoke-nexus metrics {:uri archive-uri}) + (t/is (false? (:called? @http-mock))) + (t/is (zero? (gauge-value metrics :audit-consumer-lag ["nexus"]))))))) + +(t/deftest consumer-records-metrics-on-success + (let [prof (th/create-profile* 40 {:is-active true}) + metrics (make-consumer-metrics)] + (with-redefs [cf/flags #{:nexus :admin-console}] + (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil} + http-mock {:target 'app.http.client/req :return {:status 204}}] + (insert-audit-row (:id prof) "create-project") + (insert-audit-row (:id prof) "unrelated-event") + (invoke-both metrics {:uri archive-uri}) + (t/is (= 2.0 (counter-value metrics :audit-consumer-events ["nexus" "processed"]))) + (t/is (= 1.0 (counter-value metrics :audit-consumer-events ["admin-console" "processed"]))) + (t/is (= 1.0 (counter-value metrics :audit-consumer-events ["admin-console" "skipped"]))) + (t/is (= 0 (unread-count consumer/nexus))) + (t/is (= 0 (unread-count consumer/admin-console))))))) + +(t/deftest consumer-records-the-lag-gauge + (let [prof (th/create-profile* 41 {:is-active true}) + metrics (make-consumer-metrics)] + (insert-audit-row (:id prof) "create-project") + (insert-audit-row (:id prof) "create-file") + ;; The cursor is still at zero: two confirmed events behind. + (with-open [conn (db/open th/*system*)] + (#'consumer/record-lag conn metrics {:id consumer/nexus})) + (t/is (= 2.0 (gauge-value metrics :audit-consumer-lag ["nexus"]))))) + +(t/deftest consumer-pass-counter-records-ok-and-busy + (let [prof (th/create-profile* 39 {:is-active true}) + metrics (make-consumer-metrics)] + (with-redefs [cf/flags #{:nexus}] + (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}] + (insert-audit-row (:id prof) "create-project") + (invoke-nexus metrics {:uri archive-uri}) + (t/is (= 1.0 (counter-value metrics :audit-consumer-passes ["nexus" "ok"]))) + + (insert-audit-row (:id prof) "create-file") + (with-open [conn (db/open th/*system*)] + (let [tick (hold-tick conn consumer/nexus)] + (t/is (= ::consumer/busy (invoke-nexus metrics {:uri archive-uri}))) + (release-tick conn tick))) + (t/is (= 1.0 (counter-value metrics :audit-consumer-passes ["nexus" "busy"]))) + (t/is (= 1.0 (counter-value metrics :audit-consumer-passes ["nexus" "ok"]))))))) + +(t/deftest consumer-pass-counter-records-a-failure-and-releases-the-tick + (let [prof (th/create-profile* 40 {:is-active true}) + metrics (make-consumer-metrics) + calls (atom 0)] + (with-redefs [cf/flags #{:nexus}] + (with-mocks [http-mock {:target 'app.http.client/req + :return (fn [_ _ & _] + (if (= 1 (swap! calls inc)) + (throw (ex-info "timeout" {})) + {:status 204}))}] + (insert-audit-row (:id prof) "create-project") + (t/is (thrown? clojure.lang.ExceptionInfo + (invoke-nexus metrics {:uri archive-uri}))) + (t/is (= 1.0 (counter-value metrics :audit-consumer-passes ["nexus" "failed"]))) + ;; The tick was released: the next pass drains and counts ok. + (invoke-nexus metrics {:uri archive-uri}) + (t/is (= 1.0 (counter-value metrics :audit-consumer-passes ["nexus" "ok"]))) + (t/is (= 0 (unread-count consumer/nexus))))))) + +(t/deftest consumer-pass-counter-counts-a-frontier-failure-as-failed + ;; The counter covers the whole pass, not only the chunk send: a read + ;; that fails before the first chunk is a failed pass too. + (let [metrics (make-consumer-metrics)] + (with-redefs [cf/flags #{:nexus} + consumer/sql:last-visible "SELECT 1/0 AS xid_text, 0::bigint AS seq"] + (t/is (thrown? Exception (invoke-nexus metrics {:uri archive-uri}))) + (t/is (= 1.0 (counter-value metrics :audit-consumer-passes ["nexus" "failed"]))) + (t/is (zero? (counter-value metrics :audit-consumer-passes ["nexus" "ok"])))))) + +(t/deftest consumer-pass-survives-an-unlock-failure + ;; A release that fails is logged and swallowed: it must not replace the + ;; result of the pass. The real unlock still runs, so the test leaves no + ;; session lock behind on a pooled connection. + (let [prof (th/create-profile* 46 {:is-active true}) + metrics (make-consumer-metrics) + real-unlock db/session-unlock] + (with-redefs [cf/flags #{:nexus} + db/session-unlock (fn [& args] + (apply real-unlock args) + (throw (ex-info "gone" {})))] + (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}] + (insert-audit-row (:id prof) "create-project") + (t/is (nil? (invoke-nexus metrics {:uri archive-uri}))) + (t/is (= 1.0 (counter-value metrics :audit-consumer-passes ["nexus" "ok"]))) + (t/is (= 0 (unread-count consumer/nexus))))) + ;; No advisory lock of the tick stays in the database: the mock released + ;; it for real before throwing. Read pg_locks (all sessions), because a + ;; session that already holds a lock can re-acquire it and answer true. + (let [[classid objid] (#'consumer/tick-lock-key consumer/nexus)] + (t/is (zero? (:cnt (th/db-exec-one! + ["SELECT count(*) AS cnt + FROM pg_locks + WHERE locktype = 'advisory' + AND classid = ?::int + AND objid = ?::int + AND objsubid = 2" + classid objid]))))))) + +(t/deftest consumer-pass-reports-the-chunks-that-committed-before-a-failure + ;; A refusal on the second chunk must not hide the 128 events of the + ;; first: its transaction committed and the cursor moved past it. + (let [prof (th/create-profile* 43 {:is-active true}) + metrics (make-consumer-metrics) + calls (atom 0)] + (with-redefs [cf/flags #{:nexus}] + (with-mocks [http-mock {:target 'app.http.client/req + :return (fn [_ _ & _] + (if (= 2 (swap! calls inc)) + (throw (ex-info "timeout" {})) + {:status 204}))}] + (with-open [conn (db/open th/*system*)] + (insert-rows-in-one-transaction conn "lote-" 300) + (db/exec! conn ["COMMIT"])) + (t/is (thrown? clojure.lang.ExceptionInfo + (invoke-nexus metrics {:uri archive-uri}))) + ;; The first chunk committed and the cursor moved past it. + (t/is (= 172 (unread-count consumer/nexus))) + (t/is (= 128.0 (counter-value metrics :audit-consumer-events ["nexus" "processed"]))) + (t/is (= 128.0 (counter-value metrics :audit-consumer-events ["nexus" "failed"]))) + (t/is (= 1.0 (counter-value metrics :audit-consumer-passes ["nexus" "failed"]))))))) + +(t/deftest consumer-busy-does-not-sample-the-lag + (let [prof (th/create-profile* 41 {:is-active true}) + metrics (make-consumer-metrics)] + (with-redefs [cf/flags #{:nexus}] + (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}] + (insert-audit-row (:id prof) "create-project") + (invoke-nexus metrics {:uri archive-uri}) + (t/is (zero? (gauge-value metrics :audit-consumer-lag ["nexus"]))) + + (insert-audit-row (:id prof) "create-file") + (with-open [conn (db/open th/*system*)] + (let [tick (hold-tick conn consumer/nexus)] + (t/is (= ::consumer/busy (invoke-nexus metrics {:uri archive-uri}))) + (release-tick conn tick))) + ;; Still the sample of the first pass, not the row behind. + (t/is (zero? (gauge-value metrics :audit-consumer-lag ["nexus"]))))))) + +(t/deftest consumer-pass-sends-full-chunks-under-continuous-traffic + (let [prof (th/create-profile* 42 {:is-active true}) + arrived (atom 0)] + (with-redefs [cf/flags #{:nexus}] + (with-mocks [http-mock {:target 'app.http.client/req + :return (fn [_ _ & _] + (swap! arrived inc) + (insert-audit-row (:id prof) (str "during-" @arrived)) + {:status 204})}] + (with-open [conn (db/open th/*system*)] + (insert-rows-in-one-transaction conn "lote-" 300) + (db/exec! conn ["COMMIT"])) + (run-nexus {:uri archive-uri}) + (let [bodies (map #(transit/decode (:body (second %))) + (:call-args-list @http-mock))] + (t/is (= [128 128 44] (mapv #(count (:events %)) bodies)))) + ;; The rows that arrived during the pass wait for the next one. + (t/is (= 3 (unread-count consumer/nexus))))))) + +(t/deftest consumer-component-does-not-start-on-a-read-only-database + (let [pool (make-pool 1 {::db/read-only true})] + (try + (with-redefs [cf/flags #{:nexus}] + (t/is (nil? (ig/init-key :app.loggers.audit.consumers.nexus/consumer + (merge (nexus-cfg archive-uri) {::db/pool pool}))))) + (finally + (ig/halt-key! :app.db/pool pool))))) + +(t/deftest consumer-lag-gauge-is-capped + ;; The gauge counts behind the cursor once per pass; past 10000 it + ;; saturates, so a long backlog does not turn every pass into an + ;; O(backlog) count. Remove the LIMIT and this test fails. + (let [prof (th/create-profile* 44 {:is-active true}) + metrics (make-consumer-metrics)] + (with-open [conn (db/open th/*system*)] + (insert-rows-in-one-transaction conn "lote-" 10001) + (db/exec! conn ["COMMIT"])) + (with-open [conn (db/open th/*system*)] + (#'consumer/record-lag conn metrics {:id consumer/nexus})) + (t/is (= 10000.0 (gauge-value metrics :audit-consumer-lag ["nexus"]))))) + +(t/deftest consumer-nexus-transport-failure-records-failed + (let [prof (th/create-profile* 42 {:is-active true}) + metrics (make-consumer-metrics)] + (with-redefs [cf/flags #{:nexus}] + (with-mocks [http-mock {:target 'app.http.client/req + :throw (ex-info "timeout" {})}] + (insert-audit-row (:id prof) "create-project") + (t/is (thrown? clojure.lang.ExceptionInfo + (invoke-nexus metrics {:uri archive-uri}))) + (t/is (= 1.0 (counter-value metrics :audit-consumer-events ["nexus" "failed"]))) + ;; The chunk rolled back: the cursor did not move. + (t/is (= 1 (unread-count consumer/nexus))))))) + +(t/deftest consumer-nexus-refused-chunk-keeps-the-cursor + (let [prof (th/create-profile* 43 {:is-active true}) + metrics (make-consumer-metrics)] + (with-redefs [cf/flags #{:nexus}] + (with-mocks [http-mock {:target 'app.http.client/req :return {:status 500}}] + (insert-audit-row (:id prof) "create-project") + (let [err (try + (invoke-nexus metrics {:uri archive-uri}) + nil + (catch clojure.lang.ExceptionInfo cause cause))] + (t/is (= :consumer-chunk-refused (:code (ex-data err))))) + (t/is (= 1.0 (counter-value metrics :audit-consumer-events ["nexus" "failed"]))) + (t/is (= 1 (unread-count consumer/nexus))))))) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; PRODUCER: ONE ROW, NO QUEUE +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(defn- full-event + "A complete event for submit*, which (unlike submit) assumes every + required field is filled." + [profile-id] + {:name "create-file" :type "action" :source "backend" + :profile-id profile-id :ip-addr "127.0.0.1" + :props {} :context {} + :created-at (ct/now) :tracked-at (ct/now)}) + +(t/deftest producer-inserts-one-row-and-no-queue + (let [prof (th/create-profile* 50 {:is-active true})] + (th/with-audit-config #{:nexus :admin-console} + (audit/submit* {::db/pool (:app.db/pool th/*system*)} + (full-event (:id prof))) + (t/is (= 1 (count (th/db-exec! ["SELECT * FROM audit_log"])))) + ;; The producer never fills the pending queue: the row exists, the + ;; queue stays empty (or absent, on an installation that predates it). + (let [{:keys [pending]} (th/db-exec-one! + ["SELECT to_regclass('public.audit_log_pending') AS pending"])] + (t/is (or (nil? pending) + (zero? (:cnt (th/db-exec-one! + ["SELECT count(*) AS cnt FROM audit_log_pending"])))))) + ;; The cursors did not move: the row waits for the next drain. + (t/is (= 1 (unread-count consumer/nexus))) + (t/is (= 1 (unread-count consumer/admin-console)))))) + +(t/deftest producer-rollback-leaves-no-row + (let [prof (th/create-profile* 51 {:is-active true})] + (th/with-audit-config #{:nexus :admin-console} + (db/tx-run! {::db/pool (:app.db/pool th/*system*) + ::db/rollback true} + (fn [cfg] + (audit/submit* cfg (full-event (:id prof))) + (t/is (= 1 (count (db/exec! cfg ["SELECT * FROM audit_log"])))))) + (t/is (= 0 (count (th/db-exec! ["SELECT * FROM audit_log"]))))))) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; GC: DELETE BELOW THE OLDEST CURSOR +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(t/deftest gc-deletes-only-what-every-enabled-consumer-passed + (let [prof (th/create-profile* 60 {:is-active true})] + (with-redefs [cf/flags #{:nexus :admin-console}] + (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}} + admin-mock {:target 'app.nitrate/call :return nil}] + (insert-audit-row (:id prof) "create-project") + (run-nexus {:uri archive-uri}) + ;; Nexus passed it, the admin console has not: nothing to delete. + (t/is (= 0 (run-gc))) + (t/is (= 1 (count (th/db-exec! ["SELECT * FROM audit_log"])))) + (run-admin-console) + (t/is (= 1 (run-gc))) + (t/is (= 0 (count (th/db-exec! ["SELECT * FROM audit_log"])))) + (t/is (:called? @http-mock)) + (t/is (:called? @admin-mock)))))) + +(t/deftest gc-waits-for-a-disabled-consumer + (let [prof (th/create-profile* 61 {:is-active true})] + (with-redefs [cf/flags #{:nexus}] + (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}] + (insert-audit-row (:id prof) "create-project") + (run-nexus {:uri archive-uri}) + ;; The admin console is off: its frozen cursor at (0, 0) still + ;; holds the row. A temporary disable must not let the GC delete + ;; events that consumer has not read. + (t/is (= 0 (run-gc))) + (t/is (= 1 (count (th/db-exec! ["SELECT * FROM audit_log"])))))))) + +(t/deftest gc-keeps-the-telemetry-shadows + (let [prof (th/create-profile* 62 {:is-active true})] + (with-redefs [cf/flags #{:nexus :admin-console}] + (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}} + admin-mock {:target 'app.nitrate/call :return nil}] + (insert-audit-row (:id prof) "create-project") + (insert-audit-row (:id prof) "create-project" {} "telemetry:backend") + (run-nexus {:uri archive-uri}) + (run-admin-console) + ;; The telemetry task owns the shadows: the GC leaves them. + (t/is (= 1 (run-gc))) + (let [rows (th/db-exec! ["SELECT source FROM audit_log"])] + (t/is (= ["telemetry:backend"] (mapv :source rows)))))))) + +(t/deftest gc-records-the-discarded-metric + (let [prof (th/create-profile* 63 {:is-active true}) + metrics (make-consumer-metrics) + job-def (ig/init-key :app.loggers.audit.gc/job-def + {::db/pool (:app.db/pool th/*system*) + ::mtx/metrics metrics})] + (with-redefs [cf/flags #{:nexus :admin-console}] + (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}} + admin-mock {:target 'app.nitrate/call :return nil}] + (insert-audit-row (:id prof) "create-project") + (insert-audit-row (:id prof) "create-file") + (run-nexus {:uri archive-uri}) + (run-admin-console) + (t/is (= 2 ((::jobs/handler job-def) nil {}))) + (t/is (= 2.0 (counter-value metrics :audit-consumer-discarded ["gc"]))))))) + +(t/deftest gc-deletes-in-batches + (let [prof (th/create-profile* 64 {:is-active true})] + (with-redefs [cf/flags #{:nexus :admin-console}] + (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}} + admin-mock {:target 'app.nitrate/call :return nil}] + (dotimes [i 5] + (insert-audit-row (:id prof) (str "event-" i))) + (run-nexus {:uri archive-uri}) + (run-admin-console) + (with-redefs-fn {#'gc/batch-size 2} + (fn [] + (t/is (= 5 (run-gc))))) + (t/is (= 0 (count (th/db-exec! ["SELECT * FROM audit_log"])))))))) + +(t/deftest init-component-registers-the-cursor-when-enabled + ;; The GC waits for the cursor table, so an enabled consumer must have + ;; its (0, 0) row before its first pass. A disabled consumer leaves no + ;; row behind. `start-loop` is replaced so the test leaves no live + ;; thread behind; the loop itself has its own test. + (th/db-exec! ["DELETE FROM audit_log_consumer_cursor"]) + (with-redefs [cf/flags #{:nexus} + consumer/start-loop (fn [_cfg _id _run-pass] nil)] + (consumer/init-component + {::db/pool (:app.db/pool th/*system*) + ::mtx/metrics (:app.metrics/metrics th/*system*)} + consumer/nexus + identity + (fn [_cfg] nil)) + (t/is (= 0 (:seq (cursor-of consumer/nexus)))) + (t/is (nil? (cursor-of consumer/admin-console))))) + +(t/deftest gc-waits-for-a-consumer-that-has-not-drained-yet + ;; A consumer that has just started registers its cursor at (0, 0) + ;; before its first pass. The GC treats that position as the oldest and + ;; deletes nothing until the consumer drains the history and advances. + (let [prof (th/create-profile* 65 {:is-active true})] + (with-redefs [cf/flags #{:nexus :admin-console}] + (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}} + admin-console-mock {:target 'app.nitrate/call :return nil}] + (insert-audit-row (:id prof) "create-project") + (run-nexus {:uri archive-uri}) + ;; The admin console registered at (0, 0) but has not drained yet. + (t/is (= 0 (run-gc))) + (t/is (= 1 (count (th/db-exec! ["SELECT * FROM audit_log"])))) + ;; Its first drain processes the history and advances, so the + ;; next GC can delete. + (run-admin-console) + (t/is (:called? @admin-console-mock)) + (t/is (some? (cursor-of consumer/admin-console))) + (t/is (= 1 (run-gc))) + (t/is (= 0 (count (th/db-exec! ["SELECT * FROM audit_log"])))))))) + +(t/deftest gc-min-cursor-orders-xid8-numerically + ;; The oldest cursor is chosen by the xid8 value, not by its text form: + ;; '1000' sorts before '999' as text, and the GC would take the consumer + ;; that is ahead and delete rows it has not read. + (let [[low high] (xid-pair-across-a-decimal-boundary)] + (th/db-exec! ["UPDATE audit_log_consumer_cursor + SET xid = ?::xid8, seq = 7 WHERE consumer = ?::smallint" + (str low) consumer/nexus]) + (th/db-exec! ["UPDATE audit_log_consumer_cursor + SET xid = ?::xid8, seq = 3 WHERE consumer = ?::smallint" + (str high) consumer/admin-console]) + (let [cursor (@#'gc/get-oldest-cursor {::db/pool th/*pool*})] + (t/is (= (str low) (:xid-text cursor))) + (t/is (= 7 (:seq cursor)))))) + +(t/deftest gc-re-reads-the-cursor-between-batches + ;; A consumer registered between two batches must be seen by the next + ;; batch: `delete-one-batch` re-reads the oldest cursor under the + ;; registry lock, so it does not delete rows the new consumer has not + ;; read. + (let [prof (th/create-profile* 66 {:is-active true}) + cfg {::db/pool (:app.db/pool th/*system*) + ::mtx/metrics (:app.metrics/metrics th/*system*)}] + (with-redefs [cf/flags #{:nexus}] + (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}] + (dotimes [i 4] + (insert-audit-row (:id prof) (str "event-" i))) + (run-nexus {:uri archive-uri}) + ;; The admin console has not registered yet: its cursor is absent. + (th/db-exec! ["DELETE FROM audit_log_consumer_cursor + WHERE consumer = ?::smallint" + consumer/admin-console]) + (with-redefs-fn {#'gc/batch-size 1} + (fn [] + (t/is (= {:deleted 1 :found? true} + (@#'gc/delete-one-batch cfg))) + ;; A new consumer registers at (0, 0) between the batches. + (consumer/ensure-cursor cfg consumer/admin-console) + (t/is (= {:deleted 0 :found? true} + (@#'gc/delete-one-batch cfg))) + (t/is (= 3 (count (th/db-exec! ["SELECT * FROM audit_log"])))))))))) + +(t/deftest gc-does-not-delete-past-an-unread-cursor + ;; Two enabled consumers across a decimal boundary: the GC deletes up to + ;; the oldest one. Sorting by text would pick the consumer that is ahead + ;; and remove a row it has not read. + (let [[low high] (xid-pair-across-a-decimal-boundary)] + (th/db-exec! ["UPDATE audit_log_consumer_cursor + SET xid = ?::xid8, seq = 10 WHERE consumer = ?::smallint" + (str low) consumer/nexus]) + (th/db-exec! ["UPDATE audit_log_consumer_cursor + SET xid = ?::xid8, seq = 10 WHERE consumer = ?::smallint" + (str high) consumer/admin-console]) + (insert-audit-row-at high 1) + ;; The local flags name only nexus; the admin-console cursor still + ;; holds the position, so the GC must not delete past it. + (with-redefs [cf/flags #{:nexus}] + (t/is (= 0 (run-gc))) + (t/is (= 1 (count (th/db-exec! ["SELECT * FROM audit_log"]))))))) + +(t/deftest gc-deletes-nothing-with-no-cursor + ;; The cursor table is the registry. With no cursor (the consumers have + ;; not run yet) the GC cannot know a position and deletes nothing: a + ;; drain here would remove events a consumer that is starting would miss. + (th/db-exec! ["DELETE FROM audit_log_consumer_cursor"]) + (insert-audit-row uuid/zero "create-project") + (insert-audit-row uuid/zero "create-file") + (insert-audit-row uuid/zero "create-project" {} "telemetry:backend") + (t/is (= 0 (run-gc))) + (t/is (= 3 (count (th/db-exec! ["SELECT * FROM audit_log"]))))) + +(t/deftest consumer-frontier-uses-the-numeric-max + ;; The frontier is the newest visible row, compared as xid8 and not as + ;; its text form: '999' sorts above '1000' as text, so a text ORDER BY + ;; stops the pass one row early and leaves the newest row invisible. + (let [[low high] (xid-pair-across-a-decimal-boundary)] + (with-redefs [cf/flags #{:nexus}] + (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}] + (insert-audit-row-at low 1) + (insert-audit-row-at high 1) + (run-nexus {:uri archive-uri}) + (t/is (= 2 (count (sent-event-names http-mock)))) + (t/is (= (str high) (:xid (cursor-of consumer/nexus)))))))) diff --git a/backend/test/backend_tests/migrations_test.clj b/backend/test/backend_tests/migrations_test.clj index 9a93eaecac..393a491459 100644 --- a/backend/test/backend_tests/migrations_test.clj +++ b/backend/test/backend_tests/migrations_test.clj @@ -71,6 +71,8 @@ "id" "name" "tenant" "queue" "label" "priority" "scheduled_at" "retry_num" "max_retries" "status" "created_at" "modified_at" "started_at" "completed_at" "params" + ; the jobs that resolved their key in the database + "exclusive" "dedupe" ; optional ledger columns "profile_id" "error" "result" "resource_id" "expires_at"} @@ -147,7 +149,118 @@ (t/deftest job-table-has-expected-indexes (t/is (contains? (table-indexes "job") "job__dispatcher__idx")) (t/is (contains? (table-indexes "job") "job__orphan__idx")) - (t/is (contains? (table-indexes "job") "job__profile__idx"))) + (t/is (contains? (table-indexes "job") "job__profile__idx")) + (t/is (contains? (table-indexes "job") "job__live_exclusive__uniq")) + (t/is (contains? (table-indexes "job") "job__new_dedupe__uniq"))) + +(t/deftest audit-log-has-a-consumer-cursor + (t/testing "the position columns and their covering index exist" + (t/is (contains? (table-columns "audit_log") "seq")) + (t/is (contains? (table-columns "audit_log") "xid")) + (t/is (contains? (table-indexes "audit_log") "audit_log__xid__seq__idx"))) + (t/testing "one row per consumer, with the position and its clock" + (t/is (= #{"consumer" "xid" "seq" "updated_at"} + (table-columns "audit_log_consumer_cursor"))))) + +(t/deftest the-migration-keeps-the-legacy-archive-mark-and-indexes + ;; The legacy mark and the indexes that served the old archiver stay in + ;; place: the old archiver may still be running during the rollout, and a + ;; later release removes them. The pending queue's `(created_at, id)` + ;; index is not built either. + (th/db-exec! ["ALTER TABLE audit_log ADD COLUMN IF NOT EXISTS archived_at timestamptz"]) + (th/db-exec! ["DROP INDEX IF EXISTS audit_log__created_at__idx"]) + (th/db-exec! ["CREATE INDEX audit_log__created_at__idx + ON audit_log (created_at) WHERE archived_at IS NULL"]) + (th/db-exec! ["CREATE INDEX IF NOT EXISTS audit_log__archived_at__idx + ON audit_log (archived_at) WHERE archived_at IS NOT NULL"]) + (th/db-exec! ["CREATE INDEX IF NOT EXISTS audit_log_id_archived_at_idx + ON audit_log (id, archived_at)"]) + (let [apply! (mg/resource "app/migrations/sql/0159-consumer-generalization.sql")] + (t/is (true? (apply! th/*pool*)))) + (t/is (contains? (table-columns "audit_log") "archived_at")) + (t/is (contains? (table-indexes "audit_log") "audit_log__created_at__idx")) + (t/is (contains? (table-indexes "audit_log") "audit_log__archived_at__idx")) + (t/is (contains? (table-indexes "audit_log") "audit_log_id_archived_at_idx")) + (t/is (str/includes? + (:indexdef (th/db-exec-one! + ["SELECT indexdef FROM pg_indexes + WHERE schemaname = 'public' + AND tablename = 'audit_log' + AND indexname = 'audit_log__created_at__idx'"])) + "WHERE (archived_at IS NULL)")) + (t/is (not (some #(str/includes? % "(created_at, id)") + (map :indexdef + (th/db-exec! ["SELECT indexdef FROM pg_indexes + WHERE schemaname = 'public' + AND tablename = 'audit_log'"])))))) + +(t/deftest the-migration-drops-the-old-queue-table-on-a-pre-release-db + ;; The release never creates `audit_log_pending`: the pending queue never + ;; shipped. A pre-release database that ran its step does have it, and it + ;; may still have rows; the table has to go, or the leftover rows would + ;; block the new GC when it deletes the rows they reference. Recreate the + ;; table and run the migration again to pin it. + (th/db-exec! ["CREATE TABLE IF NOT EXISTS audit_log_pending ( + audit_log_id uuid NOT NULL REFERENCES audit_log(id) DEFERRABLE, + consumer smallint NOT NULL CHECK (consumer BETWEEN 1 AND 20), + PRIMARY KEY (consumer, audit_log_id))"]) + (t/is (contains? (table-columns "audit_log_pending") "consumer")) + (let [apply! (mg/resource "app/migrations/sql/0159-consumer-generalization.sql")] + (t/is (true? (apply! th/*pool*)))) + (t/is (empty? (table-columns "audit_log_pending")))) + +(t/deftest the-migration-does-not-seed-consumer-cursors + ;; The migration cannot know which consumers are enabled, so the cursor + ;; table starts empty; `ensure-cursor` creates the row at (0, 0) on a + ;; consumer's first pass, which is what makes the first deploy replay the + ;; unarchived buffer. + (th/db-exec! ["DELETE FROM audit_log_consumer_cursor WHERE consumer IN (1, 2)"]) + (let [apply! (mg/resource "app/migrations/sql/0159-consumer-generalization.sql")] + (t/is (true? (apply! th/*pool*)))) + (let [rows (th/db-exec! ["SELECT consumer + FROM audit_log_consumer_cursor + WHERE consumer IN (1, 2)"])] + (t/is (empty? rows)))) + +(t/deftest audit-log-rows-get-seq-and-xid-by-default + ;; The previous version keeps inserting rows during the rollout, so the + ;; database must fill both columns, and the new rows must sort above the + ;; backfilled history (the reserved low range). + (let [id (uuid/next)] + (try + (th/db-exec! ["INSERT INTO audit_log (id, name, type, source, profile_id, + ip_addr, props, context, tracked_at, created_at) + VALUES (?, 'create-project', 'action', 'backend', ?, + '127.0.0.1', ?, ?, now(), now())" + id uuid/zero (db/tjson {}) (db/tjson {})]) + (let [row (th/db-exec-one! ["SELECT seq, xid::text AS xid + FROM audit_log WHERE id = ?" id])] + (t/is (some? (:seq row))) + (t/is (>= (:seq row) 1000000)) + (t/is (not= "0" (:xid row)))) + (finally + (th/db-exec! ["DELETE FROM audit_log WHERE id = ?" id]))))) + +(t/deftest audit-log-source-is-not-null + ;; `0064` added the column nullable and left `SET NOT NULL` commented; + ;; `0159` fills what is left and closes it. A NULL source would be deleted + ;; by neither the GC (`NOT LIKE 'telemetry:%'`) nor the telemetry task + ;; (`LIKE 'telemetry:%'`), so the row would stay for ever. + (let [apply! (mg/resource "app/migrations/sql/0159-consumer-generalization.sql")] + (t/is (true? (apply! th/*pool*)))) + (t/is (= "NO" (:is-nullable + (th/db-exec-one! + ["SELECT is_nullable FROM information_schema.columns + WHERE table_schema = 'public' + AND table_name = 'audit_log' + AND column_name = 'source'"])))) + (t/is (thrown? java.sql.SQLException + (th/db-exec! + ["INSERT INTO audit_log (id, name, type, source, profile_id, + ip_addr, props, context, tracked_at, created_at) + VALUES (?, 'create-project', 'action', NULL, ?, + '127.0.0.1', ?, ?, now(), now())" + (uuid/next) uuid/zero (db/tjson {}) (db/tjson {})])))) (defn- indexdef [index-name] @@ -167,12 +280,121 @@ (t/is (not (str/includes? def "(tenant, queue")))))) (t/deftest job-sweep-indexes-are-tenant-scoped + ;; The test database already ran 0159 in its old shape: re-apply it so the + ;; lost-sweep index exists with the current definition. + (th/db-exec! ["DROP INDEX IF EXISTS job__scheduled__idx"]) + ;; An early 0154 built the cron probe index; re-create it so the drop + ;; below proves the migration removes an index that already exists. + (th/db-exec! ["CREATE INDEX IF NOT EXISTS job__name_label__idx + ON job (tenant, name, queue, label) + WHERE status IN ('new', 'scheduled', 'running', 'retry')"]) + (let [apply! (mg/resource "app/migrations/sql/0159-consumer-generalization.sql")] + (t/is (true? (apply! th/*pool*)))) (t/testing "orphan sweep" (t/is (str/includes? (indexdef "job__orphan__idx") "(tenant, modified_at)"))) - (t/testing "reschedule of lost scheduled rows" - (t/is (str/includes? (indexdef "job__scheduled__idx") "(tenant, scheduled_at)"))) - (t/testing "cron no-overlap check and submit dedupe" - (t/is (str/includes? (indexdef "job__name_label__idx") "(tenant, name, queue, label)")))) + (t/testing "lost sweep: it filters by the hand-off time" + (t/is (str/includes? (indexdef "job__scheduled__idx") "(tenant, modified_at)")) + (t/is (not (str/includes? (indexdef "job__scheduled__idx") "(tenant, scheduled_at)")))) + (t/testing "the cron no-overlap probe index is gone" + (t/is (nil? (indexdef "job__name_label__idx"))))) + +(t/deftest job-new-dedupe-index-admits-one-not-started-row-per-key + ;; The dedupe of a submit is the index, not a delete before the insert: + ;; it is unique, it covers the whole key, and it only holds the rows + ;; that asked for the option and have not started yet. + (let [def (indexdef "job__new_dedupe__uniq")] + (t/is (str/includes? def "CREATE UNIQUE INDEX")) + (t/is (str/includes? def "(tenant, name, queue, label)")) + (t/is (str/includes? def "dedupe")) + (t/testing "only the not-started status is covered" + (t/is (str/includes? def "'new'")) + (doseq [status ["scheduled" "running" "retry"]] + (t/is (not (str/includes? def (str "'" status "'"))) + (str "the " status " status leaves the index")))))) + +(t/deftest consumer-generalization-migration-is-idempotent + ;; The out-of-band runbook pre-applies the whole migration (columns, + ;; CONCURRENTLY indexes, backfill), so the migration must find the schema + ;; and do nothing when it runs afterwards. + (let [apply! (mg/resource "app/migrations/sql/0159-consumer-generalization.sql")] + (t/is (true? (apply! th/*pool*))) + (t/is (contains? (table-columns "job") "exclusive")) + (t/is (contains? (table-columns "job") "dedupe")) + (t/is (contains? (table-indexes "job") "job__live_exclusive__uniq")) + (t/is (contains? (table-indexes "job") "job__new_dedupe__uniq")) + (t/is (contains? (table-columns "audit_log") "seq")) + (t/is (contains? (table-columns "audit_log") "xid")) + (t/is (contains? (table-indexes "audit_log") "audit_log__xid__seq__idx")) + (t/is (contains? (table-columns "audit_log_consumer_cursor") "consumer")))) + +(defn- sequence-last-value + [] + (:last-value (th/db-exec-one! ["SELECT last_value FROM audit_log_seq_seq"]))) + +(defn- insert-audit-row + "Insert one row with the columns the previous version filled, so the + database gives it `seq` and `xid` from their defaults." + [] + (let [id (uuid/next)] + (th/db-exec! ["INSERT INTO audit_log (id, name, type, source, profile_id, + ip_addr, props, context, tracked_at, created_at) + VALUES (?, 'create-project', 'action', 'backend', ?, + '127.0.0.1', ?, ?, now(), now())" + id uuid/zero (db/tjson {}) (db/tjson {})]) + id)) + +(t/deftest the-sequence-reserve-is-only-applied-to-a-fresh-sequence + ;; A re-run must not move a sequence that already left its initial value: + ;; on the first application it starts at 1 and the migration reserves the + ;; low range, so the history gets 1..N and the new rows start above it. + (let [apply! (mg/resource "app/migrations/sql/0159-consumer-generalization.sql") + before (sequence-last-value)] + (try + (th/db-exec! ["ALTER SEQUENCE audit_log_seq_seq RESTART WITH 1"]) + (t/is (true? (apply! th/*pool*))) + (t/is (>= (sequence-last-value) 1000000)) + (finally + (th/db-exec! ["SELECT setval('audit_log_seq_seq', ?::bigint, false)" before]))))) + +(t/deftest the-sequence-reserve-is-not-moved-on-a-rerun + ;; Inserting a row advances the sequence; deleting it drops `count(*)` + ;; again. The old unconditional setval lowered the sequence in that case. + (let [apply! (mg/resource "app/migrations/sql/0159-consumer-generalization.sql") + id (insert-audit-row)] + (try + (let [before (sequence-last-value)] + (th/db-exec! ["DELETE FROM audit_log WHERE id = ?" id]) + (t/is (true? (apply! th/*pool*))) + (t/is (= before (sequence-last-value))) + (let [max-seq (:max (th/db-exec-one! ["SELECT max(seq) AS max FROM audit_log"])) + next (:next (th/db-exec-one! ["SELECT nextval('audit_log_seq_seq') AS next"]))] + (t/is (> next (or max-seq 0))))) + (finally + (th/db-exec! ["DELETE FROM audit_log WHERE id = ?" id]))))) + +(t/deftest the-sequence-reserve-is-not-applied-to-a-used-once-sequence + ;; A sequence that was called once has last_value = 1 with is_called = true: + ;; it is not intact, so the reserve must leave it alone. + (let [apply! (mg/resource "app/migrations/sql/0159-consumer-generalization.sql") + before (sequence-last-value)] + (try + (th/db-exec! ["SELECT setval('audit_log_seq_seq', 1, true)"]) + (t/is (true? (apply! th/*pool*))) + (t/is (= 1 (sequence-last-value))) + (finally + (th/db-exec! ["SELECT setval('audit_log_seq_seq', ?::bigint, false)" before]))))) + +(t/deftest job-live-exclusive-index-admits-one-live-row-per-key + ;; The exclusion of the live jobs is the index, not a check that races + ;; an insert: it is unique, it covers the whole key, and it only + ;; holds the rows that asked for the exclusion. + (let [def (indexdef "job__live_exclusive__uniq")] + (t/is (str/includes? def "CREATE UNIQUE INDEX")) + (t/is (str/includes? def "(tenant, name, queue, label)")) + (t/is (str/includes? def "exclusive")) + (doseq [status ["new" "scheduled" "running" "retry"]] + (t/is (str/includes? def (str "'" status "'")) + (str "the live set covers the " status " status"))))) (t/deftest job-table-has-sweep-path-indexes (t/testing "jobs-GC expiration scan" @@ -311,10 +533,10 @@ {:file-id (:id file) :id data-id}))))) - (t/testing "the backfill removes the dead metadata pointer" + (t/testing "the backfill removes the migrated metadata pointer" (let [row (th/db-get :file-data {:file-id (:id file) :id data-id}) metadata (fdata/decode-metadata (:metadata row))] - (t/is (nil? (:storage-ref-id metadata))))) + (t/is (not (contains? metadata :storage-ref-id))))) (t/testing "a second application changes nothing" (let [before (th/db-get :file-data {:file-id (:id file) :id data-id})] @@ -329,6 +551,7 @@ sobject (:id (th/db-insert! :storage-object {:backend "fs"})) valid-id (uuid/next) corrupt-id (uuid/next) + orphan-soid (uuid/next) orphan-id (uuid/next) other-id (uuid/next)] @@ -340,7 +563,7 @@ :metadata (db/json {"storage-ref-id" "not-a-uuid"})}) (th/db-insert! :file-data {:file-id (:id file) :id orphan-id :type "snapshot" :backend "storage" - :metadata (db/json {"storage-ref-id" (str (uuid/next))})}) + :metadata (db/json {"storage-ref-id" (str orphan-soid)})}) (th/db-insert! :file-data {:file-id (:id file) :id other-id :type "snapshot" :backend "db" :metadata (db/json {"storage-ref-id" (str sobject)})}) @@ -354,14 +577,16 @@ :id valid-id}))))) (t/testing "a non-uuid value stays NULL" - (t/is (nil? (:storage-object-id (th/db-get :file-data - {:file-id (:id file) - :id corrupt-id}))))) + (let [row (th/db-get :file-data {:file-id (:id file) :id corrupt-id}) + metadata (fdata/decode-metadata (:metadata row))] + (t/is (nil? (:storage-object-id row))) + (t/is (= "not-a-uuid" (:storage-ref-id metadata))))) (t/testing "a pointer to a missing object stays NULL" - (t/is (nil? (:storage-object-id (th/db-get :file-data - {:file-id (:id file) - :id orphan-id}))))) + (let [row (th/db-get :file-data {:file-id (:id file) :id orphan-id}) + metadata (fdata/decode-metadata (:metadata row))] + (t/is (nil? (:storage-object-id row))) + (t/is (= (str orphan-soid) (:storage-ref-id metadata))))) (t/testing "rows of other backends are not touched" (t/is (nil? (:storage-object-id (th/db-get :file-data diff --git a/backend/test/backend_tests/nitrate_ingest_audit_log_test.clj b/backend/test/backend_tests/nitrate_ingest_audit_log_test.clj index 82971f4d93..3add9d88a8 100644 --- a/backend/test/backend_tests/nitrate_ingest_audit_log_test.clj +++ b/backend/test/backend_tests/nitrate_ingest_audit_log_test.clj @@ -85,3 +85,27 @@ (t/is (string? (:tracked-at e1'))) (t/is (= (str now) (:created-at e1'))) (t/is (= (str team-id) (str (get-in e1' [:props :team-id]))))))))) + +(t/deftest ingest-audit-log-rejects-a-redirect + ;; A 3xx is a configuration error, not an empty success: it surfaces a + ;; typed error instead of the :malli.core/invalid-schema that a nil schema + ;; throws while building the coercer. + (binding [cf/flags (conj cf/flags :admin-console) + cf/config (assoc cf/config :admin-console-uri "http://ac.example/admin-console/")] + (with-mocks [http-mock {:target 'app.http.client/req + :return {:status 302 + :headers {"location" "http://ac.example/admin-console/api/audit-log"} + :body ""}}] + (let [client (ig/init-key :app.nitrate/client + {::http/client (Object.) + ::setup/shared-keys {:admin-console "test-shared-key"}}) + cfg {:app.nitrate/client client} + err (try + (nitrate/call cfg :ingest-audit-log + {::rpc/profile-id (uuid/next) + :events []}) + nil + (catch clojure.lang.ExceptionInfo e e))] + (t/is (= 1 (:call-count @http-mock))) + (t/is (= :nitrate-http-error (:type (ex-data err)))) + (t/is (= 302 (:status (ex-data err)))))))) diff --git a/backend/test/backend_tests/objects_gc_test.clj b/backend/test/backend_tests/objects_gc_test.clj index 462cbc5f5a..cd7e2a7548 100644 --- a/backend/test/backend_tests/objects_gc_test.clj +++ b/backend/test/backend_tests/objects_gc_test.clj @@ -206,7 +206,7 @@ ;; When the GC physically deletes a team, the projection must drop its row, ;; so the event carries the purged ids. (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] - (with-redefs [cf/flags (conj cf/flags :audit-log)] + (th/with-audit-config (conj cf/flags :nexus) (let [profile (th/create-profile* 1 {:is-active true}) team (th/create-team* 71 {:profile-id (:id profile)})] (th/command! {::th/type :delete-team @@ -218,8 +218,10 @@ (th/reset-mock! audit-mock) (th/run-task! :objects-gc {:skip-delay true}) (t/is (nil? (th/db-get :team {:id (:id team)} {::db/remove-deleted false}))) - (let [events (filter #(contains? (:props %) :team-ids) + (let [events (filter #(= #{:team-ids} (set (keys (:props %)))) (events-named audit-mock "hard-delete-objects"))] + ;; Every chunk carries exactly its own entity key and nothing + ;; else: no flat team-ids are resolved after the delete. (t/is (= 1 (count events))) (let [event (first events)] (t/is (= "command" (:type event))) @@ -228,7 +230,7 @@ (t/deftest gc-emits-hard-delete-objects-for-projects (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] - (with-redefs [cf/flags (conj cf/flags :audit-log)] + (th/with-audit-config (conj cf/flags :nexus) (let [profile (th/create-profile* 1 {:is-active true}) team (th/create-team* 72 {:profile-id (:id profile)}) project (th/create-project* 72 {:profile-id (:id profile) @@ -245,12 +247,13 @@ (t/is (= 1 (count events))) (let [event (first events)] (t/is (= "command" (:type event))) + (t/is (= #{:project-ids} (set (keys (:props event))))) (t/is (= [(:id project)] (get-in event [:props :project-ids]))) (t/is (= "objects-gc" (get-in event [:context :triggered-by]))))))))) (t/deftest gc-emits-hard-delete-objects-for-files (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] - (with-redefs [cf/flags (conj cf/flags :audit-log)] + (th/with-audit-config (conj cf/flags :nexus) (let [profile (th/create-profile* 1 {:is-active true}) team (th/create-team* 73 {:profile-id (:id profile)}) project (th/create-project* 73 {:profile-id (:id profile) @@ -269,10 +272,11 @@ (t/is (= 1 (count events))) (let [event (first events)] (t/is (= "command" (:type event))) + (t/is (= #{:file-ids} (set (keys (:props event))))) (t/is (= [(:id file)] (get-in event [:props :file-ids]))) (t/is (= "objects-gc" (get-in event [:context :triggered-by]))))))))) -(t/deftest gc-emits-no-event-without-audit-log-flag +(t/deftest gc-emits-no-event-without-a-consumer ;; The GC must stay silent when the audit log is off, even though the row is ;; still purged. (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] @@ -317,7 +321,7 @@ (t/deftest gc-emits-events-only-for-committed-chunks (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] - (with-redefs [cf/flags (conj cf/flags :audit-log)] + (th/with-audit-config (conj cf/flags :nexus) (let [profile (th/create-profile* 1) rolled-back (create-deleted-file! profile 1 (ct/in-past "30m")) blocked (create-deleted-file! profile 2 (ct/in-past "20m")) diff --git a/backend/test/backend_tests/rpc_audit_test.clj b/backend/test/backend_tests/rpc_audit_test.clj index 8ae01500ad..7c9135e776 100644 --- a/backend/test/backend_tests/rpc_audit_test.clj +++ b/backend/test/backend_tests/rpc_audit_test.clj @@ -14,6 +14,7 @@ [app.db :as db] [app.jobs :as-alias jobs] [app.loggers.audit :as audit] + [app.loggers.webhooks :as-alias webhooks] [app.main :as main] [app.metrics :as mtx] [app.metrics.definition :as-alias mdef] @@ -108,7 +109,7 @@ (assoc base :app.http/auth-key-id :nexus))))))) (t/deftest push-events-1 - (with-redefs [app.config/flags #{:audit-log :nexus}] + (th/with-audit-config #{:nexus} (let [prof (th/create-profile* 1 {:is-active true}) team-id (:default-team-id prof) proj-id (:default-project-id prof) @@ -138,12 +139,10 @@ (t/is (= 1 (count rows))) (t/is (= (:id prof) (:profile-id row))) (t/is (= "navigate" (:name row))) - (t/is (= "frontend" (:source row))) - ;; The row is born pending for the enabled consumer. - (t/is (= 1 (count (th/db-exec! ["select * from audit_log_pending"])))))))) + (t/is (= "frontend" (:source row))))))) (t/deftest push-events-2 - (with-redefs [app.config/flags #{:audit-log}] + (th/with-audit-config #{:nexus} (let [prof (th/create-profile* 1 {:is-active true}) team-id (:default-team-id prof) proj-id (:default-project-id prof) @@ -178,10 +177,10 @@ ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; (t/deftest push-events-telemetry-mode-stores-anonymized-row - ;; When telemetry is enabled and audit-log is NOT, frontend events + ;; When telemetry is enabled and no consumer persists rows, frontend events ;; must be stored with source="telemetry", empty props, zeroed ip, ;; and context filtered to safe keys only. - (with-redefs [cf/flags #{:telemetry}] + (th/with-audit-config #{:telemetry} (let [prof (th/create-profile* 1 {:is-active true}) team-id (:default-team-id prof) proj-id (:default-project-id prof) @@ -239,13 +238,13 @@ (t/is (not (contains? ctx :external-session-id)))))))) (t/deftest push-events-both-flags-creates-two-rows - ;; When both :audit-log and :telemetry flags are active, two rows + ;; When a consumer persists rows and :telemetry is active, two rows ;; should be stored: one full audit entry and one telemetry entry. ;; The profile is created before enabling :admin-console: team ;; bootstrap calls the real nitrate client, which is unconfigured ;; in tests. (let [prof (th/create-profile* 1 {:is-active true})] - (with-redefs [cf/flags #{:audit-log :telemetry :nexus :admin-console}] + (th/with-audit-config #{:telemetry :nexus :admin-console} (let [params {::th/type :push-audit-events ::rpc/profile-id (:id prof) :events [{:name "navigate" @@ -273,16 +272,14 @@ (t/is (= "0.0.0.0" (str (:ip-addr row2)))) (let [day-now (ct/truncate (ct/now) :days)] (t/is (= day-now (:created-at row2))) - (t/is (= day-now (:tracked-at row2)))) - ;; Both rows are born pending for the enabled consumers. - (t/is (= 4 (count (th/db-exec! ["select * from audit_log_pending"]))))))))) + (t/is (= day-now (:tracked-at row2))))))))) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; BACKEND PROCESS-EVENT PATH (RPC commands) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; -(t/deftest backend-process-event-only-audit-log - (with-redefs [cf/flags #{:audit-log}] +(t/deftest backend-process-event-only-persisted-rows + (th/with-audit-config #{:nexus} (let [prof (th/create-profile* 1 {:is-active true}) event {:id (uuid/next) :type "action" @@ -302,7 +299,7 @@ (t/is (not= "0.0.0.0" (str (:ip-addr row)))))))) (t/deftest backend-process-event-only-telemetry - (with-redefs [cf/flags #{:telemetry}] + (th/with-audit-config #{:telemetry} (let [prof (th/create-profile* 1 {:is-active true}) event {:id (uuid/next) :type "action" @@ -321,10 +318,10 @@ (t/is (= "0.0.0.0" (str (:ip-addr row)))))))) (t/deftest backend-process-event-both-flags-creates-two-rows - ;; When both :audit-log and :telemetry are active, the backend + ;; When persistence and :telemetry are active, the backend ;; process-event must store two rows: one full audit entry and one ;; telemetry entry. - (with-redefs [cf/flags #{:audit-log :telemetry}] + (th/with-audit-config #{:nexus :telemetry} (let [prof (th/create-profile* 1 {:is-active true}) event {:id (uuid/next) :type "action" @@ -357,7 +354,7 @@ ;; the id used to be generated inside `append-audit-entry` without ;; ever reaching the worker params, so `:event-id` always landed nil. (with-mocks [submit-mock {:target 'app.jobs/submit :return nil}] - (with-redefs [cf/flags #{:audit-log :webhooks}] + (th/with-audit-config #{:nexus :webhooks} (let [prof (th/create-profile* 1 {:is-active true}) event {:type "action" :name "delete-project" @@ -379,11 +376,11 @@ (t/is (= (:id row) (:id task-event))))))))) (t/deftest backend-process-event-telemetry-row-gets-own-id - ;; When both `:audit-log` and `:telemetry` are active, the shadow + ;; When persistence and `:telemetry` are active, the shadow ;; telemetry row must not reuse the full row id (shared PK), while ;; the webhook task still carries the full row id. (with-mocks [submit-mock {:target 'app.jobs/submit :return nil}] - (with-redefs [cf/flags #{:audit-log :telemetry :webhooks}] + (th/with-audit-config #{:nexus :telemetry :webhooks} (let [prof (th/create-profile* 1 {:is-active true}) event {:type "action" :name "delete-project" @@ -404,10 +401,47 @@ (t/is (= (:id (first (filter #(= "backend" (:source %)) rows))) (:id task-event))))))))) +(t/deftest process-event-stores-no-row-without-a-consumer + ;; No cursor consumer means no full row, even when a webhook is active: + ;; the webhook job carries the event blob and does not read the buffer. + (with-mocks [submit-mock {:target 'app.jobs/submit :return nil}] + (th/with-audit-config #{:webhooks} + (let [prof (th/create-profile* 1 {:is-active true}) + event {:type "action" + :name "delete-project" + :profile-id (:id prof) + :props {:team-id (:default-team-id prof)} + :context {:version "2.0.0" :initiator "app"} + :tracked-at (ct/now) + :created-at (ct/now) + :source "backend" + :app.loggers.webhooks/event? true}] + (audit/submit* th/*system* event) + (t/is (= 0 (count (th/db-exec! ["select * from audit_log"])))) + (t/is (= 1 (:call-count @submit-mock))))))) + +(t/deftest process-event-stores-row-with-a-single-consumer + ;; A single consumer is enough to make the backend persist full rows. + (th/with-audit-config #{:nexus} + (let [prof (th/create-profile* 1 {:is-active true}) + event {:type "action" + :name "delete-project" + :profile-id (:id prof) + :props {:team-id (:default-team-id prof)} + :context {:version "2.0.0" :initiator "app"} + :tracked-at (ct/now) + :created-at (ct/now) + :source "backend"}] + (audit/submit* th/*system* event) + (let [[row :as rows] (->> (th/db-exec! ["select * from audit_log"]) + (mapv decode-row))] + (t/is (= 1 (count rows))) + (t/is (= "backend" (:source row))))))) + (t/deftest push-events-disabled-when-no-flags-and-no-telemetry - ;; When neither audit-log nor telemetry is enabled, no rows should + ;; When neither persistence nor telemetry is enabled, no rows should ;; be stored. - (with-redefs [cf/flags #{}] + (th/with-audit-config #{} (let [prof (th/create-profile* 1 {:is-active true}) params {::th/type :push-audit-events ::rpc/profile-id (:id prof) @@ -422,37 +456,59 @@ (t/is (nil? (:error out))) (t/is (= 0 (count (th/db-exec! ["select * from audit_log"]))))))) +(t/deftest push-events-error-reporting-without-a-consumer + ;; :error-reporting tells the frontend to collect events, so the ingest + ;; must accept the batch and hand crash events to the reporters even + ;; with no cursor consumer and no telemetry. No rows are stored. + (with-mocks [db-emit {:target 'app.loggers.database/emit :return nil} + mm-emit {:target 'app.loggers.mattermost/emit :return nil}] + (th/with-audit-config #{:error-reporting} + (let [prof (th/create-profile* 1 {:is-active true}) + params {::th/type :push-audit-events + ::rpc/profile-id (:id prof) + :events [{:name "unhandled-exception" + :props {:hint "crash"} + :timestamp (ct/now) + :type "action"}]} + params (with-meta params + {:app.http/request http-request}) + out (th/command! params)] + (t/is (nil? (:error out))) + (t/is (= 1 (:call-count @db-emit))) + (t/is (= 1 (:call-count @mm-emit))) + (t/is (= 0 (count (th/db-exec! ["select * from audit_log"])))))))) + ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; GET-ENVIRONMENT-DATA ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; (t/deftest get-environment-data-returns-selfhost-by-default - (with-redefs [cf/flags #{:audit-log}] + (th/with-audit-config #{:nexus} (let [out (th/command! {::th/type :get-environment-data})] (t/is (th/success? out)) (t/is (= "selfhost" (-> out :result :deployment)))))) (t/deftest get-environment-data-returns-saas-when-configured - (with-redefs [cf/flags #{:telemetry}] + (th/with-audit-config #{:telemetry} (binding [cf/config (assoc cf/config :is-saas true)] (let [out (th/command! {::th/type :get-environment-data})] (t/is (th/success? out)) (t/is (= "saas" (-> out :result :deployment))))))) (t/deftest get-environment-data-only-exposes-event-flags - (with-redefs [cf/flags #{:audit-log :telemetry :graph :nrepl-server}] + (th/with-audit-config #{:nexus :telemetry :graph :nrepl-server} (let [out (th/command! {::th/type :get-environment-data})] (t/is (th/success? out)) (t/is (= #{:audit-log :telemetry} (-> out :result :flags)))))) (t/deftest get-environment-data-returns-empty-flags-when-none-enabled - (with-redefs [cf/flags #{:graph}] + (th/with-audit-config #{:graph} (let [out (th/command! {::th/type :get-environment-data})] (t/is (th/success? out)) (t/is (= #{} (-> out :result :flags)))))) (t/deftest get-enabled-flags-still-returns-flat-set - (with-redefs [cf/flags #{:audit-log :telemetry :graph}] + (th/with-audit-config #{:nexus :telemetry :graph} (let [out (th/command! {::th/type :get-enabled-flags})] (t/is (th/success? out)) (t/is (= #{:audit-log :telemetry} (:result out)))))) @@ -655,7 +711,7 @@ (t/is (= "app" (audit/get-client-initiator (request "something-else/1.0")))))) (t/deftest push-events-assigns-initiator-from-x-client - (with-redefs [app.config/flags #{:audit-log}] + (th/with-audit-config #{:nexus} (let [prof (th/create-profile* 1 {:is-active true}) params {::th/type :push-audit-events ::rpc/profile-id (:id prof) @@ -677,7 +733,7 @@ (t/is (= "admin-console" (-> row :context :initiator))))))) (t/deftest push-events-initiator-defaults-to-app - (with-redefs [app.config/flags #{:audit-log}] + (th/with-audit-config #{:nexus} (let [prof (th/create-profile* 1 {:is-active true}) params {::th/type :push-audit-events ::rpc/profile-id (:id prof) @@ -698,15 +754,15 @@ ;; INSERT DEFAULTS ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; -(t/deftest insert-only-runs-with-audit-log-flag - ;; insert must be a no-op when :audit-log flag is not set +(t/deftest insert-only-runs-when-a-consumer-is-enabled + ;; insert must be a no-op when no consumer is enabled (with-redefs [app.config/flags #{:telemetry}] (audit/insert th/*system* {:name "test" :type "action"}) (t/is (= 0 (count (th/db-exec! ["select * from audit_log"])))))) (t/deftest insert-sets-defaults - ;; insert must set defaults and persist when :audit-log is set - (with-redefs [app.config/flags #{:audit-log}] + ;; insert must set defaults and persist when a consumer is enabled + (th/with-audit-config #{:nexus} (audit/insert th/*system* {:name "test-action" :type "action"}) (let [[row] (->> (th/db-exec! ["select * from audit_log"]) (mapv decode-row))] @@ -853,8 +909,9 @@ (t/deftest prepare-rpc-event-coerces-string-metadata-profile-id ;; `::audit/profile-id` is set by hand in a dozen commands, and some of ;; them read the value from token claims or other stringly-typed sources. - ;; The audit schema demands a uuid, and `submit*` swallows the validation - ;; error, so an unconverted string would drop the event on the floor. + ;; The audit schema demands a uuid, and `submit*` propagates the validation + ;; error, so an unconverted string would fail the response instead of only + ;; losing the event. (let [caller (th/create-profile* 1 {:is-active true}) target "33601240-a00b-11ea-ba1b-c554cc60e361" result (with-meta {:some-data "value"} @@ -926,18 +983,17 @@ (t/deftest push-handle-events-records-inserted (let [prof (th/create-profile* 1 {:is-active true}) metrics (make-ingest-metrics)] - (with-redefs [cf/flags #{:audit-log :telemetry :nexus :admin-console}] + (th/with-audit-config #{:telemetry :nexus :admin-console} (@#'cmd-audit/handle-events {::db/pool th/*pool* ::mtx/metrics metrics} (push-params (:id prof))) (t/is (= 1.0 (ingest-counter metrics "full" "inserted"))) (t/is (= 1.0 (ingest-counter metrics "shadow" "inserted"))) - (t/is (= 2 (count (th/db-exec! ["select * from audit_log"])))) - (t/is (= 4 (count (th/db-exec! ["select * from audit_log_pending"]))))))) + (t/is (= 2 (count (th/db-exec! ["select * from audit_log"]))))))) (t/deftest push-handle-events-records-failed-on-error (let [prof (th/create-profile* 1 {:is-active true}) metrics (make-ingest-metrics)] - (with-redefs [cf/flags #{:audit-log :telemetry :nexus :admin-console}] + (th/with-audit-config #{:telemetry :nexus :admin-console} (with-mocks [insert-mock {:target 'app.db/insert-many! :throw (ex-info "boom" {})}] (t/is (thrown? clojure.lang.ExceptionInfo @@ -945,19 +1001,51 @@ (push-params (:id prof))))) (t/is (= 1.0 (ingest-counter metrics "full" "failed"))) (t/is (= 1.0 (ingest-counter metrics "shadow" "failed"))) - (t/is (= 0 (count (th/db-exec! ["select * from audit_log"])))) - (t/is (= 0 (count (th/db-exec! ["select * from audit_log_pending"])))))))) + (t/is (= 0 (count (th/db-exec! ["select * from audit_log"])))))))) (t/deftest push-handle-events-single-face-emits-no-empty-series - ;; With only :audit-log on, exactly one ingest series exists: the + ;; With a consumer enabled and telemetry off, exactly one ingest series exists: the ;; pos? guard in record-ingested emits no phantom shadow series. ;; (No send flags either, so R3 leaves the row pending-less.) (let [prof (th/create-profile* 1 {:is-active true}) metrics (make-ingest-metrics)] - (with-redefs [cf/flags #{:audit-log}] + (th/with-audit-config #{:nexus} (@#'cmd-audit/handle-events {::db/pool th/*pool* ::mtx/metrics metrics} (push-params (:id prof))) (t/is (= 1.0 (ingest-counter metrics "full" "inserted"))) (t/is (= #{["full" "inserted"]} (counter-label-sets metrics))) - (t/is (= 1 (count (th/db-exec! ["select * from audit_log"])))) - (t/is (= 0 (count (th/db-exec! ["select * from audit_log_pending"]))))))) + (t/is (= 1 (count (th/db-exec! ["select * from audit_log"]))))))) + +(t/deftest backend-webhook-batch-debounces-on-the-pending-job + ;; Two events of the same batch key share one pending job: the second + ;; moves its schedule and leaves the last payload behind, so the + ;; receiver is told once about the burst and with the newest event. + (th/with-audit-config #{:webhooks} + (let [prof (th/create-profile* 1 {:is-active true}) + label "rpc:update-file:the-key" + event (fn [n batch-timeout] + {:id (uuid/next) + :type "action" + :name "update-file" + :profile-id (:id prof) + :props {:n n} + :context {:version "2.0.0" :initiator "app"} + :tracked-at (ct/now) + :created-at (ct/now) + :source "backend" + ::webhooks/event? true + ::webhooks/batch-key "the-key" + ::webhooks/batch-timeout batch-timeout})] + (audit/submit* th/*system* (event 1 (ct/duration "2m"))) + (let [[row1 :as rows] (th/db-exec! ["select * from job where label = ?" label])] + (t/is (= 1 (count rows))) + (audit/submit* th/*system* (event 2 (ct/duration "1h"))) + (let [[row2 :as rows] (th/db-exec! ["select * from job where label = ?" label])] + (t/is (= 1 (count rows)) "the burst keeps a single pending job") + (t/is (= (:id row1) (:id row2)) "and the job keeps its id") + (t/is (pos? (compare (:scheduled-at row2) (:scheduled-at row1))) + "the second event slides the window to its own delay") + (let [params (db/decode-json-pgobject (:params row2)) + blob (transit/decode-str (:event-blob params))] + (t/is (= 2 (get-in blob [:props :n])) + "and the payload delivered is the last event"))))))) diff --git a/backend/test/backend_tests/rpc_binfile_test.clj b/backend/test/backend_tests/rpc_binfile_test.clj index 0a829d23db..be8bdbb85f 100644 --- a/backend/test/backend_tests/rpc_binfile_test.clj +++ b/backend/test/backend_tests/rpc_binfile_test.clj @@ -128,7 +128,7 @@ ;; The imported files are created inside the importer, so the command emits a ;; create-file event per file for a local projection. (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] - (with-redefs [cf/flags (conj cf/flags :audit-log)] + (th/with-audit-config (conj cf/flags :nexus) (let [profile (th/create-profile* 1 {:is-active true}) team (th/create-team* 51 {:profile-id (:id profile)}) project (th/create-project* 51 {:profile-id (:id profile) diff --git a/backend/test/backend_tests/rpc_file_test.clj b/backend/test/backend_tests/rpc_file_test.clj index 81b501585e..376f412b1b 100644 --- a/backend/test/backend_tests/rpc_file_test.clj +++ b/backend/test/backend_tests/rpc_file_test.clj @@ -24,14 +24,16 @@ [app.db.sql :as sql] [app.features.fdata :as fdata] [app.http :as http] - [app.loggers.audit :as-alias audit] + [app.jobs :as-alias jobs] + [app.loggers.audit :as audit] [app.redis :as rds] [app.rpc :as-alias rpc] [app.rpc.commands.files :as files] [app.storage :as sto] [backend-tests.helpers :as th] [clojure.test :as t] - [cuerdas.core :as str])) + [cuerdas.core :as str] + [mockery.core :refer [with-mocks]])) (t/use-fixtures :once th/state-init) (t/use-fixtures :each th/database-reset) @@ -2433,15 +2435,164 @@ :id (:id library)})] (t/is (nil? (:error out))) - ;; The relations are removed at once, the absorption is done by the - ;; deletion task. - (t/is (empty? (th/db-query :file-library-rel {:library-file-id (:id library)}))) + ;; The relations stay until the deletion task runs: the job + ;; captures the dependents, removes the relations and absorbs + ;; them there. + (t/is (not (empty? (th/db-query :file-library-rel {:library-file-id (:id library)})))) (t/is (= (:id library) (cfo/get-tokens-source (:data (bfc/get-file th/*system* (:id file)))))) (th/run-pending-jobs) + (t/is (empty? (th/db-query :file-library-rel {:library-file-id (:id library)}))) (check-tokens-absorbed prepared))) +(t/deftest delete-shared-library-absorb-failure-keeps-cascade-going + ;; If the absorb blows up, the savepoint rolls it back and the + ;; cascade goes on: the file is still marked deleted and the job + ;; succeeds. The relations are dropped before the absorb runs, so + ;; they do not survive a failed absorb; the dependents keep the data + ;; they had before the absorb was attempted. + (let [profile (th/create-profile* 1) + {:keys [library]} (prepare-tokens-library profile) + out (th/command! {::th/type :delete-file + ::rpc/profile-id (:id profile) + :id (:id library)})] + (t/is (nil? (:error out))) + (with-redefs [files/absorb-library (fn [& _] (throw (ex-info "boom" {})))] + (th/run-pending-jobs)) + (t/is (empty? (th/db-query :file-library-rel {:library-file-id (:id library)}))) + (t/is (some? (:deleted-at (th/db-get :file {:id (:id library)} {::db/remove-deleted false})))))) + +(t/deftest delete-shared-library-queues-job-without-file-ids + ;; The cascade captures the dependents itself: the RPC no longer + ;; snapshots them into the job params. + (let [profile (th/create-profile* 1) + {:keys [library]} (prepare-tokens-library profile) + submitted (atom nil)] + (with-redefs [jobs/submit (fn [_ params] (reset! submitted params) (uuid/next))] + (let [out (th/command! {::th/type :delete-file + ::rpc/profile-id (:id profile) + :id (:id library)})] + (t/is (nil? (:error out))))) + (t/is (= :delete-object (::jobs/name @submitted))) + (t/is (not (contains? (::jobs/params @submitted) :file-ids))))) + +(t/deftest delete-project-skips-absorb-for-files-dying-together + ;; When the whole project dies, absorbing the shared library into + ;; its own dying files is wasted work: the cascade cleans the + ;; relations but leaves the dependents untouched. + (let [profile (th/create-profile* 1) + team-id (:default-team-id profile) + proj (th/create-project* 1 {:profile-id (:id profile) :team-id team-id}) + library (th/create-file* 1 {:profile-id (:id profile) + :project-id (:id proj) + :is-shared true}) + file (th/create-file* 2 {:profile-id (:id profile) + :project-id (:id proj)}) + _ (th/link-file-to-library* {:file-id (:id file) + :library-id (:id library)}) + revn (:revn (th/db-get :file {:id (:id file)}))] + (let [out (th/command! {::th/type :delete-project + ::rpc/profile-id (:id profile) + :id (:id proj)})] + (t/is (nil? (:error out)))) + (th/run-pending-jobs) + ;; Nothing outside the cascade uses the library, so the relations + ;; stay until the hard delete removes them with their files. + (t/is (not (empty? (th/db-query :file-library-rel {:library-file-id (:id library)})))) + (t/is (= revn (:revn (th/db-get :file {:id (:id file)} {::db/remove-deleted false})))) + ;; The hard delete cascades the relations away with their files. + (th/run-task! :objects-gc {:skip-delay true}) + (t/is (nil? (th/db-get :file {:id (:id file)} {::db/remove-deleted false}))) + (t/is (empty? (th/db-query :file-library-rel {:library-file-id (:id library)}))))) + +(t/deftest delete-object-absorbs-with-deprecated-file-ids + ;; Deploy window: a job queued by the previous release carries the + ;; dependents in :file-ids and its RPC already deleted the relations. + ;; The live lookup finds nothing, so the worker must trust the + ;; snapshot instead of skipping the absorb silently. + (let [profile (th/create-profile* 6 {:is-active true}) + team-id (:default-team-id profile) + proj (th/create-project* 6 {:profile-id (:id profile) :team-id team-id}) + library (th/create-file* 8 {:profile-id (:id profile) + :project-id (:id proj) + :is-shared true}) + dependent (th/create-file* 9 {:profile-id (:id profile) + :project-id (:id proj)}) + _ (th/link-file-to-library* {:file-id (:id dependent) + :library-id (:id library)}) + revn (:revn (th/db-get :file {:id (:id dependent)}))] + ;; What the old RPC did: drop the relations before queueing. + (th/db-delete! :file-library-rel {:library-file-id (:id library)}) + (th/run-task! :delete-object {:object :file + :id (:id library) + :deleted-at (ct/now) + :file-ids [(:id dependent)]}) + (t/is (> (:revn (th/db-get :file {:id (:id dependent)} {::db/remove-deleted false})) + revn)))) + +(t/deftest delete-project-absorbs-into-surviving-dependent + ;; The doomed-set only skips files that die in the same cascade: a + ;; dependent in another project of the team survives, so the library + ;; must be absorbed into it (and only into it). + (let [profile (th/create-profile* 5 {:is-active true}) + team-id (:default-team-id profile) + proj-a (th/create-project* 5 {:profile-id (:id profile) :team-id team-id}) + proj-b (th/create-project* 6 {:profile-id (:id profile) :team-id team-id}) + library (th/create-file* 5 {:profile-id (:id profile) + :project-id (:id proj-a) + :is-shared true}) + doomed (th/create-file* 6 {:profile-id (:id profile) + :project-id (:id proj-a)}) + survivor (th/create-file* 7 {:profile-id (:id profile) + :project-id (:id proj-b)}) + _ (th/link-file-to-library* {:file-id (:id survivor) + :library-id (:id library)}) + revn-sur (:revn (th/db-get :file {:id (:id survivor)})) + revn-doom (:revn (th/db-get :file {:id (:id doomed)}))] + (let [out (th/command! {::th/type :delete-project + ::rpc/profile-id (:id profile) + :id (:id proj-a)})] + (t/is (nil? (:error out)))) + (th/run-pending-jobs) + ;; The survivor absorbed the library; its relation is gone. + (t/is (> (:revn (th/db-get :file {:id (:id survivor)} {::db/remove-deleted false})) + revn-sur)) + (t/is (empty? (th/db-query :file-library-rel {:file-id (:id survivor)}))) + ;; The doomed file did not absorb anything. + (t/is (= revn-doom (:revn (th/db-get :file {:id (:id doomed)} {::db/remove-deleted false})))))) + +(t/deftest delete-team-skips-absorb-for-files-dying-together + ;; Team deletions skip absorb entirely (*team-deletion*): a library + ;; in project A used from project B dies with the team, so absorb + ;; never runs and the relations stay until the hard delete. + (let [profile (th/create-profile* 4 {:is-active true}) + team (th/create-team* 4 {:profile-id (:id profile)}) + proj-a (th/create-project* 4 {:profile-id (:id profile) :team-id (:id team)}) + proj-b (th/create-project* 5 {:profile-id (:id profile) :team-id (:id team)}) + library (th/create-file* 6 {:profile-id (:id profile) + :project-id (:id proj-a) + :is-shared true}) + file (th/create-file* 7 {:profile-id (:id profile) + :project-id (:id proj-b)}) + _ (th/link-file-to-library* {:file-id (:id file) + :library-id (:id library)}) + revn (:revn (th/db-get :file {:id (:id file)})) + calls (atom 0)] + (let [out (th/command! {::th/type :delete-team + ::rpc/profile-id (:id profile) + :id (:id team)})] + (t/is (nil? (:error out)))) + (with-redefs [files/absorb-library (fn [& _] (swap! calls inc) nil)] + (th/run-pending-jobs)) + (t/is (zero? @calls)) + (t/is (not (empty? (th/db-query :file-library-rel {:library-file-id (:id library)})))) + (t/is (= revn (:revn (th/db-get :file {:id (:id file)} {::db/remove-deleted false})))) + ;; The hard delete cascades the relations away with their files. + (th/run-task! :objects-gc {:skip-delay true}) + (t/is (nil? (th/db-get :file {:id (:id file)} {::db/remove-deleted false}))) + (t/is (empty? (th/db-query :file-library-rel {:library-file-id (:id library)}))))) + (t/deftest unpublish-shared-library-absorbs-its-tokens-lib (let [profile (th/create-profile* 1) {:keys [library] :as prepared} (prepare-tokens-library profile) @@ -3348,6 +3499,71 @@ (t/is (= (:id team) (get-in (meta (:result out)) [:app.loggers.audit/props :team-id]))))) +(defn- unused-media-fixture + "A file with a media object nothing references, so `:file-gc` trims it." + [i] + (let [profile (th/create-profile* i) + file (th/create-file* i {:profile-id (:id profile) + :project-id (:default-project-id profile) + :is-shared false}) + so-id (uuid/next)] + (th/db-insert! :storage-object {:id so-id :size 10 :backend "test"}) + (th/create-file-media-object* {:file-id (:id file) + :media-id so-id + :name "unused-media" + :mtype "image/svg+xml"}) + [profile file])) + +(t/deftest file-gc-emits-an-audit-event-when-it-trims-media + ;; Trimming media is a destructive action no RPC reports: the task emits + ;; `file-gc` with the file and its team so the audit trail records it. + (let [[profile file] (unused-media-fixture 1)] + (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] + (th/with-audit-config (conj cf/flags :nexus) + (t/is (true? (th/run-task! :file-gc {:file-id (:id file)}))) + (let [event (->> (:call-args-list @audit-mock) + (map second) + (filter #(= "file-gc" (:name %))) + first)] + (t/is (some? event)) + (t/is (= (:id file) (get-in event [:props :file-id]))) + (t/is (= (:default-team-id profile) (get-in event [:props :team-id]))) + (t/is (= "cron-task" (get-in event [:context :triggered-by])))))))) + +(t/deftest file-gc-does-not-emit-an-audit-event-with-the-flag-off + (let [[_ file] (unused-media-fixture 2)] + (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] + (th/with-audit-config (disj cf/flags :nexus) + (t/is (true? (th/run-task! :file-gc {:file-id (:id file)}))) + (t/is (empty? (:call-args-list @audit-mock))))))) + +(t/deftest update-file-audit-event-carries-the-file-change-id + ;; The event says that a file was saved, but not what changed: the + ;; change row (xlog) is the only place with that, so its id travels + ;; in the event. A consumer needs it to read the change (with + ;; `::db/remove-deleted false`: the row is born with its `deleted_at` + ;; in the future). + (let [profile (th/create-profile* 1 {:is-active true}) + file (th/create-file* 1 {:profile-id (:id profile) + :project-id (:default-project-id profile) + :is-shared false}) + result (update-file! :file-id (:id file) + :profile-id (:id profile) + :revn 0 + :changes []) + props (get-in (meta result) [:app.loggers.audit/replace-props]) + change-id (:file-change-id props) + rows (th/db-query :file-change {:file-id (:id file)}) + ;; The save writes two change rows: the change itself (no + ;; label) and the snapshot of the previous state (with one). + changes (remove :label rows)] + ;; The event points at the change row of this save. + (t/is (= 1 (count changes))) + (t/is (= (:id (first changes)) change-id)) + ;; The prop survives `clean-props`: a plain key with a uuid value. + (t/is (= change-id + (:file-change-id (audit/clean-props props)))))) + (t/deftest get-team-shared-files-skips-files-that-fail-to-load (let [profile (th/create-profile* 1) file1 (th/create-file* 1 {:profile-id (:id profile) diff --git a/backend/test/backend_tests/rpc_file_update_test.clj b/backend/test/backend_tests/rpc_file_update_test.clj index 34e4fe8b00..bf0df5c285 100644 --- a/backend/test/backend_tests/rpc_file_update_test.clj +++ b/backend/test/backend_tests/rpc_file_update_test.clj @@ -13,6 +13,7 @@ [app.common.logging :as l] [app.common.uuid :as uuid] [app.config :as cf] + [app.loggers.audit :as-alias audit] [app.msgbus :as mbus] [app.rpc :as-alias rpc] [app.util.blob :as blob] @@ -147,6 +148,59 @@ (t/is (= 1 (count (get-file-changes ctx)))) (t/is (empty? messages)))) +(t/deftest update-file-audit-event-carries-the-change-id-only-when-written + ;; `:file-change-id` points at the revision row of the save. A save whose + ;; first commit is rejected and keeps no prefix writes no row, so its + ;; event must not carry the id: an id that resolves to nothing cannot be + ;; told apart from a row the GC already purged. `command-raw!` keeps the + ;; result metadata, which is where the command leaves the audit props. + (let [{:keys [profile-id file-id page-id] :as ctx} (create-file) + shape (uuid/next) + save! (fn [params] + (th/command-raw! (merge {::th/type :update-file + ::rpc/profile-id profile-id + :id file-id + :session-id (uuid/next) + :vern 0 + :features cfeat/supported-features} + params))) + props (fn [out] (::audit/replace-props (meta out)))] + + (t/testing "a save that writes the revision carries its id" + (let [out (save! {:revn 0 + :changes [(th/add-rect-change page-id shape)]}) + id (:file-change-id (props out))] + (t/is (uuid? id)) + (t/is (= [id] (mapv :id (th/db-query :file-change {:id id})))) + (t/is (= "saved" (:outcome (props out)))) + (t/is (inst? (:modified-at (props out)))))) + + (t/testing "a fully rejected save carries no id and says so" + (let [commit1 (commit (break-parent page-id shape)) + commit2 (commit (th/add-rect-change page-id (uuid/next))) + out (save! {:revn 1 + :changes-with-metadata [commit1 commit2]})] + (t/is (some? (:rejected out))) + (t/is (= 1 (:revn (th/db-get :file {:id file-id})))) + (t/is (= 1 (count (get-file-changes ctx)))) + (t/is (not (contains? (props out) :file-change-id))) + (t/is (= "rejected" (:outcome (props out)))) + ;; The rejected save does not persist: the event mirrors the row. + (t/is (= (inst-ms (:modified-at (th/db-get :file {:id file-id}))) + (inst-ms (:modified-at (props out))))))) + + (t/testing "a partially rejected save says so and carries the id" + (let [commit1 (commit (th/add-rect-change page-id (uuid/next))) + commit2 (commit (conflicting-page-change)) + out (save! {:revn 1 + :changes-with-metadata [commit1 commit2]}) + row (th/db-get :file {:id file-id})] + (t/is (some? (:rejected out))) + (t/is (= "partial" (:outcome (props out)))) + (t/is (uuid? (:file-change-id (props out)))) + (t/is (= (inst-ms (:modified-at row)) + (inst-ms (:modified-at (props out))))))))) + (t/deftest rejects-a-commit-that-throws-while-applied (let [{:keys [page-id] :as ctx} (create-file) shape-a (uuid/next) diff --git a/backend/test/backend_tests/rpc_management_nitrate_test.clj b/backend/test/backend_tests/rpc_management_nitrate_test.clj index 60eae1c707..915f3ea2f1 100644 --- a/backend/test/backend_tests/rpc_management_nitrate_test.clj +++ b/backend/test/backend_tests/rpc_management_nitrate_test.clj @@ -2135,7 +2135,7 @@ (t/deftest push-audit-events-stores-backend-source (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] - (binding [cf/flags #{:audit-log}] + (th/with-audit-config #{:nexus} (let [prof (th/create-profile* 1 {:is-active true}) params {::th/type :push-audit-events :events [{:name "test-action" @@ -2160,7 +2160,7 @@ (t/deftest push-audit-events-type-defaults-to-action (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] - (binding [cf/flags #{:audit-log}] + (th/with-audit-config #{:nexus} (let [prof (th/create-profile* 1 {:is-active true}) params {::th/type :push-audit-events :events [{:name "no-type-event" @@ -2178,7 +2178,7 @@ (t/deftest push-audit-events-multiple-events (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] - (binding [cf/flags #{:audit-log}] + (th/with-audit-config #{:nexus} (let [prof (th/create-profile* 1 {:is-active true}) params {::th/type :push-audit-events :events [{:name "event-a" @@ -2201,7 +2201,7 @@ (t/deftest push-audit-events-merges-context (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] - (binding [cf/flags #{:audit-log}] + (th/with-audit-config #{:nexus} (let [prof (th/create-profile* 1 {:is-active true}) params {::th/type :push-audit-events :events [{:name "context-test" @@ -2227,7 +2227,7 @@ ;; keyword; the stored initiator must be a plain string, and a ;; caller-supplied initiator must never survive (server context wins). (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] - (binding [cf/flags #{:audit-log}] + (th/with-audit-config #{:nexus} (let [prof (th/create-profile* 1 {:is-active true}) params {::th/type :push-audit-events :events [{:name "context-test" diff --git a/backend/test/backend_tests/rpc_management_test.clj b/backend/test/backend_tests/rpc_management_test.clj index 9eefb5ef28..c1b81a32d5 100644 --- a/backend/test/backend_tests/rpc_management_test.clj +++ b/backend/test/backend_tests/rpc_management_test.clj @@ -538,6 +538,71 @@ (t/is (= (:library-file-id item) (:id file3))))))) +(t/deftest move-files-from-multiple-teams-is-rejected + ;; A single move carries one origin team in its props: mixed origins + ;; are refused before anything moves, and the move event is not + ;; submitted. The fixtures are created with the flag off so their own + ;; membership events do not pollute the mock. + (let [profile (th/create-profile* 1 {:is-active true}) + team (th/create-team* 1 {:profile-id (:id profile)}) + project1 (th/create-project* 1 {:team-id (:default-team-id profile) + :profile-id (:id profile)}) + project2 (th/create-project* 2 {:team-id (:id team) + :profile-id (:id profile)}) + file1 (th/create-file* 1 {:profile-id (:id profile) + :project-id (:id project1)}) + file3 (th/create-file* 3 {:profile-id (:id profile) + :project-id (:id project2)})] + (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] + (th/with-audit-config (conj cf/flags :nexus) + (let [out (th/command! {::th/type :move-files + ::rpc/profile-id (:id profile) + :project-id (:id project2) + :ids #{(:id file1) (:id file3)}}) + error (:error out)] + (t/is (th/ex-info? error)) + (t/is (th/ex-of-type? error :validation)) + (t/is (th/ex-of-code? error :cant-move-from-multiple-teams)))) + (t/is (not-any? #(= "move-files" (:name %)) + (map second (:call-args-list @audit-mock))))) + (t/is (= 1 (count (th/db-query :file {:project-id (:id project1)})))) + (t/is (= 1 (count (th/db-query :file {:project-id (:id project2)})))))) + + +(t/deftest move-files-carries-both-teams-in-the-event + ;; The event carries the origin and destination team, so a consumer can + ;; key on them without joining production tables. + (let [profile (th/create-profile* 1 {:is-active true}) + team (th/create-team* 1 {:profile-id (:id profile)}) + project1 (th/create-project* 1 {:team-id (:default-team-id profile) + :profile-id (:id profile)}) + project2 (th/create-project* 2 {:team-id (:id team) + :profile-id (:id profile)}) + file (th/create-file* 1 {:profile-id (:id profile) + :project-id (:id project1)}) + raw (th/command-raw! {::th/type :move-files + ::rpc/profile-id (:id profile) + :project-id (:id project2) + :ids #{(:id file)}}) + props (get-in (meta raw) [:app.loggers.audit/props])] + (t/is (= (:default-team-id profile) (:source-team-id props))) + (t/is (= (:id team) (:dest-team-id props))))) + + +(t/deftest move-project-carries-both-teams-in-the-event + (let [profile (th/create-profile* 1 {:is-active true}) + team (th/create-team* 2 {:profile-id (:id profile)}) + project (th/create-project* 1 {:team-id (:default-team-id profile) + :profile-id (:id profile)}) + raw (th/command-raw! {::th/type :move-project + ::rpc/profile-id (:id profile) + :project-id (:id project) + :team-id (:id team)}) + props (get-in (meta raw) [:app.loggers.audit/props])] + (t/is (= (:default-team-id profile) (:source-team-id props))) + (t/is (= (:id team) (:dest-team-id props))))) + + (t/deftest move-library-to-other-team (let [profile (th/create-profile* 1 {:is-active true}) team (th/create-team* 1 {:profile-id (:id profile)}) @@ -729,7 +794,7 @@ ;; The per-file `create-file` events are built by hand, so they do not pass ;; through the generic resolver: the command must copy the team id itself. (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] - (with-redefs [cf/flags (conj cf/flags :audit-log)] + (th/with-audit-config (conj cf/flags :nexus) (let [prof (th/create-profile* 1 {:is-active true}) team (th/create-team* 1 {:profile-id (:id prof)}) project (th/create-project* 1 {:profile-id (:id prof) @@ -750,7 +815,7 @@ ;; The duplicated project and files are created by internal calls, so the ;; command emits one create event per entity for a local projection. (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] - (with-redefs [cf/flags (conj cf/flags :audit-log)] + (th/with-audit-config (conj cf/flags :nexus) (let [profile (th/create-profile* 1 {:is-active true}) team (th/create-team* 52 {:profile-id (:id profile)}) project (th/create-project* 52 {:profile-id (:id profile) diff --git a/backend/test/backend_tests/rpc_team_test.clj b/backend/test/backend_tests/rpc_team_test.clj index a59d2759da..d4482a6e89 100644 --- a/backend/test/backend_tests/rpc_team_test.clj +++ b/backend/test/backend_tests/rpc_team_test.clj @@ -2491,7 +2491,7 @@ ;; Team creation adds the owner through add-profile-to-team!, which used to ;; leave no membership event. (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] - (with-redefs [cf/flags (conj cf/flags :audit-log)] + (th/with-audit-config (conj cf/flags :nexus) (let [owner (th/create-profile* 1 {:is-active true}) team (th/create-team* 21 {:profile-id (:id owner)}) events (->> (:call-args-list @audit-mock) @@ -2506,7 +2506,7 @@ ;; Adding an existing member with on-conflict-do-nothing inserts nothing and ;; must not report a membership that already exists. (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] - (with-redefs [cf/flags (conj cf/flags :audit-log)] + (th/with-audit-config (conj cf/flags :nexus) (let [owner (th/create-profile* 1 {:is-active true}) team (th/create-team* 22 {:profile-id (:id owner)})] (th/reset-mock! audit-mock) @@ -2518,7 +2518,7 @@ {::db/on-conflict-do-nothing? true}))) (t/is (empty? (:call-args-list @audit-mock))))))) -(t/deftest no-add-team-member-event-without-audit-log +(t/deftest no-add-team-member-event-without-a-consumer (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] (let [owner (th/create-profile* 1 {:is-active true}) _team (th/create-team* 23 {:profile-id (:id owner)}) @@ -2531,7 +2531,7 @@ ;; `team_profile_rel` has no role column, so no caller can pass it: the ;; emitter derives it from the flags, inverse of permissions-for-role. (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] - (with-redefs [cf/flags (conj cf/flags :audit-log)] + (th/with-audit-config (conj cf/flags :nexus) (let [owner (th/create-profile* 1 {:is-active true}) team (th/create-team* 26 {:profile-id (:id owner)}) editor (th/create-profile* 27 {:is-active true}) @@ -2560,7 +2560,7 @@ ;; Sibling events built by `prepare-rpc-event` carry the request context; ;; the hand-built membership event must not lose it when rpc params travel. (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] - (with-redefs [cf/flags (conj cf/flags :audit-log)] + (th/with-audit-config (conj cf/flags :nexus) (let [owner (th/create-profile* 1 {:is-active true}) member (th/create-profile* 2 {:is-active true}) team (th/create-team* 30 {:profile-id (:id owner)})] @@ -2585,7 +2585,7 @@ ;; With email verification off, inviting an existing profile adds the ;; member directly; the event belongs to the inviter, not the member. (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] - (with-redefs [cf/flags #{:audit-log}] + (th/with-audit-config #{:nexus} (let [owner (th/create-profile* 1 {:is-active true}) member (th/create-profile* 2 {:is-active true}) team (th/create-team* 31 {:profile-id (:id owner)})] @@ -2640,7 +2640,7 @@ ;; the manual event must carry its id for a projection to key it. (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil} email-mock {:target 'app.email/send-template :return nil}] - (with-redefs [cf/flags (conj cf/flags :audit-log)] + (th/with-audit-config (conj cf/flags :nexus) (let [profile (th/create-profile* 1 {:is-active true}) out (th/command-raw! {::th/type :create-team-with-invitations ::rpc/profile-id (:id profile) diff --git a/backend/test/backend_tests/srepl_main_test.clj b/backend/test/backend_tests/srepl_main_test.clj index 9cbf5a39af..2e313743a6 100644 --- a/backend/test/backend_tests/srepl_main_test.clj +++ b/backend/test/backend_tests/srepl_main_test.clj @@ -7,6 +7,8 @@ (ns backend-tests.srepl-main-test (:require [app.common.time :as ct] + [app.common.uuid :as uuid] + [app.db :as db] [app.jobs :as jobs] [app.srepl.main :as srepl] [app.system :as sys] @@ -113,6 +115,26 @@ (t/is (= 0 (:submitted result))) (t/is (empty? @submitted)))) +(t/deftest cascade-counts-only-the-submits-that-happen + ;; `::jobs/dedupe` returns nil when a not-started job already covers the + ;; row, so a second pass finds it but enqueues nothing. + (let [profile (create-deleted-profile! 998 (ct/in-past {:days 10})) + first-run (#'srepl/submit-deleted-cascade + th/*system* {:table "profile" :object :profile}) + second-run (#'srepl/submit-deleted-cascade + th/*system* {:table "profile" :object :profile}) + dry-run (#'srepl/submit-deleted-cascade + th/*system* {:table "profile" :object :profile :dry-run? true})] + (t/is (= 1 (:found first-run))) + (t/is (= 1 (:submitted first-run))) + (t/testing "the second pass finds the row but dedupe drops the submit" + (t/is (= 1 (:found second-run))) + (t/is (= 0 (:submitted second-run)))) + (t/testing "dry-run still counts without enqueueing" + (t/is (= 1 (:found dry-run))) + (t/is (= 0 (:submitted dry-run)))) + (t/is (= 1 (count (th/db-query :job {:label (str (:id profile))})))))) + (t/deftest public-helpers-read-their-own-table (let [profile (th/create-profile* 997) team (th/create-team* 997 {:profile-id (:id profile)}) @@ -171,6 +193,44 @@ (t/is (nil? (:profile-id event))) (t/is (= (:id profile) (get-in event [:props :id])))))))) +(t/deftest delete-file-carries-its-team + ;; The SREPL event must carry its team, so a consumer can key on it + ;; without joining production tables. + (with-mocks [audit-mock {:target 'app.loggers.audit/insert :return nil} + wrk-mock {:target 'app.jobs/invoke :return nil}] + (let [profile (th/create-profile* 1 {:is-active true}) + team-id (:default-team-id profile) + proj (th/create-project* 1 {:profile-id (:id profile) :team-id team-id}) + file (th/create-file* 1 {:profile-id (:id profile) :project-id (:id proj)})] + (with-redefs [sys/system th/*system*] + (srepl/delete-file! (str (:id file)))) + (let [event (-> @audit-mock :call-args-list first second)] + (t/is (= "delete-file" (:name event))) + (t/is (= team-id (get-in event [:props :team-id]))))))) + +(t/deftest restore-file-carries-its-team + (with-mocks [audit-mock {:target 'app.loggers.audit/insert :return nil}] + (let [profile (th/create-profile* 1 {:is-active true}) + proj (th/create-project* 1 {:profile-id (:id profile) + :team-id (:default-team-id profile)}) + file (th/create-file* 1 {:profile-id (:id profile) + :project-id (:id proj)})] + (with-redefs [sys/system th/*system*] + (srepl/restore-file! (str (:id file)))) + (let [event (-> @audit-mock :call-args-list first second)] + (t/is (= "restore-file" (:name event))) + (t/is (= (:default-team-id profile) (get-in event [:props :team-id]))))))) + +(t/deftest restore-team-carries-its-team + (with-mocks [audit-mock {:target 'app.loggers.audit/insert :return nil}] + (let [profile (th/create-profile* 1 {:is-active true}) + team (th/create-team* 3 {:profile-id (:id profile)})] + (with-redefs [sys/system th/*system*] + (srepl/restore-team! (str (:id team)))) + (let [event (-> @audit-mock :call-args-list first second)] + (t/is (= "restore-team" (:name event))) + (t/is (= (:id team) (get-in event [:props :team-id]))))))) + (t/deftest delete-team-emits-identifiable-audit-event (with-mocks [audit-mock {:target 'app.loggers.audit/insert :return nil} wrk-mock {:target 'app.jobs/invoke :return nil}] @@ -194,7 +254,11 @@ wrk-mock {:target 'app.jobs/invoke :return nil}] (with-redefs [sys/system th/*system*] (let [profile (th/create-profile* 1 {:is-active true})] - (srepl/restore-profile! (str (:id profile))) + ;; The restore path only runs for a soft-deleted profile; a direct + ;; stamp is enough here, the cascade has its own test below. + (th/db-exec! ["UPDATE profile SET deleted_at = ? WHERE id = ?" + (ct/now) (:id profile)]) + (t/is (= :restored (srepl/restore-profile! (str (:id profile))))) (let [events (->> (:call-args-list @audit-mock) (map second))] (t/is (= 1 (count events))) @@ -206,6 +270,61 @@ (t/is (nil? (:profile-id event))) (t/is (= (:id profile) (get-in event [:props :id]))))))))) +(t/deftest restore-profile-is-idempotent-and-keeps-separately-deleted-content + ;; A second restore on an active profile is a no-op. Running the cascade + ;; over live teams would un-delete a font variant deleted on its own. + (with-mocks [audit-mock {:target 'app.loggers.audit/insert :return nil}] + (let [profile (th/create-profile* 1 {:is-active true}) + team (:default-team-id profile) + variant (uuid/next)] + (th/db-insert! :team-font-variant + {:id variant + :team-id team + :profile-id (:id profile) + :font-id (uuid/next) + :font-family "somefont" + :font-weight 400 + :font-style "normal" + :deleted-at (ct/in-past {:days 1})}) + (with-redefs [sys/system th/*system*] + (t/is (= :already-active (srepl/restore-profile! (str (:id profile))))) + (t/is (= :already-active (srepl/restore-profile! (str (:id profile)))))) + (t/is (some? (:deleted-at (th/db-get :team-font-variant + {:id variant} + {::db/remove-deleted false})))) + (t/is (empty? (:call-args-list @audit-mock)))))) + +(t/deftest restore-profile-restores-only-the-teams-the-cascade-took + ;; The profile cascade stamps its live owned teams with the profile's own + ;; instant; restoring the profile brings back exactly those, not a team the + ;; owner had already deleted on its own. + (with-mocks [audit-mock {:target 'app.loggers.audit/insert :return nil}] + (let [profile (th/create-profile* 1 {:is-active true}) + owned (:default-team-id profile) + unrelated (:id (th/create-team* 2 {:profile-id (:id profile)}))] + ;; A team owned by the profile, deleted on its own long before. + (th/db-exec! ["UPDATE team SET deleted_at = ? WHERE id = ?" + (ct/in-past {:days 30}) unrelated]) + ;; The real cascade marks the profile and its live owned teams. + (th/run-task! :delete-object {:object :profile + :id (:id profile) + :deleted-at (ct/now)}) + (t/is (some? (:deleted-at (th/db-get :profile {:id (:id profile)} + {::db/remove-deleted false})))) + (t/is (some? (:deleted-at (th/db-get :team {:id owned} + {::db/remove-deleted false})))) + (with-redefs [sys/system th/*system*] + (srepl/restore-profile! (str (:id profile)))) + (t/is (nil? (:deleted-at (th/db-get :profile {:id (:id profile)} + {::db/remove-deleted false})))) + (t/is (nil? (:deleted-at (th/db-get :team {:id owned} + {::db/remove-deleted false})))) + (t/is (some? (:deleted-at (th/db-get :team {:id unrelated} + {::db/remove-deleted false})))) + (let [event (-> @audit-mock :call-args-list first second)] + (t/is (= "restore-profile" (:name event))) + (t/is (= [owned] (get-in event [:props :team-ids]))))))) + (t/deftest update-profile-email-changes-email (with-mocks [audit-mock {:target 'app.loggers.audit/insert :return nil}] (with-redefs [sys/system th/*system*] diff --git a/backend/test/backend_tests/tasks_telemetry_test.clj b/backend/test/backend_tests/tasks_telemetry_test.clj index b7b9d0c2ed..f58c48d0b0 100644 --- a/backend/test/backend_tests/tasks_telemetry_test.clj +++ b/backend/test/backend_tests/tasks_telemetry_test.clj @@ -11,7 +11,6 @@ [app.config :as cf] [app.db :as db] [app.loggers.audit :as audit] - [app.loggers.audit.consumer-task :as at] [app.tasks.telemetry :as telemetry] [app.util.json :as json] [backend-tests.helpers :as th] @@ -105,7 +104,7 @@ ;; make-legacy-request must not be called at all. (with-mocks [mock {:target 'app.tasks.telemetry/make-legacy-request :return nil}] - (with-redefs [cf/flags #{}] + (th/with-audit-config #{} (th/create-profile* 1 {:is-active true}) (th/run-task! :telemetry {:send? true}) (t/is (not (:called? @mock)))))) @@ -115,7 +114,7 @@ ;; make-legacy-request is called once with only subscriptions + version (no stats). (with-mocks [mock {:target 'app.tasks.telemetry/make-legacy-request :return nil}] - (with-redefs [cf/flags #{}] + (th/with-audit-config #{} (let [prof (th/create-profile* 1 {:is-active true :props {:newsletter-updates true}})] (th/run-task! :telemetry {:send? true}) @@ -131,7 +130,7 @@ ;; Passing send?=false must suppress all HTTP calls even when enabled. (with-mocks [mock {:target 'app.tasks.telemetry/make-legacy-request :return nil}] - (with-redefs [cf/flags #{:telemetry}] + (th/with-audit-config #{:telemetry} (th/create-profile* 1 {:is-active true}) (th/run-task! :telemetry {:send? false :enabled? true}) (t/is (not (:called? @mock)))))) @@ -157,8 +156,7 @@ "https://penpot.dev"])] (with-mocks [mock {:target 'app.tasks.telemetry/make-legacy-request :return nil}] - (with-redefs [cf/flags #{} - cf/config (assoc cf/config :public-uri public-uri)] + (th/with-audit-config #{} [cf/config (assoc cf/config :public-uri public-uri)] (th/create-profile* (+ 10 idx) {:is-active true :props {:newsletter-updates true}}) (th/run-task! :telemetry {:send? true}) @@ -169,8 +167,7 @@ ;; since nothing is going to be sent. (with-mocks [mock {:target 'app.tasks.telemetry/get-subscriptions :return []}] - (with-redefs [cf/flags #{} - cf/config (assoc cf/config :public-uri "https://design.penpot.app")] + (th/with-audit-config #{} [cf/config (assoc cf/config :public-uri "https://design.penpot.app")] (th/create-profile* 1 {:is-active true :props {:newsletter-updates true}}) (th/run-task! :telemetry {:send? true}) @@ -181,8 +178,7 @@ ;; only the implicit newsletter fallback is excluded. (with-mocks [mock {:target 'app.tasks.telemetry/make-legacy-request :return nil}] - (with-redefs [cf/flags #{:telemetry} - cf/config (assoc cf/config :public-uri "https://design.penpot.app")] + (th/with-audit-config #{:telemetry} [cf/config (assoc cf/config :public-uri "https://design.penpot.app")] (th/create-profile* 1 {:is-active true :props {:newsletter-updates true}}) (th/run-task! :telemetry {:send? true :enabled? true}) @@ -199,7 +195,7 @@ :return nil} batch-mock {:target 'app.tasks.telemetry/send-event-batch :return true}] - (with-redefs [cf/flags #{:telemetry}] + (th/with-audit-config #{:telemetry} (th/run-task! :telemetry {:send? true :enabled? true}) (t/is (:called? @legacy-mock)) (t/is (not (:called? @batch-mock)))))) @@ -211,7 +207,7 @@ :return nil} batch-mock {:target 'app.tasks.telemetry/send-event-batch :return true}] - (with-redefs [cf/flags #{:telemetry}] + (th/with-audit-config #{:telemetry} (insert-telemetry-row! "navigate") (insert-telemetry-row! "create-file") (insert-telemetry-row! "update-file") @@ -233,7 +229,7 @@ :return nil} batch-mock {:target 'app.tasks.telemetry/send-event-batch :return false}] - (with-redefs [cf/flags #{:telemetry}] + (th/with-audit-config #{:telemetry} (insert-telemetry-row! "navigate") (insert-telemetry-row! "create-file") @@ -243,14 +239,14 @@ ;; rows still present — not deleted on failure (t/is (= 2 (count-telemetry-rows)))))) -(t/deftest test-audit-events-not-collected-when-audit-log-flag-set - ;; When the :audit-log flag is active, mode C is disabled and the - ;; batch path must never run (audit-log owns those rows instead). +(t/deftest test-audit-events-not-collected-when-consumer-persists + ;; When a consumer persists rows, mode C is disabled and the + ;; batch path must never run (the buffer owns those rows instead). (with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request :return nil} batch-mock {:target 'app.tasks.telemetry/send-event-batch :return true}] - (with-redefs [cf/flags #{:telemetry :audit-log}] + (th/with-audit-config #{:telemetry :nexus} (insert-telemetry-row! "navigate") (th/run-task! :telemetry {:send? true :enabled? true}) @@ -265,11 +261,10 @@ (let [captured (atom nil)] (with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request :return nil}] - (with-redefs [cf/flags #{:telemetry} - telemetry/send-event-batch - (fn [_cfg batch] - (reset! captured batch) - true)] + (th/with-audit-config #{:telemetry} [telemetry/send-event-batch + (fn [_cfg batch] + (reset! captured batch) + true)] (insert-telemetry-row! "navigate") (insert-telemetry-row! "create-file") @@ -312,12 +307,11 @@ (let [call-count (atom 0)] (with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request :return nil}] - (with-redefs [cf/flags #{:telemetry} - telemetry/batch-size 1 - telemetry/send-event-batch - (fn [_cfg _batch] - (swap! call-count inc) - true)] + (th/with-audit-config #{:telemetry} [telemetry/batch-size 1 + telemetry/send-event-batch + (fn [_cfg _batch] + (swap! call-count inc) + true)] (insert-telemetry-row! "navigate") (insert-telemetry-row! "create-file") (insert-telemetry-row! "update-file") @@ -336,13 +330,12 @@ (let [call-count (atom 0)] (with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request :return nil}] - (with-redefs [cf/flags #{:telemetry} - telemetry/batch-size 1 - telemetry/send-event-batch - (fn [_cfg _batch] - (swap! call-count inc) - ;; fail on the second call - (not= 2 @call-count))] + (th/with-audit-config #{:telemetry} [telemetry/batch-size 1 + telemetry/send-event-batch + (fn [_cfg _batch] + (swap! call-count inc) + ;; fail on the second call + (not= 2 @call-count))] (insert-telemetry-row! "navigate") (insert-telemetry-row! "create-file") (insert-telemetry-row! "update-file") @@ -366,7 +359,7 @@ :return nil} batch-mock {:target 'app.tasks.telemetry/send-event-batch :return true}] - (with-redefs [cf/flags #{:telemetry}] + (th/with-audit-config #{:telemetry} (let [now (ct/now) eight-days (ct/minus now (ct/duration {:days 8}))] ;; Stale events (older than 7 days) @@ -389,11 +382,10 @@ (let [batch-events (atom nil)] (with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request :return nil}] - (with-redefs [cf/flags #{:telemetry} - telemetry/send-event-batch - (fn [_cfg batch] - (reset! batch-events batch) - true)] + (th/with-audit-config #{:telemetry} [telemetry/send-event-batch + (fn [_cfg batch] + (reset! batch-events batch) + true)] (let [six-days-ago (ct/minus (ct/now) (ct/duration {:days 6}))] (insert-telemetry-row! "recent-1" {:created-at six-days-ago :tracked-at six-days-ago}) (insert-telemetry-row! "recent-2" {:created-at six-days-ago :tracked-at six-days-ago})) @@ -410,11 +402,10 @@ (let [batch-events (atom nil)] (with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request :return nil}] - (with-redefs [cf/flags #{:telemetry} - telemetry/send-event-batch - (fn [_cfg batch] - (reset! batch-events batch) - true)] + (th/with-audit-config #{:telemetry} [telemetry/send-event-batch + (fn [_cfg batch] + (reset! batch-events batch) + true)] (let [eight-days (ct/minus (ct/now) (ct/duration {:days 8})) one-day (ct/minus (ct/now) (ct/duration {:days 1}))] (insert-telemetry-row! "stale" {:created-at eight-days :tracked-at eight-days}) @@ -438,7 +429,7 @@ ;; empty props, zeroed ip, profile-id=zero, source='telemetry'. ;; Safe context fields (browser, os, version, etc.) are preserved ;; but session-linking and access-token fields are stripped. - (with-redefs [cf/flags #{:telemetry}] + (th/with-audit-config #{:telemetry} (let [_prof (th/create-profile* 1 {:is-active true}) safe-ctx {:browser "Chrome" :browser-version "120.0" @@ -477,11 +468,10 @@ :external-session-id "also-stripped"}] (with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request :return nil}] - (with-redefs [cf/flags #{:telemetry} - telemetry/send-event-batch - (fn [_cfg batch] - (reset! captured-batch batch) - true)] + (th/with-audit-config #{:telemetry} [telemetry/send-event-batch + (fn [_cfg batch] + (reset! captured-batch batch) + true)] ;; Insert with safe context already pre-filtered (as the ingest path does) (th/db-insert! :audit-log {:id (uuid/next) @@ -519,7 +509,7 @@ (t/deftest test-telemetry-rows-have-day-precision-timestamps ;; Telemetry events must be stored with timestamps truncated to day ;; precision so that exact event timing cannot be inferred. - (with-redefs [cf/flags #{:telemetry}] + (th/with-audit-config #{:telemetry} (let [process-event (ns-resolve 'app.loggers.audit 'process-event) profile (th/create-profile* 1 {:is-active true}) tnow (ct/now) @@ -546,7 +536,7 @@ ;; Verify the full row shape stored by process-event in telemetry mode: ;; source=telemetry:backend, empty props, zeroed ip, context filtered to safe ;; backend keys only, profile-id preserved, timestamps truncated. - (with-redefs [cf/flags #{:telemetry}] + (th/with-audit-config #{:telemetry} (let [process-event (ns-resolve 'app.loggers.audit 'process-event) profile (th/create-profile* 1 {:is-active true}) tnow (ct/now) @@ -913,7 +903,7 @@ :return nil} http-mock {:target 'app.http.client/req :return {:status 200}}] - (with-redefs [cf/flags #{:telemetry}] + (th/with-audit-config #{:telemetry} (insert-telemetry-row! "navigate") (insert-telemetry-row! "create-file") @@ -954,7 +944,7 @@ :return nil} batch-mock {:target 'app.tasks.telemetry/send-event-batch :return true}] - (with-redefs [cf/flags #{:telemetry}] + (th/with-audit-config #{:telemetry} ;; No profiles with newsletter-updates, no telemetry rows (th/run-task! :telemetry {:send? true :enabled? true}) @@ -977,7 +967,7 @@ :return nil} batch-mock {:target 'app.tasks.telemetry/send-event-batch :return false}] - (with-redefs [cf/flags #{:telemetry}] + (th/with-audit-config #{:telemetry} (insert-telemetry-row! "navigate") (th/run-task! :telemetry {:send? true :enabled? true}) @@ -1003,7 +993,7 @@ :return nil} batch-mock {:target 'app.tasks.telemetry/send-event-batch :return false}] - (with-redefs [cf/flags #{:telemetry}] + (th/with-audit-config #{:telemetry} (let [eight-days (ct/minus (ct/now) (ct/duration {:days 8})) one-day (ct/minus (ct/now) (ct/duration {:days 1}))] ;; Stale events (should be GC'd) @@ -1054,11 +1044,10 @@ (let [send-count (atom 0)] (with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request :return nil}] - (with-redefs [cf/flags #{:telemetry} - telemetry/send-event-batch - (fn [_cfg _batch] - (swap! send-count inc) - true)] + (th/with-audit-config #{:telemetry} [telemetry/send-event-batch + (fn [_cfg _batch] + (swap! send-count inc) + true)] (insert-telemetry-row! "navigate") (insert-telemetry-row! "create-file") @@ -1073,45 +1062,3 @@ (th/run-task! :telemetry {:send? true :enabled? true}) (t/is (= 1 @send-count)) ;; still 1, not 2 (t/is (= 0 (count-telemetry-rows))))))) - -;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; -;; PENDING-QUEUE REGRESSION (telemetry deletes) -;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; - -(defn- insert-pending-pair! - "Queue the row for both consumers (from the consumer constants, not - literals): the delete paths must face pending-ful rows." - [row-id] - (doseq [consumer [at/consumer-nexus at/consumer-admin-console]] - (th/db-exec! ["INSERT INTO audit_log_pending (audit_log_id, consumer) - VALUES (?::uuid, ?::smallint) - ON CONFLICT DO NOTHING" - row-id consumer]))) - -(defn- count-pendings [] - (-> (th/db-exec-one! ["SELECT count(*) AS cnt FROM audit_log_pending"]) - :cnt - long)) - -(t/deftest test-delete-sent-events-removes-pendings-first - ;; Regression: telemetry rows carry pendings for both consumers, so - ;; deleting the row must delete its pendings first, or the FK - ;; rejects the delete and the batch is sent again next tick. - (let [row (insert-telemetry-row! "navigate")] - (insert-pending-pair! (:id row)) - (t/is (= 2 (count-pendings))) - (db/tx-run! th/*pool* - (fn [{:keys [::db/conn]}] - (@#'telemetry/delete-sent-events conn [(:id row)]))) - (t/is (= 0 (count-telemetry-rows))) - (t/is (= 0 (count-pendings))))) - -(t/deftest test-gc-events-removes-pendings-first - ;; Same guarantee for the >7-days purge. - (let [past (ct/in-past {:days 8}) - row (insert-telemetry-row! "navigate" {:created-at past :tracked-at past})] - (insert-pending-pair! (:id row)) - (t/is (= 2 (count-pendings))) - (@#'telemetry/gc-events {::db/conn th/*pool*}) - (t/is (= 0 (count-telemetry-rows))) - (t/is (= 0 (count-pendings))))) diff --git a/backend/test/backend_tests/worker_dispatcher_test.clj b/backend/test/backend_tests/worker_dispatcher_test.clj index c498214beb..c58a8d09cb 100644 --- a/backend/test/backend_tests/worker_dispatcher_test.clj +++ b/backend/test/backend_tests/worker_dispatcher_test.clj @@ -47,12 +47,12 @@ (defn- make-metrics [] (ig/init-key :app.metrics/metrics {:default (select-keys main/default-metrics - [:jobs-dispatcher-timing])})) + [:jobs-dispatcher-batch-timing])})) (defn- make-orphan-metrics [] (ig/init-key :app.metrics/metrics {:default (select-keys main/default-metrics - [:jobs-orphaned :jobs-total-timing])})) + [:jobs-total-timing])})) (defn- counter-value [metrics id labels] (let [collector (mtx/get-collector metrics id) @@ -74,7 +74,7 @@ (reduce + 0))) (defn- mk-job - [{:keys [name queue status scheduled-at modified-at tenant] + [{:keys [name queue status scheduled-at modified-at tenant dedupe label] :or {name "test-job" queue "test" tenant (cf/get :tenant) @@ -86,6 +86,8 @@ :name name :tenant tenant :queue queue + :label label + :dedupe (boolean dedupe) :params (db/json {}) :priority 100 :max-retries 3 @@ -123,6 +125,16 @@ (t/use-fixtures :each test-fixture) +(defn- fill-queue + "Put one dummy payload in a queue, as a worker that has not drained it." + [queue-name] + (let [conn (rds/connect (mk-cfg))] + (try + (rds/rpush conn (queue-key queue-name) + [(json/encode [(str (uuid/next)) (ct/format-inst (ct/now))])]) + (finally + (rds/close conn))))) + (defn- drain-queue [queue-name] (let [conn (rds/connect (mk-cfg))] @@ -176,24 +188,200 @@ (t/is (= "new" (:status (get-row future)))) (t/is (= "running" (:status (get-row running)))))) -(t/deftest dispatcher-reschedules-lost-scheduled-jobs - (let [cfg (mk-cfg) - lost-id (mk-job {:status "scheduled" - :scheduled-at (ct/minus (ct/now) - (ct/duration {:minutes 6}))}) - fresh-id (mk-job {:status "scheduled"})] +(t/deftest dispatcher-aborts-a-lost-job-whose-queue-is-empty + ;; `scheduled` means the job was handed to a queue. When the queue has + ;; nothing waiting and the row has been there longer than the grace, the + ;; message is gone and the job is terminal. Known limit: the grace is + ;; measured from the push, so a message that waited longer in a backlog + ;; and is popped while the sweep runs (the queue empty at that instant) + ;; can lose the race with its claim; the window is the worker's + ;; pop-to-claim round trip. + (let [cfg (mk-cfg) + lost (mk-job {:status "scheduled" + :modified-at (ct/minus (ct/now) + (ct/duration {:minutes 6}))})] (wdisp/run-batch cfg) - (let [lost (get-row lost-id) - fresh (get-row fresh-id)] - ;; the lost job is rescheduled to 'new' and claimed again in the - ;; same batch, so it ends as 'scheduled' with a recent scheduled_at - (t/is (= "scheduled" (:status lost))) - (t/is (> (inst-ms (:scheduled-at lost)) - (inst-ms (ct/minus (ct/now) - (ct/duration {:minutes 5}))))) - (t/testing "the rescheduled job was pushed again to the queue" - (t/is (= 1 (count (drain-queue "test")))) - (t/is (= "scheduled" (:status fresh))))))) + (let [row (get-row lost)] + (t/is (= "aborted" (:status row))) + (t/is (= jobs/lost-error (:error row))) + (t/is (= [{:kind "end" :payload {:outcome "aborted"}}] (:events row))) + (t/testing "an aborted job is terminal: it is never retried" + (t/is (zero? (jobs/retry-job cfg (:id row) 1 (ct/now) jobs/lost-error :backoff))))))) + +(t/deftest dispatcher-leaves-a-fresh-scheduled-job-alone + (let [cfg (mk-cfg) + fresh (mk-job {:status "scheduled" + :modified-at (ct/minus (ct/now) + (ct/duration {:minutes 4}))})] + (wdisp/run-batch cfg) + (let [row (get-row fresh)] + (t/is (= "scheduled" (:status row))) + (t/is (nil? (:error row))) + (t/is (= [] (:events row)))))) + +(t/deftest dispatcher-leaves-a-scheduled-job-that-is-still-in-its-queue + ;; The queue has a message waiting: the job may be behind a backlog, so + ;; it is not lost. Aborting on time alone would kill legitimate work. + (let [cfg (mk-cfg) + lost (mk-job {:status "scheduled" + :modified-at (ct/minus (ct/now) + (ct/duration {:minutes 6}))})] + (fill-queue "test") + (wdisp/run-batch cfg) + (let [row (get-row lost)] + (t/is (= "scheduled" (:status row))) + (t/is (nil? (:error row))) + (t/is (= [] (:events row)))))) + +(t/deftest dispatcher-aborts-a-scheduled-job-past-the-cap + ;; A message lost behind a backlog (or a queue with no consumer) never + ;; drains: past the cap the job is aborted anyway. + (let [cfg (mk-cfg) + lost (mk-job {:status "scheduled" + :modified-at (ct/minus (ct/now) + (ct/duration {:hours 25}))})] + (fill-queue "test") + (wdisp/run-batch cfg) + (t/is (= "aborted" (:status (get-row lost)))))) + +(t/deftest lost-sweep-pages-the-cap-branch + ;; A row past the cap is lost whatever its queue holds, and the read is + ;; bounded: with a batch of one, each pass aborts the oldest candidate + ;; and the next pass continues. Only the grace rows pay the LLEN. + (let [cfg (mk-cfg) + q "test-lost-cap" + a (mk-job {:status "scheduled" + :queue q + :modified-at (ct/minus (ct/now) (ct/duration {:hours 30}))}) + b (mk-job {:status "scheduled" + :queue q + :modified-at (ct/minus (ct/now) (ct/duration {:hours 29}))})] + (fill-queue q) + (with-redefs-fn {#'wdisp/lost-batch-size 1} + #(wdisp/run-batch cfg)) + (t/is (= 1 (count (filter #(= "aborted" (:status (get-row %))) [a b])))) + (with-redefs-fn {#'wdisp/lost-batch-size 1} + #(wdisp/run-batch cfg)) + (t/is (every? #(= "aborted" (:status (get-row %))) [a b])))) + +(t/deftest lost-sweep-pages-the-grace-window + ;; Inside the grace window the read is bounded and the queue decides: an + ;; empty queue makes the row lost, and with a batch of one the next pass + ;; continues where the previous one stopped. + (let [cfg (mk-cfg) + q "test-lost-grace" + a (mk-job {:status "scheduled" + :queue q + :modified-at (ct/minus (ct/now) (ct/duration {:minutes 7}))}) + b (mk-job {:status "scheduled" + :queue q + :modified-at (ct/minus (ct/now) (ct/duration {:minutes 6}))})] + (with-redefs-fn {#'wdisp/lost-batch-size 1} + #(wdisp/run-batch cfg)) + (t/is (= 1 (count (filter #(= "aborted" (:status (get-row %))) [a b])))) + (with-redefs-fn {#'wdisp/lost-batch-size 1} + #(wdisp/run-batch cfg)) + (t/is (every? #(= "aborted" (:status (get-row %))) [a b])))) + +(t/deftest dispatcher-does-not-abort-a-job-whose-heartbeat-renews-it + ;; The heartbeat accepts a `scheduled` row (a previous attempt may still + ;; beat): it refreshes `modified_at`, so the outer WHERE must re-check it. + (let [cfg (mk-cfg) + lost (mk-job {:status "scheduled" + :modified-at (ct/minus (ct/now) + (ct/duration {:minutes 6}))})] + (with-redefs [rds/llen (fn [_ _] + (th/db-update! :job {:modified-at (ct/now)} {:id lost}) + 0)] + (wdisp/run-batch cfg)) + (let [row (get-row lost)] + (t/is (= "scheduled" (:status row))) + (t/is (nil? (:error row))) + (t/is (= [] (:events row)))))) + +(t/deftest dispatcher-llens-only-its-tenant-queue + ;; The queue key carries the tenant: a message waiting for another + ;; environment cannot protect a lost row of this one. + (let [cfg (mk-cfg) + foreign (wrk/queue-key "other-tenant" "test") + conn (rds/connect (mk-cfg)) + lost (mk-job {:status "scheduled" + :modified-at (ct/minus (ct/now) + (ct/duration {:minutes 6}))})] + (try + (rds/rpush conn foreign + [(json/encode [(str (uuid/next)) + (ct/format-inst (ct/now))])]) + (wdisp/run-batch cfg) + (t/is (= "aborted" (:status (get-row lost)))) + (finally + (rds/del conn foreign) + (rds/close conn))))) + +(t/deftest dispatcher-aborts-two-lost-dedupe-jobs-with-the-same-key + ;; A Redis restart loses every message at once: two deduped jobs with the + ;; same key can be `scheduled` together. Neither returns to `new` (that + ;; would violate `job__new_dedupe__uniq`); both are aborted and the batch + ;; keeps going. + (let [cfg (mk-cfg) + a (mk-job {:status "scheduled" + :dedupe true + :label "k" + :modified-at (ct/minus (ct/now) + (ct/duration {:minutes 6}))}) + b (mk-job {:status "scheduled" + :dedupe true + :label "k" + :modified-at (ct/minus (ct/now) + (ct/duration {:minutes 7}))})] + (wdisp/run-batch cfg) + (t/is (= "aborted" (:status (get-row a)))) + (t/is (= "aborted" (:status (get-row b)))) + (t/is (empty? (th/db-query :job {:status "new"}))) + (t/is (empty? (drain-queue "test"))))) + +(t/deftest dispatcher-abort-lost-records-the-terminal-counter + (let [metrics (make-orphan-metrics) + cfg (assoc (mk-cfg) ::mtx/metrics metrics)] + (mk-job {:status "scheduled" + :modified-at (ct/minus (ct/now) + (ct/duration {:minutes 6}))}) + (wdisp/run-batch cfg) + (t/testing "a job lost in transit counts as an aborted terminal outcome" + (t/is (= 1 (histogram-sample-count metrics ["test-job" "aborted"])))))) + +(t/deftest dispatcher-does-not-abort-a-job-claimed-before-the-sweep + ;; The candidate is read first and the UPDATE runs later: a runner can + ;; claim the row in between. The outer WHERE must see the new status and + ;; affect no row, or the job would be aborted while running. + (let [cfg (mk-cfg) + lost (mk-job {:status "scheduled" + :modified-at (ct/minus (ct/now) + (ct/duration {:minutes 6}))})] + (with-redefs [rds/llen (fn [_ _] + (th/db-update! :job {:status "running" + :modified-at (ct/now)} + {:id lost}) + 0)] + (wdisp/run-batch cfg)) + (let [row (get-row lost)] + (t/is (= "running" (:status row))) + (t/is (nil? (:error row))) + (t/is (= [] (:events row)))))) + +(t/deftest dispatcher-aborts-a-lost-job-once + ;; Two dispatchers of the same tenant read the same candidate: only the + ;; first transition wins, so there is one end event and one sample. + (let [metrics (make-orphan-metrics) + cfg (assoc (mk-cfg) ::mtx/metrics metrics) + lost (mk-job {:status "scheduled" + :modified-at (ct/minus (ct/now) + (ct/duration {:minutes 6}))})] + (run! deref (doall (repeatedly 2 #(future (wdisp/run-batch cfg))))) + (t/is (= "aborted" (:status (get-row lost)))) + (t/is (= 1 (count (filter #(= "end" (:kind %)) + (:events (get-row lost)))))) + (t/is (= 1 (histogram-sample-count metrics ["test-job" "aborted"]))))) (t/deftest dispatcher-marks-stale-running-jobs-as-aborted-by-lease (let [cfg (mk-cfg) @@ -218,7 +406,7 @@ (t/testing "recent running job is not touched" (t/is (= "running" (:status (get-row fresh))))))) -(t/deftest dispatcher-orphan-sweep-records-orphan-and-terminal-metrics +(t/deftest dispatcher-orphan-sweep-records-the-terminal-counter (let [metrics (make-orphan-metrics) cfg (assoc (mk-cfg) ::mtx/metrics metrics) _ (mk-job {:status "running" @@ -226,8 +414,8 @@ (ct/plus (cf/get-jobs-lease) (ct/duration {:minutes 1})))})] (wdisp/run-batch cfg) - (t/is (= 1.0 (counter-value metrics :jobs-orphaned ["test"]))) - (t/is (= 1 (histogram-sample-count metrics ["test-job" "test" "aborted"]))))) + (t/testing "the _count of the aborted outcome is the orphan count" + (t/is (= 1 (histogram-sample-count metrics ["test-job" "aborted"])))))) (t/deftest dispatcher-batch-without-pending-jobs-signals-wait (let [cfg (mk-cfg)] @@ -297,19 +485,18 @@ (t/testing "and nothing was pushed to this tenant's Redis list" (t/is (empty? (drain-queue "test")))))) -(t/deftest dispatcher-does-not-reschedule-another-tenants-lost-jobs +(t/deftest dispatcher-does-not-abort-another-tenants-lost-jobs (let [cfg (mk-cfg) foreign (mk-job {:status "scheduled" :tenant "other-tenant" - :scheduled-at (ct/minus (ct/now) - (ct/duration {:minutes 6}))})] + :modified-at (ct/minus (ct/now) + (ct/duration {:minutes 61}))})] (wdisp/run-batch cfg) (t/testing "the other tenant's scheduled job keeps its status and its date" (let [row (get-row foreign)] (t/is (= "scheduled" (:status row))) - (t/is (< (inst-ms (:scheduled-at row)) - (inst-ms (ct/minus (ct/now) - (ct/duration {:minutes 5}))))))))) + (t/is (nil? (:error row))) + (t/is (= [] (:events row))))))) (t/deftest dispatcher-does-not-orphan-another-tenants-running-jobs (let [cfg (mk-cfg) diff --git a/backend/test/backend_tests/worker_runner_test.clj b/backend/test/backend_tests/worker_runner_test.clj index 2df99415a3..e3f540f5fd 100644 --- a/backend/test/backend_tests/worker_runner_test.clj +++ b/backend/test/backend_tests/worker_runner_test.clj @@ -82,7 +82,7 @@ (defn- make-metrics [] (ig/init-key :app.metrics/metrics {:default (select-keys main/default-metrics - [:jobs-queue-wait-timing + [:jobs-claim-wait-timing :jobs-execution-timing :jobs-total-timing :tasks-timing])})) @@ -241,7 +241,7 @@ (t/is (not (contains? context :queue)) "the tenant-prefixed queue is a routing detail, not context")))))) -(t/deftest runner-records-queue-wait-execution-and-total +(t/deftest runner-records-claim-wait-execution-and-total (let [metrics (make-metrics) cfg (assoc (mk-cfg {}) ::mtx/metrics metrics) at (ct/truncate (ct/now) :millisecond) @@ -249,14 +249,14 @@ (push-payload job-id at) (run-one cfg) (t/is (= 1 (histogram-sample-count metrics - :jobs-queue-wait-timing - ["echo-runner" "test"]))) + :jobs-claim-wait-timing + ["echo-runner"]))) (t/is (= 1 (histogram-sample-count metrics :jobs-execution-timing - ["echo-runner" "test"]))) + ["echo-runner"]))) (t/is (= 1 (histogram-sample-count metrics :jobs-total-timing - ["echo-runner" "test" "completed"]))) + ["echo-runner" "completed"]))) (t/testing "the legacy histogram is still emitted, with its raw name label" (t/is (= 1 (histogram-sample-count metrics :tasks-timing @@ -408,11 +408,12 @@ (t/is (= 1 (#'wrkr/claim-job (mk-cfg {}) job-id at))) (t/is (= "running" (:status (get-row job-id))))))) -(t/deftest runner-skips-stale-payload-after-reschedule +(t/deftest runner-skips-a-payload-with-a-stale-timestamp (let [stale-at (ct/truncate (ct/now) :millisecond) job-id (mk-job {:scheduled-at stale-at}) fresh-at (ct/plus stale-at (ct/duration {:minutes 6}))] - ;; dispatcher reschedules: new scheduled_at + fresh payload + ;; A duplicate payload (or one whose row was edited by hand) carries a + ;; timestamp that no longer matches the row. (th/db-update! :job {:status "scheduled" :scheduled-at fresh-at} {:id job-id}) (t/testing "stale payload is skipped, handler never invoked" diff --git a/backend/test/backend_tests/worker_test.clj b/backend/test/backend_tests/worker_test.clj index 02f525820d..7455f5739e 100644 --- a/backend/test/backend_tests/worker_test.clj +++ b/backend/test/backend_tests/worker_test.clj @@ -17,7 +17,9 @@ [app.worker.cron :as wcron] [app.worker.dispatcher :as wdisp] [app.worker.runner :as wrkr] - [clojure.test :as t])) + [backend-tests.helpers :as th] + [clojure.test :as t] + [integrant.core :as ig])) ;; The dispatcher's push and the runner's pop agree on the Redis list ;; only through this helper, so the test helpers use it too. That @@ -50,6 +52,30 @@ @#'wrkr/schema:params]] (t/is (sm/schema? (sm/schema s)))))) +(t/deftest every-worker-consumer-component-is-known-to-the-test-system + ;; The test system drops the consumer loops by key, so a new one must be + ;; added there too or it drains audit_log in the background. Deriving the + ;; list from the config (every worker key named `/consumer`) makes a + ;; third consumer fail here, not as a flaky test elsewhere. + (let [keys (->> (keys main/worker-config) + (filter #(and (keyword? %) (= "consumer" (name %)))) + (into #{}))] + (t/is (= #{:app.loggers.audit.consumers.nexus/consumer + :app.loggers.audit.consumers.admin-console/consumer} + keys) + (str "update this set and helpers/init-system's dissoc: " keys)))) + +(t/deftest audit-log-gc-cron-is-gated-by-its-own-flag + (t/testing "the GC entry is present only with :audit-log-gc" + (with-redefs [cf/flags #{:audit-log-gc}] + (t/is (some #(= :audit-log-gc (:job %)) (main/worker-cron-entries)))) + (th/with-audit-config #{} + (t/is (not (some #(= :audit-log-gc (:job %)) + (main/worker-cron-entries))))) + (with-redefs [cf/flags #{}] + (t/is (not (some #(= :audit-log-gc (:job %)) + (main/worker-cron-entries))))))) + (t/deftest the-binfile-runner-is-wired (let [cfg (get main/worker-config [:app.main/binfile ::wrk/runner])] @@ -66,3 +92,24 @@ (t/is (contains? cfg ::jobs/defs)) (t/is (contains? cfg ::mtx/metrics)) (t/is (contains? cfg ::db/pool))))) + +(t/deftest the-audit-consumers-are-wired + ;; Each consumer is a loop in a thread: the test system drops the two + ;; components by name, so a rename that leaves `helpers.clj` behind would + ;; start real loops in the background. The components live in the worker + ;; config only (a web instance does not drain the audit log) and their + ;; `ig/init-key`/`ig/halt-key!` methods must be registered. + (doseq [[k deps] [[:app.loggers.audit.consumers.nexus/consumer + [::db/pool :app.http.client/client + :app.setup/shared-keys ::mtx/metrics]] + [:app.loggers.audit.consumers.admin-console/consumer + [::db/pool :app.nitrate/client ::mtx/metrics]]]] + (t/testing (str k) + (let [cfg (get main/worker-config k)] + (t/is (some? cfg)) + (t/is (nil? (get main/system-config k))) + (t/is (some? (get-method ig/init-key k))) + (t/is (some? (get-method ig/halt-key! k))) + (t/testing "the loop resolves its dependencies" + (doseq [dep deps] + (t/is (contains? cfg dep) (str k " misses " dep)))))))) diff --git a/common/scripts/repl b/common/scripts/repl index e736b2ad99..7707e659cb 100755 --- a/common/scripts/repl +++ b/common/scripts/repl @@ -1,6 +1,6 @@ #!/usr/bin/env bash -export PENPOT_FLAGS="enable-asserts enable-audit-log $PENPOT_FLAGS" +export PENPOT_FLAGS="enable-asserts enable-nexus $PENPOT_FLAGS" export JAVA_OPTS="\ -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager \ diff --git a/common/src/app/common/flags.cljc b/common/src/app/common/flags.cljc index f97a0cadd8..e76dc92f84 100644 --- a/common/src/app/common/flags.cljc +++ b/common/src/app/common/flags.cljc @@ -59,7 +59,9 @@ (def varia "Rest of the flags" - #{:audit-log + #{;; Runs the audit-log buffer GC. Keep it off while the old and the new + ;; version coexist; the runbook turns it on after the rollout. + :audit-log-gc ;; Enables shipping audit_log chunks to Nexus. :nexus :audit-log-logger diff --git a/docs/technical-guide/developer/devenv.md b/docs/technical-guide/developer/devenv.md index 20cd2d8a3c..9cd8ed6fa3 100644 --- a/docs/technical-guide/developer/devenv.md +++ b/docs/technical-guide/developer/devenv.md @@ -219,13 +219,14 @@ keeps msgbus Pub/Sub channels (collab broadcasts, team-org notifications, file-summary cache, rate-limit counters) isolated. Background workers (`enable-backend-worker`) run only on ws0 — ws1+ overlays -disable it. ws1+ RPC handlers still enqueue tasks into the shared Postgres -`task` table; ws0's dispatcher claims them via `FOR UPDATE SKIP LOCKED` and +disable it. ws1+ RPC handlers still enqueue jobs into the shared Postgres +`job` table; ws0's dispatcher claims them via `FOR UPDATE SKIP LOCKED` and runs them against the shared DB and RustFS. Workers are fire-and-forget: -`wrk/submit!` inserts a row and returns; RPC handlers never wait on +`jobs/submit` inserts a row and returns; RPC handlers never wait on completion. The "ws0 only" policy avoids multi-instance worker races (cron -dedup is best-effort across instances, `wrk/submit!` `dedupe` is racy across -submitters). +dedup is best-effort across instances; the dedupe and exclusivity of +`jobs/submit` are resolved by unique indexes, so the submits themselves do +not race). Each workspace is independent and can be started and stopped in any order. Shared infrastructure shuts down only when no instances remain running. diff --git a/operations/2.20.0-consumer-generalization.md b/operations/2.20.0-consumer-generalization.md new file mode 100644 index 0000000000..ad7613b04a --- /dev/null +++ b/operations/2.20.0-consumer-generalization.md @@ -0,0 +1,471 @@ +# Consumer generalization: consumer handover and out-of-band migration + +Procedure to run **before upgrading** to the release that ships the audit-log +cursor consumers and the jobs substrate. Steps 1-2 (the consumer handover) +apply to every instance that ran the legacy archiver or its GC; the +out-of-band DDL is for a large instance. + +- Migration: `backend/src/app/migrations/sql/0159-consumer-generalization.sql` +- Issue: + +## When you need this + +Run the consumer handover **before upgrading** to the release that ships the +audit-log cursor consumers and the jobs substrate, while the current +version is still serving traffic. The web instances keep writing for the +whole procedure (step 4 relies on it); only the legacy archiver and its +GC stop (step 1). + +**Steps 1 and 2 are mandatory whenever the legacy archiver or its GC was +enabled in 2.19**, even when `audit_log` is small: the new consumers start at +`(0, 0)`, so the rows the old archiver already archived would be replayed and +the old GC could delete rows the new consumers have not read. Only the +out-of-band DDL is optional by size. + +You need the out-of-band DDL when `audit_log` is large. The migration runner +executes every pending step in a single transaction: the plain index builds +hold a `SHARE` lock for the whole build and every `ALTER` keeps its `ACCESS +EXCLUSIVE` until that one commit, so on a large table the upgrade stalls +writes for minutes. Pre-applying the same statements out of band and in this +order (outside a transaction block, except the jobs step, which runs as one +explicit `BEGIN`/`COMMIT`), plus registering the steps first (step 0), makes +the runner skip the group, so the deploy takes no such locks. On a small table +the migration runs fast enough on its own: skip the DDL steps, not the +handover. This procedure covers `0159-consumer-generalization` and the +`0154-add-job-table` step it depends on. + +Pre-applying does not make the migration free by itself: the guards +(`IF NOT EXISTS`, `seq IS NULL`) skip the work, not the locks. Registering the +step is what makes the runner skip the group. + +**The application keeps writing to `audit_log` for the whole procedure.** Do +not stop the web instances. The DDL takes a brief `ACCESS EXCLUSIVE` +(metadata-only) that `lock_timeout` bounds; the index builds use +`CONCURRENTLY`; the backfill takes row locks only. Stopping the archiver and its +GC is fine; stopping the writes is not. + +While the new version runs, the consumers read only rows below +`pg_snapshot_xmin(pg_current_snapshot())`, the oldest transaction of the whole +database: a long write transaction (a binfile import runs in one for up to 30 +minutes) pauses archiving and the GC until it commits, and a slow destination +does too, because a chunk sends inside its transaction and holds the cut back +until it ends. Nothing is lost and the lag gauge shows it; it is not a stuck +consumer. + +## Session settings + +Run every block below in a session with these two settings. They are plain SQL +(no `psql` meta-commands), so any client accepts them, and they last for the +session: if you paste a block into a new session, run them first. + +```sql +SET statement_timeout = 0; +SET lock_timeout = '5s'; +``` + +`statement_timeout = 0` keeps a client's default from killing the backfill or +the `CONCURRENTLY` builds. `lock_timeout = '5s'` makes an `ALTER` that waits +behind a long transaction fail and be retried, instead of queueing in front of +every later query on the table. Both are safe to repeat. + +## 0. Register the steps first + +```sql +INSERT INTO migrations (module, step) +VALUES ('main', '0154-add-job-table'), + ('main', '0159-consumer-generalization') +ON CONFLICT (module, step) DO NOTHING; +``` + +These rows are what make the out-of-band run worth it. Without them, the +runner finds the pre-applied schema and runs the statements again: the guards +skip the work, not the `ACCESS EXCLUSIVE` of each `ALTER` nor the backfill +scan. + +Registering first is the safe order for the locks: from this moment the runner +never touches the group, whatever deploy lands while you work. The other side +of the deal is that the steps below become the only thing that applies the +schema, so finish them **before you deploy the new version**. + +The insert is unconditional: a database that already has the table still gets +its row here. + +## 1. Stop the legacy archiver and its GC + +Turn off the old archiver and its garbage collection with their flags: +`:audit-log-archive` off and `:audit-log-gc` off. The new version uses +`:audit-log-gc` for its own GC, so the same flag keeps both off during the +rollout. The new version has no `:audit-log` flag: writing full rows follows +the consumer flags. Keep on the destinations this instance already uses: +`:nexus` only if it has a Nexus URI after the conversion below (it is an +internal destination), and `:admin-console` only if the instance uses it. At +least one of those flags must be on **on every instance**, web and worker: +each process computes its own persistence value at boot, so a web instance +without a consumer flag writes no rows even if a worker drains them. Do not +turn on a flag whose URI you do not have: a Nexus flag without `:nexus-uri` +fails the boot. 2.19 ignores `:nexus` (the name is new), but it does read +`:admin-console` for its Nitrate client, so keeping that flag on does not +enable the new audit consumer there. Roll the old workers with the old flags +off (the web instances keep writing through the 2.19 code; the other jobs +keep running) and wait for the active archive and GC tasks to finish. With +all consumers off the new version writes no rows, so there is no backlog to +replay later. The old and new versions do not serialize each other: a +concurrent old archiver can mark and send rows the new producer wrote, and +the old GC can delete rows the new consumers have not read yet. + +**Nexus URI.** 2.19 used `:audit-log-archive-uri` as the full HTTP endpoint +(for example `https://nexus.example/proxy/api/audit`). The new consumer reads +`:nexus-uri` as the base and appends `api/audit`: configure +`https://nexus.example/proxy` and check that `:nexus-uri` + `api/audit` is +the URL Nexus expects before the deploy. Copying the old full URL into +`:nexus-uri` duplicates the path. + +## 2. Delete what the archiver already archived + +Delete the rows the old archiver had already archived, so the replay +does not send them to Nexus again. They are the rows the old GC would delete +anyway. Delete in batches (repeat until the statement removes no rows): + +```sql +DELETE FROM audit_log + WHERE id IN (SELECT id FROM audit_log + WHERE archived_at IS NOT NULL + LIMIT 5000); +``` + +Do **not** drop `archived_at` or the indexes that served it +(`audit_log__created_at__idx`, `audit_log__archived_at__idx`, +`audit_log_id_archived_at_idx`). The old archiver may still be running on an +instance that has not been replaced, and a later release removes them. The +consumers order by `(xid, seq)` (step 7); telemetry keeps +`audit_log__source__created_at__idx` plus a sort. + +## 3. Pre-apply the jobs step + +2.19 does not have the `job` table: `0154-add-job-table` creates it and is +new in this release. Step 0 already registered the step, so the runner will +never apply it. Check whether the table exists: + +```sql +SELECT to_regclass('public.job') AS job; +``` + +If it answers a row, nothing to do here: the schema and the registration row +are both in place. If it answers null, run this block as-is (the whole step, copied from +[`backend/src/app/migrations/sql/0154-add-job-table.sql`](../backend/src/app/migrations/sql/0154-add-job-table.sql) +so you can paste it whole): + +```sql +BEGIN; +CREATE TABLE job ( + id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + name text NOT NULL, + tenant text NOT NULL, + queue text NOT NULL, + label text, + priority int NOT NULL DEFAULT 100, + scheduled_at timestamptz NOT NULL DEFAULT now(), + retry_num int NOT NULL DEFAULT 0, + max_retries int NOT NULL DEFAULT 3, + status text NOT NULL DEFAULT 'new' + CHECK (status IN ('new', 'scheduled', 'running', 'retry', + 'completed', 'failed', 'cancelled', + 'aborted')), + created_at timestamptz NOT NULL DEFAULT now(), + modified_at timestamptz NOT NULL DEFAULT now(), + started_at timestamptz, + completed_at timestamptz, + params jsonb NOT NULL DEFAULT '{}', + + -- optional ledger columns (user-facing jobs) + profile_id uuid NULL REFERENCES profile(id) ON DELETE NO ACTION DEFERRABLE, + error jsonb, + result jsonb, + resource_id uuid NULL REFERENCES storage_object(id) ON DELETE SET NULL DEFERRABLE, + expires_at timestamptz +); + +-- Dispatcher claim index: the worker selects due `new`/`retry` rows of +-- its own tenant, orders them by priority and scheduled time, and locks +-- the batch with SKIP LOCKED. The partial predicate keeps terminal rows +-- out and `status` needs no key column for that reason; `tenant` leads +-- because it is an equality the index can answer, and the leading +-- columns then carry the ORDER BY unchanged. `queue` does not lead: the +-- dispatcher claims every queue of its tenant and groups them when +-- pushing, so filtering on one queue would be wrong. +CREATE INDEX job__dispatcher__idx + ON job (tenant, priority DESC, scheduled_at) + WHERE status IN ('new', 'retry'); + +-- Dispatcher orphan sweep: the worker finds the `running` rows of its +-- own tenant whose modified_at is older than the lease and marks them +-- `aborted` (system-side terminal, never retried). +CREATE INDEX job__orphan__idx + ON job (tenant, modified_at) + WHERE status = 'running'; + +-- User-facing job ledger index: it supports filtering by profile and +-- ordering a user's jobs by newest first. No current production query +-- uses it yet; it is kept for the user-facing ledger path. +CREATE INDEX job__profile__idx + ON job (profile_id, created_at DESC) + WHERE profile_id IS NOT NULL; + +-- Storage GC reference check: it answers whether any job still points +-- to a candidate storage object. Rows without a resource are excluded. +CREATE INDEX job__resource__idx + ON job (resource_id) + WHERE resource_id IS NOT NULL; + +-- The lost-sweep index of `scheduled` jobs is created by +-- `0159-consumer-generalization`, on `modified_at` (the hand-off time), +-- which is the column the sweep filters by: building it here on +-- `scheduled_at` would only throw it away in the same transaction. + +-- Jobs GC expiration: it scans jobs past expires_at while excluding +-- running and retry jobs from the expiration delete. +CREATE INDEX job__expires__idx + ON job (expires_at) + WHERE expires_at IS NOT NULL; + +-- Jobs GC retention: it finds old internal terminal jobs with no +-- profile, which are eligible for removal after the retention delay. +CREATE INDEX job__retention__idx + ON job (modified_at) + WHERE status IN ('completed', 'failed', 'cancelled', 'aborted') + AND profile_id IS NULL; + +-- Append-only job event log: `start`, `progress`, `retry` and `end` rows in +-- insertion order. This is the only source of truth for progress; there is no +-- Redis key and no mutable progress column. The foreign key cascades so +-- deleting a job removes its whole history, and it is DEFERRABLE like every +-- other new reference of the substrate. +CREATE TABLE job_event ( + id bigserial PRIMARY KEY, + job_id uuid NOT NULL REFERENCES job(id) ON DELETE CASCADE DEFERRABLE, + kind text NOT NULL + CHECK (kind IN ('start', 'progress', 'retry', 'end')), + payload jsonb NOT NULL DEFAULT '{}', + created_at timestamptz NOT NULL DEFAULT now() +); + +-- Event history read: all events of a job, and the last progress report +-- (`WHERE job_id = ? AND kind = 'progress' ORDER BY created_at DESC LIMIT 1`). +CREATE INDEX job_event__job_kind_created_idx + ON job_event (job_id, kind, created_at DESC, id DESC); + +-- Deletion protection, like the domain tables (`team`, `profile`, the +-- file thumbnail tables). The only intended deleter of `job` rows is the +-- jobs GC, which also marks the referenced storage objects as touched in +-- the same transaction; a stray `DELETE FROM job` would drop the row +-- (and cascade `job_event`) without touching the object, orphaning it +-- forever. The jobs GC disables the guard inside its transaction with +-- `SET LOCAL rules.deletion_protection TO off`, the same escape hatch used +-- by `app.tasks.objects-gc`; `app.jobs/submit` never deletes. +CREATE OR REPLACE TRIGGER deletion_protection__tgr +BEFORE DELETE ON job FOR EACH STATEMENT + WHEN ((current_setting('rules.deletion_protection', true) IN ('on', '')) OR + (current_setting('rules.deletion_protection', true) IS NULL)) + EXECUTE PROCEDURE raise_deletion_protection(); +COMMIT; +``` + +The tables are new and empty and the old version never writes them, so every +statement is instant. A failure rolls the block back: re-run it whole +instead of completing objects by hand. The block is the whole step, not just +the tables: `job` with its six indexes (dispatcher, orphan, profile, +resource, expires, retention), `job_event` with its history index, and the +`deletion_protection__tgr` trigger. Step 9 checks every one of them. + +The old version never touches this table, but the new version needs it to +boot. + +## 4. Audit log: columns, reserved range and defaults + +Columns first (metadata-only), the reserved low range and the defaults the +old web instances need immediately: they keep inserting while the DDL and the +backfill run, and the database fills the new columns for them. The old code +ignores them. + +```sql +CREATE SEQUENCE IF NOT EXISTS audit_log_seq_seq; +ALTER TABLE audit_log ADD COLUMN IF NOT EXISTS seq bigint; +ALTER TABLE audit_log ADD COLUMN IF NOT EXISTS xid xid8 NOT NULL DEFAULT '0'::xid8; +ALTER SEQUENCE audit_log_seq_seq OWNED BY audit_log.seq; + +-- Reserve the low range for the history: new rows start above the backfill. +-- Only on the first application: a sequence that already left 1 is reserved +-- by the operator or in use, and a re-run must not move it. +SELECT setval('audit_log_seq_seq', + (SELECT count(*) + 1000000 FROM audit_log), + false) + WHERE (SELECT last_value FROM audit_log_seq_seq) = 1 + AND NOT (SELECT is_called FROM audit_log_seq_seq); + +ALTER TABLE audit_log ALTER COLUMN seq SET DEFAULT nextval('audit_log_seq_seq'); +ALTER TABLE audit_log ALTER COLUMN xid SET DEFAULT pg_current_xact_id(); +``` + +## 5. Audit log: backfill + +One statement, row locks only (never a table lock), and it filters +`seq IS NULL`, so it can be interrupted and re-run. The +archiver is stopped, but the web instances keep inserting; their new rows get +`seq` from the default, so the backfill only touches the history. It sorts by +`(created_at, id)`: the `(created_at, id)` index this branch used belonged to +the pending queue and never shipped. On a large table, raise `work_mem` for the +session before running it so the sort stays in memory: + +```sql +UPDATE audit_log + SET seq = ordered.rn + FROM (SELECT id, row_number() OVER (ORDER BY created_at, id) AS rn + FROM audit_log + WHERE seq IS NULL) ordered + WHERE audit_log.id = ordered.id; +``` + +## 6. Audit log: `seq` and `source` NOT NULL without a long `ACCESS EXCLUSIVE` scan + +No row can be `NULL` in `seq` here: the default was set in step 4. `source` is +nullable since `0064`; fill what is left first. + +```sql +ALTER TABLE audit_log ADD CONSTRAINT audit_log__seq__not_null + CHECK (seq IS NOT NULL) NOT VALID; +ALTER TABLE audit_log VALIDATE CONSTRAINT audit_log__seq__not_null; +ALTER TABLE audit_log ALTER COLUMN seq SET NOT NULL; +ALTER TABLE audit_log DROP CONSTRAINT audit_log__seq__not_null; + +UPDATE audit_log SET source = 'backend' WHERE source IS NULL; + +ALTER TABLE audit_log ADD CONSTRAINT audit_log__source__not_null + CHECK (source IS NOT NULL) NOT VALID; +ALTER TABLE audit_log VALIDATE CONSTRAINT audit_log__source__not_null; +ALTER TABLE audit_log ALTER COLUMN source SET NOT NULL; +ALTER TABLE audit_log DROP CONSTRAINT audit_log__source__not_null; +``` + +## 7. Audit log: cursor index and cursor table + +The index with `CONCURRENTLY`, and the cursor table, which starts empty: + +```sql +CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS audit_log__xid__seq__idx + ON audit_log (xid, seq); + +CREATE TABLE IF NOT EXISTS audit_log_consumer_cursor ( + consumer smallint NOT NULL, + xid xid8 NOT NULL, + seq bigint NOT NULL, + updated_at timestamptz NOT NULL DEFAULT now(), + PRIMARY KEY (consumer)); +``` + +## 8. Jobs: exclusive and dedupe columns and indexes + +The two columns and the two unique indexes, the indexes with `CONCURRENTLY`: + +```sql +ALTER TABLE job ADD COLUMN IF NOT EXISTS exclusive boolean NOT NULL DEFAULT false; +ALTER TABLE job ADD COLUMN IF NOT EXISTS dedupe boolean NOT NULL DEFAULT false; + +CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS job__live_exclusive__uniq + ON job (tenant, name, queue, label) + WHERE exclusive AND status IN ('new', 'scheduled', 'running', 'retry'); + +CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS job__new_dedupe__uniq + ON job (tenant, name, queue, label) + WHERE dedupe AND status = 'new'; +``` + +The lost sweep index changes shape: it filters by `modified_at`, not by +`scheduled_at`. `job` is not in production yet, so a plain rebuild is enough: + +```sql +DROP INDEX IF EXISTS job__scheduled__idx; +CREATE INDEX IF NOT EXISTS job__scheduled__idx + ON job (tenant, modified_at) + WHERE status = 'scheduled'; +``` + +If an early `0154` built the cron probe index, drop it too (the submit resolves +the overlap at insert time now): + +```sql +DROP INDEX IF EXISTS job__name_label__idx; +``` + +## 9. Gate before deploying + +An interrupted `CONCURRENTLY` build leaves the index `INVALID`, and +`IF NOT EXISTS` matches by name, so the migration would accept it and the +dedupe/exclusivity would stop enforcing silently. The checks: + +```sql +SELECT indexname, indexdef FROM pg_indexes + WHERE indexname IN ('job__live_exclusive__uniq', + 'job__new_dedupe__uniq', + 'job__scheduled__idx', + 'audit_log__xid__seq__idx'); + +SELECT indexrelid::regclass AS index, indisvalid, indisunique + FROM pg_index + WHERE indexrelid IN ('job__live_exclusive__uniq'::regclass, + 'job__new_dedupe__uniq'::regclass, + 'audit_log__xid__seq__idx'::regclass); + +SELECT column_name, is_nullable + FROM information_schema.columns + WHERE table_name = 'audit_log' + AND column_name IN ('seq', 'xid', 'source') + ORDER BY 1; +``` + +Any index with `indisvalid = false` must be dropped and rebuilt before the +deploy, and the three columns of `audit_log` must answer `NO`. + +The job side is a gate too: both job tables, the cursor table, the indexes of +the check and the trigger must exist before the deploy. A half-applied `0154` +or a skipped step 8 would otherwise only surface as a boot failure. + +```sql +SELECT to_regclass('public.job') IS NOT NULL AS job, + to_regclass('public.job_event') IS NOT NULL AS job_event, + to_regclass('public.audit_log_consumer_cursor') IS NOT NULL AS cursor; + +SELECT indexname FROM pg_indexes + WHERE tablename IN ('job', 'job_event') + AND indexname IN ('job__dispatcher__idx', 'job__orphan__idx', + 'job__profile__idx', 'job__resource__idx', + 'job__expires__idx', 'job__retention__idx', + 'job__scheduled__idx', 'job_event__job_kind_created_idx') + ORDER BY 1; + +SELECT column_name, is_nullable + FROM information_schema.columns + WHERE table_name = 'job' + AND column_name IN ('exclusive', 'dedupe') + ORDER BY 1; + +SELECT tgname FROM pg_trigger + WHERE tgrelid = to_regclass('public.job') + AND tgname = 'deletion_protection__tgr' + AND NOT tgisinternal; +``` + +The three tables must answer true, the eight indexes must be listed, the two +flags must answer `NO`, and the trigger must appear. The +`job__scheduled__idx` definition of the first check must read +`(tenant, modified_at)`, not `(tenant, scheduled_at)`: step 8 drops and +rebuilds it, and `CREATE INDEX IF NOT EXISTS` matches by name. This is a gate: +do not deploy if it fails. The catalog checks are instant; the +`CHECK NOT VALID` / `VALIDATE` / `SET NOT NULL` dance already proved there is +no NULL, so a `count(*)` would only scan the table again. + +## 10. After the rollout + +Once every instance runs this version, the new consumers have replayed the +unarchived buffer. Turn on `:audit-log-gc` (rolling restart of the new +workers) so the new GC deletes the rows the consumers have passed. Remove +`:audit-log-archive`; the new version does not use it. diff --git a/operations/2.20.0-file-change-partition.md b/operations/2.20.0-file-change-partition.md index 30d1185a92..de1f2ab48c 100644 --- a/operations/2.20.0-file-change-partition.md +++ b/operations/2.20.0-file-change-partition.md @@ -144,9 +144,6 @@ INSERT INTO migrations (module, step) VALUES ('main', '0158-partition-file-change'); ``` -The module is `main` and the step has no `.sql` (see -`backend/src/app/util/migrations.clj`). - ### 0.2 Add the `kind` column to the current table Until the cutover, `file_change` is the current table, and the new code writes @@ -849,10 +846,9 @@ SELECT 'partition constraints out of name', count(*), 0 AND c.conname LIKE 'file_change_new%'; ``` - The `NOT NULL` constraints (`contype = 'n'`, tracked in `pg_constraint` from - PostgreSQL 17) are excluded from the last check: their name is generated by - the column and changes with the server version, so they are not part of the - parity. The `CHECK`, the primary key and the foreign keys are. + The `NOT NULL` constraints are excluded from the last check (`contype <> + 'n'`): their names are not stable across server versions, so they are not + part of the parity. The `CHECK`, the primary key and the foreign keys are. And the names, to compare them with the migration: diff --git a/operations/2.20.0-file-data-storage-object-id.md b/operations/2.20.0-file-data-storage-object-id.md index 5355c3a909..2f87e51bcc 100644 --- a/operations/2.20.0-file-data-storage-object-id.md +++ b/operations/2.20.0-file-data-storage-object-id.md @@ -16,18 +16,25 @@ short as possible: - the column, the index and the foreign key are built online, without stopping the service; - the backfill runs in the stopped window **only if** the metadata pointer - exists; when the storage backend was never enabled there is nothing to - backfill and the new binaries roll out like any other deploy; + exists; when the storage backend is off, the pre-flight finds no pointer, + tiered offload is off and the queue is empty, there is nothing to backfill + and the new binaries roll out like any other deploy; - the step is then marked as applied in the `migrations` table, so the migration body does not run at startup. -The coordinated restart is required only when the instance uses the storage -backend (`PENPOT_FILE_DATA_BACKEND=storage`) or still has rows that carry the -metadata pointer. The previous code writes and reads `metadata.storage-ref-id`, -the new code reads the column, and a rolling deploy would mix both. With the -default `db` backend and no row carrying the pointer (see the pre-flight -query), the previous and the new code read the same `file_data.data` column, so -a rolling deploy is safe. +The coordinated restart is required when the instance uses the storage backend +(`PENPOT_FILE_DATA_BACKEND=storage`), has `:tiered-file-data-storage` on or +pending `offload-file-data` tasks, or still has rows that carry the metadata +pointer. 2.19 writes storage rows through the tiered offload even with the +default `db` backend: turn the flag off and wait for the queue before the +pre-flight. The previous code writes and reads `metadata.storage-ref-id`, the +new code reads the column, and a rolling deploy would mix both. With the +default `db` backend, no tiered offload, no pending offloads and no row +carrying the pointer (see the pre-flight query), the previous and the new code +read the same `file_data.data` column, so a rolling deploy is safe. + +**This procedure is one-way.** It has no schema rollback: after an attempt, +restore the backup. The column is the only reference for the rows 2.20 writes. **Do not run the backfill while the previous version is serving** when `PENPOT_FILE_DATA_BACKEND` is configured to `storage`, or when the table still @@ -35,22 +42,32 @@ has rows with the metadata pointer. The previous code writes the pointer only to `metadata.storage-ref-id`, and its storage GC reads the reference from there. Removing the key while that code runs leaves the rows it writes without a pointer (unreadable after the deploy), and it can make the GC delete an object -that a live row still references. When the pre-flight query returns 0 there is -nothing to backfill and this risk does not apply. +that a live row still references. When the pre-flight query returns 0, tiered +offload is off and the queue is empty, there is nothing to backfill and this +risk does not apply. ## 0. Pre-flight ```sql -SELECT version(); -- 16.x: `NOT VALID` foreign keys on a partitioned table need 18 - SELECT count(*) FROM pg_inherits WHERE inhparent = 'file_data'::regclass; -- 16 --- Whether the storage backend was ever enabled. When this is 0 there is no --- pointer in the metadata and step 6 is skipped. +-- Before the next two queries, turn off `:tiered-file-data-storage` in every +-- 2.19 worker and wait for the pending offloads: that task writes storage rows +-- with the default db backend. +-- +-- Whether the storage backend was ever used. When this is 0 there is no +-- pointer in the metadata and step 6 is skipped; it does not prove the backend +-- is off. SELECT count(*) FROM file_data WHERE backend = 'storage' AND jsonb_exists(metadata, 'storage-ref-id'); +-- offloads queued or running on the 2.19 task table. Must be 0. +SELECT count(*) AS pending_offloads + FROM task + WHERE name = 'offload-file-data' + AND status NOT IN ('completed', 'failed'); + -- Every online ALTER uses a short lock timeout and is retried, so a long -- transaction cannot make it queue and block the table behind it. SELECT pid, state, now() - xact_start AS age, left(query, 60) @@ -79,7 +96,36 @@ it did not block anything. ```sql CREATE INDEX CONCURRENTLY file_data_00__storage_object_id__idx ON file_data_00 (storage_object_id); --- ... up to file_data_15 +CREATE INDEX CONCURRENTLY file_data_01__storage_object_id__idx + ON file_data_01 (storage_object_id); +CREATE INDEX CONCURRENTLY file_data_02__storage_object_id__idx + ON file_data_02 (storage_object_id); +CREATE INDEX CONCURRENTLY file_data_03__storage_object_id__idx + ON file_data_03 (storage_object_id); +CREATE INDEX CONCURRENTLY file_data_04__storage_object_id__idx + ON file_data_04 (storage_object_id); +CREATE INDEX CONCURRENTLY file_data_05__storage_object_id__idx + ON file_data_05 (storage_object_id); +CREATE INDEX CONCURRENTLY file_data_06__storage_object_id__idx + ON file_data_06 (storage_object_id); +CREATE INDEX CONCURRENTLY file_data_07__storage_object_id__idx + ON file_data_07 (storage_object_id); +CREATE INDEX CONCURRENTLY file_data_08__storage_object_id__idx + ON file_data_08 (storage_object_id); +CREATE INDEX CONCURRENTLY file_data_09__storage_object_id__idx + ON file_data_09 (storage_object_id); +CREATE INDEX CONCURRENTLY file_data_10__storage_object_id__idx + ON file_data_10 (storage_object_id); +CREATE INDEX CONCURRENTLY file_data_11__storage_object_id__idx + ON file_data_11 (storage_object_id); +CREATE INDEX CONCURRENTLY file_data_12__storage_object_id__idx + ON file_data_12 (storage_object_id); +CREATE INDEX CONCURRENTLY file_data_13__storage_object_id__idx + ON file_data_13 (storage_object_id); +CREATE INDEX CONCURRENTLY file_data_14__storage_object_id__idx + ON file_data_14 (storage_object_id); +CREATE INDEX CONCURRENTLY file_data_15__storage_object_id__idx + ON file_data_15 (storage_object_id); ``` Run each statement in autocommit: `CREATE INDEX CONCURRENTLY` cannot run inside @@ -102,7 +148,36 @@ CREATE INDEX file_data__storage_object_id__idx ALTER INDEX file_data__storage_object_id__idx ATTACH PARTITION file_data_00__storage_object_id__idx; --- ... up to file_data_15 +ALTER INDEX file_data__storage_object_id__idx + ATTACH PARTITION file_data_01__storage_object_id__idx; +ALTER INDEX file_data__storage_object_id__idx + ATTACH PARTITION file_data_02__storage_object_id__idx; +ALTER INDEX file_data__storage_object_id__idx + ATTACH PARTITION file_data_03__storage_object_id__idx; +ALTER INDEX file_data__storage_object_id__idx + ATTACH PARTITION file_data_04__storage_object_id__idx; +ALTER INDEX file_data__storage_object_id__idx + ATTACH PARTITION file_data_05__storage_object_id__idx; +ALTER INDEX file_data__storage_object_id__idx + ATTACH PARTITION file_data_06__storage_object_id__idx; +ALTER INDEX file_data__storage_object_id__idx + ATTACH PARTITION file_data_07__storage_object_id__idx; +ALTER INDEX file_data__storage_object_id__idx + ATTACH PARTITION file_data_08__storage_object_id__idx; +ALTER INDEX file_data__storage_object_id__idx + ATTACH PARTITION file_data_09__storage_object_id__idx; +ALTER INDEX file_data__storage_object_id__idx + ATTACH PARTITION file_data_10__storage_object_id__idx; +ALTER INDEX file_data__storage_object_id__idx + ATTACH PARTITION file_data_11__storage_object_id__idx; +ALTER INDEX file_data__storage_object_id__idx + ATTACH PARTITION file_data_12__storage_object_id__idx; +ALTER INDEX file_data__storage_object_id__idx + ATTACH PARTITION file_data_13__storage_object_id__idx; +ALTER INDEX file_data__storage_object_id__idx + ATTACH PARTITION file_data_14__storage_object_id__idx; +ALTER INDEX file_data__storage_object_id__idx + ATTACH PARTITION file_data_15__storage_object_id__idx; ``` The step creates the index unconditionally: if an index with that name already @@ -124,6 +199,7 @@ SELECT count(*) FROM pg_inherits ```sql BEGIN; SET LOCAL lock_timeout = '3s'; +SET LOCAL statement_timeout = '10min'; -- measure first; see below ALTER TABLE file_data ADD CONSTRAINT file_data_storage_object_id_fkey FOREIGN KEY (storage_object_id) REFERENCES storage_object(id) @@ -131,10 +207,15 @@ ALTER TABLE file_data COMMIT; ``` -The column is NULL until step 6, so the validation finds nothing to check and -the `SHARE ROW EXCLUSIVE` lock (it blocks writes, not reads) is brief. Retry on -`lock timeout`. If the constraint already exists, do not add it again: verify -its shape with the query in step 7. Drop it first only if the shape is wrong. +The column is NULL until step 6, but PostgreSQL still scans the table (or the +index built in the previous steps) to prove there are no violations. The +`SHARE ROW EXCLUSIVE` lock blocks writes for the whole scan, not only while +the lock is acquired: `lock_timeout` bounds the wait for the lock and +`statement_timeout` bounds the scan. A timeout aborts the `ALTER`, releases +the writes and lets you retry in a quieter window; measure the scan in staging +and set the budget accordingly. Retry on `lock timeout`. If the constraint +already exists, do not add it again: verify its shape with the query in step 7. +Drop it first only if the shape is wrong. Creating it here, before the backfill, keeps the full validation scan out of the stopped window: the backfill validates each row it writes. @@ -144,43 +225,221 @@ the chunk transaction and retries the chunk; an `INITIALLY DEFERRED` check fires at commit, outside that transaction, and the GC would never move past a blocked chunk. +### Temporary index for pending backfill rows — online + +Build this partial index online, before the stop. It covers +only rows that still need migration. Each successful batch removes its rows +from the index, so later batches do not scan completed rows. Drop the parent +index after the backfill. + +Run each `CREATE INDEX CONCURRENTLY` in autocommit: + +```sql +CREATE INDEX CONCURRENTLY file_data_00__storage_ref_pending__idx + ON file_data_00 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +CREATE INDEX CONCURRENTLY file_data_01__storage_ref_pending__idx + ON file_data_01 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +CREATE INDEX CONCURRENTLY file_data_02__storage_ref_pending__idx + ON file_data_02 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +CREATE INDEX CONCURRENTLY file_data_03__storage_ref_pending__idx + ON file_data_03 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +CREATE INDEX CONCURRENTLY file_data_04__storage_ref_pending__idx + ON file_data_04 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +CREATE INDEX CONCURRENTLY file_data_05__storage_ref_pending__idx + ON file_data_05 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +CREATE INDEX CONCURRENTLY file_data_06__storage_ref_pending__idx + ON file_data_06 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +CREATE INDEX CONCURRENTLY file_data_07__storage_ref_pending__idx + ON file_data_07 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +CREATE INDEX CONCURRENTLY file_data_08__storage_ref_pending__idx + ON file_data_08 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +CREATE INDEX CONCURRENTLY file_data_09__storage_ref_pending__idx + ON file_data_09 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +CREATE INDEX CONCURRENTLY file_data_10__storage_ref_pending__idx + ON file_data_10 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +CREATE INDEX CONCURRENTLY file_data_11__storage_ref_pending__idx + ON file_data_11 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +CREATE INDEX CONCURRENTLY file_data_12__storage_ref_pending__idx + ON file_data_12 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +CREATE INDEX CONCURRENTLY file_data_13__storage_ref_pending__idx + ON file_data_13 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +CREATE INDEX CONCURRENTLY file_data_14__storage_ref_pending__idx + ON file_data_14 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +CREATE INDEX CONCURRENTLY file_data_15__storage_ref_pending__idx + ON file_data_15 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +``` + +Create and attach the matching parent index: + +```sql +CREATE INDEX file_data__storage_ref_pending__idx ON ONLY file_data (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); + +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_00__storage_ref_pending__idx; +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_01__storage_ref_pending__idx; +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_02__storage_ref_pending__idx; +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_03__storage_ref_pending__idx; +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_04__storage_ref_pending__idx; +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_05__storage_ref_pending__idx; +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_06__storage_ref_pending__idx; +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_07__storage_ref_pending__idx; +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_08__storage_ref_pending__idx; +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_09__storage_ref_pending__idx; +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_10__storage_ref_pending__idx; +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_11__storage_ref_pending__idx; +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_12__storage_ref_pending__idx; +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_13__storage_ref_pending__idx; +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_14__storage_ref_pending__idx; +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_15__storage_ref_pending__idx; + +SELECT indisvalid, indpred IS NOT NULL AS partial + FROM pg_index + WHERE indexrelid = 'file_data__storage_ref_pending__idx'::regclass; +-- t | t + +SELECT count(*) AS attached_partitions + FROM pg_inherits + WHERE inhparent = 'file_data__storage_ref_pending__idx'::regclass; +-- 16 +``` + ## 5. Stop every instance -Coordinated restart when the pre-flight count is not 0: stop all of them before -the next step. No rolling deploy. When the count is 0 there is nothing to -backfill: run steps 1-4 online, run the verification (step 7) and mark the step -as applied (step 8) without stopping the service. +Stop every instance before the next step when the coordinated restart applies +(see "When to run this"): the storage backend is on, the tiered offload is on +or has pending `offload-file-data` tasks, or the pre-flight found rows with the +metadata pointer. No rolling deploy. + +When there is no storage writer (no `PENPOT_FILE_DATA_BACKEND=storage`, no +`:tiered-file-data-storage`, no pending `offload-file-data` task) and the +pre-flight count is 0, there is nothing to backfill: run steps 1-4 online, run +the verification (step 7) and mark the step as applied (step 8) without +stopping the service. ## 6. Backfill, only if the metadata pointer exists -Skip this step when the pre-flight count is 0: the storage backend was never -enabled, no row carries the pointer, and the new code reads all file data from -the `data` column. +Skip this step only when there is no storage writer (no +`PENPOT_FILE_DATA_BACKEND=storage`, no `:tiered-file-data-storage`, no pending +`offload-file-data` task) and the pre-flight count is 0: no row carries the +pointer, and the new code reads all file data from the `data` column. If a +writer was present, stop it and repeat the count before deciding; a writer can +create the first row after the pre-flight. ```sql WITH candidates AS MATERIALIZED ( - SELECT file_id, - id, - (metadata->>'storage-ref-id')::uuid AS soid - FROM file_data - WHERE backend = 'storage' - AND storage_object_id IS NULL - AND (metadata->>'storage-ref-id') ~* + SELECT fd.file_id, + fd.id, + CASE WHEN (fd.metadata->>'storage-ref-id') ~* + '^[0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{12}$' + THEN (fd.metadata->>'storage-ref-id')::uuid END AS soid + FROM file_data AS fd + JOIN storage_object AS so + ON so.id = CASE WHEN (fd.metadata->>'storage-ref-id') ~* + '^[0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{12}$' + THEN (fd.metadata->>'storage-ref-id')::uuid END + WHERE fd.backend = 'storage' + AND fd.storage_object_id IS NULL + AND jsonb_exists(fd.metadata, 'storage-ref-id') + AND (fd.metadata->>'storage-ref-id') ~* '^[0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{12}$' + ORDER BY fd.file_id, fd.id + LIMIT 5000 + ), + updated AS ( + UPDATE file_data AS fd + SET storage_object_id = c.soid, + metadata = fd.metadata - 'storage-ref-id' + FROM candidates AS c + WHERE fd.file_id = c.file_id + AND fd.id = c.id + RETURNING 1 ) -UPDATE file_data AS fd - SET storage_object_id = c.soid, - metadata = fd.metadata - 'storage-ref-id' - FROM candidates AS c - WHERE fd.file_id = c.file_id - AND fd.id = c.id - AND EXISTS (SELECT 1 FROM storage_object AS so WHERE so.id = c.soid); +SELECT count(*) AS rows_updated FROM updated; ``` -The `MATERIALIZED` CTE keeps the uuid cast away from values that did not pass -the regex: without it PostgreSQL can inline the subquery and cast a corrupt -value, aborting the statement. The foreign key already exists, so every written -row is validated here instead of in a separate full scan. +Run the statement repeatedly, committing each batch separately, until +`rows_updated` is 0. Each batch updates at most 5,000 rows and removes the metadata +key in the same update that sets the column. The partial index keeps later +candidate searches focused on rows still pending. The `MATERIALIZED` CTE keeps +the UUID cast away from values that fail the regex. The join to +`storage_object` excludes orphan pointers, and the foreign key validates each +written row. Corrupt and orphan pointers remain untouched for investigation; +they must be resolved before the unreadable-row verification can pass. + +After all batches finish, remove the temporary index: + +```sql +DROP INDEX file_data__storage_ref_pending__idx; +``` + +The migrated key is not retained as a rollback copy. A rollback to the previous +binary after the backfill requires restoring the backup. ## 7. Verify @@ -205,6 +464,11 @@ SELECT data_type, is_nullable SELECT count(*) FROM file_data WHERE backend = 'storage' AND storage_object_id IS NULL; +-- Migrated rows no longer retain the old pointer. Must be 0. +SELECT count(*) FROM file_data + WHERE backend = 'storage' AND storage_object_id IS NOT NULL + AND jsonb_exists(metadata, 'storage-ref-id'); + -- Index: valid, plain, with every partition attached. SELECT i.indisvalid, i.indpred IS NULL AS plain, i.indexprs IS NULL AS no_expression FROM pg_index AS i @@ -242,8 +506,9 @@ migration runs at the next start and repeats the backfill and the foreign key. ## 9. Start the new instances -Start every instance you stopped in step 5. When the pre-flight count was 0 -nothing was stopped, and the rolling deploy simply finishes. +Start every instance you stopped in step 5. When there was no storage writer +and the pre-flight count was 0, nothing was stopped, and the rolling deploy +simply finishes. ## Post-deploy checks @@ -261,25 +526,3 @@ EXPLAIN (ANALYZE, BUFFERS) DELETE FROM storage_object WHERE id = ''; ROLLBACK; ``` - -## Rollback - -Before step 8: - -```sql -ALTER TABLE file_data - DROP CONSTRAINT IF EXISTS file_data_storage_object_id_fkey; - --- A partitioned index cannot be dropped with CONCURRENTLY; the plain drop --- takes a brief ACCESS EXCLUSIVE. -DROP INDEX file_data__storage_object_id__idx; - -ALTER TABLE file_data DROP COLUMN storage_object_id; -``` - -After step 8 the backfill has already removed the metadata key that the -previous code reads, so a rollback to the previous binaries is **not -supported** when the instance uses the storage backend: the files offloaded by -the new code would be unreadable, and the recovery is a data restore, not a -binary switch. With the default `db` backend and no storage rows, the previous -binaries read the same `data` column and the rollback is a plain binary switch. diff --git a/operations/2.20.0-unified.md b/operations/2.20.0-unified.md new file mode 100644 index 0000000000..cbb62f0d16 --- /dev/null +++ b/operations/2.20.0-unified.md @@ -0,0 +1,1613 @@ +# 2.20.0 — Unified upgrade runbook + +Single, self-contained procedure for upgrading an instance to the release +that ships: + +- Partition `file_change` by `HASH (file_id)` with 16 partitions, aligned + with `file_data`: `backend/src/app/migrations/sql/0158-partition-file-change.sql`, + issue . +- The file data storage pointer as a column `file_data.storage_object_id` + with a foreign key to `storage_object(id)`: + `backend/src/app/migrations/sql/0159-add-file-data-storage-object-id.sql`, + issue . +- The audit-log cursor consumers and the jobs substrate: + `backend/src/app/migrations/sql/0154-add-job-table.sql` and + `backend/src/app/migrations/sql/0159-consumer-generalization.sql`, + issue . + +Migrations apply in this order at boot when they are not registered in the +`migrations` table: `0154-add-job-table`, `0158-partition-file-change`, +`0159-add-file-data-storage-object-id`, then `0159-consumer-generalization`. + +**This document is self-contained and manual.** Every block is plain SQL, +paste-able into any client that speaks SQL; there are no shell scripts and no +`psql` meta-commands. It runs on PostgreSQL 15 or newer. It does not replace +`operations/2.20.0-file-change-partition.md`, +`operations/2.20.0-file-data-storage-object-id.md` or +`operations/2.20.0-consumer-generalization.md`: they stay as the reference for +their individual procedure. Where this document and those disagree, this +document wins for the unified execution; they win for the details of one +area only. + +## When each phase runs + +| Phase | When | Service | +|---|---|---| +| 1 | Before the upgrade; pre-flight and online preparation | 2.19.0 serving | +| 2 | The deploy to 2.20.0 | **Full stop** for the storage backfill | +| 3 | After the upgrade, 2.20.0 serving; phase 3.3 every 1-6 h for days | Online | +| 4 | **A scheduled maintenance window** for the `file_change` partition cutover | **Backend and workers stopped** | +| 5 | After the cutover | Online | + +This runbook is a complete manual procedure. Follow every phase in order, +regardless of database size; do not skip steps based on how much data a table +contains. Phase 2 is a coordinated full stop of the backend and workers for +the storage backfill during the deploy. Phase 4 is a separate scheduled +maintenance window for the `file_change` partition cutover after the deploy. +The backfill runs on every instance, even when it updates no rows. + +## Ordering constraints (do not rearrange) + +- The migration steps are pre-registered at the beginning: the 2.20.0 + migration runner skips them, and the manual procedure below is the only path + that applies their schema changes and backfills. +- Phases 3 and 4 need the **new code** running: it moves `updated_at` on + every write, the watermark of the copy passes. The deploy (phase 2) goes + before phase 3. +- The `file_data` backfill always runs in phase 2, while every old instance + is stopped and **before** the new instances start: the old code writes the + pointer only to `metadata.storage-ref-id`, the new code reads only the + column. A rolling deploy in between would mix both. +- The `0158` cutover (phase 4) takes minutes in a low-traffic window. Saves + that fall inside fail and the user retries. +- After **every** instance runs 2.20.0, turn on `:audit-log-gc` (phase 5.6). + Until then the old and new versions coexist and the new GC must stay off. + + +## Phase 1 — Before the upgrade (2.19.0 serving; online) + +### 1.1 Register every migration step first + +Run these inserts before any other step in this runbook. They prevent the +2.20.0 migration runner from starting these jobs automatically. The runbook +then owns all schema changes and backfills. Do not deploy 2.20.0 until every +manual step and verification in this document is complete. + +```sql +INSERT INTO migrations (module, step) +VALUES ('main', '0158-partition-file-change') +ON CONFLICT (module, step) DO NOTHING; + +INSERT INTO migrations (module, step) +VALUES ('main', '0154-add-job-table') +ON CONFLICT (module, step) DO NOTHING; + +INSERT INTO migrations (module, step) +VALUES ('main', '0159-consumer-generalization') +ON CONFLICT (module, step) DO NOTHING; + +INSERT INTO migrations (module, step) +VALUES ('main', '0159-add-file-data-storage-object-id') +ON CONFLICT (module, step) DO NOTHING; +``` + +These rows are pre-registration for this manual procedure, not permission to +deploy early. If the procedure is interrupted, keep 2.20.0 stopped and resume +the manual work before deploying. + +### 1.2 Pre-flight sizing + +```sql +-- total size of the database and of file_change (heap + TOAST + indexes) +SELECT pg_size_pretty(pg_database_size(current_database())) AS database; +SELECT pg_size_pretty(pg_total_relation_size('file_change')) AS file_change_total, + pg_size_pretty(pg_table_size('file_change')) AS heap_and_toast, + pg_size_pretty(pg_indexes_size('file_change')) AS indexes; +SELECT count(*) AS rows FROM file_change; + +-- audit_log volume: estimate the time for the out-of-band index builds +SELECT pg_size_pretty(pg_total_relation_size('audit_log')) AS audit_log; +SELECT count(*) FROM audit_log; + +-- backfill cost of file_change is proportional to pg_table_size, +-- not to the row count: time the first batches in phase 3 and extrapolate +``` + +These measurements help estimate the time and disk space the procedure will +need. They do not change which steps to run: this runbook always applies the +full `file_change`, `audit_log` and `file_data` procedures. + + + +After pre-flight, run subsections 1.3, 1.4 and 1.5 in order while 2.19 +serves traffic. Do not stop the web instances. Subsection 1.4 stops the +legacy archiver and its GC by flag. + +### 1.3 Partition preparation + +**1.3.1 Add the `kind` column to the current table.** Until the cutover, +`file_change` is the current table, and the new code writes `kind` on every +insert: without the column the backend does not start. + +```sql +ALTER TABLE file_change ADD COLUMN IF NOT EXISTS kind smallint; +``` + +Nullable and without a default on purpose: it is a metadata-only operation +(it does not rewrite a single row), the old code keeps inserting rows, and +the values of the rows that are already there are computed by the copy from +their `label`. + +**1.3.2 Index for the delta passes.** The delta passes filter by +`updated_at`. Without an index on that column every pass scans the whole +heap; with it, it is an index range that only touches the rows that changed. + +```sql +-- CONCURRENTLY: it does not block the writes. It cannot run inside a +-- transaction: run it alone. +CREATE INDEX CONCURRENTLY IF NOT EXISTS file_change__updated_at__idx + ON file_change (updated_at); +``` + +- If it is interrupted, the index is left `INVALID` and has to be dropped + and built again: + +```sql +SELECT i.indisvalid + FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid + WHERE c.relname = 'file_change__updated_at__idx'; +``` + +- The index goes with the old table: at the cutover it is renamed to + `file_change__updated_at__idx_legacy` and disappears with it. The new + table does not carry it (nothing queries by `updated_at`; the GC uses + `deleted_at`). + +### 1.4 Consumer generalization: audit_log and jobs + +**Session settings:** `SET statement_timeout = 0; SET lock_timeout = '5s';` +Re-issue them in each SQL session. + +**Step 1 — Stop the legacy archiver and its GC** + +1. Turn off `:audit-log-archive` and `:audit-log-gc`. There is no `:audit-log` + flag in the new version: writing full rows follows the consumer flags. +2. Enable `:nexus` only if `:nexus-uri` is configured. Keep `:admin-console` + on if used. At least one of them must be on **on every instance**, web and + worker: each process computes its own persistence value at boot, so a web + instance without a consumer flag writes no rows even if a worker drains + them. With all consumers off there is no backlog to replay later. +3. Set `:nexus-uri` to the base URL, for example + `https://nexus.example/proxy`. The new consumer appends `/api/audit`; do not + copy the old full `:audit-log-archive-uri` into `:nexus-uri`. +4. Roll the workers with these flags and wait for active archive and GC tasks + to finish. + +**Step 2 — Delete what the archiver already archived** + +```sql +DELETE FROM audit_log + WHERE id IN (SELECT id FROM audit_log + WHERE archived_at IS NOT NULL + LIMIT 5000); +``` + +**Step 3 — Pre-apply the jobs step** + +```sql +BEGIN; +CREATE TABLE job ( + id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + name text NOT NULL, + tenant text NOT NULL, + queue text NOT NULL, + label text, + priority int NOT NULL DEFAULT 100, + scheduled_at timestamptz NOT NULL DEFAULT now(), + retry_num int NOT NULL DEFAULT 0, + max_retries int NOT NULL DEFAULT 3, + status text NOT NULL DEFAULT 'new' + CHECK (status IN ('new', 'scheduled', 'running', 'retry', + 'completed', 'failed', 'cancelled', + 'aborted')), + created_at timestamptz NOT NULL DEFAULT now(), + modified_at timestamptz NOT NULL DEFAULT now(), + started_at timestamptz, + completed_at timestamptz, + params jsonb NOT NULL DEFAULT '{}', + + profile_id uuid NULL REFERENCES profile(id) ON DELETE NO ACTION DEFERRABLE, + error jsonb, + result jsonb, + resource_id uuid NULL REFERENCES storage_object(id) ON DELETE SET NULL DEFERRABLE, + expires_at timestamptz +); + +CREATE INDEX job__dispatcher__idx + ON job (tenant, priority DESC, scheduled_at) + WHERE status IN ('new', 'retry'); + +CREATE INDEX job__orphan__idx + ON job (tenant, modified_at) + WHERE status = 'running'; + +CREATE INDEX job__profile__idx + ON job (profile_id, created_at DESC) + WHERE profile_id IS NOT NULL; + +CREATE INDEX job__resource__idx + ON job (resource_id) + WHERE resource_id IS NOT NULL; + +-- Created in step 8 with the `modified_at` definition. + +CREATE INDEX job__expires__idx + ON job (expires_at) + WHERE expires_at IS NOT NULL; + +CREATE INDEX job__retention__idx + ON job (modified_at) + WHERE status IN ('completed', 'failed', 'cancelled', 'aborted') + AND profile_id IS NULL; + +CREATE TABLE job_event ( + id bigserial PRIMARY KEY, + job_id uuid NOT NULL REFERENCES job(id) ON DELETE CASCADE DEFERRABLE, + kind text NOT NULL + CHECK (kind IN ('start', 'progress', 'retry', 'end')), + payload jsonb NOT NULL DEFAULT '{}', + created_at timestamptz NOT NULL DEFAULT now() +); + +CREATE INDEX job_event__job_kind_created_idx + ON job_event (job_id, kind, created_at DESC, id DESC); + +CREATE OR REPLACE TRIGGER deletion_protection__tgr +BEFORE DELETE ON job FOR EACH STATEMENT + WHEN ((current_setting('rules.deletion_protection', true) IN ('on', '')) OR + (current_setting('rules.deletion_protection', true) IS NULL)) + EXECUTE PROCEDURE raise_deletion_protection(); +COMMIT; +``` + +If the transaction fails, rerun the whole block. + +**Step 4 — Audit log: columns, reserved range and defaults** + +```sql +CREATE SEQUENCE IF NOT EXISTS audit_log_seq_seq; +ALTER TABLE audit_log ADD COLUMN IF NOT EXISTS seq bigint; +ALTER TABLE audit_log ADD COLUMN IF NOT EXISTS xid xid8 NOT NULL DEFAULT '0'::xid8; +ALTER SEQUENCE audit_log_seq_seq OWNED BY audit_log.seq; + +-- Reserve the low range for the history: new rows start above the backfill. +-- Only on the first application: a sequence that already left 1 is reserved +-- by the operator or in use, and a re-run must not move it. +SELECT setval('audit_log_seq_seq', + (SELECT count(*) + 1000000 FROM audit_log), + false) + WHERE (SELECT last_value FROM audit_log_seq_seq) = 1 + AND NOT (SELECT is_called FROM audit_log_seq_seq); + +ALTER TABLE audit_log ALTER COLUMN seq SET DEFAULT nextval('audit_log_seq_seq'); +ALTER TABLE audit_log ALTER COLUMN xid SET DEFAULT pg_current_xact_id(); +``` + +**Step 5 — Audit log: backfill** + +Set `work_mem` high enough for the sort to stay in memory; measure the first +run and adjust if needed. + +```sql +UPDATE audit_log + SET seq = ordered.rn + FROM (SELECT id, row_number() OVER (ORDER BY created_at, id) AS rn + FROM audit_log + WHERE seq IS NULL) ordered + WHERE audit_log.id = ordered.id; +``` + +**Step 6 — Audit log: `seq` and `source` NOT NULL without a long +`ACCESS EXCLUSIVE` scan** + +```sql +ALTER TABLE audit_log ADD CONSTRAINT audit_log__seq__not_null + CHECK (seq IS NOT NULL) NOT VALID; +ALTER TABLE audit_log VALIDATE CONSTRAINT audit_log__seq__not_null; +ALTER TABLE audit_log ALTER COLUMN seq SET NOT NULL; +ALTER TABLE audit_log DROP CONSTRAINT audit_log__seq__not_null; + +UPDATE audit_log SET source = 'backend' WHERE source IS NULL; + +ALTER TABLE audit_log ADD CONSTRAINT audit_log__source__not_null + CHECK (source IS NOT NULL) NOT VALID; +ALTER TABLE audit_log VALIDATE CONSTRAINT audit_log__source__not_null; +ALTER TABLE audit_log ALTER COLUMN source SET NOT NULL; +ALTER TABLE audit_log DROP CONSTRAINT audit_log__source__not_null; +``` + +**Step 7 — Audit log: cursor index and cursor table** + +```sql +CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS audit_log__xid__seq__idx + ON audit_log (xid, seq); + +CREATE TABLE IF NOT EXISTS audit_log_consumer_cursor ( + consumer smallint NOT NULL, + xid xid8 NOT NULL, + seq bigint NOT NULL, + updated_at timestamptz NOT NULL DEFAULT now(), + PRIMARY KEY (consumer)); +``` + +**Step 8 — Jobs: exclusive and dedupe columns and indexes** + +Run in autocommit; `CREATE INDEX CONCURRENTLY` cannot run inside a transaction. + +```sql +ALTER TABLE job ADD COLUMN IF NOT EXISTS exclusive boolean NOT NULL DEFAULT false; +ALTER TABLE job ADD COLUMN IF NOT EXISTS dedupe boolean NOT NULL DEFAULT false; + +CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS job__live_exclusive__uniq + ON job (tenant, name, queue, label) + WHERE exclusive AND status IN ('new', 'scheduled', 'running', 'retry'); + +CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS job__new_dedupe__uniq + ON job (tenant, name, queue, label) + WHERE dedupe AND status = 'new'; + +DROP INDEX IF EXISTS job__scheduled__idx; +CREATE INDEX IF NOT EXISTS job__scheduled__idx + ON job (tenant, modified_at) + WHERE status = 'scheduled'; +DROP INDEX IF EXISTS job__name_label__idx; +``` + +**Step 9 — Gate before deploying** + +If an index is `INVALID`, drop and rebuild it before continuing. + +```sql +SELECT indexname, indexdef FROM pg_indexes + WHERE indexname IN ('job__live_exclusive__uniq', + 'job__new_dedupe__uniq', + 'job__scheduled__idx', + 'audit_log__xid__seq__idx'); + +SELECT indexrelid::regclass AS index, indisvalid, indisunique + FROM pg_index + WHERE indexrelid IN ('job__live_exclusive__uniq'::regclass, + 'job__new_dedupe__uniq'::regclass, + 'audit_log__xid__seq__idx'::regclass); + +SELECT column_name, is_nullable + FROM information_schema.columns + WHERE table_name = 'audit_log' + AND column_name IN ('seq', 'xid', 'source') + ORDER BY 1; + +SELECT to_regclass('public.job') IS NOT NULL AS job, + to_regclass('public.job_event') IS NOT NULL AS job_event, + to_regclass('public.audit_log_consumer_cursor') IS NOT NULL AS cursor; + +SELECT indexname FROM pg_indexes + WHERE tablename IN ('job', 'job_event') + AND indexname IN ('job__dispatcher__idx', 'job__orphan__idx', + 'job__profile__idx', 'job__resource__idx', + 'job__expires__idx', 'job__retention__idx', + 'job__scheduled__idx', 'job_event__job_kind_created_idx') + ORDER BY 1; + +SELECT column_name, is_nullable + FROM information_schema.columns + WHERE table_name = 'job' + AND column_name IN ('exclusive', 'dedupe') + ORDER BY 1; + +SELECT tgname FROM pg_trigger + WHERE tgrelid = to_regclass('public.job') + AND tgname = 'deletion_protection__tgr' + AND NOT tgisinternal; +``` + +Pass only if all three tables, eight indexes and the trigger exist; `seq`, +`xid`, `source`, `exclusive` and `dedupe` are `NOT NULL`; and the three unique +indexes are valid and unique. Confirm `job__scheduled__idx` uses +`(tenant, modified_at)`. Do not deploy if any check fails. + +### 1.5 File data storage object foreign key — online steps + +Build the schema of `0159-add-file-data-storage-object-id` online, while +2.19 still serves traffic: the `storage_object_id` column, one index per +partition, the parent index and the foreign key. The backfill is a separate +step in phase 2, inside the coordinated stop. The migration row was +pre-registered in section 1.1, so the boot migration skips this work. + +**Session for this block**: plain SQL; the online ALTERs set their own +`SET LOCAL lock_timeout = '3s'` inside their transactions. + +**Step 1 — Column** + +```sql +BEGIN; +SET LOCAL lock_timeout = '3s'; +ALTER TABLE file_data + ADD COLUMN IF NOT EXISTS storage_object_id uuid NULL; +COMMIT; +``` + +The change is catalog-only (a nullable column without a default), so it takes +`ACCESS EXCLUSIVE` for an instant. If it fails with `lock timeout`, wait for +a quieter moment and retry: it did not block anything. + +**Step 2 — One index per partition, concurrently** + +```sql +CREATE INDEX CONCURRENTLY file_data_00__storage_object_id__idx + ON file_data_00 (storage_object_id); +CREATE INDEX CONCURRENTLY file_data_01__storage_object_id__idx + ON file_data_01 (storage_object_id); +CREATE INDEX CONCURRENTLY file_data_02__storage_object_id__idx + ON file_data_02 (storage_object_id); +CREATE INDEX CONCURRENTLY file_data_03__storage_object_id__idx + ON file_data_03 (storage_object_id); +CREATE INDEX CONCURRENTLY file_data_04__storage_object_id__idx + ON file_data_04 (storage_object_id); +CREATE INDEX CONCURRENTLY file_data_05__storage_object_id__idx + ON file_data_05 (storage_object_id); +CREATE INDEX CONCURRENTLY file_data_06__storage_object_id__idx + ON file_data_06 (storage_object_id); +CREATE INDEX CONCURRENTLY file_data_07__storage_object_id__idx + ON file_data_07 (storage_object_id); +CREATE INDEX CONCURRENTLY file_data_08__storage_object_id__idx + ON file_data_08 (storage_object_id); +CREATE INDEX CONCURRENTLY file_data_09__storage_object_id__idx + ON file_data_09 (storage_object_id); +CREATE INDEX CONCURRENTLY file_data_10__storage_object_id__idx + ON file_data_10 (storage_object_id); +CREATE INDEX CONCURRENTLY file_data_11__storage_object_id__idx + ON file_data_11 (storage_object_id); +CREATE INDEX CONCURRENTLY file_data_12__storage_object_id__idx + ON file_data_12 (storage_object_id); +CREATE INDEX CONCURRENTLY file_data_13__storage_object_id__idx + ON file_data_13 (storage_object_id); +CREATE INDEX CONCURRENTLY file_data_14__storage_object_id__idx + ON file_data_14 (storage_object_id); +CREATE INDEX CONCURRENTLY file_data_15__storage_object_id__idx + ON file_data_15 (storage_object_id); +``` + +Run each statement in autocommit: `CREATE INDEX CONCURRENTLY` cannot run +inside a transaction block. An interrupted build leaves the index +`INVALID`: drop it and retry. + +```sql +SELECT c.relname, i.indisvalid + FROM pg_index AS i + JOIN pg_class AS c ON c.oid = i.indexrelid + WHERE c.relname LIKE 'file_data_%storage_object_id%' + ORDER BY c.relname; +``` + +**Step 3 — Parent index and partition attachment** + +```sql +CREATE INDEX file_data__storage_object_id__idx + ON ONLY file_data (storage_object_id); + +ALTER INDEX file_data__storage_object_id__idx + ATTACH PARTITION file_data_00__storage_object_id__idx; +ALTER INDEX file_data__storage_object_id__idx + ATTACH PARTITION file_data_01__storage_object_id__idx; +ALTER INDEX file_data__storage_object_id__idx + ATTACH PARTITION file_data_02__storage_object_id__idx; +ALTER INDEX file_data__storage_object_id__idx + ATTACH PARTITION file_data_03__storage_object_id__idx; +ALTER INDEX file_data__storage_object_id__idx + ATTACH PARTITION file_data_04__storage_object_id__idx; +ALTER INDEX file_data__storage_object_id__idx + ATTACH PARTITION file_data_05__storage_object_id__idx; +ALTER INDEX file_data__storage_object_id__idx + ATTACH PARTITION file_data_06__storage_object_id__idx; +ALTER INDEX file_data__storage_object_id__idx + ATTACH PARTITION file_data_07__storage_object_id__idx; +ALTER INDEX file_data__storage_object_id__idx + ATTACH PARTITION file_data_08__storage_object_id__idx; +ALTER INDEX file_data__storage_object_id__idx + ATTACH PARTITION file_data_09__storage_object_id__idx; +ALTER INDEX file_data__storage_object_id__idx + ATTACH PARTITION file_data_10__storage_object_id__idx; +ALTER INDEX file_data__storage_object_id__idx + ATTACH PARTITION file_data_11__storage_object_id__idx; +ALTER INDEX file_data__storage_object_id__idx + ATTACH PARTITION file_data_12__storage_object_id__idx; +ALTER INDEX file_data__storage_object_id__idx + ATTACH PARTITION file_data_13__storage_object_id__idx; +ALTER INDEX file_data__storage_object_id__idx + ATTACH PARTITION file_data_14__storage_object_id__idx; +ALTER INDEX file_data__storage_object_id__idx + ATTACH PARTITION file_data_15__storage_object_id__idx; +``` + +The step creates the index unconditionally: if an index with that name +already exists, drop it and rebuild it. Verify the result before continuing: + +```sql +SELECT i.indisvalid, i.indpred IS NULL AS plain, i.indexprs IS NULL AS no_expression + FROM pg_index AS i + WHERE i.indexrelid = 'file_data__storage_object_id__idx'::regclass; +-- t | t | t + +SELECT count(*) FROM pg_inherits + WHERE inhparent = 'file_data__storage_object_id__idx'::regclass; +-- 16 +``` + +**Step 4 — Foreign key, online** + +```sql +BEGIN; +SET LOCAL lock_timeout = '3s'; +SET LOCAL statement_timeout = '10min'; -- measure first; see below +ALTER TABLE file_data + ADD CONSTRAINT file_data_storage_object_id_fkey + FOREIGN KEY (storage_object_id) REFERENCES storage_object(id) + ON DELETE NO ACTION DEFERRABLE INITIALLY IMMEDIATE; +COMMIT; +``` + +The column is NULL until the backfill, but PostgreSQL still scans the table +(or the index built in the previous steps) to prove there are no violations. +The `SHARE ROW EXCLUSIVE` lock blocks writes for the whole scan, not only +while the lock is acquired: `lock_timeout` bounds the wait for the lock and +`statement_timeout` bounds the scan. A timeout aborts the `ALTER`, releases +the writes and lets you retry in a quieter window; measure the scan in +staging and set the budget accordingly. Retry on `lock timeout`. + +Creating it here, before the backfill, keeps the full validation scan out +of the stopped window: the backfill validates each row it writes. + +`INITIALLY IMMEDIATE` is required. The storage GC catches the violation +inside the chunk transaction and retries the chunk; an `INITIALLY DEFERRED` +check fires at commit, outside that transaction, and the GC would never +move past a blocked chunk. + +**Step 5 — Temporary index for pending backfill rows (online)** + +Build this partial index while 2.19 serves traffic, before the stop. It covers +only rows that still need migration. Each successful batch removes its rows +from the index, so later batches do not scan completed rows. Drop the parent +index after the backfill. + +Run each `CREATE INDEX CONCURRENTLY` in autocommit: + +```sql +CREATE INDEX CONCURRENTLY file_data_00__storage_ref_pending__idx + ON file_data_00 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +CREATE INDEX CONCURRENTLY file_data_01__storage_ref_pending__idx + ON file_data_01 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +CREATE INDEX CONCURRENTLY file_data_02__storage_ref_pending__idx + ON file_data_02 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +CREATE INDEX CONCURRENTLY file_data_03__storage_ref_pending__idx + ON file_data_03 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +CREATE INDEX CONCURRENTLY file_data_04__storage_ref_pending__idx + ON file_data_04 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +CREATE INDEX CONCURRENTLY file_data_05__storage_ref_pending__idx + ON file_data_05 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +CREATE INDEX CONCURRENTLY file_data_06__storage_ref_pending__idx + ON file_data_06 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +CREATE INDEX CONCURRENTLY file_data_07__storage_ref_pending__idx + ON file_data_07 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +CREATE INDEX CONCURRENTLY file_data_08__storage_ref_pending__idx + ON file_data_08 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +CREATE INDEX CONCURRENTLY file_data_09__storage_ref_pending__idx + ON file_data_09 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +CREATE INDEX CONCURRENTLY file_data_10__storage_ref_pending__idx + ON file_data_10 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +CREATE INDEX CONCURRENTLY file_data_11__storage_ref_pending__idx + ON file_data_11 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +CREATE INDEX CONCURRENTLY file_data_12__storage_ref_pending__idx + ON file_data_12 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +CREATE INDEX CONCURRENTLY file_data_13__storage_ref_pending__idx + ON file_data_13 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +CREATE INDEX CONCURRENTLY file_data_14__storage_ref_pending__idx + ON file_data_14 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +CREATE INDEX CONCURRENTLY file_data_15__storage_ref_pending__idx + ON file_data_15 (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); +``` + +Create and attach the matching parent index: + +```sql +CREATE INDEX file_data__storage_ref_pending__idx ON ONLY file_data (file_id, id) + WHERE backend = 'storage' + AND storage_object_id IS NULL + AND jsonb_exists(metadata, 'storage-ref-id'); + +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_00__storage_ref_pending__idx; +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_01__storage_ref_pending__idx; +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_02__storage_ref_pending__idx; +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_03__storage_ref_pending__idx; +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_04__storage_ref_pending__idx; +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_05__storage_ref_pending__idx; +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_06__storage_ref_pending__idx; +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_07__storage_ref_pending__idx; +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_08__storage_ref_pending__idx; +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_09__storage_ref_pending__idx; +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_10__storage_ref_pending__idx; +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_11__storage_ref_pending__idx; +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_12__storage_ref_pending__idx; +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_13__storage_ref_pending__idx; +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_14__storage_ref_pending__idx; +ALTER INDEX file_data__storage_ref_pending__idx + ATTACH PARTITION file_data_15__storage_ref_pending__idx; + +SELECT indisvalid, indpred IS NOT NULL AS partial + FROM pg_index + WHERE indexrelid = 'file_data__storage_ref_pending__idx'::regclass; +-- t | t + +SELECT count(*) AS attached_partitions + FROM pg_inherits + WHERE inhparent = 'file_data__storage_ref_pending__idx'::regclass; +-- 16 +``` + +**Step 6 — Verify after the backfill** + +Do not deploy 2.20.0 until every check passes. + +```sql +-- Column. +SELECT data_type, is_nullable + FROM information_schema.columns + WHERE table_schema = 'public' + AND table_name = 'file_data' + AND column_name = 'storage_object_id'; +-- uuid | YES + +-- Rows that will be unreadable. Must be 0 after the backfill. The backfill +-- only fills pointers it can resolve, so a non-zero count means pointers +-- that were already corrupt or orphan: inspect them and accept them before +-- continuing. A row pointing at a storage_object with deleted_at set is +-- legitimate and restorable, not unreadable: the backfill fills it on +-- purpose and the restore flow revives the object, so it is not counted. +SELECT count(*) FROM file_data + WHERE backend = 'storage' AND storage_object_id IS NULL; + +-- Migrated rows no longer retain the old pointer. Must be 0. +SELECT count(*) FROM file_data + WHERE backend = 'storage' AND storage_object_id IS NOT NULL + AND jsonb_exists(metadata, 'storage-ref-id'); + +-- Index: valid, plain, with every partition attached. +SELECT i.indisvalid, i.indpred IS NULL AS plain, i.indexprs IS NULL AS no_expression + FROM pg_index AS i + WHERE i.indexrelid = 'file_data__storage_object_id__idx'::regclass; +SELECT count(*) FROM pg_inherits + WHERE inhparent = 'file_data__storage_object_id__idx'::regclass; -- 16 + +-- Foreign key: deferrable, checked immediately, no action, validated. +SELECT condeferrable, condeferred, confdeltype, convalidated + FROM pg_constraint + WHERE conname = 'file_data_storage_object_id_fkey' + AND conrelid = 'file_data'::regclass; +-- t | f | a | t +``` + +--- + +## Phase 2 — The deploy to 2.20.0 + +### 2.1 Full stop and backfill + +The backfill runs on every instance, even when no row needs it: an instance +with nothing to backfill updates 0 rows. There is no rolling deploy: the +previous code writes the pointer only to `metadata.storage-ref-id` and its +storage GC reads the reference from there. Running the backfill while that +code serves traffic would leave the rows it writes without a pointer and +could make the GC delete an object a live row still references. + +There is no schema rollback after the deploy: the column is the only +reference for the rows 2.20 writes, so a failed upgrade is recovered from the +backup. + +**1. Stop every instance (backend and workers).** Keep +`:tiered-file-data-storage` off until every instance runs 2.20.0. + +**2. Run the backfill:** + +```sql +WITH candidates AS MATERIALIZED ( + SELECT fd.file_id, + fd.id, + CASE WHEN (fd.metadata->>'storage-ref-id') ~* + '^[0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{12}$' + THEN (fd.metadata->>'storage-ref-id')::uuid END AS soid + FROM file_data AS fd + JOIN storage_object AS so + ON so.id = CASE WHEN (fd.metadata->>'storage-ref-id') ~* + '^[0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{12}$' + THEN (fd.metadata->>'storage-ref-id')::uuid END + WHERE fd.backend = 'storage' + AND fd.storage_object_id IS NULL + AND jsonb_exists(fd.metadata, 'storage-ref-id') + AND (fd.metadata->>'storage-ref-id') ~* + '^[0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{12}$' + ORDER BY fd.file_id, fd.id + LIMIT 5000 + ), + updated AS ( + UPDATE file_data AS fd + SET storage_object_id = c.soid, + metadata = fd.metadata - 'storage-ref-id' + FROM candidates AS c + WHERE fd.file_id = c.file_id + AND fd.id = c.id + RETURNING 1 +) +SELECT count(*) AS rows_updated FROM updated; +``` + +Run the statement repeatedly, committing each batch separately, until +`rows_updated` is 0. Each batch updates at most 5,000 rows, sets the column and +removes `metadata.storage-ref-id` atomically. The index built in phase 1.5 +keeps the candidate search bounded to rows still pending. The `MATERIALIZED` +CTE keeps the UUID cast away from values that fail the regex. The join to +`storage_object` excludes orphan pointers, and the foreign key validates each +written row. Corrupt and orphan pointers remain untouched for investigation; +they must be resolved before the unreadable-row verification can pass. + +After all batches finish, remove the temporary index: + +```sql +DROP INDEX file_data__storage_ref_pending__idx; +``` + +The migrated key is not retained as a rollback copy. A rollback to the previous +binary after the backfill requires restoring the backup. + +**3. Verify.** Run the **Verify** block of phase 1.5. All checks must pass. + +**4. Confirm the pre-registered migration row exists:** + +```sql +SELECT module, step, created_at FROM migrations + WHERE module = 'main' AND step = '0159-add-file-data-storage-object-id'; +``` + +The row was inserted at the beginning of the runbook. Do not deploy 2.20.0 +until the backfill and all verification checks have passed. + +**5. Start the 2.20.0 instances.** + +### 2.2 The watermark goes live + +The deploy brings the code that updates `updated_at` on every mutation +(locks, soft deletes). Phase 3 depends on it. Confirm the watermark moved: + +```sql +SELECT max(updated_at) FROM file_change; +``` + +It has to move on every write from now on. If it does not, do not start +phase 3. + +--- + +## Phase 3 — After the upgrade (2.20.0 serving; online) + +Phase 3 copies `file_change` into the new partitioned table in the +background. It spans days: the bulk is one pass over every row (hours to +days), then short delta passes every 1-6 h until the maintenance window. +Everything here runs with the instance serving traffic. + +### 3.1 Create the partitioned table + +Create the partitioned table and its 16 partitions: + +```sql +CREATE TABLE file_change_new ( + file_id uuid NOT NULL, + id uuid NOT NULL DEFAULT gen_random_uuid(), + created_at timestamptz NOT NULL DEFAULT now(), + session_id uuid, + revn bigint NOT NULL DEFAULT 0, + data bytea, + changes bytea, + profile_id uuid, + features text[], + label text, + version integer, + created_by text NOT NULL DEFAULT 'system', + updated_at timestamptz DEFAULT now(), + deleted_at timestamptz, + migrations text[], + locked_by uuid, + -- Nullable on purpose, like the migration: the previous version inserts + -- without `kind`, and a rollback to it has to keep writing. The `NOT NULL` + -- comes in a later migration, once the rollback window is closed. + kind smallint, + CONSTRAINT file_change_kind_check CHECK (kind IN (1, 2)) +) PARTITION BY HASH (file_id); + +-- the columns keep the storage of PostgreSQL by default (migration +-- 0156-reset-pg-storage-defaults dropped the explicit EXTERNAL overrides) + +COMMENT ON COLUMN file_change_new.kind IS + 'The kind of the row: 1 is a snapshot (it has a label and its data lives in file_data) and 2 is a revision (it has no label and it holds the changes).'; +COMMENT ON COLUMN file_change_new.locked_by IS + 'Profile ID of user who has locked this version. Only the creator can lock/unlock their own versions. Locked versions cannot be deleted by others.'; + +-- 16 partitions, the same bounds as file_data. The name of the table is +-- temporary (the old one holds it until the cutover) but the name of the +-- partitions is not: it is the final one, so their indexes are born with +-- the canonical names (renaming a partition does not rename its indexes) +-- and the result is exactly the schema of the migration. +CREATE TABLE file_change_00 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 0); +CREATE TABLE file_change_01 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 1); +CREATE TABLE file_change_02 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 2); +CREATE TABLE file_change_03 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 3); +CREATE TABLE file_change_04 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 4); +CREATE TABLE file_change_05 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 5); +CREATE TABLE file_change_06 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 6); +CREATE TABLE file_change_07 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 7); +CREATE TABLE file_change_08 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 8); +CREATE TABLE file_change_09 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 9); +CREATE TABLE file_change_10 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 10); +CREATE TABLE file_change_11 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 11); +CREATE TABLE file_change_12 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 12); +CREATE TABLE file_change_13 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 13); +CREATE TABLE file_change_14 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 14); +CREATE TABLE file_change_15 PARTITION OF file_change_new + FOR VALUES WITH (MODULUS 16, REMAINDER 15); + +-- The primary key includes the partition key (required on a partitioned +-- table) +ALTER TABLE file_change_new ADD PRIMARY KEY (file_id, id); + +-- Foreign keys (no CREATE TABLE copies them). They take the canonical +-- name already: they have no index, so they do not collide with the old +-- table and the partitions inherit it. +ALTER TABLE file_change_new + ADD CONSTRAINT file_change_file_id_fkey FOREIGN KEY (file_id) + REFERENCES file(id) DEFERRABLE; +ALTER TABLE file_change_new + ADD CONSTRAINT file_change_locked_by_fkey FOREIGN KEY (locked_by) + REFERENCES profile(id) ON DELETE SET NULL DEFERRABLE; +ALTER TABLE file_change_new + ADD CONSTRAINT file_change_profile_id_fkey FOREIGN KEY (profile_id) + REFERENCES profile(id) ON DELETE SET NULL DEFERRABLE; +``` + +```sql +SELECT count(*) AS partitions FROM pg_partition_tree('file_change_new') t + WHERE t.isleaf; +-- 16 +``` + +The new indexes carry a `_new` suffix because index names are unique per +schema and the old table still holds the canonical ones until the cutover. +The same goes for the primary key of the new table: its index is named +after the constraint. The constraints that have no index (the `CHECK` and +the foreign keys) are named canonically from the start, because a +constraint name is unique per table, not per schema, so it can live with +the one of the old table; the partitions inherit it when they are created. +The partitions are created with their final name, because their indexes are +named after the partition and not after the table. + + +### 3.2 Bulk copy + +Copies every row, including the soft deleted ones: the GC manages them the +same way in the new table. `kind` is computed from `label`: a row with a +label is a snapshot, a row without one is a revision. + +The pointer of the copy lives in the database, not in the terminal: a batch +can be interrupted and resumed without losing anything. + +```sql +CREATE TABLE IF NOT EXISTS file_change_copy_state ( + name text PRIMARY KEY, + last_id uuid, + last_at timestamptz +); + +-- for a procedure that already created the table without the mark +ALTER TABLE file_change_copy_state + ADD COLUMN IF NOT EXISTS last_at timestamptz; + +INSERT INTO file_change_copy_state (name, last_id) +VALUES ('bulk', '00000000-0000-0000-0000-000000000000') +ON CONFLICT (name) DO NOTHING; + +-- The mark of the delta passes starts at the bulk: the first pass of phase +-- 3.3 covers the rows changed since here, and `DO NOTHING` keeps a re-run +-- from resetting a mark that the passes already moved. +INSERT INTO file_change_copy_state (name, last_at) VALUES ('delta', now()) +ON CONFLICT (name) DO NOTHING; + +-- The mark is mandatory: a pass without it copies nothing, in silence. The +-- `DROP ... IF EXISTS` keeps this block re-runnable. +ALTER TABLE file_change_copy_state + ADD CONSTRAINT file_change_copy_state_delta_mark + CHECK (name <> 'delta' OR last_at IS NOT NULL); +``` + +Run the create-table block of phase 3.1 first, then this block, then the batch block in a loop until it returns `0`. + +One batch (idempotent). It **always returns one row**: the number of rows +it copied, `0` when there is nothing left to copy. + +```sql +WITH batch AS MATERIALIZED ( + SELECT id FROM file_change + WHERE id > (SELECT last_id FROM file_change_copy_state WHERE name = 'bulk') + ORDER BY id + LIMIT 10000 +), +copied AS ( + INSERT INTO file_change_new (id, file_id, created_at, session_id, revn, data, changes, + profile_id, features, label, version, created_by, + updated_at, deleted_at, migrations, locked_by, kind) + SELECT id, file_id, created_at, session_id, revn, data, changes, + profile_id, features, label, version, created_by, + updated_at, deleted_at, migrations, locked_by, + CASE WHEN label IS NOT NULL THEN 1 ELSE 2 END + FROM file_change + WHERE id IN (SELECT id FROM batch) + ON CONFLICT (file_id, id) DO UPDATE + SET created_at = EXCLUDED.created_at, + session_id = EXCLUDED.session_id, + revn = EXCLUDED.revn, + data = EXCLUDED.data, + changes = EXCLUDED.changes, + profile_id = EXCLUDED.profile_id, + features = EXCLUDED.features, + label = EXCLUDED.label, + version = EXCLUDED.version, + created_by = EXCLUDED.created_by, + updated_at = EXCLUDED.updated_at, + deleted_at = EXCLUDED.deleted_at, + migrations = EXCLUDED.migrations, + locked_by = EXCLUDED.locked_by, + kind = EXCLUDED.kind + WHERE file_change_new.updated_at IS DISTINCT FROM EXCLUDED.updated_at + RETURNING 1 +), +-- `moved` advances the pointer only when the batch copied something. +-- PostgreSQL executes a data-modifying CTE exactly once even when the +-- final query does not read its output. +moved AS ( + UPDATE file_change_copy_state + SET last_id = (SELECT id FROM batch ORDER BY id DESC LIMIT 1) + WHERE name = 'bulk' + AND EXISTS (SELECT 1 FROM copied) + RETURNING 1 +) +SELECT count(*) AS copied FROM copied; +``` + +Repeat this statement until it returns `0`. The `0` comes from the final +`SELECT`, not from a missing row: a batch that finds nothing to copy still +returns it, so it is the signal to stop. + +- The pointer is a `uuid` (random order): the pagination is stable but it + does not group by physical position. The total read is the heap, in a + scattered order. +- **Estimated duration**: time the first batches and extrapolate. The bulk + reads the heap plus the live TOAST once (phase 1) and writes about the + live data. Batches are rows divided by batch size. +- Pace the batches yourself (a pause between statements) if the streaming + replica falls behind or the backend sees latency. Every batch detoastes + its share of the TOAST: the cost is I/O, not CPU. +- Watch: replica lag, `pg_stat_activity` (locks between the bulk and + autovacuum), the space of `pg_wal` and of the data dir. + +When the bulk is done, check that no row is missing (it is the check that +matters, not the total count): + +```sql +SELECT count(*) AS missing + FROM file_change o + WHERE NOT EXISTS (SELECT 1 FROM file_change_new n + WHERE n.file_id = o.file_id + AND n.id = o.id); +``` + +`missing` is zero when no row of the old table is absent from the new one; +it is the check that matters. The new table can have **more** rows than the +old one: the GC may have hard deleted a row from the source after the bulk +copied it, and it cleans that row up after the cutover. Rows written during +the bulk are missing until the delta passes pick them up. + + +### 3.3 Delta passes + +Every pass copies the rows modified since the **previous pass**. The +watermark lives in `file_change_copy_state.last_at` (the start of the last +pass; the setup block of phase 3.2 seeds it with the moment of the bulk, so +the first pass covers what changed since then), and the SQL subtracts one +hour of margin, which absorbs the clock drift between the application +(which writes `updated_at`) and the database (which writes `now()`) and the +drift between batches. A row that a pass already copied but did not change +again is not rewritten: the `WHERE` of the `ON CONFLICT` skips it. + +```sql +BEGIN; + +WITH mark AS ( + SELECT last_at - INTERVAL '1 hour' AS since + FROM file_change_copy_state + WHERE name = 'delta' +) +INSERT INTO file_change_new (id, file_id, created_at, session_id, revn, data, changes, + profile_id, features, label, version, created_by, + updated_at, deleted_at, migrations, locked_by, kind) +SELECT id, file_id, created_at, session_id, revn, data, changes, + profile_id, features, label, version, created_by, + updated_at, deleted_at, migrations, locked_by, + CASE WHEN label IS NOT NULL THEN 1 ELSE 2 END + FROM file_change, mark + WHERE updated_at > mark.since +ON CONFLICT (file_id, id) DO UPDATE + SET created_at = EXCLUDED.created_at, + session_id = EXCLUDED.session_id, + revn = EXCLUDED.revn, + data = EXCLUDED.data, + changes = EXCLUDED.changes, + profile_id = EXCLUDED.profile_id, + features = EXCLUDED.features, + label = EXCLUDED.label, + version = EXCLUDED.version, + created_by = EXCLUDED.created_by, + updated_at = EXCLUDED.updated_at, + deleted_at = EXCLUDED.deleted_at, + migrations = EXCLUDED.migrations, + locked_by = EXCLUDED.locked_by, + kind = EXCLUDED.kind + WHERE file_change_new.updated_at IS DISTINCT FROM EXCLUDED.updated_at; + +-- The mark is the start of this transaction (`now()`): the rows changed +-- after it are the window of the next pass. Both statements go in the +-- same transaction on purpose; a mark written later would leave a gap. +UPDATE file_change_copy_state SET last_at = now() WHERE name = 'delta'; + +COMMIT; +``` + +- Repeat the transaction every 1-6 h (depending on the activity) until the + pass copies very few rows. With the `updated_at` index of phase 1.3 the + pass is an index range over the window, so it takes seconds on a quiet + instance; without it, it scans the heap. To see how many rows are left + in the window: + +```sql +SELECT count(*) FROM file_change + WHERE updated_at > (SELECT last_at FROM file_change_copy_state WHERE name = 'delta'); +``` + +- The objects GC can keep running. Its soft deletes are covered by the + watermark, and its hard deletes leave a row with `deleted_at` in the new + table that the GC itself cleans up later. The hard delete of a **file** + is blocked while the old table (or the new one) has rows for it, because + the foreign key on `file_id` is `NO ACTION`. The GC deletes the pending + files in chunks of 100, and one blocked file aborts the transaction of + the whole chunk: the chunk is logged as `error on chunk deletion` and + skipped, its files wait in the trash, and the next daily run retries the + same chunk and fails again until the cutover. The other chunks keep + making progress. **Do not run the cutover at the same time as a big + batch of the GC.** + +--- + +## Phase 4 — The maintenance window (after the deploy) + +### 4.1 Before the lock (online, still 2.20.0 serving) + +- One last delta pass (the block of phase 3.3). +- Build the secondary indexes of the new table. The bulk and the delta + passes only need the primary key (`ON CONFLICT (file_id, id)`), so + building them here keeps five indexes and their WAL out of every batch. + The `deleted_at` index carries `file_id` (the primary key is + `(file_id, id)`): the objects GC needs the `file_id` to lock by pair, and + with the index it reads the rows it has to delete without touching the + table. + +```sql +-- Secondary indexes: the canonical names, with the _new suffix. There is no +-- index for the system snapshots: the old one filtered them by +-- `data IS NOT NULL`, and with the payload living in `file_data` the column +-- no longer tells snapshots apart. The new one could only filter them by +-- `deleted_at IS NULL`, and the system snapshots always carry a deletion +-- date in the future, so it would never match a row. +CREATE INDEX file_change_new__deleted_at__idx + ON file_change_new (deleted_at, file_id, id) WHERE deleted_at IS NOT NULL; +CREATE INDEX file_change_new__file_id__revn__idx + ON file_change_new (file_id, revn); +CREATE INDEX file_change_new__locked_by__idx + ON file_change_new (locked_by) WHERE locked_by IS NOT NULL; +CREATE INDEX file_change_new__profile_id__idx + ON file_change_new (profile_id) WHERE profile_id IS NOT NULL; +CREATE INDEX file_change_new__user_snapshots__idx + ON file_change_new (file_id) + WHERE label IS NOT NULL AND created_by = 'user' AND deleted_at IS NULL; +``` + +- Prepare and rehearse the cutover block in staging. +- Pick a low activity window: a freeze of the writes to `file_change` of a + couple of minutes (the final delta inside the lock copies only the rows + changed since the last delta pass, so it is quick; the saves of files + during the freeze fail and the user retries, and the rest of the + database keeps working). The window has to cover the whole cutover, with + the backend and the workers stopped and started again (see 4.2). + +### 4.2 The cutover (one transaction, a brief exclusive lock) + +- **Stop the backend and the workers before the transaction, and start + them again after it.** The pool keeps server-side prepared statements, + and a prepared statement holds the OID of the table it was prepared + with: renaming the table does not change that OID. A connection that had + already run `insert into file_change` five times (the default threshold + of the driver) would keep writing to `file_change_legacy` after the + cutover, and the pool recycles its connections every 30 minutes, so the + window of silent writes is long. +- Run the transaction, the `ANALYZE`, and the statement that records the + instant of the cutover (all of it in the block below). +- Start the backend and the workers again. + +```sql +BEGIN; + +LOCK TABLE file_change IN ACCESS EXCLUSIVE MODE; + +-- The final delta: the rows modified since the last pass (its mark, minus +-- the hour of margin). The mark is the one the last pass left. +WITH mark AS ( + SELECT last_at - INTERVAL '1 hour' AS since + FROM file_change_copy_state + WHERE name = 'delta' +) +INSERT INTO file_change_new (id, file_id, created_at, session_id, revn, data, changes, + profile_id, features, label, version, created_by, + updated_at, deleted_at, migrations, locked_by, kind) +SELECT id, file_id, created_at, session_id, revn, data, changes, + profile_id, features, label, version, created_by, + updated_at, deleted_at, migrations, locked_by, + CASE WHEN label IS NOT NULL THEN 1 ELSE 2 END + FROM file_change, mark + WHERE updated_at > mark.since +ON CONFLICT (file_id, id) DO UPDATE + SET created_at = EXCLUDED.created_at, + session_id = EXCLUDED.session_id, + revn = EXCLUDED.revn, + data = EXCLUDED.data, + changes = EXCLUDED.changes, + profile_id = EXCLUDED.profile_id, + features = EXCLUDED.features, + label = EXCLUDED.label, + version = EXCLUDED.version, + created_by = EXCLUDED.created_by, + updated_at = EXCLUDED.updated_at, + deleted_at = EXCLUDED.deleted_at, + migrations = EXCLUDED.migrations, + locked_by = EXCLUDED.locked_by, + kind = EXCLUDED.kind + WHERE file_change_new.updated_at IS DISTINCT FROM EXCLUDED.updated_at; + +-- Check inside the lock, and abort if it fails: no row of the old table +-- may be missing in the new one. The new table can carry extra rows that +-- the GC hard deleted from the source; it cleans them up after the +-- cutover. +DO $$ +DECLARE missing bigint; +BEGIN + SELECT count(*) INTO missing + FROM file_change o + WHERE NOT EXISTS (SELECT 1 FROM file_change_new n + WHERE n.file_id = o.file_id + AND n.id = o.id); + + IF missing > 0 THEN + RAISE EXCEPTION 'file_change: % rows of the old table are missing in file_change_new; cutover aborted', missing; + END IF; +END $$; + +-- The old table frees the canonical names (the same names the migration +-- 0158 uses, with the _legacy suffix) +ALTER TABLE file_change RENAME CONSTRAINT file_change_pkey + TO file_change_pkey_legacy; +ALTER TABLE file_change RENAME CONSTRAINT file_change_file_id_fkey + TO file_change_file_id_fkey_legacy; +ALTER TABLE file_change RENAME CONSTRAINT file_change_locked_by_fkey + TO file_change_locked_by_fkey_legacy; +ALTER TABLE file_change RENAME CONSTRAINT file_change_profile_id_fkey + TO file_change_profile_id_fkey_legacy; + +ALTER INDEX file_change__data_ref_id__idx + RENAME TO file_change__data_ref_id__idx_legacy; +ALTER INDEX file_change__deleted_at__idx + RENAME TO file_change__deleted_at__idx_legacy; +ALTER INDEX file_change__file_id__revn__idx + RENAME TO file_change__file_id__revn__idx_legacy; +ALTER INDEX file_change__locked_by__idx + RENAME TO file_change__locked_by__idx_legacy; +ALTER INDEX file_change__profile_id__idx + RENAME TO file_change__profile_id__idx_legacy; +ALTER INDEX file_change__system_snapshots__idx + RENAME TO file_change__system_snapshots__idx_legacy; +ALTER INDEX IF EXISTS file_change__user_snapshots__idx + RENAME TO file_change__user_snapshots__idx_legacy; +ALTER INDEX IF EXISTS file_change__updated_at__idx + RENAME TO file_change__updated_at__idx_legacy; + +-- And the new table takes them. Only the primary key needs the rename: the +-- `CHECK` and the foreign keys already carry the canonical name (they have +-- no index, so they could take it from the start), and the partitions +-- inherited it. +ALTER TABLE file_change_new RENAME CONSTRAINT file_change_new_pkey + TO file_change_pkey; + +ALTER INDEX file_change_new__deleted_at__idx + RENAME TO file_change__deleted_at__idx; +ALTER INDEX file_change_new__file_id__revn__idx + RENAME TO file_change__file_id__revn__idx; +ALTER INDEX file_change_new__locked_by__idx + RENAME TO file_change__locked_by__idx; +ALTER INDEX file_change_new__profile_id__idx + RENAME TO file_change__profile_id__idx; +ALTER INDEX file_change_new__user_snapshots__idx + RENAME TO file_change__user_snapshots__idx; + +-- And the tables are swapped. The partitions and their indexes keep the +-- names they were born with (they are named after the partition, not after +-- the table), so there is nothing else to rename. +ALTER TABLE file_change RENAME TO file_change_legacy; +ALTER TABLE file_change_new RENAME TO file_change; + +-- The foreign key of the legacy table goes away: it is `NO ACTION`, so +-- while the legacy table has rows for a file, the objects GC cannot delete +-- that file (it deletes the rows of the file first, and the legacy ones are +-- not its business). The legacy table is frozen: nothing writes to it again. +ALTER TABLE file_change_legacy + DROP CONSTRAINT file_change_file_id_fkey_legacy; + +COMMIT; + +ANALYZE file_change; + +-- The instant of the cutover, for the checks of phase 5. It goes after +-- the `COMMIT`: inside the transaction `now()` is the start of it (the +-- moment of the `LOCK`), and the rows the final delta copies could pass +-- for writes happened after the cutover. +INSERT INTO file_change_copy_state (name, last_at) VALUES ('cutover', now()) +ON CONFLICT (name) DO UPDATE SET last_at = excluded.last_at; +``` + +- `file_change_legacy` keeps its data and its indexes (with the `_legacy` + suffix): a frozen copy of the previous layout. Its foreign key on + `file_id` is dropped, so the objects GC can delete the files again. +- The writes to `file_change` during the lock fail: they are + seconds-minutes, the affected saves are retried. +- The objects GC and the four RPC commands keep working: same table name + and same index names after the rename. +- Once the backend is up again, check that the saves go to the partitioned + table and that nothing went to the table that is going away: + +```sql +-- the rows written after the cutover: this one grows with every save +SELECT count(*) FROM file_change + WHERE updated_at > (SELECT last_at FROM file_change_copy_state + WHERE name = 'cutover'); + +-- the rows that went to the legacy table: this one has to stay at 0 +SELECT count(*) FROM file_change_legacy + WHERE updated_at > (SELECT last_at FROM file_change_copy_state + WHERE name = 'cutover'); +``` + +A number above zero in the second query means that some connection was +still writing with a prepared statement of the old table: those rows are +lost when the legacy table is dropped, so they have to be copied over +before that. + +### 4.3 The storage backfill is not part of this window + +The backfill always runs in the phase 2 stop, before the new instances start. +After the deploy, any row that still carries the pointer is unreadable by +the new code, so this is not a step to defer to the phase 4 window. + +--- + +## Phase 5 — After the cutover (online) + +### 5.1 Partition pruning and access paths + +Check that the access paths prune to one partition (when the planner +chooses an index scan): + +```sql +EXPLAIN SELECT * FROM file_change + WHERE file_id = '00000000-0000-0000-0000-000000000000' + AND id = '00000000-0000-0000-0000-000000000000'; +``` + +It has to show a single partition (`file_change_NN`), not an `Append` of +16. + +### 5.2 Schema parity with the migration, and ownership + +Every row of this query has to show `got = expected`: + +```sql +SELECT 'columns' AS what, count(*) AS got, 17 AS expected + FROM pg_attribute + WHERE attrelid = 'file_change'::regclass AND attnum > 0 AND NOT attisdropped +UNION ALL +SELECT 'partitions', count(*), 16 + FROM pg_partition_tree('file_change') t WHERE t.isleaf +UNION ALL +SELECT 'primary key', count(*), 1 + FROM pg_constraint + WHERE conrelid = 'file_change'::regclass AND contype = 'p' +UNION ALL +SELECT 'foreign keys', count(*), 3 + FROM pg_constraint + WHERE conrelid = 'file_change'::regclass AND contype = 'f' +UNION ALL +SELECT 'check constraints', count(*), 1 + FROM pg_constraint + WHERE conrelid = 'file_change'::regclass AND contype = 'c' +UNION ALL +SELECT 'indexes', count(*), 6 + FROM pg_indexes + WHERE tablename = 'file_change' AND schemaname = current_schema() +UNION ALL +SELECT 'partitions out of name', count(*), 0 + FROM pg_class + WHERE relkind = 'r' AND relname ~ '^file_change_new[0-9]' +UNION ALL +SELECT 'child indexes out of name', count(*), 0 + FROM pg_class c + JOIN pg_index i ON i.indexrelid = c.oid + JOIN pg_class p ON p.oid = i.indrelid + WHERE p.relname ~ '^file_change_[0-9]{2}$' + AND c.relname NOT LIKE p.relname || '_%' +UNION ALL +SELECT 'partition constraints out of name', count(*), 0 + FROM pg_constraint c + JOIN pg_class t ON t.oid = c.conrelid + WHERE t.relname ~ '^file_change_[0-9]{2}$' + AND c.contype <> 'n' + AND c.conname LIKE 'file_change_new%'; +``` + +The `NOT NULL` constraints are excluded from the last check (`contype <> +'n'`): their names are not stable across server versions, so they are not +part of the parity. The `CHECK`, the primary key and the foreign keys are. + +And the names, to compare them with the migration: + +```sql +SELECT a.attname AS column + FROM pg_attribute a + WHERE a.attrelid = 'file_change'::regclass AND a.attnum > 0 + AND NOT a.attisdropped + ORDER BY a.attnum; + +SELECT indexname FROM pg_indexes + WHERE tablename = 'file_change' AND schemaname = current_schema() + ORDER BY indexname; + +-- the partitions, with their name +SELECT c.relname AS partition + FROM pg_class c + JOIN pg_partition_tree('file_change') t ON t.relid = c.oid + WHERE t.isleaf + ORDER BY c.relname; + +-- the indexes of the partitions: their name starts with the partition +SELECT c.relname AS child_index + FROM pg_class c + JOIN pg_index i ON i.indexrelid = c.oid + JOIN pg_class p ON p.oid = i.indrelid + WHERE p.relname ~ '^file_change_[0-9]{2}$' + ORDER BY c.relname; + +-- every table, partition and index of the family shares the owner of the +-- table +SELECT count(*) AS out_of_owner + FROM pg_class c + WHERE c.relname LIKE 'file_change%' + AND c.relkind IN ('r', 'p', 'i') + AND pg_get_userbyid(c.relowner) <> + (SELECT tableowner FROM pg_tables + WHERE schemaname = current_schema() AND tablename = 'file_change'); +``` + +The owner check has to return `0`. If it does not, fix the ownership +before opening the service (re-own the table, the 16 partitions and +their indexes) and re-apply any grants the old table had. + +### 5.3 Partition-wise join and autovacuum + +- The partition-wise join is already on: the pool of the backend sets + `enable_partitionwise_join = on` on every connection (it is in the code, + there is nothing to change in the database role). +- Autovacuum: each of the 16 partitions has its own TOAST (check their + sizes with `pg_total_relation_size` per partition); with + `autovacuum_vacuum_scale_factor = 0.2` the threshold is reached earlier + and in steps. Nothing to change, just watch `last_autovacuum` of the + partitions the first week. +- The aggregations that do not filter by `file_id` (the team snapshot + quota and the telemetry counts) read all 16 partitions. To know their + cost on your instance, time them before and after with + `EXPLAIN (ANALYZE, BUFFERS)` and compare. The team quota is already + fixed in the code: it resolves the `file_id` of the team first, which + lets the planner prune the partitions when it chooses an index scan + (with a sequential scan it reads all of them). + +### 5.4 Drop the helper table and, days later, the old table + +```sql +DROP TABLE IF EXISTS file_change_copy_state; +DROP TABLE file_change_legacy; -- out of hours: it generates WAL by + -- dropping the pages, let it roll +``` + +That is where the dead space comes back: the database shrinks by about the +size of the old table (phase 1). Copy the rows from the legacy table +first if phase 4 showed any (see the check after the cutover); they are +lost when it is dropped. + +### 5.5 Post-deploy storage checks + +```sql +SELECT conname, condeferrable, condeferred, confdeltype, convalidated + FROM pg_constraint + WHERE conname = 'file_data_storage_object_id_fkey' + AND conrelid = 'file_data'::regclass; + +SELECT count(*) FROM file_data + WHERE backend = 'storage' AND storage_object_id IS NULL; + +-- Run the foreign-key check; ROLLBACK keeps the object. +BEGIN; +EXPLAIN (ANALYZE, BUFFERS) + DELETE FROM storage_object WHERE id = ''; +ROLLBACK; +``` + +### 5.6 After every instance runs 2.20.0: turn on the new GC + +Once every instance runs this version, the new consumers have replayed +the unarchived buffer. Turn on `:audit-log-gc` (rolling restart of the +new workers) so the new GC deletes the rows the consumers have passed. +Remove `:audit-log-archive`; the new version does not use it. + +--- + +## Rollback + +- **Before the phase 2 deploy**: phase 1 may have applied out-of-band schema + changes, but the new code has not started. Keep 2.19 serving; assess and + reverse the completed phase 1 changes before removing the pre-registration + rows or switching to the automatic migration path. +- **Partition cutover**: `file_change_legacy` is a frozen copy of the + previous layout, kept for days. Rolling back before dropping it means + pointing the code at the legacy layout, which is not supported by the + release; treat the `DROP TABLE file_change_legacy` as the point of no + return and schedule it deliberately. +- **Consumer generalization**: the migration keeps the legacy + `archived_at` mark and its indexes for one release; the two versions can + coexist during the rollout. Rolling the binaries back is possible while + `:audit-log-gc` is off and the consumers are idle; keep the flag off + until you are done, and call it a rollback only within the rollout. + +## Risks + +| Risk | Mitigation | +|---|---| +| Massive WAL during the bulk copy (proportional to the copy) | Throttle between batches; watch `pg_wal`, the replication slots and the replica lag | +| Latency seen by the application during the bulk | Throttle; low traffic hours; the bulk is I/O and does not block writes | +| `updated_at` not reliable in rows changed before the deploy | The deploy goes before the bulk (phase 2.2); the delta passes cover from the start of the bulk | +| Lock/unlock/soft delete during the passes | Covered by the watermark; the cutover includes the final delta inside the lock | +| Hard delete of the GC between passes | The row is left with `deleted_at` in the new table; the GC cleans it up later | +| Orphan rows if a `file` is deleted halfway | The foreign key on `file_id` is `NO ACTION`: the GC deletes the rows of the file before the file, so a chunk that fails leaves the file in the trash and the next run deletes it | +| The files in the trash cannot be deleted while the legacy table has rows for them | The foreign key of the legacy table is dropped at the cutover; until then the files wait in the trash | +| Saves failing during the lock of the cutover | Freeze of minutes in a low traffic window; manual retries | +| Planner regression after the cutover | Immediate `ANALYZE` + the pruning `EXPLAIN`; the pool already sets the GUC | +| Clock drift between the application and the database | The delta window starts an hour before the bulk; the margin absorbs the drift | +| `CONCURRENTLY` build interrupted, index left `INVALID` | Drop it and rebuild it; `IF NOT EXISTS` matches by name and would accept the broken index: the gates (phase 1.4 step 9 and phase 1.5 verify) are mandatory | +| Running the backfill while 2.19 serves | Do not: phase 2 stops every instance before the backfill | + +## Success metrics + +- After the cutover: `file_change` (partitioned) holds at most the + `pg_table_size` of the old table (phase 1). +- After the `DROP` of the old table: the database shrinks by about the + size that table had. +- The access paths that filter by `file_id` (the lookup of a snapshot, + the quota of a file and the lock of the GC by pair) prune to one + partition when the planner chooses an index scan (`EXPLAIN`). The quota + of the team and the discovery of the GC do not filter by a single file: + they read all the 16 partitions. +- No visible change for the user. diff --git a/operations/README.md b/operations/README.md index 9590e8e865..0c3621468b 100644 --- a/operations/README.md +++ b/operations/README.md @@ -14,3 +14,4 @@ not grow with procedures that nobody will run again. |---|---| | [Migrate legacy file payloads to `file_data`](2.19.0-migrate-legacy-file-data.md) | 2.19.0 | | [Partition `file_change` by `file_id`](2.20.0-file-change-partition.md) | 2.20.0 | +| [Unified 2.20.0 upgrade](2.20.0-unified.md) | 2.20.0 |