From be2e117f670fdc3fe440e55bea541df28f1c76d9 Mon Sep 17 00:00:00 2001 From: Andrey Antukh Date: Mon, 19 Oct 2026 23:25:44 +0200 Subject: [PATCH] :recycle: Rename audit-log archivers to consumers and use base Nexus URI (#12459) * :recycle: Rename audit-log archivers to consumers The pending-queue owners are no longer only external archives: a local, non-archiving drain is about to join them, so "archiver" no longer fits. Rename the column, identifiers, jobs, metrics and docs to "consumer" in one release; migration 0157 never reached production, so a single RENAME COLUMN is enough, with no backfill. AI-assisted-by: muse-spark-1.3-contributor * :recycle: Use base Nexus URI with appended ingest path Mirror the admin-console setup: PENPOT_NEXUS_URI holds the base URI and the consumer appends api/audit itself, instead of a full endpoint URI. Keeps the explicit :uri job param as override and the :task-not-configured guard when no base is set. AI-assisted-by: muse-spark-1.3-contributor --- .serena/memories/audit-log.md | 26 +-- .serena/memories/prod-infra/core.md | 2 +- backend/scripts/_env | 2 +- backend/src/app/config.clj | 2 +- backend/src/app/loggers/audit.clj | 8 +- .../{archive_task.clj => consumer_task.clj} | 217 +++++++++--------- backend/src/app/main.clj | 26 +-- .../sql/0157-add-audit-log-pending-table.sql | 18 +- backend/src/app/rpc/commands/audit.clj | 6 +- backend/src/app/tasks/telemetry.clj | 6 +- ...j => loggers_audit_consumer_task_test.clj} | 198 ++++++++-------- backend/test/backend_tests/rpc_audit_test.clj | 4 +- .../backend_tests/tasks_telemetry_test.clj | 12 +- 13 files changed, 276 insertions(+), 251 deletions(-) rename backend/src/app/loggers/audit/{archive_task.clj => consumer_task.clj} (77%) rename backend/test/backend_tests/{loggers_audit_archive_task_test.clj => loggers_audit_consumer_task_test.clj} (82%) diff --git a/.serena/memories/audit-log.md b/.serena/memories/audit-log.md index bc33a4a190..68d4bcc021 100644 --- a/.serena/memories/audit-log.md +++ b/.serena/memories/audit-log.md @@ -45,7 +45,7 @@ Rule of thumb: if you need to know for sure "did this user do X?", look at `sour ## 3. The `audit_log` table: a buffer, not a store -The Penpot `audit_log` table **is not the final destination**. It is a waiting tray: events sit here until every active archiver (Nexus, admin console) confirms them, and whoever finishes last deletes the row. +The Penpot `audit_log` table **is not the final destination**. It is a waiting tray: events sit here until every active consumer (Nexus, admin console) confirms them, and whoever finishes last deletes the row. Main columns (migration `backend/src/app/migrations/sql/0054-add-audit-log-table.sql` and later ones): @@ -57,16 +57,16 @@ Main columns (migration `backend/src/app/migrations/sql/0054-add-audit-log-table - `source` text: `backend` / `frontend` for full rows, `telemetry:backend` / `telemetry:frontend` for anonymized copies (see section 10). - `ip_addr` inet: real IP (or `0.0.0.0` on telemetry rows). - `props` / `context` jsonb holding transit-encoded maps: action data and metadata (versions, browser, request-id...). -- No `archived_at`: the `audit_log_pending` queue (migration `0157`, archiver `1` = Nexus, `2` = admin console, ids never reused) tracks what is left. Every record is born pending only for the archivers enabled at insert time (never a fixed both): cheaper on single-archiver hosts and true to the design doc. Consequence, accepted on purpose: rows born with no send flag on get no pendings, so enabling archiving later does not pick them up — the sweep deletes them unarchived. +- No `archived_at`: the `audit_log_pending` queue (migration `0157`, consumer `1` = Nexus, `2` = admin console, ids never reused) tracks what is left. Every record is born pending only for the consumers enabled at insert time (never a fixed both): cheaper on single-consumer hosts and true to the design doc. Consequence, accepted on purpose: rows born with no send flag on get no pendings, so enabling archiving later does not pick them up — the sweep deletes them unarchived. Indexes (each consumer has its own, so a slow consumer never blocks the others): -- `audit_log_pending` PK `(archiver, audit_log_id)`: each archiver owns its queue at constant cost, even with millions pending. -- `audit_log (created_at, id)` (migration `0157`): the chronological drain order (`ORDER BY created_at ASC, id ASC`; same total order for both archivers, so concurrent drains cannot deadlock). +- `audit_log_pending` PK `(consumer, audit_log_id)`: each consumer owns its queue at constant cost, even with millions pending. +- `audit_log (created_at, id)` (migration `0157`): the chronological drain order (`ORDER BY created_at ASC, id ASC`; same total order for both consumers, so concurrent drains cannot deadlock). - `audit_log_pending (audit_log_id)`: serves the "is anything still pending?" check of the final delete. - `(source, created_at)`: used by the telemetry scan. -Life of a row: `INSERT` (row plus one pending per enabled archiver, same transaction; a rollback takes both) → (every 5 min, Nexus at second 0 and admin console at second 30) each archive job sends its own 128-row batches oldest-first and deletes its pendings in the same transaction → whoever deletes the last pending deletes the row. A batch the destination refuses rolls back and stays for the next tick. Pendings of archivers that are off are reaped at the start of each job run (covers later deactivations), and orphan rows are swept at the end of each run, so there is no separate GC task. The FK is plain `DEFERRABLE` (never `INITIALLY DEFERRED`): every deleter removes pendings before rows, including the telemetry task's own deletes. +Life of a row: `INSERT` (row plus one pending per enabled consumer, same transaction; a rollback takes both) → (every 5 min, Nexus at second 0 and admin console at second 30) each archive job sends its own 128-row batches oldest-first and deletes its pendings in the same transaction → whoever deletes the last pending deletes the row. A batch the destination refuses rolls back and stays for the next tick. Pendings of consumers that are off are reaped at the start of each job run (covers later deactivations), and orphan rows are swept at the end of each run, so there is no separate GC task. The FK is plain `DEFERRABLE` (never `INITIALLY DEFERRED`): every deleter removes pendings before rows, including the telemetry task's own deletes. ## 4. Anatomy of an event @@ -277,7 +277,7 @@ Nexus dispatches on `:type`: `"telemetry-events"` batches are coerced and insert ### How it works today -Every 5 min each archive job drains its own queue (`:audit-log-archive-nexus` on flag `:nexus`, cron `0 */5 * * * ?`; `:audit-log-archive-admin-console` on flag `:admin-console`, cron `30 */5 * * * ?`, 30s offset so the two drains never start on the same second), in 128-row chunks taken oldest-first (`ORDER BY created_at ASC, id ASC`) with `FOR UPDATE OF audit_log` (`SET LOCAL lock_timeout = '30s'`) inside the same transaction that sends and completes them, with 100 ms pauses between chunks. Batches arrive at Nexus in chronological order (it numbers by arrival for order-sensitive consumers). Nexus receives transit `{:events [...]}` with `x-shared-key: "nexus "` (`:nexus-shared-key`, or derived from the instance secret) and `origin` = public-uri; flag on but no URI raises `:task-not-configured`. The admin console receives only its allowlisted event names (enriched with team names and emails); anything filtered out counts as done for the admin-console queue. A chunk whose destination refuses it (Nexus non-204 raises `:archive-chunk-refused`) or throws (transport failure, admin-console error) rolls back and stays for the next tick, and the job aborts as failed: it archives what it can, and the cron retries the rest. Committed chunks stay sent. The send fns return counts or throw carrying the attempted count as `:events`; `run-single-archiver` records the `failed` metric from it, outside every chunk transaction (a local DB failure carries no `:events` and stays uncounted — it is not a destination failure). No `warn` before the throw: the runner reports the aborted job. Each job touches only its own queue, so a fallen Nexus never starves the admin console (nor vice versa). +Every 5 min each archive job drains its own queue (`:audit-log-consumer-nexus` on flag `:nexus`, cron `0 */5 * * * ?`; `:audit-log-consumer-admin-console` on flag `:admin-console`, cron `30 */5 * * * ?`, 30s offset so the two drains never start on the same second), in 128-row chunks taken oldest-first (`ORDER BY created_at ASC, id ASC`) with `FOR UPDATE OF audit_log` (`SET LOCAL lock_timeout = '30s'`) inside the same transaction that sends and completes them, with 100 ms pauses between chunks. Batches arrive at Nexus in chronological order (it numbers by arrival for order-sensitive consumers). Nexus receives transit `{:events [...]}` with `x-shared-key: "nexus "` (`:nexus-shared-key`, or derived from the instance secret) and `origin` = public-uri; flag on but no URI raises `:task-not-configured`. The admin console receives only its allowlisted event names (enriched with team names and emails); anything filtered out counts as done for the admin-console queue. A chunk whose destination refuses it (Nexus non-204 raises `:consumer-chunk-refused`) or throws (transport failure, admin-console error) rolls back and stays for the next tick, and the job aborts as failed: it archives what it can, and the cron retries the rest. Committed chunks stay sent. The send fns return counts or throw carrying the attempted count as `:events`; `run-single-consumer` records the `failed` metric from it, outside every chunk transaction (a local DB failure carries no `:events` and stays uncounted — it is not a destination failure). No `warn` before the throw: the runner reports the aborted job. Each job touches only its own queue, so a fallen Nexus never starves the admin console (nor vice versa). A chunk is **one POST** and the task advances only on 204, so a single oversized event wedges the whole queue: a crash report whose message embeds a whole payload (a serialized file, for example) can push the request past the proxy body limit and the rejection blocks every later event until it is pruned by hand. Frontend crash reports are therefore size-bounded at the source (`mem:frontend/routing-app-shell-subtleties`): `app.common.exceptions/get-hint` and the CLJS `format-throwable` clamp exception messages (`max-message-length`). @@ -285,7 +285,7 @@ On the other side, Nexus checks `origin` against its allowlist (403 if not allow Concurrent jobs are safe: both lock rows in `created_at ASC, id ASC` order (one total order, deliberately blocking — no `SKIP LOCKED`, so archive arrival stays chronological), so the second waits for the first to commit and then sees whether it was last. Only acknowledged work deletes pendings; an admin-console failure never blocks the Nexus send of the same rows (each job owns its queue). -Observability (`backend/src/app/loggers/audit/archive_task.clj`, definitions in `backend/src/app/main.clj:default-metrics`): one `dbg` summary per job run (`:archiver` label, sent/skipped/chunks, reaped disabled pendings, swept orphans, `:elapsed` as `"1.5s"` via `ct/format-duration`, frozen in a strict binding first because log props evaluate lazily in the logger thread), one `dbg` per admin-console send call (`ready` with totals, `sent`/`failed` with counts), plus `dbg` when the queue is empty or the job's send flag is off. Queue depth is Grafana's business: it queries the database directly, so the run performs no `GROUP BY` for the logs. No `warn` before a throw: the runner reports the aborted job (refusal details travel in the error data). Metrics, all recorded with bounded labels only: `penpot_audit_archive_events_total{ destination = nexus|admin-console, outcome = archived|failed|skipped }` (`skipped` = filtered by the admin-console allowlist) and `penpot_audit_archive_discarded_total{ reason = reaped|swept }` (entries discarded without archiving: `reaped` counts pendings dropped for disabled archivers, `swept` counts rows deleted while nobody waited for them — the silent R3 loss, now dashboard-visible instead of log-only). Timing needs no custom histogram: the jobs substrate already records execution time per job (`jobs-execution-timing`, split by job name), so each archive job is timed as itself. No queue-depth gauge: pending depth is read off the replica database directly. The bulk frontend ingest counts `penpot_audit_ingest_events_total{ kind = full|shadow, outcome = inserted|failed }` (both faces share one transaction, so a failure drops the batch on both). Archive success counts are recorded by `run-single-archiver` on the drain totals: a chunk that returns normally committed its transaction (a rollback always throws instead), so they cannot inflate. Logs never carry event data (`props`, `context`, `profile-id`, `ip-addr`, `team-name`, emails): only counts, statuses and the uri host. +Observability (`backend/src/app/loggers/audit/consumer_task.clj`, definitions in `backend/src/app/main.clj:default-metrics`): one `dbg` summary per job run (`:consumer` label, sent/skipped/chunks, reaped disabled pendings, swept orphans, `:elapsed` as `"1.5s"` via `ct/format-duration`, frozen in a strict binding first because log props evaluate lazily in the logger thread), one `dbg` per admin-console send call (`ready` with totals, `sent`/`failed` with counts), plus `dbg` when the queue is empty or the job's send flag is off. Queue depth is Grafana's business: it queries the database directly, so the run performs no `GROUP BY` for the logs. No `warn` before a throw: the runner reports the aborted job (refusal details travel in the error data). Metrics, all recorded with bounded labels only: `penpot_audit_consumer_events_total{ destination = nexus|admin-console, outcome = archived|failed|skipped }` (`skipped` = filtered by the admin-console allowlist) and `penpot_audit_consumer_discarded_total{ reason = reaped|swept }` (entries discarded without archiving: `reaped` counts pendings dropped for disabled consumers, `swept` counts rows deleted while nobody waited for them — the silent R3 loss, now dashboard-visible instead of log-only). Timing needs no custom histogram: the jobs substrate already records execution time per job (`jobs-execution-timing`, split by job name), so each archive job is timed as itself. No queue-depth gauge: pending depth is read off the replica database directly. The bulk frontend ingest counts `penpot_audit_ingest_events_total{ kind = full|shadow, outcome = inserted|failed }` (both faces share one transaction, so a failure drops the batch on both). Archive success counts are recorded by `run-single-consumer` on the drain totals: a chunk that returns normally committed its transaction (a rollback always throws instead), so they cannot inflate. Logs never carry event data (`props`, `context`, `profile-id`, `ip-addr`, `team-name`, emails): only counts, statuses and the uri host. ### How Nexus consumes the data (and forwards to PostHog) @@ -297,9 +297,9 @@ Stored rows are not the end: a consumer pipeline reads them back in sequence ord So the full journey is: Penpot buffer → Nexus `audit_log` (via either archive cron or the telemetry endpoint) → consumers → PostHog (SaaS events grouped by team, telemetry grouped by instance) plus local analytics views and materialized KPIs. -### Admin console: the second archiver +### Admin console: the second consumer -Penpot → admin-console archiving lives in its own job (`:audit-log-archive-admin-console`, archiver `2`, flag `:admin-console`, params closed empty): it drains its own pending queue and ingests allowlisted events through `nitrate/call :ingest-audit-log`, enriched as described above. What also exists (do not confuse with archiving): +Penpot → admin-console archiving lives in its own job (`:audit-log-consumer-admin-console`, consumer `2`, flag `:admin-console`, params closed empty): it drains its own pending queue and ingests allowlisted events through `nitrate/call :ingest-audit-log`, enriched as described above. What also exists (do not confuse with archiving): - **Nitrate → Penpot**: `POST /api/management/methods/push-audit-events` (`backend/src/app/rpc/management/nitrate.clj`) lets the trusted Nitrate backend inject events into the Penpot audit log (no RPC auth, `::audit/skip`, request context + `submit`). Section 7a is the main user. - **Admin-console browser → collector**: usage events from the admin-console UI traveling through the normal public `push-audit-events` (section 7b). @@ -312,13 +312,13 @@ Flags (`common/src/app/common/flags.cljc`, varia; enabled with `PENPOT_FLAGS=ena | Flag | What it does | |---|---| | `:audit-log` | Stores full rows; turns on `wrap-audit` and frontend ingest | -| `:nexus` | Send flag of the Nexus archive job `:audit-log-archive-nexus` (cron `0 */5 * * * ?`; needs `:audit-log-archive-uri`) | -| `:admin-console` | Send flag of the admin-console archive job `:audit-log-archive-admin-console` (cron `30 */5 * * * ?`, 30s after Nexus); also the module flag for the admin-console integration | +| `:nexus` | Send flag of the Nexus archive job `:audit-log-consumer-nexus` (cron `0 */5 * * * ?`; needs `:nexus-uri` (ingest path `api/audit` appended by the job)) | +| `:admin-console` | Send flag of the admin-console archive job `:audit-log-consumer-admin-console` (cron `30 */5 * * * ?`, 30s after Nexus); also the module flag for the admin-console integration | | `:audit-log-logger` | Structured `app.audit` log per event | | `:error-reporting` | Starts the error reporters (off by default); database reporter on flag alone, Mattermost on flag plus webhook URL | | `:telemetry` | Stores shadow rows and turns on shipping (auto with `:telemetry-enabled`) | -Config (`backend/src/app/config.clj`): `:audit-log-archive-uri`, `:telemetry-uri` (default `https://telemetry.penpot.app/`), `:nexus-shared-key` / instance secret, `:public-uri` (sent as `origin` to Nexus). +Config (`backend/src/app/config.clj`): `:nexus-uri`, `:telemetry-uri` (default `https://telemetry.penpot.app/`), `:nexus-shared-key` / instance secret, `:public-uri` (sent as `origin` to Nexus). ## 14. Practical guide: add or change events @@ -337,6 +337,6 @@ Config (`backend/src/app/config.clj`): `:audit-log-archive-uri`, `:telemetry-uri - Penpot frontend producer: `frontend/src/app/main/data/event.cljs` (collector, buffer, send), `frontend/src/app/main/repo.cljs` + `frontend/src/app/util/http.cljs` (headers), `frontend/src/app/main/data/nitrate.cljs` (Nitrate membership events). - Admin-console producer (private repo, no paths cited here): server-side organization/subscription events through the management `push-audit-events`, browser-side UI events through the public `push-audit-events`. - Consumers: `backend/src/app/loggers/webhooks.clj`, `backend/src/app/loggers/database.clj`, `backend/src/app/loggers/mattermost.clj`, `backend/src/app/tasks/telemetry.clj`. -- Archive (two jobs sharing one namespace + pending queue, no GC): `backend/src/app/loggers/audit/archive_task.clj` (`:audit-log-archive-nexus` / `:audit-log-archive-admin-console` job-defs, shared drain/reap/sweep helpers), producer hook in `backend/src/app/loggers/audit.clj` (`append-audit-entry`, single `queue-pendings` helper) plus the bulk ingest in `backend/src/app/rpc/commands/audit.clj` (`handle-events`), metrics/cron wiring in `backend/src/app/main.clj` (one cron entry per job, Nexus at second 0 and admin console at second 30, each gated by its own send flag), single migration `0157` (queue, drop of `archived_at`, covering `(created_at, id)` index for the chronological scan; no backfill — pre-existing rows are swept unsent, an accepted loss). +- Archive (two jobs sharing one namespace + pending queue, no GC): `backend/src/app/loggers/audit/consumer_task.clj` (`:audit-log-consumer-nexus` / `:audit-log-consumer-admin-console` job-defs, shared drain/reap/sweep helpers), producer hook in `backend/src/app/loggers/audit.clj` (`append-audit-entry`, single `queue-pendings` helper) plus the bulk ingest in `backend/src/app/rpc/commands/audit.clj` (`handle-events`), metrics/cron wiring in `backend/src/app/main.clj` (one cron entry per job, Nexus at second 0 and admin console at second 30, each gated by its own send flag), single migration `0157` (queue, drop of `archived_at`, covering `(created_at, id)` index for the chronological scan; no backfill — pre-existing rows are swept unsent, an accepted loss). - Nexus (private repo, no paths cited here): archive ingest, ordered consumers (local profile mirror, PostHog SaaS events, PostHog telemetry), telemetry endpoint, analytics views. - Tests: `backend_tests/rpc-audit-test.clj`, `rpc-team-test/accept-organization-invitation-audit-event`. diff --git a/.serena/memories/prod-infra/core.md b/.serena/memories/prod-infra/core.md index 2b3509a33f..5cd8bb707d 100644 --- a/.serena/memories/prod-infra/core.md +++ b/.serena/memories/prod-infra/core.md @@ -22,7 +22,7 @@ Postgres row locking is the only correctness primitive: `job` claims via `FOR UP Two known race patterns survive multi-backend operation: -- **Cron dedup is best-effort.** The lock on `scheduled_task` is released when the task body finishes. If two backends' cron timers fire for the same scheduled instant with a gap larger than the task body's runtime, both execute it. Penpot's cron entries are idempotent (`session-gc`, `objects-gc`, `storage-gc-*`, `storage-pending-gc`, `jobs-gc`, `file-gc-scheduler`); the exceptions are `:telemetry` (would double-report) and `:audit-log-archive` (depends on archive target idempotency). +- **Cron dedup is best-effort.** The lock on `scheduled_task` is released when the task body finishes. If two backends' cron timers fire for the same scheduled instant with a gap larger than the task body's runtime, both execute it. Penpot's cron entries are idempotent (`session-gc`, `objects-gc`, `storage-gc-*`, `storage-pending-gc`, `jobs-gc`, `file-gc-scheduler`); the exceptions are `:telemetry` (would double-report) and `:audit-log-consumer-nexus` / `:audit-log-consumer-admin-console` (depend on archive target idempotency). - **`jobs/submit ::dedupe true`** does a non-atomic `DELETE` then `INSERT`. Concurrent cross-backend submits can both bypass the `DELETE` (each sees the other's uncommitted insert as absent) and end up with duplicate `'new'` rows. Each row claims and runs once independently, so the underlying work is fine; the "at most one pending" guarantee weakens. Penpot in production lives with both: horizontal-scale deployments accept "exactly-once" as "essentially-once for idempotent operations." Devenv parallel instances handle it by running workers only on ws0 (see `mem:devenv/core`). diff --git a/backend/scripts/_env b/backend/scripts/_env index 4f7c5655bf..f777d16a8a 100644 --- a/backend/scripts/_env +++ b/backend/scripts/_env @@ -68,7 +68,7 @@ export PENPOT_FLAGS="\ # Uncomment for nexus integration testing # export PENPOT_FLAGS="$PENPOT_FLAGS enable-nexus"; -# export PENPOT_AUDIT_LOG_ARCHIVE_URI="http://localhost:6070/api/audit"; +# export PENPOT_NEXUS_URI="http://localhost:6070/"; # Default deletion delay for devenv export PENPOT_DELETION_DELAY="24h" diff --git a/backend/src/app/config.clj b/backend/src/app/config.clj index 297d3de65f..33fde1fd5f 100644 --- a/backend/src/app/config.clj +++ b/backend/src/app/config.clj @@ -211,7 +211,7 @@ [:rpc-rlimit-config {:optional true} ::fs/path] [:rpc-climit-config {:optional true} ::fs/path] - [:audit-log-archive-uri {:optional true} :string] + [:nexus-uri {:optional true} ::sm/uri] [:audit-log-http-handler-concurrency {:optional true} ::sm/int] [:default-executor-parallelism {:optional true} ::sm/int] ;; REVIEW diff --git a/backend/src/app/loggers/audit.clj b/backend/src/app/loggers/audit.clj index 619c367a49..430d369343 100644 --- a/backend/src/app/loggers/audit.clj +++ b/backend/src/app/loggers/audit.clj @@ -21,7 +21,7 @@ [app.http :as-alias http] [app.http.access-token :as-alias actoken] [app.jobs :as jobs] - [app.loggers.audit.archive-task :as archive] + [app.loggers.audit.consumer-task :as consumer] [app.loggers.audit.tasks :as-alias tasks] [app.loggers.webhooks :as-alias webhooks] [app.rpc :as-alias rpc] @@ -244,10 +244,10 @@ (update :ip-addr db/inet)) params (select-keys params event-keys) row (db/insert! cfg :audit-log params)] - ;; The archive queue follows the row in the same transaction: one - ;; pending per enabled archiver. A rollback takes both on its way + ;; The consumer queue follows the row in the same transaction: one + ;; pending per enabled consumer. A rollback takes both on its way ;; out, so a row never outlives its queue. - (archive/queue-pendings cfg [(:id row)]) + (consumer/queue-pendings cfg [(:id row)]) row)) (def ^:private xf:filter-telemetry-props diff --git a/backend/src/app/loggers/audit/archive_task.clj b/backend/src/app/loggers/audit/consumer_task.clj similarity index 77% rename from backend/src/app/loggers/audit/archive_task.clj rename to backend/src/app/loggers/audit/consumer_task.clj index a4eb14e5e6..6c4986eed5 100644 --- a/backend/src/app/loggers/audit/archive_task.clj +++ b/backend/src/app/loggers/audit/consumer_task.clj @@ -4,7 +4,7 @@ ;; ;; Copyright (c) KALEIDOS SUBSIDIARY SL -(ns app.loggers.audit.archive-task +(ns app.loggers.audit.consumer-task (:require [app.common.exceptions :as ex] [app.common.logging :as l] @@ -27,59 +27,59 @@ ;; This task drains the audit log buffer into the long-term archives. ;; ;; Two separate jobs share this namespace and the queue helpers, one -;; per destination: `:audit-log-archive-nexus` and -;; `:audit-log-archive-admin-console`. Each job drains only its own queue +;; per destination: `:audit-log-consumer-nexus` and +;; `:audit-log-consumer-admin-console`. Each job drains only its own queue ;; and deletes its pendings on completion; whoever deletes the last ;; pending also deletes the record ("last one out turns off the ;; light"), so no separate GC task is needed. Housekeeping (reaping ;; disabled pendings, sweeping orphans) lives in both jobs on purpose: ;; only the enabled job runs, so it must clean up for the rest. -;; Archiver ids are fixed constants, never reuse one: retired +;; Consumer ids are fixed constants, never reuse one: retired ;; pendings are reaped by id, so a reused id would inherit the ;; previous owner's queue. -(def archiver-nexus 1) -(def archiver-admin-console 2) +(def consumer-nexus 1) +(def consumer-admin-console 2) -(def ^:private archivers - [{:id archiver-nexus :flag :nexus :label "nexus"} - {:id archiver-admin-console :flag :admin-console :label "admin-console"}]) +(def ^:private consumers + [{:id consumer-nexus :flag :nexus :label "nexus"} + {:id consumer-admin-console :flag :admin-console :label "admin-console"}]) -(def ^:private xf:enabled-archivers - "Static transducer: keep the archivers whose send flag is on. +(def ^:private xf:enabled-consumers + "Static transducer: keep the consumers whose send flag is on. Allocated once so hot paths (every audited RPC) do not rebuild it per call; the flag lookup itself still reads the live flags." (comp (filter #(contains? cf/flags (:flag %))) (map :id))) -(defn enabled-archivers - "Archivers allowed to work right now, from the send flags. Public: - producers queue every record for exactly these archivers." +(defn enabled-consumers + "Consumers allowed to work right now, from the send flags. Public: + producers queue every record for exactly these consumers." [] - (into [] xf:enabled-archivers archivers)) + (into [] xf:enabled-consumers consumers)) (defn queue-pendings - "Queue records for every enabled archiver, nothing when there is - nothing to queue or no archiver is on. One multi-row statement for - all ids instead of one per archiver: this runs inside every audited + "Queue records for every enabled consumer, nothing when there is + nothing to queue or no consumer is on. One multi-row statement for + all ids instead of one per consumer: this runs inside every audited RPC and the bulk frontend ingest. Runs inside the row transaction: a rollback takes both." [cfg ids] - (let [archivers (enabled-archivers)] - (when (and (seq ids) (seq archivers)) + (let [consumers (enabled-consumers)] + (when (and (seq ids) (seq consumers)) (db/insert-many! cfg :audit-log-pending - [:audit-log-id :archiver] + [:audit-log-id :consumer] (for [id ids - archiver archivers] - [id archiver]) + consumer consumers] + [id consumer]) {::sql/on-conflict-do-nothing true})))) -(def ^:private archiver-labels - (into {} (map (juxt :id :label)) archivers)) +(def ^:private consumer-labels + (into {} (map (juxt :id :label)) consumers)) -(defn- archiver-label - [archiver] - (get archiver-labels archiver)) +(defn- consumer-label + [consumer] + (get consumer-labels consumer)) (defn- decode-row [{:keys [props ip-addr context] :as row}] @@ -122,7 +122,7 @@ ;; NOTE: deliberately no SKIP LOCKED here. Batches must reach the ;; destinations in chronological order (Nexus numbers by arrival), -;; so every archiver takes the same total order and a second +;; so every consumer takes the same total order and a second ;; overlapping run waits for the first to commit, then sees whether ;; it was last. The brief blocking is accepted; skipping rows could ;; reorder archive arrival. @@ -130,14 +130,14 @@ "SELECT a.* FROM audit_log_pending p JOIN audit_log a ON a.id = p.audit_log_id - WHERE p.archiver = ?::smallint + WHERE p.consumer = ?::smallint ORDER BY a.created_at ASC, a.id ASC LIMIT 128 FOR UPDATE OF a") (def ^:private sql:delete-pendings "DELETE FROM audit_log_pending - WHERE archiver = ?::smallint + WHERE consumer = ?::smallint AND audit_log_id = ANY(?)") (def ^:private sql:delete-drained @@ -149,8 +149,8 @@ )") (defn- get-pending-rows - [{:keys [::db/conn] :as cfg} archiver] - (->> (db/exec! conn [sql:pending-chunk archiver]) + [{:keys [::db/conn] :as cfg} consumer] + (->> (db/exec! conn [sql:pending-chunk consumer]) (not-empty))) (defn- complete-batch @@ -158,11 +158,21 @@ waiting for anymore. Two statements on purpose: inside a single statement the NOT EXISTS would still see the pending just deleted and never delete a record." - [{:keys [::db/conn]} archiver rows] + [{:keys [::db/conn]} consumer rows] (let [ids (db/create-array conn "uuid" (map :id rows))] - (db/exec! conn [sql:delete-pendings archiver ids]) + (db/exec! conn [sql:delete-pendings consumer ids]) (db/exec! conn [sql:delete-drained ids]))) +(defn- generate-nexus-uri + "Joins relative path segments to the Nexus backend URI, same shape + as the admin-console helper in `app.nitrate` (there the ingest path + is `api/audit-log`; here it is `api/audit`). + Segments must not start with `/`. Returns nil when no base URI is + configured, so the caller keeps the `:task-not-configured` guard." + [& segments] + (when-let [base (cf/get :nexus-uri)] + (apply cf/join-uri base segments))) + (defn- send-to-nexus "POST events to Nexus. Returns the number of events sent; throws on transport failure or non-204 refusal, carrying the attempted count @@ -190,7 +200,7 @@ (if (= (:status resp) 204) n (throw (ex/raise :type :internal - :code :archive-chunk-refused + :code :consumer-chunk-refused :hint "nexus refused archive chunk" :resp-status (:status resp) :resp-body (truncate-body (:body resp) 500) @@ -338,8 +348,8 @@ total (count rows) skipped (- total (count events)) uri-host (some-> (cf/get :admin-console-uri) u/uri u/get-domain)] - (l/dbg :hint "archive events chunk ready" - :archiver "admin-console" + (l/dbg :hint "consumer events chunk ready" + :consumer "admin-console" :uri-host uri-host :total total :skipped skipped) @@ -352,15 +362,15 @@ n (count events)] (try (nitrate/call cfg :ingest-audit-log {:events enriched}) - (l/dbg :hint "archive events chunk sent" - :archiver "admin-console" + (l/dbg :hint "consumer events chunk sent" + :consumer "admin-console" :uri-host uri-host :sent n :skipped skipped) {:sent n :skipped skipped} (catch Throwable cause - (l/dbg :hint "archive events chunk failed" - :archiver "admin-console" + (l/dbg :hint "consumer events chunk failed" + :consumer "admin-console" :uri-host uri-host :events n :code :admin-console-send-failed) @@ -370,7 +380,7 @@ :cause cause :events n)))))))) -(defn- archive-nexus-chunk +(defn- consumer-nexus-chunk "Send one pending chunk to Nexus and complete it. Returns {:sent n} or nil when the queue is empty; throws on refusal. Takes a transactional cfg: the caller opens one transaction per chunk. No @@ -378,40 +388,40 @@ the caller records the success." [cfg] (db/exec! cfg ["SET LOCAL lock_timeout = '30s'"]) - (when-let [rows (get-pending-rows cfg archiver-nexus)] + (when-let [rows (get-pending-rows cfg consumer-nexus)] (let [events (mapv row->event (mapv decode-row rows)) ;; send-to-nexus returns the count or throws: a refusal ;; aborts the job, the rows stay for the next tick. n (send-to-nexus cfg events)] - (l/trc :hint "archive events chunk" - :archiver "nexus" + (l/trc :hint "consumer events chunk" + :consumer "nexus" :events n) - (complete-batch cfg archiver-nexus rows) + (complete-batch cfg consumer-nexus rows) {:sent n}))) -(defn- archive-admin-console-chunk +(defn- consumer-admin-console-chunk "Send one pending chunk to the admin console and complete it. Returns {:sent :skipped} or nil when the queue is empty; throws on failure. Takes a transactional cfg, no metrics here: same contract as - archive-nexus-chunk." + consumer-nexus-chunk." [cfg] (db/exec! cfg ["SET LOCAL lock_timeout = '30s'"]) - (when-let [rows (get-pending-rows cfg archiver-admin-console)] + (when-let [rows (get-pending-rows cfg consumer-admin-console)] (let [decoded (mapv decode-row rows) - _ (l/trc :hint "archive events chunk" - :archiver "admin-console" + _ (l/trc :hint "consumer events chunk" + :consumer "admin-console" :events (count decoded)) admin-console-res (send-to-admin-console cfg decoded)] ;; Filtered-out rows count as processed for the ;; admin-console queue: their pending is done, the - ;; record stays until every archiver is done. - (complete-batch cfg archiver-admin-console rows) + ;; record stays until every consumer is done. + (complete-batch cfg consumer-admin-console rows) admin-console-res))) -(defn- drain-archiver - "Drain one archiver queue chunk by chunk, one transaction per chunk. +(defn- drain-consumer + "Drain one consumer queue chunk by chunk, one transaction per chunk. Returns totals; a chunk the destination refuses (or that throws) aborts the drain with a throw, and the committed chunks so far stay sent." @@ -431,11 +441,11 @@ {:sent sent :skipped skipped :chunks chunks}))) (defn- sql:reap-pendings - [archiver] + [consumer] (str "DELETE FROM audit_log_pending" - " WHERE archiver = " archiver + " WHERE consumer = " consumer " AND audit_log_id IN (SELECT audit_log_id FROM audit_log_pending" - " WHERE archiver = " archiver + " WHERE consumer = " consumer " ORDER BY audit_log_id LIMIT 5000)")) (def ^:private sql:sweep-orphans @@ -457,35 +467,35 @@ total)))) (defn- reap-disabled - "Drop the pendings of archivers that are off: a disabled destination + "Drop the pendings of consumers that are off: a disabled destination will never drain them, and without this they would pin their records forever. This also absorbs any later deactivation, with no manual step. Dropped pendings count as discarded: their rows may still be archived - by an enabled archiver, or swept unsent when nothing waits for them." + by an enabled consumer, or swept unsent when nothing waits for them." [cfg] - (let [total (->> archivers + (let [total (->> consumers (remove #(contains? cf/flags (:flag %))) (map #(delete-batched cfg (sql:reap-pendings (:id %)))) (reduce + 0))] (when (pos? total) - (record-count (mtx/instance cfg) :audit-archive-discarded ["reaped"] total)) + (record-count (mtx/instance cfg) :audit-consumer-discarded ["reaped"] total)) total)) (defn- sweep-orphans "Delete records nobody waits for anymore: rows born while no - archiver was on, rows left pending-less by the reap, and leftovers + consumer was on, rows left pending-less by the reap, and leftovers archived by the old task. Batched and normally a no-op. Deleted rows count as discarded: on a live system they were never archived." [cfg] (let [total (delete-batched cfg sql:sweep-orphans)] (when (pos? total) - (record-count (mtx/instance cfg) :audit-archive-discarded ["swept"] total)) + (record-count (mtx/instance cfg) :audit-consumer-discarded ["swept"] total)) total)) -(declare execute-audit-log-archive-nexus) -(declare execute-audit-log-archive-admin-console) +(declare execute-audit-log-consumer-nexus) +(declare execute-audit-log-consumer-admin-console) -(def schema:audit-log-archive-nexus-params +(def schema:audit-log-consumer-nexus-params "Overrides for the nexus job repl invocation defaults. Cron submits {}; system deps (pool, shared-keys, http) live on the job-def cfg closed over by the handler, not in job params. @@ -494,7 +504,7 @@ [:map {:closed true} [:uri {:optional true} ::sm/uri]]) -(def schema:audit-log-archive-admin-console-params +(def schema:audit-log-consumer-admin-console-params "Params of the admin-console job: none. Cron submits {}; system deps (pool, admin-console client) live on the job-def cfg closed over by the handler, not in job params. Closed on purpose, same as the nexus @@ -511,16 +521,16 @@ [_ cfg] ;; Effective destination, once at boot: per-run logs carry counts, ;; the runner says the job started, this says where it sends. - (l/dbg :hint "audit log archive job initialized" - :archiver "nexus" - :uri-host (some-> (cf/get :audit-log-archive-uri) u/uri u/get-domain)) - {::jobs/name :audit-log-archive-nexus - ::jobs/schema schema:audit-log-archive-nexus-params + (l/dbg :hint "audit log consumer job initialized" + :consumer "nexus" + :uri-host (some-> (cf/get :nexus-uri) u/uri u/get-domain)) + {::jobs/name :audit-log-consumer-nexus + ::jobs/schema schema:audit-log-consumer-nexus-params ::jobs/handler (fn [_context params] - (execute-audit-log-archive-nexus cfg params)) - ::jobs/decoder (sm/decoder schema:audit-log-archive-nexus-params sm/json-transformer) - ::jobs/validator (sm/validator schema:audit-log-archive-nexus-params)}) + (execute-audit-log-consumer-nexus cfg params)) + ::jobs/decoder (sm/decoder schema:audit-log-consumer-nexus-params sm/json-transformer) + ::jobs/validator (sm/validator schema:audit-log-consumer-nexus-params)}) (defmethod ig/assert-key ::admin-console-job-def [_ cfg] @@ -534,28 +544,28 @@ ;; Same as the nexus job-def: destination host once at boot. The ;; ingest uri is resolved by nitrate/call at send time (it raises ;; when missing); here it is display-only. - (l/dbg :hint "audit log archive job initialized" - :archiver "admin-console" + (l/dbg :hint "audit log consumer job initialized" + :consumer "admin-console" :uri-host (some-> (cf/get :admin-console-uri) u/uri u/get-domain)) - {::jobs/name :audit-log-archive-admin-console - ::jobs/schema schema:audit-log-archive-admin-console-params + {::jobs/name :audit-log-consumer-admin-console + ::jobs/schema schema:audit-log-consumer-admin-console-params ::jobs/handler (fn [_context params] - (execute-audit-log-archive-admin-console cfg params)) - ::jobs/decoder (sm/decoder schema:audit-log-archive-admin-console-params sm/json-transformer) - ::jobs/validator (sm/validator schema:audit-log-archive-admin-console-params)}) + (execute-audit-log-consumer-admin-console cfg params)) + ::jobs/decoder (sm/decoder schema:audit-log-consumer-admin-console-params sm/json-transformer) + ::jobs/validator (sm/validator schema:audit-log-consumer-admin-console-params)}) -(defn- run-single-archiver - "Drain one archiver queue, then housekeeping (reap disabled +(defn- run-single-consumer + "Drain one consumer queue, then housekeeping (reap disabled pendings, sweep orphans) and the per-run summary. Housekeeping lives in both jobs: only the enabled job runs, so it must reap the queues - of the archivers that are off; both halves are idempotent deletes. + of the consumers that are off; both halves are idempotent deletes. A refused chunk (or a throw) aborts the job: the committed chunks stay sent, the rest waits for the next tick, and the cron retries. Every effect is rows and logs (drain metrics land here, outside every chunk transaction)." - [cfg archiver send-chunk tpoint] - (let [label (archiver-label archiver) + [cfg consumer send-chunk tpoint] + (let [label (consumer-label consumer) metrics (mtx/instance cfg) reaped (reap-disabled cfg) ;; All drain metrics land here, outside every chunk @@ -567,17 +577,17 @@ ;; uncounted. The throw keeps rolling: a failed chunk aborts ;; the job, the cron retries. drained (try - (drain-archiver cfg send-chunk) + (drain-consumer cfg send-chunk) (catch Throwable cause (let [n (:events (ex-data cause))] (when (and (integer? n) (pos? n)) - (record-count metrics :audit-archive-events [label "failed"] n))) + (record-count metrics :audit-consumer-events [label "failed"] n))) (throw cause))) _ (let [{:keys [sent skipped]} drained] (when (pos? sent) - (record-count metrics :audit-archive-events [label "archived"] sent)) + (record-count metrics :audit-consumer-events [label "archived"] sent)) (when (pos? skipped) - (record-count metrics :audit-archive-events [label "skipped"] skipped))) + (record-count metrics :audit-consumer-events [label "skipped"] skipped))) swept (sweep-orphans cfg) sent (:sent drained) skipped (:skipped drained) @@ -590,9 +600,9 @@ (zero? reaped) (zero? swept)) (l/dbg :hint "no pending audit events" - :archiver label) + :consumer label) (l/dbg :hint "events archived" - :archiver label + :consumer label :elapsed elapsed :sent sent :skipped skipped @@ -600,16 +610,15 @@ :reaped reaped :swept swept)))) -(defn execute-audit-log-archive-nexus +(defn execute-audit-log-consumer-nexus "Nexus job handler: drain the nexus queue in chunks, then clean up (heartbeat per iteration: the sent chunk batches can be long)." [cfg params] ;; NOTE: this let allows overwrite default configured values from ;; the repl, when manually invoking the task. (if-not (contains? cf/flags :nexus) - (l/dbg :hint "audit log archive (nexus) skipped, flag off") - (let [uri (cf/get :audit-log-archive-uri) - uri (or uri (:uri params)) + (l/dbg :hint "audit log consumer (nexus) skipped, flag off") + (let [uri (or (:uri params) (generate-nexus-uri "api/audit")) ;; Normalize to an uri object; params may carry a plain string ;; on direct invocations (validation only applies on submit!). uri (u/uri uri) @@ -617,14 +626,14 @@ (when-not uri (ex/raise :type :internal :code :task-not-configured - :hint "archive task not configured, missing uri")) - (run-single-archiver cfg archiver-nexus archive-nexus-chunk (ct/tpoint))))) + :hint "consumer task not configured, missing uri")) + (run-single-consumer cfg consumer-nexus consumer-nexus-chunk (ct/tpoint))))) -(defn execute-audit-log-archive-admin-console +(defn execute-audit-log-consumer-admin-console "Admin-console job handler: drain the admin-console queue in chunks, then clean up (heartbeat per iteration: the sent chunk batches can be long)." [cfg _params] (if-not (contains? cf/flags :admin-console) - (l/dbg :hint "audit log archive (admin-console) skipped, flag off") - (run-single-archiver cfg archiver-admin-console archive-admin-console-chunk (ct/tpoint)))) + (l/dbg :hint "audit log consumer (admin-console) skipped, flag off") + (run-single-consumer cfg consumer-admin-console consumer-admin-console-chunk (ct/tpoint)))) diff --git a/backend/src/app/main.clj b/backend/src/app/main.clj index 5a387f14be..7476f9bfb2 100644 --- a/backend/src/app/main.clj +++ b/backend/src/app/main.clj @@ -373,15 +373,15 @@ ::mdef/help "Total number of requests whose body the parser could not read." ::mdef/type :counter} - :audit-archive-events - {::mdef/name "penpot_audit_archive_events_total" - ::mdef/help "Audit log events sent by the archive task, by destination and outcome." + :audit-consumer-events + {::mdef/name "penpot_audit_consumer_events_total" + ::mdef/help "Audit log events drained by the consumer task, by destination and outcome." ::mdef/labels ["destination" "outcome"] ::mdef/type :counter} - :audit-archive-discarded - {::mdef/name "penpot_audit_archive_discarded_total" - ::mdef/help "Audit log entries discarded without archiving, by reason." + :audit-consumer-discarded + {::mdef/name "penpot_audit_consumer_discarded_total" + ::mdef/help "Audit log entries discarded without being archived, by reason." ::mdef/labels ["reason"] ::mdef/type :counter} @@ -593,7 +593,7 @@ :app.nitrate/client {::http.client/client (ig/ref ::http.client/client) - ;; No ::jobs/defs: nitrate never submits jobs, and archive-task's + ;; No ::jobs/defs: nitrate never submits jobs, and consumer-task's ;; job-def needs this client — that would cycle through jobs/defs. ::setup/shared-keys (ig/ref ::setup/shared-keys)} @@ -639,8 +639,8 @@ :telemetry (ig/ref :app.tasks.telemetry/job-def) :session-gc (ig/ref ::session/job-def) :file-gc-scheduler (ig/ref :app.tasks.file-gc-scheduler/job-def) - :audit-log-archive-nexus (ig/ref :app.loggers.audit.archive-task/nexus-job-def) - :audit-log-archive-admin-console (ig/ref :app.loggers.audit.archive-task/admin-console-job-def)} + :audit-log-consumer-nexus (ig/ref :app.loggers.audit.consumer-task/nexus-job-def) + :audit-log-consumer-admin-console (ig/ref :app.loggers.audit.consumer-task/admin-console-job-def)} ::jobs/request-pool {::rds/client (ig/ref ::rds/client) @@ -729,13 +729,13 @@ {::db/pool (ig/ref ::db/pool) ::mtx/metrics (ig/ref ::mtx/metrics)} - :app.loggers.audit.archive-task/nexus-job-def + :app.loggers.audit.consumer-task/nexus-job-def {::db/pool (ig/ref ::db/pool) ::http.client/client (ig/ref ::http.client/client) ::setup/shared-keys (ig/ref ::setup/shared-keys) ::mtx/metrics (ig/ref ::mtx/metrics)} - :app.loggers.audit.archive-task/admin-console-job-def + :app.loggers.audit.consumer-task/admin-console-job-def {::db/pool (ig/ref ::db/pool) :app.nitrate/client (ig/ref :app.nitrate/client) ::mtx/metrics (ig/ref ::mtx/metrics)} @@ -861,11 +861,11 @@ (when (contains? cf/flags :nexus) {:cron #penpot/cron "0 */5 * * * ?" ;; every 5m - :job :audit-log-archive-nexus}) + :job :audit-log-consumer-nexus}) (when (contains? cf/flags :admin-console) {:cron #penpot/cron "30 */5 * * * ?" ;; every 5m, offset 30s from nexus - :job :audit-log-archive-admin-console})]} + :job :audit-log-consumer-admin-console})]} ::wrk/dispatcher {::rds/client (ig/ref ::rds/client) diff --git a/backend/src/app/migrations/sql/0157-add-audit-log-pending-table.sql b/backend/src/app/migrations/sql/0157-add-audit-log-pending-table.sql index 4c6d8d998d..ce797dd12c 100644 --- a/backend/src/app/migrations/sql/0157-add-audit-log-pending-table.sql +++ b/backend/src/app/migrations/sql/0157-add-audit-log-pending-table.sql @@ -1,8 +1,8 @@ --- Work queue for the audit log archivers: one row per (record, --- archiver) pair still pending. Inserting a record creates its --- pendings; each archiver deletes its own on completion, and whoever --- deletes the last pending also deletes the record. Archiver ids are --- fixed in code and never reused: 1 = nexus, 2 = nitrate. +-- Work queue for the audit log consumers: one row per (record, +-- consumer) pair still pending. Inserting a record creates its +-- pendings; each consumer deletes its own on completion, and whoever +-- deletes the last pending also deletes the record. Consumer ids are +-- fixed in code and never reused: 1 = nexus, 2 = admin console. CREATE TABLE audit_log_pending ( -- DEFERRABLE so the test-suite reset (which deletes table by table @@ -11,8 +11,8 @@ CREATE TABLE audit_log_pending ( -- (initially immediate) is enough: every deleter removes pendings -- before rows, so the check passes at each statement. audit_log_id uuid NOT NULL REFERENCES audit_log(id) DEFERRABLE, - archiver smallint NOT NULL CHECK (archiver BETWEEN 1 AND 20), - PRIMARY KEY (archiver, audit_log_id)); + consumer smallint NOT NULL CHECK (consumer BETWEEN 1 AND 20), + PRIMARY KEY (consumer, audit_log_id)); -- Lets the final delete ask "is anything still pending for this -- record?" without scanning the whole queue. @@ -31,8 +31,8 @@ DROP INDEX IF EXISTS audit_log__archived_at__idx; DROP INDEX IF EXISTS audit_log_id_archived_at_idx; ALTER TABLE audit_log DROP COLUMN IF EXISTS archived_at; --- Serves the archivers' chronological scan (ORDER BY created_at, id): --- one total order for every archiver, so concurrent drains cannot +-- Serves the consumers' chronological scan (ORDER BY created_at, id): +-- one total order for every consumer, so concurrent drains cannot -- deadlock on row locks. Covers the whole ORDER BY (bulk ingest -- shares request-at constantly, so ties are the norm). CREATE INDEX IF NOT EXISTS audit_log__created_at__idx diff --git a/backend/src/app/rpc/commands/audit.clj b/backend/src/app/rpc/commands/audit.clj index 771b321606..4d5f26a402 100644 --- a/backend/src/app/rpc/commands/audit.clj +++ b/backend/src/app/rpc/commands/audit.clj @@ -16,7 +16,7 @@ [app.db :as db] [app.http :as-alias http] [app.loggers.audit :as audit] - [app.loggers.audit.archive-task :as archive] + [app.loggers.audit.consumer-task :as consumer] [app.loggers.database :as loggers.db] [app.loggers.mattermost :as loggers.mm] [app.metrics :as mtx] @@ -127,8 +127,8 @@ (when (seq rows) (db/insert-many! tx :audit-log event-columns rows) ;; Same transaction as the rows: every ingested record is born - ;; pending for every enabled archiver (see archive-task). - (archive/queue-pendings tx (map first rows)))) + ;; pending for every enabled consumer (see consumer-task). + (consumer/queue-pendings tx (map first rows)))) (defn- handle-events [{:keys [::db/pool] :as cfg} params] diff --git a/backend/src/app/tasks/telemetry.clj b/backend/src/app/tasks/telemetry.clj index 8088841380..05cd056d92 100644 --- a/backend/src/app/tasks/telemetry.clj +++ b/backend/src/app/tasks/telemetry.clj @@ -233,8 +233,8 @@ so that the buffer stays bounded." [{:keys [::db/conn]}] ;; One transaction for both deletes: a crash between them would - ;; leave pending-less rows for the archive sweep to delete unsent. - ;; Pendings first: the queue holds a row per archiver for every + ;; leave pending-less rows for the consumer sweep to delete unsent. + ;; Pendings first: the queue holds a row per consumer for every ;; record, and the FK would reject deleting a row that still has ;; pendings. (db/tx-run! conn @@ -289,7 +289,7 @@ (defn- delete-sent-events "Delete rows by their ids after a successful send. One transaction for both deletes: a crash between them would leave pending-less - rows for the archive sweep to delete unsent." + rows for the consumer sweep to delete unsent." [conn ids] (db/tx-run! conn (fn [tx] diff --git a/backend/test/backend_tests/loggers_audit_archive_task_test.clj b/backend/test/backend_tests/loggers_audit_consumer_task_test.clj similarity index 82% rename from backend/test/backend_tests/loggers_audit_archive_task_test.clj rename to backend/test/backend_tests/loggers_audit_consumer_task_test.clj index c0e5a5a0e0..146f0da10b 100644 --- a/backend/test/backend_tests/loggers_audit_archive_task_test.clj +++ b/backend/test/backend_tests/loggers_audit_consumer_task_test.clj @@ -4,7 +4,7 @@ ;; ;; Copyright (c) KALEIDOS SUBSIDIARY SL -(ns backend-tests.loggers-audit-archive-task-test +(ns backend-tests.loggers-audit-consumer-task-test (:require [app.common.json :as json] [app.common.schema :as sm] @@ -17,7 +17,7 @@ [app.http.client :as-alias http] [app.jobs :as jobs] [app.loggers.audit :as audit] - [app.loggers.audit.archive-task :as archive] + [app.loggers.audit.consumer-task :as consumer] [app.main :as main] [app.metrics :as mtx] [app.metrics.definition :as-alias mdef] @@ -38,9 +38,9 @@ (defn- insert-pendings! "Queue the row with the real producer helper: pendings for the - enabled archivers, like append-audit-entry does." + enabled consumers, like append-audit-entry does." [row-id] - (archive/queue-pendings (:app.db/pool th/*system*) [row-id])) + (consumer/queue-pendings (:app.db/pool th/*system*) [row-id])) (defn- pending-count [] @@ -70,13 +70,13 @@ (:props (first (:events params))))) (defn- run-both! - "Run both archive jobs, Nexus first (the historical single-run + "Run both consumer jobs, Nexus first (the historical single-run order). The admin-console job takes no params (closed empty schema)." [params] - (th/run-task! :audit-log-archive-nexus params) - (th/run-task! :audit-log-archive-admin-console {})) + (th/run-task! :audit-log-consumer-nexus params) + (th/run-task! :audit-log-consumer-admin-console {})) -(t/deftest archive-jobs-drain-their-own-queues +(t/deftest consumer-jobs-drain-their-own-queues ;; Two separate jobs, one queue each: the nexus job sends without ;; touching the admin console, and only the admin-console job ;; deletes the last pending and the row. Create profile before @@ -88,14 +88,14 @@ http-mock {:target 'app.http.client/req :return {:status 204}}] (insert-audit-row! (:id prof) "create-project") (insert-audit-row! (:id prof) "create-file") - (th/run-task! :audit-log-archive-nexus {:uri archive-uri}) + (th/run-task! :audit-log-consumer-nexus {:uri archive-uri}) (t/is (false? (:called? @admin-console-mock))) (t/is (:called? @http-mock)) (t/is (= 1 (:call-count @http-mock))) ;; Nexus drained its own queue; both rows wait on the admin console. (t/is (= 2 (count (th/db-exec! ["select * from audit_log"])))) (t/is (= 2 (pending-count))) - (th/run-task! :audit-log-archive-admin-console {}) + (th/run-task! :audit-log-consumer-admin-console {}) (t/is (= 1 (:call-count @admin-console-mock))) (let [[_ method params] (:call-args @admin-console-mock)] (t/is (= :ingest-audit-log method)) @@ -106,7 +106,7 @@ (let [rows (th/db-exec! ["select * from audit_log"])] (t/is (= 0 (count rows)))))))) -(t/deftest archive-events-filters-admin-console-event-names +(t/deftest consumer-events-filters-admin-console-event-names (let [prof (th/create-profile* 1 {:is-active true})] (with-redefs [cf/flags #{:nexus :admin-console}] (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil} @@ -123,7 +123,7 @@ (let [rows (th/db-exec! ["select * from audit_log"])] (t/is (= 0 (count rows)))))))) -(t/deftest archive-events-skips-admin-console-when-no-matching-names +(t/deftest consumer-events-skips-admin-console-when-no-matching-names (let [prof (th/create-profile* 1 {:is-active true})] (with-redefs [cf/flags #{:nexus :admin-console}] (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil} @@ -135,31 +135,31 @@ (let [rows (th/db-exec! ["select * from audit_log"])] (t/is (= 0 (count rows)))))))) -(t/deftest archive-events-admin-console-only-marks-without-uri +(t/deftest consumer-events-admin-console-only-marks-without-uri (let [prof (th/create-profile* 1 {:is-active true})] (with-redefs [cf/flags #{:admin-console}] (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil} http-mock {:target 'app.http.client/req :return {:status 204}}] (insert-audit-row! (:id prof) "create-project") - (th/run-task! :audit-log-archive-admin-console {}) + (th/run-task! :audit-log-consumer-admin-console {}) (t/is (:called? @admin-console-mock)) (t/is (false? (:called? @http-mock))) (let [rows (th/db-exec! ["select * from audit_log"])] (t/is (= 0 (count rows)))))))) -(t/deftest archive-events-admin-console-only-marks-non-allowlisted +(t/deftest consumer-events-admin-console-only-marks-non-allowlisted (let [prof (th/create-profile* 1 {:is-active true})] (with-redefs [cf/flags #{:admin-console}] (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil} http-mock {:target 'app.http.client/req :return {:status 204}}] (insert-audit-row! (:id prof) "unrelated-event") - (th/run-task! :audit-log-archive-admin-console {}) + (th/run-task! :audit-log-consumer-admin-console {}) (t/is (false? (:called? @admin-console-mock))) (t/is (false? (:called? @http-mock))) (let [rows (th/db-exec! ["select * from audit_log"])] (t/is (= 0 (count rows)))))))) -(t/deftest archive-events-admin-console-error-still-drains-nexus +(t/deftest consumer-events-admin-console-error-still-drains-nexus ;; Mirror of the nexus-throw test: an admin-console failure aborts ;; only the admin-console job; Nexus already drained its own queue ;; in its job. The row stays on the admin-console pending. @@ -169,9 +169,9 @@ :throw (ex-info "nitrate down" {})} http-mock {:target 'app.http.client/req :return {:status 204}}] (insert-audit-row! (:id prof) "create-project") - (th/run-task! :audit-log-archive-nexus {:uri archive-uri}) + (th/run-task! :audit-log-consumer-nexus {:uri archive-uri}) (t/is (thrown? clojure.lang.ExceptionInfo - (th/run-task! :audit-log-archive-admin-console {}))) + (th/run-task! :audit-log-consumer-admin-console {}))) ;; Nexus already drained its queue; the admin-console failure no ;; longer blocks it, but the row stays on the admin-console pending. (t/is (:called? @admin-console-mock)) @@ -180,44 +180,44 @@ (t/is (= 1 (count rows)))) (t/is (= 1 (pending-count))))))) -(t/deftest archive-events-uses-nexus-without-admin-console +(t/deftest consumer-events-uses-nexus-without-admin-console (let [prof (th/create-profile* 1 {:is-active true})] (with-redefs [cf/flags #{:nexus}] (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}} admin-console-mock {:target 'app.nitrate/call :return nil}] (insert-audit-row! (:id prof) "create-project") - (th/run-task! :audit-log-archive-nexus {:uri archive-uri}) + (th/run-task! :audit-log-consumer-nexus {:uri archive-uri}) (t/is (false? (:called? @admin-console-mock))) (t/is (:called? @http-mock)) (let [rows (th/db-exec! ["select * from audit_log"])] (t/is (= 0 (count rows)))))))) -(t/deftest archive-events-skips-mark-when-nexus-fails +(t/deftest consumer-events-skips-mark-when-nexus-fails (let [prof (th/create-profile* 1 {:is-active true})] (with-redefs [cf/flags #{:nexus}] (with-mocks [http-mock {:target 'app.http.client/req :return {:status 500}}] (insert-audit-row! (:id prof) "create-project") (t/is (thrown? clojure.lang.ExceptionInfo - (th/run-task! :audit-log-archive-nexus {:uri archive-uri}))) + (th/run-task! :audit-log-consumer-nexus {:uri archive-uri}))) (t/is (:called? @http-mock)) (let [rows (th/db-exec! ["select * from audit_log"])] (t/is (= 1 (count rows)))))))) -(t/deftest archive-events-both-flags-skips-mark-when-nexus-fails +(t/deftest consumer-events-both-flags-skips-mark-when-nexus-fails (let [prof (th/create-profile* 1 {:is-active true})] (with-redefs [cf/flags #{:nexus :admin-console}] (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil} http-mock {:target 'app.http.client/req :return {:status 500}}] (insert-audit-row! (:id prof) "create-project") (t/is (thrown? clojure.lang.ExceptionInfo - (th/run-task! :audit-log-archive-nexus {:uri archive-uri}))) - (th/run-task! :audit-log-archive-admin-console {}) + (th/run-task! :audit-log-consumer-nexus {:uri archive-uri}))) + (th/run-task! :audit-log-consumer-admin-console {}) (t/is (:called? @admin-console-mock)) (t/is (:called? @http-mock)) (let [rows (th/db-exec! ["select * from audit_log"])] (t/is (= 1 (count rows)))))))) -(t/deftest archive-events-encodes-db-rows-for-admin-console +(t/deftest consumer-events-encodes-db-rows-for-admin-console (let [prof (th/create-profile* 1 {:is-active true})] (with-redefs [cf/flags #{:nexus :admin-console} cf/config (assoc cf/config @@ -229,16 +229,16 @@ client (ig/init-key :app.nitrate/client {::http/client (Object.) ::setup/shared-keys shared-keys}) - ;; Each archive job is a job-def: the handler closes over the + ;; Each consumer job is a job-def: the handler closes over the ;; cfg it was given at init and takes a nil context, because ;; an in-process invocation has no job row to describe. - nexus-job-def (ig/init-key :app.loggers.audit.archive-task/nexus-job-def + nexus-job-def (ig/init-key :app.loggers.audit.consumer-task/nexus-job-def {::db/pool (:app.db/pool th/*system*) ::setup/shared-keys shared-keys ::http/client (:app.http.client/client th/*system*) ::mtx/metrics (th/metrics-stub) :app.nitrate/client client}) - admin-console-job-def (ig/init-key :app.loggers.audit.archive-task/admin-console-job-def + admin-console-job-def (ig/init-key :app.loggers.audit.consumer-task/admin-console-job-def {::db/pool (:app.db/pool th/*system*) ::mtx/metrics (th/metrics-stub) :app.nitrate/client client})] @@ -278,16 +278,32 @@ (let [rows (th/db-exec! ["select * from audit_log"])] (t/is (= 0 (count rows)))))))))) -(t/deftest archive-nexus-job-requires-uri-when-nexus-flag-set +(t/deftest consumer-nexus-default-uri-appends-api-audit + ;; Without an explicit :uri param, the job posts to the Nexus base + ;; URI plus the ingest path, mirroring generate-nitrate-uri. + (let [prof (th/create-profile* 1 {:is-active true}) + uris (atom [])] + (with-redefs [cf/flags #{:nexus} + cf/config (assoc cf/config :nexus-uri (u/uri "http://nexus.example"))] + (with-mocks [http-mock {:target 'app.http.client/req + :return (fn [_ params & _] + (swap! uris conj (str (:uri params))) + {:status 204})}] + (insert-audit-row! (:id prof) "create-project") + (th/run-task! :audit-log-consumer-nexus {}) + (t/is (= ["http://nexus.example/api/audit"] @uris)) + (t/is (= 0 (count (th/db-exec! ["select * from audit_log"])))))))) + +(t/deftest consumer-nexus-job-requires-uri-when-nexus-flag-set (with-redefs [cf/flags #{:nexus} - cf/config (dissoc cf/config :audit-log-archive-uri)] + cf/config (dissoc cf/config :nexus-uri)] (let [data (ex-data (try - (th/run-task! :audit-log-archive-nexus {}) + (th/run-task! :audit-log-consumer-nexus {}) (catch Throwable cause cause)))] (t/is (= :task-not-configured (:code data)))))) -(t/deftest archive-events-enriches-team-name-and-emails +(t/deftest consumer-events-enriches-team-name-and-emails (let [actor (th/create-profile* 1 {:is-active true}) member (th/create-profile* 2 {:is-active true}) team (th/create-team* 1 {:profile-id (:id actor)})] @@ -300,7 +316,7 @@ :member-id (:id member) ;; Producers store this as a string UUID. :user-who-send-invitation (str (:id actor))}) - (th/run-task! :audit-log-archive-admin-console {}) + (th/run-task! :audit-log-consumer-admin-console {}) (t/is (:called? @admin-console-mock)) (let [props (admin-console-event-props admin-console-mock)] (t/is (= (:name team) (:team-name props))) @@ -311,7 +327,7 @@ (let [rows (th/db-exec! ["select * from audit_log"])] (t/is (= 0 (count rows)))))))) -(t/deftest archive-events-enriches-soft-deleted-team-and-profile +(t/deftest consumer-events-enriches-soft-deleted-team-and-profile (let [actor (th/create-profile* 1 {:is-active true}) other (th/create-profile* 2 {:is-active true}) team (th/create-team* 1 {:profile-id (:id actor)})] @@ -322,14 +338,14 @@ (insert-audit-row! (:id actor) "create-project" {:team-id (:id team) :profile-id (:id other)}) - (th/run-task! :audit-log-archive-admin-console {}) + (th/run-task! :audit-log-consumer-admin-console {}) (let [props (admin-console-event-props admin-console-mock)] (t/is (= (:name team) (:team-name props))) (t/is (= (:email other) (:profile-email props)))) (let [rows (th/db-exec! ["select * from audit_log"])] (t/is (= 0 (count rows)))))))) -(t/deftest archive-events-does-not-overwrite-existing-enrich-props +(t/deftest consumer-events-does-not-overwrite-existing-enrich-props (let [actor (th/create-profile* 1 {:is-active true}) team (th/create-team* 1 {:profile-id (:id actor)})] (with-redefs [cf/flags #{:admin-console}] @@ -345,7 +361,7 @@ :member-email "already-member@example.com" :user-who-send-invitation (str (:id actor)) :user-who-send-invitation-email "already-inviter@example.com"}) - (th/run-task! :audit-log-archive-admin-console {}) + (th/run-task! :audit-log-consumer-admin-console {}) (let [props (admin-console-event-props admin-console-mock)] (t/is (= "already-set-name" (:team-name props))) (t/is (= "already-set@example.com" (:profile-email props))) @@ -354,7 +370,7 @@ (t/is (= "already-inviter@example.com" (:user-who-send-invitation-email props)))))))) -(t/deftest archive-events-enrich-skips-missing-and-non-uuid-ids +(t/deftest consumer-events-enrich-skips-missing-and-non-uuid-ids (let [actor (th/create-profile* 1 {:is-active true})] (with-redefs [cf/flags #{:admin-console}] (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}] @@ -363,7 +379,7 @@ :profile-id (uuid/next) :user-id "not-a-uuid" :member-id 42}) - (th/run-task! :audit-log-archive-admin-console {}) + (th/run-task! :audit-log-consumer-admin-console {}) (t/is (:called? @admin-console-mock)) (let [props (admin-console-event-props admin-console-mock)] (t/is (nil? (:team-name props))) @@ -375,7 +391,7 @@ (let [rows (th/db-exec! ["select * from audit_log"])] (t/is (= 0 (count rows)))))))) -(t/deftest archive-events-forwards-export-job-not-legacy-export +(t/deftest consumer-events-forwards-export-job-not-legacy-export (let [actor (th/create-profile* 1 {:is-active true}) team (th/create-team* 1 {:profile-id (:id actor)})] (with-redefs [cf/flags #{:admin-console}] @@ -387,7 +403,7 @@ :export-type "detach-libraries"}) (insert-audit-row! (:id actor) "export-binfile" {:file-id (uuid/next)}) - (th/run-task! :audit-log-archive-admin-console {}) + (th/run-task! :audit-log-consumer-admin-console {}) (let [[_ _ params] (:call-args @admin-console-mock) events (:events params) props (:props (first events))] @@ -397,13 +413,13 @@ (let [rows (th/db-exec! ["select * from audit_log"])] (t/is (= 0 (count rows)))))))) -(t/deftest archive-params-accept-empty-cron-props +(t/deftest consumer-params-accept-empty-cron-props ;; Cron always submits (or props {}) — system deps live on the ;; job-def cfg, not in params. Requiring them here blocks every tick. ;; Both schemas are closed: a stale manual invocation must fail ;; validation loudly instead of skipping silently. - (let [nexus-valid? (sm/validator archive/schema:audit-log-archive-nexus-params) - admin-console-valid? (sm/validator archive/schema:audit-log-archive-admin-console-params)] + (let [nexus-valid? (sm/validator consumer/schema:audit-log-consumer-nexus-params) + admin-console-valid? (sm/validator consumer/schema:audit-log-consumer-admin-console-params)] (t/is (true? (nexus-valid? {}))) (t/is (false? (nexus-valid? {:enabled true}))) (t/is (true? (nexus-valid? {:uri (u/uri archive-uri)}))) @@ -419,20 +435,20 @@ ;; (pattern from `jobs-metrics-test`), so assertions read exact values ;; instead of the shared test-system registry. `th/run-task!` would use ;; the latter and its counts would leak across tests. -(def ^:private archive-metric-ids - [:audit-archive-events :audit-archive-discarded]) +(def ^:private consumer-metric-ids + [:audit-consumer-events :audit-consumer-discarded]) -(defn- make-archive-metrics +(defn- make-consumer-metrics [] (ig/init-key :app.metrics/metrics - {:default (select-keys main/default-metrics archive-metric-ids)})) + {:default (select-keys main/default-metrics consumer-metric-ids)})) (defn- invoke-nexus! - "Invoke the nexus archive handler in-process with an isolated + "Invoke the nexus consumer handler in-process with an isolated metrics registry. The handler takes a nil context because there is no job row to describe (same as `th/run-task!`)." [metrics params] - (let [job-def (ig/init-key :app.loggers.audit.archive-task/nexus-job-def + (let [job-def (ig/init-key :app.loggers.audit.consumer-task/nexus-job-def {::db/pool (:app.db/pool th/*system*) ::setup/shared-keys {:admin-console "ac-key" :nexus "nexus-key"} ::http/client (:app.http.client/client th/*system*) @@ -440,19 +456,19 @@ ((::jobs/handler job-def) nil params))) (defn- invoke-admin-console! - "Invoke the admin-console archive handler in-process with an + "Invoke the admin-console consumer handler in-process with an isolated metrics registry. The admin-console client is the (nil in tests) system one: tests mock `app.nitrate/call`, so it is never dereferenced." [metrics] - (let [job-def (ig/init-key :app.loggers.audit.archive-task/admin-console-job-def + (let [job-def (ig/init-key :app.loggers.audit.consumer-task/admin-console-job-def {::db/pool (:app.db/pool th/*system*) :app.nitrate/client (:app.nitrate/client th/*system*) ::mtx/metrics metrics})] ((::jobs/handler job-def) nil {}))) (defn- invoke-both! - "Invoke both archive handlers sharing one isolated metrics + "Invoke both consumer handlers sharing one isolated metrics registry, Nexus first (the historical single-run order)." [metrics params] (invoke-nexus! metrics params) @@ -465,23 +481,23 @@ child (.labels ^Counter instance (into-array String labels))] (.get ^Counter$Child child))) -(t/deftest archive-records-metrics-on-success +(t/deftest consumer-records-metrics-on-success ;; Smoke test: the run records its domain metrics into the wired ;; registry (isolated per test, pattern from `jobs-metrics-test`). ;; Logging itself is not asserted: `dbg` summaries run below the ;; test log level and failure `warn`s only repeat the outcome the ;; metrics already prove. (let [prof (th/create-profile* 1 {:is-active true}) - metrics (make-archive-metrics)] + metrics (make-consumer-metrics)] (with-redefs [cf/flags #{:nexus :admin-console}] (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil} http-mock {:target 'app.http.client/req :return {:status 204}}] (insert-audit-row! (:id prof) "create-project") (insert-audit-row! (:id prof) "unrelated-event") (invoke-both! metrics {:uri archive-uri}) - (t/is (= 2.0 (counter-value metrics :audit-archive-events ["nexus" "archived"]))) - (t/is (= 1.0 (counter-value metrics :audit-archive-events ["admin-console" "archived"]))) - (t/is (= 1.0 (counter-value metrics :audit-archive-events ["admin-console" "skipped"]))) + (t/is (= 2.0 (counter-value metrics :audit-consumer-events ["nexus" "archived"]))) + (t/is (= 1.0 (counter-value metrics :audit-consumer-events ["admin-console" "archived"]))) + (t/is (= 1.0 (counter-value metrics :audit-consumer-events ["admin-console" "skipped"]))) (t/is (= 0 (pending-count))) (let [rows (th/db-exec! ["select * from audit_log"])] (t/is (= 0 (count rows)))))))) @@ -499,9 +515,9 @@ :props {} :context {} :created-at (ct/now) :tracked-at (ct/now)}) -(t/deftest archive-submit-creates-pendings-for-enabled-archivers +(t/deftest consumer-submit-creates-pendings-for-enabled-consumers ;; The producer hook: a row is born pending for the enabled - ;; archivers, in the same transaction as the row itself. Goes + ;; consumers, in the same transaction as the row itself. Goes ;; through submit* because the suite stubs submit itself (see ;; helpers/init-config). (let [prof (th/create-profile* 1 {:is-active true})] @@ -511,7 +527,7 @@ (t/is (= 1 (count (th/db-exec! ["select * from audit_log"])))) (t/is (= 2 (pending-count)))))) -(t/deftest archive-submit-single-flag-creates-one-pending +(t/deftest consumer-submit-single-flag-creates-one-pending (let [prof (th/create-profile* 1 {:is-active true})] (with-redefs [cf/flags #{:audit-log :nexus}] (audit/submit* {::db/pool (:app.db/pool th/*system*)} @@ -519,7 +535,7 @@ (t/is (= 1 (count (th/db-exec! ["select * from audit_log"])))) (t/is (= 1 (pending-count)))))) -(t/deftest archive-submit-rollback-keeps-no-pendings +(t/deftest consumer-submit-rollback-keeps-no-pendings ;; A rollback takes the row and its queue together: a row never ;; outlives its pendings, and no phantom pending pins nothing. (let [prof (th/create-profile* 1 {:is-active true})] @@ -533,13 +549,13 @@ (t/is (= 0 (count (th/db-exec! ["select * from audit_log"])))) (t/is (= 0 (pending-count)))))) -(t/deftest archive-keeps-row-until-last-archiver-done +(t/deftest consumer-keeps-row-until-last-consumer-done ;; Nexus refuses the chunk: the nexus job aborts (failed count ;; recorded, isolated registry) while the admin-console job still ;; drains its own queue. The row and the nexus pending stay; once ;; nexus accepts, everything is gone. (let [prof (th/create-profile* 1 {:is-active true}) - metrics (make-archive-metrics)] + metrics (make-consumer-metrics)] (with-redefs [cf/flags #{:nexus :admin-console}] (insert-audit-row! (:id prof) "create-project") (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil} @@ -548,10 +564,10 @@ (invoke-nexus! metrics {:uri archive-uri}) nil (catch clojure.lang.ExceptionInfo cause cause))] - (t/is (= :archive-chunk-refused (:code (ex-data err))))) + (t/is (= :consumer-chunk-refused (:code (ex-data err))))) (invoke-admin-console! metrics) (t/is (:called? @admin-console-mock)) - (t/is (= 1.0 (counter-value metrics :audit-archive-events ["nexus" "failed"]))) + (t/is (= 1.0 (counter-value metrics :audit-consumer-events ["nexus" "failed"]))) (t/is (= 1 (count (th/db-exec! ["select * from audit_log"])))) (t/is (= 1 (pending-count)))) (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil} @@ -560,9 +576,9 @@ (t/is (= 0 (count (th/db-exec! ["select * from audit_log"])))) (t/is (= 0 (pending-count))))))) -(t/deftest archive-sweeps-orphan-rows +(t/deftest consumer-sweeps-orphan-rows ;; A row with no pendings (old archived leftover, or born while no - ;; archiver was on) is deleted without any send. + ;; consumer was on) is deleted without any send. (let [prof (th/create-profile* 1 {:is-active true})] (with-redefs [cf/flags #{:nexus :admin-console}] (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil} @@ -584,23 +600,23 @@ (t/is (false? (:called? @http-mock))) (t/is (= 0 (count (th/db-exec! ["select * from audit_log"])))))))) -(t/deftest archive-nexus-transport-failure-records-failed +(t/deftest consumer-nexus-transport-failure-records-failed ;; A transport exception (timeout, DNS, refused) counts the events ;; as failed and aborts the nexus job: the chunk rolls back and ;; the rows stay for the next tick. (let [prof (th/create-profile* 1 {:is-active true}) - metrics (make-archive-metrics)] + metrics (make-consumer-metrics)] (with-redefs [cf/flags #{:nexus}] (with-mocks [http-mock {:target 'app.http.client/req :throw (ex-info "timeout" {})}] (insert-audit-row! (:id prof) "create-project") (t/is (thrown? clojure.lang.ExceptionInfo (invoke-nexus! metrics {:uri archive-uri}))) - (t/is (= 1.0 (counter-value metrics :audit-archive-events ["nexus" "failed"]))) + (t/is (= 1.0 (counter-value metrics :audit-consumer-events ["nexus" "failed"]))) (t/is (= 1 (count (th/db-exec! ["select * from audit_log"])))) (t/is (= 1 (pending-count))))))) -(t/deftest archive-sends-chunks-in-chronological-order +(t/deftest consumer-sends-chunks-in-chronological-order ;; Nexus numbers by arrival for order-sensitive consumers, so ;; multi-chunk drains must arrive oldest-first no matter how the ;; queue is scanned. @@ -617,26 +633,26 @@ (th/db-update! :audit-log {:created-at (.plusSeconds ^java.time.Instant base (long i))} {:id (:id row)}))) - (th/run-task! :audit-log-archive-nexus {:uri archive-uri}) + (th/run-task! :audit-log-consumer-nexus {:uri archive-uri}) (let [received (mapcat #(:events (transit/decode %)) @bodies) times (map :created-at received)] (t/is (= 130 (count received))) (t/is (= (sort times) times)) (t/is (= 0 (count (th/db-exec! ["select * from audit_log"]))))))))) -(t/deftest archive-reaps-disabled-archiver-pendings +(t/deftest consumer-reaps-disabled-consumer-pendings ;; R3 reap by its real path: an admin-console pending born while ;; admin-console was on is dropped once it is off; nexus still ;; archives its queue and the row is gone with nothing stuck. The ;; dropped pending counts as discarded (isolated registry). (let [prof (th/create-profile* 1 {:is-active true}) - metrics (make-archive-metrics)] + metrics (make-consumer-metrics)] (with-redefs [cf/flags #{:nexus}] (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}} admin-console-mock {:target 'app.nitrate/call :return nil}] (let [row (insert-audit-row! (:id prof) "create-project")] ;; Admin console was on at insert time: both pendings on purpose. - (th/db-exec! ["INSERT INTO audit_log_pending (audit_log_id, archiver) + (th/db-exec! ["INSERT INTO audit_log_pending (audit_log_id, consumer) VALUES (?::uuid, 2) ON CONFLICT DO NOTHING" (:id row)]) @@ -644,16 +660,16 @@ (invoke-nexus! metrics {:uri archive-uri}) (t/is (:called? @http-mock)) (t/is (false? (:called? @admin-console-mock))) - (t/is (= 1.0 (counter-value metrics :audit-archive-discarded ["reaped"]))) + (t/is (= 1.0 (counter-value metrics :audit-consumer-discarded ["reaped"]))) (t/is (= 0 (count (th/db-exec! ["select * from audit_log"])))) (t/is (= 0 (pending-count)))))))) -(t/deftest archive-sweeps-rows-born-with-no-archiver-on +(t/deftest consumer-sweeps-rows-born-with-no-consumer-on ;; Accepted R3 consequence by its real path (not a hand-made ;; orphan): ingested with no send flag on, the row is swept ;; unarchived once the task runs, with zero sends. (let [prof (th/create-profile* 1 {:is-active true}) - metrics (make-archive-metrics)] + metrics (make-consumer-metrics)] (with-redefs [cf/flags #{:audit-log}] (audit/submit* {::db/pool (:app.db/pool th/*system*)} (full-event (:id prof))) @@ -665,16 +681,16 @@ (t/is (false? (:called? @http-mock))) (t/is (= 0 (count (th/db-exec! ["select * from audit_log"])))) (t/is (= 0 (pending-count))) - (t/is (= 1.0 (counter-value metrics :audit-archive-discarded ["swept"]))) - (t/is (= 0.0 (counter-value metrics :audit-archive-events ["nexus" "archived"]))))))) + (t/is (= 1.0 (counter-value metrics :audit-consumer-discarded ["swept"]))) + (t/is (= 0.0 (counter-value metrics :audit-consumer-events ["nexus" "archived"]))))))) -(t/deftest archive-nexus-throw-still-drains-admin-console +(t/deftest consumer-nexus-throw-still-drains-admin-console ;; Consumer independence, now structural: a Nexus transport throw ;; aborts only the nexus job, and the admin-console job still drains ;; its own queue. Nexus keeps its pending (so the row stays); the ;; admin console completes its own queue. (let [prof (th/create-profile* 1 {:is-active true}) - metrics (make-archive-metrics)] + metrics (make-consumer-metrics)] (with-redefs [cf/flags #{:nexus :admin-console}] (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil} http-mock {:target 'app.http.client/req @@ -684,11 +700,11 @@ (invoke-nexus! metrics {:uri archive-uri}))) (invoke-admin-console! metrics) (t/is (:called? @admin-console-mock)) - (t/is (= 1.0 (counter-value metrics :audit-archive-events ["nexus" "failed"]))) + (t/is (= 1.0 (counter-value metrics :audit-consumer-events ["nexus" "failed"]))) (t/is (= 1 (count (th/db-exec! ["select * from audit_log"])))) (t/is (= 1 (pending-count))))))) -(t/deftest archive-reap-then-sweeps-without-sends +(t/deftest consumer-reap-then-sweeps-without-sends ;; End to end of the reap path: a row born with only an ;; admin-console pending is reaped once :admin-console goes off, ;; then swept with zero sends on either destination. @@ -699,24 +715,24 @@ (with-redefs [cf/flags #{:nexus}] (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}} admin-console-mock {:target 'app.nitrate/call :return nil}] - (th/run-task! :audit-log-archive-nexus {:uri archive-uri}) + (th/run-task! :audit-log-consumer-nexus {:uri archive-uri}) (t/is (false? (:called? @http-mock))) (t/is (false? (:called? @admin-console-mock))) (t/is (= 0 (count (th/db-exec! ["select * from audit_log"])))) (t/is (= 0 (pending-count))))))) -(t/deftest archive-fully-filtered-admin-console-chunk-counts-skipped +(t/deftest consumer-fully-filtered-admin-console-chunk-counts-skipped ;; An admin-console chunk with every event outside the allowlist ;; never calls ingest, but still counts its rows as skipped and ;; completes its queue. (let [prof (th/create-profile* 1 {:is-active true}) - metrics (make-archive-metrics)] + metrics (make-consumer-metrics)] (with-redefs [cf/flags #{:nexus :admin-console}] (with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil} http-mock {:target 'app.http.client/req :return {:status 204}}] (insert-audit-row! (:id prof) "unrelated-event") (invoke-both! metrics {:uri archive-uri}) (t/is (false? (:called? @admin-console-mock))) - (t/is (= 1.0 (counter-value metrics :audit-archive-events ["admin-console" "skipped"]))) + (t/is (= 1.0 (counter-value metrics :audit-consumer-events ["admin-console" "skipped"]))) (t/is (= 0 (count (th/db-exec! ["select * from audit_log"])))) (t/is (= 0 (pending-count))))))) diff --git a/backend/test/backend_tests/rpc_audit_test.clj b/backend/test/backend_tests/rpc_audit_test.clj index 6080c483cf..8ae01500ad 100644 --- a/backend/test/backend_tests/rpc_audit_test.clj +++ b/backend/test/backend_tests/rpc_audit_test.clj @@ -139,7 +139,7 @@ (t/is (= (:id prof) (:profile-id row))) (t/is (= "navigate" (:name row))) (t/is (= "frontend" (:source row))) - ;; The row is born pending for the enabled archiver. + ;; The row is born pending for the enabled consumer. (t/is (= 1 (count (th/db-exec! ["select * from audit_log_pending"])))))))) (t/deftest push-events-2 @@ -274,7 +274,7 @@ (let [day-now (ct/truncate (ct/now) :days)] (t/is (= day-now (:created-at row2))) (t/is (= day-now (:tracked-at row2)))) - ;; Both rows are born pending for the enabled archivers. + ;; Both rows are born pending for the enabled consumers. (t/is (= 4 (count (th/db-exec! ["select * from audit_log_pending"]))))))))) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; diff --git a/backend/test/backend_tests/tasks_telemetry_test.clj b/backend/test/backend_tests/tasks_telemetry_test.clj index 4585de8c98..2bbdd7c371 100644 --- a/backend/test/backend_tests/tasks_telemetry_test.clj +++ b/backend/test/backend_tests/tasks_telemetry_test.clj @@ -11,7 +11,7 @@ [app.config :as cf] [app.db :as db] [app.loggers.audit :as audit] - [app.loggers.audit.archive-task :as at] + [app.loggers.audit.consumer-task :as at] [app.tasks.telemetry :as telemetry] [app.util.json :as json] [backend-tests.helpers :as th] @@ -1006,14 +1006,14 @@ ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; (defn- insert-pending-pair! - "Queue the row for both archivers (from the archiver constants, not + "Queue the row for both consumers (from the consumer constants, not literals): the delete paths must face pending-ful rows." [row-id] - (doseq [archiver [at/archiver-nexus at/archiver-admin-console]] - (th/db-exec! ["INSERT INTO audit_log_pending (audit_log_id, archiver) + (doseq [consumer [at/consumer-nexus at/consumer-admin-console]] + (th/db-exec! ["INSERT INTO audit_log_pending (audit_log_id, consumer) VALUES (?::uuid, ?::smallint) ON CONFLICT DO NOTHING" - row-id archiver]))) + row-id consumer]))) (defn- count-pendings [] (-> (th/db-exec-one! ["SELECT count(*) AS cnt FROM audit_log_pending"]) @@ -1021,7 +1021,7 @@ long)) (t/deftest test-delete-sent-events-removes-pendings-first - ;; Regression: telemetry rows carry pendings for both archivers, so + ;; Regression: telemetry rows carry pendings for both consumers, so ;; deleting the row must delete its pendings first, or the FK ;; rejects the delete and the batch is sent again next tick. (let [row (insert-telemetry-row! "navigate")]