mirror of
https://github.com/penpot/penpot.git
synced 2026-11-21 09:42:01 +00:00
✨ Deliver telemetry audit events independently (#13230)
* ✨ Ship telemetry shadows with a cursor consumer Send anonymized audit events through a dedicated telemetry consumer, including when Nexus or admin-console also persist full events. Keep the legacy payload and cadence of 10,000 events per chunk and three hours between passes. Nexus skips shadows to avoid archiving duplicates. Move shadow deletion to the audit GC. Delete full rows and shadows along independent cursor paths, and expire shadows after seven days even when they have not been sent. Run the deletion paths in parallel with bounded batches, separate time budgets and independent failure handling. Replace the shared consumer catalog with transport definitions and one generic pass runner. Keep the legacy telemetry report and newsletter in the scheduled task, and update the tests, memories and rollout runbooks. Addresses F1 of the audit-log consumer review. AI-assisted-by: deepseek-v4-flash AI-assisted-by: gpt-6.1-terra * 🐛 Address audit log consumer review findings Enable audit log GC by default and document how to pause it during rollout or before registering a consumer that needs a complete replay. Run GC paths in order on the runner thread. Count each committed batch and preserve partial totals on ordinary failures while propagating interruption. Skip full-row payloads in telemetry queries and decoding. Keep the age limit for local retention only, and document the shared pass contract. AI-assisted-by: gpt-6.2-sol * 🐛 Close audit log consumer review findings Verify telemetry counts processed and skipped events in a mixed batch. Clarify how to pause GC, the scope of the replay race, and what the retention warning counts. Remove private Nexus setup from the public upgrade runbook. AI-assisted-by: gpt-6.2-sol * 📚 Simplify the audit log guide Consolidate the guidance on event trust, privacy and delivery. Keep examples that explain why the design works and link to code and runbooks for details. Clarify replay limits and distinguish age-based deletion from missed delivery. AI-assisted-by: gpt-6.2-sol
This commit is contained in:
parent
c9c9a574a0
commit
e0194ce60a
@ -1,367 +1,134 @@
|
||||
# Audit Log and event processing
|
||||
|
||||
Complete guide to the Penpot event system: what gets recorded, where each piece lives, and how events travel from the moment they happen until they are archived. Start with the overview (sections 1-3); the deeper technical detail comes later (section 8 on). If you only want the idea, the first three sections are enough.
|
||||
The audit log records what Penpot observed: who made a request, what they asked to do, and when it happened. It also carries browser events for usage analytics and crash reporting. Those events are useful, but a browser can close or lose its connection before sending them. Do not use their absence as proof that something did not happen.
|
||||
|
||||
> Note about this memory: unlike other Serena memories (dense notes written for LLM agents, see `mem:memory-maintenance`), this document is also written for human developers who need full context. That is why it uses plain language and examples. It replaces the old `mem:backend/audit-log`, which is now just a redirect stub.
|
||||
## 1. Sources and trust
|
||||
|
||||
## 1. What it is (2-minute overview)
|
||||
| Channel | Who sets the actor | What the event tells you |
|
||||
|---|---|---|
|
||||
| Penpot backend RPC | The server uses the caller, an explicit command override, or `uuid/zero` when no actor is known. | The server ran the command. Check the event's outcome before treating it as a change of state. |
|
||||
| Public `push-audit-events` | Penpot uses the session, never an actor supplied in the event. | The Penpot or admin-console browser reported an action. Delivery is best-effort. |
|
||||
| Management `push-audit-events` | The authenticated admin-console service supplies the actor. | A trusted service reported an action. Authentication identifies the service, not whether its business action and the event committed together. |
|
||||
|
||||
The *audit log* answers "who did what, when, and from where". Every important user action becomes an **event**: a small object with a name, a type, who did it, from which IP, plus action data (`props`) and metadata (`context`).
|
||||
The management channel uses `POST /api/management/methods/push-audit-events` with `x-shared-key: "admin-console <key>"` and events carrying `{name, profileId, props?, context?}` in JSON. It uses server request time, not the public endpoint's client timestamp. The public channel uses `POST /api/main/methods/push-audit-events` with a user session and events carrying `{name, type, timestamp, props, context?}`. Both browsers use the same public RPC. The Nitrate senders live in the private Nitrate repo; these are the contracts Penpot receives.
|
||||
|
||||
The system has:
|
||||
|
||||
- **Three producers** (they create events): the Penpot backend (RPC layer), the Penpot frontend (browser app), and the Nitrate admin-console, which sends both server-side and client-side events.
|
||||
- **One central buffer**: the Postgres `audit_log` table in Penpot. It is temporary: it holds events for minutes/hours until they are archived, then they are deleted.
|
||||
- **Consumers** (they read events): webhooks, error reporters, telemetry, the external archives in Nexus and in the admin console.
|
||||
- **Two long-term archives**: **Nexus**, a separate service that stores events for the long term, and the **admin console** (Nitrate), which ingests its own allowlisted events through a second archive job (see section 12).
|
||||
|
||||
```
|
||||
Penpot frontend ─ POST push-audit-events (public) ─────────▶┐
|
||||
Nitrate admin-console (browser) ─ POST push-audit-events ──▶│
|
||||
Nitrate admin-console (server) ─ POST push-audit-events ──▶│ ─▶ audit_log table (buffer) ──┬─ nexus consumer ───▶ Nexus
|
||||
Penpot backend (RPC wrap-audit) ─ submit ───────────────────┘ │ ├─ admin-console ───▶ admin console
|
||||
│ └─ gc ───────────▶ delete drained rows
|
||||
│ ├──▶ webhooks
|
||||
│ ├──▶ error reporters
|
||||
└──▶ telemetry (shadow rows)
|
||||
```
|
||||
|
||||
The Penpot backend is the **center**: it checks, completes, and stores every event, no matter where it comes from. Nobody writes to the database directly; everything goes through the backend.
|
||||
|
||||
This document is the reference for all flows. One thing to keep in mind while reading: the Nitrate admin-console (both its server and its browser UI) is a **private** application, not a public client. Its events describe private SaaS management actions (organizations, subscriptions, SSO), and its server channel uses a trusted shared key instead of a user session.
|
||||
|
||||
## 2. The three kinds of events
|
||||
|
||||
Not all events are worth the same. This is the most important distinction in this document:
|
||||
|
||||
| Aspect | Backend events | Penpot frontend events | Nitrate admin-console events |
|
||||
|---|---|---|---|
|
||||
| What they describe | Real actions run on the server (create file, login, invite to team...) | Things that happen in the Penpot browser app (navigate, edit, JS errors, performance) | Organization and subscription actions in the admin-console (create/rename/delete organization, members, SSO settings, checkout, onboarding) |
|
||||
| Trust level | **High**: created inside the same request that runs the action | **Best-effort**: some loss is expected (tab closed, offline, full buffer); they show **usage**, they are not strict proof | **Mixed**: server-side events are high trust (like backend); browser events are best-effort (like frontend) |
|
||||
| Who sets `profile-id` | The server, from the logged-in session (never trusts the client) | The server too: it ignores what the client says and uses the session | Server channel: the trusted admin-console server sends it explicitly. Browser channel: the Penpot server uses the session |
|
||||
| Who sets the time | Server time (`created-at`); the source of truth | Client proposes `tracked-at`, but the server fixes it if it is in the future or more than 1h late | Same rule: server stamps `created-at`; client `tracked-at` is corrected |
|
||||
| Used for | Audit, webhooks, compliance, abuse checks | Product analytics, usage counters, crash reports | Organization audit trail (admin-console e2e spec), subscription metrics |
|
||||
|
||||
Rule of thumb: if you need to know for sure "did this user do X?", look at `source = backend`. If you want to know "how do people use the app?", also look at `source = frontend`.
|
||||
|
||||
## 3. The `audit_log` table: a buffer, not a store
|
||||
|
||||
The Penpot `audit_log` table **is not the final destination**. It is a waiting tray: events sit here until every active consumer (Nexus, admin console) confirms them, and whoever finishes last deletes the row.
|
||||
|
||||
Main columns (migration `backend/src/app/migrations/sql/0054-add-audit-log-table.sql` and later ones):
|
||||
|
||||
- `id` uuid PK (`gen_random_uuid()` today; the PK used to be `(created_at, profile_id)`).
|
||||
- `name` / `type` text NOT NULL (for example `name = "create-file"`, `type = "action"`).
|
||||
- `created_at` timestamptz NOT NULL default `now()`: **server time**, the source of truth.
|
||||
- `tracked_at` timestamptz default `now()`: what the client claimed; fixed on ingest.
|
||||
- `profile_id` uuid NOT NULL: who did it.
|
||||
- `source` text, NOT NULL (from `0159`, which backfills what is left to `backend`): `backend` / `frontend` for full rows, `telemetry:backend` / `telemetry:frontend` for anonymized copies (see section 10).
|
||||
- `ip_addr` inet: real IP (or `0.0.0.0` on telemetry rows).
|
||||
- `props` / `context` jsonb holding transit-encoded maps: action data and metadata (versions, browser, request-id...).
|
||||
- `seq` bigint (`bigserial`): the insertion order, from a sequence (no WAL cost).
|
||||
- `xid` xid8: the transaction that wrote the row (`pg_current_xact_id()` as the column default). Together with `seq` it is the position a consumer reads from; `xid8` is 64 bits and does not wrap (the row `xmin` does).
|
||||
- The legacy `archived_at` and the indexes that served the old archiver stay until a later cleanup; the new code ignores them. The drain reads no pending queue: each consumer keeps its position in `audit_log_consumer_cursor` (migration `0159-consumer-generalization`, consumer `1` = Nexus, `2` = admin console; ids are never reused, and `3` is reserved for the team-storage consumer that ships in a follow-up). The pending queue never shipped: this release does not create `audit_log_pending`; the migration drops the table if a pre-release database has it. The producer only inserts the row; it does not know the consumers. A consumer with no row registers its cursor at `(0, 0)` when its component starts (`ensure-cursor`), before its first pass, so the first deploy **replays** the buffer (the legacy archiver and its GC are stopped and the already-archived rows are deleted before deploying), instead of skipping the unarchived backlog; enabling a consumer later starts from whatever the buffer holds; with no consumer at all nothing is written, so there is no backlog to replay. `ensure-cursor` starts the cursor below any real position; the cut is applied per chunk when reading, so the first pass drains the buffer that exists and never steps over a row whose transaction was still open.
|
||||
|
||||
Position and cut: a consumer reads rows with `(xid, seq) > cursor` and `xid <= pg_snapshot_xmin(pg_current_snapshot())` (the oldest transaction still in flight), in `(xid, seq)` order. The cut is exact: a row of an open transaction is invisible until it commits, and the cursor never steps over it (a `seq`-only cursor lost those rows). A transaction that inserts many rows keeps them together: `seq` orders them inside the same `xid`. The cursor points at the last row the consumer processed, so the GC deletes `<= cursor`. The cut is conservative beyond `audit_log`: `pg_snapshot_xmin` is the oldest transaction of the whole database, so any long write transaction (a binfile import runs in one for up to 30 minutes) pauses every consumer and the GC until it commits, and a slow destination pauses them the same way because the chunk sends inside its transaction and the `FOR UPDATE` on the cursor row assigns an xid; nothing is lost or skipped, and `penpot_audit_consumer_lag_events` shows it.
|
||||
|
||||
Indexes:
|
||||
|
||||
- unique `audit_log (xid, seq)` (`audit_log__xid__seq__idx`, migration `0159-consumer-generalization`): serves the range and the order of the position scan (both predicates are `Index Cond`; the chunk also projects columns, so it is not an Index Only Scan).
|
||||
- `audit_log_consumer_cursor` PK `(consumer)`: one row per consumer, locked `FOR UPDATE` during the chunk.
|
||||
- `(source, created_at)` (`audit_log__source__created_at__idx`): narrows the telemetry scan by source; telemetry then sorts by `created_at`. The `(created_at, id)` index belonged to the pending queue and never shipped, so this release does not build it (the same path 2.18 had).
|
||||
- The legacy `archived_at` and the indexes that served it (`audit_log__created_at__idx`, `audit_log__archived_at__idx`, `audit_log_id_archived_at_idx`) stay: `0159` does not drop them. They are dead after the rollout and a later release removes them; the column/index drops are catalog operations, not a rewrite.
|
||||
|
||||
Life of a row: `INSERT` (one row, no queue) → each consumer's pass reads its own 128-row chunks from its cursor up to the frontier it took at its start, sends them and advances the cursor in the same transaction (a refusal rolls both back) → the GC job (`:audit-log-gc`) deletes the rows up to and including the oldest cursor in `audit_log_consumer_cursor`, in batches, skipping telemetry shadows (the telemetry task owns them: it deletes them after sending and purges them after 7 days). A consumer that is off keeps its cursor row and holds the GC at that position; when enabled again it rejoins from its position. To stop waiting for it, its cursor row must be deleted (retirement; no procedure ships yet). A consumer registers its cursor at `(0, 0)` when its component starts (`ensure-cursor`), before its first pass, below any real position, so the first deploy replays the unarchived backlog and enabling a consumer later starts from whatever the buffer holds; with no consumer at all nothing is written, so there is no backlog to replay. The GC treats a missing row as position 0 and deletes nothing for that run; with no cursor at all it deletes nothing (the `:audit-log-gc` flag is the switch).
|
||||
|
||||
## 4. Anatomy of an event
|
||||
|
||||
Every event ends up with this shape (schema `schema:event` in `backend/src/app/loggers/audit.clj`):
|
||||
|
||||
- `name`: what happened (`"create-file"`, `"navigate"`, `"create-organization"`...). For backend, it defaults to the RPC command name (prefixed with `<module>-` outside `main`).
|
||||
- `type`: category (`"action"`, `"identify"`, `"trigger"`). Most are `"action"`.
|
||||
- `profile-id`: **who** did it (uuid). See the invariant in section 5.
|
||||
- `ip-addr`: request IP (or `"0.0.0.0"` on telemetry).
|
||||
- `props`: action data (team/project/file ids, visited route...). **Never secrets** (see `clean-props`).
|
||||
- `context`: metadata about how/when it was captured (backend/frontend version, user-agent, locale, browser, `request-id`, `initiator`, `event-origin`...). See section 8 for the full field guide, including `initiator` (which app started it: `"app"` means Penpot itself).
|
||||
- `source`: `backend`, `frontend`, `telemetry:backend`, `telemetry:frontend`. Nitrate admin-console server events land as `backend`; its browser events land as `frontend` (see section 7).
|
||||
- `tracked-at` / `created-at`: claimed time and server time.
|
||||
- Memory-only (never stored): `::webhooks/event?`, `::webhooks/batch-key`, `::webhooks/batch-timeout` for the webhook fan-out.
|
||||
|
||||
Backend `create-file` example (simplified):
|
||||
`source` distinguishes channels: management events are `backend`, browser events are `frontend`, and telemetry copies use `telemetry:backend` or `telemetry:frontend`. It is not an outcome. For example, `update-file` can record a rejected save without a change to the file.
|
||||
|
||||
```clojure
|
||||
;; Selected fields: profile-id names the actor, props.id names the new file.
|
||||
{:name "create-file" :type "action" :source "backend"
|
||||
:profile-id #uuid "…" :ip-addr "203.0.113.7"
|
||||
:props {:project-id #uuid "…" :name "My design"}
|
||||
:context {:version "2.x" :initiator "app" :request-id "…"
|
||||
:client-version "2.x" :client-user-agent "Mozilla/…"}
|
||||
:created-at #inst "…" :tracked-at #inst "…"}
|
||||
:profile-id actor-id
|
||||
:props {:id file-id :project-id project-id :team-id team-id}
|
||||
:context {:initiator "app" :request-id request-id}}
|
||||
```
|
||||
|
||||
Frontend `navigate` example:
|
||||
|
||||
```clojure
|
||||
{:name "navigate" :type "action" :source "frontend"
|
||||
:profile-id #uuid "…" :ip-addr "203.0.113.7"
|
||||
:props {:route "dashboard" :team-id #uuid "…"}
|
||||
:context {:version "2.x" :locale "en" :browser "Chrome" :os "Linux"
|
||||
:event-origin "dashboard" :session-id "…" :session #inst "…"}
|
||||
:created-at #inst "…" :tracked-at #inst "…"}
|
||||
```
|
||||
|
||||
## 5. Backend events: the happy path
|
||||
|
||||
Most backend events need **no manual code**. The `wrap-audit` wrapper in `backend/src/app/rpc.clj` runs after every RPC command when `:webhooks`, the boot-computed persistence value, or `:telemetry` is on (unless the command sets `::audit/skip`). It builds the event with `prepare-rpc-event` and passes it to `audit/submit`.
|
||||
|
||||
Each command can customize through result metadata (`rph/with-meta`):
|
||||
|
||||
- `::audit/replace-props`: replaces props fully (auth commands use `profile->props` so a register event carries the profile, not the password).
|
||||
- `::audit/props`: merges extras over the request params.
|
||||
- `::audit/context`, `::audit/profile-id`, `::audit/name`, `::audit/type`: override the defaults.
|
||||
- `clean-props` always strips nils, namespaced keys, and `:session-id/:password/:old-password/:token/:client-secret` as a last line of defense. **Never put secrets in props.**
|
||||
|
||||
Critical `profile-id` INVARIANT: the event belongs to **whoever made the request**. It resolves as `::audit/profile-id` metadata → `::rpc/profile-id` → `uuid/zero`. It is **never** taken from the response: some results carry someone else's `:profile-id` (for example `get-error-report` returns the report with its content merged; `verify-token` on an invitation returns the inviter), and using it would blame the wrong person. Commands with `::rpc/auth false` (`login-*`, `register-profile`, `create-demo-profile`, `verify-token`, `prepare-register-profile`) have no caller to fall back on, so they **must** set `::audit/profile-id` in the metadata or the event lands on `uuid/zero`.
|
||||
|
||||
The `::audit/profile-id` override goes through `coerce-profile-id`: a string is parsed, anything that cannot become a uuid is dropped (with a warning) and the event falls back to the caller. Do not pass the value raw: `schema:event` needs a uuid and `submit*` **propagates** the validation error, and `wrap-audit` runs after the command's transaction commits (moving it inside is project P8 in `TODO.md`), so a non-uuid override would fail the response over an already committed action instead of only losing the row. Hand-built events with `event-from-rpc-params` + `submit` (clone template, accept team invitation, `management/nitrate` push-audit-events) do **not** go through that coercion: they must already carry a uuid.
|
||||
|
||||
### Team id and entity ids (projection data)
|
||||
|
||||
The event stream is meant to let a consumer (Nexus) rebuild what it needs **without joining production tables**, so events carry the ids a projection keys on:
|
||||
|
||||
- **Team id enrichment.** After `clean-props`, `prepare-rpc-event` calls `enrich-team-id`: when props have no `:team-id` but carry a `:project-id` or `:file-id`, it resolves the team with a local query and adds `:team-id`. It never resolves from `:id` (read commands put the entity id there, and that would add a query to every read), and it skips read events entirely (`get-*` names plus `has-file-libraries`: their events carry no consumer state). Commands that already loaded the team add `:team-id` themselves in `::audit/props` (`delete-file`, comments, media upload, `delete/update-webhook`, `clone-template`, ...), so the resolver does not run.
|
||||
- **Membership.** `add-profile-to-team!` emits `add-team-member` (props `:team-id`, `:member-id`, `:role`, `:is-owner`, `:is-admin`, `:can-edit`) whenever it actually inserts a row, only when full rows are persisted, inside the caller's transaction. It covers the flows that add a member with no event of their own (`create-team` owner, invitee added directly when `:email-verification` is off, approved join-request, duplicated team). With `on-conflict-do-nothing?` a conflict returns nil and emits nothing, so an existing membership is reported once. The event belongs to the actor: the two flows where the owner adds somebody else (`create-invitation`, `add-member-to-team`) pass `:actor-id` through a separate `event-opts` arg (never in `params`, which is inserted as-is); everywhere else actor and member coincide. `:role` is derived in the emitter from the flags (inverse of `permissions-for-role`: owner > admin > editor > viewer) because the table has no role column; request context travels in `event-opts` as `:rpc-params`.
|
||||
- **Created entity ids.** `create-project` and `create-file` add the new row's `:id` (and `:project-id`/`:team-id`) to `::audit/props`, so a projection can key it.
|
||||
- **Change ids.** `update-file` carries the id of the change row (xlog) it created in `:file-change-id`: the event says that a file was saved, but what changed lives in that row. A consumer that needs it must read it with `::db/remove-deleted false` (the row is born with its `deleted_at` in the future). A save writes two change rows (the change itself, without a label, and the snapshot of the previous state, with one) and the event points at the change one. A save that ends fully rejected (`:rejected` and no kept prefix) writes no row and the prop is absent: an id that resolves to nothing cannot be told apart from one the GC already purged, so do not expect it always. The event also carries `:outcome`: `saved` (every change written), `partial` (the accepted prefix written and the rest reported in the response, `:file-change-id` present) or `rejected` (nothing written, `:file-change-id` absent, `:modified-at` set to the row's persisted value instead of the request time). A consumer must look at `:outcome` before treating `update-file` as a state change.
|
||||
- **Bulk operations.** `clone-template`, `import-binfile` and `duplicate-project` create entities through internal calls that emit nothing on their own; they emit a `create-file` (and `create-project`) per entity (only when full rows are persisted). The import paths go through `audit/submit-file-created-event`, which reads the light columns (`id`, `name`, `project-id`, timestamps, `revn`) back from the row — the file must really exist, a phantom id emits nothing. `duplicate-project` already holds the rows and uses `audit/submit-create-event` directly. Both binfile versions return the same shape from `import-files!` (`{:file-ids [...]}`).
|
||||
- **Deletes.** `delete-team` still returns nil but carries the deleted row (`:id`, `:name`, `:created-at`, `:modified-at`, `:deleted-at`) in `::audit/props`, like `delete-project`. `delete-profile` carries `:id`, `:email`, `:fullname`, `:created-at`, `:modified-at` and `:deleted-at` (without this its props are empty: every param is namespaced and `clean-props` drops them). `create-team` adds `:created-at` and `:modified-at`. `delete-team-member` needs nothing: the delete is physical and the params already carry `:team-id`/`:member-id`.
|
||||
- **Moves.** `move-files` and `move-project` carry `:source-team-id` and `:dest-team-id`. A `move-files` that mixes files from more than one source team is refused (`:cant-move-from-multiple-teams`), so the single origin in the props is always correct.
|
||||
- **Hard deletes.** `objects-gc` emits one `hard-delete-objects` event per purged chunk, carrying only that chunk's entity key (`:team-ids`, `:project-ids` or `:file-ids`; one chunk transaction purges a single entity, so one event can never span sections), with context `{:triggered-by "objects-gc"}`, only when full rows are persisted. No extra team-id resolution: the event carries the key of the purge, nothing else. No other GC path emits (profiles, fonts, thumbnails); those purges stay silent. `file-gc` is the exception: it emits `file-gc` (`action`, `triggered-by: cron-task`) with the cleaned `:file-id` and its `:team-id`, so a consumer can key on it without joining production tables.
|
||||
- **SREPL deletes.** `delete-profile!` carries the subject id in props (it used to emit nothing usable); the event profile stays zero because SREPL has no actor. `delete-team!` props already carried the id, and `delete-file!` adds its `:team-id` so a consumer can key on it. All travel as actions with a `triggered-by: srepl` context.
|
||||
- **SREPL restores.** `restore-profile!` carries the full row (with id) in props with a zero event profile, plus `:team-ids` with the teams its cascade took (only those: a team the owner deleted earlier stays deleted); `restore-team!` carries its `:team-id` and `restore-project!` its id. The consumer resolves the subject from props by a uniform rule, with no SREPL special case.
|
||||
- **Cascades.** The `:delete-object` worker task emits one `delete-object-with-cascade` event per execution, type `action`, with `:object`/`:id`/`:deleted-at`, flat `:team-ids` resolved at emission plus the affected ids nested under `:cascade` (`:team-ids`, `:project-ids`, `:file-ids`), in the same transaction. The handler submits unconditionally (`submit` fans out per flag, so no persistence re-check); the `audit_log` row itself is still only written when full rows are persisted. Direct calls (objects-gc) and snapshot cascades stay silent on purpose.
|
||||
- **Ownership.** `create-team-with-invitations` carries the new team `:id` like `create-team`; `update-team-member-role` names the demoted member in `:prev-owner-id` on owner transfers.
|
||||
|
||||
Entry points:
|
||||
|
||||
- `submit`: the normal one (fills defaults, validates, runs inside `tx-run!`). A failure **propagates** (a lost event is a stale consumer, and hiding it made that invisible), and what it costs depends on where the caller runs: `wrap-audit` runs **after** the command's transaction commits, so a failure there cannot undo the action and the client gets an error over already committed work (moving the event inside the transaction is project P8 in `TODO.md`); a `submit` made inside the command's own transaction (the membership emitter, the `:delete-object` handler) throws through it and rolls the action back with the event.
|
||||
- `insert`: low level, for CLI helpers and the webhook subsystem; direct write, no webhook/telemetry fan-out, silent unless full rows are persisted.
|
||||
- Boot emits `trigger/instance-start` from `setup/props` so every restart stays visible.
|
||||
|
||||
Cases already covered by tests, do not break: `accept-organization-invitation` tells its two flows apart **with props, not with origin**: `:organization-member-add-source` is `"direct-organization-invitation"` or `"team-invitation"`, and `:belongs-to-team-on-add` repeats it as a boolean. Do not add a third prop. The `context` `:event-origin` belongs to the browser (`make-data-event` sets it and it is on the frontend allowlist; `safe-backend-context-keys` has no `:event-origin`, so a backend event has nowhere to put one).
|
||||
|
||||
## 6. Penpot frontend events: what is collected, with what guarantees
|
||||
|
||||
The browser **never writes to the table**; it POSTs transit batches to `push-audit-events` (`backend/src/app/rpc/commands/audit.clj`), which stamps server `id`, session `profile-id`, request IP, server `created-at`, and server-side `:initiator` (from the `x-client` header, see section 8; anything the client sends as `initiator` is overwritten), and distrusts the client clock (future or >1h-late `tracked-at` is reset, the original kept in `context.original-tracked-at`). Without persistence, `:telemetry` or `:error-reporting`, or on a read-only pool, the endpoint is a no-op. With `:error-reporting` alone the batch is accepted only to feed the reporters: no full row and no telemetry shadow is written.
|
||||
|
||||
Valid types (`schema:frontend-event`): `"action"`, `"identify"`, `"trigger"`. Names must match `#"[\d\w-]{1,50}"` (max 250). `props` is a free map with no schema: **tests are the only guard of the contract** (see section 13).
|
||||
|
||||
What they are for: usage metrics (navigation, editing), workspace file stats, Nitrate membership changes, crash reports (`unhandled-exception` / `exception-page`). **Not proof for audit**: collection is best-effort and includes `PerformanceObserver` noise (`performance-*`, `user-input`, `long-task`). So backend tests must never assert exact frontend event counts.
|
||||
|
||||
How the browser collects (`frontend/src/app/main/data/event.cljs`), most to least important:
|
||||
|
||||
1. **On-demand start**: `initialize` asks the backend with `get-environment-data` (old `get-enabled-flags` is deprecated) and only starts on the derived `:audit-log` wire signal or `:telemetry` (see section 13). On RPC failure it still starts in telemetry mode (the backend will reject if needed).
|
||||
2. **Potok translation**: `make-proto-event` (events with the `Event` protocol) and `make-data-event` (data-events with `::name`) turn the action into `{type, name, props, context}`. `simplify-props` flattens complex values (maps → `:placeholder/map`, and so on) and drops namespaced keys and nils. Context gains `:event-origin`, `:event-namespace`, `:event-symbol`.
|
||||
3. **Capped buffer**: queue of max 1024 events (`append-to-buffer` drops beyond that); every 2s debounce (or on logout, or `::force-persist`) a chunk of max 100 is taken, filtered to the current profile, and sent fire-and-forget to `api/main/methods/push-audit-events` as transit. On confirm, the chunk leaves the buffer (`::chunk-persisted`).
|
||||
4. **Performance noise**: `PerformanceObserver` (`event`/`longtask`) plus blocking measurement with `scheduler.postTask` create `trigger` events (`performance-blocking-event`, `user-input`, `long-task`) with 1s debounce and 1s threshold. Info only, not audit.
|
||||
5. **`skip-audit?`**: UI flag (for example resuming a pending dashboard action after Nitrate SSO, or moving a team into an organization) so one user gesture is not counted twice.
|
||||
|
||||
## 7. Nitrate admin-console events: the third producer
|
||||
|
||||
The admin-console is the third producer. It sends organization and subscription events through **two channels** that look the same once stored, but have different trust levels. All the sending code (server and browser helpers) lives in the private Nitrate repo: it is not Penpot code and no file paths are cited here. What this section documents is only the Penpot-visible side: endpoints, payloads, and which event names arrive.
|
||||
|
||||
### 7a. Server-side channel (high trust, like backend events)
|
||||
|
||||
Code: the admin-console server has a helper that POSTs to the Penpot **management** API:
|
||||
|
||||
```
|
||||
POST api/management/methods/push-audit-events
|
||||
headers: x-shared-key: "admin-console <key>", Cookie: <user cookies>
|
||||
body: {events: [{name, props, profileId, context?}]}
|
||||
```
|
||||
|
||||
Auth happens at the HTTP layer (`backend/src/app/http/middleware.clj` `wrap-shared-key-auth`): the `x-shared-key` header carries `"<key-id> <secret>"` (`admin-console` here), checked in constant time against the configured keys; the key-id lands in the event context as `:initiator`. The call goes server-to-server (ky client, internal URI, user cookies forwarded).
|
||||
|
||||
Who uses it:
|
||||
|
||||
- Organization management (best-effort wrapper): `create-organization`, `rename-organization`, `delete-organization`, member add/remove, and more. It optionally passes the UI origin in the context.
|
||||
- Subscriptions: `create-subscription`, renewals, cancellations, and more, including deployment lookup (saas/selfhost).
|
||||
|
||||
Failure policy: every send is best-effort. A failure never rolls back the business action; it is only recorded for debugging.
|
||||
|
||||
Backend side (`backend/src/app/rpc/management/nitrate.clj` `::push-audit-events`): `::audit/skip`, `::rpc/auth false` (trust comes from the shared key, not from a user session; only the `admin-console` key id is allowed). It builds context with `audit/prepare-context-from-request` + `request-id`, takes the given `profile-id` as-is (no coercion here, must already be a uuid), sets `tracked-at = request-at`, defaults `type` to `"action"`, and calls `audit/submit` (which defaults `source` to `"backend"`). So these rows land as **`source = backend`**: they are audited like backend events because a trusted service set the author.
|
||||
|
||||
### 7b. Browser channel (best-effort, like frontend events)
|
||||
|
||||
Code: the admin-console browser has a helper that POSTs to the Penpot **public** API:
|
||||
|
||||
```
|
||||
POST api/main/methods/push-audit-events
|
||||
credentials: include, keepalive: true
|
||||
headers: x-client: "penpot-admin-console/<version>"
|
||||
body: {events: [{name, type: "action", timestamp, props, context: {eventOrigin?}}]}
|
||||
```
|
||||
|
||||
Dozens of helpers cover the admin-console UI: `create-organization`, `delete-organization-member`, `create-subscription` / `create-trial-subscription`, `open-subscription-modal`, `start-nitrate-checkout`, `open-current-subscription`, SSO settings flows (`open/apply/success/fail/discard-organization-sso-changes`, activate/deactivate modals), `change-organization-advanced-permission`, `onboarding-step` / `onboarding-finish`, `close-subscription-modal`. These helpers live in the private Nitrate repo: they are not Penpot code, you will not find them in this repository. Only the event names above (what arrives at Penpot) are listed here on purpose.
|
||||
|
||||
These rows land as **`source = frontend`** with the session `profile-id`: same guarantees as Penpot frontend events (best-effort, usage analytics). Every browser helper sends an `eventOrigin` in the context: plain `"admin-console"`, with a suffix only when the same event is emitted from two places in the app (for example the SSO discard event distinguishes the config form from the unsaved-changes modal).
|
||||
|
||||
The Nitrate repo also has a shared, generic push client for the management endpoint (not tied to the admin-console server): any other Nitrate-side service that needs to record something in the Penpot audit log can reuse it instead of writing its own HTTP call. Same endpoint, same payload shape as the server channel above.
|
||||
|
||||
## 8. How the frontend helps the backend audit: HTTP headers
|
||||
|
||||
Besides explicit events, **every RPC request from the frontend carries headers** so the backend can fill its own event `context`. This is how the backend learns "which browser/version/screen" made the request:
|
||||
|
||||
- `x-frontend-version` and `x-client: penpot-frontend/<version>`: added by `default-headers` in `frontend/src/app/util/http.cljs` to **all** fetch requests (unless `omit-default-headers`). The admin-console browser channel sends `penpot-admin-console/<version>` instead (`x-frontend-version` is ignored on the event-ingest path, only `x-client` matters there).
|
||||
- `user-agent`: set by the browser alone; the backend trims it to 500 chars (`get-client-user-agent`).
|
||||
- `x-external-session-id`: external SaaS session via `cf/external-session-id`; added by `repo.cljs` (`send!`, special `cmd!`, `multipart-upload`, export...) and also stored by `persist-events` in the frontend event context. The backend checks it (max 256, ignores `"null"`/empty).
|
||||
- `x-session-id`: `cf/session-id` (tab session), only in the main `send!`.
|
||||
- `x-event-origin`: UI origin (`::ev/origin` from param metadata, for example `"dashboard"`, `"workspace"`); sent by `send!` and the OIDC/export/upload `cmd!`. The backend trims it to 200 chars and stores it as `:client-event-origin` (different from the `:event-origin` the frontend puts in its own context!).
|
||||
|
||||
On the backend, `prepare-context-from-request` (`backend/src/app/loggers/audit.clj`) joins everything into the event context: `:external-session-id`, `:initiator` (`auth-key-id` or `"app"`), `:access-token-id/type`, `:client-event-origin`, `:client-user-agent`, `:client-version`, `:version` (backend), plus `:request-id`. Only `safe-backend-context-keys` (`:version`, `:initiator`, `:client-version`, `:client-user-agent`) survive in the telemetry shadow copy; the rest (`:client-event-origin`, `:external-session-id`, `:access-token-id/type`, `:request-id`...) is stored fully in the normal row and sent to Nexus, but dropped from telemetry.
|
||||
`props` describes the action; `context` describes how Penpot captured it. `created-at` is server time. In public ingest, `tracked-at` comes from the browser: Penpot replaces future timestamps and timestamps over an hour old with server time, keeping the original in `:original-tracked-at` before any telemetry filtering.
|
||||
|
||||
### Context field guide: `initiator`
|
||||
|
||||
`initiator` labels the application that started a request. The server sets it on both channels, but only the shared-key channel authenticates the sending application:
|
||||
For backend RPCs, `:initiator` is the shared-key id authenticated by HTTP middleware, or `"app"` for ordinary requests. For public ingest, Penpot overwrites the supplied value based on `x-client`: the admin-console product (including the legacy `penpot-nitrate` alias) maps to `"admin-console"`; Penpot and unknown products map to `"app"`. A caller controls that header. Use the label for analytics, not authorization or proof of which app sent the event. The session still sets the actor.
|
||||
|
||||
- **Backend channel** (RPC + management API): it comes from the shared-key auth layer (`wrap-shared-key-auth` in `backend/src/app/http/middleware.clj`): when a trusted service calls the management API with `x-shared-key: "<key-id> <secret>"`, the key-id (lowercased) is stored on the request as `::http/auth-key-id`, and `prepare-context-from-request` copies it into `:initiator`. When there is no shared key (a normal user RPC), it falls back to `"app"`, which means Penpot itself.
|
||||
- **Frontend channel** (`push-audit-events`): `get-client-initiator` maps the `x-client` header product to an initiator: `penpot-frontend` → `"app"`, `penpot-admin-console` → `"admin-console"`, legacy `penpot-nitrate` (old admin-console releases, remove once redeployed) → `"admin-console"`, missing or unknown → `"app"`. It overwrites any `initiator` in the event context, but the caller controls the header. Any authenticated caller can claim `"admin-console"` by sending that product in `x-client`. Use this label for usage analytics; it does not prove which application sent the event or grant permissions. The session still determines `profile-id`.
|
||||
Do not confuse `:client-event-origin` (the `x-event-origin` header on an RPC request) with `:event-origin` (context on a browser event). The former can help explain which UI triggered a backend action. The latter should not carry a backend business distinction.
|
||||
|
||||
Values you will see:
|
||||
## 2. Emitting events
|
||||
|
||||
- `"app"`: normal Penpot traffic (browser → public RPC or event ingest, or backend internal work). This is the vast majority.
|
||||
- `"admin-console"`: the authenticated Nitrate admin-console on management API calls (section 7a), or a caller claiming that product on browser events (section 7b, via the `x-client` mapping).
|
||||
- `"nexus"`, `"exporter"`, `"media-processor"`: other first-party services using their own shared keys (key ids come from `::setup/shared-keys` in `backend/src/app/setup.clj`, derived from the instance secret unless overridden).
|
||||
### Backend events
|
||||
|
||||
Why it matters: it supports per-application usage counts, with the trust level described above. It survives into telemetry on both channels (one of the four backend keys, and on the frontend allowlist too), so per-initiator counts stay available even on anonymized rows.
|
||||
The RPC wrapper emits after a command returns when full-row persistence, telemetry or webhooks is enabled. Use `::audit/skip` for a command that should not emit. Use result metadata to add `::audit/props`, replace request props with `::audit/replace-props`, or set an explicit name, type or context. **Never put secrets in props or context.** `clean-props` drops some known sensitive keys on the RPC path; it is not a general privacy filter for every producer.
|
||||
|
||||
More metadata the Penpot frontend adds inside each event (not as headers): `collect-context` with `ua-parser` (browser, engine, OS, device, screen, CPU arch), `locale` (updated on language change), `:session` / `:session-id` / `:external-session-id`, and SaaS host extras (`add-external-context-info`).
|
||||
Attribute the event to the **actor**, not an arbitrary `:profile-id` in the response. An invitation response can carry the inviter's id even though the invitee made the request. Use `::audit/profile-id` when the command has established the actor and the normal caller id is absent or not the right one. The RPC wrapper normalizes valid UUID strings and drops invalid overrides before falling back to the caller or zero. Hand-built events do not go through that normalization: pass a UUID to `audit/submit`. Reserve zero for cases without a known actor, such as SREPL actions.
|
||||
|
||||
## 9. Consumers I: webhooks
|
||||
`audit/submit` propagates failures. `wrap-audit` runs **after** the command's transaction, if it has one: an audit failure can fail the response even though the business mutation is already committed. An explicit `submit` inside the caller's transaction can roll back the mutation with the event. Low-level `audit/insert` only writes a full row when persistence is on; it does not fan out to webhooks or telemetry.
|
||||
|
||||
Webhooks let an external URL get notified when something happens in a team (for example "a file was created"). The flow has three steps:
|
||||
### Events used to rebuild state
|
||||
|
||||
1. **Match.** Some audit events are marked as webhook-relevant when they are created. For each of those, Penpot looks up the active webhooks of the affected team: it reads the team straight from the event, or finds it through the project or the file in the event. Each matching webhook gets its own delivery job. If several quick events belong together, they are grouped into one delivery (the grouping rule travels with the event itself).
|
||||
2. **Deliver.** Penpot POSTs the event to the webhook URL in the format that webhook asked for: JSON (with camelCase keys), transit, or form-encoded. Every attempt is written to a delivery log with the request, the response, and the error if any, so failures can be inspected later.
|
||||
3. **Protect.** Each webhook counts its consecutive failures. After 3 in a row, Penpot switches the webhook off automatically, so a dead URL stops receiving traffic until someone fixes it. A successful delivery resets the counter to zero.
|
||||
Put the ids a projection needs in the event so the receiver does not have to join production tables. A create event needs the new entity's id. Supply `:team-id` when the command already has it; for writes the wrapper can resolve it from a project or file id, not a bare `:id`. Reads do not need that lookup. For a bulk operation whose internal calls do not emit, submit one create event per entity when full rows are persisted. Use `audit/submit-create-event` when you have the row or `audit/submit-file-created-event` when you have only a file id; the latter checks that the file exists.
|
||||
|
||||
One guarantee: sending webhooks never changes the audit event itself. If delivery fails, only the webhook (and its delivery log) is affected; the audit row stays untouched and still gets archived normally.
|
||||
- `add-team-member` reports an actual membership insert, not a conflict that made no change. The emitter submits in the caller's transaction when full rows are persisted. Keep the actor separate from `:member-id` when someone adds another person.
|
||||
- `update-file` reports `"saved"`, `"partial"` or `"rejected"`. Only the first two write a change row and carry `:file-change-id`. The id points to the change, not the previous-state snapshot. Read it with `::db/remove-deleted false` because it is born with a future deletion date; even then, GC may already have removed it.
|
||||
- For moves, preserve source and destination team ids. For cascades and hard deletes, report the affected ids at emission rather than expecting a projection to reconstruct them later from deleted rows. The `:delete-object` handler emits its cascade event in the same transaction; direct and snapshot cascades do not emit that event.
|
||||
- `accept-organization-invitation` distinguishes direct and team invitations with `:organization-member-add-source` and `:belongs-to-team-on-add`. Keep that distinction in props; do not add another origin field.
|
||||
|
||||
Every match also writes a `webhook` / `trigger` row to `audit_log` (even with zero matches): its props are `{:name <origin event name>, :event-id <origin row id>, :total-affected <matched webhook count>}`. `process-event` assigns the origin row id up-front so the trigger can reference it; the telemetry shadow row always gets its own id (reusing it would collide on the PK).
|
||||
### Browser events
|
||||
|
||||
## 10. Consumers II: error reporters
|
||||
The browser collector starts on either `:audit-log` or `:telemetry` from `get-environment-data`. If discovery fails it starts anyway; the backend still decides whether to accept events. The collector buffers a bounded number of events and sends batches for the current profile. It can lose events on shutdown, overflow or failure. HTTP success does not prove a write: if storage fails, the endpoint logs the error and still returns normally. If it does write both the full and shadow rows, they commit together.
|
||||
|
||||
Both `app.loggers.database` and `app.loggers.mattermost` listen to backend `:error` logs **and** frontend crash events (found by the `::audit/event` marker), through a sliding-buffer channel so a flood cannot stall the app. The database one stores them in `server-error-report` (source 3 = backend log, 4 = audit-log origin, 5 = rate-limit); the Mattermost one forwards a short note to `:error-report-webhook` when set.
|
||||
Use a data-event or the `Event` protocol for a new event. Use kebab-case names (the public schema caps names at 50 characters) and reuse existing prop names. Do not add a browser copy of an action the backend already reports unless the browser observes something distinct. Honor `skip-audit?` when the UI resumes the same gesture. Performance events are usage data, not audit evidence.
|
||||
|
||||
Both reporters only start when the `:error-reporting` flag is on (off by default). The database reporter starts on the flag alone; the Mattermost reporter needs the flag **plus** its webhook URL configured. Without the flag, nothing is stored and `emit` calls from `push-audit-events` are silent no-ops. Each reporter also keeps a runtime on/off switch for emergencies via REPL.
|
||||
## 3. Destinations and privacy
|
||||
|
||||
Result: crash reports only exist if the frontend collector runs and `push-audit-events` accepts `unhandled-exception`/`exception-page`. Turning off the whole pipeline also blinds frontend error reporting.
|
||||
|
||||
## 11. Consumers III: telemetry (shadow rows)
|
||||
|
||||
Telemetry reuses the same table with **anonymized shadow rows** (`source LIKE 'telemetry:%'`): day-truncated times, `0.0.0.0` IPs, props reduced to uuid/boolean/number values plus a few allowed fields (`source`, `lang`, `auth-backend`, `activation`, derived `email-domain` — never raw emails), and a minimal context allowlist. Full + shadow rows can exist together per event when both flags are on; that duplication is on purpose.
|
||||
|
||||
Filters (`filter-telemetry-props` / `filter-telemetry-context`): by default only uuid/boolean/number with a simple key; exceptions: login/register/update-profile and frontend `identify` keep `lang`/`auth-backend`/`source`/`activation`/`email-domain`; `organization-sso-auth-failed` keeps `failure-reason` if it is a known one; `instance-start` passes whole; frontend `navigate` keeps `[:route :file-id :team-id :page-id]`. Subscription actions have event-specific contracts: backend and frontend `create-subscription` / `create-trial-subscription` preserve the default uuid/boolean/number props plus `type`, `deployment`, `quantity` and `subscription-mode`; backend `update-subscription` preserves only `type`, `deployment`, `previous-status`, `status`, `previous-cancel-at-period-end`, `cancel-at-period-end`, `previous-cancel-at`, `cancel-at` and `changed-fields`; backend `subscription-summary` preserves only `type`, `deployment`, `instance-id`, `snapshot-date`, the per-status `*-count` values, `total-count`, `sync-succeeded`, `sync-failed` and `snapshot-complete`. These subscription exceptions apply only to action events. Other free text drops from telemetry but **stays in the full row and in Nexus**. In `register-profile`, `activation` is a prop (it is what reaches PostHog). `action` exists only in the `context` of the full `register-profile-retry` rows; `filter-telemetry-context` drops it from the anonymised copy, which keeps only `:version`, `:initiator`, `:client-version` and `:client-user-agent`.
|
||||
|
||||
Shipping (`backend/src/app/tasks/telemetry.clj`): the cron sends shadows to `:telemetry-uri` as JSON in 10k batches and **deletes** them on success; it purges leftovers older than 7 days. It also sends a legacy report (counters, teams, email domains...). On official hosts nothing is collected or sent (`telemetry-excluded?` covers `penpot.app`/`penpot.dev`).
|
||||
|
||||
### Where telemetry events go: `telemetry.penpot.app` is Nexus
|
||||
|
||||
The default `:telemetry-uri` (`https://telemetry.penpot.app/`) points at the **Nexus telemetry endpoint**, so telemetry is a second road into Nexus, separate from the archive cron. The payload is plain JSON:
|
||||
|
||||
```
|
||||
POST https://telemetry.penpot.app/
|
||||
{type: "telemetry-events", version, instance-id, events: [...]}
|
||||
```text
|
||||
RPC / management / public ingest
|
||||
├─ full rows ────┐ ┌─ Nexus archive
|
||||
├─ shadow rows ──┴─ audit_log buffer ─────┼─ admin-console archive
|
||||
│ └─ telemetry endpoint
|
||||
├─ webhook jobs (marked producer events)
|
||||
└─ error reporters (crash events from public ingest; also receive backend logs)
|
||||
```
|
||||
|
||||
Nexus dispatches on `:type`: `"telemetry-events"` batches are coerced and inserted into its own `audit_log`; anything else (old flat payloads, `"telemetry-legacy-report"`) is upserted into `telemetry_report` per instance + day. Three details matter:
|
||||
### Full rows and archive consumers
|
||||
|
||||
- **Anonymization happens in Penpot, before sending.** By the time events leave, times are day-truncated, IPs are already `0.0.0.0`, props/context are filtered (see above), and emails are gone (only derived `email-domain` survives). Nexus additionally forces the outcome: it prefixes any non-`telemetry:` source, sets `received-at = tracked-at` (already truncated), pins IP to `0.0.0.0`, and stamps the sending `instance-id` into every event context.
|
||||
- **No personal data is expected.** Profile ids are random UUIDs with no PII, so the telemetry consumer needs no login guard; grouping is by `instance`, not by person (see posthog-telemetry below).
|
||||
- **The endpoint always answers 200.** Failures are swallowed and only counted (`nexus_telemetry_errors_total`), so a bad batch never breaks the sending instance; successes are counted (`nexus_telemetry_events_total`, `nexus_telemetry_reports_total`). A body the parser cannot read never reaches the handler: it answers 400 and counts in `nexus_http_malformed_requests_total`. Old instances may send a fressian+zstd base64 blob instead of a plain vector; Nexus decodes both (50 MiB cap against decompression bombs).
|
||||
Nexus receives full events, not telemetry shadows. Penpot sends Transit with the instance's `public-uri` as `origin` and a Nexus shared key. In the Nexus archive contract, Penpot's `created-at` becomes `received-at`; Nexus has its own `created-at` and `tracked-at` stays the origin time. The default telemetry endpoint also points to Nexus, but it is a separate ingest path with a reduced payload.
|
||||
|
||||
## 12. The cursor consumers: Nexus and the admin console
|
||||
The admin-console archive is **Penpot → Nitrate**, distinct from either Nitrate producer channel. Its consumer selects an allowlist of full events and enriches them with names or emails needed by the destination. The producer does not know that allowlist. Nonmatching rows still advance the cursor.
|
||||
|
||||
### How it works today
|
||||
### Webhooks and error reporters
|
||||
|
||||
Each consumer is a **loop in its own thread**, not a job (nexus on flag `:nexus`, admin console on `:admin-console`; `:audit-log-gc` stays a cron job on flag `:audit-log-gc`). The pass first takes the consumer's session advisory lock (`pg_try_advisory_lock(730000, consumer id)`, the two-key form on a pooled connection held until the pass ends: a lock space of its own, so it can never collide with the one-key lock `update-file` takes from the file id), so one backend drains that consumer at a time once every instance runs this version (a rolling deploy still has the old job-based instance, which does not know the tick; the two versions do not serialize each other, see below). The pass runs entirely on that connection (frontier, chunk transactions and lag gauge), so the pool must admit at least one connection per enabled consumer; a pass that finds it taken answers `::consumer/busy` (`:app.loggers.audit.consumers/busy`) and the loop sleeps `sleep-ms` (5 s) without touching the data. A pass whose consumer flag is off answers `::consumer/disabled` before borrowing a connection: the loop is gated at boot, but a direct caller (REPL, helper) must not send or advance a cursor for a disabled consumer, and that pass records no metric. Holding the lock it reads the **frontier** of the pass (`xid <= pg_snapshot_xmin(pg_current_snapshot())`, ordered `xid DESC, seq DESC`, first row) and drains chunks only up to it, so the pass always ends and every chunk but the last is 128 rows; a row committed while the pass runs waits for the next one. The position is the pair `(xid, seq)` stored in `audit_log_consumer_cursor`; each chunk locks that row `FOR UPDATE` (`SET LOCAL lock_timeout = '30s'`), reads 128 rows with `(xid, seq) > cursor AND (xid, seq) <= frontier AND xid <= pg_snapshot_xmin(pg_current_snapshot())` in `(xid, seq)` order, sends them, and advances the cursor in the same transaction. The row lock only serializes passes of the same consumer (the cursor row takes part in no FK); no `audit_log` row is locked, so the transports never wait for each other. On the first deploy each consumer has no cursor and registers it at `(0, 0)` at boot (`ensure-cursor`, before its first pass; after stopping the legacy archiver and its GC and deleting the rows it had already archived), so the backlog is replayed instead of dropped. Batches arrive at Nexus in insertion order (it numbers by arrival for order-sensitive consumers). Nexus receives transit `{:events [...]}` with `x-shared-key: "nexus <key>"` (`:nexus-shared-key`, or derived from the instance secret) and `origin` = public-uri; flag on but no URI raises `:task-not-configured`. The admin console receives only its allowlisted event names (enriched with team names, props emails, and optional top-level `:profile-email` / `:profile-name` from the actor `:profile-id`), excluding telemetry shadows; the rows outside its filter count as `skipped` and the cursor passes over them (a selective consumer that finds nothing advances to the end of the chunk, so it never blocks the GC). A chunk whose destination refuses it (Nexus non-204 raises `:consumer-chunk-refused`) or throws (transport failure, admin-console error) rolls back the cursor advance and stays for the next pass: the pass aborts with the chunks it already committed, and the loop logs the failure and retries after its pause. Committed chunks stay sent. The send fns return counts or throw carrying the attempted count as `:events`; `drain-consumer` records the `failed` metric from it, outside every chunk transaction (a local DB failure carries no `:events` and stays uncounted — it is not a destination failure). No `warn` before the throw: the loop reports the aborted pass. Each consumer has its own cursor, so a fallen Nexus never starves the admin console (nor vice versa).
|
||||
Webhooks use jobs, not a cursor over the audit buffer. A marked event schedules matching and delivery for the affected team's webhooks. When debounce is configured, a burst delivers the **last payload**, not a list of the events. Delivery failure changes webhook status and its delivery log, not the origin audit row. The audit `webhook` trigger records the match count, including zero, only if full-row persistence is on. Its `:event-id` comes from the producer event, so it can refer back to the full row when that row exists. A telemetry copy has its own id.
|
||||
|
||||
The legacy archiver (the one this release replaces) sent **one POST** for the whole chunk and advanced only on 204, so a single oversized event wedged the whole queue: a crash report whose message embeds a whole payload (a serialized file, for example) can push the request past the proxy body limit and the rejection blocks every later event until it is pruned by hand. Frontend crash reports are therefore size-bounded at the source (`mem:frontend/routing-app-shell-subtleties`): `app.common.exceptions/get-hint` and the CLJS `format-throwable` clamp exception messages (`max-message-length`).
|
||||
With `:error-reporting` on, public ingest forwards `unhandled-exception` and `exception-page` to the reporters even if neither kind of audit row is stored. The database reporter runs on that flag alone; Mattermost also needs its webhook URL. Reporters also receive backend error logs. The browser collector still has to run for browser crashes to reach them. Bound crash reports at the source: a message that embeds a serialized file can exceed the receiver's body limit and prevent later rows in the chunk from advancing. See `mem:frontend/routing-app-shell-subtleties` for frontend report limits and classification.
|
||||
|
||||
On the other side, Nexus checks `origin` against its allowlist (403 if not allowed), coerces the events, and inserts them into its own `audit_log` (partitioned by `received_at`, with an incrementing sequence number and `source`). Watch the rename: Penpot's `created-at` is stored as **`received-at`** in Nexus; Nexus's `created-at` is when it was created there. `tracked-at` keeps the origin time. Telemetry batches arriving at `telemetry.penpot.app` land in the same table through the telemetry endpoint above.
|
||||
### Telemetry shadows
|
||||
|
||||
Concurrent passes are safe: the tick lock makes a second backend skip its pass instead of overlapping, and the events are sent in cursor order, like an event log. During a rolling deploy the old archiver and GC are stopped (flags `:audit-log-archive` off, `:audit-log-gc` off) while the web instances keep writing; the new consumers start at boot and replay the unarchived buffer. The old and the new version do not serialize their reads (the old version does not know the cursor table, and the new one takes no lock the old one respects because it only reads rows), but with the old GC off no row is deleted under the new cursors, and the archived rows are deleted before the replay. The new GC stays off (`:audit-log-gc`) until every instance runs the new version (see the migration NOTE). No consumer waits for another one, because each has its own cursor. The GC reads the cursors without locking them and deletes only up to the oldest cursor in the table, inclusive (`FOR UPDATE SKIP LOCKED` in its batch select), so it never waits for a consumer and a consumer never sees a row deleted under its cursor; a consumer registers at boot, so its row exists before its first pass, and with no cursor at all the GC deletes nothing. A disabled consumer keeps its cursor and holds the GC at that position. With no cursor at all the GC deletes nothing: the `:audit-log-gc` flag is the deployment switch, and the old drain-everything branch was removed. **Nobody writes full rows when no consumer is on**: persistence follows the consumer flags, so the buffer stays empty and enabling a consumer later starts from an empty buffer. The operator must set the consumer flags on every instance, web and worker, because each process computes its own value at boot. The buffer itself has no tenant column: it is one log per database, with the cursor rows and the tick lock shared by every instance, so any worker instance drains any consumer. Adding a tenant would touch the producer, the cursor key, the GC, the telemetry task and the webhooks, and there is no historical value to backfill; it is a plan of its own, not part of this change. The drain does not touch `audit_log_pending`: released versions never created it, and the migration drops the table on a pre-release database so a leftover pending row cannot block the new GC with its foreign key. Its path has no FK check.
|
||||
Telemetry writes a separate, reduced copy, even when full-row persistence is also on. It truncates times to the day, sets the stored IP to `0.0.0.0`, and filters props and context before storage. The telemetry POST omits row id, `created-at` and IP; it keeps `profile-id`, a stable UUID that can link events. Treat the copy as reduced data, not as proof of anonymity.
|
||||
|
||||
Observability (the transports in `backend/src/app/loggers/audit/consumers/{nexus,admin_console}.clj`, the shared drain in `backend/src/app/loggers/audit/consumers.clj`, the GC in `backend/src/app/loggers/audit/gc.clj`, definitions in `backend/src/app/main.clj:default-metrics`): one `dbg` summary per pass (`:consumer` label, processed/skipped/chunks, `:elapsed` as `"1.5s"` via `ct/format-duration`, frozen in a strict binding first because log props evaluate lazily in the logger thread), one `dbg` per admin-console ingest call (`ready` with totals, `processed`/`failed` with counts), plus `dbg` when there is nothing to read or the component did not start (flag off). No `warn` before a throw: the loop logs the aborted pass and keeps going (refusal details travel in the error data). Metrics, all recorded with bounded labels only: `penpot_audit_consumer_events_total{ destination = nexus|admin-console, outcome = processed|failed|skipped }` (all consumers report the successful rows as `processed`; `skipped` = filtered by the admin-console allowlist or a telemetry shadow) and `penpot_audit_consumer_discarded_total{ reason = gc }` (rows the GC removed once every enabled consumer had passed them). `penpot_audit_consumer_lag_events{ destination }` is a gauge with the confirmed events after a consumer's cursor, capped at 10000 (past that it reads 10000, so a long backlog is not an O(backlog) count on every pass): it rises when the stream falls behind (a slow destination, or the burst after a big transaction commits). While a transaction is open its rows are invisible to everybody, so they cannot be counted as lag. Timing needs no custom histogram and no longer comes from the jobs substrate: a pass has no job row. `penpot_audit_consumer_passes_total{ destination, outcome }` counts them with `ok` (drained or empty), `busy` (another instance holds the tick) and `failed` (the pass threw; a local DB failure appears only here and in the error log), so a `busy` rate is normal with several instances but a sustained busy with `ok` at zero on every instance means a retained tick; a disabled pass (flag off, direct caller) records nothing. The signal is that counter plus the lag gauge. The bulk frontend ingest counts `penpot_audit_ingest_events_total{ kind = full|shadow, outcome = inserted|failed }` (both faces share one transaction, so a failure drops the batch on both). Archive success counts are recorded by `drain-consumer` on the drain totals: a chunk that returns normally committed its transaction (a rollback always throws instead), so they cannot inflate, and a pass that aborts on chunk N still records the `processed`/`skipped` of chunks 1..N-1 (the totals live in a volatile updated after each commit). Logs never carry event data (`props`, `context`, `profile-id`, `ip-addr`, `team-name`, emails): only counts, statuses and the uri host.
|
||||
For example, auth events can keep a derived `email-domain`, not the raw email; a full archive row may still contain the email. Context drops request and session identifiers but keeps an allowlisted `:initiator` for counts by application. Event-specific filters allow some values the default filter would drop. Check `filter-telemetry-props` and `filter-telemetry-context` with their tests before changing a field that leaves the instance.
|
||||
|
||||
### How Nexus consumes the data (and forwards to PostHog)
|
||||
The telemetry consumer scans the same stream, skips full rows, and ships shadows independently of the full archives. It uses a three-hour pause and larger chunks to avoid flooding the endpoint with small requests. The legacy `:telemetry` job still sends the aggregate report and newsletter data; it does not drain these rows. Its event counters see only the shadows still in the buffer, not a complete day's activity.
|
||||
|
||||
Stored rows are not the end: a consumer pipeline reads them back in sequence order and processes them. Each consumer remembers its place (sequence number), fetches the next chunk, runs its handler, then saves the new position. Three consumers run in order:
|
||||
## 4. How the buffer drains
|
||||
|
||||
1. **`local`**: keeps a mirror of user profiles. It listens to profile events (`register-profile`, `update-profile`, `login-with-*`, `delete-profile`, ...) and upserts email, name, props, and delete markers. It runs first because the next consumer needs profile data.
|
||||
2. **`posthog`** (runs after `local`): forwards our own SaaS archived events to PostHog. It sends an allowlist of backend track events (teams, members, projects, files, comments, Nitrate/SSO, ...) plus frontend events minus a noise exclusion list, grouped by **team**, with person identify data from the local profile mirror and snake_case props. Both paths include the `initiator` property when the stored context has one, so PostHog can split Penpot vs admin-console traffic.
|
||||
3. **`posthog-telemetry`**: forwards events with `source = telemetry:frontend|telemetry:backend` to a separate PostHog project. Because these rows are already anonymous (no IPs, truncated times, filtered props), grouping is by **instance** (the self-hosted server, from `context.instance-id`), with one `$groupidentify` event per instance per chunk so PostHog learns per-instance properties (version, URI, flags). Each consumer has its own PostHog credentials.
|
||||
`audit_log` is a buffer, not the archive. Producers insert rows without knowing the destinations. Each consumer keeps its own position in `audit_log_consumer_cursor`. Worker instances run consumers in background threads, not jobs. A session advisory lock allows only one pass per consumer per database. The pass holds a pool connection and fixes its upper bound (the *frontier*) at the start, so it ends even while new events arrive.
|
||||
|
||||
So the full journey is: Penpot buffer → Nexus `audit_log` (via either archive cron or the telemetry endpoint) → consumers → PostHog (SaaS events grouped by team, telemetry grouped by instance) plus local analytics views and materialized KPIs.
|
||||
### Ordering and retry
|
||||
|
||||
### Admin console: the second consumer
|
||||
The cursor orders rows by `(xid, seq)`: transaction id first, then sequence within the transaction. A snapshot cut based on `pg_snapshot_xmin` keeps the reader from moving past an earlier open transaction. For example, A inserts a row and stays open. B inserts and commits while A is open; then A inserts a second row and commits. A sequence-only cursor could read B and skip A's first, still invisible row. With the cut, the drain waits and can then deliver both A rows before B. This is neither global insertion order nor commit order.
|
||||
|
||||
Penpot → admin-console archiving lives in its own consumer (id `2`, flag `:admin-console`, a loop in a thread): it reads from its own cursor and ingests the allowlisted events through `nitrate/call :ingest-audit-log`, enriched as described above. The allowlist lives in the consumer now (`backend/src/app/loggers/audit/consumers/admin_console.clj`): the producer does not queue for anybody, so rows outside it are normally `skipped`. What also exists (do not confuse with archiving):
|
||||
The cut is conservative across the database. An unrelated long write transaction or a slow send inside a cursor transaction can delay what other passes can read, even though they do not lock one another's cursors. The GC can still delete up to positions already confirmed.
|
||||
|
||||
- **Nitrate → Penpot**: `POST /api/management/methods/push-audit-events` (`backend/src/app/rpc/management/nitrate.clj`) lets the trusted Nitrate backend inject events into the Penpot audit log (no RPC auth, `::audit/skip`, request context + `submit`; only the `admin-console` shared-key id can call it). Section 7a is the main user.
|
||||
- **Admin-console browser → collector**: usage events from the admin-console UI traveling through the normal public `push-audit-events` (section 7b).
|
||||
- **Penpot frontend Nitrate bits**: membership/subscription events (`frontend/src/app/main/data/nitrate.cljs`, `nitrate-audit/*`) through the same public endpoint.
|
||||
Each chunk sends and advances its cursor in one local transaction. A refusal aborts that chunk and leaves its cursor in place; earlier committed chunks stay done. An external send cannot roll back with the local transaction: if the destination accepts it and the cursor commit fails, the next pass can send it again. Consumers must handle repeats. A filtered chunk advances even if it sends nothing.
|
||||
|
||||
## 13. Flags and config
|
||||
### Registration, deletion and replay
|
||||
|
||||
Flags (`common/src/app/common/flags.cljc`, varia; enabled with `PENPOT_FLAGS=enable-<name>`):
|
||||
The component registers its cursor at `(0, 0)` after configuration succeeds and before its loop starts, so a new consumer replays what remains in the buffer. Consumer ids are stable and never reused: `0` telemetry, `1` Nexus, `2` admin console. A disabled consumer keeps its position until retired. Turning off both full-row consumers stops new full rows; it does not empty the buffer or stop telemetry shadows.
|
||||
|
||||
| Flag | What it does |
|
||||
Only the GC deletes. For full rows it deletes through the oldest registered cursor other than `0`; for shadows it deletes through cursor `0`. A missing cursor leaves its cursor-based path idle. A separate seven-day retention path deletes shadows by age regardless of cursors. It can delete unsent shadows; an older shadow can still ship if the consumer reads it before GC removes it.
|
||||
|
||||
The GC reads cursors again for each batch but does not serialize deletion with registration. A new consumer can miss up to one batch **per concurrent GC execution that read the previous cursor state**. There is no database-wide one-batch limit: cron job exclusivity is scoped by tenant, while the buffer and cursors are shared by the database. For a complete replay of what remains, pause the GC and wait for queued and running GC jobs to finish before registering. Follow `operations/2.20.0-consumer-generalization.md` for the rollout and replay procedure.
|
||||
|
||||
### What to watch
|
||||
|
||||
- `penpot_audit_consumer_lag_events` counts visible rows after a consumer's cursor, capped at 10,000. It includes rows that the consumer will filter; it is not a count of undelivered requests.
|
||||
- Pass outcomes distinguish a failed pass from `busy` (another instance holds the lock). Some `busy` is normal with several workers; sustained `busy` with no successful passes warrants checking the lock holder. `skipped` rows were filtered by the destination, not lost.
|
||||
- `penpot_audit_consumer_discarded_total{reason="gc"}` counts all committed deletions. The `purged stale telemetry events` warning counts **age-based deletions**, not confirmed missed deliveries.
|
||||
|
||||
## 5. Configuration, changes and tests
|
||||
|
||||
| Setting | Effect |
|
||||
|---|---|
|
||||
| `:audit-log-gc` | Runs the buffer GC; the old 2.18 GC and the new one share it. Keep it off while the old and new versions coexist and turn it on after the rollout. With no consumer enabled the GC deletes nothing |
|
||||
| `:nexus` | Send flag of the Nexus consumer; it appends the ingest path to `:nexus-uri`, and the component raises `:task-not-configured` at boot when the flag is on and no URI is configured |
|
||||
| `:admin-console` | Send flag of the admin-console consumer; also the module flag for the admin-console integration |
|
||||
| `:audit-log-logger` | Structured `app.audit` log per event |
|
||||
| `:error-reporting` | Starts the error reporters (off by default); database reporter on flag alone, Mattermost on flag plus webhook URL; also opens `push-audit-events` for crash batches without any other sink |
|
||||
| `:telemetry` | Stores shadow rows and turns on shipping (auto with `:telemetry-enabled`) |
|
||||
| `:nexus`, `:admin-console` | Start the corresponding archive consumers and enable full-row persistence. Set the intended flags consistently on web and worker instances: each computes persistence at boot. Nexus needs `:nexus-uri`. |
|
||||
| `:telemetry` | Collect and ship shadow rows to `:telemetry-uri`, separately from full-row persistence. |
|
||||
| `:error-reporting` | Start error reporters and enable public ingest for crash reports without requiring stored audit rows. |
|
||||
| `:audit-log-gc` | Schedule the GC, on by default. Use `disable-audit-log-gc` during a mixed-version rollout and follow the runbook before restoring it. |
|
||||
| `:audit-log-logger` | Log backend audit events, including event data. This is not the archive or a substitute for persistence. |
|
||||
|
||||
`:audit-log` is **not a flag any more**. `app.config` computes
|
||||
`cf/audit-persist?` once at boot from the consumer flags (`:nexus` or
|
||||
`:admin-console`): when neither is on, no full rows are written anywhere, and
|
||||
telemetry shadows are the only rows the producer inserts. The other event-path
|
||||
sinks get the same treatment: `cf/webhooks-enabled?`, `cf/telemetry-enabled?`
|
||||
and `cf/error-reporting-enabled?` are computed at boot from their flags and
|
||||
read by `wrap-audit`, `process-event`, the frontend ingest and the telemetry
|
||||
task. Component startup (`ig/init-key` of the reporters) still reads the flags
|
||||
directly; it is not a per-event path. All write sites read `cf/audit-persist?`:
|
||||
`process-event`, `submit-create-event`, `insert`, the frontend ingest,
|
||||
`wrap-audit`, and the membership/GC emitters. `get-environment-data`
|
||||
still puts `:audit-log` on the wire as a derived signal (persistence,
|
||||
`:telemetry`, or `:error-reporting`), so `event.cljs` and the RPC schema do not
|
||||
change. Telemetry keeps shipping shadows and reports on its own flag; with
|
||||
persistence off the telemetry task also runs its legacy collection mode. The
|
||||
`webhook`/`trigger` row from `webhooks.clj` disappears when no consumer is on,
|
||||
but the webhook deliveries themselves still work.
|
||||
`:audit-log` is not a configurable flag. As a wire signal it means full-row persistence **or** error reporting. `:telemetry` is a separate wire signal. Public ingest skips processing on a read-only pool or when none of those destinations is enabled.
|
||||
|
||||
Config (`backend/src/app/config.clj`): `:nexus-uri`, `:telemetry-uri` (default `https://telemetry.penpot.app/`), `:nexus-shared-key` / instance secret, `:public-uri` (sent as `origin` to Nexus).
|
||||
To add a consumer, give it a non-reused id and a transport definition for the shared `run-pass` loop. Configure its `:process`, optional `:prepare` and worker component; do not add producer-specific fan-out or seed its cursor in a migration. For contract tests, drive the real pass with prepared cfg; see `mem:backend/testing` for the test system and stubs. When changing an emitter, assert actor, props and exact emission counts for the specific backend action. Checking only the first matching event misses duplicates. Do not assert exact global counts from best-effort browser collection.
|
||||
|
||||
## 14. Practical guide: add or change events
|
||||
### Where to look
|
||||
|
||||
- **New backend command that must (or must not) emit**: `wrap-audit` does it alone; to exclude it set `::audit/skip`. To customize props/name/type use result metadata. Add a test mocking `app.loggers.audit/submit` (nil; `helpers.clj` stubs it globally) and assert `:call-args-list` — see `backend_tests/rpc-audit-test.clj` (full-row, telemetry-only, dual, no-op without flags, `insert`, `prepare-rpc-event`) and `rpc-team-test/accept-organization-invitation-audit-event` (three cases, exact props; counts matter: asserting with `first` passes even when a command emits the same name twice).
|
||||
- **New Penpot frontend event**: create the data-event or implement `Event`/`PerformanceEvent` in `app.main.data.event`, with `::name`/`::type`/`::origin`. Honor `skip-audit?` if the action can resume. Remember: with no props schema, **your asserts are the contract**.
|
||||
- **New admin-console event**: server action → add the backend-log call next to the mutation, best-effort so it never breaks the action; browser action → add a new event helper plus its test.
|
||||
- **Command that creates an entity or already knows its team**: put the new row's `:id` in `::audit/props` (`create-project`/`create-file`) and set `:team-id` when the team is loaded, so the consumer never has to join production tables. For entities created inside an internal call (`clone-template`, `import-binfile`, `duplicate-project`), emit one event per entity: `audit/submit-file-created-event` when only the id is known (it reads the row back), `audit/submit-create-event` when the row is already in hand.
|
||||
- **New audit-log consumer**: add its id to `app.loggers.audit.consumers` (never reuse one), do not seed its cursor in the migration: `ensure-cursor` creates it at the beginning of the buffer on the first pass, so the consumer replays what the buffer holds, add a `::consumer` component (the thread that runs its passes, gated by its flag) and a namespace with its `process-chunk`. The producer does not change. It must be idempotent: the drain is at-least-once, so the same event can reach it twice.
|
||||
- **Never** put secrets in `props`; `clean-props` is a last defense, not an excuse.
|
||||
- **Do not duplicate events**: if the backend already emits an event for an action, do not add a frontend (or admin-console browser) event for the same action — the backend one wins because it is reliable. A second event for the same action is only acceptable with a strong reason (for example the backend cannot see it at all), and then it must be clearly justified.
|
||||
- **Follow the naming policy**: use kebab-case names (`"create-organization"`, not `"createOrganization"`), and before inventing prop keys, look at similar existing events and reuse their key names (`organization-id`, `team-id`, `file-id`, `event-origin`...). Same concept, same key spelling — that is what keeps the data queryable later.
|
||||
|
||||
## 15. Code map
|
||||
|
||||
- Backend producer: `backend/src/app/loggers/audit.clj` (`submit`, `submit*`, `insert`, `prepare-rpc-event`, `enrich-team-id`, `submit-create-event`, `event-from-rpc-params`, telemetry filters), `backend/src/app/rpc.clj` (`wrap-audit`), `backend/src/app/rpc/commands/audit.clj` (`push-audit-events`, `get-environment-data`, deprecated `get-enabled-flags`), `backend/src/app/rpc/management/nitrate.clj` (`push-audit-events` Nitrate→Penpot), `backend/src/app/http/middleware.clj` + `backend/src/app/http/management.clj` (shared-key auth).
|
||||
- Membership emitter: `backend/src/app/rpc/commands/teams.clj` (`add-profile-to-team!` → `add-team-member`).
|
||||
- Penpot frontend producer: `frontend/src/app/main/data/event.cljs` (collector, buffer, send), `frontend/src/app/main/repo.cljs` + `frontend/src/app/util/http.cljs` (headers), `frontend/src/app/main/data/nitrate.cljs` (Nitrate membership events).
|
||||
- Admin-console producer (private repo, no paths cited here): server-side organization/subscription events through the management `push-audit-events`, browser-side UI events through the public `push-audit-events`.
|
||||
- Consumers: `backend/src/app/loggers/webhooks.clj`, `backend/src/app/loggers/database.clj`, `backend/src/app/loggers/mattermost.clj`, `backend/src/app/tasks/telemetry.clj`.
|
||||
- Archive (two cursor consumers + GC): `backend/src/app/loggers/audit/consumers.clj` (the shared mechanics: consumer ids and catalog, the tick lock and loop, the cursor lock/read/advance and the frontier, the lag gauge and the metrics) plus one namespace per transport: `consumers/nexus.clj` (the shared keys/HTTP client component) and `consumers/admin_console.clj` (with the enrichment and its allowlist); each transport declares its loop as `::consumer`, so the wiring in `backend/src/app/main.clj` (the `worker-config`) points at `:app.loggers.audit.consumers.{nexus,admin-console}/consumer`, and only worker instances start them. The GC lives in `backend/src/app/loggers/audit/gc.clj` (`:audit-log-gc`, `::job-def`, still a cron job). The producer only inserts (`append-audit-entry` in `backend/src/app/loggers/audit.clj`, `insert-rows!` in `backend/src/app/rpc/commands/audit.clj`): no queue, no allowlist. The schema ships in one migration, `0159-consumer-generalization` (the runner applies every pending step in one transaction, so the whole feature is a single step): the `seq`/`xid` columns, the unique `(xid, seq)` index and the cursor table, empty on purpose: each consumer registers its row at `(0, 0)` at boot through `ensure-cursor`, before its first pass, and the job exclusivity/dedupe columns and indexes. The pending queue never shipped: the migration does not create `audit_log_pending` and drops the table if a pre-release database has it. The out-of-band runbook lives in `operations/2.20.0-consumer-generalization.md` (the migration comment points to it): it registers the steps first (step 0, `INSERT INTO migrations`, so no deploy lands while the runner could still take the locks, and the steps below become the only path that applies the schema), then stops the legacy archiver and its GC, deletes the archived rows, keeps the legacy mark and its indexes for a later cleanup, pre-applies `0154-add-job-table` (2.19 has no `job` table) and the additive `0159` schema with the `setval(count + 1000000)` reserve, the `CONCURRENTLY` indexes, the `CHECK NOT VALID` dance and a gate that also checks the job tables, its eight indexes, the trigger and the two flags (`indisvalid` included); the web writes never stop because the old producer names neither `seq` nor `xid`, so the database fills them, and the new GC is turned on only after the rollout. Both steps are registered in step 0, so the runner never applies them: everything they create must be applied by hand, and the gate is what proves it before the deploy. The two consumers start a thread gated by their flag; the GC keeps its 5-min cron entry.
|
||||
- Nexus (private repo, no paths cited here): archive ingest, ordered consumers (local profile mirror, PostHog SaaS events, PostHog telemetry), telemetry endpoint, analytics views.
|
||||
- Tests: `backend_tests/rpc-audit-test.clj`, `rpc-team-test/accept-organization-invitation-audit-event`.
|
||||
- Producer and telemetry filtering: `backend/src/app/loggers/audit.clj`; RPC wrapper: `backend/src/app/rpc.clj`.
|
||||
- Public ingest and discovery: `backend/src/app/rpc/commands/audit.clj`; browser collector: `frontend/src/app/main/data/event.cljs`; trusted ingest: `backend/src/app/rpc/management/nitrate.clj`.
|
||||
- Shared drain, transports and GC: `backend/src/app/loggers/audit/`; rollout: `operations/2.20.0-consumer-generalization.md`.
|
||||
- Contract examples in `backend/test/backend_tests/`: `rpc_audit_test.clj` for emitters and ingest, `loggers_audit_consumers_test.clj` for cursor ordering, retry and GC, and `rpc_team_test.clj` for invitation attribution.
|
||||
|
||||
@ -1,3 +1,3 @@
|
||||
# Backend Audit Log
|
||||
|
||||
Superseded by `mem:audit-log`: the full audit-log and event-processing guide (backend center, frontend collection, headers, webhooks, error reporters, telemetry, Nexus archive and planned Nitrate archive, table-as-buffer lifecycle, event anatomy, flags, tests). Read that memory instead; keep this stub so old references do not rot.
|
||||
Moved to `mem:audit-log`, the guide to event collection, delivery and retention.
|
||||
|
||||
@ -17,8 +17,8 @@ JVM `clojure.test` (kaocha runner) under `backend/test/backend_tests/`.
|
||||
- Line coverage: append the `:coverage` alias (`clojure -M:dev:test:coverage`, kaocha-cloverage). It instruments every `app.*` namespace under `src/` at load, before any test runs, so even a `--focus` run instruments them all; options live under `:cloverage/opts` in `tests.edn`, output in `target/coverage/` (lcov, HTML, and EMMA `coverage.xml` whose form counts the PR report shows as statements; cloverage measures no functions or branches) (`lcov.info`, HTML). About 2x slower and noisy with reflection warnings (instrumented forms lose type hints): never use it for normal runs. `tests-backend.yml` runs it and renders the step summary with `scripts/coverage-summary.mjs`.
|
||||
- If you need to filter output, tee to a temp file first: `clojure -M:dev:test 2>&1 | tee /tmp/penpot-test-output.txt`.
|
||||
- RPC test helpers `command!`/`management-command!` split the data map: qualified keys become server params, unqualified keys become request body params. To inject request-level context (headers, `:app.http/auth-key-id`, ip), pass a map under `:app.http/request` metadata; non-map `IRequest` stubs fall back to a dummy request.
|
||||
- The test system is `main/system-config` + `main/worker-config` with the components that must not run in the background dropped: HTTP, the srepl, cron, the dispatcher, the two audit-log consumer loops (`:app.loggers.audit.consumers.nexus/consumer`, `:app.loggers.audit.consumers.admin-console/consumer`) and **every** `[<profile> :app.worker/runner]` (`helpers/remove-job-runners`). A live runner of any queue races the tests that drive the loop by hand (`wdisp/run-batch` + `@#'wrkr/run-worker-loop`): it takes the job the test just dispatched and completes it behind the test's back, so the test's own assertions read a state it did not produce. A live consumer loop races the tests the same way: it drains `audit_log` in the background. Do not re-add any of them by editing that list by hand; the runner removal matches by key shape on purpose.
|
||||
- `th/run-task!` invokes a job by name; the audit-log consumers are no longer jobs, so a test builds the pass cfg (`::db/pool`, `::mtx/metrics` and the transport deps) and calls the transport's `run-pass` directly (`consumer.nexus/run-pass`, `consumer.admin-console/run-pass`).
|
||||
- The test system is `main/system-config` + `main/worker-config` with the components that must not run in the background dropped: HTTP, the srepl, cron, the dispatcher, the three audit-log consumer loops (`:app.loggers.audit.consumers.nexus/consumer`, `:app.loggers.audit.consumers.admin-console/consumer`, `:app.loggers.audit.consumers.telemetry/consumer`) and **every** `[<profile> :app.worker/runner]` (`helpers/remove-job-runners`). A live runner of any queue races the tests that drive the loop by hand (`wdisp/run-batch` + `@#'wrkr/run-worker-loop`): it takes the job the test just dispatched and completes it behind the test's back, so the test's own assertions read a state it did not produce. A live consumer loop races the tests the same way: it drains `audit_log` in the background. Do not re-add any of them by editing that list by hand; the runner removal matches by key shape on purpose.
|
||||
- `th/run-task!` invokes a job by name; audit-log consumers are not jobs. Call `(consumers/run-pass cfg consumer)` with the prepared cfg (`::db/pool`, `::mtx/metrics` and transport deps) and the transport's `consumer` definition. A successful pass returns nil and publishes totals as metrics; `::consumers/busy` and `::consumers/disabled` mean no work started.
|
||||
- `app.loggers.audit/submit` propagates its failure (`submit*` no longer swallows it), so a DB-free unit test that reaches a code path emitting an audit event must stub it (`with-redefs [app.loggers.audit/submit (constantly nil)]`, as the `helpers.clj` fixture does globally) or pass a cfg carrying `::db/pool`. Otherwise the throw detours the flow under test into its error path.
|
||||
- `system-config` and `worker-config` are `def`s: their `(cf/get ...)` calls are evaluated when the namespace loads, against the real environment, while test code calling `cf/get` at run time sees the `penpot-test`-prefixed config. The two can disagree — in the devenv `:tenant` is `devenv-ws0` for the defs and `default` inside a test — so a component built from `worker-config` can talk to a different Redis key prefix than a cfg the test builds itself. The symptom is a component that silently does nothing locally and races in CI (which sets no `PENPOT_TENANT`).
|
||||
- `backend-tests.helpers` has `metrics-stub` (a metrics instance with no collectors, where every `run!` is a no-op), `init-climit` (a concurrency limiter built from an inline config map, with an optional second argument for the metrics instance to publish to) and `with-climit`, the macro that builds one, runs the body and halts it. `init-climit` starts the republisher daemon thread, so a test that builds a limiter without halting it leaks that thread for the rest of the JVM run: use `with-climit`.
|
||||
|
||||
@ -45,7 +45,7 @@ Production runs the deployed toolbox, not the devenv one:
|
||||
- **One query per reference column.** Two columns in one `IN (col_a, col_b)` lose the index and scan the whole table; split them.
|
||||
- The read-only replica cuts long queries: see `mem:prod-infra/grafana`.
|
||||
- A migration that builds an index runs inside the migration transaction, so it cannot use `CONCURRENTLY`, and a plain `CREATE INDEX` locks the table against writes for the whole build. On a large table (file saves write to `file_change` on every save) build the index out of band, before deploying, with `CREATE INDEX CONCURRENTLY` on its own (never inside a transaction block); the shipped migration uses `IF NOT EXISTS`, finds it and skips the build (it still takes the `ALTER`s' locks, so the operator registers the step in `migrations`; the consumer-generalization runbook registers it first of all, before the DDL). A concurrent build interrupted midway leaves the index invalid: drop it and retry (check `pg_index` and `pg_index.indisvalid`). The runner applies **every pending step in one transaction**, so pre-applying one migration of a group does not avoid the locks of the others.
|
||||
- The 2.20 consumer generalization is the concrete case, and its runbook is the contract for a large instance: it ships as `0159-consumer-generalization` (plus `0154-add-job-table`, which a 2.19 database does not have); the runbook lives in `operations/2.20.0-consumer-generalization.md` (the migration comment points to it). It opens by registering the steps (`INSERT INTO migrations`, `module = 'main'`, `step = '0154-add-job-table'` and `'0159-consumer-generalization'`) so the runner skips the group from the first minute, whatever deploy lands in between, and the steps below become the only path that applies the schema; without that row the guards skip the work, not the `ACCESS EXCLUSIVE` locks nor the backfill scan. The steps: the `IF NOT EXISTS` ALTERs (metadata-only), the `CONCURRENTLY` unique indexes (`job__live_exclusive__uniq`, `job__new_dedupe__uniq`, `audit_log__xid__seq__idx`), the `seq` backfill (a single ordered `UPDATE`, no table lock, re-runnable) with the sequence reserved above it (`setval(count + 1000000)`, so new rows sort above the history), the `CHECK NOT VALID` + `VALIDATE` + `SET NOT NULL` for `audit_log.seq`, and the `indisvalid` gate before deploying. Before that, the runbook stops the legacy archiver **and its GC** with the flags `:audit-log-archive` off and `:audit-log-gc` off, keeps at least one consumer flag (`:nexus` or `:admin-console`) on **on every instance**, and deletes the already-archived rows. There is no `:audit-log` flag any more: each process computes its persistence value at boot from the consumer flags, so a web instance without one writes no rows even when a worker drains them. With all consumers off there is no backlog to replay. The two versions do not serialize each other: a concurrent old archiver/GC could archive and delete rows the new consumers have not read yet. The legacy mark and its indexes stay; a later release drops them with catalog operations. The new GC is gated by `:audit-log-gc` (the old 2.18 flag): keep it off while the old and new versions coexist and turn it on after the rollout; with no consumer enabled it deletes nothing. The pending queue never shipped: the migration does not create `audit_log_pending` and drops the table outright if a pre-release database has it, so leftover rows there cannot block the new GC.
|
||||
- Audit-log rollout: `operations/2.20.0-consumer-generalization.md` owns the procedure for large databases. It pre-registers `0154-add-job-table` and `0159-consumer-generalization` before doing the schema work out of band; otherwise idempotency guards do not prevent migration locks or scans. The old archiver and new consumers do not coordinate, so stop the old archiver and keep the GC explicitly disabled during the mixed-version rollout. The new GC runs by default. Turning consumers off does not stop it: registered cursors still bound cursor-based deletion, and seven-day shadow retention needs no cursor. See `mem:audit-log` for the replay and retention guarantees.
|
||||
- The 2.20.0 release ships three out-of-band migration procedures; the unified, self-contained runbook for the whole upgrade (phases from pre-flight to the maintenance window) is `operations/2.20.0-unified.md`, indexed in `operations/README.md`. The individual 2.20.0 runbooks stay as the reference per area.
|
||||
- A partitioned parent cannot take `CREATE INDEX CONCURRENTLY` (`cannot create index on partitioned table ... concurrently`). Build one index per partition with `CONCURRENTLY`, then `CREATE INDEX ... ON ONLY` on the parent and `ALTER INDEX ... ATTACH PARTITION` for each child; the parent index only becomes valid when every partition is attached.
|
||||
- Write the migration as plain, idempotent SQL: `ADD COLUMN IF NOT EXISTS`, `CREATE INDEX IF NOT EXISTS`, and `DROP CONSTRAINT IF EXISTS` followed by `ADD CONSTRAINT`. Do not encode shape checks as `pg_catalog` introspection: they are easy to get wrong (a missing `NOT condeferred` accepts an `INITIALLY DEFERRED` foreign key) and they duplicate what the runbook verifies by hand. `IF NOT EXISTS` matches the name, not the definition, so the out-of-band procedure verifies the index shape and completeness; an `INVALID` index from an interrupted build also satisfies `IF NOT EXISTS` and the dedupe/exclusivity stops enforcing silently, so the `indisvalid` gate is mandatory.
|
||||
|
||||
@ -333,8 +333,8 @@
|
||||
(when cf/telemetry-enabled?
|
||||
;; NOTE: when both persistence and telemetry are enabled, events are stored
|
||||
;; twice: once with full details (above) and once stripped of props and
|
||||
;; ip-addr, tagged with source="telemetry" so the telemetry task can
|
||||
;; collect and ship them. The profile-id is preserved (UUIDs are already
|
||||
;; ip-addr, tagged with source="telemetry:*" so the telemetry consumer
|
||||
;; (id 0) can send them. The profile-id is preserved (UUIDs are already
|
||||
;; anonymous random identifiers). Only a safe subset of context fields
|
||||
;; is kept: initiator, version, client-version and client-user-agent.
|
||||
;; Timestamps are truncated to day precision to avoid leaking exact event
|
||||
|
||||
@ -7,18 +7,21 @@
|
||||
(ns app.loggers.audit.consumers
|
||||
"The mechanics shared by the consumers of the audit log buffer.
|
||||
|
||||
Two consumers read the events of the buffer, each one from its own
|
||||
Three consumers read the events of the buffer, each one from its own
|
||||
position (a cursor) over `audit_log`:
|
||||
|
||||
- the nexus consumer sends them to Nexus.
|
||||
- the admin-console consumer sends the allowlisted ones to the admin
|
||||
console.
|
||||
- the nexus consumer sends the full rows to Nexus.
|
||||
- the admin-console consumer sends the allowlisted full rows to the
|
||||
admin console.
|
||||
- the telemetry consumer sends the anonymized copies (the shadow
|
||||
rows, `source LIKE 'telemetry:%'`) to the telemetry endpoint.
|
||||
|
||||
Each transport lives in its own namespace (`...consumers.nexus`,
|
||||
`...consumers.admin-console`); this one
|
||||
holds what they share: the consumer ids, the catalog, the cursor
|
||||
(lock, read, advance), the tick mutex, the loop and the metrics. The
|
||||
garbage collection of the buffer lives in `app.loggers.audit.gc`.
|
||||
Each transport lives in its own namespace (`...consumers.nexus`,
|
||||
`...consumers.admin-console`, `...consumers.telemetry`); this one
|
||||
holds what they share: the consumer ids, validation and generic pass
|
||||
runner, the cursor (lock, read, advance), the tick mutex, the loop and
|
||||
the metrics. Each transport exports its own consumer definition. The
|
||||
garbage collection of the buffer lives in `app.loggers.audit.gc`.
|
||||
|
||||
A consumer is a loop in a thread, not a job: the pass takes the tick
|
||||
lock, drains up to the frontier it found at its start and sleeps. The
|
||||
@ -29,7 +32,6 @@
|
||||
open. The producer does not know about consumers: it only inserts the
|
||||
row, and the GC deletes it once every enabled consumer has passed it."
|
||||
(:require
|
||||
[app.common.exceptions :as ex]
|
||||
[app.common.logging :as l]
|
||||
[app.common.time :as ct]
|
||||
[app.config :as cf]
|
||||
@ -38,27 +40,32 @@
|
||||
[promesa.exec :as px]))
|
||||
|
||||
;; Consumer ids are fixed constants, never reuse one: a retired id keeps
|
||||
;; meaning its old position in `audit_log_consumer_cursor`.
|
||||
;; meaning its old position in `audit_log_consumer_cursor`. Id 0 is the
|
||||
;; stream of anonymized copies (telemetry shadows), apart from the
|
||||
;; full-row consumers that take 1 and up.
|
||||
(def telemetry 0)
|
||||
(def nexus 1)
|
||||
(def admin-console 2)
|
||||
|
||||
(def ^:private consumers
|
||||
[{:id nexus :flag :nexus :label "nexus"}
|
||||
{:id admin-console :flag :admin-console :label "admin-console"}])
|
||||
|
||||
(def ^:private consumers-by-id
|
||||
(into {} (map (juxt :id identity)) consumers))
|
||||
(defn- check
|
||||
"Assert the shape of a consumer definition: the keys the framework needs to
|
||||
run its loop. A plain map fails at the use without this."
|
||||
[consumer]
|
||||
(assert (integer? (:id consumer)) "consumer id must be an integer")
|
||||
(assert (keyword? (:flag consumer)) "consumer flag must be a keyword")
|
||||
(assert (string? (:label consumer)) "consumer label must be a string")
|
||||
(assert (fn? (:process consumer)) "consumer process must be a function")
|
||||
(assert (or (nil? (:prepare consumer)) (fn? (:prepare consumer)))
|
||||
"consumer prepare must be a function")
|
||||
(assert (or (nil? (:options consumer)) (map? (:options consumer)))
|
||||
"consumer options must be a map"))
|
||||
|
||||
(defn enabled?
|
||||
"True when the flag of a consumer is on. The transport components use it
|
||||
to decide at boot whether to start the loop; the GC uses the cursor table
|
||||
to decide whose cursor it waits for."
|
||||
[consumer-id]
|
||||
(contains? cf/flags (get-in consumers-by-id [consumer-id :flag])))
|
||||
|
||||
(defn- consumer-label
|
||||
"True when the flag of a consumer is on. The components use it at boot to
|
||||
decide whether to start the loop; a direct pass of a disabled consumer
|
||||
answers ::disabled."
|
||||
[consumer]
|
||||
(get-in consumers-by-id [consumer :label]))
|
||||
(contains? cf/flags (:flag consumer)))
|
||||
|
||||
(defn decode-row
|
||||
"Decode the transit-encoded columns of an audit-log row. The
|
||||
@ -139,23 +146,6 @@
|
||||
VALUES (?::smallint, '0'::xid8, 0)
|
||||
ON CONFLICT (consumer) DO NOTHING")
|
||||
|
||||
(def ^:private registry-lock-classid
|
||||
"The two-key advisory lock of the cursor registry. It lives in its own
|
||||
classid, apart from the consumer tick (730000) and from the one-key form
|
||||
the file locks use."
|
||||
730001)
|
||||
|
||||
(def ^:private registry-lock-objid 0)
|
||||
|
||||
(defn lock-registry
|
||||
"Take the registry lock for the current transaction. `ensure-cursor` and
|
||||
each GC batch call it before touching the cursor table, so a cursor
|
||||
registered between two batches is seen by the next one."
|
||||
[conn]
|
||||
(db/exec-one! conn ["select pg_advisory_xact_lock(?::int, ?::int) as lock"
|
||||
registry-lock-classid registry-lock-objid])
|
||||
true)
|
||||
|
||||
(defn ensure-cursor
|
||||
"Create the cursor of a consumer when it does not exist yet: at the
|
||||
beginning of the buffer, so the first pass drains what is there. The
|
||||
@ -163,18 +153,15 @@
|
||||
`(xid, seq) > cursor`, so the starting position must sit *before* the
|
||||
oldest row: (0, 0) is below any real position and needs no extra read.
|
||||
|
||||
Serialized with the GC by `lock-registry`: a batch that reads the oldest
|
||||
cursor and deletes cannot interleave with this insert, so the rows that
|
||||
exist when the consumer registers survive its first pass. The GC waits
|
||||
only for the cursors that exist; a consumer with no row has not run yet.
|
||||
The normal drain path finds the row and does not reach this; when a drain
|
||||
finds it missing, the lock is held until that chunk transaction commits."
|
||||
The GC re-reads the cursors on every batch, so a cursor registered
|
||||
between two batches is seen by the next one. A cursor registered while a
|
||||
batch is between its read and its delete is not seen by that batch: the
|
||||
rows it deletes are lost for a consumer that starts at that exact moment
|
||||
(at most one batch per concurrent GC run that read the previous cursor).
|
||||
Two instances registering the same
|
||||
consumer are safe: `ON CONFLICT DO NOTHING` keeps the existing row."
|
||||
[cfg consumer]
|
||||
(db/tx-run!
|
||||
cfg
|
||||
(fn [cfg]
|
||||
(lock-registry (::db/conn cfg))
|
||||
(db/exec-one! cfg [sql:ensure-cursor consumer]))))
|
||||
(db/tx-run! cfg (fn [cfg] (db/exec-one! cfg [sql:ensure-cursor consumer]))))
|
||||
|
||||
(def ^:private sql:chunk
|
||||
"The next chunk of events for one consumer.
|
||||
@ -189,7 +176,7 @@
|
||||
|
||||
No `SKIP LOCKED` and no row lock: the tick mutex and the cursor lock
|
||||
already serialize the passes of this consumer, and the rows are only
|
||||
read."
|
||||
read."
|
||||
"SELECT a.id,
|
||||
a.name,
|
||||
a.type,
|
||||
@ -198,8 +185,8 @@
|
||||
a.tracked_at,
|
||||
a.profile_id,
|
||||
a.ip_addr,
|
||||
a.props,
|
||||
a.context,
|
||||
%s AS props,
|
||||
%s AS context,
|
||||
a.seq,
|
||||
a.xid::text AS xid_text
|
||||
FROM audit_log a
|
||||
@ -207,7 +194,15 @@
|
||||
AND (a.xid, a.seq) <= (?::xid8, ?::bigint)
|
||||
AND a.xid <= pg_snapshot_xmin(pg_current_snapshot())
|
||||
ORDER BY a.xid ASC, a.seq ASC
|
||||
LIMIT ?")
|
||||
LIMIT ?")
|
||||
|
||||
(def ^:private sql:full-chunk
|
||||
(format sql:chunk "a.props" "a.context"))
|
||||
|
||||
(def ^:private sql:telemetry-chunk
|
||||
(format sql:chunk
|
||||
"CASE WHEN a.source LIKE 'telemetry:%' THEN a.props END"
|
||||
"CASE WHEN a.source LIKE 'telemetry:%' THEN a.context END"))
|
||||
|
||||
(def ^:private sql:last-visible
|
||||
"The frontier of the pass: the newest row whose transaction is already
|
||||
@ -260,10 +255,10 @@
|
||||
is cleared when the connection is borrowed again, so one bad unlock
|
||||
cannot stop the consumer for good. A failed unlock is logged and
|
||||
swallowed: it must not replace the error of the pass that is unwinding."
|
||||
[{:keys [::db/pool]} consumer-id f]
|
||||
[{:keys [::db/pool]} consumer f]
|
||||
(with-open [conn (db/open pool)]
|
||||
(db/session-unlock-all conn)
|
||||
(let [[classid objid] (tick-lock-key consumer-id)]
|
||||
(let [[classid objid] (tick-lock-key (:id consumer))]
|
||||
(if (db/session-try-lock conn classid objid)
|
||||
(try
|
||||
(f conn)
|
||||
@ -272,16 +267,18 @@
|
||||
(db/session-unlock conn classid objid)
|
||||
(catch Throwable cause
|
||||
(l/wrn :hint "unable to release the audit-consumer lock"
|
||||
:consumer (consumer-label consumer-id)
|
||||
:consumer (:label consumer)
|
||||
:cause cause)))))
|
||||
::busy))))
|
||||
|
||||
(defn- get-chunk-rows
|
||||
[cfg cursor watermark]
|
||||
(->> (db/exec! cfg [sql:chunk
|
||||
[cfg consumer cursor watermark]
|
||||
(->> (db/exec! cfg [(if (= telemetry (:id consumer))
|
||||
sql:telemetry-chunk
|
||||
sql:full-chunk)
|
||||
(:xid-text cursor) (:seq cursor)
|
||||
(:xid-text watermark) (:seq watermark)
|
||||
chunk-size])
|
||||
(or (:chunk-size (:options consumer)) chunk-size)])
|
||||
(not-empty)))
|
||||
|
||||
(defn- advance-cursor
|
||||
@ -311,7 +308,7 @@
|
||||
cursor (or (lock-cursor cfg consumer-id)
|
||||
(do (ensure-cursor cfg consumer-id)
|
||||
(lock-cursor cfg consumer-id)))]
|
||||
(when-let [rows (get-chunk-rows cfg cursor watermark)]
|
||||
(when-let [rows (get-chunk-rows cfg consumer cursor watermark)]
|
||||
(let [res (process-chunk cfg rows)]
|
||||
(advance-cursor cfg consumer-id (peek rows))
|
||||
res)))))]
|
||||
@ -344,7 +341,7 @@
|
||||
(let [lag (:lag (db/exec-one! conn [sql:lag (:xid-text cursor) (:seq cursor)]))]
|
||||
(mtx/run! metrics
|
||||
:id :audit-consumer-lag
|
||||
:labels [(consumer-label (:id consumer))]
|
||||
:labels [(:label consumer)]
|
||||
:val lag))))
|
||||
|
||||
(defn- record-totals
|
||||
@ -357,7 +354,7 @@
|
||||
(when (pos? skipped)
|
||||
(record-count metrics :audit-consumer-events [label "skipped"] skipped)))
|
||||
|
||||
(defn drain-consumer
|
||||
(defn- drain
|
||||
"Run one pass of a consumer: take the tick lock, drain its cursor chunk by
|
||||
chunk (one transaction per chunk) up to the frontier of the pass, and
|
||||
record the summary. Answers ::busy without doing anything when another
|
||||
@ -382,68 +379,59 @@
|
||||
counts `ok` (drained or empty), `busy` (another instance holds the tick)
|
||||
and `failed` (anything the pass throws: the frontier read, a chunk or the
|
||||
lag query), so a failure cannot leave the family flat."
|
||||
[cfg consumer-id process-chunk]
|
||||
(cond
|
||||
(not (contains? consumers-by-id consumer-id))
|
||||
(ex/raise :type :internal
|
||||
:code :unknown-audit-consumer
|
||||
:hint "unknown audit-log consumer"
|
||||
:consumer consumer-id)
|
||||
|
||||
(not (enabled? consumer-id))
|
||||
[cfg consumer]
|
||||
(if-not (enabled? consumer)
|
||||
::disabled
|
||||
|
||||
:else
|
||||
(let [consumer (get consumers-by-id consumer-id)
|
||||
label (consumer-label consumer-id)
|
||||
metrics (mtx/instance cfg)
|
||||
totals (volatile! {:processed 0 :skipped 0 :chunks 0})
|
||||
result (with-tick-lock cfg consumer-id
|
||||
(fn [conn]
|
||||
(try
|
||||
(if-let [watermark (db/exec-one! conn [sql:last-visible])]
|
||||
(let [tpoint (ct/tpoint)
|
||||
drained (drain-chunks cfg conn consumer process-chunk watermark totals)
|
||||
_ (record-lag conn metrics consumer)
|
||||
_ (record-totals metrics label drained)
|
||||
_ (record-count metrics :audit-consumer-passes [label "ok"] 1)
|
||||
processed (:processed drained)
|
||||
skipped (:skipped drained)
|
||||
chunks (:chunks drained)
|
||||
;; Strict binding on purpose: log props evaluate lazily in
|
||||
;; the logger thread, so the elapsed must be frozen here.
|
||||
elapsed (ct/format-duration (tpoint))]
|
||||
(if (and (zero? processed)
|
||||
(zero? skipped))
|
||||
(l/dbg :hint "no new audit events"
|
||||
:consumer label)
|
||||
(l/dbg :hint "audit events processed"
|
||||
:consumer label
|
||||
:elapsed elapsed
|
||||
:processed processed
|
||||
:skipped skipped
|
||||
:chunks chunks)))
|
||||
;; Nothing was visible when the pass started: no chunk query.
|
||||
(do
|
||||
(record-lag conn metrics consumer)
|
||||
(record-count metrics :audit-consumer-passes [label "ok"] 1)
|
||||
(l/dbg :hint "no new audit events"
|
||||
:consumer label)
|
||||
nil))
|
||||
(catch Throwable cause
|
||||
;; The counter covers the whole pass: a failed frontier
|
||||
;; read or lag query counts as a failed pass too, not
|
||||
;; only a refused chunk.
|
||||
(record-totals metrics label @totals)
|
||||
(record-count metrics :audit-consumer-passes [label "failed"] 1)
|
||||
;; The failed event count comes from the error data: the
|
||||
;; send fns carry the attempted events as :events. A local
|
||||
;; DB failure carries none and is not counted as a
|
||||
;; destination failure.
|
||||
(let [n (:events (ex-data cause))]
|
||||
(when (and (integer? n) (pos? n))
|
||||
(record-count metrics :audit-consumer-events [label "failed"] n)))
|
||||
(throw cause)))))]
|
||||
(let [label (:label consumer)
|
||||
process (:process consumer)
|
||||
metrics (mtx/instance cfg)
|
||||
totals (volatile! {:processed 0 :skipped 0 :chunks 0})
|
||||
result (with-tick-lock cfg consumer
|
||||
(fn [conn]
|
||||
(try
|
||||
(if-let [watermark (db/exec-one! conn [sql:last-visible])]
|
||||
(let [tpoint (ct/tpoint)
|
||||
drained (drain-chunks cfg conn consumer process watermark totals)
|
||||
_ (record-lag conn metrics consumer)
|
||||
_ (record-totals metrics label drained)
|
||||
_ (record-count metrics :audit-consumer-passes [label "ok"] 1)
|
||||
processed (:processed drained)
|
||||
skipped (:skipped drained)
|
||||
chunks (:chunks drained)
|
||||
;; Strict binding on purpose: log props evaluate lazily in
|
||||
;; the logger thread, so the elapsed must be frozen here.
|
||||
elapsed (ct/format-duration (tpoint))]
|
||||
(if (and (zero? processed)
|
||||
(zero? skipped))
|
||||
(l/trc :hint "no new audit events"
|
||||
:consumer label)
|
||||
(l/dbg :hint "audit events processed"
|
||||
:consumer label
|
||||
:elapsed elapsed
|
||||
:processed processed
|
||||
:skipped skipped
|
||||
:chunks chunks)))
|
||||
;; Nothing was visible when the pass started: no chunk query.
|
||||
(do
|
||||
(record-lag conn metrics consumer)
|
||||
(record-count metrics :audit-consumer-passes [label "ok"] 1)
|
||||
(l/trc :hint "no new audit events"
|
||||
:consumer label)
|
||||
nil))
|
||||
(catch Throwable cause
|
||||
;; The counter covers the whole pass: a failed frontier
|
||||
;; read or lag query counts as a failed pass too, not
|
||||
;; only a refused chunk.
|
||||
(record-totals metrics label @totals)
|
||||
(record-count metrics :audit-consumer-passes [label "failed"] 1)
|
||||
;; The failed event count comes from the error data: the
|
||||
;; send fns carry the attempted events as :events. A local
|
||||
;; DB failure carries none and is not counted as a
|
||||
;; destination failure.
|
||||
(let [n (:events (ex-data cause))]
|
||||
(when (and (integer? n) (pos? n))
|
||||
(record-count metrics :audit-consumer-events [label "failed"] n)))
|
||||
(throw cause)))))]
|
||||
(if (= ::busy result)
|
||||
(do
|
||||
(record-count metrics :audit-consumer-passes [label "busy"] 1)
|
||||
@ -456,17 +444,27 @@
|
||||
|
||||
(def ^:private sleep-ms
|
||||
"Pause after every pass. Long enough to keep the database quiet while the
|
||||
buffer is empty, short enough that an event waits seconds, not minutes."
|
||||
buffer is empty, short enough that an event waits seconds, not minutes.
|
||||
A consumer may declare its own `:pause-ms` in its `:options`."
|
||||
5000)
|
||||
|
||||
(defn run-pass
|
||||
"Run one pass of a consumer with its prepared cfg. Returns nil on success
|
||||
and publishes totals as metrics. Returns ::busy when another instance
|
||||
holds the tick or ::disabled when the flag is off."
|
||||
[cfg consumer]
|
||||
(check consumer)
|
||||
(drain cfg consumer))
|
||||
|
||||
(defn start-loop
|
||||
"Start the daemon thread of one consumer: a pass, a pause, forever.
|
||||
|
||||
`run-pass` gets the cfg and may throw: the loop logs the failure and
|
||||
keeps going, because a consumer that stops on the first refused chunk
|
||||
stops for good. Answers the thread, to be interrupted on halt."
|
||||
[cfg consumer-id run-pass]
|
||||
(let [label (consumer-label consumer-id)]
|
||||
The pass may throw: the loop logs the failure and keeps going, because a
|
||||
consumer that stops on the first refused chunk stops for good. Answers the
|
||||
thread, to be interrupted on halt."
|
||||
[cfg consumer]
|
||||
(check consumer)
|
||||
(let [label (:label consumer)]
|
||||
(px/thread
|
||||
{:name (str "penpot/audit-consumer/" label)}
|
||||
(l/inf :hint "started" :consumer label)
|
||||
@ -476,7 +474,7 @@
|
||||
(throw (InterruptedException. "interrupted")))
|
||||
|
||||
(try
|
||||
(run-pass cfg)
|
||||
(run-pass cfg consumer)
|
||||
(catch InterruptedException cause
|
||||
(throw cause))
|
||||
(catch Throwable cause
|
||||
@ -484,7 +482,7 @@
|
||||
:consumer label
|
||||
:cause cause)))
|
||||
|
||||
(px/sleep sleep-ms)
|
||||
(px/sleep (or (:pause-ms (:options consumer)) sleep-ms))
|
||||
(recur))
|
||||
|
||||
(catch InterruptedException _
|
||||
@ -494,35 +492,38 @@
|
||||
(finally
|
||||
(l/inf :hint "terminated" :consumer label))))))
|
||||
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; COMPONENT
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(defn init-component
|
||||
(defn init
|
||||
"Start the loop of a consumer when its flag is on and the database is
|
||||
writable; nil otherwise. `prepare` runs only on the start path and
|
||||
answers the cfg the pass will use (Nexus resolves its uri there).
|
||||
writable; nil otherwise. The optional `:prepare` of the definition runs
|
||||
only on the start path and answers the cfg the pass will use (Nexus
|
||||
resolves its uri there).
|
||||
|
||||
The cursor is registered after `prepare` succeeds, so a consumer whose
|
||||
configuration is rejected leaves no row behind."
|
||||
[cfg consumer-id prepare run-pass]
|
||||
[cfg consumer]
|
||||
(check consumer)
|
||||
(cond
|
||||
(not (enabled? consumer-id))
|
||||
(l/dbg :hint "not started (flag off)" :consumer (consumer-label consumer-id))
|
||||
(not (enabled? consumer))
|
||||
(l/dbg :hint "not started (flag off)" :consumer (:label consumer))
|
||||
|
||||
(db/read-only? (::db/pool cfg))
|
||||
(l/wrn :hint "not started (db is read-only)" :consumer (consumer-label consumer-id))
|
||||
(l/wrn :hint "not started (db is read-only)" :consumer (:label consumer))
|
||||
|
||||
:else
|
||||
(let [cfg (prepare cfg)]
|
||||
(let [cfg (if-let [prepare (:prepare consumer)]
|
||||
(prepare cfg)
|
||||
cfg)]
|
||||
;; Register the consumer before the loop reads anything: the GC waits
|
||||
;; for its cursor, so the row must exist even if the first pass has
|
||||
;; nothing visible to drain yet.
|
||||
(ensure-cursor cfg consumer-id)
|
||||
(start-loop cfg consumer-id run-pass))))
|
||||
(ensure-cursor cfg (:id consumer))
|
||||
(start-loop cfg consumer))))
|
||||
|
||||
(defn halt-component
|
||||
(defn halt
|
||||
"Stop the loop of a consumer and wait for it. Tolerates the nil an init
|
||||
that skipped the start answers."
|
||||
[thread]
|
||||
|
||||
@ -228,12 +228,7 @@
|
||||
rows)
|
||||
total (count rows)
|
||||
skipped (- total (count events))
|
||||
uri-host (::uri-host cfg)]
|
||||
(l/dbg :hint "consumer events chunk ready"
|
||||
:consumer "admin-console"
|
||||
:uri-host uri-host
|
||||
:total total
|
||||
:skipped skipped)
|
||||
host (::host cfg)]
|
||||
(if (empty? events)
|
||||
{:processed 0 :skipped skipped}
|
||||
;; Enrich first, outside the try: a local DB/decode failure is
|
||||
@ -245,14 +240,14 @@
|
||||
(nitrate/call cfg :ingest-audit-log {:events enriched})
|
||||
(l/dbg :hint "consumer events chunk ingested"
|
||||
:consumer "admin-console"
|
||||
:uri-host uri-host
|
||||
:host host
|
||||
:processed n
|
||||
:skipped skipped)
|
||||
{:processed n :skipped skipped}
|
||||
(catch Throwable cause
|
||||
(l/dbg :hint "consumer events chunk failed"
|
||||
:consumer "admin-console"
|
||||
:uri-host uri-host
|
||||
:host host
|
||||
:events n
|
||||
:code :admin-console-send-failed)
|
||||
(throw (ex/raise :type :internal
|
||||
@ -272,15 +267,6 @@
|
||||
:events (count decoded))]
|
||||
(send-events cfg decoded)))
|
||||
|
||||
(defn run-pass
|
||||
"One pass of the admin-console consumer: drain the stream from its cursor
|
||||
up to the frontier of the pass. The loop of `::consumer` calls it once
|
||||
per pass; direct callers (REPL, tests) get the same behavior."
|
||||
[cfg]
|
||||
(consumers/drain-consumer cfg
|
||||
consumers/admin-console
|
||||
process-chunk))
|
||||
|
||||
(defmethod ig/assert-key ::consumer
|
||||
[_ cfg]
|
||||
;; No assert on the admin-console client: it inits to nil without the
|
||||
@ -300,19 +286,24 @@
|
||||
(ex/raise :type :internal
|
||||
:code :task-not-configured
|
||||
:hint "admin-console consumer not configured, missing :admin-console-uri"))
|
||||
(let [uri-host (u/get-domain uri)]
|
||||
(let [host (u/get-domain uri)]
|
||||
(l/dbg :hint "audit log consumer initialized"
|
||||
:consumer "admin-console"
|
||||
:uri-host uri-host)
|
||||
(assoc cfg ::uri-host uri-host))))
|
||||
:host host)
|
||||
(assoc cfg ::host host))))
|
||||
|
||||
(def consumer
|
||||
"The allowlisted full rows travel to the admin console."
|
||||
{:id consumers/admin-console
|
||||
:flag :admin-console
|
||||
:label "admin-console"
|
||||
:prepare prepare
|
||||
:process process-chunk})
|
||||
|
||||
(defmethod ig/init-key ::consumer
|
||||
[_ cfg]
|
||||
(consumers/init-component cfg
|
||||
consumers/admin-console
|
||||
prepare
|
||||
run-pass))
|
||||
(consumers/init cfg consumer))
|
||||
|
||||
(defmethod ig/halt-key! ::consumer
|
||||
[_ thread]
|
||||
(consumers/halt-component thread))
|
||||
(consumers/halt thread))
|
||||
|
||||
@ -7,10 +7,12 @@
|
||||
(ns app.loggers.audit.consumers.nexus
|
||||
"The Nexus transport of the audit log consumer.
|
||||
|
||||
It reads the whole stream from the Nexus cursor (the oldest events
|
||||
first) and POSTs them to Nexus in batches. A batch Nexus refuses rolls
|
||||
back and stays for the next pass: the cursor does not advance, the pass
|
||||
aborts and the loop retries after its pause.
|
||||
It reads the full rows of the stream from the Nexus cursor (the oldest
|
||||
events first) and POSTs them to Nexus in batches. The telemetry shadows
|
||||
are not its business: they count as skipped and the cursor passes over
|
||||
them, because the anonymized copy belongs to the telemetry consumer. A
|
||||
batch Nexus refuses rolls back and stays for the next pass: the cursor
|
||||
does not advance, the pass aborts and the loop retries after its pause.
|
||||
|
||||
The component starts the consumer loop at boot when its flag is on. The
|
||||
mechanics (the tick mutex, the cursor, the frontier of the pass, the GC)
|
||||
@ -23,6 +25,7 @@
|
||||
[app.config :as cf]
|
||||
[app.db :as db]
|
||||
[app.http.client :as http]
|
||||
[app.loggers.audit :as audit]
|
||||
[app.loggers.audit.consumers :as consumers]
|
||||
[app.metrics :as mtx]
|
||||
[app.setup :as-alias setup]
|
||||
@ -39,15 +42,13 @@
|
||||
(apply cf/join-uri base segments)))
|
||||
|
||||
(defn- resolve-uri
|
||||
"The Nexus endpoint of this consumer: the override on the cfg, or the one
|
||||
built from `:nexus-uri`. Raises when neither is available, at boot and on
|
||||
every pass, so a consumer without a destination fails loudly instead of
|
||||
skipping silently. Failing the boot is deliberate: a worker that starts
|
||||
with its archive destination silently disabled is worse than one that
|
||||
refuses to start."
|
||||
[cfg]
|
||||
(or (some-> (::uri cfg) u/uri)
|
||||
(generate-uri "api/audit")
|
||||
"The Nexus endpoint of this consumer, built from `:nexus-uri`. Raises at
|
||||
boot when it is not available, so a consumer without a destination fails
|
||||
loudly instead of skipping silently.
|
||||
Failing the boot is deliberate: a worker that starts with its archive
|
||||
destination silently disabled is worse than one that refuses to start."
|
||||
[]
|
||||
(or (some-> (generate-uri "api/audit") u/uri)
|
||||
(ex/raise :type :internal
|
||||
:code :task-not-configured
|
||||
:hint "consumer task not configured, missing uri")))
|
||||
@ -65,8 +66,9 @@
|
||||
transport failure or non-204 refusal, carrying the attempted count
|
||||
as :events in the error data so the caller records the failure.
|
||||
No metrics or logs here: the loop reports the aborted pass."
|
||||
[{:keys [::uri] :as cfg} events]
|
||||
(let [n (count events)
|
||||
[cfg events]
|
||||
(let [uri (::uri cfg)
|
||||
n (count events)
|
||||
skey (-> cfg ::setup/shared-keys :nexus)
|
||||
body (t/encode {:events events})
|
||||
params {:uri uri
|
||||
@ -94,30 +96,29 @@
|
||||
:events n)))))
|
||||
|
||||
(defn- process-chunk
|
||||
"Send one chunk of the stream to Nexus. Returns {:processed n}; throws
|
||||
on refusal, so the chunk rolls back and the cursor stays where it was.
|
||||
No metrics here: a normal return means the transaction committed, so
|
||||
the caller records the success."
|
||||
"Send one chunk of the stream to Nexus. The telemetry shadows count as
|
||||
skipped: the full row already carries the event, and the anonymized copy
|
||||
belongs to the telemetry consumer (sending it here would archive the same
|
||||
event twice). Returns {:processed n :skipped n}; throws on refusal, so the
|
||||
chunk rolls back and the cursor stays where it was. No metrics here: a
|
||||
normal return means the transaction committed, so the caller records the
|
||||
success."
|
||||
[cfg rows]
|
||||
(let [events (mapv (comp consumers/row->event consumers/decode-row) rows)
|
||||
;; send-events returns the count or throws: a refusal
|
||||
;; aborts the pass, the rows stay for the next one.
|
||||
n (send-events cfg events)]
|
||||
(let [decoded (mapv consumers/decode-row rows)
|
||||
events (into []
|
||||
(comp (remove (comp audit/telemetry-shadow-source? :source))
|
||||
(map consumers/row->event))
|
||||
decoded)
|
||||
skipped (- (count decoded) (count events))
|
||||
n (if (seq events)
|
||||
(send-events cfg events)
|
||||
0)]
|
||||
|
||||
(l/trc :hint "consumer events chunk"
|
||||
:consumer "nexus"
|
||||
:events n)
|
||||
(when (pos? n)
|
||||
(l/trc :hint "events chunk sent"
|
||||
:events n))
|
||||
|
||||
{:processed n :skipped 0}))
|
||||
|
||||
(defn run-pass
|
||||
"One pass of the Nexus consumer: drain the stream from its cursor up to
|
||||
the frontier of the pass. The loop of `::consumer` calls it once per
|
||||
pass; direct callers (REPL, tests) get the same behavior."
|
||||
[cfg]
|
||||
(consumers/drain-consumer (assoc cfg ::uri (resolve-uri cfg))
|
||||
consumers/nexus
|
||||
process-chunk))
|
||||
{:processed n :skipped skipped}))
|
||||
|
||||
(defmethod ig/assert-key ::consumer
|
||||
[_ cfg]
|
||||
@ -130,19 +131,24 @@
|
||||
says where the consumer sends. An incomplete configuration raises here
|
||||
(see `resolve-uri`) and the worker does not start: intended."
|
||||
[cfg]
|
||||
(let [uri (resolve-uri cfg)]
|
||||
(let [uri (resolve-uri)]
|
||||
(l/dbg :hint "audit log consumer initialized"
|
||||
:consumer "nexus"
|
||||
:uri-host (u/get-domain uri))
|
||||
:host (u/get-domain uri))
|
||||
(assoc cfg ::uri uri)))
|
||||
|
||||
(def consumer
|
||||
"The full rows travel to Nexus."
|
||||
{:id consumers/nexus
|
||||
:flag :nexus
|
||||
:label "nexus"
|
||||
:prepare prepare
|
||||
:process process-chunk})
|
||||
|
||||
(defmethod ig/init-key ::consumer
|
||||
[_ cfg]
|
||||
(consumers/init-component cfg
|
||||
consumers/nexus
|
||||
prepare
|
||||
run-pass))
|
||||
(consumers/init cfg consumer))
|
||||
|
||||
(defmethod ig/halt-key! ::consumer
|
||||
[_ thread]
|
||||
(consumers/halt-component thread))
|
||||
(consumers/halt thread))
|
||||
|
||||
154
backend/src/app/loggers/audit/consumers/telemetry.clj
Normal file
154
backend/src/app/loggers/audit/consumers/telemetry.clj
Normal file
@ -0,0 +1,154 @@
|
||||
;; This Source Code Form is subject to the terms of the Mozilla Public
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.loggers.audit.consumers.telemetry
|
||||
"The telemetry transport of the audit log consumer.
|
||||
|
||||
It reads the stream from its own cursor (id 0) and sends the anonymized
|
||||
copies (the shadow rows, `source LIKE 'telemetry:%'`) to the telemetry
|
||||
endpoint. The full rows are not its business: they count as skipped and
|
||||
the cursor passes over them, so a deployment with any full-row consumer
|
||||
still ships its telemetry.
|
||||
|
||||
The payload is the one the legacy cron sent: the events alone, without
|
||||
the row id or the cursor columns. The endpoint always answers 200 and
|
||||
counts its own errors; a refusal rolls the chunk back, so the rows stay
|
||||
for the next pass and the cursor does not advance.
|
||||
|
||||
The component starts the consumer loop at boot when its flag is on. The
|
||||
mechanics (the tick mutex, the cursor, the frontier of the pass, the GC)
|
||||
live in `app.loggers.audit.consumers`."
|
||||
(:require
|
||||
[app.common.data :as d]
|
||||
[app.common.exceptions :as ex]
|
||||
[app.common.logging :as l]
|
||||
[app.common.uri :as u]
|
||||
[app.config :as cf]
|
||||
[app.db :as db]
|
||||
[app.http.client :as http]
|
||||
[app.loggers.audit :as audit]
|
||||
[app.loggers.audit.consumers :as consumers]
|
||||
[app.metrics :as mtx]
|
||||
[app.setup :as-alias setup]
|
||||
[app.util.json :as json]
|
||||
[integrant.core :as ig]))
|
||||
|
||||
(defn- resolve-uri
|
||||
"The telemetry endpoint of this consumer, configured as `:telemetry-uri`
|
||||
(it has a default). Raises at boot when it is not available, so a consumer
|
||||
without a destination fails loudly instead of skipping silently."
|
||||
[]
|
||||
(or (some-> (cf/get :telemetry-uri) u/uri)
|
||||
(ex/raise :type :internal
|
||||
:code :task-not-configured
|
||||
:hint "telemetry consumer not configured, missing uri")))
|
||||
|
||||
(defn- row->event
|
||||
"The view of a shadow row that travels to the endpoint, the same one the
|
||||
legacy cron built: no row id, no `created-at` and no `ip-addr` (the copy
|
||||
already carries `0.0.0.0`)."
|
||||
[{:keys [name type source tracked-at profile-id props context]}]
|
||||
(d/without-nils
|
||||
{:name name
|
||||
:type type
|
||||
:source source
|
||||
:tracked-at tracked-at
|
||||
:profile-id profile-id
|
||||
:props (or props {})
|
||||
:context (or context {})}))
|
||||
|
||||
(defn- send-events
|
||||
"POST a batch of events to the telemetry endpoint. Answers the number of
|
||||
events sent; throws on transport failure or a status above 206, carrying
|
||||
the attempted count as :events in the error data so the caller records the
|
||||
failure. No metrics or logs here: the pass reports the aborted batch."
|
||||
[cfg events]
|
||||
(let [n (count events)
|
||||
uri (::uri cfg)
|
||||
body (json/encode-str {:type :telemetry-events
|
||||
:version (:full cf/version)
|
||||
:instance-id (:instance-id (::setup/props cfg))
|
||||
:events events})
|
||||
params {:uri uri
|
||||
:timeout 12000
|
||||
:method :post
|
||||
:headers {"content-type" "application/json"}
|
||||
:body body}
|
||||
resp (try
|
||||
(http/req cfg params {:skip-ssrf-check? true})
|
||||
(catch Throwable cause
|
||||
(throw (ex/raise :type :internal
|
||||
:code :telemetry-send-failed
|
||||
:hint "unable to send events to the telemetry endpoint"
|
||||
:cause cause
|
||||
:events n))))]
|
||||
(if (<= (:status resp) 206)
|
||||
n
|
||||
(throw (ex/raise :type :internal
|
||||
:code :consumer-chunk-refused
|
||||
:hint "telemetry endpoint refused the batch"
|
||||
:resp-status (:status resp)
|
||||
:events n)))))
|
||||
|
||||
(defn- process-chunk
|
||||
"Send the shadow rows of one chunk to the endpoint. Returns
|
||||
{:processed n :skipped n}: the full rows count as skipped and the cursor
|
||||
advances over them, they are not its business. Throws on refusal, so the
|
||||
chunk rolls back and the cursor stays where it was."
|
||||
[cfg rows]
|
||||
(let [events (into []
|
||||
(comp (filter (comp audit/telemetry-shadow-source? :source))
|
||||
(map (comp row->event consumers/decode-row)))
|
||||
rows)
|
||||
total (count rows)]
|
||||
(when (seq events)
|
||||
(send-events cfg events)
|
||||
(l/trc :hint "events chunk sent"
|
||||
:events (count events)))
|
||||
{:processed (count events)
|
||||
:skipped (- total (count events))}))
|
||||
|
||||
(defn- prepare
|
||||
"Resolve the destination once at boot: per-pass logs carry counts, this
|
||||
says where the consumer sends. An incomplete configuration raises here and
|
||||
the worker does not start: intended."
|
||||
[cfg]
|
||||
(let [uri (resolve-uri)]
|
||||
(l/dbg :hint "audit log consumer initialized"
|
||||
:consumer "telemetry"
|
||||
:host (u/get-domain uri))
|
||||
(assoc cfg ::uri uri)))
|
||||
|
||||
(def options
|
||||
"The copies keep the pace of the legacy cron: one pass every three hours
|
||||
with up to 10 000 events per request. A pass every five seconds would
|
||||
multiply the requests to the endpoint by thousands."
|
||||
{:chunk-size 10000
|
||||
:pause-ms 10800000})
|
||||
|
||||
(def consumer
|
||||
"The anonymized copies travel to the telemetry endpoint."
|
||||
{:id consumers/telemetry
|
||||
:flag :telemetry
|
||||
:label "telemetry"
|
||||
:options options
|
||||
:prepare prepare
|
||||
:process process-chunk})
|
||||
|
||||
(defmethod ig/init-key ::consumer
|
||||
[_ cfg]
|
||||
(consumers/init cfg consumer))
|
||||
|
||||
(defmethod ig/halt-key! ::consumer
|
||||
[_ thread]
|
||||
(consumers/halt thread))
|
||||
|
||||
(defmethod ig/assert-key ::consumer
|
||||
[_ cfg]
|
||||
(assert (db/pool? (::db/pool cfg)) "expected a valid database pool")
|
||||
(assert (http/client? (::http/client cfg)) "expected a valid http client")
|
||||
(assert (mtx/metrics? (::mtx/metrics cfg)) "expected a valid metrics instance")
|
||||
(assert (some? (::setup/props cfg)) "expected setup props to be available"))
|
||||
@ -7,127 +7,265 @@
|
||||
(ns app.loggers.audit.gc
|
||||
"The garbage collection of the audit log buffer.
|
||||
|
||||
The producer only inserts rows and each consumer keeps its position,
|
||||
so somebody has to delete the rows every registered consumer has
|
||||
already passed. This job does it in batches below the oldest cursor,
|
||||
never touching the telemetry shadows (the telemetry task owns them).
|
||||
The producer only inserts rows and each consumer keeps its position, so
|
||||
somebody has to delete the rows every registered consumer has already
|
||||
passed. This job has two paths:
|
||||
|
||||
- the full rows, the normal behavior: up to the oldest cursor of the
|
||||
consumers that read them (every id except the telemetry consumer);
|
||||
- the telemetry shadows, up to the cursor of the telemetry consumer plus
|
||||
a retention window: copies older than seven days are deleted whatever
|
||||
the cursors say, so the buffer cannot keep them forever when the
|
||||
endpoint is down or the flag is off.
|
||||
|
||||
Each path has its own cursor and its own loop, so a slow telemetry pass
|
||||
every three hours does not retain the events Nexus already archived, and
|
||||
turning telemetry off does not stop the deletion of the full rows. The
|
||||
retention is the only deletion that does not wait for a consumer, and it
|
||||
can remove copies that were never sent.
|
||||
|
||||
Each loop stops starting batches when its budget expires. A batch
|
||||
already in progress can still commit; remaining rows wait for the next
|
||||
cron run.
|
||||
|
||||
The consumer framework (ids, flags and cursor helpers) lives in
|
||||
`app.loggers.audit.consumers`."
|
||||
(:require
|
||||
[app.common.logging :as l]
|
||||
[app.common.schema :as sm]
|
||||
[app.common.time :as ct]
|
||||
[app.db :as db]
|
||||
[app.jobs :as jobs]
|
||||
[app.loggers.audit.consumers :as consumers]
|
||||
[app.metrics :as mtx]
|
||||
[integrant.core :as ig]
|
||||
[promesa.exec :as px]))
|
||||
[integrant.core :as ig]))
|
||||
|
||||
(def ^:private batch-size
|
||||
(def ^:private ^:dynamic batch-size
|
||||
"Rows per delete: a big buffer is cleaned in bounded statements, each
|
||||
one its own snapshot, so the job heartbeats between batches."
|
||||
one its own snapshot, so the job heartbeats between batches. Dynamic so a
|
||||
test can bind a smaller one."
|
||||
5000)
|
||||
|
||||
(defn- get-oldest-cursor
|
||||
"The oldest position among every cursor in the table, or nil when there
|
||||
is none.
|
||||
(def ^:private ^:dynamic run-budget
|
||||
"Wall-clock budget for starting batches in one path. The paths run in
|
||||
series, each with its own clock. Dynamic so a test can bind a different one."
|
||||
(ct/duration {:seconds 120}))
|
||||
|
||||
(defn- budget-spent?
|
||||
"True when the time measured since `tpoint` (a `ct/tpoint` created at the
|
||||
start of the run) has reached `run-budget`."
|
||||
[tpoint]
|
||||
(not (pos? (compare run-budget (tpoint)))))
|
||||
|
||||
(def ^:private retention
|
||||
"How long an unsent telemetry shadow stays in the buffer. Copies older
|
||||
than this are deleted even when no consumer has passed them."
|
||||
"7 days")
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; THE TWO PATHS: THEIR CURSORS
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(def ^:private sql:full-cursor
|
||||
"The oldest position among the cursors of the full-row consumers. The
|
||||
telemetry consumer is excluded by its id: its cursor only holds the
|
||||
anonymized copies, which have their own path."
|
||||
"SELECT xid::text AS xid_text, seq
|
||||
FROM audit_log_consumer_cursor
|
||||
WHERE consumer <> ?::smallint
|
||||
ORDER BY xid ASC, seq ASC
|
||||
LIMIT 1")
|
||||
|
||||
(def ^:private sql:telemetry-cursor
|
||||
"SELECT xid::text AS xid_text, seq
|
||||
FROM audit_log_consumer_cursor
|
||||
WHERE consumer = ?::smallint")
|
||||
|
||||
(defn- get-oldest-full-cursor
|
||||
"The position of the full-row path, or nil when no consumer of that path
|
||||
has registered a cursor yet.
|
||||
|
||||
The set is not derived from the flags of the process that runs the job:
|
||||
the cursor table is shared by every instance, so a consumer disabled on
|
||||
this worker still holds the position it reached and keeps its unread
|
||||
rows alive. A consumer registers its (0, 0) cursor at boot, before its
|
||||
first pass, so the first row it has to read is never deleted under it.
|
||||
With no cursor at all the GC deletes nothing: it cannot know whether a
|
||||
consumer is about to start."
|
||||
this worker still holds the position it reached and keeps its unread rows
|
||||
alive."
|
||||
[cfg]
|
||||
(when-let [row (first (db/exec! cfg
|
||||
["SELECT xid::text AS xid_text, seq
|
||||
FROM audit_log_consumer_cursor
|
||||
ORDER BY xid ASC, seq ASC
|
||||
LIMIT 1"]))]
|
||||
(when-let [row (first (db/exec! cfg [sql:full-cursor consumers/telemetry]))]
|
||||
(select-keys row [:xid-text :seq])))
|
||||
|
||||
(def ^:private sql:delete-batch
|
||||
"Delete one batch of rows up to and including a position, telemetry
|
||||
shadows excluded: the cursor points at the last row a consumer
|
||||
processed, so that row is done too. The telemetry task owns the
|
||||
shadows (it deletes them after sending and purges the ones older than
|
||||
its retention window), so deleting them here would lose events before
|
||||
they are shipped.
|
||||
|
||||
The inner select walks the `(xid, seq)` index and skips locked rows:
|
||||
a producer never holds an old row, and two GC runs must not wait for
|
||||
each other."
|
||||
"DELETE FROM audit_log a
|
||||
WHERE a.id IN (SELECT a2.id
|
||||
FROM audit_log a2
|
||||
WHERE (a2.xid, a2.seq) <= (?::xid8, ?::bigint)
|
||||
AND a2.source NOT LIKE 'telemetry:%'
|
||||
ORDER BY a2.xid ASC, a2.seq ASC
|
||||
LIMIT ?
|
||||
FOR UPDATE SKIP LOCKED)")
|
||||
|
||||
(defn- delete-one-batch
|
||||
"Read the oldest cursor and delete one batch under the registry lock, in
|
||||
one transaction. The read is inside the batch, so a cursor registered
|
||||
between two batches is seen before the next delete. Answers
|
||||
`{:deleted n :found? bool}`."
|
||||
(defn- get-telemetry-cursor
|
||||
"The position of the telemetry path, or nil when its consumer has not
|
||||
registered yet. A missing cursor means its first pass has not run, so the
|
||||
cursor-based deletion of this path is idle (the retention still applies)."
|
||||
[cfg]
|
||||
(db/tx-run!
|
||||
cfg
|
||||
(fn [cfg]
|
||||
(consumers/lock-registry (::db/conn cfg))
|
||||
(if-let [cursor (get-oldest-cursor cfg)]
|
||||
{:deleted (or (db/get-update-count
|
||||
(db/exec-one! cfg [sql:delete-batch
|
||||
(:xid-text cursor) (:seq cursor)
|
||||
batch-size]))
|
||||
0)
|
||||
:found? true}
|
||||
{:deleted 0 :found? false}))))
|
||||
(when-let [row (db/exec-one! cfg [sql:telemetry-cursor consumers/telemetry])]
|
||||
(select-keys row [:xid-text :seq])))
|
||||
|
||||
(defn- delete-batched
|
||||
"Run a batched DELETE until it removes nothing, re-reading the oldest
|
||||
cursor on every batch. Answers `{:total n :found? bool}`."
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; THE TWO PATHS: THEIR DELETES
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(def ^:private sql:delete-full-batch
|
||||
"Delete one batch of full rows up to and including a position. The cursor
|
||||
points at the last row a consumer processed, so that row is done too.
|
||||
|
||||
Do not replace the materialized CTE with `id IN (SELECT ... LIMIT ?
|
||||
FOR UPDATE SKIP LOCKED)`: PostgreSQL can plan that form as a nested-loop
|
||||
semi join and rescan its locking subquery for each target row. As each
|
||||
rescan observes the rows locked or deleted by the preceding one, `LIMIT`
|
||||
becomes a per-rescan bound and one statement can delete more rows than its
|
||||
batch size.
|
||||
|
||||
The materialized CTE chooses the bounded set before the delete joins it,
|
||||
so the statement cannot delete more rows than its limit. Its query walks
|
||||
the `(xid, seq)` index and skips locked rows: a producer never holds an old
|
||||
row, and two GC runs must not wait for each other."
|
||||
"WITH doomed AS MATERIALIZED
|
||||
(SELECT a2.id
|
||||
FROM audit_log a2
|
||||
WHERE (a2.xid, a2.seq) <= (?::xid8, ?::bigint)
|
||||
AND a2.source NOT LIKE 'telemetry:%'
|
||||
ORDER BY a2.xid ASC, a2.seq ASC
|
||||
LIMIT ?
|
||||
FOR UPDATE SKIP LOCKED)
|
||||
DELETE FROM audit_log a
|
||||
USING doomed
|
||||
WHERE a.id = doomed.id")
|
||||
|
||||
(def ^:private sql:delete-shadow-batch
|
||||
"Delete one batch of telemetry shadows up to and including the position of
|
||||
the telemetry consumer. Same shape as the full-row statement, with the
|
||||
predicate reversed."
|
||||
"WITH doomed AS MATERIALIZED
|
||||
(SELECT a2.id
|
||||
FROM audit_log a2
|
||||
WHERE (a2.xid, a2.seq) <= (?::xid8, ?::bigint)
|
||||
AND a2.source LIKE 'telemetry:%'
|
||||
ORDER BY a2.xid ASC, a2.seq ASC
|
||||
LIMIT ?
|
||||
FOR UPDATE SKIP LOCKED)
|
||||
DELETE FROM audit_log a
|
||||
USING doomed
|
||||
WHERE a.id = doomed.id")
|
||||
|
||||
(def ^:private sql:delete-expired-shadows
|
||||
"Delete one batch of telemetry shadows past the retention window. It does
|
||||
not read a cursor: the window is the guarantee, whatever the consumers have
|
||||
passed, so the buffer stays bounded with the endpoint down or the flag off.
|
||||
Oldest first, so the batch always makes progress."
|
||||
(str "WITH doomed AS MATERIALIZED
|
||||
(SELECT a2.id
|
||||
FROM audit_log a2
|
||||
WHERE a2.source LIKE 'telemetry:%'
|
||||
AND a2.created_at < now() - interval '" retention "'
|
||||
ORDER BY a2.created_at ASC
|
||||
LIMIT ?
|
||||
FOR UPDATE SKIP LOCKED)
|
||||
DELETE FROM audit_log a
|
||||
USING doomed
|
||||
WHERE a.id = doomed.id"))
|
||||
|
||||
(defn- delete-batch
|
||||
"One batch of one path, in one transaction: read the position again (a
|
||||
cursor registered between two batches must be seen before the next delete)
|
||||
and delete up to it. Answers the number of deleted rows, 0 when the path has
|
||||
no cursor."
|
||||
[cfg get-cursor sql]
|
||||
(db/tx-run! cfg (fn [cfg]
|
||||
(if-let [cursor (get-cursor cfg)]
|
||||
(or (db/get-update-count
|
||||
(db/exec-one! cfg [sql (:xid-text cursor) (:seq cursor) batch-size]))
|
||||
0)
|
||||
0))))
|
||||
|
||||
(defn- check-interrupted
|
||||
[]
|
||||
(when (Thread/interrupted)
|
||||
(throw (InterruptedException.))))
|
||||
|
||||
(defn- run-path
|
||||
"Drain one path on the runner thread. Return the total committed before
|
||||
the path ran out of work or time, or met an ordinary failure. Propagate
|
||||
interruptions so the runner does not start another path."
|
||||
[cfg path batch]
|
||||
(let [tpoint (ct/tpoint)
|
||||
total (volatile! 0)]
|
||||
(try
|
||||
(loop []
|
||||
(check-interrupted)
|
||||
(when-not (budget-spent? tpoint)
|
||||
(let [deleted (long (batch))]
|
||||
(when (pos? deleted)
|
||||
(vswap! total + deleted)
|
||||
(consumers/record-count (mtx/instance cfg) :audit-consumer-discarded ["gc"] deleted))
|
||||
(check-interrupted)
|
||||
(when (pos? deleted)
|
||||
(jobs/heartbeat cfg)
|
||||
(Thread/sleep 100)
|
||||
(recur)))))
|
||||
(catch InterruptedException cause
|
||||
(throw cause))
|
||||
(catch Exception cause
|
||||
(when (= :interrupt (:type (ex-data cause)))
|
||||
(throw cause))
|
||||
(check-interrupted)
|
||||
(l/err :hint "audit log GC path failed" :path path :cause cause)))
|
||||
@total))
|
||||
|
||||
(defn- delete-full-rows
|
||||
"The normal path: the full rows every one of their consumers has passed."
|
||||
[cfg]
|
||||
(loop [total 0 found? false]
|
||||
(let [{:keys [deleted] :as batch} (delete-one-batch cfg)]
|
||||
(if (pos? (long deleted))
|
||||
(do
|
||||
(jobs/heartbeat cfg)
|
||||
(px/sleep 100)
|
||||
(recur (+ total (long deleted)) (or found? (:found? batch))))
|
||||
{:total total :found? (or found? (:found? batch))}))))
|
||||
(run-path cfg "full" #(delete-batch cfg get-oldest-full-cursor sql:delete-full-batch)))
|
||||
|
||||
(defn- delete-shadow-rows
|
||||
"The telemetry path: the shadows the telemetry consumer has passed."
|
||||
[cfg]
|
||||
(run-path cfg "shadow" #(delete-batch cfg get-telemetry-cursor sql:delete-shadow-batch)))
|
||||
|
||||
(defn- delete-expired-shadows
|
||||
"The telemetry retention window: shadows older than seven days, in
|
||||
batches, whatever the cursors say. Runs after the cursor paths with its
|
||||
own budget and answers the number of deleted rows."
|
||||
[cfg]
|
||||
(let [total (run-path cfg "shadow-expired"
|
||||
#(or (db/get-update-count
|
||||
(db/exec-one! cfg [sql:delete-expired-shadows batch-size]))
|
||||
0))]
|
||||
(when (pos? total)
|
||||
(l/warn :hint "purged stale telemetry events" :count total))
|
||||
total))
|
||||
|
||||
(defn delete-consumed
|
||||
"Delete the rows every registered consumer has already passed. The
|
||||
position is the oldest cursor, inclusive (the cursor points at the last
|
||||
row the consumer processed). The set is the cursor table, not the flags
|
||||
of this worker: a consumer that is off still holds the buffer (its
|
||||
cursor freezes and rejoins from wherever it is when it is enabled
|
||||
again), so a temporary disable never deletes events it has not read.
|
||||
"Delete the rows of the two paths, each one independently:
|
||||
|
||||
A consumer registers its (0, 0) cursor when its component starts, so a
|
||||
consumer that is enabled but has not drained yet still holds the buffer
|
||||
at position 0. With no cursor at all nothing is deleted: the
|
||||
`:audit-log-gc` flag is the deployment switch, and the GC waits for the
|
||||
consumers to register. Deleting the buffer behind the back of a
|
||||
consumer that may start later loses events; the disk risk is the
|
||||
operator's, not a reason to drain.
|
||||
- the full rows, up to the oldest cursor of the consumers that read them;
|
||||
- the telemetry shadows, up to the cursor of consumer 0.
|
||||
|
||||
Returns the number of discarded rows; they count in the `discarded`
|
||||
metric with reason `gc`."
|
||||
Then apply the retention window of the shadows.
|
||||
|
||||
The set is the cursor table, not the flags of this worker: a consumer that
|
||||
is off still holds its path (its cursor freezes and rejoins from wherever
|
||||
it is when it is enabled again), so a temporary disable never deletes
|
||||
events it has not read. A consumer registers its (0, 0) cursor when its
|
||||
component starts, so a consumer that is enabled but has not drained yet
|
||||
still holds its path at position 0.
|
||||
|
||||
The three paths run in order on the runner thread. Each one owns its
|
||||
wall-clock budget for starting batches, not for finishing them. An ordinary
|
||||
failure is logged and ends only that path, preserving its committed count.
|
||||
An interruption stops the whole pass.
|
||||
|
||||
Retention is the explicit exception: copies older than seven days are
|
||||
deleted even if no cursor passed them. They still count as removed by the
|
||||
GC. The warning counts rows deleted by age, whether sent or not.
|
||||
|
||||
Returns the total number of rows deleted. Each committed batch contributes
|
||||
to the `discarded` metric with reason `gc` before its heartbeat."
|
||||
[cfg]
|
||||
(let [{:keys [total found?]} (delete-batched cfg)]
|
||||
(if-not found?
|
||||
(do
|
||||
(l/wrn :hint "audit log GC is on with no consumer cursor; the buffer will not be cleaned")
|
||||
0)
|
||||
(do
|
||||
(when (pos? total)
|
||||
(consumers/record-count (mtx/instance cfg) :audit-consumer-discarded ["gc"] total))
|
||||
total))))
|
||||
(let [cfg (db/own-transaction cfg)]
|
||||
(+ (delete-full-rows cfg)
|
||||
(delete-shadow-rows cfg)
|
||||
(delete-expired-shadows cfg))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; JOB
|
||||
@ -150,9 +288,9 @@
|
||||
::jobs/schema schema:params
|
||||
::jobs/handler
|
||||
(fn [_context _params]
|
||||
;; The no-cursor branch deletes nothing on purpose: deleting would lose
|
||||
;; events for a consumer that may start later. Without the warning the
|
||||
;; operator only sees `discarded 0` at `dbg` and the buffer grows.
|
||||
;; A path with no cursor deletes nothing on purpose: deleting would lose
|
||||
;; events for a consumer that may start later. Retention still runs,
|
||||
;; because it is bounded by age, not by a position.
|
||||
(let [total (delete-consumed cfg)]
|
||||
(l/dbg :hint "audit log buffer cleaned" :discarded total)
|
||||
total))
|
||||
|
||||
@ -33,6 +33,7 @@
|
||||
;; the config keys).
|
||||
[app.loggers.audit.consumers.admin-console]
|
||||
[app.loggers.audit.consumers.nexus]
|
||||
[app.loggers.audit.consumers.telemetry]
|
||||
[app.loggers.audit.gc]
|
||||
[app.loggers.webhooks :as-alias webhooks]
|
||||
[app.metrics :as mtx]
|
||||
@ -370,7 +371,7 @@
|
||||
|
||||
:audit-consumer-discarded
|
||||
{::mdef/name "penpot_audit_consumer_discarded_total"
|
||||
::mdef/help "Audit log entries discarded without being archived, by reason."
|
||||
::mdef/help "Audit log entries removed from the buffer by the GC, by reason."
|
||||
::mdef/labels ["reason"]
|
||||
::mdef/type :counter}
|
||||
|
||||
@ -950,6 +951,12 @@
|
||||
:app.loggers.audit.consumers.admin-console/consumer
|
||||
{::db/pool (ig/ref ::db/pool)
|
||||
:app.nitrate/client (ig/ref :app.nitrate/client)
|
||||
::mtx/metrics (ig/ref ::mtx/metrics)}
|
||||
|
||||
:app.loggers.audit.consumers.telemetry/consumer
|
||||
{::db/pool (ig/ref ::db/pool)
|
||||
::http.client/client (ig/ref ::http.client/client)
|
||||
::setup/props (ig/ref ::setup/props)
|
||||
::mtx/metrics (ig/ref ::mtx/metrics)}})
|
||||
|
||||
|
||||
|
||||
@ -16,8 +16,9 @@
|
||||
-- The legacy mark (`archived_at`) and the indexes that served it stay in
|
||||
-- place on purpose: the old archiver may still be running during the
|
||||
-- rollout, and a later release removes them. This migration does not build
|
||||
-- the `(created_at, id)` index either; the consumers order by `(xid, seq)`
|
||||
-- and telemetry keeps its `(source, created_at)` index plus a sort.
|
||||
-- the `(created_at, id)` index either; the consumers order by `(xid, seq)`,
|
||||
-- so nothing reads by `(source, created_at)` any more and that index waits
|
||||
-- for a later cleanup.
|
||||
--
|
||||
-- The migration is idempotent: every statement is IF NOT EXISTS / IF
|
||||
-- EXISTS, so it is safe to re-run. Applying it out of band does not make
|
||||
@ -102,8 +103,9 @@ ALTER TABLE audit_log
|
||||
-- `source` is nullable since 0064 (its backfill only covered the rows of
|
||||
-- that moment). The producer has written it ever since, but the GC excludes
|
||||
-- telemetry shadows with `source NOT LIKE 'telemetry:%'`, and with a NULL
|
||||
-- source that predicate is unknown: neither the GC nor the telemetry task
|
||||
-- would ever delete such a row. Fill what is left and close the column.
|
||||
-- source that predicate is unknown: neither the cursor paths nor the
|
||||
-- retention window would ever delete such a row. Fill what is left and close
|
||||
-- the column.
|
||||
UPDATE audit_log SET source = 'backend' WHERE source IS NULL;
|
||||
|
||||
ALTER TABLE audit_log
|
||||
|
||||
@ -154,6 +154,13 @@
|
||||
ORDER BY 2 DESC")
|
||||
|
||||
(defn- get-action-counters
|
||||
"The event counters of the legacy report: an aggregate of the telemetry
|
||||
shadows that are still in the buffer when the report runs. The number
|
||||
already depended on the flags (the whole day when no full-row consumer
|
||||
persisted, only since the previous run otherwise); now the consumer ships
|
||||
the shadows and the GC deletes them, so it reflects what the buffer still
|
||||
holds. The per-event data reaches Nexus as `telemetry-events` from the
|
||||
consumer; this report does not carry it."
|
||||
[conn]
|
||||
(let [counters (->> (db/exec! conn [sql:get-counters])
|
||||
(d/index-by (comp keyword :name) :count))
|
||||
@ -210,99 +217,6 @@
|
||||
|
||||
(make-legacy-request cfg data)))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; AUDIT-EVENT BATCH (TELEMETRY MODE)
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
;; Telemetry events older than this are purged by the GC step so the
|
||||
;; buffer stays bounded.
|
||||
(def ^:private batch-size 10000)
|
||||
|
||||
(def ^:private sql:aged-telemetry
|
||||
"source LIKE 'telemetry:%' AND created_at < now() - interval '7 days'")
|
||||
|
||||
(def ^:private sql:gc-events
|
||||
(str "DELETE FROM audit_log WHERE " sql:aged-telemetry))
|
||||
|
||||
(defn- gc-events
|
||||
"Delete telemetry-mode events older than `telemetry-retention-days`
|
||||
so that the buffer stays bounded. The telemetry shadows are outside
|
||||
the consumer GC on purpose: this task owns them."
|
||||
[{:keys [::db/conn]}]
|
||||
(let [result (db/exec-one! conn [sql:gc-events])]
|
||||
(when (pos? (:next.jdbc/update-count result))
|
||||
(l/warn :hint "purged stale telemetry events"
|
||||
:count (:next.jdbc/update-count result)))))
|
||||
|
||||
(def ^:private sql:fetch-telemetry-events
|
||||
"SELECT id, name, type, source, tracked_at, profile_id, props, context
|
||||
FROM audit_log
|
||||
WHERE source LIKE 'telemetry:%'
|
||||
ORDER BY created_at ASC
|
||||
LIMIT ?")
|
||||
|
||||
(defn- row->event
|
||||
[{:keys [name type source tracked-at profile-id props context]}]
|
||||
(d/without-nils
|
||||
{:name name
|
||||
:type type
|
||||
:source source
|
||||
:tracked-at tracked-at
|
||||
:profile-id profile-id
|
||||
:props (or (some-> props db/decode-transit-pgobject) {})
|
||||
:context (or (some-> context db/decode-transit-pgobject) {})}))
|
||||
|
||||
(defn send-event-batch
|
||||
"Send a single batch of events to the telemetry endpoint. Returns
|
||||
true on success. The events are sent as a plain vector of event
|
||||
maps; the JSON encoder handles UUID and temporal types natively and
|
||||
the receiver coerces them back to proper types."
|
||||
[{:keys [::setup/props] :as cfg} batch]
|
||||
(let [payload {:type :telemetry-events
|
||||
:version (:full cf/version)
|
||||
:instance-id (:instance-id props)
|
||||
:events (vec batch)}
|
||||
request {:method :post
|
||||
:uri (cf/get :telemetry-uri)
|
||||
:headers {"content-type" "application/json"}
|
||||
:body (json/encode-str payload)}
|
||||
resp (http/req cfg request {:skip-ssrf-check? true})]
|
||||
(if (<= (:status resp) 206)
|
||||
true
|
||||
(do
|
||||
(l/warn :hint "telemetry event batch send failed"
|
||||
:status (:status resp)
|
||||
:body (:body resp))
|
||||
false))))
|
||||
|
||||
(defn- delete-sent-events
|
||||
"Delete rows by their ids after a successful send. The telemetry
|
||||
shadows are outside the consumer GC, so this task is their only
|
||||
deleter: it can remove a row a consumer has not read yet (the cursor
|
||||
simply skips over the missing position)."
|
||||
[conn ids]
|
||||
(let [arr (db/create-array conn "uuid" ids)]
|
||||
(db/exec-one! conn ["DELETE FROM audit_log WHERE id = ANY(?)" arr])))
|
||||
|
||||
(defn- collect-and-send-audit-events
|
||||
"Collect anonymous telemetry-mode audit events and ship them to the
|
||||
telemetry endpoint in a loop. Each iteration fetches one page of
|
||||
`batch-size` rows, encodes and sends them, then deletes the rows on
|
||||
success. The loop stops as soon as a send returns false, leaving
|
||||
remaining rows intact for the next run."
|
||||
[{:keys [::db/conn] :as cfg}]
|
||||
(loop [counter 1]
|
||||
(when-let [rows (-> (db/exec! conn [sql:fetch-telemetry-events batch-size])
|
||||
(not-empty))]
|
||||
(let [events (mapv row->event rows)
|
||||
ids (mapv :id rows)]
|
||||
(l/dbg :hint "shipping telemetry event batch"
|
||||
:total (count events)
|
||||
:batch counter)
|
||||
(when (send-event-batch cfg events)
|
||||
(delete-sent-events conn ids)
|
||||
(recur (inc counter)))))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; TASK ENTRY POINT
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
@ -347,7 +261,6 @@
|
||||
|
||||
(if enabled?
|
||||
(when send?
|
||||
(db/run! cfg gc-events)
|
||||
;; Randomize start time to avoid thundering herd when multiple
|
||||
;; instances restart at the same time.
|
||||
(px/sleep (rand-int 10000))
|
||||
@ -357,16 +270,7 @@
|
||||
(send-legacy-data cfg stats @subs))
|
||||
(catch Exception cause
|
||||
(l/wrn :hint "unable to send legacy report"
|
||||
:cause cause)))
|
||||
|
||||
;; Ship any anonymous audit-log events accumulated in
|
||||
;; telemetry mode (only when the buffer has no consumer).
|
||||
(when-not cf/audit-persist?
|
||||
(try
|
||||
(db/run! cfg collect-and-send-audit-events)
|
||||
(catch Exception cause
|
||||
(l/wrn :hint "unable to send events"
|
||||
:cause cause)))))
|
||||
:cause cause))))
|
||||
|
||||
;; If we have telemetry disabled, but there are users that are
|
||||
;; explicitly checked the newsletter subscription on the
|
||||
|
||||
@ -176,7 +176,8 @@
|
||||
:app.worker/cron
|
||||
:app.worker/dispatcher
|
||||
:app.loggers.audit.consumers.nexus/consumer
|
||||
:app.loggers.audit.consumers.admin-console/consumer)
|
||||
:app.loggers.audit.consumers.admin-console/consumer
|
||||
:app.loggers.audit.consumers.telemetry/consumer)
|
||||
(remove-job-runners))
|
||||
_ (ig/load-namespaces system)
|
||||
system (-> (ig/expand system) (ig/init))]
|
||||
|
||||
@ -22,6 +22,7 @@
|
||||
[app.loggers.audit.consumers :as consumer]
|
||||
[app.loggers.audit.consumers.admin-console :as consumer.admin-console]
|
||||
[app.loggers.audit.consumers.nexus :as consumer.nexus]
|
||||
[app.loggers.audit.consumers.telemetry :as consumer.telemetry]
|
||||
[app.loggers.audit.gc :as gc]
|
||||
[app.main :as main]
|
||||
[app.metrics :as mtx]
|
||||
@ -50,12 +51,13 @@
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(defn- seed-cursors
|
||||
"Both cursors exist at (0, 0) before every test: the database reset
|
||||
"Every cursor exists at (0, 0) before every test: the database reset
|
||||
deletes them, so seeding keeps the starting state deterministic, lets
|
||||
the GC delete without waiting for a first pass, and leaves the
|
||||
'no cursor' case to the tests that create it on purpose."
|
||||
[next]
|
||||
(doseq [consumer-id [consumer/nexus
|
||||
(doseq [consumer-id [consumer/telemetry
|
||||
consumer/nexus
|
||||
consumer/admin-console]]
|
||||
(th/db-exec! ["INSERT INTO audit_log_consumer_cursor (consumer, xid, seq)
|
||||
VALUES (?::smallint, '0'::xid8, 0)"
|
||||
@ -106,30 +108,80 @@
|
||||
::setup/shared-keys {:admin-console "ac-key" :nexus "nexus-key"}
|
||||
::http/client (:app.http.client/client th/*system*)
|
||||
::mtx/metrics metrics}
|
||||
uri (assoc ::consumer.nexus/uri uri))))
|
||||
uri (assoc ::consumer.nexus/uri (u/uri uri)))))
|
||||
|
||||
(defn- run-nexus
|
||||
([] (run-nexus {}))
|
||||
([params]
|
||||
(consumer.nexus/run-pass
|
||||
(nexus-cfg (:uri params)))))
|
||||
(consumer/run-pass
|
||||
(nexus-cfg (:uri params))
|
||||
consumer.nexus/consumer)))
|
||||
|
||||
(defn- admin-console-cfg
|
||||
[metrics]
|
||||
{::db/pool (:app.db/pool th/*system*)
|
||||
:app.nitrate/client (:app.nitrate/client th/*system*)
|
||||
::consumer.admin-console/uri-host "admin-console.example"
|
||||
::consumer.admin-console/host "admin-console.example"
|
||||
::mtx/metrics metrics})
|
||||
|
||||
(defn- run-admin-console
|
||||
[]
|
||||
(consumer.admin-console/run-pass
|
||||
(admin-console-cfg (:app.metrics/metrics th/*system*))))
|
||||
(consumer/run-pass
|
||||
(admin-console-cfg (:app.metrics/metrics th/*system*))
|
||||
consumer.admin-console/consumer))
|
||||
|
||||
(def ^:private telemetry-uri "http://telemetry.example/")
|
||||
|
||||
(defn- telemetry-cfg
|
||||
([uri] (telemetry-cfg uri (:app.metrics/metrics th/*system*)))
|
||||
([uri metrics]
|
||||
(cond-> {::db/pool (:app.db/pool th/*system*)
|
||||
::http/client (:app.http.client/client th/*system*)
|
||||
::setup/props {:instance-id (uuid/next)}
|
||||
::mtx/metrics metrics}
|
||||
uri (assoc ::consumer.telemetry/uri (u/uri uri)))))
|
||||
|
||||
(defn- run-telemetry
|
||||
([] (run-telemetry {}))
|
||||
([params]
|
||||
(consumer/run-pass
|
||||
(telemetry-cfg (:uri params))
|
||||
consumer.telemetry/consumer)))
|
||||
|
||||
(defn- insert-shadow-row
|
||||
"Insert one anonymized copy, the row the telemetry consumer sends, at an
|
||||
explicit `created-at` so the retention tests can age it."
|
||||
([profile-id name] (insert-shadow-row profile-id name (ct/now)))
|
||||
([profile-id name created-at]
|
||||
(th/db-insert! :audit-log
|
||||
{:id (uuid/next)
|
||||
:name name
|
||||
:type "action"
|
||||
:source "telemetry:backend"
|
||||
:profile-id profile-id
|
||||
:ip-addr (db/inet "0.0.0.0")
|
||||
:props (db/tjson {})
|
||||
:context (db/tjson {})
|
||||
:tracked-at created-at
|
||||
:created-at created-at})))
|
||||
|
||||
(defn- run-gc
|
||||
[]
|
||||
(th/run-task! :audit-log-gc {}))
|
||||
|
||||
(t/deftest telemetry-only-gc-cleans-shadows-and-keeps-pending-full-rows
|
||||
(th/db-exec! ["DELETE FROM audit_log_consumer_cursor"])
|
||||
(with-redefs [cf/flags #{:telemetry}]
|
||||
(with-mocks [http-mock {:target 'app.http.client/req :return {:status 200}}]
|
||||
(insert-audit-row uuid/zero "pending-full")
|
||||
(insert-shadow-row uuid/zero "confirmed")
|
||||
(run-telemetry {:uri telemetry-uri})
|
||||
(insert-shadow-row uuid/zero "expired"
|
||||
(ct/minus (ct/now) (ct/duration {:days 8})))
|
||||
(t/is (= 2 (run-gc)))
|
||||
(t/is (= ["pending-full"]
|
||||
(mapv :name (th/db-exec! ["SELECT name FROM audit_log"])))))))
|
||||
|
||||
(defn- admin-console-event
|
||||
[admin-console-mock]
|
||||
(let [[_ _ params] (:call-args @admin-console-mock)]
|
||||
@ -232,6 +284,34 @@
|
||||
(t/is (= names (mapv :name events))))
|
||||
(t/is (= 0 (unread-count consumer/nexus)))))))
|
||||
|
||||
(t/deftest consumer-nexus-skips-the-telemetry-shadows
|
||||
;; The anonymized copy belongs to the telemetry consumer: Nexus must not
|
||||
;; archive the same event twice, so the shadow counts as skipped and the
|
||||
;; cursor passes over it.
|
||||
(let [prof (th/create-profile* 5 {:is-active true})]
|
||||
(with-redefs [cf/flags #{:nexus}]
|
||||
(with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}]
|
||||
(insert-audit-row (:id prof) "create-project")
|
||||
(insert-shadow-row (:id prof) "create-project")
|
||||
(run-nexus {:uri archive-uri})
|
||||
(let [events (-> @http-mock :call-args second :body transit/decode :events)]
|
||||
(t/is (= ["create-project"] (mapv :name events)))
|
||||
(t/is (= ["backend"] (mapv :source events))))
|
||||
;; The cursor passed both rows: the shadow was skipped, not sent.
|
||||
(t/is (= 0 (unread-count consumer/nexus)))
|
||||
(t/is (= 2 (count (th/db-exec! ["SELECT * FROM audit_log"]))))))))
|
||||
|
||||
(t/deftest consumer-nexus-does-not-post-a-chunk-with-only-shadows
|
||||
(let [prof (th/create-profile* 6 {:is-active true})]
|
||||
(with-redefs [cf/flags #{:nexus}]
|
||||
(with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}]
|
||||
(insert-shadow-row (:id prof) "create-project")
|
||||
(insert-shadow-row (:id prof) "navigate")
|
||||
(run-nexus {:uri archive-uri})
|
||||
(t/is (not (:called? @http-mock)))
|
||||
(t/is (= 0 (unread-count consumer/nexus)))
|
||||
(t/is (= 2 (count (th/db-exec! ["SELECT * FROM audit_log"]))))))))
|
||||
|
||||
(t/deftest consumer-nexus-default-uri-appends-api-audit
|
||||
(let [prof (th/create-profile* 3 {:is-active true})
|
||||
uris (atom [])]
|
||||
@ -242,7 +322,8 @@
|
||||
(swap! uris conj (str (:uri params)))
|
||||
{:status 204})}]
|
||||
(insert-audit-row (:id prof) "create-project")
|
||||
(run-nexus {})
|
||||
(consumer/run-pass ((:prepare consumer.nexus/consumer) (nexus-cfg nil))
|
||||
consumer.nexus/consumer)
|
||||
(t/is (= ["http://nexus.example/api/audit"] @uris))))))
|
||||
|
||||
(t/deftest consumer-nexus-component-requires-uri-when-flag-set
|
||||
@ -369,7 +450,7 @@
|
||||
(let [team-id (uuid/next)
|
||||
row (insert-audit-row (:id prof) "create-project"
|
||||
{:team-id team-id})]
|
||||
(consumer.admin-console/run-pass consumer-cfg)
|
||||
(consumer/run-pass consumer-cfg consumer.admin-console/consumer)
|
||||
(t/is (= 1 (count @admin-console-bodies)))
|
||||
(let [body (json/decode (first @admin-console-bodies) :key-fn json/read-kebab-key)
|
||||
event (first (:events body))]
|
||||
@ -402,7 +483,7 @@
|
||||
(with-mocks [http-mock {:target 'app.http.client/req
|
||||
:return {:status 200 :body "{}"}}]
|
||||
(insert-audit-row (:id prof) "create-project")
|
||||
(consumer.admin-console/run-pass consumer-cfg)
|
||||
(consumer/run-pass consumer-cfg consumer.admin-console/consumer)
|
||||
(t/is (= 1 (:call-count @http-mock)))
|
||||
(t/is (= 0 (unread-count consumer/admin-console))))))))
|
||||
|
||||
@ -594,6 +675,126 @@
|
||||
(t/is (= (:id team) (:team-id props)))
|
||||
(t/is (= (:name team) (:team-name props))))))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; TELEMETRY: THE ANONYMIZED COPIES
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(t/deftest consumer-telemetry-sends-only-the-anonymized-copies
|
||||
(let [prof (th/create-profile* 70 {:is-active true})]
|
||||
(with-redefs [cf/flags #{:telemetry}]
|
||||
(with-mocks [http-mock {:target 'app.http.client/req :return {:status 200}}]
|
||||
(insert-audit-row (:id prof) "create-project")
|
||||
(insert-shadow-row (:id prof) "create-project")
|
||||
(insert-shadow-row (:id prof) "navigate")
|
||||
(t/is (= 3 (unread-count consumer/telemetry)))
|
||||
(run-telemetry {:uri telemetry-uri})
|
||||
(t/is (= 1 (:call-count @http-mock)))
|
||||
(let [body (json/decode (-> @http-mock :call-args second :body)
|
||||
:key-fn json/read-kebab-key)]
|
||||
(t/is (= "telemetry-events" (name (:type body))))
|
||||
(t/is (= (:full cf/version) (:version body)))
|
||||
(t/is (some? (:instance-id body)))
|
||||
(t/is (= ["create-project" "navigate"] (mapv :name (:events body))))
|
||||
(t/is (= #{"telemetry:backend"} (into #{} (map :source) (:events body)))))
|
||||
;; The cursor passed the full row too: it was skipped, not sent.
|
||||
(t/is (= 0 (unread-count consumer/telemetry)))
|
||||
(t/is (= 1 (count (th/db-exec! ["SELECT * FROM audit_log
|
||||
WHERE source NOT LIKE 'telemetry:%'"]))))))))
|
||||
|
||||
(t/deftest consumer-telemetry-payload-keeps-the-shape-of-the-legacy-cron
|
||||
(let [prof (th/create-profile* 71 {:is-active true})]
|
||||
(with-redefs [cf/flags #{:telemetry}]
|
||||
(with-mocks [http-mock {:target 'app.http.client/req :return {:status 200}}]
|
||||
(insert-shadow-row (:id prof) "navigate")
|
||||
(run-telemetry {:uri telemetry-uri})
|
||||
(let [body (json/decode (-> @http-mock :call-args second :body)
|
||||
:key-fn json/read-kebab-key)
|
||||
event (first (:events body))]
|
||||
(t/is (= #{:name :type :source :tracked-at :profile-id :props :context}
|
||||
(set (keys event))))
|
||||
(t/is (string? (:tracked-at event)))
|
||||
(t/is (= {} (:props event)))
|
||||
(t/is (= {} (:context event)))
|
||||
(t/is (not (contains? event :id)))
|
||||
(t/is (not (contains? event :created-at)))
|
||||
(t/is (not (contains? event :ip-addr))))))))
|
||||
|
||||
(t/deftest consumer-telemetry-refused-chunk-keeps-the-cursor
|
||||
(let [prof (th/create-profile* 72 {:is-active true})]
|
||||
(with-redefs [cf/flags #{:telemetry}]
|
||||
(with-mocks [http-mock {:target 'app.http.client/req :return {:status 500}}]
|
||||
(insert-shadow-row (:id prof) "create-project")
|
||||
(let [data (ex-data (try
|
||||
(run-telemetry {:uri telemetry-uri})
|
||||
(catch Throwable cause
|
||||
cause)))]
|
||||
(t/is (= :consumer-chunk-refused (:code data)))
|
||||
(t/is (= 1 (:events data))))
|
||||
(t/is (= 1 (unread-count consumer/telemetry)))
|
||||
(t/is (= 1 (count (th/db-exec! ["SELECT * FROM audit_log"]))))))))
|
||||
|
||||
(t/deftest consumer-telemetry-pass-with-its-flag-off-sends-nothing
|
||||
(with-redefs [cf/flags #{}]
|
||||
(with-mocks [http-mock {:target 'app.http.client/req :return {:status 200}}]
|
||||
(insert-shadow-row uuid/zero "create-project")
|
||||
(t/is (= :app.loggers.audit.consumers/disabled
|
||||
(consumer/run-pass (telemetry-cfg telemetry-uri)
|
||||
consumer.telemetry/consumer)))
|
||||
(t/is (not (:called? @http-mock))))))
|
||||
|
||||
(t/deftest consumer-telemetry-chunks-by-its-own-size
|
||||
;; The catalog declares 10000 rows per chunk for telemetry. Shrinking the
|
||||
;; shared var (as the nexus test does) must not shrink it: the whole chunk
|
||||
;; travels in one request.
|
||||
(let [prof (th/create-profile* 73 {:is-active true})
|
||||
names (mapv #(str "telemetry-" %) (range 1 6))]
|
||||
(with-redefs [cf/flags #{:telemetry}
|
||||
consumer/chunk-size 3]
|
||||
(with-mocks [http-mock {:target 'app.http.client/req :return {:status 200}}]
|
||||
(doseq [name names]
|
||||
(insert-shadow-row (:id prof) name))
|
||||
(run-telemetry {:uri telemetry-uri})
|
||||
(t/is (= 1 (:call-count @http-mock)))
|
||||
(let [body (json/decode (-> @http-mock :call-args second :body)
|
||||
:key-fn json/read-kebab-key)]
|
||||
(t/is (= names (mapv :name (:events body)))))))))
|
||||
|
||||
(t/deftest consumer-telemetry-component-requires-uri-when-flag-set
|
||||
(with-redefs [cf/flags #{:telemetry}
|
||||
cf/config (dissoc cf/config :telemetry-uri)]
|
||||
(let [data (ex-data (try
|
||||
(ig/init-key :app.loggers.audit.consumers.telemetry/consumer
|
||||
(telemetry-cfg nil))
|
||||
(catch Throwable cause
|
||||
cause)))]
|
||||
(t/is (= :task-not-configured (:code data))))))
|
||||
|
||||
(t/deftest consumer-telemetry-component-does-not-start-with-the-flag-off
|
||||
(with-redefs [cf/flags #{}]
|
||||
(t/is (nil? (ig/init-key :app.loggers.audit.consumers.telemetry/consumer
|
||||
(telemetry-cfg telemetry-uri))))))
|
||||
|
||||
(t/deftest consumer-loop-uses-the-pause-of-the-consumer
|
||||
;; Each consumer declares its cadence: telemetry keeps the three hours of
|
||||
;; the legacy cron, the others keep the five seconds of the framework. The
|
||||
;; test wraps the real pause (never replaces it) so halting still has to
|
||||
;; interrupt the thread for real.
|
||||
(let [real-sleep px/sleep]
|
||||
(doseq [[definition expected] [[consumer.telemetry/consumer 10800000]
|
||||
[consumer.nexus/consumer 5000]]]
|
||||
(let [entered (promise)
|
||||
paused (atom nil)]
|
||||
(with-redefs [px/sleep (fn [ms]
|
||||
(reset! paused ms)
|
||||
(deliver entered true)
|
||||
(real-sleep ms))
|
||||
consumer/run-pass (fn [_cfg _consumer] nil)]
|
||||
(let [thread (consumer/start-loop {} definition)]
|
||||
(t/is (true? (deref entered 5000 false)))
|
||||
(consumer/halt thread)
|
||||
(t/is (not (.isAlive ^Thread thread)))
|
||||
(t/is (= expected @paused))))))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; THE SNAPSHOT CUT: A ROW OF AN OPEN TRANSACTION IS NEVER SKIPPED
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
@ -755,8 +956,9 @@
|
||||
(defn- run-nexus-on
|
||||
"Run one nexus pass merging `overrides` over the test cfg."
|
||||
[overrides]
|
||||
(consumer.nexus/run-pass
|
||||
(merge (nexus-cfg archive-uri) overrides)))
|
||||
(consumer/run-pass
|
||||
(merge (nexus-cfg archive-uri) overrides)
|
||||
consumer.nexus/consumer))
|
||||
|
||||
(t/deftest consumer-pass-needs-one-connection
|
||||
;; The frontier, every chunk transaction and the lag gauge run on the
|
||||
@ -828,18 +1030,6 @@
|
||||
(t/is (false? (:called? @http-mock)))
|
||||
(t/is (nil? (cursor-of consumer/nexus)))))))
|
||||
|
||||
(t/deftest drain-consumer-rejects-an-unknown-id
|
||||
;; The flag lookup answers false for an id outside the registry, so the
|
||||
;; unknown-consumer raise must happen first: a typo must fail, not silently
|
||||
;; disable the pass.
|
||||
(let [data (try
|
||||
(consumer/drain-consumer (nexus-cfg nil) 999 (fn [_ _] nil))
|
||||
nil
|
||||
(catch clojure.lang.ExceptionInfo e (ex-data e)))]
|
||||
(t/is (= :internal (:type data)))
|
||||
(t/is (= :unknown-audit-consumer (:code data)))
|
||||
(t/is (= 999 (:consumer data)))))
|
||||
|
||||
(t/deftest admin-console-pass-with-its-flag-off-sends-nothing
|
||||
;; `nitrate/call` returns nil when `:admin-console` is off, so without the
|
||||
;; gate the pass would count the nil as success and advance the cursor
|
||||
@ -881,12 +1071,10 @@
|
||||
|
||||
(t/deftest consumer-loop-runs-passes-until-interrupted
|
||||
(let [passes (atom 0)]
|
||||
(with-redefs-fn {#'consumer/sleep-ms 10}
|
||||
(fn []
|
||||
(let [thread (consumer/start-loop
|
||||
{}
|
||||
consumer/nexus
|
||||
(fn [_] (swap! passes inc)))]
|
||||
(with-redefs [consumer/sleep-ms 10
|
||||
consumer/run-pass (fn [_cfg _consumer] (swap! passes inc))]
|
||||
(do
|
||||
(let [thread (consumer/start-loop {} consumer.nexus/consumer)]
|
||||
(try
|
||||
(loop [i 0]
|
||||
(when (and (< @passes 3) (< i 300))
|
||||
@ -900,14 +1088,12 @@
|
||||
|
||||
(t/deftest consumer-loop-survives-a-failing-pass
|
||||
(let [passes (atom 0)]
|
||||
(with-redefs-fn {#'consumer/sleep-ms 10}
|
||||
(fn []
|
||||
(let [thread (consumer/start-loop
|
||||
{}
|
||||
consumer/nexus
|
||||
(fn [_]
|
||||
(swap! passes inc)
|
||||
(throw (ex-info "boom" {}))))]
|
||||
(with-redefs [consumer/sleep-ms 10
|
||||
consumer/run-pass (fn [_cfg _consumer]
|
||||
(swap! passes inc)
|
||||
(throw (ex-info "boom" {})))]
|
||||
(do
|
||||
(let [thread (consumer/start-loop {} consumer.nexus/consumer)]
|
||||
(try
|
||||
(loop [i 0]
|
||||
(when (and (< @passes 3) (< i 300))
|
||||
@ -921,21 +1107,22 @@
|
||||
|
||||
(t/deftest consumer-component-starts-a-loop-that-halt-stops
|
||||
;; The component is the only door to the loop: it starts a pass, keeps
|
||||
;; running it and `halt-component` stops the thread. The pass is a fake
|
||||
;; running it and `halt` stops the thread. The pass is a fake
|
||||
;; on purpose: a real drain here would keep the shared database busy in
|
||||
;; the background while the rest of the namespace runs, which is what
|
||||
;; made this suite alternate. The drain itself is covered by the
|
||||
;; `run-pass` tests.
|
||||
(let [passes (atom 0)
|
||||
cfg {::db/pool (:app.db/pool th/*system*)}]
|
||||
(with-redefs [cf/flags #{:nexus}
|
||||
consumer/sleep-ms 10]
|
||||
(let [thread (consumer/init-component cfg consumer/nexus identity
|
||||
(fn [_] (swap! passes inc)))]
|
||||
(with-redefs [cf/flags #{:nexus}
|
||||
consumer/sleep-ms 10
|
||||
consumer/run-pass (fn [_cfg _consumer] (swap! passes inc))]
|
||||
(let [thread (consumer/init cfg
|
||||
(assoc consumer.nexus/consumer :prepare identity))]
|
||||
(try
|
||||
(t/is (th/wait-for #(<= 3 @passes)))
|
||||
(finally
|
||||
(consumer/halt-component thread)))
|
||||
(consumer/halt thread)))
|
||||
(t/is (false? (.isAlive ^Thread thread)))))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
@ -959,14 +1146,16 @@
|
||||
"Run one nexus pass with an isolated metrics registry, so assertions
|
||||
read exact values instead of the shared test-system registry."
|
||||
[metrics params]
|
||||
(consumer.nexus/run-pass
|
||||
(nexus-cfg (:uri params) metrics)))
|
||||
(consumer/run-pass
|
||||
(nexus-cfg (:uri params) metrics)
|
||||
consumer.nexus/consumer))
|
||||
|
||||
(defn- invoke-admin-console
|
||||
"Run one admin-console pass with an isolated metrics registry."
|
||||
[metrics]
|
||||
(consumer.admin-console/run-pass
|
||||
(admin-console-cfg metrics)))
|
||||
(consumer/run-pass
|
||||
(admin-console-cfg metrics)
|
||||
consumer.admin-console/consumer))
|
||||
|
||||
(defn- invoke-both
|
||||
"Invoke both consumer handlers sharing one isolated metrics
|
||||
@ -982,6 +1171,33 @@
|
||||
child (.labels ^Counter instance (into-array String labels))]
|
||||
(.get ^Counter$Child child)))
|
||||
|
||||
(t/deftest telemetry-sends-old-shadows-still-in-the-buffer
|
||||
(with-redefs [cf/flags #{:telemetry}]
|
||||
(with-mocks [http-mock {:target 'app.http.client/req :return {:status 200}}]
|
||||
(insert-audit-row uuid/zero "full")
|
||||
(insert-shadow-row uuid/zero "old"
|
||||
(ct/minus (ct/now) (ct/duration {:days 8})))
|
||||
(insert-shadow-row uuid/zero "recent")
|
||||
(insert-audit-row uuid/zero "last-full")
|
||||
(let [metrics (make-consumer-metrics)
|
||||
process (:process consumer.telemetry/consumer)
|
||||
chunks (atom [])]
|
||||
(consumer/run-pass
|
||||
(telemetry-cfg telemetry-uri metrics)
|
||||
(assoc consumer.telemetry/consumer
|
||||
:process (fn [cfg rows]
|
||||
(swap! chunks into rows)
|
||||
(process cfg rows))))
|
||||
(let [body (json/decode (-> @http-mock :call-args second :body)
|
||||
:key-fn json/read-kebab-key)
|
||||
full (remove #(= "telemetry:backend" (:source %)) @chunks)]
|
||||
(t/is (= ["old" "recent"] (mapv :name (:events body))))
|
||||
(t/is (= ["full" "last-full"] (mapv :name full)))
|
||||
(t/is (every? #(and (nil? (:props %)) (nil? (:context %))) full))
|
||||
(t/is (= 0 (unread-count consumer/telemetry)))
|
||||
(t/is (= 2.0 (counter-value metrics :audit-consumer-events ["telemetry" "processed"])))
|
||||
(t/is (= 2.0 (counter-value metrics :audit-consumer-events ["telemetry" "skipped"]))))))))
|
||||
|
||||
(defn- gauge-value
|
||||
[metrics id labels]
|
||||
(let [collector (mtx/get-collector metrics id)
|
||||
@ -1014,7 +1230,7 @@
|
||||
:body ""}}]
|
||||
(insert-audit-row (:id prof) "create-project")
|
||||
(let [err (try
|
||||
(consumer.admin-console/run-pass consumer-cfg)
|
||||
(consumer/run-pass consumer-cfg consumer.admin-console/consumer)
|
||||
nil
|
||||
(catch clojure.lang.ExceptionInfo e e))]
|
||||
(t/is (= :admin-console-send-failed (:code (ex-data err))))
|
||||
@ -1054,7 +1270,7 @@
|
||||
(insert-audit-row (:id prof) "create-file")
|
||||
;; The cursor is still at zero: two confirmed events behind.
|
||||
(with-open [conn (db/open th/*system*)]
|
||||
(#'consumer/record-lag conn metrics {:id consumer/nexus}))
|
||||
(#'consumer/record-lag conn metrics consumer.nexus/consumer))
|
||||
(t/is (= 2.0 (gauge-value metrics :audit-consumer-lag ["nexus"])))))
|
||||
|
||||
(t/deftest consumer-pass-counter-records-ok-and-busy
|
||||
@ -1210,7 +1426,7 @@
|
||||
(insert-rows-in-one-transaction conn "lote-" 10001)
|
||||
(db/exec! conn ["COMMIT"]))
|
||||
(with-open [conn (db/open th/*system*)]
|
||||
(#'consumer/record-lag conn metrics {:id consumer/nexus}))
|
||||
(#'consumer/record-lag conn metrics consumer.nexus/consumer))
|
||||
(t/is (= 10000.0 (gauge-value metrics :audit-consumer-lag ["nexus"])))))
|
||||
|
||||
(t/deftest consumer-nexus-transport-failure-records-failed
|
||||
@ -1312,35 +1528,93 @@
|
||||
(t/is (= 0 (run-gc)))
|
||||
(t/is (= 1 (count (th/db-exec! ["SELECT * FROM audit_log"]))))))))
|
||||
|
||||
(t/deftest gc-keeps-the-telemetry-shadows
|
||||
(t/deftest gc-deletes-the-shadow-what-the-telemetry-consumer-passed
|
||||
(let [prof (th/create-profile* 62 {:is-active true})]
|
||||
(with-redefs [cf/flags #{:nexus :admin-console}]
|
||||
(with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}
|
||||
admin-mock {:target 'app.nitrate/call :return nil}]
|
||||
(with-redefs [cf/flags #{:telemetry}]
|
||||
(with-mocks [http-mock {:target 'app.http.client/req :return {:status 200}}]
|
||||
(insert-shadow-row (:id prof) "create-project")
|
||||
(run-telemetry {:uri telemetry-uri})
|
||||
;; The telemetry path deletes the copy: the count is the rows
|
||||
;; removed by cursor, both paths together.
|
||||
(t/is (= 1 (run-gc)))
|
||||
(t/is (= 0 (count (th/db-exec! ["SELECT * FROM audit_log"]))))))))
|
||||
|
||||
(t/deftest gc-keeps-the-shadow-past-the-telemetry-cursor
|
||||
(let [prof (th/create-profile* 62 {:is-active true})]
|
||||
;; The telemetry cursor stays at (0, 0): the copy is after it.
|
||||
(with-redefs [cf/flags #{:telemetry}]
|
||||
(insert-shadow-row (:id prof) "create-project")
|
||||
(t/is (= 0 (run-gc)))
|
||||
(t/is (= 1 (count (th/db-exec! ["SELECT * FROM audit_log"])))))))
|
||||
|
||||
(t/deftest gc-telemetry-lag-does-not-hold-the-full-rows
|
||||
;; A telemetry pass every three hours must not retain the events Nexus and
|
||||
;; the admin console already archived: each path has its own position.
|
||||
(let [prof (th/create-profile* 62 {:is-active true})]
|
||||
(with-redefs [cf/flags #{:nexus}]
|
||||
(with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}]
|
||||
(th/db-exec! ["DELETE FROM audit_log_consumer_cursor
|
||||
WHERE consumer = ?::smallint" consumer/admin-console])
|
||||
(insert-audit-row (:id prof) "create-project")
|
||||
(insert-audit-row (:id prof) "create-project" {} "telemetry:backend")
|
||||
(run-nexus {:uri archive-uri})
|
||||
(run-admin-console)
|
||||
;; The telemetry task owns the shadows: the GC leaves them.
|
||||
(insert-shadow-row (:id prof) "navigate")
|
||||
(t/is (= 1 (run-gc)))
|
||||
(let [rows (th/db-exec! ["SELECT source FROM audit_log"])]
|
||||
(t/is (= ["telemetry:backend"] (mapv :source rows))))))))
|
||||
|
||||
(t/deftest gc-retention-deletes-the-expired-shadow-without-a-cursor
|
||||
;; The retention window is the guarantee, not the cursors: with no cursor
|
||||
;; at all (nothing has ever run) an eight-day-old copy is still deleted.
|
||||
;; The run counts it with the rest of the rows it removed.
|
||||
(th/db-exec! ["DELETE FROM audit_log_consumer_cursor"])
|
||||
(insert-shadow-row uuid/zero "create-project"
|
||||
(ct/minus (ct/now) (ct/duration {:days 8})))
|
||||
(t/is (= 1 (run-gc)))
|
||||
(t/is (= 0 (count (th/db-exec! ["SELECT * FROM audit_log"])))))
|
||||
|
||||
(t/deftest gc-retention-keeps-the-young-shadow
|
||||
(th/db-exec! ["DELETE FROM audit_log_consumer_cursor"])
|
||||
(insert-shadow-row uuid/zero "create-project")
|
||||
(t/is (= 0 (run-gc)))
|
||||
(t/is (= 1 (count (th/db-exec! ["SELECT * FROM audit_log"])))))
|
||||
|
||||
(t/deftest gc-retention-runs-with-telemetry-off
|
||||
;; The flag only decides who ships; retention is about age, so a copy left
|
||||
;; behind by an earlier deployment is deleted with telemetry off.
|
||||
(with-redefs [cf/flags #{:nexus}]
|
||||
(insert-shadow-row uuid/zero "create-project"
|
||||
(ct/minus (ct/now) (ct/duration {:days 8})))
|
||||
(run-gc)
|
||||
(t/is (= 0 (count (th/db-exec! ["SELECT * FROM audit_log"]))))))
|
||||
|
||||
(t/deftest gc-retention-does-not-delete-the-full-rows
|
||||
;; The window applies to the anonymized copies only: an old full row waits
|
||||
;; for its consumers, however old it is.
|
||||
(th/db-exec! ["DELETE FROM audit_log_consumer_cursor"])
|
||||
(insert-audit-row uuid/zero "create-project")
|
||||
(th/db-exec! ["UPDATE audit_log SET created_at = now() - interval '30 days'"])
|
||||
(t/is (= 0 (run-gc)))
|
||||
(t/is (= 1 (count (th/db-exec! ["SELECT * FROM audit_log"])))))
|
||||
|
||||
(t/deftest gc-records-the-discarded-metric
|
||||
(let [prof (th/create-profile* 63 {:is-active true})
|
||||
metrics (make-consumer-metrics)
|
||||
job-def (ig/init-key :app.loggers.audit.gc/job-def
|
||||
{::db/pool (:app.db/pool th/*system*)
|
||||
::mtx/metrics metrics})]
|
||||
{::db/pool (:app.db/pool th/*system*)
|
||||
::mtx/metrics metrics})]
|
||||
(with-redefs [cf/flags #{:nexus :admin-console}]
|
||||
(with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}
|
||||
admin-mock {:target 'app.nitrate/call :return nil}]
|
||||
(insert-audit-row (:id prof) "create-project")
|
||||
(insert-audit-row (:id prof) "create-file")
|
||||
;; An expired copy: the retention window removes it and the run
|
||||
;; counts it too.
|
||||
(insert-shadow-row (:id prof) "navigate"
|
||||
(ct/minus (ct/now) (ct/duration {:days 8})))
|
||||
(run-nexus {:uri archive-uri})
|
||||
(run-admin-console)
|
||||
(t/is (= 2 ((::jobs/handler job-def) nil {})))
|
||||
(t/is (= 2.0 (counter-value metrics :audit-consumer-discarded ["gc"])))))))
|
||||
(t/is (= 3 ((::jobs/handler job-def) nil {})))
|
||||
(t/is (= 3.0 (counter-value metrics :audit-consumer-discarded ["gc"])))))))
|
||||
|
||||
(t/deftest gc-deletes-in-batches
|
||||
(let [prof (th/create-profile* 64 {:is-active true})]
|
||||
@ -1351,28 +1625,154 @@
|
||||
(insert-audit-row (:id prof) (str "event-" i)))
|
||||
(run-nexus {:uri archive-uri})
|
||||
(run-admin-console)
|
||||
(with-redefs-fn {#'gc/batch-size 2}
|
||||
(fn []
|
||||
(t/is (= 5 (run-gc)))))
|
||||
(with-bindings {#'gc/batch-size 2}
|
||||
(t/is (= 5 (run-gc))))
|
||||
(t/is (= 0 (count (th/db-exec! ["SELECT * FROM audit_log"]))))))))
|
||||
|
||||
(t/deftest init-component-registers-the-cursor-when-enabled
|
||||
(t/deftest init-registers-the-cursor-when-enabled
|
||||
;; The GC waits for the cursor table, so an enabled consumer must have
|
||||
;; its (0, 0) row before its first pass. A disabled consumer leaves no
|
||||
;; row behind. `start-loop` is replaced so the test leaves no live
|
||||
;; thread behind; the loop itself has its own test.
|
||||
(th/db-exec! ["DELETE FROM audit_log_consumer_cursor"])
|
||||
(with-redefs [cf/flags #{:nexus}
|
||||
consumer/start-loop (fn [_cfg _id _run-pass] nil)]
|
||||
(consumer/init-component
|
||||
consumer/start-loop (fn [_cfg _consumer] nil)]
|
||||
(consumer/init
|
||||
{::db/pool (:app.db/pool th/*system*)
|
||||
::mtx/metrics (:app.metrics/metrics th/*system*)}
|
||||
consumer/nexus
|
||||
identity
|
||||
(fn [_cfg] nil))
|
||||
(assoc consumer.nexus/consumer :prepare identity))
|
||||
(t/is (= 0 (:seq (cursor-of consumer/nexus))))
|
||||
(t/is (nil? (cursor-of consumer/admin-console)))))
|
||||
|
||||
(t/deftest init-prepares-the-transport-uri
|
||||
;; `prepare` is the single resolution point. The loop and every batch use
|
||||
;; the normalized URI stored in their cfg.
|
||||
(doseq [[definition cfg config-key base-uri expected-uri uri-key]
|
||||
[[consumer.nexus/consumer
|
||||
(nexus-cfg nil)
|
||||
:nexus-uri
|
||||
"http://nexus.example"
|
||||
"http://nexus.example/api/audit"
|
||||
::consumer.nexus/uri]
|
||||
[consumer.telemetry/consumer
|
||||
(telemetry-cfg nil)
|
||||
:telemetry-uri
|
||||
telemetry-uri
|
||||
telemetry-uri
|
||||
::consumer.telemetry/uri]]]
|
||||
(let [started (atom nil)]
|
||||
(with-redefs [cf/flags #{(:flag definition)}
|
||||
cf/config (assoc cf/config config-key (u/uri base-uri))
|
||||
consumer/start-loop (fn [cfg _consumer]
|
||||
(reset! started cfg))]
|
||||
(consumer/init (assoc cfg uri-key (u/uri "http://stale.example")) definition)
|
||||
(t/is (= (u/uri expected-uri) (get @started uri-key)))))))
|
||||
|
||||
(t/deftest gc-stops-at-its-time-budget
|
||||
;; A pass has a wall-clock budget: a stream that keeps feeding rows cannot
|
||||
;; hold the job alive forever, and whatever is left waits for the next run.
|
||||
(let [prof (th/create-profile* 68 {:is-active true})]
|
||||
(with-redefs [cf/flags #{:nexus}]
|
||||
(with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}]
|
||||
(th/db-exec! ["DELETE FROM audit_log_consumer_cursor
|
||||
WHERE consumer = ?::smallint" consumer/admin-console])
|
||||
(dotimes [i 3]
|
||||
(insert-audit-row (:id prof) (str "event-" i)))
|
||||
(run-nexus {:uri archive-uri})
|
||||
(with-bindings {#'gc/run-budget (ct/duration {:seconds 0})}
|
||||
(t/is (= 0 (run-gc))))
|
||||
(t/is (= 3 (count (th/db-exec! ["SELECT * FROM audit_log"]))))
|
||||
;; The next run has the budget again and drains what was left.
|
||||
(t/is (= 3 (run-gc)))
|
||||
(t/is (= 0 (count (th/db-exec! ["SELECT * FROM audit_log"]))))))))
|
||||
|
||||
(t/deftest gc-a-failing-path-does-not-stop-the-others
|
||||
;; Each path is self-sufficient: a failure logs its path and answers 0, and
|
||||
;; the others still delete and are counted.
|
||||
(let [prof (th/create-profile* 69 {:is-active true})]
|
||||
;; One expired copy: the retention path removes it. One fresh copy: its
|
||||
;; cursor has not passed it, so it stays.
|
||||
(insert-shadow-row (:id prof) "navigate"
|
||||
(ct/minus (ct/now) (ct/duration {:days 8})))
|
||||
(insert-shadow-row (:id prof) "create-project")
|
||||
(t/is (= 1 (with-redefs-fn {#'gc/delete-batch
|
||||
(fn [& _] (throw (ex-info "boom" {})))}
|
||||
#(run-gc))))
|
||||
(let [rows (th/db-exec! ["SELECT source FROM audit_log"])]
|
||||
(t/is (= ["telemetry:backend"] (mapv :source rows))))))
|
||||
|
||||
(t/deftest gc-counts-confirmed-batches-before-a-path-fails
|
||||
(let [metrics (make-consumer-metrics)
|
||||
cfg {::db/pool (:app.db/pool th/*system*) ::mtx/metrics metrics}
|
||||
delete @#'gc/delete-batch
|
||||
calls (atom 0)]
|
||||
(th/db-exec! ["DELETE FROM audit_log_consumer_cursor"])
|
||||
(with-redefs [cf/flags #{:nexus}]
|
||||
(with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}]
|
||||
(insert-audit-row uuid/zero "first")
|
||||
(insert-audit-row uuid/zero "second")
|
||||
(insert-shadow-row uuid/zero "expired"
|
||||
(ct/minus (ct/now) (ct/duration {:days 8})))
|
||||
(run-nexus {:uri archive-uri})
|
||||
(with-bindings {#'gc/batch-size 1}
|
||||
(t/is (= 2 (with-redefs-fn
|
||||
{#'gc/delete-batch
|
||||
(fn [cfg cursor sql]
|
||||
(if (and (= sql @#'gc/sql:delete-full-batch)
|
||||
(= 2 (swap! calls inc)))
|
||||
(throw (ex-info "Second batch failed" {}))
|
||||
(delete cfg cursor sql)))}
|
||||
#(gc/delete-consumed cfg)))))
|
||||
(t/is (= ["second"] (mapv :name (th/db-exec! ["SELECT name FROM audit_log"]))))
|
||||
(t/is (= 2.0 (counter-value metrics :audit-consumer-discarded ["gc"])))))))
|
||||
|
||||
(t/deftest gc-does-not-start-another-batch-after-budget-expires
|
||||
(let [metrics (make-consumer-metrics)
|
||||
cfg {::mtx/metrics metrics}
|
||||
expired (atom false)]
|
||||
(with-redefs-fn
|
||||
{#'gc/budget-spent? (fn [_] @expired)}
|
||||
#(t/is (= 1 (@#'gc/run-path cfg "full"
|
||||
(fn []
|
||||
(if (compare-and-set! expired false true)
|
||||
1
|
||||
(throw (AssertionError. "Started a late batch"))))))))
|
||||
(t/is (= 1.0 (counter-value metrics :audit-consumer-discarded ["gc"])))))
|
||||
|
||||
(t/deftest gc-interruption-counts-the-completed-batch-without-starting-another
|
||||
(let [metrics (make-consumer-metrics)
|
||||
cfg {::db/pool (:app.db/pool th/*system*) ::mtx/metrics metrics}
|
||||
entered (promise)
|
||||
release (promise)
|
||||
result (promise)
|
||||
batches (atom [])
|
||||
thread (Thread.
|
||||
(fn []
|
||||
(try
|
||||
(deliver result (gc/delete-consumed cfg))
|
||||
(catch Throwable cause
|
||||
(deliver result cause)))))]
|
||||
(with-redefs-fn
|
||||
{#'gc/delete-batch
|
||||
(fn [_ _ sql]
|
||||
(swap! batches conj sql)
|
||||
(deliver entered true)
|
||||
(while (not (realized? release))
|
||||
(Thread/onSpinWait))
|
||||
1)}
|
||||
(fn []
|
||||
(.start thread)
|
||||
(try
|
||||
(t/is (= true (deref entered 5000 ::timeout)))
|
||||
(.interrupt thread)
|
||||
(deliver release true)
|
||||
(t/is (instance? InterruptedException (deref result 5000 ::timeout)))
|
||||
(t/is (= [@#'gc/sql:delete-full-batch] @batches))
|
||||
(t/is (= 1.0 (counter-value metrics :audit-consumer-discarded ["gc"])))
|
||||
(finally
|
||||
(deliver release true)
|
||||
(.join thread 5000)))))))
|
||||
|
||||
(t/deftest gc-waits-for-a-consumer-that-has-not-drained-yet
|
||||
;; A consumer that has just started registers its cursor at (0, 0)
|
||||
;; before its first pass. The GC treats that position as the oldest and
|
||||
@ -1405,15 +1805,14 @@
|
||||
(th/db-exec! ["UPDATE audit_log_consumer_cursor
|
||||
SET xid = ?::xid8, seq = 3 WHERE consumer = ?::smallint"
|
||||
(str high) consumer/admin-console])
|
||||
(let [cursor (@#'gc/get-oldest-cursor {::db/pool th/*pool*})]
|
||||
(let [cursor (@#'gc/get-oldest-full-cursor {::db/pool th/*pool*})]
|
||||
(t/is (= (str low) (:xid-text cursor)))
|
||||
(t/is (= 7 (:seq cursor))))))
|
||||
|
||||
(t/deftest gc-re-reads-the-cursor-between-batches
|
||||
;; A consumer registered between two batches must be seen by the next
|
||||
;; batch: `delete-one-batch` re-reads the oldest cursor under the
|
||||
;; registry lock, so it does not delete rows the new consumer has not
|
||||
;; read.
|
||||
;; batch: `delete-batch` re-reads the position on every batch, so it does
|
||||
;; not delete rows the new consumer has not read.
|
||||
(let [prof (th/create-profile* 66 {:is-active true})
|
||||
cfg {::db/pool (:app.db/pool th/*system*)
|
||||
::mtx/metrics (:app.metrics/metrics th/*system*)}]
|
||||
@ -1426,15 +1825,33 @@
|
||||
(th/db-exec! ["DELETE FROM audit_log_consumer_cursor
|
||||
WHERE consumer = ?::smallint"
|
||||
consumer/admin-console])
|
||||
(with-redefs-fn {#'gc/batch-size 1}
|
||||
(fn []
|
||||
(t/is (= {:deleted 1 :found? true}
|
||||
(@#'gc/delete-one-batch cfg)))
|
||||
;; A new consumer registers at (0, 0) between the batches.
|
||||
(consumer/ensure-cursor cfg consumer/admin-console)
|
||||
(t/is (= {:deleted 0 :found? true}
|
||||
(@#'gc/delete-one-batch cfg)))
|
||||
(t/is (= 3 (count (th/db-exec! ["SELECT * FROM audit_log"]))))))))))
|
||||
(with-bindings {#'gc/batch-size 1}
|
||||
(t/is (= 1 (@#'gc/delete-batch cfg
|
||||
@#'gc/get-oldest-full-cursor
|
||||
@#'gc/sql:delete-full-batch)))
|
||||
;; A new consumer registers at (0, 0) between the batches.
|
||||
(consumer/ensure-cursor cfg consumer/admin-console)
|
||||
(t/is (= 0 (@#'gc/delete-batch cfg
|
||||
@#'gc/get-oldest-full-cursor
|
||||
@#'gc/sql:delete-full-batch)))
|
||||
(t/is (= 3 (count (th/db-exec! ["SELECT * FROM audit_log"])))))))))
|
||||
|
||||
(t/deftest gc-bounds-the-telemetry-shadow-batch
|
||||
;; A delete must not expand past its limit while joining the selected ids
|
||||
;; back to the table.
|
||||
(let [prof (th/create-profile* 67 {:is-active true})
|
||||
cfg {::db/pool (:app.db/pool th/*system*)
|
||||
::mtx/metrics (:app.metrics/metrics th/*system*)}]
|
||||
(with-redefs [cf/flags #{:telemetry}]
|
||||
(with-mocks [http-mock {:target 'app.http.client/req :return {:status 200}}]
|
||||
(dotimes [i 4]
|
||||
(insert-shadow-row (:id prof) (str "event-" i)))
|
||||
(run-telemetry {:uri telemetry-uri})
|
||||
(with-bindings {#'gc/batch-size 1}
|
||||
(t/is (= 1 (@#'gc/delete-batch cfg
|
||||
@#'gc/get-telemetry-cursor
|
||||
@#'gc/sql:delete-shadow-batch)))
|
||||
(t/is (= 3 (count (th/db-exec! ["SELECT * FROM audit_log"])))))))))
|
||||
|
||||
(t/deftest gc-does-not-delete-past-an-unread-cursor
|
||||
;; Two enabled consumers across a decimal boundary: the GC deletes up to
|
||||
|
||||
@ -10,9 +10,6 @@
|
||||
[app.common.uuid :as uuid]
|
||||
[app.config :as cf]
|
||||
[app.db :as db]
|
||||
[app.loggers.audit :as audit]
|
||||
[app.tasks.telemetry :as telemetry]
|
||||
[app.util.json :as json]
|
||||
[backend-tests.helpers :as th]
|
||||
[clojure.test :as t]
|
||||
[mockery.core :refer [with-mocks]]
|
||||
@ -30,34 +27,6 @@
|
||||
|
||||
(t/use-fixtures :each test-fixture)
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; HELPERS
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(defn- insert-telemetry-row!
|
||||
"Insert a single anonymised audit_log row as the telemetry mode does."
|
||||
([name] (insert-telemetry-row! name {}))
|
||||
([name {:keys [tracked-at created-at source]
|
||||
:or {tracked-at (ct/now)
|
||||
created-at (ct/now)
|
||||
source "telemetry:backend"}}]
|
||||
(th/db-insert! :audit-log
|
||||
{:id (uuid/next)
|
||||
:name name
|
||||
:type "action"
|
||||
:source source
|
||||
:profile-id uuid/zero
|
||||
:ip-addr (db/inet "0.0.0.0")
|
||||
:props (db/tjson {})
|
||||
:context (db/tjson {})
|
||||
:tracked-at tracked-at
|
||||
:created-at created-at})))
|
||||
|
||||
(defn- count-telemetry-rows []
|
||||
(-> (th/db-exec-one! ["SELECT count(*) AS cnt FROM audit_log WHERE source IN ('telemetry:backend', 'telemetry:frontend')"])
|
||||
:cnt
|
||||
long))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; STATS / REPORT STRUCTURE TESTS (existing behaviour, extended)
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
@ -184,242 +153,6 @@
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
(t/is (:called? @mock)))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; AUDIT-EVENT BATCH COLLECTION TESTS
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(t/deftest test-no-audit-events-no-batch-call
|
||||
;; When telemetry is enabled but there are no audit_log rows with
|
||||
;; source='telemetry', the batch send path must not be invoked.
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}
|
||||
batch-mock {:target 'app.tasks.telemetry/send-event-batch
|
||||
:return true}]
|
||||
(th/with-audit-config #{:telemetry}
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
(t/is (:called? @legacy-mock))
|
||||
(t/is (not (:called? @batch-mock))))))
|
||||
|
||||
(t/deftest test-audit-events-sent-and-deleted-on-success
|
||||
;; Happy path: telemetry rows are collected, shipped as a batch and
|
||||
;; deleted from the table when the endpoint returns success.
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}
|
||||
batch-mock {:target 'app.tasks.telemetry/send-event-batch
|
||||
:return true}]
|
||||
(th/with-audit-config #{:telemetry}
|
||||
(insert-telemetry-row! "navigate")
|
||||
(insert-telemetry-row! "create-file")
|
||||
(insert-telemetry-row! "update-file")
|
||||
|
||||
(t/is (= 3 (count-telemetry-rows)))
|
||||
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
|
||||
;; batch send was called at least once
|
||||
(t/is (:called? @batch-mock))
|
||||
|
||||
;; all rows deleted after successful send
|
||||
(t/is (= 0 (count-telemetry-rows))))))
|
||||
|
||||
(t/deftest test-audit-events-kept-on-batch-failure
|
||||
;; When the batch endpoint returns failure the rows must be retained
|
||||
;; so the next scheduled run can retry.
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}
|
||||
batch-mock {:target 'app.tasks.telemetry/send-event-batch
|
||||
:return false}]
|
||||
(th/with-audit-config #{:telemetry}
|
||||
(insert-telemetry-row! "navigate")
|
||||
(insert-telemetry-row! "create-file")
|
||||
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
|
||||
(t/is (:called? @batch-mock))
|
||||
;; rows still present — not deleted on failure
|
||||
(t/is (= 2 (count-telemetry-rows))))))
|
||||
|
||||
(t/deftest test-audit-events-not-collected-when-consumer-persists
|
||||
;; When a consumer persists rows, mode C is disabled and the
|
||||
;; batch path must never run (the buffer owns those rows instead).
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}
|
||||
batch-mock {:target 'app.tasks.telemetry/send-event-batch
|
||||
:return true}]
|
||||
(th/with-audit-config #{:telemetry :nexus}
|
||||
(insert-telemetry-row! "navigate")
|
||||
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
|
||||
(t/is (not (:called? @batch-mock)))
|
||||
;; row untouched
|
||||
(t/is (= 1 (count-telemetry-rows))))))
|
||||
|
||||
(t/deftest test-batch-payload-contains-required-fields
|
||||
;; Inspect the actual arguments forwarded to send-event-batch to
|
||||
;; verify the payload carries instance-id, version and events.
|
||||
(let [captured (atom nil)]
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}]
|
||||
(th/with-audit-config #{:telemetry} [telemetry/send-event-batch
|
||||
(fn [_cfg batch]
|
||||
(reset! captured batch)
|
||||
true)]
|
||||
(insert-telemetry-row! "navigate")
|
||||
(insert-telemetry-row! "create-file")
|
||||
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
|
||||
(t/is (some? @captured))
|
||||
(let [batch @captured]
|
||||
;; batch is a seq of event maps
|
||||
(t/is (seq batch))
|
||||
(t/is (= 2 (count batch)))
|
||||
;; each event has name, type, source — profile-id is preserved,
|
||||
;; props and ip-addr are stripped
|
||||
(let [ev (first batch)]
|
||||
(t/is (contains? ev :name))
|
||||
(t/is (contains? ev :type))
|
||||
(t/is (contains? ev :source))
|
||||
(t/is (contains? ev :profile-id))
|
||||
;; props are present but empty (stripped at ingest time)
|
||||
(t/is (= {} (:props ev)))
|
||||
(t/is (not (contains? ev :ip-addr)))))))))
|
||||
|
||||
(t/deftest test-batch-encoding-is-decodable
|
||||
;; Events are sent as a plain vector of raw event maps (no blob
|
||||
;; encoding): every batch must JSON round-trip unchanged, because
|
||||
;; the receiver coerces types from the plain JSON representation.
|
||||
(let [events [{:name "navigate" :type "action" :source "telemetry"
|
||||
:tracked-at (ct/now)}
|
||||
{:name "create-file" :type "action" :source "telemetry"
|
||||
:tracked-at (ct/now)}]
|
||||
encoded (json/encode-str {:events (vec events)})
|
||||
decoded (json/decode encoded)]
|
||||
(t/is (vector? (:events decoded)))
|
||||
(t/is (= (count events) (count (:events decoded))))
|
||||
(t/is (= "navigate" (:name (first (:events decoded)))))
|
||||
(t/is (= "create-file" (:name (second (:events decoded)))))))
|
||||
|
||||
(t/deftest test-multiple-batches-when-many-events
|
||||
;; Lower batch-size to 1 so that 3 events produce 3 separate
|
||||
;; HTTP requests and verify all are sent and all rows deleted.
|
||||
(let [call-count (atom 0)]
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}]
|
||||
(th/with-audit-config #{:telemetry} [telemetry/batch-size 1
|
||||
telemetry/send-event-batch
|
||||
(fn [_cfg _batch]
|
||||
(swap! call-count inc)
|
||||
true)]
|
||||
(insert-telemetry-row! "navigate")
|
||||
(insert-telemetry-row! "create-file")
|
||||
(insert-telemetry-row! "update-file")
|
||||
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
|
||||
;; Each event is fetched and sent in its own loop iteration
|
||||
(t/is (= 3 @call-count))
|
||||
;; All rows deleted after all iterations succeed
|
||||
(t/is (= 0 (count-telemetry-rows)))))))
|
||||
|
||||
(t/deftest test-partial-failure-stops-remaining-batches
|
||||
;; With batch-size 1, when the second send fails the loop stops.
|
||||
;; The first batch was already deleted; the two remaining rows
|
||||
;; are retained for the next run.
|
||||
(let [call-count (atom 0)]
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}]
|
||||
(th/with-audit-config #{:telemetry} [telemetry/batch-size 1
|
||||
telemetry/send-event-batch
|
||||
(fn [_cfg _batch]
|
||||
(swap! call-count inc)
|
||||
;; fail on the second call
|
||||
(not= 2 @call-count))]
|
||||
(insert-telemetry-row! "navigate")
|
||||
(insert-telemetry-row! "create-file")
|
||||
(insert-telemetry-row! "update-file")
|
||||
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
|
||||
;; Stopped at iteration 2 — third event never attempted
|
||||
(t/is (= 2 @call-count))
|
||||
;; First batch was deleted on success; 2 rows remain for retry
|
||||
(t/is (= 2 (count-telemetry-rows)))))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; GC / RETENTION-WINDOW TESTS
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(t/deftest test-gc-purges-events-older-than-7-days
|
||||
;; Insert events from 8 days ago (stale) and from today (fresh).
|
||||
;; After the task runs, stale events must be purged by GC and fresh
|
||||
;; ones shipped by the batch sender.
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}
|
||||
batch-mock {:target 'app.tasks.telemetry/send-event-batch
|
||||
:return true}]
|
||||
(th/with-audit-config #{:telemetry}
|
||||
(let [now (ct/now)
|
||||
eight-days (ct/minus now (ct/duration {:days 8}))]
|
||||
;; Stale events (older than 7 days)
|
||||
(insert-telemetry-row! "stale-1" {:created-at eight-days :tracked-at eight-days})
|
||||
(insert-telemetry-row! "stale-2" {:created-at eight-days :tracked-at eight-days})
|
||||
;; Fresh events (today)
|
||||
(insert-telemetry-row! "fresh-1" {:created-at now :tracked-at now})
|
||||
(insert-telemetry-row! "fresh-2" {:created-at now :tracked-at now})
|
||||
|
||||
(t/is (= 4 (count-telemetry-rows)))
|
||||
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
|
||||
;; GC purged the 2 stale rows, batch sender shipped the 2 fresh ones
|
||||
(t/is (= 0 (count-telemetry-rows)))))))
|
||||
|
||||
(t/deftest test-gc-keeps-events-within-7-day-window
|
||||
;; When all events are within the 7-day window, GC must not delete
|
||||
;; anything and all rows are forwarded to the batch sender.
|
||||
(let [batch-events (atom nil)]
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}]
|
||||
(th/with-audit-config #{:telemetry} [telemetry/send-event-batch
|
||||
(fn [_cfg batch]
|
||||
(reset! batch-events batch)
|
||||
true)]
|
||||
(let [six-days-ago (ct/minus (ct/now) (ct/duration {:days 6}))]
|
||||
(insert-telemetry-row! "recent-1" {:created-at six-days-ago :tracked-at six-days-ago})
|
||||
(insert-telemetry-row! "recent-2" {:created-at six-days-ago :tracked-at six-days-ago}))
|
||||
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
|
||||
;; Both events forwarded — GC left them alone
|
||||
(t/is (= 2 (count @batch-events)))
|
||||
(t/is (= 0 (count-telemetry-rows)))))))
|
||||
|
||||
(t/deftest test-gc-deletes-only-stale-events
|
||||
;; Insert a mix of stale (8 days old) and fresh (1 day old) events.
|
||||
;; After GC, only fresh events should remain for the batch sender.
|
||||
(let [batch-events (atom nil)]
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}]
|
||||
(th/with-audit-config #{:telemetry} [telemetry/send-event-batch
|
||||
(fn [_cfg batch]
|
||||
(reset! batch-events batch)
|
||||
true)]
|
||||
(let [eight-days (ct/minus (ct/now) (ct/duration {:days 8}))
|
||||
one-day (ct/minus (ct/now) (ct/duration {:days 1}))]
|
||||
(insert-telemetry-row! "stale" {:created-at eight-days :tracked-at eight-days})
|
||||
(insert-telemetry-row! "fresh" {:created-at one-day :tracked-at one-day}))
|
||||
|
||||
(t/is (= 2 (count-telemetry-rows)))
|
||||
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
|
||||
;; GC purged stale, batch shipped fresh
|
||||
(t/is (= 1 (count @batch-events)))
|
||||
(t/is (= "fresh" (:name (first @batch-events))))
|
||||
(t/is (= 0 (count-telemetry-rows)))))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; ANONYMITY TESTS
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
@ -457,55 +190,6 @@
|
||||
;; profile-id is uuid/zero — not a real user id
|
||||
(t/is (= uuid/zero (:profile-id row)))))))
|
||||
|
||||
(t/deftest test-batch-events-contain-no-pii-fields
|
||||
;; The event maps forwarded to send-event-batch must not carry props,
|
||||
;; ip-addr or profile-id. Safe context fields (browser, os, etc.) may
|
||||
;; be present but session-linking keys must be absent.
|
||||
(let [captured-batch (atom nil)
|
||||
;; Insert a row that carries safe context (as the real path does)
|
||||
safe-ctx {:browser "Firefox" :browser-version "121.0"
|
||||
:os "macOS" :session "should-be-stripped"
|
||||
:external-session-id "also-stripped"}]
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}]
|
||||
(th/with-audit-config #{:telemetry} [telemetry/send-event-batch
|
||||
(fn [_cfg batch]
|
||||
(reset! captured-batch batch)
|
||||
true)]
|
||||
;; Insert with safe context already pre-filtered (as the ingest path does)
|
||||
(th/db-insert! :audit-log
|
||||
{:id (uuid/next)
|
||||
:name "navigate"
|
||||
:type "action"
|
||||
:source "telemetry:frontend"
|
||||
:profile-id uuid/zero
|
||||
:ip-addr (db/inet "0.0.0.0")
|
||||
:props (db/tjson {})
|
||||
:context (db/tjson (dissoc safe-ctx :session :external-session-id))
|
||||
:tracked-at (ct/now)
|
||||
:created-at (ct/now)})
|
||||
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
|
||||
(t/is (= 1 (count @captured-batch)))
|
||||
(let [ev (first @captured-batch)]
|
||||
;; must have the core identity fields including profile-id
|
||||
(t/is (contains? ev :name))
|
||||
(t/is (contains? ev :type))
|
||||
(t/is (contains? ev :source))
|
||||
(t/is (contains? ev :tracked-at))
|
||||
(t/is (contains? ev :profile-id))
|
||||
;; props are present but empty (stripped at ingest time)
|
||||
(t/is (= {} (:props ev)))
|
||||
;; ip-addr is stripped
|
||||
(t/is (not (contains? ev :ip-addr)))
|
||||
;; context may be present and must not contain session-linking keys
|
||||
(when-let [ctx (:context ev)]
|
||||
(t/is (not (contains? ctx :session)))
|
||||
(t/is (not (contains? ctx :external-session-id)))
|
||||
;; safe keys should be present
|
||||
(t/is (contains? ctx :browser))))))))
|
||||
|
||||
(t/deftest test-telemetry-rows-have-day-precision-timestamps
|
||||
;; Telemetry events must be stored with timestamps truncated to day
|
||||
;; precision so that exact event timing cannot be inferred.
|
||||
@ -891,61 +575,17 @@
|
||||
(t/is (= (dissoc props :email :unsafe-label)
|
||||
(:props result)))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; SEND-EVENT-BATCH PAYLOAD STRUCTURE
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(t/deftest test-send-event-batch-payload-structure
|
||||
;; Verify the HTTP request sent by send-event-batch carries the
|
||||
;; correct outer wrapper: :type, :version, :instance-id, :events.
|
||||
(let [captured-request (atom nil)]
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}
|
||||
http-mock {:target 'app.http.client/req
|
||||
:return {:status 200}}]
|
||||
(th/with-audit-config #{:telemetry}
|
||||
(insert-telemetry-row! "navigate")
|
||||
(insert-telemetry-row! "create-file")
|
||||
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
|
||||
;; http/req was called (by both send-legacy-data and send-event-batch)
|
||||
(t/is (:called? @http-mock))
|
||||
;; Find the call whose body contains :telemetry-events
|
||||
(let [calls (filter (fn [args]
|
||||
(let [[_ request] args
|
||||
body (:body request)]
|
||||
(and (string? body)
|
||||
(re-find #"telemetry-events" body))))
|
||||
(:call-args-list @http-mock))]
|
||||
(t/is (= 1 (count calls)))
|
||||
(let [[_ request] (first calls)
|
||||
body (json/decode (:body request))]
|
||||
;; Outer payload fields
|
||||
(t/is (= "telemetry-events" (name (:type body))))
|
||||
(t/is (string? (:version body)))
|
||||
(t/is (some? (:instance-id body)))
|
||||
;; :events is a plain vector of raw event maps
|
||||
(t/is (vector? (:events body)))
|
||||
(t/is (pos? (count (:events body))))
|
||||
(doseq [ev (:events body)]
|
||||
(t/is (string? (:name ev)))
|
||||
(t/is (string? (:source ev)))
|
||||
(t/is (string? (:tracked-at ev))))))))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; TASK BRANCH COVERAGE
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(t/deftest test-enabled-no-subs-no-events-legacy-still-sends
|
||||
;; When telemetry is enabled, there are no newsletter subscriptions
|
||||
;; and no audit_log rows, the legacy report must still be sent.
|
||||
;; and no audit_log rows, the legacy report must still be sent. The
|
||||
;; shadows no longer travel here: the telemetry consumer sends them.
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}
|
||||
batch-mock {:target 'app.tasks.telemetry/send-event-batch
|
||||
:return true}]
|
||||
:return nil}]
|
||||
(th/with-audit-config #{:telemetry}
|
||||
;; No profiles with newsletter-updates, no telemetry rows
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
|
||||
;; Legacy report was sent
|
||||
@ -954,111 +594,4 @@
|
||||
(t/is (= :telemetry-legacy-report (:type data)))
|
||||
(t/is (contains? data :stats))
|
||||
;; No subscriptions in the payload
|
||||
(t/is (not (contains? data :subscriptions))))
|
||||
|
||||
;; No events to batch-send
|
||||
(t/is (not (:called? @batch-mock))))))
|
||||
|
||||
(t/deftest test-legacy-succeeds-batch-fails
|
||||
;; The legacy report and event batch are independent paths.
|
||||
;; When the batch endpoint fails, the legacy report must still
|
||||
;; have been sent successfully.
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}
|
||||
batch-mock {:target 'app.tasks.telemetry/send-event-batch
|
||||
:return false}]
|
||||
(th/with-audit-config #{:telemetry}
|
||||
(insert-telemetry-row! "navigate")
|
||||
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
|
||||
;; Legacy report was sent
|
||||
(t/is (:called? @legacy-mock))
|
||||
(let [[_ data] (:call-args @legacy-mock)]
|
||||
(t/is (= :telemetry-legacy-report (:type data))))
|
||||
|
||||
;; Batch send was attempted but failed
|
||||
(t/is (:called? @batch-mock))
|
||||
;; Row still present (not deleted on failure)
|
||||
(t/is (= 1 (count-telemetry-rows))))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; GC + BATCH FAILURE INTERACTION
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(t/deftest test-gc-runs-even-when-batch-fails
|
||||
;; GC must purge stale events regardless of whether the subsequent
|
||||
;; batch send succeeds or fails.
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}
|
||||
batch-mock {:target 'app.tasks.telemetry/send-event-batch
|
||||
:return false}]
|
||||
(th/with-audit-config #{:telemetry}
|
||||
(let [eight-days (ct/minus (ct/now) (ct/duration {:days 8}))
|
||||
one-day (ct/minus (ct/now) (ct/duration {:days 1}))]
|
||||
;; Stale events (should be GC'd)
|
||||
(insert-telemetry-row! "stale-1" {:created-at eight-days :tracked-at eight-days})
|
||||
(insert-telemetry-row! "stale-2" {:created-at eight-days :tracked-at eight-days})
|
||||
;; Fresh event (should survive GC but fail to send)
|
||||
(insert-telemetry-row! "fresh" {:created-at one-day :tracked-at one-day})
|
||||
|
||||
(t/is (= 3 (count-telemetry-rows)))
|
||||
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
|
||||
;; Batch send was attempted (and failed)
|
||||
(t/is (:called? @batch-mock))
|
||||
;; Stale rows were purged by GC, fresh row remains
|
||||
(t/is (= 1 (count-telemetry-rows)))
|
||||
(t/is (= "fresh" (:name (first (th/db-exec! ["SELECT name FROM audit_log WHERE source LIKE 'telemetry:%'"])))))))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; ROW->EVENT CONTEXT GUARANTEE
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(t/deftest test-row->event-always-includes-context
|
||||
;; row->event must always include :context as a map, even when the
|
||||
;; DB column contains an empty transit object.
|
||||
(let [row->event (ns-resolve 'app.tasks.telemetry 'row->event)]
|
||||
;; With non-empty context
|
||||
(let [ev (row->event {:name "test" :type "action" :source "telemetry:backend"
|
||||
:tracked-at (ct/now) :profile-id uuid/zero
|
||||
:context (db/tjson {:browser "Chrome"})})]
|
||||
(t/is (contains? ev :context))
|
||||
(t/is (= {:browser "Chrome"} (:context ev))))
|
||||
|
||||
;; With empty context ({} in transit)
|
||||
(let [ev (row->event {:name "test" :type "action" :source "telemetry:backend"
|
||||
:tracked-at (ct/now) :profile-id uuid/zero
|
||||
:context (db/tjson {})})]
|
||||
(t/is (contains? ev :context))
|
||||
(t/is (= {} (:context ev))))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; NO DUPLICATE EVENTS ON SUCCESS
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(t/deftest test-no-duplicate-events-after-successful-send
|
||||
;; After a successful batch send, the sent rows must be deleted.
|
||||
;; Running the task again must NOT re-send the same events.
|
||||
(let [send-count (atom 0)]
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}]
|
||||
(th/with-audit-config #{:telemetry} [telemetry/send-event-batch
|
||||
(fn [_cfg _batch]
|
||||
(swap! send-count inc)
|
||||
true)]
|
||||
(insert-telemetry-row! "navigate")
|
||||
(insert-telemetry-row! "create-file")
|
||||
|
||||
(t/is (= 2 (count-telemetry-rows)))
|
||||
|
||||
;; First run: sends and deletes
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
(t/is (= 1 @send-count))
|
||||
(t/is (= 0 (count-telemetry-rows)))
|
||||
|
||||
;; Second run: no events to send
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
(t/is (= 1 @send-count)) ;; still 1, not 2
|
||||
(t/is (= 0 (count-telemetry-rows)))))))
|
||||
(t/is (not (contains? data :subscriptions)))))))
|
||||
|
||||
@ -17,7 +17,6 @@
|
||||
[app.worker.cron :as wcron]
|
||||
[app.worker.dispatcher :as wdisp]
|
||||
[app.worker.runner :as wrkr]
|
||||
[backend-tests.helpers :as th]
|
||||
[clojure.test :as t]
|
||||
[integrant.core :as ig]))
|
||||
|
||||
@ -55,24 +54,23 @@
|
||||
(t/deftest every-worker-consumer-component-is-known-to-the-test-system
|
||||
;; The test system drops the consumer loops by key, so a new one must be
|
||||
;; added there too or it drains audit_log in the background. Deriving the
|
||||
;; list from the config (every worker key named `/consumer`) makes a
|
||||
;; third consumer fail here, not as a flaky test elsewhere.
|
||||
;; list from the config (every worker key named `/consumer`) makes a new
|
||||
;; consumer fail here, not as a flaky test elsewhere.
|
||||
(let [keys (->> (keys main/worker-config)
|
||||
(filter #(and (keyword? %) (= "consumer" (name %))))
|
||||
(into #{}))]
|
||||
(t/is (= #{:app.loggers.audit.consumers.nexus/consumer
|
||||
:app.loggers.audit.consumers.admin-console/consumer}
|
||||
:app.loggers.audit.consumers.admin-console/consumer
|
||||
:app.loggers.audit.consumers.telemetry/consumer}
|
||||
keys)
|
||||
(str "update this set and helpers/init-system's dissoc: " keys))))
|
||||
|
||||
(t/deftest audit-log-gc-cron-is-gated-by-its-own-flag
|
||||
(t/testing "the GC entry is present only with :audit-log-gc"
|
||||
(with-redefs [cf/flags #{:audit-log-gc}]
|
||||
(t/is (some #(= :audit-log-gc (:job %)) (main/worker-cron-entries))))
|
||||
(th/with-audit-config #{}
|
||||
(t/is (not (some #(= :audit-log-gc (:job %))
|
||||
(main/worker-cron-entries)))))
|
||||
(with-redefs [cf/flags #{}]
|
||||
(t/testing "default configuration schedules the GC"
|
||||
(binding [cf/flags (#'cf/parse-flags {})]
|
||||
(t/is (some #(= :audit-log-gc (:job %)) (main/worker-cron-entries)))))
|
||||
(t/testing "an explicit disable suppresses the GC even with telemetry enabled"
|
||||
(binding [cf/flags (#'cf/parse-flags {:flags [:enable-telemetry :disable-audit-log-gc]})]
|
||||
(t/is (not (some #(= :audit-log-gc (:job %))
|
||||
(main/worker-cron-entries)))))))
|
||||
|
||||
@ -94,7 +92,7 @@
|
||||
(t/is (contains? cfg ::db/pool)))))
|
||||
|
||||
(t/deftest the-audit-consumers-are-wired
|
||||
;; Each consumer is a loop in a thread: the test system drops the two
|
||||
;; Each consumer is a loop in a thread: the test system drops the three
|
||||
;; components by name, so a rename that leaves `helpers.clj` behind would
|
||||
;; start real loops in the background. The components live in the worker
|
||||
;; config only (a web instance does not drain the audit log) and their
|
||||
@ -103,7 +101,10 @@
|
||||
[::db/pool :app.http.client/client
|
||||
:app.setup/shared-keys ::mtx/metrics]]
|
||||
[:app.loggers.audit.consumers.admin-console/consumer
|
||||
[::db/pool :app.nitrate/client ::mtx/metrics]]]]
|
||||
[::db/pool :app.nitrate/client ::mtx/metrics]]
|
||||
[:app.loggers.audit.consumers.telemetry/consumer
|
||||
[::db/pool :app.http.client/client
|
||||
:app.setup/props ::mtx/metrics]]]]
|
||||
(t/testing (str k)
|
||||
(let [cfg (get main/worker-config k)]
|
||||
(t/is (some? cfg))
|
||||
|
||||
@ -59,8 +59,8 @@
|
||||
|
||||
(def varia
|
||||
"Rest of the flags"
|
||||
#{;; Runs the audit-log buffer GC. Keep it off while the old and the new
|
||||
;; version coexist; the runbook turns it on after the rollout.
|
||||
#{;; Runs the audit-log buffer GC by default. Disable it explicitly while
|
||||
;; the old and new versions coexist.
|
||||
:audit-log-gc
|
||||
;; Enables shipping audit_log chunks to Nexus.
|
||||
:nexus
|
||||
@ -224,6 +224,7 @@
|
||||
:enable-backend-api-doc
|
||||
:enable-backend-openapi-doc
|
||||
:enable-backend-worker
|
||||
:enable-audit-log-gc
|
||||
:enable-secure-session-cookies
|
||||
:enable-email-verification
|
||||
:enable-onboarding
|
||||
|
||||
15
common/test/common_tests/flags_test.cljc
Normal file
15
common/test/common_tests/flags_test.cljc
Normal file
@ -0,0 +1,15 @@
|
||||
;; This Source Code Form is subject to the terms of the Mozilla Public
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns common-tests.flags-test
|
||||
(:require
|
||||
[app.common.flags :as flags]
|
||||
[clojure.test :as t]))
|
||||
|
||||
(t/deftest audit-log-gc-default-can-be-disabled
|
||||
(t/is (contains? (flags/parse flags/default) :audit-log-gc))
|
||||
(t/is (not (contains? (flags/parse flags/default [:disable-audit-log-gc])
|
||||
:audit-log-gc))))
|
||||
@ -38,6 +38,7 @@
|
||||
[common-tests.files.tokens-test]
|
||||
[common-tests.files.validate-test]
|
||||
[common-tests.files.variant-test]
|
||||
[common-tests.flags-test]
|
||||
[common-tests.geom-align-test]
|
||||
[common-tests.geom-bounds-layout-nil-test]
|
||||
[common-tests.geom-bounds-map-test]
|
||||
@ -160,6 +161,7 @@
|
||||
'common-tests.files.tokens-test
|
||||
'common-tests.files.validate-test
|
||||
'common-tests.files.variant-test
|
||||
'common-tests.flags-test
|
||||
'common-tests.geom-align-test
|
||||
'common-tests.geom-bounds-layout-nil-test
|
||||
'common-tests.geom-bounds-map-test
|
||||
|
||||
@ -92,10 +92,13 @@ its row here.
|
||||
|
||||
## 1. Stop the legacy archiver and its GC
|
||||
|
||||
Turn off the old archiver and its garbage collection with their flags:
|
||||
`:audit-log-archive` off and `:audit-log-gc` off. The new version uses
|
||||
`:audit-log-gc` for its own GC, so the same flag keeps both off during the
|
||||
rollout. The new version has no `:audit-log` flag: writing full rows follows
|
||||
Before any 2.20.0 worker starts, set `disable-audit-log-archive` and
|
||||
`disable-audit-log-gc` in `PENPOT_FLAGS` on all workers. Remove any existing
|
||||
`enable-audit-log-gc`; do not leave both tokens. After restarting, check
|
||||
each worker's startup log and confirm `audit-log-gc` is not active. Keep
|
||||
`disable-audit-log-gc` throughout the mixed-version rollout: the new GC is
|
||||
on by default, so removing `enable-audit-log-gc` alone does not stop it.
|
||||
The new version has no `:audit-log` flag: writing full rows follows
|
||||
the consumer flags. Keep on the destinations this instance already uses:
|
||||
`:nexus` only if it has a Nexus URI after the conversion below (it is an
|
||||
internal destination), and `:admin-console` only if the instance uses it. At
|
||||
@ -137,8 +140,8 @@ Do **not** drop `archived_at` or the indexes that served it
|
||||
(`audit_log__created_at__idx`, `audit_log__archived_at__idx`,
|
||||
`audit_log_id_archived_at_idx`). The old archiver may still be running on an
|
||||
instance that has not been replaced, and a later release removes them. The
|
||||
consumers order by `(xid, seq)` (step 7); telemetry keeps
|
||||
`audit_log__source__created_at__idx` plus a sort.
|
||||
consumers order by `(xid, seq)` (step 7), so `audit_log__source__created_at__idx`
|
||||
is not used and waits for the same cleanup.
|
||||
|
||||
## 3. Pre-apply the jobs step
|
||||
|
||||
@ -465,7 +468,59 @@ no NULL, so a `count(*)` would only scan the table again.
|
||||
|
||||
## 10. After the rollout
|
||||
|
||||
Once every instance runs this version, the new consumers have replayed the
|
||||
unarchived buffer. Turn on `:audit-log-gc` (rolling restart of the new
|
||||
workers) so the new GC deletes the rows the consumers have passed. Remove
|
||||
`:audit-log-archive`; the new version does not use it.
|
||||
Once every instance runs this version and the new consumers have replayed
|
||||
the unarchived buffer, remove `disable-audit-log-gc` and roll the workers
|
||||
to restore the default GC schedule. The GC deletes rows the consumers have
|
||||
passed and applies the 7-day retention of telemetry shadows. Remove the
|
||||
old archiver flag setting; the new version does not use it.
|
||||
|
||||
### Registering a consumer after the rollout
|
||||
|
||||
The GC reads the cursors again for each batch, but it does not lock out
|
||||
registration. If a consumer registers while a cursor-based delete batch is
|
||||
in progress, it can miss up to one batch per concurrent GC run that already
|
||||
read the previous cursor limit. The 7-day shadow retention deletes
|
||||
independently of cursors.
|
||||
|
||||
For a complete replay of the buffer, keep only `disable-audit-log-gc` on all
|
||||
workers sharing the database. Restart them, verify the effective flag in
|
||||
each startup log, and wait for all queued or running GC jobs to finish
|
||||
before enabling the new consumer. Do not invoke the GC manually during the
|
||||
pause. Confirm the consumer has registered and replayed the buffer before
|
||||
removing `disable-audit-log-gc` and rolling the workers again. This cannot
|
||||
recover rows the GC already deleted.
|
||||
|
||||
### The telemetry shadows
|
||||
|
||||
The anonymized copies (`source LIKE 'telemetry:%'`) have their own consumer
|
||||
since this version: id `0`, flag `:telemetry`. It ships them with the pace of
|
||||
the old cron (10 000 rows per request, a pass every 3 hours) and the GC
|
||||
deletes them once its cursor has passed them; no task deletes them any more.
|
||||
|
||||
- On the first boot of a worker with `:telemetry` on, the consumer registers
|
||||
its cursor at `(0, 0)` and its first pass ships every shadow the buffer
|
||||
holds. While the old version still runs its own cron, the same copy can
|
||||
reach the endpoint twice during the overlap window; telemetry is
|
||||
at-least-once and nothing deduplicates it.
|
||||
- The GC has two cursor paths: the full rows up to the oldest cursor of the
|
||||
consumers that read them, and the shadows up to the cursor `0`. A slow
|
||||
telemetry pass (three hours) does not retain the events Nexus already
|
||||
archived, and turning telemetry off does not stop the deletion of the full
|
||||
rows.
|
||||
- Shadows older than 7 days are deleted by the GC whatever the cursor says,
|
||||
with the endpoint down included. That window only runs with
|
||||
`:audit-log-gc` on, and it can drop copies that were never sent.
|
||||
It limits local retention, not sending: the consumer can send an older
|
||||
shadow if it reads it before the GC deletes it.
|
||||
- To retire the telemetry consumer, turn `:telemetry` off on every instance
|
||||
first, let the GC run once with the cursor frozen (it deletes the copies
|
||||
the consumer had already passed), and then delete the cursor:
|
||||
|
||||
```sql
|
||||
DELETE FROM audit_log_consumer_cursor WHERE consumer = 0;
|
||||
```
|
||||
|
||||
The statement deletes no shadow by itself: the copies the consumer had not
|
||||
read wait for the retention window (or a manual delete). Turning
|
||||
`:telemetry` back on registers the cursor at `(0, 0)` again, so the first
|
||||
pass replays whatever the buffer still holds and skips the full rows.
|
||||
|
||||
@ -60,8 +60,8 @@ The backfill runs on every instance, even when it updates no rows.
|
||||
column. A rolling deploy in between would mix both.
|
||||
- The `0158` cutover (phase 4) takes minutes in a low-traffic window. Saves
|
||||
that fall inside fail and the user retries.
|
||||
- After **every** instance runs 2.20.0, turn on `:audit-log-gc` (phase 5.6).
|
||||
Until then the old and new versions coexist and the new GC must stay off.
|
||||
- The new GC runs by default. Keep `disable-audit-log-gc` in `PENPOT_FLAGS`
|
||||
throughout the mixed-version rollout. Restore the default in phase 5.6.
|
||||
|
||||
|
||||
## Phase 1 — Before the upgrade (2.19.0 serving; online)
|
||||
@ -170,18 +170,24 @@ Re-issue them in each SQL session.
|
||||
|
||||
**Step 1 — Stop the legacy archiver and its GC**
|
||||
|
||||
1. Turn off `:audit-log-archive` and `:audit-log-gc`. There is no `:audit-log`
|
||||
flag in the new version: writing full rows follows the consumer flags.
|
||||
2. Enable `:nexus` only if `:nexus-uri` is configured. Keep `:admin-console`
|
||||
on if used. At least one of them must be on **on every instance**, web and
|
||||
worker: each process computes its own persistence value at boot, so a web
|
||||
instance without a consumer flag writes no rows even if a worker drains
|
||||
them. With all consumers off there is no backlog to replay later.
|
||||
3. Set `:nexus-uri` to the base URL, for example
|
||||
`https://nexus.example/proxy`. The new consumer appends `/api/audit`; do not
|
||||
copy the old full `:audit-log-archive-uri` into `:nexus-uri`.
|
||||
4. Roll the workers with these flags and wait for active archive and GC tasks
|
||||
to finish.
|
||||
1. Before any 2.20.0 worker starts, set `disable-audit-log-archive` and
|
||||
`disable-audit-log-gc` in `PENPOT_FLAGS` on all workers. Remove any existing
|
||||
`enable-audit-log-gc`; do not leave both tokens. Removing `enable` alone
|
||||
does not stop the new GC. There is no
|
||||
`:audit-log` flag in the new version: writing full rows follows the
|
||||
consumer flags.
|
||||
2. If the deployment uses the admin console, keep `:admin-console` on
|
||||
**on every instance**, web and worker. Each process decides at startup
|
||||
whether to write full rows; enabling the flag only on workers does not
|
||||
make web instances write them.
|
||||
The `:telemetry` consumer (id `0`) ships the anonymized copies
|
||||
with the pace of the old cron (10 000 rows, a pass every 3 hours) and its
|
||||
first pass replays the shadows the buffer holds. While the old version
|
||||
still runs its cron, a copy can reach the endpoint twice; telemetry is
|
||||
at-least-once.
|
||||
3. Roll the workers with these flags. Check each worker's startup log to
|
||||
confirm `audit-log-gc` is not active, and wait for active archive and GC
|
||||
tasks to finish. Keep `disable-audit-log-gc` in place until phase 5.6.
|
||||
|
||||
**Step 2 — Delete what the archiver already archived**
|
||||
|
||||
@ -1558,10 +1564,37 @@ ROLLBACK;
|
||||
|
||||
### 5.6 After every instance runs 2.20.0: turn on the new GC
|
||||
|
||||
Once every instance runs this version, the new consumers have replayed
|
||||
the unarchived buffer. Turn on `:audit-log-gc` (rolling restart of the
|
||||
new workers) so the new GC deletes the rows the consumers have passed.
|
||||
Remove `:audit-log-archive`; the new version does not use it.
|
||||
Once every instance runs this version and the new consumers have replayed
|
||||
the unarchived buffer, remove `disable-audit-log-gc` and roll the workers to
|
||||
restore the default GC schedule. The GC deletes the rows the consumers have
|
||||
passed and applies the 7-day retention of the telemetry shadows. The GC has
|
||||
two cursor paths: the full rows up to the oldest cursor of the consumers
|
||||
that read them, and the shadows up to the cursor `0`; a slow telemetry pass
|
||||
does not retain the full rows. Remove the old archiver flag setting; the new
|
||||
version does not use it.
|
||||
|
||||
The seven-day window limits local retention, not sending. The telemetry
|
||||
consumer can send older shadows if it reads them before the GC deletes them.
|
||||
|
||||
If a consumer registers while a cursor-based GC batch is in progress, it
|
||||
can miss up to one batch per concurrent GC run that already read the previous
|
||||
cursor limit. The GC reads the cursors again for each batch but does not lock
|
||||
out registration. The 7-day shadow retention deletes
|
||||
independently of cursors. For a complete replay of the buffer, set
|
||||
`disable-audit-log-gc` on all workers sharing the database and remove any
|
||||
`enable-audit-log-gc` token. Restart the workers, verify the effective flag
|
||||
in each startup log, and wait for all queued or running GC jobs to finish
|
||||
before enabling the new consumer. Do not invoke the
|
||||
GC manually during the pause. Confirm the consumer has registered and
|
||||
replayed the buffer before removing `disable-audit-log-gc` and rolling the
|
||||
workers again. This cannot recover rows the GC already deleted.
|
||||
|
||||
To retire the telemetry consumer, turn `:telemetry` off on every instance,
|
||||
let the GC run once with its cursor frozen and then delete the cursor
|
||||
(`DELETE FROM audit_log_consumer_cursor WHERE consumer = 0;`). The statement
|
||||
deletes no shadow by itself: the copies the consumer had not read wait for the
|
||||
retention window. Turning the flag back on registers the cursor at `(0, 0)`
|
||||
again and replays whatever the buffer still holds.
|
||||
|
||||
---
|
||||
|
||||
@ -1579,8 +1612,8 @@ Remove `:audit-log-archive`; the new version does not use it.
|
||||
- **Consumer generalization**: the migration keeps the legacy
|
||||
`archived_at` mark and its indexes for one release; the two versions can
|
||||
coexist during the rollout. Rolling the binaries back is possible while
|
||||
`:audit-log-gc` is off and the consumers are idle; keep the flag off
|
||||
until you are done, and call it a rollback only within the rollout.
|
||||
`disable-audit-log-gc` is set and the consumers are idle. Keep that
|
||||
setting until you are done, and call it a rollback only within the rollout.
|
||||
|
||||
## Risks
|
||||
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user