📚 Add backend audit-log memory documentation

Add a new memory file documenting the backend audit log system:
purpose, storage schema, RPC producers, frontend ingestion,
webhooks/error-reporter/telemetry consumers, and Nexus archival.

Also wire a reference to it from the backend core memory so it
is discoverable through the memory graph.

AI-assisted-by: longcat-2.0
This commit is contained in:
Andrey Antukh 2026-09-22 15:49:46 +00:00
parent 5e99e89e5b
commit cc0941343d
2 changed files with 52 additions and 0 deletions

View File

@ -0,0 +1,51 @@
# Backend Audit Log
Penpot records what users do as events in the Postgres `audit_log` table. There are two producers (the backend RPC layer and the frontend app) and four consumers (webhooks, error reporters, telemetry shipping, and the Nexus archive). Everything below follows that flow: purpose, storage, producers, consumers, archival.
## Purpose
- The audit log answers "who did what, when, from where": every RPC mutation and selected frontend actions become a row with `name`, `type`, `profile-id`, `ip-addr`, `props` and `context`. Product analytics, abuse investigation and compliance exports all read from here, so keep events truthful and never put secrets in `props`.
- It is also the trigger bus for side effects: the same event object fans out to webhooks, error reporting and telemetry without the RPC handler knowing. New features should reuse this bus instead of building parallel notification paths.
## Storage
- Live `audit_log` columns: `id` uuid PK default `gen_random_uuid()`; `name`/`type` text NOT NULL; `created_at` timestamptz NOT NULL default `now()` (server time, the source of truth); `tracked_at` timestamptz default `now()` (client-claimed time, corrected on ingest); `profile_id` uuid NOT NULL; `source` text telling full rows (`backend`/`frontend`) apart from anonymized copies (`telemetry:backend`/`telemetry:frontend`); `ip_addr` inet; `props`/`context` jsonb holding transit-encoded maps; `archived_at` timestamptz set once Nexus acknowledges the row.
- Indexes: PK on `(id)`; partial `created_at WHERE archived_at IS NULL` serving the archive scan; partial `archived_at WHERE archived_at IS NOT NULL` serving the GC; `(source, created_at)` serving the telemetry scan. Each consumer has its own index, so a slow consumer never blocks the others.
## Backend producers (`app.loggers.audit`)
- Most backend events need no manual code: `wrap-audit` in `app.rpc` runs after every RPC handler when `:webhooks`, `:audit-log` or `:telemetry` is on (unless the command sets `::audit/skip`) and builds the event via `prepare-rpc-event`. The event name defaults to the command name (prefixed with `<module>-` outside `main`), props default to the request params, and timestamps come from the server request time.
- Commands customize through result metadata (`rph/with-meta`): `::audit/replace-props` swaps the props wholesale (auth commands use `profile->props` so a register event carries the profile, not the password), `::audit/props` merges extras, `::audit/context`/`profile-id`/`name`/`type` override the defaults. `clean-props` always strips nils, qualified keys and `:session-id/:password/:old-password/:token/:client-secret` as a last line of defense.
- `submit` is the normal entry point (fills defaults, validates `schema:event`, runs inside `tx-run!`, logs failures without failing the RPC). `insert` is the low-level one for CLI/helpers and the webhook subsystem: direct write, no webhook/telemetry fan-out, silent unless `:audit-log` is on. Boot emits `trigger/instance-start` from `setup/props` so every restart is visible in the log.
## Consumers I: webhooks (`app.loggers.webhooks`)
- Webhooks are the first dependent: when an event carries `::webhooks/event?`, `process-event` (worker task `:process-webhook-event`) finds the team's active webhooks from the event props (`team-id`, else `project-id`, else `file-id`), records a `trigger webhook` row, and enqueues one `:run-webhook` delivery per match. Batching and dedupe come from the audit event itself (`batch-key` + `batch-timeout`), not from webhook config.
- `:run-webhook` POSTs the event in the webhook's `mtype` (JSON camelCase, transit, or form-encoded), logs each attempt in `webhook-delivery`, and disables the webhook after 3 consecutive errors. Delivery problems never touch the audit row itself.
## Consumers II: error reporters (`app.loggers.database`, `app.loggers.mattermost`)
- Both reporters listen for backend `:error` log records and for frontend crash events (recognized by the `::audit/event` marker), through a sliding-buffer channel so a flood of errors cannot stall the app. The database reporter persists them into `server-error-report` (source 4 = audit-log origin); the Mattermost reporter forwards a short notification to `:error-report-webhook` when configured.
- Consequence for producers: crash reports only exist if the frontend collector is running and `push-audit-events` accepts `unhandled-exception`/`exception-page` events. Disabling the whole pipeline also blinds error reporting from the frontend.
## Consumers III: telemetry (`app.loggers.audit` + `app.tasks.telemetry`)
- Telemetry reuses the same table with anonymized shadow rows (`source LIKE 'telemetry:%'`): day-truncated timestamps, `0.0.0.0` IPs, props reduced to uuid/boolean/number values plus a few allowlisted fields (`lang`, `auth-backend`, derived `email-domain`, never raw emails), and a minimal context allowlist. Both full and shadow rows can coexist per event; that duplication is intentional.
- The telemetry cron ships shadow rows to `:telemetry-uri` as JSON in 10k batches, deletes them on success, and purges leftovers older than 7d. Nothing is collected or sent on official hosts (`telemetry-excluded?` covers `penpot.app`/`penpot.dev`).
## Frontend ingestion (`app.rpc.commands.audit`, `app.main.data.event`)
- The browser cannot write to the table directly; it POSTs transit batches to `push-audit-events`, which stamps server `id`, session `profile-id`, request ip and server `created-at`, and distrusts the client clock (future or >1h-lagging `tracked-at` is reset, original preserved in context). The endpoint is a no-op without `:audit-log`/`:telemetry` or on a read-only pool.
- The in-browser collector (`app.main.data.event`) only starts after `get-enabled-flags` confirms the backend wants events. It turns Potok events and explicit `ev/event` calls (nitrate membership changes, workspace file stats, crash reports) into a capped buffer (1024, chunks of 100, 2s debounce, current profile only) and sends fire-and-forget. `skip-audit?` exists for resumed dashboard actions so one user gesture is not counted twice.
- Because collection is best-effort and includes `PerformanceObserver` noise (`performance-*` triggers), backend tests must never assert exact frontend event counts.
## Archival to Nexus and retention
- Long-term storage lives outside Penpot in Nexus. Every 5m the `:audit-log-archive` cron takes chunks of 128 unarchived rows (`FOR UPDATE SKIP LOCKED`), POSTs them as transit `{:events [...]}` authenticated with `x-shared-key: "nexus <key>"` (`:nexus-shared-key`, else derived from the instance secret), and marks `archived_at=now()` only on HTTP 204, in the same transaction. Anything else is retried on the next run; a missing URI with the flag on raises `:task-not-configured`.
- Every 5m the `:audit-log-gc` cron deletes all archived rows (no age filter), so archive must run before GC or data ships never. Cron dedup is best-effort (`mem:prod-infra/core`): two backends can fire the archiver twice, which is why the Nexus endpoint must be idempotent and the DB only marks acknowledged rows.
- Flags live in `common/flags.cljc` varia and are enabled as `PENPOT_FLAGS=enable-<name>`: `:audit-log`, `:audit-log-archive`, `:audit-log-gc`, `:audit-log-logger` (structured `app.audit` log). `:telemetry-enabled` config auto-adds `:enable-telemetry`.
## Tests
- `backend_tests/rpc-audit-test.clj` exercises the whole backend path (full-row insert, telemetry-only and dual-row modes, `submit*`, no-op without flags, `insert` gating, `prepare-rpc-event` resolution) with `with-redefs [cf/flags #{...}]` against real `audit_log` rows.
- Other RPC suites mock `app.loggers.audit/submit` (nil return; `helpers.clj` stubs it globally) and assert on `:call-args-list`; any new command that must (or must not) emit an event needs the same treatment.

View File

@ -8,6 +8,7 @@ Backend: JVM Clojure; Integrant; PostgreSQL; Redis/Valkey; RPC; HTTP; storage; m
- Storage abstraction, logical buckets, object lifecycle, deduplication, access, and garbage collection: `mem:backend/storage`.
- HTTP sessions, config, media processing, and file data persistence: `mem:backend/http-storage-filedata-subtleties`.
- Auth flows, permission model, teams, projects, invitations, comments, webhooks, audit: `mem:backend/auth-permissions-product-domains`
- Audit-log event collection (RPC wrapper, frontend ingestion), telemetry duality, webhook fan-out, error reporters, Nexus archival and retention: `mem:backend/audit-log`
- Services, task-queue/Pub-Sub topology constraints -> `mem:prod-infra/core`.
## Stable namespace map