diff --git a/.serena/memories/backend/audit-log.md b/.serena/memories/backend/audit-log.md new file mode 100644 index 0000000000..fe401cef0c --- /dev/null +++ b/.serena/memories/backend/audit-log.md @@ -0,0 +1,51 @@ +# Backend Audit Log + +Penpot records what users do as events in the Postgres `audit_log` table. There are two producers (the backend RPC layer and the frontend app) and four consumers (webhooks, error reporters, telemetry shipping, and the Nexus archive). Everything below follows that flow: purpose, storage, producers, consumers, archival. + +## Purpose + +- The audit log answers "who did what, when, from where": every RPC mutation and selected frontend actions become a row with `name`, `type`, `profile-id`, `ip-addr`, `props` and `context`. Product analytics, abuse investigation and compliance exports all read from here, so keep events truthful and never put secrets in `props`. +- It is also the trigger bus for side effects: the same event object fans out to webhooks, error reporting and telemetry without the RPC handler knowing. New features should reuse this bus instead of building parallel notification paths. + +## Storage + +- Live `audit_log` columns: `id` uuid PK default `gen_random_uuid()`; `name`/`type` text NOT NULL; `created_at` timestamptz NOT NULL default `now()` (server time, the source of truth); `tracked_at` timestamptz default `now()` (client-claimed time, corrected on ingest); `profile_id` uuid NOT NULL; `source` text telling full rows (`backend`/`frontend`) apart from anonymized copies (`telemetry:backend`/`telemetry:frontend`); `ip_addr` inet; `props`/`context` jsonb holding transit-encoded maps; `archived_at` timestamptz set once Nexus acknowledges the row. +- Indexes: PK on `(id)`; partial `created_at WHERE archived_at IS NULL` serving the archive scan; partial `archived_at WHERE archived_at IS NOT NULL` serving the GC; `(source, created_at)` serving the telemetry scan. Each consumer has its own index, so a slow consumer never blocks the others. + +## Backend producers (`app.loggers.audit`) + +- Most backend events need no manual code: `wrap-audit` in `app.rpc` runs after every RPC handler when `:webhooks`, `:audit-log` or `:telemetry` is on (unless the command sets `::audit/skip`) and builds the event via `prepare-rpc-event`. The event name defaults to the command name (prefixed with `-` outside `main`), props default to the request params, and timestamps come from the server request time. +- Commands customize through result metadata (`rph/with-meta`): `::audit/replace-props` swaps the props wholesale (auth commands use `profile->props` so a register event carries the profile, not the password), `::audit/props` merges extras, `::audit/context`/`profile-id`/`name`/`type` override the defaults. `clean-props` always strips nils, qualified keys and `:session-id/:password/:old-password/:token/:client-secret` as a last line of defense. +- `submit` is the normal entry point (fills defaults, validates `schema:event`, runs inside `tx-run!`, logs failures without failing the RPC). `insert` is the low-level one for CLI/helpers and the webhook subsystem: direct write, no webhook/telemetry fan-out, silent unless `:audit-log` is on. Boot emits `trigger/instance-start` from `setup/props` so every restart is visible in the log. + +## Consumers I: webhooks (`app.loggers.webhooks`) + +- Webhooks are the first dependent: when an event carries `::webhooks/event?`, `process-event` (worker task `:process-webhook-event`) finds the team's active webhooks from the event props (`team-id`, else `project-id`, else `file-id`), records a `trigger webhook` row, and enqueues one `:run-webhook` delivery per match. Batching and dedupe come from the audit event itself (`batch-key` + `batch-timeout`), not from webhook config. +- `:run-webhook` POSTs the event in the webhook's `mtype` (JSON camelCase, transit, or form-encoded), logs each attempt in `webhook-delivery`, and disables the webhook after 3 consecutive errors. Delivery problems never touch the audit row itself. + +## Consumers II: error reporters (`app.loggers.database`, `app.loggers.mattermost`) + +- Both reporters listen for backend `:error` log records and for frontend crash events (recognized by the `::audit/event` marker), through a sliding-buffer channel so a flood of errors cannot stall the app. The database reporter persists them into `server-error-report` (source 4 = audit-log origin); the Mattermost reporter forwards a short notification to `:error-report-webhook` when configured. +- Consequence for producers: crash reports only exist if the frontend collector is running and `push-audit-events` accepts `unhandled-exception`/`exception-page` events. Disabling the whole pipeline also blinds error reporting from the frontend. + +## Consumers III: telemetry (`app.loggers.audit` + `app.tasks.telemetry`) + +- Telemetry reuses the same table with anonymized shadow rows (`source LIKE 'telemetry:%'`): day-truncated timestamps, `0.0.0.0` IPs, props reduced to uuid/boolean/number values plus a few allowlisted fields (`lang`, `auth-backend`, derived `email-domain`, never raw emails), and a minimal context allowlist. Both full and shadow rows can coexist per event; that duplication is intentional. +- The telemetry cron ships shadow rows to `:telemetry-uri` as JSON in 10k batches, deletes them on success, and purges leftovers older than 7d. Nothing is collected or sent on official hosts (`telemetry-excluded?` covers `penpot.app`/`penpot.dev`). + +## Frontend ingestion (`app.rpc.commands.audit`, `app.main.data.event`) + +- The browser cannot write to the table directly; it POSTs transit batches to `push-audit-events`, which stamps server `id`, session `profile-id`, request ip and server `created-at`, and distrusts the client clock (future or >1h-lagging `tracked-at` is reset, original preserved in context). The endpoint is a no-op without `:audit-log`/`:telemetry` or on a read-only pool. +- The in-browser collector (`app.main.data.event`) only starts after `get-enabled-flags` confirms the backend wants events. It turns Potok events and explicit `ev/event` calls (nitrate membership changes, workspace file stats, crash reports) into a capped buffer (1024, chunks of 100, 2s debounce, current profile only) and sends fire-and-forget. `skip-audit?` exists for resumed dashboard actions so one user gesture is not counted twice. +- Because collection is best-effort and includes `PerformanceObserver` noise (`performance-*` triggers), backend tests must never assert exact frontend event counts. + +## Archival to Nexus and retention + +- Long-term storage lives outside Penpot in Nexus. Every 5m the `:audit-log-archive` cron takes chunks of 128 unarchived rows (`FOR UPDATE SKIP LOCKED`), POSTs them as transit `{:events [...]}` authenticated with `x-shared-key: "nexus "` (`:nexus-shared-key`, else derived from the instance secret), and marks `archived_at=now()` only on HTTP 204, in the same transaction. Anything else is retried on the next run; a missing URI with the flag on raises `:task-not-configured`. +- Every 5m the `:audit-log-gc` cron deletes all archived rows (no age filter), so archive must run before GC or data ships never. Cron dedup is best-effort (`mem:prod-infra/core`): two backends can fire the archiver twice, which is why the Nexus endpoint must be idempotent and the DB only marks acknowledged rows. +- Flags live in `common/flags.cljc` varia and are enabled as `PENPOT_FLAGS=enable-`: `:audit-log`, `:audit-log-archive`, `:audit-log-gc`, `:audit-log-logger` (structured `app.audit` log). `:telemetry-enabled` config auto-adds `:enable-telemetry`. + +## Tests + +- `backend_tests/rpc-audit-test.clj` exercises the whole backend path (full-row insert, telemetry-only and dual-row modes, `submit*`, no-op without flags, `insert` gating, `prepare-rpc-event` resolution) with `with-redefs [cf/flags #{...}]` against real `audit_log` rows. +- Other RPC suites mock `app.loggers.audit/submit` (nil return; `helpers.clj` stubs it globally) and assert on `:call-args-list`; any new command that must (or must not) emit an event needs the same treatment. diff --git a/.serena/memories/backend/core.md b/.serena/memories/backend/core.md index 9a567932b7..816a322a3b 100644 --- a/.serena/memories/backend/core.md +++ b/.serena/memories/backend/core.md @@ -8,6 +8,7 @@ Backend: JVM Clojure; Integrant; PostgreSQL; Redis/Valkey; RPC; HTTP; storage; m - Storage abstraction, logical buckets, object lifecycle, deduplication, access, and garbage collection: `mem:backend/storage`. - HTTP sessions, config, media processing, and file data persistence: `mem:backend/http-storage-filedata-subtleties`. - Auth flows, permission model, teams, projects, invitations, comments, webhooks, audit: `mem:backend/auth-permissions-product-domains` +- Audit-log event collection (RPC wrapper, frontend ingestion), telemetry duality, webhook fan-out, error reporters, Nexus archival and retention: `mem:backend/audit-log` - Services, task-queue/Pub-Sub topology constraints -> `mem:prod-infra/core`. ## Stable namespace map