From 03e24d18ce64242811978dba02258248b4f5526a Mon Sep 17 00:00:00 2001 From: Pablo Alba Date: Thu, 1 Oct 2026 17:17:11 +0200 Subject: [PATCH] :sparkles: Dual-send audit log archive to Nitrate and Nexus (#12037) Run the audit archive cron when :nexus or :admin-console is on. Ship allowlisted events to Admin Console first (with row ids for idempotency), then the full chunk to Nexus when :nexus is set. Mark archived_at for the whole chunk on success, including nitrate-only mode. Rename the gate flag from :audit-log-archive to :nexus. AI-assisted-by: Composer Co-authored-by: Cursor --- .serena/memories/backend/audit-log.md | 16 +- backend/scripts/_env | 2 +- .../src/app/loggers/audit/archive_task.clj | 97 ++++++-- backend/src/app/main.clj | 9 +- backend/src/app/nitrate.clj | 69 ++++-- .../loggers_audit_archive_task_test.clj | 227 ++++++++++++++++++ .../nitrate_ingest_audit_log_test.clj | 81 +++++++ common/src/app/common/flags.cljc | 4 +- 8 files changed, 455 insertions(+), 50 deletions(-) create mode 100644 backend/test/backend_tests/loggers_audit_archive_task_test.clj create mode 100644 backend/test/backend_tests/nitrate_ingest_audit_log_test.clj diff --git a/.serena/memories/backend/audit-log.md b/.serena/memories/backend/audit-log.md index ccd4153655..a77c3e75a4 100644 --- a/.serena/memories/backend/audit-log.md +++ b/.serena/memories/backend/audit-log.md @@ -5,11 +5,11 @@ Penpot records what users do as events in the Postgres `audit_log` table. There ## Purpose - The audit log answers "who did what, when, from where": every RPC mutation and selected frontend actions become a row with `name`, `type`, `profile-id`, `ip-addr`, `props` and `context`. Product analytics, abuse investigation and compliance exports all read from here, so keep events truthful and never put secrets in `props`. -- It is also the trigger bus for side effects: the same event object fans out to webhooks, error reporting and telemetry without the RPC handler knowing. New features should reuse this bus instead of building parallel notification paths. +- It is also the trigger bus for side effects: the same event object fans out to webhooks, error reporting and telemetry without the RPC handler knowing. New features should reuse this bus instead of building parallel notification paths. Archival consumers: Nexus and optionally Nitrate (Admin Console) via the archive task. ## Storage -- Live `audit_log` columns: `id` uuid PK default `gen_random_uuid()`; `name`/`type` text NOT NULL; `created_at` timestamptz NOT NULL default `now()` (server time, the source of truth); `tracked_at` timestamptz default `now()` (client-claimed time, corrected on ingest); `profile_id` uuid NOT NULL; `source` text telling full rows (`backend`/`frontend`) apart from anonymized copies (`telemetry:backend`/`telemetry:frontend`); `ip_addr` inet; `props`/`context` jsonb holding transit-encoded maps; `archived_at` timestamptz set once Nexus acknowledges the row. +- Live `audit_log` columns: `id` uuid PK default `gen_random_uuid()`; `name`/`type` text NOT NULL; `created_at` timestamptz NOT NULL default `now()` (server time, the source of truth); `tracked_at` timestamptz default `now()` (client-claimed time, corrected on ingest); `profile_id` uuid NOT NULL; `source` text telling full rows (`backend`/`frontend`) apart from anonymized copies (`telemetry:backend`/`telemetry:frontend`); `ip_addr` inet; `props`/`context` jsonb holding transit-encoded maps; `archived_at` timestamptz set once a successful archive-task chunk finishes (Nexus ack when `:nexus` is on; also set after nitrate-only when `:admin-console` is on without Nexus — see Archival). - Indexes: PK on `(id)`; partial `created_at WHERE archived_at IS NULL` serving the archive scan; partial `archived_at WHERE archived_at IS NOT NULL` serving the GC; `(source, created_at)` serving the telemetry scan. Each consumer has its own index, so a slow consumer never blocks the others. ## Backend producers (`app.loggers.audit`) @@ -42,11 +42,15 @@ Penpot records what users do as events in the Postgres `audit_log` table. There - The in-browser collector (`app.main.data.event`) only starts after `get-enabled-flags` confirms the backend wants events. It turns Potok events and explicit `ev/event` calls (nitrate membership changes, workspace file stats, crash reports) into a capped buffer (1024, chunks of 100, 2s debounce, current profile only) and sends fire-and-forget. `skip-audit?` exists for resumed dashboard actions so one user gesture is not counted twice. - Because collection is best-effort and includes `PerformanceObserver` noise (`performance-*` triggers), backend tests must never assert exact frontend event counts. -## Archival to Nexus and retention +## Archival to Nexus / Nitrate and retention -- Long-term storage lives outside Penpot in Nexus. Every 5m the `:audit-log-archive` cron takes chunks of 128 unarchived rows (`FOR UPDATE SKIP LOCKED`), POSTs them as transit `{:events [...]}` authenticated with `x-shared-key: "nexus "` (`:nexus-shared-key`, else derived from the instance secret), and marks `archived_at=now()` only on HTTP 204, in the same transaction. Anything else is retried on the next run; a missing URI with the flag on raises `:task-not-configured`. -- Every 5m the `:audit-log-gc` cron deletes all archived rows (no age filter), so archive must run before GC or data ships never. Cron dedup is best-effort (`mem:prod-infra/core`): two backends can fire the archiver twice, which is why the Nexus endpoint must be idempotent and the DB only marks acknowledged rows. -- Flags live in `common/flags.cljc` varia and are enabled as `PENPOT_FLAGS=enable-`: `:audit-log`, `:audit-log-archive`, `:audit-log-gc`, `:audit-log-logger` (structured `app.audit` log). `:telemetry-enabled` config auto-adds `:enable-telemetry`. +- Every 5m the `:audit-log-archive` **task** runs when **`:nexus` OR `:admin-console`** is on. It takes chunks of 128 unarchived rows (`FOR UPDATE SKIP LOCKED`). +- **Nitrate (Admin Console):** when `:admin-console` is on, allowlisted event names are POSTed via `nitrate/call :ingest-audit-log` (JSON batch `{:events [...]}`). The allowlist only shapes the HTTP body. +- **Nexus:** when `:nexus` is on, the full chunk is POSTed as transit `{:events [...]}` with `x-shared-key: "nexus "`. Missing URI with that flag raises `:task-not-configured`. +- **Both flags:** nitrate first, then Nexus on the same chunk. +- **`archived_at`:** set for **every row in the chunk** after success. With Nexus off, Nexus is treated as skipped success so nitrate-only still marks (including non-allowlisted names that were never POSTed). With Nexus on, mark only on Nexus HTTP 204; nitrate/Nexus failures leave rows unarchived for retry (Nitrate ingest must be idempotent). +- Every 5m the **`:audit-log-gc`** cron runs when **`:audit-log-gc` OR `:nexus` OR `:admin-console`**, and deletes all rows with `archived_at IS NOT NULL` (no age filter). Cron dedup is best-effort (`mem:prod-infra/core`). +- Flags: `:audit-log`, `:nexus`, `:audit-log-gc`, `:audit-log-logger`, `:admin-console`. Enabled as `PENPOT_FLAGS=enable-`. `:telemetry-enabled` config auto-adds `:enable-telemetry`. ## Tests diff --git a/backend/scripts/_env b/backend/scripts/_env index 0544911a02..4f7c5655bf 100644 --- a/backend/scripts/_env +++ b/backend/scripts/_env @@ -67,7 +67,7 @@ export PENPOT_FLAGS="\ enable-subscriptions"; # Uncomment for nexus integration testing -# export PENPOT_FLAGS="$PENPOT_FLAGS enable-audit-log-archive"; +# export PENPOT_FLAGS="$PENPOT_FLAGS enable-nexus"; # export PENPOT_AUDIT_LOG_ARCHIVE_URI="http://localhost:6070/api/audit"; # Default deletion delay for devenv diff --git a/backend/src/app/loggers/audit/archive_task.clj b/backend/src/app/loggers/audit/archive_task.clj index 1c5f953d44..b5190cff2b 100644 --- a/backend/src/app/loggers/audit/archive_task.clj +++ b/backend/src/app/loggers/audit/archive_task.clj @@ -13,6 +13,7 @@ [app.config :as cf] [app.db :as db] [app.http.client :as http] + [app.nitrate :as nitrate] [app.setup :as-alias setup] [integrant.core :as ig] [promesa.exec :as px])) @@ -69,15 +70,61 @@ :resp-body (:body resp)) false)))) +(def ^:private event-names-for-nitrate + #{"accept-organization-invitation" + "accept-team-invitation" + "accept-team-invitation-from" + "add-member-to-organization" + "add-team-to-organization" + "cancel-organization-invitation" + "change-organization-advanced-permission" + "create-file" + "create-organization" + "create-organization-invitation" + "create-project" + "create-team" + "create-team-access-request" + "create-team-invitation" + "create-team-invitations" + "create-webhook" + "delete-font" + "delete-organization" + "delete-project" + "delete-team" + "delete-team-invitation" + "delete-team-member" + "leave-team" + "move-project" + "move-team-to-organization" + "organization-sso-auth-failed" + "organization-sso-auth-started" + "organization-sso-auth-succeeded" + "permanently-delete-team-files" + "remove-organization-team" + "remove-team-from-organization" + "rename-organization" + "rename-project" + "restore-deleted-team-files" + "update-organization-invitation" + "update-organization-permissions" + "update-team-invitation" + "update-team-invitation-role" + "update-team-member-role" + "update-team-photo" + "verify-token"}) + +(defn- send-to-nitrate! + [cfg rows] + (when-let [events (->> rows + (filterv #(contains? event-names-for-nitrate (:name %))) + (not-empty))] + (nitrate/call cfg :ingest-audit-log {:events events}))) + (defn- mark-archived! [{:keys [::db/conn]} rows] (let [ids (db/create-array conn "uuid" (map :id rows))] (db/exec-one! conn ["update audit_log set archived_at=now() where id = ANY(?)" ids]))) -(def ^:private xf:create-event - (comp (map decode-row) - (map row->event))) - (def ^:private sql:get-audit-log-chunk "SELECT * FROM audit_log @@ -96,17 +143,30 @@ [{:keys [::uri] :as cfg}] (db/tx-run! cfg (fn [cfg] (when-let [rows (get-event-rows cfg)] - (let [events (into [] xf:create-event rows)] + (let [decoded (mapv decode-row rows) + events (mapv row->event decoded)] (l/trc :hint "archive events chunk" :uri uri :events (count events)) - (when (send! cfg events) - (mark-archived! cfg rows) - (count events))))))) + + (when (contains? cf/flags :admin-console) + (send-to-nitrate! cfg decoded)) + + ;; When :nexus is off, treat Nexus as skipped + ;; success so nitrate-only still marks the chunk. + ;; REPL :enabled true with neither send flag also marks + ;; with no outbound send — always set at least one flag. + (let [nexus-ok? (if (contains? cf/flags :nexus) + (send! cfg events) + true)] + (when nexus-ok? + (mark-archived! cfg rows) + (count events)))))))) (def ^:private schema:handler-params [:map ::db/pool ::setup/shared-keys - ::http/client]) + ::http/client + [:app.nitrate/client {:optional true} [:maybe :map]]]) (defmethod ig/assert-key ::handler [_ params] @@ -114,17 +174,23 @@ (defmethod ig/init-key ::handler [_ cfg] - (fn [params] + (fn [{:keys [props] :as params}] ;; NOTE: this let allows overwrite default configured values from - ;; the repl, when manually invoking the task. - (let [enabled (or (contains? cf/flags :audit-log-archive) - (:enabled params false)) + ;; the repl, when manually invoking the task. Prefer setting a send + ;; flag (:admin-console and/or :nexus); :enabled alone + ;; marks chunks without shipping. `invoke!` passes params under + ;; `:props`; direct REPL calls may pass a flat map. + (let [props (or props params) + enabled (or (contains? cf/flags :nexus) + (contains? cf/flags :admin-console) + (:enabled props false)) uri (cf/get :audit-log-archive-uri) - uri (or uri (:uri params)) + uri (or uri (:uri props)) cfg (assoc cfg ::uri uri)] - (when (and enabled (not uri)) + (when (and (contains? cf/flags :nexus) + (not uri)) (ex/raise :type :internal :code :task-not-configured :hint "archive task not configured, missing uri")) @@ -138,4 +204,3 @@ (when (pos? total) (l/dbg :hint "events archived" :total total)))))))) - diff --git a/backend/src/app/main.clj b/backend/src/app/main.clj index de72b9a0ef..c60f1362ed 100644 --- a/backend/src/app/main.clj +++ b/backend/src/app/main.clj @@ -575,7 +575,8 @@ :app.loggers.audit.archive-task/handler {::setup/shared-keys (ig/ref ::setup/shared-keys) ::http.client/client (ig/ref ::http.client/client) - ::db/pool (ig/ref ::db/pool)} + ::db/pool (ig/ref ::db/pool) + :app.nitrate/client (ig/ref :app.nitrate/client)} :app.loggers.audit.gc-task/handler {::db/pool (ig/ref ::db/pool)} @@ -654,11 +655,13 @@ {:cron #penpot/cron "0 30 */3,23 * * ?" :task :telemetry} - (when (contains? cf/flags :audit-log-archive) + (when (or (contains? cf/flags :nexus) + (contains? cf/flags :admin-console)) {:cron #penpot/cron "0 */5 * * * ?" ;; every 5m :task :audit-log-archive}) - (when (contains? cf/flags :audit-log-gc) + (when (or (contains? cf/flags :nexus) + (contains? cf/flags :admin-console)) {:cron #penpot/cron "30 */5 * * * ?" ;; every 5m :task :audit-log-gc})]} diff --git a/backend/src/app/nitrate.clj b/backend/src/app/nitrate.clj index 3b4b41eff0..a759e63c57 100644 --- a/backend/src/app/nitrate.clj +++ b/backend/src/app/nitrate.clj @@ -7,6 +7,7 @@ (ns app.nitrate "Module that make calls to the external nitrate aplication" (:require + [app.common.data :as d] [app.common.data.macros :as dm] [app.common.exceptions :as ex] [app.common.json :as json] @@ -496,6 +497,29 @@ schema:redeem-result (assoc params :throw-on-error? true))) +(def ^:private audit-event-keys + [:id :name :type :profile-id :props :context + :created-at :tracked-at :source :ip-addr]) + +(defn- ->audit-event + [event] + (-> (select-keys event audit-event-keys) + (d/without-nils))) + +(defn- ingest-audit-log-api + "POST a batch of audit events to Admin Console `/api/audit-log`. + Expects HTTP 204; returns nil on success. 4xx raises when + `:throw-on-error?` is set." + [cfg {:keys [events] :as params}] + (let [request-params {:events (mapv ->audit-event events)} + params (assoc params + :request-params request-params + :throw-on-error? true)] + (request-to-nitrate cfg :post + (generate-nitrate-uri "api/audit-log") + nil + params))) + ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; INITIALIZATION ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; @@ -505,28 +529,29 @@ (when (contains? cf/flags :admin-console) (when (nil? (cf/get :admin-console-uri)) (l/warn :hint "admin console is not configured; nitrate calls will fail until the setup is complete")) - {:get-team-organization (partial get-team-organization-api cfg) - :get-teams-organizations (partial get-teams-organizations-api cfg) - :set-team-organization (partial set-team-organization-api cfg) - :get-organization-membership (partial get-organization-membership-api cfg) - :get-organization-membership-by-team (partial get-organization-membership-by-team-api cfg) - :get-organization-summary (partial get-organization-summary-api cfg) - :get-owned-organizations (partial get-owned-organizations-api cfg) - :get-owned-organizations-summary (partial get-owned-organizations-summary-api cfg) - :get-organization-members (partial get-organization-members-api cfg) - :cleanup-deleted-penpot-user (partial cleanup-deleted-penpot-user-api cfg) - :add-profile-to-organization (partial add-profile-to-organization-api cfg) - :remove-profile-from-organization (partial remove-profile-from-organization-api cfg) - :get-organization-permissions (partial get-organization-permissions-api cfg) - :get-organization-sso-by-team (partial get-organization-sso-by-team-api cfg) - :get-organization-sso (partial get-organization-sso-api cfg) - :delete-team (partial delete-team-api cfg) - :remove-team-from-organization (partial remove-team-from-organization-api cfg) - :get-subscription (partial get-subscription-api cfg) - :get-subscription-warning (partial get-subscription-warning-api cfg) - :connectivity (partial get-connectivity-api cfg) - :get-identity (partial get-identity-api cfg) - :redeem-activation-code (partial redeem-activation-code-api cfg)})) + {:get-team-organization (partial get-team-organization-api cfg) + :get-teams-organizations (partial get-teams-organizations-api cfg) + :set-team-organization (partial set-team-organization-api cfg) + :get-organization-membership (partial get-organization-membership-api cfg) + :get-organization-membership-by-team (partial get-organization-membership-by-team-api cfg) + :get-organization-summary (partial get-organization-summary-api cfg) + :get-owned-organizations (partial get-owned-organizations-api cfg) + :get-owned-organizations-summary (partial get-owned-organizations-summary-api cfg) + :get-organization-members (partial get-organization-members-api cfg) + :cleanup-deleted-penpot-user (partial cleanup-deleted-penpot-user-api cfg) + :add-profile-to-organization (partial add-profile-to-organization-api cfg) + :remove-profile-from-organization (partial remove-profile-from-organization-api cfg) + :get-organization-permissions (partial get-organization-permissions-api cfg) + :get-organization-sso-by-team (partial get-organization-sso-by-team-api cfg) + :get-organization-sso (partial get-organization-sso-api cfg) + :delete-team (partial delete-team-api cfg) + :remove-team-from-organization (partial remove-team-from-organization-api cfg) + :get-subscription (partial get-subscription-api cfg) + :get-subscription-warning (partial get-subscription-warning-api cfg) + :connectivity (partial get-connectivity-api cfg) + :get-identity (partial get-identity-api cfg) + :redeem-activation-code (partial redeem-activation-code-api cfg) + :ingest-audit-log (partial ingest-audit-log-api cfg)})) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; UTILS diff --git a/backend/test/backend_tests/loggers_audit_archive_task_test.clj b/backend/test/backend_tests/loggers_audit_archive_task_test.clj new file mode 100644 index 0000000000..f8011c8b34 --- /dev/null +++ b/backend/test/backend_tests/loggers_audit_archive_task_test.clj @@ -0,0 +1,227 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns backend-tests.loggers-audit-archive-task-test + (:require + [app.common.json :as json] + [app.common.time :as ct] + [app.common.uuid :as uuid] + [app.config :as cf] + [app.db :as db] + [app.http.client :as-alias http] + [app.setup :as-alias setup] + [backend-tests.helpers :as th] + [clojure.test :as t] + [cuerdas.core :as str] + [integrant.core :as ig] + [mockery.core :refer [with-mocks]])) + +(t/use-fixtures :once th/state-init) +(t/use-fixtures :each th/database-reset) + +(def ^:private archive-uri "http://nexus.example/archive") + +(defn- insert-audit-row! + [profile-id name] + (th/db-insert! :audit-log + {:id (uuid/next) + :name name + :type "action" + :source "backend" + :profile-id profile-id + :ip-addr (db/inet "127.0.0.1") + :props (db/tjson {:team-id (uuid/next)}) + :context (db/tjson {}) + :tracked-at (ct/now) + :created-at (ct/now)})) + +(t/deftest archive-events-sends-to-nitrate-then-nexus + ;; Create profile before enabling :admin-console — system nitrate + ;; client is nil in tests, and team bootstrap would call it. + (let [prof (th/create-profile* 1 {:is-active true}) + order (atom [])] + (with-redefs [cf/flags #{:nexus :admin-console}] + (with-mocks [nitrate-mock {:target 'app.nitrate/call + :return (fn [& _] + (swap! order conj :nitrate) + nil)} + http-mock {:target 'app.http.client/req + :return (fn [& _] + (swap! order conj :nexus) + {:status 204})}] + (insert-audit-row! (:id prof) "create-project") + (insert-audit-row! (:id prof) "create-file") + (th/run-task! :audit-log-archive {:uri archive-uri}) + (t/is (:called? @nitrate-mock)) + (t/is (:called? @http-mock)) + (t/is (= 1 (:call-count @nitrate-mock))) + (t/is (= 1 (:call-count @http-mock))) + (t/is (= [:nitrate :nexus] @order)) + (let [[_ method params] (:call-args @nitrate-mock)] + (t/is (= :ingest-audit-log method)) + (t/is (= 2 (count (:events params)))) + (t/is (= #{"create-project" "create-file"} + (into #{} (map :name) (:events params)))) + (t/is (every? uuid? (map :id (:events params))))) + (let [rows (th/db-exec! ["select * from audit_log where archived_at is not null"])] + (t/is (= 2 (count rows)))))))) + +(t/deftest archive-events-filters-nitrate-event-names + (let [prof (th/create-profile* 1 {:is-active true})] + (with-redefs [cf/flags #{:nexus :admin-console}] + (with-mocks [nitrate-mock {:target 'app.nitrate/call :return nil} + http-mock {:target 'app.http.client/req :return {:status 204}}] + (insert-audit-row! (:id prof) "create-project") + (insert-audit-row! (:id prof) "unrelated-event") + (th/run-task! :audit-log-archive {:uri archive-uri}) + (t/is (:called? @nitrate-mock)) + (t/is (:called? @http-mock)) + (let [[_ _ params] (:call-args @nitrate-mock)] + (t/is (= ["create-project"] (mapv :name (:events params))))) + (let [rows (th/db-exec! ["select * from audit_log where archived_at is not null"])] + (t/is (= 2 (count rows)))))))) + +(t/deftest archive-events-skips-nitrate-when-no-matching-names + (let [prof (th/create-profile* 1 {:is-active true})] + (with-redefs [cf/flags #{:nexus :admin-console}] + (with-mocks [nitrate-mock {:target 'app.nitrate/call :return nil} + http-mock {:target 'app.http.client/req :return {:status 204}}] + (insert-audit-row! (:id prof) "unrelated-event") + (th/run-task! :audit-log-archive {:uri archive-uri}) + (t/is (false? (:called? @nitrate-mock))) + (t/is (:called? @http-mock)) + (let [rows (th/db-exec! ["select * from audit_log where archived_at is not null"])] + (t/is (= 1 (count rows)))))))) + +(t/deftest archive-events-admin-console-only-marks-without-uri + (let [prof (th/create-profile* 1 {:is-active true})] + (with-redefs [cf/flags #{:admin-console}] + (with-mocks [nitrate-mock {:target 'app.nitrate/call :return nil} + http-mock {:target 'app.http.client/req :return {:status 204}}] + (insert-audit-row! (:id prof) "create-project") + (th/run-task! :audit-log-archive {}) + (t/is (:called? @nitrate-mock)) + (t/is (false? (:called? @http-mock))) + (let [rows (th/db-exec! ["select * from audit_log where archived_at is not null"])] + (t/is (= 1 (count rows)))))))) + +(t/deftest archive-events-admin-console-only-marks-non-allowlisted + (let [prof (th/create-profile* 1 {:is-active true})] + (with-redefs [cf/flags #{:admin-console}] + (with-mocks [nitrate-mock {:target 'app.nitrate/call :return nil} + http-mock {:target 'app.http.client/req :return {:status 204}}] + (insert-audit-row! (:id prof) "unrelated-event") + (th/run-task! :audit-log-archive {}) + (t/is (false? (:called? @nitrate-mock))) + (t/is (false? (:called? @http-mock))) + (let [rows (th/db-exec! ["select * from audit_log where archived_at is not null"])] + (t/is (= 1 (count rows)))))))) + +(t/deftest archive-events-propagates-nitrate-error + (let [prof (th/create-profile* 1 {:is-active true})] + (with-redefs [cf/flags #{:nexus :admin-console}] + (with-mocks [nitrate-mock {:target 'app.nitrate/call + :throw (ex-info "nitrate down" {})} + http-mock {:target 'app.http.client/req :return {:status 204}}] + (insert-audit-row! (:id prof) "create-project") + (t/is (thrown? clojure.lang.ExceptionInfo + (th/run-task! :audit-log-archive {:uri archive-uri}))) + (t/is (:called? @nitrate-mock)) + (t/is (false? (:called? @http-mock))) + (let [rows (th/db-exec! ["select * from audit_log where archived_at is null"])] + (t/is (= 1 (count rows)))))))) + +(t/deftest archive-events-uses-nexus-without-admin-console + (let [prof (th/create-profile* 1 {:is-active true})] + (with-redefs [cf/flags #{:nexus}] + (with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}} + nitrate-mock {:target 'app.nitrate/call :return nil}] + (insert-audit-row! (:id prof) "create-project") + (th/run-task! :audit-log-archive {:uri archive-uri}) + (t/is (false? (:called? @nitrate-mock))) + (t/is (:called? @http-mock)) + (let [rows (th/db-exec! ["select * from audit_log where archived_at is not null"])] + (t/is (= 1 (count rows)))))))) + +(t/deftest archive-events-skips-mark-when-nexus-fails + (let [prof (th/create-profile* 1 {:is-active true})] + (with-redefs [cf/flags #{:nexus}] + (with-mocks [http-mock {:target 'app.http.client/req :return {:status 500}}] + (insert-audit-row! (:id prof) "create-project") + (th/run-task! :audit-log-archive {:uri archive-uri}) + (t/is (:called? @http-mock)) + (let [rows (th/db-exec! ["select * from audit_log where archived_at is null"])] + (t/is (= 1 (count rows)))))))) + +(t/deftest archive-events-both-flags-skips-mark-when-nexus-fails + (let [prof (th/create-profile* 1 {:is-active true})] + (with-redefs [cf/flags #{:nexus :admin-console}] + (with-mocks [nitrate-mock {:target 'app.nitrate/call :return nil} + http-mock {:target 'app.http.client/req :return {:status 500}}] + (insert-audit-row! (:id prof) "create-project") + (th/run-task! :audit-log-archive {:uri archive-uri}) + (t/is (:called? @nitrate-mock)) + (t/is (:called? @http-mock)) + (let [rows (th/db-exec! ["select * from audit_log where archived_at is null"])] + (t/is (= 1 (count rows)))))))) + +(t/deftest archive-events-encodes-db-rows-for-nitrate + (let [prof (th/create-profile* 1 {:is-active true})] + (with-redefs [cf/flags #{:nexus :admin-console} + cf/config (assoc cf/config + :admin-console-uri "http://ac.example/admin-console/" + :admin-console-shared-key "ac-key" + :nexus-shared-key "nexus-key")] + (let [nitrate-bodies (atom []) + shared-keys {:admin-console "ac-key" :nexus "nexus-key"} + client (ig/init-key :app.nitrate/client + {::http/client (Object.) + ::setup/shared-keys shared-keys}) + handler (ig/init-key :app.loggers.audit.archive-task/handler + {::db/pool (:app.db/pool th/*system*) + ::setup/shared-keys shared-keys + ::http/client (:app.http.client/client th/*system*) + :app.nitrate/client client})] + (with-mocks [http-mock {:target 'app.http.client/req + :return + (fn [_ req & _] + (when (str/includes? (str (:uri req)) "api/audit-log") + (swap! nitrate-bodies conj (:body req))) + {:status 204})}] + (let [team-id (uuid/next) + row (th/db-insert! :audit-log + {:id (uuid/next) + :name "create-project" + :type "action" + :source "backend" + :profile-id (:id prof) + :ip-addr (db/inet "127.0.0.1") + :props (db/tjson {:team-id team-id}) + :context (db/tjson {}) + :tracked-at (ct/now) + :created-at (ct/now)})] + (handler {:props {:uri archive-uri}}) + (t/is (= 1 (count @nitrate-bodies))) + (let [body (json/decode (first @nitrate-bodies) :key-fn json/read-kebab-key) + event (first (:events body))] + (t/is (= 1 (count (:events body)))) + (t/is (= (str (:id row)) (str (:id event)))) + (t/is (= "create-project" (:name event))) + (t/is (= "127.0.0.1" (:ip-addr event))) + (t/is (string? (:created-at event))) + (t/is (string? (:tracked-at event))) + (t/is (= (str team-id) (str (get-in event [:props :team-id]))))) + (let [rows (th/db-exec! ["select * from audit_log where archived_at is not null"])] + (t/is (= 1 (count rows)))))))))) + +(t/deftest archive-events-requires-uri-when-nexus-flag-set + (with-redefs [cf/flags #{:nexus} + cf/config (dissoc cf/config :audit-log-archive-uri)] + (let [data (ex-data (try + (th/run-task! :audit-log-archive {}) + (catch Throwable cause + cause)))] + (t/is (= :task-not-configured (:code data)))))) diff --git a/backend/test/backend_tests/nitrate_ingest_audit_log_test.clj b/backend/test/backend_tests/nitrate_ingest_audit_log_test.clj new file mode 100644 index 0000000000..b8a679d842 --- /dev/null +++ b/backend/test/backend_tests/nitrate_ingest_audit_log_test.clj @@ -0,0 +1,81 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns backend-tests.nitrate-ingest-audit-log-test + (:require + [app.common.json :as json] + [app.common.time :as ct] + [app.common.uuid :as uuid] + [app.config :as cf] + [app.http.client :as-alias http] + [app.nitrate :as nitrate] + [app.rpc :as-alias rpc] + [app.setup :as-alias setup] + [backend-tests.helpers :as th] + [clojure.test :as t] + [cuerdas.core :as str] + [integrant.core :as ig] + [mockery.core :refer [with-mocks]])) + +(t/use-fixtures :once th/state-init) + +(t/deftest ingest-audit-log-posts-events-batch + (binding [cf/flags (conj cf/flags :admin-console) + cf/config (assoc cf/config :admin-console-uri "http://ac.example/admin-console/")] + (with-mocks [http-mock {:target 'app.http.client/req + :return {:status 204}}] + (let [client (ig/init-key :app.nitrate/client + {::http/client (Object.) + ::setup/shared-keys {:admin-console "test-shared-key"}}) + cfg {:app.nitrate/client client} + p1 (uuid/next) + p2 (uuid/next) + e1 (uuid/next) + e2 (uuid/next) + team-id (uuid/next) + now (ct/now)] + (nitrate/call cfg :ingest-audit-log + {::rpc/profile-id p1 + :events [{:id e1 + :name "create-project" + :type "action" + :profile-id p1 + :props {:team-id team-id} + :context {} + :created-at now + :tracked-at now + :source "backend" + :ip-addr "127.0.0.1"} + {:id e2 + :name "create-file" + :type "action" + :profile-id p2 + :props {} + :context {} + :created-at now + :tracked-at now + :source "backend" + :ip-addr "10.0.0.1"}]}) + (t/is (= 1 (:call-count @http-mock))) + (let [[_ req] (:call-args @http-mock) + body (json/decode (:body req) :key-fn json/read-kebab-key) + e1' (first (:events body)) + e2' (second (:events body))] + (t/is (= :post (:method req))) + (t/is (str/ends-with? (str (:uri req)) "api/audit-log")) + (t/is (= 2 (count (:events body)))) + (t/is (= "create-project" (:name e1'))) + (t/is (= "create-file" (:name e2'))) + (t/is (= (str e1) (str (:id e1')))) + (t/is (= (str e2) (str (:id e2')))) + (t/is (= (str p1) (str (:profile-id e1')))) + (t/is (= (str p2) (str (:profile-id e2')))) + (t/is (= "127.0.0.1" (:ip-addr e1'))) + (t/is (= "10.0.0.1" (:ip-addr e2'))) + (t/is (string? (:created-at e1'))) + (t/is (string? (:tracked-at e1'))) + (t/is (= (str now) (:created-at e1'))) + (t/is (= (str team-id) (str (get-in e1' [:props :team-id]))))))))) diff --git a/common/src/app/common/flags.cljc b/common/src/app/common/flags.cljc index 60353f730f..7589a23668 100644 --- a/common/src/app/common/flags.cljc +++ b/common/src/app/common/flags.cljc @@ -60,8 +60,8 @@ (def varia "Rest of the flags" #{:audit-log - :audit-log-archive - :audit-log-gc + ;; Enables shipping audit_log chunks to Nexus. + :nexus :audit-log-logger :auto-file-snapshot ;; enables the `/api/doc` endpoint that lists all the rpc methods available.