github/penpot
1
0
Fork 0
mirror of https://github.com/penpot/penpot.git synced 2026-10-29 05:50:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

✨ Add profile name and email to nitrate audit log (#12636)

Browse Source 
* ✨ Add profile name and email to nitrate audit log

* ✨ Mr changes
This commit is contained in:
Pablo Alba 2026-10-27 09:30:28 +01:00 committed by GitHub
parent 4b13562d02
commit e4c1b154e3
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
5 changed files with 157 additions and 20 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.serena/memories/audit-log.md
View File

@ -277,7 +277,7 @@ Nexus dispatches on `:type`: `"telemetry-events"` batches are coerced and insert
### How it works today
Every 5 min each archive job drains its own queue (`:audit-log-consumer-nexus` on flag `:nexus`, cron `0 */5 * * * ?`; `:audit-log-consumer-admin-console` on flag `:admin-console`, cron `30 */5 * * * ?`, 30s offset so the two drains never start on the same second), in 128-row chunks taken oldest-first (`ORDER BY created_at ASC, id ASC`) with `FOR UPDATE OF audit_log` (`SET LOCAL lock_timeout = '30s'`) inside the same transaction that sends and completes them, with 100 ms pauses between chunks. Batches arrive at Nexus in chronological order (it numbers by arrival for order-sensitive consumers). Nexus receives transit `{:events [...]}` with `x-shared-key: "nexus <key>"` (`:nexus-shared-key`, or derived from the instance secret) and `origin` = public-uri; flag on but no URI raises `:task-not-configured`. The admin console receives only its allowlisted event names (enriched with team names and emails); anything filtered out counts as done for the admin-console queue. A chunk whose destination refuses it (Nexus non-204 raises `:consumer-chunk-refused`) or throws (transport failure, admin-console error) rolls back and stays for the next tick, and the job aborts as failed: it archives what it can, and the cron retries the rest. Committed chunks stay sent. The send fns return counts or throw carrying the attempted count as `:events`; `run-single-consumer` records the `failed` metric from it, outside every chunk transaction (a local DB failure carries no `:events` and stays uncounted — it is not a destination failure). No `warn` before the throw: the runner reports the aborted job. Each job touches only its own queue, so a fallen Nexus never starves the admin console (nor vice versa).
Every 5 min each archive job drains its own queue (`:audit-log-consumer-nexus` on flag `:nexus`, cron `0 */5 * * * ?`; `:audit-log-consumer-admin-console` on flag `:admin-console`, cron `30 */5 * * * ?`, 30s offset so the two drains never start on the same second), in 128-row chunks taken oldest-first (`ORDER BY created_at ASC, id ASC`) with `FOR UPDATE OF audit_log` (`SET LOCAL lock_timeout = '30s'`) inside the same transaction that sends and completes them, with 100 ms pauses between chunks. Batches arrive at Nexus in chronological order (it numbers by arrival for order-sensitive consumers). Nexus receives transit `{:events [...]}` with `x-shared-key: "nexus <key>"` (`:nexus-shared-key`, or derived from the instance secret) and `origin` = public-uri; flag on but no URI raises `:task-not-configured`. The admin console receives only its allowlisted event names (enriched with team names, props emails, and optional top-level `:profile-email` / `:profile-name` from the actor `:profile-id`); anything filtered out counts as done for the admin-console queue. A chunk whose destination refuses it (Nexus non-204 raises `:consumer-chunk-refused`) or throws (transport failure, admin-console error) rolls back and stays for the next tick, and the job aborts as failed: it archives what it can, and the cron retries the rest. Committed chunks stay sent. The send fns return counts or throw carrying the attempted count as `:events`; `run-single-consumer` records the `failed` metric from it, outside every chunk transaction (a local DB failure carries no `:events` and stays uncounted — it is not a destination failure). No `warn` before the throw: the runner reports the aborted job. Each job touches only its own queue, so a fallen Nexus never starves the admin console (nor vice versa).
A chunk is **one POST** and the task advances only on 204, so a single oversized event wedges the whole queue: a crash report whose message embeds a whole payload (a serialized file, for example) can push the request past the proxy body limit and the rejection blocks every later event until it is pruned by hand. Frontend crash reports are therefore size-bounded at the source (`mem:frontend/routing-app-shell-subtleties`): `app.common.exceptions/get-hint` and the CLJS `format-throwable` clamp exception messages (`max-message-length`).

61
backend/src/app/loggers/audit/consumer_task.clj
View File

@ -289,6 +289,19 @@
id->email-prop)))
events))
(defn- collect-actor-ids-needing-profile
"Actor ids (event :profile-id) that still need top-level :profile-email
and/or :profile-name. Skips uuid/zero and non-uuids."
[events]
(into #{}
(keep (fn [event]
(when (or (not (contains? event :profile-email))
(not (contains? event :profile-name)))
(when-let [id (uuid/coerce (:profile-id event))]
(when-not (uuid/zero? id)
id)))))
events))
(defn- load-team-names
[conn ids]
(if (seq ids)
@ -297,12 +310,13 @@
(into {} (map (juxt :id :name) rows)))
{}))
(defn- load-profile-emails
(defn- load-profiles
"Batch-load profile email and fullname (including soft-deleted)."
[conn ids]
(if (seq ids)
(let [arr (db/create-array conn "uuid" ids)
rows (db/exec! conn ["SELECT id, email FROM profile WHERE id = ANY(?)" arr])]
(into {} (map (juxt :id :email) rows)))
rows (db/exec! conn ["SELECT id, email, fullname FROM profile WHERE id = ANY(?)" arr])]
(into {} (map (juxt :id identity) rows)))
{}))
(defn- maybe-assoc-team-name
@ -315,33 +329,58 @@
event)))
(defn- maybe-assoc-emails
[event emails]
[event profiles]
(update event :props
(fn [props]
(reduce-kv (fn [props id-key email-key]
(if-let [email (and (not (contains? props email-key))
(when-let [id (uuid/coerce (get props id-key))]
(get emails id)))]
(:email (get profiles id))))]
(assoc props email-key email)
props))
props
id->email-prop))))
(defn- maybe-assoc-actor-profile
"Fill-only top-level :profile-email / :profile-name from the event
actor (:profile-id). Omits keys when the id is missing, zero, the
profile row is absent, or the corresponding column is empty
(fullname defaults to \"\" in the schema)."
[event profiles]
(let [id (uuid/coerce (:profile-id event))]
(if (or (nil? id) (uuid/zero? id))
event
(if-let [profile (get profiles id)]
(cond-> event
(and (not (contains? event :profile-email))
(not-empty (:email profile)))
(assoc :profile-email (:email profile))
(and (not (contains? event :profile-name))
(not-empty (:fullname profile)))
(assoc :profile-name (:fullname profile)))
event))))
(defn- enrich-events-for-admin-console
"Enrich allowlisted events before the admin-console ingest:
1. Fill missing :team-name from team id (including soft-deleted).
2. Fill missing emails from matching id keys (including soft-deleted
profiles): :profile-email, :user-email, :member-email,
:user-who-send-invitation-email."
2. Fill missing emails from matching id keys in props (including
soft-deleted profiles): :profile-email, :user-email, :member-email,
:user-who-send-invitation-email.
3. Fill missing top-level :profile-email / :profile-name from the
event actor (:profile-id), including soft-deleted profiles; omit
when there is no usable actor id or the value is empty."
[{:keys [::db/conn]} events]
(let [team-ids (collect-team-ids-needing-name events)
profile-ids (collect-profile-ids-needing-email events)
profile-ids (into (collect-profile-ids-needing-email events)
(collect-actor-ids-needing-profile events))
team-names (load-team-names conn team-ids)
emails (load-profile-emails conn profile-ids)]
profiles (load-profiles conn profile-ids)]
(mapv (fn [event]
(-> event
(maybe-assoc-team-name team-names)
(maybe-assoc-emails emails)))
(maybe-assoc-emails profiles)
(maybe-assoc-actor-profile profiles)))
events)))
(defn- send-to-admin-console

4
backend/src/app/nitrate.clj
View File

@ -483,8 +483,8 @@
(assoc params :throw-on-error? true)))
(def ^:private audit-event-keys
[:id :name :type :profile-id :props :context
:created-at :tracked-at :source :ip-addr])
[:id :name :type :profile-id :profile-email :profile-name
:props :context :created-at :tracked-at :source :ip-addr])
(defn- ->audit-event
[event]

104
backend/test/backend_tests/loggers_audit_consumer_task_test.clj
View File

@ -64,10 +64,14 @@
(insert-pendings! (:id row))
row)))
(defn- admin-console-event-props
(defn- admin-console-event
[admin-console-mock]
(let [[_ _ params] (:call-args @admin-console-mock)]
(:props (first (:events params)))))
(first (:events params))))
(defn- admin-console-event-props
[admin-console-mock]
(:props (admin-console-event admin-console-mock)))
(defn- run-both!
"Run both consumer jobs, Nexus first (the historical single-run
@ -319,12 +323,16 @@
:user-who-send-invitation (str (:id actor))})
(th/run-task! :audit-log-consumer-admin-console {})
(t/is (:called? @admin-console-mock))
(let [props (admin-console-event-props admin-console-mock)]
(let [event (admin-console-event admin-console-mock)
props (:props event)]
(t/is (= (:name team) (:team-name props)))
(t/is (= (:email actor) (:profile-email props)))
(t/is (= (:email member) (:user-email props)))
(t/is (= (:email member) (:member-email props)))
(t/is (= (:email actor) (:user-who-send-invitation-email props))))
(t/is (= (:email actor) (:user-who-send-invitation-email props)))
;; Top-level actor fields (event :profile-id), distinct from props.
(t/is (= (:email actor) (:profile-email event)))
(t/is (= (:fullname actor) (:profile-name event))))
(let [rows (th/db-exec! ["select * from audit_log"])]
(t/is (= 0 (count rows))))))))
@ -340,12 +348,48 @@
{:team-id (:id team)
:profile-id (:id other)})
(th/run-task! :audit-log-consumer-admin-console {})
(let [props (admin-console-event-props admin-console-mock)]
(let [event (admin-console-event admin-console-mock)
props (:props event)]
(t/is (= (:name team) (:team-name props)))
(t/is (= (:email other) (:profile-email props))))
(t/is (= (:email other) (:profile-email props)))
(t/is (= (:email actor) (:profile-email event)))
(t/is (= (:fullname actor) (:profile-name event))))
(let [rows (th/db-exec! ["select * from audit_log"])]
(t/is (= 0 (count rows))))))))
(t/deftest consumer-events-enriches-soft-deleted-actor-profile
(let [actor (th/create-profile* 1 {:is-active true})
team (th/create-team* 1 {:profile-id (:id actor)})]
(th/db-update! :profile {:deleted-at (ct/now)} {:id (:id actor)})
(with-redefs [cf/flags #{:admin-console}]
(with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}]
(insert-audit-row! (:id actor) "create-project"
{:team-id (:id team)})
(th/run-task! :audit-log-consumer-admin-console {})
(let [event (admin-console-event admin-console-mock)]
(t/is (= (:email actor) (:profile-email event)))
(t/is (= (:fullname actor) (:profile-name event))))))))
(t/deftest consumer-events-omits-actor-profile-when-no-usable-id
(let [actor (th/create-profile* 1 {:is-active true})
team (th/create-team* 1 {:profile-id (:id actor)})
missing (uuid/next)]
(with-redefs [cf/flags #{:admin-console}]
(with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}]
(insert-audit-row! uuid/zero "create-project" {:team-id (:id team)})
(insert-audit-row! missing "create-team" {:team-id (:id team)})
(th/run-task! :audit-log-consumer-admin-console {})
(t/is (:called? @admin-console-mock))
(let [[_ _ params] (:call-args @admin-console-mock)
events (:events params)
by-name (into {} (map (juxt :name identity) events))
zero-evt (get by-name "create-project")
missing-evt (get by-name "create-team")]
(t/is (not (contains? zero-evt :profile-email)))
(t/is (not (contains? zero-evt :profile-name)))
(t/is (not (contains? missing-evt :profile-email)))
(t/is (not (contains? missing-evt :profile-name))))))))
(t/deftest consumer-events-does-not-overwrite-existing-enrich-props
(let [actor (th/create-profile* 1 {:is-active true})
team (th/create-team* 1 {:profile-id (:id actor)})]
@ -371,6 +415,54 @@
(t/is (= "already-inviter@example.com"
(:user-who-send-invitation-email props))))))))
(t/deftest consumer-events-does-not-overwrite-existing-actor-profile-fields
;; Top-level :profile-email / :profile-name are not stored on the row;
;; exercise fill-only via the enrich helper directly.
(let [actor (th/create-profile* 1 {:is-active true})
event {:profile-id (:id actor)
:profile-email "preset@example.com"
:profile-name "Preset Name"
:props {}}]
(db/tx-run! th/*system*
(fn [cfg]
(let [enriched (first (#'consumer/enrich-events-for-admin-console
cfg [event]))]
(t/is (= "preset@example.com" (:profile-email enriched)))
(t/is (= "Preset Name" (:profile-name enriched))))))))
(t/deftest consumer-events-fills-missing-actor-profile-field-independently
;; Each top-level key fills on its own; guards an or→and regression in
;; collect-actor-ids-needing-profile.
(let [actor (th/create-profile* 1 {:is-active true})]
(db/tx-run! th/*system*
(fn [cfg]
(let [email-only (first (#'consumer/enrich-events-for-admin-console
cfg [{:profile-id (:id actor)
:profile-email "preset@example.com"
:props {}}]))
name-only (first (#'consumer/enrich-events-for-admin-console
cfg [{:profile-id (:id actor)
:profile-name "Preset Name"
:props {}}]))]
(t/is (= "preset@example.com" (:profile-email email-only)))
(t/is (= (:fullname actor) (:profile-name email-only)))
(t/is (= (:email actor) (:profile-email name-only)))
(t/is (= "Preset Name" (:profile-name name-only))))))))
(t/deftest consumer-events-omits-empty-actor-profile-name
;; profile.fullname is NOT NULL DEFAULT ''; empty must not become
;; :profile-name "".
(let [actor (th/create-profile* 1 {:is-active true})
team (th/create-team* 1 {:profile-id (:id actor)})]
(th/db-update! :profile {:fullname ""} {:id (:id actor)})
(with-redefs [cf/flags #{:admin-console}]
(with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}]
(insert-audit-row! (:id actor) "create-project" {:team-id (:id team)})
(th/run-task! :audit-log-consumer-admin-console {})
(let [event (admin-console-event admin-console-mock)]
(t/is (= (:email actor) (:profile-email event)))
(t/is (not (contains? event :profile-name))))))))
(t/deftest consumer-events-enrich-skips-missing-and-non-uuid-ids
(let [actor (th/create-profile* 1 {:is-active true})]
(with-redefs [cf/flags #{:admin-console}]

6
backend/test/backend_tests/nitrate_ingest_audit_log_test.clj
View File

@ -43,6 +43,8 @@
:name "create-project"
:type "action"
:profile-id p1
:profile-email "ada@example.com"
:profile-name "Ada"
:props {:team-id team-id}
:context {}
:created-at now
@ -73,6 +75,10 @@
(t/is (= (str e2) (str (:id e2'))))
(t/is (= (str p1) (str (:profile-id e1'))))
(t/is (= (str p2) (str (:profile-id e2'))))
(t/is (= "ada@example.com" (:profile-email e1')))
(t/is (= "Ada" (:profile-name e1')))
(t/is (nil? (:profile-email e2')))
(t/is (nil? (:profile-name e2')))
(t/is (= "127.0.0.1" (:ip-addr e1')))
(t/is (= "10.0.0.1" (:ip-addr e2')))
(t/is (string? (:created-at e1')))