* 🎉 Add telemetry anonymous event collection
Rewrite the audit logging subsystem to support three operating modes and
add anonymous telemetry event collection:
Modes:
- A (audit-log only): events persisted with full context
- B (audit-log + telemetry): same as A, plus events are collected for
telemetry shipping
- C (telemetry-only): events stored anonymously with PII stripped,
telemetry flag active, audit-log flag inactive
Audit system refactoring (app.loggers.audit):
- Replace qualified map keys (::audit/name etc.) with plain keywords
- Rename submit! -> submit, insert! -> insert, prepare-event ->
prepare-rpc-event
- Add submit* as a lower-level public API
- Add process-event dispatch function that handles all three modes and
webhooks in a single tx-run!
- Add :id to event schema (auto-generated if omitted)
- Add filter-telemetry-props: anonymises event props per event type.
Keeps UUID/boolean/number values; for login/identify events preserves
lang, auth-backend, email-domain; for navigate events preserves route,
file-id, team-id, page-id; instance-start trigger passes through.
- Add filter-telemetry-context: retains only safe context keys.
Backend: version, initiator, client-version, client-user-agent.
Frontend: browser, os, locale, screen metrics, event-origin.
- Timestamps truncated to day precision via ct/truncate for telemetry
storage
- PII stripped: props emptied, ip-addr zeroed, session-linking and
access-token fields removed from context
Config (app.config):
- Derive :enable-telemetry flag from telemetry-enabled config option
Email utilities (app.email):
- Add email/clean and email/get-domain helper functions for domain
extraction from email addresses
Setup (app.setup):
- Emit instance-start trigger event at system startup
- Simplify handle-instance-id (remove read-only check)
RPC layer (app.rpc):
- wrap-audit now activates when :telemetry flag is set
- Add :request-id to RPC params context for event correlation
RPC commands (management, teams_invitations, verify_token, OIDC auth,
webhooks): migrate all audit call sites to use the new plain-key API
SREPL (app.srepl.main):
- Migrate all audit/insert! calls to audit/insert with plain keys
Telemetry task (app.tasks.telemetry):
- Restructure legacy report into make-legacy-request; distinguish
payload type as :telemetry-legacy-report
- Add collect-and-send-audit-events: loop fetching up to 10,000 rows
per iteration, encodes and sends each page, deletes on success,
stops immediately on failure for retry
- Add send-event-batch: POSTs fressian+zstd batch (base64 via
blob/encode-str) to the telemetry endpoint with instance-id per event
- Add gc-telemetry-events: enforces 100,000-row safety cap by dropping
oldest rows first
- Add delete-sent-events: deletes successfully shipped rows by id
Blob utilities (app.util.blob):
- Add encode-str/decode-str: combine fressian+zstd encoding with URL-
safe base64 for JSON-safe string transport
Database:
- Add migration 0145: index on audit_log (source, created_at ASC) for
efficient telemetry batch collection queries
Frontend:
- Always initialize event system regardless of :audit-log flag
- Defer auth events (signin identify) to after profile is set
- Refactor event subsystem for telemetry support
Tests (21 test vars, 94 assertions in tasks-telemetry-test):
- Cover all code paths: disabled/enabled telemetry, no-events no-op,
happy-path batch send and delete, failure retention, payload anonymity,
context stripping, timestamp day precision, batch encoding round-trip,
multi-page iteration, GC cap enforcement, partial failure handling
- blob encode-str/decode-str round-trip tests (14 test vars)
- RPC audit integration tests (5 test vars)
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 📎 Add pr feedback changes
---------
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
The malli schema for the LDAP provider params (`schema:params` in
`backend/src/app/auth/ldap.clj`) declared the bind-password slot as
`:bind-passwor` (missing trailing `d`). The runtime code in the same
file uses `:bind-password` everywhere — `prepare-params` reads
`(:bind-password cfg)` on line 21 and `try-connectivity` reads
`(:bind-password cfg)` on line 89. Effects of the typo:
1. The schema slot for `:bind-password` is missing, so a wrong type
(e.g. a number or vector instead of a string) for the actual key
slips through `check-params` unvalidated. Malli `[:map ...]` is
open by default, so the genuine `:bind-password` key is silently
accepted as an unknown extra key.
2. Anyone reading the schema (operator, future contributor, or
tooling generating docs) sees a non-existent `:bind-passwor`
parameter and could legitimately set that key — schema would
accept it, runtime would never read it, LDAP bind would silently
fail with a confusing "no password" error.
Cross-checked against the pre-malli `clojure.spec` shape removed in
commit 88fb5e7ab (2024-10-29, "♻️ Update integrant to latest
version", which carried the spec→malli migration). The deleted spec
defined `(s/def ::bind-password ::us/string)` correctly — the typo
was introduced when re-typing the keys into the new malli vector-of-
tuples form.
Add a CHANGES.md entry under the 2.17.0 Unreleased 🐛 Bugs fixed
section.
One-character fix.
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
Co-authored-by: Andrey Antukh <niwi@niwi.nz>
Closes#9108.
The `case` expression in `get-info` (`backend/src/app/auth/oidc.clj`)
dispatched on `:token` and `:userinfo` keywords, but the provider map's
`:user-info-source` value is a string — both from config (the malli
schema in `app.config` pins it to one of `"token"`, `"userinfo"`,
`"auto"`) and from the hard-coded Google / GitHub provider maps (which
already write `"userinfo"`). Strings never equal keywords in Clojure
`case`, so every call fell through to the auto-fallback that prefers
ID-token claims and only hits the UserInfo endpoint when claims are
empty. The net effect: setting `PENPOT_OIDC_USER_INFO_SOURCE=userinfo`
did nothing, and OIDC flows whose IdP requires the UserInfo endpoint
(so claims come back empty/partial) failed with "incomplete user info".
- Extract a pure helper `select-user-info-source` that maps the raw
config string to a dispatch keyword (`:token`, `:userinfo`, `:auto`),
falling back to `:auto` for unknown / missing / accidentally-keyword
values
- Rewrite `get-info`'s `case` to dispatch on the helper's output so
the arms unambiguously match the normalised keyword
- Add vitest-style deftests in `auth_oidc_test.clj` pinning the three
valid strings, the nil / "auto" / unknown fallback, and the reverse
regression (a keyword input must not slip through as if it were the
matching string)
- Add a CHANGES.md entry under the 2.17.0 Unreleased `🐛 Bugs fixed`
section linking back to #9108
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
Co-authored-by: Andrey Antukh <niwi@niwi.nz>
* ✨ Add newsletter opt-in checkbox to registration validation form
Add accept-newsletter-updates support through the full registration
token flow. The newsletter checkbox is now available on the
registration validation form, allowing users to opt-in during the
email verification step.
Backend changes:
- Refactor prepare-register to consolidate UTM params and newsletter
preference into props at token creation time
- Add accept-newsletter-updates to prepare-register-profile and
register-profile schemas
- Handle newsletter-updates in register-profile by updating token
claims props on second step
Frontend changes:
- Add newsletter-options component to register-validate-form
- Add accept-newsletter-updates to validation schema
- Fix subscription finalize/error handling in register form
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* ♻️ Refactor auth register components to modern style
Migrate all components in app.main.ui.auth.register and
app.main.ui.auth.login/demo-warning to use the modern * suffix
convention, removing deprecated ::mf/props :obj metadata and
updating all invocations from [:& name] to [:> name*] syntax.
Components updated:
- terms-and-privacy -> terms-and-privacy*
- register-form -> register-form*
- register-methods -> register-methods*
- register-page -> register-page*
- register-success-page -> register-success-page*
- terms-register -> terms-register*
- register-validate-form -> register-validate-form*
- register-validate-page -> register-validate-page*
- demo-warning -> demo-warning*
Also remove unused old context-notification import in login.cljs.
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 🔥 Remove unused onboarding-newsletter component
The newsletter opt-in is now handled directly in the registration
form via the newsletter-options* component, making the standalone
onboarding-newsletter modal obsolete.
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 🐛 Fix register test for UTM params to use prepare-register step
UTM params are now extracted and stored in token props during the
prepare-register step, not at register-profile time. Move utm_campaign
and mtm_campaign from the register-profile call to the
prepare-register-profile call in the test.
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
---------
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
Mainly make it receive the whol cfg/system instead only props. This
makes the api more flexible for a future extending without the need
to change the api again.
This upgrade also includes complete elimination of use spec
from the backend codebase, completing the long running migration
to fully use malli for validation and decoding.
This will happen only if:
- the oidc provider properly reports that user has a non-verified email
- the oidc does not provides any way to know the email verification
status
This commit separates the email cleaning mechanism to a separated
function, and enables a proper cleaning of `mailto:` prefix, usually
found on invitations because users just copy and paste from external
source.
If jwks-uri is provided or properly discovered, they will be used
for unsign JWT token and get use info data from that token instead
of making an additional call to the userinfo endpoint
