* 🎉 Add telemetry anonymous event collection
Rewrite the audit logging subsystem to support three operating modes and
add anonymous telemetry event collection:
Modes:
- A (audit-log only): events persisted with full context
- B (audit-log + telemetry): same as A, plus events are collected for
telemetry shipping
- C (telemetry-only): events stored anonymously with PII stripped,
telemetry flag active, audit-log flag inactive
Audit system refactoring (app.loggers.audit):
- Replace qualified map keys (::audit/name etc.) with plain keywords
- Rename submit! -> submit, insert! -> insert, prepare-event ->
prepare-rpc-event
- Add submit* as a lower-level public API
- Add process-event dispatch function that handles all three modes and
webhooks in a single tx-run!
- Add :id to event schema (auto-generated if omitted)
- Add filter-telemetry-props: anonymises event props per event type.
Keeps UUID/boolean/number values; for login/identify events preserves
lang, auth-backend, email-domain; for navigate events preserves route,
file-id, team-id, page-id; instance-start trigger passes through.
- Add filter-telemetry-context: retains only safe context keys.
Backend: version, initiator, client-version, client-user-agent.
Frontend: browser, os, locale, screen metrics, event-origin.
- Timestamps truncated to day precision via ct/truncate for telemetry
storage
- PII stripped: props emptied, ip-addr zeroed, session-linking and
access-token fields removed from context
Config (app.config):
- Derive :enable-telemetry flag from telemetry-enabled config option
Email utilities (app.email):
- Add email/clean and email/get-domain helper functions for domain
extraction from email addresses
Setup (app.setup):
- Emit instance-start trigger event at system startup
- Simplify handle-instance-id (remove read-only check)
RPC layer (app.rpc):
- wrap-audit now activates when :telemetry flag is set
- Add :request-id to RPC params context for event correlation
RPC commands (management, teams_invitations, verify_token, OIDC auth,
webhooks): migrate all audit call sites to use the new plain-key API
SREPL (app.srepl.main):
- Migrate all audit/insert! calls to audit/insert with plain keys
Telemetry task (app.tasks.telemetry):
- Restructure legacy report into make-legacy-request; distinguish
payload type as :telemetry-legacy-report
- Add collect-and-send-audit-events: loop fetching up to 10,000 rows
per iteration, encodes and sends each page, deletes on success,
stops immediately on failure for retry
- Add send-event-batch: POSTs fressian+zstd batch (base64 via
blob/encode-str) to the telemetry endpoint with instance-id per event
- Add gc-telemetry-events: enforces 100,000-row safety cap by dropping
oldest rows first
- Add delete-sent-events: deletes successfully shipped rows by id
Blob utilities (app.util.blob):
- Add encode-str/decode-str: combine fressian+zstd encoding with URL-
safe base64 for JSON-safe string transport
Database:
- Add migration 0145: index on audit_log (source, created_at ASC) for
efficient telemetry batch collection queries
Frontend:
- Always initialize event system regardless of :audit-log flag
- Defer auth events (signin identify) to after profile is set
- Refactor event subsystem for telemetry support
Tests (21 test vars, 94 assertions in tasks-telemetry-test):
- Cover all code paths: disabled/enabled telemetry, no-events no-op,
happy-path batch send and delete, failure retention, payload anonymity,
context stripping, timestamp day precision, batch encoding round-trip,
multi-page iteration, GC cap enforcement, partial failure handling
- blob encode-str/decode-str round-trip tests (14 test vars)
- RPC audit integration tests (5 test vars)
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 📎 Add pr feedback changes
---------
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 🎉 Add telemetry anonymous event collection
Rewrite the audit logging subsystem to support three operating modes and
add anonymous telemetry event collection:
Modes:
- A (audit-log only): events persisted with full context
- B (audit-log + telemetry): same as A, plus events are collected for
telemetry shipping
- C (telemetry-only): events stored anonymously with PII stripped,
telemetry flag active, audit-log flag inactive
Audit system refactoring (app.loggers.audit):
- Replace qualified map keys (::audit/name etc.) with plain keywords
- Rename submit! -> submit, insert! -> insert, prepare-event ->
prepare-rpc-event
- Add submit* as a lower-level public API
- Add process-event dispatch function that handles all three modes and
webhooks in a single tx-run!
- Add :id to event schema (auto-generated if omitted)
- Add filter-telemetry-props: anonymises event props per event type.
Keeps UUID/boolean/number values; for login/identify events preserves
lang, auth-backend, email-domain; for navigate events preserves route,
file-id, team-id, page-id; instance-start trigger passes through.
- Add filter-telemetry-context: retains only safe context keys.
Backend: version, initiator, client-version, client-user-agent.
Frontend: browser, os, locale, screen metrics, event-origin.
- Timestamps truncated to day precision via ct/truncate for telemetry
storage
- PII stripped: props emptied, ip-addr zeroed, session-linking and
access-token fields removed from context
Config (app.config):
- Derive :enable-telemetry flag from telemetry-enabled config option
Email utilities (app.email):
- Add email/clean and email/get-domain helper functions for domain
extraction from email addresses
Setup (app.setup):
- Emit instance-start trigger event at system startup
- Simplify handle-instance-id (remove read-only check)
RPC layer (app.rpc):
- wrap-audit now activates when :telemetry flag is set
- Add :request-id to RPC params context for event correlation
RPC commands (management, teams_invitations, verify_token, OIDC auth,
webhooks): migrate all audit call sites to use the new plain-key API
SREPL (app.srepl.main):
- Migrate all audit/insert! calls to audit/insert with plain keys
Telemetry task (app.tasks.telemetry):
- Restructure legacy report into make-legacy-request; distinguish
payload type as :telemetry-legacy-report
- Add collect-and-send-audit-events: loop fetching up to 10,000 rows
per iteration, encodes and sends each page, deletes on success,
stops immediately on failure for retry
- Add send-event-batch: POSTs fressian+zstd batch (base64 via
blob/encode-str) to the telemetry endpoint with instance-id per event
- Add gc-telemetry-events: enforces 100,000-row safety cap by dropping
oldest rows first
- Add delete-sent-events: deletes successfully shipped rows by id
Blob utilities (app.util.blob):
- Add encode-str/decode-str: combine fressian+zstd encoding with URL-
safe base64 for JSON-safe string transport
Database:
- Add migration 0145: index on audit_log (source, created_at ASC) for
efficient telemetry batch collection queries
Frontend:
- Always initialize event system regardless of :audit-log flag
- Defer auth events (signin identify) to after profile is set
- Refactor event subsystem for telemetry support
Tests (21 test vars, 94 assertions in tasks-telemetry-test):
- Cover all code paths: disabled/enabled telemetry, no-events no-op,
happy-path batch send and delete, failure retention, payload anonymity,
context stripping, timestamp day precision, batch encoding round-trip,
multi-page iteration, GC cap enforcement, partial failure handling
- blob encode-str/decode-str round-trip tests (14 test vars)
- RPC audit integration tests (5 test vars)
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 📎 Add pr feedback changes
---------
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
The recursive `read-items` function in `app.util.sse/read-stream`
caused a synchronous stack overflow when reading buffered stream
data. Each `rx/mapcat` call chained another recursive invocation
on the same call stack without yielding to the event loop.
Replace the recursive pattern with an `rx/create`-based async pump
that uses Promise `.then()` chaining, keeping the call stack depth
constant regardless of stream size.
Also add progress reporting with names and IDs during binfile
export and import, and bump `eventsource-parser` dependency.
Closes#9470
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
The recursive `read-items` function in `app.util.sse/read-stream`
caused a synchronous stack overflow when reading buffered stream
data. Each `rx/mapcat` call chained another recursive invocation
on the same call stack without yielding to the event loop.
Replace the recursive pattern with an `rx/create`-based async pump
that uses Promise `.then()` chaining, keeping the call stack depth
constant regardless of stream size.
Also add progress reporting with names and IDs during binfile
export and import, and bump `eventsource-parser` dependency.
Closes#9470
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 🐛 Fix incorrect invitation token handling on register process
- Reject prepare-register-profile when an active profile already
exists for the requested email.
- Stop embedding an existing profile's :profile-id into the
prepared-register JWE. Profile resolution in register-profile is
now done exclusively by email lookup, never by a JWE claim.
- Add created? guard to the invitation-success branch in
register-profile, so existing profiles (active or not) cannot
reach session creation via anonymous registration.
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* ♻️ Restructure invitation handling inside register-profile
Move the invitation-success branch into the created? sub-cond so it
sits alongside the other post-creation branches, making the control
flow consistent.
- Active new profile + matching invitation: mint session and return
:invitation-token (frontend redirects to :auth-verify-token).
- Not-yet-active new profile + matching invitation: embed the
invitation token inside the verify-email JWE and send the
verification email. When the user clicks the link, they get
logged in and the frontend completes the team-invitation flow.
- Extend send-email-verification! with an optional invitation-token
parameter propagated into the verify-email JWE claims.
- Update the frontend verify-email handler to navigate to
:auth-verify-token when the response carries :invitation-token.
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 🐛 Handle email-already-exists error on registration form
Add a specific handler for the [:validation :email-already-exists] error
code in the registration form's on-error callback. The backend raises
this error when an active profile already exists for the requested email,
but the frontend was falling through to the generic error message.
Now it shows the existing "Email already used" i18n message instead of
the generic "Something wrong has happened" toast.
* 🐛 Reset submitted state on registration form error
The on-error handler in the registration form was not resetting the
submitted? state, causing the submit button to remain disabled after
any error. The completion callback in rx/subs! only fires on success,
not on error.
Add (reset! submitted? false) at the beginning of the on-error handler
so the form becomes submittable again after any error, allowing the user
to fix their input and retry.
---------
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* ✨ Show specific error messages for invitation token failures
Surface distinct error messages for the three invitation-token failure
modes that the backend already distinguishes: email mismatch, expired
token, and invalid/corrupted token. Replaces the single generic
could not accept invitation message with actionable text so the
user knows what went wrong and how to recover.
Signed-off-by: jack-stormentswe <crazycoder131@gmail.com>
* 💄 Update CHANGE.md
Signed-off-by: jack-stormentswe <crazycoder131@gmail.com>
* 💄 Address review feedback on invitation-error messages
Signed-off-by: jack-stormentswe <crazycoder131@gmail.com>
---------
Signed-off-by: jack-stormentswe <crazycoder131@gmail.com>
Co-authored-by: Andrey Antukh <niwi@niwi.nz>
The malli schema for the LDAP provider params (`schema:params` in
`backend/src/app/auth/ldap.clj`) declared the bind-password slot as
`:bind-passwor` (missing trailing `d`). The runtime code in the same
file uses `:bind-password` everywhere — `prepare-params` reads
`(:bind-password cfg)` on line 21 and `try-connectivity` reads
`(:bind-password cfg)` on line 89. Effects of the typo:
1. The schema slot for `:bind-password` is missing, so a wrong type
(e.g. a number or vector instead of a string) for the actual key
slips through `check-params` unvalidated. Malli `[:map ...]` is
open by default, so the genuine `:bind-password` key is silently
accepted as an unknown extra key.
2. Anyone reading the schema (operator, future contributor, or
tooling generating docs) sees a non-existent `:bind-passwor`
parameter and could legitimately set that key — schema would
accept it, runtime would never read it, LDAP bind would silently
fail with a confusing "no password" error.
Cross-checked against the pre-malli `clojure.spec` shape removed in
commit 88fb5e7ab (2024-10-29, "♻️ Update integrant to latest
version", which carried the spec→malli migration). The deleted spec
defined `(s/def ::bind-password ::us/string)` correctly — the typo
was introduced when re-typing the keys into the new malli vector-of-
tuples form.
Add a CHANGES.md entry under the 2.17.0 Unreleased 🐛 Bugs fixed
section.
One-character fix.
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
Co-authored-by: Andrey Antukh <niwi@niwi.nz>
login-with-ldap raised a :restriction exception with the message
"ldap auth provider is not initialized" stored under :hide instead
of :hint. ex/raise (common/src/app/common/exceptions.cljc:33-34)
uses :hint as the ExceptionInfo message and the downstream error
formatters only read :hint (line 250, 312) — :hide is unread
anywhere in the codebase (0 other occurrences vs 447 for :hint).
Effect: when LDAP is misconfigured, operators saw the generic
"restriction" error message instead of the diagnostic string. The
typo has been present since the LDAP command was first introduced
by commit 14d1cb90bd (2022-06-30, "Refactor auth code") and was
carried forward through 6cdf696fc (2023-01-05, "Fix issues on ldap
provider and rpc method") without ever surfacing as a code-review
comment.
One-character fix: :hide -> :hint. Add a CHANGES.md entry under
the 2.17.0 Unreleased 🐛 Bugs fixed section.
Closes#9108.
The `case` expression in `get-info` (`backend/src/app/auth/oidc.clj`)
dispatched on `:token` and `:userinfo` keywords, but the provider map's
`:user-info-source` value is a string — both from config (the malli
schema in `app.config` pins it to one of `"token"`, `"userinfo"`,
`"auto"`) and from the hard-coded Google / GitHub provider maps (which
already write `"userinfo"`). Strings never equal keywords in Clojure
`case`, so every call fell through to the auto-fallback that prefers
ID-token claims and only hits the UserInfo endpoint when claims are
empty. The net effect: setting `PENPOT_OIDC_USER_INFO_SOURCE=userinfo`
did nothing, and OIDC flows whose IdP requires the UserInfo endpoint
(so claims come back empty/partial) failed with "incomplete user info".
- Extract a pure helper `select-user-info-source` that maps the raw
config string to a dispatch keyword (`:token`, `:userinfo`, `:auto`),
falling back to `:auto` for unknown / missing / accidentally-keyword
values
- Rewrite `get-info`'s `case` to dispatch on the helper's output so
the arms unambiguously match the normalised keyword
- Add vitest-style deftests in `auth_oidc_test.clj` pinning the three
valid strings, the nil / "auto" / unknown fallback, and the reverse
regression (a keyword input must not slip through as if it were the
matching string)
- Add a CHANGES.md entry under the 2.17.0 Unreleased `🐛 Bugs fixed`
section linking back to #9108
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
Co-authored-by: Andrey Antukh <niwi@niwi.nz>
The viewer-side `obfuscate-email` helper used by `anonymize-member` when
building share-link bundles called `clojure.string/split` on the raw
email input and then on the extracted domain. Two failure modes:
1. When the stored email had no `@` (legacy data, LDAP-sourced UIDs, direct
DB inserts, or fixtures that bypassed `::sm/email`), destructuring
left `domain` bound to `nil` and the follow-up `(str/split nil "." 2)`
raised `NullPointerException`. Because `obfuscate-email` runs inside
`get-view-only-bundle`, the exception aborted the whole RPC response
for share-link viewers, not just the field.
2. When the stored email used a single-label domain (`alice@localhost`),
`(str/split "localhost" "." 2)` returned `["localhost"]`; destructuring
bound `rest` to `nil` and the final `(str name "@****." rest)` produced
a dangling-dot output `"****@****."` (nil coerces to empty in `str`).
Guard both split calls with `(or x "")` so the chain is nil-safe, and
emit the trailing `.<tld>` segment only when `rest` is present. Add three
`deftest` groups covering the happy path, dotless domains, and malformed
inputs (nil / empty / no-`@`), plus a CHANGES.md entry under the 2.17.0
Unreleased bugs-fixed section.
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
Co-authored-by: Andrey Antukh <niwi@niwi.nz>
