* 🐛 Return the full profile from the LDAP invitation branch
The invitation branch responded with only {:invitation-token token},
so the session start received a map without id or auth-backend. With
the new auth-origin vocabulary that map fails validation and the login
breaks in development and tests; in production the event would be
emitted without a channel.
Return the profile plus the token, like the regular branch, and cover
both branches with RPC-level tests.
Closes#12849
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Mint an auth token in the OIDC callback with the invitation inside
The callback used the invitation JWE as the session token, so the
frontend landed directly on token verification and never ran the
regular session start: no signin event, no teams, no host hook.
Always mint an :auth token and carry the invitation in its claims.
Token verification returns the claims verbatim, so login-from-token
attaches the invitation to the profile and logged-in does the rest,
the same pattern the verify-email token already uses.
Closes#12849
AI-assisted-by: deepseek-v4.1-flash
* ✨ Keep the session origin in the telemetry shadow rows
The anonymisation filter drops every string unless it is on a short
allowlist. Add source and activation next to lang and auth-backend so
the shadow copy keeps them; the email and the full name are still
dropped and the email domain is still derived from the email.
The filter keeps its two bindings: the transducer discards everything
that is not a UUID, a boolean or a number, so the allowlist is applied
on the filtered map, never on the raw props.
Closes#12849
AI-assisted-by: deepseek-v4.1-flash
* ✨ Describe the signup outcome with activation, not with action
The three success branches of register-profile labelled a new account
as "login", "email-verification" or "accept-invitation", so the event
contradicted its own name and the value only lived in context, where
the analytics consumers never read it.
Add activation to the event props and drop action from those branches:
the name already says it is a signup, and a constant action would only
add noise. The retry branches keep their action in context because
there it distinguishes four different outcomes.
Closes#12849
AI-assisted-by: deepseek-v4.1-flash
* ✨ Describe the session origin in the signin event
The event carried the profile but never said why the session started:
the only field that tried was frontend metadata that never leaves the
browser, and several flows filled it with the wrong value.
Add a small closed vocabulary (source, auth-backend, activation,
from-invitation) and compose it from the profile when signin is
emitted. The assertion fails in development and tests and is elided in
the release build, so a bad value cannot break a real login.
Closes#12849
AI-assisted-by: deepseek-v4.1-flash
* ✨ Emit the session origin from every login entry point
The four emitters stamped a free-form string as metadata, so the
session event could not rely on it: two flows that were signups wrote
the same value as a login.
Route them through origin-meta: password and LDAP logins declare
source login, register declares source register with activation
immediate, and login-from-token derives both from the token issuer:
:auth for a login and :verify-email for a signup that needed the
email confirmation.
Closes#12849
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Send the invitation signups through the regular session start
Registering from a team invitation link navigated by hand to token
verification, so the session start never ran: no signin event, no
teams, no host hook. The same navigation was copied in four places.
Teach login-from-register to carry an optional invitation token,
extract the profile decoration into with-register-origin, and call it
from the four sites instead of the hand-written navigation.
Closes#12849
AI-assisted-by: deepseek-v4.1-flash
* 🐛 Resolve the invitation without waiting for the teams
Registering from an invitation link now goes through the session
start, and that start only navigates after the teams request
completes. fetch-teams has no error handling, so a network failure
there would leave the invitation unaccepted and the user on the signup
screen with a session already minted.
Move the invitation navigation out of the teams path: it emits as soon
as the profile carries a token, and the teams path emits nothing for
an invited profile so it cannot move the user away from the token
verification screen.
Closes#12849
AI-assisted-by: deepseek-v4.1-flash
* 📚 Document the session origin vocabulary and the invitation flow
Add a focused memory with the four fields, where each value comes from,
the table of flows, the LDAP exception and the invitation path through
logged-in, and link it from the frontend routing.
Update the audit-log memory: source and activation are now on the
telemetry allowlist, activation lives in props and action only exists
on the retry rows.
Closes#12849
AI-assisted-by: deepseek-v4.1-flash
* 📎 Cover the invitation session start and fix the auth memories
The invitation flows and the token decoration in login-from-token were
only covered by helper tests, so removing the guard that keeps the
teams path from navigating, or the token attach, broke the behaviour
silently.
Run the events through ptk/watch with a teams-fetched event as input:
an invited profile navigates to the token verification screen and
never to the dashboard, and the signin keeps from-invitation. Add the
login-from-token cases with and without the token, and fix the memory
statements about action, the flow table and from-invitation.
Closes#12849
AI-assisted-by: deepseek-v4.1-flash
* ♻️ Rename audit-log archivers to consumers
The pending-queue owners are no longer only external archives: a
local, non-archiving drain is about to join them, so "archiver" no
longer fits. Rename the column, identifiers, jobs, metrics and docs
to "consumer" in one release; migration 0157 never reached
production, so a single RENAME COLUMN is enough, with no backfill.
AI-assisted-by: muse-spark-1.3-contributor
* ♻️ Use base Nexus URI with appended ingest path
Mirror the admin-console setup: PENPOT_NEXUS_URI holds the base
URI and the consumer appends api/audit itself, instead of a full
endpoint URI. Keeps the explicit :uri job param as override and the
:task-not-configured guard when no base is set.
AI-assisted-by: muse-spark-1.3-contributor
* ♻️ Archive audit log from a pending queue with two jobs
Replace the mark-and-sweep audit archive with a pending queue
(audit_log_pending, migration 0157): every row is born pending
for the archivers enabled at insert time, in the same
transaction, so a rollback takes both. Two jobs share one
namespace, one queue each: :audit-log-archive-nexus and
:audit-log-archive-nitrate, each with its own closed params
schema, deps and every-5m cron entry gated by its send flag.
Each job drains its queue oldest-first in 128-row chunks (one
transaction per chunk, same total order, no SKIP LOCKED so
arrival stays chronological) and deletes its pendings on
completion; whoever deletes the last pending deletes the row.
Housekeeping lives in both jobs: reap of disabled pendings
plus sweep of orphans, both idempotent. A refused chunk raises
:archive-chunk-refused and aborts the job; the cron retries.
No separate GC task.
Metrics with bounded labels only: archived/failed/skipped per
destination, discarded per reason (reaped pendings, swept rows:
the silent R3 loss, now dashboard-visible). Timing comes from
the jobs substrate; queue depth is read off the replica DB.
No gauge, no custom histogram. Destination hosts log once at
job-def init; per-run logs carry counts plus frozen elapsed
(log props evaluate lazily in the logger thread).
Pre-existing rows are swept unsent (no backfill, accepted
loss); rows born with no archiver on share that fate (R3).
Memory mem:audit-log documents the design.
AI-assisted-by: muse-spark-1.3-contributor
* ♻️ Address review comments on audit log archive jobs
Rename the second archive destination from nitrate to admin-console
(jobs, metric and log labels, error code), reusing the :admin-console
flag. Unify the pending-queue insert into a single helper used by
both producers. Stagger the archive crons (Nexus at second 0, admin
console at second 30). Add per-call dbg logs to the admin-console
send. Update the audit-log memory and the affected tests.
Closes#12372
AI-assisted-by: muse-spark-1.3-contributor
* ✨ Count the application logs as a metric, by logger and level
The error panels had no metric to read: they counted log lines in Loki,
because the backend published nothing about the errors of the application
(the only thing alive in Prometheus was the http listener error count,
which measures another thing). Now there is
`penpot_log_events_total{logger, level}`.
The counter comes from a new component, `app.loggers.metrics/reporter`,
which hooks a watch on the log record like the database and mattermost
reporters do, but without a thread or a channel: the increment is in
memory, so it runs in the watcher itself. Only `warn`, `error` and `fatal`
are counted, and only the records that go through `app.common.logging`:
the lines a library writes straight to log4j2 are not counted (one in a
thousand today).
The body of the watcher is wrapped in a `try/catch` that swallows the
error on purpose: the watcher fires from the `swap!` of the log record,
inside the logging path, so an exception there would reach whoever was
logging and would take the logging executor down with it. A test pins that
a failing publication does not propagate.
The suite removes the reporter (it would count the logs of the tests and
the recording of their `mtx/run!`), so a test of its own pins the wiring.
The memories carry the counter, its two limits and why the `try/catch` is
not noise; the grafana one also brings the notes of this afternoon about
the service account, the dashboard search and Cloudflare, which were
waiting for a commit.
AI-assisted-by: deepseek-v4.1-flash
* 📚 Address the review findings on the log-events metric
The review of the counter found that the reason written for the
`try/catch` was not the real one, so the memory and the comments say now
what happens if it is removed: with `::l/sync?` the exception reaches
whoever was logging, and on the async path it dies in the uncaught
handler of the logging executor -- outside the application's logging --
where the pool replaces the worker, so logging continues and that line is
lost. They also say that a publication failing every time is silent: the
counter stays flat and nothing is written about it.
The rest are small things the review asked for: the test of the `fatal`
level (it was in the counted set and nothing verified it), the HELP of the
metric naming the three levels, the require of the test without the unused
alias that clj-kondo flagged, and the note that the `level` is lowercase
here and uppercase in Loki.
The grafana memory loses what was said twice: the two pairs of bullets
about reading the API (the service account 403s and the search that lies)
are merged, the CLIMIT notes are one bullet instead of two, and the four
bullets about editing dashboards are grouped in their own section. Its
host name is gone as well, which the first line of that file forbids.
AI-assisted-by: deepseek-v4.1-flash
* ✨ Enrich backend audit events with team and entity ids
Backend audit events now carry the data a consumer needs to rebuild
state without joining production tables:
- prepare-rpc-event adds :team-id when the event carries a project or
file id; commands that already loaded the team set it themselves.
- add-profile-to-team! emits add-team-member on every real insert.
- create-project and create-file emit the new row id.
- clone-template, import-binfile and duplicate-project emit one event
per created entity.
- both binfile versions return {:file-ids [...]} from import-files!.
Closes#12219
AI-assisted-by: deepseek-v4.1-flash
* ✨ Emit delete and hard-delete audit events with entity data
The event stream must let Nexus rebuild local state without joining
production tables. This completes the enrichment with the delete paths:
- delete-team keeps returning nil but now carries the deleted row
(id, name, timestamps) in the audit props, like delete-project.
- create-team adds created-at to its audit props.
- delete-profile carries id, email, fullname and deleted-at; its
props used to be empty because every param is namespaced.
- objects-gc emits hard-delete-team/project/file per purged batch,
with the deleted ids, only when the audit log is on.
- delete-team-member needs no change: the delete is physical and the
params already carry team-id and member-id.
The nitrate push-audit-events tests now drop the membership events
emitted by their own profile setup before counting.
Closes#12219
AI-assisted-by: muse-spark-1.3-free
* ✨ Announce cascade deletes and missing ids in audit events
The :delete-object worker task marked teams, projects and files without
leaving any event, so an event-sourced projection kept them alive for
weeks. It now emits one cascade-deleted event per execution with every
affected id, in the same transaction, only with :audit-log on. Rows
purged by objects-gc and snapshot cascades stay silent on purpose.
Also: create-team-with-invitations carries the new team id, and
update-team-member-role names the demoted member on owner transfers.
Closes#12219
AI-assisted-by: muse-spark-1.3-free
* 🐛 Identify the SREPL profile delete in audit events
The SREPL delete-profile! emitted no profile-id and empty props, so the
row could not be attributed by any consumer. It now carries the profile
id in both fields. delete-team! named the wrong cause in its context.
Closes#12219
AI-assisted-by: muse-spark-1.3-free
* 🐛 Identify the SREPL profile restore in audit events
The SREPL restore-profile! emitted no profile-id, so the consumer
keyed on the zero profile instead of the restored one. It now carries
the profile id like a normal profile event, and no consumer special
case is needed.
Closes#12219
AI-assisted-by: muse-spark-1.3-free
* 🐛 Stop attributing SREPL events to the subject profile
profile-id names the actor, and SREPL calls have none: setting the
subject id blamed the user for an admin action and would have leaked
those rows into PostHog as user activity. The subject id stays in
props, where the consumer resolves it, and the event profile stays
zero.
Closes#12219
AI-assisted-by: muse-spark-1.3-free
* ✨ Expose file analysis data for the nexus pipeline
The nexus file-security pipeline needs file content and scheduling
state from the backend. This adds a read-only management method,
`get-file-for-analysis`, returning the semantic projection of one
file (pages, frames, texts, links, interactions; ids as strings;
capped with a truncated flag), plus identity, revn, timestamps and
share state. No prior projection code existed, so the method ships
its own pure `file->projection`.
Audit events now carry what the `:local` consumer schedules from:
revn and modified-at on update-file, name and timestamps on every
create-file emitter, deleted-at on delete-file, and new
create/delete-share-link events with file and link ids.
AI-assisted-by: muse-spark-1.3-contributor
* ♻️ Rename deletion audit events to action names
Event names are actions, not past tense: `cascade-deleted` becomes
`delete-object-with-cascade` with the same shape and transaction.
The three `hard-delete-*` events merge into `hard-delete-objects`:
one event per purged chunk carrying only that chunk's entity key
(`:team-ids`, `:project-ids` or `:file-ids`), because each GC
section purges in its own chunk transactions and a single event
could never share them all. Every submit stays inside its delete
transaction, `triggered-by` stays annotated, and silent paths stay
silent.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Read worker parallelism knobs with plain config keys
The runner wiring looked up `:app.main/worker-*-parallelism`
(namespaced) while the schema and the environment provide plain
keys, so `PENPOT_WORKER_DEFAULT_PARALLELISM` and
`PENPOT_WORKER_WEBHOOK_PARALLELISM` never applied and workers
always ran with the fallback. Read the keys the config holds.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Fix binfile audit test with a real file row
`submit-file-created-event` reads the light columns back from the
row, so the mocked importer must return an existing file id: a
phantom id emits no event and the test saw zero `create-file`
events. Create the file first and use its id. The audit-log memory
now tells both bulk-event helpers apart.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Fix audit event fidelity from review findings
The review of the audit enrichment found data-fidelity gaps; fix
them while the branch is fresh.
Membership events now belong to the actor: the flows where the
owner adds somebody else pass an explicit actor id, the role is
derived from the flags (the table has no role column), and the
event carries the request context like its siblings.
Also: team-id resolution skips read events, duplicate-project
emits the project name and timestamps, delete-profile is
idempotent again, the GC tests assert physical purge, and
get-file-for-analysis leaves an audit trace.
AI-assisted-by: muse-spark-1.3-contributor
* ♻️ Nest cascade ids under :cascade in delete event
The cascade event carried every affected id flat next to the
object identity; nest them under `:cascade` so the payload reads
as header plus affected set. The event is an action like its
siblings, not a command. Drop the `:audit-log` flag re-check:
`submit` already fans out per flag.
AI-assisted-by: muse-spark-1.3-contributor
* 📎 Update copyright name on file header
The company name changed; a single header in this branch still
carried the old one. Align it with the current form.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Fix nil event-id on webhook trigger audit rows
process-event generated the row id inside append-audit-entry, so the
:process-webhook-event task never received it and the trigger row
stored a nil :event-id. Assign the id up-front and share it between
the stored row and the worker params; the telemetry shadow row keeps
its own id to avoid a PK collision. Add regression tests covering
the handoff and the end-to-end trigger row.
AI-assisted-by: muse-spark-1.3-contributor
After the unified jobs migration, cron submitted empty params that
failed validation because the job schema still required Integrant
system keys. Narrow the schema to optional REPL overrides, wire
:app.nitrate/client onto the archive job-def, and drop the unused
::jobs/defs ref from nitrate that created an Integrant cycle.
AI-assisted-by: composer
* 🎉 Move backend jobs to the unified job table substrate
Replace the generic 'task' table with a unified 'job' table as the
durable substrate for all backend jobs, and migrate every producer
and consumer to it. The legacy 'task' table stops receiving writes
but stays dormant (still cleaned by tasks-gc); upload-session
chunking and the other post-branch upstream changes are preserved.
New 0154 migration creates the 'job' table: dispatch and lifecycle
columns (name, queue, priority, scheduled_at, retries, status,
timestamps, props) plus optional user-facing ledger columns
(profile_id, target, progress, error, result, resource_id,
expires_at). No modified_at trigger: the application updates it on
every write for lease and orphan detection. Partial indexes cover
the dispatcher claim query, orphan scanning, and per-profile
listing.
New app.jobs public API: submit!/invoke!, heartbeats and progress
reporting, a precompiled job-def registry (name, schema, handler,
decoder, validator), an ephemeral request/response mode, and a
management API for external workers. The dispatcher claims
new/retry rows (FOR UPDATE SKIP LOCKED) with a plain JSON redis
payload and marks lease-expired running rows as orphaned; the
runner executes per-queue with the same retry conventions.
New jobs GC (daily): deletes expired rows and retains internal
terminal rows per :jobs-retention, touching referenced storage
objects so storage-gc-touched reclaims them. Job resources live in
a dedicated job-resource bucket with a reclaim branch in
storage-gc-touched.
All production call-sites (sendmail, delete-object, demo-purge,
file-gc, offload, webhooks, quotes, nitrate bulk delete,
snapshots, telemetry) and the ten cron tasks now submit through
app.jobs; cron is a pure scheduler and the legacy worker registry,
task-table inserts, and per-namespace handlers are removed.
Includes migrations, dispatcher, runner, jobs, request/response,
jobs GC, management API, and cron test suites.
AI-assisted-by: glm-5.3-flash
AI-assisted-by: muse-spark-1.3-contributor
AI-assisted-by: kimi-k3
AI-assisted-by: mimo-v2.5
* ✨ Address review comments on jobs substrate
Resolves the four open review threads on the jobs PR. Orphaned jobs
are now marked aborted, a system-side terminal state with no retry:
the dispatcher sweep stores the end event with outcome aborted in a
single bulk insert in the same transaction, records the terminal
outcome next to the orphaned counter, and reports an error log;
orphans keep alerting through the log and the counter, with no
user notification.
The aborted state is terminal everywhere: the status CHECK, the
sweep, the lifecycle writers, the jobs GC retention, the backlog
gauges and the user status mapping, with tests covering each
transition. The claim comment, the 2.20 doc version and the legacy
task table notes are corrected, and objects-gc heartbeats per chunk.
AI-assisted-by: muse-spark-1.3-contributor
* ✨ Gate error reporters behind :error-reporting flag
Error reporters used to start on every boot with no way to
switch them off. They now only start when the new
:error-reporting flag is on (off by default, enable with
PENPOT_FLAGS=enable-error-reporting). The database reporter
needs the flag alone; mattermost needs the flag plus its
webhook url. Shutdown with the flag off is safe, and the
database loop now honors its runtime switch.
AI-assisted-by: muse-spark-1.3-contributor
* 📚 Add audit-log reference memory and update dev tooling
Adds the top-level audit-log reference memory with all event
flows and turns backend/audit-log into a redirect stub.
Also includes the staged dev-tooling updates (playwright
deps, mdts args, gitignore).
AI-assisted-by: muse-spark-1.3-contributor
* ✨ Assign server-side initiator to frontend audit events
Frontend audit events now carry a server-assigned initiator
derived from the x-client header (penpot-frontend goes to
app, penpot-admin-console goes to admin-console, legacy
penpot-nitrate stays on admin-console, all else goes to
app). Client-sent values are always overwritten, and the
key survives on telemetry shadow rows.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Keep initiator on frontend telemetry context
The strips-pii-keys test still asserted the old contract
(initiator stripped). The initiator only names the sending
app, so it is not personal data and must survive the
telemetry filter like on the backend channel.
AI-assisted-by: muse-spark-1.3-contributor
* 📚 Document initiator on PostHog and browser origins
Complete the audit-log memory for the latest changes: the
initiator property forwarded to PostHog on both channels
and the eventOrigin rule for admin-console browser events.
AI-assisted-by: muse-spark-1.3-contributor
* 🐛 Fix unscaped argument in script
---------
Co-authored-by: alonso.torres <alonso.torres@kaleidos.net>
Run the audit archive cron when :nexus or :admin-console is on. Ship
allowlisted events to Admin Console first (with row ids for
idempotency), then the full chunk to Nexus when :nexus is set. Mark
archived_at for the whole chunk on success, including nitrate-only
mode. Rename the gate flag from :audit-log-archive to :nexus.
AI-assisted-by: Composer
Co-authored-by: Cursor <cursoragent@cursor.com>
* 🐛 Attribute audit events to the caller, not the response
prepare-rpc-event took the event profile-id from the result map
whenever it carried one, before falling back to the caller. Any
command returning a response with a :profile-id key silently
credited the action to somebody else.
get-error-report returns the report with its decoded content
merged in, and that content holds the profile that owned the
report, so privileged reads were logged against the users whose
crashes were being inspected. verify-token on a team invitation
returns the inviter's profile-id, so accepting an invitation was
logged against the inviter.
Resolution is now ::audit/profile-id metadata, then
::rpc/profile-id, then the zero uuid; the response is never
consulted. The two verify-token branches that relied on it now
declare the profile in the result metadata. Every other command
either already declared it or returns no :profile-id; all 30
registered command namespaces were checked.
The tests that pinned the old behavior are replaced by ones
covering the new contract.
AI-assisted-by: space-bunny-free
* 🐛 Coerce the audit profile-id override to a uuid
The only sanctioned way for a command to override the profile of an
audit event is the ::audit/profile-id metadata, and the value is set
by hand in a dozen commands, some of them reading it from token
claims or other sources we do not type.
schema:event requires a uuid and submit* swallows the validation
error, so a string did not fail loudly: the row was dropped silently.
Values that cannot become a uuid are now discarded with a warning
and the event falls back to the caller, which is always a valid uuid.
A uuid, the common case, exits on the first check.
AI-assisted-by: space-bunny-free
Shared-key callers (exporter, admin-console) arrive as keywords on
auth-key-id, so transit persisted them as ~:exporter while regular traffic
stored app. Coerce with d/name at the single origin so every audit and
telemetry copy carries a plain string. Adds regression tests for the origin
and the push-audit-events path, including caller spoofing precedence.
Closes#11628
AI-assisted-by: muse-spark-1.3-contributor
Add escape-markdown to common/data.cljc that escapes Markdown
special characters (*, _, ~, `, [, ], >, #, @, etc.) by prefixing
them with backslash. Apply it to user-controlled fields (:hint,
:href) in the Mattermost error reporter before constructing the
notification message.
This is an internal-only feature not accessible to end users.
AI-assisted-by: mimo-v2.5-pro
The audit event validation was failing when processing error reports that
contain string profile-id values. The error report storage converts
profile-id to string format, but the audit schema expects a UUID.
Changes:
- Modified prepare-rpc-event to convert string profile-id to UUID using
uuid/parse* (exception-safe parsing)
- Updated access token middleware to set ::id and ::type on request so
audit context includes token identification
- Added tests for profile-id conversion and token context population
Closes#10897
AI-assisted-by: qwen3.7-plus
Implement RPC methods for querying server error reports with pagination
and filtering. Add CLI tool (tools/error-reports.mjs) for convenient
access with table and JSON output formats. Extract profile-id from audit
events and logging context for better error categorization. Build
improved HREF using request path when available.
AI-assisted-by: qwen3.7-plus
* 🎉 Add telemetry anonymous event collection
Rewrite the audit logging subsystem to support three operating modes and
add anonymous telemetry event collection:
Modes:
- A (audit-log only): events persisted with full context
- B (audit-log + telemetry): same as A, plus events are collected for
telemetry shipping
- C (telemetry-only): events stored anonymously with PII stripped,
telemetry flag active, audit-log flag inactive
Audit system refactoring (app.loggers.audit):
- Replace qualified map keys (::audit/name etc.) with plain keywords
- Rename submit! -> submit, insert! -> insert, prepare-event ->
prepare-rpc-event
- Add submit* as a lower-level public API
- Add process-event dispatch function that handles all three modes and
webhooks in a single tx-run!
- Add :id to event schema (auto-generated if omitted)
- Add filter-telemetry-props: anonymises event props per event type.
Keeps UUID/boolean/number values; for login/identify events preserves
lang, auth-backend, email-domain; for navigate events preserves route,
file-id, team-id, page-id; instance-start trigger passes through.
- Add filter-telemetry-context: retains only safe context keys.
Backend: version, initiator, client-version, client-user-agent.
Frontend: browser, os, locale, screen metrics, event-origin.
- Timestamps truncated to day precision via ct/truncate for telemetry
storage
- PII stripped: props emptied, ip-addr zeroed, session-linking and
access-token fields removed from context
Config (app.config):
- Derive :enable-telemetry flag from telemetry-enabled config option
Email utilities (app.email):
- Add email/clean and email/get-domain helper functions for domain
extraction from email addresses
Setup (app.setup):
- Emit instance-start trigger event at system startup
- Simplify handle-instance-id (remove read-only check)
RPC layer (app.rpc):
- wrap-audit now activates when :telemetry flag is set
- Add :request-id to RPC params context for event correlation
RPC commands (management, teams_invitations, verify_token, OIDC auth,
webhooks): migrate all audit call sites to use the new plain-key API
SREPL (app.srepl.main):
- Migrate all audit/insert! calls to audit/insert with plain keys
Telemetry task (app.tasks.telemetry):
- Restructure legacy report into make-legacy-request; distinguish
payload type as :telemetry-legacy-report
- Add collect-and-send-audit-events: loop fetching up to 10,000 rows
per iteration, encodes and sends each page, deletes on success,
stops immediately on failure for retry
- Add send-event-batch: POSTs fressian+zstd batch (base64 via
blob/encode-str) to the telemetry endpoint with instance-id per event
- Add gc-telemetry-events: enforces 100,000-row safety cap by dropping
oldest rows first
- Add delete-sent-events: deletes successfully shipped rows by id
Blob utilities (app.util.blob):
- Add encode-str/decode-str: combine fressian+zstd encoding with URL-
safe base64 for JSON-safe string transport
Database:
- Add migration 0145: index on audit_log (source, created_at ASC) for
efficient telemetry batch collection queries
Frontend:
- Always initialize event system regardless of :audit-log flag
- Defer auth events (signin identify) to after profile is set
- Refactor event subsystem for telemetry support
Tests (21 test vars, 94 assertions in tasks-telemetry-test):
- Cover all code paths: disabled/enabled telemetry, no-events no-op,
happy-path batch send and delete, failure retention, payload anonymity,
context stripping, timestamp day precision, batch encoding round-trip,
multi-page iteration, GC cap enforcement, partial failure handling
- blob encode-str/decode-str round-trip tests (14 test vars)
- RPC audit integration tests (5 test vars)
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 📎 Add pr feedback changes
---------
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 🎉 Add telemetry anonymous event collection
Rewrite the audit logging subsystem to support three operating modes and
add anonymous telemetry event collection:
Modes:
- A (audit-log only): events persisted with full context
- B (audit-log + telemetry): same as A, plus events are collected for
telemetry shipping
- C (telemetry-only): events stored anonymously with PII stripped,
telemetry flag active, audit-log flag inactive
Audit system refactoring (app.loggers.audit):
- Replace qualified map keys (::audit/name etc.) with plain keywords
- Rename submit! -> submit, insert! -> insert, prepare-event ->
prepare-rpc-event
- Add submit* as a lower-level public API
- Add process-event dispatch function that handles all three modes and
webhooks in a single tx-run!
- Add :id to event schema (auto-generated if omitted)
- Add filter-telemetry-props: anonymises event props per event type.
Keeps UUID/boolean/number values; for login/identify events preserves
lang, auth-backend, email-domain; for navigate events preserves route,
file-id, team-id, page-id; instance-start trigger passes through.
- Add filter-telemetry-context: retains only safe context keys.
Backend: version, initiator, client-version, client-user-agent.
Frontend: browser, os, locale, screen metrics, event-origin.
- Timestamps truncated to day precision via ct/truncate for telemetry
storage
- PII stripped: props emptied, ip-addr zeroed, session-linking and
access-token fields removed from context
Config (app.config):
- Derive :enable-telemetry flag from telemetry-enabled config option
Email utilities (app.email):
- Add email/clean and email/get-domain helper functions for domain
extraction from email addresses
Setup (app.setup):
- Emit instance-start trigger event at system startup
- Simplify handle-instance-id (remove read-only check)
RPC layer (app.rpc):
- wrap-audit now activates when :telemetry flag is set
- Add :request-id to RPC params context for event correlation
RPC commands (management, teams_invitations, verify_token, OIDC auth,
webhooks): migrate all audit call sites to use the new plain-key API
SREPL (app.srepl.main):
- Migrate all audit/insert! calls to audit/insert with plain keys
Telemetry task (app.tasks.telemetry):
- Restructure legacy report into make-legacy-request; distinguish
payload type as :telemetry-legacy-report
- Add collect-and-send-audit-events: loop fetching up to 10,000 rows
per iteration, encodes and sends each page, deletes on success,
stops immediately on failure for retry
- Add send-event-batch: POSTs fressian+zstd batch (base64 via
blob/encode-str) to the telemetry endpoint with instance-id per event
- Add gc-telemetry-events: enforces 100,000-row safety cap by dropping
oldest rows first
- Add delete-sent-events: deletes successfully shipped rows by id
Blob utilities (app.util.blob):
- Add encode-str/decode-str: combine fressian+zstd encoding with URL-
safe base64 for JSON-safe string transport
Database:
- Add migration 0145: index on audit_log (source, created_at ASC) for
efficient telemetry batch collection queries
Frontend:
- Always initialize event system regardless of :audit-log flag
- Defer auth events (signin identify) to after profile is set
- Refactor event subsystem for telemetry support
Tests (21 test vars, 94 assertions in tasks-telemetry-test):
- Cover all code paths: disabled/enabled telemetry, no-events no-op,
happy-path batch send and delete, failure retention, payload anonymity,
context stripping, timestamp day precision, batch encoding round-trip,
multi-page iteration, GC cap enforcement, partial failure handling
- blob encode-str/decode-str round-trip tests (14 test vars)
- RPC audit integration tests (5 test vars)
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 📎 Add pr feedback changes
---------
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
