mirror of
https://github.com/penpot/penpot.git
synced 2026-11-21 09:42:01 +00:00
Merge remote-tracking branch 'origin/staging' into develop
This commit is contained in:
commit
adb61a9b96
@ -1,342 +1,134 @@
|
||||
# Audit Log and event processing
|
||||
|
||||
Complete guide to the Penpot event system: what gets recorded, where each piece lives, and how events travel from the moment they happen until they are archived. Start with the overview (sections 1-3); the deeper technical detail comes later (section 8 on). If you only want the idea, the first three sections are enough.
|
||||
The audit log records what Penpot observed: who made a request, what they asked to do, and when it happened. It also carries browser events for usage analytics and crash reporting. Those events are useful, but a browser can close or lose its connection before sending them. Do not use their absence as proof that something did not happen.
|
||||
|
||||
> Note about this memory: unlike other Serena memories (dense notes written for LLM agents, see `mem:memory-maintenance`), this document is also written for human developers who need full context. That is why it uses plain language and examples. It replaces the old `mem:backend/audit-log`, which is now just a redirect stub.
|
||||
## 1. Sources and trust
|
||||
|
||||
## 1. What it is (2-minute overview)
|
||||
| Channel | Who sets the actor | What the event tells you |
|
||||
|---|---|---|
|
||||
| Penpot backend RPC | The server uses the caller, an explicit command override, or `uuid/zero` when no actor is known. | The server ran the command. Check the event's outcome before treating it as a change of state. |
|
||||
| Public `push-audit-events` | Penpot uses the session, never an actor supplied in the event. | The Penpot or admin-console browser reported an action. Delivery is best-effort. |
|
||||
| Management `push-audit-events` | The authenticated admin-console service supplies the actor. | A trusted service reported an action. Authentication identifies the service, not whether its business action and the event committed together. |
|
||||
|
||||
The *audit log* answers "who did what, when, and from where". Every important user action becomes an **event**: a small object with a name, a type, who did it, from which IP, plus action data (`props`) and metadata (`context`).
|
||||
The management channel uses `POST /api/management/methods/push-audit-events` with `x-shared-key: "admin-console <key>"` and events carrying `{name, profileId, props?, context?}` in JSON. It uses server request time, not the public endpoint's client timestamp. The public channel uses `POST /api/main/methods/push-audit-events` with a user session and events carrying `{name, type, timestamp, props, context?}`. Both browsers use the same public RPC. The Nitrate senders live in the private Nitrate repo; these are the contracts Penpot receives.
|
||||
|
||||
The system has:
|
||||
|
||||
- **Three producers** (they create events): the Penpot backend (RPC layer), the Penpot frontend (browser app), and the Nitrate admin-console, which sends both server-side and client-side events.
|
||||
- **One central buffer**: the Postgres `audit_log` table in Penpot. It is temporary: it holds events for minutes/hours until they are archived, then they are deleted.
|
||||
- **Four consumers** (they read events): webhooks, error reporters, telemetry, and the external archives in Nexus and in the admin console.
|
||||
- **Two long-term archives**: **Nexus**, a separate service that stores events for the long term, and the **admin console** (Nitrate), which ingests its own allowlisted events through a second archive job (see section 12).
|
||||
|
||||
```
|
||||
Penpot frontend ─ POST push-audit-events (public) ─────────▶┐
|
||||
Nitrate admin-console (browser) ─ POST push-audit-events ──▶│
|
||||
Nitrate admin-console (server) ─ POST push-audit-events ──▶│ ─▶ audit_log table (buffer) ──┬─ cron 0s every 5m ──▶ Nexus (archive)
|
||||
Penpot backend (RPC wrap-audit) ─ submit ───────────────────┘ │ └─ cron 30s every 5m ─▶ admin console (archive)
|
||||
│ ├──▶ webhooks
|
||||
│ ├──▶ error reporters
|
||||
└──▶ telemetry (shadow rows)
|
||||
```
|
||||
|
||||
The Penpot backend is the **center**: it checks, completes, and stores every event, no matter where it comes from. Nobody writes to the database directly; everything goes through the backend.
|
||||
|
||||
This document is the reference for all flows. One thing to keep in mind while reading: the Nitrate admin-console (both its server and its browser UI) is a **private** application, not a public client. Its events describe private SaaS management actions (organizations, subscriptions, SSO), and its server channel uses a trusted shared key instead of a user session.
|
||||
|
||||
## 2. The three kinds of events
|
||||
|
||||
Not all events are worth the same. This is the most important distinction in this document:
|
||||
|
||||
| Aspect | Backend events | Penpot frontend events | Nitrate admin-console events |
|
||||
|---|---|---|---|
|
||||
| What they describe | Real actions run on the server (create file, login, invite to team...) | Things that happen in the Penpot browser app (navigate, edit, JS errors, performance) | Organization and subscription actions in the admin-console (create/rename/delete organization, members, SSO settings, checkout, onboarding) |
|
||||
| Trust level | **High**: created inside the same request that runs the action | **Best-effort**: some loss is expected (tab closed, offline, full buffer); they show **usage**, they are not strict proof | **Mixed**: server-side events are high trust (like backend); browser events are best-effort (like frontend) |
|
||||
| Who sets `profile-id` | The server, from the logged-in session (never trusts the client) | The server too: it ignores what the client says and uses the session | Server channel: the trusted admin-console server sends it explicitly. Browser channel: the Penpot server uses the session |
|
||||
| Who sets the time | Server time (`created-at`); the source of truth | Client proposes `tracked-at`, but the server fixes it if it is in the future or more than 1h late | Same rule: server stamps `created-at`; client `tracked-at` is corrected |
|
||||
| Used for | Audit, webhooks, compliance, abuse checks | Product analytics, usage counters, crash reports | Organization audit trail (admin-console e2e spec), subscription metrics |
|
||||
|
||||
Rule of thumb: if you need to know for sure "did this user do X?", look at `source = backend`. If you want to know "how do people use the app?", also look at `source = frontend`.
|
||||
|
||||
## 3. The `audit_log` table: a buffer, not a store
|
||||
|
||||
The Penpot `audit_log` table **is not the final destination**. It is a waiting tray: events sit here until every active consumer (Nexus, admin console) confirms them, and whoever finishes last deletes the row.
|
||||
|
||||
Main columns (migration `backend/src/app/migrations/sql/0054-add-audit-log-table.sql` and later ones):
|
||||
|
||||
- `id` uuid PK (`gen_random_uuid()` today; the PK used to be `(created_at, profile_id)`).
|
||||
- `name` / `type` text NOT NULL (for example `name = "create-file"`, `type = "action"`).
|
||||
- `created_at` timestamptz NOT NULL default `now()`: **server time**, the source of truth.
|
||||
- `tracked_at` timestamptz default `now()`: what the client claimed; fixed on ingest.
|
||||
- `profile_id` uuid NOT NULL: who did it.
|
||||
- `source` text: `backend` / `frontend` for full rows, `telemetry:backend` / `telemetry:frontend` for anonymized copies (see section 10).
|
||||
- `ip_addr` inet: real IP (or `0.0.0.0` on telemetry rows).
|
||||
- `props` / `context` jsonb holding transit-encoded maps: action data and metadata (versions, browser, request-id...).
|
||||
- No `archived_at`: the `audit_log_pending` queue (migration `0157`, consumer `1` = Nexus, `2` = admin console, ids never reused) tracks what is left. Every record is born pending only for the consumers enabled at insert time (never a fixed both): cheaper on single-consumer hosts and true to the design doc. Consequence, accepted on purpose: rows born with no send flag on get no pendings, so enabling archiving later does not pick them up — the sweep deletes them unarchived.
|
||||
|
||||
Indexes (each consumer has its own, so a slow consumer never blocks the others):
|
||||
|
||||
- `audit_log_pending` PK `(consumer, audit_log_id)`: each consumer owns its queue at constant cost, even with millions pending.
|
||||
- `audit_log (created_at, id)` (migration `0157`): the chronological drain order (`ORDER BY created_at ASC, id ASC`; same total order for both consumers, so concurrent drains cannot deadlock).
|
||||
- `audit_log_pending (audit_log_id)`: serves the "is anything still pending?" check of the final delete.
|
||||
- `(source, created_at)`: used by the telemetry scan.
|
||||
|
||||
Life of a row: `INSERT` (row plus one pending per enabled consumer, same transaction; a rollback takes both) → (every 5 min, Nexus at second 0 and admin console at second 30) each archive job sends its own 128-row batches oldest-first and deletes its pendings in the same transaction → whoever deletes the last pending deletes the row. A batch the destination refuses rolls back and stays for the next tick. Pendings of consumers that are off are reaped at the start of each job run (covers later deactivations), and orphan rows are swept at the end of each run, so there is no separate GC task. The FK is plain `DEFERRABLE` (never `INITIALLY DEFERRED`): every deleter removes pendings before rows, including the telemetry task's own deletes.
|
||||
|
||||
## 4. Anatomy of an event
|
||||
|
||||
Every event ends up with this shape (schema `schema:event` in `backend/src/app/loggers/audit.clj`):
|
||||
|
||||
- `name`: what happened (`"create-file"`, `"navigate"`, `"create-organization"`...). For backend, it defaults to the RPC command name (prefixed with `<module>-` outside `main`).
|
||||
- `type`: category (`"action"`, `"identify"`, `"trigger"`). Most are `"action"`.
|
||||
- `profile-id`: **who** did it (uuid). See the invariant in section 5.
|
||||
- `ip-addr`: request IP (or `"0.0.0.0"` on telemetry).
|
||||
- `props`: action data (team/project/file ids, visited route...). **Never secrets** (see `clean-props`).
|
||||
- `context`: metadata about how/when it was captured (backend/frontend version, user-agent, locale, browser, `request-id`, `initiator`, `event-origin`...). See section 8 for the full field guide, including `initiator` (which app started it: `"app"` means Penpot itself).
|
||||
- `source`: `backend`, `frontend`, `telemetry:backend`, `telemetry:frontend`. Nitrate admin-console server events land as `backend`; its browser events land as `frontend` (see section 7).
|
||||
- `source`: `backend`, `frontend`, `telemetry:backend`, `telemetry:frontend`.
|
||||
- `tracked-at` / `created-at`: claimed time and server time.
|
||||
- Memory-only (never stored): `::webhooks/event?`, `::webhooks/batch-key`, `::webhooks/batch-timeout` for the webhook fan-out.
|
||||
|
||||
Backend `create-file` example (simplified):
|
||||
`source` distinguishes channels: management events are `backend`, browser events are `frontend`, and telemetry copies use `telemetry:backend` or `telemetry:frontend`. It is not an outcome. For example, `update-file` can record a rejected save without a change to the file.
|
||||
|
||||
```clojure
|
||||
;; Selected fields: profile-id names the actor, props.id names the new file.
|
||||
{:name "create-file" :type "action" :source "backend"
|
||||
:profile-id #uuid "…" :ip-addr "203.0.113.7"
|
||||
:props {:project-id #uuid "…" :name "My design"}
|
||||
:context {:version "2.x" :initiator "app" :request-id "…"
|
||||
:client-version "2.x" :client-user-agent "Mozilla/…"}
|
||||
:created-at #inst "…" :tracked-at #inst "…"}
|
||||
:profile-id actor-id
|
||||
:props {:id file-id :project-id project-id :team-id team-id}
|
||||
:context {:initiator "app" :request-id request-id}}
|
||||
```
|
||||
|
||||
Frontend `navigate` example:
|
||||
|
||||
```clojure
|
||||
{:name "navigate" :type "action" :source "frontend"
|
||||
:profile-id #uuid "…" :ip-addr "203.0.113.7"
|
||||
:props {:route "dashboard" :team-id #uuid "…"}
|
||||
:context {:version "2.x" :locale "en" :browser "Chrome" :os "Linux"
|
||||
:event-origin "dashboard" :session-id "…" :session #inst "…"}
|
||||
:created-at #inst "…" :tracked-at #inst "…"}
|
||||
```
|
||||
|
||||
## 5. Backend events: the happy path
|
||||
|
||||
Most backend events need **no manual code**. The `wrap-audit` wrapper in `backend/src/app/rpc.clj` runs after every RPC command when `:webhooks`, `:audit-log`, or `:telemetry` is on (unless the command sets `::audit/skip`). It builds the event with `prepare-rpc-event` and passes it to `audit/submit`.
|
||||
|
||||
Each command can customize through result metadata (`rph/with-meta`):
|
||||
|
||||
- `::audit/replace-props`: replaces props fully (auth commands use `profile->props` so a register event carries the profile, not the password).
|
||||
- `::audit/props`: merges extras over the request params.
|
||||
- `::audit/context`, `::audit/profile-id`, `::audit/name`, `::audit/type`: override the defaults.
|
||||
- `clean-props` always strips nils, namespaced keys, and `:session-id/:password/:old-password/:token/:client-secret` as a last line of defense. **Never put secrets in props.**
|
||||
|
||||
Critical `profile-id` INVARIANT: the event belongs to **whoever made the request**. It resolves as `::audit/profile-id` metadata → `::rpc/profile-id` → `uuid/zero`. It is **never** taken from the response: some results carry someone else's `:profile-id` (for example `get-error-report` returns the report with its content merged; `verify-token` on an invitation returns the inviter), and using it would blame the wrong person. Commands with `::rpc/auth false` (`login-*`, `register-profile`, `create-demo-profile`, `verify-token`, `prepare-register-profile`) have no caller to fall back on, so they **must** set `::audit/profile-id` in the metadata or the event lands on `uuid/zero`.
|
||||
|
||||
The `::audit/profile-id` override goes through `coerce-profile-id`: a string is parsed, anything that cannot become a uuid is dropped (with a warning) and the event falls back to the caller. Do not pass the value raw: `schema:event` needs a uuid and `submit*` swallows the validation error (it only logs, the RPC still succeeds), so a non-uuid override **loses the row silently**. Hand-built events with `event-from-rpc-params` + `submit` (clone template, accept team invitation, `management/nitrate` push-audit-events) do **not** go through that coercion: they must already carry a uuid.
|
||||
|
||||
### Team id and entity ids (projection data)
|
||||
|
||||
The event stream is meant to let a consumer (Nexus) rebuild what it needs **without joining production tables**, so events carry the ids a projection keys on:
|
||||
|
||||
- **Team id enrichment.** After `clean-props`, `prepare-rpc-event` calls `enrich-team-id`: when props have no `:team-id` but carry a `:project-id` or `:file-id`, it resolves the team with a local query and adds `:team-id`. It never resolves from `:id` (read commands put the entity id there, and that would add a query to every read), and it skips read events entirely (`get-*` names plus `has-file-libraries`: their events carry no consumer state). Commands that already loaded the team add `:team-id` themselves in `::audit/props` (`delete-file`, comments, media upload, `delete/update-webhook`, `clone-template`, ...), so the resolver does not run.
|
||||
- **Membership.** `add-profile-to-team!` emits `add-team-member` (props `:team-id`, `:member-id`, `:role`, `:is-owner`, `:is-admin`, `:can-edit`) whenever it actually inserts a row, only with `:audit-log` on, inside the caller's transaction. It covers the flows that add a member with no event of their own (`create-team` owner, invitee added directly when `:email-verification` is off, approved join-request, duplicated team). With `on-conflict-do-nothing?` a conflict returns nil and emits nothing, so an existing membership is reported once. The event belongs to the actor: the two flows where the owner adds somebody else (`create-invitation`, `add-member-to-team`) pass `:actor-id` through a separate `event-opts` arg (never in `params`, which is inserted as-is); everywhere else actor and member coincide. `:role` is derived in the emitter from the flags (inverse of `permissions-for-role`: owner > admin > editor > viewer) because the table has no role column; request context travels in `event-opts` as `:rpc-params`.
|
||||
- **Created entity ids.** `create-project` and `create-file` add the new row's `:id` (and `:project-id`/`:team-id`) to `::audit/props`, so a projection can key it.
|
||||
- **Bulk operations.** `clone-template`, `import-binfile` and `duplicate-project` create entities through internal calls that emit nothing on their own; they emit a `create-file` (and `create-project`) per entity (only with `:audit-log`). The import paths go through `audit/submit-file-created-event`, which reads the light columns (`id`, `name`, `project-id`, timestamps, `revn`) back from the row — the file must really exist, a phantom id emits nothing. `duplicate-project` already holds the rows and uses `audit/submit-create-event` directly. Both binfile versions return the same shape from `import-files!` (`{:file-ids [...]}`).
|
||||
- **Deletes.** `delete-team` still returns nil but carries the deleted row (`:id`, `:name`, `:created-at`, `:modified-at`, `:deleted-at`) in `::audit/props`, like `delete-project`. `delete-profile` carries `:id`, `:email`, `:fullname`, `:created-at`, `:modified-at` and `:deleted-at` (without this its props are empty: every param is namespaced and `clean-props` drops them). `create-team` adds `:created-at` and `:modified-at`. `delete-team-member` needs nothing: the delete is physical and the params already carry `:team-id`/`:member-id`.
|
||||
- **Hard deletes.** `objects-gc` emits one `hard-delete-objects` event per purged chunk, carrying only that chunk's entity key (`:team-ids`, `:project-ids` or `:file-ids`; one chunk transaction purges a single entity, so one event can never span sections) with context `{:triggered-by "objects-gc"}`, only with `:audit-log` on. No other GC path emits (profiles, fonts, thumbnails); those purges stay silent.
|
||||
- **SREPL deletes.** `delete-profile!` carries the subject id in props (it used to emit nothing usable); the event profile stays zero because SREPL has no actor. `delete-team!` props already carried the id. Both travel as actions with a `triggered-by: srepl` context.
|
||||
- **SREPL restores.** `restore-profile!` carries the full row (with id) in props with a zero event profile; `restore-team!` and `restore-project!` already carried their ids. The consumer resolves the subject from props by a uniform rule, with no SREPL special case.
|
||||
- **Cascades.** The `:delete-object` worker task emits one `delete-object-with-cascade` event per execution, type `action`, with `:object`/`:id`/`:deleted-at` plus the affected ids nested under `:cascade` (`:team-ids`, `:project-ids`, `:file-ids`), in the same transaction. The handler submits unconditionally (`submit` fans out per flag, so no `:audit-log` re-check); the `audit_log` row itself is still only written with `:audit-log` on. Direct calls (objects-gc) and snapshot cascades stay silent on purpose.
|
||||
- **Ownership.** `create-team-with-invitations` carries the new team `:id` like `create-team`; `update-team-member-role` names the demoted member in `:prev-owner-id` on owner transfers.
|
||||
|
||||
Entry points:
|
||||
|
||||
- `submit`: the normal one (fills defaults, validates, runs inside `tx-run!`, never breaks the RPC).
|
||||
- `insert`: low level, for CLI helpers and the webhook subsystem; direct write, no webhook/telemetry fan-out, silent unless `:audit-log` is on.
|
||||
- Boot emits `trigger/instance-start` from `setup/props` so every restart stays visible.
|
||||
|
||||
Cases already covered by tests, do not break: `accept-organization-invitation` tells its two flows apart **with props, not with origin**: `:organization-member-add-source` is `"direct-organization-invitation"` or `"team-invitation"`, and `:belongs-to-team-on-add` repeats it as a boolean. Do not add a third prop. The `context` `:event-origin` belongs to the browser (`make-data-event` sets it and it is on the frontend allowlist; `safe-backend-context-keys` has no `:event-origin`, so a backend event has nowhere to put one).
|
||||
|
||||
## 6. Penpot frontend events: what is collected, with what guarantees
|
||||
|
||||
The browser **never writes to the table**; it POSTs transit batches to `push-audit-events` (`backend/src/app/rpc/commands/audit.clj`), which stamps server `id`, session `profile-id`, request IP, server `created-at`, and server-side `:initiator` (from the `x-client` header, see section 8; anything the client sends as `initiator` is overwritten), and distrusts the client clock (future or >1h-late `tracked-at` is reset, the original kept in `context.original-tracked-at`). Without `:audit-log`/`:telemetry`, or on a read-only pool, the endpoint is a no-op.
|
||||
|
||||
Valid types (`schema:frontend-event`): `"action"`, `"identify"`, `"trigger"`. Names must match `#"[\d\w-]{1,50}"` (max 250). `props` is a free map with no schema: **tests are the only guard of the contract** (see section 13).
|
||||
|
||||
What they are for: usage metrics (navigation, editing), workspace file stats, Nitrate membership changes, crash reports (`unhandled-exception` / `exception-page`). **Not proof for audit**: collection is best-effort and includes `PerformanceObserver` noise (`performance-*`, `user-input`, `long-task`). So backend tests must never assert exact frontend event counts.
|
||||
|
||||
How the browser collects (`frontend/src/app/main/data/event.cljs`), most to least important:
|
||||
|
||||
1. **On-demand start**: `initialize` asks the backend with `get-environment-data` (old `get-enabled-flags` is deprecated) and only starts on `:audit-log` or `:telemetry`. On RPC failure it still starts in telemetry mode (the backend will reject if needed).
|
||||
2. **Potok translation**: `make-proto-event` (events with the `Event` protocol) and `make-data-event` (data-events with `::name`) turn the action into `{type, name, props, context}`. `simplify-props` flattens complex values (maps → `:placeholder/map`, and so on) and drops namespaced keys and nils. Context gains `:event-origin`, `:event-namespace`, `:event-symbol`.
|
||||
3. **Capped buffer**: queue of max 1024 events (`append-to-buffer` drops beyond that); every 2s debounce (or on logout, or `::force-persist`) a chunk of max 100 is taken, filtered to the current profile, and sent fire-and-forget to `api/main/methods/push-audit-events` as transit. On confirm, the chunk leaves the buffer (`::chunk-persisted`).
|
||||
4. **Performance noise**: `PerformanceObserver` (`event`/`longtask`) plus blocking measurement with `scheduler.postTask` create `trigger` events (`performance-blocking-event`, `user-input`, `long-task`) with 1s debounce and 1s threshold. Info only, not audit.
|
||||
5. **`skip-audit?`**: UI flag (for example resuming a pending dashboard action after Nitrate SSO, or moving a team into an organization) so one user gesture is not counted twice.
|
||||
|
||||
## 7. Nitrate admin-console events: the third producer
|
||||
|
||||
The admin-console is the third producer. It sends organization and subscription events through **two channels** that look the same once stored, but have different trust levels. All the sending code (server and browser helpers) lives in the private Nitrate repo: it is not Penpot code and no file paths are cited here. What this section documents is only the Penpot-visible side: endpoints, payloads, and which event names arrive.
|
||||
|
||||
### 7a. Server-side channel (high trust, like backend events)
|
||||
|
||||
Code: the admin-console server has a helper that POSTs to the Penpot **management** API:
|
||||
|
||||
```
|
||||
POST api/management/methods/push-audit-events
|
||||
headers: x-shared-key: "admin-console <key>", Cookie: <user cookies>
|
||||
body: {events: [{name, props, profileId, context?}]}
|
||||
```
|
||||
|
||||
Auth happens at the HTTP layer (`backend/src/app/http/middleware.clj` `wrap-shared-key-auth`): the `x-shared-key` header carries `"<key-id> <secret>"` (`admin-console` here), checked in constant time against the configured keys; the key-id lands in the event context as `:initiator`. The call goes server-to-server (ky client, internal URI, user cookies forwarded).
|
||||
|
||||
Who uses it:
|
||||
|
||||
- Organization management (best-effort wrapper): `create-organization`, `rename-organization`, `delete-organization`, member add/remove, and more. It optionally passes the UI origin in the context.
|
||||
- Subscriptions: `create-subscription`, renewals, cancellations, and more, including deployment lookup (saas/selfhost).
|
||||
|
||||
Failure policy: every send is best-effort. A failure never rolls back the business action; it is only recorded for debugging.
|
||||
|
||||
Backend side (`backend/src/app/rpc/management/nitrate.clj` `::push-audit-events`): `::audit/skip`, `::rpc/auth false` (trust comes from the shared key, not from a user session; only the `admin-console` key id is allowed). It builds context with `audit/prepare-context-from-request` + `request-id`, takes the given `profile-id` as-is (no coercion here, must already be a uuid), sets `tracked-at = request-at`, defaults `type` to `"action"`, and calls `audit/submit` (which defaults `source` to `"backend"`). So these rows land as **`source = backend`**: they are audited like backend events because a trusted service set the author.
|
||||
|
||||
### 7b. Browser channel (best-effort, like frontend events)
|
||||
|
||||
Code: the admin-console browser has a helper that POSTs to the Penpot **public** API:
|
||||
|
||||
```
|
||||
POST api/main/methods/push-audit-events
|
||||
credentials: include, keepalive: true
|
||||
headers: x-client: "penpot-admin-console/<version>"
|
||||
body: {events: [{name, type: "action", timestamp, props, context: {eventOrigin?}}]}
|
||||
```
|
||||
|
||||
Dozens of helpers cover the admin-console UI: `create-organization`, `delete-organization-member`, `create-subscription` / `create-trial-subscription`, `open-subscription-modal`, `start-nitrate-checkout`, `open-current-subscription`, SSO settings flows (`open/apply/success/fail/discard-organization-sso-changes`, activate/deactivate modals), `change-organization-advanced-permission`, `onboarding-step` / `onboarding-finish`, `close-subscription-modal`. These helpers live in the private Nitrate repo: they are not Penpot code, you will not find them in this repository. Only the event names above (what arrives at Penpot) are listed here on purpose.
|
||||
|
||||
These rows land as **`source = frontend`** with the session `profile-id`: same guarantees as Penpot frontend events (best-effort, usage analytics). Every browser helper sends an `eventOrigin` in the context: plain `"admin-console"`, with a suffix only when the same event is emitted from two places in the app (for example the SSO discard event distinguishes the config form from the unsaved-changes modal).
|
||||
|
||||
The Nitrate repo also has a shared, generic push client for the management endpoint (not tied to the admin-console server): any other Nitrate-side service that needs to record something in the Penpot audit log can reuse it instead of writing its own HTTP call. Same endpoint, same payload shape as the server channel above.
|
||||
|
||||
## 8. How the frontend helps the backend audit: HTTP headers
|
||||
|
||||
Besides explicit events, **every RPC request from the frontend carries headers** so the backend can fill its own event `context`. This is how the backend learns "which browser/version/screen" made the request:
|
||||
|
||||
- `x-frontend-version` and `x-client: penpot-frontend/<version>`: added by `default-headers` in `frontend/src/app/util/http.cljs` to **all** fetch requests (unless `omit-default-headers`). The admin-console browser channel sends `penpot-admin-console/<version>` instead (`x-frontend-version` is ignored on the event-ingest path, only `x-client` matters there).
|
||||
- `user-agent`: set by the browser alone; the backend trims it to 500 chars (`get-client-user-agent`).
|
||||
- `x-external-session-id`: external SaaS session via `cf/external-session-id`; added by `repo.cljs` (`send!`, special `cmd!`, `multipart-upload`, export...) and also stored by `persist-events` in the frontend event context. The backend checks it (max 256, ignores `"null"`/empty).
|
||||
- `x-session-id`: `cf/session-id` (tab session), only in the main `send!`.
|
||||
- `x-event-origin`: UI origin (`::ev/origin` from param metadata, for example `"dashboard"`, `"workspace"`); sent by `send!` and the OIDC/export/upload `cmd!`. The backend trims it to 200 chars and stores it as `:client-event-origin` (different from the `:event-origin` the frontend puts in its own context!).
|
||||
|
||||
On the backend, `prepare-context-from-request` (`backend/src/app/loggers/audit.clj`) joins everything into the event context: `:external-session-id`, `:initiator` (`auth-key-id` or `"app"`), `:access-token-id/type`, `:client-event-origin`, `:client-user-agent`, `:client-version`, `:version` (backend), plus `:request-id`. Only `safe-backend-context-keys` (`:version`, `:initiator`, `:client-version`, `:client-user-agent`) survive in the telemetry shadow copy; the rest (`:client-event-origin`, `:external-session-id`, `:access-token-id/type`, `:request-id`...) is stored fully in the normal row and sent to Nexus, but dropped from telemetry.
|
||||
`props` describes the action; `context` describes how Penpot captured it. `created-at` is server time. In public ingest, `tracked-at` comes from the browser: Penpot replaces future timestamps and timestamps over an hour old with server time, keeping the original in `:original-tracked-at` before any telemetry filtering.
|
||||
|
||||
### Context field guide: `initiator`
|
||||
|
||||
`initiator` labels the application that started a request. The server sets it on both channels, but only the shared-key channel authenticates the sending application:
|
||||
For backend RPCs, `:initiator` is the shared-key id authenticated by HTTP middleware, or `"app"` for ordinary requests. For public ingest, Penpot overwrites the supplied value based on `x-client`: the admin-console product (including the legacy `penpot-nitrate` alias) maps to `"admin-console"`; Penpot and unknown products map to `"app"`. A caller controls that header. Use the label for analytics, not authorization or proof of which app sent the event. The session still sets the actor.
|
||||
|
||||
- **Backend channel** (RPC + management API): it comes from the shared-key auth layer (`wrap-shared-key-auth` in `backend/src/app/http/middleware.clj`): when a trusted service calls the management API with `x-shared-key: "<key-id> <secret>"`, the key-id (lowercased) is stored on the request as `::http/auth-key-id`, and `prepare-context-from-request` copies it into `:initiator`. When there is no shared key (a normal user RPC), it falls back to `"app"`, which means Penpot itself.
|
||||
- **Frontend channel** (`push-audit-events`): `get-client-initiator` maps the `x-client` header product to an initiator: `penpot-frontend` → `"app"`, `penpot-admin-console` → `"admin-console"`, legacy `penpot-nitrate` (old admin-console releases, remove once redeployed) → `"admin-console"`, missing or unknown → `"app"`. It overwrites any `initiator` in the event context, but the caller controls the header. Any authenticated caller can claim `"admin-console"` by sending that product in `x-client`. Use this label for usage analytics; it does not prove which application sent the event or grant permissions. The session still determines `profile-id`.
|
||||
Do not confuse `:client-event-origin` (the `x-event-origin` header on an RPC request) with `:event-origin` (context on a browser event). The former can help explain which UI triggered a backend action. The latter should not carry a backend business distinction.
|
||||
|
||||
Values you will see:
|
||||
## 2. Emitting events
|
||||
|
||||
- `"app"`: normal Penpot traffic (browser → public RPC or event ingest, or backend internal work). This is the vast majority.
|
||||
- `"admin-console"`: the authenticated Nitrate admin-console on management API calls (section 7a), or a caller claiming that product on browser events (section 7b, via the `x-client` mapping).
|
||||
- `"nexus"`, `"exporter"`, `"media-processor"`: other first-party services using their own shared keys (key ids come from `::setup/shared-keys` in `backend/src/app/setup.clj`, derived from the instance secret unless overridden).
|
||||
### Backend events
|
||||
|
||||
Why it matters: it supports per-application usage counts, with the trust level described above. It survives into telemetry on both channels (one of the four backend keys, and on the frontend allowlist too), so per-initiator counts stay available even on anonymized rows.
|
||||
The RPC wrapper emits after a command returns when full-row persistence, telemetry or webhooks is enabled. Use `::audit/skip` for a command that should not emit. Use result metadata to add `::audit/props`, replace request props with `::audit/replace-props`, or set an explicit name, type or context. **Never put secrets in props or context.** `clean-props` drops some known sensitive keys on the RPC path; it is not a general privacy filter for every producer.
|
||||
|
||||
More metadata the Penpot frontend adds inside each event (not as headers): `collect-context` with `ua-parser` (browser, engine, OS, device, screen, CPU arch), `locale` (updated on language change), `:session` / `:session-id` / `:external-session-id`, and SaaS host extras (`add-external-context-info`).
|
||||
Attribute the event to the **actor**, not an arbitrary `:profile-id` in the response. An invitation response can carry the inviter's id even though the invitee made the request. Use `::audit/profile-id` when the command has established the actor and the normal caller id is absent or not the right one. The RPC wrapper normalizes valid UUID strings and drops invalid overrides before falling back to the caller or zero. Hand-built events do not go through that normalization: pass a UUID to `audit/submit`. Reserve zero for cases without a known actor, such as SREPL actions.
|
||||
|
||||
## 9. Consumers I: webhooks
|
||||
`audit/submit` propagates failures. `wrap-audit` runs **after** the command's transaction, if it has one: an audit failure can fail the response even though the business mutation is already committed. An explicit `submit` inside the caller's transaction can roll back the mutation with the event. Low-level `audit/insert` only writes a full row when persistence is on; it does not fan out to webhooks or telemetry.
|
||||
|
||||
Webhooks let an external URL get notified when something happens in a team (for example "a file was created"). The flow has three steps:
|
||||
### Events used to rebuild state
|
||||
|
||||
1. **Match.** Some audit events are marked as webhook-relevant when they are created. For each of those, Penpot looks up the active webhooks of the affected team: it reads the team straight from the event, or finds it through the project or the file in the event. Each matching webhook gets its own delivery job. If several quick events belong together, they are grouped into one delivery (the grouping rule travels with the event itself).
|
||||
2. **Deliver.** Penpot POSTs the event to the webhook URL in the format that webhook asked for: JSON (with camelCase keys), transit, or form-encoded. Every attempt is written to a delivery log with the request, the response, and the error if any, so failures can be inspected later.
|
||||
3. **Protect.** Each webhook counts its consecutive failures. After 3 in a row, Penpot switches the webhook off automatically, so a dead URL stops receiving traffic until someone fixes it. A successful delivery resets the counter to zero.
|
||||
Put the ids a projection needs in the event so the receiver does not have to join production tables. A create event needs the new entity's id. Supply `:team-id` when the command already has it; for writes the wrapper can resolve it from a project or file id, not a bare `:id`. Reads do not need that lookup. For a bulk operation whose internal calls do not emit, submit one create event per entity when full rows are persisted. Use `audit/submit-create-event` when you have the row or `audit/submit-file-created-event` when you have only a file id; the latter checks that the file exists.
|
||||
|
||||
One guarantee: sending webhooks never changes the audit event itself. If delivery fails, only the webhook (and its delivery log) is affected; the audit row stays untouched and still gets archived normally.
|
||||
- `add-team-member` reports an actual membership insert, not a conflict that made no change. The emitter submits in the caller's transaction when full rows are persisted. Keep the actor separate from `:member-id` when someone adds another person.
|
||||
- `update-file` reports `"saved"`, `"partial"` or `"rejected"`. Only the first two write a change row and carry `:file-change-id`. The id points to the change, not the previous-state snapshot. Read it with `::db/remove-deleted false` because it is born with a future deletion date; even then, GC may already have removed it.
|
||||
- For moves, preserve source and destination team ids. For cascades and hard deletes, report the affected ids at emission rather than expecting a projection to reconstruct them later from deleted rows. The `:delete-object` handler emits its cascade event in the same transaction; direct and snapshot cascades do not emit that event.
|
||||
- `accept-organization-invitation` distinguishes direct and team invitations with `:organization-member-add-source` and `:belongs-to-team-on-add`. Keep that distinction in props; do not add another origin field.
|
||||
|
||||
Every match also writes a `webhook` / `trigger` row to `audit_log` (even with zero matches): its props are `{:name <origin event name>, :event-id <origin row id>, :total-affected <matched webhook count>}`. `process-event` assigns the origin row id up-front so the trigger can reference it; the telemetry shadow row always gets its own id (reusing it would collide on the PK).
|
||||
### Browser events
|
||||
|
||||
## 10. Consumers II: error reporters
|
||||
The browser collector starts on either `:audit-log` or `:telemetry` from `get-environment-data`. If discovery fails it starts anyway; the backend still decides whether to accept events. The collector buffers a bounded number of events and sends batches for the current profile. It can lose events on shutdown, overflow or failure. HTTP success does not prove a write: if storage fails, the endpoint logs the error and still returns normally. If it does write both the full and shadow rows, they commit together.
|
||||
|
||||
Both `app.loggers.database` and `app.loggers.mattermost` listen to backend `:error` logs **and** frontend crash events (found by the `::audit/event` marker), through a sliding-buffer channel so a flood cannot stall the app. The database one stores them in `server-error-report` (source 3 = backend log, 4 = audit-log origin, 5 = rate-limit); the Mattermost one forwards a short note to `:error-report-webhook` when set.
|
||||
Use a data-event or the `Event` protocol for a new event. Use kebab-case names (the public schema caps names at 50 characters) and reuse existing prop names. Do not add a browser copy of an action the backend already reports unless the browser observes something distinct. Honor `skip-audit?` when the UI resumes the same gesture. Performance events are usage data, not audit evidence.
|
||||
|
||||
Both reporters only start when the `:error-reporting` flag is on (off by default). The database reporter starts on the flag alone; the Mattermost reporter needs the flag **plus** its webhook URL configured. Without the flag, nothing is stored and `emit` calls from `push-audit-events` are silent no-ops. Each reporter also keeps a runtime on/off switch for emergencies via REPL.
|
||||
## 3. Destinations and privacy
|
||||
|
||||
Result: crash reports only exist if the frontend collector runs and `push-audit-events` accepts `unhandled-exception`/`exception-page`. Turning off the whole pipeline also blinds frontend error reporting.
|
||||
|
||||
## 11. Consumers III: telemetry (shadow rows)
|
||||
|
||||
Telemetry reuses the same table with **anonymized shadow rows** (`source LIKE 'telemetry:%'`): day-truncated times, `0.0.0.0` IPs, props reduced to uuid/boolean/number values plus a few allowed fields (`source`, `lang`, `auth-backend`, `activation`, derived `email-domain` — never raw emails), and a minimal context allowlist. Full + shadow rows can exist together per event when both flags are on; that duplication is on purpose.
|
||||
|
||||
Filters (`filter-telemetry-props` / `filter-telemetry-context`): by default only uuid/boolean/number with a simple key; exceptions: login/register/update-profile and frontend `identify` keep `lang`/`auth-backend`/`source`/`activation`/`email-domain`; `organization-sso-auth-failed` keeps `failure-reason` if it is a known one; `instance-start` passes whole; frontend `navigate` keeps `[:route :file-id :team-id :page-id]`. Subscription actions have event-specific contracts: backend and frontend `create-subscription` / `create-trial-subscription` preserve the default uuid/boolean/number props plus `type`, `deployment`, `quantity` and `subscription-mode`; backend `update-subscription` preserves only `type`, `deployment`, `previous-status`, `status`, `previous-cancel-at-period-end`, `cancel-at-period-end`, `previous-cancel-at`, `cancel-at` and `changed-fields`; backend `subscription-summary` preserves only `type`, `deployment`, `instance-id`, `snapshot-date`, the per-status `*-count` values, `total-count`, `sync-succeeded`, `sync-failed` and `snapshot-complete`. These subscription exceptions apply only to action events. Other free text drops from telemetry but **stays in the full row and in Nexus**. In `register-profile`, `activation` is a prop (it is what reaches PostHog). `action` exists only in the `context` of the full `register-profile-retry` rows; `filter-telemetry-context` drops it from the anonymised copy, which keeps only `:version`, `:initiator`, `:client-version` and `:client-user-agent`.
|
||||
|
||||
Shipping (`backend/src/app/tasks/telemetry.clj`): the cron sends shadows to `:telemetry-uri` as JSON in 10k batches and **deletes** them on success; it purges leftovers older than 7 days. It also sends a legacy report (counters, teams, email domains...). On official hosts nothing is collected or sent (`telemetry-excluded?` covers `penpot.app`/`penpot.dev`).
|
||||
|
||||
### Where telemetry events go: `telemetry.penpot.app` is Nexus
|
||||
|
||||
The default `:telemetry-uri` (`https://telemetry.penpot.app/`) points at the **Nexus telemetry endpoint**, so telemetry is a second road into Nexus, separate from the archive cron. The payload is plain JSON:
|
||||
|
||||
```
|
||||
POST https://telemetry.penpot.app/
|
||||
{type: "telemetry-events", version, instance-id, events: [...]}
|
||||
```text
|
||||
RPC / management / public ingest
|
||||
├─ full rows ────┐ ┌─ Nexus archive
|
||||
├─ shadow rows ──┴─ audit_log buffer ─────┼─ admin-console archive
|
||||
│ └─ telemetry endpoint
|
||||
├─ webhook jobs (marked producer events)
|
||||
└─ error reporters (crash events from public ingest; also receive backend logs)
|
||||
```
|
||||
|
||||
Nexus dispatches on `:type`: `"telemetry-events"` batches are coerced and inserted into its own `audit_log`; anything else (old flat payloads, `"telemetry-legacy-report"`) is upserted into `telemetry_report` per instance + day. Three details matter:
|
||||
### Full rows and archive consumers
|
||||
|
||||
- **Anonymization happens in Penpot, before sending.** By the time events leave, times are day-truncated, IPs are already `0.0.0.0`, props/context are filtered (see above), and emails are gone (only derived `email-domain` survives). Nexus additionally forces the outcome: it prefixes any non-`telemetry:` source, sets `received-at = tracked-at` (already truncated), pins IP to `0.0.0.0`, and stamps the sending `instance-id` into every event context.
|
||||
- **No personal data is expected.** Profile ids are random UUIDs with no PII, so the telemetry consumer needs no login guard; grouping is by `instance`, not by person (see posthog-telemetry below).
|
||||
- **The endpoint always answers 200.** Failures are swallowed and only counted (`nexus_telemetry_errors_total`), so a bad batch never breaks the sending instance; successes are counted (`nexus_telemetry_events_total`, `nexus_telemetry_reports_total`). A body the parser cannot read never reaches the handler: it answers 400 and counts in `nexus_http_malformed_requests_total`. Old instances may send a fressian+zstd base64 blob instead of a plain vector; Nexus decodes both (50 MiB cap against decompression bombs).
|
||||
Nexus receives full events, not telemetry shadows. Penpot sends Transit with the instance's `public-uri` as `origin` and a Nexus shared key. In the Nexus archive contract, Penpot's `created-at` becomes `received-at`; Nexus has its own `created-at` and `tracked-at` stays the origin time. The default telemetry endpoint also points to Nexus, but it is a separate ingest path with a reduced payload.
|
||||
|
||||
## 12. Archive to Nexus and to the admin console
|
||||
The admin-console archive is **Penpot → Nitrate**, distinct from either Nitrate producer channel. Its consumer selects an allowlist of full events and enriches them with names or emails needed by the destination. The producer does not know that allowlist. Nonmatching rows still advance the cursor.
|
||||
|
||||
### How it works today
|
||||
### Webhooks and error reporters
|
||||
|
||||
Every 5 min each archive job drains its own queue (`:audit-log-consumer-nexus` on flag `:nexus`, cron `0 */5 * * * ?`; `:audit-log-consumer-admin-console` on flag `:admin-console`, cron `30 */5 * * * ?`, 30s offset so the two drains never start on the same second), in 128-row chunks taken oldest-first (`ORDER BY created_at ASC, id ASC`) with `FOR UPDATE OF audit_log` (`SET LOCAL lock_timeout = '30s'`) inside the same transaction that sends and completes them, with 100 ms pauses between chunks. Batches arrive at Nexus in chronological order (it numbers by arrival for order-sensitive consumers). Nexus receives transit `{:events [...]}` with `x-shared-key: "nexus <key>"` (`:nexus-shared-key`, or derived from the instance secret) and `origin` = public-uri; flag on but no URI raises `:task-not-configured`. The admin console receives only its allowlisted event names (enriched with team names, props emails, and optional top-level `:profile-email` / `:profile-name` from the actor `:profile-id`); anything filtered out counts as done for the admin-console queue. A chunk whose destination refuses it (Nexus non-204 raises `:consumer-chunk-refused`) or throws (transport failure, admin-console error) rolls back and stays for the next tick, and the job aborts as failed: it archives what it can, and the cron retries the rest. Committed chunks stay sent. The send fns return counts or throw carrying the attempted count as `:events`; `run-single-consumer` records the `failed` metric from it, outside every chunk transaction (a local DB failure carries no `:events` and stays uncounted — it is not a destination failure). No `warn` before the throw: the runner reports the aborted job. Each job touches only its own queue, so a fallen Nexus never starves the admin console (nor vice versa).
|
||||
Webhooks use jobs, not a cursor over the audit buffer. A marked event schedules matching and delivery for the affected team's webhooks. When debounce is configured, a burst delivers the **last payload**, not a list of the events. Delivery failure changes webhook status and its delivery log, not the origin audit row. The audit `webhook` trigger records the match count, including zero, only if full-row persistence is on. Its `:event-id` comes from the producer event, so it can refer back to the full row when that row exists. A telemetry copy has its own id.
|
||||
|
||||
A chunk is **one POST** and the task advances only on 204, so a single oversized event wedges the whole queue: a crash report whose message embeds a whole payload (a serialized file, for example) can push the request past the proxy body limit and the rejection blocks every later event until it is pruned by hand. Frontend crash reports are therefore size-bounded at the source (`mem:frontend/routing-app-shell-subtleties`): `app.common.exceptions/get-hint` and the CLJS `format-throwable` clamp exception messages (`max-message-length`).
|
||||
With `:error-reporting` on, public ingest forwards `unhandled-exception` and `exception-page` to the reporters even if neither kind of audit row is stored. The database reporter runs on that flag alone; Mattermost also needs its webhook URL. Reporters also receive backend error logs. The browser collector still has to run for browser crashes to reach them. Bound crash reports at the source: a message that embeds a serialized file can exceed the receiver's body limit and prevent later rows in the chunk from advancing. See `mem:frontend/routing-app-shell-subtleties` for frontend report limits and classification.
|
||||
|
||||
On the other side, Nexus checks `origin` against its allowlist (403 if not allowed), coerces the events, and inserts them into its own `audit_log` (partitioned by `received_at`, with an incrementing sequence number and `source`). Watch the rename: Penpot's `created-at` is stored as **`received-at`** in Nexus; Nexus's `created-at` is when it was created there. `tracked-at` keeps the origin time. Telemetry batches arriving at `telemetry.penpot.app` land in the same table through the telemetry endpoint above.
|
||||
### Telemetry shadows
|
||||
|
||||
Concurrent jobs are safe: both lock rows in `created_at ASC, id ASC` order (one total order, deliberately blocking — no `SKIP LOCKED`, so archive arrival stays chronological), so the second waits for the first to commit and then sees whether it was last. Only acknowledged work deletes pendings; an admin-console failure never blocks the Nexus send of the same rows (each job owns its queue).
|
||||
Telemetry writes a separate, reduced copy, even when full-row persistence is also on. It truncates times to the day, sets the stored IP to `0.0.0.0`, and filters props and context before storage. The telemetry POST omits row id, `created-at` and IP; it keeps `profile-id`, a stable UUID that can link events. Treat the copy as reduced data, not as proof of anonymity.
|
||||
|
||||
Observability (`backend/src/app/loggers/audit/consumer_task.clj`, definitions in `backend/src/app/main.clj:default-metrics`): one `dbg` summary per job run (`:consumer` label, sent/skipped/chunks, reaped disabled pendings, swept orphans, `:elapsed` as `"1.5s"` via `ct/format-duration`, frozen in a strict binding first because log props evaluate lazily in the logger thread), one `dbg` per admin-console send call (`ready` with totals, `sent`/`failed` with counts), plus `dbg` when the queue is empty or the job's send flag is off. Queue depth is Grafana's business: it queries the database directly, so the run performs no `GROUP BY` for the logs. No `warn` before a throw: the runner reports the aborted job (refusal details travel in the error data). Metrics, all recorded with bounded labels only: `penpot_audit_consumer_events_total{ destination = nexus|admin-console, outcome = archived|failed|skipped }` (`skipped` = filtered by the admin-console allowlist) and `penpot_audit_consumer_discarded_total{ reason = reaped|swept }` (entries discarded without archiving: `reaped` counts pendings dropped for disabled consumers, `swept` counts rows deleted while nobody waited for them — the silent R3 loss, now dashboard-visible instead of log-only). Timing needs no custom histogram: the jobs substrate already records execution time per job (`jobs-execution-timing`, split by job name), so each archive job is timed as itself. No queue-depth gauge: pending depth is read off the replica database directly. The bulk frontend ingest counts `penpot_audit_ingest_events_total{ kind = full|shadow, outcome = inserted|failed }` (both faces share one transaction, so a failure drops the batch on both). Archive success counts are recorded by `run-single-consumer` on the drain totals: a chunk that returns normally committed its transaction (a rollback always throws instead), so they cannot inflate. Logs never carry event data (`props`, `context`, `profile-id`, `ip-addr`, `team-name`, emails): only counts, statuses and the uri host.
|
||||
For example, auth events can keep a derived `email-domain`, not the raw email; a full archive row may still contain the email. Context drops request and session identifiers but keeps an allowlisted `:initiator` for counts by application. Event-specific filters allow some values the default filter would drop. Check `filter-telemetry-props` and `filter-telemetry-context` with their tests before changing a field that leaves the instance.
|
||||
|
||||
### How Nexus consumes the data (and forwards to PostHog)
|
||||
The telemetry consumer scans the same stream, skips full rows, and ships shadows independently of the full archives. It uses a three-hour pause and larger chunks to avoid flooding the endpoint with small requests. The legacy `:telemetry` job still sends the aggregate report and newsletter data; it does not drain these rows. Its event counters see only the shadows still in the buffer, not a complete day's activity.
|
||||
|
||||
Stored rows are not the end: a consumer pipeline reads them back in sequence order and processes them. Each consumer remembers its place (sequence number), fetches the next chunk, runs its handler, then saves the new position. Three consumers run in order:
|
||||
## 4. How the buffer drains
|
||||
|
||||
1. **`local`**: keeps a mirror of user profiles. It listens to profile events (`register-profile`, `update-profile`, `login-with-*`, `delete-profile`, ...) and upserts email, name, props, and delete markers. It runs first because the next consumer needs profile data.
|
||||
2. **`posthog`** (runs after `local`): forwards our own SaaS archived events to PostHog. It sends an allowlist of backend track events (teams, members, projects, files, comments, Nitrate/SSO, ...) plus frontend events minus a noise exclusion list, grouped by **team**, with person identify data from the local profile mirror and snake_case props. Both paths include the `initiator` property when the stored context has one, so PostHog can split Penpot vs admin-console traffic.
|
||||
3. **`posthog-telemetry`**: forwards events with `source = telemetry:frontend|telemetry:backend` to a separate PostHog project. Because these rows are already anonymous (no IPs, truncated times, filtered props), grouping is by **instance** (the self-hosted server, from `context.instance-id`), with one `$groupidentify` event per instance per chunk so PostHog learns per-instance properties (version, URI, flags). Each consumer has its own PostHog credentials.
|
||||
`audit_log` is a buffer, not the archive. Producers insert rows without knowing the destinations. Each consumer keeps its own position in `audit_log_consumer_cursor`. Worker instances run consumers in background threads, not jobs. A session advisory lock allows only one pass per consumer per database. The pass holds a pool connection and fixes its upper bound (the *frontier*) at the start, so it ends even while new events arrive.
|
||||
|
||||
So the full journey is: Penpot buffer → Nexus `audit_log` (via either archive cron or the telemetry endpoint) → consumers → PostHog (SaaS events grouped by team, telemetry grouped by instance) plus local analytics views and materialized KPIs.
|
||||
### Ordering and retry
|
||||
|
||||
### Admin console: the second consumer
|
||||
The cursor orders rows by `(xid, seq)`: transaction id first, then sequence within the transaction. A snapshot cut based on `pg_snapshot_xmin` keeps the reader from moving past an earlier open transaction. For example, A inserts a row and stays open. B inserts and commits while A is open; then A inserts a second row and commits. A sequence-only cursor could read B and skip A's first, still invisible row. With the cut, the drain waits and can then deliver both A rows before B. This is neither global insertion order nor commit order.
|
||||
|
||||
Penpot → admin-console archiving lives in its own job (`:audit-log-consumer-admin-console`, consumer `2`, flag `:admin-console`, params closed empty): it drains its own pending queue and ingests allowlisted events through `nitrate/call :ingest-audit-log`, enriched as described above. What also exists (do not confuse with archiving):
|
||||
The cut is conservative across the database. An unrelated long write transaction or a slow send inside a cursor transaction can delay what other passes can read, even though they do not lock one another's cursors. The GC can still delete up to positions already confirmed.
|
||||
|
||||
- **Nitrate → Penpot**: `POST /api/management/methods/push-audit-events` (`backend/src/app/rpc/management/nitrate.clj`) lets the trusted Nitrate backend inject events into the Penpot audit log (no RPC auth, `::audit/skip`, request context + `submit`; only the `admin-console` shared-key id can call it). Section 7a is the main user.
|
||||
- **Admin-console browser → collector**: usage events from the admin-console UI traveling through the normal public `push-audit-events` (section 7b).
|
||||
- **Penpot frontend Nitrate bits**: membership/subscription events (`frontend/src/app/main/data/nitrate.cljs`, `nitrate-audit/*`) through the same public endpoint.
|
||||
Each chunk sends and advances its cursor in one local transaction. A refusal aborts that chunk and leaves its cursor in place; earlier committed chunks stay done. An external send cannot roll back with the local transaction: if the destination accepts it and the cursor commit fails, the next pass can send it again. Consumers must handle repeats. A filtered chunk advances even if it sends nothing.
|
||||
|
||||
## 13. Flags and config
|
||||
### Registration, deletion and replay
|
||||
|
||||
Flags (`common/src/app/common/flags.cljc`, varia; enabled with `PENPOT_FLAGS=enable-<name>`):
|
||||
The component registers its cursor at `(0, 0)` after configuration succeeds and before its loop starts, so a new consumer replays what remains in the buffer. Consumer ids are stable and never reused: `0` telemetry, `1` Nexus, `2` admin console. A disabled consumer keeps its position until retired. Turning off both full-row consumers stops new full rows; it does not empty the buffer or stop telemetry shadows.
|
||||
|
||||
| Flag | What it does |
|
||||
Only the GC deletes. For full rows it deletes through the oldest registered cursor other than `0`; for shadows it deletes through cursor `0`. A missing cursor leaves its cursor-based path idle. A separate seven-day retention path deletes shadows by age regardless of cursors. It can delete unsent shadows; an older shadow can still ship if the consumer reads it before GC removes it.
|
||||
|
||||
The GC reads cursors again for each batch but does not serialize deletion with registration. A new consumer can miss up to one batch **per concurrent GC execution that read the previous cursor state**. There is no database-wide one-batch limit: cron job exclusivity is scoped by tenant, while the buffer and cursors are shared by the database. For a complete replay of what remains, pause the GC and wait for queued and running GC jobs to finish before registering. Follow `operations/2.20.0-consumer-generalization.md` for the rollout and replay procedure.
|
||||
|
||||
### What to watch
|
||||
|
||||
- `penpot_audit_consumer_lag_events` counts visible rows after a consumer's cursor, capped at 10,000. It includes rows that the consumer will filter; it is not a count of undelivered requests.
|
||||
- Pass outcomes distinguish a failed pass from `busy` (another instance holds the lock). Some `busy` is normal with several workers; sustained `busy` with no successful passes warrants checking the lock holder. `skipped` rows were filtered by the destination, not lost.
|
||||
- `penpot_audit_consumer_discarded_total{reason="gc"}` counts all committed deletions. The `purged stale telemetry events` warning counts **age-based deletions**, not confirmed missed deliveries.
|
||||
|
||||
## 5. Configuration, changes and tests
|
||||
|
||||
| Setting | Effect |
|
||||
|---|---|
|
||||
| `:audit-log` | Stores full rows; turns on `wrap-audit` and frontend ingest |
|
||||
| `:nexus` | Send flag of the Nexus archive job `:audit-log-consumer-nexus` (cron `0 */5 * * * ?`; needs `:nexus-uri` (ingest path `api/audit` appended by the job)) |
|
||||
| `:admin-console` | Send flag of the admin-console archive job `:audit-log-consumer-admin-console` (cron `30 */5 * * * ?`, 30s after Nexus); also the module flag for the admin-console integration |
|
||||
| `:audit-log-logger` | Structured `app.audit` log per event |
|
||||
| `:error-reporting` | Starts the error reporters (off by default); database reporter on flag alone, Mattermost on flag plus webhook URL |
|
||||
| `:telemetry` | Stores shadow rows and turns on shipping (auto with `:telemetry-enabled`) |
|
||||
| `:nexus`, `:admin-console` | Start the corresponding archive consumers and enable full-row persistence. Set the intended flags consistently on web and worker instances: each computes persistence at boot. Nexus needs `:nexus-uri`. |
|
||||
| `:telemetry` | Collect and ship shadow rows to `:telemetry-uri`, separately from full-row persistence. |
|
||||
| `:error-reporting` | Start error reporters and enable public ingest for crash reports without requiring stored audit rows. |
|
||||
| `:audit-log-gc` | Schedule the GC, on by default. Use `disable-audit-log-gc` during a mixed-version rollout and follow the runbook before restoring it. |
|
||||
| `:audit-log-logger` | Log backend audit events, including event data. This is not the archive or a substitute for persistence. |
|
||||
|
||||
Config (`backend/src/app/config.clj`): `:nexus-uri`, `:telemetry-uri` (default `https://telemetry.penpot.app/`), `:nexus-shared-key` / instance secret, `:public-uri` (sent as `origin` to Nexus).
|
||||
`:audit-log` is not a configurable flag. As a wire signal it means full-row persistence **or** error reporting. `:telemetry` is a separate wire signal. Public ingest skips processing on a read-only pool or when none of those destinations is enabled.
|
||||
|
||||
## 14. Practical guide: add or change events
|
||||
To add a consumer, give it a non-reused id and a transport definition for the shared `run-pass` loop. Configure its `:process`, optional `:prepare` and worker component; do not add producer-specific fan-out or seed its cursor in a migration. For contract tests, drive the real pass with prepared cfg; see `mem:backend/testing` for the test system and stubs. When changing an emitter, assert actor, props and exact emission counts for the specific backend action. Checking only the first matching event misses duplicates. Do not assert exact global counts from best-effort browser collection.
|
||||
|
||||
- **New backend command that must (or must not) emit**: `wrap-audit` does it alone; to exclude it set `::audit/skip`. To customize props/name/type use result metadata. Add a test mocking `app.loggers.audit/submit` (nil; `helpers.clj` stubs it globally) and assert `:call-args-list` — see `backend_tests/rpc-audit-test.clj` (full-row, telemetry-only, dual, no-op without flags, `insert`, `prepare-rpc-event`) and `rpc-team-test/accept-organization-invitation-audit-event` (three cases, exact props; counts matter: asserting with `first` passes even when a command emits the same name twice).
|
||||
- **New Penpot frontend event**: create the data-event or implement `Event`/`PerformanceEvent` in `app.main.data.event`, with `::name`/`::type`/`::origin`. Honor `skip-audit?` if the action can resume. Remember: with no props schema, **your asserts are the contract**.
|
||||
- **New admin-console event**: server action → add the backend-log call next to the mutation, best-effort so it never breaks the action; browser action → add a new event helper plus its test.
|
||||
- **Command that creates an entity or already knows its team**: put the new row's `:id` in `::audit/props` (`create-project`/`create-file`) and set `:team-id` when the team is loaded, so the consumer never has to join production tables. For entities created inside an internal call (`clone-template`, `import-binfile`, `duplicate-project`), emit one event per entity: `audit/submit-file-created-event` when only the id is known (it reads the row back), `audit/submit-create-event` when the row is already in hand.
|
||||
- **Never** put secrets in `props`; `clean-props` is a last defense, not an excuse.
|
||||
- **Do not duplicate events**: if the backend already emits an event for an action, do not add a frontend (or admin-console browser) event for the same action — the backend one wins because it is reliable. A second event for the same action is only acceptable with a strong reason (for example the backend cannot see it at all), and then it must be clearly justified.
|
||||
- **Follow the naming policy**: use kebab-case names (`"create-organization"`, not `"createOrganization"`), and before inventing prop keys, look at similar existing events and reuse their key names (`organization-id`, `team-id`, `file-id`, `event-origin`...). Same concept, same key spelling — that is what keeps the data queryable later.
|
||||
### Where to look
|
||||
|
||||
## 15. Code map
|
||||
|
||||
- Backend producer: `backend/src/app/loggers/audit.clj` (`submit`, `submit*`, `insert`, `prepare-rpc-event`, `enrich-team-id`, `submit-create-event`, `event-from-rpc-params`, telemetry filters), `backend/src/app/rpc.clj` (`wrap-audit`), `backend/src/app/rpc/commands/audit.clj` (`push-audit-events`, `get-environment-data`, deprecated `get-enabled-flags`), `backend/src/app/rpc/management/nitrate.clj` (`push-audit-events` Nitrate→Penpot), `backend/src/app/http/middleware.clj` + `backend/src/app/http/management.clj` (shared-key auth).
|
||||
- Membership emitter: `backend/src/app/rpc/commands/teams.clj` (`add-profile-to-team!` → `add-team-member`).
|
||||
- Penpot frontend producer: `frontend/src/app/main/data/event.cljs` (collector, buffer, send), `frontend/src/app/main/repo.cljs` + `frontend/src/app/util/http.cljs` (headers), `frontend/src/app/main/data/nitrate.cljs` (Nitrate membership events).
|
||||
- Admin-console producer (private repo, no paths cited here): server-side organization/subscription events through the management `push-audit-events`, browser-side UI events through the public `push-audit-events`.
|
||||
- Consumers: `backend/src/app/loggers/webhooks.clj`, `backend/src/app/loggers/database.clj`, `backend/src/app/loggers/mattermost.clj`, `backend/src/app/tasks/telemetry.clj`.
|
||||
- Archive (two jobs sharing one namespace + pending queue, no GC): `backend/src/app/loggers/audit/consumer_task.clj` (`:audit-log-consumer-nexus` / `:audit-log-consumer-admin-console` job-defs, shared drain/reap/sweep helpers), producer hook in `backend/src/app/loggers/audit.clj` (`append-audit-entry`, single `queue-pendings` helper) plus the bulk ingest in `backend/src/app/rpc/commands/audit.clj` (`handle-events`), metrics/cron wiring in `backend/src/app/main.clj` (one cron entry per job, Nexus at second 0 and admin console at second 30, each gated by its own send flag), single migration `0157` (queue, drop of `archived_at`, covering `(created_at, id)` index for the chronological scan; no backfill — pre-existing rows are swept unsent, an accepted loss).
|
||||
- Nexus (private repo, no paths cited here): archive ingest, ordered consumers (local profile mirror, PostHog SaaS events, PostHog telemetry), telemetry endpoint, analytics views.
|
||||
- Tests: `backend_tests/rpc-audit-test.clj`, `rpc-team-test/accept-organization-invitation-audit-event`.
|
||||
- Producer and telemetry filtering: `backend/src/app/loggers/audit.clj`; RPC wrapper: `backend/src/app/rpc.clj`.
|
||||
- Public ingest and discovery: `backend/src/app/rpc/commands/audit.clj`; browser collector: `frontend/src/app/main/data/event.cljs`; trusted ingest: `backend/src/app/rpc/management/nitrate.clj`.
|
||||
- Shared drain, transports and GC: `backend/src/app/loggers/audit/`; rollout: `operations/2.20.0-consumer-generalization.md`.
|
||||
- Contract examples in `backend/test/backend_tests/`: `rpc_audit_test.clj` for emitters and ingest, `loggers_audit_consumers_test.clj` for cursor ordering, retry and GC, and `rpc_team_test.clj` for invitation attribution.
|
||||
|
||||
@ -1,3 +1,3 @@
|
||||
# Backend Audit Log
|
||||
|
||||
Superseded by `mem:audit-log`: the full audit-log and event-processing guide (backend center, frontend collection, headers, webhooks, error reporters, telemetry, Nexus archive and planned Nitrate archive, table-as-buffer lifecycle, event anatomy, flags, tests). Read that memory instead; keep this stub so old references do not rot.
|
||||
Moved to `mem:audit-log`, the guide to event collection, delivery and retention.
|
||||
|
||||
@ -29,6 +29,7 @@
|
||||
- The organization-owner fallback (`get-organization-owner-permissions` via Nitrate over HTTP) grants read-only access for a non-member organization owner. It resolves the owning team through `resolve-team-id`, which refuses soft-deleted file/project/team: the file and project lookups filter `deleted_at`, and a final `db/get*` check covers the team row itself — `nil` means no access. `db/get*` only filters soft-deleted rows when `deleted_at` is in the projection. The re-checks of open websocket subscriptions rely on this (`mem:backend/websocket-subscriptions`).
|
||||
- File/project/comment checks are implemented in the owning command namespaces, often via helpers imported from `files`, `teams`, or `projects`; do not bypass those helpers with direct DB lookups unless preserving their not-found semantics.
|
||||
- Comment permission includes both logged-in state and the file/team comment policy. Shared viewer paths may pass `share-id`; preserve that path when changing comment queries. RPCs that accept `share-id` enforce the link's page scope before mutating (`share-link-pages` + `check-page-scope`): a thread or page outside the link raises `:not-found`, and that check runs before the ownership checks. `get-profiles-for-file-comments` filters its query by the same pages. `mark-all-threads-as-read` does not take `share-id`.
|
||||
- Share-link view-only bundles are page-scoped for components too: `get-view-only-bundle` filters `pages`/`pages-index` to the link, trims the primary file's `:components` to live components referenced transitively from the allowed pages (deleted components never travel), and emits them inline; membership bundles keep the full primary components map. Linked libraries are trimmed to the components the emitted data references, and that walk reads the emitted data itself (scoped pages plus the trimmed components map) rather than the full realized file, so neither excluded pages nor deleted primary components — whose stored `:objects` may instance library components — retain a library component; library pages never travel either. `get-file-fragment` (`check-fragment-scope`) authorizes only fragments of the link's pages for share links (`:not-found` otherwise): no root-level pointer, components included, is ever authorized.
|
||||
|
||||
## Teams, projects, and invitations
|
||||
|
||||
|
||||
@ -43,7 +43,7 @@ Backend RPC command areas without focused memories include access tokens (auth r
|
||||
- Next.js/JDBC caveat: where-maps don't support value vectors (no `IN`); use `status = ANY(?)` with `db/create-array` inside `db/tx-run!`.
|
||||
- Use `db/run!` for multiple operations on one connection.
|
||||
- Use `db/tx-run!` for transactions.
|
||||
- `job`-substrate specifics (unified jobs): the only writers of a job status are `claim`, `retry-job`, `complete`, `fail` and `cancel` in `app.jobs`; each one is a conditional update plus its `job_event` row in the same transaction, so history can never contradict the row. A writer that affects no row writes no event and no metric. They never join the caller's transaction, because a lifecycle write reports on work that already happened: a rolled back caller cannot undo a transition, and an independent connection cannot see the caller's uncommitted job row, so a job must be committed before it can be claimed. GC deletes return resource ids via `RETURNING` inside one transaction.
|
||||
- `job`-substrate specifics (unified jobs): the lifecycle writers in `app.jobs` are `claim`, `retry-job`, `complete`, `fail` and `cancel`, and the dispatcher owns the transition to `scheduled` and the two abort sweeps (`lost` and `orphan`), and writes their events itself; each one is a conditional update plus its `job_event` row in the same transaction, so history can never contradict the row. A writer that affects no row writes no event and no metric. They never join the caller's transaction, because a lifecycle write reports on work that already happened: a rolled back caller cannot undo a transition, and an independent connection cannot see the caller's uncommitted job row, so a job must be committed before it can be claimed. GC deletes return resource ids via `RETURNING` inside one transaction.
|
||||
- Every public function of `app.jobs` takes a cfg map. A bare pool or connection is a caller bug and is not coerced: the substrate has no conversion of its own, and the ones in `app.db` are the db layer's business.
|
||||
- Every new foreign key of the substrate is `DEFERRABLE`, `job_event.job_id` included.
|
||||
- Some tables carry a SQL deletion guard (`raise_deletion_protection` + a `deletion_protection__tgr` statement trigger): `profile`, `team`, `team_font_variant`, `file_media_object`, `file_thumbnail`, `file_tagged_object_thumbnail` and `job`. A `DELETE` raises unless the transaction runs `SET LOCAL rules.deletion_protection TO off` (the guard is on when the GUC is unset, empty or `on`); see `app.tasks.objects-gc`, `app.jobs.gc` and `th/db-force-delete`.
|
||||
@ -82,14 +82,14 @@ For worker dispatch, cron, retry semantics (`ex/raise :type ::wrk/retry` with `:
|
||||
|
||||
## Jobs metrics
|
||||
|
||||
- `app.jobs.metrics` owns the jobs metric names, bounded labels and the periodic backlog sampler.
|
||||
- Metrics use only `name`, `queue`, `outcome`, `reason`, `stage` and `kind` labels. Job IDs, profile IDs, params, reply bodies and exception text must never be labels. Every label goes as-is (lowercased where case varies): values come from fixed call sites, so a new value in Prometheus means new code, not noise. Only a missing value falls back (`other`, or `failed` for outcomes).
|
||||
- `app.jobs.metrics` owns the jobs metric names and bounded labels. It has no gauge and no sampler: to see the current state, the operator runs one SQL query (`SELECT status, count(*) FROM job GROUP BY status`); the namespace runs no query and starts no thread.
|
||||
- Metrics use `name`, `queue`, `outcome`, `reason`, `stage`, `kind`, `action`, `type` and `result` labels, each on its own family. `queue` survives only on `penpot_jobs_dispatcher_batch_size`: of the measurements that used to carry it, every other one is now keyed by the job name, because the job-def owns its routing (`::jobs/queue-name`) and the name already determines the queue; the GC, cron, request, event and file-data-offload families keep their own dimensions. Job IDs, profile IDs, params, reply bodies and exception text must never be labels. Every label goes as-is (lowercased where case varies): values come from fixed call sites, so a new value in Prometheus means new code, not noise. Only a missing value falls back, and each label has its own default: `other` (name, queue, GC kind, offload type), `failed` (the total and batch outcomes, offload result), `error` (cron and request outcomes), `backoff` (retry reason), `execution` (dispatcher stage), `deleted` (GC action) and `none` (cron reason). The submit outcome has no fallback on purpose: its three values come from fixed call sites.
|
||||
- The canonical job name is a keyword (`app.jobs/schema:job-name`): `submit` and `invoke` reject strings. The row stores text, so `jobs/get-job-def` still coerces (it is the runner's boundary), cron entries carry the job under `:job` with `d/name` only at the SQL/log boundary, and test helpers keyword-ize row names before invoking.
|
||||
- The durable lifecycle metrics are submitted, dispatched, completed, retries, orphaned and rescheduled counters; queue-wait, execution and total-time histograms; dispatcher and backlog state; GC, cron and ephemeral request metrics.
|
||||
- The durable lifecycle metrics are one submit counter with `outcome` (`created`, `skipped`, `refreshed`), dispatched and retries counters; claim-wait, execution and total-time histograms; dispatcher batch timing and size; GC, cron and ephemeral request metrics. `penpot_jobs_dispatched_total` and the claim-wait histogram are keyed by the job name.
|
||||
- `app.metrics/run!` is safe for recording failures, but metrics components still require a valid metrics instance. Durable submit call sites carry `::mtx/metrics` through RPC or job-def configuration. The jobs substrate takes the **cfg**, not the instance, and resolves it with `app.metrics/instance`, which raises `:missing-metrics` when absent: `app.jobs`, `app.jobs.metrics`, the runner, the dispatcher, cron and the jobs GC all follow that. `app.http`, `app.rpc.climit` and `app.storage.s3.metrics` still take the instance and were left out of scope on purpose, so do not read the codebase as uniform. The six job writers resolve at the top of their body, before writing, because their metric lands in a `db/after-commit` callback that swallows exceptions. The GC, the dispatcher and cron resolve inside that callback instead, so their guarantee rests on the component schema instead. A job that calls `heartbeat` from a component cfg must carry `::mtx/metrics` on it, and the event helper records the counter from the cfg the transaction hands it.
|
||||
- SQL-backed job metrics are registered with `app.db/after-commit` from INSIDE the transaction that changed the row, so a rollback cannot inflate submitted, terminal, retry, event or dispatcher counters. Every lifecycle writer owns that transaction (`app.jobs/own-transaction` drops `::db/conn`), and `db/transact!` gives a transaction that joins no other its own callback context, so the write and its metric commit or vanish together. `submit` is the exception and joins the caller on purpose: creating a job is part of the caller's business work.
|
||||
- SQL-backed job metrics are registered with `app.db/after-commit` from INSIDE the transaction that changed the row, so a rollback cannot inflate submit, terminal, retry, event or dispatcher counters. Every lifecycle writer owns that transaction (`app.db/own-transaction` drops `::db/conn` and `::db/read-only`, so a write can own its transaction even inside a read-only caller), and `db/transact!` gives a transaction that joins no other its own callback context, so the write and its metric commit or vanish together. `submit` is the exception and joins the caller on purpose: creating a job is part of the caller's business work.
|
||||
- `penpot_jobs_events_total` counts stored job events with no labels: the interesting split is `kind`, and grouping a growing append-only table to build a metric is not worth the query on the write path.
|
||||
- The backlog sampler is `::jobs-metrics/sampler`; it runs every 30 seconds on worker-enabled, writable systems, groups by status and publishes the oldest pending age, and does not start on a read-only database.
|
||||
- The terminal counter per outcome is the `_count` of `penpot_jobs_total_ms` (`completed`, `failed`, `cancelled`, `aborted`), so there is no separate terminal or orphan counter: `_count{outcome="aborted"}` counts the system-side aborts: an expired orphan lease, or a `scheduled` job whose queue has nothing waiting after the grace or that passed the 24-hour cap. There is no backlog gauge and no pending-age gauge: the backlog is a SQL query, and `created_at` is not renewed by a debounce refresh, so an age gauge would lie.
|
||||
- `penpot_tasks_timing` remains exported for compatibility while the jobs-specific histograms are adopted.
|
||||
|
||||
|
||||
@ -128,8 +128,8 @@ Fixtures can populate local data for manual testing/perf work. From the backend
|
||||
|
||||
* **Type Hinting:** Use explicit JVM type hints (e.g. `^String`, `^long`) in performance-critical paths to avoid reflection overhead.
|
||||
* **Batch inserts:** Use `db/insert-many!` for bulk row inserts — generates a single SQL with multiple parameter tuples. Avoid on very large datasets (SQL length / parameter count limits).
|
||||
* **Server-side cursors:** Use `db/plan` (fetch-size 1000, forward-only, read-only) for large result sets. Never fetch large collections into memory at once. `db/plan` is an `IReduceInit`: `reduce`/`transduce` over it (`sequence` does not work on it), and with `partition-all` the transducer's completing arity is what flushes the last partial chunk. A plan row is a **view over the open result set**, valid only inside the reducing function: read the columns you need right there — buffering rows to read them later (a `partition-all` buffer, a lazy seq) silently yields nil columns. `db/cursor` (manual `DECLARE`/`FETCH`) is a leftover with no callers left.
|
||||
* **Transaction discipline:** Use `tx-run!` for writes (opens a transaction), `run!` for reads (single connection, no transaction). Set `:read-only` on `tx-run!` when applicable to let PostgreSQL optimize.
|
||||
* **Server-side cursors:** Use `db/plan` (fetch-size 1000, forward-only, read-only) for large result sets, always inside `tx-run!` (prefer `::db/read-only true`): the pool runs with auto-commit on, and the PostgreSQL JDBC driver only honors `fetch-size` (real streaming cursor) with auto-commit off, so a `plan` outside a transaction silently materializes the whole `ResultSet` in memory. Never fetch large collections into memory at once. `db/plan` is an `IReduceInit`: `reduce`/`transduce` over it (`sequence` does not work on it), and with `partition-all` the transducer's completing arity is what flushes the last partial chunk; the `reduce` holds one connection (and, inside `tx-run!`, the whole transaction) open until it finishes, so keep chunk transactions separate from the plan one when each chunk must commit on its own (the objects GC does). A plan row is a **view over the open result set**, valid only inside the reducing function: read the columns you need right there — buffering rows to read them later (a `partition-all` buffer, a lazy seq) silently yields nil columns. `db/cursor` (manual `DECLARE`/`FETCH`) is a leftover with no callers left.
|
||||
* **Transaction discipline:** Use `tx-run!` for writes (opens a transaction), `run!` for reads (single connection, no transaction) — except `db/plan`, which always needs `tx-run!` (see above). Set `:read-only` on `tx-run!` when applicable to let PostgreSQL optimize.
|
||||
|
||||
## Lint and Format
|
||||
|
||||
|
||||
@ -123,7 +123,7 @@ Since `put-object!` uses backend-specific operations (`impl/resolve-backend` + `
|
||||
- `db` stores encoded data in `file_data.data`.
|
||||
- `storage` stores encoded data in storage subsystem with `file-data` bucket, keeps `data` nil in `file_data` table, and points at the object with `file_data.storage_object_id`.
|
||||
- `file_data.storage_object_id` is a `NO ACTION DEFERRABLE INITIALLY IMMEDIATE` foreign key to `storage_object(id)`: the database refuses to physically delete a referenced object. The `IMMEDIATE` is load-bearing: the RI violation fires inside the `storage-gc-deleted` chunk transaction, so the chunk rolls back, is logged at error level with a generic hint plus the `constraint` and `table` the database reports (any `NO ACTION` foreign key on `storage_object` can block the delete, not only `file_data`'s) and is retried on the next run; with `INITIALLY DEFERRED` the check would fire at commit, outside the catch, and the sweep would never move past the blocked chunk.
|
||||
- The column is the only source of truth. The release is a coordinated restart (no rolling deploy) when the instance uses the storage backend (`PENPOT_FILE_DATA_BACKEND=storage`) or still has rows carrying `metadata.storage-ref-id`; with the default `db` backend and no such rows the previous and the new code read the same `file_data.data` column, so a rolling deploy is safe. `metadata.storage-ref-id` is no longer written or read, and the `file-data` metadata schema does not declare it. The migration backfills the rows it can resolve (valid uuid with an existing `storage_object`) and removes the key from those rows; a corrupt or orphan pointer keeps the key, which nothing reads. A large instance runs `operations/2.20.0-file-data-storage-object-id.md`: column, index and FK online, then the backfill (only if a row still carries `metadata.storage-ref-id`) with every instance stopped, verifies, and marks the migration as applied (`INSERT INTO migrations`). The backfill must never run while the previous code serves when the storage backend is in use (or rows with the pointer remain), because that code writes and scans the metadata pointer. A rollback to the previous code cannot read the rows written by this release when the storage backend is in use.
|
||||
- The column is the only source of truth. The release is a coordinated restart (no rolling deploy) when the instance uses the storage backend (`PENPOT_FILE_DATA_BACKEND=storage`), has `:tiered-file-data-storage` on or pending `offload-file-data` tasks (that task writes storage rows with the default `db` backend), or still has rows carrying `metadata.storage-ref-id`; with the default `db` backend, the tiered offload off, the queue empty and no such rows the previous and the new code read the same `file_data.data` column, so a rolling deploy is safe. `metadata.storage-ref-id` is no longer written or read, and the `file-data` metadata schema does not declare it. The automatic migration and the large-instance runbook both backfill only valid UUIDs with an existing `storage_object`, set `storage_object_id` and remove the old key in the same update. Corrupt and orphan pointers are not modified and must be resolved before the unreadable-row check passes. The migration runs as one statement; the large-instance runbook uses batches of 5,000 and a temporary partial index on each partition for rows with the metadata key and a NULL column pointer. A successful batch removes its rows from that index; drop the parent index after the backfill. The backfill runs with every instance stopped because old code writes and scans the metadata pointer. The key is not retained as a rollback copy: rollback to the previous binary after backfill requires restoring the backup. The unified runbook `operations/2.20.0-unified.md` pre-registers all migration steps at the start, then runs every manual procedure before deploying 2.20.0.
|
||||
- `fdata/upsert!` touches the `storage_object_id` it receives in the params (the pointer the persisted row had) before replacing it, so the replaced blob is reclaimed. Callers pass the column from the file map (main/restore), `nil` for a new row (fragments, snapshots, new files) or the row itself (offload); there is no query in the upsert path, which is why every caller reads the file under the row lock first (the migration-on-read path included, see `mem:backend/subtleties`). It returns `{:written <bool> :storage-object-id <uuid or nil>}`, the pointer of the object the write created (nil for `db`/`legacy-db`), and `binfile.common/update-file!` propagates it so a migrating read returns the map with the pointer of the row it persisted instead of the pre-write one. The write also compares the expected pointer with `IS NOT DISTINCT FROM` (the caller's value, not the new one) and raises `:stale-file-data-pointer` when the row changed under it, so a stale map fails instead of leaking the replaced blob; the check lives in `upsert-in-database` because the `legacy-db` branches do not use the upsert SQL.
|
||||
- `binfile.common/update-file!` takes the file map read from the database plus the fields to change (`[cfg file & params]`), merges them and builds the `file_data` params from the merged map, so the pointer travels in the read map and a caller cannot lose it by building a partial map; the params cannot override `:storage-object-id`. `insert-file!` always drops it (an INSERT never replaces a persisted pointer; `duplicate-file` hands the source map there). `bfc/get-file` keeps the pointer for the write paths, and every response goes through `bfc/file->response`, which strips it and any residual `storage-ref-id` left in the metadata of the rows the migration does not backfill: each `file-data` object is authentication-only and any logged-in profile could otherwise fetch it from `/assets/by-id`.
|
||||
- The in-place binfile import was removed: `import-files!` raises `:not-supported` when the cfg carries `::bfc/file-id`, so no caller can overwrite a storage-backed row without carrying its previous pointer.
|
||||
|
||||
@ -30,22 +30,24 @@
|
||||
- Whether a call is nested is decided by the CONNECTION, not by an ambient flag: `db/transact!` binds `*after-commit-conn*` around the body, and a transactable that is not identical to that connection is an independent unit of work that owns its own after-commit context. A `tx-run!` that passes a pool while a transaction is open therefore commits on its own and drains its own callbacks, instead of leaving them in a context that may roll back.
|
||||
- `after-commit` callbacks run on commit only. A transaction marked `::db/rollback` (used by `srepl` and `app.tasks/offload-file-data`) commits nothing, so it runs none of them.
|
||||
- `db/run!` opens/reuses one connection but does not create a transaction.
|
||||
- `db/own-transaction` returns a cfg that owns its next transaction: it drops `::db/conn` (so `tx-run!` opens its own) and `::db/read-only` (so a caller running read-only, like the storage scheduler streaming a plan cursor, cannot make the nested write read-only too). Lifecycle writes use it so a rollback of the surrounding work can undo neither the write nor its metric.
|
||||
- `db/tjson` is Transit JSON for jsonb storage; `db/json` is plain JSON. Job params (`job.params`) are plain JSON, decoded with the job-def decoder (`app.jobs/decode-params`); only the legacy `task` props were Transit (`decode-transit-pgobject`). Job results (`job.result`) are plain JSON too: they are read with the result decoder of the job-def (`app.jobs/decode-result`), and a result map keyed by uuid must declare its schema (`[:map-of ::sm/uuid ...]`, the import resolution does) or every key comes back as a keyword and every uuid as the text it is on disk.
|
||||
- Advisory transaction locks accept UUIDs or ints. UUID locks are hashed using a zero-UUID seeded siphash.
|
||||
- Advisory locks (`app.db`) come in two independent axes. **Scope**: the transaction ones (`xact-lock!`, `xact-try-lock!`) vanish with the transaction, and the session ones (`session-try-lock`, `session-unlock`, `session-unlock-all`) live until unlocked or the session ends — a pooled connection returned with one keeps it. **Form**: the one-key form takes a uuid or an int (uuids hashed with a zero-UUID seeded siphash) and the two-key form takes a `classid`/`objid` pair; PostgreSQL keys them by `objsubid` (1 vs 2), so the two spaces never conflict even with the same numbers. `update-file` and the setup take the one-key form; the audit consumers take the two-key one with `classid` `730000` and the consumer id as `objid`, which is why a consumer tick can never block a file update, and they clear whatever a borrowed connection carries before taking it.
|
||||
|
||||
## Workers and cron
|
||||
|
||||
- `job` carries `tenant` (the instance the job belongs to) and a bare `queue`. Several instances share one database and the tenant is what keeps an instance from claiming another environment's jobs, so every query that acts on one instance's work filters on `tenant`: dispatch claim, orphan sweep, lost-job reschedule, cron no-overlap count, submit dedupe. The `job__dispatcher__idx` leads with `tenant` (equality) then `(priority DESC, scheduled_at)`; `queue` does not lead because the dispatcher claims every queue of its tenant. The `<tenant>:<queue>` prefix survives only in the Redis key, composed once by `app.worker/queue-key`.
|
||||
- Submit dedupe only removes not-yet-due `new` jobs of the same tenant with the same name/queue/label; it does not dedupe due, scheduled, retry, running, or completed work. The `::jobs/skip-if-active` submit option goes further: it skips the insert (returning `nil`, recording `penpot_jobs_skipped_total`) when a live job (`new`/`scheduled`/`running`/`retry`) already exists for the same tenant/name/queue/label, requires an explicit non-blank label, and carries the same best-effort cross-backend race caveat as dedupe.
|
||||
- Jobs metrics rule (one event, one family): counters exist only for events without latency; when an event has a histogram, its `_count` is the counter (no twin counter). `dispatched` (pushes) vs `queue-wait` (claims) are deliberately different events — do not merge them.
|
||||
- The dispatcher selects `new`/`retry` jobs with `FOR UPDATE SKIP LOCKED`, marks them `scheduled`, and publishes the JSON payload `[id scheduled-at]` to the `penpot.worker.queue:<tenant>:<queue>` Redis list. The runner skips Redis messages whose scheduled timestamp no longer matches DB state.
|
||||
- Lost `scheduled` jobs are rescheduled after 5 minutes; `running` jobs untouched longer than `:jobs-lease` (default 30 min) are marked `aborted` with `app.jobs/orphan-error` (`type :internal`, `code :orphan`). `aborted` is a system-side terminal state like `cancelled` (never retried, swept by retention like other terminals, reported with an error log); long jobs must `heartbeat` per chunk so a healthy backlog is never mistaken for a dead process. The `:progress` report is optional and only feeds the durable log. A beat whose write finds the job no longer active raises an `:interrupt` (`code :job-interrupted`) instead of counting zero: it is the internal stop-the-work signal, not an error, so neither the listener nor the runner logs it as one.
|
||||
- An orphan gets its `end` event (`outcome "aborted"`) from the sweep itself: one bulk `INSERT ... SELECT` in the same transaction as the bulk `UPDATE`, never one insert per orphan. No msgbus notification goes out for it; orphans alert through the error log and the orphaned counter.
|
||||
- `job` carries `tenant` (the instance the job belongs to) and a bare `queue`. Several instances share one database, and the tenant is what keeps an instance from claiming another environment's jobs: every query that acts on one instance's work filters on it (dispatch claim, lost sweep, orphan sweep, cron submit), and the tenant is part of the key of the dedupe and exclusivity indexes of `submit` (the cron no-overlap check is one of those inserts).
|
||||
- Two submit options resolve their key in the database instead of reading before writing, each through its own partial unique index over (tenant, name, queue, label). `::jobs/skip-if-active` inserts the row as **exclusive** and `job__live_exclusive__uniq` admits a single live row (`new`/`scheduled`/`running`/`retry`). `::jobs/dedupe` inserts it as **deduped** and `job__new_dedupe__uniq` admits a single not-started row (`new`) whatever its `scheduled_at`; a conflicting insert affects no row, the submit returns `nil` and records `penpot_jobs_submit_total{outcome="skipped"}`. The pending row is never deleted nor modified: it runs when it is due, with its own params. `::jobs/debounce` uses the same key and index but resolves the conflict with `ON CONFLICT ... DO UPDATE`, refreshing `scheduled_at` (to `now + delay`), `params` and `modified_at` of the pending row and answering with the id it already had: that is the sliding window of the webhook batches, where the last payload is the one delivered. A refresh is not a new job: the submit records `outcome="refreshed"`, apart from the `created` ones, and the id it reports is the id the statement wrote. Dedupe and debounce are mutually exclusive. Both require an explicit non-blank label and only cover the jobs that asked for the option: a live job that was not submitted with it (a webhook delivery, a notification) does not block a flagged submit, and two plain jobs can share a label. The `exclusive` and `dedupe` columns are what keep that distinction; the label stays a label.
|
||||
- A single insert carries a single `ON CONFLICT` clause, so `skip-if-active` cannot be combined with `dedupe`/`debounce`: an exclusive job already admits one live row, PostgreSQL admits one conflict target per statement, and the submit raises `:skip-if-active-cannot-be-combined`.
|
||||
- Jobs metrics rule (one event, one family): counters exist only for events without latency; when an event has a histogram, its `_count` is the counter (no twin counter). `dispatched` (pushes) vs `claim_wait` (claims) are deliberately different events — do not merge them. `claim_wait` measures `now - scheduled_at` at the claim, so it is how late the claim happened, including the dispatcher gap, not time inside a queue. `dispatcher_batch_ms` times one dispatcher pass and pairs with `dispatcher_batch_size`; `gc_pass_ms` times one GC pass (`expired` or `retained`).
|
||||
- The dispatcher claims `new`/`retry` jobs atomically, marks them `scheduled` and hands them to their queue; the runner skips a message whose scheduled timestamp no longer matches the database state (a rescheduled or reclaimed job).
|
||||
- A job handed to a queue (`scheduled`) is marked `aborted` with `app.jobs/lost-error` (`code :lost`) when its queue has nothing waiting (`LLEN`) and it has been `scheduled` longer than the grace (5 min; it also covers the pop-to-claim gap of a message popped right after the hand-off, but not one that waited longer in the queue and is popped while the sweep runs: the round trip can lose that race), or when it has been `scheduled` past the cap (24 h, for a message lost behind a backlog or a queue without a consumer; a daily cron can miss one run). Leftover rows are never re-dispatched, so nothing returns to `new` and the dispatcher cannot violate `job__new_dedupe__uniq` (the thresholds are `lost-grace` and `lost-cap`, and `lost-batch-size` bounds each read in `app.worker.dispatcher`: past-cap rows are aborted without an `LLEN` and grace rows with one, so a younger lost row behind a large legitimate backlog may wait for the cap instead of the grace). `running` jobs untouched past `:jobs-lease` are marked `aborted` with `app.jobs/orphan-error` (`type :internal`, `code :orphan`). `aborted` is a system-side terminal state like `cancelled` (never retried, swept by retention like other terminals, reported with a log line: `warn` for lost, `error` for orphan); long jobs must `heartbeat` per chunk so a healthy backlog is never mistaken for a dead process. The `:progress` report is optional and only feeds the durable log. A beat whose write finds the job no longer active raises an `:interrupt` (`code :job-interrupted`) instead of counting zero: it is the internal stop-the-work signal, not an error, so neither the listener nor the runner logs it as one.
|
||||
- Both abort sweeps (lost and orphan) write the `end` event (`outcome "aborted"`) in the same transaction as the status change, never one insert per row. No msgbus notification goes out for them; the aborted rows alert through the log and `penpot_jobs_total_ms_count{outcome="aborted"}`, which counts both causes (there is no separate counter).
|
||||
- `job_event` is append-only and deleted by cascade with the job. `progress` is a row there, not a column: no Redis and no mutable column to keep in sync.
|
||||
- `job` is deletion-protected at the SQL level (`deletion_protection__tgr`): a stray `DELETE FROM job` raises instead of dropping the row and its `job_event` cascade without touching the referenced `storage_object`. The only intended deleters, the jobs GC and the submit dedupe, disable the guard with `SET LOCAL rules.deletion_protection TO off` inside their transaction; tests use `th/db-force-delete`.
|
||||
- `job` is deletion-protected at the SQL level: a stray `DELETE FROM job` raises instead of dropping the row and its `job_event` cascade without touching the referenced `storage_object`. The intended deleter is the jobs GC, which disables the guard inside its transaction; tests use `th/db-force-delete`. `app.jobs/submit` never deletes: its dedupe is a conflict of the insert.
|
||||
- A missing job-def raises (`:no-job-definition`) instead of completing. Throwing with `ex-data :type ::retry` still controls retry behavior; `:strategy ::noop` retries without incrementing retry count.
|
||||
- Cron entries claim their `scheduled_task` row with `FOR UPDATE SKIP LOCKED`, disable statement/idle-in-transaction timeouts locally, submit one `job` row per entry when no active instance exists (no-overlap), and reschedule themselves in `finally` unless interrupted. Worker, dispatcher, and cron components do not start when the DB pool is read-only.
|
||||
- Integrant params schemas name injected components as bare `::ns/key` entries (pool, metrics, msgbus, job defs), and a bare entry is a malli registry lookup, not `any?`: the owning namespace must `sm/register!` it at load, or the backend crashes at boot with `:malli.core/invalid-schema` instead of asserting. Unit tests never build those keys, so `backend-tests.worker-test` compiles the three worker schemas and `backend-tests.http-websocket-test` compiles the revocation-watcher one to catch it. The assertion only runs with `PENPOT_FLAGS=enable-backend-asserts`, so without it a broken schema reaches production and only crashes at boot.
|
||||
- Cron entries claim their `scheduled_task` row with `FOR UPDATE SKIP LOCKED`, disable statement/idle-in-transaction timeouts locally, and submit one exclusive `job` row per entry when no live instance exists. The database's partial unique index decides overlap across backends, not a count. Entries reschedule themselves in `finally` unless interrupted. Worker, dispatcher, and cron components do not start when the DB pool is read-only.
|
||||
- Integrant params schemas name injected components as bare `::ns/key` entries (pool, metrics, msgbus, job defs), and a bare entry is a malli registry lookup, not `any?`: the owning namespace must `sm/register!` it at load, or the backend crashes at boot with `:malli.core/invalid-schema` instead of asserting. Unit tests never build those keys, so `backend-tests.worker-test` compiles the worker schemas and `backend-tests.http-websocket-test` compiles the revocation-watcher schema to catch it. The assertion only runs with `PENPOT_FLAGS=enable-backend-asserts`, so without it a broken schema reaches production and only crashes at boot.
|
||||
- Whether a bare entry works has nothing to do with its position or with how many of them there are: it is a registry lookup, so it works only if that exact keyword was `sm/register!`'d. `[:map ::db/pool ::metrics/metrics]` is fine, `[:map :app.aaa/unregistered :app.ccc/unregistered]` throws. When a component is **not** registered as a malli type, spell it as an explicit pair instead of registering a type just to satisfy the lookup — `[:app.nitrate/client {:optional true} [:maybe :map]]` is how `auth.oidc/schema:routes-params` and `loggers.audit.archive-task/schema:audit-log-archive-params` declare it. `{:optional true}` matters because `init-key` yields `nil` when the owning feature flag (`:admin-console`) is off, and a required key would then fail the assert on a perfectly normal boot.
|
||||
- `sm/valid?` catches every `Throwable` and returns `false`, so an uncompilable schema and a genuinely invalid value both surface as a bare `Assert failed:` with no cause. When an `assert-key` fires with no message, compile the schema by hand to see the real exception.
|
||||
|
||||
@ -85,6 +87,7 @@
|
||||
## Storage and media
|
||||
|
||||
- The `media-storage-bytes-per-team` quota check on `upload-file-media-object` is TEMPORARILY DISABLED (2026-10-19, kept commented in `app.rpc.commands.media` with its `sql:get-team-id-for-file` lookup still active for the audit props): the quote performs a full `SUM` over `storage_object` per upload and was too slow in production. The `upload-sessions-per-profile` quota is unaffected and stays active. Re-enable only with a cheaper approximation.
|
||||
- `file-gc` trims emit `file-gc` with the cleaned `:file-id` and its `:team-id`; `delete-object-with-cascade` carries flat `:team-ids` next to the nested `:cascade` map. A consumer can key on those props without joining production tables.
|
||||
- Storage abstraction, backend configuration, logical buckets, object lifecycle, deduplication, access rules, and garbage collection: `mem:backend/storage`.
|
||||
- SVG validation strips DOCTYPE and uses secure SAX parsing. Basic SVG info falls back to 100x100 dimensions when width/height/viewBox are missing.
|
||||
- The SVG sanitizer (`app.media.svg`) is a **blocklist**, not an allowlist: it drops dangerous elements and attributes by name and escapes every text and attribute value when it writes the tree back, so a payload hidden in a value cannot become live markup again. The allowlist rework is [#12190](https://github.com/penpot/penpot/issues/12190) and lands in media-processor, not here. `sanitize-svg` also tracks the namespace bindings through the recursion and drops any element that is not in the SVG namespace, so a document whose root is not an `svg` element in the SVG namespace is rejected and a child that rebinds `xmlns` to XHTML (`iframe`/`srcdoc`/`meta`) is dropped with its subtree; it compares the scheme of an URL the way the browser does (dropping tab/CR/LF/spaces), so `java	script:` does not slip through the `javascript:` filter. The paths that store an uploaded SVG as its final object go through `app.media.svg/sanitize-svg-upload` (media upload, the URL download, both thumbnail writers and the organization logo); the three binfile imports go through `app.binfile.common/sanitize-imported-svg`, which calls `sanitize-svg` directly. The profile picture is not an SVG: `import-profile-picture` ignores an `image/svg+xml` download, because the avatar is always rendered as an image. The caller checks the size before calling it: the helper reads the whole file, and the size limit applies to the upload, not to the sanitized copy, which can be up to six times bigger.
|
||||
@ -112,7 +115,7 @@
|
||||
- File snapshots (versions) keep their metadata row in `file_change` and their payload in `file_data` (`type = 'snapshot'`). With the `db` backend `file_change.data` stays NULL: that column is only read by the `legacy-db` backend (it is no longer written), plus the rows written before the 2025-07 file storage refactor.
|
||||
- Tell a snapshot apart by `label IS NOT NULL` (with `created_by` / `deleted_at` as needed), never by `file_change.data IS NOT NULL`: the label holds with every file data backend, and the payload column stopped meaning "this is a snapshot" when the payload moved out. Queries that identify versions must use the label. `0157-migrate-legacy-file-data` moved the labelled leftovers to `file_data` and deleted the unlabelled payload rows, so from 2.19 on the label is the only marker there is.
|
||||
- The version quota counts of `app.rpc.quotes` (`sql:get-snapshots-per-file`, `sql:get-snapshots-per-team`) are served by the partial index `file_change__user_snapshots__idx`, whose predicate mirrors their where clause; changing one without the other drops the index only scan and the count walks the whole table again.
|
||||
- Revisions live only in `file_change` and have no `file_data` counterpart. They are born already scheduled for deletion one hour later (`app.rpc.commands.files-update`) and the snapshot paths cannot reach one, because they resolve their row through a `label IS NOT NULL` filter.
|
||||
- Revisions live only in `file_change` and have no `file_data` counterpart. They are born already scheduled for deletion one hour later (`app.rpc.commands.files-update`) and the snapshot paths cannot reach one, because they resolve their row through a `label IS NOT NULL` filter. The revision row created by a save travels to the audit event in `:file-change-id` (a consumer reads it with `::db/remove-deleted false`, because the row is born with its `deleted_at` in the future); a save writes two rows, the revision (no `label`) and the snapshot of the previous state (with one).
|
||||
- A deletion date is assigned once and never moved: `restore-files` (`app.rpc.commands.files`) unmarks only durable history, which is `label IS NOT NULL` (the unlabelled rows that still carry a legacy payload are removed by `0157-migrate-legacy-file-data`, so restore leaves them dated), the `:file` method of `app.tasks.delete-object` dates only rows that have no date yet, and `objects-gc` collects only rows that carry one. Unmarking a revision leaves it invisible to `objects-gc` for ever, because nothing sets that date again.
|
||||
- Binfile v1/v3 import and the v3 export `get-file` step run inside one long transaction. They raise `idle_in_transaction_session_timeout` to 30 min (`app.binfile.common/transaction-idle-timeout-ms`) instead of disabling it, so a connection left idle while the ZIP is decoded or the file data is realized is bounded; `statement_timeout` stays at 300 s.
|
||||
- `update-file` with `:changes-with-metadata` groups that all carry `:commit-id` is partial: fast path applies and validates everything once; on a change failure it bisects the commits for the longest prefix that applies and validates (each attempt applies a prefix from the original data and runs referential integrity + file schema; ~log2 K attempts, none for a single commit, the first attempt's error reused), keeping that prefix (a commit that repairs an earlier one can keep both) and answering 200 with `:rejected {:commit-ids :failed-commit-id :error}`, plus `:preexisting-damage true` when the data the first rejected commit touches already fails validation without it (`preexisting-damage?`: one more referential + schema check of its pages/components on the base file, the accepted prefix or the original; the frontend then says the page data is damaged). Only accepted changes reach `file_change`, persistence and the msgbus; a fully rejected request writes and broadcasts nothing (revn unchanged). Flat `:changes` or groups without ids stay all-or-nothing. Before each attempt (`apply-prefix`, which returns `{:file :tracked :refs}` or the cause) `pmap/*tracked*` and the `cpc/*state*` atom are reset to their pre-attempt values (at the end `apply-commits` sets those of the kept prefix): a pointer map already `modified?` keeps its id when changed again, so the failed version would be persisted. DB/IO/interrupt errors and `:not-found` fragments (`change-failure?`) fail the whole request, never reject a commit. A rejection whose ex-info has no `:type` reports `:internal :unexpected-error`. `app.http.errors` answers 503 (`:server-error` + code) for transient SQLSTATEs (classes 08/53, 57P01–3, 40001, 40P01), `SQLTransientException`, `InterruptedException` and only the `IOException`s that pass on their own (`transient-io-exception?`: `SocketException`, `InterruptedIOException` incl. socket timeouts, `UnknownHostException`, `HttpTimeoutException`), which the client's save retries (and GETs retry by themselves); any other `IOException` (EOF from a corrupt blob, Jackson parse errors, missing files) stays 500 `:io-exception`, as do other SQL errors. With the `:debug-save-faults` flag, a `:set-plugin-data` file change in namespace `:penpot-debug` keyed `save-fault:<kind>` raises that fault in `apply-changes` (used by `debug.failNextSave`/`commitInvalidChange`). With a schema check running (hard schema validation without skip-validate, or soft) and 2+ commits, the touched pages' objects are decoded on the original file first (`decode-objects`; referential validation alone reads few objects, so no pre-decode then): objects-map caches decoded values per instance and a derived map copies the cache, so otherwise every attempt re-decodes the page (~260 ms at 10k shapes). Validation step timings (`with-validation-timing`, upstream `penpot_rpc_update_file_validation_timing` + the `update-file` debug log) come from the first attempt only (retries run with a cfg without metrics and timings); the whole bisection is one `retry` step sample, and the rejection log carries `:attempts`/`:elapsed`.
|
||||
|
||||
@ -12,26 +12,14 @@ JVM `clojure.test` (kaocha runner) under `backend/test/backend_tests/`.
|
||||
- Coverage: if code is added or modified in `src/`, corresponding tests in `test/backend_tests/` must be added or updated.
|
||||
- Isolated run: `clojure -M:dev:test --focus backend-tests.my-ns-test` for a specific test namespace, or `clojure -M:dev:test --focus backend-tests.my-ns-test/my-test-var` for a specific test var. Keep the `PENPOT_FLAGS="enable-backend-asserts"` prefix on these too; a focused run without it hides assert-dependent failures exactly like a full one.
|
||||
- A **whole-suite** crash reported as `Execution error ... Assert failed: ...` with no `FAIL in` line is not a test failure: an `ig/assert-key` ran while the test system was being built, which happens outside `clojure.test`, so kaocha reports it as a load/build error and the summary has no counts. Fix the offending schema (see `mem:backend/subtleties`) rather than hunting for a test.
|
||||
- **Log lines are not tested.** A log record is for the operator, not a contract: do not add a test that captures it (`add-watch` on `app.common.logging/log-record`, or `with-redefs` on `enabled?`/`emit-log`) to assert its message, hint or props, and do not keep such a test alive when the line changes. Whatever must be guaranteed belongs in a returned value, an event or a metric, and the test asserts that; a log line may change its wording at any time.
|
||||
- Multiple focus symbols are supported by REPEATING the flag (kaocha accumulates them, last-wins does not apply): `clojure -M:dev:test --focus backend-tests.ns-one --focus backend-tests.ns-two` — preferred over a shell loop of isolated runs.
|
||||
- Regression run: `clojure -M:dev:test` to ensure no regressions in related functional areas.
|
||||
- Line coverage: append the `:coverage` alias (`clojure -M:dev:test:coverage`, kaocha-cloverage). It instruments every `app.*` namespace under `src/` at load, before any test runs, so even a `--focus` run instruments them all; options live under `:cloverage/opts` in `tests.edn`, output in `target/coverage/` (lcov, HTML, and EMMA `coverage.xml` whose form counts the PR report shows as statements; cloverage measures no functions or branches) (`lcov.info`, HTML). About 2x slower and noisy with reflection warnings (instrumented forms lose type hints): never use it for normal runs. `tests-backend.yml` runs it and renders the step summary with `scripts/coverage-summary.mjs`.
|
||||
- If you need to filter output, tee to a temp file first: `clojure -M:dev:test 2>&1 | tee /tmp/penpot-test-output.txt`.
|
||||
- RPC test helpers `command!`/`management-command!` split the data map: qualified keys become server params, unqualified keys become request body params. To inject request-level context (headers, `:app.http/auth-key-id`, ip), pass a map under `:app.http/request` metadata; non-map `IRequest` stubs fall back to a dummy request.
|
||||
- The test system is `main/system-config` + `main/worker-config` with the
|
||||
components that must not run in the background dropped: HTTP, the srepl,
|
||||
cron, the dispatcher and **every** `[<profile> :app.worker/runner]`
|
||||
(`helpers/remove-job-runners`). A live runner of any queue races the tests
|
||||
that drive the loop by hand (`wdisp/run-batch` + `@#'wrkr/run-worker-loop`):
|
||||
it takes the job the test just dispatched and completes it behind the test's
|
||||
back, so the test's own assertions read a state it did not produce. Do not
|
||||
re-add a runner by editing that list by hand; the removal matches by key
|
||||
shape on purpose.
|
||||
- `system-config` and `worker-config` are `def`s: their `(cf/get ...)` calls
|
||||
are evaluated when the namespace loads, against the real environment, while
|
||||
test code calling `cf/get` at run time sees the `penpot-test`-prefixed
|
||||
config. The two can disagree — in the devenv `:tenant` is `devenv-ws0` for
|
||||
the defs and `default` inside a test — so a component built from
|
||||
`worker-config` can talk to a different Redis key prefix than a cfg the test
|
||||
builds itself. The symptom is a component that silently does nothing
|
||||
locally and races in CI (which sets no `PENPOT_TENANT`).
|
||||
- `backend-tests.helpers` has `metrics-stub` (a metrics instance with no collectors, where every `run!` is a no-op), `init-climit` (a concurrency limiter built from an inline config map, with an optional second argument for the metrics instance to publish to) and `with-climit`, the macro that builds one, runs the body and halts it. `init-climit` starts the republisher daemon thread, so a test that builds a limiter without halting it leaks that thread for the rest of the JVM run: use `with-climit`. Log lines are capturable with `add-watch` on `app.common.logging/log-record` (it holds the last record; the watcher sees every one, but only of the levels the logger has enabled: a line the app writes at TRACE never gets there). The logger swaps the record on an executor thread, after the call that logged it returns: keep the watch installed until the record arrives (poll for it) and only then remove it, or the record lands on nobody under load.
|
||||
- The test system is `main/system-config` + `main/worker-config` with the components that must not run in the background dropped: HTTP, the srepl, cron, the dispatcher, the three audit-log consumer loops (`:app.loggers.audit.consumers.nexus/consumer`, `:app.loggers.audit.consumers.admin-console/consumer`, `:app.loggers.audit.consumers.telemetry/consumer`) and **every** `[<profile> :app.worker/runner]` (`helpers/remove-job-runners`). A live runner of any queue races the tests that drive the loop by hand (`wdisp/run-batch` + `@#'wrkr/run-worker-loop`): it takes the job the test just dispatched and completes it behind the test's back, so the test's own assertions read a state it did not produce. A live consumer loop races the tests the same way: it drains `audit_log` in the background. Do not re-add any of them by editing that list by hand; the runner removal matches by key shape on purpose.
|
||||
- `th/run-task!` invokes a job by name; audit-log consumers are not jobs. Call `(consumers/run-pass cfg consumer)` with the prepared cfg (`::db/pool`, `::mtx/metrics` and transport deps) and the transport's `consumer` definition. A successful pass returns nil and publishes totals as metrics; `::consumers/busy` and `::consumers/disabled` mean no work started.
|
||||
- `app.loggers.audit/submit` propagates its failure (`submit*` no longer swallows it), so a DB-free unit test that reaches a code path emitting an audit event must stub it (`with-redefs [app.loggers.audit/submit (constantly nil)]`, as the `helpers.clj` fixture does globally) or pass a cfg carrying `::db/pool`. Otherwise the throw detours the flow under test into its error path.
|
||||
- `system-config` and `worker-config` are `def`s: their `(cf/get ...)` calls are evaluated when the namespace loads, against the real environment, while test code calling `cf/get` at run time sees the `penpot-test`-prefixed config. The two can disagree — in the devenv `:tenant` is `devenv-ws0` for the defs and `default` inside a test — so a component built from `worker-config` can talk to a different Redis key prefix than a cfg the test builds itself. The symptom is a component that silently does nothing locally and races in CI (which sets no `PENPOT_TENANT`).
|
||||
- `backend-tests.helpers` has `metrics-stub` (a metrics instance with no collectors, where every `run!` is a no-op), `init-climit` (a concurrency limiter built from an inline config map, with an optional second argument for the metrics instance to publish to) and `with-climit`, the macro that builds one, runs the body and halts it. `init-climit` starts the republisher daemon thread, so a test that builds a limiter without halting it leaks that thread for the rest of the JVM run: use `with-climit`.
|
||||
|
||||
@ -2,9 +2,9 @@
|
||||
|
||||
How to shape a function in this codebase. Each rule came from a review round; do not re-derive from taste.
|
||||
|
||||
- **HARD RULE: no new `!` names.** Do not introduce a function or macro ending in `!`; side effects alone do not justify the suffix. Allowed exceptions, and only these: mutating public fns in `app.db` (`insert!`, `update!`, `delete!`, `run!`, `tx-run!`, ...) and fns whose contract is to work over transients (stdlib `conj!`/`assoc!`/`dissoc!`/`persistent!` and custom helpers over them). Add a new exception to `mem:clojure/idioms` with its justification **before** using it. Existing `!` elsewhere (e.g. `emit!`, `send!`, `submit!`, `validate-file!`) stays until its refactor, so a family stays consistent meanwhile; that is a reason to keep an old name, never to add a new one. Reporting helpers (log, sample-and-publish) never take `!`: the mutation happens in the `!` API they call.
|
||||
- **Preconditions belong to the boundary, not to the core.** A function documents what it needs (hints, docstring) and assumes it. Absence checks live where the value enters the system: the caller that resolves optional objects, `ig/init-key`, the request handler — including `nil` that is optional-by-design (`when-let [cs (some-> server ...)]` at the entry point, as with `ConnectorStatistics` when the server option is off). A `(when (and (some? a) (some? b)) ...)` inside a leaf function means the guard is in the wrong place — if you cannot get `a` or `b`, you should not be calling it.
|
||||
- **A static set of operations is written statically.** With a fixed, small set (four metrics) write the four calls. Do not build a collection and iterate it (`doseq` over a literal vector): it allocates, adds indirection and hides each operation. Use a collection when the set is dynamic (config, registry, input).
|
||||
- **Default to no `!`; existing `!` is legacy being removed gradually.** Do not introduce new `!` names except the allowed cases documented in `mem:clojure/idioms` (add and justify the special case there first). Allowed: mutating public fns in `app.db` (`insert!`, `update!`, `delete!`, `run!`, `tx-run!`, ...) and fns whose contract is to work over transients (stdlib `conj!`/`assoc!`/`dissoc!`/`persistent!` and custom helpers over them). Existing `!` elsewhere (e.g. `emit!`, `send!`, `submit!`, `validate-file!`) stays until its refactor; keep a family consistent in the meantime. Reporting helpers (log, sample-and-publish) never take `!`: the mutation happens in the `!` API they call.
|
||||
- **Functions that obtain data start with `get-` or `fetch-`.** A reader must see at a glance that the function reads (`get-file`, `get-snapshot`, `get-file-json`); a bare noun hides it (`file-json` says nothing about reading). New helpers that fetch by id follow this, never a `current-` prefix: the id parameter already says which one it is.
|
||||
- **`?` marks a predicate, not a data prop.** A var or function that answers a question takes `?` (`export-error?`, `(defn editable? …)`); a map key that names a property does not (`:export-error`, never `:export-error?`), even when its value is a boolean. Existing `?` keys are legacy to remove gradually; do not add a new one.
|
||||
- **Do not shape production code for tests.** No return values added just to assert them; assert on the observable effect. When preconditions move, move the tests with them.
|
||||
|
||||
@ -15,7 +15,7 @@ Compose-based dev environment under `docker/devenv/`, driven by `manage.sh`. Par
|
||||
- For ws1+, `instance-env-overrides` computes the per-instance overrides (container/volume names, host ports offset `10000·N`, `PENPOT_PUBLIC_URI`, `PENPOT_REDIS_URI`, `PENPOT_TEST_DATABASE_URI`, `PENPOT_TEST_REDIS_URI`) and `instance-compose` injects them as env vars at compose time — never written to disk, recomputed each call so they can't drift. ws0 uses `defaults.env` as-is.
|
||||
- Per-instance test databases are created by `ensure-instance-test-database` (called from `start-instance`): `CREATE DATABASE` guarded by a `pg_database` existence check, so it is idempotent. Needed because `postgresql_init.sql` only runs on first volume creation. The backend applies migrations itself on first use, so an empty database is enough.
|
||||
- `backend/scripts/_env`: backend-internal only — secret keys, `PENPOT_FLAGS` (with `enable-backend-worker` gated on `PENPOT_BACKEND_WORKER`), `JAVA_OPTS`, `setup_s3_bucket()`. Never duplicates `defaults.env`.
|
||||
- Frontend admin-console is off by default in devenv: `docker-compose.main.yml` sets `PENPOT_DEVENV=true`, which makes `app.config` (`goog-define devenv-mode`) prepend `disable-admin-console` ahead of the user `config.js` flags; flags parse in order and user flags win, so `enable-admin-console` in `frontend/resources/public/js/config.js` re-enables it. Outside devenv (prod builds) the default is unset and the admin-console stays on, matching `flags/default`.
|
||||
- Frontend admin-console is off by default in devenv: `docker-compose.main.yml` sets `PENPOT_DEVENV=true`, which makes `app.config` (`goog-define devenv-mode`) prepend `disable-admin-console` ahead of the user `config.js` flags; flags parse in order and user flags win, so `enable-admin-console` in `frontend/resources/public/js/config.js` re-enables it. Outside devenv (prod builds) the default is unset and the admin-console stays on, matching `flags/default`. The backend `_env` also adds `disable-admin-console` and keeps `enable-nexus` commented out, so both cursor consumers are off in devenv: `cf/audit-persist?` is false and no full audit rows are written (telemetry is disabled there too). Uncomment `enable-nexus` in `_env` when integration testing the consumer.
|
||||
- Compose files use pure `${VAR}` substitution; missing var = compose fails.
|
||||
|
||||
## Invariants
|
||||
@ -30,7 +30,7 @@ Compose-based dev environment under `docker/devenv/`, driven by `manage.sh`. Par
|
||||
|
||||
## Worker policy
|
||||
|
||||
Backend workers run only on ws0. `_env` gates `enable-backend-worker` on `PENPOT_BACKEND_WORKER`; ws1+ inject it as false. Workers are pure fire-and-forget: `jobs/submit` inserts a row into the shared Postgres `job` table and returns (the legacy `task` table stays dormant); RPC handlers never wait on completion and workers never publish to msgbus. The reason for "ws0 only" is avoiding multi-instance worker races (cron dedup is best-effort across instances, `jobs/submit` `dedupe` is racy across submitters); details in `mem:prod-infra/core`.
|
||||
Backend workers run only on ws0. `_env` gates `enable-backend-worker` on `PENPOT_BACKEND_WORKER`; ws1+ inject it as false. Workers are fire-and-forget from the RPC's point of view: `jobs/submit` inserts a row into the shared Postgres `job` table and returns (the legacy `task` table stays dormant) and RPC handlers never wait on completion. A job with an owner profile does publish `:job-event` on that profile's msgbus topic after the commit, on the msgbus of the workspace that executes it; with per-workspace valkey, a job enqueued from ws1 and run on ws0 notifies ws0's bus, not ws1's. The reason for "ws0 only" is avoiding multi-instance worker races (cron dedup is best-effort across instances; `jobs/submit` resolves its dedupe and exclusivity in the database, so the submit itself does not race); details in `mem:prod-infra/core`.
|
||||
|
||||
Each workspace is independent and can be started/stopped in any order. Shared infra (Postgres, RustFS, etc.) is shut down only when no instances remain running.
|
||||
|
||||
|
||||
@ -15,6 +15,16 @@ Recommended order after CLJ/CLJC/CLJS source edits:
|
||||
|
||||
`clj_check_parentheses` analyzes one Clojure/ClojureScript source file and reports the area likely responsible for unclosed parentheses/brackets/braces. Use it when compiler output points near EOF, points at a misleading later form, or says delimiter-related syntax errors.
|
||||
|
||||
## Test build warnings
|
||||
|
||||
`clojure -M:dev:shadow-cljs compile test` (from `frontend/`) is the compile that
|
||||
prints the analyzer warnings CI reports. Warnings are printed only for the
|
||||
namespaces the compiler actually recompiles, so a cached build reports none
|
||||
while old ones are still latent: force it with
|
||||
`find test -name '*.cljs' -exec touch {} +` (or by clearing
|
||||
`.shadow-cljs/builds/test`) before claiming a clean tree. Why `:fn-arity`
|
||||
cannot be ignored: `mem:frontend/testing`.
|
||||
|
||||
## Hot reload notes
|
||||
|
||||
When the frontend shadow-cljs watch process is running, edits to CLJC files in `common/` are normally recompiled into the browser automatically. Do not restart the frontend before checking `cljs_compiler_output`; stale behavior is often a failed build.
|
||||
|
||||
@ -16,6 +16,6 @@
|
||||
- `app.main.ui.ui` parses the viewer's `file-id` query param with `uuid/parse*`, so a missing or malformed id arrives as nil. `viewer-page*` calls `check-file-id!` (same namespace) during render, before mounting `viewer*`: it raises `ex/raise :type :not-found :code :missing-file-id`, which the error boundary turns into the not-found page. Without the guard the `dv/initialize` assertion fails on the nil file-id and the user gets an internal-error page. Test: `frontend-tests.ui.viewer-test/check-file-id-raises-not-found-for-invalid-file-id`.
|
||||
- Viewer initialization sets `:current-file-id`, `:current-share-id`, and `:viewer-local`, then fetches the view-only bundle. Comment threads are fetched only for logged-in users.
|
||||
- Viewer bundle fetch sends the full supported feature set because anonymous shared viewers may not know team-enabled features.
|
||||
- View-only bundles can contain pointer values in `:pages-index` and file data. Viewer resolves those fragments with `:get-file-fragment` before storing the bundle, through `app.main.features.pointer-map/resolve-pointer` (share-link scoped; the workspace and library loaders use `resolve-file`, which resolves the same two levels). Both fail loudly: a fragment that cannot be resolved rejects the stream and is logged with file id and fragment id instead of silently dropping the page.
|
||||
- View-only bundles can contain pointer values in `:pages-index` and file data (membership bundles; share-link bundles carry components inline and deny their fragment). Viewer resolves those fragments with `:get-file-fragment` before storing the bundle, through `app.main.features.pointer-map/resolve-pointer` (share-link scoped; the workspace and library loaders use `resolve-file`, which resolves the same two levels). Both fail loudly: a fragment that cannot be resolved rejects the stream and is logged with file id and fragment id instead of silently dropping the page.
|
||||
- `bundle-fetched` indexes pages and precomputes viewer frames/all-frames, stores libraries/users/thumbnails/permissions under `:viewer`, then navigates to frame id, query index, or auto-selected frame.
|
||||
- Viewer zoom and interaction mode changes update both `:viewer-local` and the `:viewer` route query params. `update-zoom-querystring` guards with a query-param comparison (`not= current expected`) and uses `::rt/replace`; without that guard the load sequence (`bundle-fetched` → `zoom-to-fill` → `update-zoom-querystring` → `rt/nav`) re-runs on every navigation and crashes with React "maximum update depth exceeded" (2.18.0-RC5 regression, fixed in `31b73460c3`; regression test: `bundle-fetched-with-zoom-fill-url-does-not-navigate`).
|
||||
@ -31,6 +31,10 @@ To assert over emitted event sequences, observe termination: subscribe through `
|
||||
- `cljs.test` keeps its env in a `set!` var: assertions inside deferred ticks count. `run-block`: double `done` only warns; missing `done` stalls.
|
||||
- `t/async` discards the body promise — completion signals ONLY via `done`. `t/deftest ^:async` adds auto async-context + auto `done`, but awaiting stays the author's job; without it the test passes empty.
|
||||
- `take 1` is per-subscription on a hot subject: subscribe-before-push or hang. `end!`/`.error` with pending takes only forwards valueless completion.
|
||||
- Single-arity and variadic `defn`s compile to JS functions with no argument-count check (any target), so missing required args arrive as `undefined` instead of throwing; the analyzer
|
||||
`:fn-arity` warning is the only signal. Treat `:fn-arity` in `compile test` as a defect, never as noise.
|
||||
- `:infer-warning` on a JS interop fixes with `^js` on the target (`(.-foo ^js x)`, `(.method ^js x ...)`). Inside `set!` the hint must sit on a symbol: `(set! (.-foo ^js (f x)) v)` still warns, so bind `(let [^js y (f x)] (set! (.-foo y) v))`.
|
||||
- A helper that returns a JS object carries the hint on its name (`(defn- ^js catalog [id] ...)`), and call sites then need no hint at all — `(.method (catalog id))` and `(.-prop (catalog id))` stop warning. This is the cheapest way to keep interop out of the test bodies; `^js` on the arglist instead of the name does not work.
|
||||
|
||||
### Traps that bit
|
||||
|
||||
|
||||
@ -21,7 +21,7 @@
|
||||
Dense agent notes, not prose docs. Prefer invariants, terse bullets.
|
||||
Avoid obvious context, rationale, and examples unless they prevent likely mistakes.
|
||||
Keep guidance durable and generalizable, not task-local.
|
||||
No manual line wrapping: one line per bullet or paragraph, however long. Memories render adapting to the viewport; hard-wrapped lines degrade rendering and diffs.
|
||||
**No manual line wrapping**, in new and edited memories alike: one line per bullet or paragraph, however long. Memories render adapting to the viewport; hard-wrapped lines degrade rendering and diffs. When you touch a memory that is hard-wrapped, unwrap the whole file in the same change (join each paragraph and each bullet's continuation lines with a single space), so the wrapping does not survive merely because nobody rewrote the file.
|
||||
|
||||
## Add/update threshold
|
||||
|
||||
|
||||
@ -4,36 +4,37 @@ Backend (`app.config`, `PENPOT_*` env vars) is parameterized; deployments choose
|
||||
|
||||
## Services
|
||||
|
||||
- **PostgreSQL**: durable store. Profiles, teams, files, sessions, audit, `storage_object` metadata, the `job` work queue (plus the dormant legacy `task` table), `scheduled_task` cron registry, migrations. File-data also lives here when the file-data backend is `legacy-db`/`db`. One shared DB across all backends.
|
||||
- **Redis (Valkey-compatible)**: per-backend message bus and cache. Concrete uses: msgbus Pub/Sub for collaborative-editing broadcasts and team/profile-org notifications fired by RPC handlers (`app.rpc.notifications`, `files_update`, `teams`, `websocket`); file-summary cache gated by `enable-redis-cache`; rate-limit counters; and the dispatcher→runner work hand-off list `penpot.worker.queue:<tenant>:<queue>`. `PENPOT_REDIS_URI`. A burst of client-side `RedisCommandTimeoutException` (2s on `DEL`/`SET`, 10s on `BLPOP`) with the exporter healthy is the JVM paused or starved, not Redis: the timeout fires because nothing reads the reply.
|
||||
- **PostgreSQL**: durable store: profiles, teams, files, sessions, audit, the jobs work queue, the cron registry and migrations. File data also lives here depending on the file-data backend. One shared DB across all backends.
|
||||
- **Redis (Valkey-compatible)**: per-backend message bus and cache: collaborative-editing broadcasts and team/profile notifications, the file-summary cache, rate-limit counters and the dispatcher→runner hand-off. `PENPOT_REDIS_URI`. A burst of client-side `RedisCommandTimeoutException` with the exporter healthy is the JVM paused or starved, not Redis: the timeout fires because nothing reads the reply.
|
||||
- **Object storage**: backends `:s3` and `:fs`. S3 in prod; devenv uses RustFS. Holds uploaded media, file-data when the file-data backend is `storage`, exports. Backend-side details (resolve, dedup, bucket set, object lifecycle, and file-data backends): `mem:backend/storage`.
|
||||
- **SMTP mailer**: invitations, password resets, email verification (sent via the `:sendmail` worker job).
|
||||
- **LDAP** (optional auth provider): helpers in `app.auth.*`, gated by `enable-login-with-ldap`.
|
||||
|
||||
## Task queue and worker model
|
||||
|
||||
Async jobs are enqueued via `jobs/submit` (`app.jobs`), which inserts a row into the shared Postgres `job` table tagged with `queue = "<tenant>:<queue-name>"`. Submission is **fire-and-forget** — RPC handlers never poll and never wait. The completion signal is the `job` row's `status` / `completed_at` columns, which the creation commands read back to answer with the state of the job they just created. A job submitted with `::jobs/profile-id` also publishes a `:job-event` message on that profile's msgbus topic after each transaction commits, and the binfile commands (`create-export-binfile-job`, `create-import-binfile-job`) are the callers that pass that owner, so the progress of their jobs does reach the msgbus. Soft-delete RPCs return immediately after marking the top-level row, leaving the cascade and reaping to workers.
|
||||
Async jobs are enqueued via `jobs/submit` (`app.jobs`), which inserts a row into the shared Postgres `job` table tagged with its tenant and queue. Submission is **fire-and-forget**: RPC handlers never poll and never wait. The completion signal is the job row's status, which the creation commands read back to answer with the state of the job they just created. A job submitted with an owner profile publishes its progress on that profile's msgbus topic after each transaction commits; the binfile commands are the callers that pass the owner. Soft-delete RPCs return immediately after marking the top-level row, leaving the cascade and reaping to workers.
|
||||
|
||||
Workers run on backends with `enable-backend-worker` in `PENPOT_FLAGS`. Each worker-enabled backend has a `dispatcher` (polls `job` with `FOR UPDATE SKIP LOCKED`, marks status='scheduled', RPUSHes the JSON `[id scheduled-at]` payloads into **its own** Redis list) and one or more `runner`s per queue (BLPOP from that same local list, execute, update the Postgres row). The Redis hand-off list is purely intra-backend — cross-backend coordination happens at the Postgres row level. The queues are `default`, `webhooks`, `cron` and `binfile`: the last one carries the heavy work of a user job (`:worker-binfile-parallelism`, default 1), so a long export or import does not block the default runner.
|
||||
Workers run on backends with `enable-backend-worker` in `PENPOT_FLAGS`. Each worker-enabled backend has a dispatcher that claims jobs and pushes them to the shared Redis list `penpot.worker.queue:<tenant>:<queue>` (`app.worker/queue-key`), and one or more runners per queue that pop from the same list and update the Postgres row. The tenant segment keeps two tenants sharing one Redis apart; a job handed off by one backend can be popped by any backend running that tenant's queue, and the Postgres row is the only claim. The queues are `default`, `webhooks`, `cron` and `binfile`: `binfile` carries the heavy work of a user job, so a long export or import does not block the default runner that serves the delete cascades.
|
||||
|
||||
## Cross-backend safety
|
||||
|
||||
Postgres row locking is the only correctness primitive: `job` claims via `FOR UPDATE SKIP LOCKED`, cron firing via `FOR UPDATE SKIP LOCKED` on the `scheduled_task` row, plus job-handler-internal locks (e.g. `file_gc_scheduler` locks candidate file rows). This makes the work-claim path safe across any number of worker-enabled backends.
|
||||
Postgres row locking is the only correctness primitive: job claims, cron firing and the job-handler-internal locks (for example, the file-gc scheduler locks its candidate rows) all rely on it. This makes the work-claim path safe across any number of worker-enabled backends.
|
||||
|
||||
Two known race patterns survive multi-backend operation:
|
||||
|
||||
- **Cron dedup is best-effort.** The lock on `scheduled_task` is released when the task body finishes. If two backends' cron timers fire for the same scheduled instant with a gap larger than the task body's runtime, both execute it. Penpot's cron entries are idempotent (`session-gc`, `objects-gc`, `storage-gc-*`, `storage-pending-gc`, `jobs-gc`, `file-gc-scheduler`); the exceptions are `:telemetry` (would double-report) and `:audit-log-consumer-nexus` / `:audit-log-consumer-admin-console` (depend on archive target idempotency).
|
||||
- **`jobs/submit ::dedupe true`** does a non-atomic `DELETE` then `INSERT`. Concurrent cross-backend submits can both bypass the `DELETE` (each sees the other's uncommitted insert as absent) and end up with duplicate `'new'` rows. Each row claims and runs once independently, so the underlying work is fine; the "at most one pending" guarantee weakens.
|
||||
- **Cron dedup is best-effort.** The lock on `scheduled_task` is released when the task body finishes. If two backends' cron timers fire for the same scheduled instant with a gap larger than the task body's runtime, both execute it. Penpot's cron entries are idempotent; the exception is `:telemetry`, which would double-report. The audit consumers are not served by cron any more: each one is a loop in a thread and takes a session advisory lock for the whole pass, so a double-fire of a cron entry cannot overlap them.
|
||||
- **`jobs/submit ::dedupe` / `::debounce`** are not affected by it: the row is inserted as deduped and a partial unique index (`status = 'new'`) admits a single not-started row per tenant/name/queue/label. The insert resolves the conflict with `ON CONFLICT DO NOTHING` (skip) or `DO UPDATE` (refresh the schedule and the params of the pending row), so there is no delete before the insert and no cross-backend duplicate. Neither gives "at most one execution": a row that already left `new` (dispatched, running, retrying) does not block the next submit.
|
||||
- **`jobs/submit ::skip-if-active true`** works the same way one level up: the row is inserted as exclusive and a partial unique index admits a single live row (`new`/`scheduled`/`running`/`retry`) per tenant/name/queue/label, resolved with `ON CONFLICT DO NOTHING`. What it does not give is "at most one execution": the row can go terminal while its work is still running (a retry, an `aborted` orphan), so the work itself must be claimed on the data it touches.
|
||||
|
||||
Penpot in production lives with both: horizontal-scale deployments accept "exactly-once" as "essentially-once for idempotent operations." Devenv parallel instances handle it by running workers only on ws0 (see `mem:devenv/core`).
|
||||
|
||||
## Jobs observability
|
||||
|
||||
Worker-enabled, writable backends publish unified jobs metrics through the normal Prometheus registry. Counters describe submit, dispatch, terminal outcomes, retries, orphan recovery, rescheduling and total stored job events (unlabelled); histograms separate queue wait, handler execution and total job age. SQL-backed events are recorded after the outer PostgreSQL transaction commits. The `::jobs-metrics/sampler` component publishes backlog by status and the oldest pending-job age every 30 seconds. All labels are bounded operational values; job IDs, profile IDs, params and error text are excluded. Read-only backends do not start the sampler or the workers.
|
||||
Worker-enabled, writable backends publish unified jobs metrics through the normal Prometheus registry. One submit counter splits by `outcome` (`created`, `skipped`, `refreshed`), and the dispatch and retry counters are keyed by job name; terminal outcomes are the `_count` of the total-time histogram (`completed`, `failed`, `cancelled`, `aborted`), which counts the system-side aborts (an expired orphan lease or a job never claimed after being scheduled), so there is no separate terminal or orphan counter. Histograms separate claim wait (how late the claim happened), handler execution and total job age, plus dispatcher batch timing and size. `queue` survives only on the batch size: a batch mixes job names, and every other measurement is keyed by a name that already determines its queue. SQL-backed events are recorded after the outer PostgreSQL transaction commits. There is no gauge and no sampler: the backlog and the age of the oldest pending job are one SQL query when an operator asks. All labels are bounded operational values; job IDs, profile IDs, params and error text are excluded. Read-only backends do not start the workers.
|
||||
|
||||
## Draining one instance
|
||||
|
||||
Each backend has a per-process availability flag (`app.health`, a global atom). While offline, `/readyz` answers `503 OFFLINE` without querying the database, so the load balancer removes that instance from rotation while requests in flight still finish. Operator surface: `app.srepl.main/set-offline` / `set-online`; a restart comes back online. In-flight worker jobs are safe: jobs claimed as `scheduled` return to `new` after 5 minutes via `sql:reschedule-lost` and another instance picks them up.
|
||||
Each backend has a per-process availability flag (`app.health`, a global atom). While offline, `/readyz` answers `503 OFFLINE` without querying the database, so the load balancer removes that instance from rotation while requests in flight still finish. Operator surface: `app.srepl.main/set-offline` / `set-online`; a restart comes back online. Nothing stays stuck after an instance leaves: a job handed to a queue (`scheduled`) that no worker claims within the lost window is aborted with `:lost`, and a `running` job whose lease expires is aborted with `:orphan`. The in-flight work is abandoned, not resumed, and the sweep never retries it.
|
||||
|
||||
## Touching production data
|
||||
|
||||
@ -43,12 +44,14 @@ Production runs the deployed toolbox, not the devenv one:
|
||||
- **Start from the ids, not from an inventory.** The ids come from the error or the logs. Unbounded queries over tables with tens of millions of rows (`storage_object`, `file_media_object`) do not finish in a reasonable time; bounded to an id list they are index lookups.
|
||||
- **One query per reference column.** Two columns in one `IN (col_a, col_b)` lose the index and scan the whole table; split them.
|
||||
- The read-only replica cuts long queries: see `mem:prod-infra/grafana`.
|
||||
- A migration that builds an index runs inside the migration transaction, so it cannot use `CONCURRENTLY`, and a plain `CREATE INDEX` locks the table against writes for the whole build. On a large table build the index out of band, before deploying, with `CREATE INDEX CONCURRENTLY` on its own (never inside a transaction block). A concurrent build interrupted midway leaves the index invalid: drop it and retry (check `pg_index` and `pg_index.indisvalid`).
|
||||
- A migration that builds an index runs inside the migration transaction, so it cannot use `CONCURRENTLY`, and a plain `CREATE INDEX` locks the table against writes for the whole build. On a large table (file saves write to `file_change` on every save) build the index out of band, before deploying, with `CREATE INDEX CONCURRENTLY` on its own (never inside a transaction block); the shipped migration uses `IF NOT EXISTS`, finds it and skips the build (it still takes the `ALTER`s' locks, so the operator registers the step in `migrations`; the consumer-generalization runbook registers it first of all, before the DDL). A concurrent build interrupted midway leaves the index invalid: drop it and retry (check `pg_index` and `pg_index.indisvalid`). The runner applies **every pending step in one transaction**, so pre-applying one migration of a group does not avoid the locks of the others.
|
||||
- Audit-log rollout: `operations/2.20.0-consumer-generalization.md` owns the procedure for large databases. It pre-registers `0154-add-job-table` and `0159-consumer-generalization` before doing the schema work out of band; otherwise idempotency guards do not prevent migration locks or scans. The old archiver and new consumers do not coordinate, so stop the old archiver and keep the GC explicitly disabled during the mixed-version rollout. The new GC runs by default. Turning consumers off does not stop it: registered cursors still bound cursor-based deletion, and seven-day shadow retention needs no cursor. See `mem:audit-log` for the replay and retention guarantees.
|
||||
- The 2.20.0 release ships three out-of-band migration procedures; the unified, self-contained runbook for the whole upgrade (phases from pre-flight to the maintenance window) is `operations/2.20.0-unified.md`, indexed in `operations/README.md`. The individual 2.20.0 runbooks stay as the reference per area.
|
||||
- A partitioned parent cannot take `CREATE INDEX CONCURRENTLY` (`cannot create index on partitioned table ... concurrently`). Build one index per partition with `CONCURRENTLY`, then `CREATE INDEX ... ON ONLY` on the parent and `ALTER INDEX ... ATTACH PARTITION` for each child; the parent index only becomes valid when every partition is attached.
|
||||
- Write the migration as plain, idempotent SQL: `ADD COLUMN IF NOT EXISTS`, `CREATE INDEX IF NOT EXISTS`, and `DROP CONSTRAINT IF EXISTS` followed by `ADD CONSTRAINT`. Do not encode shape checks as `pg_catalog` introspection: they are easy to get wrong (a missing `NOT condeferred` accepts an `INITIALLY DEFERRED` foreign key) and they duplicate what the runbook verifies by hand. `IF NOT EXISTS` matches the name, not the definition, so the out-of-band procedure verifies the index shape and completeness.
|
||||
- The runbook is the contract for a large instance: it builds and verifies the out-of-band work and then marks the migration step as applied (`INSERT INTO migrations (module, step) ...`), so the body does not run at startup. Verify the column, the pending rows, the index (valid, no predicate or expression, every partition attached) and the constraint shape before inserting the row; the marker is the only thing that skips the migration.
|
||||
- Write the migration as plain, idempotent SQL: `ADD COLUMN IF NOT EXISTS`, `CREATE INDEX IF NOT EXISTS`, and `DROP CONSTRAINT IF EXISTS` followed by `ADD CONSTRAINT`. Do not encode shape checks as `pg_catalog` introspection: they are easy to get wrong (a missing `NOT condeferred` accepts an `INITIALLY DEFERRED` foreign key) and they duplicate what the runbook verifies by hand. `IF NOT EXISTS` matches the name, not the definition, so the out-of-band procedure verifies the index shape and completeness; an `INVALID` index from an interrupted build also satisfies `IF NOT EXISTS` and the dedupe/exclusivity stops enforcing silently, so the `indisvalid` gate is mandatory.
|
||||
- The runbook is the contract for a large instance: it pre-registers the migration steps (`INSERT INTO migrations (module, step) ...`) at the start, then builds and verifies the out-of-band work before deployment. The rows make the runner skip the migrations, so the manual runbook must finish every schema change and backfill before 2.20.0 starts. The unified 2.20.0 runbook registers all steps together at the beginning.
|
||||
- A reference that a GC task resolves must be `DEFERRABLE INITIALLY IMMEDIATE`, never `INITIALLY DEFERRED`: the immediate check fires inside the transaction that catches the violation and retries the chunk; the deferred one fires at commit, outside the catch. A validated foreign key belongs out of band: `ADD FOREIGN KEY` runs as an `ALTER TABLE` and it scans the whole referencing table; `NOT VALID` on a partitioned table only exists since PostgreSQL 18.
|
||||
- A backfill that removes a key an old reader uses (for example `metadata.storage-ref-id`, which the previous storage GC scans) must run with the old code stopped. Otherwise the old reader loses the reference and can delete a live object, and the rows it writes in the meantime miss the backfill.
|
||||
- A backfill that removes a key an old reader uses (for example `metadata.storage-ref-id`, which the previous storage GC scans) must run with the old code stopped. Otherwise the old reader loses the reference and can delete a live object, and the rows it writes in the meantime miss the backfill. The 2.20 file-data migration does this atomically per row; the large-instance runbook batches updates, clears the key with the column write, uses a temporary partial index for pending rows, and requires restoring the backup to roll back to the old binary.
|
||||
- `ADD COLUMN` takes a brief `ACCESS EXCLUSIVE` even when `IF NOT EXISTS` skips it, and `ADD FOREIGN KEY` takes `ACCESS EXCLUSIVE` on the referencing table and `SHARE ROW EXCLUSIVE` on the referenced one. A migration accepts those locks; a deployment that cannot take them runs the runbook and marks the step as applied.
|
||||
|
||||
## See also
|
||||
|
||||
@ -17,6 +17,10 @@ assertion is compiled out. GitHub Actions sets the same flag.
|
||||
finishing (AGENTS.md: run the applicable lint and format checks).
|
||||
- When `common/` changed: validate its consumers too (frontend, backend,
|
||||
exporter; see the dependency graph in `mem:critical-info`).
|
||||
- Pass every module you touched, `common` included: the backend `lint:clj`
|
||||
lints `../common/src` with clj-kondo but does not run `cljfmt` over it, so
|
||||
a `common/` formatting error only shows in the `common:fmt` task (its own
|
||||
`check-fmt:clj`), which is what CI runs.
|
||||
- To fix formatting across a module (`--fix`) or repair delimiters
|
||||
(`--paren-repair`) before linting.
|
||||
|
||||
|
||||
@ -106,6 +106,11 @@ Use your test framework's parametrize/table-driven mechanism to test multiple
|
||||
scenarios with a single test body. Keeps tests concise and surfaces all cases
|
||||
at a glance.
|
||||
|
||||
Only when the body is short and identical for every row. A loop that builds its
|
||||
own world per row (fresh store, fixtures, several awaits) or that nests two
|
||||
dimensions hides which case failed and shares state across cases: write one
|
||||
test per case instead, with a shared fixture for the arrangement.
|
||||
|
||||
## Test Pyramid
|
||||
|
||||
```
|
||||
|
||||
@ -15,6 +15,13 @@
|
||||
- **Never pipe test output directly to filters** (`| head`, `| tail`, `| grep`, etc.).
|
||||
Always redirect to a file first: `command > /tmp/output.txt 2>&1`, then read/grep the file.
|
||||
This prevents hiding test failures. See `mem:testing` for details.
|
||||
- **Never introduce a new function or macro name ending in `!`.** Side
|
||||
effects alone do not justify it: only the exceptions documented in
|
||||
`mem:clojure/design-rules` (public mutators of `app.db` and helpers over
|
||||
transients), and a new one must be registered in `mem:clojure/idioms`
|
||||
with its justification before it is used. Existing names stay until their
|
||||
refactor; a new one fails review. See `mem:clojure/design-rules` before
|
||||
naming anything.
|
||||
- **Read `AGENTS.local.md` after this file whenever it exists.** It carries
|
||||
the developer's steering for this project and overrides this file on prose style and
|
||||
workflow, but not on project policy, where instructions here prevail.
|
||||
|
||||
@ -42,7 +42,6 @@ export PENPOT_FLAGS="\
|
||||
enable-backend-asserts \
|
||||
disable-feature-fdata-pointer-map \
|
||||
enable-feature-fdata-objects-map \
|
||||
enable-audit-log \
|
||||
enable-transit-readable-response \
|
||||
disable-remote-media-processing \
|
||||
enable-demo-users \
|
||||
|
||||
@ -396,7 +396,18 @@
|
||||
;; When telemetry-enabled config is true, add :telemetry flag
|
||||
(true? (c/get config :telemetry-enabled))
|
||||
(conj :enable-telemetry))]
|
||||
(flags/parse flags/default extra-flags (:flags config))))
|
||||
;; :audit-log is no longer configurable: persistence follows the
|
||||
;; consumer flags, so a stale enable-audit-log in the environment is
|
||||
;; dropped instead of lingering in the set as a misleading no-op.
|
||||
(-> (flags/parse flags/default extra-flags (:flags config))
|
||||
(disj :audit-log))))
|
||||
|
||||
(defn persist-audit-events?
|
||||
"True when at least one audit-log consumer is enabled. The result is
|
||||
computed once at boot into `audit-persist?`; event paths read that value
|
||||
instead of checking the flags on every event."
|
||||
[flags]
|
||||
(boolean (some #{:nexus :admin-console} flags)))
|
||||
|
||||
(defn read-env
|
||||
[prefix]
|
||||
@ -453,6 +464,10 @@
|
||||
|
||||
(defonce ^:dynamic config (read-config :default default))
|
||||
(defonce ^:dynamic flags (parse-flags config))
|
||||
(defonce ^:dynamic audit-persist? (persist-audit-events? flags))
|
||||
(defonce ^:dynamic webhooks-enabled? (contains? flags :webhooks))
|
||||
(defonce ^:dynamic telemetry-enabled? (contains? flags :telemetry))
|
||||
(defonce ^:dynamic error-reporting-enabled? (contains? flags :error-reporting))
|
||||
|
||||
(defn validate!
|
||||
"Validate the currently loaded configuration data."
|
||||
|
||||
@ -683,6 +683,19 @@
|
||||
(finally
|
||||
(reset! context [])))))))
|
||||
|
||||
(defn own-transaction
|
||||
"A cfg for a write that must own its transaction: drops `::db/conn`
|
||||
so the next `tx-run!` opens its own transaction on its own
|
||||
connection, and `::db/read-only` so that transaction is writable
|
||||
even when the caller runs read-only (as the storage scheduler does
|
||||
for its plan cursor). It never joins the transaction of whoever
|
||||
called it; because that transaction joins no other, it owns its
|
||||
after-commit context too, so the write and its metric commit or
|
||||
vanish together and a rollback of the surrounding work can undo
|
||||
neither."
|
||||
[cfg]
|
||||
(dissoc cfg ::conn ::read-only))
|
||||
|
||||
(defn tx-run!
|
||||
"Run a function in a transaction."
|
||||
[system f & params]
|
||||
@ -820,6 +833,52 @@
|
||||
row (exec-one! conn ["select pg_try_advisory_xact_lock(?::bigint) as lock" n])]
|
||||
(:lock row)))
|
||||
|
||||
(defn- check-lock-key
|
||||
"A two-key advisory lock takes int4 keys; anything else (a uuid, a value
|
||||
out of range) would reach the database as an opaque cast error."
|
||||
[n]
|
||||
(if (and (int? n) (<= Integer/MIN_VALUE (long n) Integer/MAX_VALUE))
|
||||
n
|
||||
(throw (IllegalArgumentException. "int4 key required"))))
|
||||
|
||||
(defn session-try-lock
|
||||
"Take a session-level advisory lock of two keys without waiting.
|
||||
|
||||
`classid` is the namespace of the lock and `objid` the key inside it.
|
||||
This two-key form lives in its own lock space, apart from the one-key
|
||||
`pg_advisory_lock(bigint)` of `xact-lock!` (the file locks and the setup
|
||||
one): PostgreSQL keys a lock by `objsubid` (1 for one key, 2 for two), so
|
||||
the same numbers in the two forms do not conflict. That is how a
|
||||
subsystem takes a namespace of its own without touching the others.
|
||||
|
||||
Unlike the `xact-*` variants the lock survives the transaction: it lives
|
||||
until `session-unlock` releases it, until the connection is really closed
|
||||
or until the session ends. A pooled connection returned with a lock held
|
||||
keeps it, so the caller must release it; `session-unlock-all` clears
|
||||
whatever a borrowed connection carries."
|
||||
[conn classid objid]
|
||||
(let [row (exec-one! conn ["select pg_try_advisory_lock(?::int, ?::int) as lock"
|
||||
(check-lock-key classid)
|
||||
(check-lock-key objid)])]
|
||||
(:lock row)))
|
||||
|
||||
(defn session-unlock
|
||||
"Release one two-key session-level advisory lock held by this connection.
|
||||
Answers false when this connection did not hold that lock."
|
||||
[conn classid objid]
|
||||
(let [row (exec-one! conn ["select pg_advisory_unlock(?::int, ?::int) as lock"
|
||||
(check-lock-key classid)
|
||||
(check-lock-key objid)])]
|
||||
(:lock row)))
|
||||
|
||||
(defn session-unlock-all
|
||||
"Release every session-level advisory lock held by this connection.
|
||||
Used before taking one on a borrowed connection, so a lock some earlier
|
||||
borrower failed to release cannot outlive it."
|
||||
[conn]
|
||||
(exec-one! conn ["select pg_advisory_unlock_all()"])
|
||||
true)
|
||||
|
||||
(defn sql-exception?
|
||||
[cause]
|
||||
(instance? java.sql.SQLException cause))
|
||||
|
||||
@ -281,37 +281,52 @@
|
||||
;; SUBMIT API
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
;; Two columns mark the jobs whose key (tenant, name, queue, label) is
|
||||
;; resolved by the database instead of by the caller: `exclusive` admits a
|
||||
;; single live row (`job__live_exclusive__uniq`) and `dedupe` admits a
|
||||
;; single not-started one (`job__new_dedupe__uniq`). Both are resolved by
|
||||
;; the insert: PostgreSQL admits a single `ON CONFLICT` clause per
|
||||
;; statement, so a submit is one of these four and never two at once.
|
||||
;;
|
||||
;; The conflict target repeats the index predicate on purpose: that is
|
||||
;; how PostgreSQL infers a partial unique index.
|
||||
(def ^:private sql:insert-new-job
|
||||
"insert into job (id, name, tenant, params, queue, label, priority,
|
||||
max_retries, profile_id, resource_id, expires_at,
|
||||
created_at, modified_at, scheduled_at)
|
||||
values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
created_at, modified_at, scheduled_at, exclusive, dedupe)
|
||||
values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)")
|
||||
|
||||
;; A single live row per key: `::jobs/skip-if-active` affects no row and
|
||||
;; the submit reports the job as skipped.
|
||||
(def ^:private sql:conflict-exclusive
|
||||
"on conflict (tenant, name, queue, label)
|
||||
where exclusive
|
||||
and status in ('new', 'scheduled', 'running', 'retry')
|
||||
do nothing
|
||||
returning id")
|
||||
|
||||
;; `tenant` is part of the match, not an extra: without it this DELETE
|
||||
;; reaches the not-started jobs of every other instance sharing the
|
||||
;; database.
|
||||
(def ^:private sql:remove-not-started-jobs
|
||||
"DELETE FROM job
|
||||
WHERE name=?
|
||||
AND tenant=?
|
||||
AND queue=?
|
||||
AND label=?
|
||||
AND status = 'new'
|
||||
AND scheduled_at > ?")
|
||||
;; A single not-started row per key: `::jobs/dedupe` leaves it alone and
|
||||
;; `::jobs/debounce` refreshes it in place. The refreshed row keeps its
|
||||
;; id, its events and its `created_at`; only the schedule and the params
|
||||
;; travel from the new submission.
|
||||
(def ^:private sql:conflict-dedupe
|
||||
"on conflict (tenant, name, queue, label)
|
||||
where dedupe and status = 'new'
|
||||
do nothing
|
||||
returning id")
|
||||
|
||||
;; The cron no-overlap check and the skip-if-active submit option share
|
||||
;; this shape: both look for jobs of one tenant with the same name, queue
|
||||
;; and label that have not reached a terminal state yet. The
|
||||
;; `job__name_label__idx` partial index serves both queries.
|
||||
(def ^:private sql:count-active-jobs
|
||||
"SELECT count(*) AS n
|
||||
FROM job
|
||||
WHERE name = ?
|
||||
AND tenant = ?
|
||||
AND queue = ?
|
||||
AND label = ?
|
||||
AND status IN ('new', 'scheduled', 'running', 'retry')")
|
||||
(def ^:private sql:conflict-debounce
|
||||
"on conflict (tenant, name, queue, label)
|
||||
where dedupe and status = 'new'
|
||||
do update set params = excluded.params,
|
||||
scheduled_at = excluded.scheduled_at,
|
||||
modified_at = excluded.modified_at
|
||||
returning id")
|
||||
|
||||
;; A plain submit has no conflict target but still answers with the id of
|
||||
;; the row it wrote: every insert returns it, whichever clause it carries.
|
||||
(def ^:private sql:returning-id
|
||||
"returning id")
|
||||
|
||||
(def ^:private schema:options
|
||||
[:map {:title "submit-options"}
|
||||
@ -322,7 +337,15 @@
|
||||
[::queue {:optional true} [:or ::sm/text :keyword]]
|
||||
[::priority {:optional true} ::sm/int]
|
||||
[::max-retries {:optional true} ::sm/int]
|
||||
;; Skip the insert when a not-started job already covers the same
|
||||
;; (tenant, name, queue, label), whatever its `scheduled_at`. Requires
|
||||
;; an explicit label, like the other two.
|
||||
[::dedupe {:optional true} ::sm/boolean]
|
||||
;; Same key as `dedupe`, but instead of skipping, move the pending job
|
||||
;; to `now + delay` and store the new params. That is the sliding
|
||||
;; window of a batch: the receiver gets the last payload, not the
|
||||
;; first. Mutually exclusive with `dedupe`.
|
||||
[::debounce {:optional true} ::sm/boolean]
|
||||
;; Skip the insert when a live job already covers the same
|
||||
;; (tenant, name, queue, label). Requires an explicit label: the
|
||||
;; default empty one would block every flagged submission at once.
|
||||
@ -352,6 +375,40 @@
|
||||
(sm/check (::schema job-def) params))
|
||||
params)
|
||||
|
||||
(defn- check-option-label
|
||||
"Raise unless the option that keys the submit of a job has an explicit
|
||||
label: the label is the key of the dedupe and of the exclusion, and the
|
||||
default empty one would block every flagged submission at once."
|
||||
[option label job-name]
|
||||
(when (or (not (string? label)) (str/blank? label))
|
||||
(ex/raise :type :validation
|
||||
:code (keyword (str option "-requires-label"))
|
||||
:hint (str option " requires an explicit non-blank label")
|
||||
:name job-name)))
|
||||
|
||||
(defn- check-keyed-options
|
||||
"Validate the options that need an explicit label and the ones that
|
||||
contradict each other. Runs before the schema check so the failure names
|
||||
the option, not the shape: the schema would reject a blank label with a
|
||||
generic \"check error\" either way."
|
||||
[{::keys [name label dedupe debounce skip-if-active]}]
|
||||
(when (and dedupe debounce)
|
||||
(ex/raise :type :validation
|
||||
:code :dedupe-and-debounce-are-exclusive
|
||||
:hint "dedupe and debounce cannot be used together"
|
||||
:name (d/name name)))
|
||||
(when (and skip-if-active (or dedupe debounce))
|
||||
(ex/raise :type :validation
|
||||
:code :skip-if-active-cannot-be-combined
|
||||
:hint "skip-if-active cannot be combined with dedupe or debounce"
|
||||
:name (d/name name)))
|
||||
(when dedupe
|
||||
(check-option-label "dedupe" label (d/name name)))
|
||||
(when debounce
|
||||
(check-option-label "debounce" label (d/name name)))
|
||||
(when skip-if-active
|
||||
(check-option-label "skip-if-active" label (d/name name))))
|
||||
|
||||
(defn submit
|
||||
"Submit a durable job: validates the params with the job-def validator,
|
||||
encodes them as plain JSON and inserts a row into the `job` table.
|
||||
@ -361,36 +418,59 @@
|
||||
`::expires-at`) are validated and stored as columns; nothing is inferred
|
||||
from `params`.
|
||||
|
||||
With `::skip-if-active`, the insert is skipped (and `nil` returned)
|
||||
when a live job (`new`, `scheduled`, `running` or `retry`) already
|
||||
exists for the same tenant, name, queue and label. The check and the
|
||||
insert share one transaction. A skip deletes nothing and inserts
|
||||
nothing; it records a skipped metric instead of a submitted one, so
|
||||
dashboards can tell withheld work apart from new work. The option
|
||||
needs an explicit non-blank `::label`, otherwise it raises.
|
||||
With `::skip-if-active`, the row is inserted as exclusive: the
|
||||
database admits a single live row (a `new`, `scheduled`, `running` or
|
||||
`retry` one) per tenant, name, queue and label, and the insert is
|
||||
skipped (and `nil` returned) when there is already one. The decision
|
||||
is the unique index `job__live_exclusive__uniq`, so it holds across
|
||||
backends and does not depend on a check racing an insert. A skip
|
||||
inserts nothing and records the submit with `outcome=\"skipped\"`
|
||||
instead of `created`, so dashboards can tell withheld work apart from
|
||||
new work. The option needs an explicit non-blank `::label`, otherwise
|
||||
it raises: the label is the key of the exclusion.
|
||||
|
||||
NOTE: the dedupe DELETE and the INSERT run atomically: joined to
|
||||
the caller's transaction when the cfg provides `::db/conn`, wrapped
|
||||
in their own transaction otherwise. Concurrent cross-backend
|
||||
submissions can, in rare race conditions, produce duplicated 'new'
|
||||
rows (accepted risk, see prod-infra documentation). The same caveat
|
||||
covers the skip-if-active check: a race between backends can slip
|
||||
two inserts past the count."
|
||||
The exclusion is between the jobs that asked for it: a live job that
|
||||
was not submitted as exclusive (a webhook delivery, a notification)
|
||||
does not block an exclusive submit, and two such jobs can share a
|
||||
label. That is why the flag is a column and not a rule about the
|
||||
label: the label stays a label.
|
||||
|
||||
With `::dedupe`, the row is inserted as deduplicated: the database
|
||||
admits a single not-started (`new`) row per tenant, name, queue and
|
||||
label, whatever its `scheduled_at`, and the insert is skipped (and
|
||||
`nil` returned) when there is already one. The pending job is not
|
||||
touched: it runs when it is due, with its own params. The decision is
|
||||
the unique index `job__new_dedupe__uniq`, so it holds across backends.
|
||||
`::debounce` uses the same key and the same index but, instead of
|
||||
skipping, refreshes the pending row: it moves `scheduled_at` to
|
||||
`now + delay`, stores the new params and returns the id the row already
|
||||
had. That is the sliding window of a batch job, where the last event is
|
||||
the one worth delivering. A refresh is not a new job, so the submit
|
||||
records it with `outcome=\"refreshed\"`, apart from the ones that created
|
||||
a row. Both options need an explicit non-blank `::label` and are
|
||||
mutually exclusive.
|
||||
|
||||
Only the jobs submitted with `::dedupe` or `::debounce` are part of
|
||||
that key, so a plain job never blocks a deduplicated one, and two plain
|
||||
jobs can share a label.
|
||||
|
||||
An insert carries a single `ON CONFLICT` clause, so `::skip-if-active`
|
||||
cannot be combined with `::dedupe` or `::debounce`: they are different
|
||||
keys, an exclusive job already admits a single live row, and the
|
||||
combination raises.
|
||||
|
||||
The submit runs joined to the caller's transaction when the cfg
|
||||
provides `::db/conn`, wrapped in its own otherwise, so a rollback takes
|
||||
the row and the metric with it. There is no read before the write: the
|
||||
unique index of the option decides."
|
||||
[cfg {:keys [::params ::name ::delay ::queue ::priority ::max-retries
|
||||
::dedupe ::skip-if-active ::label
|
||||
::dedupe ::debounce ::skip-if-active ::label
|
||||
::profile-id ::resource-id ::expires-at]
|
||||
:or {delay 0 queue :default priority 100 max-retries 3 label ""}
|
||||
:as options}]
|
||||
|
||||
(assert (mtx/instance cfg) "missing ::mtx/metrics on the jobs cfg")
|
||||
;; Before the schema check so the failure names the option, not the
|
||||
;; shape: the schema would reject a blank label with a generic
|
||||
;; "check error" either way.
|
||||
(when (and skip-if-active (or (not (string? label)) (str/blank? label)))
|
||||
(ex/raise :type :validation
|
||||
:code :skip-if-active-requires-label
|
||||
:hint "skip-if-active requires an explicit non-blank label"
|
||||
:name (d/name name)))
|
||||
(check-keyed-options options)
|
||||
(check-options options)
|
||||
(let [job-def (get-job-def (get-defs cfg) name)
|
||||
params (validate-params job-def params)
|
||||
@ -413,62 +493,62 @@
|
||||
tenant (cf/get :tenant)
|
||||
job-name (d/name name)
|
||||
queue (d/name queue)
|
||||
;; Dedupe is best-effort: we delete not-started jobs of this
|
||||
;; tenant with the same name/queue/label, then insert. A race
|
||||
;; between backends could create duplicates, but this is
|
||||
;; acceptable: cross-backend races are rare, jobs are idempotent,
|
||||
;; and dedupe is best-effort.
|
||||
;; Skip-if-active is best-effort like dedupe: the count and the
|
||||
;; insert share this transaction, but a race between backends
|
||||
;; can still slip two inserts past the count.
|
||||
active? (fn [conn]
|
||||
(when skip-if-active
|
||||
(pos? (or (:n (db/exec-one! conn [sql:count-active-jobs
|
||||
job-name tenant queue label]))
|
||||
0))))
|
||||
insert! (fn [conn]
|
||||
(let [deleted (when dedupe
|
||||
;; the DELETE below is intentional:
|
||||
;; disable the `job` deletion-protection
|
||||
;; guard inside this transaction
|
||||
(db/exec-one! conn ["SET LOCAL rules.deletion_protection TO off"])
|
||||
(-> (db/exec-one! conn [sql:remove-not-started-jobs
|
||||
job-name tenant queue label now])
|
||||
(db/get-update-count)))]
|
||||
(l/trc :hint "submit"
|
||||
:name job-name
|
||||
:job-id (str id)
|
||||
:tenant tenant
|
||||
:queue queue
|
||||
:label label
|
||||
:dedupe (boolean dedupe)
|
||||
:delay (ct/format-duration delay)
|
||||
:replace (or deleted 0))
|
||||
(db/exec-one! conn [sql:insert-new-job id job-name tenant payload queue
|
||||
label priority max-retries
|
||||
profile-id resource-id expires-at
|
||||
now now scheduled-at])))]
|
||||
;; Both statements always run inside db/tx-run!: joined to the
|
||||
;; caller's transaction when the cfg provides a connection,
|
||||
;; wrapped in their own otherwise (a failed INSERT can never
|
||||
;; orphan a committed DELETE, even on an autocommit caller conn).
|
||||
;; A skip returns nil: no DELETE, no INSERT, no submitted metric.
|
||||
;; A single ON CONFLICT clause per insert: the option picks it,
|
||||
;; and the unique index of that option decides. The insert is the
|
||||
;; only statement, so there is nothing to delete and nothing to
|
||||
;; roll back on its own.
|
||||
conflict (cond
|
||||
skip-if-active sql:conflict-exclusive
|
||||
debounce sql:conflict-debounce
|
||||
dedupe sql:conflict-dedupe
|
||||
:else nil)
|
||||
sql (str/join " " [sql:insert-new-job
|
||||
(or conflict sql:returning-id)])]
|
||||
;; The insert runs inside db/tx-run!: joined to the caller's
|
||||
;; transaction when the cfg provides a connection, wrapped in its own
|
||||
;; otherwise. The statement is what decides: a conflicting one affects
|
||||
;; no row (the skip) or refreshes the pending row (the debounce) and
|
||||
;; answers with its id, so the row it names is always a real one.
|
||||
(db/tx-run! cfg
|
||||
(fn [{:keys [::db/conn]}]
|
||||
(if (active? conn)
|
||||
(do (db/after-commit
|
||||
#(metrics/record-skipped cfg job-name queue))
|
||||
(l/trc :hint "submit skipped, active job exists"
|
||||
:name job-name
|
||||
:tenant tenant
|
||||
:queue queue
|
||||
:label label
|
||||
:skipped true))
|
||||
(let [result (insert! conn)]
|
||||
(if-let [result
|
||||
(db/exec-one! conn [sql id job-name tenant payload
|
||||
queue label priority max-retries
|
||||
profile-id resource-id expires-at
|
||||
now now scheduled-at
|
||||
(boolean skip-if-active)
|
||||
(boolean (or dedupe debounce))])]
|
||||
(let [created? (= id (:id result))]
|
||||
(db/after-commit
|
||||
#(metrics/record-submitted cfg job-name queue))
|
||||
result
|
||||
id))))))
|
||||
#(if created?
|
||||
(metrics/record-created cfg job-name)
|
||||
(metrics/record-refreshed cfg job-name)))
|
||||
(l/trc :hint "submit"
|
||||
:outcome (if created? "created" "refreshed")
|
||||
:name job-name
|
||||
:job-id (str (:id result))
|
||||
:tenant tenant
|
||||
:queue queue
|
||||
:label label
|
||||
:dedupe (boolean dedupe)
|
||||
:debounce (boolean debounce)
|
||||
:exclusive (boolean skip-if-active)
|
||||
:delay (ct/format-duration delay))
|
||||
(:id result))
|
||||
(do
|
||||
(db/after-commit
|
||||
#(metrics/record-skipped cfg job-name))
|
||||
(l/trc :hint "submit"
|
||||
:outcome "skipped"
|
||||
:name job-name
|
||||
:tenant tenant
|
||||
:queue queue
|
||||
:label label
|
||||
:dedupe (boolean dedupe)
|
||||
:debounce (boolean debounce)
|
||||
:exclusive (boolean skip-if-active)
|
||||
:delay (ct/format-duration delay))
|
||||
nil))))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; JOB API
|
||||
@ -526,9 +606,9 @@
|
||||
|
||||
(def ^:private known-outcomes
|
||||
;; mirrors the `job_event` end outcomes. `aborted` is only ever written
|
||||
;; by the dispatcher orphan sweep (bulk insert in the same transaction,
|
||||
;; see `app.worker.dispatcher/mark-orphan-jobs`): no `app.jobs` writer
|
||||
;; emits it.
|
||||
;; by the dispatcher sweeps, orphan and lost (one bulk insert per sweep
|
||||
;; in the same transaction of its status change, see
|
||||
;; `app.worker.dispatcher`): no `app.jobs` writer emits it.
|
||||
#{"completed" "failed" "cancelled" "aborted"})
|
||||
|
||||
(def ^:private known-retry-reasons
|
||||
@ -716,21 +796,6 @@
|
||||
:created-at created-at})))
|
||||
{:event-id id :created-at created-at :kind kind :payload payload})))
|
||||
|
||||
(defn- own-transaction
|
||||
"A cfg for a job write that must own its transaction.
|
||||
|
||||
A lifecycle write reports on work that already happened elsewhere, so
|
||||
it never joins the transaction of whoever called it: the `::db/conn` is
|
||||
dropped and `db/tx-run!` opens its own transaction on its own
|
||||
connection. Because that transaction joins no other, it owns its
|
||||
after-commit context too, so the write and its metric commit or vanish
|
||||
together and a rollback of the surrounding work can undo neither.
|
||||
|
||||
`submit` is the deliberate exception: creating a job is part of the
|
||||
business unit of work of whoever asks for it."
|
||||
[cfg]
|
||||
(dissoc cfg ::db/conn))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; HEARTBEAT / PROGRESS
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
@ -825,10 +890,10 @@
|
||||
here, so a progress event can never land on an already terminal job.
|
||||
Returns 1 when the event was stored.
|
||||
|
||||
Like every lifecycle write it owns its transaction (see `own-transaction`),
|
||||
Like every lifecycle write it owns its transaction (see `db/own-transaction`),
|
||||
so a beat survives the rollback of the work around it."
|
||||
[cfg job-id progress]
|
||||
(db/tx-run! (own-transaction cfg)
|
||||
(db/tx-run! (db/own-transaction cfg)
|
||||
(fn [{:keys [::db/conn] :as tx-cfg}]
|
||||
(if (db/exec-one! conn [sql:lock-active-job job-id])
|
||||
(do
|
||||
@ -942,7 +1007,7 @@
|
||||
;; Every metric is registered with `db/after-commit` from inside the
|
||||
;; transaction, next to the event, so it lands on the outermost commit
|
||||
;; whatever that is: a rolled back transition is never counted. Each
|
||||
;; writer owns that transaction (see `own-transaction`), so the commit
|
||||
;; writer owns that transaction (see `db/own-transaction`), so the commit
|
||||
;; in question is always its own.
|
||||
;;
|
||||
;; That callback swallows exceptions, which is why every writer below
|
||||
@ -969,18 +1034,19 @@
|
||||
"Record the total time of a terminal outcome. Runs after the commit
|
||||
that made the job terminal, so a rolled back transition is never
|
||||
counted. The histogram `_count` is the terminal total per outcome:
|
||||
no separate counter. Takes the cfg and resolves the metrics
|
||||
instance from it, so the callback is not tied to an instance read
|
||||
at registration time."
|
||||
no separate counter, and `outcome=\"aborted\"` counts the system-side
|
||||
aborts (an orphan lease or a job lost on its way to the queue).
|
||||
The row always carries `created_at` (`NOT NULL` in the table, and every
|
||||
caller reads it), so there is no guard: a caller that forgets it fails
|
||||
in `db/after-commit`, which logs it. Takes the cfg and resolves the
|
||||
metrics instance from it, so the callback is not tied to an instance
|
||||
read at registration time."
|
||||
[cfg job outcome]
|
||||
(when job
|
||||
(when (ct/inst? (:created-at job))
|
||||
(metrics/record-total
|
||||
cfg
|
||||
(:name job)
|
||||
(:queue job)
|
||||
outcome
|
||||
(- (inst-ms (ct/now)) (inst-ms (:created-at job)))))))
|
||||
(metrics/record-total
|
||||
cfg
|
||||
(:name job)
|
||||
outcome
|
||||
(- (inst-ms (ct/now)) (inst-ms (:created-at job)))))
|
||||
|
||||
(def schema:job-error
|
||||
"Structured error of a job that did not complete. `type` and `code` are
|
||||
@ -1007,6 +1073,16 @@
|
||||
:code :orphan
|
||||
:hint "job execution lease expired"}))
|
||||
|
||||
(def lost-error
|
||||
"Error of a job that was handed to its queue and no worker claimed it
|
||||
within the dispatcher window. The queue message is gone (`scheduled`
|
||||
means pushed), so the job is marked `aborted` (system-side terminal,
|
||||
never retried) with its `end` event in the same transaction instead of
|
||||
going back to `new`."
|
||||
(check-job-error {:type :internal
|
||||
:code :lost
|
||||
:hint "job was not claimed after being scheduled"}))
|
||||
|
||||
(def ^:private decode-job-error-value
|
||||
(sm/decoder schema:job-error sm/json-transformer))
|
||||
|
||||
@ -1036,15 +1112,15 @@
|
||||
|
||||
Only transitions a pending row (new/scheduled/retry) to `running`, and
|
||||
requires an exact `scheduled-at` match with the value advertised in the
|
||||
queue payload, so a stale payload (row rescheduled or claimed in the
|
||||
meantime) affects 0 rows and must be skipped. Returns the number of
|
||||
affected rows.
|
||||
queue payload, so a duplicate or stale payload (the row was claimed in
|
||||
the meantime, or is already aborted) affects 0 rows and must be
|
||||
skipped. Returns the number of affected rows.
|
||||
|
||||
The event carries `attempt` = `retry_num + 1`: the number of the
|
||||
attempt this run consumes from the retry budget."
|
||||
[cfg job-id scheduled-at]
|
||||
(assert (mtx/instance cfg) "missing ::mtx/metrics on the jobs cfg")
|
||||
(db/tx-run! (own-transaction cfg)
|
||||
(db/tx-run! (db/own-transaction cfg)
|
||||
(fn [{:keys [::db/conn] :as tx-cfg}]
|
||||
(if-let [{:keys [retry-num]}
|
||||
(db/exec-one! conn [sql:claim-job job-id scheduled-at])]
|
||||
@ -1076,7 +1152,7 @@
|
||||
(assert (mtx/instance cfg) "missing ::mtx/metrics on the jobs cfg")
|
||||
(let [job (get-job cfg job-id)
|
||||
encoded (db/json (check-job-error error))]
|
||||
(db/tx-run! (own-transaction cfg)
|
||||
(db/tx-run! (db/own-transaction cfg)
|
||||
(fn [{:keys [::db/conn] :as tx-cfg}]
|
||||
(if-let [{:keys [retry-num]}
|
||||
(db/exec-one! conn [sql:retry-job (ct/now) scheduled-at
|
||||
@ -1088,7 +1164,6 @@
|
||||
(db/after-commit
|
||||
#(metrics/record-retry tx-cfg
|
||||
(:name job)
|
||||
(:queue job)
|
||||
reason))
|
||||
1)
|
||||
0)))))
|
||||
@ -1151,7 +1226,7 @@
|
||||
[cfg & {:keys [job-id result resource-id]}]
|
||||
(assert (mtx/instance cfg) "missing ::mtx/metrics on the jobs cfg")
|
||||
(check-resource-id resource-id)
|
||||
(let [n (db/tx-run! (own-transaction cfg)
|
||||
(let [n (db/tx-run! (db/own-transaction cfg)
|
||||
(fn [{:keys [::db/conn] :as tx-cfg}]
|
||||
(let [job (some-> (db/exec-one! conn [sql:lock-job job-id])
|
||||
decode-row)
|
||||
@ -1206,7 +1281,7 @@
|
||||
(assert (mtx/instance cfg) "missing ::mtx/metrics on the jobs cfg")
|
||||
(let [job (get-job cfg job-id)
|
||||
encoded (db/json (check-job-error error))
|
||||
n (db/tx-run! (own-transaction cfg)
|
||||
n (db/tx-run! (db/own-transaction cfg)
|
||||
(fn [{:keys [::db/conn] :as tx-cfg}]
|
||||
(if (db/exec-one! conn [sql:fail-job (ct/now) encoded job-id])
|
||||
(do
|
||||
@ -1239,7 +1314,7 @@
|
||||
[cfg job-id]
|
||||
(assert (mtx/instance cfg) "missing ::mtx/metrics on the jobs cfg")
|
||||
(let [job (get-job cfg job-id)
|
||||
n (db/tx-run! (own-transaction cfg)
|
||||
n (db/tx-run! (db/own-transaction cfg)
|
||||
(fn [{:keys [::db/conn] :as tx-cfg}]
|
||||
(if (db/exec-one! conn [sql:cancel-job (ct/now) job-id])
|
||||
(do
|
||||
|
||||
@ -5,27 +5,22 @@
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.jobs.metrics
|
||||
"Metric names, labels and the jobs backlog sampler.
|
||||
"Metric names and labels used by the jobs substrate.
|
||||
|
||||
Labels are bounded by construction, not by allowlist: every value
|
||||
recorded here comes from a fixed call site (a job name from the
|
||||
registry, a queue from submit, an outcome from the lifecycle
|
||||
writers), never from params, IDs or exception text. A new value
|
||||
showing up in Prometheus means new code, which is exactly what you
|
||||
want to see: folding it into some other bucket would only hide it.
|
||||
registry, a queue from the dispatcher batch, an outcome from submit or
|
||||
the lifecycle writers), never from params, IDs or exception text. A new
|
||||
value showing up in Prometheus means new code, which is exactly what
|
||||
you want to see: folding it into some other bucket would only hide it.
|
||||
|
||||
This namespace is intentionally free of job handlers. Callers pass only
|
||||
values already known at their boundary; IDs, props and exception text never
|
||||
become metric labels."
|
||||
(:require
|
||||
[app.common.data :as d]
|
||||
[app.common.logging :as l]
|
||||
[app.common.schema :as sm]
|
||||
[app.db :as db]
|
||||
[app.metrics :as mtx]
|
||||
[cuerdas.core :as str]
|
||||
[integrant.core :as ig]
|
||||
[promesa.exec :as px]))
|
||||
[cuerdas.core :as str]))
|
||||
|
||||
(set! *warn-on-reflection* true)
|
||||
|
||||
@ -59,29 +54,40 @@
|
||||
(defn- record-count [metrics id labels amount]
|
||||
(mtx/run! metrics :id id :labels labels :inc amount))
|
||||
|
||||
(defn record-submitted
|
||||
[cfg name queue]
|
||||
(record-count (mtx/instance cfg) :jobs-submitted
|
||||
[(name-label name) (queue-label queue)]
|
||||
(defn- record-submit
|
||||
"One measurement with three outcomes: the submit created a row, skipped
|
||||
it because a live job covered the key (`::jobs/skip-if-active`, any status
|
||||
from `new` to `retry`), or refreshed the not-started row that covered it
|
||||
(`::jobs/debounce`). The outcome comes from a fixed call site, so it has
|
||||
no fallback, unlike `outcome-label`, which would say `failed` for a
|
||||
submission that cannot fail."
|
||||
[cfg outcome name]
|
||||
(record-count (mtx/instance cfg) :jobs-submit
|
||||
[(name-label name) (d/name outcome)]
|
||||
1))
|
||||
|
||||
(defn record-created
|
||||
[cfg name]
|
||||
(record-submit cfg :created name))
|
||||
|
||||
(defn record-skipped
|
||||
[cfg name queue]
|
||||
(record-count (mtx/instance cfg) :jobs-skipped
|
||||
[(name-label name) (queue-label queue)]
|
||||
1))
|
||||
[cfg name]
|
||||
(record-submit cfg :skipped name))
|
||||
|
||||
(defn record-refreshed
|
||||
[cfg name]
|
||||
(record-submit cfg :refreshed name))
|
||||
|
||||
(defn record-dispatched
|
||||
[cfg name queue amount]
|
||||
[cfg name amount]
|
||||
(record-count (mtx/instance cfg) :jobs-dispatched
|
||||
[(name-label name) (queue-label queue)]
|
||||
[(name-label name)]
|
||||
amount))
|
||||
|
||||
(defn record-retry
|
||||
[cfg name queue reason]
|
||||
[cfg name reason]
|
||||
(record-count (mtx/instance cfg) :jobs-retries
|
||||
[(name-label name) (queue-label queue)
|
||||
(retry-reason-label reason)]
|
||||
[(name-label name) (retry-reason-label reason)]
|
||||
1))
|
||||
|
||||
(defn record-event
|
||||
@ -91,53 +97,41 @@
|
||||
[cfg]
|
||||
(record-count (mtx/instance cfg) :jobs-events-total [] 1))
|
||||
|
||||
(defn record-orphan
|
||||
[cfg queue]
|
||||
(record-count (mtx/instance cfg) :jobs-orphaned
|
||||
[(queue-label queue)]
|
||||
1))
|
||||
|
||||
(defn record-rescheduled
|
||||
[cfg queue]
|
||||
(record-count (mtx/instance cfg) :jobs-rescheduled
|
||||
[(queue-label queue)]
|
||||
1))
|
||||
|
||||
(defn record-queue-wait
|
||||
[cfg name queue millis]
|
||||
(record-value (mtx/instance cfg) :jobs-queue-wait-timing
|
||||
[(name-label name) (queue-label queue)]
|
||||
(defn record-claim-wait
|
||||
"How late a claim happened: `now - scheduled_at` at the moment the
|
||||
runner takes the job, including the gap the dispatcher added. It is not
|
||||
time spent inside a queue, hence the name."
|
||||
[cfg name millis]
|
||||
(record-value (mtx/instance cfg) :jobs-claim-wait-timing
|
||||
[(name-label name)]
|
||||
(max 0 (long millis))))
|
||||
|
||||
(defn record-execution
|
||||
[cfg name queue millis]
|
||||
[cfg name millis]
|
||||
(record-value (mtx/instance cfg) :jobs-execution-timing
|
||||
[(name-label name) (queue-label queue)]
|
||||
[(name-label name)]
|
||||
(max 0 (long millis))))
|
||||
|
||||
(defn record-legacy-execution
|
||||
"Histogram `penpot_tasks_timing`, from the pre-jobs `task` queue.
|
||||
|
||||
It only carries the job name label, with no queue to tell jobs of the
|
||||
same name apart. Use `record-execution`, which adds the queue label,
|
||||
for anything new. Kept exported only so the dashboards and the alerts
|
||||
that already read it keep working: like every other jobs metric,
|
||||
the label is the raw job name."
|
||||
It carries the job name label, like every other jobs metric. Kept
|
||||
exported only so the dashboards and the alerts that already read it
|
||||
keep working: the label is the raw job name."
|
||||
[cfg name millis]
|
||||
(record-value (mtx/instance cfg) :tasks-timing
|
||||
[(mtx/label name "other")]
|
||||
(max 0 (long millis))))
|
||||
|
||||
(defn record-total
|
||||
[cfg name queue outcome millis]
|
||||
[cfg name outcome millis]
|
||||
(record-value (mtx/instance cfg) :jobs-total-timing
|
||||
[(name-label name) (queue-label queue)
|
||||
(outcome-label outcome)]
|
||||
[(name-label name) (outcome-label outcome)]
|
||||
(max 0 (long millis))))
|
||||
|
||||
(defn record-dispatcher-batch
|
||||
[cfg stage outcome millis]
|
||||
(record-value (mtx/instance cfg) :jobs-dispatcher-timing
|
||||
(record-value (mtx/instance cfg) :jobs-dispatcher-batch-timing
|
||||
[(stage-label stage) (outcome-label outcome)]
|
||||
(max 0 (long millis))))
|
||||
|
||||
@ -155,8 +149,9 @@
|
||||
amount))
|
||||
|
||||
(defn record-gc-duration
|
||||
"One pass of the jobs garbage collector, `expired` or `retained`."
|
||||
[cfg kind millis]
|
||||
(record-value (mtx/instance cfg) :jobs-gc-timing
|
||||
(record-value (mtx/instance cfg) :jobs-gc-pass-timing
|
||||
[(gc-kind-label kind)]
|
||||
(max 0 (long millis))))
|
||||
|
||||
@ -215,74 +210,3 @@
|
||||
(record-value (mtx/instance cfg) :file-data-offload-timing
|
||||
[(offload-result-label result)]
|
||||
(max 0 (long millis))))
|
||||
|
||||
(def ^:private backlog-statuses
|
||||
["new" "scheduled" "running" "retry" "completed" "failed" "cancelled" "aborted"])
|
||||
|
||||
(def ^:private sql:backlog
|
||||
"SELECT status, count(*) AS n
|
||||
FROM job
|
||||
GROUP BY status")
|
||||
|
||||
(def ^:private sql:oldest-pending
|
||||
"SELECT coalesce(extract(epoch FROM (now() - min(created_at))), 0) AS age
|
||||
FROM job
|
||||
WHERE status IN ('new', 'scheduled', 'retry')")
|
||||
|
||||
(defn sample-backlog
|
||||
"Update the current job backlog gauges. The query is intentionally small
|
||||
and runs periodically rather than from the worker hot path."
|
||||
[{:keys [::db/pool] :as cfg}]
|
||||
(let [rows (db/exec! pool [sql:backlog])
|
||||
by-id (into {} (map (juxt :status #(or (:n %) 0))) rows)
|
||||
age (-> (db/exec-one! pool [sql:oldest-pending])
|
||||
:age
|
||||
(max 0)
|
||||
(double))
|
||||
metrics (mtx/instance cfg)]
|
||||
(doseq [status backlog-statuses]
|
||||
(record-value metrics :jobs-backlog [status] (get by-id status 0)))
|
||||
(record-value metrics :jobs-oldest-pending-age [] age)
|
||||
age))
|
||||
|
||||
(defn- sample-jobs-metrics [cfg]
|
||||
(try
|
||||
(sample-backlog cfg)
|
||||
(catch Throwable cause
|
||||
(l/warn :hint "unable to sample jobs metrics" :cause cause))))
|
||||
|
||||
(def ^:private sample-interval-ms 30000)
|
||||
|
||||
(defn create-sampler
|
||||
"Create the daemon scheduler used by the jobs metrics component."
|
||||
[cfg]
|
||||
(let [scheduler (px/scheduled-executor
|
||||
:parallelism 1
|
||||
:factory (px/thread-factory :prefix "penpot/jobs-metrics/"
|
||||
:daemon true))
|
||||
sample (fn sample []
|
||||
(try
|
||||
(sample-jobs-metrics cfg)
|
||||
(finally
|
||||
(px/schedule scheduler sample-interval-ms sample))))]
|
||||
(px/schedule scheduler 0 sample)
|
||||
scheduler))
|
||||
|
||||
(def ^:private schema:sampler
|
||||
[:map
|
||||
::db/pool
|
||||
::mtx/metrics])
|
||||
|
||||
(defmethod ig/assert-key ::sampler
|
||||
[_ cfg]
|
||||
(assert (sm/check schema:sampler cfg)))
|
||||
|
||||
(defmethod ig/init-key ::sampler
|
||||
[_ {:keys [::db/pool] :as cfg}]
|
||||
(if (db/read-only? pool)
|
||||
(l/warn :hint "not started (db is read-only)")
|
||||
(create-sampler cfg)))
|
||||
|
||||
(defmethod ig/halt-key! ::sampler
|
||||
[_ sampler]
|
||||
(some-> sampler px/shutdown-now))
|
||||
|
||||
@ -21,8 +21,6 @@
|
||||
[app.http :as-alias http]
|
||||
[app.http.access-token :as-alias actoken]
|
||||
[app.jobs :as jobs]
|
||||
[app.loggers.audit.consumer-task :as consumer]
|
||||
[app.loggers.audit.tasks :as-alias tasks]
|
||||
[app.loggers.webhooks :as-alias webhooks]
|
||||
[app.rpc :as-alias rpc]
|
||||
[app.setup :as-alias setup]
|
||||
@ -88,6 +86,17 @@
|
||||
:sync-failed
|
||||
:snapshot-complete})
|
||||
|
||||
(def ^:private telemetry-sources
|
||||
#{"telemetry:backend" "telemetry:frontend"})
|
||||
|
||||
(defn telemetry-shadow-source?
|
||||
"True for the anonymized shadow sources of the telemetry consumer. A
|
||||
selective consumer must skip them: the full row carries the event
|
||||
already, and the shadow keeps the event name, so processing it would
|
||||
ingest the same event twice (once full, once anonymized)."
|
||||
[source]
|
||||
(contains? telemetry-sources (str source)))
|
||||
|
||||
(def ^:private safe-backend-context-keys
|
||||
#{:version
|
||||
:initiator
|
||||
@ -282,13 +291,13 @@
|
||||
(update :props db/tjson)
|
||||
(update :context db/tjson)
|
||||
(update :ip-addr db/inet))
|
||||
params (select-keys params event-keys)
|
||||
row (db/insert! cfg :audit-log params)]
|
||||
;; The consumer queue follows the row in the same transaction: one
|
||||
;; pending per enabled consumer. A rollback takes both on its way
|
||||
;; out, so a row never outlives its queue.
|
||||
(consumer/queue-pendings cfg [(:id row)])
|
||||
row))
|
||||
params (select-keys params event-keys)]
|
||||
;; The producer knows nothing about the consumers: one row, and each
|
||||
;; consumer reads it from its own cursor. The row and the cursor
|
||||
;; position commit independently, so a consumer can process a row
|
||||
;; only after it is visible here (the snapshot cut in the consumer
|
||||
;; guarantees it).
|
||||
(db/insert! cfg :audit-log params)))
|
||||
|
||||
(def ^:private xf:filter-telemetry-props
|
||||
"Transducer that keeps only map entries whose values are UUIDs,
|
||||
@ -318,14 +327,14 @@
|
||||
:props (json/encode (:props event) :key-fn json/write-camel-key)
|
||||
:context (json/encode (:context event) :key-fn json/write-camel-key)))
|
||||
|
||||
(when (contains? cf/flags :audit-log)
|
||||
(when cf/audit-persist?
|
||||
(append-audit-entry cfg event))
|
||||
|
||||
(when (contains? cf/flags :telemetry)
|
||||
;; NOTE: when both audit-log and telemetry are enabled, events are stored
|
||||
(when cf/telemetry-enabled?
|
||||
;; NOTE: when both persistence and telemetry are enabled, events are stored
|
||||
;; twice: once with full details (above) and once stripped of props and
|
||||
;; ip-addr, tagged with source="telemetry" so the telemetry task can
|
||||
;; collect and ship them. The profile-id is preserved (UUIDs are already
|
||||
;; ip-addr, tagged with source="telemetry:*" so the telemetry consumer
|
||||
;; (id 0) can send them. The profile-id is preserved (UUIDs are already
|
||||
;; anonymous random identifiers). Only a safe subset of context fields
|
||||
;; is kept: initiator, version, client-version and client-user-agent.
|
||||
;; Timestamps are truncated to day precision to avoid leaking exact event
|
||||
@ -341,7 +350,7 @@
|
||||
(assoc :ip-addr "0.0.0.0"))]
|
||||
(append-audit-entry cfg event)))
|
||||
|
||||
(when (and (contains? cf/flags :webhooks)
|
||||
(when (and cf/webhooks-enabled?
|
||||
(::webhooks/event? event))
|
||||
(let [batch-key (::webhooks/batch-key event)
|
||||
batch-timeout (::webhooks/batch-timeout event)
|
||||
@ -350,14 +359,17 @@
|
||||
(ifn? batch-key) (dm/str label ":" (batch-key (::rpc/params event)))
|
||||
(string? batch-key) (dm/str label ":" batch-key)
|
||||
:else label)
|
||||
dedupe? (boolean (and batch-key batch-timeout))]
|
||||
debounce? (boolean (and batch-key batch-timeout))]
|
||||
|
||||
(jobs/submit cfg
|
||||
{::jobs/name :process-webhook-event
|
||||
::jobs/queue :webhooks
|
||||
::jobs/max-retries 0
|
||||
::jobs/delay (or batch-timeout 0)
|
||||
::jobs/dedupe dedupe?
|
||||
;; A batch slides its window with every event and
|
||||
;; delivers the last payload, so the receiver is
|
||||
;; told once about the whole burst.
|
||||
::jobs/debounce debounce?
|
||||
::jobs/label label
|
||||
;; The event travels as an opaque transit blob:
|
||||
;; transit preserves the instant/UUID/set types
|
||||
@ -373,13 +385,13 @@
|
||||
event))
|
||||
|
||||
(defn submit*
|
||||
"A public API, lower-level than submit, assumes all required fields are filled"
|
||||
"A public API, lower-level than submit, assumes all required fields are
|
||||
filled. A failure propagates: an event that cannot be stored never
|
||||
reaches any consumer, and nothing records that it was lost. Swallowing
|
||||
it here made that invisible."
|
||||
[cfg event]
|
||||
(try
|
||||
(let [event (check-event event)]
|
||||
(db/tx-run! cfg process-event event))
|
||||
(catch Throwable cause
|
||||
(l/error :hint "unexpected error processing event" :cause cause))))
|
||||
(let [event (check-event event)]
|
||||
(db/tx-run! cfg process-event event)))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; PUBLIC API
|
||||
@ -454,12 +466,14 @@
|
||||
(defn- coerce-profile-id
|
||||
"Normalize a hand-written `::audit/profile-id` override to a uuid.
|
||||
|
||||
`schema:event` requires a uuid and `submit*` swallows the validation
|
||||
error, so a value that is not a uuid loses the event instead of
|
||||
failing loudly. Commands read the override from places that are not
|
||||
typed by us (token claims, stringly-typed drivers), so a string has
|
||||
to be accepted. Anything that cannot become a uuid is discarded, and
|
||||
the event falls back to the caller, which is always a valid uuid."
|
||||
`schema:event` requires a uuid and `submit*` propagates the validation
|
||||
error; `wrap-audit` runs after the command commits (moving it inside the
|
||||
transaction is project P8 in `TODO.md`), so an invalid override would fail
|
||||
the response over an already committed action. Commands read the
|
||||
override from places that are not typed by us (token claims,
|
||||
stringly-typed drivers), so a string has to be accepted. Anything that
|
||||
cannot become a uuid is discarded, and the event falls back to the
|
||||
caller, which is always a valid uuid."
|
||||
[v]
|
||||
(let [coerced (cond
|
||||
;; Fast path: the override comes straight from a `profile`
|
||||
@ -478,15 +492,57 @@
|
||||
|
||||
coerced))
|
||||
|
||||
(def ^:private sql:team-id-by-project
|
||||
"SELECT id FROM team
|
||||
WHERE id = (SELECT team_id FROM project WHERE id = ?)")
|
||||
(def ^:private sql:team-ids-by-projects
|
||||
"SELECT team_id
|
||||
FROM project
|
||||
WHERE id = ANY(?::uuid[])")
|
||||
|
||||
(def ^:private sql:team-id-by-file
|
||||
"SELECT t.id FROM team AS t
|
||||
JOIN project AS p ON (p.team_id = t.id)
|
||||
JOIN file AS f ON (f.project_id = p.id)
|
||||
WHERE f.id = ?")
|
||||
(def ^:private sql:team-ids-by-files
|
||||
"SELECT p.team_id AS team_id
|
||||
FROM file AS f
|
||||
JOIN project AS p ON (p.id = f.project_id)
|
||||
WHERE f.id = ANY(?::uuid[])")
|
||||
|
||||
(def ^:private xf:team-ids
|
||||
"Transducer: pull the team id out of a team/project/file resolution
|
||||
row. Allocated once so hot paths (every audited RPC) do not rebuild
|
||||
it per call."
|
||||
(map :team-id))
|
||||
|
||||
(defn- resolve-team-ids
|
||||
"Resolve one bag of project or file ids to team ids with one ANY(?)
|
||||
query on a plain connection. The result holds one tiny row per id,
|
||||
so neither a cursor nor a transaction is needed."
|
||||
[cfg kind ids]
|
||||
;; Fetch inside the connection, reduce outside it: the rows are
|
||||
;; already materialized, so the pool connection is held as briefly
|
||||
;; as possible.
|
||||
(->> (db/run! cfg
|
||||
(fn [{:keys [::db/conn]}]
|
||||
(db/exec! conn [(case kind
|
||||
:project sql:team-ids-by-projects
|
||||
:file sql:team-ids-by-files)
|
||||
(db/create-array conn "uuid" ids)])))
|
||||
(into #{} xf:team-ids)))
|
||||
|
||||
(defn get-team-ids
|
||||
"Resolve a mixed bag of team, project and file ids to team ids with
|
||||
two batched lookups. Producers of cascade-style events call it so
|
||||
consumers never join production tables. The pure part runs without
|
||||
touching the database; each lookup resolves its connection lazily,
|
||||
only when it has ids (reusing the caller transaction). Every id
|
||||
reaching this function is a UUID already (RPC params and frontend
|
||||
ingest travel as transit, jobs read DB rows): anything else is a
|
||||
bug at the source, never coerced here."
|
||||
[cfg {:keys [team-ids project-ids file-ids]}]
|
||||
(let [direct (into #{} team-ids)]
|
||||
(-> direct
|
||||
(into (if (seq project-ids)
|
||||
(resolve-team-ids cfg :project project-ids)
|
||||
#{}))
|
||||
(into (if (seq file-ids)
|
||||
(resolve-team-ids cfg :file file-ids)
|
||||
#{})))))
|
||||
|
||||
(def ^:private read-event-names
|
||||
"Read commands excluded from team-id resolution on top of the `get-*`
|
||||
@ -513,9 +569,9 @@
|
||||
(let [project-id (:project-id props)
|
||||
file-id (:file-id props)
|
||||
team-id (or (when (some? project-id)
|
||||
(:id (db/exec-one! cfg [sql:team-id-by-project project-id])))
|
||||
(first (get-team-ids cfg {:project-ids [project-id]})))
|
||||
(when (some? file-id)
|
||||
(:id (db/exec-one! cfg [sql:team-id-by-file file-id]))))]
|
||||
(first (get-team-ids cfg {:file-ids [file-id]}))))]
|
||||
(cond-> props
|
||||
(some? team-id) (assoc :team-id team-id)))))
|
||||
|
||||
@ -622,7 +678,7 @@
|
||||
event is built by hand and does not pass through `prepare-rpc-event`, so its
|
||||
props are passed as-is minus nils."
|
||||
[cfg params event-name props]
|
||||
(when (contains? cf/flags :audit-log)
|
||||
(when cf/audit-persist?
|
||||
(submit cfg
|
||||
(-> (event-from-rpc-params params)
|
||||
(assoc :name event-name)
|
||||
@ -647,9 +703,9 @@
|
||||
command line helpers because this skips all webhooks and telemetry
|
||||
logic."
|
||||
[cfg event]
|
||||
(when (contains? cf/flags :audit-log)
|
||||
;; Transactional (not a bare connection): the row and its
|
||||
;; pendings commit or vanish together (see append-audit-entry).
|
||||
(when cf/audit-persist?
|
||||
;; Transactional (not a bare connection): the row commits or
|
||||
;; vanishes whole (see append-audit-entry).
|
||||
(let [tnow (ct/now)
|
||||
event (-> event
|
||||
(assoc :created-at tnow)
|
||||
|
||||
@ -1,691 +0,0 @@
|
||||
;; This Source Code Form is subject to the terms of the Mozilla Public
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.loggers.audit.consumer-task
|
||||
(:require
|
||||
[app.common.exceptions :as ex]
|
||||
[app.common.logging :as l]
|
||||
[app.common.schema :as sm]
|
||||
[app.common.time :as ct]
|
||||
[app.common.transit :as t]
|
||||
[app.common.uri :as u]
|
||||
[app.common.uuid :as uuid]
|
||||
[app.config :as cf]
|
||||
[app.db :as db]
|
||||
[app.db.sql :as sql]
|
||||
[app.http.client :as http]
|
||||
[app.jobs :as jobs]
|
||||
[app.metrics :as mtx]
|
||||
[app.nitrate :as nitrate]
|
||||
[app.setup :as-alias setup]
|
||||
[integrant.core :as ig]
|
||||
[promesa.exec :as px]))
|
||||
|
||||
;; This task drains the audit log buffer into the long-term archives.
|
||||
;;
|
||||
;; Two separate jobs share this namespace and the queue helpers, one
|
||||
;; per destination: `:audit-log-consumer-nexus` and
|
||||
;; `:audit-log-consumer-admin-console`. Each job drains only its own queue
|
||||
;; and deletes its pendings on completion; whoever deletes the last
|
||||
;; pending also deletes the record ("last one out turns off the
|
||||
;; light"), so no separate GC task is needed. Housekeeping (reaping
|
||||
;; disabled pendings, sweeping orphans) lives in both jobs on purpose:
|
||||
;; only the enabled job runs, so it must clean up for the rest.
|
||||
|
||||
;; Consumer ids are fixed constants, never reuse one: retired
|
||||
;; pendings are reaped by id, so a reused id would inherit the
|
||||
;; previous owner's queue.
|
||||
(def consumer-nexus 1)
|
||||
(def consumer-admin-console 2)
|
||||
|
||||
(def ^:private consumers
|
||||
[{:id consumer-nexus :flag :nexus :label "nexus"}
|
||||
{:id consumer-admin-console :flag :admin-console :label "admin-console"}])
|
||||
|
||||
(def ^:private xf:enabled-consumers
|
||||
"Static transducer: keep the consumers whose send flag is on.
|
||||
Allocated once so hot paths (every audited RPC) do not rebuild
|
||||
it per call; the flag lookup itself still reads the live flags."
|
||||
(comp (filter #(contains? cf/flags (:flag %)))
|
||||
(map :id)))
|
||||
|
||||
(defn enabled-consumers
|
||||
"Consumers allowed to work right now, from the send flags. Public:
|
||||
producers queue every record for exactly these consumers."
|
||||
[]
|
||||
(into [] xf:enabled-consumers consumers))
|
||||
|
||||
(defn queue-pendings
|
||||
"Queue records for every enabled consumer, nothing when there is
|
||||
nothing to queue or no consumer is on. One multi-row statement for
|
||||
all ids instead of one per consumer: this runs inside every audited
|
||||
RPC and the bulk frontend ingest. Runs inside the row transaction:
|
||||
a rollback takes both."
|
||||
[cfg ids]
|
||||
(let [consumers (enabled-consumers)]
|
||||
(when (and (seq ids) (seq consumers))
|
||||
(db/insert-many! cfg :audit-log-pending
|
||||
[:audit-log-id :consumer]
|
||||
(for [id ids
|
||||
consumer consumers]
|
||||
[id consumer])
|
||||
{::sql/on-conflict-do-nothing true}))))
|
||||
|
||||
(def ^:private consumer-labels
|
||||
(into {} (map (juxt :id :label)) consumers))
|
||||
|
||||
(defn- consumer-label
|
||||
[consumer]
|
||||
(get consumer-labels consumer))
|
||||
|
||||
(defn- decode-row
|
||||
[{:keys [props ip-addr context] :as row}]
|
||||
(cond-> row
|
||||
(db/pgobject? props)
|
||||
(assoc :props (db/decode-transit-pgobject props))
|
||||
|
||||
(db/pgobject? context)
|
||||
(assoc :context (db/decode-transit-pgobject context))
|
||||
|
||||
(db/pgobject? ip-addr "inet")
|
||||
(assoc :ip-addr (db/decode-inet ip-addr))))
|
||||
|
||||
(def ^:private event-keys
|
||||
[:type
|
||||
:name
|
||||
:source
|
||||
:created-at
|
||||
:tracked-at
|
||||
:profile-id
|
||||
:ip-addr
|
||||
:props
|
||||
:context])
|
||||
|
||||
(defn- row->event
|
||||
[row]
|
||||
(select-keys row event-keys))
|
||||
|
||||
(defn- record-count
|
||||
[metrics id labels amount]
|
||||
(mtx/run! metrics :id id :labels labels :inc amount))
|
||||
|
||||
(defn- truncate-body
|
||||
[body limit]
|
||||
(when (some? body)
|
||||
(let [s (str body)]
|
||||
(if (> (count s) limit)
|
||||
(subs s 0 limit)
|
||||
s))))
|
||||
|
||||
;; NOTE: deliberately no SKIP LOCKED here. Batches must reach the
|
||||
;; destinations in chronological order (Nexus numbers by arrival),
|
||||
;; so every consumer takes the same total order and a second
|
||||
;; overlapping run waits for the first to commit, then sees whether
|
||||
;; it was last. The brief blocking is accepted; skipping rows could
|
||||
;; reorder archive arrival.
|
||||
(def ^:private sql:pending-chunk
|
||||
"SELECT a.*
|
||||
FROM audit_log_pending p
|
||||
JOIN audit_log a ON a.id = p.audit_log_id
|
||||
WHERE p.consumer = ?::smallint
|
||||
ORDER BY a.created_at ASC, a.id ASC
|
||||
LIMIT 128
|
||||
FOR UPDATE OF a")
|
||||
|
||||
(def ^:private sql:delete-pendings
|
||||
"DELETE FROM audit_log_pending
|
||||
WHERE consumer = ?::smallint
|
||||
AND audit_log_id = ANY(?)")
|
||||
|
||||
(def ^:private sql:delete-drained
|
||||
"DELETE FROM audit_log a
|
||||
WHERE a.id = ANY(?)
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM audit_log_pending p
|
||||
WHERE p.audit_log_id = a.id
|
||||
)")
|
||||
|
||||
(defn- get-pending-rows
|
||||
[{:keys [::db/conn] :as cfg} consumer]
|
||||
(->> (db/exec! conn [sql:pending-chunk consumer])
|
||||
(not-empty)))
|
||||
|
||||
(defn- complete-batch
|
||||
"Delete our pendings for the batch, then the records nobody is
|
||||
waiting for anymore. Two statements on purpose: inside a single
|
||||
statement the NOT EXISTS would still see the pending just deleted
|
||||
and never delete a record."
|
||||
[{:keys [::db/conn]} consumer rows]
|
||||
(let [ids (db/create-array conn "uuid" (map :id rows))]
|
||||
(db/exec! conn [sql:delete-pendings consumer ids])
|
||||
(db/exec! conn [sql:delete-drained ids])))
|
||||
|
||||
(defn- generate-nexus-uri
|
||||
"Joins relative path segments to the Nexus backend URI, same shape
|
||||
as the admin-console helper in `app.nitrate` (there the ingest path
|
||||
is `api/audit-log`; here it is `api/audit`).
|
||||
Segments must not start with `/`. Returns nil when no base URI is
|
||||
configured, so the caller keeps the `:task-not-configured` guard."
|
||||
[& segments]
|
||||
(when-let [base (cf/get :nexus-uri)]
|
||||
(apply cf/join-uri base segments)))
|
||||
|
||||
(defn- send-to-nexus
|
||||
"POST events to Nexus. Returns the number of events sent; throws on
|
||||
transport failure or non-204 refusal, carrying the attempted count
|
||||
as :events in the error data so the caller records the failure.
|
||||
No metrics or logs here: the runner reports the aborted job."
|
||||
[{:keys [::uri] :as cfg} events]
|
||||
(let [n (count events)
|
||||
skey (-> cfg ::setup/shared-keys :nexus)
|
||||
body (t/encode {:events events})
|
||||
params {:uri uri
|
||||
:timeout 12000
|
||||
:method :post
|
||||
:headers {"content-type" "application/transit+json"
|
||||
"origin" (str (cf/get :public-uri))
|
||||
"x-shared-key" (str "nexus " skey)}
|
||||
:body body}
|
||||
resp (try
|
||||
(http/req cfg params {:skip-ssrf-check? true})
|
||||
(catch Throwable cause
|
||||
(throw (ex/raise :type :internal
|
||||
:code :nexus-send-failed
|
||||
:hint "unable to send events to nexus"
|
||||
:cause cause
|
||||
:events n))))]
|
||||
(if (= (:status resp) 204)
|
||||
n
|
||||
(throw (ex/raise :type :internal
|
||||
:code :consumer-chunk-refused
|
||||
:hint "nexus refused archive chunk"
|
||||
:resp-status (:status resp)
|
||||
:resp-body (truncate-body (:body resp) 500)
|
||||
:events n)))))
|
||||
|
||||
(def ^:private event-names-for-admin-console
|
||||
#{"accept-organization-invitation"
|
||||
"accept-organization-invitation-from"
|
||||
"accept-team-invitation"
|
||||
"accept-team-invitation-from"
|
||||
"add-member-to-organization"
|
||||
"add-team-to-organization"
|
||||
"assign-organization-attribute"
|
||||
"auditlog-exported"
|
||||
"auditlog-guide-downloaded"
|
||||
"cancel-organization-invitation"
|
||||
"create-export-binfile-job"
|
||||
"create-file"
|
||||
"create-organization"
|
||||
"create-organization-attribute"
|
||||
"create-organization-invitation"
|
||||
"create-project"
|
||||
"create-team"
|
||||
"create-team-access-request"
|
||||
"create-team-invitation"
|
||||
"create-team-invitations"
|
||||
"create-webhook"
|
||||
"delete-file"
|
||||
"delete-font"
|
||||
"delete-organization"
|
||||
"delete-organization-attribute"
|
||||
"delete-organization-member"
|
||||
"delete-project"
|
||||
"delete-team"
|
||||
"delete-team-invitation"
|
||||
"delete-team-member"
|
||||
"leave-team"
|
||||
"move-project"
|
||||
"move-team-to-organization"
|
||||
"organization-sso-auth-failed"
|
||||
"organization-sso-auth-started"
|
||||
"organization-sso-auth-succeeded"
|
||||
"organization-sso-token-issued"
|
||||
"organization-sso-token-revoked"
|
||||
"permanently-delete-team-files"
|
||||
"remove-organization-team"
|
||||
"remove-team-from-organization"
|
||||
"rename-organization"
|
||||
"rename-project"
|
||||
"restore-deleted-team-files"
|
||||
"sso-config-deleted"
|
||||
"sso-config-tested"
|
||||
"success-activate-organization-sso"
|
||||
"success-apply-organization-sso-changes"
|
||||
"transfer-team-ownership"
|
||||
"unassign-organization-attribute"
|
||||
"update-organization-attribute"
|
||||
"update-organization-invitation"
|
||||
"update-organization-member-role"
|
||||
"update-organization-permissions"
|
||||
"update-team-invitation"
|
||||
"update-team-invitation-role"
|
||||
"update-team-member-role"
|
||||
"update-team-photo"
|
||||
"verify-token"})
|
||||
|
||||
;; Props id key -> email prop filled from profile.email (admin-console path only).
|
||||
;; Some producers store ids as UUID strings (e.g. :user-who-send-invitation).
|
||||
(def ^:private id->email-prop
|
||||
{:profile-id :profile-email
|
||||
:user-id :user-email
|
||||
:member-id :member-email
|
||||
:user-who-send-invitation :user-who-send-invitation-email})
|
||||
|
||||
(defn- collect-team-ids-needing-name
|
||||
[events]
|
||||
(into #{}
|
||||
(comp (map :props)
|
||||
(keep (fn [props]
|
||||
(when-let [team-id (and (not (contains? props :team-name))
|
||||
(uuid/coerce (:team-id props)))]
|
||||
team-id))))
|
||||
events))
|
||||
|
||||
(defn- collect-profile-ids-needing-email
|
||||
[events]
|
||||
(into #{}
|
||||
(mapcat (fn [{:keys [props]}]
|
||||
(keep (fn [[id-key email-key]]
|
||||
(when-not (contains? props email-key)
|
||||
(uuid/coerce (get props id-key))))
|
||||
id->email-prop)))
|
||||
events))
|
||||
|
||||
(defn- collect-actor-ids-needing-profile
|
||||
"Actor ids (event :profile-id) that still need top-level :profile-email
|
||||
and/or :profile-name. Skips uuid/zero and non-uuids."
|
||||
[events]
|
||||
(into #{}
|
||||
(keep (fn [event]
|
||||
(when (or (not (contains? event :profile-email))
|
||||
(not (contains? event :profile-name)))
|
||||
(when-let [id (uuid/coerce (:profile-id event))]
|
||||
(when-not (uuid/zero? id)
|
||||
id)))))
|
||||
events))
|
||||
|
||||
(defn- load-team-names
|
||||
[conn ids]
|
||||
(if (seq ids)
|
||||
(let [arr (db/create-array conn "uuid" ids)
|
||||
rows (db/exec! conn ["SELECT id, name FROM team WHERE id = ANY(?)" arr])]
|
||||
(into {} (map (juxt :id :name) rows)))
|
||||
{}))
|
||||
|
||||
(defn- load-profiles
|
||||
"Batch-load profile email and fullname (including soft-deleted)."
|
||||
[conn ids]
|
||||
(if (seq ids)
|
||||
(let [arr (db/create-array conn "uuid" ids)
|
||||
rows (db/exec! conn ["SELECT id, email, fullname FROM profile WHERE id = ANY(?)" arr])]
|
||||
(into {} (map (juxt :id identity) rows)))
|
||||
{}))
|
||||
|
||||
(defn- maybe-assoc-team-name
|
||||
[event team-names]
|
||||
(let [props (:props event)]
|
||||
(if-let [team-name (and (not (contains? props :team-name))
|
||||
(when-let [team-id (uuid/coerce (:team-id props))]
|
||||
(get team-names team-id)))]
|
||||
(assoc-in event [:props :team-name] team-name)
|
||||
event)))
|
||||
|
||||
(defn- maybe-assoc-emails
|
||||
[event profiles]
|
||||
(update event :props
|
||||
(fn [props]
|
||||
(reduce-kv (fn [props id-key email-key]
|
||||
(if-let [email (and (not (contains? props email-key))
|
||||
(when-let [id (uuid/coerce (get props id-key))]
|
||||
(:email (get profiles id))))]
|
||||
(assoc props email-key email)
|
||||
props))
|
||||
props
|
||||
id->email-prop))))
|
||||
|
||||
(defn- maybe-assoc-actor-profile
|
||||
"Fill-only top-level :profile-email / :profile-name from the event
|
||||
actor (:profile-id). Omits keys when the id is missing, zero, the
|
||||
profile row is absent, or the corresponding column is empty
|
||||
(fullname defaults to \"\" in the schema)."
|
||||
[event profiles]
|
||||
(let [id (uuid/coerce (:profile-id event))]
|
||||
(if (or (nil? id) (uuid/zero? id))
|
||||
event
|
||||
(if-let [profile (get profiles id)]
|
||||
(cond-> event
|
||||
(and (not (contains? event :profile-email))
|
||||
(not-empty (:email profile)))
|
||||
(assoc :profile-email (:email profile))
|
||||
|
||||
(and (not (contains? event :profile-name))
|
||||
(not-empty (:fullname profile)))
|
||||
(assoc :profile-name (:fullname profile)))
|
||||
event))))
|
||||
|
||||
(defn- enrich-events-for-admin-console
|
||||
"Enrich allowlisted events before the admin-console ingest:
|
||||
1. Fill missing :team-name from team id (including soft-deleted).
|
||||
2. Fill missing emails from matching id keys in props (including
|
||||
soft-deleted profiles): :profile-email, :user-email, :member-email,
|
||||
:user-who-send-invitation-email.
|
||||
3. Fill missing top-level :profile-email / :profile-name from the
|
||||
event actor (:profile-id), including soft-deleted profiles; omit
|
||||
when there is no usable actor id or the value is empty."
|
||||
[{:keys [::db/conn]} events]
|
||||
(let [team-ids (collect-team-ids-needing-name events)
|
||||
profile-ids (into (collect-profile-ids-needing-email events)
|
||||
(collect-actor-ids-needing-profile events))
|
||||
team-names (load-team-names conn team-ids)
|
||||
profiles (load-profiles conn profile-ids)]
|
||||
(mapv (fn [event]
|
||||
(-> event
|
||||
(maybe-assoc-team-name team-names)
|
||||
(maybe-assoc-emails profiles)
|
||||
(maybe-assoc-actor-profile profiles)))
|
||||
events)))
|
||||
|
||||
(defn- send-to-admin-console
|
||||
"Ingest rows into the admin console. Returns {:sent :skipped}
|
||||
(filtered-out rows count as processed for the queue); throws
|
||||
carrying the attempted count as :events, like send-to-nexus. Enrich
|
||||
failures propagate untouched: a local DB/decode failure is ours, not
|
||||
the destination's, and must not count as a send failure."
|
||||
[cfg rows]
|
||||
(let [events (filterv #(contains? event-names-for-admin-console (:name %)) rows)
|
||||
total (count rows)
|
||||
skipped (- total (count events))
|
||||
uri-host (some-> (cf/get :admin-console-uri) u/uri u/get-domain)]
|
||||
(l/dbg :hint "consumer events chunk ready"
|
||||
:consumer "admin-console"
|
||||
:uri-host uri-host
|
||||
:total total
|
||||
:skipped skipped)
|
||||
(if (empty? events)
|
||||
{:sent 0 :skipped skipped}
|
||||
;; Enrich first, outside the try: a local DB/decode failure is
|
||||
;; ours, not the destination's, and must not count as
|
||||
;; admin-console/failed.
|
||||
(let [enriched (enrich-events-for-admin-console cfg events)
|
||||
n (count events)]
|
||||
(try
|
||||
(nitrate/call cfg :ingest-audit-log {:events enriched})
|
||||
(l/dbg :hint "consumer events chunk sent"
|
||||
:consumer "admin-console"
|
||||
:uri-host uri-host
|
||||
:sent n
|
||||
:skipped skipped)
|
||||
{:sent n :skipped skipped}
|
||||
(catch Throwable cause
|
||||
(l/dbg :hint "consumer events chunk failed"
|
||||
:consumer "admin-console"
|
||||
:uri-host uri-host
|
||||
:events n
|
||||
:code :admin-console-send-failed)
|
||||
(throw (ex/raise :type :internal
|
||||
:code :admin-console-send-failed
|
||||
:hint "unable to archive events to the admin console"
|
||||
:cause cause
|
||||
:events n))))))))
|
||||
|
||||
(defn- consumer-nexus-chunk
|
||||
"Send one pending chunk to Nexus and complete it. Returns {:sent n}
|
||||
or nil when the queue is empty; throws on refusal. Takes a
|
||||
transactional cfg: the caller opens one transaction per chunk. No
|
||||
metrics here: a normal return means the transaction committed, so
|
||||
the caller records the success."
|
||||
[cfg]
|
||||
(db/exec! cfg ["SET LOCAL lock_timeout = '30s'"])
|
||||
(when-let [rows (get-pending-rows cfg consumer-nexus)]
|
||||
(let [events (mapv row->event (mapv decode-row rows))
|
||||
;; send-to-nexus returns the count or throws: a refusal
|
||||
;; aborts the job, the rows stay for the next tick.
|
||||
n (send-to-nexus cfg events)]
|
||||
|
||||
(l/trc :hint "consumer events chunk"
|
||||
:consumer "nexus"
|
||||
:events n)
|
||||
|
||||
(complete-batch cfg consumer-nexus rows)
|
||||
{:sent n})))
|
||||
|
||||
(defn- consumer-admin-console-chunk
|
||||
"Send one pending chunk to the admin console and complete it. Returns
|
||||
{:sent :skipped} or nil when the queue is empty; throws on failure.
|
||||
Takes a transactional cfg, no metrics here: same contract as
|
||||
consumer-nexus-chunk."
|
||||
[cfg]
|
||||
(db/exec! cfg ["SET LOCAL lock_timeout = '30s'"])
|
||||
(when-let [rows (get-pending-rows cfg consumer-admin-console)]
|
||||
(let [decoded (mapv decode-row rows)
|
||||
_ (l/trc :hint "consumer events chunk"
|
||||
:consumer "admin-console"
|
||||
:events (count decoded))
|
||||
admin-console-res (send-to-admin-console cfg decoded)]
|
||||
;; Filtered-out rows count as processed for the
|
||||
;; admin-console queue: their pending is done, the
|
||||
;; record stays until every consumer is done.
|
||||
(complete-batch cfg consumer-admin-console rows)
|
||||
admin-console-res)))
|
||||
|
||||
(defn- drain-consumer
|
||||
"Drain one consumer queue chunk by chunk, one transaction per chunk.
|
||||
Returns totals; a chunk the destination refuses (or that throws)
|
||||
aborts the drain with a throw, and the committed chunks so far
|
||||
stay sent."
|
||||
[cfg send-chunk]
|
||||
(loop [sent 0
|
||||
skipped 0
|
||||
chunks 0]
|
||||
;; One transaction per chunk: fetch, send and complete commit (or
|
||||
;; roll back) together, and the next chunk sees a fresh snapshot.
|
||||
(if-let [res (db/tx-run! cfg send-chunk)]
|
||||
(do
|
||||
(jobs/heartbeat cfg)
|
||||
(px/sleep 100)
|
||||
(recur (+ sent ^long (:sent res))
|
||||
(+ skipped ^long (:skipped res 0))
|
||||
(inc chunks)))
|
||||
{:sent sent :skipped skipped :chunks chunks})))
|
||||
|
||||
(defn- sql:reap-pendings
|
||||
[consumer]
|
||||
(str "DELETE FROM audit_log_pending"
|
||||
" WHERE consumer = " consumer
|
||||
" AND audit_log_id IN (SELECT audit_log_id FROM audit_log_pending"
|
||||
" WHERE consumer = " consumer
|
||||
" ORDER BY audit_log_id LIMIT 5000)"))
|
||||
|
||||
(def ^:private sql:sweep-orphans
|
||||
"DELETE FROM audit_log a
|
||||
WHERE a.id IN (SELECT a2.id FROM audit_log a2
|
||||
WHERE NOT EXISTS (SELECT 1 FROM audit_log_pending p
|
||||
WHERE p.audit_log_id = a2.id)
|
||||
ORDER BY a2.id LIMIT 5000)")
|
||||
|
||||
(defn- delete-batched
|
||||
"Run a batched DELETE until it removes nothing. Returns the total."
|
||||
[cfg sql]
|
||||
(loop [total 0]
|
||||
(let [n (db/get-update-count (db/exec-one! cfg [sql]))]
|
||||
(if (pos? n)
|
||||
(do
|
||||
(px/sleep 100)
|
||||
(recur (+ total ^long n)))
|
||||
total))))
|
||||
|
||||
(defn- reap-disabled
|
||||
"Drop the pendings of consumers that are off: a disabled destination
|
||||
will never drain them, and without this they would pin their records
|
||||
forever. This also absorbs any later deactivation, with no manual step.
|
||||
Dropped pendings count as discarded: their rows may still be archived
|
||||
by an enabled consumer, or swept unsent when nothing waits for them."
|
||||
[cfg]
|
||||
(let [total (->> consumers
|
||||
(remove #(contains? cf/flags (:flag %)))
|
||||
(map #(delete-batched cfg (sql:reap-pendings (:id %))))
|
||||
(reduce + 0))]
|
||||
(when (pos? total)
|
||||
(record-count (mtx/instance cfg) :audit-consumer-discarded ["reaped"] total))
|
||||
total))
|
||||
|
||||
(defn- sweep-orphans
|
||||
"Delete records nobody waits for anymore: rows born while no
|
||||
consumer was on, rows left pending-less by the reap, and leftovers
|
||||
archived by the old task. Batched and normally a no-op. Deleted rows
|
||||
count as discarded: on a live system they were never archived."
|
||||
[cfg]
|
||||
(let [total (delete-batched cfg sql:sweep-orphans)]
|
||||
(when (pos? total)
|
||||
(record-count (mtx/instance cfg) :audit-consumer-discarded ["swept"] total))
|
||||
total))
|
||||
|
||||
(declare execute-audit-log-consumer-nexus)
|
||||
(declare execute-audit-log-consumer-admin-console)
|
||||
|
||||
(def schema:audit-log-consumer-nexus-params
|
||||
"Overrides for the nexus job repl invocation defaults.
|
||||
Cron submits {}; system deps (pool, shared-keys, http) live on the
|
||||
job-def cfg closed over by the handler, not in job params.
|
||||
Closed on purpose: a stale key (like the old :enabled) must fail
|
||||
validation loudly instead of skipping silently."
|
||||
[:map {:closed true}
|
||||
[:uri {:optional true} ::sm/uri]])
|
||||
|
||||
(def schema:audit-log-consumer-admin-console-params
|
||||
"Params of the admin-console job: none. Cron submits {}; system deps
|
||||
(pool, admin-console client) live on the job-def cfg closed over by
|
||||
the handler, not in job params. Closed on purpose, same as the nexus
|
||||
schema."
|
||||
[:map {:closed true}])
|
||||
|
||||
(defmethod ig/assert-key ::nexus-job-def
|
||||
[_ cfg]
|
||||
(assert (db/pool? (::db/pool cfg)) "expected a valid database pool")
|
||||
(assert (http/client? (::http/client cfg)) "expected a valid http client")
|
||||
(assert (mtx/metrics? (::mtx/metrics cfg)) "expected a valid metrics instance"))
|
||||
|
||||
(defmethod ig/init-key ::nexus-job-def
|
||||
[_ cfg]
|
||||
;; Effective destination, once at boot: per-run logs carry counts,
|
||||
;; the runner says the job started, this says where it sends.
|
||||
(l/dbg :hint "audit log consumer job initialized"
|
||||
:consumer "nexus"
|
||||
:uri-host (some-> (cf/get :nexus-uri) u/uri u/get-domain))
|
||||
{::jobs/name :audit-log-consumer-nexus
|
||||
::jobs/schema schema:audit-log-consumer-nexus-params
|
||||
::jobs/handler
|
||||
(fn [_context params]
|
||||
(execute-audit-log-consumer-nexus cfg params))
|
||||
::jobs/decoder (sm/decoder schema:audit-log-consumer-nexus-params sm/json-transformer)
|
||||
::jobs/validator (sm/validator schema:audit-log-consumer-nexus-params)})
|
||||
|
||||
(defmethod ig/assert-key ::admin-console-job-def
|
||||
[_ cfg]
|
||||
;; No assert on the admin-console client: it inits to nil without the
|
||||
;; :admin-console flag, and the job skips on its flag anyway.
|
||||
(assert (db/pool? (::db/pool cfg)) "expected a valid database pool")
|
||||
(assert (mtx/metrics? (::mtx/metrics cfg)) "expected a valid metrics instance"))
|
||||
|
||||
(defmethod ig/init-key ::admin-console-job-def
|
||||
[_ cfg]
|
||||
;; Same as the nexus job-def: destination host once at boot. The
|
||||
;; ingest uri is resolved by nitrate/call at send time (it raises
|
||||
;; when missing); here it is display-only.
|
||||
(l/dbg :hint "audit log consumer job initialized"
|
||||
:consumer "admin-console"
|
||||
:uri-host (some-> (cf/get :admin-console-uri) u/uri u/get-domain))
|
||||
{::jobs/name :audit-log-consumer-admin-console
|
||||
::jobs/schema schema:audit-log-consumer-admin-console-params
|
||||
::jobs/handler
|
||||
(fn [_context params]
|
||||
(execute-audit-log-consumer-admin-console cfg params))
|
||||
::jobs/decoder (sm/decoder schema:audit-log-consumer-admin-console-params sm/json-transformer)
|
||||
::jobs/validator (sm/validator schema:audit-log-consumer-admin-console-params)})
|
||||
|
||||
(defn- run-single-consumer
|
||||
"Drain one consumer queue, then housekeeping (reap disabled
|
||||
pendings, sweep orphans) and the per-run summary. Housekeeping lives
|
||||
in both jobs: only the enabled job runs, so it must reap the queues
|
||||
of the consumers that are off; both halves are idempotent deletes.
|
||||
A refused chunk (or a throw) aborts the job: the committed chunks
|
||||
stay sent, the rest waits for the next tick, and the cron retries.
|
||||
Every effect is rows and logs (drain metrics land here, outside
|
||||
every chunk transaction)."
|
||||
[cfg consumer send-chunk tpoint]
|
||||
(let [label (consumer-label consumer)
|
||||
metrics (mtx/instance cfg)
|
||||
reaped (reap-disabled cfg)
|
||||
;; All drain metrics land here, outside every chunk
|
||||
;; transaction. Success is recorded on the totals: a normal
|
||||
;; chunk return means its transaction committed (a rollback
|
||||
;; always throws, taking the failure path instead). The send
|
||||
;; fns carry the attempted count as :events; anything else (a
|
||||
;; local DB failure) is not a destination failure and stays
|
||||
;; uncounted. The throw keeps rolling: a failed chunk aborts
|
||||
;; the job, the cron retries.
|
||||
drained (try
|
||||
(drain-consumer cfg send-chunk)
|
||||
(catch Throwable cause
|
||||
(let [n (:events (ex-data cause))]
|
||||
(when (and (integer? n) (pos? n))
|
||||
(record-count metrics :audit-consumer-events [label "failed"] n)))
|
||||
(throw cause)))
|
||||
_ (let [{:keys [sent skipped]} drained]
|
||||
(when (pos? sent)
|
||||
(record-count metrics :audit-consumer-events [label "archived"] sent))
|
||||
(when (pos? skipped)
|
||||
(record-count metrics :audit-consumer-events [label "skipped"] skipped)))
|
||||
swept (sweep-orphans cfg)
|
||||
sent (:sent drained)
|
||||
skipped (:skipped drained)
|
||||
chunks (:chunks drained)
|
||||
;; Strict binding on purpose: log props evaluate lazily in
|
||||
;; the logger thread, so the elapsed must be frozen here.
|
||||
elapsed (ct/format-duration (tpoint))]
|
||||
(if (and (zero? sent)
|
||||
(zero? skipped)
|
||||
(zero? reaped)
|
||||
(zero? swept))
|
||||
(l/dbg :hint "no pending audit events"
|
||||
:consumer label)
|
||||
(l/dbg :hint "events archived"
|
||||
:consumer label
|
||||
:elapsed elapsed
|
||||
:sent sent
|
||||
:skipped skipped
|
||||
:chunks chunks
|
||||
:reaped reaped
|
||||
:swept swept))))
|
||||
|
||||
(defn execute-audit-log-consumer-nexus
|
||||
"Nexus job handler: drain the nexus queue in chunks, then clean up
|
||||
(heartbeat per iteration: the sent chunk batches can be long)."
|
||||
[cfg params]
|
||||
;; NOTE: this let allows overwrite default configured values from
|
||||
;; the repl, when manually invoking the task.
|
||||
(if-not (contains? cf/flags :nexus)
|
||||
(l/dbg :hint "audit log consumer (nexus) skipped, flag off")
|
||||
(let [uri (or (:uri params) (generate-nexus-uri "api/audit"))
|
||||
;; Normalize to an uri object; params may carry a plain string
|
||||
;; on direct invocations (validation only applies on submit!).
|
||||
uri (u/uri uri)
|
||||
cfg (assoc cfg ::uri uri)]
|
||||
(when-not uri
|
||||
(ex/raise :type :internal
|
||||
:code :task-not-configured
|
||||
:hint "consumer task not configured, missing uri"))
|
||||
(run-single-consumer cfg consumer-nexus consumer-nexus-chunk (ct/tpoint)))))
|
||||
|
||||
(defn execute-audit-log-consumer-admin-console
|
||||
"Admin-console job handler: drain the admin-console queue in chunks,
|
||||
then clean up (heartbeat per iteration: the sent chunk batches can
|
||||
be long)."
|
||||
[cfg _params]
|
||||
(if-not (contains? cf/flags :admin-console)
|
||||
(l/dbg :hint "audit log consumer (admin-console) skipped, flag off")
|
||||
(run-single-consumer cfg consumer-admin-console consumer-admin-console-chunk (ct/tpoint))))
|
||||
532
backend/src/app/loggers/audit/consumers.clj
Normal file
532
backend/src/app/loggers/audit/consumers.clj
Normal file
@ -0,0 +1,532 @@
|
||||
;; This Source Code Form is subject to the terms of the Mozilla Public
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.loggers.audit.consumers
|
||||
"The mechanics shared by the consumers of the audit log buffer.
|
||||
|
||||
Three consumers read the events of the buffer, each one from its own
|
||||
position (a cursor) over `audit_log`:
|
||||
|
||||
- the nexus consumer sends the full rows to Nexus.
|
||||
- the admin-console consumer sends the allowlisted full rows to the
|
||||
admin console.
|
||||
- the telemetry consumer sends the anonymized copies (the shadow
|
||||
rows, `source LIKE 'telemetry:%'`) to the telemetry endpoint.
|
||||
|
||||
Each transport lives in its own namespace (`...consumers.nexus`,
|
||||
`...consumers.admin-console`, `...consumers.telemetry`); this one
|
||||
holds what they share: the consumer ids, validation and generic pass
|
||||
runner, the cursor (lock, read, advance), the tick mutex, the loop and
|
||||
the metrics. Each transport exports its own consumer definition. The
|
||||
garbage collection of the buffer lives in `app.loggers.audit.gc`.
|
||||
|
||||
A consumer is a loop in a thread, not a job: the pass takes the tick
|
||||
lock, drains up to the frontier it found at its start and sleeps. The
|
||||
position is the pair `(xid, seq)`: the transaction that wrote the row
|
||||
and its insertion order. A consumer reads only rows below
|
||||
`pg_snapshot_xmin(pg_current_snapshot())` (the oldest transaction still
|
||||
in flight), so it never steps over a row whose transaction is still
|
||||
open. The producer does not know about consumers: it only inserts the
|
||||
row, and the GC deletes it once every enabled consumer has passed it."
|
||||
(:require
|
||||
[app.common.logging :as l]
|
||||
[app.common.time :as ct]
|
||||
[app.config :as cf]
|
||||
[app.db :as db]
|
||||
[app.metrics :as mtx]
|
||||
[promesa.exec :as px]))
|
||||
|
||||
;; Consumer ids are fixed constants, never reuse one: a retired id keeps
|
||||
;; meaning its old position in `audit_log_consumer_cursor`. Id 0 is the
|
||||
;; stream of anonymized copies (telemetry shadows), apart from the
|
||||
;; full-row consumers that take 1 and up.
|
||||
(def telemetry 0)
|
||||
(def nexus 1)
|
||||
(def admin-console 2)
|
||||
|
||||
(defn- check
|
||||
"Assert the shape of a consumer definition: the keys the framework needs to
|
||||
run its loop. A plain map fails at the use without this."
|
||||
[consumer]
|
||||
(assert (integer? (:id consumer)) "consumer id must be an integer")
|
||||
(assert (keyword? (:flag consumer)) "consumer flag must be a keyword")
|
||||
(assert (string? (:label consumer)) "consumer label must be a string")
|
||||
(assert (fn? (:process consumer)) "consumer process must be a function")
|
||||
(assert (or (nil? (:prepare consumer)) (fn? (:prepare consumer)))
|
||||
"consumer prepare must be a function")
|
||||
(assert (or (nil? (:options consumer)) (map? (:options consumer)))
|
||||
"consumer options must be a map"))
|
||||
|
||||
(defn enabled?
|
||||
"True when the flag of a consumer is on. The components use it at boot to
|
||||
decide whether to start the loop; a direct pass of a disabled consumer
|
||||
answers ::disabled."
|
||||
[consumer]
|
||||
(contains? cf/flags (:flag consumer)))
|
||||
|
||||
(defn decode-row
|
||||
"Decode the transit-encoded columns of an audit-log row. The
|
||||
transports read the rows of their chunk, so this is shared."
|
||||
[{:keys [props ip-addr context] :as row}]
|
||||
(cond-> row
|
||||
(db/pgobject? props)
|
||||
(assoc :props (db/decode-transit-pgobject props))
|
||||
|
||||
(db/pgobject? context)
|
||||
(assoc :context (db/decode-transit-pgobject context))
|
||||
|
||||
(db/pgobject? ip-addr "inet")
|
||||
(assoc :ip-addr (db/decode-inet ip-addr))))
|
||||
|
||||
(def ^:private event-keys
|
||||
[:type
|
||||
:name
|
||||
:source
|
||||
:created-at
|
||||
:tracked-at
|
||||
:profile-id
|
||||
:ip-addr
|
||||
:props
|
||||
:context])
|
||||
|
||||
(defn row->event
|
||||
"The view of an audit-log row that travels to an archive: the decoded
|
||||
event alone, without the row id or the cursor columns."
|
||||
[row]
|
||||
(select-keys row event-keys))
|
||||
|
||||
(defn record-count
|
||||
"Record a counter of the consumers metrics."
|
||||
[metrics id labels amount]
|
||||
(mtx/run! metrics :id id :labels labels :inc amount))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; CURSOR
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(def ^:private chunk-size
|
||||
"Rows per chunk. Kept in a var so tests can shrink it."
|
||||
128)
|
||||
|
||||
(def ^:private sql:lock-cursor
|
||||
"Lock the position of one consumer for the duration of the chunk.
|
||||
The lock is on the cursor row, which takes part in no foreign key:
|
||||
two passes of the same consumer serialize here, and no row lock on
|
||||
`audit_log` is needed (the GC never locks the cursor). The tick mutex
|
||||
is the advisory lock of `with-tick-lock`; this row lock is the safety
|
||||
net for a pass that bypasses the loop (a REPL against a running
|
||||
consumer, for instance).
|
||||
|
||||
`xid::text` is the JDBC bridge, not a sorting choice: `xid8` is an
|
||||
unsigned 64-bit transaction counter with no Java type and no standard
|
||||
driver mapping, so it crosses as text and goes back to SQL as
|
||||
`?::xid8`. The alias is `xid_text` on purpose: an output column named
|
||||
`xid` would make `ORDER BY xid` sort the text."
|
||||
"SELECT xid::text AS xid_text, seq
|
||||
FROM audit_log_consumer_cursor
|
||||
WHERE consumer = ?::smallint
|
||||
FOR UPDATE")
|
||||
|
||||
(def ^:private sql:get-cursor
|
||||
"SELECT xid::text AS xid_text, seq
|
||||
FROM audit_log_consumer_cursor
|
||||
WHERE consumer = ?::smallint")
|
||||
|
||||
(def ^:private sql:ensure-cursor
|
||||
"A consumer with no cursor starts at the beginning of the buffer: it
|
||||
processes what the buffer holds, not only what arrives from now on.
|
||||
(0, 0) sits below any real position, so the first chunk includes the
|
||||
oldest row; the table starts empty because the migration cannot know
|
||||
which consumers are enabled. ON CONFLICT keeps an existing position
|
||||
untouched."
|
||||
"INSERT INTO audit_log_consumer_cursor (consumer, xid, seq)
|
||||
VALUES (?::smallint, '0'::xid8, 0)
|
||||
ON CONFLICT (consumer) DO NOTHING")
|
||||
|
||||
(defn ensure-cursor
|
||||
"Create the cursor of a consumer when it does not exist yet: at the
|
||||
beginning of the buffer, so the first pass drains what is there. The
|
||||
cursor means 'last row processed' and the chunk query is a strict
|
||||
`(xid, seq) > cursor`, so the starting position must sit *before* the
|
||||
oldest row: (0, 0) is below any real position and needs no extra read.
|
||||
|
||||
The GC re-reads the cursors on every batch, so a cursor registered
|
||||
between two batches is seen by the next one. A cursor registered while a
|
||||
batch is between its read and its delete is not seen by that batch: the
|
||||
rows it deletes are lost for a consumer that starts at that exact moment
|
||||
(at most one batch per concurrent GC run that read the previous cursor).
|
||||
Two instances registering the same
|
||||
consumer are safe: `ON CONFLICT DO NOTHING` keeps the existing row."
|
||||
[cfg consumer]
|
||||
(db/tx-run! cfg (fn [cfg] (db/exec-one! cfg [sql:ensure-cursor consumer]))))
|
||||
|
||||
(def ^:private sql:chunk
|
||||
"The next chunk of events for one consumer.
|
||||
|
||||
The tuple comparison walks the `(xid, seq)` index from the cursor; the
|
||||
snapshot cut leaves out every row whose transaction is still in flight,
|
||||
and the watermark bounds the pass to what was visible when it started,
|
||||
so the pass always ends. The order is the insertion order that commits
|
||||
respect: a transaction with a lower `xid` committed (or aborted) before
|
||||
a higher one can be visible, and `seq` unties the rows of a single
|
||||
transaction.
|
||||
|
||||
No `SKIP LOCKED` and no row lock: the tick mutex and the cursor lock
|
||||
already serialize the passes of this consumer, and the rows are only
|
||||
read."
|
||||
"SELECT a.id,
|
||||
a.name,
|
||||
a.type,
|
||||
a.source,
|
||||
a.created_at,
|
||||
a.tracked_at,
|
||||
a.profile_id,
|
||||
a.ip_addr,
|
||||
%s AS props,
|
||||
%s AS context,
|
||||
a.seq,
|
||||
a.xid::text AS xid_text
|
||||
FROM audit_log a
|
||||
WHERE (a.xid, a.seq) > (?::xid8, ?::bigint)
|
||||
AND (a.xid, a.seq) <= (?::xid8, ?::bigint)
|
||||
AND a.xid <= pg_snapshot_xmin(pg_current_snapshot())
|
||||
ORDER BY a.xid ASC, a.seq ASC
|
||||
LIMIT ?")
|
||||
|
||||
(def ^:private sql:full-chunk
|
||||
(format sql:chunk "a.props" "a.context"))
|
||||
|
||||
(def ^:private sql:telemetry-chunk
|
||||
(format sql:chunk
|
||||
"CASE WHEN a.source LIKE 'telemetry:%' THEN a.props END"
|
||||
"CASE WHEN a.source LIKE 'telemetry:%' THEN a.context END"))
|
||||
|
||||
(def ^:private sql:last-visible
|
||||
"The frontier of the pass: the newest row whose transaction is already
|
||||
committed, with the same visibility rule the chunk uses. The pass drains
|
||||
up to it, so it always ends and every chunk but the last is full.
|
||||
|
||||
The `ORDER BY` names the table column: `xid` alone would resolve to the
|
||||
text projection (`xid_text`) and sort '999' above '1000'."
|
||||
"SELECT xid::text AS xid_text, seq
|
||||
FROM audit_log
|
||||
WHERE xid <= pg_snapshot_xmin(pg_current_snapshot())
|
||||
ORDER BY audit_log.xid DESC, seq DESC
|
||||
LIMIT 1")
|
||||
|
||||
(def ^:private sql:advance-cursor
|
||||
"UPDATE audit_log_consumer_cursor
|
||||
SET xid = ?::xid8, seq = ?::bigint, updated_at = now()
|
||||
WHERE consumer = ?::smallint")
|
||||
|
||||
(defn- lock-cursor
|
||||
[cfg consumer]
|
||||
(db/exec-one! cfg [sql:lock-cursor consumer]))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; TICK
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(def ^:private tick-lock-classid
|
||||
"Namespace of the two-key advisory locks of the consumers. The two-key
|
||||
form lives in its own lock space, disjoint from the one-key
|
||||
`pg_advisory_lock(bigint)` that `update-file` and the setup take, so a
|
||||
consumer tick can never block a file update (nor the other way around)
|
||||
even if the numbers coincide."
|
||||
730000)
|
||||
|
||||
(defn- tick-lock-key
|
||||
"The two-key advisory lock of one consumer: the namespace of the audit
|
||||
log and the consumer id."
|
||||
[consumer-id]
|
||||
[tick-lock-classid consumer-id])
|
||||
|
||||
(defn- with-tick-lock
|
||||
"Run `f` with the session advisory lock of the consumer held, passing it
|
||||
the connection that holds the lock; answer ::busy without running it
|
||||
when another instance is draining.
|
||||
|
||||
The lock lives on a connection borrowed from the pool and is released
|
||||
before that connection goes back, because a pooled connection returned
|
||||
with a session lock keeps it. A lock an earlier pass failed to release
|
||||
is cleared when the connection is borrowed again, so one bad unlock
|
||||
cannot stop the consumer for good. A failed unlock is logged and
|
||||
swallowed: it must not replace the error of the pass that is unwinding."
|
||||
[{:keys [::db/pool]} consumer f]
|
||||
(with-open [conn (db/open pool)]
|
||||
(db/session-unlock-all conn)
|
||||
(let [[classid objid] (tick-lock-key (:id consumer))]
|
||||
(if (db/session-try-lock conn classid objid)
|
||||
(try
|
||||
(f conn)
|
||||
(finally
|
||||
(try
|
||||
(db/session-unlock conn classid objid)
|
||||
(catch Throwable cause
|
||||
(l/wrn :hint "unable to release the audit-consumer lock"
|
||||
:consumer (:label consumer)
|
||||
:cause cause)))))
|
||||
::busy))))
|
||||
|
||||
(defn- get-chunk-rows
|
||||
[cfg consumer cursor watermark]
|
||||
(->> (db/exec! cfg [(if (= telemetry (:id consumer))
|
||||
sql:telemetry-chunk
|
||||
sql:full-chunk)
|
||||
(:xid-text cursor) (:seq cursor)
|
||||
(:xid-text watermark) (:seq watermark)
|
||||
(or (:chunk-size (:options consumer)) chunk-size)])
|
||||
(not-empty)))
|
||||
|
||||
(defn- advance-cursor
|
||||
[{:keys [::db/conn]} consumer row]
|
||||
(db/exec-one! conn [sql:advance-cursor (:xid-text row) (:seq row) consumer]))
|
||||
|
||||
(defn- drain-chunks
|
||||
"Drain one consumer cursor chunk by chunk, one transaction per chunk, up
|
||||
to the watermark of the pass. The processor receives the rows of each
|
||||
chunk and returns {:processed n :skipped n}; the cursor advances past
|
||||
every row of the chunk in the same transaction, so a refusal (or a
|
||||
throw) rolls the advance back and the chunk is retried on the next pass.
|
||||
The send runs inside that transaction too: the `FOR UPDATE` on the
|
||||
cursor row assigns an xid, so a slow destination holds
|
||||
`pg_snapshot_xmin` back for every consumer until the chunk ends.
|
||||
`totals` is updated after each committed chunk, so a pass that aborts
|
||||
later still reports what it already archived. Answers the totals."
|
||||
[cfg conn consumer process-chunk watermark totals]
|
||||
(loop []
|
||||
;; One transaction per chunk on the connection that holds the tick:
|
||||
;; fetch, process and advance commit (or roll back) together, and the
|
||||
;; next chunk sees a fresh snapshot. A pass needs no second connection.
|
||||
(if-let [res (db/tx-run! (-> cfg (dissoc ::db/pool) (assoc ::db/conn conn))
|
||||
(fn [cfg]
|
||||
(db/exec! cfg ["SET LOCAL lock_timeout = '30s'"])
|
||||
(let [consumer-id (:id consumer)
|
||||
cursor (or (lock-cursor cfg consumer-id)
|
||||
(do (ensure-cursor cfg consumer-id)
|
||||
(lock-cursor cfg consumer-id)))]
|
||||
(when-let [rows (get-chunk-rows cfg consumer cursor watermark)]
|
||||
(let [res (process-chunk cfg rows)]
|
||||
(advance-cursor cfg consumer-id (peek rows))
|
||||
res)))))]
|
||||
(do
|
||||
;; Committed: its rows count even if a later chunk fails.
|
||||
(vswap! totals (fn [{:keys [processed skipped chunks]}]
|
||||
{:processed (+ (long processed) (long (or (:processed res) 0)))
|
||||
:skipped (+ (long skipped) (long (or (:skipped res) 0)))
|
||||
:chunks (inc (long chunks))}))
|
||||
(px/sleep 100)
|
||||
(recur))
|
||||
@totals)))
|
||||
|
||||
(def ^:private sql:lag
|
||||
"Events confirmed in the buffer that this consumer has not processed
|
||||
yet, capped: past 10000 the gauge reads 10000, so a long backlog does
|
||||
not turn every pass into an O(backlog) count. It rises when the stream
|
||||
falls behind (a slow destination, or a big transaction that just
|
||||
committed); while a transaction is open its rows are invisible to
|
||||
everybody, so they cannot be counted here."
|
||||
"SELECT count(*) AS lag
|
||||
FROM (SELECT 1
|
||||
FROM audit_log a
|
||||
WHERE (a.xid, a.seq) > (?::xid8, ?::bigint)
|
||||
LIMIT 10000) s")
|
||||
|
||||
(defn- record-lag
|
||||
[conn metrics consumer]
|
||||
(when-let [cursor (db/exec-one! conn [sql:get-cursor (:id consumer)])]
|
||||
(let [lag (:lag (db/exec-one! conn [sql:lag (:xid-text cursor) (:seq cursor)]))]
|
||||
(mtx/run! metrics
|
||||
:id :audit-consumer-lag
|
||||
:labels [(:label consumer)]
|
||||
:val lag))))
|
||||
|
||||
(defn- record-totals
|
||||
"Record the event counters of the chunks a pass committed. Called on the
|
||||
happy path and from the catch, so a pass that aborts on chunk N still
|
||||
counts chunks 1..N-1."
|
||||
[metrics label {:keys [processed skipped]}]
|
||||
(when (pos? processed)
|
||||
(record-count metrics :audit-consumer-events [label "processed"] processed))
|
||||
(when (pos? skipped)
|
||||
(record-count metrics :audit-consumer-events [label "skipped"] skipped)))
|
||||
|
||||
(defn- drain
|
||||
"Run one pass of a consumer: take the tick lock, drain its cursor chunk by
|
||||
chunk (one transaction per chunk) up to the frontier of the pass, and
|
||||
record the summary. Answers ::busy without doing anything when another
|
||||
instance is draining this consumer, and ::disabled without touching the
|
||||
data when the consumer's flag is off: the loop is gated at boot, but a
|
||||
direct caller (REPL, tests, a helper) must not advance a cursor or send
|
||||
events for a consumer the deployment has disabled.
|
||||
|
||||
The processor receives the rows of each chunk and returns
|
||||
{:processed n :skipped n}; the cursor advances past every row of the
|
||||
chunk, so a selective consumer still advances over the events it does
|
||||
not take (they are counted as skipped, not lost: the next chunk
|
||||
continues from there).
|
||||
|
||||
The frontier is the newest visible row when the pass starts, so the pass
|
||||
always ends and a row committed while it runs waits for the next one. A
|
||||
refused chunk (or a throw) aborts the pass: the committed chunks stay
|
||||
processed, the rest waits for the next pass. The whole pass runs on the
|
||||
connection that holds the tick (frontier, chunk transactions and lag), so
|
||||
the pool needs one connection per enabled consumer. The metrics land
|
||||
here, outside every chunk transaction: `penpot_audit_consumer_passes_total`
|
||||
counts `ok` (drained or empty), `busy` (another instance holds the tick)
|
||||
and `failed` (anything the pass throws: the frontier read, a chunk or the
|
||||
lag query), so a failure cannot leave the family flat."
|
||||
[cfg consumer]
|
||||
(if-not (enabled? consumer)
|
||||
::disabled
|
||||
(let [label (:label consumer)
|
||||
process (:process consumer)
|
||||
metrics (mtx/instance cfg)
|
||||
totals (volatile! {:processed 0 :skipped 0 :chunks 0})
|
||||
result (with-tick-lock cfg consumer
|
||||
(fn [conn]
|
||||
(try
|
||||
(if-let [watermark (db/exec-one! conn [sql:last-visible])]
|
||||
(let [tpoint (ct/tpoint)
|
||||
drained (drain-chunks cfg conn consumer process watermark totals)
|
||||
_ (record-lag conn metrics consumer)
|
||||
_ (record-totals metrics label drained)
|
||||
_ (record-count metrics :audit-consumer-passes [label "ok"] 1)
|
||||
processed (:processed drained)
|
||||
skipped (:skipped drained)
|
||||
chunks (:chunks drained)
|
||||
;; Strict binding on purpose: log props evaluate lazily in
|
||||
;; the logger thread, so the elapsed must be frozen here.
|
||||
elapsed (ct/format-duration (tpoint))]
|
||||
(if (and (zero? processed)
|
||||
(zero? skipped))
|
||||
(l/trc :hint "no new audit events"
|
||||
:consumer label)
|
||||
(l/dbg :hint "audit events processed"
|
||||
:consumer label
|
||||
:elapsed elapsed
|
||||
:processed processed
|
||||
:skipped skipped
|
||||
:chunks chunks)))
|
||||
;; Nothing was visible when the pass started: no chunk query.
|
||||
(do
|
||||
(record-lag conn metrics consumer)
|
||||
(record-count metrics :audit-consumer-passes [label "ok"] 1)
|
||||
(l/trc :hint "no new audit events"
|
||||
:consumer label)
|
||||
nil))
|
||||
(catch Throwable cause
|
||||
;; The counter covers the whole pass: a failed frontier
|
||||
;; read or lag query counts as a failed pass too, not
|
||||
;; only a refused chunk.
|
||||
(record-totals metrics label @totals)
|
||||
(record-count metrics :audit-consumer-passes [label "failed"] 1)
|
||||
;; The failed event count comes from the error data: the
|
||||
;; send fns carry the attempted events as :events. A local
|
||||
;; DB failure carries none and is not counted as a
|
||||
;; destination failure.
|
||||
(let [n (:events (ex-data cause))]
|
||||
(when (and (integer? n) (pos? n))
|
||||
(record-count metrics :audit-consumer-events [label "failed"] n)))
|
||||
(throw cause)))))]
|
||||
(if (= ::busy result)
|
||||
(do
|
||||
(record-count metrics :audit-consumer-passes [label "busy"] 1)
|
||||
::busy)
|
||||
result))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; LOOP
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(def ^:private sleep-ms
|
||||
"Pause after every pass. Long enough to keep the database quiet while the
|
||||
buffer is empty, short enough that an event waits seconds, not minutes.
|
||||
A consumer may declare its own `:pause-ms` in its `:options`."
|
||||
5000)
|
||||
|
||||
(defn run-pass
|
||||
"Run one pass of a consumer with its prepared cfg. Returns nil on success
|
||||
and publishes totals as metrics. Returns ::busy when another instance
|
||||
holds the tick or ::disabled when the flag is off."
|
||||
[cfg consumer]
|
||||
(check consumer)
|
||||
(drain cfg consumer))
|
||||
|
||||
(defn start-loop
|
||||
"Start the daemon thread of one consumer: a pass, a pause, forever.
|
||||
|
||||
The pass may throw: the loop logs the failure and keeps going, because a
|
||||
consumer that stops on the first refused chunk stops for good. Answers the
|
||||
thread, to be interrupted on halt."
|
||||
[cfg consumer]
|
||||
(check consumer)
|
||||
(let [label (:label consumer)]
|
||||
(px/thread
|
||||
{:name (str "penpot/audit-consumer/" label)}
|
||||
(l/inf :hint "started" :consumer label)
|
||||
(try
|
||||
(loop []
|
||||
(when (px/interrupted?)
|
||||
(throw (InterruptedException. "interrupted")))
|
||||
|
||||
(try
|
||||
(run-pass cfg consumer)
|
||||
(catch InterruptedException cause
|
||||
(throw cause))
|
||||
(catch Throwable cause
|
||||
(l/err :hint "consumer pass failed, retrying after the pause"
|
||||
:consumer label
|
||||
:cause cause)))
|
||||
|
||||
(px/sleep (or (:pause-ms (:options consumer)) sleep-ms))
|
||||
(recur))
|
||||
|
||||
(catch InterruptedException _
|
||||
(l/dbg :hint "interrupted" :consumer label))
|
||||
(catch Throwable cause
|
||||
(l/err :hint "unexpected exception" :consumer label :cause cause))
|
||||
(finally
|
||||
(l/inf :hint "terminated" :consumer label))))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; COMPONENT
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(defn init
|
||||
"Start the loop of a consumer when its flag is on and the database is
|
||||
writable; nil otherwise. The optional `:prepare` of the definition runs
|
||||
only on the start path and answers the cfg the pass will use (Nexus
|
||||
resolves its uri there).
|
||||
|
||||
The cursor is registered after `prepare` succeeds, so a consumer whose
|
||||
configuration is rejected leaves no row behind."
|
||||
[cfg consumer]
|
||||
(check consumer)
|
||||
(cond
|
||||
(not (enabled? consumer))
|
||||
(l/dbg :hint "not started (flag off)" :consumer (:label consumer))
|
||||
|
||||
(db/read-only? (::db/pool cfg))
|
||||
(l/wrn :hint "not started (db is read-only)" :consumer (:label consumer))
|
||||
|
||||
:else
|
||||
(let [cfg (if-let [prepare (:prepare consumer)]
|
||||
(prepare cfg)
|
||||
cfg)]
|
||||
;; Register the consumer before the loop reads anything: the GC waits
|
||||
;; for its cursor, so the row must exist even if the first pass has
|
||||
;; nothing visible to drain yet.
|
||||
(ensure-cursor cfg (:id consumer))
|
||||
(start-loop cfg consumer))))
|
||||
|
||||
(defn halt
|
||||
"Stop the loop of a consumer and wait for it. Tolerates the nil an init
|
||||
that skipped the start answers."
|
||||
[thread]
|
||||
(when thread
|
||||
(px/interrupt! thread)
|
||||
(.join ^Thread thread 5000)))
|
||||
309
backend/src/app/loggers/audit/consumers/admin_console.clj
Normal file
309
backend/src/app/loggers/audit/consumers/admin_console.clj
Normal file
@ -0,0 +1,309 @@
|
||||
;; This Source Code Form is subject to the terms of the Mozilla Public
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.loggers.audit.consumers.admin-console
|
||||
"The admin console transport of the audit log consumer.
|
||||
|
||||
It reads the stream from its own cursor and ingests the allowlisted
|
||||
events, enriched with the name of their team and the emails of the
|
||||
profiles they mention (the admin console shows people, not ids). The
|
||||
events outside the allowlist, and the telemetry shadows, count as
|
||||
skipped for the cursor: they are passed over and never reach the
|
||||
destination.
|
||||
|
||||
The allowlist lives here: it is this consumer's contract, not the
|
||||
producer's (the producer only inserts rows). The component starts the
|
||||
consumer loop at boot when its flag is on; the mechanics (the tick
|
||||
mutex, the cursor, the frontier of the pass, the GC) live in
|
||||
`app.loggers.audit.consumers`."
|
||||
(:require
|
||||
[app.common.exceptions :as ex]
|
||||
[app.common.logging :as l]
|
||||
[app.common.uri :as u]
|
||||
[app.common.uuid :as uuid]
|
||||
[app.config :as cf]
|
||||
[app.db :as db]
|
||||
[app.loggers.audit :as audit]
|
||||
[app.loggers.audit.consumers :as consumers]
|
||||
[app.metrics :as mtx]
|
||||
[app.nitrate :as nitrate]
|
||||
[integrant.core :as ig]))
|
||||
|
||||
(def event-names
|
||||
"The event names the admin console archives. Each consumer owns its
|
||||
filter now: the producer only inserts the row, and this consumer
|
||||
decides on read which events concern it."
|
||||
#{"accept-organization-invitation"
|
||||
"accept-organization-invitation-from"
|
||||
"accept-team-invitation"
|
||||
"accept-team-invitation-from"
|
||||
"add-member-to-organization"
|
||||
"add-team-to-organization"
|
||||
"assign-organization-attribute"
|
||||
"auditlog-exported"
|
||||
"auditlog-guide-downloaded"
|
||||
"cancel-organization-invitation"
|
||||
"create-export-binfile-job"
|
||||
"create-file"
|
||||
"create-organization"
|
||||
"create-organization-attribute"
|
||||
"create-organization-invitation"
|
||||
"create-project"
|
||||
"create-team"
|
||||
"create-team-access-request"
|
||||
"create-team-invitation"
|
||||
"create-team-invitations"
|
||||
"create-webhook"
|
||||
"delete-file"
|
||||
"delete-font"
|
||||
"delete-organization"
|
||||
"delete-organization-attribute"
|
||||
"delete-organization-member"
|
||||
"delete-project"
|
||||
"delete-team"
|
||||
"delete-team-invitation"
|
||||
"delete-team-member"
|
||||
"leave-team"
|
||||
"move-project"
|
||||
"move-team-to-organization"
|
||||
"organization-sso-auth-failed"
|
||||
"organization-sso-auth-started"
|
||||
"organization-sso-auth-succeeded"
|
||||
"organization-sso-token-issued"
|
||||
"organization-sso-token-revoked"
|
||||
"permanently-delete-team-files"
|
||||
"remove-organization-team"
|
||||
"remove-team-from-organization"
|
||||
"rename-organization"
|
||||
"rename-project"
|
||||
"restore-deleted-team-files"
|
||||
"sso-config-deleted"
|
||||
"sso-config-tested"
|
||||
"success-activate-organization-sso"
|
||||
"success-apply-organization-sso-changes"
|
||||
"unassign-organization-attribute"
|
||||
"update-organization-attribute"
|
||||
"update-organization-invitation"
|
||||
"update-organization-permissions"
|
||||
"update-team-invitation"
|
||||
"update-team-invitation-role"
|
||||
"update-team-member-role"
|
||||
"update-team-photo"
|
||||
"verify-token"})
|
||||
|
||||
;; Props id key -> email prop filled from profile.email (admin-console path only).
|
||||
;; Some producers store ids as UUID strings (e.g. :user-who-send-invitation).
|
||||
(def ^:private id->email-prop
|
||||
{:profile-id :profile-email
|
||||
:user-id :user-email
|
||||
:member-id :member-email
|
||||
:user-who-send-invitation :user-who-send-invitation-email})
|
||||
|
||||
(defn- collect-team-ids-needing-name
|
||||
[events]
|
||||
(into #{}
|
||||
(comp (map :props)
|
||||
(keep (fn [props]
|
||||
(when-let [team-id (and (not (contains? props :team-name))
|
||||
(uuid/coerce (:team-id props)))]
|
||||
team-id))))
|
||||
events))
|
||||
|
||||
(defn- collect-profile-ids-needing-email
|
||||
[events]
|
||||
(into #{}
|
||||
(mapcat (fn [{:keys [props]}]
|
||||
(keep (fn [[id-key email-key]]
|
||||
(when-not (contains? props email-key)
|
||||
(uuid/coerce (get props id-key))))
|
||||
id->email-prop)))
|
||||
events))
|
||||
|
||||
(defn- collect-actor-ids-needing-profile
|
||||
"Actor ids (event :profile-id) that still need top-level :profile-email
|
||||
and/or :profile-name. Skips uuid/zero and non-uuids."
|
||||
[events]
|
||||
(into #{}
|
||||
(keep (fn [event]
|
||||
(when (or (not (contains? event :profile-email))
|
||||
(not (contains? event :profile-name)))
|
||||
(when-let [id (uuid/coerce (:profile-id event))]
|
||||
(when-not (uuid/zero? id)
|
||||
id)))))
|
||||
events))
|
||||
|
||||
(defn- load-team-names
|
||||
[conn ids]
|
||||
(if (seq ids)
|
||||
(let [arr (db/create-array conn "uuid" ids)
|
||||
rows (db/exec! conn ["SELECT id, name FROM team WHERE id = ANY(?)" arr])]
|
||||
(into {} (map (juxt :id :name) rows)))
|
||||
{}))
|
||||
|
||||
(defn- load-profiles
|
||||
"Batch-load profile email and fullname (including soft-deleted)."
|
||||
[conn ids]
|
||||
(if (seq ids)
|
||||
(let [arr (db/create-array conn "uuid" ids)
|
||||
rows (db/exec! conn ["SELECT id, email, fullname FROM profile WHERE id = ANY(?)" arr])]
|
||||
(into {} (map (juxt :id identity) rows)))
|
||||
{}))
|
||||
|
||||
(defn- maybe-assoc-team-name
|
||||
[event team-names]
|
||||
(let [props (:props event)]
|
||||
(if-let [team-name (and (not (contains? props :team-name))
|
||||
(when-let [team-id (uuid/coerce (:team-id props))]
|
||||
(get team-names team-id)))]
|
||||
(assoc-in event [:props :team-name] team-name)
|
||||
event)))
|
||||
|
||||
(defn- maybe-assoc-emails
|
||||
[event profiles]
|
||||
(update event :props
|
||||
(fn [props]
|
||||
(reduce-kv (fn [props id-key email-key]
|
||||
(if-let [email (and (not (contains? props email-key))
|
||||
(when-let [id (uuid/coerce (get props id-key))]
|
||||
(:email (get profiles id))))]
|
||||
(assoc props email-key email)
|
||||
props))
|
||||
props
|
||||
id->email-prop))))
|
||||
|
||||
(defn- maybe-assoc-actor-profile
|
||||
"Fill-only top-level :profile-email / :profile-name from the event
|
||||
actor (:profile-id). Omits keys when the id is missing, zero, the
|
||||
profile row is absent, or the corresponding column is empty
|
||||
(fullname defaults to \"\" in the schema)."
|
||||
[event profiles]
|
||||
(let [id (uuid/coerce (:profile-id event))]
|
||||
(if (or (nil? id) (uuid/zero? id))
|
||||
event
|
||||
(if-let [profile (get profiles id)]
|
||||
(cond-> event
|
||||
(and (not (contains? event :profile-email))
|
||||
(not-empty (:email profile)))
|
||||
(assoc :profile-email (:email profile))
|
||||
|
||||
(and (not (contains? event :profile-name))
|
||||
(not-empty (:fullname profile)))
|
||||
(assoc :profile-name (:fullname profile)))
|
||||
event))))
|
||||
|
||||
(defn- enrich-events
|
||||
"Enrich allowlisted events before the admin-console ingest:
|
||||
1. Fill missing :team-name from team id (including soft-deleted).
|
||||
2. Fill missing emails from matching id keys in props (including
|
||||
soft-deleted profiles): :profile-email, :user-email, :member-email,
|
||||
:user-who-send-invitation-email.
|
||||
3. Fill missing top-level :profile-email / :profile-name from the
|
||||
event actor (:profile-id), including soft-deleted profiles; omit
|
||||
when there is no usable actor id or the value is empty."
|
||||
[{:keys [::db/conn]} events]
|
||||
(let [team-ids (collect-team-ids-needing-name events)
|
||||
profile-ids (into (collect-profile-ids-needing-email events)
|
||||
(collect-actor-ids-needing-profile events))
|
||||
team-names (load-team-names conn team-ids)
|
||||
profiles (load-profiles conn profile-ids)]
|
||||
(mapv (fn [event]
|
||||
(-> event
|
||||
(maybe-assoc-team-name team-names)
|
||||
(maybe-assoc-emails profiles)
|
||||
(maybe-assoc-actor-profile profiles)))
|
||||
events)))
|
||||
|
||||
(defn- send-events
|
||||
"Ingest rows into the admin console. Returns {:processed :skipped}
|
||||
(filtered-out rows count as skipped); throws carrying the attempted
|
||||
count as :events. Enrich failures propagate untouched: a local
|
||||
DB/decode failure is ours, not the destination's, and must not count as
|
||||
a send failure."
|
||||
[cfg rows]
|
||||
(let [events (filterv #(and (contains? event-names (:name %))
|
||||
(not (audit/telemetry-shadow-source? (:source %))))
|
||||
rows)
|
||||
total (count rows)
|
||||
skipped (- total (count events))
|
||||
host (::host cfg)]
|
||||
(if (empty? events)
|
||||
{:processed 0 :skipped skipped}
|
||||
;; Enrich first, outside the try: a local DB/decode failure is
|
||||
;; ours, not the destination's, and must not count as
|
||||
;; admin-console/failed.
|
||||
(let [enriched (enrich-events cfg events)
|
||||
n (count events)]
|
||||
(try
|
||||
(nitrate/call cfg :ingest-audit-log {:events enriched})
|
||||
(l/dbg :hint "consumer events chunk ingested"
|
||||
:consumer "admin-console"
|
||||
:host host
|
||||
:processed n
|
||||
:skipped skipped)
|
||||
{:processed n :skipped skipped}
|
||||
(catch Throwable cause
|
||||
(l/dbg :hint "consumer events chunk failed"
|
||||
:consumer "admin-console"
|
||||
:host host
|
||||
:events n
|
||||
:code :admin-console-send-failed)
|
||||
(throw (ex/raise :type :internal
|
||||
:code :admin-console-send-failed
|
||||
:hint "unable to archive events to the admin console"
|
||||
:cause cause
|
||||
:events n))))))))
|
||||
|
||||
(defn- process-chunk
|
||||
"Send one chunk to the admin console. Returns {:processed :skipped}:
|
||||
the row outside the allowlist (and the telemetry shadows) count as
|
||||
skipped. Throws on failure, same contract as the nexus one."
|
||||
[cfg rows]
|
||||
(let [decoded (mapv consumers/decode-row rows)
|
||||
_ (l/trc :hint "consumer events chunk"
|
||||
:consumer "admin-console"
|
||||
:events (count decoded))]
|
||||
(send-events cfg decoded)))
|
||||
|
||||
(defmethod ig/assert-key ::consumer
|
||||
[_ cfg]
|
||||
;; No assert on the admin-console client: it inits to nil without the
|
||||
;; :admin-console flag, and the component skips on its flag anyway.
|
||||
(assert (db/pool? (::db/pool cfg)) "expected a valid database pool")
|
||||
(assert (mtx/metrics? (::mtx/metrics cfg)) "expected a valid metrics instance"))
|
||||
|
||||
(defn- prepare
|
||||
"Resolve the destination once at boot. An incomplete configuration
|
||||
raises here and the worker does not start: a consumer whose ingest can
|
||||
never reach its destination would fail every pass without advancing
|
||||
its cursor, and while enabled it also holds the audit-log GC. The host
|
||||
travels in the cfg so the per-chunk logs do not resolve it again."
|
||||
[cfg]
|
||||
(let [uri (some-> (cf/get :admin-console-uri) u/uri)]
|
||||
(when (nil? uri)
|
||||
(ex/raise :type :internal
|
||||
:code :task-not-configured
|
||||
:hint "admin-console consumer not configured, missing :admin-console-uri"))
|
||||
(let [host (u/get-domain uri)]
|
||||
(l/dbg :hint "audit log consumer initialized"
|
||||
:consumer "admin-console"
|
||||
:host host)
|
||||
(assoc cfg ::host host))))
|
||||
|
||||
(def consumer
|
||||
"The allowlisted full rows travel to the admin console."
|
||||
{:id consumers/admin-console
|
||||
:flag :admin-console
|
||||
:label "admin-console"
|
||||
:prepare prepare
|
||||
:process process-chunk})
|
||||
|
||||
(defmethod ig/init-key ::consumer
|
||||
[_ cfg]
|
||||
(consumers/init cfg consumer))
|
||||
|
||||
(defmethod ig/halt-key! ::consumer
|
||||
[_ thread]
|
||||
(consumers/halt thread))
|
||||
154
backend/src/app/loggers/audit/consumers/nexus.clj
Normal file
154
backend/src/app/loggers/audit/consumers/nexus.clj
Normal file
@ -0,0 +1,154 @@
|
||||
;; This Source Code Form is subject to the terms of the Mozilla Public
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.loggers.audit.consumers.nexus
|
||||
"The Nexus transport of the audit log consumer.
|
||||
|
||||
It reads the full rows of the stream from the Nexus cursor (the oldest
|
||||
events first) and POSTs them to Nexus in batches. The telemetry shadows
|
||||
are not its business: they count as skipped and the cursor passes over
|
||||
them, because the anonymized copy belongs to the telemetry consumer. A
|
||||
batch Nexus refuses rolls back and stays for the next pass: the cursor
|
||||
does not advance, the pass aborts and the loop retries after its pause.
|
||||
|
||||
The component starts the consumer loop at boot when its flag is on. The
|
||||
mechanics (the tick mutex, the cursor, the frontier of the pass, the GC)
|
||||
live in `app.loggers.audit.consumers`."
|
||||
(:require
|
||||
[app.common.exceptions :as ex]
|
||||
[app.common.logging :as l]
|
||||
[app.common.transit :as t]
|
||||
[app.common.uri :as u]
|
||||
[app.config :as cf]
|
||||
[app.db :as db]
|
||||
[app.http.client :as http]
|
||||
[app.loggers.audit :as audit]
|
||||
[app.loggers.audit.consumers :as consumers]
|
||||
[app.metrics :as mtx]
|
||||
[app.setup :as-alias setup]
|
||||
[integrant.core :as ig]))
|
||||
|
||||
(defn- generate-uri
|
||||
"Joins relative path segments to the Nexus backend URI, same shape
|
||||
as the admin-console helper in `app.nitrate` (there the ingest path
|
||||
is `api/audit-log`; here it is `api/audit`).
|
||||
Segments must not start with `/`. Returns nil when no base URI is
|
||||
configured, so the caller keeps the `:task-not-configured` guard."
|
||||
[& segments]
|
||||
(when-let [base (cf/get :nexus-uri)]
|
||||
(apply cf/join-uri base segments)))
|
||||
|
||||
(defn- resolve-uri
|
||||
"The Nexus endpoint of this consumer, built from `:nexus-uri`. Raises at
|
||||
boot when it is not available, so a consumer without a destination fails
|
||||
loudly instead of skipping silently.
|
||||
Failing the boot is deliberate: a worker that starts with its archive
|
||||
destination silently disabled is worse than one that refuses to start."
|
||||
[]
|
||||
(or (some-> (generate-uri "api/audit") u/uri)
|
||||
(ex/raise :type :internal
|
||||
:code :task-not-configured
|
||||
:hint "consumer task not configured, missing uri")))
|
||||
|
||||
(defn- truncate-body
|
||||
[body limit]
|
||||
(when (some? body)
|
||||
(let [s (str body)]
|
||||
(if (> (count s) limit)
|
||||
(subs s 0 limit)
|
||||
s))))
|
||||
|
||||
(defn- send-events
|
||||
"POST events to Nexus. Returns the number of events sent; throws on
|
||||
transport failure or non-204 refusal, carrying the attempted count
|
||||
as :events in the error data so the caller records the failure.
|
||||
No metrics or logs here: the loop reports the aborted pass."
|
||||
[cfg events]
|
||||
(let [uri (::uri cfg)
|
||||
n (count events)
|
||||
skey (-> cfg ::setup/shared-keys :nexus)
|
||||
body (t/encode {:events events})
|
||||
params {:uri uri
|
||||
:timeout 12000
|
||||
:method :post
|
||||
:headers {"content-type" "application/transit+json"
|
||||
"origin" (str (cf/get :public-uri))
|
||||
"x-shared-key" (str "nexus " skey)}
|
||||
:body body}
|
||||
resp (try
|
||||
(http/req cfg params {:skip-ssrf-check? true})
|
||||
(catch Throwable cause
|
||||
(throw (ex/raise :type :internal
|
||||
:code :nexus-send-failed
|
||||
:hint "unable to send events to nexus"
|
||||
:cause cause
|
||||
:events n))))]
|
||||
(if (= (:status resp) 204)
|
||||
n
|
||||
(throw (ex/raise :type :internal
|
||||
:code :consumer-chunk-refused
|
||||
:hint "nexus refused archive chunk"
|
||||
:resp-status (:status resp)
|
||||
:resp-body (truncate-body (:body resp) 500)
|
||||
:events n)))))
|
||||
|
||||
(defn- process-chunk
|
||||
"Send one chunk of the stream to Nexus. The telemetry shadows count as
|
||||
skipped: the full row already carries the event, and the anonymized copy
|
||||
belongs to the telemetry consumer (sending it here would archive the same
|
||||
event twice). Returns {:processed n :skipped n}; throws on refusal, so the
|
||||
chunk rolls back and the cursor stays where it was. No metrics here: a
|
||||
normal return means the transaction committed, so the caller records the
|
||||
success."
|
||||
[cfg rows]
|
||||
(let [decoded (mapv consumers/decode-row rows)
|
||||
events (into []
|
||||
(comp (remove (comp audit/telemetry-shadow-source? :source))
|
||||
(map consumers/row->event))
|
||||
decoded)
|
||||
skipped (- (count decoded) (count events))
|
||||
n (if (seq events)
|
||||
(send-events cfg events)
|
||||
0)]
|
||||
|
||||
(when (pos? n)
|
||||
(l/trc :hint "events chunk sent"
|
||||
:events n))
|
||||
|
||||
{:processed n :skipped skipped}))
|
||||
|
||||
(defmethod ig/assert-key ::consumer
|
||||
[_ cfg]
|
||||
(assert (db/pool? (::db/pool cfg)) "expected a valid database pool")
|
||||
(assert (http/client? (::http/client cfg)) "expected a valid http client")
|
||||
(assert (mtx/metrics? (::mtx/metrics cfg)) "expected a valid metrics instance"))
|
||||
|
||||
(defn- prepare
|
||||
"Resolve the destination once at boot: per-pass logs carry counts, this
|
||||
says where the consumer sends. An incomplete configuration raises here
|
||||
(see `resolve-uri`) and the worker does not start: intended."
|
||||
[cfg]
|
||||
(let [uri (resolve-uri)]
|
||||
(l/dbg :hint "audit log consumer initialized"
|
||||
:consumer "nexus"
|
||||
:host (u/get-domain uri))
|
||||
(assoc cfg ::uri uri)))
|
||||
|
||||
(def consumer
|
||||
"The full rows travel to Nexus."
|
||||
{:id consumers/nexus
|
||||
:flag :nexus
|
||||
:label "nexus"
|
||||
:prepare prepare
|
||||
:process process-chunk})
|
||||
|
||||
(defmethod ig/init-key ::consumer
|
||||
[_ cfg]
|
||||
(consumers/init cfg consumer))
|
||||
|
||||
(defmethod ig/halt-key! ::consumer
|
||||
[_ thread]
|
||||
(consumers/halt thread))
|
||||
154
backend/src/app/loggers/audit/consumers/telemetry.clj
Normal file
154
backend/src/app/loggers/audit/consumers/telemetry.clj
Normal file
@ -0,0 +1,154 @@
|
||||
;; This Source Code Form is subject to the terms of the Mozilla Public
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.loggers.audit.consumers.telemetry
|
||||
"The telemetry transport of the audit log consumer.
|
||||
|
||||
It reads the stream from its own cursor (id 0) and sends the anonymized
|
||||
copies (the shadow rows, `source LIKE 'telemetry:%'`) to the telemetry
|
||||
endpoint. The full rows are not its business: they count as skipped and
|
||||
the cursor passes over them, so a deployment with any full-row consumer
|
||||
still ships its telemetry.
|
||||
|
||||
The payload is the one the legacy cron sent: the events alone, without
|
||||
the row id or the cursor columns. The endpoint always answers 200 and
|
||||
counts its own errors; a refusal rolls the chunk back, so the rows stay
|
||||
for the next pass and the cursor does not advance.
|
||||
|
||||
The component starts the consumer loop at boot when its flag is on. The
|
||||
mechanics (the tick mutex, the cursor, the frontier of the pass, the GC)
|
||||
live in `app.loggers.audit.consumers`."
|
||||
(:require
|
||||
[app.common.data :as d]
|
||||
[app.common.exceptions :as ex]
|
||||
[app.common.logging :as l]
|
||||
[app.common.uri :as u]
|
||||
[app.config :as cf]
|
||||
[app.db :as db]
|
||||
[app.http.client :as http]
|
||||
[app.loggers.audit :as audit]
|
||||
[app.loggers.audit.consumers :as consumers]
|
||||
[app.metrics :as mtx]
|
||||
[app.setup :as-alias setup]
|
||||
[app.util.json :as json]
|
||||
[integrant.core :as ig]))
|
||||
|
||||
(defn- resolve-uri
|
||||
"The telemetry endpoint of this consumer, configured as `:telemetry-uri`
|
||||
(it has a default). Raises at boot when it is not available, so a consumer
|
||||
without a destination fails loudly instead of skipping silently."
|
||||
[]
|
||||
(or (some-> (cf/get :telemetry-uri) u/uri)
|
||||
(ex/raise :type :internal
|
||||
:code :task-not-configured
|
||||
:hint "telemetry consumer not configured, missing uri")))
|
||||
|
||||
(defn- row->event
|
||||
"The view of a shadow row that travels to the endpoint, the same one the
|
||||
legacy cron built: no row id, no `created-at` and no `ip-addr` (the copy
|
||||
already carries `0.0.0.0`)."
|
||||
[{:keys [name type source tracked-at profile-id props context]}]
|
||||
(d/without-nils
|
||||
{:name name
|
||||
:type type
|
||||
:source source
|
||||
:tracked-at tracked-at
|
||||
:profile-id profile-id
|
||||
:props (or props {})
|
||||
:context (or context {})}))
|
||||
|
||||
(defn- send-events
|
||||
"POST a batch of events to the telemetry endpoint. Answers the number of
|
||||
events sent; throws on transport failure or a status above 206, carrying
|
||||
the attempted count as :events in the error data so the caller records the
|
||||
failure. No metrics or logs here: the pass reports the aborted batch."
|
||||
[cfg events]
|
||||
(let [n (count events)
|
||||
uri (::uri cfg)
|
||||
body (json/encode-str {:type :telemetry-events
|
||||
:version (:full cf/version)
|
||||
:instance-id (:instance-id (::setup/props cfg))
|
||||
:events events})
|
||||
params {:uri uri
|
||||
:timeout 12000
|
||||
:method :post
|
||||
:headers {"content-type" "application/json"}
|
||||
:body body}
|
||||
resp (try
|
||||
(http/req cfg params {:skip-ssrf-check? true})
|
||||
(catch Throwable cause
|
||||
(throw (ex/raise :type :internal
|
||||
:code :telemetry-send-failed
|
||||
:hint "unable to send events to the telemetry endpoint"
|
||||
:cause cause
|
||||
:events n))))]
|
||||
(if (<= (:status resp) 206)
|
||||
n
|
||||
(throw (ex/raise :type :internal
|
||||
:code :consumer-chunk-refused
|
||||
:hint "telemetry endpoint refused the batch"
|
||||
:resp-status (:status resp)
|
||||
:events n)))))
|
||||
|
||||
(defn- process-chunk
|
||||
"Send the shadow rows of one chunk to the endpoint. Returns
|
||||
{:processed n :skipped n}: the full rows count as skipped and the cursor
|
||||
advances over them, they are not its business. Throws on refusal, so the
|
||||
chunk rolls back and the cursor stays where it was."
|
||||
[cfg rows]
|
||||
(let [events (into []
|
||||
(comp (filter (comp audit/telemetry-shadow-source? :source))
|
||||
(map (comp row->event consumers/decode-row)))
|
||||
rows)
|
||||
total (count rows)]
|
||||
(when (seq events)
|
||||
(send-events cfg events)
|
||||
(l/trc :hint "events chunk sent"
|
||||
:events (count events)))
|
||||
{:processed (count events)
|
||||
:skipped (- total (count events))}))
|
||||
|
||||
(defn- prepare
|
||||
"Resolve the destination once at boot: per-pass logs carry counts, this
|
||||
says where the consumer sends. An incomplete configuration raises here and
|
||||
the worker does not start: intended."
|
||||
[cfg]
|
||||
(let [uri (resolve-uri)]
|
||||
(l/dbg :hint "audit log consumer initialized"
|
||||
:consumer "telemetry"
|
||||
:host (u/get-domain uri))
|
||||
(assoc cfg ::uri uri)))
|
||||
|
||||
(def options
|
||||
"The copies keep the pace of the legacy cron: one pass every three hours
|
||||
with up to 10 000 events per request. A pass every five seconds would
|
||||
multiply the requests to the endpoint by thousands."
|
||||
{:chunk-size 10000
|
||||
:pause-ms 10800000})
|
||||
|
||||
(def consumer
|
||||
"The anonymized copies travel to the telemetry endpoint."
|
||||
{:id consumers/telemetry
|
||||
:flag :telemetry
|
||||
:label "telemetry"
|
||||
:options options
|
||||
:prepare prepare
|
||||
:process process-chunk})
|
||||
|
||||
(defmethod ig/init-key ::consumer
|
||||
[_ cfg]
|
||||
(consumers/init cfg consumer))
|
||||
|
||||
(defmethod ig/halt-key! ::consumer
|
||||
[_ thread]
|
||||
(consumers/halt thread))
|
||||
|
||||
(defmethod ig/assert-key ::consumer
|
||||
[_ cfg]
|
||||
(assert (db/pool? (::db/pool cfg)) "expected a valid database pool")
|
||||
(assert (http/client? (::http/client cfg)) "expected a valid http client")
|
||||
(assert (mtx/metrics? (::mtx/metrics cfg)) "expected a valid metrics instance")
|
||||
(assert (some? (::setup/props cfg)) "expected setup props to be available"))
|
||||
298
backend/src/app/loggers/audit/gc.clj
Normal file
298
backend/src/app/loggers/audit/gc.clj
Normal file
@ -0,0 +1,298 @@
|
||||
;; This Source Code Form is subject to the terms of the Mozilla Public
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.loggers.audit.gc
|
||||
"The garbage collection of the audit log buffer.
|
||||
|
||||
The producer only inserts rows and each consumer keeps its position, so
|
||||
somebody has to delete the rows every registered consumer has already
|
||||
passed. This job has two paths:
|
||||
|
||||
- the full rows, the normal behavior: up to the oldest cursor of the
|
||||
consumers that read them (every id except the telemetry consumer);
|
||||
- the telemetry shadows, up to the cursor of the telemetry consumer plus
|
||||
a retention window: copies older than seven days are deleted whatever
|
||||
the cursors say, so the buffer cannot keep them forever when the
|
||||
endpoint is down or the flag is off.
|
||||
|
||||
Each path has its own cursor and its own loop, so a slow telemetry pass
|
||||
every three hours does not retain the events Nexus already archived, and
|
||||
turning telemetry off does not stop the deletion of the full rows. The
|
||||
retention is the only deletion that does not wait for a consumer, and it
|
||||
can remove copies that were never sent.
|
||||
|
||||
Each loop stops starting batches when its budget expires. A batch
|
||||
already in progress can still commit; remaining rows wait for the next
|
||||
cron run.
|
||||
|
||||
The consumer framework (ids, flags and cursor helpers) lives in
|
||||
`app.loggers.audit.consumers`."
|
||||
(:require
|
||||
[app.common.logging :as l]
|
||||
[app.common.schema :as sm]
|
||||
[app.common.time :as ct]
|
||||
[app.db :as db]
|
||||
[app.jobs :as jobs]
|
||||
[app.loggers.audit.consumers :as consumers]
|
||||
[app.metrics :as mtx]
|
||||
[integrant.core :as ig]))
|
||||
|
||||
(def ^:private ^:dynamic batch-size
|
||||
"Rows per delete: a big buffer is cleaned in bounded statements, each
|
||||
one its own snapshot, so the job heartbeats between batches. Dynamic so a
|
||||
test can bind a smaller one."
|
||||
5000)
|
||||
|
||||
(def ^:private ^:dynamic run-budget
|
||||
"Wall-clock budget for starting batches in one path. The paths run in
|
||||
series, each with its own clock. Dynamic so a test can bind a different one."
|
||||
(ct/duration {:seconds 120}))
|
||||
|
||||
(defn- budget-spent?
|
||||
"True when the time measured since `tpoint` (a `ct/tpoint` created at the
|
||||
start of the run) has reached `run-budget`."
|
||||
[tpoint]
|
||||
(not (pos? (compare run-budget (tpoint)))))
|
||||
|
||||
(def ^:private retention
|
||||
"How long an unsent telemetry shadow stays in the buffer. Copies older
|
||||
than this are deleted even when no consumer has passed them."
|
||||
"7 days")
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; THE TWO PATHS: THEIR CURSORS
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(def ^:private sql:full-cursor
|
||||
"The oldest position among the cursors of the full-row consumers. The
|
||||
telemetry consumer is excluded by its id: its cursor only holds the
|
||||
anonymized copies, which have their own path."
|
||||
"SELECT xid::text AS xid_text, seq
|
||||
FROM audit_log_consumer_cursor
|
||||
WHERE consumer <> ?::smallint
|
||||
ORDER BY xid ASC, seq ASC
|
||||
LIMIT 1")
|
||||
|
||||
(def ^:private sql:telemetry-cursor
|
||||
"SELECT xid::text AS xid_text, seq
|
||||
FROM audit_log_consumer_cursor
|
||||
WHERE consumer = ?::smallint")
|
||||
|
||||
(defn- get-oldest-full-cursor
|
||||
"The position of the full-row path, or nil when no consumer of that path
|
||||
has registered a cursor yet.
|
||||
|
||||
The set is not derived from the flags of the process that runs the job:
|
||||
the cursor table is shared by every instance, so a consumer disabled on
|
||||
this worker still holds the position it reached and keeps its unread rows
|
||||
alive."
|
||||
[cfg]
|
||||
(when-let [row (first (db/exec! cfg [sql:full-cursor consumers/telemetry]))]
|
||||
(select-keys row [:xid-text :seq])))
|
||||
|
||||
(defn- get-telemetry-cursor
|
||||
"The position of the telemetry path, or nil when its consumer has not
|
||||
registered yet. A missing cursor means its first pass has not run, so the
|
||||
cursor-based deletion of this path is idle (the retention still applies)."
|
||||
[cfg]
|
||||
(when-let [row (db/exec-one! cfg [sql:telemetry-cursor consumers/telemetry])]
|
||||
(select-keys row [:xid-text :seq])))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; THE TWO PATHS: THEIR DELETES
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(def ^:private sql:delete-full-batch
|
||||
"Delete one batch of full rows up to and including a position. The cursor
|
||||
points at the last row a consumer processed, so that row is done too.
|
||||
|
||||
Do not replace the materialized CTE with `id IN (SELECT ... LIMIT ?
|
||||
FOR UPDATE SKIP LOCKED)`: PostgreSQL can plan that form as a nested-loop
|
||||
semi join and rescan its locking subquery for each target row. As each
|
||||
rescan observes the rows locked or deleted by the preceding one, `LIMIT`
|
||||
becomes a per-rescan bound and one statement can delete more rows than its
|
||||
batch size.
|
||||
|
||||
The materialized CTE chooses the bounded set before the delete joins it,
|
||||
so the statement cannot delete more rows than its limit. Its query walks
|
||||
the `(xid, seq)` index and skips locked rows: a producer never holds an old
|
||||
row, and two GC runs must not wait for each other."
|
||||
"WITH doomed AS MATERIALIZED
|
||||
(SELECT a2.id
|
||||
FROM audit_log a2
|
||||
WHERE (a2.xid, a2.seq) <= (?::xid8, ?::bigint)
|
||||
AND a2.source NOT LIKE 'telemetry:%'
|
||||
ORDER BY a2.xid ASC, a2.seq ASC
|
||||
LIMIT ?
|
||||
FOR UPDATE SKIP LOCKED)
|
||||
DELETE FROM audit_log a
|
||||
USING doomed
|
||||
WHERE a.id = doomed.id")
|
||||
|
||||
(def ^:private sql:delete-shadow-batch
|
||||
"Delete one batch of telemetry shadows up to and including the position of
|
||||
the telemetry consumer. Same shape as the full-row statement, with the
|
||||
predicate reversed."
|
||||
"WITH doomed AS MATERIALIZED
|
||||
(SELECT a2.id
|
||||
FROM audit_log a2
|
||||
WHERE (a2.xid, a2.seq) <= (?::xid8, ?::bigint)
|
||||
AND a2.source LIKE 'telemetry:%'
|
||||
ORDER BY a2.xid ASC, a2.seq ASC
|
||||
LIMIT ?
|
||||
FOR UPDATE SKIP LOCKED)
|
||||
DELETE FROM audit_log a
|
||||
USING doomed
|
||||
WHERE a.id = doomed.id")
|
||||
|
||||
(def ^:private sql:delete-expired-shadows
|
||||
"Delete one batch of telemetry shadows past the retention window. It does
|
||||
not read a cursor: the window is the guarantee, whatever the consumers have
|
||||
passed, so the buffer stays bounded with the endpoint down or the flag off.
|
||||
Oldest first, so the batch always makes progress."
|
||||
(str "WITH doomed AS MATERIALIZED
|
||||
(SELECT a2.id
|
||||
FROM audit_log a2
|
||||
WHERE a2.source LIKE 'telemetry:%'
|
||||
AND a2.created_at < now() - interval '" retention "'
|
||||
ORDER BY a2.created_at ASC
|
||||
LIMIT ?
|
||||
FOR UPDATE SKIP LOCKED)
|
||||
DELETE FROM audit_log a
|
||||
USING doomed
|
||||
WHERE a.id = doomed.id"))
|
||||
|
||||
(defn- delete-batch
|
||||
"One batch of one path, in one transaction: read the position again (a
|
||||
cursor registered between two batches must be seen before the next delete)
|
||||
and delete up to it. Answers the number of deleted rows, 0 when the path has
|
||||
no cursor."
|
||||
[cfg get-cursor sql]
|
||||
(db/tx-run! cfg (fn [cfg]
|
||||
(if-let [cursor (get-cursor cfg)]
|
||||
(or (db/get-update-count
|
||||
(db/exec-one! cfg [sql (:xid-text cursor) (:seq cursor) batch-size]))
|
||||
0)
|
||||
0))))
|
||||
|
||||
(defn- check-interrupted
|
||||
[]
|
||||
(when (Thread/interrupted)
|
||||
(throw (InterruptedException.))))
|
||||
|
||||
(defn- run-path
|
||||
"Drain one path on the runner thread. Return the total committed before
|
||||
the path ran out of work or time, or met an ordinary failure. Propagate
|
||||
interruptions so the runner does not start another path."
|
||||
[cfg path batch]
|
||||
(let [tpoint (ct/tpoint)
|
||||
total (volatile! 0)]
|
||||
(try
|
||||
(loop []
|
||||
(check-interrupted)
|
||||
(when-not (budget-spent? tpoint)
|
||||
(let [deleted (long (batch))]
|
||||
(when (pos? deleted)
|
||||
(vswap! total + deleted)
|
||||
(consumers/record-count (mtx/instance cfg) :audit-consumer-discarded ["gc"] deleted))
|
||||
(check-interrupted)
|
||||
(when (pos? deleted)
|
||||
(jobs/heartbeat cfg)
|
||||
(Thread/sleep 100)
|
||||
(recur)))))
|
||||
(catch InterruptedException cause
|
||||
(throw cause))
|
||||
(catch Exception cause
|
||||
(when (= :interrupt (:type (ex-data cause)))
|
||||
(throw cause))
|
||||
(check-interrupted)
|
||||
(l/err :hint "audit log GC path failed" :path path :cause cause)))
|
||||
@total))
|
||||
|
||||
(defn- delete-full-rows
|
||||
"The normal path: the full rows every one of their consumers has passed."
|
||||
[cfg]
|
||||
(run-path cfg "full" #(delete-batch cfg get-oldest-full-cursor sql:delete-full-batch)))
|
||||
|
||||
(defn- delete-shadow-rows
|
||||
"The telemetry path: the shadows the telemetry consumer has passed."
|
||||
[cfg]
|
||||
(run-path cfg "shadow" #(delete-batch cfg get-telemetry-cursor sql:delete-shadow-batch)))
|
||||
|
||||
(defn- delete-expired-shadows
|
||||
"The telemetry retention window: shadows older than seven days, in
|
||||
batches, whatever the cursors say. Runs after the cursor paths with its
|
||||
own budget and answers the number of deleted rows."
|
||||
[cfg]
|
||||
(let [total (run-path cfg "shadow-expired"
|
||||
#(or (db/get-update-count
|
||||
(db/exec-one! cfg [sql:delete-expired-shadows batch-size]))
|
||||
0))]
|
||||
(when (pos? total)
|
||||
(l/warn :hint "purged stale telemetry events" :count total))
|
||||
total))
|
||||
|
||||
(defn delete-consumed
|
||||
"Delete the rows of the two paths, each one independently:
|
||||
|
||||
- the full rows, up to the oldest cursor of the consumers that read them;
|
||||
- the telemetry shadows, up to the cursor of consumer 0.
|
||||
|
||||
Then apply the retention window of the shadows.
|
||||
|
||||
The set is the cursor table, not the flags of this worker: a consumer that
|
||||
is off still holds its path (its cursor freezes and rejoins from wherever
|
||||
it is when it is enabled again), so a temporary disable never deletes
|
||||
events it has not read. A consumer registers its (0, 0) cursor when its
|
||||
component starts, so a consumer that is enabled but has not drained yet
|
||||
still holds its path at position 0.
|
||||
|
||||
The three paths run in order on the runner thread. Each one owns its
|
||||
wall-clock budget for starting batches, not for finishing them. An ordinary
|
||||
failure is logged and ends only that path, preserving its committed count.
|
||||
An interruption stops the whole pass.
|
||||
|
||||
Retention is the explicit exception: copies older than seven days are
|
||||
deleted even if no cursor passed them. They still count as removed by the
|
||||
GC. The warning counts rows deleted by age, whether sent or not.
|
||||
|
||||
Returns the total number of rows deleted. Each committed batch contributes
|
||||
to the `discarded` metric with reason `gc` before its heartbeat."
|
||||
[cfg]
|
||||
(let [cfg (db/own-transaction cfg)]
|
||||
(+ (delete-full-rows cfg)
|
||||
(delete-shadow-rows cfg)
|
||||
(delete-expired-shadows cfg))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; JOB
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(def schema:params
|
||||
"Params of the GC job: none. Cron submits {}; system deps (pool,
|
||||
metrics) live on the job-def cfg closed over by the handler."
|
||||
[:map {:closed true}])
|
||||
|
||||
(defmethod ig/assert-key ::job-def
|
||||
[_ cfg]
|
||||
(assert (db/pool? (::db/pool cfg)) "expected a valid database pool")
|
||||
(assert (mtx/metrics? (::mtx/metrics cfg)) "expected a valid metrics instance"))
|
||||
|
||||
(defmethod ig/init-key ::job-def
|
||||
[_ cfg]
|
||||
(l/dbg :hint "audit log GC job initialized")
|
||||
{::jobs/name :audit-log-gc
|
||||
::jobs/schema schema:params
|
||||
::jobs/handler
|
||||
(fn [_context _params]
|
||||
;; A path with no cursor deletes nothing on purpose: deleting would lose
|
||||
;; events for a consumer that may start later. Retention still runs,
|
||||
;; because it is bounded by age, not by a position.
|
||||
(let [total (delete-consumed cfg)]
|
||||
(l/dbg :hint "audit log buffer cleaned" :discarded total)
|
||||
total))
|
||||
::jobs/decoder (sm/decoder schema:params sm/json-transformer)
|
||||
::jobs/validator (sm/validator schema:params)})
|
||||
@ -27,7 +27,14 @@
|
||||
[app.http.websocket :as http.ws]
|
||||
[app.jobs :as-alias jobs]
|
||||
[app.jobs.gc :as-alias jobs.gc]
|
||||
[app.jobs.metrics :as jobs-metrics]
|
||||
;; The consumer transports live in their own namespaces; requiring them
|
||||
;; registers their `ig/init-key`/`ig/halt-key!` methods before the
|
||||
;; worker components start (`ig/load-namespaces` also loads them from
|
||||
;; the config keys).
|
||||
[app.loggers.audit.consumers.admin-console]
|
||||
[app.loggers.audit.consumers.nexus]
|
||||
[app.loggers.audit.consumers.telemetry]
|
||||
[app.loggers.audit.gc]
|
||||
[app.loggers.webhooks :as-alias webhooks]
|
||||
[app.metrics :as mtx]
|
||||
[app.metrics.definition :as-alias mdef]
|
||||
@ -121,12 +128,11 @@
|
||||
|
||||
:tasks-timing
|
||||
{::mdef/name "penpot_tasks_timing"
|
||||
;; Deprecated: superseded by `penpot_jobs_execution_ms`, which adds a
|
||||
;; queue label. Kept only for existing dashboards and alerts. The help
|
||||
;; text is what someone reading /metrics sees, so it is where the
|
||||
;; redirect belongs: the name and the labels stay untouched, so no
|
||||
;; panel or alert breaks.
|
||||
::mdef/help "Deprecated: use penpot_jobs_execution_ms, which adds a queue label."
|
||||
;; Deprecated: superseded by `penpot_jobs_execution_ms`. Kept only for
|
||||
;; existing dashboards and alerts. The help text is what someone reading
|
||||
;; /metrics sees, so it is where the redirect belongs: the name and the
|
||||
;; labels stay untouched, so no panel or alert breaks.
|
||||
::mdef/help "Deprecated: use penpot_jobs_execution_ms. Kept for existing dashboards and alerts."
|
||||
::mdef/labels ["name"]
|
||||
::mdef/buckets mtx/default-tasks-histogram-buckets
|
||||
::mdef/type :histogram}
|
||||
@ -262,28 +268,22 @@
|
||||
::mdef/help "Total number of handler errors in the http listener."
|
||||
::mdef/type :counter}
|
||||
|
||||
:jobs-submitted
|
||||
{::mdef/name "penpot_jobs_submitted_total"
|
||||
::mdef/help "Total number of durable jobs submitted."
|
||||
::mdef/labels ["name" "queue"]
|
||||
::mdef/type :counter}
|
||||
|
||||
:jobs-skipped
|
||||
{::mdef/name "penpot_jobs_skipped_total"
|
||||
::mdef/help "Total number of durable job submissions skipped by skip-if-active."
|
||||
::mdef/labels ["name" "queue"]
|
||||
:jobs-submit
|
||||
{::mdef/name "penpot_jobs_submit_total"
|
||||
::mdef/help "Total number of durable job submissions by outcome: created, skipped or refreshed."
|
||||
::mdef/labels ["name" "outcome"]
|
||||
::mdef/type :counter}
|
||||
|
||||
:jobs-dispatched
|
||||
{::mdef/name "penpot_jobs_dispatched_total"
|
||||
::mdef/help "Total number of jobs pushed to a worker queue. Counts pushes, not claims: compare with the _count of penpot_jobs_queue_wait_ms."
|
||||
::mdef/labels ["name" "queue"]
|
||||
::mdef/help "Total number of jobs pushed to a worker queue. Counts pushes, not claims: compare with the _count of penpot_jobs_claim_wait_ms."
|
||||
::mdef/labels ["name"]
|
||||
::mdef/type :counter}
|
||||
|
||||
:jobs-retries
|
||||
{::mdef/name "penpot_jobs_retries_total"
|
||||
::mdef/help "Total number of durable jobs rescheduled for retry."
|
||||
::mdef/labels ["name" "queue" "reason"]
|
||||
::mdef/labels ["name" "reason"]
|
||||
::mdef/type :counter}
|
||||
|
||||
:jobs-events-total
|
||||
@ -291,41 +291,29 @@
|
||||
::mdef/help "Total number of job events stored."
|
||||
::mdef/type :counter}
|
||||
|
||||
:jobs-orphaned
|
||||
{::mdef/name "penpot_jobs_orphaned_total"
|
||||
::mdef/help "Total number of running jobs marked as orphaned."
|
||||
::mdef/labels ["queue"]
|
||||
::mdef/type :counter}
|
||||
|
||||
:jobs-rescheduled
|
||||
{::mdef/name "penpot_jobs_rescheduled_total"
|
||||
::mdef/help "Total number of scheduled jobs recovered after transit loss."
|
||||
::mdef/labels ["queue"]
|
||||
::mdef/type :counter}
|
||||
|
||||
:jobs-queue-wait-timing
|
||||
{::mdef/name "penpot_jobs_queue_wait_ms"
|
||||
:jobs-claim-wait-timing
|
||||
{::mdef/name "penpot_jobs_claim_wait_ms"
|
||||
::mdef/help "Time from job scheduling to worker claim (milliseconds). Its _count counts claims, not pushes: compare with penpot_jobs_dispatched_total."
|
||||
::mdef/labels ["name" "queue"]
|
||||
::mdef/labels ["name"]
|
||||
::mdef/type :histogram
|
||||
::mdef/buckets [1 10 50 100 500 1000 5000 10000 30000 60000 300000 600000]}
|
||||
|
||||
:jobs-execution-timing
|
||||
{::mdef/name "penpot_jobs_execution_ms"
|
||||
::mdef/help "Time spent executing a job handler (milliseconds)."
|
||||
::mdef/labels ["name" "queue"]
|
||||
::mdef/labels ["name"]
|
||||
::mdef/type :histogram
|
||||
::mdef/buckets [1 10 50 100 500 1000 5000 10000 30000 60000 300000 600000]}
|
||||
|
||||
:jobs-total-timing
|
||||
{::mdef/name "penpot_jobs_total_ms"
|
||||
::mdef/help "Time from job creation to a terminal state (milliseconds). Its _count is the terminal total per outcome."
|
||||
::mdef/labels ["name" "queue" "outcome"]
|
||||
::mdef/labels ["name" "outcome"]
|
||||
::mdef/type :histogram
|
||||
::mdef/buckets [10 50 100 500 1000 5000 10000 30000 60000 300000 600000 3600000]}
|
||||
|
||||
:jobs-dispatcher-timing
|
||||
{::mdef/name "penpot_jobs_dispatcher_ms"
|
||||
:jobs-dispatcher-batch-timing
|
||||
{::mdef/name "penpot_jobs_dispatcher_batch_ms"
|
||||
::mdef/help "Dispatcher batch processing time (milliseconds)."
|
||||
::mdef/labels ["stage" "outcome"]
|
||||
::mdef/type :histogram
|
||||
@ -338,25 +326,14 @@
|
||||
::mdef/type :histogram
|
||||
::mdef/buckets [1 5 10 25 50 100 250 500 1000]}
|
||||
|
||||
:jobs-backlog
|
||||
{::mdef/name "penpot_jobs_backlog"
|
||||
::mdef/help "Current number of jobs grouped by status."
|
||||
::mdef/labels ["status"]
|
||||
::mdef/type :gauge}
|
||||
|
||||
:jobs-oldest-pending-age
|
||||
{::mdef/name "penpot_jobs_oldest_pending_age_seconds"
|
||||
::mdef/help "Age in seconds of the oldest pending job."
|
||||
::mdef/type :gauge}
|
||||
|
||||
:jobs-gc-rows
|
||||
{::mdef/name "penpot_jobs_gc_rows_total"
|
||||
::mdef/help "Rows removed or resources touched by the jobs garbage collector."
|
||||
::mdef/labels ["kind" "action"]
|
||||
::mdef/type :counter}
|
||||
|
||||
:jobs-gc-timing
|
||||
{::mdef/name "penpot_jobs_gc_ms"
|
||||
:jobs-gc-pass-timing
|
||||
{::mdef/name "penpot_jobs_gc_pass_ms"
|
||||
::mdef/help "Time spent in each jobs garbage collection pass (milliseconds)."
|
||||
::mdef/labels ["kind"]
|
||||
::mdef/type :histogram
|
||||
@ -382,16 +359,28 @@
|
||||
|
||||
:audit-consumer-events
|
||||
{::mdef/name "penpot_audit_consumer_events_total"
|
||||
::mdef/help "Audit log events drained by the consumer task, by destination and outcome."
|
||||
::mdef/help "Audit log events drained by the audit-log consumers, by destination and outcome."
|
||||
::mdef/labels ["destination" "outcome"]
|
||||
::mdef/type :counter}
|
||||
|
||||
:audit-consumer-passes
|
||||
{::mdef/name "penpot_audit_consumer_passes_total"
|
||||
::mdef/help "Audit log consumer passes, by destination and outcome."
|
||||
::mdef/labels ["destination" "outcome"]
|
||||
::mdef/type :counter}
|
||||
|
||||
:audit-consumer-discarded
|
||||
{::mdef/name "penpot_audit_consumer_discarded_total"
|
||||
::mdef/help "Audit log entries discarded without being archived, by reason."
|
||||
::mdef/help "Audit log entries removed from the buffer by the GC, by reason."
|
||||
::mdef/labels ["reason"]
|
||||
::mdef/type :counter}
|
||||
|
||||
:audit-consumer-lag
|
||||
{::mdef/name "penpot_audit_consumer_lag_events"
|
||||
::mdef/help "Audit log events confirmed in the buffer that a consumer has not processed yet (capped at 10000)."
|
||||
::mdef/labels ["destination"]
|
||||
::mdef/type :gauge}
|
||||
|
||||
:audit-ingest-events
|
||||
{::mdef/name "penpot_audit_ingest_events_total"
|
||||
::mdef/help "Audit log frontend events ingested, by kind and outcome."
|
||||
@ -635,8 +624,7 @@
|
||||
|
||||
:app.nitrate/client
|
||||
{::http.client/client (ig/ref ::http.client/client)
|
||||
;; No ::jobs/defs: nitrate never submits jobs, and consumer-task's
|
||||
;; job-def needs this client — that would cycle through jobs/defs.
|
||||
;; No ::jobs/defs: nitrate never submits jobs.
|
||||
::setup/shared-keys (ig/ref ::setup/shared-keys)}
|
||||
|
||||
:app.rpc/management-methods
|
||||
@ -664,26 +652,25 @@
|
||||
::setup/shared-keys (ig/ref ::setup/shared-keys)}
|
||||
|
||||
::jobs/defs
|
||||
{:sendmail (ig/ref ::email/job-def)
|
||||
:send-templated-mail (ig/ref ::email/templated-job-def)
|
||||
:delete-object (ig/ref :app.tasks.delete-object/job-def)
|
||||
:demo-purge (ig/ref :app.tasks.demo-purge/job-def)
|
||||
:run-webhook (ig/ref ::webhooks/run-webhook-job-def)
|
||||
:process-webhook-event (ig/ref ::webhooks/process-webhook-event-job-def)
|
||||
:file-gc (ig/ref :app.tasks.file-gc/job-def)
|
||||
:offload-file-data (ig/ref :app.tasks.offload-file-data/job-def)
|
||||
:objects-gc (ig/ref :app.tasks.objects-gc/job-def)
|
||||
:export-binfile (ig/ref :app.tasks.export-binfile/job-def)
|
||||
:import-binfile (ig/ref :app.tasks.import-binfile/job-def)
|
||||
:storage-gc-deleted (ig/ref ::sto.gc-deleted/job-def)
|
||||
:storage-gc-touched (ig/ref ::sto.gc-touched/job-def)
|
||||
:storage-pending-gc (ig/ref ::sto.pending-gc/job-def)
|
||||
:jobs-gc (ig/ref :app.jobs.gc/job-def)
|
||||
:telemetry (ig/ref :app.tasks.telemetry/job-def)
|
||||
:session-gc (ig/ref ::session/job-def)
|
||||
:file-gc-scheduler (ig/ref :app.tasks.file-gc-scheduler/job-def)
|
||||
:audit-log-consumer-nexus (ig/ref :app.loggers.audit.consumer-task/nexus-job-def)
|
||||
:audit-log-consumer-admin-console (ig/ref :app.loggers.audit.consumer-task/admin-console-job-def)}
|
||||
{:sendmail (ig/ref ::email/job-def)
|
||||
:send-templated-mail (ig/ref ::email/templated-job-def)
|
||||
:delete-object (ig/ref :app.tasks.delete-object/job-def)
|
||||
:demo-purge (ig/ref :app.tasks.demo-purge/job-def)
|
||||
:run-webhook (ig/ref ::webhooks/run-webhook-job-def)
|
||||
:process-webhook-event (ig/ref ::webhooks/process-webhook-event-job-def)
|
||||
:file-gc (ig/ref :app.tasks.file-gc/job-def)
|
||||
:offload-file-data (ig/ref :app.tasks.offload-file-data/job-def)
|
||||
:objects-gc (ig/ref :app.tasks.objects-gc/job-def)
|
||||
:export-binfile (ig/ref :app.tasks.export-binfile/job-def)
|
||||
:import-binfile (ig/ref :app.tasks.import-binfile/job-def)
|
||||
:storage-gc-deleted (ig/ref ::sto.gc-deleted/job-def)
|
||||
:storage-gc-touched (ig/ref ::sto.gc-touched/job-def)
|
||||
:storage-pending-gc (ig/ref ::sto.pending-gc/job-def)
|
||||
:jobs-gc (ig/ref :app.jobs.gc/job-def)
|
||||
:telemetry (ig/ref :app.tasks.telemetry/job-def)
|
||||
:session-gc (ig/ref ::session/job-def)
|
||||
:file-gc-scheduler (ig/ref :app.tasks.file-gc-scheduler/job-def)
|
||||
:audit-log-gc (ig/ref :app.loggers.audit.gc/job-def)}
|
||||
|
||||
::jobs/request-pool
|
||||
{::rds/client (ig/ref ::rds/client)
|
||||
@ -776,16 +763,9 @@
|
||||
{::db/pool (ig/ref ::db/pool)
|
||||
::mtx/metrics (ig/ref ::mtx/metrics)}
|
||||
|
||||
:app.loggers.audit.consumer-task/nexus-job-def
|
||||
{::db/pool (ig/ref ::db/pool)
|
||||
::http.client/client (ig/ref ::http.client/client)
|
||||
::setup/shared-keys (ig/ref ::setup/shared-keys)
|
||||
::mtx/metrics (ig/ref ::mtx/metrics)}
|
||||
|
||||
:app.loggers.audit.consumer-task/admin-console-job-def
|
||||
{::db/pool (ig/ref ::db/pool)
|
||||
:app.nitrate/client (ig/ref :app.nitrate/client)
|
||||
::mtx/metrics (ig/ref ::mtx/metrics)}
|
||||
:app.loggers.audit.gc/job-def
|
||||
{::db/pool (ig/ref ::db/pool)
|
||||
::mtx/metrics (ig/ref ::mtx/metrics)}
|
||||
|
||||
::email/blacklist
|
||||
{}
|
||||
@ -876,43 +856,46 @@
|
||||
{::sto.fs/directory (or (cf/get :storage-assets-fs-directory)
|
||||
(cf/get :objects-storage-fs-directory))}})
|
||||
|
||||
(defn worker-cron-entries
|
||||
"Cron entries of the worker. The audit-log GC is a separate switch: the
|
||||
old and the new version share `:audit-log-gc`, so keeping it off during
|
||||
the rollout stops both until the consumers are the only readers."
|
||||
[]
|
||||
(into [] (remove nil?)
|
||||
[{:cron #penpot/cron "0 0 0 * * ?" ;; daily
|
||||
:job :session-gc}
|
||||
|
||||
{:cron #penpot/cron "0 0 0 * * ?" ;; daily
|
||||
:job :objects-gc}
|
||||
|
||||
{:cron #penpot/cron "0 0 0 * * ?" ;; daily
|
||||
:job :storage-gc-deleted}
|
||||
|
||||
{:cron #penpot/cron "0 0 0 * * ?" ;; daily
|
||||
:job :storage-gc-touched}
|
||||
|
||||
{:cron #penpot/cron "0 0 0 * * ?" ;; daily
|
||||
:job :storage-pending-gc}
|
||||
|
||||
{:cron #penpot/cron "0 0 0 * * ?" ;; daily
|
||||
:job :jobs-gc}
|
||||
|
||||
{:cron #penpot/cron "0 0 2 * * ?" ;; daily
|
||||
:job :file-gc-scheduler}
|
||||
|
||||
{:cron #penpot/cron "0 30 */3,23 * * ?"
|
||||
:job :telemetry}
|
||||
|
||||
(when (contains? cf/flags :audit-log-gc)
|
||||
{:cron #penpot/cron "45 */5 * * * ?" ;; every 5m
|
||||
:job :audit-log-gc})]))
|
||||
|
||||
(def worker-config
|
||||
{::wrk/cron
|
||||
{::jobs/defs (ig/ref ::jobs/defs)
|
||||
::db/pool (ig/ref ::db/pool)
|
||||
::mtx/metrics (ig/ref ::mtx/metrics)
|
||||
::wrk/entries
|
||||
[{:cron #penpot/cron "0 0 0 * * ?" ;; daily
|
||||
:job :session-gc}
|
||||
|
||||
{:cron #penpot/cron "0 0 0 * * ?" ;; daily
|
||||
:job :objects-gc}
|
||||
|
||||
{:cron #penpot/cron "0 0 0 * * ?" ;; daily
|
||||
:job :storage-gc-deleted}
|
||||
|
||||
{:cron #penpot/cron "0 0 0 * * ?" ;; daily
|
||||
:job :storage-gc-touched}
|
||||
|
||||
{:cron #penpot/cron "0 0 0 * * ?" ;; daily
|
||||
:job :storage-pending-gc}
|
||||
|
||||
{:cron #penpot/cron "0 0 0 * * ?" ;; daily
|
||||
:job :jobs-gc}
|
||||
|
||||
{:cron #penpot/cron "0 0 2 * * ?" ;; daily
|
||||
:job :file-gc-scheduler}
|
||||
|
||||
{:cron #penpot/cron "0 30 */3,23 * * ?"
|
||||
:job :telemetry}
|
||||
|
||||
(when (contains? cf/flags :nexus)
|
||||
{:cron #penpot/cron "0 */5 * * * ?" ;; every 5m
|
||||
:job :audit-log-consumer-nexus})
|
||||
|
||||
(when (contains? cf/flags :admin-console)
|
||||
{:cron #penpot/cron "30 */5 * * * ?" ;; every 5m, offset 30s from nexus
|
||||
:job :audit-log-consumer-admin-console})]}
|
||||
::wrk/entries (worker-cron-entries)}
|
||||
|
||||
::wrk/dispatcher
|
||||
{::rds/client (ig/ref ::rds/client)
|
||||
@ -920,10 +903,6 @@
|
||||
::db/pool (ig/ref ::db/pool)
|
||||
::wrk/tenant (cf/get :tenant)}
|
||||
|
||||
::jobs-metrics/sampler
|
||||
{::db/pool (ig/ref ::db/pool)
|
||||
::mtx/metrics (ig/ref ::mtx/metrics)}
|
||||
|
||||
[::default ::wrk/runner]
|
||||
{::wrk/parallelism (cf/get :worker-default-parallelism 1)
|
||||
::wrk/queue :default
|
||||
@ -963,7 +942,27 @@
|
||||
::jobs/defs (ig/ref ::jobs/defs)
|
||||
::mtx/metrics (ig/ref ::mtx/metrics)
|
||||
::db/pool (ig/ref ::db/pool)
|
||||
::mbus/msgbus (ig/ref ::mbus/msgbus)}})
|
||||
::mbus/msgbus (ig/ref ::mbus/msgbus)}
|
||||
|
||||
;; The audit-log consumers are loops, not jobs: each one starts a
|
||||
;; thread that takes its tick lock, drains up to the frontier of the
|
||||
;; pass and sleeps. Only worker instances run them.
|
||||
:app.loggers.audit.consumers.nexus/consumer
|
||||
{::db/pool (ig/ref ::db/pool)
|
||||
::http.client/client (ig/ref ::http.client/client)
|
||||
::setup/shared-keys (ig/ref ::setup/shared-keys)
|
||||
::mtx/metrics (ig/ref ::mtx/metrics)}
|
||||
|
||||
:app.loggers.audit.consumers.admin-console/consumer
|
||||
{::db/pool (ig/ref ::db/pool)
|
||||
:app.nitrate/client (ig/ref :app.nitrate/client)
|
||||
::mtx/metrics (ig/ref ::mtx/metrics)}
|
||||
|
||||
:app.loggers.audit.consumers.telemetry/consumer
|
||||
{::db/pool (ig/ref ::db/pool)
|
||||
::http.client/client (ig/ref ::http.client/client)
|
||||
::setup/props (ig/ref ::setup/props)
|
||||
::mtx/metrics (ig/ref ::mtx/metrics)}})
|
||||
|
||||
|
||||
(defn start
|
||||
|
||||
@ -531,9 +531,6 @@
|
||||
{:name "0156-reset-pg-storage-defaults"
|
||||
:fn (mg/resource "app/migrations/sql/0156-reset-pg-storage-defaults.sql")}
|
||||
|
||||
{:name "0157-add-audit-log-pending-table"
|
||||
:fn (mg/resource "app/migrations/sql/0157-add-audit-log-pending-table.sql")}
|
||||
|
||||
{:name "0157-migrate-legacy-file-data"
|
||||
:fn (mg/resource "app/migrations/sql/0157-migrate-legacy-file-data.sql")}
|
||||
|
||||
@ -544,7 +541,10 @@
|
||||
:fn (mg/resource "app/migrations/sql/0158-partition-file-change.sql")}
|
||||
|
||||
{:name "0159-add-file-data-storage-object-id"
|
||||
:fn (mg/resource "app/migrations/sql/0159-add-file-data-storage-object-id.sql")}])
|
||||
:fn (mg/resource "app/migrations/sql/0159-add-file-data-storage-object-id.sql")}
|
||||
|
||||
{:name "0159-consumer-generalization"
|
||||
:fn (mg/resource "app/migrations/sql/0159-consumer-generalization.sql")}])
|
||||
|
||||
(defn apply-migrations!
|
||||
[pool name migrations]
|
||||
|
||||
@ -19,8 +19,10 @@
|
||||
-- database and the tenant is what keeps them from claiming each other's work:
|
||||
-- as an equality the filter is something an index can serve, where a
|
||||
-- `<tenant>:<queue>` prefix could only be matched with a LIKE over text. Every
|
||||
-- query that acts on one instance's work (dispatch, orphan sweep, reschedule,
|
||||
-- cron no-overlap check, submit dedupe) filters on `tenant` for that reason.
|
||||
-- query that acts on one instance's work (dispatch, lost sweep, orphan sweep,
|
||||
-- cron no-overlap check) filters on `tenant` for that reason, and the tenant
|
||||
-- leads the keys of the submit dedupe and exclusivity indexes (see the
|
||||
-- consumer-generalization migration).
|
||||
-- The prefix still exists, but only in the Redis key the queue's payloads
|
||||
-- travel through, composed in one place by `app.worker/queue-key`.
|
||||
--
|
||||
@ -84,24 +86,16 @@ CREATE INDEX job__profile__idx
|
||||
ON job (profile_id, created_at DESC)
|
||||
WHERE profile_id IS NOT NULL;
|
||||
|
||||
-- Cron no-overlap check and submit dedupe: both look for jobs of one
|
||||
-- tenant with the same name and queue and label that have not started
|
||||
-- yet, so one index serves the two queries.
|
||||
CREATE INDEX job__name_label__idx
|
||||
ON job (tenant, name, queue, label)
|
||||
WHERE status IN ('new', 'scheduled', 'running', 'retry');
|
||||
|
||||
-- Storage GC reference check: it answers whether any job still points
|
||||
-- to a candidate storage object. Rows without a resource are excluded.
|
||||
CREATE INDEX job__resource__idx
|
||||
ON job (resource_id)
|
||||
WHERE resource_id IS NOT NULL;
|
||||
|
||||
-- Dispatcher recovery: it finds the `scheduled` jobs of its own tenant
|
||||
-- that were pushed to Redis but not claimed within the recovery window.
|
||||
CREATE INDEX job__scheduled__idx
|
||||
ON job (tenant, scheduled_at)
|
||||
WHERE status = 'scheduled';
|
||||
-- The lost-sweep index of `scheduled` jobs is created by
|
||||
-- `0159-consumer-generalization`, on `modified_at` (the hand-off time),
|
||||
-- which is the column the sweep filters by: building it here on
|
||||
-- `scheduled_at` would only throw it away in the same transaction.
|
||||
|
||||
-- Jobs GC expiration: it scans jobs past expires_at while excluding
|
||||
-- running and retry jobs from the expiration delete.
|
||||
@ -140,9 +134,9 @@ CREATE INDEX job_event__job_kind_created_idx
|
||||
-- jobs GC, which also marks the referenced storage objects as touched in
|
||||
-- the same transaction; a stray `DELETE FROM job` would drop the row
|
||||
-- (and cascade `job_event`) without touching the object, orphaning it
|
||||
-- forever. The GC and the submit dedupe disable the guard inside their
|
||||
-- transaction with `SET LOCAL rules.deletion_protection TO off`, the
|
||||
-- same escape hatch used by `app.tasks.objects-gc`.
|
||||
-- forever. The jobs GC disables the guard inside its transaction with
|
||||
-- `SET LOCAL rules.deletion_protection TO off`, the same escape hatch used
|
||||
-- by `app.tasks.objects-gc`; `app.jobs/submit` never deletes.
|
||||
CREATE OR REPLACE TRIGGER deletion_protection__tgr
|
||||
BEFORE DELETE ON job FOR EACH STATEMENT
|
||||
WHEN ((current_setting('rules.deletion_protection', true) IN ('on', '')) OR
|
||||
|
||||
@ -1,39 +0,0 @@
|
||||
-- Work queue for the audit log consumers: one row per (record,
|
||||
-- consumer) pair still pending. Inserting a record creates its
|
||||
-- pendings; each consumer deletes its own on completion, and whoever
|
||||
-- deletes the last pending also deletes the record. Consumer ids are
|
||||
-- fixed in code and never reused: 1 = nexus, 2 = admin console.
|
||||
|
||||
CREATE TABLE audit_log_pending (
|
||||
-- DEFERRABLE so the test-suite reset (which deletes table by table
|
||||
-- with SET CONSTRAINTS ALL DEFERRED) keeps working; the guard still
|
||||
-- fires when a record with pendings is deleted. Plain DEFERRABLE
|
||||
-- (initially immediate) is enough: every deleter removes pendings
|
||||
-- before rows, so the check passes at each statement.
|
||||
audit_log_id uuid NOT NULL REFERENCES audit_log(id) DEFERRABLE,
|
||||
consumer smallint NOT NULL CHECK (consumer BETWEEN 1 AND 20),
|
||||
PRIMARY KEY (consumer, audit_log_id));
|
||||
|
||||
-- Lets the final delete ask "is anything still pending for this
|
||||
-- record?" without scanning the whole queue.
|
||||
CREATE INDEX ON audit_log_pending (audit_log_id);
|
||||
|
||||
-- NOTE: no backfill on purpose. Rows predating this queue (sent or
|
||||
-- not) get no pendings, so the orphan sweep deletes them unsent.
|
||||
-- Losing the unarchived ones at upgrade time is accepted: this
|
||||
-- table is a minutes/hours buffer, not a store.
|
||||
|
||||
-- The pending queue replaces the archived_at mark: whoever deletes
|
||||
-- the last pending deletes the record, so the flag column and the
|
||||
-- indexes that served the old mark-and-sweep scans go away.
|
||||
DROP INDEX IF EXISTS audit_log__created_at__idx;
|
||||
DROP INDEX IF EXISTS audit_log__archived_at__idx;
|
||||
DROP INDEX IF EXISTS audit_log_id_archived_at_idx;
|
||||
ALTER TABLE audit_log DROP COLUMN IF EXISTS archived_at;
|
||||
|
||||
-- Serves the consumers' chronological scan (ORDER BY created_at, id):
|
||||
-- one total order for every consumer, so concurrent drains cannot
|
||||
-- deadlock on row locks. Covers the whole ORDER BY (bulk ingest
|
||||
-- shares request-at constantly, so ties are the norm).
|
||||
CREATE INDEX IF NOT EXISTS audit_log__created_at__idx
|
||||
ON audit_log (created_at ASC, id);
|
||||
@ -27,13 +27,14 @@
|
||||
ALTER TABLE file_data
|
||||
ADD COLUMN IF NOT EXISTS storage_object_id uuid;
|
||||
|
||||
-- The CTE is MATERIALIZED so the uuid cast only sees rows that passed the
|
||||
-- regex. A plain subquery can be inlined, and a corrupt value would reach the
|
||||
-- cast and abort the migration.
|
||||
-- The cast is guarded as well as materialized: corrupt values cannot reach it
|
||||
-- even if PostgreSQL reorders expressions in the WHERE clause.
|
||||
WITH candidates AS MATERIALIZED (
|
||||
SELECT file_id,
|
||||
id,
|
||||
(metadata->>'storage-ref-id')::uuid AS soid
|
||||
CASE WHEN (metadata->>'storage-ref-id') ~*
|
||||
'^[0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{12}$'
|
||||
THEN (metadata->>'storage-ref-id')::uuid END AS soid
|
||||
FROM file_data
|
||||
WHERE backend = 'storage'
|
||||
AND storage_object_id IS NULL
|
||||
@ -48,6 +49,9 @@ UPDATE file_data AS fd
|
||||
AND fd.id = c.id
|
||||
AND EXISTS (SELECT 1 FROM storage_object AS so WHERE so.id = c.soid);
|
||||
|
||||
-- The old key is removed from every row migrated by this statement. A rollback
|
||||
-- to the previous code after the backfill requires restoring the backup.
|
||||
|
||||
-- The deletion check of the foreign key looks a row up by this column for
|
||||
-- every deleted object; without the index it scans all partitions.
|
||||
CREATE INDEX IF NOT EXISTS file_data__storage_object_id__idx
|
||||
|
||||
218
backend/src/app/migrations/sql/0159-consumer-generalization.sql
Normal file
218
backend/src/app/migrations/sql/0159-consumer-generalization.sql
Normal file
@ -0,0 +1,218 @@
|
||||
-- Consumer generalization: the schema of the whole feature in one step.
|
||||
--
|
||||
-- This single migration replaces the steps this branch used to ship
|
||||
-- separately (exclusive jobs, the audit cursor and dedupe jobs). The
|
||||
-- migration runner executes every pending step inside ONE transaction, so
|
||||
-- several files bought no isolation: their locks piled up until the same
|
||||
-- commit, and pre-applying one of them out of band did not avoid the
|
||||
-- others.
|
||||
--
|
||||
-- The pending-queue step that sat between them never shipped and is gone:
|
||||
-- released versions never created `audit_log_pending`, so this migration
|
||||
-- creates neither the table nor the `(created_at, id)` index it added. The
|
||||
-- table itself is dropped below, for a pre-release database that happens
|
||||
-- to have it.
|
||||
--
|
||||
-- The legacy mark (`archived_at`) and the indexes that served it stay in
|
||||
-- place on purpose: the old archiver may still be running during the
|
||||
-- rollout, and a later release removes them. This migration does not build
|
||||
-- the `(created_at, id)` index either; the consumers order by `(xid, seq)`,
|
||||
-- so nothing reads by `(source, created_at)` any more and that index waits
|
||||
-- for a later cleanup.
|
||||
--
|
||||
-- The migration is idempotent: every statement is IF NOT EXISTS / IF
|
||||
-- EXISTS, so it is safe to re-run. Applying it out of band does not make
|
||||
-- it free, though: the guards turn the statements into no-ops, not into
|
||||
-- lock-free ones (an ALTER still takes ACCESS EXCLUSIVE and the backfill
|
||||
-- still scans `audit_log`), so the operator registers the step in
|
||||
-- `migrations` once the schema is pre-applied (step 0 of the runbook,
|
||||
-- `operations/2.20.0-consumer-generalization.md`).
|
||||
--
|
||||
-- The statements take the locks in the same order the audit-log producer
|
||||
-- uses (audit_log, then audit_log_pending, then job), so a deploy that
|
||||
-- still has the previous version running cannot deadlock against it: the
|
||||
-- producer inserts the audit row, then its pending (when that version has
|
||||
-- it), then submits the webhook job, and this migration walks the same
|
||||
-- order. What it takes last (`job`) is what the producer takes last.
|
||||
|
||||
-- ---------------------------------------------------------------------
|
||||
-- Audit log cursors: each consumer keeps a position over `audit_log`
|
||||
-- instead of a pending queue, and the row is deleted when every enabled
|
||||
-- consumer has passed it.
|
||||
--
|
||||
-- `seq` is the insertion order (a sequence, so it costs no extra WAL);
|
||||
-- `xid` is the transaction that wrote the row. The pair gives a total
|
||||
-- order that real commits respect: the consumer reads only up to
|
||||
-- pg_snapshot_xmin(), the oldest live transaction, so it never steps
|
||||
-- over a row whose transaction is still open. `xid8` (64 bits) does not
|
||||
-- wrap, unlike the row `xmin` (32 bits); it exists since PostgreSQL 13.
|
||||
-- ---------------------------------------------------------------------
|
||||
|
||||
CREATE SEQUENCE IF NOT EXISTS audit_log_seq_seq;
|
||||
|
||||
ALTER TABLE audit_log
|
||||
ADD COLUMN IF NOT EXISTS seq bigint,
|
||||
-- Fast default (no table rewrite): the rows that already exist get 0
|
||||
-- and sort before every real xid8. They are the history, and the first
|
||||
-- deploy replays them (the consumers create their cursor at (0, 0) on
|
||||
-- their first pass).
|
||||
ADD COLUMN IF NOT EXISTS xid xid8 NOT NULL DEFAULT '0'::xid8;
|
||||
|
||||
ALTER SEQUENCE audit_log_seq_seq OWNED BY audit_log.seq;
|
||||
|
||||
-- Reserve the low range for the history: the new rows start above the
|
||||
-- backfill, so any `ORDER BY seq` stays chronological and the backfill
|
||||
-- can never collide with a value produced by nextval. The margin covers
|
||||
-- the rows the previous version may insert between this count and the
|
||||
-- SET DEFAULT (a few statements away).
|
||||
--
|
||||
-- Only the first application sets it: a sequence that was never called
|
||||
-- (`last_value = 1` with `is_called = false`) is intact; any other state
|
||||
-- (reserved by the operator, in use, or already reserved by a previous
|
||||
-- run) must be left alone, because moving it back would reuse values above
|
||||
-- the history and break the reserve.
|
||||
SELECT setval('audit_log_seq_seq',
|
||||
(SELECT count(*) + 1000000 FROM audit_log),
|
||||
false)
|
||||
WHERE (SELECT last_value FROM audit_log_seq_seq) = 1
|
||||
AND NOT (SELECT is_called FROM audit_log_seq_seq);
|
||||
|
||||
-- From here no new row can be NULL: the previous version keeps inserting
|
||||
-- during the rollout and must leave the columns filled.
|
||||
ALTER TABLE audit_log
|
||||
ALTER COLUMN seq SET DEFAULT nextval('audit_log_seq_seq'),
|
||||
ALTER COLUMN xid SET DEFAULT pg_current_xact_id();
|
||||
|
||||
-- Backfill the history with 1..N, in the order the previous version sent
|
||||
-- it (`created_at, id`), so the replay keeps the archive order. A single
|
||||
-- statement: it takes row locks, never a table lock, so the previous
|
||||
-- version keeps inserting and the statement can be interrupted and
|
||||
-- re-run (it filters `seq IS NULL`). The sort is on purpose: the
|
||||
-- `(created_at, id)` index belonged to the pending queue and never
|
||||
-- shipped, so there is no index to serve this order.
|
||||
UPDATE audit_log
|
||||
SET seq = ordered.rn
|
||||
FROM (SELECT id, row_number() OVER (ORDER BY created_at, id) AS rn
|
||||
FROM audit_log
|
||||
WHERE seq IS NULL) ordered
|
||||
WHERE audit_log.id = ordered.id;
|
||||
|
||||
ALTER TABLE audit_log
|
||||
ALTER COLUMN seq SET NOT NULL;
|
||||
|
||||
-- `source` is nullable since 0064 (its backfill only covered the rows of
|
||||
-- that moment). The producer has written it ever since, but the GC excludes
|
||||
-- telemetry shadows with `source NOT LIKE 'telemetry:%'`, and with a NULL
|
||||
-- source that predicate is unknown: neither the cursor paths nor the
|
||||
-- retention window would ever delete such a row. Fill what is left and close
|
||||
-- the column.
|
||||
UPDATE audit_log SET source = 'backend' WHERE source IS NULL;
|
||||
|
||||
ALTER TABLE audit_log
|
||||
ALTER COLUMN source SET NOT NULL;
|
||||
|
||||
-- Covers both predicates of the scan: the cursor comparison and the
|
||||
-- snapshot cut are Index Cond, so the index serves the range and the
|
||||
-- (xid, seq) order of the scan.
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS audit_log__xid__seq__idx
|
||||
ON audit_log (xid, seq);
|
||||
|
||||
-- One row per consumer: its position, updated in the same transaction
|
||||
-- that sends the chunk. Consumer ids live in app.loggers.audit.consumers
|
||||
-- and are never reused: 1 = nexus, 2 = admin console.
|
||||
--
|
||||
-- The table starts empty on purpose: the migration does not know which
|
||||
-- consumers are enabled. `ensure-cursor` creates the row at (0, 0) on a
|
||||
-- consumer's first pass, so the first deploy **replays** the unarchived
|
||||
-- buffer instead of dropping it (the old archiver is disabled and the
|
||||
-- already-archived rows are deleted before deploying, see the NOTE).
|
||||
CREATE TABLE IF NOT EXISTS audit_log_consumer_cursor (
|
||||
consumer smallint NOT NULL,
|
||||
xid xid8 NOT NULL,
|
||||
seq bigint NOT NULL,
|
||||
updated_at timestamptz NOT NULL DEFAULT now(),
|
||||
PRIMARY KEY (consumer));
|
||||
|
||||
-- ---------------------------------------------------------------------
|
||||
-- The old pending queue never shipped: this release's producer does not
|
||||
-- fill it, and no released version ever created the table. Only a
|
||||
-- pre-release database that ran the pending-queue step has it, and that
|
||||
-- one may keep rows left behind. Drop the table: after the rollout nobody
|
||||
-- drains the queue, so a pending row would make the new GC fail on every
|
||||
-- pass when it deletes the row it references. A released database does
|
||||
-- not have it, so this is a no-op there.
|
||||
-- ---------------------------------------------------------------------
|
||||
|
||||
DROP TABLE IF EXISTS audit_log_pending;
|
||||
|
||||
-- ---------------------------------------------------------------------
|
||||
-- Jobs: at most one live exclusive job and at most one not-started
|
||||
-- dedupe job per (tenant, name, queue, label).
|
||||
--
|
||||
-- `exclusive` marks the row of a `::jobs/skip-if-active` submit, and the
|
||||
-- index admits a single live row (new, scheduled, running or retry) for
|
||||
-- its key. `dedupe` marks a `::jobs/dedupe`/`::jobs/debounce` submit, and
|
||||
-- its index admits a single `new` row. The submit no longer counts or
|
||||
-- deletes before inserting: the partial unique index decides and the
|
||||
-- insert resolves the conflict with `ON CONFLICT`, so the exclusion holds
|
||||
-- across backends.
|
||||
--
|
||||
-- Both columns default to false, so the rows that never asked for the
|
||||
-- option stay out of the indexes (the label stays a label) and the
|
||||
-- indexes start empty: no cleanup of duplicates is needed.
|
||||
-- ---------------------------------------------------------------------
|
||||
|
||||
ALTER TABLE job
|
||||
ADD COLUMN IF NOT EXISTS exclusive boolean NOT NULL DEFAULT false;
|
||||
|
||||
ALTER TABLE job
|
||||
ADD COLUMN IF NOT EXISTS dedupe boolean NOT NULL DEFAULT false;
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS job__live_exclusive__uniq
|
||||
ON job (tenant, name, queue, label)
|
||||
WHERE exclusive
|
||||
AND status IN ('new', 'scheduled', 'running', 'retry');
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS job__new_dedupe__uniq
|
||||
ON job (tenant, name, queue, label)
|
||||
WHERE dedupe
|
||||
AND status = 'new';
|
||||
|
||||
-- The plain index over (tenant, name, queue, label) served the old cron
|
||||
-- no-overlap probe. Nothing reads it now: the cron resolves the overlap at
|
||||
-- insert time with a `::jobs/skip-if-active` submit (that is,
|
||||
-- `job__live_exclusive__uniq`), which is one index instead of a probe plus
|
||||
-- a race. Bases built from an early 0154 drop it here.
|
||||
DROP INDEX IF EXISTS job__name_label__idx;
|
||||
|
||||
-- The lost sweep looks for `scheduled` rows by their hand-off time
|
||||
-- (`modified_at`), not by their due time: the old index served the removed
|
||||
-- reschedule and only narrows by tenant now. The `job` table is not in
|
||||
-- production yet, so the rebuild does not need CONCURRENTLY.
|
||||
DROP INDEX IF EXISTS job__scheduled__idx;
|
||||
|
||||
CREATE INDEX IF NOT EXISTS job__scheduled__idx
|
||||
ON job (tenant, modified_at)
|
||||
WHERE status = 'scheduled';
|
||||
|
||||
-- ---------------------------------------------------------------------
|
||||
-- NOTE: on an instance with a large `job` or `audit_log` table, apply this
|
||||
-- migration out of band before deploying. The runner executes all pending
|
||||
-- steps in a single transaction, so the plain indexes below hold a SHARE
|
||||
-- lock on the table for the whole build and every ALTER keeps its ACCESS
|
||||
-- EXCLUSIVE until that one commit; on tables that keep taking writes (every
|
||||
-- file save inserts a job and an audit row) that is minutes of stalled
|
||||
-- writes.
|
||||
--
|
||||
-- The complete procedure (lock timeouts, the `CHECK NOT VALID` dance, the
|
||||
-- `indisvalid` gate, registering the step and the optional post-rollout
|
||||
-- drop for a pre-release database) lives in
|
||||
-- `operations/2.20.0-consumer-generalization.md`. Pre-applying does not make
|
||||
-- the migration free by itself: the guards skip the work, not the locks, so
|
||||
-- registering the step is what makes the runner skip the group.
|
||||
--
|
||||
-- The new consumers read only rows below
|
||||
-- `pg_snapshot_xmin(pg_current_snapshot())`, the oldest transaction of the
|
||||
-- whole database: a long write transaction (a binfile import runs in one for
|
||||
-- up to 30 minutes) pauses archiving and the GC until it commits. Nothing is
|
||||
-- lost and the lag gauge shows it; it is not a stuck consumer.
|
||||
@ -113,8 +113,22 @@
|
||||
:status status
|
||||
:hint (str "nitrate HTTP " status " at " uri))
|
||||
nil))
|
||||
(= status 204) ;; 204 doesn't return any body
|
||||
;; A 2xx with no expected body is success: 204 by contract, and any
|
||||
;; other 2xx for the methods that never return a body. Building a
|
||||
;; coercer from a nil schema throws before the try below can catch it.
|
||||
(and (<= 200 status 299)
|
||||
(or (= status 204) (nil? schema)))
|
||||
nil
|
||||
;; A redirect is a configuration error (a proxy normalising the url,
|
||||
;; a trailing slash, an auth hop), never a success: raise a typed
|
||||
;; error so the caller does not read it as an empty or confirmed
|
||||
;; response. This sits outside `with-retries` on purpose.
|
||||
(<= 300 status 399)
|
||||
(do
|
||||
(l/error :hint "nitrate request was redirected" :uri uri :status status)
|
||||
(ex/raise :type :nitrate-http-error
|
||||
:status status
|
||||
:hint (str "nitrate redirected, HTTP " status " at " uri)))
|
||||
:else ;; For success status codes, validate the response
|
||||
(let [coercer-http (sm/coercer schema
|
||||
:type :validation
|
||||
@ -508,8 +522,8 @@
|
||||
|
||||
(defn- ingest-audit-log-api
|
||||
"POST a batch of audit events to Admin Console `/api/audit-log`.
|
||||
Expects HTTP 204; returns nil on success. 4xx raises when
|
||||
`:throw-on-error?` is set."
|
||||
Any 2xx is success and returns nil (the endpoint answers 204 by
|
||||
contract). 4xx raises when `:throw-on-error?` is set."
|
||||
[cfg {:keys [events] :as params}]
|
||||
(let [request-params {:events (mapv ->audit-event events)}
|
||||
params (assoc params
|
||||
|
||||
@ -71,6 +71,7 @@
|
||||
"Public API of default redis connection"
|
||||
(-publish [_ topic message])
|
||||
(-rpush [_ key payload])
|
||||
(-llen [_ key])
|
||||
(-blpop [_ timeout keys])
|
||||
(-eval [_ script])
|
||||
(-get [_ key])
|
||||
@ -195,6 +196,9 @@
|
||||
(catch RedisCommandInterruptedException cause
|
||||
(throw (InterruptedException. (ex-message cause))))))
|
||||
|
||||
(-llen [_ key]
|
||||
(.llen cmd ^String key))
|
||||
|
||||
(-blpop [_ keys timeout]
|
||||
(try
|
||||
(let [keys (into-array String keys)]
|
||||
@ -445,6 +449,11 @@
|
||||
(assert (every? string? keys) "only string keys allowed")
|
||||
(-blpop conn keys timeout)))
|
||||
|
||||
(defn llen
|
||||
[conn key]
|
||||
(assert (string? key) "key must be string instance")
|
||||
(-llen conn key))
|
||||
|
||||
(defn rpush
|
||||
[conn key elements]
|
||||
(assert (string? key) "key must be string instance")
|
||||
|
||||
@ -212,6 +212,13 @@
|
||||
:else
|
||||
(f cfg params))))))
|
||||
|
||||
;; Only the methods that declare `::db/transaction true` get a transaction;
|
||||
;; every wrapper applied after this point in `wrap` runs outside it. That is
|
||||
;; the case of `wrap-audit`: the event is written after the command commits,
|
||||
;; so a failure there cannot roll the action back and the client gets an
|
||||
;; error over committed work. Making the transaction the default for every
|
||||
;; command, so the audit event can move inside it, is project P8 in TODO.md
|
||||
;; and is out of scope here.
|
||||
(defn- wrap-db-transaction
|
||||
[_ f mdata]
|
||||
(if (::db/transaction mdata)
|
||||
@ -221,9 +228,9 @@
|
||||
|
||||
(defn- wrap-audit
|
||||
[_ f mdata]
|
||||
(if (or (contains? cf/flags :webhooks)
|
||||
(contains? cf/flags :audit-log)
|
||||
(contains? cf/flags :telemetry))
|
||||
(if (or cf/webhooks-enabled?
|
||||
cf/audit-persist?
|
||||
cf/telemetry-enabled?)
|
||||
(if-not (::audit/skip mdata)
|
||||
(fn [cfg params]
|
||||
(let [result (f cfg params)]
|
||||
|
||||
@ -16,7 +16,6 @@
|
||||
[app.db :as db]
|
||||
[app.http :as-alias http]
|
||||
[app.loggers.audit :as audit]
|
||||
[app.loggers.audit.consumer-task :as consumer]
|
||||
[app.loggers.database :as loggers.db]
|
||||
[app.loggers.mattermost :as loggers.mm]
|
||||
[app.metrics :as mtx]
|
||||
@ -25,8 +24,7 @@
|
||||
[app.rpc.doc :as-alias doc]
|
||||
[app.rpc.helpers :as rph]
|
||||
[app.util.inet :as inet]
|
||||
[app.util.services :as sv]
|
||||
[clojure.set :as set]))
|
||||
[app.util.services :as sv]))
|
||||
|
||||
(def ^:private event-columns
|
||||
[:id
|
||||
@ -125,10 +123,10 @@
|
||||
(defn- insert-rows!
|
||||
[tx rows]
|
||||
(when (seq rows)
|
||||
(db/insert-many! tx :audit-log event-columns rows)
|
||||
;; Same transaction as the rows: every ingested record is born
|
||||
;; pending for every enabled consumer (see consumer-task).
|
||||
(consumer/queue-pendings tx (map first rows))))
|
||||
;; The producer only inserts: each consumer reads the row from its
|
||||
;; own cursor, so there is no queue to fill here (and no consumer
|
||||
;; contract to keep in sync with this namespace).
|
||||
(db/insert-many! tx :audit-log event-columns rows)))
|
||||
|
||||
(defn- handle-events
|
||||
[{:keys [::db/pool] :as cfg} params]
|
||||
@ -141,15 +139,15 @@
|
||||
(run! (partial loggers.db/emit cfg) events)
|
||||
(run! (partial loggers.mm/emit cfg) events))
|
||||
|
||||
;; Rows and their pendings commit together in one transaction: a
|
||||
;; row never outlives its queue. Both faces are best-effort, so a
|
||||
;; failure drops the whole batch (counted below as failed); the
|
||||
;; The rows of both faces commit together (the consumers read them
|
||||
;; from their own cursors afterwards). Both faces are best-effort,
|
||||
;; so a failure drops the whole batch (counted below as failed); the
|
||||
;; endpoint still answers without breaking the request.
|
||||
(let [metrics (mtx/instance cfg)
|
||||
full-rows (if (contains? cf/flags :audit-log)
|
||||
full-rows (if cf/audit-persist?
|
||||
(into [] xf:map-event-row events)
|
||||
[])
|
||||
shadow-rows (if (contains? cf/flags :telemetry)
|
||||
shadow-rows (if cf/telemetry-enabled?
|
||||
(into [] xf:map-telemetry-event-row events)
|
||||
[])]
|
||||
(try
|
||||
@ -196,10 +194,14 @@
|
||||
::doc/skip true
|
||||
::doc/added "1.17"}
|
||||
[{:keys [::db/pool] :as cfg} params]
|
||||
(let [telemetry? (contains? cf/flags :telemetry)
|
||||
audit-log? (contains? cf/flags :audit-log)
|
||||
(let [telemetry? cf/telemetry-enabled?
|
||||
;; :error-reporting also needs the batch: the crash events reach
|
||||
;; the reporters here, before any row is written. The wire signal
|
||||
;; in get-environment-data includes the same case.
|
||||
enabled? (and (not (db/read-only? pool))
|
||||
(or audit-log? telemetry?))]
|
||||
(or cf/audit-persist?
|
||||
telemetry?
|
||||
cf/error-reporting-enabled?))]
|
||||
(when enabled?
|
||||
(try
|
||||
(handle-events cfg params)
|
||||
@ -213,15 +215,19 @@
|
||||
;; GET-ENVIRONMENT-DATA
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(def ^:private enabled-flags
|
||||
"Subset of the backend flags that are safe to expose publicly and that
|
||||
drive frontend event collection."
|
||||
#{:audit-log :telemetry})
|
||||
|
||||
(defn- environment-data
|
||||
"The audit-log signal on the wire means the backend stores full rows. It is
|
||||
derived from the boot-computed persistence value, not from a configurable
|
||||
flag, and also covers error reporting so frontend crash events keep flowing."
|
||||
[]
|
||||
{:deployment (if (cf/get :is-saas) "saas" "selfhost")
|
||||
:flags (set/intersection cf/flags enabled-flags)})
|
||||
:flags (cond-> #{}
|
||||
(or cf/audit-persist?
|
||||
cf/error-reporting-enabled?)
|
||||
(conj :audit-log)
|
||||
|
||||
cf/telemetry-enabled?
|
||||
(conj :telemetry))})
|
||||
|
||||
(def ^:private schema:get-environment-data-result
|
||||
[:map
|
||||
|
||||
@ -257,12 +257,15 @@
|
||||
(feat.fdata/decode-file-data cfg))]
|
||||
(select-keys fragment (sm/keys schema:file-fragment)))))
|
||||
|
||||
(defn- check-fragment-scope!
|
||||
(defn- check-fragment-scope
|
||||
"Checks that the fragment is reachable from the pages authorized by
|
||||
the share-link. Raises a :not-found exception if the fragment is not reachable."
|
||||
the share-link. Raises a :not-found exception if the fragment is not reachable.
|
||||
Only the fragments of the link's pages are authorized: root-level pointers
|
||||
(such as the file's components) never pass."
|
||||
[cfg file-id fragment-id pages]
|
||||
(let [fdata (-> (bfc/get-file cfg file-id :read-only? true)
|
||||
(get :data)
|
||||
(select-keys [:pages-index])
|
||||
(update :pages-index select-keys pages))]
|
||||
(when-not (contains? (feat.fdata/get-used-pointer-ids fdata) fragment-id)
|
||||
(ex/raise :type :not-found
|
||||
@ -280,7 +283,7 @@
|
||||
(let [perms (perms/get-file-read-permissions cfg profile-id file-id share-id)]
|
||||
(check-read-permissions! perms)
|
||||
(when (= :share-link (:type perms))
|
||||
(check-fragment-scope! cfg file-id fragment-id (:pages perms)))
|
||||
(check-fragment-scope cfg file-id fragment-id (:pages perms)))
|
||||
(-> (get-file-fragment cfg file-id fragment-id)
|
||||
(rph/with-http-cache long-cache-duration))))))
|
||||
|
||||
@ -1034,7 +1037,10 @@
|
||||
:modified-at (ct/now)
|
||||
:has-media-trimmed false))))
|
||||
|
||||
(defn- get-referenced-files
|
||||
(defn get-referenced-files
|
||||
"Ids of the live files using the given file as a library, oldest
|
||||
first. Public so the delete cascade can capture the dependents
|
||||
before removing the relations."
|
||||
[cfg library-id]
|
||||
(->> (db/exec! cfg [sql:get-referenced-files library-id])
|
||||
(into [] bfc/xf-map-id)))
|
||||
@ -1148,23 +1154,17 @@
|
||||
file (db/update! conn :file
|
||||
{:deleted-at (ct/in-future delay)}
|
||||
{:id file-id}
|
||||
{::db/return-keys [:id :name :is-shared :deleted-at
|
||||
:project-id :created-at :modified-at]})
|
||||
;; Files using the library, captured before their relations are removed
|
||||
file-ids (when (:is-shared file)
|
||||
(get-referenced-files conn file-id))]
|
||||
|
||||
;; Remove all possible relations for that file
|
||||
(db/delete! conn :file-library-rel
|
||||
{:library-file-id file-id})
|
||||
|
||||
{::db/return-keys [:name :deleted-at
|
||||
:project-id :created-at :modified-at]})]
|
||||
;; Only the mark happens here, so the file disappears at once. The
|
||||
;; cascade job captures the files using this one as a library,
|
||||
;; removes the relations and absorbs them: the relations must still
|
||||
;; exist when the job runs.
|
||||
(jobs/submit cfg
|
||||
{::jobs/name :delete-object
|
||||
::jobs/params (cond-> {:object :file
|
||||
:deleted-at (:deleted-at file)
|
||||
:id file-id}
|
||||
(some? file-ids)
|
||||
(assoc :file-ids file-ids))})
|
||||
::jobs/params {:object :file
|
||||
:deleted-at (:deleted-at file)
|
||||
:id file-id}})
|
||||
file))
|
||||
|
||||
(def ^:private
|
||||
|
||||
@ -340,6 +340,12 @@
|
||||
deleted-at
|
||||
(ct/plus timestamp (ct/duration {:hours 1}))
|
||||
|
||||
;; The id of the change row (xlog) goes to the audit event
|
||||
;; too: a consumer that reads the event needs to know what the
|
||||
;; save changed, and that row is the only place with it.
|
||||
file-change-id
|
||||
(uuid/next)
|
||||
|
||||
repair
|
||||
(::repair file)
|
||||
|
||||
@ -397,7 +403,7 @@
|
||||
;; file are not repeated here (nobody reads them from a revision, and
|
||||
;; they are duplicated on every save).
|
||||
(db/insert! conn :file-change
|
||||
{:id (uuid/next)
|
||||
{:id file-change-id
|
||||
:kind fchanges/revision-kind
|
||||
:session-id session-id
|
||||
:profile-id profile-id
|
||||
@ -424,18 +430,35 @@
|
||||
(some? rejected)
|
||||
(assoc :rejected rejected))
|
||||
{::audit/replace-props
|
||||
{:id (:id file)
|
||||
:name (:name file)
|
||||
:features (:features file)
|
||||
:project-id (:project-id file)
|
||||
:team-id (:team-id file)
|
||||
;; NOTE: `file` carries the post-update revn here (`revn`
|
||||
;; above is the pre-update one kept for the response), and
|
||||
;; `modified-at` mirrors `persist-file!`: same `::timestamp`
|
||||
;; fallback, so both agree to the millisecond when the
|
||||
;; request timestamp exists.
|
||||
:revn (if saved? (:revn file) revn)
|
||||
:modified-at (or timestamp (ct/now))}}))))
|
||||
(cond-> {:id (:id file)
|
||||
:name (:name file)
|
||||
:features (:features file)
|
||||
:project-id (:project-id file)
|
||||
:team-id (:team-id file)
|
||||
;; NOTE: `file` carries the post-update revn here (`revn`
|
||||
;; above is the pre-update one kept for the response).
|
||||
:revn (if saved? (:revn file) revn)
|
||||
;; How the save ended: every change written, the accepted
|
||||
;; prefix written and the rest rejected, or nothing
|
||||
;; written. A consumer must not read `update-file` as a
|
||||
;; state change without looking at this.
|
||||
:outcome (cond
|
||||
(not saved?) "rejected"
|
||||
(some? rejected) "partial"
|
||||
:else "saved")
|
||||
;; A rejected save does not call `persist-file!`, so the
|
||||
;; row keeps its previous instant: mirror the persisted
|
||||
;; value instead of claiming a modification.
|
||||
:modified-at (if saved?
|
||||
(or timestamp (ct/now))
|
||||
(:modified-at file))}
|
||||
|
||||
;; The id of the change row inserted above: what the save
|
||||
;; changed lives in that row, not in the event. A fully rejected
|
||||
;; save writes no row, and an id that resolves to nothing cannot
|
||||
;; be told apart from one the GC already purged.
|
||||
saved?
|
||||
(assoc :file-change-id file-change-id))}))))
|
||||
|
||||
(defn get-file
|
||||
"Get not-decoded file, only decodes the features set.
|
||||
|
||||
@ -25,8 +25,9 @@
|
||||
[app.rpc.commands.projects :as proj]
|
||||
[app.rpc.commands.teams :as teams]
|
||||
[app.rpc.doc :as-alias doc]
|
||||
[app.rpc.notifications :as ntf]
|
||||
[app.rpc.permissions :as perms]
|
||||
[app.rpc.notifications :as ntf]
|
||||
[app.rpc.permissions :as perms]
|
||||
[app.rpc.helpers :as rph]
|
||||
[app.setup :as-alias setup]
|
||||
[app.setup.templates :as tmpl]
|
||||
[app.storage.tmp :as tmp]
|
||||
@ -288,7 +289,10 @@
|
||||
(ntf/notify-team-permissions-changed cfg team-id)))
|
||||
|
||||
(def sql:get-files
|
||||
"select id, features, project_id from file where id = ANY(?)")
|
||||
"select f.id, f.features, f.project_id, p.team_id
|
||||
from file as f
|
||||
join project as p on (p.id = f.project_id)
|
||||
where f.id = ANY(?)")
|
||||
|
||||
(def sql:move-files
|
||||
"update file set project_id = ? where id = ANY(?)")
|
||||
@ -316,7 +320,15 @@
|
||||
(map files/decode-row))
|
||||
source (into #{} (map :project-id) files)
|
||||
pids (->> (conj source project-id)
|
||||
(db/create-array conn "uuid"))]
|
||||
(db/create-array conn "uuid"))
|
||||
;; A move carries a single origin team in its props: refuse
|
||||
;; mixed-origin requests up front instead of guessing.
|
||||
source-teams (into #{} (keep :team-id) files)]
|
||||
|
||||
(when (> (count source-teams) 1)
|
||||
(ex/raise :type :validation
|
||||
:code :cant-move-from-multiple-teams
|
||||
:hint "Unable to move files from different teams at once"))
|
||||
|
||||
(when (contains? source project-id)
|
||||
(ex/raise :type :validation
|
||||
@ -330,7 +342,9 @@
|
||||
(doseq [project-id source]
|
||||
(proj/check-edition-permissions! conn profile-id project-id))
|
||||
|
||||
;; Check the team compatibility
|
||||
;; Check the team compatibility, then move: the destination team
|
||||
;; is already loaded here, so the event props below need no extra
|
||||
;; query.
|
||||
(let [orig-team (teams/get-team conn :profile-id profile-id :project-id (first source))
|
||||
dest-team (teams/get-team conn :profile-id profile-id :project-id project-id)]
|
||||
(cfeat/check-teams-compatibility! orig-team dest-team)
|
||||
@ -338,24 +352,24 @@
|
||||
;; Check if all pending to move files are compaib
|
||||
(let [features (cfeat/get-team-enabled-features cf/flags dest-team)]
|
||||
(doseq [file files]
|
||||
(cfeat/check-file-features! features (:features file)))))
|
||||
(cfeat/check-file-features! features (:features file))))
|
||||
|
||||
;; move all files to the project
|
||||
(db/exec-one! conn [sql:move-files project-id fids])
|
||||
;; move all files to the project
|
||||
(db/exec-one! conn [sql:move-files project-id fids])
|
||||
|
||||
;; delete possible broken relations on moved files
|
||||
(db/exec-one! conn [sql:delete-broken-relations pids])
|
||||
;; delete possible broken relations on moved files
|
||||
(db/exec-one! conn [sql:delete-broken-relations pids])
|
||||
|
||||
;; Update the modification date of the all affected projects
|
||||
;; ensuring that the destination project is the most recent one.
|
||||
(loop [project-ids (into (list project-id) source)
|
||||
modified-at (ct/now)]
|
||||
(when-let [project-id (first project-ids)]
|
||||
(db/update! conn :project
|
||||
{:modified-at modified-at}
|
||||
{:id project-id})
|
||||
(recur (rest project-ids)
|
||||
(ct/plus modified-at 10))))
|
||||
;; Update the modification date of the all affected projects
|
||||
;; ensuring that the destination project is the most recent one.
|
||||
(loop [project-ids (into (list project-id) source)
|
||||
modified-at (ct/now)]
|
||||
(when-let [project-id (first project-ids)]
|
||||
(db/update! conn :project
|
||||
{:modified-at modified-at}
|
||||
{:id project-id})
|
||||
(recur (rest project-ids)
|
||||
(ct/plus modified-at 10))))
|
||||
|
||||
;; Moving files can take them out of the reach of the team they came
|
||||
;; from. Only the open connections know who was relying on them, so
|
||||
@ -365,7 +379,11 @@
|
||||
cfg
|
||||
(keep (partial perms/get-team-id-for-project conn) source))
|
||||
|
||||
nil))
|
||||
;; Both teams travel explicitly in the event props so a consumer
|
||||
;; never joins production tables.
|
||||
(rph/with-meta (rph/wrap)
|
||||
{::audit/props {:source-team-id (first source-teams)
|
||||
:dest-team-id (:id dest-team)}}))))
|
||||
|
||||
(def ^:private
|
||||
schema:move-files
|
||||
@ -376,6 +394,7 @@
|
||||
(sv/defmethod ::move-files
|
||||
"Move a set of files from one project to other."
|
||||
{::doc/added "1.16"
|
||||
::doc/changes [["2.20" "Files from more than one source team are rejected with :cant-move-from-multiple-teams"]]
|
||||
::webhooks/event? true
|
||||
::sm/params schema:move-files}
|
||||
[cfg {:keys [::rpc/profile-id] :as params}]
|
||||
@ -438,7 +457,11 @@
|
||||
;; destination team only gains access, so it is not announced.
|
||||
(notify-team-access-changed cfg [(:team-id project)])
|
||||
|
||||
nil))
|
||||
;; Both teams travel explicitly in the event props so a consumer
|
||||
;; never joins production tables.
|
||||
(rph/with-meta (rph/wrap)
|
||||
{::audit/props {:source-team-id (:team-id project)
|
||||
:dest-team-id team-id}})))
|
||||
|
||||
(def ^:private
|
||||
schema:move-project
|
||||
|
||||
@ -645,6 +645,11 @@
|
||||
|
||||
;; --- HELPERS
|
||||
|
||||
;; Owned teams with their participant count. The predicate over
|
||||
;; `t.deleted_at` is injected: the read path asks for live teams, and the
|
||||
;; restore path asks for the ones the delete cascade stamped with the
|
||||
;; profile's own instant. Both predicates are constants of this namespace,
|
||||
;; never operator input.
|
||||
(def sql:owned-teams
|
||||
"WITH owner_teams AS (
|
||||
SELECT tpr.team_id AS id
|
||||
@ -652,7 +657,7 @@
|
||||
JOIN team AS t ON (t.id = tpr.team_id)
|
||||
WHERE tpr.is_owner IS TRUE
|
||||
AND tpr.profile_id = ?
|
||||
AND t.deleted_at IS NULL
|
||||
AND %s
|
||||
)
|
||||
SELECT tpr.team_id AS id,
|
||||
count(tpr.profile_id) - 1 AS participants
|
||||
@ -661,8 +666,16 @@
|
||||
GROUP BY 1")
|
||||
|
||||
(defn get-owned-teams
|
||||
[conn profile-id]
|
||||
(db/exec! conn [sql:owned-teams profile-id]))
|
||||
"Teams the profile owns. `:deleted-at` returns the ones soft-deleted at
|
||||
that exact instant: the delete cascade stamps the children with the
|
||||
parent's date, so the restore path uses it to bring back only what the
|
||||
cascade took. Without it the answer holds live teams only."
|
||||
[conn profile-id & {:keys [deleted-at]}]
|
||||
(if (some? deleted-at)
|
||||
(db/exec! conn [(format sql:owned-teams "t.deleted_at = ?::timestamptz")
|
||||
profile-id deleted-at])
|
||||
(db/exec! conn [(format sql:owned-teams "t.deleted_at IS NULL")
|
||||
profile-id])))
|
||||
|
||||
(def ^:private sql:profile-existence
|
||||
"select exists (select * from profile
|
||||
|
||||
@ -656,7 +656,7 @@
|
||||
action owner, defaulting to the member) and `:rpc-params` (request context,
|
||||
which sibling events get from `prepare-rpc-event`)."
|
||||
[cfg {:keys [profile-id team-id] :as params} {:keys [actor-id rpc-params]}]
|
||||
(when (contains? cf/flags :audit-log)
|
||||
(when cf/audit-persist?
|
||||
(audit/submit cfg
|
||||
(-> (audit/event-from-rpc-params (or rpc-params {}))
|
||||
(assoc :name "add-team-member"
|
||||
|
||||
@ -12,6 +12,7 @@
|
||||
[app.common.features :as cfeat]
|
||||
[app.common.files.helpers :as cfh]
|
||||
[app.common.schema :as sm]
|
||||
[app.common.types.file :as ctf]
|
||||
[app.config :as cf]
|
||||
[app.db :as db]
|
||||
[app.features.fdata :as fdata]
|
||||
@ -37,8 +38,19 @@
|
||||
(update :pages (fn [pages] (filterv #(contains? allowed %) pages)))
|
||||
(update :pages-index select-keys allowed)))
|
||||
|
||||
(defn- same-file-component-ref?
|
||||
"Whether `shape` references a component of this same file. A reference
|
||||
missing `:component-file` means a same-file reference."
|
||||
[file-id shape]
|
||||
(and (associative? shape)
|
||||
(some? (:component-id shape))
|
||||
(or (nil? (:component-file shape))
|
||||
(= file-id (:component-file shape)))))
|
||||
|
||||
(defn- find-library-refs
|
||||
"Return `{lib-id #{component-id}}` referenced by the given shapes."
|
||||
"Return `{lib-id #{component-id}}` referenced by the given shapes.
|
||||
Counterpart of `same-file-refs`, kept for the library side: this one
|
||||
requires a non-nil `:component-file`; same-file references are none."
|
||||
[shapes]
|
||||
(reduce (fn [acc shape]
|
||||
(if (and (map? shape)
|
||||
@ -67,10 +79,12 @@
|
||||
(cfh/get-children-with-self objects main-instance-id)))))
|
||||
|
||||
(defn- collect-used-library-components
|
||||
"Walk the objects of the (already page-scoped) primary file data and of
|
||||
the referenced library components themselves, and return a map of
|
||||
`{lib-id #{component-id}}` with every library component the allowed
|
||||
pages need. Follows nested references to a fixpoint so trimming never
|
||||
"Walk the objects of the primary file data the view-only bundle emits,
|
||||
and of the referenced library components themselves, and return a map
|
||||
of `{lib-id #{component-id}}` with every library component the emitted
|
||||
data needs. The caller passes that data: pages scoped to the link and
|
||||
the already trimmed components map, so deleted components are never
|
||||
walked. Follows nested references to a fixpoint so trimming never
|
||||
drops a component that is only reached through another component."
|
||||
[primary-data libs-by-id]
|
||||
(let [root-shapes (concat (mapcat (comp vals :objects) (vals (:pages-index primary-data)))
|
||||
@ -100,6 +114,48 @@
|
||||
(assoc-in [:data :pages-index] {})
|
||||
(update-in [:data :components] select-keys (get used-refs id #{}))))
|
||||
|
||||
(defn- same-file-refs
|
||||
"Return the ids of this file's components referenced by `shapes`.
|
||||
Counterpart of `find-library-refs`, which handles the library side.
|
||||
A reference missing `:component-file` means a same-file reference;
|
||||
library references are handled by `trim-library-data` and skipped."
|
||||
[file-id shapes]
|
||||
(into #{}
|
||||
(keep (fn [shape]
|
||||
(when (same-file-component-ref? file-id shape)
|
||||
(:component-id shape))))
|
||||
shapes))
|
||||
|
||||
(defn- referenced-component-ids
|
||||
"Return the ids of the primary file's live components a share-link
|
||||
viewer of `allowed` pages must receive: every live component referenced
|
||||
transitively from those pages. A referenced component that is deleted
|
||||
is neither emitted nor expanded; the bundle never carries stored
|
||||
shapes of deleted components. References are traced through the full
|
||||
realized data — a component's subtree may live on an excluded page —
|
||||
but only the ids are returned; the emitted bundle carries the trimmed
|
||||
map inline. Terminates on cycles."
|
||||
[realized-file allowed]
|
||||
(let [file-id (:id realized-file)
|
||||
seed-shapes (->> (vals (select-keys (:pages-index (:data realized-file)) allowed))
|
||||
(mapcat (comp vals :objects)))]
|
||||
(loop [seen #{}
|
||||
pending (vec (same-file-refs file-id seed-shapes))]
|
||||
(if (empty? pending)
|
||||
seen
|
||||
(let [comp-id (peek pending)
|
||||
pending' (pop pending)]
|
||||
(if (contains? seen comp-id)
|
||||
(recur seen pending')
|
||||
(let [component (get-in realized-file [:data :components comp-id])
|
||||
live? (and (some? component) (not (:deleted component)))
|
||||
extra (when live?
|
||||
(same-file-refs file-id
|
||||
(ctf/get-component-shapes (:data realized-file)
|
||||
component)))]
|
||||
(recur (if live? (conj seen comp-id) seen)
|
||||
(into pending' extra)))))))))
|
||||
|
||||
(defn obfuscate-email
|
||||
"Obfuscate the `email` for share-link members so the viewer only sees a
|
||||
partially redacted address. Accepts any string shape (including nil,
|
||||
@ -153,9 +209,23 @@
|
||||
(cfeat/check-client-features! (:features params))
|
||||
(cfeat/check-file-features! (:features file)))
|
||||
|
||||
;; The share-link bundle carries the primary file's components
|
||||
;; too: reuse the realization the library trim already needs and
|
||||
;; keep the emitted map inline. The walk may cross onto excluded
|
||||
;; pages (a retained component's subtree can live there), so the
|
||||
;; whole file is realized on each share-link bundle request.
|
||||
;; Pointer-map is operator-enabled and not active in production;
|
||||
;; revisit if it ships.
|
||||
realized-data
|
||||
(when (= :share-link (:type perms))
|
||||
(fdata/realize-pointers cfg file))
|
||||
|
||||
file (cond-> file
|
||||
(= :share-link (:type perms))
|
||||
(update :data remove-not-allowed-pages (:pages perms))
|
||||
(-> (update :data remove-not-allowed-pages (:pages perms))
|
||||
(assoc-in [:data :components]
|
||||
(select-keys (:components (:data realized-data))
|
||||
(referenced-component-ids realized-data (:pages perms)))))
|
||||
|
||||
:always
|
||||
(update :data select-keys view-only-data-keys))
|
||||
@ -166,15 +236,20 @@
|
||||
(bfc/file->response)))))
|
||||
|
||||
;; A share link never grants pages of a linked library, so trim
|
||||
;; each library to the components the allowed pages reference.
|
||||
;; File data may hold lazy pointer-mapped pages/components, so
|
||||
;; realize each file with its own loader before walking it.
|
||||
;; Membership bundles keep the full libraries.
|
||||
;; each library to the components the emitted bundle references:
|
||||
;; pages are scoped to the link and the components map already
|
||||
;; dropped deleted components, so neither of them seeds library
|
||||
;; references. The pages must come from the realized file, not
|
||||
;; from `file`: lazy pointer-mapped pages only load in their own
|
||||
;; loader. Membership bundles keep the full libraries.
|
||||
libs (if (= :share-link (:type perms))
|
||||
(let [file' (fdata/realize-pointers cfg file)
|
||||
libs' (mapv #(fdata/realize-pointers cfg %) libs)
|
||||
libs-by-id (d/index-by :id libs')
|
||||
used (collect-used-library-components (:data file') libs-by-id)]
|
||||
(let [allowed-data (-> realized-data
|
||||
(update :data remove-not-allowed-pages (:pages perms))
|
||||
(assoc-in [:data :components]
|
||||
(get-in file [:data :components])))
|
||||
libs' (mapv #(fdata/realize-pointers cfg %) libs)
|
||||
libs-by-id (d/index-by :id libs')
|
||||
used (collect-used-library-components (:data allowed-data) libs-by-id)]
|
||||
(mapv #(trim-library-data used %) libs'))
|
||||
libs)
|
||||
|
||||
|
||||
@ -767,12 +767,16 @@
|
||||
"Mark a project for deletion"
|
||||
[file-id]
|
||||
(let [file-id (h/parse-uuid file-id)
|
||||
tnow (ct/now)]
|
||||
tnow (ct/now)
|
||||
;; The event carries its team, so a consumer can key on it
|
||||
;; without joining production tables.
|
||||
team-id (first (audit/get-team-ids sys/system {:file-ids [file-id]}))]
|
||||
|
||||
(audit/insert sys/system
|
||||
{:name "delete-file"
|
||||
:type "action"
|
||||
:props {:id file-id}
|
||||
:props (cond-> {:id file-id}
|
||||
(some? team-id) (assoc :team-id team-id))
|
||||
:context {:triggered-by "srepl"
|
||||
:cause "explicit call to delete-file!"}
|
||||
:tracked-at tnow})
|
||||
@ -792,13 +796,17 @@
|
||||
(when-let [file (db/get* system :file
|
||||
{:id file-id}
|
||||
{::db/remove-deleted false
|
||||
::sql/columns [:id :name]})]
|
||||
(audit/insert system
|
||||
{:name "restore-file"
|
||||
:type "action"
|
||||
:props file
|
||||
:context {:triggered-by "srepl"
|
||||
:cause "explicit call to restore-file!"}})
|
||||
::sql/columns [:id :name :project-id]})]
|
||||
(let [team-id (first (audit/get-team-ids
|
||||
system
|
||||
{:project-ids [(:project-id file)]}))]
|
||||
(audit/insert system
|
||||
{:name "restore-file"
|
||||
:type "action"
|
||||
:props (cond-> file
|
||||
(some? team-id) (assoc :team-id team-id))
|
||||
:context {:triggered-by "srepl"
|
||||
:cause "explicit call to restore-file!"}}))
|
||||
|
||||
(#'files/restore-files conn [file-id]))
|
||||
:restored))))
|
||||
@ -920,7 +928,7 @@
|
||||
(audit/insert system
|
||||
{:name "restore-team"
|
||||
:type "action"
|
||||
:props team
|
||||
:props (assoc team :team-id (:id team))
|
||||
:context {:triggered-by "srepl"
|
||||
:cause "explicit call to restore-team!"}})
|
||||
|
||||
@ -948,7 +956,10 @@
|
||||
:deleted))
|
||||
|
||||
(defn restore-profile!
|
||||
"Mark a team and all related objects as not deleted"
|
||||
"Mark a profile and the teams its deletion cascade soft-deleted as not
|
||||
deleted. Answers `:already-active` without touching anything when the
|
||||
profile is not deleted: running the cascade over live teams would revive
|
||||
font variants and files deleted on their own."
|
||||
[profile-id]
|
||||
(let [profile-id (h/parse-uuid profile-id)]
|
||||
(db/tx-run! sys/system
|
||||
@ -957,22 +968,34 @@
|
||||
{:id profile-id}
|
||||
{::db/remove-deleted false})
|
||||
(profile/decode-row))]
|
||||
(audit/insert system
|
||||
{:name "restore-profile"
|
||||
:type "action"
|
||||
:props (audit/profile->props profile)
|
||||
:context {:triggered-by "srepl"
|
||||
:cause "explicit call to restore-profile!"}})
|
||||
(if (nil? (:deleted-at profile))
|
||||
:already-active
|
||||
(do
|
||||
(db/update! system :profile
|
||||
{:deleted-at nil}
|
||||
{:id profile-id}
|
||||
{::db/return-keys false})
|
||||
|
||||
(db/update! system :profile
|
||||
{:deleted-at nil}
|
||||
{:id profile-id}
|
||||
{::db/return-keys false})
|
||||
;; Owned teams are still soft-deleted here, so resolve
|
||||
;; them through the membership table, and only the ones
|
||||
;; the cascade stamped with the profile's own instant: a
|
||||
;; team the owner deleted earlier stays deleted.
|
||||
(let [team-ids (mapv :id (profile/get-owned-teams system (:id profile)
|
||||
:deleted-at (:deleted-at profile)))]
|
||||
(doseq [id team-ids]
|
||||
(restore-team* system id))
|
||||
|
||||
(doseq [{:keys [id]} (profile/get-owned-teams system profile-id)]
|
||||
(restore-team* system id))
|
||||
;; The event carries the restored teams explicitly,
|
||||
;; so a consumer never joins production tables.
|
||||
(audit/insert system
|
||||
{:name "restore-profile"
|
||||
:type "action"
|
||||
:props (assoc (audit/profile->props profile)
|
||||
:team-ids team-ids)
|
||||
:context {:triggered-by "srepl"
|
||||
:cause "explicit call to restore-profile!"}}))
|
||||
|
||||
:restored)))))
|
||||
:restored)))))))
|
||||
|
||||
(defn delete-profiles-in-bulk!
|
||||
[system path]
|
||||
@ -1007,13 +1030,15 @@
|
||||
the whole table, and enqueues the job instead of running it inline, so it
|
||||
can be used against production in bounded batches. The job is idempotent (it
|
||||
only marks the children with the row's own date), `label` is the row id and
|
||||
`dedupe` drops not-yet-due duplicates.
|
||||
`dedupe` skips the row when its job is still not started.
|
||||
|
||||
Options: `:limit` (max rows read), `:ids` (restrict the run to these row ids)
|
||||
and `:dry-run?` (count without enqueueing). `table` and `object` are internal
|
||||
constants, never operator input.
|
||||
|
||||
Returns `{:found n :submitted m}`.
|
||||
Returns `{:found n :submitted m}`. `m` counts the rows whose job was
|
||||
really enqueued: a row whose `dedupe` key still holds a not-started job
|
||||
is counted in `n` but not in `m`.
|
||||
|
||||
Note: the submissions are given the outer `system` (with the pool) on
|
||||
purpose; handing them the read-only transaction cfg would write on it."
|
||||
@ -1028,20 +1053,22 @@
|
||||
(fn [{:keys [::db/conn]}]
|
||||
(let [query (cond-> [sql]
|
||||
(seq ids) (conj (db/create-array conn "uuid" ids)))
|
||||
found (reduce (fn [total {:keys [id deleted-at]}]
|
||||
(when-not dry-run?
|
||||
(jobs/submit system
|
||||
{::jobs/name :delete-object
|
||||
::jobs/params {:object object
|
||||
:deleted-at deleted-at
|
||||
:id id}
|
||||
::jobs/label (str id)
|
||||
::jobs/dedupe true}))
|
||||
(inc total))
|
||||
0
|
||||
(db/plan conn query))]
|
||||
[found submitted]
|
||||
(reduce (fn [[found submitted] {:keys [id deleted-at]}]
|
||||
(if (and (not dry-run?)
|
||||
(jobs/submit system
|
||||
{::jobs/name :delete-object
|
||||
::jobs/params {:object object
|
||||
:deleted-at deleted-at
|
||||
:id id}
|
||||
::jobs/label (str id)
|
||||
::jobs/dedupe true}))
|
||||
[(inc found) (inc submitted)]
|
||||
[(inc found) submitted]))
|
||||
[0 0]
|
||||
(db/plan conn query))]
|
||||
{:found found
|
||||
:submitted (if dry-run? 0 found)})))))
|
||||
:submitted submitted})))))
|
||||
|
||||
(defn process-deleted-profiles-cascade
|
||||
"Replay the deletion cascade of every soft-deleted profile."
|
||||
|
||||
@ -7,6 +7,7 @@
|
||||
(ns app.tasks.delete-object
|
||||
"A generic task for object deletion cascade handling"
|
||||
(:require
|
||||
[app.common.data :as d]
|
||||
[app.common.logging :as l]
|
||||
[app.common.schema :as sm]
|
||||
[app.common.time :as ct]
|
||||
@ -21,6 +22,14 @@
|
||||
|
||||
(def ^:dynamic *team-deletion* false)
|
||||
|
||||
(def ^:dynamic *deleting-file-ids*
|
||||
"Ids of the files the running cascade is going to delete. Bound by
|
||||
the `:project` method with its own files so `:file` only absorbs the
|
||||
dependents that survive: absorbing into files that die in the same
|
||||
run is wasted work. Team deletions need no such set: they bind
|
||||
`*team-deletion*` instead, which skips absorb entirely."
|
||||
nil)
|
||||
|
||||
(def ^:dynamic *cascade*
|
||||
"Accumulates the ids marked deleted by the current cascade execution,
|
||||
grouped by entity. Bound by the task entry point; nil on direct calls
|
||||
@ -79,14 +88,45 @@
|
||||
{::db/return-keys false})
|
||||
|
||||
(when (and (:is-shared file)
|
||||
;; Team deletions skip absorb entirely: every file dies
|
||||
;; in the same cascade, so no survivor can absorb.
|
||||
(not *team-deletion*))
|
||||
;; NOTE: we don't prevent file deletion on absorb operation failure
|
||||
(try
|
||||
(db/tx-run! cfg files/absorb-library id :file-ids file-ids)
|
||||
(catch Throwable cause
|
||||
(l/warn :hint "error on absorbing library"
|
||||
:file-id id
|
||||
:cause cause))))
|
||||
;; NOTE: we don't prevent file deletion on absorb operation failure.
|
||||
;; Only live dependents outside this cascade matter: if the whole
|
||||
;; closure dies together there is nothing to absorb and the
|
||||
;; relations die with their files (both FKs are ON DELETE CASCADE).
|
||||
;; DEPRECATED `:file-ids`: jobs queued by the previous release
|
||||
;; already deleted the relations in the RPC, so the live lookup
|
||||
;; finds nothing. Trust their snapshot for one release; new jobs
|
||||
;; never carry the key.
|
||||
(let [doomed (or *deleting-file-ids* #{})
|
||||
dependents (if (seq file-ids)
|
||||
(into [] (remove doomed) file-ids)
|
||||
(into [] (remove doomed)
|
||||
(files/get-referenced-files conn id)))]
|
||||
(when (seq dependents)
|
||||
;; The relations are dropped unconditionally, before the absorb
|
||||
;; runs: they are dead weight once the library is being
|
||||
;; deleted, regardless of whether the absorb succeeds. A
|
||||
;; failure here fails the job instead of being swallowed,
|
||||
;; like every other write in the cascade.
|
||||
(db/delete! conn :file-library-rel {:library-file-id id})
|
||||
;; Runs in the caller transaction: a nested transaction here
|
||||
;; would join it anyway, so the call goes directly. The
|
||||
;; savepoint contains an absorb failure: it rolls back the
|
||||
;; whole absorb and the cascade goes on.
|
||||
;; Nothing retries the absorb itself: the job succeeds and the
|
||||
;; dependents keep the data they had before the absorb ran.
|
||||
(let [sp (db/savepoint conn "absorb-library")]
|
||||
(try
|
||||
(files/absorb-library cfg id :file-ids dependents)
|
||||
(catch Throwable cause
|
||||
(db/rollback! conn sp)
|
||||
(l/warn :hint "error on absorbing library"
|
||||
:file-id id
|
||||
:cause cause))
|
||||
(finally
|
||||
(db/release! conn sp)))))))
|
||||
|
||||
;; Mark file change to be deleted, but only the rows that do not carry a
|
||||
;; date yet. A revision is born already scheduled for deletion one hour
|
||||
@ -134,12 +174,16 @@
|
||||
{:id id}
|
||||
{::db/return-keys false})
|
||||
|
||||
(doseq [file (db/query conn :file
|
||||
{:project-id id}
|
||||
{::db/columns [:id :deleted-at]})]
|
||||
(delete-object cfg (assoc file
|
||||
:object :file
|
||||
:deleted-at deleted-at)))
|
||||
(let [files (db/query conn :file
|
||||
{:project-id id}
|
||||
{::db/columns [:id :deleted-at]})]
|
||||
;; Every file below dies in this run: absorbing a library into
|
||||
;; them is wasted work, so the files only absorb survivors.
|
||||
(binding [*deleting-file-ids* (into #{} d/xf:map-id files)]
|
||||
(doseq [file files]
|
||||
(delete-object cfg (assoc file
|
||||
:object :file
|
||||
:deleted-at deleted-at)))))
|
||||
|
||||
(record :projects id))
|
||||
|
||||
@ -193,29 +237,38 @@
|
||||
[:deleted-at ::ct/inst]
|
||||
[:id ::sm/uuid]
|
||||
[:file-id {:optional true} ::sm/uuid]
|
||||
;; DEPRECATED: only jobs queued before the cascade owned the
|
||||
;; relations carry it (their RPC already deleted them). Remove once
|
||||
;; no such rows can be left.
|
||||
[:file-ids {:optional true} [:vector ::sm/uuid]]])
|
||||
|
||||
(defn execute-delete-object
|
||||
"Plain job handler: run the delete-object multimethod on the provided
|
||||
params inside a single transaction, tracking the cascade and reporting
|
||||
it as an audit event when anything was marked."
|
||||
params, tracking the cascade and reporting it as an audit event when
|
||||
anything was marked. Runs inside the caller transaction (the job-def
|
||||
handler opens it with `tx-run!`)."
|
||||
[cfg params]
|
||||
(db/tx-run! cfg
|
||||
(fn [cfg]
|
||||
(when (top-exists? cfg params)
|
||||
(binding [*cascade* (atom {:teams [] :projects [] :files []})]
|
||||
(delete-object cfg params)
|
||||
(let [{:keys [teams projects files]} @*cascade*]
|
||||
(when (or (seq teams) (seq projects) (seq files))
|
||||
(audit/submit cfg {:name "delete-object-with-cascade"
|
||||
:type "action"
|
||||
:props {:object (:object params)
|
||||
:id (:id params)
|
||||
:deleted-at (:deleted-at params)
|
||||
:cascade {:team-ids teams
|
||||
:project-ids projects
|
||||
:file-ids files}}
|
||||
:context {:triggered-by "delete-object"}}))))))))
|
||||
(when (top-exists? cfg params)
|
||||
(binding [*cascade* (atom {:teams [] :projects [] :files []})]
|
||||
(delete-object cfg params)
|
||||
(let [{:keys [teams projects files]} @*cascade*]
|
||||
(when (or (seq teams) (seq projects) (seq files))
|
||||
;; Flat team ids, so a consumer can key on them without
|
||||
;; joining production tables.
|
||||
(let [team-ids (audit/get-team-ids cfg
|
||||
{:team-ids teams
|
||||
:project-ids projects
|
||||
:file-ids files})]
|
||||
(audit/submit cfg {:name "delete-object-with-cascade"
|
||||
:type "action"
|
||||
:props {:object (:object params)
|
||||
:id (:id params)
|
||||
:deleted-at (:deleted-at params)
|
||||
:team-ids (vec team-ids)
|
||||
:cascade {:team-ids teams
|
||||
:project-ids projects
|
||||
:file-ids files}}
|
||||
:context {:triggered-by "delete-object"}})))))))
|
||||
|
||||
(defmethod ig/assert-key ::job-def
|
||||
[_ params]
|
||||
@ -228,6 +281,6 @@
|
||||
::jobs/schema schema:delete-object-params
|
||||
::jobs/handler
|
||||
(fn [_context params]
|
||||
(execute-delete-object cfg params))
|
||||
(db/tx-run! cfg execute-delete-object params))
|
||||
::jobs/decoder (sm/decoder schema:delete-object-params sm/json-transformer)
|
||||
::jobs/validator (sm/validator schema:delete-object-params)})
|
||||
|
||||
@ -26,6 +26,7 @@
|
||||
[app.features.fdata :as feat.fdata]
|
||||
[app.features.file-snapshots :as fsnap]
|
||||
[app.jobs :as jobs]
|
||||
[app.loggers.audit :as audit]
|
||||
[app.storage :as sto]
|
||||
[integrant.core :as ig]))
|
||||
|
||||
@ -72,10 +73,21 @@
|
||||
(into #{} (map :id)))]
|
||||
|
||||
(jobs/heartbeat cfg)
|
||||
(doseq [id unused-media]
|
||||
(doseq [media-id unused-media]
|
||||
(l/trc :obj "media-object"
|
||||
:file-id (str id)
|
||||
:id (str id)))
|
||||
:id (str media-id)))
|
||||
|
||||
;; A trimmed file is a destructive action that no RPC reports: emit
|
||||
;; it so the audit trail records it. The team id travels in props so
|
||||
;; a consumer never joins production tables.
|
||||
(when (and (seq unused-media) cf/audit-persist?)
|
||||
(let [team-id (first (audit/get-team-ids cfg {:file-ids [id]}))]
|
||||
(audit/submit cfg {:name "file-gc"
|
||||
:type "action"
|
||||
:props (cond-> {:file-id id}
|
||||
(some? team-id) (assoc :team-id team-id))
|
||||
:context {:triggered-by "cron-task"}})))
|
||||
|
||||
file))
|
||||
|
||||
@ -260,9 +272,14 @@
|
||||
(feat.fdata/with-pointers cfg file-id
|
||||
(let [liveness (collect-snapshot-liveness cfg file-id)
|
||||
original (feat.fdata/original-pointers (:data file))
|
||||
file (bfl/clean-file file)
|
||||
;; With retention on, kept revision rows can name media and
|
||||
;; components that neither the head nor the snapshots name,
|
||||
;; and the media cleaners do not read those rows.
|
||||
file (if (contains? cf/flags :file-change-retention)
|
||||
file
|
||||
(clean-media! cfg liveness file))
|
||||
file (->> file
|
||||
(bfl/clean-file)
|
||||
(clean-media! cfg liveness)
|
||||
(feat.fdata/reattach-unchanged original)
|
||||
(clean-fragments! cfg (:pointers liveness)))
|
||||
file (assoc file :has-media-trimmed true)]
|
||||
|
||||
@ -44,7 +44,7 @@
|
||||
their transactions. Runs inside the caller transaction, next to the
|
||||
deletes: if the purge aborts, the event aborts with it."
|
||||
[cfg ids-key ids]
|
||||
(when (contains? cf/flags :audit-log)
|
||||
(when cf/audit-persist?
|
||||
(audit/submit cfg
|
||||
{:name "hard-delete-objects"
|
||||
:type "command"
|
||||
|
||||
@ -154,6 +154,13 @@
|
||||
ORDER BY 2 DESC")
|
||||
|
||||
(defn- get-action-counters
|
||||
"The event counters of the legacy report: an aggregate of the telemetry
|
||||
shadows that are still in the buffer when the report runs. The number
|
||||
already depended on the flags (the whole day when no full-row consumer
|
||||
persisted, only since the previous run otherwise); now the consumer ships
|
||||
the shadows and the GC deletes them, so it reflects what the buffer still
|
||||
holds. The per-event data reaches Nexus as `telemetry-events` from the
|
||||
consumer; this report does not carry it."
|
||||
[conn]
|
||||
(let [counters (->> (db/exec! conn [sql:get-counters])
|
||||
(d/index-by (comp keyword :name) :count))
|
||||
@ -210,114 +217,6 @@
|
||||
|
||||
(make-legacy-request cfg data)))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; AUDIT-EVENT BATCH (TELEMETRY MODE)
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
;; Telemetry events older than this are purged by the GC step so the
|
||||
;; buffer stays bounded.
|
||||
(def ^:private batch-size 10000)
|
||||
|
||||
(def ^:private sql:aged-telemetry
|
||||
"source LIKE 'telemetry:%' AND created_at < now() - interval '7 days'")
|
||||
|
||||
(def ^:private sql:gc-pendings
|
||||
(str "DELETE FROM audit_log_pending WHERE audit_log_id IN"
|
||||
" (SELECT id FROM audit_log WHERE " sql:aged-telemetry ")"))
|
||||
|
||||
(def ^:private sql:gc-events
|
||||
(str "DELETE FROM audit_log WHERE " sql:aged-telemetry))
|
||||
|
||||
(defn- gc-events
|
||||
"Delete telemetry-mode events older than `telemetry-retention-days`
|
||||
so that the buffer stays bounded."
|
||||
[{:keys [::db/conn]}]
|
||||
;; One transaction for both deletes: a crash between them would
|
||||
;; leave pending-less rows for the consumer sweep to delete unsent.
|
||||
;; Pendings first: the queue holds a row per consumer for every
|
||||
;; record, and the FK would reject deleting a row that still has
|
||||
;; pendings.
|
||||
(db/tx-run! conn
|
||||
(fn [{:keys [::db/conn]}]
|
||||
(db/exec-one! conn [sql:gc-pendings])
|
||||
(let [result (db/exec-one! conn [sql:gc-events])]
|
||||
(when (pos? (:next.jdbc/update-count result))
|
||||
(l/warn :hint "purged stale telemetry events"
|
||||
:count (:next.jdbc/update-count result)))))))
|
||||
|
||||
(def ^:private sql:fetch-telemetry-events
|
||||
"SELECT id, name, type, source, tracked_at, profile_id, props, context
|
||||
FROM audit_log
|
||||
WHERE source LIKE 'telemetry:%'
|
||||
ORDER BY created_at ASC
|
||||
LIMIT ?")
|
||||
|
||||
(defn- row->event
|
||||
[{:keys [name type source tracked-at profile-id props context]}]
|
||||
(d/without-nils
|
||||
{:name name
|
||||
:type type
|
||||
:source source
|
||||
:tracked-at tracked-at
|
||||
:profile-id profile-id
|
||||
:props (or (some-> props db/decode-transit-pgobject) {})
|
||||
:context (or (some-> context db/decode-transit-pgobject) {})}))
|
||||
|
||||
(defn send-event-batch
|
||||
"Send a single batch of events to the telemetry endpoint. Returns
|
||||
true on success. The events are sent as a plain vector of event
|
||||
maps; the JSON encoder handles UUID and temporal types natively and
|
||||
the receiver coerces them back to proper types."
|
||||
[{:keys [::setup/props] :as cfg} batch]
|
||||
(let [payload {:type :telemetry-events
|
||||
:version (:full cf/version)
|
||||
:instance-id (:instance-id props)
|
||||
:events (vec batch)}
|
||||
request {:method :post
|
||||
:uri (cf/get :telemetry-uri)
|
||||
:headers {"content-type" "application/json"}
|
||||
:body (json/encode-str payload)}
|
||||
resp (http/req cfg request {:skip-ssrf-check? true})]
|
||||
(if (<= (:status resp) 206)
|
||||
true
|
||||
(do
|
||||
(l/warn :hint "telemetry event batch send failed"
|
||||
:status (:status resp)
|
||||
:body (:body resp))
|
||||
false))))
|
||||
|
||||
(defn- delete-sent-events
|
||||
"Delete rows by their ids after a successful send. One transaction
|
||||
for both deletes: a crash between them would leave pending-less
|
||||
rows for the consumer sweep to delete unsent."
|
||||
[conn ids]
|
||||
(db/tx-run! conn
|
||||
(fn [tx]
|
||||
(let [tx-conn (::db/conn tx)
|
||||
arr (db/create-array tx-conn "uuid" ids)]
|
||||
;; Pendings first: same FK reason as in `gc-events`.
|
||||
(db/exec-one! tx-conn ["DELETE FROM audit_log_pending WHERE audit_log_id = ANY(?)" arr])
|
||||
(db/exec-one! tx-conn ["DELETE FROM audit_log WHERE id = ANY(?)" arr])))))
|
||||
|
||||
(defn- collect-and-send-audit-events
|
||||
"Collect anonymous telemetry-mode audit events and ship them to the
|
||||
telemetry endpoint in a loop. Each iteration fetches one page of
|
||||
`batch-size` rows, encodes and sends them, then deletes the rows on
|
||||
success. The loop stops as soon as a send returns false, leaving
|
||||
remaining rows intact for the next run."
|
||||
[{:keys [::db/conn] :as cfg}]
|
||||
(loop [counter 1]
|
||||
(when-let [rows (-> (db/exec! conn [sql:fetch-telemetry-events batch-size])
|
||||
(not-empty))]
|
||||
(let [events (mapv row->event rows)
|
||||
ids (mapv :id rows)]
|
||||
(l/dbg :hint "shipping telemetry event batch"
|
||||
:total (count events)
|
||||
:batch counter)
|
||||
(when (send-event-batch cfg events)
|
||||
(delete-sent-events conn ids)
|
||||
(recur (inc counter)))))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; TASK ENTRY POINT
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
@ -350,7 +249,7 @@
|
||||
[cfg params]
|
||||
(let [send? (get params :send? true)
|
||||
enabled? (or (get params :enabled? false)
|
||||
(contains? cf/flags :telemetry))
|
||||
cf/telemetry-enabled?)
|
||||
;; Deferred so the query runs only when a report is
|
||||
;; actually going to be sent.
|
||||
subs (delay (get-subscriptions cfg))]
|
||||
@ -362,7 +261,6 @@
|
||||
|
||||
(if enabled?
|
||||
(when send?
|
||||
(db/run! cfg gc-events)
|
||||
;; Randomize start time to avoid thundering herd when multiple
|
||||
;; instances restart at the same time.
|
||||
(px/sleep (rand-int 10000))
|
||||
@ -372,16 +270,7 @@
|
||||
(send-legacy-data cfg stats @subs))
|
||||
(catch Exception cause
|
||||
(l/wrn :hint "unable to send legacy report"
|
||||
:cause cause)))
|
||||
|
||||
;; Ship any anonymous audit-log events accumulated in
|
||||
;; telemetry mode (only when audit-log feature is off).
|
||||
(when-not (contains? cf/flags :audit-log)
|
||||
(try
|
||||
(db/run! cfg collect-and-send-audit-events)
|
||||
(catch Exception cause
|
||||
(l/wrn :hint "unable to send events"
|
||||
:cause cause)))))
|
||||
:cause cause))))
|
||||
|
||||
;; If we have telemetry disabled, but there are users that are
|
||||
;; explicitly checked the newsletter subscription on the
|
||||
|
||||
@ -91,13 +91,12 @@
|
||||
{::db/return-keys false})
|
||||
|
||||
;; The no-overlap check lives in the submit itself
|
||||
;; (`::jobs/skip-if-active`): the check and the
|
||||
;; insert share this transaction, so a job could
|
||||
;; still transition from running to completed
|
||||
;; between the check and the insert. This is
|
||||
;; acceptable: the next tick will submit if
|
||||
;; needed. The FOR UPDATE SKIP LOCKED on the
|
||||
;; scheduled_task row prevents race conditions
|
||||
;; (`::jobs/skip-if-active`): the partial unique
|
||||
;; index decides on the insert, so there is no
|
||||
;; check racing it. The only residual race is a
|
||||
;; live job going terminal right before the insert,
|
||||
;; which the next tick covers. The FOR UPDATE SKIP
|
||||
;; LOCKED on the scheduled_task row prevents races
|
||||
;; between nodes.
|
||||
(if-let [job-id (submit-cron-job cfg entry)]
|
||||
(let [elapsed (ct/format-duration (tpoint))]
|
||||
|
||||
@ -59,23 +59,61 @@
|
||||
"UPDATE job SET status = 'scheduled', modified_at = ?
|
||||
WHERE id = ANY(?)")
|
||||
|
||||
;; Both sweeps are bulk UPDATEs that run before every dispatch batch, so
|
||||
;; they are the two queries where a missing tenant filter does the most
|
||||
;; damage: without it any instance with a worker requeues another
|
||||
;; instance's scheduled rows and, worse, aborts its running jobs once
|
||||
;; their lease expires.
|
||||
(def ^:private sql:reschedule-lost
|
||||
;; The sweeps are bulk UPDATEs that run before every dispatch batch, so
|
||||
;; they are the queries where a missing tenant filter does the most
|
||||
;; damage: without it any instance with a worker aborts another
|
||||
;; instance's jobs.
|
||||
(def ^:private lost-batch-size
|
||||
"Rows per lost-sweep read. The dispatcher loops without pausing while it
|
||||
finds work, so an unbounded read would turn a big `scheduled` backlog
|
||||
into a repeated scan. Aborting a row takes it out of the candidate set,
|
||||
so the next pass continues where this one stopped."
|
||||
1000)
|
||||
|
||||
(def ^:private sql:lost-past-cap
|
||||
"The `scheduled` rows of one tenant that have waited past the cap, oldest
|
||||
first. They are lost behind a backlog or in a queue with no consumer
|
||||
whatever the queue holds, so the sweep aborts them without an `LLEN`.
|
||||
`modified_at` is the hand-off time (the push), so it measures time in the
|
||||
queue, not the due time."
|
||||
"SELECT id, queue, modified_at
|
||||
FROM job
|
||||
WHERE status = 'scheduled'
|
||||
AND tenant = ?
|
||||
AND modified_at < ?::timestamptz
|
||||
ORDER BY modified_at ASC
|
||||
LIMIT ?")
|
||||
|
||||
(def ^:private sql:lost-in-grace
|
||||
"The `scheduled` rows of one tenant inside the grace window, oldest first.
|
||||
These are the only candidates that need the per-queue `LLEN`: they are
|
||||
lost when nothing is waiting for them."
|
||||
"SELECT id, queue, modified_at
|
||||
FROM job
|
||||
WHERE status = 'scheduled'
|
||||
AND tenant = ?
|
||||
AND modified_at >= ?::timestamptz
|
||||
AND modified_at < ?::timestamptz
|
||||
ORDER BY modified_at ASC
|
||||
LIMIT ?")
|
||||
|
||||
(def ^:private sql:abort-lost
|
||||
"Abort the lost rows by id. The status, the tenant and the hand-off time
|
||||
are repeated in the outer WHERE so a claim, a heartbeat or a competing
|
||||
dispatcher that lands between the candidate read and this UPDATE affects
|
||||
no row."
|
||||
"UPDATE job
|
||||
SET status='new', scheduled_at=?::timestamptz
|
||||
FROM (SELECT t.id
|
||||
FROM job AS t
|
||||
WHERE status = 'scheduled'
|
||||
AND t.tenant = ?
|
||||
AND t.scheduled_at < ?::timestamptz - '5 min'::interval) AS subquery
|
||||
WHERE job.id=subquery.id
|
||||
RETURNING job.id, job.queue")
|
||||
SET status='aborted', modified_at=?::timestamptz, error=?::jsonb
|
||||
WHERE id = ANY(?)
|
||||
AND status = 'scheduled'
|
||||
AND tenant = ?
|
||||
AND modified_at < ?::timestamptz
|
||||
RETURNING job.id, job.queue, job.name, job.created_at")
|
||||
|
||||
(def ^:private sql:mark-orphan
|
||||
"Abort the running rows past the lease. The predicate is repeated in the
|
||||
outer WHERE so a heartbeat that renews the row between the subquery and
|
||||
the update affects no row."
|
||||
"UPDATE job
|
||||
SET status='aborted', modified_at=?::timestamptz, error=?::jsonb
|
||||
FROM (SELECT t.id
|
||||
@ -84,18 +122,38 @@ RETURNING job.id, job.queue")
|
||||
AND t.tenant = ?
|
||||
AND t.modified_at < ?::timestamptz) AS subquery
|
||||
WHERE job.id=subquery.id
|
||||
AND job.status = 'running'
|
||||
AND job.modified_at < ?::timestamptz
|
||||
RETURNING job.id, job.queue, job.name, job.created_at")
|
||||
|
||||
(def ^:private lost-grace
|
||||
"How long a `scheduled` row may wait before its queue is inspected. It
|
||||
also covers the pop-to-claim gap of a message popped right after the
|
||||
hand-off. A message that waited longer in the queue and is popped while
|
||||
the sweep runs is not covered: the window is the pop-to-claim round trip
|
||||
and the abort can win it (see `abort-lost-jobs`)."
|
||||
(ct/duration "5m"))
|
||||
|
||||
(def ^:private lost-cap
|
||||
"Beyond this age a `scheduled` row is aborted even if its queue still has
|
||||
messages: the job is lost behind a backlog, or its queue has no consumer.
|
||||
A daily cron lost in a backed-up queue can miss one run, because its live
|
||||
row blocks the next submit until the cap."
|
||||
(ct/duration "24h"))
|
||||
|
||||
;; the error travels as text and PostgreSQL casts it: a PGobject parameter
|
||||
;; on a `::jsonb` placeholder is not what the driver expects here. Same
|
||||
;; for the orphan `end` event payload below.
|
||||
;; for the `end` event payload below.
|
||||
(def ^:private orphan-error-json
|
||||
(json/encode jobs/orphan-error))
|
||||
|
||||
(def ^:private lost-error-json
|
||||
(json/encode jobs/lost-error))
|
||||
|
||||
(def ^:private aborted-outcome-json
|
||||
(json/encode {:outcome "aborted"}))
|
||||
|
||||
(def ^:private sql:insert-orphan-events
|
||||
(def ^:private sql:insert-aborted-events
|
||||
"INSERT INTO job_event (job_id, kind, payload)
|
||||
SELECT unnest(?::uuid[]), 'end', ?::jsonb")
|
||||
|
||||
@ -103,42 +161,92 @@ RETURNING job.id, job.queue, job.name, job.created_at")
|
||||
[{:keys [id scheduled-at]}]
|
||||
(json/encode [(str id) (ct/format-inst scheduled-at)]))
|
||||
|
||||
(defn- reschedule-lost-jobs
|
||||
[{:keys [::db/conn ::timestamp ::wrk/tenant] :as cfg}]
|
||||
(let [rows (db/exec! conn [sql:reschedule-lost timestamp tenant timestamp]
|
||||
{:return-keys true})]
|
||||
(db/after-commit
|
||||
(fn []
|
||||
(doseq [{:keys [id queue]} rows]
|
||||
(metrics/record-rescheduled cfg queue)
|
||||
(l/wrn :hint "reschedule"
|
||||
:id (str id)
|
||||
:tenant tenant
|
||||
:queue queue))))))
|
||||
(defn- record-aborted
|
||||
"The durable side of one abort sweep: the `end` events in the same
|
||||
transaction (one bulk insert, never one per row) and the terminal metric
|
||||
after the commit. No msgbus notification: an aborted job alerts through
|
||||
the error log and `penpot_jobs_total_ms{outcome=\"aborted\"}`."
|
||||
[{:keys [::db/conn ::timestamp] :as cfg} rows]
|
||||
(when (seq rows)
|
||||
(db/exec-one! conn [sql:insert-aborted-events
|
||||
(db/create-array conn "uuid" (map :id rows))
|
||||
aborted-outcome-json]))
|
||||
(db/after-commit
|
||||
(fn []
|
||||
(doseq [{:keys [name created-at]} rows]
|
||||
(metrics/record-total cfg name :aborted
|
||||
(- (inst-ms timestamp) (inst-ms created-at)))))))
|
||||
|
||||
(defn- get-waiting
|
||||
"Messages waiting in each of the given queues, one LLEN per queue."
|
||||
[{:keys [::rds/conn ::wrk/tenant]} queues]
|
||||
(into {}
|
||||
(map (fn [queue]
|
||||
[queue (rds/llen conn (wrk/queue-key tenant queue))]))
|
||||
queues))
|
||||
|
||||
(defn- abort-lost-jobs
|
||||
"Abort the rows that were handed to a queue and never claimed. A row is
|
||||
lost when it has been `scheduled` past the cap (behind a backlog, or in a
|
||||
queue without a consumer), or when its queue has nothing waiting and it
|
||||
has been `scheduled` longer than the grace (a worker that popped it died
|
||||
before claiming). Lost rows are never re-dispatched: nothing returns to
|
||||
`new`, which keeps the dedupe index out of this path.
|
||||
|
||||
The two branches read a bounded page of candidates, oldest first: the
|
||||
dispatcher loops without pausing while it finds work, so an unbounded
|
||||
read would scan a big `scheduled` backlog on every iteration. Aborting
|
||||
removes a row from the next read. Only the grace rows pay the `LLEN`; a
|
||||
row past the cap is lost whatever its queue holds, so a younger lost row
|
||||
behind a large legitimate backlog may wait for the cap instead of the
|
||||
grace.
|
||||
|
||||
The grace is measured from the push, so a worker that pops a message
|
||||
older than the grace and stalls before claiming it can still lose that
|
||||
job: the sweep cannot tell a slow worker from a dead one."
|
||||
[cfg]
|
||||
(let [{:keys [::db/conn ::timestamp ::wrk/tenant]} cfg
|
||||
grace (ct/minus timestamp lost-grace)
|
||||
cap (ct/minus timestamp lost-cap)
|
||||
capped (db/exec! conn [sql:lost-past-cap tenant cap lost-batch-size])
|
||||
in-grace (db/exec! conn [sql:lost-in-grace tenant cap grace lost-batch-size])
|
||||
waiting (get-waiting cfg (into #{} (map :queue) in-grace))
|
||||
lost (into (vec capped)
|
||||
(filter (fn [{:keys [queue]}]
|
||||
(zero? (get waiting queue))))
|
||||
in-grace)]
|
||||
(when (seq lost)
|
||||
(let [aborted (db/exec! conn [sql:abort-lost
|
||||
timestamp
|
||||
lost-error-json
|
||||
(db/create-array conn "uuid" (map :id lost))
|
||||
tenant
|
||||
grace]
|
||||
{:return-keys true})]
|
||||
(record-aborted cfg aborted)
|
||||
(db/after-commit
|
||||
(fn []
|
||||
(doseq [{:keys [id queue name]} aborted]
|
||||
(l/wrn :hint "marked job as aborted (never claimed after being scheduled)"
|
||||
:name name
|
||||
:id (str id)
|
||||
:tenant tenant
|
||||
:queue queue))))))))
|
||||
|
||||
(defn- mark-orphan-jobs
|
||||
"Abort the running jobs whose process stopped reporting: their lease
|
||||
expired, so nothing alive can finish them."
|
||||
[{:keys [::db/conn ::timestamp ::lease ::wrk/tenant] :as cfg}]
|
||||
(let [cutoff (ct/minus timestamp (or lease (cf/get-jobs-lease)))
|
||||
rows (db/exec! conn [sql:mark-orphan timestamp orphan-error-json
|
||||
tenant cutoff]
|
||||
tenant cutoff cutoff]
|
||||
{:return-keys true})]
|
||||
;; The sweep is the only status change outside `app.jobs`, so it owns
|
||||
;; its `end` events too: one bulk insert in the same transaction (never
|
||||
;; one insert per orphan) keeps the history consistent with the rows.
|
||||
;; No msgbus notification: orphans alert through the error log and the
|
||||
;; orphaned counter, like before.
|
||||
(when (seq rows)
|
||||
(db/exec-one! conn [sql:insert-orphan-events
|
||||
(db/create-array conn "uuid" (map :id rows))
|
||||
aborted-outcome-json]))
|
||||
(record-aborted cfg rows)
|
||||
(db/after-commit
|
||||
(fn []
|
||||
(doseq [{:keys [id queue name created-at]} rows]
|
||||
(metrics/record-orphan cfg queue)
|
||||
(when (ct/inst? created-at)
|
||||
(metrics/record-total cfg name queue :aborted
|
||||
(- (inst-ms timestamp) (inst-ms created-at))))
|
||||
(doseq [{:keys [id queue name]} rows]
|
||||
(l/err :hint "marked job as aborted (orphan lease expired)"
|
||||
:name name
|
||||
:id (str id)
|
||||
:tenant tenant
|
||||
:queue queue))))))
|
||||
@ -159,8 +267,9 @@ RETURNING job.id, job.queue, job.name, job.created_at")
|
||||
(defn- push-jobs
|
||||
[{:keys [::rds/conn ::wrk/tenant] :as cfg} [queue jobs]]
|
||||
;; Mark first, push last: a crash between mark and push leaves a
|
||||
;; `scheduled` row that reschedule-lost-jobs re-queues, while the
|
||||
;; reverse order leaves a duplicate payload nothing deduplicates.
|
||||
;; `scheduled` row that the lost sweep aborts after its window, while the
|
||||
;; reverse order leaves a payload for a row the dispatcher will push
|
||||
;; again (two deliveries of the same job).
|
||||
(mark-as-scheduled cfg jobs)
|
||||
(let [items (mapv encode-payload jobs)
|
||||
key (wrk/queue-key tenant queue)]
|
||||
@ -168,7 +277,7 @@ RETURNING job.id, job.queue, job.name, job.created_at")
|
||||
(rds/rpush conn key items)
|
||||
(metrics/record-dispatcher-size cfg queue (count jobs))
|
||||
(doseq [{:keys [id name queue]} jobs]
|
||||
(metrics/record-dispatched cfg name queue 1)
|
||||
(metrics/record-dispatched cfg name 1)
|
||||
(l/trc :hist "schedule"
|
||||
:tenant tenant
|
||||
:queue queue
|
||||
@ -178,9 +287,10 @@ RETURNING job.id, job.queue, job.name, job.created_at")
|
||||
[cfg]
|
||||
(let [cfg (assoc cfg ::timestamp (ct/now))
|
||||
tpoint (ct/tpoint)]
|
||||
;; Reschedule lost in transit jobs (can happen when
|
||||
;; redis server is restarted just after job is pushed)
|
||||
(reschedule-lost-jobs cfg)
|
||||
;; The jobs that were handed to a queue and never claimed are
|
||||
;; terminal: a lost message is not re-dispatched, so no job ever goes
|
||||
;; back to `new` and the dedupe index stays out of this path.
|
||||
(abort-lost-jobs cfg)
|
||||
|
||||
;; Mark as aborted all jobs that are still marked as running but
|
||||
;; their last modification (heartbeat or progress) is older than
|
||||
@ -207,9 +317,10 @@ RETURNING job.id, job.queue, job.name, job.created_at")
|
||||
(px/sleep (or (::timeout cfg) (ct/duration "10s"))))
|
||||
|
||||
(defn run-batch
|
||||
"Execute a single dispatch batch: reschedule lost jobs, mark orphans
|
||||
(lease-based) and claim pending jobs into their Redis queues. Exposed
|
||||
as a function for testability; the dispatcher thread loops on it."
|
||||
"Execute a single dispatch batch: abort the lost scheduled jobs, abort
|
||||
the orphans (lease-based) and claim pending jobs into their Redis
|
||||
queues. Exposed as a function for testability; the dispatcher thread
|
||||
loops on it."
|
||||
[cfg]
|
||||
(let [tpoint (ct/tpoint)
|
||||
tenant (::wrk/tenant cfg)]
|
||||
|
||||
@ -37,9 +37,8 @@
|
||||
terminal job produces
|
||||
0 affected rows and is skipped without touching its state
|
||||
(first-terminal-wins companion). Also predicates the payload
|
||||
scheduled_at: a row rescheduled after the payload was pushed
|
||||
(dispatcher re-pushes on reschedule) is never claimed with the stale
|
||||
payload."
|
||||
scheduled_at: a duplicate or stale payload is never claimed with a
|
||||
timestamp that no longer matches the row."
|
||||
[cfg job-id scheduled-at]
|
||||
(jobs/claim cfg job-id scheduled-at))
|
||||
|
||||
@ -111,10 +110,9 @@
|
||||
|
||||
(do
|
||||
(let [job-def (jobs/get-job-def defs (:name job))
|
||||
_ (jobs-metrics/record-queue-wait
|
||||
_ (jobs-metrics/record-claim-wait
|
||||
cfg
|
||||
(::jobs/name job-def)
|
||||
queue
|
||||
(- (inst-ms (ct/now)) (inst-ms (:scheduled-at job))))
|
||||
params (try
|
||||
(->> (:params job)
|
||||
@ -140,7 +138,6 @@
|
||||
(jobs-metrics/record-execution
|
||||
cfg
|
||||
(:name job)
|
||||
queue
|
||||
(inst-ms (tpoint)))
|
||||
(jobs-metrics/record-legacy-execution
|
||||
cfg
|
||||
@ -236,7 +233,7 @@
|
||||
|
||||
(not= (inst-ms scheduled-at)
|
||||
(inst-ms (:scheduled-at job)))
|
||||
(l/wrn :hint "skipping job, rescheduled"
|
||||
(l/wrn :hint "skipping job, stale payload"
|
||||
:job-id (str job-id)
|
||||
:runner-id id
|
||||
:tenant tenant
|
||||
|
||||
@ -821,7 +821,8 @@
|
||||
(binding [cf/config {:public-uri "https://penpot.example.com"}]
|
||||
(with-redefs [app.nitrate/call (constantly {:active true})
|
||||
app.auth.oidc/prepare-organization-sso-provider (constantly {:type "oidc"})
|
||||
app.auth.oidc/get-info (constantly {})]
|
||||
app.auth.oidc/get-info (constantly {})
|
||||
app.loggers.audit/submit (constantly nil)]
|
||||
(let [result (#'oidc/callback-handler cfg request)]
|
||||
(t/is (= "https://penpot.example.com/" (redirect-location result))))))))
|
||||
|
||||
@ -838,7 +839,8 @@
|
||||
app.auth.oidc/prepare-organization-sso-provider (constantly {:type "oidc"})
|
||||
app.auth.oidc/get-info (fn [& _]
|
||||
(ex/raise :type :internal
|
||||
:code :unable-to-retrieve-user-info))]
|
||||
:code :unable-to-retrieve-user-info))
|
||||
app.loggers.audit/submit (constantly nil)]
|
||||
(let [result (#'oidc/callback-handler cfg request)
|
||||
loc (redirect-location result)]
|
||||
(t/is (not (.contains loc "evil.example")))
|
||||
|
||||
@ -153,3 +153,19 @@
|
||||
(t/deftest read-config-takes-the-default-back-end
|
||||
(t/is (= "db" (c/get (read-config {} :default cf/default)
|
||||
:file-data-backend))))
|
||||
|
||||
(t/deftest audit-persistence-follows-consumer-flags
|
||||
(t/testing "a consumer flag enables persistence"
|
||||
(t/is (cf/persist-audit-events? #{:nexus}))
|
||||
(t/is (cf/persist-audit-events? #{:admin-console})))
|
||||
(t/testing "other sinks alone do not enable persistence"
|
||||
(doseq [flags [#{} #{:webhooks} #{:telemetry} #{:error-reporting}
|
||||
#{:audit-log-logger}]]
|
||||
(t/is (not (cf/persist-audit-events? flags)) (str flags))))
|
||||
(t/testing "the boot value is derived from the loaded flags"
|
||||
(t/is (= (cf/persist-audit-events? cf/flags) cf/audit-persist?)))
|
||||
(t/testing "a stale enable-audit-log in the environment is dropped"
|
||||
(let [flags (#'cf/parse-flags {:flags #{"enable-audit-log" "enable-nexus"}})]
|
||||
(t/is (not (contains? flags :audit-log)))
|
||||
(t/is (contains? flags :nexus))
|
||||
(t/is (cf/persist-audit-events? flags)))))
|
||||
|
||||
@ -195,3 +195,93 @@
|
||||
["SELECT current_setting('enable_partitionwise_join') AS partitionwise_join"])]
|
||||
|
||||
(t/is (= "on" (:partitionwise-join row)))))
|
||||
|
||||
(t/deftest session-locks-are-held-per-connection
|
||||
(let [classid 730000
|
||||
objid 1]
|
||||
(with-open [a (db/open th/*pool*)
|
||||
b (db/open th/*pool*)]
|
||||
(t/testing "the first connection takes it, the second cannot"
|
||||
(t/is (true? (db/session-try-lock a classid objid)))
|
||||
(t/is (false? (db/session-try-lock b classid objid))))
|
||||
(t/testing "releasing it lets the other take it"
|
||||
(t/is (true? (db/session-unlock a classid objid)))
|
||||
(t/is (true? (db/session-try-lock b classid objid)))
|
||||
(t/is (true? (db/session-unlock b classid objid)))))))
|
||||
|
||||
(t/deftest session-locks-keep-the-two-key-space-apart
|
||||
;; PostgreSQL documents the split: a bigint key is shown with its
|
||||
;; high-order half in classid, its low-order half in objid and
|
||||
;; objsubid 1, while a two-int key keeps both numbers and objsubid 2.
|
||||
;; The two forms are therefore different locks even with the same
|
||||
;; numbers, which is what the consumers rely on to never block a file
|
||||
;; update. The test proves each step instead of assuming it.
|
||||
(let [classid 730000
|
||||
objid 2
|
||||
bigint-key (+ (* (long classid) 4294967296) objid)]
|
||||
(with-open [a (db/open th/*pool*)
|
||||
b (db/open th/*pool*)
|
||||
c (db/open th/*pool*)]
|
||||
(t/testing "the one-key form really holds that identity"
|
||||
(t/is (true? (:lock (db/exec-one! a ["select pg_try_advisory_lock(?::bigint) as lock"
|
||||
bigint-key]))))
|
||||
(t/is (false? (:lock (db/exec-one! b ["select pg_try_advisory_lock(?::bigint) as lock"
|
||||
bigint-key])))))
|
||||
|
||||
(t/testing "PostgreSQL displays that bigint as the (classid, objid) pair"
|
||||
(let [{:keys [classid objid objsubid]}
|
||||
(db/exec-one! a ["select classid, objid, objsubid from pg_locks
|
||||
where pid = pg_backend_pid()
|
||||
and locktype = 'advisory'
|
||||
and objsubid = 1
|
||||
and granted"])]
|
||||
(t/is (= [730000 2 1] [classid objid objsubid]))
|
||||
;; and the displayed halves reassemble to the same bigint
|
||||
(t/is (= bigint-key (+ (* (long classid) 4294967296) objid)))))
|
||||
|
||||
(t/testing "the same numbers in the two-key form are another lock"
|
||||
(t/is (true? (db/session-try-lock c classid objid)))
|
||||
(t/is (true? (db/session-unlock c classid objid))))
|
||||
|
||||
(t/is (true? (:lock (db/exec-one! a ["select pg_advisory_unlock(?::bigint) as lock"
|
||||
bigint-key])))))))
|
||||
|
||||
(t/deftest session-locks-reject-a-key-that-is-not-int4
|
||||
(with-open [a (db/open th/*pool*)]
|
||||
(t/is (thrown? IllegalArgumentException
|
||||
(db/session-try-lock a uuid/zero 1)))
|
||||
(t/is (thrown? IllegalArgumentException
|
||||
(db/session-unlock a 730000 uuid/zero)))
|
||||
(t/is (thrown? IllegalArgumentException
|
||||
(db/session-try-lock a 730000 (inc Integer/MAX_VALUE))))))
|
||||
|
||||
(t/deftest session-unlock-all-releases-every-lock-of-the-connection
|
||||
(let [classid 730000]
|
||||
(with-open [a (db/open th/*pool*)
|
||||
b (db/open th/*pool*)]
|
||||
(t/is (true? (db/session-try-lock a classid 1)))
|
||||
(t/is (true? (db/session-try-lock a classid 2)))
|
||||
(t/is (false? (db/session-try-lock b classid 1)))
|
||||
(db/session-unlock-all a)
|
||||
(t/is (true? (db/session-try-lock b classid 1)))
|
||||
(t/is (true? (db/session-try-lock b classid 2)))
|
||||
(db/session-unlock-all b))))
|
||||
|
||||
(t/deftest session-unlock-answers-false-for-a-lock-not-held
|
||||
(with-open [a (db/open th/*pool*)]
|
||||
(t/is (false? (db/session-unlock a 730000 3)))))
|
||||
|
||||
(t/deftest own-transaction-drops-the-connection-and-the-read-only-flag
|
||||
;; The contract has two halves: the write must not join the caller's
|
||||
;; connection, and it must be writable even when the caller is
|
||||
;; read-only (`::db/read-only` dropped).
|
||||
(let [caller-conn (Object.)
|
||||
cfg {::db/pool th/*pool*
|
||||
::db/conn caller-conn
|
||||
::db/read-only true}]
|
||||
(t/testing "the connection and the read-only flag are gone"
|
||||
(t/is (= {::db/pool th/*pool*} (db/own-transaction cfg))))
|
||||
(t/testing "so the write opens its own transaction on another connection"
|
||||
(db/tx-run! (db/own-transaction cfg)
|
||||
(fn [{:keys [::db/conn]}]
|
||||
(t/is (not (identical? conn caller-conn))))))))
|
||||
|
||||
@ -32,7 +32,7 @@
|
||||
;; carries every affected id so a projection can follow without reading
|
||||
;; production tables.
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}]
|
||||
(with-redefs [cf/flags (conj cf/flags :audit-log)]
|
||||
(th/with-audit-config (conj cf/flags :nexus)
|
||||
(let [profile (th/create-profile* 1 {:is-active true})
|
||||
team (th/create-team* 91 {:profile-id (:id profile)})
|
||||
project (th/create-project* 91 {:profile-id (:id profile)
|
||||
@ -56,9 +56,13 @@
|
||||
(t/is (= "team" (name (:object props))))
|
||||
(t/is (= (:id team) (:id props)))
|
||||
(t/is (= deleted-at (:deleted-at props)))
|
||||
(t/is (not (contains? props :team-ids)))
|
||||
(t/is (not (contains? props :project-ids)))
|
||||
(t/is (not (contains? props :file-ids)))
|
||||
;; The shape is the contract: only the team ids go flat, the
|
||||
;; rest stays under `:cascade`.
|
||||
(t/is (= #{:object :id :deleted-at :team-ids :cascade}
|
||||
(set (keys props))))
|
||||
;; Flat team ids for team-keyed consumers, next to the
|
||||
;; per-entity cascade breakdown.
|
||||
(t/is (= #{(:id team)} (set (:team-ids props))))
|
||||
(t/is (= #{(:id team)} (set (get-in props [:cascade :team-ids]))))
|
||||
(t/is (= #{(:default-project-id team) (:id project)}
|
||||
(set (get-in props [:cascade :project-ids]))))
|
||||
@ -66,7 +70,7 @@
|
||||
|
||||
(t/deftest delete-object-file-cascade-emits-event
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}]
|
||||
(with-redefs [cf/flags (conj cf/flags :audit-log)]
|
||||
(th/with-audit-config (conj cf/flags :nexus)
|
||||
(let [profile (th/create-profile* 1 {:is-active true})
|
||||
team (th/create-team* 92 {:profile-id (:id profile)})
|
||||
project (th/create-project* 92 {:profile-id (:id profile)
|
||||
@ -85,12 +89,13 @@
|
||||
(t/is (= (:id file) (:id props)))
|
||||
(t/is (= [] (get-in props [:cascade :team-ids])))
|
||||
(t/is (= [] (get-in props [:cascade :project-ids])))
|
||||
(t/is (= [(:id file)] (get-in props [:cascade :file-ids])))))))))
|
||||
(t/is (= [(:id file)] (get-in props [:cascade :file-ids])))
|
||||
(t/is (= #{(:id team)} (set (:team-ids props))))))))))
|
||||
|
||||
(t/deftest delete-object-missing-row-emits-nothing
|
||||
;; A cascade over an unknown id marks nothing, so there is nothing to report.
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}]
|
||||
(with-redefs [cf/flags (conj cf/flags :audit-log)]
|
||||
(th/with-audit-config (conj cf/flags :nexus)
|
||||
(th/run-task! :delete-object {:object :team
|
||||
:id (uuid/next)
|
||||
:deleted-at (ct/now)})
|
||||
@ -99,7 +104,7 @@
|
||||
(t/deftest delete-object-snapshot-emits-nothing
|
||||
;; Snapshots have no projection table; their cascade stays silent.
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}]
|
||||
(with-redefs [cf/flags (conj cf/flags :audit-log)]
|
||||
(th/with-audit-config (conj cf/flags :nexus)
|
||||
(th/run-task! :delete-object {:object :snapshot
|
||||
:id (uuid/next)
|
||||
:file-id (uuid/next)
|
||||
|
||||
@ -63,6 +63,23 @@
|
||||
(def ^:dynamic *system* nil)
|
||||
(def ^:dynamic *pool* nil)
|
||||
|
||||
(defmacro with-audit-config
|
||||
"Rebind the flags and the values derived from them at boot. An optional
|
||||
binding vector rebinds other vars in the same scope."
|
||||
[flags & body]
|
||||
(let [extra (if (and (seq body) (vector? (first body)))
|
||||
(first body)
|
||||
[])
|
||||
body (if (seq extra) (rest body) body)]
|
||||
`(let [flags# ~flags]
|
||||
(with-redefs [cf/flags flags#
|
||||
cf/audit-persist? (cf/persist-audit-events? flags#)
|
||||
cf/webhooks-enabled? (contains? flags# :webhooks)
|
||||
cf/telemetry-enabled? (contains? flags# :telemetry)
|
||||
cf/error-reporting-enabled? (contains? flags# :error-reporting)
|
||||
~@extra]
|
||||
~@body))))
|
||||
|
||||
;; Fallback values used when no PENPOT_TEST_* env vars are set (e.g. CI sets
|
||||
;; them explicitly, see .github/workflows/tests-backend.yml). Inside the
|
||||
;; devenv each wsN container receives per-instance values via
|
||||
@ -94,8 +111,12 @@
|
||||
([next]
|
||||
(init-config nil next))
|
||||
([extra-flags next]
|
||||
(let [flags (into default-flags extra-flags)]
|
||||
(with-redefs [app.config/flags (flags/parse flags/default flags)
|
||||
(let [flags (flags/parse flags/default (into default-flags extra-flags))]
|
||||
(with-redefs [app.config/flags flags
|
||||
app.config/audit-persist? (cf/persist-audit-events? flags)
|
||||
app.config/webhooks-enabled? (contains? flags :webhooks)
|
||||
app.config/telemetry-enabled? (contains? flags :telemetry)
|
||||
app.config/error-reporting-enabled? (contains? flags :error-reporting)
|
||||
app.config/config config
|
||||
app.loggers.audit/submit (constantly nil)
|
||||
app.auth/derive-password identity
|
||||
@ -153,7 +174,10 @@
|
||||
:app.loggers.database/reporter
|
||||
:app.loggers.metrics/reporter
|
||||
:app.worker/cron
|
||||
:app.worker/dispatcher)
|
||||
:app.worker/dispatcher
|
||||
:app.loggers.audit.consumers.nexus/consumer
|
||||
:app.loggers.audit.consumers.admin-console/consumer
|
||||
:app.loggers.audit.consumers.telemetry/consumer)
|
||||
(remove-job-runners))
|
||||
_ (ig/load-namespaces system)
|
||||
system (-> (ig/expand system) (ig/init))]
|
||||
@ -658,6 +682,18 @@
|
||||
(assoc ::jobs/context (jobs/make-context row))
|
||||
(assoc ::jobs/job-id (:id row)))))))))
|
||||
|
||||
(defn wait-for
|
||||
"Poll `pred` until it holds, up to `timeout` ms (5 s by default). Answers
|
||||
true when it held, false on timeout."
|
||||
([pred] (wait-for pred 5000))
|
||||
([pred timeout]
|
||||
(let [deadline (+ (System/currentTimeMillis) timeout)]
|
||||
(loop []
|
||||
(cond
|
||||
(pred) true
|
||||
(< (System/currentTimeMillis) deadline) (do (Thread/sleep 10) (recur))
|
||||
:else false)))))
|
||||
|
||||
(defn metrics-stub
|
||||
"A metrics instance with no collectors, where every `run!` is a no-op."
|
||||
[]
|
||||
|
||||
@ -6,9 +6,6 @@
|
||||
|
||||
(ns backend-tests.jobs-metrics-test
|
||||
(:require
|
||||
[app.common.time :as ct]
|
||||
[app.common.uuid :as uuid]
|
||||
[app.config :as cf]
|
||||
[app.db :as db]
|
||||
[app.jobs :as jobs]
|
||||
[app.jobs.metrics :as jobs-metrics]
|
||||
@ -24,29 +21,22 @@
|
||||
io.prometheus.client.Collector$MetricFamilySamples$Sample
|
||||
io.prometheus.client.CollectorRegistry
|
||||
io.prometheus.client.Counter
|
||||
io.prometheus.client.Counter$Child
|
||||
io.prometheus.client.Gauge
|
||||
io.prometheus.client.Gauge$Child))
|
||||
io.prometheus.client.Counter$Child))
|
||||
|
||||
(t/use-fixtures :once th/state-init)
|
||||
(t/use-fixtures :each th/database-reset)
|
||||
|
||||
(def ^:private metric-ids
|
||||
[:jobs-submitted
|
||||
:jobs-skipped
|
||||
[:jobs-submit
|
||||
:jobs-dispatched
|
||||
:jobs-retries
|
||||
:jobs-orphaned
|
||||
:jobs-rescheduled
|
||||
:jobs-queue-wait-timing
|
||||
:jobs-claim-wait-timing
|
||||
:jobs-execution-timing
|
||||
:jobs-total-timing
|
||||
:jobs-dispatcher-timing
|
||||
:jobs-dispatcher-batch-timing
|
||||
:jobs-dispatcher-batch-size
|
||||
:jobs-backlog
|
||||
:jobs-oldest-pending-age
|
||||
:jobs-gc-rows
|
||||
:jobs-gc-timing
|
||||
:jobs-gc-pass-timing
|
||||
:jobs-cron-total
|
||||
:jobs-request-timing
|
||||
:jobs-events-total])
|
||||
@ -90,29 +80,34 @@
|
||||
;; assertions use, and (= 0 0.0) is false in Clojure
|
||||
(reduce + 0.0))))
|
||||
|
||||
(defn- gauge-value [metrics id labels]
|
||||
(let [collector (mtx/get-collector metrics id)
|
||||
instance (::mdef/instance collector)
|
||||
child (.labels ^Gauge instance (into-array String labels))]
|
||||
(.get ^Gauge$Child child)))
|
||||
(defn- echo-defs
|
||||
"The minimal job-def the tests submit and drive through the lifecycle."
|
||||
[]
|
||||
{:echo {::jobs/name :echo
|
||||
::jobs/schema [:map]
|
||||
::jobs/handler (fn [_context params] params)
|
||||
::jobs/decoder identity
|
||||
::jobs/validator (constantly true)}})
|
||||
|
||||
(t/deftest default-metrics-cover-the-jobs-observability-contract
|
||||
(doseq [id metric-ids]
|
||||
(let [definition (get main/default-metrics id)]
|
||||
(t/is (some? definition) id)
|
||||
(t/is (re-find #"^penpot_jobs_" (::mdef/name definition)) id)))
|
||||
(t/is (= ["name" "queue" "outcome"]
|
||||
(t/is (= ["name" "outcome"]
|
||||
(::mdef/labels (:jobs-submit main/default-metrics))))
|
||||
(t/is (= ["name" "outcome"]
|
||||
(::mdef/labels (:jobs-total-timing main/default-metrics))))
|
||||
(t/is (= ["status"]
|
||||
(::mdef/labels (:jobs-backlog main/default-metrics)))))
|
||||
(t/is (= ["name"]
|
||||
(::mdef/labels (:jobs-claim-wait-timing main/default-metrics))))
|
||||
(t/testing "one submit family and one terminal counter, with no twin ids"
|
||||
(t/is (not (contains? main/default-metrics :jobs-submitted)))
|
||||
(t/is (not (contains? main/default-metrics :jobs-skipped)))
|
||||
(t/is (not (contains? main/default-metrics :jobs-orphaned)))))
|
||||
|
||||
(t/deftest submit-metric-is-not-recorded-when-the-outer-transaction-rolls-back
|
||||
(let [metrics (make-metrics)
|
||||
defs {:echo {::jobs/name :echo
|
||||
::jobs/schema [:map]
|
||||
::jobs/handler (fn [_context params] params)
|
||||
::jobs/decoder identity
|
||||
::jobs/validator (constantly true)}}
|
||||
defs (echo-defs)
|
||||
cfg {::jobs/defs defs
|
||||
::db/pool th/*pool*
|
||||
::mtx/metrics metrics}]
|
||||
@ -123,68 +118,114 @@
|
||||
{::jobs/name :echo
|
||||
::jobs/params {}})
|
||||
(throw (ex-info "rollback" {}))))))
|
||||
(t/is (= 0.0 (counter-value metrics :jobs-submitted ["echo" "default"])))))
|
||||
(t/is (= 0.0 (counter-value metrics :jobs-submit ["echo" "created"])))))
|
||||
|
||||
(t/deftest submit-and-terminal-writers-record-production-events
|
||||
(let [metrics (make-metrics)
|
||||
defs {:echo {::jobs/name :echo
|
||||
::jobs/schema [:map]
|
||||
::jobs/handler (fn [_context params] params)
|
||||
::jobs/decoder identity
|
||||
::jobs/validator (constantly true)}}
|
||||
defs (echo-defs)
|
||||
cfg {::jobs/defs defs
|
||||
::db/pool th/*pool*
|
||||
::mtx/metrics metrics}
|
||||
job-id (jobs/submit cfg {::jobs/name :echo ::jobs/params {}})]
|
||||
(jobs/claim cfg job-id (:scheduled-at (jobs/get-job cfg job-id)))
|
||||
(jobs/complete cfg :job-id job-id :result {:ok true})
|
||||
(t/is (= 1.0 (counter-value metrics :jobs-submitted ["echo" "default"])))
|
||||
(t/is (= 1.0 (histogram-count metrics :jobs-total-timing ["echo" "default" "completed"])))))
|
||||
(t/is (= 1.0 (counter-value metrics :jobs-submit ["echo" "created"])))
|
||||
(t/is (= 1.0 (histogram-count metrics :jobs-total-timing ["echo" "completed"])))))
|
||||
|
||||
(t/deftest skip-if-active-records-skipped-metric
|
||||
(let [metrics (make-metrics)
|
||||
defs {:echo {::jobs/name :echo
|
||||
::jobs/schema [:map]
|
||||
::jobs/handler (fn [_context params] params)
|
||||
::jobs/decoder identity
|
||||
::jobs/validator (constantly true)}}
|
||||
defs (echo-defs)
|
||||
cfg {::jobs/defs defs
|
||||
::db/pool th/*pool*
|
||||
::mtx/metrics metrics}
|
||||
_ (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params {}
|
||||
::jobs/label "skip-metric"})
|
||||
skipped (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params {}
|
||||
::jobs/label "skip-metric"
|
||||
_ (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params {}
|
||||
::jobs/label "skip-metric"
|
||||
::jobs/skip-if-active true})
|
||||
skipped (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params {}
|
||||
::jobs/label "skip-metric"
|
||||
::jobs/skip-if-active true})]
|
||||
(t/is (nil? skipped))
|
||||
(t/is (= 1.0 (counter-value metrics :jobs-skipped ["echo" "default"])))
|
||||
(t/is (= 1.0 (counter-value metrics :jobs-submitted ["echo" "default"]))
|
||||
"the withheld submit records no submission")))
|
||||
(t/is (= 1.0 (counter-value metrics :jobs-submit ["echo" "skipped"])))
|
||||
(t/is (= 1.0 (counter-value metrics :jobs-submit ["echo" "created"]))
|
||||
"the withheld submit records no creation")))
|
||||
|
||||
(t/deftest dedupe-skip-is-counted-apart-from-a-creation
|
||||
(let [metrics (make-metrics)
|
||||
cfg {::jobs/defs (echo-defs)
|
||||
::db/pool th/*pool*
|
||||
::mtx/metrics metrics}
|
||||
options {::jobs/name :echo
|
||||
::jobs/params {}
|
||||
::jobs/label "dedupe-metric"}
|
||||
created (jobs/submit cfg (assoc options ::jobs/dedupe true))
|
||||
skipped (jobs/submit cfg (assoc options ::jobs/dedupe true))]
|
||||
(t/is (some? created))
|
||||
(t/is (nil? skipped))
|
||||
(t/is (= 1.0 (counter-value metrics :jobs-submit ["echo" "created"])))
|
||||
(t/is (= 1.0 (counter-value metrics :jobs-submit ["echo" "skipped"])))
|
||||
(t/is (= 0.0 (counter-value metrics :jobs-submit ["echo" "refreshed"])))))
|
||||
|
||||
(t/deftest debounce-refresh-is-counted-apart-from-a-creation
|
||||
(let [metrics (make-metrics)
|
||||
cfg {::jobs/defs (echo-defs)
|
||||
::db/pool th/*pool*
|
||||
::mtx/metrics metrics}
|
||||
options {::jobs/name :echo
|
||||
::jobs/params {}
|
||||
::jobs/label "debounce-metric"
|
||||
::jobs/debounce true}
|
||||
created (jobs/submit cfg options)
|
||||
refreshed (jobs/submit cfg options)]
|
||||
(t/is (some? created))
|
||||
(t/is (= created refreshed)
|
||||
"a refresh answers with the id the row already had")
|
||||
(t/is (= 1.0 (counter-value metrics :jobs-submit ["echo" "created"])))
|
||||
(t/is (= 1.0 (counter-value metrics :jobs-submit ["echo" "refreshed"])))
|
||||
(t/is (= 0.0 (counter-value metrics :jobs-submit ["echo" "skipped"])))))
|
||||
|
||||
(t/deftest the-terminal-histogram-count-is-the-counter-of-every-outcome
|
||||
(let [metrics (make-metrics)
|
||||
cfg {::jobs/defs (echo-defs)
|
||||
::db/pool th/*pool*
|
||||
::mtx/metrics metrics}
|
||||
claim! (fn [id]
|
||||
(jobs/claim cfg id (:scheduled-at (jobs/get-job cfg id))))
|
||||
completed (jobs/submit cfg {::jobs/name :echo ::jobs/params {}})
|
||||
failed (jobs/submit cfg {::jobs/name :echo ::jobs/params {}})
|
||||
cancelled (jobs/submit cfg {::jobs/name :echo ::jobs/params {}})]
|
||||
(claim! completed)
|
||||
(jobs/complete cfg :job-id completed :result {:ok true})
|
||||
(claim! failed)
|
||||
(jobs/fail cfg failed {:type :internal :code :boom :hint "boom"})
|
||||
(claim! cancelled)
|
||||
(jobs/cancel cfg cancelled)
|
||||
|
||||
(t/testing "one observation per outcome, with no twin counter"
|
||||
(t/is (= 1.0 (histogram-count metrics :jobs-total-timing ["echo" "completed"])))
|
||||
(t/is (= 1.0 (histogram-count metrics :jobs-total-timing ["echo" "failed"])))
|
||||
(t/is (= 1.0 (histogram-count metrics :jobs-total-timing ["echo" "cancelled"])))
|
||||
(t/is (= 0.0 (histogram-count metrics :jobs-total-timing ["echo" "aborted"]))))))
|
||||
|
||||
(t/deftest lifecycle-helpers-use-labels-as-is
|
||||
(let [metrics (make-metrics)
|
||||
cfg (metrics-cfg metrics)]
|
||||
(jobs-metrics/record-submitted cfg "delete-object" "default")
|
||||
(jobs-metrics/record-skipped cfg "delete-object" "default")
|
||||
(jobs-metrics/record-dispatched cfg "delete-object" "default" 2)
|
||||
(jobs-metrics/record-total cfg "delete-object" "default" :completed 42)
|
||||
(jobs-metrics/record-retry cfg "delete-object" "default" :backoff)
|
||||
(jobs-metrics/record-orphan cfg "webhooks")
|
||||
(jobs-metrics/record-rescheduled cfg "custom")
|
||||
(jobs-metrics/record-created cfg "delete-object")
|
||||
(jobs-metrics/record-skipped cfg "delete-object")
|
||||
(jobs-metrics/record-dispatched cfg "delete-object" 2)
|
||||
(jobs-metrics/record-total cfg "delete-object" :completed 42)
|
||||
(jobs-metrics/record-retry cfg "delete-object" :backoff)
|
||||
(jobs-metrics/record-gc-rows cfg :expired :deleted 3)
|
||||
(jobs-metrics/record-cron cfg :submitted :none)
|
||||
(jobs-metrics/record-request cfg :replied 12)
|
||||
(jobs-metrics/record-legacy-execution cfg "delete-object" 42)
|
||||
|
||||
(t/is (= 1.0 (counter-value metrics :jobs-submitted ["delete-object" "default"])))
|
||||
(t/is (= 1.0 (counter-value metrics :jobs-skipped ["delete-object" "default"])))
|
||||
(t/is (= 2.0 (counter-value metrics :jobs-dispatched ["delete-object" "default"])))
|
||||
(t/is (= 1.0 (histogram-count metrics :jobs-total-timing ["delete-object" "default" "completed"])))
|
||||
(t/is (= 1.0 (counter-value metrics :jobs-retries ["delete-object" "default" "backoff"])))
|
||||
(t/is (= 1.0 (counter-value metrics :jobs-orphaned ["webhooks"])))
|
||||
(t/is (= 1.0 (counter-value metrics :jobs-rescheduled ["custom"])))
|
||||
(t/is (= 1.0 (counter-value metrics :jobs-submit ["delete-object" "created"])))
|
||||
(t/is (= 1.0 (counter-value metrics :jobs-submit ["delete-object" "skipped"])))
|
||||
(t/is (= 2.0 (counter-value metrics :jobs-dispatched ["delete-object"])))
|
||||
(t/is (= 1.0 (histogram-count metrics :jobs-total-timing ["delete-object" "completed"])))
|
||||
(t/is (= 1.0 (counter-value metrics :jobs-retries ["delete-object" "backoff"])))
|
||||
(t/is (= 3.0 (counter-value metrics :jobs-gc-rows ["expired" "deleted"])))
|
||||
(t/is (= 1.0 (counter-value metrics :jobs-cron-total ["submitted" "none"])))
|
||||
(t/is (= 1.0 (histogram-count metrics :jobs-request-timing ["replied"])))
|
||||
@ -197,15 +238,15 @@
|
||||
(t/is (= 0.0 (histogram-count metrics :tasks-timing ["other"]))))))
|
||||
|
||||
(t/deftest labels-go-to-metrics-as-is
|
||||
(t/testing "queues are lowercased, with other only when missing"
|
||||
(t/testing "the queue of a batch is lowercased, other only when missing"
|
||||
(t/is (= "unexpected" (jobs-metrics/queue-label "unexpected")))
|
||||
(t/is (= "default" (jobs-metrics/queue-label "default")))
|
||||
(t/is (= "default" (jobs-metrics/queue-label :default)))
|
||||
(t/is (= "other" (jobs-metrics/queue-label nil))))
|
||||
(let [metrics (make-metrics)
|
||||
cfg (metrics-cfg metrics)]
|
||||
(jobs-metrics/record-total cfg "echo" "unexpected" :unexpected 42)
|
||||
(t/is (= 1.0 (histogram-count metrics :jobs-total-timing ["echo" "unexpected" "unexpected"])))
|
||||
(jobs-metrics/record-total cfg "echo" :unexpected 42)
|
||||
(t/is (= 1.0 (histogram-count metrics :jobs-total-timing ["echo" "unexpected"])))
|
||||
(t/testing "a new value is recorded, never folded or dropped"
|
||||
(jobs-metrics/record-gc-rows cfg :unknown :deleted 1)
|
||||
(jobs-metrics/record-cron cfg :brand-new :whatever)
|
||||
@ -214,57 +255,17 @@
|
||||
(t/is (= 1.0 (counter-value metrics :jobs-cron-total ["brand-new" "whatever"])))
|
||||
(t/is (= 1.0 (histogram-count metrics :jobs-request-timing ["brand-new"]))))))
|
||||
|
||||
(t/deftest sampler-is-wired-and-does-not-start-on-read-only
|
||||
(t/is (contains? main/worker-config :app.jobs.metrics/sampler))
|
||||
(let [metrics (make-metrics)
|
||||
cfg {::db/pool th/*pool*
|
||||
::mtx/metrics metrics}]
|
||||
(with-redefs [db/read-only? (constantly true)]
|
||||
(t/is (nil? (ig/init-key :app.jobs.metrics/sampler cfg))))
|
||||
(let [sampler (ig/init-key :app.jobs.metrics/sampler cfg)]
|
||||
(try
|
||||
(t/is (some? sampler))
|
||||
(finally
|
||||
(ig/halt-key! :app.jobs.metrics/sampler sampler))))))
|
||||
|
||||
(t/deftest unknown-names-go-to-metrics-as-is
|
||||
(let [metrics (make-metrics)
|
||||
cfg (metrics-cfg metrics)]
|
||||
(jobs-metrics/record-submitted cfg "unknown-job" "default")
|
||||
(t/is (= 1.0 (counter-value metrics :jobs-submitted ["unknown-job" "default"])))))
|
||||
|
||||
(t/deftest backlog-sampler-updates-status-and-age-gauges
|
||||
(let [metrics (make-metrics)
|
||||
cfg {::db/pool th/*pool*
|
||||
::mtx/metrics metrics}
|
||||
now (ct/now)]
|
||||
(th/db-insert! :job {:id (uuid/next)
|
||||
:name "echo"
|
||||
:tenant (cf/get :tenant)
|
||||
:queue "default"
|
||||
:params (db/json {})
|
||||
:priority 100
|
||||
:max-retries 3
|
||||
:retry-num 0
|
||||
:status "new"
|
||||
:scheduled-at now
|
||||
:created-at (ct/in-past {:minutes 2})
|
||||
:modified-at now})
|
||||
(jobs-metrics/sample-backlog cfg)
|
||||
(t/is (= 1.0 (gauge-value metrics :jobs-backlog ["new"])))
|
||||
(t/is (= 0.0 (gauge-value metrics :jobs-backlog ["completed"])))
|
||||
(t/is (= 0.0 (gauge-value metrics :jobs-backlog ["aborted"])))
|
||||
(t/is (>= (gauge-value metrics :jobs-oldest-pending-age []) 120.0))))
|
||||
(jobs-metrics/record-created cfg "unknown-job")
|
||||
(t/is (= 1.0 (counter-value metrics :jobs-submit ["unknown-job" "created"])))))
|
||||
|
||||
(t/deftest event-counter-has-no-labels-and-follows-the-events
|
||||
(let [metrics (make-metrics)
|
||||
cfg {::db/pool th/*pool*
|
||||
::mtx/metrics metrics}
|
||||
defs {:echo {::jobs/name :echo
|
||||
::jobs/schema [:map]
|
||||
::jobs/handler (fn [_context params] params)
|
||||
::jobs/decoder identity
|
||||
::jobs/validator (constantly true)}}
|
||||
defs (echo-defs)
|
||||
cfg2 (assoc cfg ::jobs/defs defs)
|
||||
job-id (jobs/submit cfg2 {::jobs/name :echo ::jobs/params {}})]
|
||||
(jobs/claim cfg2 job-id (:scheduled-at (jobs/get-job cfg2 job-id)))
|
||||
@ -278,11 +279,7 @@
|
||||
|
||||
(t/deftest event-counter-follows-a-lifecycle-write-that-outlives-the-caller
|
||||
(let [metrics (make-metrics)
|
||||
defs {:echo {::jobs/name :echo
|
||||
::jobs/schema [:map]
|
||||
::jobs/handler (fn [_context params] params)
|
||||
::jobs/decoder identity
|
||||
::jobs/validator (constantly true)}}
|
||||
defs (echo-defs)
|
||||
cfg {::jobs/defs defs
|
||||
::db/pool th/*pool*
|
||||
::mtx/metrics metrics}
|
||||
@ -302,15 +299,11 @@
|
||||
(t/is (= 2 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job_event"])))))
|
||||
|
||||
(t/testing "the terminal counter followed its own commit, not the caller one"
|
||||
(t/is (= 1.0 (histogram-count metrics :jobs-total-timing ["echo" "default" "completed"]))))))
|
||||
(t/is (= 1.0 (histogram-count metrics :jobs-total-timing ["echo" "completed"]))))))
|
||||
|
||||
(t/deftest progress-counter-follows-the-report-that-outlives-the-caller
|
||||
(let [metrics (make-metrics)
|
||||
defs {:echo {::jobs/name :echo
|
||||
::jobs/schema [:map]
|
||||
::jobs/handler (fn [_context params] params)
|
||||
::jobs/decoder identity
|
||||
::jobs/validator (constantly true)}}
|
||||
defs (echo-defs)
|
||||
cfg {::jobs/defs defs
|
||||
::db/pool th/*pool*
|
||||
::mtx/metrics metrics}
|
||||
@ -329,7 +322,7 @@
|
||||
job-id]))))
|
||||
(t/is (= (inc before) (counter-value metrics :jobs-events-total [])))))))
|
||||
|
||||
(t/deftest the-binfile-queue-is-labelled-like-any-other
|
||||
(t/deftest the-submit-carries-no-queue-and-the-batch-size-still-does
|
||||
(let [metrics (make-metrics)
|
||||
defs {:export-binfile {::jobs/name :export-binfile
|
||||
::jobs/schema [:map]
|
||||
@ -344,6 +337,15 @@
|
||||
::jobs/params {}
|
||||
::jobs/queue :binfile})
|
||||
|
||||
(t/testing "the queue of the job is its label, never other"
|
||||
(t/is (= 1.0 (counter-value metrics :jobs-submitted ["export-binfile" "binfile"])))
|
||||
(t/is (= 0.0 (counter-value metrics :jobs-submitted ["export-binfile" "other"]))))))
|
||||
(t/testing "the submit is keyed by the job name and what it did"
|
||||
(t/is (= 1.0 (counter-value metrics :jobs-submit ["export-binfile" "created"])))
|
||||
(t/is (= ["name" "outcome"]
|
||||
(::mdef/labels (:jobs-submit main/default-metrics)))))
|
||||
|
||||
(t/testing "a batch mixes job names, so it keeps the queue"
|
||||
(jobs-metrics/record-dispatcher-size cfg "binfile" 3)
|
||||
(t/is (= 1.0 (histogram-count metrics
|
||||
:jobs-dispatcher-batch-size
|
||||
["binfile"])))
|
||||
(t/is (= ["queue"]
|
||||
(::mdef/labels (:jobs-dispatcher-batch-size main/default-metrics)))))))
|
||||
|
||||
@ -32,6 +32,20 @@
|
||||
[f]
|
||||
(try (f) nil (catch Throwable cause cause)))
|
||||
|
||||
(defn- race
|
||||
"Run every thunk on a thread of its own and return what each returned,
|
||||
in the order they were given. They all wait on the same gate, so they
|
||||
contend for whatever they share instead of running one after another."
|
||||
[& thunks]
|
||||
(let [gate (promise)
|
||||
results (mapv (fn [f]
|
||||
(future
|
||||
@gate
|
||||
(f)))
|
||||
thunks)]
|
||||
(deliver gate true)
|
||||
(mapv deref results)))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; JOB-DEF (plain handler + precompiled init-key)
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
@ -108,14 +122,6 @@
|
||||
:id (uuid/next)
|
||||
:file-id (uuid/next)})
|
||||
|
||||
(defn- insert-job-statement?
|
||||
"True when a `db/exec-one!` call is the job INSERT, whatever the
|
||||
argument shape (a `[sql & params]` vector or a bare SQL string)."
|
||||
[args]
|
||||
(let [stmt (second args)
|
||||
sql (if (string? stmt) stmt (first stmt))]
|
||||
(and (string? sql) (str/starts-with? sql "insert into job"))))
|
||||
|
||||
(def ^:private test-error
|
||||
{:type :internal
|
||||
:code :boom
|
||||
@ -433,70 +439,74 @@
|
||||
(t/is (uuid? (:file-id decoded)))
|
||||
(t/is (ct/inst? (:deleted-at decoded)))))))))
|
||||
|
||||
(t/deftest submit-dedupe-replaces-not-due-new-rows
|
||||
;; --- DEDUPE / DEBOUNCE
|
||||
|
||||
(t/deftest submit-dedupe-skips-a-pending-job
|
||||
(let [cfg (make-cfg (get-job-defs))
|
||||
params (make-params)
|
||||
label "dedupe-label"
|
||||
id1 (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label label
|
||||
::jobs/dedupe true
|
||||
::jobs/delay 10000})
|
||||
id2 (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label label
|
||||
::jobs/dedupe true
|
||||
::jobs/delay 10000})]
|
||||
opts {::jobs/name :echo
|
||||
::jobs/label label
|
||||
::jobs/dedupe true
|
||||
::jobs/delay 10000}
|
||||
id1 (jobs/submit cfg (assoc opts ::jobs/params (make-params)))
|
||||
row1 (jobs/get-job cfg id1)
|
||||
id2 (jobs/submit cfg (assoc opts
|
||||
::jobs/params (assoc (make-params)
|
||||
:text "second")))]
|
||||
|
||||
(t/is (uuid? id1))
|
||||
(t/is (uuid? id2))
|
||||
(t/is (not= id1 id2))
|
||||
(t/testing "the pending job is not touched and nothing is inserted"
|
||||
(t/is (nil? id2))
|
||||
(t/is (= 1 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job
|
||||
WHERE label = ?" label]))))
|
||||
(let [row2 (jobs/get-job cfg id1)]
|
||||
(t/is (= (:scheduled-at row1) (:scheduled-at row2))
|
||||
"the schedule of the pending job does not move")
|
||||
(t/is (= "hello" (:text (:params row2)))
|
||||
"and it keeps its own params")))))
|
||||
|
||||
(t/testing "the first new row (not due yet) is removed"
|
||||
(t/is (nil? (jobs/get-job cfg id1)))
|
||||
(t/deftest submit-dedupe-checks-any-schedule
|
||||
(let [cfg (make-cfg (get-job-defs))
|
||||
params (make-params)]
|
||||
|
||||
(t/testing "and only the new row remains"
|
||||
(t/testing "a due `new` row still counts as pending"
|
||||
(let [label "dedupe-due-label"
|
||||
id1 (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label label
|
||||
::jobs/dedupe true
|
||||
::jobs/delay 10000})
|
||||
_ (th/db-update! :job {:scheduled-at (ct/minus (ct/now)
|
||||
(ct/duration {:seconds 5}))}
|
||||
{:id id1})
|
||||
id2 (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label label
|
||||
::jobs/dedupe true})]
|
||||
(t/is (nil? id2))
|
||||
(t/is (= 1 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job
|
||||
WHERE label = ?" label]))))))))
|
||||
WHERE label = ?" label]))))))
|
||||
|
||||
(t/deftest submit-dedupe-keeps-due-and-running-rows
|
||||
(let [cfg (make-cfg (get-job-defs))
|
||||
params (make-params)
|
||||
label "dedupe-active-label"
|
||||
id1 (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label label})
|
||||
_ (th/db-update! :job {:status "running"} {:id id1})
|
||||
id2 (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label label
|
||||
::jobs/dedupe true})]
|
||||
|
||||
(t/is (some? (jobs/get-job cfg id1)))
|
||||
(t/is (= 2 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job
|
||||
WHERE label = ?" label])))))
|
||||
|
||||
(let [cfg (make-cfg (get-job-defs))
|
||||
params (make-params)
|
||||
label "dedupe-due-label"
|
||||
id1 (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label label})
|
||||
_ (th/db-update! :job {:scheduled-at (ct/minus (ct/now)
|
||||
(ct/duration {:seconds 5}))}
|
||||
{:id id1})
|
||||
id2 (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label label
|
||||
::jobs/dedupe true})]
|
||||
|
||||
(t/is (some? (jobs/get-job cfg id1)))
|
||||
(t/is (= 2 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job
|
||||
WHERE label = ?" label]))))))
|
||||
(t/testing "a started row does not block the next one"
|
||||
(doseq [status ["scheduled" "running" "retry"]]
|
||||
(let [label (str "dedupe-started-" status)
|
||||
id1 (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label label
|
||||
::jobs/dedupe true})
|
||||
_ (th/db-update! :job {:status status} {:id id1})
|
||||
id2 (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label label
|
||||
::jobs/dedupe true})]
|
||||
(t/is (uuid? id2))
|
||||
(t/is (= 2 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job
|
||||
WHERE label = ?" label])))))))))
|
||||
|
||||
(t/deftest submit-dedupe-leaves-another-tenants-rows-alone
|
||||
;; The DELETE matches name, queue and label. Without the tenant in the
|
||||
;; match it would reach into another instance's not-started jobs.
|
||||
;; The tenant is part of the key of the index. Without it this submit
|
||||
;; would see the not-started job of another instance sharing the
|
||||
;; database and skip its own work.
|
||||
(let [cfg (make-cfg (get-job-defs))
|
||||
params (make-params)
|
||||
label "dedupe-shared-label"
|
||||
@ -511,22 +521,149 @@
|
||||
:priority 100
|
||||
:max-retries 3
|
||||
:retry-num 0
|
||||
:dedupe true
|
||||
:scheduled-at (ct/plus (ct/now) (ct/duration {:seconds 10}))
|
||||
:created-at (ct/now)
|
||||
:modified-at (ct/now)})
|
||||
(jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/queue :default
|
||||
::jobs/label label
|
||||
::jobs/dedupe true
|
||||
::jobs/delay 10000})
|
||||
(t/is (some? (jobs/get-job cfg foreign-id))
|
||||
"another tenant's matching job must survive our dedupe")
|
||||
(t/testing "and only our own row was written"
|
||||
(let [id (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/queue :default
|
||||
::jobs/label label
|
||||
::jobs/dedupe true
|
||||
::jobs/delay 10000})]
|
||||
(t/is (uuid? id))
|
||||
(t/is (some? (jobs/get-job cfg foreign-id))
|
||||
"another tenant's matching job must survive our submit")
|
||||
(t/testing "and only our own row was written"
|
||||
(t/is (= 1 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job
|
||||
WHERE label = ?
|
||||
AND tenant = ?"
|
||||
label (cf/get :tenant)]))))))))
|
||||
|
||||
(t/deftest submit-dedupe-does-not-block-on-plain-jobs
|
||||
;; Only the jobs submitted with the option are part of its key, so the
|
||||
;; label of an ordinary job stays a label.
|
||||
(let [cfg (make-cfg (get-job-defs))
|
||||
params (make-params)
|
||||
label "dedupe-over-plain"]
|
||||
(t/is (uuid? (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label label})))
|
||||
(t/is (uuid? (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label label
|
||||
::jobs/dedupe true})))
|
||||
(t/is (= 2 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job
|
||||
WHERE label = ?" label]))))))
|
||||
|
||||
(t/deftest two-concurrent-dedupe-submits-leave-one-row
|
||||
;; The exclusion is the index, not a read before the write: two
|
||||
;; backends submitting the same key at the same time cannot both win.
|
||||
(let [cfg (make-cfg (get-job-defs))
|
||||
params (make-params)
|
||||
label "dedupe-race"
|
||||
submit #(jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label label
|
||||
::jobs/dedupe true})]
|
||||
(let [[a b] (race submit submit)]
|
||||
(t/is (= 1 (count (filter uuid? [a b])))
|
||||
"exactly one of the two submits got a job")
|
||||
(t/is (= 1 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job
|
||||
WHERE label = ?
|
||||
AND tenant = ?"
|
||||
label (cf/get :tenant)])))))))
|
||||
WHERE label = ?" label])))))))
|
||||
|
||||
(t/deftest submit-debounce-refreshes-the-pending-job
|
||||
(let [cfg (make-cfg (get-job-defs))
|
||||
label "debounce-label"
|
||||
opts {::jobs/name :echo
|
||||
::jobs/label label
|
||||
::jobs/debounce true}
|
||||
id1 (jobs/submit cfg (assoc opts
|
||||
::jobs/params (make-params)
|
||||
::jobs/delay 10000))
|
||||
row1 (jobs/get-job cfg id1)
|
||||
id2 (jobs/submit cfg (assoc opts
|
||||
::jobs/params (assoc (make-params)
|
||||
:text "second")
|
||||
::jobs/delay 60000))
|
||||
row2 (jobs/get-job cfg id1)]
|
||||
|
||||
(t/is (= id1 id2)
|
||||
"a refresh answers with the id the row already had")
|
||||
(t/is (= 1 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job
|
||||
WHERE label = ?" label]))))
|
||||
(t/testing "the pending job keeps its id and its creation date"
|
||||
(t/is (some? row2))
|
||||
(t/is (= (:id row1) (:id row2)))
|
||||
(t/is (= (:created-at row1) (:created-at row2))))
|
||||
(t/testing "and takes the new schedule and the new params"
|
||||
(t/is (pos? (compare (:scheduled-at row2) (:scheduled-at row1))))
|
||||
(t/is (= "second" (:text (:params row2)))))))
|
||||
|
||||
(t/deftest submit-debounce-inserts-when-nothing-is-pending
|
||||
(let [cfg (make-cfg (get-job-defs))
|
||||
params (make-params)
|
||||
label "debounce-idle-label"
|
||||
id (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label label
|
||||
::jobs/debounce true})]
|
||||
(t/is (uuid? id))
|
||||
(t/is (= 1 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job
|
||||
WHERE label = ?" label]))))))
|
||||
|
||||
(t/deftest submit-debounce-inserts-past-started-rows
|
||||
(doseq [status ["scheduled" "running" "retry"]]
|
||||
(t/testing (str "started status " status)
|
||||
(let [cfg (make-cfg (get-job-defs))
|
||||
params (make-params)
|
||||
label (str "debounce-started-" status)
|
||||
id1 (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label label
|
||||
::jobs/debounce true})
|
||||
_ (th/db-update! :job {:status status} {:id id1})
|
||||
id2 (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label label
|
||||
::jobs/debounce true})]
|
||||
(t/is (uuid? id2))
|
||||
(t/is (not= id1 id2))
|
||||
(t/is (= 2 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job
|
||||
WHERE label = ?" label]))))))))
|
||||
|
||||
(t/deftest submit-keyed-options-need-a-label-and-are-exclusive
|
||||
(let [cfg (make-cfg (get-job-defs))
|
||||
params (make-params)]
|
||||
(t/testing "dedupe"
|
||||
(t/is (thrown-with-msg? Exception #"dedupe requires"
|
||||
(jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/dedupe true})))
|
||||
(t/is (thrown-with-msg? Exception #"dedupe requires"
|
||||
(jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label " "
|
||||
::jobs/dedupe true}))))
|
||||
(t/testing "debounce"
|
||||
(t/is (thrown-with-msg? Exception #"debounce requires"
|
||||
(jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/debounce true}))))
|
||||
(t/testing "and they contradict each other"
|
||||
(t/is (thrown-with-msg? Exception #"cannot be used together"
|
||||
(jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label "both-label"
|
||||
::jobs/dedupe true
|
||||
::jobs/debounce true}))))
|
||||
(t/testing "and neither fits an exclusive submit"
|
||||
(t/is (thrown-with-msg? Exception #"cannot be combined"
|
||||
(jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label "both-label"
|
||||
::jobs/dedupe true
|
||||
::jobs/skip-if-active true}))))))
|
||||
|
||||
(t/deftest submit-skip-if-active-skips-live-jobs
|
||||
(doseq [status ["new" "scheduled" "running" "retry"]]
|
||||
@ -534,13 +671,14 @@
|
||||
(let [cfg (make-cfg (get-job-defs))
|
||||
params (make-params)
|
||||
label (str "skip-active-" status)
|
||||
id1 (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label label})
|
||||
id1 (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label label
|
||||
::jobs/skip-if-active true})
|
||||
_ (th/db-update! :job {:status status} {:id id1})
|
||||
id2 (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label label
|
||||
id2 (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label label
|
||||
::jobs/skip-if-active true})]
|
||||
(t/is (nil? id2))
|
||||
(t/testing "no row deleted, none inserted"
|
||||
@ -554,18 +692,109 @@
|
||||
(let [cfg (make-cfg (get-job-defs))
|
||||
params (make-params)
|
||||
label (str "skip-terminal-" status)
|
||||
id1 (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label label})
|
||||
id1 (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label label
|
||||
::jobs/skip-if-active true})
|
||||
_ (th/db-update! :job {:status status} {:id id1})
|
||||
id2 (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label label
|
||||
id2 (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label label
|
||||
::jobs/skip-if-active true})]
|
||||
(t/is (uuid? id2))
|
||||
(t/is (= 2 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job
|
||||
WHERE label = ?" label]))))))))
|
||||
|
||||
(t/deftest submit-skip-if-active-does-not-block-on-plain-jobs
|
||||
;; The exclusion is between the jobs that asked for it: a live job
|
||||
;; that was not submitted as exclusive (a webhook delivery, a
|
||||
;; notification) does not block an exclusive submit. The label stays
|
||||
;; a label, and two plain jobs can share it.
|
||||
(let [cfg (make-cfg (get-job-defs))
|
||||
params (make-params)
|
||||
label "skip-over-plain"
|
||||
plain (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label label})
|
||||
excl (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label label
|
||||
::jobs/skip-if-active true})]
|
||||
(t/is (uuid? plain))
|
||||
(t/is (uuid? excl))
|
||||
(t/is (= 2 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job
|
||||
WHERE label = ?" label]))))
|
||||
(t/testing "and two plain jobs share the label too"
|
||||
(let [other (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label label})]
|
||||
(t/is (uuid? other))
|
||||
(t/is (= 3 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job
|
||||
WHERE label = ?" label]))))))))
|
||||
|
||||
(t/deftest exclusive-jobs-are-scoped-by-their-key
|
||||
;; The key of the exclusion is (tenant, name, queue, label): a job
|
||||
;; that differs in any of them is a different job and lives with the
|
||||
;; first one.
|
||||
(let [cfg (make-cfg (get-job-defs))
|
||||
params (make-params)
|
||||
base {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/skip-if-active true}]
|
||||
(t/is (uuid? (jobs/submit cfg (assoc base ::jobs/label "scope-a"))))
|
||||
(t/is (uuid? (jobs/submit cfg (assoc base ::jobs/label "scope-a"
|
||||
::jobs/queue :storage))))
|
||||
(t/is (uuid? (jobs/submit cfg (assoc base ::jobs/label "scope-a"
|
||||
::jobs/name :typed))))
|
||||
(t/is (nil? (jobs/submit cfg (assoc base ::jobs/label "scope-a"))))))
|
||||
|
||||
(t/deftest the-job-table-refuses-two-live-exclusive-rows
|
||||
;; The exclusion does not live in the submit: the unique index of the
|
||||
;; live exclusive jobs refuses the second row, whoever writes it.
|
||||
(let [cfg (make-cfg (get-job-defs))
|
||||
params (make-params)
|
||||
label "db-enforced"
|
||||
id1 (jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label label
|
||||
::jobs/skip-if-active true})
|
||||
row {:name "echo"
|
||||
:tenant (cf/get :tenant)
|
||||
:queue "default"
|
||||
:label label
|
||||
:params (db/json params)
|
||||
:priority 100
|
||||
:max-retries 3
|
||||
:retry-num 0
|
||||
:exclusive true
|
||||
:scheduled-at (ct/now)
|
||||
:created-at (ct/now)
|
||||
:modified-at (ct/now)}]
|
||||
(t/is (uuid? id1))
|
||||
(t/is (thrown? Exception
|
||||
(th/db-insert! :job (assoc row :id (uuid/next)))))
|
||||
(t/testing "and the row that was there is untouched"
|
||||
(t/is (some? (jobs/get-job cfg id1)))
|
||||
(t/is (= 1 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job
|
||||
WHERE label = ?" label])))))))
|
||||
|
||||
(t/deftest two-concurrent-exclusive-submits-leave-one-row
|
||||
;; The check and the insert are the same statement, so two backends
|
||||
;; submitting the same exclusive key at the same time cannot both
|
||||
;; win: the second one is rejected by the index.
|
||||
(let [cfg (make-cfg (get-job-defs))
|
||||
params (make-params)
|
||||
label "exclusive-race"
|
||||
submit #(jobs/submit cfg {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/label label
|
||||
::jobs/skip-if-active true})]
|
||||
(let [[a b] (race submit submit)]
|
||||
(t/is (= 1 (count (filter uuid? [a b])))
|
||||
"exactly one of the two submits got a job")
|
||||
(t/is (= 1 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job
|
||||
WHERE label = ?" label])))))))
|
||||
|
||||
(t/deftest submit-skip-if-active-requires-an-explicit-label
|
||||
(let [cfg (make-cfg (get-job-defs))
|
||||
params (make-params)]
|
||||
@ -591,55 +820,6 @@
|
||||
(t/is (= 1 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job
|
||||
WHERE label = ?" label]))))))
|
||||
|
||||
(t/deftest submit-dedupe-rolls-back-on-insert-failure
|
||||
(let [cfg (make-cfg (get-job-defs))
|
||||
params (make-params)
|
||||
opts {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/dedupe true
|
||||
::jobs/label "atomic-label"}
|
||||
kept (jobs/submit cfg opts)
|
||||
orig @#'db/exec-one!]
|
||||
;; fault the INSERT: the DELETE before it must roll back too
|
||||
(alter-var-root #'db/exec-one!
|
||||
(constantly (fn [& args]
|
||||
(when (insert-job-statement? args)
|
||||
(throw (ex-info "boom" {})))
|
||||
(apply orig args))))
|
||||
(try
|
||||
(t/is (thrown? Exception (jobs/submit cfg opts)))
|
||||
(t/testing "the original row survives, no duplicate left behind"
|
||||
(t/is (some? (jobs/get-job cfg kept)))
|
||||
(t/is (= 1 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job WHERE label = ?" "atomic-label"])))))
|
||||
(finally
|
||||
(alter-var-root #'db/exec-one! (constantly orig))))))
|
||||
|
||||
(t/deftest submit-dedupe-atomic-on-autocommit-conn
|
||||
(let [cfg (make-cfg (get-job-defs))
|
||||
params (make-params)
|
||||
opts {::jobs/name :echo
|
||||
::jobs/params params
|
||||
::jobs/dedupe true
|
||||
::jobs/label "atomic-autocommit-label"}
|
||||
kept (jobs/submit cfg opts)
|
||||
orig @#'db/exec-one!]
|
||||
;; a raw connection outside any transaction: DELETE+INSERT must
|
||||
;; still share one transaction opened on it
|
||||
(with-open [conn (db/open th/*pool*)]
|
||||
(alter-var-root #'db/exec-one!
|
||||
(constantly (fn [& args]
|
||||
(when (insert-job-statement? args)
|
||||
(throw (ex-info "boom" {})))
|
||||
(apply orig args))))
|
||||
(try
|
||||
(t/is (thrown? Exception
|
||||
(jobs/submit (assoc cfg ::db/conn conn) opts)))
|
||||
(t/testing "the original row survives, no duplicate left behind"
|
||||
(t/is (some? (jobs/get-job cfg kept)))
|
||||
(t/is (= 1 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job WHERE label = ?" "atomic-autocommit-label"])))))
|
||||
(finally
|
||||
(alter-var-root #'db/exec-one! (constantly orig)))))))
|
||||
|
||||
(t/deftest submit-requires-metrics-on-bare-connectable
|
||||
(let [defs (get-job-defs)
|
||||
prev @@#'jobs/defs-registry]
|
||||
@ -1683,20 +1863,6 @@
|
||||
;; on. Both are only meaningful when two callers arrive at once, so the
|
||||
;; tests below put real threads in front of them.
|
||||
|
||||
(defn- race
|
||||
"Run every thunk on a thread of its own and return what each returned,
|
||||
in the order they were given. They all wait on the same gate, so they
|
||||
contend for whatever they share instead of running one after another."
|
||||
[& thunks]
|
||||
(let [gate (promise)
|
||||
results (mapv (fn [f]
|
||||
(future
|
||||
@gate
|
||||
(f)))
|
||||
thunks)]
|
||||
(deliver gate true)
|
||||
(mapv deref results)))
|
||||
|
||||
(t/deftest the-first-terminal-transition-wins
|
||||
(let [cfg (make-cfg (get-job-defs))
|
||||
job-id (mk-running cfg)]
|
||||
|
||||
@ -1,837 +0,0 @@
|
||||
;; This Source Code Form is subject to the terms of the Mozilla Public
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns backend-tests.loggers-audit-consumer-task-test
|
||||
(:require
|
||||
[app.common.json :as json]
|
||||
[app.common.schema :as sm]
|
||||
[app.common.time :as ct]
|
||||
[app.common.transit :as transit]
|
||||
[app.common.uri :as u]
|
||||
[app.common.uuid :as uuid]
|
||||
[app.config :as cf]
|
||||
[app.db :as db]
|
||||
[app.http.client :as-alias http]
|
||||
[app.jobs :as jobs]
|
||||
[app.loggers.audit :as audit]
|
||||
[app.loggers.audit.consumer-task :as consumer]
|
||||
[app.main :as main]
|
||||
[app.metrics :as mtx]
|
||||
[app.metrics.definition :as-alias mdef]
|
||||
[app.setup :as-alias setup]
|
||||
[backend-tests.helpers :as th]
|
||||
[clojure.test :as t]
|
||||
[cuerdas.core :as str]
|
||||
[integrant.core :as ig]
|
||||
[mockery.core :refer [with-mocks]])
|
||||
(:import
|
||||
io.prometheus.client.Counter
|
||||
io.prometheus.client.Counter$Child))
|
||||
|
||||
(t/use-fixtures :once th/state-init)
|
||||
(t/use-fixtures :each th/database-reset)
|
||||
|
||||
(def ^:private archive-uri "http://nexus.example/archive")
|
||||
|
||||
(defn- insert-pendings!
|
||||
"Queue the row with the real producer helper: pendings for the
|
||||
enabled consumers, like append-audit-entry does."
|
||||
[row-id]
|
||||
(consumer/queue-pendings (:app.db/pool th/*system*) [row-id]))
|
||||
|
||||
(defn- pending-count
|
||||
[]
|
||||
(:pending (first (th/db-exec! ["SELECT count(*) AS pending FROM audit_log_pending"]))))
|
||||
|
||||
(defn- insert-audit-row!
|
||||
([profile-id name]
|
||||
(insert-audit-row! profile-id name {:team-id (uuid/next)}))
|
||||
([profile-id name props]
|
||||
(let [row (th/db-insert! :audit-log
|
||||
{:id (uuid/next)
|
||||
:name name
|
||||
:type "action"
|
||||
:source "backend"
|
||||
:profile-id profile-id
|
||||
:ip-addr (db/inet "127.0.0.1")
|
||||
:props (db/tjson props)
|
||||
:context (db/tjson {})
|
||||
:tracked-at (ct/now)
|
||||
:created-at (ct/now)})]
|
||||
(insert-pendings! (:id row))
|
||||
row)))
|
||||
|
||||
(defn- admin-console-event
|
||||
[admin-console-mock]
|
||||
(let [[_ _ params] (:call-args @admin-console-mock)]
|
||||
(first (:events params))))
|
||||
|
||||
(defn- admin-console-event-props
|
||||
[admin-console-mock]
|
||||
(:props (admin-console-event admin-console-mock)))
|
||||
|
||||
(defn- run-both!
|
||||
"Run both consumer jobs, Nexus first (the historical single-run
|
||||
order). The admin-console job takes no params (closed empty schema)."
|
||||
[params]
|
||||
(th/run-task! :audit-log-consumer-nexus params)
|
||||
(th/run-task! :audit-log-consumer-admin-console {}))
|
||||
|
||||
(t/deftest consumer-jobs-drain-their-own-queues
|
||||
;; Two separate jobs, one queue each: the nexus job sends without
|
||||
;; touching the admin console, and only the admin-console job
|
||||
;; deletes the last pending and the row. Create profile before
|
||||
;; enabling :admin-console — the system admin-console client is nil
|
||||
;; in tests, and team bootstrap would call it.
|
||||
(let [prof (th/create-profile* 1 {:is-active true})]
|
||||
(with-redefs [cf/flags #{:nexus :admin-console}]
|
||||
(with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}
|
||||
http-mock {:target 'app.http.client/req :return {:status 204}}]
|
||||
(insert-audit-row! (:id prof) "create-project")
|
||||
(insert-audit-row! (:id prof) "create-file")
|
||||
(th/run-task! :audit-log-consumer-nexus {:uri archive-uri})
|
||||
(t/is (false? (:called? @admin-console-mock)))
|
||||
(t/is (:called? @http-mock))
|
||||
(t/is (= 1 (:call-count @http-mock)))
|
||||
;; Nexus drained its own queue; both rows wait on the admin console.
|
||||
(t/is (= 2 (count (th/db-exec! ["select * from audit_log"]))))
|
||||
(t/is (= 2 (pending-count)))
|
||||
(th/run-task! :audit-log-consumer-admin-console {})
|
||||
(t/is (= 1 (:call-count @admin-console-mock)))
|
||||
(let [[_ method params] (:call-args @admin-console-mock)]
|
||||
(t/is (= :ingest-audit-log method))
|
||||
(t/is (= 2 (count (:events params))))
|
||||
(t/is (= #{"create-project" "create-file"}
|
||||
(into #{} (map :name) (:events params))))
|
||||
(t/is (every? uuid? (map :id (:events params)))))
|
||||
(let [rows (th/db-exec! ["select * from audit_log"])]
|
||||
(t/is (= 0 (count rows))))))))
|
||||
|
||||
(t/deftest consumer-events-filters-admin-console-event-names
|
||||
(let [prof (th/create-profile* 1 {:is-active true})]
|
||||
(with-redefs [cf/flags #{:nexus :admin-console}]
|
||||
(with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}
|
||||
http-mock {:target 'app.http.client/req :return {:status 204}}]
|
||||
(insert-audit-row! (:id prof) "create-project")
|
||||
(insert-audit-row! (:id prof) "create-organization-attribute")
|
||||
(insert-audit-row! (:id prof) "delete-organization-attribute")
|
||||
(insert-audit-row! (:id prof) "update-organization-attribute")
|
||||
(insert-audit-row! (:id prof) "auditlog-exported")
|
||||
(insert-audit-row! (:id prof) "unrelated-event")
|
||||
(run-both! {:uri archive-uri})
|
||||
(t/is (:called? @admin-console-mock))
|
||||
(t/is (:called? @http-mock))
|
||||
(let [[_ _ params] (:call-args @admin-console-mock)]
|
||||
(t/is (= #{"create-project"
|
||||
"create-organization-attribute"
|
||||
"delete-organization-attribute"
|
||||
"update-organization-attribute"
|
||||
"auditlog-exported"}
|
||||
(into #{} (map :name) (:events params)))))
|
||||
(let [rows (th/db-exec! ["select * from audit_log"])]
|
||||
(t/is (= 0 (count rows))))))))
|
||||
|
||||
(t/deftest consumer-events-skips-admin-console-when-no-matching-names
|
||||
(let [prof (th/create-profile* 1 {:is-active true})]
|
||||
(with-redefs [cf/flags #{:nexus :admin-console}]
|
||||
(with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}
|
||||
http-mock {:target 'app.http.client/req :return {:status 204}}]
|
||||
(insert-audit-row! (:id prof) "unrelated-event")
|
||||
(run-both! {:uri archive-uri})
|
||||
(t/is (false? (:called? @admin-console-mock)))
|
||||
(t/is (:called? @http-mock))
|
||||
(let [rows (th/db-exec! ["select * from audit_log"])]
|
||||
(t/is (= 0 (count rows))))))))
|
||||
|
||||
(t/deftest consumer-events-admin-console-only-marks-without-uri
|
||||
(let [prof (th/create-profile* 1 {:is-active true})]
|
||||
(with-redefs [cf/flags #{:admin-console}]
|
||||
(with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}
|
||||
http-mock {:target 'app.http.client/req :return {:status 204}}]
|
||||
(insert-audit-row! (:id prof) "create-project")
|
||||
(th/run-task! :audit-log-consumer-admin-console {})
|
||||
(t/is (:called? @admin-console-mock))
|
||||
(t/is (false? (:called? @http-mock)))
|
||||
(let [rows (th/db-exec! ["select * from audit_log"])]
|
||||
(t/is (= 0 (count rows))))))))
|
||||
|
||||
(t/deftest consumer-events-admin-console-only-marks-non-allowlisted
|
||||
(let [prof (th/create-profile* 1 {:is-active true})]
|
||||
(with-redefs [cf/flags #{:admin-console}]
|
||||
(with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}
|
||||
http-mock {:target 'app.http.client/req :return {:status 204}}]
|
||||
(insert-audit-row! (:id prof) "unrelated-event")
|
||||
(th/run-task! :audit-log-consumer-admin-console {})
|
||||
(t/is (false? (:called? @admin-console-mock)))
|
||||
(t/is (false? (:called? @http-mock)))
|
||||
(let [rows (th/db-exec! ["select * from audit_log"])]
|
||||
(t/is (= 0 (count rows))))))))
|
||||
|
||||
(t/deftest consumer-events-admin-console-error-still-drains-nexus
|
||||
;; Mirror of the nexus-throw test: an admin-console failure aborts
|
||||
;; only the admin-console job; Nexus already drained its own queue
|
||||
;; in its job. The row stays on the admin-console pending.
|
||||
(let [prof (th/create-profile* 1 {:is-active true})]
|
||||
(with-redefs [cf/flags #{:nexus :admin-console}]
|
||||
(with-mocks [admin-console-mock {:target 'app.nitrate/call
|
||||
:throw (ex-info "nitrate down" {})}
|
||||
http-mock {:target 'app.http.client/req :return {:status 204}}]
|
||||
(insert-audit-row! (:id prof) "create-project")
|
||||
(th/run-task! :audit-log-consumer-nexus {:uri archive-uri})
|
||||
(t/is (thrown? clojure.lang.ExceptionInfo
|
||||
(th/run-task! :audit-log-consumer-admin-console {})))
|
||||
;; Nexus already drained its queue; the admin-console failure no
|
||||
;; longer blocks it, but the row stays on the admin-console pending.
|
||||
(t/is (:called? @admin-console-mock))
|
||||
(t/is (:called? @http-mock))
|
||||
(let [rows (th/db-exec! ["select * from audit_log"])]
|
||||
(t/is (= 1 (count rows))))
|
||||
(t/is (= 1 (pending-count)))))))
|
||||
|
||||
(t/deftest consumer-events-uses-nexus-without-admin-console
|
||||
(let [prof (th/create-profile* 1 {:is-active true})]
|
||||
(with-redefs [cf/flags #{:nexus}]
|
||||
(with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}
|
||||
admin-console-mock {:target 'app.nitrate/call :return nil}]
|
||||
(insert-audit-row! (:id prof) "create-project")
|
||||
(th/run-task! :audit-log-consumer-nexus {:uri archive-uri})
|
||||
(t/is (false? (:called? @admin-console-mock)))
|
||||
(t/is (:called? @http-mock))
|
||||
(let [rows (th/db-exec! ["select * from audit_log"])]
|
||||
(t/is (= 0 (count rows))))))))
|
||||
|
||||
(t/deftest consumer-events-skips-mark-when-nexus-fails
|
||||
(let [prof (th/create-profile* 1 {:is-active true})]
|
||||
(with-redefs [cf/flags #{:nexus}]
|
||||
(with-mocks [http-mock {:target 'app.http.client/req :return {:status 500}}]
|
||||
(insert-audit-row! (:id prof) "create-project")
|
||||
(t/is (thrown? clojure.lang.ExceptionInfo
|
||||
(th/run-task! :audit-log-consumer-nexus {:uri archive-uri})))
|
||||
(t/is (:called? @http-mock))
|
||||
(let [rows (th/db-exec! ["select * from audit_log"])]
|
||||
(t/is (= 1 (count rows))))))))
|
||||
|
||||
(t/deftest consumer-events-both-flags-skips-mark-when-nexus-fails
|
||||
(let [prof (th/create-profile* 1 {:is-active true})]
|
||||
(with-redefs [cf/flags #{:nexus :admin-console}]
|
||||
(with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}
|
||||
http-mock {:target 'app.http.client/req :return {:status 500}}]
|
||||
(insert-audit-row! (:id prof) "create-project")
|
||||
(t/is (thrown? clojure.lang.ExceptionInfo
|
||||
(th/run-task! :audit-log-consumer-nexus {:uri archive-uri})))
|
||||
(th/run-task! :audit-log-consumer-admin-console {})
|
||||
(t/is (:called? @admin-console-mock))
|
||||
(t/is (:called? @http-mock))
|
||||
(let [rows (th/db-exec! ["select * from audit_log"])]
|
||||
(t/is (= 1 (count rows))))))))
|
||||
|
||||
(t/deftest consumer-events-encodes-db-rows-for-admin-console
|
||||
(let [prof (th/create-profile* 1 {:is-active true})]
|
||||
(with-redefs [cf/flags #{:nexus :admin-console}
|
||||
cf/config (assoc cf/config
|
||||
:admin-console-uri "http://ac.example/admin-console/"
|
||||
:admin-console-shared-key "ac-key"
|
||||
:nexus-shared-key "nexus-key")]
|
||||
(let [admin-console-bodies (atom [])
|
||||
shared-keys {:admin-console "ac-key" :nexus "nexus-key"}
|
||||
client (ig/init-key :app.nitrate/client
|
||||
{::http/client (Object.)
|
||||
::setup/shared-keys shared-keys})
|
||||
;; Each consumer job is a job-def: the handler closes over the
|
||||
;; cfg it was given at init and takes a nil context, because
|
||||
;; an in-process invocation has no job row to describe.
|
||||
nexus-job-def (ig/init-key :app.loggers.audit.consumer-task/nexus-job-def
|
||||
{::db/pool (:app.db/pool th/*system*)
|
||||
::setup/shared-keys shared-keys
|
||||
::http/client (:app.http.client/client th/*system*)
|
||||
::mtx/metrics (th/metrics-stub)
|
||||
:app.nitrate/client client})
|
||||
admin-console-job-def (ig/init-key :app.loggers.audit.consumer-task/admin-console-job-def
|
||||
{::db/pool (:app.db/pool th/*system*)
|
||||
::mtx/metrics (th/metrics-stub)
|
||||
:app.nitrate/client client})]
|
||||
(with-mocks [http-mock {:target 'app.http.client/req
|
||||
:return
|
||||
(fn [_ req & _]
|
||||
(when (str/includes? (str (:uri req)) "api/audit-log")
|
||||
(swap! admin-console-bodies conj (:body req)))
|
||||
{:status 204})}]
|
||||
(let [team-id (uuid/next)
|
||||
row (th/db-insert! :audit-log
|
||||
{:id (uuid/next)
|
||||
:name "create-project"
|
||||
:type "action"
|
||||
:source "backend"
|
||||
:profile-id (:id prof)
|
||||
:ip-addr (db/inet "127.0.0.1")
|
||||
:props (db/tjson {:team-id team-id})
|
||||
:context (db/tjson {})
|
||||
:tracked-at (ct/now)
|
||||
:created-at (ct/now)})]
|
||||
(insert-pendings! (:id row))
|
||||
;; Admin console first so its body is captured before Nexus
|
||||
;; deletes the last pending and the row.
|
||||
((::jobs/handler admin-console-job-def) nil {})
|
||||
((::jobs/handler nexus-job-def) nil {:uri archive-uri})
|
||||
(t/is (= 1 (count @admin-console-bodies)))
|
||||
(let [body (json/decode (first @admin-console-bodies) :key-fn json/read-kebab-key)
|
||||
event (first (:events body))]
|
||||
(t/is (= 1 (count (:events body))))
|
||||
(t/is (= (str (:id row)) (str (:id event))))
|
||||
(t/is (= "create-project" (:name event)))
|
||||
(t/is (= "127.0.0.1" (:ip-addr event)))
|
||||
(t/is (string? (:created-at event)))
|
||||
(t/is (string? (:tracked-at event)))
|
||||
(t/is (= (str team-id) (str (get-in event [:props :team-id])))))
|
||||
(let [rows (th/db-exec! ["select * from audit_log"])]
|
||||
(t/is (= 0 (count rows))))))))))
|
||||
|
||||
(t/deftest consumer-nexus-default-uri-appends-api-audit
|
||||
;; Without an explicit :uri param, the job posts to the Nexus base
|
||||
;; URI plus the ingest path, mirroring generate-nitrate-uri.
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
uris (atom [])]
|
||||
(with-redefs [cf/flags #{:nexus}
|
||||
cf/config (assoc cf/config :nexus-uri (u/uri "http://nexus.example"))]
|
||||
(with-mocks [http-mock {:target 'app.http.client/req
|
||||
:return (fn [_ params & _]
|
||||
(swap! uris conj (str (:uri params)))
|
||||
{:status 204})}]
|
||||
(insert-audit-row! (:id prof) "create-project")
|
||||
(th/run-task! :audit-log-consumer-nexus {})
|
||||
(t/is (= ["http://nexus.example/api/audit"] @uris))
|
||||
(t/is (= 0 (count (th/db-exec! ["select * from audit_log"]))))))))
|
||||
|
||||
(t/deftest consumer-nexus-job-requires-uri-when-nexus-flag-set
|
||||
(with-redefs [cf/flags #{:nexus}
|
||||
cf/config (dissoc cf/config :nexus-uri)]
|
||||
(let [data (ex-data (try
|
||||
(th/run-task! :audit-log-consumer-nexus {})
|
||||
(catch Throwable cause
|
||||
cause)))]
|
||||
(t/is (= :task-not-configured (:code data))))))
|
||||
|
||||
(t/deftest consumer-events-enriches-team-name-and-emails
|
||||
(let [actor (th/create-profile* 1 {:is-active true})
|
||||
member (th/create-profile* 2 {:is-active true})
|
||||
team (th/create-team* 1 {:profile-id (:id actor)})]
|
||||
(with-redefs [cf/flags #{:admin-console}]
|
||||
(with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}]
|
||||
(insert-audit-row! (:id actor) "create-project"
|
||||
{:team-id (:id team)
|
||||
:profile-id (:id actor)
|
||||
:user-id (:id member)
|
||||
:member-id (:id member)
|
||||
;; Producers store this as a string UUID.
|
||||
:user-who-send-invitation (str (:id actor))})
|
||||
(th/run-task! :audit-log-consumer-admin-console {})
|
||||
(t/is (:called? @admin-console-mock))
|
||||
(let [event (admin-console-event admin-console-mock)
|
||||
props (:props event)]
|
||||
(t/is (= (:name team) (:team-name props)))
|
||||
(t/is (= (:email actor) (:profile-email props)))
|
||||
(t/is (= (:email member) (:user-email props)))
|
||||
(t/is (= (:email member) (:member-email props)))
|
||||
(t/is (= (:email actor) (:user-who-send-invitation-email props)))
|
||||
;; Top-level actor fields (event :profile-id), distinct from props.
|
||||
(t/is (= (:email actor) (:profile-email event)))
|
||||
(t/is (= (:fullname actor) (:profile-name event))))
|
||||
(let [rows (th/db-exec! ["select * from audit_log"])]
|
||||
(t/is (= 0 (count rows))))))))
|
||||
|
||||
(t/deftest consumer-events-enriches-soft-deleted-team-and-profile
|
||||
(let [actor (th/create-profile* 1 {:is-active true})
|
||||
other (th/create-profile* 2 {:is-active true})
|
||||
team (th/create-team* 1 {:profile-id (:id actor)})]
|
||||
(th/db-update! :team {:deleted-at (ct/now)} {:id (:id team)})
|
||||
(th/db-update! :profile {:deleted-at (ct/now)} {:id (:id other)})
|
||||
(with-redefs [cf/flags #{:admin-console}]
|
||||
(with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}]
|
||||
(insert-audit-row! (:id actor) "create-project"
|
||||
{:team-id (:id team)
|
||||
:profile-id (:id other)})
|
||||
(th/run-task! :audit-log-consumer-admin-console {})
|
||||
(let [event (admin-console-event admin-console-mock)
|
||||
props (:props event)]
|
||||
(t/is (= (:name team) (:team-name props)))
|
||||
(t/is (= (:email other) (:profile-email props)))
|
||||
(t/is (= (:email actor) (:profile-email event)))
|
||||
(t/is (= (:fullname actor) (:profile-name event))))
|
||||
(let [rows (th/db-exec! ["select * from audit_log"])]
|
||||
(t/is (= 0 (count rows))))))))
|
||||
|
||||
(t/deftest consumer-events-enriches-soft-deleted-actor-profile
|
||||
(let [actor (th/create-profile* 1 {:is-active true})
|
||||
team (th/create-team* 1 {:profile-id (:id actor)})]
|
||||
(th/db-update! :profile {:deleted-at (ct/now)} {:id (:id actor)})
|
||||
(with-redefs [cf/flags #{:admin-console}]
|
||||
(with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}]
|
||||
(insert-audit-row! (:id actor) "create-project"
|
||||
{:team-id (:id team)})
|
||||
(th/run-task! :audit-log-consumer-admin-console {})
|
||||
(let [event (admin-console-event admin-console-mock)]
|
||||
(t/is (= (:email actor) (:profile-email event)))
|
||||
(t/is (= (:fullname actor) (:profile-name event))))))))
|
||||
|
||||
(t/deftest consumer-events-omits-actor-profile-when-no-usable-id
|
||||
(let [actor (th/create-profile* 1 {:is-active true})
|
||||
team (th/create-team* 1 {:profile-id (:id actor)})
|
||||
missing (uuid/next)]
|
||||
(with-redefs [cf/flags #{:admin-console}]
|
||||
(with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}]
|
||||
(insert-audit-row! uuid/zero "create-project" {:team-id (:id team)})
|
||||
(insert-audit-row! missing "create-team" {:team-id (:id team)})
|
||||
(th/run-task! :audit-log-consumer-admin-console {})
|
||||
(t/is (:called? @admin-console-mock))
|
||||
(let [[_ _ params] (:call-args @admin-console-mock)
|
||||
events (:events params)
|
||||
by-name (into {} (map (juxt :name identity) events))
|
||||
zero-evt (get by-name "create-project")
|
||||
missing-evt (get by-name "create-team")]
|
||||
(t/is (not (contains? zero-evt :profile-email)))
|
||||
(t/is (not (contains? zero-evt :profile-name)))
|
||||
(t/is (not (contains? missing-evt :profile-email)))
|
||||
(t/is (not (contains? missing-evt :profile-name))))))))
|
||||
|
||||
(t/deftest consumer-events-does-not-overwrite-existing-enrich-props
|
||||
(let [actor (th/create-profile* 1 {:is-active true})
|
||||
team (th/create-team* 1 {:profile-id (:id actor)})]
|
||||
(with-redefs [cf/flags #{:admin-console}]
|
||||
(with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}]
|
||||
(insert-audit-row! (:id actor) "create-project"
|
||||
{:team-id (:id team)
|
||||
:team-name "already-set-name"
|
||||
:profile-id (:id actor)
|
||||
:profile-email "already-set@example.com"
|
||||
:user-id (:id actor)
|
||||
:user-email "already-user@example.com"
|
||||
:member-id (:id actor)
|
||||
:member-email "already-member@example.com"
|
||||
:user-who-send-invitation (str (:id actor))
|
||||
:user-who-send-invitation-email "already-inviter@example.com"})
|
||||
(th/run-task! :audit-log-consumer-admin-console {})
|
||||
(let [props (admin-console-event-props admin-console-mock)]
|
||||
(t/is (= "already-set-name" (:team-name props)))
|
||||
(t/is (= "already-set@example.com" (:profile-email props)))
|
||||
(t/is (= "already-user@example.com" (:user-email props)))
|
||||
(t/is (= "already-member@example.com" (:member-email props)))
|
||||
(t/is (= "already-inviter@example.com"
|
||||
(:user-who-send-invitation-email props))))))))
|
||||
|
||||
(t/deftest consumer-events-does-not-overwrite-existing-actor-profile-fields
|
||||
;; Top-level :profile-email / :profile-name are not stored on the row;
|
||||
;; exercise fill-only via the enrich helper directly.
|
||||
(let [actor (th/create-profile* 1 {:is-active true})
|
||||
event {:profile-id (:id actor)
|
||||
:profile-email "preset@example.com"
|
||||
:profile-name "Preset Name"
|
||||
:props {}}]
|
||||
(db/tx-run! th/*system*
|
||||
(fn [cfg]
|
||||
(let [enriched (first (#'consumer/enrich-events-for-admin-console
|
||||
cfg [event]))]
|
||||
(t/is (= "preset@example.com" (:profile-email enriched)))
|
||||
(t/is (= "Preset Name" (:profile-name enriched))))))))
|
||||
|
||||
(t/deftest consumer-events-fills-missing-actor-profile-field-independently
|
||||
;; Each top-level key fills on its own; guards an or→and regression in
|
||||
;; collect-actor-ids-needing-profile.
|
||||
(let [actor (th/create-profile* 1 {:is-active true})]
|
||||
(db/tx-run! th/*system*
|
||||
(fn [cfg]
|
||||
(let [email-only (first (#'consumer/enrich-events-for-admin-console
|
||||
cfg [{:profile-id (:id actor)
|
||||
:profile-email "preset@example.com"
|
||||
:props {}}]))
|
||||
name-only (first (#'consumer/enrich-events-for-admin-console
|
||||
cfg [{:profile-id (:id actor)
|
||||
:profile-name "Preset Name"
|
||||
:props {}}]))]
|
||||
(t/is (= "preset@example.com" (:profile-email email-only)))
|
||||
(t/is (= (:fullname actor) (:profile-name email-only)))
|
||||
(t/is (= (:email actor) (:profile-email name-only)))
|
||||
(t/is (= "Preset Name" (:profile-name name-only))))))))
|
||||
|
||||
(t/deftest consumer-events-omits-empty-actor-profile-name
|
||||
;; profile.fullname is NOT NULL DEFAULT ''; empty must not become
|
||||
;; :profile-name "".
|
||||
(let [actor (th/create-profile* 1 {:is-active true})
|
||||
team (th/create-team* 1 {:profile-id (:id actor)})]
|
||||
(th/db-update! :profile {:fullname ""} {:id (:id actor)})
|
||||
(with-redefs [cf/flags #{:admin-console}]
|
||||
(with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}]
|
||||
(insert-audit-row! (:id actor) "create-project" {:team-id (:id team)})
|
||||
(th/run-task! :audit-log-consumer-admin-console {})
|
||||
(let [event (admin-console-event admin-console-mock)]
|
||||
(t/is (= (:email actor) (:profile-email event)))
|
||||
(t/is (not (contains? event :profile-name))))))))
|
||||
|
||||
(t/deftest consumer-events-enrich-skips-missing-and-non-uuid-ids
|
||||
(let [actor (th/create-profile* 1 {:is-active true})]
|
||||
(with-redefs [cf/flags #{:admin-console}]
|
||||
(with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}]
|
||||
(insert-audit-row! (:id actor) "create-project"
|
||||
{:team-id (uuid/next)
|
||||
:profile-id (uuid/next)
|
||||
:user-id "not-a-uuid"
|
||||
:member-id 42})
|
||||
(th/run-task! :audit-log-consumer-admin-console {})
|
||||
(t/is (:called? @admin-console-mock))
|
||||
(let [props (admin-console-event-props admin-console-mock)]
|
||||
(t/is (nil? (:team-name props)))
|
||||
(t/is (nil? (:profile-email props)))
|
||||
(t/is (nil? (:user-email props)))
|
||||
(t/is (nil? (:member-email props)))
|
||||
(t/is (= "not-a-uuid" (:user-id props)))
|
||||
(t/is (= 42 (:member-id props))))
|
||||
(let [rows (th/db-exec! ["select * from audit_log"])]
|
||||
(t/is (= 0 (count rows))))))))
|
||||
|
||||
(t/deftest consumer-events-forwards-export-job-not-legacy-export
|
||||
(let [actor (th/create-profile* 1 {:is-active true})
|
||||
team (th/create-team* 1 {:profile-id (:id actor)})]
|
||||
(with-redefs [cf/flags #{:admin-console}]
|
||||
(with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}]
|
||||
(insert-audit-row! (:id actor) "create-export-binfile-job"
|
||||
{:team-id (:id team)
|
||||
:job-id (uuid/next)
|
||||
:files 1
|
||||
:export-type "detach-libraries"})
|
||||
(insert-audit-row! (:id actor) "export-binfile"
|
||||
{:file-id (uuid/next)})
|
||||
(th/run-task! :audit-log-consumer-admin-console {})
|
||||
(let [[_ _ params] (:call-args @admin-console-mock)
|
||||
events (:events params)
|
||||
props (:props (first events))]
|
||||
(t/is (= ["create-export-binfile-job"] (mapv :name events)))
|
||||
(t/is (= (:id team) (:team-id props)))
|
||||
(t/is (= (:name team) (:team-name props))))
|
||||
(let [rows (th/db-exec! ["select * from audit_log"])]
|
||||
(t/is (= 0 (count rows))))))))
|
||||
|
||||
(t/deftest consumer-params-accept-empty-cron-props
|
||||
;; Cron always submits (or props {}) — system deps live on the
|
||||
;; job-def cfg, not in params. Requiring them here blocks every tick.
|
||||
;; Both schemas are closed: a stale manual invocation must fail
|
||||
;; validation loudly instead of skipping silently.
|
||||
(let [nexus-valid? (sm/validator consumer/schema:audit-log-consumer-nexus-params)
|
||||
admin-console-valid? (sm/validator consumer/schema:audit-log-consumer-admin-console-params)]
|
||||
(t/is (true? (nexus-valid? {})))
|
||||
(t/is (false? (nexus-valid? {:enabled true})))
|
||||
(t/is (true? (nexus-valid? {:uri (u/uri archive-uri)})))
|
||||
(t/is (true? (admin-console-valid? {})))
|
||||
(t/is (false? (admin-console-valid? {:enabled true})))
|
||||
(t/is (false? (admin-console-valid? {:uri (u/uri archive-uri)})))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; METRICS SMOKE TEST
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
;; The job-def cfg of these tests carries an isolated metrics registry
|
||||
;; (pattern from `jobs-metrics-test`), so assertions read exact values
|
||||
;; instead of the shared test-system registry. `th/run-task!` would use
|
||||
;; the latter and its counts would leak across tests.
|
||||
(def ^:private consumer-metric-ids
|
||||
[:audit-consumer-events :audit-consumer-discarded])
|
||||
|
||||
(defn- make-consumer-metrics
|
||||
[]
|
||||
(ig/init-key :app.metrics/metrics
|
||||
{:default (select-keys main/default-metrics consumer-metric-ids)}))
|
||||
|
||||
(defn- invoke-nexus!
|
||||
"Invoke the nexus consumer handler in-process with an isolated
|
||||
metrics registry. The handler takes a nil context because there is
|
||||
no job row to describe (same as `th/run-task!`)."
|
||||
[metrics params]
|
||||
(let [job-def (ig/init-key :app.loggers.audit.consumer-task/nexus-job-def
|
||||
{::db/pool (:app.db/pool th/*system*)
|
||||
::setup/shared-keys {:admin-console "ac-key" :nexus "nexus-key"}
|
||||
::http/client (:app.http.client/client th/*system*)
|
||||
::mtx/metrics metrics})]
|
||||
((::jobs/handler job-def) nil params)))
|
||||
|
||||
(defn- invoke-admin-console!
|
||||
"Invoke the admin-console consumer handler in-process with an
|
||||
isolated metrics registry. The admin-console client is the (nil in
|
||||
tests) system one: tests mock `app.nitrate/call`, so it is never
|
||||
dereferenced."
|
||||
[metrics]
|
||||
(let [job-def (ig/init-key :app.loggers.audit.consumer-task/admin-console-job-def
|
||||
{::db/pool (:app.db/pool th/*system*)
|
||||
:app.nitrate/client (:app.nitrate/client th/*system*)
|
||||
::mtx/metrics metrics})]
|
||||
((::jobs/handler job-def) nil {})))
|
||||
|
||||
(defn- invoke-both!
|
||||
"Invoke both consumer handlers sharing one isolated metrics
|
||||
registry, Nexus first (the historical single-run order)."
|
||||
[metrics params]
|
||||
(invoke-nexus! metrics params)
|
||||
(invoke-admin-console! metrics))
|
||||
|
||||
(defn- counter-value
|
||||
[metrics id labels]
|
||||
(let [collector (mtx/get-collector metrics id)
|
||||
instance (::mdef/instance collector)
|
||||
child (.labels ^Counter instance (into-array String labels))]
|
||||
(.get ^Counter$Child child)))
|
||||
|
||||
(t/deftest consumer-records-metrics-on-success
|
||||
;; Smoke test: the run records its domain metrics into the wired
|
||||
;; registry (isolated per test, pattern from `jobs-metrics-test`).
|
||||
;; Logging itself is not asserted: `dbg` summaries run below the
|
||||
;; test log level and failure `warn`s only repeat the outcome the
|
||||
;; metrics already prove.
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
metrics (make-consumer-metrics)]
|
||||
(with-redefs [cf/flags #{:nexus :admin-console}]
|
||||
(with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}
|
||||
http-mock {:target 'app.http.client/req :return {:status 204}}]
|
||||
(insert-audit-row! (:id prof) "create-project")
|
||||
(insert-audit-row! (:id prof) "unrelated-event")
|
||||
(invoke-both! metrics {:uri archive-uri})
|
||||
(t/is (= 2.0 (counter-value metrics :audit-consumer-events ["nexus" "archived"])))
|
||||
(t/is (= 1.0 (counter-value metrics :audit-consumer-events ["admin-console" "archived"])))
|
||||
(t/is (= 1.0 (counter-value metrics :audit-consumer-events ["admin-console" "skipped"])))
|
||||
(t/is (= 0 (pending-count)))
|
||||
(let [rows (th/db-exec! ["select * from audit_log"])]
|
||||
(t/is (= 0 (count rows))))))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; PENDING-QUEUE TESTS
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(defn- full-event
|
||||
"A complete event for submit*, which (unlike submit) assumes every
|
||||
required field is filled."
|
||||
[profile-id]
|
||||
{:name "create-file" :type "action" :source "backend"
|
||||
:profile-id profile-id :ip-addr "127.0.0.1"
|
||||
:props {} :context {}
|
||||
:created-at (ct/now) :tracked-at (ct/now)})
|
||||
|
||||
(t/deftest consumer-submit-creates-pendings-for-enabled-consumers
|
||||
;; The producer hook: a row is born pending for the enabled
|
||||
;; consumers, in the same transaction as the row itself. Goes
|
||||
;; through submit* because the suite stubs submit itself (see
|
||||
;; helpers/init-config).
|
||||
(let [prof (th/create-profile* 1 {:is-active true})]
|
||||
(with-redefs [cf/flags #{:audit-log :nexus :admin-console}]
|
||||
(audit/submit* {::db/pool (:app.db/pool th/*system*)}
|
||||
(full-event (:id prof)))
|
||||
(t/is (= 1 (count (th/db-exec! ["select * from audit_log"]))))
|
||||
(t/is (= 2 (pending-count))))))
|
||||
|
||||
(t/deftest consumer-submit-single-flag-creates-one-pending
|
||||
(let [prof (th/create-profile* 1 {:is-active true})]
|
||||
(with-redefs [cf/flags #{:audit-log :nexus}]
|
||||
(audit/submit* {::db/pool (:app.db/pool th/*system*)}
|
||||
(full-event (:id prof)))
|
||||
(t/is (= 1 (count (th/db-exec! ["select * from audit_log"]))))
|
||||
(t/is (= 1 (pending-count))))))
|
||||
|
||||
(t/deftest consumer-submit-rollback-keeps-no-pendings
|
||||
;; A rollback takes the row and its queue together: a row never
|
||||
;; outlives its pendings, and no phantom pending pins nothing.
|
||||
(let [prof (th/create-profile* 1 {:is-active true})]
|
||||
(with-redefs [cf/flags #{:audit-log :nexus :admin-console}]
|
||||
(db/tx-run! {::db/pool (:app.db/pool th/*system*)
|
||||
::db/rollback true}
|
||||
(fn [cfg]
|
||||
(audit/submit* cfg (full-event (:id prof)))
|
||||
(t/is (= 1 (count (db/exec! cfg ["select * from audit_log"]))))
|
||||
(t/is (= 2 (count (db/exec! cfg ["select * from audit_log_pending"]))))))
|
||||
(t/is (= 0 (count (th/db-exec! ["select * from audit_log"]))))
|
||||
(t/is (= 0 (pending-count))))))
|
||||
|
||||
(t/deftest consumer-keeps-row-until-last-consumer-done
|
||||
;; Nexus refuses the chunk: the nexus job aborts (failed count
|
||||
;; recorded, isolated registry) while the admin-console job still
|
||||
;; drains its own queue. The row and the nexus pending stay; once
|
||||
;; nexus accepts, everything is gone.
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
metrics (make-consumer-metrics)]
|
||||
(with-redefs [cf/flags #{:nexus :admin-console}]
|
||||
(insert-audit-row! (:id prof) "create-project")
|
||||
(with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}
|
||||
http-mock {:target 'app.http.client/req :return {:status 500}}]
|
||||
(let [err (try
|
||||
(invoke-nexus! metrics {:uri archive-uri})
|
||||
nil
|
||||
(catch clojure.lang.ExceptionInfo cause cause))]
|
||||
(t/is (= :consumer-chunk-refused (:code (ex-data err)))))
|
||||
(invoke-admin-console! metrics)
|
||||
(t/is (:called? @admin-console-mock))
|
||||
(t/is (= 1.0 (counter-value metrics :audit-consumer-events ["nexus" "failed"])))
|
||||
(t/is (= 1 (count (th/db-exec! ["select * from audit_log"]))))
|
||||
(t/is (= 1 (pending-count))))
|
||||
(with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}
|
||||
http-mock {:target 'app.http.client/req :return {:status 204}}]
|
||||
(run-both! {:uri archive-uri})
|
||||
(t/is (= 0 (count (th/db-exec! ["select * from audit_log"]))))
|
||||
(t/is (= 0 (pending-count)))))))
|
||||
|
||||
(t/deftest consumer-sweeps-orphan-rows
|
||||
;; A row with no pendings (old archived leftover, or born while no
|
||||
;; consumer was on) is deleted without any send.
|
||||
(let [prof (th/create-profile* 1 {:is-active true})]
|
||||
(with-redefs [cf/flags #{:nexus :admin-console}]
|
||||
(with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}
|
||||
http-mock {:target 'app.http.client/req :return {:status 204}}]
|
||||
;; Raw insert, no pendings on purpose.
|
||||
(th/db-insert! :audit-log
|
||||
{:id (uuid/next)
|
||||
:name "create-project"
|
||||
:type "action"
|
||||
:source "backend"
|
||||
:profile-id (:id prof)
|
||||
:ip-addr (db/inet "127.0.0.1")
|
||||
:props (db/tjson {})
|
||||
:context (db/tjson {})
|
||||
:tracked-at (ct/now)
|
||||
:created-at (ct/now)})
|
||||
(run-both! {:uri archive-uri})
|
||||
(t/is (false? (:called? @admin-console-mock)))
|
||||
(t/is (false? (:called? @http-mock)))
|
||||
(t/is (= 0 (count (th/db-exec! ["select * from audit_log"]))))))))
|
||||
|
||||
(t/deftest consumer-nexus-transport-failure-records-failed
|
||||
;; A transport exception (timeout, DNS, refused) counts the events
|
||||
;; as failed and aborts the nexus job: the chunk rolls back and
|
||||
;; the rows stay for the next tick.
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
metrics (make-consumer-metrics)]
|
||||
(with-redefs [cf/flags #{:nexus}]
|
||||
(with-mocks [http-mock {:target 'app.http.client/req
|
||||
:throw (ex-info "timeout" {})}]
|
||||
(insert-audit-row! (:id prof) "create-project")
|
||||
(t/is (thrown? clojure.lang.ExceptionInfo
|
||||
(invoke-nexus! metrics {:uri archive-uri})))
|
||||
(t/is (= 1.0 (counter-value metrics :audit-consumer-events ["nexus" "failed"])))
|
||||
(t/is (= 1 (count (th/db-exec! ["select * from audit_log"]))))
|
||||
(t/is (= 1 (pending-count)))))))
|
||||
|
||||
(t/deftest consumer-sends-chunks-in-chronological-order
|
||||
;; Nexus numbers by arrival for order-sensitive consumers, so
|
||||
;; multi-chunk drains must arrive oldest-first no matter how the
|
||||
;; queue is scanned.
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
base (ct/now)
|
||||
bodies (atom [])]
|
||||
(with-redefs [cf/flags #{:nexus}]
|
||||
(with-mocks [http-mock {:target 'app.http.client/req
|
||||
:return (fn [_ params & _]
|
||||
(swap! bodies conj (:body params))
|
||||
{:status 204})}]
|
||||
(doseq [i (range 130)]
|
||||
(let [row (insert-audit-row! (:id prof) (str "event-" i))]
|
||||
(th/db-update! :audit-log
|
||||
{:created-at (.plusSeconds ^java.time.Instant base (long i))}
|
||||
{:id (:id row)})))
|
||||
(th/run-task! :audit-log-consumer-nexus {:uri archive-uri})
|
||||
(let [received (mapcat #(:events (transit/decode %)) @bodies)
|
||||
times (map :created-at received)]
|
||||
(t/is (= 130 (count received)))
|
||||
(t/is (= (sort times) times))
|
||||
(t/is (= 0 (count (th/db-exec! ["select * from audit_log"])))))))))
|
||||
|
||||
(t/deftest consumer-reaps-disabled-consumer-pendings
|
||||
;; R3 reap by its real path: an admin-console pending born while
|
||||
;; admin-console was on is dropped once it is off; nexus still
|
||||
;; archives its queue and the row is gone with nothing stuck. The
|
||||
;; dropped pending counts as discarded (isolated registry).
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
metrics (make-consumer-metrics)]
|
||||
(with-redefs [cf/flags #{:nexus}]
|
||||
(with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}
|
||||
admin-console-mock {:target 'app.nitrate/call :return nil}]
|
||||
(let [row (insert-audit-row! (:id prof) "create-project")]
|
||||
;; Admin console was on at insert time: both pendings on purpose.
|
||||
(th/db-exec! ["INSERT INTO audit_log_pending (audit_log_id, consumer)
|
||||
VALUES (?::uuid, 2)
|
||||
ON CONFLICT DO NOTHING"
|
||||
(:id row)])
|
||||
(t/is (= 2 (pending-count)))
|
||||
(invoke-nexus! metrics {:uri archive-uri})
|
||||
(t/is (:called? @http-mock))
|
||||
(t/is (false? (:called? @admin-console-mock)))
|
||||
(t/is (= 1.0 (counter-value metrics :audit-consumer-discarded ["reaped"])))
|
||||
(t/is (= 0 (count (th/db-exec! ["select * from audit_log"]))))
|
||||
(t/is (= 0 (pending-count))))))))
|
||||
|
||||
(t/deftest consumer-sweeps-rows-born-with-no-consumer-on
|
||||
;; Accepted R3 consequence by its real path (not a hand-made
|
||||
;; orphan): ingested with no send flag on, the row is swept
|
||||
;; unarchived once the task runs, with zero sends.
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
metrics (make-consumer-metrics)]
|
||||
(with-redefs [cf/flags #{:audit-log}]
|
||||
(audit/submit* {::db/pool (:app.db/pool th/*system*)}
|
||||
(full-event (:id prof)))
|
||||
(t/is (= 1 (count (th/db-exec! ["select * from audit_log"]))))
|
||||
(t/is (= 0 (pending-count))))
|
||||
(with-redefs [cf/flags #{:audit-log :nexus}]
|
||||
(with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}]
|
||||
(invoke-nexus! metrics {:uri archive-uri})
|
||||
(t/is (false? (:called? @http-mock)))
|
||||
(t/is (= 0 (count (th/db-exec! ["select * from audit_log"]))))
|
||||
(t/is (= 0 (pending-count)))
|
||||
(t/is (= 1.0 (counter-value metrics :audit-consumer-discarded ["swept"])))
|
||||
(t/is (= 0.0 (counter-value metrics :audit-consumer-events ["nexus" "archived"])))))))
|
||||
|
||||
(t/deftest consumer-nexus-throw-still-drains-admin-console
|
||||
;; Consumer independence, now structural: a Nexus transport throw
|
||||
;; aborts only the nexus job, and the admin-console job still drains
|
||||
;; its own queue. Nexus keeps its pending (so the row stays); the
|
||||
;; admin console completes its own queue.
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
metrics (make-consumer-metrics)]
|
||||
(with-redefs [cf/flags #{:nexus :admin-console}]
|
||||
(with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}
|
||||
http-mock {:target 'app.http.client/req
|
||||
:throw (ex-info "timeout" {})}]
|
||||
(insert-audit-row! (:id prof) "create-project")
|
||||
(t/is (thrown? clojure.lang.ExceptionInfo
|
||||
(invoke-nexus! metrics {:uri archive-uri})))
|
||||
(invoke-admin-console! metrics)
|
||||
(t/is (:called? @admin-console-mock))
|
||||
(t/is (= 1.0 (counter-value metrics :audit-consumer-events ["nexus" "failed"])))
|
||||
(t/is (= 1 (count (th/db-exec! ["select * from audit_log"]))))
|
||||
(t/is (= 1 (pending-count)))))))
|
||||
|
||||
(t/deftest consumer-reap-then-sweeps-without-sends
|
||||
;; End to end of the reap path: a row born with only an
|
||||
;; admin-console pending is reaped once :admin-console goes off,
|
||||
;; then swept with zero sends on either destination.
|
||||
(let [prof (th/create-profile* 1 {:is-active true})]
|
||||
(with-redefs [cf/flags #{:admin-console}]
|
||||
(insert-audit-row! (:id prof) "create-project")
|
||||
(t/is (= 1 (pending-count))))
|
||||
(with-redefs [cf/flags #{:nexus}]
|
||||
(with-mocks [http-mock {:target 'app.http.client/req :return {:status 204}}
|
||||
admin-console-mock {:target 'app.nitrate/call :return nil}]
|
||||
(th/run-task! :audit-log-consumer-nexus {:uri archive-uri})
|
||||
(t/is (false? (:called? @http-mock)))
|
||||
(t/is (false? (:called? @admin-console-mock)))
|
||||
(t/is (= 0 (count (th/db-exec! ["select * from audit_log"]))))
|
||||
(t/is (= 0 (pending-count)))))))
|
||||
|
||||
(t/deftest consumer-fully-filtered-admin-console-chunk-counts-skipped
|
||||
;; An admin-console chunk with every event outside the allowlist
|
||||
;; never calls ingest, but still counts its rows as skipped and
|
||||
;; completes its queue.
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
metrics (make-consumer-metrics)]
|
||||
(with-redefs [cf/flags #{:nexus :admin-console}]
|
||||
(with-mocks [admin-console-mock {:target 'app.nitrate/call :return nil}
|
||||
http-mock {:target 'app.http.client/req :return {:status 204}}]
|
||||
(insert-audit-row! (:id prof) "unrelated-event")
|
||||
(invoke-both! metrics {:uri archive-uri})
|
||||
(t/is (false? (:called? @admin-console-mock)))
|
||||
(t/is (= 1.0 (counter-value metrics :audit-consumer-events ["admin-console" "skipped"])))
|
||||
(t/is (= 0 (count (th/db-exec! ["select * from audit_log"]))))
|
||||
(t/is (= 0 (pending-count)))))))
|
||||
1896
backend/test/backend_tests/loggers_audit_consumers_test.clj
Normal file
1896
backend/test/backend_tests/loggers_audit_consumers_test.clj
Normal file
File diff suppressed because it is too large
Load Diff
@ -71,6 +71,8 @@
|
||||
"id" "name" "tenant" "queue" "label" "priority" "scheduled_at"
|
||||
"retry_num" "max_retries" "status" "created_at" "modified_at"
|
||||
"started_at" "completed_at" "params"
|
||||
; the jobs that resolved their key in the database
|
||||
"exclusive" "dedupe"
|
||||
; optional ledger columns
|
||||
"profile_id" "error" "result"
|
||||
"resource_id" "expires_at"}
|
||||
@ -147,7 +149,118 @@
|
||||
(t/deftest job-table-has-expected-indexes
|
||||
(t/is (contains? (table-indexes "job") "job__dispatcher__idx"))
|
||||
(t/is (contains? (table-indexes "job") "job__orphan__idx"))
|
||||
(t/is (contains? (table-indexes "job") "job__profile__idx")))
|
||||
(t/is (contains? (table-indexes "job") "job__profile__idx"))
|
||||
(t/is (contains? (table-indexes "job") "job__live_exclusive__uniq"))
|
||||
(t/is (contains? (table-indexes "job") "job__new_dedupe__uniq")))
|
||||
|
||||
(t/deftest audit-log-has-a-consumer-cursor
|
||||
(t/testing "the position columns and their covering index exist"
|
||||
(t/is (contains? (table-columns "audit_log") "seq"))
|
||||
(t/is (contains? (table-columns "audit_log") "xid"))
|
||||
(t/is (contains? (table-indexes "audit_log") "audit_log__xid__seq__idx")))
|
||||
(t/testing "one row per consumer, with the position and its clock"
|
||||
(t/is (= #{"consumer" "xid" "seq" "updated_at"}
|
||||
(table-columns "audit_log_consumer_cursor")))))
|
||||
|
||||
(t/deftest the-migration-keeps-the-legacy-archive-mark-and-indexes
|
||||
;; The legacy mark and the indexes that served the old archiver stay in
|
||||
;; place: the old archiver may still be running during the rollout, and a
|
||||
;; later release removes them. The pending queue's `(created_at, id)`
|
||||
;; index is not built either.
|
||||
(th/db-exec! ["ALTER TABLE audit_log ADD COLUMN IF NOT EXISTS archived_at timestamptz"])
|
||||
(th/db-exec! ["DROP INDEX IF EXISTS audit_log__created_at__idx"])
|
||||
(th/db-exec! ["CREATE INDEX audit_log__created_at__idx
|
||||
ON audit_log (created_at) WHERE archived_at IS NULL"])
|
||||
(th/db-exec! ["CREATE INDEX IF NOT EXISTS audit_log__archived_at__idx
|
||||
ON audit_log (archived_at) WHERE archived_at IS NOT NULL"])
|
||||
(th/db-exec! ["CREATE INDEX IF NOT EXISTS audit_log_id_archived_at_idx
|
||||
ON audit_log (id, archived_at)"])
|
||||
(let [apply! (mg/resource "app/migrations/sql/0159-consumer-generalization.sql")]
|
||||
(t/is (true? (apply! th/*pool*))))
|
||||
(t/is (contains? (table-columns "audit_log") "archived_at"))
|
||||
(t/is (contains? (table-indexes "audit_log") "audit_log__created_at__idx"))
|
||||
(t/is (contains? (table-indexes "audit_log") "audit_log__archived_at__idx"))
|
||||
(t/is (contains? (table-indexes "audit_log") "audit_log_id_archived_at_idx"))
|
||||
(t/is (str/includes?
|
||||
(:indexdef (th/db-exec-one!
|
||||
["SELECT indexdef FROM pg_indexes
|
||||
WHERE schemaname = 'public'
|
||||
AND tablename = 'audit_log'
|
||||
AND indexname = 'audit_log__created_at__idx'"]))
|
||||
"WHERE (archived_at IS NULL)"))
|
||||
(t/is (not (some #(str/includes? % "(created_at, id)")
|
||||
(map :indexdef
|
||||
(th/db-exec! ["SELECT indexdef FROM pg_indexes
|
||||
WHERE schemaname = 'public'
|
||||
AND tablename = 'audit_log'"]))))))
|
||||
|
||||
(t/deftest the-migration-drops-the-old-queue-table-on-a-pre-release-db
|
||||
;; The release never creates `audit_log_pending`: the pending queue never
|
||||
;; shipped. A pre-release database that ran its step does have it, and it
|
||||
;; may still have rows; the table has to go, or the leftover rows would
|
||||
;; block the new GC when it deletes the rows they reference. Recreate the
|
||||
;; table and run the migration again to pin it.
|
||||
(th/db-exec! ["CREATE TABLE IF NOT EXISTS audit_log_pending (
|
||||
audit_log_id uuid NOT NULL REFERENCES audit_log(id) DEFERRABLE,
|
||||
consumer smallint NOT NULL CHECK (consumer BETWEEN 1 AND 20),
|
||||
PRIMARY KEY (consumer, audit_log_id))"])
|
||||
(t/is (contains? (table-columns "audit_log_pending") "consumer"))
|
||||
(let [apply! (mg/resource "app/migrations/sql/0159-consumer-generalization.sql")]
|
||||
(t/is (true? (apply! th/*pool*))))
|
||||
(t/is (empty? (table-columns "audit_log_pending"))))
|
||||
|
||||
(t/deftest the-migration-does-not-seed-consumer-cursors
|
||||
;; The migration cannot know which consumers are enabled, so the cursor
|
||||
;; table starts empty; `ensure-cursor` creates the row at (0, 0) on a
|
||||
;; consumer's first pass, which is what makes the first deploy replay the
|
||||
;; unarchived buffer.
|
||||
(th/db-exec! ["DELETE FROM audit_log_consumer_cursor WHERE consumer IN (1, 2)"])
|
||||
(let [apply! (mg/resource "app/migrations/sql/0159-consumer-generalization.sql")]
|
||||
(t/is (true? (apply! th/*pool*))))
|
||||
(let [rows (th/db-exec! ["SELECT consumer
|
||||
FROM audit_log_consumer_cursor
|
||||
WHERE consumer IN (1, 2)"])]
|
||||
(t/is (empty? rows))))
|
||||
|
||||
(t/deftest audit-log-rows-get-seq-and-xid-by-default
|
||||
;; The previous version keeps inserting rows during the rollout, so the
|
||||
;; database must fill both columns, and the new rows must sort above the
|
||||
;; backfilled history (the reserved low range).
|
||||
(let [id (uuid/next)]
|
||||
(try
|
||||
(th/db-exec! ["INSERT INTO audit_log (id, name, type, source, profile_id,
|
||||
ip_addr, props, context, tracked_at, created_at)
|
||||
VALUES (?, 'create-project', 'action', 'backend', ?,
|
||||
'127.0.0.1', ?, ?, now(), now())"
|
||||
id uuid/zero (db/tjson {}) (db/tjson {})])
|
||||
(let [row (th/db-exec-one! ["SELECT seq, xid::text AS xid
|
||||
FROM audit_log WHERE id = ?" id])]
|
||||
(t/is (some? (:seq row)))
|
||||
(t/is (>= (:seq row) 1000000))
|
||||
(t/is (not= "0" (:xid row))))
|
||||
(finally
|
||||
(th/db-exec! ["DELETE FROM audit_log WHERE id = ?" id])))))
|
||||
|
||||
(t/deftest audit-log-source-is-not-null
|
||||
;; `0064` added the column nullable and left `SET NOT NULL` commented;
|
||||
;; `0159` fills what is left and closes it. A NULL source would be deleted
|
||||
;; by neither the GC (`NOT LIKE 'telemetry:%'`) nor the telemetry task
|
||||
;; (`LIKE 'telemetry:%'`), so the row would stay for ever.
|
||||
(let [apply! (mg/resource "app/migrations/sql/0159-consumer-generalization.sql")]
|
||||
(t/is (true? (apply! th/*pool*))))
|
||||
(t/is (= "NO" (:is-nullable
|
||||
(th/db-exec-one!
|
||||
["SELECT is_nullable FROM information_schema.columns
|
||||
WHERE table_schema = 'public'
|
||||
AND table_name = 'audit_log'
|
||||
AND column_name = 'source'"]))))
|
||||
(t/is (thrown? java.sql.SQLException
|
||||
(th/db-exec!
|
||||
["INSERT INTO audit_log (id, name, type, source, profile_id,
|
||||
ip_addr, props, context, tracked_at, created_at)
|
||||
VALUES (?, 'create-project', 'action', NULL, ?,
|
||||
'127.0.0.1', ?, ?, now(), now())"
|
||||
(uuid/next) uuid/zero (db/tjson {}) (db/tjson {})]))))
|
||||
|
||||
(defn- indexdef
|
||||
[index-name]
|
||||
@ -167,12 +280,121 @@
|
||||
(t/is (not (str/includes? def "(tenant, queue"))))))
|
||||
|
||||
(t/deftest job-sweep-indexes-are-tenant-scoped
|
||||
;; The test database already ran 0159 in its old shape: re-apply it so the
|
||||
;; lost-sweep index exists with the current definition.
|
||||
(th/db-exec! ["DROP INDEX IF EXISTS job__scheduled__idx"])
|
||||
;; An early 0154 built the cron probe index; re-create it so the drop
|
||||
;; below proves the migration removes an index that already exists.
|
||||
(th/db-exec! ["CREATE INDEX IF NOT EXISTS job__name_label__idx
|
||||
ON job (tenant, name, queue, label)
|
||||
WHERE status IN ('new', 'scheduled', 'running', 'retry')"])
|
||||
(let [apply! (mg/resource "app/migrations/sql/0159-consumer-generalization.sql")]
|
||||
(t/is (true? (apply! th/*pool*))))
|
||||
(t/testing "orphan sweep"
|
||||
(t/is (str/includes? (indexdef "job__orphan__idx") "(tenant, modified_at)")))
|
||||
(t/testing "reschedule of lost scheduled rows"
|
||||
(t/is (str/includes? (indexdef "job__scheduled__idx") "(tenant, scheduled_at)")))
|
||||
(t/testing "cron no-overlap check and submit dedupe"
|
||||
(t/is (str/includes? (indexdef "job__name_label__idx") "(tenant, name, queue, label)"))))
|
||||
(t/testing "lost sweep: it filters by the hand-off time"
|
||||
(t/is (str/includes? (indexdef "job__scheduled__idx") "(tenant, modified_at)"))
|
||||
(t/is (not (str/includes? (indexdef "job__scheduled__idx") "(tenant, scheduled_at)"))))
|
||||
(t/testing "the cron no-overlap probe index is gone"
|
||||
(t/is (nil? (indexdef "job__name_label__idx")))))
|
||||
|
||||
(t/deftest job-new-dedupe-index-admits-one-not-started-row-per-key
|
||||
;; The dedupe of a submit is the index, not a delete before the insert:
|
||||
;; it is unique, it covers the whole key, and it only holds the rows
|
||||
;; that asked for the option and have not started yet.
|
||||
(let [def (indexdef "job__new_dedupe__uniq")]
|
||||
(t/is (str/includes? def "CREATE UNIQUE INDEX"))
|
||||
(t/is (str/includes? def "(tenant, name, queue, label)"))
|
||||
(t/is (str/includes? def "dedupe"))
|
||||
(t/testing "only the not-started status is covered"
|
||||
(t/is (str/includes? def "'new'"))
|
||||
(doseq [status ["scheduled" "running" "retry"]]
|
||||
(t/is (not (str/includes? def (str "'" status "'")))
|
||||
(str "the " status " status leaves the index"))))))
|
||||
|
||||
(t/deftest consumer-generalization-migration-is-idempotent
|
||||
;; The out-of-band runbook pre-applies the whole migration (columns,
|
||||
;; CONCURRENTLY indexes, backfill), so the migration must find the schema
|
||||
;; and do nothing when it runs afterwards.
|
||||
(let [apply! (mg/resource "app/migrations/sql/0159-consumer-generalization.sql")]
|
||||
(t/is (true? (apply! th/*pool*)))
|
||||
(t/is (contains? (table-columns "job") "exclusive"))
|
||||
(t/is (contains? (table-columns "job") "dedupe"))
|
||||
(t/is (contains? (table-indexes "job") "job__live_exclusive__uniq"))
|
||||
(t/is (contains? (table-indexes "job") "job__new_dedupe__uniq"))
|
||||
(t/is (contains? (table-columns "audit_log") "seq"))
|
||||
(t/is (contains? (table-columns "audit_log") "xid"))
|
||||
(t/is (contains? (table-indexes "audit_log") "audit_log__xid__seq__idx"))
|
||||
(t/is (contains? (table-columns "audit_log_consumer_cursor") "consumer"))))
|
||||
|
||||
(defn- sequence-last-value
|
||||
[]
|
||||
(:last-value (th/db-exec-one! ["SELECT last_value FROM audit_log_seq_seq"])))
|
||||
|
||||
(defn- insert-audit-row
|
||||
"Insert one row with the columns the previous version filled, so the
|
||||
database gives it `seq` and `xid` from their defaults."
|
||||
[]
|
||||
(let [id (uuid/next)]
|
||||
(th/db-exec! ["INSERT INTO audit_log (id, name, type, source, profile_id,
|
||||
ip_addr, props, context, tracked_at, created_at)
|
||||
VALUES (?, 'create-project', 'action', 'backend', ?,
|
||||
'127.0.0.1', ?, ?, now(), now())"
|
||||
id uuid/zero (db/tjson {}) (db/tjson {})])
|
||||
id))
|
||||
|
||||
(t/deftest the-sequence-reserve-is-only-applied-to-a-fresh-sequence
|
||||
;; A re-run must not move a sequence that already left its initial value:
|
||||
;; on the first application it starts at 1 and the migration reserves the
|
||||
;; low range, so the history gets 1..N and the new rows start above it.
|
||||
(let [apply! (mg/resource "app/migrations/sql/0159-consumer-generalization.sql")
|
||||
before (sequence-last-value)]
|
||||
(try
|
||||
(th/db-exec! ["ALTER SEQUENCE audit_log_seq_seq RESTART WITH 1"])
|
||||
(t/is (true? (apply! th/*pool*)))
|
||||
(t/is (>= (sequence-last-value) 1000000))
|
||||
(finally
|
||||
(th/db-exec! ["SELECT setval('audit_log_seq_seq', ?::bigint, false)" before])))))
|
||||
|
||||
(t/deftest the-sequence-reserve-is-not-moved-on-a-rerun
|
||||
;; Inserting a row advances the sequence; deleting it drops `count(*)`
|
||||
;; again. The old unconditional setval lowered the sequence in that case.
|
||||
(let [apply! (mg/resource "app/migrations/sql/0159-consumer-generalization.sql")
|
||||
id (insert-audit-row)]
|
||||
(try
|
||||
(let [before (sequence-last-value)]
|
||||
(th/db-exec! ["DELETE FROM audit_log WHERE id = ?" id])
|
||||
(t/is (true? (apply! th/*pool*)))
|
||||
(t/is (= before (sequence-last-value)))
|
||||
(let [max-seq (:max (th/db-exec-one! ["SELECT max(seq) AS max FROM audit_log"]))
|
||||
next (:next (th/db-exec-one! ["SELECT nextval('audit_log_seq_seq') AS next"]))]
|
||||
(t/is (> next (or max-seq 0)))))
|
||||
(finally
|
||||
(th/db-exec! ["DELETE FROM audit_log WHERE id = ?" id])))))
|
||||
|
||||
(t/deftest the-sequence-reserve-is-not-applied-to-a-used-once-sequence
|
||||
;; A sequence that was called once has last_value = 1 with is_called = true:
|
||||
;; it is not intact, so the reserve must leave it alone.
|
||||
(let [apply! (mg/resource "app/migrations/sql/0159-consumer-generalization.sql")
|
||||
before (sequence-last-value)]
|
||||
(try
|
||||
(th/db-exec! ["SELECT setval('audit_log_seq_seq', 1, true)"])
|
||||
(t/is (true? (apply! th/*pool*)))
|
||||
(t/is (= 1 (sequence-last-value)))
|
||||
(finally
|
||||
(th/db-exec! ["SELECT setval('audit_log_seq_seq', ?::bigint, false)" before])))))
|
||||
|
||||
(t/deftest job-live-exclusive-index-admits-one-live-row-per-key
|
||||
;; The exclusion of the live jobs is the index, not a check that races
|
||||
;; an insert: it is unique, it covers the whole key, and it only
|
||||
;; holds the rows that asked for the exclusion.
|
||||
(let [def (indexdef "job__live_exclusive__uniq")]
|
||||
(t/is (str/includes? def "CREATE UNIQUE INDEX"))
|
||||
(t/is (str/includes? def "(tenant, name, queue, label)"))
|
||||
(t/is (str/includes? def "exclusive"))
|
||||
(doseq [status ["new" "scheduled" "running" "retry"]]
|
||||
(t/is (str/includes? def (str "'" status "'"))
|
||||
(str "the live set covers the " status " status")))))
|
||||
|
||||
(t/deftest job-table-has-sweep-path-indexes
|
||||
(t/testing "jobs-GC expiration scan"
|
||||
@ -311,10 +533,10 @@
|
||||
{:file-id (:id file)
|
||||
:id data-id})))))
|
||||
|
||||
(t/testing "the backfill removes the dead metadata pointer"
|
||||
(t/testing "the backfill removes the migrated metadata pointer"
|
||||
(let [row (th/db-get :file-data {:file-id (:id file) :id data-id})
|
||||
metadata (fdata/decode-metadata (:metadata row))]
|
||||
(t/is (nil? (:storage-ref-id metadata)))))
|
||||
(t/is (not (contains? metadata :storage-ref-id)))))
|
||||
|
||||
(t/testing "a second application changes nothing"
|
||||
(let [before (th/db-get :file-data {:file-id (:id file) :id data-id})]
|
||||
@ -329,6 +551,7 @@
|
||||
sobject (:id (th/db-insert! :storage-object {:backend "fs"}))
|
||||
valid-id (uuid/next)
|
||||
corrupt-id (uuid/next)
|
||||
orphan-soid (uuid/next)
|
||||
orphan-id (uuid/next)
|
||||
other-id (uuid/next)]
|
||||
|
||||
@ -340,7 +563,7 @@
|
||||
:metadata (db/json {"storage-ref-id" "not-a-uuid"})})
|
||||
(th/db-insert! :file-data
|
||||
{:file-id (:id file) :id orphan-id :type "snapshot" :backend "storage"
|
||||
:metadata (db/json {"storage-ref-id" (str (uuid/next))})})
|
||||
:metadata (db/json {"storage-ref-id" (str orphan-soid)})})
|
||||
(th/db-insert! :file-data
|
||||
{:file-id (:id file) :id other-id :type "snapshot" :backend "db"
|
||||
:metadata (db/json {"storage-ref-id" (str sobject)})})
|
||||
@ -354,14 +577,16 @@
|
||||
:id valid-id})))))
|
||||
|
||||
(t/testing "a non-uuid value stays NULL"
|
||||
(t/is (nil? (:storage-object-id (th/db-get :file-data
|
||||
{:file-id (:id file)
|
||||
:id corrupt-id})))))
|
||||
(let [row (th/db-get :file-data {:file-id (:id file) :id corrupt-id})
|
||||
metadata (fdata/decode-metadata (:metadata row))]
|
||||
(t/is (nil? (:storage-object-id row)))
|
||||
(t/is (= "not-a-uuid" (:storage-ref-id metadata)))))
|
||||
|
||||
(t/testing "a pointer to a missing object stays NULL"
|
||||
(t/is (nil? (:storage-object-id (th/db-get :file-data
|
||||
{:file-id (:id file)
|
||||
:id orphan-id})))))
|
||||
(let [row (th/db-get :file-data {:file-id (:id file) :id orphan-id})
|
||||
metadata (fdata/decode-metadata (:metadata row))]
|
||||
(t/is (nil? (:storage-object-id row)))
|
||||
(t/is (= (str orphan-soid) (:storage-ref-id metadata)))))
|
||||
|
||||
(t/testing "rows of other backends are not touched"
|
||||
(t/is (nil? (:storage-object-id (th/db-get :file-data
|
||||
|
||||
@ -85,3 +85,27 @@
|
||||
(t/is (string? (:tracked-at e1')))
|
||||
(t/is (= (str now) (:created-at e1')))
|
||||
(t/is (= (str team-id) (str (get-in e1' [:props :team-id])))))))))
|
||||
|
||||
(t/deftest ingest-audit-log-rejects-a-redirect
|
||||
;; A 3xx is a configuration error, not an empty success: it surfaces a
|
||||
;; typed error instead of the :malli.core/invalid-schema that a nil schema
|
||||
;; throws while building the coercer.
|
||||
(binding [cf/flags (conj cf/flags :admin-console)
|
||||
cf/config (assoc cf/config :admin-console-uri "http://ac.example/admin-console/")]
|
||||
(with-mocks [http-mock {:target 'app.http.client/req
|
||||
:return {:status 302
|
||||
:headers {"location" "http://ac.example/admin-console/api/audit-log"}
|
||||
:body ""}}]
|
||||
(let [client (ig/init-key :app.nitrate/client
|
||||
{::http/client (Object.)
|
||||
::setup/shared-keys {:admin-console "test-shared-key"}})
|
||||
cfg {:app.nitrate/client client}
|
||||
err (try
|
||||
(nitrate/call cfg :ingest-audit-log
|
||||
{::rpc/profile-id (uuid/next)
|
||||
:events []})
|
||||
nil
|
||||
(catch clojure.lang.ExceptionInfo e e))]
|
||||
(t/is (= 1 (:call-count @http-mock)))
|
||||
(t/is (= :nitrate-http-error (:type (ex-data err))))
|
||||
(t/is (= 302 (:status (ex-data err))))))))
|
||||
|
||||
@ -206,7 +206,7 @@
|
||||
;; When the GC physically deletes a team, the projection must drop its row,
|
||||
;; so the event carries the purged ids.
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}]
|
||||
(with-redefs [cf/flags (conj cf/flags :audit-log)]
|
||||
(th/with-audit-config (conj cf/flags :nexus)
|
||||
(let [profile (th/create-profile* 1 {:is-active true})
|
||||
team (th/create-team* 71 {:profile-id (:id profile)})]
|
||||
(th/command! {::th/type :delete-team
|
||||
@ -218,8 +218,10 @@
|
||||
(th/reset-mock! audit-mock)
|
||||
(th/run-task! :objects-gc {:skip-delay true})
|
||||
(t/is (nil? (th/db-get :team {:id (:id team)} {::db/remove-deleted false})))
|
||||
(let [events (filter #(contains? (:props %) :team-ids)
|
||||
(let [events (filter #(= #{:team-ids} (set (keys (:props %))))
|
||||
(events-named audit-mock "hard-delete-objects"))]
|
||||
;; Every chunk carries exactly its own entity key and nothing
|
||||
;; else: no flat team-ids are resolved after the delete.
|
||||
(t/is (= 1 (count events)))
|
||||
(let [event (first events)]
|
||||
(t/is (= "command" (:type event)))
|
||||
@ -228,7 +230,7 @@
|
||||
|
||||
(t/deftest gc-emits-hard-delete-objects-for-projects
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}]
|
||||
(with-redefs [cf/flags (conj cf/flags :audit-log)]
|
||||
(th/with-audit-config (conj cf/flags :nexus)
|
||||
(let [profile (th/create-profile* 1 {:is-active true})
|
||||
team (th/create-team* 72 {:profile-id (:id profile)})
|
||||
project (th/create-project* 72 {:profile-id (:id profile)
|
||||
@ -245,12 +247,13 @@
|
||||
(t/is (= 1 (count events)))
|
||||
(let [event (first events)]
|
||||
(t/is (= "command" (:type event)))
|
||||
(t/is (= #{:project-ids} (set (keys (:props event)))))
|
||||
(t/is (= [(:id project)] (get-in event [:props :project-ids])))
|
||||
(t/is (= "objects-gc" (get-in event [:context :triggered-by])))))))))
|
||||
|
||||
(t/deftest gc-emits-hard-delete-objects-for-files
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}]
|
||||
(with-redefs [cf/flags (conj cf/flags :audit-log)]
|
||||
(th/with-audit-config (conj cf/flags :nexus)
|
||||
(let [profile (th/create-profile* 1 {:is-active true})
|
||||
team (th/create-team* 73 {:profile-id (:id profile)})
|
||||
project (th/create-project* 73 {:profile-id (:id profile)
|
||||
@ -269,10 +272,11 @@
|
||||
(t/is (= 1 (count events)))
|
||||
(let [event (first events)]
|
||||
(t/is (= "command" (:type event)))
|
||||
(t/is (= #{:file-ids} (set (keys (:props event)))))
|
||||
(t/is (= [(:id file)] (get-in event [:props :file-ids])))
|
||||
(t/is (= "objects-gc" (get-in event [:context :triggered-by])))))))))
|
||||
|
||||
(t/deftest gc-emits-no-event-without-audit-log-flag
|
||||
(t/deftest gc-emits-no-event-without-a-consumer
|
||||
;; The GC must stay silent when the audit log is off, even though the row is
|
||||
;; still purged.
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}]
|
||||
@ -317,7 +321,7 @@
|
||||
|
||||
(t/deftest gc-emits-events-only-for-committed-chunks
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}]
|
||||
(with-redefs [cf/flags (conj cf/flags :audit-log)]
|
||||
(th/with-audit-config (conj cf/flags :nexus)
|
||||
(let [profile (th/create-profile* 1)
|
||||
rolled-back (create-deleted-file! profile 1 (ct/in-past "30m"))
|
||||
blocked (create-deleted-file! profile 2 (ct/in-past "20m"))
|
||||
|
||||
@ -14,6 +14,7 @@
|
||||
[app.db :as db]
|
||||
[app.jobs :as-alias jobs]
|
||||
[app.loggers.audit :as audit]
|
||||
[app.loggers.webhooks :as-alias webhooks]
|
||||
[app.main :as main]
|
||||
[app.metrics :as mtx]
|
||||
[app.metrics.definition :as-alias mdef]
|
||||
@ -108,7 +109,7 @@
|
||||
(assoc base :app.http/auth-key-id :nexus)))))))
|
||||
|
||||
(t/deftest push-events-1
|
||||
(with-redefs [app.config/flags #{:audit-log :nexus}]
|
||||
(th/with-audit-config #{:nexus}
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
team-id (:default-team-id prof)
|
||||
proj-id (:default-project-id prof)
|
||||
@ -138,12 +139,10 @@
|
||||
(t/is (= 1 (count rows)))
|
||||
(t/is (= (:id prof) (:profile-id row)))
|
||||
(t/is (= "navigate" (:name row)))
|
||||
(t/is (= "frontend" (:source row)))
|
||||
;; The row is born pending for the enabled consumer.
|
||||
(t/is (= 1 (count (th/db-exec! ["select * from audit_log_pending"]))))))))
|
||||
(t/is (= "frontend" (:source row)))))))
|
||||
|
||||
(t/deftest push-events-2
|
||||
(with-redefs [app.config/flags #{:audit-log}]
|
||||
(th/with-audit-config #{:nexus}
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
team-id (:default-team-id prof)
|
||||
proj-id (:default-project-id prof)
|
||||
@ -178,10 +177,10 @@
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(t/deftest push-events-telemetry-mode-stores-anonymized-row
|
||||
;; When telemetry is enabled and audit-log is NOT, frontend events
|
||||
;; When telemetry is enabled and no consumer persists rows, frontend events
|
||||
;; must be stored with source="telemetry", empty props, zeroed ip,
|
||||
;; and context filtered to safe keys only.
|
||||
(with-redefs [cf/flags #{:telemetry}]
|
||||
(th/with-audit-config #{:telemetry}
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
team-id (:default-team-id prof)
|
||||
proj-id (:default-project-id prof)
|
||||
@ -239,13 +238,13 @@
|
||||
(t/is (not (contains? ctx :external-session-id))))))))
|
||||
|
||||
(t/deftest push-events-both-flags-creates-two-rows
|
||||
;; When both :audit-log and :telemetry flags are active, two rows
|
||||
;; When a consumer persists rows and :telemetry is active, two rows
|
||||
;; should be stored: one full audit entry and one telemetry entry.
|
||||
;; The profile is created before enabling :admin-console: team
|
||||
;; bootstrap calls the real nitrate client, which is unconfigured
|
||||
;; in tests.
|
||||
(let [prof (th/create-profile* 1 {:is-active true})]
|
||||
(with-redefs [cf/flags #{:audit-log :telemetry :nexus :admin-console}]
|
||||
(th/with-audit-config #{:telemetry :nexus :admin-console}
|
||||
(let [params {::th/type :push-audit-events
|
||||
::rpc/profile-id (:id prof)
|
||||
:events [{:name "navigate"
|
||||
@ -273,16 +272,14 @@
|
||||
(t/is (= "0.0.0.0" (str (:ip-addr row2))))
|
||||
(let [day-now (ct/truncate (ct/now) :days)]
|
||||
(t/is (= day-now (:created-at row2)))
|
||||
(t/is (= day-now (:tracked-at row2))))
|
||||
;; Both rows are born pending for the enabled consumers.
|
||||
(t/is (= 4 (count (th/db-exec! ["select * from audit_log_pending"])))))))))
|
||||
(t/is (= day-now (:tracked-at row2)))))))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; BACKEND PROCESS-EVENT PATH (RPC commands)
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(t/deftest backend-process-event-only-audit-log
|
||||
(with-redefs [cf/flags #{:audit-log}]
|
||||
(t/deftest backend-process-event-only-persisted-rows
|
||||
(th/with-audit-config #{:nexus}
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
event {:id (uuid/next)
|
||||
:type "action"
|
||||
@ -302,7 +299,7 @@
|
||||
(t/is (not= "0.0.0.0" (str (:ip-addr row))))))))
|
||||
|
||||
(t/deftest backend-process-event-only-telemetry
|
||||
(with-redefs [cf/flags #{:telemetry}]
|
||||
(th/with-audit-config #{:telemetry}
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
event {:id (uuid/next)
|
||||
:type "action"
|
||||
@ -321,10 +318,10 @@
|
||||
(t/is (= "0.0.0.0" (str (:ip-addr row))))))))
|
||||
|
||||
(t/deftest backend-process-event-both-flags-creates-two-rows
|
||||
;; When both :audit-log and :telemetry are active, the backend
|
||||
;; When persistence and :telemetry are active, the backend
|
||||
;; process-event must store two rows: one full audit entry and one
|
||||
;; telemetry entry.
|
||||
(with-redefs [cf/flags #{:audit-log :telemetry}]
|
||||
(th/with-audit-config #{:nexus :telemetry}
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
event {:id (uuid/next)
|
||||
:type "action"
|
||||
@ -357,7 +354,7 @@
|
||||
;; the id used to be generated inside `append-audit-entry` without
|
||||
;; ever reaching the worker params, so `:event-id` always landed nil.
|
||||
(with-mocks [submit-mock {:target 'app.jobs/submit :return nil}]
|
||||
(with-redefs [cf/flags #{:audit-log :webhooks}]
|
||||
(th/with-audit-config #{:nexus :webhooks}
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
event {:type "action"
|
||||
:name "delete-project"
|
||||
@ -379,11 +376,11 @@
|
||||
(t/is (= (:id row) (:id task-event)))))))))
|
||||
|
||||
(t/deftest backend-process-event-telemetry-row-gets-own-id
|
||||
;; When both `:audit-log` and `:telemetry` are active, the shadow
|
||||
;; When persistence and `:telemetry` are active, the shadow
|
||||
;; telemetry row must not reuse the full row id (shared PK), while
|
||||
;; the webhook task still carries the full row id.
|
||||
(with-mocks [submit-mock {:target 'app.jobs/submit :return nil}]
|
||||
(with-redefs [cf/flags #{:audit-log :telemetry :webhooks}]
|
||||
(th/with-audit-config #{:nexus :telemetry :webhooks}
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
event {:type "action"
|
||||
:name "delete-project"
|
||||
@ -404,10 +401,47 @@
|
||||
(t/is (= (:id (first (filter #(= "backend" (:source %)) rows)))
|
||||
(:id task-event)))))))))
|
||||
|
||||
(t/deftest process-event-stores-no-row-without-a-consumer
|
||||
;; No cursor consumer means no full row, even when a webhook is active:
|
||||
;; the webhook job carries the event blob and does not read the buffer.
|
||||
(with-mocks [submit-mock {:target 'app.jobs/submit :return nil}]
|
||||
(th/with-audit-config #{:webhooks}
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
event {:type "action"
|
||||
:name "delete-project"
|
||||
:profile-id (:id prof)
|
||||
:props {:team-id (:default-team-id prof)}
|
||||
:context {:version "2.0.0" :initiator "app"}
|
||||
:tracked-at (ct/now)
|
||||
:created-at (ct/now)
|
||||
:source "backend"
|
||||
:app.loggers.webhooks/event? true}]
|
||||
(audit/submit* th/*system* event)
|
||||
(t/is (= 0 (count (th/db-exec! ["select * from audit_log"]))))
|
||||
(t/is (= 1 (:call-count @submit-mock)))))))
|
||||
|
||||
(t/deftest process-event-stores-row-with-a-single-consumer
|
||||
;; A single consumer is enough to make the backend persist full rows.
|
||||
(th/with-audit-config #{:nexus}
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
event {:type "action"
|
||||
:name "delete-project"
|
||||
:profile-id (:id prof)
|
||||
:props {:team-id (:default-team-id prof)}
|
||||
:context {:version "2.0.0" :initiator "app"}
|
||||
:tracked-at (ct/now)
|
||||
:created-at (ct/now)
|
||||
:source "backend"}]
|
||||
(audit/submit* th/*system* event)
|
||||
(let [[row :as rows] (->> (th/db-exec! ["select * from audit_log"])
|
||||
(mapv decode-row))]
|
||||
(t/is (= 1 (count rows)))
|
||||
(t/is (= "backend" (:source row)))))))
|
||||
|
||||
(t/deftest push-events-disabled-when-no-flags-and-no-telemetry
|
||||
;; When neither audit-log nor telemetry is enabled, no rows should
|
||||
;; When neither persistence nor telemetry is enabled, no rows should
|
||||
;; be stored.
|
||||
(with-redefs [cf/flags #{}]
|
||||
(th/with-audit-config #{}
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
params {::th/type :push-audit-events
|
||||
::rpc/profile-id (:id prof)
|
||||
@ -422,37 +456,59 @@
|
||||
(t/is (nil? (:error out)))
|
||||
(t/is (= 0 (count (th/db-exec! ["select * from audit_log"])))))))
|
||||
|
||||
(t/deftest push-events-error-reporting-without-a-consumer
|
||||
;; :error-reporting tells the frontend to collect events, so the ingest
|
||||
;; must accept the batch and hand crash events to the reporters even
|
||||
;; with no cursor consumer and no telemetry. No rows are stored.
|
||||
(with-mocks [db-emit {:target 'app.loggers.database/emit :return nil}
|
||||
mm-emit {:target 'app.loggers.mattermost/emit :return nil}]
|
||||
(th/with-audit-config #{:error-reporting}
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
params {::th/type :push-audit-events
|
||||
::rpc/profile-id (:id prof)
|
||||
:events [{:name "unhandled-exception"
|
||||
:props {:hint "crash"}
|
||||
:timestamp (ct/now)
|
||||
:type "action"}]}
|
||||
params (with-meta params
|
||||
{:app.http/request http-request})
|
||||
out (th/command! params)]
|
||||
(t/is (nil? (:error out)))
|
||||
(t/is (= 1 (:call-count @db-emit)))
|
||||
(t/is (= 1 (:call-count @mm-emit)))
|
||||
(t/is (= 0 (count (th/db-exec! ["select * from audit_log"]))))))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; GET-ENVIRONMENT-DATA
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(t/deftest get-environment-data-returns-selfhost-by-default
|
||||
(with-redefs [cf/flags #{:audit-log}]
|
||||
(th/with-audit-config #{:nexus}
|
||||
(let [out (th/command! {::th/type :get-environment-data})]
|
||||
(t/is (th/success? out))
|
||||
(t/is (= "selfhost" (-> out :result :deployment))))))
|
||||
|
||||
(t/deftest get-environment-data-returns-saas-when-configured
|
||||
(with-redefs [cf/flags #{:telemetry}]
|
||||
(th/with-audit-config #{:telemetry}
|
||||
(binding [cf/config (assoc cf/config :is-saas true)]
|
||||
(let [out (th/command! {::th/type :get-environment-data})]
|
||||
(t/is (th/success? out))
|
||||
(t/is (= "saas" (-> out :result :deployment)))))))
|
||||
|
||||
(t/deftest get-environment-data-only-exposes-event-flags
|
||||
(with-redefs [cf/flags #{:audit-log :telemetry :graph :nrepl-server}]
|
||||
(th/with-audit-config #{:nexus :telemetry :graph :nrepl-server}
|
||||
(let [out (th/command! {::th/type :get-environment-data})]
|
||||
(t/is (th/success? out))
|
||||
(t/is (= #{:audit-log :telemetry} (-> out :result :flags))))))
|
||||
|
||||
(t/deftest get-environment-data-returns-empty-flags-when-none-enabled
|
||||
(with-redefs [cf/flags #{:graph}]
|
||||
(th/with-audit-config #{:graph}
|
||||
(let [out (th/command! {::th/type :get-environment-data})]
|
||||
(t/is (th/success? out))
|
||||
(t/is (= #{} (-> out :result :flags))))))
|
||||
|
||||
(t/deftest get-enabled-flags-still-returns-flat-set
|
||||
(with-redefs [cf/flags #{:audit-log :telemetry :graph}]
|
||||
(th/with-audit-config #{:nexus :telemetry :graph}
|
||||
(let [out (th/command! {::th/type :get-enabled-flags})]
|
||||
(t/is (th/success? out))
|
||||
(t/is (= #{:audit-log :telemetry} (:result out))))))
|
||||
@ -655,7 +711,7 @@
|
||||
(t/is (= "app" (audit/get-client-initiator (request "something-else/1.0"))))))
|
||||
|
||||
(t/deftest push-events-assigns-initiator-from-x-client
|
||||
(with-redefs [app.config/flags #{:audit-log}]
|
||||
(th/with-audit-config #{:nexus}
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
params {::th/type :push-audit-events
|
||||
::rpc/profile-id (:id prof)
|
||||
@ -677,7 +733,7 @@
|
||||
(t/is (= "admin-console" (-> row :context :initiator)))))))
|
||||
|
||||
(t/deftest push-events-initiator-defaults-to-app
|
||||
(with-redefs [app.config/flags #{:audit-log}]
|
||||
(th/with-audit-config #{:nexus}
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
params {::th/type :push-audit-events
|
||||
::rpc/profile-id (:id prof)
|
||||
@ -698,15 +754,15 @@
|
||||
;; INSERT DEFAULTS
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(t/deftest insert-only-runs-with-audit-log-flag
|
||||
;; insert must be a no-op when :audit-log flag is not set
|
||||
(t/deftest insert-only-runs-when-a-consumer-is-enabled
|
||||
;; insert must be a no-op when no consumer is enabled
|
||||
(with-redefs [app.config/flags #{:telemetry}]
|
||||
(audit/insert th/*system* {:name "test" :type "action"})
|
||||
(t/is (= 0 (count (th/db-exec! ["select * from audit_log"]))))))
|
||||
|
||||
(t/deftest insert-sets-defaults
|
||||
;; insert must set defaults and persist when :audit-log is set
|
||||
(with-redefs [app.config/flags #{:audit-log}]
|
||||
;; insert must set defaults and persist when a consumer is enabled
|
||||
(th/with-audit-config #{:nexus}
|
||||
(audit/insert th/*system* {:name "test-action" :type "action"})
|
||||
(let [[row] (->> (th/db-exec! ["select * from audit_log"])
|
||||
(mapv decode-row))]
|
||||
@ -853,8 +909,9 @@
|
||||
(t/deftest prepare-rpc-event-coerces-string-metadata-profile-id
|
||||
;; `::audit/profile-id` is set by hand in a dozen commands, and some of
|
||||
;; them read the value from token claims or other stringly-typed sources.
|
||||
;; The audit schema demands a uuid, and `submit*` swallows the validation
|
||||
;; error, so an unconverted string would drop the event on the floor.
|
||||
;; The audit schema demands a uuid, and `submit*` propagates the validation
|
||||
;; error, so an unconverted string would fail the response instead of only
|
||||
;; losing the event.
|
||||
(let [caller (th/create-profile* 1 {:is-active true})
|
||||
target "33601240-a00b-11ea-ba1b-c554cc60e361"
|
||||
result (with-meta {:some-data "value"}
|
||||
@ -926,18 +983,17 @@
|
||||
(t/deftest push-handle-events-records-inserted
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
metrics (make-ingest-metrics)]
|
||||
(with-redefs [cf/flags #{:audit-log :telemetry :nexus :admin-console}]
|
||||
(th/with-audit-config #{:telemetry :nexus :admin-console}
|
||||
(@#'cmd-audit/handle-events {::db/pool th/*pool* ::mtx/metrics metrics}
|
||||
(push-params (:id prof)))
|
||||
(t/is (= 1.0 (ingest-counter metrics "full" "inserted")))
|
||||
(t/is (= 1.0 (ingest-counter metrics "shadow" "inserted")))
|
||||
(t/is (= 2 (count (th/db-exec! ["select * from audit_log"]))))
|
||||
(t/is (= 4 (count (th/db-exec! ["select * from audit_log_pending"])))))))
|
||||
(t/is (= 2 (count (th/db-exec! ["select * from audit_log"])))))))
|
||||
|
||||
(t/deftest push-handle-events-records-failed-on-error
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
metrics (make-ingest-metrics)]
|
||||
(with-redefs [cf/flags #{:audit-log :telemetry :nexus :admin-console}]
|
||||
(th/with-audit-config #{:telemetry :nexus :admin-console}
|
||||
(with-mocks [insert-mock {:target 'app.db/insert-many!
|
||||
:throw (ex-info "boom" {})}]
|
||||
(t/is (thrown? clojure.lang.ExceptionInfo
|
||||
@ -945,19 +1001,51 @@
|
||||
(push-params (:id prof)))))
|
||||
(t/is (= 1.0 (ingest-counter metrics "full" "failed")))
|
||||
(t/is (= 1.0 (ingest-counter metrics "shadow" "failed")))
|
||||
(t/is (= 0 (count (th/db-exec! ["select * from audit_log"]))))
|
||||
(t/is (= 0 (count (th/db-exec! ["select * from audit_log_pending"]))))))))
|
||||
(t/is (= 0 (count (th/db-exec! ["select * from audit_log"]))))))))
|
||||
|
||||
(t/deftest push-handle-events-single-face-emits-no-empty-series
|
||||
;; With only :audit-log on, exactly one ingest series exists: the
|
||||
;; With a consumer enabled and telemetry off, exactly one ingest series exists: the
|
||||
;; pos? guard in record-ingested emits no phantom shadow series.
|
||||
;; (No send flags either, so R3 leaves the row pending-less.)
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
metrics (make-ingest-metrics)]
|
||||
(with-redefs [cf/flags #{:audit-log}]
|
||||
(th/with-audit-config #{:nexus}
|
||||
(@#'cmd-audit/handle-events {::db/pool th/*pool* ::mtx/metrics metrics}
|
||||
(push-params (:id prof)))
|
||||
(t/is (= 1.0 (ingest-counter metrics "full" "inserted")))
|
||||
(t/is (= #{["full" "inserted"]} (counter-label-sets metrics)))
|
||||
(t/is (= 1 (count (th/db-exec! ["select * from audit_log"]))))
|
||||
(t/is (= 0 (count (th/db-exec! ["select * from audit_log_pending"])))))))
|
||||
(t/is (= 1 (count (th/db-exec! ["select * from audit_log"])))))))
|
||||
|
||||
(t/deftest backend-webhook-batch-debounces-on-the-pending-job
|
||||
;; Two events of the same batch key share one pending job: the second
|
||||
;; moves its schedule and leaves the last payload behind, so the
|
||||
;; receiver is told once about the burst and with the newest event.
|
||||
(th/with-audit-config #{:webhooks}
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
label "rpc:update-file:the-key"
|
||||
event (fn [n batch-timeout]
|
||||
{:id (uuid/next)
|
||||
:type "action"
|
||||
:name "update-file"
|
||||
:profile-id (:id prof)
|
||||
:props {:n n}
|
||||
:context {:version "2.0.0" :initiator "app"}
|
||||
:tracked-at (ct/now)
|
||||
:created-at (ct/now)
|
||||
:source "backend"
|
||||
::webhooks/event? true
|
||||
::webhooks/batch-key "the-key"
|
||||
::webhooks/batch-timeout batch-timeout})]
|
||||
(audit/submit* th/*system* (event 1 (ct/duration "2m")))
|
||||
(let [[row1 :as rows] (th/db-exec! ["select * from job where label = ?" label])]
|
||||
(t/is (= 1 (count rows)))
|
||||
(audit/submit* th/*system* (event 2 (ct/duration "1h")))
|
||||
(let [[row2 :as rows] (th/db-exec! ["select * from job where label = ?" label])]
|
||||
(t/is (= 1 (count rows)) "the burst keeps a single pending job")
|
||||
(t/is (= (:id row1) (:id row2)) "and the job keeps its id")
|
||||
(t/is (pos? (compare (:scheduled-at row2) (:scheduled-at row1)))
|
||||
"the second event slides the window to its own delay")
|
||||
(let [params (db/decode-json-pgobject (:params row2))
|
||||
blob (transit/decode-str (:event-blob params))]
|
||||
(t/is (= 2 (get-in blob [:props :n]))
|
||||
"and the payload delivered is the last event")))))))
|
||||
|
||||
@ -128,7 +128,7 @@
|
||||
;; The imported files are created inside the importer, so the command emits a
|
||||
;; create-file event per file for a local projection.
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}]
|
||||
(with-redefs [cf/flags (conj cf/flags :audit-log)]
|
||||
(th/with-audit-config (conj cf/flags :nexus)
|
||||
(let [profile (th/create-profile* 1 {:is-active true})
|
||||
team (th/create-team* 51 {:profile-id (:id profile)})
|
||||
project (th/create-project* 51 {:profile-id (:id profile)
|
||||
|
||||
@ -24,14 +24,16 @@
|
||||
[app.db.sql :as sql]
|
||||
[app.features.fdata :as fdata]
|
||||
[app.http :as http]
|
||||
[app.loggers.audit :as-alias audit]
|
||||
[app.jobs :as-alias jobs]
|
||||
[app.loggers.audit :as audit]
|
||||
[app.redis :as rds]
|
||||
[app.rpc :as-alias rpc]
|
||||
[app.rpc.commands.files :as files]
|
||||
[app.storage :as sto]
|
||||
[backend-tests.helpers :as th]
|
||||
[clojure.test :as t]
|
||||
[cuerdas.core :as str]))
|
||||
[cuerdas.core :as str]
|
||||
[mockery.core :refer [with-mocks]]))
|
||||
|
||||
(t/use-fixtures :once th/state-init)
|
||||
(t/use-fixtures :each th/database-reset)
|
||||
@ -2433,15 +2435,164 @@
|
||||
:id (:id library)})]
|
||||
(t/is (nil? (:error out)))
|
||||
|
||||
;; The relations are removed at once, the absorption is done by the
|
||||
;; deletion task.
|
||||
(t/is (empty? (th/db-query :file-library-rel {:library-file-id (:id library)})))
|
||||
;; The relations stay until the deletion task runs: the job
|
||||
;; captures the dependents, removes the relations and absorbs
|
||||
;; them there.
|
||||
(t/is (not (empty? (th/db-query :file-library-rel {:library-file-id (:id library)}))))
|
||||
(t/is (= (:id library)
|
||||
(cfo/get-tokens-source (:data (bfc/get-file th/*system* (:id file))))))
|
||||
|
||||
(th/run-pending-jobs)
|
||||
(t/is (empty? (th/db-query :file-library-rel {:library-file-id (:id library)})))
|
||||
(check-tokens-absorbed prepared)))
|
||||
|
||||
(t/deftest delete-shared-library-absorb-failure-keeps-cascade-going
|
||||
;; If the absorb blows up, the savepoint rolls it back and the
|
||||
;; cascade goes on: the file is still marked deleted and the job
|
||||
;; succeeds. The relations are dropped before the absorb runs, so
|
||||
;; they do not survive a failed absorb; the dependents keep the data
|
||||
;; they had before the absorb was attempted.
|
||||
(let [profile (th/create-profile* 1)
|
||||
{:keys [library]} (prepare-tokens-library profile)
|
||||
out (th/command! {::th/type :delete-file
|
||||
::rpc/profile-id (:id profile)
|
||||
:id (:id library)})]
|
||||
(t/is (nil? (:error out)))
|
||||
(with-redefs [files/absorb-library (fn [& _] (throw (ex-info "boom" {})))]
|
||||
(th/run-pending-jobs))
|
||||
(t/is (empty? (th/db-query :file-library-rel {:library-file-id (:id library)})))
|
||||
(t/is (some? (:deleted-at (th/db-get :file {:id (:id library)} {::db/remove-deleted false}))))))
|
||||
|
||||
(t/deftest delete-shared-library-queues-job-without-file-ids
|
||||
;; The cascade captures the dependents itself: the RPC no longer
|
||||
;; snapshots them into the job params.
|
||||
(let [profile (th/create-profile* 1)
|
||||
{:keys [library]} (prepare-tokens-library profile)
|
||||
submitted (atom nil)]
|
||||
(with-redefs [jobs/submit (fn [_ params] (reset! submitted params) (uuid/next))]
|
||||
(let [out (th/command! {::th/type :delete-file
|
||||
::rpc/profile-id (:id profile)
|
||||
:id (:id library)})]
|
||||
(t/is (nil? (:error out)))))
|
||||
(t/is (= :delete-object (::jobs/name @submitted)))
|
||||
(t/is (not (contains? (::jobs/params @submitted) :file-ids)))))
|
||||
|
||||
(t/deftest delete-project-skips-absorb-for-files-dying-together
|
||||
;; When the whole project dies, absorbing the shared library into
|
||||
;; its own dying files is wasted work: the cascade cleans the
|
||||
;; relations but leaves the dependents untouched.
|
||||
(let [profile (th/create-profile* 1)
|
||||
team-id (:default-team-id profile)
|
||||
proj (th/create-project* 1 {:profile-id (:id profile) :team-id team-id})
|
||||
library (th/create-file* 1 {:profile-id (:id profile)
|
||||
:project-id (:id proj)
|
||||
:is-shared true})
|
||||
file (th/create-file* 2 {:profile-id (:id profile)
|
||||
:project-id (:id proj)})
|
||||
_ (th/link-file-to-library* {:file-id (:id file)
|
||||
:library-id (:id library)})
|
||||
revn (:revn (th/db-get :file {:id (:id file)}))]
|
||||
(let [out (th/command! {::th/type :delete-project
|
||||
::rpc/profile-id (:id profile)
|
||||
:id (:id proj)})]
|
||||
(t/is (nil? (:error out))))
|
||||
(th/run-pending-jobs)
|
||||
;; Nothing outside the cascade uses the library, so the relations
|
||||
;; stay until the hard delete removes them with their files.
|
||||
(t/is (not (empty? (th/db-query :file-library-rel {:library-file-id (:id library)}))))
|
||||
(t/is (= revn (:revn (th/db-get :file {:id (:id file)} {::db/remove-deleted false}))))
|
||||
;; The hard delete cascades the relations away with their files.
|
||||
(th/run-task! :objects-gc {:skip-delay true})
|
||||
(t/is (nil? (th/db-get :file {:id (:id file)} {::db/remove-deleted false})))
|
||||
(t/is (empty? (th/db-query :file-library-rel {:library-file-id (:id library)})))))
|
||||
|
||||
(t/deftest delete-object-absorbs-with-deprecated-file-ids
|
||||
;; Deploy window: a job queued by the previous release carries the
|
||||
;; dependents in :file-ids and its RPC already deleted the relations.
|
||||
;; The live lookup finds nothing, so the worker must trust the
|
||||
;; snapshot instead of skipping the absorb silently.
|
||||
(let [profile (th/create-profile* 6 {:is-active true})
|
||||
team-id (:default-team-id profile)
|
||||
proj (th/create-project* 6 {:profile-id (:id profile) :team-id team-id})
|
||||
library (th/create-file* 8 {:profile-id (:id profile)
|
||||
:project-id (:id proj)
|
||||
:is-shared true})
|
||||
dependent (th/create-file* 9 {:profile-id (:id profile)
|
||||
:project-id (:id proj)})
|
||||
_ (th/link-file-to-library* {:file-id (:id dependent)
|
||||
:library-id (:id library)})
|
||||
revn (:revn (th/db-get :file {:id (:id dependent)}))]
|
||||
;; What the old RPC did: drop the relations before queueing.
|
||||
(th/db-delete! :file-library-rel {:library-file-id (:id library)})
|
||||
(th/run-task! :delete-object {:object :file
|
||||
:id (:id library)
|
||||
:deleted-at (ct/now)
|
||||
:file-ids [(:id dependent)]})
|
||||
(t/is (> (:revn (th/db-get :file {:id (:id dependent)} {::db/remove-deleted false}))
|
||||
revn))))
|
||||
|
||||
(t/deftest delete-project-absorbs-into-surviving-dependent
|
||||
;; The doomed-set only skips files that die in the same cascade: a
|
||||
;; dependent in another project of the team survives, so the library
|
||||
;; must be absorbed into it (and only into it).
|
||||
(let [profile (th/create-profile* 5 {:is-active true})
|
||||
team-id (:default-team-id profile)
|
||||
proj-a (th/create-project* 5 {:profile-id (:id profile) :team-id team-id})
|
||||
proj-b (th/create-project* 6 {:profile-id (:id profile) :team-id team-id})
|
||||
library (th/create-file* 5 {:profile-id (:id profile)
|
||||
:project-id (:id proj-a)
|
||||
:is-shared true})
|
||||
doomed (th/create-file* 6 {:profile-id (:id profile)
|
||||
:project-id (:id proj-a)})
|
||||
survivor (th/create-file* 7 {:profile-id (:id profile)
|
||||
:project-id (:id proj-b)})
|
||||
_ (th/link-file-to-library* {:file-id (:id survivor)
|
||||
:library-id (:id library)})
|
||||
revn-sur (:revn (th/db-get :file {:id (:id survivor)}))
|
||||
revn-doom (:revn (th/db-get :file {:id (:id doomed)}))]
|
||||
(let [out (th/command! {::th/type :delete-project
|
||||
::rpc/profile-id (:id profile)
|
||||
:id (:id proj-a)})]
|
||||
(t/is (nil? (:error out))))
|
||||
(th/run-pending-jobs)
|
||||
;; The survivor absorbed the library; its relation is gone.
|
||||
(t/is (> (:revn (th/db-get :file {:id (:id survivor)} {::db/remove-deleted false}))
|
||||
revn-sur))
|
||||
(t/is (empty? (th/db-query :file-library-rel {:file-id (:id survivor)})))
|
||||
;; The doomed file did not absorb anything.
|
||||
(t/is (= revn-doom (:revn (th/db-get :file {:id (:id doomed)} {::db/remove-deleted false}))))))
|
||||
|
||||
(t/deftest delete-team-skips-absorb-for-files-dying-together
|
||||
;; Team deletions skip absorb entirely (*team-deletion*): a library
|
||||
;; in project A used from project B dies with the team, so absorb
|
||||
;; never runs and the relations stay until the hard delete.
|
||||
(let [profile (th/create-profile* 4 {:is-active true})
|
||||
team (th/create-team* 4 {:profile-id (:id profile)})
|
||||
proj-a (th/create-project* 4 {:profile-id (:id profile) :team-id (:id team)})
|
||||
proj-b (th/create-project* 5 {:profile-id (:id profile) :team-id (:id team)})
|
||||
library (th/create-file* 6 {:profile-id (:id profile)
|
||||
:project-id (:id proj-a)
|
||||
:is-shared true})
|
||||
file (th/create-file* 7 {:profile-id (:id profile)
|
||||
:project-id (:id proj-b)})
|
||||
_ (th/link-file-to-library* {:file-id (:id file)
|
||||
:library-id (:id library)})
|
||||
revn (:revn (th/db-get :file {:id (:id file)}))
|
||||
calls (atom 0)]
|
||||
(let [out (th/command! {::th/type :delete-team
|
||||
::rpc/profile-id (:id profile)
|
||||
:id (:id team)})]
|
||||
(t/is (nil? (:error out))))
|
||||
(with-redefs [files/absorb-library (fn [& _] (swap! calls inc) nil)]
|
||||
(th/run-pending-jobs))
|
||||
(t/is (zero? @calls))
|
||||
(t/is (not (empty? (th/db-query :file-library-rel {:library-file-id (:id library)}))))
|
||||
(t/is (= revn (:revn (th/db-get :file {:id (:id file)} {::db/remove-deleted false}))))
|
||||
;; The hard delete cascades the relations away with their files.
|
||||
(th/run-task! :objects-gc {:skip-delay true})
|
||||
(t/is (nil? (th/db-get :file {:id (:id file)} {::db/remove-deleted false})))
|
||||
(t/is (empty? (th/db-query :file-library-rel {:library-file-id (:id library)})))))
|
||||
|
||||
(t/deftest unpublish-shared-library-absorbs-its-tokens-lib
|
||||
(let [profile (th/create-profile* 1)
|
||||
{:keys [library] :as prepared} (prepare-tokens-library profile)
|
||||
@ -3348,6 +3499,87 @@
|
||||
(t/is (= (:id team)
|
||||
(get-in (meta (:result out)) [:app.loggers.audit/props :team-id])))))
|
||||
|
||||
(defn- unused-media-fixture
|
||||
"A file with a media object nothing references, so `:file-gc` trims it."
|
||||
[i]
|
||||
(let [profile (th/create-profile* i)
|
||||
file (th/create-file* i {:profile-id (:id profile)
|
||||
:project-id (:default-project-id profile)
|
||||
:is-shared false})
|
||||
so-id (uuid/next)]
|
||||
(th/db-insert! :storage-object {:id so-id :size 10 :backend "test"})
|
||||
(th/create-file-media-object* {:file-id (:id file)
|
||||
:media-id so-id
|
||||
:name "unused-media"
|
||||
:mtype "image/svg+xml"})
|
||||
[profile file]))
|
||||
|
||||
(t/deftest file-gc-emits-an-audit-event-when-it-trims-media
|
||||
;; Trimming media is a destructive action no RPC reports: the task emits
|
||||
;; `file-gc` with the file and its team so the audit trail records it.
|
||||
(let [[profile file] (unused-media-fixture 1)]
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}]
|
||||
(th/with-audit-config (conj cf/flags :nexus)
|
||||
(t/is (true? (th/run-task! :file-gc {:file-id (:id file)})))
|
||||
(let [event (->> (:call-args-list @audit-mock)
|
||||
(map second)
|
||||
(filter #(= "file-gc" (:name %)))
|
||||
first)]
|
||||
(t/is (some? event))
|
||||
(t/is (= (:id file) (get-in event [:props :file-id])))
|
||||
(t/is (= (:default-team-id profile) (get-in event [:props :team-id])))
|
||||
(t/is (= "cron-task" (get-in event [:context :triggered-by]))))))))
|
||||
|
||||
(t/deftest file-gc-does-not-emit-an-audit-event-with-the-flag-off
|
||||
(let [[_ file] (unused-media-fixture 2)]
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}]
|
||||
(th/with-audit-config (disj cf/flags :nexus)
|
||||
(t/is (true? (th/run-task! :file-gc {:file-id (:id file)})))
|
||||
(t/is (empty? (:call-args-list @audit-mock)))))))
|
||||
|
||||
(t/deftest file-gc-keeps-unused-media-with-the-retention-flag
|
||||
;; A kept revision row can name a media object that the head no longer
|
||||
;; names, and file-gc does not read revision rows, so the flag turns its
|
||||
;; media clean-up off.
|
||||
(let [[_ file-1] (unused-media-fixture 1)
|
||||
[_ file-2] (unused-media-fixture 2)
|
||||
media-row (fn [file] (th/db-get :file-media-object {:file-id (:id file)}
|
||||
{::db/remove-deleted false}))]
|
||||
(t/is (true? (th/run-task! :file-gc {:file-id (:id file-1)})))
|
||||
(with-redefs [cf/flags (conj cf/flags :file-change-retention)]
|
||||
(t/is (true? (th/run-task! :file-gc {:file-id (:id file-2)}))))
|
||||
|
||||
(t/is (ct/inst? (:deleted-at (media-row file-1))))
|
||||
(t/is (nil? (:deleted-at (media-row file-2))))
|
||||
(t/is (true? (:has-media-trimmed (th/db-get :file {:id (:id file-2)}))))))
|
||||
|
||||
(t/deftest update-file-audit-event-carries-the-file-change-id
|
||||
;; The event says that a file was saved, but not what changed: the
|
||||
;; change row (xlog) is the only place with that, so its id travels
|
||||
;; in the event. A consumer needs it to read the change (with
|
||||
;; `::db/remove-deleted false`: the row is born with its `deleted_at`
|
||||
;; in the future).
|
||||
(let [profile (th/create-profile* 1 {:is-active true})
|
||||
file (th/create-file* 1 {:profile-id (:id profile)
|
||||
:project-id (:default-project-id profile)
|
||||
:is-shared false})
|
||||
result (update-file! :file-id (:id file)
|
||||
:profile-id (:id profile)
|
||||
:revn 0
|
||||
:changes [])
|
||||
props (get-in (meta result) [:app.loggers.audit/replace-props])
|
||||
change-id (:file-change-id props)
|
||||
rows (th/db-query :file-change {:file-id (:id file)})
|
||||
;; The save writes two change rows: the change itself (no
|
||||
;; label) and the snapshot of the previous state (with one).
|
||||
changes (remove :label rows)]
|
||||
;; The event points at the change row of this save.
|
||||
(t/is (= 1 (count changes)))
|
||||
(t/is (= (:id (first changes)) change-id))
|
||||
;; The prop survives `clean-props`: a plain key with a uuid value.
|
||||
(t/is (= change-id
|
||||
(:file-change-id (audit/clean-props props))))))
|
||||
|
||||
(t/deftest get-team-shared-files-skips-files-that-fail-to-load
|
||||
(let [profile (th/create-profile* 1)
|
||||
file1 (th/create-file* 1 {:profile-id (:id profile)
|
||||
|
||||
@ -13,6 +13,7 @@
|
||||
[app.common.logging :as l]
|
||||
[app.common.uuid :as uuid]
|
||||
[app.config :as cf]
|
||||
[app.loggers.audit :as-alias audit]
|
||||
[app.msgbus :as mbus]
|
||||
[app.rpc :as-alias rpc]
|
||||
[app.util.blob :as blob]
|
||||
@ -147,6 +148,59 @@
|
||||
(t/is (= 1 (count (get-file-changes ctx))))
|
||||
(t/is (empty? messages))))
|
||||
|
||||
(t/deftest update-file-audit-event-carries-the-change-id-only-when-written
|
||||
;; `:file-change-id` points at the revision row of the save. A save whose
|
||||
;; first commit is rejected and keeps no prefix writes no row, so its
|
||||
;; event must not carry the id: an id that resolves to nothing cannot be
|
||||
;; told apart from a row the GC already purged. `command-raw!` keeps the
|
||||
;; result metadata, which is where the command leaves the audit props.
|
||||
(let [{:keys [profile-id file-id page-id] :as ctx} (create-file)
|
||||
shape (uuid/next)
|
||||
save! (fn [params]
|
||||
(th/command-raw! (merge {::th/type :update-file
|
||||
::rpc/profile-id profile-id
|
||||
:id file-id
|
||||
:session-id (uuid/next)
|
||||
:vern 0
|
||||
:features cfeat/supported-features}
|
||||
params)))
|
||||
props (fn [out] (::audit/replace-props (meta out)))]
|
||||
|
||||
(t/testing "a save that writes the revision carries its id"
|
||||
(let [out (save! {:revn 0
|
||||
:changes [(th/add-rect-change page-id shape)]})
|
||||
id (:file-change-id (props out))]
|
||||
(t/is (uuid? id))
|
||||
(t/is (= [id] (mapv :id (th/db-query :file-change {:id id}))))
|
||||
(t/is (= "saved" (:outcome (props out))))
|
||||
(t/is (inst? (:modified-at (props out))))))
|
||||
|
||||
(t/testing "a fully rejected save carries no id and says so"
|
||||
(let [commit1 (commit (break-parent page-id shape))
|
||||
commit2 (commit (th/add-rect-change page-id (uuid/next)))
|
||||
out (save! {:revn 1
|
||||
:changes-with-metadata [commit1 commit2]})]
|
||||
(t/is (some? (:rejected out)))
|
||||
(t/is (= 1 (:revn (th/db-get :file {:id file-id}))))
|
||||
(t/is (= 1 (count (get-file-changes ctx))))
|
||||
(t/is (not (contains? (props out) :file-change-id)))
|
||||
(t/is (= "rejected" (:outcome (props out))))
|
||||
;; The rejected save does not persist: the event mirrors the row.
|
||||
(t/is (= (inst-ms (:modified-at (th/db-get :file {:id file-id})))
|
||||
(inst-ms (:modified-at (props out)))))))
|
||||
|
||||
(t/testing "a partially rejected save says so and carries the id"
|
||||
(let [commit1 (commit (th/add-rect-change page-id (uuid/next)))
|
||||
commit2 (commit (conflicting-page-change))
|
||||
out (save! {:revn 1
|
||||
:changes-with-metadata [commit1 commit2]})
|
||||
row (th/db-get :file {:id file-id})]
|
||||
(t/is (some? (:rejected out)))
|
||||
(t/is (= "partial" (:outcome (props out))))
|
||||
(t/is (uuid? (:file-change-id (props out))))
|
||||
(t/is (= (inst-ms (:modified-at row))
|
||||
(inst-ms (:modified-at (props out)))))))))
|
||||
|
||||
(t/deftest rejects-a-commit-that-throws-while-applied
|
||||
(let [{:keys [page-id] :as ctx} (create-file)
|
||||
shape-a (uuid/next)
|
||||
|
||||
@ -2135,7 +2135,7 @@
|
||||
|
||||
(t/deftest push-audit-events-stores-backend-source
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}]
|
||||
(binding [cf/flags #{:audit-log}]
|
||||
(th/with-audit-config #{:nexus}
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
params {::th/type :push-audit-events
|
||||
:events [{:name "test-action"
|
||||
@ -2160,7 +2160,7 @@
|
||||
|
||||
(t/deftest push-audit-events-type-defaults-to-action
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}]
|
||||
(binding [cf/flags #{:audit-log}]
|
||||
(th/with-audit-config #{:nexus}
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
params {::th/type :push-audit-events
|
||||
:events [{:name "no-type-event"
|
||||
@ -2178,7 +2178,7 @@
|
||||
|
||||
(t/deftest push-audit-events-multiple-events
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}]
|
||||
(binding [cf/flags #{:audit-log}]
|
||||
(th/with-audit-config #{:nexus}
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
params {::th/type :push-audit-events
|
||||
:events [{:name "event-a"
|
||||
@ -2201,7 +2201,7 @@
|
||||
|
||||
(t/deftest push-audit-events-merges-context
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}]
|
||||
(binding [cf/flags #{:audit-log}]
|
||||
(th/with-audit-config #{:nexus}
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
params {::th/type :push-audit-events
|
||||
:events [{:name "context-test"
|
||||
@ -2227,7 +2227,7 @@
|
||||
;; keyword; the stored initiator must be a plain string, and a
|
||||
;; caller-supplied initiator must never survive (server context wins).
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}]
|
||||
(binding [cf/flags #{:audit-log}]
|
||||
(th/with-audit-config #{:nexus}
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
params {::th/type :push-audit-events
|
||||
:events [{:name "context-test"
|
||||
|
||||
@ -532,6 +532,71 @@
|
||||
(t/is (= (:library-file-id item) (:id file3)))))))
|
||||
|
||||
|
||||
(t/deftest move-files-from-multiple-teams-is-rejected
|
||||
;; A single move carries one origin team in its props: mixed origins
|
||||
;; are refused before anything moves, and the move event is not
|
||||
;; submitted. The fixtures are created with the flag off so their own
|
||||
;; membership events do not pollute the mock.
|
||||
(let [profile (th/create-profile* 1 {:is-active true})
|
||||
team (th/create-team* 1 {:profile-id (:id profile)})
|
||||
project1 (th/create-project* 1 {:team-id (:default-team-id profile)
|
||||
:profile-id (:id profile)})
|
||||
project2 (th/create-project* 2 {:team-id (:id team)
|
||||
:profile-id (:id profile)})
|
||||
file1 (th/create-file* 1 {:profile-id (:id profile)
|
||||
:project-id (:id project1)})
|
||||
file3 (th/create-file* 3 {:profile-id (:id profile)
|
||||
:project-id (:id project2)})]
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}]
|
||||
(th/with-audit-config (conj cf/flags :nexus)
|
||||
(let [out (th/command! {::th/type :move-files
|
||||
::rpc/profile-id (:id profile)
|
||||
:project-id (:id project2)
|
||||
:ids #{(:id file1) (:id file3)}})
|
||||
error (:error out)]
|
||||
(t/is (th/ex-info? error))
|
||||
(t/is (th/ex-of-type? error :validation))
|
||||
(t/is (th/ex-of-code? error :cant-move-from-multiple-teams))))
|
||||
(t/is (not-any? #(= "move-files" (:name %))
|
||||
(map second (:call-args-list @audit-mock)))))
|
||||
(t/is (= 1 (count (th/db-query :file {:project-id (:id project1)}))))
|
||||
(t/is (= 1 (count (th/db-query :file {:project-id (:id project2)}))))))
|
||||
|
||||
|
||||
(t/deftest move-files-carries-both-teams-in-the-event
|
||||
;; The event carries the origin and destination team, so a consumer can
|
||||
;; key on them without joining production tables.
|
||||
(let [profile (th/create-profile* 1 {:is-active true})
|
||||
team (th/create-team* 1 {:profile-id (:id profile)})
|
||||
project1 (th/create-project* 1 {:team-id (:default-team-id profile)
|
||||
:profile-id (:id profile)})
|
||||
project2 (th/create-project* 2 {:team-id (:id team)
|
||||
:profile-id (:id profile)})
|
||||
file (th/create-file* 1 {:profile-id (:id profile)
|
||||
:project-id (:id project1)})
|
||||
raw (th/command-raw! {::th/type :move-files
|
||||
::rpc/profile-id (:id profile)
|
||||
:project-id (:id project2)
|
||||
:ids #{(:id file)}})
|
||||
props (get-in (meta raw) [:app.loggers.audit/props])]
|
||||
(t/is (= (:default-team-id profile) (:source-team-id props)))
|
||||
(t/is (= (:id team) (:dest-team-id props)))))
|
||||
|
||||
|
||||
(t/deftest move-project-carries-both-teams-in-the-event
|
||||
(let [profile (th/create-profile* 1 {:is-active true})
|
||||
team (th/create-team* 2 {:profile-id (:id profile)})
|
||||
project (th/create-project* 1 {:team-id (:default-team-id profile)
|
||||
:profile-id (:id profile)})
|
||||
raw (th/command-raw! {::th/type :move-project
|
||||
::rpc/profile-id (:id profile)
|
||||
:project-id (:id project)
|
||||
:team-id (:id team)})
|
||||
props (get-in (meta raw) [:app.loggers.audit/props])]
|
||||
(t/is (= (:default-team-id profile) (:source-team-id props)))
|
||||
(t/is (= (:id team) (:dest-team-id props)))))
|
||||
|
||||
|
||||
(t/deftest move-library-to-other-team
|
||||
(let [profile (th/create-profile* 1 {:is-active true})
|
||||
team (th/create-team* 1 {:profile-id (:id profile)})
|
||||
@ -723,7 +788,7 @@
|
||||
;; The per-file `create-file` events are built by hand, so they do not pass
|
||||
;; through the generic resolver: the command must copy the team id itself.
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}]
|
||||
(with-redefs [cf/flags (conj cf/flags :audit-log)]
|
||||
(th/with-audit-config (conj cf/flags :nexus)
|
||||
(let [prof (th/create-profile* 1 {:is-active true})
|
||||
team (th/create-team* 1 {:profile-id (:id prof)})
|
||||
project (th/create-project* 1 {:profile-id (:id prof)
|
||||
@ -744,7 +809,7 @@
|
||||
;; The duplicated project and files are created by internal calls, so the
|
||||
;; command emits one create event per entity for a local projection.
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}]
|
||||
(with-redefs [cf/flags (conj cf/flags :audit-log)]
|
||||
(th/with-audit-config (conj cf/flags :nexus)
|
||||
(let [profile (th/create-profile* 1 {:is-active true})
|
||||
team (th/create-team* 52 {:profile-id (:id profile)})
|
||||
project (th/create-project* 52 {:profile-id (:id profile)
|
||||
|
||||
@ -2491,7 +2491,7 @@
|
||||
;; Team creation adds the owner through add-profile-to-team!, which used to
|
||||
;; leave no membership event.
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}]
|
||||
(with-redefs [cf/flags (conj cf/flags :audit-log)]
|
||||
(th/with-audit-config (conj cf/flags :nexus)
|
||||
(let [owner (th/create-profile* 1 {:is-active true})
|
||||
team (th/create-team* 21 {:profile-id (:id owner)})
|
||||
events (->> (:call-args-list @audit-mock)
|
||||
@ -2506,7 +2506,7 @@
|
||||
;; Adding an existing member with on-conflict-do-nothing inserts nothing and
|
||||
;; must not report a membership that already exists.
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}]
|
||||
(with-redefs [cf/flags (conj cf/flags :audit-log)]
|
||||
(th/with-audit-config (conj cf/flags :nexus)
|
||||
(let [owner (th/create-profile* 1 {:is-active true})
|
||||
team (th/create-team* 22 {:profile-id (:id owner)})]
|
||||
(th/reset-mock! audit-mock)
|
||||
@ -2518,7 +2518,7 @@
|
||||
{::db/on-conflict-do-nothing? true})))
|
||||
(t/is (empty? (:call-args-list @audit-mock)))))))
|
||||
|
||||
(t/deftest no-add-team-member-event-without-audit-log
|
||||
(t/deftest no-add-team-member-event-without-a-consumer
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}]
|
||||
(let [owner (th/create-profile* 1 {:is-active true})
|
||||
_team (th/create-team* 23 {:profile-id (:id owner)})
|
||||
@ -2531,7 +2531,7 @@
|
||||
;; `team_profile_rel` has no role column, so no caller can pass it: the
|
||||
;; emitter derives it from the flags, inverse of permissions-for-role.
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}]
|
||||
(with-redefs [cf/flags (conj cf/flags :audit-log)]
|
||||
(th/with-audit-config (conj cf/flags :nexus)
|
||||
(let [owner (th/create-profile* 1 {:is-active true})
|
||||
team (th/create-team* 26 {:profile-id (:id owner)})
|
||||
editor (th/create-profile* 27 {:is-active true})
|
||||
@ -2560,7 +2560,7 @@
|
||||
;; Sibling events built by `prepare-rpc-event` carry the request context;
|
||||
;; the hand-built membership event must not lose it when rpc params travel.
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}]
|
||||
(with-redefs [cf/flags (conj cf/flags :audit-log)]
|
||||
(th/with-audit-config (conj cf/flags :nexus)
|
||||
(let [owner (th/create-profile* 1 {:is-active true})
|
||||
member (th/create-profile* 2 {:is-active true})
|
||||
team (th/create-team* 30 {:profile-id (:id owner)})]
|
||||
@ -2585,7 +2585,7 @@
|
||||
;; With email verification off, inviting an existing profile adds the
|
||||
;; member directly; the event belongs to the inviter, not the member.
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}]
|
||||
(with-redefs [cf/flags #{:audit-log}]
|
||||
(th/with-audit-config #{:nexus}
|
||||
(let [owner (th/create-profile* 1 {:is-active true})
|
||||
member (th/create-profile* 2 {:is-active true})
|
||||
team (th/create-team* 31 {:profile-id (:id owner)})]
|
||||
@ -2640,7 +2640,7 @@
|
||||
;; the manual event must carry its id for a projection to key it.
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}
|
||||
email-mock {:target 'app.email/send-template :return nil}]
|
||||
(with-redefs [cf/flags (conj cf/flags :audit-log)]
|
||||
(th/with-audit-config (conj cf/flags :nexus)
|
||||
(let [profile (th/create-profile* 1 {:is-active true})
|
||||
out (th/command-raw! {::th/type :create-team-with-invitations
|
||||
::rpc/profile-id (:id profile)
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@ -7,6 +7,8 @@
|
||||
(ns backend-tests.srepl-main-test
|
||||
(:require
|
||||
[app.common.time :as ct]
|
||||
[app.common.uuid :as uuid]
|
||||
[app.db :as db]
|
||||
[app.jobs :as jobs]
|
||||
[app.srepl.main :as srepl]
|
||||
[app.system :as sys]
|
||||
@ -113,6 +115,26 @@
|
||||
(t/is (= 0 (:submitted result)))
|
||||
(t/is (empty? @submitted))))
|
||||
|
||||
(t/deftest cascade-counts-only-the-submits-that-happen
|
||||
;; `::jobs/dedupe` returns nil when a not-started job already covers the
|
||||
;; row, so a second pass finds it but enqueues nothing.
|
||||
(let [profile (create-deleted-profile! 998 (ct/in-past {:days 10}))
|
||||
first-run (#'srepl/submit-deleted-cascade
|
||||
th/*system* {:table "profile" :object :profile})
|
||||
second-run (#'srepl/submit-deleted-cascade
|
||||
th/*system* {:table "profile" :object :profile})
|
||||
dry-run (#'srepl/submit-deleted-cascade
|
||||
th/*system* {:table "profile" :object :profile :dry-run? true})]
|
||||
(t/is (= 1 (:found first-run)))
|
||||
(t/is (= 1 (:submitted first-run)))
|
||||
(t/testing "the second pass finds the row but dedupe drops the submit"
|
||||
(t/is (= 1 (:found second-run)))
|
||||
(t/is (= 0 (:submitted second-run))))
|
||||
(t/testing "dry-run still counts without enqueueing"
|
||||
(t/is (= 1 (:found dry-run)))
|
||||
(t/is (= 0 (:submitted dry-run))))
|
||||
(t/is (= 1 (count (th/db-query :job {:label (str (:id profile))}))))))
|
||||
|
||||
(t/deftest public-helpers-read-their-own-table
|
||||
(let [profile (th/create-profile* 997)
|
||||
team (th/create-team* 997 {:profile-id (:id profile)})
|
||||
@ -171,6 +193,44 @@
|
||||
(t/is (nil? (:profile-id event)))
|
||||
(t/is (= (:id profile) (get-in event [:props :id]))))))))
|
||||
|
||||
(t/deftest delete-file-carries-its-team
|
||||
;; The SREPL event must carry its team, so a consumer can key on it
|
||||
;; without joining production tables.
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/insert :return nil}
|
||||
wrk-mock {:target 'app.jobs/invoke :return nil}]
|
||||
(let [profile (th/create-profile* 1 {:is-active true})
|
||||
team-id (:default-team-id profile)
|
||||
proj (th/create-project* 1 {:profile-id (:id profile) :team-id team-id})
|
||||
file (th/create-file* 1 {:profile-id (:id profile) :project-id (:id proj)})]
|
||||
(with-redefs [sys/system th/*system*]
|
||||
(srepl/delete-file! (str (:id file))))
|
||||
(let [event (-> @audit-mock :call-args-list first second)]
|
||||
(t/is (= "delete-file" (:name event)))
|
||||
(t/is (= team-id (get-in event [:props :team-id])))))))
|
||||
|
||||
(t/deftest restore-file-carries-its-team
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/insert :return nil}]
|
||||
(let [profile (th/create-profile* 1 {:is-active true})
|
||||
proj (th/create-project* 1 {:profile-id (:id profile)
|
||||
:team-id (:default-team-id profile)})
|
||||
file (th/create-file* 1 {:profile-id (:id profile)
|
||||
:project-id (:id proj)})]
|
||||
(with-redefs [sys/system th/*system*]
|
||||
(srepl/restore-file! (str (:id file))))
|
||||
(let [event (-> @audit-mock :call-args-list first second)]
|
||||
(t/is (= "restore-file" (:name event)))
|
||||
(t/is (= (:default-team-id profile) (get-in event [:props :team-id])))))))
|
||||
|
||||
(t/deftest restore-team-carries-its-team
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/insert :return nil}]
|
||||
(let [profile (th/create-profile* 1 {:is-active true})
|
||||
team (th/create-team* 3 {:profile-id (:id profile)})]
|
||||
(with-redefs [sys/system th/*system*]
|
||||
(srepl/restore-team! (str (:id team))))
|
||||
(let [event (-> @audit-mock :call-args-list first second)]
|
||||
(t/is (= "restore-team" (:name event)))
|
||||
(t/is (= (:id team) (get-in event [:props :team-id])))))))
|
||||
|
||||
(t/deftest delete-team-emits-identifiable-audit-event
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/insert :return nil}
|
||||
wrk-mock {:target 'app.jobs/invoke :return nil}]
|
||||
@ -194,7 +254,11 @@
|
||||
wrk-mock {:target 'app.jobs/invoke :return nil}]
|
||||
(with-redefs [sys/system th/*system*]
|
||||
(let [profile (th/create-profile* 1 {:is-active true})]
|
||||
(srepl/restore-profile! (str (:id profile)))
|
||||
;; The restore path only runs for a soft-deleted profile; a direct
|
||||
;; stamp is enough here, the cascade has its own test below.
|
||||
(th/db-exec! ["UPDATE profile SET deleted_at = ? WHERE id = ?"
|
||||
(ct/now) (:id profile)])
|
||||
(t/is (= :restored (srepl/restore-profile! (str (:id profile)))))
|
||||
(let [events (->> (:call-args-list @audit-mock)
|
||||
(map second))]
|
||||
(t/is (= 1 (count events)))
|
||||
@ -206,6 +270,61 @@
|
||||
(t/is (nil? (:profile-id event)))
|
||||
(t/is (= (:id profile) (get-in event [:props :id])))))))))
|
||||
|
||||
(t/deftest restore-profile-is-idempotent-and-keeps-separately-deleted-content
|
||||
;; A second restore on an active profile is a no-op. Running the cascade
|
||||
;; over live teams would un-delete a font variant deleted on its own.
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/insert :return nil}]
|
||||
(let [profile (th/create-profile* 1 {:is-active true})
|
||||
team (:default-team-id profile)
|
||||
variant (uuid/next)]
|
||||
(th/db-insert! :team-font-variant
|
||||
{:id variant
|
||||
:team-id team
|
||||
:profile-id (:id profile)
|
||||
:font-id (uuid/next)
|
||||
:font-family "somefont"
|
||||
:font-weight 400
|
||||
:font-style "normal"
|
||||
:deleted-at (ct/in-past {:days 1})})
|
||||
(with-redefs [sys/system th/*system*]
|
||||
(t/is (= :already-active (srepl/restore-profile! (str (:id profile)))))
|
||||
(t/is (= :already-active (srepl/restore-profile! (str (:id profile))))))
|
||||
(t/is (some? (:deleted-at (th/db-get :team-font-variant
|
||||
{:id variant}
|
||||
{::db/remove-deleted false}))))
|
||||
(t/is (empty? (:call-args-list @audit-mock))))))
|
||||
|
||||
(t/deftest restore-profile-restores-only-the-teams-the-cascade-took
|
||||
;; The profile cascade stamps its live owned teams with the profile's own
|
||||
;; instant; restoring the profile brings back exactly those, not a team the
|
||||
;; owner had already deleted on its own.
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/insert :return nil}]
|
||||
(let [profile (th/create-profile* 1 {:is-active true})
|
||||
owned (:default-team-id profile)
|
||||
unrelated (:id (th/create-team* 2 {:profile-id (:id profile)}))]
|
||||
;; A team owned by the profile, deleted on its own long before.
|
||||
(th/db-exec! ["UPDATE team SET deleted_at = ? WHERE id = ?"
|
||||
(ct/in-past {:days 30}) unrelated])
|
||||
;; The real cascade marks the profile and its live owned teams.
|
||||
(th/run-task! :delete-object {:object :profile
|
||||
:id (:id profile)
|
||||
:deleted-at (ct/now)})
|
||||
(t/is (some? (:deleted-at (th/db-get :profile {:id (:id profile)}
|
||||
{::db/remove-deleted false}))))
|
||||
(t/is (some? (:deleted-at (th/db-get :team {:id owned}
|
||||
{::db/remove-deleted false}))))
|
||||
(with-redefs [sys/system th/*system*]
|
||||
(srepl/restore-profile! (str (:id profile))))
|
||||
(t/is (nil? (:deleted-at (th/db-get :profile {:id (:id profile)}
|
||||
{::db/remove-deleted false}))))
|
||||
(t/is (nil? (:deleted-at (th/db-get :team {:id owned}
|
||||
{::db/remove-deleted false}))))
|
||||
(t/is (some? (:deleted-at (th/db-get :team {:id unrelated}
|
||||
{::db/remove-deleted false}))))
|
||||
(let [event (-> @audit-mock :call-args-list first second)]
|
||||
(t/is (= "restore-profile" (:name event)))
|
||||
(t/is (= [owned] (get-in event [:props :team-ids])))))))
|
||||
|
||||
(t/deftest update-profile-email-changes-email
|
||||
(with-mocks [audit-mock {:target 'app.loggers.audit/insert :return nil}]
|
||||
(with-redefs [sys/system th/*system*]
|
||||
|
||||
@ -10,10 +10,6 @@
|
||||
[app.common.uuid :as uuid]
|
||||
[app.config :as cf]
|
||||
[app.db :as db]
|
||||
[app.loggers.audit :as audit]
|
||||
[app.loggers.audit.consumer-task :as at]
|
||||
[app.tasks.telemetry :as telemetry]
|
||||
[app.util.json :as json]
|
||||
[backend-tests.helpers :as th]
|
||||
[clojure.test :as t]
|
||||
[mockery.core :refer [with-mocks]]
|
||||
@ -31,34 +27,6 @@
|
||||
|
||||
(t/use-fixtures :each test-fixture)
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; HELPERS
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(defn- insert-telemetry-row!
|
||||
"Insert a single anonymised audit_log row as the telemetry mode does."
|
||||
([name] (insert-telemetry-row! name {}))
|
||||
([name {:keys [tracked-at created-at source]
|
||||
:or {tracked-at (ct/now)
|
||||
created-at (ct/now)
|
||||
source "telemetry:backend"}}]
|
||||
(th/db-insert! :audit-log
|
||||
{:id (uuid/next)
|
||||
:name name
|
||||
:type "action"
|
||||
:source source
|
||||
:profile-id uuid/zero
|
||||
:ip-addr (db/inet "0.0.0.0")
|
||||
:props (db/tjson {})
|
||||
:context (db/tjson {})
|
||||
:tracked-at tracked-at
|
||||
:created-at created-at})))
|
||||
|
||||
(defn- count-telemetry-rows []
|
||||
(-> (th/db-exec-one! ["SELECT count(*) AS cnt FROM audit_log WHERE source IN ('telemetry:backend', 'telemetry:frontend')"])
|
||||
:cnt
|
||||
long))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; STATS / REPORT STRUCTURE TESTS (existing behaviour, extended)
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
@ -105,7 +73,7 @@
|
||||
;; make-legacy-request must not be called at all.
|
||||
(with-mocks [mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}]
|
||||
(with-redefs [cf/flags #{}]
|
||||
(th/with-audit-config #{}
|
||||
(th/create-profile* 1 {:is-active true})
|
||||
(th/run-task! :telemetry {:send? true})
|
||||
(t/is (not (:called? @mock))))))
|
||||
@ -115,7 +83,7 @@
|
||||
;; make-legacy-request is called once with only subscriptions + version (no stats).
|
||||
(with-mocks [mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}]
|
||||
(with-redefs [cf/flags #{}]
|
||||
(th/with-audit-config #{}
|
||||
(let [prof (th/create-profile* 1 {:is-active true
|
||||
:props {:newsletter-updates true}})]
|
||||
(th/run-task! :telemetry {:send? true})
|
||||
@ -131,7 +99,7 @@
|
||||
;; Passing send?=false must suppress all HTTP calls even when enabled.
|
||||
(with-mocks [mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}]
|
||||
(with-redefs [cf/flags #{:telemetry}]
|
||||
(th/with-audit-config #{:telemetry}
|
||||
(th/create-profile* 1 {:is-active true})
|
||||
(th/run-task! :telemetry {:send? false :enabled? true})
|
||||
(t/is (not (:called? @mock))))))
|
||||
@ -157,8 +125,7 @@
|
||||
"https://penpot.dev"])]
|
||||
(with-mocks [mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}]
|
||||
(with-redefs [cf/flags #{}
|
||||
cf/config (assoc cf/config :public-uri public-uri)]
|
||||
(th/with-audit-config #{} [cf/config (assoc cf/config :public-uri public-uri)]
|
||||
(th/create-profile* (+ 10 idx) {:is-active true
|
||||
:props {:newsletter-updates true}})
|
||||
(th/run-task! :telemetry {:send? true})
|
||||
@ -169,8 +136,7 @@
|
||||
;; since nothing is going to be sent.
|
||||
(with-mocks [mock {:target 'app.tasks.telemetry/get-subscriptions
|
||||
:return []}]
|
||||
(with-redefs [cf/flags #{}
|
||||
cf/config (assoc cf/config :public-uri "https://design.penpot.app")]
|
||||
(th/with-audit-config #{} [cf/config (assoc cf/config :public-uri "https://design.penpot.app")]
|
||||
(th/create-profile* 1 {:is-active true
|
||||
:props {:newsletter-updates true}})
|
||||
(th/run-task! :telemetry {:send? true})
|
||||
@ -181,254 +147,12 @@
|
||||
;; only the implicit newsletter fallback is excluded.
|
||||
(with-mocks [mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}]
|
||||
(with-redefs [cf/flags #{:telemetry}
|
||||
cf/config (assoc cf/config :public-uri "https://design.penpot.app")]
|
||||
(th/with-audit-config #{:telemetry} [cf/config (assoc cf/config :public-uri "https://design.penpot.app")]
|
||||
(th/create-profile* 1 {:is-active true
|
||||
:props {:newsletter-updates true}})
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
(t/is (:called? @mock)))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; AUDIT-EVENT BATCH COLLECTION TESTS
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(t/deftest test-no-audit-events-no-batch-call
|
||||
;; When telemetry is enabled but there are no audit_log rows with
|
||||
;; source='telemetry', the batch send path must not be invoked.
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}
|
||||
batch-mock {:target 'app.tasks.telemetry/send-event-batch
|
||||
:return true}]
|
||||
(with-redefs [cf/flags #{:telemetry}]
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
(t/is (:called? @legacy-mock))
|
||||
(t/is (not (:called? @batch-mock))))))
|
||||
|
||||
(t/deftest test-audit-events-sent-and-deleted-on-success
|
||||
;; Happy path: telemetry rows are collected, shipped as a batch and
|
||||
;; deleted from the table when the endpoint returns success.
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}
|
||||
batch-mock {:target 'app.tasks.telemetry/send-event-batch
|
||||
:return true}]
|
||||
(with-redefs [cf/flags #{:telemetry}]
|
||||
(insert-telemetry-row! "navigate")
|
||||
(insert-telemetry-row! "create-file")
|
||||
(insert-telemetry-row! "update-file")
|
||||
|
||||
(t/is (= 3 (count-telemetry-rows)))
|
||||
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
|
||||
;; batch send was called at least once
|
||||
(t/is (:called? @batch-mock))
|
||||
|
||||
;; all rows deleted after successful send
|
||||
(t/is (= 0 (count-telemetry-rows))))))
|
||||
|
||||
(t/deftest test-audit-events-kept-on-batch-failure
|
||||
;; When the batch endpoint returns failure the rows must be retained
|
||||
;; so the next scheduled run can retry.
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}
|
||||
batch-mock {:target 'app.tasks.telemetry/send-event-batch
|
||||
:return false}]
|
||||
(with-redefs [cf/flags #{:telemetry}]
|
||||
(insert-telemetry-row! "navigate")
|
||||
(insert-telemetry-row! "create-file")
|
||||
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
|
||||
(t/is (:called? @batch-mock))
|
||||
;; rows still present — not deleted on failure
|
||||
(t/is (= 2 (count-telemetry-rows))))))
|
||||
|
||||
(t/deftest test-audit-events-not-collected-when-audit-log-flag-set
|
||||
;; When the :audit-log flag is active, mode C is disabled and the
|
||||
;; batch path must never run (audit-log owns those rows instead).
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}
|
||||
batch-mock {:target 'app.tasks.telemetry/send-event-batch
|
||||
:return true}]
|
||||
(with-redefs [cf/flags #{:telemetry :audit-log}]
|
||||
(insert-telemetry-row! "navigate")
|
||||
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
|
||||
(t/is (not (:called? @batch-mock)))
|
||||
;; row untouched
|
||||
(t/is (= 1 (count-telemetry-rows))))))
|
||||
|
||||
(t/deftest test-batch-payload-contains-required-fields
|
||||
;; Inspect the actual arguments forwarded to send-event-batch to
|
||||
;; verify the payload carries instance-id, version and events.
|
||||
(let [captured (atom nil)]
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}]
|
||||
(with-redefs [cf/flags #{:telemetry}
|
||||
telemetry/send-event-batch
|
||||
(fn [_cfg batch]
|
||||
(reset! captured batch)
|
||||
true)]
|
||||
(insert-telemetry-row! "navigate")
|
||||
(insert-telemetry-row! "create-file")
|
||||
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
|
||||
(t/is (some? @captured))
|
||||
(let [batch @captured]
|
||||
;; batch is a seq of event maps
|
||||
(t/is (seq batch))
|
||||
(t/is (= 2 (count batch)))
|
||||
;; each event has name, type, source — profile-id is preserved,
|
||||
;; props and ip-addr are stripped
|
||||
(let [ev (first batch)]
|
||||
(t/is (contains? ev :name))
|
||||
(t/is (contains? ev :type))
|
||||
(t/is (contains? ev :source))
|
||||
(t/is (contains? ev :profile-id))
|
||||
;; props are present but empty (stripped at ingest time)
|
||||
(t/is (= {} (:props ev)))
|
||||
(t/is (not (contains? ev :ip-addr)))))))))
|
||||
|
||||
(t/deftest test-batch-encoding-is-decodable
|
||||
;; Events are sent as a plain vector of raw event maps (no blob
|
||||
;; encoding): every batch must JSON round-trip unchanged, because
|
||||
;; the receiver coerces types from the plain JSON representation.
|
||||
(let [events [{:name "navigate" :type "action" :source "telemetry"
|
||||
:tracked-at (ct/now)}
|
||||
{:name "create-file" :type "action" :source "telemetry"
|
||||
:tracked-at (ct/now)}]
|
||||
encoded (json/encode-str {:events (vec events)})
|
||||
decoded (json/decode encoded)]
|
||||
(t/is (vector? (:events decoded)))
|
||||
(t/is (= (count events) (count (:events decoded))))
|
||||
(t/is (= "navigate" (:name (first (:events decoded)))))
|
||||
(t/is (= "create-file" (:name (second (:events decoded)))))))
|
||||
|
||||
(t/deftest test-multiple-batches-when-many-events
|
||||
;; Lower batch-size to 1 so that 3 events produce 3 separate
|
||||
;; HTTP requests and verify all are sent and all rows deleted.
|
||||
(let [call-count (atom 0)]
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}]
|
||||
(with-redefs [cf/flags #{:telemetry}
|
||||
telemetry/batch-size 1
|
||||
telemetry/send-event-batch
|
||||
(fn [_cfg _batch]
|
||||
(swap! call-count inc)
|
||||
true)]
|
||||
(insert-telemetry-row! "navigate")
|
||||
(insert-telemetry-row! "create-file")
|
||||
(insert-telemetry-row! "update-file")
|
||||
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
|
||||
;; Each event is fetched and sent in its own loop iteration
|
||||
(t/is (= 3 @call-count))
|
||||
;; All rows deleted after all iterations succeed
|
||||
(t/is (= 0 (count-telemetry-rows)))))))
|
||||
|
||||
(t/deftest test-partial-failure-stops-remaining-batches
|
||||
;; With batch-size 1, when the second send fails the loop stops.
|
||||
;; The first batch was already deleted; the two remaining rows
|
||||
;; are retained for the next run.
|
||||
(let [call-count (atom 0)]
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}]
|
||||
(with-redefs [cf/flags #{:telemetry}
|
||||
telemetry/batch-size 1
|
||||
telemetry/send-event-batch
|
||||
(fn [_cfg _batch]
|
||||
(swap! call-count inc)
|
||||
;; fail on the second call
|
||||
(not= 2 @call-count))]
|
||||
(insert-telemetry-row! "navigate")
|
||||
(insert-telemetry-row! "create-file")
|
||||
(insert-telemetry-row! "update-file")
|
||||
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
|
||||
;; Stopped at iteration 2 — third event never attempted
|
||||
(t/is (= 2 @call-count))
|
||||
;; First batch was deleted on success; 2 rows remain for retry
|
||||
(t/is (= 2 (count-telemetry-rows)))))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; GC / RETENTION-WINDOW TESTS
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(t/deftest test-gc-purges-events-older-than-7-days
|
||||
;; Insert events from 8 days ago (stale) and from today (fresh).
|
||||
;; After the task runs, stale events must be purged by GC and fresh
|
||||
;; ones shipped by the batch sender.
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}
|
||||
batch-mock {:target 'app.tasks.telemetry/send-event-batch
|
||||
:return true}]
|
||||
(with-redefs [cf/flags #{:telemetry}]
|
||||
(let [now (ct/now)
|
||||
eight-days (ct/minus now (ct/duration {:days 8}))]
|
||||
;; Stale events (older than 7 days)
|
||||
(insert-telemetry-row! "stale-1" {:created-at eight-days :tracked-at eight-days})
|
||||
(insert-telemetry-row! "stale-2" {:created-at eight-days :tracked-at eight-days})
|
||||
;; Fresh events (today)
|
||||
(insert-telemetry-row! "fresh-1" {:created-at now :tracked-at now})
|
||||
(insert-telemetry-row! "fresh-2" {:created-at now :tracked-at now})
|
||||
|
||||
(t/is (= 4 (count-telemetry-rows)))
|
||||
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
|
||||
;; GC purged the 2 stale rows, batch sender shipped the 2 fresh ones
|
||||
(t/is (= 0 (count-telemetry-rows)))))))
|
||||
|
||||
(t/deftest test-gc-keeps-events-within-7-day-window
|
||||
;; When all events are within the 7-day window, GC must not delete
|
||||
;; anything and all rows are forwarded to the batch sender.
|
||||
(let [batch-events (atom nil)]
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}]
|
||||
(with-redefs [cf/flags #{:telemetry}
|
||||
telemetry/send-event-batch
|
||||
(fn [_cfg batch]
|
||||
(reset! batch-events batch)
|
||||
true)]
|
||||
(let [six-days-ago (ct/minus (ct/now) (ct/duration {:days 6}))]
|
||||
(insert-telemetry-row! "recent-1" {:created-at six-days-ago :tracked-at six-days-ago})
|
||||
(insert-telemetry-row! "recent-2" {:created-at six-days-ago :tracked-at six-days-ago}))
|
||||
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
|
||||
;; Both events forwarded — GC left them alone
|
||||
(t/is (= 2 (count @batch-events)))
|
||||
(t/is (= 0 (count-telemetry-rows)))))))
|
||||
|
||||
(t/deftest test-gc-deletes-only-stale-events
|
||||
;; Insert a mix of stale (8 days old) and fresh (1 day old) events.
|
||||
;; After GC, only fresh events should remain for the batch sender.
|
||||
(let [batch-events (atom nil)]
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}]
|
||||
(with-redefs [cf/flags #{:telemetry}
|
||||
telemetry/send-event-batch
|
||||
(fn [_cfg batch]
|
||||
(reset! batch-events batch)
|
||||
true)]
|
||||
(let [eight-days (ct/minus (ct/now) (ct/duration {:days 8}))
|
||||
one-day (ct/minus (ct/now) (ct/duration {:days 1}))]
|
||||
(insert-telemetry-row! "stale" {:created-at eight-days :tracked-at eight-days})
|
||||
(insert-telemetry-row! "fresh" {:created-at one-day :tracked-at one-day}))
|
||||
|
||||
(t/is (= 2 (count-telemetry-rows)))
|
||||
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
|
||||
;; GC purged stale, batch shipped fresh
|
||||
(t/is (= 1 (count @batch-events)))
|
||||
(t/is (= "fresh" (:name (first @batch-events))))
|
||||
(t/is (= 0 (count-telemetry-rows)))))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; ANONYMITY TESTS
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
@ -438,7 +162,7 @@
|
||||
;; empty props, zeroed ip, profile-id=zero, source='telemetry'.
|
||||
;; Safe context fields (browser, os, version, etc.) are preserved
|
||||
;; but session-linking and access-token fields are stripped.
|
||||
(with-redefs [cf/flags #{:telemetry}]
|
||||
(th/with-audit-config #{:telemetry}
|
||||
(let [_prof (th/create-profile* 1 {:is-active true})
|
||||
safe-ctx {:browser "Chrome"
|
||||
:browser-version "120.0"
|
||||
@ -466,60 +190,10 @@
|
||||
;; profile-id is uuid/zero — not a real user id
|
||||
(t/is (= uuid/zero (:profile-id row)))))))
|
||||
|
||||
(t/deftest test-batch-events-contain-no-pii-fields
|
||||
;; The event maps forwarded to send-event-batch must not carry props,
|
||||
;; ip-addr or profile-id. Safe context fields (browser, os, etc.) may
|
||||
;; be present but session-linking keys must be absent.
|
||||
(let [captured-batch (atom nil)
|
||||
;; Insert a row that carries safe context (as the real path does)
|
||||
safe-ctx {:browser "Firefox" :browser-version "121.0"
|
||||
:os "macOS" :session "should-be-stripped"
|
||||
:external-session-id "also-stripped"}]
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}]
|
||||
(with-redefs [cf/flags #{:telemetry}
|
||||
telemetry/send-event-batch
|
||||
(fn [_cfg batch]
|
||||
(reset! captured-batch batch)
|
||||
true)]
|
||||
;; Insert with safe context already pre-filtered (as the ingest path does)
|
||||
(th/db-insert! :audit-log
|
||||
{:id (uuid/next)
|
||||
:name "navigate"
|
||||
:type "action"
|
||||
:source "telemetry:frontend"
|
||||
:profile-id uuid/zero
|
||||
:ip-addr (db/inet "0.0.0.0")
|
||||
:props (db/tjson {})
|
||||
:context (db/tjson (dissoc safe-ctx :session :external-session-id))
|
||||
:tracked-at (ct/now)
|
||||
:created-at (ct/now)})
|
||||
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
|
||||
(t/is (= 1 (count @captured-batch)))
|
||||
(let [ev (first @captured-batch)]
|
||||
;; must have the core identity fields including profile-id
|
||||
(t/is (contains? ev :name))
|
||||
(t/is (contains? ev :type))
|
||||
(t/is (contains? ev :source))
|
||||
(t/is (contains? ev :tracked-at))
|
||||
(t/is (contains? ev :profile-id))
|
||||
;; props are present but empty (stripped at ingest time)
|
||||
(t/is (= {} (:props ev)))
|
||||
;; ip-addr is stripped
|
||||
(t/is (not (contains? ev :ip-addr)))
|
||||
;; context may be present and must not contain session-linking keys
|
||||
(when-let [ctx (:context ev)]
|
||||
(t/is (not (contains? ctx :session)))
|
||||
(t/is (not (contains? ctx :external-session-id)))
|
||||
;; safe keys should be present
|
||||
(t/is (contains? ctx :browser))))))))
|
||||
|
||||
(t/deftest test-telemetry-rows-have-day-precision-timestamps
|
||||
;; Telemetry events must be stored with timestamps truncated to day
|
||||
;; precision so that exact event timing cannot be inferred.
|
||||
(with-redefs [cf/flags #{:telemetry}]
|
||||
(th/with-audit-config #{:telemetry}
|
||||
(let [process-event (ns-resolve 'app.loggers.audit 'process-event)
|
||||
profile (th/create-profile* 1 {:is-active true})
|
||||
tnow (ct/now)
|
||||
@ -546,7 +220,7 @@
|
||||
;; Verify the full row shape stored by process-event in telemetry mode:
|
||||
;; source=telemetry:backend, empty props, zeroed ip, context filtered to safe
|
||||
;; backend keys only, profile-id preserved, timestamps truncated.
|
||||
(with-redefs [cf/flags #{:telemetry}]
|
||||
(th/with-audit-config #{:telemetry}
|
||||
(let [process-event (ns-resolve 'app.loggers.audit 'process-event)
|
||||
profile (th/create-profile* 1 {:is-active true})
|
||||
tnow (ct/now)
|
||||
@ -901,61 +575,17 @@
|
||||
(t/is (= (dissoc props :email :unsafe-label)
|
||||
(:props result)))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; SEND-EVENT-BATCH PAYLOAD STRUCTURE
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(t/deftest test-send-event-batch-payload-structure
|
||||
;; Verify the HTTP request sent by send-event-batch carries the
|
||||
;; correct outer wrapper: :type, :version, :instance-id, :events.
|
||||
(let [captured-request (atom nil)]
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}
|
||||
http-mock {:target 'app.http.client/req
|
||||
:return {:status 200}}]
|
||||
(with-redefs [cf/flags #{:telemetry}]
|
||||
(insert-telemetry-row! "navigate")
|
||||
(insert-telemetry-row! "create-file")
|
||||
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
|
||||
;; http/req was called (by both send-legacy-data and send-event-batch)
|
||||
(t/is (:called? @http-mock))
|
||||
;; Find the call whose body contains :telemetry-events
|
||||
(let [calls (filter (fn [args]
|
||||
(let [[_ request] args
|
||||
body (:body request)]
|
||||
(and (string? body)
|
||||
(re-find #"telemetry-events" body))))
|
||||
(:call-args-list @http-mock))]
|
||||
(t/is (= 1 (count calls)))
|
||||
(let [[_ request] (first calls)
|
||||
body (json/decode (:body request))]
|
||||
;; Outer payload fields
|
||||
(t/is (= "telemetry-events" (name (:type body))))
|
||||
(t/is (string? (:version body)))
|
||||
(t/is (some? (:instance-id body)))
|
||||
;; :events is a plain vector of raw event maps
|
||||
(t/is (vector? (:events body)))
|
||||
(t/is (pos? (count (:events body))))
|
||||
(doseq [ev (:events body)]
|
||||
(t/is (string? (:name ev)))
|
||||
(t/is (string? (:source ev)))
|
||||
(t/is (string? (:tracked-at ev))))))))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; TASK BRANCH COVERAGE
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(t/deftest test-enabled-no-subs-no-events-legacy-still-sends
|
||||
;; When telemetry is enabled, there are no newsletter subscriptions
|
||||
;; and no audit_log rows, the legacy report must still be sent.
|
||||
;; and no audit_log rows, the legacy report must still be sent. The
|
||||
;; shadows no longer travel here: the telemetry consumer sends them.
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}
|
||||
batch-mock {:target 'app.tasks.telemetry/send-event-batch
|
||||
:return true}]
|
||||
(with-redefs [cf/flags #{:telemetry}]
|
||||
;; No profiles with newsletter-updates, no telemetry rows
|
||||
:return nil}]
|
||||
(th/with-audit-config #{:telemetry}
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
|
||||
;; Legacy report was sent
|
||||
@ -964,154 +594,4 @@
|
||||
(t/is (= :telemetry-legacy-report (:type data)))
|
||||
(t/is (contains? data :stats))
|
||||
;; No subscriptions in the payload
|
||||
(t/is (not (contains? data :subscriptions))))
|
||||
|
||||
;; No events to batch-send
|
||||
(t/is (not (:called? @batch-mock))))))
|
||||
|
||||
(t/deftest test-legacy-succeeds-batch-fails
|
||||
;; The legacy report and event batch are independent paths.
|
||||
;; When the batch endpoint fails, the legacy report must still
|
||||
;; have been sent successfully.
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}
|
||||
batch-mock {:target 'app.tasks.telemetry/send-event-batch
|
||||
:return false}]
|
||||
(with-redefs [cf/flags #{:telemetry}]
|
||||
(insert-telemetry-row! "navigate")
|
||||
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
|
||||
;; Legacy report was sent
|
||||
(t/is (:called? @legacy-mock))
|
||||
(let [[_ data] (:call-args @legacy-mock)]
|
||||
(t/is (= :telemetry-legacy-report (:type data))))
|
||||
|
||||
;; Batch send was attempted but failed
|
||||
(t/is (:called? @batch-mock))
|
||||
;; Row still present (not deleted on failure)
|
||||
(t/is (= 1 (count-telemetry-rows))))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; GC + BATCH FAILURE INTERACTION
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(t/deftest test-gc-runs-even-when-batch-fails
|
||||
;; GC must purge stale events regardless of whether the subsequent
|
||||
;; batch send succeeds or fails.
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}
|
||||
batch-mock {:target 'app.tasks.telemetry/send-event-batch
|
||||
:return false}]
|
||||
(with-redefs [cf/flags #{:telemetry}]
|
||||
(let [eight-days (ct/minus (ct/now) (ct/duration {:days 8}))
|
||||
one-day (ct/minus (ct/now) (ct/duration {:days 1}))]
|
||||
;; Stale events (should be GC'd)
|
||||
(insert-telemetry-row! "stale-1" {:created-at eight-days :tracked-at eight-days})
|
||||
(insert-telemetry-row! "stale-2" {:created-at eight-days :tracked-at eight-days})
|
||||
;; Fresh event (should survive GC but fail to send)
|
||||
(insert-telemetry-row! "fresh" {:created-at one-day :tracked-at one-day})
|
||||
|
||||
(t/is (= 3 (count-telemetry-rows)))
|
||||
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
|
||||
;; Batch send was attempted (and failed)
|
||||
(t/is (:called? @batch-mock))
|
||||
;; Stale rows were purged by GC, fresh row remains
|
||||
(t/is (= 1 (count-telemetry-rows)))
|
||||
(t/is (= "fresh" (:name (first (th/db-exec! ["SELECT name FROM audit_log WHERE source LIKE 'telemetry:%'"])))))))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; ROW->EVENT CONTEXT GUARANTEE
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(t/deftest test-row->event-always-includes-context
|
||||
;; row->event must always include :context as a map, even when the
|
||||
;; DB column contains an empty transit object.
|
||||
(let [row->event (ns-resolve 'app.tasks.telemetry 'row->event)]
|
||||
;; With non-empty context
|
||||
(let [ev (row->event {:name "test" :type "action" :source "telemetry:backend"
|
||||
:tracked-at (ct/now) :profile-id uuid/zero
|
||||
:context (db/tjson {:browser "Chrome"})})]
|
||||
(t/is (contains? ev :context))
|
||||
(t/is (= {:browser "Chrome"} (:context ev))))
|
||||
|
||||
;; With empty context ({} in transit)
|
||||
(let [ev (row->event {:name "test" :type "action" :source "telemetry:backend"
|
||||
:tracked-at (ct/now) :profile-id uuid/zero
|
||||
:context (db/tjson {})})]
|
||||
(t/is (contains? ev :context))
|
||||
(t/is (= {} (:context ev))))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; NO DUPLICATE EVENTS ON SUCCESS
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(t/deftest test-no-duplicate-events-after-successful-send
|
||||
;; After a successful batch send, the sent rows must be deleted.
|
||||
;; Running the task again must NOT re-send the same events.
|
||||
(let [send-count (atom 0)]
|
||||
(with-mocks [legacy-mock {:target 'app.tasks.telemetry/make-legacy-request
|
||||
:return nil}]
|
||||
(with-redefs [cf/flags #{:telemetry}
|
||||
telemetry/send-event-batch
|
||||
(fn [_cfg _batch]
|
||||
(swap! send-count inc)
|
||||
true)]
|
||||
(insert-telemetry-row! "navigate")
|
||||
(insert-telemetry-row! "create-file")
|
||||
|
||||
(t/is (= 2 (count-telemetry-rows)))
|
||||
|
||||
;; First run: sends and deletes
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
(t/is (= 1 @send-count))
|
||||
(t/is (= 0 (count-telemetry-rows)))
|
||||
|
||||
;; Second run: no events to send
|
||||
(th/run-task! :telemetry {:send? true :enabled? true})
|
||||
(t/is (= 1 @send-count)) ;; still 1, not 2
|
||||
(t/is (= 0 (count-telemetry-rows)))))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; PENDING-QUEUE REGRESSION (telemetry deletes)
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(defn- insert-pending-pair!
|
||||
"Queue the row for both consumers (from the consumer constants, not
|
||||
literals): the delete paths must face pending-ful rows."
|
||||
[row-id]
|
||||
(doseq [consumer [at/consumer-nexus at/consumer-admin-console]]
|
||||
(th/db-exec! ["INSERT INTO audit_log_pending (audit_log_id, consumer)
|
||||
VALUES (?::uuid, ?::smallint)
|
||||
ON CONFLICT DO NOTHING"
|
||||
row-id consumer])))
|
||||
|
||||
(defn- count-pendings []
|
||||
(-> (th/db-exec-one! ["SELECT count(*) AS cnt FROM audit_log_pending"])
|
||||
:cnt
|
||||
long))
|
||||
|
||||
(t/deftest test-delete-sent-events-removes-pendings-first
|
||||
;; Regression: telemetry rows carry pendings for both consumers, so
|
||||
;; deleting the row must delete its pendings first, or the FK
|
||||
;; rejects the delete and the batch is sent again next tick.
|
||||
(let [row (insert-telemetry-row! "navigate")]
|
||||
(insert-pending-pair! (:id row))
|
||||
(t/is (= 2 (count-pendings)))
|
||||
(db/tx-run! th/*pool*
|
||||
(fn [{:keys [::db/conn]}]
|
||||
(@#'telemetry/delete-sent-events conn [(:id row)])))
|
||||
(t/is (= 0 (count-telemetry-rows)))
|
||||
(t/is (= 0 (count-pendings)))))
|
||||
|
||||
(t/deftest test-gc-events-removes-pendings-first
|
||||
;; Same guarantee for the >7-days purge.
|
||||
(let [past (ct/in-past {:days 8})
|
||||
row (insert-telemetry-row! "navigate" {:created-at past :tracked-at past})]
|
||||
(insert-pending-pair! (:id row))
|
||||
(t/is (= 2 (count-pendings)))
|
||||
(@#'telemetry/gc-events {::db/conn th/*pool*})
|
||||
(t/is (= 0 (count-telemetry-rows)))
|
||||
(t/is (= 0 (count-pendings)))))
|
||||
(t/is (not (contains? data :subscriptions)))))))
|
||||
|
||||
@ -47,12 +47,12 @@
|
||||
(defn- make-metrics []
|
||||
(ig/init-key :app.metrics/metrics
|
||||
{:default (select-keys main/default-metrics
|
||||
[:jobs-dispatcher-timing])}))
|
||||
[:jobs-dispatcher-batch-timing])}))
|
||||
|
||||
(defn- make-orphan-metrics []
|
||||
(ig/init-key :app.metrics/metrics
|
||||
{:default (select-keys main/default-metrics
|
||||
[:jobs-orphaned :jobs-total-timing])}))
|
||||
[:jobs-total-timing])}))
|
||||
|
||||
(defn- counter-value [metrics id labels]
|
||||
(let [collector (mtx/get-collector metrics id)
|
||||
@ -74,7 +74,7 @@
|
||||
(reduce + 0)))
|
||||
|
||||
(defn- mk-job
|
||||
[{:keys [name queue status scheduled-at modified-at tenant]
|
||||
[{:keys [name queue status scheduled-at modified-at tenant dedupe label]
|
||||
:or {name "test-job"
|
||||
queue "test"
|
||||
tenant (cf/get :tenant)
|
||||
@ -86,6 +86,8 @@
|
||||
:name name
|
||||
:tenant tenant
|
||||
:queue queue
|
||||
:label label
|
||||
:dedupe (boolean dedupe)
|
||||
:params (db/json {})
|
||||
:priority 100
|
||||
:max-retries 3
|
||||
@ -123,6 +125,16 @@
|
||||
|
||||
(t/use-fixtures :each test-fixture)
|
||||
|
||||
(defn- fill-queue
|
||||
"Put one dummy payload in a queue, as a worker that has not drained it."
|
||||
[queue-name]
|
||||
(let [conn (rds/connect (mk-cfg))]
|
||||
(try
|
||||
(rds/rpush conn (queue-key queue-name)
|
||||
[(json/encode [(str (uuid/next)) (ct/format-inst (ct/now))])])
|
||||
(finally
|
||||
(rds/close conn)))))
|
||||
|
||||
(defn- drain-queue
|
||||
[queue-name]
|
||||
(let [conn (rds/connect (mk-cfg))]
|
||||
@ -176,24 +188,200 @@
|
||||
(t/is (= "new" (:status (get-row future))))
|
||||
(t/is (= "running" (:status (get-row running))))))
|
||||
|
||||
(t/deftest dispatcher-reschedules-lost-scheduled-jobs
|
||||
(let [cfg (mk-cfg)
|
||||
lost-id (mk-job {:status "scheduled"
|
||||
:scheduled-at (ct/minus (ct/now)
|
||||
(ct/duration {:minutes 6}))})
|
||||
fresh-id (mk-job {:status "scheduled"})]
|
||||
(t/deftest dispatcher-aborts-a-lost-job-whose-queue-is-empty
|
||||
;; `scheduled` means the job was handed to a queue. When the queue has
|
||||
;; nothing waiting and the row has been there longer than the grace, the
|
||||
;; message is gone and the job is terminal. Known limit: the grace is
|
||||
;; measured from the push, so a message that waited longer in a backlog
|
||||
;; and is popped while the sweep runs (the queue empty at that instant)
|
||||
;; can lose the race with its claim; the window is the worker's
|
||||
;; pop-to-claim round trip.
|
||||
(let [cfg (mk-cfg)
|
||||
lost (mk-job {:status "scheduled"
|
||||
:modified-at (ct/minus (ct/now)
|
||||
(ct/duration {:minutes 6}))})]
|
||||
(wdisp/run-batch cfg)
|
||||
(let [lost (get-row lost-id)
|
||||
fresh (get-row fresh-id)]
|
||||
;; the lost job is rescheduled to 'new' and claimed again in the
|
||||
;; same batch, so it ends as 'scheduled' with a recent scheduled_at
|
||||
(t/is (= "scheduled" (:status lost)))
|
||||
(t/is (> (inst-ms (:scheduled-at lost))
|
||||
(inst-ms (ct/minus (ct/now)
|
||||
(ct/duration {:minutes 5})))))
|
||||
(t/testing "the rescheduled job was pushed again to the queue"
|
||||
(t/is (= 1 (count (drain-queue "test"))))
|
||||
(t/is (= "scheduled" (:status fresh)))))))
|
||||
(let [row (get-row lost)]
|
||||
(t/is (= "aborted" (:status row)))
|
||||
(t/is (= jobs/lost-error (:error row)))
|
||||
(t/is (= [{:kind "end" :payload {:outcome "aborted"}}] (:events row)))
|
||||
(t/testing "an aborted job is terminal: it is never retried"
|
||||
(t/is (zero? (jobs/retry-job cfg (:id row) 1 (ct/now) jobs/lost-error :backoff)))))))
|
||||
|
||||
(t/deftest dispatcher-leaves-a-fresh-scheduled-job-alone
|
||||
(let [cfg (mk-cfg)
|
||||
fresh (mk-job {:status "scheduled"
|
||||
:modified-at (ct/minus (ct/now)
|
||||
(ct/duration {:minutes 4}))})]
|
||||
(wdisp/run-batch cfg)
|
||||
(let [row (get-row fresh)]
|
||||
(t/is (= "scheduled" (:status row)))
|
||||
(t/is (nil? (:error row)))
|
||||
(t/is (= [] (:events row))))))
|
||||
|
||||
(t/deftest dispatcher-leaves-a-scheduled-job-that-is-still-in-its-queue
|
||||
;; The queue has a message waiting: the job may be behind a backlog, so
|
||||
;; it is not lost. Aborting on time alone would kill legitimate work.
|
||||
(let [cfg (mk-cfg)
|
||||
lost (mk-job {:status "scheduled"
|
||||
:modified-at (ct/minus (ct/now)
|
||||
(ct/duration {:minutes 6}))})]
|
||||
(fill-queue "test")
|
||||
(wdisp/run-batch cfg)
|
||||
(let [row (get-row lost)]
|
||||
(t/is (= "scheduled" (:status row)))
|
||||
(t/is (nil? (:error row)))
|
||||
(t/is (= [] (:events row))))))
|
||||
|
||||
(t/deftest dispatcher-aborts-a-scheduled-job-past-the-cap
|
||||
;; A message lost behind a backlog (or a queue with no consumer) never
|
||||
;; drains: past the cap the job is aborted anyway.
|
||||
(let [cfg (mk-cfg)
|
||||
lost (mk-job {:status "scheduled"
|
||||
:modified-at (ct/minus (ct/now)
|
||||
(ct/duration {:hours 25}))})]
|
||||
(fill-queue "test")
|
||||
(wdisp/run-batch cfg)
|
||||
(t/is (= "aborted" (:status (get-row lost))))))
|
||||
|
||||
(t/deftest lost-sweep-pages-the-cap-branch
|
||||
;; A row past the cap is lost whatever its queue holds, and the read is
|
||||
;; bounded: with a batch of one, each pass aborts the oldest candidate
|
||||
;; and the next pass continues. Only the grace rows pay the LLEN.
|
||||
(let [cfg (mk-cfg)
|
||||
q "test-lost-cap"
|
||||
a (mk-job {:status "scheduled"
|
||||
:queue q
|
||||
:modified-at (ct/minus (ct/now) (ct/duration {:hours 30}))})
|
||||
b (mk-job {:status "scheduled"
|
||||
:queue q
|
||||
:modified-at (ct/minus (ct/now) (ct/duration {:hours 29}))})]
|
||||
(fill-queue q)
|
||||
(with-redefs-fn {#'wdisp/lost-batch-size 1}
|
||||
#(wdisp/run-batch cfg))
|
||||
(t/is (= 1 (count (filter #(= "aborted" (:status (get-row %))) [a b]))))
|
||||
(with-redefs-fn {#'wdisp/lost-batch-size 1}
|
||||
#(wdisp/run-batch cfg))
|
||||
(t/is (every? #(= "aborted" (:status (get-row %))) [a b]))))
|
||||
|
||||
(t/deftest lost-sweep-pages-the-grace-window
|
||||
;; Inside the grace window the read is bounded and the queue decides: an
|
||||
;; empty queue makes the row lost, and with a batch of one the next pass
|
||||
;; continues where the previous one stopped.
|
||||
(let [cfg (mk-cfg)
|
||||
q "test-lost-grace"
|
||||
a (mk-job {:status "scheduled"
|
||||
:queue q
|
||||
:modified-at (ct/minus (ct/now) (ct/duration {:minutes 7}))})
|
||||
b (mk-job {:status "scheduled"
|
||||
:queue q
|
||||
:modified-at (ct/minus (ct/now) (ct/duration {:minutes 6}))})]
|
||||
(with-redefs-fn {#'wdisp/lost-batch-size 1}
|
||||
#(wdisp/run-batch cfg))
|
||||
(t/is (= 1 (count (filter #(= "aborted" (:status (get-row %))) [a b]))))
|
||||
(with-redefs-fn {#'wdisp/lost-batch-size 1}
|
||||
#(wdisp/run-batch cfg))
|
||||
(t/is (every? #(= "aborted" (:status (get-row %))) [a b]))))
|
||||
|
||||
(t/deftest dispatcher-does-not-abort-a-job-whose-heartbeat-renews-it
|
||||
;; The heartbeat accepts a `scheduled` row (a previous attempt may still
|
||||
;; beat): it refreshes `modified_at`, so the outer WHERE must re-check it.
|
||||
(let [cfg (mk-cfg)
|
||||
lost (mk-job {:status "scheduled"
|
||||
:modified-at (ct/minus (ct/now)
|
||||
(ct/duration {:minutes 6}))})]
|
||||
(with-redefs [rds/llen (fn [_ _]
|
||||
(th/db-update! :job {:modified-at (ct/now)} {:id lost})
|
||||
0)]
|
||||
(wdisp/run-batch cfg))
|
||||
(let [row (get-row lost)]
|
||||
(t/is (= "scheduled" (:status row)))
|
||||
(t/is (nil? (:error row)))
|
||||
(t/is (= [] (:events row))))))
|
||||
|
||||
(t/deftest dispatcher-llens-only-its-tenant-queue
|
||||
;; The queue key carries the tenant: a message waiting for another
|
||||
;; environment cannot protect a lost row of this one.
|
||||
(let [cfg (mk-cfg)
|
||||
foreign (wrk/queue-key "other-tenant" "test")
|
||||
conn (rds/connect (mk-cfg))
|
||||
lost (mk-job {:status "scheduled"
|
||||
:modified-at (ct/minus (ct/now)
|
||||
(ct/duration {:minutes 6}))})]
|
||||
(try
|
||||
(rds/rpush conn foreign
|
||||
[(json/encode [(str (uuid/next))
|
||||
(ct/format-inst (ct/now))])])
|
||||
(wdisp/run-batch cfg)
|
||||
(t/is (= "aborted" (:status (get-row lost))))
|
||||
(finally
|
||||
(rds/del conn foreign)
|
||||
(rds/close conn)))))
|
||||
|
||||
(t/deftest dispatcher-aborts-two-lost-dedupe-jobs-with-the-same-key
|
||||
;; A Redis restart loses every message at once: two deduped jobs with the
|
||||
;; same key can be `scheduled` together. Neither returns to `new` (that
|
||||
;; would violate `job__new_dedupe__uniq`); both are aborted and the batch
|
||||
;; keeps going.
|
||||
(let [cfg (mk-cfg)
|
||||
a (mk-job {:status "scheduled"
|
||||
:dedupe true
|
||||
:label "k"
|
||||
:modified-at (ct/minus (ct/now)
|
||||
(ct/duration {:minutes 6}))})
|
||||
b (mk-job {:status "scheduled"
|
||||
:dedupe true
|
||||
:label "k"
|
||||
:modified-at (ct/minus (ct/now)
|
||||
(ct/duration {:minutes 7}))})]
|
||||
(wdisp/run-batch cfg)
|
||||
(t/is (= "aborted" (:status (get-row a))))
|
||||
(t/is (= "aborted" (:status (get-row b))))
|
||||
(t/is (empty? (th/db-query :job {:status "new"})))
|
||||
(t/is (empty? (drain-queue "test")))))
|
||||
|
||||
(t/deftest dispatcher-abort-lost-records-the-terminal-counter
|
||||
(let [metrics (make-orphan-metrics)
|
||||
cfg (assoc (mk-cfg) ::mtx/metrics metrics)]
|
||||
(mk-job {:status "scheduled"
|
||||
:modified-at (ct/minus (ct/now)
|
||||
(ct/duration {:minutes 6}))})
|
||||
(wdisp/run-batch cfg)
|
||||
(t/testing "a job lost in transit counts as an aborted terminal outcome"
|
||||
(t/is (= 1 (histogram-sample-count metrics ["test-job" "aborted"]))))))
|
||||
|
||||
(t/deftest dispatcher-does-not-abort-a-job-claimed-before-the-sweep
|
||||
;; The candidate is read first and the UPDATE runs later: a runner can
|
||||
;; claim the row in between. The outer WHERE must see the new status and
|
||||
;; affect no row, or the job would be aborted while running.
|
||||
(let [cfg (mk-cfg)
|
||||
lost (mk-job {:status "scheduled"
|
||||
:modified-at (ct/minus (ct/now)
|
||||
(ct/duration {:minutes 6}))})]
|
||||
(with-redefs [rds/llen (fn [_ _]
|
||||
(th/db-update! :job {:status "running"
|
||||
:modified-at (ct/now)}
|
||||
{:id lost})
|
||||
0)]
|
||||
(wdisp/run-batch cfg))
|
||||
(let [row (get-row lost)]
|
||||
(t/is (= "running" (:status row)))
|
||||
(t/is (nil? (:error row)))
|
||||
(t/is (= [] (:events row))))))
|
||||
|
||||
(t/deftest dispatcher-aborts-a-lost-job-once
|
||||
;; Two dispatchers of the same tenant read the same candidate: only the
|
||||
;; first transition wins, so there is one end event and one sample.
|
||||
(let [metrics (make-orphan-metrics)
|
||||
cfg (assoc (mk-cfg) ::mtx/metrics metrics)
|
||||
lost (mk-job {:status "scheduled"
|
||||
:modified-at (ct/minus (ct/now)
|
||||
(ct/duration {:minutes 6}))})]
|
||||
(run! deref (doall (repeatedly 2 #(future (wdisp/run-batch cfg)))))
|
||||
(t/is (= "aborted" (:status (get-row lost))))
|
||||
(t/is (= 1 (count (filter #(= "end" (:kind %))
|
||||
(:events (get-row lost))))))
|
||||
(t/is (= 1 (histogram-sample-count metrics ["test-job" "aborted"])))))
|
||||
|
||||
(t/deftest dispatcher-marks-stale-running-jobs-as-aborted-by-lease
|
||||
(let [cfg (mk-cfg)
|
||||
@ -218,7 +406,7 @@
|
||||
(t/testing "recent running job is not touched"
|
||||
(t/is (= "running" (:status (get-row fresh)))))))
|
||||
|
||||
(t/deftest dispatcher-orphan-sweep-records-orphan-and-terminal-metrics
|
||||
(t/deftest dispatcher-orphan-sweep-records-the-terminal-counter
|
||||
(let [metrics (make-orphan-metrics)
|
||||
cfg (assoc (mk-cfg) ::mtx/metrics metrics)
|
||||
_ (mk-job {:status "running"
|
||||
@ -226,8 +414,8 @@
|
||||
(ct/plus (cf/get-jobs-lease)
|
||||
(ct/duration {:minutes 1})))})]
|
||||
(wdisp/run-batch cfg)
|
||||
(t/is (= 1.0 (counter-value metrics :jobs-orphaned ["test"])))
|
||||
(t/is (= 1 (histogram-sample-count metrics ["test-job" "test" "aborted"])))))
|
||||
(t/testing "the _count of the aborted outcome is the orphan count"
|
||||
(t/is (= 1 (histogram-sample-count metrics ["test-job" "aborted"]))))))
|
||||
|
||||
(t/deftest dispatcher-batch-without-pending-jobs-signals-wait
|
||||
(let [cfg (mk-cfg)]
|
||||
@ -297,19 +485,18 @@
|
||||
(t/testing "and nothing was pushed to this tenant's Redis list"
|
||||
(t/is (empty? (drain-queue "test"))))))
|
||||
|
||||
(t/deftest dispatcher-does-not-reschedule-another-tenants-lost-jobs
|
||||
(t/deftest dispatcher-does-not-abort-another-tenants-lost-jobs
|
||||
(let [cfg (mk-cfg)
|
||||
foreign (mk-job {:status "scheduled"
|
||||
:tenant "other-tenant"
|
||||
:scheduled-at (ct/minus (ct/now)
|
||||
(ct/duration {:minutes 6}))})]
|
||||
:modified-at (ct/minus (ct/now)
|
||||
(ct/duration {:minutes 61}))})]
|
||||
(wdisp/run-batch cfg)
|
||||
(t/testing "the other tenant's scheduled job keeps its status and its date"
|
||||
(let [row (get-row foreign)]
|
||||
(t/is (= "scheduled" (:status row)))
|
||||
(t/is (< (inst-ms (:scheduled-at row))
|
||||
(inst-ms (ct/minus (ct/now)
|
||||
(ct/duration {:minutes 5})))))))))
|
||||
(t/is (nil? (:error row)))
|
||||
(t/is (= [] (:events row)))))))
|
||||
|
||||
(t/deftest dispatcher-does-not-orphan-another-tenants-running-jobs
|
||||
(let [cfg (mk-cfg)
|
||||
|
||||
@ -82,7 +82,7 @@
|
||||
(defn- make-metrics []
|
||||
(ig/init-key :app.metrics/metrics
|
||||
{:default (select-keys main/default-metrics
|
||||
[:jobs-queue-wait-timing
|
||||
[:jobs-claim-wait-timing
|
||||
:jobs-execution-timing
|
||||
:jobs-total-timing
|
||||
:tasks-timing])}))
|
||||
@ -241,7 +241,7 @@
|
||||
(t/is (not (contains? context :queue))
|
||||
"the tenant-prefixed queue is a routing detail, not context"))))))
|
||||
|
||||
(t/deftest runner-records-queue-wait-execution-and-total
|
||||
(t/deftest runner-records-claim-wait-execution-and-total
|
||||
(let [metrics (make-metrics)
|
||||
cfg (assoc (mk-cfg {}) ::mtx/metrics metrics)
|
||||
at (ct/truncate (ct/now) :millisecond)
|
||||
@ -249,14 +249,14 @@
|
||||
(push-payload job-id at)
|
||||
(run-one cfg)
|
||||
(t/is (= 1 (histogram-sample-count metrics
|
||||
:jobs-queue-wait-timing
|
||||
["echo-runner" "test"])))
|
||||
:jobs-claim-wait-timing
|
||||
["echo-runner"])))
|
||||
(t/is (= 1 (histogram-sample-count metrics
|
||||
:jobs-execution-timing
|
||||
["echo-runner" "test"])))
|
||||
["echo-runner"])))
|
||||
(t/is (= 1 (histogram-sample-count metrics
|
||||
:jobs-total-timing
|
||||
["echo-runner" "test" "completed"])))
|
||||
["echo-runner" "completed"])))
|
||||
(t/testing "the legacy histogram is still emitted, with its raw name label"
|
||||
(t/is (= 1 (histogram-sample-count metrics
|
||||
:tasks-timing
|
||||
@ -408,11 +408,12 @@
|
||||
(t/is (= 1 (#'wrkr/claim-job (mk-cfg {}) job-id at)))
|
||||
(t/is (= "running" (:status (get-row job-id)))))))
|
||||
|
||||
(t/deftest runner-skips-stale-payload-after-reschedule
|
||||
(t/deftest runner-skips-a-payload-with-a-stale-timestamp
|
||||
(let [stale-at (ct/truncate (ct/now) :millisecond)
|
||||
job-id (mk-job {:scheduled-at stale-at})
|
||||
fresh-at (ct/plus stale-at (ct/duration {:minutes 6}))]
|
||||
;; dispatcher reschedules: new scheduled_at + fresh payload
|
||||
;; A duplicate payload (or one whose row was edited by hand) carries a
|
||||
;; timestamp that no longer matches the row.
|
||||
(th/db-update! :job {:status "scheduled" :scheduled-at fresh-at} {:id job-id})
|
||||
|
||||
(t/testing "stale payload is skipped, handler never invoked"
|
||||
|
||||
@ -17,7 +17,8 @@
|
||||
[app.worker.cron :as wcron]
|
||||
[app.worker.dispatcher :as wdisp]
|
||||
[app.worker.runner :as wrkr]
|
||||
[clojure.test :as t]))
|
||||
[clojure.test :as t]
|
||||
[integrant.core :as ig]))
|
||||
|
||||
;; The dispatcher's push and the runner's pop agree on the Redis list
|
||||
;; only through this helper, so the test helpers use it too. That
|
||||
@ -50,6 +51,29 @@
|
||||
@#'wrkr/schema:params]]
|
||||
(t/is (sm/schema? (sm/schema s))))))
|
||||
|
||||
(t/deftest every-worker-consumer-component-is-known-to-the-test-system
|
||||
;; The test system drops the consumer loops by key, so a new one must be
|
||||
;; added there too or it drains audit_log in the background. Deriving the
|
||||
;; list from the config (every worker key named `/consumer`) makes a new
|
||||
;; consumer fail here, not as a flaky test elsewhere.
|
||||
(let [keys (->> (keys main/worker-config)
|
||||
(filter #(and (keyword? %) (= "consumer" (name %))))
|
||||
(into #{}))]
|
||||
(t/is (= #{:app.loggers.audit.consumers.nexus/consumer
|
||||
:app.loggers.audit.consumers.admin-console/consumer
|
||||
:app.loggers.audit.consumers.telemetry/consumer}
|
||||
keys)
|
||||
(str "update this set and helpers/init-system's dissoc: " keys))))
|
||||
|
||||
(t/deftest audit-log-gc-cron-is-gated-by-its-own-flag
|
||||
(t/testing "default configuration schedules the GC"
|
||||
(binding [cf/flags (#'cf/parse-flags {})]
|
||||
(t/is (some #(= :audit-log-gc (:job %)) (main/worker-cron-entries)))))
|
||||
(t/testing "an explicit disable suppresses the GC even with telemetry enabled"
|
||||
(binding [cf/flags (#'cf/parse-flags {:flags [:enable-telemetry :disable-audit-log-gc]})]
|
||||
(t/is (not (some #(= :audit-log-gc (:job %))
|
||||
(main/worker-cron-entries)))))))
|
||||
|
||||
(t/deftest the-binfile-runner-is-wired
|
||||
(let [cfg (get main/worker-config [:app.main/binfile ::wrk/runner])]
|
||||
|
||||
@ -66,3 +90,27 @@
|
||||
(t/is (contains? cfg ::jobs/defs))
|
||||
(t/is (contains? cfg ::mtx/metrics))
|
||||
(t/is (contains? cfg ::db/pool)))))
|
||||
|
||||
(t/deftest the-audit-consumers-are-wired
|
||||
;; Each consumer is a loop in a thread: the test system drops the three
|
||||
;; components by name, so a rename that leaves `helpers.clj` behind would
|
||||
;; start real loops in the background. The components live in the worker
|
||||
;; config only (a web instance does not drain the audit log) and their
|
||||
;; `ig/init-key`/`ig/halt-key!` methods must be registered.
|
||||
(doseq [[k deps] [[:app.loggers.audit.consumers.nexus/consumer
|
||||
[::db/pool :app.http.client/client
|
||||
:app.setup/shared-keys ::mtx/metrics]]
|
||||
[:app.loggers.audit.consumers.admin-console/consumer
|
||||
[::db/pool :app.nitrate/client ::mtx/metrics]]
|
||||
[:app.loggers.audit.consumers.telemetry/consumer
|
||||
[::db/pool :app.http.client/client
|
||||
:app.setup/props ::mtx/metrics]]]]
|
||||
(t/testing (str k)
|
||||
(let [cfg (get main/worker-config k)]
|
||||
(t/is (some? cfg))
|
||||
(t/is (nil? (get main/system-config k)))
|
||||
(t/is (some? (get-method ig/init-key k)))
|
||||
(t/is (some? (get-method ig/halt-key! k)))
|
||||
(t/testing "the loop resolves its dependencies"
|
||||
(doseq [dep deps]
|
||||
(t/is (contains? cfg dep) (str k " misses " dep))))))))
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
export PENPOT_FLAGS="enable-asserts enable-audit-log $PENPOT_FLAGS"
|
||||
export PENPOT_FLAGS="enable-asserts enable-nexus $PENPOT_FLAGS"
|
||||
|
||||
export JAVA_OPTS="\
|
||||
-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager \
|
||||
|
||||
@ -160,7 +160,7 @@
|
||||
([objects id]
|
||||
(get-children-ids objects id {}))
|
||||
([objects id {:keys [ignore-children-fn]
|
||||
;;ignore-children-fn should receive a shape and return a boolean
|
||||
;;ignore-children-fn should receive a shape id and return a boolean
|
||||
:or {ignore-children-fn (constantly false)}}]
|
||||
(letfn [(collect [result id processed]
|
||||
(if (contains? processed id)
|
||||
|
||||
@ -2450,7 +2450,8 @@
|
||||
;; editor treats those as separators, and a later edit strips them.
|
||||
;; Files saved before that strip can also hold two identical names, two
|
||||
;; identical value combinations, or a label that was not saved with the
|
||||
;; values. Rewrite the text in place and refresh the label.
|
||||
;; values. Rewrite the text in place and refresh the label. On a clash,
|
||||
;; the variant at the top of the layers panel keeps its values.
|
||||
(defn- variant-container-entries
|
||||
[data]
|
||||
(concat
|
||||
|
||||
@ -170,10 +170,14 @@
|
||||
|
||||
(defn update-variant-container-properties
|
||||
"Apply `f` to the property vectors of the members of `container`, in
|
||||
`:shapes` order, and write each result with `update-variant-properties`.
|
||||
`f` takes and returns a vector of property vectors."
|
||||
layers-panel order (top first, the reverse of `:shapes`), and write each
|
||||
result with `update-variant-properties`. `f` takes and returns a vector
|
||||
of property vectors."
|
||||
[data objects container f]
|
||||
(let [components (mapv :component (qualifying-variant-members data objects container))
|
||||
(let [components (->> (qualifying-variant-members data objects container)
|
||||
(map :component)
|
||||
(reverse)
|
||||
(vec))
|
||||
props-seq (f (mapv #(vec (or (:variant-properties %) [])) components))]
|
||||
(reduce (fn [data [component props]]
|
||||
(update-variant-properties data component (constantly props)))
|
||||
|
||||
@ -59,7 +59,9 @@
|
||||
|
||||
(def varia
|
||||
"Rest of the flags"
|
||||
#{:audit-log
|
||||
#{;; Runs the audit-log buffer GC by default. Disable it explicitly while
|
||||
;; the old and new versions coexist.
|
||||
:audit-log-gc
|
||||
;; Enables shipping audit_log chunks to Nexus.
|
||||
:nexus
|
||||
:audit-log-logger
|
||||
@ -88,6 +90,11 @@
|
||||
;; on the database (server-error-report) and notified to mattermost
|
||||
;; (only when the webhook url is also configured).
|
||||
:error-reporting
|
||||
;; Disabled by default. When enabled, file-gc skips its media clean-up,
|
||||
;; which does not read revision rows, and the revision writer keeps
|
||||
;; revision rows with no expiry (penpot#13187). Enable it only where
|
||||
;; both are deployed.
|
||||
:file-change-retention
|
||||
;; Activates the schema validation during update file.
|
||||
:file-schema-validation
|
||||
;; Reports the schema validation errors internally.
|
||||
@ -219,6 +226,7 @@
|
||||
:enable-backend-api-doc
|
||||
:enable-backend-openapi-doc
|
||||
:enable-backend-worker
|
||||
:enable-audit-log-gc
|
||||
:enable-secure-session-cookies
|
||||
:enable-email-verification
|
||||
:enable-onboarding
|
||||
|
||||
@ -408,10 +408,18 @@
|
||||
[changes file container libraries shape-id]
|
||||
(log/trace :msg " -> advance-nesting-level")
|
||||
(let [detached-ids (atom #{})
|
||||
children (cfh/get-children-with-self (:objects container) shape-id) ;; TODO: this function should be refactored to be a recursive tree traversal.
|
||||
skip-near (fn [changes shape] ;; this way we could shake the tree more easily when detaching shapes
|
||||
(shape-log :trace (:id shape) container ;; and perhaps even allow to recover nested instances that have been
|
||||
:msg " * advancing" :shape-id (:id shape)) ;; swapped and so we can access the main instance again.
|
||||
;; Nested heads swapped in this copy, and their descendants, already
|
||||
;; point to the swapped-in main, so they must not be advanced
|
||||
children (let [objects (:objects container)
|
||||
swapped-head? #(let [shape (get objects %)]
|
||||
(and (ctk/instance-head? shape)
|
||||
(some? (ctk/get-swap-slot shape))))]
|
||||
(->> (cfh/get-children-ids objects shape-id {:ignore-children-fn swapped-head?})
|
||||
(cons shape-id)
|
||||
(map (d/getf objects))))
|
||||
skip-near (fn [changes shape]
|
||||
(shape-log :trace (:id shape) container
|
||||
:msg " * advancing" :shape-id (:id shape))
|
||||
(if (contains? @detached-ids (:id shape))
|
||||
(do (log/trace :msg " (detached)")
|
||||
changes)
|
||||
|
||||
@ -245,8 +245,8 @@
|
||||
allow-altering-copies
|
||||
;; We will delete the shapes and its descendants.
|
||||
;; ignore-children-fn is used to ignore some descendants
|
||||
;; on the deletion process. It should receive a shape and
|
||||
;; return a boolean
|
||||
;; on the deletion process. It should receive a shape id
|
||||
;; and return a boolean
|
||||
ignore-children-fn
|
||||
ignore-mask
|
||||
ignore-flows-for]
|
||||
|
||||
@ -37,23 +37,32 @@
|
||||
|
||||
|
||||
(defn generate-remove-property
|
||||
[changes variant-id pos]
|
||||
(let [data (pcb/get-library-data changes)
|
||||
objects (pcb/get-objects changes)
|
||||
related-components (cfv/find-variant-components data objects variant-id)
|
||||
props (-> related-components first :variant-properties)]
|
||||
"Remove the property at `pos` from every variant of the set. Then, unless
|
||||
`sanitize-set?` is false, sanitize the remaining set as a whole, so two
|
||||
variants that differed only in the removed property do not end up with
|
||||
the same values."
|
||||
[changes variant-id pos & {:keys [sanitize-set?] :or {sanitize-set? true}}]
|
||||
(let [data (pcb/get-library-data changes)
|
||||
objects (pcb/get-objects changes)
|
||||
;; Layers-panel order, the reverse of :shapes: on a clash the variant
|
||||
;; higher in the panel keeps its values and the lower one gets " (n)".
|
||||
components (->> (cfv/qualifying-variant-members data objects (get objects variant-id))
|
||||
(map :component)
|
||||
(reverse))
|
||||
props (-> components first :variant-properties)]
|
||||
(if (and (seq props) (<= 0 pos) (< pos (count props)))
|
||||
(reduce (fn [changes component]
|
||||
(let [props (:variant-properties component)
|
||||
props (d/remove-at-index props pos)
|
||||
main-id (:main-instance-id component)
|
||||
name (ctv/properties-to-name props)]
|
||||
(let [stripped (mapv #(d/remove-at-index (:variant-properties %) pos) components)
|
||||
props-seq (cond-> stripped
|
||||
sanitize-set? ctv/sanitize-properties-set)]
|
||||
(reduce (fn [changes [component props]]
|
||||
(-> changes
|
||||
(pcb/update-component (:id component) #(assoc % :variant-properties props)
|
||||
(pcb/update-component (:id component)
|
||||
#(assoc % :variant-properties props)
|
||||
{:apply-changes-local-library? true})
|
||||
(pcb/update-shapes [main-id] #(assoc % :variant-name name)))))
|
||||
changes
|
||||
related-components)
|
||||
(pcb/update-shapes [(:main-instance-id component)]
|
||||
#(assoc % :variant-name (ctv/properties-to-name props)))))
|
||||
changes
|
||||
(map vector components props-seq)))
|
||||
changes)))
|
||||
|
||||
|
||||
|
||||
@ -185,6 +185,7 @@
|
||||
It will return the shape or nil if it doesn't found any"
|
||||
[container libraries shape parent-id]
|
||||
(let [ref-shape (ctf/find-ref-shape nil container libraries shape
|
||||
:include-deleted? true
|
||||
:with-context? true)
|
||||
|
||||
ref-shape-container (when ref-shape (:container (meta ref-shape)))
|
||||
|
||||
@ -1283,10 +1283,34 @@
|
||||
container container]
|
||||
(if (= (:id current) target-ref)
|
||||
chain
|
||||
(if-let [ref (find-ref-shape nil container libraries current :with-context? true)]
|
||||
(if-let [ref (find-ref-shape nil container libraries current
|
||||
:include-deleted? true
|
||||
:with-context? true)]
|
||||
(recur (conj chain ref) ref (:container (meta ref)))
|
||||
chain))))
|
||||
|
||||
(defn get-swap-level
|
||||
"Returns the level (0 is the shape itself) of the first shape in the ref
|
||||
chain of `shape` that has been swapped there, looking at most
|
||||
`max-levels` levels up. Returns `max-levels` if none is found."
|
||||
[container libraries shape max-levels]
|
||||
(loop [level 0
|
||||
current shape
|
||||
container container]
|
||||
(cond
|
||||
(>= level max-levels)
|
||||
max-levels
|
||||
|
||||
(some? (ctk/get-swap-slot current))
|
||||
level
|
||||
|
||||
:else
|
||||
(if-let [ref (find-ref-shape nil container libraries current
|
||||
:include-deleted? true
|
||||
:with-context? true)]
|
||||
(recur (inc level) ref (:container (meta ref)))
|
||||
max-levels))))
|
||||
|
||||
(defn get-touched-from-ref-chain-until-target-ref
|
||||
"Returns a set with the :touched of all the items on the shape
|
||||
ref chain until target-ref, including itself"
|
||||
|
||||
@ -343,8 +343,9 @@
|
||||
|
||||
(defn sanitize-properties-set
|
||||
"Sanitize the property vectors of all the variants of one set, given in
|
||||
board order. Every vector gets the same names in the same order, and no
|
||||
two vectors share a value combination. Earlier vectors keep their values."
|
||||
layers-panel order (top first). Every vector gets the same names in the
|
||||
same order, and no two vectors share a value combination. Earlier
|
||||
vectors keep their values."
|
||||
[props-seq]
|
||||
(let [cleaned (mapv sanitize-properties props-seq)
|
||||
names (properties-names cleaned)]
|
||||
|
||||
@ -91,8 +91,9 @@
|
||||
|
||||
(t/is (= "big" (get-in data' [:components deleted-id :objects main-id :variant-name])))))
|
||||
|
||||
(t/deftest test-sanitize-variant-container-cleans-members-in-board-order
|
||||
;; :shapes of the board is [m02 m01], so c02 keeps its value.
|
||||
(t/deftest test-sanitize-variant-container-cleans-members-in-layers-panel-order
|
||||
;; :shapes of the board is [m02 m01], so m01 is the top variant in the
|
||||
;; layers panel and c01 keeps its value.
|
||||
(let [props [{:name "Property 1" :value "a=b"}]
|
||||
file (-> (variant-file)
|
||||
(thc/update-component :c01 {:variant-properties props})
|
||||
@ -103,10 +104,10 @@
|
||||
file' (assoc file :data data')]
|
||||
|
||||
(t/is (= [{:name "Property 1" :value "ab"}]
|
||||
(:variant-properties (thc/get-component file' :c02))))
|
||||
(t/is (= [{:name "Property 1" :value "ab (1)"}]
|
||||
(:variant-properties (thc/get-component file' :c01))))
|
||||
(t/is (= "ab (1)" (:variant-name (ths/get-shape file' :m01))))))
|
||||
(t/is (= [{:name "Property 1" :value "ab (1)"}]
|
||||
(:variant-properties (thc/get-component file' :c02))))
|
||||
(t/is (= "ab (1)" (:variant-name (ths/get-shape file' :m02))))))
|
||||
|
||||
(t/deftest test-sanitize-variant-component
|
||||
(let [file (-> (variant-file)
|
||||
|
||||
@ -42,8 +42,13 @@
|
||||
(-> (thf/sample-file :file1)
|
||||
(thv/add-variant :variant1 :component1 :root1 :component2 :root2)))
|
||||
|
||||
(defn- prop-named
|
||||
[props name]
|
||||
(some #(when (= name (:name %)) %) props))
|
||||
|
||||
(t/deftest migration-0038-strips-forbidden-characters
|
||||
;; root2 is the first child of :shapes, so its names lead the set.
|
||||
;; root1 is the last child of :shapes (top of the layers panel), so its
|
||||
;; names lead the set.
|
||||
(let [file (-> (variant-file)
|
||||
(thk/update-component :component2 {:variant-properties [{:name "Size=Big"
|
||||
:value "a=b,"
|
||||
@ -58,13 +63,11 @@
|
||||
copy' (ths/get-shape file' :copy2)
|
||||
data' (ctf/file-data file')]
|
||||
|
||||
(t/is (= "SizeBig" (-> comp2' :variant-properties first :name)))
|
||||
(t/is (= "ab" (-> comp2' :variant-properties first :value)))
|
||||
(t/is (true? (-> comp2' :variant-properties first :hidden)))
|
||||
(t/is (= ["Property 1" "SizeBig"] (mapv :name (:variant-properties comp2'))))
|
||||
(t/is (= "ab" (:value (prop-named (:variant-properties comp2') "SizeBig"))))
|
||||
(t/is (true? (:hidden (prop-named (:variant-properties comp2') "SizeBig"))))
|
||||
(t/is (= "ab" (:variant-name root2')))
|
||||
(t/is (= "Value1" (->> (:variant-properties comp1')
|
||||
(some #(when (= "Property 1" (:name %)) %))
|
||||
:value)))
|
||||
(t/is (= "Value1" (:value (prop-named (:variant-properties comp1') "Property 1"))))
|
||||
(t/is (= copy copy') "copies are left untouched")
|
||||
(t/is (= data' (cfm/migrate-data data' migration-id)))))
|
||||
|
||||
@ -86,6 +89,25 @@
|
||||
(t/is (= (ctv/properties-to-name props) (:variant-name root')))
|
||||
(t/is (= data' (cfm/migrate-data data' migration-id)))))
|
||||
|
||||
(t/deftest migration-0038-keeps-the-top-variant-on-a-clash
|
||||
;; root1 is the last child of :shapes, so it is the top variant in the
|
||||
;; layers panel and the primary one: it keeps its values.
|
||||
(let [props [{:name "Size" :value "Big"}]
|
||||
file (-> (variant-file)
|
||||
(thk/update-component :component1 {:variant-properties props})
|
||||
(thk/update-component :component2 {:variant-properties props})
|
||||
(ths/update-shape :root1 :variant-name "Big")
|
||||
(ths/update-shape :root2 :variant-name "Big"))
|
||||
file' (migrate-text file)
|
||||
data' (ctf/file-data file')]
|
||||
|
||||
(t/is (= props (:variant-properties (thk/get-component file' :component1))))
|
||||
(t/is (= "Big" (:variant-name (ths/get-shape file' :root1))))
|
||||
(t/is (= [{:name "Size" :value "Big (1)"}]
|
||||
(:variant-properties (thk/get-component file' :component2))))
|
||||
(t/is (= "Big (1)" (:variant-name (ths/get-shape file' :root2))))
|
||||
(t/is (= data' (cfm/migrate-data data' migration-id)))))
|
||||
|
||||
(t/deftest migration-0038-does-not-take-names-from-a-detached-component
|
||||
(let [deleted-id (uuid/next)
|
||||
main-id (uuid/next)
|
||||
@ -133,5 +155,5 @@
|
||||
file' (cfm/migrate-file file {})]
|
||||
|
||||
(t/is (contains? (:migrations file') migration-id))
|
||||
(t/is (= "SizeBig" (-> (thk/get-component file' :component2) :variant-properties first :name)))
|
||||
(t/is (some? (prop-named (:variant-properties (thk/get-component file' :component2)) "SizeBig")))
|
||||
(t/is (some? (ctf/check-file-data (:data file'))))))
|
||||
|
||||
15
common/test/common_tests/flags_test.cljc
Normal file
15
common/test/common_tests/flags_test.cljc
Normal file
@ -0,0 +1,15 @@
|
||||
;; This Source Code Form is subject to the terms of the Mozilla Public
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns common-tests.flags-test
|
||||
(:require
|
||||
[app.common.flags :as flags]
|
||||
[clojure.test :as t]))
|
||||
|
||||
(t/deftest audit-log-gc-default-can-be-disabled
|
||||
(t/is (contains? (flags/parse flags/default) :audit-log-gc))
|
||||
(t/is (not (contains? (flags/parse flags/default [:disable-audit-log-gc])
|
||||
:audit-log-gc))))
|
||||
@ -226,6 +226,127 @@
|
||||
(t/is (= (:shape-ref copy-nested-rectangle) (thi/id :nested-rectangle)))
|
||||
(t/is (nil? (ctk/get-swap-slot copy-nested-rectangle)))))
|
||||
|
||||
(t/deftest test-dont-advance-under-swapped-nested-copy
|
||||
(let [;; ==== Setup
|
||||
;; :copy-big-board [:name Big Board] #--> :big-board
|
||||
;; :copy-h-board-with-ellipse [:name Board with ellipse] @--> :h-board-with-ellipse
|
||||
;; :copy-board-with-rectangle [:name Board with rectangle, :swap-slot-label :nested-h-ellipse] @--> :board-with-rectangle
|
||||
;; :copy-nested-h-rectangle [:name Rectangle] @--> :nested-h-rectangle
|
||||
;; :copy-nested-rectangle [:name rectangle] ---> :nested-rectangle
|
||||
file (-> (setup-file)
|
||||
(thc/instantiate-component :c-big-board
|
||||
:copy-big-board
|
||||
:children-labels [:copy-h-board-with-ellipse
|
||||
:copy-nested-h-ellipse
|
||||
:copy-nested-ellipse])
|
||||
(thc/component-swap :copy-nested-h-ellipse
|
||||
:c-board-with-rectangle
|
||||
:copy-board-with-rectangle
|
||||
:children-labels [:copy-nested-h-rectangle
|
||||
:copy-nested-rectangle]))
|
||||
|
||||
page (thf/current-page file)
|
||||
|
||||
;; ==== Action
|
||||
changes (cll/generate-detach-instance (-> (pcb/empty-changes nil)
|
||||
(pcb/with-page page)
|
||||
(pcb/with-objects (:objects page)))
|
||||
page
|
||||
{(:id file) file}
|
||||
(thi/id :copy-big-board))
|
||||
file' (thf/apply-changes file changes)
|
||||
|
||||
;; ==== Get
|
||||
copy-h-board-with-ellipse (ths/get-shape file' :copy-h-board-with-ellipse)
|
||||
copy-board-with-rectangle (ths/get-shape file' :copy-board-with-rectangle)
|
||||
copy-nested-h-rectangle (ths/get-shape file' :copy-nested-h-rectangle)
|
||||
copy-nested-rectangle (ths/get-shape file' :copy-nested-rectangle)]
|
||||
|
||||
;; ==== Check
|
||||
|
||||
;; The first level copy becomes a root copy and advances one level.
|
||||
(t/is (ctk/instance-root? copy-h-board-with-ellipse))
|
||||
(t/is (= (:shape-ref copy-h-board-with-ellipse) (thi/id :board-with-ellipse)))
|
||||
(t/is (nil? (ctk/get-swap-slot copy-h-board-with-ellipse)))
|
||||
|
||||
;; The nested copy swapped in the detached copy already points to the
|
||||
;; main of board-with-rectangle, so it and its children must not advance.
|
||||
(t/is (= (:shape-ref copy-board-with-rectangle) (thi/id :board-with-rectangle)))
|
||||
(t/is (= (ctk/get-swap-slot copy-board-with-rectangle) (thi/id :nested-h-ellipse)))
|
||||
|
||||
(t/is (= (:shape-ref copy-nested-h-rectangle) (thi/id :nested-h-rectangle)))
|
||||
(t/is (= (:shape-ref copy-nested-rectangle) (thi/id :nested-rectangle)))))
|
||||
|
||||
(t/deftest test-dont-advance-under-deep-swapped-nested-copy
|
||||
(let [;; ==== Setup
|
||||
;; {:huge-board} [:name Huge Board] # [Component :c-huge-board]
|
||||
;; :big-board-in-huge [:name Big Board] @--> :big-board
|
||||
;; :h-board-in-huge [:name Board with ellipse] @--> :h-board-with-ellipse
|
||||
;; :nested3-h-ellipse [:name Ellipse] @--> :nested2-h-ellipse
|
||||
;; :nested3-ellipse [:name Ellipse] ---> :nested2-ellipse
|
||||
;; :copy-huge-board [:name Huge Board] #--> :huge-board
|
||||
;; :copy-big-board [:name Big Board] @--> :big-board-in-huge
|
||||
;; :copy-h-board-with-ellipse [:name Board with ellipse] @--> :h-board-in-huge
|
||||
;; :copy-board-with-rectangle [:name Board with rectangle, :swap-slot-label :nested-h-ellipse] @--> :board-with-rectangle
|
||||
;; :copy-nested-h-rectangle [:name Rectangle] @--> :nested-h-rectangle
|
||||
;; :copy-nested-rectangle [:name rectangle] ---> :nested-rectangle
|
||||
file (-> (setup-file)
|
||||
(tho/add-frame :huge-board :name "Huge Board")
|
||||
(thc/instantiate-component :c-big-board
|
||||
:big-board-in-huge
|
||||
:parent-label :huge-board
|
||||
:children-labels [:h-board-in-huge
|
||||
:nested3-h-ellipse
|
||||
:nested3-ellipse])
|
||||
(thc/make-component :c-huge-board :huge-board)
|
||||
(thc/instantiate-component :c-huge-board
|
||||
:copy-huge-board
|
||||
:children-labels [:copy-big-board
|
||||
:copy-h-board-with-ellipse
|
||||
:copy-nested-h-ellipse
|
||||
:copy-nested-ellipse])
|
||||
(thc/component-swap :copy-nested-h-ellipse
|
||||
:c-board-with-rectangle
|
||||
:copy-board-with-rectangle
|
||||
:children-labels [:copy-nested-h-rectangle
|
||||
:copy-nested-rectangle]))
|
||||
|
||||
page (thf/current-page file)
|
||||
|
||||
;; ==== Action
|
||||
changes (cll/generate-detach-instance (-> (pcb/empty-changes nil)
|
||||
(pcb/with-page page)
|
||||
(pcb/with-objects (:objects page)))
|
||||
page
|
||||
{(:id file) file}
|
||||
(thi/id :copy-huge-board))
|
||||
file' (thf/apply-changes file changes)
|
||||
|
||||
;; ==== Get
|
||||
copy-big-board (ths/get-shape file' :copy-big-board)
|
||||
copy-h-board-with-ellipse (ths/get-shape file' :copy-h-board-with-ellipse)
|
||||
copy-board-with-rectangle (ths/get-shape file' :copy-board-with-rectangle)
|
||||
copy-nested-h-rectangle (ths/get-shape file' :copy-nested-h-rectangle)
|
||||
copy-nested-rectangle (ths/get-shape file' :copy-nested-rectangle)]
|
||||
|
||||
;; ==== Check
|
||||
|
||||
;; The first level copy becomes a root copy and advances one level.
|
||||
(t/is (ctk/instance-root? copy-big-board))
|
||||
(t/is (= (:shape-ref copy-big-board) (thi/id :big-board)))
|
||||
(t/is (nil? (ctk/get-swap-slot copy-big-board)))
|
||||
|
||||
;; The non-swapped nested copy under it advances one level too.
|
||||
(t/is (= (:shape-ref copy-h-board-with-ellipse) (thi/id :h-board-with-ellipse)))
|
||||
|
||||
;; The nested copy swapped two levels down already points to the main
|
||||
;; of board-with-rectangle, so it and its children must not advance.
|
||||
(t/is (= (:shape-ref copy-board-with-rectangle) (thi/id :board-with-rectangle)))
|
||||
(t/is (= (ctk/get-swap-slot copy-board-with-rectangle) (thi/id :nested-h-ellipse)))
|
||||
|
||||
(t/is (= (:shape-ref copy-nested-h-rectangle) (thi/id :nested-h-rectangle)))
|
||||
(t/is (= (:shape-ref copy-nested-rectangle) (thi/id :nested-rectangle)))))
|
||||
|
||||
(t/deftest test-propagate-slot-when-swapped-main
|
||||
(let [;; ==== Setup
|
||||
file (-> (setup-file)
|
||||
|
||||
@ -1621,6 +1621,84 @@
|
||||
;; The rect still has width 25 after the switch
|
||||
(t/is (= (:width rect02') 25))))
|
||||
|
||||
(defn- setup-touched-child-in-deleted-outer
|
||||
"Variants `:c01` / `:c02` with a child rect, and a component `:external01`
|
||||
whose main holds a copy of `:c01` with `update-fn` applied to its rect.
|
||||
A copy of `:external01` is placed with its nested variant copy labelled
|
||||
`:c01-in-copy`, and then `:external01` is deleted, so the ref chain of the
|
||||
copy goes through a deleted component."
|
||||
[variant-params update-fn]
|
||||
(let [file (-> (thf/sample-file :file1)
|
||||
(thv/add-variant-with-child
|
||||
:v01 :c01 :m01 :c02 :m02 :r01 :r02
|
||||
variant-params)
|
||||
(tho/add-simple-component :external01 :external01-root :external01-child)
|
||||
(thc/instantiate-component :c01
|
||||
:c01-in-root
|
||||
:children-labels [:r01-in-c01-in-root]
|
||||
:parent-label :external01-root))
|
||||
page (thf/current-page file)
|
||||
changes (cls/generate-update-shapes (pcb/empty-changes nil (:id page))
|
||||
#{(thi/id :r01-in-c01-in-root)}
|
||||
update-fn
|
||||
(:objects page)
|
||||
{})]
|
||||
(-> (thf/apply-changes file changes)
|
||||
(thc/instantiate-component :external01
|
||||
:external-copy01
|
||||
:children-labels [:external-copy01-rect :c01-in-copy])
|
||||
(tho/delete-shape :external01-root))))
|
||||
|
||||
(t/deftest test-switch-keeps-main-width-override-when-outer-component-deleted
|
||||
(let [;; ==== Setup
|
||||
file (setup-touched-child-in-deleted-outer
|
||||
{:child1-params {:width 5}
|
||||
:child2-params {:width 5}}
|
||||
#(assoc % :width 25))
|
||||
page (thf/current-page file)
|
||||
c01-in-copy (ths/get-shape file :c01-in-copy)
|
||||
rect01 (get-in page [:objects (-> c01-in-copy :shapes first)])
|
||||
|
||||
;; ==== Action
|
||||
file' (tho/swap-component-in-shape file :c01-in-copy :c02 {:new-shape-label :c02-in-copy :keep-touched? true})
|
||||
|
||||
page' (thf/current-page file')
|
||||
c02-in-copy' (ths/get-shape file' :c02-in-copy)
|
||||
rect02' (get-in page' [:objects (-> c02-in-copy' :shapes first)])]
|
||||
|
||||
;; The outer component is deleted
|
||||
(t/is (nil? (thc/get-component file :external01)))
|
||||
(t/is (some? (thc/get-component file :external01 :include-deleted? true)))
|
||||
;; The rect had width 25 before the switch
|
||||
(t/is (= (:width rect01) 25))
|
||||
;; The rect still has width 25 after the switch
|
||||
(t/is (= (:width rect02') 25))))
|
||||
|
||||
(t/deftest test-switch-keeps-main-fill-override-when-outer-component-deleted
|
||||
(let [;; ==== Setup
|
||||
fills (ths/sample-fills-color :fill-color "#fabada")
|
||||
file (setup-touched-child-in-deleted-outer
|
||||
{}
|
||||
#(assoc % :fills fills))
|
||||
page (thf/current-page file)
|
||||
c01-in-copy (ths/get-shape file :c01-in-copy)
|
||||
rect01 (get-in page [:objects (-> c01-in-copy :shapes first)])
|
||||
|
||||
;; ==== Action
|
||||
file' (tho/swap-component-in-shape file :c01-in-copy :c02 {:new-shape-label :c02-in-copy :keep-touched? true})
|
||||
|
||||
page' (thf/current-page file')
|
||||
c02-in-copy' (ths/get-shape file' :c02-in-copy)
|
||||
rect02' (get-in page' [:objects (-> c02-in-copy' :shapes first)])]
|
||||
|
||||
;; The outer component is deleted
|
||||
(t/is (nil? (thc/get-component file :external01)))
|
||||
(t/is (some? (thc/get-component file :external01 :include-deleted? true)))
|
||||
;; The rect had the fill override before the switch
|
||||
(t/is (= (:fills rect01) fills))
|
||||
;; The rect still has the fill override after the switch
|
||||
(t/is (= (:fills rect02') fills))))
|
||||
|
||||
;; ============================================================
|
||||
;; LAYOUT ITEM SIZING - HORIZONTAL (fix, auto, fill, none)
|
||||
;; ============================================================
|
||||
|
||||
@ -211,7 +211,76 @@
|
||||
(t/is (= (count (:variant-properties comp01')) 1))
|
||||
(t/is (= (count (:variant-properties comp02)) 2))
|
||||
(t/is (= (count (:variant-properties comp02')) 1))
|
||||
(t/is (= (-> comp01' :variant-properties first :name) "Property 2"))))
|
||||
(t/is (= (-> comp01' :variant-properties first :name) "Property 2"))
|
||||
;; :m01 is higher in the layers panel, so it keeps its value.
|
||||
(t/is (= "" (-> comp01' :variant-properties first :value)))
|
||||
(t/is (= "(1)" (-> comp02' :variant-properties first :value)))
|
||||
(t/is (= "" (:variant-name (ths/get-shape file' :m01))))
|
||||
(t/is (= "(1)" (:variant-name (ths/get-shape file' :m02))))))
|
||||
|
||||
(defn- size-color-shape
|
||||
[size color shape]
|
||||
[{:name "Size" :value size}
|
||||
{:name "Color" :value color}
|
||||
{:name "Shape" :value shape}])
|
||||
|
||||
(defn- variant-set-file
|
||||
"Two variants; :m01 is higher in the layers panel than :m02."
|
||||
[props1 props2]
|
||||
(-> (thf/sample-file :file1)
|
||||
(thv/add-variant :v01 :c01 :m01 :c02 :m02)
|
||||
(thc/update-component :c01 {:variant-properties props1})
|
||||
(thc/update-component :c02 {:variant-properties props2})
|
||||
(ths/update-shape :m01 :variant-name (ctv/properties-to-name props1))
|
||||
(ths/update-shape :m02 :variant-name (ctv/properties-to-name props2))))
|
||||
|
||||
(defn- remove-property-changes
|
||||
[file pos & {:keys [sanitize-set?] :or {sanitize-set? true}}]
|
||||
(let [page (thf/current-page file)]
|
||||
(-> (pcb/empty-changes nil)
|
||||
(pcb/with-page-id (:id page))
|
||||
(pcb/with-library-data (:data file))
|
||||
(pcb/with-objects (:objects page))
|
||||
(clvp/generate-remove-property (thi/id :v01) pos :sanitize-set? sanitize-set?))))
|
||||
|
||||
(defn- values
|
||||
[file component-label]
|
||||
(mapv :value (:variant-properties (thc/get-component file component-label))))
|
||||
|
||||
(t/deftest test-remove-property-numbers-the-lower-variant-on-a-clash
|
||||
(let [file (variant-set-file (size-color-shape "Big" "red" "round")
|
||||
(size-color-shape "Big" "blue" "round"))
|
||||
file' (thf/apply-changes file (remove-property-changes file 1))]
|
||||
|
||||
(t/is (= ["Big" "round"] (values file' :c01)))
|
||||
(t/is (= ["Big" "round (1)"] (values file' :c02)))
|
||||
(t/is (= "Big, round" (:variant-name (ths/get-shape file' :m01))))
|
||||
(t/is (= "Big, round (1)" (:variant-name (ths/get-shape file' :m02))))))
|
||||
|
||||
(t/deftest test-remove-property-without-a-clash-keeps-the-values
|
||||
(let [file (variant-set-file (size-color-shape "Big" "red" "round")
|
||||
(size-color-shape "Small" "blue" "square"))
|
||||
file' (thf/apply-changes file (remove-property-changes file 1))]
|
||||
|
||||
(t/is (= ["Big" "round"] (values file' :c01)))
|
||||
(t/is (= ["Small" "square"] (values file' :c02)))
|
||||
(t/is (= "Small, square" (:variant-name (ths/get-shape file' :m02))))))
|
||||
|
||||
(t/deftest test-remove-property-without-sanitize-keeps-a-duplicate
|
||||
(let [file (variant-set-file (size-color-shape "Big" "red" "round")
|
||||
(size-color-shape "Big" "blue" "round"))
|
||||
file' (thf/apply-changes file (remove-property-changes file 1 :sanitize-set? false))]
|
||||
|
||||
(t/is (= ["Big" "round"] (values file' :c01)))
|
||||
(t/is (= ["Big" "round"] (values file' :c02)))
|
||||
(t/is (= "Big, round" (:variant-name (ths/get-shape file' :m02))))))
|
||||
|
||||
(t/deftest test-remove-property-out-of-range-does-nothing
|
||||
(let [file (variant-set-file (size-color-shape "Big" "red" "round")
|
||||
(size-color-shape "Big" "blue" "round"))]
|
||||
|
||||
(t/is (empty? (:redo-changes (remove-property-changes file -1))))
|
||||
(t/is (empty? (:redo-changes (remove-property-changes file 3))))))
|
||||
|
||||
(t/deftest test-update-property-value
|
||||
(let [;; ==== Setup
|
||||
|
||||
@ -40,6 +40,7 @@
|
||||
[common-tests.files.tokens-test]
|
||||
[common-tests.files.validate-test]
|
||||
[common-tests.files.variant-test]
|
||||
[common-tests.flags-test]
|
||||
[common-tests.geom-align-test]
|
||||
[common-tests.geom-bounds-layout-nil-test]
|
||||
[common-tests.geom-bounds-map-test]
|
||||
@ -112,6 +113,7 @@
|
||||
[common-tests.types.components-test]
|
||||
[common-tests.types.container-test]
|
||||
[common-tests.types.file-load-component-objects-test]
|
||||
[common-tests.types.file-test]
|
||||
[common-tests.types.fill-test]
|
||||
[common-tests.types.font-test]
|
||||
[common-tests.types.modifiers-test]
|
||||
@ -164,6 +166,7 @@
|
||||
'common-tests.files.tokens-test
|
||||
'common-tests.files.validate-test
|
||||
'common-tests.files.variant-test
|
||||
'common-tests.flags-test
|
||||
'common-tests.geom-align-test
|
||||
'common-tests.geom-bounds-layout-nil-test
|
||||
'common-tests.geom-bounds-map-test
|
||||
@ -236,6 +239,7 @@
|
||||
'common-tests.types.components-test
|
||||
'common-tests.types.container-test
|
||||
'common-tests.types.file-load-component-objects-test
|
||||
'common-tests.types.file-test
|
||||
'common-tests.types.fill-test
|
||||
'common-tests.types.font-test
|
||||
'common-tests.types.modifiers-test
|
||||
|
||||
97
common/test/common_tests/types/file_test.cljc
Normal file
97
common/test/common_tests/types/file_test.cljc
Normal file
@ -0,0 +1,97 @@
|
||||
;; This Source Code Form is subject to the terms of the Mozilla Public
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns common-tests.types.file-test
|
||||
(:require
|
||||
[app.common.test-helpers.components :as thc]
|
||||
[app.common.test-helpers.compositions :as tho]
|
||||
[app.common.test-helpers.files :as thf]
|
||||
[app.common.test-helpers.ids-map :as thi]
|
||||
[app.common.test-helpers.shapes :as ths]
|
||||
[app.common.types.container :as ctn]
|
||||
[app.common.types.file :as ctf]
|
||||
[clojure.test :as t]))
|
||||
|
||||
(t/use-fixtures :each thi/test-fixture)
|
||||
|
||||
(defn- setup-file
|
||||
[]
|
||||
;; {:frame-base} [:name frame-base] # [Component :base]
|
||||
;; {:frame-effects} [:name frame-effects] # [Component :effects]
|
||||
;; :base-in-effects [:name frame-base] @--> frame-base
|
||||
;; {:frame-plain} [:name frame-plain] # [Component :plain]
|
||||
;; {:frame-inner} [:name frame-inner] # [Component :inner]
|
||||
;; :plain-in-inner [:name frame-plain] @--> frame-plain
|
||||
;; {:frame-mid} [:name frame-mid] # [Component :mid]
|
||||
;; :inner-in-mid [:name frame-inner] @--> frame-inner
|
||||
;; :effects-in-mid [:name frame-effects, :swap-slot-label :plain-in-inner] @--> frame-effects
|
||||
;; [:name frame-base] @--> :base-in-effects
|
||||
;; {:frame-top} [:name frame-top] # [Component :top]
|
||||
;; :mid-in-top [:name frame-mid] @--> frame-mid
|
||||
;; :inner-in-top [:name frame-inner] @--> :inner-in-mid
|
||||
;; :effects-in-top [:name frame-effects] @--> :effects-in-mid
|
||||
;; :base-in-top [:name frame-base] @--> (base in :effects-in-mid)
|
||||
;; :top-copy [:name frame-top] #--> frame-top
|
||||
;; :mid-copy [:name frame-mid] @--> :mid-in-top
|
||||
;; :inner-copy [:name frame-inner] @--> :inner-in-top
|
||||
;; :effects-copy [:name frame-effects] @--> :effects-in-top
|
||||
;; :base-copy [:name frame-base] @--> :base-in-top
|
||||
(-> (thf/sample-file :file1)
|
||||
(tho/add-frame :frame-base :name "frame-base")
|
||||
(thc/make-component :base :frame-base)
|
||||
(tho/add-frame :frame-effects :name "frame-effects")
|
||||
(thc/make-component :effects :frame-effects)
|
||||
(thc/instantiate-component :base :base-in-effects :parent-label :frame-effects)
|
||||
(tho/add-frame :frame-plain :name "frame-plain")
|
||||
(thc/make-component :plain :frame-plain)
|
||||
(tho/add-frame :frame-inner :name "frame-inner")
|
||||
(thc/make-component :inner :frame-inner)
|
||||
(thc/instantiate-component :plain :plain-in-inner :parent-label :frame-inner)
|
||||
(tho/add-frame :frame-mid :name "frame-mid")
|
||||
(thc/make-component :mid :frame-mid)
|
||||
(thc/instantiate-component :inner :inner-in-mid :parent-label :frame-mid
|
||||
:children-labels [:plain-in-mid])
|
||||
(thc/component-swap :plain-in-mid :effects :effects-in-mid)
|
||||
(tho/add-frame :frame-top :name "frame-top")
|
||||
(thc/make-component :top :frame-top)
|
||||
(thc/instantiate-component :mid :mid-in-top :parent-label :frame-top
|
||||
:children-labels [:inner-in-top :effects-in-top :base-in-top])
|
||||
(thc/instantiate-component :top :top-copy
|
||||
:children-labels [:mid-copy :inner-copy
|
||||
:effects-copy :base-copy])))
|
||||
|
||||
(defn- swap-level
|
||||
[file label max-levels]
|
||||
(let [page (thf/current-page file)]
|
||||
(ctf/get-swap-level (ctn/make-container page :page)
|
||||
{(:id file) file}
|
||||
(ths/get-shape file label)
|
||||
max-levels)))
|
||||
|
||||
(t/deftest test-get-swap-level-shape-swapped-itself
|
||||
(let [file (setup-file)]
|
||||
(t/is (= 0 (swap-level file :effects-in-mid 5)))))
|
||||
|
||||
(t/deftest test-get-swap-level-swapped-up-the-ref-chain
|
||||
(let [file (setup-file)]
|
||||
(t/is (= 1 (swap-level file :effects-in-top 5)))
|
||||
(t/is (= 2 (swap-level file :effects-copy 5)))))
|
||||
|
||||
(t/deftest test-get-swap-level-none-found
|
||||
(let [file (setup-file)]
|
||||
;; The ref chain of the copy has no swap and reaches the main
|
||||
(t/is (= 5 (swap-level file :inner-copy 5)))
|
||||
;; The base main has no ref chain at all
|
||||
(t/is (= 5 (swap-level file :frame-base 5)))))
|
||||
|
||||
(t/deftest test-get-swap-level-capped-by-max-levels
|
||||
(let [file (setup-file)]
|
||||
;; The swap sits exactly at max-levels
|
||||
(t/is (= 2 (swap-level file :effects-copy 2)))
|
||||
;; The swap sits beyond max-levels
|
||||
(t/is (= 1 (swap-level file :effects-copy 1)))
|
||||
;; With no levels to look at, not even the shape itself is checked
|
||||
(t/is (= 0 (swap-level file :effects-in-mid 0)))))
|
||||
@ -219,13 +219,14 @@ keeps msgbus Pub/Sub channels (collab broadcasts, team-org notifications,
|
||||
file-summary cache, rate-limit counters) isolated.
|
||||
|
||||
Background workers (`enable-backend-worker`) run only on ws0 — ws1+ overlays
|
||||
disable it. ws1+ RPC handlers still enqueue tasks into the shared Postgres
|
||||
`task` table; ws0's dispatcher claims them via `FOR UPDATE SKIP LOCKED` and
|
||||
disable it. ws1+ RPC handlers still enqueue jobs into the shared Postgres
|
||||
`job` table; ws0's dispatcher claims them via `FOR UPDATE SKIP LOCKED` and
|
||||
runs them against the shared DB and RustFS. Workers are fire-and-forget:
|
||||
`wrk/submit!` inserts a row and returns; RPC handlers never wait on
|
||||
`jobs/submit` inserts a row and returns; RPC handlers never wait on
|
||||
completion. The "ws0 only" policy avoids multi-instance worker races (cron
|
||||
dedup is best-effort across instances, `wrk/submit!` `dedupe` is racy across
|
||||
submitters).
|
||||
dedup is best-effort across instances; the dedupe and exclusivity of
|
||||
`jobs/submit` are resolved by unique indexes, so the submits themselves do
|
||||
not race).
|
||||
|
||||
Each workspace is independent and can be started and stopped in any order.
|
||||
Shared infrastructure shuts down only when no instances remain running.
|
||||
|
||||
@ -56,6 +56,9 @@ desc: Learn how to organize your work in Penpot. Create, manage and organize pro
|
||||
</figure>
|
||||
<p>When creating a file, you'll be asked to give it a name. The file will open in the workspace where you can start designing immediately.</p>
|
||||
|
||||
<h3 id="open-new-tab">Open in a new tab</h3>
|
||||
<p><kbd>Ctrl</kbd> + click (<kbd>⌘</kbd> + click on macOS) opens a file or a project in a new browser tab. It works on file cards in grid and list view, on project titles and <strong>Show all files</strong> in the projects view, and on projects and Drafts in the sidebar. You can also open a file in a new tab from its menu with <strong>Open file in a new tab</strong>.</p>
|
||||
|
||||
<h3 id="file-view">Grid and list view</h3>
|
||||
<p>In the dashboard header you can switch how files are shown: a thumbnail <strong>Grid view</strong> or a compact <strong>List view</strong>. The choice is kept in the browser and applies both to the team files view and inside a project.</p>
|
||||
<p>List view shows the file name, whether the file is a shared library, the last modification time, and the same options menu as the file cards.</p>
|
||||
|
||||
@ -520,6 +520,7 @@ ExtraBold Italic
|
||||
<figure>
|
||||
<img src="/img/design-tokens/34-tokens-composite-typography-alias.webp" alt="Typography composite token" />
|
||||
</figure>
|
||||
<p>To stop using the reference and edit the properties one by one, hover the reference field and click the detach icon (<strong>Detach reference</strong>). The form switches to the individual properties, filled with the values or references of the referenced token. Nothing changes until you click <strong>Save</strong>; <strong>Cancel</strong> keeps the reference.</p>
|
||||
|
||||
<h4 id="line-height-property">Line height property</h4>
|
||||
<p>The <strong>Typography Token</strong> includes a <em>line-height</em> property, which is not available as an individual token. This is because line-height depends on the font size to be calculated properly. Make sure the <em>font-size</em> property is defined before setting <em>line-height</em>.</p>
|
||||
@ -571,6 +572,7 @@ ExtraBold Italic
|
||||
</figure>
|
||||
<p>When creating a shadow with individual values, the color value starts empty, but the other inputs have default values (X: 4, Y: 4, Blur: 4, Spread: 0). You can reorder shadows by hovering over a shadow form and using the reorder button to drag it to a different position.</p>
|
||||
<p>You can also reference another existing shadow token instead of defining each property manually. When doing so, Penpot resolves all shadow properties from the referenced token.</p>
|
||||
<p>To edit the shadows of a referenced token without the reference, hover the reference field and click the detach icon (<strong>Detach reference</strong>). The form switches to individual values, filled with the shadows of the referenced token. Nothing changes until you click <strong>Save</strong>.</p>
|
||||
|
||||
<h4 id="design-tokens-shadow-apply">Applying shadow tokens</h4>
|
||||
<p>Shadow tokens can be applied to any layer type. Clicking on a shadow token will apply it to the selected layer. Right-clicking on a shadow token shows the context menu with the <strong>Shadow</strong> option to apply it.</p>
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
x
Reference in New Issue
Block a user