diff --git a/.serena/memories/audit-log.md b/.serena/memories/audit-log.md index 8097bd901c..3739612214 100644 --- a/.serena/memories/audit-log.md +++ b/.serena/memories/audit-log.md @@ -118,6 +118,21 @@ Critical `profile-id` INVARIANT: the event belongs to **whoever made the request The `::audit/profile-id` override goes through `coerce-profile-id`: a string is parsed, anything that cannot become a uuid is dropped (with a warning) and the event falls back to the caller. Do not pass the value raw: `schema:event` needs a uuid and `submit*` swallows the validation error (it only logs, the RPC still succeeds), so a non-uuid override **loses the row silently**. Hand-built events with `event-from-rpc-params` + `submit` (clone template, accept team invitation, `management/nitrate` push-audit-events) do **not** go through that coercion: they must already carry a uuid. +### Team id and entity ids (projection data) + +The event stream is meant to let a consumer (Nexus) rebuild what it needs **without joining production tables**, so events carry the ids a projection keys on: + +- **Team id enrichment.** After `clean-props`, `prepare-rpc-event` calls `enrich-team-id`: when props have no `:team-id` but carry a `:project-id` or `:file-id`, it resolves the team with a local query and adds `:team-id`. It never resolves from `:id` (read commands put the entity id there, and that would add a query to every read), and it skips read events entirely (`get-*` names plus `has-file-libraries`: their events carry no consumer state). Commands that already loaded the team add `:team-id` themselves in `::audit/props` (`delete-file`, comments, media upload, `delete/update-webhook`, `clone-template`, ...), so the resolver does not run. +- **Membership.** `add-profile-to-team!` emits `add-team-member` (props `:team-id`, `:member-id`, `:role`, `:is-owner`, `:is-admin`, `:can-edit`) whenever it actually inserts a row, only with `:audit-log` on, inside the caller's transaction. It covers the flows that add a member with no event of their own (`create-team` owner, invitee added directly when `:email-verification` is off, approved join-request, duplicated team). With `on-conflict-do-nothing?` a conflict returns nil and emits nothing, so an existing membership is reported once. The event belongs to the actor: the two flows where the owner adds somebody else (`create-invitation`, `add-member-to-team`) pass `:actor-id` through a separate `event-opts` arg (never in `params`, which is inserted as-is); everywhere else actor and member coincide. `:role` is derived in the emitter from the flags (inverse of `permissions-for-role`: owner > admin > editor > viewer) because the table has no role column; request context travels in `event-opts` as `:rpc-params`. +- **Created entity ids.** `create-project` and `create-file` add the new row's `:id` (and `:project-id`/`:team-id`) to `::audit/props`, so a projection can key it. +- **Bulk operations.** `clone-template`, `import-binfile` and `duplicate-project` create entities through internal calls that emit nothing on their own; they emit a `create-file` (and `create-project`) per entity (only with `:audit-log`). The import paths go through `audit/submit-file-created-event`, which reads the light columns (`id`, `name`, `project-id`, timestamps, `revn`) back from the row — the file must really exist, a phantom id emits nothing. `duplicate-project` already holds the rows and uses `audit/submit-create-event` directly. Both binfile versions return the same shape from `import-files!` (`{:file-ids [...]}`). +- **Deletes.** `delete-team` still returns nil but carries the deleted row (`:id`, `:name`, `:created-at`, `:modified-at`, `:deleted-at`) in `::audit/props`, like `delete-project`. `delete-profile` carries `:id`, `:email`, `:fullname`, `:created-at`, `:modified-at` and `:deleted-at` (without this its props are empty: every param is namespaced and `clean-props` drops them). `create-team` adds `:created-at` and `:modified-at`. `delete-team-member` needs nothing: the delete is physical and the params already carry `:team-id`/`:member-id`. +- **Hard deletes.** `objects-gc` emits one `hard-delete-objects` event per purged chunk, carrying only that chunk's entity key (`:team-ids`, `:project-ids` or `:file-ids`; one chunk transaction purges a single entity, so one event can never span sections) with context `{:triggered-by "objects-gc"}`, only with `:audit-log` on. No other GC path emits (profiles, fonts, thumbnails); those purges stay silent. +- **SREPL deletes.** `delete-profile!` carries the subject id in props (it used to emit nothing usable); the event profile stays zero because SREPL has no actor. `delete-team!` props already carried the id. Both travel as actions with a `triggered-by: srepl` context. +- **SREPL restores.** `restore-profile!` carries the full row (with id) in props with a zero event profile; `restore-team!` and `restore-project!` already carried their ids. The consumer resolves the subject from props by a uniform rule, with no SREPL special case. +- **Cascades.** The `:delete-object` worker task emits one `delete-object-with-cascade` event per execution, type `action`, with `:object`/`:id`/`:deleted-at` plus the affected ids nested under `:cascade` (`:team-ids`, `:project-ids`, `:file-ids`), in the same transaction. The handler submits unconditionally (`submit` fans out per flag, so no `:audit-log` re-check); the `audit_log` row itself is still only written with `:audit-log` on. Direct calls (objects-gc) and snapshot cascades stay silent on purpose. +- **Ownership.** `create-team-with-invitations` carries the new team `:id` like `create-team`; `update-team-member-role` names the demoted member in `:prev-owner-id` on owner transfers. + Entry points: - `submit`: the normal one (fills defaults, validates, runs inside `tx-run!`, never breaks the RPC). @@ -223,6 +238,8 @@ Webhooks let an external URL get notified when something happens in a team (for One guarantee: sending webhooks never changes the audit event itself. If delivery fails, only the webhook (and its delivery log) is affected; the audit row stays untouched and still gets archived normally. +Every match also writes a `webhook` / `trigger` row to `audit_log` (even with zero matches): its props are `{:name , :event-id , :total-affected }`. `process-event` assigns the origin row id up-front so the trigger can reference it; the telemetry shadow row always gets its own id (reusing it would collide on the PK). + ## 10. Consumers II: error reporters Both `app.loggers.database` and `app.loggers.mattermost` listen to backend `:error` logs **and** frontend crash events (found by the `::audit/event` marker), through a sliding-buffer channel so a flood cannot stall the app. The database one stores them in `server-error-report` (source 3 = backend log, 4 = audit-log origin, 5 = rate-limit); the Mattermost one forwards a short note to `:error-report-webhook` when set. @@ -260,6 +277,8 @@ Nexus dispatches on `:type`: `"telemetry-events"` batches are coerced and insert Nexus is the long-term store. Every 5 min the `:audit-log-archive` cron takes 128-row chunks of unarchived rows (`FOR UPDATE SKIP LOCKED`, ordered by `created_at`), POSTs them as transit `{:events [...]}` with `x-shared-key: "nexus "` (`:nexus-shared-key`, or derived from the instance secret) and `origin` = public-uri, and marks `archived_at = now()` **only on HTTP 204**, in the same transaction. Anything else is retried next run; flag on but no URI raises `:task-not-configured`. It loops with 100 ms pauses until empty. +A chunk is **one POST** and the task advances only on 204, so a single oversized event wedges the whole queue: a crash report whose message embeds a whole payload (a serialized file, for example) can push the request past the proxy body limit and the rejection blocks every later event until it is pruned by hand. Frontend crash reports are therefore size-bounded at the source (`mem:frontend/routing-app-shell-subtleties`): `app.common.exceptions/get-hint` and the CLJS `format-throwable` clamp exception messages (`max-message-length`). + On the other side, Nexus checks `origin` against its allowlist (403 if not allowed), coerces the events, and inserts them into its own `audit_log` (partitioned by `received_at`, with an incrementing sequence number and `source`). Watch the rename: Penpot's `created-at` is stored as **`received-at`** in Nexus; Nexus's `created-at` is when it was created there. `tracked-at` keeps the origin time. Telemetry batches arriving at `telemetry.penpot.app` land in the same table through the telemetry endpoint above. Concurrent runs are safe: rows are locked with `FOR UPDATE SKIP LOCKED` inside the same transaction that sends and marks them, so two archivers running at once each take different rows. Only rows Nexus acknowledges (HTTP 204) get marked; anything else is retried on the next run. @@ -306,13 +325,15 @@ Config (`backend/src/app/config.clj`): `:audit-log-archive-uri`, `:telemetry-uri - **New backend command that must (or must not) emit**: `wrap-audit` does it alone; to exclude it set `::audit/skip`. To customize props/name/type use result metadata. Add a test mocking `app.loggers.audit/submit` (nil; `helpers.clj` stubs it globally) and assert `:call-args-list` — see `backend_tests/rpc-audit-test.clj` (full-row, telemetry-only, dual, no-op without flags, `insert`, `prepare-rpc-event`) and `rpc-team-test/accept-organization-invitation-audit-event` (three cases, exact props; counts matter: asserting with `first` passes even when a command emits the same name twice). - **New Penpot frontend event**: create the data-event or implement `Event`/`PerformanceEvent` in `app.main.data.event`, with `::name`/`::type`/`::origin`. Honor `skip-audit?` if the action can resume. Remember: with no props schema, **your asserts are the contract**. - **New admin-console event**: server action → add the backend-log call next to the mutation, best-effort so it never breaks the action; browser action → add a new event helper plus its test. +- **Command that creates an entity or already knows its team**: put the new row's `:id` in `::audit/props` (`create-project`/`create-file`) and set `:team-id` when the team is loaded, so the consumer never has to join production tables. For entities created inside an internal call (`clone-template`, `import-binfile`, `duplicate-project`), emit one event per entity: `audit/submit-file-created-event` when only the id is known (it reads the row back), `audit/submit-create-event` when the row is already in hand. - **Never** put secrets in `props`; `clean-props` is a last defense, not an excuse. - **Do not duplicate events**: if the backend already emits an event for an action, do not add a frontend (or admin-console browser) event for the same action — the backend one wins because it is reliable. A second event for the same action is only acceptable with a strong reason (for example the backend cannot see it at all), and then it must be clearly justified. - **Follow the naming policy**: use kebab-case names (`"create-organization"`, not `"createOrganization"`), and before inventing prop keys, look at similar existing events and reuse their key names (`organization-id`, `team-id`, `file-id`, `event-origin`...). Same concept, same key spelling — that is what keeps the data queryable later. ## 15. Code map -- Backend producer: `backend/src/app/loggers/audit.clj` (`submit`, `submit*`, `insert`, `prepare-rpc-event`, `event-from-rpc-params`, telemetry filters), `backend/src/app/rpc.clj` (`wrap-audit`), `backend/src/app/rpc/commands/audit.clj` (`push-audit-events`, `get-environment-data`, deprecated `get-enabled-flags`), `backend/src/app/rpc/management/nitrate.clj` (`push-audit-events` Nitrate→Penpot), `backend/src/app/http/middleware.clj` + `backend/src/app/http/management.clj` (shared-key auth). +- Backend producer: `backend/src/app/loggers/audit.clj` (`submit`, `submit*`, `insert`, `prepare-rpc-event`, `enrich-team-id`, `submit-create-event`, `event-from-rpc-params`, telemetry filters), `backend/src/app/rpc.clj` (`wrap-audit`), `backend/src/app/rpc/commands/audit.clj` (`push-audit-events`, `get-environment-data`, deprecated `get-enabled-flags`), `backend/src/app/rpc/management/nitrate.clj` (`push-audit-events` Nitrate→Penpot), `backend/src/app/http/middleware.clj` + `backend/src/app/http/management.clj` (shared-key auth). +- Membership emitter: `backend/src/app/rpc/commands/teams.clj` (`add-profile-to-team!` → `add-team-member`). - Penpot frontend producer: `frontend/src/app/main/data/event.cljs` (collector, buffer, send), `frontend/src/app/main/repo.cljs` + `frontend/src/app/util/http.cljs` (headers), `frontend/src/app/main/data/nitrate.cljs` (Nitrate membership events). - Admin-console producer (private repo, no paths cited here): server-side organization/subscription events through the management `push-audit-events`, browser-side UI events through the public `push-audit-events`. - Consumers: `backend/src/app/loggers/webhooks.clj`, `backend/src/app/loggers/database.clj`, `backend/src/app/loggers/mattermost.clj`, `backend/src/app/tasks/telemetry.clj`. diff --git a/.serena/memories/backend/storage.md b/.serena/memories/backend/storage.md index beb591e810..f2e582c826 100644 --- a/.serena/memories/backend/storage.md +++ b/.serena/memories/backend/storage.md @@ -25,8 +25,9 @@ - Existing object values can remain readable until physical deletion. - `:expired-at` blocks reads after the expiration time. - `del-object!` sets `deleted_at` on live rows only (`deleted_at IS NULL`): a repeated call returns `false`. It does not remove backend content. -- `storage-gc-deleted` removes the database row and backend content after the deletion delay, streaming the pending rows with `db/plan` inside a read-only transaction and deleting them in chunks of 25, each chunk in its own transaction: a chunk that fails (for example, an object a foreign key still references) is logged and skipped instead of aborting the run. A physical-deletion failure inside a chunk defers its rows one day (`deletion_attempts + 1`) and gives up after 7 attempts, accepting the orphan blob; chunk mappings (`upload_session_chunk`) are removed before the `storage_object` rows (NO ACTION foreign keys). +- `storage-gc-deleted` removes the database row and backend content after the deletion delay, streaming the pending rows with `db/plan` inside a read-only transaction and deleting them in chunks of 25, each chunk in its own transaction: a chunk that fails (for example, an object a foreign key still references) is logged and skipped instead of aborting the run. A physical-deletion failure inside a chunk defers its rows one day (`deletion_attempts + 1`) and gives up after 7 attempts, accepting the orphan blob. Deferred rows keep their references until the final give-up. The freeze path re-verifies references under row locks, so a concurrent purge can delay a freeze but never swallow a touch. - `storage-gc-touched` finds references before it sets `deleted_at`. +- Orphan watch: `SELECT COALESCE(metadata->>'bucket', metadata->>'~:bucket') AS bucket, count(*) FROM storage_object so WHERE touched_at IS NULL AND deleted_at IS NULL AND status = 'valid' AND NOT EXISTS (SELECT 1 FROM upload_session_chunk usc WHERE usc.object_id = so.id) GROUP BY 1;` Only the `upload-session` row of that output is a leak signal (its rows always have a mapping or a pending touch/deleted mark); other buckets use domain-table references, not this mapping. - `objects-gc` removes deleted domain rows and touches their storage object IDs. Each proc streams its pending keys with `db/plan` inside a read-only transaction and deletes them in chunks, each chunk in its own transaction with `FOR UPDATE SKIP LOCKED`: a chunk that fails is logged and skipped instead of aborting the run. - Use `::db/reuse-conn true` with `sto/resolve` inside a database transaction. @@ -91,7 +92,7 @@ Since `put-object!` uses backend-specific operations (`impl/resolve-backend` + ` | `profile` | User and team profile photos. References: `profile.photo_id` and `team.photo_id`. | Yes | Authentication required | Reference scan. | | `organization` | Organization logos uploaded by the Nitrate management API. | Yes | Public | No reference scan. A touched object is deleted. | | `tempfile` | Export files and temporary font downloads. | No | Authentication required | No reference scan. A touched object uses a two-hour deletion delay. | -| `upload-session` | Chunked-upload chunks. References: `upload_session_chunk.object_id` and `upload_session_chunk.session_id` (both NO ACTION DEFERRABLE: restrict semantics, procedural deletion). | No | Authentication required | No reference scan. A touched object is deleted after the delay; `gc-deleted` removes mappings before rows. | +| `upload-session` | Chunked-upload chunks. References: `upload_session_chunk.object_id` and `upload_session_chunk.session_id` (both NO ACTION DEFERRABLE: restrict semantics, procedural deletion). | No | Authentication required | Reference scan on `upload_session_chunk.object_id`; a referenced chunk freezes, an orphan is deleted after the delay. Deferred rows keep their mappings until give-up. | | `file-data` | Encoded file data when `file-data-backend` is `storage`. Reference metadata has `storage-ref-id`, `file-id`, and the `file_data` row ID. | Yes | Authentication required | Reference scan. | | `file-data-fragment` | Compatibility value for file-data fragments. The current backend has no dedicated producer for this bucket. | No current write semantics | Public | No touched-object collector case. | | `file-change` | Compatibility value for file changes. Current snapshots store data in `file_data`, not this bucket. | No current write semantics | Authentication required | No touched-object collector case. | diff --git a/.serena/memories/backend/subtleties.md b/.serena/memories/backend/subtleties.md index c95dd5d174..d046fb6f71 100644 --- a/.serena/memories/backend/subtleties.md +++ b/.serena/memories/backend/subtleties.md @@ -73,6 +73,10 @@ - `app.msgbus` (`pub!`, `sub!`, `purge!`) keeps taking the **instance** on purpose: msgbus is what produces it, and `app.rpc.notifications` already shows the caller passing a cfg. `app.jobs` resolves `::mbus/msgbus` from the cfg internally, and `insert-event` keeps its explicit pre-insert check so a profile job never loses its notification. - `app.db/after-commit` is the boundary for SQL-backed metrics: callbacks are drained by the commit of the transaction that owns the connection they were registered on, and discarded when it rolls back. Because nesting follows the connection, that transaction is the outermost one only for a real nested call. +## Log-derived metrics + +- `app.loggers.metrics/reporter` counts the log records the application emits and publishes them as `penpot_log_events_total{logger, level}` (counter), so the error panels do not have to count text in Loki. It is a component that hooks `add-watch` on `app.common.logging/log-record`, like `app.loggers.database`, but with no thread and no channel: the increment is in memory, so it goes straight in the watcher. Only `warn`, `error` and `fatal` are counted, and only the records that go through `app.common.logging` (the application's own logs; a library writing straight to log4j2 is not counted, and today that is one line in a thousand). The body of the watcher is wrapped in a `try/catch` that swallows the error on purpose: the watcher fires from the `swap!` of the log record, inside the logging path. With `::l/sync?` (`app.binfile.v1`, `app.msgbus`) an exception there would reach whoever was logging; on the async path it escapes to the logging executor worker, whose trace goes to the thread's uncaught handler -- outside the application's logging -- and the pool replaces the dead worker, so logging continues and that line is lost. The catch also makes a publication that fails every time invisible: `penpot_log_events_total` stays flat and nothing is written about it. The published `level` values are lowercase (`warn`), unlike the `level` label of the Loki logs (`ERROR`), so a panel that moves from one source to the other has to change it. Cardinality is `logger` (fixed names taken from the code) times `level`; a logger built from a dynamic value would break that. + ## Storage and media - Storage abstraction, backend configuration, logical buckets, object lifecycle, deduplication, access rules, and garbage collection: `mem:backend/storage`. diff --git a/.serena/memories/clojure/idioms.md b/.serena/memories/clojure/idioms.md index d7f40ce740..7e81fba7ac 100644 --- a/.serena/memories/clojure/idioms.md +++ b/.serena/memories/clojure/idioms.md @@ -10,3 +10,4 @@ Behaviors confirmed against the language/stdlib — do not re-derive from assump - `^:async` fns never throw synchronously (rejected promises instead); `try/catch/finally` supported; continuations are microtasks, timers and RxJS schedulers are macrotasks (FIFO). - `!` suffix allowed cases (only exceptions to the no-new-`!` rule in `mem:clojure/design-rules`): mutating public `app.db` API (`insert!`, `update!`, `delete!`, `run!`, `tx-run!`, `exec!`, ...) and fns whose contract is to work over transients (stdlib `conj!`, `assoc!`, `dissoc!`, `disj!`, `persistent!` and custom helpers over them); any further case must be registered here with justification before use. - Penpot extends `clojure.core/Inst` in `app.common.time` ( JVM-only section), so `inst-ms` also accepts `Duration` (returns `.toMillis`, NOT epoch millis), `FileTime` and `OffsetDateTime`. It only works where that namespace is loaded; on a bare `Duration`, prefer explicit `.toMillis`, which reads as what it is (an amount, not an instant). +- `cuerdas.core/index-of` returns `nil` (not `-1`) when the substring is absent, like `clojure.string/index-of`. `(pos? nil)` **throws on the JVM** but evaluates to `false` in CLJS (JS coercion), so `(if (pos? (str/index-of s "\n")) …)` works in the browser and crashes the backend. Guard with `some?` before `pos?` (this was `app.common.exceptions/first-line`). diff --git a/.serena/memories/frontend/routing-app-shell-subtleties.md b/.serena/memories/frontend/routing-app-shell-subtleties.md index 989b1d8bad..8c43a30161 100644 --- a/.serena/memories/frontend/routing-app-shell-subtleties.md +++ b/.serena/memories/frontend/routing-app-shell-subtleties.md @@ -14,9 +14,14 @@ - Frontend error handling treats stale cross-build JS chunk failures specially: messages containing `$cljs$cst$` or `$cljs$core$I` plus undefined/null/not-a-function signatures trigger throttled reload. - Plugin-originated uncaught errors are identified through the plugin runtime hook and logged rather than turning into the global exception page. - `app.main.errors/submit-report` is governed by a dedup governor: each report carries a fingerprint (`report-name|type|code|hint|first stack frame`; the report name is part of it so a handled report never coalesces with an unhandled/exception-page one), the first occurrence is always emitted, repeats within 2 minutes are counted and included in the next emitted report as `:occurrences`, and the fingerprint cache is bounded (first-inserted entry evicted, FIFO, via `:order` queue) so memory stays fixed. It applies to `handled-exception`, `unhandled-exception` and `exception-page`; a report without an exception cause is ignored and does not consume a reservation. -- Errors caused by the environment (`environment-error-types`: `:network`, `:offline`, `:bad-gateway`, `:service-unavailable`, `:nitrate-unavailable`, `:nitrate-not-configured`) are not application defects: they are reported as audit-only `handled-exception` (never `unhandled-exception`/`exception-page`, so they skip internal reports and alerts), with a compact report and a fingerprint that drops the stack frame. `:offline` has its own handler and no longer falls through to `:default`; `:network`/`:offline` show the `errors.connection-error` toast. +- Errors caused by the environment (`environment-error-types`: `:network`, `:offline`, `:bad-gateway`, `:service-unavailable`, `:gateway-error`, `:nitrate-unavailable`, `:nitrate-not-configured`) are not application defects: they are reported as audit-only `handled-exception` (never `unhandled-exception`/`exception-page`, so they skip internal reports and alerts), with a compact report and a fingerprint that drops the stack frame. `:offline` has its own handler and no longer falls through to `:default`; `:network`/`:offline`/`:gateway-error` show the `errors.connection-error` toast. +- `repo/handle-response` classifies responses the backend never wrote (body is not a transit map) by status: 504 and Cloudflare 520–524 → `:gateway-error` (retryable, so GETs and saves retry); 429 → `:rate-limit`; any other ≥400 → `:unexpected-response` (both toast-only via `handle-transport-error`, never the exception page). `:internal :unable-to-process-repository-response` is left for a <400 response that does not decode. Uninterpretable bodies are capped at 500 chars in `:data`. - `generate-report` accepts an explicit `{:format :compact|:full}` (default `:full`) chosen by its caller (`flash`, `exception-section*`): `:compact` keeps the context header plus type/code/uri, and skips the stack, the `ex-data` dump and the last-events list. It is total: if formatting fails it returns a minimal fallback string instead of nil, so an already reserved emission is never dropped. - `flash` runs its whole body (report pipeline first, toast after) inside a single `ts/schedule` callback: nothing executes synchronously on the error handler's stack. The report is reserved before being generated, so suppressed occurrences do not pay the `generate-report` cost; a reporting or notification failure is logged to the console and never propagates, and the toast is still attempted. `flash` returns a total promise (never rejects) resolving with the generated report, or nil when nothing is emitted, once the callback completes; production callers ignore it, tests await it. `flash` derives only the payload format from the cause (`environment-error?` → `:compact`); the audit event name is the canonical one requested by `:type` (`handled-exception`/`unhandled-exception`) and is never reclassified, because external tools filter on those names. `exception-section*` picks `handled-exception`/`exception-page` explicitly per cause; `flash-persistence` only adds the toast hint. +- `flash` accepts a `:links` vector of `{:label :callback}` actions, appended after the download link that `:report-link?` produces. `flash-persistence` passes a reload link, so a terminal save failure hands the user both the report and the only action that restarts persistence for the session. `labels.reload-page` plus `js/location.reload` is the established full-restart pattern; the link cannot be invoked under the test runner because `js/location` does not exist there, so it is covered by its label and by being callable. +- `flash-persistence` only delegates to another handler for `:authentication`, `:not-found` and `:code :vern-conflict`. Everything else it reports itself, and the report type is derived from the cause: `#{:validation :server-error :internal}` are application defects and report as `unhandled-exception` so they reach the internal reports and the alerts; the rest stay `handled-exception`. `:cause-type` wins over `:type` when classifying, because `persistence-failed` wraps the original cause under `:type :persistence`. Environment failures are deliberately kept handled: they land here after the retry budget is spent and the user's connection is not our bug. `:assertion` is also kept handled, matching its own handler. +- The report governor is global and stateful across tests: two reports with the same fingerprint inside `report-window-ms` collapse. Call `errors/reset-report-governor!` between assertions that expect an emission. +- Error messages are bounded at the source: `app.common.exceptions/get-hint` clamps to `max-message-length` (1000 chars) and the CLJS `format-throwable` clamps the message line of the stack trace and honors `:trace-length`. A message can embed a whole payload (a serialized file, for example) and reach megabytes; the bound keeps the audit `:hint` prop and the `:report` string small. The `Context: Hint:` header in `generate-report` prints `get-hint` (not the raw `:hint` from `ex-data`) for the same reason. ## Store and websocket @@ -24,4 +29,4 @@ For general store mechanics such as `emit!`, `last-events`, persistence, and und - Websocket initialization uses `cf/public-uri` joined with `ws/notifications`, converting `http/https` to `ws/wss`, and includes the current `session-id` as query param. - Reinitializing or finalizing websocket stops the previous receive stream. Incoming websocket payloads become Potok data events under `app.main.data.websocket/message`. -- A job the client created is followed with `app.main.data.jobs/watch-job`: it emits the `:job-event` messages of that job (filtered by `:job-id`) and one final outcome read from the row with `get-job`, which is read when the stream starts and again on every `opened` of the websocket, so a job that ended before the client started following it, or while the socket was down, still answers. The stream ends after that outcome, and a read that fails reaches the caller: the repo retries the transient transport errors of a GET, and anything else is a real failure the flow has to show. The `:stage` of a progress event is a stable key of the vocabulary of the worker, never a display string: `app.main.ui.jobs.progress` resolves it with a `case` of literal `tr` calls, falls back to a generic label for a stage it does not know, and the counter of the activity is named after its own stage (the counter of `:files` is the outer context). \ No newline at end of file +- A job the client created is followed with `app.main.data.jobs/watch-job`: it emits the `:job-event` messages of that job (filtered by `:job-id`) and one final outcome read from the row with `get-job`, which is read when the stream starts and again on every `opened` of the websocket, so a job that ended before the client started following it, or while the socket was down, still answers. The stream ends after that outcome, and a read that fails reaches the caller: the repo retries the transient transport errors of a GET, and anything else is a real failure the flow has to show. The `:stage` of a progress event is a stable key of the vocabulary of the worker, never a display string: `app.main.ui.jobs.progress` resolves it with a `case` of literal `tr` calls, falls back to a generic label for a stage it does not know, and the counter of the activity is named after its own stage (the counter of `:files` is the outer context). diff --git a/.serena/memories/frontend/workspace-state-persistence-subtleties.md b/.serena/memories/frontend/workspace-state-persistence-subtleties.md index be612121e1..57094e1994 100644 --- a/.serena/memories/frontend/workspace-state-persistence-subtleties.md +++ b/.serena/memories/frontend/workspace-state-persistence-subtleties.md @@ -11,21 +11,27 @@ ## Repo calls - `app.main.repo/send!` uses GET only when the RPC name starts with `get-`, when all params are query params, or for configured special cases. Only GET requests are retried. -- GET retry is limited to transient `:network`, `:bad-gateway`, `:service-unavailable`, and `:offline` errors with exponential backoff. Mutations are not retried. +- `repo/retryable-types` is public and is the single source of truth for the transient transport classification. `persistence/transient-error?` and `errors/environment-error-types` both derive from it; extend that set, never copy it. +- GET retry is limited to transient `:network`, `:bad-gateway`, `:service-unavailable`, `:gateway-error` (504, Cloudflare 52x), and `:offline` errors with exponential backoff. Mutations are not retried. - A server SSE response is only accepted when the command is configured `:stream?`; otherwise it raises an unexpected-response assertion. ## Commits, undo, persistence -- `commit-changes` refuses to create commits unless `:permissions :can-edit` is true. It captures file revn/vern, selected-before, features, tags, undo group, and translation flag into a `::commit` event. +- `commit-changes` refuses to create commits unless `:permissions :can-edit` is true. It captures file revn/vern, selected-before, features, tags, undo group, and translation flag into a `::commit` event. A refusal answers with an empty stream, never nil, so callers subscribe uniformly instead of branching on nil. - Applying a remote commit first rolls back pending local commits, applies the remote changes, then replays pending local redo changes. Index updates are emitted for undo, remote redo, and replayed redo paths. - Local commits are independently consumed by undo, persistence, WASM model updates, thumbnail/library watchers, and text position-data recalculation. - Persistence buffers local commits: status becomes pending after about 200ms, commits are flushed after about 3s or `::force-persist`, and buffered commits are merged per file before `:update-file`. - Persistence sends revn as the max of the commit revn and locally tracked latest revn; remote commits update that revn tracker. - Persistence is skipped in version preview/read-only mode or without edit permission. -- Save failures split transient vs terminal (`transient-error?`: the repo retryable types `:network`/`:offline`/`:bad-gateway`/`:service-unavailable` plus `:invalid-save-response`; everything else is terminal). Terminal keeps the `:error` halt + `flash-persistence` path; transient enters a `:retrying` episode: the head commit stays queued and resends with backoff 2s/8s/20s (3 retries, then today's terminal path). Resends rotate the `::request-id` stamp only when the old request left `active-requests`, carry the same `:commit-id`, and never double-send while one request is in flight (`:in-flight` stays silent). Retry timers carry the episode token; a superseded token stays silent. Status stays `:retrying` through re-entries (`next-status` refuses `:pending`/`:saving` from it); waiters (`wait-persisted-or-error`) wait through it and reject only on `:error`. +- A terminal failure halts the queue on purpose: the edits stay queued so `wait-persisted-or-error` keeps rejecting, and `workspace/versions.cljs` relies on that to avoid snapshotting a file whose edits were never saved. `recover-persistence` lifts the halt when a file is opened again or the role changes. A transport failure still inside its retry window (`slow-retry-pending?`) is resent once (`resume-persistence true`), never released; any other halt emits `release-failed-file` (drops every queued commit of the file that failed) then `resume-persistence`. Opening a file also replays the queued edits of its version onto the loaded data (`pending-file-changes` → `restore-pending-changes` in `bundle-fetched`); a replay that throws keeps the server data and the queue. That is what unblocks the editor without a reload; because the queue is shared, dropping the failed file is also what lets other files' pending edits drain. `release-failed-file` runs at recovery time, never at failure time — clearing the error eagerly would make waiters resolve. +- The split between `initialize-persistence-state` and `recover-persistence` is deliberate: initialization deliberately does NOT lift a halt, so undeliverable edits are released when the user acts, not when the view mounts. +- **Every non-transport terminal save failure is permanent.** There is no "stale but valid" category: permission lost, file gone, and server-rejected changes all fail again on a resend. Losing edit permission is not temporary — the backend raises `:not-found` for it (`check-edition-permissions!` → `perms/make-check-fn`), and `:not-found` routes to the exception page. So `:save-permission-denied` (a client-side `can-edit` check) must never be resumed: replaying the save to discover that routes the user to the exception page. Release it instead. +- Save failures split transient vs terminal (`transient-error?`: the repo retryable types `:network`/`:offline`/`:bad-gateway`/`:service-unavailable`/`:gateway-error` plus `:invalid-save-response`; everything else is terminal). Terminal keeps the `:error` halt + `flash-persistence` path; transient enters a `:retrying` episode: the head commit stays queued and resends with backoff 2s/8s/20s (3 retries, then the `:error` path). A transient failure that outlasts the backoff arms `slow-retry-cycle`: one silent resend (`:recovering`) every `slow-retry-delay-ms` (30s) until a save lands, `retry-give-up-ms` (24h since `:failing-since`) passes, or the head commit collects `max-slow-failures` (3) attempts that failed after `slow-failure-ms` (90s) or more (`:slow-failures`, counted in `persistence-transient-failure` from the attempt's elapsed time): a gateway timeout may leave the backend applying the save, so a save that keeps failing slowly must not be replayed for a day, while fast failures (offline) never count; the cycle stops on `::resume-persistence`, `::initialize-persistence` and `finalize-workspace`. Leaving the workspace also emits `suspend-retries` (`:suspended` in `:persistence`, drops `:retry-token` so a pending backoff timer goes stale): no backoff, slow cycle or `online` resend runs outside the workspace; only `workspace-initialized` (not a role change) lifts it via `recover-persistence` (`opened-file?`), which also resends a `:retrying` episode whose backoff was cut short. Its toast is `errors.save-still-retrying` without a reload link; a landing save hides the failure toast (`errors/persistence-failed-tag`). Replaying a save is safe because changes are idempotent, so there is no unknown-outcome guard: a commit whose earlier request left `active-requests` is simply sent again under the same `:commit-id`; only a request still in flight blocks a resend (`:in-flight` stays silent). Tests hold the slow cycle by stubbing `rx/timer` to never fire for `slow-retry-delay-ms`, or instant timers loop forever. Retry timers carry the episode token; a superseded token stays silent. Status stays `:retrying` through re-entries (`next-status` refuses `:pending`/`:saving` from it); waiters (`wait-persisted-or-error`) wait through it and reject only on `:error`. - One reconnect notice per episode: sticky toast tagged `:persistence-reconnecting` (single-toast store, re-show replaces), hidden by tag on drain (`:saved`) and on terminal failure; recovery is silent. Header indicator has a `:retrying` state (`workspace.header.retrying`). - Resume triggers: backoff timer and the browser `online` event (guarded by `exists? js/window`; re-enters the runner only for a live `:retrying` episode). New local edits during `:retrying` only join the queue: the episode keeps its `:run-id`, so `append-commit` does not re-enter, and the live runner (still waiting on the head's `commit-persisted`) sends them after the head. Re-entering on edits would bypass the backoff and spend an attempt per batched commit. - Tests instant-trigger retries by stubbing `rx/timer` (recording delays to assert the schedule); dynamic bindings do not survive `await` continuations, so no dynamic var for delays. +- `save-request-timeout-ms` is 7 minutes and wraps the `update-file` request in `rx/timeout`, failing it as a transient `:network` error so the retry path resends instead of blocking the queue forever. It is deliberately longer than `saving-stall-timeout-ms` (5 min) so the stalled report fires before the retry. +- `sync-unload-guard` keeps `:persistence` in `st/ongoing-tasks` while edits are buffered or queued, which is what holds the browser leave-page warning on. It syncs on `::append-commit`, `::discard-commit` and `::release-failed-file` (every queue change). It is the one effect that must stay in `store.cljs`'s `omitset`, and `store.cljs`'s `ongoing-tasks` watcher is guarded by `exists? js/window` because tests run without a window. - Undo transactions can stay open only temporarily; timed-out pending transactions are force-committed after about 20s. Undo entries are capped at 50. - Undo/redo are ignored while a normal editor/drawing interaction is active, except grid-layout edition handles undo through this path. - After local commits and when render-wasm is active, text shapes get derived `:position-data` recomputed in a separate commit tagged `#{:position-data}`; that tag is excluded from the position-data watcher to avoid loops. diff --git a/.serena/memories/prod-infra/grafana.md b/.serena/memories/prod-infra/grafana.md index fbc40ec6e5..10af277db2 100644 --- a/.serena/memories/prod-infra/grafana.md +++ b/.serena/memories/prod-infra/grafana.md @@ -6,15 +6,15 @@ Keep this memory free of internal names: no host names, no deployed versions, no ## Datasources -- Metrics — **Prometheus**, uid `xkRmyFBGk`. PromQL. Metric families: `penpot_*` (RPC, task timings, CLIMIT, HTTP, websocket, storage), `hikaricp_*`, `jvm_*` and `process_*` on the backend (`job="app"`); `pg_*` from the Postgres exporter (`job="postgres"`); `redis_*` from the Redis exporter (`job="redis"`); `node_*` and `container_*` from node-exporter and cAdvisor. Backend histograms are in **milliseconds**. -- Logs — **Loki**, uid `h3CGKhEMk`. LogQL, and filter by the structured metadata `logger`/`level` (`{job="penpot"} | logger="..." | level="ERROR"`). Signals that only exist as logs: the profile behind a CLIMIT queue (`logger="app.rpc.climit"`, whose `id` carries the key). CLIMIT rejections have their own counters (`penpot_rpc_climit_rejected_total`, by `name` and `reason`), and their `hint="rejected"` line goes out at WARN only when something stays queued, so counting the lines is not counting the rejections. +- Metrics — **Prometheus**, uid `xkRmyFBGk`. PromQL. Metric families: `penpot_*` (RPC, task timings, CLIMIT, HTTP, websocket, storage, and the application log events of the backend), `hikaricp_*`, `jvm_*` and `process_*` on the backend (`job="app"`); `pg_*` from the Postgres exporter (`job="postgres"`); `redis_*` from the Redis exporter (`job="redis"`); `node_*` and `container_*` from node-exporter and cAdvisor. Backend histograms are in **milliseconds**. +- Logs — **Loki**, uid `h3CGKhEMk`. LogQL, and filter by the structured metadata `logger`/`level` (`{job="penpot"} | logger="..." | level="ERROR"`). Two things to remember when counting from it: the CLIMIT rejections have their own counters (`penpot_rpc_climit_rejected_total`, by `name` and `reason`) and their `hint="rejected"` line goes out at WARN only when something is left queued behind the call (a lone caller that times out is written at TRACE), so counting lines is not counting rejections -- the same goes for `hint="enqueued"` and `hint="acquired"`; and the only signal that exists nowhere else is the profile behind a queue (`logger="app.rpc.climit"`, whose `id` carries the key), chased there on demand. - SQL — **Postgres**, uid `y8ea9tfMk`. Panels need `rawQuery: true`, `editorMode: "code"` and `format: "table"`. The `task` table is the only truth about the queue depth. It is a **read-only replica**: it rejects any write, `CREATE TEMP TABLE` included, so queries through it must be plain `SELECT`s. It also cuts long-running queries (the MCP path has a short timeout), so bound them (e.g. a list of ids); an empty answer there is not proof that there is no data. - **Postgres (telemetry)**, uid `9YAUuM_Gz`. - `$__rate_interval` and `$__interval` are substituted by Grafana; use them instead of fixed windows in graphs. ## Main dashboard -- `Production status`, uid `penpot-production-status`. Sections: general state and deploy, RPC and CLIMIT, PostgreSQL and pool, Redis/msgbus, tasks, JVM/process. Variables `$env` and `$host` (custom), no auto refresh. +- `Production status`, uid `penpot-production-status`. Sections: general state and deploy, `RPC y CLIMIT`, PostgreSQL and pool, Redis/msgbus, tasks, JVM/process. Variables `$env` and `$host` (custom), no auto refresh. Every section opens collapsed except `RPC y CLIMIT`, which is the one read on a normal day. - Per-host investigation dashboards are separate, and the alert rules deeplink into them. ## Conventions @@ -26,6 +26,20 @@ Keep this memory free of internal names: no host names, no deployed versions, no - Cron "did it run" comes from the task metrics, not from the cron registry table (it keeps rows from old versions). - Dashboard thresholds must match the alert rule thresholds. +## Editing dashboards through the API + +- Saving through the API normalizes the dashboard: submit the native form, rows with `panels: []` and their panels top-level with contiguous `y`. A row that has to start collapsed (or open) is sent holding its children in `panels` with `collapsed` set, and the top-level `y` are recomputed by hand: one unit per collapsed row, with the expanded block kept contiguous. Grafana accepts it and returns the same shape. +- A `remove` operation on a panel leaves its `gridPos` hole behind: Grafana does not refill the layout (the UI behaves the same), so a neighbouring panel has to be moved by hand when the gap matters. +- Check `list_dashboard_versions` before writing and prefer `operations` patches: the user edits dashboards by hand and a full-JSON overwrite silently reverts that work. + +## Reading the API by hand + +- The MCP authenticates with a service account token (`GRAFANA_SERVICE_ACCOUNT_TOKEN`, set in the environment of the MCP server), and a service account has no user behind it: a 403 is never the token being bad. On `/api/user` and `/api/user/orgs` it is normal; on a single dashboard (`Access denied to this dashboard`, which is not a 404) it means that identity cannot see it, a permission on that dashboard or its folder. Restarting the MCP or rotating the token changes nothing, and `X-Grafana-Org-Id` does not help when the identity belongs to another organization. +- Listing is reliable, filtering is not: a dashboard search with a text query answers empty in this instance (through the tool and through `/api/search?query=...` alike), and `list_prometheus_metric_names` also lists names that survive only in the retained series of a metric that stopped being published. Use the listing without a query, and `count()` to tell whether a metric is alive today. +- The MCP Prometheus tool can answer empty for every query, `up` included, while the datasource itself is fine: run PromQL by hand with `grafana_api_request` against `/api/datasources/uid//resources/api/v1/query?query=...`. +- `curl` and `python` from the shell cannot reach the Grafana host of the deployment: Cloudflare blocks them (`error code: 1010`, which looks like a 403 and has nothing to do with Grafana). Probe the API through the MCP (`grafana_api_request`), which is what the deployment lets through. +- A Loki query by hand goes through `GET /api/datasources/proxy/uid//loki/api/v1/query_range?query=&start=&end=&step=` (the range comes back as `resultType: matrix`, one series per group). For Loki the `/api/datasources/uid//resources/...` route answers 500, and `/api/ds/query` is not reachable from `grafana_api_request` at all (404), even though that is the path the dashboards use. + ## Gotchas - PromQL literal dots need `\\.`; simpler to use `[.]` or a plain dot. @@ -34,9 +48,5 @@ Keep this memory free of internal names: no host names, no deployed versions, no - Prometheus: `label_replace(, "host", "$1", "instance", "([^.]+)[.][^.]+.*")` plus `legendFormat: "... {{host}}"`. - Loki metric query: `| label_format host="{{ regexReplaceAll "[.][^.]*[.][^.]*[.][^.]*$" .instance "" }}"` plus `legendFormat: "{{host}}"`. Quotes inside the template must be escaped as `\"`, or LogQL closes the string early (`not a valid duration string`). - Tables from a Prometheus raw selector carry a `__name__` column; hide it with `organize`. -- Saving through the API normalizes the dashboard (rows may be flattened, `y` recomputed): submit the native form (rows with `panels: []`, panels top-level, contiguous `y`). - Panel images cannot be rendered (no image renderer installed) and there is no browser here: verify a dashboard by running its queries with the variables substituted. - Signals that exist only as logs are counted with Loki, and the count is then cross-checked against the nginx access log for the same window and the same status code (`job="frontend"`, e.g. a `429`): two independent sources must agree before a number is trusted. -- The app writes its own logs at WARN only when something is left queued behind the call and at TRACE otherwise, so production Loki holds just the under-pressure lines: a count of `hint="enqueued"` or `hint="acquired"` is not the number of calls, and neither is a count of `hint="rejected"` lines, because a rejection that leaves the queue empty (a lone caller that times out) is written at TRACE. The rejection counters are the real number. -- The MCP Prometheus tool can answer empty for every query, `up` included, while the datasource itself is fine: an empty result from it is not proof that a panel has no data, and blaming the datasource for it is wrong. Run PromQL by hand with `grafana_api_request` against `/api/datasources/uid//resources/api/v1/query?query=...` (or `POST /api/ds/query`, the path dashboards use), and check when a series is expected but the tool says nothing. -- **Check `list_dashboard_versions` before writing.** The user edits dashboards by hand and a full-JSON overwrite silently reverts that work; prefer `operations` patches per panel. diff --git a/backend/src/app/binfile/common.clj b/backend/src/app/binfile/common.clj index 42085f7647..f9cb23cdef 100644 --- a/backend/src/app/binfile/common.clj +++ b/backend/src/app/binfile/common.clj @@ -13,6 +13,7 @@ [app.common.features :as cfeat] [app.common.files.helpers :as cfh] [app.common.files.migrations :as fmg] + [app.common.files.tokens :as cfo] [app.common.files.validate :as fval] [app.common.logging :as l] [app.common.media :as cm] @@ -678,15 +679,34 @@ colors colors)) +(defn absorb-unlinked-tokens-source + "Make the file its own tokens source when its tokens source is not one + of the given library-ids, as happens when the library was deleted. The + tokens lib of the source is copied while the library still exists, even + if deleted; otherwise the tokens source and status are removed." + [cfg data library-ids] + (let [source-id (cfo/get-tokens-source data)] + (if (and (some? source-id) + (not (contains? library-ids source-id))) + (if-let [library (get-file cfg source-id + :include-deleted? true + :throw-if-not-exists? false)] + (cfo/absorb-tokens-lib data (:data library)) + (dissoc data :tokens-source :tokens-status)) + data))) + (defn embed-assets [cfg data file-id] - (let [library-ids (get-libraries cfg [file-id])] - (reduce (fn [data library-id] - (if-let [library (get-file cfg library-id :include-deleted? true)] - (ctf/absorb-assets data (:data library)) - data)) - data - library-ids))) + (let [library-ids (get-libraries cfg [file-id]) + data (reduce (fn [data library-id] + (if-let [library (get-file cfg library-id :include-deleted? true)] + (cond-> (ctf/absorb-assets data (:data library)) + (= library-id (cfo/get-tokens-source data)) + (cfo/absorb-tokens-lib (:data library))) + data)) + data + library-ids)] + (absorb-unlinked-tokens-source cfg data library-ids))) (def ^:const import-transaction-timeout-ms "Ceiling for binfile import transactions (20 minutes). Interpolated diff --git a/backend/src/app/binfile/jobs.clj b/backend/src/app/binfile/jobs.clj index c947ae858a..950d157220 100644 --- a/backend/src/app/binfile/jobs.clj +++ b/backend/src/app/binfile/jobs.clj @@ -161,7 +161,7 @@ :input input})] (with-progress cfg context #(let [result (case (int version) - 1 {:file-ids (vec (bf.v1/import-files! cfg))} + 1 (bf.v1/import-files! cfg) 3 (bf.v3/import-files! cfg) (ex/raise :type :validation :code :unsupported-version diff --git a/backend/src/app/binfile/v1.clj b/backend/src/app/binfile/v1.clj index d7468eaa38..83423e771a 100644 --- a/backend/src/app/binfile/v1.clj +++ b/backend/src/app/binfile/v1.clj @@ -482,9 +482,9 @@ (bfm/apply-pending-migrations! cfg) ;; Knowing that the ids of the created files are in index, - ;; just lookup them and return it as a set + ;; just lookup them and return them in the same shape as v3. (let [files (-> bfc/*state* deref :files)] - (into #{} (keep #(get-in @bfc/*state* [:index %])) files)))))) + {:file-ids (into [] (keep #(get-in @bfc/*state* [:index %])) files)}))))) (defmethod read-import :v1 [options] diff --git a/backend/src/app/binfile/v3.clj b/backend/src/app/binfile/v3.clj index 338a4a8a6a..3ddcc0c1df 100644 --- a/backend/src/app/binfile/v3.clj +++ b/backend/src/app/binfile/v3.clj @@ -285,6 +285,10 @@ embed? (update :data #(bfc/embed-assets cfg % file-id)) + (not (or detach? embed?)) + (update :data #(bfc/absorb-unlinked-tokens-source + cfg % (bfc/get-libraries cfg [file-id]))) + :always (bfc/clean-file-features)))))) @@ -843,7 +847,7 @@ {}) (not-empty)))) -(defn- read-file-tokens-lib +(defn- read-file-tokens-lib* [{:keys [::bfc/input ::entries-index] :as cfg} file-id] (when-let [{:keys [entry]} (first (get-in entries-index [:tokens (str file-id)]))] (tap-progress cfg {:section :tokens-lib :file-id file-id}) @@ -851,6 +855,19 @@ (decode-tokens-lib) (validate-tokens-lib)))) +(defn- read-file-tokens-lib + "Read the tokens lib of a file once per import. Decoding generates new + set ids, so the tokens lib of a library must be the same instance when + it is imported and when it resolves the tokens status of a file that + uses it as tokens source." + [cfg file-id] + (let [cache (:tokens-libs @bfc/*state*)] + (if (contains? cache file-id) + (get cache file-id) + (let [tokens-lib (read-file-tokens-lib* cfg file-id)] + (vswap! bfc/*state* assoc-in [:tokens-libs file-id] tokens-lib) + tokens-lib)))) + (defn- read-file-tokens-status [{:keys [::bfc/input ::entries-index] :as cfg} file-id tokens-lib] (when-let [{:keys [entry]} (first (get-in entries-index [:tokens-status (str file-id)]))] @@ -1369,7 +1386,8 @@ format. There are some options available for customize the output: `::bfc/export-type`: determines how linked libraries are handled. - Valid values: `:include-libraries` (include linked libraries), + Valid values: + `:include-libraries` (include linked libraries), `:merge-libraries` (embed library assets in the file), `:detach-libraries` (treat assets as basic objects), `:link-later` (preserve component metadata for relinking on import)." diff --git a/backend/src/app/loggers/audit.clj b/backend/src/app/loggers/audit.clj index 7a7d95511a..7c77a710d7 100644 --- a/backend/src/app/loggers/audit.clj +++ b/backend/src/app/loggers/audit.clj @@ -235,7 +235,9 @@ (defn- append-audit-entry [cfg params] (let [params (-> params - (assoc :id (uuid/next)) + ;; Keep a pre-assigned id (process-event assigns one so + ;; the webhook task can reference the stored row). + (update :id #(or % (uuid/next))) (update :props db/tjson) (update :context db/tjson) (update :ip-addr db/inet)) @@ -254,67 +256,75 @@ (defn- process-event [cfg event] - (when (contains? cf/flags :audit-log-logger) - (l/log! ::l/logger "app.audit" - ::l/level :info - :profile-id (str (:profile-id event)) - :ip-addr (str (:ip-addr event)) - :type (:type event) - :name (:name event) - :props (json/encode (:props event) :key-fn json/write-camel-key) - :context (json/encode (:context event) :key-fn json/write-camel-key))) + ;; Assign the row id up-front so every sink below shares it: the + ;; stored row and the `:process-webhook-event` task params. The + ;; webhook trigger row links back to its origin event through props + ;; `:event-id`, which used to land nil because the id was generated + ;; inside `append-audit-entry` without reaching the worker params. + (let [event (assoc event :id (or (:id event) (uuid/next)))] + (when (contains? cf/flags :audit-log-logger) + (l/log! ::l/logger "app.audit" + ::l/level :info + :profile-id (str (:profile-id event)) + :ip-addr (str (:ip-addr event)) + :type (:type event) + :name (:name event) + :props (json/encode (:props event) :key-fn json/write-camel-key) + :context (json/encode (:context event) :key-fn json/write-camel-key))) - (when (contains? cf/flags :audit-log) - (append-audit-entry cfg event)) + (when (contains? cf/flags :audit-log) + (append-audit-entry cfg event)) - (when (contains? cf/flags :telemetry) - ;; NOTE: when both audit-log and telemetry are enabled, events are stored - ;; twice: once with full details (above) and once stripped of props and - ;; ip-addr, tagged with source="telemetry" so the telemetry task can - ;; collect and ship them. The profile-id is preserved (UUIDs are already - ;; anonymous random identifiers). Only a safe subset of context fields - ;; is kept: initiator, version, client-version and client-user-agent. - ;; Timestamps are truncated to day precision to avoid leaking exact event - ;; timing. - (let [event (-> event - (filter-telemetry-props) - (filter-telemetry-context) - (update :created-at ct/truncate :days) - (update :tracked-at ct/truncate :days) - (assoc :source "telemetry:backend") - (assoc :ip-addr "0.0.0.0"))] - (append-audit-entry cfg event))) + (when (contains? cf/flags :telemetry) + ;; NOTE: when both audit-log and telemetry are enabled, events are stored + ;; twice: once with full details (above) and once stripped of props and + ;; ip-addr, tagged with source="telemetry" so the telemetry task can + ;; collect and ship them. The profile-id is preserved (UUIDs are already + ;; anonymous random identifiers). Only a safe subset of context fields + ;; is kept: initiator, version, client-version and client-user-agent. + ;; Timestamps are truncated to day precision to avoid leaking exact event + ;; timing. The shadow row gets its own id: reusing the full row id + ;; would collide on the primary key. + (let [event (-> event + (assoc :id (uuid/next)) + (filter-telemetry-props) + (filter-telemetry-context) + (update :created-at ct/truncate :days) + (update :tracked-at ct/truncate :days) + (assoc :source "telemetry:backend") + (assoc :ip-addr "0.0.0.0"))] + (append-audit-entry cfg event))) - (when (and (contains? cf/flags :webhooks) - (::webhooks/event? event)) - (let [batch-key (::webhooks/batch-key event) - batch-timeout (::webhooks/batch-timeout event) - label (dm/str "rpc:" (:name event)) - label (cond - (ifn? batch-key) (dm/str label ":" (batch-key (::rpc/params event))) - (string? batch-key) (dm/str label ":" batch-key) - :else label) - dedupe? (boolean (and batch-key batch-timeout))] + (when (and (contains? cf/flags :webhooks) + (::webhooks/event? event)) + (let [batch-key (::webhooks/batch-key event) + batch-timeout (::webhooks/batch-timeout event) + label (dm/str "rpc:" (:name event)) + label (cond + (ifn? batch-key) (dm/str label ":" (batch-key (::rpc/params event))) + (string? batch-key) (dm/str label ":" batch-key) + :else label) + dedupe? (boolean (and batch-key batch-timeout))] - (jobs/submit cfg - {::jobs/name :process-webhook-event - ::jobs/queue :webhooks - ::jobs/max-retries 0 - ::jobs/delay (or batch-timeout 0) - ::jobs/dedupe dedupe? - ::jobs/label label - ;; The event travels as an opaque transit blob: - ;; transit preserves the instant/UUID/set types - ;; that plain JSON props cannot carry. - ::jobs/params {:event-blob - (t/encode-str - (-> event - (d/without-qualified) - (dissoc :source) - (dissoc :context) - (dissoc :ip-addr) - (dissoc :type)))}}))) - event) + (jobs/submit cfg + {::jobs/name :process-webhook-event + ::jobs/queue :webhooks + ::jobs/max-retries 0 + ::jobs/delay (or batch-timeout 0) + ::jobs/dedupe dedupe? + ::jobs/label label + ;; The event travels as an opaque transit blob: + ;; transit preserves the instant/UUID/set types + ;; that plain JSON props cannot carry. + ::jobs/params {:event-blob + (t/encode-str + (-> event + (d/without-qualified) + (dissoc :source) + (dissoc :context) + (dissoc :ip-addr) + (dissoc :type)))}}))) + event)) (defn submit* "A public API, lower-level than submit, assumes all required fields are filled" @@ -402,6 +412,47 @@ coerced)) +(def ^:private sql:team-id-by-project + "SELECT id FROM team + WHERE id = (SELECT team_id FROM project WHERE id = ?)") + +(def ^:private sql:team-id-by-file + "SELECT t.id FROM team AS t + JOIN project AS p ON (p.team_id = t.id) + JOIN file AS f ON (f.project_id = p.id) + WHERE f.id = ?") + +(def ^:private read-event-names + "Read commands excluded from team-id resolution on top of the `get-*` + prefix: their events carry no consumer state, so the lookup would only + tax the read path." + #{"has-file-libraries"}) + +(defn- read-event? + [event-name] + (or (str/starts-with? event-name "get-") + (contains? read-event-names event-name))) + +(defn- enrich-team-id + "Adds the team id to the event props when it is missing and can be resolved + locally, so consumers do not have to join production tables. Only events that + already carry a project or file id are considered: read commands put the + entity id under `:id`, and resolving those would add a query to every read." + [cfg event-name props] + (if (or (:team-id props) + (read-event? event-name) + (and (nil? (:project-id props)) + (nil? (:file-id props)))) + props + (let [project-id (:project-id props) + file-id (:file-id props) + team-id (or (when (some? project-id) + (:id (db/exec-one! cfg [sql:team-id-by-project project-id]))) + (when (some? file-id) + (:id (db/exec-one! cfg [sql:team-id-by-file file-id]))))] + (cond-> props + (some? team-id) (assoc :team-id team-id))))) + (defn prepare-rpc-event [cfg mdata params result] (let [resultm (meta result) @@ -416,25 +467,32 @@ (::rpc/profile-id params) uuid/zero) + module (get cfg ::rpc/module) + event-name (or (::name resultm) + (let [sname (::sv/name mdata)] + (if (not= module "main") + (str module "-" sname) + sname))) + props (-> (or (::replace-props resultm) (merge params (::props resultm))) (clean-props)) + ;; Resolve the team locally when the event carries a project or file + ;; id, so the consumers do not need to join production tables. + ;; Reads are excluded: their events carry no consumer state. + props (enrich-team-id cfg event-name props) + context (-> (::context resultm) (merge (prepare-context-from-request request)) (assoc :request-id (::rpc/request-id params)) (d/without-nils)) - ip-addr (inet/parse-request request) - module (get cfg ::rpc/module)] + ip-addr (inet/parse-request request)] {:type (or (::type resultm) (::rpc/type cfg)) - :name (or (::name resultm) - (let [sname (::sv/name mdata)] - (if (not= module "main") - (str module "-" sname) - sname))) + :name event-name :profile-id profile-id :ip-addr ip-addr @@ -491,6 +549,33 @@ (d/without-nils))] (submit* cfg event))) +(defn submit-create-event + "Emits a `create-*` audit event for an entity created as a side effect of a + bulk operation (import, duplicate, clone). Those entities get no event from + their own command, so a local projection cannot key them otherwise. The + event is built by hand and does not pass through `prepare-rpc-event`, so its + props are passed as-is minus nils." + [cfg params event-name props] + (when (contains? cf/flags :audit-log) + (submit cfg + (-> (event-from-rpc-params params) + (assoc :name event-name) + (assoc :props (d/without-nils props)))))) + +(def ^:private sql:get-file-event-columns + "SELECT id, name, project_id, created_at, modified_at, revn + FROM file WHERE id = ?;") + +(defn submit-file-created-event + "Emit a `create-file` event for a file born inside a bulk operation, + with the same prop shape as the `create-file` command event. Only + the file id is needed: the light columns are read back (no data + blob), so projections that key on name, revn or timestamps work. + `extra` carries what the row lacks (the team id)." + [cfg params file-id extra] + (when-let [row (db/exec-one! cfg [sql:get-file-event-columns file-id])] + (submit-create-event cfg params "create-file" (merge row extra)))) + (defn insert "Submit audit event to the collector, intended to be used only from command line helpers because this skips all webhooks and telemetry diff --git a/backend/src/app/loggers/metrics.clj b/backend/src/app/loggers/metrics.clj new file mode 100644 index 0000000000..07c97debd1 --- /dev/null +++ b/backend/src/app/loggers/metrics.clj @@ -0,0 +1,46 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS INC Sucursal en España SL + +(ns app.loggers.metrics + "A reporter that counts the log records of this application, by logger and + level, and publishes them as a counter." + (:require + [app.common.logging :as l] + [app.metrics :as mtx] + [integrant.core :as ig])) + +(def ^:private counted-levels + "The levels worth counting: `info` is emitted often and says nothing about + the health of the application." + #{:warn :error :fatal}) + +(defmethod ig/assert-key ::reporter + [_ params] + (assert (mtx/metrics? (::mtx/metrics params)) "expect valid metrics instance")) + +(defmethod ig/init-key ::reporter + [_ {::mtx/keys [metrics]}] + (add-watch l/log-record ::reporter + (fn [_ _ _ record] + ;; This runs inside the logging path: the watcher fires from + ;; the `swap!` of the log record, so an exception here would + ;; reach whoever was logging on the synchronous path, and on + ;; the async one it would die in the executor's uncaught + ;; handler, taking that log line out of Loki. Swallowed on + ;; purpose either way. + (try + (let [level (::l/level record)] + (when (contains? counted-levels level) + (mtx/run! metrics + :id :log-events + :inc 1 + :labels (into-array String [(::l/logger record) (name level)])))) + (catch Throwable _ nil)))) + {::mtx/metrics metrics}) + +(defmethod ig/halt-key! ::reporter + [_ _] + (remove-watch l/log-record ::reporter)) diff --git a/backend/src/app/main.clj b/backend/src/app/main.clj index 2c34fc8c94..0565ce3448 100644 --- a/backend/src/app/main.clj +++ b/backend/src/app/main.clj @@ -54,7 +54,13 @@ (:gen-class)) (def default-metrics - {:update-file-changes + {:log-events + {::mdef/name "penpot_log_events_total" + ::mdef/help "Log records at WARN, ERROR and FATAL emitted by the application, by logger and level." + ::mdef/labels ["logger" "level"] + ::mdef/type :counter} + + :update-file-changes {::mdef/name "penpot_rpc_update_file_changes_total" ::mdef/help "A total number of changes submitted to update-file." ::mdef/type :counter} @@ -745,6 +751,9 @@ :app.loggers.database/reporter {::db/pool (ig/ref ::db/pool)} + :app.loggers.metrics/reporter + {::mtx/metrics (ig/ref ::mtx/metrics)} + ::sto/storage {::db/pool (ig/ref ::db/pool) ::mtx/metrics (ig/ref ::mtx/metrics) diff --git a/backend/src/app/rpc.clj b/backend/src/app/rpc.clj index d1b5247b95..8d48a8c984 100644 --- a/backend/src/app/rpc.clj +++ b/backend/src/app/rpc.clj @@ -427,7 +427,8 @@ [cfg] (let [cfg (assoc cfg ::module "management" ::type "command" ::metrics-id :rpc-management-timing) mods (cond->> (list 'app.rpc.management.exporter - 'app.rpc.management.jobs) + 'app.rpc.management.jobs + 'app.rpc.management.files) (contains? cf/flags :admin-console) (cons 'app.rpc.management.nitrate))] diff --git a/backend/src/app/rpc/commands/binfile.clj b/backend/src/app/rpc/commands/binfile.clj index 67163c1568..97e6c8fd44 100644 --- a/backend/src/app/rpc/commands/binfile.clj +++ b/backend/src/app/rpc/commands/binfile.clj @@ -16,7 +16,7 @@ [app.config :as cf] [app.db :as db] [app.http.sse :as sse] - [app.loggers.audit :as-alias audit] + [app.loggers.audit :as audit] [app.loggers.webhooks :as-alias webhooks] [app.media.validation :as media.v] [app.rpc :as-alias rpc] @@ -83,7 +83,7 @@ ;; --- Command: import-binfile (defn- import-binfile - [{:keys [::db/pool] :as cfg} {:keys [profile-id project-id version name file upload-id]}] + [{:keys [::db/pool] :as cfg} {:keys [profile-id project-id version name file upload-id] :as params}] (let [team (teams/get-team pool :profile-id profile-id @@ -116,6 +116,14 @@ {:id project-id} {::db/return-keys false}) + ;; The imported files get no event from their own command, so emit one per + ;; file inside a single transaction. + (db/tx-run! cfg + (fn [cfg] + (doseq [file-id (:file-ids result)] + (audit/submit-file-created-event cfg params file-id + {:team-id (:id team)})))) + result)) (def ^:private schema:import-binfile diff --git a/backend/src/app/rpc/commands/comments.clj b/backend/src/app/rpc/commands/comments.clj index 459ebbeaa0..fbd460cbb2 100644 --- a/backend/src/app/rpc/commands/comments.clj +++ b/backend/src/app/rpc/commands/comments.clj @@ -489,7 +489,7 @@ (let [params (assoc params ::file file) thread (db/tx-run! cfg create-comment-thread params)] - (vary-meta thread assoc ::audit/props thread)))) + (vary-meta thread assoc ::audit/props (assoc thread :team-id team-id))))) (defn- create-comment-thread [{:keys [::db/conn] :as cfg} @@ -653,7 +653,7 @@ (send-comment-emails! cfg profile comment thread file) - (vary-meta comment assoc ::audit/props comment)))) + (vary-meta comment assoc ::audit/props (assoc comment :team-id team-id))))) ;; --- COMMAND: Update Comment diff --git a/backend/src/app/rpc/commands/files.clj b/backend/src/app/rpc/commands/files.clj index 5d236a8e04..88547d25ce 100644 --- a/backend/src/app/rpc/commands/files.clj +++ b/backend/src/app/rpc/commands/files.clj @@ -988,7 +988,11 @@ (let [file (-> (bfc/get-file cfg file-id :include-deleted? true :lock-for-update? true) - (update :data ctf/absorb-assets ldata))] + (update :data ctf/absorb-assets ldata) + (update :data (fn [data] + (cond-> data + (cfo/effective-tokens-source? data (:id ldata)) + (cfo/absorb-tokens-lib ldata)))))] (l/trc :hint "library absorbed" :library-id (str (:id ldata)) @@ -1001,16 +1005,21 @@ :modified-at (ct/now) :has-media-trimmed false})))) +(defn- get-referenced-files + [cfg library-id] + (->> (db/exec! cfg [sql:get-referenced-files library-id]) + (into [] bfc/xf-map-id))) + (defn- absorb-library* - "Find all files using a shared library, and absorb all library assets - into the file local libraries" - [cfg {:keys [id data] :as library}] + "Absorb all library assets, and its tokens lib if it is their tokens + source, into the local libraries of the files that use it. The files + are the given file-ids, or the ones linked to the library if nil." + [cfg {:keys [id data] :as library} file-ids] (assert (db/connection-map? cfg) "expected cfg with valid connection") - (let [ids (->> (db/exec! cfg [sql:get-referenced-files id]) - (sequence bfc/xf-map-id))] + (let [ids (or file-ids (get-referenced-files cfg id))] (l/trc :hint "absorbing library" :library-id (str id) @@ -1020,7 +1029,9 @@ library)) (defn absorb-library - [{:keys [::db/conn] :as cfg} id] + "Absorb the library into the files that use it. The files are the given + file-ids, or the ones linked to the library if omitted." + [{:keys [::db/conn] :as cfg} id & {:keys [file-ids]}] (let [file (-> (bfc/get-file cfg id :realize? true :lock-for-update? true @@ -1036,7 +1047,7 @@ (-> (cfeat/get-team-enabled-features cf/flags team) (cfeat/check-file-features! (:features file))) - (absorb-library* cfg file))) + (absorb-library* cfg file file-ids))) (defn- set-file-shared [{:keys [::db/conn] :as cfg} {:keys [profile-id id] :as params}] @@ -1109,7 +1120,10 @@ {:deleted-at (ct/in-future delay)} {:id file-id} {::db/return-keys [:id :name :is-shared :deleted-at - :project-id :created-at :modified-at]})] + :project-id :created-at :modified-at]}) + ;; Files using the library, captured before their relations are removed + file-ids (when (:is-shared file) + (get-referenced-files conn file-id))] ;; Remove all possible relations for that file (db/delete! conn :file-library-rel @@ -1117,9 +1131,11 @@ (jobs/submit cfg {::jobs/name :delete-object - ::jobs/params {:object :file - :deleted-at (:deleted-at file) - :id file-id}}) + ::jobs/params (cond-> {:object :file + :deleted-at (:deleted-at file) + :id file-id} + (some? file-ids) + (assoc :file-ids file-ids))}) file)) (def ^:private @@ -1147,7 +1163,8 @@ :project-id (:project-id file) :name (:name file) :created-at (:created-at file) - :modified-at (:modified-at file)}}))) + :modified-at (:modified-at file) + :deleted-at (:deleted-at file)}}))) (sv/defmethod ::delete-file {::doc/added "1.17" diff --git a/backend/src/app/rpc/commands/files_create.clj b/backend/src/app/rpc/commands/files_create.clj index d1b14283a5..23959989af 100644 --- a/backend/src/app/rpc/commands/files_create.clj +++ b/backend/src/app/rpc/commands/files_create.clj @@ -137,5 +137,11 @@ {:id team-id} {::db/return-keys false})))) - (-> (create-file cfg params) - (vary-meta assoc ::audit/props {:team-id team-id})))) + (let [file (create-file cfg params)] + (vary-meta file assoc ::audit/props {:id (:id file) + :project-id project-id + :team-id team-id + :name (:name file) + :created-at (:created-at file) + :modified-at (:modified-at file) + :revn (:revn file)})))) diff --git a/backend/src/app/rpc/commands/files_share.clj b/backend/src/app/rpc/commands/files_share.clj index 6643554a06..66bb1447ad 100644 --- a/backend/src/app/rpc/commands/files_share.clj +++ b/backend/src/app/rpc/commands/files_share.clj @@ -12,9 +12,11 @@ [app.common.schema :as sm] [app.common.uuid :as uuid] [app.db :as db] + [app.loggers.audit :as-alias audit] [app.rpc :as-alias rpc] [app.rpc.commands.files :as files] [app.rpc.doc :as-alias doc] + [app.rpc.helpers :as rph] [app.util.services :as sv])) ;; --- MUTATION: Create Share Link @@ -39,7 +41,11 @@ ::db/transaction true} [{:keys [::db/conn]} {:keys [::rpc/profile-id file-id] :as params}] (files/check-edition-permissions! conn profile-id file-id) - (create-share-link conn (assoc params :profile-id profile-id))) + (let [slink (create-share-link conn (assoc params :profile-id profile-id))] + ;; The consumers key share scope by file, so the event carries + ;; both ids; `:file-id` is already in params, `:share-id` is new. + (rph/with-meta slink + {::audit/props {:share-id (:id slink)}}))) (defn create-share-link [conn {:keys [profile-id file-id pages who-comment who-inspect]}] @@ -80,4 +86,8 @@ :hint "You can only delete share-links you created"))) (db/delete! conn :share-link {:id id}) - nil)) + ;; The delete params carry only the link id, so resolve the file + ;; from the row fetched above before it is gone. + (rph/with-meta (rph/wrap) + {::audit/props {:file-id (:file-id slink) + :share-id id}}))) diff --git a/backend/src/app/rpc/commands/files_update.clj b/backend/src/app/rpc/commands/files_update.clj index 9d996c6dd4..373a06717f 100644 --- a/backend/src/app/rpc/commands/files_update.clj +++ b/backend/src/app/rpc/commands/files_update.clj @@ -283,11 +283,18 @@ (with-meta {:revn revn :lagged (get-lagged-changes conn params)} {::audit/replace-props - {:id (:id file) - :name (:name file) - :features (:features file) - :project-id (:project-id file) - :team-id (:team-id file)}})))) + {:id (:id file) + :name (:name file) + :features (:features file) + :project-id (:project-id file) + :team-id (:team-id file) + ;; NOTE: `file` carries the post-update revn here (`revn` + ;; above is the pre-update one kept for the response), and + ;; `modified-at` mirrors `persist-file!`: same `::timestamp` + ;; fallback, so both agree to the millisecond when the + ;; request timestamp exists. + :revn (:revn file) + :modified-at (or timestamp (ct/now))}})))) (defn get-file "Get not-decoded file, only decodes the features set." diff --git a/backend/src/app/rpc/commands/management.clj b/backend/src/app/rpc/commands/management.clj index 62d5a88330..b0ac390b57 100644 --- a/backend/src/app/rpc/commands/management.clj +++ b/backend/src/app/rpc/commands/management.clj @@ -142,12 +142,29 @@ (when (uuid? profile-id) (teams/create-project-role conn profile-id (:id project) :owner)) + ;; The new entities are created by internal calls, so they carry no + ;; event of their own; emit one per entity for a local projection. + (audit/submit-create-event cfg params "create-project" + {:id (:id project) + :team-id (:team-id project) + :name (:name project) + :created-at (:created-at project) + :modified-at (:modified-at project)}) + (doseq [file-id files] (let [params (-> params (dissoc :name) (assoc :file-id file-id) - (assoc :reset-shared-flag false))] - (duplicate-file cfg params))) + (assoc :reset-shared-flag false)) + file (duplicate-file cfg params)] + (audit/submit-create-event cfg params "create-file" + {:id (:id file) + :project-id (:project-id file) + :team-id (:team-id project) + :name (:name file) + :created-at (:created-at file) + :modified-at (:modified-at file) + :revn (:revn file)}))) project)))) @@ -211,7 +228,7 @@ (update :team-id bfc/lookup-index) (assoc :created-at timestamp) (assoc :modified-at timestamp))] - (teams/add-profile-to-team! cfg params {::db/return-keys false}))) + (teams/add-profile-to-team! cfg params))) ;; Duplicate team fonts (doseq [font fonts] @@ -444,14 +461,12 @@ {:id project-id} {::db/return-keys false}) - (let [props (audit/clean-props params)] - (doseq [file-id result] - (let [props (assoc props :id file-id) - event (-> (audit/event-from-rpc-params params) - (assoc :profile-id profile-id) - (assoc :name "create-file") - (assoc :props props))] - (audit/submit cfg event)))))) + (doseq [file-id (:file-ids result)] + (audit/submit-file-created-event + cfg + (assoc params :profile-id profile-id) + file-id + {:team-id (:id team)})))) result)) diff --git a/backend/src/app/rpc/commands/media.clj b/backend/src/app/rpc/commands/media.clj index e914ff3302..1c9e07ed01 100644 --- a/backend/src/app/rpc/commands/media.clj +++ b/backend/src/app/rpc/commands/media.clj @@ -68,27 +68,28 @@ (quotes/check! cfg {::quotes/id ::quotes/media-storage-bytes-per-team ::quotes/profile-id profile-id ::quotes/team-id team-id - ::quotes/incr (:size content)})) + ::quotes/incr (:size content)}) - (db/run! cfg (fn [{:keys [::db/conn] :as cfg}] - ;; We get the minimal file for proper checking if - ;; file is not already deleted - (let [_ (files/get-minimal-file conn file-id) - mobj (create-file-media-object cfg params)] + (db/run! cfg (fn [{:keys [::db/conn] :as cfg}] + ;; We get the minimal file for proper checking if + ;; file is not already deleted + (let [_ (files/get-minimal-file conn file-id) + mobj (create-file-media-object cfg params)] - (db/update! conn :file - {:modified-at (ct/now) - :has-media-trimmed false} - {:id file-id} - {::db/return-keys false}) + (db/update! conn :file + {:modified-at (ct/now) + :has-media-trimmed false} + {:id file-id} + {::db/return-keys false}) - (with-meta mobj - {::audit/replace-props - {:name (:name params) - :file-id file-id - :is-local (:is-local params) - :size (:size content) - :mtype (:mtype content)}}))))) + (with-meta mobj + {::audit/replace-props + {:name (:name params) + :file-id file-id + :team-id team-id + :is-local (:is-local params) + :size (:size content) + :mtype (:mtype content)}})))))) (defn- big-enough-for-thumbnail? "Checks if the provided image info is big enough for diff --git a/backend/src/app/rpc/commands/profile.clj b/backend/src/app/rpc/commands/profile.clj index f388d0dfc0..c6e38d510e 100644 --- a/backend/src/app/rpc/commands/profile.clj +++ b/backend/src/app/rpc/commands/profile.clj @@ -540,7 +540,8 @@ {::doc/added "1.0" ::db/transaction true} [{:keys [::db/conn] :as cfg} {:keys [::rpc/profile-id] :as params}] - (let [teams (get-owned-teams conn profile-id) + (let [profile (db/get* conn :profile {:id profile-id} {::db/remove-deleted false}) + teams (get-owned-teams conn profile-id) deleted-at (ct/now)] ;; If we found owned teams with participants, we don't allow @@ -581,6 +582,12 @@ (session/invalidate-all cfg profile-id) (-> (rph/wrap nil) + (rph/with-meta {::audit/props {:id (:id profile) + :email (:email profile) + :fullname (:fullname profile) + :created-at (:created-at profile) + :modified-at (:modified-at profile) + :deleted-at deleted-at}}) (rph/with-transform (session/delete-fn cfg))))) (def sql:get-subscription-editors diff --git a/backend/src/app/rpc/commands/projects.clj b/backend/src/app/rpc/commands/projects.clj index db366020a2..18e5c282f0 100644 --- a/backend/src/app/rpc/commands/projects.clj +++ b/backend/src/app/rpc/commands/projects.clj @@ -222,8 +222,9 @@ ::quotes/profile-id profile-id ::quotes/team-id team-id}) - (let [params (assoc params :profile-id profile-id)] - (db/tx-run! cfg create-project params))) + (let [params (assoc params :profile-id profile-id) + project (db/tx-run! cfg create-project params)] + (rph/with-meta project {::audit/props {:id (:id project)}}))) ;; --- MUTATION: Toggle Project Pin diff --git a/backend/src/app/rpc/commands/teams.clj b/backend/src/app/rpc/commands/teams.clj index ea8787ff67..1e01c21742 100644 --- a/backend/src/app/rpc/commands/teams.clj +++ b/backend/src/app/rpc/commands/teams.clj @@ -564,7 +564,9 @@ team (db/tx-run! cfg create-team params)] (with-meta team - {::audit/props {:id (:id team)}}))) + {::audit/props {:id (:id team) + :created-at (:created-at team) + :modified-at (:modified-at team)}}))) (defn create-default-organization-team @@ -619,10 +621,42 @@ :default-team-id default-team-id})) default-team-id)))))))) +(defn- params->role + "Derive the team role from membership flags, inverse of + `types.team/permissions-for-role`. `team_profile_rel` carries no role + column, so no caller can pass it in `params`: derive it here so the event + always carries one." + [{:keys [is-owner is-admin can-edit]}] + (cond is-owner :owner is-admin :admin can-edit :editor :else :viewer)) + +(defn- submit-membership-event + "Emits an audit event for a new team membership, so the event stream carries + every membership add, including the ones that only happen as a side effect of + another command (team creation, invitation accepted without email roundtrip, + duplicated team). `event-opts` carries what the row lacks: `:actor-id` (the + action owner, defaulting to the member) and `:rpc-params` (request context, + which sibling events get from `prepare-rpc-event`)." + [cfg {:keys [profile-id team-id] :as params} {:keys [actor-id rpc-params]}] + (when (contains? cf/flags :audit-log) + (audit/submit cfg + (-> (audit/event-from-rpc-params (or rpc-params {})) + (assoc :name "add-team-member" + :type "action" + :profile-id (or actor-id profile-id)) + (assoc :props (d/without-nils + {:team-id team-id + :member-id profile-id + :role (params->role params) + :is-owner (boolean (:is-owner params)) + :is-admin (boolean (:is-admin params)) + :can-edit (boolean (:can-edit params))})))))) + (defn add-profile-to-team! ([cfg params] - (add-profile-to-team! cfg params nil)) - ([{:keys [::db/conn] :as cfg} {:keys [:profile-id :team-id] :as params} options] + (add-profile-to-team! cfg params nil nil)) + ([cfg params options] + (add-profile-to-team! cfg params options nil)) + ([{:keys [::db/conn] :as cfg} {:keys [:profile-id :team-id] :as params} options event-opts] (assert (db/connection-map? cfg) "expected cfg with valid connection") (when (contains? cf/flags :admin-console) @@ -632,7 +666,13 @@ (some? (:organization-id membership)) ;; the team do belong to an organization (not (:is-member membership))) ;; the user is not a member of the organization yet (initialize-user-in-organization cfg profile-id (:organization-id membership))))) - (db/insert! conn :team-profile-rel (assoc params :id (uuid/next)) options))) + (let [options (merge options {::db/return-keys [:id]}) + result (db/insert! conn :team-profile-rel (assoc params :id (uuid/next)) options)] + ;; Only emit when a row was actually inserted: with `on-conflict-do-nothing` + ;; an existing membership returns nil and must not be reported twice. + (when (some? result) + (submit-membership-event cfg params event-opts)) + result))) (defn create-team "This is a complete team creation process, it creates the team @@ -885,7 +925,7 @@ ::jobs/params {:object :team :deleted-at (:deleted-at team) :id team-id}}) - (rph/with-meta (rph/wrap) + (rph/with-meta (rph/wrap team) {::audit/props {:organization-id organization-id :team-id team-id :team-name team-name}})))) @@ -900,7 +940,19 @@ ::sm/params schema:delete-team ::db/transaction true} [cfg {:keys [::rpc/profile-id id] :as params}] - (delete-team cfg {:team-id id :profile-id profile-id})) + (let [result (delete-team cfg {:team-id id :profile-id profile-id}) + team (rph/unwrap result)] + ;; The client keeps receiving nil; the audit event carries the deleted row. + ;; Both prop sets are kept: the organization ones (develop) and the entity + ;; identity and timestamps (staging); `with-meta` merges shallowly, so the + ;; maps are merged explicitly. + (rph/with-meta (rph/wrap) + {::audit/props (merge (::audit/props (meta result)) + {:id (:id team) + :name (:name team) + :created-at (:created-at team) + :modified-at (:modified-at team) + :deleted-at (:deleted-at team)})}))) ;; --- Mutation: Team Update Role @@ -967,7 +1019,13 @@ {::doc/added "1.17" ::sm/params schema:update-team-member-role} [cfg {:keys [::rpc/profile-id] :as params}] - (db/tx-run! cfg update-team-member-role (assoc params :profile-id profile-id))) + (db/tx-run! cfg update-team-member-role (assoc params :profile-id profile-id)) + ;; The client keeps receiving nil; when ownership moves, the audit event + ;; names the demoted member so a projection can keep a single owner. + (rph/with-meta (rph/wrap) + (if (= :owner (:role params)) + {::audit/props {:prev-owner-id profile-id}} + {}))) ;; --- Mutation: Delete Team Member diff --git a/backend/src/app/rpc/commands/teams_invitations.clj b/backend/src/app/rpc/commands/teams_invitations.clj index 6277506bda..6cabb06ec5 100644 --- a/backend/src/app/rpc/commands/teams_invitations.clj +++ b/backend/src/app/rpc/commands/teams_invitations.clj @@ -202,7 +202,10 @@ (when (contains? cf/flags :admin-console) (teams/initialize-user-in-organization cfg (:id member) (:id organization) email)) ;; Insert the invited member to the team - (teams/add-profile-to-team! cfg params {::db/on-conflict-do-nothing? true})) + (teams/add-profile-to-team! cfg params + {::db/on-conflict-do-nothing? true} + {:actor-id (:id profile) + :rpc-params params})) ;; If profile is not yet verified, mark it as verified because ;; accepting an invitation link serves as verification. @@ -318,7 +321,7 @@ :role :editor)))) (defn- add-member-to-team - [{:keys [::db/conn] :as cfg} profile team role member] + [{:keys [::db/conn] :as cfg} profile team role member event-opts] (assert (db/connection-map? cfg) "expected cfg with valid connection") @@ -340,7 +343,9 @@ ::quotes/team-id team-id}) ;; Insert the member to the team - (teams/add-profile-to-team! cfg params {::db/on-conflict-do-nothing? true}) + (teams/add-profile-to-team! cfg params + {::db/on-conflict-do-nothing? true} + event-opts) ;; Delete any request (db/delete! conn :team-access-request @@ -438,7 +443,9 @@ (filter #(contains? invitation-emails (key %))) (map (fn [[email member]] (let [role (:role (first (filter #(= (:email %) email) invitation-data)))] - (add-member-to-team cfg profile team role member)))) + (add-member-to-team cfg profile team role member + {:actor-id (:id profile) + :rpc-params params})))) (doall)) invitations)) @@ -593,7 +600,7 @@ :hint "the maximum of invitation on single request is reached" :threshold max-invitations-by-request-threshold)) - (let [props {:name name :features features} + (let [props {:id (:id team) :name name :features features} event (-> (audit/event-from-rpc-params params) (assoc :name "create-team") (assoc :props props))] @@ -609,7 +616,8 @@ (map (fn [email] (assoc params :email email))) (map (partial create-invitation cfg)))] - (vary-meta team assoc ::audit/props {:invitations (count invitations)})))) + (vary-meta team assoc ::audit/props {:id (:id team) + :invitations (count invitations)})))) ;; --- Query: get-team-invitation-token diff --git a/backend/src/app/rpc/commands/webhooks.clj b/backend/src/app/rpc/commands/webhooks.clj index 2476128e7c..f22a693fd2 100644 --- a/backend/src/app/rpc/commands/webhooks.clj +++ b/backend/src/app/rpc/commands/webhooks.clj @@ -14,10 +14,12 @@ [app.common.uuid :as uuid] [app.db :as db] [app.http.client :as http] + [app.loggers.audit :as-alias audit] [app.loggers.webhooks :as webhooks] [app.rpc :as-alias rpc] [app.rpc.commands.teams :refer [check-read-permissions!] :as t] [app.rpc.doc :as-alias doc] + [app.rpc.helpers :as rph] [app.rpc.permissions :as perms] [app.util.services :as sv] [cuerdas.core :as str])) @@ -137,7 +139,8 @@ (let [whook (-> (db/get pool :webhook {:id id}) (decode-row))] (check-webhook-edition-permissions! pool profile-id (:team-id whook)) (validate-webhook! cfg whook params) - (update-webhook! cfg whook params))) + (rph/with-meta (update-webhook! cfg whook params) + {::audit/props {:team-id (:team-id whook)}}))) (def ^:private schema:delete-webhook [:map {:title "delete-webhook"} @@ -151,7 +154,8 @@ (let [whook (-> (db/get conn :webhook {:id id}) decode-row)] (check-webhook-edition-permissions! conn profile-id (:team-id whook)) (db/delete! conn :webhook {:id id}) - nil)) + (rph/with-meta (rph/wrap) + {::audit/props {:team-id (:team-id whook)}}))) ;; --- Query: Webhooks diff --git a/backend/src/app/rpc/management/files.clj b/backend/src/app/rpc/management/files.clj new file mode 100644 index 0000000000..d947ec7549 --- /dev/null +++ b/backend/src/app/rpc/management/files.clj @@ -0,0 +1,238 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns app.rpc.management.files + "Read-only file content for the nexus file-security pipeline. + + The single method `get-file-for-analysis` returns the semantic + projection of one file (pages, frames, text, links, interactions): + small, neutral data the decision model consumes. Full file binaries + never cross this boundary. Every call leaves an audit event like any + other management call, so content reads stay traceable; the volume + scales with analyzed files, and the nexus consumer ignores the event." + (:require + [app.binfile.common :as bfc] + [app.common.data :as d] + [app.common.exceptions :as ex] + [app.common.schema :as sm] + [app.common.time :as ct] + [app.common.types.text :as txt] + [app.db :as db] + [app.rpc :as-alias rpc] + [app.rpc.doc :as doc] + [app.util.services :as sv] + [cuerdas.core :as str])) + +(def projection-version 1) + +;; Bounds for the data that crosses to nexus on every sweep. The +;; projection stays small by construction; anything past the caps sets +;; `:truncated?` instead of growing the payload. +(def ^:const max-text-chars 30000) +(def ^:const max-links 1000) +(def ^:const max-interactions 2000) + +(defn- page-shapes + "All shapes of one page with their page id attached." + [{:keys [id objects]}] + (let [page-id (str id)] + (->> (vals objects) + (filter map?) + (map #(assoc % :page-id page-id))))) + +(defn- shape-text + "Plain text of a text shape, or nil when it has none. Never throws: + corrupt content skips the shape instead of failing the file." + [shape] + (when (and (= :text (:type shape)) + (some? (:content shape))) + (let [text (ex/ignoring (txt/content->text (:content shape)))] + (when-not (str/blank? text) + text)))) + +(defn- open-url? + [{:keys [action-type url]}] + (and (= :open-url action-type) + (not (str/blank? url)))) + +(defn file->projection + "Turn decoded file data into the neutral projection the decision + model consumes. Pure: takes the `:data` map, returns plain data + with all ids as strings." + [{:keys [pages pages-index]}] + (let [shapes (mapcat page-shapes + (keep #(get pages-index %) pages)) + frames (into [] + (comp (filter #(= :frame (:type %))) + (map (fn [{:keys [id name page-id]}] + (d/without-nils + {:id (str id) + :page-id page-id + :name name})))) + shapes) + texts (into [] + (comp (map (juxt :id :page-id shape-text)) + (remove #(nil? (nth % 2))) + (map (fn [[id page-id text]] + {:shape-id (str id) + :page-id page-id + :text text}))) + shapes) + ;; Texts are capped by total characters, the unit the model + ;; bills; links and interactions by count. + texts (loop [acc [] + chars 0 + [item & rest] texts] + (cond + (nil? item) + {:items acc :truncated? false} + + (> (+ chars (count (:text item))) max-text-chars) + {:items acc :truncated? true} + + :else + (recur (conj acc item) + (+ chars (count (:text item))) + rest))) + inter (into [] + (comp (mapcat (fn [shape] + (->> (:interactions shape) + (filter map?) + (filter :action-type) + (filter :event-type) + (map #(assoc % :shape-id (:id shape) + :page-id (:page-id shape)))))) + (map (fn [{:keys [shape-id page-id event-type action-type destination url]}] + (d/without-nils + {:shape-id (str shape-id) + :page-id page-id + :event (d/name event-type) + :action (d/name action-type) + :target (or url (some-> destination str))})))) + shapes) + links (into [] + (comp (mapcat (fn [shape] + (->> (:interactions shape) + (filter map?) + (filter open-url?) + (map #(assoc % :shape-id (:id shape) + :page-id (:page-id shape)))))) + (map (fn [{:keys [shape-id page-id url]}] + {:shape-id (str shape-id) + :page-id page-id + :url url}))) + shapes)] + {:version projection-version + :pages (mapv (fn [page-id] + (let [page (get pages-index page-id)] + (d/without-nils + {:id (str page-id) + :name (:name page)}))) + pages) + :frames frames + :texts texts + :links (if (> (count links) max-links) + {:items (subvec links 0 max-links) :truncated? true} + {:items links :truncated? false}) + :interactions (if (> (count inter) max-interactions) + {:items (subvec inter 0 max-interactions) :truncated? true} + {:items inter :truncated? false})})) + +(def ^:private sql:has-share-link + "SELECT EXISTS (SELECT 1 FROM share_link WHERE file_id = ?) AS shared;") + +(def ^:private sql:get-team-id + "SELECT team_id FROM project WHERE id = ?;") + +(def ^:private + schema:get-file-for-analysis-params + [:map {:title "get-file-for-analysis-params"} + [:file-id ::sm/uuid]]) + +(def ^:private schema:projection-page + [:map + [:id :string] + [:name {:optional true} :string]]) + +(def ^:private schema:projection-frame + [:map + [:id :string] + [:page-id :string] + [:name {:optional true} :string]]) + +(def ^:private schema:projection-text + [:map + [:shape-id :string] + [:page-id :string] + [:text :string]]) + +(def ^:private schema:projection-link + [:map + [:shape-id :string] + [:page-id :string] + [:url :string]]) + +(def ^:private schema:projection-interaction + [:map + [:shape-id :string] + [:page-id :string] + [:event :string] + [:action :string] + [:target {:optional true} :string]]) + +(def ^:private schema:projection + [:map {:title "file-projection"} + [:version ::sm/int] + [:pages [:vector schema:projection-page]] + [:frames [:vector schema:projection-frame]] + [:texts [:map + [:items [:vector schema:projection-text]] + [:truncated? :boolean]]] + [:links [:map + [:items [:vector schema:projection-link]] + [:truncated? :boolean]]] + [:interactions [:map + [:items [:vector schema:projection-interaction]] + [:truncated? :boolean]]]]) + +(def ^:private + schema:get-file-for-analysis-result + [:map {:title "get-file-for-analysis-result"} + [:id ::sm/uuid] + [:revn ::sm/int] + [:modified-at ::ct/inst] + [:project-id ::sm/uuid] + [:team-id ::sm/uuid] + [:has-share-link :boolean] + [:projection-version ::sm/int] + [:projection schema:projection]]) + +(sv/defmethod ::get-file-for-analysis + "Get the semantic projection of one file for security analysis. + + Read-only: returns the file identity, revision, share state and the + neutral projection (pages, frames, text, links, interactions). + Called by nexus with the shared-key identity; takes no profile. + Every call leaves an audit event like any other management call." + {::doc/added "2.20" + ::sm/params schema:get-file-for-analysis-params + ::sm/result schema:get-file-for-analysis-result + ::rpc/auth false} + [cfg {:keys [file-id]}] + ;; `realize?` loads the whole object tree eagerly: the projection + ;; walks every shape, so lazy pointers (no `load-fn` bound here) + ;; would fail on first touch. Same choice as the file exporter. + (let [file (bfc/get-file cfg file-id :realize? true) + shared (:shared (db/exec-one! cfg [sql:has-share-link file-id])) + team-id (:team-id (db/exec-one! cfg [sql:get-team-id (:project-id file)]))] + {:id (:id file) + :revn (:revn file) + :modified-at (:modified-at file) + :project-id (:project-id file) + :team-id team-id + :has-share-link (boolean shared) + :projection-version projection-version + :projection (file->projection (:data file))})) diff --git a/backend/src/app/srepl/main.clj b/backend/src/app/srepl/main.clj index a205320242..51075ecda0 100644 --- a/backend/src/app/srepl/main.clj +++ b/backend/src/app/srepl/main.clj @@ -783,7 +783,7 @@ :type "action" :props {:id team-id} :context {:triggered-by "srepl" - :cause "explicit call to delete-profile!"} + :cause "explicit call to delete-team!"} :tracked-at tnow}) (jobs/invoke (-> sys/system @@ -852,6 +852,7 @@ (audit/insert sys/system {:name "delete-profile" :type "action" + :props {:id profile-id} :context {:triggered-by "srepl" :cause "explicit call to delete-profile!"} :tracked-at tnow}) @@ -1040,4 +1041,4 @@ (let [params (-> rel (assoc :id (uuid/next)) (assoc :team-id (:id team)))] - (teams/add-profile-to-team! cfg params {::db/return-keys false})))))))) + (teams/add-profile-to-team! cfg params)))))))) diff --git a/backend/src/app/storage/gc_deleted.clj b/backend/src/app/storage/gc_deleted.clj index bdc3cc3bef..ce2ce83556 100644 --- a/backend/src/app/storage/gc_deleted.clj +++ b/backend/src/app/storage/gc_deleted.clj @@ -59,18 +59,6 @@ (-> (db/exec-one! conn [sql:delete-sobjects ids]) (db/get-update-count)))) -(def ^:private sql:delete-upload-session-chunks - "DELETE FROM upload_session_chunk - WHERE object_id = ANY(?::uuid[])") - -(defn- delete-upload-session-chunks - "Remove the chunk mappings for the given storage object ids. This must run - before the storage_object rows are deleted: the upload_session_chunk - foreign keys are ON DELETE NO ACTION." - [conn ids] - (let [ids (db/create-array conn "uuid" ids)] - (db/exec-one! conn [sql:delete-upload-session-chunks ids]))) - (def ^:private sql:increment-attempts-and-defer "UPDATE storage_object SET deletion_attempts = deletion_attempts + 1, @@ -119,21 +107,17 @@ :backend (name backend-id))) (when (seq ok-ids) - ;; NOTE: the chunk mappings must be removed before the - ;; storage_object rows (NO ACTION foreign keys). It only affects - ;; objects of the upload-session bucket; for any other bucket the - ;; delete matches no rows. - (delete-upload-session-chunks conn ok-ids) + ;; NOTE: rows arrive here with no live references (the mark-time + ;; scan in gc-touched owns that check, also for the upload-session + ;; bucket). If a NO ACTION reference still exists, this DELETE + ;; aborts the chunk loud instead of breaking anyone silently. (delete-sobjects conn ok-ids)) (when (seq fail-ids) + ;; NOTE: deferred rows keep both their row and their references + ;; (upload-session mappings included) until the final give-up, so + ;; a backend outage does not break live sessions. (increment-attempts-and-defer conn fail-ids) - ;; NOTE: same NO ACTION ordering as above: the give-up DELETE below - ;; removes storage_object rows, so chunk mappings must go first. - ;; Deferred objects keep their rows; only the mapping of a - ;; permanently given-up object disappears early, and that object is - ;; already deleted-marked. - (delete-upload-session-chunks conn fail-ids) (let [given-up (delete-give-up conn fail-ids)] (when (pos? (db/get-update-count given-up)) (l/wrn :hint "giving up on orphan blob after max attempts" diff --git a/backend/src/app/storage/gc_touched.clj b/backend/src/app/storage/gc_touched.clj index 82394e8f7b..80e411d9da 100644 --- a/backend/src/app/storage/gc_touched.clj +++ b/backend/src/app/storage/gc_touched.clj @@ -96,6 +96,15 @@ (-> (db/exec-one! conn [sql:has-job-resource-refs id]) (get :has-refs))) +(def ^:private + sql:has-upload-session-chunk-refs + "SELECT EXISTS (SELECT 1 FROM upload_session_chunk WHERE object_id = ?) AS has_refs") + +(defn- has-upload-session-chunk-refs? + [conn {:keys [id]}] + (-> (db/exec-one! conn [sql:has-upload-session-chunk-refs id]) + (get :has-refs))) + (def sql:exists-file-data-refs "SELECT EXISTS ( SELECT 1 FROM file_data @@ -165,6 +174,31 @@ :id (str id)) sto/default-bucket))) +(def ^:private sql:lock-storage-objects + "SELECT id FROM storage_object WHERE id = ANY(?::uuid[]) ORDER BY id FOR UPDATE") + +(defn- verify-frozen + "Re-checks, holding the row locks, that the about-to-freeze objects are + still referenced. A concurrent purge (mapping delete plus touch) either + commits before the check and the object moves to deletion, or blocks on + the lock and its touch wins last, so no touch is ever swallowed by a + stale freeze. Ids are locked in order to avoid lock-order inversions + between workers. Returns [still-frozen-ids now-deletable-ids]." + [conn has-refs? bucket objects] + (let [by-id (into {} (map (juxt :id identity)) objects) + ids (db/create-array conn "uuid" (sort (keys by-id))) + locked (into #{} (map :id) (db/exec! conn [sql:lock-storage-objects ids]))] + (reduce (fn [[frozen deletable] {:keys [id] :as object}] + (if (and (contains? locked id) (has-refs? conn object)) + [(conj frozen id) deletable] + (do + (l/dbg :id (str id) + :status "delete-after-recheck" + :bucket bucket) + [frozen (conj deletable id)]))) + [#{} #{}] + objects))) + (defn- process-objects [conn has-refs? bucket objects] (loop [to-freeze #{} @@ -176,17 +210,24 @@ (l/dbg :id (str id) :status "freeze" :bucket bucket) - (recur (conj to-freeze id) to-delete (rest objects))) + (recur (conj to-freeze object) to-delete (rest objects))) (do (l/dbg :id (str id) :status "delete" :bucket bucket) (recur to-freeze (conj to-delete id) (rest objects)))) - (do - (some->> (seq to-freeze) (mark-freeze-in-bulk conn)) - (some->> (seq to-delete) (mark-delete-in-bulk conn)) - [(count to-freeze) (count to-delete)])))) + (let [[frozen deletable] (if (seq to-freeze) + (verify-frozen conn has-refs? bucket to-freeze) + [#{} #{}]) + doomed (into to-delete deletable)] + (some->> (seq frozen) (mark-freeze-in-bulk conn)) + (some->> (seq doomed) (mark-delete-in-bulk conn)) + [(count frozen) (count doomed)])))) +;; NOTE: upload-session chunks are 1:1 with their mapping and never +;; shared, but the scan owns the liveness decision anyway: a referenced +;; chunk freezes, so gc-deleted never meets a live NO ACTION reference. +;; The freeze path re-verifies under row locks (see verify-frozen). (defn- process-bucket [conn bucket objects] (cond @@ -197,7 +238,7 @@ (= bucket "profile") (process-objects conn has-profile-refs? bucket objects) (= bucket "file-data") (process-objects conn has-file-data-refs? bucket objects) (= bucket sto/tempfile-bucket) (process-objects conn (constantly false) sto/tempfile-bucket objects) - (= bucket sto/upload-session-bucket) (process-objects conn (constantly false) sto/upload-session-bucket objects) + (= bucket sto/upload-session-bucket) (process-objects conn has-upload-session-chunk-refs? sto/upload-session-bucket objects) (= bucket sto/job-resource-bucket) (process-objects conn has-job-resource-refs? bucket objects) (= bucket "organization") (process-objects conn (constantly false) bucket objects) :else diff --git a/backend/src/app/tasks/delete_object.clj b/backend/src/app/tasks/delete_object.clj index 730a2364c9..267f8da51a 100644 --- a/backend/src/app/tasks/delete_object.clj +++ b/backend/src/app/tasks/delete_object.clj @@ -13,6 +13,7 @@ [app.db :as db] [app.db.sql :as-alias sql] [app.jobs :as jobs] + [app.loggers.audit :as audit] [app.rpc.commands.files :as files] [app.rpc.commands.profile :as profile] [app.storage :as sto] @@ -20,6 +21,29 @@ (def ^:dynamic *team-deletion* false) +(def ^:dynamic *cascade* + "Accumulates the ids marked deleted by the current cascade execution, + grouped by entity. Bound by the task entry point; nil on direct calls + (e.g. objects-gc, whose purges are announced by their own procs)." + nil) + +(defn- record + [kind id] + (when (some? *cascade*) + (swap! *cascade* update kind conj id))) + +(defn- top-exists? + "The cascade marks blindly, so an unknown top id would report phantom + rows. Every queue submission and SREPL call names a real row; only + check, never change behavior: a missing row just skips the run." + [{:keys [::db/conn]} {:keys [object id]}] + (let [object (when (some? object) (keyword (name object)))] + (if (contains? #{:team :project :file :profile} object) + (some? (db/get* conn object {:id id} + {::db/remove-deleted false + ::sql/columns [:id]})) + true))) + (defmulti delete-object (fn [_ props] (:object props))) @@ -39,7 +63,7 @@ {::db/return-keys false})) (defmethod delete-object :file - [{:keys [::db/conn] :as cfg} {:keys [id deleted-at]}] + [{:keys [::db/conn] :as cfg} {:keys [id deleted-at file-ids]}] (when-let [file (db/get* conn :file {:id id} {::db/remove-deleted false ::sql/columns [:id :is-shared]})] @@ -57,7 +81,7 @@ (not *team-deletion*)) ;; NOTE: we don't prevent file deletion on absorb operation failure (try - (db/tx-run! cfg files/absorb-library id) + (db/tx-run! cfg files/absorb-library id :file-ids file-ids) (catch Throwable cause (l/warn :hint "error on absorbing library" :file-id id @@ -90,7 +114,9 @@ (db/update! conn :file-tagged-object-thumbnail {:deleted-at deleted-at} {:file-id id} - {::db/return-keys false}))) + {::db/return-keys false}) + + (record :files id))) (defmethod delete-object :project [{:keys [::db/conn] :as cfg} {:keys [id deleted-at]}] @@ -107,7 +133,9 @@ {::db/columns [:id :deleted-at]})] (delete-object cfg (assoc file :object :file - :deleted-at deleted-at)))) + :deleted-at deleted-at))) + + (record :projects id)) (defmethod delete-object :team [{:keys [::db/conn] :as cfg} {:keys [id deleted-at]}] @@ -129,7 +157,9 @@ {::db/columns [:id :deleted-at]})] (delete-object cfg (assoc project :object :project - :deleted-at deleted-at))))) + :deleted-at deleted-at)))) + + (record :teams id)) (defmethod delete-object :profile [{:keys [::db/conn] :as cfg} {:keys [id deleted-at]}] @@ -156,13 +186,30 @@ [:object [:enum :snapshot :team :project :profile :file]] [:deleted-at ::ct/inst] [:id ::sm/uuid] - [:file-id {:optional true} ::sm/uuid]]) + [:file-id {:optional true} ::sm/uuid] + [:file-ids {:optional true} [:vector ::sm/uuid]]]) (defn execute-delete-object "Plain job handler: run the delete-object multimethod on the provided - params inside a single transaction." + params inside a single transaction, tracking the cascade and reporting + it as an audit event when anything was marked." [cfg params] - (db/tx-run! cfg delete-object params)) + (db/tx-run! cfg + (fn [cfg] + (when (top-exists? cfg params) + (binding [*cascade* (atom {:teams [] :projects [] :files []})] + (delete-object cfg params) + (let [{:keys [teams projects files]} @*cascade*] + (when (or (seq teams) (seq projects) (seq files)) + (audit/submit cfg {:name "delete-object-with-cascade" + :type "action" + :props {:object (:object params) + :id (:id params) + :deleted-at (:deleted-at params) + :cascade {:team-ids teams + :project-ids projects + :file-ids files}} + :context {:triggered-by "delete-object"}})))))))) (defmethod ig/assert-key ::job-def [_ params] diff --git a/backend/src/app/tasks/objects_gc.clj b/backend/src/app/tasks/objects_gc.clj index 1e530ea750..1fca7bf321 100644 --- a/backend/src/app/tasks/objects_gc.clj +++ b/backend/src/app/tasks/objects_gc.clj @@ -11,9 +11,11 @@ [app.common.logging :as l] [app.common.schema :as sm] [app.common.time :as ct] + [app.config :as cf] [app.db :as db] [app.features.fdata :as fdata] [app.jobs :as jobs] + [app.loggers.audit :as audit] [app.storage :as sto] [app.tasks.delete-object :as dobj] [integrant.core :as ig])) @@ -33,6 +35,22 @@ ;; through the pool and never through the chunk transaction, so a long backlog ;; in a single proc cannot outrun the orphan lease. +(defn- submit-hard-delete-event + "Emits one hard-delete audit event for the batch of rows just purged + by this task, so a projection of the audit stream can drop its local + rows. Each GC section purges in its own chunk transactions, so each + chunk carries only its own entity key (`:team-ids`, `:project-ids` + or `:file-ids`); a single event spanning sections could never share + their transactions. Runs inside the caller transaction, next to the + deletes: if the purge aborts, the event aborts with it." + [cfg ids-key ids] + (when (contains? cf/flags :audit-log) + (audit/submit cfg + {:name "hard-delete-objects" + :type "command" + :props {ids-key (vec ids)} + :context {:triggered-by "objects-gc"}}))) + (def ^:private sql:get-upload-sessions "SELECT us.id FROM upload_session AS us @@ -191,23 +209,28 @@ [cfg ids] (try (db/tx-run! cfg - (fn [{:keys [::db/conn ::threshold ::sto/storage]}] + (fn [{:keys [::db/conn ::threshold ::sto/storage] :as cfg}] (db/exec-one! conn ["SET LOCAL rules.deletion_protection TO off"]) - (let [ids (db/create-array conn "uuid" ids)] - (->> (db/exec! conn [sql:lock-teams ids threshold]) - (reduce (fn [total {:keys [id photo-id deleted-at]}] - (l/trc :obj "team" - :id (str id) - :deleted-at (ct/format-inst deleted-at)) + (let [ids (db/create-array conn "uuid" ids) + [total deleted-ids] + (->> (db/exec! conn [sql:lock-teams ids threshold]) + (reduce (fn [[total deleted-ids] {:keys [id photo-id deleted-at]}] + (l/trc :obj "team" + :id (str id) + :deleted-at (ct/format-inst deleted-at)) - ;; Mark as deleted the storage object - (some->> photo-id (sto/touch-object! storage)) + ;; Mark as deleted the storage object + (some->> photo-id (sto/touch-object! storage)) - ;; And finally, permanently delete the team. - (let [affected (-> (db/delete! conn :team {:id id}) - (db/get-update-count))] - (+ total affected))) - 0))))) + ;; And finally, permanently delete the team. + (let [affected (-> (db/delete! conn :team {:id id}) + (db/get-update-count))] + [(+ total affected) + (cond-> deleted-ids (pos? affected) (conj id))])) + [0 []]))] + (when (seq deleted-ids) + (submit-hard-delete-event cfg :team-ids deleted-ids)) + total))) (catch Throwable cause (l/err :obj "team" :hint "error on chunk deletion, skipping it" @@ -305,20 +328,25 @@ [cfg ids] (try (db/tx-run! cfg - (fn [{:keys [::db/conn ::threshold]}] + (fn [{:keys [::db/conn ::threshold] :as cfg}] (db/exec-one! conn ["SET LOCAL rules.deletion_protection TO off"]) - (let [ids (db/create-array conn "uuid" ids)] - (->> (db/exec! conn [sql:lock-projects ids threshold]) - (reduce (fn [total {:keys [id team-id deleted-at]}] - (l/trc :obj "project" - :id (str id) - :team-id (str team-id) - :deleted-at (ct/format-inst deleted-at)) + (let [ids (db/create-array conn "uuid" ids) + [total deleted-ids] + (->> (db/exec! conn [sql:lock-projects ids threshold]) + (reduce (fn [[total deleted-ids] {:keys [id team-id deleted-at]}] + (l/trc :obj "project" + :id (str id) + :team-id (str team-id) + :deleted-at (ct/format-inst deleted-at)) - (let [affected (-> (db/delete! conn :project {:id id}) - (db/get-update-count))] - (+ total affected))) - 0))))) + (let [affected (-> (db/delete! conn :project {:id id}) + (db/get-update-count))] + [(+ total affected) + (cond-> deleted-ids (pos? affected) (conj id))])) + [0 []]))] + (when (seq deleted-ids) + (submit-hard-delete-event cfg :project-ids deleted-ids)) + total))) (catch Throwable cause (l/err :obj "project" :hint "error on chunk deletion, skipping it" @@ -357,20 +385,25 @@ [cfg ids] (try (db/tx-run! cfg - (fn [{:keys [::db/conn ::threshold]}] + (fn [{:keys [::db/conn ::threshold] :as cfg}] (db/exec-one! conn ["SET LOCAL rules.deletion_protection TO off"]) - (let [ids (db/create-array conn "uuid" ids)] - (->> (db/exec! conn [sql:lock-files ids threshold]) - (reduce (fn [total {:keys [id deleted-at project-id]}] - (l/trc :obj "file" - :id (str id) - :project-id (str project-id) - :deleted-at (ct/format-inst deleted-at)) + (let [ids (db/create-array conn "uuid" ids) + [total deleted-ids] + (->> (db/exec! conn [sql:lock-files ids threshold]) + (reduce (fn [[total deleted-ids] {:keys [id deleted-at project-id]}] + (l/trc :obj "file" + :id (str id) + :project-id (str project-id) + :deleted-at (ct/format-inst deleted-at)) - (let [affected (-> (db/delete! conn :file {:id id}) - (db/get-update-count))] - (+ total affected))) - 0))))) + (let [affected (-> (db/delete! conn :file {:id id}) + (db/get-update-count))] + [(+ total affected) + (cond-> deleted-ids (pos? affected) (conj id))])) + [0 []]))] + (when (seq deleted-ids) + (submit-hard-delete-event cfg :file-ids deleted-ids)) + total))) (catch Throwable cause (l/err :obj "file" :hint "error on chunk deletion, skipping it" diff --git a/backend/test/backend_tests/binfile_svg_test.clj b/backend/test/backend_tests/binfile_svg_test.clj index f992f155d2..5094a0b2dc 100644 --- a/backend/test/backend_tests/binfile_svg_test.clj +++ b/backend/test/backend_tests/binfile_svg_test.clj @@ -375,13 +375,14 @@ of its single media object." [profile file] (let [bundle (export-v1 profile file)] - (let [result (-> th/*system* - (assoc ::bfc/project-id (:default-project-id profile)) - (assoc ::bfc/profile-id (:id profile)) - (assoc ::bfc/input (io/input-stream bundle)) - (v1/import-files!))] - (t/is (= 1 (count result))) - (let [rows (th/db-query :file-media-object {:file-id (first result)}) + (let [result (-> th/*system* + (assoc ::bfc/project-id (:default-project-id profile)) + (assoc ::bfc/profile-id (:id profile)) + (assoc ::bfc/input (io/input-stream bundle)) + (v1/import-files!)) + file-id (first (:file-ids result))] + (t/is (= 1 (count (:file-ids result)))) + (let [rows (th/db-query :file-media-object {:file-id file-id}) storage (:app.storage/storage th/*system*)] (t/is (= 1 (count rows))) (let [object (sto/get-object storage (:media-id (first rows)))] diff --git a/backend/test/backend_tests/binfile_test.clj b/backend/test/backend_tests/binfile_test.clj index 40baeb63ee..ff42106519 100644 --- a/backend/test/backend_tests/binfile_test.clj +++ b/backend/test/backend_tests/binfile_test.clj @@ -12,17 +12,18 @@ [app.binfile.v3 :as v3] [app.common.data :as d] [app.common.features :as cfeat] + [app.common.files.changes-builder :as pcb] + [app.common.files.tokens :as cfo] [app.common.files.validate :as cfv] - [app.common.pprint :as pp] [app.common.thumbnails :as thc] [app.common.time :as ct] [app.common.types.shape :as cts] + [app.common.types.token :as cto] [app.common.types.tokens-lib :as ctob] + [app.common.types.tokens-status :as ctos] [app.common.uuid :as uuid] [app.config :as cf] [app.db :as db] - [app.db.sql :as sql] - [app.http :as http] [app.rpc :as-alias rpc] [app.rpc.commands.binfile :as binfile] [app.storage :as sto] @@ -36,7 +37,6 @@ [datoteka.io :as io]) (:import java.io.ByteArrayInputStream - java.io.DataInputStream java.io.OutputStreamWriter java.io.Writer java.util.zip.Deflater @@ -105,6 +105,117 @@ (dissoc file :data)))) +(defn- prepare-file-with-tokens-lib + "A shared library and a linked file with different active token sets. + The file has a rect with two applied tokens. Returns the created files, + and the ids and tokens needed to check them." + [profile] + (let [lib-page-id (uuid/custom 5 1) + file-page-id (uuid/custom 5 2) + shape-id (uuid/custom 5 3) + set-id (uuid/custom 5 4) + theme-id (uuid/custom 5 5) + other-set-id (uuid/custom 5 8) + other-theme-id (uuid/custom 5 9) + + token-radius (ctob/make-token :id (uuid/custom 5 6) + :name "token-radius" + :type :border-radius + :value 10) + + token-opacity (ctob/make-token :id (uuid/custom 5 7) + :name "token-opacity" + :type :opacity + :value 0.7) + + tokens-lib (-> (ctob/make-tokens-lib) + (ctob/add-set (ctob/make-token-set :id set-id + :name "test-token-set")) + (ctob/add-theme (ctob/make-token-theme :id theme-id + :name "test-theme" + :sets #{"test-token-set"})) + (ctob/add-set (ctob/make-token-set :id other-set-id + :name "other-token-set")) + (ctob/add-theme (ctob/make-token-theme :id other-theme-id + :name "other-theme" + :sets #{"other-token-set"})) + (ctob/add-token other-set-id (ctob/make-token :id (uuid/custom 5 10) + :name "other-token" + :type :opacity + :value 0.4)) + (ctob/add-token set-id token-radius) + (ctob/add-token set-id token-opacity)) + + library (th/create-file* 1 {:profile-id (:id profile) + :project-id (:default-project-id profile) + :is-shared true + :name "Tokens Library"}) + + file (th/create-file* 2 {:profile-id (:id profile) + :project-id (:default-project-id profile) + :is-shared false})] + + (th/link-file-to-library* {:file-id (:id file) + :library-id (:id library)}) + + (update-file! + :file-id (:id library) + :profile-id (:id profile) + :revn 0 + :changes + (-> (pcb/empty-changes nil) + (pcb/with-library-data (:data library)) + (pcb/add-empty-page lib-page-id "library page") + (pcb/set-tokens-lib tokens-lib) + (pcb/set-tokens-status (ctos/make-tokens-status :active-theme-ids #{other-theme-id} + :active-set-ids #{other-set-id})) + (:redo-changes))) + + (update-file! + :file-id (:id file) + :profile-id (:id profile) + :revn 0 + :changes + (-> (pcb/empty-changes nil) + (pcb/with-library-data (:data file)) + (pcb/add-empty-page file-page-id "file page") + (pcb/set-tokens-source (:id library)) + (pcb/set-tokens-status (ctos/make-tokens-status :active-theme-ids #{theme-id} + :active-set-ids #{set-id})) + (:redo-changes))) + + (let [page (-> (bfc/get-file th/*system* (:id file) {:realize? true}) + (get-in [:data :pages-index file-page-id])) + shape (as-> (cts/setup-shape + {:id shape-id + :name "rect" + :frame-id uuid/zero + :parent-id uuid/zero + :type :rect}) $ + (cto/apply-token-to-shape {:shape $ + :token token-radius + :attributes [:r1 :r2 :r3 :r4]}) + (cto/apply-token-to-shape {:shape $ + :token token-opacity + :attributes [:opacity]}))] + (update-file! + :file-id (:id file) + :profile-id (:id profile) + :revn 0 + :changes + (-> (pcb/empty-changes nil) + (pcb/with-page page) + (pcb/with-objects (:objects page)) + (pcb/add-object shape) + (:redo-changes)))) + + {:library library + :file file + :page-id file-page-id + :shape-id shape-id + :token-radius token-radius + :token-opacity token-opacity})) + (def ^:private svg-raw-page-id (uuid/custom 1 1)) (def ^:private svg-raw-root-id (uuid/custom 3 1)) (def ^:private svg-raw-child-id (uuid/custom 3 2)) @@ -376,6 +487,299 @@ (t/is (= "icons-library" (:slug ext-lib))) (t/is (= [(:id file)] (:used-by ext-lib)))))))) +(defn- zip-entry-names + [path] + (with-open [zin (ZipFile. (fs/file path))] + (into #{} (map #(.getName ^ZipEntry %)) (iterator-seq (.entries zin))))) + +(t/deftest export-detach-libraries-removes-tokens-library-although-keeps-applied-tokens + (let [profile (th/create-profile* 1) + {:keys [library file page-id shape-id token-radius token-opacity]} + (prepare-file-with-tokens-lib profile)] + + ;; Precondition: before the export, the file really depends on the + ;; tokens of the library + (let [file-data (:data (bfc/get-file th/*system* (:id file)))] + (t/is (= (:id library) (cfo/get-tokens-source file-data))) + (t/is (some? (cfo/get-tokens-status file-data))) + (t/is (nil? (cfo/get-tokens-lib file-data)))) + + ;; Do the export, with type :detach-libraries + (let [output (tmp/tempfile :suffix ".zip")] + (v3/export-files! + (-> th/*system* + (assoc ::bfc/ids #{(:id file)}) + (assoc ::bfc/export-type :detach-libraries)) + (io/output-stream output)) + + ;; Only the file is exported, not the library + (let [manifest (v3/get-manifest th/*system* output) + names (zip-entry-names output)] + (t/is (= [(:id file)] (mapv :id (:files manifest)))) + (t/is (empty? (:relations manifest))) + (t/is (empty? (:external-libraries manifest))) + (t/is (contains? names (str "files/" (:id file) ".json"))) + (t/is (not-any? #(str/includes? % (str (:id library))) names))) + + ;; The exported file has neither tokens library nor tokens status + (let [names (zip-entry-names output)] + (t/is (not (contains? names (str "files/" (:id file) "/tokens.json")))) + (t/is (not (contains? names (str "files/" (:id file) "/tokens-status.json"))))) + + ;; Import the file again + (let [result (-> th/*system* + (assoc ::bfc/project-id (:default-project-id profile)) + (assoc ::bfc/team-id (:default-team-id profile)) + (assoc ::bfc/profile-id (:id profile)) + (assoc ::bfc/input output) + (v3/import-files!)) + imported (bfc/get-file th/*system* (first (:file-ids result))) + file-data (:data imported) + shape (get-in file-data [:pages-index page-id :objects shape-id])] + + (t/is (= 1 (count (:file-ids result)))) + + ;; ...so the imported file has no tokens source, status or library + (t/is (nil? (cfo/get-tokens-source file-data))) + (t/is (nil? (cfo/get-tokens-status file-data))) + (t/is (nil? (cfo/get-tokens-lib file-data))) + + ;; The shape keeps the applied tokens + (t/is (some? shape)) + (t/is (cfo/token-applied? token-radius shape [:r1 :r2 :r3 :r4])) + (t/is (cfo/token-applied? token-opacity shape [:opacity])))))) + +(t/deftest export-merge-libraries-embeds-tokens-library-and-keeps-applied-tokens + (let [profile (th/create-profile* 1) + {:keys [library file page-id shape-id token-radius token-opacity]} + (prepare-file-with-tokens-lib profile)] + + ;; Precondition: before the export, the file depends on the tokens + ;; of the library and has no tokens library of its own + (let [file-data (:data (bfc/get-file th/*system* (:id file)))] + (t/is (= (:id library) (cfo/get-tokens-source file-data))) + (t/is (nil? (cfo/get-tokens-lib file-data)))) + + (let [library-tokens-lib (-> (bfc/get-file th/*system* (:id library)) + (:data) + (cfo/get-tokens-lib)) + output (tmp/tempfile :suffix ".zip")] + + (t/is (some? library-tokens-lib)) + + (v3/export-files! + (-> th/*system* + (assoc ::bfc/ids #{(:id file)}) + (assoc ::bfc/export-type :merge-libraries)) + (io/output-stream output)) + + ;; Only the file is exported, not the library + (let [manifest (v3/get-manifest th/*system* output) + names (zip-entry-names output)] + (t/is (= [(:id file)] (mapv :id (:files manifest)))) + (t/is (empty? (:relations manifest))) + (t/is (empty? (:external-libraries manifest))) + (t/is (contains? names (str "files/" (:id file) ".json"))) + (t/is (not-any? #(str/includes? % (str (:id library))) names))) + + ;; The exported file carries its own tokens library + (t/is (contains? (zip-entry-names output) + (str "files/" (:id file) "/tokens.json"))) + + (let [result (-> th/*system* + (assoc ::bfc/project-id (:default-project-id profile)) + (assoc ::bfc/team-id (:default-team-id profile)) + (assoc ::bfc/profile-id (:id profile)) + (assoc ::bfc/input output) + (v3/import-files!)) + imported (bfc/get-file th/*system* (first (:file-ids result))) + file-data (:data imported) + shape (get-in file-data [:pages-index page-id :objects shape-id])] + + (t/is (= 1 (count (:file-ids result)))) + + ;; The tokens library of the library file is now in the file... + ;; (the import gives new ids to the sets and tokens, so compare + ;; their content) + (t/is (some? (cfo/get-tokens-lib file-data))) + (t/is (= (ctob/export-dtcg-json library-tokens-lib) + (ctob/export-dtcg-json (cfo/get-tokens-lib file-data)))) + + ;; ...so the file is its own tokens source + (t/is (nil? (cfo/get-tokens-source file-data))) + + ;; Activation follows the consuming file, using the imported ids. + (t/is (= {:active-themes #{"/test-theme"} + :active-sets #{"test-token-set"}} + (cfo/ids->names (cfo/get-tokens-status file-data) + (cfo/get-tokens-lib file-data)))) + (t/is (= #{"token-radius" "token-opacity"} + (set (keys (cfo/get-tokens-in-active-sets + (cfo/get-tokens-status file-data) + (cfo/get-tokens-lib file-data)))))) + + ;; The shape keeps the applied tokens + (t/is (some? shape)) + (t/is (cfo/token-applied? token-radius shape [:r1 :r2 :r3 :r4])) + (t/is (cfo/token-applied? token-opacity shape [:opacity])))))) + +(t/deftest export-detach-libraries-preserves-own-tokens-and-status + (let [profile (th/create-profile* 1) + {:keys [library file page-id shape-id token-radius token-opacity]} + (prepare-file-with-tokens-lib profile) + file-data (:data (bfc/get-file th/*system* (:id file))) + library-data (:data (bfc/get-file th/*system* (:id library))) + tokens-lib (cfo/get-tokens-lib library-data) + output (tmp/tempfile :suffix ".zip")] + + ;; Keep the linked library, but use the file's own tokens. + (update-file! + :file-id (:id file) + :profile-id (:id profile) + :changes + (-> (pcb/empty-changes nil) + (pcb/with-library-data file-data) + (pcb/set-tokens-source nil) + (pcb/set-tokens-lib tokens-lib) + (:redo-changes))) + + (v3/export-files! + (assoc th/*system* + ::bfc/ids #{(:id file)} + ::bfc/export-type :detach-libraries) + (io/output-stream output)) + + (let [names (zip-entry-names output)] + (t/is (contains? names (str "files/" (:id file) "/tokens.json"))) + (t/is (contains? names (str "files/" (:id file) "/tokens-status.json")))) + + (let [result (v3/import-files! + (assoc th/*system* + ::bfc/project-id (:default-project-id profile) + ::bfc/team-id (:default-team-id profile) + ::bfc/profile-id (:id profile) + ::bfc/input output)) + imported (bfc/get-file th/*system* (first (:file-ids result))) + file-data (:data imported) + shape (get-in file-data [:pages-index page-id :objects shape-id])] + (t/is (nil? (cfo/get-tokens-source file-data))) + (t/is (= (ctob/export-dtcg-json tokens-lib) + (ctob/export-dtcg-json (cfo/get-tokens-lib file-data)))) + (t/is (some? (cfo/get-tokens-status file-data))) + (when-let [status (cfo/get-tokens-status file-data)] + (t/is (= {:active-themes #{"/test-theme"} + :active-sets #{"test-token-set"}} + (cfo/ids->names status (cfo/get-tokens-lib file-data))))) + (t/is (cfo/token-applied? token-radius shape [:r1 :r2 :r3 :r4])) + (t/is (cfo/token-applied? token-opacity shape [:opacity]))))) + +(defn- import-output + "Import the exported output in the default project of the profile and + return the imported files, indexed by name." + [profile output] + (let [result (v3/import-files! + (assoc th/*system* + ::bfc/project-id (:default-project-id profile) + ::bfc/team-id (:default-team-id profile) + ::bfc/profile-id (:id profile) + ::bfc/input output))] + (into {} + (map (fn [id] + (let [file (bfc/get-file th/*system* id)] + [(:name file) file]))) + (:file-ids result)))) + +(t/deftest export-include-libraries-keeps-tokens-status-of-the-file + (let [profile (th/create-profile* 1) + {:keys [library file]} (prepare-file-with-tokens-lib profile) + output (tmp/tempfile :suffix ".zip")] + + (v3/export-files! + (assoc th/*system* + ::bfc/ids #{(:id file)} + ::bfc/export-type :include-libraries) + (io/output-stream output)) + + (let [files (import-output profile output) + library' (get files (:name library)) + file' (get files (:name file)) + file-data (:data file') + library-lib (cfo/get-tokens-lib (:data library')) + file-status (cfo/get-tokens-status file-data)] + + (t/is (= 2 (count files))) + (t/is (= (:id library') (cfo/get-tokens-source file-data))) + + ;; The active sets of the file are resolved against the sets of the + ;; imported library, so they are not lost. + (t/is (= {:active-themes #{"/test-theme"} + :active-sets #{"test-token-set"}} + (cfo/ids->names file-status library-lib))) + (t/is (= #{"token-radius" "token-opacity"} + (set (keys (cfo/get-tokens-in-active-sets file-status library-lib)))))))) + +(t/deftest export-file-with-deleted-tokens-library-embeds-its-tokens + (let [profile (th/create-profile* 1) + {:keys [library file]} (prepare-file-with-tokens-lib profile) + tokens-lib (-> (bfc/get-file th/*system* (:id library)) + (:data) + (cfo/get-tokens-lib)) + output (tmp/tempfile :suffix ".zip")] + + ;; The state left by deleting the library: in the trash and unlinked, + ;; but still the tokens source of the file. + (th/db-update! :file {:deleted-at (ct/in-future {:days 7})} {:id (:id library)}) + (th/db-delete! :file-library-rel {:library-file-id (:id library)}) + + (v3/export-files! + (assoc th/*system* + ::bfc/ids #{(:id file)} + ::bfc/export-type :include-libraries) + (io/output-stream output)) + + (t/is (contains? (zip-entry-names output) + (str "files/" (:id file) "/tokens.json"))) + + (let [files (import-output profile output) + file-data (:data (get files (:name file)))] + (t/is (= 1 (count files))) + (t/is (nil? (cfo/get-tokens-source file-data))) + (t/is (= (ctob/export-dtcg-json tokens-lib) + (ctob/export-dtcg-json (cfo/get-tokens-lib file-data)))) + (t/is (= {:active-themes #{"/test-theme"} + :active-sets #{"test-token-set"}} + (cfo/ids->names (cfo/get-tokens-status file-data) + (cfo/get-tokens-lib file-data))))))) + +(t/deftest export-file-with-missing-tokens-library-removes-tokens-source + (let [profile (th/create-profile* 1) + {:keys [library file]} (prepare-file-with-tokens-lib profile) + output (tmp/tempfile :suffix ".zip")] + + ;; The tokens source points to a library that does not exist anymore + (th/db-delete! :file-library-rel {:library-file-id (:id library)}) + (update-file! + :file-id (:id file) + :profile-id (:id profile) + :changes + (-> (pcb/empty-changes nil) + (pcb/with-library-data (:data (bfc/get-file th/*system* (:id file)))) + (pcb/set-tokens-source (uuid/next)) + (:redo-changes))) + + (v3/export-files! + (assoc th/*system* + ::bfc/ids #{(:id file)} + ::bfc/export-type :include-libraries) + (io/output-stream output)) + + (let [files (import-output profile output) + file-data (:data (get files (:name file)))] + (t/is (= 1 (count files))) + (t/is (nil? (cfo/get-tokens-source file-data))) + (t/is (nil? (cfo/get-tokens-status file-data))) + (t/is (nil? (cfo/get-tokens-lib file-data)))))) + (t/deftest import-auto-links-single-candidate (let [profile (th/create-profile* 1) ;; Create a shared library file diff --git a/backend/test/backend_tests/delete_object_test.clj b/backend/test/backend_tests/delete_object_test.clj new file mode 100644 index 0000000000..878bea305d --- /dev/null +++ b/backend/test/backend_tests/delete_object_test.clj @@ -0,0 +1,107 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns backend-tests.delete-object-test + (:require + [app.common.time :as ct] + [app.common.uuid :as uuid] + [app.config :as cf] + [app.db :as db] + [backend-tests.helpers :as th] + [clojure.test :as t] + [mockery.core :refer [with-mocks]])) + +(t/use-fixtures :once th/state-init) +(t/use-fixtures :each th/database-reset) + +(defn- submitted-events + [audit-mock] + (->> (:call-args-list @audit-mock) + (map second))) + +(defn- cascade-events + [audit-mock] + (filter #(= "delete-object-with-cascade" (:name %)) + (submitted-events audit-mock))) + +(t/deftest delete-object-team-cascade-emits-event + ;; The cascade marks the team, its projects and their files; the event + ;; carries every affected id so a projection can follow without reading + ;; production tables. + (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] + (with-redefs [cf/flags (conj cf/flags :audit-log)] + (let [profile (th/create-profile* 1 {:is-active true}) + team (th/create-team* 91 {:profile-id (:id profile)}) + project (th/create-project* 91 {:profile-id (:id profile) + :team-id (:id team)}) + file (th/create-file* 91 {:profile-id (:id profile) + :project-id (:id project)}) + deleted-at (ct/now)] + (th/run-task! :delete-object {:object :team + :id (:id team) + :deleted-at deleted-at}) + (t/is (some? (:deleted-at (th/db-get :team {:id (:id team)} + {::db/remove-deleted false})))) + (t/is (some? (:deleted-at (th/db-get :project {:id (:id project)} + {::db/remove-deleted false})))) + (t/is (some? (:deleted-at (th/db-get :file {:id (:id file)} + {::db/remove-deleted false})))) + (let [events (cascade-events audit-mock)] + (t/is (= 1 (count events))) + (t/is (= "action" (:type (first events)))) + (let [props (:props (first events))] + (t/is (= "team" (name (:object props)))) + (t/is (= (:id team) (:id props))) + (t/is (= deleted-at (:deleted-at props))) + (t/is (not (contains? props :team-ids))) + (t/is (not (contains? props :project-ids))) + (t/is (not (contains? props :file-ids))) + (t/is (= #{(:id team)} (set (get-in props [:cascade :team-ids])))) + (t/is (= #{(:default-project-id team) (:id project)} + (set (get-in props [:cascade :project-ids])))) + (t/is (= #{(:id file)} (set (get-in props [:cascade :file-ids])))))))))) + +(t/deftest delete-object-file-cascade-emits-event + (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] + (with-redefs [cf/flags (conj cf/flags :audit-log)] + (let [profile (th/create-profile* 1 {:is-active true}) + team (th/create-team* 92 {:profile-id (:id profile)}) + project (th/create-project* 92 {:profile-id (:id profile) + :team-id (:id team)}) + file (th/create-file* 92 {:profile-id (:id profile) + :project-id (:id project)}) + deleted-at (ct/now)] + (th/run-task! :delete-object {:object :file + :id (:id file) + :deleted-at deleted-at}) + (let [events (cascade-events audit-mock)] + (t/is (= 1 (count events))) + (t/is (= "action" (:type (first events)))) + (let [props (:props (first events))] + (t/is (= "file" (name (:object props)))) + (t/is (= (:id file) (:id props))) + (t/is (= [] (get-in props [:cascade :team-ids]))) + (t/is (= [] (get-in props [:cascade :project-ids]))) + (t/is (= [(:id file)] (get-in props [:cascade :file-ids]))))))))) + +(t/deftest delete-object-missing-row-emits-nothing + ;; A cascade over an unknown id marks nothing, so there is nothing to report. + (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] + (with-redefs [cf/flags (conj cf/flags :audit-log)] + (th/run-task! :delete-object {:object :team + :id (uuid/next) + :deleted-at (ct/now)}) + (t/is (empty? (:call-args-list @audit-mock)))))) + +(t/deftest delete-object-snapshot-emits-nothing + ;; Snapshots have no projection table; their cascade stays silent. + (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] + (with-redefs [cf/flags (conj cf/flags :audit-log)] + (th/run-task! :delete-object {:object :snapshot + :id (uuid/next) + :file-id (uuid/next) + :deleted-at (ct/now)}) + (t/is (empty? (:call-args-list @audit-mock)))))) diff --git a/backend/test/backend_tests/helpers.clj b/backend/test/backend_tests/helpers.clj index bbcc71173c..1f04cb5021 100644 --- a/backend/test/backend_tests/helpers.clj +++ b/backend/test/backend_tests/helpers.clj @@ -149,6 +149,7 @@ :app.notifications/handler :app.loggers.mattermost/reporter :app.loggers.database/reporter + :app.loggers.metrics/reporter :app.worker/cron :app.worker/dispatcher) (remove-job-runners)) @@ -524,6 +525,20 @@ (let [params (prepare-rpc-params data)] (try-on! (method-fn params))))) +(defn command-raw! + "Like `command!`, but returns the raw result without unwrapping it, so a test + can read the result metadata (for example the audit props a command sets on a + nil result). The HTTP layer unwraps it the same way in production." + [{:keys [::type] :as data}] + (let [[_ method-fn] (get-in *system* [:app.rpc/methods type])] + (when-not method-fn + (ex/raise :type :assertion + :code :rpc-method-not-found + :hint (str/ffmt "rpc method '%' not found" (name type)))) + + (let [params (prepare-rpc-params data)] + (method-fn params)))) + (defn management-command! [{:keys [::type] :as data}] (let [[_ method-fn] (get-in *system* [:app.rpc/management-methods type])] diff --git a/backend/test/backend_tests/loggers_metrics_test.clj b/backend/test/backend_tests/loggers_metrics_test.clj new file mode 100644 index 0000000000..984ad3b0f8 --- /dev/null +++ b/backend/test/backend_tests/loggers_metrics_test.clj @@ -0,0 +1,135 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS INC Sucursal en España SL + +(ns backend-tests.loggers-metrics-test + "The reporter that turns log records into a counter." + (:require + [app.common.logging :as l] + [app.loggers.metrics] ;; registers the integrant methods of the reporter key + [app.main :as main] + [app.metrics :as mtx] + [clojure.test :as t] + [integrant.core :as ig]) + (:import + io.prometheus.client.CollectorRegistry)) + +(def ^:private logger + "The logger of a record is the namespace of the code that logs, and that is + this namespace." + (str *ns*)) + +(defn- init-metrics + [] + (ig/init-key :app.metrics/metrics {:default main/default-metrics})) + +(defn- init-reporter + [metrics] + (ig/init-key :app.loggers.metrics/reporter {::mtx/metrics metrics})) + +(defn- halt-reporter! + [reporter] + (ig/halt-key! :app.loggers.metrics/reporter reporter)) + +(defn- sample + "Value of the counter for one logger and level, read from the registry. Nil + when that combination has never been published." + [metrics logger level] + (.getSampleValue ^CollectorRegistry (mtx/get-registry metrics) + "penpot_log_events_total" + (into-array String ["logger" "level"]) + (into-array String [logger level]))) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; COUNTING +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(t/deftest warn-and-error-are-counted-by-logger-and-level + (let [metrics (init-metrics) + reporter (init-reporter metrics)] + (try + ;; `:sync?` so the record is emitted on this thread and the assertion + ;; does not depend on the logging executor. + (l/warn :hint "una prueba" ::l/sync? true) + (l/error :hint "otra prueba" ::l/sync? true) + + (t/is (= 1.0 (sample metrics logger "warn"))) + (t/is (= 1.0 (sample metrics logger "error"))) + + (l/error :hint "y otra mas" ::l/sync? true) + (t/is (= 2.0 (sample metrics logger "error"))) + + (finally + (halt-reporter! reporter))))) + +(t/deftest info-is-not-counted + (let [metrics (init-metrics) + reporter (init-reporter metrics)] + (try + (l/info :hint "un mensaje normal" ::l/sync? true) + (t/is (nil? (sample metrics logger "info"))) + + (finally + (halt-reporter! reporter))))) + +(t/deftest fatal-is-counted + (let [metrics (init-metrics) + reporter (init-reporter metrics)] + (try + ;; No macro of the logging emits `:fatal`; this is the only way a record + ;; gets that level. + (l/log :fatal :hint "algo muy malo" ::l/sync? true) + (t/is (= 1.0 (sample metrics logger "fatal"))) + + (finally + (halt-reporter! reporter))))) + +(t/deftest halting-the-reporter-stops-counting + (let [metrics (init-metrics) + reporter (init-reporter metrics)] + + (halt-reporter! reporter) + + (l/error :hint "despues de parar" ::l/sync? true) + (t/is (nil? (sample metrics logger "error"))))) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; SAFETY +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(t/deftest a-failing-publication-does-not-break-the-logging + ;; The watcher fires from the `swap!` of the log record, inside the logging + ;; path. With `::l/sync?` an exception here would reach whoever was logging; + ;; on the async path it would end in the executor's uncaught handler, which + ;; is outside the application's logging, taking that log line out of Loki and + ;; out of this counter. + (let [metrics (init-metrics) + reporter (init-reporter metrics)] + (try + (with-redefs [mtx/run! (fn [& _] (throw (ex-info "publicando mal" {})))] + (t/is (nil? (l/error :hint "no debe propagar" ::l/sync? true)))) + + (finally + (halt-reporter! reporter))))) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; DEFINITION +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(t/deftest the-log-events-metric-is-defined + (let [metrics (init-metrics)] + (t/is (= ["logger" "level"] + (:app.metrics.definition/labels (mtx/get-collector metrics :log-events)))))) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; WIRING +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(t/deftest the-system-declares-the-reporter + ;; The reporter hooks itself to the logging, so declaring it in the system is + ;; all the wiring there is. The tests of the suite remove it, so this is the + ;; only place that pins it. + (t/is (= {::mtx/metrics (ig/ref ::mtx/metrics)} + (:app.loggers.metrics/reporter main/system-config)))) diff --git a/backend/test/backend_tests/loggers_webhooks_test.clj b/backend/test/backend_tests/loggers_webhooks_test.clj index fd3a1bd5bf..e2c540119c 100644 --- a/backend/test/backend_tests/loggers_webhooks_test.clj +++ b/backend/test/backend_tests/loggers_webhooks_test.clj @@ -9,6 +9,7 @@ [app.common.time :as ct] [app.common.transit :as transit] [app.common.uuid :as uuid] + [app.config :as cf] [app.db :as db] [app.http :as http] [app.jobs :as jobs] @@ -20,6 +21,12 @@ (t/use-fixtures :once th/state-init) (t/use-fixtures :each th/database-reset) +(defn decode-row + [{:keys [props context] :as row}] + (cond-> row + (db/pgobject? props) (assoc :props (db/decode-transit-pgobject props)) + (db/pgobject? context) (assoc :context (db/decode-transit-pgobject context)))) + (t/deftest process-event-handler-with-no-webhooks (with-mocks [submit-mock {:target 'app.jobs/submit :return nil}] (let [prof (th/create-profile* 1 {:is-active true}) diff --git a/backend/test/backend_tests/objects_gc_test.clj b/backend/test/backend_tests/objects_gc_test.clj index 99a1314d13..6696a7f2f0 100644 --- a/backend/test/backend_tests/objects_gc_test.clj +++ b/backend/test/backend_tests/objects_gc_test.clj @@ -8,11 +8,14 @@ (:require [app.common.logging :as l] [app.common.time :as ct] + [app.config :as cf] [app.db :as db] + [app.rpc :as-alias rpc] [app.storage :as sto] [app.tasks.objects-gc :as ogc] [backend-tests.helpers :as th] - [clojure.test :as t])) + [clojure.test :as t] + [mockery.core :refer [with-mocks]])) (t/use-fixtures :once th/state-init) (t/use-fixtures :each th/database-reset) @@ -123,3 +126,119 @@ (t/testing "the run finishes" (t/is (pos? (:processed result)))))) + +(defn- submitted-events + [audit-mock] + (->> (:call-args-list @audit-mock) + (map second))) + +(defn- events-named + [audit-mock event-name] + (filter #(= event-name (:name %)) + (submitted-events audit-mock))) + +(t/deftest gc-emits-hard-delete-objects-for-teams + ;; When the GC physically deletes a team, the projection must drop its row, + ;; so the event carries the purged ids. + (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] + (with-redefs [cf/flags (conj cf/flags :audit-log)] + (let [profile (th/create-profile* 1 {:is-active true}) + team (th/create-team* 71 {:profile-id (:id profile)})] + (th/command! {::th/type :delete-team + ::rpc/profile-id (:id profile) + :id (:id team)}) + ;; Run the queued cascade so the default project is marked deleted too, + ;; then purge everything the way production does. + (th/run-pending-jobs) + (th/reset-mock! audit-mock) + (th/run-task! :objects-gc {:skip-delay true}) + (t/is (nil? (th/db-get :team {:id (:id team)} {::db/remove-deleted false}))) + (let [events (filter #(contains? (:props %) :team-ids) + (events-named audit-mock "hard-delete-objects"))] + (t/is (= 1 (count events))) + (let [event (first events)] + (t/is (= "command" (:type event))) + (t/is (= [(:id team)] (get-in event [:props :team-ids]))) + (t/is (= "objects-gc" (get-in event [:context :triggered-by]))))))))) + +(t/deftest gc-emits-hard-delete-objects-for-projects + (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] + (with-redefs [cf/flags (conj cf/flags :audit-log)] + (let [profile (th/create-profile* 1 {:is-active true}) + team (th/create-team* 72 {:profile-id (:id profile)}) + project (th/create-project* 72 {:profile-id (:id profile) + :team-id (:id team)})] + (th/command! {::th/type :delete-project + ::rpc/profile-id (:id profile) + :id (:id project)}) + (th/run-pending-jobs) + (th/reset-mock! audit-mock) + (th/run-task! :objects-gc {:skip-delay true}) + (t/is (nil? (th/db-get :project {:id (:id project)} {::db/remove-deleted false}))) + (let [events (filter #(contains? (:props %) :project-ids) + (events-named audit-mock "hard-delete-objects"))] + (t/is (= 1 (count events))) + (let [event (first events)] + (t/is (= "command" (:type event))) + (t/is (= [(:id project)] (get-in event [:props :project-ids]))) + (t/is (= "objects-gc" (get-in event [:context :triggered-by]))))))))) + +(t/deftest gc-emits-hard-delete-objects-for-files + (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] + (with-redefs [cf/flags (conj cf/flags :audit-log)] + (let [profile (th/create-profile* 1 {:is-active true}) + team (th/create-team* 73 {:profile-id (:id profile)}) + project (th/create-project* 73 {:profile-id (:id profile) + :team-id (:id team)}) + file (th/create-file* 73 {:profile-id (:id profile) + :project-id (:id project)})] + (th/command! {::th/type :delete-file + ::rpc/profile-id (:id profile) + :id (:id file)}) + (th/run-pending-jobs) + (th/reset-mock! audit-mock) + (th/run-task! :objects-gc {:skip-delay true}) + (t/is (nil? (th/db-get :file {:id (:id file)} {::db/remove-deleted false}))) + (let [events (filter #(contains? (:props %) :file-ids) + (events-named audit-mock "hard-delete-objects"))] + (t/is (= 1 (count events))) + (let [event (first events)] + (t/is (= "command" (:type event))) + (t/is (= [(:id file)] (get-in event [:props :file-ids]))) + (t/is (= "objects-gc" (get-in event [:context :triggered-by]))))))))) + +(t/deftest gc-emits-no-event-without-audit-log-flag + ;; The GC must stay silent when the audit log is off, even though the row is + ;; still purged. + (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] + (let [profile (th/create-profile* 1 {:is-active true}) + team (th/create-team* 74 {:profile-id (:id profile)})] + (th/command! {::th/type :delete-team + ::rpc/profile-id (:id profile) + :id (:id team)}) + (th/run-pending-jobs) + (th/reset-mock! audit-mock) + (th/run-task! :objects-gc {:skip-delay true}) + (t/is (nil? (th/db-get :team {:id (:id team)} {::db/remove-deleted false}))) + (t/is (empty? (:call-args-list @audit-mock)))))) + +(t/deftest gc-emits-events-only-for-committed-chunks + (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] + (with-redefs [cf/flags (conj cf/flags :audit-log)] + (let [profile (th/create-profile* 1) + rolled-back (create-deleted-file! profile 1 (ct/in-past "30m")) + blocked (create-deleted-file! profile 2 (ct/in-past "20m")) + clean (create-deleted-file! profile 3 (ct/in-past "10m"))] + (block-file! blocked) + (th/reset-mock! audit-mock) + (let [result (th/run-task! :objects-gc {:chunk-size 2}) + events (filter #(contains? (:props %) :file-ids) + (events-named audit-mock "hard-delete-objects"))] + (t/testing "the failed chunk keeps both files" + (t/is (some? (find-file (:id rolled-back)))) + (t/is (some? (find-file (:id blocked))))) + (t/testing "the next chunk commits and reports only its deleted file" + (t/is (nil? (find-file (:id clean)))) + (t/is (pos? (:processed result))) + (t/is (= 1 (count events))) + (t/is (= [(:id clean)] (get-in (first events) [:props :file-ids]))))))))) diff --git a/backend/test/backend_tests/rpc_audit_test.clj b/backend/test/backend_tests/rpc_audit_test.clj index 50ee79f2e8..7c99c0093c 100644 --- a/backend/test/backend_tests/rpc_audit_test.clj +++ b/backend/test/backend_tests/rpc_audit_test.clj @@ -8,15 +8,18 @@ (:require [app.common.pprint :as pp] [app.common.time :as ct] + [app.common.transit :as transit] [app.common.uuid :as uuid] [app.config :as cf] [app.db :as db] + [app.jobs :as-alias jobs] [app.loggers.audit :as audit] [app.rpc :as-alias rpc] [app.rpc.climit :as climit] [app.util.services :as sv] [backend-tests.helpers :as th] [clojure.test :as t] + [mockery.core :refer [with-mocks]] [yetti.request])) (t/use-fixtures :once th/state-init) @@ -329,6 +332,60 @@ (t/is (= day-now (:created-at row2))) (t/is (= day-now (:tracked-at row2)))))))) +(t/deftest backend-process-event-forwards-id-to-webhook-task + ;; The `webhook/trigger` row links back to its origin event through + ;; props `:event-id`, so `process-event` must forward the stored row + ;; id to the `:process-webhook-event` worker task. Regression test: + ;; the id used to be generated inside `append-audit-entry` without + ;; ever reaching the worker params, so `:event-id` always landed nil. + (with-mocks [submit-mock {:target 'app.jobs/submit :return nil}] + (with-redefs [cf/flags #{:audit-log :webhooks}] + (let [prof (th/create-profile* 1 {:is-active true}) + event {:type "action" + :name "delete-project" + :profile-id (:id prof) + :props {:team-id (:default-team-id prof)} + :context {:version "2.0.0" :initiator "app"} + :tracked-at (ct/now) + :created-at (ct/now) + :source "backend" + :app.loggers.webhooks/event? true}] + (audit/submit* th/*system* event) + (let [[row :as rows] (->> (th/db-exec! ["select * from audit_log"]) + (mapv decode-row))] + (t/is (= 1 (count rows))) + (t/is (= 1 (:call-count @submit-mock))) + (let [params (-> @submit-mock :call-args-list first second ::jobs/params) + task-event (transit/decode-str (:event-blob params))] + (t/is (some? (:id task-event))) + (t/is (= (:id row) (:id task-event))))))))) + +(t/deftest backend-process-event-telemetry-row-gets-own-id + ;; When both `:audit-log` and `:telemetry` are active, the shadow + ;; telemetry row must not reuse the full row id (shared PK), while + ;; the webhook task still carries the full row id. + (with-mocks [submit-mock {:target 'app.jobs/submit :return nil}] + (with-redefs [cf/flags #{:audit-log :telemetry :webhooks}] + (let [prof (th/create-profile* 1 {:is-active true}) + event {:type "action" + :name "delete-project" + :profile-id (:id prof) + :props {:team-id (:default-team-id prof)} + :context {:version "2.0.0" :initiator "app"} + :tracked-at (ct/now) + :created-at (ct/now) + :source "backend" + :app.loggers.webhooks/event? true}] + (audit/submit* th/*system* event) + (let [[row1 row2 :as rows] (->> (th/db-exec! ["select * from audit_log order by source"]) + (mapv decode-row))] + (t/is (= 2 (count rows))) + (t/is (not= (:id row1) (:id row2))) + (let [params (-> @submit-mock :call-args-list first second ::jobs/params) + task-event (transit/decode-str (:event-blob params))] + (t/is (= (:id (first (filter #(= "backend" (:source %)) rows))) + (:id task-event))))))))) + (t/deftest push-events-disabled-when-no-flags-and-no-telemetry ;; When neither audit-log nor telemetry is enabled, no rows should ;; be stored. @@ -651,17 +708,22 @@ (defn- prepare-event "Call prepare-rpc-event with a bare request, returning the built event." - [caller result] - (let [mdata {::sv/name "test-cmd"} - params {::rpc/profile-id caller - ::rpc/request-id (uuid/next) - ::rpc/request-at (ct/now)} - mock-req (reify - yetti.request/IRequest - (get-header [_ _] nil) - (remote-addr [_] "127.0.0.1")) - params (with-meta params {:app.http/request mock-req})] - (audit/prepare-rpc-event th/*system* mdata params result))) + ([caller result] + (prepare-event caller "test-cmd" result)) + ([caller cmd-name result] + (let [mdata {::sv/name cmd-name} + params {::rpc/profile-id caller + ::rpc/request-id (uuid/next) + ::rpc/request-at (ct/now)} + mock-req (reify + yetti.request/IRequest + (get-header [_ _] nil) + (remote-addr [_] "127.0.0.1")) + params (with-meta params {:app.http/request mock-req})] + ;; Production always resolves commands with `::module` set ("main" or + ;; "management"): the event name (and the read exclusion) depends on it. + (audit/prepare-rpc-event (assoc th/*system* ::rpc/module "main") + mdata params result)))) (t/deftest prepare-rpc-event-ignores-profile-id-from-result ;; An audit event belongs to the caller, never to whatever `:profile-id` @@ -696,6 +758,76 @@ (t/is (= uuid/zero (:profile-id (prepare-event nil {:some-data "value"}))))) +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; PREPARE-RPC-EVENT TEAM-ID ENRICHMENT +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(t/deftest prepare-rpc-event-enriches-team-id-from-project-id + ;; An event that carries a project id resolves its team locally, so the + ;; downstream consumers do not need to join production tables. + (let [profile (th/create-profile* 1 {:is-active true}) + team (th/create-team* 10 {:profile-id (:id profile)}) + project (th/create-project* 10 {:profile-id (:id profile) + :team-id (:id team)}) + result (with-meta {} {::audit/props {:project-id (:id project)}}) + event (prepare-event (:id profile) result)] + (t/is (= (:id team) (get-in event [:props :team-id]))))) + +(t/deftest prepare-rpc-event-enriches-team-id-from-file-id + (let [profile (th/create-profile* 1 {:is-active true}) + team (th/create-team* 11 {:profile-id (:id profile)}) + project (th/create-project* 11 {:profile-id (:id profile) + :team-id (:id team)}) + file (th/create-file* 11 {:profile-id (:id profile) + :project-id (:id project)}) + result (with-meta {} {::audit/props {:file-id (:id file)}}) + event (prepare-event (:id profile) result)] + (t/is (= (:id team) (get-in event [:props :team-id]))))) + +(t/deftest prepare-rpc-event-keeps-existing-team-id + ;; The resolver never overwrites a team id the command already set. + (let [profile (th/create-profile* 1 {:is-active true}) + team (th/create-team* 12 {:profile-id (:id profile)}) + project (th/create-project* 12 {:profile-id (:id profile) + :team-id (:id team)}) + result (with-meta {} {::audit/props {:team-id (:id team) + :project-id (:id project)}}) + event (prepare-event (:id profile) result)] + (t/is (= (:id team) (get-in event [:props :team-id]))))) + +(t/deftest prepare-rpc-event-does-not-resolve-without-project-or-file + ;; Read commands put the entity id under `:id`; enriching those would add a + ;; query to every read event, so the resolver must leave them untouched. + (let [profile (th/create-profile* 1 {:is-active true}) + result (with-meta {} {::audit/props {:id (uuid/next)}}) + event (prepare-event (:id profile) result)] + (t/is (not (contains? (:props event) :team-id))))) + +(t/deftest prepare-rpc-event-resolves-default-team-id + ;; The resolver adds the team id whenever it can resolve it, including the + ;; personal (default) team. + (let [profile (th/create-profile* 1 {:is-active true}) + result (with-meta {} {::audit/props {:project-id (:default-project-id profile)}}) + event (prepare-event (:id profile) result)] + (t/is (= (:default-team-id profile) (get-in event [:props :team-id]))))) + +(t/deftest prepare-rpc-event-skips-team-id-resolution-for-reads + ;; Read events carry no consumer state: resolving their team would add a + ;; query to every read, so `get-*` names (plus the known non-get reads) + ;; are left untouched even when they carry a project or file id. + (let [profile (th/create-profile* 1 {:is-active true}) + team (th/create-team* 13 {:profile-id (:id profile)}) + project (th/create-project* 13 {:profile-id (:id profile) + :team-id (:id team)}) + file (th/create-file* 13 {:profile-id (:id profile) + :project-id (:id project)})] + (doseq [cmd-name ["get-file-fragment" "get-comments" "has-file-libraries"] + props [{:file-id (:id file)} {:project-id (:id project)}]] + (let [result (with-meta {} {::audit/props props}) + event (prepare-event (:id profile) cmd-name result)] + (t/is (not (contains? (:props event) :team-id)) + (str cmd-name " with " (keys props) " must not resolve team-id")))))) + ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; PREPARE-RPC-EVENT PROFILE-ID COERCION ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; diff --git a/backend/test/backend_tests/rpc_binfile_test.clj b/backend/test/backend_tests/rpc_binfile_test.clj index b1a144ab70..ccdcc7fafa 100644 --- a/backend/test/backend_tests/rpc_binfile_test.clj +++ b/backend/test/backend_tests/rpc_binfile_test.clj @@ -6,13 +6,17 @@ (ns backend-tests.rpc-binfile-test (:require + [app.binfile.v3 :as bf.v3] [app.common.schema :as sm] [app.common.uuid :as uuid] + [app.config :as cf] + [app.db :as db] [app.rpc :as-alias rpc] [app.rpc.commands.binfile :as binfile] [backend-tests.helpers :as th] [clojure.test :as t] - [datoteka.fs :as fs])) + [datoteka.fs :as fs] + [mockery.core :refer [with-mocks]])) (t/use-fixtures :once th/state-init) (t/use-fixtures :each th/database-reset) @@ -72,3 +76,38 @@ ;; Version 4 should be rejected (t/is (false? (validator (assoc base-params :version 4))) "version 4 should be rejected"))) + +(t/deftest import-binfile-audit-events-carry-entity-ids + ;; The imported files are created inside the importer, so the command emits a + ;; create-file event per file for a local projection. + (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] + (with-redefs [cf/flags (conj cf/flags :audit-log)] + (let [profile (th/create-profile* 1 {:is-active true}) + team (th/create-team* 51 {:profile-id (:id profile)}) + project (th/create-project* 51 {:profile-id (:id profile) + :team-id (:id team)}) + ;; A real row: `submit-file-created-event` reads the light + ;; columns back, so a phantom id would emit no event. + file (th/create-file* 99 {:profile-id (:id profile) + :project-id (:id project)}) + file-id (:id file)] + (with-redefs [bf.v3/import-files! (fn [_cfg] {:file-ids [file-id]})] + (#'binfile/import-binfile + th/*system* + {::rpc/profile-id (:id profile) + :profile-id (:id profile) + :project-id (:id project) + :version 3 + :name "imported" + :file {:filename "package.penpot" + :path (th/tempfile "backend_tests/test_files/svg-attrs-camel-case.penpot") + :mtype "application/zip" + :size 1}})) + + (let [events (->> (:call-args-list @audit-mock) + (map second) + (filter #(= "create-file" (:name %))))] + (t/is (= 1 (count events))) + (t/is (= file-id (get-in (first events) [:props :id]))) + (t/is (= (:id project) (get-in (first events) [:props :project-id]))) + (t/is (= (:id team) (get-in (first events) [:props :team-id])))))))) diff --git a/backend/test/backend_tests/rpc_comment_test.clj b/backend/test/backend_tests/rpc_comment_test.clj index ea64723545..0c58687751 100644 --- a/backend/test/backend_tests/rpc_comment_test.clj +++ b/backend/test/backend_tests/rpc_comment_test.clj @@ -478,3 +478,32 @@ :content "member comment" :frame-id uuid/zero})] (t/is (th/success? out)))))) + +(t/deftest comment-audit-events-carry-team-id + ;; Comment events carry the file id, so the resolver could find the team; the + ;; command already has the file with its team id and sets it without a join. + (with-mocks [mock {:target 'app.config/get + :return (th/config-get-mock + {:quotes-teams-per-profile 200})}] + (let [profile (th/create-profile* 1 {:is-active true}) + team (th/create-team* 1 {:profile-id (:id profile)}) + project (th/create-project* 1 {:team-id (:id team) + :profile-id (:id profile)}) + file (th/create-file* 1 {:profile-id (:id profile) + :project-id (:id project)}) + page-id (get-in file [:data :pages 0]) + thread (th/command! {::th/type :create-comment-thread + ::rpc/profile-id (:id profile) + :file-id (:id file) + :page-id page-id + :position (gpt/point 0) + :content "hello" + :frame-id uuid/zero}) + comment (th/command! {::th/type :create-comment + ::rpc/profile-id (:id profile) + :thread-id (get-in thread [:result :id]) + :content "comment"})] + (t/is (= (:id team) + (get-in (meta (:result thread)) [:app.loggers.audit/props :team-id]))) + (t/is (= (:id team) + (get-in (meta (:result comment)) [:app.loggers.audit/props :team-id])))))) diff --git a/backend/test/backend_tests/rpc_file_security_test.clj b/backend/test/backend_tests/rpc_file_security_test.clj new file mode 100644 index 0000000000..15ac56c53d --- /dev/null +++ b/backend/test/backend_tests/rpc_file_security_test.clj @@ -0,0 +1,368 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS INC Sucursal en España SL + +(ns backend-tests.rpc-file-security-test + "Tests for the nexus file-security backend prerequisite: the + `get-file-for-analysis` management method, the semantic projection, + and the audit prop coverage the `:local` consumer schedules from." + (:require + [app.binfile.common :as bfc] + [app.common.features :as cfeat] + [app.common.time :as ct] + [app.common.types.shape :as cts] + [app.common.uuid :as uuid] + [app.loggers.audit :as audit] + [app.rpc :as-alias rpc] + [app.rpc.management.files :as mgmt.files] + [app.util.services :as sv] + [backend-tests.helpers :as th] + [clojure.test :as t] + [yetti.request])) + +(t/use-fixtures :once th/state-init) +(t/use-fixtures :each th/database-reset) + +;; --- Projection unit tests (pure, no database) + +(def ^:private page-id (uuid/random)) +(def ^:private frame-id (uuid/random)) +(def ^:private text-id (uuid/random)) +(def ^:private link-id (uuid/random)) + +(defn- test-data + [objects] + {:pages [page-id] + :pages-index {page-id {:id page-id + :name "Page 1" + :objects objects}}}) + +(t/deftest projection-empty-file + (let [projection (mgmt.files/file->projection (test-data {}))] + (t/is (= 1 (:version projection))) + (t/is (= [{:id (str page-id) :name "Page 1"}] (:pages projection))) + (t/is (= [] (:frames projection))) + (t/is (= {:items [] :truncated? false} (:texts projection))) + (t/is (= {:items [] :truncated? false} (:links projection))) + (t/is (= {:items [] :truncated? false} (:interactions projection))))) + +(t/deftest projection-picks-frames-text-links-and-interactions + (let [data (test-data + {frame-id {:id frame-id :name "Frame" :type :frame} + text-id {:id text-id :name "Text" :type :text + :content {:type "root" + :children [{:type "paragraph" + :children [{:text "hello"} + {:text " world"}]}]}} + link-id {:id link-id :name "Button" :type :rect + :interactions [{:event-type :click + :action-type :open-url + :url "https://phish.example/login"} + {:event-type :click + :action-type :navigate + :destination page-id} + ;; No event, cannot trigger: + ;; collected as link, never + ;; as interaction. + {:action-type :open-url + :url "https://quiet.example"}]}}) + projection (mgmt.files/file->projection data)] + (t/is (= [{:id (str frame-id) :page-id (str page-id) :name "Frame"}] + (:frames projection))) + (t/is (= {:items [{:shape-id (str text-id) + :page-id (str page-id) + :text "hello world"}] + :truncated? false} + (:texts projection))) + (t/is (= {:items [{:shape-id (str link-id) + :page-id (str page-id) + :url "https://phish.example/login"} + {:shape-id (str link-id) + :page-id (str page-id) + :url "https://quiet.example"}] + :truncated? false} + (:links projection))) + (t/is (= {:items [{:shape-id (str link-id) + :page-id (str page-id) + :event "click" + :action "open-url" + :target "https://phish.example/login"} + {:shape-id (str link-id) + :page-id (str page-id) + :event "click" + :action "navigate" + :target (str page-id)}] + :truncated? false} + (:interactions projection))))) + +(t/deftest projection-skips-blank-text-and-corrupt-content + (let [bad-id (uuid/random) + projection (mgmt.files/file->projection + (test-data + {text-id {:id text-id :type :text + :content {:type "root" + :children [{:type "paragraph" + :children [{:text " "}]}]}} + bad-id {:id bad-id :type :text + :content {:type "root" :children 42}}}))] + ;; Blank text contributes nothing; corrupt content skips the shape + ;; instead of failing the file. + (t/is (= {:items [] :truncated? false} (:texts projection))))) + +(t/deftest projection-truncates-links + (let [objects (into {} + (map (fn [i] + (let [id (uuid/random)] + [id {:id id :type :rect + :interactions [{:event-type :click + :action-type :open-url + :url (str "https://example.com/" i)}]}]))) + (range (inc mgmt.files/max-links))) + projection (mgmt.files/file->projection (test-data objects))] + (t/is (= mgmt.files/max-links (count (get-in projection [:links :items])))) + (t/is (true? (get-in projection [:links :truncated?]))) + (t/is (false? (get-in projection [:interactions :truncated?]))))) + +;; --- Management method (through the worker stack) + +(def text-content + {:type "root" + :children [{:type "paragraph-set" + :children [{:type "paragraph" + :children [{:text "cheap meds"}]}]}]}) + +(defn- create-file! + [profile-id project-id] + (let [out (th/command! {::th/type :create-file + ::rpc/profile-id profile-id + :project-id project-id + :name "Shop" + :is-shared false})] + (t/is (nil? (:error out))) + (get-in out [:result :id]))) + +(defn- file-page-id + [file-id] + (get-in (bfc/get-file th/*system* file-id) [:data :pages 0])) + +(defn- update-file! + [profile-id file-id page-id changes revn] + (let [out (th/command! {::th/type :update-file + ::rpc/profile-id profile-id + :id file-id + :session-id (uuid/random) + :revn revn + :vern 0 + :features cfeat/supported-features + :changes changes})] + (t/is (nil? (:error out))) + (:result out))) + +(defn- add-text-shape-change + [page-id shape-id] + [{:type :add-obj + :page-id page-id + :id shape-id + :parent-id uuid/zero + :frame-id uuid/zero + :components-v2 true + :obj (cts/setup-shape + {:id shape-id + :name "Buy now" + :frame-id uuid/zero + :parent-id uuid/zero + :type :text + :content text-content})}]) + +(t/deftest get-file-for-analysis-returns-projection + (let [profile (th/create-profile* 1 {:is-active true}) + team (th/create-team* 1 {:profile-id (:id profile)}) + project (th/create-project* 1 {:profile-id (:id profile) + :team-id (:id team)}) + file-id (create-file! (:id profile) (:id project)) + page-id (file-page-id file-id)] + ;; One edit so revn moves past zero and the file has content. + (update-file! (:id profile) file-id page-id + (add-text-shape-change page-id (uuid/random)) + 0) + (let [out (th/management-command! {::th/type :get-file-for-analysis + :file-id file-id})] + (t/is (nil? (:error out))) + (let [result (:result out)] + (t/is (= file-id (:id result))) + (t/is (= 1 (:revn result))) + (t/is (some? (:modified-at result))) + (t/is (= (:id project) (:project-id result))) + (t/is (= (:id team) (:team-id result))) + (t/is (false? (:has-share-link result))) + (t/is (= mgmt.files/projection-version (:projection-version result))) + (t/is (= "cheap meds" + (-> result :projection :texts :items first :text))))) + ;; A share link flips the share state, no re-edit needed. + (th/command! {::th/type :create-share-link + ::rpc/profile-id (:id profile) + :file-id file-id + :pages #{page-id} + :who-comment "all" + :who-inspect "all"}) + (let [out (th/management-command! {::th/type :get-file-for-analysis + :file-id file-id})] + (t/is (nil? (:error out))) + (t/is (true? (:has-share-link (:result out))))))) + +(t/deftest get-file-for-analysis-rejects-unknown-file + (let [out (th/management-command! {::th/type :get-file-for-analysis + :file-id (uuid/random)})] + (t/is (some? (:error out))) + (t/is (= :not-found (th/ex-type (:error out)))))) + +;; --- Audit prop coverage (through the commands, into real events) +;; +;; NOTE: the test system neutralizes `audit/submit`, so no audit row +;; is ever written. The observable contract of a command is the +;; metadata it sets on its result, which `prepare-rpc-event` turns +;; into the stored event: feed one into the other with the real +;; method metadata and assert the final props. + +(def mock-request + (reify + yetti.request/IRequest + (get-header [_ _] nil) + (remote-addr [_] "127.0.0.1"))) + +(defn- event-for + "Run a main RPC command and return the audit event it would emit." + [type params] + (let [[mdata _] (get-in th/*system* [:app.rpc/methods type]) + result (th/command-raw! (assoc params ::th/type type)) + params (-> params + (assoc ::rpc/request-id (uuid/next) + ::rpc/request-at (ct/now)) + (with-meta {:app.http/request mock-request})) + ;; In production the caller assocs the module before + ;; preparing the event; without it every name would come out + ;; with a leading dash. + cfg (assoc th/*system* ::rpc/module "main")] + (audit/prepare-rpc-event cfg mdata params result))) + +(t/deftest get-file-for-analysis-emits-audit-event + ;; Content reads stay traceable: with the skip gone, the call flows + ;; through `wrap-audit` like any other management method. `wrap-audit` + ;; itself is fixed at system init (no audit flags there), so assert the + ;; metadata flip plus the event shape through `prepare-rpc-event` with + ;; the real method metadata. + (let [profile (th/create-profile* 1 {:is-active true}) + team (th/create-team* 1 {:profile-id (:id profile)}) + project (th/create-project* 1 {:profile-id (:id profile) + :team-id (:id team)}) + file-id (create-file! (:id profile) (:id project)) + [mdata _] (get-in th/*system* [:app.rpc/management-methods :get-file-for-analysis])] + (t/is (not (::audit/skip mdata))) + (let [out (th/management-command! {::th/type :get-file-for-analysis + :file-id file-id}) + result (:result out) + params (-> {:file-id file-id} + (assoc ::rpc/request-id (uuid/next) + ::rpc/request-at (ct/now)) + (with-meta {:app.http/request mock-request})) + cfg (assoc th/*system* ::rpc/module "management") + event (audit/prepare-rpc-event cfg mdata params result)] + (t/is (nil? (:error out))) + (t/is (= "management-get-file-for-analysis" (:name event))) + (t/is (= file-id (get-in event [:props :file-id]))) + (t/is (= (:id team) (get-in event [:props :team-id])))))) + +(t/deftest file-events-carry-scheduling-props + (let [profile (th/create-profile* 2 {:is-active true}) + team (th/create-team* 2 {:profile-id (:id profile)}) + project (th/create-project* 2 {:profile-id (:id profile) + :team-id (:id team)}) + base {::rpc/profile-id (:id profile)}] + (t/testing "create-file" + (let [event (event-for :create-file + (assoc base + :project-id (:id project) + :name "Shop" + :is-shared false)) + props (:props event)] + (t/is (= "create-file" (:name event))) + (t/is (some? (:id props))) + (t/is (= "Shop" (:name props))) + (t/is (some? (:created-at props))) + (t/is (some? (:modified-at props))) + (t/is (= 0 (:revn props))))) + (t/testing "update-file" + ;; A file to edit, created through the command so the event + ;; below belongs to a real row. + (let [created (event-for :create-file + (assoc base + :project-id (:id project) + :name "Edit me" + :is-shared false)) + file-id (get-in created [:props :id]) + page-id (file-page-id file-id) + shape-id (uuid/random) + event (event-for :update-file + (assoc base + :id file-id + :session-id (uuid/random) + :revn 0 + :vern 0 + :features cfeat/supported-features + :changes [{:type :add-obj + :page-id page-id + :id shape-id + :parent-id uuid/zero + :frame-id uuid/zero + :components-v2 true + :obj (cts/setup-shape + {:id shape-id + :name "rect" + :frame-id uuid/zero + :parent-id uuid/zero + :type :rect})}])) + props (:props event)] + (t/is (= "update-file" (:name event))) + (t/is (= file-id (:id props))) + (t/is (= 1 (:revn props))) + (t/is (some? (:modified-at props))))) + (t/testing "share-link create and delete" + (let [created (event-for :create-file + (assoc base + :project-id (:id project) + :name "Shared" + :is-shared false)) + file-id (get-in created [:props :id]) + page-id (file-page-id file-id) + event (event-for :create-share-link + (assoc base + :file-id file-id + :pages #{page-id} + :who-comment "all" + :who-inspect "all")) + props (:props event) + slink-id (:share-id props)] + (t/is (= "create-share-link" (:name event))) + (t/is (= file-id (:file-id props))) + (t/is (some? slink-id)) + (let [event (event-for :delete-share-link + (assoc base :id slink-id)) + props (:props event)] + (t/is (= "delete-share-link" (:name event))) + (t/is (= file-id (:file-id props))) + (t/is (= slink-id (:share-id props)))))) + (t/testing "delete-file" + (let [created (event-for :create-file + (assoc base + :project-id (:id project) + :name "Doomed" + :is-shared false)) + file-id (get-in created [:props :id]) + event (event-for :delete-file + (assoc base :id file-id)) + props (:props event)] + (t/is (= "delete-file" (:name event))) + (t/is (= file-id (:id props))) + (t/is (some? (:deleted-at props))))))) diff --git a/backend/test/backend_tests/rpc_file_test.clj b/backend/test/backend_tests/rpc_file_test.clj index 305108003c..000bebe8d9 100644 --- a/backend/test/backend_tests/rpc_file_test.clj +++ b/backend/test/backend_tests/rpc_file_test.clj @@ -6,11 +6,16 @@ (ns backend-tests.rpc-file-test (:require + [app.binfile.common :as bfc] [app.common.features :as cfeat] + [app.common.files.changes-builder :as pcb] + [app.common.files.tokens :as cfo] [app.common.pprint :as pp] [app.common.thumbnails :as thc] [app.common.time :as ct] [app.common.types.shape :as cts] + [app.common.types.tokens-lib :as ctob] + [app.common.types.tokens-status :as ctos] [app.common.uuid :as uuid] [app.config :as cf] [app.db :as db] @@ -2254,6 +2259,93 @@ (get-in out [:result :data :pages-index f-page :objects inst-id :component-file]))))))))) +(defn- prepare-tokens-library + "A shared library with a tokens lib, a file linked to it that uses it + as its tokens source, and another linked file that does not." + [profile] + (let [set-id (uuid/next) + theme-id (uuid/next) + tokens-lib (-> (ctob/make-tokens-lib) + (ctob/add-set (ctob/make-token-set :id set-id :name "test-set")) + (ctob/add-theme (ctob/make-token-theme :id theme-id + :name "test-theme" + :sets #{"test-set"})) + (ctob/add-token set-id (ctob/make-token :name "token-radius" + :type :border-radius + :value 10))) + params {:profile-id (:id profile) + :project-id (:default-project-id profile)} + library (th/create-file* 1 (assoc params :is-shared true)) + file (th/create-file* 2 params) + other (th/create-file* 3 params)] + + (th/link-file-to-library* {:file-id (:id file) :library-id (:id library)}) + (th/link-file-to-library* {:file-id (:id other) :library-id (:id library)}) + + (update-file! + :file-id (:id library) + :profile-id (:id profile) + :changes + (-> (pcb/empty-changes nil) + (pcb/with-library-data (:data library)) + (pcb/set-tokens-lib tokens-lib) + (:redo-changes))) + + (update-file! + :file-id (:id file) + :profile-id (:id profile) + :changes + (-> (pcb/empty-changes nil) + (pcb/with-library-data (:data file)) + (pcb/set-tokens-source (:id library)) + (pcb/set-tokens-status (ctos/make-tokens-status :active-theme-ids #{theme-id} + :active-set-ids #{set-id})) + (:redo-changes))) + + {:library library :file file :other other :tokens-lib tokens-lib})) + +(defn- check-tokens-absorbed + "The file is its own tokens source, with a copy of the tokens lib, and + keeps its tokens status; the other file is not changed." + [{:keys [file other tokens-lib]}] + (let [file-data (:data (bfc/get-file th/*system* (:id file))) + other-data (:data (bfc/get-file th/*system* (:id other)))] + (t/is (nil? (cfo/get-tokens-source file-data))) + (t/is (= (ctob/export-dtcg-json tokens-lib) + (ctob/export-dtcg-json (cfo/get-tokens-lib file-data)))) + (t/is (= {:active-themes #{"/test-theme"} + :active-sets #{"test-set"}} + (cfo/ids->names (cfo/get-tokens-status file-data) + (cfo/get-tokens-lib file-data)))) + (t/is (nil? (cfo/get-tokens-lib other-data))))) + +(t/deftest delete-shared-library-absorbs-its-tokens-lib + (let [profile (th/create-profile* 1) + {:keys [library file] :as prepared} (prepare-tokens-library profile) + out (th/command! {::th/type :delete-file + ::rpc/profile-id (:id profile) + :id (:id library)})] + (t/is (nil? (:error out))) + + ;; The relations are removed at once, the absorption is done by the + ;; deletion task. + (t/is (empty? (th/db-query :file-library-rel {:library-file-id (:id library)}))) + (t/is (= (:id library) + (cfo/get-tokens-source (:data (bfc/get-file th/*system* (:id file)))))) + + (th/run-pending-jobs) + (check-tokens-absorbed prepared))) + +(t/deftest unpublish-shared-library-absorbs-its-tokens-lib + (let [profile (th/create-profile* 1) + {:keys [library] :as prepared} (prepare-tokens-library profile) + out (th/command! {::th/type :set-file-shared + ::rpc/profile-id (:id profile) + :id (:id library) + :is-shared false})] + (t/is (nil? (:error out))) + (check-tokens-absorbed prepared))) + (t/deftest deleted-files-permanently-delete (let [prof (th/create-profile* 1 {:is-active true}) team-id (:default-team-id prof) @@ -2994,3 +3086,36 @@ :file-id (:id file)})] (t/is (th/ex-info? (:error out))) (t/is (th/ex-of-type? (:error out) :not-found)))) + +(t/deftest delete-file-audit-event-carries-team-id + ;; The event already carries the project id, so the resolver could find the + ;; team; the command has the team row and sets the id without a lookup. + (let [profile (th/create-profile* 1 {:is-active true}) + team (th/create-team* 1 {:profile-id (:id profile)}) + project (th/create-project* 1 {:profile-id (:id profile) + :team-id (:id team)}) + file (th/create-file* 1 {:profile-id (:id profile) + :project-id (:id project)}) + wrapped (th/command-raw! {::th/type :delete-file + ::rpc/profile-id (:id profile) + :id (:id file)})] + (t/is (= (:id team) + (get-in (meta wrapped) [:app.loggers.audit/props :team-id]))))) + +(t/deftest create-file-audit-event-carries-its-id + ;; The command returns the new file but did not emit its id, so a consumer + ;; could not key a local projection. + (let [profile (th/create-profile* 1 {:is-active true}) + team (th/create-team* 1 {:profile-id (:id profile)}) + project (th/create-project* 1 {:profile-id (:id profile) + :team-id (:id team)}) + out (th/command! {::th/type :create-file + ::rpc/profile-id (:id profile) + :project-id (:id project) + :name "foobar" + :is-shared false})] + (t/is (th/success? out)) + (t/is (= (get-in out [:result :id]) + (get-in (meta (:result out)) [:app.loggers.audit/props :id]))) + (t/is (= (:id team) + (get-in (meta (:result out)) [:app.loggers.audit/props :team-id]))))) diff --git a/backend/test/backend_tests/rpc_font_test.clj b/backend/test/backend_tests/rpc_font_test.clj index 90dabf62a0..bd81112bcb 100644 --- a/backend/test/backend_tests/rpc_font_test.clj +++ b/backend/test/backend_tests/rpc_font_test.clj @@ -140,7 +140,7 @@ (let [res (binding [ct/*clock* (ct/fixed-clock (ct/in-future {:hours 3}))] (th/run-task! :storage-gc-touched {}))] - (t/is (= 6 (:freeze res)))) + (t/is (= 8 (:freeze res)))) (let [params {::th/type :delete-font ::rpc/profile-id (:id prof) @@ -209,7 +209,7 @@ (let [res (binding [ct/*clock* (ct/fixed-clock (ct/in-future {:hours 3}))] (th/run-task! :storage-gc-touched {}))] - (t/is (= 6 (:freeze res)))) + (t/is (= 8 (:freeze res)))) (let [params {::th/type :delete-font ::rpc/profile-id (:id prof) @@ -260,7 +260,7 @@ ;; freeze with hours 3 clock (let [res (binding [ct/*clock* (ct/fixed-clock (ct/in-future {:hours 3}))] (th/run-task! :storage-gc-touched {}))] - (t/is (= 6 (:freeze res)))) + (t/is (= 8 (:freeze res)))) (let [params {::th/type :delete-font-variant ::rpc/profile-id (:id prof) :team-id team-id :id (-> out1 :result :id)} diff --git a/backend/test/backend_tests/rpc_management_nitrate_test.clj b/backend/test/backend_tests/rpc_management_nitrate_test.clj index c00930857c..bb21ce7dcd 100644 --- a/backend/test/backend_tests/rpc_management_nitrate_test.clj +++ b/backend/test/backend_tests/rpc_management_nitrate_test.clj @@ -1989,6 +1989,9 @@ :type "action"}]} params (with-meta params {::http/request http-request}) + ;; The profile setup above emits membership events; drop them so + ;; only the pushed events are counted below. + _ (th/reset-mock! audit-mock) out (th/management-command! params)] (t/is (nil? (:error out))) (t/is (:called? @audit-mock)) @@ -2009,6 +2012,9 @@ :profile-id (:id prof)}]} params (with-meta params {::http/request http-request}) + ;; The profile setup above emits membership events; drop them so + ;; only the pushed events are asserted below. + _ (th/reset-mock! audit-mock) out (th/management-command! params)] (t/is (nil? (:error out))) (let [[_ event] (:call-args @audit-mock)] @@ -2028,6 +2034,9 @@ :type "action"}]} params (with-meta params {::http/request http-request}) + ;; The profile setup above emits membership events; drop them so + ;; only the pushed events are counted below. + _ (th/reset-mock! audit-mock) out (th/management-command! params)] (t/is (nil? (:error out))) (t/is (= 2 (:call-count @audit-mock))) @@ -2047,6 +2056,9 @@ :foo "bar"}}]} params (with-meta params {::http/request http-request}) + ;; The profile setup above emits membership events; drop them so + ;; only the pushed events are asserted below. + _ (th/reset-mock! audit-mock) out (th/management-command! params)] (t/is (nil? (:error out))) (let [[_ event] (:call-args @audit-mock)] @@ -2071,6 +2083,9 @@ params (with-meta params {::http/request (assoc http-request ::http/auth-key-id :admin-console)}) + ;; The profile setup above emits membership events; drop them so + ;; only the pushed events are asserted below. + _ (th/reset-mock! audit-mock) out (th/management-command! params)] (t/is (nil? (:error out))) (let [[_ event] (:call-args @audit-mock)] diff --git a/backend/test/backend_tests/rpc_management_test.clj b/backend/test/backend_tests/rpc_management_test.clj index 4b449af993..e1f6e81950 100644 --- a/backend/test/backend_tests/rpc_management_test.clj +++ b/backend/test/backend_tests/rpc_management_test.clj @@ -9,6 +9,7 @@ [app.binfile.common :as bfc] [app.common.features :as cfeat] [app.common.pprint :as pp] + [app.common.time :as ct] [app.common.types.shape :as cts] [app.common.uuid :as uuid] [app.config :as cf] @@ -22,7 +23,8 @@ [clojure.test :as t] [cuerdas.core :as str] [datoteka.fs :as fs] - [datoteka.io :as io])) + [datoteka.io :as io] + [mockery.core :refer [with-mocks]])) (t/use-fixtures :once th/state-init) (t/use-fixtures :each th/database-reset) @@ -666,3 +668,55 @@ (t/is (vector? result)) (t/is (= 1 (count result))) (t/is (= "test" (:id (first result))))))) + +(t/deftest clone-template-audit-events-carry-team-id + ;; The per-file `create-file` events are built by hand, so they do not pass + ;; through the generic resolver: the command must copy the team id itself. + (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] + (with-redefs [cf/flags (conj cf/flags :audit-log)] + (let [prof (th/create-profile* 1 {:is-active true}) + team (th/create-team* 1 {:profile-id (:id prof)}) + project (th/create-project* 1 {:profile-id (:id prof) + :team-id (:id team)}) + out (th/command! {::th/type :clone-template + ::rpc/profile-id (:id prof) + :project-id (:id project) + :template-id "test"})] + (t/is (th/success? out)) + (th/consume-sse (:result out)) + (let [events (->> (:call-args-list @audit-mock) + (map second) + (filter #(= "create-file" (:name %))))] + (t/is (seq events)) + (t/is (every? #(= (:id team) (get-in % [:props :team-id])) events))))))) + +(t/deftest duplicate-project-audit-events-carry-entity-ids + ;; The duplicated project and files are created by internal calls, so the + ;; command emits one create event per entity for a local projection. + (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] + (with-redefs [cf/flags (conj cf/flags :audit-log)] + (let [profile (th/create-profile* 1 {:is-active true}) + team (th/create-team* 52 {:profile-id (:id profile)}) + project (th/create-project* 52 {:profile-id (:id profile) + :team-id (:id team)}) + _file (th/create-file* 52 {:profile-id (:id profile) + :project-id (:id project)}) + out (th/command! {::th/type :duplicate-project + ::rpc/profile-id (:id profile) + :project-id (:id project) + :name "copy"}) + events (->> (:call-args-list @audit-mock) + (map second) + (filter #(contains? #{"create-project" "create-file"} (:name %))))] + (t/is (th/success? out)) + (t/is (some #(and (= "create-project" (:name %)) + (= (get-in out [:result :id]) (get-in % [:props :id])) + (= (:id team) (get-in % [:props :team-id])) + (= "copy" (get-in % [:props :name]))) + events)) + (t/is (some #(and (= "create-file" (:name %)) + (string? (get-in % [:props :name])) + (int? (get-in % [:props :revn])) + (ct/inst? (get-in % [:props :modified-at]))) + events)) + (t/is (every? #(= (:id team) (get-in % [:props :team-id])) events)))))) diff --git a/backend/test/backend_tests/rpc_media_test.clj b/backend/test/backend_tests/rpc_media_test.clj index fb34396a09..985fb68fc2 100644 --- a/backend/test/backend_tests/rpc_media_test.clj +++ b/backend/test/backend_tests/rpc_media_test.clj @@ -1226,3 +1226,27 @@ (t/is (th/ex-info? (:error out))) (t/is (th/ex-of-type? (:error out) :validation)) (t/is (th/ex-of-code? (:error out) :params-validation)))) + +(t/deftest media-audit-event-carries-team-id + ;; The media event carries the file id, so the resolver could find the team; + ;; the command already resolved the team for the quota check and sets it. + (let [prof (th/create-profile* 1) + team (th/create-team* 1 {:profile-id (:id prof)}) + project (th/create-project* 1 {:profile-id (:id prof) + :team-id (:id team)}) + file (th/create-file* 1 {:profile-id (:id prof) + :project-id (:id project) + :is-shared false}) + mfile {:filename "sample.jpg" + :path (th/tempfile "backend_tests/test_files/sample.jpg") + :mtype "image/jpeg" + :size 312043} + out (th/command! {::th/type :upload-file-media-object + ::rpc/profile-id (:id prof) + :file-id (:id file) + :is-local true + :name "testfile" + :content mfile})] + (t/is (th/success? out)) + (t/is (= (:id team) + (get-in (meta (:result out)) [:app.loggers.audit/replace-props :team-id]))))) diff --git a/backend/test/backend_tests/rpc_profile_test.clj b/backend/test/backend_tests/rpc_profile_test.clj index fbdef2f160..dc44b6eeaf 100644 --- a/backend/test/backend_tests/rpc_profile_test.clj +++ b/backend/test/backend_tests/rpc_profile_test.clj @@ -1802,3 +1802,30 @@ (t/is (th/success? out)) (t/is (= (:id member) event-pid)) (t/is (not= (:id owner) event-pid)))))) + +(t/deftest delete-profile-audit-event-carries-profile-data + ;; The event used to carry empty props because every param is namespaced and + ;; clean-props drops them; the projection needs the deleted identity. + (let [profile (th/create-profile* 1 {:is-active true}) + out (th/command-raw! {::th/type :delete-profile + ::rpc/profile-id (:id profile)}) + props (:app.loggers.audit/props (meta out))] + (t/is (= (:id profile) (:id props))) + (t/is (= "profile1.test@nodomain.com" (:email props))) + (t/is (= "Profile 1" (:fullname props))) + (t/is (ct/inst? (:created-at props))) + (t/is (ct/inst? (:modified-at props))) + (t/is (ct/inst? (:deleted-at props))))) + +(t/deftest delete-profile-is-idempotent + ;; Deleting an already-deleted profile (client retry after a lost response) + ;; succeeds: the row is still there, only marked. + (let [profile (th/create-profile* 1 {:is-active true}) + params {::th/type :delete-profile + ::rpc/profile-id (:id profile)} + first-out (th/command-raw! params)] + (t/is (th/success? first-out)) + (let [second-out (th/command-raw! params)] + (t/is (th/success? second-out)) + (t/is (= (:id profile) + (get-in (meta second-out) [:app.loggers.audit/props :id])))))) diff --git a/backend/test/backend_tests/rpc_project_test.clj b/backend/test/backend_tests/rpc_project_test.clj index f5855a43e8..c27bb31868 100644 --- a/backend/test/backend_tests/rpc_project_test.clj +++ b/backend/test/backend_tests/rpc_project_test.clj @@ -274,3 +274,16 @@ (t/is (th/ex-info? (:error out))) (t/is (th/ex-of-type? (:error out) :validation)) (t/is (th/ex-of-code? (:error out) :params-validation)))) + +(t/deftest create-project-audit-event-carries-its-id + ;; The command returns the new project but did not emit its id, so a consumer + ;; could not key a local projection. + (let [profile (th/create-profile* 1) + team (th/create-team* 1 {:profile-id (:id profile)}) + out (th/command! {::th/type :create-project + ::rpc/profile-id (:id profile) + :team-id (:id team) + :name "test project"})] + (t/is (th/success? out)) + (t/is (= (get-in out [:result :id]) + (get-in (meta (:result out)) [:app.loggers.audit/props :id]))))) diff --git a/backend/test/backend_tests/rpc_team_test.clj b/backend/test/backend_tests/rpc_team_test.clj index 7781605860..cc80909d64 100644 --- a/backend/test/backend_tests/rpc_team_test.clj +++ b/backend/test/backend_tests/rpc_team_test.clj @@ -6,8 +6,10 @@ (ns backend-tests.rpc-team-test (:require + [app.common.data :as d] [app.common.logging :as l] [app.common.time :as ct] + [app.common.types.team :as types.team] [app.common.uuid :as uuid] [app.config :as cf] [app.db :as db] @@ -17,6 +19,7 @@ [app.nitrate :as nitrate] [app.rpc :as-alias rpc] [app.rpc.commands.teams :as teams] + [app.rpc.helpers :as rph] [app.storage :as sto] [app.tokens :as tokens] [backend-tests.helpers :as th] @@ -1784,3 +1787,195 @@ ::rpc/profile-id (:id stranger)})] (t/is (th/success? out)) (t/is (nil? (some #(= (:id project) (:id %)) (:result out)))))))) + +(t/deftest create-team-emits-add-team-member-audit-event + ;; Team creation adds the owner through add-profile-to-team!, which used to + ;; leave no membership event. + (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] + (with-redefs [cf/flags (conj cf/flags :audit-log)] + (let [owner (th/create-profile* 1 {:is-active true}) + team (th/create-team* 21 {:profile-id (:id owner)}) + events (->> (:call-args-list @audit-mock) + (map second) + (filter #(= "add-team-member" (:name %))) + (filter #(= (:id team) (get-in % [:props :team-id]))))] + (t/is (= 1 (count events))) + (t/is (= (:id owner) (get-in (first events) [:props :member-id]))) + (t/is (true? (get-in (first events) [:props :is-owner]))))))) + +(t/deftest add-team-member-event-skipped-when-membership-exists + ;; Adding an existing member with on-conflict-do-nothing inserts nothing and + ;; must not report a membership that already exists. + (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] + (with-redefs [cf/flags (conj cf/flags :audit-log)] + (let [owner (th/create-profile* 1 {:is-active true}) + team (th/create-team* 22 {:profile-id (:id owner)})] + (th/reset-mock! audit-mock) + (db/tx-run! th/*system* + (fn [cfg] + (teams/add-profile-to-team! cfg + {:team-id (:id team) + :profile-id (:id owner)} + {::db/on-conflict-do-nothing? true}))) + (t/is (empty? (:call-args-list @audit-mock))))))) + +(t/deftest no-add-team-member-event-without-audit-log + (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] + (let [owner (th/create-profile* 1 {:is-active true}) + _team (th/create-team* 23 {:profile-id (:id owner)}) + events (->> (:call-args-list @audit-mock) + (map second) + (filter #(= "add-team-member" (:name %))))] + (t/is (empty? events))))) + +(t/deftest add-team-member-event-carries-derived-role + ;; `team_profile_rel` has no role column, so no caller can pass it: the + ;; emitter derives it from the flags, inverse of permissions-for-role. + (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] + (with-redefs [cf/flags (conj cf/flags :audit-log)] + (let [owner (th/create-profile* 1 {:is-active true}) + team (th/create-team* 26 {:profile-id (:id owner)}) + editor (th/create-profile* 27 {:is-active true}) + admin (th/create-profile* 28 {:is-active true}) + viewer (th/create-profile* 29 {:is-active true})] + (db/tx-run! th/*system* + (fn [cfg] + (doseq [[member role] [[editor :editor] + [admin :admin] + [viewer :viewer]]] + (teams/add-profile-to-team! + cfg + (merge {:team-id (:id team) + :profile-id (:id member)} + (get types.team/permissions-for-role role)))))) + (let [by-member (->> (:call-args-list @audit-mock) + (map second) + (filter #(= "add-team-member" (:name %))) + (d/index-by #(get-in % [:props :member-id])))] + (t/is (= :owner (get-in by-member [(:id owner) :props :role]))) + (t/is (= :editor (get-in by-member [(:id editor) :props :role]))) + (t/is (= :admin (get-in by-member [(:id admin) :props :role]))) + (t/is (= :viewer (get-in by-member [(:id viewer) :props :role])))))))) + +(t/deftest add-team-member-event-carries-request-context + ;; Sibling events built by `prepare-rpc-event` carry the request context; + ;; the hand-built membership event must not lose it when rpc params travel. + (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] + (with-redefs [cf/flags (conj cf/flags :audit-log)] + (let [owner (th/create-profile* 1 {:is-active true}) + member (th/create-profile* 2 {:is-active true}) + team (th/create-team* 30 {:profile-id (:id owner)})] + (th/reset-mock! audit-mock) + (db/tx-run! th/*system* + (fn [cfg] + (teams/add-profile-to-team! + cfg + (merge {:team-id (:id team) + :profile-id (:id member)} + (get types.team/permissions-for-role :editor)) + nil + {:rpc-params {::rpc/profile-id (:id owner) + ::rpc/request-id "req-1"}}))) + (let [events (->> (:call-args-list @audit-mock) + (map second) + (filter #(= "add-team-member" (:name %))))] + (t/is (= 1 (count events))) + (t/is (= "req-1" (get-in (first events) [:context :request-id])))))))) + +(t/deftest invitation-direct-add-attributes-event-to-inviter + ;; With email verification off, inviting an existing profile adds the + ;; member directly; the event belongs to the inviter, not the member. + (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil}] + (with-redefs [cf/flags #{:audit-log}] + (let [owner (th/create-profile* 1 {:is-active true}) + member (th/create-profile* 2 {:is-active true}) + team (th/create-team* 31 {:profile-id (:id owner)})] + (th/reset-mock! audit-mock) + (let [out (th/command! {::th/type :create-team-invitations + ::rpc/profile-id (:id owner) + :team-id (:id team) + :role :editor + :emails [(:email member)]})] + (t/is (th/success? out)) + (let [events (->> (:call-args-list @audit-mock) + (map second) + (filter #(= "add-team-member" (:name %))) + (filter #(= (:id member) (get-in % [:props :member-id]))))] + (t/is (= 1 (count events))) + (t/is (= (:id owner) (:profile-id (first events)))) + (t/is (= :editor (get-in (first events) [:props :role]))))))))) + +(t/deftest create-team-audit-event-carries-created-at + ;; The projection keys the new team on the event id and needs its creation + ;; time from the same event. + (let [profile (th/create-profile* 1 {:is-active true}) + out (th/command-raw! {::th/type :create-team + ::rpc/profile-id (:id profile) + :name "team"}) + props (:app.loggers.audit/props (meta out))] + (t/is (= (:id out) (:id props))) + (t/is (= (:created-at out) (:created-at props))) + (t/is (ct/inst? (:created-at props))) + (t/is (= (:modified-at out) (:modified-at props))) + (t/is (ct/inst? (:modified-at props))))) + +(t/deftest delete-team-audit-event-carries-entity-data + ;; The projection marks the team deleted from this single event, so it needs + ;; the identity, the name and the timestamps. + (let [profile (th/create-profile* 1 {:is-active true}) + team (th/create-team* 31 {:profile-id (:id profile)}) + out (th/command-raw! {::th/type :delete-team + ::rpc/profile-id (:id profile) + :id (:id team)}) + props (:app.loggers.audit/props (meta out))] + ;; The API response is unchanged: the command still returns nil. + (t/is (nil? (rph/unwrap out))) + (t/is (= (:id team) (:id props))) + (t/is (= (:name team) (:name props))) + (t/is (= (:created-at team) (:created-at props))) + (t/is (ct/inst? (:modified-at props))) + (t/is (ct/inst? (:deleted-at props))))) + +(t/deftest create-team-with-invitations-audit-event-carries-team-id + ;; The team is created by an internal call, so both the command event and + ;; the manual event must carry its id for a projection to key it. + (with-mocks [audit-mock {:target 'app.loggers.audit/submit :return nil} + email-mock {:target 'app.email/send :return nil}] + (with-redefs [cf/flags (conj cf/flags :audit-log)] + (let [profile (th/create-profile* 1 {:is-active true}) + out (th/command-raw! {::th/type :create-team-with-invitations + ::rpc/profile-id (:id profile) + :name "team" + :emails #{"invitee@example.com"} + :role :editor}) + props (:app.loggers.audit/props (meta out)) + events (->> (:call-args-list @audit-mock) + (map second) + (filter #(= "create-team" (:name %))))] + (t/is (= (:id out) (:id props))) + (t/is (= 1 (count events))) + (t/is (= (:id out) (get-in (first events) [:props :id]))))))) + +(t/deftest update-team-member-role-to-owner-names-demoted-member + ;; Promoting a member to owner demotes the actor; the event names them so a + ;; projection can keep a single owner. + (let [owner (th/create-profile* 1 {:is-active true}) + editor (th/create-profile* 2 {:is-active true}) + team (th/create-team* 82 {:profile-id (:id owner)})] + (th/create-team-role* {:team-id (:id team) + :profile-id (:id editor) + :role :editor}) + (let [out (th/command-raw! {::th/type :update-team-member-role + ::rpc/profile-id (:id owner) + :team-id (:id team) + :member-id (:id editor) + :role :editor}) + props (:app.loggers.audit/props (meta out))] + (t/is (nil? (:prev-owner-id props)))) + (let [out (th/command-raw! {::th/type :update-team-member-role + ::rpc/profile-id (:id owner) + :team-id (:id team) + :member-id (:id editor) + :role :owner}) + props (:app.loggers.audit/props (meta out))] + (t/is (= (:id owner) (:prev-owner-id props)))))) diff --git a/backend/test/backend_tests/rpc_webhooks_test.clj b/backend/test/backend_tests/rpc_webhooks_test.clj index 5b15d46111..de5227aeb2 100644 --- a/backend/test/backend_tests/rpc_webhooks_test.clj +++ b/backend/test/backend_tests/rpc_webhooks_test.clj @@ -374,3 +374,29 @@ (t/is (nil? (:result out))) (let [rows (th/db-exec! ["select * from webhook"])] (t/is (= 0 (count rows))))))))))) + +(t/deftest webhook-audit-events-carry-team-id + ;; `update-webhook` and `delete-webhook` have no team id in their params, so + ;; the command copies it from the stored webhook into the audit props. + (with-mocks [http-mock {:target 'app.http.client/req :return {:status 200}}] + (let [prof (th/create-profile* 1 {:is-active true}) + team (th/create-team* 1 {:profile-id (:id prof)}) + whook (-> (th/command! {::th/type :create-webhook + ::rpc/profile-id (:id prof) + :team-id (:id team) + :uri (u/uri "http://example.com") + :mtype "application/json"}) + :result) + upd (th/command! {::th/type :update-webhook + ::rpc/profile-id (:id prof) + :id (:id whook) + :uri (u/uri "http://example.com/updated") + :mtype "application/json" + :is-active true}) + del (th/command-raw! {::th/type :delete-webhook + ::rpc/profile-id (:id prof) + :id (:id whook)})] + (t/is (= (:id team) + (get-in (meta (:result upd)) [:app.loggers.audit/props :team-id]))) + (t/is (= (:id team) + (get-in (meta del) [:app.loggers.audit/props :team-id])))))) diff --git a/backend/test/backend_tests/srepl_main_test.clj b/backend/test/backend_tests/srepl_main_test.clj index b44c20bf57..1d131d1fe3 100644 --- a/backend/test/backend_tests/srepl_main_test.clj +++ b/backend/test/backend_tests/srepl_main_test.clj @@ -11,7 +11,8 @@ [app.srepl.main :as srepl] [app.system :as sys] [backend-tests.helpers :as th] - [clojure.test :as t])) + [clojure.test :as t] + [mockery.core :refer [with-mocks]])) (t/use-fixtures :once th/state-init) (t/use-fixtures :each th/database-reset) @@ -150,3 +151,57 @@ #(srepl/process-deleted-files-cascade :ids [(:id file)]))] (t/is (= [{:object :file :id (:id file)}] (mapv #(select-keys % [:object :id]) (params-of submitted)))))))))) + +(t/deftest delete-profile-emits-identifiable-audit-event + ;; SREPL has no actor, so the event profile stays zero; the subject id in + ;; props is what makes the row attributable. Both insert and cascade are + ;; mocked, so this has no side effects. + (with-mocks [audit-mock {:target 'app.loggers.audit/insert :return nil} + wrk-mock {:target 'app.jobs/invoke :return nil}] + (let [profile (th/create-profile* 1 {:is-active true})] + (srepl/delete-profile! (str (:id profile))) + (let [events (->> (:call-args-list @audit-mock) + (map second))] + (t/is (= 1 (count events))) + (let [event (first events)] + (t/is (= "delete-profile" (:name event))) + (t/is (= "action" (:type event))) + ;; No actor on SREPL calls; insert defaults the missing profile + ;; to the zero one. + (t/is (nil? (:profile-id event))) + (t/is (= (:id profile) (get-in event [:props :id])))))))) + +(t/deftest delete-team-emits-identifiable-audit-event + (with-mocks [audit-mock {:target 'app.loggers.audit/insert :return nil} + wrk-mock {:target 'app.jobs/invoke :return nil}] + (let [profile (th/create-profile* 1 {:is-active true}) + team (th/create-team* 95 {:profile-id (:id profile)})] + (srepl/delete-team! (str (:id team))) + (let [events (->> (:call-args-list @audit-mock) + (map second))] + (t/is (= 1 (count events))) + (let [event (first events)] + (t/is (= "delete-team" (:name event))) + (t/is (= "action" (:type event))) + (t/is (= (:id team) (get-in event [:props :id]))) + (t/is (= "explicit call to delete-team!" + (get-in event [:context :cause])))))))) + +(t/deftest restore-profile-emits-identifiable-audit-event + ;; SREPL has no actor, so the event profile stays zero; the subject id in + ;; props is what makes the row attributable. + (with-mocks [audit-mock {:target 'app.loggers.audit/insert :return nil} + wrk-mock {:target 'app.jobs/invoke :return nil}] + (with-redefs [sys/system th/*system*] + (let [profile (th/create-profile* 1 {:is-active true})] + (srepl/restore-profile! (str (:id profile))) + (let [events (->> (:call-args-list @audit-mock) + (map second))] + (t/is (= 1 (count events))) + (let [event (first events)] + (t/is (= "restore-profile" (:name event))) + (t/is (= "action" (:type event))) + ;; No actor on SREPL calls; insert defaults the missing profile + ;; to the zero one. + (t/is (nil? (:profile-id event))) + (t/is (= (:id profile) (get-in event [:props :id]))))))))) diff --git a/backend/test/backend_tests/storage_test.clj b/backend/test/backend_tests/storage_test.clj index ba7c2bb914..5db678d6a8 100644 --- a/backend/test/backend_tests/storage_test.clj +++ b/backend/test/backend_tests/storage_test.clj @@ -17,6 +17,7 @@ [app.storage :as sto] [app.storage.fs :as-alias sto.fs] [app.storage.gc-deleted :as gcd] + [app.storage.gc-touched :as gc-touched] [app.storage.impl :as impl] [app.storage.s3 :as-alias sto.s3] [app.storage.schema :as stsch] @@ -367,8 +368,10 @@ ;; run the touched gc task (let [res (binding [ct/*clock* (ct/fixed-clock (ct/in-future {:hours 3}))] (th/run-task! :storage-gc-touched {}))] - (t/is (= 5 (:freeze res))) - (t/is (= 1 (:delete res))) + (t/is (= 6 (:freeze res))) + ;; the consumed-but-not-yet-purged session keeps its mapping, + ;; so its chunk freezes instead of being marked + (t/is (= 0 (:delete res))) (let [result-1 (:result out1) result-2 (:result out2)] @@ -1177,6 +1180,168 @@ (t/is (= 1 (:deleted res)))) (t/is (= 0 (:count (th/db-exec-one! ["select count(*) from storage_object"])))))) +(t/deftest upload-session-chunk-frozen-while-referenced + ;; A chunk whose session mapping is still alive must be frozen, not + ;; marked for deletion: the reference scan owns the liveness decision. + (let [prof (th/create-profile* 1) + _ (th/create-project* 1 {:profile-id (:id prof) + :team-id (:default-team-id prof)}) + _ (th/create-file* 1 {:profile-id (:id prof) + :project-id (:default-project-id prof) + :is-shared false}) + mfile {:filename "chunk" + :path (th/tempfile "backend_tests/test_files/sample.jpg") + :mtype "image/jpeg" + :size 312043} + session-id (-> (th/command! {::th/type :create-upload-session + ::rpc/profile-id (:id prof) + :total-chunks 1}) + :result :session-id) + out (th/command! {::th/type :upload-chunk + ::rpc/profile-id (:id prof) + :session-id session-id + :index 0 + :content mfile})] + (t/is (nil? (:error out))) + (t/is (= 1 (:count (th/db-exec-one! ["select count(*) from upload_session_chunk where session_id = ?" + session-id])))) + + ;; the session is neither consumed nor stalled: the mapping is alive + (let [res (binding [ct/*clock* (ct/fixed-clock (ct/in-future {:hours 3}))] + (th/run-task! :storage-gc-touched {}))] + (t/is (= 1 (:freeze res))) + (t/is (= 0 (:delete res)))) + + ;; frozen: untouched, not marked, mapping intact + (t/is (= 0 (:count (th/db-exec-one! ["select count(*) from storage_object where touched_at is not null"])))) + (t/is (= 0 (:count (th/db-exec-one! ["select count(*) from storage_object where deleted_at is not null"])))) + (t/is (= 1 (:count (th/db-exec-one! ["select count(*) from upload_session_chunk where session_id = ?" + session-id])))))) + +(t/deftest upload-session-freeze-reverify + ;; The freeze recheck runs under row locks: an object whose mapping + ;; disappeared (plus a concurrent touch) after the first check is + ;; reclassified to deletion instead of swallowing the touch. + (let [prof (th/create-profile* 1) + _ (th/create-project* 1 {:profile-id (:id prof) + :team-id (:default-team-id prof)}) + _ (th/create-file* 1 {:profile-id (:id prof) + :project-id (:default-project-id prof) + :is-shared false}) + mfile {:filename "chunk" + :path (th/tempfile "backend_tests/test_files/sample.jpg") + :mtype "image/jpeg" + :size 312043} + session-id (-> (th/command! {::th/type :create-upload-session + ::rpc/profile-id (:id prof) + :total-chunks 1}) + :result :session-id) + out (th/command! {::th/type :upload-chunk + ::rpc/profile-id (:id prof) + :session-id session-id + :index 0 + :content mfile}) + object-id (:object-id (th/db-exec-one! ["select object_id from upload_session_chunk where session_id = ?" + session-id])) + check @#'gc-touched/has-upload-session-chunk-refs? + verify @#'gc-touched/verify-frozen] + (t/is (nil? (:error out))) + + ;; mapping alive: stays frozen + (let [[frozen deletable] (db/transact! th/*pool* (fn [conn] (verify conn check "upload-session" [{:id object-id}])))] + (t/is (= #{object-id} frozen)) + (t/is (= #{} deletable))) + + ;; simulate a purge that committed after the first check: mapping + ;; gone, touch bumped + (th/db-exec! ["delete from upload_session_chunk where session_id = ?" session-id]) + (th/db-exec! ["update storage_object set touched_at = ? where id = ?" (ct/now) object-id]) + + ;; recheck moves it to deletion + (let [[frozen deletable] (db/transact! th/*pool* (fn [conn] (verify conn check "upload-session" [{:id object-id}])))] + (t/is (= #{} frozen)) + (t/is (= #{object-id} deletable))))) + +(t/deftest upload-session-gc-deleted-skips-live-mapping + ;; A row marked for deletion while its mapping is still alive must NOT + ;; be reaped silently: the chunk is skipped loud (logged) and retried + ;; once objects-gc purges the session. This pins the mark-time + ;; invariant instead of breaking sessions quietly. + (let [prof (th/create-profile* 1) + _ (th/create-project* 1 {:profile-id (:id prof) + :team-id (:default-team-id prof)}) + _ (th/create-file* 1 {:profile-id (:id prof) + :project-id (:default-project-id prof) + :is-shared false}) + mfile {:filename "chunk" + :path (th/tempfile "backend_tests/test_files/sample.jpg") + :mtype "image/jpeg" + :size 312043} + session-id (-> (th/command! {::th/type :create-upload-session + ::rpc/profile-id (:id prof) + :total-chunks 1}) + :result :session-id) + out (th/command! {::th/type :upload-chunk + ::rpc/profile-id (:id prof) + :session-id session-id + :index 0 + :content mfile}) + object-id (:object-id (th/db-exec-one! ["select object_id from upload_session_chunk where session_id = ?" + session-id]))] + (t/is (nil? (:error out))) + + ;; force the anomaly: mark deleted while the mapping is alive + (th/db-exec! ["update storage_object set deleted_at = ? where id = ?" (ct/now) object-id]) + + (let [res (th/run-task! :storage-gc-deleted {})] + (t/is (= 0 (:deleted res)))) + + ;; row and mapping both intact for the retry + (t/is (= 1 (:count (th/db-exec-one! ["select count(*) from storage_object where id = ?" object-id])))) + (t/is (= 1 (:count (th/db-exec-one! ["select count(*) from upload_session_chunk where session_id = ?" + session-id])))))) + +(t/deftest upload-session-deferred-keeps-mapping + ;; A physical-deletion failure defers the row; the session mapping is + ;; kept so the session survives the backend outage and the blob is + ;; retried the next day. + (let [prof (th/create-profile* 1) + _ (th/create-project* 1 {:profile-id (:id prof) + :team-id (:default-team-id prof)}) + _ (th/create-file* 1 {:profile-id (:id prof) + :project-id (:default-project-id prof) + :is-shared false}) + mfile {:filename "chunk" + :path (th/tempfile "backend_tests/test_files/sample.jpg") + :mtype "image/jpeg" + :size 312043} + session-id (-> (th/command! {::th/type :create-upload-session + ::rpc/profile-id (:id prof) + :total-chunks 1}) + :result :session-id) + out (th/command! {::th/type :upload-chunk + ::rpc/profile-id (:id prof) + :session-id session-id + :index 0 + :content mfile}) + object-id (:object-id (th/db-exec-one! ["select object_id from upload_session_chunk where session_id = ?" + session-id]))] + (t/is (nil? (:error out))) + (th/db-exec! ["update storage_object set deleted_at = ? where id = ?" (ct/now) object-id]) + + (with-mocks [_mock {:target 'app.storage.impl/del-objects-in-bulk + :return (fn [_ ids] (set ids))}] + (let [res (th/run-task! :storage-gc-deleted {})] + (t/is (= 0 (:deleted res))))) + + ;; row kept and deferred one day + (let [row (th/db-exec-one! ["select deletion_attempts, touched_at from storage_object where id = ?" object-id])] + (t/is (= 1 (:deletion-attempts row))) + (t/is (some? (:touched-at row)))) + ;; mapping kept: the session is intact for the retry + (t/is (= 1 (:count (th/db-exec-one! ["select count(*) from upload_session_chunk where session_id = ?" + session-id])))))) + (t/deftest upload-session-consumed-purge ;; An assembled session is marked as consumed and objects-gc purges it ;; right away, without waiting for the stalled threshold. diff --git a/common/src/app/common/exceptions.cljc b/common/src/app/common/exceptions.cljc index 5a8a9acaec..f27d1d531d 100644 --- a/common/src/app/common/exceptions.cljc +++ b/common/src/app/common/exceptions.cljc @@ -29,6 +29,22 @@ (def ^:dynamic *data-length* 8) (def ^:dynamic *data-level* 8) +(def max-message-length + "Upper bound, in characters, for an exception message surfaced as a hint or + embedded in a formatted stack trace. A message can carry a whole payload (a + serialized file, for instance) and grow to megabytes; only the head helps to + identify the failure, and the rest is noise that bloats reports and the audit + events built from them." + 1000) + +(defn- truncate + "Truncates a string to at most `n` characters, appending an ellipsis when it + had to cut. Values that are not strings pass through unchanged." + [s n] + (if (and (string? s) (> (count s) n)) + (str (subs s 0 n) "...") + s)) + (defmacro error [& {:keys [type hint] :as params}] `(ex-info ~(or hint (name type)) @@ -243,7 +259,7 @@ :cljs (defn format-throwable - [cause & {:as opts}] + [cause & {:keys [trace-length]}] (with-out-str (when-let [exdata (ex-data cause)] (when-let [hint (or (get exdata :hint) @@ -251,7 +267,7 @@ (when (str/index-of hint "\n") (println "Hint:") (println "--------------------") - (println hint) + (println (truncate hint max-message-length)) (println))) (when-let [explain (get exdata ::sm/explain)] @@ -272,14 +288,20 @@ (println)) (when-let [trace (.-stack cause)] - (println "Trace:") - (println "--------------------") - (println (.-stack cause)))))) + (let [lines (str/lines trace) + head (truncate (first lines) max-message-length) + tail (rest lines)] + (println "Trace:") + (println "--------------------") + (when head + (println head)) + (doseq [line (if trace-length (take trace-length tail) tail)] + (println line))))))) (defn first-line [s] (let [break-index (str/index-of s "\n")] - (if (pos? break-index) + (if (and (some? break-index) (pos? break-index)) (subs s 0 break-index) s))) @@ -308,5 +330,6 @@ (defn get-hint [cause] - (or (some-> (ex-data cause) (get :hint) first-line) - (some-> (ex-message cause) first-line))) + (truncate (or (some-> (ex-data cause) (get :hint) first-line) + (some-> (ex-message cause) first-line)) + max-message-length)) diff --git a/common/src/app/common/files/builder.cljc b/common/src/app/common/files/builder.cljc index 49d93407a0..2e02166a93 100644 --- a/common/src/app/common/files/builder.cljc +++ b/common/src/app/common/files/builder.cljc @@ -16,6 +16,7 @@ [app.common.files.helpers :as cph] [app.common.files.migrations :as fmig] [app.common.geom.shapes :as gsh] + [app.common.path-names :as cpn] [app.common.schema :as sm] [app.common.svg :as csvg] [app.common.time :as dt] @@ -102,7 +103,8 @@ (str/capital (d/name type))))) :always - (update :name unique-name state))) + (-> (update :name cpn/clean-path) + (update :name unique-name state)))) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; SCHEMAS diff --git a/common/src/app/common/files/migrations.cljc b/common/src/app/common/files/migrations.cljc index dbe0b0abd7..d0312f5a56 100644 --- a/common/src/app/common/files/migrations.cljc +++ b/common/src/app/common/files/migrations.cljc @@ -21,21 +21,24 @@ [app.common.geom.shapes.text :as gsht] [app.common.logging :as l] [app.common.math :as mth] + [app.common.path-names :as cpn] [app.common.schema :as sm] [app.common.svg :as csvg] [app.common.types.color :as types.color] [app.common.types.component :as ctk] + [app.common.types.components-list :as ctkl] [app.common.types.container :as ctn] [app.common.types.file :as ctf] [app.common.types.fills :as types.fills] [app.common.types.path :as path] [app.common.types.path.segment :as path.segment] [app.common.types.shape :as cts] + [app.common.types.shape-tree :as ctt] [app.common.types.shape.interactions :as ctsi] [app.common.types.shape.shadow :as ctss] [app.common.types.shape.text :as ctst] [app.common.types.text :as types.text] - [app.common.types.token :as ctt] + [app.common.types.token :as cto] [app.common.types.tokens-lib :as ctob] [app.common.types.variant :as ctv] [app.common.uuid :as uuid] @@ -223,8 +226,10 @@ [data _] (letfn [(fix-line-paths [shape] (if (= (:type shape) :path) - (let [{:keys [width height]} (grc/points->rect (:points shape))] - (if (or (mth/almost-zero? width) (mth/almost-zero? height)) + ;; A path with missing or empty points has no rect; rebuild + ;; its geometry from the content as for line-like paths. + (let [{:keys [width height] :as rect} (grc/points->rect (:points shape))] + (if (or (nil? rect) (mth/almost-zero? width) (mth/almost-zero? height)) (let [selrect (path.segment/content->selrect (:content shape)) points (grc/rect->points selrect) transform (gmt/matrix) @@ -2167,7 +2172,7 @@ (if (some? token) (update shape :applied-tokens (fn [applied] - (-> (zipmap ctt/per-side-stroke-width-keys (repeat token)) + (-> (zipmap cto/per-side-stroke-width-keys (repeat token)) (merge applied) (dissoc :stroke-width)))) shape))) @@ -2217,6 +2222,66 @@ (update :pages-index d/update-vals update-container) (d/update-when :components d/update-vals update-container)))) +(defmethod migrate-data "0033-normalize-main-instance-names" + [data _] + ;; In penpot shape names always have "/" characters normalized with one space between and after, + ;; to form paths. But this is not enforced and may be places that still admit names like + ;; "Group/Name" or "Group / Name". With normal shapes this is not a problem. But for main + ;; instances, the name of the root shape must match exactly the path+name of the component. + ;; Here we normalize the names of all main instances, and a new file integrity validation has + ;; been added to enforce this from now on. + ;; Main instances in variants are a special case because they need to match the variant name. + (letfn [(update-shape [container shape-id] + (let [shape (ctn/get-shape container shape-id) + + shape (cond + ;; Variant container must have a normalized name + (ctk/is-variant-container? shape) + (let [name (:name shape) + clean-name (cpn/clean-path name) + clean-name (if (str/blank? clean-name) + "(empty)" ;; this may occur if the name contains only separators + clean-name)] + (if (= name clean-name) + shape + (assoc shape :name clean-name))) + + ;; Main instance inside a variant must have the same name as the container + (ctk/main-instance? shape) + (if (ctk/is-variant? shape) + (let [variant-container (ctn/get-shape container (:parent-id shape))] + (if (or (= (:name shape) (:name variant-container)) + (nil? variant-container)) + shape + (assoc shape :name (:name variant-container)))) + + ;; Other main instances must have the same name as the path+name of the component + (let [component (ctkl/get-component data (:component-id shape) true) + expected-name (when component + (cpn/merge-path-item (:path component) (:name component)))] + (if (or (= expected-name (:name shape)) + (nil? expected-name)) + shape + (assoc shape :name expected-name)))) + + ;; Any other shape does not need processing + :else shape)] + + (reduce update-shape + (ctt/set-shape container shape) + (:shapes shape)))) + + (update-container + [container] + (let [root (ctn/get-container-root container)] + (if root + (update-shape container (:id root)) + container)))] + + (-> data + (update :pages-index d/update-vals update-container) + (d/update-when :components d/update-vals update-container)))) + (def available-migrations (into (d/ordered-set) ["legacy-2" @@ -2306,4 +2371,5 @@ "0029-move-background-blur-out-of-blur" "0030-remove-stroke-per-side-attr" "0031-migrate-stroke-width-token-attr" - "0032-fix-children-not-in-parent"])) + "0032-fix-children-not-in-parent" + "0033-normalize-main-instance-names"])) diff --git a/common/src/app/common/files/repair.cljc b/common/src/app/common/files/repair.cljc index a8d98a2d87..4756f8a758 100644 --- a/common/src/app/common/files/repair.cljc +++ b/common/src/app/common/files/repair.cljc @@ -7,6 +7,7 @@ (ns app.common.files.repair (:require [app.common.data :as d] + [app.common.data.macros :as dm] [app.common.files.changes-builder :as pcb] [app.common.files.helpers :as cfh] [app.common.files.tokens :as cfo] @@ -215,6 +216,20 @@ (pcb/with-library-data file-data) (pcb/update-component (:component-id shape) repair-component)))) +(defmethod repair-error :invalid-main-instance-name + [_ {:keys [shape page-id args] :as error} file-data _] + (let [repair-shape + (fn [shape] + ;; Change the name to match the component + (let [component-name (:component-name args)] + (log/debug :hint (str " -> rename shape " (:id shape) " to " component-name)) + (assoc shape :name component-name)))] + + (log/debug :hint "repairing shape :invalid-main-instance-name" :id (:id shape) :name (:name shape) :page-id page-id) + (-> (pcb/empty-changes nil page-id) + (pcb/with-file-data file-data) + (pcb/update-shapes [(:id shape)] repair-shape)))) + (defmethod repair-error :invalid-main-instance [_ {:keys [shape page-id] :as error} file-data _] (let [repair-shape @@ -729,6 +744,42 @@ (pcb/with-file-data file-data) (pcb/update-shapes [parent-id] repair-fn)))) +(defmethod repair-error :variant-container-bad-name + [_ {:keys [shape page-id args]} file-data _] + (let [clean-name (:clean-name args) + + repair-shape + (fn [shape] + (log/debug :hint " -> set :name of shape" :id (:id shape) :name clean-name) + (assoc shape :name clean-name)) + + repair-component + (fn [component] + (let [[path name] (cpn/split-group-name clean-name)] + (log/debug :hint " -> set :path and :name of component" :id (:id component) :path path :name name) + (assoc component :path path :name name))) + + all-shape-ids + (cons (:id shape) (:shapes shape)) + + objects + (dm/get-in file-data [:pages-index page-id :objects]) + + all-main-instances + (map (d/getf objects) (:shapes shape)) + + all-component-ids + (map :component-id all-main-instances)] + + (log/debug :hint "repairing shape :variant-container-bad-name" :id (:id shape) :name (:name shape) :page-id page-id) + (as-> (pcb/empty-changes nil page-id) $ + (pcb/with-file-data $ file-data) + (pcb/with-library-data $ file-data) + (pcb/update-shapes $ all-shape-ids repair-shape) + (reduce #(pcb/update-component %1 %2 repair-component) + $ + all-component-ids)))) + (defmethod repair-error :variant-main-bad-name [_ {:keys [shape page-id args]} file-data _] (let [repair-fn @@ -771,14 +822,22 @@ (fn [shape] (let [variant-id (:variant-id args)] ;; Set the desired variant-id - (log/debug :hint (str " -> set variant-id to " variant-id)) - (assoc shape - :variant-id variant-id)))] + (log/debug :hint (str " -> set variant-id in shape to " variant-id)) + (assoc shape :variant-id variant-id))) + + repair-component + (fn [component] + (let [variant-id (:variant-id args)] + ;; Set the desired variant-id + (log/debug :hint (str " -> set variant-id in component to " variant-id)) + (assoc component :variant-id variant-id)))] (log/debug :hint "repairing shape :variant-component-bad-id" :id (:id shape) :name (:name shape) :page-id page-id) (-> (pcb/empty-changes nil page-id) (pcb/with-file-data file-data) - (pcb/update-shapes [(:id shape)] repair-shape)))) + (pcb/with-library-data file-data) + (pcb/update-shapes [(:id shape)] repair-shape) + (pcb/update-component (:component-id shape) repair-component)))) (defmethod repair-error :missing-tokens-status [_ _ file-data _] diff --git a/common/src/app/common/files/tokens.cljc b/common/src/app/common/files/tokens.cljc index 34c5b760d6..e78adf2c47 100644 --- a/common/src/app/common/files/tokens.cljc +++ b/common/src/app/common/files/tokens.cljc @@ -547,6 +547,32 @@ [file-data f & args] (d/update-when file-data :tokens-status #(apply f % args))) +(defn absorb-tokens-lib + "Copy the tokens-lib of the library into the file, replacing the + previous one if any, and unset the tokens-source, so the file becomes + its own tokens source. + + The tokens-status of the file is preserved. If the file has none, the + one of the library is copied along with the tokens-lib. + + If the library has no tokens-lib, only the tokens-source is unset, and + the file keeps its own tokens-lib and tokens-status. If there is no + library-data, there is nothing to absorb and the file is returned + unchanged." + [file-data library-data] + (if (nil? library-data) + file-data + (let [tokens-lib (get-tokens-lib library-data)] + (if (nil? tokens-lib) + (set-tokens-source file-data nil) + (let [tokens-status (or (get-tokens-status file-data) + (get-tokens-status library-data))] + (cond-> (-> file-data + (set-tokens-source nil) + (assoc :tokens-lib tokens-lib)) + (some? tokens-status) + (assoc :tokens-status tokens-status))))))) + ;; Tokens status with tokens lib (defn- calculate-active-sets diff --git a/common/src/app/common/files/validate.cljc b/common/src/app/common/files/validate.cljc index 72a1e112ff..6654e28508 100644 --- a/common/src/app/common/files/validate.cljc +++ b/common/src/app/common/files/validate.cljc @@ -41,6 +41,7 @@ :duplicate-slot :invalid-main-instance-id :invalid-main-instance-page + :invalid-main-instance-name :invalid-main-instance :invalid-parent :component-main @@ -70,6 +71,7 @@ :invalid-variant-properties :variant-not-main :parent-not-variant + :variant-container-bad-name :variant-main-bad-name :variant-main-bad-variant-name :variant-component-bad-name @@ -302,7 +304,14 @@ shape file page) (report-error :invalid-main-instance-page (str/ffmt "Main instance page of component % is not valid" (:component-id shape)) - shape file page)))))))) + shape file page)))) + (when-not (ctk/is-variant? component) ;; Variants are checked elsewhere + (let [component-name (cpn/merge-path-item (:path component) (:name component))] + (when-not (= (:name shape) component-name) + (report-error :invalid-main-instance-name + (str/ffmt "Main instance % has an invalid name" (:id shape)) + shape file page + :component-name component-name)))))))) (defn- check-component-not-main-head "Validate shape is a not-main instance head, component @@ -570,14 +579,25 @@ (defn- check-variant-container "Shape is a variant container, so: -all its children should be variants with variant-id equals to the shape-id - -all the components should have the same properties" + -all the components should have the same properties + -the name of the container should be normalized for paths (separator / with one space at each side)." [shape file page] - (let [shape-id (:id shape) - shapes (:shapes shape) - objects (:objects page) - file-data (:data file) - first-child (get objects (first shapes)) - prop-names (cfv/extract-properties-names first-child file-data)] + (let [shape-id (:id shape) + container-name (:name shape) + shapes (:shapes shape) + objects (:objects page) + file-data (:data file) + first-child (get objects (first shapes)) + prop-names (cfv/extract-properties-names first-child file-data) + clean-path (cpn/clean-path container-name) + clean-path (if (str/blank? clean-path) + "(empty)" ;; this may occur if the name contains only separators + clean-path)] + (when-not (= container-name clean-path) + (report-error :variant-container-bad-name + (str/ffmt "Variant container % has a bad name %" shape-id container-name) + shape file page + :clean-name clean-path)) (run! (fn [child-id] (when-let [child (get objects child-id)] (if (not (ctk/is-variant? child)) @@ -598,6 +618,11 @@ :prop-names prop-names)))))) shapes))) +(defn- renamed-container? + [container] + (and (some? container) + (not= (:name container) (cpn/clean-path (:name container))))) + (defn- check-variant "Shape is a variant, so -it should be a main component @@ -622,21 +647,23 @@ (str/ffmt "Variant % has an invalid variant-name" (:id shape)) shape file page :variant-name variant-name)) - (when-not (= (:name parent) (:name shape)) - (report-error :variant-main-bad-name - (str/ffmt "Main instance inside variant % has an invalid name" (:id shape)) - shape file page - :variant-name (:name parent))) - (when-not (= (:name parent) (cpn/merge-path-item (:path component) (:name component))) - (report-error :variant-component-bad-name - (str/ffmt "Component % has an invalid name" (:id shape)) - shape file page - :variant-container-name (:name parent))) + ;; If the container is going to be renamed all main and components will be renamed, too + (when-not (renamed-container? parent) + (when-not (= (:name parent) (:name shape)) + (report-error :variant-main-bad-name + (str/ffmt "Main instance inside variant % has an invalid name" (:id shape)) + shape file page + :variant-name (:name parent))) + (when-not (= (:name parent) (cpn/merge-path-item (:path component) (:name component))) + (report-error :variant-component-bad-name + (str/ffmt "Component % has an invalid name" (:id shape)) + shape file page + :variant-container-name (:name parent)))) (when-not (= (:variant-id component) (:variant-id shape)) (report-error :variant-component-bad-id - (str/ffmt "Variant % has adifferent variant-id than its component" (:id shape)) + (str/ffmt "Variant % has a :variant-id that does not point to the container" (:id shape)) shape file page - :variant-id (:variant-id component))))) + :variant-id (:parent-id shape))))) (defn- check-shape "Validate referential integrity and semantic coherence of diff --git a/common/src/app/common/geom/image_crop.cljc b/common/src/app/common/geom/image_crop.cljc new file mode 100644 index 0000000000..fdd9f89527 --- /dev/null +++ b/common/src/app/common/geom/image_crop.cljc @@ -0,0 +1,38 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns app.common.geom.image-crop + "Geometry of the image crop (resize with the mod key on a shape with an + image fill). The shape is a window over a picture that keeps its size + and position; the window never flips." + (:require + [app.common.math :as mth])) + +(defn crop-axis + "Resize one axis of the crop window. `size` is the current window size, + `origin` the fixed point relative to the window start and `scale` the + resize factor, negative when the dragged edge crosses the fixed point. + Returns `[offset size]`: the new window start relative to the old one, + and the new size, always positive." + [size origin scale] + (let [start (* origin (- 1 scale)) + end (+ origin (* (- size origin) scale))] + [(mth/min start end) (mth/abs (- end start))])) + +(defn update-image-transform + "Return the image fill transform that keeps the picture in place when + the window of `width` x `height` moves by `dx`, `dy` and gets a size of + `width'` x `height'`. The transform values are relative to the window + size, and a nil transform means the picture fills the window." + [transform width height dx dy width' height'] + (let [x (get transform :x 0.0) + y (get transform :y 0.0) + w (get transform :width 1.0) + h (get transform :height 1.0)] + {:x (/ (- (* x width) dx) width') + :y (/ (- (* y height) dy) height') + :width (/ (* w width) width') + :height (/ (* h height) height')})) diff --git a/common/src/app/common/test_helpers/variants.cljc b/common/src/app/common/test_helpers/variants.cljc index 8bbadf4c40..34f0cc8467 100644 --- a/common/src/app/common/test_helpers/variants.cljc +++ b/common/src/app/common/test_helpers/variants.cljc @@ -62,7 +62,6 @@ (thc/make-component component2-label root2-label) (thc/update-component component2-label {:variant-id variant-id :variant-properties [{:name "Property 1" :value "Value2"}]})))) - (defn add-variant-with-copy [file variant-label component1-label root1-label component2-label root2-label child1-label child2-label component-copy-label] (let [file (ths/add-sample-shape file variant-label :type :frame :is-variant-container true) @@ -77,8 +76,6 @@ (thc/make-component component2-label root2-label) (thc/update-component component2-label {:variant-id variant-id :variant-properties [{:name "Property 1" :value "Value2"}]})))) - - (defn add-variant-with-text [file variant-label component1-label root1-label component2-label root2-label child1-label child2-label text1 text2 & {:keys [text1-params text2-params]}] diff --git a/common/src/app/common/types/file.cljc b/common/src/app/common/types/file.cljc index c4ab29e3da..3a52dd79ca 100644 --- a/common/src/app/common/types/file.cljc +++ b/common/src/app/common/types/file.cljc @@ -11,6 +11,7 @@ [app.common.features :as cfeat] [app.common.files.defaults :refer [version]] [app.common.files.helpers :as cfh] + [app.common.files.tokens :as cfo] [app.common.geom.point :as gpt] [app.common.geom.shapes :as gsh] [app.common.geom.shapes.tree-seq :as gsts] @@ -1238,7 +1239,9 @@ (fn [pages-index] (d/update-vals pages-index #(update % :objects detach-objects)))] - (update-in file [:data :pages-index] detach-pages))) + (cond-> (update-in file [:data :pages-index] detach-pages) + (not (cfo/effective-tokens-source? (:data file) file-id)) + (update :data dissoc :tokens-source :tokens-status)))) ;; Base font size diff --git a/common/test/common_tests/exceptions_test.cljc b/common/test/common_tests/exceptions_test.cljc new file mode 100644 index 0000000000..6f9df8da60 --- /dev/null +++ b/common/test/common_tests/exceptions_test.cljc @@ -0,0 +1,57 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns common-tests.exceptions-test + "Exception message bounding in `app.common.exceptions`. + + A message can carry a whole payload (a serialized file, for instance) and + grow to megabytes. `get-hint` and the CLJS `format-throwable` keep a bounded + head so neither the audit `:hint` prop nor the report can explode." + (:require + #?(:cljs [cuerdas.core :as str]) + [app.common.exceptions :as ex] + [clojure.test :as t])) + +(defn- huge-string + [n] + (apply str (repeat n "x"))) + +(t/deftest first-line-keeps-a-single-line-message-intact-test + (t/is (= "boom" (ex/first-line "boom"))) + (t/is (= "boom" (ex/first-line "boom\nrest of the message")))) + +(t/deftest get-hint-returns-the-first-line-test + (t/is (= "boom" (ex/get-hint (ex-info "boom\nsecond line" {}))))) + +(t/deftest get-hint-prefers-the-ex-data-hint-test + (t/is (= "from-data" (ex/get-hint (ex-info "from-message" {:hint "from-data"}))))) + +(t/deftest get-hint-bounds-a-huge-message-test + (let [huge (huge-string (* 4 ex/max-message-length)) + hint (ex/get-hint (ex-info huge {}))] + (t/is (string? hint)) + (t/is (<= (count hint) (+ ex/max-message-length 3))) + (t/is (= (subs hint 0 32) (subs huge 0 32))))) + +(t/deftest get-hint-bounds-a-huge-ex-data-hint-test + (let [huge (huge-string (* 4 ex/max-message-length)) + hint (ex/get-hint (ex-info "short" {:hint huge}))] + (t/is (<= (count hint) (+ ex/max-message-length 3))) + (t/is (= (subs hint 0 32) (subs huge 0 32))))) + +#?(:cljs + (t/deftest format-throwable-bounds-a-huge-trace-message-test + (let [huge (huge-string (* 4 ex/max-message-length)) + out (ex/format-throwable (ex-info huge {})) + msg (some #(when (str/includes? % (subs huge 0 16)) %) (str/lines out))] + (t/is (some? msg)) + (t/is (<= (count msg) (+ ex/max-message-length 16)))))) + +#?(:cljs + (t/deftest format-throwable-honors-trace-length-test + (let [full (ex/format-throwable (ex-info "boom" {})) + short (ex/format-throwable (ex-info "boom" {}) {:trace-length 1})] + (t/is (< (count (str/lines short)) (count (str/lines full))))))) diff --git a/common/test/common_tests/files/repair_test.cljc b/common/test/common_tests/files/repair_test.cljc index d5aba47c98..27f6964de2 100644 --- a/common/test/common_tests/files/repair_test.cljc +++ b/common/test/common_tests/files/repair_test.cljc @@ -15,6 +15,7 @@ [app.common.files.repair :as cfr] [app.common.files.validate :as cfv] [app.common.test-helpers.components :as thc] + [app.common.test-helpers.compositions :as tho] [app.common.test-helpers.files :as thf] [app.common.test-helpers.ids-map :as thi] [app.common.test-helpers.shapes :as ths] @@ -24,6 +25,48 @@ (t/use-fixtures :each thi/test-fixture) +(t/deftest repair-invalid-main-instance-name + (t/testing "detect and repair a main instance whose name doesn't match the component's name" + (let [file (-> (thf/sample-file :file1 :page-label :page1) + (tho/add-simple-component-with-copy :component1 :root1 :child1 :copy1) + (tho/add-simple-component-with-copy :component2 :root2 :child2 :copy2) + + ;; Update component paths and names + (thc/update-component :component1 {:path "Group / Subgroup" :name "Component1"}) + (thc/update-component :component2 {:path "Group / Subgroup" :name "Component2"}) + + ;; Update component1 shapes name to have path structure + (ths/update-shape :root1 :name "Group / Subgroup / Component1") + (ths/update-shape :copy1 :name "Group / Subgroup / Component1") + + ;; Update component2's name to have unnormalized path + ;; Note that the copy name is allowed to be different + (ths/update-shape :root2 :name "Group/Subgroup / Component2") + (ths/update-shape :copy2 :name "Group/Subgroup / Component2")) + + errors (cfv/validate-file file {}) + changes (cfr/repair-file file {} errors) + file' (thf/apply-changes file {:redo-changes changes} :validate? false) + errors' (cfv/validate-file file' {}) + root1' (ths/get-shape file' :root1) + child1' (ths/get-shape file' :child1) + copy1' (ths/get-shape file' :copy1) + root2' (ths/get-shape file' :root2) + child2' (ths/get-shape file' :child2) + copy2' (ths/get-shape file' :copy2)] + + (t/is (= 1 (count errors))) + (t/is (= :invalid-main-instance-name (:code (first errors)))) + (t/is (nil? errors')) + + (t/is (= "Group / Subgroup / Component1" (:name root1'))) + (t/is (= "Rect1" (:name child1'))) + (t/is (= "Group / Subgroup / Component1" (:name copy1'))) + + (t/is (= "Group / Subgroup / Component2" (:name root2'))) + (t/is (= "Rect1" (:name child2'))) + (t/is (= "Group/Subgroup / Component2" (:name copy2')))))) + (t/deftest repair-main-instance-not-a-variant (t/testing "detect and repair a variant component whose root shape is not a variant" (let [file (-> (thf/sample-file :file1 :page-label :page1) @@ -44,7 +87,7 @@ (t/is (= (thi/id :variant1) (:variant-id root1')))))) (t/deftest repair-invalid-variant-id-variant-component-bad-id - (t/testing "detect and repair a variant component whose variant id does not match the container's id" + (t/testing "detect and repair a variant main whose variant id does not match the container's id" (let [file (-> (thf/sample-file :file1 :page-label :page1) (thv/add-variant :variant1 :component1 :root1 :component2 :root2) (ths/update-shape :root1 :variant-id (uuid/next))) @@ -61,7 +104,25 @@ (t/is (= :variant-component-bad-id (:code (second errors)))) (t/is (nil? errors')) - (t/is (= (thi/id :variant1) (:variant-id root1')))))) + (t/is (= (thi/id :variant1) (:variant-id root1'))))) + + (t/testing "detect and repair a variant component whose variant id does not match the container's id" + (let [file (-> (thf/sample-file :file1 :page-label :page1) + (thv/add-variant :variant1 :component1 :root1 :component2 :root2) + (thc/update-component :component1 {:variant-id (uuid/next)})) + + errors (cfv/validate-file file {}) + changes (cfr/repair-file file {} errors) + file' (thf/apply-changes file {:redo-changes changes} :validate? false) + errors' (cfv/validate-file file' {}) + + component1' (thc/get-component file' :component1)] + + (t/is (= 1 (count errors))) + (t/is (= :variant-component-bad-id (:code (first errors)))) + + (t/is (nil? errors')) + (t/is (= (thi/id :variant1) (:variant-id component1')))))) (t/deftest repair-invalid-variant-properties (t/testing "detect and repair a second variant component whose properties do not match the first variant component's properties" @@ -161,23 +222,60 @@ (t/is (true? (:is-variant-container container')))))) +(t/deftest repair-variant-container-bad-name + (t/testing "detect and repair a variant container who has an unnormalized name" + (let [file (-> (thf/sample-file :file1 :page-label :page1) + (thv/add-variant :variant1 :component1 :root1 :component2 :root2) + ;; Variant container has a name that is not normalized and thus it cannot + ;; match the component names, that are separated in path and group. + (ths/update-shape :variant1 :name "Group / Subgroup/Component") + (ths/update-shape :root1 :name "Group / Subgroup/Component") + (ths/update-shape :root2 :name "Group / Subgroup/Component") + (thc/update-component :component1 {:path "Group / Subgroup" :name "Component"}) + (thc/update-component :component2 {:path "Group/Subgroup" :name "Component"})) + + errors (cfv/validate-file file {}) + changes (cfr/repair-file file {} errors) + file' (thf/apply-changes file {:redo-changes changes} :validate? false) + errors' (cfv/validate-file file' {}) + var1' (ths/get-shape file' :variant1) + root1' (ths/get-shape file' :root1) + comp1' (thc/get-component file' :component1) + root2' (ths/get-shape file' :root2) + comp2' (thc/get-component file' :component2)] + + (t/is (= 1 (count errors))) + (t/is (= :variant-container-bad-name (:code (first errors)))) + (t/is (nil? errors')) + (t/is (= "Group / Subgroup / Component" (:name var1'))) + (t/is (= "Group / Subgroup / Component" (:name root1'))) + (t/is (= "Group / Subgroup" (:path comp1'))) + (t/is (= "Component" (:name comp1'))) + (t/is (= "Group / Subgroup / Component" (:name root2'))) + (t/is (= "Group / Subgroup" (:path comp2'))) + (t/is (= "Component" (:name comp2')))))) + (t/deftest repair-variant-main-bad-name (t/testing "detect and repair a main instance whose name doesn't match the variant container's name" (let [file (-> (thf/sample-file :file1 :page-label :page1) (thv/add-variant :variant1 :component1 :root1 :component2 :root2) - ;; Change root1's name so it doesn't match the container + ;; Change component's name so it doesn't match the container + (thc/update-component :component1 :name "WrongName") (ths/update-shape :root1 :name "WrongName")) errors (cfv/validate-file file {}) changes (cfr/repair-file file {} errors) file' (thf/apply-changes file {:redo-changes changes} :validate? false) errors' (cfv/validate-file file' {}) - root1' (ths/get-shape file' :root1)] + root1' (ths/get-shape file' :root1) + comp1' (thc/get-component file' :component1)] - (t/is (= 1 (count errors))) + (t/is (= 2 (count errors))) (t/is (= :variant-main-bad-name (:code (first errors)))) + (t/is (= :variant-component-bad-name (:code (second errors)))) (t/is (nil? errors')) - (t/is (= "Board" (:name root1')))))) + (t/is (= "Board" (:name root1'))) + (t/is (= "Board" (:name comp1')))))) (t/deftest repair-variant-main-bad-variant-name (t/testing "detect and repair a variant shape whose :variant-name doesn't match the component's properties" @@ -211,23 +309,28 @@ (ths/update-shape :variant1 :name "Group / Subgroup / Component") (ths/update-shape :root1 :name "Group / Subgroup / Component") (ths/update-shape :root2 :name "Group / Subgroup / Component") - ;; Update component paths and names (thc/update-component :component1 {:path "Group / Subgroup" :name "Component"}) - (thc/update-component :component2 {:path "Group / Subgroup" :name "Component"}) - ;; Break component1's name - (thc/update-component :component1 {:name "WrongName"})) + ;; Break component2's name + (thc/update-component :component2 {:name "WrongName"})) errors (cfv/validate-file file {}) changes (cfr/repair-file file {} errors) file' (thf/apply-changes file {:redo-changes changes} :validate? false) errors' (cfv/validate-file file' {}) - comp1' (thc/get-component file' :component1)] + root1' (ths/get-shape file' :root1) + comp1' (thc/get-component file' :component1) + root2' (ths/get-shape file' :root2) + comp2' (thc/get-component file' :component2)] (t/is (= 1 (count errors))) (t/is (= :variant-component-bad-name (:code (first errors)))) (t/is (nil? errors')) + (t/is (= "Group / Subgroup / Component" (:name root1'))) (t/is (= "Group / Subgroup" (:path comp1'))) - (t/is (= "Component" (:name comp1')))))) + (t/is (= "Component" (:name comp1'))) + (t/is (= "Group / Subgroup / Component" (:name root2'))) + (t/is (= "Group / Subgroup" (:path comp2'))) + (t/is (= "Component" (:name comp2')))))) (t/deftest repair-child-not-in-parent (t/testing "detect and repair shapes whose parent exists but does not list them" diff --git a/common/test/common_tests/files/tokens_test.cljc b/common/test/common_tests/files/tokens_test.cljc index 76da45bd4e..8d6a28b393 100644 --- a/common/test/common_tests/files/tokens_test.cljc +++ b/common/test/common_tests/files/tokens_test.cljc @@ -1494,3 +1494,218 @@ ;; set-theme-active false removes non-existent id (let [status' (cfo/set-theme-active all-invalid tokens-lib invalid-theme-id false)] (t/is (not (ctos/theme-active? status' invalid-theme-id)))))))) + +;; absorb-tokens-lib + +(defn- make-sample-tokens-lib + "A tokens-lib with a set (and a token in it) for each label. Each set + is named as its label, and its id is registered in thi under it." + [& set-labels] + (reduce (fn [tokens-lib set-label] + (let [set-id (thi/new-id! set-label)] + (-> tokens-lib + (ctob/add-set (ctob/make-token-set :id set-id + :name (name set-label))) + (ctob/add-token set-id (ctob/make-token :name (str (name set-label) "-token") + :type :border-radius + :value 10))))) + (ctob/make-tokens-lib) + set-labels)) + +(defn- make-sample-tokens-status + "A tokens-status with the sets of the given labels active. The sets + must have been created before by make-sample-tokens-lib." + [& set-labels] + (ctos/make-tokens-status :active-set-ids (into #{} (map thi/id) set-labels))) + +(defn- make-sample-data + "File data with the given tokens-lib, tokens-status and tokens-source. + Those not given are left unset." + [label & {:keys [tokens-lib tokens-status tokens-source]}] + (-> (ctf/file-data (thf/sample-file label)) + (cfo/ensure-tokens-lib) + (cfo/update-tokens-lib (constantly tokens-lib)) + (cfo/update-tokens-status (constantly tokens-status)) + (cfo/set-tokens-source tokens-source))) + +(t/deftest test-absorb-tokens-lib + (t/testing "copies the tokens-lib of the library and unsets the tokens-source" + (let [library-data (make-sample-data :library + :tokens-lib (make-sample-tokens-lib :lib-set)) + file-data (make-sample-data :file1 + :tokens-source (:id library-data)) + file-data' (cfo/absorb-tokens-lib file-data library-data)] + (t/is (= (cfo/get-tokens-lib library-data) (cfo/get-tokens-lib file-data'))) + (t/is (nil? (cfo/get-tokens-source file-data'))) + (t/is (true? (cfo/effective-tokens-source? file-data' (:id file-data')))) + (t/is (true? (cfo/editable-tokens? file-data'))))) + + (t/testing "replaces the own tokens-lib of the file" + (let [library-data (make-sample-data :library + :tokens-lib (make-sample-tokens-lib :lib-set)) + file-data (make-sample-data :file1 + :tokens-lib (make-sample-tokens-lib :own-set) + :tokens-source (:id library-data)) + file-data' (cfo/absorb-tokens-lib file-data library-data)] + (t/is (= (cfo/get-tokens-lib library-data) (cfo/get-tokens-lib file-data'))) + (t/is (nil? (cfo/get-tokens-source file-data'))))) + + (t/testing "copies an empty tokens-lib of the library" + (let [empty-lib (ctob/make-tokens-lib) + library-data (make-sample-data :library :tokens-lib empty-lib) + file-data (make-sample-data :file1 + :tokens-lib (make-sample-tokens-lib :own-set) + :tokens-source (:id library-data)) + file-data' (cfo/absorb-tokens-lib file-data library-data)] + (t/is (= empty-lib (cfo/get-tokens-lib file-data'))) + (t/is (nil? (cfo/get-tokens-source file-data'))))) + + (t/testing "unsets the tokens-source even if the file had no tokens-source" + (let [library-data (make-sample-data :library + :tokens-lib (make-sample-tokens-lib :lib-set)) + file-data (make-sample-data :file1) + file-data' (cfo/absorb-tokens-lib file-data library-data)] + (t/is (= (cfo/get-tokens-lib library-data) (cfo/get-tokens-lib file-data'))) + (t/is (nil? (cfo/get-tokens-source file-data'))))) + + (t/testing "does not change anything else in the file" + (let [library-data (make-sample-data :library + :tokens-lib (make-sample-tokens-lib :lib-set)) + file-data (make-sample-data :file1 + :tokens-source (:id library-data)) + file-data' (cfo/absorb-tokens-lib file-data library-data)] + (t/is (= (dissoc file-data :tokens-lib :tokens-status :tokens-source) + (dissoc file-data' :tokens-lib :tokens-status :tokens-source)))))) + +(t/deftest test-absorb-tokens-lib-status + (t/testing "preserves the tokens-status of the file" + (let [library-lib (make-sample-tokens-lib :lib-set :other-set) + file-status (make-sample-tokens-status :lib-set) + library-status (make-sample-tokens-status :other-set) + library-data (make-sample-data :library + :tokens-lib library-lib + :tokens-status library-status) + file-data (make-sample-data :file1 + :tokens-status file-status + :tokens-source (:id library-data)) + file-data' (cfo/absorb-tokens-lib file-data library-data)] + (t/is (= file-status (cfo/get-tokens-status file-data'))) + (t/is (not= library-status (cfo/get-tokens-status file-data'))))) + + (t/testing "preserves an empty tokens-status of the file" + (let [library-lib (make-sample-tokens-lib :lib-set) + file-status (ctos/make-tokens-status) + library-status (make-sample-tokens-status :lib-set) + library-data (make-sample-data :library + :tokens-lib library-lib + :tokens-status library-status) + file-data (make-sample-data :file1 + :tokens-status file-status + :tokens-source (:id library-data)) + file-data' (cfo/absorb-tokens-lib file-data library-data)] + (t/is (= file-status (cfo/get-tokens-status file-data'))))) + + (t/testing "copies the tokens-status of the library if the file has none" + (let [library-lib (make-sample-tokens-lib :lib-set) + library-status (make-sample-tokens-status :lib-set) + library-data (make-sample-data :library + :tokens-lib library-lib + :tokens-status library-status) + file-data (make-sample-data :file1 + :tokens-source (:id library-data)) + file-data' (cfo/absorb-tokens-lib file-data library-data)] + (t/is (= library-status (cfo/get-tokens-status file-data'))) + (t/is (= #{(thi/id :lib-set)} + (ctos/get-active-set-ids (cfo/get-tokens-status file-data')))) + (t/is (some? (ctob/get-set (cfo/get-tokens-lib file-data') (thi/id :lib-set)))))) + + (t/testing "leaves the tokens-status unset if neither the file nor the library have one" + (let [library-data (make-sample-data :library + :tokens-lib (make-sample-tokens-lib :lib-set)) + file-data (make-sample-data :file1 + :tokens-source (:id library-data)) + file-data' (cfo/absorb-tokens-lib file-data library-data)] + (t/is (nil? (cfo/get-tokens-status file-data')))))) + +(t/deftest test-absorb-tokens-lib-library-without-tokens + (t/testing "library without tokens-lib: keeps the tokens-lib and tokens-status of the file, and unsets the tokens-source" + (let [file-lib (make-sample-tokens-lib :own-set) + file-status (make-sample-tokens-status :own-set) + library-status (ctos/make-tokens-status :active-set-ids #{(uuid/next)}) + library-data (make-sample-data :library + :tokens-status library-status) + file-data (make-sample-data :file1 + :tokens-lib file-lib + :tokens-status file-status + :tokens-source (:id library-data)) + file-data' (cfo/absorb-tokens-lib file-data library-data)] + (t/is (= file-lib (cfo/get-tokens-lib file-data'))) + (t/is (= file-status (cfo/get-tokens-status file-data'))) + (t/is (nil? (cfo/get-tokens-source file-data'))))) + + (t/testing "library without tokens-lib: does not copy the tokens-status of the library" + (let [library-status (ctos/make-tokens-status :active-set-ids #{(uuid/next)}) + library-data (make-sample-data :library + :tokens-status library-status) + file-data (make-sample-data :file1 + :tokens-source (:id library-data)) + file-data' (cfo/absorb-tokens-lib file-data library-data)] + (t/is (nil? (cfo/get-tokens-lib file-data'))) + (t/is (nil? (cfo/get-tokens-status file-data'))) + (t/is (nil? (cfo/get-tokens-source file-data'))))) + + (t/testing "library without any tokens attribute" + (let [library-data (ctf/file-data (thf/sample-file :library)) + file-lib (make-sample-tokens-lib :own-set) + file-data (make-sample-data :file1 + :tokens-lib file-lib + :tokens-source (:id library-data)) + file-data' (cfo/absorb-tokens-lib file-data library-data)] + (t/is (= file-lib (cfo/get-tokens-lib file-data'))) + (t/is (nil? (cfo/get-tokens-source file-data'))))) + + (t/testing "neither the file nor the library have tokens" + (let [library-data (ctf/file-data (thf/sample-file :library)) + file-data (ctf/file-data (thf/sample-file :file1)) + file-data' (cfo/absorb-tokens-lib file-data library-data)] + (t/is (nil? (cfo/get-tokens-lib file-data'))) + (t/is (nil? (cfo/get-tokens-status file-data'))) + (t/is (nil? (cfo/get-tokens-source file-data')))))) + +(t/deftest test-absorb-tokens-lib-no-library-data + (t/testing "nil library-data: the file is returned unchanged" + (let [file-lib (make-sample-tokens-lib :own-set) + file-data (make-sample-data :file1 + :tokens-lib file-lib + :tokens-status (make-sample-tokens-status :own-set) + :tokens-source (uuid/next))] + (t/is (identical? file-data (cfo/absorb-tokens-lib file-data nil))))) + + (t/testing "nil library-data and a file without tokens" + (let [file-data (ctf/file-data (thf/sample-file :file1))] + (t/is (identical? file-data (cfo/absorb-tokens-lib file-data nil)))))) + +(t/deftest test-detach-external-references-preserves-own-tokens + (let [file (thf/sample-file :file1) + lib (make-sample-tokens-lib :own-set) + status (make-sample-tokens-status :own-set) + file (-> file + (assoc-in [:data :tokens-lib] lib) + (assoc-in [:data :tokens-status] status))] + (doseq [source [nil (:id file)]] + (t/testing (str "local tokens source " source) + (let [file (cond-> file + source (assoc-in [:data :tokens-source] source)) + detached (ctf/detach-external-references file (:id file))] + (t/is (= source (cfo/get-tokens-source (:data detached)))) + (t/is (identical? lib (cfo/get-tokens-lib (:data detached)))) + (t/is (= status (cfo/get-tokens-status (:data detached))))))))) + +(t/deftest test-detach-external-references-removes-external-tokens-status + (let [file (-> (thf/sample-file :file1) + (assoc-in [:data :tokens-source] (uuid/next)) + (assoc-in [:data :tokens-status] (ctos/make-tokens-status))) + detached (ctf/detach-external-references file (:id file))] + (t/is (not (contains? (:data detached) :tokens-source))) + (t/is (not (contains? (:data detached) :tokens-status))) + (t/is (= detached (ctf/check-file detached))))) diff --git a/common/test/common_tests/files_builder_test.cljc b/common/test/common_tests/files_builder_test.cljc index 707d56d3f0..23891a583b 100644 --- a/common/test/common_tests/files_builder_test.cljc +++ b/common/test/common_tests/files_builder_test.cljc @@ -70,3 +70,20 @@ (t/is (= "Image" (:name media))) (t/is (= 128 (:width media))) (t/is (= 64 (:height media))))) + +(t/deftest add-shape-normalizes-path-name + (let [file-id (uuid/next) + page-id (uuid/next) + shape-id (uuid/next) + state (-> (fb/create-state) + (fb/add-file {:id file-id :name "Test file"}) + (fb/add-page {:id page-id :name "Page 1"}) + (fb/add-shape {:id shape-id + :type :frame + :name "Group/Subgroup / Name" + :x 0 + :y 0 + :width 10 + :height 10})) + shape (fb/get-shape state shape-id)] + (t/is (= "Group / Subgroup / Name" (:name shape))))) diff --git a/common/test/common_tests/files_migrations_test.cljc b/common/test/common_tests/files_migrations_test.cljc index 0ac8084708..61ba0e7486 100644 --- a/common/test/common_tests/files_migrations_test.cljc +++ b/common/test/common_tests/files_migrations_test.cljc @@ -9,12 +9,20 @@ [app.common.data :as d] [app.common.files.migrations :as cfm] [app.common.files.tokens :as cfo] + [app.common.test-helpers.components :as thk] + [app.common.test-helpers.compositions :as tho] + [app.common.test-helpers.files :as thf] + [app.common.test-helpers.ids-map :as thi] + [app.common.test-helpers.shapes :as ths] + [app.common.test-helpers.variants :as thv] [app.common.types.file :as ctf] [app.common.types.shape :as cts] [app.common.types.token :as ctt] [app.common.uuid :as uuid] [clojure.test :as t])) +(t/use-fixtures :each thi/test-fixture) + (defmethod cfm/migrate-data "test/1" [data _] (update data :sum inc)) (defmethod cfm/migrate-data "test/2" [data _] (update data :sum inc)) (defmethod cfm/migrate-data "test/3" [data _] (update data :sum inc)) @@ -605,3 +613,140 @@ data' (cfm/migrate-data data migration-id)] (t/is (= [child-id] (get-in data' [:components component-id :objects main-id :shapes]))))) + +(defn- make-path-shape + [shape-id] + (cts/setup-shape {:id shape-id + :type :path + :content [{:command :move-to :params {:x 10 :y 20}} + {:command :line-to :params {:x 110 :y 70}}]})) + +(t/deftest migration-legacy-6-repairs-paths-without-points + (let [migration-id "legacy-6" + page-id (uuid/next) + shape-id (uuid/next) + shape (make-path-shape shape-id)] + + (doseq [points [nil []]] + (let [data {:pages-index {page-id {:objects {shape-id (assoc shape :points points)}}}} + data' (cfm/migrate-data data migration-id) + shape' (get-in data' [:pages-index page-id :objects shape-id])] + + (t/is (= (:points shape) (:points shape')) + (str "points rebuilt from content when they were " (pr-str points))) + (t/is (= (:selrect shape) (:selrect shape')) + "selrect rebuilt from content"))))) + +(t/deftest migration-legacy-6-runs-through-file-migration + ;; A file saved before version 6 runs every migration from legacy-6 on, + ;; and the whole chain must succeed for the file to load. + (let [migration-id "legacy-6" + shape-id (uuid/next) + file (ctf/make-file {:name "Legacy path without points"}) + page-id (first (get-in file [:data :pages])) + shape (-> (make-path-shape shape-id) + (assoc :points [])) + file (-> file + (dissoc :migrations) + (assoc :version 5) + (assoc-in [:data :pages-index page-id :objects shape-id] shape)) + file' (cfm/migrate-file file (delay {}))] + + (t/is (contains? (:migrations file') migration-id) "migration id persisted") + (t/is (= 4 (count (get-in file' [:data :pages-index page-id :objects shape-id :points]))) + "path points repaired before schema validation"))) + +(t/deftest migration-0033-normalize-main-instance-diff-from-component + (let [migration-id "0033-normalize-main-instance-names" + file (-> (thf/sample-file :file1) + (tho/add-simple-component :component1 :root1 :child1 + :component-params {:path "Group / Subgroup" + :name "Name"} + :root-params {:name "Group/Subgroup / Name"}) + (tho/add-simple-component :component2 :root2 :child2 + :component-params {:path "Group / Subgroup" + :name "Other Name"} + :root-params {:name "Group / Subgroup / Other Name"}) + (ths/add-sample-shape :shape1 :name "Other Group/Other Subgroup / Other Name")) + + data (ctf/file-data file) + data' (cfm/migrate-data data migration-id) + file' (assoc file :data data') + + root1' (ths/get-shape file' :root1) + component1' (thk/get-component file' :component1) + root2' (ths/get-shape file' :root2) + component2' (thk/get-component file' :component2) + shape1' (ths/get-shape file' :shape1)] + + (t/is (= "Group / Subgroup" (:path component1'))) + (t/is (= "Name" (:name component1'))) + (t/is (= "Group / Subgroup / Name" (:name root1'))) + (t/is (= "Group / Subgroup" (:path component2'))) + (t/is (= "Other Name" (:name component2'))) + (t/is (= "Group / Subgroup / Other Name" (:name root2'))) + (t/is (= "Other Group/Other Subgroup / Other Name" (:name shape1'))))) + +(t/deftest migration-0033-normalize-variant-instance-diff-from-container + (let [migration-id "0033-normalize-main-instance-names" + file (-> (thf/sample-file :file1) + (thv/add-variant :variant1 :component1 :root1 :component2 :root2) + (ths/update-shape :variant1 :name "Group / Subgroup / Component") + (ths/update-shape :root1 :name "Group / Subgroup / Component") + (ths/update-shape :root2 :name "Group / Subgroup/Component") ;; Bad path + (thk/update-component :component1 {:path "Group / Subgroup" :name "Component"}) + (thk/update-component :component2 {:path "Group / Subgroup" :name "Component"}) + (thk/instantiate-component :component2 :copy2) + (ths/update-shape :copy2 :name "Group / Subgroup/Component")) ;; Allowed name in copies + + data (ctf/file-data file) + data' (cfm/migrate-data data migration-id) + file' (assoc file :data data') + + variant1' (ths/get-shape file' :variant1) + root1' (ths/get-shape file' :root1) + component1' (thk/get-component file' :component1) + root2' (ths/get-shape file' :root2) + component2' (thk/get-component file' :component2) + copy2' (ths/get-shape file' :copy2)] + + (t/is (= "Group / Subgroup / Component" (:name variant1'))) + (t/is (= "Group / Subgroup / Component" (:name root1'))) + (t/is (= "Group / Subgroup / Component" (:name root2'))) + (t/is (= "Group / Subgroup" (:path component1'))) + (t/is (= "Group / Subgroup" (:path component2'))) + (t/is (= "Component" (:name component1'))) + (t/is (= "Component" (:name component2'))) + (t/is (= "Group / Subgroup/Component" (:name copy2'))))) + +(t/deftest migration-0033-normalize-variant-name-not-normalized + (let [migration-id "0033-normalize-main-instance-names" + file (-> (thf/sample-file :file1) + (thv/add-variant :variant1 :component1 :root1 :component2 :root2) + (ths/update-shape :variant1 :name "Group/Subgroup / Component") + (ths/update-shape :root1 :name "Group/Subgroup / Component") + (ths/update-shape :root2 :name "Group/Subgroup / Component") ;; Bad path + (thk/update-component :component1 {:path "Group / Subgroup" :name "Component"}) + (thk/update-component :component2 {:path "Group / Subgroup" :name "Component"}) + (thk/instantiate-component :component2 :copy2) + (ths/update-shape :copy2 :name "Group / Subgroup/Component")) ;; Allowed name in copies + + data (ctf/file-data file) + data' (cfm/migrate-data data migration-id) + file' (assoc file :data data') + + variant1' (ths/get-shape file' :variant1) + root1' (ths/get-shape file' :root1) + component1' (thk/get-component file' :component1) + root2' (ths/get-shape file' :root2) + component2' (thk/get-component file' :component2) + copy2' (ths/get-shape file' :copy2)] + + (t/is (= "Group / Subgroup / Component" (:name variant1'))) + (t/is (= "Group / Subgroup / Component" (:name root1'))) + (t/is (= "Group / Subgroup / Component" (:name root2'))) + (t/is (= "Group / Subgroup" (:path component1'))) + (t/is (= "Group / Subgroup" (:path component2'))) + (t/is (= "Component" (:name component1'))) + (t/is (= "Component" (:name component2'))) + (t/is (= "Group / Subgroup/Component" (:name copy2'))))) diff --git a/common/test/common_tests/geom_image_crop_test.cljc b/common/test/common_tests/geom_image_crop_test.cljc new file mode 100644 index 0000000000..cdd30e444f --- /dev/null +++ b/common/test/common_tests/geom_image_crop_test.cljc @@ -0,0 +1,67 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns common-tests.geom-image-crop-test + (:require + [app.common.geom.image-crop :as gic] + [app.common.math :as mth] + [clojure.test :as t])) + +(defn- close-vals? + [a b] + (every? true? (map mth/close? a b))) + +(defn- picture-box + "Picture position and size relative to the original window start, for a + window at `offset` with `size` and the given image transform." + [{:keys [x y width height]} [dx dy] [w h]] + [(+ dx (* x w)) (+ dy (* y h)) (* width w) (* height h)]) + +(t/deftest crop-axis-test + (t/testing "shrinks the window towards the fixed edge" + ;; Top handler: the bottom edge (300) stays fixed. + (t/is (close-vals? [60 240] (gic/crop-axis 300 300 0.8)))) + + (t/testing "grows the window away from the fixed edge" + (t/is (close-vals? [-60 360] (gic/crop-axis 300 300 1.2)))) + + (t/testing "crossing the fixed edge moves the window to the other side" + ;; Top handler dragged 100px below the bottom edge. + (t/is (close-vals? [300 100] (gic/crop-axis 300 300 (/ -1 3)))) + ;; Bottom handler dragged 150px above the top edge. + (t/is (close-vals? [-150 150] (gic/crop-axis 300 0 -0.5)))) + + (t/testing "resize from the center keeps the center when crossing" + (t/is (close-vals? [75 150] (gic/crop-axis 300 150 -0.5))))) + +(t/deftest update-image-transform-test + (let [width 400 + height 300 + tf {:x 0 :y -0.2 :width 1 :height 1.4}] + + (t/testing "the picture keeps its position and size" + (doseq [[dx dy w' h'] [[0 60 400 240] + [0 300 400 100] + [0 -150 400 150] + [-50 75 500 150]]] + (let [tf' (gic/update-image-transform tf width height dx dy w' h')] + (t/is (close-vals? (picture-box tf [0 0] [width height]) + (picture-box tf' [dx dy] [w' h'])) + (str "window " [dx dy w' h']))))) + + (t/testing "a missing transform means the picture fills the window" + (t/is (close-vals? [0 -0.25 1 1.25] + ((juxt :x :y :width :height) + (gic/update-image-transform nil 400 300 0 60 400 240))))) + + (t/testing "moving the window back restores the original transform" + (let [[dy h'] (gic/crop-axis height height (/ -1 3)) + crossed (gic/update-image-transform tf width height 0 dy width h') + [dy2 h2] (gic/crop-axis h' 0 (/ height h' -1)) + back (gic/update-image-transform crossed width h' 0 dy2 width h2)] + (t/is (close-vals? [(- dy) height] [dy2 h2]) "window back at its original place") + (t/is (close-vals? ((juxt :x :y :width :height) tf) + ((juxt :x :y :width :height) back))))))) diff --git a/common/test/common_tests/logic/comp_creation_test.cljc b/common/test/common_tests/logic/comp_creation_test.cljc index d28e5f3b31..95f158e2a3 100644 --- a/common/test/common_tests/logic/comp_creation_test.cljc +++ b/common/test/common_tests/logic/comp_creation_test.cljc @@ -19,6 +19,7 @@ [app.common.test-helpers.shapes :as ths] [app.common.types.component :as ctk] [app.common.types.components-list :as ctkl] + [app.common.types.file :as ctf] [app.common.types.shape-tree :as ctst] [app.common.uuid :as uuid] [clojure.test :as t])) @@ -250,18 +251,25 @@ component (thc/get-component file :component1) ;; ==== Action - changes (cll/generate-rename-component (pcb/empty-changes) - (:id component) - "Test component after" - (:data file)) + changes (-> (pcb/empty-changes nil (thf/current-page-id file)) + (pcb/with-file-data (ctf/file-data file)) + (cll/generate-rename-component + (:id component) + "Test component after" + (:data file)) + (pcb/update-shapes + [(thi/id :main1-root)] + #(assoc % :name "Test component after"))) file' (thf/apply-changes file changes) ;; ==== Get - component' (thc/get-component file' :component1)] + component' (thc/get-component file' :component1) + main1-root' (ths/get-shape file' :main1-root)] ;; ==== Check - (t/is (= (:name component') "Test component after")))) + (t/is (= (:name component') "Test component after")) + (t/is (= (:name main1-root') "Test component after")))) (t/deftest test-duplicate-component (let [;; ==== Setup diff --git a/common/test/common_tests/runner.cljc b/common/test/common_tests/runner.cljc index eb2e3bb1c7..88e10df432 100644 --- a/common/test/common_tests/runner.cljc +++ b/common/test/common_tests/runner.cljc @@ -16,6 +16,7 @@ [common-tests.buffer-test] [common-tests.colors-test] [common-tests.data-test] + [common-tests.exceptions-test] [common-tests.files-builder-test] [common-tests.files-changes-test] [common-tests.files-migrations-0025-test] @@ -34,6 +35,7 @@ [common-tests.geom-grid-layout-test] [common-tests.geom-grid-test] [common-tests.geom-image-bounds-resize-test] + [common-tests.geom-image-crop-test] [common-tests.geom-line-test] [common-tests.geom-modif-tree-test] [common-tests.geom-modifiers-test] @@ -118,6 +120,7 @@ 'common-tests.buffer-test 'common-tests.colors-test 'common-tests.data-test + 'common-tests.exceptions-test 'common-tests.files-builder-test 'common-tests.files-changes-test 'common-tests.files-migrations-0025-test @@ -136,6 +139,7 @@ 'common-tests.geom-grid-layout-test 'common-tests.geom-grid-test 'common-tests.geom-image-bounds-resize-test + 'common-tests.geom-image-crop-test 'common-tests.geom-line-test 'common-tests.geom-modif-tree-test 'common-tests.geom-modifiers-test diff --git a/frontend/playwright/ui/specs/text-editor-v3.spec.js b/frontend/playwright/ui/specs/text-editor-v3.spec.js index 04efe9f73c..fa49a0704a 100644 --- a/frontend/playwright/ui/specs/text-editor-v3.spec.js +++ b/frontend/playwright/ui/specs/text-editor-v3.spec.js @@ -408,10 +408,13 @@ test.describe("BUG 10934 - Double-clicking a text side handle sets auto-size", ( await workspace.goToWorkspace(); await workspace.waitForFirstRender(); - // Select the text and zoom to fit, so it is fully visible in the viewport + // Select the text and zoom to fit, so it is fully visible in the viewport. + // Wait for the zoom label to change so the handles are at their final spot. await workspace.clickLeafLayer("Fixed text"); + const zoomLabel = page.getByTitle("Zoom", { exact: true }); + const initialZoom = await zoomLabel.innerText(); await page.keyboard.press("Shift+1"); - await workspace.waitForIdle(); + await expect(zoomLabel).not.toHaveText(initialZoom); return workspace; } diff --git a/frontend/resources/images/icons/hourglass.svg b/frontend/resources/images/icons/hourglass.svg new file mode 100644 index 0000000000..f20d4e15f1 --- /dev/null +++ b/frontend/resources/images/icons/hourglass.svg @@ -0,0 +1,6 @@ + + + + + + diff --git a/frontend/src/app/main/data/common.cljs b/frontend/src/app/main/data/common.cljs index 6c948eef09..b04228694c 100644 --- a/frontend/src/app/main/data/common.cljs +++ b/frontend/src/app/main/data/common.cljs @@ -257,6 +257,8 @@ (watch [_ state _] (let [current-team-id (:current-team-id state) organization (:organization team) + team-name (or (:name team) + (dm/get-in state [:teams (:id team) :name])) current-team? (= (:id team) current-team-id)] (when (and (contains? cf/flags :admin-console) current-team?) @@ -266,14 +268,14 @@ ;; (tr "dashboard.team-belong-organization") ;; (tr "dashboard.team-no-longer-belong-organization") ;; notification is a server-provided key, dynamic by design + ;; args: team name, organization name #_{:clj-kondo/ignore [:penpot/tr-dynamic]} - (rx/of (ntf/show {:content (tr notification (:name organization)) + (rx/of (ntf/show {:content (tr notification team-name (:name organization)) :type :toast :level :info :timeout nil}))) (when (:id organization) - (check-team-sso current-team-id)))))) - ptk/UpdateEvent + (check-team-sso current-team-id)))))) ptk/UpdateEvent (update [_ state] (if (contains? cf/flags :admin-console) (let [team-id (:id team) diff --git a/frontend/src/app/main/data/modal.cljs b/frontend/src/app/main/data/modal.cljs index 458eba37ad..7db55ee4d7 100644 --- a/frontend/src/app/main/data/modal.cljs +++ b/frontend/src/app/main/data/modal.cljs @@ -17,6 +17,19 @@ ;; TODO: rename `:type` to `:name` +(defn- summarize-export-shapes + "Builds a compact body for the `:export-shapes` modal event. We know the + shape of these props here, so instead of dumping every export (each one + carrying its full shape, which can be megabytes) we emit counters: total + exports, distinct shapes and one count per export type." + [props] + (let [exports (:exports props)] + (merge + {:page (:name props) + :num-shapes (count (distinct (map :object-id exports))) + :num-exports (count exports)} + (frequencies (map :type exports))))) + (defn show ([props] (show (uuid/next) (:type props) props)) @@ -26,9 +39,17 @@ (ptk/reify ::show-modal ev/Event (-data [_] - (-> props - (dissoc :type) - (assoc :name type))) + (let [origin (:origin props) + data (if (= type :export-shapes) + (assoc (summarize-export-shapes props) :name type) + (-> props + (dissoc :type) + (assoc :name type)))] + ;; The origin is event metadata, not a prop: sending it as + ;; ::ev/origin makes `make-proto-event` put it on the event context + ;; (`:event-origin`); as a plain `:origin` it only stayed as a prop. + (cond-> (dissoc data :origin) + (some? origin) (assoc ::ev/origin origin)))) ptk/UpdateEvent (update [_ state] diff --git a/frontend/src/app/main/data/persistence.cljs b/frontend/src/app/main/data/persistence.cljs index 6dd69e5c7c..99e708d5bd 100644 --- a/frontend/src/app/main/data/persistence.cljs +++ b/frontend/src/app/main/data/persistence.cljs @@ -27,6 +27,8 @@ (declare ^:private run-persistence-task) (declare ^:private persist-commit) +(declare ^:private resume-persistence) +(declare ^:private slow-retry-cycle) (log/set-level! :warn) @@ -36,6 +38,25 @@ (def force-persist? #(= % ::force-persist)) +(def slow-retry-delay-ms + "Pause between the attempts of a queue whose backoff is spent: one attempt + per cycle, until it saves or the retry window closes." + 30000) + +(def retry-give-up-ms + "How long a queue that keeps failing on transport errors is sent again." + (* 24 60 60 1000)) + +(def slow-failure-ms + "An attempt that fails after this long may have kept the backend busy, + as a gateway timeout does, so it counts towards `max-slow-failures`." + (* 90 1000)) + +(def max-slow-failures + "Slow failures a queued commit may collect before the slow cycle stops + sending it. Fast failures, such as being offline, never count." + 3) + (def ^:private saving-stall-timeout-ms (* 5 60 1000)) (def ^:private saving-check-interval-ms 30000) (def ^:private save-wait-timeout-ms (* 2 60 1000)) @@ -142,6 +163,23 @@ save or on terminal failure." :persistence-reconnecting) +(defn- retry-window-open? + "True while a failing queue may still be sent again." + [pstate] + (let [since (:failing-since pstate)] + (or (nil? since) + (< (- (inst-ms (ct/now)) since) retry-give-up-ms)))) + +(defn- slow-retry-pending? + "True when the queue is halted by a transport failure that the slow retry + cycle keeps sending. A commit whose attempts keep failing slowly is not + sent again: each attempt may cost the backend the whole save." + [{:keys [status error slow-failures] :as pstate}] + (and (= :error status) + (transient-error? error) + (retry-window-open? pstate) + (< (or slow-failures 0) max-slow-failures))) + (defn- report-stalled-persistence [now] (ptk/reify ::report-stalled-persistence @@ -212,8 +250,9 @@ (throw (ex-info "invalid state" {}))))) (update :index dissoc commit-id) (assoc :last-progress-at (inst-ms (ct/now))) - (dissoc :stall-reported - :attempts :retry-token :retry-for))))))) + (dissoc :stall-reported :failing-since :recovering + :slow-failures :attempts :retry-token + :retry-for))))))) (defn- append-commit "Event used internally to append the current change to the @@ -243,6 +282,23 @@ (rx/of (update-status :saving) (run-persistence-task)))))))) +(defn- drop-file-commits + "Removes every queued commit belonging to `file-id`, together with its + index entry." + [pstate file-id] + (if-not file-id + pstate + (let [dropped (into #{} + (keep (fn [commit-id] + (when (= file-id (get-in pstate [:index commit-id :file-id])) + commit-id))) + (:queue pstate))] + (if (empty? dropped) + pstate + (-> pstate + (update :queue #(into (empty %) (remove dropped) %)) + (update :index #(apply dissoc % dropped))))))) + (defn- persistence-failed [commit-id cause] (ptk/reify ::persistence-failed @@ -260,28 +316,45 @@ :commit-id commit-id :hint (ex-message cause) ::errors/handled? true)) + (update :failing-since d/nilv (inst-ms (ct/now))) (dissoc :run-id :last-progress-at :stall-reported :attempts :retry-token :retry-for)))))) ptk/WatchEvent - (watch [_ _ _] - ;; The terminal toast supersedes the reconnect notice; hide it - ;; explicitly instead of relying on the single-toast replacement. - (rx/of (ptk/data-event ::error cause) - (ntf/hide :tag reconnecting-tag))) + (watch [_ state stream] + (rx/merge + ;; The terminal toast supersedes the reconnect notice; hide it + ;; explicitly instead of relying on the single-toast replacement. + (rx/of (ptk/data-event ::error cause) + (ntf/hide :tag reconnecting-tag)) + ;; A transport failure that outlasts the backoff can still pass, so + ;; the queue keeps trying at a slow pace. + (let [pstate (:persistence state)] + (if (and (not (:suspended pstate)) + (slow-retry-pending? pstate)) + (slow-retry-cycle stream) + (rx/empty))))) ptk/EffectEvent - (effect [_ _ _] - ;; Report without invoking global handlers that may reload the file or - ;; navigate away before the user can recover the retained changes. - (errors/flash-persistence cause)))) + (effect [_ state _] + ;; A failed slow attempt repeats a failure the user was already warned + ;; about. Report without invoking global handlers that may reload the + ;; file or navigate away before the user can recover the retained + ;; changes. + (let [pstate (:persistence state)] + (when-not (and (:recovering pstate) + (slow-retry-pending? pstate)) + (errors/flash-persistence cause :retrying (slow-retry-pending? pstate))))))) (defn- persistence-transient-failure "Transient save failure: the head commit stays queued and a retry is scheduled with backoff instead of parking the save in `:error`. Once the - budget (`retry-delays-ms`) is exhausted, the failure falls through to the - terminal `persistence-failed` path unchanged." - [commit-id cause] + budget (`retry-delays-ms`) is exhausted, or when the failed attempt was + one of the slow cycle, the failure falls through to the `persistence-failed` + path, which schedules the next slow attempt. While retries are suspended + nothing is scheduled: the episode waits for a file to be opened again. + `elapsed-ms` is how long the failed attempt took." + [commit-id cause elapsed-ms] (ptk/reify ::persistence-transient-failure ptk/UpdateEvent (update [_ state] @@ -296,13 +369,22 @@ :retry-token (uuid/next) :retry-for commit-id :last-progress-at (inst-ms (ct/now))) + (update :failing-since d/nilv (inst-ms (ct/now))) + (cond-> (>= elapsed-ms slow-failure-ms) + (update :slow-failures (fnil inc 0))) (dissoc :stall-reported)))))) ptk/WatchEvent (watch [_ state _] - (let [attempts (dm/get-in state [:persistence :attempts])] - (if (> attempts (count retry-delays-ms)) + (let [{:keys [attempts recovering suspended]} (:persistence state)] + (cond + (or recovering (> attempts (count retry-delays-ms))) (rx/of (persistence-failed commit-id cause)) + + suspended + (rx/empty) + + :else (rx/merge ;; One notice per episode: shown on the first attempt, ;; re-showing would only replace the identical toast. @@ -316,19 +398,6 @@ (->> (rx/timer delay-ms) (rx/map (fn [_] (persist-commit commit-id {:token token}))))))))))) -(defn- rotate-stalled-stamp - "Clears a previous attempt stamp when it is safe to resend: the commit is - the current retry head and its request is no longer in flight. A still - in-flight request is left alone — its own result drives the next step." - [state commit-id] - (let [commit (dm/get-in state [:persistence :index commit-id])] - (if (and (= :retrying (dm/get-in state [:persistence :status])) - (= commit-id (dm/get-in state [:persistence :retry-for])) - (::request-id commit) - (not (contains? @active-requests (::request-id commit)))) - (update-in state [:persistence :index commit-id] dissoc ::request-id) - state))) - (defn- commit-persisted [commit] (ptk/reify ::commit-persisted @@ -360,30 +429,29 @@ "Classifies what should happen with a queued commit before sending it. The attempt stamp and the send decision both read this, so a commit is only ever stamped with a request that is actually going to be sent." - [state commit-id request-id] + [state commit-id] (let [commit (dm/get-in state [:persistence :index commit-id])] (cond (= :error (dm/get-in state [:persistence :status])) :halted (nil? commit) :missing-commit (::acknowledged commit) :acknowledged (contains? @active-requests (::request-id commit)) :in-flight - (and (::request-id commit) - (not= request-id (::request-id commit))) :unknown-outcome (not (dm/get-in state [:permissions :can-edit])) :permission-denied :else :ready))) (defn- send-queued-commit "Sends one queued commit and maps its outcome to persistence events." [request-id session-id {:keys [id file-id file-revn file-vern changes features] :as commit}] - (let [params {:id file-id - :revn (max file-revn (get @revn-data file-id 0)) - :vern file-vern - :session-id session-id - :origin (:origin commit) - :created-at (:created-at commit) - :commit-id id - :changes (vec changes) - :features features}] + (let [sent-at (inst-ms (ct/now)) + params {:id file-id + :revn (max file-revn (get @revn-data file-id 0)) + :vern file-vern + :session-id session-id + :origin (:origin commit) + :created-at (:created-at commit) + :commit-id id + :changes (vec changes) + :features features}] ;; UI read-only mode does not invalidate already queued edits. (->> (update-file-request request-id params) ;; Fails an unanswered request as a transient network error, so the @@ -406,30 +474,29 @@ :code :invalid-save-response :file-id file-id}))))) (rx/catch (fn [cause] - (rx/of ((if (transient-error? (ex-data cause)) - persistence-transient-failure - persistence-failed) - id cause))))))) + (rx/of (if (transient-error? (ex-data cause)) + (persistence-transient-failure + id cause (- (inst-ms (ct/now)) sent-at)) + (persistence-failed id cause)))))))) (defn- persist-commit - "Sends the queued commit, stamping the attempt first. Inside a `:retrying` - episode a previous stamp is rotated when its request is no longer in - flight, so a retry resends the same commit instead of failing as - `:unknown-outcome`. Carries an optional `:token`: retry timers pass the - episode token, and a stale token (superseded episode) stays silent - instead of sending or failing." + "Sends the queued commit, stamping the attempt first. A commit whose + earlier request is no longer in flight is sent again: changes are + idempotent, so a save the backend already applied can be applied again. + Carries an optional `:token`: retry timers pass the episode token, and a + stale token (superseded episode) stays silent instead of sending or + failing." [commit-id & {:keys [token]}] (let [request-id (uuid/next)] (ptk/reify ::persist-commit ptk/UpdateEvent (update [_ state] (let [token-ok? (or (nil? token) - (= token (dm/get-in state [:persistence :retry-token]))) - state (if token-ok? (rotate-stalled-stamp state commit-id) state)] + (= token (dm/get-in state [:persistence :retry-token])))] (if (and token-ok? - (= :ready (attempt-state state commit-id request-id))) - ;; Record the attempt before starting I/O. An interrupted request - ;; may have reached the server and must not be replayed blindly. + (= :ready (attempt-state state commit-id))) + ;; Stamp the attempt before any I/O, so a request in flight is + ;; never sent twice in parallel. (assoc-in state [:persistence :index commit-id ::request-id] request-id) state))) @@ -446,16 +513,12 @@ (not= token (dm/get-in state [:persistence :retry-token]))) ;; Stale retry timer: its episode was superseded. Stay silent. (rx/empty) - (case (attempt-state state commit-id request-id) + (case (attempt-state state commit-id) :halted (rx/empty) :missing-commit (fail :missing-commit "A queued save has no change data") :acknowledged (rx/of (commit-persisted commit)) ;; The replacement runner listens for the original request's result. :in-flight (rx/empty) - ;; Fail-safe for anything outside a live retry episode (retry - ;; rotation in `update` already cleared the stamp when resending - ;; is safe). Keep the attempt stamp to prevent replay. - :unknown-outcome (fail :save-outcome-unknown "An interrupted save has an unknown outcome") :permission-denied (fail :save-permission-denied "Edit permission was lost before changes could be saved") :ready (send-queued-commit request-id (:session-id state) commit)))))))) @@ -484,7 +547,10 @@ (rx/filter #(= commit-id (:id %))) (rx/take 1) (rx/mapcat (fn [_] + ;; A save that lands shows the connection + ;; is back: the failure warning goes away. (rx/of (discard-commit commit-id) + (ntf/hide :tag errors/persistence-failed-tag) (run-persistence-task))))) (rx/of (persist-commit commit-id))) (rx/take-until stoper-s))) @@ -494,43 +560,186 @@ (ntf/hide :tag reconnecting-tag))))))) (defn- resume-persistence + "Starts the queue again. A `recovering` resume is an attempt of the slow + cycle, which sends once instead of starting a backoff episode." + ([] (resume-persistence false)) + ([recovering] + (ptk/reify ::resume-persistence + ptk/UpdateEvent + (update [_ state] + (update state :persistence + (fn [pstate] + (-> pstate + (dissoc :error :attempts :retry-token :retry-for) + (assoc :run-id (uuid/next) + :status :saving + :recovering recovering) + (update :last-progress-at d/nilv (inst-ms (ct/now))))))) + ptk/WatchEvent + (watch [_ _ _] + (rx/of (run-persistence-task)))))) + +(defn- slow-retry-cycle + "Resumes a failed queue after `slow-retry-delay-ms`, unless the queue is + resumed some other way, persistence restarts or the workspace closes." + [stream] + (let [stopper-s (rx/merge + (rx/filter (ptk/type? ::resume-persistence) stream) + (rx/filter (ptk/type? ::initialize-persistence) stream) + (rx/filter (ptk/type? ::dw/finalize-workspace) stream))] + (->> (rx/timer slow-retry-delay-ms) + (rx/map (fn [_] (resume-persistence true))) + (rx/take-until stopper-s)))) + +(defn- suspend-retries + "Stops retrying when the workspace closes. A pending backoff timer goes + stale, and no backoff or slow cycle is armed until a file is opened + again. Queued edits stay queued and a request in flight still lands." [] - (ptk/reify ::resume-persistence + (ptk/reify ::suspend-retries ptk/UpdateEvent (update [_ state] (update state :persistence - (fn [pstate] - (-> pstate - (dissoc :error :attempts :retry-token :retry-for) - (assoc :run-id (uuid/next) :status :saving) - (update :last-progress-at d/nilv (inst-ms (ct/now))))))) - ptk/WatchEvent - (watch [_ _ _] - (rx/of (run-persistence-task))))) + #(-> % + (assoc :suspended true) + (dissoc :retry-token)))))) -(defn- recover-persistence +(defn- restore-retries + "Lets failed saves be retried again once a file is opened." [] - (ptk/reify ::recover-persistence + (ptk/reify ::restore-retries + ptk/UpdateEvent + (update [_ state] + (update state :persistence dissoc :suspended)))) + +(defn- release-failed-file + "Drops the queued commits of the file whose save failed terminally. + + This runs when the user opens a file again, not at failure time. The + failure itself stays visible, because `wait-persisted-or-error` must keep + rejecting: callers use it to avoid snapshotting a file whose edits were + never saved. What must not survive is the halt. Left in place, the failed + commits sit at the head of the queue forever and, because the queue is + shared, they also block the pending edits of every other file." + [] + (ptk/reify ::release-failed-file + ptk/UpdateEvent + (update [_ state] + (let [commit-id (get-in state [:persistence :error :commit-id]) + file-id (get-in state [:persistence :index commit-id :file-id])] + (update state :persistence + #(-> % + (drop-file-commits file-id) + (dissoc :failing-since :slow-failures))))))) + +(defn- queue-needs-a-runner? + "True when the queue still holds commits that nobody is sending. + + The runner does not survive a subscription teardown, so both + initialization and recovery hit this. A halted queue is excluded: the + halt is lifted by `recover-persistence`, which has its own triggers. A + retrying episode is excluded too: its backoff timer owns the recovery, + or `recover-persistence` resumes it when retries were suspended." + [{:keys [queue status]}] + (and (seq queue) + (not= :error status) + (not= :retrying status))) + +(defn pending-file-changes + "Queued edits to restore into a freshly loaded file, in queue order. + Only edits of the loaded version belong in its data. A permanently + failed file is excluded because recovery releases its edits." + [{:keys [persistence]} {:keys [id vern]}] + (let [{:keys [queue index status error]} persistence + failed-file-id (get-in index [(:commit-id error) :file-id])] + (if (and (= :error status) + (= id failed-file-id) + (not (slow-retry-pending? persistence))) + [] + (into [] + (comp (map #(get index %)) + (filter #(and (= id (:file-id %)) + (= vern (:file-vern %)))) + (mapcat :redo-changes)) + queue)))) + +(defn- initialize-persistence-state + "Brings persistence to a coherent starting shape when the workspace + mounts. + + Deliberately leaves a halted or retrying queue alone. Lifting a halt is + `recover-persistence`'s job, and it waits for the user to open a file, so + doing it here would release edits the moment the view mounts rather than + when the user acts on them." + [] + (ptk/reify ::initialize-persistence-state ptk/WatchEvent (watch [_ state _] - (let [{:keys [queue index status error run-id]} (:persistence state) - commit (get index (peek queue))] + (let [{:keys [queue status run-id]} (:persistence state)] (cond - ;; A retrying episode owns its recovery through the backoff - ;; scheduler; resuming here would bypass the attempt budget. - (and (seq queue) - (not= status :retrying) - (or (not= status :error) - (and (= :save-permission-denied (:code error)) - (not (::request-id commit)) - (= (:file-id commit) (:current-file-id state)) - (dm/get-in state [:permissions :can-edit])))) + ;; A commit left queued by a previous session, with no runner. + (queue-needs-a-runner? (:persistence state)) (rx/of (resume-persistence)) + ;; Nothing queued and nothing in flight: there is nothing to save. (and (empty? queue) (not= status :error) (or run-id (#{:pending :saving} status))) - (rx/of (update-status :saved))))))) + (rx/of (update-status :saved)) + + :else + (rx/empty)))))) + +(defn- recover-persistence + "Lifts a halted queue when a file is opened again or the role changes. + + Leaving the workspace suspends retries (`suspend-retries`), so opening a + file (`opened-file?`) restores them, resends a retrying episode whose + backoff was cut short, and sends a failure the slow cycle still covers + (`slow-retry-pending?`) once more instead of dropping it. + + Every other terminal failure is permanent for that file: permission lost, + file gone, or changes the server refuses. None of them can succeed on a + resend, so opening a file releases them instead of replaying them. + Retaining them until here is deliberate: it is what keeps + `wait-persisted-or-error` rejecting, which is what stops + `workspace/versions.cljs` from building a snapshot out of unsaved edits. + + A lost permission is not a temporary condition. The backend raises + `:not-found` for it, and replaying the save to find that out routes the + user to the exception page." + [opened-file?] + (ptk/reify ::recover-persistence + ptk/WatchEvent + (watch [_ state _] + (let [{:keys [queue status run-id suspended]} (:persistence state)] + (rx/concat + (if (and opened-file? suspended) + (rx/of (restore-retries)) + (rx/empty)) + (cond + ;; A live retry episode owns its recovery through the backoff + ;; scheduler; resuming here would bypass the attempt budget. + ;; A suspended one has no scheduler left. + (= :retrying status) + (if (and opened-file? suspended) + (rx/of (resume-persistence)) + (rx/empty)) + + (slow-retry-pending? (:persistence state)) + (rx/of (resume-persistence true)) + + (= :error status) + (rx/of (release-failed-file) (resume-persistence)) + + (queue-needs-a-runner? (:persistence state)) + (rx/of (resume-persistence)) + + (and (empty? queue) (or run-id (#{:pending :saving} status))) + (rx/of (update-status :saved)) + + :else + (rx/empty))))))) (def ^:private xf-mapcat-undo (mapcat :undo-changes)) @@ -552,13 +761,14 @@ (defn- resume-on-online "Re-enters the runner for a retrying episode when the browser reports connectivity back, instead of waiting out the backoff. Terminal failures - stay terminal: only a live `:retrying` episode resumes." + stay terminal: only a live `:retrying` episode resumes, never one + suspended outside the workspace." [] (ptk/reify ::resume-on-online ptk/WatchEvent (watch [_ state _] - (let [{:keys [status queue]} (:persistence state)] - (if (and (seq queue) (= :retrying status)) + (let [{:keys [status queue suspended]} (:persistence state)] + (if (and (seq queue) (= :retrying status) (not suspended)) (rx/of (run-persistence-task)) (rx/empty)))))) @@ -601,12 +811,17 @@ (rx/filter #(= % ::force-persist))))] (rx/merge - (rx/of (recover-persistence)) + (rx/of (initialize-persistence-state)) + + (->> stream + (rx/filter (ptk/type? ::dw/finalize-workspace)) + (rx/map (fn [_] (suspend-retries))) + (rx/take-until stoper-s)) (->> stream (rx/filter #(or (ptk/type? ::dc/change-team-role %) (ptk/type? ::dw/workspace-initialized %))) - (rx/map (fn [_] (recover-persistence))) + (rx/map #(recover-persistence (ptk/type? ::dw/workspace-initialized %))) (rx/take-until stoper-s)) (->> (rx/interval saving-check-interval-ms) @@ -627,14 +842,16 @@ (rx/take-until stoper-s)) ;; Syncs the leave-page warning on every local commit and on every - ;; queue change (append after a flush, discard after a save). + ;; queue change (append after a flush, discard after a save, + ;; release of a failed file). (->> (rx/merge (rx/of false) (->> local-commits-s (rx/map (fn [_] (vreset! buffered? true)))) (->> stream (rx/filter #(or (ptk/type? ::append-commit %) - (ptk/type? ::discard-commit %))) + (ptk/type? ::discard-commit %) + (ptk/type? ::release-failed-file %))) (rx/map (fn [_] @buffered?)))) (rx/map sync-unload-guard) (rx/take-until stoper-s)) diff --git a/frontend/src/app/main/data/workspace.cljs b/frontend/src/app/main/data/workspace.cljs index e4b79d7b66..c8e538560f 100644 --- a/frontend/src/app/main/data/workspace.cljs +++ b/frontend/src/app/main/data/workspace.cljs @@ -9,6 +9,7 @@ [app.common.attrs :as attrs] [app.common.data :as d] [app.common.data.macros :as dm] + [app.common.files.changes :as cpc] [app.common.files.changes-builder :as pcb] [app.common.files.helpers :as cfh] [app.common.files.variant :as cfv] @@ -313,6 +314,22 @@ :file-id file-id :team-id team-id))))))) +(defn- restore-pending-changes + "Applies the edits still waiting to be saved to a freshly loaded `file`. + An edit that no longer applies to the loaded data leaves the file as the + server sent it; the save queue keeps the edits either way." + [state file] + (let [changes (dps/pending-file-changes state file)] + (if (empty? changes) + file + (try + (update file :data cpc/process-changes changes false) + (catch :default cause + (log/wrn :hint "unable to restore pending changes" + :file-id (dm/str (:id file)) + :cause cause) + file))))) + (defn- bundle-fetched [{:keys [file file-id thumbnails] :as bundle}] (ptk/reify ::bundle-fetched @@ -322,9 +339,10 @@ ptk/UpdateEvent (update [_ state] - (-> state - (assoc :thumbnails (d/update-vals thumbnails (fn [uri] {:uri uri :rendered-at nil}))) - (update :files assoc file-id file))))) + (let [file (restore-pending-changes state file)] + (-> state + (assoc :thumbnails (d/update-vals thumbnails (fn [uri] {:uri uri :rendered-at nil}))) + (update :files assoc file-id file)))))) (defn zoom-to-frame [] diff --git a/frontend/src/app/main/data/workspace/transforms.cljs b/frontend/src/app/main/data/workspace/transforms.cljs index e5bcaa1c8a..48b74d7023 100644 --- a/frontend/src/app/main/data/workspace/transforms.cljs +++ b/frontend/src/app/main/data/workspace/transforms.cljs @@ -11,6 +11,7 @@ [app.common.data.macros :as dm] [app.common.files.changes-builder :as pcb] [app.common.files.helpers :as cfh] + [app.common.geom.image-crop :as gic] [app.common.geom.matrix :as gmt] [app.common.geom.modifiers :as gm] [app.common.geom.point :as gpt] @@ -213,10 +214,29 @@ ;; Resize origin point given the selected handler selrect (dm/get-prop shape :selrect) handler-origin (get-handler-resize-origin selrect handler) + selrect-start (gpt/point (dm/get-prop selrect :x) (dm/get-prop selrect :y)) + + ;; Image crop (mod key): the shape is a window over a + ;; picture that keeps its size and position. Dragging an + ;; edge past the fixed one moves the window to the other + ;; side instead of flipping it. + crop-origin (if ^boolean center? + (gpt/point (/ width 2) (/ height 2)) + (gpt/subtract handler-origin selrect-start)) + [crop-dx crop-w] (gic/crop-axis width (dm/get-prop crop-origin :x) (dm/get-prop scalev :x)) + [crop-dy crop-h] (gic/crop-axis height (dm/get-prop crop-origin :y) (dm/get-prop scalev :y)) + + scalev (if ^boolean bounds-resize? + (gpt/point (/ crop-w width) (/ crop-h height)) + scalev) ;; If we want resize from center, displace the shape ;; so it is still centered after resize. - displacement (when ^boolean center? + displacement (cond + ^boolean bounds-resize? + (gpt/transform (gpt/point crop-dx crop-dy) shape-transform) + + ^boolean center? (-> shape-center (gpt/subtract handler-origin) (gpt/multiply scalev) @@ -225,7 +245,10 @@ (gpt/multiply (gpt/point -1 -1)) (gpt/transform shape-transform))) - resize-origin (gmt/transform-point-center handler-origin shape-center shape-transform) + resize-origin (gmt/transform-point-center + (if ^boolean bounds-resize? selrect-start handler-origin) + shape-center + shape-transform) resize-origin (if (some? displacement) (gpt/add resize-origin displacement) resize-origin) @@ -243,57 +266,23 @@ change-height? (not (mth/close? (dm/get-prop scalev :y) 1)) - ;; Calculate independent image bounds resize transform - sx (dm/get-prop scalev :x) - sy (dm/get-prop scalev :y) - w-new (* width sx) - h-new (* height sy) - - bounds-resize? (and ^boolean bounds-resize? - (pos? w-new) - (pos? h-new)) - - [dx dy] (if ^boolean center? - [(/ (* width (- 1.0 sx)) 2.0) - (/ (* height (- 1.0 sy)) 2.0)] - [(case handler - (:left :bottom-left :top-left) (* width (- 1.0 sx)) - 0.0) - (case handler - (:top :top-left :top-right) (* height (- 1.0 sy)) - 0.0)]) + ;; Keep the picture in place inside the new crop window + update-image + (fn [img-fill] + (update img-fill :transform gic/update-image-transform + width height crop-dx crop-dy crop-w crop-h)) new-fills - (when (and bounds-resize? (seq (:fills shape))) + (when (and ^boolean bounds-resize? (seq (:fills shape))) (mapv (fn [fill] - (if-let [img-fill (:fill-image fill)] - (let [tf (get img-fill :transform) - nx0 (get tf :x 0.0) - ny0 (get tf :y 0.0) - nw0 (get tf :width 1.0) - nh0 (get tf :height 1.0) - nx' (/ (- (* nx0 width) dx) w-new) - ny' (/ (- (* ny0 height) dy) h-new) - nw' (/ nw0 sx) - nh' (/ nh0 sy)] - (assoc-in fill [:fill-image :transform] - {:x nx' :y ny' :width nw' :height nh'})) - fill)) + (cond-> fill + (some? (:fill-image fill)) + (update :fill-image update-image))) (:fills shape))) new-fill-image - (when (and bounds-resize? (some? (:fill-image shape))) - (let [img-fill (:fill-image shape) - tf (get img-fill :transform) - nx0 (get tf :x 0.0) - ny0 (get tf :y 0.0) - nw0 (get tf :width 1.0) - nh0 (get tf :height 1.0) - nx' (/ (- (* nx0 width) dx) w-new) - ny' (/ (- (* ny0 height) dy) h-new) - nw' (/ nw0 sx) - nh' (/ nh0 sy)] - (assoc img-fill :transform {:x nx' :y ny' :width nw' :height nh'})))] + (when (and ^boolean bounds-resize? (some? (:fill-image shape))) + (update-image (:fill-image shape)))] (cond-> (ctm/empty) (some? displacement) diff --git a/frontend/src/app/main/errors.cljs b/frontend/src/app/main/errors.cljs index d2c4976024..54401c94af 100644 --- a/frontend/src/app/main/errors.cljs +++ b/frontend/src/app/main/errors.cljs @@ -192,7 +192,7 @@ (println "Context:") (println "--------------------") (println "Timestamp:" (ct/format-inst (ct/now) :rfc1123)) - (println "Hint: " (or (:hint data) (ex-message cause) "--")) + (println "Hint: " (or (ex/get-hint cause) "--")) (println "Prof ID: " (str (or profile-id "--"))) (println "Team ID: " (str (or team-id "--"))) (when-let [file-id (or (:file-id data) file-id)] @@ -490,6 +490,7 @@ (defn flash "Show error notification banner and emit error report. A nil timeout keeps the notification visible until dismissed or replaced. + A `:tag` lets the notification be hidden later by that tag. The payload format is derived from the cause: environment failures get a compact report. The audit event name is the canonical one requested by @@ -520,7 +521,7 @@ synchronously from inside an error handler creates a re-entrant event-processing cycle that can exhaust the JS call stack (RangeError: Maximum call stack size exceeded)." - [& {:keys [type hint cause timeout report-link?] + [& {:keys [type hint cause timeout tag report-link? links] :or {type :handled timeout 5000}}] (js/Promise. (fn [resolve _reject] @@ -533,16 +534,26 @@ link-report (when report-link? (or report (when (ex/exception? cause) - (generate-report cause {:format (report-format cause)}))))] + (generate-report cause {:format (report-format cause)})))) + ;; The download link comes first because `:report-link?` is + ;; what puts the report on the toast at all; caller links are + ;; the recovery actions and follow it. + all-links (cond-> [] + link-report + (conj {:label (tr "labels.download" "report.txt") + :callback (partial download-report! link-report)}) + (seq links) (into links))] (try (st/emit! (ntf/show (cond-> {:content (or ^boolean hint (tr "errors.generic")) :type :toast :level :error :timeout timeout} - link-report - (assoc :links [{:label (tr "labels.download" "report.txt") - :callback (partial download-report! link-report)}])))) + (some? tag) + (assoc :tag tag) + + (seq all-links) + (assoc :links all-links)))) (catch :default err (.error js/console "error on emitting toast" err))) (resolve report))))))) @@ -558,6 +569,15 @@ :type :handled :hint (tr "errors.connection-error"))) +(defn- handle-transport-error + "Reports a failure whose answer did not come from the backend. These affect + one request and pass on their own, so the user keeps the page and gets a + toast." + [error prefix] + (when-let [cause (::instance error)] + (ex/print-throwable cause :prefix prefix)) + (flash :cause (::instance error) :type :handled)) + (defmethod ptk/handle-error :network [error] ;; Transient network errors (e.g. lost connectivity, DNS failure) @@ -571,6 +591,18 @@ ;; that would report it as an unhandled application error. (handle-connectivity-error error "Offline Error")) +(defmethod ptk/handle-error :gateway-error + [error] + (handle-connectivity-error error "Gateway Error")) + +(defmethod ptk/handle-error :rate-limit + [error] + (handle-transport-error error "Rate Limit Error")) + +(defmethod ptk/handle-error :unexpected-response + [error] + (handle-transport-error error "Unexpected Response")) + (def ^:private delegated-persistence-types "Save failure causes routed to their own error handler: retaining the changes cannot resolve them." @@ -583,18 +615,72 @@ ;; The retained changes no longer apply to the restored version. (= :vern-conflict code))) +(def ^:private unhandled-persistence-types + "Save failure causes that are application defects, not expected + outcomes: the client asked for something the server refused, or the + server or our own code broke. They must reach the internal reports and + the alerts, so they are reported as unhandled. + Environment failures are deliberately absent: the user's connection is + not our bug, and they land here once the retry budget is spent. + `:assertion` is absent on purpose too, because its own handler already + reports it as handled." + #{:validation :server-error :internal}) + +(defn- persistence-report-type + "The audit event name a save failure gets. `persistence-failed` wraps the + original cause under `:type :persistence` and keeps the real type in + `:cause-type`, so `:cause-type` wins: reading `:type` alone would match the + wrapper and never the cause." + [{:keys [type cause-type]}] + (if (contains? unhandled-persistence-types (or cause-type type)) + :unhandled + :handled)) + +(defn- reload-page-link + "Recovery action for a save failure: reload the window. + + Persistence stops for the session once a save fails terminally, and + reloading is the only way to get a fresh one without the user having to + know they can also leave the file and come back. `location.reload` is the + pattern used elsewhere in the app for a full restart." + [] + {:label (tr "labels.reload-page") + :callback (fn [event] + (dom/prevent-default event) + (js/location.reload))}) + +(def persistence-failed-tag + "Tag of the save failure notification, hidden once a save lands." + :persistence-failed) + (defn flash-persistence - [cause] + "Notifies a save failure. With `:retrying` the failure is transient and + the queue keeps trying in the background, so the toast says the changes + are kept and offers no reload, which would lose them." + [cause & {:keys [retrying]}] (let [data (ex-data cause)] - (if (delegated-persistence-failure? data) + (cond + (delegated-persistence-failure? data) ;; The persistence state wraps the failure and records the original ;; type under :cause-type; dispatch on it to reach the cause's handler. (on-error (-> (exception->error-data cause) (assoc :type (or (:cause-type data) (:type data))))) + + retrying (flash :cause cause - :type :handled + :type (persistence-report-type data) :timeout nil + :tag persistence-failed-tag :report-link? true + :hint (tr "errors.save-still-retrying")) + + :else + (flash :cause cause + :type (persistence-report-type data) + :timeout nil + :tag persistence-failed-tag + :report-link? true + :links [(reload-page-link)] :hint (tr "errors.save-failed"))))) (defmethod ptk/handle-error :persistence diff --git a/frontend/src/app/main/repo.cljs b/frontend/src/app/main/repo.cljs index 938e6aaa16..c3dca3b882 100644 --- a/frontend/src/app/main/repo.cljs +++ b/frontend/src/app/main/repo.cljs @@ -32,6 +32,7 @@ #{:network ; js/fetch network-level failure :bad-gateway ; 502 :service-unavailable ; 503 + :gateway-error ; 504 and the Cloudflare 52x family :offline}) ; status 0 (browser offline) (defn retryable-error? @@ -79,6 +80,28 @@ ;; -- Response handling ------------------------------------------------------- +(def ^:private gateway-statuses + "Statuses raised by an edge proxy when the request never reached the + backend or its answer never came back." + #{504 ; gateway timeout + 520 ; cloudflare: unknown origin error + 521 ; cloudflare: origin is down + 522 ; cloudflare: connection timed out + 523 ; cloudflare: origin unreachable + 524}) ; cloudflare: origin timed out + +(def ^:private max-body-excerpt + "Number of characters kept from a body that we could not interpret." + 500) + +(defn- body-excerpt + "Keeps the head of an uninterpretable body. Error pages from proxies run + into tens of kilobytes and the excerpt is enough to recognize them." + [body] + (if (and (string? body) (> (count body) max-body-excerpt)) + (str (subs body 0 max-body-excerpt) "...") + body)) + (defn handle-response [{:keys [status body headers uri] :as response}] (cond @@ -121,6 +144,35 @@ (and (>= status 400) (map? body)) (rx/throw (ex-info "http error" (assoc body :uri uri :status status))) + ;; The branches below handle responses the backend never wrote: the + ;; body carries no error code, so the status is all we can read. + + (contains? gateway-statuses status) + (rx/throw + (ex/error :type :gateway-error + :code :gateway-error + :hint "the request did not reach the backend" + :uri uri + :status status)) + + (= 429 status) + (rx/throw + (ex/error :type :rate-limit + :code :rate-limited + :hint "too many requests" + :uri uri + :status status)) + + (>= status 400) + (rx/throw + (ex/error :type :unexpected-response + :code :unexpected-error-response + :hint "unexpected error response" + :uri uri + :status status + :headers headers + :data (body-excerpt body))) + :else (rx/throw (ex/error :type :internal @@ -129,7 +181,7 @@ :uri uri :status status :headers headers - :data body)))) + :data (body-excerpt body))))) (def default-options {:update-file {:query-params [:id]} diff --git a/frontend/src/app/main/ui/dashboard/organization_team_switch.scss b/frontend/src/app/main/ui/dashboard/organization_team_switch.scss index c2aef5e875..3fee83b7a4 100644 --- a/frontend/src/app/main/ui/dashboard/organization_team_switch.scss +++ b/frontend/src/app/main/ui/dashboard/organization_team_switch.scss @@ -208,7 +208,12 @@ gap: var(--sp-s); top: calc(var(--sp-s) + #{$sz-64} + var(--sp-xs)); left: var(--sp-s); - height: px2rem(416); + + // Floor matches the previous fixed height; grow for ~6 org rows + left + // footer; scroll + fade kick in above the cap. + block-size: fit-content; + min-block-size: px2rem(416); + max-block-size: $sz-520; overflow: hidden; } diff --git a/frontend/src/app/main/ui/ds/foundations/assets/icon.cljs b/frontend/src/app/main/ui/ds/foundations/assets/icon.cljs index da902d690e..991532c963 100644 --- a/frontend/src/app/main/ui/ds/foundations/assets/icon.cljs +++ b/frontend/src/app/main/ui/ds/foundations/assets/icon.cljs @@ -168,6 +168,7 @@ (def ^:icon-id help "help") (def ^:icon-id hide "hide") (def ^:icon-id history "history") +(def ^:icon-id hourglass "hourglass") (def ^:icon-id hsva "hsva") (def ^:icon-id hug-content "hug-content") (def ^:icon-id icon "icon") diff --git a/frontend/src/app/main/ui/icons.cljs b/frontend/src/app/main/ui/icons.cljs index 9e62720d02..7dec276e5d 100644 --- a/frontend/src/app/main/ui/icons.cljs +++ b/frontend/src/app/main/ui/icons.cljs @@ -150,6 +150,7 @@ (def ^:icon help (icon-xref :help)) (def ^:icon hide (icon-xref :hide)) (def ^:icon history (icon-xref :history)) +(def ^:icon hourglass (icon-xref :hourglass)) (def ^:icon hsva (icon-xref :hsva)) (def ^:icon hug-content (icon-xref :hug-content)) (def ^:icon icon (icon-xref :icon)) diff --git a/frontend/src/app/main/ui/settings/subscription.cljs b/frontend/src/app/main/ui/settings/subscription.cljs index f502f5285f..d1db4cab5f 100644 --- a/frontend/src/app/main/ui/settings/subscription.cljs +++ b/frontend/src/app/main/ui/settings/subscription.cljs @@ -469,7 +469,14 @@ (= subscription-type "enterprise") (tr "subscription.settings.enterprise")) subscription-is-trial? - (= (:status subscription) "trialing") + (if nitrate? + (= (:status nitrate-license) "trialing") + (= (:status subscription) "trialing")) + + trial-end-date + (if nitrate? + (ct/format-inst (:trial-end nitrate-license) "d MMMM, yyyy") + (ct/format-inst (:trial-end subscription) "d MMMM, yyyy")) member-since (ct/format-inst (:created-at profile) "d MMMM, yyyy") @@ -701,12 +708,19 @@ [:div {:class (stl/css :membership-container)} (when (or nitrate? (and subscribed-since (not= subscription-type "professional"))) - [:div {:class (stl/css :membership)} - [:> icon* {:class (stl/css :subscription-member) - :icon-id "crown" - :size "m"}] - [:span {:class (stl/css :membership-date)} - (tr "subscription.settings.subscribed-since" subscription-name subscribed-since)]]) + (if subscription-is-trial? + [:div {:class (stl/css :membership)} + [:> icon* {:class (stl/css :subscription-member) + :icon-id "hourglass" + :size "m"}] + [:span {:class (stl/css :membership-date)} + (tr "subscription.settings.trial-ends-on" trial-end-date)]] + [:div {:class (stl/css :membership)} + [:> icon* {:class (stl/css :subscription-member) + :icon-id "crown" + :size "m"}] + [:span {:class (stl/css :membership-date)} + (tr "subscription.settings.subscribed-since" subscription-name subscribed-since)]])) [:div {:class (stl/css :membership)} [:> icon* {:class (stl/css :penpot-member) diff --git a/frontend/src/app/main/ui/workspace/sidebar/options/menus/layout_item.cljs b/frontend/src/app/main/ui/workspace/sidebar/options/menus/layout_item.cljs index 10b9c42db7..659aea2fb3 100644 --- a/frontend/src/app/main/ui/workspace/sidebar/options/menus/layout_item.cljs +++ b/frontend/src/app/main/ui/workspace/sidebar/options/menus/layout_item.cljs @@ -602,7 +602,6 @@ :input-type :min-width :nillable true :applied-token applied-token-to-min-w - :tooltip-class (stl/css :tooltip-wrapper) :value min-w}] [:div {:class (stl/css :layout-item-min-w) @@ -632,7 +631,6 @@ :attr :layout-item-max-w :property (tr "workspace.options.layout-item.layout-item-max-w") :nillable true - :tooltip-class (stl/css :tooltip-wrapper) :applied-token applied-token-to-max-w :value max-w}] @@ -663,7 +661,6 @@ :attr :layout-item-min-h :property (tr "workspace.options.layout-item.layout-item-min-h") :nillable true - :tooltip-class (stl/css :tooltip-wrapper) :applied-token applied-token-to-min-h :value min-h}] @@ -694,7 +691,6 @@ :attr :layout-item-max-h :property (tr "workspace.options.layout-item.layout-item-max-h") :nillable true - :tooltip-class (stl/css :tooltip-wrapper) :applied-token applied-token-to-max-h :value max-h}] diff --git a/frontend/src/app/main/ui/workspace/sidebar/options/menus/layout_item.scss b/frontend/src/app/main/ui/workspace/sidebar/options/menus/layout_item.scss index b362df51d0..52b6a165e3 100644 --- a/frontend/src/app/main/ui/workspace/sidebar/options/menus/layout_item.scss +++ b/frontend/src/app/main/ui/workspace/sidebar/options/menus/layout_item.scss @@ -192,7 +192,3 @@ .inputs-wrapper { grid-column: 1 / span 2; } - -.tooltip-wrapper { - inline-size: 100%; -} diff --git a/frontend/src/app/main/ui/workspace/sidebar/options/menus/measures.cljs b/frontend/src/app/main/ui/workspace/sidebar/options/menus/measures.cljs index b398f099fc..3f56fbbc23 100644 --- a/frontend/src/app/main/ui/workspace/sidebar/options/menus/measures.cljs +++ b/frontend/src/app/main/ui/workspace/sidebar/options/menus/measures.cljs @@ -154,6 +154,8 @@ (get n-values :rotation)) (identical? (get o-values :hide-in-viewer) (get n-values :hide-in-viewer)) + (identical? (get o-values :show-content) + (get n-values :show-content)) (identical? (get o-values :width) (get n-values :width)) (identical? (get o-values :height) diff --git a/frontend/src/app/main/ui/workspace/sidebar/options/menus/typography.cljs b/frontend/src/app/main/ui/workspace/sidebar/options/menus/typography.cljs index f4974abc7b..2a6b472977 100644 --- a/frontend/src/app/main/ui/workspace/sidebar/options/menus/typography.cljs +++ b/frontend/src/app/main/ui/workspace/sidebar/options/menus/typography.cljs @@ -291,21 +291,14 @@ (mf/defc font-item* {::mf/wrap [mf/memo]} - [{:keys [font is-current on-click style]}] - (let [item-ref (mf/use-ref) - on-click (mf/use-fn (mf/deps font) #(on-click font))] - - (mf/with-effect [is-current] - (when is-current - (let [element (mf/ref-val item-ref)] - (when-not (dom/is-in-viewport? element) - (dom/scroll-into-view! element))))) - + [{:keys [font is-current is-focused on-click style]}] + (let [on-click (mf/use-fn (mf/deps font) #(on-click font))] [:div {:class (stl/css :font-wrapper) :style style - :ref item-ref :on-click on-click} - [:div {:class (stl/css-case :font-item true :selected is-current)} + [:div {:class (stl/css-case :font-item true + :selected is-current + :focused is-focused)} [:> font-item-preview* {:font font}] (when is-current [:> icon* {:icon-id i/tick @@ -326,11 +319,20 @@ (mf/defc font-selector* [{:keys [on-select on-close current-font show-recent full-size]}] - (let [selected (mf/use-state current-font) + (let [;; The tick marks the font the text had when the selector opened. + ;; The focus (green border) marks the font Enter applies: arrows move + ;; it and preview the font, Esc and an outside click undo the preview + initial-font (mf/with-memo [] current-font) + focused* (mf/use-state current-font) + focused (deref focused*) + previewed? (mf/use-ref false) + preview? (mf/use-ref false) + state* (mf/use-state #(do {:term "" :backends #{}})) state (deref state*) + root (mf/use-ref) flist (mf/use-ref) input (mf/use-ref) @@ -353,52 +355,57 @@ (filter #(contains? installed-ids (:id %))) (filter-fonts state))) - ;; When the active font is not in the filtered results, pre-select - ;; the first match visually so the user can confirm it with Enter — - ;; without live-applying it on every keystroke. - effective-selected - (if (and (seq fonts) (not (d/seek #(= (:id %) (:id @selected)) fonts))) - (first fonts) - @selected) - full-size? (boolean (and full-size show-recent)) - select-next - (mf/use-fn - (mf/deps fonts) - (fn [event] - (dom/stop-propagation event) - (dom/prevent-default event) - (swap! selected get-next-font fonts))) + ;; `recent-fonts` is a vector, so check it with `seq`: an empty one + ;; is truthy and would show the heading over nothing + show-recent? (boolean (and show-recent (seq recent-fonts))) - select-prev + ;; Arrows only move the focus, from its latest value: a held key + ;; fires again before the listener sees the new state. The preview + ;; follows in an effect + focus-and-preview (mf/use-fn (mf/deps fonts) - (fn [event] + (fn [event get-font] (dom/stop-propagation event) (dom/prevent-default event) - (swap! selected get-prev-font fonts))) + (when (seq fonts) + (mf/set-ref-val! preview? true) + (swap! focused* get-font fonts)))) + + on-cancel + (mf/use-fn + (mf/deps initial-font on-select on-close) + (fn [] + (when (mf/ref-val previewed?) + (on-select initial-font)) + (on-close))) on-key-down (mf/use-fn - (mf/deps fonts on-select on-close) + (mf/deps focused focus-and-preview on-cancel on-select on-close) (fn [event] (cond - (kbd/up-arrow? event) (select-prev event) - (kbd/down-arrow? event) (select-next event) - (kbd/esc? event) (on-close) - (kbd/enter? event) - (let [first-result (when-not (d/seek #(= (:id %) (:id @selected)) fonts) - (first fonts))] - (if first-result - (do (on-select first-result) (on-close)) - (on-close))) + (kbd/up-arrow? event) (focus-and-preview event get-prev-font) + (kbd/down-arrow? event) (focus-and-preview event get-next-font) + (kbd/esc? event) (on-cancel) + (kbd/enter? event) (do (when (some? focused) + (on-select focused)) + (on-close)) :else (dom/focus! (mf/ref-val input))))) on-filter-change (mf/use-fn - (fn [event] - (swap! state* assoc :term event))) + (mf/deps state all-fonts) + (fn [term] + ;; Move the focus to the first result, without previewing it. + ;; `swap!` on a `use-state` returns nil, so build the new state here + (let [state (assoc state :term term)] + (reset! state* state) + (mf/set-ref-val! preview? false) + (when (seq term) + (reset! focused* (first (filter-fonts state all-fonts))))))) on-select-and-close (mf/use-fn @@ -407,10 +414,29 @@ (on-select font) (on-close)))] - (mf/with-effect [fonts on-key-down] + (mf/with-effect [on-key-down] (let [key (events/listen js/document "keydown" on-key-down)] #(events/unlistenByKey key))) + ;; Close on an outside click. Starts on the next tick to skip the + ;; opening click; detached targets (a just-selected row) are ignored + (mf/with-effect [on-cancel] + (let [listening? (volatile! false) + on-click (fn [event] + (let [target (dom/get-target event) + node (mf/ref-val root)] + (when (and @listening? + (some? node) + (.-isConnected ^js target) + (not (dom/child? target node))) + (on-cancel)))) + task (tm/schedule #(vreset! listening? true)) + keys [(events/listen js/document "click" on-click) + (events/listen js/document "contextmenu" on-click)]] + (fn [] + (tm/dispose! task) + (run! events/unlistenByKey keys)))) + ;; Materialize the preview sprite into the DOM only while the picker is open ;; (markup is prefetched on workspace load), removing it on close so its ;; ~2000 nodes aren't kept around idle. The attachment is deferred so the @@ -427,16 +453,26 @@ (when-some [n @node*] (fonts/detach-preview-sprite! n)))))) - (mf/with-effect [@selected] + ;; Preview the focused font once the arrows stop for a moment. Applying + ;; every font of a held key floods the store and the text editor with + ;; updates, until React stops with "Maximum update depth exceeded" + (mf/with-effect [focused] + (when (mf/ref-val preview?) + (mf/set-ref-val! preview? false) + (let [task (tm/schedule 100 (fn [] + (mf/set-ref-val! previewed? true) + (on-select focused)))] + #(tm/dispose! task)))) + + ;; The only scroll to the focused row: the list is virtualized, so the row + ;; may not be rendered yet + (mf/with-effect [focused fonts] (let [node (mf/ref-val flist) - index (:index @selected)] + index (d/index-of-pred fonts #(= (:id %) (:id focused)))] ;; This is nil safe operation, do nothing if node or index are ;; invalid. (dom/scroll-to-row node index))) - (mf/with-effect [@selected] - (on-select @selected)) - (mf/with-effect [] (st/emit! (dsc/push-shortcuts :typography {} :workspace)) (fn [] @@ -451,7 +487,8 @@ (dom/scroll-to-position node offset))))) [:div {:class [(stl/css-case :font-selector true - :fonts-on-modal (not full-size?))]} + :fonts-on-modal (not full-size?))] + :ref root} [:div {:class (stl/css-case :font-selector-dropdown true :font-selector-dropdown-full-size full-size?)} [:div {:class (stl/css :header)} @@ -459,7 +496,7 @@ :value (:term state) :auto-focus true :placeholder (tr "workspace.options.search-font")}] - (when (and recent-fonts show-recent) + (when show-recent? [:section {:class (stl/css :show-recent)} [:p {:class (stl/css :header-title)} (tr "workspace.options.recent-fonts")] (for [[idx font] (d/enumerate recent-fonts)] @@ -467,15 +504,21 @@ :font font :style {} :on-click on-select-and-close - :is-current (= (:id font) (:id effective-selected))}])])] + ;; Recent fonts are reached with the mouse or by + ;; filtering, never with the arrow keys, so no focus + :is-current (= (:id font) (:id initial-font))}])])] [:div {:class (stl/css-case :fonts-list true - :fonts-list-full-size full-size?)} + :fonts-list-full-size full-size? + :fonts-list-below-recent show-recent?)} + (when (empty? fonts) + [:p {:class (stl/css :fonts-list-empty)} + (tr "workspace.options.no-font-results")]) [:> rvt/AutoSizer {} (fn [props] (let [width (unchecked-get props "width") height (unchecked-get props "height") - render #(row-renderer fonts effective-selected on-select-and-close %)] + render #(row-renderer fonts (:id initial-font) (:id focused) on-select-and-close %)] (mf/html [:> rvt/List #js {:height height :ref flist @@ -485,7 +528,7 @@ :rowRenderer render}])))]]]])) (defn row-renderer - [fonts selected on-select props] + [fonts current-id focused-id on-select props] (let [index (unchecked-get props "index") key (unchecked-get props "key") style (unchecked-get props "style") @@ -495,7 +538,8 @@ :font font :style style :on-click on-select - :is-current (= (:id font) (:id selected))}]))) + :is-current (= (:id font) current-id) + :is-focused (= (:id font) focused-id)}]))) (mf/defc font-options* [{:keys [values on-change on-blur show-recent full-size-selector]}] diff --git a/frontend/src/app/main/ui/workspace/sidebar/options/menus/typography.scss b/frontend/src/app/main/ui/workspace/sidebar/options/menus/typography.scss index 606303bf47..40e0374813 100644 --- a/frontend/src/app/main/ui/workspace/sidebar/options/menus/typography.scss +++ b/frontend/src/app/main/ui/workspace/sidebar/options/menus/typography.scss @@ -13,6 +13,8 @@ // Must match BOX_HEIGHT in scripts/build-fonts-preview.js (the sprite bakes each // font name into that box at 1 unit == 1px). +// NOTE: kept in px on purpose, not px2rem(): the sprite is baked in pixels, so +// the box must not scale with the root font size. $font-preview-box-height: 28px; .typography-entry { @@ -116,6 +118,10 @@ $font-preview-box-height: 28px; } .typography-sample-input { + // No DS typography on purpose: the "Ag" sample is drawn in the + // typography's own font + font-size: px2rem(16); + line-height: 1; display: flex; justify-content: center; align-items: center; @@ -130,6 +136,9 @@ $font-preview-box-height: 28px; } .adv-typography-name { + // TODO: No exact DS typography match found. Rename input of a + // typography; only the font size is set. + // Closest match: body-small (12px). font-size: px2rem(12); color: var(--font-name-wrapper-foreground-color); flex-grow: 1; @@ -161,26 +170,31 @@ $font-preview-box-height: 28px; } .typography-name-wrapper { - @extend %asset-element; + @include t.use-typography("body-small"); display: grid; grid-template-columns: minmax($sz-24, auto) auto 1fr $sz-28; column-gap: var(--sp-xs); flex: 1; + align-items: center; block-size: $sz-32; inline-size: 100%; padding: 0 0 0 var(--sp-m); + border-radius: $br-8; background-color: var(--color-background-quaternary); + color: var(--color-foreground-primary); margin-block-end: var(--sp-xs); } .typography-sample { + // No DS typography on purpose: the "Ag" sample is drawn in the + // typography's own font + font-size: px2rem(16); + line-height: 1; display: flex; justify-content: center; align-items: center; min-inline-size: $sz-24; - font-size: px2rem(16); - line-height: 1; block-size: $sz-32; white-space: nowrap; padding: 0; @@ -194,8 +208,8 @@ $font-preview-box-height: 28px; margin: 0; min-inline-size: px2rem(6); - padding: 8px 0 8px 6px; - max-width: 100%; + padding: var(--sp-s) 0 var(--sp-s) px2rem(6); + max-inline-size: 100%; color: var(--color-foreground-primary); } @@ -218,7 +232,7 @@ $font-preview-box-height: 28px; display: grid; grid-template-columns: 50% 50%; block-size: $sz-32; - padding-left: var(--sp-xs); + padding-inline-start: var(--sp-xs); } .info-label, @@ -243,11 +257,22 @@ $font-preview-box-height: 28px; } .font-option { - @include t.use-typography("body-small"); - @extend %asset-element; + --font-option-background-color: var(--color-background-tertiary); + @include t.use-typography("body-small"); + + display: flex; + align-items: center; + block-size: $sz-32; padding: var(--sp-s); + border-radius: $br-8; + background-color: var(--font-option-background-color); + color: var(--color-foreground-primary); cursor: pointer; + + &:hover { + --font-option-background-color: var(--color-background-quaternary); + } } .font-option-name { @@ -263,15 +288,26 @@ $font-preview-box-height: 28px; } .font-size-options { - @extend %asset-element; + --font-size-options-background-color: var(--color-background-tertiary); + @include t.use-typography("body-small"); + display: flex; flex-grow: 1; + align-items: center; + block-size: $sz-32; inline-size: px2rem(60); margin: 0; padding: 0; border: $b-1 solid var(--color-background-tertiary); + border-radius: $br-8; + background-color: var(--font-size-options-background-color); + color: var(--color-foreground-primary); position: relative; + + &:hover { + --font-size-options-background-color: var(--color-background-quaternary); + } } .font-variant-options { @@ -291,6 +327,9 @@ $font-preview-box-height: 28px; gap: var(--sp-xs); } +// TODO: Kept on the deprecated %input-element and %button-icon-small +// placeholders, which style the icon svg and the input elements. They go +// away when these controls move to the DS numeric-input*. .line-height, .letter-spacing { @extend %input-element; @@ -309,11 +348,22 @@ $font-preview-box-height: 28px; } .text-transform { - @extend %asset-element; + --text-transform-background-color: var(--color-background-tertiary); + @include t.use-typography("body-small"); + + display: flex; + align-items: center; + block-size: $sz-32; inline-size: fit-content; padding: 0; - background-color: var(--color-background-tertiary); + border-radius: $br-8; + background-color: var(--text-transform-background-color); + color: var(--color-foreground-primary); + + &:hover { + --text-transform-background-color: var(--color-background-quaternary); + } } .font-size-select { @@ -328,10 +378,24 @@ $font-preview-box-height: 28px; outline: none; .numeric-input { - @extend %input-base; @include t.use-typography("body-small"); + @include text-ellipsis; + block-size: $sz-28; + inline-size: 100%; + flex-grow: 1; + margin: var(--sp-xxs) 0; padding: 0; + border: none; + border-radius: $br-8; + background: none; + outline: none; + color: var(--color-foreground-primary); + + &[disabled] { + opacity: 0.5; + pointer-events: none; + } } } @@ -340,16 +404,15 @@ $font-preview-box-height: 28px; justify-content: center; align-items: center; position: absolute; - top: calc(-1 * #{px2rem(18)}); - left: 0; - right: 0; + inset-block-start: calc(-1 * #{px2rem(18)}); + inset-inline: 0; block-size: 100%; inline-size: 100%; z-index: var(--z-index-dropdown); } .fonts-on-modal { - top: 0; + inset-block-start: 0; } .show-recent { @@ -385,22 +448,39 @@ $font-preview-box-height: 28px; } .font-wrapper { - padding-bottom: var(--sp-xs); + padding-block-end: var(--sp-xs); cursor: pointer; } .font-item { - @extend %asset-element; + --font-item-background-color: var(--color-background-tertiary); + + @include t.use-typography("body-small"); - width: px2rem(280); - margin-bottom: var(--sp-xs); - border-radius: $br-8; display: flex; + block-size: $sz-32; + inline-size: 100%; + margin-block-end: var(--sp-xs); + padding: var(--sp-s) var(--sp-m); + border-radius: $br-8; + background-color: var(--font-item-background-color); + color: var(--color-foreground-primary); + + &:hover { + --font-item-background-color: var(--color-background-quaternary); + } &.selected { color: var(--color-foreground-primary); } + // The font Enter applies, same border as the focused option of the + // select dropdowns + &.focused { + outline: $b-1 solid var(--color-accent-primary); + outline-offset: calc(-1 * #{$b-1}); + } + // Center & clip so a previewed font's own metrics never grow/overflow the row. align-items: center; overflow: hidden; @@ -456,9 +536,21 @@ $font-preview-box-height: 28px; } .fonts-list-full-size { + border: $b-1 solid var(--color-background-quaternary); +} + +// Joins the list to the "Recent" section above it +.fonts-list-below-recent { border-start-start-radius: 0; border-start-end-radius: 0; - border: $b-1 solid var(--color-background-quaternary); +} + +.fonts-list-empty { + @include t.use-typography("body-small"); + + color: var(--color-foreground-secondary); + margin: 0; + padding: var(--sp-m); } .dropdown-icon { diff --git a/frontend/test/frontend_tests/data/dashboard_test.cljs b/frontend/test/frontend_tests/data/dashboard_test.cljs index 9d5fbd1ce3..c97e3742f9 100644 --- a/frontend/test/frontend_tests/data/dashboard_test.cljs +++ b/frontend/test/frontend_tests/data/dashboard_test.cljs @@ -10,14 +10,57 @@ [app.config :as cf] [app.main.data.common :as dcm] [app.main.data.dashboard :as dd] + [app.main.data.notifications :as ntf] [app.main.data.websocket :as dws] [app.main.repo :as rp] [app.main.router :as rt] + [app.util.i18n :as i18n] [beicon.v2.core :as rx] [cljs.test :as t :include-macros true] [frontend-tests.helpers.async :as async] [frontend-tests.helpers.mock :as mock] [potok.v2.core :as ptk])) +(defn- tr-args-for-team-org-toast + "Runs handle-change-team-organization and returns recorded i18n/tr calls." + [notification organization] + (let [team-id (uuid/next) + current-url (str "https://penpot.example.com/#/dashboard/recent?team-id=" team-id) + state {:current-team-id team-id + :teams {team-id {:id team-id :name "Team-Apple"}}} + event (dcm/handle-change-team-organization + {:team {:id team-id :organization organization} + :notification notification}) + tr-calls (atom [])] + (-> (mock/with-mocks* + {cf/flags (conj cf/flags :admin-console) + ;; plain variadic fn: mock/stub does not cover IFn variadic dispatch + i18n/tr (fn [key & args] + (swap! tr-calls conj [key (vec args)]) + (str key ":" (pr-str args))) + rp/cmd! (mock/stub + (fn [cmd params] + (if (= [:check-nitrate-sso + {:team-id team-id :url current-url}] + [cmd params]) + (rx/of {:authorized true}) + (rx/throw (ex-info "unexpected RPC" {:cmd cmd :params params}))))) + rt/get-current-href (mock/stub (constantly current-url))} + (await (async/observe (ptk/watch event state nil) + :on-next (fn [ev] + (t/is (= ::ntf/show (ptk/type ev))))))) + (.then (fn [_] @tr-calls))))) (t/deftest ^:async team-org-change-toast-names-team-when-joining-org + (let [tr-calls (await (tr-args-for-team-org-toast + "dashboard.team-belong-organization" + {:id (uuid/next) :name "OrgB"}))] + (t/is (= [["dashboard.team-belong-organization" ["Team-Apple" "OrgB"]]] + tr-calls)))) + +(t/deftest ^:async team-org-change-toast-names-team-when-leaving-org + (let [tr-calls (await (tr-args-for-team-org-toast + "dashboard.team-no-longer-belong-organization" + {:name "OrgA"}))] + (t/is (= [["dashboard.team-no-longer-belong-organization" ["Team-Apple" "OrgA"]]] + tr-calls)))) (t/deftest ^:async duplicated-project-has-a-file-count-and-is-unpinned (let [id (uuid/next) @@ -69,7 +112,6 @@ (fn [] (done'))))) done)))) - (t/deftest organization-sso-activation-redirects-current-team (t/async done (let [team-id (uuid/next) diff --git a/frontend/test/frontend_tests/data/fix_deleted_fonts_test.cljs b/frontend/test/frontend_tests/data/fix_deleted_fonts_test.cljs index 4592c90ab3..b2b5c2e083 100644 --- a/frontend/test/frontend_tests/data/fix_deleted_fonts_test.cljs +++ b/frontend/test/frontend_tests/data/fix_deleted_fonts_test.cljs @@ -2,7 +2,7 @@ ;; License, v. 2.0. If a copy of the MPL was not distributed with this ;; file, You can obtain one at http://mozilla.org/MPL/2.0/. ;; -;; Copyright (c) KALEIDOS SUBSIDIARY SL +;; Copyright (c) KALEIDOS INC Sucursal en España SL (ns frontend-tests.data.fix-deleted-fonts-test (:require diff --git a/frontend/test/frontend_tests/data/modal_test.cljs b/frontend/test/frontend_tests/data/modal_test.cljs new file mode 100644 index 0000000000..e3b0fd4325 --- /dev/null +++ b/frontend/test/frontend_tests/data/modal_test.cljs @@ -0,0 +1,55 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns frontend-tests.data.modal-test + (:require + [app.common.uuid :as uuid] + [app.main.data.event :as ev] + [app.main.data.modal :as modal] + [cljs.test :as t :include-macros true])) + +(t/deftest export-shapes-modal-event-carries-a-summary + (let [shape-a (uuid/next) + shape-b (uuid/next) + exports [{:object-id shape-a :type :png} + {:object-id shape-a :type :svg} + {:object-id shape-b :type :png}] + event (modal/show :export-shapes + {:exports exports + :origin "workspace:sidebar" + :name "Page 1"}) + data (ev/-data event)] + + (t/testing "the raw exports never leak into the event" + (t/is (not (contains? data :exports)))) + + (t/testing "the event carries counters instead" + (t/is (= 3 (:num-exports data))) + (t/is (= 2 (:num-shapes data))) + (t/is (= 2 (:png data))) + (t/is (= 1 (:svg data)))) + + (t/testing "the modal context is preserved" + (t/is (= "Page 1" (:page data))) + (t/is (= :export-shapes (:name data)))) + + (t/testing "origin is emitted as event metadata, not as a prop" + (t/is (= "workspace:sidebar" (::ev/origin data))) + (t/is (not (contains? data :origin)))))) + +(t/deftest other-modals-keep-their-props + (let [event (modal/show :some-modal {:foo "bar" :items [1 2 3]}) + data (ev/-data event)] + (t/is (= "bar" (:foo data))) + (t/is (= [1 2 3] (:items data))) + (t/is (= :some-modal (:name data))))) + +(t/deftest modal-origin-becomes-event-metadata + (let [event (modal/show :some-modal {:foo "bar" :origin "dashboard"}) + data (ev/-data event)] + (t/is (= "dashboard" (::ev/origin data))) + (t/is (not (contains? data :origin))) + (t/is (= "bar" (:foo data))))) diff --git a/frontend/test/frontend_tests/data/persistence_retry_test.cljs b/frontend/test/frontend_tests/data/persistence_retry_test.cljs index c2ddab917e..6bbfbedf47 100644 --- a/frontend/test/frontend_tests/data/persistence_retry_test.cljs +++ b/frontend/test/frontend_tests/data/persistence_retry_test.cljs @@ -2,7 +2,7 @@ ;; License, v. 2.0. If a copy of the MPL was not distributed with this ;; file, You can obtain one at http://mozilla.org/MPL/2.0/. ;; -;; Copyright (c) KALEIDOS SUBSIDIARY SL +;; Copyright (c) KALEIDOS INC Sucursal en España SL (ns frontend-tests.data.persistence-retry-test "Retry-episode tests for the file persistence save pipeline @@ -83,18 +83,26 @@ (rx/dispose! store) (rx/end! response)))))) +(defn- instant-backoff + "Timer double: the backoff delays fire at once and the slow cycle never + does, so a failure that outlasts the backoff settles in `:error`." + [ms] + (if (= ms dps/slow-retry-delay-ms) + (rx/subject) + (rx/of :tick))) + (defn- check-failed-save-response "Feeds `result` as the transport response and asserts the commit stays queued as a failed save instead of being treated as persisted. Retry - timers fire instantly (stubbed `rx/timer`), so a permanently bad answer - exhausts the 3-retry budget and lands terminal: `:error` carrying + timers fire instantly (stubbed `rx/timer`, slow cycle held), so a + permanently bad answer exhausts the 3-retry budget and lands terminal: `:error` carrying `:invalid-save-response`, queue intact, exactly 4 sends." [result] (with-persistence (^:async fn [{:keys [file-id requests store]}] (await (mock/with-mocks* - {rx/timer (mock/stub (fn [_] (rx/of :tick)))} + {rx/timer (mock/stub instant-backoff)} (ptk/emit! store (local-commit file-id) ::dps/force-persist) (await (async/wait-for #(and (= :error (get-in @store [:persistence :status])) (= :invalid-save-response @@ -141,7 +149,7 @@ (^:async fn [{:keys [file-id requests store]}] (await (mock/with-mocks* - {rx/timer (mock/stub (fn [_] (rx/of :tick)))} + {rx/timer (mock/stub instant-backoff)} (ptk/emit! store (local-commit file-id) ::dps/force-persist) (await (async/wait-for #(and (= :saved (get-in @store [:persistence :status])) (empty? (get-in @store [:persistence :queue]))) @@ -158,6 +166,39 @@ (rx/throw (ex-info "offline" {:type :offline})) (rx/of {:revn 1}))))))) +(defn- check-gateway-response-retries + "Answers the first send with a proxy error page for `status`, built by the + real `rp/handle-response`, and the retry with a valid revision. The save + must take the retry path and land, never halt in `:error`." + [status] + (let [calls (atom 0)] + (with-persistence + (^:async fn [{:keys [file-id failures requests store]}] + (await + (mock/with-mocks* + {rx/timer (mock/stub instant-backoff)} + (ptk/emit! store (local-commit file-id) ::dps/force-persist) + (await (async/wait-for #(and (= :saved (get-in @store [:persistence :status])) + (empty? (get-in @store [:persistence :queue]))) + (str "a " status " is retried and the save lands"))) + (t/is (= 2 (count @requests)) "the proxy error plus one retry") + (t/is (empty? @failures) "a retried proxy error never reaches the failure toast")))) + (fn [_ _] + (if (= 1 (swap! calls inc)) + (rp/handle-response {:status status + :headers {"content-type" "text/html"} + :body "Gateway Timeout" + :uri "https://penpot.example.com/api/main/methods/update-file"}) + (rx/of {:revn 1})))))) + +;; Variant: the proxy in front of the backend timed out. +(t/deftest ^:async gateway-timeout-save-is-retried + (await (check-gateway-response-retries 504))) + +;; Variant: Cloudflare gave up waiting for the origin. +(t/deftest ^:async cloudflare-origin-timeout-save-is-retried + (await (check-gateway-response-retries 524))) + ;; Scenario: every send fails transiently. The 2s / 8s / 20s retries fire ;; and then the failure falls through to the exact terminal path: `:error` ;; carrying the cause, queue intact, one flash. Proves: the budget bounds @@ -169,7 +210,7 @@ (^:async fn [{:keys [file-id failures requests store]}] (await (mock/with-mocks* - {rx/timer (mock/stub (fn [ms] (swap! delays conj ms) (rx/of :tick)))} + {rx/timer (mock/stub (fn [ms] (swap! delays conj ms) (instant-backoff ms)))} (ptk/emit! store (local-commit file-id) ::dps/force-persist) (await (async/wait-for #(and (= :error (get-in @store [:persistence :status])) (= 4 (count @requests))) @@ -177,7 +218,8 @@ (t/is (= :error (get-in @store [:persistence :status]))) (t/is (= :save-failed (get-in @store [:persistence :error :code]))) (t/is (= 1 (count (get-in @store [:persistence :queue])))) - (t/is (= [2000 8000 20000] @delays) "the backoff schedule fires in order") + (t/is (= [2000 8000 20000 dps/slow-retry-delay-ms] @delays) + "the backoff schedule fires in order, then the slow cycle is armed") (t/is (= 1 (count @failures)) "exhaustion flashes exactly once")))) (fn [_ _] (rx/throw (ex-info "offline" {:type :offline}))))))) @@ -221,9 +263,9 @@ (rx/subject))))))) ;; Scenario: a persist-commit arrives with a superseded episode token after -;; a transient failure. Without the token guard it would fail terminally as -;; `:save-outcome-unknown`; with it, nothing happens. Proves: stale retry -;; timers stay silent. +;; a transient failure. Without the token guard it would send outside the +;; backoff schedule; with it, nothing happens. Proves: stale retry timers +;; stay silent. (t/deftest ^:async stale-retry-token-stays-silent (await (with-persistence @@ -333,7 +375,7 @@ (^:async fn [{:keys [file-id store]}] (await (mock/with-mocks* - {rx/timer (mock/stub (fn [_] (rx/of :tick)))} + {rx/timer (mock/stub instant-backoff)} (ptk/emit! store (local-commit file-id) ::dps/force-persist) (await (async/wait-for #(= :error (get-in @store [:persistence :status])) "retries exhaust into terminal error")) @@ -503,7 +545,7 @@ (^:async fn [{:keys [file-id requests store]}] (await (mock/with-mocks* - {rx/timer (mock/stub (fn [_] (rx/of :tick))) + {rx/timer (mock/stub instant-backoff) rx/timeout (mock/stub (fn [ms with ob] (swap! timeouts conj ms) (timeout 0 with ob)))} @@ -519,3 +561,324 @@ (if (= 1 (swap! calls inc)) (rx/subject) (rx/of {:revn 1}))))))) + +;; --------------------------------------------------------------------------- +;; slow cycle +;; --------------------------------------------------------------------------- + +(defn- slow-cycle-timer + "Timer double: the backoff delays fire at once, and every slow cycle waits + on a subject collected in `cycles`, so the test fires them one by one." + [cycles] + (fn [ms] + (if (= ms dps/slow-retry-delay-ms) + (let [cycle-s (rx/subject)] + (swap! cycles conj cycle-s) + cycle-s) + (rx/of :tick)))) + +;; Scenario: the backoff is spent while the network is down. The queue keeps +;; trying on the slow cycle, one send per cycle, without warning the user +;; again, and saves once the network is back. Proves: a session outlasting +;; the backoff still recovers on its own. +(t/deftest ^:async a-spent-backoff-keeps-trying-on-a-slow-cycle + (let [calls (atom 0) + cycles (atom [])] + (await + (with-persistence + (^:async fn [{:keys [file-id failures requests store]}] + (await + (mock/with-mocks* + {rx/timer (mock/stub (slow-cycle-timer cycles))} + (ptk/emit! store (local-commit file-id) ::dps/force-persist) + (await (async/wait-for #(and (= :error (get-in @store [:persistence :status])) + (= 1 (count @cycles))) + "the spent backoff arms the slow cycle")) + (t/is (= 4 (count @requests))) + (t/is (= 1 (count @failures))) + + (rx/push! (last @cycles) :tick) + (await (async/wait-for #(= 2 (count @cycles)) + "a failed slow attempt arms the next cycle")) + (t/is (= 5 (count @requests)) "a slow cycle sends once") + (t/is (= :error (get-in @store [:persistence :status]))) + (t/is (= 1 (count @failures)) "the user is not warned again") + + ;; The warning the first failure left on screen. + (ptk/emit! store #(assoc % :notification {:tag errors/persistence-failed-tag + :level :error})) + (rx/push! (last @cycles) :tick) + (await (async/wait-for #(and (= :saved (get-in @store [:persistence :status])) + (empty? (get-in @store [:persistence :queue]))) + "the slow attempt saves")) + (t/is (= 6 (count @requests))) + (t/is (apply = (map (comp :commit-id second) @requests)) + "every attempt carries the same commit id") + (t/is (nil? (get-in @store [:persistence :recovering]))) + (t/is (nil? (get-in @store [:persistence :failing-since]))) + (t/is (nil? (get @store :notification)) + "the failure warning goes away once the save lands")))) + (fn [_ _] + (if (< (swap! calls inc) 6) + (rx/throw (ex-info "offline" {:type :offline})) + (rx/of {:revn 1}))))))) + +;; Scenario: a gateway error outlasts the backoff, then the proxy recovers. +;; Proves: each gateway status takes the slow cycle and saves the held edit. +(t/deftest ^:async a-gateway-error-arms-the-slow-cycle + (doseq [status [504 520 521 522 523 524]] + (let [calls (atom 0) + cycles (atom [])] + (await + (with-persistence + (^:async fn [{:keys [file-id requests store]}] + (await + (mock/with-mocks* + {rx/timer (mock/stub (slow-cycle-timer cycles))} + (ptk/emit! store (local-commit file-id) ::dps/force-persist) + (await (async/wait-for #(= 1 (count @cycles)) + (str status " arms the slow cycle"))) + (t/is (= 4 (count @requests))) + (t/is (= :gateway-error (get-in @store [:persistence :error :cause-type]))) + (t/is (= 1 (count (get-in @store [:persistence :queue])))) + + (rx/push! (last @cycles) :tick) + (await (async/wait-for #(and (= :saved (get-in @store [:persistence :status])) + (empty? (get-in @store [:persistence :queue]))) + (str status " recovers on the slow cycle"))) + (t/is (= 5 (count @requests))) + (t/is (apply = (map (comp :commit-id second) @requests)) + "the slow cycle saves the held commit")))) + (fn [_ _] + (if (<= (swap! calls inc) 4) + (rp/handle-response {:status status :body "Gateway error" :uri "/api"}) + (rx/of {:revn 1})))))))) + +;; Scenario: a save fails for a reason no attempt can get past. Proves: only +;; transport failures arm the slow cycle. +(t/deftest ^:async a-terminal-failure-does-not-arm-the-slow-cycle + (let [cycles (atom [])] + (await + (with-persistence + (^:async fn [{:keys [file-id requests store]}] + (await + (mock/with-mocks* + {rx/timer (mock/stub (slow-cycle-timer cycles))} + (ptk/emit! store (local-commit file-id) ::dps/force-persist) + (await (async/wait-for #(= :error (get-in @store [:persistence :status])) + "the save fails")) + (await (async/settle)) + (t/is (= 1 (count @requests))) + (t/is (empty? @cycles))))) + (fn [_ _] (rx/throw (ex-info "Validation failed" {:type :validation}))))))) + +;; Scenario: a gateway answers 504 only after its own timeout, while the +;; backend may still be applying the save. Proves: slow failures below the +;; limit still arm the slow cycle; at the limit the save stops being sent, +;; and opening a file releases it instead of sending it again. +(t/deftest ^:async slow-failures-stop-the-slow-cycle-at-the-limit + (doseq [[slow-count armed?] [[(dec dps/max-slow-failures) true] + [dps/max-slow-failures false]]] + (let [clock (atom 0) + calls (atom 0) + cycles (atom [])] + (await + (with-persistence + (^:async fn [{:keys [file-id requests store]}] + (await + (mock/with-mocks* + {rx/timer (mock/stub (slow-cycle-timer cycles)) + ct/now (mock/stub #(ct/inst @clock))} + (ptk/emit! store (local-commit file-id) ::dps/force-persist) + (await (async/wait-for #(= :error (get-in @store [:persistence :status])) + (str slow-count " slow failures spend the backoff"))) + (await (async/settle)) + (t/is (= 4 (count @requests))) + (t/is (= slow-count (get-in @store [:persistence :slow-failures]))) + (t/is (= (if armed? 1 0) (count @cycles)) + (str slow-count " slow failures " (if armed? "arm" "do not arm") " the slow cycle")) + + (when-not armed? + (ptk/emit! store (ptk/data-event :app.main.data.common/change-team-role)) + (await (async/wait-for #(empty? (get-in @store [:persistence :queue])) + "recovery releases the held edit")) + (t/is (= 4 (count @requests)) "the released edit is not sent again"))))) + (fn [_ _] + (when (<= (swap! calls inc) slow-count) + (swap! clock + dps/slow-failure-ms)) + (rp/handle-response {:status 504 :body "Gateway timeout" :uri "/api"}))))))) + +;; Scenario: the slow cycle is still failing when the retry window closes. +;; Proves: the cycle gives up, keeping the edits queued. +(t/deftest ^:async the-slow-cycle-stops-when-the-retry-window-closes + (let [clock (atom 0) + cycles (atom [])] + (await + (with-persistence + (^:async fn [{:keys [file-id requests store]}] + (await + (mock/with-mocks* + {rx/timer (mock/stub (slow-cycle-timer cycles)) + ct/now (mock/stub #(ct/inst @clock))} + (ptk/emit! store (local-commit file-id) ::dps/force-persist) + (await (async/wait-for #(= 1 (count @cycles)) "the slow cycle is armed")) + + (reset! clock (+ dps/retry-give-up-ms 1000)) + (rx/push! (last @cycles) :tick) + (await (async/wait-for #(= 5 (count @requests)) "the last slow attempt")) + (await (async/wait-for #(= :error (get-in @store [:persistence :status])) + "the attempt fails")) + (await (async/settle)) + (t/is (= 1 (count @cycles)) "no further cycle is armed") + (t/is (= 1 (count (get-in @store [:persistence :queue]))))))) + (fn [_ _] (rx/throw (ex-info "offline" {:type :offline}))))))) + +;; Scenario: the backoff is spent and the user leaves the workspace, which +;; stops the slow cycle, then opens a file again. Recovery sends the queue +;; once more instead of releasing it, and it saves. Proves: edits held by a +;; transport failure are never dropped by recovery. +(t/deftest ^:async recovery-resends-a-transport-failure-instead-of-releasing-it + (let [calls (atom 0) + cycles (atom [])] + (await + (with-persistence + (^:async fn [{:keys [file-id requests store]}] + (await + (mock/with-mocks* + {rx/timer (mock/stub (slow-cycle-timer cycles))} + (ptk/emit! store (local-commit file-id) ::dps/force-persist) + (await (async/wait-for #(= 1 (count @cycles)) "the slow cycle is armed")) + (ptk/emit! store (ptk/data-event :app.main.data.workspace/finalize-workspace)) + (ptk/emit! store (ptk/data-event :app.main.data.common/change-team-role)) + (await (async/wait-for #(and (= :saved (get-in @store [:persistence :status])) + (empty? (get-in @store [:persistence :queue]))) + "recovery saves the held edit")) + (t/is (= 5 (count @requests))) + (t/is (apply = (map (comp :commit-id second) @requests)) + "recovery sends the same commit, not a released queue")))) + (fn [_ _] + (if (< (swap! calls inc) 5) + (rx/throw (ex-info "offline" {:type :offline})) + (rx/of {:revn 1}))))))) + +;; Leaving the workspace tests. + +;; Scenario: a save fails on a transport error and the user leaves the +;; workspace before the backoff fires. The pending retry stays silent, and +;; opening a file sends the held edit again. Proves: the backoff does not +;; run outside the workspace and resumes once a file is opened. +(t/deftest ^:async leaving-the-workspace-stops-the-backoff + (let [calls (atom 0) + timers (atom [])] + (await + (with-persistence + (^:async fn [{:keys [file-id requests store]}] + (await + (mock/with-mocks* + {rx/timer (mock/stub (fn [_] + (let [timer-s (rx/subject)] + (swap! timers conj timer-s) + timer-s)))} + (ptk/emit! store (local-commit file-id) ::dps/force-persist) + (await (async/wait-for #(= 1 (count @timers)) "the failure schedules a retry")) + + (ptk/emit! store (ptk/data-event :app.main.data.workspace/finalize-workspace)) + (await (async/wait-for #(get-in @store [:persistence :suspended]) + "leaving the workspace suspends retries")) + (rx/push! (first @timers) :tick) + (await (async/settle)) + (t/is (= 1 (count @requests)) "the pending retry stays silent") + (t/is (= :retrying (get-in @store [:persistence :status]))) + (t/is (= 1 (count (get-in @store [:persistence :queue])))) + + (ptk/emit! store (ptk/data-event :app.main.data.workspace/workspace-initialized)) + (await (async/wait-for #(and (= :saved (get-in @store [:persistence :status])) + (empty? (get-in @store [:persistence :queue]))) + "opening a file saves the held edit")) + (t/is (= 2 (count @requests))) + (t/is (apply = (map (comp :commit-id second) @requests)) + "the same commit is sent again") + (t/is (nil? (get-in @store [:persistence :suspended])))))) + (fn [_ _] + (if (= 1 (swap! calls inc)) + (rx/throw (ex-info "offline" {:type :offline})) + (rx/of {:revn 1}))))))) + +;; Scenario: the user leaves the workspace while a save is in flight, and the +;; save then fails on a transport error. No backoff is scheduled, the online +;; event does not resend, and opening a file does. Proves: a failure that +;; lands outside the workspace waits for the user instead of retrying there. +(t/deftest ^:async a-failure-after-leaving-the-workspace-arms-no-retry + (let [calls (atom 0) + timers (atom []) + first-s (rx/subject)] + (await + (with-persistence + (^:async fn [{:keys [file-id requests store]}] + (await + (mock/with-mocks* + {rx/timer (mock/stub (fn [ms] (swap! timers conj ms) (rx/subject)))} + (ptk/emit! store (local-commit file-id) ::dps/force-persist) + (await (async/wait-for #(= 1 (count @requests)) "the edit is sent")) + + (ptk/emit! store (ptk/data-event :app.main.data.workspace/finalize-workspace)) + (await (async/wait-for #(get-in @store [:persistence :suspended]) + "leaving the workspace suspends retries")) + (rx/error! first-s (ex-info "offline" {:type :offline})) + (await (async/wait-for #(= :retrying (get-in @store [:persistence :status])) + "the failure keeps the edit queued")) + (await (async/settle)) + (t/is (empty? @timers) "no backoff is scheduled") + + (ptk/emit! store (#'dps/resume-on-online)) + (await (async/settle)) + (t/is (= 1 (count @requests)) "the online event does not resend") + + (ptk/emit! store (ptk/data-event :app.main.data.workspace/workspace-initialized)) + (await (async/wait-for #(and (= :saved (get-in @store [:persistence :status])) + (empty? (get-in @store [:persistence :queue]))) + "opening a file saves the held edit")) + (t/is (= 2 (count @requests)))))) + (fn [_ _] + (if (= 1 (swap! calls inc)) + first-s + (rx/of {:revn 1}))))))) + +;; Scenario: the last backoff attempt is in flight when the user leaves the +;; workspace, and it fails. The queue halts without arming the slow cycle, +;; and opening a file sends it once more. Proves: the slow cycle never +;; starts outside the workspace. +(t/deftest ^:async a-spent-backoff-outside-the-workspace-arms-no-slow-cycle + (let [calls (atom 0) + cycles (atom []) + last-s (rx/subject)] + (await + (with-persistence + (^:async fn [{:keys [file-id requests store]}] + (await + (mock/with-mocks* + {rx/timer (mock/stub (slow-cycle-timer cycles))} + (ptk/emit! store (local-commit file-id) ::dps/force-persist) + (await (async/wait-for #(= 4 (count @requests)) "the last retry is sent")) + + (ptk/emit! store (ptk/data-event :app.main.data.workspace/finalize-workspace)) + (await (async/wait-for #(get-in @store [:persistence :suspended]) + "leaving the workspace suspends retries")) + (rx/error! last-s (ex-info "offline" {:type :offline})) + (await (async/wait-for #(= :error (get-in @store [:persistence :status])) + "the spent backoff halts the queue")) + (await (async/settle)) + (t/is (empty? @cycles) "no slow cycle is armed") + (t/is (= 1 (count (get-in @store [:persistence :queue])))) + + (ptk/emit! store (ptk/data-event :app.main.data.workspace/workspace-initialized)) + (await (async/wait-for #(and (= :saved (get-in @store [:persistence :status])) + (empty? (get-in @store [:persistence :queue]))) + "opening a file saves the held edit")) + (t/is (= 5 (count @requests)))))) + (fn [_ _] + (let [n (swap! calls inc)] + (cond + (< n 4) (rx/throw (ex-info "offline" {:type :offline})) + (= n 4) last-s + :else (rx/of {:revn 1})))))))) diff --git a/frontend/test/frontend_tests/data/persistence_test.cljs b/frontend/test/frontend_tests/data/persistence_test.cljs index 0f72a59025..7fa453aaf4 100644 --- a/frontend/test/frontend_tests/data/persistence_test.cljs +++ b/frontend/test/frontend_tests/data/persistence_test.cljs @@ -18,10 +18,12 @@ `mock/with-mocks` with an explicit `done` chain." (:require [app.common.time :as ct] + [app.common.types.shape :as cts] [app.common.uuid :as uuid] [app.main.data.changes :as dch] [app.main.data.persistence :as dps] [app.main.data.render-wasm :as drw] + [app.main.data.workspace :as dw] [app.main.errors :as errors] [app.main.repo :as rp] [app.main.store :as st] @@ -32,6 +34,74 @@ [frontend-tests.helpers.mock :as mock] [potok.v2.core :as ptk])) +(t/deftest reopening-restores-only-the-retained-edits-of-the-loaded-file + (doseq [[status cause-type vern restored?] + [[:retrying nil 0 true] + [:saving nil 0 true] + [:error :offline 0 true] + [:error :validation 0 false] + [:retrying nil 1 false]]] + (t/testing (str status " / " cause-type " / version " vern) + (let [file-id (uuid/next) + page-id (uuid/next) + shape-id (uuid/next) + first-id (uuid/next) + next-id (uuid/next) + other-id (uuid/next) + shape (cts/setup-shape {:id shape-id :type :rect :x 0 :y 0}) + file {:id file-id + :vern vern + :data {:pages [page-id] + :pages-index {page-id {:id page-id :name "Page" + :objects {uuid/zero + (cts/setup-shape {:id uuid/zero :type :frame})}}}}} + rename (fn [name] + {:type :mod-obj :id shape-id :page-id page-id + :operations [{:type :set :attr :name :val name}]}) + pstate {:status status + :error {:cause-type cause-type :commit-id first-id} + :queue (into #queue [] [first-id next-id other-id]) + :index {first-id {:file-id file-id :file-vern 0 + :redo-changes [{:type :add-obj :id shape-id :obj shape + :page-id page-id :parent-id uuid/zero + :frame-id uuid/zero}]} + next-id {:file-id file-id :file-vern 0 + :redo-changes [(rename "Retained rectangle")]} + other-id {:file-id (uuid/next) :file-vern 0 + :redo-changes [(rename "Another file")]}}} + state {:persistence pstate} + loaded (ptk/update (#'dw/bundle-fetched {:file file :file-id file-id :thumbnails {}}) + state) + restored (get-in loaded [:files file-id :data :pages-index page-id :objects shape-id])] + (t/is (= restored? (some? restored))) + (when restored? + (t/is (= "Retained rectangle" (:name restored)) + "queued edits apply in order and exclude other files")) + (t/is (identical? pstate (:persistence loaded)) + "restoring the editor must preserve the save queue"))))) + +;; Scenario: a queued edit no longer applies to the data the server sent, so +;; replaying it throws. Proves: the file still opens, as the server sent it, +;; and the edits stay queued. +(t/deftest reopening-with-an-edit-that-no-longer-applies-keeps-the-server-file + (let [file-id (uuid/next) + page-id (uuid/next) + commit-id (uuid/next) + data {:pages [page-id] + :pages-index {page-id {:id page-id :name "Page" + :objects {uuid/zero + (cts/setup-shape {:id uuid/zero :type :frame})}}}} + file {:id file-id :vern 0 :data data} + pstate {:status :saving + :queue (into #queue [] [commit-id]) + :index {commit-id {:file-id file-id :file-vern 0 + :redo-changes [{:type :mod-obj :id uuid/zero :page-id page-id + :operations [{:type :no-longer-supported}]}]}}} + loaded (ptk/update (#'dw/bundle-fetched {:file file :file-id file-id :thumbnails {}}) + {:persistence pstate})] + (t/is (identical? data (get-in loaded [:files file-id :data]))) + (t/is (identical? pstate (:persistence loaded))))) + (defn- local-commit "Builds a synthetic local commit event for `file-id`: a page edit as redo, no undo, from the `:local` source." @@ -430,8 +500,8 @@ ;; - Transport failures and malformed answers (`nil`, empty, bad revision) ;; fail the save but preserve the queue for retry. ;; - Initialization recovers gracefully: dangling runners error instead of -;; skipping, unsent commits are picked up, unknown outcomes are reported -;; without replaying. +;; skipping, unsent commits are picked up, unknown outcomes are sent again +;; under the same commit id. ;; ;; Same async pattern as the watchdog tests: triggers stay bare, every ;; assert block is preceded by `wait-for` on its leading signal (or a bare @@ -477,11 +547,10 @@ (rx/end! response)))))) ;; Scenario: edit permission is revoked with an edit queued. The save fails -;; without sending anything, the queue keeps the edit, and the user is -;; notified once; further edits keep queueing behind the failure instead of -;; replacing it. Proves: permission loss fails safe — no send, no discard, -;; one flash. -(t/deftest ^:async permission-loss-fails-without-discarding-queued-edits +;; without sending anything, the queue keeps the edit and the user is +;; notified once, so `wait-persisted-or-error` still rejects and nothing is +;; snapshotted from unsaved state. Proves: permission loss fails safe. +(t/deftest ^:async permission-loss-retains-the-queue-and-notifies-once (await (with-persistence (^:async fn [{:keys [file-id requests failures store]}] @@ -492,37 +561,76 @@ (= 1 (count (get-in @store [:persistence :queue])))) "permission loss fails the save")) (t/is (= :error (get-in @store [:persistence :status]))) - (t/is (= 1 (count (get-in @store [:persistence :queue])))) + (t/is (= 1 (count (get-in @store [:persistence :queue]))) + "the undeliverable edit is retained, so waiters keep rejecting") (t/is (empty? @requests)) - (t/is (= 1 (count @failures))) - (ptk/emit! store (local-commit file-id) ::dps/force-persist - (#'dps/update-status :pending)) - (await (async/wait-for #(and (= :error (get-in @store [:persistence :status])) - (= 2 (count (get-in @store [:persistence :queue])))) - "second edit queues behind the failure")) - (t/is (= :error (get-in @store [:persistence :status]))) - (t/is (= 2 (count (get-in @store [:persistence :queue])))))))) + (t/is (= 1 (count @failures))))))) -;; Scenario: a request with two queued commits fails terminally at the -;; transport. Both edits stay queued as failed; reinitializing persistence -;; must not replay the dead request. Proves: failed requests preserve the -;; queue and are never retried on initialization. -(t/deftest ^:async failed-request-retains-the-queue-and-is-not-retried-on-initialization +;; Scenario: the same terminal failure, then the user opens a file again. +;; Opening a file releases the edit that can never be saved and settles the +;; editor, instead of leaving it halted until a reload. Proves: the halt does +;; not outlive the visit that caused it. +(t/deftest ^:async opening-a-file-again-releases-a-terminal-failure (await (with-persistence - (^:async fn [{:keys [file-id response requests store]}] - (ptk/emit! store (local-commit file-id) ::dps/force-persist - (local-commit file-id) ::dps/force-persist) - (.error response (ex-info "Validation failed" {:type :validation})) - (await (async/wait-for #(= :error (get-in @store [:persistence :status])) - "failed request errors the save")) - (ptk/emit! store (dps/initialize-persistence)) + (^:async fn [{:keys [file-id requests failures store]}] + (ptk/emit! store (local-commit file-id) + #(assoc-in % [:permissions :can-edit] false) + ::dps/force-persist) (await (async/wait-for #(and (= :error (get-in @store [:persistence :status])) - (= 2 (count (get-in @store [:persistence :queue])))) - "initialization preserves the failed queue")) - (t/is (= :error (get-in @store [:persistence :status]))) - (t/is (= 2 (count (get-in @store [:persistence :queue])))) - (t/is (= 1 (count @requests))))))) + (= 1 (count (get-in @store [:persistence :queue])))) + "the save fails terminally")) + ;; The user acts again: a team-role change is one of the two real + ;; recovery triggers (the other is opening a file, whose event does + ;; real work and is not usable as a bare signal here). + (ptk/emit! store (ptk/data-event :app.main.data.common/change-team-role)) + (await (async/wait-for #(and (= :saved (get-in @store [:persistence :status])) + (empty? (get-in @store [:persistence :queue]))) + "re-entering the file settles the queue")) + (t/is (nil? (get-in @store [:persistence :error])) + "the halt is cleared once the user comes back") + (t/is (empty? @requests) + "the edit that cannot be saved is not replayed on re-entry") + (t/is (= 1 (count @failures)) "the user is still notified exactly once"))))) + +;; Scenario: two files have pending edits and the first send fails +;; terminally. Opening a file again releases the failed file, the other file +;; still saves, and the editor settles saved instead of staying halted. +;; Proves: one file that cannot be saved does not block every other file. +(t/deftest ^:async terminal-failure-does-not-block-other-files + (let [files (into [] (repeatedly 2 uuid/next)) + sends (atom 0)] + (await + (with-persistence + (^:async fn [{:keys [requests store]}] + (ptk/emit! store + #(assoc % :files (into (:files %) + (map (fn [id] [id {:id id :revn 0}])) + files)) + (local-commit (nth files 0)) + (local-commit (nth files 1)) + ::dps/force-persist) + (await (async/wait-for #(seq @requests) "the first file is sent")) + (let [failed-file (get-in (second (first @requests)) [:id])] + (await (async/wait-for #(and (= :error (get-in @store [:persistence :status])) + (not (empty? (get-in @store [:persistence :queue])))) + "the terminal failure retains the queue")) + ;; The user acts again: recovery releases the failed file and drains + ;; the rest. + (ptk/emit! store (ptk/data-event :app.main.data.common/change-team-role)) + (await (async/wait-for #(and (= :saved (get-in @store [:persistence :status])) + (empty? (get-in @store [:persistence :queue]))) + "the released file frees the rest of the queue")) + (t/is (= 2 (count @requests)) + "the other file was saved after the failed one was released") + (t/is (= 1 (count (filter #(= failed-file (get-in (second %) [:id])) @requests))) + "the failed file is sent once, never replayed"))) + (fn [_ _] + ;; Fail the first send terminally and answer the rest inline, so no + ;; hot-subject race can swallow the answer. + (if (= 1 (swap! sends inc)) + (rx/throw (ex-info "rejected" {:type :validation :code :invalid-data})) + (rx/of {:revn 1}))))))) (defn- check-save-failure-warning "Fails a queued save with `cause-type` and asserts the hint shown to the @@ -687,35 +795,53 @@ (t/deftest ^:async save-error-does-not-resend-an-active-request (await (check-active-request-not-resent [(ptk/data-event ::dps/error)]))) -;; Scenario: two commits go out, permission is lost mid-flight (first request -;; fails, one edit stays queued as failed), then permission is restored with -;; a team-role change. Only the unsent edit is resent; once answered, the -;; file saves with an empty queue. Proves: restoration resumes exactly the -;; unsent edits. -(t/deftest ^:async permission-restoration-resumes-only-unsent-edits +;; Scenario: permission is lost mid-flight, so the save fails terminally. +;; Even if the permission comes back, recovery releases the undeliverable +;; edits instead of replaying them. The backend raises `:not-found` for a +;; lost edit permission, so a resend would only route the user to the +;; exception page. Proves: permission loss is treated as permanent, like +;; every other terminal failure. +(t/deftest ^:async permission-loss-is-released-not-resumed (await (with-persistence - (^:async fn [{:keys [file-id response requests store]}] - (ptk/emit! store (local-commit file-id) ::dps/force-persist - (local-commit file-id) ::dps/force-persist - #(assoc-in % [:permissions :can-edit] false)) - (rx/push! response {:revn 1}) + (^:async fn [{:keys [file-id requests store]}] + (ptk/emit! store (local-commit file-id) + #(assoc-in % [:permissions :can-edit] false) + ::dps/force-persist) (await (async/wait-for #(and (= :error (get-in @store [:persistence :status])) (= 1 (count (get-in @store [:persistence :queue])))) "permission loss fails the save")) - (t/is (= :error (get-in @store [:persistence :status]))) - (t/is (= 1 (count (get-in @store [:persistence :queue])))) - (ptk/emit! store - #(assoc-in % [:permissions :can-edit] true) + ;; Permission comes back and a file is opened again. + (ptk/emit! store #(assoc-in % [:permissions :can-edit] true) (ptk/data-event :app.main.data.common/change-team-role)) - (await (async/wait-for #(= 2 (count @requests)) "restoration resends")) - (t/is (= 2 (count @requests))) - (rx/push! response {:revn 2}) (await (async/wait-for #(and (= :saved (get-in @store [:persistence :status])) (empty? (get-in @store [:persistence :queue]))) - "resumed save lands")) - (t/is (= :saved (get-in @store [:persistence :status]))) - (t/is (empty? (get-in @store [:persistence :queue]))))))) + "recovery releases the undeliverable edit")) + (t/is (empty? @requests) + "the undeliverable edit is never replayed, not even once the + permission is back") + (t/is (not= :error (get-in @store [:persistence :status])) + "the editor is not left halted"))))) + +;; Scenario: a terminal failure is waiting, and persistence is re-initialized. +;; Initialization must not lift the halt: only opening a file does. Otherwise +;; the edits would be released the moment the workspace view mounts, instead +;; of when the user acts on them. Proves: the init/recovery split holds. +(t/deftest ^:async initialization-does-not-lift-a-halt + (await + (with-persistence + (^:async fn [{:keys [file-id response store]}] + (ptk/emit! store (local-commit file-id) ::dps/force-persist) + (.error response (ex-info "Validation failed" {:type :validation})) + (await (async/wait-for #(and (= :error (get-in @store [:persistence :status])) + (= 1 (count (get-in @store [:persistence :queue])))) + "the save fails terminally")) + (ptk/emit! store (dps/initialize-persistence)) + (await (async/settle)) + (t/is (= :error (get-in @store [:persistence :status])) + "initialization leaves the halt alone") + (t/is (= 1 (count (get-in @store [:persistence :queue]))) + "and keeps the undeliverable edits queued until the user acts"))))) ;; Scenario: the answer arrives but no runner is tracking the request ;; anymore. The acknowledged commit stays queued instead of being dropped; @@ -740,27 +866,26 @@ (t/is (empty? (get-in @store [:persistence :queue]))))))) ;; Scenario: a queued commit carries a request id but persistence has no -;; run id for it when initializing — its outcome is unknowable. It errors -;; as `:save-outcome-unknown`, stays queued, and nothing is sent. -;; Proves: unknown outcomes report without replaying (never assume persisted, -;; never resend blindly). -(t/deftest ^:async recovery-reports-an-unknown-request-outcome-without-replaying-it +;; run id for it when initializing, so its outcome is unknown. It is sent +;; again under the same commit id and saves. Proves: an unknown outcome is +;; resent, since changes are idempotent. +(t/deftest ^:async recovery-sends-again-a-save-with-an-unknown-outcome (await (with-persistence - (^:async fn [{:keys [file-id requests store]}] + (^:async fn [{:keys [file-id response requests store]}] (let [commit (assoc @(local-commit file-id) ::dps/request-id (uuid/next)) id (:id commit)] (ptk/emit! store #(assoc % :persistence {:queue (conj #queue [] id) :index {id commit} :status :saving}) (dps/initialize-persistence)) - (await (async/wait-for #(= :save-outcome-unknown - (get-in @store [:persistence :error :code])) - "unknown outcome errors")) - (t/is (= :error (get-in @store [:persistence :status]))) - (t/is (= :save-outcome-unknown (get-in @store [:persistence :error :code]))) - (t/is (= [id] (vec (get-in @store [:persistence :queue])))) - (t/is (empty? @requests))))))) + (await (async/wait-for #(= 1 (count @requests)) "the commit is sent again")) + (t/is (= id (:commit-id (second (first @requests))))) + (rx/push! response {:revn 1}) + (await (async/wait-for #(and (= :saved (get-in @store [:persistence :status])) + (empty? (get-in @store [:persistence :queue]))) + "the resent commit saves")) + (t/is (= 1 (count @requests)))))))) ;; Scenario: an edit is queued before persistence even initializes. On ;; initialization plus force-persist it is sent exactly once (still queued @@ -842,15 +967,24 @@ "the second save turns the guard off")) (t/is (empty? (get-in @store [:persistence :queue]))))))) -;; Scenario: the save fails terminally. The edit stays queued, and so does -;; the guard. Proves: retained edits are not dropped silently on leave. -(t/deftest ^:async failed-save-keeps-the-leave-page-guard +;; Scenario: a save fails terminally, then recovery releases the edit. The +;; guard stays on while the failed edit is retained, and turns off once it +;; is released. Proves: a release that empties the queue also turns the +;; leave-page warning off. +(t/deftest ^:async releasing-a-failed-edit-turns-the-leave-page-guard-off (await (with-persistence (^:async fn [{:keys [file-id store tasks]}] - (ptk/emit! store (local-commit file-id) ::dps/force-persist) + (ptk/emit! store (local-commit file-id) + #(assoc-in % [:permissions :can-edit] false) + ::dps/force-persist) (await (async/wait-for #(= :error (get-in @store [:persistence :status])) - "the save fails")) - (t/is (= 1 (count (get-in @store [:persistence :queue])))) - (t/is (contains? @tasks :persistence) "the guard stays on for retained edits")) - (fn [_ _] (rx/throw (ex-info "invalid" {:type :validation})))))) + "the save fails terminally")) + (await (async/settle)) + (t/is (contains? @tasks :persistence) "the retained edit keeps the guard on") + (ptk/emit! store (ptk/data-event :app.main.data.common/change-team-role)) + (await (async/wait-for #(empty? (get-in @store [:persistence :queue])) + "recovery releases the edit")) + (await (async/wait-for #(not (contains? @tasks :persistence)) + "the release turns the guard off")) + (t/is (= :saved (get-in @store [:persistence :status]))))))) diff --git a/frontend/test/frontend_tests/data/repo_test.cljs b/frontend/test/frontend_tests/data/repo_test.cljs index 13066fbe9e..5334f1b719 100644 --- a/frontend/test/frontend_tests/data/repo_test.cljs +++ b/frontend/test/frontend_tests/data/repo_test.cljs @@ -8,7 +8,8 @@ (:require [app.main.repo :as repo] [beicon.v2.core :as rx] - [cljs.test :as t :include-macros true])) + [cljs.test :as t :include-macros true] + [cuerdas.core :as str])) ;; --------------------------------------------------------------------------- ;; retryable-error? tests (synchronous) @@ -29,6 +30,11 @@ (let [err (ex-info "service unavailable" {:type :service-unavailable})] (t/is (true? (repo/retryable-error? err)))))) +(t/deftest retryable-error-gateway-error + (t/testing "504 and the Cloudflare 52x family are retryable" + (let [err (ex-info "http error" {:type :gateway-error})] + (t/is (true? (repo/retryable-error? err)))))) + (t/deftest retryable-error-offline (t/testing "offline (status 0) is retryable" (let [err (ex-info "offline" {:type :offline})] @@ -215,3 +221,123 @@ (t/is (= 2 @call-count)) (t/is (= :authentication (:type (ex-data err)))) (done)))))))) + +;; --------------------------------------------------------------------------- +;; handle-response classification tests +;; --------------------------------------------------------------------------- + +(defn- capture-error + "Subscribes to `handle-response` for `response` and calls `k` with the + `ex-data` of the raised error." + [response k] + (->> (repo/handle-response response) + (rx/subs! + (fn [_] (t/is false "expected an error response") (k nil)) + (fn [cause] (k (ex-data cause)))))) + +(def ^:private html-error-page + "Web Page Blockedblocked") + +(t/deftest handle-response-gateway-statuses + (t/testing "every edge status that never reached the backend is a gateway error" + (t/async done + (let [statuses [504 520 521 522 523 524] + pending (atom (count statuses))] + (doseq [status statuses] + (capture-error {:status status + :body html-error-page + :headers {"content-type" "text/html"} + :uri "https://design.penpot.app/api/main/methods/get-teams"} + (fn [data] + (t/is (= :gateway-error (:type data)) + (str "status " status " should be a gateway error")) + (t/is (= status (:status data)) + (str "status " status " should be carried through")) + (when (zero? (swap! pending dec)) + (done))))))))) + +(t/deftest handle-response-rate-limit + (t/testing "a 429 with no interpretable body is a rate-limit error" + (t/async done + (capture-error {:status 429 + :body "" + :headers {} + :uri "https://design.penpot.app/api/main/methods/get-teams"} + (fn [data] + (t/is (= :rate-limit (:type data))) + (t/is (= 429 (:status data))) + (done)))))) + +(t/deftest handle-response-intercepted-by-proxy + (t/testing "an error page from a filtering proxy is not an internal error" + (t/async done + (capture-error {:status 403 + :body html-error-page + :headers {"content-type" "text/html" + "server" "Zscaler/6.2"} + :uri "https://design.penpot.app/api/main/methods/update-file"} + (fn [data] + (t/is (= :unexpected-response (:type data))) + (t/is (= 403 (:status data))) + (done)))))) + +(t/deftest handle-response-keeps-body-excerpt + (t/testing "a body too long to keep is cut short and marked as cut" + (t/async done + (capture-error {:status 403 + :body (apply str (repeat 5000 "x")) + :headers {"content-type" "text/html"} + :uri "https://design.penpot.app/api/main/methods/update-file"} + (fn [data] + (t/is (= 503 (count (:data data))) "500 characters and an ellipsis") + (t/is (str/ends-with? (:data data) "...")) + (done)))))) + +(t/deftest handle-response-keeps-a-short-body-whole + (t/testing "a body short enough to read is passed through untouched" + (t/async done + (capture-error {:status 403 + :body html-error-page + :headers {"content-type" "text/html"} + :uri "https://design.penpot.app/api/main/methods/update-file"} + (fn [data] + (t/is (= html-error-page (:data data))) + (done)))))) + +(t/deftest handle-response-challenge-still-authorization + (t/testing "a Cloudflare challenge keeps its own type" + (t/async done + (capture-error {:status 403 + :body html-error-page + :headers {"server" "cloudflare" + "cf-mitigated" "challenge"} + :uri "https://design.penpot.app/api/main/methods/get-teams"} + (fn [data] + (t/is (= :authorization (:type data))) + (t/is (= :challenge-required (:code data))) + (done)))))) + +(t/deftest handle-response-keeps-backend-error + (t/testing "an error the backend itself reported is passed through untouched" + (t/async done + (capture-error {:status 400 + :body {:type :validation :code :missing-param} + :headers {"content-type" "application/transit+json"} + :uri "https://design.penpot.app/api/main/methods/update-file"} + (fn [data] + (t/is (= :validation (:type data))) + (t/is (= :missing-param (:code data))) + (t/is (= 400 (:status data))) + (done)))))) + +(t/deftest handle-response-undecodable-non-error-is-internal + (t/testing "a non-error response we cannot interpret is still an internal error" + (t/async done + (capture-error {:status 301 + :body html-error-page + :headers {"content-type" "text/html"} + :uri "https://design.penpot.app/api/main/methods/get-teams"} + (fn [data] + (t/is (= :internal (:type data))) + (t/is (= :unable-to-process-repository-response (:code data))) + (done)))))) diff --git a/frontend/test/frontend_tests/errors_governor_test.cljs b/frontend/test/frontend_tests/errors_governor_test.cljs index c0998a7280..a51f775a43 100644 --- a/frontend/test/frontend_tests/errors_governor_test.cljs +++ b/frontend/test/frontend_tests/errors_governor_test.cljs @@ -2,7 +2,7 @@ ;; License, v. 2.0. If a copy of the MPL was not distributed with this ;; file, You can obtain one at http://mozilla.org/MPL/2.0/. ;; -;; Copyright (c) KALEIDOS SUBSIDIARY SL +;; Copyright (c) KALEIDOS INC Sucursal en España SL (ns frontend-tests.errors-governor-test "Unit tests for the error report governor (`app.main.errors`). diff --git a/frontend/test/frontend_tests/helpers/async.cljs b/frontend/test/frontend_tests/helpers/async.cljs index 36bbaed769..1e52a49c25 100644 --- a/frontend/test/frontend_tests/helpers/async.cljs +++ b/frontend/test/frontend_tests/helpers/async.cljs @@ -2,7 +2,7 @@ ;; License, v. 2.0. If a copy of the MPL was not distributed with this ;; file, You can obtain one at http://mozilla.org/MPL/2.0/. ;; -;; Copyright (c) KALEIDOS SUBSIDIARY SL +;; Copyright (c) KALEIDOS INC Sucursal en España SL (ns frontend-tests.helpers.async "Async-first plumbing for ClojureScript tests. diff --git a/frontend/test/frontend_tests/helpers/mock.cljc b/frontend/test/frontend_tests/helpers/mock.cljc index b55b8e24bd..c5a2d8eb49 100644 --- a/frontend/test/frontend_tests/helpers/mock.cljc +++ b/frontend/test/frontend_tests/helpers/mock.cljc @@ -29,7 +29,11 @@ Requires a done-chained context (usually `t/async`): a synchronous test completes before its deferred body runs. - Usage: `(with-mocks {ns/sym mock-fn, ...} test-fn done)`" + Usage: `(with-mocks {ns/sym mock-fn, ...} test-fn done)`. + + `with-mocks*` is the preferred form for new tests: it wraps the body in a + generated `^:async` fn and evaluates to a promise, so there is no `done` to + thread. See [[run-mocked]] for the restore-on-settle machinery." #?(:cljs (:require [beicon.v2.core :as rx] [cljs.test :as t])) @@ -150,6 +154,50 @@ (^:async fn [] ~@body) (fn [] ~@restore-exprs))))))) +#?(:clj + (defmacro with-mocks* + "Installs `mocks` via `set!`, then evaluates `body` inside a generated + `^:async` fn awaited via `run-mocked`: when the body settles, the + originals are restored — always. A rejection is reported as an `:error`. + + Evaluates to a promise resolving once the body settles and the mocks + are restored. `await` it, either directly or by returning it from an + `^:async` fn; nested scopes must be awaited too. + + Example: + + (t/deftest ^:async my-async-test + (await (mock/with-mocks* + {app.main.repo/cmd! mock/rpc-cmd-mock} + (await (some-async-flow)) + ...)))" + [mocks & body] + (let [entries (map identity mocks) + gen-pairs (mapv (fn [[qsym _mock]] + {:qsym qsym + :osym (gensym "orig-")}) + entries) + let-bindings (vec (mapcat + (fn [{:keys [qsym osym]}] + [osym qsym]) + gen-pairs)) + install-exprs (mapv + (fn [[_qsym mock-fn] {:keys [qsym]}] + `(set! ~qsym ~mock-fn)) + entries + gen-pairs) + restore-exprs (mapv + (fn [{:keys [qsym osym]}] + `(set! ~qsym ~osym)) + gen-pairs)] + `(do + (frontend-tests.helpers.mock/reset-state!) + (let ~let-bindings + ~@install-exprs + (frontend-tests.helpers.mock/run-mocked + (^:async fn [] ~@body) + (fn [] ~@restore-exprs))))))) + ;; ═══════════════════════════════════════════════════════════════ ;; Runtime (ClojureScript only) ;; ═══════════════════════════════════════════════════════════════ diff --git a/frontend/test/frontend_tests/main_errors_test.cljs b/frontend/test/frontend_tests/main_errors_test.cljs index c622647f84..5954eab457 100644 --- a/frontend/test/frontend_tests/main_errors_test.cljs +++ b/frontend/test/frontend_tests/main_errors_test.cljs @@ -135,6 +135,33 @@ [events] (filter #(= ::ev/event (ptk/type %)) events)) +(defn- ^:async report-names + "Drives `flash-persistence` with an `ex-data` map of `cause-data` and + returns the audit event names it emitted, in order. + + The report generator is stubbed and the scheduler runs its callback + inline, so the whole flash body completes before returning. The emitter + is the only observable." + [cause-data] + (let [events (atom []) + cause (ex-info "Save failed" (assoc cause-data :hint "Save failed"))] + ;; The governor is global and remembers fingerprints for two minutes, so + ;; a second identical report would be suppressed. Clear it per case. + (errors/reset-report-governor!) + (await + (mock/with-mocks* + {errors/generate-report (fn [_ & _] "generated error report") + st/emit! (mock/stub (fn [& emitted] + (swap! events into emitted))) + ;; `schedule` is multi-arity, so the mock needs every arity the + ;; flash body may call it with. + tm/schedule (mock/stub (fn + ([] nil) + ([callback] (callback)) + ([_ms callback] (callback))))} + (errors/flash-persistence cause))) + (mapv #(-> % deref ::ev/name) (report-events @events)))) + ;; --------------------------------------------------------------------------- ;; Environment failures ;; @@ -145,7 +172,7 @@ (t/deftest environment-error-classification (t/testing "environment failures are recognised" (doseq [type [:network :offline :bad-gateway :service-unavailable - :nitrate-unavailable :nitrate-not-configured]] + :gateway-error :nitrate-unavailable :nitrate-not-configured]] (t/is (true? (errors/environment-error? (error-cause :type type))) (str "expected environment error: " type)))) (t/testing "application defects are not environment failures" @@ -191,12 +218,29 @@ ;; evaluates to a promise, so the body must settle one. (await (async/settle)))))) +(t/deftest ^:async generate-report-bounds-a-huge-exception-message + ;; A message can embed a whole payload (a serialized file) and reach + ;; megabytes. The report must keep a bounded head in both the context and + ;; the trace, so the audit event built from it can never bloat. + (await + (mock/with-mocks* + {st/format-last-events (mock/stub (fn [& _] "(stub last events)")) + rt/get-current-href (constantly "https://penpot.example.com/#/workspace")} + (let [huge (apply str (repeat 200000 "z")) + report (errors/generate-report (js/Error. huge))] + (t/is (string? report)) + (t/is (< (count report) 50000)) + (t/is (not (str/includes? report (subs huge 0 1500)))) + ;; Trailing settle: the body is synchronous, but `with-mocks*` + ;; evaluates to a promise, so the body must settle one. + (await (async/settle)))))) + (t/deftest ^:async connectivity-handlers-report-governed-compact-audit-events ;; Scenario: a network failure and an offline status through the global ;; handler. Each is reported as a governed compact audit event plus its ;; connection toast. Proves: per type, report first and toast right after, ;; with no stack in the report and the dedicated toast message. - (doseq [type [:network :offline]] + (doseq [type [:network :offline :gateway-error]] (errors/reset-report-governor!) (let [events (atom []) cause (ex-info "http error" {:type type :hint "http error"})] @@ -803,7 +847,67 @@ ((:callback download) nil) (t/is (= [["report" "text/plain" "blob:report"]] @downloads)) (doseq [callback @idle-callbacks] (callback)) - (t/is (= ["blob:report"] @revoked))))))) + (t/is (= ["blob:report"] @revoked)) + ;; Autosave is stopped for the whole session, so the toast also + ;; offers the only action that restarts it without the user having + ;; to know they could leave the file and come back. + (let [reload (get-in state [:notification :links 1])] + (t/is (= "labels.reload-page:" (:label reload)) + "the `tr` stub renders a one-argument key with a trailing colon") + (t/is (fn? (:callback reload)) "the reload link has a callback"))))))) + +;; Scenario: a save fails terminally. The toast offers the report download +;; and a reload. Proves: the recovery action reaches the user. +;; +;; The reload callback itself is not invoked here: `js/location` does not +;; exist under the test runner, so the link is verified by its label and by +;; being a callable, not by calling it. +(t/deftest ^:async save-failure-toast-offers-a-reload-action + (let [events (atom [])] + (await + (mock/with-mocks* + {errors/generate-report (fn [_ & _] "generated error report") + st/emit! (mock/stub (fn [& emitted] + (swap! events into emitted))) + tm/schedule (mock/stub (fn [callback] (callback)))} + (errors/flash-persistence (ex-info "Save failed" + {:type :validation :hint "Save failed"})) + (await (async/settle)) + ;; The report is emitted first and the toast right after, so the toast + ;; is the last event collected. + (let [state (ptk/update (last @events) {}) + links (get-in state [:notification :links])] + ;; `tr` is not stubbed here, and the test runner does not load the + ;; catalogue, so a key comes back verbatim without its arguments. + (t/is (= ["labels.download" "labels.reload-page"] (mapv :label links)) + "the toast carries the report and a reload, download first") + (t/is (every? #(fn? (:callback %)) links) + "both actions are callable") + (t/is (nil? (get-in state [:notification :timeout])) + "the toast stays until the user acts on it")))))) + +;; Scenario: a save fails on the transport and the queue keeps trying on +;; the slow cycle. The toast says the changes are kept and offers no reload, +;; which would lose them. Proves: the retrying variant tells the truth. +(t/deftest ^:async retrying-save-failure-toast-offers-no-reload + (let [events (atom [])] + (await + (mock/with-mocks* + {errors/generate-report (fn [_ & _] "generated error report") + st/emit! (mock/stub (fn [& emitted] + (swap! events into emitted))) + tm/schedule (mock/stub (fn [callback] (callback)))} + (errors/flash-persistence (ex-info "Save failed" + {:type :gateway-error :hint "Save failed"}) + :retrying true) + (await (async/settle)) + (let [state (ptk/update (last @events) {})] + (t/is (= "errors.save-still-retrying" (get-in state [:notification :content]))) + (t/is (= ["labels.download"] (mapv :label (get-in state [:notification :links]))) + "the toast carries the report and no reload") + (t/is (= errors/persistence-failed-tag (get-in state [:notification :tag])) + "a save that lands can hide it") + (t/is (nil? (get-in state [:notification :timeout])))))))) (t/deftest ^:async persistence-waiters-do-not-report-an-already-handled-failure (let [reports (atom []) @@ -906,25 +1010,86 @@ (t/is (= [::dw/reload-current-file] (mapv ptk/type @events))) (t/is (empty? @scheduled) "A restored version shows no save notification"))))) -(t/deftest ^:async other-validation-failures-during-save-keep-the-notification - (t/testing "a validation failure without a recovery of its own is still notified" - (let [events (atom []) - scheduled (atom [])] +(t/deftest ^:async save-failure-report-type-follows-the-cause + ;; The user still gets the same sticky toast and report link; only the audit + ;; event name changes, and that is what decides whether the incident reaches + ;; the internal reports and the alerts. + (t/testing "a validation failure is a defect, so it reports unhandled" + (t/is (= ["unhandled-exception"] + (await (report-names {:type :validation :code :invalid-data}))))) + + (t/testing "the wrapped shape reads the cause, not the wrapper" + ;; This is the shape that actually reaches flash-persistence: + ;; persistence-failed sets :type :persistence and keeps the real type + ;; under :cause-type. Reading :type alone would match the wrapper and + ;; leave the cause as handled. + (t/is (= ["unhandled-exception"] + (await (report-names {:type :persistence + :cause-type :validation + :code :invalid-data}))))) + + (t/testing "environment failures stay handled after the retry budget" + ;; They land here once the backoff is spent, and the user's connection is + ;; not our bug, so they must not alert. + (doseq [cause-type [:network :offline :bad-gateway :service-unavailable + :gateway-error]] + (t/is (= ["handled-exception"] + (await (report-names {:type :persistence + :cause-type cause-type}))) + (str cause-type " is an environment failure, not a defect")))) + + (t/testing "responses the backend never wrote stay handled" + (doseq [cause-type [:rate-limit :unexpected-response]] + (t/is (= ["handled-exception"] + (await (report-names {:type :persistence + :cause-type cause-type}))) + (str cause-type " comes from a proxy, not from our code")))) + + (t/testing "an unclassified failure keeps the handled path" + (t/is (= ["handled-exception"] + (await (report-names {:type :persistence + :cause-type :unknown-thing}))))) + + (t/testing "a bare :type still classifies, without a wrapper" + (t/is (= ["unhandled-exception"] + (await (report-names {:type :validation}))))) + + (t/testing "a server-side or internal defect reports unhandled" + (doseq [cause-type [:server-error :internal]] + (t/is (= ["unhandled-exception"] + (await (report-names {:type :persistence + :cause-type cause-type}))) + (str cause-type " is our bug"))))) + +;; --------------------------------------------------------------------------- +;; transport errors +;; --------------------------------------------------------------------------- + +(t/deftest ^:async transport-errors-are-handled-without-replacing-the-page + ;; Scenario: a rate limit and an error page the backend did not write, + ;; through the global handler. Proves: neither replaces the page, each is + ;; a handled report followed by one toast. + (doseq [type [:rate-limit :unexpected-response]] + (errors/reset-report-governor!) + (let [events (atom []) + assigned (atom []) + cause (ex-info "http error" + {:type type + :hint "http error" + :status 429 + :uri "https://design.penpot.app/api/main/methods/get-teams"})] (await (mock/with-mocks* - {errors/generate-report (fn [_ & _] "generated error report") - errors/submit-report (fn [& _]) - st/emit! (mock/stub (fn [& emitted] - (swap! events into emitted))) - tm/schedule (mock/stub (fn [callback] - (swap! scheduled conj callback)))} - (errors/flash-persistence (ex-info "Invalid data" {:type :validation - :code :invalid-data})) - (doseq [callback @scheduled] (callback)) + {st/emit! (mock/stub (fn [& emitted] (swap! events into emitted))) + rt/get-current-href (constantly "https://penpot.example.com/#/workspace") + rt/assign-exception (fn [error] (swap! assigned conj error) error) + tm/schedule (mock/stub (fn [f] (f))) + st/format-last-events (mock/stub (fn [& _] "(stub last events)"))} + (errors/on-error cause) (await (async/settle)) - ;; The report is emitted first and the toast right after: one - ;; audit event plus the sticky toast carrying the report link. - (t/is (= 1 (count (report-events @events)))) - (let [state (ptk/update (last @events) {})] - (t/is (nil? (get-in state [:notification :timeout]))) - (t/is (some? (get-in state [:notification :links 0]))))))))) + (t/is (empty? @assigned) + (str type " must not replace the page with an error screen")) + (t/is (= ["handled-exception"] + (map (comp ::ev/name deref) (report-events @events))) + (str type " is reported as handled, not unhandled")) + (t/is (= 2 (count @events)) (str type " must show one notification"))))))) diff --git a/frontend/test/frontend_tests/runner.cljs b/frontend/test/frontend_tests/runner.cljs index 93bae2969f..df0fdb2c93 100644 --- a/frontend/test/frontend_tests/runner.cljs +++ b/frontend/test/frontend_tests/runner.cljs @@ -17,6 +17,7 @@ [frontend-tests.data.fix-deleted-fonts-test] [frontend-tests.data.imports-test] [frontend-tests.data.jobs-test] + [frontend-tests.data.modal-test] [frontend-tests.data.nitrate-test] [frontend-tests.data.persistence-retry-test] [frontend-tests.data.persistence-test] @@ -171,6 +172,7 @@ 'frontend-tests.data.dashboard-test 'frontend-tests.data.event-test 'frontend-tests.data.fix-deleted-fonts-test + 'frontend-tests.data.modal-test 'frontend-tests.data.nitrate-test 'frontend-tests.data.persistence-retry-test 'frontend-tests.data.persistence-test diff --git a/frontend/test/frontend_tests/ui/measures_menu_props_test.cljs b/frontend/test/frontend_tests/ui/measures_menu_props_test.cljs index 0f509d3a08..790b749aef 100644 --- a/frontend/test/frontend_tests/ui/measures_menu_props_test.cljs +++ b/frontend/test/frontend_tests/ui/measures_menu_props_test.cljs @@ -55,4 +55,9 @@ ;; force a re-render so the X/Y inputs enabled/disabled state updates. (t/is (false? (check-measures-menu-props (props (assoc base-values :layout-item-absolute false)) - (props (assoc base-values :layout-item-absolute true))))))) + (props (assoc base-values :layout-item-absolute true)))))) + + (t/testing "re-renders when clip content changes but nothing else does" + (t/is (false? (check-measures-menu-props + (props (assoc base-values :show-content false)) + (props (assoc base-values :show-content true))))))) diff --git a/frontend/translations/ca.po b/frontend/translations/ca.po index 2c3702dc0c..900e2048d1 100644 --- a/frontend/translations/ca.po +++ b/frontend/translations/ca.po @@ -1238,7 +1238,7 @@ msgstr "S'ha mogut el projecte" #: src/app/main/data/common.cljs:266 #, backend msgid "dashboard.team-belong-organization" -msgstr "Aquest equip ara forma part de l'organització %s" +msgstr "%s ara forma part de %s" #: src/app/main/ui/dashboard/team.cljs:1669 msgid "dashboard.team-info" @@ -1251,7 +1251,7 @@ msgstr "Membres de l'equip" #: src/app/main/data/common.cljs:267 #, backend msgid "dashboard.team-no-longer-belong-organization" -msgstr "Aquest equip ja no forma part de l'organització %s" +msgstr "%s ja no forma part de %s" #: src/app/main/ui/dashboard/team.cljs:1676 msgid "dashboard.team-organization" diff --git a/frontend/translations/en.po b/frontend/translations/en.po index 68e9f34c95..f5cfa817f4 100644 --- a/frontend/translations/en.po +++ b/frontend/translations/en.po @@ -1222,7 +1222,7 @@ msgstr "Your project has been moved successfully" #: src/app/main/data/common.cljs:266 #, backend msgid "dashboard.team-belong-organization" -msgstr "This team is now part of the organization %s" +msgstr "%s now belongs to %s" #: src/app/main/ui/dashboard/team.cljs:1669 msgid "dashboard.team-info" @@ -1235,7 +1235,7 @@ msgstr "Team members" #: src/app/main/data/common.cljs:267 #, backend msgid "dashboard.team-no-longer-belong-organization" -msgstr "This team is no longer part of the organization %s" +msgstr "%s is no longer part of %s" #: src/app/main/ui/dashboard/team.cljs:1676 msgid "dashboard.team-organization" @@ -1504,7 +1504,7 @@ msgstr "There was an error with the comment" msgid "errors.connection-error" msgstr "Cannot reach the server. Check your connection and try again." -#: src/app/main/errors.cljs:851 +#: src/app/main/errors.cljs:414 msgid "errors.deprecated" msgstr "" "Sorry! This is an old file that uses a deprecated type of Penpot assets and " @@ -1730,8 +1730,16 @@ msgstr "The registration is currently disabled." #: src/app/main/errors.cljs:538 msgid "errors.save-failed" msgstr "" -"Autosave is not working due to an error. Contact support to report the " -"error and reload to continue from your last saved version." +"Autosave has stopped for this session, so your changes since the last successful save cannot be saved. Download the report to help us fix it, then reload the page, or leave this file and open it again, to keep editing." + +#: src/app/main/data/persistence.cljs:310 +msgid "errors.save-retrying" +msgstr "Connection lost. Retrying to save your changes." + +#: src/app/main/errors.cljs +msgid "errors.save-still-retrying" +msgstr "" +"Your changes are not saved yet because Penpot cannot reach the server. They are kept, and Penpot will keep trying to save them. Keep this tab open until they are saved." #: src/app/main/data/persistence.cljs:307 msgid "errors.save-retrying" @@ -5806,7 +5814,7 @@ msgstr "" #: src/app/main/ui/settings/subscription.cljs:714 msgid "subscription.settings.member-since" -msgstr "Penpot member since: %s" +msgstr "Penpot member since: %s." #: src/app/main/ui/settings/subscription.cljs:736, src/app/main/ui/settings/subscription.cljs:751, src/app/main/ui/settings/subscription.cljs:767, src/app/main/ui/settings/subscription.cljs:787 msgid "subscription.settings.more-information" @@ -5876,6 +5884,9 @@ msgstr "Subscribe" msgid "subscription.settings.subscribed-since" msgstr "You've been on the %s plan since %s." +msgid "subscription.settings.trial-ends-on" +msgstr "Your free trial ends on %s." + #: src/app/main/ui/settings/subscription.cljs:399 msgid "subscription.settings.success.dialog.description" msgstr "" @@ -6645,7 +6656,7 @@ msgstr "Reset" msgid "workspace.header.retrying" msgstr "Retrying to save" -#: src/app/main/ui/workspace/left_header.cljs:126 +#: src/app/main/ui/workspace/left_header.cljs:123 msgid "workspace.header.save-error" msgstr "Error on saving" @@ -7731,6 +7742,10 @@ msgstr "More library colors" msgid "workspace.options.more-token-colors" msgstr "More color tokens" +#: src/app/main/ui/workspace/sidebar/options/menus/typography.cljs:502 +msgid "workspace.options.no-font-results" +msgstr "No matching fonts" + #: src/app/main/ui/workspace/sidebar/options/drawing/frame.cljs:117, src/app/main/ui/workspace/sidebar/options/menus/measures.cljs:518 msgid "workspace.options.no-size-preset-results" msgstr "No matching size preset" diff --git a/frontend/translations/es.po b/frontend/translations/es.po index 103b2725b3..4ef52e1afe 100644 --- a/frontend/translations/es.po +++ b/frontend/translations/es.po @@ -1236,7 +1236,7 @@ msgstr "Tu proyecto ha sido movido con éxito" #: src/app/main/data/common.cljs:266 #, backend msgid "dashboard.team-belong-organization" -msgstr "Este equipo ahora es parte de la organización %s" +msgstr "%s ahora pertenece a %s" #: src/app/main/ui/dashboard/team.cljs:1669 msgid "dashboard.team-info" @@ -1249,7 +1249,7 @@ msgstr "Integrantes del equipo" #: src/app/main/data/common.cljs:267 #, backend msgid "dashboard.team-no-longer-belong-organization" -msgstr "Este equipo ya no es parte de la organización %s" +msgstr "%s ya no forma parte de %s" #: src/app/main/ui/dashboard/team.cljs:1676 msgid "dashboard.team-organization" @@ -1756,9 +1756,12 @@ msgstr "El registro está actualmente desactivado." #: src/app/main/errors.cljs:538 msgid "errors.save-failed" msgstr "" -"El autoguardado se ha detenido debido a un error. Contacta soporte para " -"informar del error y recarga para continuar desde los últimos cambios " -"guardados." +"El autoguardado se ha detenido en esta sesión, así que los cambios desde el último guardado correcto no se podrán guardar. Descarga el informe para ayudarnos a arreglarlo y recarga la página, o sal de este fichero y vuelve a abrirlo, para seguir editando." + +#: src/app/main/errors.cljs +msgid "errors.save-still-retrying" +msgstr "" +"Tus cambios aún no se han guardado porque Penpot no puede conectar con el servidor. Se conservan y Penpot seguirá intentando guardarlos. Mantén esta pestaña abierta hasta que se guarden." #: src/app/main/data/persistence.cljs:307 msgid "errors.save-retrying" @@ -5890,7 +5893,7 @@ msgstr "" #: src/app/main/ui/settings/subscription.cljs:714 msgid "subscription.settings.member-since" -msgstr "Miembro de penpot desde: %s" +msgstr "Miembro de penpot desde: %s." #: src/app/main/ui/settings/subscription.cljs:736, src/app/main/ui/settings/subscription.cljs:751, src/app/main/ui/settings/subscription.cljs:767, src/app/main/ui/settings/subscription.cljs:787 msgid "subscription.settings.more-information" @@ -5962,6 +5965,9 @@ msgstr "Suscríbete" msgid "subscription.settings.subscribed-since" msgstr "Llevas en el plan %s desde %s." +msgid "subscription.settings.trial-ends-on" +msgstr "Tu prueba gratuita acaba el %s." + #: src/app/main/ui/settings/subscription.cljs:399 msgid "subscription.settings.success.dialog.description" msgstr "" @@ -7823,6 +7829,10 @@ msgstr "Más colores de la biblioteca" msgid "workspace.options.more-token-colors" msgstr "Más tokens de color" +#: src/app/main/ui/workspace/sidebar/options/menus/typography.cljs:502 +msgid "workspace.options.no-font-results" +msgstr "No hay fuentes coincidentes" + #: src/app/main/ui/workspace/sidebar/options/drawing/frame.cljs:117, src/app/main/ui/workspace/sidebar/options/menus/measures.cljs:518 msgid "workspace.options.no-size-preset-results" msgstr "No hay tamaños predefinidos coincidentes" diff --git a/frontend/translations/fa.po b/frontend/translations/fa.po index 2acda65b41..bf01868ef6 100644 --- a/frontend/translations/fa.po +++ b/frontend/translations/fa.po @@ -1220,7 +1220,7 @@ msgstr "پروژه شما با موفقیت منتقل شد" #: src/app/main/data/common.cljs:266 #, backend msgid "dashboard.team-belong-organization" -msgstr "این تیم اکنون به %s تعلق دارد" +msgstr "%s اکنون به %s تعلق دارد" #: src/app/main/ui/dashboard/team.cljs:1669 msgid "dashboard.team-info" @@ -1233,7 +1233,7 @@ msgstr "اعضای تیم" #: src/app/main/data/common.cljs:267 #, backend msgid "dashboard.team-no-longer-belong-organization" -msgstr "این تیم دیگر به سازمان %s تعلق ندارد" +msgstr "%s دیگر به %s تعلق ندارد" #: src/app/main/ui/dashboard/team.cljs:1676 msgid "dashboard.team-organization" diff --git a/frontend/translations/fr.po b/frontend/translations/fr.po index 507dfc85bb..5686f05331 100644 --- a/frontend/translations/fr.po +++ b/frontend/translations/fr.po @@ -1248,7 +1248,7 @@ msgstr "Votre projet a bien été déplacé" #: src/app/main/data/common.cljs:266 #, backend msgid "dashboard.team-belong-organization" -msgstr "Cette équipe appartient désormais à %s" +msgstr "%s appartient désormais à %s" #: src/app/main/ui/dashboard/team.cljs:1669 msgid "dashboard.team-info" @@ -1261,7 +1261,7 @@ msgstr "Membres de l’équipe" #: src/app/main/data/common.cljs:267 #, backend msgid "dashboard.team-no-longer-belong-organization" -msgstr "Cette équipe n'appartient plus à l'organisation %s" +msgstr "%s n'appartient plus à %s" #: src/app/main/ui/dashboard/team.cljs:1676 msgid "dashboard.team-organization" diff --git a/frontend/translations/fr_CA.po b/frontend/translations/fr_CA.po index 65470f996a..79fb1a0323 100644 --- a/frontend/translations/fr_CA.po +++ b/frontend/translations/fr_CA.po @@ -1122,7 +1122,7 @@ msgstr "Le projet a été déplacé avec succès" #: src/app/main/data/common.cljs:266 #, backend msgid "dashboard.team-belong-organization" -msgstr "Cette équipe appartient maintenant à l'organisation %s" +msgstr "%s appartient maintenant à %s" #: src/app/main/ui/dashboard/team.cljs:1669 msgid "dashboard.team-info" @@ -1135,7 +1135,7 @@ msgstr "Membres de l'équipe" #: src/app/main/data/common.cljs:267 #, backend msgid "dashboard.team-no-longer-belong-organization" -msgstr "Cette équipe n'appartient plus à l'organisation %s" +msgstr "%s n'appartient plus à %s" #: src/app/main/ui/dashboard/team.cljs:1676 msgid "dashboard.team-organization" diff --git a/frontend/translations/he.po b/frontend/translations/he.po index 468980a6f6..0e9eabca97 100644 --- a/frontend/translations/he.po +++ b/frontend/translations/he.po @@ -1093,7 +1093,7 @@ msgstr "המיזם שלך הועבר בהצלחה" #: src/app/main/data/common.cljs:266 #, backend msgid "dashboard.team-belong-organization" -msgstr "הצוות הזה שייך מעתה ל־%s" +msgstr "%s שייך מעתה ל־%s" #: src/app/main/ui/dashboard/team.cljs:1669 msgid "dashboard.team-info" @@ -1106,7 +1106,7 @@ msgstr "חברי הצוות" #: src/app/main/data/common.cljs:267 #, backend msgid "dashboard.team-no-longer-belong-organization" -msgstr "הצוות הזה לא שייך יותר לארגון %s" +msgstr "%s לא שייך יותר ל־%s" #: src/app/main/ui/dashboard/team.cljs:1676 msgid "dashboard.team-organization" diff --git a/frontend/translations/it.po b/frontend/translations/it.po index 992dc61a58..3512c5d007 100644 --- a/frontend/translations/it.po +++ b/frontend/translations/it.po @@ -1103,7 +1103,7 @@ msgstr "Il tuo progetto è stato spostato con successo" #: src/app/main/data/common.cljs:266 #, backend msgid "dashboard.team-belong-organization" -msgstr "Questo team ora appartiene a %s" +msgstr "%s ora appartiene a %s" #: src/app/main/ui/dashboard/team.cljs:1669 msgid "dashboard.team-info" @@ -1116,7 +1116,7 @@ msgstr "Membri del team" #: src/app/main/data/common.cljs:267 #, backend msgid "dashboard.team-no-longer-belong-organization" -msgstr "Questo team non appartiene più all'organizzazione %s" +msgstr "%s non appartiene più a %s" #: src/app/main/ui/dashboard/team.cljs:1676 msgid "dashboard.team-organization" diff --git a/frontend/translations/jpn_JP.po b/frontend/translations/jpn_JP.po index 1d633bb75d..7b0a3d78b9 100644 --- a/frontend/translations/jpn_JP.po +++ b/frontend/translations/jpn_JP.po @@ -1202,7 +1202,7 @@ msgstr "プロジェクトを移動しました" #: src/app/main/data/common.cljs:266 #, backend msgid "dashboard.team-belong-organization" -msgstr "このチームは組織「%s」に所属しました" +msgstr "%s は組織「%s」に所属しました" #: src/app/main/ui/dashboard/team.cljs:1669 msgid "dashboard.team-info" @@ -1215,7 +1215,7 @@ msgstr "チームのメンバー" #: src/app/main/data/common.cljs:267 #, backend msgid "dashboard.team-no-longer-belong-organization" -msgstr "このチームは組織「%s」に所属しなくなりました" +msgstr "%s は組織「%s」に所属しなくなりました" #: src/app/main/ui/dashboard/team.cljs:1676 msgid "dashboard.team-organization" diff --git a/frontend/translations/nl.po b/frontend/translations/nl.po index 1de2d611ec..3d55474017 100644 --- a/frontend/translations/nl.po +++ b/frontend/translations/nl.po @@ -1230,7 +1230,7 @@ msgstr "Je project is succesvol verplaatst" #: src/app/main/data/common.cljs:266 #, backend msgid "dashboard.team-belong-organization" -msgstr "Dit team behoort nu tot %s" +msgstr "%s behoort nu tot %s" #: src/app/main/ui/dashboard/team.cljs:1669 msgid "dashboard.team-info" @@ -1243,7 +1243,7 @@ msgstr "Teamleden" #: src/app/main/data/common.cljs:267 #, backend msgid "dashboard.team-no-longer-belong-organization" -msgstr "Dit team behoort niet langer tot de organisatie %s" +msgstr "%s behoort niet langer tot %s" #: src/app/main/ui/dashboard/team.cljs:1676 msgid "dashboard.team-organization" diff --git a/frontend/translations/sv.po b/frontend/translations/sv.po index b74e6d85a0..293b654895 100644 --- a/frontend/translations/sv.po +++ b/frontend/translations/sv.po @@ -1221,7 +1221,7 @@ msgstr "Ditt projekt har flyttats" #: src/app/main/data/common.cljs:266 #, backend msgid "dashboard.team-belong-organization" -msgstr "Detta team tillhör nu %s" +msgstr "%s tillhör nu %s" #: src/app/main/ui/dashboard/team.cljs:1669 msgid "dashboard.team-info" @@ -1234,7 +1234,7 @@ msgstr "Teammedlemmar" #: src/app/main/data/common.cljs:267 #, backend msgid "dashboard.team-no-longer-belong-organization" -msgstr "Detta team tillhör inte längre organisation %s" +msgstr "%s tillhör inte längre %s" #: src/app/main/ui/dashboard/team.cljs:1676 msgid "dashboard.team-organization" diff --git a/frontend/translations/tr.po b/frontend/translations/tr.po index 82cd5fe9a6..adbc51754c 100644 --- a/frontend/translations/tr.po +++ b/frontend/translations/tr.po @@ -1226,7 +1226,7 @@ msgstr "Projen başarıyla taşındı" #: src/app/main/data/common.cljs:266 #, backend msgid "dashboard.team-belong-organization" -msgstr "Bu takım artık %s organizasyonuna ait" +msgstr "%s artık %s organizasyonuna ait" #: src/app/main/ui/dashboard/team.cljs:1669 msgid "dashboard.team-info" @@ -1239,7 +1239,7 @@ msgstr "Takım üyeleri" #: src/app/main/data/common.cljs:267 #, backend msgid "dashboard.team-no-longer-belong-organization" -msgstr "Bu takım artık %s organizasyonuna ait değil" +msgstr "%s artık %s organizasyonuna ait değil" #: src/app/main/ui/dashboard/team.cljs:1676 msgid "dashboard.team-organization" diff --git a/frontend/translations/zh_Hant.po b/frontend/translations/zh_Hant.po index 94bb9de6fa..ebb8577137 100644 --- a/frontend/translations/zh_Hant.po +++ b/frontend/translations/zh_Hant.po @@ -1163,7 +1163,7 @@ msgstr "已成功移動您的專案" #: src/app/main/data/common.cljs:266 #, backend msgid "dashboard.team-belong-organization" -msgstr "此團隊現已隸屬於組織 %s" +msgstr "%s 現已隸屬於 %s" #: src/app/main/ui/dashboard/team.cljs:1669 msgid "dashboard.team-info" @@ -1176,7 +1176,7 @@ msgstr "團隊成員" #: src/app/main/data/common.cljs:267 #, backend msgid "dashboard.team-no-longer-belong-organization" -msgstr "此團隊已不再隸屬於組織 %s" +msgstr "%s 已不再隸屬於 %s" #: src/app/main/ui/dashboard/team.cljs:1676 msgid "dashboard.team-organization"