github/penpot
1
0
Fork 0
mirror of https://github.com/penpot/penpot.git synced 2026-10-28 21:40:34 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/staging' into develop

Browse Source
This commit is contained in:
Andrey Antukh 2026-10-19 22:18:27 +02:00
commit fffc3ee222
115 changed files with 5570 additions and 1655 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
.agents/README.md
View File

@ -410,6 +410,10 @@ append-only `Review Log` (UTC ISO 8601, one line per entry):
A plan approved with no changes goes `draft` → `done` directly.
A plan is never staged or committed: the `done` status and its
`Review Log` are session bookkeeping, not part of the change. Their
absence from the commit is expected.
## 7. Connecting `gh` CLI with a token
The `create-issue` and `create-pr` flows need an authenticated `gh`

10
.gitignore vendored
View File

@ -16,7 +16,10 @@
.repl
opencode.json
.opencode/package-lock.json
/*.html
/*.jpg
/*.json
!/package.json
/*.md
!CHANGES.md
!CONTRIBUTING.md
@ -92,6 +95,12 @@ opencode.json
/playwright/.cache/
/render-wasm/target/
/render-wasm/.render-wasm-build.lock
# render-wasm build artifacts copied into the exporter (autogenerated)
/exporter/resources/wasm/render-wasm.js
/exporter/resources/wasm/render-wasm.wasm
/exporter/src/app/wasm/shared.js
/media-processor/dist/
/media-processor/target/
/**/node_modules
@ -114,5 +123,6 @@ opencode.json
/.ci-logs
/tmp/
/.codex/
/scripts/__pycache__/
/tools/__pycache__
/scripts/__pycache__

2
.serena/memories/backend/auth-permissions-product-domains.md
View File

@ -13,7 +13,7 @@
- Logout may return an OIDC provider redirect URI when the session claims include provider/session data and the provider has a logout URI.
- Invitation tokens are verified through token issuers and only accepted when the token member id/email matches the authenticated profile; otherwise login proceeds without consuming the invitation.
- When public registration is disabled, invitation-based password registration also requires a matching, non-expired `team_invitation` row; registration revalidates it under a row lock before creating the profile.
- HTTP/session parsing details such as cookie/header precedence and SameSite behavior are in `mem:backend/subtleties`.
- HTTP/session parsing details such as cookie/header precedence, JWT session token versions, and SameSite behavior are in `mem:backend/subtleties`.
## Permission model

2
.serena/memories/backend/core.md
View File

@ -4,7 +4,7 @@ Backend: JVM Clojure; Integrant; PostgreSQL; Redis/Valkey; RPC; HTTP; storage; m
## Focused memories
- Cross-cutting backend subtleties (RPC, DB, workers, cron, CLIMIT, HTTP/sessions, storage, file data): `mem:backend/subtleties`
- Cross-cutting backend subtleties (RPC, DB helpers, workers, cron, CLIMIT, config and HTTP/session middleware, http self-metrics, media processing, and file data persistence): `mem:backend/subtleties`
- Storage abstraction, logical buckets, object lifecycle, deduplication, access, and garbage collection: `mem:backend/storage`.
- Embedded Ladybug graph experiment, projection, incremental sync, console, and risks: `mem:backend/graph-experiment`
- Session lifetime config, token `:exp`, and idle/absolute GC: `mem:backend/session-expiration`.

4
.serena/memories/backend/storage.md
View File

@ -25,7 +25,7 @@
- Existing object values can remain readable until physical deletion.
- `:expired-at` blocks reads after the expiration time.
- `del-object!` sets `deleted_at` on live rows only (`deleted_at IS NULL`): a repeated call returns `false`. It does not remove backend content.
- `storage-gc-deleted` removes the database row and backend content after the deletion delay, streaming the pending rows with `db/plan` inside a read-only transaction and deleting them in chunks of 25, each chunk in its own transaction: a chunk that fails (for example, an object a foreign key still references) is logged and skipped instead of aborting the run. A physical-deletion failure inside a chunk defers its rows one day (`deletion_attempts + 1`) and gives up after 7 attempts, accepting the orphan blob. Deferred rows keep their references until the final give-up. The freeze path re-verifies references under row locks, so a concurrent purge can delay a freeze but never swallow a touch.
- `storage-gc-deleted` removes the database row and backend content after the deletion delay, processing the pending objects in chunks of 25 with keyset pagination (`(deleted_at, id)` cursor, matching the `storage_object(deleted_at, id)` index). Only `status = 'valid'` objects are read: pending ones (still being written, see the status lifecycle) stay untouched even when already marked as deleted. Each chunk is one short read-write transaction: `SELECT ... FOR UPDATE SKIP LOCKED` (so two concurrent runs never take the same chunk; no separate read transaction exists and the run does not block the table vacuum) followed by the database delete and the backend delete grouped by backend. The cursor advances to the `(deleted_at, id)` of the last row read regardless of the deletion result, so a chunk that fails is skipped and not visited again in the same run: the error aborts the transaction (nothing of the chunk is deleted, locks are released), the handler catches it inside the transaction function and rolls it back explicitly — the commit that `tx-run!` performs afterwards is a JDBC-driver no-op on the already-rolled-back connection — and the loop continues with the returned rows. Accepted quirks, both benign and self-healing: (1) with rows of several backends in one chunk, a failure after one group's physical delete leaves those rows pending with already-removed content — pending rows are excluded from reads and the next run finishes them, since the physical delete is idempotent (S3 ignores missing keys, FS uses `deleteIfExists`); (2) rows locked by a concurrent run are skipped via `SKIP LOCKED` and deferred to the next scheduled run, and the `deleted_at <= now` cutoff is sampled per chunk. It emits the `penpot_storage_gc_deleted_objects_total{backend}` counter (one increment per deleted backend group, reported only after the chunk commits, so rolled-back chunks are never counted twice); run duration is covered by the generic `penpot_tasks_timing{name="storage-gc-deleted"}` histogram and chunk failures by the log-derived `penpot_log_events_total`. Every 5000 deleted objects the run pauses 5 seconds (throttle between chunk transactions, never holding one open). It runs as the `storage-gc-deleted` job (`::job-def`) and beats `jobs/heartbeat` after each committed chunk, so a long sweep keeps its lease and never holds it during a chunk transaction.
- `storage-gc-touched` finds references before it sets `deleted_at`.
- Orphan watch: `SELECT COALESCE(metadata->>'bucket', metadata->>'~:bucket') AS bucket, count(*) FROM storage_object so WHERE touched_at IS NULL AND deleted_at IS NULL AND status = 'valid' AND NOT EXISTS (SELECT 1 FROM upload_session_chunk usc WHERE usc.object_id = so.id) GROUP BY 1;` Only the `upload-session` row of that output is a leak signal (its rows always have a mapping or a pending touch/deleted mark); other buckets use domain-table references, not this mapping.
- `objects-gc` removes deleted domain rows and touches their storage object IDs. Each proc streams its pending keys with `db/plan` inside a read-only transaction and deletes them in chunks, each chunk in its own transaction with `FOR UPDATE SKIP LOCKED`: a chunk that fails is logged and skipped instead of aborting the run.
@ -35,7 +35,7 @@
- `sto/touch-object!` is one `UPDATE storage_object SET touched_at`: DB-only, never touches S3/FS backend content.
- Physical FK cascades are avoided by convention; domain deletion is logical (`deleted_at`) everywhere. GC removes small objects first, then parents, then grandparents (see the call order in `app.tasks.objects-gc/execute-objects-gc`). Per-row delete cost is small and bounded.
- `objects-gc` sweeps in chunks of 100 (`::chunk-size`, overridable per job params); `storage-gc-deleted` in chunks of 25. The cursor that streams the pending keys runs in a read-only transaction of its own (`db/plan`), and every chunk is deleted in a transaction of its own: the cursor fetch and the deletes are never the same transaction.
- `objects-gc` sweeps in chunks of 100 (`::chunk-size`, overridable per job params); `storage-gc-deleted` in chunks of 25. `objects-gc` (and the other `db/plan` sweeps) stream the pending keys in a read-only transaction of their own and delete every chunk in a transaction of its own: the cursor fetch and the deletes are never the same transaction. `storage-gc-deleted` has no separate cursor transaction: it reads, locks and deletes each chunk in the same short transaction (see the keyset pagination above).
- Consequence: per-chunk cost is DB-only and bounded, so these sweeps cannot outrun the 30-minute jobs lease in practice (would need millions of pending rows in one proc). Per-chunk `heartbeat` where present is belt-and-braces, not load-bearing.
## Connection Reuse Details

11
.serena/memories/backend/subtleties.md
View File

@ -80,6 +80,7 @@
## Storage and media
- The `media-storage-bytes-per-team` quota check on `upload-file-media-object` is TEMPORARILY DISABLED (2026-10-19, kept commented in `app.rpc.commands.media` with its `sql:get-team-id-for-file` lookup still active for the audit props): the quote performs a full `SUM` over `storage_object` per upload and was too slow in production. The `upload-sessions-per-profile` quota is unaffected and stays active. Re-enable only with a cheaper approximation.
- Storage abstraction, backend configuration, logical buckets, object lifecycle, deduplication, access rules, and garbage collection: `mem:backend/storage`.
- SVG validation strips DOCTYPE and uses secure SAX parsing. Basic SVG info falls back to 100x100 dimensions when width/height/viewBox are missing.
- Raster metadata is shell-derived with ImageMagick `identify`, verifies detected MIME against the supplied MIME, and swaps dimensions for EXIF orientations 6/8.
@ -91,4 +92,12 @@
- File data backends are `legacy-db`, `db`, and `storage`. The storage backend keeps encoded file data in storage bucket `file-data`; the DB row stores metadata with `storage-ref-id` and nil data.
- `fdata/upsert!` touches any storage object referenced by incoming metadata before storing the new row/blob.
- Pointer-map fragments are persisted separately as type `fragment`, and only modified pointer maps are written.
- `fdata/realize` combines pointer realization and object-map realization. Use it before operations that need complete in-memory file data instead of pointer placeholders.
- `fdata/realize` combines pointer realization and object-map realization. Use it before operations that need complete in-memory file data instead of pointer placeholders.
- A `file_data` row is one of three classes, told apart by `type`: `main` is the file's current state and carries `id = file_id`; `snapshot` is a point-in-time copy whose `id` is the one its `file_change` row shares; `fragment` is a piece of the pointer-map, referenced from the file's data.
- Their lifecycles differ: `main` and `snapshot` live with the file (dated when it is deleted, undated when it is restored), while a `fragment` carries no date and is collected by `file_gc` once the file stops referencing it. Neither is a session buffer, so do not read a `fragment` as an ephemeral revision.
- File snapshots (versions) keep their metadata row in `file_change` and their payload in `file_data` (`type = 'snapshot'`). With the `db` backend `file_change.data` stays NULL: that column is only written by the `legacy-db` backend, plus the rows written before the 2025-07 file storage refactor.
- Tell a snapshot apart by `label IS NOT NULL` (with `created_by` / `deleted_at` as needed), never by `file_change.data IS NOT NULL`: the label holds with every file data backend, and the payload column stopped meaning "this is a snapshot" when the payload moved out. Queries that identify versions must use the label.
- The version quota counts of `app.rpc.quotes` (`sql:get-snapshots-per-file`, `sql:get-snapshots-per-team`) are served by the partial index `file_change__user_snapshots__idx`, whose predicate mirrors their where clause; changing one without the other drops the index only scan and the count walks the whole table again.
- Revisions live only in `file_change` and have no `file_data` counterpart. They are born already scheduled for deletion one hour later (`app.rpc.commands.files-update`) and the snapshot paths cannot reach one, because they resolve their row through a `label IS NOT NULL` filter.
- A deletion date is assigned once and never moved: `restore-files` (`app.rpc.commands.files`) unmarks only durable history, the `:file` method of `app.tasks.delete-object` dates only rows that have no date yet, and `objects-gc` collects only rows that carry one. Unmarking a revision leaves it invisible to `objects-gc` for ever, because nothing sets that date again.
- Binfile v1/v3 import and the v3 export `get-file` step run inside one long transaction. They raise `idle_in_transaction_session_timeout` to 30 min (`app.binfile.common/transaction-idle-timeout-ms`) instead of disabling it, so a connection left idle while the ZIP is decoded or the file data is realized is bounded; `statement_timeout` stays at 300 s.

4
.serena/memories/clojure/design-rules.md
View File

@ -2,10 +2,10 @@
How to shape a function in this codebase. Each rule came from a review round; do not re-derive from taste.
- **Preconditions belong to the boundary, not to the core.** A function documents what it needs (hints, docstring) and assumes it. Absence checks live where the value enters the system: the caller that resolves optional objects, `ig/init-key`, the request handler. A `(when (and (some? a) (some? b)) ...)` inside a leaf function means the guard is in the wrong place — if you cannot get `a` or `b`, you should not be calling it.
- **Optional-by-design data is guarded where the optionality is born.** Example: `ConnectorStatistics` is nil when the server option is off, so the guard is `when-let [cs (some-> server ...)]` at the entry point and the consumer assumes `cs`.
- **Preconditions belong to the boundary, not to the core.** A function documents what it needs (hints, docstring) and assumes it. Absence checks live where the value enters the system: the caller that resolves optional objects, `ig/init-key`, the request handler — including `nil` that is optional-by-design (`when-let [cs (some-> server ...)]` at the entry point, as with `ConnectorStatistics` when the server option is off). A `(when (and (some? a) (some? b)) ...)` inside a leaf function means the guard is in the wrong place — if you cannot get `a` or `b`, you should not be calling it.
- **A static set of operations is written statically.** With a fixed, small set (four metrics) write the four calls. Do not build a collection and iterate it (`doseq` over a literal vector): it allocates, adds indirection and hides each operation. Use a collection when the set is dynamic (config, registry, input).
- **Default to no `!`; existing `!` is legacy being removed gradually.** Do not introduce new `!` names except the allowed cases documented in `mem:clojure/idioms` (add and justify the special case there first). Allowed: mutating public fns in `app.db` (`insert!`, `update!`, `delete!`, `run!`, `tx-run!`, ...) and fns whose contract is to work over transients (stdlib `conj!`/`assoc!`/`dissoc!`/`persistent!` and custom helpers over them). Existing `!` elsewhere (e.g. `emit!`, `send!`, `submit!`, `validate-file!`) stays until its refactor; keep a family consistent in the meantime. Reporting helpers (log, sample-and-publish) never take `!`: the mutation happens in the `!` API they call.
- **Functions that obtain data start with `get-` or `fetch-`.** A reader must see at a glance that the function reads (`get-file`, `get-snapshot`, `get-file-json`); a bare noun hides it (`file-json` says nothing about reading). New helpers that fetch by id follow this, never a `current-` prefix: the id parameter already says which one it is.
- **`?` marks a predicate, not a data prop.** A var or function that answers a question takes `?` (`export-error?`, `(defn editable? …)`); a map key that names a property does not (`:export-error`, never `:export-error?`), even when its value is a boolean. Existing `?` keys are legacy to remove gradually; do not add a new one.
- **Do not shape production code for tests.** No return values added just to assert them; assert on the observable effect. When preconditions move, move the tests with them.

2
.serena/memories/critical-info.md
View File

@ -7,7 +7,7 @@ You are working on the GitHub project `penpot/penpot`, a monorepo.
- Edits/stale refs/duplication cleanup: `mem:memory-maintenance`.
- Cross-cutting testing principles, TDD workflow, and anti-patterns: `mem:testing`.
- Verified Clojure language behaviors that contradict common assumptions (e.g. `int?` covers `Long`; `integer?` is the general predicate): `mem:clojure/idioms` — read before assuming stdlib predicate semantics.
- When designing a solution or an API, read `mem:clojure/design-rules` (function shape, boundary checks, naming).
- Read `mem:clojure/design-rules` (function shape, boundary checks, naming) before writing or modifying Clojure/ClojureScript code — not only when designing a solution or an API.
# Development workflow

1
.serena/memories/exporter/core.md
View File

@ -5,6 +5,7 @@
## Layout and commands
- Source: `exporter/src/`; config: `deps.edn`, `shadow-cljs.edn`, `package.json`; runtime helpers/assets: `vendor/`, `scripts/`.
- `exporter/resources/wasm/render-wasm.{js,wasm}` and `exporter/src/app/wasm/shared.js` are generated render-wasm build output, not source: they are gitignored. Never hand-edit or commit them; regenerate by rebuilding render-wasm.
- From `exporter/`: setup `./scripts/setup`; watch `pnpm run watch` or `pnpm run watch:app`; production build `pnpm run build`; test bundle `pnpm run build:test`; tests `pnpm run test` or `pnpm run test:quiet`; lint `pnpm run lint:clj`; format check/fix `pnpm run check-fmt:clj` / `pnpm run fmt:clj`.
- Because exporter consumes `common/`, shared file/shape/model changes may need exporter verification even when the immediate change is not under `exporter/`.
- Cross-cutting testing principles and anti-patterns: `mem:testing`.

5
.serena/memories/frontend/workspace-state-persistence-subtleties.md
View File

@ -26,12 +26,15 @@
- A terminal failure halts the queue on purpose: the edits stay queued so `wait-persisted-or-error` keeps rejecting, and `workspace/versions.cljs` relies on that to avoid snapshotting a file whose edits were never saved. `recover-persistence` lifts the halt when a file is opened again or the role changes. A transport failure still inside its retry window (`slow-retry-pending?`) is resent once (`resume-persistence true`), never released; any other halt emits `release-failed-file` (drops every queued commit of the file that failed) then `resume-persistence`. Opening a file also replays the queued edits of its version onto the loaded data (`pending-file-changes` → `restore-pending-changes` in `bundle-fetched`); a replay that throws keeps the server data and the queue. That is what unblocks the editor without a reload; because the queue is shared, dropping the failed file is also what lets other files' pending edits drain. `release-failed-file` runs at recovery time, never at failure time — clearing the error eagerly would make waiters resolve.
- The split between `initialize-persistence-state` and `recover-persistence` is deliberate: initialization deliberately does NOT lift a halt, so undeliverable edits are released when the user acts, not when the view mounts.
- **Every non-transport terminal save failure is permanent.** There is no "stale but valid" category: permission lost, file gone, and server-rejected changes all fail again on a resend. Losing edit permission is not temporary — the backend raises `:not-found` for it (`check-edition-permissions!` → `perms/make-check-fn`), and `:not-found` routes to the exception page. So `:save-permission-denied` (a client-side `can-edit` check) must never be resumed: replaying the save to discover that routes the user to the exception page. Release it instead.
- Save failures split transient vs terminal (`transient-error?`: the repo retryable types `:network`/`:offline`/`:bad-gateway`/`:service-unavailable`/`:gateway-error` plus `:invalid-save-response`; everything else is terminal). Terminal keeps the `:error` halt + `flash-persistence` path; transient enters a `:retrying` episode: the head commit stays queued and resends with backoff 2s/8s/20s (3 retries, then the `:error` path). A transient failure that outlasts the backoff arms `slow-retry-cycle`: one silent resend (`:recovering`) every `slow-retry-delay-ms` (30s) until a save lands, `retry-give-up-ms` (24h since `:failing-since`) passes, or the head commit collects `max-slow-failures` (3) attempts that failed after `slow-failure-ms` (90s) or more (`:slow-failures`, counted in `persistence-transient-failure` from the attempt's elapsed time): a gateway timeout may leave the backend applying the save, so a save that keeps failing slowly must not be replayed for a day, while fast failures (offline) never count; the cycle stops on `::resume-persistence`, `::initialize-persistence` and `finalize-workspace`. Leaving the workspace also emits `suspend-retries` (`:suspended` in `:persistence`, drops `:retry-token` so a pending backoff timer goes stale): no backoff, slow cycle or `online` resend runs outside the workspace; only `workspace-initialized` (not a role change) lifts it via `recover-persistence` (`opened-file?`), which also resends a `:retrying` episode whose backoff was cut short. Its toast is `errors.save-still-retrying` without a reload link; a landing save hides the failure toast (`errors/persistence-failed-tag`). Replaying a save is safe because changes are idempotent, so there is no unknown-outcome guard: a commit whose earlier request left `active-requests` is simply sent again under the same `:commit-id`; only a request still in flight blocks a resend (`:in-flight` stays silent). Tests hold the slow cycle by stubbing `rx/timer` to never fire for `slow-retry-delay-ms`, or instant timers loop forever. Retry timers carry the episode token; a superseded token stays silent. Status stays `:retrying` through re-entries (`next-status` refuses `:pending`/`:saving` from it); waiters (`wait-persisted-or-error`) wait through it and reject only on `:error`.
- Save failures split transient vs terminal (`transient-error?`: the repo retryable types `:network`/`:offline`/`:bad-gateway`/`:service-unavailable`/`:gateway-error`, plus `:invalid-save-response`, plus save-only `busy-server-error?`: `:rate-limit` (429 from rlimit/climit — a save parked on another save's file lock holds the profile's single `:update-file/by-profile` permit, so the user's saves to other files 429 after 60 s) and backend-written 504 `:server-error` with `:statement-timeout`/`:idle-in-transaction-timeout` (Postgres cancels the lock wait at 300 s, before nginx; rolled back, so resend is safe). Kept out of `rp/retryable-types` on purpose so GETs do not retry them; everything else is terminal). Terminal keeps the `:error` halt + `flash-persistence` path; transient enters a `:retrying` episode: the head commit stays queued and resends with backoff 2s/8s/20s (3 retries, then the `:error` path). A transient failure that outlasts the backoff arms `slow-retry-cycle`: one silent resend (`:recovering`) every `slow-retry-delay-ms` (30s) until a save lands, `retry-give-up-ms` (24h since `:failing-since`) passes, or the head commit collects `max-slow-failures` (3) attempts that failed after `slow-failure-ms` (90s) or more (`:slow-failures`, counted in `persistence-transient-failure` from the attempt's elapsed time): a gateway timeout may leave the backend applying the save, so a save that keeps failing slowly must not be replayed for a day, while fast failures (offline) never count; the cycle stops on `::resume-persistence`, `::initialize-persistence` and `finalize-workspace`. Leaving the workspace also emits `suspend-retries` (`:suspended` in `:persistence`, drops `:retry-token` so a pending backoff timer goes stale): no backoff, slow cycle or `online` resend runs outside the workspace; only `workspace-initialized` (not a role change) lifts it via `recover-persistence` (`opened-file?`), which also resends a `:retrying` episode whose backoff was cut short. Its toast is `errors.save-still-retrying` without a reload link; a landing save hides the failure toast (`errors/persistence-failed-tag`). Replaying a save is safe because changes are idempotent, so there is no unknown-outcome guard: a commit whose earlier request left `active-requests` is simply sent again under the same `:commit-id`; only a request still in flight blocks a resend (`:in-flight` stays silent). Tests hold the slow cycle by stubbing `rx/timer` to never fire for `slow-retry-delay-ms`, or instant timers loop forever. Retry timers carry the episode token; a superseded token stays silent. Status stays `:retrying` through re-entries (`next-status` refuses `:pending`/`:saving` from it); waiters (`wait-persisted-or-error`) wait through it and reject only on `:error`.
- One reconnect notice per episode: sticky toast tagged `:persistence-reconnecting` (single-toast store, re-show replaces), hidden by tag on drain (`:saved`) and on terminal failure; recovery is silent. Header indicator has a `:retrying` state (`workspace.header.retrying`).
- Resume triggers: backoff timer and the browser `online` event (guarded by `exists? js/window`; re-enters the runner only for a live `:retrying` episode). New local edits during `:retrying` only join the queue: the episode keeps its `:run-id`, so `append-commit` does not re-enter, and the live runner (still waiting on the head's `commit-persisted`) sends them after the head. Re-entering on edits would bypass the backoff and spend an attempt per batched commit.
- Tests instant-trigger retries by stubbing `rx/timer` (recording delays to assert the schedule); dynamic bindings do not survive `await` continuations, so no dynamic var for delays.
- `save-request-timeout-ms` is 7 minutes and wraps the `update-file` request in `rx/timeout`, failing it as a transient `:network` error so the retry path resends instead of blocking the queue forever. It is deliberately longer than `saving-stall-timeout-ms` (5 min) so the stalled report fires before the retry.
- `sync-unload-guard` keeps `:persistence` in `st/ongoing-tasks` while edits are buffered or queued, which is what holds the browser leave-page warning on. It syncs on `::append-commit`, `::discard-commit` and `::release-failed-file` (every queue change). It is the one effect that must stay in `store.cljs`'s `omitset`, and `store.cljs`'s `ongoing-tasks` watcher is guarded by `exists? js/window` because tests run without a window.
- `leave-signals` (document `visibilitychange`→hidden, window `blur`) emits `::force-persist` when edits are buffered, so leaving the tab/window sends the 3 s batch at once; tests mock `dps/leave-signals` BEFORE `initialize-persistence` runs. pagehide/beforeunload are too late for an async save.
- Exports and the `.penpot` download call `force-persist-files-and-wait file-ids timeout` (waits only for those files, ends on `:error` or timeout, never raises) and then `unsaved-changes? state file-ids` to warn; it reads `refs/persistence`, so tests mock it with `(l/derived :persistence store)`.
- `persistence-failed` emits a `save-failed` audit event (`:cause-type :code :retrying :queued-commits`) whenever the user is notified (`notify-failure?`, same rule as the toast), including delegated auth/not-found failures.
- Undo transactions can stay open only temporarily; timed-out pending transactions are force-committed after about 20s. Undo entries are capped at 50.
- Undo/redo are ignored while a normal editor/drawing interaction is active, except grid-layout edition handles undo through this path.
- After local commits and when render-wasm is active, text shapes get derived `:position-data` recomputed in a separate commit tagged `#{:position-data}`; that tag is excluded from the position-data watcher to avoid loops.

1
.serena/memories/prod-infra/core.md
View File

@ -43,6 +43,7 @@ Production runs the deployed toolbox, not the devenv one:
- **Start from the ids, not from an inventory.** The ids come from the error or the logs. Unbounded queries over tables with tens of millions of rows (`storage_object`, `file_media_object`) do not finish in a reasonable time; bounded to an id list they are index lookups.
- **One query per reference column.** Two columns in one `IN (col_a, col_b)` lose the index and scan the whole table; split them.
- The read-only replica cuts long queries: see `mem:prod-infra/grafana`.
- A migration that builds an index runs inside the migration transaction, so it cannot use `CONCURRENTLY`, and a plain `CREATE INDEX` locks the table against writes for the whole build. On a large table (file saves write to `file_change` on every save) build the index out of band, before deploying, with `CREATE INDEX CONCURRENTLY` on its own (never inside a transaction block); the shipped migration uses `IF NOT EXISTS`, finds it and does nothing. A concurrent build interrupted midway leaves the index invalid: drop it and retry (check `pg_indexes` and `pg_index.indisvalid`).
## See also

10
.serena/memories/prod-infra/grafana.md
View File

@ -6,9 +6,14 @@ Keep this memory free of internal names: no host names, no deployed versions, no
## Datasources
- Metrics — **Prometheus**, uid `xkRmyFBGk`. PromQL. Metric families: `penpot_*` (RPC, task timings, CLIMIT, HTTP, websocket, storage, and the application log events of the backend), `hikaricp_*`, `jvm_*` and `process_*` on the backend (`job="app"`); `pg_*` from the Postgres exporter (`job="postgres"`); `redis_*` from the Redis exporter (`job="redis"`); `node_*` and `container_*` from node-exporter and cAdvisor. Backend histograms are in **milliseconds**.
- Metrics — **Prometheus**, uid `xkRmyFBGk`. PromQL. Metric families: `penpot_*` (RPC, task timings, CLIMIT, HTTP, websocket, storage, and the application log events of the backend), `hikaricp_*`, `jvm_*` and `process_*` on the backend (`job="app"`); `pg_*` from the Postgres exporter (`job="postgres"`); `redis_*` from the Redis exporter (`job="redis"`); `node_*` and `container_*` from node-exporter and cAdvisor. Backend histograms are in **milliseconds**. Tails (ms): requests and waits to 60000, background tasks to 300000, Redis eval to 10000; the sub-7500 buckets are identical everywhere, so old quantiles stay comparable.
- The Postgres exporter scrapes **the primary as well as the read-only replica**, so the primary's `pg_stat_activity`, `pg_settings_*` and `pg_stat_bgwriter_*` are in Prometheus even though the SQL datasource cannot reach them (the replica's views describe the replica, not the primary).
- Per-database data from the exporter is **aggregate only**: `pg_settings_*` (the whole server configuration, so autovacuum, checkpoint and memory parameters are readable from Prometheus), `pg_stat_database_*`, `pg_stat_bgwriter_*`, `pg_stat_activity_*`, and replication/lock counts. It publishes **no per-table series** (`pg_stat_user_tables_*` resolves empty), so dead tuples, per-table reloptions and relation sizes are SQL-only questions.
- It publishes **no `pg_stat_progress_*` collector** and `pg_stat_activity_count` carries no `backend_type` label, so a session is only identifiable by its remaining labels: autovacuum workers are the bucket with `usename` and `application_name` empty plus `state="active"`, and `pg_wait_events_connections{wait_event="AutoVacuumMain"}` is the autovacuum launcher alive (absent on the replica).
- A metric the exporter does not publish itself has to be added as a custom query block in its YAML queries file on the exporter host (that is where `pg_wait_events_connections` comes from), and in this build the file is read at startup: restart the exporter after editing. There is no per-metric flag to turn on.
- Logs — **Loki**, uid `h3CGKhEMk`. LogQL, and filter by the structured metadata `logger`/`level` (`{job="penpot"} | logger="..." | level="ERROR"`). Two things to remember when counting from it: the CLIMIT rejections have their own counters (`penpot_rpc_climit_rejected_total`, by `name` and `reason`) and their `hint="rejected"` line goes out at WARN only when something is left queued behind the call (a lone caller that times out is written at TRACE), so counting lines is not counting rejections -- the same goes for `hint="enqueued"` and `hint="acquired"`; and the only signal that exists nowhere else is the profile behind a queue (`logger="app.rpc.climit"`, whose `id` carries the key), chased there on demand.
- SQL — **Postgres**, uid `y8ea9tfMk`. Panels need `rawQuery: true`, `editorMode: "code"` and `format: "table"`. The `task` table is the only truth about the queue depth. It is a **read-only replica**: it rejects any write, `CREATE TEMP TABLE` included, so queries through it must be plain `SELECT`s. It also cuts long-running queries (the MCP path has a short timeout), so bound them (e.g. a list of ids); an empty answer there is not proof that there is no data.
- The Postgres server also ships its own log (`job="postgresql"`, primary only). Those streams carry just `filename`, `instance`, `job` and `level` — no `environment`/`hostname` — so selectors need `instance=~"$host.$env.*"`. `log_autovacuum_min_duration` is low enough that every autovacuum leaves an `automatic vacuum of table "db.schema.table"` line with pages, tuples, index scans and `elapsed`: a per-table history of **finished** vacuums (retrospective, not a view of what runs now). A day of vacuums is only a couple of hundred lines, so pulling them raw over `query_range` and parsing the table, the pages scanned, `tuples: N removed` and the `removable cutoff` XID age in the client works well — less fighting than making LogQL's `pattern`/`regexp` chew a multi-line message.
- SQL — **Postgres**, uid `y8ea9tfMk`. Panels need `rawQuery: true`, `editorMode: "code"` and `format: "table"`. The `task` table is the only truth about the queue depth. It is a **read-only replica**: it rejects any write, `CREATE TEMP TABLE` included, so queries through it must be plain `SELECT`s. Catalog data (`pg_class`, `pg_partition_tree`, `pg_stats`, and `reloptions`) describes the primary faithfully, because catalogs replicate; the statistics views (`pg_stat_user_tables`, `pg_stat_all_tables`) read as zero there, since statistics are per-instance and are not replicated. It also cuts long-running queries (the MCP path has a short timeout), so bound them (e.g. a list of ids); an empty answer there is not proof that there is no data.
- **Postgres (telemetry)**, uid `9YAUuM_Gz`.
- `$__rate_interval` and `$__interval` are substituted by Grafana; use them instead of fixed windows in graphs.
@ -16,6 +21,7 @@ Keep this memory free of internal names: no host names, no deployed versions, no
- `Production status`, uid `penpot-production-status`. Sections: general state and deploy, `RPC y CLIMIT`, PostgreSQL and pool, Redis/msgbus, tasks, JVM/process. Variables `$env` and `$host` (custom), no auto refresh. Every section opens collapsed except `RPC y CLIMIT`, which is the one read on a normal day.
- Per-host investigation dashboards are separate, and the alert rules deeplink into them.
- The PostgreSQL section ends with a block of vacuum panels: two stats (autovacuum workers active on the primary, and the launcher alive), a Loki timeseries of finished vacuums per interval and table, and then the `Vacuum TXID age` table, which is the last panel of the section. The worker stat is an approximation — it counts the `pg_stat_activity` bucket described above, because the exporter publishes no `backend_type` — and the Loki panel is retrospective by construction. A per-table, in-flight view (table, phase, progress) needs a `pg_stat_progress_vacuum` collector added to the exporter itself.
## Conventions

1
.serena/memories/render-wasm/core.md
View File

@ -25,6 +25,7 @@
- The lock covers setup, Cargo build, and artifact copy. The watch process releases it while waiting for changes. The operating system releases it on success, error, or signal; the file remains.
- The lock only coordinates repository scripts. Manual Cargo commands do not take it.
- Set `RENDER_WASM_LOCK_FILE` to the same path only when separate checkouts intentionally share a `CARGO_TARGET_DIR`.
- Build output is gitignored and stays untracked everywhere it lands: `frontend/resources/public/js/render-wasm.{js,wasm}` and `frontend/src/app/render_wasm/api/shared.js` from `./build`, plus `exporter/resources/wasm/render-wasm.{js,wasm}` and `exporter/src/app/wasm/shared.js`. Never commit generated JS/WASM; rebuild instead.
## Commands

1
.serena/memories/scripts/gh.md
View File

@ -119,6 +119,7 @@ python3 scripts/gh.py advisories GHSA-xvj6-fh9w-gjw7
## Key principles
- `issues --compare` collects every `#NNNN` token in the file regardless of its link target, so a same-numbered reference to another tracker (e.g. `[Taiga #12312](...)`) or an inline prose number falsely marks that GitHub issue as already listed; verify against `changelog.py cross-ref` when a number collides.
- Changelog-specific checks (merge-status verification, milestone-vs-changelog cross-reference, anomaly report) live in `scripts/changelog.py`, which calls this script via subprocess — see `mem:scripts/changelog`.
- All output is JSON — pipe into `jq` or other tools for further processing.

6
.serena/memories/workflow/creating-commits.md
View File

@ -2,6 +2,12 @@
Commit only on explicit request. Before commit: `git status`; exclude unrelated user changes.
**Never stage a plan file.** A plan is session bookkeeping, not part of the
change: the commit carries the code and the tests. The `implement-plan` flow
still closes the plan (`Status: done` plus one `Review Log` line with the issue
URL); that is not a file to ship, and its absence from the commit is correct,
not a missing piece.
Do not guess or hallucinate git author information (Name or Email). Never include the
`--author` flag in git commands unless specifically instructed by the user for a unique
case; assume the local environment is already configured. Allow git commit to

4
CHANGES.md
View File

@ -129,11 +129,15 @@
- Fix file restore leaving its storage objects marked for deletion [#12278](https://github.com/penpot/penpot/issues/12278) (PR: [#12269](https://github.com/penpot/penpot/pull/12269))
- Fix backend workers staying parked forever when nested calls take the same concurrency limits [#12279](https://github.com/penpot/penpot/issues/12279) (PR: [#12269](https://github.com/penpot/penpot/pull/12269))
- Fix one profile's activity blocking the audit events of every other profile [#12280](https://github.com/penpot/penpot/issues/12280) (PR: [#12269](https://github.com/penpot/penpot/pull/12269))
- Fix storage garbage collection blocking database vacuum during long deletions [#12396](https://github.com/penpot/penpot/issues/12396) (PR: [#12397](https://github.com/penpot/penpot/pull/12397))
- Fix saving a version taking seconds and the version quota never counting saved versions [#12404](https://github.com/penpot/penpot/issues/12404) (PR: [#12405](https://github.com/penpot/penpot/pull/12405))
- Fix file_change accumulating revisions that the objects GC never collects [#12418](https://github.com/penpot/penpot/issues/12418) (PR: [#12419](https://github.com/penpot/penpot/pull/12419))
### :sparkles: New features & Enhancements
- Add a graceful instance drain through the /readyz endpoint [#12281](https://github.com/penpot/penpot/issues/12281) (PR: [#12269](https://github.com/penpot/penpot/pull/12269))
- Add an accumulated queue gauge to the CLIMIT metrics [#12282](https://github.com/penpot/penpot/issues/12282) (PR: [#12269](https://github.com/penpot/penpot/pull/12269))
- Expose the backend log events as a metric by logger and level [#12312](https://github.com/penpot/penpot/issues/12312) (PR: [#12313](https://github.com/penpot/penpot/pull/12313))
## 2.18.3

19
backend/scripts/decode-file-data Executable file
View File

@ -0,0 +1,19 @@
#!/usr/bin/env bash
# Decode a file_data blob (app.util.blob) to JSON for inspection with jq.
#
# Offline use (no DB):
# backend/scripts/decode-file-data --input /tmp/blob.bin --format summary | jq .
#
# DB use (backend running, via nREPL):
# ./scripts/nrepl-eval.mjs "(require '[app.scripts.decode-file-data :as dfd] :reload)"
# ./scripts/nrepl-eval.mjs '(dfd/get-file-json sys/system #uuid "..." :format :summary)'
SCRIPT_DIR=$(dirname "$0")
source "$SCRIPT_DIR/_env"
if [ -f "$SCRIPT_DIR/_env.local" ]; then
source "$SCRIPT_DIR/_env.local"
fi
exec clojure -M:dev -m app.scripts.decode-file-data "$@"

24
backend/src/app/binfile/common.clj
View File

@ -708,16 +708,28 @@
library-ids)]
(absorb-unlinked-tokens-source cfg data library-ids)))
(def ^:const import-transaction-timeout-ms
"Ceiling for binfile import transactions (20 minutes). Interpolated
directly into SQL: compile-time constant, never user input."
(* 20 60 1000))
(def transaction-idle-timeout-ms
"Idle-in-transaction guard for the long binfile transactions: 30
minutes. The import and the export of a file run inside one
transaction that can go quiet while the ZIP is decoded and the file
data is realized. The longest operation observed in production is
under 8 minutes, so this leaves room for bigger files and still
aborts a connection that is left idle for half an hour."
1800000)
(defn configure-database-timeouts!
(defn set-transaction-idle-timeout
"Set `idle_in_transaction_session_timeout` for the current
transaction. Must be called inside a transaction: `SET LOCAL` is
scoped to it."
[cfg]
(let [conn (db/get-connection cfg)]
(db/exec-one! conn [(str "SET LOCAL idle_in_transaction_session_timeout = "
import-transaction-timeout-ms)])
transaction-idle-timeout-ms)])))
(defn configure-import-transaction
[cfg]
(set-transaction-idle-timeout cfg)
(let [conn (db/get-connection cfg)]
(db/exec-one! conn ["SET CONSTRAINTS ALL DEFERRED"])))
(defn process-file

2
backend/src/app/binfile/v1.clj
View File

@ -455,7 +455,7 @@
(defn- read-import-v1
[{:keys [::db/conn ::bfc/project-id ::bfc/profile-id ::bfc/input] :as cfg}]
(bfc/configure-database-timeouts! cfg)
(bfc/configure-import-transaction cfg)
(pu/with-open [input (zstd-input-stream input)
input (io/data-input-stream input)]

5
backend/src/app/binfile/v3.clj
View File

@ -274,6 +274,7 @@
(let [detach? (= export-type :detach-libraries)
embed? (= export-type :merge-libraries)]
(db/tx-run! cfg (fn [cfg]
(bfc/set-transaction-idle-timeout cfg)
(cond-> (bfc/get-file cfg file-id
{:realize? true
:include-deleted? true
@ -1234,7 +1235,7 @@
(defn- import-files*
[{:keys [::manifest] :as cfg}]
(bfc/configure-database-timeouts! cfg)
(bfc/configure-import-transaction cfg)
(vswap! bfc/*state* update :index bfc/update-index (:files manifest) :id)
@ -1305,7 +1306,7 @@
:hint "unable to perform in-place update with binfile containing more than 1 file"
:manifest manifest))
(bfc/configure-database-timeouts! cfg)
(bfc/configure-import-transaction cfg)
(let [ref-file (bfc/get-minimal-file cfg file-id ::db/for-update true)
file (first (get manifest :files))

22
backend/src/app/main.clj
View File

@ -28,7 +28,7 @@
[app.jobs.gc :as-alias jobs.gc]
[app.jobs.metrics :as jobs-metrics]
[app.loggers.webhooks :as-alias webhooks]
[app.metrics :as-alias mtx]
[app.metrics :as mtx]
[app.metrics.definition :as-alias mdef]
[app.msgbus :as-alias mbus]
[app.redis :as-alias rds]
@ -70,16 +70,25 @@
::mdef/help "A total number of bytes processed by update-file."
::mdef/type :counter}
:update-file-validation-timing
{::mdef/name "penpot_rpc_update_file_validation_timing"
::mdef/help "update-file validation step timings (milliseconds)."
::mdef/labels ["step"]
::mdef/buckets mtx/default-request-histogram-buckets
::mdef/type :histogram}
:rpc-main-timing
{::mdef/name "penpot_rpc_main_timing"
::mdef/help "RPC command method call timing for main"
::mdef/labels ["name"]
::mdef/buckets mtx/default-request-histogram-buckets
::mdef/type :histogram}
:rpc-management-timing
{::mdef/name "penpot_rpc_management_timing"
::mdef/help "RPC command method call timing for management."
::mdef/labels ["name"]
::mdef/buckets mtx/default-request-histogram-buckets
::mdef/type :histogram}
:websocket-active-connections
@ -112,12 +121,20 @@
;; panel or alert breaks.
::mdef/help "Deprecated: use penpot_jobs_execution_ms, which adds a queue label."
::mdef/labels ["name"]
::mdef/buckets mtx/default-tasks-histogram-buckets
::mdef/type :histogram}
:storage-gc-deleted-objects
{::mdef/name "penpot_storage_gc_deleted_objects_total"
::mdef/help "Storage objects permanently deleted by storage-gc-deleted, by backend."
::mdef/labels ["backend"]
::mdef/type :counter}
:redis-eval-timing
{::mdef/name "penpot_redis_eval_timing"
::mdef/help "Redis EVAL commands execution timings (ms)"
::mdef/labels ["name"]
::mdef/buckets mtx/default-redis-histogram-buckets
::mdef/type :histogram}
:rpc-climit-waiting
@ -136,6 +153,7 @@
{::mdef/name "penpot_rpc_climit_timing"
::mdef/help "Summary of the time between queuing and executing on the CLIMIT"
::mdef/labels ["name"]
::mdef/buckets mtx/default-request-histogram-buckets
::mdef/type :histogram}
:audit-http-handler-queue-size
@ -154,6 +172,7 @@
{::mdef/name "penpot_audit_http_handler_timing"
::mdef/help "Summary of the time between queuing and executing on the audit log http handler"
::mdef/labels []
::mdef/buckets mtx/default-request-histogram-buckets
::mdef/type :histogram}
:storage-s3-requests
@ -203,6 +222,7 @@
{::mdef/name "penpot_http_server_dispatch_timing"
::mdef/help "Histogram of dispatch handler"
::mdef/labels []
::mdef/buckets mtx/default-request-histogram-buckets
::mdef/type :histogram}
:http-worker-queue-size

19
backend/src/app/metrics.clj
View File

@ -148,6 +148,25 @@
(number? value) (str value)
:else fallback))
;; Tail for request and wait timings (RPC, CLIMIT waits, audit handler,
;; HTTP dispatch): covers the 30s/60s CLIMIT timeouts while keeping the
;; sub-7.5s resolution of the default buckets untouched.
(def default-request-histogram-buckets
[1 5 10 25 50 75 100 250 500 750 1000 2500 5000 7500
10000 15000 30000 60000])
;; Tail for background tasks (GC sweeps, file workers): they run until
;; done with no CLIMIT timeout, bounded in practice by the 300s postgres
;; statement timeout.
(def default-tasks-histogram-buckets
[1 5 10 25 50 75 100 250 500 750 1000 2500 5000 7500
10000 15000 30000 60000 120000 300000])
;; Tail for Redis eval: sub-second in practice, one extra bucket covers
;; the pathological (timeout) case without adding idle series.
(def default-redis-histogram-buckets
[1 5 10 25 50 75 100 250 500 750 1000 2500 5000 7500 10000])
(defmulti run-collector! (fn [mdef _] (::mdef/type mdef)))
(defmulti create-collector ::mdef/type)

8
backend/src/app/migrations.clj
View File

@ -523,7 +523,13 @@
:fn (mg/resource "app/migrations/sql/0153-del-orphan-profile-rels-after-team-leave.sql")}
{:name "0154-del-legacy-file-object-thumbnail-table"
:fn (mg/resource "app/migrations/sql/0154-del-legacy-file-object-thumbnail-table.sql")}])
:fn (mg/resource "app/migrations/sql/0154-del-legacy-file-object-thumbnail-table.sql")}
{:name "0155-add-idx-to-file-change"
:fn (mg/resource "app/migrations/sql/0155-add-idx-to-file-change.sql")}
{:name "0156-reset-pg-storage-defaults"
:fn (mg/resource "app/migrations/sql/0156-reset-pg-storage-defaults.sql")}])
(defn apply-migrations!
[pool name migrations]

38
backend/src/app/migrations/sql/0155-add-idx-to-file-change.sql Normal file
View File

@ -0,0 +1,38 @@
-- Index for the version (snapshot) quota checks of app.rpc.quotes:
-- sql:get-snapshots-per-file and sql:get-snapshots-per-team count the live
-- versions created by a user, and this partial index answers both counts
-- with an index only scan, without walking the whole file_change table.
--
-- The predicate must stay identical to the queries: the planner only uses a
-- partial index when the query where clause implies the index predicate.
-- Versions are told apart by label (every snapshot has one); file_change.data
-- is a legacy column, only written by the legacy-db file data backend.
CREATE INDEX IF NOT EXISTS file_change__user_snapshots__idx
ON file_change (file_id)
WHERE label IS NOT NULL
AND created_by = 'user'
AND deleted_at IS NULL;
-- NOTE: on instances where file_change is large, run the statement above
-- out of band, before deploying, with CONCURRENTLY instead: this migration
-- runs inside a transaction, and a plain CREATE INDEX locks the table
-- against writes for the whole build (in Penpot every file save writes a row
-- here), while CONCURRENTLY does not. Run it on its own, not inside a
-- transaction block:
--
-- CREATE INDEX CONCURRENTLY IF NOT EXISTS file_change__user_snapshots__idx
-- ON file_change (file_id)
-- WHERE label IS NOT NULL
-- AND created_by = 'user'
-- AND deleted_at IS NULL;
--
-- A CONCURRENTLY build interrupted midway leaves the index INVALID: drop it
-- and retry. Verify with:
--
-- SELECT indexdef FROM pg_indexes
-- WHERE indexname = 'file_change__user_snapshots__idx';
-- SELECT indisvalid FROM pg_index
-- WHERE indexrelid = 'file_change__user_snapshots__idx'::regclass;
--
-- Once the index exists, this migration finds it and does nothing.

129
backend/src/app/migrations/sql/0156-reset-pg-storage-defaults.sql Normal file
View File

@ -0,0 +1,129 @@
-- Reset explicit `STORAGE EXTERNAL` settings back to the PostgreSQL defaults.
--
-- Those were set when PostgreSQL ran on top of ZFS with transparent
-- compression (see 0041-mod-pg-storage-options.sql and the per-table
-- migrations that followed it): skipping the TOAST compression done by
-- PostgreSQL made sense there. On regular filesystems it only wastes
-- space and I/O bandwidth, so restore the default storage strategy of
-- each column type (EXTENDED for every column below: text, text[],
-- jsonb and bytea).
--
-- This only touches the catalog, it does not rewrite the tables:
-- already stored TOAST values keep their strategy until the row is
-- rewritten, new writes use the default.
--
-- NOTE: DEFAULT restores the storage mode of the column's data type
-- (EXTENDED for every column below: text, text[], jsonb and bytea). It
-- requires PostgreSQL 16 or newer, which is the minimum version the
-- project supports (see the database section of the technical guide).
-- RESET STORAGE is not valid syntax for a column.
ALTER TABLE access_token
ALTER COLUMN name SET STORAGE DEFAULT,
ALTER COLUMN perms SET STORAGE DEFAULT,
ALTER COLUMN token SET STORAGE DEFAULT;
ALTER TABLE audit_log
ALTER COLUMN context SET STORAGE DEFAULT,
ALTER COLUMN name SET STORAGE DEFAULT,
ALTER COLUMN props SET STORAGE DEFAULT,
ALTER COLUMN source SET STORAGE DEFAULT,
ALTER COLUMN type SET STORAGE DEFAULT;
ALTER TABLE comment
ALTER COLUMN content SET STORAGE DEFAULT;
ALTER TABLE comment_thread
ALTER COLUMN page_name SET STORAGE DEFAULT,
ALTER COLUMN participants SET STORAGE DEFAULT;
ALTER TABLE file
ALTER COLUMN data SET STORAGE DEFAULT,
ALTER COLUMN data_backend SET STORAGE DEFAULT,
ALTER COLUMN features SET STORAGE DEFAULT,
ALTER COLUMN name SET STORAGE DEFAULT;
ALTER TABLE file_change
ALTER COLUMN changes SET STORAGE DEFAULT,
ALTER COLUMN data SET STORAGE DEFAULT,
ALTER COLUMN features SET STORAGE DEFAULT,
ALTER COLUMN label SET STORAGE DEFAULT;
ALTER TABLE file_data_fragment
ALTER COLUMN metadata SET STORAGE DEFAULT;
ALTER TABLE file_media_object
ALTER COLUMN mtype SET STORAGE DEFAULT,
ALTER COLUMN name SET STORAGE DEFAULT;
ALTER TABLE file_thumbnail
ALTER COLUMN data SET STORAGE DEFAULT,
ALTER COLUMN props SET STORAGE DEFAULT;
ALTER TABLE global_complaint_report
ALTER COLUMN content SET STORAGE DEFAULT,
ALTER COLUMN type SET STORAGE DEFAULT;
ALTER TABLE profile
ALTER COLUMN auth_backend SET STORAGE DEFAULT,
ALTER COLUMN email SET STORAGE DEFAULT,
ALTER COLUMN fullname SET STORAGE DEFAULT,
ALTER COLUMN lang SET STORAGE DEFAULT,
ALTER COLUMN password SET STORAGE DEFAULT,
ALTER COLUMN props SET STORAGE DEFAULT,
ALTER COLUMN theme SET STORAGE DEFAULT;
ALTER TABLE profile_complaint_report
ALTER COLUMN content SET STORAGE DEFAULT,
ALTER COLUMN type SET STORAGE DEFAULT;
ALTER TABLE project
ALTER COLUMN name SET STORAGE DEFAULT;
ALTER TABLE scheduled_task
ALTER COLUMN cron_expr SET STORAGE DEFAULT,
ALTER COLUMN id SET STORAGE DEFAULT;
ALTER TABLE server_error_report
ALTER COLUMN content SET STORAGE DEFAULT;
ALTER TABLE server_prop
ALTER COLUMN content SET STORAGE DEFAULT,
ALTER COLUMN id SET STORAGE DEFAULT;
ALTER TABLE storage_object
ALTER COLUMN backend SET STORAGE DEFAULT,
ALTER COLUMN metadata SET STORAGE DEFAULT;
ALTER TABLE task
ALTER COLUMN error SET STORAGE DEFAULT,
ALTER COLUMN label SET STORAGE DEFAULT,
ALTER COLUMN name SET STORAGE DEFAULT,
ALTER COLUMN props SET STORAGE DEFAULT,
ALTER COLUMN queue SET STORAGE DEFAULT,
ALTER COLUMN status SET STORAGE DEFAULT;
ALTER TABLE team
ALTER COLUMN name SET STORAGE DEFAULT;
ALTER TABLE team_font_variant
ALTER COLUMN font_family SET STORAGE DEFAULT,
ALTER COLUMN font_style SET STORAGE DEFAULT;
ALTER TABLE team_invitation
ALTER COLUMN email_to SET STORAGE DEFAULT,
ALTER COLUMN role SET STORAGE DEFAULT;
ALTER TABLE usage_quote
ALTER COLUMN target SET STORAGE DEFAULT;
ALTER TABLE webhook
ALTER COLUMN error_code SET STORAGE DEFAULT,
ALTER COLUMN mtype SET STORAGE DEFAULT,
ALTER COLUMN secret_key SET STORAGE DEFAULT,
ALTER COLUMN uri SET STORAGE DEFAULT;
ALTER TABLE webhook_delivery
ALTER COLUMN error_code SET STORAGE DEFAULT,
ALTER COLUMN req_data SET STORAGE DEFAULT,
ALTER COLUMN rsp_data SET STORAGE DEFAULT;

10
backend/src/app/rpc/commands/files.clj
View File

@ -1384,8 +1384,16 @@
WHERE file_id = ANY(?::uuid[])")
(def ^:private sql:restore-file-changes
;; Only durable history comes back: a snapshot (carries `label`) or a legacy
;; payload row (carries `data`, the column the new model replaces with
;; `file_data`). Revisions have neither and are deliberately left alone: they
;; are born already scheduled for deletion one hour later, and the collector
;; only looks at rows that carry a date, so unmarking one here would keep it
;; in the table for ever. The `data` half is a guard for the rows still
;; pending migration, not a dependency: drop it with the last one.
"UPDATE file_change SET deleted_at = null
WHERE file_id = ANY(?::uuid[])")
WHERE file_id = ANY(?::uuid[])
AND (label IS NOT NULL OR data IS NOT NULL)")
(def ^:private sql:restore-file-data
"UPDATE file_data SET deleted_at = null

119
backend/src/app/rpc/commands/files_update.clj
View File

@ -168,7 +168,8 @@
(assoc :file file)
(assoc :changes changes))
cfg (assoc cfg ::timestamp (ct/now))
cfg (assoc cfg ::timestamp (ct/now)
::validation-timings (atom {}))
tpoint (ct/tpoint)]
@ -209,9 +210,39 @@
(binding [l/*context* (some-> (meta params)
(get :app.http/request)
(errors/request->context))]
(-> (update-file* cfg params)
(rph/with-defer #(let [elapsed (tpoint)]
(l/trace :hint "update-file" :time (ct/format-duration elapsed))))))))
(try
(-> (update-file* cfg params)
(rph/with-defer #(let [elapsed (tpoint)
timings @(::validation-timings cfg)]
(l/dbg :hint "update-file"
:file-id (str (:id file))
:project-id (str (:project-id file))
:team-id (str (:team-id file))
:status "success"
:time (ct/format-duration elapsed)
:resolve-libs-ms (get timings :resolve-libs-ms)
:process-changes-ms (get timings :process-changes-ms)
:soft-validate-affected-ms (get timings :soft-validate-affected-ms)
:soft-validate-schema-ms (get timings :soft-validate-schema-ms)
:validate-affected-ms (get timings :validate-affected-ms)
:validate-schema-ms (get timings :validate-schema-ms)))))
(catch Throwable cause
;; NOTE: `with-defer` hooks only run on the success path
;; (`handle-response`), so timings collected before a hard
;; validation failure would be lost without this inline log.
(let [timings @(::validation-timings cfg)]
(l/dbg :hint "update-file"
:file-id (str (:id file))
:project-id (str (:project-id file))
:team-id (str (:team-id file))
:status "error"
:resolve-libs-ms (get timings :resolve-libs-ms)
:process-changes-ms (get timings :process-changes-ms)
:soft-validate-affected-ms (get timings :soft-validate-affected-ms)
:soft-validate-schema-ms (get timings :soft-validate-schema-ms)
:validate-affected-ms (get timings :validate-affected-ms)
:validate-schema-ms (get timings :validate-schema-ms)))
(throw cause))))))
(defn- update-file*
"Internal function, part of the update-file process, that encapsulates
@ -385,6 +416,16 @@
(catch Throwable cause
(l/error :hint "file schema validation error" :cause cause))))
(defn- record-validation-timing
[cfg step-key step-label elapsed-ms]
(when-let [metrics (::mtx/metrics cfg)]
(when (mtx/metrics? metrics)
(mtx/run! metrics {:id :update-file-validation-timing
:labels [step-label]
:val (double elapsed-ms)})))
(when-let [timings (::validation-timings cfg)]
(swap! timings assoc step-key elapsed-ms)))
(defn- soft-validate-file!
[file libs changes]
(try
@ -395,11 +436,21 @@
(defn- process-changes-and-validate
[cfg file changes skip-validate]
(let [libs
(when (and (or (contains? cf/flags :file-validation)
(contains? cf/flags :soft-file-validation))
(not skip-validate))
(bfc/get-resolved-file-libraries cfg file))
(let [should-resolve-libs?
(and (or (contains? cf/flags :file-validation)
(contains? cf/flags :soft-file-validation))
(not skip-validate))
tlibs
(ct/tpoint)
libs
(try
(when should-resolve-libs?
(bfc/get-resolved-file-libraries cfg file))
(finally
(when should-resolve-libs?
(record-validation-timing cfg :resolve-libs-ms "resolve-libs" (inst-ms (tlibs))))))
;; The main purpose of this atom is provide a contextual state
;; for the changes subsystem where optionally some hints can
@ -409,32 +460,56 @@
state
(atom {})
file
(binding [cpc/*state* state]
(-> (files/check-version! file)
(update :revn inc)
(update :data cpc/process-changes changes)
(update :data d/without-nils)))
tproc
(ct/tpoint)
file
(if-let [media-refs (-> @state :media-refs not-empty)]
(bfc/update-media-references! cfg file media-refs)
file)]
(try
(let [file
(binding [cpc/*state* state]
(-> (files/check-version! file)
(update :revn inc)
(update :data cpc/process-changes changes)
(update :data d/without-nils)))
file
(if-let [media-refs (-> @state :media-refs not-empty)]
(bfc/update-media-references! cfg file media-refs)
file)]
file)
(finally
(record-validation-timing cfg :process-changes-ms "process-changes" (inst-ms (tproc)))))]
(binding [pmap/*tracked* nil]
(when (contains? cf/flags :soft-file-validation)
(soft-validate-file! file libs changes))
(let [t (ct/tpoint)]
(try
(soft-validate-file! file libs changes)
(finally
(record-validation-timing cfg :soft-validate-affected-ms "soft-validate-affected" (inst-ms (t)))))))
(when (contains? cf/flags :soft-file-schema-validation)
(soft-validate-file-schema! file))
(let [t (ct/tpoint)]
(try
(soft-validate-file-schema! file)
(finally
(record-validation-timing cfg :soft-validate-schema-ms "soft-validate-schema" (inst-ms (t)))))))
(when (and (contains? cf/flags :file-validation)
(not skip-validate))
(val/validate-file-affected! file libs changes))
(let [t (ct/tpoint)]
(try
(val/validate-file-affected! file libs changes)
(finally
(record-validation-timing cfg :validate-affected-ms "validate-affected" (inst-ms (t)))))))
(when (and (contains? cf/flags :file-schema-validation)
(not skip-validate))
(val/validate-file-schema! file)))
(let [t (ct/tpoint)]
(try
(val/validate-file-schema! file)
(finally
(record-validation-timing cfg :validate-schema-ms "validate-schema" (inst-ms (t))))))))
file))

17
backend/src/app/rpc/commands/media.clj
View File

@ -40,6 +40,9 @@
(declare create-file-media-object)
;; TEMPORARILY DISABLED (2026-10-19): the media-storage-bytes-per-team
;; quote check that used this lookup is disabled below. Kept active
;; because the audit props of the upload still report the team id.
(def ^:private sql:get-team-id-for-file
"SELECT p.team_id
FROM file AS f
@ -66,11 +69,15 @@
(media.v/validate-media-size! content)
(let [team-id (:team-id (db/exec-one! pool [sql:get-team-id-for-file file-id]))]
(quotes/check! cfg {::quotes/id ::quotes/media-storage-bytes-per-team
::quotes/profile-id profile-id
::quotes/team-id team-id
::quotes/incr (:size content)})
;; TEMPORARILY DISABLED (2026-10-19): media-storage-bytes-per-team
;; quote performs a full SUM over storage_object per upload (3-12s in
;; prod on a 3.5TB DB, ~34 slow executions/hour). Disabled until we
;; land a cheaper approximation (cached counter / rewritten query).
;; TODO: re-enable with the new implementation.
;; (quotes/check! cfg {::quotes/id ::quotes/media-storage-bytes-per-team
;; ::quotes/profile-id profile-id
;; ::quotes/team-id team-id
;; ::quotes/incr (:size content)})
(db/run! cfg (fn [{:keys [::db/conn] :as cfg}]
;; We get the minimal file for proper checking if
;; file is not already deleted

4
backend/src/app/rpc/quotes.clj
View File

@ -428,7 +428,7 @@
WHERE fc.file_id = ?
AND fc.created_by = 'user'
AND fc.deleted_at IS NULL
AND fc.data IS NOT NULL")
AND fc.label IS NOT NULL")
(defmethod check-quote ::snapshots-per-file
[{:keys [::profile-id ::file-id ::team-id ::project-id ::target] :as cfg}]
@ -461,7 +461,7 @@
WHERE p.team_id = ?
AND fc.created_by = 'user'
AND fc.deleted_at IS NULL
AND fc.data IS NOT NULL")
AND fc.label IS NOT NULL")
(defmethod check-quote ::snapshots-per-team
[{:keys [::profile-id ::team-id ::target] :as cfg}]

300
backend/src/app/scripts/decode_file_data.clj Normal file
View File

@ -0,0 +1,300 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
(ns app.scripts.decode-file-data
"Decode a file's `file_data` binary blob to `jq`-readable JSON.
The content of `file_data.data` (and `file.data` on the `legacy-db`
backend, or the `file-data` object on storage) is not text: it is a blob
encoded with `app.util.blob` (versions 1, 3, 4 and 5; transit+json or
fressian plus lz4/zstd). That is why it cannot be inspected with SQL
directly.
Two uses:
1. Offline (no DB, just the blob): dump the `bytea` to a file with
`psql` (`\\copy ...` or `encode(data, 'base64')`) and pass it with `--input`.
Needs no system and no `file_data` read permissions.
backend/scripts/decode-file-data --input /tmp/blob.bin --format summary | jq .
backend/scripts/decode-file-data --input /tmp/blob.b64 --from-base64 --format pages | jq .
2. Inside the penpot context (nREPL with the backend running, where
`sys/system` already exists): call `get-file-json` or `get-snapshot-json`.
They resolve the backend (`legacy-db`, `db`, `storage`), realize the
pointer-maps and objects-maps and return JSON.
(require '[app.scripts.decode-file-data :as dfd] :reload)
(spit \"/tmp/maquette.json\"
(dfd/get-file-json sys/system #uuid \"9e056cfe-f83f-81f8-8008-cc885732f00d\"
:format :full))
(spit \"/tmp/snap-518.json\"
(dfd/get-snapshot-json sys/system #uuid \"9e056cfe-f83f-81f8-8008-cc885732f00d\"
#uuid \"<snapshot-id>\" :format :summary))
Then filter with `jq`: `.pages`, `.data[\"pages-index\"] | keys`,
`[.data[\"pages-index\"][] | {id, name}]`, etc."
(:require
[app.binfile.common :as bfc]
[app.common.json :as json]
[app.db :as db]
[app.features.file-snapshots :as fsnap]
[app.util.blob :as blob]
[app.util.objects-map :as omap-legacy]
[app.util.pointer-map :as pmap]
[clojure.java.io :as io]
[clojure.string :as str])
(:import
java.util.Base64))
(set! *warn-on-reflection* true)
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; JSON sanitization
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defn- sanitize-key
[k]
(cond
(keyword? k) (subs (str k) 1)
(symbol? k) (str k)
(uuid? k) (str k)
:else (str k)))
(declare sanitize)
(defn- sanitize-map
[m]
(persistent!
(reduce-kv (fn [acc k v]
(assoc! acc (sanitize-key k) (sanitize v)))
(transient {})
m)))
(defn sanitize
"Convert a blob-decoded value to plain JSON-ready data.
- Unrealized pointers become `{\"$pointer\": \"<uuid>\"}`.
- `objects-map` instances are materialized to a plain map.
- Bytes become `{\"$bytes\": \"<base64>\"}`.
- UUIDs, instants, keywords and symbols become strings."
[x]
(cond
(nil? x) nil
(string? x) x
(boolean? x) x
(number? x)
(if (ratio? x) (double x) x)
(keyword? x) (subs (str x) 1)
(symbol? x) (str x)
(uuid? x) (str x)
(inst? x) (str x)
(bytes? x)
{"$bytes" (.encodeToString (Base64/getEncoder) ^bytes x)}
(pmap/pointer-map? x)
{"$pointer" (str (pmap/get-id x))}
(omap-legacy/objects-map? x)
(sanitize (into {} x))
(map? x)
(sanitize-map x)
(set? x)
(mapv sanitize x)
(coll? x)
(mapv sanitize x)
:else
(str x)))
(defn data->json
"Encode sanitizable data to a JSON string."
[data]
(json/encode (sanitize data)))
(defn blob->json
"Decode a `file_data` blob (`app.util.blob`) and return it as JSON."
[^bytes blob-bytes]
(data->json (blob/decode blob-bytes)))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; jq-oriented investigation summary
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defn summarize-data
"Small, stable summary for `jq`: pages (in `:pages` order) with object
counts, plus component, color and typography counters. Takes the already
decoded `:data` (not the JSON)."
[data]
(let [pages (:pages data)
pages-index (:pages-index data)
page-entry (fn [pid]
{:id (str pid)
:name (get-in pages-index [pid :name])
:objects (count (get-in pages-index [pid :objects]))})]
{:pages (mapv page-entry (or pages []))
:stats {:pages (count (or pages []))
:components (count (:components data))
:colors (count (:colors data))
:typographies (count (:typographies data))}}))
(defn file-data-summary-json
[data]
(data->json (summarize-data data)))
(defn file-data-pages-json
[data]
(data->json (:pages (summarize-data data))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; DB access (for use from nREPL with the backend running)
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defn- write-output
[^String payload output]
(if output
(do (spit output payload) output)
(print payload))
payload)
(defn get-file-json
"Return a file as JSON (realized). For nREPL.
`format` can be `:full` (whole file), `:data` (`:data` only),
`:summary` or `:pages`. With `:output` it also writes it to a file."
[system file-id & {:keys [format output] :or {format :full}}]
(let [file (db/run! system bfc/get-file file-id :realize? true)
data (case format
:full file
:data (:data file)
:summary (summarize-data (:data file))
:pages (:pages (summarize-data (:data file))))]
(write-output (data->json data) output)))
(defn get-snapshot-json
"Return a snapshot as JSON (realized). For nREPL.
Takes the same `format` and `output` as `get-file-json`."
[system file-id snapshot-id & {:keys [format output] :or {format :full}}]
(let [snapshot (db/run! system
(fn [cfg]
(fsnap/get-snapshot cfg file-id snapshot-id)))
data (case format
:full snapshot
:data (:data snapshot)
:summary (summarize-data (:data snapshot))
:pages (:pages (summarize-data (:data snapshot))))]
(write-output (data->json data) output)))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Offline CLI: decode a file holding the blob
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(def ^:private cli-usage
(str "Usage: decode-file-data --input PATH [--output PATH] [--format full|data|summary|pages]\n"
" [--from-base64|--from-hex]\n"
"\n"
"Decode a file_data blob (app.util.blob) to JSON for inspection with jq.\n"
"\n"
"Examples:\n"
" psql -c \"\\\\copy (select data from file_data where file_id='...' and id='...') to '/tmp/blob.bin' with binary\"\n"
" decode-file-data --input /tmp/blob.bin --format summary | jq .\n"
" decode-file-data --input /tmp/blob.b64 --from-base64 --format pages | jq .\n"))
(defn- parse-args
[args]
(loop [args args
opts {:format "full"}]
(if (empty? args)
opts
(let [[a b & rest-args] args]
(cond
(contains? #{"-h" "--help"} a)
(assoc opts :help true)
(= a "--input")
(recur rest-args (assoc opts :input b))
(= a "--output")
(recur rest-args (assoc opts :output b))
(= a "--format")
(recur rest-args (assoc opts :format b))
(= a "--from-base64")
(recur (cons b rest-args) (assoc opts :from :base64))
(= a "--from-hex")
(recur (cons b rest-args) (assoc opts :from :hex))
:else
(assoc opts :error (str "unknown argument: " a)))))))
(defn- hex->bytes
[^String s]
(let [s (str/trim s)
s (cond
(str/starts-with? s "\\x") (subs s 2)
(str/starts-with? s "0x") (subs s 2)
:else s)
s (str/replace s #"\s+" "")
n (/ (count s) 2)
out (byte-array n)]
(dotimes [i n]
(aset-byte out i (unchecked-byte (Integer/parseInt (subs s (* i 2) (+ (* i 2) 2)) 16))))
out))
(defn- read-input-bytes
[{:keys [input from]}]
(cond
(nil? input)
(throw (ex-info "missing --input" {}))
(= from :base64)
(.decode (Base64/getDecoder) ^String (str/trim (slurp (io/file input))))
(= from :hex)
(hex->bytes (slurp (io/file input)))
:else
(java.nio.file.Files/readAllBytes (.toPath (io/file input)))))
(defn -main
[& args]
(let [{:keys [help error input format output] :as opts} (parse-args (vec args))]
(cond
help
(do (print cli-usage) nil)
error
(do (binding [*out* *err*] (println error) (println cli-usage)) (System/exit 2))
(nil? input)
(do (binding [*out* *err*] (println "missing --input") (println cli-usage)) (System/exit 2))
(not (contains? #{"full" "data" "summary" "pages"} (or format "full")))
(do (binding [*out* *err*] (println (str "unknown format: " format))) (System/exit 2))
:else
(let [blob-bytes (read-input-bytes opts)
decoded (blob/decode blob-bytes)
payload (case format
"full" decoded
"data" (:data decoded (:data decoded decoded))
"summary" (summarize-data (:data decoded decoded))
"pages" (:pages (summarize-data (:data decoded decoded))))
;; NOTE: the file_data blob already IS the file's :data; but if
;; someone dumps the whole serialized row, tolerate both.
out (data->json payload)]
(if output
(spit output out)
(println out))))))

300
backend/src/app/storage/gc_deleted.clj
View File

@ -19,185 +19,175 @@
[app.common.time :as ct]
[app.db :as db]
[app.jobs :as jobs]
[app.metrics :as mtx]
[app.storage :as sto]
[app.storage.impl :as impl]
[clojure.set :as set]
[integrant.core :as ig]))
(def ^:private max-attempts
"Maximum number of deletion attempts before giving up and accepting
the orphan blob."
7)
(def ^:private chunk-size
"Pending objects deleted in each transaction."
25)
(def ^:private sql:lock-sobjects
"SELECT id FROM storage_object
WHERE id = ANY(?::uuid[])
FOR UPDATE
SKIP LOCKED")
(defn- lock-ids
"Perform a select before delete for proper object locking and
prevent concurrent operations and we proceed only with successfully
locked objects."
[conn ids]
(let [ids (db/create-array conn "uuid" ids)]
(->> (db/exec! conn [sql:lock-sobjects ids])
(into #{} (map :id))
(not-empty))))
(def ^:private sql:delete-sobjects
"DELETE FROM storage_object
WHERE id = ANY(?::uuid[])")
(defn- delete-sobjects
[conn ids]
(let [ids (db/create-array conn "uuid" ids)]
(-> (db/exec-one! conn [sql:delete-sobjects ids])
(db/get-update-count))))
(defn- delete-by-backend
"Deletes the locked objects of one backend group: the database rows and
their backend content. Runs inside the caller's transaction. Returns the
number of deleted rows; reporting it (metrics) is the caller's job, once
the transaction has committed."
[{:keys [::db/conn ::sto/storage]} backend-id ids]
(let [ids-array (db/create-array conn "uuid" ids)
total (-> (db/exec-one! conn [sql:delete-sobjects ids-array])
(db/get-update-count))]
(-> (impl/resolve-backend storage backend-id)
(impl/del-objects-in-bulk ids))
(def ^:private sql:increment-attempts-and-defer
"UPDATE storage_object
SET deletion_attempts = deletion_attempts + 1,
deleted_at = NOW() + INTERVAL '1 day'
WHERE id = ANY(?::uuid[])")
(defn- increment-attempts-and-defer
[conn ids]
(let [ids (db/create-array conn "uuid" ids)]
(db/exec-one! conn [sql:increment-attempts-and-defer ids])))
(def ^:private sql:delete-give-up
"DELETE FROM storage_object
WHERE id = ANY(?::uuid[])
AND deletion_attempts >= ?")
(defn- delete-give-up
[conn ids]
(let [ids (db/create-array conn "uuid" ids)]
(db/exec-one! conn [sql:delete-give-up ids max-attempts])))
(defn- process-chunk
"Attempt to delete a chunk of storage objects from a specific backend.
This function runs inside the chunk's own transaction (see
`delete-in-bulk`) — it does NOT open its own transaction. The rows are
locked via FOR UPDATE SKIP LOCKED before deleting.
Returns the number of successfully deleted objects, or 0 if no rows
could be locked."
[conn storage backend-id ids]
(if-let [locked-ids (lock-ids conn ids)]
(let [fail-ids (try
(-> (impl/resolve-backend storage backend-id)
(impl/del-objects-in-bulk locked-ids))
(catch Throwable cause
(l/err :hint "error on physical deletion, will retry"
:ids locked-ids
:cause cause)
locked-ids))
ok-ids (set/difference locked-ids fail-ids)]
(doseq [id ok-ids]
(l/dbg :hint "permanently delete storage object"
:id (str id)
:backend (name backend-id)))
(when (seq ok-ids)
;; NOTE: rows arrive here with no live references (the mark-time
;; scan in gc-touched owns that check, also for the upload-session
;; bucket). If a NO ACTION reference still exists, this DELETE
;; aborts the chunk loud instead of breaking anyone silently.
(delete-sobjects conn ok-ids))
(when (seq fail-ids)
;; NOTE: deferred rows keep both their row and their references
;; (upload-session mappings included) until the final give-up, so
;; a backend outage does not break live sessions.
(increment-attempts-and-defer conn fail-ids)
(let [given-up (delete-give-up conn fail-ids)]
(when (pos? (db/get-update-count given-up))
(l/wrn :hint "giving up on orphan blob after max attempts"
:ids fail-ids
:max-attempts max-attempts))))
(count ok-ids))
0))
(defn- delete-in-bulk
[cfg backend-id ids]
;; Each chunk runs in its own transaction, on its own connection: a chunk
;; that fails (for example, an object that a foreign key still references) is
;; logged and skipped, and it cannot poison the cursor that feeds the rest.
(try
(db/tx-run! cfg
(fn [{:keys [::db/conn ::sto/storage]}]
(process-chunk conn storage backend-id ids)))
(catch Throwable cause
(l/err :hint "unexpected error on bulk deletion, skipping chunk"
:ids ids
:cause cause)
0)))
(doseq [id ids]
(l/dbg :hint "permanently delete storage object"
:id (str id)
:backend (name backend-id)))
total))
(defn- group-by-backend
[items]
(d/group-by (comp keyword :backend) :id #{} items))
[rows]
(d/group-by (comp keyword :backend) :id #{} rows))
(defn- process-chunk-deletion
"Deletes the locked rows of one chunk, grouped by backend. Runs inside the
caller's transaction: any failure here aborts it, so the whole chunk is
rolled back (nothing is deleted and the locks are released) and skipped by
the caller. Returns a map of backend id to deleted row count."
[tx rows]
(->> (group-by-backend rows)
(map (fn [[backend-id ids]]
[backend-id (delete-by-backend tx backend-id ids)]))
(into {})))
(def ^:private sql:get-deleted-sobjects
"SELECT id, backend
FROM storage_object
WHERE deleted_at IS NOT NULL
AND deleted_at <= ?
AND status = 'valid'
ORDER BY deleted_at ASC")
"SELECT s.id, s.backend, s.deleted_at
FROM storage_object AS s
WHERE s.deleted_at IS NOT NULL
AND s.deleted_at <= ?
AND s.status = 'valid'
AND (s.deleted_at, s.id) > (?, ?)
ORDER BY s.deleted_at ASC, s.id ASC
LIMIT ?
FOR UPDATE SKIP LOCKED")
(defn- pending-row
"Takes the columns the GC needs from a `db/plan` row.
(def ^:private sql:get-first-deleted-sobjects
"SELECT s.id, s.backend, s.deleted_at
FROM storage_object AS s
WHERE s.deleted_at IS NOT NULL
AND s.deleted_at <= ?
AND s.status = 'valid'
ORDER BY s.deleted_at ASC, s.id ASC
LIMIT ?
FOR UPDATE SKIP LOCKED")
A plan row is a view over the open result set, valid only while the reduction
is running, so it cannot be stored for later: this has to happen inside the
reducing function."
[row]
{:id (:id row)
:backend (:backend row)})
(def ^:private chunk-size
"Pending objects read, locked and deleted in each transaction."
25)
(def ^:private throttle-every
"Deleted objects between throttle pauses."
5000)
(def ^:private throttle-ms
"Pause applied every `throttle-every` deleted objects, letting the
database (and its replicas) breathe on huge backlogs."
5000)
(defn- throttle
"Sleeps `throttle-ms` when the deleted count crosses a `throttle-every`
boundary. Runs between chunk transactions, never holding one open."
[prev-total new-total]
(when (> (quot new-total throttle-every)
(quot prev-total throttle-every))
(l/inf :hint "throttling storage gc"
:deleted new-total
:pause-ms throttle-ms)
(Thread/sleep (long throttle-ms))))
(defn- fetch-and-lock-chunk
"Reads and locks one chunk of pending objects in `(deleted_at, id)` order.
Objects already locked by another process are skipped, so two concurrent
runs never take the same chunk. Runs inside the caller's transaction."
[conn last-deleted-at last-id]
(let [[sql params] (if (some? last-deleted-at)
[sql:get-deleted-sobjects
[(ct/now) last-deleted-at last-id chunk-size]]
[sql:get-first-deleted-sobjects
[(ct/now) chunk-size]])]
(db/exec! conn (into [sql] params))))
(defn- clean-deleted
"Walks the pending objects with `db/plan`, a server-side cursor that streams
the rows instead of materialising them, and deletes them chunk by chunk.
"Processes the pending objects chunk by chunk with keyset pagination: each
chunk is one short transaction that reads and locks the next `chunk-size`
objects after the `(deleted_at, id)` cursor and deletes them (database rows
and backend content). Only `valid` objects are read: pending ones (still
being written, see the status lifecycle) stay untouched even when already
marked as deleted.
The rows are read in their own read-only transaction and every chunk of
`chunk-size` objects is deleted in its own one (see `delete-in-bulk`), so a
chunk that fails is logged and skipped instead of aborting the whole run, and
the work of the previous chunks is already committed. A physical-deletion
failure inside a chunk defers its rows one day and gives up after
`max-attempts` attempts (see `process-chunk`)."
[cfg]
(let [xform (comp (map pending-row)
(partition-all chunk-size)
(mapcat group-by-backend))
rf (fn
([total] total)
([total [backend-id ids]]
(let [deleted (or (delete-in-bulk cfg backend-id ids) 0)]
;; Beat the job after every chunk: each chunk commits on its
;; own, so a long sweep neither loses work on a late failure
;; nor outruns the orphan lease.
(jobs/heartbeat cfg)
(+ total deleted))))]
(db/tx-run! (assoc cfg ::db/read-only true)
(fn [{:keys [::db/conn]}]
(transduce xform rf 0
(db/plan conn [sql:get-deleted-sobjects (ct/now)]))))))
A chunk that fails aborts its transaction: nothing of the chunk is deleted,
its locks are released, and it is logged and skipped. The cursor advances
to the `(deleted_at, id)` of the last row of each read chunk regardless of
the deletion result, so a failed chunk is not visited again in the same
run. There is no long-lived read transaction: each transaction lives only
as long as one chunk, letting the vacuum proceed between chunks. Rows
locked by a concurrent run are skipped and deferred to the next run; a
chunk that fails after a partial backend delete self-heals on the next
run (physical deletes are idempotent). Every `throttle-every` deleted
objects the run pauses `throttle-ms` (see `throttle`)."
[{:keys [::mtx/metrics] :as cfg}]
(loop [total 0
last-deleted-at nil
last-id nil]
(let [[rows counts]
(db/tx-run! cfg
(fn [{:keys [::db/conn] :as tx}]
(let [rows (fetch-and-lock-chunk conn last-deleted-at last-id)]
(try
[rows (process-chunk-deletion tx rows)]
(catch Throwable cause
;; Explicit rollback, then return normally (do
;; not rethrow): the caller needs `rows` to
;; advance the keyset cursor past the failed
;; chunk. The commit that `tx-run!` performs
;; afterwards is a JDBC no-op on the already
;; rolled-back connection.
(db/rollback! conn)
(l/err :hint "unexpected error on chunk deletion"
:ids (map :id rows)
:cause cause)
[rows {}])))))]
;; Reported only after the chunk commits: a rolled-back chunk leaves
;; an empty count map, so retried rows are never counted twice.
(doseq [[backend-id n] counts]
(mtx/run! metrics
:id :storage-gc-deleted-objects
:inc n
:labels [(name backend-id)]))
;; Beat the job after every chunk: each chunk commits on its own, so a
;; long sweep neither loses work on a late failure nor outruns the
;; orphan lease.
(jobs/heartbeat cfg)
(if-let [cursor (peek rows)]
(let [new-total (+ total (reduce + 0 (vals counts)))]
(throttle total new-total)
(recur new-total
(:deleted-at cursor)
(:id cursor)))
total))))
(declare execute-storage-gc-deleted)
(defmethod ig/assert-key ::job-def
[_ params]
(assert (sto/valid-storage? (::sto/storage params)) "expect valid storage")
(assert (db/pool? (::db/pool params)) "expect valid db pool"))
(assert (db/pool? (::db/pool params)) "expect valid db pool")
(assert (mtx/metrics? (::mtx/metrics params)) "expect valid metrics instance"))
(def schema:storage-gc-deleted-params
"Params map (no params needed; cfg-provided config only)."

8
backend/src/app/tasks/delete_object.clj
View File

@ -87,10 +87,14 @@
:file-id id
:cause cause))))
;; Mark file change to be deleted
;; Mark file change to be deleted, but only the rows that do not carry a
;; date yet. A revision is born already scheduled for deletion one hour
;; later, and rewriting its date here would push it out to the trash
;; lifetime; the collector would still take it, just much later than it
;; should.
(db/update! conn :file-change
{:deleted-at deleted-at}
{:file-id id}
{:file-id id :deleted-at nil}
{::db/return-keys false})
;; Mark file data fragment to be deleted

40
backend/test/backend_tests/binfile_test.clj
View File

@ -2650,24 +2650,28 @@
(t/is (= :validation (:type out)))
(t/is (= :max-file-size-reached (:code out))))))
(t/deftest import-configures-finite-transaction-idle-timeout
;; The binfile import transaction must set a finite ceiling for
;; idle_in_transaction_session_timeout (20 min) instead of disabling
;; it entirely: a stalled import must not retain a pool connection
;; without any upper bound. The ceiling must be scoped with SET
;; LOCAL: once the transaction ends, the pool session default must
;; be restored for subsequent transactions.
(let [pg-setting-sql ["SELECT setting FROM pg_settings WHERE name = 'idle_in_transaction_session_timeout'"]
in-tx (db/tx-run! th/*system*
(fn [cfg]
(bfc/configure-database-timeouts! cfg)
(:setting (db/exec-one! cfg pg-setting-sql))))
;; SET LOCAL must not leak past the transaction boundary
after (db/tx-run! th/*system*
(fn [cfg]
(:setting (db/exec-one! cfg pg-setting-sql))))]
(t/is (= "1200000" in-tx))
(t/is (= "300000" after))))
(def ^:private sql:current-idle-timeout
"SELECT current_setting('idle_in_transaction_session_timeout')::interval
= interval '30 minutes' AS ok")
(t/deftest binfile-transactions-use-a-finite-idle-timeout
;; The binfile import and export transactions must set a finite
;; ceiling for idle_in_transaction_session_timeout (30 min) instead
;; of disabling it entirely: a stalled transaction must not retain a
;; pool connection without any upper bound. The ceiling is scoped
;; with SET LOCAL: once the transaction ends, the pool session
;; default is restored for subsequent transactions.
(db/tx-run! th/*system*
(fn [{::db/keys [conn] :as cfg}]
(t/testing "export transaction"
(bfc/set-transaction-idle-timeout cfg)
(t/is (true? (-> (db/exec-one! conn [sql:current-idle-timeout])
:ok))))
(t/testing "import transaction"
(bfc/configure-import-transaction cfg)
(t/is (true? (-> (db/exec-one! conn [sql:current-idle-timeout])
:ok)))))))
(def ^:private index-test-file-id "22222222-2222-2222-2222-222222222222")
(def ^:private index-test-page-id "44444444-4444-4444-4444-444444444444")

1
backend/test/backend_tests/climit_test.clj
View File

@ -663,7 +663,6 @@
(fn []
(#'climit/republish-waiting (th/metrics-stub) counters)
(t/is (= [3] (waiting-value-of name)))))))
(t/deftest halting-the-component-stops-the-republisher
;; The republisher is a daemon thread started with the component: halting it
;; has to stop the thread, or every restart leaves another one running.

209
backend/test/backend_tests/file_change_retention_test.clj Normal file
View File

@ -0,0 +1,209 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
(ns backend-tests.file-change-retention-test
"Retention of `file_change` rows through the file lifecycle.
A file has two kinds of row. Durable history: the snapshots, which a restore
brings back. And ephemeral revisions: the xlog of an edit, born already
scheduled for deletion one hour later, which nobody needs once the sessions
that were going to read them are gone.
The invariant under test is that a deletion date is assigned once and never
moved: restoring a file must not unmark a revision, and deleting a file must
not push a revision's date out to the trash lifetime. When either happens the
revision becomes invisible to `objects-gc`, which only looks at rows that
carry a date, and it stays in the table for ever."
(:require
[app.common.time :as ct]
[app.common.uuid :as uuid]
[app.db :as db]
[app.rpc :as-alias rpc]
[app.rpc.commands.files :as files]
[app.util.blob :as blob]
[backend-tests.helpers :as th]
[clojure.test :as t]))
(t/use-fixtures :once th/state-init)
(t/use-fixtures :each th/database-reset)
(def ^:private test-now
"The instant every test pins the clock to. Whole seconds on purpose: a value
with a sub-second part does not survive the round trip through the database,
and the test could not compare it for equality."
(ct/inst "2026-10-01T00:00:00Z"))
(def ^:private revision-lifetime
"The lifetime the `update-file` flow gives a revision when it inserts it."
(ct/duration {:hours 1}))
(defn- create-file
[i]
(let [profile (th/create-profile* i {:is-active true})]
[profile (th/create-file* i {:profile-id (:id profile)
:project-id (:default-project-id profile)})]))
(defn- insert-change
"Inserts a `file_change` row directly, so the test builds the exact shape it
needs. The row class is decided by what the caller passes: a revision carries
only `changes`, a snapshot carries `label`, and a legacy payload row carries
`data` without a label. Returns the new row id."
[file-id {:keys [label data changes created-at deleted-at created-by]}]
(let [created-at (or created-at (ct/now))
id (uuid/next)]
(th/db-insert! :file-change
{:id id
:file-id file-id
:created-at created-at
:updated-at created-at
:deleted-at deleted-at
:revn 1
:version 0
:created-by (or created-by "system")
:label label
:data data
:changes changes}
{::db/return-keys false})
id))
(defn- insert-revision
"A revision exactly as the `update-file` flow writes it: no label, no data and
`changes` filled, already scheduled for deletion one hour after creation."
[file-id created-at]
(insert-change file-id {:created-at created-at
:changes (blob/encode {:foo "bar"})
:deleted-at (ct/plus created-at revision-lifetime)}))
(defn- find-change
[id]
(th/db-get :file-change {:id id} {::db/remove-deleted false}))
(defn- find-file
[id]
(th/db-get :file {:id id} {::db/remove-deleted false}))
(defn- restore-file
[file-id]
(db/tx-run! th/*system*
(fn [{:keys [::db/conn]}]
(#'files/restore-files conn [file-id]))))
(t/deftest restoring-a-file-leaves-the-revision-schedule-alone
(binding [ct/*clock* (ct/fixed-clock test-now)]
(let [[_ file] (create-file 1)
file-id (:id file)
trash-at (ct/in-future {:days 7})
old-at (ct/in-past {:hours 2})
revision-id (insert-revision file-id old-at)
revision-at (:deleted-at (find-change revision-id))
snapshot-id (insert-change file-id {:created-at old-at
:created-by "user"
:label "a snapshot"
:deleted-at trash-at})
legacy-id (insert-change file-id {:created-at old-at
:data (blob/encode {:legacy true})
:deleted-at trash-at})]
(restore-file file-id)
(t/testing "the revision keeps the schedule it was born with"
(t/is (= revision-at (:deleted-at (find-change revision-id)))))
(t/testing "the snapshot comes back"
(t/is (nil? (:deleted-at (find-change snapshot-id)))))
(t/testing "the legacy payload row comes back"
(t/is (nil? (:deleted-at (find-change legacy-id))))))))
(t/deftest deleting-a-file-does-not-push-the-revision-schedule-out
(binding [ct/*clock* (ct/fixed-clock test-now)]
(let [[_ file] (create-file 1)
file-id (:id file)
trash-at (ct/in-future {:days 7})
old-at (ct/in-past {:hours 2})
system-at (ct/in-future {:days 2})
revision-id (insert-revision file-id old-at)
revision-at (:deleted-at (find-change revision-id))
snapshot-id (insert-change file-id {:created-at old-at
:created-by "user"
:label "a snapshot"})
system-id (insert-change file-id {:created-at old-at
:label "internal/snapshot/1"
:deleted-at system-at})]
(th/run-task! :delete-object {:object :file
:deleted-at trash-at
:id file-id})
(t/testing "the revision keeps the schedule it was born with"
(t/is (= revision-at (:deleted-at (find-change revision-id)))))
(t/testing "the snapshot takes the file's deletion date"
(t/is (= (:deleted-at (find-file file-id))
(:deleted-at (find-change snapshot-id)))))
(t/testing "the system snapshot keeps the date it already had"
(t/is (= system-at (:deleted-at (find-change system-id))))))))
(t/deftest updating-a-snapshot-cannot-reach-a-revision
;; The snapshot paths clear the deletion mark of the row they touch, which is
;; only legitimate for durable history. They reach a row through
;; `sql:snapshots`, which filters `label IS NOT NULL`, and a revision never
;; carries one. This test pins that: the row has to keep the date it was born
;; with, and the RPC has to reject the id. If a future change loosens that
;; read filter, this goes red instead of the revision losing its date in
;; silence.
(binding [ct/*clock* (ct/fixed-clock test-now)]
(let [[profile file] (create-file 1)
file-id (:id file)
revision-id (insert-revision file-id (ct/in-past {:hours 2}))
revision-at (:deleted-at (find-change revision-id))
out (th/command! {::th/type :update-file-snapshot
::rpc/profile-id (:id profile)
:id revision-id
:label "renamed"})]
(t/testing "the RPC rejects the id"
(t/is (th/ex-info? (:error out)))
(t/is (= :not-found (:type (ex-data (:error out))))))
(t/testing "and the revision keeps the date it was born with"
(t/is (= revision-at (:deleted-at (find-change revision-id))))))))
(t/deftest a-deleted-then-restored-file-does-not-leak-its-revisions
;; The lifecycle that leaked 358,517 rows in production: a file is edited, it
;; ends up in the trash, it comes back, and the revisions that were scheduled
;; to die an hour after they were written have to die anyway.
(binding [ct/*clock* (ct/fixed-clock test-now)]
(let [[_ file] (create-file 1)
file-id (:id file)
revision-id (insert-revision file-id (ct/in-past {:hours 2}))
revision-at (:deleted-at (find-change revision-id))]
(th/run-task! :delete-object {:object :file
:deleted-at (ct/in-future {:days 7})
:id file-id})
(t/testing "deleting the file leaves the revision schedule alone"
(t/is (= revision-at (:deleted-at (find-change revision-id)))))
(restore-file file-id)
(t/testing "restoring the file leaves it alone too"
(t/is (= revision-at (:deleted-at (find-change revision-id)))))
(th/run-task! :objects-gc {})
(t/testing "so the collector takes it"
(t/is (nil? (find-change revision-id)))))))

128
backend/test/backend_tests/files_update_validation_timing_test.clj Normal file
View File

@ -0,0 +1,128 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
(ns backend-tests.files-update-validation-timing-test
(:require
[app.binfile.common :as bfc]
[app.common.files.validate :as val]
[app.common.uuid :as uuid]
[app.config :as cf]
[app.main :as main]
[app.metrics :as mtx]
[app.metrics.definition :as-alias mdef]
[app.rpc.commands.files-update :as files-update]
[clojure.test :as t]
[integrant.core :as ig])
(:import
io.prometheus.client.CollectorRegistry))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Helpers
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defn- bucket-value
[metrics mname label-names label-values le]
(.getSampleValue ^CollectorRegistry (mtx/get-registry metrics)
(str mname "_bucket")
(into-array String (conj (vec label-names) "le"))
(into-array String (conj (vec label-values) le))))
(defn- test-file
[]
{:id (uuid/next)
:revn 0
:version 0
:data {}})
(defn- test-cfg
[metrics]
{::mtx/metrics metrics
::files-update/validation-timings (atom {})})
(def ^:private all-flags
#{:file-validation
:file-schema-validation
:soft-file-validation
:soft-file-schema-validation})
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Tests
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(t/deftest validation-timing-metric-uses-request-buckets
(t/is (= mtx/default-request-histogram-buckets
(get-in main/default-metrics [:update-file-validation-timing ::mdef/buckets])))
(t/is (= "penpot_rpc_update_file_validation_timing"
(get-in main/default-metrics [:update-file-validation-timing ::mdef/name]))))
(t/deftest process-changes-and-validate-records-all-steps
(let [metrics (ig/init-key :app.metrics/metrics {:default main/default-metrics})
cfg (test-cfg metrics)]
(binding [cf/flags all-flags]
(with-redefs [bfc/get-resolved-file-libraries (fn [_ _] {})
val/validate-file-affected! (fn [_ _ _] nil)
val/validate-file-schema! (fn [_] nil)]
(let [file (@#'files-update/process-changes-and-validate cfg (test-file) [] nil)
timings @(::files-update/validation-timings cfg)]
(t/is (map? file))
(t/is (number? (:resolve-libs-ms timings)))
(t/is (number? (:process-changes-ms timings)))
(t/is (number? (:soft-validate-affected-ms timings)))
(t/is (number? (:soft-validate-schema-ms timings)))
(t/is (number? (:validate-affected-ms timings)))
(t/is (number? (:validate-schema-ms timings)))
(doseq [step ["resolve-libs" "process-changes"
"soft-validate-affected" "soft-validate-schema"
"validate-affected" "validate-schema"]]
(t/is (= 1.0 (bucket-value metrics "penpot_rpc_update_file_validation_timing"
["step"] [step] "60000.0"))
(str "expected one sample for step " step))))))))
(t/deftest process-changes-and-validate-skips-when-skip-validate
(let [metrics (ig/init-key :app.metrics/metrics {:default main/default-metrics})
cfg (test-cfg metrics)]
(binding [cf/flags all-flags]
(with-redefs [bfc/get-resolved-file-libraries (fn [_ _]
(throw (ex-info "must not resolve libs" {})))
val/validate-file-affected! (fn [_ _ _]
(throw (ex-info "must not validate" {})))
val/validate-file-schema! (fn [_]
(throw (ex-info "must not validate schema" {})))]
;; soft validations still run: they only check flags, not skip-validate
(let [file (@#'files-update/process-changes-and-validate cfg (test-file) [] true)
timings @(::files-update/validation-timings cfg)]
(t/is (map? file))
(t/is (nil? (:resolve-libs-ms timings)))
(t/is (number? (:process-changes-ms timings)))
(t/is (number? (:soft-validate-affected-ms timings)))
(t/is (number? (:soft-validate-schema-ms timings)))
(t/is (nil? (:validate-affected-ms timings)))
(t/is (nil? (:validate-schema-ms timings)))
(t/is (= 1.0 (bucket-value metrics "penpot_rpc_update_file_validation_timing"
["step"] ["process-changes"] "60000.0"))))))))
(t/deftest process-changes-and-validate-records-timing-on-error
(let [metrics (ig/init-key :app.metrics/metrics {:default main/default-metrics})
cfg (test-cfg metrics)]
(binding [cf/flags all-flags]
(with-redefs [bfc/get-resolved-file-libraries (fn [_ _] {})
val/validate-file-affected! (fn [_ _ _]
(throw (ex-info "bad file" {:code :referential-integrity})))
val/validate-file-schema! (fn [_] nil)]
(let [timings-atom (::files-update/validation-timings cfg)]
(t/is (thrown? clojure.lang.ExceptionInfo
(@#'files-update/process-changes-and-validate cfg (test-file) [] nil)))
;; the failing step still recorded its timing via finally
(t/is (number? (:validate-affected-ms @timings-atom)))
(t/is (number? (:resolve-libs-ms @timings-atom)))
(t/is (number? (:process-changes-ms @timings-atom))))))))
(t/deftest record-validation-timing-tolerates-missing-collaborators
;; production code must not blow up when called without metrics
;; (e.g. scripts) or without the timings collector.
(let [record @#'files-update/record-validation-timing]
(t/is (nil? (record {} :resolve-libs-ms "resolve-libs" 12)))
(t/is (nil? (record {::mtx/metrics nil} :resolve-libs-ms "resolve-libs" 12)))))

70
backend/test/backend_tests/metrics_test.clj
View File

@ -7,6 +7,7 @@
(ns backend-tests.metrics-test
(:require
[app.common.exceptions :as ex]
[app.main :as main]
[app.metrics :as mtx]
[app.metrics.definition :as-alias mdef]
[clojure.test :as t]
@ -163,3 +164,72 @@
(finally
(px/shutdown-now sampler)))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; HISTOGRAM BUCKETS
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(t/deftest histogram-bucket-tails-extend-past-7-5s
;; The sub-7.5s prefix stays identical, so p50/p90 resolution and the
;; history remain comparable; only the tail grows, per latency profile.
(t/is (= mtx/default-histogram-buckets
(subvec mtx/default-request-histogram-buckets 0 14)))
(t/is (= mtx/default-histogram-buckets
(subvec mtx/default-tasks-histogram-buckets 0 14)))
(t/is (= mtx/default-histogram-buckets
(subvec mtx/default-redis-histogram-buckets 0 14)))
(t/is (= [10000 15000 30000 60000]
(subvec mtx/default-request-histogram-buckets 14)))
(t/is (= [10000 15000 30000 60000 120000 300000]
(subvec mtx/default-tasks-histogram-buckets 14)))
(t/is (= [10000]
(subvec mtx/default-redis-histogram-buckets 14))))
(t/deftest default-metrics-wire-bucket-tails-per-profile
;; Requests and waits cover the 30s/60s CLIMIT timeouts, tasks run
;; until done (300s postgres statement timeout), Redis only needs the
;; pathological case covered.
(doseq [id [:rpc-main-timing :rpc-management-timing :rpc-climit-timing
:audit-http-handler-timing :http-server-dispatch-timing
:update-file-validation-timing]]
(t/is (= mtx/default-request-histogram-buckets
(get-in main/default-metrics [id ::mdef/buckets]))
(str "request buckets on " id)))
(t/is (= mtx/default-tasks-histogram-buckets
(get-in main/default-metrics [:tasks-timing ::mdef/buckets])))
(t/is (= mtx/default-redis-histogram-buckets
(get-in main/default-metrics [:redis-eval-timing ::mdef/buckets]))))
(defn- bucket-value
"Cumulative count of the `le` bucket of a histogram, read from the
registry. Nil when that boundary does not exist."
[metrics mname label-names label-values le]
(.getSampleValue ^CollectorRegistry (mtx/get-registry metrics)
(str mname "_bucket")
(into-array String (conj (vec label-names) "le"))
(into-array String (conj (vec label-values) le))))
(t/deftest histogram-tails-are-registered
;; A 9s observation must land in the new tail instead of piling up in
;; +Inf: every histogram used to stop at 7500.
(let [metrics (ig/init-key :app.metrics/metrics {:default main/default-metrics})]
(mtx/run! metrics :id :rpc-main-timing :val 9000 :labels (into-array String ["update-file"]))
(t/is (= 0.0 (bucket-value metrics "penpot_rpc_main_timing" ["name"] ["update-file"] "7500.0")))
(t/is (= 1.0 (bucket-value metrics "penpot_rpc_main_timing" ["name"] ["update-file"] "10000.0")))
(t/is (nil? (bucket-value metrics "penpot_rpc_main_timing" ["name"] ["update-file"] "120000.0")))
(mtx/run! metrics :id :tasks-timing :val 200000 :labels (into-array String ["file-gc"]))
(t/is (= 0.0 (bucket-value metrics "penpot_tasks_timing" ["name"] ["file-gc"] "60000.0")))
(t/is (= 1.0 (bucket-value metrics "penpot_tasks_timing" ["name"] ["file-gc"] "300000.0")))
(mtx/run! metrics :id :redis-eval-timing :val 9000 :labels (into-array String ["claim"]))
(t/is (= 1.0 (bucket-value metrics "penpot_redis_eval_timing" ["name"] ["claim"] "10000.0")))
(t/is (nil? (bucket-value metrics "penpot_redis_eval_timing" ["name"] ["claim"] "15000.0")))
(mtx/run! metrics :id :http-server-dispatch-timing :val 45000)
(t/is (= 1.0 (bucket-value metrics "penpot_http_server_dispatch_timing" [] [] "60000.0")))
(mtx/run! metrics :id :update-file-validation-timing :val 9000 :labels (into-array String ["validate-affected"]))
(t/is (= 0.0 (bucket-value metrics "penpot_rpc_update_file_validation_timing" ["step"] ["validate-affected"] "7500.0")))
(t/is (= 1.0 (bucket-value metrics "penpot_rpc_update_file_validation_timing" ["step"] ["validate-affected"] "10000.0")))))

106
backend/test/backend_tests/rpc_quotes_test.clj
View File

@ -445,6 +445,7 @@
(t/is (true? true) "deduped storage counted once, under quota"))))
(t/deftest media-upload-enforces-storage-quote
;; TEMPORARILY DISABLED (2026-10-19), see body.
(with-mocks [mock {:target 'app.config/get
:return (th/config-get-mock
{:quotes-media-storage-bytes-per-team 100})}]
@ -469,12 +470,104 @@
out (th/multipart-command! params)]
;; 312043 bytes > 100 byte limit → should be rejected
(t/is (not (th/success? out)))
(let [error (:error out)]
(t/is (= :restriction (th/ex-type error)))
(t/is (= :max-quote-reached (th/ex-code error)))
(t/is (= "media-storage-bytes-per-team" (:target (ex-data error))))))))
;; TEMPORARILY DISABLED (2026-10-19): upload path no longer checks
;; media-storage-bytes-per-team, so even over quota the upload
;; succeeds. TODO: restore rejection with cheaper implementation.
(t/is (th/success? out)))))
;; --- SNAPSHOTS (VERSIONS) QUOTES
(defn- check-snapshot-ok
"Save a version of the file and check that it was created."
[profile file label]
(let [out (th/command! {::th/type :create-file-snapshot
::rpc/profile-id (:id profile)
:file-id (:id file)
:label label})]
;; (th/print-result! out)
(t/is (nil? (:error out)))
(t/is (= label (get-in out [:result :label])))
out))
(defn- check-snapshot-ko
"Try to save a version of the file and check that the quota refused it."
[profile file label target]
(let [out (th/command! {::th/type :create-file-snapshot
::rpc/profile-id (:id profile)
:file-id (:id file)
:label label})
error (:error out)]
;; (th/print-result! out)
(t/is (not (th/success? out)))
(t/is (= :restriction (th/ex-type error)))
(t/is (= :max-quote-reached (th/ex-code error)))
(t/is (= target (:target (ex-data error))))
out))
(t/deftest snapshots-per-file-quote
(with-mocks [mock {:target 'app.config/get
:return (th/config-get-mock
{:quotes-snapshots-per-file 2
:quotes-snapshots-per-team 1000})}]
(let [profile (th/create-profile* 1)
project (th/create-project* 1 {:profile-id (:id profile)
:team-id (:default-team-id profile)})
file (th/create-file* 1 {:profile-id (:id profile)
:project-id (:id project)
:is-shared false})]
(check-snapshot-ok profile file "version1")
(check-snapshot-ok profile file "version2")
(check-snapshot-ko profile file "version3" "snapshots-per-file"))))
(t/deftest snapshots-per-team-quote
(with-mocks [mock {:target 'app.config/get
:return (th/config-get-mock
{:quotes-snapshots-per-file 1000
:quotes-snapshots-per-team 3})}]
(let [profile (th/create-profile* 1)
project (th/create-project* 1 {:profile-id (:id profile)
:team-id (:default-team-id profile)})
file-1 (th/create-file* 1 {:profile-id (:id profile)
:project-id (:id project)
:is-shared false})
file-2 (th/create-file* 2 {:profile-id (:id profile)
:project-id (:id project)
:is-shared false})]
;; the quota adds up the versions of every file of the team
(check-snapshot-ok profile file-1 "version1")
(check-snapshot-ok profile file-2 "version2")
(check-snapshot-ok profile file-1 "version3")
(check-snapshot-ko profile file-2 "version4" "snapshots-per-team"))))
(t/deftest snapshots-quote-freed-on-delete
(with-mocks [mock {:target 'app.config/get
:return (th/config-get-mock
{:quotes-snapshots-per-file 2
:quotes-snapshots-per-team 1000})}]
(let [profile (th/create-profile* 1)
project (th/create-project* 1 {:profile-id (:id profile)
:team-id (:default-team-id profile)})
file (th/create-file* 1 {:profile-id (:id profile)
:project-id (:id project)
:is-shared false})
snapshot (:result (check-snapshot-ok profile file "version1"))]
(check-snapshot-ok profile file "version2")
(check-snapshot-ko profile file "version3" "snapshots-per-file")
;; deleting a version takes it out of the count right away
(let [out (th/command! {::th/type :delete-file-snapshot
::rpc/profile-id (:id profile)
:id (:id snapshot)})]
(t/is (nil? (:error out)))
(t/is (true? (:result out)))
(check-snapshot-ok profile file "version4")))))
(t/deftest binfile-jobs-per-profile-quote
(with-mocks [mock {:target 'app.config/get
@ -550,3 +643,4 @@
(t/is (nil? (check! cfg ::quotes/export-jobs-per-profile (:id future))))
(insert! (:id future) "export-assets" "new")
(t/is (some? (error cfg ::quotes/export-jobs-per-profile (:id future)))))))))

82
backend/test/backend_tests/scripts_decode_file_data_test.clj Normal file
View File

@ -0,0 +1,82 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC
(ns backend-tests.scripts-decode-file-data-test
(:require
[app.common.json :as json]
[app.common.uuid :as uuid]
[app.scripts.decode-file-data :as dfd]
[app.util.blob :as blob]
[clojure.java.io :as io]
[clojure.test :as t])
(:import
java.util.Base64))
(defn- sample-file-data
[]
(let [p1 (uuid/next)
p2 (uuid/next)]
{:pages [p1 p2]
:pages-index
{p1 {:id p1 :name "Accueil1" :objects {:a 1 :b 2}}
p2 {:id p2 :name "Contact" :objects {:c 3}}}
:components {(uuid/next) {:name "comp"}}
:colors {(uuid/next) {:name "red"}}
:typographies {}}))
(t/deftest sanitize-converts-special-types
(let [id (uuid/next)
out (dfd/sanitize {:id id
:kw :pages-index
:when (java.util.Date. 0)
:tags #{:a :b}
:raw (.getBytes "hi" "UTF-8")})]
(t/is (= (str id) (get out "id")))
(t/is (= "pages-index" (get out "kw")))
(t/is (string? (get out "when")))
(t/is (= ["a" "b"] (sort (get out "tags"))))
(t/is (= {"$bytes" (.encodeToString (Base64/getEncoder) (.getBytes "hi" "UTF-8"))}
(get out "raw")))))
(t/deftest sanitize-keeps-integers-as-integers
(t/is (= 2 (dfd/sanitize 2)))
(t/is (= 2.5 (dfd/sanitize 2.5))))
(t/deftest blob-roundtrip-all-versions-to-json
(doseq [version [1 3 4 5]]
(let [data (sample-file-data)
encoded (blob/encode data {:version version})
payload (dfd/blob->json encoded)
parsed (json/decode payload)]
(t/is (map? parsed) (str "version " version " produces a JSON map"))
(t/is (= 2 (count (get parsed "pages"))) (str "version " version " keeps pages")))))
(t/deftest summary-lists-pages-in-order-with-object-counts
(let [data (sample-file-data)
summary (dfd/summarize-data data)
pages (:pages summary)]
(t/is (= 2 (count pages)))
(t/is (= ["Accueil1" "Contact"] (mapv :name pages)))
(t/is (= [2 1] (mapv :objects pages)))
(t/is (= {:pages 2 :components 1 :colors 1 :typographies 0}
(:stats summary)))
(let [pages-json (json/decode (dfd/file-data-pages-json data))]
(t/is (= ["Accueil1" "Contact"] (mapv #(get % "name") pages-json))))))
(t/deftest cli-decodes-blob-file-to-json
(let [data (sample-file-data)
tmp-in (java.io.File/createTempFile "blob" ".bin")
tmp-out (java.io.File/createTempFile "out" ".json")
_ (.deleteOnExit tmp-in)
_ (.deleteOnExit tmp-out)
_ (with-open [os (io/output-stream tmp-in)]
(.write os ^bytes (blob/encode data {:version 5})))
_ (dfd/-main "--input" (.getAbsolutePath tmp-in)
"--output" (.getAbsolutePath tmp-out)
"--format" "summary")
parsed (json/decode (slurp tmp-out))]
(t/is (== 2 (get-in parsed ["stats" "pages"])))
(t/is (= ["Accueil1" "Contact"] (mapv #(get % "name") (get parsed "pages"))))))

64
backend/test/backend_tests/storage_gc_deleted_test.clj
View File

@ -10,6 +10,7 @@
[app.common.uuid :as uuid]
[app.db :as db]
[app.storage :as sto]
[app.storage.impl :as impl]
[backend-tests.helpers :as th]
[clojure.test :as t]))
@ -91,3 +92,66 @@
(t/testing "the object of the next chunk is gone"
(t/is (nil? (find-object last-id))))))
(t/deftest failed-batch-is-not-visited-again-in-the-same-run
(let [profile (th/create-profile* 1)
file (th/create-file* 1 {:profile-id (:id profile)
:project-id (:default-project-id profile)})
;; A full first batch of deletable objects.
batch1 (mapv (fn [_] (put-object! (ct/in-past "30m"))) (range chunk-size))
;; The blocked object leads a second full batch, in `deleted_at` order
;; after the first one: the whole batch fails and must be skipped, and
;; the cursor must not come back to any of its objects.
blocked (blocked-object! (:id file))
fillers (mapv (fn [_] (put-object! (ct/in-past "20m"))) (range (dec chunk-size)))
;; Objects after the failed batch: the next read batch starts right
;; after the `(deleted_at, id)` of the last row of the failed one.
batch3 (mapv (fn [_] (put-object! (ct/in-past "5m"))) (range 5))
result (th/run-task! :storage-gc-deleted {})]
(t/testing "the first and the last batches are fully deleted"
(t/is (= (+ (count batch1) (count batch3)) (:deleted result))))
(t/testing "the objects of the first batch are gone"
(t/is (every? #(nil? (find-object (:id %))) batch1)))
(t/testing "the objects of the failed batch are still pending"
(t/is (some? (find-object (:id blocked))))
(t/is (every? #(some? (find-object (:id %))) fillers)))
(t/testing "the objects after the failed batch are gone"
(t/is (every? #(nil? (find-object (:id %))) batch3)))))
(t/deftest physical-failure-rolls-back-the-chunk-and-continues
"Pins the chunk rollback semantics: the physical delete of the first
chunk fails (mocked) after its rows were already deleted inside the
transaction, so the transaction must roll back (the rows stay pending)
while the run skips the failed chunk and deletes the rest."
(let [;; A full first chunk of deletable objects, older in
;; `deleted_at` order than the rest, so the failing physical
;; delete hits exactly this chunk.
old (mapv (fn [_] (put-object! (ct/in-past "30m"))) (range chunk-size))
recent (mapv (fn [_] (put-object! (ct/in-past "5m"))) (range 5))
failing (atom true)
orig impl/del-objects-in-bulk
result (with-redefs [impl/del-objects-in-bulk
(fn [backend ids]
(if @failing
(do (reset! failing false)
(throw (ex-info "simulated backend failure" {})))
(orig backend ids)))]
(th/run-task! :storage-gc-deleted {}))]
(t/testing "the run finishes and only the objects after the failed chunk are deleted"
(t/is (= (count recent) (:deleted result))))
(t/testing "the objects of the failed chunk are still pending"
(t/is (every? #(some? (find-object (:id %))) old)))
(t/testing "the objects after the failed chunk are gone"
(t/is (every? #(nil? (find-object (:id %))) recent)))))

50
backend/test/backend_tests/storage_test.clj
View File

@ -693,9 +693,9 @@
(t/is (= 26 (count ids)))
(th/db-exec! ["update storage_object set deleted_at = ?" (ct/now)])
(let [orig @#'gcd/process-chunk
(let [orig @#'gcd/process-chunk-deletion
calls (atom 0)]
(alter-var-root #'gcd/process-chunk
(alter-var-root #'gcd/process-chunk-deletion
(constantly (fn [& args]
(when (= 2 (swap! calls inc))
(throw (ex-info "boom" {})))
@ -708,7 +708,7 @@
(t/testing "first chunk committed before the fault"
(t/is (= 1 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM storage_object WHERE deleted_at IS NOT NULL"])))))
(finally
(alter-var-root #'gcd/process-chunk (constantly orig)))))))
(alter-var-root #'gcd/process-chunk-deletion (constantly orig)))))))
(t/deftest objects-gc-task-skip-delay
(let [storage (-> (:app.storage/storage th/*system*)
@ -993,25 +993,6 @@
(:id object)])]
(t/is (= 1 (:count row))))))
(t/deftest gc-deleted-gives-up-after-max-attempts
(let [storage (-> (:app.storage/storage th/*system*)
(configure-storage-backend))
content (sto/content "content")
object (sto/put-object! storage {::sto/content content
:content-type "text/plain"})]
(th/db-update! :storage-object {:deleted-at (ct/in-past {:minutes 1})
:deletion_attempts 6}
{:id (:id object)})
(with-mocks [_mock {:target 'app.storage.impl/del-objects-in-bulk
:return (fn [_ ids] (set ids))}]
(let [res (th/run-task! :storage-gc-deleted {})]
(t/is (= 0 (:deleted res)))))
(let [row (th/db-exec-one! ["select count(*) from storage_object where id = ?" (:id object)])]
(t/is (= 0 (:count row))))))
(t/deftest dedup-reuses-existing-blob-with-touch
(let [storage (-> (:app.storage/storage th/*system*)
(configure-storage-backend))
@ -1301,10 +1282,13 @@
(t/is (= 1 (:count (th/db-exec-one! ["select count(*) from upload_session_chunk where session_id = ?"
session-id]))))))
(t/deftest upload-session-deferred-keeps-mapping
;; A physical-deletion failure defers the row; the session mapping is
;; kept so the session survives the backend outage and the blob is
;; retried the next day.
(t/deftest upload-session-mapping-blocks-gc-deleted
;; A live upload-session chunk mapping references its object with a NO
;; ACTION foreign key: gc-deleted cannot purge that object while the
;; session is intact. The blocked chunk fails and is skipped, and both
;; the object row and the mapping stay in place, so the stalled session
;; stays retryable; objects-gc owns purging the mapping before the
;; object (see upload-session-consumed-purge below).
(let [prof (th/create-profile* 1)
_ (th/create-project* 1 {:profile-id (:id prof)
:team-id (:default-team-id prof)})
@ -1327,17 +1311,13 @@
object-id (:object-id (th/db-exec-one! ["select object_id from upload_session_chunk where session_id = ?"
session-id]))]
(t/is (nil? (:error out)))
(th/db-exec! ["update storage_object set deleted_at = ? where id = ?" (ct/now) object-id])
(th/db-exec! ["update storage_object set deleted_at = ? where id = ?" (ct/in-past {:minutes 5}) object-id])
(with-mocks [_mock {:target 'app.storage.impl/del-objects-in-bulk
:return (fn [_ ids] (set ids))}]
(let [res (th/run-task! :storage-gc-deleted {})]
(t/is (= 0 (:deleted res)))))
(th/run-task! :storage-gc-deleted {})
;; row kept and deferred one day
(let [row (th/db-exec-one! ["select deletion_attempts, touched_at from storage_object where id = ?" object-id])]
(t/is (= 1 (:deletion-attempts row)))
(t/is (some? (:touched-at row))))
;; the object row survives the run: the mapping blocks its deletion
(t/is (= 1 (:count (th/db-exec-one! ["select count(*) from storage_object where id = ?"
object-id]))))
;; mapping kept: the session is intact for the retry
(t/is (= 1 (:count (th/db-exec-one! ["select count(*) from upload_session_chunk where session_id = ?"
session-id]))))))

16
backend/test/backend_tests/tasks/binfile_jobs_integration_test.clj
View File

@ -67,14 +67,14 @@
::rpc/profile-id (:id profile)
:total-chunks 1})
:result :session-id)
out (th/command! {::th/type :upload-chunk
::rpc/profile-id (:id profile)
:session-id session-id
:index 0
:content {:filename "package.penpot"
:path (th/tempfile fixture)
:mtype "application/zip"
:size (.length source)}})]
out (th/multipart-command! {::th/type :upload-chunk
::rpc/profile-id (:id profile)
:session-id session-id
:index 0
:content {:filename "package.penpot"
:path (th/tempfile fixture)
:mtype "application/zip"
:size (.length source)}})]
(t/is (nil? (:error out)))
session-id))

12
common/src/app/common/data.cljc
View File

@ -1202,12 +1202,20 @@
(str/replace (str s) #"([*_~`\[\]()>#+=\-|{}.!@\\])" (fn [[_ c]] (str "\\" c)))
""))
;; \p{L} (Unicode letter) works in Java regex natively, but in JavaScript it
;; requires the "u" flag which ClojureScript regex literals don't support.
;; Do not use the "g" flag: a shared global RegExp mutates lastIndex and
;; corrupts later get-initials calls (org avatar lists in the team switcher).
(def ^:private letter-re
#?(:clj #"\p{L}"
:cljs (js/RegExp. "\\p{L}" "u")))
(defn get-initials
"Returns up to two uppercase initials extracted from a string.
Non-letter prefixes in each token are ignored."
Non-letter prefixes in each token are ignored (Unicode letters)."
[name]
(->> (str/split (str/trim (or name "")) #"\s+")
(keep #(first (re-seq #"[a-zA-Z]" %)))
(keep #(re-find letter-re %))
(take 2)
(map str/upper)
(apply str)))

16
common/src/app/common/geom/image_crop.cljc
View File

@ -9,6 +9,7 @@
image fill). The shape is a window over a picture that keeps its size
and position; the window never flips."
(:require
[app.common.geom.matrix :as gmt]
[app.common.math :as mth]))
(defn crop-axis
@ -36,3 +37,18 @@
:y (/ (- (* y height) dy) height')
:width (/ (* w width) width')
:height (/ (* h height) height')}))
(defn preview-image-matrix
"Return the matrix that takes the picture of a `width` x `height` window
from its box under `transform` to its box under `transform'`, both in
the coordinates of the window before the resize. While dragging, the
SVG preview resizes the whole shape with one matrix; this one, applied
to the picture first, undoes that resize so the picture stays in place."
[transform transform' width height]
(let [x (* (get transform :x 0.0) width)
y (* (get transform :y 0.0) height)
x' (* (get transform' :x 0.0) width)
y' (* (get transform' :y 0.0) height)
sx (/ (get transform' :width 1.0) (get transform :width 1.0))
sy (/ (get transform' :height 1.0) (get transform :height 1.0))]
(gmt/matrix sx 0 0 sy (- x' (* x sx)) (- y' (* y sy)))))

147
common/src/app/common/path_names.cljc
View File

@ -118,7 +118,10 @@ Some naming conventions:
[path]
(last (split-path path)))
(defn inside-path? [child parent]
(defn inside-path?
"Returns true when `child` equals `parent` or is nested inside it,
comparing both as path segments (e.g. 'a / b / c' is inside 'a / b')."
[child parent]
(let [child-path (split-path child)
parent-path (split-path parent)]
(and (<= (count parent-path) (count child-path))
@ -135,90 +138,94 @@ Some naming conventions:
;; Tree building functions --------------------------------------------------
"Build tree structure from flat list of paths"
"`build-tree-root` is the main function to build the tree."
"Receives a list of segments with 'name' properties representing paths,
and a separator string."
"E.g segments = [{... :name 'one/two/three'} {... :name 'one/two/four'} {... :name 'one/five'}]"
"Transforms into a tree structure like:
[{:name 'one'
:path 'one'
:depth 0
:leaf nil
:children [{:name 'two'
:path 'one.two'
:depth 1
:leaf nil
:children [{... :name 'three'} {... :name 'four'}]}
{:name 'five'
:path 'one.five'
:depth 1
:leaf {... :name 'five'}
:children nil}]}]"
(defn- sort-by-children
"Sorts segments so that those with children come first."
[segments separator]
(sort-by (fn [segment]
(let [path (split-path (:name segment) :separator separator)
path-length (count path)]
(if (= path-length 1)
1
0)))
segments))
(defn- attach-remaining-path
"Annotates a segment with ::remaining-path: the vector of path segments
still to descend after the current level. Computed once, at the root,
from the segment :name."
[segment separator]
(assoc segment ::remaining-path (split-path (:name segment) :separator separator)))
(defn- group-by-first-segment
"Groups segments by their first path segment and update segment name."
[segments separator]
"Groups segments by the first element of their ::remaining-path and
advances that path by one element.
A segment whose ::remaining-path becomes empty is a token that ends at
the current node; a non-empty one still has descendants below it."
[segments]
(reduce (fn [acc segment]
(let [[first-segment & remaining-segments] (split-path (:name segment) :separator separator)
rest-path (when (seq remaining-segments) (join-path remaining-segments :separator separator :with-spaces? false))]
(update acc first-segment (fnil conj [])
(if rest-path
(assoc segment :name rest-path)
segment))))
(let [path (::remaining-path segment)]
(update acc (first path) (fnil conj [])
(update segment ::remaining-path subvec 1))))
{}
segments))
(defn- sort-and-group-segments
"Sorts elements and groups them by their first path segment."
[segments separator]
(let [sorted (sort-by-children segments separator)
grouped (group-by-first-segment sorted separator)]
(into (sorted-map) grouped)))
"Groups elements by their first path segment, keyed in sorted order."
[segments]
(into (sorted-map) (group-by-first-segment segments)))
(defn- build-tree-node
"Builds a single tree node with computed children."
[segment-name remaining-segments separator parent-path depth]
[segment-name remaining-segments parent-path depth]
(let [current-path (if parent-path
(str parent-path "." segment-name)
segment-name)
is-leaf? (and (seq remaining-segments)
(every? (fn [segment]
(let [remaining-segment-name (first (split-path (:name segment) :separator separator))]
(= segment-name remaining-segment-name)))
remaining-segments))
leaf-segment (some (fn [segment]
(when (empty? (::remaining-path segment))
(-> segment
(assoc :name segment-name)
(dissoc ::remaining-path))))
remaining-segments)
leaf-segment (when is-leaf? (first remaining-segments))
node {:name segment-name
:path current-path
:depth depth
:leaf leaf-segment
:children (when-not is-leaf?
(let [grouped-elements (sort-and-group-segments remaining-segments separator)]
(mapv (fn [[child-segment-name remaining-child-segments]]
(build-tree-node child-segment-name remaining-child-segments separator current-path (inc depth)))
grouped-elements)))}]
node))
child-segments (filter #(seq (::remaining-path %)) remaining-segments)
children (when (seq child-segments)
(let [grouped-elements (sort-and-group-segments child-segments)]
(mapv (fn [[child-segment-name remaining-child-segments]]
(build-tree-node child-segment-name remaining-child-segments current-path (inc depth)))
grouped-elements)))]
{:name segment-name
:path current-path
:depth depth
:leaf leaf-segment
:children children}))
(defn- attach-remaining-paths
"Annotates every segment with its ::remaining-path vector."
[segments separator]
(map #(attach-remaining-path % separator) segments))
(defn- build-root-node
"Builds a root-level tree node from a grouped element."
[[segment-name remaining-segments]]
(build-tree-node segment-name remaining-segments nil 0))
(defn build-tree-root
"Builds the root level of the tree."
"Builds the root level of the tree from a flat list of path segments.
Receives a list of segments with a :name that represents a path, and
the separator used to split each name, e.g.:
[{... :name 'one/two/three'} {... :name 'one/two/four'} {... :name 'one/five'}]
Transforms them into a tree of nodes shaped like:
[{:name 'one'
:path 'one'
:depth 0
:leaf nil
:children [{:name 'two'
:path 'one.two'
:depth 1
:leaf nil
:children [{... :name 'three'} {... :name 'four'}]}
{:name 'five'
:path 'one.five'
:depth 1
:leaf {... :name 'five'}
:children nil}]}]"
[segments separator]
(let [grouped-elements (sort-and-group-segments segments separator)]
(mapv (fn [[segment-name remaining-segments]]
(build-tree-node segment-name remaining-segments separator nil 0))
grouped-elements)))
(let [grouped-elements (sort-and-group-segments (attach-remaining-paths segments separator))]
(mapv build-root-node grouped-elements)))

8
common/src/app/common/types/tokens_lib.cljc
View File

@ -147,7 +147,9 @@
(check-token-attrs)
(map->Token)))
(def ^:private token-separator ".")
(def token-separator
"Separator used to split a token name into its path segments."
".")
(defn get-token-path
[token]
@ -167,9 +169,9 @@
(rename-path node nil new-name))
([node token new-name]
(let [element (if token (:name token) (:path node))
split-path (cpn/split-path element :separator ".")
split-path (cpn/split-path element :separator token-separator)
updated-split-element-name (assoc split-path (:depth node) new-name)
new-element-path (cpn/join-path updated-split-element-name :separator "." :with-spaces? false)]
new-element-path (cpn/join-path updated-split-element-name :separator token-separator :with-spaces? false)]
new-element-path)))
;; === Token Set

13
common/test/common_tests/data_test.cljc
View File

@ -34,7 +34,18 @@
(t/is (= "AB" (d/get-initials "123 Alpha ## beta")))
(t/is (= "PD" (d/get-initials " penpot design tool ")))
(t/is (= "" (d/get-initials nil)))
(t/is (= "" (d/get-initials "!!! ???"))))
(t/is (= "" (d/get-initials "!!! ???")))
;; Accented / Unicode letters (org default avatars)
(t/is (= "Á" (d/get-initials "Ábaco")))
(t/is (= "Ó" (d/get-initials "Órbita")))
(t/is (= "ÁN" (d/get-initials "Álvaro Núñez")))
;; Sequential calls must stay stable (CLJS global RegExp lastIndex)
(t/is (= ["Á" "BB" "JG" "L" "Á"]
(mapv d/get-initials ["Ábaco"
"Black Bird"
"Jean Grey Academy"
"Library"
"Ábaco"]))))
(t/deftest normalize-string-test
;; nil input returns empty string

28
common/test/common_tests/geom_image_crop_test.cljc
View File

@ -7,6 +7,8 @@
(ns common-tests.geom-image-crop-test
(:require
[app.common.geom.image-crop :as gic]
[app.common.geom.matrix :as gmt]
[app.common.geom.point :as gpt]
[app.common.math :as mth]
[clojure.test :as t]))
@ -65,3 +67,29 @@
(t/is (close-vals? [(- dy) height] [dy2 h2]) "window back at its original place")
(t/is (close-vals? ((juxt :x :y :width :height) tf)
((juxt :x :y :width :height) back)))))))
(t/deftest preview-image-matrix-test
(let [width 400
height 300
tf {:x 0 :y -0.2 :width 1 :height 1.4}]
(t/testing "the picture keeps its place once the window is resized"
(doseq [[dx dy w' h'] [[0 60 400 240]
[0 300 400 100]
[-50 75 500 150]]]
(let [tf' (gic/update-image-transform tf width height dx dy w' h')
matrix (gic/preview-image-matrix tf tf' width height)
;; Resize the preview applies to the whole shape: the old
;; window goes to the new one.
window (-> (gmt/matrix)
(gmt/translate (gpt/point dx dy))
(gmt/scale (gpt/point (/ w' width) (/ h' height))))
[x y w h] (picture-box tf [0 0] [width height])
p1 (gpt/transform (gpt/point x y) (gmt/multiply window matrix))
p2 (gpt/transform (gpt/point (+ x w) (+ y h)) (gmt/multiply window matrix))]
(t/is (close-vals? [x y (+ x w) (+ y h)] [(:x p1) (:y p1) (:x p2) (:y p2)])
(str "window " [dx dy w' h'])))))
(t/testing "no change gives the identity"
(t/is (gmt/close? (gmt/matrix) (gic/preview-image-matrix tf tf width height)))
(t/is (gmt/close? (gmt/matrix) (gic/preview-image-matrix nil nil width height))))))

97
common/test/common_tests/path_names_tree_test.cljc Normal file
View File

@ -0,0 +1,97 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns common-tests.path-names-tree-test
(:require
[app.common.files.tokens :as cfo]
[app.common.path-names :as cpn]
[app.common.types.tokens-lib :as ctob]
[clojure.test :as t]))
(defn- single-chain-terminal
"Follows the first child down a single-token tree to its terminal node."
[root]
(loop [node root]
(if (:children node)
(recur (first (:children node)))
node)))
(defn- tree-payloads
"Collects every node and :leaf map in a tree for marker-leak checks."
[node]
(cons node
(concat (when (:leaf node) [(:leaf node)])
(mapcat tree-payloads (:children node)))))
(defn- tree-leaves
"Collects every token leaf from a tree."
[nodes]
(mapcat (fn [node]
(concat (when (:leaf node) [(:leaf node)])
(tree-leaves (:children node))))
nodes))
(t/deftest build-tree-root-preserves-repeated-segments
(t/testing "#11588: adjacent repeated path segments keep their depth"
(let [tree (cpn/build-tree-root [{:name "test.pointer.pointer" :id "t1"}] ".")
terminal (single-chain-terminal (first tree))]
(t/is (= "test" (-> tree first :name)))
(t/is (= "pointer" (-> tree first :children first :name)))
(t/is (nil? (-> tree first :children first :leaf)))
(t/is (= "pointer" (:name terminal)))
(t/is (= "test.pointer.pointer" (:path terminal)))
(t/is (= {:name "pointer" :id "t1"} (:leaf terminal)))
(t/is (nil? (:children terminal)))
(t/is (every? #(not (contains? % :app.common.path-names/remaining-path))
(tree-payloads (first tree))))))
(t/testing "control case without repeated segments is unchanged"
(let [tree (cpn/build-tree-root [{:name "test.thing.other-thing" :id "t2"}] ".")
terminal (single-chain-terminal (first tree))]
(t/is (= "test" (-> tree first :name)))
(t/is (= "thing" (-> tree first :children first :name)))
(t/is (= "other-thing" (:name terminal)))
(t/is (= {:name "other-thing" :id "t2"} (:leaf terminal)))))
(t/testing "repeated and non-repeated siblings coexist"
(let [tree (cpn/build-tree-root [{:name "test.pointer.pointer" :id "t1"}
{:name "test.thing.other-thing" :id "t2"}] ".")
children (-> tree first :children)]
(t/is (= ["pointer" "thing"] (mapv :name children)))
(t/is (= {:name "pointer" :id "t1"} (-> children first :children first :leaf)))))
(t/testing "longer repeated runs keep every depth"
(doseq [[path id] [["test.test.pointer" "t3"] ["test.test.test" "t4"]
["a.a" "t5"] ["a.b.b" "t6"] ["a.a.a" "t8"]]]
(let [segments (cpn/split-path path :separator ".")
root (first (cpn/build-tree-root [{:name path :id id}] "."))
terminal (single-chain-terminal root)]
(t/is (= (last segments) (:name terminal)) path)
(t/is (= (inc (count segments)) (count (tree-payloads root))) path)
(t/is (= {:name (last segments) :id id} (:leaf terminal)) path)))))
(t/deftest build-tree-root-preserves-active-set-ancestor-and-descendant
(t/testing "#11588: an ancestor token and its descendant in the active set are both kept"
(let [tokens-lib (ctob/read-multi-set-dtcg
{"Global" {"a.a" {"$value" "#aaa" "$type" "color"}
"a.a.a" {"$value" "#bbb" "$type" "color"}}
"$themes" []
"$metadata" {"tokenSetOrder" ["Global"]
"activeSets" ["Global"]}})
tokens-status (cfo/make-tokens-status-from-lib tokens-lib)
active-tokens (cfo/get-tokens-in-active-sets tokens-status tokens-lib)
tree (cpn/build-tree-root (vec (vals active-tokens)) ".")
collision (-> tree first :children first)
leaves (tree-leaves tree)]
(t/is (= #{"a.a" "a.a.a"} (set (keys active-tokens))))
(t/is (= 2 (count leaves)))
(t/is (= #{(get-in active-tokens ["a.a" :id])
(get-in active-tokens ["a.a.a" :id])}
(set (map :id leaves))))
(t/is (= "a.a" (:path collision)))
(t/is (= (get-in active-tokens ["a.a" :id])
(get-in collision [:leaf :id])))
(t/is (= (get-in active-tokens ["a.a.a" :id])
(get-in collision [:children 0 :leaf :id])))
(t/is (every? #(not (contains? % :app.common.path-names/remaining-path))
(mapcat tree-payloads tree))))))

2
common/test/common_tests/runner.cljc
View File

@ -80,6 +80,7 @@
[common-tests.math-test]
[common-tests.media-test]
[common-tests.path-names-test]
[common-tests.path-names-tree-test]
[common-tests.record-test]
[common-tests.schema-test]
[common-tests.spec-test]
@ -184,6 +185,7 @@
'common-tests.math-test
'common-tests.media-test
'common-tests.path-names-test
'common-tests.path-names-tree-test
'common-tests.record-test
'common-tests.schema-test
'common-tests.spec-test

2
docs/technical-guide/configuration.md
View File

@ -735,7 +735,7 @@ flags, and by whether `PENPOT_PUBLIC_URI` is served over HTTPS. See
### Database
Penpot only supports PostgreSQL and we highly recommend >=13 version. If you are using official
Penpot only supports PostgreSQL and we highly recommend >=16 version. If you are using official
docker images this is already solved for you.
Essential database configuration:

4
docs/user-guide/first-steps/shortcuts.njk
View File

@ -35,7 +35,9 @@ desc: Get quickstart tips, shortcuts, and tutorials for Penpot! Learn interface
<p class="advice"><strong>Paste</strong> cannot be customized.</p>
<h3 id="customize-shortcuts-layouts">Keyboard layouts</h3>
<p>Punctuation keys (such as <kbd>-</kbd>, <kbd>[</kbd>, <kbd>/</kbd> or <kbd>'</kbd>) are recorded by their physical position on the keyboard, so they work the same on AZERTY, QWERTZ and other layouts. Letters and numbers are recorded by the character they type. With <kbd>Shift</kbd>, the base key is recorded, not the shifted character.</p>
<p>Letters are recorded by the character they type, so they follow your layout (on AZERTY, the <kbd>A</kbd> key records <kbd>a</kbd>). With <kbd>Shift</kbd>, digit keys record the digit itself (<kbd>Shift</kbd><kbd>0</kbd>, not the shifted symbol), on every layout and in every browser.</p>
<p>Punctuation keys (such as <kbd>-</kbd>, <kbd>[</kbd>, <kbd>/</kbd> or <kbd>'</kbd>) are recorded by their physical position on the keyboard, so the same key works on AZERTY, QWERTZ and other layouts — but only in browsers that expose the keyboard layout (Chromium). There, <kbd>Shift</kbd> with the <kbd>+</kbd> key records <kbd>Shift</kbd><kbd>+</kbd>. In browsers without that support, such as Firefox or Safari, punctuation falls back to the typed character (<kbd>Shift</kbd><kbd>*</kbd> instead): those shortcuts still work in the browser where you recorded them, but <kbd>Shift</kbd> combinations with punctuation may stop working if you switch browsers. If that affects you, record those shortcuts in a Chromium browser.</p>
<p>Two known limitations: number keys set opacity (<kbd>0</kbd>–<kbd>9</kbd>) only where digits are typed without <kbd>Shift</kbd>, so AZERTY layouts need the numeric keypad for them; and <kbd>\</kbd> needs <kbd>AltGr</kbd> on some layouts (Spanish), where it is recorded with <kbd>Alt</kbd> and its exact form depends on the browser.</p>
<h3 id="customize-shortcuts-reset">Reset to default</h3>
<p>When you have custom shortcuts, a <strong>Reset all</strong> button appears next to the search field. It lists every shortcut that will go back to its default before you confirm.</p>

5
exporter/src/app/wasm.cljs
View File

@ -287,11 +287,10 @@
(defn render-shape-pdf
"Renders the shape subtree to PDF bytes (Uint8Array)."
[shape-id scale]
[shape-id]
(let [buf (uuid/get-u32 shape-id)]
(-> (h/call wasm/internal-module "_render_shape_pdf"
(aget buf 0) (aget buf 1) (aget buf 2) (aget buf 3)
scale)
(aget buf 0) (aget buf 1) (aget buf 2) (aget buf 3))
(read-render-result LEN-HEADER-BYTES))))
(defn render-shape-svg

2
exporter/src/app/wasm/render.cljs
View File

@ -418,7 +418,7 @@
(defn- render-object-bytes
[type id scale]
(case type
:pdf (let [bytes (wasm/render-shape-pdf id scale)]
:pdf (let [bytes (wasm/render-shape-pdf id)]
(l/dbg :hint "PDF generated via Skia (render-wasm headless)"
:object-id (str id)
:backend "skia-wasm"

19
frontend/packages/mousetrap/README.md Normal file
View File

@ -0,0 +1,19 @@
# @penpot/mousetrap
Vendored keyboard-shortcut library (based on Mousetrap 1.6.5), migrated
from `code`/`keyCode` to `event.key` with lowercase canonical tokens so
shortcuts behave the same across keyboard layouts.
## Tests
Unit suite with the Node standard runner, no browser and no extra
dependencies:
```bash
pnpm test # node --test "test/*.test.js"
```
The suite builds real `Mousetrap` instances over fake targets and
synthetic events, with fake timers for sequences. It runs in the frontend
CI through `frontend/scripts/test` and must stay deterministic (no
network, no real time, no platform reads).

1362
frontend/packages/mousetrap/index.js
View File

File diff suppressed because it is too large Load Diff

3
frontend/packages/mousetrap/package.json
View File

@ -4,6 +4,9 @@
"description": "Simple library for handling keyboard shortcuts",
"main": "index.js",
"type": "module",
"scripts": {
"test": "node --test \"test/*.test.js\""
},
"packageManager": "pnpm@12.14.0+sha512.c18ba6d87993d3d164cb5e8948f32b2c06ee854487e06cd8d86781342fb3e5bb4634367034711e86802baa9c37bea906ce1929ad08d70884c97d801df13389a6",
"author": "Craig Campbell",
"license": "Apache-2.0 WITH LLVM-exception"

183
frontend/packages/mousetrap/test/bind-dispatch.test.js Normal file
View File

@ -0,0 +1,183 @@
import { describe, it } from "node:test";
import assert from "node:assert/strict";
import { freshInstance, fire, makeKeyEvent, makeSpiedEvent } from "./helpers.js";
describe("bind and dispatch", () => {
it("binds a single key", () => {
const { mousetrap, target } = freshInstance();
let calls = 0;
mousetrap.bind("m", () => {
calls++;
});
fire(target, makeKeyEvent({ key: "m", code: "KeyM" }));
assert.equal(calls, 1);
});
it("binds an array of combos to the same callback", () => {
const { mousetrap, target } = freshInstance();
let calls = 0;
const cb = () => {
calls++;
};
mousetrap.bind(["del", "backspace"], cb);
fire(target, makeKeyEvent({ key: "Delete", code: "Delete" }));
fire(target, makeKeyEvent({ key: "Backspace", code: "Backspace" }));
assert.equal(calls, 2);
});
it("supports the plus forms", () => {
const { mousetrap, target } = freshInstance();
let plus = 0;
let shiftPlus = 0;
mousetrap.bind("+", () => {
plus++;
});
mousetrap.bind("shift++", () => {
shiftPlus++;
});
fire(target, makeKeyEvent({ key: "+", code: "Equal" }));
fire(target, makeKeyEvent({ key: "*", code: "BracketRight", shiftKey: true }));
// "*" normalizes by glyph, so "shift++" (shift + "+" key) does not fire;
// this pins the layout-independent behavior for the "+" physical key.
assert.equal(plus, 1);
assert.equal(shiftPlus, 0);
});
it("resolves bind-string aliases to canonical tokens", () => {
const cases = [
["control+z", { key: "z", code: "KeyZ", ctrlKey: true }],
["command+z", { key: "z", code: "KeyZ", metaKey: true }],
["option+q", { key: "q", code: "KeyQ", altKey: true }],
["mod+z", { key: "z", code: "KeyZ", ctrlKey: true }],
["return", { key: "Enter", code: "Enter" }],
["esc", { key: "Escape", code: "Escape" }],
["plus", { key: "+", code: "Equal" }],
];
for (const [combo, eventOver] of cases) {
const { mousetrap, target } = freshInstance();
let calls = 0;
mousetrap.bind(combo, () => {
calls++;
});
fire(target, makeKeyEvent(eventOver));
assert.equal(calls, 1, combo);
}
});
it("defaults to keydown and honors explicit keyup", () => {
const { mousetrap, target } = freshInstance();
let downs = 0;
let ups = 0;
mousetrap.bind("x", () => {
downs++;
});
mousetrap.bind(
"y",
() => {
ups++;
},
"keyup",
);
fire(target, makeKeyEvent({ key: "x", code: "KeyX", type: "keyup" }));
fire(target, makeKeyEvent({ key: "y", code: "KeyY" }));
fire(target, makeKeyEvent({ key: "y", code: "KeyY", type: "keyup" }));
assert.equal(downs, 0);
assert.equal(ups, 1);
});
it("fires a bare modifier on keyup", () => {
const { mousetrap, target } = freshInstance();
let calls = 0;
mousetrap.bind(
"shift",
() => {
calls++;
},
"keyup",
);
fire(target, makeKeyEvent({ key: "Shift", code: "ShiftLeft", type: "keyup", shiftKey: true }));
assert.equal(calls, 1);
});
it("calls preventDefault and stopPropagation when the callback returns false", () => {
const { mousetrap, target } = freshInstance();
mousetrap.bind("ctrl+s", () => false);
const { event, calls } = makeSpiedEvent({ key: "s", code: "KeyS", ctrlKey: true });
fire(target, event);
assert.equal(calls.preventDefault, 1);
assert.equal(calls.stopPropagation, 1);
});
it("reset() clears every binding", () => {
const { mousetrap, target } = freshInstance();
let calls = 0;
mousetrap.bind("a", () => {
calls++;
});
mousetrap.reset();
fire(target, makeKeyEvent({ key: "a", code: "KeyA" }));
assert.equal(calls, 0);
});
it("overwrite replaces the previous callback for the combo", () => {
const { mousetrap, target } = freshInstance();
const seen = [];
mousetrap.bind("ctrl+z", () => {
seen.push("old");
});
mousetrap.bind(
"ctrl+z",
() => {
seen.push("new");
},
undefined,
true,
);
fire(target, makeKeyEvent({ key: "z", code: "KeyZ", ctrlKey: true }));
assert.deepEqual(seen, ["new"]);
});
it("unbind pins the current noop behavior (TODO: really remove)", () => {
// NOTE: unbind() binds an empty function on top instead of removing
// the original, so the original still fires. Pinned as-is; fixing it
// is a behavior change out of scope.
const { mousetrap, target } = freshInstance();
let calls = 0;
mousetrap.bind("a", () => {
calls++;
});
mousetrap.unbind("a");
fire(target, makeKeyEvent({ key: "a", code: "KeyA" }));
assert.equal(calls, 1);
});
it("trigger dispatches a bind registered with an explicit action", () => {
const { mousetrap } = freshInstance();
let calls = 0;
let seenCombo = null;
mousetrap.bind(
"ctrl+q",
(e, combo) => {
calls++;
seenCombo = combo;
},
"keydown",
);
mousetrap.trigger("ctrl+q", "keydown");
assert.equal(calls, 1);
assert.equal(seenCombo, "ctrl+q");
});
it("trigger documents the default-action quirk (TODO)", () => {
// NOTE: binds without an explicit action are stored under
// "combo:undefined", so trigger(combo, "keydown") misses them.
// Pinned as-is; fixing it is a behavior change out of scope.
const { mousetrap } = freshInstance();
let calls = 0;
mousetrap.bind("ctrl+q", () => {
calls++;
});
mousetrap.trigger("ctrl+q", "keydown");
assert.equal(calls, 0);
});
});

89
frontend/packages/mousetrap/test/helpers.js Normal file
View File

@ -0,0 +1,89 @@
import { Mousetrap } from "../index.js";
/**
* Fake event target capturing listeners. The Mousetrap class only needs
* `addEventListener`, so tests run in plain Node without a DOM.
*/
export function makeTarget() {
const listeners = {};
const target = {
listeners,
addEventListener(type, callback) {
listeners[type] = listeners[type] || [];
listeners[type].push(callback);
},
};
return target;
}
/**
* Fake focused element for `stopCallback`. Covers every property the
* implementation reads: dataset, className, tagName, contentEditable.
*/
export function makeElement(over = {}) {
return {
dataset: {},
className: "",
tagName: "DIV",
contentEditable: "false",
...over,
};
}
/**
* Synthetic keyboard event with sane defaults. Override any field per
* case (`key`, `code`, `shiftKey`, ...). The `target` defaults to a plain
* DIV so `stopCallback` lets the event through.
*/
export function makeKeyEvent(over = {}) {
return {
type: "keydown",
key: "a",
code: "KeyA",
shiftKey: false,
ctrlKey: false,
altKey: false,
metaKey: false,
target: makeElement(),
...over,
};
}
/**
* Dispatches a synthetic event through every listener registered for
* its type on a fake target.
*/
export function fire(target, event) {
for (const callback of target.listeners[event.type] || []) {
callback(event);
}
}
/**
* Event spies recording `preventDefault`/`stopPropagation` calls for the
* `callback → false` convention.
*/
export function makeSpiedEvent(over = {}) {
const calls = { preventDefault: 0, stopPropagation: 0 };
const event = makeKeyEvent({
preventDefault() {
calls.preventDefault++;
},
stopPropagation() {
calls.stopPropagation++;
},
...over,
});
return { event, calls };
}
/**
* Fresh isolated instance plus its fake target. Timers used by sequences
* are real by default; tests needing determinism enable `mock.timers`
* themselves and restore it afterwards.
*/
export function freshInstance() {
const target = makeTarget();
const mousetrap = new Mousetrap(target);
return { mousetrap, target };
}

95
frontend/packages/mousetrap/test/layout-map.test.js Normal file
View File

@ -0,0 +1,95 @@
import { describe, it } from "node:test";
import assert from "node:assert/strict";
import { freshInstance, fire, makeKeyEvent } from "./helpers.js";
function firedWithMap(map, bindStr, eventOver) {
const { mousetrap, target } = freshInstance();
mousetrap.setLayoutMap(map);
let calls = 0;
mousetrap.bind(bindStr, () => {
calls++;
});
fire(target, makeKeyEvent(eventOver));
return calls;
}
const spanishMap = {
get(code) {
return code === "BracketRight" ? "+" : undefined;
},
};
describe("layout map", () => {
it("resolves punctuation by position, so Shift+* fires a shift++ bind", () => {
const event = { key: "*", code: "BracketRight", shiftKey: true };
assert.equal(firedWithMap(spanishMap, "shift++", event), 1);
assert.equal(firedWithMap(spanishMap, "shift+*", event), 0);
});
it("matches the unshifted primary on the same position", () => {
const event = { key: "+", code: "BracketRight" };
assert.equal(firedWithMap(spanishMap, "+", event), 1);
});
it("falls back to the glyph when the map misses the code", () => {
const miss = { get: () => undefined };
const event = { key: "*", code: "BracketRight", shiftKey: true };
assert.equal(firedWithMap(miss, "shift+*", event), 1);
assert.equal(firedWithMap(miss, "shift++", event), 0);
});
it("falls back to the glyph when the map throws", () => {
const broken = {
get() {
throw new Error("no map");
},
};
assert.equal(
firedWithMap(broken, "shift+*", { key: "*", code: "BracketRight", shiftKey: true }),
1,
);
});
it("setLayoutMap(null) restores the glyph fallback", () => {
const { mousetrap, target } = freshInstance();
mousetrap.setLayoutMap(spanishMap);
mousetrap.setLayoutMap(null);
let calls = 0;
mousetrap.bind("shift+*", () => {
calls++;
});
fire(target, makeKeyEvent({ key: "*", code: "BracketRight", shiftKey: true }));
assert.equal(calls, 1);
});
it("maps stay per instance", () => {
const a = freshInstance();
const b = freshInstance();
a.mousetrap.setLayoutMap(spanishMap);
let callsA = 0;
let callsB = 0;
a.mousetrap.bind("shift++", () => {
callsA++;
});
b.mousetrap.bind("shift+*", () => {
callsB++;
});
const event = { key: "*", code: "BracketRight", shiftKey: true };
fire(a.target, makeKeyEvent(event));
fire(b.target, makeKeyEvent(event));
assert.equal(callsA, 1);
assert.equal(callsB, 1);
});
it("letters still resolve by glyph with a map set", () => {
const { mousetrap, target } = freshInstance();
mousetrap.setLayoutMap(spanishMap);
let calls = 0;
mousetrap.bind("a", () => {
calls++;
});
// AZERTY: the "A" label sits on the KeyQ position.
fire(target, makeKeyEvent({ key: "a", code: "KeyQ" }));
assert.equal(calls, 1);
});
});

63
frontend/packages/mousetrap/test/modifiers.test.js Normal file
View File

@ -0,0 +1,63 @@
import { describe, it } from "node:test";
import assert from "node:assert/strict";
import { freshInstance, fire, makeKeyEvent } from "./helpers.js";
function fired(bindStr, eventOver) {
const { mousetrap, target } = freshInstance();
let calls = 0;
mousetrap.bind(bindStr, () => {
calls++;
});
fire(target, makeKeyEvent(eventOver));
return calls;
}
describe("modifiers", () => {
it("matches each flag on its own", () => {
assert.equal(fired("shift+x", { key: "x", code: "KeyX", shiftKey: true }), 1);
assert.equal(fired("alt+x", { key: "x", code: "KeyX", altKey: true }), 1);
assert.equal(fired("ctrl+x", { key: "x", code: "KeyX", ctrlKey: true }), 1);
assert.equal(fired("meta+x", { key: "x", code: "KeyX", metaKey: true }), 1);
});
it("does not fire without the bound modifier", () => {
assert.equal(fired("shift+x", { key: "x", code: "KeyX" }), 0);
assert.equal(fired("ctrl+x", { key: "x", code: "KeyX", altKey: true }), 0);
});
it("combines modifiers regardless of bind order", () => {
const event = { key: "q", code: "KeyQ", ctrlKey: true, altKey: true, shiftKey: true };
assert.equal(fired("ctrl+alt+shift+q", event), 1);
assert.equal(fired("shift+ctrl+alt+q", event), 1);
});
it("drops the phantom ctrl while AltGraph is active", () => {
const altgr = {
key: "]",
code: "BracketRight",
ctrlKey: true,
altKey: true,
getModifierState: (mod) => mod === "AltGraph",
};
assert.equal(fired("ctrl+alt+]", altgr), 0);
assert.equal(fired("alt+]", altgr), 1);
});
it("keeps a real ctrl+alt without AltGraph", () => {
const event = {
key: "]",
code: "BracketRight",
ctrlKey: true,
altKey: true,
getModifierState: () => false,
};
assert.equal(fired("ctrl+alt+]", event), 1);
});
it("works when the event has no getModifierState at all", () => {
assert.equal(
fired("ctrl+x", { key: "x", code: "KeyX", ctrlKey: true, getModifierState: undefined }),
1,
);
});
});

109
frontend/packages/mousetrap/test/normalize.test.js Normal file
View File

@ -0,0 +1,109 @@
import { describe, it } from "node:test";
import assert from "node:assert/strict";
import { freshInstance, fire, makeKeyEvent } from "./helpers.js";
function fired(bindStr, eventOver) {
const { mousetrap, target } = freshInstance();
let calls = 0;
mousetrap.bind(bindStr, () => {
calls++;
});
fire(target, makeKeyEvent(eventOver));
return calls;
}
describe("canonical key normalization", () => {
// [native event.key, code, canonical bind] — every _KEY_NAMES entry.
const table = [
[" ", "Space", "space"],
["Spacebar", "Space", "space"],
["ArrowUp", "ArrowUp", "up"],
["Up", "ArrowUp", "up"],
["ArrowDown", "ArrowDown", "down"],
["Down", "ArrowDown", "down"],
["ArrowLeft", "ArrowLeft", "left"],
["Left", "ArrowLeft", "left"],
["ArrowRight", "ArrowRight", "right"],
["Right", "ArrowRight", "right"],
["Escape", "Escape", "escape"],
["Esc", "Escape", "escape"],
["Enter", "Enter", "enter"],
["Backspace", "Backspace", "backspace"],
["Delete", "Delete", "del"],
["Del", "Delete", "del"],
["Insert", "Insert", "ins"],
["Ins", "Insert", "ins"],
["Tab", "Tab", "tab"],
["Home", "Home", "home"],
["End", "End", "end"],
["PageUp", "PageUp", "pageup"],
["PageDown", "PageDown", "pagedown"],
["CapsLock", "CapsLock", "capslock"],
];
for (const [nativeKey, code, canonical] of table) {
it(`maps ${nativeKey} to ${canonical}`, () => {
assert.equal(fired(canonical, { key: nativeKey, code }), 1);
});
}
it("lowercases single characters so Shift+Z matches shift+z", () => {
assert.equal(fired("shift+z", { key: "Z", code: "KeyZ", shiftKey: true }), 1);
assert.equal(fired("a", { key: "A", code: "KeyA" }), 1);
});
it("resolves a shifted digit row key to the digit itself", () => {
// US Shift+0 types ")", AZERTY Shift+0 types "0": shift+0 must mean
// the physical digit key on every layout (production keyCode behavior).
assert.equal(fired("shift+0", { key: ")", code: "Digit0", shiftKey: true }), 1);
assert.equal(fired("shift+1", { key: "!", code: "Digit1", shiftKey: true }), 1);
assert.equal(fired("shift+2", { key: "@", code: "Digit2", shiftKey: true }), 1);
assert.equal(fired("shift+8", { key: "*", code: "Digit8", shiftKey: true }), 1);
// ...so the shifted glyphs are not bindable tokens for digit codes.
assert.equal(fired("shift+)", { key: ")", code: "Digit0", shiftKey: true }), 0);
assert.equal(fired("shift+*", { key: "*", code: "Digit8", shiftKey: true }), 0);
// AZERTY shifted digits already type the digit: unchanged.
assert.equal(fired("shift+0", { key: "0", code: "Digit0", shiftKey: true }), 1);
// Unshifted digits are untouched.
assert.equal(fired("0", { key: "0", code: "Digit0" }), 1);
assert.equal(fired("&", { key: "&", code: "Digit1" }), 1);
});
it("without a layout map, punctuation resolves by glyph", () => {
// Spanish layout: Shift + the physical "+" key (BracketRight) types "*".
assert.equal(fired("shift+*", { key: "*", code: "BracketRight", shiftKey: true }), 1);
assert.equal(fired("shift+]", { key: "*", code: "BracketRight", shiftKey: true }), 0);
assert.equal(fired("+", { key: "+", code: "BracketRight" }), 1);
// US layout: Shift + BracketRight types "}".
assert.equal(fired("shift+}", { key: "}", code: "BracketRight", shiftKey: true }), 1);
assert.equal(fired("shift+]", { key: "}", code: "BracketRight", shiftKey: true }), 0);
assert.equal(fired("?", { key: "?", code: "Slash" }), 1);
});
it("lowercases function keys", () => {
assert.equal(fired("f1", { key: "F1", code: "F1" }), 1);
assert.equal(fired("f13", { key: "F13", code: "F13" }), 1);
});
it("ignores Dead keys and missing/empty keys without throwing", () => {
assert.equal(fired("dead", { key: "Dead", code: "Quote" }), 0);
assert.equal(fired("a", { key: undefined, code: "KeyA" }), 0);
assert.equal(fired("a", { key: "", code: "KeyA" }), 0);
});
it("keeps numpad digits positional as num0..num9", () => {
assert.equal(fired("num0", { key: "0", code: "Numpad0" }), 1);
assert.equal(fired("num9", { key: "9", code: "Numpad9" }), 1);
// NumLock off: the key says Insert but the position wins.
assert.equal(fired("num0", { key: "Insert", code: "Numpad0" }), 1);
});
it("falls back to the glyph for non-digit numpad codes", () => {
assert.equal(fired("+", { key: "+", code: "NumpadAdd" }), 1);
});
it("lowercases unknown long keys", () => {
assert.equal(fired("foo", { key: "Foo", code: "Foo" }), 1);
assert.equal(fired("ñ", { key: "Ñ", code: "Semicolon" }), 1);
});
});

108
frontend/packages/mousetrap/test/sequences.test.js Normal file
View File

@ -0,0 +1,108 @@
import { describe, it, beforeEach, afterEach, mock } from "node:test";
import assert from "node:assert/strict";
import { freshInstance, fire, makeKeyEvent } from "./helpers.js";
function down(over) {
return makeKeyEvent({ type: "keydown", ...over });
}
function up(over) {
return makeKeyEvent({ type: "keyup", ...over });
}
describe("sequences", () => {
beforeEach(() => {
mock.timers.enable({ apis: ["setTimeout"] });
});
afterEach(() => {
mock.timers.reset();
});
it("fires after the full sequence, not on the prefix alone", () => {
const { mousetrap, target } = freshInstance();
let calls = 0;
mousetrap.bind("g v", () => {
calls++;
});
fire(target, down({ key: "g", code: "KeyG" }));
assert.equal(calls, 0);
fire(target, down({ key: "v", code: "KeyV" }));
assert.equal(calls, 1);
});
it("advances every sequence sharing a prefix", () => {
const { mousetrap, target } = freshInstance();
const seen = [];
mousetrap.bind("g i", () => {
seen.push("i");
});
mousetrap.bind("g t", () => {
seen.push("t");
});
fire(target, down({ key: "g", code: "KeyG" }));
fire(target, down({ key: "t", code: "KeyT" }));
assert.deepEqual(seen, ["t"]);
});
it("resets on a wrong key mid-sequence", () => {
const { mousetrap, target } = freshInstance();
let calls = 0;
mousetrap.bind("g v", () => {
calls++;
});
fire(target, down({ key: "g", code: "KeyG" }));
fire(target, down({ key: "e", code: "KeyE" }));
fire(target, down({ key: "v", code: "KeyV" }));
assert.equal(calls, 0);
// and the sequence works again afterwards
fire(target, down({ key: "g", code: "KeyG" }));
fire(target, down({ key: "v", code: "KeyV" }));
assert.equal(calls, 1);
});
it("expires after one second without the next key", () => {
const { mousetrap, target } = freshInstance();
let calls = 0;
mousetrap.bind("g v", () => {
calls++;
});
fire(target, down({ key: "g", code: "KeyG" }));
mock.timers.tick(1001);
fire(target, down({ key: "v", code: "KeyV" }));
assert.equal(calls, 0);
});
it("supports modifiers inside sequences", () => {
const { mousetrap, target } = freshInstance();
let calls = 0;
mousetrap.bind("ctrl+k u", () => {
calls++;
});
fire(target, down({ key: "k", code: "KeyK", ctrlKey: true }));
fire(target, down({ key: "u", code: "KeyU" }));
assert.equal(calls, 1);
});
it("supports keyup sequences and ignores the final keyup", () => {
const { mousetrap, target } = freshInstance();
const seen = [];
mousetrap.bind(
"g v",
() => {
seen.push("seq");
},
"keyup",
);
mousetrap.bind(
"v",
() => {
seen.push("keyup-v");
},
"keyup",
);
fire(target, up({ key: "g", code: "KeyG" }));
fire(target, up({ key: "v", code: "KeyV" }));
assert.deepEqual(seen, ["seq"]);
});
});

31
frontend/packages/mousetrap/test/smoke.test.js Normal file
View File

@ -0,0 +1,31 @@
import { describe, it } from "node:test";
import assert from "node:assert/strict";
import { freshInstance, fire, makeKeyEvent } from "./helpers.js";
describe("smoke", () => {
it("fires a bound combo and ignores it without the modifier", () => {
const { mousetrap, target } = freshInstance();
let calls = 0;
mousetrap.bind("ctrl+z", () => {
calls++;
});
fire(target, makeKeyEvent({ key: "z", code: "KeyZ", ctrlKey: true }));
fire(target, makeKeyEvent({ key: "z", code: "KeyZ" }));
assert.equal(calls, 1);
});
it("reset() leaves the instance deaf", () => {
const { mousetrap, target } = freshInstance();
let calls = 0;
mousetrap.bind("ctrl+z", () => {
calls++;
});
mousetrap.reset();
fire(target, makeKeyEvent({ key: "z", code: "KeyZ", ctrlKey: true }));
assert.equal(calls, 0);
});
});

102
frontend/packages/mousetrap/test/stop-callback.test.js Normal file
View File

@ -0,0 +1,102 @@
import { describe, it } from "node:test";
import assert from "node:assert/strict";
import { Mousetrap } from "../index.js";
import {
freshInstance,
fire,
makeKeyEvent,
makeElement,
} from "./helpers.js";
describe("stopCallback", () => {
function stopped(elementOver, eventOver = {}, combo = "ctrl+z") {
const { mousetrap } = freshInstance();
const event = makeKeyEvent({ target: makeElement(elementOver), ...eventOver });
return mousetrap.stopCallback(event, event.target, combo);
}
it("stops inside form fields", () => {
for (const tagName of ["INPUT", "SELECT", "TEXTAREA"]) {
assert.equal(stopped({ tagName }), true, tagName);
}
});
it("stops buttons only for tab combos, tolerating a missing combo", () => {
assert.equal(stopped({ tagName: "BUTTON" }, {}, "shift+tab"), true);
assert.equal(stopped({ tagName: "BUTTON" }, {}, "ctrl+z"), false);
assert.equal(
stopped({ tagName: "BUTTON" }, {}, undefined),
false,
);
});
it("stops editable content", () => {
assert.equal(stopped({ contentEditable: "true" }), true);
assert.equal(stopped({ contentEditable: "plaintext-only" }), true);
assert.equal(stopped({ contentEditable: "false" }), false);
assert.equal(stopped({ contentEditable: "inherit" }), false);
});
it("lets events through on mousetrap-marked elements", () => {
assert.equal(stopped({ className: "mousetrap" }), false);
assert.equal(stopped({ className: "foo mousetrap bar" }), false);
assert.equal(stopped({ dataset: { mousetrapDontStop: "" } }), false);
});
it("always lets keyup through", () => {
assert.equal(stopped({ tagName: "INPUT" }, { type: "keyup" }), false);
});
it("retargets through composedPath in open shadow trees", () => {
const inner = makeElement({ tagName: "DIV" });
const event = makeKeyEvent({
target: makeElement({ tagName: "INPUT" }),
composedPath: () => [inner],
});
const { mousetrap } = freshInstance();
assert.equal(mousetrap.stopCallback(event, event.target, "ctrl+z"), false);
});
it("gates dispatch end to end: typing in an input fires nothing", () => {
const { mousetrap, target } = freshInstance();
let calls = 0;
mousetrap.bind("ctrl+z", () => {
calls++;
});
fire(
target,
makeKeyEvent({
key: "z",
code: "KeyZ",
ctrlKey: true,
target: makeElement({ tagName: "INPUT" }),
}),
);
assert.equal(calls, 0);
});
});
describe("exports", () => {
it("exposes the shared default instance", async () => {
const mod = await import("../index.js");
assert.equal(typeof mod.default.bind, "function");
assert.equal(typeof mod.default.reset, "function");
});
it("constructs without a target in Node: no listeners, no throw", () => {
const mousetrap = new Mousetrap();
assert.deepEqual(mousetrap._callbacks, {});
});
it("wires keydown and keyup on the given target", async () => {
const { Mousetrap: Klass } = await import("../index.js");
const seen = [];
const target = {
addEventListener(type) {
seen.push(type);
},
};
new Klass(target);
assert.deepEqual(seen, ["keydown", "keyup"]);
});
});

727
frontend/playwright/data/text-editor/get-file-12325.json Normal file
View File

@ -0,0 +1,727 @@
{
"~:features": {
"~#set": [
"fdata/path-data",
"plugins/runtime",
"design-tokens/v1",
"layout/grid",
"styles/v2",
"fdata/pointer-map",
"fdata/objects-map",
"components/v2",
"fdata/shape-data-type",
"text-editor/v2"
]
},
"~:team-id": "~u9e6e22b2-db76-81d6-8006-75d7cdbb8bad",
"~:permissions": {
"~:type": "~:membership",
"~:is-owner": true,
"~:is-admin": true,
"~:can-edit": true,
"~:can-read": true,
"~:is-logged": true
},
"~:has-media-trimmed": false,
"~:comment-thread-seqn": 0,
"~:name": "Bug 12325",
"~:revn": 3,
"~:modified-at": "~m1753957736516",
"~:vern": 0,
"~:id": "~u238a17e0-75ff-8075-8006-934586ea2230",
"~:is-shared": false,
"~:migrations": {
"~#ordered-set": [
"legacy-2",
"legacy-3",
"legacy-5",
"legacy-6",
"legacy-7",
"legacy-8",
"legacy-9",
"legacy-10",
"legacy-11",
"legacy-12",
"legacy-13",
"legacy-14",
"legacy-16",
"legacy-17",
"legacy-18",
"legacy-19",
"legacy-25",
"legacy-26",
"legacy-27",
"legacy-28",
"legacy-29",
"legacy-31",
"legacy-32",
"legacy-33",
"legacy-34",
"legacy-36",
"legacy-37",
"legacy-38",
"legacy-39",
"legacy-40",
"legacy-41",
"legacy-42",
"legacy-43",
"legacy-44",
"legacy-45",
"legacy-46",
"legacy-47",
"legacy-48",
"legacy-49",
"legacy-50",
"legacy-51",
"legacy-52",
"legacy-53",
"legacy-54",
"legacy-55",
"legacy-56",
"legacy-57",
"legacy-59",
"legacy-62",
"legacy-65",
"legacy-66",
"legacy-67",
"0001-remove-tokens-from-groups",
"0002-normalize-bool-content-v2",
"0002-clean-shape-interactions",
"0003-fix-root-shape",
"0003-convert-path-content-v2",
"0004-clean-shadow-color",
"0005-deprecate-image-type",
"0006-fix-old-texts-fills",
"0007-clear-invalid-strokes-and-fills-v2",
"0008-fix-library-colors-v4",
"0009-clean-library-colors",
"0009-add-partial-text-touched-flags"
]
},
"~:version": 67,
"~:project-id": "~u9e6e22b2-db76-81d6-8006-75d7cdc30669",
"~:created-at": "~m1753957644225",
"~:data": {
"~:pages": ["~u238a17e0-75ff-8075-8006-934586ea2231"],
"~:pages-index": {
"~u238a17e0-75ff-8075-8006-934586ea2231": {
"~:objects": {
"~u00000000-0000-0000-0000-000000000000": {
"~#shape": {
"~:y": 0,
"~:hide-fill-on-export": false,
"~:transform": {
"~#matrix": {
"~:a": 1.0,
"~:b": 0.0,
"~:c": 0.0,
"~:d": 1.0,
"~:e": 0.0,
"~:f": 0.0
}
},
"~:rotation": 0,
"~:name": "Root Frame",
"~:width": 0.01,
"~:type": "~:frame",
"~:points": [
{
"~#point": {
"~:x": 0.0,
"~:y": 0.0
}
},
{
"~#point": {
"~:x": 0.01,
"~:y": 0.0
}
},
{
"~#point": {
"~:x": 0.01,
"~:y": 0.01
}
},
{
"~#point": {
"~:x": 0.0,
"~:y": 0.01
}
}
],
"~:r2": 0,
"~:proportion-lock": false,
"~:transform-inverse": {
"~#matrix": {
"~:a": 1.0,
"~:b": 0.0,
"~:c": 0.0,
"~:d": 1.0,
"~:e": 0.0,
"~:f": 0.0
}
},
"~:r3": 0,
"~:r1": 0,
"~:id": "~u00000000-0000-0000-0000-000000000000",
"~:parent-id": "~u00000000-0000-0000-0000-000000000000",
"~:frame-id": "~u00000000-0000-0000-0000-000000000000",
"~:strokes": [],
"~:x": 0,
"~:proportion": 1.0,
"~:r4": 0,
"~:selrect": {
"~#rect": {
"~:x": 0,
"~:y": 0,
"~:width": 0.01,
"~:height": 0.01,
"~:x1": 0,
"~:y1": 0,
"~:x2": 0.01,
"~:y2": 0.01
}
},
"~:fills": [
{
"~:fill-color": "#FFFFFF",
"~:fill-opacity": 1
}
],
"~:flip-x": null,
"~:height": 0.01,
"~:flip-y": null,
"~:shapes": [
"~uc0a701d6-7df9-80ac-8008-c9e6c5b7f7f4",
"~uc0a701d6-7df9-80ac-8008-c9e6c8b2a9e2"
]
}
},
"~uc0a701d6-7df9-80ac-8008-c9e6c5b7f7f4": {
"~#shape": {
"~:id": "~uc0a701d6-7df9-80ac-8008-c9e6c5b7f7f4",
"~:name": "Text board",
"~:type": "~:frame",
"~:parent-id": "~u00000000-0000-0000-0000-000000000000",
"~:frame-id": "~u00000000-0000-0000-0000-000000000000",
"~:shapes": ["~uc0a701d6-7df9-80ac-8008-c9e6c47fbce4"],
"~:fills": [
{
"~:fill-color": "#B1B2B5",
"~:fill-opacity": 1
}
],
"~:strokes": [],
"~:hide-fill-on-export": false,
"~:show-content": true,
"~:proportion-lock": false,
"~:proportion": 1,
"~:r1": 0,
"~:r2": 0,
"~:r3": 0,
"~:r4": 0,
"~:layout": "~:flex",
"~:layout-flex-dir": "~:row",
"~:layout-wrap-type": "~:nowrap",
"~:layout-gap-type": "~:multiple",
"~:layout-gap": {
"~:row-gap": 0,
"~:column-gap": 0
},
"~:layout-padding-type": "~:simple",
"~:layout-padding": {
"~:p1": 0,
"~:p2": 0,
"~:p3": 0,
"~:p4": 0
},
"~:layout-align-items": "~:start",
"~:layout-justify-content": "~:start",
"~:layout-align-content": "~:stretch",
"~:layout-item-h-sizing": "~:auto",
"~:layout-item-v-sizing": "~:auto",
"~:x": 0,
"~:y": 0,
"~:width": 99.99999904632568,
"~:height": 17,
"~:transform": {
"~#matrix": {
"~:a": 1,
"~:b": 0,
"~:c": 0,
"~:d": 1,
"~:e": 0,
"~:f": 0
}
},
"~:transform-inverse": {
"~#matrix": {
"~:a": 1,
"~:b": 0,
"~:c": 0,
"~:d": 1,
"~:e": 0,
"~:f": 0
}
},
"~:rotation": 0,
"~:points": [
{
"~#point": {
"~:x": 0,
"~:y": 0
}
},
{
"~#point": {
"~:x": 99.99999904632568,
"~:y": 0
}
},
{
"~#point": {
"~:x": 99.99999904632568,
"~:y": 17
}
},
{
"~#point": {
"~:x": 0,
"~:y": 17
}
}
],
"~:selrect": {
"~#rect": {
"~:x": 0,
"~:y": 0,
"~:width": 99.99999904632568,
"~:height": 17,
"~:x1": 0,
"~:y1": 0,
"~:x2": 99.99999904632568,
"~:y2": 17
}
}
}
},
"~uc0a701d6-7df9-80ac-8008-c9e6c47fbce4": {
"~#shape": {
"~:id": "~uc0a701d6-7df9-80ac-8008-c9e6c47fbce4",
"~:name": "Hello wide world",
"~:type": "~:text",
"~:grow-type": "~:auto-width",
"~:parent-id": "~uc0a701d6-7df9-80ac-8008-c9e6c5b7f7f4",
"~:frame-id": "~uc0a701d6-7df9-80ac-8008-c9e6c5b7f7f4",
"~:content": {
"~:type": "root",
"~:children": [
{
"~:type": "paragraph-set",
"~:children": [
{
"~:type": "paragraph",
"~:children": [
{
"~:line-height": "1.2",
"~:font-style": "normal",
"~:typography-ref-id": null,
"~:text-transform": "none",
"~:text-align": "left",
"~:font-id": "sourcesanspro",
"~:font-size": "14",
"~:font-weight": "400",
"~:typography-ref-file": null,
"~:text-direction": "ltr",
"~:font-variant-id": "regular",
"~:text-decoration": "none",
"~:letter-spacing": "0",
"~:fills": [
{
"~:fill-color": "#000000",
"~:fill-opacity": 1
}
],
"~:font-family": "sourcesanspro",
"~:text": "Hello wide world"
}
]
}
]
}
]
},
"~:x": 0,
"~:y": 0,
"~:width": 99.99999904632568,
"~:height": 17,
"~:transform": {
"~#matrix": {
"~:a": 1,
"~:b": 0,
"~:c": 0,
"~:d": 1,
"~:e": 0,
"~:f": 0
}
},
"~:transform-inverse": {
"~#matrix": {
"~:a": 1,
"~:b": 0,
"~:c": 0,
"~:d": 1,
"~:e": 0,
"~:f": 0
}
},
"~:rotation": 0,
"~:points": [
{
"~#point": {
"~:x": 0,
"~:y": 0
}
},
{
"~#point": {
"~:x": 99.99999904632568,
"~:y": 0
}
},
{
"~#point": {
"~:x": 99.99999904632568,
"~:y": 17
}
},
{
"~#point": {
"~:x": 0,
"~:y": 17
}
}
],
"~:selrect": {
"~#rect": {
"~:x": 0,
"~:y": 0,
"~:width": 99.99999904632568,
"~:height": 17,
"~:x1": 0,
"~:y1": 0,
"~:x2": 99.99999904632568,
"~:y2": 17
}
}
}
},
"~uc0a701d6-7df9-80ac-8008-c9e6c8b2a9e2": {
"~#shape": {
"~:id": "~uc0a701d6-7df9-80ac-8008-c9e6c8b2a9e2",
"~:name": "Group board",
"~:type": "~:frame",
"~:parent-id": "~u00000000-0000-0000-0000-000000000000",
"~:frame-id": "~u00000000-0000-0000-0000-000000000000",
"~:shapes": ["~uc0a701d6-7df9-80ac-8008-c9e6ca7a304a"],
"~:fills": [
{
"~:fill-color": "#B1B2B5",
"~:fill-opacity": 1
}
],
"~:strokes": [],
"~:hide-fill-on-export": false,
"~:show-content": true,
"~:proportion-lock": false,
"~:proportion": 1,
"~:r1": 0,
"~:r2": 0,
"~:r3": 0,
"~:r4": 0,
"~:layout": "~:flex",
"~:layout-flex-dir": "~:row",
"~:layout-wrap-type": "~:nowrap",
"~:layout-gap-type": "~:multiple",
"~:layout-gap": {
"~:row-gap": 0,
"~:column-gap": 0
},
"~:layout-padding-type": "~:simple",
"~:layout-padding": {
"~:p1": 0,
"~:p2": 0,
"~:p3": 0,
"~:p4": 0
},
"~:layout-align-items": "~:start",
"~:layout-justify-content": "~:start",
"~:layout-align-content": "~:stretch",
"~:layout-item-h-sizing": "~:auto",
"~:layout-item-v-sizing": "~:auto",
"~:x": 0,
"~:y": 100,
"~:width": 120.99999821186066,
"~:height": 17,
"~:transform": {
"~#matrix": {
"~:a": 1,
"~:b": 0,
"~:c": 0,
"~:d": 1,
"~:e": 0,
"~:f": 0
}
},
"~:transform-inverse": {
"~#matrix": {
"~:a": 1,
"~:b": 0,
"~:c": 0,
"~:d": 1,
"~:e": 0,
"~:f": 0
}
},
"~:rotation": 0,
"~:points": [
{
"~#point": {
"~:x": 0,
"~:y": 100
}
},
{
"~#point": {
"~:x": 120.99999821186066,
"~:y": 100
}
},
{
"~#point": {
"~:x": 120.99999821186066,
"~:y": 117
}
},
{
"~#point": {
"~:x": 0,
"~:y": 117
}
}
],
"~:selrect": {
"~#rect": {
"~:x": 0,
"~:y": 100,
"~:width": 120.99999821186066,
"~:height": 17,
"~:x1": 0,
"~:y1": 100,
"~:x2": 120.99999821186066,
"~:y2": 117
}
}
}
},
"~uc0a701d6-7df9-80ac-8008-c9e6ca7a304a": {
"~#shape": {
"~:id": "~uc0a701d6-7df9-80ac-8008-c9e6ca7a304a",
"~:name": "Group",
"~:type": "~:group",
"~:parent-id": "~uc0a701d6-7df9-80ac-8008-c9e6c8b2a9e2",
"~:frame-id": "~uc0a701d6-7df9-80ac-8008-c9e6c8b2a9e2",
"~:shapes": ["~uc0a701d6-7df9-80ac-8008-c9e6c781a50f"],
"~:fills": [],
"~:strokes": [],
"~:proportion-lock": false,
"~:proportion": 1,
"~:x": 0,
"~:y": 100,
"~:width": 120.99999821186066,
"~:height": 17,
"~:transform": {
"~#matrix": {
"~:a": 1,
"~:b": 0,
"~:c": 0,
"~:d": 1,
"~:e": 0,
"~:f": 0
}
},
"~:transform-inverse": {
"~#matrix": {
"~:a": 1,
"~:b": 0,
"~:c": 0,
"~:d": 1,
"~:e": 0,
"~:f": 0
}
},
"~:rotation": 0,
"~:points": [
{
"~#point": {
"~:x": 0,
"~:y": 100
}
},
{
"~#point": {
"~:x": 120.99999821186066,
"~:y": 100
}
},
{
"~#point": {
"~:x": 120.99999821186066,
"~:y": 117
}
},
{
"~#point": {
"~:x": 0,
"~:y": 117
}
}
],
"~:selrect": {
"~#rect": {
"~:x": 0,
"~:y": 100,
"~:width": 120.99999821186066,
"~:height": 17,
"~:x1": 0,
"~:y1": 100,
"~:x2": 120.99999821186066,
"~:y2": 117
}
}
}
},
"~uc0a701d6-7df9-80ac-8008-c9e6c781a50f": {
"~#shape": {
"~:id": "~uc0a701d6-7df9-80ac-8008-c9e6c781a50f",
"~:name": "Hello grouped world",
"~:type": "~:text",
"~:grow-type": "~:auto-width",
"~:parent-id": "~uc0a701d6-7df9-80ac-8008-c9e6ca7a304a",
"~:frame-id": "~uc0a701d6-7df9-80ac-8008-c9e6c8b2a9e2",
"~:content": {
"~:type": "root",
"~:children": [
{
"~:type": "paragraph-set",
"~:children": [
{
"~:type": "paragraph",
"~:children": [
{
"~:line-height": "1.2",
"~:font-style": "normal",
"~:typography-ref-id": null,
"~:text-transform": "none",
"~:text-align": "left",
"~:font-id": "sourcesanspro",
"~:font-size": "14",
"~:font-weight": "400",
"~:typography-ref-file": null,
"~:text-direction": "ltr",
"~:font-variant-id": "regular",
"~:text-decoration": "none",
"~:letter-spacing": "0",
"~:fills": [
{
"~:fill-color": "#000000",
"~:fill-opacity": 1
}
],
"~:font-family": "sourcesanspro",
"~:text": "Hello grouped world"
}
]
}
]
}
]
},
"~:x": 0,
"~:y": 100,
"~:width": 120.99999821186066,
"~:height": 17,
"~:transform": {
"~#matrix": {
"~:a": 1,
"~:b": 0,
"~:c": 0,
"~:d": 1,
"~:e": 0,
"~:f": 0
}
},
"~:transform-inverse": {
"~#matrix": {
"~:a": 1,
"~:b": 0,
"~:c": 0,
"~:d": 1,
"~:e": 0,
"~:f": 0
}
},
"~:rotation": 0,
"~:points": [
{
"~#point": {
"~:x": 0,
"~:y": 100
}
},
{
"~#point": {
"~:x": 120.99999821186066,
"~:y": 100
}
},
{
"~#point": {
"~:x": 120.99999821186066,
"~:y": 117
}
},
{
"~#point": {
"~:x": 0,
"~:y": 117
}
}
],
"~:selrect": {
"~#rect": {
"~:x": 0,
"~:y": 100,
"~:width": 120.99999821186066,
"~:height": 17,
"~:x1": 0,
"~:y1": 100,
"~:x2": 120.99999821186066,
"~:y2": 117
}
}
}
}
},
"~:id": "~u238a17e0-75ff-8075-8006-934586ea2231",
"~:name": "Page 1"
}
},
"~:id": "~u238a17e0-75ff-8075-8006-934586ea2230",
"~:options": {
"~:components-v2": true,
"~:base-font-size": "16px"
}
}
}

74
frontend/playwright/ui/specs/text-editor-v3.spec.js
View File

@ -23,7 +23,6 @@ async function openEditorAndSelectAll(workspace) {
await workspace.page.keyboard.press("ControlOrMeta+a");
}
test("Typography at a collapsed caret only styles newly typed text", async ({
page,
}) => {
@ -218,8 +217,6 @@ test.describe("BUG 11083 - Changing typography must not quit the editor", () =>
});
});
test("BUG 10467 - Auto-width text captures every typed character", async ({
page,
}) => {
@ -681,3 +678,74 @@ test.describe("Text Editor context menu", () => {
await expect(menuItem(page, "Select all")).not.toHaveAttribute("disabled");
});
});
test.describe("BUG 12325 - Fit-content flex board follows an auto-width text while typing", () => {
// Two grey fit-content flex boards, one per test. The content
// (0, 0)-(121, 117) loads centred in the viewport at zoom 1.
const CONTENT_CENTER = { x: 60.5, y: 58.5 };
// Screenshot of a small canvas patch centred on (x, y), in canvas
// coordinates.
async function patchAt(workspace, x, y) {
const viewport = await workspace.viewport.boundingBox();
const px = viewport.x + viewport.width / 2 + x - CONTENT_CENTER.x;
const py = viewport.y + viewport.height / 2 + y - CONTENT_CENTER.y;
return workspace.page.screenshot({
clip: { x: px - 2, y: py - 2, width: 4, height: 4 },
});
}
async function openFile(page) {
const workspace = new WasmWorkspacePage(page, { textEditor: true });
await workspace.setupEmptyFile();
await workspace.mockGetFile("text-editor/get-file-12325.json");
await workspace.goToWorkspace();
await workspace.waitForFirstRender();
return workspace;
}
// Shrinks `text` to its first character while editing and checks that a spot
// in the right part of its board, at `boardY`, shows the empty canvas.
async function expectBoardShrinksWhileTyping(workspace, text, boardY) {
const x = 80;
const inside = boardY + 8;
const outside = boardY + 50;
expect(await patchAt(workspace, x, inside)).not.toEqual(
await patchAt(workspace, x, outside),
);
await workspace.textEditor.startEditing();
await workspace.page.keyboard.press("End");
for (let i = 0; i < text.length - 1; i++) {
await workspace.page.keyboard.press("Backspace");
}
// Without leaving the editor, the board already shrank around the text, so
// the spot shows the empty canvas.
await expect
.poll(async () =>
(await patchAt(workspace, x, inside)).equals(
await patchAt(workspace, x, outside),
),
)
.toBe(true);
}
test("The board shrinks with the text", async ({ page }) => {
const workspace = await openFile(page);
await workspace.clickToggableLayer("Text board");
await workspace.clickLeafLayer("Hello wide world");
await expectBoardShrinksWhileTyping(workspace, "Hello wide world", 0);
});
test("The board shrinks with a text inside a group", async ({ page }) => {
const workspace = await openFile(page);
await workspace.clickToggableLayer("Group board");
// The first "Group" match is the board row.
await workspace.clickToggableLayer("Group", {}, 1);
await workspace.clickLeafLayer("Hello grouped world");
await expectBoardShrinksWhileTyping(workspace, "Hello grouped world", 100);
});
});

5
frontend/playwright/ui/specs/viewer-inspect-exports.spec.js
View File

@ -23,11 +23,8 @@ test("[View mode] Export presets are preserved when navigating between boards in
// Enter inspect (code) mode
await viewer.showCode();
// Wait for the inspect panel to load
await page.waitForSelector(".main_ui_inspect_exports__add-export");
// Add an export preset via the "+" button in the Export section
const addExportButton = page.locator(".main_ui_inspect_exports__add-export");
const addExportButton = page.getByRole("button", { name: "Add export" });
await addExportButton.click();
// Verify the "Export 1 element" button appears, confirming the preset was added

1
frontend/scripts/check-translations/words.es.txt
View File

@ -281,6 +281,7 @@ lectores
letras
manifiesto
menos
navegadores
nosotros
otras
pero

5
frontend/scripts/test
View File

@ -10,3 +10,8 @@ popd
pnpm run lint:scss;
pnpm run test;
# Mousetrap JS unit tests (node:test, zero deps)
pushd packages/mousetrap;
pnpm test;
popd

3
frontend/src/app/main/data/dashboard/shortcuts.cljs
View File

@ -50,7 +50,8 @@
(def shortcut-create-new-project
{:create-new-project {:tooltip "+"
:label (fn [] (tr "shortcuts.create-new-project"))
:command "+"
:command ["+" "="]
:show-command "+"
:section [:dashboard]
:subsections [:generic]
:fn #(st/emit! (dd/create-element))}})

198
frontend/src/app/main/data/exports/assets.cljs
View File

@ -12,11 +12,14 @@
[app.main.data.exports.wasm :as wasm.exports]
[app.main.data.helpers :as dsh]
[app.main.data.modal :as modal]
[app.main.data.notifications :as ntf]
[app.main.data.persistence :as dwp]
[app.main.errors :as errors]
[app.main.features :as features]
[app.main.repo :as rp]
[app.main.store :as st]
[app.util.dom :as dom]
[app.util.i18n :refer [tr]]
[app.util.websocket :as ws]
[beicon.v2.core :as rx]
[cuerdas.core :as str]
@ -24,6 +27,68 @@
(def default-timeout 5000)
;; Longest wait for the pending changes of the exported files to be saved.
(def save-wait-timeout-ms 10000)
(defn- export-file-ids
[exports]
(into #{} (keep :file-id) exports))
(defn- warn-unsaved-changes
"Warns that the export may miss the changes of `file-ids` not saved yet."
[file-ids]
(ptk/reify ::warn-unsaved-changes
ptk/WatchEvent
(watch [_ state _]
(when (dwp/unsaved-changes? state file-ids)
(rx/of (ntf/show {:content (tr "workspace.export.unsaved-changes")
:type :toast
:level :warning
:timeout 8000}))))))
(defn- export-failed
"Shows an export failure as a toast, keeping the workspace open: the
failure is never raised to the global error handler, which would
replace the workspace with an error page."
[cause]
(ptk/reify ::export-failed
ptk/EffectEvent
(effect [_ _ _]
(errors/flash :cause cause
:type (if (#{:server-error :internal} (:type (ex-data cause)))
:unhandled
:handled)
:hint (tr "errors.export-failed")))))
(defn- export-widget-failed
"Reports a failure that happened before the export ever started: the
widget settles instead of waiting for progress that will never arrive,
and the toast says what happened. Nothing is raised to the global
error handler, which would replace the workspace with an error page."
[exports cmd cause]
(ptk/reify ::export-widget-failed
ptk/UpdateEvent
(update [_ state]
(assoc state :export {:in-progress false
:widget-visible true
:detail-visible true
:healthy? true
:progress 0
:total (count exports)
:exports exports
:cmd cmd
:error (or (ex-message cause) true)
:error-code (:code (ex-data cause))
:last-update (ct/now)}))
ptk/EffectEvent
(effect [_ _ _]
(errors/flash :cause cause
:type (if (#{:server-error :internal} (:type (ex-data cause)))
:unhandled
:handled)
:hint (tr "errors.export-failed")))))
(defn normalize-export
[{:keys [object-id name] :as export}]
(assoc export :name (if (str/blank? name)
@ -200,29 +265,6 @@
(when (= status "ended")
(dom/trigger-download-uri filename mtype resource-uri)))))
;; The exporter is at capacity. Not a crash: the widget says so and the user
;; retries, instead of the generic error dialog.
(def ^:private saturation-codes #{:queue-full})
(defn- export-failed
"Reports a failure that happened before the export ever started, so the widget
settles instead of waiting for progress that will never arrive."
[exports cmd cause]
(ptk/reify ::export-failed
ptk/UpdateEvent
(update [_ state]
(assoc state :export {:in-progress false
:widget-visible true
:detail-visible true
:healthy? true
:progress 0
:total (count exports)
:exports exports
:cmd cmd
:error (or (ex-message cause) true)
:error-code (:code (ex-data cause))
:last-update (ct/now)}))))
(defn cancel-export
"Stops the running export. Only reachable on the job API path, where the
exporter can actually abort the work.
@ -295,16 +337,16 @@
:wait true
:is-wasm (wasm-export-enabled? state)})]
(rx/concat
(dwp/force-persist-and-wait 400)
(dwp/force-persist-files-and-wait (export-file-ids [export]) save-wait-timeout-ms)
(rx/of (warn-unsaved-changes (export-file-ids [export])))
(->> (rp/cmd! :export params)
(rx/map (fn [{:keys [filename mtype uri]}]
(dom/trigger-download-uri filename mtype uri)
(clear-export-state uuid/zero)))
(rx/catch (fn [cause]
(rx/concat
(rx/of (clear-export-state uuid/zero))
(rx/throw cause))))))))))))
(rx/of (clear-export-state uuid/zero)
(export-failed cause))))))))))))
(defn request-multiple-export
[{:keys [exports cmd name]
@ -315,6 +357,8 @@
ptk/WatchEvent
(watch [_ state _]
(let [resource-id (volatile! nil)
;; Emits when the export request fails.
failed-s (rx/subject)
profile-id (:profile-id state)
ws-conn (:ws-conn state)
params (cond->
@ -332,67 +376,67 @@
(rx/map :payload)
(rx/filter #(= :export-update (:type %)))
(rx/filter #(= @resource-id (:resource-id %)))
(rx/take-until failed-s)
(rx/share))
stopper
(rx/filter #(or (= "ended" (:status %))
(= "error" (:status %))
(= "cancelled" (:status %)))
progress-stream)]
progress-stream)
file-ids
(export-file-ids exports)]
(swap! st/ongoing-tasks conj :export)
(rx/merge
;; Force that all data is persisted; best effort.
(rx/of ::dwp/force-persist)
(rx/concat
(dwp/force-persist-files-and-wait file-ids save-wait-timeout-ms)
(rx/of (warn-unsaved-changes file-ids))
(rx/merge
;; Updates the export state with each progress message, until
;; shortly after the export ends or fails.
(->> progress-stream
(rx/map update-export-status)
(rx/take-until (rx/delay 500 stopper))
(rx/finalize (fn []
(swap! st/ongoing-tasks disj :export))))
;; Launch the exportation process and stores the resource id
;; locally. With wasm export active the job API is used instead: it
;; answers with the exporter's own object count and gives a handle
;; to cancel.
(->> (if (wasm-export-enabled? state)
(->> (rp/cmd! :create-export-job params)
(rx/map (fn [{job-id :id :keys [total] :as job}]
(vreset! resource-id (:resource-id job))
(initialize-export-status exports cmd
{:id (:resource-id job)}
{:job-id job-id
:total total
:status (:state job)
:backend (:backend job)}))))
(->> (rp/cmd! :export params)
(rx/map (fn [{:keys [id] :as resource}]
(vreset! resource-id id)
(initialize-export-status exports cmd resource nil)))))
(rx/catch (fn [cause]
;; Saturation is an answer, not a fault.
(if (contains? saturation-codes (:code (ex-data cause)))
(rx/of (export-failed exports cmd cause))
(rx/concat
(rx/of (export-failed exports cmd cause))
(rx/throw cause))))))
;; Clears the export state a while after the export ends or is
;; cancelled.
(->> progress-stream
(rx/filter #(or (= "ended" (:status %))
(= "cancelled" (:status %))))
(rx/take 1)
(rx/delay default-timeout)
(rx/map #(clear-export-state @resource-id))
(rx/take-until (rx/delay 6000 stopper)))
;; We proceed to update the export state with incoming
;; progress updates. We delay the stopper for give some time
;; to update the status with ended or errored status before
;; close the stream.
(->> progress-stream
(rx/map update-export-status)
(rx/take-until (rx/delay 500 stopper))
(rx/finalize (fn []
(swap! st/ongoing-tasks disj :export))))
;; We hide need to hide the ui elements of the export after
;; some interval. We also delay a little bit more the stopper
;; for ensure that after some security time, the stream is
;; completely closed.
(->> progress-stream
(rx/filter #(or (= "ended" (:status %))
(= "cancelled" (:status %))))
(rx/take 1)
(rx/delay default-timeout)
(rx/map #(clear-export-state @resource-id))
(rx/take-until (rx/delay 6000 stopper)))))))))
;; Launches the exportation process and stores the resource id
;; locally. With wasm export active the job API is used instead:
;; it answers with the exporter's own object count and gives a
;; handle to cancel. It goes last so a failure reaches the
;; progress listeners above: the push to `failed-s` releases the
;; leave-page warning even though no progress message will ever
;; arrive, and nothing is raised to the global error handler,
;; which would replace the workspace with an error page.
(->> (if (wasm-export-enabled? state)
(->> (rp/cmd! :create-export-job params)
(rx/map (fn [{job-id :id :keys [total] :as job}]
(vreset! resource-id (:resource-id job))
(initialize-export-status exports cmd
{:id (:resource-id job)}
{:job-id job-id
:total total
:status (:state job)
:backend (:backend job)}))))
(->> (rp/cmd! :export params)
(rx/map (fn [{:keys [id] :as resource}]
(vreset! resource-id id)
(initialize-export-status exports cmd resource nil)))))
(rx/catch (fn [cause]
(rx/push! failed-s cause)
(rx/of (export-widget-failed exports cmd cause))))))))))))
(defn request-export
[{:keys [exports] :as params}]

40
frontend/src/app/main/data/exports/files.cljs
View File

@ -12,6 +12,7 @@
[app.main.data.event :as ev]
[app.main.data.jobs :as dj]
[app.main.data.modal :as modal]
[app.main.data.persistence :as dps]
[app.main.repo :as rp]
[app.main.store :as st]
[beicon.v2.core :as rx]
@ -37,9 +38,25 @@
(def check-export-files
(sm/check-fn schema:export-files))
;; Longest wait for the pending changes of the downloaded files to be saved.
(def save-wait-timeout-ms 10000)
(defn- show-export-dialog
"Opens the download dialog, flagging whether the files have changes not
saved yet."
[team-id files]
(ptk/reify ::show-export-dialog
ptk/WatchEvent
(watch [_ state _]
(rx/of (modal/show {:type ::export-files
:team-id team-id
:files files
:unsaved (dps/unsaved-changes? state (into #{} (map :id) files))})))))
(defn open-export-dialog
[files]
(let [files (check-export-files files)]
(let [files (check-export-files files)
file-ids (into #{} (map :id) files)]
(ptk/reify ::export-files
ptk/WatchEvent
(watch [_ state _]
@ -49,16 +66,17 @@
::ev/origin "dashboard"
:format "binfile-v3"
:num-files (count files)}))
(->> (rx/from files)
(rx/mapcat
(fn [file]
(->> (rp/cmd! :has-file-libraries {:file-id (:id file)})
(rx/map #(assoc file :has-libraries %)))))
(rx/reduce conj [])
(rx/map (fn [files]
(modal/show {:type ::export-files
:team-id team-id
:files files}))))))))))
;; The download is built from the server copy: send the pending
;; changes first.
(rx/concat
(dps/force-persist-files-and-wait file-ids save-wait-timeout-ms)
(->> (rx/from files)
(rx/mapcat
(fn [file]
(->> (rp/cmd! :has-file-libraries {:file-id (:id file)})
(rx/map #(assoc file :has-libraries %)))))
(rx/reduce conj [])
(rx/map (partial show-export-dialog team-id))))))))))
(defn- export-file
"One file, one job: the file is exported on its own, so a failure does

4
frontend/src/app/main/data/exports/wasm.cljs
View File

@ -30,8 +30,8 @@
nil))
(defn export-pdf-uri
[{:keys [scale object-id]}]
(let [bytes (wasm.api/render-shape-pdf object-id (or scale 1))
[{:keys [object-id]}]
(let [bytes (wasm.api/render-shape-pdf object-id)
blob (wapi/create-blob bytes "application/pdf")]
(wapi/create-uri blob)))

108
frontend/src/app/main/data/persistence.cljs
View File

@ -14,6 +14,7 @@
[app.common.uuid :as uuid]
[app.main.data.changes :as dch]
[app.main.data.common :as-alias dc]
[app.main.data.event :as ev]
[app.main.data.helpers :as dsh]
[app.main.data.notifications :as ntf]
[app.main.data.workspace :as-alias dw]
@ -108,6 +109,30 @@
([timeout-ms]
(rx/concat (rx/of ::force-persist) (wait-persisted timeout-ms))))
(defn- files-queued?
"True when the queue in `pstate` still holds changes of any of `file-ids`."
[{:keys [queue index]} file-ids]
(boolean (some #(contains? file-ids (get-in index [% :file-id])) queue)))
(defn unsaved-changes?
"True when any of `file-ids` has changes the server does not have yet."
[state file-ids]
(files-queued? (:persistence state) file-ids))
(defn force-persist-files-and-wait
"Sends the pending changes and waits until no change of `file-ids` is
queued, saving fails, or `timeout-ms` passes. Never raises; check
`unsaved-changes?` afterwards."
[file-ids timeout-ms]
(rx/concat
(rx/of ::force-persist)
(->> (rx/from-atom refs/persistence {:emit-current-value? true})
(rx/filter #(or (= :error (:status %))
(not (files-queued? % file-ids))))
(rx/take 1)
(rx/ignore)
(rx/timeout timeout-ms (rx/empty)))))
(defn- next-status
"Refuses downgrades: a save in progress stays :saving, a failed save
stays :error until persistence is resumed, and a retrying episode stays
@ -138,17 +163,31 @@
(#{:error :saved} status)
(dissoc :run-id :last-progress-at :stall-reported))))))))
(def ^:private busy-server-codes
"Server error codes for a database timeout that rolled the save back."
#{:statement-timeout :idle-in-transaction-timeout})
(defn- busy-server-error?
"True when the backend was too busy to apply the save: it refused it
(429) or a database timeout rolled it back. Resending it is safe."
[{:keys [type cause-type code]}]
(let [type (or cause-type type)]
(or (= :rate-limit type)
(and (= :server-error type)
(contains? busy-server-codes code)))))
(defn transient-error?
"True when a save failure is worth retrying with backoff: a transient
transport failure (from the shared `repo/retryable-types` set, which says
nothing about the edits themselves) or an unusable save response.
Everything else (auth, validation, state) is terminal and keeps the
`:error` path. Checked against both `:type` and `:cause-type` because
nothing about the edits themselves), a busy backend, or an unusable save
response. Everything else (auth, validation, state) is terminal and keeps
the `:error` path. Checked against both `:type` and `:cause-type` because
`persistence-failed` wraps the original cause under `:type :persistence`."
[{:keys [type cause-type code]}]
[{:keys [type cause-type code] :as error}]
(boolean
(or (contains? rp/retryable-types type)
(contains? rp/retryable-types cause-type)
(busy-server-error? error)
(= :invalid-save-response code))))
(def ^:private retry-delays-ms
@ -299,6 +338,23 @@
(update :queue #(into (empty %) (remove dropped) %))
(update :index #(apply dissoc % dropped)))))))
(defn- notify-failure?
"False for a failed attempt of the slow cycle: it repeats a failure the
user was already told about."
[pstate]
(not (and (:recovering pstate)
(slow-retry-pending? pstate))))
(defn- save-failed-event
"Audit event for a save failure the user is told about, so failed saves
can be counted."
[{:keys [error queue] :as pstate}]
(ev/event {::ev/name "save-failed"
:cause-type (:cause-type error)
:code (:code error)
:retrying (slow-retry-pending? pstate)
:queued-commits (count queue)}))
(defn- persistence-failed
[commit-id cause]
(ptk/reify ::persistence-failed
@ -322,14 +378,16 @@
ptk/WatchEvent
(watch [_ state stream]
(rx/merge
;; The terminal toast supersedes the reconnect notice; hide it
;; explicitly instead of relying on the single-toast replacement.
(rx/of (ptk/data-event ::error cause)
(ntf/hide :tag reconnecting-tag))
;; A transport failure that outlasts the backoff can still pass, so
;; the queue keeps trying at a slow pace.
(let [pstate (:persistence state)]
(let [pstate (:persistence state)]
(rx/merge
;; The terminal toast replaces the reconnect notice, hidden by tag.
(rx/of (ptk/data-event ::error cause)
(ntf/hide :tag reconnecting-tag))
(if (notify-failure? pstate)
(rx/of (save-failed-event pstate))
(rx/empty))
;; A transport failure that outlasts the backoff can still pass, so
;; the queue keeps trying at a slow pace.
(if (and (not (:suspended pstate))
(slow-retry-pending? pstate))
(slow-retry-cycle stream)
@ -337,13 +395,10 @@
ptk/EffectEvent
(effect [_ state _]
;; A failed slow attempt repeats a failure the user was already warned
;; about. Report without invoking global handlers that may reload the
;; file or navigate away before the user can recover the retained
;; changes.
;; Notifies without the global handlers, which may reload the file or
;; navigate away and lose the retained changes.
(let [pstate (:persistence state)]
(when-not (and (:recovering pstate)
(slow-retry-pending? pstate))
(when (notify-failure? pstate)
(errors/flash-persistence cause :retrying (slow-retry-pending? pstate)))))))
(defn- persistence-transient-failure
@ -783,6 +838,17 @@
(swap! st/ongoing-tasks conj :persistence)
(swap! st/ongoing-tasks disj :persistence)))))
(defn leave-signals
"Emits when the tab is hidden or the window loses focus: another tab or
app, a plugin iframe, a closed laptop. Empty outside the browser."
[]
(if (and (exists? js/window) (exists? js/document))
(rx/merge
(->> (rx/from-event js/document "visibilitychange")
(rx/filter #(= "hidden" (.-visibilityState js/document))))
(rx/from-event js/window "blur"))
(rx/empty)))
(defn initialize-persistence
[]
(ptk/reify ::initialize-persistence
@ -841,6 +907,12 @@
(rx/map #(ptk/data-event ::persistence-notification))
(rx/take-until stoper-s))
;; Sends the buffered edits when the user leaves the tab or window.
(->> (leave-signals)
(rx/filter (fn [_] @buffered?))
(rx/map (fn [_] ::force-persist))
(rx/take-until stoper-s))
;; Syncs the leave-page warning on every local commit and on every
;; queue change (append after a flush, discard after a save,
;; release of a failed file).

31
frontend/src/app/main/data/shortcuts.cljs
View File

@ -17,6 +17,37 @@
(log/set-level! :warn)
;; Keyboard layout map shared by the shortcut recorder
;; (`app.main.ui.shortcuts`) and the Mousetrap dispatcher: punctuation
;; resolves by physical position through this map, so recording and
;; dispatch use the same token. `nil` until `getLayoutMap` resolves (or
;; when the browser has no keyboard API); both sides then fall back to
;; the typed glyph, which keeps them consistent with and without a map.
(defonce layout-map (atom nil))
(defn layout-map-supported?
"Synchronous keyboard-layout API probe: true when the browser exposes
navigator.keyboard.getLayoutMap (Chromium). Punctuation then resolves
by physical position; without it both sides fall back to the typed
glyph, so Shift+punctuation shortcuts may not fire."
([] (layout-map-supported?
(when (exists? js/navigator) js/navigator)))
([navigator]
(let [keyboard (when (some? navigator) (.-keyboard ^js navigator))]
(boolean
(and (some? keyboard)
(exists? (.-getLayoutMap ^js keyboard)))))))
(let [navigator (when (exists? js/navigator) js/navigator)]
(when (layout-map-supported? navigator)
(-> (.getLayoutMap ^js (.-keyboard ^js navigator))
(.then (fn [m]
(cljs.core/reset! layout-map m)
(.setLayoutMap mousetrap m))
(fn [_]
(cljs.core/reset! layout-map nil)
(.setLayoutMap mousetrap nil))))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Helpers
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

3
frontend/src/app/main/data/viewer/shortcuts.cljs
View File

@ -17,7 +17,8 @@
(def shortcuts
{:increase-zoom {:tooltip "+"
:label (fn [] (tr "shortcuts.increase-zoom"))
:command "+"
:command ["+" "="]
:show-command "+"
:subsections [:zoom-viewer]
:section [:viewer]
:fn #(st/emit! dv/increase-zoom)}

3
frontend/src/app/main/data/workspace/grid_layout/shortcuts.cljs
View File

@ -55,7 +55,8 @@
:increase-zoom {:tooltip "+"
:label (fn [] (tr "shortcuts.increase-zoom"))
:command "+"
:command ["+" "="]
:show-command "+"
:section [:workspace]
:fn #(st/emit! (dw/increase-zoom nil))}

11
frontend/src/app/main/data/workspace/modifiers.cljs
View File

@ -744,8 +744,9 @@
#_:clj-kondo/ignore
(defn set-wasm-modifiers
[modif-tree & {:keys [ignore-constraints ignore-snap-pixel snap-ignore-axis
subtree-ids-by-id selection-rect-cache]
:or {ignore-constraints false ignore-snap-pixel false snap-ignore-axis nil}
subtree-ids-by-id selection-rect-cache interactive?]
:or {ignore-constraints false ignore-snap-pixel false snap-ignore-axis nil
interactive? true}
:as params}]
(let [modif-tree (without-nil-ids modif-tree)]
(ptk/reify ::set-wasm-modifiers
@ -764,8 +765,10 @@
;; the renderer into fast + atlas-backdrop mode so the live
;; preview is cheap, tiles never appear sequentially and the main
;; thread is not blocked. The pair is closed in
;; `clear-local-transform`.
(ensure-interactive-transform-start!)
;; `clear-local-transform`. Previews outside a gesture pass
;; `:interactive? false` to keep full quality rendering.
(when interactive?
(ensure-interactive-transform-start!))
(let [snap-pixel? (and (not ignore-snap-pixel) (contains? (:workspace-layout state) :snap-pixel-grid))
translation? (every? #(ctm/only-move? (:modifiers %)) (vals modif-tree))]

9
frontend/src/app/main/data/workspace/path/shortcuts.cljs
View File

@ -59,7 +59,8 @@
:add-node {:tooltip (ds/shift "+")
:label (fn [] (tr "shortcuts.add-node"))
:command "shift++"
:command ["shift++" "shift+="]
:show-command "shift++"
:subsections [:path-editor]
:section [:workspace]
:fn #(st/emit! (drp/add-node))}
@ -117,8 +118,7 @@
:snap-nodes {:tooltip (ds/meta "'")
:label (fn [] (tr "shortcuts.snap-nodes"))
;;https://github.com/ccampbell/mousetrap/issues/85
:command [(ds/c-mod "'") (ds/c-mod "219")]
:command (ds/c-mod "'")
:subsections [:path-editor]
:section [:workspace]
:fn #(st/emit! (drp/toggle-snap))}
@ -207,7 +207,8 @@
:increase-zoom {:tooltip "+"
:label (fn [] (tr "shortcuts.increase-zoom"))
:command "+"
:command ["+" "="]
:show-command "+"
:section [:workspace]
:fn #(st/emit! (dw/increase-zoom nil))}

12
frontend/src/app/main/data/workspace/shortcuts.cljs
View File

@ -567,8 +567,7 @@
:toggle-guides {:tooltip (ds/meta "'")
:label (fn [] (tr "shortcuts.toggle-guides"))
;;https://github.com/ccampbell/mousetrap/issues/85
:command [(ds/c-mod "'") (ds/c-mod "219")]
:command (ds/c-mod "'")
:show-command (ds/c-mod "'")
:subsections [:main-menu]
:section [:workspace]
@ -619,8 +618,7 @@
:toggle-snap-guides {:tooltip (ds/meta-shift "'")
:label (fn [] (tr "shortcuts.toggle-snap-guides"))
;;https://github.com/ccampbell/mousetrap/issues/85
:command [(ds/c-mod "shift+'") (ds/c-mod "shift+219")]
:command (ds/c-mod "shift+'")
:show-command (ds/c-mod "shift+'")
:subsections [:main-menu]
:section [:workspace]
@ -679,7 +677,8 @@
:hide-ui {:tooltip "\\"
:label (fn [] (tr "shortcuts.hide-ui"))
:command "\\"
:command ["\\" "alt+\\" "alt+º"]
:show-command "\\"
:subsections [:panels :basics]
:section [:workspace :basics]
:fn #(st/emit! (toggle-layout-flag :hide-ui))}
@ -716,7 +715,8 @@
:zoom-selected {:tooltip (ds/shift "2")
:label (fn [] (tr "shortcuts.zoom-selected"))
:command ["shift+2" "shift+num2" "@" "\""]
:command ["shift+2" "shift+num2" "shift+'" "shift+\""]
:show-command "shift+2"
:subsections [:zoom-workspace]
:section [:workspace]
:fn #(st/emit! dw/zoom-to-selected-shape)}

66
frontend/src/app/main/data/workspace/texts.cljs
View File

@ -1468,12 +1468,72 @@
:name name
:finalize? finalize?)))
(defn- layout-item
"The ancestor of `shape` (or `shape` itself) that is a direct child of its board."
[objects shape]
(loop [item shape]
(let [parent-id (:parent-id item)]
(if (or (= parent-id (:frame-id shape))
(nil? (get objects parent-id)))
item
(recur (get objects parent-id))))))
(defn- v3-sync-text-layout
"Previews an auto-grow text's live size in its layout, so a fit-content parent
follows it while typing. Geometry is committed on finalize."
[id]
(ptk/reify ::v3-sync-text-layout
ptk/WatchEvent
(watch [_ state _]
(let [objects (dsh/lookup-page-objects state)
shape (get objects id)
preview? (and (cfh/text-shape? shape)
(not= :fixed (:grow-type shape))
(ctl/any-layout-descent? objects shape)
(not (ctl/position-absolute? (layout-item objects shape))))
size (when preview?
(dwwt/get-wasm-text-new-size shape))
previous (or (get-in state [::layout-preview id])
(select-keys (:selrect shape) [:width :height]))]
;; Modifiers are costly, so skip keystrokes that keep the size.
(if (and (some? size)
(not (and (mth/close? (:width size) (:width previous))
(mth/close? (:height size) (:height previous)))))
(rx/of #(assoc-in % [::layout-preview id] size)
(dwm/set-wasm-modifiers (dwwt/resize-wasm-text-modifiers shape)
:interactive? false))
(rx/empty))))))
(defn v3-stop-text-layout-preview
"Drops the layout preview of the text `id` and leaves interactive render mode."
[id]
(ptk/reify ::v3-stop-text-layout-preview
ptk/WatchEvent
(watch [_ state _]
(if (contains? (get state ::layout-preview) id)
(rx/of #(update % ::layout-preview dissoc id)
(dwm/clear-local-transform))
(rx/empty)))))
(defn v3-update-editor-content
"Commits `content` from the WASM editor to the text `shape-id` and, while
editing, previews its new size in the layout."
[shape-id content & {:keys [finalize?]}]
(let [update-event (editor-content-update shape-id content :finalize? finalize?)]
(if finalize?
update-event
(ptk/reify ::v3-update-editor-content
ptk/WatchEvent
(watch [_ _ _]
(rx/of update-event
(v3-sync-text-layout shape-id)))))))
(defn v3-sync-editor-content
"Event pushing the WASM editor content back into the shape, or nil when there is
nothing to sync. Every text edit commits through it, menu or keystroke alike."
[& {:keys [finalize?]}]
(when-let [{:keys [shape-id content]} (wasm.text-editor/text-editor-sync-content)]
(editor-content-update shape-id content :finalize? finalize?)))
(v3-update-editor-content shape-id content :finalize? finalize?)))
(defn v3-apply-pending-caret-styles
"Event saving an insertion made at `before` that consumed the pending caret style:
@ -1483,7 +1543,7 @@
;; Sync first so the cached content stays index-aligned with WASM.
(wasm.text-editor/text-editor-sync-content)
(if-let [{:keys [content]} (wasm.api/apply-pending-caret-styles! shape-id range)]
(editor-content-update shape-id content)
(v3-update-editor-content shape-id content)
(v3-sync-editor-content))))
(defn- sync-editor-content-stream
@ -1636,7 +1696,7 @@
;; Sync first so the cached content stays index-aligned with WASM.
(wasm.text-editor/text-editor-sync-content)
(if-let [{:keys [content]} (wasm.api/apply-paste-styles shape-id fragment start penpot?)]
(editor-content-update shape-id content)
(v3-update-editor-content shape-id content)
(v3-sync-editor-content)))
pending?

10
frontend/src/app/main/ui/exports/files.cljs
View File

@ -20,6 +20,7 @@
[app.main.ui.ds.foundations.typography :as t]
[app.main.ui.ds.foundations.typography.heading :refer [heading*]]
[app.main.ui.ds.foundations.typography.text :refer [text*]]
[app.main.ui.ds.notifications.context-notification :refer [context-notification*]]
[app.main.ui.ds.product.loader :refer [loader*]]
[app.main.ui.jobs.progress :as jp]
[app.main.ui.notifications.context-notification :refer [context-notification]]
@ -134,7 +135,7 @@
{::mf/register modal/components
::mf/register-as ::fexp/export-files
::mf/props :obj}
[{:keys [team-id files]}]
[{:keys [team-id files unsaved]}]
(let [state* (mf/use-state (partial initialize-state files))
has-libs? (some :has-libraries files)
export-types (cond-> fexp/valid-types
@ -248,6 +249,13 @@
:on-click on-cancel
:class (stl/css :modal-close-btn)
:icon i/close}]]
(when unsaved
[:> context-notification* {:level :warning
:appearance :ghost
:class (stl/css :unsaved-warning)}
(tr "files-download-modal.unsaved-changes")])
(cond
(= status :prepare)
[:*

4
frontend/src/app/main/ui/exports/files.scss
View File

@ -39,6 +39,10 @@
max-block-size: calc(10 * px2rem(80));
}
.unsaved-warning {
margin-block-end: var(--sp-xxl);
}
.modal-content {
display: flex;
flex-direction: column;

37
frontend/src/app/main/ui/inspect/exports.cljs
View File

@ -14,7 +14,9 @@
[app.main.store :as st]
[app.main.ui.components.select :refer [select]]
[app.main.ui.components.title-bar :refer [title-bar*]]
[app.main.ui.icons :as deprecated-icon]
[app.main.ui.ds.buttons.button :refer [button*]]
[app.main.ui.ds.buttons.icon-button :refer [icon-button*]]
[app.main.ui.ds.foundations.assets.icon :as i]
[app.util.dom :as dom]
[app.util.i18n :refer [tr c]]
[app.util.keyboard :as kbd]
@ -161,17 +163,19 @@
[:> title-bar* {:collapsable false
:title (tr "workspace.options.export")
:class (stl/css :title-spacing-export-viewer)}
[:button {:class (stl/css :add-export)
:on-click add-export} deprecated-icon/add]]]
[:> icon-button* {:variant "ghost"
:aria-label (tr "workspace.options.export.add-export")
:on-click add-export
:icon i/add}]]]
(cond
(= :multiple exports)
[:div {:class (stl/css :multiple-exports)}
[:div {:class (stl/css :label)} (tr "settings.multiple")]
[:div {:class (stl/css :actions)}
[:button {:class (stl/css :action-btn)
:on-click ()}
deprecated-icon/remove-icon]]]
[:> icon-button* {:variant "ghost"
:aria-label (tr "workspace.options.export.remove-export")
:icon i/remove}]]]
(seq @exports)
[:div {:class (stl/css :element-set-content)}
@ -183,14 +187,14 @@
[:& select
{:default-value (d/name (:type export))
:options format-options
:dropdown-class (stl/css :dropdown-upwards)
:dropdown-class (stl/css :dropdown)
:on-change (partial on-type-change index)}]]
(when (scale-enabled? export)
[:div {:class (stl/css :size-select)}
[:& select
{:default-value (str (:scale export))
:options size-options
:dropdown-class (stl/css :dropdown-upwards)
:dropdown-class (stl/css :dropdown)
:on-change (partial on-scale-change index)}]])
[:label {:class (stl/css :suffix-input)
:for "suffix-export-input"}
@ -203,16 +207,15 @@
:on-change on-suffix-change
:on-key-down manage-key-down}]]]
[:button {:class (stl/css :action-btn)
:on-click (partial delete-export index)}
deprecated-icon/remove-icon]])])
[:> icon-button* {:variant "ghost"
:aria-label (tr "workspace.options.export.remove-export")
:on-click (partial delete-export index)
:icon i/remove}]])])
(when (or (= :multiple exports) (seq @exports))
[:button
{:on-click (when-not in-progress? on-download)
:class (stl/css-case
:export-btn true
:btn-disabled in-progress?)
:disabled in-progress?}
[:> button* {:variant "secondary"
:class (stl/css :export-btn)
:on-click (when-not in-progress? on-download)
:disabled in-progress?}
(if in-progress?
(tr "workspace.options.exporting-object")
(tr "workspace.options.export-object" (c (count shapes))))])]))

73
frontend/src/app/main/ui/inspect/exports.scss
View File

@ -5,9 +5,11 @@
// Copyright (c) KALEIDOS SUBSIDIARY SL
@use "refactor/common-refactor.scss" as deprecated;
@use "ds/typography.scss" as *;
@use "ds/_utils.scss" as *;
.element-set {
padding-bottom: deprecated.$s-16;
padding-block-end: var(--sp-l);
margin: 0;
}
@ -17,33 +19,22 @@
.title-spacing-export-viewer {
margin: 0;
color: var(--entry-foreground-color-hover);
margin-inline-start: calc(-1 * deprecated.$s-8);
width: calc(100% + deprecated.$s-8);
}
.add-export {
@extend %button-tertiary;
height: deprecated.$s-32;
width: deprecated.$s-28;
svg {
@extend %button-icon;
stroke: var(--icon-foreground);
}
color: var(--color-foreground-primary);
margin-inline-start: calc(-1 * var(--sp-s));
inline-size: calc(100% + var(--sp-s));
}
.element-set-content {
@include deprecated.flex-column;
margin-bottom: deprecated.$s-4;
display: flex;
flex-direction: column;
gap: var(--sp-xs);
margin-block-end: var(--sp-xs);
}
.multiple-exports {
@include deprecated.flex-row;
display: flex;
align-items: center;
gap: var(--sp-xs);
grid-column: 1 / span 9;
}
@ -52,24 +43,15 @@
}
.actions {
@include deprecated.flex-row;
display: flex;
align-items: center;
gap: var(--sp-xs);
}
.element-group {
display: grid;
grid-template-columns: repeat(9, 1fr);
column-gap: deprecated.$s-4;
.action-btn {
@extend %button-tertiary;
height: deprecated.$s-32;
width: deprecated.$s-28;
svg {
@extend %button-icon;
}
}
column-gap: var(--sp-xs);
}
.input-wrapper {
@ -82,10 +64,8 @@
grid-column: span 3;
padding: 0;
.dropdown-upwards {
bottom: deprecated.$s-36;
width: deprecated.$s-80;
top: unset;
.dropdown {
inline-size: px2rem(80);
}
}
@ -93,24 +73,19 @@
grid-column: span 2;
padding: 0;
.dropdown-upwards {
bottom: deprecated.$s-36;
top: unset;
width: deprecated.$s-80;
.dropdown {
inline-size: px2rem(80);
}
}
.suffix-input {
@extend %input-element;
@include deprecated.body-small-typography;
@include use-typography("body-small");
grid-column: span 3;
}
.export-btn {
@extend %button-secondary;
@include deprecated.uppercase-title-typography;
height: deprecated.$s-32;
width: 100%;
justify-content: center;
inline-size: 100%;
}

5
frontend/src/app/main/ui/settings/shortcuts.cljs
View File

@ -23,6 +23,7 @@
[app.main.ui.ds.foundations.typography.heading :refer [heading*]]
[app.main.ui.ds.foundations.typography.text :refer [text*]]
[app.main.ui.ds.layout.tab-switcher :refer [tab-switcher*]]
[app.main.ui.ds.notifications.context-notification :refer [context-notification*]]
[app.main.ui.ds.product.empty-state :refer [empty-state*]]
[app.main.ui.settings.import-shortcuts-diff-modal]
[app.main.ui.settings.restore-shortcuts-modal]
@ -477,6 +478,10 @@
[:> icon* {:icon-id i/info
:class (stl/css :shortcuts-description-icon)}]
(tr "shortcuts.reload-hint")]
(when-not (ds/layout-map-supported?)
[:> context-notification* {:level :warning
:class (stl/css :layout-map-warning)}
(tr "shortcuts.layout-map-missing")])
[:> (mf/provider ctx/shortcuts-ctx) {:value shortcuts-ctx-value}
[:div {:class (stl/css :shortcuts-content)}

4
frontend/src/app/main/ui/settings/shortcuts.scss
View File

@ -314,3 +314,7 @@
padding: var(--sp-s) 0;
margin-block-end: var(--sp-l);
}
.layout-map-warning {
margin-block-end: var(--sp-m);
}

155
frontend/src/app/main/ui/shortcuts.cljs
View File

@ -34,41 +34,40 @@
(def ^:private modifier-keys
#{"Control" "Shift" "Alt" "Meta"})
(def ^:private punctuation-codes
#{"Minus" "Equal" "BracketLeft" "BracketRight" "Backslash"
"Semicolon" "Quote" "Backquote" "Comma" "Period" "Slash"})
(def ^:private key-name-map
{"ArrowUp" "up"
"Up" "up"
"ArrowDown" "down"
"Down" "down"
"ArrowLeft" "left"
"Left" "left"
"ArrowRight" "right"
"Right" "right"
"Escape" "escape"
"Esc" "escape"
"Enter" "enter"
"Backspace" "backspace"
"Delete" "del"
"Del" "del"
"Insert" "ins"
"Ins" "ins"
"Tab" "tab"
"Home" "home"
"End" "end"
"PageUp" "pageup"
"PageDown" "pagedown"
"CapsLock" "capslock"
"Spacebar" "space"
"Control" "ctrl"
"Shift" "shift"
"Alt" "alt"
"Meta" "meta"
" " "space"})
;; Mousetrap (the runtime shortcut matcher, vendored at
;; frontend/packages/mousetrap) resolves a keydown positionally - via keyCode
;; - ONLY for this punctuation set (its _KEYCODE_MAP); it falls back to
;; `event.key.toLowerCase()` (glyph-based) for letters and digits, which
;; aren't in that map. Recording punctuation from `code` therefore matches
;; what Mousetrap triggers regardless of layout (e.g. the key next to Right
;; Shift on a US layout is always "Period", whatever glyph it types on a
;; given ISO layout). With modifiers, letters and digits come from keyCode
;; like Mousetrap's keydown match (see `letter-or-digit`); otherwise from the
;; glyph, like its keypress match.
(def ^:private code-name-map
{"Minus" "-"
"Equal" "="
"BracketLeft" "["
"BracketRight" "]"
"Backslash" "\\"
"Semicolon" ";"
"Quote" "'"
"Backquote" "`"
"Comma" ","
"Period" "."
"Slash" "/"})
(defn- event->native
"Unwraps a goog.events.BrowserEvent to the native event, since the
wrapper copies `key` but not `code`."
@ -77,42 +76,100 @@
(or (.getBrowserEvent event) event)
event))
(defn- letter-or-digit
"Letter or digit named by keyCode when a modifier is held, ignoring the glyph
it makes the key type (Option+Shift+5 types \"fi\" on macOS)."
(defn- alt-graph?
[^js event]
(let [key-code (.-keyCode event)]
(when (and (or (.-ctrlKey event) (.-metaKey event)
(.-altKey event) (.-shiftKey event))
(or (<= 48 key-code 57)
(<= 65 key-code 90)))
(.toLowerCase (js/String.fromCharCode key-code)))))
(boolean
(or (when (fn? (.-getModifierState event))
(.getModifierState event "AltGraph"))
(let [native (event->native event)]
(when (and (not (identical? native event))
(fn? (.-getModifierState native)))
(.getModifierState native "AltGraph"))))))
(defn- numpad-code?
[code]
(and (string? code)
(str/starts-with? code "Numpad")
(= 1 (count (subs code 6)))))
(defn- numpad-alias
[code]
(let [suffix (subs code 6)]
(when (re-matches #"[0-9]" suffix)
(str "num" suffix))))
(defn- punctuation-alias
"Resolves punctuation by physical position through the shared layout
map (`app.main.data.shortcuts/layout-map`, also pushed into Mousetrap),
so recording and dispatch use the same token. Without a map the typed
glyph applies, which keeps both sides consistent."
[code key]
(let [mapped (when (some? @ds/layout-map)
(.get ^js @ds/layout-map code))]
(if (string? mapped)
(str/lower mapped)
(str/lower key))))
(defn- digit-code?
[code]
(and (string? code)
(boolean (re-matches #"Digit[0-9]" code))))
(defn- digit-alias
"Shifted digits resolve to the digit itself: with Shift held, Digit0
types ) on US layouts but 0 on AZERTY, so shift+0 must mean the
physical digit key everywhere, like shift+z means z."
[^js event code key]
(when (and (.-shiftKey ^js event)
(digit-code? code)
(= 1 (count key)))
(subs code 5)))
(defn- normalize-key
"Normalizes `event.key` to the lowercase canonical token mousetrap
matches on. Letters and unshifted digits resolve by glyph (`a`, not
`q` on AZERTY); a digit row key held with Shift resolves to the digit
itself (`shift+0`, never `shift+)`); punctuation resolves by physical
position through the layout map, so `Shift+*` on a Spanish layout
records `shift++`, the primary key the user sees."
[^js event]
(let [key (.-key event)]
(when (and key (not (contains? modifier-keys key)))
(or (get key-name-map key)
(get code-name-map (.-code (event->native event)))
(letter-or-digit event)
(some-> key .toLowerCase)))))
(let [native (event->native event)
key (.-key ^js event)
code (.-code ^js native)]
(when (and (string? key) (not (contains? modifier-keys key)))
(cond
;; Numpad digits keep their own `numN` bindings, like mousetrap.
(numpad-code? code)
(numpad-alias code)
;; Punctuation: primary key of the physical position.
(contains? punctuation-codes code)
(punctuation-alias code key)
:else
(or (get key-name-map key)
;; Shifted digit row: the digit itself, on every layout.
(digit-alias event code key)
(str/lower key))))))
(defn- event->modifier-parts
[^js event]
(cond-> []
(and (.-ctrlKey event)
(not (cf/check-platform? :macos)))
(conj "ctrl")
(let [alt-graph? (alt-graph? event)]
(cond-> []
(and (.-ctrlKey event)
(not (cf/check-platform? :macos))
(not alt-graph?))
(conj "ctrl")
(and (.-metaKey event)
(cf/check-platform? :macos))
(conj "command")
(and (.-metaKey event)
(cf/check-platform? :macos))
(conj "command")
(.-altKey event)
(conj "alt")
(.-altKey event)
(conj "alt")
(.-shiftKey event)
(conj "shift")))
(.-shiftKey event)
(conj "shift"))))
(defn- keyboard-event->mousetrap
[^js event]

64
frontend/src/app/main/ui/workspace/shapes/frame/dynamic_modifiers.cljs
View File

@ -9,6 +9,7 @@
[app.common.data :as d]
[app.common.data.macros :as dm]
[app.common.files.helpers :as cfh]
[app.common.geom.image-crop :as gic]
[app.common.geom.matrix :as gmt]
[app.common.geom.point :as gpt]
[app.common.geom.rect :as grc]
@ -20,6 +21,7 @@
[app.util.debug :as dbg]
[app.util.dom :as dom]
[app.util.timers :as ts]
[cuerdas.core :as str]
[rumext.v2 :as mf]))
(defn get-shape-node
@ -31,6 +33,14 @@
base-node
(dom/query base-node (dm/str "#shape-" id)))))
(defn- get-fill-image-nodes
"Retrieve the pictures of the shape image fills, inside its own fill
patterns"
[shape shape-node]
(when (or (some :fill-image (:fills shape))
(some? (:fill-image shape)))
(dom/query-all shape-node ":scope > defs > pattern image")))
(defn get-nodes
"Retrieve the DOM nodes to apply the matrix transformation"
[base-node {:keys [id parent-id] :as shape}]
@ -43,11 +53,13 @@
masking-child? (:masking-child? (meta shape))]
(cond
frame?
[shape-node
(dom/query shape-node ".frame-children")
(dom/query (dm/str "#thumbnail-container-" id))
(dom/query (dm/str "#thumbnail-" id))
(dom/query (dm/str "#frame-title-" id))]
(d/concat-vec
[shape-node
(dom/query shape-node ".frame-children")
(dom/query (dm/str "#thumbnail-container-" id))
(dom/query (dm/str "#thumbnail-" id))
(dom/query (dm/str "#frame-title-" id))]
(get-fill-image-nodes shape shape-node))
;; For groups we don't want to transform the whole group but only
;; its filters/masks
@ -72,9 +84,38 @@
(dom/query (dm/str "#frame-title-" id)))]
:else
[shape-node
(when (dbg/enabled? :shape-titles)
(dom/query (dm/str "#frame-title-" id)))]))))
(d/concat-vec
[shape-node
(when (dbg/enabled? :shape-titles)
(dom/query (dm/str "#frame-title-" id)))]
(get-fill-image-nodes shape shape-node))))))
(defn- fill-image-matrix
"Return the matrix that keeps the picture of an image fill `node` in
place while `modifiers` change the fill transform (resize with the mod
key). Return nil when the modifiers do not change the fill."
[shape modifiers node]
(let [changes (->> (:structure-parent modifiers)
(filter #(= :change-property (:type %)))
(d/index-by :property :value))
selrect (dm/get-prop shape :selrect)
id (dom/get-attribute node "id")
[transform transform' :as change]
(if (some? id)
;; Fill images have an id ending with the fill index
(when-let [fills' (get changes :fills)]
(let [index (d/parse-integer (last (str/split id "-")))]
[(dm/get-in (nth (:fills shape) index nil) [:fill-image :transform])
(dm/get-in (nth fills' index nil) [:fill-image :transform])]))
(when-let [fill-image' (get changes :fill-image)]
[(dm/get-in shape [:fill-image :transform])
(get fill-image' :transform)]))]
(when (some? change)
(gic/preview-image-matrix transform transform'
(dm/get-prop selrect :width)
(dm/get-prop selrect :height)))))
(defn transform-region!
[node modifiers]
@ -167,6 +208,13 @@
(= (dom/get-tag-name node) "pattern")
(set-transform-att! node "patternTransform" transform)
;; Picture of an image fill: it gets the shape transform
;; through the shape node, and only needs its own when the
;; fill transform changes
(= (dom/get-tag-name node) "image")
(when-let [matrix (fill-image-matrix shape modifiers node)]
(set-transform-att! node "transform" matrix))
(and (some? transform) (some? node))
(set-transform-att! node "transform" transform)))))))

1
frontend/src/app/main/ui/workspace/shapes/text/v3_editor.cljs
View File

@ -624,6 +624,7 @@
(fn []
(on-blur)
(st/emit! dw/hide-context-menu
(dwt/v3-stop-text-layout-preview shape-id)
(dwu/commit-undo-transaction shape-id))
(text-editor/text-editor-dispose)
(wasm.api/request-render-preserving-target "text-editor-dispose"))))

14
frontend/src/app/main/ui/workspace/tokens/management.cljs
View File

@ -26,6 +26,13 @@
[cuerdas.core :as str]
[rumext.v2 :as mf]))
(defn token-in-path?
"True when `token-name` is exactly `path` or nested below it. This keeps an
ancestor token inside the operations of its own path-based group."
[path token-name]
(or (= token-name path)
(str/starts-with? token-name (str path ctob/token-separator))))
(defn- get-sorted-token-groups
"Separate token-types into groups of `empty` or `filled` depending if
tokens exist for that type. Sort each group alphabetically (by their type)."
@ -130,9 +137,9 @@
(fn [tokens-filtered-by-type node]
(->> tokens-filtered-by-type
(filter (fn [token]
(let [token-path (cpn/split-path (:name token) :separator ".")]
(let [token-path (cpn/split-path (:name token) :separator ctob/token-separator)]
(and (> (count token-path) 0)
(str/starts-with? (:name token) (str (:path node) ".")))))))))
(token-in-path? (:path node) (:name token)))))))))
;; Filter tokens by their path and return their ids
filter-tokens-by-path-ids
@ -142,7 +149,8 @@
(->> selected-token-set-tokens
(filter (fn [token]
(let [[_ token-value] token]
(and (= (:type token-value) type) (str/starts-with? (:name token-value) (str path "."))))))
(and (= (:type token-value) type)
(token-in-path? path (:name token-value))))))
(mapv (fn [token]
(let [[_ token-value] token]
(:id token-value)))))))

185
frontend/src/app/main/ui/workspace/tokens/management/token_tree.cljs
View File

@ -31,8 +31,56 @@
[:on-node-context-menu {:optional true} fn?]
[:can-edit {:optional true} :boolean]])
(defn- leaf-node?
"A tree node is a leaf when it holds a token and has no children."
[node]
(and (:leaf node) (not (seq (:children node)))))
(defn- sort-nodes
"Sorts tree nodes so that leaf tokens come before groups, sorting each
set naturally by name."
[nodes]
(let [[leafs groups]
(reduce (fn [[leafs groups] node]
(if (leaf-node? node)
[(conj leafs node) groups]
[leafs (conj groups node)]))
[[] []]
nodes)]
(concat (d/natural-sort-by :name leafs)
(d/natural-sort-by :name groups))))
(def ^:private schema:leaf-node
[:map
[:leaf :any]
[:tokens-lib {:optional true} :any]
[:selected-token-set-id {:optional true} :any]
[:selected-shapes :any]
[:is-selected-inside-layout {:optional true} :boolean]
[:active-theme-tokens {:optional true} :any]
[:can-edit {:optional true} :boolean]
[:on-token-pill-click {:optional true} fn?]
[:on-pill-context-menu {:optional true} fn?]])
(mf/defc leaf-node*
{::mf/private true
::mf/schema schema:leaf-node}
[{:keys [leaf tokens-lib selected-token-set-id selected-shapes
is-selected-inside-layout active-theme-tokens can-edit
on-token-pill-click on-pill-context-menu]}]
(let [id (:id leaf)
token (ctob/get-token tokens-lib selected-token-set-id id)]
[:> token-pill* {:token token
:selected-shapes selected-shapes
:is-selected-inside-layout is-selected-inside-layout
:active-theme-tokens active-theme-tokens
:on-click on-token-pill-click
:can-edit can-edit
:on-context-menu on-pill-context-menu}]))
(mf/defc folder-node*
{::mf/schema schema:folder-node}
{::mf/private true
::mf/schema schema:folder-node}
[{:keys [node
type
folded-token-paths
@ -47,20 +95,12 @@
can-edit]}]
(let [full-path (str (name type) "." (:path node))
is-folder-expanded (not (contains? (set (or folded-token-paths [])) full-path))
children (:children node)
sorted-children
(mf/with-memo [children]
(let [[leafs groups]
(reduce (fn [[l g] item]
(if (:leaf item)
[(conj l item) g]
[l (conj g item)]))
[[] []]
children)
sorted-leafs (d/natural-sort-by :name leafs)
sorted-groups (d/natural-sort-by :name groups)]
(concat sorted-leafs sorted-groups)))
(sort-nodes children))
swap-folder-expanded
(mf/use-fn
@ -80,40 +120,50 @@
:expanded is-folder-expanded
:aria-expanded is-folder-expanded
:aria-controls (str "folder-children-" (:path node))
:is-expandable (not (:leaf node))
:is-expandable (boolean (seq children))
:on-toggle-expand swap-folder-expanded
:on-context-menu node-context-menu-prep}]
(when is-folder-expanded
[:div {:class (stl/css :folder-children-wrapper)
:id (str "folder-children-" (:path node))}
(when (seq children)
(for [child sorted-children]
(if (not (:leaf child))
[:ul {:class (stl/css :node-parent)
:key (:path child)}
[:> folder-node* {:type type
:node child
:folded-token-paths folded-token-paths
:selected-shapes selected-shapes
:is-selected-inside-layout is-selected-inside-layout
:active-theme-tokens active-theme-tokens
:on-token-pill-click on-token-pill-click
:on-pill-context-menu on-pill-context-menu
:on-node-context-menu on-node-context-menu
:tokens-lib tokens-lib
:can-edit can-edit
:selected-token-set-id selected-token-set-id}]]
(let [id (:id (:leaf child))
token (ctob/get-token tokens-lib selected-token-set-id id)]
[:> token-pill*
{:key id
:token token
:selected-shapes selected-shapes
:is-selected-inside-layout is-selected-inside-layout
:active-theme-tokens active-theme-tokens
:on-click on-token-pill-click
:can-edit can-edit
:on-context-menu on-pill-context-menu}]))))])]))
(when-let [leaf (:leaf node)]
[:> leaf-node* {:key (:id leaf)
:leaf leaf
:tokens-lib tokens-lib
:selected-token-set-id selected-token-set-id
:selected-shapes selected-shapes
:is-selected-inside-layout is-selected-inside-layout
:active-theme-tokens active-theme-tokens
:can-edit can-edit
:on-token-pill-click on-token-pill-click
:on-pill-context-menu on-pill-context-menu}])
(for [child sorted-children]
(if (leaf-node? child)
[:> leaf-node* {:key (:id (:leaf child))
:leaf (:leaf child)
:tokens-lib tokens-lib
:selected-token-set-id selected-token-set-id
:selected-shapes selected-shapes
:is-selected-inside-layout is-selected-inside-layout
:active-theme-tokens active-theme-tokens
:can-edit can-edit
:on-token-pill-click on-token-pill-click
:on-pill-context-menu on-pill-context-menu}]
[:ul {:class (stl/css :node-parent)
:key (:path child)}
[:> folder-node* {:type type
:node child
:folded-token-paths folded-token-paths
:selected-shapes selected-shapes
:is-selected-inside-layout is-selected-inside-layout
:active-theme-tokens active-theme-tokens
:on-token-pill-click on-token-pill-click
:on-pill-context-menu on-pill-context-menu
:on-node-context-menu on-node-context-menu
:tokens-lib tokens-lib
:can-edit can-edit
:selected-token-set-id selected-token-set-id}]]))])]))
(def ^:private schema:token-tree
[:map
@ -146,42 +196,35 @@
on-node-context-menu
can-edit
can-edit-tokens]}]
(let [separator "."
raw-tree
(let [raw-tree
(mf/with-memo [tokens]
(cpn/build-tree-root tokens separator))
(cpn/build-tree-root tokens ctob/token-separator))
on-node-context-menu (mf/use-fn
(mf/deps can-edit-tokens on-node-context-menu)
(fn [event node]
(when can-edit-tokens
(on-node-context-menu event node))))
ordered-nodes
(mf/with-memo [raw-tree]
(sort-nodes raw-tree))
on-node-context-menu
(mf/use-fn
(mf/deps can-edit-tokens on-node-context-menu)
(fn [event node]
(when can-edit-tokens
(on-node-context-menu event node))))]
ordered-nodes (mf/with-memo [raw-tree]
(let [[leafs groups]
(reduce (fn [[l g] item]
(if (:leaf item)
[(conj l item) g]
[l (conj g item)]))
[[] []]
raw-tree)
sorted-leafs (d/natural-sort-by :name leafs)
sorted-groups (d/natural-sort-by :name groups)]
(concat sorted-leafs sorted-groups)))]
[:div {:class (stl/css :token-tree-wrapper)}
(for [node ordered-nodes]
(if (:leaf node)
(let [token (ctob/get-token tokens-lib selected-token-set-id (get-in node [:leaf :id]))]
[:> token-pill*
{:token token
:key (:id (:leaf node))
:selected-shapes selected-shapes
:is-selected-inside-layout is-selected-inside-layout
:active-theme-tokens active-theme-tokens
:on-click on-token-pill-click
:can-edit can-edit
:on-context-menu on-pill-context-menu}])
;; Render segment folder
(if (leaf-node? node)
[:> leaf-node* {:key (:id (:leaf node))
:leaf (:leaf node)
:tokens-lib tokens-lib
:selected-token-set-id selected-token-set-id
:selected-shapes selected-shapes
:is-selected-inside-layout is-selected-inside-layout
:active-theme-tokens active-theme-tokens
:can-edit can-edit
:on-token-pill-click on-token-pill-click
:on-pill-context-menu on-pill-context-menu}]
[:ul {:class (stl/css :node-parent)
:key (:path node)}
[:> folder-node* {:node node

2
frontend/src/app/main/ui/workspace/tokens/sidebar.scss
View File

@ -28,7 +28,7 @@
overflow-y: auto;
scrollbar-gutter: stable;
position: relative;
padding-block: var(--sp-xs) var(--sp-l);
padding-block-start: var(--sp-xs);
}
.tokens-section-wrapper {

5
frontend/src/app/render_wasm/api.cljs
View File

@ -2943,7 +2943,7 @@
{:x x :y y :width w :height h})))))
(defn render-shape-pdf
[shape-id scale]
[shape-id]
(when (initialized?)
(let [buffer (uuid/get-u32 shape-id)
@ -2952,8 +2952,7 @@
(aget buffer 0)
(aget buffer 1)
(aget buffer 2)
(aget buffer 3)
scale)
(aget buffer 3))
heap (mem/get-heap-u8)
heapu32 (mem/get-heap-u32)

108
frontend/test/frontend_tests/data/exports_assets_test.cljs
View File

@ -9,12 +9,15 @@
[app.common.uuid :as uuid]
[app.main.data.exports.assets :as de]
[app.main.data.persistence :as dwp]
[app.main.errors :as errors]
[app.main.repo :as repo]
[app.main.store :as st]
[app.util.dom :as dom]
[app.util.i18n :as i18n]
[app.util.websocket :as ws]
[beicon.v2.core :as rx]
[cljs.test :as t :include-macros true]
[frontend-tests.helpers.async :as async]
[frontend-tests.helpers.events :as the]
[frontend-tests.helpers.mock :as mock]
[potok.v2.core :as ptk]))
@ -55,7 +58,7 @@
(rx/of {:filename "export.png"
:mtype "image/png"
:uri "blob:export"})))
dwp/force-persist-and-wait (mock/stub (fn [_] (rx/of ::force-persisted)))
dwp/force-persist-files-and-wait (mock/stub (fn [_ _] (rx/of ::force-persisted)))
dom/trigger-download-uri (mock/stub (fn [& _] nil))}
(fn [done']
(let [completed (fn [_state]
@ -97,3 +100,106 @@
(de/request-multiple-export {:exports exports})
:the/end)))
done))))
(defn- export-store
"A store for the export tests that records store errors in `store-errors`."
[state store-errors]
(ptk/store {:state state :on-error #(swap! store-errors conj %)}))
(defn- unsaved-state
"Test state whose save queue still holds a change of `file-id`."
[file-id]
(let [commit-id (uuid/next)]
(assoc (test-state)
:persistence {:queue (conj #queue [] commit-id)
:index {commit-id {:id commit-id :file-id file-id}}})))
;; Scenario: the exporter fails a single export. Proves: the user gets a
;; toast and stays in the workspace; the failure is not raised to the global
;; error handler, which would replace the workspace with an error page.
(t/deftest ^:async failed-simple-export-shows-a-toast-instead-of-raising
(let [store-errors (atom [])
flashed (atom [])
store (export-store (test-state) store-errors)]
(await
(mock/with-mocks*
{repo/cmd! (mock/stub (fn [_ _]
(rx/throw (ex-info "exporter failed"
{:type :server-error
:code :unhandled}))))
dwp/force-persist-files-and-wait (mock/stub (fn [_ _] (rx/empty)))
errors/flash (fn [& {:keys [hint type]}]
(swap! flashed conj [hint type]))
i18n/tr (mock/stub #(str "translated:" %))}
(ptk/emit! store (de/request-simple-export {:export export}))
(await (async/wait-for #(seq @flashed) "the failure is shown"))
(await (async/settle))
(t/is (= [["translated:errors.export-failed" :unhandled]] @flashed)
"an exporter defect reports as unhandled")
(t/is (empty? @store-errors) "nothing reaches the global error handler")
(t/is (nil? (:export @store)) "the export state is cleared")))))
;; Scenario: the export request of a multiple export fails. Proves: the user
;; gets a toast, and the leave-page warning the export turned on is released
;; even though no progress message will ever arrive.
(t/deftest ^:async failed-multiple-export-shows-a-toast-and-releases-the-leave-warning
(let [store-errors (atom [])
flashed (atom [])
tasks (atom #{})
store (export-store (test-state) store-errors)]
(await
(mock/with-mocks*
{repo/cmd! (mock/stub (fn [_ _]
(rx/throw (ex-info "exporter failed"
{:type :server-error}))))
dwp/force-persist-files-and-wait (mock/stub (fn [_ _] (rx/empty)))
ws/get-rcv-stream (mock/stub (fn [_] (rx/subject)))
st/ongoing-tasks tasks
errors/flash (fn [& {:keys [hint]}] (swap! flashed conj hint))
i18n/tr (mock/stub #(str "translated:" %))}
(ptk/emit! store (de/request-multiple-export {:exports [(assoc export :enabled true)]}))
(await (async/wait-for #(seq @flashed) "the failure is shown"))
(await (async/wait-for #(not (contains? @tasks :export))
"the leave-page warning is released"))
(t/is (= ["translated:errors.export-failed"] @flashed))
(t/is (empty? @store-errors) "nothing reaches the global error handler")))))
;; Scenario: an export starts while its file has a change not saved yet.
;; Proves: the user is warned that the export may miss it, and the export
;; runs anyway.
(t/deftest ^:async export-warns-when-the-file-has-unsaved-changes
(let [file-id (uuid/next)
downloaded (atom false)
store (export-store (unsaved-state file-id) (atom []))]
(await
(mock/with-mocks*
{repo/cmd! (mock/stub (fn [_ _]
(rx/of {:filename "export.png"
:mtype "image/png"
:uri "blob:export"})))
dwp/force-persist-files-and-wait (mock/stub (fn [_ _] (rx/empty)))
dom/trigger-download-uri (mock/stub (fn [& _] (reset! downloaded true)))
i18n/tr (mock/stub #(str "translated:" %))}
(ptk/emit! store (de/request-simple-export {:export (assoc export :file-id file-id)}))
(await (async/wait-for #(true? @downloaded) "the export runs anyway"))
(t/is (= "translated:workspace.export.unsaved-changes"
(get-in @store [:notification :content])))
(t/is (= :warning (get-in @store [:notification :level])))))))
;; Scenario: an export starts with every change of its file saved. Proves:
;; no warning is shown.
(t/deftest ^:async export-shows-no-warning-when-the-file-is-saved
(let [downloaded (atom false)
store (export-store (test-state) (atom []))]
(await
(mock/with-mocks*
{repo/cmd! (mock/stub (fn [_ _]
(rx/of {:filename "export.png"
:mtype "image/png"
:uri "blob:export"})))
dwp/force-persist-files-and-wait (mock/stub (fn [_ _] (rx/empty)))
dom/trigger-download-uri (mock/stub (fn [& _] (reset! downloaded true)))}
(ptk/emit! store (de/request-simple-export {:export (assoc export :file-id (uuid/next))}))
(await (async/wait-for #(true? @downloaded) "the export runs"))
(await (async/settle))
(t/is (nil? (:notification @store)))))))

67
frontend/test/frontend_tests/data/exports_files_test.cljs
View File

@ -5,18 +5,23 @@
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns frontend-tests.data.exports-files-test
"Exporting files as jobs: one file, one job and one answer, with the
progress of the job reported while it runs."
"Exporting files as jobs (one file, one job and one answer, with the progress
of the job reported while it runs), and opening the `.penpot` download dialog
(the pending changes are sent first, and the dialog is told when some are
still not saved)."
(:require
[app.common.uuid :as uuid]
[app.main.data.exports.files :as fexp]
[app.main.data.modal :as modal]
[app.main.data.persistence :as dps]
[app.main.repo :as rp]
[app.main.store :as st]
[app.util.websocket :as ws]
[beicon.v2.core :as rx]
[cljs.test :as t :include-macros true]
[frontend-tests.helpers.async :as hva]
[frontend-tests.helpers.mock :as mock]))
[frontend-tests.helpers.mock :as mock]
[potok.v2.core :as ptk]))
(defn- file
[name]
@ -284,3 +289,59 @@
(await done)
(t/is (= {:file-id (:id target) :cancelled true} (last @seen)))
(t/is (not-any? :uri @seen))))))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; DOWNLOAD DIALOG
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defn- file-param
[file-id]
{:id file-id :name "File" :project-id (uuid/next) :is-shared false})
(defn- with-queued-change
"Adds to `state` a save queue that still holds a change of `file-id`."
[state file-id]
(let [commit-id (uuid/next)]
(assoc state :persistence {:queue (conj #queue [] commit-id)
:index {commit-id {:id commit-id :file-id file-id}}})))
(defn- ^:async dialog-props
"Opens the download dialog for `file-id` from `state` and returns the
props the dialog is shown with, plus the file ids the save wait got."
[state file-id]
(let [store (ptk/store {:state state :on-error #(t/is false (str %))})
waited (atom nil)]
(await
(mock/with-mocks*
{rp/cmd! (mock/stub (fn [_ _] (rx/of false)))
dps/force-persist-files-and-wait (mock/stub (fn [file-ids _]
(reset! waited file-ids)
(rx/empty)))}
(ptk/emit! store (fexp/open-export-dialog [(file-param file-id)]))
(await (hva/wait-for #(some? (get-in @store [::modal/modal :props]))
"the dialog opens"))))
(assoc (get-in @store [::modal/modal :props]) ::waited @waited)))
;; Scenario: the user downloads a file whose changes are all saved. Proves:
;; the pending changes of that file are sent first, and the dialog shows no
;; warning.
(t/deftest ^:async download-of-a-saved-file-has-no-warning
(let [file-id (uuid/next)
props (await (dialog-props {} file-id))]
(t/is (= #{file-id} (::waited props)) "the save wait covers the downloaded file")
(t/is (false? (:unsaved props)))))
;; Scenario: the file still has a change the server does not have after the
;; wait. Proves: the dialog is told, so it can warn that the download will
;; not include it.
(t/deftest ^:async download-of-a-file-with-unsaved-changes-warns
(let [file-id (uuid/next)
props (await (dialog-props (with-queued-change {} file-id) file-id))]
(t/is (true? (:unsaved props)))))
;; Scenario: another file has unsaved changes. Proves: only the downloaded
;; file's changes count.
(t/deftest ^:async unsaved-changes-of-another-file-do-not-warn
(let [file-id (uuid/next)
props (await (dialog-props (with-queued-change {} (uuid/next)) file-id))]
(t/is (false? (:unsaved props)))))

62
frontend/test/frontend_tests/data/persistence_retry_test.cljs
View File

@ -199,6 +199,68 @@
(t/deftest ^:async cloudflare-origin-timeout-save-is-retried
(await (check-gateway-response-retries 524)))
(defn- check-busy-server-response-retries
"Answers the first send with `response`, a refusal the backend wrote
because it was busy, built by the real `rp/handle-response`, and the retry
with a valid revision. The save was never applied, so it must take the
retry path and land, never halt in `:error`."
[response]
(let [calls (atom 0)]
(with-persistence
(^:async fn [{:keys [file-id failures requests store]}]
(await
(mock/with-mocks*
{rx/timer (mock/stub instant-backoff)}
(ptk/emit! store (local-commit file-id) ::dps/force-persist)
(await (async/wait-for #(and (= :saved (get-in @store [:persistence :status]))
(empty? (get-in @store [:persistence :queue])))
(str "a busy " (:status response) " is retried and the save lands")))
(t/is (= 2 (count @requests)) "the busy answer plus one retry")
(t/is (empty? @failures) "a retried busy answer never reaches the failure toast"))))
(fn [_ _]
(if (= 1 (swap! calls inc))
(rp/handle-response (assoc response :uri "https://penpot.example.com/api/main/methods/update-file"))
(rx/of {:revn 1}))))))
;; Variant: the save waited too long for a concurrency slot (no body).
(t/deftest ^:async concurrency-limit-save-is-retried
(await (check-busy-server-response-retries {:status 429 :body nil})))
;; Variant: the database cancelled the save while it waited for the file lock.
(t/deftest ^:async statement-timeout-save-is-retried
(await (check-busy-server-response-retries
{:status 504
:body {:type :server-error
:code :statement-timeout
:hint "The operation took too long and was cancelled"}})))
;; Variant: the database ended the save's transaction after it sat idle.
(t/deftest ^:async idle-transaction-timeout-save-is-retried
(await (check-busy-server-response-retries
{:status 504
:body {:type :server-error
:code :idle-in-transaction-timeout
:hint "The transaction was idle too long and was cancelled"}})))
;; Scenario: the backend fails the save for any other reason. Proves: only
;; the busy-server codes retry; a server defect stops saving at once.
(t/deftest ^:async other-server-errors-stay-terminal
(await
(with-persistence
(^:async fn [{:keys [file-id requests store]}]
(await
(mock/with-mocks*
{rx/timer (mock/stub instant-backoff)}
(ptk/emit! store (local-commit file-id) ::dps/force-persist)
(await (async/wait-for #(= :error (get-in @store [:persistence :status]))
"the save fails"))
(t/is (= :server-error (get-in @store [:persistence :error :cause-type])))
(t/is (= 1 (count @requests)) "no retry"))))
(fn [_ _]
(rp/handle-response {:status 500
:body {:type :server-error :code :unhandled}
:uri "https://penpot.example.com/api/main/methods/update-file"})))))
;; Scenario: every send fails transiently. The 2s / 8s / 20s retries fire
;; and then the failure falls through to the exact terminal path: `:error`
;; carrying the cause, queue intact, one flash. Proves: the budget bounds

148
frontend/test/frontend_tests/data/persistence_test.cljs
View File

@ -21,10 +21,12 @@
[app.common.types.shape :as cts]
[app.common.uuid :as uuid]
[app.main.data.changes :as dch]
[app.main.data.event :as ev]
[app.main.data.persistence :as dps]
[app.main.data.render-wasm :as drw]
[app.main.data.workspace :as dw]
[app.main.errors :as errors]
[app.main.refs :as refs]
[app.main.repo :as rp]
[app.main.store :as st]
[app.util.i18n :as i18n]
@ -32,6 +34,7 @@
[cljs.test :as t :include-macros true]
[frontend-tests.helpers.async :as async]
[frontend-tests.helpers.mock :as mock]
[okulary.core :as l]
[potok.v2.core :as ptk]))
(t/deftest reopening-restores-only-the-retained-edits-of-the-loaded-file
@ -693,6 +696,19 @@
(t/is (not (dps/transient-error? {:type :persistence :code :save-permission-denied})))
(t/is (not (dps/transient-error? {}))))
;; A busy backend refused or rolled back the save, so it is safe to resend:
;; a rate or concurrency limit, or a database timeout. Any other server
;; error stays terminal.
(t/deftest busy-server-classification
(t/is (dps/transient-error? {:type :rate-limit}))
(t/is (dps/transient-error? {:type :persistence :cause-type :rate-limit :code :rate-limited}))
(t/is (dps/transient-error? {:type :server-error :code :statement-timeout}))
(t/is (dps/transient-error? {:type :persistence :cause-type :server-error :code :statement-timeout}))
(t/is (dps/transient-error? {:type :persistence :cause-type :server-error :code :idle-in-transaction-timeout}))
(t/is (not (dps/transient-error? {:type :server-error})))
(t/is (not (dps/transient-error? {:type :persistence :cause-type :server-error :code :unhandled})))
(t/is (not (dps/transient-error? {:type :persistence :cause-type :validation :code :statement-timeout}))))
;; The transient set is owned by `repo/retryable-types`: every transport
;; type retryable at the HTTP layer must also be transient for the save
;; pipeline, in both the direct and the wrapped (`persistence-failed`)
@ -988,3 +1004,135 @@
(await (async/wait-for #(not (contains? @tasks :persistence))
"the release turns the guard off"))
(t/is (= :saved (get-in @store [:persistence :status])))))))
;; Scenario: an edit waits in the 3 s buffer and the user switches tabs or
;; windows. Proves: leaving sends the buffered edit at once, well before
;; the buffer would (the bounded `wait-for` gives up long before 3 s).
(t/deftest ^:async leaving-the-tab-sends-buffered-edits-at-once
(let [leave-s (rx/subject)]
(await
(mock/with-mocks*
{dps/leave-signals (fn [] leave-s)}
(await
(with-persistence
(^:async fn [{:keys [file-id requests store]}]
(ptk/emit! store (local-commit file-id))
(await (async/settle))
(t/is (empty? @requests) "the edit waits in the buffer")
(rx/push! leave-s :hidden)
(await (async/wait-for #(= 1 (count @requests))
"leaving sends the edit at once"))
(t/is (= 1 (count @requests))))))))))
;; Scenario: nothing is buffered when the user leaves. Proves: a leave
;; signal alone sends nothing.
(t/deftest ^:async leaving-with-nothing-buffered-sends-nothing
(let [leave-s (rx/subject)]
(await
(mock/with-mocks*
{dps/leave-signals (fn [] leave-s)}
(await
(with-persistence
(^:async fn [{:keys [requests store]}]
(rx/push! leave-s :hidden)
(await (async/settle))
(t/is (empty? @requests))
(t/is (nil? (get-in @store [:persistence :status]))))))))))
(defn- wait-files-saved
"Emits into `store` an event that runs `force-persist-files-and-wait` for
`file-ids` and then flips `done`."
[store file-ids timeout-ms done]
(ptk/emit! store
(ptk/reify ::wait-files-saved
ptk/WatchEvent
(watch [_ _ _]
(rx/concat
(dps/force-persist-files-and-wait file-ids timeout-ms)
(rx/of (ptk/reify ::files-saved
ptk/EffectEvent
(effect [_ _ _] (reset! done true)))))))))
;; Scenario: an edit waits in the buffer when an export asks for its file
;; to be saved. Proves: the wait sends the edit and only ends once the save
;; lands.
(t/deftest ^:async waiting-for-a-file-sends-and-waits-for-its-edits
(let [done (atom false)]
(await
(with-persistence
(^:async fn [{:keys [file-id response requests store]}]
(await
(mock/with-mocks*
{refs/persistence (l/derived :persistence store)}
(ptk/emit! store (local-commit file-id))
(wait-files-saved store #{file-id} 60000 done)
(await (async/wait-for #(= 1 (count @requests)) "the edit is sent at once"))
(await (async/settle))
(t/is (false? @done) "the wait holds while the save is in flight")
(rx/push! response {:revn 1})
(await (async/wait-for #(true? @done) "the landed save ends the wait"))
(t/is (not (dps/unsaved-changes? @store #{file-id}))))))))))
;; Scenario: a save of one file hangs while another file is exported.
;; Proves: the wait only covers the exported files.
(t/deftest ^:async waiting-for-a-file-ignores-other-files
(let [done (atom false)]
(await
(with-persistence
(^:async fn [{:keys [file-id requests store]}]
(await
(mock/with-mocks*
{refs/persistence (l/derived :persistence store)}
(ptk/emit! store (local-commit file-id) ::dps/force-persist)
(await (async/wait-for #(= 1 (count @requests)) "the other file's save hangs"))
(wait-files-saved store #{(uuid/next)} 60000 done)
(await (async/wait-for #(true? @done) "the wait ends at once")))))))))
;; Scenario: the file's save hangs past the wait. Proves: the wait gives up
;; without raising, and the file still reports unsaved changes.
(t/deftest ^:async waiting-for-a-file-gives-up-after-the-timeout
(let [done (atom false)]
(await
(with-persistence
(^:async fn [{:keys [file-id requests store]}]
(await
(mock/with-mocks*
{refs/persistence (l/derived :persistence store)}
(ptk/emit! store (local-commit file-id))
(wait-files-saved store #{file-id} 1 done)
(await (async/wait-for #(true? @done) "the wait times out"))
(t/is (= 1 (count @requests)))
(t/is (dps/unsaved-changes? @store #{file-id})))))))))
(defn- save-failed-events
"Starts collecting the `save-failed` audit events that `store` processes."
[store]
(let [events (atom [])]
(->> (ptk/input-stream store)
(rx/filter #(= ::ev/event (ptk/type %)))
(rx/map deref)
(rx/filter #(= "save-failed" (::ev/name %)))
(rx/subs! #(swap! events conj %)))
events))
;; Scenario: the backend rejects a save as invalid data. Proves: the user's
;; failure toast comes with one `save-failed` audit event that records why.
(t/deftest ^:async a-failed-save-emits-one-save-failed-event
(await
(with-persistence
(^:async fn [{:keys [file-id store]}]
(let [events (save-failed-events store)]
(ptk/emit! store (local-commit file-id) ::dps/force-persist)
(await (async/wait-for #(= :error (get-in @store [:persistence :status]))
"the save fails"))
(await (async/wait-for #(seq @events) "the event is emitted"))
(await (async/settle))
(t/is (= 1 (count @events)))
(t/is (= {:cause-type :validation
:code :referential-integrity
:retrying false
:queued-commits 1}
(select-keys (first @events)
[:cause-type :code :retrying :queued-commits])))))
(fn [_ _]
(rx/throw (ex-info "invalid" {:type :validation :code :referential-integrity}))))))

4
frontend/test/frontend_tests/runner.cljs
View File

@ -109,6 +109,7 @@
[frontend-tests.tokens.referenced-token-value-test]
[frontend-tests.tokens.style-dictionary-test]
[frontend-tests.tokens.token-errors-test]
[frontend-tests.tokens.token-path-test]
[frontend-tests.tokens.workspace-tokens-remap-test]
[frontend-tests.ui.check-updates-test]
[frontend-tests.ui.color-token-position-test]
@ -134,7 +135,6 @@
[frontend-tests.ui.settings-shortcuts-test]
[frontend-tests.ui.shape-filters-test]
[frontend-tests.ui.shortcuts-labels-test]
[frontend-tests.ui.shortcuts-recording-test]
[frontend-tests.ui.sidebar-scroll-test]
[frontend-tests.ui.stroke-menu-test]
[frontend-tests.ui.text-attrs-multiple-test]
@ -266,6 +266,7 @@
'frontend-tests.tokens.referenced-token-value-test
'frontend-tests.tokens.style-dictionary-test
'frontend-tests.tokens.token-errors-test
'frontend-tests.tokens.token-path-test
'frontend-tests.tokens.logic.tokens-status-test
'frontend-tests.tokens.workspace-tokens-remap-test
'frontend-tests.ui.check-updates-test
@ -294,7 +295,6 @@
'frontend-tests.ui.settings-shortcuts-test
'frontend-tests.ui.shape-filters-test
'frontend-tests.ui.shortcuts-labels-test
'frontend-tests.ui.shortcuts-recording-test
'frontend-tests.ui.sidebar-scroll-test
'frontend-tests.ui.stroke-menu-test
'frontend-tests.ui.text-attrs-multiple-test

Some files were not shown because too many files have changed in this diff Show More