mirror of
https://github.com/penpot/penpot.git
synced 2026-11-19 08:30:42 +00:00
140 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
3f6c69bb6a | Merge remote-tracking branch 'origin/staging' into develop | ||
|
|
38a9878981
|
🔥 Remove the unused SQLite-based binfile v2 format (#12947)
The SQLite-based v2 binfile package was never released, and its only caller was the developer REPL helper `app.srepl.binfile`, which exported and imported a whole team. With the format gone, the helper, the functions of `app.binfile.common` without a caller (`get-fonts`, `get-file-thumbnail`, `get-team`, `get-teams`, `collect-summary`) and the `sqlite-jdbc` dependency go with it. The public surface does not change: the import RPC accepts only versions 1 and 3, and the test that pins the rejection of version 2 stays as it is. AI-assisted-by: deepseek-v4.1-flash |
||
|
|
86eb0bf805 | Merge remote-tracking branch 'origin/staging' into develop | ||
|
|
e23ca02aa2
|
✨ Reference the file data storage object with a foreign key (#12769)
* ✨ Reference the file data storage object with a foreign key The file data pointer lived in `file_data.metadata.storage-ref-id`, a JSON key. The storage GC parsed it to find live references, and nothing stopped it from physically deleting an object a live row still pointed at: a failure in that scan lost the blob. Migration 0159 moves the pointer to a nullable `storage_object_id` column on `file_data`, backfills it from the metadata key, removes the key from the rows it backfills, indexes the column and rebuilds the foreign key to `storage_object(id)` as `NO ACTION DEFERRABLE INITIALLY IMMEDIATE`. `INITIALLY IMMEDIATE` is deliberate: the GC catches the violation inside the chunk transaction, while a deferred check would fire at commit, outside it, and the sweep would never move past the blocked chunk. The column is the only source of truth. The `storage-ref-id` entry is gone from the file metadata schema, and the single point that serializes the metadata drops it, so no write can resurrect it. `upsert!` does not query the row: the pointer of the row being replaced travels in its params, and the object is touched before the row is written. Fragments and snapshots pass nil because a modified pointer always gets a new id; the offload passes its own row; `insert-file!` drops it, since an INSERT never replaces a persisted object. `update-file!` takes the file map read from the database plus the fields to change and merges them, so the pointer comes from the read map and no caller can lose it by building a partial map. A map without `:data` raises `:missing-file-data`: it is not `assert`, which is compiled out in the deployed binary. The params cannot override the pointer or retarget the write with `:id`/`:file-id`. `bfc/file->response` strips the pointer from every response (`get-file`, `create-file`, `duplicate-file`, the view-only bundle, the team access request, the comments read, the library summary, the snapshot read and the debug download). A `file-data` object is authentication-only, so a leaked id would let any logged-in profile fetch the blob from `/assets/by-id`. `get-file-fragment` resolves through the file data backend, so a fragment on cold storage no longer hands nil to `blob/decode`, and answers only the keys of its result schema. The storage GC reference scan, the restore union and `objects-gc` read the column. The offload reads `SELECT fd.*`, so its rows go through a `decode-row` that decodes jsonb metadata and drops a JSON null, which the row schema rejected; a row without a readable object raises `:missing-storage-object` with the file and row ids. A chunk blocked by the foreign key (SQLState 23503) is a real anomaly, since the reference scan should have frozen the object: it stays an error with its own hint and the chunk is retried on the next run until the reference is fixed. The release is a coordinated restart, not a rolling deploy, and not rollback-safe for the rows it writes: the previous code writes and scans the metadata pointer, so a key it writes misses the backfill and removing it loses the reference. `operations/2.20.0-file-data-storage-object-id.md` builds the column, the index and the foreign key online and runs the backfill with every instance stopped, then marks the step as applied; a fresh or small instance does the whole job in the migration. The runbook keeps the validation scan out of the stopped window and adds a lock timeout and retry to the online statements. Tests cover the foreign key shape, the backfill and its corrupt and orphan guards, the rebuild of a missing index and of a wrong, non-deferrable or deferred constraint, the deletion refused by a live reference, the metadata cleanup, the offload row normalizations, the touched objects of the unshare and the restore, the response stripping on every call site and the chunk blocked by the foreign key. The backend memories and the developer guide document the pointer, the GC behavior and the two migration paths. Closes #12768 AI-assisted-by: deepseek-v4.1-flash AI-assisted-by: inkling-2-small-free * 🐛 Hold the file row lock across the migration write `get-file` applies the pending migrations in memory and, when it finds any, persists the result. That write read the row without a lock and `update-file!` is an unconditional UPDATE, so a save that committed in between was overwritten: the saved content, `revn` and `modified-at` went back, the next save from that client failed with `:revn-conflict` and the object the save left became an orphan the storage GC cannot revisit once it is frozen. The read now runs in a transaction of its own (`get-file` calls `get-file*` inside `db/tx-run!` and accepts a cfg map only), so the `file` row lock covers the re-check and the write. `get-file*` reads the row once (`sql:get-file` plus `resolve-applied-migrations`) and completes it with `attach-file-data`; only when a migration is pending it reads the file again under `FOR UPDATE`, re-checks `need-migration?` on that fresh map and persists it (`migrate-file`). The common case stays at one row read and one migrations resolution, the migration case keeps a single decoded copy in the heap, and a row that changed in between is not overwritten. A migration found on the unlocked read is applied in memory only, for the next read to persist. `migrate-file` returns the reattached map when it persists, with the `:storage-object-id` of the row the write left, and the realized map when it does not; `:realize?` is what returns the read fully plain. `fdata/upsert!` answers `{:written ... :storage-object-id ...}` instead of a boolean, `update-file!` propagates it and the migrating read puts it back in the map it returns, so a caller that persists again touches the current object instead of the replaced one. `insert-file!` keeps nil: an INSERT never replaces a persisted object. The write also compares the stored pointer with the caller's value (`IS NOT DISTINCT FROM`) and raises `:stale-file-data-pointer` when it affects no row. Every writer holds the `file` row lock, so a pointer that no longer matches is a protocol violation, not a race to retry. The expected pointer is a separate argument because the storage branch replaces `:storage-object-id` before the upsert, and the check lives in `upsert-in-database`: the `legacy-db` branches do not use that SQL and their count can legitimately be zero. The save, snapshot and restore-backup reads take `FOR UPDATE` instead of `FOR SHARE`: they inherit the lock, and the migration branch asks for the exclusive one in the same transaction, so a shared lock that upgrades deadlocks two concurrent readers. A data read rejects `:lock-for-share?` for the same reason; only row-only reads accept it. `sql:get-file-without-data` is a plain select now, not a CTE, which makes locking legal without loading the data. Realization has a single owner: `attach-file-data` only resolves and decodes, and `get-file*` realizes once per path. The applied migrations no longer travel in the `get-file-with-summary` response, cache hit and cache miss alike. `embed-assets` reads each library without `:realize?`, so its `:components` is an unloaded pointer map; `absorb-assets` walks it with `vals`, `PointerMap.seq` calls `load!` and that raised with no loader bound, so every export with `:merge-libraries` of a library with components failed. The absorb runs under the library's own loader now. The in-place binfile import (`import-file-and-overwrite*`, `::overwrite`, `::bfc/file-id`) had no product caller and cannot carry the previous `storage_object_id`, so overwriting a storage-backed row would leave its object unreclaimed. `import-files!` rejects a cfg with `::bfc/file-id` (`:not-supported`) and the helpers only that path used go with it (`invalidate-thumbnails`, `*reference-file*`, the `::overwrite` id branch of `index-object` and of `save-file!`, the `on-conflict-do-nothing?` flag of the media insert). Tests: a new `file-migration-read-test` pins the single read, the locked re-read, the concurrent save interleaving, the double migration and the read-only path; the upsert tests cover the return shape and the stale pointer; the SQL recorder they share lives in `backend-tests.helpers`, and the save, snapshot, restore and summary sequences follow the new reads. The backend memories document the read contract and the pointer. AI-assisted-by: deepseek-v4.1-flash * 🐛 Harden the get-file guard and skip its read-only transaction The cfg guard of `get-file` was an `assert`, which `app.config` elides when `:backend-asserts` is off, so a deployed binary accepted a bare pool or connection while the docstring promised it raises. It throws `IllegalArgumentException` now, the class the option guards of the same namespace already use, and the existing test expects it instead of `AssertionError`. A read with `:read-only?` applies the pending migrations in memory and writes nothing, so the transaction the migration write needs only costs two round trips there. `get-file` runs it with `db/run!` instead: with a cfg that carries a connection it still joins the caller's transaction, and only the reads that would have opened their own skip it. A lock option needs a read-write transaction from the caller, because PostgreSQL rejects row locks in a read-only one; the docstring states that. Tests: the guard test expects the new class; a cfg marked `::db/read-only` reads a pending migration without persisting it (the config flag that fills that key is legacy, but the key is set by hand); and a read-only read is pinned as not opening a transaction while the same call without the option does. The backend memory documents the split and the guard. AI-assisted-by: deepseek-v4.1-flash |
||
|
|
9c5260314a
|
🐛 Revoke websocket subscriptions when access is revoked (#12189)
* 🐛 Revoke websocket subscriptions when access is revoked GHSA-m53j-2766-6jqw. A websocket subscription was authorized once, when it opened, and that decision was trusted for the rest of the connection's life, so a member whose access was revoked kept receiving live file content as long as the tab stayed open. The revocation travels over the message bus instead of closing the socket from the mutation: the connection registry is local to one backend instance, while the RPC that revokes access runs on whichever instance received the request. Every instance re-verifies the subscriptions it owns with the same `has-read-permissions?` predicates that `:subscribe-file` and `:subscribe-team` authorize with, so a downgrade to viewer keeps its subscription and a non-member organization owner keeps read-only access. Two mechanisms cover each other: - Access-changing mutations announce after the write. delete-team- member, leave-team, update-team-member-role, delete-file, delete-project, move-files, move-project and delete-team announce. delete-share-link is deliberately excluded: a share-link visitor never passes the subscribe permission check, so there is no subscription to revoke. Team-wide changes publish one event per team; the watcher resolves the members itself, and both publishers defer through `db/after-commit` so the watcher never decides from pre-commit state. - Every relay re-checks access every 5 minutes by default (`subscription-revalidation-interval`), driven from the relay loop with `alts!` over the subscription channel and a one-shot tick channel, so a steady stream of edits cannot push the check away forever. Supporting changes: - The registry holds `:connections` and `:by-profile` in one atom, kept consistent by register/unregister in a single swap!; a profile lookup is one map access, and a profile whose last connection leaves is dropped from the index. - `close-file-subscription` and `close-team-subscription` announce the :leave-* message, close the relay channel and purge the bus subscription, and are idempotent. Closing the channel ends the relay loop and its re-check task. - The permission query the re-check uses filters soft-deleted files, projects and teams, matching its sibling queries. - A transient database error fails open: it logs and keeps the subscription; the next tick retries. - The revocation watcher's params schema declared the nitrate client as a bare keyword inside [:map ...], which malli resolved as a registry lookup and which took the whole system build down under enable-backend-asserts. It is now an explicit pair, optional because init-key yields nil with the admin-console flag off. The tests reproduce the leak against the real message bus, including the negative control for the interval re-check. The written records live in mem:backend/websocket-subscriptions. Closes #12155 AI-assisted-by: space-bunny-free AI-assisted-by: muse-spark-1.3-contributor * 🐛 Mirror the subscribe gates in the revocation re-checks Addresses the second review round on PR #12189 (pabloalba). The re-checks only asked whether the profile could read. The membership legs of sql:file-permissions filter soft-deleted rows since the F2 fix, but the organization-owner fallback grants read-only access over HTTP without looking at deleted_at, so a soft-deleted file or team kept an expelled org owner subscribed indefinitely. The re-checks now gate on existence as well: file-readable? and team-readable? require a live row (db/get*, non-throwing, soft-deleted rows excluded) plus the read permission, mirroring the :subscribe-file gates and extending them to teams. Both relays and revalidate-profile-subscriptions use them; the subscribe handlers keep their exact behavior. Also from the same review: - watcher-consumes-revocation-events-from-the-bus no longer races the command's after-commit announcement: it takes access away with a direct row delete and drives the watcher with a synthetic bus event, instead of deleting through the command and publishing a second notify-* that would mask a broken command announcement. - moving-a-project-closes-source-team-subscriptions pins the move-project e2e: the file subscription drops, the source-team subscription survives (membership unchanged). - soft-deleted-file/team-closes-org-owner-subscription cover the new gate with red-before/green-after tests. mem:backend/websocket-subscriptions updated. Closes #12155 AI-assisted-by: inkling-2-small-free * 🐛 Refuse dead teams in resolve-team-id (PR #12189 review) Second review round on PR #12189 (pabloalba). The file re-check added in 87ca7aca53 stops at the file row, but the organization-owner fallback grants read-only access to non-member org owners via Nitrate over HTTP, which knows nothing about Penpot soft-deletion. resolve-team-id resolved the team from unfiltered SQL, so an expelled org owner kept the file subscription open for an indefinitely soft-deleted file, project or team. Now both SQL lookups filter deleted_at, and resolve-team-id checks the team row itself (db/get*, which filters soft-deleted rows) for every caller — teams, projects and files alike. Also from the same review: - file-readable? now uses the same sql:get-minimal-file as the subscribe gate (non-throwing via throw-if-not-exists false), and team-readable? projects id and deleted-at, so db/get* can filter. - The test suite extracts the thrice-duplicated raw delete into delete-team-membership!; a sibling of the fail-open pin now redefs the existence probe to throw; memory and docstring aligned with the code. - docs/technical-guide/developer/architecture/backend.md attributed the websocket to notifications.clj; it now names app.http.websocket as the owner and notifications.clj as the revocation publisher. - mem:backend/auth-permissions-product-domains documents the fallback mechanism and the resolve-team-id contract. Closes #12155 AI-assisted-by: inkling-2-small-free * 🐛 Test resolve-team-id refusal and tidy review findings Third pass over PR #12189 after review-code on 225eb98de2. resolve-team-id now has a direct test: all three branches resolve while rows are live, and file, project and team soft-deletes each refuse their branch — the two new SQL predicates and the final team-row check are all pinned by it. From the same review: - failing-existence-check-keeps-the-subscription and the sibling a-failing-check-keeps-the-subscription are deleted: they could not distinguish a live relay from a dead one. The fail-open policy remains documented in the docstring and the memory; the memory no longer claims the pins exist. - team-readable? projects ::sql/columns like the rest of the backend; the fully-qualified keyword went away with the new alias. - The dead when-not guard in get-organization-owner-viewer-team is removed (resolve-team-id guarantees the row live); the let binding in resolve-team-id no longer shadows the destructured team-id. - docs/technical-guide names rpc/notifications.clj as the publisher and the tree lists the real top-level modules. Closes #12155 AI-assisted-by: inkling-2-small-free * 🐛 Make the revocation tests deterministic Third review round on PR #12189 (pabloalba). Both tests took access away through a command that announces, so the live watcher could close the subscriptions before the test looked: - revocation-closes-subscriptions-that-are-no-longer-authorized asserted an exact count of 2 closes after delete-team-member, and that count reads 0 whenever the watcher got there first. It now removes the member with delete-team-membership!, the raw row delete that publishes nothing, and asserts both subscriptions are still open before calling revalidate-profile-subscriptions. - team-event-closes-every-member-subscription ran delete-team and then published a second notify-team-permissions-changed, so a missing announcement in delete-team still passed. It now soft-deletes the team row (sql:team-members joins team_profile_rel and profile only, so the watcher still resolves every member), asserts nothing closed, and drives one synthetic team event. The command paths stay covered by the announce and end-to-end tests for delete-team-member and delete-team. The file split the reviewer suggested stays the optional follow-up, tracked outside this PR and waiting for the merge. Closes #12155 AI-assisted-by: deepseek-v4.1-flash |
||
|
|
bcac14a4a8
|
🐛 Skip circular library links on binfile import (#12062)
* 🐛 Skip circular library links on binfile import File import recreated circular library references that the link-file-to-library command already rejects. The v3 importer now skips manifest relations that would close a cycle, or link a file to itself, and logs each skipped link. Closes #12052. AI-assisted-by: muse-spark-1.3-contributor-free 🐛 Address review findings on circular import guard F1 extracts the cycle check into bfc/library-link-status and reuses it in link-file-to-library and v3 import, keeping UI errors intact. F3 orders get-files-rels and manifest relations so the first edge wins deterministically, and documents it. F2 adds a three-node transitive cycle test; F4 asserts file_library_sync has no orphan rows for skipped links and matches rels otherwise. AI-assisted-by: muse-spark-1.3-contributor 🐛 Pass team-id in binfile v3 circular library tests The four circular library relation tests (import-skips-circular, import-skips-transitive-cycle, import-skips-self-referencing, import-keeps-acyclic) called v3/import-files! without ::bfc/team-id. resolve-and-link-libraries asserts (uuid? team-id), so with enable-backend-asserts the tests errored instead of running. Add (assoc ::bfc/team-id (:default-team-id profile)) to each test, mirroring the production RPC import path and the earlier fix in a36193e54b for the other binfile v3 import tests. Verified: 1338 tests, 6910 assertions, 0 failures. AI-assisted-by: mistral-large-4 * 🐛 Guard the v1 import and restore the RPC check order The version 1 importer recreated the circular library references that the version 3 importer already skips, and a user reaches it through `import-binfile` and the import job. It now asks `bfc/library-link-status` before inserting each relation, so a self link or an edge that would close a cycle is skipped and logged, and it processes the relations in sorted order like v3 does. Closes #12052. The version 2 importer stays out on purpose: only the developer REPL reaches it, and a note in the code records the gap. `link-file-to-library` keeps the shared helper but restores the order of the base commit: the self link guard, then the edition permission and team ownership checks, and only then the cycle check. A caller without permission gets `not-found` again instead of the circular error, and a rejected request no longer runs the recursive library query first. Tests: a v1 import that recreates a cycle, the self link error of the command, and the permission check running before the cycle check. AI-assisted-by: deepseek-v4.1-flash * 🐛 Tolerate repeated file entries in the v1 binfile import The v1 exporter unioned the requested files with their libraries without deduplicating, so a package exported from a file inside a library cycle (see #12052) lists that file twice in the metadata. The v1 import created one file per entry, so the second insert hit the primary key of `file` and aborted the whole import with "file already exists". The importer now reads the pair of objects of every entry to keep the stream in sync but imports the first occurrence only, warns about the repeats, and answers with one id per imported file. The v1 export keeps its behavior: it has no caller in the product, and only the tests and the developer REPL use it. Tests: a package with a repeated entry, built with the v1 export from a cyclic library pair, imports the file once and leaves no duplicated rows. AI-assisted-by: deepseek-v4.1-flash |
||
|
|
a77cb2608e
|
🐛 Normalize stored SVG attribute keys to kebab-case (#12006)
* 🐛 Normalize stored SVG attribute keys to kebab-case The v3 reader already emits kebab-case keys, but birth stored the camelCase React prop names, so import and export never converged: fill rules painted wrongly and filter defs lost their blur. Birth now normalizes every stored attribute key to kebab-case through the `svg-props` whitelist (`attrs->kebab-props`), applied on all element paths of the svg importer, on the file builder and on defs. svg-raw `:content` is kebab-ized in spelling only and keeps unknown keys, because the sidebar reads them. Non-spec attrs Penpot must keep live in `penpot-extra-attrs`, seeded with `:stroke-style`. Render boundaries translate back to camelCase for React and for the wasm props, and the shape schema declares the closed kebab contract derived from that same whitelist, so birth and validation cannot drift. Migration `0035-fix-shape-svg-attrs-and-defs` converts old files once with the birth transforms, and carries back the svg-viewbox rect normalization the withdrawn `0021` had. That old id leaves `available-migrations`, so files recording it migrate again through this one, and the binfile cleaner drops its own svg repair. Review fixes included here: `setup-opacity` and the blend mode read the stored keys, gradient stops accept `:style` as a CSS string or a map, `inherit-attributes` reads `:style` so a child's inline style beats the inherited group attr per CSS, and the svg-attrs menu shows spec spelling (`viewBox`, `class`) through `stored-attr-display-name`. Tests cover the migration, the closed svg-attr schemas and the import/export round trip; the svg-content schema test pins its sample seed, which dealt no svg-raw shape on unlucky runs. Closes #11947 AI-assisted-by: muse-spark-1.3-contributor * 🐛 Keep legacy svg node tags and transforms valid in migration 0035 The closed svg node schemas require keyword tags and map attrs, but stored files hold def and svg-raw content tags as strings, and some node `:attrs` as nil. Migration 0035 left them as they were, so the file data check that runs after migrating rejected those files and they could not be opened. `normalize-node-tree` now turns string tags into keywords and nil attrs into an empty map, at birth and in 0035. With keyword tags, the svg defs renderer applies the shape transform to user-space defs. Legacy `:svg-transform` values are plain maps, which multiply into NaN matrices. `gmt/coerce` turns them into matrices, in 0035 and in the binfile cleaner, because every v3 import decodes the transform as a plain map again. Legacy masks and gradients now render as designed in the svg renderer. The binfile test helper calls `export-files!` with the current config keys. AI-assisted-by: claude-opus-5-5 --------- |
||
|
|
6a1b58c310 | Merge remote-tracking branch 'origin/staging' into develop | ||
|
|
0b8a428d6b
|
🐛 Validate binfile library relations against the package (#12862)
Ignore imported library relations that point outside the package or self-link a file on the v3 import path. Delete leftover cross-team and self relations from file_library_rel and their matching file_library_sync rows with migration 0158, sync rows first. Move the get-files-rels SQL to a var, without behaviour change. Cover the write-side invariants with state-based tests: tampered manifests, package containment, self links and the migration cleanup. AI-assisted-by: deepseek-v4.1-flash AI-assisted-by: muse-spark-1.3-contributor |
||
|
|
93d36d04d8
|
✨ Stabilize fragmented file data across snapshots and GC (#12420)
* ✨ Stabilize fragmented file data across snapshots and GC Snapshots, automatic and user, now store pointer-based rows that share fragments with the live file instead of duplicating page content. `get-file-snapshot` resolves those pointers on the server so version previews keep rendering plain data, and `restore!` reuses the shared rows. Export still materializes (the package carries no fragments) and import regenerates them. `file-gc` treats the main file and every unpurged snapshot as liveness roots, reads the file unrealized, and reattaches the original pointer wherever the cleaned value is unchanged, so a pass over an untouched file writes no fragments and a pass over an edited file keeps the ids of the pages it did not touch. Snapshot creation and the restore backup take the file row `FOR SHARE`, which serializes them with the pass and closes the race that let a new snapshot reference fragments the pass was about to mark. All backend pointer bindings go through the `fdata/with-pointers`, `with-tracked`, `with-tracked-pointers` and `with-untracked` macros. The loader reads fragments a previous pass marked until `objects-gc` purges them, and the frontend fails loudly with retry instead of dropping an unresolved fragment. Review fixes in this branch: the automatic snapshot attaches before pending migrations, `realize-pointers` is the single verb, `reattach-unchanged` takes the identity fast path, the restore guard takes its revn from the backup read, and `PointerMap.withMeta` generates a fresh id when metadata changes. Tests cover both snapshot writers (red without the lock), snapshot sharing and pin release through the real delete and purge paths, reattach against orphan marking, preview after offload, the pointer binding contracts and the with-meta id change. Closes #12410 AI-assisted-by: muse-spark-1.3-contributor AI-assisted-by: deepseek-v4.1-flash ♻️ Pin no-churn behavior for untouched pages Pin the behavior the churn verification measured: a save keeps the fragment id of every page it does not touch, with a page of eleven keys (PersistentHashMap), a 46-shape objects map and a nine-page index (hash map). Also pin the mixed case: a data-level objects-map flip rewrites the old pointer pages once and the next save keeps the new ids. The memory records the identity rule behind it: re-assoc with the same value returns the same map in array and hash maps alike. Closes #12410 AI-assisted-by: deepseek-v4.1-flash ⚡ Stop loading untouched pages when enabling objects-map `enable-objects-map` wrapped every page on every save, and reading a page's objects forced its PointerMap to load the fragment. A save of one page in a seven-page file therefore made seven fragment reads. Skip unloaded pointers: their fragment was already wrapped by the encode-file that wrote it, so loading it only to wrap it again is a no-op. The save now loads only the pages it edits. A page with a plain fragment (a data-level feature flip) keeps its plain fragment until it is first loaded or edited; a migration that enables the feature must rewrite the data itself. Cover the change with load-count tests for save and restore, update the flip test to the new contract, and drop the stale restore comment. Closes #12410 AI-assisted-by: deepseek-v4.1-flash ⚡ Keep page fragment ids through pending migrations Applying a pending migration on read realized the file, ran the migration and persisted it again; the persist wrapped every plain page in a new PointerMap, so every page got a new fragment id even when the migration did not change it. That leaves the fragment cache cold and rewrites every fragment on the first read after a deploy. Capture the original pointers before realizing, reattach the unchanged positions after migrating (same value comparison file-gc already uses), persist the reattached file and return the realized one: the read contract is plain data. Read-only connections skip the persist and the reattach. A migration that adds objects-map skips the reattach so the pages get wrapped. Move the three helpers to fdata so file-gc and the migration path share them, and cover the path with a real 0031 migration test: the unchanged page keeps its id, the changed page gets a new one, the caller gets realized data and a second read does not migrate again. Closes #12410 AI-assisted-by: deepseek-v4.1-flash * 🐛 Realize library pointers with each library's own loader Fragment rows are keyed by file id, and every caller binds the fragment loader of the file it works on. get-resolved-file-libraries returned libraries with their pages still as pointers, so the first read of a library page under fdata/pointer-map asked the consuming file for the fragment and failed with :fragment-not-found. That broke migrations and file validation for every file using a pointer-mapped library. The weak-map loader now realizes each library's pointers with the library's own id. The test fails without the change. AI-assisted-by: claude-opus-5-5 Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 🐛 Restore the pointer-map require the repair path still needs The branch dropped `app.util.pointer-map` from the namespace while the auto-repair path (added to develop before this branch) still rebinds `pmap/*tracked*` around validation and repair in `repair-preexisting-damage` and `process-changes-and-validate`. The namespace did not compile, so the whole backend failed to load. Those toggles do not fit the fdata macros: `with-untracked` binds nil and `with-tracked` creates a fresh atom, while the repair path has to turn tracking off and restore the existing atom (and read it before the repair starts). Restore the require and keep the bindings explicit. Also pin the feature-off path with a test: with `fdata/pointer-map` off (the default), a save creates no fragments, file-gc writes nothing and the snapshot preview still returns plain data. Drop a stray `\n` char literal left at the end of one deftest. Update the memories with the exception to the macro rule, the snapshot FOR SHARE locking and the pre-migration automatic snapshot. Closes #12410 AI-assisted-by: deepseek-v4.1-flash * 🐛 Validate debug files with their own pointer loader The `/dbg/actions` file validation passed the file id where `get-resolved-file-libraries` expects the file map, so the library map was seeded with a nil key. It also validated a file read without realizing it, so the first page lookup of a pointer-mapped file raised "load is not supported when *load-fn* is not bind" and the endpoint could not validate exactly the files the feature touches. Read the file realized (validation walks every page and component anyway) and build the library map from the file. The library loader already resolves each library's pointers with its own id, so the realized file and its libraries validate as plain data. Covered by a handler-level test that fails before the change. AI-assisted-by: deepseek-v4.1-flash * ♻️ Bind the snapshot loader through the fdata helper `reduce-snapshots` hand-rolled the same `binding` that `fdata/with-pointers` builds (it also loaded the function's own pointer loader before entering the reduction). Use the macro so the rule holds there too and the namespace no longer needs `app.util.pointer-map`; the loader is identical (same cfg and file-id) and snapshot walking behaves the same. AI-assisted-by: deepseek-v4.1-flash * 📚 Note the pointer-binding exceptions and snapshot ordering Record what the feature review settled: - Two direct `binding`s stay on purpose: the fine-grained `*tracked*` toggles of `app.rpc.commands.files-update` (validation/repair turn tracking off and restore the existing atom) and `binfile.common/encode-file` (fresh tracked set inheriting the current one). Atom-scoped macros would only be worth it if the set grows. - The automatic snapshot is taken before the pending migration, and the pre-migration window is narrow: the `get-file` RPC migrates on read and its ETag hashes the available migrations, so a post-deploy read cannot be a 304 from the pre-deploy state. The old-format snapshot only appears when the save is the first request to touch the file after the deploy, and `get-file-snapshot` previews it as stored. AI-assisted-by: deepseek-v4.1-flash * 🐛 Apply pending migrations when previewing a version Automatic snapshots are attached before the save applies pending file migrations, so a version can store the old format. The preview RPC handed that data to the UI as stored, and the viewport renders it with the current code. Realize the snapshot with the live file id (fragments are keyed by file, not by snapshot), normalize its migrations to a set (the decoder keeps the pgarray as a vector, and `set/difference` would compare indexes) and apply the pending migrations with the pure `fmg/migrate-file` and the live libraries. Nothing is persisted: the stored version keeps its data and restoring it re-migrates on the next read. The response now carries the snapshot's migrations. Use `:features` the way `get-file` does for `fdata/objects-map`: a client that does not declare it gets plain objects. The historical feature set is not checked against the enabled one on purpose: a version may predate a feature change. Cover the preview of a snapshot with a pending migration and the objects realization; both pin that the stored row is untouched. Update the file-data memory with the preview contract and the vector/set trap. AI-assisted-by: deepseek-v4.1-flash * 🐛 Decode snapshot migrations as an ordered set `file-snapshots/decode-snapshot` handed the `file_change.migrations` pgarray as a vector. `set/difference` falls back to index lookups when the first collection is shorter than the second, so the pending set was right only by accident: a stored subset happens to work, while a stored set larger than `available-migrations` (a snapshot taken by a newer build) marks every available migration as pending and re-applies it to already migrated data. Decode it as a `d/ordered-set`, the type the rest of the code and `schema:file` expect. This also covers `restore!` and the snapshot list, and it lets `get-file-snapshot` drop the `(into #{} ...)` normalization it was carrying. Re-wrap the union in `fmg/migrate` so the invariant holds even when a caller hands a plain set. A snapshot with NULL `migrations` predates SQL migration 0140 and keeps the format version of its time. Pin that legacy path: the preview regenerates the applied set from that version, migrates in memory and leaves the stored row untouched. Update the file-data memory with the invariant and the legacy path. AI-assisted-by: deepseek-v4.1-flash * 🐛 Fix flaky waiting-gauge assertion in climit test The throwable-while-waiting test asserted the exact list of waiting-gauge publications, but the component sampler republishes the same count when its first tick lands inside the recorded window under load. Assert only the last value, which is the contract the helper documents. AI-assisted-by: muse-spark-1.3 --------- Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> Co-authored-by: Álvaro Tejero Cantero <alvorithm@teje.ro> |
||
|
|
29373fff9f
|
✨ Rework the progress of the import and export dialogs (#12192)
* ✨ Rework the import dialog The import wizard paints a Check/Upload/Import stepper derived from the dialog status, plus a phase title, a description and a progress bar per phase, and its rows follow the new design: folder icon, file name and a size subtitle. While checking, the dialog shows no rows and no spinner, because the phase header already carries the message. The wizard writes the job milestone once, in the header (`jp/current-milestone-text`), since every file of a container reports the same one. A row keeps its name and its size, says "Ready" once its container finished uploading, and shows the error text when it fails; the button that drops a file only appears while the wizard still offers its import step. The `jobs.queued` label now reads "Waiting job slot". The bar measures the whole import: determinate while the containers are uploaded, weighted by their size (`imp/upload-pct`), and animated otherwise, because the stages of a job are not comparable. The phase title counts the files, and `imp/imported-count` counts them done with the job's own counter, so the number moves while a container rebuilds its files one by one. The analysis reads the entries of a version 3 package in one pass: a file shows what it occupies inside its container, its own entries plus the storage objects its media and thumbnails reference, while what the container weighs stays in `:container-size`, the size the upload bar is weighed on. The ready step waits for the end of the analysis (`imp/ready-to-import?`), not for every known file, so Continue never leaves out the files that arrive later. Tests cover the file sizes, the upload percentage, the done counter, the wizard step and the milestone in course. The change also adds `-XX:NativeMemoryTracking=summary` to the backend devenv so `jcmd` shows native memory, and documents the wizard, the promise call styles (`mem:clojure/promesa`) and two CSS pitfalls: a logical `inline-size` wins over an inline physical `width`, and Rumext leaves a plain `:style` map raw, so inline styles leave CLJS as `#js` objects. AI-assisted-by: glm-5.3-flash AI-assisted-by: deepseek-v4.1-flash * ✨ Rework the export dialog Every exported file is its own job, so each row reports its own state: folder icon, name and a size-plus-`.penpot` subtitle; waiting for a worker; running with the milestone of the job and an animated bar; or exported with a tick. A failed row paints red, takes the close icon and shows the hint of the job, or a generic label, through `files-export-modal.export-error`. Success, failure and cancellation replace each other, so a row never reports two outcomes at once. The ids of the messages are canonicalized to string before they match a row, because they cross serialization boundaries. The export result now carries `:size`, so the subtitle says the format until the job produces the artifact and the size once it exists; a transport failure builds a synthetic error, so a nil `ex-data` no longer leaves the row hanging. The header counts the files and says whether the export runs or ended; since every file has its own job, it paints no global bar. Closing changes too: the Close button is gone. While files are in flight the footer offers Cancel export and the X cancels as well, which marks the loading rows cancelled and leaves them in place, so their outcome can be read; once every row reached a terminal state the footer offers Done, which dismisses the dialog. The retry button of the finished design is left out, because the client has no retry mechanism yet. Styles come along: the rows scroll while the header and the footer stay put, and the new phase header, row and bar follow the tokens of the import wizard. The title key becomes `files-export-modal.title`, and the new keys cover the header, the row states and the buttons. AI-assisted-by: muse-spark-1.3 * ♻️ Inline the binfile job adapters into their task handlers The wrappers in app.binfile.jobs only forwarded to the core, so each task handler now composes its own run: frozen cfg, active check, progress bridge and format dispatch. export-cfg freezes the whole export (file-ids, export-type, output) and v3/export-files! takes it as its single argument; ::bfc/ids becomes ::bfc/file-ids. The bridge keeps one job: bfj/run forwards the cfg to its callback, and check-active answers with it to thread the run. AI-assisted-by: muse-spark-1.3 * 🐛 Migrate binfile tests to the single-arg export-files! The refactor made v3/export-files! take a single cfg carrying ::bfc/file-ids and ::bfc/output, but six calls in binfile_test (added by the tokens handling change) still used the old two-arg form with ::bfc/ids. Move them to the new form, as the rest of the file already does. AI-assisted-by: muse-spark-1.3 * 🐛 Fix export artifact expectations to include file size The export flow now reports the artifact size, but the stale expectations dropped it and failed the frontend suite. Include the size in both assertions so they match the emitted message. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Migrate remaining export-files! test callers to the single-arg form rpc_binfile_test and offload_file_data_test still passed the output apart and used ::bfc/ids; move both into the cfg as ::bfc/file-ids/::bfc/output. * 🐛 Hide the downloads hint when no file was exported After Cancel or when every job failed, the export header said "0 files exported · Check your downloads folder."; drop the hint then. * ♻️ Read the object of a container entry once The index of a container asked twice for the object of the same zip entry: once to test it and once as the key of the update. Resolve it once, in the branch that owns it, so the entries of a file never pay for a question that does not apply to them. Addresses a review note on #12192. AI-assisted-by: deepseek-v4.1-flash * ♻️ Never size an import row with its container's weight A row of a version 3 package took the size of the container whenever the container held no entry for its file, while the comment beside it promised the opposite. An unknown size now stays nil, which the row already understands: it paints no subtitle instead of a number that belongs to the whole package. The test carries a file the manifest lists and the container does not hold, and the memory records what the row does then. Addresses a review note on #12192. AI-assisted-by: deepseek-v4.1-flash * 🐛 Retry pool stats read until snapshot is consistent Hikari exposes each pool counter through a separate MXBean call, so a borrow or return between the reads can leave one snapshot off by one. The test read once and failed in CI with total 2 vs active + idle 3. Read up to 20 times with a 10ms pause until total equals active + idle, then assert on the stable snapshot. The invariant stays strict; only the transient race is tolerated. AI-assisted-by: muse-spark-1.3-contributor --------- Co-authored-by: Elena Torro <elenatorro@gmail.com> |
||
|
|
a8e8a54a2b | Merge remote-tracking branch 'origin/staging' into develop | ||
|
|
86e01d56a2
|
🐛 Update canvas and keep local sets when switching tokens source (#12442)
Setting a library as tokens source did not update shapes that use tokens, because set-tokens-source never ran the token propagation that every other token operation runs. Run it after the commit. Going back to the local tokens, or unlinking the library that was the source, left no local set active: the file has a single tokens status, and switching to a library replaced it with the library's one. Keep the status of the file's own tokens lib aside in a new optional :own-tokens-status attribute while an external library is the source, and restore it when the file is its own source again. Exporting without libraries and absorbing a deleted source library restore it as well. Closes #12430 AI-assisted-by: claude-opus-5-5 |
||
|
|
fffc3ee222 | Merge remote-tracking branch 'origin/staging' into develop | ||
|
|
fdd66978fd | Merge branch 'main' into staging | ||
|
|
5abf66dbf0 |
🐛 Set a finite idle timeout on binfile transactions
Binfile v1/v3 import and the v3 export get-file step hold a single transaction while the ZIP is decoded and the file data is realized. Disabling idle_in_transaction_session_timeout left a stuck connection holding locks (and a pool slot) forever. Raise it to 30 minutes instead. The longest import/export observed in production is under 8 minutes, so the limit still bounds a stuck session. statement_timeout stays at 300 s. Add set-transaction-idle-timeout and configure-import-transaction in app.binfile.common, use them from both paths, and cover the helper with a test. AI-assisted-by: deepseek-v4.1-flash |
||
|
|
dfc55a7192 | Merge remote-tracking branch 'origin/staging' into develop | ||
|
|
feb73f0fd8
|
✨ Handle tokens libraries when exporting .penpot files (#12077)
* ✨ Handle tokens libraries when exporting .penpot files Exporting a file that takes its tokens from an external library left the :tokens-source of the file pointing at a library that is not in the export, and the tokens lib of that library was not exported. With :detach-libraries, unset the :tokens-source and the tokens status in detach-external-references. The file is exported without any link to the library, and the shapes keep their applied tokens. With :merge-libraries, add cfo/absorb-tokens-lib, which copies the tokens lib of a library into a file and unsets its :tokens-source, so the file becomes its own tokens source. The tokens status of the file is kept; if it has none, the one of the library is copied. If the library has no tokens lib, only the source is unset. embed-assets calls it for the library that is the tokens source of the file, and ignores the other linked libraries. Add unit tests for absorb-tokens-lib, and binfile tests that export a file with tokens from an external library with both modes, checking the exported files, the tokens lib, the tokens source and the applied tokens. Also reformat the export-type docstring in binfile v3 and remove a leftover debug prn in the export dialog. AI-assisted-by: claude-sonnet-5-5 * 🐛 Preserve active token sets when importing linked libraries Cache each decoded tokens library for the duration of an import so files and their tokens source resolve active sets against the same generated IDs. Add a round-trip regression test for exports that include linked libraries. * 🐛 Preserve tokens when deleting or unpublishing libraries Absorb the tokens library into files that use it as their tokens source. Keep referencing file IDs in the deletion task before removing relations. Embed tokens from deleted, unlinked sources during export, and clear stale source and status data when the library no longer exists. Add regression tests for deletion, unpublishing, and exports with unavailable libraries. --------- Co-authored-by: alonso.torres <alonso.torres@kaleidos.net> |
||
|
|
a53481c3b7
|
✨ Enrich backend audit events with team and entity ids (#12233)
* ✨ Enrich backend audit events with team and entity ids Backend audit events now carry the data a consumer needs to rebuild state without joining production tables: - prepare-rpc-event adds :team-id when the event carries a project or file id; commands that already loaded the team set it themselves. - add-profile-to-team! emits add-team-member on every real insert. - create-project and create-file emit the new row id. - clone-template, import-binfile and duplicate-project emit one event per created entity. - both binfile versions return {:file-ids [...]} from import-files!. Closes #12219 AI-assisted-by: deepseek-v4.1-flash * ✨ Emit delete and hard-delete audit events with entity data The event stream must let Nexus rebuild local state without joining production tables. This completes the enrichment with the delete paths: - delete-team keeps returning nil but now carries the deleted row (id, name, timestamps) in the audit props, like delete-project. - create-team adds created-at to its audit props. - delete-profile carries id, email, fullname and deleted-at; its props used to be empty because every param is namespaced. - objects-gc emits hard-delete-team/project/file per purged batch, with the deleted ids, only when the audit log is on. - delete-team-member needs no change: the delete is physical and the params already carry team-id and member-id. The nitrate push-audit-events tests now drop the membership events emitted by their own profile setup before counting. Closes #12219 AI-assisted-by: muse-spark-1.3-free * ✨ Announce cascade deletes and missing ids in audit events The :delete-object worker task marked teams, projects and files without leaving any event, so an event-sourced projection kept them alive for weeks. It now emits one cascade-deleted event per execution with every affected id, in the same transaction, only with :audit-log on. Rows purged by objects-gc and snapshot cascades stay silent on purpose. Also: create-team-with-invitations carries the new team id, and update-team-member-role names the demoted member on owner transfers. Closes #12219 AI-assisted-by: muse-spark-1.3-free * 🐛 Identify the SREPL profile delete in audit events The SREPL delete-profile! emitted no profile-id and empty props, so the row could not be attributed by any consumer. It now carries the profile id in both fields. delete-team! named the wrong cause in its context. Closes #12219 AI-assisted-by: muse-spark-1.3-free * 🐛 Identify the SREPL profile restore in audit events The SREPL restore-profile! emitted no profile-id, so the consumer keyed on the zero profile instead of the restored one. It now carries the profile id like a normal profile event, and no consumer special case is needed. Closes #12219 AI-assisted-by: muse-spark-1.3-free * 🐛 Stop attributing SREPL events to the subject profile profile-id names the actor, and SREPL calls have none: setting the subject id blamed the user for an admin action and would have leaked those rows into PostHog as user activity. The subject id stays in props, where the consumer resolves it, and the event profile stays zero. Closes #12219 AI-assisted-by: muse-spark-1.3-free * ✨ Expose file analysis data for the nexus pipeline The nexus file-security pipeline needs file content and scheduling state from the backend. This adds a read-only management method, `get-file-for-analysis`, returning the semantic projection of one file (pages, frames, texts, links, interactions; ids as strings; capped with a truncated flag), plus identity, revn, timestamps and share state. No prior projection code existed, so the method ships its own pure `file->projection`. Audit events now carry what the `:local` consumer schedules from: revn and modified-at on update-file, name and timestamps on every create-file emitter, deleted-at on delete-file, and new create/delete-share-link events with file and link ids. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Rename deletion audit events to action names Event names are actions, not past tense: `cascade-deleted` becomes `delete-object-with-cascade` with the same shape and transaction. The three `hard-delete-*` events merge into `hard-delete-objects`: one event per purged chunk carrying only that chunk's entity key (`:team-ids`, `:project-ids` or `:file-ids`), because each GC section purges in its own chunk transactions and a single event could never share them all. Every submit stays inside its delete transaction, `triggered-by` stays annotated, and silent paths stay silent. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Read worker parallelism knobs with plain config keys The runner wiring looked up `:app.main/worker-*-parallelism` (namespaced) while the schema and the environment provide plain keys, so `PENPOT_WORKER_DEFAULT_PARALLELISM` and `PENPOT_WORKER_WEBHOOK_PARALLELISM` never applied and workers always ran with the fallback. Read the keys the config holds. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Fix binfile audit test with a real file row `submit-file-created-event` reads the light columns back from the row, so the mocked importer must return an existing file id: a phantom id emits no event and the test saw zero `create-file` events. Create the file first and use its id. The audit-log memory now tells both bulk-event helpers apart. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Fix audit event fidelity from review findings The review of the audit enrichment found data-fidelity gaps; fix them while the branch is fresh. Membership events now belong to the actor: the flows where the owner adds somebody else pass an explicit actor id, the role is derived from the flags (the table has no role column), and the event carries the request context like its siblings. Also: team-id resolution skips read events, duplicate-project emits the project name and timestamps, delete-profile is idempotent again, the GC tests assert physical purge, and get-file-for-analysis leaves an audit trace. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Nest cascade ids under :cascade in delete event The cascade event carried every affected id flat next to the object identity; nest them under `:cascade` so the payload reads as header plus affected set. The event is an action like its siblings, not a command. Drop the `:audit-log` flag re-check: `submit` already fans out per flag. AI-assisted-by: muse-spark-1.3-contributor * 📎 Update copyright name on file header The company name changed; a single header in this branch still carried the old one. Align it with the current form. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Fix nil event-id on webhook trigger audit rows process-event generated the row id inside append-audit-entry, so the :process-webhook-event task never received it and the trigger row stored a nil :event-id. Assign the id up-front and share it between the stored row and the worker params; the telemetry shadow row keeps its own id to avoid a PK collision. Add regression tests covering the handoff and the end-to-end trigger row. AI-assisted-by: muse-spark-1.3-contributor |
||
|
|
9abcfebf60
|
✨ Run binfile import and export as durable jobs (#12056)
* ✨ Run binfile import and export as durable jobs Import and export of `.penpot` files become durable jobs: two commands create them and answer at once with a job id the caller follows, while a worker of the `binfile` queue does the work. Each job keeps its own ledger (owner, params frozen at creation, result, progress history and an expiry), and the legacy commands and their SSE keep working. Why: both operations ran inside the request. A large import held it open for minutes, progress only existed while the caller kept the stream open, and a client that went away left work nobody recorded, retried or could cancel. The unified `job` substrate already dispatched, leased, retried and logged events, but it had no user-facing ledger. How: - The substrate gains the pieces a user-facing job needs: optional job-def metadata (family, resource-role), an expiry on submit, the owner in the runner context, and job resources in their own bucket, written with the profile of the job and never deduplicated. - One generic command per family: the caller names the job and sends its business params, the registry checks the family, the job-def decodes and validates, and permission and quota are checked first. - Progress taps of the core become milestones of the job: a shared stage vocabulary with self-contained counters, throttled heartbeats, and an interrupt that stops the run inside its own transaction on cancel. - Cancel reaches pending and running jobs from both modals, and closing a modal mid-run cancels the job it was following. - The export re-checks the read permission at run time and answers with the package descriptor; the import re-checks the edition permission, runs the core in a transaction so a cancellation rolls it back, and releases the consumed package and its temporary copy. AI-assisted-by: deepseek-v4.1-flash * ♻️ Handle externally-cancelled binfile jobs end to end The `cancelled` outcome of a job nobody followed no longer hangs its screens: the data layers emit an explicit terminal message for it. - The export emits `{:cancelled true}` per file and the modal paints a neutral cancelled row instead of success; a message without an artifact never reaches the download branch. - The import closes every entry with a dedicated cancelled message through the existing error rendering, so the wizard completes. - Both flows are pinned by data-layer tests, including the red state they replace. Also in this change: the progress-contract comment of the management API, early business-params validation in create-import-job (before assembling the upload), a `release-resource` helper without fabricated job ids, a note on the heartbeat write cost, and a memory cleanup. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Pre-translate the cancelled import message The cancelled entry of the import wizard carried its message as a key for the UI to translate, which needed a rehash marker comment. Like its neighboring errors, it now translates eagerly in the data layer: the literal call keeps rehash clean with no marker, and the tests assert through the same call instead of the raw key. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Move cancelled strings under generic labels keys The dashboard-specific cancelled key never shipped: before use it moves under `labels.*` next to the generic strings both flows already use, with Spanish translations following the established wording. AI-assisted-by: muse-spark-1.3-contributor * ✨ Add minor improvements * ♻️ Rename export job command to create-export-binfile-job Export now has one command per job type, the shape the :export-assets command needs: create-export-binfile-job freezes exactly the :export-binfile job, so :name no longer travels in the body and only :params does. Import keeps the generic envelope. submit-job reads the queue from the job-def metadata ::jobs/queue-name (falling back to :binfile), read by the command and never by the substrate, so a job-def can route its work to another queue without a change in the command. The rename reaches the frontend and the tests: the binfile export is asked for under its own name, and the exporter service keeps create-export-job. AI-assisted-by: deepseek-v4.1-flash * ✨ Polish the import and export status rows The status row of a file already carries what the file is doing, so the generic "Uploading file…"/"Downloading file…" lines are gone: the import wizard drops its own (the export one went with the jobs refactor), along with the now-unused style. In the export, every file is queued up front, even the ones whose job is created only after the previous one is over, and a file stays queued until its first milestone: a worker picking the job up no longer ends the queue. The queued text is a step smaller than the milestone. AI-assisted-by: deepseek-v4.1-flash * 🐛 Show the cancelled export and fix the import error The cancelled export was unreachable: mark-file-cancelled sets :loading false and the label lived inside the loading branch, so a cancelled file only painted a neutral notification. The terminal state now carries the label, and the export flags lose the `?` suffix (predicates keep it; a data prop is not a question). The import wizard rendered (:error entry) through tr, but every producer writes user-facing text (a hint or a translated label), never a key: it renders the text as-is now, and the dynamic-key ignores and the stale comment are gone. The total-text-limit comment no longer references the 200MiB it replaced; the value is 500MiB in develop already. AI-assisted-by: deepseek-v4.1-flash * ♻️ Reflect the real job params in the creation commands The creation commands declared their params as an open empty map, so the RPC contract and the generated docs said nothing about them. Each command now imports the schema from the job namespace (`export-binfile/schema:params`, `import-binfile/schema:create-params`), and the RPC validates the business params before the command runs. The job-def still owns the params and re-validates the assembled ones at run time. The commands no longer decode the params again: the RPC did it. Import joins export as one command per job type: `create-import-job` becomes `create-import-binfile-job`, its body no longer carries `:name` and the job-def is resolved by name like export's. Its creation schema keeps the version optional, because the command reads it from the package header when the caller does not say it, and fills the manifest metadata kept for audit. A version out of range is now refused by the schema (`:params-validation`) instead of the command's own `:unsupported-version` check, which is gone. AI-assisted-by: deepseek-v4.1-flash * 🐛 Stop the test system from running job runners The test system is main/system-config plus main/worker-config with a few components dissoc'ed. The runners of default, webhook and cron were listed by hand; the binfile runner added with the durable binfile jobs was not, so a real penpot/job-runner/binfile/0 thread ran during the whole suite. That runner races the tests that drive the loop by hand (run-batch plus run-worker-loop): it takes the job the test just dispatched and completes it, and the test then reads a state it did not produce. That is the reported flake of binfile-jobs-integration-test, whose middle assertion waits for a runner of another queue to not take the job. It only reproduced in CI: in the devenv PENPOT_TENANT makes the tenant baked into main/worker-config differ from the one the test builds, so the two used different queue keys locally. Drop every [<profile> :app.worker/runner] by key shape instead of listing the known queues, so a new queue cannot bring the race back. AI-assisted-by: deepseek-v4.1-flash * ✨ Upload the package one chunk at a time and as a percentage The upload milestone counts chunks, not units of work, so the row wrote "Uploading the package 3/9" over a file the user had uploaded once: it read as if nine packages were on their way. Write the stage as a percentage of the file instead ("Uploading file… (33%)"), and make the English text of the stage say file, not package. The chunks also go one at a time now (concat-all instead of merge-all with two in flight), so the percentage moves in a straight line instead of jumping. upload-blob-chunked is the generic uploader, so media and font uploads become sequential too. AI-assisted-by: deepseek-v4.1-flash * 🌐 Complete the Spanish catalogue es.po was 25 keys short of en.po: the 15 that the durable binfile jobs introduced (14 jobs.progress.stage.* and labels.queued) plus 10 gaps that predate them (errors.connection-error, errors.save-retrying, the find shortcuts, workspace.header.retrying, workspace.tokens.stroke-width and the mcp session labels). sync -l es then sorts the entries, syncs the #: comments from en and drops the 7 dashboard.import.progress.* keys en no longer has. The msgid sets of en and es are now identical (0 missing, 0 extra). AI-assisted-by: deepseek-v4.1-flash * ♻️ Move the job status keys into the jobs namespace The three status words the durable import and export rows render were filed under labels.*, next to the generic labels bucket, while their progress stages live under jobs.progress.stage.*: the strings of one feature sat in two places. Rename them so the whole job vocabulary is one namespace: labels.queued -> jobs.queued labels.export-cancelled -> jobs.export-cancelled labels.import-cancelled -> jobs.import-cancelled No other locale has these keys yet, so the rename costs no translation. The convention (a key names the feature that renders it; labels.* holds the generic reusable words) is written down in the frontend/translations memory. AI-assisted-by: deepseek-v4.1-flash * 🔥 Remove the translation keys nothing references Six keys of en.po (and of es.po, which now mirrors it) have no (tr ...) call site left in frontend/src or common/src, and no literal reference either: labels.uploading-file a duplicate of jobs.progress.stage.upload since the upload stage says "Uploading file…" labels.downloading-file dead since the durable jobs replaced the old import and export progress rows inspect.attributes.stroke.alignment.center inspect.attributes.stroke.alignment.inner inspect.attributes.stroke.alignment.outer workspace.toolbar.mcp-connect-here The first two are leftovers of the durable binfile jobs, the other four predate them. Both catalogues keep an identical msgid set afterwards. AI-assisted-by: deepseek-v4.1-flash |
||
|
|
51ae83202e | Merge remote-tracking branch 'origin/main' into staging | ||
|
|
10504ff2bf
|
🐛 Sanitize SVG on binfile import (#12105)
* 🐛 Sanitize smuggled SVG on binfile import Route imported SVG storage objects through the SVG sanitizer on all three binfile paths (v3, v1, v2), closing the second bypass of GHSA-ffhp-m958-qxvr. Integrity checks still run on the raw bundle bytes first; only the persisted copy is the sanitized one. Adds a shared sanitize-imported-svg helper with unit tests and a v3 export-tamper-import test proving smuggled scripts no longer survive the import. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Normalize SVG content-type on binfile import Close the residual GHSA-ffhp-m958-qxvr bypass where a crafted content-type spelling (uppercase, parameters) skipped the import sanitizer. Detection now uses a shared case-insensitive predicate, stored values are canonicalized, and the v3 re-read honors the import size limit. Extends the tamper tests to obfuscated spellings and adds exhaustive v1 coverage, including the tempfile branch. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Gate binfile content-type on known storage types Define the complete set of storage content-types in app.common.media and enforce it on the binfile import schema, so unknown types fail closed instead of passing through. Detection keeps a single normalization at the boundary with an exact predicate, and the string helpers use cuerdas. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Keep only produced types in storage-object-types Drop apng, avif, penpot and plain text from the set: no flow stores them, they only exist in the extension mapping table. Every member must have a producing code path; unproduced types stay out until some flow actually stores them. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Reject unknown content-type on v1 binfile import Enforce the storage content-type allowlist on the v1 import path, like v3 already does via schema. Bundles declaring types outside the set, or none at all, fail closed with the same media-type-not-allowed error as uploads. The sanitize branch is explicit: SVG bytes are sanitized, anything else passes through. AI-assisted-by: muse-spark-1.3-contributor |
||
|
|
753e98bb9f |
🐛 Raise default binfile import text total to 500MiB
Giant legitimate exports hold hundreds of MB of JSON across hundreds of thousands of KB-sized entries without tripping any per-entry cap, so the 200MiB cumulative default rejected them out of the box. Raise it to 500MiB; the 20MiB per-entry cap still bounds single-entry inflation. Pin the floor with a regression test. AI-assisted-by: muse-spark-1.3-contributor |
||
|
|
ab58922f2c
|
🚑 Revoke file access when a member leaves the team (#12097)
Delete the team-scoped file, project and pin relations on leave-team and delete-team-member, require a live team membership in the file, project, search and listing permission queries, and clean pre-existing orphan rows with a migration. Former members now get not-found on file actions instead of keeping read and rename access (GHSA-v9r9-h77c-55m2). AI-assisted-by: muse-spark-1.3-contributor |
||
|
|
94d6f5a25c | Merge remote-tracking branch 'origin/main' into staging | ||
|
|
0de865748d
|
🐛 Keep camelCase svg attribute names when importing a penpot file (#11974)
The json reader of the binfile v3 import rewrites every key of every entry to kebab-case, nested maps included. Shape `:svg-attrs` is the one map whose keys are camelCase react prop names, because the svg import path runs them through `attrs->props`, so an attribute exported as `fillRule` came back as `:fill-rule` and was stored that way. The renderer looks the attribute up by its camelCase name, does not find it and falls back to the default fill rule, so a shape exported with `fillRule: evenodd` was painted without its hole, and the attributes panel showed `fill-rule`. `clean-shape-post-decode` already runs on every shape right after the schema decode, for page shapes and component objects alike, so the repair goes there: run `:svg-attrs` back through `attrs->props`, the same transform that built the keys. It is idempotent, so shapes that arrive correct are left untouched. The new tests import a real export that carries the attribute, for page shapes and component shapes. Closes #11954 AI-assisted-by: space-bunny-free |
||
|
|
cdb0b3950d | Merge remote-tracking branch 'origin/staging' into develop | ||
|
|
773fc2295e
|
🐛 Raise validation error for incompatible binfile export options (#10341)
Requesting a binfile export with both `include-libraries` and `embed-assets` enabled threw a raw `IllegalArgumentException`, which the RPC layer surfaced to API clients as a generic `:server-error`/`:unexpected` (HTTP 500) with no actionable code -- the opaque failure reported in #7649 (perceived as an "empty" export). The two options are genuinely mutually exclusive, so the request is invalid input, not a server fault. Replace the raw throw in both the v1 and v3 export paths with `ex/raise :type :validation :code :incompatible-options`, matching how the rest of the binfile code reports bad input. Also fix the "mutally" -> "mutually" typo in the hint. Verified on a local devenv: the export RPC now returns `{:type :validation :code :incompatible-options}` instead of a 500, and valid single-option exports are unaffected. Adds a regression test. Fixes #7649 Signed-off-by: Filip Sajdak <filip.sajdak@siili.com> Signed-off-by: Andrey Antukh <niwi@niwi.nz> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Andrey Antukh <niwi@niwi.nz> |
||
|
|
ea7e5d2473
|
🐛 Add tokens status to the penpot import (#11850) | ||
|
|
7e1d5f3e88
|
✨ Bound binfile import db timeout and zip entry scanning (#11584)
Binfile import ran the whole import in a single transaction with idle_in_transaction_session_timeout disabled (= 0), so a stalled import could retain a connection pool slot indefinitely. Set a finite 20 minutes ceiling via SET LOCAL instead (a compile-time constant interpolated into the SQL; PostgreSQL does not accept bind parameters on SET). The v3 importer also located each file data by rescanning the full zip entry collection once per manifest file and once per page, making the cost close to quadratic on large files. Replace the per-file regex matchers with a single classification pass that groups entries by their raw path shape; consumers now lookup their entries per file and page. As a deliberate tightening, the .json suffix is matched literally: the previous regexes left the dot unescaped, so crafted paths like files/<f>/tokensXjson or objects/x-json matched by accident and are now ignored. Closes #11579 AI-assisted-by: omen-alpha |
||
|
|
38212e757c
|
🐛 Bound decompressed size of JSON entries on binfile v3 import (#11607)
* 🐛 Bound decompressed size of JSON entries on binfile v3 import Every JSON/text zip entry (manifest, files, pages, shapes, colors, components, typographies, tokens, plugin-data) was decompressed without any size limit, letting a small .penpot archive exhaust the backend heap (GHSA-qcw7-v626-g6cf). Only binary storage blobs were guarded. Reuse the existing size-limiting-stream guard on the text path: 20 MiB cap per entry, 200 MiB cumulative budget per import job, plus a cheap declared-size pre-check. Both limits are configurable and wired through the binfile, management and debug entry points. Adds zip-bomb regression tests for the file entry, the synchronous manifest read and the cumulative budget. Closes #11606 AI-assisted-by: muse-spark-1.3-contributor * ♻️ Uniform binfile import limits behind init-limits Move the binfile import limits to a single source of truth in app.binfile.common (default-* vars) and drop the duplicated entries from config/default; env overrides keep working through the schema. Resolve all limits once per job with init-limits (::max-size, ::total-max, ::current-size, ::max-object-size, ::max-zip-entries) instead of rebuilding the map per zip entry. Thread cfg as the first arg through the v3 readers, collapse read-plain-entry into read-entry, and give size-limiting-stream a single explicit-counter arity. v1 keeps using the compiled default (mechanical rename only). No behavior change. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Rename binfile limits to text-entry/binary-entry terms Use text-entry/binary-entry vocabulary consistently across config keys, bfc input keys, default-* vars and the limits resolved by setup-limits (::max-text-entry-size, ::max-text-total-size, ::current-text-size, ::max-binary-entry-size). Rename init-limits to setup-limits. No behavior change. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Rename cumulative text counter and document binary limit Rename ::current-text-size to ::accumulated-total-text-size for clarity and expand the default-max-binary-entry-size comment to match the other limit vars. No behavior change. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Harden binfile guards and prove budget accumulation Add a regression test that only passes when text bytes accumulate across entries (budget between largest entry and summed total; verified red against a per-entry atom). Include the entry name in streaming-guard errors, count skipped bytes against the budget with a direct unit test, and forward all four limit keys in get-manifest. No behavior change. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Thread cfg through get-manifest get-manifest now takes the caller cfg and resolves limits with setup-limits like the import job itself, instead of building a single-use mini-cfg from cf/get. No behavior change. AI-assisted-by: muse-spark-1.3-contributor |
||
|
|
a4becb5d1f |
🐛 Fix size-limiting-stream read arity on v3 binfile import (#11468)
The FilterInputStream proxy only implemented read() and read(byte[], int, int). Buffered reads call read(byte[]) (and read(byte[], int) via Clojure interop), causing ArityException while hashing storage objects and breaking v3 imports. Implement all read overloads and extract shared byte-count logic. |
||
|
|
b46ed37141
|
🐛 Fix size-limiting-stream read arity on v3 binfile import (#11468)
The FilterInputStream proxy only implemented read() and read(byte[], int, int). Buffered reads call read(byte[]) (and read(byte[], int) via Clojure interop), causing ArityException while hashing storage objects and breaking v3 imports. Implement all read overloads and extract shared byte-count logic. |
||
|
|
f5aad7b1ae | Merge remote-tracking branch 'origin/staging' into develop | ||
|
|
92c2079ae2
|
🐛 Add configurable limits for ZIP entry count and object size in v3 import (#11022)
* 🐛 Add configurable limits for ZIP entry count and object size in v3 import Add binfile-import-max-zip-entries (default 500,000) and binfile-import-max-object-size (default 100 MiB) config entries. Both are configurable via PENPOT_BINFILE_IMPORT_MAX_ZIP_ENTRIES and PENPOT_BINFILE_IMPORT_MAX_OBJECT_SIZE env vars. Entry count is checked before processing begins. Per-object size is checked after each storage object content is resolved. AI-assisted-by: mimo-v2.5-pro * 🐛 Enforce actual decompressed byte limits on v3 import The previous object-size check trusted the ZIP entry header's declared size (ZipEntry.getSize()), which a malicious zip-bomb can forge. The check would pass, then the full decompressed payload would be read anyway during hashing and storage persistence. Add size-limiting-stream, a FilterInputStream wrapper that counts actual bytes read and raises :validation :max-file-size-reached when the configured limit is exceeded. Wire it into zip-entry-storage-content so both the hash calculation and storage write paths are bounded by real decompressed bytes, not declared header size. Also wire import limits into management.clj (clone-template) and debug.clj (import-handler + clone path) for defense-in-depth, and add a test that exercises the object-size limit with a real storage object in the exported ZIP. AI-assisted-by: mimo-v2.5-pro |
||
|
|
0e388442a1
|
✨ Add storage object status lifecycle and verified dedup (#11345)
* ♻️ Simplify storage GC delays and add skip-delay task params The touched GC no longer applies an extra deletion-delay when marking storage objects as deleted. By the time a storage object is touched, its referencing domain row has already passed its own deletion delay, and the reference scan is the only safety check needed. Touched objects are now marked with deleted_at = now, so the deleted GC removes them on the next run. For the tempfile bucket, upload chunks now set touched-at in the future (1h, aligned with the upload-session-gc TTL) instead of relying on a special-case deletion delay. Task handlers now read their task props: - storage-gc-touched accepts :skip-delay to process all touched objects immediately, bypassing the min-age threshold. - objects-gc accepts :chunk-size and :skip-delay to process recently deleted rows without waiting for the deletion delay. This allows running the deletion cascade immediately from the REPL via run-task! with the skip-delay option. AI-assisted-by: deepseek-v4-flash * ✨ Add storage object status lifecycle, verified dedup, and deletion retry tracking Storage object lifecycle hardening: - Add status column ('valid' | 'pending') as write-ahead marker for object creation. put-object! inserts in 'pending' state, writes blob, then promotes to 'valid'. Failed writes remove the pending row. - Add :storage-pending-gc task to reclaim orphaned pending rows (e.g. after crash between blob write and promotion). - Verify blob existence on every dedup hit via exists-object? (fs stat / s3 headObject). Missing blobs mark the row as deleted and create fresh object. - Add deletion_attempts column (migration 0154) to track physical blob deletion attempts. Restructure gc_deleted to use chunked processing with per-chunk transactions (short lock duration). Failed deletions are deferred to tomorrow (deleted_at = NOW() + 1 day) to prevent infinite loops. After 7 attempts, give up and accept orphan. - Change del-objects-in-bulk contract to return #{fail-ids} for precise per-id tracking (fs and s3 backends updated). - Use tmp/tempfile for fs atomic writes with cleanup queue registration (crashed-JVM temp files swept ~60min later). Document ATOMIC_MOVE POSIX-only assumption. - Add linear backoff to s3 exists-object? retries (100ms/200ms/300ms). - Wrap compensating delete in put-object! catch block to prevent masking original error when connection is aborted. - Fix assert messages in pending_gc.clj and gc_deleted.clj (pool assertion said 'expected valid storage' instead of 'db pool'). - Add pending-objects-excluded-from-gc-deleted test. Use unique path in put-object-write-failure-leaves-no-row test to avoid collisions. AI-assisted-by: qwen3.7-plus * 🐛 Fix review comments on gc-deleted and storage - Fix process-chunk! returning nil causing (+ acc nil) crash - Add FOR UPDATE SKIP LOCKED to sql:get-deleted-chunk to prevent infinite loop when another worker holds locks - Pass :cause to log messages in gc_deleted.clj and s3.clj - Fix extra space in log hint string - Remove unused ::blob-missing? reference from storage memory - Rename test to match actual behavior (leaves pending row) - Add test for gc-deleted giving up after max attempts AI-assisted-by: qwen3.7-plus |
||
|
|
03cd3fa70f
|
♻️ Consolidate auto-link libraries with unified export-type and fix ref integrity (#9958)
* ✨ Auto-link libraries during import based on slugified name When a Penpot file is exported without bundled libraries and then imported into a different environment, external library links are broken because library UUIDs differ across environments. This feature adds a heuristic to auto-relink libraries by matching slugified library names against shared files in the target team: - Export: embed external library metadata (id, name, slug, used-by) in the manifest when libraries are not included in the export. - Import: resolve external libraries by slugifying shared file names in the destination team and matching against manifest slugs. - Single match: auto-link silently (creates file-library-rel row). - Multiple matches: emit SSE event so the frontend shows a selection dialog for the user to pick the correct library. - No match: import continues without linking (current behavior). Backend changes: - Extended manifest schema with optional :external-libraries field - Added slugify-name, get-files-names, get-shared-files-for-team, find-shared-files-by-slug helpers in app.binfile.common - Threaded team-id into import cfg from RPC layer - Added resolve-external-libraries and auto-link-libraries in v3 - Emit :library-candidates SSE event for multi-match cases Frontend changes: - Worker captures library-candidates SSE events and forwards them - Import dialog shows auto-link notification and multi-match selection UI with select dropdowns - Added link-files-to-library! RPC helper for user selections - Added en/es translations for new UI strings Closes #9263 Signed-off-by: Andrey Antukh <niwi@niwi.nz> * ✨ Add UI for the auto-link plumbing * ♻️ Consolidate auto-link libraries with unified export-type and fix ref integrity Consolidates the auto-link libraries feature into a single coherent implementation: - Unify boolean flags (embed-assets, include-libraries, link-later) into single ::bfc/export-type parameter - Fix critical reference-integrity bug: pre-resolution no longer remaps :component-file refs when no link is created (multi-match / no-permission) - compute-link-decisions as single source of truth for auto-link logic - 80+ backend tests covering round-trip, cross-team, permissions, edge cases, and reference integrity AI-assisted-by: longcat-2.0 --------- Signed-off-by: Andrey Antukh <niwi@niwi.nz> Co-authored-by: Eva Marco <evamarcod@gmail.com> |
||
|
|
81c3b3cd56
|
📎 Update copyright name on file header (#11346) | ||
|
|
47d599fe34
|
✨ Persist binfile manifest and emit workspace audit events (#11106) (#11138)
Persist binfile manifest metadata in file_data on import so file statistics are available at open-workspace time. Emit a new open-workspace-file audit event enriched with file statistics: page count, shape count, component count, linked libraries, design tokens, and whether the file is a shared library. Closes #11106 AI-assisted-by: mimo-v2.5-pro |
||
|
|
5906312dff
|
🐛 Normalize error response on duplicate file ID (#11050)
Capture unique constraint violation in insert-file! and return generic :not-found error instead of propagating raw PostgreSQL exception, preventing file existence oracle. AI-assisted-by: mimo-v2.5-pro |
||
|
|
689d3a1be2
|
🐛 Add max-object-size guard to read-obj! in v1 parser (#11018)
Prevent unbounded memory allocation when a crafted binfile specifies an excessively large object size. Apply the same 100 MiB limit that read-stream! already enforces. AI-assisted-by: mimo-v2.5 |
||
|
|
4a8fb5af53 | Merge remote-tracking branch 'origin/staging' into develop | ||
|
|
c5de4c27b0 | Merge remote-tracking branch 'origin/main' into staging | ||
|
|
ddba2ffa75
|
📎 Update Kaleidos Copyright (#9929) | ||
|
|
56d8dc678c
|
🐛 Populate is-indirect flag on file libraries from relation graph (#9289)
Co-authored-by: Andrey Antukh <niwi@niwi.nz> |
||
|
|
52588412c7 | Merge remote-tracking branch 'origin/staging' into develop | ||
|
|
fffafdab93
|
🐛 Fix library updates reappear after file is reloaded (#9563)
* 🐛 Fix library updates reappear after file is reloaded Summary Migrate synced_at timestamps to a standalone file_library_sync table to ensure sync state is tracked for both direct and transitive libraries. Problem Transitive libraries (libraries imported by other libraries) are not stored as direct rows in file_library_rel. Because the system previously coupled synced_at directly to the file_library_rel schema, transitive libraries lacked a persistent location for their sync timestamps. This caused sync states to be lost or incorrectly reported for nested dependencies. Changes Schema Migration: Created file_library_sync and migrated existing synced_at values from file_library_rel. Decoupling: Removed tight Foreign Key coupling to allow sync rows to exist independently of specific relationship records. Persistent Writes: Added upsert-file-library-sync! helper. Updated all import, duplication, and RPC write paths (v1/v2/v3 importers, link-file-library) to ensure every write persists a sync row. Unified Reads: Updated both direct and recursive/transitive library queries to fetch synced_at from the new table. Testing: Added regression tests to verify that sync rows are correctly created/updated even when a transitive relation is absent in file_library_rel. Impact This fix ensures that the system accurately records and retrieves sync states for the entire library dependency tree, resolving the bug where nested libraries appeared out of sync. * ✨ MR review |
||
|
|
06986e25a3 | Merge remote-tracking branch 'origin/staging' into develop | ||
|
|
15379f37f5 |
🐛 Fix maximum call stack size exceeded in SSE read-stream (#9484)
The recursive `read-items` function in `app.util.sse/read-stream` caused a synchronous stack overflow when reading buffered stream data. Each `rx/mapcat` call chained another recursive invocation on the same call stack without yielding to the event loop. Replace the recursive pattern with an `rx/create`-based async pump that uses Promise `.then()` chaining, keeping the call stack depth constant regardless of stream size. Also add progress reporting with names and IDs during binfile export and import, and bump `eventsource-parser` dependency. Closes #9470 Signed-off-by: Andrey Antukh <niwi@niwi.nz> |
||
|
|
ec0d692856
|
🐛 Fix maximum call stack size exceeded in SSE read-stream (#9484)
The recursive `read-items` function in `app.util.sse/read-stream` caused a synchronous stack overflow when reading buffered stream data. Each `rx/mapcat` call chained another recursive invocation on the same call stack without yielding to the event loop. Replace the recursive pattern with an `rx/create`-based async pump that uses Promise `.then()` chaining, keeping the call stack depth constant regardless of stream size. Also add progress reporting with names and IDs during binfile export and import, and bump `eventsource-parser` dependency. Closes #9470 Signed-off-by: Andrey Antukh <niwi@niwi.nz> |