mirror of
https://github.com/penpot/penpot.git
synced 2026-11-19 08:30:42 +00:00
199 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
e23ca02aa2
|
✨ Reference the file data storage object with a foreign key (#12769)
* ✨ Reference the file data storage object with a foreign key The file data pointer lived in `file_data.metadata.storage-ref-id`, a JSON key. The storage GC parsed it to find live references, and nothing stopped it from physically deleting an object a live row still pointed at: a failure in that scan lost the blob. Migration 0159 moves the pointer to a nullable `storage_object_id` column on `file_data`, backfills it from the metadata key, removes the key from the rows it backfills, indexes the column and rebuilds the foreign key to `storage_object(id)` as `NO ACTION DEFERRABLE INITIALLY IMMEDIATE`. `INITIALLY IMMEDIATE` is deliberate: the GC catches the violation inside the chunk transaction, while a deferred check would fire at commit, outside it, and the sweep would never move past the blocked chunk. The column is the only source of truth. The `storage-ref-id` entry is gone from the file metadata schema, and the single point that serializes the metadata drops it, so no write can resurrect it. `upsert!` does not query the row: the pointer of the row being replaced travels in its params, and the object is touched before the row is written. Fragments and snapshots pass nil because a modified pointer always gets a new id; the offload passes its own row; `insert-file!` drops it, since an INSERT never replaces a persisted object. `update-file!` takes the file map read from the database plus the fields to change and merges them, so the pointer comes from the read map and no caller can lose it by building a partial map. A map without `:data` raises `:missing-file-data`: it is not `assert`, which is compiled out in the deployed binary. The params cannot override the pointer or retarget the write with `:id`/`:file-id`. `bfc/file->response` strips the pointer from every response (`get-file`, `create-file`, `duplicate-file`, the view-only bundle, the team access request, the comments read, the library summary, the snapshot read and the debug download). A `file-data` object is authentication-only, so a leaked id would let any logged-in profile fetch the blob from `/assets/by-id`. `get-file-fragment` resolves through the file data backend, so a fragment on cold storage no longer hands nil to `blob/decode`, and answers only the keys of its result schema. The storage GC reference scan, the restore union and `objects-gc` read the column. The offload reads `SELECT fd.*`, so its rows go through a `decode-row` that decodes jsonb metadata and drops a JSON null, which the row schema rejected; a row without a readable object raises `:missing-storage-object` with the file and row ids. A chunk blocked by the foreign key (SQLState 23503) is a real anomaly, since the reference scan should have frozen the object: it stays an error with its own hint and the chunk is retried on the next run until the reference is fixed. The release is a coordinated restart, not a rolling deploy, and not rollback-safe for the rows it writes: the previous code writes and scans the metadata pointer, so a key it writes misses the backfill and removing it loses the reference. `operations/2.20.0-file-data-storage-object-id.md` builds the column, the index and the foreign key online and runs the backfill with every instance stopped, then marks the step as applied; a fresh or small instance does the whole job in the migration. The runbook keeps the validation scan out of the stopped window and adds a lock timeout and retry to the online statements. Tests cover the foreign key shape, the backfill and its corrupt and orphan guards, the rebuild of a missing index and of a wrong, non-deferrable or deferred constraint, the deletion refused by a live reference, the metadata cleanup, the offload row normalizations, the touched objects of the unshare and the restore, the response stripping on every call site and the chunk blocked by the foreign key. The backend memories and the developer guide document the pointer, the GC behavior and the two migration paths. Closes #12768 AI-assisted-by: deepseek-v4.1-flash AI-assisted-by: inkling-2-small-free * 🐛 Hold the file row lock across the migration write `get-file` applies the pending migrations in memory and, when it finds any, persists the result. That write read the row without a lock and `update-file!` is an unconditional UPDATE, so a save that committed in between was overwritten: the saved content, `revn` and `modified-at` went back, the next save from that client failed with `:revn-conflict` and the object the save left became an orphan the storage GC cannot revisit once it is frozen. The read now runs in a transaction of its own (`get-file` calls `get-file*` inside `db/tx-run!` and accepts a cfg map only), so the `file` row lock covers the re-check and the write. `get-file*` reads the row once (`sql:get-file` plus `resolve-applied-migrations`) and completes it with `attach-file-data`; only when a migration is pending it reads the file again under `FOR UPDATE`, re-checks `need-migration?` on that fresh map and persists it (`migrate-file`). The common case stays at one row read and one migrations resolution, the migration case keeps a single decoded copy in the heap, and a row that changed in between is not overwritten. A migration found on the unlocked read is applied in memory only, for the next read to persist. `migrate-file` returns the reattached map when it persists, with the `:storage-object-id` of the row the write left, and the realized map when it does not; `:realize?` is what returns the read fully plain. `fdata/upsert!` answers `{:written ... :storage-object-id ...}` instead of a boolean, `update-file!` propagates it and the migrating read puts it back in the map it returns, so a caller that persists again touches the current object instead of the replaced one. `insert-file!` keeps nil: an INSERT never replaces a persisted object. The write also compares the stored pointer with the caller's value (`IS NOT DISTINCT FROM`) and raises `:stale-file-data-pointer` when it affects no row. Every writer holds the `file` row lock, so a pointer that no longer matches is a protocol violation, not a race to retry. The expected pointer is a separate argument because the storage branch replaces `:storage-object-id` before the upsert, and the check lives in `upsert-in-database`: the `legacy-db` branches do not use that SQL and their count can legitimately be zero. The save, snapshot and restore-backup reads take `FOR UPDATE` instead of `FOR SHARE`: they inherit the lock, and the migration branch asks for the exclusive one in the same transaction, so a shared lock that upgrades deadlocks two concurrent readers. A data read rejects `:lock-for-share?` for the same reason; only row-only reads accept it. `sql:get-file-without-data` is a plain select now, not a CTE, which makes locking legal without loading the data. Realization has a single owner: `attach-file-data` only resolves and decodes, and `get-file*` realizes once per path. The applied migrations no longer travel in the `get-file-with-summary` response, cache hit and cache miss alike. `embed-assets` reads each library without `:realize?`, so its `:components` is an unloaded pointer map; `absorb-assets` walks it with `vals`, `PointerMap.seq` calls `load!` and that raised with no loader bound, so every export with `:merge-libraries` of a library with components failed. The absorb runs under the library's own loader now. The in-place binfile import (`import-file-and-overwrite*`, `::overwrite`, `::bfc/file-id`) had no product caller and cannot carry the previous `storage_object_id`, so overwriting a storage-backed row would leave its object unreclaimed. `import-files!` rejects a cfg with `::bfc/file-id` (`:not-supported`) and the helpers only that path used go with it (`invalidate-thumbnails`, `*reference-file*`, the `::overwrite` id branch of `index-object` and of `save-file!`, the `on-conflict-do-nothing?` flag of the media insert). Tests: a new `file-migration-read-test` pins the single read, the locked re-read, the concurrent save interleaving, the double migration and the read-only path; the upsert tests cover the return shape and the stale pointer; the SQL recorder they share lives in `backend-tests.helpers`, and the save, snapshot, restore and summary sequences follow the new reads. The backend memories document the read contract and the pointer. AI-assisted-by: deepseek-v4.1-flash * 🐛 Harden the get-file guard and skip its read-only transaction The cfg guard of `get-file` was an `assert`, which `app.config` elides when `:backend-asserts` is off, so a deployed binary accepted a bare pool or connection while the docstring promised it raises. It throws `IllegalArgumentException` now, the class the option guards of the same namespace already use, and the existing test expects it instead of `AssertionError`. A read with `:read-only?` applies the pending migrations in memory and writes nothing, so the transaction the migration write needs only costs two round trips there. `get-file` runs it with `db/run!` instead: with a cfg that carries a connection it still joins the caller's transaction, and only the reads that would have opened their own skip it. A lock option needs a read-write transaction from the caller, because PostgreSQL rejects row locks in a read-only one; the docstring states that. Tests: the guard test expects the new class; a cfg marked `::db/read-only` reads a pending migration without persisting it (the config flag that fills that key is legacy, but the key is set by hand); and a read-only read is pinned as not opening a transaction while the same call without the option does. The backend memory documents the split and the guard. AI-assisted-by: deepseek-v4.1-flash |
||
|
|
34c702bc80
|
✨ Add table partitioning for file-change (#12417)
* ✨ Partition file_change by file_id Partition `file_change` by `HASH (file_id)` in 16 partitions, with the same bounds as `file_data`, so a join between them is resolved partition by partition. The primary key becomes `(file_id, id)`, which a partitioned table needs, and every access filters by `file_id`. The new table carries a `kind` column (1 snapshot, 2 revision) as a reserve: the queries still tell them apart by the label, and `kind` takes over when they are rewritten. The payload of a snapshot stays in `file_data`; the `data` column is kept for reading old `legacy-db` payloads. The four snapshot commands require `file-id`, because it prunes the partition. The objects GC locks and deletes by the pair, in batches. The old table is kept as `file_change_legacy` as the way back. The automatic migration refuses to run above 10 GiB or 2M rows and points to the runbook; big instances partition online with that runbook, which copies by id, follows the `updated_at` watermark in delta passes and checks that no row of the source is missing from the target. Closes #12416 AI-assisted-by: deepseek-v4.1-flash * 🐛 Address the review of the file_change partition The production runbook promised a rollback it did not have: the section was a sketch, and its statements could write to the table about to be dropped. The rollback is removed, and so is every comment that described it. `file_change_legacy` stays as a frozen copy of the previous layout. The cutover check now aborts the transaction when a row of the old table is missing in the new one, instead of leaving the decision to the operator. The five secondary indexes of the new table are built after the last delta pass: the bulk and the delta passes only need the primary key, so the batches no longer maintain them. The copy of 0158 is now replayed in a test against the pre-0158 shape with a labelled and an unlabelled row, plus an empty instance that must leave no legacy table behind. The updated_at assertion of the snapshot mutations is bounded by the test clock, and the repair revision stops writing the `features` of the file, like the normal save. AI-assisted-by: deepseek-v4.1-flash * 📎 Fix linter issues * 🐛 Fix stale helper call in rpc_file_snapshot_test The suite does not load: `th/run-pending-tasks!` does not exist in backend-tests.helpers. The pending jobs runner is `run-pending-jobs` (no arguments), the name every other call site of the suite uses. AI-assisted-by: inkling-2-small-free * 📚 Align the partition change with the 2.20.0 milestone 2.19.0 is already shipped; this work targets 2.20.0. The operations file of the partition procedure carries the 2.20.0 prefix, the references that point at it (the migration, the runbook table and the memory) follow, and the RPC doc/changes tags of the snapshot commands record 2.20 as the release that makes `file-id` required. AI-assisted-by: inkling-2-small-free * 🐛 Fix review findings on the partition rebase The 0157 legacy test was missed by the rebase and inserted file_change rows without the new mandatory `kind` column, so the backend suite fails against the migrated schema. The inserts now carry `:kind` (snapshot-kind for the labelled rows, revision-kind for the rest). The post-cutover verification block of the runbook terminates the first query with a semicolon so the block pastes into psql without a syntax error. The guard message of the migration names the step (0158-partition-file-change) instead of "0158", ambiguous since two migrations share that number. Closes #12416 AI-assisted-by: inkling-2-small-free * 🐛 Fix the stale label in the snapshot restore test The rename block of snapshots-crud runs before the restore block, so the restored version is already named "label1-updated" when the restore publishes its message. The command names the version by the label the row has at that moment, not by the one it had when it was created, so the test was asserting a stale value. Assert the current label, and say why in the comment. AI-assisted-by: deepseek-v4.1-flash * ♻️ Split the version event tests into one test per command The workspace version events and the plugin proxy send the file-id of the open file on every snapshot command, because the command needs it to prune the partition of file_change. Both tests checked that with a doseq and t/testing blocks, so a failure named the whole group and not the command that broke. Give each command its own test. The workspace ones share a helper that runs the event and returns the command and params it sends; the plugin ones keep their own mock of rp/cmd!. AI-assisted-by: deepseek-v4.1-flash |
||
|
|
93d36d04d8
|
✨ Stabilize fragmented file data across snapshots and GC (#12420)
* ✨ Stabilize fragmented file data across snapshots and GC Snapshots, automatic and user, now store pointer-based rows that share fragments with the live file instead of duplicating page content. `get-file-snapshot` resolves those pointers on the server so version previews keep rendering plain data, and `restore!` reuses the shared rows. Export still materializes (the package carries no fragments) and import regenerates them. `file-gc` treats the main file and every unpurged snapshot as liveness roots, reads the file unrealized, and reattaches the original pointer wherever the cleaned value is unchanged, so a pass over an untouched file writes no fragments and a pass over an edited file keeps the ids of the pages it did not touch. Snapshot creation and the restore backup take the file row `FOR SHARE`, which serializes them with the pass and closes the race that let a new snapshot reference fragments the pass was about to mark. All backend pointer bindings go through the `fdata/with-pointers`, `with-tracked`, `with-tracked-pointers` and `with-untracked` macros. The loader reads fragments a previous pass marked until `objects-gc` purges them, and the frontend fails loudly with retry instead of dropping an unresolved fragment. Review fixes in this branch: the automatic snapshot attaches before pending migrations, `realize-pointers` is the single verb, `reattach-unchanged` takes the identity fast path, the restore guard takes its revn from the backup read, and `PointerMap.withMeta` generates a fresh id when metadata changes. Tests cover both snapshot writers (red without the lock), snapshot sharing and pin release through the real delete and purge paths, reattach against orphan marking, preview after offload, the pointer binding contracts and the with-meta id change. Closes #12410 AI-assisted-by: muse-spark-1.3-contributor AI-assisted-by: deepseek-v4.1-flash ♻️ Pin no-churn behavior for untouched pages Pin the behavior the churn verification measured: a save keeps the fragment id of every page it does not touch, with a page of eleven keys (PersistentHashMap), a 46-shape objects map and a nine-page index (hash map). Also pin the mixed case: a data-level objects-map flip rewrites the old pointer pages once and the next save keeps the new ids. The memory records the identity rule behind it: re-assoc with the same value returns the same map in array and hash maps alike. Closes #12410 AI-assisted-by: deepseek-v4.1-flash ⚡ Stop loading untouched pages when enabling objects-map `enable-objects-map` wrapped every page on every save, and reading a page's objects forced its PointerMap to load the fragment. A save of one page in a seven-page file therefore made seven fragment reads. Skip unloaded pointers: their fragment was already wrapped by the encode-file that wrote it, so loading it only to wrap it again is a no-op. The save now loads only the pages it edits. A page with a plain fragment (a data-level feature flip) keeps its plain fragment until it is first loaded or edited; a migration that enables the feature must rewrite the data itself. Cover the change with load-count tests for save and restore, update the flip test to the new contract, and drop the stale restore comment. Closes #12410 AI-assisted-by: deepseek-v4.1-flash ⚡ Keep page fragment ids through pending migrations Applying a pending migration on read realized the file, ran the migration and persisted it again; the persist wrapped every plain page in a new PointerMap, so every page got a new fragment id even when the migration did not change it. That leaves the fragment cache cold and rewrites every fragment on the first read after a deploy. Capture the original pointers before realizing, reattach the unchanged positions after migrating (same value comparison file-gc already uses), persist the reattached file and return the realized one: the read contract is plain data. Read-only connections skip the persist and the reattach. A migration that adds objects-map skips the reattach so the pages get wrapped. Move the three helpers to fdata so file-gc and the migration path share them, and cover the path with a real 0031 migration test: the unchanged page keeps its id, the changed page gets a new one, the caller gets realized data and a second read does not migrate again. Closes #12410 AI-assisted-by: deepseek-v4.1-flash * 🐛 Realize library pointers with each library's own loader Fragment rows are keyed by file id, and every caller binds the fragment loader of the file it works on. get-resolved-file-libraries returned libraries with their pages still as pointers, so the first read of a library page under fdata/pointer-map asked the consuming file for the fragment and failed with :fragment-not-found. That broke migrations and file validation for every file using a pointer-mapped library. The weak-map loader now realizes each library's pointers with the library's own id. The test fails without the change. AI-assisted-by: claude-opus-5-5 Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> * 🐛 Restore the pointer-map require the repair path still needs The branch dropped `app.util.pointer-map` from the namespace while the auto-repair path (added to develop before this branch) still rebinds `pmap/*tracked*` around validation and repair in `repair-preexisting-damage` and `process-changes-and-validate`. The namespace did not compile, so the whole backend failed to load. Those toggles do not fit the fdata macros: `with-untracked` binds nil and `with-tracked` creates a fresh atom, while the repair path has to turn tracking off and restore the existing atom (and read it before the repair starts). Restore the require and keep the bindings explicit. Also pin the feature-off path with a test: with `fdata/pointer-map` off (the default), a save creates no fragments, file-gc writes nothing and the snapshot preview still returns plain data. Drop a stray `\n` char literal left at the end of one deftest. Update the memories with the exception to the macro rule, the snapshot FOR SHARE locking and the pre-migration automatic snapshot. Closes #12410 AI-assisted-by: deepseek-v4.1-flash * 🐛 Validate debug files with their own pointer loader The `/dbg/actions` file validation passed the file id where `get-resolved-file-libraries` expects the file map, so the library map was seeded with a nil key. It also validated a file read without realizing it, so the first page lookup of a pointer-mapped file raised "load is not supported when *load-fn* is not bind" and the endpoint could not validate exactly the files the feature touches. Read the file realized (validation walks every page and component anyway) and build the library map from the file. The library loader already resolves each library's pointers with its own id, so the realized file and its libraries validate as plain data. Covered by a handler-level test that fails before the change. AI-assisted-by: deepseek-v4.1-flash * ♻️ Bind the snapshot loader through the fdata helper `reduce-snapshots` hand-rolled the same `binding` that `fdata/with-pointers` builds (it also loaded the function's own pointer loader before entering the reduction). Use the macro so the rule holds there too and the namespace no longer needs `app.util.pointer-map`; the loader is identical (same cfg and file-id) and snapshot walking behaves the same. AI-assisted-by: deepseek-v4.1-flash * 📚 Note the pointer-binding exceptions and snapshot ordering Record what the feature review settled: - Two direct `binding`s stay on purpose: the fine-grained `*tracked*` toggles of `app.rpc.commands.files-update` (validation/repair turn tracking off and restore the existing atom) and `binfile.common/encode-file` (fresh tracked set inheriting the current one). Atom-scoped macros would only be worth it if the set grows. - The automatic snapshot is taken before the pending migration, and the pre-migration window is narrow: the `get-file` RPC migrates on read and its ETag hashes the available migrations, so a post-deploy read cannot be a 304 from the pre-deploy state. The old-format snapshot only appears when the save is the first request to touch the file after the deploy, and `get-file-snapshot` previews it as stored. AI-assisted-by: deepseek-v4.1-flash * 🐛 Apply pending migrations when previewing a version Automatic snapshots are attached before the save applies pending file migrations, so a version can store the old format. The preview RPC handed that data to the UI as stored, and the viewport renders it with the current code. Realize the snapshot with the live file id (fragments are keyed by file, not by snapshot), normalize its migrations to a set (the decoder keeps the pgarray as a vector, and `set/difference` would compare indexes) and apply the pending migrations with the pure `fmg/migrate-file` and the live libraries. Nothing is persisted: the stored version keeps its data and restoring it re-migrates on the next read. The response now carries the snapshot's migrations. Use `:features` the way `get-file` does for `fdata/objects-map`: a client that does not declare it gets plain objects. The historical feature set is not checked against the enabled one on purpose: a version may predate a feature change. Cover the preview of a snapshot with a pending migration and the objects realization; both pin that the stored row is untouched. Update the file-data memory with the preview contract and the vector/set trap. AI-assisted-by: deepseek-v4.1-flash * 🐛 Decode snapshot migrations as an ordered set `file-snapshots/decode-snapshot` handed the `file_change.migrations` pgarray as a vector. `set/difference` falls back to index lookups when the first collection is shorter than the second, so the pending set was right only by accident: a stored subset happens to work, while a stored set larger than `available-migrations` (a snapshot taken by a newer build) marks every available migration as pending and re-applies it to already migrated data. Decode it as a `d/ordered-set`, the type the rest of the code and `schema:file` expect. This also covers `restore!` and the snapshot list, and it lets `get-file-snapshot` drop the `(into #{} ...)` normalization it was carrying. Re-wrap the union in `fmg/migrate` so the invariant holds even when a caller hands a plain set. A snapshot with NULL `migrations` predates SQL migration 0140 and keeps the format version of its time. Pin that legacy path: the preview regenerates the applied set from that version, migrates in memory and leaves the stored row untouched. Update the file-data memory with the invariant and the legacy path. AI-assisted-by: deepseek-v4.1-flash * 🐛 Fix flaky waiting-gauge assertion in climit test The throwable-while-waiting test asserted the exact list of waiting-gauge publications, but the component sampler republishes the same count when its first tick lands inside the recorded window under load. Assert only the last value, which is the contract the helper documents. AI-assisted-by: muse-spark-1.3 --------- Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro> Co-authored-by: Álvaro Tejero Cantero <alvorithm@teje.ro> |
||
|
|
29373fff9f
|
✨ Rework the progress of the import and export dialogs (#12192)
* ✨ Rework the import dialog The import wizard paints a Check/Upload/Import stepper derived from the dialog status, plus a phase title, a description and a progress bar per phase, and its rows follow the new design: folder icon, file name and a size subtitle. While checking, the dialog shows no rows and no spinner, because the phase header already carries the message. The wizard writes the job milestone once, in the header (`jp/current-milestone-text`), since every file of a container reports the same one. A row keeps its name and its size, says "Ready" once its container finished uploading, and shows the error text when it fails; the button that drops a file only appears while the wizard still offers its import step. The `jobs.queued` label now reads "Waiting job slot". The bar measures the whole import: determinate while the containers are uploaded, weighted by their size (`imp/upload-pct`), and animated otherwise, because the stages of a job are not comparable. The phase title counts the files, and `imp/imported-count` counts them done with the job's own counter, so the number moves while a container rebuilds its files one by one. The analysis reads the entries of a version 3 package in one pass: a file shows what it occupies inside its container, its own entries plus the storage objects its media and thumbnails reference, while what the container weighs stays in `:container-size`, the size the upload bar is weighed on. The ready step waits for the end of the analysis (`imp/ready-to-import?`), not for every known file, so Continue never leaves out the files that arrive later. Tests cover the file sizes, the upload percentage, the done counter, the wizard step and the milestone in course. The change also adds `-XX:NativeMemoryTracking=summary` to the backend devenv so `jcmd` shows native memory, and documents the wizard, the promise call styles (`mem:clojure/promesa`) and two CSS pitfalls: a logical `inline-size` wins over an inline physical `width`, and Rumext leaves a plain `:style` map raw, so inline styles leave CLJS as `#js` objects. AI-assisted-by: glm-5.3-flash AI-assisted-by: deepseek-v4.1-flash * ✨ Rework the export dialog Every exported file is its own job, so each row reports its own state: folder icon, name and a size-plus-`.penpot` subtitle; waiting for a worker; running with the milestone of the job and an animated bar; or exported with a tick. A failed row paints red, takes the close icon and shows the hint of the job, or a generic label, through `files-export-modal.export-error`. Success, failure and cancellation replace each other, so a row never reports two outcomes at once. The ids of the messages are canonicalized to string before they match a row, because they cross serialization boundaries. The export result now carries `:size`, so the subtitle says the format until the job produces the artifact and the size once it exists; a transport failure builds a synthetic error, so a nil `ex-data` no longer leaves the row hanging. The header counts the files and says whether the export runs or ended; since every file has its own job, it paints no global bar. Closing changes too: the Close button is gone. While files are in flight the footer offers Cancel export and the X cancels as well, which marks the loading rows cancelled and leaves them in place, so their outcome can be read; once every row reached a terminal state the footer offers Done, which dismisses the dialog. The retry button of the finished design is left out, because the client has no retry mechanism yet. Styles come along: the rows scroll while the header and the footer stay put, and the new phase header, row and bar follow the tokens of the import wizard. The title key becomes `files-export-modal.title`, and the new keys cover the header, the row states and the buttons. AI-assisted-by: muse-spark-1.3 * ♻️ Inline the binfile job adapters into their task handlers The wrappers in app.binfile.jobs only forwarded to the core, so each task handler now composes its own run: frozen cfg, active check, progress bridge and format dispatch. export-cfg freezes the whole export (file-ids, export-type, output) and v3/export-files! takes it as its single argument; ::bfc/ids becomes ::bfc/file-ids. The bridge keeps one job: bfj/run forwards the cfg to its callback, and check-active answers with it to thread the run. AI-assisted-by: muse-spark-1.3 * 🐛 Migrate binfile tests to the single-arg export-files! The refactor made v3/export-files! take a single cfg carrying ::bfc/file-ids and ::bfc/output, but six calls in binfile_test (added by the tokens handling change) still used the old two-arg form with ::bfc/ids. Move them to the new form, as the rest of the file already does. AI-assisted-by: muse-spark-1.3 * 🐛 Fix export artifact expectations to include file size The export flow now reports the artifact size, but the stale expectations dropped it and failed the frontend suite. Include the size in both assertions so they match the emitted message. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Migrate remaining export-files! test callers to the single-arg form rpc_binfile_test and offload_file_data_test still passed the output apart and used ::bfc/ids; move both into the cfg as ::bfc/file-ids/::bfc/output. * 🐛 Hide the downloads hint when no file was exported After Cancel or when every job failed, the export header said "0 files exported · Check your downloads folder."; drop the hint then. * ♻️ Read the object of a container entry once The index of a container asked twice for the object of the same zip entry: once to test it and once as the key of the update. Resolve it once, in the branch that owns it, so the entries of a file never pay for a question that does not apply to them. Addresses a review note on #12192. AI-assisted-by: deepseek-v4.1-flash * ♻️ Never size an import row with its container's weight A row of a version 3 package took the size of the container whenever the container held no entry for its file, while the comment beside it promised the opposite. An unknown size now stays nil, which the row already understands: it paints no subtitle instead of a number that belongs to the whole package. The test carries a file the manifest lists and the container does not hold, and the memory records what the row does then. Addresses a review note on #12192. AI-assisted-by: deepseek-v4.1-flash * 🐛 Retry pool stats read until snapshot is consistent Hikari exposes each pool counter through a separate MXBean call, so a borrow or return between the reads can leave one snapshot off by one. The test read once and failed in CI with total 2 vs active + idle 3. Read up to 20 times with a 10ms pause until total equals active + idle, then assert on the stable snapshot. The invariant stays strict; only the transient race is tolerated. AI-assisted-by: muse-spark-1.3-contributor --------- Co-authored-by: Elena Torro <elenatorro@gmail.com> |
||
|
|
ba78ccb8c9
|
✨ Harden file data offload job with lock, summary and metrics (#12403)
Take the file row with FOR UPDATE SKIP LOCKED when the offload job starts and skip the run when the row is locked or gone, so the job never runs while the file is being edited. Every live hot-storage row of the file moves to cold storage inside one all-or-nothing transaction, and the handler answers a summary map with the offloaded rows, bytes and per-type stats. Add bounded metrics for the job: rows and bytes counters labeled by row type on successful runs only, and a timing histogram with the ok, skip and error outcomes. The job definition now requires metrics, the recorders live in app.jobs.metrics, and the existing ephemeral job timing histogram pins its buckets. Cover the job with a new test namespace: the happy path, the locked row, missing and deleted files, the rows and bytes summary with its no-op rerun, the whole-file abort on storage failure, the metric definitions and emissions, and the submit-time params validation. An end-to-end suite reads, snapshots, writes, exports, imports and deletes offloaded files; concurrency tests pin the stale-revn skip, the blocked-edit retry and the reclaimable orphans of a partial failure. The file-gc scheduler test pins one job per eligible file, and the scheduler private helper drops its bang. Document the lock invariant, the deliberate no-pacing decision and the metric label sets in the backend storage memory and the configuration guide. Related to #12398. AI-assisted-by: muse-spark-1.3-contributor |
||
|
|
be2e117f67
|
♻️ Rename audit-log archivers to consumers and use base Nexus URI (#12459)
* ♻️ Rename audit-log archivers to consumers The pending-queue owners are no longer only external archives: a local, non-archiving drain is about to join them, so "archiver" no longer fits. Rename the column, identifiers, jobs, metrics and docs to "consumer" in one release; migration 0157 never reached production, so a single RENAME COLUMN is enough, with no backfill. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Use base Nexus URI with appended ingest path Mirror the admin-console setup: PENPOT_NEXUS_URI holds the base URI and the consumer appends api/audit itself, instead of a full endpoint URI. Keeps the explicit :uri job param as override and the :task-not-configured guard when no base is set. AI-assisted-by: muse-spark-1.3-contributor |
||
|
|
ccf99c7dcf
|
♻️ Archive audit log from a pending queue with two jobs (#12373)
* ♻️ Archive audit log from a pending queue with two jobs Replace the mark-and-sweep audit archive with a pending queue (audit_log_pending, migration 0157): every row is born pending for the archivers enabled at insert time, in the same transaction, so a rollback takes both. Two jobs share one namespace, one queue each: :audit-log-archive-nexus and :audit-log-archive-nitrate, each with its own closed params schema, deps and every-5m cron entry gated by its send flag. Each job drains its queue oldest-first in 128-row chunks (one transaction per chunk, same total order, no SKIP LOCKED so arrival stays chronological) and deletes its pendings on completion; whoever deletes the last pending deletes the row. Housekeeping lives in both jobs: reap of disabled pendings plus sweep of orphans, both idempotent. A refused chunk raises :archive-chunk-refused and aborts the job; the cron retries. No separate GC task. Metrics with bounded labels only: archived/failed/skipped per destination, discarded per reason (reaped pendings, swept rows: the silent R3 loss, now dashboard-visible). Timing comes from the jobs substrate; queue depth is read off the replica DB. No gauge, no custom histogram. Destination hosts log once at job-def init; per-run logs carry counts plus frozen elapsed (log props evaluate lazily in the logger thread). Pre-existing rows are swept unsent (no backfill, accepted loss); rows born with no archiver on share that fate (R3). Memory mem:audit-log documents the design. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Address review comments on audit log archive jobs Rename the second archive destination from nitrate to admin-console (jobs, metric and log labels, error code), reusing the :admin-console flag. Unify the pending-queue insert into a single helper used by both producers. Stagger the archive crons (Nexus at second 0, admin console at second 30). Add per-call dbg logs to the admin-console send. Update the audit-log memory and the affected tests. Closes #12372 AI-assisted-by: muse-spark-1.3-contributor |
||
|
|
fffc3ee222 | Merge remote-tracking branch 'origin/staging' into develop | ||
|
|
fdd66978fd | Merge branch 'main' into staging | ||
|
|
688c0fc398
|
🐛 Keep file_change revisions on their own deletion schedule (#12419)
* 🐛 Keep file_change revisions on their own deletion schedule Restoring a file from the trash cleared the deletion mark on every file_change row of that file, revisions included. A revision is born already scheduled for deletion one hour later, and objects-gc only looks at rows that carry a date, so unmarking one left it in the table for ever: production had 358,517 of them across 3,614 files. The file deletion cascade had the mirror problem: it rewrote the date of rows that already had one, pushing a revision out to the trash lifetime. Restrict the restore to durable history and date only the rows that have no date yet, so a revision keeps the schedule it was born with. A pinning test covers the snapshot update path, which cannot reach a revision because it resolves its row through a read filter. The backend memories now carry the model of the three file_data classes and this invariant, in place of the two superseded stubs. Closes #12418 AI-assisted-by: deepseek-v4.1-flash * 📚 Add autovacuum and Postgres exporter notes to Grafana memory Record what this Grafana instance can and cannot show about autovacuum so the next session does not have to re-derive it: - The Postgres exporter scrapes the primary as well as the read-only replica, so the primary's pg_stat_activity and pg_settings_* are in Prometheus even though the SQL datasource cannot reach them. - It publishes no pg_stat_progress_* collector, and pg_stat_activity_count has no backend_type label: autovacuum workers are the usename/application_name-less bucket with state=active. The launcher shows up as the AutoVacuumMain wait event, and it is absent on the replica. - A metric the exporter does not publish needs a custom query block in its YAML queries file and a restart of the exporter. - The Postgres server ships its own log to Loki (the job is postgresql, primary only), whose streams carry only filename, instance, job and level. The completed vacuum lines live there: a per-table history, not a view of what runs now. - The PostgreSQL section now ends with the vacuum panels, with the Vacuum TXID age table last; an in-flight per-table view would need a pg_stat_progress_vacuum collector in the exporter itself. AI-assisted-by: deepseek-v4.1-flash |
||
|
|
dfc55a7192 | Merge remote-tracking branch 'origin/staging' into develop | ||
|
|
57fa280209
|
🐛 Validate and decode a job result with its declared schema (#12350)
* 🐛 Decode job results with the schema of the job-def A job result is stored as plain JSON, so a uuid comes back as the text it is on disk and a uuid used as a map key comes back as a keyword. The import resolution is a map keyed by file id, and the wizard lost that identity: choosing a library in the link step created no link at all, with no error and with the modal closing as if it had worked. A job-def can now declare a result schema and its precompiled decoder, the pair the params already have, and the substrate applies them when the row is read. A def that declares neither answers with the JSON the row holds, as before; a row whose job-def is gone is read untouched instead of raising. Relates to #12339 AI-assisted-by: deepseek-v4.1-flash Signed-off-by: Andrey Antukh <niwi@niwi.nz> * 🐛 Type the import result so its resolution keeps uuid keys The resolution of a version 3 import is a map keyed by the new file id, and the result column is plain JSON: without a schema of its own every key came back as a keyword while the :id of the entry came back as text, so the dashboard wizard could not match the file it was linking and created no library link at all. Declare the shape of the resolution (files, done and pending entries and their candidates) and give the job-def its result schema and decoder. The check before the result is stored keeps validating the same schema. Closes #12339 AI-assisted-by: deepseek-v4.1-flash Signed-off-by: Andrey Antukh <niwi@niwi.nz> * 🐛 Take the imported file id from the entry, not the map key The wizard looked a file up in the selection with the key of the resolution map while it had indexed the selection by the :id of the entry. The key is keywordized on its way through the job result, so the lookup missed and the confirm button created no library link at all, with no error and with the modal closing as if it had worked. Take the id from the entry, the source the rest of the wizard already uses, and move the decision to a pure function covered by tests of both shapes of the resolution. Relates to #12339 AI-assisted-by: deepseek-v4.1-flash Signed-off-by: Andrey Antukh <niwi@niwi.nz> * 🐛 Test that an import result keeps its ids as uuids The regression this branch fixes is only visible through the read path of the substrate: the resolution of an import is a map keyed by the new file id, and a test that works with the value in memory never sees the keys coming back as keywords from the plain JSON column. Import a package whose library needs a replacement chosen by the user, read the result with `jobs/get-job` as the client does, and check the keys, the file ids and the ids of the candidates. Without the result decoder of the import job-def the test answers with text ids. Relates to #12339 AI-assisted-by: deepseek-v4.1-flash Signed-off-by: Andrey Antukh <niwi@niwi.nz> * 📚 Document the result schema of a job-def The memories and the docstring of the substrate said a job result has no schema of its own and comes back as the JSON the row holds. A job-def can now declare its result schema and decoder, and the substrate applies them when the row is read. Keep the rule next to it: a result map keyed by uuid needs its schema declared, or every key comes back as a keyword, and the link-later step of the import wizard takes the file id from the entry and never from the key of the resolution map. Relates to #12339 AI-assisted-by: deepseek-v4.1-flash Signed-off-by: Andrey Antukh <niwi@niwi.nz> * 📚 Fix the documented contract of a job result The docstring of get-job said a result has no schema of its own and comes back as the JSON the row holds, which is what this branch changes: the result is decoded with the schema its job-def declares, and a result that does not conform never reaches the column because the write validates it. Say the same in the management command that stores the result of an external worker, the other writer of the column. Relates to #12339 AI-assisted-by: deepseek-v4.1-flash Signed-off-by: Andrey Antukh <niwi@niwi.nz> * 🐛 Check the result of a job against its schema before storing it The result of a job crosses a plain JSON column, so a uuid comes back as text and a uuid used as a map key as a keyword. The job-def now declares the schema of its result and the substrate compiles the decoder from it, which is what the read path uses to get the types back. The same schema is the contract of the write path: complete refuses a result that does not conform, with the malli explanation, so nothing invalid reaches the column, and the answer is the same for the runner and for the external worker that completes a job through the management RPC. A job that is no longer completable is skipped without checking, and the read path decodes only: a row written by another version stays readable. Relates to #12339 AI-assisted-by: deepseek-v4.1-flash Signed-off-by: Andrey Antukh <niwi@niwi.nz> * 🐛 Check the result of an import inside its transaction The check ran after the transaction of the import had committed, so a result the schema does not accept failed the job with the files already created in the destination project: a half imported project and an error. The core decides whether the whole import is kept or rolled back, so the check belongs inside its transaction, which is what turns a contract violation into a rollback. The substrate checks the result again when the job is completed, which is the only net for a job that does not validate its own result. Relates to #12339 AI-assisted-by: deepseek-v4.1-flash Signed-off-by: Andrey Antukh <niwi@niwi.nz> * 🐛 Cover the edges of an import result read by the client The shape of a result with no libraries to resolve, and the one of a version 1 import, is the most common in production and neither was covered by the tests that read a result the way the client does. Cover both through jobs/get-job, and the case of a registry that no longer knows the kind of the row: it answers with the JSON of the column, keywordized keys and uuid as text, instead of raising. Say in the two tests that depend on the decoder what implementation without it makes them fail. Relates to #12339 AI-assisted-by: deepseek-v4.1-flash Signed-off-by: Andrey Antukh <niwi@niwi.nz> * 📚 Fix the prose and the fixtures of the result contract The result schema of a job-def is what the read decodes with and what the write checks: say that in the memories, with the rule that any uuid of a result needs it, whether a value or a map key. The motive of the wizard helper was a fact this branch makes false: what a key of the resolution map holds depends on the result schema of the job-def that wrote it, so the rule is to read the id from the entry. Same correction in its memory and in the test that pins the shape. The candidates of a pending entry match by slugified name, not by name, and every test row now states the kind of its job, because the name is what resolves its job-def when the result is read or checked. Relates to #12339 AI-assisted-by: deepseek-v4.1-flash Signed-off-by: Andrey Antukh <niwi@niwi.nz> * ♻️ Keep the wizard link helper private and cover its shape The helper that decides which links the wizard creates belongs to its namespace, as its neighbours do: reach it from the test with @#', the pattern the frontend tests already use for a private helper. Cover the shapes it had no test for: a file with two libraries pending resolution, a file whose libraries were auto-linked, and a selection where only one of two entries was resolved. Relates to #12339 AI-assisted-by: deepseek-v4.1-flash Signed-off-by: Andrey Antukh <niwi@niwi.nz> * ♻️ Share one job-row helper across the job tests Three test namespaces had their own copy of the same insert, differing only in the kind of the job and in whether the row holds the resource it consumes. Move it to the test helpers as create-job*, with the kind of the job as its first argument: it is what resolves the job-def of the row when its result is read or checked, so every test states it. The bang of the name goes away with it: the design rules keep it for the mutating API of app.db, and this is a fixture of the tests. The three namespaces lose the requires that only their removed copy used. Relates to #12339 AI-assisted-by: deepseek-v4.1-flash Signed-off-by: Andrey Antukh <niwi@niwi.nz> --------- Signed-off-by: Andrey Antukh <niwi@niwi.nz> |
||
|
|
feb73f0fd8
|
✨ Handle tokens libraries when exporting .penpot files (#12077)
* ✨ Handle tokens libraries when exporting .penpot files Exporting a file that takes its tokens from an external library left the :tokens-source of the file pointing at a library that is not in the export, and the tokens lib of that library was not exported. With :detach-libraries, unset the :tokens-source and the tokens status in detach-external-references. The file is exported without any link to the library, and the shapes keep their applied tokens. With :merge-libraries, add cfo/absorb-tokens-lib, which copies the tokens lib of a library into a file and unsets its :tokens-source, so the file becomes its own tokens source. The tokens status of the file is kept; if it has none, the one of the library is copied. If the library has no tokens lib, only the source is unset. embed-assets calls it for the library that is the tokens source of the file, and ignores the other linked libraries. Add unit tests for absorb-tokens-lib, and binfile tests that export a file with tokens from an external library with both modes, checking the exported files, the tokens lib, the tokens source and the applied tokens. Also reformat the export-type docstring in binfile v3 and remove a leftover debug prn in the export dialog. AI-assisted-by: claude-sonnet-5-5 * 🐛 Preserve active token sets when importing linked libraries Cache each decoded tokens library for the duration of an import so files and their tokens source resolve active sets against the same generated IDs. Add a round-trip regression test for exports that include linked libraries. * 🐛 Preserve tokens when deleting or unpublishing libraries Absorb the tokens library into files that use it as their tokens source. Keep referencing file IDs in the deletion task before removing relations. Embed tokens from deleted, unlinked sources during export, and clear stale source and status data when the library no longer exists. Add regression tests for deletion, unpublishing, and exports with unavailable libraries. --------- Co-authored-by: alonso.torres <alonso.torres@kaleidos.net> |
||
|
|
a53481c3b7
|
✨ Enrich backend audit events with team and entity ids (#12233)
* ✨ Enrich backend audit events with team and entity ids Backend audit events now carry the data a consumer needs to rebuild state without joining production tables: - prepare-rpc-event adds :team-id when the event carries a project or file id; commands that already loaded the team set it themselves. - add-profile-to-team! emits add-team-member on every real insert. - create-project and create-file emit the new row id. - clone-template, import-binfile and duplicate-project emit one event per created entity. - both binfile versions return {:file-ids [...]} from import-files!. Closes #12219 AI-assisted-by: deepseek-v4.1-flash * ✨ Emit delete and hard-delete audit events with entity data The event stream must let Nexus rebuild local state without joining production tables. This completes the enrichment with the delete paths: - delete-team keeps returning nil but now carries the deleted row (id, name, timestamps) in the audit props, like delete-project. - create-team adds created-at to its audit props. - delete-profile carries id, email, fullname and deleted-at; its props used to be empty because every param is namespaced. - objects-gc emits hard-delete-team/project/file per purged batch, with the deleted ids, only when the audit log is on. - delete-team-member needs no change: the delete is physical and the params already carry team-id and member-id. The nitrate push-audit-events tests now drop the membership events emitted by their own profile setup before counting. Closes #12219 AI-assisted-by: muse-spark-1.3-free * ✨ Announce cascade deletes and missing ids in audit events The :delete-object worker task marked teams, projects and files without leaving any event, so an event-sourced projection kept them alive for weeks. It now emits one cascade-deleted event per execution with every affected id, in the same transaction, only with :audit-log on. Rows purged by objects-gc and snapshot cascades stay silent on purpose. Also: create-team-with-invitations carries the new team id, and update-team-member-role names the demoted member on owner transfers. Closes #12219 AI-assisted-by: muse-spark-1.3-free * 🐛 Identify the SREPL profile delete in audit events The SREPL delete-profile! emitted no profile-id and empty props, so the row could not be attributed by any consumer. It now carries the profile id in both fields. delete-team! named the wrong cause in its context. Closes #12219 AI-assisted-by: muse-spark-1.3-free * 🐛 Identify the SREPL profile restore in audit events The SREPL restore-profile! emitted no profile-id, so the consumer keyed on the zero profile instead of the restored one. It now carries the profile id like a normal profile event, and no consumer special case is needed. Closes #12219 AI-assisted-by: muse-spark-1.3-free * 🐛 Stop attributing SREPL events to the subject profile profile-id names the actor, and SREPL calls have none: setting the subject id blamed the user for an admin action and would have leaked those rows into PostHog as user activity. The subject id stays in props, where the consumer resolves it, and the event profile stays zero. Closes #12219 AI-assisted-by: muse-spark-1.3-free * ✨ Expose file analysis data for the nexus pipeline The nexus file-security pipeline needs file content and scheduling state from the backend. This adds a read-only management method, `get-file-for-analysis`, returning the semantic projection of one file (pages, frames, texts, links, interactions; ids as strings; capped with a truncated flag), plus identity, revn, timestamps and share state. No prior projection code existed, so the method ships its own pure `file->projection`. Audit events now carry what the `:local` consumer schedules from: revn and modified-at on update-file, name and timestamps on every create-file emitter, deleted-at on delete-file, and new create/delete-share-link events with file and link ids. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Rename deletion audit events to action names Event names are actions, not past tense: `cascade-deleted` becomes `delete-object-with-cascade` with the same shape and transaction. The three `hard-delete-*` events merge into `hard-delete-objects`: one event per purged chunk carrying only that chunk's entity key (`:team-ids`, `:project-ids` or `:file-ids`), because each GC section purges in its own chunk transactions and a single event could never share them all. Every submit stays inside its delete transaction, `triggered-by` stays annotated, and silent paths stay silent. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Read worker parallelism knobs with plain config keys The runner wiring looked up `:app.main/worker-*-parallelism` (namespaced) while the schema and the environment provide plain keys, so `PENPOT_WORKER_DEFAULT_PARALLELISM` and `PENPOT_WORKER_WEBHOOK_PARALLELISM` never applied and workers always ran with the fallback. Read the keys the config holds. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Fix binfile audit test with a real file row `submit-file-created-event` reads the light columns back from the row, so the mocked importer must return an existing file id: a phantom id emits no event and the test saw zero `create-file` events. Create the file first and use its id. The audit-log memory now tells both bulk-event helpers apart. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Fix audit event fidelity from review findings The review of the audit enrichment found data-fidelity gaps; fix them while the branch is fresh. Membership events now belong to the actor: the flows where the owner adds somebody else pass an explicit actor id, the role is derived from the flags (the table has no role column), and the event carries the request context like its siblings. Also: team-id resolution skips read events, duplicate-project emits the project name and timestamps, delete-profile is idempotent again, the GC tests assert physical purge, and get-file-for-analysis leaves an audit trace. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Nest cascade ids under :cascade in delete event The cascade event carried every affected id flat next to the object identity; nest them under `:cascade` so the payload reads as header plus affected set. The event is an action like its siblings, not a command. Drop the `:audit-log` flag re-check: `submit` already fans out per flag. AI-assisted-by: muse-spark-1.3-contributor * 📎 Update copyright name on file header The company name changed; a single header in this branch still carried the old one. Align it with the current form. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Fix nil event-id on webhook trigger audit rows process-event generated the row id inside append-audit-entry, so the :process-webhook-event task never received it and the trigger row stored a nil :event-id. Assign the id up-front and share it between the stored row and the worker params; the telemetry shadow row keeps its own id to avoid a PK collision. Add regression tests covering the handoff and the end-to-end trigger row. AI-assisted-by: muse-spark-1.3-contributor |
||
|
|
0c0336da42 | Merge remote-tracking branch 'origin/staging' into develop | ||
|
|
eafa483a18 | Merge remote-tracking branch 'origin/main' into staging | ||
|
|
0947e666c0 |
🐛 Keep objects-gc running past a blocked chunk
objects-gc deleted every kind of object inside one transaction per proc, looping until empty, so a single row that a foreign key still references aborted the whole run: the task died on its first file every night, after the earlier steps had already spent around twenty minutes. The per-file savepoint only saved the files proc and left the same shape everywhere else. Now every proc streams its pending keys with db/plan inside a read-only transaction of its own and deletes them chunk by chunk, each chunk in a transaction of its own with FOR UPDATE SKIP LOCKED. A chunk that fails is reported with l/err and skipped, so the run reaches the end and the next run retries the skipped rows. AI-assisted-by: muse-spark-1.3-contributor |
||
|
|
9abcfebf60
|
✨ Run binfile import and export as durable jobs (#12056)
* ✨ Run binfile import and export as durable jobs Import and export of `.penpot` files become durable jobs: two commands create them and answer at once with a job id the caller follows, while a worker of the `binfile` queue does the work. Each job keeps its own ledger (owner, params frozen at creation, result, progress history and an expiry), and the legacy commands and their SSE keep working. Why: both operations ran inside the request. A large import held it open for minutes, progress only existed while the caller kept the stream open, and a client that went away left work nobody recorded, retried or could cancel. The unified `job` substrate already dispatched, leased, retried and logged events, but it had no user-facing ledger. How: - The substrate gains the pieces a user-facing job needs: optional job-def metadata (family, resource-role), an expiry on submit, the owner in the runner context, and job resources in their own bucket, written with the profile of the job and never deduplicated. - One generic command per family: the caller names the job and sends its business params, the registry checks the family, the job-def decodes and validates, and permission and quota are checked first. - Progress taps of the core become milestones of the job: a shared stage vocabulary with self-contained counters, throttled heartbeats, and an interrupt that stops the run inside its own transaction on cancel. - Cancel reaches pending and running jobs from both modals, and closing a modal mid-run cancels the job it was following. - The export re-checks the read permission at run time and answers with the package descriptor; the import re-checks the edition permission, runs the core in a transaction so a cancellation rolls it back, and releases the consumed package and its temporary copy. AI-assisted-by: deepseek-v4.1-flash * ♻️ Handle externally-cancelled binfile jobs end to end The `cancelled` outcome of a job nobody followed no longer hangs its screens: the data layers emit an explicit terminal message for it. - The export emits `{:cancelled true}` per file and the modal paints a neutral cancelled row instead of success; a message without an artifact never reaches the download branch. - The import closes every entry with a dedicated cancelled message through the existing error rendering, so the wizard completes. - Both flows are pinned by data-layer tests, including the red state they replace. Also in this change: the progress-contract comment of the management API, early business-params validation in create-import-job (before assembling the upload), a `release-resource` helper without fabricated job ids, a note on the heartbeat write cost, and a memory cleanup. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Pre-translate the cancelled import message The cancelled entry of the import wizard carried its message as a key for the UI to translate, which needed a rehash marker comment. Like its neighboring errors, it now translates eagerly in the data layer: the literal call keeps rehash clean with no marker, and the tests assert through the same call instead of the raw key. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Move cancelled strings under generic labels keys The dashboard-specific cancelled key never shipped: before use it moves under `labels.*` next to the generic strings both flows already use, with Spanish translations following the established wording. AI-assisted-by: muse-spark-1.3-contributor * ✨ Add minor improvements * ♻️ Rename export job command to create-export-binfile-job Export now has one command per job type, the shape the :export-assets command needs: create-export-binfile-job freezes exactly the :export-binfile job, so :name no longer travels in the body and only :params does. Import keeps the generic envelope. submit-job reads the queue from the job-def metadata ::jobs/queue-name (falling back to :binfile), read by the command and never by the substrate, so a job-def can route its work to another queue without a change in the command. The rename reaches the frontend and the tests: the binfile export is asked for under its own name, and the exporter service keeps create-export-job. AI-assisted-by: deepseek-v4.1-flash * ✨ Polish the import and export status rows The status row of a file already carries what the file is doing, so the generic "Uploading file…"/"Downloading file…" lines are gone: the import wizard drops its own (the export one went with the jobs refactor), along with the now-unused style. In the export, every file is queued up front, even the ones whose job is created only after the previous one is over, and a file stays queued until its first milestone: a worker picking the job up no longer ends the queue. The queued text is a step smaller than the milestone. AI-assisted-by: deepseek-v4.1-flash * 🐛 Show the cancelled export and fix the import error The cancelled export was unreachable: mark-file-cancelled sets :loading false and the label lived inside the loading branch, so a cancelled file only painted a neutral notification. The terminal state now carries the label, and the export flags lose the `?` suffix (predicates keep it; a data prop is not a question). The import wizard rendered (:error entry) through tr, but every producer writes user-facing text (a hint or a translated label), never a key: it renders the text as-is now, and the dynamic-key ignores and the stale comment are gone. The total-text-limit comment no longer references the 200MiB it replaced; the value is 500MiB in develop already. AI-assisted-by: deepseek-v4.1-flash * ♻️ Reflect the real job params in the creation commands The creation commands declared their params as an open empty map, so the RPC contract and the generated docs said nothing about them. Each command now imports the schema from the job namespace (`export-binfile/schema:params`, `import-binfile/schema:create-params`), and the RPC validates the business params before the command runs. The job-def still owns the params and re-validates the assembled ones at run time. The commands no longer decode the params again: the RPC did it. Import joins export as one command per job type: `create-import-job` becomes `create-import-binfile-job`, its body no longer carries `:name` and the job-def is resolved by name like export's. Its creation schema keeps the version optional, because the command reads it from the package header when the caller does not say it, and fills the manifest metadata kept for audit. A version out of range is now refused by the schema (`:params-validation`) instead of the command's own `:unsupported-version` check, which is gone. AI-assisted-by: deepseek-v4.1-flash * 🐛 Stop the test system from running job runners The test system is main/system-config plus main/worker-config with a few components dissoc'ed. The runners of default, webhook and cron were listed by hand; the binfile runner added with the durable binfile jobs was not, so a real penpot/job-runner/binfile/0 thread ran during the whole suite. That runner races the tests that drive the loop by hand (run-batch plus run-worker-loop): it takes the job the test just dispatched and completes it, and the test then reads a state it did not produce. That is the reported flake of binfile-jobs-integration-test, whose middle assertion waits for a runner of another queue to not take the job. It only reproduced in CI: in the devenv PENPOT_TENANT makes the tenant baked into main/worker-config differ from the one the test builds, so the two used different queue keys locally. Drop every [<profile> :app.worker/runner] by key shape instead of listing the known queues, so a new queue cannot bring the race back. AI-assisted-by: deepseek-v4.1-flash * ✨ Upload the package one chunk at a time and as a percentage The upload milestone counts chunks, not units of work, so the row wrote "Uploading the package 3/9" over a file the user had uploaded once: it read as if nine packages were on their way. Write the stage as a percentage of the file instead ("Uploading file… (33%)"), and make the English text of the stage say file, not package. The chunks also go one at a time now (concat-all instead of merge-all with two in flight), so the percentage moves in a straight line instead of jumping. upload-blob-chunked is the generic uploader, so media and font uploads become sequential too. AI-assisted-by: deepseek-v4.1-flash * 🌐 Complete the Spanish catalogue es.po was 25 keys short of en.po: the 15 that the durable binfile jobs introduced (14 jobs.progress.stage.* and labels.queued) plus 10 gaps that predate them (errors.connection-error, errors.save-retrying, the find shortcuts, workspace.header.retrying, workspace.tokens.stroke-width and the mcp session labels). sync -l es then sorts the entries, syncs the #: comments from en and drops the 7 dashboard.import.progress.* keys en no longer has. The msgid sets of en and es are now identical (0 missing, 0 extra). AI-assisted-by: deepseek-v4.1-flash * ♻️ Move the job status keys into the jobs namespace The three status words the durable import and export rows render were filed under labels.*, next to the generic labels bucket, while their progress stages live under jobs.progress.stage.*: the strings of one feature sat in two places. Rename them so the whole job vocabulary is one namespace: labels.queued -> jobs.queued labels.export-cancelled -> jobs.export-cancelled labels.import-cancelled -> jobs.import-cancelled No other locale has these keys yet, so the rename costs no translation. The convention (a key names the feature that renders it; labels.* holds the generic reusable words) is written down in the frontend/translations memory. AI-assisted-by: deepseek-v4.1-flash * 🔥 Remove the translation keys nothing references Six keys of en.po (and of es.po, which now mirrors it) have no (tr ...) call site left in frontend/src or common/src, and no literal reference either: labels.uploading-file a duplicate of jobs.progress.stage.upload since the upload stage says "Uploading file…" labels.downloading-file dead since the durable jobs replaced the old import and export progress rows inspect.attributes.stroke.alignment.center inspect.attributes.stroke.alignment.inner inspect.attributes.stroke.alignment.outer workspace.toolbar.mcp-connect-here The first two are leftovers of the durable binfile jobs, the other four predate them. Both catalogues keep an identical msgid set afterwards. AI-assisted-by: deepseek-v4.1-flash |
||
|
|
d3ca9a0b83
|
✨ Add unified jobs substrate replacing legacy one-shot task system (#11542)
* 🎉 Move backend jobs to the unified job table substrate Replace the generic 'task' table with a unified 'job' table as the durable substrate for all backend jobs, and migrate every producer and consumer to it. The legacy 'task' table stops receiving writes but stays dormant (still cleaned by tasks-gc); upload-session chunking and the other post-branch upstream changes are preserved. New 0154 migration creates the 'job' table: dispatch and lifecycle columns (name, queue, priority, scheduled_at, retries, status, timestamps, props) plus optional user-facing ledger columns (profile_id, target, progress, error, result, resource_id, expires_at). No modified_at trigger: the application updates it on every write for lease and orphan detection. Partial indexes cover the dispatcher claim query, orphan scanning, and per-profile listing. New app.jobs public API: submit!/invoke!, heartbeats and progress reporting, a precompiled job-def registry (name, schema, handler, decoder, validator), an ephemeral request/response mode, and a management API for external workers. The dispatcher claims new/retry rows (FOR UPDATE SKIP LOCKED) with a plain JSON redis payload and marks lease-expired running rows as orphaned; the runner executes per-queue with the same retry conventions. New jobs GC (daily): deletes expired rows and retains internal terminal rows per :jobs-retention, touching referenced storage objects so storage-gc-touched reclaims them. Job resources live in a dedicated job-resource bucket with a reclaim branch in storage-gc-touched. All production call-sites (sendmail, delete-object, demo-purge, file-gc, offload, webhooks, quotes, nitrate bulk delete, snapshots, telemetry) and the ten cron tasks now submit through app.jobs; cron is a pure scheduler and the legacy worker registry, task-table inserts, and per-namespace handlers are removed. Includes migrations, dispatcher, runner, jobs, request/response, jobs GC, management API, and cron test suites. AI-assisted-by: glm-5.3-flash AI-assisted-by: muse-spark-1.3-contributor AI-assisted-by: kimi-k3 AI-assisted-by: mimo-v2.5 * ✨ Address review comments on jobs substrate Resolves the four open review threads on the jobs PR. Orphaned jobs are now marked aborted, a system-side terminal state with no retry: the dispatcher sweep stores the end event with outcome aborted in a single bulk insert in the same transaction, records the terminal outcome next to the orphaned counter, and reports an error log; orphans keep alerting through the log and the counter, with no user notification. The aborted state is terminal everywhere: the status CHECK, the sweep, the lifecycle writers, the jobs GC retention, the backlog gauges and the user status mapping, with tests covering each transition. The claim comment, the 2.20 doc version and the legacy task table notes are corrected, and objects-gc heartbeats per chunk. AI-assisted-by: muse-spark-1.3-contributor |
||
|
|
aeeca59b1a |
🐛 Fix shared library deletion with storage-backed file data
Inject the storage component into the delete-object task handler and validate it on initialization, so library absorption no longer fails with "expected valid storage to be provided" for storage-backed file data. Also strengthen the regression test: it now creates a real component in the shared library and an instance in the referencing file, and verifies the component is absorbed after deletion instead of only checking the revision bump. The test disables the fdata/pointer-map feature to stay focused on the storage fix. Signed-off-by: Yuito Akatsuki (Tani Yutaka) <yuito@yuito-it.jp> AI-assisted-by: deepseek-v4.1-flash |
||
|
|
ad7e035b63
|
✨ Add indirection for upload-chunks storage via new table (#11651)
* ✨ Add upload_session_chunk indirection for chunked uploads Chunks now live in the upload_session_chunk table with non-deleting foreign keys to storage_object and upload_session, instead of tempfile objects with session metadata. Reads go through a JOIN, so chunk state never scans storage_object. Uploads validate the live session, reject duplicate indexes, and store objects in the new upload-session bucket without extra metadata. Assemble removes mappings and marks the session consumed; objects-gc procedurally purges consumed and stalled sessions, touching referenced objects first. Touched-gc and deleted-gc handle the new bucket, and upload-session-gc is removed. Closes #11644 AI-assisted-by: muse-spark-1.3-contributor * 🐛 Fix quota, give-up and coverage for session chunks Exclude consumed sessions from the sessions-per-profile quota so finished uploads free their slot at once. Remove chunk mappings before the gc-deleted give-up delete to respect the NO ACTION keys. Catch java.sql.SQLException for duplicate chunks. Cover the profile-owned session purge and the UNIQUE race backstop with tests. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Align chunked upload tests with upload_session_chunk Drop the duplicate-index tests written against metadata-backed chunks; the UNIQUE mapping makes those cases unrepresentable and the new tests cover them. Rewrite the rejected-duplicate tests to expect :validation/:chunk-already-exists and assert against the upload_session_chunk table, and scope the chunk-too-large "nothing stored" check to the mapping table. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Use NO ACTION DEFERRABLE session FKs in single migration Fold the profile FK change into 0154 so the feature ships one migration. All three upload session FKs use ON DELETE NO ACTION DEFERRABLE: identical to RESTRICT in normal operation, but deferrable for tooling that relies on SET CONSTRAINTS ALL DEFERRED. Extend the RESTRICT test to the direct profile delete. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Reserve chunk slot before writing blob in upload-chunk Make object_id nullable and insert the mapping with NULL inside the session-locking transaction, then write the blob outside it and link it with a conditional update. A failed write removes the mapping and reraises; a mid-flight death leaves a NULL row and the client starts a new session. AI-assisted-by: muse-spark-1.3-contributor * 🔥 Remove redundant session_id index on upload_session_chunk The UNIQUE(session_id, chunk_index) btree already serves session_id-only lookups and the session FK check through its leftmost column, so the standalone index only taxed the per-chunk INSERT path. Verified with EXPLAIN on an equivalent table shape. AI-assisted-by: muse-spark-1.3-contributor * ⚡ Merge chunk touch and delete into single RETURNING query Replace the SELECT-then-DELETE round-trip in delete-upload-sessions! with DELETE ... RETURNING object_id, touching each returned object. Same semantics, one less query per purged session. Follows the RETURNING pattern already used in file-gc. AI-assisted-by: muse-spark-1.3-contributor * ♻️ Let objects-gc own chunk mapping deletion Assemble-chunks now only marks the session as consumed; the chunk mappings stay until objects-gc purges them (touching the chunk objects first), leaving a single procedural deletion path for consumed, stalled and profile-purge sessions. AI-assisted-by: muse-spark-1.3-contributor * 🐛 Fix font-deletion GC expectations for chunk objects Update final storage-gc-touched counts to include the two chunk objects touched by objects-gc when purging consumed upload sessions (8/5/5 instead of 6/3/3). AI-assisted-by: muse-spark-1.3-contributor * 🐛 Release chunk reservation when the link UPDATE fails Review feedback on #11651: the link UPDATE in upload-chunk could leave a NULL reservation behind, blocking retries of the same index with :chunk-already-exists. Remove the reservation when the link fails so the client can retry in the same session; the orphaned blob stays touched for touched-gc. Also realign the process-bucket! cond branches in gc-touched. Tests: chunked-upload-link-failure-releases-slot and chunked-upload-null-reservation-blocks-retry. AI-assisted-by: muse-spark-1.3-contributor |
||
|
|
ba608f8c77 | Merge remote-tracking branch 'origin/staging' into develop | ||
|
|
e5f375edbc |
✨ Disable newsletter telemetry fallback on official hosts
Skip the limited newsletter report when the public-uri host belongs to penpot.dev or penpot.app, so the SaaS never sends subscriber emails to its own telemetry endpoint. Defer the subscriptions query with delay so it only runs when a report is actually going to be sent. AI-assisted-by: muse-spark-1.3-contributor |
||
|
|
66fb4a69ba | 📎 Update copyright headers | ||
|
|
c1bd3cb9f0 | Merge remote-tracking branch 'origin/staging' into develop | ||
|
|
e96a75d366 |
✨ Send telemetry event batch as plain JSON vector
The event batch sent to the telemetry server was encoded as a fressian+zstd base64 blob. Send it as a plain vector of event maps instead: the JSON encoder handles UUID and temporal types natively, the payload becomes inspectable, and the receiver schema coerces values back to proper types. The receiver (penpot-telemetry) now accepts both the blob and the plain vector, so it must be deployed before this backend change. AI-assisted-by: omen-alpha |
||
|
|
f5aad7b1ae | Merge remote-tracking branch 'origin/staging' into develop | ||
|
|
810f8ef671
|
🐛 Cascade profile deletion in objects-gc task (#11395)
The objects-gc task was performing a hard delete on profiles without cascading the soft-delete to owned teams, projects, and files. This left orphaned objects that were never cleaned up. Now the task invokes delete-object before the hard delete, ensuring all owned resources are properly marked for deletion and cleaned up in subsequent GC iterations. AI-assisted-by: qwen3.7-plus |
||
|
|
3df039abc3
|
✨ Add several improvements for demo profile creation mechanism (#11257)
* ⚡ Optimize demo user setup for performance tests Use UUID-based demo emails to prevent concurrent profile collisions.\nUse fast PBKDF2 hashing for demo profiles while keeping regular user hashing unchanged.\nAdd focused coverage for hashing, email uniqueness, and the feature flag.\n\nAI-assisted-by: gpt-5.6-luna * 🐛 Harden font upload test setup Report upload-session errors before chunk validation. Skip chunk uploads when the session ID is invalid. Remove unnecessary Mockery state from the foreign-font test. AI-assisted-by: gpt-5.6-luna * ✨ Add demo profile purge task Schedule delayed deletion for demo profiles through the worker system. Restore normal profile filtering and cover the purge handler with tests. AI-assisted-by: gpt-5.6-luna |
||
|
|
0e388442a1
|
✨ Add storage object status lifecycle and verified dedup (#11345)
* ♻️ Simplify storage GC delays and add skip-delay task params The touched GC no longer applies an extra deletion-delay when marking storage objects as deleted. By the time a storage object is touched, its referencing domain row has already passed its own deletion delay, and the reference scan is the only safety check needed. Touched objects are now marked with deleted_at = now, so the deleted GC removes them on the next run. For the tempfile bucket, upload chunks now set touched-at in the future (1h, aligned with the upload-session-gc TTL) instead of relying on a special-case deletion delay. Task handlers now read their task props: - storage-gc-touched accepts :skip-delay to process all touched objects immediately, bypassing the min-age threshold. - objects-gc accepts :chunk-size and :skip-delay to process recently deleted rows without waiting for the deletion delay. This allows running the deletion cascade immediately from the REPL via run-task! with the skip-delay option. AI-assisted-by: deepseek-v4-flash * ✨ Add storage object status lifecycle, verified dedup, and deletion retry tracking Storage object lifecycle hardening: - Add status column ('valid' | 'pending') as write-ahead marker for object creation. put-object! inserts in 'pending' state, writes blob, then promotes to 'valid'. Failed writes remove the pending row. - Add :storage-pending-gc task to reclaim orphaned pending rows (e.g. after crash between blob write and promotion). - Verify blob existence on every dedup hit via exists-object? (fs stat / s3 headObject). Missing blobs mark the row as deleted and create fresh object. - Add deletion_attempts column (migration 0154) to track physical blob deletion attempts. Restructure gc_deleted to use chunked processing with per-chunk transactions (short lock duration). Failed deletions are deferred to tomorrow (deleted_at = NOW() + 1 day) to prevent infinite loops. After 7 attempts, give up and accept orphan. - Change del-objects-in-bulk contract to return #{fail-ids} for precise per-id tracking (fs and s3 backends updated). - Use tmp/tempfile for fs atomic writes with cleanup queue registration (crashed-JVM temp files swept ~60min later). Document ATOMIC_MOVE POSIX-only assumption. - Add linear backoff to s3 exists-object? retries (100ms/200ms/300ms). - Wrap compensating delete in put-object! catch block to prevent masking original error when connection is aborted. - Fix assert messages in pending_gc.clj and gc_deleted.clj (pool assertion said 'expected valid storage' instead of 'db pool'). - Add pending-objects-excluded-from-gc-deleted test. Use unique path in put-object-write-failure-leaves-no-row test to avoid collisions. AI-assisted-by: qwen3.7-plus * 🐛 Fix review comments on gc-deleted and storage - Fix process-chunk! returning nil causing (+ acc nil) crash - Add FOR UPDATE SKIP LOCKED to sql:get-deleted-chunk to prevent infinite loop when another worker holds locks - Pass :cause to log messages in gc_deleted.clj and s3.clj - Fix extra space in log hint string - Remove unused ::blob-missing? reference from storage memory - Rename test to match actual behavior (leaves pending row) - Add test for gc-deleted giving up after max attempts AI-assisted-by: qwen3.7-plus |
||
|
|
81c3b3cd56
|
📎 Update copyright name on file header (#11346) | ||
|
|
ddba2ffa75
|
📎 Update Kaleidos Copyright (#9929) | ||
|
|
7d4be33d4f |
🎉 Add telemetry anonymous event collection (#9483)
* 🎉 Add telemetry anonymous event collection Rewrite the audit logging subsystem to support three operating modes and add anonymous telemetry event collection: Modes: - A (audit-log only): events persisted with full context - B (audit-log + telemetry): same as A, plus events are collected for telemetry shipping - C (telemetry-only): events stored anonymously with PII stripped, telemetry flag active, audit-log flag inactive Audit system refactoring (app.loggers.audit): - Replace qualified map keys (::audit/name etc.) with plain keywords - Rename submit! -> submit, insert! -> insert, prepare-event -> prepare-rpc-event - Add submit* as a lower-level public API - Add process-event dispatch function that handles all three modes and webhooks in a single tx-run! - Add :id to event schema (auto-generated if omitted) - Add filter-telemetry-props: anonymises event props per event type. Keeps UUID/boolean/number values; for login/identify events preserves lang, auth-backend, email-domain; for navigate events preserves route, file-id, team-id, page-id; instance-start trigger passes through. - Add filter-telemetry-context: retains only safe context keys. Backend: version, initiator, client-version, client-user-agent. Frontend: browser, os, locale, screen metrics, event-origin. - Timestamps truncated to day precision via ct/truncate for telemetry storage - PII stripped: props emptied, ip-addr zeroed, session-linking and access-token fields removed from context Config (app.config): - Derive :enable-telemetry flag from telemetry-enabled config option Email utilities (app.email): - Add email/clean and email/get-domain helper functions for domain extraction from email addresses Setup (app.setup): - Emit instance-start trigger event at system startup - Simplify handle-instance-id (remove read-only check) RPC layer (app.rpc): - wrap-audit now activates when :telemetry flag is set - Add :request-id to RPC params context for event correlation RPC commands (management, teams_invitations, verify_token, OIDC auth, webhooks): migrate all audit call sites to use the new plain-key API SREPL (app.srepl.main): - Migrate all audit/insert! calls to audit/insert with plain keys Telemetry task (app.tasks.telemetry): - Restructure legacy report into make-legacy-request; distinguish payload type as :telemetry-legacy-report - Add collect-and-send-audit-events: loop fetching up to 10,000 rows per iteration, encodes and sends each page, deletes on success, stops immediately on failure for retry - Add send-event-batch: POSTs fressian+zstd batch (base64 via blob/encode-str) to the telemetry endpoint with instance-id per event - Add gc-telemetry-events: enforces 100,000-row safety cap by dropping oldest rows first - Add delete-sent-events: deletes successfully shipped rows by id Blob utilities (app.util.blob): - Add encode-str/decode-str: combine fressian+zstd encoding with URL- safe base64 for JSON-safe string transport Database: - Add migration 0145: index on audit_log (source, created_at ASC) for efficient telemetry batch collection queries Frontend: - Always initialize event system regardless of :audit-log flag - Defer auth events (signin identify) to after profile is set - Refactor event subsystem for telemetry support Tests (21 test vars, 94 assertions in tasks-telemetry-test): - Cover all code paths: disabled/enabled telemetry, no-events no-op, happy-path batch send and delete, failure retention, payload anonymity, context stripping, timestamp day precision, batch encoding round-trip, multi-page iteration, GC cap enforcement, partial failure handling - blob encode-str/decode-str round-trip tests (14 test vars) - RPC audit integration tests (5 test vars) Signed-off-by: Andrey Antukh <niwi@niwi.nz> * 📎 Add pr feedback changes --------- Signed-off-by: Andrey Antukh <niwi@niwi.nz> |
||
|
|
279231240d
|
🐛 Harden outbound HTTP requests against SSRF and restrict assets handlers (#9390)
* ⬆️ Update root deps * 🐛 Harden outbound HTTP requests against SSRF and restrict unauthenticated asset access - Add app.util.ssrf URL/host validator that resolves hostnames and blocks loopback, link-local, site-local, cloud metadata, and operator-supplied CIDRs - Add app.media.sanitize image EOF truncator that strips trailing data after PNG IEND, JPEG EOI, GIF trailer, and WebP RIFF markers - Disable HTTP client auto-redirect; add req-with-redirects! helper that revalidates every redirect hop against the SSRF blocklist - Wire SSRF validation and EOF sanitization into media/download-image - Validate webhook URLs and OIDC profile picture URLs against SSRF - Restrict /assets/by-id to require authentication for non-public buckets (profile) while keeping public access for file-media-object, file-object-thumbnail, team-font-variant, and file-data-fragment - Add config knobs: ssrf-protection-enabled, ssrf-allowed-hosts, ssrf-extra-blocked-cidrs Signed-off-by: Andrey Antukh <niwi@niwi.nz> --------- Signed-off-by: Andrey Antukh <niwi@niwi.nz> |
||
|
|
e5f9c1e863 |
🎉 Add chunked upload API for large media and binary files
Introduce a purpose-agnostic three-step session-based upload API that
allows uploading large binary blobs (media files and .penpot imports)
without hitting multipart size limits.
Backend:
- Migration 0147: new `upload_session` table (profile_id, total_chunks,
created_at) with indexes on profile_id and created_at.
- Three new RPC commands in media.clj:
* `create-upload-session` – allocates a session row; enforces
`upload-sessions-per-profile` and `upload-chunks-per-session`
quota limits (configurable in config.clj, defaults 5 / 20).
* `upload-chunk` – stores each slice as a storage object;
validates chunk index bounds and profile ownership.
* `assemble-file-media-object` – reassembles chunks via the shared
`assemble-chunks!` helper and creates the final media object.
- `assemble-chunks!` is a public helper in media.clj shared by both
`assemble-file-media-object` and `import-binfile`.
- `import-binfile` (binfile.clj): accepts an optional `upload-id` param;
when provided, materialises the temp file from chunks instead of
expecting an inline multipart body, removing the 200 MiB body limit
on .penpot imports. Schema updated with an `:and` validator requiring
either `:file` or `:upload-id`.
- quotes.clj: new `upload-sessions-per-profile` quota check.
- Background GC task (`tasks/upload_session_gc.clj`): deletes stalled
(never-completed) sessions older than 1 hour; scheduled daily at
midnight via the cron system in main.clj.
- backend/AGENTS.md: document the background-task wiring pattern.
Frontend:
- New `app.main.data.uploads` namespace: generic `upload-blob-chunked`
helper drives steps 1–2 (create session + upload all chunks with a
concurrency cap of 2) and emits `{:session-id uuid}` for callers.
- `config.cljs`: expose `upload-chunk-size` (default 25 MiB, overridable
via `penpotUploadChunkSize` global).
- `workspace/media.cljs`: blobs ≥ chunk-size go through the chunked path
(`upload-blob-chunked` → `assemble-file-media-object`); smaller blobs
use the existing direct `upload-file-media-object` path.
`handle-media-error` simplified; `on-error` callback removed.
- `worker/import.cljs`: new `import-blob-via-upload` helper replaces the
inline multipart approach for both binfile-v1 and binfile-v3 imports.
- `repo.cljs`: `:upload-chunk` derived as a `::multipart-upload`;
`form-data?` removed from `import-binfile` (JSON params only).
Tests:
- Backend (rpc_media_test.clj): happy path, idempotency, permission
isolation, invalid media type, missing chunks, session-not-found,
chunk-index out-of-range, and quota-limit scenarios.
- Frontend (uploads_test.cljs): session creation and chunk-count
correctness for `upload-blob-chunked`.
- Frontend (workspace_media_test.cljs): direct-upload path for small
blobs, chunked path for large blobs, and chunk-count correctness for
`process-blobs`.
- `helpers/http.cljs`: shared fetch-mock helpers (`install-fetch-mock!`,
`make-json-response`, `make-transit-response`, `url->cmd`).
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
|
||
|
|
81b1b253f1
|
✨ Add unique email domains to telemetry report (#8819)
Extend the telemetry payload with a sorted list of unique email domains extracted from all registered profile email addresses. The new :email-domains field is populated via a single SQL query using split_part and DISTINCT, and is included in the stats sent when telemetry is enabled. Also update the tasks-telemetry-test to assert the new field is present and contains the expected domain values. |
||
|
|
7b5817f407
|
♻️ Make several adjustments to the dashboard deleted page (#7999)
* ♻️ Make several sustantial adjustments to the dashboard deleted page * 📎 Add PR feedback changes |
||
|
|
94719eebf8 |
♻️ Make storage and other objects deletion task vclock aware
This simplifes the mental model on how it works and simplifies testing of the related code. This also normalizes storage object deletion in the same way as the rest of objects in penpot (now future deletion date on storage object also means storage object to be deleted). |
||
|
|
7d5c1c9b5f |
✨ Make file-gc-scheduler task compatible with virtual clock
And simplify implementation |
||
|
|
5717708b56 |
♻️ Refactor file storage
Make it more scallable and make it easily extensible |
||
|
|
17ec360720 | ⚡ Add several missing type hints for avoid reflection and autoboxing | ||
|
|
283eb0419c |
♻️ Refactor time related namespaces
Mainly removes the custom app.util.time namespace from frontend and backend and normalize all to use the app.common.time namespace |
||
|
|
a706907b26 |
♻️ Normalize logical deletion to future dates
Instead of managing the ... |
||
|
|
694a2084e2 | ✨ Add file cleaner to file-gc process | ||
|
|
1187d64f69 | Merge remote-tracking branch 'origin/staging' into develop | ||
|
|
3074fc9ab5
|
♻️ Remove deprecated with-atomic and refactor tx-run! (#5915)
* ♻️ Remove deprecated with-atomic and refactor tx-run! * ✨ Do not hold open connection for the whole clone-template operation |
||
|
|
d2c800fc0f | 🐛 Add missing handling of file_change on delete-object task | ||
|
|
624750ad16
|
🐛 Fix incorrect order of execution of internal procs on objects-gc (#5929)
* ⚡ Improve efficiency on objects-gc tasks Replacing db/cursor with db/plan * 🐛 Use correct order of subprocs on objects-gc Mainly affects the file deletion * 📎 Increase chunk-size to 100 on objects-gc |
||
|
|
15f7fac8f4 |
✨ Add more reasonable default fetch size for db/plan
Making it more general purpose and not only for streaming |
||
|
|
16934b734b | 🎉 Add srepl fix function for process media refs on a file |