Introduce a purpose-agnostic three-step session-based upload API that
allows uploading large binary blobs (media files and .penpot imports)
without hitting multipart size limits.
Backend:
- Migration 0147: new `upload_session` table (profile_id, total_chunks,
created_at) with indexes on profile_id and created_at.
- Three new RPC commands in media.clj:
* `create-upload-session` – allocates a session row; enforces
`upload-sessions-per-profile` and `upload-chunks-per-session`
quota limits (configurable in config.clj, defaults 5 / 20).
* `upload-chunk` – stores each slice as a storage object;
validates chunk index bounds and profile ownership.
* `assemble-file-media-object` – reassembles chunks via the shared
`assemble-chunks!` helper and creates the final media object.
- `assemble-chunks!` is a public helper in media.clj shared by both
`assemble-file-media-object` and `import-binfile`.
- `import-binfile` (binfile.clj): accepts an optional `upload-id` param;
when provided, materialises the temp file from chunks instead of
expecting an inline multipart body, removing the 200 MiB body limit
on .penpot imports. Schema updated with an `:and` validator requiring
either `:file` or `:upload-id`.
- quotes.clj: new `upload-sessions-per-profile` quota check.
- Background GC task (`tasks/upload_session_gc.clj`): deletes stalled
(never-completed) sessions older than 1 hour; scheduled daily at
midnight via the cron system in main.clj.
- backend/AGENTS.md: document the background-task wiring pattern.
Frontend:
- New `app.main.data.uploads` namespace: generic `upload-blob-chunked`
helper drives steps 1–2 (create session + upload all chunks with a
concurrency cap of 2) and emits `{:session-id uuid}` for callers.
- `config.cljs`: expose `upload-chunk-size` (default 25 MiB, overridable
via `penpotUploadChunkSize` global).
- `workspace/media.cljs`: blobs ≥ chunk-size go through the chunked path
(`upload-blob-chunked` → `assemble-file-media-object`); smaller blobs
use the existing direct `upload-file-media-object` path.
`handle-media-error` simplified; `on-error` callback removed.
- `worker/import.cljs`: new `import-blob-via-upload` helper replaces the
inline multipart approach for both binfile-v1 and binfile-v3 imports.
- `repo.cljs`: `:upload-chunk` derived as a `::multipart-upload`;
`form-data?` removed from `import-binfile` (JSON params only).
Tests:
- Backend (rpc_media_test.clj): happy path, idempotency, permission
isolation, invalid media type, missing chunks, session-not-found,
chunk-index out-of-range, and quota-limit scenarios.
- Frontend (uploads_test.cljs): session creation and chunk-count
correctness for `upload-blob-chunked`.
- Frontend (workspace_media_test.cljs): direct-upload path for small
blobs, chunked path for large blobs, and chunk-count correctness for
`process-blobs`.
- `helpers/http.cljs`: shared fetch-mock helpers (`install-fetch-mock!`,
`make-json-response`, `make-transit-response`, `url->cmd`).
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
Skip storage objects touched less than 2 hours ago, matching the pattern
used by upload-session-gc. Update all affected tests to advance the clock
past the threshold using ct/*clock* bindings.
* ✨ Add newsletter opt-in checkbox to registration validation form
Add accept-newsletter-updates support through the full registration
token flow. The newsletter checkbox is now available on the
registration validation form, allowing users to opt-in during the
email verification step.
Backend changes:
- Refactor prepare-register to consolidate UTM params and newsletter
preference into props at token creation time
- Add accept-newsletter-updates to prepare-register-profile and
register-profile schemas
- Handle newsletter-updates in register-profile by updating token
claims props on second step
Frontend changes:
- Add newsletter-options component to register-validate-form
- Add accept-newsletter-updates to validation schema
- Fix subscription finalize/error handling in register form
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* ♻️ Refactor auth register components to modern style
Migrate all components in app.main.ui.auth.register and
app.main.ui.auth.login/demo-warning to use the modern * suffix
convention, removing deprecated ::mf/props :obj metadata and
updating all invocations from [:& name] to [:> name*] syntax.
Components updated:
- terms-and-privacy -> terms-and-privacy*
- register-form -> register-form*
- register-methods -> register-methods*
- register-page -> register-page*
- register-success-page -> register-success-page*
- terms-register -> terms-register*
- register-validate-form -> register-validate-form*
- register-validate-page -> register-validate-page*
- demo-warning -> demo-warning*
Also remove unused old context-notification import in login.cljs.
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 🔥 Remove unused onboarding-newsletter component
The newsletter opt-in is now handled directly in the registration
form via the newsletter-options* component, making the standalone
onboarding-newsletter modal obsolete.
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* 🐛 Fix register test for UTM params to use prepare-register step
UTM params are now extracted and stored in token props during the
prepare-register step, not at register-profile time. Move utm_campaign
and mtm_campaign from the register-profile call to the
prepare-register-profile call in the test.
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
---------
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
Extend the telemetry payload with a sorted list of unique email domains
extracted from all registered profile email addresses. The new
:email-domains field is populated via a single SQL query using
split_part and DISTINCT, and is included in the stats sent when
telemetry is enabled.
Also update the tasks-telemetry-test to assert the new field is present
and contains the expected domain values.
The download-image function in app.media silently succeeded when the
remote image URL was unreachable or returned an error status code,
causing create-file-media-object-from-url to report success with no
actual image stored.
Add exception handling for connection refused, timeouts, and I/O errors
around the HTTP request, and validate the HTTP status code in
parse-and-validate before processing the response body.
Fixes#8499
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
The download-image function in app.media silently succeeded when the
remote image URL was unreachable or returned an error status code,
causing create-file-media-object-from-url to report success with no
actual image stored.
Add exception handling for connection refused, timeouts, and I/O errors
around the HTTP request, and validate the HTTP status code in
parse-and-validate before processing the response body.
Fixes#8499
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
* ✨ Add core changes for mcp server
* ✨ Changes to plugins-runtime to add mcp extensions
* ✨ Changes to MCP plugin
* ✨ Changes post-review and ci fixes
This simplifes the mental model on how it works and simplifies testing
of the related code.
This also normalizes storage object deletion in the same way as the
rest of objects in penpot (now future deletion date on storage object
also means storage object to be deleted).
The main idea behind this refactor is make the
API less especialized for specific use of out internal
submidules and make it more general and usable
for more general purposes (per example cache)
Mainly make it receive the whol cfg/system instead only props. This
makes the api more flexible for a future extending without the need
to change the api again.
Replace general usage of virtual threads with platform threads
and use virtual threads for lightweight procs such that websocket
connections. This decision is made mainly because virtual threads
does not appear on thread dumps in an easy way so debugging issues
becomes very difficult.
The threads requirement of penpot for serving http requests
is not very big so having so this decision does not really affects
the resource usage.
