* 🎉 Move backend jobs to the unified job table substrate
Replace the generic 'task' table with a unified 'job' table as the
durable substrate for all backend jobs, and migrate every producer
and consumer to it. The legacy 'task' table stops receiving writes
but stays dormant (still cleaned by tasks-gc); upload-session
chunking and the other post-branch upstream changes are preserved.
New 0154 migration creates the 'job' table: dispatch and lifecycle
columns (name, queue, priority, scheduled_at, retries, status,
timestamps, props) plus optional user-facing ledger columns
(profile_id, target, progress, error, result, resource_id,
expires_at). No modified_at trigger: the application updates it on
every write for lease and orphan detection. Partial indexes cover
the dispatcher claim query, orphan scanning, and per-profile
listing.
New app.jobs public API: submit!/invoke!, heartbeats and progress
reporting, a precompiled job-def registry (name, schema, handler,
decoder, validator), an ephemeral request/response mode, and a
management API for external workers. The dispatcher claims
new/retry rows (FOR UPDATE SKIP LOCKED) with a plain JSON redis
payload and marks lease-expired running rows as orphaned; the
runner executes per-queue with the same retry conventions.
New jobs GC (daily): deletes expired rows and retains internal
terminal rows per :jobs-retention, touching referenced storage
objects so storage-gc-touched reclaims them. Job resources live in
a dedicated job-resource bucket with a reclaim branch in
storage-gc-touched.
All production call-sites (sendmail, delete-object, demo-purge,
file-gc, offload, webhooks, quotes, nitrate bulk delete,
snapshots, telemetry) and the ten cron tasks now submit through
app.jobs; cron is a pure scheduler and the legacy worker registry,
task-table inserts, and per-namespace handlers are removed.
Includes migrations, dispatcher, runner, jobs, request/response,
jobs GC, management API, and cron test suites.
AI-assisted-by: glm-5.3-flash
AI-assisted-by: muse-spark-1.3-contributor
AI-assisted-by: kimi-k3
AI-assisted-by: mimo-v2.5
* ✨ Address review comments on jobs substrate
Resolves the four open review threads on the jobs PR. Orphaned jobs
are now marked aborted, a system-side terminal state with no retry:
the dispatcher sweep stores the end event with outcome aborted in a
single bulk insert in the same transaction, records the terminal
outcome next to the orphaned counter, and reports an error log;
orphans keep alerting through the log and the counter, with no
user notification.
The aborted state is terminal everywhere: the status CHECK, the
sweep, the lifecycle writers, the jobs GC retention, the backlog
gauges and the user status mapping, with tests covering each
transition. The claim comment, the 2.20 doc version and the legacy
task table notes are corrected, and objects-gc heartbeats per chunk.
AI-assisted-by: muse-spark-1.3-contributor
Delete the team-scoped file, project and pin relations on leave-team
and delete-team-member, require a live team membership in the file,
project, search and listing permission queries, and clean
pre-existing orphan rows with a migration. Former members now get
not-found on file actions instead of keeping read and rename access
(GHSA-v9r9-h77c-55m2).
AI-assisted-by: muse-spark-1.3-contributor
The event was written twice per accepted organization invitation. The
backend submitted it, and the browser then re-submitted a copy of the
props that the backend had already put in the response under
`:organization-invitation-audit` (`handle-token :team-invitation` in
`verify-token.cljs`). Both rows carried the same name with different prop
vocabularies, and the browser copy only existed when the browser finished
the flow.
Emit the event from the backend only. It now also carries the three props
that lived in the browser copy: the organization member count before the
add, the add source, and whether the invitee also joined a team. The
origin moves to the event context as `:event-origin`. The response no
longer includes `:organization-invitation-audit`, so the browser stops
emitting the event and `verify-token.cljs` drops its
`app.main.data.event` require.
The `accept-*` events of this command now share one prop vocabulary:
`:profile-id` for the accepting profile, `:invited-by` for the inviter
and `:profile-email` for the email, replacing the mix of
`:user-id`/`:user-who-send-invitation` and `:email`.
Audit consumers of `accept-organization-invitation` now see one row per
acceptance instead of two, and must read the new prop names.
AI-assisted-by: space-bunny-free
* ♻️ Build organization invitation audit event in frontend
* ♻️ Align invitation token profile-id with created-by
The invitation token carried the minter in :profile-id while the
invitation row tracks the creator in :created-by. Both mean the
inviter, so re-sends or re-requested links made them disagree and
forced a second response key, :user-who-send-invitation.
Mint :profile-id from :created-by in both token creators, backfill
it from the row on accept (covers stale in-flight tokens), and drop
the duplicate response key. The frontend maps :profile-id to the
unchanged :user-who-send-invitation audit prop.
AI-assisted-by: Muse Spark 1.3 Free
* ♻️ Reuse token ids in invitation accept response
Backfill :member-id with the accepting profile and drop the
:user-id duplicate from the verify-token response, mirroring the
:profile-id/:user-who-send-invitation cleanup. The frontend maps
:member-id to the unchanged :user-id audit prop.
AI-assisted-by: Muse Spark 1.3 Free
---------
Co-authored-by: Andrey Antukh <niwi@niwi.nz>
The seven creation commands no longer accept an optional client
id: create-file, create-project, create-team,
create-team-with-invitations, upload-file-media-object,
create-file-media-object-from-url and assemble-file-media-object.
The server always generates the identifier; a sent id is ignored.
Malli maps are open and the RPC layer never strips unknown params,
so the handlers that would still honor an id (create-file,
create-project) now drop it explicitly. Internal callers that pass
remapped ids (project duplicate, binfile import) keep working.
Closes#11783
AI-assisted-by: muse-spark-1.3-contributor
* ✨ Restrict optional RPC ids to user-provided UUIDs
Add ::sm/user-provided-uuid, backed by a version and variant
aware regex that only accepts v4, v7 and v8 instances. Use it
for the optional :id of the creation RPC commands so reserved
versions such as v3 are rejected at validation time. Reads
such as get-team keep the lax ::sm/uuid. Cover the predicate
and the schema on both JVM and JS runtimes.
AI-assisted-by: muse-spark-1.3-contributor
* ✨ Cover id version restriction at the RPC boundary
Add backend regression tests proving the seven creation commands
reject reserved-version ids (v3) with :params-validation and
accept v4 ids (plus v7/v8 on create-team) through the real
decode and validate path. Also drop two duplicated assertions
and document the version and variant of every fixture UUID in
user-provided-test.
AI-assisted-by: muse-spark-1.3-contributor
Add owner protection to ::delete-team-member RPC command.
Previously, a team admin could remove the team owner, permanently
locking them out of their team and all resources.
Changes:
- Fetch target member data before deletion
- Validate member exists (return :not-found if not)
- Reject removal if target is owner and caller is not owner
This mirrors the existing protection in update-team-member-role.
Closes#11367
AI-assisted-by: qwen3.7-plus
The handler previously allowed any team member (including viewers) to
generate invitation tokens. Now requires at least edition-level
permissions (can-edit, admin, or owner).
Closes#11358
AI-assisted-by: longcat-2.0
Add role-ceiling check to create-team-invitations and
update-team-invitation-role methods. These RPC methods allowed
team admins to grant or elevate invitations to :owner role,
bypassing the protection that exists in update-team-member-role.
The fix replicates the existing check from update-team-member-role:
reject promotion to :owner when the caller is not an owner.
Closes#11098
AI-assisted-by: qwen3.7-plus
* ♻️ Rename nitrate config to admin-console
Rename user-facing configuration from 'nitrate' to 'admin-console':
- Feature flags: :nitrate -> :admin-console, :nitrate-bulk-create-profiles -> :admin-console-bulk-create-profiles
- Config keys: :nitrate-shared-key -> :admin-console-shared-key, :nitrate-backend-uri -> :admin-console-uri
- Shared-keys map entry: :nitrate -> :admin-console (setup.clj + main.clj)
- Env vars: PENPOT_NITRATE_SHARED_KEY -> PENPOT_ADMIN_CONSOLE_SHARED_KEY, PENPOT_NITRATE_BACKEND_URI removed (consolidated into PENPOT_ADMIN_CONSOLE_URI)
- Docker/nginx: PENPOT_NITRATE_URI -> PENPOT_ADMIN_CONSOLE_URI
Code namespaces, file paths, CSS classes, and i18n keys stay as-is.
AI-assisted-by: mimo-v2.5-pro
* ♻️ Rename initialize-user-in-nitrate-organization to initialize-user-in-organization
Part of the nitrate -> admin-console rename series. The function and all 9 references across 6 files have been renamed.
* ♻️ Rename :nitrate-bulk-create-profiles-not-allowed to :bulk-create-profiles-not-allowed
* ♻️ Inline nitrate-permissions into app.common.types.organization
- Delete app.common.types.nitrate-permissions and its test
- Move permission rules (allowed?, can-send-invitations?, etc.) into organization.cljc
- Harmonize all consumers to use alias cto for app.common.types.organization
- Update test runner and create organization_test.cljc
This simplifes the mental model on how it works and simplifies testing
of the related code.
This also normalizes storage object deletion in the same way as the
rest of objects in penpot (now future deletion date on storage object
also means storage object to be deleted).
Mainly make it receive the whol cfg/system instead only props. This
makes the api more flexible for a future extending without the need
to change the api again.
* 📎 Add minor improvements to team tests
* 🐛 Fix incorrect handling request access with deleted profiles
* 🐛 Fix redirect loop on empty route
Happens when the current profile is deleted from team
* 🐛 Fix urls on request access emails
* 📎 Revert url changes on emails
this allows almost all api operations to success usin application/json
encoding with the exception of the update-file, which we need to
approach a bit differently;
the reason update-file is different, is because the operations vector
is right now defined without the context of shape type, so we are just
unable to properly parse the value to correct type using the schema
decoding mechanism
The main issue was the long running gc operation that
affects storage objects with deduplication. The long running
transacion ends locking some storage object rows which collaterally
made operations like import-binfile become blocked indefinitelly
because of the same rows (because of deduplication).
The solution used in this commit is split operations on small
chunks so we no longer use long running transactions that holds
too many locks. With this approach we will make a window to work
concurrently all operarate the distinct operations that requires
locks on the same rows.
The main objective is prevent deletion of objects that can leave
unreachable orphan objects which we are unable to correctly track.
Additionally, this commit includes:
1. Properly implement safe cascade deletion of all participating
tables on soft deletion in the objects-gc task;
2. Make the file thumbnail related tables also participate in the
touch/refcount mechanism applyign to the same safety checks;
3. Add helper for db query lazy iteration using PostgreSQL support
for server side cursors;
4. Fix efficiency issues on gc related task using server side
cursors instead of custom chunked iteration for processing data.
The problem resided when a large chunk of rows that has identical
value on the deleted_at column and the chunk size is small (the
default); when the custom chunked iteration only reads a first N
items and skip the rest of the set to the next run.
This has caused many objects to remain pending to be eliminated,
taking up space for longer than expected. The server side cursor
based iteration does not has this problem and iterates correctly
over all objects.
5. Fix refcount issues on font variant deletion RPC methods
Mainly the followin changes:
- Pass majority of code to the old and plain synchronous style
and start using virtual threads for the RPC (and partially some
HTTP server middlewares).
- Make some improvements on how CLIMIT is handled, simplifying code
- Improve considerably performance reducing the reflection and
unnecesary funcion calls on the whole stack-trace of an RPC call.
- Improve efficiency reducing considerably the total threads number.
