Merge remote-tracking branch 'origin/develop' into staging

.agents/skills/migrate-ds-area/SKILL.md

@@ -0,0 +1,330 @@
+---
+name: migrate-ds-area
+description: Migrate one Penpot UI area to the Design System components for one group of the DS migration epic (#12375): buttons and icons (icon-button*, button*, icon*), selects (select*) and later groups. Moves the area's SCSS off the deprecated styles, proves every change with a zero-tolerance visual test, then opens the draft PR and updates the group issue (PRs table row, QA comment). Use when the user asks to migrate an area or module to the DS, continue a DS migration group, or names an area of a group issue.
+---
+
+# Skill: migrate-ds-area
+
+Move one UI area (one `.cljs` + `.scss` pair, or a few related modules)
+from legacy components and raw HTML to the Design System components,
+prove with pixels what changed, and ship it as a draft PR that QA can
+test.
+
+The epic is #12375. Each control type is a group with its own issue
+(buttons and icons #12376, selects #12377, radios #12378, …). The group
+issue holds the plan (order, waiting areas), the changes QA should
+expect, and the PRs table where people claim areas. Read it first.
+
+## Input
+
+- The group issue (e.g. `#12376`) and the area: a module path (e.g.
+  `dashboard/fonts`) or an area named in the group issue.
+- No area given: take the next unclaimed area in the group issue's plan
+  order, and say which one.
+
+## Ground rules
+
+- Never `git push`. The user pushes; then you open the PR.
+- PRs go to `develop`; branches start from `origin/develop`.
+- **Stay in the group.** Only the controls of this group change. In the
+  buttons/icons group, selects, radios, numeric/text inputs, checkboxes,
+  menus (`%menu-*`, `%dropdown-*`) and modal structure (`%modal-*`) stay;
+  a checkbox tick or an input label icon stays legacy.
+- **Accept the DS look.** Do not add CSS to imitate legacy sizes or
+  colors. Do keep positions: icons and labels must not move unless the
+  DS size forces it, and then say so.
+- No negative margins to compensate; use positive DS spacing or
+  `px2rem()`.
+- **Gaps in the DS wait.** An area that needs an icon or a feature the
+  DS lacks keeps its legacy control; say so in the PR.
+- Skip legacy shared components that another group will replace
+  (`context_menu_a11y`, `forms`, `select`, `dropdown`, `radio_buttons`,
+  `search_bar`, `title_bar`, …): they wait for that group's cleanup,
+  where they are updated if still used or deleted if not.
+- Debug-only UI gets no QA comment and no compare image.
+
+## Devenv
+
+Run every test, lint and `pnpm` command inside the workspace container,
+never on the host (a host `pnpm` relinks `node_modules` and breaks the
+container's watcher). The container is `penpot-devenv-ws<N>-main`, where
+`<N>` is the workspace number (`ws0` is the main repo; `wsN` lives in
+`~/.penpot/penpot_workspaces/wsN`). Write output to a file, then read it:
+
+```bash
+C=penpot-devenv-ws1-main   # your workspace
+docker exec -u penpot -w /home/penpot/penpot/frontend $C \
+  bash -lc '<command>' > <scratch>/out.txt 2>&1
+```
+
+When the app does not behave as the code says, check the build first:
+
+- **Stale build after a branch switch** ("is not a function", or a page
+  with empty `stl/css` classes): `touch` the stale `.cljs`/`.scss`, wait
+  for the compiled file under `frontend/resources/public/` to update,
+  hard-reload.
+- **A whole area unstyled:** a new `.scss` added after the watcher
+  started is missing from `main.css` (grep `main.css` for the module
+  prefix), or `main.css` was written corrupted after many `.scss`
+  changed at once. Touch a `.scss`; if the module is still missing,
+  restart the watcher in the workspace's tmux session (`penpot:0`,
+  `./scripts/watch app`).
+- **Blank app after switching between `develop` and `staging`:**
+  dependencies differ. Run `pnpm install --frozen-lockfile` in the
+  container and restart the watcher.
+
+## Workflow
+
+### 1. Claim the area and branch
+
+1. Read the group issue: plan, waiting areas, expected changes, PRs table.
+2. Add a row to the PRs table with no PR number, the user's @login as
+   Owner and the state `Claimed` (`gh issue view/edit <n> --body-file`).
+   Read the body right before editing, and read it again after saving:
+   when two people edit at the same time, GitHub keeps only the last
+   edit, so a row can vanish. Put yours back if it is gone.
+   Skip an area someone else already claimed, unless the claim has had
+   no PR for 7 days and its owner did not answer a comment asking.
+3. `git switch -c <user>-ds-<group>-<area> origin/develop`.
+4. Survey the area: every control of the group in the `.cljs`, the
+   legacy icons, the placeholders and mixins the `.scss` uses, the
+   `div`/`span` with `:on-click`.
+
+### 2. Visual baseline
+
+Add a `test.describe("<Area> controls", …)` block to the matching spec
+in `frontend/playwright/ui/visual-specs/` (project `ds`, dark theme).
+Put it next to the blocks of the same screen, not always at the end of
+the file: two PRs that both append at the end conflict.
+
+- Every shot: `toHaveScreenshot({ maxDiffPixelRatio: 0, threshold: 0 })`.
+- Show every changed control at rest, on hover and open or pressed.
+  Capture the element or the smallest stable container.
+- Locators must work before and after the change: text, role, testid,
+  `[class*='…']`, or `a.or(b)` when the element type changes (raw
+  `[class*='menu-btn']` before, `getByRole("button", { name: "Options" })`
+  after). Scope `.or()` to the area so it cannot match elsewhere.
+- Missing data? Add a fixture under `frontend/playwright/data/` and mock
+  it (`dashboardPage.mockRPC(...)`, `DashboardPage.mockConfigFlags` for
+  flags). Watch for expired or inconsistent fixtures (dates in the past,
+  wrong file for a mock).
+- Capture with `--update-snapshots=all`:
+  ```bash
+  npx playwright test --project=ds <spec> -g "<describe>" --update-snapshots=all
+  ```
+  Plain `--update-snapshots` keeps a snapshot within the project's 0.5%
+  tolerance, so a real diff looks like 0.
+- Copy the snapshots (`<spec>.spec.js-snapshots/<Describe>-*`) to
+  `<scratch>/diffs/<area>/before/` and look at them: the state you
+  wanted must really be on screen.
+
+### 3. Controls commit
+
+#### Buttons and icons
+
+| Legacy | DS |
+|---|---|
+| icon-only raw button | `[:> icon-button* {:variant "ghost" :aria-label (tr …) :icon i/x :on-click f}]` (always pass `:variant`; `"action"` for 24px icon toggles) |
+| pressed/selected | `:aria-pressed (boolean x)` |
+| text button | `[:> button* {:variant "primary"/"secondary"/"ghost"/"destructive" :type "button"} label]` |
+| `[:input {:type "button"}]` | `button*` |
+| modal close | `icon-button*` ghost, `:aria-label (tr "labels.close")`, `:tooltip-class` carrying the corner position |
+| legacy icon outside a button | `[:> icon* {:icon-id i/x :class …}]` |
+
+- Not every raw button changes: rows, tabs, swatches, avatar widgets,
+  full-height segments, round custom buttons and link-style text buttons
+  stay raw (their legacy icon still changes). List the kept ones in the
+  PR.
+- A `div`/`span` with `:on-click` in the area becomes a `[:button]`
+  (with `aria-expanded`/`aria-haspopup` when it opens something). An
+  `[:a]` without `href` stays; list it in the PR.
+- **Tooltip wrapper.** `icon-button*` renders inside a tooltip trigger
+  wrapper. Position (`position: absolute`, grid placement, margins
+  against siblings) goes on `:tooltip-class`; look (`--button-height`,
+  colors) goes on `:class`.
+- **Tooltip at the right edge.** Small icon buttons at the right end of
+  a sidebar row or title bar get `:tooltip-placement "top-left"`; list
+  it as an expected change.
+- **Sizes through custom properties:** `--button-height: #{px2rem(64)}` /
+  `--button-width`. Sass does not evaluate functions inside custom
+  properties without `#{}`.
+- **A grown `button*` keeps its label at the start**: add
+  `justify-content: center` when it is wider than its content.
+- **Icon colors:** DS icons use `stroke: currentColor`. Set `color` on
+  the icon class, not `stroke`/`fill`. A legacy `color: transparent` left
+  on the button or wrapper makes the DS icon invisible: drop it. A class
+  that redefines the color variable on the icon breaks the hover that
+  changes it on the button: define the variable on the button.
+- **Every DS icon needs a color of its own** (on its class or a parent):
+  one that inherits none shows in the devenv debug yellow.
+- **Icon sizes:** `"s"` draws 12px inside a 16px box (compensate the
+  margin by 2px per side to keep the drawing in place); `"m"` 16px; `"l"`
+  24px; `"xl"` 32px.
+- **Stroke width:** DS icons keep the DS stroke width. Never set
+  `stroke-width` on a DS icon to match the legacy one; a thinner or
+  thicker line is part of the DS look.
+- **Legacy placeholders that style `svg` descendants** (e.g.
+  `%dropdown-element-base` → `span svg { 12px box; stroke }`) override a
+  DS icon placed inside them: give the icon class its own box
+  (`inline-size`/`block-size` 16px) and `stroke: currentcolor` with more
+  specificity.
+- **Flex shrink:** a DS svg inside a small flex button can shrink and be
+  cut; add `flex-shrink: 0` to the icon class.
+- **Rotated legacy arrows:** prefer `i/arrow-left`/`-right`/`-up`/`-down`
+  over rotating `i/arrow` (same path), unless a rotation class drives the
+  open/closed state.
+- **Menus anchored by the button size** move when the button grows; open
+  them from the top (`inset-block-start: calc(100% + var(--sp-xs))`) or
+  fix the context menu offset, and say so.
+
+#### Selects
+
+`[:& select {:default-value v :options [{:value :label}] :on-change f}]`
+→ `[:> select* {:default-selected <id> :options <memoized vector of {:id :label}> :on-change f}]`
+(`app.main.ui.ds.controls.select`).
+
+- **Ids are non-blank strings.** A blank id shows a dimmed "--", and an
+  id that matches no option falls back to the first one. Give mixed,
+  none and auto their own ids (`"multiple"`, `"none"`, `"auto"`) and map
+  them back in the handler. Use `""` only where "--" is the wanted look.
+- **Convert the value.** The legacy select ran `d/read-string`, so
+  handlers got keywords, numbers or uuids; `select*` passes the `:id`
+  string. Convert with `keyword`, `d/parse-integer` or `uuid/parse`.
+- **Memoized options, live selection.** `select*` resets its selection
+  when `:default-selected` or the `:options` identity changes. Build
+  options with `mf/with-memo` and `mapv` (the schema wants a non-empty
+  vector) and pass `:default-selected` from the live value, never a
+  constant.
+- **Never pass nil.** In dev builds a present prop with value nil
+  (`:class`, `:wrapper-class`, `:disabled`, `:default-selected`) throws
+  `invalid props on component select*`. Use `(boolean x)`, `d/nilv`, or
+  add the prop only when it has a value.
+- **Props the legacy select ignored stay out**: `select*` passes rest
+  props to its button, so they would reach the DOM.
+- **Classes:** `:wrapper-class` for layout, `:class` for look.
+- **Portal:** add `:has-portal true` only where the visual test shows the
+  list cut at the bottom of a panel, and say so in the QA comment.
+- **Locators:** the button is a `combobox` with no accessible name; use
+  `getByRole("combobox").filter({ hasText: "Linear" })` (it matches the
+  legacy select too) and `getByRole("option", { name: … })`.
+- **Keyboard:** the arrows no longer change the value while the list is
+  closed. It is an expected change, already listed in #12377.
+
+#### All groups
+
+- New labels: add the key to `en.po` and `es.po` by hand, in sorted
+  position, with a `#:` reference to the cljs line. Never run the
+  translations sync.
+- The `.scss` only gets what the new controls need in this commit.
+- Some e2e specs are not Prettier-clean on develop: never run
+  `prettier --write` on a whole spec; format only your block.
+
+Then wait for the watcher (grep the compiled JS/CSS under
+`frontend/resources/public/` for the change), recapture with `=all`,
+copy to `after/`, diff:
+
+```bash
+SK=.agents/skills/migrate-ds-area/scripts
+python3 -I $SK/diff.py <before> <after>              # per-file pixel count + bbox
+python3 -I $SK/trip.py <b> <a> <out.png> <scale>     # before | after | red diff
+```
+
+`trip.py` pads images of different sizes, so a taller "after" still
+renders.
+Explain every diff. Fix what is not a DS-look change (moved icons, cut
+icons, lost hover color). Rerun a shot that changes on unchanged code
+before trusting it; flaky spots seen so far: the invite modal textarea
+border, the search input caret, the user account page, the path handler
+menu trigger.
+
+Lint, then commit (read `mem:workflow/creating-commits` first):
+
+```bash
+npx prettier --check <spec> <fixtures>; npx stylelint <scss>; clj-kondo --lint <cljs>; cljfmt check <cljs>
+git add -f <cljs> <scss> <po> <spec> <fixtures>   # -f: visual-specs/ is gitignored
+./scripts/check-commit
+```
+
+A plain `git add` with an ignored path in the list adds nothing; always
+`-f`, and check `git show --stat`. Never commit the screenshots.
+Subject: `:recycle: Use DS <controls> in <area>`. A pre-existing bug
+found and fixed on the way goes in its own `:bug:` commit.
+
+### 4. SCSS commit
+
+Run the `refactor-scss` skill on every `.scss` the PR touched (only
+those), in its own commit. Keep `refactor/` only for component-family
+placeholders (`%modal-*`, `%menu-*`, `%input-*`, `%checkbox-*`) and say
+why in a comment above the `@use`. Unused selectors are deleted, not
+commented out (list them in the commit body). Typography without an
+exact match uses the closest DS typography. Run the skill's
+**Self-review** before committing: no bare element selectors (add
+classes in the `.cljs`), no copied properties that do nothing, z-index
+tokens by purpose. Compare against the controls commit: it should be
+0 px; explain anything else (typically a typography or border change).
+For files that several modules share or that are huge, stop and propose
+a separate PR instead.
+
+### 5. Hand off, PR and tracking
+
+1. Report to the user: commits, visible changes, kept raw controls,
+   pre-existing bugs, and the compare image path
+   (`<scratch>/diffs/<area>/compare.png`, before | after | diff). Give
+   the push command: `git push -u origin <branch>` (the branch tracks
+   `origin/develop`, so a bare `git push` is refused).
+2. After the user pushes, check `git rev-parse HEAD origin/<branch>`,
+   then (read `mem:workflow/creating-prs` first):
+   ```bash
+   gh pr create --repo penpot/penpot --draft --base develop --head <branch> \
+     --title "<commit subject>" --body-file <body> \
+     --label "WG adoption" --label "no issue required" \
+     --milestone <group issue milestone> --project Main
+   gh pr edit <n> --repo penpot/penpot --remove-label "needs triage"
+   ```
+   Never add `no changelog` to a PR: that label goes on issues.
+   Body: AI note, `Part of #<group> (DS migration epic #12375).`, a
+   short What / How, the kept raw controls, an empty `## Screenshots`
+   when something is visible (the user attaches the image). Never
+   `Closes #<group>`. Keep it short: reviewers find long PRs hard to read.
+3. In the group issue's PRs table, fill your row:
+   `| #<n> | <area>: <controls> (<visible changes>) | @<owner> | Draft |`.
+4. Post a QA comment on the group issue: `**#<n>: <area>**`, *Where* (how
+   to reach it), *Expected changes* (every visible change; QA treats
+   anything not listed as a bug), *No other change* (what still works
+   the same).
+
+Do not mark the draft ready or ask for review: the user decides when.
+
+### 6. Review fixes
+
+When the reviewer comments on a PR:
+
+1. Fix each comment in a new commit on the branch (never amend pushed
+   commits). A comment you do not apply gets a short reason the user can
+   post, with an image when it is visual.
+2. Rerun the area's visual spec before and after the fix with `=all`.
+   Build a compare image for every visible change and give the user its
+   full path, saying which PR it goes to.
+3. Update everything the fix makes stale, in the same turn: the QA
+   comment (*Expected changes* must list the new visible changes), the
+   PRs table row, and the PR body.
+4. When working on several PRs in one go, report per PR: branch, new
+   commit, what changed, image path. Never mix two branches in one list.
+5. Give the push command per branch. If a rebase is coming soon, say
+   that pushing now costs one extra CI run; the user decides.
+6. After a round of review fixes, check whether the same issue exists in
+   the user's other open PRs of the group, and fix it there too.
+
+### 7. After merges
+
+Whenever PRs merge (or go from draft to ready), sync the state column of
+the group's PRs table with GitHub for **every** row, not only your own:
+read each row's PR (`gh pr view <n> --json state,isDraft`) and set
+Merged / In review / Draft. The table drifts when other people merge.
+
+## Output
+
+- Branch with 2 commits (controls, SCSS) plus any `:bug:` commit.
+- Visual spec block with zero-tolerance shots.
+- Compare image in the scratch folder.
+- After the push: draft PR, row and QA comment in the group issue.

.agents/skills/migrate-ds-area/scripts/diff.py

@@ -0,0 +1,25 @@
+"""Pixel diff of two folders of PNG snapshots with the same file names.
+
+Usage: python3 -I diff.py <before-dir> <after-dir>
+Prints, per file, the number of differing pixels and their bounding box,
+or the two sizes when they differ.
+"""
+import os
+import sys
+
+from PIL import Image, ImageChops
+
+before, after = sys.argv[1], sys.argv[2]
+for name in sorted(os.listdir(before)):
+    other = os.path.join(after, name)
+    if not os.path.exists(other):
+        print(name, "missing in", after)
+        continue
+    a = Image.open(os.path.join(before, name)).convert("RGB")
+    b = Image.open(other).convert("RGB")
+    if a.size != b.size:
+        print(name, "size", a.size, b.size)
+        continue
+    diff = ImageChops.difference(a, b)
+    count = sum(1 for p in diff.getdata() if p != (0, 0, 0))
+    print(name, count, diff.getbbox())

.agents/skills/migrate-ds-area/scripts/trip.py

@@ -0,0 +1,33 @@
+"""Before | after | red diff image, one row per snapshot.
+
+Usage: python3 -I trip.py <before-dir> <after-dir> <out.png> <scale>
+Both folders hold PNGs with the same names (crop them first if needed).
+"""
+import os
+import sys
+
+from PIL import Image, ImageChops
+
+before, after, out, scale = sys.argv[1], sys.argv[2], sys.argv[3], int(sys.argv[4])
+rows = []
+for name in sorted(os.listdir(before)):
+    a = Image.open(os.path.join(before, name)).convert("RGB")
+    b = Image.open(os.path.join(after, name)).convert("RGB")
+    w, h = max(a.width, b.width), max(a.height, b.height)
+    pad_a = Image.new("RGB", (w, h), (200, 200, 200)); pad_a.paste(a, (0, 0))
+    pad_b = Image.new("RGB", (w, h), (200, 200, 200)); pad_b.paste(b, (0, 0))
+    mask = ImageChops.difference(pad_a, pad_b).convert("L").point(lambda v: 255 if v else 0)
+    red = Image.new("RGB", (w, h), (255, 255, 255))
+    red.paste((255, 0, 0), mask=mask)
+    row = Image.new("RGB", (w * 3 + 20, h), (200, 200, 200))
+    for i, im in enumerate([pad_a, pad_b, red]):
+        row.paste(im, (i * (w + 10), 0))
+    rows.append(row.resize((row.width * scale, row.height * scale), Image.NEAREST))
+width = max(r.width for r in rows)
+height = sum(r.height + 10 for r in rows)
+canvas = Image.new("RGB", (width, height), (255, 255, 255))
+y = 0
+for r in rows:
+    canvas.paste(r, (0, y))
+    y += r.height + 10
+canvas.save(out)

.agents/skills/refactor-scss/SKILL.md

@@ -0,0 +1,421 @@
+---
+name: refactor-scss
+description: Migrate one Penpot SCSS module off the deprecated styles under frontend/resources/styles/common/refactor/ to the current Design System (ds/ tokens, mixins and px2rem). Pure refactor, no visual change. Use when the user asks to refactor, migrate or clean up a .scss file to the DS.
+---
+
+# Skill: refactor-scss
+
+Refactor a single SCSS file so that it no longer depends on the deprecated
+styling system, using the current Design System instead.
+
+This is a **pure refactor**. Do NOT introduce visual or behavioral changes.
+Preserve the generated CSS behavior exactly unless a rule below requires
+otherwise. The rules that allow a small visible change (closest DS
+typography, DS border width) want it shown in the visual check and listed
+in the report and the commit body.
+
+Reviewers read every line of the result. A line that is
+copied but does nothing, a bare element selector or a raw z-index comes
+back as a review comment, so the rules below are strict about them.
+
+## Input
+
+- The path to one `.scss` file (e.g. `frontend/src/app/main/ui/foo/bar.scss`).
+- If no path is given, ask for it. Do not pick a file yourself.
+
+## Workflow
+
+1. Read the target `.scss` file and the `.cljs` file with the same name next
+   to it.
+2. Read every deprecated source the file uses, so you inline real values,
+   never remembered ones:
+   - `frontend/resources/styles/common/refactor/mixins.scss`
+   - `frontend/resources/styles/common/refactor/design-tokens.scss`
+   - `frontend/resources/styles/common/refactor/spacing.scss`
+   - any other file under `frontend/resources/styles/common/refactor/` that
+     the stylesheet pulls in (placeholders, borders, z-index, fonts...).
+3. Read the DS modules you will map to:
+   - `frontend/src/app/main/ui/ds/_utils.scss` (`px2rem()`)
+   - `frontend/src/app/main/ui/ds/spacing.scss`
+   - `frontend/src/app/main/ui/ds/_sizes.scss`
+   - `frontend/src/app/main/ui/ds/typography.scss` (`use-typography()`)
+   - `frontend/src/app/main/ui/ds/_borders.scss`
+   - `frontend/src/app/main/ui/ds/z-index.scss`
+4. Apply the rules below and edit the file in place.
+5. Verify (see **Verification** and **Visual check**).
+6. Report (see **Output**). Do not commit. When the refactor goes with other
+   changes, it belongs in its own commit, after the task's commits.
+
+## Primary goal
+
+After the refactor, the SCSS file must not depend on anything under
+`frontend/resources/styles/common/refactor/`. This includes imports, mixins,
+placeholder selectors, functions, deprecated variables and design-token
+aliases.
+
+### Partial refactor
+
+Some placeholders stand for a whole component family that will move to the DS
+in its own PR: `%input-element`, `%input-with-label`, `%input-checkbox`,
+`%modal-*`, `%menu-dropdown`, `%menu-item-base`, `%shortcut-*` and similar.
+Inlining them in a small task produces big, risky diffs. In that case:
+
+- migrate everything else (tokens, sizes, spacing, small mixins);
+- keep the `refactor/` import only for those placeholders;
+- list each remaining dependency and the selectors that use it in the report,
+  so the user can plan it as a separate PR.
+
+## General rules
+
+- Keep the resulting SCSS as small and readable as possible.
+- Avoid duplicated declarations and duplicated selectors.
+- Do not change class names.
+- Do not change DOM structure.
+- Do not change specificity unless required.
+- Follow the project's SCSS formatting (Prettier, see **Verification**).
+
+## Imports
+
+Use the DS modules. Import them with `as *` when they expose mixins,
+variables or functions:
+
+```scss
+@use "ds/_utils.scss" as *;
+```
+
+not
+
+```scss
+@use "ds/_utils.scss" as utils;
+```
+
+Do not introduce namespaces such as `utils.`, `typography.`, `borders.`,
+`spacing.` or `sizes.` unless a module explicitly requires one.
+
+Remove every unused import.
+
+## Deprecated mixins
+
+If the stylesheet uses mixins from
+`frontend/resources/styles/common/refactor/mixins.scss`:
+
+- locate the mixin;
+- inline its generated CSS (then apply the rest of these rules to it);
+- remove the `@include`;
+- remove the import.
+
+Do not leave deprecated mixins.
+
+## Placeholders
+
+Replace every `@extend %placeholder` with the generated CSS. Remove every
+deprecated placeholder, except the component-family ones described in
+**Partial refactor**.
+
+Inline only what does something on that element. Do not paste a
+placeholder (or mixin) body as it is. For each property, check:
+
+- **It applies to the element.** `fill`, `stroke` and `stroke-width` only
+  act on SVG shapes: drop them from a `span`, `div` or `button`.
+  `color: transparent` on a wrapper that only holds an icon does nothing
+  useful: drop it and set the real color.
+- **It is not the default or already given.** `stroke-width: 1px` is the
+  SVG default; `fill: none` and `stroke: currentColor` are already set by
+  the DS `icon*` (`.icon`). Drop them.
+- **A later rule does not override it.**
+- **It is not a stroke width on a DS icon.** DS icons keep the DS
+  stroke width; drop any `stroke-width` that only imitates the legacy
+  icon (1.33px, 2.25px...).
+
+`%button-icon` is the usual case: on a wrapper it leaves `color:
+transparent; fill: none; stroke-width: 1px`, and none of the three is
+needed.
+
+## Element selectors
+
+Do not style HTML elements directly (`svg {}`, `h3 {}`, `button {}`,
+`input.x {}`, or `.component { svg {} }`).
+
+When no class exists, **add one in the `.cljs`**: a DS component takes
+`:class (stl/css :x)` (`icon*`, `heading*`, `button*`…), and a plain
+element takes `{:class (stl/css :x)}`. Touching the `.cljs` for this is
+part of the refactor. A comment like "the icon has no class" is not an
+answer.
+
+Prefer the DS component when one fits: `heading*` for `h1`–`h6`
+(`:level`, `:typography`), `text*` for text, `icon*` for icons.
+
+`element.class` (e.g. `input.element-title`) is not needed to beat the
+input defaults: try the class alone and check it in the visual check.
+
+An element selector stays only for markup you cannot reach (injected
+HTML, third-party widgets), with a comment saying so.
+
+**Icon colors.** A DS icon draws with `stroke: currentColor`. Set
+`color` on the icon class or its parent, never `stroke`/`fill` on an
+`svg` selector.
+
+## Color variables
+
+Whenever a color changes across states, introduce a component-local CSS
+variable.
+
+Instead of
+
+```scss
+.button {
+  color: var(--color-foreground-primary);
+
+  &:hover {
+    color: var(--color-accent-primary);
+  }
+
+  &:disabled {
+    color: var(--color-foreground-disabled);
+  }
+}
+```
+
+write
+
+```scss
+.button {
+  --button-foreground-color: var(--color-foreground-primary);
+
+  color: var(--button-foreground-color);
+
+  &:hover {
+    --button-foreground-color: var(--color-accent-primary);
+  }
+
+  &:disabled {
+    --button-foreground-color: var(--color-foreground-disabled);
+  }
+}
+```
+
+Use this pattern for `color`, `background-color`, `border-color`,
+`outline-color`, `fill`, `stroke` and any similar property.
+
+## Design token aliases
+
+Remove every alias defined in
+`frontend/resources/styles/common/refactor/design-tokens.scss`. Resolve each
+alias to its final DS token. Do not keep intermediate aliases.
+
+Instead of `color: var(--button-foreground-hover);` write
+`color: var(--color-accent-primary);`.
+
+## Spacing
+
+Replace deprecated spacing variables using
+`frontend/src/app/main/ui/ds/spacing.scss` for `margin`, `padding`, `gap` and
+`inset`.
+
+If no exact spacing token exists, use `px2rem()`. Do not approximate with the
+closest token.
+
+## Sizes
+
+Replace deprecated size variables using
+`frontend/src/app/main/ui/ds/_sizes.scss` for `inline-size`, `block-size`,
+`width`, `height` and min/max sizes.
+
+If no exact size exists, use `px2rem()` from
+`frontend/src/app/main/ui/ds/_utils.scss`. Do not leave hardcoded pixel values
+unless absolutely necessary.
+
+## Typography
+
+When any of `font-size`, `font-weight`, `line-height`, `font-family` or
+`letter-spacing` appear, check whether together they match one of the
+definitions in `frontend/src/app/main/ui/ds/typography.scss`.
+
+If they match, replace all the individual declarations with:
+
+```scss
+@include use-typography("<typography-name>");
+```
+
+If there is no exact match, use the **closest DS typography** (same role:
+a panel title in uppercase → `headline-small`, body text → `body-*`),
+even when more than one detail differs. Show the change in the visual
+check and state it in the report and in the commit body (e.g. "11px
+regular → 12px medium").
+
+Keep the original declarations only when no DS typography fits the role
+(e.g. a 32px display number in a 14px-only area); then leave a TODO
+naming the closest one and ask the user.
+
+## Borders
+
+Replace deprecated border radius and border width values using
+`frontend/src/app/main/ui/ds/_borders.scss`. A raw width close to a token
+(`1.75px`) takes the token (`$b-2`); say so in the report. If nothing is
+close, use `px2rem()`.
+
+## Logical properties
+
+Prefer logical properties whenever they preserve the existing behavior:
+
+- `width` → `inline-size`
+- `height` → `block-size`
+- `max-width` → `max-inline-size`
+- `margin-left` → `margin-inline-start`
+- `padding-right` → `padding-inline-end`
+- `top` → `inset-block-start`
+- `left` → `inset-inline-start`
+
+## Z-index
+
+Never leave magic z-index values. Use tokens from
+`frontend/src/app/main/ui/ds/z-index.scss`, and pick the token **by what
+the element is, not by its number**: a dropdown, menu or select →
+`--z-index-dropdown`; a toolbar, palette, sidebar or panel →
+`--z-index-panels`; and so on (read the comments in that file). A raw `4`
+on a dropdown becomes `var(--z-index-dropdown)` even though the number
+changes. Check the visual check for overlap.
+
+Leave a TODO only when the element fits no token's purpose.
+
+Legacy z-index values between sibling elements that do not overlap can be
+removed, but only when the **Visual check** shows no pixel change in every
+state that matters (open menus, collapsed, hidden). Say so in the report.
+
+## Duplicated properties
+
+Within the same selector, avoid duplicated properties, duplicated CSS
+variables and duplicated declarations introduced by the refactor. Allow
+duplicates only when they are intentional browser fallbacks.
+
+## Unused classes
+
+Open the `.cljs` file with the same name as the SCSS file and check whether
+every CSS class is referenced (as `(stl/css :class-name)`,
+`(stl/css-case :class-name ...)` or similar).
+
+CSS modules prefix every class with its module
+(`main_ui_<path>__<class>`), so only the `.cljs` of the same module can use
+it. Before calling a selector unused, also search the whole `frontend/src`
+for the class name as a string, a `:global` use, or a dynamic class built
+at runtime.
+
+If a selector is unused after that check:
+
+- delete it (do not leave it commented out, do not add a TODO);
+- list every deleted selector in the report and in the commit body.
+
+A rule that matches nothing because the markup changed (e.g. `.icon svg`
+when the DS icon is the `svg` itself) is deleted the same way. Before
+deleting a descendant rule, check every element it matched, not only
+the one you changed: `.placeholder .icon svg` can also color a second
+icon, which then has no color (in the devenv it shows in the debug
+yellow).
+
+Every DS icon needs a color of its own (`color` on its class or on a
+parent with a defined color): an icon that inherits nothing shows in
+the debug yellow in the devenv.
+
+Classes used by markup injected from outside (e.g. an external widget)
+stay, with a comment saying so.
+
+Also check the other direction: every class the `.cljs` passes to
+`stl/css` / `stl/css-case` must exist in the `.scss`. A missing one silently
+does nothing; report it.
+
+## Sass and CSS pitfalls
+
+- Sass does not evaluate functions inside CSS custom properties. Write
+  `--button-height: calc(var(--height) - #{px2rem(16)});`, not
+  `... - px2rem(16))`, or the browser gets the literal `px2rem(16)`.
+- A `/` between Sass size variables is read as a division
+  (`grid-template: $sz-32 / auto` breaks). Split it into
+  `grid-template-rows` / `grid-template-columns` (or areas).
+- A CSS-wide keyword cannot be part of a shorthand: `inset-inline: unset 0`
+  is invalid and stylelint does not catch it. Use the longhands.
+- After editing, check the compiled CSS for any of these
+  (`frontend/resources/public/css/main.css`, search the module class).
+
+## Uncertainty
+
+If any transformation is uncertain, never guess. Leave a TODO in the code
+and explain why in the report.
+
+## Self-review
+
+Before the report, read the final diff as the reviewer would and search
+the file for these; each hit must be fixed or have a real reason:
+
+```bash
+rg -n 'svg \{|^\s*(h[1-6]|p|hr|input|button|span|a)\b[^-_]|[a-z]\.[a-z-]+ \{' <file>
+rg -n 'stroke-width|color: transparent|fill: none|z-index: [0-9]|TODO' <file>
+rg -n '[0-9]\.[0-9]+px|[^(]\b[0-9]+px' <file>
+```
+
+- an element selector → add a class (see **Element selectors**);
+- a property that does not apply or is a default → drop it (see
+  **Placeholders**);
+- a raw z-index → token by purpose;
+- a typography TODO → closest DS typography;
+- a raw px value → token or `px2rem()`.
+
+## Verification
+
+Run the project scripts from `frontend/` inside the devenv container
+(`penpot-devenv-ws<N>-main` for workspace `wsN`; never run `pnpm` on the
+host), writing output to a file first (never pipe lint output into
+filters):
+
+```bash
+docker exec -u penpot -w /home/penpot/penpot/frontend penpot-devenv-ws<N>-main \
+  bash -lc 'pnpm run lint:scss; echo LINT=$?; pnpm run check-fmt:scss; echo FMT=$?' \
+  > <scratch>/scss-check.txt 2>&1
+```
+
+If the format check fails, run `pnpm run fmt:scss` and check again.
+
+**Dependency check.** A grep for `refactor/` is not enough. For the final
+file, check:
+
+- `@use` / `@import` sources: all must be `ds/...` (or the sidebar common
+  module, which is DS-only);
+- every `@include`, `@extend` and `%placeholder`;
+- every `$variable`: which imported module defines it (`$br-*` exists in both
+  `ds/_borders.scss` and `refactor/borders.scss`; the `@use` decides);
+- every `var(--token)`: defined in the DS (colors, spacing, sizes) or only as a
+  `refactor/` alias.
+
+Report any failure as it is; do not hide it.
+
+## Visual check
+
+"No visual change" must be shown, not assumed. When a visual spec covers the
+component (`frontend/playwright/ui/visual-specs/`, project `ds`), or one can be
+added:
+
+1. Capture before (base branch styles) and after with
+   `pnpm exec playwright test --project=ds -g "<test>" --update-snapshots=all`
+   in the container. `=all` is required: plain `--update-snapshots` only
+   rewrites past the 0.5% tolerance, so a real diff can look like 0.
+2. Before each capture, wait until the compiled CSS/JS shows the expected
+   version (search `main.css` for the changed rule).
+3. Compare with zero tolerance. If pixels change, build the before | after |
+   diff (red) image and explain every change in the report.
+
+## Output
+
+1. The diff of the file (`git diff -- <file>`).
+2. A summary with:
+   - removed imports;
+   - removed deprecated mixins;
+   - removed placeholders;
+   - typography replacements;
+   - design-token aliases removed;
+   - logical properties introduced;
+   - deleted unused selectors;
+   - TODOs requiring manual review, each with its reason;
+   - remaining `refactor/` dependencies (partial refactor) and where they are
+     used;
+   - classes missing from the `.scss`, if any.
+3. The verification results, including the visual check (pixel count, or why
+   it was not possible).

.agents/skills/review-plan/SKILL.md

@@ -6,64 +6,40 @@ slash: true
 
 # Review Plan
 
-Act as a senior software engineer and perform a thorough review of an
-implementation plan.
+Act as a senior software engineer and perform a thorough review of an implementation plan.
 
 ## When to use
 
-- The user asks to review a plan, in any phrasing: "review this plan",
-  "does this plan look right?", "second opinion on the plan" — or runs
-  `/review-plan`.
-- A plan was just produced (typically by `/make-a-plan`) and the user
-  wants it evaluated before executing it.
+- The user asks to review a plan, in any phrasing: "review this plan", "does this plan look right?", "second opinion on the plan" — or runs `/review-plan`.
+- A plan was just produced (typically by `/make-a-plan`) and the user wants it evaluated before executing it.
 
 ## Instructions
 
-1. **Determine the plan under review** from the session context (for example, a
-   plan just produced by `/make-a-plan`) or from a plan file path given by the
-   user (typically under `.agents/plans/`). If a file path is given, read the
-   file first so the complete plan is in context.
-2. Delegate the review to the `general` subagent (via the task tool), unless the
-   user specifies another agent. Include in the prompt the
-   **`plan-review-criteria`** skill name and all user context.
-3. When the subagent returns, output the review to the user verbatim. Do not
-   summarize it and do not act on its findings.
-4. Right after the review, suggest the next step based on the verdict. These
-   are suggestions — the user decides, and any instruction overrides them:
+1. **Determine the plan under review** from the session context (for example, a plan just produced by `/make-a-plan`) or from a plan file path given by the user (typically under `.agents/plans/`). If a file path is given, read the file first so the complete plan is in context.
+2. Delegate the review to the `general` subagent (via the task tool), unless the user specifies another agent. Include in the prompt the **`plan-review-criteria`** skill name and all user context.
+3. When the subagent returns, output the review to the user verbatim. Do not summarize it and do not act on its findings.
+4. Right after the review, suggest the next step based on the verdict. These are suggestions — the user decides, and any instruction overrides them:
    - **Approve** → suggest `/implement-plan` to execute it.
-   - **Request changes** → suggest `/make-a-plan` to make a plan to address the
-     findings.
+   - **Request changes** → suggest `/make-a-plan` to make a plan to address the findings.
 
 ### Hard rule — read-only while reviewing
 
-This flow is read-only **for the duration of the review**: from the moment it
-starts until the user considers the review finished (including any feedback,
-questions, or clarifications about it). During that period, never fix,
-implement, edit files or create commits — not even "obvious" fixes derived from
-the findings. Once the user explicitly states the review is done (or moves on to
-a different task), this rule no longer applies and you act as a normal build
-agent again.
+This flow is read-only **for the duration of the review**: from the moment it starts until the user considers the review finished (including any feedback, questions, or clarifications about it). During that period, never fix, implement, edit files or create commits — not even "obvious" fixes derived from the findings. Once the user explicitly states the review is done (or moves on to a different task), this rule no longer applies and you act as a normal build agent again.
 
 ## Instructions for the subagent
 
-1. Load the **`plan-review-criteria`** skill and follow its process and output
-   format.
-2. Read `AGENTS.md` (if present) and follow its instructions for finding and
-   reading all related documentation and testing memories before reviewing.
-3. Return in your final message the COMPLETE review, verbatim, exactly as the
-   skill instructs it to be produced. Do not summarize it — include the full
-   structured review.
+1. Load the **`plan-review-criteria`** skill and follow its process and output format.
+2. Read `AGENTS.md` (if present) and follow its instructions for finding and reading all related documentation and testing memories before reviewing.
+3. Cada hallazgo debe tener un identificador único y estable, con el formato `F1`, `F2`, etc. El identificador debe aparecer en el título del hallazgo y no debe reutilizarse dentro de la misma revisión. Esto permite que el usuario pueda responder sobre un hallazgo concreto.
+4. Return in your final message the COMPLETE review, verbatim, exactly as the skill instructs it to be produced. Do not summarize it — include the full structured review.
 
 ### Strong rules for the subagent
 
 1. Do not invent problems. Every finding must be real and actionable.
-2. Read-only: do not modify any file and do not create a commit — reviewing
-   never writes.
-3. Be specific and constructive. "This could be better" is not helpful — explain
-   why and how.
+2. Read-only: do not modify any file and do not create a commit — reviewing never writes.
+3. Be specific and constructive. "This could be better" is not helpful — explain why and how.
 4. Prioritize by impact. One structural issue outweighs ten nits.
-5. Judge the plan as the implementer would: every task executable without
-   guessing, ordering follows the dependency graph, risks named.
+5. Judge the plan as the implementer would: every task executable without guessing, ordering follows the dependency graph, risks named.
 
 ## User input, overrides and additional context
 

.devenv/README.md

@@ -85,6 +85,10 @@ target instance is running and its MCP config has been generated:
 # 'main', 'ws1' and similar spellings are rejected.
 ./manage.sh start-coding-agent claude --ws 1
 ./manage.sh start-coding-agent opencode --ws 2
+
+# Run a different executable (e.g. a wrapper script) with the same MCP
+# wiring. --ws and --bin may come in any order.
+./manage.sh start-coding-agent claude --bin claude-alt
 ```
 
 Equivalents by hand (run from inside the workspace directory):

.github/dependabot.yml

@@ -1,20 +0,0 @@
-# https://docs.github.com/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file
-version: 2
-updates:
-  # Actions are pinned to a commit SHA with a `# vX.Y.Z` comment; Dependabot
-  # bumps both. All updates are grouped into a single monthly PR.
-  - package-ecosystem: "github-actions"
-    directory: "/"
-    schedule:
-      interval: "monthly"
-    open-pull-requests-limit: 5
-    labels:
-      - "dependencies"
-      - "team infra"
-      - "no issue required"
-    commit-message:
-      prefix: ":arrow_up:"
-    groups:
-      github-actions:
-        patterns:
-          - "*"

.github/workflows/build-docker-devenv.yml

@@ -38,13 +38,13 @@ jobs:
         uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
 
       - name: Login to Docker Registry (push destination)
-        uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
+        uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
         with:
           username: ${{ secrets.PUB_DOCKER_USERNAME }}
           password: ${{ secrets.PUB_DOCKER_PASSWORD }}
 
       - name: Login to Docker Hardened Images registry (base image pull)
-        uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
+        uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
         with:
           registry: dhi.io
           username: ${{ secrets.PUB_DOCKER_USERNAME }}

.github/workflows/build-docker.yml

@@ -200,7 +200,7 @@ jobs:
           ref: ${{ inputs.sha != '' && inputs.sha || inputs.gh_ref }}
 
       - name: Login to Docker Registry
-        uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
+        uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
         with:
           registry: ${{ secrets.DOCKER_REGISTRY }}
           username: ${{ secrets.DOCKER_USERNAME }}
@@ -210,7 +210,7 @@ jobs:
       # images from DockerHub for unregistered users.
       # https://docs.docker.com/docker-hub/usage/
       - name: Login to DockerHub Registry
-        uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
+        uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
         with:
           username: ${{ secrets.PUB_DOCKER_USERNAME }}
           password: ${{ secrets.PUB_DOCKER_PASSWORD }}
@@ -222,7 +222,7 @@ jobs:
       # reuses the same PUB_DOCKER_* credentials as the DockerHub
       # login above.
       - name: Login to Docker Hardened Images registry (base image pull)
-        uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
+        uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
         with:
           registry: dhi.io
           username: ${{ secrets.PUB_DOCKER_USERNAME }}
@@ -301,7 +301,7 @@ jobs:
         uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
 
       - name: Login to Docker Registry
-        uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
+        uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
         with:
           registry: ${{ secrets.DOCKER_REGISTRY }}
           username: ${{ secrets.DOCKER_USERNAME }}
@@ -371,7 +371,7 @@ jobs:
         uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
 
       - name: Login to Docker Registry
-        uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
+        uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
         with:
           registry: ${{ secrets.DOCKER_REGISTRY }}
           username: ${{ secrets.DOCKER_USERNAME }}

.github/workflows/build-renovate.yml

@@ -114,20 +114,20 @@ jobs:
           persist-credentials: false
 
       - name: Login to Docker Registry
-        uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
+        uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
         with:
           registry: ${{ secrets.DOCKER_REGISTRY }}
           username: ${{ secrets.DOCKER_USERNAME }}
           password: ${{ secrets.DOCKER_PASSWORD }}
 
       - name: Login to DockerHub Registry
-        uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
+        uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
         with:
           username: ${{ secrets.PUB_DOCKER_USERNAME }}
           password: ${{ secrets.PUB_DOCKER_PASSWORD }}
 
       - name: Login to Docker Hardened Images registry
-        uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
+        uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
         with:
           registry: dhi.io
           username: ${{ secrets.PUB_DOCKER_USERNAME }}
@@ -191,13 +191,13 @@ jobs:
           persist-credentials: false
 
       - name: Login to DockerHub Registry
-        uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
+        uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
         with:
           username: ${{ secrets.PUB_DOCKER_USERNAME }}
           password: ${{ secrets.PUB_DOCKER_PASSWORD }}
 
       - name: Login to Docker Hardened Images registry
-        uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
+        uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
         with:
           registry: dhi.io
           username: ${{ secrets.PUB_DOCKER_USERNAME }}
@@ -242,7 +242,7 @@ jobs:
           persist-credentials: false
 
       - name: Login to Docker Hardened Images registry
-        uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
+        uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
         with:
           registry: dhi.io
           username: ${{ secrets.PUB_DOCKER_USERNAME }}

.github/workflows/plugins-deploy-api-doc.yml

@@ -48,10 +48,10 @@ jobs:
       # No dependency cache here: a deploy must not restore a cache that
       # other workflows can write to.
       - name: Setup pnpm + Node.js
-        uses: pnpm/setup@a0881977cd8ea73b1ed212236c99ca10dcc80cf4 # v3.1.1
+        uses: pnpm/setup@fa36a5ef7dcdefc3a7cb25655c122e00cf6e636e # v3.1.2
         with:
           working-directory: plugins
-          runtime: node@24.22.2
+          runtime: node@26.13.0
           install: false
 
       - name: Install deps

.github/workflows/plugins-deploy-styles-doc.yml

@@ -46,10 +46,10 @@ jobs:
       # No dependency cache here: a deploy must not restore a cache that
       # other workflows can write to.
       - name: Setup pnpm + Node.js
-        uses: pnpm/setup@a0881977cd8ea73b1ed212236c99ca10dcc80cf4 # v3.1.1
+        uses: pnpm/setup@fa36a5ef7dcdefc3a7cb25655c122e00cf6e636e # v3.1.2
         with:
           working-directory: plugins
-          runtime: node@24.22.2
+          runtime: node@26.13.0
           install: false
 
       - name: Install deps

.serena/memories/audit-log.md

@@ -12,14 +12,15 @@ The system has:
 
 - **Three producers** (they create events): the Penpot backend (RPC layer), the Penpot frontend (browser app), and the Nitrate admin-console, which sends both server-side and client-side events.
 - **One central buffer**: the Postgres `audit_log` table in Penpot. It is temporary: it holds events for minutes/hours until they are archived, then they are deleted.
-- **Four consumers** (they read events): webhooks, error reporters, telemetry, and the external archive in Nexus.
-- **One long-term archive**: **Nexus**, a separate service that stores events for the long term. Archiving to **Nitrate** is in development and **does not exist on the current branch** (see section 11).
+- **Four consumers** (they read events): webhooks, error reporters, telemetry, and the external archives in Nexus and in the admin console.
+- **Two long-term archives**: **Nexus**, a separate service that stores events for the long term, and the **admin console** (Nitrate), which ingests its own allowlisted events through a second archive job (see section 12).
 
 ```
 Penpot frontend ─ POST push-audit-events (public) ─────────▶┐
 Nitrate admin-console (browser) ─ POST push-audit-events ──▶│
-Nitrate admin-console (server) ─ POST push-audit-events  ──▶│ ─▶ audit_log table (buffer) ── cron every 5m ──▶ Nexus (archive)
-Penpot backend (RPC wrap-audit) ─ submit ───────────────────┘         │   ├──▶ webhooks
+Nitrate admin-console (server) ─ POST push-audit-events  ──▶│ ─▶ audit_log table (buffer) ──┬─ cron 0s every 5m ──▶ Nexus (archive)
+Penpot backend (RPC wrap-audit) ─ submit ───────────────────┘         │                      └─ cron 30s every 5m ─▶ admin console (archive)
+                                                                      │   ├──▶ webhooks
                                                                       │   ├──▶ error reporters
                                                                       └──▶ telemetry (shadow rows)
 ```
@@ -44,7 +45,7 @@ Rule of thumb: if you need to know for sure "did this user do X?", look at `sour
 
 ## 3. The `audit_log` table: a buffer, not a store
 
-The Penpot `audit_log` table **is not the final destination**. It is a waiting tray: events sit here until the archive cron sends them to Nexus and the cleanup cron (GC) deletes them.
+The Penpot `audit_log` table **is not the final destination**. It is a waiting tray: events sit here until every active consumer (Nexus, admin console) confirms them, and whoever finishes last deletes the row.
 
 Main columns (migration `backend/src/app/migrations/sql/0054-add-audit-log-table.sql` and later ones):
 
@@ -56,15 +57,16 @@ Main columns (migration `backend/src/app/migrations/sql/0054-add-audit-log-table
 - `source` text: `backend` / `frontend` for full rows, `telemetry:backend` / `telemetry:frontend` for anonymized copies (see section 10).
 - `ip_addr` inet: real IP (or `0.0.0.0` on telemetry rows).
 - `props` / `context` jsonb holding transit-encoded maps: action data and metadata (versions, browser, request-id...).
-- `archived_at` timestamptz: NULL until Nexus confirms receipt; only then the row can be deleted.
+- No `archived_at`: the `audit_log_pending` queue (migration `0157`, consumer `1` = Nexus, `2` = admin console, ids never reused) tracks what is left. Every record is born pending only for the consumers enabled at insert time (never a fixed both): cheaper on single-consumer hosts and true to the design doc. Consequence, accepted on purpose: rows born with no send flag on get no pendings, so enabling archiving later does not pick them up — the sweep deletes them unarchived.
 
 Indexes (each consumer has its own, so a slow consumer never blocks the others):
 
-- Partial `created_at WHERE archived_at IS NULL`: used by the archive scan.
-- Partial `archived_at WHERE archived_at IS NOT NULL`: used by the GC.
+- `audit_log_pending` PK `(consumer, audit_log_id)`: each consumer owns its queue at constant cost, even with millions pending.
+- `audit_log (created_at, id)` (migration `0157`): the chronological drain order (`ORDER BY created_at ASC, id ASC`; same total order for both consumers, so concurrent drains cannot deadlock).
+- `audit_log_pending (audit_log_id)`: serves the "is anything still pending?" check of the final delete.
 - `(source, created_at)`: used by the telemetry scan.
 
-Life of a row: `INSERT` → (every 5 min) archive sets `archived_at = now()` **only if Nexus answers 204**, in the same transaction → (every 5 min) GC runs `DELETE WHERE archived_at IS NOT NULL` **with no age filter**. So archive must run before GC, or data is deleted without being sent.
+Life of a row: `INSERT` (row plus one pending per enabled consumer, same transaction; a rollback takes both) → (every 5 min, Nexus at second 0 and admin console at second 30) each archive job sends its own 128-row batches oldest-first and deletes its pendings in the same transaction → whoever deletes the last pending deletes the row. A batch the destination refuses rolls back and stays for the next tick. Pendings of consumers that are off are reaped at the start of each job run (covers later deactivations), and orphan rows are swept at the end of each run, so there is no separate GC task. The FK is plain `DEFERRABLE` (never `INITIALLY DEFERRED`): every deleter removes pendings before rows, including the telemetry task's own deletes.
 
 ## 4. Anatomy of an event
 
@@ -180,7 +182,7 @@ Who uses it:
 
 Failure policy: every send is best-effort. A failure never rolls back the business action; it is only recorded for debugging.
 
-Backend side (`backend/src/app/rpc/management/nitrate.clj` `::push-audit-events`): `::audit/skip`, `::rpc/auth false` (trust comes from the shared key, not from a user session). It builds context with `audit/prepare-context-from-request` + `request-id`, takes the given `profile-id` as-is (no coercion here, must already be a uuid), sets `tracked-at = request-at`, defaults `type` to `"action"`, and calls `audit/submit` (which defaults `source` to `"backend"`). So these rows land as **`source = backend`**: they are audited like backend events because a trusted service set the author.
+Backend side (`backend/src/app/rpc/management/nitrate.clj` `::push-audit-events`): `::audit/skip`, `::rpc/auth false` (trust comes from the shared key, not from a user session; only the `admin-console` key id is allowed). It builds context with `audit/prepare-context-from-request` + `request-id`, takes the given `profile-id` as-is (no coercion here, must already be a uuid), sets `tracked-at = request-at`, defaults `type` to `"action"`, and calls `audit/submit` (which defaults `source` to `"backend"`). So these rows land as **`source = backend`**: they are audited like backend events because a trusted service set the author.
 
 ### 7b. Browser channel (best-effort, like frontend events)
 
@@ -252,7 +254,7 @@ Result: crash reports only exist if the frontend collector runs and `push-audit-
 
 Telemetry reuses the same table with **anonymized shadow rows** (`source LIKE 'telemetry:%'`): day-truncated times, `0.0.0.0` IPs, props reduced to uuid/boolean/number values plus a few allowed fields (`source`, `lang`, `auth-backend`, `activation`, derived `email-domain` — never raw emails), and a minimal context allowlist. Full + shadow rows can exist together per event when both flags are on; that duplication is on purpose.
 
-Filters (`filter-telemetry-props` / `filter-telemetry-context`): by default only uuid/boolean/number with a simple key; exceptions: login/register/update-profile and frontend `identify` keep `source`/`auth-backend`/`activation`/`email-domain`; `organization-sso-auth-failed` keeps `failure-reason` if it is a known one; `instance-start` passes whole; frontend `navigate` keeps `[:route :file-id :team-id :page-id]`. Free text drops from telemetry but **stays in the full row and in Nexus**. In `register-profile`, `activation` is a prop (it is what reaches PostHog). `action` exists only in the `context` of the full `register-profile-retry` rows; `filter-telemetry-context` drops it from the anonymised copy, which keeps only `:version`, `:initiator`, `:client-version` and `:client-user-agent`.
+Filters (`filter-telemetry-props` / `filter-telemetry-context`): by default only uuid/boolean/number with a simple key; exceptions: login/register/update-profile and frontend `identify` keep `lang`/`auth-backend`/`source`/`activation`/`email-domain`; `organization-sso-auth-failed` keeps `failure-reason` if it is a known one; `instance-start` passes whole; frontend `navigate` keeps `[:route :file-id :team-id :page-id]`. Subscription actions have event-specific contracts: backend and frontend `create-subscription` / `create-trial-subscription` preserve the default uuid/boolean/number props plus `type`, `deployment`, `quantity` and `subscription-mode`; backend `update-subscription` preserves only `type`, `deployment`, `previous-status`, `status`, `previous-cancel-at-period-end`, `cancel-at-period-end`, `previous-cancel-at`, `cancel-at` and `changed-fields`; backend `subscription-summary` preserves only `type`, `deployment`, `instance-id`, `snapshot-date`, the per-status `*-count` values, `total-count`, `sync-succeeded`, `sync-failed` and `snapshot-complete`. These subscription exceptions apply only to action events. Other free text drops from telemetry but **stays in the full row and in Nexus**. In `register-profile`, `activation` is a prop (it is what reaches PostHog). `action` exists only in the `context` of the full `register-profile-retry` rows; `filter-telemetry-context` drops it from the anonymised copy, which keeps only `:version`, `:initiator`, `:client-version` and `:client-user-agent`.
 
 Shipping (`backend/src/app/tasks/telemetry.clj`): the cron sends shadows to `:telemetry-uri` as JSON in 10k batches and **deletes** them on success; it purges leftovers older than 7 days. It also sends a legacy report (counters, teams, email domains...). On official hosts nothing is collected or sent (`telemetry-excluded?` covers `penpot.app`/`penpot.dev`).
 
@@ -271,17 +273,19 @@ Nexus dispatches on `:type`: `"telemetry-events"` batches are coerced and insert
 - **No personal data is expected.** Profile ids are random UUIDs with no PII, so the telemetry consumer needs no login guard; grouping is by `instance`, not by person (see posthog-telemetry below).
 - **The endpoint always answers 200.** Failures are swallowed and only counted (`nexus_telemetry_errors_total`), so a bad batch never breaks the sending instance; successes are counted (`nexus_telemetry_events_total`, `nexus_telemetry_reports_total`). A body the parser cannot read never reaches the handler: it answers 400 and counts in `nexus_http_malformed_requests_total`. Old instances may send a fressian+zstd base64 blob instead of a plain vector; Nexus decodes both (50 MiB cap against decompression bombs).
 
-## 12. Archive to Nexus (today) and to Nitrate (future)
+## 12. Archive to Nexus and to the admin console
 
-### Nexus: how it works today
+### How it works today
 
-Nexus is the long-term store. Every 5 min the `:audit-log-archive` cron takes 128-row chunks of unarchived rows (`FOR UPDATE SKIP LOCKED`, ordered by `created_at`), POSTs them as transit `{:events [...]}` with `x-shared-key: "nexus <key>"` (`:nexus-shared-key`, or derived from the instance secret) and `origin` = public-uri, and marks `archived_at = now()` **only on HTTP 204**, in the same transaction. Anything else is retried next run; flag on but no URI raises `:task-not-configured`. It loops with 100 ms pauses until empty.
+Every 5 min each archive job drains its own queue (`:audit-log-consumer-nexus` on flag `:nexus`, cron `0 */5 * * * ?`; `:audit-log-consumer-admin-console` on flag `:admin-console`, cron `30 */5 * * * ?`, 30s offset so the two drains never start on the same second), in 128-row chunks taken oldest-first (`ORDER BY created_at ASC, id ASC`) with `FOR UPDATE OF audit_log` (`SET LOCAL lock_timeout = '30s'`) inside the same transaction that sends and completes them, with 100 ms pauses between chunks. Batches arrive at Nexus in chronological order (it numbers by arrival for order-sensitive consumers). Nexus receives transit `{:events [...]}` with `x-shared-key: "nexus <key>"` (`:nexus-shared-key`, or derived from the instance secret) and `origin` = public-uri; flag on but no URI raises `:task-not-configured`. The admin console receives only its allowlisted event names (enriched with team names, props emails, and optional top-level `:profile-email` / `:profile-name` from the actor `:profile-id`); anything filtered out counts as done for the admin-console queue. A chunk whose destination refuses it (Nexus non-204 raises `:consumer-chunk-refused`) or throws (transport failure, admin-console error) rolls back and stays for the next tick, and the job aborts as failed: it archives what it can, and the cron retries the rest. Committed chunks stay sent. The send fns return counts or throw carrying the attempted count as `:events`; `run-single-consumer` records the `failed` metric from it, outside every chunk transaction (a local DB failure carries no `:events` and stays uncounted — it is not a destination failure). No `warn` before the throw: the runner reports the aborted job. Each job touches only its own queue, so a fallen Nexus never starves the admin console (nor vice versa).
 
 A chunk is **one POST** and the task advances only on 204, so a single oversized event wedges the whole queue: a crash report whose message embeds a whole payload (a serialized file, for example) can push the request past the proxy body limit and the rejection blocks every later event until it is pruned by hand. Frontend crash reports are therefore size-bounded at the source (`mem:frontend/routing-app-shell-subtleties`): `app.common.exceptions/get-hint` and the CLJS `format-throwable` clamp exception messages (`max-message-length`).
 
 On the other side, Nexus checks `origin` against its allowlist (403 if not allowed), coerces the events, and inserts them into its own `audit_log` (partitioned by `received_at`, with an incrementing sequence number and `source`). Watch the rename: Penpot's `created-at` is stored as **`received-at`** in Nexus; Nexus's `created-at` is when it was created there. `tracked-at` keeps the origin time. Telemetry batches arriving at `telemetry.penpot.app` land in the same table through the telemetry endpoint above.
 
-Concurrent runs are safe: rows are locked with `FOR UPDATE SKIP LOCKED` inside the same transaction that sends and marks them, so two archivers running at once each take different rows. Only rows Nexus acknowledges (HTTP 204) get marked; anything else is retried on the next run.
+Concurrent jobs are safe: both lock rows in `created_at ASC, id ASC` order (one total order, deliberately blocking — no `SKIP LOCKED`, so archive arrival stays chronological), so the second waits for the first to commit and then sees whether it was last. Only acknowledged work deletes pendings; an admin-console failure never blocks the Nexus send of the same rows (each job owns its queue).
+
+Observability (`backend/src/app/loggers/audit/consumer_task.clj`, definitions in `backend/src/app/main.clj:default-metrics`): one `dbg` summary per job run (`:consumer` label, sent/skipped/chunks, reaped disabled pendings, swept orphans, `:elapsed` as `"1.5s"` via `ct/format-duration`, frozen in a strict binding first because log props evaluate lazily in the logger thread), one `dbg` per admin-console send call (`ready` with totals, `sent`/`failed` with counts), plus `dbg` when the queue is empty or the job's send flag is off. Queue depth is Grafana's business: it queries the database directly, so the run performs no `GROUP BY` for the logs. No `warn` before a throw: the runner reports the aborted job (refusal details travel in the error data). Metrics, all recorded with bounded labels only: `penpot_audit_consumer_events_total{ destination = nexus|admin-console, outcome = archived|failed|skipped }` (`skipped` = filtered by the admin-console allowlist) and `penpot_audit_consumer_discarded_total{ reason = reaped|swept }` (entries discarded without archiving: `reaped` counts pendings dropped for disabled consumers, `swept` counts rows deleted while nobody waited for them — the silent R3 loss, now dashboard-visible instead of log-only). Timing needs no custom histogram: the jobs substrate already records execution time per job (`jobs-execution-timing`, split by job name), so each archive job is timed as itself. No queue-depth gauge: pending depth is read off the replica database directly. The bulk frontend ingest counts `penpot_audit_ingest_events_total{ kind = full|shadow, outcome = inserted|failed }` (both faces share one transaction, so a failure drops the batch on both). Archive success counts are recorded by `run-single-consumer` on the drain totals: a chunk that returns normally committed its transaction (a rollback always throws instead), so they cannot inflate. Logs never carry event data (`props`, `context`, `profile-id`, `ip-addr`, `team-name`, emails): only counts, statuses and the uri host.
 
 ### How Nexus consumes the data (and forwards to PostHog)
 
@@ -291,20 +295,16 @@ Stored rows are not the end: a consumer pipeline reads them back in sequence ord
 2. **`posthog`** (runs after `local`): forwards our own SaaS archived events to PostHog. It sends an allowlist of backend track events (teams, members, projects, files, comments, Nitrate/SSO, ...) plus frontend events minus a noise exclusion list, grouped by **team**, with person identify data from the local profile mirror and snake_case props. Both paths include the `initiator` property when the stored context has one, so PostHog can split Penpot vs admin-console traffic.
 3. **`posthog-telemetry`**: forwards events with `source = telemetry:frontend|telemetry:backend` to a separate PostHog project. Because these rows are already anonymous (no IPs, truncated times, filtered props), grouping is by **instance** (the self-hosted server, from `context.instance-id`), with one `$groupidentify` event per instance per chunk so PostHog learns per-instance properties (version, URI, flags). Each consumer has its own PostHog credentials.
 
-So the full journey is: Penpot buffer → Nexus `audit_log` (via archive cron or telemetry endpoint) → consumers → PostHog (SaaS events grouped by team, telemetry grouped by instance) plus local analytics views and materialized KPIs.
+So the full journey is: Penpot buffer → Nexus `audit_log` (via either archive cron or the telemetry endpoint) → consumers → PostHog (SaaS events grouped by team, telemetry grouped by instance) plus local analytics views and materialized KPIs.
 
-### Nitrate: what exists and what is planned
+### Admin console: the second consumer
 
-Today **only Nexus archives**. Archiving Penpot → Nitrate is in development and not on this branch.
+Penpot → admin-console archiving lives in its own job (`:audit-log-consumer-admin-console`, consumer `2`, flag `:admin-console`, params closed empty): it drains its own pending queue and ingests allowlisted events through `nitrate/call :ingest-audit-log`, enriched as described above. What also exists (do not confuse with archiving):
 
-What already exists (do not confuse with archiving):
-
-- **Nitrate → Penpot**: `POST /api/management/methods/push-audit-events` (`backend/src/app/rpc/management/nitrate.clj`) lets the trusted Nitrate backend inject events into the Penpot audit log (no RPC auth, `::audit/skip`, request context + `submit`). Section 7a is the main user.
+- **Nitrate → Penpot**: `POST /api/management/methods/push-audit-events` (`backend/src/app/rpc/management/nitrate.clj`) lets the trusted Nitrate backend inject events into the Penpot audit log (no RPC auth, `::audit/skip`, request context + `submit`; only the `admin-console` shared-key id can call it). Section 7a is the main user.
 - **Admin-console browser → collector**: usage events from the admin-console UI traveling through the normal public `push-audit-events` (section 7b).
 - **Penpot frontend Nitrate bits**: membership/subscription events (`frontend/src/app/main/data/nitrate.cljs`, `nitrate-audit/*`) through the same public endpoint.
 
-The future Penpot → Nitrate archive will follow the Nexus pattern (cron reading the buffer and POSTing batches), but there is no contract or flag on this branch yet. When it lands, document here: endpoint, auth, format, which `source` values travel, and how it shares the GC.
-
 ## 13. Flags and config
 
 Flags (`common/src/app/common/flags.cljc`, varia; enabled with `PENPOT_FLAGS=enable-<name>`):
@@ -312,13 +312,13 @@ Flags (`common/src/app/common/flags.cljc`, varia; enabled with `PENPOT_FLAGS=ena
 | Flag | What it does |
 |---|---|
 | `:audit-log` | Stores full rows; turns on `wrap-audit` and frontend ingest |
-| `:audit-log-archive` | Turns on the cron that sends to Nexus (needs `:audit-log-archive-uri`) |
-| `:audit-log-gc` | Turns on the cron that deletes archived rows |
+| `:nexus` | Send flag of the Nexus archive job `:audit-log-consumer-nexus` (cron `0 */5 * * * ?`; needs `:nexus-uri` (ingest path `api/audit` appended by the job)) |
+| `:admin-console` | Send flag of the admin-console archive job `:audit-log-consumer-admin-console` (cron `30 */5 * * * ?`, 30s after Nexus); also the module flag for the admin-console integration |
 | `:audit-log-logger` | Structured `app.audit` log per event |
 | `:error-reporting` | Starts the error reporters (off by default); database reporter on flag alone, Mattermost on flag plus webhook URL |
 | `:telemetry` | Stores shadow rows and turns on shipping (auto with `:telemetry-enabled`) |
 
-Config (`backend/src/app/config.clj`): `:audit-log-archive-uri`, `:telemetry-uri` (default `https://telemetry.penpot.app/`), `:nexus-shared-key` / instance secret, `:public-uri` (sent as `origin` to Nexus).
+Config (`backend/src/app/config.clj`): `:nexus-uri`, `:telemetry-uri` (default `https://telemetry.penpot.app/`), `:nexus-shared-key` / instance secret, `:public-uri` (sent as `origin` to Nexus).
 
 ## 14. Practical guide: add or change events
 
@@ -337,6 +337,6 @@ Config (`backend/src/app/config.clj`): `:audit-log-archive-uri`, `:telemetry-uri
 - Penpot frontend producer: `frontend/src/app/main/data/event.cljs` (collector, buffer, send), `frontend/src/app/main/repo.cljs` + `frontend/src/app/util/http.cljs` (headers), `frontend/src/app/main/data/nitrate.cljs` (Nitrate membership events).
 - Admin-console producer (private repo, no paths cited here): server-side organization/subscription events through the management `push-audit-events`, browser-side UI events through the public `push-audit-events`.
 - Consumers: `backend/src/app/loggers/webhooks.clj`, `backend/src/app/loggers/database.clj`, `backend/src/app/loggers/mattermost.clj`, `backend/src/app/tasks/telemetry.clj`.
-- Archive/GC: `backend/src/app/loggers/audit/archive_task.clj`, `backend/src/app/loggers/audit/gc_task.clj`, wiring and crons in `backend/src/app/main.clj`.
+- Archive (two jobs sharing one namespace + pending queue, no GC): `backend/src/app/loggers/audit/consumer_task.clj` (`:audit-log-consumer-nexus` / `:audit-log-consumer-admin-console` job-defs, shared drain/reap/sweep helpers), producer hook in `backend/src/app/loggers/audit.clj` (`append-audit-entry`, single `queue-pendings` helper) plus the bulk ingest in `backend/src/app/rpc/commands/audit.clj` (`handle-events`), metrics/cron wiring in `backend/src/app/main.clj` (one cron entry per job, Nexus at second 0 and admin console at second 30, each gated by its own send flag), single migration `0157` (queue, drop of `archived_at`, covering `(created_at, id)` index for the chronological scan; no backfill — pre-existing rows are swept unsent, an accepted loss).
 - Nexus (private repo, no paths cited here): archive ingest, ordered consumers (local profile mirror, PostHog SaaS events, PostHog telemetry), telemetry endpoint, analytics views.
 - Tests: `backend_tests/rpc-audit-test.clj`, `rpc-team-test/accept-organization-invitation-audit-event`.

.serena/memories/backend/auth-permissions-product-domains.md

@@ -13,6 +13,10 @@
 - Logout may return an OIDC provider redirect URI when the session claims include provider/session data and the provider has a logout URI.
 - Invitation tokens are verified through token issuers and only accepted when the token member id/email matches the authenticated profile; otherwise login proceeds without consuming the invitation.
 - When public registration is disabled, invitation-based password registration also requires a matching, non-expired `team_invitation` row; registration revalidates it under a row lock before creating the profile.
+- Access tokens (`app.http.access-token`): personal tokens have JWT issuer `access-token` and DB type other than `mcp`; MCP tokens have issuer `urn:penpot:mcp-token` (`actoken/mcp-token-issuer`) and DB type `mcp`. Authentication requires issuer and type to agree, and the claim `uid` to match the row's profile; other issuers and undecryptable tokens are treated as anonymous.
+- Management RPC (`/api/management/methods/:name`) is shared-key only. The middleware authenticates `<id> <secret>`; the RPC wrapper `wrap-shared-key-authorization` authorizes the key id against the method's `::rpc/allowed-shared-keys` set. Absent metadata accepts any configured key; all current methods declare one (`exporter`/`media-processor` for `upload-tempfile`, `nexus` for file analysis, `exporter` for jobs, `admin-console` for Nitrate). A valid key outside the set answers 403 `:shared-key-not-allowed`; denied calls do not reach the audit wrapper.
+- Personal tokens need flag `:access-tokens`. MCP tokens need `:mcp` or `:access-tokens`, and only on routes whose `actoken/authz` gets `::actoken/allow-mcp true` (the RPC routes); elsewhere (e.g. assets) they are anonymous.
+- On RPC, an MCP token reaches only commands with `::rpc/token-type "mcp"` (`wrap-authentication` in `app.rpc`); every other command answers 403 `:token-auth-required`. A command with that metadata also rejects sessions and personal tokens. Currently only `send-mcp-feedback`. Blocking a profile invalidates sessions, not tokens, so such commands check `:is-blocked`/`:is-active` themselves.
 - HTTP/session parsing details such as cookie/header precedence, JWT session token versions, and SameSite behavior are in `mem:backend/subtleties`.
 
 ## Permission model

.serena/memories/backend/core.md

@@ -32,29 +32,64 @@ Backend: JVM Clojure; Integrant; PostgreSQL; Redis/Valkey; RPC; HTTP; storage; m
 
 RPC commands are defined with `app.util.services/defmethod` and schemas. Use `get-` prefixes for read operations. Command metadata usually includes auth, docs version, params schema, and result schema. Return plain maps/vectors or raise structured exceptions from `app.common.exceptions`.
 
-Backend RPC command areas without focused memories include access tokens, binfile, demo, feedback, file snapshots, fonts, management, Nitrate, and webhooks beyond the notes in `mem:backend/auth-permissions-product-domains`; inspect nearby command tests and command metadata before changing them.
+Backend RPC command areas without focused memories include access tokens (auth rules, incl. MCP tokens, in `mem:backend/auth-permissions-product-domains`), binfile, demo, feedback, file snapshots, fonts, management, Nitrate, and webhooks beyond the notes in `mem:backend/auth-permissions-product-domains`; inspect nearby command tests and command metadata before changing them.
 
 ## DB conventions
 
 `app.db` helpers accept cfg, pool, or conn in most places and convert kebab-case to snake_case:
 - `db/get`, `db/get*`, `db/query`, `db/insert!`, `db/update!`, `db/delete!`.
+- `db/get*`/`db/query` signature is `(ds table where-params & {:as opts})`: trailing keywords become an opts MAP (ignored by the select builder — use `{::db/remove-deleted false}` map as the single vararg to also see rows with `deleted_at` set), while a trailing ODD number of keywords crashes with "Don't know how to create ISeq from: Keyword". Extra keywords are NOT a column filter: rows always come back with all columns.
+- Next.js/JDBC caveat: where-maps don't support value vectors (no `IN`); use `status = ANY(?)` with `db/create-array` inside `db/tx-run!`.
 - Use `db/run!` for multiple operations on one connection.
 - Use `db/tx-run!` for transactions.
+- `job`-substrate specifics (unified jobs): the only writers of a job status are `claim`, `retry-job`, `complete`, `fail` and `cancel` in `app.jobs`; each one is a conditional update plus its `job_event` row in the same transaction, so history can never contradict the row. A writer that affects no row writes no event and no metric. They never join the caller's transaction, because a lifecycle write reports on work that already happened: a rolled back caller cannot undo a transition, and an independent connection cannot see the caller's uncommitted job row, so a job must be committed before it can be claimed. GC deletes return resource ids via `RETURNING` inside one transaction.
+- Every public function of `app.jobs` takes a cfg map. A bare pool or connection is a caller bug and is not coerced: the substrate has no conversion of its own, and the ones in `app.db` are the db layer's business.
+- Every new foreign key of the substrate is `DEFERRABLE`, `job_event.job_id` included.
+- Some tables carry a SQL deletion guard (`raise_deletion_protection` + a `deletion_protection__tgr` statement trigger): `profile`, `team`, `team_font_variant`, `file_media_object`, `file_thumbnail`, `file_tagged_object_thumbnail` and `job`. A `DELETE` raises unless the transaction runs `SET LOCAL rules.deletion_protection TO off` (the guard is on when the GUC is unset, empty or `on`); see `app.tasks.objects-gc`, `app.jobs.gc` and `th/db-force-delete`.
 
 Database migrations live in `backend/src/app/migrations/`; pure SQL migrations are under `backend/src/app/migrations/sql/`. SQL filenames conventionally start with a sequence and verb/table description, e.g. `0026-mod-profile-table-add-is-active-field`. Applied migrations are tracked in the `migrations` table.
 
 Migrations must stay within PostgreSQL 15 syntax: the official docker images (`docker/images/docker-compose.yaml`) pin `postgres:15`, so anything newer breaks those deployments (`SET STORAGE DEFAULT` is PostgreSQL 16, for instance). The documented floor is >=15.
 
-For interactive PostgreSQL access with correct dev defaults, use `scripts/psql`; to dump
-the current DDL schema, use `scripts/db-schema` (see `mem:scripts/psql`).
+For interactive PostgreSQL access with correct dev defaults, use `scripts/psql`; to dump the current DDL schema, use `scripts/db-schema` (see `mem:scripts/psql`).
 
 For deeper details on transaction semantics, advisory locks, Transit vs JSON helpers, and dev/test DB URLs: `mem:backend/subtleties`.
 
-## Background tasks
+## Background tasks (unified jobs)
 
-A task handler is an Integrant component with `ig/assert-key`, `ig/expand-key`, and `ig/init-key`, returning the function run by the worker. New tasks also need wiring in `app.main`: handler config, worker registry entry, and cron entry if scheduled.
+Every background job is a job-def: a plain `(defn execute-X [cfg params] ...)` in its namespace + a malli params schema + an `ig/init-key` that returns the job-def map `{::jobs/name, ::jobs/schema, ::jobs/handler, ::jobs/decoder, ::jobs/validator}` (decoder/validator precompiled at init), optionally with `::jobs/result-schema` when the result carries types plain JSON does not keep (a uuid, whether a value or a map key): the substrate compiles the decoder from that schema on registry init and checks a result against the schema before storing it. The registry is the `::jobs/defs` wiring in `app.main`, which also populates a module-level registry used by `jobs/submit` as fallback — job-def components cannot ig/ref `::jobs/defs` (wiring cycle). Submit from RPC code passes its RPC cfg (it carries `::jobs/defs` via ig/ref).
+
+- `::jobs/handler` is `(fn [context params] ...)`. It is NOT given the runner cfg: a job-def closes over its own dependencies, and handing it the runner cfg would let a handler reach the pool or any other component. A wrapper passes the context to its implementation only when the implementation takes it; most do not, and their implementation stays `[cfg params]`.
+- The context is built by `jobs/make-context` and has exactly five keys: `id`, `name`, `label`, `resource-id`, `profile-id`, behind a closed schema. `profile-id` is the owner of a user-facing job and nil for an internal one, and it is there so a handler can revalidate permissions and check who owns its resource without a second query. `queue` is absent on purpose (a routing detail of the dispatcher, not a property of the job) and so are `retry-num`/`max-retries`/`attempt` (retry policy belongs to the runner, and an attempt number cannot be durable because the `noop` strategy re-runs without incrementing the counter).
+- `jobs/invoke` is the in-process escape hatch: it delivers a nil context without a row, validates and delivers `::jobs/context` when given, and keeps `::jobs/job-id` independent. `::jobs/context` and `::jobs/job-id` are cfg options, not params.
+- Business params live in the `params` column (plain JSON, decoded with the job-def decoder). `submit` takes optional `::jobs/profile-id`, `::jobs/resource-id` (both validated as UUIDs) and `::jobs/expires-at` (an instant: the retention the jobs GC sweeps by), all stored as columns; nothing is inferred from `params`. The binfile creation commands are the callers that pass a profile-id and a resource.
+- A job-def may carry optional metadata keys read from the registry by its consumers, never by the substrate: `::jobs/family` (e.g. `:export`) groups the jobs of a family for quotes, `::jobs/resource-role` (`:input` or `:output`) says whether the job's `resource_id` is what it consumes or what it produces, and `::jobs/queue-name` names the queue the creation command routes the job to (`submit-job` falls back to `:binfile` when the def does not set it).
+- Job resources (`app.jobs.storage`) are the objects a job owns through `job.resource_id`: they live in the `job-resource` bucket, are written with the profile of the job as owner (only that profile can read them back from `/assets/by-id`) and are never deduplicated. They carry no expiry of their own, because `expired-at` is stored as `deleted_at` and an object with it is unreadable from birth: the retention of the artifact is the retention of its job row. The writer marks the object as touched right away, so an artifact that no row references (a crash between the upload and the completion) is reclaimed by `storage-gc-touched`, and a row that does reference it freezes it until the jobs GC deletes the row and touches the object.
+- The binfile jobs (`:export-binfile` and `:import-binfile`) run the core of `bf.v1`/`bf.v3` through `app.binfile.jobs`, which turns the `:progress` taps of the core into `progress` events of the job with `jobs/heartbeat` (the `modified_at` touch at most once per second, progress events at up to ~200ms). Each event is a self-contained milestone: a `:stage` keyword of one vocabulary shared by the two formats (the same work has a different name in v1 and v3) and an optional `:counters` map, with the counter of the activity named after its own stage and the outer scopes in course as context, which the tap carries itself (the reader puts `::outer-counters` in every tap inside them — today only `:files`, extendable to `:teams`/`:projects` for multi-scope exports — so the adapter remembers nothing between taps and a tap that happens outside any scope has no context); the ids and names the taps carry never reach the event. Every tap is reported, so a tap with no units to count still keeps the job alive, and a milestone the throttle drops loses nothing because the next one carries the same state. The adapter checks that the job is still active before the run, and the beat raises an `:interrupt` right there on the run thread when the job is no longer active, so the work stops at its next event point inside the transaction of its handler, which rolls back. A run whose last tap predates the interruption is discarded by the check once it returns. The adapter passes the job id from the handler context. Their job-def components carry pool, storage, metrics and msgbus, because a job with a `profile_id` cannot store an event without a msgbus on its cfg. Both readers answer with the same shape (`:file-ids` and, in version 3, the resolution of the libraries), because the version 1 reader returns a bare set of file ids.
+- Both binfile handlers revalidate their permission at run time, not only when the job is created, because it can be revoked while the job waits: the export checks the read permission of every file before reading anything and stores the package in `job-resource` as the resource of the job (answering with the completion envelope), and the import checks the edition permission of the destination and runs the core inside a transaction, which is what makes a cancelled import roll back instead of leaving half of a file. The import releases its package (marks it as deleted and drops the reference from the job row in the same transaction) and deletes its temporary copy once the job is terminal, whatever the outcome. Both handlers check their own result against a schema before returning it (the import job-def declares that same schema as its result schema, which is what types the resolution when the row is read).
+- The v3 import only materializes a library relation when both ids belong to the package and differ; a relation that points outside the package or a self link is ignored with a warning. Migration `0158-del-forbidden-file-library-rels` deletes the legacy cross-team and self rows from `file_library_rel` and their `file_library_sync` rows, sync rows first.
+- The domain cfg of a binfile request lives in `app.binfile.common` (`export-cfg`, `import-cfg`), so the legacy RPC and the jobs freeze the same ids, resolve the same team features and apply the same anti-zip-bomb limits. `resolve-export-type` is the translation the legacy RPC needs for its deprecated boolean flags; the creation command asks the caller for the type instead of guessing a default.
+- Two retention knobs, easy to confuse: `:jobs-retention` sweeps terminal internal rows (the ones without a profile), while `:jobs-user-ttl` is the default expiry a user-facing job gets when it is created; the jobs GC deletes the row once `expires_at` passes and touches its resource. The heavy work of a user job runs on its own `:binfile` queue (`:worker-binfile-parallelism`, default 1) so it does not block the default runner.
+- The quote of a family of jobs (`::export-jobs-per-profile`, `::import-jobs-per-profile`) counts the jobs of the profile that are still on their way (`new`/`scheduled`/`running`/`retry`) whose name the registry gives for that family, so a new job of the family is covered without touching the quote.
+- A user-facing job is created through `app.rpc.commands.jobs`. One command per job type: `create-export-binfile-job` and `create-import-binfile-job` name the job they create and their body carries only `:params`, whose schema the command imports from the job namespace (`::sm/params`), so the RPC validates the business params up front, before the command runs; the job-def is still the place that owns them and validates them again at run time. The commands check the permission and the quote of the family before the expensive part, submit with `max-retries 0`, an expiry from `:jobs-user-ttl` and the queue the job-def routes to through `::jobs/queue-name` (`:binfile` when the def says nothing; read by the command, never by the substrate), and answer with a summary of the row. The import command has its own creation schema (`import-binfile/schema:create-params`, version optional) because the request is not the stored params: the command fills the version (from the caller or the package header) and the manifest metadata kept for audit. It assembles the upload, reads the manifest of a version 3 package for the audit (never its content) and stores the package in `job-resource` as the resource of the job, releasing it when the job never comes to own it.
+- `get-job` is the read surface a profile has over its own jobs: it answers with the same summary the creation commands return plus the `result` of a finished job or its public error (through `decode-job-error`, so the keywords come back), and a job of another profile is not distinguishable from one that does not exist: both answer `not-found`. `decode-row` decodes `params`, `result` and `error`, and the result is then decoded with the decoder the substrate compiled from the result schema the job-def the row names declares (`decode-result`; a def without one, or a row whose job-def is gone, keeps the plain JSON the row holds: a uuid as the text it is on disk, a uuid used as a map key as a keyword). The event log still has no read surface: a client that follows a job takes its progress from the websocket.
+- `heartbeat` uses named options `[cfg & {:keys [job-id progress] :as options}]` and returns the number of durable writes, so 0 means there was nothing to write: no job context, or the throttle did not allow it. A write that finds the job no longer active (terminal or gone) raises an `:interrupt` (`code :job-interrupted`, carrying the job id and the status it found) through the shared `jobs/check-active` instead of counting zero, so a task in course stops at its next beat; the interrupt is internal and never reaches HTTP (the management progress report answers `:skip` for it). The job id falls back to `::jobs/job-id` on the cfg and then to the runner-bound `*job-id*`. The `modified_at` touch is throttled to ~1s and the progress event to ~200ms, so the check fires at most once per second with no extra query (the read of the row happens only when a write already found nothing). A progress report is a milestone (`schema:progress`): a required `:stage` keyword plus an optional `:counters` map of `scope -> {:current, :total?}`, and no other key. Counters are non-negative, a `total` is positive and never lower than its `current`, and stage and counter keys are names of at most 64 characters. A milestone without counters is valid (a point of the run with no units to count), the stage vocabulary belongs to the worker and not to the substrate, and a client must degrade for a stage it does not know. Read a stored payload back with `jobs/decode-progress`: the database holds the keywords as strings. The 1s beat (down from 60s, so a cancelled heavy job is seen as gone at its next beat) costs one `modified_at` touch per beating job per second: cheap while the only heavy beater runs alone on its queue, revisit if concurrent beating jobs ever grow.
+- `job.error` conforms to `app.jobs/schema:job-error` (`type` and `code` keywords, `hint` text, extra details allowed). Read it back with `jobs/decode-job-error`; a plain `db/decode-json-pgobject` leaves the keywords as strings. `complete` takes named options and an optional `resource-id`, which is only ever set, never replaced: it reads the row locked inside its own transaction and applies the rules in a fixed order, skipping a job that is no longer completable before refusing one for its resource, so the answer is a function of the row and not of the order two callers arrive in. It checks the result against the schema the job-def of the row declares before it writes (a violation raises `:invalid-result` and nothing is stored), and only for a transition that is going to write: a terminal job answers `:skip` without validating. `complete` is the single door of a result, so the check covers the runner and the external worker that completes a job through the management RPC (`::complete-job`), which stores a result the substrate would otherwise take blind.
+
+For worker dispatch, cron, retry semantics (`ex/raise :type ::wrk/retry` with `:delay`/`:strategy`), deduplication, job events, and queue internals: `mem:backend/subtleties`.
+
+## Jobs metrics
+
+- `app.jobs.metrics` owns the jobs metric names, bounded labels and the periodic backlog sampler.
+- Metrics use only `name`, `queue`, `outcome`, `reason`, `stage` and `kind` labels. Job IDs, profile IDs, params, reply bodies and exception text must never be labels. Every label goes as-is (lowercased where case varies): values come from fixed call sites, so a new value in Prometheus means new code, not noise. Only a missing value falls back (`other`, or `failed` for outcomes).
+- The canonical job name is a keyword (`app.jobs/schema:job-name`): `submit` and `invoke` reject strings. The row stores text, so `jobs/get-job-def` still coerces (it is the runner's boundary), cron entries carry the job under `:job` with `d/name` only at the SQL/log boundary, and test helpers keyword-ize row names before invoking.
+- The durable lifecycle metrics are submitted, dispatched, completed, retries, orphaned and rescheduled counters; queue-wait, execution and total-time histograms; dispatcher and backlog state; GC, cron and ephemeral request metrics.
+- `app.metrics/run!` is safe for recording failures, but metrics components still require a valid metrics instance. Durable submit call sites carry `::mtx/metrics` through RPC or job-def configuration. The jobs substrate takes the **cfg**, not the instance, and resolves it with `app.metrics/instance`, which raises `:missing-metrics` when absent: `app.jobs`, `app.jobs.metrics`, the runner, the dispatcher, cron and the jobs GC all follow that. `app.http`, `app.rpc.climit` and `app.storage.s3.metrics` still take the instance and were left out of scope on purpose, so do not read the codebase as uniform. The six job writers resolve at the top of their body, before writing, because their metric lands in a `db/after-commit` callback that swallows exceptions. The GC, the dispatcher and cron resolve inside that callback instead, so their guarantee rests on the component schema instead. A job that calls `heartbeat` from a component cfg must carry `::mtx/metrics` on it, and the event helper records the counter from the cfg the transaction hands it.
+- SQL-backed job metrics are registered with `app.db/after-commit` from INSIDE the transaction that changed the row, so a rollback cannot inflate submitted, terminal, retry, event or dispatcher counters. Every lifecycle writer owns that transaction (`app.jobs/own-transaction` drops `::db/conn`), and `db/transact!` gives a transaction that joins no other its own callback context, so the write and its metric commit or vanish together. `submit` is the exception and joins the caller on purpose: creating a job is part of the caller's business work.
+- `penpot_jobs_events_total` counts stored job events with no labels: the interesting split is `kind`, and grouping a growing append-only table to build a metric is not worth the query on the write path.
+- The backlog sampler is `::jobs-metrics/sampler`; it runs every 30 seconds on worker-enabled, writable systems, groups by status and publishes the oldest pending age, and does not start on a read-only database.
+- `penpot_tasks_timing` remains exported for compatibility while the jobs-specific histograms are adopted.
 
-For worker dispatch, cron, retry semantics, deduplication, and queue internals: `mem:backend/subtleties`.
 
 ## REPL
 
@@ -101,9 +136,7 @@ IMPORTANT: all CLI commands must be executed from the `backend/` subdirectory.
 * **Linting:** `pnpm run lint:clj`.
 * **Formatting:** `pnpm run check-fmt:clj` to check, `pnpm run fmt:clj` to fix. After running `fmt:clj`, `check-fmt:clj` is redundant. Avoid unrelated whitespace diffs.
 
-**Before linting:** if delimiter errors are suspected (after LLM edits), run
-`scripts/paren-repair` on the affected files first. Delimiter errors produce
-misleading linter/compiler output. See `mem:scripts/paren-repair`.
+**Before linting:** if delimiter errors are suspected (after LLM edits), run `scripts/paren-repair` on the affected files first. Delimiter errors produce misleading linter/compiler output. See `mem:scripts/paren-repair`.
 
 ## Testing
 

.serena/memories/backend/storage.md

@@ -25,12 +25,19 @@
 - Existing object values can remain readable until physical deletion.
 - `:expired-at` blocks reads after the expiration time.
 - `del-object!` sets `deleted_at` on live rows only (`deleted_at IS NULL`): a repeated call returns `false`. It does not remove backend content.
-- `storage-gc-deleted` removes the database row and backend content after the deletion delay, processing the pending objects in chunks of 25 with keyset pagination (`(deleted_at, id)` cursor, matching the `storage_object(deleted_at, id)` index). Only `status = 'valid'` objects are read: pending ones (still being written, see the status lifecycle) stay untouched even when already marked as deleted. Each chunk is one short read-write transaction: `SELECT ... FOR UPDATE SKIP LOCKED` (so two concurrent runs never take the same chunk; no separate read transaction exists and the run does not block the table vacuum) followed by the database delete and the backend delete grouped by backend. The cursor advances to the `(deleted_at, id)` of the last row read regardless of the deletion result, so a chunk that fails is skipped and not visited again in the same run: the error aborts the transaction (nothing of the chunk is deleted, locks are released), the handler catches it inside the transaction function and rolls it back explicitly — the commit that `tx-run!` performs afterwards is a JDBC-driver no-op on the already-rolled-back connection — and the loop continues with the returned rows. Accepted quirks, both benign and self-healing: (1) with rows of several backends in one chunk, a failure after one group's physical delete leaves those rows pending with already-removed content — pending rows are excluded from reads and the next run finishes them, since the physical delete is idempotent (S3 ignores missing keys, FS uses `deleteIfExists`); (2) rows locked by a concurrent run are skipped via `SKIP LOCKED` and deferred to the next scheduled run, and the `deleted_at <= now` cutoff is sampled per chunk. It emits the `penpot_storage_gc_deleted_objects_total{backend}` counter (one increment per deleted backend group, reported only after the chunk commits, so rolled-back chunks are never counted twice); run duration is covered by the generic `penpot_tasks_timing{name="storage-gc-deleted"}` histogram and chunk failures by the log-derived `penpot_log_events_total`. Every 5000 deleted objects the run pauses 5 seconds (throttle between chunk transactions, never holding one open).
+- `storage-gc-deleted` removes the database row and backend content after the deletion delay, processing the pending objects in chunks of 25 with keyset pagination (`(deleted_at, id)` cursor, matching the `storage_object(deleted_at, id)` index). Only `status = 'valid'` objects are read: pending ones (still being written, see the status lifecycle) stay untouched even when already marked as deleted. Each chunk is one short read-write transaction: `SELECT ... FOR UPDATE SKIP LOCKED` (so two concurrent runs never take the same chunk; no separate read transaction exists and the run does not block the table vacuum) followed by the database delete and the backend delete grouped by backend. The cursor advances to the `(deleted_at, id)` of the last row read regardless of the deletion result, so a chunk that fails is skipped and not visited again in the same run: the error aborts the transaction (nothing of the chunk is deleted, locks are released), the handler catches it inside the transaction function and rolls it back explicitly — the commit that `tx-run!` performs afterwards is a JDBC-driver no-op on the already-rolled-back connection — and the loop continues with the returned rows. Accepted quirks, both benign and self-healing: (1) with rows of several backends in one chunk, a failure after one group's physical delete leaves those rows pending with already-removed content — pending rows are excluded from reads and the next run finishes them, since the physical delete is idempotent (S3 ignores missing keys, FS uses `deleteIfExists`); (2) rows locked by a concurrent run are skipped via `SKIP LOCKED` and deferred to the next scheduled run, and the `deleted_at <= now` cutoff is sampled per chunk. It emits the `penpot_storage_gc_deleted_objects_total{backend}` counter (one increment per deleted backend group, reported only after the chunk commits, so rolled-back chunks are never counted twice); run duration is covered by the generic `penpot_tasks_timing{name="storage-gc-deleted"}` histogram and chunk failures by the log-derived `penpot_log_events_total`. Every 5000 deleted objects the run pauses 5 seconds (throttle between chunk transactions, never holding one open). It runs as the `storage-gc-deleted` job (`::job-def`) and beats `jobs/heartbeat` after each committed chunk, so a long sweep keeps its lease and never holds it during a chunk transaction.
 - `storage-gc-touched` finds references before it sets `deleted_at`.
 - Orphan watch: `SELECT COALESCE(metadata->>'bucket', metadata->>'~:bucket') AS bucket, count(*) FROM storage_object so WHERE touched_at IS NULL AND deleted_at IS NULL AND status = 'valid' AND NOT EXISTS (SELECT 1 FROM upload_session_chunk usc WHERE usc.object_id = so.id) GROUP BY 1;` Only the `upload-session` row of that output is a leak signal (its rows always have a mapping or a pending touch/deleted mark); other buckets use domain-table references, not this mapping.
 - `objects-gc` removes deleted domain rows and touches their storage object IDs. Each proc streams its pending keys with `db/plan` inside a read-only transaction and deletes them in chunks, each chunk in its own transaction with `FOR UPDATE SKIP LOCKED`: a chunk that fails is logged and skipped instead of aborting the run.
 - Use `::db/reuse-conn true` with `sto/resolve` inside a database transaction.
 
+## GC Cost Model
+
+- `sto/touch-object!` is one `UPDATE storage_object SET touched_at`: DB-only, never touches S3/FS backend content.
+- Physical FK cascades are avoided by convention; domain deletion is logical (`deleted_at`) everywhere. GC removes small objects first, then parents, then grandparents (see the call order in `app.tasks.objects-gc/execute-objects-gc`). Per-row delete cost is small and bounded.
+- `objects-gc` sweeps in chunks of 100 (`::chunk-size`, overridable per job params); `storage-gc-deleted` in chunks of 25. `objects-gc` (and the other `db/plan` sweeps) stream the pending keys in a read-only transaction of their own and delete every chunk in a transaction of its own: the cursor fetch and the deletes are never the same transaction. `storage-gc-deleted` has no separate cursor transaction: it reads, locks and deletes each chunk in the same short transaction (see the keyset pagination above).
+- Consequence: per-chunk cost is DB-only and bounded, so these sweeps cannot outrun the 30-minute jobs lease in practice (would need millions of pending rows in one proc). Per-chunk `heartbeat` where present is belt-and-braces, not load-bearing.
+
 ## Connection Reuse Details
 
 ### `app.storage/resolve` patterns:
@@ -71,6 +78,7 @@ Since `put-object!` uses backend-specific operations (`impl/resolve-backend` + `
 - Objects can therefore share content across users and files within one bucket.
 - Deleted objects are not reused.
 - `tempfile` objects never use deduplication, even when the caller requests it.
+- `job-resource` objects never use deduplication either, whatever the caller asks: the object belongs to one profile, and a shared blob would serve the artifact of one user to the job of another. `put-object!` excludes the bucket from the dedupable set, so the invariant does not depend on call sites.
 - Use `sto/wrap-with-hash` when the caller already calculated the content hash.
 
 ## Bucket Rules
@@ -88,12 +96,13 @@ Since `put-object!` uses backend-specific operations (`impl/resolve-backend` + `
 | `file-data` | Encoded file data when `file-data-backend` is `storage`. Reference metadata has `storage-ref-id`, `file-id`, and the `file_data` row ID. | Yes | Authentication required | Reference scan. |
 | `file-data-fragment` | Compatibility value for file-data fragments. The current backend has no dedicated producer for this bucket. | No current write semantics | Public | No touched-object collector case. |
 | `file-change` | Compatibility value for file changes. Current snapshots store data in `file_data`, not this bucket. | No current write semantics | Authentication required | No touched-object collector case. |
+| `job-resource` | Storage objects owned by job rows (`job.resource_id`), written by `app.jobs.storage`. | No | Authentication required, owner-scoped | jobs-GC touch → reference scan. |
 
-- The valid bucket set lives in `app.storage/valid-buckets`.
+- The valid bucket set is `app.storage.schema/metadata-buckets` (derived from `bucket-requirements`, where a new bucket and its required metadata keys are one entry); `app.storage/valid-buckets` aliases it.
 - `file-media-object` is the default bucket for old rows without bucket metadata.
 - Do not assign a new bucket without adding its access and cleanup behavior.
 - The touched-object collector raises an internal error for an unknown bucket.
-- It supports `file-media-object`, `team-font-variant`, `file-object-thumbnail`, `file-thumbnail`, `profile`, `file-data`, `tempfile`, `upload-session`, and `organization`.
+- It supports `file-media-object`, `team-font-variant`, `file-object-thumbnail`, `file-thumbnail`, `profile`, `file-data`, `tempfile`, `upload-session`, `job-resource`, and `organization`.
 - It does not support `file-data-fragment` or `file-change`.
 
 ## Access Rules
@@ -101,6 +110,7 @@ Since `put-object!` uses backend-specific operations (`impl/resolve-backend` + `
 - `app.http.assets` decides direct object authentication from the bucket.
 - Public buckets are `file-media-object`, `file-object-thumbnail`, `team-font-variant`, `file-data-fragment`, and `organization`.
 - Other valid buckets require a session or access-token profile ID.
+- `tempfile` and `job-resource` are owner-scoped: the request must come from the profile stored in the object metadata, and a mismatch answers 404 (counted as `unauthorized`). A missing owner is tolerated only in `tempfile`, where legacy objects predate the metadata; `job-resource` always carries one, so an object without it is never served.
 - File-media routes also require file read permission.
 - Non-public direct responses set `content-disposition: attachment`.
 - FS responses use `x-accel-redirect` for the configured asset path.
@@ -115,6 +125,11 @@ Since `put-object!` uses backend-specific operations (`impl/resolve-backend` + `
 - The `file_data.metadata.storage-ref-id` value points to the storage object.
 - `fdata/upsert!` touches a storage object from incoming metadata before it stores the new row.
 - File snapshots use `file_data` for snapshot data and `file_change` for snapshot metadata.
+- Tiered offload (`:tiered-file-data-storage` flag + `db` backend): `file-gc` submits an `:offload-file-data` job (priority 10, delay 1000, same params) only when it actually processed the file; the handler takes the `file` row with `FOR UPDATE SKIP LOCKED` and skips (`{:skipped 1}`, no error) when the row is locked or gone, then moves every live `file_data` row of that file with `backend = 'db'` to `backend = 'storage'` via `fdata/upsert!` (one `put-object!` in bucket `file-data` per row, `data` becomes nil) inside a single all-or-nothing transaction per file. The skip only serializes because every `file_data` writer holds at least `FOR SHARE` on the `file` row. A storage failure aborts only that file and bubbles for a runner retry; orphans go to `storage-gc-touched`. A second run is a no-op because the query only matches `backend = 'db'`.
+- Offload pacing: none by deliberate decision. Sleeps only make sense when batching multiple transactions; the per-file transaction is short by construction and the scheduler run takes minutes per production metrics, so neither loop sleeps. `jobs/heartbeat` (pool-based liveness, ~1 write/s) stays on both paths.
+- Offload metrics (bounded labels only, never ids or error text): `penpot_file_data_offload_rows_total{type}` and `penpot_file_data_offload_bytes_total{type}` (`type`: `main`/`snapshot`/`fragment`, successful runs only) plus `penpot_file_data_offload_timing_ms{result}` (`result`: `ok`/`skip`/`error`). Helpers live in `app.jobs.metrics`; the job-def requires `::mtx/metrics`.
+- Writes always resolve the backend through `fdata/default-backend` (`cf/get :file-data-backend`), so post-offload edits land back in hot (`db`) storage without extra code; reads stay transparent through `fdata/resolve-file-data` + `decode-file-data` + `realize`.
+- Deleting an offloaded row touches its storage object first (`objects-gc/delete-file-data-chunk` via `fdata/process-metadata`), so `storage-gc-touched` reclaims the blob with the usual delay; restoring a file unmarks the referenced objects.
 
 ## Metrics
 

.serena/memories/backend/subtleties.md

@@ -9,6 +9,7 @@
 - Params with `::sm/params` are decoded/conformed through the JSON transformer and successful IObj results get `:encode/json` metadata. Legacy spec conforming only applies when no Malli params schema exists. Client params are stripped of qualified keys (`d/without-qualified`) before merging with the server auth context, so request bodies cannot override `::profile-id`, `::auth-type`, or `::token-perms`.
 - Params schemas are open by default, so undeclared client keys reach the handler unless the map is `:closed true`. Creation commands (`create-file`, `create-project`, `create-team`, `create-team-with-invitations`, `upload-file-media-object`, `create-file-media-object-from-url`, `assemble-file-media-object`) use closed schemas: a client-provided `:id` fails with `:params-validation`. Their internal creation functions still accept an optional explicit `:id` for imports, duplicates and deterministic test fixtures.
 - Nil RPC bodies become HTTP 204 unless explicit status metadata is present. Stream bodies default to `application/octet-stream` when no content type is set.
+- The upload schema (`media.v/schema:upload`) takes a `:path` from the caller, and a Transit body can tag any string as a `path`, so a command that reads such an upload reads a file the client named. `create-import-binfile-job` avoids that boundary by accepting only a chunked upload (`:upload-id`); the commands that still take a plain file (the legacy `import-binfile`, the media and font uploads) trust it, and the guard belongs where uploads are accepted, not in each command.
 
 ## Concurrency limiter (CLIMIT)
 
@@ -26,27 +27,36 @@
 
 - Most `app.db` helpers accept a pool, connection, or map containing `::db/pool` / `::db/conn`; preserve that convention in shared code.
 - `db/tx-run!` uses `next.jdbc.transaction/*nested-tx* :ignore`: nested transaction calls reuse the outer transaction, not a savepoint. Use explicit savepoints when nested rollback semantics matter.
+- Whether a call is nested is decided by the CONNECTION, not by an ambient flag: `db/transact!` binds `*after-commit-conn*` around the body, and a transactable that is not identical to that connection is an independent unit of work that owns its own after-commit context. A `tx-run!` that passes a pool while a transaction is open therefore commits on its own and drains its own callbacks, instead of leaving them in a context that may roll back.
+- `after-commit` callbacks run on commit only. A transaction marked `::db/rollback` (used by `srepl` and `app.tasks/offload-file-data`) commits nothing, so it runs none of them.
 - `db/run!` opens/reuses one connection but does not create a transaction.
-- `db/tjson` is Transit JSON for jsonb storage; `db/json` is plain JSON. Worker task props use Transit and are decoded with `decode-transit-pgobject`.
+- `db/tjson` is Transit JSON for jsonb storage; `db/json` is plain JSON. Job params (`job.params`) are plain JSON, decoded with the job-def decoder (`app.jobs/decode-params`); only the legacy `task` props were Transit (`decode-transit-pgobject`). Job results (`job.result`) are plain JSON too: they are read with the result decoder of the job-def (`app.jobs/decode-result`), and a result map keyed by uuid must declare its schema (`[:map-of ::sm/uuid ...]`, the import resolution does) or every key comes back as a keyword and every uuid as the text it is on disk.
 - Advisory transaction locks accept UUIDs or ints. UUID locks are hashed using a zero-UUID seeded siphash.
 
 ## Workers and cron
 
-- Task queues are tenant-prefixed. Submit dedupe only removes not-yet-due `new` tasks with the same name/queue/label; it does not dedupe due, scheduled, retry, running, or completed work.
-- The dispatcher selects `new`/`retry` tasks with `FOR UPDATE SKIP LOCKED`, marks them `scheduled`, and publishes Redis payload `[id scheduled-at]`. The runner skips Redis messages whose scheduled timestamp no longer matches DB state.
-- Lost `scheduled` tasks are rescheduled after about 5 minutes; `running` tasks older than about 24 hours are marked failed as orphans.
-- A task handler that is missing or returns an invalid result currently defaults to completed after warning. Throwing with `ex-data :type ::retry` controls retry behavior; `:strategy ::noop` retries without incrementing retry count.
-- Cron jobs lock their `scheduled_task` row with `FOR UPDATE SKIP LOCKED`, disable statement/idle-in-transaction timeouts locally, and reschedule themselves in `finally` unless interrupted. Worker, dispatcher, and cron components do not start when the DB pool is read-only.
+- `job` carries `tenant` (the instance the job belongs to) and a bare `queue`. Several instances share one database and the tenant is what keeps an instance from claiming another environment's jobs, so every query that acts on one instance's work filters on `tenant`: dispatch claim, orphan sweep, lost-job reschedule, cron no-overlap count, submit dedupe. The `job__dispatcher__idx` leads with `tenant` (equality) then `(priority DESC, scheduled_at)`; `queue` does not lead because the dispatcher claims every queue of its tenant. The `<tenant>:<queue>` prefix survives only in the Redis key, composed once by `app.worker/queue-key`.
+- Submit dedupe only removes not-yet-due `new` jobs of the same tenant with the same name/queue/label; it does not dedupe due, scheduled, retry, running, or completed work. The `::jobs/skip-if-active` submit option goes further: it skips the insert (returning `nil`, recording `penpot_jobs_skipped_total`) when a live job (`new`/`scheduled`/`running`/`retry`) already exists for the same tenant/name/queue/label, requires an explicit non-blank label, and carries the same best-effort cross-backend race caveat as dedupe.
+- Jobs metrics rule (one event, one family): counters exist only for events without latency; when an event has a histogram, its `_count` is the counter (no twin counter). `dispatched` (pushes) vs `queue-wait` (claims) are deliberately different events — do not merge them.
+- The dispatcher selects `new`/`retry` jobs with `FOR UPDATE SKIP LOCKED`, marks them `scheduled`, and publishes the JSON payload `[id scheduled-at]` to the `penpot.worker.queue:<tenant>:<queue>` Redis list. The runner skips Redis messages whose scheduled timestamp no longer matches DB state.
+- Lost `scheduled` jobs are rescheduled after 5 minutes; `running` jobs untouched longer than `:jobs-lease` (default 30 min) are marked `aborted` with `app.jobs/orphan-error` (`type :internal`, `code :orphan`). `aborted` is a system-side terminal state like `cancelled` (never retried, swept by retention like other terminals, reported with an error log); long jobs must `heartbeat` per chunk so a healthy backlog is never mistaken for a dead process. The `:progress` report is optional and only feeds the durable log. A beat whose write finds the job no longer active raises an `:interrupt` (`code :job-interrupted`) instead of counting zero: it is the internal stop-the-work signal, not an error, so neither the listener nor the runner logs it as one.
+- An orphan gets its `end` event (`outcome "aborted"`) from the sweep itself: one bulk `INSERT ... SELECT` in the same transaction as the bulk `UPDATE`, never one insert per orphan. No msgbus notification goes out for it; orphans alert through the error log and the orphaned counter.
+- `job_event` is append-only and deleted by cascade with the job. `progress` is a row there, not a column: no Redis and no mutable column to keep in sync.
+- `job` is deletion-protected at the SQL level (`deletion_protection__tgr`): a stray `DELETE FROM job` raises instead of dropping the row and its `job_event` cascade without touching the referenced `storage_object`. The only intended deleters, the jobs GC and the submit dedupe, disable the guard with `SET LOCAL rules.deletion_protection TO off` inside their transaction; tests use `th/db-force-delete`.
+- A missing job-def raises (`:no-job-definition`) instead of completing. Throwing with `ex-data :type ::retry` still controls retry behavior; `:strategy ::noop` retries without incrementing retry count.
+- Cron entries claim their `scheduled_task` row with `FOR UPDATE SKIP LOCKED`, disable statement/idle-in-transaction timeouts locally, submit one `job` row per entry when no active instance exists (no-overlap), and reschedule themselves in `finally` unless interrupted. Worker, dispatcher, and cron components do not start when the DB pool is read-only.
+- Integrant params schemas name injected components as bare `::ns/key` entries (pool, metrics, msgbus, job defs), and a bare entry is a malli registry lookup, not `any?`: the owning namespace must `sm/register!` it at load, or the backend crashes at boot with `:malli.core/invalid-schema` instead of asserting. Unit tests never build those keys, so `backend-tests.worker-test` compiles the three worker schemas to catch it.
 
 ## Config and HTTP/session middleware
 
 - `app.config/config` and `flags` are dynamic `defonce` vars populated from `PENPOT_*` env vars through the shared schema string transformer. Tests and tooling can bind them.
+- Duration config values are parsed as ISO-8601 with the `PT` prefix, so a day is written in hours (`:jobs-retention` and `:jobs-user-ttl` default to a week, and an operator writes `168h`): `7d` does not parse, and the value fails validation at boot instead of being silently ignored.
 - `parse-flags` automatically adds `:disable-secure-session-cookies` when `public-uri` is plain HTTP and not localhost. This changes cookie defaults without an explicit env flag.
 - The backend sets Clojure `*assert*` globally from the `:backend-asserts` feature flag. Assertion-dependent checks can therefore differ by runtime flags.
 - Request body parsing is mostly POST-oriented and supports Transit JSON plus plain JSON. Plain JSON request keys are kebab-decoded before being merged into `:params`. A body the parser cannot read is a client error, classified at the parse site and never by exception class (`clojure.data.json` signals its syntax errors with a bare `java.lang.Exception`): 400 `:validation`/`:malformed-json`, one `warn` line (`malformed request body`) carrying path, peer address and content-type instead of a stack trace, and `penpot_http_malformed_requests_total`. `RequestTooBigException` keeps its 413.
 - Response formatting negotiates with `Accept` or `_fmt=json`. Transit is the default for collection/boolean bodies; JSON encoding has special pointer-map handling.
 - Auth prefers the session cookie token before the `Authorization` header. Headers may be `Token` or `Bearer`. Only `kid=1`/`ver=1` tokens are decoded as session tokens; anything else is left unauthenticated (legacy v1 tokens were removed).
-- Shared-key auth requires `x-shared-key` as `<key-id> <key>` and stores the lowercased key id on the request. If no shared keys are configured it always rejects.
+- Shared-key auth requires `x-shared-key` as `<key-id> <key>` and stores the lowercased key id on the request. If no shared keys are configured it always rejects. Management methods declare the key ids they accept with `::rpc/allowed-shared-keys`; a valid key outside that set answers 403 `:shared-key-not-allowed`, and a method without the metadata accepts any configured key.
 - Session management uses DB storage unless the DB pool is read-only, then falls back to the in-memory manager. Sessions use only the v2 UUID model (`http_session_v2`); legacy string ids were removed.
 - Session cookies are renewed when `modified-at` is older than the 6h renewal interval. SameSite is `none` for CORS, otherwise strict/lax based on config. Session lifetime config and GC: `mem:backend/session-expiration`.
 
@@ -60,7 +70,9 @@
 
 ## Metrics recording
 
-- `app.metrics/run!` is safe by default: a recording failure never throws (a metrics bug must not change the behavior of the measured operation). The first failure per metric id logs at `warn`, later ones at `debug`. The `instance` precondition is a plain assert (the backend enables `:backend-asserts`), and the collector lookup sits outside the recording guard, so a missing instance fails hard even when asserts are disabled. `::mtx/metrics` is required by the storage, s3-backend, and db-pool schemas; `app.db` wires the prometheus `MetricsTrackerFactory` unconditionally. Storage-specific metric contracts: `mem:backend/storage`.
+- `app.metrics/run!` is safe by default: a recording failure never throws (a metrics bug must not change the behavior of the measured operation). The first failure per metric id logs at `warn`, later ones at `debug`. The `instance` precondition is a plain assert (the backend enables `:backend-asserts`), and the collector lookup sits outside the recording guard, so a missing instance fails hard even when asserts are disabled. `::mtx/metrics` is required by the storage, s3-backend, db-pool and jobs APIs; jobs helpers never silently skip a missing metrics instance. The resolver is `app.metrics/instance` and it raises, unlike `run!`.
+- `app.msgbus` (`pub!`, `sub!`, `purge!`) keeps taking the **instance** on purpose: msgbus is what produces it, and `app.rpc.notifications` already shows the caller passing a cfg. `app.jobs` resolves `::mbus/msgbus` from the cfg internally, and `insert-event` keeps its explicit pre-insert check so a profile job never loses its notification.
+- `app.db/after-commit` is the boundary for SQL-backed metrics: callbacks are drained by the commit of the transaction that owns the connection they were registered on, and discarded when it rolls back. Because nesting follows the connection, that transaction is the outermost one only for a real nested call.
 - `penpot_http_malformed_requests_total` (counter, no labels) counts requests whose body the parser could not read. `app.http.middleware/wrap-parse-request` emits it where it classifies the error, so the instance travels as a middleware argument (`[mw/parse-request (::mtx/metrics cfg)]`) and each service declares the definition with its own prefix (nexus: `nexus_http_malformed_requests_total`). The per-endpoint request counters of nexus do not count these requests: they sit inside the parse middleware, so the failure happens before them.
 
 ## Log-derived metrics
@@ -80,7 +92,15 @@
 
 - File data backends are `legacy-db`, `db`, and `storage`. The storage backend keeps encoded file data in storage bucket `file-data`; the DB row stores metadata with `storage-ref-id` and nil data.
 - `fdata/upsert!` touches any storage object referenced by incoming metadata before storing the new row/blob.
-- Pointer-map fragments are persisted separately as type `fragment`, and only modified pointer maps are written.
+- Pointer-map fragments are persisted separately as type `fragment`, and only modified pointer maps are written. A pointer keeps its id while nothing changes; content (`assoc`/`without`) and metadata (`with-meta`) changes produce a fresh id, so a fragment row is never rewritten after it is written. Re-assoc with an identical value returns the same map in array and hash maps alike, so a save keeps the fragment id of every page it does not change.
+- `enable-objects-map` skips unloaded pointer pages: their fragment was already wrapped when it was written, and loading it only to wrap it again costs one page read per untouched page on every save. A page with a plain fragment is wrapped when it is first loaded or edited; a data migration that enables `fdata/objects-map` must rewrite the data itself.
+- Snapshots share fragment rows with the live file (they are stored with pointers, not materialized). `file-gc` marks every fragment referenced neither by the current main file nor by an unpurged snapshot; the file is read unrealized and unchanged pages keep their ids via reattach, so a pass over an unchanged file writes no new fragments.
+- A save attaches its automatic snapshot before applying pending file migrations: the snapshot keeps the pre-migration pointers, data and migration set, and restoring it leaves the next read to migrate the file. This also holds for files without the feature. The preview (`get-file-snapshot`) applies the pending migrations in memory with the live file's libraries and never persists them: the stored version keeps its data, the payload carries the snapshot's `:migrations` (an ordered set) and `fdata/objects-map` is realized for clients that do not declare it. The historical feature set is deliberately not checked against the enabled one: a version may predate a feature change. The window is narrow: the `get-file` RPC reads with `migrate?` on and its ETag hashes `available-migrations`, so a post-deploy read cannot be a 304 from the pre-deploy state and migrates before any save in the normal flow; the pre-migration snapshot only appears when the save is the first request to touch the file after the deploy (an open tab that saves without re-reading, or a non-UI writer). `create-file-snapshot` and the restore backup read with `migrate?` on, so those versions are stored already migrated.
+- `:migrations` is always a `d/ordered-set`: `file-migrations/resolve-applied-migrations` builds one, `fmg/migrate` re-wraps the union after applying, and `file-snapshots/decode-snapshot` converts the `file_change` pgarray (as a plain vector, `set/difference` and `contains?` would compare indexes instead of names). A snapshot with NULL `:migrations` and an old `:version` predates the `file_change.migrations` column (SQL migration 0140): the preview regenerates the set from the stored version and applies that full chain, while `restore!` uses a fixed 67 baseline, so preview and restore can disagree for very old snapshots.
+- The migration read path (`binfile.common/migrate-file`) captures `fdata/original-pointers` before realizing, reattaches unchanged positions after applying the migrations, and persists the reattached file; it returns the realized file because reads expose plain data. A migration that adds `fdata/objects-map` skips the reattach so the pages get wrapped.
+- `fdata/load-pointer` reads rows that a previous `file-gc` pass marked on purpose (explicit `::db/remove-deleted false`): the fragment is only gone once `objects-gc` purges it. The `get-file-fragment` RPC keeps rejecting marked rows.
+- Bind the fragment loader through `fdata/with-pointers` (reads), `fdata/with-tracked-pointers` (writes with a loader) or `fdata/with-tracked` (writes with nothing to load yet, e.g. file creation), and `fdata/with-untracked` (validation passes that must track nothing) instead of hand-rolled `binding` forms. Two direct bindings remain on purpose: the fine-grained tracked toggles of `app.rpc.commands.files-update` (validation/repair turn tracking off, restore the existing atom and read it before repairing), and `binfile.common/encode-file` (a fresh tracked set inheriting the current one). Converge them only if atom-scoped macros (`with-tracking`, `with-inherited-tracking`) are added; today the extra indirection buys nothing.
+- Snapshot creation takes `FOR SHARE` on the file row: both `create-file-snapshot` and the backup read of `restore-file-snapshot` hold it from the read to the insert, so they serialize with a `file-gc` pass (`FOR UPDATE ... SKIP LOCKED`) instead of pinning fragments the pass is about to mark. `get-file-snapshot` realizes the snapshot pointers with the live file id (fragment rows are keyed by file, not by snapshot).
 - `fdata/realize` combines pointer realization and object-map realization. Use it before operations that need complete in-memory file data instead of pointer placeholders.
 - A `file_data` row is one of three classes, told apart by `type`: `main` is the file's current state and carries `id = file_id`; `snapshot` is a point-in-time copy whose `id` is the one its `file_change` row shares; `fragment` is a piece of the pointer-map, referenced from the file's data.
 - Their lifecycles differ: `main` and `snapshot` live with the file (dated when it is deleted, undated when it is restored), while a `fragment` carries no date and is collected by `file_gc` once the file stops referencing it. Neither is a session buffer, so do not read a `fragment` as an ephemeral revision.
@@ -90,3 +110,4 @@
 - Revisions live only in `file_change` and have no `file_data` counterpart. They are born already scheduled for deletion one hour later (`app.rpc.commands.files-update`) and the snapshot paths cannot reach one, because they resolve their row through a `label IS NOT NULL` filter.
 - A deletion date is assigned once and never moved: `restore-files` (`app.rpc.commands.files`) unmarks only durable history, which is `label IS NOT NULL` (the unlabelled rows that still carry a legacy payload are removed by `0157-migrate-legacy-file-data`, so restore leaves them dated), the `:file` method of `app.tasks.delete-object` dates only rows that have no date yet, and `objects-gc` collects only rows that carry one. Unmarking a revision leaves it invisible to `objects-gc` for ever, because nothing sets that date again.
 - Binfile v1/v3 import and the v3 export `get-file` step run inside one long transaction. They raise `idle_in_transaction_session_timeout` to 30 min (`app.binfile.common/transaction-idle-timeout-ms`) instead of disabling it, so a connection left idle while the ZIP is decoded or the file data is realized is bounded; `statement_timeout` stays at 300 s.
+- `update-file` with `:changes-with-metadata` groups that all carry `:commit-id` is partial: fast path applies and validates everything once; on a change failure it bisects the commits for the longest prefix that applies and validates (each attempt applies a prefix from the original data and runs referential integrity + file schema; ~log2 K attempts, none for a single commit, the first attempt's error reused), keeping that prefix (a commit that repairs an earlier one can keep both) and answering 200 with `:rejected {:commit-ids :failed-commit-id :error}`, plus `:preexisting-damage true` when the data the first rejected commit touches already fails validation without it (`preexisting-damage?`: one more referential + schema check of its pages/components on the base file, the accepted prefix or the original; the frontend then says the page data is damaged). Only accepted changes reach `file_change`, persistence and the msgbus; a fully rejected request writes and broadcasts nothing (revn unchanged). Flat `:changes` or groups without ids stay all-or-nothing. Before each attempt (`apply-prefix`, which returns `{:file :tracked :refs}` or the cause) `pmap/*tracked*` and the `cpc/*state*` atom are reset to their pre-attempt values (at the end `apply-commits` sets those of the kept prefix): a pointer map already `modified?` keeps its id when changed again, so the failed version would be persisted. DB/IO/interrupt errors and `:not-found` fragments (`change-failure?`) fail the whole request, never reject a commit. A rejection whose ex-info has no `:type` reports `:internal :unexpected-error`. `app.http.errors` answers 503 (`:server-error` + code) for transient SQLSTATEs (classes 08/53, 57P01–3, 40001, 40P01), `SQLTransientException`, `InterruptedException` and only the `IOException`s that pass on their own (`transient-io-exception?`: `SocketException`, `InterruptedIOException` incl. socket timeouts, `UnknownHostException`, `HttpTimeoutException`), which the client's save retries (and GETs retry by themselves); any other `IOException` (EOF from a corrupt blob, Jackson parse errors, missing files) stays 500 `:io-exception`, as do other SQL errors. With the `:debug-save-faults` flag, a `:set-plugin-data` file change in namespace `:penpot-debug` keyed `save-fault:<kind>` raises that fault in `apply-changes` (used by `debug.failNextSave`/`commitInvalidChange`). With a schema check running (hard schema validation without skip-validate, or soft) and 2+ commits, the touched pages' objects are decoded on the original file first (`decode-objects`; referential validation alone reads few objects, so no pre-decode then): objects-map caches decoded values per instance and a derived map copies the cache, so otherwise every attempt re-decodes the page (~260 ms at 10k shapes). Validation step timings (`with-validation-timing`, upstream `penpot_rpc_update_file_validation_timing` + the `update-file` debug log) come from the first attempt only (retries run with a cfg without metrics and timings); the whole bisection is one `retry` step sample, and the rejection log carries `:attempts`/`:elapsed`.

.serena/memories/backend/testing.md

@@ -5,10 +5,29 @@ JVM `clojure.test` (kaocha runner) under `backend/test/backend_tests/`.
 - READ `mem:testing` FIRST — it defines the execution discipline (no piping, tee to file, preferred commands) that applies to all JVM test runs.
 - All CLI commands must be executed from the `backend/` subdirectory.
 - Tests are invoked directly via `clojure -M:dev:test` (kaocha) — there is no pnpm wrapper. Kaocha auto-discovers test namespaces, so no runner registration is needed.
-- `clojure.core/assert` is elided at compile time unless `PENPOT_FLAGS` contains `enable-backend-asserts` (`app.config` sets `*assert*` from it). `scripts/ci` and `tests-backend.yml` export the flag; a bare `clojure -M:dev:test` (or `--focus` run) without it silently skips every precondition assert.
+- **Preferred way to run them: `scripts/ci <module> --test`**, which runs lint, tests and format per module and writes a log per task under `.ci-logs/`. It exports `PENPOT_FLAGS` with `enable-backend-asserts` appended, so use it rather than calling `clojure -M:dev:test` by hand.
+- **Why the flag matters:** `clojure.core/assert` is elided at compile time when `*assert*` is false, and `app.config` only sets that var from the flags when it loads. The test fixture redefines the flags after that, so they never reach it: without the flag the suite runs with every assertion compiled out, and a test that depends on an assert only means something with it set. `scripts/ci` and `tests-backend.yml` export the flag (GitHub Actions sets it too). When running the suite directly — including a `--focus` run — the command is `PENPOT_FLAGS="enable-backend-asserts" clojure -M:dev:test`.
 - Coverage: if code is added or modified in `src/`, corresponding tests in `test/backend_tests/` must be added or updated.
 - Isolated run: `clojure -M:dev:test --focus backend-tests.my-ns-test` for a specific test namespace, or `clojure -M:dev:test --focus backend-tests.my-ns-test/my-test-var` for a specific test var.
+- Multiple focus symbols are supported by REPEATING the flag (kaocha accumulates them, last-wins does not apply): `clojure -M:dev:test --focus backend-tests.ns-one --focus backend-tests.ns-two` — preferred over a shell loop of isolated runs.
 - Regression run: `clojure -M:dev:test` to ensure no regressions in related functional areas.
 - If you need to filter output, tee to a temp file first: `clojure -M:dev:test 2>&1 | tee /tmp/penpot-test-output.txt`.
 - RPC test helpers `command!`/`management-command!` split the data map: qualified keys become server params, unqualified keys become request body params. To inject request-level context (headers, `:app.http/auth-key-id`, ip), pass a map under `:app.http/request` metadata; non-map `IRequest` stubs fall back to a dummy request.
+- The test system is `main/system-config` + `main/worker-config` with the
+  components that must not run in the background dropped: HTTP, the srepl,
+  cron, the dispatcher and **every** `[<profile> :app.worker/runner]`
+  (`helpers/remove-job-runners`). A live runner of any queue races the tests
+  that drive the loop by hand (`wdisp/run-batch` + `@#'wrkr/run-worker-loop`):
+  it takes the job the test just dispatched and completes it behind the test's
+  back, so the test's own assertions read a state it did not produce. Do not
+  re-add a runner by editing that list by hand; the removal matches by key
+  shape on purpose.
+- `system-config` and `worker-config` are `def`s: their `(cf/get ...)` calls
+  are evaluated when the namespace loads, against the real environment, while
+  test code calling `cf/get` at run time sees the `penpot-test`-prefixed
+  config. The two can disagree — in the devenv `:tenant` is `devenv-ws0` for
+  the defs and `default` inside a test — so a component built from
+  `worker-config` can talk to a different Redis key prefix than a cfg the test
+  builds itself. The symptom is a component that silently does nothing
+  locally and races in CI (which sets no `PENPOT_TENANT`).
 - `backend-tests.helpers` has `metrics-stub` (a metrics instance with no collectors, where every `run!` is a no-op), `init-climit` (a concurrency limiter built from an inline config map, with an optional second argument for the metrics instance to publish to) and `with-climit`, the macro that builds one, runs the body and halts it. `init-climit` starts the republisher daemon thread, so a test that builds a limiter without halting it leaks that thread for the rest of the JVM run: use `with-climit`. Log lines are capturable with `add-watch` on `app.common.logging/log-record` (it holds the last record; the watcher sees every one, but only of the levels the logger has enabled: a line the app writes at TRACE never gets there).

.serena/memories/clojure/idioms.md

@@ -8,5 +8,7 @@ Behaviors confirmed against the language/stdlib — do not re-derive from assump
 - The analyzer reads `:async` only from the fn name meta and the `fn` operator meta; list-level meta is ignored (would make a MetaFn). `(fn ^:async [] …)` puts the meta on argv (pre/post only, NOT async).
 - Valid: `(defn ^:async f)`, `(defn- ^:async f)`, `(^:async fn [] …)` (the latter is what stock `t/async` generates itself).
 - `^:async` fns never throw synchronously (rejected promises instead); `try/catch/finally` supported; continuations are microtasks, timers and RxJS schedulers are macrotasks (FIFO).
+- Inside an `^:async` fn, a nested call that needs a temp binding (e.g. calling a local fn `(g (f x))`, whose arg gets a `G__` binding) compiles to `(await (async function(){…})())`: `await` unwraps a returned promise, so `(settled (upload (make-ctx)))` blocks until the upload promise settles instead of passing the promise on (a test that pushes the answer later hangs, idle). Bind each step in its own `let` binding.
 - `!` suffix allowed cases (only exceptions to the no-new-`!` rule in `mem:clojure/design-rules`): mutating public `app.db` API (`insert!`, `update!`, `delete!`, `run!`, `tx-run!`, `exec!`, ...) and fns whose contract is to work over transients (stdlib `conj!`, `assoc!`, `dissoc!`, `disj!`, `persistent!` and custom helpers over them); any further case must be registered here with justification before use.
+- Penpot extends `clojure.core/Inst` in `app.common.time` ( JVM-only section), so `inst-ms` also accepts `Duration` (returns `.toMillis`, NOT epoch millis), `FileTime` and `OffsetDateTime`. It only works where that namespace is loaded; on a bare `Duration`, prefer explicit `.toMillis`, which reads as what it is (an amount, not an instant).
 - `cuerdas.core/index-of` returns `nil` (not `-1`) when the substring is absent, like `clojure.string/index-of`. `(pos? nil)` **throws on the JVM** but evaluates to `false` in CLJS (JS coercion), so `(if (pos? (str/index-of s "\n")) …)` works in the browser and crashes the backend. Guard with `some?` before `pos?` (this was `app.common.exceptions/first-line`).

.serena/memories/clojure/promesa.md

@@ -0,0 +1,9 @@
+# Promesa
+
+Promise library of the monorepo (frontend CLJS and common, `funcool/promesa` 12.0.1 in `common/deps.edn`). Its API has two call styles and mixing them in one expression is the trap: `fmap`/`mcat`/`merr` are **fn first, promise last** (`->>` friendly) and `then`/`then'`/`catch`/`handle` are **promise first, fn last** (`->` friendly). Convention: `fmap`/`mcat`/`merr` inside `->>` pipelines, `then`/`catch` with `->`. A `then` written inside a `->>` reads backwards and hides whether the step carries a value or a promise.
+- `fmap` (alias `map`, both `(f p)`) is functor map: `f` returns a plain value.
+- `mcat` (alias `mapcat`, both `(f p)`) is monadic bind: `f` must return a promise, and the result mirrors it.
+- `then` also unwraps a promise returned by `f`, which is why it covers both cases at the cost of not naming which one it is; `then'` maps without unwrapping.
+- `merr` (fn first) and `catch` (promise first) are the error side.
+- `all` takes a collection of promises (a lazy seq works); `let` chains sequential bindings.
+- Seam with the rx pipelines of the frontend: `beicon.v2.core/from` builds an observable from "arrays, collections, and promise instance" (`external/beicon`), so `(->> p (p/mcat f) (rx/from) (rx/map g))` is the usual bridge (`app.main.data.imports`, `app.util.zip`).

.serena/memories/common/data-model-change-checklist.md

@@ -6,7 +6,7 @@
 - Do not treat nil as a distinct persisted state from absence. Import/export and cleanup paths may filter nil attrs away.
 - Avoid Clojure-special naming in exported object attrs, especially boolean names ending in `?`; exported/imported data must survive JSON/SVG/Transit and external tooling.
 - Any new shape attr that participates in component sync must be listed in `app.common.types.component/sync-attrs` with the correct touched group. Attrs absent from `sync-attrs` are ignored by component synchronization.
-- Shape `:svg-attrs` is the one map whose keys are camelCase React prop names (`fillRule`, not `fill-rule`), because the SVG import path runs them through `app.common.svg/attrs->props`. The `.penpot` reader (`app.common.json/read-kebab-key`) rewrites every json key of every entry to kebab-case, nested maps included, so any code that decodes a binary file must run `:svg-attrs` back through `attrs->props` or the names are lost. `:svg-defs` and the `:content` tree of `svg-raw` shapes keep their source names and round-trip on their own.
+- Shape `:svg-attrs` keys are kebab-case (`fill-rule`, not `fillRule`): birth (`app.common.svg/attrs->kebab-props`, applied everywhere data lands on a shape) normalizes any outside spelling through the `svg-props` whitelist, and render boundaries (`get-svg-props`, `render-xml*`, `svg-node*`) translate back to camelCase for React. `:svg-defs` node attrs follow the same rule recursively; svg-raw `:content` attrs are kebab-ized spelling-only, never filtered — the sidebar reads `:stroke-style`, which is outside the whitelist. Non-spec attrs Penpot keeps live in `penpot-extra-attrs` (seeded with `:stroke-style`) and travel in source spelling. Excluded, each with its reason: `:tag` values (React and tag sets need them verbatim; the schema decoder restores keywords from stored strings), outer `:svg-defs` keys (id strings at birth, keywords or strings after JSON decode — accept both), all values (the kebab reader never touches values). Repair sites: birth plus migration `0035-fix-shape-svg-attrs-and-defs` (replaces the withdrawn `0021`); the binfile cleaner carries no svg repair.
 
 ## Cross-module update checklist
 

.serena/memories/common/file-change-validation-migration-subtleties.md

@@ -20,8 +20,10 @@
 
 - Full referential/semantic validation currently runs only when file features contain `"components/v2"`.
 - Validation starts at root plus orphan shapes, then validates component records. `validate-file!` raises `:validation :referential-integrity` with collected details.
+- `update-file` validates per save with the change-scoped variants: `validate-file-affected!` (referential, whole touched pages/components) and `validate-file-schema-affected!` (file schema with `:pages-index` reduced to the touched pages; non-page data checked whole). The schema variant maps change types to pages itself (`touched-page-ids`): a new change type that writes a page must be added there, or to `file-level-change-types` if it touches no page; an unknown type makes it check the whole file. The whole-file schema check loads and decodes every page, ~75–80% of a save on multi-page files.
 - `repair-file` does not mutate data directly; it reduces validation errors into redo changes using `changes-builder`. Callers must apply or persist those changes.
 - `repair-file-until-valid` stops on: no errors; unchanged `:data` after apply (backend page/`PointerMap` `=` is pointer identity); stable error fingerprint `(frequencies (map #(dissoc % :shape) errors))` vs previous pass (check fingerprint before `repair-file`/`process-changes`); or `max-iterations`. Each `repair-error` builds from the original `file-data`, so N `:set`-based errors on one shape take N passes.
+- `:ref-shape-not-found` repair: when the copy root's local component has a `:main-instance-id` that resolves to no shape but a `:main-instance` shape of that component sits on its `:main-instance-page`, it emits `mod-component` pointing the id back to that main (keeps the copy linked; a repair scoped to the copy's page never sees the main's `:invalid-main-instance-id`). Otherwise near-main reassign, else detach the copy subtree.
 - `:missing-slot` fires only for a REAL swap: a copy sub-head whose `shape-ref` is no longer a child of the near main parent. A pure positional mismatch (ref still a sibling elsewhere) is a reorder — valid, realigned by the async component sync; do not "repair" it by assigning swap slots (a slot freezes the child out of normal sync). `fix-missing-swap-slots` (migration 0019) follows the same membership rule.
 
 ## Migrations
@@ -30,4 +32,5 @@
 - Migrations are an ordered set mixing legacy version-derived ids and newer named ids. Keep append order stable; `migrate` applies the set difference between available migrations and file migrations.
 - `migrate-file` synthesizes legacy migration ids from old numeric versions when `:migrations` is absent, migrates legacy features, and records feature flags created through `cfeat/*new*`.
 - When a file had no previous `:migrations`, `migrate-file` marks all migrations as migrated in metadata so callers persist the complete migration set, not only transformations that changed data.
-- Closed schemas (`schema:stroke-attrs` and others) reject a key that was dropped from the declaration, so removing a declared attr needs a migration in the same change, for data already written with it.
+- Closed schemas (`schema:stroke-attrs` and others) reject a key that was dropped from the declaration, so removing a declared attr needs a migration in the same change, for data already written with it.
+- Rebase collision on a migration number: upstream keeps the id. Renumber the branch migration to the next free one, keep it last in `available-migrations`, and rename everything that spells the number — its test file and namespace (`files_migrations_00NN_test.cljc`), the test names, and comments in production code or memories that name it.

.serena/memories/common/testing.md

@@ -38,6 +38,10 @@ Useful builders:
 
 `add-variant-with-copy` does not accept position params for children; use `gsh/absolute-move` after creation if positions matter.
 
+## Generative sampling caution
+
+- `sg/sample` is unseeded per run. If a test filters samples by type (e.g. keep only `:svg-raw`) then `first` can yield nil on unlucky seeds; `assoc` onto nil builds a typeless map that fails `:multi` dispatch on `:type`. Pin `:seed` in `sg/sample` opts whenever a test depends on a filtered subtype being present (verified seed 42 deals svg-raw shapes on JVM and JS).
+
 ## Driving production paths
 
 For shape mutations, prefer production-path helpers such as `cls/generate-update-shapes` plus `thf/apply-changes`. For component swaps with keep-touched behavior, use `tho/swap-component-in-shape` with `{:keep-touched? true}`.

.serena/memories/critical-info.md

@@ -8,6 +8,7 @@ You are working on the GitHub project `penpot/penpot`, a monorepo.
 - Cross-cutting testing principles, TDD workflow, and anti-patterns: `mem:testing`.
 - Verified Clojure language behaviors that contradict common assumptions (e.g. `int?` covers `Long`; `integer?` is the general predicate): `mem:clojure/idioms` — read before assuming stdlib predicate semantics.
 - Read `mem:clojure/design-rules` (function shape, boundary checks, naming) before writing or modifying Clojure/ClojureScript code — not only when designing a solution or an API.
+- Promises: which call style to use where (`fmap`/`mcat`/`merr` vs `then`/`catch`) and how they bridge to rx: `mem:clojure/promesa` — read before chaining promises or mixing them with observables.
 
 # Development workflow
 
@@ -17,7 +18,7 @@ You are working on the GitHub project `penpot/penpot`, a monorepo.
   - Before `gh pr create` / `gh pr edit` → `mem:workflow/creating-prs` (title format, body structure, "Note:" line)
 - Before updating `CHANGES.md` → `mem:workflow/updating-changelog` (sources of truth, entry rules, the `:boom:`/`:rocket:` label contracts, pre-flight checks, anomaly report)
 - Before a repo-wide pnpm version update → `mem:workflow/updating-pnpm` (workspace layout, `scripts/sync-pnpm-version` flow, the stamp-missing-field and ignored-builds gotchas, verification steps)
-- Before a Node version update → `mem:workflow/updating-node` (the six pin sites in `.nvmrc`, the devenv and image Dockerfiles and the plugins-deploy workflows, the DHI tag gotcha, what to verify)
+- Before a Node version update → `mem:workflow/updating-node` (the six pin sites in `.nvmrc`, the devenv and image Dockerfiles and the plugins-deploy workflows, the DHI tag and digest rules, what to verify)
 - Before bumping a non-JS dependency pinned in `docker/images/` (JDK, ImageMagick) → `mem:workflow/docker-images`
 - **Never `git push`, force-push, or modify `git origin`** (or any other remote). The user pushes from their own shell; if a push is required, say so and wait. Never amend a commit that the user has already pushed unless explicitly asked.
 - **Never edit `CHANGES.md` by hand.** The changelog is generated from GitHub milestones during the release process; update it only via `mem:workflow/updating-changelog` (the `update-changelog` skill flow) or on explicit user request.
@@ -64,7 +65,7 @@ The memory is structured in a way that you can get the critical information abou
 - `scripts/error-reports.mjs` — Query error reports via RPC API with token authentication. Supports list/get operations with filtering and pagination. See `mem:scripts/error-reports`.
 - `scripts/clean-node-modules` — Remove stale `node_modules` from all pnpm workspaces (root, modules, member packages). Keeps the shared pnpm store at `<repo>/.pnpm-store` unless `--store`; ignores `external/` and `.opencode/`. Usage and reinstall steps: `mem:workflow/updating-pnpm`.
 - `scripts/setup-opencode` — Install the opencode CLI in-container as a global pnpm package in `$PNPM_HOME`. Not part of the devenv image; see `mem:devenv/core`.
-- `scripts/ci` — CI orchestration script: runs lint, tests and format checks per module, plus the production build on request (`--build`). Modules: `frontend backend common render-wasm exporter mcp plugins library media-processor`. Logs go to `.ci-logs/`; read the log file on failure. See `mem:scripts/ci`.
+- `scripts/ci` — CI orchestration script: runs lint, tests, and format checks per module, plus the production build on request (`--build`). Modules: `frontend backend common render-wasm exporter mcp plugins library media-processor`. Logs go to `.ci-logs/`; read the log file on failure. It is the preferred way to run the backend tests, and it exports `PENPOT_FLAGS` with `enable-backend-asserts`, which is what makes `clojure.core/assert` exist in the compiled test code. See `mem:scripts/ci`.
 
 # Dependency graph
 

.serena/memories/devenv/core.md

@@ -5,7 +5,7 @@ Compose-based dev environment under `docker/devenv/`, driven by `manage.sh`. Par
 ## Compose project layout
 
 - `penpotdev-infra`: shared `postgres`, `rustfs`, `valkey`, `mailer`, `ldap`. File: `docker-compose.infra.yml`.
-- `penpotdev-wsN` (N=0,1,…): per-instance `main` + `redis` (Valkey). File: `docker-compose.main.yml`. ws0 (a.k.a. `main`) binds `$PWD`; ws1+ bind clones at `${PENPOT_WORKSPACES_DIR}/wsN/` (default `~/.penpot/penpot_workspaces/`), maintained by the developer.
+- `penpotdev-wsN` (N=0,1,…): per-instance `main` + `redis` (Valkey). File: `docker-compose.main.yml`. ws0 (a.k.a. `main`) binds `$PWD`; ws1+ bind clones at `${PENPOT_WORKSPACES_DIR}/wsN/` (default `${XDG_DATA_HOME:-~/.local/share}/penpot-devenv/workspaces/`, or the former `~/.penpot/penpot_workspaces/` when it already exists), maintained by the developer.
 - Optional overlay `docker-compose.opencode.yml`: added by `instance-compose` as an extra `-f` only when `PENPOT_OPENCODE_CONFIG_DIR` is set (i.e. `run-devenv --opencode-config-dir DIR` ran in this process). Bind-mounts the host dir at `/home/penpot/.config/opencode` (`:z`). Flag-only, per-call; not read from ambient env. Parser `parse-opencode-config-dir` absolutizes (`~`, realpath) because compose resolves relative bind sources against the compose file's dir. Only instances brought up with the flag get the mount.
 - All projects join external network `penpot_shared`. Created idempotently by `ensure-devenv-network`, never removed by lifecycle commands.
 
@@ -29,7 +29,7 @@ Compose-based dev environment under `docker/devenv/`, driven by `manage.sh`. Par
 
 ## Worker policy
 
-Backend workers run only on ws0. `_env` gates `enable-backend-worker` on `PENPOT_BACKEND_WORKER`; ws1+ inject it as false. Workers are pure fire-and-forget: `wrk/submit!` inserts a row into the shared Postgres `task` table and returns; RPC handlers never wait on completion and workers never publish to msgbus. The reason for "ws0 only" is avoiding multi-instance worker races (cron dedup is best-effort across instances, `wrk/submit!` `dedupe` is racy across submitters); details in `mem:prod-infra/core`.
+Backend workers run only on ws0. `_env` gates `enable-backend-worker` on `PENPOT_BACKEND_WORKER`; ws1+ inject it as false. Workers are pure fire-and-forget: `jobs/submit` inserts a row into the shared Postgres `job` table and returns (the legacy `task` table stays dormant); RPC handlers never wait on completion and workers never publish to msgbus. The reason for "ws0 only" is avoiding multi-instance worker races (cron dedup is best-effort across instances, `jobs/submit` `dedupe` is racy across submitters); details in `mem:prod-infra/core`.
 
 Each workspace is independent and can be started/stopped in any order. Shared infra (Postgres, RustFS, etc.) is shut down only when no instances remain running.
 
@@ -77,3 +77,42 @@ No `--delete` on the working-tree pass: gitignored caches in the workspace survi
 - `start-devenv` / `log-devenv` / `drop-devenv`: legacy paths around ws0 + shared infra. `drop-devenv` never removes volumes.
 
 `exporter/scripts/run` and `wait-and-start.sh` source `backend/scripts/_env` then `_env.local` if present.
+
+## Image and dependency pins
+
+Two Dockerfile trees, both multi-arch (`linux/amd64,linux/arm64`):
+`docker/devenv` (dev) and `docker/images` + `docker/imagemagick` (prod).
+
+- Hand-pinned versions, one source each:
+  - `docker/devenv/Dockerfile`: `NODE_VERSION` (+ per-arch `NODE_SHA256`),
+    `PNPM_VERSION` (+ `PNPM_SHA256`), `CADDY_VERSION`, `CLOJURE_VERSION`,
+    Zulu JDK URL + `ESUM`, `RUSTUP_VERSION`/`RUST_VERSION`/
+    `EMSCRIPTEN_VERSION`, `CLJKONDO_VERSION`, `BABASHKA_VERSION`,
+    `CLJFMT_VERSION`, `PIXI_VERSION`, `GITHUB_CLI_VERSION`,
+    `AWS_CLI_VERSION` (+ `BINARY_SHA256`), `UV_VERSION`, `SERENA_VERSION`,
+    `postgresql-client-18`, and the `penpotapp/imagemagick:<tag>`
+    `COPY --from`.
+  - `docker/images/Dockerfile.backend`: Zulu JDK URL + `ESUM`. The `jlink`
+    `--add-modules` list must keep `jdk.management`
+    (`mem:backend/subtleties`).
+  - `docker/images/Dockerfile.{exporter,mcp,media-processor}`: `ARG
+    PNPM_VERSION` (+ arch SHAs, same values as devenv).
+- Node and pnpm must agree in five places: `.nvmrc`, the devenv tarball pin,
+  the two `plugins-deploy-*` workflows that set `runtime: node@<version>`, and
+  the four Dockerfile pins. pnpm procedure: `mem:workflow/updating-pnpm`.
+- `dhi.io` base images are pinned `tag@sha256:<index digest>`; Renovate moves
+  them (`penpot-renovate[bot]`, commits titled `:arrow_up: Update DHI base
+  images`). Tag and digest move together: a new tag with the old digest still
+  pulls the old image, and the digest must be the multi-arch index, not a
+  per-platform one.
+- Every hand-pinned checksum is per architecture; one wrong value breaks half
+  of the multi-arch build.
+- `SERENA_UPDATE_VERSION` in `docker-compose.main.yml` must never be lower than the Dockerfile's `SERENA_VERSION`. Any value makes `start-tmux.sh` install that Serena release on start, overriding the version baked into the image; a stale value silently downgrades Serena (it once replaced 1.7.0 with 1.6.1). If the image already has that version the install is an offline no-op; otherwise it downloads on every start (`run-devenv` recreates the container) and, offline, fails after ~25s and keeps the image version. Bump both together; empty means use the image version.
+- Infra images live in `docker-compose.infra.yml`: `rustfs/rustfs` (owns the
+  S3 data volume), `axllent/mailpit` (SQLite in a volume), `valkey/valkey`,
+  `postgres:18.6`, `rroemhild/test-openldap`. Keep the volume names stable
+  when bumping them; replacement services get new volumes and leave the old
+  data untouched.
+- Build locally: `docker build -f docker/devenv/Dockerfile docker/devenv/`
+  and `cd docker/images && ./build.sh <image> --local`. CI:
+  `build-docker-devenv.yml`, `build-docker.yml`.

.serena/memories/frontend/auth-events.md

@@ -55,8 +55,11 @@ never carries `activation`. Do not "fix" it by inferring a signup.
 - `login-from-register` accepts an optional `:invitation-token`;
   `with-register-origin` decorates the profile and `logged-in` does the rest.
   The four places that used to navigate by hand to `:auth-verify-token`
-  (`ui/auth/register.cljs` twice, `ui/viewer/login.cljs`, `ui/static.cljs`) emit
-  `login-from-register` with the token.
+  (`ui/auth/register.cljs` twice, `ui/static.cljs`, `ui/auth/login_modal.cljs`)
+  emit `login-from-register` with the token. The modal used to live in
+  `ui/viewer/login.cljs`; it was extracted to the shared
+  `ui/auth/login_modal.cljs` (`login-register-modal*`, also used by
+  `ui/mcp_authorize.cljs`).
 - The invitation navigates through `get-invitation-nav`, which does not wait for
   `::dtm/teams-fetched`: `fetch-teams` has no error handling, so depending on it
   would leave the invitation unaccepted when `get-teams` fails. For invited
@@ -66,9 +69,11 @@ never carries `activation`. Do not "fix" it by inferring a signup.
   invitation JWE in its claims (`backend/src/app/auth/oidc.clj`), so the
   frontend follows the regular path (`login-from-token` → `logged-in`). That
   flow visits the token verification screen twice.
-- `ui/viewer/login.cljs` and `ui/static.cljs` `:is-active` branches still bypass
-  `logged-in` (they hide the modal or reload the page); they are a separate
-  decision, not an oversight.
+- `ui/auth/login_modal.cljs` (the register modal: `:is-active` branch calls
+  `on-close`, or `on-register` when the caller passes one, as in the MCP
+  authorize flow) and `ui/static.cljs` (`rt/reload true`) `:is-active`
+  branches still bypass `logged-in`; they are a separate decision, not an
+  oversight.
 
 ## Backend side
 

.serena/memories/frontend/cljs-repl.md

@@ -49,16 +49,23 @@ Get `team-id` from `(:current-team-id @app.main.store/state)`. Get file ids from
 
 ## Runtime patching with `set!`
 
-Some frontend vars are deliberately mutable escape hatches for runtime instrumentation or circular-dependency patching. From `cljs_repl`, use `set!` for temporary debugging of CLJS vars such as `app.main.store/on-event`, `app.main.errors/reload-file`, `app.main.errors/is-plugin-error?`, `app.main.errors/last-report`, or `app.main.errors/last-exception`. These patches affect only the live browser runtime and disappear on reload or recompilation.
+Some frontend vars are deliberately mutable escape hatches for runtime instrumentation or circular-dependency patching. From `cljs_repl`, use `set!` for temporary debugging of CLJS vars such as `app.main.errors/reload-file`, `app.main.errors/is-plugin-error?`, `app.main.errors/last-report`, or `app.main.errors/last-exception`. These patches affect only the live browser runtime and disappear on reload or recompilation.
+
+`set!` on `app.main.store/on-event` does nothing at runtime: the store captured the function when it was created. To watch events, subscribe to the store stream and dispose the subscription after:
 
 ```clojure
 ;; Log non-noisy Potok events temporarily.
-(set! app.main.store/on-event
-      (fn [event]
-        (when (potok.v2.core/event? event)
-          (.log js/console (potok.v2.core/repr-event event)))))
+(def sub
+  (beicon.v2.core/subs!
+   (fn [event]
+     (when (potok.v2.core/event? event)
+       (.log js/console (potok.v2.core/repr-event event))))
+   app.main.store/stream))
+;; (beicon.v2.core/dispose! sub)
 ```
 
+`cljs_repl` starts every call in `cljs.user`; begin a call with `(in-ns 'my-ns)` to reuse helpers defined in another namespace.
+
 Restore mutable hooks after debugging, or reload the frontend. Use JVM `alter-var-root` only for JVM Clojure; it is not the normal way to patch live CLJS browser vars.
 
 ## Browser-console debug namespace

.serena/memories/frontend/composable-component-tests.md

@@ -150,5 +150,5 @@ The frontend bundle is built once per SHA and restored by this job (`mem:fronten
 Details: README, "Running in CI".
 
 ## Substrate
-`mem:common/test-setup`, `mem:common/component-data-model`, `mem:common/component-swap-pipeline`,
+`mem:common/testing`, `mem:common/component-data-model`, `mem:common/component-swap-pipeline`,
 `mem:frontend/testing`.

.serena/memories/frontend/dashboard-viewer-subtleties.md

@@ -5,6 +5,7 @@
 - Dashboard initialization fetches projects and fonts for the team, then listens to websocket messages only for global topic `uuid/zero` or the current profile id.
 - Project fetch replaces each project map completely instead of merging, so fields such as `deleted-at` can disappear cleanly.
 - Dashboard file/project mutations are often optimistic local updates with fire-and-forget RPC watchers. Bulk permanent delete/restore paths use SSE progress and progress notifications.
+- The import wizard runs the package as a job of the backend: it reads the manifest of the file in the main thread (`app.main.data.imports`, no web worker) and follows the job, so its header paints the milestone of the job (the stage and its counter) instead of the messages the old SSE stream sent; every file of a container reports the same milestone, so `jp/current-milestone-text` writes it once and the rows never repeat it. Before the job exists, the header paints a client-side `:upload` milestone built from the chunked upload (`uploads/upload-blob-chunked` `:on-progress`, chunks done/total), which never comes from the backend; the chunks go one at a time and it writes them as a percentage of the file, not as a counter. Once created and still waiting for a worker, no row says anything: the header says the import waits for a slot while no job of it has started. The bar of the header measures the whole import, never a file: indeterminate while checking, waiting for a slot or rebuilding, because the client cannot know a global index and total of operations. The upload is the exception — the chunked upload knows its chunks, so the bar is determinate while the containers are sent, weighted by their size (`imp/upload-pct` over `:container-size`, the only percentage the client really knows). The export modal follows the same one-job-per-file model, but every file is queued up front (even the ones whose job is created only after the previous one is over) and its queue ends only with a milestone: `:start`/`:retry` do not end it there. Neither modal paints a generic "…ing file…" line on the rows any more: the rows of the import wizard carry only what is not progress — the name, the size, `Ready` once their container finished uploading, and the error text —, and the button that drops a file only shows while the wizard still offers its import step (the confirmation screen). Above the rows, the import wizard paints a stepper (Check/Upload/Import) plus one phase line: `imp/import-step` derives the step from the dialog status and the entries (upload holds while any file is pending, either `:import-ready` or carrying its own `:upload` milestone), so Check covers analyze, Upload covers the chunked client-side upload, and Import covers queue plus job. A file of a version 3 package shows what it occupies inside its container, computed while analyzing (`imp/file-sizes`): its own `files/<id>…` entries plus the storage objects its media and thumbnails reference, whose metadata is the only link to `objects/<id>…`. The container's own size stays in `:container-size` (what the dropped file weighs) and is what the upload bar weighs, never what a row shows. A file the manifest lists but the container holds no entry of has no size at all, and its row then shows no subtitle: what the package weighs never stands in for it. The subtitle is the size alone, with no extension. While analyzing with no failures nothing renders below the phase header — no rows, no spinner (`imp/awaiting-analysis?`); the header text is the only "Checking…" message. Rows appear once the analysis resolves (one per file inside the package), and a failed analysis takes the error-summary path instead. The ready step and its Continue wait for the end of the analysis (`imp/ready-to-import?`, fed with the completion of `imp/analyze`): a ready file is not enough, because the files still to arrive would never be imported. While rebuilding, the phase title counts the files and `imp/imported-count` counts them done with the job's own `:files` counter, so the number moves while a container rebuilds its files one by one. The `:error` of an import entry is user-facing text (the backend hint, or a translated label), never an i18n key: the wizard renders it as-is, it does not translate it again. The link-later step takes the file id from the `:id` of its resolution entry, never from the key of the resolution map: the map comes from the job result, so what a key holds depends on the result schema of the job-def that wrote it, while the wizard indexes its selection by the id of the entry. The legacy SSE import stays in the backend for other clients.
 - File creation/duplication strips file `:data` before putting file summaries into dashboard state.
 
 ## Viewer
@@ -12,6 +13,6 @@
 - `app.main.ui.ui` parses the viewer's `file-id` query param with `uuid/parse*`, so a missing or malformed id arrives as nil. `viewer-page*` calls `check-file-id!` (same namespace) during render, before mounting `viewer*`: it raises `ex/raise :type :not-found :code :missing-file-id`, which the error boundary turns into the not-found page. Without the guard the `dv/initialize` assertion fails on the nil file-id and the user gets an internal-error page. Test: `frontend-tests.ui.viewer-test/check-file-id-raises-not-found-for-invalid-file-id`.
 - Viewer initialization sets `:current-file-id`, `:current-share-id`, and `:viewer-local`, then fetches the view-only bundle. Comment threads are fetched only for logged-in users.
 - Viewer bundle fetch sends the full supported feature set because anonymous shared viewers may not know team-enabled features.
-- View-only bundles can contain pointer values in `:pages-index` and file data. Viewer resolves those fragments with `:get-file-fragment` before storing the bundle.
+- View-only bundles can contain pointer values in `:pages-index` and file data. Viewer resolves those fragments with `:get-file-fragment` before storing the bundle, through `app.main.features.pointer-map/resolve-pointer` (share-link scoped; the workspace and library loaders use `resolve-file`, which resolves the same two levels). Both fail loudly: a fragment that cannot be resolved rejects the stream and is logged with file id and fragment id instead of silently dropping the page.
 - `bundle-fetched` indexes pages and precomputes viewer frames/all-frames, stores libraries/users/thumbnails/permissions under `:viewer`, then navigates to frame id, query index, or auto-selected frame.
 - Viewer zoom and interaction mode changes update both `:viewer-local` and the `:viewer` route query params. `update-zoom-querystring` guards with a query-param comparison (`not= current expected`) and uses `::rt/replace`; without that guard the load sequence (`bundle-fetched` → `zoom-to-fill` → `update-zoom-querystring` → `rt/nav`) re-runs on every navigation and crashes with React "maximum update depth exceeded" (2.18.0-RC5 regression, fixed in `31b73460c3`; regression test: `bundle-fetched-with-zoom-fill-url-does-not-navigate`).

.serena/memories/frontend/plugin-api-to-cljs-binding.md

@@ -23,6 +23,9 @@
 
 - Public Plugin API objects are lightweight handles, not durable snapshots. Most getters locate fresh state from `app.main.store/state` using hidden `$id`, `$file`, `$page`, etc.
 - `not-valid` logs by default but throws when the plugin flag `throwValidationErrors` is enabled. The MCP execute-code handler deliberately enables that flag while running code.
+- A write to a removed shape always throws `Shape not found`, whatever that flag: setters through the shape proxy's `:on-set u/check-shape-writable` (runs outside the error handling), and methods that must not vanish silently (`resize`, `rotate`) call `u/check-shape-located` first. Methods run inside `:on-error (u/handle-error ...)`, which logs instead of throwing when the flag is off, so it rethrows errors whose `ex-data :type` is `::u/shape-not-found` or `::r/read-only`.
+- A write while the workspace cannot take it (`r/read-only-reason`: save-failure lock `[:workspace-global :persistence-lock]`, `[:permissions :can-edit]` false, version preview `:preview-id`, lost render context `[:render-state :lost]`) always throws, whatever that flag. The `[:workspace-global :read-only?]` flag alone does not block: the Inspect tab sets it and plugins may still write. `r/check-permission` with `content:write`/`library:write` calls `r/check-writable` first, which covers most writers; writers without a permission check (most top-level context methods in `api.cljs`: `create*`, `group`, `align*`, …; z-order methods; text range `applyTypography`; ruler guide `remove`) call `r/check-writable` themselves, and shape setters get it through `u/check-shape-writable`. A new writer without a write-permission check must call it too. An async writer calls it again after each server await, before the next server write or commit (`rx/tap` on the response stream; `dwm/create-svg-shape-with-images` takes it as `:check-writable`), so a lock that lands mid-way rejects the promise with the read-only error. `r/has-permission?` is the check without it, used by the undo blocks so a block opened before a lock still closes. Comments (`comment:write`) stay writable: they do not go through file saves.
+- `penpotMgmt` (`app.plugins.management`) checks permissions with `r/has-permission?` (`granted?`), never `r/check-permission`: its calls act on files through the backend, often from the dashboard with no file open, so the workspace read-only check (`check-writable`) must not block them.
 - `naturalChildOrdering` and `throwValidationErrors` are stored per plugin under `[:plugins :flags plugin-id ...]`; changing default behavior affects automation and MCP diagnostics.
 - Plugin data is stored under keyword namespaces: private data uses `(keyword "plugin" plugin-id)`, shared data uses `(keyword "shared" namespace)`.
 
@@ -31,4 +34,5 @@
 - Plugin listeners are watches on the global store and callbacks are debounced about 10ms. Callback exceptions are caught and logged so plugin code does not crash the app.
 - `selectionchange` callbacks receive arrays of shape id strings, while `filechange`, `pagechange`, and `shapechange` return proxies.
 - `contentsave` fires only when persistence status transitions to `:saved`; it calls the callback with no value.
+- `contentrollback` is not a state watch: it subscribes to `st/stream` for `::dps/rolled-back` (kept in `subscriptions`, disposed by `remove-listener`), not debounced, delivered async so the reverted state is in place. Payload `{fileIds pageIds shapeIds}` from `pcm/rollback-targets` (page shapes only, from the reverted commits' redo changes). When the rollback falls back to reloading the file, `::dps/rolled-back` waits for the reloaded data, so the callback never reads the rejected data. The runtime's `validEvents` must list every event name, or `penpot.on` rejects it; a stale runtime build rejects new events. The MCP plugin forwards these rollbacks with the next task response and the server puts a warning first in that tool response (`RollbackNotices`). Unsent ones survive a tab or plugin reload in `penpot.localStorage` (key `rollbacks/<file-id>`, `LocalStorageRollbackStore`; the MCP manifest in `app.main.data.workspace.mcp` and `mcp/packages/plugin/public/manifest.json` grant `allow:localstorage`, which the runtime checks on access); the entry goes once a response carried them.
 - Plugin history `undoBlockBegin` creates a workspace undo transaction with a JS `Symbol`; `undoBlockFinish` commits that symbol. Missing finish eventually relies on the workspace transaction timeout.

.serena/memories/frontend/routing-app-shell-subtleties.md

@@ -29,3 +29,4 @@ For general store mechanics such as `emit!`, `last-events`, persistence, and und
 
 - Websocket initialization uses `cf/public-uri` joined with `ws/notifications`, converting `http/https` to `ws/wss`, and includes the current `session-id` as query param.
 - Reinitializing or finalizing websocket stops the previous receive stream. Incoming websocket payloads become Potok data events under `app.main.data.websocket/message`.
+- A job the client created is followed with `app.main.data.jobs/watch-job`: it emits the `:job-event` messages of that job (filtered by `:job-id`) and one final outcome read from the row with `get-job`, which is read when the stream starts and again on every `opened` of the websocket, so a job that ended before the client started following it, or while the socket was down, still answers. The stream ends after that outcome, and a read that fails reaches the caller: the repo retries the transient transport errors of a GET, and anything else is a real failure the flow has to show. The `:stage` of a progress event is a stable key of the vocabulary of the worker, never a display string: `app.main.ui.jobs.progress` resolves it with a `case` of literal `tr` calls, falls back to a generic label for a stage it does not know, and the counter of the activity is named after its own stage (the counter of `:files` is the outer context).

.serena/memories/frontend/testing.md

@@ -39,6 +39,7 @@ To assert over emitted event sequences, observe termination: subscribe through `
 - Auto-answering mocks lose deadline expressiveness (can't time answers), need teardown timer-cancellation, and post-teardown deliveries hit real implementations — explicit pushes + async delivery + `wait-for` won on every axis.
 - Teardown belongs to the terminal continuation, never to `finally`-around-triggers (it would dispose in-flight flows).
 - A body that awaits must be `(^:async fn …)` even if the rest is sync; sync sequences are atomic vs the event loop.
+- The `:esm` test build cannot load the CommonJS `sax` module behind `tubax`, so SVG parsing fails in tests (`dwm/valid-svg-string?` answers false, `dwm/svg->clj` errors): mock both with the parsed data.
 
 From `frontend/`:
 - Full unit test run (always builds, suppressed output): `pnpm run test:quiet`.

.serena/memories/frontend/translations.md

@@ -7,10 +7,11 @@ high-coverage support reference, never the base.
 ## Workflow
 
 - Canonicalize with `node ./scripts/translations.js sync -l <locale>` from `frontend/`: sorts entries, syncs `#:` comments from `en`, deletes keys missing in `en`.
+- `sync` never ADDS a key missing in the locale: it only syncs comments (for keys that already exist) and drops keys absent from `en`. A key added to `en` alone stays missing in `es` and its text silently falls back to English. Every new key is born in `en` AND `es` in the same change; compare the `msgid` sets to check (`comm -23 <(grep '^msgid ' en.po | sort -u) <(grep '^msgid ' es.po | sort -u)` must be empty).
 - `sync` copies ALL comment flags from `en`, including `#, fuzzy`. Translated entries must NOT stay fuzzy: strip the flag after translating (mirror `es.po`, which keeps fuzzy only on genuinely untranslated entries). Fuzzy entries are excluded from `msgfmt --statistics` translated counts.
 - Canonical files carry NO `#~` obsolete blocks (`en`/`es` have zero). Drop them; `sync` does not resurrect them.
-- Definition of done: `msgfmt --check <locale>.po` exit 0 and `msgfmt --statistics` shows 0 fuzzy, 0 untranslated.
-- `rehash` scans `frontend/src` AND `common/src` for `(tr "key"` occurrences (any line counts, including `;;` comments) and refreshes `#:` refs in `en`, marking unreferenced keys `#, unused`.
+- Definition of done: `msgfmt --check <locale>.po` exit 0, `msgfmt --statistics` shows 0 fuzzy, 0 untranslated, and the locale's `msgid` set is identical to `en.po`'s (no key missing, none extra).
+- `rehash` scans `frontend/src` AND `common/src` for `(tr "key"` occurrences (any line counts, including `;;` comments) and refreshes `#:` refs in `en`, marking unreferenced keys `#, unused`. The refs in `en` are broadly stale, so a repo-wide `rehash` rewrites hundreds of lines (measured: ~487 on `en` alone): it is a maintenance action on its own, never a step to run inside a change that only adds or edits a key.
 - Dynamic keys are invisible to `rehash`: eliminate them instead of
   declaring them. Preference order: literal `(tr "key")` args only;
   never put a branch inside `tr`, hoist it out (`(if cond (tr "a") (tr "b"))`);
@@ -28,6 +29,12 @@ high-coverage support reference, never the base.
 
 ## Entry rules
 
+- Placement: a key names the feature that renders it (`dashboard.import.*`,
+  `files-export-modal.*`, `jobs.*`); `labels.*` is the bucket for the short
+  generic words reused across screens (Accept, Cancel, Download…), never the
+  home of a feature's own vocabulary. Keep a feature's strings together: the
+  durable jobs write their progress stages as `jobs.progress.stage.*` and
+  their row statuses as `jobs.*`, not half in `labels.*`.
 - Keys resolved dynamically from server-provided data carry a `#, backend`
   flag (set it in `en`, `sync` copies it to the locales): do not remove or
   rename these entries, and keep their `%s` placeholders (the backend only
@@ -40,7 +47,6 @@ high-coverage support reference, never the base.
 - Preserve verbatim: `%s`/`%d`, `{var}`/`{{...}}`, markdown `[text](%s)`, HTML tags, `\n` positions, brand names (Penpot), key names (Ctrl/Shift/Alt), technical terms (SVG, CSS, HSV, RGB).
 
 ## Catalan (ca) conventions
-
 - Normative IEC/Termcat Catalan. Address the user in VOSALTRES (2nd person plural): "Deseu", "Creeu", "Ja teniu un compte?". Buttons/menus use short imperatives ("Crea", "Mou", "Restaura").
 - Established glossary (reuse exactly, do not re-coin): layer=capa, board=tauler, stroke=traç, fill=Emplenat, blur=Difuminat, shadow=Ombra, clipboard=porta-retalls, delete=Elimina, rename=Canvia el nom, shortcut=drecera, grid=graella (keep "grid" where the file already does, e.g. grid-layout editing), plugins/extensions UI=extensions, layout=Disposició, gradient=Degradat, wireframing kept as loanword.
 - Ela geminada uses the middle dot: Cancel·la, paral·lel, al·lega.
@@ -48,6 +54,20 @@ high-coverage support reference, never the base.
 - Shortcut/action names (`shortcuts.*`) are noun/infinitive labels, not sentences. Error strings are direct, no hedging.
 - Same English source in different contexts may legitimately differ (verb "Copia" vs noun "Còpia"; "Desactivat" vs "Deshabilitada" agreeing with "drecera"). Normalize only true duplicates.
 
+## Russian (ru) conventions
+
+- Formal address with capital ВЫ ("Вы уверены…", "Вы собираетесь…"). Buttons/menus use infinitives ("Создать", "Удалить", "Дублировать"), never imperatives.
+- Established glossary (reuse exactly, do not re-coin): layer=слой, board=доска (EN `board` is NEVER кадр — normalized in lote1: toolbar/viewer/shape-menu/header-menu/export-frames/undo/num-of-frames), path/draw tool=контур, stroke=обводка, fill=заливка, blur=размытие, shadow=тень, grid=сетка, clipboard=буфер обмена, shortcuts=сочетания клавиш (normalized in lote1: title/show-shortcuts/not-found), plugin=плагин, layout=макет (`labels.layout` fixed in lote1), gradient=градиент, token=токен, set=набор, theme=тема, file=файл, project=проект, team=команда, library=библиотека, component=компонент.
+- No external RU style guide applies verbatim (Mozilla ru guide is marketing-only, GNOME glossary unmaintained, Adobe uses монтажная область for artboard but `ru.po` majority already says доска — keep доска for consistency).
+- `words.ru.txt` `[function]` DELIBERATELY omits 1-letter words (в/с/к/о/у/а/и) and ending-colliding particles (на/по/не/ли/же/бы/его/их/те/или…): Russian inflects (проект→проекта, запросили=запрос+или), so listing them flags every inflected noun/verb. Never "complete" that list; put tripped valid words in `[ok]`.
+- Never abbreviate т. д. / т. е. in `ru.po`: the lone т/д tokens poison the glue detector's frequency map and flag short words (тон→т+он). Write и так далее out.
+- Tokens-domain glossary (lote2, reuse): alias=алиас, remap=переназначить, viewport=вьюпорт, set=набор, theme=тема, spread (shadow)=распространение, margins=внешние отступы, paddings=внутренние отступы, gaps=промежутки, type identifiers stay in English (Line Height, Font Size, innerShadow/dropShadow, none/Uppercase/Lowercase/Capitalize, thin/light/regular/bold/Italic).
+- Editor-domain glossary (lote3, reuse): variant=вариант, property=свойство, guide=направляющая, board guide=направляющая доски, MCP server=MCP-сервер, size preset=пресет размера, border-radius section=скругление, path handler=маркер, canvas=холст, render=рендер/рендеринг, plugin (never extensión)=плагин.
+- Avoid standalone те (those) in phrasing ("те же" → "такие же"): one lone те validates те+мы splits and flags every темы. Same poisoning class as т. д.
+- `words.ru.txt` `[function]` also omits пока (dropped in lote4): with standalone за anywhere in the file, пока+за flags показа. Rare-typo detection loss accepted.
+- Billing-domain glossary (lote4, reuse): plan names follow shipped `ru.po`, NOT es: Enterprise=Корпоративный, Professional=Профессиональный, Unlimited=Безлимитный, trial=пробный; subscription=подписка, storage=хранилище, seat editors=редакторы, viewer role=наблюдатель, amounts as `175 $/мес.` / `10 ГБ`; access token=токен доступа, MCP key=MCP-ключ (regenerate=перевыпустить), markdown `[text|target:self](%s)` kept verbatim.
+- Closing-domain glossary (lote5, reuse): shortcut noun everywhere=сочетание (shortcuts section titles are noun/infinitive labels, never ярлыки); node=узел, segment=сегмент, measurements=размеры; WebGL unavailable flow uses гайд/руководство по неполадкам + рендеринг WebGL выключен/включён; `ru.po` reached 2275/2275 keys (100%) with 0 fuzzy, 0 untranslated per `msgfmt --statistics`.
+
 ## QA before commit
 
 - Run `node ./scripts/translations.js check -l <locale>` from `frontend/` (no default locale: pass `-l` explicitly): 0 errors required; review warnings by hand. Word lists live in `frontend/scripts/check-translations/words.<locale>.txt` (`[elision]` `[function]` `[common]` `[ok]` `[brands]`); new valid words that trip the gate go to `[ok]`; `check --self-test` covers the detector rules. Without a catalog only the universal checks run. `#, fuzzy` entries are skipped (known-pending, owned elsewhere).

.serena/memories/frontend/ui-conventions-and-style-system.md

@@ -12,7 +12,7 @@
 
 - Co-located SCSS modules are preferred. Use `app.main.style/stl` helpers from CLJS and design-system SCSS tokens/mixins instead of legacy global selectors or high-specificity nesting.
 - Keep CSS specificity low. Avoid nested selectors unless they target elements the component owns; CSS Modules already prevent class-name collisions.
-- Prefer CSS logical properties for directional spacing/layout (`padding-inline-start`, etc.). Physical `width`/`height` are still acceptable where they are clearer.
+- Prefer CSS logical properties for directional spacing/layout (`padding-inline-start`, etc.). Physical `width`/`height` are still acceptable where they are clearer. Never mix: a stylesheet logical property (`inline-size`) and an inline physical `width` on the same element cascade separately, and the logical mapping wins — a dynamically driven width must be `width` on both sides (import modal progress bar hit this). And an inline style passed from CLJS must be a JS object (`#js {:width …}`): for plain `[:div …]` elements the Rumext macro does not convert the `:style` map at runtime — a plain CLJS map reaches React raw and React never finds the keys (static style literals are converted at compile time, which is why `:style` maps seem to work elsewhere).
 - Use named design-system variables/tokens for spacing, borders, fixed dimensions, colors, and typography. Avoid hardcoded px/rem values and deprecated `resources/styles/common/refactor/spacing.scss` variables.
 - Use component-local CSS custom properties for variants and theming instead of one-off Sass variables when a component has multiple visual states.
 - Prefer DS typography components (`heading*`, `text*`) and typography mixins instead of plain text wrappers or deprecated typography mixins.

.serena/memories/frontend/workspace-transform-subtleties.md

@@ -6,6 +6,7 @@
 - `apply-modifiers*` is the lower-level commit path once object/text modifiers are ready. It updates frame guides, frame comment threads, and then emits `update-shapes` with `:reg-objects? true`.
 - Transform commits restrict diff attrs to `transform-attrs` to avoid scanning unrelated shape attrs.
 - Text transforms may carry derived `:position-data`; `assoc-position-data` attaches it while preserving the original text shape context.
+- A drag commits when its `mse/drag-stopper` fires, `:interrupt` (Escape) included. The save-failure lock emits `::plk/cancel-gestures` first: drags built on `plk/gesture-stopper` (start-move, start-resize, start-rotate, start-move-line-point, box/curve/line drawing) see a `cancelled?` flag set before the stop and emit `dwt/cancel-transform` (`clear-local-transform` + `finish-transform`) or `cancel-drawing` instead. A new drag gesture must use it (or, in a component, drop the drag when read-only turns on): `commit-changes` refuses the commit anyway, but the preview (WASM modifiers/props) would stay.
 
 ## Component-copy touched suppression
 

.serena/memories/mcp/core.md

@@ -13,19 +13,26 @@ enabling advanced AI-driven features in Penpot.
 - Build Tool: TypeScript Compiler (tsc) + esbuild
 - Package Manager: pnpm
 
-## General Principles
+## Principles & Code Style
 
 IMPORTANT: Use an idiomatic, object-oriented style.
-In particular, this implies that, for any non-trivial interfaces, you use interfaces that expect explicitly typed abstractions
-rather than mere functions (i.e. use the strategy pattern, for example).
+* You keep each concern in exactly one home.
+  A mechanism whose parts are only correct in combination is implemented as a single component/class, and its parts are private to it;
+  expose a minimal public surface. Java-like encapsulation: If helper functions and constants are only used by one abstraction,
+  make them internal to the respective class.
+* For any non-trivial interfaces, you use functions/methods that expect explicitly typed abstractions
+  rather than mere functions (i.e. use the strategy pattern, for example).
+* Prefer declaring and instantiating concrete classes over defining only interfaces and using object literals.
 
 Comments:
-When describing parameters, methods/functions and classes, you use a precise style, where the initial (elliptical) phrase
-clearly defines *what* it is. Any details then follow in subsequent sentences.
+* When describing parameters, methods/functions and classes, you use a precise style, where the initial (elliptical) phrase
+  clearly defines *what* it is. Any details then follow in subsequent sentences.
+* When describing what blocks of code do, you also use an elliptical style and start with a lower-case letter unless
+  the comment is a lengthy explanation with at least two sentences (in which case you start with a capital letter, as is
+  required for sentences).
 
-When describing what blocks of code do, you also use an elliptical style and start with a lower-case letter unless
-the comment is a lengthy explanation with at least two sentences (in which case you start with a capital letter, as is
-required for sentences).
+Syntax: 
+* Never use if blocks (or other blocks) without braces, even for single-line cases.
 
 ## Project Structure (Excerpt)
 
@@ -102,6 +109,19 @@ From the `mcp/` directory, run
   stdout directly. `NODE_TEST_CONTEXT` is the env var `node --test` sets.
 * Cross-cutting testing principles and anti-patterns: `mem:testing`.
 
+## OAuth for remote MCP clients
+
+- Authorization/discovery URLs, self-contained client IDs, consent and PKCE/code binding, backend Redis state, deployment wiring, and compatibility limits: `mem:mcp/oauth`.
+
+## Backend tools (`submit_feedback`)
+
+- Tools that call the Penpot backend directly (not through the plugin) register only when `PENPOT_MCP_BACKEND_URI` is nonblank (`PenpotMcpServer.initTools`). Currently only `SubmitFeedbackTool`.
+- `PenpotBackendClient` owns the HTTP call and the status-to-message mapping; it posts to `api/main/methods/send-mcp-feedback` with `Authorization: Token <MCP user token>`. The token comes from the session context (OAuth bearer or `?userToken=`); without one the tool fails before calling the backend.
+- Tool zod schemas mirror the backend malli schema (lengths, blank, control characters); keep both in sync. The backend is the authority.
+- Success means the mail job was queued, not delivered; the tool text says so. `formatArgs` hides the feedback from logs.
+- Backend side: `send-mcp-feedback` in `app.rpc.commands.feedback`, configured by `PENPOT_MCP_FEEDBACK_EMAIL`; MCP-token auth rules in `mem:backend/auth-permissions-product-domains`.
+- User docs: `mcp/README.md` (env tables), `docs/mcp/index.md` (tool list), `docs/technical-guide/configuration.md` (MCP feedback section).
+
 ## Devenv plugin/server wiring
 
 In the normal Penpot devenv MCP path, the browser plugin does not discover or route through Postgres. The frontend provides the plugin extension API with `mcp.getServerUrl()`, currently derived from `frontend/src/app/config.cljs` as `penpotMcpServerURI` if set, otherwise `<public-uri>/mcp/ws`. The MCP plugin opens a direct WebSocket to that URL and appends the current MCP access token as a query parameter.
@@ -113,5 +133,10 @@ For parallel devenvs, prefer same-origin MCP routing: each Penpot instance shoul
 ## Plugin reconnect policy
 
 - The plugin treats WebSocket close code `1008` (policy violation) as terminal: it stops auto-reconnecting and stays disconnected until the user explicitly reconnects. Other close codes keep the capped-backoff retry. The decision lives in `ReconnectPolicy.ts` (`shouldReconnectAfterClose`), kept as a pure module so it is unit-testable without DOM/CSS.
-- The MCP server emits `1008` for a duplicate connection on the same user token (`PluginBridge`) and for a missing `userToken` in multi-user mode.
-- A tab rejected with `1008` never reaches `connected`, so the frontend's 60s reconnect watcher (`start-reconnect-watcher` in `app.main.data.workspace.mcp`, started only on `connected`) does not engage; recovery is manual via "Connect here".
+- The MCP server emits `1008` for a missing `userToken` in multi-user mode, for missing/invalid connection metadata, and to close a connection displaced by a newer one with the same user and session ID.
+- Session IDs are deterministic per tab and file, so a reconnect reuses its ID while the old socket may be half-open: the newest connection wins (`PluginBridge.displaceConnection`). Across instances, each new connection publishes a session claim on a per-user Redis channel; an instance displaces its local connection only on a foreign claim that arrives after its own claim (channel order, no clocks). Discovery deduplicates by session ID.
+- A tab rejected with `1008` never reaches `connected`, so the frontend's 60s reconnect watcher (`start-reconnect-watcher` in `app.main.data.workspace.mcp`, started only on `connected`) does not engage; recovery is manual via the MCP menu.
+
+## Rollback warnings
+
+- The plugin's `RollbackLog` (`packages/plugin/src/RollbackLog.ts`) holds the `contentrollback` events not yet sent with a task response; `Task.sendResponse` attaches them and acknowledges them once sent, and the server's `unclaimed-rollbacks` message (a response that came after its task timed out) puts them back. With a `RollbackStore` (`LocalStorageRollbackStore` over `penpot.localStorage`, one entry per file) every change is written through, so a reload of the tab or plugin keeps them; storage failures fall back to memory. Plugin tests run with `node --experimental-strip-types`, which rejects TypeScript parameter properties: declare fields explicitly.

.serena/memories/mcp/oauth.md

@@ -0,0 +1,47 @@
+# MCP OAuth
+
+## Contract and scope
+
+- MCP-specific authorization; existing Penpot login identifies the user on the approval page. No application vetting or persistent consent policy. Access requires user approval; registration alone grants nothing.
+- Return the existing fixed MCP user token. On approval, enable `:mcp-enabled` and create the token only if no usable one exists; preserve an existing usable token. A token is unusable, and replaced, when expired (settings can give MCP tokens an expiry), when its JWT issuer is not `urn:penpot:mcp-token` (tokens issued before MCP tokens had their own issuer), or when this instance cannot decrypt it (e.g. secret key changed); the backend rejects all three. No refresh token, `expires_in`, or new token lifetime.
+- No new client database table or schema migration. Client IDs must remain usable across separate authorization flows without server-side registration storage. Do not replace them with expiring Redis registrations.
+- MCP transport continues matching user tokens to connected Penpot plugins; no new per-request token introspection or independent semantic validation. The logged-in Penpot instance remains the content-access boundary. Do not silently add resource-server validation or change token semantics.
+- README scope: user/developer usage instructions; document `PENPOT_MCP_AUTH_URI` in its environment-variable table. Keep OAuth architecture and protocol details here and non-obvious invariants in source comments.
+
+## Deployment and discovery
+
+- MCP process: `PENPOT_MCP_AUTH_URI=https://penpot.example/api/mcp/oauth` with `--multi-user`. Unset/empty disables OAuth; configured OAuth requires multi-user. Absolute HTTPS, or HTTP on loopback for development; no userinfo, query, or fragment. Invalid configuration fails startup; startup logs OAuth status.
+- Devenv: `mcp/scripts/start-mcp-devenv-oauth` starts multi-user mode with `PENPOT_MCP_AUTH_URI` derived from `PENPOT_PUBLIC_URI`. Manual opt-in only: agentic tooling needs the single-user `start-mcp-devenv`.
+- Public MCP transport: `<Penpot public URI>/mcp/stream`, proxied to Node `/mcp`. Missing credentials with OAuth enabled: 401 plus `WWW-Authenticate` referencing `<auth base>/resource` and scope `mcp`.
+- Bearer header takes precedence over legacy `?userToken=`. Malformed headers do not fall back to the query token. Existing query-token clients remain supported.
+- Protected-resource metadata: `/api/mcp/oauth/resource`; issuer: `/api/mcp/oauth`; authorization-server metadata: `/.well-known/oauth-authorization-server/api/mcp/oauth`. All reach the backend. Bundled devenv/production nginx configs forward the well-known path; custom proxies must do likewise. Keep Penpot public URI and external URLs consistent.
+- Node implementation: `mcp/packages/server/src/McpAuthentication.ts`, integrated by `PenpotMcpServer.ts`; OAuth-enabled tools advertise `_meta.securitySchemes`.
+
+## Client identification
+
+- `backend/src/app/mcp/oauth_clients.clj`: dynamic registration at `/api/mcp/oauth/register`, HTTPS client metadata documents (CIMD), and registrations obtained in advance via `(app.mcp.oauth-clients/register! {} metadata)`.
+- DCR returns `penpot-mcp-v1.` plus unpadded base64url JSON: normalized callback URLs, optional name, authentication method, random nonce, optional secret hash. Fixed grant/response/scope values are reconstructed. Nonce distinguishes registrations with otherwise identical metadata.
+- IDs are public, unsigned, unencrypted, non-expiring metadata; validate on every resolution. Client secrets are generated randomly and returned separately; only the hash enters the ID. No signing/encryption key or client record in PostgreSQL/Redis. Old experimental database-backed UUID IDs require registration again.
+- An edited payload changes the ID. Authorization-code exchange compares the exact ID string against the approved request before resolving metadata. Do not replace this with comparison of decoded fields; code ownership and PKCE binding must survive metadata edits.
+- Supported authentication: `none`, `client_secret_basic`, `client_secret_post`; no `private_key_jwt`. CIMD selects public-client `none` from the document's supported methods; other listed methods do not imply support here. Scope `mcp`; grant `authorization_code`; response `code`. CIMD may advertise additional grants (e.g. Codex lists `refresh_token`); require `authorization_code` support and normalize to the grant Penpot implements, rather than reject broader client capabilities.
+- Entire ID capped at 2,048 ASCII characters. DCR returns `invalid_client_metadata` if encoded metadata exceeds this; shorter/fewer callback URLs reduce size. OAuth defines no fixed ID-length maximum (RFC 6749 §2.2); stateless IDs are explicitly described by RFC 7591 §A.5.2. Installed MCP client SDK 2.3.1 accepts a 2,048-character ID; this does not establish Codex/Claude compatibility.
+- Redirect matching is exact except loopback HTTP port changes for native clients (`127.0.0.1`, `[::1]`, exact `localhost`). Host, path and query must still match; reject fragments/userinfo and lookalike localhost domains. Allow HTTPS, loopback HTTP, and reverse-domain native schemes.
+- CIMD: fetch with existing SSRF-checked `app.http.client/req`, no redirect following, require matching document `client_id`, bounded 64-KiB body and request/body-read timeouts. Cache max 256 entries/one minute, further limited by HTTP max-age; no-store/no-cache/no max-age means no reuse.
+
+## Approval and exchange
+
+- `backend/src/app/http/mcp_oauth.clj`: separate router branch from ordinary RPC middleware; preserve OAuth snake_case JSON and form encoding. Session auth applies to approval request/decision only. Public metadata/register/token endpoints allow noncredentialed CORS; MCP transport CORS was not added.
+- Errors: every failure answers with an OAuth `{"error": ...}` body, never the shared Penpot error shape. Failures without `:oauth-error` are also passed to `app.http.errors/handle` for logging only; its response is discarded. `wrap-errors` also wraps the session middleware of `/request` and `/decision`. Safe to log because these routes never set request `:params`.
+- `/authorize`: validate client/callback, `code`, PKCE S256, scope/resource/state; store pending request for 600 seconds; redirect to `?screen=mcp-authorize&request=<opaque handle>`.
+- `frontend/src/app/main/ui/mcp_authorize.cljs`: existing login with return URL; show application and account details; submit explicit Allow/Cancel. No fixed MCP token in frontend state or browser redirect. Approval refreshes profile and broadcasts MCP enablement; does not connect a plugin automatically.
+- `/request`: first authenticated read binds request to profile plus hashed login-session ID and CSRF value, using compare-and-replace. Re-read after the attempt: concurrent GETs may win/lose binding; only the matching session may proceed.
+- `/decision`: check exact Origin, session binding and CSRF, then atomically consume the request. Denial creates no token/enables nothing. Approval calls transactional `access-token/ensure-mcp-access!`, then stores code for 120 seconds. Code state holds token row ID, never the fixed token. Callback echoes OAuth `state` and RFC 9207 `iss`.
+- `/token`: validate grant, exact client ID/callback/resource and PKCE; resolve current client metadata/authentication method and verify any secret; atomically consume validated code; fetch the unexpired token by original row ID/profile/type. Regeneration deletes the old token row, so a pending code cannot silently yield the replacement token. Return fixed `access_token`, `token_type: Bearer`, scope `mcp`.
+- `backend/src/app/mcp/oauth_state.clj` and Lua: tenant/kind/hash-prefixed keys; random handles; SET NX EX must succeed before returning handle. Reads retain raw JSON for atomic comparison. Consume/delete and bind/replace compare the validated raw record in Lua; replacement preserves TTL. Redis failures fail closed. Backend Redis is required independently of Node's optional Redis plugin bridge.
+- `backend/src/app/rpc/commands/access_token.clj`: `ensure-mcp-access!` decodes the stored token with `ex/ignoring` to apply the usable-token rule above. Lock profile row in both MCP generation and OAuth provisioning, including when no token exists. Serializes simultaneous approvals and settings regeneration. Provisioning must run within a transaction.
+
+## Verification and limits
+
+- Tests: `mcp_oauth_clients_test.clj`, `mcp_oauth_http_test.clj`, `mcp_oauth_state_test.clj`, OAuth provisioning in `rpc_access_tokens_test.clj`; MCP authentication/HTTP tests; frontend approval tests. Cover no-storage registration, size boundaries, edited-ID rejection, code replay/concurrency, session/CSRF binding, and fixed-token reuse.
+- Live OpenAI/Codex/Claude compatibility and marketplace acceptance remain unverified. In particular, verify token persistence without `expires_in` and acceptance of the existing matching-only resource boundary; do not claim protocol conformance from SDK tests alone.
+- Protocol references: https://www.rfc-editor.org/rfc/rfc6749.html#section-2.2 ; https://www.rfc-editor.org/rfc/rfc7591.html#appendix-A.5.2 ; https://www.rfc-editor.org/rfc/rfc7636.html ; https://www.rfc-editor.org/rfc/rfc9207.html .

.serena/memories/prod-infra/core.md

@@ -4,32 +4,36 @@ Backend (`app.config`, `PENPOT_*` env vars) is parameterized; deployments choose
 
 ## Services
 
-- **PostgreSQL**: durable store. Profiles, teams, files, sessions, audit, `storage_object` metadata, the `task` queue, `scheduled_task` cron registry, migrations. File-data also lives here when the file-data backend is `legacy-db`/`db`. One shared DB across all backends.
+- **PostgreSQL**: durable store. Profiles, teams, files, sessions, audit, `storage_object` metadata, the `job` work queue (plus the dormant legacy `task` table), `scheduled_task` cron registry, migrations. File-data also lives here when the file-data backend is `legacy-db`/`db`. One shared DB across all backends.
 - **Redis (Valkey-compatible)**: per-backend message bus and cache. Concrete uses: msgbus Pub/Sub for collaborative-editing broadcasts and team/profile-org notifications fired by RPC handlers (`app.rpc.notifications`, `files_update`, `teams`, `websocket`); file-summary cache gated by `enable-redis-cache`; rate-limit counters; and the dispatcher→runner work hand-off list `penpot.worker.queue:<tenant>:<queue>`. `PENPOT_REDIS_URI`. A burst of client-side `RedisCommandTimeoutException` (2s on `DEL`/`SET`, 10s on `BLPOP`) with the exporter healthy is the JVM paused or starved, not Redis: the timeout fires because nothing reads the reply.
 - **Object storage**: backends `:s3` and `:fs`. S3 in prod; devenv uses RustFS. Holds uploaded media, file-data when the file-data backend is `storage`, exports. Backend-side details (resolve, dedup, bucket set, object lifecycle, and file-data backends): `mem:backend/storage`.
-- **SMTP mailer**: invitations, password resets, email verification (sent via the `:sendmail` worker task).
+- **SMTP mailer**: invitations, password resets, email verification (sent via the `:sendmail` worker job).
 - **LDAP** (optional auth provider): helpers in `app.auth.*`, gated by `enable-login-with-ldap`.
 
 ## Task queue and worker model
 
-Async tasks are enqueued via `wrk/submit!` (`app.worker`), which inserts a row into the shared Postgres `task` table tagged with `queue = "<tenant>:<queue-name>"`. Submission is **fire-and-forget** — RPC handlers never poll, never wait, and workers never publish to msgbus. The only completion signal is the `task` row's `status` / `completed_at` columns, which nothing in `rpc/` reads. Soft-delete RPCs return immediately after marking the top-level row, leaving the cascade and reaping to workers.
+Async jobs are enqueued via `jobs/submit` (`app.jobs`), which inserts a row into the shared Postgres `job` table tagged with `queue = "<tenant>:<queue-name>"`. Submission is **fire-and-forget** — RPC handlers never poll and never wait. The completion signal is the `job` row's `status` / `completed_at` columns, which the creation commands read back to answer with the state of the job they just created. A job submitted with `::jobs/profile-id` also publishes a `:job-event` message on that profile's msgbus topic after each transaction commits, and the binfile commands (`create-export-binfile-job`, `create-import-binfile-job`) are the callers that pass that owner, so the progress of their jobs does reach the msgbus. Soft-delete RPCs return immediately after marking the top-level row, leaving the cascade and reaping to workers.
 
-Workers run on backends with `enable-backend-worker` in `PENPOT_FLAGS`. Each worker-enabled backend has a `dispatcher` (polls `task` with `FOR UPDATE SKIP LOCKED`, marks status='scheduled', RPUSHes claimed task IDs into **its own** Redis list) and one or more `runner`s per queue (BLPOP from that same local list, execute, update the Postgres row). The Redis hand-off list is purely intra-backend — cross-backend coordination happens at the Postgres row level.
+Workers run on backends with `enable-backend-worker` in `PENPOT_FLAGS`. Each worker-enabled backend has a `dispatcher` (polls `job` with `FOR UPDATE SKIP LOCKED`, marks status='scheduled', RPUSHes the JSON `[id scheduled-at]` payloads into **its own** Redis list) and one or more `runner`s per queue (BLPOP from that same local list, execute, update the Postgres row). The Redis hand-off list is purely intra-backend — cross-backend coordination happens at the Postgres row level. The queues are `default`, `webhooks`, `cron` and `binfile`: the last one carries the heavy work of a user job (`:worker-binfile-parallelism`, default 1), so a long export or import does not block the default runner.
 
 ## Cross-backend safety
 
-Postgres row locking is the only correctness primitive: `task` claims via `FOR UPDATE SKIP LOCKED`, cron firing via `FOR UPDATE SKIP LOCKED` on the `scheduled_task` row, plus task-handler-internal locks (e.g. `file_gc_scheduler` locks candidate file rows). This makes the work-claim path safe across any number of worker-enabled backends.
+Postgres row locking is the only correctness primitive: `job` claims via `FOR UPDATE SKIP LOCKED`, cron firing via `FOR UPDATE SKIP LOCKED` on the `scheduled_task` row, plus job-handler-internal locks (e.g. `file_gc_scheduler` locks candidate file rows). This makes the work-claim path safe across any number of worker-enabled backends.
 
 Two known race patterns survive multi-backend operation:
 
-- **Cron dedup is best-effort.** The lock on `scheduled_task` is released when the task body finishes. If two backends' cron timers fire for the same scheduled instant with a gap larger than the task body's runtime, both execute it. Penpot's cron entries are idempotent (`session-gc`, `objects-gc`, `storage-gc-*`, `tasks-gc`, `upload-session-gc`, `file-gc-scheduler`); the exceptions are `:telemetry` (would double-report) and `:audit-log-archive` (depends on archive target idempotency).
-- **`wrk/submit! ::dedupe true`** does a non-atomic `DELETE` then `INSERT`. Concurrent cross-backend submits can both bypass the `DELETE` (each sees the other's uncommitted insert as absent) and end up with duplicate `'new'` rows. Each row claims and runs once independently, so the underlying work is fine; the "at most one pending" guarantee weakens.
+- **Cron dedup is best-effort.** The lock on `scheduled_task` is released when the task body finishes. If two backends' cron timers fire for the same scheduled instant with a gap larger than the task body's runtime, both execute it. Penpot's cron entries are idempotent (`session-gc`, `objects-gc`, `storage-gc-*`, `storage-pending-gc`, `jobs-gc`, `file-gc-scheduler`); the exceptions are `:telemetry` (would double-report) and `:audit-log-consumer-nexus` / `:audit-log-consumer-admin-console` (depend on archive target idempotency).
+- **`jobs/submit ::dedupe true`** does a non-atomic `DELETE` then `INSERT`. Concurrent cross-backend submits can both bypass the `DELETE` (each sees the other's uncommitted insert as absent) and end up with duplicate `'new'` rows. Each row claims and runs once independently, so the underlying work is fine; the "at most one pending" guarantee weakens.
 
 Penpot in production lives with both: horizontal-scale deployments accept "exactly-once" as "essentially-once for idempotent operations." Devenv parallel instances handle it by running workers only on ws0 (see `mem:devenv/core`).
 
+## Jobs observability
+
+Worker-enabled, writable backends publish unified jobs metrics through the normal Prometheus registry. Counters describe submit, dispatch, terminal outcomes, retries, orphan recovery, rescheduling and total stored job events (unlabelled); histograms separate queue wait, handler execution and total job age. SQL-backed events are recorded after the outer PostgreSQL transaction commits. The `::jobs-metrics/sampler` component publishes backlog by status and the oldest pending-job age every 30 seconds. All labels are bounded operational values; job IDs, profile IDs, params and error text are excluded. Read-only backends do not start the sampler or the workers.
+
 ## Draining one instance
 
-Each backend has a per-process availability flag (`app.health`, a global atom). While offline, `/readyz` answers `503 OFFLINE` without querying the database, so the load balancer removes that instance from rotation while requests in flight still finish. Operator surface: `app.srepl.main/set-offline` / `set-online`; a restart comes back online. In-flight worker tasks are safe: tasks claimed as `scheduled` return to `new` after 5 minutes via `sql:reschedule-lost` and another instance picks them up.
+Each backend has a per-process availability flag (`app.health`, a global atom). While offline, `/readyz` answers `503 OFFLINE` without querying the database, so the load balancer removes that instance from rotation while requests in flight still finish. Operator surface: `app.srepl.main/set-offline` / `set-online`; a restart comes back online. In-flight worker jobs are safe: jobs claimed as `scheduled` return to `new` after 5 minutes via `sql:reschedule-lost` and another instance picks them up.
 
 ## Touching production data
 
@@ -46,5 +50,4 @@ Production runs the deployed toolbox, not the devenv one:
 - Devenv composition and the ws0-only worker placement: `mem:devenv/core`.
 - Storage backend resolution, dedup, bucket behavior, object lifecycle, and file-data lifecycle: `mem:backend/storage`.
 - Storage operation metrics (S3 API calls, logical ops, dedup, asset requests) and the logical-bucket vs physical-target distinction: `mem:backend/storage` (Metrics section).
-- Storage operation metrics (S3 API calls, logical ops, dedup, asset requests) and the logical-bucket vs physical-target distinction: `mem:backend/storage` (Metrics section).
 - Grafana (datasource ids and how to use them, main dashboard, panel conventions and gotchas): `mem:prod-infra/grafana`.

.serena/memories/scripts/ci.md

@@ -5,6 +5,12 @@ the production build of each module, for one or more monorepo modules, and
 prints a per-task summary. It is the local equivalent of CI; use it to
 verify changes before declaring work done.
 
+It also exports `PENPOT_FLAGS` with `enable-backend-asserts` appended (keeping
+whatever the caller already set), which is what puts `clojure.core/assert`
+into the compiled test code. Prefer it over calling `clojure -M:dev:test` by
+hand for that reason alone: without the flag the suite passes while every
+assertion is compiled out. GitHub Actions sets the same flag.
+
 ## When to use
 
 - After implementing or fixing code in a module: run its checks before

.serena/memories/workflow/updating-node.md

@@ -1,15 +1,16 @@
 # Updating Node Across the Toolchain
 
-No single source of truth: nothing reads `.nvmrc` at build time, and the version is repeated in a different shape per file. Six sites move together.
+No single source of truth: nothing reads `.nvmrc` at build time, and the version is repeated in a different shape per file. Six sites move together, and the image ones carry a digest beside the tag.
 
 ## Pin sites
 
 - `.nvmrc` → `v<X.Y.Z>`, with the leading `v`.
-- `docker/devenv/Dockerfile`, `setup-node` stage → `ENV NODE_VERSION=v<X.Y.Z>`; downloads `https://nodejs.org/dist/${NODE_VERSION}/node-${NODE_VERSION}-linux-{arm64,x64}.tar.gz` and verifies each with `NODE_SHA256` in the same arch case. Bumping Node means recomputing both hashes from the published tarballs; only the arch SHAs change, the structure stays.
+- `docker/devenv/Dockerfile`, `setup-node` stage → `ENV NODE_VERSION=v<X.Y.Z>`; downloads `https://nodejs.org/dist/${NODE_VERSION}/node-${NODE_VERSION}-linux-{arm64,x64}.tar.gz`, one tarball per arch, and verifies each against a `NODE_SHA256` right below its `BINARY_URL`. Both hashes come from `SHASUMS256.txt` in that same dist directory, so a bump must recompute the two; only the arch SHAs change, the structure stays.
 - `docker/images/Dockerfile.mcp` → two stages: `FROM dhi.io/node:<X.Y.Z>-debian13-dev AS build` and `FROM dhi.io/node:<X.Y.Z>-debian13 AS image`. The release stage carries no `-dev`.
 - `docker/images/Dockerfile.media-processor` and `docker/images/Dockerfile.exporter` → single `FROM dhi.io/node:<X.Y.Z>-debian13-dev`.
 - `.github/workflows/plugins-deploy-api-doc.yml` and `.github/workflows/plugins-deploy-styles-doc.yml` → `runtime: node@<X.Y.Z>` in the `pnpm/setup` step. Its `working-directory: plugins` makes pnpm's own version come from the manifest, so this pin is the Node one; keep it equal to `.nvmrc` minus the `v`.
 - DHI tags take the bare version; only the devenv `NODE_VERSION` carries the `v`.
+- Every `dhi.io` base image in the repo (the node ones above, plus `debian-base` and `nginx`) is pinned by digest: `FROM dhi.io/<image>:<tag>@sha256:<digest>`. Tag and digest are a pair; Renovate's `Update DHI base images` commits rewrite both in one go, so a hand-made tag bump that leaves the digest behind keeps the old image.
 
 ## Gotchas
 
@@ -22,5 +23,5 @@ No single source of truth: nothing reads `.nvmrc` at build time, and the version
 
 ## Verification
 
-- `rg '<old version>'` finds nothing outside `CHANGES.md` and untracked scratch files (a repo-wide grep without path filters hits multi-megabyte ClojureScript analyzer caches: always scope it).
+- `rg '<old version>'` finds nothing outside `CHANGES.md` and untracked scratch files (a repo-wide grep without path filters hits multi-megabyte ClojureScript analyzer caches: always scope it). Grep `dhi\.io/node` as well: a line whose tag moved but whose digest did not still pulls the old image.
 - After a devenv rebuild, `node --version` matches `.nvmrc` and `pnpm --version` matches `PNPM_VERSION` in `docker/devenv/Dockerfile`.

.serena/memories/workflow/updating-pnpm.md

@@ -23,7 +23,7 @@ file (never pipe tool output through filters).
 - The store survives `node_modules` cleans. It is content-addressed and
   integrity-verified, so it cannot go stale; staleness lives in
   node_modules. Only `scripts/clean-node-modules --store` removes it.
-- Every first-party `package.json` (35 files: workspace roots plus
+- Every first-party `package.json` (34 files: workspace roots plus
   members) must carry a `packageManager` field with the identical
   `pnpm@<version>+sha512.<hash>` value. Do not let them drift.
   `external/` (vendored trees with their own lifecycles), `.opencode/`,
@@ -60,7 +60,7 @@ file (never pipe tool output through filters).
 3. From the repo root, on a host whose system pnpm is the target version
    (rebuilt devenv), run `scripts/sync-pnpm-version`. It resolves the
    integrity hash via `pnpm view` + node (no npm, no corepack, no
-   python3) and stamps the identical field into all 35 files, replacing
+   python3) and stamps the identical field into all 34 files, replacing
    only the value line (each file keeps its own indent) and inserting
    the key after `"name"`/`"version"` where missing. Explicit version
    instead: `scripts/sync-pnpm-version <version>`. The explicit form works
@@ -89,8 +89,8 @@ file (never pipe tool output through filters).
   so a nested one silently forks install and lockfile behavior. Do not
   reintroduce it.
 - `media-processor/target/package.json` is a generated, gitignored copy that
-  the script also stamps, so stamping and `--check` report 36 files while
-  only 35 are tracked. A stale copy there makes `--check` fail with nothing
+  the script also stamps, so stamping and `--check` report 35 files while
+  only 34 are tracked. A stale copy there makes `--check` fail with nothing
   wrong in the repo: delete `media-processor/target` and re-run.
 - Expect metadata-only lockfile diffs when only the pnpm version moves:
   the pnpm self-reference entries, plus a new `packageManagerDependencies`

.serena/project.yml

@@ -1,42 +1,54 @@
 # the name by which the project can be referenced within Serena/when chatting with the LLM.
 project_name: "penpot"
 
-# list of languages for which language servers are started (LSP backend only); choose from:
-#   ada                 al                  angular             ansible             bash
-#   bsl                 clojure             cpp                 cpp_ccls            crystal
-#   csharp              csharp_omnisharp    cue                 dart                elixir
-#   elm                 erlang              fortran             fsharp              gdscript
-#   go                  groovy              haskell             haxe                hlsl
-#   html                java                json                julia               kotlin
-#   latex               lean4               lua                 luau                markdown
-#   matlab              msl                 nix                 ocaml               pascal
-#   perl                php                 php_phpactor        php_phpantom        powershell
-#   python              python_jedi         python_pyrefly      python_ty           r
-#   rego                ruby                ruby_solargraph     rust                scala
-#   scss                solidity            svelte              swift               systemverilog
-#   terraform           toml                typescript          typescript_vts      vue
-#   yaml                zig
+# list of language servers to start when using the LSP backend; choose from:
+#   ada                    al                     angular                ansible                astro
+#   bash                   bsl                    clojure                cpp                    cpp_ccls
+#   crystal                csharp                 csharp_omnisharp       cue                    dart
+#   deno                   elixir                 elm                    erlang                 fortran
+#   fsharp                 gdscript               gleam                  go                     groovy
+#   haskell                haxe                   hlsl                   html                   java
+#   json                   julia                  julia_fatou            kotlin                 latex
+#   lean4                  lua                    lua_emmylua            luau                   markdown
+#   matlab                 msl                    nextflow               nix                    ocaml
+#   pascal                 perl                   php                    php_devsense           php_phpactor
+#   php_phpantom           powershell             python                 python_basedpyright    python_jedi
+#   python_pyrefly         python_ty              qml                    r                      rego
+#   ruby                   ruby_solargraph        rust                   scala                  scss
+#   solidity               svelte                 swift                  systemverilog          terraform
+#   toml                   typescript             typescript_vts         vue                    wolfram
+#   yaml                   zig
 #   (This list may be outdated; generated with scripts/print_language_list.py;
-#   For the current list, see values of Language enum here:
+#   For the current list, see values of the LanguageServerId enum here:
 #   https://github.com/oraios/serena/blob/main/src/solidlsp/ls_config.py)
-# For some languages, there are alternative language servers, e.g. csharp_omnisharp, ruby_solargraph.)
+# For some languages, there are several alternative language servers, e.g. csharp_omnisharp, ruby_solargraph.)
 # Note:
 #   - For C, use cpp
 #   - For JavaScript, use typescript
 #   - For Angular projects, use angular (subsumes typescript+html; requires `npm install` in the project root)
+#   - For Astro projects, use astro (subsumes typescript/javascript for .astro projects; requires npm)
 #   - For Svelte projects, use svelte (subsumes typescript/javascript for .svelte projects; requires npm)
+#   - For Deno projects, use deno (serves the same .ts/.js files as typescript; requires the deno CLI on PATH)
 #   - For SCSS / Sass / plain CSS, use scss (some-sass-language-server handles all three)
 #   - For Free Pascal/Lazarus, use pascal
+#   - External Python adapters may add further registered IDs; install the adapter package first
+#     and then use its ID here, for example: example
 # Special requirements:
-#   Some languages require additional setup/installations.
+#   Some language servers require additional setup/installations.
 #   See here for details: https://oraios.github.io/serena/01-about/020_programming-languages.html#language-servers
-# When using multiple languages, the first language server that supports a given file will be used for that file.
-# The first language is the default language and the respective language server will be used as a fallback.
+# When using multiple language servers, the first language server that supports a given file will be used for that file.
+# The first language server is the default language and the respective language server will be used as a fallback.
 # Note that when using the JetBrains backend, language servers are not used and this list is correspondingly ignored.
+language_servers:
+  - clojure
+  - typescript
+  - rust
+
+# same list for Serena 1.x, which only reads this key; keep both lists equal
 languages:
-- clojure
-- typescript
-- rust
+  - clojure
+  - typescript
+  - rust
 
 # the encoding used by text files in the project
 # For a list of possible encodings, see https://docs.python.org/3.11/library/codecs.html#standard-encodings
@@ -177,3 +189,18 @@ activation_command:
 # maximum time in seconds to wait for activation_command to complete before killing it (default 180s).
 # must be a positive number.
 activation_command_timeout: 180.0
+
+# list of APIs (facades or facade methods, e.g. "lsp" or "lsp.get_diagnostics_for_symbol") to include in the REPL
+# that would otherwise be disabled (particularly optional methods, which are disabled by default).
+# This extends the existing inclusions (e.g. from the global configuration).
+included_apis: []
+
+# list of APIs (facades or facade methods, e.g. "lsp" or "lsp.find_symbol") to exclude from the REPL.
+# This extends the existing exclusions (e.g. from the global configuration).
+excluded_apis: []
+
+# The interface through which the agent (LLM) accesses Serena's functionality (overrides the global setting).
+# Valid values: tools, REPL (see the global configuration for details); leave empty to use the global setting.
+# Note: the interface is fixed at startup. If a project is activated post-init, its setting is not applied.
+agent_interface: REPL
+

AGENTS.md

@@ -33,7 +33,7 @@
 
 Do this **before planning, before coding, before touching any file**:
 
-1. Read `critical-info` (use `serena_read_memory critical-info` or read `.serena/memories/critical-info.md`).
+1. Read `critical-info` through Serena (see "Accessing memories" below) or read `.serena/memories/critical-info.md`.
    It describes the project structure and tells you which modules exist.
 2. From `critical-info`, identify which modules your task affects.
 3. Read each affected module's **core memory** — the name is `<module>/core`
@@ -104,9 +104,12 @@ When working on a task:
 
 ## Accessing memories
 
-- **If `serena_read_memory` / `serena_list_memories` tools are available**: use them.
-  `serena_read_memory` takes a memory name (e.g. `critical-info`, `frontend/core`).
-- **If tools are NOT available**: read the filesystem directly.
+- **If Serena's `serena_repl` tool is available** (Serena 2.x): call `s.mem.read_memory(name)`
+  and `s.mem.list_memories()` through it.
+- **If Serena's `read_memory` / `list_memories` tools are available** (Serena 1.x; some clients
+  show them as `serena_read_memory` / `serena_list_memories`): use them.
+- Either way, `read_memory` takes a memory name (e.g. `critical-info`, `frontend/core`).
+- **If Serena is NOT available**: read the filesystem directly.
   Memory name `mem:foo/bar` maps to file `.serena/memories/foo/bar.md`.
 
 ## Cross-reference convention

backend/package.json

@@ -4,7 +4,7 @@
   "license": "MPL-2.0",
   "author": "Kaleidos INC Sucursal en España SL",
   "private": true,
-  "packageManager": "pnpm@12.16.1+sha512.091d1dee7927575661b25dab3c2ae31cafc1c54d6b71ed6a6600cf0c1cb10434a447eae14015f1b23e2e6bedeed79df94a932e2164d2832280dce3fb32f656d1",
+  "packageManager": "pnpm@12.18.0+sha512.9d7882d5dc82147ecfc4e062aa34834e0ad0296710a1837a1115ae012127d36b7109e321ac1598a1dd0a65597681b1e99917e12ed0d749eae91187943363beb3",
   "repository": {
     "type": "git",
     "url": "https://github.com/penpot/penpot"

backend/pnpm-lock.yaml

@@ -7,153 +7,153 @@ importers:
     configDependencies: {}
     packageManagerDependencies:
       pnpm:
-        specifier: 12.16.1
-        version: 12.16.1
+        specifier: 12.18.0
+        version: 12.18.0
 
 packages:
 
-  '@pnpm/exe.android-arm64@12.16.1':
-    resolution: {integrity: sha512-PKf78Ew+i0CBYf0Xc+eMcqQiqyBT+u8XCVg1F5roc98/6wWfdYlWhrUMmEBiKWaJ3JqHxrtt8l1NwZ7qfJt2/Q==}
+  '@pnpm/exe.android-arm64@12.18.0':
+    resolution: {integrity: sha512-fOF6IwgvnUKEmVedGqiAsNPfN2Ry8GvaqzSQOiF3RJFbpBfPKO/6YT54s1GjToyYinRxw9preHHPe4y2N3SzhA==}
     cpu: [arm64]
     os: [android]
 
-  '@pnpm/exe.android-x64@12.16.1':
-    resolution: {integrity: sha512-xdu7gMbB98pdvhwMeA6612Of515FDlRUtfA/1f7nWPZfPLnEdSRBO2u+1N0XBtQ/ennhPH/lrxB1gC6gnR/cdg==}
+  '@pnpm/exe.android-x64@12.18.0':
+    resolution: {integrity: sha512-6OiolnaCsB3/GcJ9XNrenbUH3zFHRJ+5QynrMUUEyThbgnnCzWxa199hpQlOmxN4rAXuYQ0IYgWFNUZxyzg31Q==}
     cpu: [x64]
     os: [android]
 
-  '@pnpm/exe.darwin-arm64@12.16.1':
-    resolution: {integrity: sha512-pVbmIIbhixiLvADLu1WA3TaW8FZy/D72R1De27WbbFx8/2mxvnsTyFVm4fO9kjpjMY7RzJd48CiQ96p8fO3ebQ==}
+  '@pnpm/exe.darwin-arm64@12.18.0':
+    resolution: {integrity: sha512-zkq+4YzVIX39ETYLyC3W7LY7MpHEU/J3nRhHl4i2yPld2Cf2+rEKRh3ejJ4AmaQ77HHJLCktY4Z3jq0fpcxFXg==}
     cpu: [arm64]
     os: [darwin]
 
-  '@pnpm/exe.darwin-x64@12.16.1':
-    resolution: {integrity: sha512-xZvQBFqcZjO7vFKTK3XsJpqY6lnBL9EWSLCzVUPq3kZ94+D96gwJCa76xxGNK8Cj32f4eeM91Qg+yNRR4sRpDw==}
+  '@pnpm/exe.darwin-x64@12.18.0':
+    resolution: {integrity: sha512-qMsL9qx92unOTa6hgkfiTJv8ceZaTx0IQg7InhYDVX2ZrTECujaVEEVypnJK5m3eGUsP1KD7Y4qIgleCwfRsYw==}
     cpu: [x64]
     os: [darwin]
 
-  '@pnpm/exe.freebsd-x64@12.16.1':
-    resolution: {integrity: sha512-zTtunfi9mBhRXiXEJYfwiJj0tk7MgTSfqDSRh+Az9FZ9THazVIiQiVNS8jpcsftDOqu7LKNffQdjmgwHDThfUg==}
+  '@pnpm/exe.freebsd-x64@12.18.0':
+    resolution: {integrity: sha512-/9vy/4ms52cU4MjZ5NhHd6N2d9L9W0kARf5/ff2tweAHu0VeUzs/0qp/7ZHFuddm2Wa/m0AKQoPum9AVUCqKsw==}
     cpu: [x64]
     os: [freebsd]
 
-  '@pnpm/exe.linux-arm64-musl@12.16.1':
-    resolution: {integrity: sha512-BJ3mOmcmQgcUo92Ee/WfEOt1pRiv3y2MMAkhEiYHd85UiPpHTfFhv+SpCbALmfExzXk3UlaRtMYwHlkQymc4VA==}
+  '@pnpm/exe.linux-arm64-musl@12.18.0':
+    resolution: {integrity: sha512-Vp+TIMPPyackVCXkOdEKH/RmAn/r07EcIyrgZYQK7VrkksqZOie0GnWIx+85mPqeFEYCRtXIZAFJ9L4ceTndDg==}
     cpu: [arm64]
     os: [linux]
     libc: [musl]
 
-  '@pnpm/exe.linux-arm64@12.16.1':
-    resolution: {integrity: sha512-JaRd5gNxwdNGBBkrZFt8bknrK4YmHVJBW0JMhUQ0fJ0dEYp6C2GT9ExaCqOO8EwYEyqM+jpKgJIK+8WFaJy3ew==}
+  '@pnpm/exe.linux-arm64@12.18.0':
+    resolution: {integrity: sha512-2dpyVBCHT+FfpuQh9i+7MXAqNWYLCvD5vYr5p6Eimxoga8USLnjbUgyj72AOrO0hjc5ri3TEFP3jiyBHjkSZpQ==}
     cpu: [arm64]
     os: [linux]
     libc: [glibc]
 
-  '@pnpm/exe.linux-ppc64@12.16.1':
-    resolution: {integrity: sha512-FnSV0Jv40S1cMv7vFNWwvu0X567Lec4L4HCRead3CFcD6zQDNbGw81s8N5bcKPahtPD9ApVuhsbmFCx4rw6RMw==}
+  '@pnpm/exe.linux-ppc64@12.18.0':
+    resolution: {integrity: sha512-BBOOOfNJ2QwIIya3eKqOjiN4WNTszfRz7HCvw9XggcT7+ftMjNP8XJ/GfW+ltCBeUFR9d8WBVsr2m/NLjX2CZQ==}
     cpu: [ppc64]
     os: [linux]
     libc: [glibc]
 
-  '@pnpm/exe.linux-riscv64@12.16.1':
-    resolution: {integrity: sha512-C1Sww4FN5IttD4atRRhAKmQtSGWVp04ZAKNxO8pfu2bkRPwOW+7v+2DH1Xs1W+kZYYpO86BYDvUcadefF+AgaQ==}
+  '@pnpm/exe.linux-riscv64@12.18.0':
+    resolution: {integrity: sha512-7kMN77xSOYJX8GPSH5RbNAc5oM61LMmcSUukDDUwllnZPXm8GRPskCNsTWy8SCw01Kr1FB+5SwGNMiTdiziLNQ==}
     cpu: [riscv64]
     os: [linux]
     libc: [glibc]
 
-  '@pnpm/exe.linux-s390x@12.16.1':
-    resolution: {integrity: sha512-kn2fautsTcxIMODoovQVqZyfpDRdNgGkoeVqNdrKqbeBBvrEl+fXLkmZiq5xDht1LMr4sYiCNcsMfNMT6JPFmg==}
+  '@pnpm/exe.linux-s390x@12.18.0':
+    resolution: {integrity: sha512-htuSCpEcmDzhKoJqXbPkvfr176dBHU0QIF5mWcRiqxRXAB/1JoYiMYotfjWpk+ztgpB9qlx4XnWlLFAf7gzc6g==}
     cpu: [s390x]
     os: [linux]
     libc: [glibc]
 
-  '@pnpm/exe.linux-x64-musl@12.16.1':
-    resolution: {integrity: sha512-PM6ccHCOe/O9SsTflTJTYWfwogJZa6EmUvpnwZgiPguJBddh20wDQKgxUPXf48rDsPm9vMsIij0uHswwwbXL8g==}
+  '@pnpm/exe.linux-x64-musl@12.18.0':
+    resolution: {integrity: sha512-Uj+vzfIQ62kDP10x36yHZIXVs5ohdNtiPYwn5FhCLlNScHPo0OAI+ShGDtLofiBXKE7umu71Jmjm7N8a8zVcDw==}
     cpu: [x64]
     os: [linux]
     libc: [musl]
 
-  '@pnpm/exe.linux-x64@12.16.1':
-    resolution: {integrity: sha512-RK//8ZLcXGW8A2k8bQOXZDBN1Qu92zN4dYnG1bzj1zoQ1fkZ6jNzZ1AvXtsfGVma795CJtvoaF5bLXf4l/FIUw==}
+  '@pnpm/exe.linux-x64@12.18.0':
+    resolution: {integrity: sha512-OJ+X2wcraldCwj75pCIerWARQSRpdfj0eBTs/2ODyhfeW46ylOnjTIE/GKra6Mn6E72/gBmXS8igrUioa+F9fQ==}
     cpu: [x64]
     os: [linux]
     libc: [glibc]
 
-  '@pnpm/exe.win32-arm64@12.16.1':
-    resolution: {integrity: sha512-Ox3kNNVxtGw6pK7SVejmeOUgHhUTZgAM7NQQ6cU7frPw3ix9+j7d2MGbRg3fyEFAaKWq8ZAZLTQGcDyLnAu+qA==}
+  '@pnpm/exe.win32-arm64@12.18.0':
+    resolution: {integrity: sha512-wEY6jiACGcbhhi5yD59doHqNTndS7AaujaPmtWAu1DKkHru2hy5fJ9yUGecW72Ypaw2/I3BEprfMOopap2NQtw==}
     cpu: [arm64]
     os: [win32]
 
-  '@pnpm/exe.win32-x64@12.16.1':
-    resolution: {integrity: sha512-k+FhRJ0ttphFoFIVB1Sk4v76YJAulafmA/Ly57cysesUhhIzVlApeOShUz0jXubKx317TThlEfpQrukldnqORA==}
+  '@pnpm/exe.win32-x64@12.18.0':
+    resolution: {integrity: sha512-5FJkghO7RLRkQpWxeVEygW9XsmnNuxzPfXbgwFWCHsFyFhpDn/vOdL88ToQEM0Kt5igdW5s6U0/IDH8Dvkt3pg==}
     cpu: [x64]
     os: [win32]
 
-  pnpm@12.16.1:
-    resolution: {integrity: sha512-CR0d7nknV1Zhsl2rPCrjHK/BxU1rce1qZgDPDByxBDSkR+rhQBXxsj4ua+3u1535SpMuIWTSgyKA3OP7MvZW0Q==}
+  pnpm@12.18.0:
+    resolution: {integrity: sha512-nXiC1dyCFH7PxOBiqjSDTgrQKWcQoYN6ERWuASEn02txCeMhrBWYod0KZVl2gbHpmRfhLtDXSerpEYeUM2O+sw==}
     engines: {node: '>=18'}
     hasBin: true
 
 snapshots:
 
-  '@pnpm/exe.android-arm64@12.16.1':
+  '@pnpm/exe.android-arm64@12.18.0':
     optional: true
 
-  '@pnpm/exe.android-x64@12.16.1':
+  '@pnpm/exe.android-x64@12.18.0':
     optional: true
 
-  '@pnpm/exe.darwin-arm64@12.16.1':
+  '@pnpm/exe.darwin-arm64@12.18.0':
     optional: true
 
-  '@pnpm/exe.darwin-x64@12.16.1':
+  '@pnpm/exe.darwin-x64@12.18.0':
     optional: true
 
-  '@pnpm/exe.freebsd-x64@12.16.1':
+  '@pnpm/exe.freebsd-x64@12.18.0':
     optional: true
 
-  '@pnpm/exe.linux-arm64-musl@12.16.1':
+  '@pnpm/exe.linux-arm64-musl@12.18.0':
     optional: true
 
-  '@pnpm/exe.linux-arm64@12.16.1':
+  '@pnpm/exe.linux-arm64@12.18.0':
     optional: true
 
-  '@pnpm/exe.linux-ppc64@12.16.1':
+  '@pnpm/exe.linux-ppc64@12.18.0':
     optional: true
 
-  '@pnpm/exe.linux-riscv64@12.16.1':
+  '@pnpm/exe.linux-riscv64@12.18.0':
     optional: true
 
-  '@pnpm/exe.linux-s390x@12.16.1':
+  '@pnpm/exe.linux-s390x@12.18.0':
     optional: true
 
-  '@pnpm/exe.linux-x64-musl@12.16.1':
+  '@pnpm/exe.linux-x64-musl@12.18.0':
     optional: true
 
-  '@pnpm/exe.linux-x64@12.16.1':
+  '@pnpm/exe.linux-x64@12.18.0':
     optional: true
 
-  '@pnpm/exe.win32-arm64@12.16.1':
+  '@pnpm/exe.win32-arm64@12.18.0':
     optional: true
 
-  '@pnpm/exe.win32-x64@12.16.1':
+  '@pnpm/exe.win32-x64@12.18.0':
     optional: true
 
-  pnpm@12.16.1:
+  pnpm@12.18.0:
     optionalDependencies:
-      '@pnpm/exe.android-arm64': 12.16.1
-      '@pnpm/exe.android-x64': 12.16.1
-      '@pnpm/exe.darwin-arm64': 12.16.1
-      '@pnpm/exe.darwin-x64': 12.16.1
-      '@pnpm/exe.freebsd-x64': 12.16.1
-      '@pnpm/exe.linux-arm64': 12.16.1
-      '@pnpm/exe.linux-arm64-musl': 12.16.1
-      '@pnpm/exe.linux-ppc64': 12.16.1
-      '@pnpm/exe.linux-riscv64': 12.16.1
-      '@pnpm/exe.linux-s390x': 12.16.1
-      '@pnpm/exe.linux-x64': 12.16.1
-      '@pnpm/exe.linux-x64-musl': 12.16.1
-      '@pnpm/exe.win32-arm64': 12.16.1
-      '@pnpm/exe.win32-x64': 12.16.1
+      '@pnpm/exe.android-arm64': 12.18.0
+      '@pnpm/exe.android-x64': 12.18.0
+      '@pnpm/exe.darwin-arm64': 12.18.0
+      '@pnpm/exe.darwin-x64': 12.18.0
+      '@pnpm/exe.freebsd-x64': 12.18.0
+      '@pnpm/exe.linux-arm64': 12.18.0
+      '@pnpm/exe.linux-arm64-musl': 12.18.0
+      '@pnpm/exe.linux-ppc64': 12.18.0
+      '@pnpm/exe.linux-riscv64': 12.18.0
+      '@pnpm/exe.linux-s390x': 12.18.0
+      '@pnpm/exe.linux-x64': 12.18.0
+      '@pnpm/exe.linux-x64-musl': 12.18.0
+      '@pnpm/exe.win32-arm64': 12.18.0
+      '@pnpm/exe.win32-x64': 12.18.0
 
 ---
 lockfileVersion: '9.0'

backend/resources/climit.edn

@@ -69,5 +69,11 @@
  :import-binfile/by-profile
  {:permits 1 :queue 2 :timeout 60000}
 
+ :create-import-binfile-job/global
+ {:permits 4 :queue 10 :timeout 60000}
+
+ :create-import-binfile-job/by-profile
+ {:permits 1 :queue 2 :timeout 60000}
+
  :profile-plugin-ops/by-profile
  {:permits 1 :queue 2 :timeout 30000}}

backend/scripts/_env

@@ -62,13 +62,14 @@ export PENPOT_FLAGS="\
        disable-tiered-file-data-storage \
        enable-file-validation \
        enable-file-schema-validation \
+       enable-debug-save-faults \
        enable-redis-cache \
        enable-link-preview \
        enable-subscriptions";
 
 # Uncomment for nexus integration testing
-# export PENPOT_FLAGS="$PENPOT_FLAGS enable-audit-log-archive";
-# export PENPOT_AUDIT_LOG_ARCHIVE_URI="http://localhost:6070/api/audit";
+# export PENPOT_FLAGS="$PENPOT_FLAGS enable-nexus";
+# export PENPOT_NEXUS_URI="http://localhost:6070/";
 
 # Default deletion delay for devenv
 export PENPOT_DELETION_DELAY="24h"
@@ -92,6 +93,7 @@ export JAVA_OPTS="\
        -XX:+UseShenandoahGC \
        -XX:+UseCompactObjectHeaders \
        -XX:ShenandoahGCMode=generational \
+       -XX:NativeMemoryTracking=summary \
        -XX:-OmitStackTraceInFastThrow  \
        --sun-misc-unsafe-memory-access=allow \
        --enable-preview \

backend/src/app/auth/oidc.clj

@@ -629,7 +629,7 @@
         (l/warn :hint "received incomplete user info object (please set correct scopes)" :info info)
         (ex/raise :type :internal
                   :code :incomplete-user-info
-                  :hint "inconmplete user info"
+                  :hint "incomplete user info"
                   :info info)))
 
     ;; If the provider is OIDC, we can proceed to check
@@ -1037,7 +1037,9 @@
         (when (and session organization-id)
           (let [props (-> (or (:props session) {})
                           (update :sso assoc organization-id exp))]
-            (session/update-session (::session/manager cfg) (assoc session :props props))))
+            (session/update-session (::session/manager cfg) (assoc session :props props))
+            (submit-organization-sso-auth-event
+             cfg request (:profile-id session) organization-id "organization-sso-token-issued")))
         (submit-organization-sso-auth-event
          cfg request (:profile-id session) organization-id "organization-sso-auth-succeeded")
         (redirect-response dest-url))

backend/src/app/binfile/cleaner.clj

@@ -9,7 +9,7 @@
   for recently imported shapes."
   (:require
    [app.common.data :as d]
-   [app.common.svg :as csvg]
+   [app.common.geom.matrix :as gmt]
    [app.common.types.shape :as cts]
    [app.common.uuid :as uuid]))
 
@@ -106,14 +106,11 @@
                      :reverse-column :column-reverse
                      dir))))
 
-(defn- fix-svg-attrs
-  "The json reader of the binfile rewrites every key of every entry to
-  kebab-case, but `:svg-attrs` keys are react prop names and are stored
-  in camelCase. `attrs->props` is the transform the svg import path
-  already applies to them, so running it again restores the names; it
-  is idempotent, so shapes that come in correct are left untouched."
+(defn- fix-svg-transform
+  "The json reader decodes `:svg-transform` as a plain map; the svg defs
+  renderer multiplies it as a matrix, so turn it back into one."
   [shape]
-  (d/update-when shape :svg-attrs csvg/attrs->props))
+  (d/update-when shape :svg-transform gmt/coerce))
 
 (defn clean-shape-post-decode
   "A shape procesor that expected to be executed after schema decoding
@@ -123,7 +120,7 @@
       (fix-shape-shadow-color)
       (fix-root-shape)
       (fix-legacy-flex-dir)
-      (fix-svg-attrs)))
+      (fix-svg-transform)))
 
 (defn- fix-container
   [container]

backend/src/app/binfile/common.clj

@@ -74,11 +74,11 @@
 ;; Maximum total decompressed size allowed for all JSON/text zip entries
 ;; combined within a single import job: 500MiB. Bounds the case where many
 ;; entries, each individually under default-max-text-entry-size, still sum
-;; to an unreasonable total. Set well above the 200MiB it replaces because
-;; legitimate giant exports exist: hundreds of thousands of KB-sized
-;; entries (pages, shapes) that never trip the per-entry cap but sum past
-;; 400MiB. The per-entry cap still bounds single-entry DEFLATE
-;; amplification, so this only moves the cumulative ceiling.
+;; to an unreasonable total. Set this high because legitimate giant
+;; exports exist: hundreds of thousands of KB-sized entries (pages,
+;; shapes) that never trip the per-entry cap but sum past 400MiB. The
+;; per-entry cap still bounds single-entry DEFLATE amplification, so this
+;; only moves the cumulative ceiling.
 (def ^:const default-max-text-total-size
   (* 1024 1024 500))
 
@@ -239,24 +239,30 @@
 
 (defn- migrate-file
   [{:keys [::db/conn] :as cfg} {:keys [read-only?]} {:keys [id] :as file}]
-  (binding [pmap/*load-fn* (partial fdata/load-pointer cfg id)
-            pmap/*tracked* (pmap/create-tracked)]
-    (let [libs (delay (get-resolved-file-libraries cfg file))
-          ;; For avoid unnecesary overhead of creating multiple
-          ;; pointers and handly internally with objects map in their
-          ;; worst case (when probably all shapes and all pointers
-          ;; will be readed in any case), we just realize/resolve them
-          ;; before applying the migration to the file.
-          file (-> (fdata/realize cfg file)
-                   (fmg/migrate-file libs))]
+  (fdata/with-tracked-pointers cfg id
+    (let [libs                (delay (get-resolved-file-libraries cfg file))
+          ;; Capture the pointer layout before realizing: the migration
+          ;; runs on plain data, and unchanged positions must keep their
+          ;; original fragment ids instead of being re-wrapped with new
+          ;; ones on persist.
+          original            (fdata/original-pointers (:data file))
+          objects-map-before? (contains? (:features file) "fdata/objects-map")
+          realized            (-> (fdata/realize cfg file)
+                                  (fmg/migrate-file libs))
+          objects-map-added?  (and (not objects-map-before?)
+                                   (contains? (:features realized) "fdata/objects-map"))]
 
       (if (or read-only? (db/read-only? conn))
-        file
-        (do ;; When file is migrated, we break the rule of no
-          ;; perform mutations on get operations and update the
-          ;; file with all migrations applied
+        realized
+        (let [file (if objects-map-added?
+                     realized
+                     (fdata/reattach-unchanged original realized))]
+          ;; When the file is migrated, we break the rule of not
+          ;; performing mutations on get operations and update the file
+          ;; with all migrations applied. The caller still receives the
+          ;; realized file: the read contract is plain data.
           (update-file! cfg file)
-          (fmigr/resolve-applied-migrations cfg file))))))
+          (fmigr/resolve-applied-migrations cfg realized))))))
 
 (defn- get-file*
   [{:keys [::db/conn] :as cfg} id
@@ -281,7 +287,7 @@
 
   (assert (db/connection? conn) "expected cfg with valid connection")
   (when (and (not load-data?)
-             (or lock-for-share? lock-for-share? skip-locked?))
+             (or lock-for-share? lock-for-update? skip-locked?))
     (throw (IllegalArgumentException. "locking is incompatible when `load-data?` is false")))
 
   (let [sql
@@ -946,7 +952,12 @@
   (let [library-ids (->> (get-file-libraries conn (:id file))
                          (map :id)
                          (cons (:id file)))
-        load-fn     #(get-file cfg % :migrate? false)]
+        ;; Fragment rows are file-local and callers bind the loader of
+        ;; the file they work on, so each library realizes its pointers
+        ;; here, with its own id, before anything walks its data.
+        load-fn     (fn [library-id]
+                      (->> (get-file cfg library-id :migrate? false)
+                           (fdata/realize-pointers cfg)))]
     (weak/loadable-weak-value-map library-ids load-fn {id file})))
 
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
@@ -1064,3 +1075,46 @@
       {:bytes bytes
        :size  (alength ^bytes bytes)
        :hash  (sto/calculate-hash (java.io.ByteArrayInputStream. bytes))})))
+
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;; REQUEST CFG
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+
+(defn resolve-export-type
+  "The canonical export type of a request on the legacy RPC: an explicit
+  `type` wins, and the deprecated boolean flags are translated. The
+  creation command of a job does not need it, because it asks the caller
+  for the type instead of guessing it."
+  [{:keys [type include-libraries embed-assets]}]
+  (cond
+    (some? type)              type
+    (true? include-libraries) :include-libraries
+    (true? embed-assets)      :merge-libraries
+    :else                     :detach-libraries))
+
+(defn export-cfg
+  "The domain cfg of an export: the frozen set of files, how their
+  libraries are handled, and where the package is written."
+  [cfg {:keys [file-ids export-type output]}]
+  (-> cfg
+      (assoc ::file-ids file-ids)
+      (assoc ::export-type export-type)
+      (assoc ::output output)))
+
+(defn import-cfg
+  "The domain cfg of an import: the destination team/profile/project, the
+  name, the input and the anti-zip-bomb limits. The caller resolves the
+  team, because what allows it to import there differs between the RPC and
+  a job; what both must share is fixed here."
+  [cfg {:keys [profile-id project-id team name input]}]
+  (-> cfg
+      (assoc ::features (cfeat/get-team-enabled-features cf/flags team))
+      (assoc ::project-id project-id)
+      (assoc ::profile-id profile-id)
+      (assoc ::team-id (:id team))
+      (assoc ::name name)
+      (assoc ::input input)
+      (assoc ::import-max-binary-entry-size (cf/get :binfile-import-max-binary-entry-size))
+      (assoc ::import-max-text-entry-size (cf/get :binfile-import-max-text-entry-size))
+      (assoc ::import-max-text-total-size (cf/get :binfile-import-max-text-total-size))
+      (assoc ::import-max-zip-entries (cf/get :binfile-import-max-zip-entries))))

backend/src/app/binfile/jobs.clj

@@ -0,0 +1,120 @@
+;; This Source Code Form is subject to the terms of the Mozilla Public
+;; License, v. 2.0. If a copy of the MPL was not distributed with this
+;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
+;;
+;; Copyright (c) KALEIDOS SUBSIDIARY SL
+
+(ns app.binfile.jobs
+  "The progress bridge of the binfile jobs.
+
+  A job has no SSE channel, so the `:progress` points that `bf.v1` and
+  `bf.v3` emit become `progress` events of the job through
+  `jobs/heartbeat`, which throttles them and drops the ones that do not
+  fit. The adapter turns a tap into a self-contained milestone: a `:stage`
+  of a single vocabulary (the two formats name the same work differently)
+  and the counters the tap carries, with the outer scopes in course as
+  context. The reader puts the outer counters in every tap inside them,
+  so the adapter holds no state between taps. The ids the taps carry are
+  not part of the job event contract.
+
+  The handlers in `app.tasks.export-binfile` and
+  `app.tasks.import-binfile` run the core through `run` and
+  guard it with `check-active`: a job that was cancelled, or that
+  already reached a terminal state, raises an `:interrupt` right there on
+  the run thread, so the work stops at its next event point and the
+  transaction of its handler rolls back with it. A run whose last tap
+  happened before the interruption is discarded once it returns, which is
+  what makes that transaction roll back too."
+  (:require
+   [app.jobs :as jobs]
+   [app.util.events :as events]))
+
+(defn check-active
+  "Raise an `:interrupt` when the job is no longer active.
+
+  The substrate owns both the check and the error; this is the adapter's
+  entry point to it, used by the two handlers before their run and by the
+  tests. Answers with the cfg, so a handler can thread it."
+  [cfg context]
+  (jobs/check-active cfg (:id context)))
+
+(def ^:private stage-of
+  "The stage a tap section becomes: one vocabulary for the two formats, so
+  a client resolves each stage once. The counter of the activity is named
+  after its own stage, which is what lets a client render
+  `counters[stage]` without a table of equivalences."
+  {:file            :files
+   :page            :pages
+   :media           :media
+   :thumbnail       :thumbnails
+   :thumbnails      :thumbnails
+   :component       :components
+   :color           :colors
+   :typography      :typographies
+   :tokens-lib      :tokens
+   :tokens-status   :tokens
+   :storage-object  :storage-objects
+   :storage-objects :storage-objects
+   :relations       :relations
+   :manifest        :manifest
+   :v1/metadata     :metadata
+   :v1/files        :files
+   :v1/rels         :relations
+   :v1/sobjects     :storage-objects})
+
+(defn- stage
+  "The stage of a tap section; a section without a name of its own keeps
+  it, so a new tap is never dropped from the stream."
+  [section]
+  (get stage-of section section))
+
+(defn- counter
+  "The counter a tap reports, when it reports one: a tap that only marks a
+  point of the run has no units to count."
+  [{:keys [current total]}]
+  (when (some? current)
+    (cond-> {:current current}
+      (some? total)
+      (assoc :total total))))
+
+(defn- milestone
+  "The milestone a tap becomes: what the job is doing now and the counters
+  the tap carries, with the outer scopes in course as context.
+
+  The tap of a scope carries its outer counters, so the milestone is
+  self-contained and the adapter remembers nothing between taps: a client
+  that misses one (the substrate throttles the events) still renders the
+  truth from the next."
+  [{:keys [section outer-counters] :as tap}]
+  (let [stage (stage section)
+        own   (counter tap)]
+    (cond-> {:stage stage}
+      (or (some? outer-counters) (some? own))
+      (assoc :counters (cond-> (or outer-counters {})
+                         (some? own) (assoc stage own))))))
+
+(defn run
+  "Run `f` with the progress taps of the core turned into milestones of the
+  job.
+
+  `f` is called with the cfg, so the adapter forwards what it receives
+  and a caller can pass the core directly.
+
+  Every tap is reported: the substrate throttles the durable events, and
+  a milestone that does not fit loses no information because the next one
+  carries the same state.
+
+  The taps run on this same thread: the sink calls the heartbeat inline,
+  so an `:interrupt` it raises aborts the core right there, inside the
+  transaction of its handler, which rolls back."
+  [cfg context f]
+  (let [job-id (:id context)
+        sink   (fn [[type data]]
+                 (when (= :progress type)
+                   (jobs/heartbeat cfg
+                                   :job-id job-id
+                                   :progress (milestone data))))]
+    (binding [events/*sink* sink]
+      (let [result (f cfg)]
+        (check-active cfg context)
+        result))))

backend/src/app/binfile/v1.clj

@@ -329,13 +329,13 @@
             [:v1/metadata :v1/files :v1/rels :v1/sobjects]))))
 
 (defmethod write-section :v1/metadata
-  [{:keys [::output ::bfc/ids ::bfc/include-libraries] :as cfg}]
-  (if-let [fids (get-files cfg ids)]
+  [{:keys [::output ::bfc/file-ids ::bfc/include-libraries] :as cfg}]
+  (if-let [fids (get-files cfg file-ids)]
     (let [lids (when include-libraries
-                 (bfc/get-libraries cfg ids))
-          ids  (into fids lids)]
-      (write-obj! output {:version cf/version :files ids})
-      (vswap! bfc/*state* assoc :files ids))
+                 (bfc/get-libraries cfg file-ids))
+          file-ids (into fids lids)]
+      (write-obj! output {:version cf/version :files file-ids})
+      (vswap! bfc/*state* assoc :files file-ids))
     (ex/raise :type :not-found
               :code :files-not-found
               :hint "unable to retrieve files for export")))
@@ -689,12 +689,12 @@
   `::bfc/embed-assets`: instead of including the libraries, embed in the
   same file library all assets used from external libraries."
 
-  [{:keys [::bfc/ids] :as cfg} output]
+  [{:keys [::bfc/file-ids] :as cfg} output]
 
   (dm/assert!
-   "expected a set of uuid's for `::bfc/ids` parameter"
-   (and (set? ids)
-        (every? uuid? ids)))
+   "expected a set of uuid's for `::bfc/file-ids` parameter"
+   (and (set? file-ids)
+        (every? uuid? file-ids)))
 
   (dm/assert!
    "expected instance of jio/IOFactory for `input`"

backend/src/app/binfile/v3.clj

@@ -231,6 +231,32 @@
 (def validate-file-thumbnail
   (sm/check-fn schema:file-thumbnail))
 
+;; --- PROGRESS
+
+(defn- tap-progress
+  "Emit a progress tap of the core, with the counters of the outer scopes
+  in course when the cfg has any.
+
+  The outer counters live in `::outer-counters` of the cfg while a scope
+  is being processed (today only `:files`, tomorrow `:teams` and
+  `:projects` too), so every tap inside carries them as `:outer-counters`:
+  the stages inside do not repeat them, and whoever reads the taps (the
+  jobs adapter) builds its report from one tap alone instead of
+  remembering the last position it saw. A tap that happens outside any
+  scope carries no outer counters, which is the truth about it."
+  [cfg data]
+  (events/tap :progress
+              (cond-> data
+                (some? (::outer-counters cfg))
+                (assoc :outer-counters (::outer-counters cfg)))))
+
+(defn- with-outer-counter
+  "Return cfg with the counter of `scope` set to `position`, keeping the
+  counters of the other outer scopes already in course, so entering a
+  nested scope never wipes the ones above it."
+  [cfg scope position]
+  (assoc-in cfg [::outer-counters scope] position))
+
 ;; --- EXPORT IMPL
 
 (defn- write-entry!
@@ -269,8 +295,10 @@
 
 (defn- export-storage-objects
   [{:keys [::output] :as cfg}]
-  (let [storage (sto/resolve cfg)]
-    (doseq [id (-> bfc/*state* deref :storage-objects not-empty)]
+  (let [storage (sto/resolve cfg)
+        ids     (-> bfc/*state* deref :storage-objects not-empty)
+        total   (count ids)]
+    (doseq [[index id] (d/enumerate ids)]
       (let [sobject (sto/get-object storage id)
             smeta   (meta sobject)
             ext     (cmedia/mtype->extension (:content-type smeta))
@@ -282,7 +310,8 @@
 
         (write-entry! output path params)
 
-        (events/tap :progress {:section :storage-object :id id})
+        (tap-progress cfg {:section :storage-object :id id
+                           :current (inc index) :total total})
 
         (with-open [input (sto/get-object-data storage sobject)]
           (.putNextEntry ^ZipOutputStream output (ZipEntry. (str "objects/" id ext)))
@@ -310,7 +339,7 @@
 
         thumbnails    (bfc/get-file-object-thumbnails cfg file-id)]
 
-    (events/tap :progress {:section :file :id file-id :name (:name file)})
+    (tap-progress cfg {:section :file :id file-id :name (:name file)})
 
     (vswap! bfc/*state* update :files assoc file-id
             {:id file-id
@@ -344,7 +373,8 @@
 
         (write-entry! output path page)
 
-        (events/tap :progress {:section :page :id page-id :name (:name page) :file-id file-id})
+        (tap-progress cfg {:section :page :id page-id :name (:name page) :file-id file-id
+                           :current (inc index) :total (count pages)})
 
         (doseq [[shape-id shape] objects]
           (let [path  (str "files/" file-id "/pages/" page-id "/" shape-id ".json")
@@ -355,30 +385,33 @@
     (vswap! bfc/*state* bfc/collect-storage-objects media)
     (vswap! bfc/*state* bfc/collect-storage-objects thumbnails)
 
-    (doseq [{:keys [id] :as media} media]
-      (let [path  (str "files/" file-id "/media/" id ".json")
-            media (encode-media media)]
+    (doseq [[index {:keys [id] :as item}] (d/enumerate media)]
+      (let [path    (str "files/" file-id "/media/" id ".json")
+            encoded (encode-media item)]
 
-        (events/tap :progress {:section :media :id id  :file-id file-id})
-        (write-entry! output path media)))
+        (tap-progress cfg {:section :media :id id :file-id file-id
+                           :current (inc index) :total (count media)})
+        (write-entry! output path encoded)))
 
-    (doseq [thumbnail thumbnails]
+    (doseq [[index thumbnail] (d/enumerate thumbnails)]
       (let [data (cth/parse-object-id (:object-id thumbnail))
             path (str "files/" file-id "/thumbnails/" (:tag data) "/" (:page-id data)
                       "/" (:frame-id data) ".json")
             data (-> data
                      (assoc :media-id (:media-id thumbnail))
                      (encode-file-thumbnail))]
-        (events/tap :progress {:section :thumbnails :id (:object-id thumbnail) :file-id file-id})
+        (tap-progress cfg {:section :thumbnails :id (:object-id thumbnail) :file-id file-id
+                           :current (inc index) :total (count thumbnails)})
         (write-entry! output path data)))
 
-    (doseq [[id component] components]
+    (doseq [[index [id component]] (d/enumerate components)]
       (let [path      (str "files/" file-id "/components/" id ".json")
             component (encode-component component)]
-        (events/tap :progress {:section :component :id id :file-id file-id})
+        (tap-progress cfg {:section :component :id id :file-id file-id
+                           :current (inc index) :total (count components)})
         (write-entry! output path component)))
 
-    (doseq [[id color] colors]
+    (doseq [[index [id color]] (d/enumerate colors)]
       (let [path  (str "files/" file-id "/colors/" id ".json")
             color (-> (encode-color color)
                       (dissoc :file-id))
@@ -386,20 +419,22 @@
                     (and (contains? color :path)
                          (str/empty? (:path color)))
                     (dissoc :path))]
-        (events/tap :progress {:section :color :id id :file-id file-id})
+        (tap-progress cfg {:section :color :id id :file-id file-id
+                           :current (inc index) :total (count colors)})
         (write-entry! output path color)))
 
-    (doseq [[id object] typographies]
+    (doseq [[index [id object]] (d/enumerate typographies)]
       (let [path       (str "files/" file-id "/typographies/" id ".json")
             typography (encode-typography object)]
-        (events/tap :progress {:section :typography :id id :file-id file-id})
+        (tap-progress cfg {:section :typography :id id :file-id file-id
+                           :current (inc index) :total (count typographies)})
         (write-entry! output path typography)))
 
     (when (and tokens-lib
                (not (ctob/empty-lib? tokens-lib)))
       (let [path           (str "files/" file-id "/tokens.json")
             encoded-tokens (encode-tokens-lib tokens-lib)]
-        (events/tap :progress {:section :tokens-lib :file-id file-id})
+        (tap-progress cfg {:section :tokens-lib :file-id file-id})
         (write-entry! output path encoded-tokens)))
 
     (when tokens-status
@@ -415,18 +450,19 @@
             encoded-status        (-> (encode-tokens-status tokens-status)
                                       (cfo/ids->names effective-tokens-lib))]
 
-        (events/tap :progress {:section :tokens-status :file-id file-id})
+        (tap-progress cfg {:section :tokens-status :file-id file-id})
         (write-entry! output path encoded-status)))))
 
 (defn- export-files
-  [{:keys [::bfc/ids ::bfc/export-type ::output] :as cfg}]
+  [{:keys [::bfc/file-ids ::bfc/export-type ::output] :as cfg}]
 
-  (let [original-ids ids
-        ids  (into ids (when (= export-type :include-libraries) (bfc/get-libraries cfg ids)))
-        rels (if (= export-type :include-libraries)
-               (->> (bfc/get-files-rels cfg ids)
-                    (mapv (juxt :file-id :library-file-id)))
-               [])
+  (let [original-ids file-ids
+        file-ids     (into file-ids (when (= export-type :include-libraries)
+                                      (bfc/get-libraries cfg file-ids)))
+        rels         (if (= export-type :include-libraries)
+                       (->> (bfc/get-files-rels cfg file-ids)
+                            (mapv (juxt :file-id :library-file-id)))
+                       [])
 
         ;; Compute external libraries: referenced by original files but
         ;; not included in the export set. Only relevant for :link-later.
@@ -451,10 +487,11 @@
     (vswap! bfc/*state* assoc :files (d/ordered-map))
 
     ;; Write all the exporting files
-    (doseq [[index file-id] (d/enumerate ids)]
+    (doseq [[index file-id] (d/enumerate file-ids)]
       (-> cfg
           (assoc ::file-id file-id)
-          (assoc ::file-seqn index)
+          (with-outer-counter :files {:current (inc index)
+                                      :total (count file-ids)})
           (export-file)))
 
     ;; Write manifest file
@@ -747,17 +784,19 @@
 
 (defn- read-file-colors
   [{:keys [::bfc/input ::entries-index] :as cfg} file-id]
-  (->> (get-in entries-index [:colors (str file-id)])
-       (reduce (fn [result {:keys [id entry]}]
-                 (let [object (->> (read-entry cfg input entry)
-                                   (decode-color)
-                                   (validate-color))]
-                   (events/tap :progress {:section :color :id id :file-id file-id})
-                   (if (= id (:id object))
-                     (assoc result id object)
-                     result)))
-               {})
-       (not-empty)))
+  (let [entries (get-in entries-index [:colors (str file-id)])]
+    (->> (d/enumerate entries)
+         (reduce (fn [result [index {:keys [id entry]}]]
+                   (let [object (->> (read-entry cfg input entry)
+                                     (decode-color)
+                                     (validate-color))]
+                     (tap-progress cfg {:section :color :id id :file-id file-id
+                                        :current (inc index) :total (count entries)})
+                     (if (= id (:id object))
+                       (assoc result id object)
+                       result)))
+                 {})
+         (not-empty))))
 
 (defn- read-file-components
   [{:keys [::bfc/input ::entries-index] :as cfg} file-id]
@@ -776,15 +815,18 @@
                            (reduce-kv (fn [objects id shape]
                                         (assoc objects id (bfl/clean-shape-pre-decode shape)))
                                       objects
-                                      objects))))]
+                                      objects))))
+        entries
+        (get-in entries-index [:components (str file-id)])]
 
-    (->> (get-in entries-index [:components (str file-id)])
-         (reduce (fn [result {:keys [id entry]}]
+    (->> (d/enumerate entries)
+         (reduce (fn [result [index {:keys [id entry]}]]
                    (let [object (->> (read-entry cfg input entry)
                                      (clean-component-pre-decode)
                                      (decode-component)
                                      (clean-component-post-decode))]
-                     (events/tap :progress {:section :component :id id :file-id file-id})
+                     (tap-progress cfg {:section :component :id id :file-id file-id
+                                        :current (inc index) :total (count entries)})
                      (if (= id (:id object))
                        (assoc result id object)
                        result)))
@@ -793,22 +835,24 @@
 
 (defn- read-file-typographies
   [{:keys [::bfc/input ::entries-index] :as cfg} file-id]
-  (->> (get-in entries-index [:typographies (str file-id)])
-       (reduce (fn [result {:keys [id entry]}]
-                 (let [object (->> (read-entry cfg input entry)
-                                   (decode-typography)
-                                   (validate-typography))]
-                   (events/tap :progress {:section :typography :id id :file-id file-id})
-                   (if (= id (:id object))
-                     (assoc result id object)
-                     result)))
-               {})
-       (not-empty)))
+  (let [entries (get-in entries-index [:typographies (str file-id)])]
+    (->> (d/enumerate entries)
+         (reduce (fn [result [index {:keys [id entry]}]]
+                   (let [object (->> (read-entry cfg input entry)
+                                     (decode-typography)
+                                     (validate-typography))]
+                     (tap-progress cfg {:section :typography :id id :file-id file-id
+                                        :current (inc index) :total (count entries)})
+                     (if (= id (:id object))
+                       (assoc result id object)
+                       result)))
+                 {})
+         (not-empty))))
 
 (defn- read-file-tokens-lib*
   [{:keys [::bfc/input ::entries-index] :as cfg} file-id]
   (when-let [{:keys [entry]} (first (get-in entries-index [:tokens (str file-id)]))]
-    (events/tap :progress {:section :tokens-lib :file-id file-id})
+    (tap-progress cfg {:section :tokens-lib :file-id file-id})
     (->> (read-entry cfg input entry nil)
          (decode-tokens-lib)
          (validate-tokens-lib))))
@@ -829,7 +873,7 @@
 (defn- read-file-tokens-status
   [{:keys [::bfc/input ::entries-index] :as cfg} file-id tokens-lib]
   (when-let [{:keys [entry]} (first (get-in entries-index [:tokens-status (str file-id)]))]
-    (events/tap :progress {:section :tokens-status :file-id file-id})
+    (tap-progress cfg {:section :tokens-status :file-id file-id})
     (->> (read-entry cfg input entry nil)
          (cfo/names->ids tokens-lib)
          (decode-tokens-status)
@@ -851,19 +895,21 @@
 
 (defn- read-file-pages
   [{:keys [::bfc/input ::entries-index] :as cfg} file-id]
-  (->> (get-in entries-index [:pages (str file-id)])
-       (keep (fn [{:keys [id entry]}]
-               (let [page (->> (read-entry cfg input entry)
-                               (decode-page))
-                     page (dissoc page :options)]
-                 (events/tap :progress {:section :page :id id :file-id file-id})
-                 (when (= id (:id page))
-                   (let [objects (read-file-shapes cfg file-id id)]
-                     (assoc page :objects objects))))))
-       (sort-by :index)
-       (reduce (fn [result {:keys [id] :as page}]
-                 (assoc result id (dissoc page :index)))
-               (d/ordered-map))))
+  (let [entries (get-in entries-index [:pages (str file-id)])]
+    (->> (d/enumerate entries)
+         (keep (fn [[index {:keys [id entry]}]]
+                 (let [page (->> (read-entry cfg input entry)
+                                 (decode-page))
+                       page (dissoc page :options)]
+                   (tap-progress cfg {:section :page :id id :file-id file-id
+                                      :current (inc index) :total (count entries)})
+                   (when (= id (:id page))
+                     (let [objects (read-file-shapes cfg file-id id)]
+                       (assoc page :objects objects))))))
+         (sort-by :index)
+         (reduce (fn [result {:keys [id] :as page}]
+                   (assoc result id (dissoc page :index)))
+                 (d/ordered-map)))))
 
 (defn- read-file-thumbnails
   [{:keys [::bfc/input ::entries-index] :as cfg} file-id]
@@ -904,8 +950,11 @@
      :plugin-data plugin-data}))
 
 (defn- import-file
-  [{:keys [::db/conn ::bfc/project-id ::manifest] :as cfg} {file-id :id file-name :name}]
-  (let [file-id'   (bfc/lookup-index file-id)
+  [{:keys [::db/conn ::bfc/project-id ::manifest] :as cfg}
+   {file-id :id file-name :name}
+   position]
+  (let [cfg        (with-outer-counter cfg :files position)
+        file-id'   (bfc/lookup-index file-id)
         file       (read-file cfg file-id)
         media      (read-file-media cfg file-id)
         thumbnails (read-file-thumbnails cfg file-id)]
@@ -917,16 +966,19 @@
            :version (:version file)
            ::l/sync? true)
 
+    (tap-progress cfg {:section :file :file-id file-id})
+
     (vswap! bfc/*state* update :index bfc/update-index media :id)
 
-    (doseq [item media]
+    (doseq [[index item] (d/enumerate media)]
       (let [params (-> item
                        (update :id bfc/lookup-index)
                        (assoc :file-id file-id')
                        (d/update-when :media-id bfc/lookup-index)
                        (d/update-when :thumbnail-id bfc/lookup-index))]
 
-        (events/tap :progress {:section :media :id (:id params) :file-id file-id})
+        (tap-progress cfg {:section :media :id (:id params) :file-id file-id
+                           :current (inc index) :total (count media)})
 
         (l/dbg :hint "inserting media object"
                :file-id (str file-id')
@@ -939,7 +991,7 @@
         (db/insert! conn :file-media-object params
                     ::db/on-conflict-do-nothing? (::bfc/overwrite cfg))))
 
-    (doseq [item thumbnails]
+    (doseq [[index item] (d/enumerate thumbnails)]
       (let [media-id  (bfc/lookup-index (:media-id item))
             object-id (-> (assoc item :file-id file-id')
                           (cth/fmt-object-id))
@@ -953,13 +1005,12 @@
                :media-id (str media-id)
                ::l/sync? true)
 
-        (events/tap :progress {:section :thumbnail :file-id file-id :object-id object-id})
+        (tap-progress cfg {:section :thumbnail :file-id file-id :object-id object-id
+                           :current (inc index) :total (count thumbnails)})
 
         (db/insert! conn :file-tagged-object-thumbnail params
                     ::db/on-conflict-do-nothing? true)))
 
-    (events/tap :progress {:section :file :file-id file-id})
-
     (let [data (-> (read-file-data cfg file-id (:tokens-source file))
                    (d/without-nils)
                    (assoc :id file-id')
@@ -988,9 +1039,12 @@
 
 (defn- import-file-relations
   [{:keys [::db/conn ::manifest ::bfc/timestamp] :as cfg}]
-  (events/tap :progress {:section :relations})
-  (let [file-ids (into #{} (map :id) (:files manifest))]
-    (doseq [[file-id libr-id] (:relations manifest)]
+  (let [relations (:relations manifest)
+        file-ids  (into #{} (map :id) (:files manifest))]
+    (doseq [[index [file-id libr-id]] (d/enumerate relations)]
+      (tap-progress cfg {:section :relations
+                         :current (inc index) :total (count relations)})
+
       (if (and (not= file-id libr-id)
                (contains? file-ids file-id)
                (contains? file-ids libr-id))
@@ -1010,8 +1064,6 @@
 
 (defn- import-storage-objects
   [{:keys [::bfc/input ::entries-index ::bfc/timestamp] :as cfg}]
-  (events/tap :progress {:section :storage-objects})
-
   ;; IMPORTANT: we strongly do not reuse the main connection that can
   ;; run inside a transaction because the storage upload process can
   ;; fail in the middle of uploading and leave garbage on the underlying
@@ -1020,9 +1072,12 @@
   ;; what the storage subsystem registers in other parallel
   ;; transaction
   (let [storage (sto/resolve cfg)
-        entries (:objects entries-index)]
+        entries (:objects entries-index)
+        total   (count entries)]
 
-    (doseq [{:keys [id entry]} entries]
+    (doseq [[index {:keys [id entry]}] (d/enumerate entries)]
+      (tap-progress cfg {:section :storage-objects
+                         :current (inc index) :total total})
       (let [object  (-> (read-entry cfg input entry)
                         (decode-storage-object)
                         (d/update-when :content-type bfc/normalize-content-type)
@@ -1204,12 +1259,15 @@
       (vswap! bfc/*state* update :index assoc old-lib-id library-id)))
 
   (let [files    (get manifest :files)
-        file-ids (reduce (fn [result file]
+        total    (count files)
+        file-ids (reduce (fn [result [index file]]
                            (let [name' (get file :name)
-                                 file (assoc file :name name')]
-                             (conj result (import-file cfg file))))
+                                 file  (assoc file :name name')]
+                             (conj result (import-file cfg file
+                                                       {:current (inc index)
+                                                        :total total}))))
                          []
-                         files)
+                         (d/enumerate files))
         ;; Build map of file-id to file-name for resolution
         files-info (into {} (map (fn [file-id manifest-file]
                                    [file-id (:name manifest-file)])
@@ -1266,7 +1324,7 @@
               bfc/*reference-file* ref-file]
 
       (import-storage-objects cfg)
-      (import-file cfg file)
+      (import-file cfg file {:current 1 :total 1})
 
       (invalidate-thumbnails cfg file-id)
       (bfm/apply-pending-migrations! cfg)
@@ -1321,7 +1379,7 @@
 
         (cfeat/check-supported-features! features)))
 
-    (events/tap :progress {:section :manifest})
+    (tap-progress cfg {:section :manifest})
 
     (binding [bfc/*state* (volatile! {:media [] :index {}})]
       (if (::bfc/file-id cfg)
@@ -1332,24 +1390,25 @@
 
 (defn export-files!
   "Do the exportation of a specified file in custom penpot binary
-  format. There are some options available for customize the output:
+  format. The cfg carries what `bfc/export-cfg` froze: `::bfc/file-ids`
+  (the files to export), `::bfc/export-type` (how linked libraries are
+  handled) and `::bfc/output` (where the package is written).
 
-  `::bfc/export-type`: determines how linked libraries are handled.
-  Valid values:
-  `:include-libraries` (include linked libraries),
+  `::bfc/export-type` determines how linked libraries are handled.
+  Valid values: `:include-libraries` (include linked libraries),
   `:merge-libraries` (embed library assets in the file),
   `:detach-libraries` (treat assets as basic objects),
   `:link-later` (preserve component metadata for relinking on import)."
 
-  [{:keys [::bfc/ids] :as cfg} output]
+  [{:keys [::bfc/file-ids ::bfc/output] :as cfg}]
 
   (assert
-   (and (set? ids) (every? uuid? ids))
-   "expected a set of uuid's for `::bfc/ids` parameter")
+   (and (set? file-ids) (every? uuid? file-ids))
+   "expected a set of uuid's for `::bfc/file-ids` parameter")
 
   (assert
    (satisfies? jio/IOFactory output)
-   "expected instance of jio/IOFactory for `input`")
+   "expected instance of jio/IOFactory for `::bfc/output`")
 
   (let [id (uuid/next)
         tp (ct/tpoint)
@@ -1375,6 +1434,11 @@
         (vreset! ab true)
         nil)
 
+      (catch clojure.lang.ExceptionInfo cause
+        (if (= :interrupt (:type (ex-data cause)))
+          (vreset! ab true)
+          (throw cause)))
+
       (catch Throwable cause
         (vreset! cs cause)
         (vreset! ab true)

backend/src/app/config.clj

@@ -77,6 +77,11 @@
 
    :telemetry-uri "https://telemetry.penpot.app/"
 
+   :jobs-lease (ct/duration {:minutes 30})
+   :jobs-retention (ct/duration {:days 7})
+   :jobs-user-ttl (ct/duration {:days 7})
+   :jobs-request-timeout (ct/duration {:minutes 2})
+
    :media-max-file-size (* 1024 1024 30) ; 30MiB
    :font-max-file-size  (* 1024 1024 30) ; 30MiB
 
@@ -97,12 +102,44 @@
 
    :quotes-upload-sessions-per-profile 5
    :quotes-upload-chunks-per-session 20
+   :quotes-export-jobs-per-profile 10
+   :quotes-import-jobs-per-profile 10
    :upload-max-chunk-size (* 1024 1024 30) ; 30MiB
 
    ;; SSRF protection
    :ssrf-allowed-hosts #{}
    :ssrf-extra-blocked-cidrs #{}})
 
+(def schema:tenant
+  "Tenant identifier: hostname-label-style, lowercase (letters, digits
+  and hyphens). The value is normalized to lowercase when the
+  configuration is read, so an operator writing `PENPOT_TENANT=ACME` and
+  one writing `acme` name the same tenant: the value is compared as an
+  exact string in the keys and rows it lands in, and one uppercase
+  letter would otherwise split one tenant into two.
+
+  It is interpolated into Redis keys separated by dots (msgbus
+  topics, rate-limit buckets, exporter job keys, which are also matched
+  with a glob pattern), so `.` and whitespace are rejected at startup
+  because one tenant would then be able to read or overwrite another's
+  keys.
+
+  `%`, `_` and `:` are rejected too, and the reason they were added is
+  gone: the job queue used to be filtered with a LIKE over a
+  `<tenant>:<queue>` prefix and now has a `tenant` column. They are kept
+  because narrowing the rule is a startup contract change with no real
+  tenant asking for it, and a tenant that has one would have to be
+  renamed. Tenants using those characters must be renamed before
+  upgrading."
+  [:re #"^[a-z0-9-]+$"])
+
+(defn- normalize-tenant
+  "Lowercase the configured tenant before it is validated, so
+  `PENPOT_TENANT=ACME` and `acme` mean the same tenant everywhere."
+  [config]
+  (cond-> config
+    (string? (:tenant config)) (update :tenant str/lower)))
+
 (def schema:config
   (do #_sm/optional-keys
    [:map {:title "config"}
@@ -110,7 +147,7 @@
     [:admins {:optional true} [::sm/set ::sm/email]]
     [:secret-key {:optional true} :string]
 
-    [:tenant {:optional false} :string]
+    [:tenant {:optional false} schema:tenant]
     [:is-saas ::sm/boolean]
     [:public-uri {:optional false} ::sm/uri]
     [:host {:optional false} :string]
@@ -173,22 +210,29 @@
 
     [:deletion-delay {:optional true} ::ct/duration]
     [:file-clean-delay {:optional true} ::ct/duration]
+    [:jobs-lease {:optional true} ::ct/duration]
+    [:jobs-retention {:optional true} ::ct/duration]
+    [:jobs-user-ttl {:optional true} ::ct/duration]
+    [:jobs-request-timeout {:optional true} ::ct/duration]
     [:telemetry-enabled {:optional true} ::sm/boolean]
     [:default-blob-version {:optional true} ::sm/int]
     [:allow-demo-users {:optional true} ::sm/boolean]
     [:error-report-webhook {:optional true} :string]
     [:user-feedback-destination {:optional true} :string]
+    [:mcp-feedback-email {:optional true} ::sm/email]
 
     [:default-rpc-rlimit {:optional true} [::sm/vec :string]]
     [:rpc-rlimit-config {:optional true} ::fs/path]
     [:rpc-climit-config {:optional true} ::fs/path]
 
-    [:audit-log-archive-uri {:optional true} :string]
+    [:nexus-uri {:optional true} ::sm/uri]
     [:audit-log-http-handler-concurrency {:optional true} ::sm/int]
 
     [:default-executor-parallelism {:optional true} ::sm/int] ;; REVIEW
     [:worker-default-parallelism {:optional true} ::sm/int]
     [:worker-webhook-parallelism {:optional true} ::sm/int]
+    [:worker-cron-parallelism {:optional true} ::sm/int]
+    [:worker-binfile-parallelism {:optional true} ::sm/int]
 
     [:database-password {:optional true} [:maybe :string]]
     [:database-uri {:optional true} ::sm/uri]
@@ -207,6 +251,8 @@
     [:quotes-font-variants-per-team {:optional true} ::sm/int]
     [:quotes-comment-threads-per-file {:optional true} ::sm/int]
     [:quotes-comments-per-file {:optional true} ::sm/int]
+    [:quotes-export-jobs-per-profile {:optional true} ::sm/int]
+    [:quotes-import-jobs-per-profile {:optional true} ::sm/int]
     [:quotes-snapshots-per-file {:optional true} ::sm/int]
     [:quotes-snapshots-per-team {:optional true} ::sm/int]
     [:quotes-team-access-requests-per-team {:optional true} ::sm/int]
@@ -368,6 +414,7 @@
   [& {:keys [prefix default] :or {prefix "penpot"}}]
   (->> (read-env prefix)
        (merge default)
+       (normalize-tenant)
        (decode-config)))
 
 (def version
@@ -422,6 +469,41 @@
   [& segments]
   (apply join-uri (c/get config :public-uri) segments))
 
+(defn get-jobs-lease
+  "Max time a job can run without touching modified_at (heartbeat or
+  progress) before the dispatcher marks it as `aborted` (system-side
+  terminal, never retried, reported with an error log)."
+  []
+  (or (c/get config :jobs-lease)
+      (ct/duration {:minutes 30})))
+
+(defn get-jobs-request-timeout
+  "Default timeout for the ephemeral request! calls (waiting for the
+  reply-key blpop); can be overridden per call. Any override is applied
+  by raising the pooled connection command timeout for the duration of
+  the call, which the pool restores on return."
+  []
+  (or (c/get config :jobs-request-timeout)
+      (ct/duration {:minutes 2})))
+
+(defn get-jobs-retention
+  "How long terminal (completed/failed/cancelled/aborted) internal job rows are
+  kept before the jobs GC deletes them. The legacy `task` table is not
+  touched by the jobs GC: while both versions run in parallel it is
+  cleaned by the legacy `tasks-gc` of that version."
+  []
+  (or (c/get config :jobs-retention)
+      (ct/duration {:days 7})))
+
+(defn get-jobs-user-ttl
+  "How long a user-facing job, and the artifact it owns, is kept before
+  the jobs GC deletes the row and marks the object for the storage GC.
+  Not the same as `:jobs-retention`, which only sweeps terminal internal
+  rows."
+  []
+  (or (c/get config :jobs-user-ttl)
+      (ct/duration {:days 7})))
+
 (defn get
   "A configuration getter. Helps code be more testable."
   ([key]

backend/src/app/db.clj

@@ -583,12 +583,70 @@
      (l/trc :hint "explicit rollback requested (savepoint)")
      (.rollback conn sp))))
 
+(def ^:dynamic *after-commit-context* nil)
+
+;; the connection the ambient transaction owns, bound by `transact!` for
+;; the extent of its body. It is what tells a nested call (same
+;; connection) from an independent one (any other).
+(def ^:dynamic *after-commit-conn* nil)
+
+(defn- run-after-commit-callbacks
+  [callbacks]
+  (doseq [callback callbacks]
+    (try
+      (callback)
+      (catch Throwable cause
+        (l/wrn :hint "after-commit callback failed" :cause cause)))))
+
+(defn after-commit
+  "Run a callback after the current transaction commits.
+
+  The callback runs immediately when called outside a transaction.
+
+  Nesting follows the connection, not an ambient flag. A transaction
+  opened on the connection the surrounding one owns is part of it and
+  shares its context, so callbacks are never drained before the
+  transaction that owns the connection commits. A transaction opened on
+  any other connection is an independent unit of work: it owns its
+  context, drains its callbacks at its own commit and drops them if it
+  rolls back."
+  [f]
+  (if *after-commit-context*
+    (swap! *after-commit-context* conj f)
+    (f)))
+
+(defn- joins-transaction?
+  "True when `transactable` is the very connection the ambient
+  transaction owns, which is what makes this call part of it."
+  [transactable]
+  (boolean
+   (and *after-commit-conn*
+        (identical? transactable *after-commit-conn*))))
+
 (defn transact!
   "A lower-level function for executing function in a transaction"
   ([transactable f] (transact! transactable f {}))
   ([transactable f opts]
-   (binding [next.jdbc.transaction/*nested-tx* :ignore]
-     (jdbc/transact transactable f opts))))
+   (if (joins-transaction? transactable)
+     (binding [next.jdbc.transaction/*nested-tx* :ignore]
+       (jdbc/transact transactable f opts))
+     (let [context (atom [])]
+       (try
+         (let [result (binding [*after-commit-context* context]
+                        (binding [next.jdbc.transaction/*nested-tx* :ignore]
+                          (jdbc/transact
+                           transactable
+                           (fn [conn]
+                             (binding [*after-commit-conn* conn]
+                               (f conn)))
+                           opts)))]
+           ;; a rollback-only transaction commits nothing, so its
+           ;; callbacks have nothing to follow
+           (when-not (:rollback-only opts)
+             (run-after-commit-callbacks @context))
+           result)
+         (finally
+           (reset! context [])))))))
 
 (defn tx-run!
   "Run a function in a transaction."

backend/src/app/email.clj

@@ -6,6 +6,7 @@
 
 (ns app.email
   "Main api for send emails."
+  (:refer-clojure :exclude [send])
   (:require
    [app.common.data :as d]
    [app.common.data.macros :as dm]
@@ -16,8 +17,8 @@
    [app.config :as cf]
    [app.db :as db]
    [app.db.sql :as sql]
+   [app.jobs :as jobs]
    [app.util.template :as tmpl]
-   [app.worker :as wrk]
    [clojure.java.io :as io]
    [cuerdas.core :as str]
    [integrant.core :as ig])
@@ -296,23 +297,39 @@
   [email-factory params]
   (email-factory params))
 
-(defn send!
-  "Schedule an already defined email to be sent using asynchronously
-  using worker task."
-  [{:keys [::conn ::factory] :as params}]
-  (assert (db/connectable? conn) "expected a valid database connection or pool")
+(defn send
+  "Schedule an already defined email to be sent asynchronously
+  using the unified jobs machinery. The first `cfg` parameter is the
+  connectable context that provides the `::jobs/defs` registry (an RPC
+  method cfg or the system) and can provide a default connection; the
+  second `params` provides the email data and optionally `::factory`
+  and `::reuse-conn`. When `::reuse-conn` is true the submission
+  reuses the caller's existing `::db/conn` (from cfg) instead of
+  acquiring a fresh one from the pool.
 
+  NOTE: this renders the email before submitting, so the full subject
+  and body are stored as job params. Prefer `send-template` for
+  templated emails: it stores only the template id plus the raw data
+  and renders inside the worker. `send` is kept for raw (non
+  templated) emails and for draining jobs submitted in the old
+  format."
+  [cfg {:keys [::reuse-conn ::factory] :as params}]
+  (when (and reuse-conn (nil? (::db/conn cfg)))
+    (ex/raise :type :validation
+              :code :missing-connection
+              :hint "reuse-conn requires the caller cfg to carry ::db/conn (call inside a transaction)"))
   (let [email (if factory
                 (factory params)
                 (-> params
-                    (dissoc params)
+                    (dissoc ::reuse-conn ::factory)
                     (check-params)))]
-    (wrk/submit! {::wrk/task :sendmail
-                  ::wrk/delay 0
-                  ::wrk/max-retries 4
-                  ::wrk/priority 200
-                  ::db/conn conn
-                  ::wrk/params email})))
+    (jobs/submit (cond-> (dissoc cfg ::db/conn)
+                   reuse-conn (assoc ::db/conn (::db/conn cfg)))
+                 {::jobs/name :sendmail
+                  ::jobs/delay 0
+                  ::jobs/max-retries 4
+                  ::jobs/priority 200
+                  ::jobs/params email})))
 
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;; SENDMAIL FN / TASK HANDLER
@@ -320,39 +337,44 @@
 
 (declare send-to-logger!)
 
+(defn- sendmail-impl
+  [cfg params]
+  (when (contains? cf/flags :smtp)
+    (let [session (create-smtp-session cfg)]
+      (with-open [transport (.getTransport session (if (::ssl cfg) "smtps" "smtp"))]
+        (.connect ^Transport transport
+                  ^String (::host cfg)
+                  ^String (::port cfg)
+                  ^String (::username cfg)
+                  ^String (::password cfg))
+
+        (let [^MimeMessage message (create-smtp-message cfg session params)]
+          (l/dbg :hint "sendmail"
+                 :id (:id params)
+                 :to (:to params)
+                 :subject (str/trim (:subject params)))
+
+          (.sendMessage ^Transport transport
+                        ^MimeMessage message
+                        (.getAllRecipients message))))))
+
+  (when (contains? cf/flags :log-emails)
+    (send-to-logger! cfg params)))
+
 (defmethod ig/init-key ::sendmail
   [_ cfg]
-  (fn [params]
-    (when (contains? cf/flags :smtp)
-      (let [session (create-smtp-session cfg)]
-        (with-open [transport (.getTransport session (if (::ssl cfg) "smtps" "smtp"))]
-          (.connect ^Transport transport
-                    ^String (::host cfg)
-                    ^String (::port cfg)
-                    ^String (::username cfg)
-                    ^String (::password cfg))
+  ;; the job handler: [context params]. The smtp deps are closed over
+  ;; here, and this job has no use for the context.
+  (fn [_context params]
+    (sendmail-impl cfg params)))
 
-          (let [^MimeMessage message (create-smtp-message cfg session params)]
-            (l/dbg :hint "sendmail"
-                   :id (:id params)
-                   :to (:to params)
-                   :subject (str/trim (:subject params)))
-
-            (.sendMessage ^Transport transport
-                          ^MimeMessage message
-                          (.getAllRecipients message))))))
-
-    (when (contains? cf/flags :log-emails)
-      (send-to-logger! cfg params))))
-
-(defmethod ig/assert-key ::handler
-  [_ params]
-  (assert (fn? (::sendmail params)) "expected valid sendmail handler"))
-
-(defmethod ig/init-key ::handler
-  [_ {:keys [::sendmail]}]
-  (fn [{:keys [props] :as task}]
-    (sendmail props)))
+(defmethod ig/init-key ::job-def
+  [_ {sendmail ::sendmail}]
+  {::jobs/name      :sendmail
+   ::jobs/schema    schema:params
+   ::jobs/handler   sendmail
+   ::jobs/decoder   (sm/decoder schema:params sm/json-transformer)
+   ::jobs/validator (sm/validator schema:params)})
 
 (defn- send-to-logger!
   [_ email]
@@ -379,7 +401,7 @@
    [:feedback-subject ::sm/text]
    [:feedback-type ::sm/text]
    [:feedback-content ::sm/text]
-   [:profile :map]])
+   [:profile [:map [:id ::sm/uuid]]]])
 
 (def user-feedback
   "A profile feedback email."
@@ -578,6 +600,125 @@
    :id ::comment-notification
    :schema schema:comment-notification))
 
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;; TEMPLATED SEND (render inside the worker)
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+
+(defn- template-entry
+  "Registry entry for a templated email: the factory that renders it and
+  the JSON encoder/decoder of its data schema. The job `params` column
+  is plain JSON, so values without a JSON form (e.g. the URIs of an
+  organization) are encoded to strings before submit and decoded back
+  to their types before the factory validates and renders them."
+  [factory schema]
+  {::factory factory
+   ::encode  (sm/encoder schema sm/json-transformer)
+   ::decode  (sm/decoder schema sm/json-transformer)})
+
+;; Registry of template id to entry. This map is the single source of
+;; truth for templated mail: `send-template` validates membership and
+;; encodes the data here before submitting (so a job that could never
+;; be rendered is rejected up front), and the worker decodes the data
+;; and renders it with the entry factory. Callers pass the id string
+;; itself (e.g. `"invite-to-team"`); keys stay `::eml/`-namespaced and
+;; values stay plain data, following the params convention.
+(def templates
+  {"feedback"                            (template-entry user-feedback schema:feedback)
+   "register"                            (template-entry register schema:register)
+   "password-recovery"                   (template-entry password-recovery schema:password-recovery)
+   "password-changed"                    (template-entry password-changed schema:password-changed)
+   "change-email"                        (template-entry change-email schema:change-email)
+   "invite-to-team"                      (template-entry invite-to-team schema:invite-to-team)
+   "invite-to-organization"              (template-entry invite-to-organization schema:invite-to-organization)
+   "organization-setup-sso"              (template-entry organization-setup-sso schema:organization-setup-sso)
+   "renewal-notice"                      (template-entry renewal-notice schema:renewal-notice)
+   "join-team"                           (template-entry join-team schema:join-team)
+   "request-file-access"                 (template-entry request-file-access schema:request-file-access)
+   "request-file-access-yourpenpot"      (template-entry request-file-access-yourpenpot schema:request-file-access)
+   "request-file-access-yourpenpot-view" (template-entry request-file-access-yourpenpot-view schema:request-file-access)
+   "request-team-access"                 (template-entry request-team-access schema:request-team-access)
+   "comment-mention"                     (template-entry comment-mention schema:comment-mention)
+   "comment-thread"                      (template-entry comment-thread schema:comment-thread)
+   "comment-notification"                (template-entry comment-notification schema:comment-notification)})
+
+(defn render-template
+  "Render the templated email `template-id` from its JSON-decoded data,
+  as the worker does before sending it."
+  [template-id params]
+  (let [{:keys [::factory ::decode]} (get templates template-id)]
+    (factory (decode params))))
+
+;; Envelope of the job params only: the template data travels encoded
+;; by the template schema and is decoded by `render-template`.
+(def ^:private schema:templated-params
+  [:map {:title "Templated Email Params"}
+   [:template (into [:enum] (sort (keys templates)))]
+   [:to [:or ::sm/email [::sm/vec ::sm/email]]]
+   [:reply-to {:optional true} ::sm/email]
+   [:from {:optional true} ::sm/email]
+   [:lang {:optional true} ::sm/text]
+   [:priority {:optional true} [:enum :high :low]]
+   [:extra-data {:optional true} ::sm/text]
+   [:attachments {:optional true}
+    [:map-of :string :string]]])
+
+(defn send-template
+  "Schedule a templated email to be sent asynchronously using the
+  unified jobs machinery. Unlike `send`, this does NOT render the
+  email: it stores only the template id plus the raw template data as
+  `:send-templated-mail` job params, and the worker renders and sends
+  the email. The `::template` param is the template id string (one of
+  the keys of `templates`, e.g. `invite-to-team`);
+  `::reuse-conn` behaves like in `send`."
+  [cfg {:keys [::template ::reuse-conn] :as params}]
+  (when (and reuse-conn (nil? (::db/conn cfg)))
+    (ex/raise :type :validation
+              :code :missing-connection
+              :hint "reuse-conn requires the caller cfg to carry ::db/conn (call inside a transaction)"))
+  (when (nil? template)
+    (ex/raise :type :validation
+              :code :missing-template
+              :hint "template is mandatory for templated emails"))
+  (when-not (contains? templates template)
+    (ex/raise :type :validation
+              :code :unknown-email-template
+              :hint (str/ffmt "unknown email template: %", template)))
+  (let [encode (get-in templates [template ::encode])
+        recipe (-> params
+                   (dissoc ::reuse-conn ::template)
+                   ;; Present-but-nil extras (e.g. `:file-id` when there is
+                   ;; no file) would fail the job schema, while the old
+                   ;; inline render simply ignored them.
+                   (d/without-nils)
+                   (encode))]
+    (jobs/submit (cond-> (dissoc cfg ::db/conn)
+                   reuse-conn (assoc ::db/conn (::db/conn cfg)))
+                 {::jobs/name :send-templated-mail
+                  ::jobs/delay 0
+                  ::jobs/max-retries 4
+                  ::jobs/priority 200
+                  ::jobs/params (cond-> (assoc recipe :template template)
+                                  ;; `:public-uri` arrives as a URI record
+                                  ;; from config, which plain JSON cannot
+                                  ;; roundtrip; the templates only need its
+                                  ;; string form.
+                                  (some? (:public-uri recipe))
+                                  (update :public-uri str))})))
+
+(defmethod ig/init-key ::templated-job-def
+  [_ {sendmail ::sendmail}]
+  {::jobs/name      :send-templated-mail
+   ::jobs/schema    schema:templated-params
+   ::jobs/handler   (fn [context params]
+                      (let [template-id (:template params)]
+                        (when-not (contains? templates template-id)
+                          (ex/raise :type :internal
+                                    :code :unknown-email-template
+                                    :template-id template-id))
+                        (sendmail context (render-template template-id (dissoc params :template)))))
+   ::jobs/decoder   (sm/decoder schema:templated-params sm/json-transformer)
+   ::jobs/validator (sm/validator schema:templated-params)})
+
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;; BOUNCE/COMPLAINS HELPERS
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

backend/src/app/features/fdata.clj

@@ -51,10 +51,19 @@
   (update file :data process-objects (partial into {})))
 
 (defn enable-objects-map
+  "Ensure every page holds an ObjectsMap. Unloaded pointers are left
+  untouched: the fragment they point to was written by an `encode-file`
+  that already wrapped its objects, and loading it only to wrap it again
+  costs one page read per untouched page on every save. A page whose
+  fragment predates the feature is wrapped when it is first loaded or
+  edited; a data migration that enables the feature must rewrite the
+  data itself."
   [file & _opts]
   (let [update-page
         (fn [page]
-          (update page :objects omap/wrap))
+          (if (and (pmap/pointer-map? page) (not (pmap/loaded? page)))
+            page
+            (update page :objects omap/wrap)))
 
         update-data
         (fn [fdata]
@@ -264,9 +273,13 @@
 (defn load-pointer
   "A database loader pointer helper"
   [cfg file-id id]
+  ;; Keep the include-marked behavior explicit: `db/get*` only filters
+  ;; rows by `deleted_at` when that column is selected, and a fragment a
+  ;; previous pass marked must still load until `objects-gc` purges it.
   (let [fragment (some-> (db/get* cfg :file-data
                                   {:id id :file-id file-id :type "fragment"}
-                                  {::sql/columns [:data :backend :id :metadata]})
+                                  {::sql/columns [:data :backend :id :metadata]
+                                   ::db/remove-deleted false})
                          (update :metadata decode-metadata))]
 
     (l/trc :hint "load pointer"
@@ -297,6 +310,40 @@
                       :type "fragment"
                       :data content})))))
 
+(defmacro with-pointers
+  "Bind the fragment loader for `file-id` around `body`. All pointer
+  derefs must happen inside the body; a pointer derefed outside of it
+  raises because there is no loader bound."
+  [cfg file-id & body]
+  `(binding [pmap/*load-fn* (partial load-pointer ~cfg ~file-id)]
+     ~@body))
+
+(defmacro with-tracked
+  "Bind a fresh tracked set around `body`, without a fragment loader.
+  For write paths that create pointers before any file exists to load
+  from (e.g. file creation): wrapping tracks new pointers for
+  persistence, while any deref still fails loudly."
+  [& body]
+  `(binding [pmap/*tracked* (pmap/create-tracked)]
+     ~@body))
+
+(defmacro with-untracked
+  "Run `body` with pointer tracking disabled. For validation passes
+   that must not track anything for persistence: pointers created
+   inside are not collected, and derefs still go through the bound
+   loader when there is one."
+  [& body]
+  `(binding [pmap/*tracked* nil]
+     ~@body))
+
+(defmacro with-tracked-pointers
+  "Bind a fresh tracked set plus the fragment loader for `file-id`
+  around `body`. For write paths that create or modify pointers."
+  [cfg file-id & body]
+  `(binding [pmap/*tracked* (pmap/create-tracked)
+             pmap/*load-fn* (partial load-pointer ~cfg ~file-id)]
+     ~@body))
+
 (defn process-pointers
   "Apply a function to all pointers on the file. Usuly used for
   dereference the pointer to a plain value before some processing."
@@ -311,14 +358,20 @@
 
 (defn realize-pointers
   "Process a file and remove all instances of pointers realizing them to
-  a plain data. Used in operation where is more efficient have the
-  whole file loaded in memory."
-  [cfg {:keys [id] :as file}]
-  (binding [pmap/*load-fn* (partial load-pointer cfg id)]
-    (update file :data process-pointers deref)))
+  plain data. Accepts a file map (arity 1) or raw file data with its
+  file id (arity 2): a snapshot map carries the snapshot id, so always
+  pass the live file id explicitly."
+  ([cfg {:keys [id] :as file}]
+   (update file :data #(realize-pointers cfg id %)))
+  ([cfg file-id data]
+   (with-pointers cfg file-id
+     (process-pointers data deref))))
 
 (defn get-used-pointer-ids
-  "Given a file, return all pointer ids used in the data."
+  "Given a file data map, return all pointer ids used in the data.
+  This is a plain readout with no liveness meaning of its own: file-gc
+  unions it over the main file and every unpurged snapshot (see
+  `app.tasks.file-gc/clean-fragments!` for the actual liveness rule)."
   [fdata]
   (->> (concat (vals fdata)
                (vals (:pages-index fdata)))
@@ -345,3 +398,50 @@
   (->> file
        (realize-pointers cfg)
        (realize-objects cfg)))
+
+(defn realize-entry
+  "Realize a file data entry (a page or the components map) to plain
+  data for value comparison: deref pointers and convert objects-maps
+  with the same `into {}` production uses. Requires *load-fn* bound."
+  [entry]
+  (let [entry (if (pmap/pointer-map? entry) (deref entry) entry)]
+    (if (and (map? entry) (contains? entry :objects))
+      (let [objects (:objects entry)]
+        (assoc entry :objects (if (omap/objects-map? objects)
+                                (into {} objects)
+                                objects)))
+      entry)))
+
+(defn original-pointers
+  "Snapshot the current pointer layout of file data, mapping each
+  pointer position to its PointerMap. No loading involved."
+  [fdata]
+  {:root  (into {} (filter (fn [[_ v]] (pmap/pointer-map? v))) fdata)
+   :pages (into {} (filter (fn [[_ v]] (pmap/pointer-map? v)))
+                (:pages-index fdata {}))})
+
+(defn reattach-unchanged
+  "Put back the original fragment pointers wherever the current value
+  is value-equal to the stored fragment, so unchanged positions keep
+  their ids and no new fragment rows are persisted for them. `current`
+  may be a PointerMap (file-gc) or a plain page (the migration read
+  path); `orig` always comes from `original-pointers`. Anything added,
+  removed or actually changed keeps the current value. Requires
+  *load-fn* bound."
+  [{:keys [root pages]} file]
+  (letfn [(reattach [current orig]
+            (if (or (identical? current orig)
+                    (= (realize-entry current) (realize-entry orig)))
+              orig
+              current))
+          (reattach-all [m origs]
+            (reduce-kv (fn [m k orig]
+                         (if (contains? m k)
+                           (update m k reattach orig)
+                           m))
+                       m origs))]
+    (update file :data (fn [data]
+                         (cond-> (reattach-all data root)
+                           (contains? data :pages-index)
+                           (update :pages-index reattach-all pages))))))
+

backend/src/app/features/file_snapshots.clj

@@ -19,9 +19,9 @@
    [app.db :as db]
    [app.db.sql :as-alias sql]
    [app.features.fdata :as fdata]
+   [app.jobs :as jobs]
    [app.storage :as sto]
    [app.util.blob :as blob]
-   [app.worker :as wrk]
    [cuerdas.core :as str]))
 
 (def sql:snapshots
@@ -48,11 +48,19 @@
                                    AND fd.type = 'snapshot')
     WHERE c.label IS NOT NULL")
 
+;; `:migrations` is an ordered set everywhere (see `common.types.file`),
+;; and `need-migration?`/`migrate` use set semantics on it: the pgarray
+;; must never surface as a vector, where `contains?`/`set/difference`
+;; would compare indexes instead of names.
+(defn- decode-migrations
+  [value]
+  (into (d/ordered-set) (db/decode-pgarray value [])))
+
 (defn- decode-snapshot
   [snapshot]
   (some-> snapshot
           (-> (d/update-when :metadata fdata/decode-metadata)
-              (d/update-when :migrations db/decode-pgarray [])
+              (d/update-when :migrations decode-migrations)
               (d/update-when :features db/decode-pgarray #{}))))
 
 (def ^:private sql:get-minimal-file
@@ -376,8 +384,13 @@
                :snapshot-id (str (:id snapshot)))
 
         ;; In the same way, on reseting the file data, we need to restore
-        ;; the applied migrations on the moment of taking the snapshot
-        (bfc/update-file! cfg file ::bfc/reset-migrations? true)
+        ;; the applied migrations on the moment of taking the snapshot.
+        ;; The snapshot data may hold shared pointer-mapped pages; the
+        ;; loader stays bound while persisting so any page the encode
+        ;; path does load keeps working (`enable-objects-map` skips the
+        ;; unloaded ones).
+        (fdata/with-pointers cfg file-id
+          (bfc/update-file! cfg file ::bfc/reset-migrations? true))
 
         ;; FIXME: this should be separated functions, we should not have
         ;; inline sql here.
@@ -406,12 +419,12 @@
   (assert (uuid? file-id) "missing file-id")
   (assert (ct/inst? deleted-at) "missing deleted-at")
 
-  (wrk/submit! {::db/conn (db/get-connection cfg)
-                ::wrk/task :delete-object
-                ::wrk/params {:object :snapshot
-                              :deleted-at deleted-at
-                              :file-id file-id
-                              :id id}})
+  (jobs/submit cfg
+               {::jobs/name :delete-object
+                ::jobs/params {:object :snapshot
+                               :deleted-at deleted-at
+                               :file-id file-id
+                               :id id}})
   (db/update! cfg :file-change
               {:deleted-at deleted-at}
               {:id id :file-id file-id}
@@ -441,7 +454,12 @@
 
 (defn reduce-snapshots
   "Process the file snapshots using efficient reduction; the file
-  reduction comes with all snapshots, including maked as deleted"
+  reduction comes with all snapshots, including maked as deleted.
+
+  Snapshot data may hold pointer-mapped pages, so the fragment loader
+  is bound while reducing: consumers that walk page content (media
+  refs, stats) transparently load fragments, consumers that only read
+  pointer ids never trigger a load."
   [cfg file-id xform f init]
   (let [conn  (db/get-connection cfg)
         xform (comp
@@ -449,5 +467,6 @@
                (map (partial fdata/decode-file-data cfg))
                xform)]
 
-    (->> (db/plan conn [sql:get-snapshots file-id] {:fetch-size 1})
-         (transduce xform f init))))
+    (fdata/with-pointers cfg file-id
+      (->> (db/plan conn [sql:get-snapshots file-id] {:fetch-size 1})
+           (transduce xform f init)))))

backend/src/app/http.clj

@@ -19,6 +19,7 @@
    [app.http.errors :as errors]
    [app.http.link-preview :as-alias link-preview]
    [app.http.management :as mgmt]
+   [app.http.mcp-oauth :as-alias mcp-oauth]
    [app.http.middleware :as mw]
    [app.http.security :as sec]
    [app.http.session :as session]
@@ -236,6 +237,7 @@
    [::ws/routes schema:routes]
    [::rpc/routes schema:routes]
    [::oidc/routes schema:routes]
+   [::mcp-oauth/routes schema:routes]
    [::assets/routes schema:routes]
    [::link-preview/routes schema:routes]
    [::debug/routes schema:routes]
@@ -278,4 +280,7 @@
 
      (::ws/routes cfg)
      (::oidc/routes cfg)
-     (::rpc/routes cfg)]]))
+     (::rpc/routes cfg)]
+    ;; OAuth owns form parsing, protocol JSON names and cross-origin handling;
+    ;; it must not inherit the RPC transformations and session middleware above.
+    (::mcp-oauth/routes cfg)]))

backend/src/app/http/access_token.clj

@@ -14,14 +14,20 @@
    [app.setup :as-alias setup]
    [app.tokens :as tokens]))
 
+(def mcp-token-issuer "urn:penpot:mcp-token")
+
+(def ^:private valid-issuers
+  #{"access-token" mcp-token-issuer})
+
 (defn decode-token
   [cfg token]
   (try
-    (tokens/verify cfg {:token token :iss "access-token"})
-    (catch Throwable cause
-      (l/trc :hint "exception on decoding token"
-             :token token
-             :cause cause))))
+    (let [claims (tokens/verify cfg {:token token})]
+      (when (contains? valid-issuers (:iss claims))
+        claims))
+    ;; The cause is not logged because its data carries the token
+    (catch Throwable _
+      (l/trc :hint "exception on decoding token"))))
 
 (def sql:get-token-data
   "SELECT perms, profile_id, expires_at, type
@@ -34,32 +40,48 @@
   [pool claims]
   (when-not (db/read-only? pool)
     (when-let [token-id (get claims :tid)]
-      (some-> (db/exec-one! pool [sql:get-token-data token-id])
-              (update :perms db/decode-pgarray #{})))))
+      (let [token       (db/exec-one! pool [sql:get-token-data token-id])
+            mcp-claims? (= mcp-token-issuer (:iss claims))
+            mcp-token?  (= "mcp" (:type token))]
+        (when (and (= (:uid claims) (:profile-id token))
+                   (= mcp-claims? mcp-token?))
+          (update token :perms db/decode-pgarray #{}))))))
 
 (defn- wrap-authz
-  [handler {:keys [::db/pool]}]
-  (fn [request]
-    (let [{:keys [type claims]} (get request ::http/auth-data)]
-      (if (= :token type)
-        (let [{:keys [perms profile-id expires-at type]} (some->> claims (get-token-data pool))
-              token-id (get claims :tid)]
-          (handler (cond-> request
-                     (some? perms)
-                     (assoc ::perms perms)
-                     (some? profile-id)
-                     (assoc ::profile-id profile-id)
-                     (some? expires-at)
-                     (assoc ::expires-at expires-at)
-                     (some? token-id)
-                     (assoc ::id token-id)
-                     (some? type)
-                     (assoc ::type type))))
+  "Authenticate requests carrying an access token. Personal tokens need the
+  `access-tokens` flag. MCP tokens are restricted to specific commands, so
+  only RPC routes opt in (the RPC wrapper enforces those restrictions); they
+  need either the `mcp` or the `access-tokens` flag."
+  [handler {:keys [::db/pool ::allow-mcp]}]
+  (let [personal-enabled? (contains? cf/flags :access-tokens)
+        mcp-enabled?      (and allow-mcp
+                               (or personal-enabled?
+                                   (contains? cf/flags :mcp)))]
+    (fn [request]
+      (let [{:keys [type claims]} (get request ::http/auth-data)]
+        (if (and (= :token type)
+                 (if (= mcp-token-issuer (:iss claims))
+                   mcp-enabled?
+                   personal-enabled?))
+          (let [{:keys [perms profile-id expires-at type]} (some->> claims (get-token-data pool))
+                token-id (get claims :tid)]
+            (handler (cond-> request
+                       (some? perms)
+                       (assoc ::perms perms)
+                       (some? profile-id)
+                       (assoc ::profile-id profile-id)
+                       (some? expires-at)
+                       (assoc ::expires-at expires-at)
+                       (some? token-id)
+                       (assoc ::id token-id)
+                       (some? type)
+                       (assoc ::type type))))
 
-        (handler request)))))
+          (handler request))))))
 
 (def authz
   {:name ::authz
    :compile (fn [& _]
-              (when (contains? cf/flags :access-tokens)
+              (when (or (contains? cf/flags :access-tokens)
+                        (contains? cf/flags :mcp))
                 wrap-authz))})

backend/src/app/http/assets.clj

@@ -100,7 +100,7 @@
         headers (cond-> {"location" (str url)
                          "x-host"   (cond-> host port (str ":" port))
                          "x-mtype"  (-> obj meta :content-type)
-                         "cache-control" (str "max-age=" (inst-ms cch-max-age))}
+                         "cache-control" (str "max-age=" (ct/seconds cch-max-age))}
                   (not public?)
                   (assoc "content-disposition" "attachment"))]
     {::yres/status  307
@@ -115,7 +115,7 @@
         bucket  (:bucket mdata)
         headers (cond-> {"x-accel-redirect" (:path purl)
                          "content-type" (:content-type mdata)
-                         "cache-control" (str "max-age=" (inst-ms cch-max-age))}
+                         "cache-control" (str "max-age=" (ct/seconds cch-max-age))}
                   (not (public-bucket? bucket))
                   (assoc "content-disposition" "attachment"))]
     {::yres/status 204
@@ -150,14 +150,22 @@
   [request]
   (some? (request-profile-id request)))
 
-(defn- tempfile-owner-match?
-  "Check if the request's profile-id matches the tempfile's stored owner.
-   Returns true if no profile-id was stored (legacy objects)."
+(def ^:private owner-scoped-buckets
+  "Buckets whose objects belong to a profile: a request from anybody else
+  must not reach them."
+  #{sto/tempfile-bucket sto/job-resource-bucket})
+
+(defn- owner-match?
+  "Check that the request belongs to the profile that owns the object.
+  A missing owner is only tolerated in `tempfile`, where objects predate
+  the metadata; the `job-resource` bucket always carries one, so an
+  object without it fails closed."
   [obj request]
-  (let [stored-profile-id (:profile-id (meta obj))
-        request-profile-id (request-profile-id request)]
-    (or (nil? stored-profile-id)
-        (= stored-profile-id request-profile-id))))
+  (let [stored  (-> obj meta :profile-id)
+        owner   (request-profile-id request)
+        legacy? (= sto/tempfile-bucket (-> obj meta :bucket))]
+    (or (and legacy? (nil? stored))
+        (= stored owner))))
 
 (defn- serve-object-measured
   "Serve `obj`, recording one asset metric per outcome. A failure is
@@ -175,7 +183,9 @@
   "Handler that serves storage objects by id.
    For non-public buckets (e.g. profile), requires authentication
    via session cookie or access token.
-   For tempfile bucket, also requires ownership (profile-id match)."
+   For the owner-scoped buckets (tempfile, job-resource), also requires
+   ownership: the request must come from the profile the object belongs
+   to."
   [{:keys [::sto/storage] :as cfg} request]
   (let [id  (get-id request)
         obj (sto/get-object storage id)]
@@ -193,8 +203,8 @@
 
       ;; The response stays 404 to avoid leaking existence, but the
       ;; metric records the internal 401 outcome.
-      (and (= (-> obj meta :bucket) sto/tempfile-bucket)
-           (not (tempfile-owner-match? obj request)))
+      (and (contains? owner-scoped-buckets (-> obj meta :bucket))
+           (not (owner-match? obj request)))
       (do
         (emit-asset! cfg "by-id" obj 401)
         {::yres/status 404})

backend/src/app/http/debug.clj

@@ -359,10 +359,11 @@
     (let [path (tmp/tempfile :prefix "penpot.export." :min-age "30m")]
       (with-open [output (io/output-stream path)]
         (-> cfg
-            (assoc ::bfc/ids file-ids)
+            (assoc ::bfc/file-ids file-ids)
             (assoc ::bfc/embed-assets embed?)
             (assoc ::bfc/include-libraries libs?)
-            (bf.v3/export-files! output)))
+            (assoc ::bfc/output output)
+            (bf.v3/export-files!)))
 
       (if clone?
         (let [profile    (profile/get-profile pool profile-id)
@@ -784,8 +785,12 @@
 
     (db/tx-run! (assoc cfg ::db/rollback true)
                 (fn [cfg]
-                  (let [file (bfc/get-file cfg file-id)
-                        libs (bfc/get-resolved-file-libraries cfg file-id)]
+                  ;; Validation walks every page and component, so the
+                  ;; file is read realized; the library map is built
+                  ;; from the file (not its id) and its own loader
+                  ;; resolves each library's pointers.
+                  (let [file (bfc/get-file cfg file-id :realize? true)
+                        libs (bfc/get-resolved-file-libraries cfg file)]
                     (if file
                       (let [errors (cfv/validate-file file libs)]
                         {::yres/status  200

backend/src/app/http/errors.clj

@@ -17,6 +17,7 @@
    [app.http.session :as-alias session]
    [app.util.inet :as inet]
    [clojure.spec.alpha :as s]
+   [cuerdas.core :as str]
    [yetti.request :as yreq]
    [yetti.response :as yres]))
 
@@ -223,6 +224,25 @@
     "25P03" "The transaction was idle too long and was cancelled"
     "A database error occurred"))
 
+(defn- transient-sql-state?
+  "True for SQLSTATE codes of failures that pass on their own: lost or
+  refused connections, lack of resources, a server shutting down, and
+  serialization failures or deadlocks of concurrent transactions."
+  [state]
+  (and (string? state)
+       (or (str/starts-with? state "08")
+           (str/starts-with? state "53")
+           (contains? #{"57P01" "57P02" "57P03" "40001" "40P01"} state))))
+
+(defn- unavailable
+  "Response for a failure that passes on its own: the request can be sent
+  again."
+  [code hint]
+  {::yres/status 503
+   ::yres/body {:type :server-error
+                :code code
+                :hint hint}})
+
 (defmethod handle-exception org.postgresql.util.PSQLException
   [error request parent-cause]
   (let [state (.getSQLState ^java.sql.SQLException error)
@@ -231,6 +251,9 @@
       (l/error :hint "postgresql error"
                :cause cause)
       (cond
+        (transient-sql-state? state)
+        (unavailable :database-unavailable "The database is not available right now")
+
         (= state "57014")
         {::yres/status 504
          ::yres/body {:type :server-error
@@ -271,13 +294,37 @@
                          (assoc :type :server-error)
                          (update :code #(or % :unhandled)))}))))
 
+(defmethod handle-exception java.sql.SQLTransientException
+  [cause request _]
+  (binding [l/*context* (request->context request)]
+    (l/wrn :hint "transient database error" :cause cause)
+    (unavailable :database-unavailable "The database is not available right now")))
+
+(defn- transient-io-exception?
+  "True for I/O failures that pass on their own: a lost, refused or timed
+  out connection, or a host name that did not resolve. A parse error or
+  data that cannot be read fails again on every attempt."
+  [cause]
+  (or (instance? java.net.SocketException cause)
+      (instance? java.io.InterruptedIOException cause)
+      (instance? java.net.UnknownHostException cause)
+      (instance? java.net.http.HttpTimeoutException cause)))
+
 (defmethod handle-exception java.io.IOException
   [cause request _]
   (binding [l/*context* (request->context request)]
     (l/wrn :hint "io exception" :cause cause)
-    {::yres/status 500
-     ::yres/body {:type :server-error
-                  :code :io-exception}}))
+    (if (transient-io-exception? cause)
+      (unavailable :io-exception "An input/output operation failed")
+      {::yres/status 500
+       ::yres/body {:type :server-error
+                    :code :io-exception}})))
+
+(defmethod handle-exception InterruptedException
+  [cause request _]
+  (binding [l/*context* (request->context request)]
+    (l/wrn :hint "interrupted request" :cause cause)
+    (unavailable :interrupted "The request was interrupted")))
 
 (defmethod handle-exception java.util.concurrent.CompletionException
   [cause request _]

backend/src/app/http/mcp_oauth.clj

@@ -0,0 +1,353 @@
+;; This Source Code Form is subject to the terms of the Mozilla Public
+;; License, v. 2.0. If a copy of the MPL was not distributed with this
+;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
+;;
+;; Copyright (c) KALEIDOS SUBSIDIARY SL
+
+(ns app.http.mcp-oauth
+  "MCP-specific OAuth authorization server.
+
+  The authorize endpoint stages a request; the existing Penpot session identifies
+  the user on the approval page. Approval creates a one-use code, and token
+  exchange returns the existing fixed MCP user token after verifying PKCE.
+  Client registration, login sessions, and authorization codes are distinct."
+  (:require
+   [app.common.uuid :as uuid]
+   [app.config :as cf]
+   [app.db :as db]
+   [app.http.errors :as errors]
+   [app.http.middleware :as mw]
+   [app.http.session :as session]
+   [app.mcp.oauth-clients :as clients]
+   [app.mcp.oauth-state :as state]
+   [app.rpc.commands.access-token :as access-token]
+   [app.util.json :as json]
+   [clojure.string :as str]
+   [integrant.core :as ig]
+   [yetti.request :as yreq]
+   [yetti.response :as yres])
+  (:import
+   java.io.InputStream
+   java.net.URI
+   java.net.URLDecoder
+   java.net.URLEncoder
+   java.nio.charset.StandardCharsets
+   java.util.Base64))
+
+(defn issuer [] (str (cf/get-public-uri "api/mcp/oauth")))
+(defn resource [] (str (cf/get-public-uri "mcp/stream")))
+
+(defn authorization-metadata
+  "OAuth capabilities and endpoints advertised to MCP clients."
+  []
+  {:issuer (issuer)
+   :authorization_endpoint (str (issuer) "/authorize")
+   :token_endpoint (str (issuer) "/token")
+   :registration_endpoint (str (issuer) "/register")
+   :response_types_supported ["code"]
+   :grant_types_supported ["authorization_code"]
+   :code_challenge_methods_supported ["S256"]
+   :token_endpoint_auth_methods_supported ["none" "client_secret_basic" "client_secret_post"]
+   :scopes_supported ["mcp"]
+   :authorization_response_iss_parameter_supported true
+   :client_id_metadata_document_supported true})
+
+(defn resource-metadata
+  "Protected-resource metadata referenced by the MCP server's 401 challenge."
+  []
+  {:resource (resource)
+   :authorization_servers [(issuer)]
+   :scopes_supported ["mcp"]
+   :bearer_methods_supported ["header"]
+   :resource_name "Penpot MCP"})
+
+(defn- response
+  ([body] (response 200 body))
+  ([status body]
+   {::yres/status status
+    ::yres/headers {"content-type" "application/json"
+                    "cache-control" "no-store"
+                    "pragma" "no-cache"
+                    "referrer-policy" "no-referrer"}
+    ::yres/body (json/encode-str body)}))
+
+(defn- redirect
+  [url]
+  {::yres/status 303
+   ::yres/headers {"location" url "cache-control" "no-store"
+                   "referrer-policy" "no-referrer"}
+   ::yres/body ""})
+
+(defn- encode [^String value] (URLEncoder/encode value StandardCharsets/UTF_8))
+(defn- decode
+  [^String value]
+  (try (URLDecoder/decode value StandardCharsets/UTF_8)
+       (catch IllegalArgumentException _ (clients/invalid! "invalid_request"))))
+
+(defn- parse-form
+  "Bounded OAuth form/query parser preserving protocol parameter names.
+  Rejects duplicate keys so validation never depends on which value wins."
+  [query]
+  (when (> (count query) 16384) (clients/invalid! "invalid_request"))
+  (reduce (fn [result part]
+            (let [[key value] (str/split part #"=" 2)
+                  key         (keyword (decode key))]
+              (when (contains? result key) (clients/invalid! "invalid_request"))
+              (assoc result key (decode (or value "")))))
+          {}
+          (remove str/blank? (str/split (or query "") #"&"))))
+
+(defn- read-body
+  [request content-type]
+  (when-not (str/starts-with? (or (yreq/get-header request "content-type") "") content-type)
+    (clients/invalid! "invalid_request"))
+  (with-open [^InputStream body (yreq/body request)]
+    (when-not body (clients/invalid! "invalid_request"))
+    (let [bytes (.readNBytes body 16385)]
+      (when (> (alength bytes) 16384) (clients/invalid! "invalid_request"))
+      (String. bytes StandardCharsets/UTF_8))))
+
+(defn- append-query
+  [url params]
+  (str url (if (str/includes? url "?") "&" "?")
+       (str/join "&" (for [[key value] params :when (some? value)]
+                       (str (encode (name key)) "=" (encode value))))))
+
+(defn- limit!
+  [cfg request kind limit]
+  (when-not (state/allow-request! cfg kind (str (yreq/remote-addr request)) limit)
+    (clients/invalid! "temporarily_unavailable" 429)))
+
+(defn- authorize
+  "Validate and stage an authorization request before user login or consent.
+  The browser receives only an opaque handle to the ten-minute Redis record."
+  [cfg request]
+  (limit! cfg request :authorize-rate 60)
+  (let [params   (parse-form (yreq/query request))
+        client   (clients/resolve-client cfg (:client_id params))
+        callback (:redirect_uri params)]
+    ;; Never redirect until the client and its callback have been checked.
+    (when-not (clients/redirect-matches? client callback)
+      (clients/invalid! "invalid_request"))
+    (when-not (and (= "code" (:response_type params))
+                   (= "S256" (:code_challenge_method params))
+                   (string? (:code_challenge params))
+                   (re-matches #"[A-Za-z0-9_-]{43}" (:code_challenge params))
+                   (= "mcp" (get params :scope "mcp"))
+                   (= (resource) (get params :resource (resource)))
+                   (<= (count (:state params)) 2048))
+      (clients/invalid! "invalid_request"))
+    (let [handle (state/create! cfg :request
+                                {:client-id (:client_id client)
+                                 :client-name (:client_name client)
+                                 :logo-uri (:logo_uri client)
+                                 :redirect-uri callback
+                                 :challenge (:code_challenge params)
+                                 :state (:state params)
+                                 :resource (resource)
+                                 :scope "mcp"}
+                                600)]
+      (redirect (append-query (str (cf/get-public-uri))
+                              {:screen "mcp-authorize" :request handle})))))
+
+(defn- session-binding
+  "Profile and login-session identity used to bind one approval request."
+  [request]
+  (let [session (session/get-session request)]
+    (when-not (:profile-id session) (clients/invalid! "login_required" 401))
+    {:profile-id (str (:profile-id session))
+     :session-id (state/digest (str (:id session)))}))
+
+(defn- approval-request
+  "Bind a pending request to its first authenticated session and return UI data.
+  Compare-and-replace prevents another session from taking over that request."
+  [cfg request]
+  (let [binding (session-binding request)
+        handle  (:request (parse-form (yreq/query request)))
+        record  (state/read! cfg :request handle)]
+    (when-not record (clients/invalid! "invalid_request"))
+    (let [value (:value record)]
+      (if (:session-id value)
+        (when-not (= binding (select-keys value [:profile-id :session-id]))
+          (clients/invalid! "access_denied" 403))
+        (state/replace! cfg :request handle record
+                        (merge value binding {:csrf (state/random-secret)}))))
+    ;; Another GET may have won the binding race. Read the winning record so
+    ;; same-session retries share its CSRF value; other sessions cannot proceed.
+    (let [value (:value (state/read! cfg :request handle))]
+      (when-not (= binding (select-keys value [:profile-id :session-id]))
+        (clients/invalid! "invalid_request"))
+      (response (select-keys value [:client-id :client-name :logo-uri :csrf :scope])))))
+
+(defn- same-origin?
+  [request]
+  (let [public (URI. (str (cf/get-public-uri)))
+        origin (str (.getScheme public) "://" (.getRawAuthority public))]
+    (= origin (yreq/get-header request "origin"))))
+
+(defn- decision
+  "Accept one session-bound approval/denial and return the client callback.
+  Origin and CSRF checks protect the cookie-authenticated decision endpoint."
+  [cfg request]
+  (when-not (same-origin? request) (clients/invalid! "access_denied" 403))
+  (let [binding (session-binding request)
+        params  (parse-form (read-body request "application/x-www-form-urlencoded"))
+        handle  (:request params)
+        record  (state/read! cfg :request handle)
+        value   (:value record)]
+    (when-not (and record
+                   (= binding (select-keys value [:profile-id :session-id]))
+                   (state/secret= (:csrf value) (:csrf params))
+                   (contains? #{"approve" "deny"} (:decision params)))
+      (clients/invalid! "invalid_request"))
+    ;; Consume before provisioning: concurrent clicks must not issue two codes.
+    ;; A later failure requires a fresh authorization attempt.
+    (when-not (state/consume! cfg :request handle record)
+      (clients/invalid! "invalid_request"))
+    (let [result (if (= "deny" (:decision params))
+                   {:error "access_denied"}
+                   (let [token (db/tx-run! cfg access-token/ensure-mcp-access!
+                                           (uuid/parse (:profile-id value)))
+                         ;; Retain the token row ID, never the fixed token itself.
+                         code  (state/create! cfg :code
+                                              (-> value
+                                                  (dissoc :csrf :session-id :client-name)
+                                                  (assoc :token-id (str (:id token))))
+                                              120)]
+                     {:code code}))]
+      (response {:redirect-uri (append-query (:redirect-uri value)
+                                             (merge result {:state (:state value)
+                                                            :iss (issuer)}))}))))
+
+(defn- client-authentication
+  "Extract one OAuth client-authentication method, rejecting mixed credentials.
+  Basic credentials are form-decoded after splitting the decoded header."
+  [request params]
+  (let [header (yreq/get-header request "authorization")]
+    (if header
+      (do
+        (when (or (:client_secret params) (not (str/starts-with? header "Basic ")))
+          (clients/invalid! "invalid_client" 401))
+        (let [decoded     (try (String. (.decode (Base64/getDecoder) (subs header 6))
+                                        StandardCharsets/UTF_8)
+                               (catch IllegalArgumentException _
+                                 (clients/invalid! "invalid_client" 401)))
+              [id secret] (str/split decoded #":" 2)
+              id          (decode id)]
+          (when (or (nil? secret)
+                    (and (:client_id params) (not= id (:client_id params))))
+            (clients/invalid! "invalid_client" 401))
+          {:id id :secret (decode secret) :method "client_secret_basic"}))
+      {:id (:client_id params) :secret (:client_secret params)
+       :method (if (:client_secret params) "client_secret_post" "none")})))
+
+(defn- token
+  "Exchange a code for the fixed MCP token after client and PKCE verification."
+  [cfg request]
+  (limit! cfg request :token-rate 120)
+  (let [params (parse-form (read-body request "application/x-www-form-urlencoded"))
+        auth   (client-authentication request params)
+        record (state/read! cfg :code (:code params))
+        value  (:value record)]
+    (when-not (= "authorization_code" (:grant_type params))
+      (clients/invalid! "unsupported_grant_type"))
+    ;; Compare the exact client ID, not decoded metadata: a self-contained ID
+    ;; edited after approval identifies a different client for this exchange.
+    (when-not (and record
+                   (= (:id auth) (:client-id value))
+                   (= (:redirect_uri params) (:redirect-uri value))
+                   (= (:resource value) (get params :resource (:resource value)))
+                   (string? (:code_verifier params))
+                   (re-matches #"[A-Za-z0-9._~-]{43,128}" (:code_verifier params))
+                   (state/secret= (:challenge value) (state/digest (:code_verifier params))))
+      (clients/invalid! "invalid_grant"))
+    ;; Resolve again because an HTTPS metadata document may have changed
+    ;; since approval; its current callback and authentication method apply.
+    (let [client (clients/resolve-client cfg (:id auth))]
+      (when-not (and (= (:method auth) (:token_endpoint_auth_method client))
+                     (clients/redirect-matches? client (:redirect-uri value))
+                     (or (= "none" (:method auth))
+                         (and (string? (:secret auth))
+                              (state/secret= (:secret-hash client)
+                                             (state/digest (:secret auth))))))
+        (clients/invalid! "invalid_client" 401)))
+    ;; Validate before consuming so an invalid verifier/secret cannot burn the
+    ;; code. Atomic consumption allows only one valid exchange to succeed.
+    (when-not (state/consume! cfg :code (:code params) record)
+      (clients/invalid! "invalid_grant"))
+    ;; Regeneration deletes the old token row. Never substitute the profile's
+    ;; latest token for the one this approval authorized, nor return it expired.
+    (let [row (db/exec-one! (::db/pool cfg)
+                            [(str "SELECT token FROM access_token"
+                                  " WHERE id = ? AND profile_id = ? AND type = 'mcp'"
+                                  " AND (expires_at IS NULL OR expires_at > now())")
+                             (uuid/parse (:token-id value))
+                             (uuid/parse (:profile-id value))])]
+      (when-not row (clients/invalid! "invalid_grant"))
+      (response {:access_token (:token row) :token_type "Bearer" :scope "mcp"}))))
+
+(defn- register
+  "Issue a self-contained client registration; does not grant user access."
+  [cfg request]
+  (limit! cfg request :registration-rate 10)
+  (let [body     (read-body request "application/json")
+        metadata (try (json/decode body)
+                      (catch Exception _ (clients/invalid! "invalid_request")))]
+    (response 201 (clients/register! cfg metadata))))
+
+(defn- wrap-errors
+  "Answer every failure with an OAuth error body. Unexpected failures also go
+  through the shared HTTP error handler, only for its logging; these routes
+  never set request params, so codes and credentials stay out of the log."
+  [handler]
+  (fn [request]
+    (try (handler request)
+         (catch Exception cause
+           (let [{:keys [oauth-error status]} (ex-data cause)
+                 status                       (or status (if (and oauth-error (not= oauth-error "temporarily_unavailable")) 400 503))]
+             (when-not oauth-error
+               (errors/handle cause request))
+             (cond-> (response status {:error (or oauth-error "temporarily_unavailable")})
+               (and (= status 401) (= oauth-error "invalid_client"))
+               (assoc-in [::yres/headers "www-authenticate"] "Basic realm=\"Penpot MCP\"")))))))
+
+(defn- wrap-public
+  "CORS for discovery, registration and exchange without browser credentials.
+  Cookie-authenticated approval endpoints must not use this wrapper."
+  [handler]
+  (fn [request]
+    (let [response (if (= :options (yreq/method request))
+                     {::yres/status 204}
+                     (handler request))]
+      (update response ::yres/headers merge
+              {"access-control-allow-origin" "*"
+               "access-control-allow-methods" "GET, POST, OPTIONS"
+               "access-control-allow-headers" "Content-Type, Authorization"
+               "access-control-expose-headers" "WWW-Authenticate"}))))
+
+(defn- endpoint
+  [cfg handler method public?]
+  {:handler (partial handler cfg)
+   :middleware (into (if public? [{:compile (constantly wrap-public)}] [])
+                     [{:compile (constantly wrap-errors)} [mw/restrict-methods]])
+   :allowed-methods (cond-> #{method} public? (conj :options))})
+
+;; OAuth endpoints parse forms and emit snake_case JSON independently of RPC
+;; middleware. Only the approval UI endpoints require a Penpot login session.
+(defmethod ig/init-key ::routes
+  [_ cfg]
+  [""
+   ["/.well-known/oauth-authorization-server/api/mcp/oauth"
+    (endpoint cfg (fn [_ _] (response (authorization-metadata))) :get true)]
+   ["/api/mcp/oauth"
+    ["/resource" (endpoint cfg (fn [_ _] (response (resource-metadata))) :get true)]
+    ["/authorize" (endpoint cfg authorize :get false)]
+    ["/register" (endpoint cfg register :post true)]
+    ["/token" (endpoint cfg token :post true)]
+    ;; Error handling wraps the session middleware too, so its failures also
+    ;; get OAuth error bodies and logging.
+    ["" {:middleware [{:compile (constantly wrap-errors)}
+                      [mw/auth {:cookie (partial session/decode-token cfg)}]
+                      [session/authz cfg]]}
+     ["/request" (endpoint cfg approval-request :get false)]
+     ["/decision" (endpoint cfg decision :post false)]]]])

backend/src/app/http/session.clj

@@ -18,6 +18,7 @@
    [app.http :as-alias http]
    [app.http.auth :as-alias http.auth]
    [app.http.session.tasks :as-alias tasks]
+   [app.jobs :as jobs]
    [app.main :as-alias main]
    [app.setup :as-alias setup]
    [app.setup.clock :as clock]
@@ -329,7 +330,7 @@
 ;; TASK: SESSION GC
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 
-(defmethod ig/assert-key ::tasks/gc
+(defmethod ig/assert-key ::job-def
   [_ params]
   (assert (db/pool? (::db/pool params)) "expected valid database pool")
   (assert (ct/duration? (::tasks/max-age params)))
@@ -341,7 +342,7 @@
     (throw (IllegalArgumentException.
             "absolute session max-age must be greater than or equal to idle max-age"))))
 
-(defmethod ig/expand-key ::tasks/gc
+(defmethod ig/expand-key ::job-def
   [k v]
   (let [max-age          (cf/get :auth-token-cookie-max-age default-cookie-max-age)
         max-age-absolute (cf/get :auth-token-cookie-max-age-absolute
@@ -368,10 +369,25 @@
            :deleted result)
     result))
 
-(defmethod ig/init-key ::tasks/gc
-  [_ {:keys [::tasks/max-age ::tasks/max-age-absolute] :as cfg}]
-  (l/dbg :hint "initializing session gc task"
-         :max-age max-age
-         :max-age-absolute max-age-absolute)
-  (fn [_]
-    (db/tx-run! cfg collect-expired-tasks)))
+(declare execute-session-gc)
+
+(def schema:session-gc-params
+  "Params map (no params needed; config-derived only)."
+  [:map {:closed true}])
+
+(defmethod ig/init-key ::job-def
+  [_ cfg]
+  {::jobs/name      :session-gc
+   ::jobs/schema    schema:session-gc-params
+   ::jobs/handler
+   (fn [_context params]
+     (execute-session-gc cfg params))
+   ::jobs/decoder   (sm/decoder schema:session-gc-params sm/json-transformer)
+   ::jobs/validator (sm/validator schema:session-gc-params)})
+
+(defn execute-session-gc
+  "Plain job handler: delete expired http sessions."
+  ([cfg] (execute-session-gc cfg nil))
+  ([cfg _params]
+   (jobs/heartbeat cfg)
+   (db/tx-run! cfg collect-expired-tasks)))

backend/src/app/http/sse.clj

@@ -60,14 +60,13 @@
                                     (partial write! output)
                                     (partial pu/close! output))]
                       (try
-                        (binding [events/*channel* channel]
-                          (let [result (handler)]
-                            (events/tap :end result)))
-
-                        (catch Throwable cause
-                          (let [result (errors/handle' cause request)]
-                            (events/tap channel :error result)))
-
+                        (binding [events/*sink* channel]
+                          (try
+                            (let [result (handler)]
+                              (events/tap :end result))
+                            (catch Throwable cause
+                              (let [result (errors/handle' cause request)]
+                                (events/tap :error result)))))
                         (finally
                           (sp/close! channel)
                           (px/await! listener))))))}))

backend/src/app/jobs/gc.clj

@@ -0,0 +1,164 @@
+;; This Source Code Form is subject to the terms of the Mozilla Public
+;; License, v. 2.0. If a copy of the MPL was not distributed with this
+;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
+;;
+;; Copyright (c) KALEIDOS INC
+
+(ns app.jobs.gc
+  "A maintenance cron task for the unified `job` table:
+
+  - expiration: deletes rows with `expires_at` in the past (any status,
+    user or internal). Expiration is the mechanism for user-facing jobs
+    and for jobs carrying volatile resources.
+  - retention: deletes internal terminal rows (no profile, status in
+    completed/failed/cancelled/aborted) older than the `:jobs-retention` delay.
+    The dormant legacy `task` table is cleaned by the parallel legacy
+    version until the next release, which is what allows PRE and HOURLY to
+    run side by side on the same database and the deploy to roll back.
+    User-facing terminal rows (profile_id NOT NULL) are
+    NOT swept by retention: they are governed by `expires_at` (or stay
+    as ledger history).
+
+  Rows carrying a `resource_id` touch their storage object (touched_at
+  set in the same transaction, mirroring `sto/touch-object!`) so
+  `storage-gc-touched` reclaims it: deleting the job row does not touch
+  the `storage_object` (ON DELETE SET NULL only fires when the object
+  row is deleted, not the job row), and gc-touched only analyzes objects
+  already marked with `touched_at` — without the explicit touch the
+  object would be orphaned.
+
+  Each batch deletes and touches in its own transaction: a crash
+  leaves the remaining batches for the next tick, and an object can
+  never lose its referencing row without being marked for reclaim.
+
+  `job` carries a deletion-protection trigger, so each batch disables
+  it first (`SET LOCAL rules.deletion_protection TO off`): the GC is
+  the intended deleter, and the guard only exists so a stray
+  `DELETE FROM job` cannot drop a row without touching its object."
+  (:require
+   [app.common.logging :as l]
+   [app.common.schema :as sm]
+   [app.common.time :as ct]
+   [app.config :as cf]
+   [app.db :as db]
+   [app.jobs :as jobs]
+   [app.jobs.metrics :as metrics]
+   [integrant.core :as ig]))
+
+(def ^:private sql:touch-objects
+  "UPDATE storage_object
+      SET touched_at = ?
+    WHERE id = ANY(?::uuid[])")
+
+(def ^:private gc-batch-size 1000)
+
+(def ^:private sql:delete-expired-jobs
+  "DELETE FROM job
+     WHERE id IN (SELECT id
+                    FROM job
+                   WHERE expires_at < now()
+                     AND status NOT IN ('running', 'retry')
+                   LIMIT ?)
+   RETURNING resource_id")
+
+(def ^:private sql:delete-retained-jobs
+  "DELETE FROM job
+     WHERE id IN (SELECT id
+                    FROM job
+                   WHERE status IN ('completed', 'failed', 'cancelled', 'aborted')
+                     AND profile_id IS NULL
+                     AND modified_at < now() - ?::interval
+                   LIMIT ?)
+   RETURNING resource_id")
+
+(defn- touch-resources
+  [conn resource-ids]
+  (if (seq resource-ids)
+    (-> (db/exec-one! conn [sql:touch-objects (ct/now)
+                            (db/create-array conn "uuid" resource-ids)])
+        (db/get-update-count))
+    0))
+
+(defn- delete-jobs
+  [cfg kind sql & params]
+  (loop [deleted 0
+         touched 0]
+    (let [[rows touched-now]
+          (db/tx-run! cfg
+                      (fn [{:keys [::db/conn]}]
+                        (db/exec-one! conn ["SET LOCAL rules.deletion_protection TO off"])
+                        (let [rows         (db/exec! conn (conj (into [sql] params)
+                                                                gc-batch-size))
+                              resource-ids (into [] (keep :resource-id) rows)
+                              touched      (touch-resources conn resource-ids)]
+                          (db/after-commit
+                           #(do
+                              (metrics/record-gc-rows
+                               cfg kind :deleted (count rows))
+                              (metrics/record-gc-rows
+                               cfg kind :touched touched)))
+                          [rows touched])))
+          deleted' (+ deleted (count rows))
+          touched' (+ touched touched-now)]
+      (if (< (count rows) gc-batch-size)
+        [deleted' touched']
+        (do (jobs/heartbeat cfg)
+            (recur deleted' touched'))))))
+
+(declare execute-jobs-gc)
+
+(def schema:jobs-gc-params
+  "min-age: duration object, integer millis or duration string
+  in-process; integer millis or duration string over the job pipeline
+  (a duration object does not survive JSON encoding). The handler
+  always normalizes with ct/duration."
+  [:map
+   [:min-age {:optional true} [:or :int :string ::ct/duration]]])
+
+(defmethod ig/init-key ::job-def
+  [_ cfg]
+  {::jobs/name      :jobs-gc
+   ::jobs/schema    schema:jobs-gc-params
+   ::jobs/handler   (fn [_context params]
+                      (execute-jobs-gc cfg params))
+   ::jobs/decoder   (sm/decoder schema:jobs-gc-params sm/json-transformer)
+   ::jobs/validator (sm/validator schema:jobs-gc-params)})
+
+(defn execute-jobs-gc
+  "Plain job handler: delete expired rows (expires_at) and retained
+  internal terminal rows, marking the storage resources of the deleted
+  rows as touched.
+
+  Deletes run in bounded batches of 1000 rows, each batch in its own
+  transaction, with a heartbeat between batches so long sweeps neither
+  spike the WAL nor outrun the job lease."
+  [cfg params]
+  (let [min-age
+        (ct/duration (or (:min-age params)
+                         (cf/get-jobs-retention)))
+
+        expired-tpoint
+        (ct/tpoint)
+
+        [deleted-expired touched-expired]
+        (delete-jobs cfg :expired sql:delete-expired-jobs)
+
+        retained-tpoint
+        (ct/tpoint)
+
+        [deleted-retained touched-retained]
+        (delete-jobs cfg :retained sql:delete-retained-jobs
+                     (db/interval min-age))]
+
+    (metrics/record-gc-duration cfg :expired (inst-ms (expired-tpoint)))
+    (metrics/record-gc-duration cfg :retained (inst-ms (retained-tpoint)))
+    (l/dbg :hint "finished"
+           :deleted-expired deleted-expired
+           :touched-expired touched-expired
+           :deleted-retained deleted-retained
+           :touched-retained touched-retained)
+    {:deleted-expired    deleted-expired
+     :touched-expired    touched-expired
+     :deleted-retained   deleted-retained
+     :touched-retained   touched-retained}))
+

backend/src/app/jobs/metrics.clj

@@ -0,0 +1,288 @@
+;; This Source Code Form is subject to the terms of the Mozilla Public
+;; License, v. 2.0. If a copy of the MPL was not distributed with this
+;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
+;;
+;; Copyright (c) KALEIDOS SUBSIDIARY SL
+
+(ns app.jobs.metrics
+  "Metric names, labels and the jobs backlog sampler.
+
+  Labels are bounded by construction, not by allowlist: every value
+  recorded here comes from a fixed call site (a job name from the
+  registry, a queue from submit, an outcome from the lifecycle
+  writers), never from params, IDs or exception text. A new value
+  showing up in Prometheus means new code, which is exactly what you
+  want to see: folding it into some other bucket would only hide it.
+
+  This namespace is intentionally free of job handlers. Callers pass only
+  values already known at their boundary; IDs, props and exception text never
+  become metric labels."
+  (:require
+   [app.common.data :as d]
+   [app.common.logging :as l]
+   [app.common.schema :as sm]
+   [app.db :as db]
+   [app.metrics :as mtx]
+   [cuerdas.core :as str]
+   [integrant.core :as ig]
+   [promesa.exec :as px]))
+
+(set! *warn-on-reflection* true)
+
+(defn queue-label
+  "Coerce a queue to its label: lowercase, with \"other\" only when there
+  is no queue at all. A new queue shows up as-is: that means new code."
+  [queue]
+  (str/lower (mtx/label queue "other")))
+
+;; Job names go to the metric as-is: the registry is fixed (one job-def
+;; per name, wired in `app.main`), so cardinality is bounded by it.
+;; Anything without a name (nil) still falls back to "other".
+(defn- name-label [name]
+  (mtx/label name "other"))
+
+(defn- gc-kind-label [kind]
+  (str/lower (or (some-> kind d/name) "other")))
+
+(defn- outcome-label [outcome]
+  (str/lower (or (some-> outcome d/name) "failed")))
+
+(defn- retry-reason-label [reason]
+  (str/lower (or (some-> reason d/name) "backoff")))
+
+(defn- stage-label [stage]
+  (str/lower (or (some-> stage d/name) "execution")))
+
+(defn- record-value [metrics id labels value]
+  (mtx/run! metrics :id id :labels labels :val value))
+
+(defn- record-count [metrics id labels amount]
+  (mtx/run! metrics :id id :labels labels :inc amount))
+
+(defn record-submitted
+  [cfg name queue]
+  (record-count (mtx/instance cfg) :jobs-submitted
+                [(name-label name) (queue-label queue)]
+                1))
+
+(defn record-skipped
+  [cfg name queue]
+  (record-count (mtx/instance cfg) :jobs-skipped
+                [(name-label name) (queue-label queue)]
+                1))
+
+(defn record-dispatched
+  [cfg name queue amount]
+  (record-count (mtx/instance cfg) :jobs-dispatched
+                [(name-label name) (queue-label queue)]
+                amount))
+
+(defn record-retry
+  [cfg name queue reason]
+  (record-count (mtx/instance cfg) :jobs-retries
+                [(name-label name) (queue-label queue)
+                 (retry-reason-label reason)]
+                1))
+
+(defn record-event
+  "Total number of job events stored. No labels: the interesting split is
+  `kind`, and grouping a growing append-only table to build a metric is not
+  worth the query on the write path."
+  [cfg]
+  (record-count (mtx/instance cfg) :jobs-events-total [] 1))
+
+(defn record-orphan
+  [cfg queue]
+  (record-count (mtx/instance cfg) :jobs-orphaned
+                [(queue-label queue)]
+                1))
+
+(defn record-rescheduled
+  [cfg queue]
+  (record-count (mtx/instance cfg) :jobs-rescheduled
+                [(queue-label queue)]
+                1))
+
+(defn record-queue-wait
+  [cfg name queue millis]
+  (record-value (mtx/instance cfg) :jobs-queue-wait-timing
+                [(name-label name) (queue-label queue)]
+                (max 0 (long millis))))
+
+(defn record-execution
+  [cfg name queue millis]
+  (record-value (mtx/instance cfg) :jobs-execution-timing
+                [(name-label name) (queue-label queue)]
+                (max 0 (long millis))))
+
+(defn record-legacy-execution
+  "Histogram `penpot_tasks_timing`, from the pre-jobs `task` queue.
+
+  It only carries the job name label, with no queue to tell jobs of the
+  same name apart. Use `record-execution`, which adds the queue label,
+  for anything new. Kept exported only so the dashboards and the alerts
+  that already read it keep working: like every other jobs metric,
+  the label is the raw job name."
+  [cfg name millis]
+  (record-value (mtx/instance cfg) :tasks-timing
+                [(mtx/label name "other")]
+                (max 0 (long millis))))
+
+(defn record-total
+  [cfg name queue outcome millis]
+  (record-value (mtx/instance cfg) :jobs-total-timing
+                [(name-label name) (queue-label queue)
+                 (outcome-label outcome)]
+                (max 0 (long millis))))
+
+(defn record-dispatcher-batch
+  [cfg stage outcome millis]
+  (record-value (mtx/instance cfg) :jobs-dispatcher-timing
+                [(stage-label stage) (outcome-label outcome)]
+                (max 0 (long millis))))
+
+(defn record-dispatcher-size
+  [cfg queue size]
+  (record-value (mtx/instance cfg) :jobs-dispatcher-batch-size
+                [(queue-label queue)]
+                size))
+
+(defn record-gc-rows
+  [cfg kind action amount]
+  (record-count (mtx/instance cfg) :jobs-gc-rows
+                [(gc-kind-label kind)
+                 (str/lower (or (some-> action d/name) "deleted"))]
+                amount))
+
+(defn record-gc-duration
+  [cfg kind millis]
+  (record-value (mtx/instance cfg) :jobs-gc-timing
+                [(gc-kind-label kind)]
+                (max 0 (long millis))))
+
+(defn record-cron
+  [cfg outcome reason]
+  (record-count (mtx/instance cfg) :jobs-cron-total
+                [(str/lower (or (some-> outcome d/name) "error"))
+                 (str/lower (or (some-> reason d/name) "none"))]
+                1))
+
+(defn record-request
+  [cfg outcome millis]
+  (let [outcome (str/lower (or (some-> outcome d/name) "error"))
+        metrics (mtx/instance cfg)]
+    ;; No separate counter: the timing `_count` is the request total
+    ;; per outcome.
+    (record-value metrics :jobs-request-timing [outcome]
+                  (max 0 (long millis)))))
+
+(defn- offload-result-label
+  "Coerce an offload outcome to its label: lowercase, with `failed`
+  only when there is no outcome at all. Values come from the fixed
+  call sites of the offload handler (`ok`, `skip`, `error`), never
+  from params, IDs or exception text."
+  [result]
+  (str/lower (mtx/label result "failed")))
+
+(defn- offload-type-label
+  "Coerce a file data type to its label: lowercase, with `other` only
+  when there is no type at all. Values are the fixed `file_data.type`
+  values (`main`, `snapshot`, `fragment`)."
+  [type]
+  (str/lower (mtx/label type "other")))
+
+(defn record-offload-rows
+  "Count file data rows moved by the offload job, broken down by row
+  type. Only successful runs emit rows, so there is no result label."
+  [cfg type amount]
+  (record-count (mtx/instance cfg) :file-data-offload-rows
+                [(offload-type-label type)]
+                amount))
+
+(defn record-offload-bytes
+  "Count file data bytes moved by the offload job, broken down by row
+  type. Only successful runs emit bytes, so there is no result label."
+  [cfg type amount]
+  (record-count (mtx/instance cfg) :file-data-offload-bytes
+                [(offload-type-label type)]
+                amount))
+
+(defn record-offload-duration
+  "Time one offload run from lock to summary (milliseconds). Recorded
+  for every outcome (`ok`, `skip`, `error`), so its count is the run
+  total per outcome."
+  [cfg result millis]
+  (record-value (mtx/instance cfg) :file-data-offload-timing
+                [(offload-result-label result)]
+                (max 0 (long millis))))
+
+(def ^:private backlog-statuses
+  ["new" "scheduled" "running" "retry" "completed" "failed" "cancelled" "aborted"])
+
+(def ^:private sql:backlog
+  "SELECT status, count(*) AS n
+     FROM job
+    GROUP BY status")
+
+(def ^:private sql:oldest-pending
+  "SELECT coalesce(extract(epoch FROM (now() - min(created_at))), 0) AS age
+     FROM job
+    WHERE status IN ('new', 'scheduled', 'retry')")
+
+(defn sample-backlog
+  "Update the current job backlog gauges. The query is intentionally small
+  and runs periodically rather than from the worker hot path."
+  [{:keys [::db/pool] :as cfg}]
+  (let [rows    (db/exec! pool [sql:backlog])
+        by-id   (into {} (map (juxt :status #(or (:n %) 0))) rows)
+        age     (-> (db/exec-one! pool [sql:oldest-pending])
+                    :age
+                    (max 0)
+                    (double))
+        metrics (mtx/instance cfg)]
+    (doseq [status backlog-statuses]
+      (record-value metrics :jobs-backlog [status] (get by-id status 0)))
+    (record-value metrics :jobs-oldest-pending-age [] age)
+    age))
+
+(defn- sample-jobs-metrics [cfg]
+  (try
+    (sample-backlog cfg)
+    (catch Throwable cause
+      (l/warn :hint "unable to sample jobs metrics" :cause cause))))
+
+(def ^:private sample-interval-ms 30000)
+
+(defn create-sampler
+  "Create the daemon scheduler used by the jobs metrics component."
+  [cfg]
+  (let [scheduler (px/scheduled-executor
+                   :parallelism 1
+                   :factory (px/thread-factory :prefix "penpot/jobs-metrics/"
+                                               :daemon true))
+        sample    (fn sample []
+                    (try
+                      (sample-jobs-metrics cfg)
+                      (finally
+                        (px/schedule scheduler sample-interval-ms sample))))]
+    (px/schedule scheduler 0 sample)
+    scheduler))
+
+(def ^:private schema:sampler
+  [:map
+   ::db/pool
+   ::mtx/metrics])
+
+(defmethod ig/assert-key ::sampler
+  [_ cfg]
+  (assert (sm/check schema:sampler cfg)))
+
+(defmethod ig/init-key ::sampler
+  [_ {:keys [::db/pool] :as cfg}]
+  (if (db/read-only? pool)
+    (l/warn :hint "not started (db is read-only)")
+    (create-sampler cfg)))
+
+(defmethod ig/halt-key! ::sampler
+  [_ sampler]
+  (some-> sampler px/shutdown-now))

backend/src/app/jobs/storage.clj

@@ -0,0 +1,152 @@
+;; This Source Code Form is subject to the terms of the Mozilla Public
+;; License, v. 2.0. If a copy of the MPL was not distributed with this
+;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
+;;
+;; Copyright (c) KALEIDOS SUBSIDIARY SL
+
+(ns app.jobs.storage
+  "Storage helpers for the objects a job owns through `job.resource_id`.
+
+  A job resource lives in the `job-resource` bucket and belongs to the
+  profile that owns the job, so only that profile can read it back (the
+  asset route checks the owner, not just the session).
+
+  The job row is what keeps the object alive: `storage-gc-touched`
+  freezes it while a row references it and deletes it once no row does,
+  and the jobs GC marks it as touched when it removes the row. The object
+  carries no expiry of its own: `expired-at` is stored as `deleted_at`,
+  which makes the object unreadable from birth, so the retention of the
+  artifact is the retention of its job."
+  (:require
+   [app.common.exceptions :as ex]
+   [app.common.logging :as l]
+   [app.common.time :as ct]
+   [app.config :as cf]
+   [app.db :as db]
+   [app.storage :as sto]
+   [app.storage.tmp :as tmp]))
+
+(defn- resource-uri
+  "The url a user downloads the artifact from. A string, because the
+  descriptor ends up in the JSON result of the job."
+  [id]
+  (cf/get-public-uri "assets/by-id/" (str id)))
+
+(defn put-resource
+  "Store an artifact of a job in the `job-resource` bucket and return the
+  descriptor that identifies it.
+
+  Named options:
+
+  - `:content`  the artifact, an `app.storage/content` value built from a
+                temporary file or from bytes in memory
+  - `:filename` the name the user will see
+  - `:mtype`    the content type of the artifact
+
+  `profile-id` is the owner of the object: the profile that owns the job,
+  for both the package a job consumes and the artifact it produces.
+
+  The object is marked as touched right away, so an artifact that no job
+  row ever references (a crash between the upload and the completion) is
+  reclaimed by the storage GC instead of leaking; while a row references
+  it, `storage-gc-touched` freezes it."
+  [cfg profile-id {:keys [content filename mtype]}]
+  (assert (uuid? profile-id) "expected the profile that owns the resource")
+  (assert (string? mtype) "expected the content type of the resource")
+  (let [storage (sto/resolve cfg)
+        object  (sto/put-object! storage
+                                 {::sto/content    content
+                                  ::sto/touched-at (ct/now)
+                                  :bucket          sto/job-resource-bucket
+                                  :profile-id      profile-id
+                                  :content-type    mtype})]
+    {:resource-id  (:id object)
+     :resource-uri (resource-uri (:id object))
+     :filename     filename
+     :mtype        mtype
+     :size         (:size object)}))
+
+(defn- check-input
+  "The object a job consumes must exist, be a job resource and belong to
+  the profile that owns the job. Raises `:not-found` otherwise: a handler
+  never reads another profile's package."
+  [context object]
+  (when (or (nil? object)
+            (not= sto/job-resource-bucket (:bucket (meta object)))
+            (not= (:profile-id context) (:profile-id (meta object))))
+    (ex/raise :type :not-found
+              :code :job-resource-not-found
+              :hint "the input of the job is not available"
+              :job-id (:id context)
+              :resource-id (:resource-id context)))
+  object)
+
+(defn load-input
+  "Download the package a job consumes to a local temporary file and
+  return its path, ready for the binfile reader.
+
+  An object that was released or expired is already invisible to
+  `sto/get-object`, so it lands in the same `:not-found` as a missing
+  one."
+  [cfg context]
+  (let [storage     (sto/resolve cfg)
+        resource-id (:resource-id context)
+        object      (check-input context
+                                 (when (uuid? resource-id)
+                                   (sto/get-object storage resource-id)))]
+    (if-let [input (sto/get-object-data storage object)]
+      (tmp/tempfile-from input)
+      (ex/raise :type :not-found
+                :code :job-resource-not-found
+                :hint "the input of the job is not readable"
+                :job-id (:id context)
+                :resource-id resource-id))))
+
+(defn release-resource
+  "Release a job resource.
+
+  The object is marked as deleted and, when it belongs to a job, the
+  row stops pointing at it in the same transaction: a row never
+  references an object that is already gone, and the foreign key is not
+  what ends up clearing the reference. A resource no job owns (the
+  package of a creation that never submitted) is released with a nil
+  job id, and then no row is touched at all.
+
+  Best effort: a failure is logged and never changes the terminal state
+  of the job, and releasing without a resource is a no-op. Returns true
+  when the object was marked as deleted or a stale reference was
+  cleared."
+  ([cfg resource-id]
+   (release-resource cfg resource-id nil))
+  ([cfg resource-id job-id]
+   (when (uuid? resource-id)
+     (try
+       (db/tx-run!
+        cfg
+        (fn [tx-cfg]
+          (let [storage  (sto/resolve tx-cfg ::db/reuse-conn true)
+                deleted? (sto/del-object! storage resource-id)
+                ;; the job no longer owns the package: the pointer goes with
+                ;; it, so clearing it does not depend on the object being
+                ;; physically deleted later
+                cleared? (when (some? job-id)
+                           (db/update! tx-cfg :job
+                                       {:resource-id nil :modified-at (ct/now)}
+                                       ["id = ? AND resource_id = ?"
+                                        job-id resource-id]
+                                       {::db/return-keys [:id]}))]
+            (or deleted? (some? cleared?)))))
+       (catch Throwable cause
+         (l/wrn :hint "unable to release the input of a job"
+                :job-id (str job-id)
+                :resource-id (str resource-id)
+                :cause cause)
+         false)))))
+
+(defn release-input
+  "Release the package a job consumed.
+
+  Delegates to `release-resource` with the id of the job: the object
+  and the pointer are dropped together. Best effort, like there."
+  [cfg context]
+  (release-resource cfg (:resource-id context) (:id context)))

backend/src/app/loggers/audit.clj

@@ -13,19 +13,21 @@
    [app.common.logging :as l]
    [app.common.schema :as sm]
    [app.common.time :as ct]
+   [app.common.transit :as t]
    [app.common.uuid :as uuid]
    [app.config :as cf]
    [app.db :as db]
    [app.email :as email]
    [app.http :as-alias http]
    [app.http.access-token :as-alias actoken]
+   [app.jobs :as jobs]
+   [app.loggers.audit.consumer-task :as consumer]
    [app.loggers.audit.tasks :as-alias tasks]
    [app.loggers.webhooks :as-alias webhooks]
    [app.rpc :as-alias rpc]
    [app.setup :as-alias setup]
    [app.util.inet :as inet]
    [app.util.services :as-alias sv]
-   [app.worker :as wrk]
    [cuerdas.core :as str]
    [yetti.request :as yreq]))
 
@@ -46,6 +48,46 @@
     "incomplete-user-info"
     "unexpected-error"})
 
+(def ^:private subscription-creation-events
+  #{"create-subscription" "create-trial-subscription"})
+
+(def ^:private subscription-creation-telemetry-props
+  #{:type
+    :deployment
+    :quantity
+    :subscription-mode})
+
+(def ^:private subscription-update-telemetry-props
+  #{:type
+    :deployment
+    :previous-status
+    :status
+    :previous-cancel-at-period-end
+    :cancel-at-period-end
+    :previous-cancel-at
+    :cancel-at
+    :changed-fields})
+
+(def ^:private subscription-summary-telemetry-props
+  #{:type
+    :deployment
+    :instance-id
+    :snapshot-date
+    :trialing-count
+    :active-count
+    :active-cancel-scheduled-count
+    :past-due-count
+    :canceled-count
+    :unpaid-count
+    :incomplete-count
+    :incomplete-expired-count
+    :paused-count
+    :other-count
+    :total-count
+    :sync-succeeded
+    :sync-failed
+    :snapshot-complete})
+
 (def ^:private safe-backend-context-keys
   #{:version
     :initiator
@@ -240,8 +282,13 @@
                    (update :props db/tjson)
                    (update :context db/tjson)
                    (update :ip-addr db/inet))
-        params (select-keys params event-keys)]
-    (db/insert! cfg :audit-log params)))
+        params (select-keys params event-keys)
+        row    (db/insert! cfg :audit-log params)]
+    ;; The consumer queue follows the row in the same transaction: one
+    ;; pending per enabled consumer. A rollback takes both on its way
+    ;; out, so a row never outlives its queue.
+    (consumer/queue-pendings cfg [(:id row)])
+    row))
 
 (def ^:private xf:filter-telemetry-props
   "Transducer that keeps only map entries whose values are UUIDs,
@@ -305,19 +352,24 @@
                             :else               label)
             dedupe?       (boolean (and batch-key batch-timeout))]
 
-        (wrk/submit! (-> cfg
-                         (assoc ::wrk/task :process-webhook-event)
-                         (assoc ::wrk/queue :webhooks)
-                         (assoc ::wrk/max-retries 0)
-                         (assoc ::wrk/delay (or batch-timeout 0))
-                         (assoc ::wrk/dedupe dedupe?)
-                         (assoc ::wrk/label label)
-                         (assoc ::wrk/params (-> event
-                                                 (d/without-qualified)
-                                                 (dissoc :source)
-                                                 (dissoc :context)
-                                                 (dissoc :ip-addr)
-                                                 (dissoc :type)))))))
+        (jobs/submit cfg
+                     {::jobs/name :process-webhook-event
+                      ::jobs/queue :webhooks
+                      ::jobs/max-retries 0
+                      ::jobs/delay (or batch-timeout 0)
+                      ::jobs/dedupe dedupe?
+                      ::jobs/label label
+                      ;; The event travels as an opaque transit blob:
+                      ;; transit preserves the instant/UUID/set types
+                      ;; that plain JSON props cannot carry.
+                      ::jobs/params {:event-blob
+                                     (t/encode-str
+                                      (-> event
+                                          (d/without-qualified)
+                                          (dissoc :source)
+                                          (dissoc :context)
+                                          (dissoc :ip-addr)
+                                          (dissoc :type)))}})))
     event))
 
 (defn submit*
@@ -336,6 +388,24 @@
 (defn filter-telemetry-props
   [{:keys [source name props type] :as params}]
   (cond
+    (and (= type "action")
+         (contains? subscription-creation-events name)
+         (or (= source "backend")
+             (= source "frontend")))
+    (let [props' (into {} xf:filter-telemetry-props props)
+          props' (merge props' (select-keys props subscription-creation-telemetry-props))]
+      (assoc params :props props'))
+
+    (and (= source "backend")
+         (= type "action")
+         (= name "update-subscription"))
+    (assoc params :props (select-keys props subscription-update-telemetry-props))
+
+    (and (= source "backend")
+         (= type "action")
+         (= name "subscription-summary"))
+    (assoc params :props (select-keys props subscription-summary-telemetry-props))
+
     (and (= source "backend")
          (= name "organization-sso-auth-failed"))
     (let [props' (into {} xf:filter-telemetry-props props)
@@ -578,6 +648,8 @@
   logic."
   [cfg event]
   (when (contains? cf/flags :audit-log)
+    ;; Transactional (not a bare connection): the row and its
+    ;; pendings commit or vanish together (see append-audit-entry).
     (let [tnow  (ct/now)
           event (-> event
                     (assoc :created-at tnow)
@@ -588,4 +660,4 @@
                     (update :source d/nilv "backend")
                     (select-keys event-keys)
                     (check-event))]
-      (db/run! cfg append-audit-entry event))))
+      (db/tx-run! cfg append-audit-entry event))))

backend/src/app/loggers/audit/archive_task.clj

@@ -1,141 +0,0 @@
-;; This Source Code Form is subject to the terms of the Mozilla Public
-;; License, v. 2.0. If a copy of the MPL was not distributed with this
-;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
-;;
-;; Copyright (c) KALEIDOS SUBSIDIARY SL
-
-(ns app.loggers.audit.archive-task
-  (:require
-   [app.common.exceptions :as ex]
-   [app.common.logging :as l]
-   [app.common.schema :as sm]
-   [app.common.transit :as t]
-   [app.config :as cf]
-   [app.db :as db]
-   [app.http.client :as http]
-   [app.setup :as-alias setup]
-   [integrant.core :as ig]
-   [promesa.exec :as px]))
-
-;; This is a task responsible to send the accumulated events to
-;; external service for archival.
-
-(defn- decode-row
-  [{:keys [props ip-addr context] :as row}]
-  (cond-> row
-    (db/pgobject? props)
-    (assoc :props (db/decode-transit-pgobject props))
-
-    (db/pgobject? context)
-    (assoc :context (db/decode-transit-pgobject context))
-
-    (db/pgobject? ip-addr "inet")
-    (assoc :ip-addr (db/decode-inet ip-addr))))
-
-(def ^:private event-keys
-  [:type
-   :name
-   :source
-   :created-at
-   :tracked-at
-   :profile-id
-   :ip-addr
-   :props
-   :context])
-
-(defn- row->event
-  [row]
-  (select-keys row event-keys))
-
-(defn- send!
-  [{:keys [::uri] :as cfg} events]
-  (let [skey    (-> cfg ::setup/shared-keys :nexus)
-        body    (t/encode {:events events})
-        headers {"content-type" "application/transit+json"
-                 "origin" (str (cf/get :public-uri))
-                 "x-shared-key" (str "nexus " skey)}
-        params  {:uri uri
-                 :timeout 12000
-                 :method :post
-                 :headers headers
-                 :body body}
-        resp    (http/req cfg params {:skip-ssrf-check? true})]
-
-    (if (= (:status resp) 204)
-      true
-      (do
-        (l/error :hint "unable to archive events"
-                 :resp-status (:status resp)
-                 :resp-body (:body resp))
-        false))))
-
-(defn- mark-archived!
-  [{:keys [::db/conn]} rows]
-  (let [ids (db/create-array conn "uuid" (map :id rows))]
-    (db/exec-one! conn ["update audit_log set archived_at=now() where id = ANY(?)" ids])))
-
-(def ^:private xf:create-event
-  (comp (map decode-row)
-        (map row->event)))
-
-(def ^:private sql:get-audit-log-chunk
-  "SELECT *
-     FROM audit_log
-    WHERE archived_at IS NULL
-    ORDER BY created_at ASC
-    LIMIT 128
-      FOR UPDATE
-     SKIP LOCKED")
-
-(defn- get-event-rows
-  [{:keys [::db/conn] :as cfg}]
-  (->> (db/exec! conn [sql:get-audit-log-chunk])
-       (not-empty)))
-
-(defn- archive-events!
-  [{:keys [::uri] :as cfg}]
-  (db/tx-run! cfg (fn [cfg]
-                    (when-let [rows (get-event-rows cfg)]
-                      (let [events (into [] xf:create-event rows)]
-                        (l/trc :hint "archive events chunk" :uri uri :events (count events))
-                        (when (send! cfg events)
-                          (mark-archived! cfg rows)
-                          (count events)))))))
-
-(def ^:private schema:handler-params
-  [:map
-   ::db/pool
-   ::setup/shared-keys
-   ::http/client])
-
-(defmethod ig/assert-key ::handler
-  [_ params]
-  (assert (sm/valid? schema:handler-params params) "valid params expected for handler"))
-
-(defmethod ig/init-key ::handler
-  [_ cfg]
-  (fn [params]
-    ;; NOTE: this let allows overwrite default configured values from
-    ;; the repl, when manually invoking the task.
-    (let [enabled (or (contains? cf/flags :audit-log-archive)
-                      (:enabled params false))
-
-          uri     (cf/get :audit-log-archive-uri)
-          uri     (or uri (:uri params))
-          cfg     (assoc cfg ::uri uri)]
-
-      (when (and enabled (not uri))
-        (ex/raise :type :internal
-                  :code :task-not-configured
-                  :hint "archive task not configured, missing uri"))
-
-      (when enabled
-        (loop [total 0]
-          (if-let [n (archive-events! cfg)]
-            (do
-              (px/sleep 100)
-              (recur (+ total ^long n)))
-
-            (when (pos? total)
-              (l/dbg :hint "events archived" :total total))))))))
-

backend/src/app/loggers/audit/consumer_task.clj

@@ -0,0 +1,688 @@
+;; This Source Code Form is subject to the terms of the Mozilla Public
+;; License, v. 2.0. If a copy of the MPL was not distributed with this
+;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
+;;
+;; Copyright (c) KALEIDOS SUBSIDIARY SL
+
+(ns app.loggers.audit.consumer-task
+  (:require
+   [app.common.exceptions :as ex]
+   [app.common.logging :as l]
+   [app.common.schema :as sm]
+   [app.common.time :as ct]
+   [app.common.transit :as t]
+   [app.common.uri :as u]
+   [app.common.uuid :as uuid]
+   [app.config :as cf]
+   [app.db :as db]
+   [app.db.sql :as sql]
+   [app.http.client :as http]
+   [app.jobs :as jobs]
+   [app.metrics :as mtx]
+   [app.nitrate :as nitrate]
+   [app.setup :as-alias setup]
+   [integrant.core :as ig]
+   [promesa.exec :as px]))
+
+;; This task drains the audit log buffer into the long-term archives.
+;;
+;; Two separate jobs share this namespace and the queue helpers, one
+;; per destination: `:audit-log-consumer-nexus` and
+;; `:audit-log-consumer-admin-console`. Each job drains only its own queue
+;; and deletes its pendings on completion; whoever deletes the last
+;; pending also deletes the record ("last one out turns off the
+;; light"), so no separate GC task is needed. Housekeeping (reaping
+;; disabled pendings, sweeping orphans) lives in both jobs on purpose:
+;; only the enabled job runs, so it must clean up for the rest.
+
+;; Consumer ids are fixed constants, never reuse one: retired
+;; pendings are reaped by id, so a reused id would inherit the
+;; previous owner's queue.
+(def consumer-nexus 1)
+(def consumer-admin-console 2)
+
+(def ^:private consumers
+  [{:id consumer-nexus          :flag :nexus         :label "nexus"}
+   {:id consumer-admin-console :flag :admin-console :label "admin-console"}])
+
+(def ^:private xf:enabled-consumers
+  "Static transducer: keep the consumers whose send flag is on.
+  Allocated once so hot paths (every audited RPC) do not rebuild
+  it per call; the flag lookup itself still reads the live flags."
+  (comp (filter #(contains? cf/flags (:flag %)))
+        (map :id)))
+
+(defn enabled-consumers
+  "Consumers allowed to work right now, from the send flags. Public:
+  producers queue every record for exactly these consumers."
+  []
+  (into [] xf:enabled-consumers consumers))
+
+(defn queue-pendings
+  "Queue records for every enabled consumer, nothing when there is
+  nothing to queue or no consumer is on. One multi-row statement for
+  all ids instead of one per consumer: this runs inside every audited
+  RPC and the bulk frontend ingest. Runs inside the row transaction:
+  a rollback takes both."
+  [cfg ids]
+  (let [consumers (enabled-consumers)]
+    (when (and (seq ids) (seq consumers))
+      (db/insert-many! cfg :audit-log-pending
+                       [:audit-log-id :consumer]
+                       (for [id ids
+                             consumer consumers]
+                         [id consumer])
+                       {::sql/on-conflict-do-nothing true}))))
+
+(def ^:private consumer-labels
+  (into {} (map (juxt :id :label)) consumers))
+
+(defn- consumer-label
+  [consumer]
+  (get consumer-labels consumer))
+
+(defn- decode-row
+  [{:keys [props ip-addr context] :as row}]
+  (cond-> row
+    (db/pgobject? props)
+    (assoc :props (db/decode-transit-pgobject props))
+
+    (db/pgobject? context)
+    (assoc :context (db/decode-transit-pgobject context))
+
+    (db/pgobject? ip-addr "inet")
+    (assoc :ip-addr (db/decode-inet ip-addr))))
+
+(def ^:private event-keys
+  [:type
+   :name
+   :source
+   :created-at
+   :tracked-at
+   :profile-id
+   :ip-addr
+   :props
+   :context])
+
+(defn- row->event
+  [row]
+  (select-keys row event-keys))
+
+(defn- record-count
+  [metrics id labels amount]
+  (mtx/run! metrics :id id :labels labels :inc amount))
+
+(defn- truncate-body
+  [body limit]
+  (when (some? body)
+    (let [s (str body)]
+      (if (> (count s) limit)
+        (subs s 0 limit)
+        s))))
+
+;; NOTE: deliberately no SKIP LOCKED here. Batches must reach the
+;; destinations in chronological order (Nexus numbers by arrival),
+;; so every consumer takes the same total order and a second
+;; overlapping run waits for the first to commit, then sees whether
+;; it was last. The brief blocking is accepted; skipping rows could
+;; reorder archive arrival.
+(def ^:private sql:pending-chunk
+  "SELECT a.*
+     FROM audit_log_pending p
+     JOIN audit_log a ON a.id = p.audit_log_id
+    WHERE p.consumer = ?::smallint
+    ORDER BY a.created_at ASC, a.id ASC
+    LIMIT 128
+      FOR UPDATE OF a")
+
+(def ^:private sql:delete-pendings
+  "DELETE FROM audit_log_pending
+    WHERE consumer = ?::smallint
+      AND audit_log_id = ANY(?)")
+
+(def ^:private sql:delete-drained
+  "DELETE FROM audit_log a
+    WHERE a.id = ANY(?)
+      AND NOT EXISTS (
+        SELECT 1 FROM audit_log_pending p
+        WHERE p.audit_log_id = a.id
+      )")
+
+(defn- get-pending-rows
+  [{:keys [::db/conn] :as cfg} consumer]
+  (->> (db/exec! conn [sql:pending-chunk consumer])
+       (not-empty)))
+
+(defn- complete-batch
+  "Delete our pendings for the batch, then the records nobody is
+  waiting for anymore. Two statements on purpose: inside a single
+  statement the NOT EXISTS would still see the pending just deleted
+  and never delete a record."
+  [{:keys [::db/conn]} consumer rows]
+  (let [ids (db/create-array conn "uuid" (map :id rows))]
+    (db/exec! conn [sql:delete-pendings consumer ids])
+    (db/exec! conn [sql:delete-drained ids])))
+
+(defn- generate-nexus-uri
+  "Joins relative path segments to the Nexus backend URI, same shape
+  as the admin-console helper in `app.nitrate` (there the ingest path
+  is `api/audit-log`; here it is `api/audit`).
+  Segments must not start with `/`. Returns nil when no base URI is
+  configured, so the caller keeps the `:task-not-configured` guard."
+  [& segments]
+  (when-let [base (cf/get :nexus-uri)]
+    (apply cf/join-uri base segments)))
+
+(defn- send-to-nexus
+  "POST events to Nexus. Returns the number of events sent; throws on
+  transport failure or non-204 refusal, carrying the attempted count
+  as :events in the error data so the caller records the failure.
+  No metrics or logs here: the runner reports the aborted job."
+  [{:keys [::uri] :as cfg} events]
+  (let [n      (count events)
+        skey   (-> cfg ::setup/shared-keys :nexus)
+        body   (t/encode {:events events})
+        params {:uri uri
+                :timeout 12000
+                :method :post
+                :headers {"content-type" "application/transit+json"
+                          "origin" (str (cf/get :public-uri))
+                          "x-shared-key" (str "nexus " skey)}
+                :body body}
+        resp   (try
+                 (http/req cfg params {:skip-ssrf-check? true})
+                 (catch Throwable cause
+                   (throw (ex/raise :type :internal
+                                    :code :nexus-send-failed
+                                    :hint "unable to send events to nexus"
+                                    :cause cause
+                                    :events n))))]
+    (if (= (:status resp) 204)
+      n
+      (throw (ex/raise :type :internal
+                       :code :consumer-chunk-refused
+                       :hint "nexus refused archive chunk"
+                       :resp-status (:status resp)
+                       :resp-body (truncate-body (:body resp) 500)
+                       :events n)))))
+
+(def ^:private event-names-for-admin-console
+  #{"accept-organization-invitation"
+    "accept-organization-invitation-from"
+    "accept-team-invitation"
+    "accept-team-invitation-from"
+    "add-member-to-organization"
+    "add-team-to-organization"
+    "assign-organization-attribute"
+    "auditlog-exported"
+    "auditlog-guide-downloaded"
+    "cancel-organization-invitation"
+    "create-export-binfile-job"
+    "create-file"
+    "create-organization"
+    "create-organization-attribute"
+    "create-organization-invitation"
+    "create-project"
+    "create-team"
+    "create-team-access-request"
+    "create-team-invitation"
+    "create-team-invitations"
+    "create-webhook"
+    "delete-file"
+    "delete-font"
+    "delete-organization"
+    "delete-organization-member"
+    "delete-project"
+    "delete-team"
+    "delete-team-invitation"
+    "delete-team-member"
+    "leave-team"
+    "move-project"
+    "move-team-to-organization"
+    "organization-sso-auth-failed"
+    "organization-sso-auth-started"
+    "organization-sso-auth-succeeded"
+    "organization-sso-token-issued"
+    "organization-sso-token-revoked"
+    "permanently-delete-team-files"
+    "remove-organization-team"
+    "remove-team-from-organization"
+    "rename-organization"
+    "rename-project"
+    "restore-deleted-team-files"
+    "sso-config-deleted"
+    "sso-config-tested"
+    "success-activate-organization-sso"
+    "success-apply-organization-sso-changes"
+    "unassign-organization-attribute"
+    "update-organization-attribute"
+    "update-organization-invitation"
+    "update-organization-permissions"
+    "update-team-invitation"
+    "update-team-invitation-role"
+    "update-team-member-role"
+    "update-team-photo"
+    "verify-token"})
+
+;; Props id key -> email prop filled from profile.email (admin-console path only).
+;; Some producers store ids as UUID strings (e.g. :user-who-send-invitation).
+(def ^:private id->email-prop
+  {:profile-id                 :profile-email
+   :user-id                    :user-email
+   :member-id                  :member-email
+   :user-who-send-invitation   :user-who-send-invitation-email})
+
+(defn- collect-team-ids-needing-name
+  [events]
+  (into #{}
+        (comp (map :props)
+              (keep (fn [props]
+                      (when-let [team-id (and (not (contains? props :team-name))
+                                              (uuid/coerce (:team-id props)))]
+                        team-id))))
+        events))
+
+(defn- collect-profile-ids-needing-email
+  [events]
+  (into #{}
+        (mapcat (fn [{:keys [props]}]
+                  (keep (fn [[id-key email-key]]
+                          (when-not (contains? props email-key)
+                            (uuid/coerce (get props id-key))))
+                        id->email-prop)))
+        events))
+
+(defn- collect-actor-ids-needing-profile
+  "Actor ids (event :profile-id) that still need top-level :profile-email
+  and/or :profile-name. Skips uuid/zero and non-uuids."
+  [events]
+  (into #{}
+        (keep (fn [event]
+                (when (or (not (contains? event :profile-email))
+                          (not (contains? event :profile-name)))
+                  (when-let [id (uuid/coerce (:profile-id event))]
+                    (when-not (uuid/zero? id)
+                      id)))))
+        events))
+
+(defn- load-team-names
+  [conn ids]
+  (if (seq ids)
+    (let [arr  (db/create-array conn "uuid" ids)
+          rows (db/exec! conn ["SELECT id, name FROM team WHERE id = ANY(?)" arr])]
+      (into {} (map (juxt :id :name) rows)))
+    {}))
+
+(defn- load-profiles
+  "Batch-load profile email and fullname (including soft-deleted)."
+  [conn ids]
+  (if (seq ids)
+    (let [arr  (db/create-array conn "uuid" ids)
+          rows (db/exec! conn ["SELECT id, email, fullname FROM profile WHERE id = ANY(?)" arr])]
+      (into {} (map (juxt :id identity) rows)))
+    {}))
+
+(defn- maybe-assoc-team-name
+  [event team-names]
+  (let [props (:props event)]
+    (if-let [team-name (and (not (contains? props :team-name))
+                            (when-let [team-id (uuid/coerce (:team-id props))]
+                              (get team-names team-id)))]
+      (assoc-in event [:props :team-name] team-name)
+      event)))
+
+(defn- maybe-assoc-emails
+  [event profiles]
+  (update event :props
+          (fn [props]
+            (reduce-kv (fn [props id-key email-key]
+                         (if-let [email (and (not (contains? props email-key))
+                                             (when-let [id (uuid/coerce (get props id-key))]
+                                               (:email (get profiles id))))]
+                           (assoc props email-key email)
+                           props))
+                       props
+                       id->email-prop))))
+
+(defn- maybe-assoc-actor-profile
+  "Fill-only top-level :profile-email / :profile-name from the event
+  actor (:profile-id). Omits keys when the id is missing, zero, the
+  profile row is absent, or the corresponding column is empty
+  (fullname defaults to \"\" in the schema)."
+  [event profiles]
+  (let [id (uuid/coerce (:profile-id event))]
+    (if (or (nil? id) (uuid/zero? id))
+      event
+      (if-let [profile (get profiles id)]
+        (cond-> event
+          (and (not (contains? event :profile-email))
+               (not-empty (:email profile)))
+          (assoc :profile-email (:email profile))
+
+          (and (not (contains? event :profile-name))
+               (not-empty (:fullname profile)))
+          (assoc :profile-name (:fullname profile)))
+        event))))
+
+(defn- enrich-events-for-admin-console
+  "Enrich allowlisted events before the admin-console ingest:
+  1. Fill missing :team-name from team id (including soft-deleted).
+  2. Fill missing emails from matching id keys in props (including
+     soft-deleted profiles): :profile-email, :user-email, :member-email,
+     :user-who-send-invitation-email.
+  3. Fill missing top-level :profile-email / :profile-name from the
+     event actor (:profile-id), including soft-deleted profiles; omit
+     when there is no usable actor id or the value is empty."
+  [{:keys [::db/conn]} events]
+  (let [team-ids    (collect-team-ids-needing-name events)
+        profile-ids (into (collect-profile-ids-needing-email events)
+                          (collect-actor-ids-needing-profile events))
+        team-names  (load-team-names conn team-ids)
+        profiles    (load-profiles conn profile-ids)]
+    (mapv (fn [event]
+            (-> event
+                (maybe-assoc-team-name team-names)
+                (maybe-assoc-emails profiles)
+                (maybe-assoc-actor-profile profiles)))
+          events)))
+
+(defn- send-to-admin-console
+  "Ingest rows into the admin console. Returns {:sent :skipped}
+  (filtered-out rows count as processed for the queue); throws
+  carrying the attempted count as :events, like send-to-nexus. Enrich
+  failures propagate untouched: a local DB/decode failure is ours, not
+  the destination's, and must not count as a send failure."
+  [cfg rows]
+  (let [events   (filterv #(contains? event-names-for-admin-console (:name %)) rows)
+        total    (count rows)
+        skipped  (- total (count events))
+        uri-host (some-> (cf/get :admin-console-uri) u/uri u/get-domain)]
+    (l/dbg :hint "consumer events chunk ready"
+           :consumer "admin-console"
+           :uri-host uri-host
+           :total total
+           :skipped skipped)
+    (if (empty? events)
+      {:sent 0 :skipped skipped}
+      ;; Enrich first, outside the try: a local DB/decode failure is
+      ;; ours, not the destination's, and must not count as
+      ;; admin-console/failed.
+      (let [enriched (enrich-events-for-admin-console cfg events)
+            n        (count events)]
+        (try
+          (nitrate/call cfg :ingest-audit-log {:events enriched})
+          (l/dbg :hint "consumer events chunk sent"
+                 :consumer "admin-console"
+                 :uri-host uri-host
+                 :sent n
+                 :skipped skipped)
+          {:sent n :skipped skipped}
+          (catch Throwable cause
+            (l/dbg :hint "consumer events chunk failed"
+                   :consumer "admin-console"
+                   :uri-host uri-host
+                   :events n
+                   :code :admin-console-send-failed)
+            (throw (ex/raise :type :internal
+                             :code :admin-console-send-failed
+                             :hint "unable to archive events to the admin console"
+                             :cause cause
+                             :events n))))))))
+
+(defn- consumer-nexus-chunk
+  "Send one pending chunk to Nexus and complete it. Returns {:sent n}
+  or nil when the queue is empty; throws on refusal. Takes a
+  transactional cfg: the caller opens one transaction per chunk. No
+  metrics here: a normal return means the transaction committed, so
+  the caller records the success."
+  [cfg]
+  (db/exec! cfg ["SET LOCAL lock_timeout = '30s'"])
+  (when-let [rows (get-pending-rows cfg consumer-nexus)]
+    (let [events (mapv row->event (mapv decode-row rows))
+          ;; send-to-nexus returns the count or throws: a refusal
+          ;; aborts the job, the rows stay for the next tick.
+          n      (send-to-nexus cfg events)]
+
+      (l/trc :hint "consumer events chunk"
+             :consumer "nexus"
+             :events n)
+
+      (complete-batch cfg consumer-nexus rows)
+      {:sent n})))
+
+(defn- consumer-admin-console-chunk
+  "Send one pending chunk to the admin console and complete it. Returns
+  {:sent :skipped} or nil when the queue is empty; throws on failure.
+  Takes a transactional cfg, no metrics here: same contract as
+  consumer-nexus-chunk."
+  [cfg]
+  (db/exec! cfg ["SET LOCAL lock_timeout = '30s'"])
+  (when-let [rows (get-pending-rows cfg consumer-admin-console)]
+    (let [decoded     (mapv decode-row rows)
+          _           (l/trc :hint "consumer events chunk"
+                             :consumer "admin-console"
+                             :events (count decoded))
+          admin-console-res (send-to-admin-console cfg decoded)]
+      ;; Filtered-out rows count as processed for the
+      ;; admin-console queue: their pending is done, the
+      ;; record stays until every consumer is done.
+      (complete-batch cfg consumer-admin-console rows)
+      admin-console-res)))
+
+(defn- drain-consumer
+  "Drain one consumer queue chunk by chunk, one transaction per chunk.
+  Returns totals; a chunk the destination refuses (or that throws)
+  aborts the drain with a throw, and the committed chunks so far
+  stay sent."
+  [cfg send-chunk]
+  (loop [sent    0
+         skipped 0
+         chunks  0]
+    ;; One transaction per chunk: fetch, send and complete commit (or
+    ;; roll back) together, and the next chunk sees a fresh snapshot.
+    (if-let [res (db/tx-run! cfg send-chunk)]
+      (do
+        (jobs/heartbeat cfg)
+        (px/sleep 100)
+        (recur (+ sent ^long (:sent res))
+               (+ skipped ^long (:skipped res 0))
+               (inc chunks)))
+      {:sent sent :skipped skipped :chunks chunks})))
+
+(defn- sql:reap-pendings
+  [consumer]
+  (str "DELETE FROM audit_log_pending"
+       " WHERE consumer = " consumer
+       "   AND audit_log_id IN (SELECT audit_log_id FROM audit_log_pending"
+       " WHERE consumer = " consumer
+       " ORDER BY audit_log_id LIMIT 5000)"))
+
+(def ^:private sql:sweep-orphans
+  "DELETE FROM audit_log a
+    WHERE a.id IN (SELECT a2.id FROM audit_log a2
+                    WHERE NOT EXISTS (SELECT 1 FROM audit_log_pending p
+                                       WHERE p.audit_log_id = a2.id)
+                    ORDER BY a2.id LIMIT 5000)")
+
+(defn- delete-batched
+  "Run a batched DELETE until it removes nothing. Returns the total."
+  [cfg sql]
+  (loop [total 0]
+    (let [n (db/get-update-count (db/exec-one! cfg [sql]))]
+      (if (pos? n)
+        (do
+          (px/sleep 100)
+          (recur (+ total ^long n)))
+        total))))
+
+(defn- reap-disabled
+  "Drop the pendings of consumers that are off: a disabled destination
+  will never drain them, and without this they would pin their records
+  forever. This also absorbs any later deactivation, with no manual step.
+  Dropped pendings count as discarded: their rows may still be archived
+  by an enabled consumer, or swept unsent when nothing waits for them."
+  [cfg]
+  (let [total (->> consumers
+                   (remove #(contains? cf/flags (:flag %)))
+                   (map #(delete-batched cfg (sql:reap-pendings (:id %))))
+                   (reduce + 0))]
+    (when (pos? total)
+      (record-count (mtx/instance cfg) :audit-consumer-discarded ["reaped"] total))
+    total))
+
+(defn- sweep-orphans
+  "Delete records nobody waits for anymore: rows born while no
+  consumer was on, rows left pending-less by the reap, and leftovers
+  archived by the old task. Batched and normally a no-op. Deleted rows
+  count as discarded: on a live system they were never archived."
+  [cfg]
+  (let [total (delete-batched cfg sql:sweep-orphans)]
+    (when (pos? total)
+      (record-count (mtx/instance cfg) :audit-consumer-discarded ["swept"] total))
+    total))
+
+(declare execute-audit-log-consumer-nexus)
+(declare execute-audit-log-consumer-admin-console)
+
+(def schema:audit-log-consumer-nexus-params
+  "Overrides for the nexus job repl invocation defaults.
+  Cron submits {}; system deps (pool, shared-keys, http) live on the
+  job-def cfg closed over by the handler, not in job params.
+  Closed on purpose: a stale key (like the old :enabled) must fail
+  validation loudly instead of skipping silently."
+  [:map {:closed true}
+   [:uri {:optional true} ::sm/uri]])
+
+(def schema:audit-log-consumer-admin-console-params
+  "Params of the admin-console job: none. Cron submits {}; system deps
+  (pool, admin-console client) live on the job-def cfg closed over by
+  the handler, not in job params. Closed on purpose, same as the nexus
+  schema."
+  [:map {:closed true}])
+
+(defmethod ig/assert-key ::nexus-job-def
+  [_ cfg]
+  (assert (db/pool? (::db/pool cfg)) "expected a valid database pool")
+  (assert (http/client? (::http/client cfg)) "expected a valid http client")
+  (assert (mtx/metrics? (::mtx/metrics cfg)) "expected a valid metrics instance"))
+
+(defmethod ig/init-key ::nexus-job-def
+  [_ cfg]
+  ;; Effective destination, once at boot: per-run logs carry counts,
+  ;; the runner says the job started, this says where it sends.
+  (l/dbg :hint "audit log consumer job initialized"
+         :consumer "nexus"
+         :uri-host (some-> (cf/get :nexus-uri) u/uri u/get-domain))
+  {::jobs/name      :audit-log-consumer-nexus
+   ::jobs/schema    schema:audit-log-consumer-nexus-params
+   ::jobs/handler
+   (fn [_context params]
+     (execute-audit-log-consumer-nexus cfg params))
+   ::jobs/decoder   (sm/decoder schema:audit-log-consumer-nexus-params sm/json-transformer)
+   ::jobs/validator (sm/validator schema:audit-log-consumer-nexus-params)})
+
+(defmethod ig/assert-key ::admin-console-job-def
+  [_ cfg]
+  ;; No assert on the admin-console client: it inits to nil without the
+  ;; :admin-console flag, and the job skips on its flag anyway.
+  (assert (db/pool? (::db/pool cfg)) "expected a valid database pool")
+  (assert (mtx/metrics? (::mtx/metrics cfg)) "expected a valid metrics instance"))
+
+(defmethod ig/init-key ::admin-console-job-def
+  [_ cfg]
+  ;; Same as the nexus job-def: destination host once at boot. The
+  ;; ingest uri is resolved by nitrate/call at send time (it raises
+  ;; when missing); here it is display-only.
+  (l/dbg :hint "audit log consumer job initialized"
+         :consumer "admin-console"
+         :uri-host (some-> (cf/get :admin-console-uri) u/uri u/get-domain))
+  {::jobs/name      :audit-log-consumer-admin-console
+   ::jobs/schema    schema:audit-log-consumer-admin-console-params
+   ::jobs/handler
+   (fn [_context params]
+     (execute-audit-log-consumer-admin-console cfg params))
+   ::jobs/decoder   (sm/decoder schema:audit-log-consumer-admin-console-params sm/json-transformer)
+   ::jobs/validator (sm/validator schema:audit-log-consumer-admin-console-params)})
+
+(defn- run-single-consumer
+  "Drain one consumer queue, then housekeeping (reap disabled
+  pendings, sweep orphans) and the per-run summary. Housekeeping lives
+  in both jobs: only the enabled job runs, so it must reap the queues
+  of the consumers that are off; both halves are idempotent deletes.
+  A refused chunk (or a throw) aborts the job: the committed chunks
+  stay sent, the rest waits for the next tick, and the cron retries.
+  Every effect is rows and logs (drain metrics land here, outside
+  every chunk transaction)."
+  [cfg consumer send-chunk tpoint]
+  (let [label   (consumer-label consumer)
+        metrics (mtx/instance cfg)
+        reaped  (reap-disabled cfg)
+        ;; All drain metrics land here, outside every chunk
+        ;; transaction. Success is recorded on the totals: a normal
+        ;; chunk return means its transaction committed (a rollback
+        ;; always throws, taking the failure path instead). The send
+        ;; fns carry the attempted count as :events; anything else (a
+        ;; local DB failure) is not a destination failure and stays
+        ;; uncounted. The throw keeps rolling: a failed chunk aborts
+        ;; the job, the cron retries.
+        drained (try
+                  (drain-consumer cfg send-chunk)
+                  (catch Throwable cause
+                    (let [n (:events (ex-data cause))]
+                      (when (and (integer? n) (pos? n))
+                        (record-count metrics :audit-consumer-events [label "failed"] n)))
+                    (throw cause)))
+        _       (let [{:keys [sent skipped]} drained]
+                  (when (pos? sent)
+                    (record-count metrics :audit-consumer-events [label "archived"] sent))
+                  (when (pos? skipped)
+                    (record-count metrics :audit-consumer-events [label "skipped"] skipped)))
+        swept   (sweep-orphans cfg)
+        sent    (:sent drained)
+        skipped (:skipped drained)
+        chunks  (:chunks drained)
+        ;; Strict binding on purpose: log props evaluate lazily in
+        ;; the logger thread, so the elapsed must be frozen here.
+        elapsed (ct/format-duration (tpoint))]
+    (if (and (zero? sent)
+             (zero? skipped)
+             (zero? reaped)
+             (zero? swept))
+      (l/dbg :hint "no pending audit events"
+             :consumer label)
+      (l/dbg :hint "events archived"
+             :consumer label
+             :elapsed elapsed
+             :sent sent
+             :skipped skipped
+             :chunks chunks
+             :reaped reaped
+             :swept swept))))
+
+(defn execute-audit-log-consumer-nexus
+  "Nexus job handler: drain the nexus queue in chunks, then clean up
+  (heartbeat per iteration: the sent chunk batches can be long)."
+  [cfg params]
+  ;; NOTE: this let allows overwrite default configured values from
+  ;; the repl, when manually invoking the task.
+  (if-not (contains? cf/flags :nexus)
+    (l/dbg :hint "audit log consumer (nexus) skipped, flag off")
+    (let [uri (or (:uri params) (generate-nexus-uri "api/audit"))
+          ;; Normalize to an uri object; params may carry a plain string
+          ;; on direct invocations (validation only applies on submit!).
+          uri (u/uri uri)
+          cfg (assoc cfg ::uri uri)]
+      (when-not uri
+        (ex/raise :type :internal
+                  :code :task-not-configured
+                  :hint "consumer task not configured, missing uri"))
+      (run-single-consumer cfg consumer-nexus consumer-nexus-chunk (ct/tpoint)))))
+
+(defn execute-audit-log-consumer-admin-console
+  "Admin-console job handler: drain the admin-console queue in chunks,
+  then clean up (heartbeat per iteration: the sent chunk batches can
+  be long)."
+  [cfg _params]
+  (if-not (contains? cf/flags :admin-console)
+    (l/dbg :hint "audit log consumer (admin-console) skipped, flag off")
+    (run-single-consumer cfg consumer-admin-console consumer-admin-console-chunk (ct/tpoint))))

backend/src/app/loggers/audit/gc_task.clj

@@ -1,31 +0,0 @@
-;; This Source Code Form is subject to the terms of the Mozilla Public
-;; License, v. 2.0. If a copy of the MPL was not distributed with this
-;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
-;;
-;; Copyright (c) KALEIDOS SUBSIDIARY SL
-
-(ns app.loggers.audit.gc-task
-  (:require
-   [app.common.logging :as l]
-   [app.db :as db]
-   [integrant.core :as ig]))
-
-(def ^:private sql:clean-archived
-  "DELETE FROM audit_log
-    WHERE archived_at IS NOT NULL")
-
-(defn- clean-archived!
-  [{:keys [::db/pool]}]
-  (let [result (db/exec-one! pool [sql:clean-archived])
-        result (db/get-update-count result)]
-    (l/debug :hint "delete archived audit log entries" :deleted result)
-    result))
-
-(defmethod ig/assert-key ::handler
-  [_ params]
-  (assert (db/pool? (::db/pool params)) "valid database pool expected"))
-
-(defmethod ig/init-key ::handler
-  [_ cfg]
-  (fn [_]
-    (clean-archived! cfg)))

backend/src/app/loggers/database.clj

@@ -96,12 +96,11 @@
   "Convert the log record into a report object and persist it on the database"
   [{:keys [::db/pool]} {:keys [::l/id] :as record}]
   (try
-    (let [uri    (cf/get :public-uri)
-          report (-> record log-record->report d/without-nils)]
+    (let [report (-> record log-record->report d/without-nils)]
       (l/dbg :hint "registering error on database"
              :id (str id)
              :src "logging"
-             :uri (str uri "/dbg/error/" id))
+             :uri (cf/get-public-uri "dbg/error/" (str id)))
       (persist-on-database! pool id 3 report))
     (catch Throwable cause
       (l/warn :hint "unexpected exception on database error logger" :cause cause))))
@@ -139,12 +138,11 @@
   "Convert the log record into a report object and persist it on the database"
   [{:keys [::db/pool]} {:keys [id] :as event}]
   (try
-    (let [uri    (cf/get :public-uri)
-          report (-> event audit-event->report d/without-nils)]
+    (let [report (-> event audit-event->report d/without-nils)]
       (l/dbg :hint "registering error on database"
              :id (str id)
              :src "audit-log"
-             :uri (str uri "/dbg/error/" id))
+             :uri (cf/get-public-uri "dbg/error/" (str id)))
       (persist-on-database! pool id 4 report))
     (catch Throwable cause
       (l/warn :hint "unexpected exception on database error logger" :cause cause))))
@@ -172,18 +170,17 @@
      :value            (pp/pprint-str result :length 50)
      :tenant           (cf/get :tenant)
      :version          (:full cf/version)
-     :href             (str (cf/get :public-uri))}))
+     :href             (cf/get-public-uri)}))
 
 (defn- handle-rlimit-event
   "Convert the log record into a report object and persist it on the database"
   [{:keys [::db/pool]} {:keys [::rlimit/id] :as event}]
   (try
-    (let [uri    (cf/get :public-uri)
-          report (-> event rlimit-event->report d/without-nils)]
+    (let [report (-> event rlimit-event->report d/without-nils)]
       (l/dbg :hint "registering rate limit rejection"
              :id (str id)
              :src "rlimit"
-             :uri (str uri "/dbg/error/" id))
+             :uri (cf/get-public-uri "dbg/error/" (str id)))
       (persist-on-database! pool id 5 report))
     (catch Throwable cause
       (l/warn :hint "unexpected exception on database error logger" :cause cause))))

backend/src/app/loggers/mattermost.clj

@@ -11,7 +11,6 @@
    [app.common.exceptions :as ex]
    [app.common.logging :as l]
    [app.common.pprint :as pp]
-   [app.common.uri :as u]
    [app.config :as cf]
    [app.http.client :as http]
    [app.loggers.audit :as audit]
@@ -28,7 +27,7 @@
   (let [type (get report :type)
         text (str "#" type " | " (d/escape-markdown (get report :hint)) "\n"
                   (when id
-                    (str (u/join (cf/get :public-uri) "/dbg/error/" id) " "))
+                    (str (cf/get-public-uri "dbg/error/" id) " "))
 
                   (when-let [pid (:profile-id report)]
                     (if (uuid? pid)

backend/src/app/loggers/webhooks.clj

@@ -10,14 +10,15 @@
    [app.common.data :as d]
    [app.common.data.macros :as dm]
    [app.common.logging :as l]
+   [app.common.schema :as sm]
    [app.common.time :as ct]
    [app.common.transit :as t]
    [app.common.uri :as uri]
    [app.config :as cf]
    [app.db :as db]
    [app.http.client :as http]
+   [app.jobs :as jobs]
    [app.loggers.audit :as audit]
-   [app.worker :as wrk]
    [clojure.data.json :as json]
    [cuerdas.core :as str]
    [integrant.core :as ig]))
@@ -60,35 +61,54 @@
       (some->> (:project-id props) (lookup-webhooks-by-project pool))
       (some->> (:file-id props) (lookup-webhooks-by-file pool))))
 
-(defmethod ig/assert-key ::process-event-handler
+(defmethod ig/assert-key ::process-webhook-event-job-def
   [_ params]
   (assert (db/pool? (::db/pool params)) "expect valid database pool")
   (assert (http/client? (::http/client params)) "expect valid http client"))
 
-(defmethod ig/init-key ::process-event-handler
+(defn process-event-impl
+  [cfg {:keys [event-blob]}]
+  ;; The blob carries the original typed event; decode it once for the
+  ;; lookups below and forward it untouched so deliveries keep the
+  ;; exact instant/UUID/set types of the legacy worker.
+  (let [props (t/decode-str event-blob)
+        items (lookup-webhooks cfg props)
+        event {:profile-id (:profile-id props)
+               :name "webhook"
+               :type "trigger"
+               :props {:name (get props :name)
+                       :event-id (get props :id)
+                       :total-affected (count items)}}]
+
+    (audit/insert cfg event)
+
+    (when items
+      (l/trc :hint "webhooks found for event" :total (count items))
+      (db/tx-run! cfg (fn [cfg]
+                        (doseq [item items]
+                          (jobs/submit cfg
+                                       {::jobs/name :run-webhook
+                                        ::jobs/queue :webhooks
+                                        ::jobs/max-retries 3
+                                        ::jobs/params {:event event-blob
+                                                       :config (select-keys item [:id :uri :mtype])}})))))))
+
+
+(def schema:process-webhook-event-params
+  "The audited event travels as an opaque transit blob: transit
+  preserves the instant/UUID/set types that plain JSON props cannot
+  carry, so no per-key type declarations are needed."
+  [:map [:event-blob :string]])
+
+(defmethod ig/init-key ::process-webhook-event-job-def
   [_ cfg]
-  (fn [{:keys [props] :as task}]
-
-    (let [items (lookup-webhooks cfg props)
-          event {:profile-id (:profile-id props)
-                 :name "webhook"
-                 :type "trigger"
-                 :props {:name (get props :name)
-                         :event-id (get props :id)
-                         :total-affected (count items)}}]
-
-      (audit/insert cfg event)
-
-      (when items
-        (l/trc :hint "webhooks found for event" :total (count items))
-        (db/tx-run! cfg (fn [cfg]
-                          (doseq [item items]
-                            (wrk/submit! (-> cfg
-                                             (assoc ::wrk/task :run-webhook)
-                                             (assoc ::wrk/queue :webhooks)
-                                             (assoc ::wrk/max-retries 3)
-                                             (assoc ::wrk/params {:event props
-                                                                  :config item}))))))))))
+  {::jobs/name      :process-webhook-event
+   ::jobs/schema    schema:process-webhook-event-params
+   ::jobs/handler
+   (fn [_context params]
+     (process-event-impl cfg params))
+   ::jobs/decoder   (sm/decoder schema:process-webhook-event-params sm/json-transformer)
+   ::jobs/validator (sm/validator schema:process-webhook-event-params)})
 ;; --- RUN
 
 (declare interpret-exception)
@@ -98,17 +118,17 @@
   {:key-fn str/camel
    :indent true})
 
-(defmethod ig/assert-key ::run-webhook-handler
+(defmethod ig/assert-key ::run-webhook-job-def
   [_ params]
   (assert (db/pool? (::db/pool params)) "expect valid database pool")
   (assert (http/client? (::http/client params)) "expect valid http client"))
 
-(defmethod ig/expand-key ::run-webhook-handler
+(defmethod ig/expand-key ::run-webhook-job-def
   [k v]
   {k (merge {::max-errors 3} (d/without-nils v))})
 
-(defmethod ig/init-key ::run-webhook-handler
-  [_ {:keys [::db/pool ::max-errors] :as cfg}]
+(defn run-webhook-impl
+  [{:keys [::db/pool ::max-errors] :as cfg} props]
   (letfn [(update-webhook! [whook err]
             (if err
               (let [sql [(str "update webhook "
@@ -137,39 +157,68 @@
                          :req-data (db/tjson req)
                          :rsp-data (db/tjson rsp)}))]
 
-    (fn [{:keys [props] :as task}]
-      (let [event (:event props)
-            whook (:config props)
+    (let [event (t/decode-str (:event props))
+          whook (:config props)
 
-            body  (case (:mtype whook)
-                    "application/json" (json/write-str event json-write-opts)
-                    "application/transit+json" (t/encode-str event)
-                    "application/x-www-form-urlencoded" (uri/map->query-string event))]
+          body  (case (:mtype whook)
+                  "application/json" (json/write-str event json-write-opts)
+                  "application/transit+json" (t/encode-str event)
+                  "application/x-www-form-urlencoded" (uri/map->query-string event))]
 
-        (l/dbg :hint "run webhook"
-               :event-name (:name event)
-               :webhook-id (str (:id whook))
-               :webhook-uri (:uri whook)
-               :webhook-mtype (:mtype whook))
+      (l/dbg :hint "run webhook"
+             :event-name (:name event)
+             :webhook-id (str (:id whook))
+             :webhook-uri (:uri whook)
+             :webhook-mtype (:mtype whook))
 
-        (let [req {:uri (:uri whook)
-                   :headers {"content-type" (:mtype whook)
-                             "user-agent" (str/ffmt "penpot/%" (:main cf/version))}
-                   :timeout (ct/duration "4s")
-                   :method :post
-                   :body body}]
-          (try
-            (let [rsp (http/req cfg req {:response-type :input-stream :sync? true})
-                  err (interpret-response rsp)]
-              (report-delivery! whook req rsp err)
-              (update-webhook! whook err))
-            (catch Throwable cause
-              (let [err (interpret-exception cause)]
-                (report-delivery! whook req nil err)
-                (update-webhook! whook err)
-                (when (= err "unknown")
-                  (l/err :hint "unknown error on webhook request"
-                         :cause cause))))))))))
+      (let [req {:uri (:uri whook)
+                 :headers {"content-type" (:mtype whook)
+                           "user-agent" (str/ffmt "penpot/%" (:main cf/version))}
+                 :timeout (ct/duration "4s")
+                 :method :post
+                 :body body}]
+        (try
+          (let [rsp (http/req cfg req {:response-type :input-stream :sync? true})
+                err (interpret-response rsp)]
+            (report-delivery! whook req rsp err)
+            (update-webhook! whook err))
+          (catch Throwable cause
+            (let [err (interpret-exception cause)]
+              (report-delivery! whook req nil err)
+              (update-webhook! whook err)
+              (when (= err "unknown")
+                (l/err :hint "unknown error on webhook request"
+                       :cause cause)))))))))
+
+
+(def schema:run-webhook-config
+  "Subset of the webhook row consumed by the run-webhook job. The uri
+  stays text on purpose: configs come straight from the database and
+  submit! validates raw params without coercion."
+  [:map
+   [:id ::sm/uuid]
+   [:uri ::sm/text]
+   [:mtype ::sm/text]])
+
+(def schema:run-webhook-params
+  "The event travels as the original transit blob inside the JSON job
+  props: process-webhook-event receives the transit-encoded audited
+  event and forwards it untouched, and the handler transit-decodes it
+  before delivery. This keeps the wire payload identical to the legacy
+  worker for every mtype, whatever shape the event has."
+  [:map
+   [:event :string]
+   [:config schema:run-webhook-config]])
+
+(defmethod ig/init-key ::run-webhook-job-def
+  [_ cfg]
+  {::jobs/name      :run-webhook
+   ::jobs/schema    schema:run-webhook-params
+   ::jobs/handler
+   (fn [_context params]
+     (run-webhook-impl cfg params))
+   ::jobs/decoder   (sm/decoder schema:run-webhook-params sm/json-transformer)
+   ::jobs/validator (sm/validator schema:run-webhook-params)})
 
 (defn interpret-response
   [{:keys [status] :as response}]

backend/src/app/main.clj

@@ -22,9 +22,12 @@
    [app.http.debug :as-alias http.debug]
    [app.http.link-preview :as-alias http.link-preview]
    [app.http.management :as mgmt]
+   [app.http.mcp-oauth :as-alias http.mcp-oauth]
    [app.http.session :as session]
-   [app.http.session.tasks :as-alias session.tasks]
    [app.http.websocket :as http.ws]
+   [app.jobs :as-alias jobs]
+   [app.jobs.gc :as-alias jobs.gc]
+   [app.jobs.metrics :as jobs-metrics]
    [app.loggers.webhooks :as-alias webhooks]
    [app.metrics :as mtx]
    [app.metrics.definition :as-alias mdef]
@@ -118,7 +121,12 @@
 
    :tasks-timing
    {::mdef/name "penpot_tasks_timing"
-    ::mdef/help "Background tasks timing (milliseconds)."
+    ;; Deprecated: superseded by `penpot_jobs_execution_ms`, which adds a
+    ;; queue label. Kept only for existing dashboards and alerts. The help
+    ;; text is what someone reading /metrics sees, so it is where the
+    ;; redirect belongs: the name and the labels stay untouched, so no
+    ;; panel or alert breaks.
+    ::mdef/help "Deprecated: use penpot_jobs_execution_ms, which adds a queue label."
     ::mdef/labels ["name"]
     ::mdef/buckets mtx/default-tasks-histogram-buckets
     ::mdef/type :histogram}
@@ -254,10 +262,160 @@
     ::mdef/help "Total number of handler errors in the http listener."
     ::mdef/type :counter}
 
+   :jobs-submitted
+   {::mdef/name "penpot_jobs_submitted_total"
+    ::mdef/help "Total number of durable jobs submitted."
+    ::mdef/labels ["name" "queue"]
+    ::mdef/type :counter}
+
+   :jobs-skipped
+   {::mdef/name "penpot_jobs_skipped_total"
+    ::mdef/help "Total number of durable job submissions skipped by skip-if-active."
+    ::mdef/labels ["name" "queue"]
+    ::mdef/type :counter}
+
+   :jobs-dispatched
+   {::mdef/name "penpot_jobs_dispatched_total"
+    ::mdef/help "Total number of jobs pushed to a worker queue. Counts pushes, not claims: compare with the _count of penpot_jobs_queue_wait_ms."
+    ::mdef/labels ["name" "queue"]
+    ::mdef/type :counter}
+
+   :jobs-retries
+   {::mdef/name "penpot_jobs_retries_total"
+    ::mdef/help "Total number of durable jobs rescheduled for retry."
+    ::mdef/labels ["name" "queue" "reason"]
+    ::mdef/type :counter}
+
+   :jobs-events-total
+   {::mdef/name "penpot_jobs_events_total"
+    ::mdef/help "Total number of job events stored."
+    ::mdef/type :counter}
+
+   :jobs-orphaned
+   {::mdef/name "penpot_jobs_orphaned_total"
+    ::mdef/help "Total number of running jobs marked as orphaned."
+    ::mdef/labels ["queue"]
+    ::mdef/type :counter}
+
+   :jobs-rescheduled
+   {::mdef/name "penpot_jobs_rescheduled_total"
+    ::mdef/help "Total number of scheduled jobs recovered after transit loss."
+    ::mdef/labels ["queue"]
+    ::mdef/type :counter}
+
+   :jobs-queue-wait-timing
+   {::mdef/name "penpot_jobs_queue_wait_ms"
+    ::mdef/help "Time from job scheduling to worker claim (milliseconds). Its _count counts claims, not pushes: compare with penpot_jobs_dispatched_total."
+    ::mdef/labels ["name" "queue"]
+    ::mdef/type :histogram
+    ::mdef/buckets [1 10 50 100 500 1000 5000 10000 30000 60000 300000 600000]}
+
+   :jobs-execution-timing
+   {::mdef/name "penpot_jobs_execution_ms"
+    ::mdef/help "Time spent executing a job handler (milliseconds)."
+    ::mdef/labels ["name" "queue"]
+    ::mdef/type :histogram
+    ::mdef/buckets [1 10 50 100 500 1000 5000 10000 30000 60000 300000 600000]}
+
+   :jobs-total-timing
+   {::mdef/name "penpot_jobs_total_ms"
+    ::mdef/help "Time from job creation to a terminal state (milliseconds). Its _count is the terminal total per outcome."
+    ::mdef/labels ["name" "queue" "outcome"]
+    ::mdef/type :histogram
+    ::mdef/buckets [10 50 100 500 1000 5000 10000 30000 60000 300000 600000 3600000]}
+
+   :jobs-dispatcher-timing
+   {::mdef/name "penpot_jobs_dispatcher_ms"
+    ::mdef/help "Dispatcher batch processing time (milliseconds)."
+    ::mdef/labels ["stage" "outcome"]
+    ::mdef/type :histogram
+    ::mdef/buckets [1 10 50 100 500 1000 5000 10000 30000 60000]}
+
+   :jobs-dispatcher-batch-size
+   {::mdef/name "penpot_jobs_dispatcher_batch_size"
+    ::mdef/help "Number of jobs in a dispatcher queue batch."
+    ::mdef/labels ["queue"]
+    ::mdef/type :histogram
+    ::mdef/buckets [1 5 10 25 50 100 250 500 1000]}
+
+   :jobs-backlog
+   {::mdef/name "penpot_jobs_backlog"
+    ::mdef/help "Current number of jobs grouped by status."
+    ::mdef/labels ["status"]
+    ::mdef/type :gauge}
+
+   :jobs-oldest-pending-age
+   {::mdef/name "penpot_jobs_oldest_pending_age_seconds"
+    ::mdef/help "Age in seconds of the oldest pending job."
+    ::mdef/type :gauge}
+
+   :jobs-gc-rows
+   {::mdef/name "penpot_jobs_gc_rows_total"
+    ::mdef/help "Rows removed or resources touched by the jobs garbage collector."
+    ::mdef/labels ["kind" "action"]
+    ::mdef/type :counter}
+
+   :jobs-gc-timing
+   {::mdef/name "penpot_jobs_gc_ms"
+    ::mdef/help "Time spent in each jobs garbage collection pass (milliseconds)."
+    ::mdef/labels ["kind"]
+    ::mdef/type :histogram
+    ::mdef/buckets [1 10 50 100 500 1000 5000 10000 30000 60000 300000]}
+
+   :jobs-cron-total
+   {::mdef/name "penpot_jobs_cron_total"
+    ::mdef/help "Total number of cron scheduler events."
+    ::mdef/labels ["outcome" "reason"]
+    ::mdef/type :counter}
+
+   :jobs-request-timing
+   {::mdef/name "penpot_jobs_request_ms"
+    ::mdef/help "Time spent waiting for an ephemeral job reply (milliseconds). Its _count is the request total per outcome."
+    ::mdef/labels ["outcome"]
+    ::mdef/type :histogram
+    ::mdef/buckets [1 10 50 100 500 1000 5000 10000 30000 60000 300000]}
+
    :http-malformed-requests-total
    {::mdef/name "penpot_http_malformed_requests_total"
     ::mdef/help "Total number of requests whose body the parser could not read."
-    ::mdef/type :counter}})
+    ::mdef/type :counter}
+
+   :audit-consumer-events
+   {::mdef/name "penpot_audit_consumer_events_total"
+    ::mdef/help "Audit log events drained by the consumer task, by destination and outcome."
+    ::mdef/labels ["destination" "outcome"]
+    ::mdef/type :counter}
+
+   :audit-consumer-discarded
+   {::mdef/name "penpot_audit_consumer_discarded_total"
+    ::mdef/help "Audit log entries discarded without being archived, by reason."
+    ::mdef/labels ["reason"]
+    ::mdef/type :counter}
+
+   :audit-ingest-events
+   {::mdef/name "penpot_audit_ingest_events_total"
+    ::mdef/help "Audit log frontend events ingested, by kind and outcome."
+    ::mdef/labels ["kind" "outcome"]
+    ::mdef/type :counter}
+
+   :file-data-offload-rows
+   {::mdef/name "penpot_file_data_offload_rows_total"
+    ::mdef/help "File data rows moved from hot to cold storage by the offload job."
+    ::mdef/labels ["type"]
+    ::mdef/type :counter}
+
+   :file-data-offload-bytes
+   {::mdef/name "penpot_file_data_offload_bytes_total"
+    ::mdef/help "File data bytes moved from hot to cold storage by the offload job."
+    ::mdef/labels ["type"]
+    ::mdef/type :counter}
+
+   :file-data-offload-timing
+   {::mdef/name "penpot_file_data_offload_timing_ms"
+    ::mdef/help "Time spent offloading one file from hot to cold storage (milliseconds)."
+    ::mdef/labels ["result"]
+    ::mdef/type :histogram
+    ::mdef/buckets [1 10 50 100 500 1000 5000 10000 30000 60000 300000]}})
 
 (def system-config
   {::db/pool
@@ -304,28 +462,12 @@
    :app.storage.tmp/cleaner
    {::wrk/executor (ig/ref ::wrk/executor)}
 
-   ::sto.gc-deleted/handler
-   {::db/pool      (ig/ref ::db/pool)
-    ::sto/storage  (ig/ref ::sto/storage)
-    ::mtx/metrics  (ig/ref ::mtx/metrics)}
-
-   ::sto.gc-touched/handler
-   {::db/pool      (ig/ref ::db/pool)
-    ::mtx/metrics  (ig/ref ::mtx/metrics)}
-
-   ::sto.pending-gc/handler
-   {::db/pool     (ig/ref ::db/pool)
-    ::sto/storage (ig/ref ::sto/storage)}
-
    ::http.client/client
    {::wrk/executor (ig/ref ::wrk/executor)}
 
    ::session/manager
    {::db/pool (ig/ref ::db/pool)}
 
-   ::session.tasks/gc
-   {::db/pool (ig/ref ::db/pool)}
-
    ::http.awsns/routes
    {::setup/props        (ig/ref ::setup/props)
     ::db/pool            (ig/ref ::db/pool)
@@ -374,6 +516,7 @@
 
    ::oidc/routes
    {::http.client/client (ig/ref ::http.client/client)
+    ::jobs/defs          (ig/ref ::jobs/defs)
     ::db/pool            (ig/ref ::db/pool)
     ::setup/props        (ig/ref ::setup/props)
     ::oidc/providers     (ig/ref ::oidc/providers)
@@ -398,9 +541,17 @@
     ::http.debug/routes  (ig/ref ::http.debug/routes)
     ::http.assets/routes (ig/ref ::http.assets/routes)
     ::http.link-preview/routes (ig/ref ::http.link-preview/routes)
+    ::http.mcp-oauth/routes (ig/ref ::http.mcp-oauth/routes)
     ::http.ws/routes     (ig/ref ::http.ws/routes)
     ::http.awsns/routes  (ig/ref ::http.awsns/routes)}
 
+   ::http.mcp-oauth/routes
+   {::db/pool (ig/ref ::db/pool)
+    ::rds/pool (ig/ref ::rds/pool)
+    ::setup/props (ig/ref ::setup/props)
+    ::session/manager (ig/ref ::session/manager)
+    ::http.client/client (ig/ref ::http.client/client)}
+
    ::http.link-preview/routes
    {::db/pool         (ig/ref ::db/pool)}
 
@@ -447,6 +598,7 @@
 
    :app.rpc/methods
    {::http.client/client (ig/ref ::http.client/client)
+    ::jobs/defs           (ig/ref ::jobs/defs)
     ::db/pool            (ig/ref ::db/pool)
     ::rds/pool           (ig/ref ::rds/pool)
     :app.nitrate/client  (ig/ref :app.nitrate/client)
@@ -475,10 +627,13 @@
 
    :app.nitrate/client
    {::http.client/client (ig/ref ::http.client/client)
+    ;; No ::jobs/defs: nitrate never submits jobs, and consumer-task's
+    ;; job-def needs this client — that would cycle through jobs/defs.
     ::setup/shared-keys  (ig/ref ::setup/shared-keys)}
 
    :app.rpc/management-methods
    {::http.client/client (ig/ref ::http.client/client)
+    ::jobs/defs          (ig/ref ::jobs/defs)
     ::db/pool            (ig/ref ::db/pool)
     ::rds/pool           (ig/ref ::rds/pool)
     ::wrk/executor       (ig/ref ::wrk/executor)
@@ -500,31 +655,129 @@
     ::setup/props            (ig/ref ::setup/props)
     ::setup/shared-keys      (ig/ref ::setup/shared-keys)}
 
-   ::wrk/registry
-   {::mtx/metrics (ig/ref ::mtx/metrics)
-    ::wrk/tasks
-    {:sendmail           (ig/ref ::email/handler)
-     :objects-gc         (ig/ref :app.tasks.objects-gc/handler)
-     :file-gc            (ig/ref :app.tasks.file-gc/handler)
-     :file-gc-scheduler  (ig/ref :app.tasks.file-gc-scheduler/handler)
-     :offload-file-data  (ig/ref :app.tasks.offload-file-data/handler)
-     :tasks-gc           (ig/ref :app.tasks.tasks-gc/handler)
-     :telemetry          (ig/ref :app.tasks.telemetry/handler)
-     :storage-gc-deleted (ig/ref ::sto.gc-deleted/handler)
-     :storage-gc-touched (ig/ref ::sto.gc-touched/handler)
-     :storage-pending-gc (ig/ref ::sto.pending-gc/handler)
-     :session-gc         (ig/ref ::session.tasks/gc)
-     :audit-log-archive  (ig/ref :app.loggers.audit.archive-task/handler)
-     :audit-log-gc       (ig/ref :app.loggers.audit.gc-task/handler)
+   ::jobs/defs
+   {:sendmail                         (ig/ref ::email/job-def)
+    :send-templated-mail              (ig/ref ::email/templated-job-def)
+    :delete-object                    (ig/ref :app.tasks.delete-object/job-def)
+    :demo-purge                       (ig/ref :app.tasks.demo-purge/job-def)
+    :run-webhook                      (ig/ref ::webhooks/run-webhook-job-def)
+    :process-webhook-event            (ig/ref ::webhooks/process-webhook-event-job-def)
+    :file-gc                          (ig/ref :app.tasks.file-gc/job-def)
+    :offload-file-data                (ig/ref :app.tasks.offload-file-data/job-def)
+    :objects-gc                       (ig/ref :app.tasks.objects-gc/job-def)
+    :export-binfile                   (ig/ref :app.tasks.export-binfile/job-def)
+    :import-binfile                   (ig/ref :app.tasks.import-binfile/job-def)
+    :storage-gc-deleted               (ig/ref ::sto.gc-deleted/job-def)
+    :storage-gc-touched               (ig/ref ::sto.gc-touched/job-def)
+    :storage-pending-gc               (ig/ref ::sto.pending-gc/job-def)
+    :jobs-gc                          (ig/ref :app.jobs.gc/job-def)
+    :telemetry                        (ig/ref :app.tasks.telemetry/job-def)
+    :session-gc                       (ig/ref ::session/job-def)
+    :file-gc-scheduler                (ig/ref :app.tasks.file-gc-scheduler/job-def)
+    :audit-log-consumer-nexus         (ig/ref :app.loggers.audit.consumer-task/nexus-job-def)
+    :audit-log-consumer-admin-console (ig/ref :app.loggers.audit.consumer-task/admin-console-job-def)}
 
-     :delete-object
-     (ig/ref :app.tasks.delete-object/handler)
-     :demo-purge
-     (ig/ref :app.tasks.demo-purge/handler)
-     :process-webhook-event
-     (ig/ref ::webhooks/process-event-handler)
-     :run-webhook
-     (ig/ref ::webhooks/run-webhook-handler)}}
+   ::jobs/request-pool
+   {::rds/client  (ig/ref ::rds/client)
+    ::mtx/metrics (ig/ref ::mtx/metrics)}
+
+   :app.email/job-def
+   {::email/sendmail (ig/ref ::email/sendmail)
+    ::mtx/metrics   (ig/ref ::mtx/metrics)}
+
+   :app.email/templated-job-def
+   {::email/sendmail (ig/ref ::email/sendmail)
+    ::mtx/metrics   (ig/ref ::mtx/metrics)}
+
+   :app.tasks.delete-object/job-def
+   {::db/pool     (ig/ref ::db/pool)
+    ::sto/storage (ig/ref ::sto/storage)
+    ::mtx/metrics (ig/ref ::mtx/metrics)}
+
+   :app.tasks.demo-purge/job-def
+   {::db/pool     (ig/ref ::db/pool)
+    ::mtx/metrics (ig/ref ::mtx/metrics)}
+
+   :app.loggers.webhooks/run-webhook-job-def
+   {::db/pool     (ig/ref ::db/pool)
+    ::http.client/client (ig/ref ::http.client/client)
+    ::mtx/metrics (ig/ref ::mtx/metrics)}
+
+   :app.loggers.webhooks/process-webhook-event-job-def
+   {::db/pool     (ig/ref ::db/pool)
+    ::http.client/client (ig/ref ::http.client/client)
+    ::mtx/metrics (ig/ref ::mtx/metrics)}
+
+   :app.tasks.file-gc/job-def
+   {::db/pool     (ig/ref ::db/pool)
+    ::sto/storage (ig/ref ::sto/storage)
+    ::mtx/metrics (ig/ref ::mtx/metrics)}
+
+   :app.tasks.offload-file-data/job-def
+   {::db/pool     (ig/ref ::db/pool)
+    ::sto/storage (ig/ref ::sto/storage)
+    ::mtx/metrics (ig/ref ::mtx/metrics)}
+
+   :app.tasks.objects-gc/job-def
+   {::db/pool     (ig/ref ::db/pool)
+    ::sto/storage (ig/ref ::sto/storage)
+    ::mtx/metrics (ig/ref ::mtx/metrics)}
+
+   :app.tasks.export-binfile/job-def
+   {::db/pool      (ig/ref ::db/pool)
+    ::sto/storage  (ig/ref ::sto/storage)
+    ::mtx/metrics  (ig/ref ::mtx/metrics)
+    ;; the job belongs to a profile, so its events are published
+    ::mbus/msgbus  (ig/ref ::mbus/msgbus)}
+
+   :app.tasks.import-binfile/job-def
+   {::db/pool      (ig/ref ::db/pool)
+    ::sto/storage  (ig/ref ::sto/storage)
+    ::mtx/metrics  (ig/ref ::mtx/metrics)
+    ::mbus/msgbus  (ig/ref ::mbus/msgbus)}
+
+   :app.storage.gc-deleted/job-def
+   {::db/pool     (ig/ref ::db/pool)
+    ::sto/storage (ig/ref ::sto/storage)
+    ::mtx/metrics (ig/ref ::mtx/metrics)}
+
+   :app.storage.gc-touched/job-def
+   {::db/pool     (ig/ref ::db/pool)
+    ::mtx/metrics (ig/ref ::mtx/metrics)}
+
+   :app.storage.pending-gc/job-def
+   {::db/pool     (ig/ref ::db/pool)
+    ::sto/storage (ig/ref ::sto/storage)
+    ::mtx/metrics (ig/ref ::mtx/metrics)}
+
+   :app.jobs.gc/job-def
+   {::db/pool     (ig/ref ::db/pool)
+    ::mtx/metrics (ig/ref ::mtx/metrics)}
+
+   :app.tasks.telemetry/job-def
+   {::db/pool     (ig/ref ::db/pool)
+    ::http.client/client (ig/ref ::http.client/client)
+    ::setup/props (ig/ref ::setup/props)
+    ::mtx/metrics (ig/ref ::mtx/metrics)}
+
+   ::session/job-def
+   {::db/pool     (ig/ref ::db/pool)
+    ::mtx/metrics (ig/ref ::mtx/metrics)}
+
+   :app.tasks.file-gc-scheduler/job-def
+   {::db/pool     (ig/ref ::db/pool)
+    ::mtx/metrics (ig/ref ::mtx/metrics)}
+
+   :app.loggers.audit.consumer-task/nexus-job-def
+   {::db/pool            (ig/ref ::db/pool)
+    ::http.client/client (ig/ref ::http.client/client)
+    ::setup/shared-keys  (ig/ref ::setup/shared-keys)
+    ::mtx/metrics        (ig/ref ::mtx/metrics)}
+
+   :app.loggers.audit.consumer-task/admin-console-job-def
+   {::db/pool            (ig/ref ::db/pool)
+    :app.nitrate/client  (ig/ref :app.nitrate/client)
+    ::mtx/metrics        (ig/ref ::mtx/metrics)}
 
    ::email/blacklist
    {}
@@ -542,38 +795,6 @@
     ::email/default-reply-to (cf/get :smtp-default-reply-to)
     ::email/default-from     (cf/get :smtp-default-from)}
 
-   ::email/handler
-   {::email/sendmail (ig/ref ::email/sendmail)}
-
-   :app.tasks.tasks-gc/handler
-   {::db/pool (ig/ref ::db/pool)}
-
-   :app.tasks.objects-gc/handler
-   {::db/pool     (ig/ref ::db/pool)
-    ::sto/storage (ig/ref ::sto/storage)}
-
-   :app.tasks.delete-object/handler
-   {::db/pool     (ig/ref ::db/pool)
-    ::sto/storage (ig/ref ::sto/storage)}
-
-   :app.tasks.demo-purge/handler
-   {::db/pool (ig/ref ::db/pool)}
-
-   :app.tasks.file-gc/handler
-   {::db/pool     (ig/ref ::db/pool)
-    ::sto/storage (ig/ref ::sto/storage)}
-
-   :app.tasks.file-gc-scheduler/handler
-   {::db/pool (ig/ref ::db/pool)}
-
-   :app.tasks.offload-file-data/handler
-   {::db/pool     (ig/ref ::db/pool)
-    ::sto/storage (ig/ref ::sto/storage)}
-
-   :app.tasks.telemetry/handler
-   {::db/pool            (ig/ref ::db/pool)
-    ::http.client/client (ig/ref ::http.client/client)
-    ::setup/props        (ig/ref ::setup/props)}
 
    ::srepl/urepl
    {:port (cf/get :urepl-port 6062)
@@ -607,22 +828,6 @@
    ::setup/clock
    {}
 
-   :app.loggers.audit.archive-task/handler
-   {::setup/shared-keys  (ig/ref ::setup/shared-keys)
-    ::http.client/client (ig/ref ::http.client/client)
-    ::db/pool            (ig/ref ::db/pool)}
-
-   :app.loggers.audit.gc-task/handler
-   {::db/pool (ig/ref ::db/pool)}
-
-   ::webhooks/process-event-handler
-   {::db/pool            (ig/ref ::db/pool)
-    ::http.client/client (ig/ref ::http.client/client)}
-
-   ::webhooks/run-webhook-handler
-   {::db/pool            (ig/ref ::db/pool)
-    ::http.client/client (ig/ref ::http.client/client)}
-
    :app.loggers.mattermost/reporter
    {::http.client/client (ig/ref ::http.client/client)}
 
@@ -665,41 +870,41 @@
 
 (def worker-config
   {::wrk/cron
-   {::wrk/registry            (ig/ref ::wrk/registry)
+   {::jobs/defs               (ig/ref ::jobs/defs)
     ::db/pool                 (ig/ref ::db/pool)
+    ::mtx/metrics             (ig/ref ::mtx/metrics)
     ::wrk/entries
     [{:cron #penpot/cron "0 0 0 * * ?" ;; daily
-      :task :session-gc}
+      :job :session-gc}
 
      {:cron #penpot/cron "0 0 0 * * ?" ;; daily
-      :task :objects-gc}
-
-     ;; FIXME: temporary commented
-     ;; {:cron #penpot/cron "0 0 0 * * ?" ;; daily
-     ;;  :task :storage-gc-deleted}
+      :job :objects-gc}
 
      {:cron #penpot/cron "0 0 0 * * ?" ;; daily
-      :task :storage-gc-touched}
+      :job :storage-gc-deleted}
 
      {:cron #penpot/cron "0 0 0 * * ?" ;; daily
-      :task :storage-pending-gc}
+      :job :storage-gc-touched}
 
      {:cron #penpot/cron "0 0 0 * * ?" ;; daily
-      :task :tasks-gc}
+      :job :storage-pending-gc}
+
+     {:cron #penpot/cron "0 0 0 * * ?" ;; daily
+      :job :jobs-gc}
 
      {:cron #penpot/cron "0 0 2 * * ?" ;; daily
-      :task :file-gc-scheduler}
+      :job :file-gc-scheduler}
 
      {:cron #penpot/cron "0 30 */3,23 * * ?"
-      :task :telemetry}
+      :job :telemetry}
 
-     (when (contains? cf/flags :audit-log-archive)
+     (when (contains? cf/flags :nexus)
        {:cron #penpot/cron "0 */5 * * * ?" ;; every 5m
-        :task :audit-log-archive})
+        :job :audit-log-consumer-nexus})
 
-     (when (contains? cf/flags :audit-log-gc)
-       {:cron #penpot/cron "30 */5 * * * ?" ;; every 5m
-        :task :audit-log-gc})]}
+     (when (contains? cf/flags :admin-console)
+       {:cron #penpot/cron "30 */5 * * * ?" ;; every 5m, offset 30s from nexus
+        :job :audit-log-consumer-admin-console})]}
 
    ::wrk/dispatcher
    {::rds/client  (ig/ref ::rds/client)
@@ -707,23 +912,50 @@
     ::db/pool     (ig/ref ::db/pool)
     ::wrk/tenant  (cf/get :tenant)}
 
+   ::jobs-metrics/sampler
+   {::db/pool     (ig/ref ::db/pool)
+    ::mtx/metrics (ig/ref ::mtx/metrics)}
+
    [::default ::wrk/runner]
    {::wrk/parallelism (cf/get :worker-default-parallelism 1)
     ::wrk/queue       :default
     ::wrk/tenant      (cf/get :tenant)
     ::rds/client      (ig/ref ::rds/client)
-    ::wrk/registry    (ig/ref ::wrk/registry)
+    ::jobs/defs       (ig/ref ::jobs/defs)
     ::mtx/metrics     (ig/ref ::mtx/metrics)
-    ::db/pool         (ig/ref ::db/pool)}
+    ::db/pool         (ig/ref ::db/pool)
+    ::mbus/msgbus     (ig/ref ::mbus/msgbus)}
 
    [::webhook ::wrk/runner]
    {::wrk/parallelism (cf/get :worker-webhook-parallelism 1)
     ::wrk/queue       :webhooks
     ::wrk/tenant      (cf/get :tenant)
     ::rds/client      (ig/ref ::rds/client)
-    ::wrk/registry    (ig/ref ::wrk/registry)
+    ::jobs/defs       (ig/ref ::jobs/defs)
     ::mtx/metrics     (ig/ref ::mtx/metrics)
-    ::db/pool         (ig/ref ::db/pool)}})
+    ::db/pool         (ig/ref ::db/pool)
+    ::mbus/msgbus     (ig/ref ::mbus/msgbus)}
+
+   ;; the heavy work of a user job does not share the default runner
+   [::binfile ::wrk/runner]
+   {::wrk/parallelism (cf/get :worker-binfile-parallelism 1)
+    ::wrk/queue       :binfile
+    ::wrk/tenant      (cf/get :tenant)
+    ::rds/client      (ig/ref ::rds/client)
+    ::jobs/defs       (ig/ref ::jobs/defs)
+    ::mtx/metrics     (ig/ref ::mtx/metrics)
+    ::db/pool         (ig/ref ::db/pool)
+    ::mbus/msgbus     (ig/ref ::mbus/msgbus)}
+
+   [::cron ::wrk/runner]
+   {::wrk/parallelism (cf/get :worker-cron-parallelism 2)
+    ::wrk/queue       :cron
+    ::wrk/tenant      (cf/get :tenant)
+    ::rds/client      (ig/ref ::rds/client)
+    ::jobs/defs       (ig/ref ::jobs/defs)
+    ::mtx/metrics     (ig/ref ::mtx/metrics)
+    ::db/pool         (ig/ref ::db/pool)
+    ::mbus/msgbus     (ig/ref ::mbus/msgbus)}})
 
 
 (defn start

backend/src/app/mcp/oauth_clients.clj

@@ -0,0 +1,220 @@
+;; This Source Code Form is subject to the terms of the Mozilla Public
+;; License, v. 2.0. If a copy of the MPL was not distributed with this
+;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
+;;
+;; Copyright (c) KALEIDOS SUBSIDIARY SL
+
+(ns app.mcp.oauth-clients
+  "Client metadata from self-contained registrations or HTTPS documents.
+  Registration describes an application; it does not authorize a Penpot user.
+  Both sources resolve to the same metadata shape without a client database."
+  (:require
+   [app.http.client :as http]
+   [app.mcp.oauth-state :as state]
+   [app.util.cache :as cache]
+   [app.util.json :as json]
+   [clojure.string :as str])
+  (:import
+   java.io.InputStream
+   java.net.URI
+   java.util.Base64))
+
+(defn invalid!
+  ([error] (invalid! error 400))
+  ([error status]
+   (throw (ex-info "MCP OAuth request rejected" {:oauth-error error :status status}))))
+
+(defn uri
+  [value]
+  (when (and (string? value) (<= 1 (count value) 2048))
+    (try (URI. value) (catch Exception _ nil))))
+
+(defn- loopback?
+  [^URI uri]
+  (and (= "http" (.getScheme uri))
+       (contains? #{"127.0.0.1" "[::1]" "localhost"} (.getHost uri))))
+
+(defn redirect-uri?
+  [value]
+  (when-let [uri (uri value)]
+    (and (nil? (.getFragment uri))
+         (nil? (.getUserInfo uri))
+         (or (and (= "https" (.getScheme uri)) (some? (.getHost uri)))
+             (loopback? uri)
+             ;; Native applications use a reverse-domain scheme.
+             (and (some? (.getScheme uri))
+                  (str/includes? (.getScheme uri) ".")
+                  (not (.isOpaque uri)))))))
+
+(defn redirect-matches?
+  "Exact callback match, except for the port of loopback HTTP URLs.
+  Native applications may allocate a different local port for each login."
+  [registered requested]
+  (and (redirect-uri? requested)
+       (some (fn [value]
+               (or (= value requested)
+                   (let [a (uri value)
+                         b (uri requested)]
+                     (and (loopback? a) (loopback? b)
+                          (= (.getHost a) (.getHost b))
+                          (= (.getRawPath a) (.getRawPath b))
+                          (= (.getRawQuery a) (.getRawQuery b))))))
+             (:redirect_uris registered))))
+
+(defn- logo-uri?
+  [value]
+  (when-let [uri (uri value)]
+    (and (= "https" (.getScheme uri))
+         (some? (.getHost uri))
+         (nil? (.getUserInfo uri))
+         (nil? (.getFragment uri)))))
+
+(defn validate-metadata
+  "Validate untrusted metadata and retain the fields this server supports.
+  CIMD selects public-client authentication from the advertised methods;
+  dynamic registration also supports Basic and form-post client secrets."
+  [metadata cimd?]
+  (let [redirects (:redirect_uris metadata)
+        grants    (get metadata :grant_types ["authorization_code"])
+        methods   (if cimd?
+                    (or (:token_endpoint_auth_methods_supported metadata)
+                        [(get metadata :token_endpoint_auth_method "none")])
+                    [(get metadata :token_endpoint_auth_method "client_secret_basic")])
+        method    (if cimd? "none" (first methods))]
+    (when-not (and (map? metadata)
+                   (vector? redirects) (<= 1 (count redirects) 20)
+                   (every? redirect-uri? redirects)
+                   (or (nil? (:client_name metadata))
+                       (and (string? (:client_name metadata))
+                            (<= 1 (count (:client_name metadata)) 200)))
+                   ;; Published metadata describes client capabilities across
+                   ;; servers. Extra grants do not require us to issue them.
+                   (if cimd?
+                     (and (vector? grants) (every? string? grants)
+                          (some #{"authorization_code"} grants))
+                     (= ["authorization_code"] grants))
+                   (= ["code"] (get metadata :response_types ["code"]))
+                   (or (nil? (:scope metadata)) (= "mcp" (:scope metadata)))
+                   (if cimd?
+                     (and (vector? methods) (some #{"none"} methods))
+                     (contains? #{"none" "client_secret_basic" "client_secret_post"} method)))
+      (invalid! "invalid_client_metadata"))
+    (cond-> {:redirect_uris redirects
+             :grant_types ["authorization_code"]
+             :response_types ["code"]
+             :scope "mcp"
+             :token_endpoint_auth_method method}
+      (:client_name metadata) (assoc :client_name (:client_name metadata))
+      ;; Logos are optional browser-loaded images. Ignore unusable URLs
+      ;; instead of preventing authorization because of cosmetic metadata.
+      (logo-uri? (:logo_uri metadata)) (assoc :logo_uri (:logo_uri metadata)))))
+
+(defn register!
+  "Issue a reusable client ID and, when applicable, a separate client secret.
+  No client record is stored. The public ID carries only the secret hash;
+  the random nonce distinguishes registrations with otherwise equal metadata."
+  [_cfg metadata]
+  (let [metadata (validate-metadata metadata false)
+        secret   (when-not (= "none" (:token_endpoint_auth_method metadata))
+                   (state/random-secret))
+        ;; Omit fixed protocol values to keep IDs short; validation restores
+        ;; them on resolution. The version prefix defines the payload format.
+        payload  (cond-> (-> metadata
+                             (dissoc :grant_types :response_types :scope)
+                             (assoc :nonce (state/random-secret)))
+                   secret (assoc :secret-hash (state/digest secret)))
+        id       (str "penpot-mcp-v1."
+                      (.encodeToString (.withoutPadding (Base64/getUrlEncoder))
+                                       (json/encode payload)))]
+    ;; Enforce the resolver's size cap here rather than issue an unusable ID.
+    (when (> (count id) 2048)
+      (invalid! "invalid_client_metadata"))
+    (cond-> (assoc metadata :client_id id
+                   :client_id_issued_at (quot (System/currentTimeMillis) 1000))
+      secret (assoc :client_secret secret :client_secret_expires_at 0))))
+
+(defn- decode-client-id
+  [client-id]
+  ;; Open registration accepts client-supplied metadata, not a trusted identity.
+  ;; Changing these details changes the ID; codes are bound to the exact ID
+  ;; approved by the user, so an edited ID cannot redeem an existing code.
+  (try
+    (let [[_ encoded] (re-matches #"penpot-mcp-v1\.([A-Za-z0-9_-]+)" client-id)
+          _           (when-not encoded (invalid! "invalid_client"))
+          payload     (json/decode (.decode (Base64/getUrlDecoder) ^String encoded))
+          metadata    (validate-metadata payload false)
+          nonce       (:nonce payload)
+          secret-hash (:secret-hash payload)]
+      (when-not (and (string? nonce) (re-matches #"[A-Za-z0-9_-]{43}" nonce)
+                     (if (= "none" (:token_endpoint_auth_method metadata))
+                       (nil? secret-hash)
+                       (and (string? secret-hash)
+                            (re-matches #"[A-Za-z0-9_-]{43}" secret-hash))))
+        (invalid! "invalid_client"))
+      (cond-> (assoc metadata :client_id client-id)
+        secret-hash (assoc :secret-hash secret-hash)))
+    (catch Exception _
+      (invalid! "invalid_client"))))
+
+(defonce ^:private metadata-cache
+  (cache/create :expire "1m" :max-size 256))
+
+(defn- fetch-metadata
+  "Fetch a client's own HTTPS metadata document with bounded size and freshness.
+  The shared HTTP client checks SSRF targets and does not follow redirects."
+  [cfg client-id]
+  (let [response (http/req cfg {:method :get :uri client-id :timeout 5000
+                                :headers {"accept" "application/json"}}
+                           {:response-type :input-stream})]
+    (with-open [^InputStream body (:body response)]
+      (when-not (and (= 200 (:status response))
+                     (str/starts-with? (get-in response [:headers "content-type"] "")
+                                       "application/json"))
+        (invalid! "invalid_client"))
+      ;; Bound both bytes read and time spent receiving the response body.
+      (let [reader (future (.readNBytes body 65537))
+            bytes  (deref reader 5000 ::timeout)]
+        (when (= ::timeout bytes)
+          (future-cancel reader)
+          (invalid! "invalid_client"))
+        (when (> (alength ^bytes bytes) 65536)
+          (invalid! "invalid_client"))
+        (let [metadata (json/decode bytes)
+              control  (get-in response [:headers "cache-control"] "")
+              ;; The local cache ceiling never extends the publisher's TTL.
+              ;; Missing cache lifetime is treated as immediately stale.
+              max-age  (some-> (re-find #"(?i)max-age=(\d+)" control) second parse-long)
+              ttl      (if (re-find #"(?i)no-store|no-cache" control)
+                         0
+                         (min 60 (or max-age 0)))]
+          (when-not (= client-id (:client_id metadata))
+            (invalid! "invalid_client"))
+          {:client (assoc (validate-metadata metadata true) :client_id client-id)
+           :expires (+ (System/currentTimeMillis) (* ttl 1000))})))))
+
+(defn resolve-client
+  "Resolve a client ID to validated metadata for authorization or exchange.
+  HTTPS IDs use the document cache; versioned IDs decode locally without
+  PostgreSQL or Redis. Neither form establishes user approval."
+  [cfg client-id]
+  (when-not (and (string? client-id) (<= 1 (count client-id) 2048))
+    (invalid! "invalid_client"))
+  (if-let [url (when (str/starts-with? client-id "https://") (uri client-id))]
+    (do
+      (when-not (and (.getHost url)
+                     (nil? (.getUserInfo url))
+                     (nil? (.getFragment url))
+                     (not (str/blank? (.getPath url))))
+        (invalid! "invalid_client"))
+      (try
+        (let [cached (cache/get metadata-cache client-id)
+              entry  (if (and cached (> (:expires cached) (System/currentTimeMillis)))
+                       cached
+                       (fetch-metadata cfg client-id))]
+          (when-not (identical? cached entry)
+            (cache/invalidate metadata-cache client-id)
+            (cache/get metadata-cache client-id (constantly entry)))
+          (:client entry))
+        (catch Exception _
+          (invalid! "invalid_client"))))
+    (decode-client-id client-id)))

backend/src/app/mcp/oauth_rate.lua

@@ -0,0 +1,5 @@
+-- Copyright (c) KALEIDOS SUBSIDIARY SL
+-- MPL-2.0
+local count = redis.call('INCR', KEYS[1])
+if count == 1 then redis.call('EXPIRE', KEYS[1], ARGV[1]) end
+return {count}

backend/src/app/mcp/oauth_state.clj

@@ -0,0 +1,105 @@
+;; This Source Code Form is subject to the terms of the Mozilla Public
+;; License, v. 2.0. If a copy of the MPL was not distributed with this
+;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
+;;
+;; Copyright (c) KALEIDOS SUBSIDIARY SL
+
+(ns app.mcp.oauth-state
+  "Temporary approval requests and authorization codes in Redis/Valkey.
+  Handles are opaque; tenant, record kind and handle hash form the key.
+  Mutations compare the exact record that the HTTP handler validated."
+  (:require
+   [app.config :as cf]
+   [app.redis :as rds]
+   [app.redis.script :as-alias script]
+   [app.util.json :as json])
+  (:import
+   java.nio.charset.StandardCharsets
+   java.security.MessageDigest
+   java.security.SecureRandom
+   java.util.Base64))
+
+(defn random-secret
+  "32 random bytes encoded as a 43-character, unpadded base64url string."
+  []
+  (let [bytes (byte-array 32)]
+    (.nextBytes (SecureRandom.) bytes)
+    (.encodeToString (.withoutPadding (Base64/getUrlEncoder)) bytes)))
+
+(defn digest
+  "Unpadded base64url SHA-256 digest, also used for PKCE S256 challenges."
+  [^String value]
+  (-> (.digest (MessageDigest/getInstance "SHA-256")
+               (.getBytes value StandardCharsets/UTF_8))
+      (#(.encodeToString (.withoutPadding (Base64/getUrlEncoder)) %))))
+
+(defn secret=
+  [a b]
+  (and (string? a) (string? b)
+       (MessageDigest/isEqual (.getBytes ^String a StandardCharsets/UTF_8)
+                              (.getBytes ^String b StandardCharsets/UTF_8))))
+
+(defn- state-key
+  [kind handle]
+  (str "penpot:" (cf/get :tenant) ":mcp-oauth:" (name kind) ":" (digest handle)))
+
+(defn create!
+  "Store a new record for ttl seconds and return its random handle.
+  The handle is returned only after Redis confirms SET NX EX succeeded."
+  [cfg kind value ttl]
+  (let [handle (random-secret)
+        result (rds/run! cfg
+                         (fn [{::rds/keys [conn]}]
+                           (rds/set conn (state-key kind handle)
+                                    (json/encode-str value) {:ex ttl :nx true})))]
+    (when-not (= "OK" result)
+      (throw (ex-info "Unable to store OAuth state" {:oauth-error "temporarily_unavailable"})))
+    handle))
+
+(defn read!
+  "Read an unexpired record, or nil for a missing/malformed handle.
+  Return both decoded :value for validation and original :raw for atomic
+  comparison; serializing the decoded map again is not an equivalent check."
+  [cfg kind handle]
+  (when (and (string? handle) (re-matches #"[A-Za-z0-9_-]{43}" handle))
+    (when-let [raw (rds/run! cfg (fn [{::rds/keys [conn]}]
+                                   (rds/get conn (state-key kind handle))))]
+      {:raw raw :value (json/decode raw)})))
+
+(defn- compare-and-update!
+  "Atomically change only the exact record the caller read and validated.
+  Lua deletes for nil replacement, or replaces while preserving the TTL.
+  False means the record expired or another request changed/consumed it."
+  [cfg kind handle record replacement]
+  (let [result (rds/run! cfg
+                         (fn [{::rds/keys [conn]}]
+                           (rds/eval conn
+                                     {::script/name ::compare-and-update
+                                      ::script/path "app/mcp/oauth_state.lua"
+                                      ::script/keys [(state-key kind handle)]
+                                      ::script/vals [(:raw record) (or replacement "")]})))]
+    (= [1] result)))
+
+(defn consume!
+  "Delete a validated record once; true only for the request that consumes it."
+  [cfg kind handle record]
+  (compare-and-update! cfg kind handle record nil))
+
+(defn replace!
+  "Replace a validated record without extending its original expiration."
+  [cfg kind handle record value]
+  (compare-and-update! cfg kind handle record (json/encode-str value)))
+
+(defn allow-request!
+  "Limit requests per endpoint kind, tenant and remote address.
+  The one-minute window begins with the first request; subsequent requests
+  increment the count without extending that window."
+  [cfg kind remote-address limit]
+  (let [[count] (rds/run! cfg
+                          (fn [{::rds/keys [conn]}]
+                            (rds/eval conn
+                                      {::script/name ::rate
+                                       ::script/path "app/mcp/oauth_rate.lua"
+                                       ::script/keys [(state-key kind (or remote-address "unknown"))]
+                                       ::script/vals [60]})))]
+    (and (number? count) (<= count limit))))

backend/src/app/mcp/oauth_state.lua

@@ -0,0 +1,11 @@
+-- Copyright (c) KALEIDOS SUBSIDIARY SL
+-- MPL-2.0
+if redis.call('GET', KEYS[1]) ~= ARGV[1] then
+  return {0}
+end
+if ARGV[2] == '' then
+  redis.call('DEL', KEYS[1])
+else
+  redis.call('SET', KEYS[1], ARGV[2], 'KEEPTTL')
+end
+return {1}

backend/src/app/metrics.clj

@@ -7,6 +7,7 @@
 (ns app.metrics
   (:refer-clojure :exclude [run!])
   (:require
+   [app.common.exceptions :as ex]
    [app.common.logging :as l]
    [app.common.schema :as sm]
    [app.metrics.definition :as-alias mdef]
@@ -179,6 +180,25 @@
       (swap! warned-hints conj hint)
       (l/wrn :hint hint :cause cause))))
 
+(defn instance
+  "Resolve the metrics instance from a cfg; raises when it is absent.
+
+  The shared resolver: a module that records metrics takes a cfg and
+  calls this, so no caller has to know where the instance lives or
+  capture it beforehand. That matters most inside an
+  `app.db/after-commit` callback, which closes over whatever it was
+  given and cannot read the cfg again later.
+
+  Raises on purpose. A cfg without metrics is a wiring bug, and it must
+  surface where the metric is recorded rather than disappear: a caller
+  that resolved this at the top of a write and dropped the value is
+  precisely the pattern this replaces."
+  [cfg]
+  (or (::metrics cfg)
+      (ex/raise :type :assertion
+                :code :missing-metrics
+                :hint "missing ::mtx/metrics on the provided cfg")))
+
 (defn run!
   "Record a metric.
 

backend/src/app/migrations.clj

@@ -507,6 +507,9 @@
    {:name "0154-add-upload-session-chunk-table"
     :fn (mg/resource "app/migrations/sql/0154-add-upload-session-chunk-table.sql")}
 
+   {:name "0154-add-job-table"
+    :fn (mg/resource "app/migrations/sql/0154-add-job-table.sql")}
+
    {:name "0155-normalize-storage-object-metadata"
     :fn (mg/resource "app/migrations/sql/0155-normalize-storage-object-metadata.sql")}
 
@@ -528,6 +531,9 @@
    {:name "0156-reset-pg-storage-defaults"
     :fn (mg/resource "app/migrations/sql/0156-reset-pg-storage-defaults.sql")}
 
+   {:name "0157-add-audit-log-pending-table"
+    :fn (mg/resource "app/migrations/sql/0157-add-audit-log-pending-table.sql")}
+
    {:name "0157-migrate-legacy-file-data"
     :fn (mg/resource "app/migrations/sql/0157-migrate-legacy-file-data.sql")}
 

backend/src/app/migrations/sql/0154-add-job-table.sql

@@ -0,0 +1,150 @@
+-- Migration: Add unified `job` table.
+--
+-- A single row per unit of work, replacing the generic `task` table as the
+-- durable substrate for backend jobs. Columns combine the dispatch/lifecycle
+-- fields of `task` with optional user-facing ledger fields (borrowed from the
+-- planned export_job/import_job tables, which this migration supersedes).
+--
+-- Job kinds are identified by `name` (the job-def registry key); user jobs are
+-- marked by `profile_id IS NOT NULL` and expirable jobs by `expires_at IS NOT
+-- NULL`. There is no `modified_at` trigger: the application code updates it on
+-- every UPDATE (used for the unified lease/orphan detection).
+--
+-- `params` carries the business payload (plain JSON) of the job. Progress is
+-- not a column: it is an append-only `job_event` row, so a job keeps its whole
+-- progress history instead of only the last value.
+--
+-- `tenant` is the instance the job belongs to and `queue` is its bare queue
+-- name. They are separate columns because several instances can share one
+-- database and the tenant is what keeps them from claiming each other's work:
+-- as an equality the filter is something an index can serve, where a
+-- `<tenant>:<queue>` prefix could only be matched with a LIKE over text. Every
+-- query that acts on one instance's work (dispatch, orphan sweep, reschedule,
+-- cron no-overlap check, submit dedupe) filters on `tenant` for that reason.
+-- The prefix still exists, but only in the Redis key the queue's payloads
+-- travel through, composed in one place by `app.worker/queue-key`.
+--
+-- The legacy `task` table stays in place (dormant) on purpose until the
+-- next version, so PRE (legacy) and HOURLY (jobs) can run in parallel on
+-- the same database and the deploy can roll back. During that window
+-- each table is cleaned by its own version: `task` rows by the parallel
+-- legacy `tasks-gc`, `job` rows by the new `:jobs-gc`.
+
+CREATE TABLE job (
+    id           uuid PRIMARY KEY DEFAULT gen_random_uuid(),
+    name         text NOT NULL,
+    tenant       text NOT NULL,
+    queue        text NOT NULL,
+    label        text,
+    priority     int NOT NULL DEFAULT 100,
+    scheduled_at timestamptz NOT NULL DEFAULT now(),
+    retry_num    int NOT NULL DEFAULT 0,
+    max_retries  int NOT NULL DEFAULT 3,
+    status       text NOT NULL DEFAULT 'new'
+                 CHECK (status IN ('new', 'scheduled', 'running', 'retry',
+                                   'completed', 'failed', 'cancelled',
+                                   'aborted')),
+    created_at   timestamptz NOT NULL DEFAULT now(),
+    modified_at  timestamptz NOT NULL DEFAULT now(),
+    started_at   timestamptz,
+    completed_at timestamptz,
+    params       jsonb NOT NULL DEFAULT '{}',
+
+    -- optional ledger columns (user-facing jobs)
+    profile_id   uuid NULL REFERENCES profile(id) ON DELETE NO ACTION DEFERRABLE,
+    error        jsonb,
+    result       jsonb,
+    resource_id  uuid NULL REFERENCES storage_object(id) ON DELETE SET NULL DEFERRABLE,
+    expires_at   timestamptz
+);
+
+-- Dispatcher claim index: the worker selects due `new`/`retry` rows of
+-- its own tenant, orders them by priority and scheduled time, and locks
+-- the batch with SKIP LOCKED. The partial predicate keeps terminal rows
+-- out and `status` needs no key column for that reason; `tenant` leads
+-- because it is an equality the index can answer, and the leading
+-- columns then carry the ORDER BY unchanged. `queue` does not lead: the
+-- dispatcher claims every queue of its tenant and groups them when
+-- pushing, so filtering on one queue would be wrong.
+CREATE INDEX job__dispatcher__idx
+    ON job (tenant, priority DESC, scheduled_at)
+    WHERE status IN ('new', 'retry');
+
+-- Dispatcher orphan sweep: the worker finds the `running` rows of its
+-- own tenant whose modified_at is older than the lease and marks them
+-- `aborted` (system-side terminal, never retried).
+CREATE INDEX job__orphan__idx
+    ON job (tenant, modified_at)
+    WHERE status = 'running';
+
+-- User-facing job ledger index: it supports filtering by profile and
+-- ordering a user's jobs by newest first. No current production query
+-- uses it yet; it is kept for the user-facing ledger path.
+CREATE INDEX job__profile__idx
+    ON job (profile_id, created_at DESC)
+    WHERE profile_id IS NOT NULL;
+
+-- Cron no-overlap check and submit dedupe: both look for jobs of one
+-- tenant with the same name and queue and label that have not started
+-- yet, so one index serves the two queries.
+CREATE INDEX job__name_label__idx
+    ON job (tenant, name, queue, label)
+    WHERE status IN ('new', 'scheduled', 'running', 'retry');
+
+-- Storage GC reference check: it answers whether any job still points
+-- to a candidate storage object. Rows without a resource are excluded.
+CREATE INDEX job__resource__idx
+    ON job (resource_id)
+    WHERE resource_id IS NOT NULL;
+
+-- Dispatcher recovery: it finds the `scheduled` jobs of its own tenant
+-- that were pushed to Redis but not claimed within the recovery window.
+CREATE INDEX job__scheduled__idx
+    ON job (tenant, scheduled_at)
+    WHERE status = 'scheduled';
+
+-- Jobs GC expiration: it scans jobs past expires_at while excluding
+-- running and retry jobs from the expiration delete.
+CREATE INDEX job__expires__idx
+    ON job (expires_at)
+    WHERE expires_at IS NOT NULL;
+
+-- Jobs GC retention: it finds old internal terminal jobs with no
+-- profile, which are eligible for removal after the retention delay.
+CREATE INDEX job__retention__idx
+    ON job (modified_at)
+    WHERE status IN ('completed', 'failed', 'cancelled', 'aborted')
+      AND profile_id IS NULL;
+
+-- Append-only job event log: `start`, `progress`, `retry` and `end` rows in
+-- insertion order. This is the only source of truth for progress; there is no
+-- Redis key and no mutable progress column. The foreign key cascades so
+-- deleting a job removes its whole history, and it is DEFERRABLE like every
+-- other new reference of the substrate.
+CREATE TABLE job_event (
+    id         bigserial PRIMARY KEY,
+    job_id     uuid NOT NULL REFERENCES job(id) ON DELETE CASCADE DEFERRABLE,
+    kind       text NOT NULL
+               CHECK (kind IN ('start', 'progress', 'retry', 'end')),
+    payload    jsonb NOT NULL DEFAULT '{}',
+    created_at timestamptz NOT NULL DEFAULT now()
+);
+
+-- Event history read: all events of a job, and the last progress report
+-- (`WHERE job_id = ? AND kind = 'progress' ORDER BY created_at DESC LIMIT 1`).
+CREATE INDEX job_event__job_kind_created_idx
+    ON job_event (job_id, kind, created_at DESC, id DESC);
+
+-- Deletion protection, like the domain tables (`team`, `profile`, the
+-- file thumbnail tables). The only intended deleter of `job` rows is the
+-- jobs GC, which also marks the referenced storage objects as touched in
+-- the same transaction; a stray `DELETE FROM job` would drop the row
+-- (and cascade `job_event`) without touching the object, orphaning it
+-- forever. The GC and the submit dedupe disable the guard inside their
+-- transaction with `SET LOCAL rules.deletion_protection TO off`, the
+-- same escape hatch used by `app.tasks.objects-gc`.
+CREATE OR REPLACE TRIGGER deletion_protection__tgr
+BEFORE DELETE ON job FOR EACH STATEMENT
+  WHEN ((current_setting('rules.deletion_protection', true) IN ('on', '')) OR
+        (current_setting('rules.deletion_protection', true) IS NULL))
+  EXECUTE PROCEDURE raise_deletion_protection();

backend/src/app/migrations/sql/0157-add-audit-log-pending-table.sql

@@ -0,0 +1,39 @@
+-- Work queue for the audit log consumers: one row per (record,
+-- consumer) pair still pending. Inserting a record creates its
+-- pendings; each consumer deletes its own on completion, and whoever
+-- deletes the last pending also deletes the record. Consumer ids are
+-- fixed in code and never reused: 1 = nexus, 2 = admin console.
+
+CREATE TABLE audit_log_pending (
+  -- DEFERRABLE so the test-suite reset (which deletes table by table
+  -- with SET CONSTRAINTS ALL DEFERRED) keeps working; the guard still
+  -- fires when a record with pendings is deleted. Plain DEFERRABLE
+  -- (initially immediate) is enough: every deleter removes pendings
+  -- before rows, so the check passes at each statement.
+  audit_log_id uuid     NOT NULL REFERENCES audit_log(id) DEFERRABLE,
+  consumer     smallint NOT NULL CHECK (consumer BETWEEN 1 AND 20),
+  PRIMARY KEY (consumer, audit_log_id));
+
+-- Lets the final delete ask "is anything still pending for this
+-- record?" without scanning the whole queue.
+CREATE INDEX ON audit_log_pending (audit_log_id);
+
+-- NOTE: no backfill on purpose. Rows predating this queue (sent or
+-- not) get no pendings, so the orphan sweep deletes them unsent.
+-- Losing the unarchived ones at upgrade time is accepted: this
+-- table is a minutes/hours buffer, not a store.
+
+-- The pending queue replaces the archived_at mark: whoever deletes
+-- the last pending deletes the record, so the flag column and the
+-- indexes that served the old mark-and-sweep scans go away.
+DROP INDEX IF EXISTS audit_log__created_at__idx;
+DROP INDEX IF EXISTS audit_log__archived_at__idx;
+DROP INDEX IF EXISTS audit_log_id_archived_at_idx;
+ALTER TABLE audit_log DROP COLUMN IF EXISTS archived_at;
+
+-- Serves the consumers' chronological scan (ORDER BY created_at, id):
+-- one total order for every consumer, so concurrent drains cannot
+-- deadlock on row locks. Covers the whole ORDER BY (bulk ingest
+-- shares request-at constantly, so ties are the norm).
+CREATE INDEX IF NOT EXISTS audit_log__created_at__idx
+    ON audit_log (created_at ASC, id);

backend/src/app/nitrate.clj

@@ -7,6 +7,7 @@
 (ns app.nitrate
   "Module that make calls to the external nitrate aplication"
   (:require
+   [app.common.data :as d]
    [app.common.data.macros :as dm]
    [app.common.exceptions :as ex]
    [app.common.json :as json]
@@ -16,7 +17,6 @@
    [app.common.time :as ct]
    [app.common.types.organization :as cto
     :refer [schema:nitrate-sso]]
-   [app.common.uri :as u]
    [app.config :as cf]
    [app.http.client :as http]
    [app.http.session :as session]
@@ -31,31 +31,17 @@
 ;; HELPERS
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 
-(defn- join-path-segments
-  "Build a single relative path from Nitrate URI segments, normalizing slashes."
-  [segments]
-  (let [path (->> segments (map str) (str/join "/"))]
-    (->> (str/split path #"/")
-         (remove str/blank?)
-         (str/join "/"))))
-
-(defn- join-base-uri
-  "Join path segments to a base URI."
-  [base-uri & segments]
-  (u/join (u/ensure-path-slash base-uri)
-          (join-path-segments segments)))
-
 (defn- generate-nitrate-uri
   "Joins relative path segments to the Nitrate backend URI.
    Segments must not start with `/`"
   [& segments]
-  (apply join-base-uri (cf/get :admin-console-uri) segments))
+  (apply cf/join-uri (cf/get :admin-console-uri) segments))
 
 (defn- generate-public-uri
   "Joins relative path segments to the public backend URI.
    Segments must not start with `/`"
   [& segments]
-  (apply join-base-uri (cf/get :public-uri) segments))
+  (apply cf/get-public-uri segments))
 
 (defn- request-builder
   [cfg method uri shared-key profile-id request-params]
@@ -279,7 +265,7 @@
   (request-to-nitrate cfg :get
                       (generate-nitrate-uri
                        "api/organizations/"
-                       organization-id
+                       (str organization-id "/")
                        "members/"
                        profile-id)
                       schema:profile-organization params))
@@ -289,7 +275,7 @@
   (request-to-nitrate cfg :get
                       (generate-nitrate-uri
                        "api/teams/"
-                       team-id
+                       (str team-id "/")
                        "users/"
                        profile-id)
                       schema:profile-organization params))
@@ -299,7 +285,7 @@
   (request-to-nitrate cfg :get
                       (generate-nitrate-uri
                        "api/organizations/"
-                       organization-id
+                       (str organization-id "/")
                        "summary")
                       schema:organization-summary params))
 
@@ -308,7 +294,7 @@
   (request-to-nitrate cfg :get
                       (generate-nitrate-uri
                        "api/users/"
-                       profile-id
+                       (str profile-id "/")
                        "owned-organizations")
                       [:vector schema:organization-summary]
                       params))
@@ -328,7 +314,7 @@
   (let [organizations (request-to-nitrate cfg :get
                                           (generate-nitrate-uri
                                            "api/users/"
-                                           profile-id
+                                           (str profile-id "/")
                                            "owned-organizations-summary")
                                           [:vector schema:organization-summary-counts]
                                           params)]
@@ -343,7 +329,7 @@
   (request-to-nitrate cfg :post
                       (generate-nitrate-uri
                        "api/users/"
-                       profile-id
+                       (str profile-id "/")
                        "cleanup-after-deletion")
                       nil params))
 
@@ -354,7 +340,7 @@
         team (request-to-nitrate cfg :post
                                  (generate-nitrate-uri
                                   "api/organizations/"
-                                  organization-id
+                                  (str organization-id "/")
                                   "add-team")
                                  cto/schema:team-with-organization params)
         custom-photo (when-let [logo-id (dm/get-in team [:organization :logo-id])]
@@ -371,7 +357,7 @@
     (request-to-nitrate cfg :post
                         (generate-nitrate-uri
                          "api/organizations/"
-                         organization-id
+                         (str organization-id "/")
                          "add-user")
                         schema:profile-organization params)))
 
@@ -386,7 +372,7 @@
     (request-to-nitrate cfg :post
                         (generate-nitrate-uri
                          "api/organizations/"
-                         organization-id
+                         (str organization-id "/")
                          "remove-user")
                         nil params)))
 
@@ -396,7 +382,7 @@
     (request-to-nitrate cfg :post
                         (generate-nitrate-uri
                          "api/organizations/"
-                         organization-id
+                         (str organization-id "/")
                          "remove-team")
                         nil params)))
 
@@ -453,12 +439,13 @@
   (request-to-nitrate cfg :get
                       (generate-nitrate-uri
                        "api/organizations/"
-                       organization-id
+                       (str organization-id "/")
                        "permissions")
                       [:map
                        [:organization-id ::sm/uuid]
                        [:owner-id ::sm/uuid]
-                       [:permissions [:map-of :keyword :string]]]
+                       [:permissions [:map-of :keyword :string]]
+                       [:restrictions-met {:optional true} [:maybe [:map-of :keyword :boolean]]]]
                       params))
 
 (defn- get-organization-sso-api
@@ -467,7 +454,7 @@
   (request-to-nitrate cfg :get
                       (generate-nitrate-uri
                        "api/organizations/"
-                       organization-id
+                       (str organization-id "/")
                        "sso")
                       schema:nitrate-sso
                       params))
@@ -475,7 +462,7 @@
 (defn- get-organization-sso-by-team-api
   [cfg {:keys [team-id] :as params}]
   (request-to-nitrate cfg :get
-                      (generate-nitrate-uri "api/teams/" team-id "sso")
+                      (generate-nitrate-uri "api/teams/" (str team-id "/") "sso")
                       schema:nitrate-sso
                       params))
 
@@ -484,7 +471,7 @@
   (request-to-nitrate cfg :get
                       (generate-nitrate-uri
                        "api/organizations/"
-                       organization-id
+                       (str organization-id "/")
                        "members-list")
                       [:vector ::sm/uuid]
                       params))
@@ -496,6 +483,29 @@
                       schema:redeem-result
                       (assoc params :throw-on-error? true)))
 
+(def ^:private audit-event-keys
+  [:id :name :type :profile-id :profile-email :profile-name
+   :props :context :created-at :tracked-at :source :ip-addr])
+
+(defn- ->audit-event
+  [event]
+  (-> (select-keys event audit-event-keys)
+      (d/without-nils)))
+
+(defn- ingest-audit-log-api
+  "POST a batch of audit events to Admin Console `/api/audit-log`.
+  Expects HTTP 204; returns nil on success. 4xx raises when
+  `:throw-on-error?` is set."
+  [cfg {:keys [events] :as params}]
+  (let [request-params {:events (mapv ->audit-event events)}
+        params (assoc params
+                      :request-params request-params
+                      :throw-on-error? true)]
+    (request-to-nitrate cfg :post
+                        (generate-nitrate-uri "api/audit-log")
+                        nil
+                        params)))
+
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;; INITIALIZATION
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
@@ -505,28 +515,29 @@
   (when (contains? cf/flags :admin-console)
     (when (nil? (cf/get :admin-console-uri))
       (l/warn :hint "admin console is not configured; nitrate calls will fail until the setup is complete"))
-    {:get-team-organization                 (partial get-team-organization-api cfg)
-     :get-teams-organizations               (partial get-teams-organizations-api cfg)
-     :set-team-organization                 (partial set-team-organization-api cfg)
-     :get-organization-membership           (partial get-organization-membership-api cfg)
-     :get-organization-membership-by-team   (partial get-organization-membership-by-team-api cfg)
-     :get-organization-summary              (partial get-organization-summary-api cfg)
-     :get-owned-organizations               (partial get-owned-organizations-api cfg)
-     :get-owned-organizations-summary       (partial get-owned-organizations-summary-api cfg)
-     :get-organization-members              (partial get-organization-members-api cfg)
-     :cleanup-deleted-penpot-user  (partial cleanup-deleted-penpot-user-api cfg)
-     :add-profile-to-organization           (partial add-profile-to-organization-api cfg)
-     :remove-profile-from-organization      (partial remove-profile-from-organization-api cfg)
-     :get-organization-permissions          (partial get-organization-permissions-api cfg)
-     :get-organization-sso-by-team          (partial get-organization-sso-by-team-api cfg)
-     :get-organization-sso                  (partial get-organization-sso-api cfg)
-     :delete-team                  (partial delete-team-api cfg)
-     :remove-team-from-organization         (partial remove-team-from-organization-api cfg)
-     :get-subscription             (partial get-subscription-api cfg)
-     :get-subscription-warning     (partial get-subscription-warning-api cfg)
-     :connectivity                 (partial get-connectivity-api cfg)
-     :get-identity                 (partial get-identity-api cfg)
-     :redeem-activation-code       (partial redeem-activation-code-api cfg)}))
+    {:get-team-organization               (partial get-team-organization-api cfg)
+     :get-teams-organizations             (partial get-teams-organizations-api cfg)
+     :set-team-organization               (partial set-team-organization-api cfg)
+     :get-organization-membership         (partial get-organization-membership-api cfg)
+     :get-organization-membership-by-team (partial get-organization-membership-by-team-api cfg)
+     :get-organization-summary            (partial get-organization-summary-api cfg)
+     :get-owned-organizations             (partial get-owned-organizations-api cfg)
+     :get-owned-organizations-summary     (partial get-owned-organizations-summary-api cfg)
+     :get-organization-members            (partial get-organization-members-api cfg)
+     :cleanup-deleted-penpot-user         (partial cleanup-deleted-penpot-user-api cfg)
+     :add-profile-to-organization         (partial add-profile-to-organization-api cfg)
+     :remove-profile-from-organization    (partial remove-profile-from-organization-api cfg)
+     :get-organization-permissions        (partial get-organization-permissions-api cfg)
+     :get-organization-sso-by-team        (partial get-organization-sso-by-team-api cfg)
+     :get-organization-sso                (partial get-organization-sso-api cfg)
+     :delete-team                         (partial delete-team-api cfg)
+     :remove-team-from-organization       (partial remove-team-from-organization-api cfg)
+     :get-subscription                    (partial get-subscription-api cfg)
+     :get-subscription-warning            (partial get-subscription-warning-api cfg)
+     :connectivity                        (partial get-connectivity-api cfg)
+     :get-identity                        (partial get-identity-api cfg)
+     :redeem-activation-code              (partial redeem-activation-code-api cfg)
+     :ingest-audit-log                    (partial ingest-audit-log-api cfg)}))
 
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;; UTILS
@@ -642,6 +653,35 @@
                    :cause cause)
           team)))))
 
+(defn add-organization-info-to-team-strict
+  "Like `add-organization-info-to-team`, but fails closed for authorization
+  checks: the team is returned unchanged only when Nitrate answers 404 (the
+  team belongs to no organization). Any other failure, including a 4xx or an
+  invalid response body, raises instead of looking like a team without an
+  organization."
+  [cfg team params]
+  (let [params (assoc (or params {})
+                      :team-id (:id team)
+                      :throw-on-error? true)
+        team-with-organization
+        (try
+          (call cfg :get-team-organization params)
+          (catch Throwable cause
+            (let [{:keys [type status]} (ex-data cause)]
+              (if (and (= :nitrate-http-error type) (= 404 status))
+                ::no-organization
+                (throw cause)))))]
+    (cond
+      (= ::no-organization team-with-organization)
+      team
+
+      (nil? team-with-organization)
+      (ex/raise :type :nitrate-unavailable
+                :hint "nitrate did not return a valid team organization response")
+
+      :else
+      (apply-organization-info-to-team team team-with-organization))))
+
 (defn add-organization-info-to-teams
   "Enriches teams with organization information using one batched Nitrate request.
   Teams absent from the Nitrate response are returned unchanged.

backend/src/app/redis.clj

@@ -78,6 +78,7 @@
   (-hget [_ key field])
   (-set [_ key val args])
   (-del [_ key-or-keys])
+  (-expire [_ key seconds])
   (-ping [_]))
 
 (defprotocol IPubSubConnection
@@ -234,6 +235,10 @@
     (let [keys (into-array String keys)]
       (.del cmd ^String/1 keys)))
 
+  (-expire [_ key seconds]
+    (assert (string? key) "key expected to be string")
+    (.expire cmd ^String key ^long seconds))
+
   (-ping [_]
     (.ping cmd))
 
@@ -511,6 +516,11 @@
   [cause]
   (instance? RedisCommandTimeoutException cause))
 
+(defn expire
+  [conn key seconds]
+  (assert (string? key) "key must be string instance")
+  (-expire conn key seconds))
+
 (defn exception?
   [cause]
   (instance? RedisException cause))

backend/src/app/rpc.clj

@@ -13,7 +13,6 @@
    [app.common.schema :as sm]
    [app.common.spec :as us]
    [app.common.time :as ct]
-   [app.common.uri :as u]
    [app.common.uuid :as uuid]
    [app.config :as cf]
    [app.db :as db]
@@ -125,7 +124,8 @@
                              (cond-> (and (not (uuid? session-pid))
                                           (uuid? token-pid))
                                (-> (assoc ::auth-type :token)
-                                   (assoc ::token-perms (set (::actoken/perms request #{})))))
+                                   (assoc ::token-perms (set (::actoken/perms request #{})))
+                                   (assoc ::token-type (::actoken/type request))))
                              (cond-> key-id
                                (assoc ::auth-key-id key-id))
                              ;; Server-side only: request bodies cannot forge
@@ -172,10 +172,12 @@
   [_ f mdata]
   (let [required-auth?      (::auth mdata true)
         required-auth-type  (::auth-type mdata)
+        required-token-type (::token-type mdata)
         required-perms      (into #{} (::perms mdata))]
     (fn [cfg params]
       (let [profile-id  (::profile-id params)
             auth-type   (::auth-type params)
+            token-type  (::token-type params)
             token-perms (set (::token-perms params #{}))]
         (cond
           (and required-auth? (not (uuid? profile-id)))
@@ -183,8 +185,13 @@
                     :code :authentication-required
                     :hint "authentication required for this endpoint")
 
-          (and (= required-auth-type :token)
-               (not= auth-type :token))
+          (or (and (= required-auth-type :token)
+                   (not= auth-type :token))
+              (and required-token-type
+                   (or (not= auth-type :token)
+                       (not= required-token-type token-type)))
+              (and (= token-type "mcp")
+                   (not= required-token-type "mcp")))
           (ex/raise :type :authorization
                     :code :token-auth-required
                     :hint "access token authentication required for this endpoint")
@@ -359,6 +366,28 @@
     (wrap-params-validation cfg $ mdata)
     (wrap-authentication cfg $ mdata)))
 
+(defn- wrap-shared-key-authorization
+  "Authorizes a shared-key caller against the method's `::allowed-shared-keys`.
+
+  A management method declares the shared-key ids allowed to call it with
+  `::allowed-shared-keys`. When the metadata is absent any configured shared key is
+  accepted, preserving the previous behavior. When it is present and empty no
+  key is accepted."
+  [_ f mdata]
+  (if (contains? mdata ::allowed-shared-keys)
+    (let [allowed-keys (::allowed-shared-keys mdata)]
+      (fn [cfg params]
+        (if (contains? allowed-keys (::auth-key-id params))
+          (f cfg params)
+          (do
+            (l/wrn :hint "shared key is not allowed for this method"
+                   :handler-name (::handler-name params)
+                   :key-id (::auth-key-id params))
+            (ex/raise :type :authorization
+                      :code :shared-key-not-allowed
+                      :hint "shared key is not allowed for this method")))))
+    f))
+
 (defn- wrap-management
   [cfg f mdata]
   (as-> f $
@@ -369,7 +398,10 @@
     (wrap-audit cfg $ mdata)
     (wrap-nitrate-sso cfg $ mdata)
     (wrap-params-validation cfg $ mdata)
-    (wrap-authentication cfg $ mdata)))
+    (wrap-authentication cfg $ mdata)
+    ;; Outermost on purpose: a denied key never opens a transaction,
+    ;; validates params or runs the method body.
+    (wrap-shared-key-authorization cfg $ mdata)))
 
 (defn- process-method
   [cfg wrap-fn [f mdata]]
@@ -401,6 +433,7 @@
           'app.rpc.commands.files-update
           'app.rpc.commands.files-snapshot
           'app.rpc.commands.files-thumbnails
+          'app.rpc.commands.jobs
           'app.rpc.commands.ldap
           'app.rpc.commands.management
           'app.rpc.commands.media
@@ -448,6 +481,7 @@
   [cfg]
   (let [cfg  (assoc cfg ::module "management" ::type "command" ::metrics-id :rpc-management-timing)
         mods (cond->> (list 'app.rpc.management.exporter
+                            'app.rpc.management.jobs
                             'app.rpc.management.files)
                (contains? cf/flags :admin-console)
                (cons 'app.rpc.management.nitrate))]
@@ -504,42 +538,40 @@
 
 (defmethod ig/init-key ::routes
   [_ {:keys [::methods ::management-methods ::setup/shared-keys] :as cfg}]
+  ["/api"
+   ["/management"
+    ["/methods/:method-name"
+     {:middleware [[mw/shared-key-auth shared-keys]
+                   [session/authz cfg]]
+      :handler (make-rpc-handler management-methods)}]
 
-  (let [public-uri (cf/get :public-uri)]
-    ["/api"
-     ["/management"
-      ["/methods/:method-name"
-       {:middleware [[mw/shared-key-auth shared-keys]
-                     [session/authz cfg]]
-        :handler (make-rpc-handler management-methods)}]
+    (doc/routes :methods management-methods
+                :label "management"
+                :base-uri (cf/get-public-uri "api/management")
+                :description "MANAGEMENT API")]
 
-      (doc/routes :methods management-methods
-                  :label "management"
-                  :base-uri (u/join public-uri "/api/management")
-                  :description "MANAGEMENT API")]
+   ["/main"
+    ["/methods/:method-name"
+     {:middleware [[mw/cors]
+                   [sec/client-header-check]
+                   [session/authz cfg]
+                   [actoken/authz (assoc cfg ::actoken/allow-mcp true)]]
+      :handler (make-rpc-handler methods)}]
 
-     ["/main"
-      ["/methods/:method-name"
-       {:middleware [[mw/cors]
-                     [sec/client-header-check]
-                     [session/authz cfg]
-                     [actoken/authz cfg]]
-        :handler (make-rpc-handler methods)}]
+    (doc/routes :methods methods
+                :label "main"
+                :base-uri (cf/get-public-uri "api/main")
+                :description "MAIN API")]
 
-      (doc/routes :methods methods
-                  :label "main"
-                  :base-uri (u/join public-uri "/api/main")
-                  :description "MAIN API")]
+   ;; BACKWARD COMPATIBILITY
+   ["/_doc" {:handler (redirect (cf/get-public-uri "api/main/doc"))}]
+   ["/doc" {:handler (redirect (cf/get-public-uri "api/main/doc"))}]
+   ["/openapi" {:handler (redirect (cf/get-public-uri "api/main/doc/openapi"))}]
+   ["/openapi.join" {:handler (redirect (cf/get-public-uri "api/main/doc/openapi.json"))}]
 
-     ;; BACKWARD COMPATIBILITY
-     ["/_doc" {:handler (redirect (u/join public-uri "/api/main/doc"))}]
-     ["/doc" {:handler (redirect (u/join public-uri "/api/main/doc"))}]
-     ["/openapi" {:handler (redirect (u/join public-uri "/api/main/doc/openapi"))}]
-     ["/openapi.join" {:handler (redirect (u/join public-uri "/api/main/doc/openapi.json"))}]
-
-     ["/rpc/command/:method-name"
-      {:middleware [[mw/cors]
-                    [sec/client-header-check]
-                    [session/authz cfg]
-                    [actoken/authz cfg]]
-       :handler (make-rpc-handler methods)}]]))
+   ["/rpc/command/:method-name"
+    {:middleware [[mw/cors]
+                  [sec/client-header-check]
+                  [session/authz cfg]
+                  [actoken/authz (assoc cfg ::actoken/allow-mcp true)]]
+     :handler (make-rpc-handler methods)}]])

backend/src/app/rpc/commands/access_token.clj

@@ -6,12 +6,15 @@
 
 (ns app.rpc.commands.access-token
   (:require
+   [app.common.exceptions :as ex]
    [app.common.schema :as sm]
    [app.common.time :as ct]
    [app.common.uuid :as uuid]
    [app.db :as db]
+   [app.http.access-token :as actoken]
    [app.main :as-alias main]
    [app.rpc :as-alias rpc]
+   [app.rpc.commands.profile :as profile]
    [app.rpc.doc :as-alias doc]
    [app.rpc.quotes :as quotes]
    [app.setup :as-alias setup]
@@ -28,16 +31,27 @@
       AND id != ?
       AND type = 'mcp'")
 
+(def ^:private sql:get-valid-mcp-token
+  "SELECT * FROM access_token
+    WHERE profile_id = ?
+      AND type = 'mcp'
+      AND (expires_at IS NULL OR expires_at > now())
+    ORDER BY created_at DESC
+    LIMIT 1")
+
 (defn create-access-token
   "Create an access token with empty perms.
 
   Elevated permissions (e.g. error-reports:read) are not assignable via
   the public API; grant them with SQL or `repl:grant-access-token-perm`."
   [{:keys [::db/conn] :as cfg} profile-id name expiration type]
+  ;; Serialize regeneration with OAuth provisioning for the same profile.
+  (when (= type "mcp")
+    (db/get-by-id conn :profile profile-id {::db/for-update true}))
   (let [token-id   (uuid/next)
         expires-at (some-> expiration (ct/in-future))
         created-at (ct/now)
-        token-iss  (if (= type "mcp") "urn:penpot:mcp-token" "access-token")
+        token-iss  (if (= type "mcp") actoken/mcp-token-issuer "access-token")
         token      (tokens/generate cfg {:iss token-iss
                                          :uid profile-id
                                          :iat created-at
@@ -62,6 +76,27 @@
 
     (decode-row token)))
 
+(defn ensure-mcp-access!
+  "Enable MCP and return the profile's unexpired token, creating it if absent.
+  Must run in a transaction. Lock the profile even when no token exists, so
+  concurrent approvals and settings regeneration use the same lock. Enabling
+  MCP must not regenerate a token already held by a connected client; an
+  expired token is replaced because no client can use it, and so is a token
+  issued before MCP tokens had their own issuer, or one this instance cannot
+  decrypt, because the backend rejects both."
+  [{:keys [::db/conn] :as cfg} profile-id]
+  (let [profile (profile/get-profile conn profile-id ::db/for-update true)
+        token   (db/exec-one! conn [sql:get-valid-mcp-token profile-id])
+        ;; A token this instance cannot decrypt is as unusable as an expired one
+        token   (when (= actoken/mcp-token-issuer
+                         (ex/ignoring (:iss (tokens/decode cfg (:token token)))))
+                  token)]
+    (when-not (:mcp-enabled (:props profile))
+      (profile/update-profile-props cfg profile-id {:mcp-enabled true}))
+    (if token
+      (decode-row token)
+      (create-access-token cfg profile-id "MCP" nil "mcp"))))
+
 (defn repl:create-access-token
   [cfg profile-id name expiration]
   (db/tx-run! cfg create-access-token profile-id name expiration))

backend/src/app/rpc/commands/audit.clj

@@ -16,8 +16,10 @@
    [app.db :as db]
    [app.http :as-alias http]
    [app.loggers.audit :as audit]
+   [app.loggers.audit.consumer-task :as consumer]
    [app.loggers.database :as loggers.db]
    [app.loggers.mattermost :as loggers.mm]
+   [app.metrics :as mtx]
    [app.rpc :as-alias rpc]
    [app.rpc.climit :as-alias climit]
    [app.rpc.doc :as-alias doc]
@@ -71,6 +73,9 @@
 (def ^:private xf:map-event-row
   (comp
    (map adjust-timestamp)
+   ;; Each row mints its own id: the full row and its telemetry
+   ;; shadow are distinct records that must not share it.
+   (map #(assoc % :id (uuid/next)))
    (map event->row)))
 
 (defn- prepare-events
@@ -80,8 +85,7 @@
         initiator (audit/get-client-initiator request)
         xform     (comp
                    (map (fn [event]
-                          {:id (uuid/next)
-                           :type (:type event)
+                          {:type (:type event)
                            :name (:name event)
                            :props (:props event)
                            ;; The initiator always comes from the
@@ -113,6 +117,19 @@
               (assoc :source "telemetry:frontend"))))
    (map event->row)))
 
+(defn- record-ingested
+  [metrics kind outcome amount]
+  (when (pos? amount)
+    (mtx/run! metrics :id :audit-ingest-events :labels [kind outcome] :inc amount)))
+
+(defn- insert-rows!
+  [tx rows]
+  (when (seq rows)
+    (db/insert-many! tx :audit-log event-columns rows)
+    ;; Same transaction as the rows: every ingested record is born
+    ;; pending for every enabled consumer (see consumer-task).
+    (consumer/queue-pendings tx (map first rows))))
+
 (defn- handle-events
   [{:keys [::db/pool] :as cfg} params]
   (let [events (prepare-events params)]
@@ -124,19 +141,30 @@
       (run! (partial loggers.db/emit cfg) events)
       (run! (partial loggers.mm/emit cfg) events))
 
-    (when (contains? cf/flags :audit-log)
-      ;; Process and save full audit events when audit-log flag is active
-      (when-let [rows (-> (sequence xf:map-event-row events)
-                          (not-empty))]
-        (db/insert-many! pool :audit-log event-columns rows)))
-
-    (when (contains? cf/flags :telemetry)
-      ;; Store anonymized frontend events so the telemetry task can ship them
-      ;; in batches. Runs independently from the audit-log insert above so
-      ;; both modes can be active simultaneously.
-      (when-let [rows (-> (sequence xf:map-telemetry-event-row events)
-                          (not-empty))]
-        (db/insert-many! pool :audit-log event-columns rows)))))
+    ;; Rows and their pendings commit together in one transaction: a
+    ;; row never outlives its queue. Both faces are best-effort, so a
+    ;; failure drops the whole batch (counted below as failed); the
+    ;; endpoint still answers without breaking the request.
+    (let [metrics     (mtx/instance cfg)
+          full-rows   (if (contains? cf/flags :audit-log)
+                        (into [] xf:map-event-row events)
+                        [])
+          shadow-rows (if (contains? cf/flags :telemetry)
+                        (into [] xf:map-telemetry-event-row events)
+                        [])]
+      (try
+        (db/tx-run! pool
+                    (fn [tx]
+                      (insert-rows! tx full-rows)
+                      (insert-rows! tx shadow-rows)))
+        (record-ingested metrics "full" "inserted" (count full-rows))
+        (record-ingested metrics "shadow" "inserted" (count shadow-rows))
+        (catch Throwable cause
+          ;; Whatever never landed counts as failed per attempted
+          ;; kind; the endpoint logs the cause as before.
+          (record-ingested metrics "full" "failed" (count full-rows))
+          (record-ingested metrics "shadow" "failed" (count shadow-rows))
+          (throw cause))))))
 
 (def ^:private valid-event-types
   #{"action" "identify" "trigger"})

backend/src/app/rpc/commands/auth.clj

@@ -202,11 +202,11 @@
                            (update-password conn))]
       (when profile
         (login-lockout/clear-attempts! cfg (:id profile))
-        (eml/send! {::eml/conn conn
-                    ::eml/factory eml/password-changed
-                    :public-uri (cf/get :public-uri)
-                    :to (:email profile)
-                    :name (:fullname profile)})))
+        (eml/send-template cfg {::eml/reuse-conn true
+                                ::eml/template "password-changed"
+                                :public-uri (cf/get :public-uri)
+                                :to (:email profile)
+                                :name (:fullname profile)})))
 
     nil))
 
@@ -389,12 +389,17 @@
              :cause cause)
       nil)))
 
+(def ^:private default-renderer-new-profiles
+  "Renderer stored on new profiles. Profiles without one fall back to SVG."
+  :wasm)
+
 (defn create-profile
   "Create the profile entry on the database with limited set of input
   attrs (all the other attrs are filled with default values)."
   [{:keys [::db/conn] :as cfg} {:keys [email] :as params}]
   (let [id        (or (:id params) (uuid/next))
         props     (-> (:props params)
+                      (update :renderer d/nilv default-renderer-new-profiles)
                       (merge {:viewed-tutorial? false
                               :viewed-walkthrough? false
                               :nudge {:big 10 :small 1}
@@ -467,7 +472,7 @@
 
 (defn send-email-verification!
   ([cfg profile] (send-email-verification! cfg profile nil))
-  ([{:keys [::db/conn] :as cfg} profile invitation-token]
+  ([cfg profile invitation-token]
    (let [vclaims (cond-> {:iss :verify-email
                           :exp (ct/in-future "72h")
                           :profile-id (:id profile)
@@ -485,13 +490,13 @@
                                   {:iss :profile-identity
                                    :profile-id (:id profile)
                                    :exp (ct/in-future {:days 30})})]
-     (eml/send! {::eml/conn conn
-                 ::eml/factory eml/register
-                 :public-uri (cf/get :public-uri)
-                 :to (:email profile)
-                 :name (:fullname profile)
-                 :token vtoken
-                 :extra-data ptoken}))))
+     (eml/send-template cfg {::eml/reuse-conn true
+                             ::eml/template "register"
+                             :public-uri (cf/get :public-uri)
+                             :to (:email profile)
+                             :name (:fullname profile)
+                             :token vtoken
+                             :extra-data ptoken}))))
 
 (defn register-profile
   [{:keys [::db/conn] :as cfg} {:keys [token] :as params}]
@@ -516,8 +521,9 @@
 
         created?   (-> profile meta :created true?)
 
-        props      (-> (audit/profile->props profile)
-                       (assoc :from-invitation (some? invitation)))]
+        props      (cond-> (audit/profile->props profile)
+                     :always  (assoc :from-invitation (some? invitation))
+                     created? (assoc :renderer-default (-> profile :props :renderer)))]
 
     (cond
       ;; When profile is blocked, we just ignore it and return plain data
@@ -650,18 +656,18 @@
                                           :profile-id id})]
               (assoc profile :token token)))
 
-          (send-email-notification [conn profile]
+          (send-email-notification [profile]
             (let [ptoken (tokens/generate cfg
                                           {:iss :profile-identity
                                            :profile-id (:id profile)
                                            :exp (ct/in-future {:days 30})})]
-              (eml/send! {::eml/conn conn
-                          ::eml/factory eml/password-recovery
-                          :public-uri (cf/get :public-uri)
-                          :to (:email profile)
-                          :token (:token profile)
-                          :name (:fullname profile)
-                          :extra-data ptoken})
+              (eml/send-template cfg {::eml/reuse-conn true
+                                      ::eml/template "password-recovery"
+                                      :public-uri (cf/get :public-uri)
+                                      :to (:email profile)
+                                      :token (:token profile)
+                                      :name (:fullname profile)
+                                      :extra-data ptoken})
               nil))]
 
     (let [profile (->> (profile/clean-email email)
@@ -699,7 +705,7 @@
                       {:id (:id profile)})
           (->> profile
                (create-recovery-token)
-               (send-email-notification conn)))))))
+               (send-email-notification)))))))
 
 (def schema:request-profile-recovery
   [:map {:title "request-profile-recovery"}

backend/src/app/rpc/commands/binfile.clj

@@ -10,7 +10,6 @@
    [app.binfile.common :as bfc]
    [app.binfile.v1 :as bf.v1]
    [app.binfile.v3 :as bf.v3]
-   [app.common.features :as cfeat]
    [app.common.schema :as sm]
    [app.common.time :as ct]
    [app.common.uri :as u]
@@ -47,20 +46,14 @@
    [:embed-assets {:optional true} ::sm/boolean]])
 
 (defn- export-binfile
-  [{:keys [::sto/storage] :as cfg} {:keys [type file-id include-libraries embed-assets]}]
-  (let [output (tmp/tempfile*)
-        ;; Convert legacy boolean flags to unified export-type
-        export-type (cond
-                      (some? type) type
-                      (true? include-libraries) :include-libraries
-                      (true? embed-assets) :merge-libraries
-                      :else :detach-libraries)]
+  [{:keys [::sto/storage] :as cfg} {:keys [file-id] :as params}]
+  (let [output (tmp/tempfile*)]
 
     (try
-      (-> cfg
-          (assoc ::bfc/ids #{file-id})
-          (assoc ::bfc/export-type export-type)
-          (bf.v3/export-files! output))
+      (-> (bfc/export-cfg cfg {:file-ids    #{file-id}
+                               :export-type (bfc/resolve-export-type params)
+                               :output      output})
+          (bf.v3/export-files!))
 
       (let [data   (sto/content output)
             object (sto/put-object! storage
@@ -98,22 +91,13 @@
                         :project-id project-id)
 
         cfg
-        (-> cfg
-            (assoc ::bfc/features (cfeat/get-team-enabled-features cf/flags team))
-            (assoc ::bfc/project-id project-id)
-            (assoc ::bfc/profile-id profile-id)
-            (assoc ::bfc/team-id (:id team))
-            (assoc ::bfc/name name)
-            (assoc ::bfc/import-max-binary-entry-size (cf/get :binfile-import-max-binary-entry-size))
-            (assoc ::bfc/import-max-text-entry-size (cf/get :binfile-import-max-text-entry-size))
-            (assoc ::bfc/import-max-text-total-size (cf/get :binfile-import-max-text-total-size))
-            (assoc ::bfc/import-max-zip-entries (cf/get :binfile-import-max-zip-entries)))
+        (bfc/import-cfg cfg {:profile-id profile-id
+                             :project-id project-id
+                             :team team
+                             :name name
+                             :input (:path file)})
 
-        input-path (:path file)
-        owned?     (some? upload-id)
-
-        cfg
-        (assoc cfg ::bfc/input input-path)
+        owned? (some? upload-id)
 
         result
         (try
@@ -126,7 +110,7 @@
                              :version version})))
           (finally
             (when owned?
-              (fs/delete input-path))))]
+              (fs/delete (:path file)))))]
 
     (db/update! pool :project
                 {:modified-at (ct/now)}

backend/src/app/rpc/commands/comments.clj

@@ -29,7 +29,6 @@
    [app.rpc.doc :as-alias doc]
    [app.rpc.quotes :as quotes]
    [app.rpc.retry :as rtry]
-   [app.util.pointer-map :as pmap]
    [app.util.services :as sv]
    [clojure.set :as set]
    [cuerdas.core :as str]))
@@ -83,7 +82,7 @@
   (not= :none (-> props :notifications :email-comments)))
 
 (defn send-comment-emails!
-  [conn profile comment thread file]
+  [{:keys [::db/conn] :as cfg} profile comment thread file]
   (let [team-users        (get-team-users conn (:team-id file))
         comment-reference (format-comment-ref thread file)
         comment-content   (format-comment comment)
@@ -113,9 +112,10 @@
     (doseq [mention comment-mentions]
       (let [{:keys [fullname email props]} (get team-users mention)]
         (when (mention-email? props)
-          (eml/send!
-           {::eml/conn conn
-            ::eml/factory eml/comment-mention
+          (eml/send-template
+           cfg
+           {::eml/reuse-conn true
+            ::eml/template "comment-mention"
             :public-uri (cf/get :public-uri)
             :to email
             :name fullname
@@ -128,9 +128,10 @@
     (doseq [mention thread-mentions]
       (let [{:keys [fullname email props]} (get team-users mention)]
         (when (mention-email? props)
-          (eml/send!
-           {::eml/conn conn
-            ::eml/factory eml/comment-thread
+          (eml/send-template
+           cfg
+           {::eml/reuse-conn true
+            ::eml/template "comment-thread"
             :public-uri (cf/get :public-uri)
             :to email
             :name fullname
@@ -143,9 +144,10 @@
     (doseq [user-id notificate-users-ids]
       (let [{:keys [id fullname email props]} (get team-users user-id)]
         (when (notification-email? id (:owner-id thread) props)
-          (eml/send!
-           {::eml/conn conn
-            ::eml/factory eml/comment-notification
+          (eml/send-template
+           cfg
+           {::eml/reuse-conn true
+            ::eml/template "comment-notification"
             :public-uri (cf/get :public-uri)
             :to email
             :name fullname
@@ -167,7 +169,7 @@
 (defn- get-file
   "A specialized version of get-file for comments module."
   [cfg file-id page-id]
-  (binding [pmap/*load-fn* (partial feat.fdata/load-pointer cfg file-id)]
+  (feat.fdata/with-pointers cfg file-id
     (let [file (bfc/get-file cfg file-id)
           data (get file :data)]
       (-> file
@@ -538,7 +540,7 @@
     (update-thread-seqn conn file-id seqn)
 
     ;; Send mentions emails
-    (send-comment-emails! conn profile comment thread file)
+    (send-comment-emails! cfg profile comment thread file)
 
     (-> thread
         (add-owner profile)
@@ -648,7 +650,7 @@
       ;; Update the current profile status in relation to the current thread
       (upsert-comment-thread-status! conn profile-id thread-id)
 
-      (send-comment-emails! conn profile comment thread file)
+      (send-comment-emails! cfg profile comment thread file)
 
       (vary-meta comment assoc ::audit/props (assoc comment :team-id team-id)))))
 

backend/src/app/rpc/commands/demo.clj

@@ -14,12 +14,12 @@
    [app.common.uuid :as uuid]
    [app.config :as cf]
    [app.db :as db]
+   [app.jobs :as jobs]
    [app.loggers.audit :as audit]
    [app.rpc :as-alias rpc]
    [app.rpc.commands.auth :as auth]
    [app.rpc.doc :as-alias doc]
    [app.util.services :as sv]
-   [app.worker :as wrk]
    [buddy.core.codecs :as bc]
    [buddy.core.nonce :as bn]))
 
@@ -66,7 +66,8 @@
    ::doc/changes [["1.15" "This method is migrated from mutations to commands."]
                   ["2.18" "Add optional `skip-onboarding` param. When true, the profile is created with `onboarding-viewed` and `release-notes-viewed` (current version) set, skipping the onboarding flow."]
                   ["2.18" "Add optional `expires-in` param. When set, the demo purge is scheduled that long after creation instead of the global deletion delay. Only values between 5 minutes and the global delay are accepted."]
-                  ["2.18.1" "Add optional `renderer` param. When set to `wasm` or `svg`, the profile is created with that `renderer` in props. When omitted, no renderer is stored and the Penpot default applies."]]
+                  ["2.18.1" "Add optional `renderer` param. When set to `wasm` or `svg`, the profile is created with that `renderer` in props. When omitted, no renderer is stored and the Penpot default applies."]
+                  ["2.20" "When `renderer` is omitted, the profile is created with the default renderer for new accounts (`wasm`)."]]
    ::sm/params schema:create-demo-profile}
   [cfg {:keys [skip-onboarding expires-in renderer]}]
 
@@ -101,10 +102,10 @@
                                          (->> (auth/create-profile cfg params)
                                               (auth/create-profile-rels cfg))))]
 
-    (wrk/submit! (-> cfg
-                     (assoc ::wrk/task :demo-purge)
-                     (assoc ::wrk/delay deletion-delay)
-                     (assoc ::wrk/params {:profile-id (:id profile)})))
+    (jobs/submit cfg
+                 {::jobs/name :demo-purge
+                  ::jobs/delay deletion-delay
+                  ::jobs/params {:profile-id (:id profile)}})
 
     (with-meta {:email email
                 :password password}

backend/src/app/rpc/commands/feedback.clj

@@ -13,11 +13,57 @@
    [app.config :as cf]
    [app.db :as db]
    [app.email :as eml]
+   [app.loggers.audit :as-alias audit]
    [app.rpc :as-alias rpc]
    [app.rpc.climit :as-alias climit]
    [app.rpc.commands.profile :as profile]
    [app.rpc.doc :as-alias doc]
-   [app.util.services :as sv]))
+   [app.rpc.helpers :as rph]
+   [app.util.services :as sv]
+   [clojure.string :as str]))
+
+(def schema:send-mcp-feedback
+  [:map {:title "send-mcp-feedback" :closed true}
+   [:subject [:and [:string {:min 1 :max 120}]
+              [:fn #(and (not (str/blank? %))
+                         (not (re-find #"\p{Cc}" %)))]]]
+   ;; Line breaks and tabs are the only control characters allowed
+   [:feedback [:and [:string {:min 1 :max 5000}]
+               [:fn #(and (not (str/blank? %))
+                          (not (re-find #"[\p{Cc}&&[^\t\n\r]]" %)))]]]])
+
+(sv/defmethod ::send-mcp-feedback
+  {::rpc/token-type "mcp"
+   ::db/transaction true
+   ::doc/added "2.20"
+   ::sm/params schema:send-mcp-feedback}
+  [cfg {:keys [::rpc/profile-id subject feedback]}]
+  (let [profile     (profile/get-profile cfg profile-id)
+        destination (cf/get :mcp-feedback-email)]
+    ;; Blocking a profile invalidates its sessions but not its access tokens
+    (when (:is-blocked profile)
+      (ex/raise :type :authorization
+                :code :profile-blocked
+                :hint "profile is marked as blocked"))
+    (when-not (:is-active profile)
+      (ex/raise :type :authorization
+                :code :profile-not-active
+                :hint "profile is not active"))
+    (when (str/blank? destination)
+      (ex/raise :type :restriction
+                :code :feedback-disabled
+                :hint "MCP feedback is not configured"))
+    (eml/send cfg {::eml/reuse-conn true
+                   :to destination
+                   :reply-to (:email profile)
+                   :subject (str "[MCP feedback] " subject)
+                   :body {"text/plain" (str "Submitted by: " (:fullname profile)
+                                            " <" (:email profile) ">\n"
+                                            "Profile ID: " (:id profile) "\n\n"
+                                            "Feedback:\n\n" feedback)}})
+    ;; The audit event keeps who sent feedback, not its text
+    (rph/with-meta {:status "queued"}
+      {::audit/replace-props {}})))
 
 (declare ^:private send-user-feedback!)
 
@@ -32,20 +78,22 @@
 (sv/defmethod ::send-user-feedback
   {::climit/id [[:send-user-feedback/by-profile ::rpc/profile-id]
                 [:send-user-feedback/global]]
+   ::db/transaction true
    ::doc/added "1.18"
    ::sm/params schema:send-user-feedback}
-  [{:keys [::db/pool]} {:keys [::rpc/profile-id] :as params}]
+  [cfg {:keys [::rpc/profile-id] :as params}]
   (when-not (contains? cf/flags :user-feedback)
     (ex/raise :type :restriction
               :code :feedback-disabled
               :hint "feedback not enabled"))
 
-  (let [profile (profile/get-profile pool profile-id)]
-    (send-user-feedback! pool profile params)
+  (let [pool    (::db/pool cfg)
+        profile (profile/get-profile pool profile-id)]
+    (send-user-feedback! cfg profile params)
     nil))
 
 (defn- send-user-feedback!
-  [pool profile params]
+  [cfg profile params]
   (let [destination
         (or (cf/get :user-feedback-destination)
             ;; LEGACY
@@ -55,16 +103,16 @@
         (d/without-nils
          {"error-report.txt" (:error-report params)})]
 
-    (eml/send! {::eml/conn pool
-                ::eml/factory eml/user-feedback
-                :to       destination
-                :reply-to (:email profile)
-                :email    (:email profile)
-                :attachments attachments
+    (eml/send-template cfg {::eml/reuse-conn true
+                            ::eml/template "feedback"
+                            :to       destination
+                            :reply-to (:email profile)
+                            :email    (:email profile)
+                            :attachments attachments
 
-                :feedback-subject (:subject params)
-                :feedback-type (:type params "not-specified")
-                :feedback-content (:content params)
-                :feedback-error-href (:error-href params)
-                :profile profile})
+                            :feedback-subject (:subject params)
+                            :feedback-type (:type params "not-specified")
+                            :feedback-content (:content params)
+                            :feedback-error-href (:error-href params)
+                            :profile (select-keys profile [:id :fullname :email])})
     nil))

backend/src/app/rpc/commands/files.clj

@@ -22,6 +22,7 @@
    [app.common.transit :as t]
    [app.common.types.components-list :as ctkl]
    [app.common.types.file :as ctf]
+   [app.common.types.plugins :as ctpg]
    [app.common.types.tokens-lib :as ctob]
    [app.common.uri :as uri]
    [app.config :as cf]
@@ -30,6 +31,7 @@
    [app.features.fdata :as feat.fdata]
    [app.features.logical-deletion :as ldel]
    [app.http.sse :as sse]
+   [app.jobs :as jobs]
    [app.loggers.audit :as-alias audit]
    [app.loggers.webhooks :as-alias webhooks]
    [app.msgbus :as mbus]
@@ -45,7 +47,6 @@
    [app.util.events :as events]
    [app.util.pointer-map :as pmap]
    [app.util.services :as sv]
-   [app.worker :as wrk]
    [cuerdas.core :as str]))
 
 ;; --- FEATURES
@@ -430,7 +431,7 @@
                   (cfeat/check-client-features! (:features params))
                   (cfeat/check-file-features! (:features file)))
 
-        page  (binding [pmap/*load-fn* (partial feat.fdata/load-pointer cfg file-id)]
+        page  (feat.fdata/with-pointers cfg file-id
                 (let [page (dm/get-in file [:data :pages-index resolved-page-id])]
                   (if (pmap/pointer-map? page)
                     (deref page)
@@ -560,7 +561,7 @@
         calculate-from-db
         (fn []
           (let [file   (bfc/get-file cfg id)
-                result (binding [pmap/*load-fn* (partial feat.fdata/load-pointer cfg id)]
+                result (feat.fdata/with-pointers cfg id
                          (calculate-library-summary file))]
             (-> file
                 (assoc :library-summary result)
@@ -684,7 +685,7 @@
 (defn- get-file-stats
   [{:keys [::db/conn] :as cfg} file-id]
   (let [file    (bfc/get-file cfg file-id)
-        base    (binding [pmap/*load-fn* (partial feat.fdata/load-pointer cfg file-id)]
+        base    (feat.fdata/with-pointers cfg file-id
                   (cfs/calc-file-stats (:data file)))
         lib-cnt (get-file-stats-library-counts conn file-id)]
     (-> base
@@ -731,6 +732,53 @@
   (get-file-stats cfg id))
 
 
+;; --- COMMAND QUERY: get-file-plugin-data
+
+(def ^:private schema:get-file-plugin-data
+  [:map {:title "get-file-plugin-data"}
+   [:id ::sm/uuid]
+   [:namespace {:optional true} :keyword]])
+
+(sv/defmethod ::get-file-plugin-data
+  "Return the file-level plugin data: the arbitrary key/value metadata
+   stored on the file root through the Plugin API. Optionally narrowed
+   to a single `namespace`. Lets external tooling read metadata such as
+   issue or commit references without downloading the whole file."
+  {::doc/added "2.20"
+   ::sm/params schema:get-file-plugin-data
+   ::sm/result ctpg/schema:plugin-data
+   ::db/transaction true}
+  [{:keys [::db/conn] :as cfg} {:keys [::rpc/profile-id id namespace]}]
+  (check-read-permissions! conn profile-id id)
+  (let [plugin-data (-> (bfc/get-file cfg id)
+                        (get-in [:data :plugin-data]))]
+    (if namespace
+      (select-keys plugin-data [namespace])
+      (or plugin-data {}))))
+
+;; --- COMMAND QUERY: get-file-tokens
+
+(def ^:private schema:get-file-tokens
+  [:map {:title "get-file-tokens"}
+   [:file-id ::sm/uuid]])
+
+(sv/defmethod ::get-file-tokens
+  "Return the design tokens of a file as a DTCG document, or nil when
+   the file has none. Active themes and sets come from the file's
+   tokens status. Cheap alternative to `get-file` when only the tokens
+   are needed."
+  {::doc/added "2.20"
+   ::sm/params schema:get-file-tokens
+   ::sm/result [:maybe [:map-of :string :any]]}
+  [cfg {:keys [::rpc/profile-id file-id]}]
+  (bfc/check-file-exists cfg file-id)
+  (check-read-permissions! cfg profile-id file-id)
+  (let [{:keys [data]} (bfc/get-file cfg file-id)]
+    (feat.fdata/with-pointers cfg file-id
+      (when-let [tokens-lib (cfo/get-tokens-lib data)]
+        (ctob/export-dtcg-json tokens-lib (cfo/get-tokens-status data))))))
+
+
 ;; --- COMMAND QUERY: get-file-libraries
 
 (def ^:private schema:get-file-libraries
@@ -945,8 +993,7 @@
   (assert (db/connection-map? cfg)
           "expected cfg with valid connection")
 
-  (binding [pmap/*load-fn* (partial feat.fdata/load-pointer cfg file-id)
-            pmap/*tracked* (pmap/create-tracked)]
+  (feat.fdata/with-tracked-pointers cfg file-id
     (let [file (-> (bfc/get-file cfg file-id
                                  :include-deleted? true
                                  :lock-for-update? true)
@@ -1077,13 +1124,13 @@
 ;; --- MUTATION COMMAND: delete-file
 
 (defn- mark-file-deleted
-  [conn team file-id]
-  (let [delay    (ldel/get-deletion-delay team)
-        file     (db/update! conn :file
-                             {:deleted-at (ct/in-future delay)}
-                             {:id file-id}
-                             {::db/return-keys [:id :name :is-shared :deleted-at
-                                                :project-id :created-at :modified-at]})
+  [{:keys [::db/conn] :as cfg} team file-id]
+  (let [delay (ldel/get-deletion-delay team)
+        file  (db/update! conn :file
+                          {:deleted-at (ct/in-future delay)}
+                          {:id file-id}
+                          {::db/return-keys [:id :name :is-shared :deleted-at
+                                             :project-id :created-at :modified-at]})
         ;; Files using the library, captured before their relations are removed
         file-ids (when (:is-shared file)
                    (get-referenced-files conn file-id))]
@@ -1092,13 +1139,13 @@
     (db/delete! conn :file-library-rel
                 {:library-file-id file-id})
 
-    (wrk/submit! {::db/conn conn
-                  ::wrk/task :delete-object
-                  ::wrk/params (cond-> {:object :file
-                                        :deleted-at (:deleted-at file)
-                                        :id file-id}
-                                 (some? file-ids)
-                                 (assoc :file-ids file-ids))})
+    (jobs/submit cfg
+                 {::jobs/name :delete-object
+                  ::jobs/params (cond-> {:object :file
+                                         :deleted-at (:deleted-at file)
+                                         :id file-id}
+                                  (some? file-ids)
+                                  (assoc :file-ids file-ids))})
     file))
 
 (def ^:private
@@ -1112,7 +1159,7 @@
   (let [team (teams/get-team conn
                              :profile-id profile-id
                              :file-id id)
-        file (mark-file-deleted conn team id)
+        file (mark-file-deleted cfg team id)
         msgbus (::mbus/msgbus cfg)]
 
     (mbus/pub! msgbus
@@ -1122,12 +1169,12 @@
                          :profile-id profile-id})
 
     (rph/with-meta (rph/wrap)
-      {::audit/props {:project-id (:project-id file)
+      {::audit/props {:team-id (:id team)
+                      :project-id (:project-id file)
                       :name (:name file)
                       :created-at (:created-at file)
                       :modified-at (:modified-at file)
-                      :deleted-at (:deleted-at file)
-                      :team-id (:id team)}})))
+                      :deleted-at (:deleted-at file)}})))
 
 (sv/defmethod ::delete-file
   {::doc/added "1.17"
@@ -1294,7 +1341,7 @@
    [:ids [::sm/set ::sm/uuid]]])
 
 (defn- permanently-delete-team-files
-  [{:keys [::db/conn]} {:keys [::rpc/request-at team-id ids]}]
+  [{:keys [::db/conn] :as cfg} {:keys [::rpc/request-at team-id ids]}]
   (let [ids (into #{}
                   d/xf:map-id
                   (db/exec! conn [sql:get-delete-team-files-candidates team-id
@@ -1306,11 +1353,11 @@
                           {:deleted-at request-at}
                           {:id id}
                           {::db/return-keys false})
-              (wrk/submit! {::db/conn conn
-                            ::wrk/task :delete-object
-                            ::wrk/params {:object :file
-                                          :deleted-at request-at
-                                          :id id}})
+              (jobs/submit cfg
+                           {::jobs/name :delete-object
+                            ::jobs/params {:object :file
+                                           :deleted-at request-at
+                                           :id id}})
               (conj acc id))
             #{}
             ids)))

backend/src/app/rpc/commands/files_create.clj

@@ -14,6 +14,7 @@
    [app.common.types.file :as ctf]
    [app.config :as cf]
    [app.db :as db]
+   [app.features.fdata :as fdata]
    [app.loggers.audit :as-alias audit]
    [app.loggers.webhooks :as-alias webhooks]
    [app.rpc :as-alias rpc]
@@ -22,7 +23,6 @@
    [app.rpc.doc :as-alias doc]
    [app.rpc.permissions :as perms]
    [app.rpc.quotes :as quotes]
-   [app.util.pointer-map :as pmap]
    [app.util.services :as sv]
    [clojure.set :as set]))
 
@@ -44,32 +44,31 @@
 
   (assert (db/connection? conn) "expected a valid connection")
 
-  (binding [pmap/*tracked* (pmap/create-tracked)
-            cfeat/*current* features]
+  (binding [cfeat/*current* features]
+    (fdata/with-tracked
+      (let [file (ctf/make-file {:id id
+                                 :project-id project-id
+                                 :name name
+                                 :revn revn
+                                 :is-shared is-shared
+                                 :features features
+                                 :migrations fmg/available-migrations
+                                 :ignore-sync-until ignore-sync-until
+                                 :created-at modified-at
+                                 :deleted-at deleted-at}
+                                {:create-page create-page
+                                 :page-id page-id})]
 
-    (let [file (ctf/make-file {:id id
-                               :project-id project-id
-                               :name name
-                               :revn revn
-                               :is-shared is-shared
-                               :features features
-                               :migrations fmg/available-migrations
-                               :ignore-sync-until ignore-sync-until
-                               :created-at modified-at
-                               :deleted-at deleted-at}
-                              {:create-page create-page
-                               :page-id page-id})]
+        (bfc/insert-file! cfg file)
 
-      (bfc/insert-file! cfg file)
+        (->> (assoc params :file-id (:id file) :role :owner)
+             (create-file-role! conn))
 
-      (->> (assoc params :file-id (:id file) :role :owner)
-           (create-file-role! conn))
+        (db/update! conn :project
+                    {:modified-at (ct/now)}
+                    {:id project-id})
 
-      (db/update! conn :project
-                  {:modified-at (ct/now)}
-                  {:id project-id})
-
-      (bfc/get-file cfg (:id file)))))
+        (bfc/get-file cfg (:id file))))))
 
 (def ^:private schema:create-file
   [:map {:title "create-file" :closed true}

backend/src/app/rpc/commands/files_snapshot.clj

@@ -9,10 +9,12 @@
    [app.binfile.common :as bfc]
    [app.common.exceptions :as ex]
    [app.common.features :as-alias cfeat]
+   [app.common.files.migrations :as fmg]
    [app.common.schema :as sm]
    [app.common.time :as ct]
    [app.db :as db]
    [app.db.sql :as-alias sql]
+   [app.features.fdata :as fdata]
    [app.features.file-snapshots :as fsnap]
    [app.features.logical-deletion :as ldel]
    [app.main :as-alias main]
@@ -53,7 +55,7 @@
    ::sm/params schema:get-file-snapshot
    ::sm/result files/schema:file-with-permissions
    ::db/transaction true}
-  [cfg {:keys [::rpc/profile-id file-id id] :as params}]
+  [cfg {:keys [::rpc/profile-id file-id id features] :as params}]
   (let [perms (perms/get-file-read-permissions cfg profile-id file-id)]
     (files/check-read-permissions! perms)
     (let [snapshot (fsnap/get-snapshot cfg file-id id)]
@@ -66,12 +68,41 @@
       ;; Load current file metadata only (no data decoding) then overlay
       ;; the snapshot data so the client receives the same shape as a
       ;; normal get-file response but with historical page/object content.
-      (let [base-file (bfc/get-file cfg file-id :load-data? false)]
+      ;; Snapshots may hold pointer-mapped pages shared with the live
+      ;; file, so materialize them here: the preview UI applies the
+      ;; response straight into state and cannot resolve pointers.
+      ;; NOTE: the loader binds the live file-id, not the snapshot id.
+      ;;
+      ;; A snapshot taken while a migration was pending stores the old
+      ;; format. Apply the pending migrations in memory, the same pure
+      ;; step a normal read runs, and never persist them: the stored
+      ;; version keeps its data and restoring it re-migrates on the
+      ;; next read. `decode-snapshot` hands an ordered set, which is
+      ;; what `need-migration?`/`migrate` require; a stored NULL is
+      ;; regenerated from the snapshot's version.
+      (let [base-file (bfc/get-file cfg file-id :load-data? false)
+            snapshot  (assoc snapshot :id file-id)
+            file      (if (fmg/need-migration? snapshot)
+                        (let [file (fdata/realize cfg snapshot)]
+                          (fmg/migrate-file
+                           file
+                           (delay (bfc/get-resolved-file-libraries cfg file))))
+                        (update snapshot :data
+                                #(fdata/realize-pointers cfg file-id %)))
+            ;; Works like in `get-file`: a client that does not declare
+            ;; `fdata/objects-map` gets plain objects. The historical
+            ;; feature set is deliberately not checked against the
+            ;; enabled one: a version may predate a feature change.
+            file      (if (and (contains? (:features file) "fdata/objects-map")
+                               (not (contains? features "fdata/objects-map")))
+                        (fdata/realize-objects cfg file)
+                        file)]
         (-> base-file
-            (assoc :data (:data snapshot))
-            (assoc :version (:version snapshot))
-            (assoc :features (:features snapshot))
-            (assoc :revn (:revn snapshot))
+            (assoc :data (:data file))
+            (assoc :version (:version file))
+            (assoc :features (:features file))
+            (assoc :migrations (:migrations file))
+            (assoc :revn (:revn file))
             (assoc :vern (rand-int 100000))
             (assoc :permissions perms))))))
 
@@ -84,10 +115,22 @@
   {::doc/added "1.20"
    ::sm/params schema:create-file-snapshot
    ::climit/id [[:create-file-snapshot/by-profile ::rpc/profile-id]
-                [:create-file-snapshot/global]]}
+                [:create-file-snapshot/global]]
+   ::db/transaction true}
   [cfg {:keys [::rpc/profile-id file-id label]}]
   (files/check-edition-permissions! cfg profile-id file-id)
-  (let [file    (bfc/get-file cfg file-id :realize? true)
+  ;; NOTE: the file is intentionally NOT realized here: the snapshot
+  ;; shares fragment rows with the live file (same as automatic
+  ;; snapshots) instead of storing a materialized copy. file-gc treats
+  ;; every unpurged snapshot as a liveness root, and readers
+  ;; (get-file-snapshot, restore!, reduce-snapshots) resolve pointers
+  ;; on demand.
+  ;; NOTE: the file is read with FOR SHARE inside this transaction so
+  ;; snapshot creation serializes with a concurrent :file-gc pass
+  ;; (which takes FOR UPDATE + SKIP LOCKED on the same row). Without
+  ;; the lock, a version pinned mid-pass could reference fragment rows
+  ;; the pass is recreating.
+  (let [file    (bfc/get-file cfg file-id :lock-for-share? true)
         project (db/get-by-id cfg :project (:project-id file))]
 
     (-> cfg
@@ -115,38 +158,58 @@
                 [:restore-file-snapshot/global]]}
   [{:keys [::db/pool ::mbus/msgbus] :as cfg} {:keys [::rpc/profile-id ::rpc/session-id file-id id] :as params}]
 
-  ;; Check permissions and read current file state (short-lived, outside restore transaction)
+  ;; Check permissions
   (files/check-edition-permissions! pool profile-id file-id)
-  (let [file  (bfc/get-file cfg file-id)
-        team  (teams/get-team pool
+  (let [team  (teams/get-team pool
                               :profile-id profile-id
                               :file-id file-id)
         delay (ldel/get-deletion-delay team)
-        file-revn (:revn file)]
 
-    ;; Create backup snapshot of the current state (committed immediately
-    ;; independently of the restore outcome)
-    (fsnap/create! cfg file
-                   {:profile-id profile-id
-                    :deleted-at (ct/in-future delay)
-                    :created-by "system"})
+        ;; Create backup snapshot of the current state (committed immediately
+        ;; independent of the restore outcome). The read takes FOR SHARE so a
+        ;; concurrent :file-gc pass (which holds FOR UPDATE + SKIP LOCKED on the
+        ;; file row) either finishes first -- and we snapshot the new fragments --
+        ;; or skips the file and retries later. Without the lock, the backup
+        ;; could reference fragments the pass is about to mark as deleted,
+        ;; breaking the backup permanently.
+        ;;
+        ;; The backup read also provides the revn the guard below compares
+        ;; against. An edit committed before this read is captured in the
+        ;; backup and no longer aborts the restore; only edits committed
+        ;; after the backup conflict.
+        backup-revn
+        (db/tx-run! cfg
+                    (fn [cfg]
+                      (let [file (bfc/get-file cfg file-id :lock-for-share? true)]
+                        (fsnap/create! cfg file
+                                       {:profile-id profile-id
+                                        :deleted-at (ct/in-future delay)
+                                        :created-by "system"})
+                        (:revn file))))]
 
     ;; Restore snapshot inside its own transaction; the revn check
     ;; ensures no data is lost if the file was edited concurrently
     (db/tx-run! cfg
                 (fn [{:keys [::db/conn] :as cfg}]
                   (let [current (bfc/get-minimal-file conn file-id {::db/for-update true})]
-                    (when (not= (:revn current) file-revn)
+                    (when (not= (:revn current) backup-revn)
                       (ex/raise :type :conflict
                                 :code :file-modified
                                 :hint "the file was modified during the restore process, please retry")))
-                  (let [vern (fsnap/restore! cfg file-id id)]
+                  (let [snapshot (fsnap/get-minimal-snapshot cfg id)
+                        vern     (fsnap/restore! cfg file-id id)]
+                    ;; Peers name the restored version by its label when a
+                    ;; user named it, else by its date
                     (mbus/pub! msgbus
-                               :topic (:id file)
-                               :message {:type :file-restored
-                                         :session-id session-id
-                                         :file-id (:id file)
-                                         :vern vern})
+                               :topic file-id
+                               :message (cond-> {:type :file-restored
+                                                 :session-id session-id
+                                                 :profile-id profile-id
+                                                 :file-id file-id
+                                                 :vern vern
+                                                 :created-at (:created-at snapshot)}
+                                          (= "user" (:created-by snapshot))
+                                          (assoc :label (:label snapshot))))
                     nil)))))
 
 (def ^:private schema:update-file-snapshot
@@ -171,7 +234,7 @@
   {::doc/added "1.20"
    ::sm/params schema:remove-file-snapshot
    ::db/transaction true}
-  [{:keys [::db/conn]} {:keys [::rpc/profile-id id]}]
+  [{:keys [::db/conn] :as cfg} {:keys [::rpc/profile-id id]}]
   (let [snapshot (fsnap/get-minimal-snapshot conn id)]
     (files/check-edition-permissions! conn profile-id (:file-id snapshot))
 
@@ -194,7 +257,7 @@
                                 :profile-id profile-id
                                 :file-id (:file-id snapshot))
           delay (ldel/get-deletion-delay team)]
-      (fsnap/delete! conn (assoc snapshot :deleted-at (ct/in-future delay))))))
+      (fsnap/delete! cfg (assoc snapshot :deleted-at (ct/in-future delay))))))
 
 ;;; Lock/unlock version endpoints