github/penpot
1
0
Fork 0
mirror of https://github.com/penpot/penpot.git synced 2026-11-12 04:50:42 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'develop' into alvorithm-issue-12666

Browse Source 
Develop moved the per-save validation of files_update.clj into
`validate-file`, which `apply-changes` runs on every attempt: the whole
save first, then each commit prefix while bisecting a failing save.
`process-changes-and-validate` no longer has the `fdata/with-untracked`
block where this branch ran its soft check of pages added through the
`{:page ...}` form of `:add-page`.

- files_update.clj keeps develop's `validate-file`, debug-fault helpers,
  and `process-changes-and-validate` as they are, and keeps this
  branch's `get-added-page-ids` and `validate-added-pages`.
- The soft check moves into the new `soft-validate-added-pages!`, which
  `validate-file` calls first, under the same condition develop uses to
  resolve libraries: `file-validation` or `soft-file-validation`, and
  no `skip-validate`. It logs and never raises, as before.

Develop's `validate-file-schema-affected!` already counts these pages
among the touched ones, so the schema check covers them. The
referential `validate-file-affected` still collects only the top-level
`:id`, so the soft referential check of this branch is still needed.

AI-assisted-by: claude-opus-5-5
Signed-off-by: Álvaro Tejero Cantero <alvorithm@teje.ro>
This commit is contained in:
Álvaro Tejero Cantero 2026-11-11 17:38:54 +01:00
commit 3258623f74
No known key found for this signature in database
626 changed files with 69371 additions and 44663 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

330
.agents/skills/migrate-ds-area/SKILL.md Normal file
View File

@ -0,0 +1,330 @@
---
name: migrate-ds-area
description: Migrate one Penpot UI area to the Design System components for one group of the DS migration epic (#12375): buttons and icons (icon-button*, button*, icon*), selects (select*) and later groups. Moves the area's SCSS off the deprecated styles, proves every change with a zero-tolerance visual test, then opens the draft PR and updates the group issue (PRs table row, QA comment). Use when the user asks to migrate an area or module to the DS, continue a DS migration group, or names an area of a group issue.
---
# Skill: migrate-ds-area
Move one UI area (one `.cljs` + `.scss` pair, or a few related modules)
from legacy components and raw HTML to the Design System components,
prove with pixels what changed, and ship it as a draft PR that QA can
test.
The epic is #12375. Each control type is a group with its own issue
(buttons and icons #12376, selects #12377, radios #12378, …). The group
issue holds the plan (order, waiting areas), the changes QA should
expect, and the PRs table where people claim areas. Read it first.
## Input
- The group issue (e.g. `#12376`) and the area: a module path (e.g.
`dashboard/fonts`) or an area named in the group issue.
- No area given: take the next unclaimed area in the group issue's plan
order, and say which one.
## Ground rules
- Never `git push`. The user pushes; then you open the PR.
- PRs go to `develop`; branches start from `origin/develop`.
- **Stay in the group.** Only the controls of this group change. In the
buttons/icons group, selects, radios, numeric/text inputs, checkboxes,
menus (`%menu-*`, `%dropdown-*`) and modal structure (`%modal-*`) stay;
a checkbox tick or an input label icon stays legacy.
- **Accept the DS look.** Do not add CSS to imitate legacy sizes or
colors. Do keep positions: icons and labels must not move unless the
DS size forces it, and then say so.
- No negative margins to compensate; use positive DS spacing or
`px2rem()`.
- **Gaps in the DS wait.** An area that needs an icon or a feature the
DS lacks keeps its legacy control; say so in the PR.
- Skip legacy shared components that another group will replace
(`context_menu_a11y`, `forms`, `select`, `dropdown`, `radio_buttons`,
`search_bar`, `title_bar`, …): they wait for that group's cleanup,
where they are updated if still used or deleted if not.
- Debug-only UI gets no QA comment and no compare image.
## Devenv
Run every test, lint and `pnpm` command inside the workspace container,
never on the host (a host `pnpm` relinks `node_modules` and breaks the
container's watcher). The container is `penpot-devenv-ws<N>-main`, where
`<N>` is the workspace number (`ws0` is the main repo; `wsN` lives in
`~/.penpot/penpot_workspaces/wsN`). Write output to a file, then read it:
```bash
C=penpot-devenv-ws1-main # your workspace
docker exec -u penpot -w /home/penpot/penpot/frontend $C \
bash -lc '<command>' > <scratch>/out.txt 2>&1
```
When the app does not behave as the code says, check the build first:
- **Stale build after a branch switch** ("is not a function", or a page
with empty `stl/css` classes): `touch` the stale `.cljs`/`.scss`, wait
for the compiled file under `frontend/resources/public/` to update,
hard-reload.
- **A whole area unstyled:** a new `.scss` added after the watcher
started is missing from `main.css` (grep `main.css` for the module
prefix), or `main.css` was written corrupted after many `.scss`
changed at once. Touch a `.scss`; if the module is still missing,
restart the watcher in the workspace's tmux session (`penpot:0`,
`./scripts/watch app`).
- **Blank app after switching between `develop` and `staging`:**
dependencies differ. Run `pnpm install --frozen-lockfile` in the
container and restart the watcher.
## Workflow
### 1. Claim the area and branch
1. Read the group issue: plan, waiting areas, expected changes, PRs table.
2. Add a row to the PRs table with no PR number, the user's @login as
Owner and the state `Claimed` (`gh issue view/edit <n> --body-file`).
Read the body right before editing, and read it again after saving:
when two people edit at the same time, GitHub keeps only the last
edit, so a row can vanish. Put yours back if it is gone.
Skip an area someone else already claimed, unless the claim has had
no PR for 7 days and its owner did not answer a comment asking.
3. `git switch -c <user>-ds-<group>-<area> origin/develop`.
4. Survey the area: every control of the group in the `.cljs`, the
legacy icons, the placeholders and mixins the `.scss` uses, the
`div`/`span` with `:on-click`.
### 2. Visual baseline
Add a `test.describe("<Area> controls", …)` block to the matching spec
in `frontend/playwright/ui/visual-specs/` (project `ds`, dark theme).
Put it next to the blocks of the same screen, not always at the end of
the file: two PRs that both append at the end conflict.
- Every shot: `toHaveScreenshot({ maxDiffPixelRatio: 0, threshold: 0 })`.
- Show every changed control at rest, on hover and open or pressed.
Capture the element or the smallest stable container.
- Locators must work before and after the change: text, role, testid,
`[class*='…']`, or `a.or(b)` when the element type changes (raw
`[class*='menu-btn']` before, `getByRole("button", { name: "Options" })`
after). Scope `.or()` to the area so it cannot match elsewhere.
- Missing data? Add a fixture under `frontend/playwright/data/` and mock
it (`dashboardPage.mockRPC(...)`, `DashboardPage.mockConfigFlags` for
flags). Watch for expired or inconsistent fixtures (dates in the past,
wrong file for a mock).
- Capture with `--update-snapshots=all`:
```bash
npx playwright test --project=ds <spec> -g "<describe>" --update-snapshots=all
```
Plain `--update-snapshots` keeps a snapshot within the project's 0.5%
tolerance, so a real diff looks like 0.
- Copy the snapshots (`<spec>.spec.js-snapshots/<Describe>-*`) to
`<scratch>/diffs/<area>/before/` and look at them: the state you
wanted must really be on screen.
### 3. Controls commit
#### Buttons and icons
| Legacy | DS |
|---|---|
| icon-only raw button | `[:> icon-button* {:variant "ghost" :aria-label (tr …) :icon i/x :on-click f}]` (always pass `:variant`; `"action"` for 24px icon toggles) |
| pressed/selected | `:aria-pressed (boolean x)` |
| text button | `[:> button* {:variant "primary"/"secondary"/"ghost"/"destructive" :type "button"} label]` |
| `[:input {:type "button"}]` | `button*` |
| modal close | `icon-button*` ghost, `:aria-label (tr "labels.close")`, `:tooltip-class` carrying the corner position |
| legacy icon outside a button | `[:> icon* {:icon-id i/x :class …}]` |
- Not every raw button changes: rows, tabs, swatches, avatar widgets,
full-height segments, round custom buttons and link-style text buttons
stay raw (their legacy icon still changes). List the kept ones in the
PR.
- A `div`/`span` with `:on-click` in the area becomes a `[:button]`
(with `aria-expanded`/`aria-haspopup` when it opens something). An
`[:a]` without `href` stays; list it in the PR.
- **Tooltip wrapper.** `icon-button*` renders inside a tooltip trigger
wrapper. Position (`position: absolute`, grid placement, margins
against siblings) goes on `:tooltip-class`; look (`--button-height`,
colors) goes on `:class`.
- **Tooltip at the right edge.** Small icon buttons at the right end of
a sidebar row or title bar get `:tooltip-placement "top-left"`; list
it as an expected change.
- **Sizes through custom properties:** `--button-height: #{px2rem(64)}` /
`--button-width`. Sass does not evaluate functions inside custom
properties without `#{}`.
- **A grown `button*` keeps its label at the start**: add
`justify-content: center` when it is wider than its content.
- **Icon colors:** DS icons use `stroke: currentColor`. Set `color` on
the icon class, not `stroke`/`fill`. A legacy `color: transparent` left
on the button or wrapper makes the DS icon invisible: drop it. A class
that redefines the color variable on the icon breaks the hover that
changes it on the button: define the variable on the button.
- **Every DS icon needs a color of its own** (on its class or a parent):
one that inherits none shows in the devenv debug yellow.
- **Icon sizes:** `"s"` draws 12px inside a 16px box (compensate the
margin by 2px per side to keep the drawing in place); `"m"` 16px; `"l"`
24px; `"xl"` 32px.
- **Stroke width:** DS icons keep the DS stroke width. Never set
`stroke-width` on a DS icon to match the legacy one; a thinner or
thicker line is part of the DS look.
- **Legacy placeholders that style `svg` descendants** (e.g.
`%dropdown-element-base` → `span svg { 12px box; stroke }`) override a
DS icon placed inside them: give the icon class its own box
(`inline-size`/`block-size` 16px) and `stroke: currentcolor` with more
specificity.
- **Flex shrink:** a DS svg inside a small flex button can shrink and be
cut; add `flex-shrink: 0` to the icon class.
- **Rotated legacy arrows:** prefer `i/arrow-left`/`-right`/`-up`/`-down`
over rotating `i/arrow` (same path), unless a rotation class drives the
open/closed state.
- **Menus anchored by the button size** move when the button grows; open
them from the top (`inset-block-start: calc(100% + var(--sp-xs))`) or
fix the context menu offset, and say so.
#### Selects
`[:& select {:default-value v :options [{:value :label}] :on-change f}]`
→ `[:> select* {:default-selected <id> :options <memoized vector of {:id :label}> :on-change f}]`
(`app.main.ui.ds.controls.select`).
- **Ids are non-blank strings.** A blank id shows a dimmed "--", and an
id that matches no option falls back to the first one. Give mixed,
none and auto their own ids (`"multiple"`, `"none"`, `"auto"`) and map
them back in the handler. Use `""` only where "--" is the wanted look.
- **Convert the value.** The legacy select ran `d/read-string`, so
handlers got keywords, numbers or uuids; `select*` passes the `:id`
string. Convert with `keyword`, `d/parse-integer` or `uuid/parse`.
- **Memoized options, live selection.** `select*` resets its selection
when `:default-selected` or the `:options` identity changes. Build
options with `mf/with-memo` and `mapv` (the schema wants a non-empty
vector) and pass `:default-selected` from the live value, never a
constant.
- **Never pass nil.** In dev builds a present prop with value nil
(`:class`, `:wrapper-class`, `:disabled`, `:default-selected`) throws
`invalid props on component select*`. Use `(boolean x)`, `d/nilv`, or
add the prop only when it has a value.
- **Props the legacy select ignored stay out**: `select*` passes rest
props to its button, so they would reach the DOM.
- **Classes:** `:wrapper-class` for layout, `:class` for look.
- **Portal:** add `:has-portal true` only where the visual test shows the
list cut at the bottom of a panel, and say so in the QA comment.
- **Locators:** the button is a `combobox` with no accessible name; use
`getByRole("combobox").filter({ hasText: "Linear" })` (it matches the
legacy select too) and `getByRole("option", { name: … })`.
- **Keyboard:** the arrows no longer change the value while the list is
closed. It is an expected change, already listed in #12377.
#### All groups
- New labels: add the key to `en.po` and `es.po` by hand, in sorted
position, with a `#:` reference to the cljs line. Never run the
translations sync.
- The `.scss` only gets what the new controls need in this commit.
- Some e2e specs are not Prettier-clean on develop: never run
`prettier --write` on a whole spec; format only your block.
Then wait for the watcher (grep the compiled JS/CSS under
`frontend/resources/public/` for the change), recapture with `=all`,
copy to `after/`, diff:
```bash
SK=.agents/skills/migrate-ds-area/scripts
python3 -I $SK/diff.py <before> <after> # per-file pixel count + bbox
python3 -I $SK/trip.py <b> <a> <out.png> <scale> # before | after | red diff
```
`trip.py` pads images of different sizes, so a taller "after" still
renders.
Explain every diff. Fix what is not a DS-look change (moved icons, cut
icons, lost hover color). Rerun a shot that changes on unchanged code
before trusting it; flaky spots seen so far: the invite modal textarea
border, the search input caret, the user account page, the path handler
menu trigger.
Lint, then commit (read `mem:workflow/creating-commits` first):
```bash
npx prettier --check <spec> <fixtures>; npx stylelint <scss>; clj-kondo --lint <cljs>; cljfmt check <cljs>
git add -f <cljs> <scss> <po> <spec> <fixtures> # -f: visual-specs/ is gitignored
./scripts/check-commit
```
A plain `git add` with an ignored path in the list adds nothing; always
`-f`, and check `git show --stat`. Never commit the screenshots.
Subject: `:recycle: Use DS <controls> in <area>`. A pre-existing bug
found and fixed on the way goes in its own `:bug:` commit.
### 4. SCSS commit
Run the `refactor-scss` skill on every `.scss` the PR touched (only
those), in its own commit. Keep `refactor/` only for component-family
placeholders (`%modal-*`, `%menu-*`, `%input-*`, `%checkbox-*`) and say
why in a comment above the `@use`. Unused selectors are deleted, not
commented out (list them in the commit body). Typography without an
exact match uses the closest DS typography. Run the skill's
**Self-review** before committing: no bare element selectors (add
classes in the `.cljs`), no copied properties that do nothing, z-index
tokens by purpose. Compare against the controls commit: it should be
0 px; explain anything else (typically a typography or border change).
For files that several modules share or that are huge, stop and propose
a separate PR instead.
### 5. Hand off, PR and tracking
1. Report to the user: commits, visible changes, kept raw controls,
pre-existing bugs, and the compare image path
(`<scratch>/diffs/<area>/compare.png`, before | after | diff). Give
the push command: `git push -u origin <branch>` (the branch tracks
`origin/develop`, so a bare `git push` is refused).
2. After the user pushes, check `git rev-parse HEAD origin/<branch>`,
then (read `mem:workflow/creating-prs` first):
```bash
gh pr create --repo penpot/penpot --draft --base develop --head <branch> \
--title "<commit subject>" --body-file <body> \
--label "WG adoption" --label "no issue required" \
--milestone <group issue milestone> --project Main
gh pr edit <n> --repo penpot/penpot --remove-label "needs triage"
```
Never add `no changelog` to a PR: that label goes on issues.
Body: AI note, `Part of #<group> (DS migration epic #12375).`, a
short What / How, the kept raw controls, an empty `## Screenshots`
when something is visible (the user attaches the image). Never
`Closes #<group>`. Keep it short: reviewers find long PRs hard to read.
3. In the group issue's PRs table, fill your row:
`| #<n> | <area>: <controls> (<visible changes>) | @<owner> | Draft |`.
4. Post a QA comment on the group issue: `**#<n>: <area>**`, *Where* (how
to reach it), *Expected changes* (every visible change; QA treats
anything not listed as a bug), *No other change* (what still works
the same).
Do not mark the draft ready or ask for review: the user decides when.
### 6. Review fixes
When the reviewer comments on a PR:
1. Fix each comment in a new commit on the branch (never amend pushed
commits). A comment you do not apply gets a short reason the user can
post, with an image when it is visual.
2. Rerun the area's visual spec before and after the fix with `=all`.
Build a compare image for every visible change and give the user its
full path, saying which PR it goes to.
3. Update everything the fix makes stale, in the same turn: the QA
comment (*Expected changes* must list the new visible changes), the
PRs table row, and the PR body.
4. When working on several PRs in one go, report per PR: branch, new
commit, what changed, image path. Never mix two branches in one list.
5. Give the push command per branch. If a rebase is coming soon, say
that pushing now costs one extra CI run; the user decides.
6. After a round of review fixes, check whether the same issue exists in
the user's other open PRs of the group, and fix it there too.
### 7. After merges
Whenever PRs merge (or go from draft to ready), sync the state column of
the group's PRs table with GitHub for **every** row, not only your own:
read each row's PR (`gh pr view <n> --json state,isDraft`) and set
Merged / In review / Draft. The table drifts when other people merge.
## Output
- Branch with 2 commits (controls, SCSS) plus any `:bug:` commit.
- Visual spec block with zero-tolerance shots.
- Compare image in the scratch folder.
- After the push: draft PR, row and QA comment in the group issue.

25
.agents/skills/migrate-ds-area/scripts/diff.py Normal file
View File

@ -0,0 +1,25 @@
"""Pixel diff of two folders of PNG snapshots with the same file names.
Usage: python3 -I diff.py <before-dir> <after-dir>
Prints, per file, the number of differing pixels and their bounding box,
or the two sizes when they differ.
"""
import os
import sys
from PIL import Image, ImageChops
before, after = sys.argv[1], sys.argv[2]
for name in sorted(os.listdir(before)):
other = os.path.join(after, name)
if not os.path.exists(other):
print(name, "missing in", after)
continue
a = Image.open(os.path.join(before, name)).convert("RGB")
b = Image.open(other).convert("RGB")
if a.size != b.size:
print(name, "size", a.size, b.size)
continue
diff = ImageChops.difference(a, b)
count = sum(1 for p in diff.getdata() if p != (0, 0, 0))
print(name, count, diff.getbbox())

33
.agents/skills/migrate-ds-area/scripts/trip.py Normal file
View File

@ -0,0 +1,33 @@
"""Before | after | red diff image, one row per snapshot.
Usage: python3 -I trip.py <before-dir> <after-dir> <out.png> <scale>
Both folders hold PNGs with the same names (crop them first if needed).
"""
import os
import sys
from PIL import Image, ImageChops
before, after, out, scale = sys.argv[1], sys.argv[2], sys.argv[3], int(sys.argv[4])
rows = []
for name in sorted(os.listdir(before)):
a = Image.open(os.path.join(before, name)).convert("RGB")
b = Image.open(os.path.join(after, name)).convert("RGB")
w, h = max(a.width, b.width), max(a.height, b.height)
pad_a = Image.new("RGB", (w, h), (200, 200, 200)); pad_a.paste(a, (0, 0))
pad_b = Image.new("RGB", (w, h), (200, 200, 200)); pad_b.paste(b, (0, 0))
mask = ImageChops.difference(pad_a, pad_b).convert("L").point(lambda v: 255 if v else 0)
red = Image.new("RGB", (w, h), (255, 255, 255))
red.paste((255, 0, 0), mask=mask)
row = Image.new("RGB", (w * 3 + 20, h), (200, 200, 200))
for i, im in enumerate([pad_a, pad_b, red]):
row.paste(im, (i * (w + 10), 0))
rows.append(row.resize((row.width * scale, row.height * scale), Image.NEAREST))
width = max(r.width for r in rows)
height = sum(r.height + 10 for r in rows)
canvas = Image.new("RGB", (width, height), (255, 255, 255))
y = 0
for r in rows:
canvas.paste(r, (0, y))
y += r.height + 10
canvas.save(out)

421
.agents/skills/refactor-scss/SKILL.md Normal file
View File

@ -0,0 +1,421 @@
---
name: refactor-scss
description: Migrate one Penpot SCSS module off the deprecated styles under frontend/resources/styles/common/refactor/ to the current Design System (ds/ tokens, mixins and px2rem). Pure refactor, no visual change. Use when the user asks to refactor, migrate or clean up a .scss file to the DS.
---
# Skill: refactor-scss
Refactor a single SCSS file so that it no longer depends on the deprecated
styling system, using the current Design System instead.
This is a **pure refactor**. Do NOT introduce visual or behavioral changes.
Preserve the generated CSS behavior exactly unless a rule below requires
otherwise. The rules that allow a small visible change (closest DS
typography, DS border width) want it shown in the visual check and listed
in the report and the commit body.
Reviewers read every line of the result. A line that is
copied but does nothing, a bare element selector or a raw z-index comes
back as a review comment, so the rules below are strict about them.
## Input
- The path to one `.scss` file (e.g. `frontend/src/app/main/ui/foo/bar.scss`).
- If no path is given, ask for it. Do not pick a file yourself.
## Workflow
1. Read the target `.scss` file and the `.cljs` file with the same name next
to it.
2. Read every deprecated source the file uses, so you inline real values,
never remembered ones:
- `frontend/resources/styles/common/refactor/mixins.scss`
- `frontend/resources/styles/common/refactor/design-tokens.scss`
- `frontend/resources/styles/common/refactor/spacing.scss`
- any other file under `frontend/resources/styles/common/refactor/` that
the stylesheet pulls in (placeholders, borders, z-index, fonts...).
3. Read the DS modules you will map to:
- `frontend/src/app/main/ui/ds/_utils.scss` (`px2rem()`)
- `frontend/src/app/main/ui/ds/spacing.scss`
- `frontend/src/app/main/ui/ds/_sizes.scss`
- `frontend/src/app/main/ui/ds/typography.scss` (`use-typography()`)
- `frontend/src/app/main/ui/ds/_borders.scss`
- `frontend/src/app/main/ui/ds/z-index.scss`
4. Apply the rules below and edit the file in place.
5. Verify (see **Verification** and **Visual check**).
6. Report (see **Output**). Do not commit. When the refactor goes with other
changes, it belongs in its own commit, after the task's commits.
## Primary goal
After the refactor, the SCSS file must not depend on anything under
`frontend/resources/styles/common/refactor/`. This includes imports, mixins,
placeholder selectors, functions, deprecated variables and design-token
aliases.
### Partial refactor
Some placeholders stand for a whole component family that will move to the DS
in its own PR: `%input-element`, `%input-with-label`, `%input-checkbox`,
`%modal-*`, `%menu-dropdown`, `%menu-item-base`, `%shortcut-*` and similar.
Inlining them in a small task produces big, risky diffs. In that case:
- migrate everything else (tokens, sizes, spacing, small mixins);
- keep the `refactor/` import only for those placeholders;
- list each remaining dependency and the selectors that use it in the report,
so the user can plan it as a separate PR.
## General rules
- Keep the resulting SCSS as small and readable as possible.
- Avoid duplicated declarations and duplicated selectors.
- Do not change class names.
- Do not change DOM structure.
- Do not change specificity unless required.
- Follow the project's SCSS formatting (Prettier, see **Verification**).
## Imports
Use the DS modules. Import them with `as *` when they expose mixins,
variables or functions:
```scss
@use "ds/_utils.scss" as *;
```
not
```scss
@use "ds/_utils.scss" as utils;
```
Do not introduce namespaces such as `utils.`, `typography.`, `borders.`,
`spacing.` or `sizes.` unless a module explicitly requires one.
Remove every unused import.
## Deprecated mixins
If the stylesheet uses mixins from
`frontend/resources/styles/common/refactor/mixins.scss`:
- locate the mixin;
- inline its generated CSS (then apply the rest of these rules to it);
- remove the `@include`;
- remove the import.
Do not leave deprecated mixins.
## Placeholders
Replace every `@extend %placeholder` with the generated CSS. Remove every
deprecated placeholder, except the component-family ones described in
**Partial refactor**.
Inline only what does something on that element. Do not paste a
placeholder (or mixin) body as it is. For each property, check:
- **It applies to the element.** `fill`, `stroke` and `stroke-width` only
act on SVG shapes: drop them from a `span`, `div` or `button`.
`color: transparent` on a wrapper that only holds an icon does nothing
useful: drop it and set the real color.
- **It is not the default or already given.** `stroke-width: 1px` is the
SVG default; `fill: none` and `stroke: currentColor` are already set by
the DS `icon*` (`.icon`). Drop them.
- **A later rule does not override it.**
- **It is not a stroke width on a DS icon.** DS icons keep the DS
stroke width; drop any `stroke-width` that only imitates the legacy
icon (1.33px, 2.25px...).
`%button-icon` is the usual case: on a wrapper it leaves `color:
transparent; fill: none; stroke-width: 1px`, and none of the three is
needed.
## Element selectors
Do not style HTML elements directly (`svg {}`, `h3 {}`, `button {}`,
`input.x {}`, or `.component { svg {} }`).
When no class exists, **add one in the `.cljs`**: a DS component takes
`:class (stl/css :x)` (`icon*`, `heading*`, `button*`…), and a plain
element takes `{:class (stl/css :x)}`. Touching the `.cljs` for this is
part of the refactor. A comment like "the icon has no class" is not an
answer.
Prefer the DS component when one fits: `heading*` for `h1`–`h6`
(`:level`, `:typography`), `text*` for text, `icon*` for icons.
`element.class` (e.g. `input.element-title`) is not needed to beat the
input defaults: try the class alone and check it in the visual check.
An element selector stays only for markup you cannot reach (injected
HTML, third-party widgets), with a comment saying so.
**Icon colors.** A DS icon draws with `stroke: currentColor`. Set
`color` on the icon class or its parent, never `stroke`/`fill` on an
`svg` selector.
## Color variables
Whenever a color changes across states, introduce a component-local CSS
variable.
Instead of
```scss
.button {
color: var(--color-foreground-primary);
&:hover {
color: var(--color-accent-primary);
}
&:disabled {
color: var(--color-foreground-disabled);
}
}
```
write
```scss
.button {
--button-foreground-color: var(--color-foreground-primary);
color: var(--button-foreground-color);
&:hover {
--button-foreground-color: var(--color-accent-primary);
}
&:disabled {
--button-foreground-color: var(--color-foreground-disabled);
}
}
```
Use this pattern for `color`, `background-color`, `border-color`,
`outline-color`, `fill`, `stroke` and any similar property.
## Design token aliases
Remove every alias defined in
`frontend/resources/styles/common/refactor/design-tokens.scss`. Resolve each
alias to its final DS token. Do not keep intermediate aliases.
Instead of `color: var(--button-foreground-hover);` write
`color: var(--color-accent-primary);`.
## Spacing
Replace deprecated spacing variables using
`frontend/src/app/main/ui/ds/spacing.scss` for `margin`, `padding`, `gap` and
`inset`.
If no exact spacing token exists, use `px2rem()`. Do not approximate with the
closest token.
## Sizes
Replace deprecated size variables using
`frontend/src/app/main/ui/ds/_sizes.scss` for `inline-size`, `block-size`,
`width`, `height` and min/max sizes.
If no exact size exists, use `px2rem()` from
`frontend/src/app/main/ui/ds/_utils.scss`. Do not leave hardcoded pixel values
unless absolutely necessary.
## Typography
When any of `font-size`, `font-weight`, `line-height`, `font-family` or
`letter-spacing` appear, check whether together they match one of the
definitions in `frontend/src/app/main/ui/ds/typography.scss`.
If they match, replace all the individual declarations with:
```scss
@include use-typography("<typography-name>");
```
If there is no exact match, use the **closest DS typography** (same role:
a panel title in uppercase → `headline-small`, body text → `body-*`),
even when more than one detail differs. Show the change in the visual
check and state it in the report and in the commit body (e.g. "11px
regular → 12px medium").
Keep the original declarations only when no DS typography fits the role
(e.g. a 32px display number in a 14px-only area); then leave a TODO
naming the closest one and ask the user.
## Borders
Replace deprecated border radius and border width values using
`frontend/src/app/main/ui/ds/_borders.scss`. A raw width close to a token
(`1.75px`) takes the token (`$b-2`); say so in the report. If nothing is
close, use `px2rem()`.
## Logical properties
Prefer logical properties whenever they preserve the existing behavior:
- `width` → `inline-size`
- `height` → `block-size`
- `max-width` → `max-inline-size`
- `margin-left` → `margin-inline-start`
- `padding-right` → `padding-inline-end`
- `top` → `inset-block-start`
- `left` → `inset-inline-start`
## Z-index
Never leave magic z-index values. Use tokens from
`frontend/src/app/main/ui/ds/z-index.scss`, and pick the token **by what
the element is, not by its number**: a dropdown, menu or select →
`--z-index-dropdown`; a toolbar, palette, sidebar or panel →
`--z-index-panels`; and so on (read the comments in that file). A raw `4`
on a dropdown becomes `var(--z-index-dropdown)` even though the number
changes. Check the visual check for overlap.
Leave a TODO only when the element fits no token's purpose.
Legacy z-index values between sibling elements that do not overlap can be
removed, but only when the **Visual check** shows no pixel change in every
state that matters (open menus, collapsed, hidden). Say so in the report.
## Duplicated properties
Within the same selector, avoid duplicated properties, duplicated CSS
variables and duplicated declarations introduced by the refactor. Allow
duplicates only when they are intentional browser fallbacks.
## Unused classes
Open the `.cljs` file with the same name as the SCSS file and check whether
every CSS class is referenced (as `(stl/css :class-name)`,
`(stl/css-case :class-name ...)` or similar).
CSS modules prefix every class with its module
(`main_ui_<path>__<class>`), so only the `.cljs` of the same module can use
it. Before calling a selector unused, also search the whole `frontend/src`
for the class name as a string, a `:global` use, or a dynamic class built
at runtime.
If a selector is unused after that check:
- delete it (do not leave it commented out, do not add a TODO);
- list every deleted selector in the report and in the commit body.
A rule that matches nothing because the markup changed (e.g. `.icon svg`
when the DS icon is the `svg` itself) is deleted the same way. Before
deleting a descendant rule, check every element it matched, not only
the one you changed: `.placeholder .icon svg` can also color a second
icon, which then has no color (in the devenv it shows in the debug
yellow).
Every DS icon needs a color of its own (`color` on its class or on a
parent with a defined color): an icon that inherits nothing shows in
the debug yellow in the devenv.
Classes used by markup injected from outside (e.g. an external widget)
stay, with a comment saying so.
Also check the other direction: every class the `.cljs` passes to
`stl/css` / `stl/css-case` must exist in the `.scss`. A missing one silently
does nothing; report it.
## Sass and CSS pitfalls
- Sass does not evaluate functions inside CSS custom properties. Write
`--button-height: calc(var(--height) - #{px2rem(16)});`, not
`... - px2rem(16))`, or the browser gets the literal `px2rem(16)`.
- A `/` between Sass size variables is read as a division
(`grid-template: $sz-32 / auto` breaks). Split it into
`grid-template-rows` / `grid-template-columns` (or areas).
- A CSS-wide keyword cannot be part of a shorthand: `inset-inline: unset 0`
is invalid and stylelint does not catch it. Use the longhands.
- After editing, check the compiled CSS for any of these
(`frontend/resources/public/css/main.css`, search the module class).
## Uncertainty
If any transformation is uncertain, never guess. Leave a TODO in the code
and explain why in the report.
## Self-review
Before the report, read the final diff as the reviewer would and search
the file for these; each hit must be fixed or have a real reason:
```bash
rg -n 'svg \{|^\s*(h[1-6]|p|hr|input|button|span|a)\b[^-_]|[a-z]\.[a-z-]+ \{' <file>
rg -n 'stroke-width|color: transparent|fill: none|z-index: [0-9]|TODO' <file>
rg -n '[0-9]\.[0-9]+px|[^(]\b[0-9]+px' <file>
```
- an element selector → add a class (see **Element selectors**);
- a property that does not apply or is a default → drop it (see
**Placeholders**);
- a raw z-index → token by purpose;
- a typography TODO → closest DS typography;
- a raw px value → token or `px2rem()`.
## Verification
Run the project scripts from `frontend/` inside the devenv container
(`penpot-devenv-ws<N>-main` for workspace `wsN`; never run `pnpm` on the
host), writing output to a file first (never pipe lint output into
filters):
```bash
docker exec -u penpot -w /home/penpot/penpot/frontend penpot-devenv-ws<N>-main \
bash -lc 'pnpm run lint:scss; echo LINT=$?; pnpm run check-fmt:scss; echo FMT=$?' \
> <scratch>/scss-check.txt 2>&1
```
If the format check fails, run `pnpm run fmt:scss` and check again.
**Dependency check.** A grep for `refactor/` is not enough. For the final
file, check:
- `@use` / `@import` sources: all must be `ds/...` (or the sidebar common
module, which is DS-only);
- every `@include`, `@extend` and `%placeholder`;
- every `$variable`: which imported module defines it (`$br-*` exists in both
`ds/_borders.scss` and `refactor/borders.scss`; the `@use` decides);
- every `var(--token)`: defined in the DS (colors, spacing, sizes) or only as a
`refactor/` alias.
Report any failure as it is; do not hide it.
## Visual check
"No visual change" must be shown, not assumed. When a visual spec covers the
component (`frontend/playwright/ui/visual-specs/`, project `ds`), or one can be
added:
1. Capture before (base branch styles) and after with
`pnpm exec playwright test --project=ds -g "<test>" --update-snapshots=all`
in the container. `=all` is required: plain `--update-snapshots` only
rewrites past the 0.5% tolerance, so a real diff can look like 0.
2. Before each capture, wait until the compiled CSS/JS shows the expected
version (search `main.css` for the changed rule).
3. Compare with zero tolerance. If pixels change, build the before | after |
diff (red) image and explain every change in the report.
## Output
1. The diff of the file (`git diff -- <file>`).
2. A summary with:
- removed imports;
- removed deprecated mixins;
- removed placeholders;
- typography replacements;
- design-token aliases removed;
- logical properties introduced;
- deleted unused selectors;
- TODOs requiring manual review, each with its reason;
- remaining `refactor/` dependencies (partial refactor) and where they are
used;
- classes missing from the `.scss`, if any.
3. The verification results, including the visual check (pixel count, or why
it was not possible).

312
.agents/skills/update-changelog/SKILL.md
View File

@ -5,16 +5,18 @@ description: Update the project CHANGES.md with issues from a given GitHub miles
# Skill: update-changelog
Update `CHANGES.md` with entries for all issues and PRs in a given GitHub
milestone. Each entry references the user-facing issue (not the PR) as the
primary link, with the fix PR inline on the same line.
Entry point for all changelog work. All rules — sections, exclusions, entry
format, the `:boom:`, `:rocket:` and `:lock:` contracts, pre-flight, and the
anomaly report — live in `mem:workflow/updating-changelog` (file:
`.serena/memories/workflow/updating-changelog.md`). This skill routes to that
memory; it does not repeat it.
## When to Use
- Before a new release, to populate the changelog with all fixed issues
- When new issues are added to an existing milestone and the changelog needs
to be refreshed
- To ensure every entry follows the correct format for the changelog
- To audit or repair an existing version section
## Prerequisites
@ -23,289 +25,33 @@ primary link, with the fix PR inline on the same line.
- `scripts/gh.py` for GitHub queries, `scripts/changelog.py` for changelog
checks (run `python3 scripts/changelog.py <command> --help` for usage)
## Workflow
## What to Read
### 1. Determine the target version
1. `mem:workflow/updating-changelog` — the canonical flow: sources of truth,
layout, entry rules, the `## Breaking change note` comment contract, the
advisory (`:lock:`) contract, pre-flight checks, anomaly types, and pitfalls.
2. `mem:scripts/changelog` — `boom`, `security`, `check-merged`, `cross-ref`,
`report`.
3. `mem:scripts/gh` — `issues`, `issue`, `comments`, `prs`, `advisories`.
The version is typically a semver string like `2.15.3`. Confirm with the user
if not specified.
### 2. Fetch all issues in the milestone
## Commands
```bash
# All closed issues (default)
python3 scripts/gh.py issues "2.16.0"
# Include open issues too
python3 scripts/gh.py issues "2.16.0" --state all
# Exclude entries that should not go in the changelog
python3 scripts/gh.py issues "2.16.0" --exclude "release blocker,no changelog"
python3 scripts/gh.py issues "<VERSION>" --exclude "release blocker,no changelog"
python3 scripts/gh.py advisories --patched-version "<VERSION>"
python3 scripts/changelog.py check-merged <PR numbers...>
python3 scripts/changelog.py boom "<VERSION>" # print the :boom: section
python3 scripts/changelog.py boom "<VERSION>" --write # replace it in CHANGES.md
python3 scripts/changelog.py security "<VERSION>" # print the :lock: section
python3 scripts/changelog.py security "<VERSION>" --write # replace it in CHANGES.md
python3 scripts/changelog.py cross-ref "<VERSION>"
python3 scripts/changelog.py report "<VERSION>"
```
**Exclusion rules (issue-level):**
- `no changelog` label — chore/refactor work, no entry needed
- `release blocker` label — blocked issues not yet ready for changelog
- `Task` issue type — internal chores, not user-facing; excluded by `gh.py`
(use `--include-tasks` to override)
- **Rejected project status** — issues with "Rejected" status on the "Main"
project board are excluded by `gh.py` (use `--include-rejected` to override).
This status is independent of the GitHub issue `state`.
## Hard Rules
**Exclusion rules (PR-level):** PRs with these labels stay out regardless of
their linked issue's labels: `release blocker`, `no issue required`.
Each entry carries `number`, `title`, `state`, `issue_type`, `labels`,
`closing_prs`, and `project_status`.
### 3. Identify missing entries (optional)
```bash
python3 scripts/gh.py issues "2.16.0" --exclude "release blocker,no changelog" --compare CHANGES.md
```
Returns only milestone issues not yet referenced in the changelog. Note: it
compares **issues** only. For unreferenced merged **PRs**, use the
cross-reference in step 8.
### 4. Fetch additional PR details when needed
```bash
# One or more PR numbers (also: --file prs.txt, or --stdin)
python3 scripts/gh.py prs 9179 9204 9311
# All merged PRs in a milestone (default); --state all/open/closed for others
python3 scripts/gh.py prs --milestone "2.16.0"
```
Returns `number`, `title`, `body`, `state`, `merged_at`, `author`, `labels`,
and `closing_issues`. Milestone mode uses paginated GraphQL (100 per page).
### 5. Categorize entries — strictly by issue type, never by labels or emoji
Use the **Issue Type** field (`issue_type` in the `gh.py` output). No separate
query is needed.
> **⚠️ CRITICAL: Never use labels or title emoji prefixes for categorization.**
> Labels like `bug`/`enhancement` and prefixes like `:bug:`/`:sparkles:` are
> often wrong or missing. `issue_type` is the single source of truth.
| `issue_type` value | Changelog section |
|--------------------|-------------------|
| `Bug` | `### :bug: Bugs fixed` |
| `Feature` or `Enhancement` | `### :sparkles: New features & Enhancements` |
| `Task` | **Exclude** — internal chores, not user-facing |
| `null` (not set) | Fallback to labels: `bug` label → bugs, otherwise enhancements |
**Breaking changes override everything:** an issue with the `breaking change`
label goes under `### :boom: Breaking changes & Deprecations`, no matter its
issue type. This and `release highlight` below are the only label-based
rules — explicit exceptions to the "never by labels" principle above. The
`:boom:` subsection goes first in the version, before `:rocket:` (matching
the existing precedent).
**Highlights come from the `release highlight` label:** an issue with that
label also gets an entry under `### :rocket: Epics and highlights`, on top of
its regular entry (usually `:sparkles:`). In the section being refreshed,
`:rocket:` lists exactly the labelled issues: never pick highlights yourself
and never keep an unlabelled entry there. Product decides the highlights by
labelling issues on GitHub, so a regeneration cannot undo that choice. Older
version sections are left as they are.
**Community attribution:** if the issue or its fix PR has the
`community contribution` label, add `(by @<github_username>)` on the entry
line, **before** the issue/PR references. Use the **PR author** (the `author`
field from step 4), not the issue author:
```markdown
- Fix description of the bug (by @username) [#<ISSUE>](...) (PR: [#<PR>](...))
```
**Only closed issues are included**, even if open ones are tracked in the
milestone.
**Pairing rules:**
| Pattern | Changelog format |
|---------|-----------------|
| Closed issue + one or more fix PRs | Primary link = issue, PRs inline comma-separated |
| PR with no linked issue | Link the issue if a matching closed one exists in the milestone; otherwise skip (the issue is the changelog unit) |
| Closed issue with no fix PR in milestone | Link the issue directly, no PR reference |
> **False-positive associations:** a PR may wrongly claim to close an issue
> from another context (ancient PR, cross-project reference). If titles are
> clearly unrelated or the PR predates the issue by years, treat it as a data
> glitch and skip it.
### 5a. Verify PR merge status before writing
A closed issue may list closing PRs that were **closed without merging**
(e.g. a superseded community PR). Only **merged** PRs go in the changelog:
```bash
python3 scripts/changelog.py check-merged <ALL_PR_NUMBERS>
# also accepts: --file prs.txt, or numbers via --stdin
```
If a closing PR is closed-unmerged, find the merged PR that superseded it
(other PRs in the issue's closing list, similar titles, or pointers in the
closed PR's timeline) and reference that one instead.
### 5b. Security advisory (GHSA) entries
Advisories fixed in a release go in the changelog even though they are
**neither milestone issues nor PRs**. The GHSA ID and description come from
the user or the release notes — never from the milestone fetch.
```markdown
- Fix <user-facing description> (https://github.com/penpot/penpot/security/advisories/GHSA-XXXX-XXXX-XXXX)
```
Rules: place under `### :bug: Bugs fixed` with **no issue or PR link**; do
**not** fetch or verify the URL (it may be draft/unpublished and 404); write
the description in imperative mood from the advisory title. These entries are
invisible to the automation — add them by hand, and in step 9 apply only the
backport/duplicate check to them.
### 6. Read the current CHANGES.md and run pre-flight checks
Newest version goes at the top, right after the `# CHANGELOG` header:
```markdown
## <VERSION>
### :boom: Breaking changes & Deprecations
- <breaking change or deprecation> [#<ISSUE>](https://github.com/penpot/penpot/issues/<ISSUE>) (PR: [#<PR>](https://github.com/penpot/penpot/pull/<PR>))
### :bug: Bugs fixed
- Fix description of the bug [#<ISSUE>](https://github.com/penpot/penpot/issues/<ISSUE>) (PR: [#<PR>](https://github.com/penpot/penpot/pull/<PR>))
- Fix another bug (by @contributor) [#<ISSUE>](https://github.com/penpot/penpot/issues/<ISSUE>) (PR: [#<PR>](https://github.com/penpot/penpot/pull/<PR>))
### :sparkles: New features & Enhancements
- Add new feature description [#<ISSUE>](https://github.com/penpot/penpot/issues/<ISSUE>) (PR: [#<PR>](https://github.com/penpot/penpot/pull/<PR>))
```
Format details: entries start with `- ` plus a short imperative description;
PR refs stay inline (`(PR: [#<N>](<url>))`, comma-separated for several);
`(by @<username>)` goes before the issue link; only include non-empty
sections; blank line between a section's last entry and the next title; never
duplicate an entry from an earlier version section (the earlier version wins
for backports).
**Pre-flight checks — fix violations directly in `CHANGES.md` before writing
the new section.** Reconcile every existing entry (any version section) and
every milestone candidate against the current milestone state (re-fetch, do
not trust cached data):
1. **Duplicate across versions** → remove from the current section.
2. **Stale milestone assignment** (issue moved out of this milestone) →
remove the entry (or drop it if the section does not exist yet).
3. **Newly applied exclusion labels** (`no changelog`, `release blocker`) →
remove the entry.
4. **Issue no longer closed/deleted/Rejected** → remove the entry.
5. **Unmerged or moved PR reference** → fix the reference or remove the
entry (a PR merged in a *different* milestone is a step-9 anomaly, do not
silently remove it).
6. **Issue type changed to `Task`** → remove the entry.
7. **Breaking change misplaced** (issue has the `breaking change` label but
sits in another section) → move the entry to `:boom:`.
8. **Highlight out of sync** (`:rocket:` entry without the `release
highlight` label, or a labelled issue missing from `:rocket:`) → remove
the `:rocket:` entry or add it (step 7b). The regular entry stays.
9. **Missing valid issues** (closed, non-excluded, unreferenced anywhere) →
add them to the current section per step 5.
### 7. Build the description text
Derive it from the **issue title**, not the PR title. Strip leading emoji
prefixes (`:bug:`, `:sparkles:`, `:tada:`) and describe the user-facing
behavior:
| Issue title | Changelog description |
|-------------|----------------------|
| `Plugin API token methods fail with schema validation error on PRO` | `Fix Plugin API token methods failing with schema validation error on PRO` |
| `Comment content is not sanitized before rendering, enabling stored XSS` | `Sanitize comment content on rendering` |
| `Custom uploaded font family names are not sanitized` | `Sanitize font family names on custom uploaded fonts` |
Insert the new version section right after the `# CHANGELOG` header with the
`edit` tool and enough context for a unique match.
### 7b. Populate `:rocket: Epics and highlights` from the label
List the milestone issues that pass step 2 exclusions and carry the
`release highlight` label:
```bash
python3 scripts/gh.py issues "2.16.0" --exclude "release blocker,no changelog" \
| python3 -c "import sys,json; [print(i['number'], i['title']) for i in json.load(sys.stdin) if 'release highlight' in i['labels']]"
```
Create the subsection if missing (place it before `### :sparkles:`) with one
entry per labelled issue, same text and references as its regular entry.
Every `:rocket:` entry MUST carry issue AND PR references. If no issue is
labelled, leave `:rocket:` out and tell the user: the highlights are
product's call, not the agent's. To change them, add or remove the label on
GitHub and re-run; never edit `:rocket:` by hand.
### 8. Cross-reference milestone PRs against the changelog
```bash
python3 scripts/changelog.py cross-ref "<MILESTONE>" [--changes CHANGES.md]
```
Lists merged milestone PRs missing from the changelog section (decide per PR:
add it or confirm its exclusion labels) and warns about CLOSED (unmerged) PRs
in the milestone. GHSA entries never appear here — that absence is expected.
### 9. Generate the anomaly report
```bash
python3 scripts/changelog.py report "<MILESTONE>" [--changes CHANGES.md --output CHANGES-ISSUES.md]
```
Overwrites `CHANGES-ISSUES.md` with the current state. Every number renders as
a full `[#N](https://github.com/penpot/penpot/issues/N)` or
`[#N](https://github.com/penpot/penpot/pull/N)` link.
**An anomaly is a milestone mismatch or a `:boom:` mislabel** (the changelog
pairing is misleading and a human must judge intent):
1. Issue in this milestone, referenced PR in another milestone (or none).
2. PR in this milestone, closed issue in another milestone — except an issue
with *no* milestone, which belongs to another (probably private) project
and is neither anomaly nor changelog candidate.
3. `:boom:` entry whose issue lacks the `breaking change` label. These stay
listed in the report **and** in the changelog: either label the issue or
move the entry to its regular section.
4. `:rocket:` out of sync with the `release highlight` label. Step 6
pre-flight should have fixed it: if it shows up, re-run the workflow.
**Highlight gaps are warnings, not anomalies:** missing `:rocket:` on a
released X.Y.0 (patches never carry highlights); `:rocket:` entry without
issue AND PR references. Gaps never count toward the anomaly total.
**Anything else is a rule violation**, not a report item: fix it in step 6
pre-flight. If one shows up in the report, re-run the workflow.
## Key Principles
- **Issue = changelog unit**, PR = implementation detail inline.
- **Latest version first**, below the `# CHANGELOG` header.
- **Issue Type decides the section — exclusively**, except `breaking change`
label → `:boom:` first, and `release highlight` label → also `:rocket:`.
- **User-facing descriptions** in imperative mood.
- **Community attribution** uses the PR author, placed before the issue link.
- **Only closed issues**; **Rejected** project status excludes.
- **`no changelog` and `Task`** stay out.
- **Multiple fix PRs** go comma-separated inline.
- **Duplicates**: the earlier version section wins.
- **Taiga references**: resolve to the GitHub issue via description text or
PRs mentioning the Taiga URL, then link issue + PR.
- **GHSA entries** live under `:bug:` with only the advisory URL (see 5b).
- **Re-fetch before editing**; prefer `scripts/gh.py` over raw `gh api`.
- **Verify PR merge status** (`check-merged`); PR-level exclusions apply.
- **Cross-reference PRs, not just issues** (`cross-ref`); watch for
false-positive PR-to-issue links.
- `CHANGES.md` is edited only on explicit request or during the release process.
- Only closed, non-excluded milestone issues are included; the issue is the changelog unit and the PR is inline.
- `:boom:` and `:rocket:` are label-generated and `:lock:` is advisory-generated: never hand-edit them; edit the label, the `## Breaking change note` comment or the advisory and re-run the command.
- Verify PR merge status before writing; only merged PRs are referenced.
- Fill `patched_versions` in the advisories before running `security`; unpublished advisories are valid.

4
.github/workflows/build-docker-devenv.yml vendored
View File

@ -38,13 +38,13 @@ jobs:
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
- name: Login to Docker Registry (push destination)
uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
with:
username: ${{ secrets.PUB_DOCKER_USERNAME }}
password: ${{ secrets.PUB_DOCKER_PASSWORD }}
- name: Login to Docker Hardened Images registry (base image pull)
uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
with:
registry: dhi.io
username: ${{ secrets.PUB_DOCKER_USERNAME }}

10
.github/workflows/build-docker.yml vendored
View File

@ -200,7 +200,7 @@ jobs:
ref: ${{ inputs.sha != '' && inputs.sha || inputs.gh_ref }}
- name: Login to Docker Registry
uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
with:
registry: ${{ secrets.DOCKER_REGISTRY }}
username: ${{ secrets.DOCKER_USERNAME }}
@ -210,7 +210,7 @@ jobs:
# images from DockerHub for unregistered users.
# https://docs.docker.com/docker-hub/usage/
- name: Login to DockerHub Registry
uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
with:
username: ${{ secrets.PUB_DOCKER_USERNAME }}
password: ${{ secrets.PUB_DOCKER_PASSWORD }}
@ -222,7 +222,7 @@ jobs:
# reuses the same PUB_DOCKER_* credentials as the DockerHub
# login above.
- name: Login to Docker Hardened Images registry (base image pull)
uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
with:
registry: dhi.io
username: ${{ secrets.PUB_DOCKER_USERNAME }}
@ -301,7 +301,7 @@ jobs:
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
- name: Login to Docker Registry
uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
with:
registry: ${{ secrets.DOCKER_REGISTRY }}
username: ${{ secrets.DOCKER_USERNAME }}
@ -371,7 +371,7 @@ jobs:
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
- name: Login to Docker Registry
uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
with:
registry: ${{ secrets.DOCKER_REGISTRY }}
username: ${{ secrets.DOCKER_USERNAME }}

12
.github/workflows/build-renovate.yml vendored
View File

@ -114,20 +114,20 @@ jobs:
persist-credentials: false
- name: Login to Docker Registry
uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
with:
registry: ${{ secrets.DOCKER_REGISTRY }}
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Login to DockerHub Registry
uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
with:
username: ${{ secrets.PUB_DOCKER_USERNAME }}
password: ${{ secrets.PUB_DOCKER_PASSWORD }}
- name: Login to Docker Hardened Images registry
uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
with:
registry: dhi.io
username: ${{ secrets.PUB_DOCKER_USERNAME }}
@ -191,13 +191,13 @@ jobs:
persist-credentials: false
- name: Login to DockerHub Registry
uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
with:
username: ${{ secrets.PUB_DOCKER_USERNAME }}
password: ${{ secrets.PUB_DOCKER_PASSWORD }}
- name: Login to Docker Hardened Images registry
uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
with:
registry: dhi.io
username: ${{ secrets.PUB_DOCKER_USERNAME }}
@ -242,7 +242,7 @@ jobs:
persist-credentials: false
- name: Login to Docker Hardened Images registry
uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
with:
registry: dhi.io
username: ${{ secrets.PUB_DOCKER_USERNAME }}

2
.github/workflows/lint-workflows.yml vendored
View File

@ -34,7 +34,7 @@ permissions: {}
env:
ACTIONLINT_VERSION: '1.7.12'
ZIZMOR_VERSION: '1.31.1'
ZIZMOR_VERSION: '1.32.0'
jobs:
lint:

2
.github/workflows/tests-wasm.yml vendored
View File

@ -49,7 +49,7 @@ jobs:
- name: Format
working-directory: ./render-wasm
run: |
cargo fmt --check
./fmt --check
- name: Lint
working-directory: ./render-wasm

8
.serena/memories/audit-log.md
View File

@ -182,7 +182,7 @@ Who uses it:
Failure policy: every send is best-effort. A failure never rolls back the business action; it is only recorded for debugging.
Backend side (`backend/src/app/rpc/management/nitrate.clj` `::push-audit-events`): `::audit/skip`, `::rpc/auth false` (trust comes from the shared key, not from a user session). It builds context with `audit/prepare-context-from-request` + `request-id`, takes the given `profile-id` as-is (no coercion here, must already be a uuid), sets `tracked-at = request-at`, defaults `type` to `"action"`, and calls `audit/submit` (which defaults `source` to `"backend"`). So these rows land as **`source = backend`**: they are audited like backend events because a trusted service set the author.
Backend side (`backend/src/app/rpc/management/nitrate.clj` `::push-audit-events`): `::audit/skip`, `::rpc/auth false` (trust comes from the shared key, not from a user session; only the `admin-console` key id is allowed). It builds context with `audit/prepare-context-from-request` + `request-id`, takes the given `profile-id` as-is (no coercion here, must already be a uuid), sets `tracked-at = request-at`, defaults `type` to `"action"`, and calls `audit/submit` (which defaults `source` to `"backend"`). So these rows land as **`source = backend`**: they are audited like backend events because a trusted service set the author.
### 7b. Browser channel (best-effort, like frontend events)
@ -252,9 +252,9 @@ Result: crash reports only exist if the frontend collector runs and `push-audit-
## 11. Consumers III: telemetry (shadow rows)
Telemetry reuses the same table with **anonymized shadow rows** (`source LIKE 'telemetry:%'`): day-truncated times, `0.0.0.0` IPs, props reduced to uuid/boolean/number values plus a few allowed fields (`lang`, `auth-backend`, derived `email-domain` — never raw emails), and a minimal context allowlist. Full + shadow rows can exist together per event when both flags are on; that duplication is on purpose.
Telemetry reuses the same table with **anonymized shadow rows** (`source LIKE 'telemetry:%'`): day-truncated times, `0.0.0.0` IPs, props reduced to uuid/boolean/number values plus a few allowed fields (`source`, `lang`, `auth-backend`, `activation`, derived `email-domain` — never raw emails), and a minimal context allowlist. Full + shadow rows can exist together per event when both flags are on; that duplication is on purpose.
Filters (`filter-telemetry-props` / `filter-telemetry-context`): by default only uuid/boolean/number with a simple key; exceptions: login/register/update-profile and frontend `identify` keep `lang`/`auth-backend`/`email-domain`; `organization-sso-auth-failed` keeps `failure-reason` if it is a known one; `instance-start` passes whole; frontend `navigate` keeps `[:route :file-id :team-id :page-id]`. Free text drops from telemetry but **stays in the full row and in Nexus**.
Filters (`filter-telemetry-props` / `filter-telemetry-context`): by default only uuid/boolean/number with a simple key; exceptions: login/register/update-profile and frontend `identify` keep `lang`/`auth-backend`/`source`/`activation`/`email-domain`; `organization-sso-auth-failed` keeps `failure-reason` if it is a known one; `instance-start` passes whole; frontend `navigate` keeps `[:route :file-id :team-id :page-id]`. Subscription actions have event-specific contracts: backend and frontend `create-subscription` / `create-trial-subscription` preserve the default uuid/boolean/number props plus `type`, `deployment`, `quantity` and `subscription-mode`; backend `update-subscription` preserves only `type`, `deployment`, `previous-status`, `status`, `previous-cancel-at-period-end`, `cancel-at-period-end`, `previous-cancel-at`, `cancel-at` and `changed-fields`; backend `subscription-summary` preserves only `type`, `deployment`, `instance-id`, `snapshot-date`, the per-status `*-count` values, `total-count`, `sync-succeeded`, `sync-failed` and `snapshot-complete`. These subscription exceptions apply only to action events. Other free text drops from telemetry but **stays in the full row and in Nexus**. In `register-profile`, `activation` is a prop (it is what reaches PostHog). `action` exists only in the `context` of the full `register-profile-retry` rows; `filter-telemetry-context` drops it from the anonymised copy, which keeps only `:version`, `:initiator`, `:client-version` and `:client-user-agent`.
Shipping (`backend/src/app/tasks/telemetry.clj`): the cron sends shadows to `:telemetry-uri` as JSON in 10k batches and **deletes** them on success; it purges leftovers older than 7 days. It also sends a legacy report (counters, teams, email domains...). On official hosts nothing is collected or sent (`telemetry-excluded?` covers `penpot.app`/`penpot.dev`).
@ -301,7 +301,7 @@ So the full journey is: Penpot buffer → Nexus `audit_log` (via either archive
Penpot → admin-console archiving lives in its own job (`:audit-log-consumer-admin-console`, consumer `2`, flag `:admin-console`, params closed empty): it drains its own pending queue and ingests allowlisted events through `nitrate/call :ingest-audit-log`, enriched as described above. What also exists (do not confuse with archiving):
- **Nitrate → Penpot**: `POST /api/management/methods/push-audit-events` (`backend/src/app/rpc/management/nitrate.clj`) lets the trusted Nitrate backend inject events into the Penpot audit log (no RPC auth, `::audit/skip`, request context + `submit`). Section 7a is the main user.
- **Nitrate → Penpot**: `POST /api/management/methods/push-audit-events` (`backend/src/app/rpc/management/nitrate.clj`) lets the trusted Nitrate backend inject events into the Penpot audit log (no RPC auth, `::audit/skip`, request context + `submit`; only the `admin-console` shared-key id can call it). Section 7a is the main user.
- **Admin-console browser → collector**: usage events from the admin-console UI traveling through the normal public `push-audit-events` (section 7b).
- **Penpot frontend Nitrate bits**: membership/subscription events (`frontend/src/app/main/data/nitrate.cljs`, `nitrate-audit/*`) through the same public endpoint.

4
.serena/memories/backend/auth-permissions-product-domains.md
View File

@ -14,6 +14,7 @@
- Invitation tokens are verified through token issuers and only accepted when the token member id/email matches the authenticated profile; otherwise login proceeds without consuming the invitation.
- When public registration is disabled, invitation-based password registration also requires a matching, non-expired `team_invitation` row; registration revalidates it under a row lock before creating the profile.
- Access tokens (`app.http.access-token`): personal tokens have JWT issuer `access-token` and DB type other than `mcp`; MCP tokens have issuer `urn:penpot:mcp-token` (`actoken/mcp-token-issuer`) and DB type `mcp`. Authentication requires issuer and type to agree, and the claim `uid` to match the row's profile; other issuers and undecryptable tokens are treated as anonymous.
- Management RPC (`/api/management/methods/:name`) is shared-key only. The middleware authenticates `<id> <secret>`; the RPC wrapper `wrap-shared-key-authorization` authorizes the key id against the method's `::rpc/allowed-shared-keys` set. Absent metadata accepts any configured key; all current methods declare one (`exporter`/`media-processor` for `upload-tempfile`, `nexus` for file analysis, `exporter` for jobs, `admin-console` for Nitrate). A valid key outside the set answers 403 `:shared-key-not-allowed`; denied calls do not reach the audit wrapper.
- Personal tokens need flag `:access-tokens`. MCP tokens need `:mcp` or `:access-tokens`, and only on routes whose `actoken/authz` gets `::actoken/allow-mcp true` (the RPC routes); elsewhere (e.g. assets) they are anonymous.
- On RPC, an MCP token reaches only commands with `::rpc/token-type "mcp"` (`wrap-authentication` in `app.rpc`); every other command answers 403 `:token-auth-required`. A command with that metadata also rejects sessions and personal tokens. Currently only `send-mcp-feedback`. Blocking a profile invalidates sessions, not tokens, so such commands check `:is-blocked`/`:is-active` themselves.
- HTTP/session parsing details such as cookie/header precedence, JWT session token versions, and SameSite behavior are in `mem:backend/subtleties`.
@ -24,12 +25,13 @@
- Team role flags are normalized as owner > admin > editor > viewer. Owner/admin imply edit; any membership row implies read.
- File/project role rows only count with a live team membership: the file/project permission queries require a matching `team_profile_rel` row, and leaving or removing a member deletes their team-scoped `file`/`project`/`team-project-profile-rel` rows (GHSA-v9r9-h77c-55m2).
- File/project/comment checks are implemented in the owning command namespaces, often via helpers imported from `files`, `teams`, or `projects`; do not bypass those helpers with direct DB lookups unless preserving their not-found semantics.
- Comment permission includes both logged-in state and the file/team comment policy. Shared viewer paths may pass `share-id`; preserve that path when changing comment queries.
- Comment permission includes both logged-in state and the file/team comment policy. Shared viewer paths may pass `share-id`; preserve that path when changing comment queries. RPCs that accept `share-id` enforce the link's page scope before mutating (`share-link-pages` + `check-page-scope`): a thread or page outside the link raises `:not-found`, and that check runs before the ownership checks. `get-profiles-for-file-comments` filters its query by the same pages. `mark-all-threads-as-read` does not take `share-id`.
## Teams, projects, and invitations
- Team/project commands mix DB changes, email, message bus notifications, media/storage cleanup, feature flags, quotas, and audit metadata. Keep mutations transactional when the existing command does so.
- Invitation flows validate muted/bounced emails before sending and use tokenized invitation state. Accepting an invitation is tied to the invited member identity, not just possession of a token.
- The whole invitation family is team-admin only: creating, updating and deleting invitations, minting the invitation token (`get-team-invitation-token`) and listing the pending ones (`get-team-invitations`, which carries the invited email and the offered role). The token of an invitation that grants the `owner` role is restricted to the team owner, because an admin cannot create nor promote an owner invitation (`cant-promote-to-owner`). This family is the exception to the rule above: it answers `:validation` / `:insufficient-permissions` instead of `:not-found`, so the five commands speak one language.
- Logical deletion is used for many product objects; prefer existing logical-deletion helpers over hard deletes unless the command already performs permanent cleanup.
- Bounced/spam-complaint emails can mute/block a profile for login/registration and email sending. Devenv Mailpit is the normal local path for registration/email-flow testing.

1
.serena/memories/backend/core.md
View File

@ -67,6 +67,7 @@ Every background job is a job-def: a plain `(defn execute-X [cfg params] ...)` i
- Job resources (`app.jobs.storage`) are the objects a job owns through `job.resource_id`: they live in the `job-resource` bucket, are written with the profile of the job as owner (only that profile can read them back from `/assets/by-id`) and are never deduplicated. They carry no expiry of their own, because `expired-at` is stored as `deleted_at` and an object with it is unreadable from birth: the retention of the artifact is the retention of its job row. The writer marks the object as touched right away, so an artifact that no row references (a crash between the upload and the completion) is reclaimed by `storage-gc-touched`, and a row that does reference it freezes it until the jobs GC deletes the row and touches the object.
- The binfile jobs (`:export-binfile` and `:import-binfile`) run the core of `bf.v1`/`bf.v3` through `app.binfile.jobs`, which turns the `:progress` taps of the core into `progress` events of the job with `jobs/heartbeat` (the `modified_at` touch at most once per second, progress events at up to ~200ms). Each event is a self-contained milestone: a `:stage` keyword of one vocabulary shared by the two formats (the same work has a different name in v1 and v3) and an optional `:counters` map, with the counter of the activity named after its own stage and the outer scopes in course as context, which the tap carries itself (the reader puts `::outer-counters` in every tap inside them — today only `:files`, extendable to `:teams`/`:projects` for multi-scope exports — so the adapter remembers nothing between taps and a tap that happens outside any scope has no context); the ids and names the taps carry never reach the event. Every tap is reported, so a tap with no units to count still keeps the job alive, and a milestone the throttle drops loses nothing because the next one carries the same state. The adapter checks that the job is still active before the run, and the beat raises an `:interrupt` right there on the run thread when the job is no longer active, so the work stops at its next event point inside the transaction of its handler, which rolls back. A run whose last tap predates the interruption is discarded by the check once it returns. The adapter passes the job id from the handler context. Their job-def components carry pool, storage, metrics and msgbus, because a job with a `profile_id` cannot store an event without a msgbus on its cfg. Both readers answer with the same shape (`:file-ids` and, in version 3, the resolution of the libraries), because the version 1 reader returns a bare set of file ids.
- Both binfile handlers revalidate their permission at run time, not only when the job is created, because it can be revoked while the job waits: the export checks the read permission of every file before reading anything and stores the package in `job-resource` as the resource of the job (answering with the completion envelope), and the import checks the edition permission of the destination and runs the core inside a transaction, which is what makes a cancelled import roll back instead of leaving half of a file. The import releases its package (marks it as deleted and drops the reference from the job row in the same transaction) and deletes its temporary copy once the job is terminal, whatever the outcome. Both handlers check their own result against a schema before returning it (the import job-def declares that same schema as its result schema, which is what types the resolution when the row is read).
- The v3 import only materializes a library relation when both ids belong to the package and differ; a relation that points outside the package or a self link is ignored with a warning. Migration `0158-del-forbidden-file-library-rels` deletes the legacy cross-team and self rows from `file_library_rel` and their `file_library_sync` rows, sync rows first.
- The domain cfg of a binfile request lives in `app.binfile.common` (`export-cfg`, `import-cfg`), so the legacy RPC and the jobs freeze the same ids, resolve the same team features and apply the same anti-zip-bomb limits. `resolve-export-type` is the translation the legacy RPC needs for its deprecated boolean flags; the creation command asks the caller for the type instead of guessing a default.
- Two retention knobs, easy to confuse: `:jobs-retention` sweeps terminal internal rows (the ones without a profile), while `:jobs-user-ttl` is the default expiry a user-facing job gets when it is created; the jobs GC deletes the row once `expires_at` passes and touches its resource. The heavy work of a user job runs on its own `:binfile` queue (`:worker-binfile-parallelism`, default 1) so it does not block the default runner.
- The quote of a family of jobs (`::export-jobs-per-profile`, `::import-jobs-per-profile`) counts the jobs of the profile that are still on their way (`new`/`scheduled`/`running`/`retry`) whose name the registry gives for that family, so a new job of the family is covered without touching the quote.

9
.serena/memories/backend/subtleties.md
View File

@ -56,7 +56,7 @@
- Request body parsing is mostly POST-oriented and supports Transit JSON plus plain JSON. Plain JSON request keys are kebab-decoded before being merged into `:params`. A body the parser cannot read is a client error, classified at the parse site and never by exception class (`clojure.data.json` signals its syntax errors with a bare `java.lang.Exception`): 400 `:validation`/`:malformed-json`, one `warn` line (`malformed request body`) carrying path, peer address and content-type instead of a stack trace, and `penpot_http_malformed_requests_total`. `RequestTooBigException` keeps its 413.
- Response formatting negotiates with `Accept` or `_fmt=json`. Transit is the default for collection/boolean bodies; JSON encoding has special pointer-map handling.
- Auth prefers the session cookie token before the `Authorization` header. Headers may be `Token` or `Bearer`. Only `kid=1`/`ver=1` tokens are decoded as session tokens; anything else is left unauthenticated (legacy v1 tokens were removed).
- Shared-key auth requires `x-shared-key` as `<key-id> <key>` and stores the lowercased key id on the request. If no shared keys are configured it always rejects.
- Shared-key auth requires `x-shared-key` as `<key-id> <key>` and stores the lowercased key id on the request. If no shared keys are configured it always rejects. Management methods declare the key ids they accept with `::rpc/allowed-shared-keys`; a valid key outside that set answers 403 `:shared-key-not-allowed`, and a method without the metadata accepts any configured key.
- Session management uses DB storage unless the DB pool is read-only, then falls back to the in-memory manager. Sessions use only the v2 UUID model (`http_session_v2`); legacy string ids were removed.
- Session cookies are renewed when `modified-at` is older than the 6h renewal interval. SameSite is `none` for CORS, otherwise strict/lax based on config. Session lifetime config and GC: `mem:backend/session-expiration`.
@ -84,6 +84,8 @@
- The `media-storage-bytes-per-team` quota check on `upload-file-media-object` is TEMPORARILY DISABLED (2026-10-19, kept commented in `app.rpc.commands.media` with its `sql:get-team-id-for-file` lookup still active for the audit props): the quote performs a full `SUM` over `storage_object` per upload and was too slow in production. The `upload-sessions-per-profile` quota is unaffected and stays active. Re-enable only with a cheaper approximation.
- Storage abstraction, backend configuration, logical buckets, object lifecycle, deduplication, access rules, and garbage collection: `mem:backend/storage`.
- SVG validation strips DOCTYPE and uses secure SAX parsing. Basic SVG info falls back to 100x100 dimensions when width/height/viewBox are missing.
- The SVG sanitizer (`app.media.svg`) is a **blocklist**, not an allowlist: it drops dangerous elements and attributes by name and escapes every text and attribute value when it writes the tree back, so a payload hidden in a value cannot become live markup again. The allowlist rework is [#12190](https://github.com/penpot/penpot/issues/12190) and lands in media-processor, not here. `sanitize-svg` also tracks the namespace bindings through the recursion and drops any element that is not in the SVG namespace, so a document whose root is not an `svg` element in the SVG namespace is rejected and a child that rebinds `xmlns` to XHTML (`iframe`/`srcdoc`/`meta`) is dropped with its subtree; it compares the scheme of an URL the way the browser does (dropping tab/CR/LF/spaces), so `java&#x09;script:` does not slip through the `javascript:` filter. The paths that store an uploaded SVG as its final object go through `app.media.svg/sanitize-svg-upload` (media upload, the URL download, both thumbnail writers and the organization logo); the three binfile imports go through `app.binfile.common/sanitize-imported-svg`, which calls `sanitize-svg` directly. The profile picture is not an SVG: `import-profile-picture` ignores an `image/svg+xml` download, because the avatar is always rendered as an image. The caller checks the size before calling it: the helper reads the whole file, and the size limit applies to the upload, not to the sanitized copy, which can be up to six times bigger.
- Upload handlers run the `media.v` checks in a fixed order: `check-upload-path`, then `check-media-type`, then `check-media-size` (or `check-font-size` for fonts). The path check gates every later read: `check-media-type` sniffs the bytes at `:path` and `check-media-size` reads `fs/size`, so a path the client named outside `allowed-upload-prefixes` must be rejected before either touches the disk. `check-media-size` leans on that and answers a file it cannot read with the path code (`:params-validation`), not `:media-max-file-size-reached`. Keep the order when touching or adding a handler; two branches that reorder these lines conflict.
- Raster metadata is shell-derived with ImageMagick `identify`, verifies detected MIME against the supplied MIME, and swaps dimensions for EXIF orientations 6/8.
- Remote image download requires 2xx status, `content-length`, a known MIME, and size under the configured maximum before writing the temp file; mismatched byte count is an internal error.
- Font processing shells out to FontForge and WOFF conversion tools and can derive TTF/OTF/WOFF variants from uploaded fonts.
@ -105,8 +107,9 @@
- A `file_data` row is one of three classes, told apart by `type`: `main` is the file's current state and carries `id = file_id`; `snapshot` is a point-in-time copy whose `id` is the one its `file_change` row shares; `fragment` is a piece of the pointer-map, referenced from the file's data.
- Their lifecycles differ: `main` and `snapshot` live with the file (dated when it is deleted, undated when it is restored), while a `fragment` carries no date and is collected by `file_gc` once the file stops referencing it. Neither is a session buffer, so do not read a `fragment` as an ephemeral revision.
- File snapshots (versions) keep their metadata row in `file_change` and their payload in `file_data` (`type = 'snapshot'`). With the `db` backend `file_change.data` stays NULL: that column is only written by the `legacy-db` backend, plus the rows written before the 2025-07 file storage refactor.
- Tell a snapshot apart by `label IS NOT NULL` (with `created_by` / `deleted_at` as needed), never by `file_change.data IS NOT NULL`: the label holds with every file data backend, and the payload column stopped meaning "this is a snapshot" when the payload moved out. Queries that identify versions must use the label.
- Tell a snapshot apart by `label IS NOT NULL` (with `created_by` / `deleted_at` as needed), never by `file_change.data IS NOT NULL`: the label holds with every file data backend, and the payload column stopped meaning "this is a snapshot" when the payload moved out. Queries that identify versions must use the label. `0157-migrate-legacy-file-data` moved the labelled leftovers to `file_data` and deleted the unlabelled payload rows, so from 2.19 on the label is the only marker there is.
- The version quota counts of `app.rpc.quotes` (`sql:get-snapshots-per-file`, `sql:get-snapshots-per-team`) are served by the partial index `file_change__user_snapshots__idx`, whose predicate mirrors their where clause; changing one without the other drops the index only scan and the count walks the whole table again.
- Revisions live only in `file_change` and have no `file_data` counterpart. They are born already scheduled for deletion one hour later (`app.rpc.commands.files-update`) and the snapshot paths cannot reach one, because they resolve their row through a `label IS NOT NULL` filter.
- A deletion date is assigned once and never moved: `restore-files` (`app.rpc.commands.files`) unmarks only durable history, the `:file` method of `app.tasks.delete-object` dates only rows that have no date yet, and `objects-gc` collects only rows that carry one. Unmarking a revision leaves it invisible to `objects-gc` for ever, because nothing sets that date again.
- A deletion date is assigned once and never moved: `restore-files` (`app.rpc.commands.files`) unmarks only durable history, which is `label IS NOT NULL` (the unlabelled rows that still carry a legacy payload are removed by `0157-migrate-legacy-file-data`, so restore leaves them dated), the `:file` method of `app.tasks.delete-object` dates only rows that have no date yet, and `objects-gc` collects only rows that carry one. Unmarking a revision leaves it invisible to `objects-gc` for ever, because nothing sets that date again.
- Binfile v1/v3 import and the v3 export `get-file` step run inside one long transaction. They raise `idle_in_transaction_session_timeout` to 30 min (`app.binfile.common/transaction-idle-timeout-ms`) instead of disabling it, so a connection left idle while the ZIP is decoded or the file data is realized is bounded; `statement_timeout` stays at 300 s.
- `update-file` with `:changes-with-metadata` groups that all carry `:commit-id` is partial: fast path applies and validates everything once; on a change failure it bisects the commits for the longest prefix that applies and validates (each attempt applies a prefix from the original data and runs referential integrity + file schema; ~log2 K attempts, none for a single commit, the first attempt's error reused), keeping that prefix (a commit that repairs an earlier one can keep both) and answering 200 with `:rejected {:commit-ids :failed-commit-id :error}`, plus `:preexisting-damage true` when the data the first rejected commit touches already fails validation without it (`preexisting-damage?`: one more referential + schema check of its pages/components on the base file, the accepted prefix or the original; the frontend then says the page data is damaged). Only accepted changes reach `file_change`, persistence and the msgbus; a fully rejected request writes and broadcasts nothing (revn unchanged). Flat `:changes` or groups without ids stay all-or-nothing. Before each attempt (`apply-prefix`, which returns `{:file :tracked :refs}` or the cause) `pmap/*tracked*` and the `cpc/*state*` atom are reset to their pre-attempt values (at the end `apply-commits` sets those of the kept prefix): a pointer map already `modified?` keeps its id when changed again, so the failed version would be persisted. DB/IO/interrupt errors and `:not-found` fragments (`change-failure?`) fail the whole request, never reject a commit. A rejection whose ex-info has no `:type` reports `:internal :unexpected-error`. `app.http.errors` answers 503 (`:server-error` + code) for transient SQLSTATEs (classes 08/53, 57P01–3, 40001, 40P01), `SQLTransientException`, `InterruptedException` and only the `IOException`s that pass on their own (`transient-io-exception?`: `SocketException`, `InterruptedIOException` incl. socket timeouts, `UnknownHostException`, `HttpTimeoutException`), which the client's save retries (and GETs retry by themselves); any other `IOException` (EOF from a corrupt blob, Jackson parse errors, missing files) stays 500 `:io-exception`, as do other SQL errors. With the `:debug-save-faults` flag, a `:set-plugin-data` file change in namespace `:penpot-debug` keyed `save-fault:<kind>` raises that fault in `apply-changes` (used by `debug.failNextSave`/`commitInvalidChange`). With a schema check running (hard schema validation without skip-validate, or soft) and 2+ commits, the touched pages' objects are decoded on the original file first (`decode-objects`; referential validation alone reads few objects, so no pre-decode then): objects-map caches decoded values per instance and a derived map copies the cache, so otherwise every attempt re-decodes the page (~260 ms at 10k shapes). Validation step timings (`with-validation-timing`, upstream `penpot_rpc_update_file_validation_timing` + the `update-file` debug log) come from the first attempt only (retries run with a cfg without metrics and timings); the whole bisection is one `retry` step sample, and the rejection log carries `:attempts`/`:elapsed`.

1
.serena/memories/clojure/idioms.md
View File

@ -8,6 +8,7 @@ Behaviors confirmed against the language/stdlib — do not re-derive from assump
- The analyzer reads `:async` only from the fn name meta and the `fn` operator meta; list-level meta is ignored (would make a MetaFn). `(fn ^:async [] …)` puts the meta on argv (pre/post only, NOT async).
- Valid: `(defn ^:async f)`, `(defn- ^:async f)`, `(^:async fn [] …)` (the latter is what stock `t/async` generates itself).
- `^:async` fns never throw synchronously (rejected promises instead); `try/catch/finally` supported; continuations are microtasks, timers and RxJS schedulers are macrotasks (FIFO).
- Inside an `^:async` fn, a nested call that needs a temp binding (e.g. calling a local fn `(g (f x))`, whose arg gets a `G__` binding) compiles to `(await (async function(){…})())`: `await` unwraps a returned promise, so `(settled (upload (make-ctx)))` blocks until the upload promise settles instead of passing the promise on (a test that pushes the answer later hangs, idle). Bind each step in its own `let` binding.
- `!` suffix allowed cases (only exceptions to the no-new-`!` rule in `mem:clojure/design-rules`): mutating public `app.db` API (`insert!`, `update!`, `delete!`, `run!`, `tx-run!`, `exec!`, ...) and fns whose contract is to work over transients (stdlib `conj!`, `assoc!`, `dissoc!`, `disj!`, `persistent!` and custom helpers over them); any further case must be registered here with justification before use.
- Penpot extends `clojure.core/Inst` in `app.common.time` ( JVM-only section), so `inst-ms` also accepts `Duration` (returns `.toMillis`, NOT epoch millis), `FileTime` and `OffsetDateTime`. It only works where that namespace is loaded; on a bare `Duration`, prefer explicit `.toMillis`, which reads as what it is (an amount, not an instant).
- `cuerdas.core/index-of` returns `nil` (not `-1`) when the substring is absent, like `clojure.string/index-of`. `(pos? nil)` **throws on the JVM** but evaluates to `false` in CLJS (JS coercion), so `(if (pos? (str/index-of s "\n")) …)` works in the browser and crashes the backend. Guard with `some?` before `pos?` (this was `app.common.exceptions/first-line`).

2
.serena/memories/common/data-model-change-checklist.md
View File

@ -6,7 +6,7 @@
- Do not treat nil as a distinct persisted state from absence. Import/export and cleanup paths may filter nil attrs away.
- Avoid Clojure-special naming in exported object attrs, especially boolean names ending in `?`; exported/imported data must survive JSON/SVG/Transit and external tooling.
- Any new shape attr that participates in component sync must be listed in `app.common.types.component/sync-attrs` with the correct touched group. Attrs absent from `sync-attrs` are ignored by component synchronization.
- Shape `:svg-attrs` is the one map whose keys are camelCase React prop names (`fillRule`, not `fill-rule`), because the SVG import path runs them through `app.common.svg/attrs->props`. The `.penpot` reader (`app.common.json/read-kebab-key`) rewrites every json key of every entry to kebab-case, nested maps included, so any code that decodes a binary file must run `:svg-attrs` back through `attrs->props` or the names are lost. `:svg-defs` and the `:content` tree of `svg-raw` shapes keep their source names and round-trip on their own.
- Shape `:svg-attrs` keys are kebab-case (`fill-rule`, not `fillRule`): birth (`app.common.svg/attrs->kebab-props`, applied everywhere data lands on a shape) normalizes any outside spelling through the `svg-props` whitelist, and render boundaries (`get-svg-props`, `render-xml*`, `svg-node*`) translate back to camelCase for React. `:svg-defs` node attrs follow the same rule recursively; svg-raw `:content` attrs are kebab-ized spelling-only, never filtered — the sidebar reads `:stroke-style`, which is outside the whitelist. Non-spec attrs Penpot keeps live in `penpot-extra-attrs` (seeded with `:stroke-style`) and travel in source spelling. Excluded, each with its reason: `:tag` values (React and tag sets need them verbatim; the schema decoder restores keywords from stored strings), outer `:svg-defs` keys (id strings at birth, keywords or strings after JSON decode — accept both), all values (the kebab reader never touches values). Repair sites: birth plus migration `0035-fix-shape-svg-attrs-and-defs` (replaces the withdrawn `0021`); the binfile cleaner carries no svg repair.
## Cross-module update checklist

5
.serena/memories/common/file-change-validation-migration-subtleties.md
View File

@ -20,8 +20,10 @@
- Full referential/semantic validation currently runs only when file features contain `"components/v2"`.
- Validation starts at root plus orphan shapes, then validates component records. `validate-file!` raises `:validation :referential-integrity` with collected details.
- `update-file` validates per save with the change-scoped variants: `validate-file-affected!` (referential, whole touched pages/components) and `validate-file-schema-affected!` (file schema with `:pages-index` reduced to the touched pages; non-page data checked whole). The schema variant maps change types to pages itself (`touched-page-ids`): a new change type that writes a page must be added there, or to `file-level-change-types` if it touches no page; an unknown type makes it check the whole file. The whole-file schema check loads and decodes every page, ~75–80% of a save on multi-page files.
- `repair-file` does not mutate data directly; it reduces validation errors into redo changes using `changes-builder`. Callers must apply or persist those changes.
- `repair-file-until-valid` stops on: no errors; unchanged `:data` after apply (backend page/`PointerMap` `=` is pointer identity); stable error fingerprint `(frequencies (map #(dissoc % :shape) errors))` vs previous pass (check fingerprint before `repair-file`/`process-changes`); or `max-iterations`. Each `repair-error` builds from the original `file-data`, so N `:set`-based errors on one shape take N passes.
- `:ref-shape-not-found` repair: when the copy root's local component has a `:main-instance-id` that resolves to no shape but a `:main-instance` shape of that component sits on its `:main-instance-page`, it emits `mod-component` pointing the id back to that main (keeps the copy linked; a repair scoped to the copy's page never sees the main's `:invalid-main-instance-id`). Otherwise near-main reassign, else detach the copy subtree.
- `:missing-slot` fires only for a REAL swap: a copy sub-head whose `shape-ref` is no longer a child of the near main parent. A pure positional mismatch (ref still a sibling elsewhere) is a reorder — valid, realigned by the async component sync; do not "repair" it by assigning swap slots (a slot freezes the child out of normal sync). `fix-missing-swap-slots` (migration 0019) follows the same membership rule.
## Migrations
@ -30,4 +32,5 @@
- Migrations are an ordered set mixing legacy version-derived ids and newer named ids. Keep append order stable; `migrate` applies the set difference between available migrations and file migrations.
- `migrate-file` synthesizes legacy migration ids from old numeric versions when `:migrations` is absent, migrates legacy features, and records feature flags created through `cfeat/*new*`.
- When a file had no previous `:migrations`, `migrate-file` marks all migrations as migrated in metadata so callers persist the complete migration set, not only transformations that changed data.
- Closed schemas (`schema:stroke-attrs` and others) reject a key that was dropped from the declaration, so removing a declared attr needs a migration in the same change, for data already written with it.
- Closed schemas (`schema:stroke-attrs` and others) reject a key that was dropped from the declaration, so removing a declared attr needs a migration in the same change, for data already written with it.
- Rebase collision on a migration number: upstream keeps the id. Renumber the branch migration to the next free one, keep it last in `available-migrations`, and rename everything that spells the number — its test file and namespace (`files_migrations_00NN_test.cljc`), the test names, and comments in production code or memories that name it.

4
.serena/memories/common/testing.md
View File

@ -38,6 +38,10 @@ Useful builders:
`add-variant-with-copy` does not accept position params for children; use `gsh/absolute-move` after creation if positions matter.
## Generative sampling caution
- `sg/sample` is unseeded per run. If a test filters samples by type (e.g. keep only `:svg-raw`) then `first` can yield nil on unlucky seeds; `assoc` onto nil builds a typeless map that fails `:multi` dispatch on `:type`. Pin `:seed` in `sg/sample` opts whenever a test depends on a filtered subtype being present (verified seed 42 deals svg-raw shapes on JVM and JS).
## Driving production paths
For shape mutations, prefer production-path helpers such as `cls/generate-update-shapes` plus `thf/apply-changes`. For component swaps with keep-touched behavior, use `tho/swap-component-in-shape` with `{:keep-touched? true}`.

5
.serena/memories/critical-info.md
View File

@ -16,11 +16,12 @@ You are working on the GitHub project `penpot/penpot`, a monorepo.
- Before `git commit` → `mem:workflow/creating-commits` (subject/body format, 80-char body wrapping enforced by `scripts/check-commit`, `AI-assisted-by: model-name` trailer)
- Before `gh issue create` → `mem:workflow/creating-issues` (title derivation, body template, labels, Issue Type)
- Before `gh pr create` / `gh pr edit` → `mem:workflow/creating-prs` (title format, body structure, "Note:" line)
- Before updating `CHANGES.md` → `mem:workflow/updating-changelog` (sources of truth, entry rules, the `:boom:`/`:rocket:` label contracts, pre-flight checks, anomaly report)
- Before a repo-wide pnpm version update → `mem:workflow/updating-pnpm` (workspace layout, `scripts/sync-pnpm-version` flow, the stamp-missing-field and ignored-builds gotchas, verification steps)
- Before a Node version update → `mem:workflow/updating-node` (the six pin sites in `.nvmrc`, the devenv and image Dockerfiles and the plugins-deploy workflows, the DHI tag and digest rules, what to verify)
- Before bumping a non-JS dependency pinned in `docker/images/` (JDK, ImageMagick) → `mem:workflow/docker-images`
- **Never `git push`, force-push, or modify `git origin`** (or any other remote). The user pushes from their own shell; if a push is required, say so and wait. Never amend a commit that the user has already pushed unless explicitly asked.
- **Never edit `CHANGES.md` by hand.** The changelog is generated from GitHub milestones during the release process; update it only via the `update-changelog` skill flow or on explicit user request.
- **Never edit `CHANGES.md` by hand.** The changelog is generated from GitHub milestones during the release process; update it only via `mem:workflow/updating-changelog` (the `update-changelog` skill flow) or on explicit user request.
- You have access to the GitHub CLI `gh` or corresponding MCP tools.
- Issues are also managed on Taiga. Read issues using the `read_taiga_issue` tool.
- Before writing code, analyze the task in depth and describe your plan. If the task is complex, break it down into atomic steps. After making changes, run the applicable lint and format checks for the affected module before considering the work done (per example `mem:backend/core` or `mem:frontend/core`).
@ -60,7 +61,7 @@ The memory is structured in a way that you can get the critical information abou
- `scripts/taiga.py` — Fetch public issues, user stories, and tasks from the Penpot Taiga project without authentication. See `mem:scripts/taiga`.
- `scripts/gh.py` — GitHub operations helper: list milestone issues, fetch PR details, compare against CHANGES.md. Requires `gh` CLI. See `mem:scripts/gh`.
- `scripts/project-anomalies.py` — Main-board anomaly check for a milestone (`check` writes `tmp/<MILESTONE>-ANOMALIES.md`). Bulk resolution via `scripts/gh.py`. See `mem:scripts/project-anomalies`.
- `scripts/changelog.py` — Changelog checks for the `update-changelog` skill: `check-merged` (PR merge status), `cross-ref` (milestone PRs vs changelog section), `report` (anomaly report to CHANGES-ISSUES.md). Calls `scripts/gh.py` via subprocess. See `mem:scripts/changelog`.
- `scripts/changelog.py` — Changelog checks for `mem:workflow/updating-changelog`: `boom` (generate the `:boom:` section from `breaking change` issues), `check-merged` (PR merge status), `cross-ref` (milestone PRs vs changelog section), `report` (anomaly report to CHANGES-ISSUES.md). Calls `scripts/gh.py` via subprocess. See `mem:scripts/changelog`.
- `scripts/error-reports.mjs` — Query error reports via RPC API with token authentication. Supports list/get operations with filtering and pagination. See `mem:scripts/error-reports`.
- `scripts/clean-node-modules` — Remove stale `node_modules` from all pnpm workspaces (root, modules, member packages). Keeps the shared pnpm store at `<repo>/.pnpm-store` unless `--store`; ignores `external/` and `.opencode/`. Usage and reinstall steps: `mem:workflow/updating-pnpm`.
- `scripts/setup-opencode` — Install the opencode CLI in-container as a global pnpm package in `$PNPM_HOME`. Not part of the devenv image; see `mem:devenv/core`.

2
.serena/memories/devenv/core.md
View File

@ -5,7 +5,7 @@ Compose-based dev environment under `docker/devenv/`, driven by `manage.sh`. Par
## Compose project layout
- `penpotdev-infra`: shared `postgres`, `rustfs`, `valkey`, `mailer`, `ldap`. File: `docker-compose.infra.yml`.
- `penpotdev-wsN` (N=0,1,…): per-instance `main` + `redis` (Valkey). File: `docker-compose.main.yml`. ws0 (a.k.a. `main`) binds `$PWD`; ws1+ bind clones at `${PENPOT_WORKSPACES_DIR}/wsN/` (default `~/.penpot/penpot_workspaces/`), maintained by the developer.
- `penpotdev-wsN` (N=0,1,…): per-instance `main` + `redis` (Valkey). File: `docker-compose.main.yml`. ws0 (a.k.a. `main`) binds `$PWD`; ws1+ bind clones at `${PENPOT_WORKSPACES_DIR}/wsN/` (default `${XDG_DATA_HOME:-~/.local/share}/penpot-devenv/workspaces/`, or the former `~/.penpot/penpot_workspaces/` when it already exists), maintained by the developer.
- Optional overlay `docker-compose.opencode.yml`: added by `instance-compose` as an extra `-f` only when `PENPOT_OPENCODE_CONFIG_DIR` is set (i.e. `run-devenv --opencode-config-dir DIR` ran in this process). Bind-mounts the host dir at `/home/penpot/.config/opencode` (`:z`). Flag-only, per-call; not read from ambient env. Parser `parse-opencode-config-dir` absolutizes (`~`, realpath) because compose resolves relative bind sources against the compose file's dir. Only instances brought up with the flag get the mount.
- All projects join external network `penpot_shared`. Created idempotently by `ensure-devenv-network`, never removed by lifecycle commands.

106
.serena/memories/frontend/auth-events.md Normal file
View File

@ -0,0 +1,106 @@
# Auth Events and the Session Origin
`signin` is the frontend `identify` event emitted once per session from
`logged-in` (`app.main.data.auth`). It carries the profile plus a closed
vocabulary that says why the session started.
## Vocabulary
- `source`: `login` | `register`. Always present.
- `auth-backend`: `penpot` | `ldap` | `google` | `github` | `gitlab` | `oidc`.
Same key the profile already uses. Always present.
- `activation`: `immediate` | `email-verification`. Only on signups.
- `from-invitation`: boolean. Always present in `signin`
(`(some? (:invitation-token profile))`); `auth-origin` omits it when `nil`.
`auth-origin` (public, validated against `schema:auth-origin`) drops the nil
fields and checks with `(assert (sm/check ...))`: it throws in development and
tests, and `:elide-asserts true` removes it from the release build, so a bad
value can never break a real login. Emitters never call it directly.
## Where each value comes from
- `source` and `activation` travel as profile metadata (`::ev/source`,
`::ev/activation`), written by `origin-meta`. `login-from-token` derives both
from the token issuer with `iss->origin`: `:auth` → `login`; `:verify-email` →
`register` + `email-verification`. An unknown `:iss` produces no metadata, so
the assertion fires in development and the event goes without `:source` in
production.
- `auth-backend` and `from-invitation` are read from the profile.
- `profile-origin` composes the four and `logged-in` merges them into the event;
the emitters only declare the fact.
| Flow | `source` | `activation` | `from-invitation` |
|---|---|---|---|
| Password login | login | | false |
| Password login from an invitation link | login | | true |
| LDAP login | login | | false |
| Social login, existing active profile | login | | false |
| Signup, email verification disabled | register | immediate | false |
| Signup, email verification enabled | register | email-verification | false |
| Social signup, active profile | register | immediate | false |
| Social signup, inactive profile | register | email-verification | false |
| Signup from an invitation link, email verification disabled | register | immediate | true |
| Signup from an invitation link, email verification enabled | register | email-verification | true |
| Social login from an invitation link | login | | true |
| Social signup from an invitation link, email verification enabled | register | email-verification | true |
| LDAP login from an invitation link | login | | true |
LDAP is the explicit exception: the directory provisions the account on the
first login, but there is no user signup, so the flow is always `login` and
never carries `activation`. Do not "fix" it by inferring a signup.
## Invitations go through `logged-in`
- `login-from-register` accepts an optional `:invitation-token`;
`with-register-origin` decorates the profile and `logged-in` does the rest.
The four places that used to navigate by hand to `:auth-verify-token`
(`ui/auth/register.cljs` twice, `ui/static.cljs`, `ui/auth/login_modal.cljs`)
emit `login-from-register` with the token. The modal used to live in
`ui/viewer/login.cljs`; it was extracted to the shared
`ui/auth/login_modal.cljs` (`login-register-modal*`, also used by
`ui/mcp_authorize.cljs`).
- The invitation navigates through `get-invitation-nav`, which does not wait for
`::dtm/teams-fetched`: `fetch-teams` has no error handling, so depending on it
would leave the invitation unaccepted when `get-teams` fails. For invited
profiles `get-redirect-events` emits `rx/empty`, so the teams path cannot move
the user away from the token verification screen.
- The OIDC callback with an invitation mints an `:auth` token that carries the
invitation JWE in its claims (`backend/src/app/auth/oidc.clj`), so the
frontend follows the regular path (`login-from-token` → `logged-in`). That
flow visits the token verification screen twice.
- `ui/auth/login_modal.cljs` (the register modal: `:is-active` branch calls
`on-close`, or `on-register` when the caller passes one, as in the MCP
authorize flow) and `ui/static.cljs` (`rt/reload true`) `:is-active`
branches still bypass `logged-in`; they are a separate decision, not an
oversight.
## Backend side
- The `register-profile` success branches
(`backend/src/app/rpc/commands/auth.clj`) put `activation` in the event props
and emit **no** `action`: the name already says it is a signup, and a constant
action would only add noise. The retry branches
(`register-profile-retry`) keep `action` in `context`
(`ignore-because-blocked`, `ignore`, `resend-email-verification`,
`ignore-because-complaints`), where it distinguishes outcomes.
- The LDAP invitation branch of `login-with-ldap` returns the full profile plus
`:invitation-token`; a partial map would break the `auth-origin` assertion.
- `register-profile` does not accept the invitation: it mints the session and
returns the token. The acceptance is `accept-team-invitation`, emitted from
`verify-token`.
- `filter-telemetry-props` keeps `source` and `activation` in the anonymised
row; `:action` is not on the allowlist.
## Analytics
- `register-profile` is a track event, so the props adapter forwards
`activation` with no extra work.
- `signin` is an identify event built from a closed list of profile keys. It
only reaches analytics if the identify branch of Nexus forwards `source`,
`activation` and `from_invitation` (`external/nexus`, another repository).
## `::ev/source` is no longer dead metadata
It is read by `profile-origin`. Do not remove it as unused, and keep the
vocabulary in sync with `schema:auth-origin` and the table above.

17
.serena/memories/frontend/cljs-repl.md
View File

@ -49,16 +49,23 @@ Get `team-id` from `(:current-team-id @app.main.store/state)`. Get file ids from
## Runtime patching with `set!`
Some frontend vars are deliberately mutable escape hatches for runtime instrumentation or circular-dependency patching. From `cljs_repl`, use `set!` for temporary debugging of CLJS vars such as `app.main.store/on-event`, `app.main.errors/reload-file`, `app.main.errors/is-plugin-error?`, `app.main.errors/last-report`, or `app.main.errors/last-exception`. These patches affect only the live browser runtime and disappear on reload or recompilation.
Some frontend vars are deliberately mutable escape hatches for runtime instrumentation or circular-dependency patching. From `cljs_repl`, use `set!` for temporary debugging of CLJS vars such as `app.main.errors/reload-file`, `app.main.errors/is-plugin-error?`, `app.main.errors/last-report`, or `app.main.errors/last-exception`. These patches affect only the live browser runtime and disappear on reload or recompilation.
`set!` on `app.main.store/on-event` does nothing at runtime: the store captured the function when it was created. To watch events, subscribe to the store stream and dispose the subscription after:
```clojure
;; Log non-noisy Potok events temporarily.
(set! app.main.store/on-event
(fn [event]
(when (potok.v2.core/event? event)
(.log js/console (potok.v2.core/repr-event event)))))
(def sub
(beicon.v2.core/subs!
(fn [event]
(when (potok.v2.core/event? event)
(.log js/console (potok.v2.core/repr-event event))))
app.main.store/stream))
;; (beicon.v2.core/dispose! sub)
```
`cljs_repl` starts every call in `cljs.user`; begin a call with `(in-ns 'my-ns)` to reuse helpers defined in another namespace.
Restore mutable hooks after debugging, or reload the frontend. Use JVM `alter-var-root` only for JVM Clojure; it is not the normal way to patch live CLJS browser vars.
## Browser-console debug namespace

1
.serena/memories/frontend/core.md
View File

@ -46,6 +46,7 @@ Workspace behavior:
App shell and product flows:
- Routing, root app shell, websocket, and global errors: `mem:frontend/routing-app-shell-subtleties`.
- Authentication events, the session origin vocabulary, and the invitation flow: `mem:frontend/auth-events`.
- Dashboard and viewer flows: `mem:frontend/dashboard-viewer-subtleties`.
- Plugin JS API runtime inside the frontend app: `mem:frontend/plugin-api-to-cljs-binding`.

1
.serena/memories/frontend/dashboard-viewer-subtleties.md
View File

@ -7,6 +7,7 @@
- Dashboard file/project mutations are often optimistic local updates with fire-and-forget RPC watchers. Bulk permanent delete/restore paths use SSE progress and progress notifications.
- The import wizard runs the package as a job of the backend: it reads the manifest of the file in the main thread (`app.main.data.imports`, no web worker) and follows the job, so its header paints the milestone of the job (the stage and its counter) instead of the messages the old SSE stream sent; every file of a container reports the same milestone, so `jp/current-milestone-text` writes it once and the rows never repeat it. Before the job exists, the header paints a client-side `:upload` milestone built from the chunked upload (`uploads/upload-blob-chunked` `:on-progress`, chunks done/total), which never comes from the backend; the chunks go one at a time and it writes them as a percentage of the file, not as a counter. Once created and still waiting for a worker, no row says anything: the header says the import waits for a slot while no job of it has started. The bar of the header measures the whole import, never a file: indeterminate while checking, waiting for a slot or rebuilding, because the client cannot know a global index and total of operations. The upload is the exception — the chunked upload knows its chunks, so the bar is determinate while the containers are sent, weighted by their size (`imp/upload-pct` over `:container-size`, the only percentage the client really knows). The export modal follows the same one-job-per-file model, but every file is queued up front (even the ones whose job is created only after the previous one is over) and its queue ends only with a milestone: `:start`/`:retry` do not end it there. Neither modal paints a generic "…ing file…" line on the rows any more: the rows of the import wizard carry only what is not progress — the name, the size, `Ready` once their container finished uploading, and the error text —, and the button that drops a file only shows while the wizard still offers its import step (the confirmation screen). Above the rows, the import wizard paints a stepper (Check/Upload/Import) plus one phase line: `imp/import-step` derives the step from the dialog status and the entries (upload holds while any file is pending, either `:import-ready` or carrying its own `:upload` milestone), so Check covers analyze, Upload covers the chunked client-side upload, and Import covers queue plus job. A file of a version 3 package shows what it occupies inside its container, computed while analyzing (`imp/file-sizes`): its own `files/<id>…` entries plus the storage objects its media and thumbnails reference, whose metadata is the only link to `objects/<id>…`. The container's own size stays in `:container-size` (what the dropped file weighs) and is what the upload bar weighs, never what a row shows. A file the manifest lists but the container holds no entry of has no size at all, and its row then shows no subtitle: what the package weighs never stands in for it. The subtitle is the size alone, with no extension. While analyzing with no failures nothing renders below the phase header — no rows, no spinner (`imp/awaiting-analysis?`); the header text is the only "Checking…" message. Rows appear once the analysis resolves (one per file inside the package), and a failed analysis takes the error-summary path instead. The ready step and its Continue wait for the end of the analysis (`imp/ready-to-import?`, fed with the completion of `imp/analyze`): a ready file is not enough, because the files still to arrive would never be imported. While rebuilding, the phase title counts the files and `imp/imported-count` counts them done with the job's own `:files` counter, so the number moves while a container rebuilds its files one by one. The `:error` of an import entry is user-facing text (the backend hint, or a translated label), never an i18n key: the wizard renders it as-is, it does not translate it again. The link-later step takes the file id from the `:id` of its resolution entry, never from the key of the resolution map: the map comes from the job result, so what a key holds depends on the result schema of the job-def that wrote it, while the wizard indexes its selection by the id of the entry. The legacy SSE import stays in the backend for other clients.
- File creation/duplication strips file `:data` before putting file summaries into dashboard state.
- The pending invitations are a team-admin only list (backend `:is-admin` already includes the owner). The dashboard hides the invitations entry from the header tabs and the team-options dropdown for non-admins, `invitations-section-content` keys the section on `:is-admin`, and `fetch-invitations` answers `insufficient-permissions` by emitting `invitations-fetched` with `[]`, so a demoted admin does not keep the cached emails and the header does not paint the invite button from a stale list. The organization `send-invitations` policy only gates inviting and resending, not the list.
## Viewer

4
.serena/memories/frontend/plugin-api-to-cljs-binding.md
View File

@ -23,6 +23,9 @@
- Public Plugin API objects are lightweight handles, not durable snapshots. Most getters locate fresh state from `app.main.store/state` using hidden `$id`, `$file`, `$page`, etc.
- `not-valid` logs by default but throws when the plugin flag `throwValidationErrors` is enabled. The MCP execute-code handler deliberately enables that flag while running code.
- A write to a removed shape always throws `Shape not found`, whatever that flag: setters through the shape proxy's `:on-set u/check-shape-writable` (runs outside the error handling), and methods that must not vanish silently (`resize`, `rotate`) call `u/check-shape-located` first. Methods run inside `:on-error (u/handle-error ...)`, which logs instead of throwing when the flag is off, so it rethrows errors whose `ex-data :type` is `::u/shape-not-found` or `::r/read-only`.
- A write while the workspace cannot take it (`r/read-only-reason`: save-failure lock `[:workspace-global :persistence-lock]`, `[:permissions :can-edit]` false, version preview `:preview-id`, lost render context `[:render-state :lost]`) always throws, whatever that flag. The `[:workspace-global :read-only?]` flag alone does not block: the Inspect tab sets it and plugins may still write. `r/check-permission` with `content:write`/`library:write` calls `r/check-writable` first, which covers most writers; writers without a permission check (most top-level context methods in `api.cljs`: `create*`, `group`, `align*`, …; z-order methods; text range `applyTypography`; ruler guide `remove`) call `r/check-writable` themselves, and shape setters get it through `u/check-shape-writable`. A new writer without a write-permission check must call it too. An async writer calls it again after each server await, before the next server write or commit (`rx/tap` on the response stream; `dwm/create-svg-shape-with-images` takes it as `:check-writable`), so a lock that lands mid-way rejects the promise with the read-only error. `r/has-permission?` is the check without it, used by the undo blocks so a block opened before a lock still closes. Comments (`comment:write`) stay writable: they do not go through file saves.
- `penpotMgmt` (`app.plugins.management`) checks permissions with `r/has-permission?` (`granted?`), never `r/check-permission`: its calls act on files through the backend, often from the dashboard with no file open, so the workspace read-only check (`check-writable`) must not block them.
- `naturalChildOrdering` and `throwValidationErrors` are stored per plugin under `[:plugins :flags plugin-id ...]`; changing default behavior affects automation and MCP diagnostics.
- Plugin data is stored under keyword namespaces: private data uses `(keyword "plugin" plugin-id)`, shared data uses `(keyword "shared" namespace)`.
@ -31,4 +34,5 @@
- Plugin listeners are watches on the global store and callbacks are debounced about 10ms. Callback exceptions are caught and logged so plugin code does not crash the app.
- `selectionchange` callbacks receive arrays of shape id strings, while `filechange`, `pagechange`, and `shapechange` return proxies.
- `contentsave` fires only when persistence status transitions to `:saved`; it calls the callback with no value.
- `contentrollback` is not a state watch: it subscribes to `st/stream` for `::dps/rolled-back` (kept in `subscriptions`, disposed by `remove-listener`), not debounced, delivered async so the reverted state is in place. Payload `{fileIds pageIds shapeIds}` from `pcm/rollback-targets` (page shapes only, from the reverted commits' redo changes). When the rollback falls back to reloading the file, `::dps/rolled-back` waits for the reloaded data, so the callback never reads the rejected data. The runtime's `validEvents` must list every event name, or `penpot.on` rejects it; a stale runtime build rejects new events. The MCP plugin forwards these rollbacks with the next task response and the server puts a warning first in that tool response (`RollbackNotices`). Unsent ones survive a tab or plugin reload in `penpot.localStorage` (key `rollbacks/<file-id>`, `LocalStorageRollbackStore`; the MCP manifest in `app.main.data.workspace.mcp` and `mcp/packages/plugin/public/manifest.json` grant `allow:localstorage`, which the runtime checks on access); the entry goes once a response carried them.
- Plugin history `undoBlockBegin` creates a workspace undo transaction with a JS `Symbol`; `undoBlockFinish` commits that symbol. Missing finish eventually relies on the workspace transaction timeout.

1
.serena/memories/frontend/testing.md
View File

@ -39,6 +39,7 @@ To assert over emitted event sequences, observe termination: subscribe through `
- Auto-answering mocks lose deadline expressiveness (can't time answers), need teardown timer-cancellation, and post-teardown deliveries hit real implementations — explicit pushes + async delivery + `wait-for` won on every axis.
- Teardown belongs to the terminal continuation, never to `finally`-around-triggers (it would dispose in-flight flows).
- A body that awaits must be `(^:async fn …)` even if the rest is sync; sync sequences are atomic vs the event loop.
- The `:esm` test build cannot load the CommonJS `sax` module behind `tubax`, so SVG parsing fails in tests (`dwm/valid-svg-string?` answers false, `dwm/svg->clj` errors): mock both with the parsed data.
From `frontend/`:
- Full unit test run (always builds, suppressed output): `pnpm run test:quiet`.

32
.serena/memories/frontend/workspace-state-persistence-subtleties.md
View File

File diff suppressed because one or more lines are too long

1
.serena/memories/frontend/workspace-transform-subtleties.md
View File

@ -6,6 +6,7 @@
- `apply-modifiers*` is the lower-level commit path once object/text modifiers are ready. It updates frame guides, frame comment threads, and then emits `update-shapes` with `:reg-objects? true`.
- Transform commits restrict diff attrs to `transform-attrs` to avoid scanning unrelated shape attrs.
- Text transforms may carry derived `:position-data`; `assoc-position-data` attaches it while preserving the original text shape context.
- A drag commits when its `mse/drag-stopper` fires, `:interrupt` (Escape) included. The save-failure lock emits `::plk/cancel-gestures` first: drags built on `plk/gesture-stopper` (start-move, start-resize, start-rotate, start-move-line-point, box/curve/line drawing) see a `cancelled?` flag set before the stop and emit `dwt/cancel-transform` (`clear-local-transform` + `finish-transform`) or `cancel-drawing` instead. A new drag gesture must use it (or, in a component, drop the drag when read-only turns on): `commit-changes` refuses the commit anyway, but the preview (WASM modifiers/props) would stay.
## Component-copy touched suppression

13
.serena/memories/mcp/core.md
View File

@ -57,7 +57,7 @@ mcp/
│ ├── src/
│ │ ├── main.ts # handles communication
│ │ └── plugin.ts # plugin implementation
│ └── package.json # Includes @penpot-mcp/common dependency
│ └── package.json # Plugin deps: published plugin-styles, linked plugin-types
└── prepare-api-docs # Python project for the generation of API docs
```
@ -107,6 +107,13 @@ From the `mcp/` directory, run
(node reports `Promise resolution is still pending but the event loop has
already resolved` for the file's root test). Inside tests the logger writes to
stdout directly. `NODE_TEST_CONTEXT` is the env var `node --test` sets.
* `packages/plugin` takes `@penpot/plugin-styles` from the registry but links
`@penpot/plugin-types` to `plugins/libs/plugin-types` (`link:` in the manifest
and in `mcp/pnpm-lock.yaml`, with no registry entry, because a linked package
has no resolution): the published types do not declare the `penpotMgmt`
global the plugin needs. On a plugins libs version bump, only the styles
version moves. After editing the manifest, `pnpm install --lockfile-only` in
`mcp/` must leave the lock untouched.
* Cross-cutting testing principles and anti-patterns: `mem:testing`.
## OAuth for remote MCP clients
@ -136,3 +143,7 @@ For parallel devenvs, prefer same-origin MCP routing: each Penpot instance shoul
- The MCP server emits `1008` for a missing `userToken` in multi-user mode, for missing/invalid connection metadata, and to close a connection displaced by a newer one with the same user and session ID.
- Session IDs are deterministic per tab and file, so a reconnect reuses its ID while the old socket may be half-open: the newest connection wins (`PluginBridge.displaceConnection`). Across instances, each new connection publishes a session claim on a per-user Redis channel; an instance displaces its local connection only on a foreign claim that arrives after its own claim (channel order, no clocks). Discovery deduplicates by session ID.
- A tab rejected with `1008` never reaches `connected`, so the frontend's 60s reconnect watcher (`start-reconnect-watcher` in `app.main.data.workspace.mcp`, started only on `connected`) does not engage; recovery is manual via the MCP menu.
## Rollback warnings
- The plugin's `RollbackLog` (`packages/plugin/src/RollbackLog.ts`) holds the `contentrollback` events not yet sent with a task response; `Task.sendResponse` attaches them and acknowledges them once sent, and the server's `unclaimed-rollbacks` message (a response that came after its task timed out) puts them back. With a `RollbackStore` (`LocalStorageRollbackStore` over `penpot.localStorage`, one entry per file) every change is written through, so a reload of the tab or plugin keeps them; storage failures fall back to memory. Plugin tests run with `node --experimental-strip-types`, which rejects TypeScript parameter properties: declare fields explicitly.

3
.serena/memories/render-wasm/core.md
View File

@ -36,6 +36,7 @@ From `render-wasm/`:
- Rust tests: `./test` or `cargo test <name>`.
- Cross-cutting testing principles and anti-patterns: `mem:testing`.
- Lint: `./lint`.
- Format check: `cargo fmt --check`.
- Format: `./fmt`; check only: `./fmt --check`.
- `./build`, `./test`, `./lint` and `./fmt` all run `setup`, which selects the Rust version pinned in `_build_env` (skip with `PENPOT_WASM_PREPARED=1`).
Do not change exported WASM function signatures without updating the corresponding frontend bridge and verifying the frontend renderer path.

11
.serena/memories/scripts/changelog.md
View File

@ -1,12 +1,14 @@
# Changelog helper
`scripts/changelog.py` holds the checks used by the `update-changelog` skill (extracted from the skill file so the skill stays workflow-only). Calls `scripts/gh.py` via subprocess; never call the GitHub API directly from this script.
`scripts/changelog.py` holds the checks used by the changelog flow. The rules and the step-by-step live in `mem:workflow/updating-changelog`; this memory documents the subcommands. Calls `scripts/gh.py` via subprocess; never call the GitHub API directly from this script.
## Subcommands
- `check-merged <PR...>` (`--file`, `--stdin` accepted) — warn on any non-merged PR, exit 1 when found; run before writing changelog entries.
- `cross-ref <MILESTONE> [--changes CHANGES.md]` — fetch milestone PRs with `--state all` once, report merged PRs missing from the version section plus CLOSED (unmerged) warnings; GHSA entries never appear here by design.
- `report <MILESTONE> [--changes CHANGES.md --output CHANGES-ISSUES.md]` — overwrite the anomaly report (milestone mismatches type A/B plus `:rocket:` gaps C/D); rule violations never belong in the report, they are fixed in CHANGES.md first.
- `boom <MILESTONE> [--changes CHANGES.md --write]` — generate `### :boom: Breaking changes & Deprecations` from the closed, non-excluded milestone issues labelled `breaking change`, sorted by issue number. Each issue becomes a `#### <issue title>` block: content = body of the issue's last `## Breaking change note` comment (whole markdown, headings demoted to `#####` outside code fences; fallback = heading plus refs with a stderr warning), refs = issue link plus merged closing PRs on their own line, with `(by @author)` for `community contribution`. `--write` replaces only that subsection of the version (inserts it after the version header when missing, keeps the heading when there are no entries; idempotent). Print-only by default. Uses `gh.py comments`.
- `security <MILESTONE> [--changes CHANGES.md --write]` — generate `### :lock: Security fixes` from the advisories whose `patched_versions` include the version (via `gh.py advisories --patched-version`), sorted by GHSA id. Entry: `- Fix <description> (<advisory URL>) (PR: [#N](...))`; description = `Fix` + advisory summary, first word decapitalized unless it starts with an acronym; PRs = merged milestone PRs labelled `pr pt` whose body contains the GHSA id, comma-separated, and no PR is legal. Warns on stderr when a `pr pt` PR references an advisory with no matching `patched_versions`. `--write` replaces only that subsection, placed after `:rocket:` (or `:boom:`, or the version header when neither exists); empty entries remove the subsection; idempotent. Print-only by default.
- `cross-ref <MILESTONE> [--changes CHANGES.md]` — fetch milestone PRs with `--state all` once, report merged PRs missing from the version section plus CLOSED (unmerged) warnings. PR links inside `:lock:` entries count as referenced; advisory URLs never appear here by design.
- `report <MILESTONE> [--changes CHANGES.md --output CHANGES-ISSUES.md]` — overwrite the anomaly report: A/B milestone mismatches, E `:boom:` entry without the label, F `:rocket:` sync, G `:boom:` sync, plus the C/D `:rocket:` gaps (warnings). `:boom:`/`:rocket:` rule violations are fixed by regenerating or relabelling, not by editing entries in place.
## Conventions
@ -15,4 +17,5 @@
- Version-section lookup accepts an optional `(suffix)` after the version header (e.g. `(Unreleased)`); a missing section is an error for `cross-ref` and an empty section for `report`.
- PR/issue reference parsing covers standard `[#N](.../pull/N)` / `[#N](.../issues/N)` links plus legacy `PR:[N]` / `[Github #N]` forms.
- Shared parsing helpers (`extract_version_section`, `extract_subsection`, `collect_changelog_prs/issues`) live at module top so future subcommands reuse them instead of duplicating regexes.
- Report anomaly types: A/B milestone mismatches, E `:boom:` entry whose issue lacks the `breaking change` label (counted, entries preserved — human labels the issue or moves the entry); C/D are `:rocket:` gaps (warnings, never counted).
- Report anomaly types: A/B milestone mismatches, E `:boom:` entry whose issue lacks the `breaking change` label (counted, entries preserved — human labels the issue or moves the entry), G `breaking change` issue missing from `:boom:` (counted, same contract as F); C/D are `:rocket:` gaps (warnings, never counted).
- `issue_excluded` (closed, non-Task, non-Rejected, no `release blocker`/`no changelog` label) is module-level and shared by `report` and `boom` so the exclusions cannot drift.

2
.serena/memories/scripts/ci.md
View File

@ -106,7 +106,7 @@ produce deployable artifacts.
## Notes
- `mcp` has no lint task and `media-processor` has no lint task; both show
as skipped. `render-wasm` uses `./lint`, `./test`, and `cargo fmt`.
as skipped. `render-wasm` uses `./lint`, `./test`, and `./fmt`.
- `scripts/test_ci.py` (unittest, `python3 scripts/test_ci.py`) covers the
flag wiring, one build command per module and the help text through
`--dry-run`; it executes no real build.

18
.serena/memories/scripts/gh.md
View File

@ -11,6 +11,7 @@ repository via GraphQL and REST APIs through the authenticated `gh` CLI.
- Comparing milestone issues against CHANGES.md to find missing entries.
- Explicitly linking a GitHub issue to a pull request.
- Listing or inspecting GitHub Security Advisories (GHSA).
- Fetching issue comments (the `## Breaking change note` markdown for the changelog).
## Prerequisites
@ -82,6 +83,16 @@ Fetch issue details by number in batches of 50 (same input styles as `prs`: numb
**Output**: JSON array to stdout; progress to stderr.
### `comments`
Fetch every comment of one or more issues via REST (paginated, 100 per page), oldest first. Same input styles as `issue`: numbers, `--file`, `--stdin`.
```bash
python3 scripts/gh.py comments 11235 11236
```
**Output fields per comment**: `issue`, `id`, `author`, `created_at`, `body`. JSON array to stdout; progress to stderr. `changelog.py boom` reads the `## Breaking change note` comment from this output.
### `link-issue`
Explicitly assign a GitHub issue to a pull request:
@ -91,7 +102,7 @@ python3 scripts/gh.py link-issue <ISSUE_NUMBER> <PR_NUMBER>
# Short alias: python3 scripts/gh.py link <ISSUE_NUMBER> <PR_NUMBER>
```
The command resolves both node IDs and calls `addCloseIssueReferences`. It trusts the successful mutation instead of re-querying, because GitHub does not reliably report mutation-created links through `closedByPullRequestsReferences(userLinkedOnly: true)`. It is safe to rerun, works for merged PRs, and does not close an issue retroactively. JSON goes to stdout; progress and errors go to stderr; a missing issue/PR or a failed mutation exits non-zero.
The command resolves both node IDs and calls `addCloseIssueReferences`. It trusts the successful mutation instead of re-querying, because GitHub does not reliably report mutation-created links through `closedByPullRequestsReferences(userLinkedOnly: true)`. It is safe to rerun and works for merged PRs. It does not guarantee the issue stays open: adding `Closes #N` to the body of an already-merged PR closed the issue immediately when tried (#12896 / PR #12067, 2026-11-09). JSON goes to stdout; progress and errors go to stderr; a missing issue/PR or a failed mutation exits non-zero.
### `advisories`
@ -107,6 +118,9 @@ python3 scripts/gh.py advisories --severity critical
# Filter by state
python3 scripts/gh.py advisories --state triage
# Filter by patched version declared in the advisory
python3 scripts/gh.py advisories --patched-version 2.19.1
# Get full detail for a single advisory
python3 scripts/gh.py advisories GHSA-xvj6-fh9w-gjw7
```
@ -115,6 +129,8 @@ python3 scripts/gh.py advisories GHSA-xvj6-fh9w-gjw7
**Detail output** (single advisory) adds: description, vulnerabilities (package, version ranges), credits, timestamps.
`--patched-version` filters the raw advisories before formatting: the match is an exact token of `vulnerabilities[*].patched_versions` (split on `,`/`;`), so `2.19.1` never matches `2.19.10`. Every state is included (`triage`, `draft`, `published`, `closed`, `withdrawn`). The raw `vulnerabilities` stay internal to the filter: the list output keeps the summary schema.
**Output**: JSON to stdout; progress to stderr.
## Key principles

34
.serena/memories/workflow/creating-issues.md
View File

@ -35,11 +35,36 @@ Command what should be built. Format: `[Imperative verb] [what] in/on [where]`.
| Field | Rule |
|-------|------|
| **Labels** | `community contribution` (PRs from non-core) · skip workflow labels (`backport candidate`, `team-qa`) · do **not** add `bug` or `enhancement` labels (use Issue Type instead) |
| **Labels** | `community contribution` (PRs from non-core) · `breaking change` when the change breaks compatibility (see below) · skip workflow labels (`backport candidate`, `team-qa`) · do **not** add `bug` or `enhancement` labels (use Issue Type instead) |
| **Milestone** | Use the current or next planned milestone. Fetch available milestones: `gh api repos/penpot/penpot/milestones --jq '.[].title'`. If unsure, omit. |
| **Project** | Always `Main` (project number 8). Use `--project "Main"` flag. |
| **Issue Type** | See Issue Type section below. Cannot be set via `gh issue create` — use GraphQL after creation. |
## Breaking changes
When the issue introduces a breaking change or a deprecation, it is the unit
`mem:workflow/updating-changelog` turns into a block under
`### :boom: Breaking changes & Deprecations`. The section is generated from
the issue (`scripts/changelog.py boom`), so the issue must carry both:
1. The `breaking change` label.
2. A `## Breaking change note` **comment** (not a body section). Everything
after that marker line is copied to the changelog: paragraphs, bullet
lists, code blocks and headings are kept (headings are demoted to level 5
in `CHANGES.md`). Write it for self-hosters: what changes, what they must
do, and what breaks if they do nothing. Keep it short: explain the
implications and link the runbook or docs for the detail, never paste the
runbook itself. Links must point to a long-lived branch (`staging` or
`main`), never to a feature branch.
Post the note as a new comment. Do not add a breaking-change section to the
issue body and do not hand-write the changelog block. When the issue comes
from a PR, reuse the text of the PR's breaking-change section.
Do not hard-wrap the note: one line per paragraph or bullet.
The full changelog rules (sections, exclusions, `:rocket:`, pre-flight and
anomaly report) live in `mem:workflow/updating-changelog`.
## Issue Body Template
Write the body to a temp file to avoid shell quoting issues:
@ -237,6 +262,9 @@ Identify:
description). Omit internal details. Use the templates above.
- **Issue Type:** use the mapping table above (also handles `:bug:` /
`:sparkles:` / `:tada:` title prefixes).
- **Breaking changes:** when the PR body has a breaking-change section, post
its user-facing text as a `## Breaking change note` comment on the issue and
add the `breaking change` label (see **Breaking changes** above).
### Create the issue
@ -287,7 +315,7 @@ Add `Closes #<ISSUE_NUMBER>` to the PR body for readable context, then run the e
python3 scripts/gh.py link-issue <ISSUE_NUMBER> <PR_NUMBER>
```
The command creates the GitHub Development link by calling `addCloseIssueReferences` and trusts the successful mutation (GitHub does not reliably report mutation-created links back through the API). It is safe to rerun and does not close an issue retroactively when the PR is already merged. Do not rely on the body keyword as the assignment operation.
The command creates the GitHub Development link by calling `addCloseIssueReferences` and trusts the successful mutation (GitHub does not reliably report mutation-created links back through the API). It is safe to rerun. Do not rely on it to keep the issue open: adding `Closes #N` to the body of an already-merged PR closed the issue immediately when tried (#12896 / PR #12067, 2026-11-09; the timeline shows a cross-reference plus a ClosedEvent with no closer). For changelog units that outcome is fine — a merged fix means a closed issue — but be aware when linking old merged PRs. Do not rely on the body keyword as the assignment operation.
### Clean up
@ -303,6 +331,8 @@ rm -f /tmp/issue-body.md /tmp/pr-body.md
label or the author is not a core team member, add the label to the issue.
- **Don't put implementation details in the issue body** — the issue is for
users, QA, and changelog readers.
- **Drop the PR's `no issue required` label** once the issue exists: it only
marks PRs that need no changelog unit, and it is stale from then on.
## Creating Issues from Draft Body

4
.serena/memories/workflow/creating-prs.md
View File

@ -66,7 +66,7 @@ The "Note:" line is required at the top. Adjust if this is a manual (non-AI) PR.
- **Write for humans.** The diff shows what changed. The description explains why.
- **Be concise.** Focus on reasoning: What was the problem? Why did it happen? How did you solve it?
- **Prefer bullets over paragraphs.** Short bullet items, grouped by area with bold lead-ins where helpful, are far easier to digest than prose; keep any remaining paragraph to a few sentences.
- **No manual line wraps.** Markdown renders adapting to the viewport; hard-wrapped lines degrade rendering. One line per paragraph or bullet, however long.
- **No manual line wraps.** Markdown renders adapting to the viewport; hard-wrapped lines degrade rendering. One line per paragraph or bullet, however long. This applies to PR descriptions and to PR/issue comments and review replies alike.
- **Skip the obvious.** Don't explain what `git diff` already shows.
### What NOT to Include
@ -80,7 +80,7 @@ The "Note:" line is required at the top. Adjust if this is a manual (non-AI) PR.
## Explicit Issue Assignment
- For each GitHub issue that a PR resolves, run `python3 scripts/gh.py link-issue <ISSUE_NUMBER> <PR_NUMBER>` after creating or editing the PR. Do not rely on `Closes #NNNN` in the body; it is only human-readable context.
- The command calls `addCloseIssueReferences` and trusts the successful mutation: GitHub does not reliably report mutation-created links back through `closedByPullRequestsReferences(userLinkedOnly: true)`, so the command exits non-zero only when a link target is missing or the mutation fails. It is safe to rerun and also works for an already merged PR; it does not close an issue retroactively.
- The command calls `addCloseIssueReferences` and trusts the successful mutation: GitHub does not reliably report mutation-created links back through `closedByPullRequestsReferences(userLinkedOnly: true)`, so the command exits non-zero only when a link target is missing or the mutation fails. It is safe to rerun and also works for an already merged PR. Do not rely on it to keep the issue open: adding `Closes #N` to the body of an already-merged PR closed the issue immediately when tried (#12896 / PR #12067, 2026-11-09).
- Skip this process for `Relates to #NNNN` and Taiga references, which do not represent a closing relationship.
## Before Opening

208
.serena/memories/workflow/updating-changelog.md Normal file
View File

@ -0,0 +1,208 @@
# Updating the changelog
> Note about this memory: unlike other Serena memories (dense notes written for LLM agents, see `mem:memory-maintenance`), this document is also written for human release managers who need full context. That is why it uses plain language and examples. It is the canonical home of the changelog flow; the `update-changelog` skill is only an entry point that routes here.
`CHANGES.md` (repository root) is the release notes file. Every entry comes from GitHub: a milestone issue, its merged PR, or a security advisory whose `patched_versions` include the version. It is never written from scratch by hand; the only manual parts are the wording of a description when the title is not enough and the advisory summaries (polished on GitHub, the source of the generated security entries).
The commands live in `scripts/gh.py` (GitHub queries) and `scripts/changelog.py` (changelog checks). Their flags and internals are documented in `mem:scripts/gh` and `mem:scripts/changelog`.
## 1. When to run it
- Before a release, to fill the section of the version being released.
- When new issues are added to an existing milestone.
- When asked to audit or refresh `CHANGES.md`.
Editing `CHANGES.md` happens only on explicit request or as part of the release process; see `mem:critical-info`.
## 2. Sources of truth
| Question | Source |
|---|---|
| Which issues belong to the version? | GitHub milestone |
| Which security fixes? | Advisories whose `patched_versions` include the version |
| Which section? | Issue Type |
| Is it a breaking change? | `breaking change` label |
| Is it a highlight? | `release highlight` label |
- **Issue Type decides the section**: `Bug` → `:bug:`, `Feature`/`Enhancement` → `:sparkles:`, `Task` is excluded (internal chores). Labels `bug`/`enhancement` and title emoji prefixes are often wrong or missing; do not use them.
- **Security fixes come from advisories**: an advisory whose `patched_versions` include the version becomes a `:lock:` entry (section 7). `triage`, `draft` and `published` are all valid states; the advisory may still be unpublished when the changelog is written. A security fix may also carry a regular bug issue; both entries are legal and independent (the issue lives in `:bug:` as any other bug).
- **Two labels override the Issue Type**: `breaking change` → an entry only under `:boom:`, never in its regular section; `release highlight` → an extra entry under `:rocket:` on top of its regular entry.
- **Only closed issues**. An open issue in the milestone is not generated, even if someone added a provisional entry by hand before the merge.
- **Excluded issues**: `no changelog`, `release blocker`, `Task` issue type, `Rejected` project status.
- **Excluded PRs**: a PR with `release blocker` or `no issue required` does not bring an entry of its own.
- **An issue with no milestone** belongs to another (probably private) project: it is neither changelog candidate nor anomaly.
## 3. Layout of `CHANGES.md`
The newest version goes at the top, right after the `# CHANGELOG` header. Inside a version, the section order is fixed:
```markdown
# CHANGELOG
## 2.19.0 (Unreleased)
### :boom: Breaking changes & Deprecations
#### <Breaking change title>
<content>
[#ISSUE](https://github.com/penpot/penpot/issues/ISSUE) (PR: [#PR](https://github.com/penpot/penpot/pull/PR))
### :rocket: Epics and highlights
- Add configurable keyboard shortcuts [#9924](https://github.com/penpot/penpot/issues/9924) (PR: [#10237](https://github.com/penpot/penpot/pull/10237))
### :lock: Security fixes
- Fix <user-facing description of the vulnerability> (<advisory URL>) (PR: [#PR](https://github.com/penpot/penpot/pull/PR))
### :bug: Bugs fixed
- Fix <user-facing description of the bug> [#ISSUE](https://github.com/penpot/penpot/issues/ISSUE) (PR: [#PR](https://github.com/penpot/penpot/pull/PR))
### :sparkles: New features & Enhancements
- Add <user-facing description of the feature> [#ISSUE](https://github.com/penpot/penpot/issues/ISSUE) (PR: [#PR](https://github.com/penpot/penpot/pull/PR))
```
- `:boom:` first, then `:rocket:`, `:lock:`, `:bug:`, `:sparkles:`. Other headings exist in old versions (`:heart:` community contributions); do not add them to new versions.
- Include a section only when it has entries. Historical versions may have an empty `:boom:` or `:rocket:` heading; `boom --write` keeps the `:boom:` heading even when there are no entries. `:lock:` is omitted (and removed on `--write`) when the version has no security advisories.
- One blank line between the last entry of a section and the next heading.
- No manual line wrapping: one line per bullet or paragraph, however long.
## 4. Regular entries (`:bug:`, `:sparkles:`)
The entry shape is:
```markdown
- <description> (by @github_username) [#ISSUE](https://github.com/penpot/penpot/issues/ISSUE) (PR: [#PR](https://github.com/penpot/penpot/pull/PR), [#PR2](https://github.com/penpot/penpot/pull/PR2))
```
- **Description**: derived from the issue title, not the PR title. Strip emoji prefixes (`:bug:`, `:sparkles:`, `:tada:`) and write the user-facing behaviour in imperative mood. Examples: `Plugin API token methods fail with schema validation error` → `Fix Plugin API token methods failing with schema validation error`; `Custom uploaded font family names are not sanitized` → `Sanitize font family names on custom uploaded fonts`.
- **Issue is the primary link**: the issue is the changelog unit, the PR is the implementation. Multiple fix PRs go comma-separated inline.
- **Community attribution**: when the issue or its PR carries `community contribution`, add `(by @username)` before the issue link. Use the PR author, not the issue author.
- **Closed issue with no PR** in the milestone: link the issue only, no PR reference.
- **PR with no linked issue**: link a matching closed issue in the milestone when one exists; otherwise skip it, because the issue is the changelog unit.
- **False-positive PR links**: a PR may claim to close an unrelated issue (ancient PR, cross-project reference). If the titles are clearly unrelated or the PR predates the issue by years, treat it as a data glitch and ignore it.
## 5. `:boom:`: breaking changes and deprecations
Each breaking change is a `####` block, not a bullet. The title is the issue title without its emoji prefix; the content comes from a comment on the issue; the references go on their own line at the end.
The **issue** must carry:
1. The `breaking change` label.
2. A comment whose first line is `## Breaking change note`. Everything after that marker line is the changelog content: paragraphs, bullet lists, code blocks, tables, headings (they are demoted to `#####` on write), and so on. The marker itself is case-insensitive and accepts `#`, `##` or `###`; use `##`.
Example comment:
```markdown
## Breaking change note
The Admin Console is now enabled by default and the `enable-admin-console` flag is no longer needed.
Self-hosted instances must deploy the `penpot-admin-console` service and set `PENPOT_ADMIN_CONSOLE_URI` in frontend and backend (check the [upgrade guide](https://help.penpot.app/technical-guide/getting-started/docker/#upgrade-to-2.19)).
```
Generated block:
```markdown
#### Enable the Admin Console by default
The Admin Console is now enabled by default and the `enable-admin-console` flag is no longer needed.
Self-hosted instances must deploy the `penpot-admin-console` service and set `PENPOT_ADMIN_CONSOLE_URI` in frontend and backend (check the [upgrade guide](https://help.penpot.app/technical-guide/getting-started/docker/#upgrade-to-2.19)).
[#12807](https://github.com/penpot/penpot/issues/12807) (PR: [#12475](https://github.com/penpot/penpot/pull/12475))
```
Rules for the note:
- **Write it for admins**: what changes, what they must do, and what breaks if they do nothing. Do not paste the full runbook; explain the implications and link the runbook or docs.
- **Link a long-lived branch**, `staging` or `main`, never a feature branch (`issue-1234`). The link may 404 until the PR merges; that is expected.
- **Do not add issue or PR links**: the generator appends them.
- **Do not hard-wrap the text**: one line per paragraph or bullet. The generator copies lines verbatim, so wrapped lines end up wrapped in `CHANGES.md`.
- **One note per issue**. If several comments carry the marker, the generator uses the last one and warns.
- **No note**: the generator warns on stderr and emits only the heading and references; fix the issue, not `CHANGES.md`.
- **`####`, not `###`**: `###` is the level of the changelog sections. An entry at `###` would render as a sibling section and the scripts that read and rewrite `:boom:` would cut the section there. The note author never writes the `####`: the generator adds it.
- **Only merged closing PRs** are referenced; unmerged or superseded PRs are skipped.
- **Never hand-edit the section**. Generate it with `boom` (section 8). The note lives on GitHub; edit it there.
## 6. `:rocket:`: epics and highlights
- Lists exactly the milestone issues with the `release highlight` label, and nothing else. Product decides highlights by labelling issues; a regeneration cannot undo that choice.
- Every entry copies the text and references of the issue's regular entry and MUST carry both an issue link and a PR reference.
- If no issue is labelled, leave the section out; do not pick highlights yourself.
- Never edit `:rocket:` by hand: add or remove the label and regenerate.
## 7. `:lock:`: security fixes
Generated from the advisories that declare the version in `patched_versions`; never written by hand.
- **Source and filter**: `python3 scripts/gh.py advisories --patched-version <VERSION>` keeps the advisories whose `vulnerabilities[*].patched_versions` contain the version as an exact token (so `2.19.1` never matches `2.19.10`). Any state is valid — `triage`, `draft`, `published` — because the advisory may still be unpublished when the changelog is written.
- **Fill the advisories first**: `patched_versions` must be filled before generating the changelog. The release process publishes the advisories with (or after) the release; the changelog does not wait for publication.
- **PR reference**: a merged PR in the milestone labelled `pr pt` whose body contains the GHSA id contributes its number inline; several PRs go comma-separated. An advisory with no PR is legal: the entry links only the advisory.
- **Missing `patched_versions`**: when a `pr pt` PR references an advisory that declares no matching `patched_versions`, `security` prints a warning on stderr and leaves it out. Fix the advisory, not `CHANGES.md`.
- **Entry format**: `- Fix <description> (<advisory URL>) (PR: [#PR](...))`. The description is `Fix` plus the advisory summary, with the first word decapitalized unless it starts with an acronym (`SVG`, `CVE-...`).
- **Wording**: the flow may polish the description. Re-running `security --write` rewrites it from the summary, so the durable place for wording is the advisory summary on GitHub; keep it a problem statement, because the generator adds `Fix`.
- **Empty version**: no matching advisories → the subsection is not inserted (and is removed by `--write`).
- **Never hand-edit the section**: generate it with `security` (section 8).
## 8. Flow
1. **Confirm the version** (a semver string such as `2.19.0`) with the user when it is not obvious.
2. **Fetch the milestone issues**: `python3 scripts/gh.py issues "<VERSION>" --exclude "release blocker,no changelog"` (closed by default; add `--state all` only to inspect open issues). `gh.py` already drops `Task` and `Rejected`; pass `--include-tasks` / `--include-rejected` only to inspect them. To find what is missing from an existing section, add `--compare CHANGES.md`.
3. **Fetch PR details when needed**: `python3 scripts/gh.py prs <numbers...>` (also `--milestone`). Use it for the PR author, labels and merge state.
4. **Categorize** by Issue Type and the two labels (section 2), and **build the descriptions** (section 4).
5. **Verify PR merge status** before writing: `python3 scripts/changelog.py check-merged <PR numbers...>`. Only merged PRs go in the changelog; if a closing PR was closed unmerged, look for the PR that superseded it.
6. **Reconcile the existing text** (section 9).
7. **Generate `:boom:`**: `python3 scripts/changelog.py boom "<VERSION>"` prints the section; add `--write` to replace it in `CHANGES.md`. The write replaces only that subsection, inserts it after the version header when missing, keeps the heading when there are no entries, and is idempotent.
8. **Generate `:rocket:`** from the `release highlight` label (section 6).
9. **Generate `:lock:`**: `python3 scripts/changelog.py security "<VERSION>"` prints the section; add `--write` to replace it in `CHANGES.md`. The contract lives in section 7.
10. **Cross-check PRs**: `python3 scripts/changelog.py cross-ref "<VERSION>"`. It lists merged milestone PRs missing from the section and warns about closed (unmerged) PRs. PRs linked from `:lock:` count as referenced; an advisory with no PR is invisible here by design.
11. **Generate the anomaly report**: `python3 scripts/changelog.py report "<VERSION>"`. It overwrites `CHANGES-ISSUES.md` with the current state (section 10).
## 9. Pre-flight: reconcile before writing
Re-fetch the milestone (do not trust cached data) and fix these in `CHANGES.md` before adding the new section:
1. **Duplicate across versions** → remove it from the current section; the earlier version wins for backports. The same holds after merging a patch branch into `staging`: the fix keeps its released patch entry and leaves the newer, unreleased section (the merge conflict in `CHANGES.md` is resolved by keeping both sections, newest first, minus the duplicate).
2. **Stale milestone assignment** (the issue moved out of this milestone) → remove the entry.
3. **Newly applied exclusion labels** (`no changelog`, `release blocker`) → remove the entry.
4. **Issue no longer closed, deleted, or now `Rejected`** → remove the entry.
5. **Unmerged or moved PR reference** → fix the reference or remove the entry. A PR merged in a different milestone is an anomaly (section 10); do not remove it silently.
6. **Issue Type changed to `Task`** → remove the entry.
7. **Breaking change out of sync** (a `breaking change` issue outside `:boom:`, or a `:boom:` entry without the label) → regenerate the section; a labelled issue has no regular entry.
8. **Highlight out of sync** (`:rocket:` entry without the label, or a labelled issue missing) → remove it or add it.
9. **Missing valid issues** (closed, non-excluded, referenced nowhere) → add them to their section.
## 10. Anomaly report
`report` classifies problems it cannot decide alone. The total counts A, B, E, F and G; C and D are warnings.
- **A — issue in this milestone, referenced PR in another milestone (or none).** The claim "this issue is fixed by this PR in this milestone" is inconsistent. Fix the milestone on the issue or the PR, or remove the entry.
- **B — PR in this milestone whose closing issue is in another milestone.** Same fix. An issue with no milestone is not an anomaly (another, probably private, project).
- **E — `:boom:` entry whose issue lacks the `breaking change` label.** The entry stays in the changelog: either label the issue or move the entry to its regular section.
- **F — `:rocket:` out of sync with the `release highlight` label.** A rule violation: `:rocket:` must list exactly the labelled issues. Fix the label or re-run the flow.
- **G — `breaking change` issue missing from `:boom:`.** A rule violation: `:boom:` must list exactly the labelled issues. Regenerate the section or relabel.
- **C — released X.Y.0 version without a `:rocket:` section** (gap; patches never carry highlights).
- **D — `:rocket:` entry without issue AND PR references** (gap).
Anything else is a rule violation, not a report item: fix it in the pre-flight and re-run.
## 11. Pitfalls
- **Open issues with provisional entries**: a PR author may add the entry before the merge. `boom` will not generate it until the issue closes; keep the hand-written block identical to what the generator would produce.
- **Labels, not emoji**: title prefixes are cosmetic and often missing; the section comes from Issue Type and the two labels.
- **Taiga references**: old entries link to Taiga. Resolve them to the GitHub issue via the description text or the PRs that mention the Taiga URL, then link issue and PR.
- **Long-lived links only**: runbooks and docs referenced from a note must point to `staging` or `main`, never to a feature branch that will be deleted.
- **Generated sections**: `:boom:`, `:rocket:` and `:lock:` are generated. Never patch them by hand; edit the label, the note comment or the advisory and re-run.
- **Deleted or transferred issues**: an issue referenced in the changelog that returns 410/404 (deleted or moved out of the repo) must lose its entry in the pre-flight. Its security fixes, if any, stay in `:lock:` through the advisory.
## 12. Where the code is
- `scripts/gh.py`: `issues`, `issue`, `comments`, `prs`, `link-issue`, `advisories` (with `--patched-version`). Details: `mem:scripts/gh`.
- `scripts/changelog.py`: `check-merged`, `boom`, `security`, `cross-ref`, `report`. Details: `mem:scripts/changelog`.
- Tests: `scripts/test_gh.py`, `scripts/test_changelog.py`.
- The note contract for issue authors: `mem:workflow/creating-issues`, section **Breaking changes**.

3
AGENTS.md
View File

@ -10,7 +10,8 @@
If the user pushes, treat that commit as final from the agent's side.
- **Never edit `CHANGES.md` by hand** in commits or PRs. The changelog is
generated from GitHub milestones during the release process; update it only
via the `update-changelog` skill flow or on explicit user request.
via `mem:workflow/updating-changelog` (the `update-changelog` skill flow) or
on explicit user request.
- **Never pipe test output directly to filters** (`| head`, `| tail`, `| grep`, etc.).
Always redirect to a file first: `command > /tmp/output.txt 2>&1`, then read/grep the file.
This prevents hiding test failures. See `mem:testing` for details.

150
CHANGES.md
View File

@ -1,6 +1,95 @@
# CHANGELOG
## 2.19.0 (Unreleased)
## 2.20.0
### :bug: Bugs fixed
- Fix dashboard file thumbnails loading slow for large multi-page files [#12053](https://github.com/penpot/penpot/issues/12053) (PR: [#12054](https://github.com/penpot/penpot/pull/12054))
- Fix Plugin API resolving a font family token to its separate words when the family name has several words [#11748](https://github.com/penpot/penpot/issues/11748) (PR: [#11750](https://github.com/penpot/penpot/pull/11750))
- Fix asset redirect responses advertising a cache lifetime of about 1000 days instead of 24 hours (by @ners) [#12761](https://github.com/penpot/penpot/issues/12761) (PR: [#12758](https://github.com/penpot/penpot/pull/12758))
- Fix an internal error on typography tokens due to circular dependencies on other token types [#9580](https://github.com/penpot/penpot/issues/9580)
- Fix the MCP server rejecting request bodies over 100kb (express.json default) with PayloadTooLargeError (by @GitNimay) [#12080](https://github.com/penpot/penpot/issues/12080) (PR: [#12450](https://github.com/penpot/penpot/pull/12450))
- Fix library component, color and typography counters showing zero when opening files from the dashboard (by @lukel97) [#12826](https://github.com/penpot/penpot/issues/12826) (PR: [#12787](https://github.com/penpot/penpot/pull/12787))
- Fix workspace crash with Maximum call stack size exceeded when applying a library update [#12576](https://github.com/penpot/penpot/issues/12576) (PR: [#12661](https://github.com/penpot/penpot/pull/12661))
- Fix canvas rendering an imported shape without its hole when the SVG fill rule is evenodd [#11947](https://github.com/penpot/penpot/issues/11947) (PR: [#12006](https://github.com/penpot/penpot/pull/12006))
- Fix Plugin API component.instance() and createShapeFromSvg() nesting the new shape into whatever board is at (0, 0) of the current page (by @makesomethingshit) [#12086](https://github.com/penpot/penpot/issues/12086) (PR: [#12499](https://github.com/penpot/penpot/pull/12499))
- Fix view mode breaking zoom, board navigation, and comments when the URL has no page id [#12684](https://github.com/penpot/penpot/issues/12684) (PR: [#12687](https://github.com/penpot/penpot/pull/12687))
- Fix autosave stopping and valid edits being lost when part of a save fails validation [#12553](https://github.com/penpot/penpot/issues/12553) (PR: [#12555](https://github.com/penpot/penpot/pull/12555))
- Fix dashboard duplicate copying a file without the edits that are still saving [#12603](https://github.com/penpot/penpot/issues/12603) (PR: [#12555](https://github.com/penpot/penpot/pull/12555))
- Fix Alt measurements still showing a shape that was deleted [#12605](https://github.com/penpot/penpot/issues/12605) (PR: [#12555](https://github.com/penpot/penpot/pull/12555))
- Fix Plugin API ignoring changes to a deleted shape without an error [#12606](https://github.com/penpot/penpot/issues/12606) (PR: [#12555](https://github.com/penpot/penpot/pull/12555))
- Fix typography sample overflowing its row in locales with a wider sample (ko, ar) (by @desty) [#11486](https://github.com/penpot/penpot/issues/11486) (PR: [#11487](https://github.com/penpot/penpot/pull/11487))
- Fix token not showing when selecting two text layers (by @ShreyashAgare26) [#11924](https://github.com/penpot/penpot/issues/11924) (PR: [#11944](https://github.com/penpot/penpot/pull/11944))
- Fix inner shadow fill not working in the SVG render while it works in WebGL [#11967](https://github.com/penpot/penpot/issues/11967) (PR: [#12141](https://github.com/penpot/penpot/pull/12141))
- Fix @penpot/plugin-styles loading Work Sans from a third-party CDN (by @ump45nose) [#12023](https://github.com/penpot/penpot/issues/12023) (PR: [#12087](https://github.com/penpot/penpot/pull/12087))
### :sparkles: New features & Enhancements
- Add a feedback channel to the MCP server [#12490](https://github.com/penpot/penpot/issues/12490) (PR: [#12551](https://github.com/penpot/penpot/pull/12551))
- Auto-scroll sortable lists when dragging an item near their top or bottom edge (by @ShreyashAgare26) [#10478](https://github.com/penpot/penpot/issues/10478) (PR: [#11595](https://github.com/penpot/penpot/pull/11595))
- Make WebGL the default renderer for new accounts [#11855](https://github.com/penpot/penpot/issues/11855) (PR: [#12880](https://github.com/penpot/penpot/pull/12880))
- Allow MCP to work with multiple open Penpot files [#11759](https://github.com/penpot/penpot/issues/11759)
- Complete the Russian translations [#11955](https://github.com/penpot/penpot/issues/11955) (PR: [#11956](https://github.com/penpot/penpot/pull/11956))
- Implement full text format paste within Penpot [#12047](https://github.com/penpot/penpot/issues/12047) (PR: [#12137](https://github.com/penpot/penpot/pull/12137))
- Prevent the hosted MCP server from running execute_code twice when the plugin is open in two tabs [#12008](https://github.com/penpot/penpot/issues/12008) (PR: [#11927](https://github.com/penpot/penpot/pull/11927))
- Allow detaching typography tokens [#11887](https://github.com/penpot/penpot/issues/11887) (PR: [#11969](https://github.com/penpot/penpot/pull/11969))
- Add team, project and file management to the Plugin API for global plugins [#12170](https://github.com/penpot/penpot/issues/12170) (PR: [#12169](https://github.com/penpot/penpot/pull/12169))
- Run .penpot file import and export as durable background jobs [#12194](https://github.com/penpot/penpot/issues/12194) (PR: [#12056](https://github.com/penpot/penpot/pull/12056), [#12192](https://github.com/penpot/penpot/pull/12192))
- Open files and projects in a new tab with Ctrl/Cmd+click in the dashboard [#12228](https://github.com/penpot/penpot/issues/12228) (PR: [#12230](https://github.com/penpot/penpot/pull/12230))
- Add an RPC method to read file-level plugin data [#12258](https://github.com/penpot/penpot/issues/12258) (PR: [#10171](https://github.com/penpot/penpot/pull/10171))
- Add a get-file-tokens RPC method to read a file's design tokens [#12260](https://github.com/penpot/penpot/issues/12260) (PR: [#12172](https://github.com/penpot/penpot/pull/12172))
- Show the progress and the size of each file in the import and export dialogs [#12340](https://github.com/penpot/penpot/issues/12340) (PR: [#12192](https://github.com/penpot/penpot/pull/12192))
- Focus the search field when opening the color tokens tab of the color picker [#12344](https://github.com/penpot/penpot/issues/12344) (PR: [#12345](https://github.com/penpot/penpot/pull/12345))
- Harden the file data offload job with a row lock, per-file transactions, throttle and metrics [#12398](https://github.com/penpot/penpot/issues/12398) (PR: [#12403](https://github.com/penpot/penpot/pull/12403))
- Render email templates inside the mail job in the backend [#12445](https://github.com/penpot/penpot/issues/12445) (PR: [#12448](https://github.com/penpot/penpot/pull/12448))
- Serve Penpot under a subpath [#3194](https://github.com/penpot/penpot/issues/3194) (PR: [#12358](https://github.com/penpot/penpot/pull/12358))
- Run local WASM exports for shapes and boards without freezing the UI [#12512](https://github.com/penpot/penpot/issues/12512) (PR: [#12515](https://github.com/penpot/penpot/pull/12515))
- Use WASM to generate SVG markup in the code inspector when render-wasm is active [#12567](https://github.com/penpot/penpot/issues/12567) (PR: [#12608](https://github.com/penpot/penpot/pull/12608))
- Close the workspace search with the Escape key (by @filipsajdak) [#9540](https://github.com/penpot/penpot/issues/9540) (PR: [#12374](https://github.com/penpot/penpot/pull/12374))
- Allow MCP to access and manage Penpot Dashboard resources [#11761](https://github.com/penpot/penpot/issues/11761)
- Add OAuth support for MCP client connections [#12034](https://github.com/penpot/penpot/issues/12034) (PR: [#12480](https://github.com/penpot/penpot/pull/12480))
- Restrict management RPC methods to their declared shared keys [#12730](https://github.com/penpot/penpot/issues/12730) (PR: [#12736](https://github.com/penpot/penpot/pull/12736))
- Reduce memory used by duplicate shape ids in the workspace [#12791](https://github.com/penpot/penpot/issues/12791) (PR: [#12792](https://github.com/penpot/penpot/pull/12792))
- Reduce memory consumption on shape decoding [#12789](https://github.com/penpot/penpot/issues/12789) (PR: [#12790](https://github.com/penpot/penpot/pull/12790))
- Notify other users when a snapshot is restored [#12425](https://github.com/penpot/penpot/issues/12425) (PR: [#12555](https://github.com/penpot/penpot/pull/12555))
- Normalize the configured tenant to lowercase [#12842](https://github.com/penpot/penpot/issues/12842) (PR: [#12843](https://github.com/penpot/penpot/pull/12843))
- Speed up saves on large files by checking only the pages a save changes [#12602](https://github.com/penpot/penpot/issues/12602) (PR: [#12555](https://github.com/penpot/penpot/pull/12555))
- Retry saves after short server failures and keep the workspace open when a save fails [#12601](https://github.com/penpot/penpot/issues/12601) (PR: [#12555](https://github.com/penpot/penpot/pull/12555))
- Switch the workspace to read-only mode when the file can't be saved [#12507](https://github.com/penpot/penpot/issues/12507) (PR: [#12555](https://github.com/penpot/penpot/pull/12555))
## 2.19.1
### :lock: Security fixes
- Fix editor-role team members retrieving the signed invitation token of any pending invitation (https://github.com/penpot/penpot/security/advisories/GHSA-4mmc-j8rh-74pf) (PR: [#12892](https://github.com/penpot/penpot/pull/12892))
- Fix share-link page-scope escape in `get-profiles-for-file-comments` leaking the identity of comment authors outside the link's scope (https://github.com/penpot/penpot/security/advisories/GHSA-89cq-3wv7-987r) (PR: [#12888](https://github.com/penpot/penpot/pull/12888))
- Fix missing authorization and unbounded array in mark-all-threads-as-read that let any authenticated user write the status of inaccessible threads (https://github.com/penpot/penpot/security/advisories/GHSA-j4vc-cmf6-jhr7) (PR: [#12894](https://github.com/penpot/penpot/pull/12894))
- Fix incomplete SVG sanitizer escaping that let a "sanitized" SVG carry live script (CVE-2026-105694) (https://github.com/penpot/penpot/security/advisories/GHSA-qwhj-rchj-58mg) (PR: [#12887](https://github.com/penpot/penpot/pull/12887))
### :bug: Bugs fixed
- Fix component instance (variant) inside a layout resetting to its original size after a shared library update of an unrelated component [#12670](https://github.com/penpot/penpot/issues/12670) (PR: [#12851](https://github.com/penpot/penpot/pull/12851))
## 2.19.0
### :boom: Breaking changes & Deprecations
#### Migrate legacy file payloads to file_data and delete orphan revisions
File payloads now live in one place: every file state and snapshot is kept in `file_data`, and the orphan `file_change` rows that kept old payloads alive forever are removed. No read path reached those rows and nothing ever collected them, so the upgrade also reclaims the database and TOAST space they held. There is no API or UI change.
On small and medium instances this happens automatically on boot and there is nothing to do. On large instances (millions of snapshots, hundreds of GB in TOAST) the work does not fit in the startup transaction: run it by hand **before upgrading to 2.19.0** following the [large-instance runbook](https://github.com/penpot/penpot/blob/staging/operations/2.19.0-migrate-legacy-file-data.md). The runbook tells you whether your instance needs it, keeps the instance online while the data moves, and verifies the result before you deploy.
[#12784](https://github.com/penpot/penpot/issues/12784) (PR: [#12785](https://github.com/penpot/penpot/pull/12785))
#### Enable the Admin Console by default
The Admin Console is now enabled by default and the `enable-admin-console` flag is no longer needed. Self-hosted instances must deploy the `penpot-admin-console` service and set `PENPOT_ADMIN_CONSOLE_URI` in frontend and backend (check the [upgrade guide](https://help.penpot.app/technical-guide/getting-started/docker/#upgrade-to-2.19)).
[#12807](https://github.com/penpot/penpot/issues/12807) (PR: [#12475](https://github.com/penpot/penpot/pull/12475))
### :rocket: Epics and highlights
@ -74,6 +163,38 @@
- Fix thumbnails failing on older Chromium browsers and each thumbnail rendering several times [#11878](https://github.com/penpot/penpot/issues/11878) (PR: [#11879](https://github.com/penpot/penpot/pull/11879))
- Fix internal error when opening the libraries modal [#11943](https://github.com/penpot/penpot/issues/11943) (PR: [#11946](https://github.com/penpot/penpot/pull/11946))
- Fix Plugin API `connectLibrary` linking unpublished files as libraries [#12001](https://github.com/penpot/penpot/issues/12001) (PR: [#12002](https://github.com/penpot/penpot/pull/12002))
- Fix incorrect padding at the top of the tokens and themes section [#9585](https://github.com/penpot/penpot/issues/9585) (PR: [#12422](https://github.com/penpot/penpot/pull/12422))
- Fix token tree path handling for repeated and nested segments (by @makesomethingshit) [#11588](https://github.com/penpot/penpot/issues/11588) (PR: [#11591](https://github.com/penpot/penpot/pull/11591))
- Fix comment input duplicating text when confirming a Japanese IME composition with Enter [#11757](https://github.com/penpot/penpot/issues/11757) (PR: [#11768](https://github.com/penpot/penpot/pull/11768))
- Fix file not saving after deleting a variant [#11758](https://github.com/penpot/penpot/issues/11758) (PR: [#12216](https://github.com/penpot/penpot/pull/12216))
- Fix component copies not syncing when a shape is moved out of a main component and another change follows at once [#11978](https://github.com/penpot/penpot/issues/11978) (PR: [#11981](https://github.com/penpot/penpot/pull/11981))
- Fix MCP plugin showing a version mismatch warning on every patch release (by @robertkodra) [#12007](https://github.com/penpot/penpot/issues/12007) (PR: [#12009](https://github.com/penpot/penpot/pull/12009))
- Fix workspace crash when switching to the path tool while drawing a shape [#12100](https://github.com/penpot/penpot/issues/12100) (PR: [#12101](https://github.com/penpot/penpot/pull/12101))
- Fix referential integrity error when swapping a component [#12144](https://github.com/penpot/penpot/issues/12144) (PR: [#12333](https://github.com/penpot/penpot/pull/12333))
- Fix workspace saving text shapes on every page open when a custom font is missing [#12151](https://github.com/penpot/penpot/issues/12151) (PR: [#12164](https://github.com/penpot/penpot/pull/12164))
- Fix swapped nested components losing their absolute position after a library update [#12176](https://github.com/penpot/penpot/issues/12176) (PR: [#12556](https://github.com/penpot/penpot/pull/12556))
- Fix theme selector showing the editor message when the tokens source is a library [#12177](https://github.com/penpot/penpot/issues/12177) (PR: [#12203](https://github.com/penpot/penpot/pull/12203))
- Fix untranslated texts in the This file tab of the libraries modal [#12178](https://github.com/penpot/penpot/issues/12178) (PR: [#12204](https://github.com/penpot/penpot/pull/12204))
- Fix workspace staying in the saving state and losing unsaved edits when a save request gets no answer [#12222](https://github.com/penpot/penpot/issues/12222) (PR: [#12223](https://github.com/penpot/penpot/pull/12223))
- Fix scrub cursor showing on numeric inputs with mixed values in the sidebar [#12227](https://github.com/penpot/penpot/issues/12227) (PR: [#12229](https://github.com/penpot/penpot/pull/12229))
- Fix internal error when copying a swapped nested component [#12289](https://github.com/penpot/penpot/issues/12289) (PR: [#12434](https://github.com/penpot/penpot/pull/12434))
- Show an error when uploading a custom font fails [#12304](https://github.com/penpot/penpot/issues/12304) (PR: [#12315](https://github.com/penpot/penpot/pull/12315))
- Fix React warning about the unknown `tooltipClass` prop in layout-item size inputs [#12311](https://github.com/penpot/penpot/issues/12311) (PR: [#12329](https://github.com/penpot/penpot/pull/12329))
- Fix guides inside a board still being interactive [#12322](https://github.com/penpot/penpot/issues/12322) (PR: [#12460](https://github.com/penpot/penpot/pull/12460))
- Fix pasted text not being visible in an empty text box until it is resized [#12438](https://github.com/penpot/penpot/issues/12438) (PR: [#12859](https://github.com/penpot/penpot/pull/12859))
- Fix files failing referential integrity validation staying unsaveable in every session [#12464](https://github.com/penpot/penpot/issues/12464) (PR: [#12524](https://github.com/penpot/penpot/pull/12524), [#12662](https://github.com/penpot/penpot/pull/12662))
- Fix unicode characters not rendering in text layers with the new render engine [#12469](https://github.com/penpot/penpot/issues/12469) (PR: [#12471](https://github.com/penpot/penpot/pull/12471))
- Fix text image fills staying blank when media loads after text layout [#12563](https://github.com/penpot/penpot/issues/12563) (PR: [#12564](https://github.com/penpot/penpot/pull/12564))
- Fix swapped nested icons moving outside the layout after a library update [#12657](https://github.com/penpot/penpot/issues/12657) (PR: [#12716](https://github.com/penpot/penpot/pull/12716))
- Fix garbled bullets before the element counts in the libraries modal [#12677](https://github.com/penpot/penpot/issues/12677) (PR: [#12678](https://github.com/penpot/penpot/pull/12678))
- Fix token dropdown of numeric inputs showing the scrub cursor [#12679](https://github.com/penpot/penpot/issues/12679) (PR: [#12680](https://github.com/penpot/penpot/pull/12680))
- Fix variant property names and values accepting `,` and `=` [#12683](https://github.com/penpot/penpot/issues/12683) (PR: [#12688](https://github.com/penpot/penpot/pull/12688))
- Fix changing a variant property from another page leaving the main component label out of date [#12692](https://github.com/penpot/penpot/issues/12692) (PR: [#12693](https://github.com/penpot/penpot/pull/12693))
- Fix `:referential-integrity` errors when moving variant mains in or out of a set [#12696](https://github.com/penpot/penpot/issues/12696) (PR: [#12706](https://github.com/penpot/penpot/pull/12706))
- Fix `:referential-integrity` errors when creating, combining or renaming variant sets [#12734](https://github.com/penpot/penpot/issues/12734) (PR: [#12779](https://github.com/penpot/penpot/pull/12779))
- Fix save errors after masking a variant with Ctrl+M [#12748](https://github.com/penpot/penpot/issues/12748) (PR: [#12749](https://github.com/penpot/penpot/pull/12749))
- Fix icons in component instances not being shown on the canvas [#12752](https://github.com/penpot/penpot/issues/12752)
- Fix upload endpoints accepting non-multipart upload paths and validation error bodies being truncated [#12896](https://github.com/penpot/penpot/issues/12896) (PR: [#12067](https://github.com/penpot/penpot/pull/12067))
### :sparkles: New features & Enhancements
@ -89,7 +210,7 @@
- Apply tokens to stroke per side [#9842](https://github.com/penpot/penpot/issues/9842) (PR: [#10913](https://github.com/penpot/penpot/pull/10913))
- Add individual stroke widths per side (top, right, bottom, left) [#9854](https://github.com/penpot/penpot/issues/9854) (PR: [#10648](https://github.com/penpot/penpot/pull/10648))
- Expand the token tree right after importing it (by @AKnassa) [#9819](https://github.com/penpot/penpot/issues/9819) (PR: [#10664](https://github.com/penpot/penpot/pull/10664))
- Update the selection size badge live during resize, replacing the resize tooltip [#10259](https://github.com/penpot/penpot/issues/10259)
- Update the selection size badge live during resize, replacing the resize tooltip [#10259](https://github.com/penpot/penpot/issues/10259) (PR: [#11884](https://github.com/penpot/penpot/pull/11884))
- Add color variants, rotation correction, viewport flip and hide-when-too-small to the selection size badge [#10264](https://github.com/penpot/penpot/issues/10264) (PR: [#11884](https://github.com/penpot/penpot/pull/11884))
- Support single-file SVG export with the WebGL render [#10546](https://github.com/penpot/penpot/issues/10546)
- Persist the hide resolved comments preference and separate the filter options in the comments menu [#10686](https://github.com/penpot/penpot/issues/10686) (PR: [#10694](https://github.com/penpot/penpot/pull/10694))
@ -113,13 +234,36 @@
- Make "Fix when scrolling" work in the WebGL render [#11684](https://github.com/penpot/penpot/issues/11684) (PR: [#11752](https://github.com/penpot/penpot/pull/11752))
- Complete Catalan translations [#11739](https://github.com/penpot/penpot/issues/11739) (PR: [#11741](https://github.com/penpot/penpot/pull/11741))
- Remove the optional client-provided id from creation RPC endpoints [#11783](https://github.com/penpot/penpot/issues/11783) (PR: [#11784](https://github.com/penpot/penpot/pull/11784))
- Allow color tokens for the canvas background [#11791](https://github.com/penpot/penpot/issues/11791) (PR: [#11923](https://github.com/penpot/penpot/pull/11923))
- Allow color tokens for the canvas background [#11791](https://github.com/penpot/penpot/issues/11791) (PR: [#11923](https://github.com/penpot/penpot/pull/11923), [#11950](https://github.com/penpot/penpot/pull/11950))
- Show a "no assets yet" message for empty files in the libraries modal [#11811](https://github.com/penpot/penpot/issues/11811) (PR: [#11818](https://github.com/penpot/penpot/pull/11818))
- Upgrade the MCP SDK to v2 and adopt the new protocol version [#11827](https://github.com/penpot/penpot/issues/11827) (PR: [#11841](https://github.com/penpot/penpot/pull/11841))
- Remove legacy SSE support from the MCP server [#11846](https://github.com/penpot/penpot/issues/11846) (PR: [#11841](https://github.com/penpot/penpot/pull/11841))
- Delete selected path handles with Backspace or Delete [#11858](https://github.com/penpot/penpot/issues/11858) (PR: [#11896](https://github.com/penpot/penpot/pull/11896))
- Use the tooltip component in the token pill info [#11886](https://github.com/penpot/penpot/issues/11886) (PR: [#11900](https://github.com/penpot/penpot/pull/11900))
- Hide the duplicate cursor when alt-dragging a shape that cannot be duplicated (by @0xTHAC0) [#11905](https://github.com/penpot/penpot/issues/11905) (PR: [#11195](https://github.com/penpot/penpot/pull/11195))
- Document the team auto-add behavior with `disable-email-verification` [#11376](https://github.com/penpot/penpot/issues/11376) (PR: [#11425](https://github.com/penpot/penpot/pull/11425))
- Make the workspace recover from interrupted saves and show a proper error instead of the internal error page [#11799](https://github.com/penpot/penpot/issues/11799)
- Add component management operations to the Plugin API [#12301](https://github.com/penpot/penpot/issues/12301) (PR: [#12303](https://github.com/penpot/penpot/pull/12303))
- Record the imported file ids in the import-binfile audit event [#12485](https://github.com/penpot/penpot/issues/12485) (PR: [#12491](https://github.com/penpot/penpot/pull/12491))
## 2.18.7
### :bug: Bugs fixed
- Fix binfile import accepting library relations outside the package and self-relations [#12895](https://github.com/penpot/penpot/issues/12895) (PR: [#12862](https://github.com/penpot/penpot/pull/12862))
## 2.18.6
### :bug: Bugs fixed
- Fix upload endpoints storing server files when the path points outside the temp dir [#12771](https://github.com/penpot/penpot/issues/12771) (PR: [#12772](https://github.com/penpot/penpot/pull/12772))
## 2.18.5
### :bug: Bugs fixed
- Fix the Admin Console failing on every URL on Docker deployments by reverting the subpath support [#12358](https://github.com/penpot/penpot/pull/12358)
## 2.18.4
### :bug: Bugs fixed

2
backend/deps.edn
View File

@ -54,6 +54,8 @@
org.jsoup/jsoup {:mvn/version "1.23.2"}
com.j256.simplemagic/simplemagic {:mvn/version "1.17"}
at.yawk.lz4/lz4-java
{:mvn/version "1.12.0"}

2
backend/package.json
View File

@ -4,7 +4,7 @@
"license": "MPL-2.0",
"author": "Kaleidos INC Sucursal en España SL",
"private": true,
"packageManager": "pnpm@12.16.1+sha512.091d1dee7927575661b25dab3c2ae31cafc1c54d6b71ed6a6600cf0c1cb10434a447eae14015f1b23e2e6bedeed79df94a932e2164d2832280dce3fb32f656d1",
"packageManager": "pnpm@12.18.0+sha512.9d7882d5dc82147ecfc4e062aa34834e0ad0296710a1837a1115ae012127d36b7109e321ac1598a1dd0a65597681b1e99917e12ed0d749eae91187943363beb3",
"repository": {
"type": "git",
"url": "https://github.com/penpot/penpot"

122
backend/pnpm-lock.yaml generated
View File

@ -7,153 +7,153 @@ importers:
configDependencies: {}
packageManagerDependencies:
pnpm:
specifier: 12.16.1
version: 12.16.1
specifier: 12.18.0
version: 12.18.0
packages:
'@pnpm/exe.android-arm64@12.16.1':
resolution: {integrity: sha512-PKf78Ew+i0CBYf0Xc+eMcqQiqyBT+u8XCVg1F5roc98/6wWfdYlWhrUMmEBiKWaJ3JqHxrtt8l1NwZ7qfJt2/Q==}
'@pnpm/exe.android-arm64@12.18.0':
resolution: {integrity: sha512-fOF6IwgvnUKEmVedGqiAsNPfN2Ry8GvaqzSQOiF3RJFbpBfPKO/6YT54s1GjToyYinRxw9preHHPe4y2N3SzhA==}
cpu: [arm64]
os: [android]
'@pnpm/exe.android-x64@12.16.1':
resolution: {integrity: sha512-xdu7gMbB98pdvhwMeA6612Of515FDlRUtfA/1f7nWPZfPLnEdSRBO2u+1N0XBtQ/ennhPH/lrxB1gC6gnR/cdg==}
'@pnpm/exe.android-x64@12.18.0':
resolution: {integrity: sha512-6OiolnaCsB3/GcJ9XNrenbUH3zFHRJ+5QynrMUUEyThbgnnCzWxa199hpQlOmxN4rAXuYQ0IYgWFNUZxyzg31Q==}
cpu: [x64]
os: [android]
'@pnpm/exe.darwin-arm64@12.16.1':
resolution: {integrity: sha512-pVbmIIbhixiLvADLu1WA3TaW8FZy/D72R1De27WbbFx8/2mxvnsTyFVm4fO9kjpjMY7RzJd48CiQ96p8fO3ebQ==}
'@pnpm/exe.darwin-arm64@12.18.0':
resolution: {integrity: sha512-zkq+4YzVIX39ETYLyC3W7LY7MpHEU/J3nRhHl4i2yPld2Cf2+rEKRh3ejJ4AmaQ77HHJLCktY4Z3jq0fpcxFXg==}
cpu: [arm64]
os: [darwin]
'@pnpm/exe.darwin-x64@12.16.1':
resolution: {integrity: sha512-xZvQBFqcZjO7vFKTK3XsJpqY6lnBL9EWSLCzVUPq3kZ94+D96gwJCa76xxGNK8Cj32f4eeM91Qg+yNRR4sRpDw==}
'@pnpm/exe.darwin-x64@12.18.0':
resolution: {integrity: sha512-qMsL9qx92unOTa6hgkfiTJv8ceZaTx0IQg7InhYDVX2ZrTECujaVEEVypnJK5m3eGUsP1KD7Y4qIgleCwfRsYw==}
cpu: [x64]
os: [darwin]
'@pnpm/exe.freebsd-x64@12.16.1':
resolution: {integrity: sha512-zTtunfi9mBhRXiXEJYfwiJj0tk7MgTSfqDSRh+Az9FZ9THazVIiQiVNS8jpcsftDOqu7LKNffQdjmgwHDThfUg==}
'@pnpm/exe.freebsd-x64@12.18.0':
resolution: {integrity: sha512-/9vy/4ms52cU4MjZ5NhHd6N2d9L9W0kARf5/ff2tweAHu0VeUzs/0qp/7ZHFuddm2Wa/m0AKQoPum9AVUCqKsw==}
cpu: [x64]
os: [freebsd]
'@pnpm/exe.linux-arm64-musl@12.16.1':
resolution: {integrity: sha512-BJ3mOmcmQgcUo92Ee/WfEOt1pRiv3y2MMAkhEiYHd85UiPpHTfFhv+SpCbALmfExzXk3UlaRtMYwHlkQymc4VA==}
'@pnpm/exe.linux-arm64-musl@12.18.0':
resolution: {integrity: sha512-Vp+TIMPPyackVCXkOdEKH/RmAn/r07EcIyrgZYQK7VrkksqZOie0GnWIx+85mPqeFEYCRtXIZAFJ9L4ceTndDg==}
cpu: [arm64]
os: [linux]
libc: [musl]
'@pnpm/exe.linux-arm64@12.16.1':
resolution: {integrity: sha512-JaRd5gNxwdNGBBkrZFt8bknrK4YmHVJBW0JMhUQ0fJ0dEYp6C2GT9ExaCqOO8EwYEyqM+jpKgJIK+8WFaJy3ew==}
'@pnpm/exe.linux-arm64@12.18.0':
resolution: {integrity: sha512-2dpyVBCHT+FfpuQh9i+7MXAqNWYLCvD5vYr5p6Eimxoga8USLnjbUgyj72AOrO0hjc5ri3TEFP3jiyBHjkSZpQ==}
cpu: [arm64]
os: [linux]
libc: [glibc]
'@pnpm/exe.linux-ppc64@12.16.1':
resolution: {integrity: sha512-FnSV0Jv40S1cMv7vFNWwvu0X567Lec4L4HCRead3CFcD6zQDNbGw81s8N5bcKPahtPD9ApVuhsbmFCx4rw6RMw==}
'@pnpm/exe.linux-ppc64@12.18.0':
resolution: {integrity: sha512-BBOOOfNJ2QwIIya3eKqOjiN4WNTszfRz7HCvw9XggcT7+ftMjNP8XJ/GfW+ltCBeUFR9d8WBVsr2m/NLjX2CZQ==}
cpu: [ppc64]
os: [linux]
libc: [glibc]
'@pnpm/exe.linux-riscv64@12.16.1':
resolution: {integrity: sha512-C1Sww4FN5IttD4atRRhAKmQtSGWVp04ZAKNxO8pfu2bkRPwOW+7v+2DH1Xs1W+kZYYpO86BYDvUcadefF+AgaQ==}
'@pnpm/exe.linux-riscv64@12.18.0':
resolution: {integrity: sha512-7kMN77xSOYJX8GPSH5RbNAc5oM61LMmcSUukDDUwllnZPXm8GRPskCNsTWy8SCw01Kr1FB+5SwGNMiTdiziLNQ==}
cpu: [riscv64]
os: [linux]
libc: [glibc]
'@pnpm/exe.linux-s390x@12.16.1':
resolution: {integrity: sha512-kn2fautsTcxIMODoovQVqZyfpDRdNgGkoeVqNdrKqbeBBvrEl+fXLkmZiq5xDht1LMr4sYiCNcsMfNMT6JPFmg==}
'@pnpm/exe.linux-s390x@12.18.0':
resolution: {integrity: sha512-htuSCpEcmDzhKoJqXbPkvfr176dBHU0QIF5mWcRiqxRXAB/1JoYiMYotfjWpk+ztgpB9qlx4XnWlLFAf7gzc6g==}
cpu: [s390x]
os: [linux]
libc: [glibc]
'@pnpm/exe.linux-x64-musl@12.16.1':
resolution: {integrity: sha512-PM6ccHCOe/O9SsTflTJTYWfwogJZa6EmUvpnwZgiPguJBddh20wDQKgxUPXf48rDsPm9vMsIij0uHswwwbXL8g==}
'@pnpm/exe.linux-x64-musl@12.18.0':
resolution: {integrity: sha512-Uj+vzfIQ62kDP10x36yHZIXVs5ohdNtiPYwn5FhCLlNScHPo0OAI+ShGDtLofiBXKE7umu71Jmjm7N8a8zVcDw==}
cpu: [x64]
os: [linux]
libc: [musl]
'@pnpm/exe.linux-x64@12.16.1':
resolution: {integrity: sha512-RK//8ZLcXGW8A2k8bQOXZDBN1Qu92zN4dYnG1bzj1zoQ1fkZ6jNzZ1AvXtsfGVma795CJtvoaF5bLXf4l/FIUw==}
'@pnpm/exe.linux-x64@12.18.0':
resolution: {integrity: sha512-OJ+X2wcraldCwj75pCIerWARQSRpdfj0eBTs/2ODyhfeW46ylOnjTIE/GKra6Mn6E72/gBmXS8igrUioa+F9fQ==}
cpu: [x64]
os: [linux]
libc: [glibc]
'@pnpm/exe.win32-arm64@12.16.1':
resolution: {integrity: sha512-Ox3kNNVxtGw6pK7SVejmeOUgHhUTZgAM7NQQ6cU7frPw3ix9+j7d2MGbRg3fyEFAaKWq8ZAZLTQGcDyLnAu+qA==}
'@pnpm/exe.win32-arm64@12.18.0':
resolution: {integrity: sha512-wEY6jiACGcbhhi5yD59doHqNTndS7AaujaPmtWAu1DKkHru2hy5fJ9yUGecW72Ypaw2/I3BEprfMOopap2NQtw==}
cpu: [arm64]
os: [win32]
'@pnpm/exe.win32-x64@12.16.1':
resolution: {integrity: sha512-k+FhRJ0ttphFoFIVB1Sk4v76YJAulafmA/Ly57cysesUhhIzVlApeOShUz0jXubKx317TThlEfpQrukldnqORA==}
'@pnpm/exe.win32-x64@12.18.0':
resolution: {integrity: sha512-5FJkghO7RLRkQpWxeVEygW9XsmnNuxzPfXbgwFWCHsFyFhpDn/vOdL88ToQEM0Kt5igdW5s6U0/IDH8Dvkt3pg==}
cpu: [x64]
os: [win32]
pnpm@12.16.1:
resolution: {integrity: sha512-CR0d7nknV1Zhsl2rPCrjHK/BxU1rce1qZgDPDByxBDSkR+rhQBXxsj4ua+3u1535SpMuIWTSgyKA3OP7MvZW0Q==}
pnpm@12.18.0:
resolution: {integrity: sha512-nXiC1dyCFH7PxOBiqjSDTgrQKWcQoYN6ERWuASEn02txCeMhrBWYod0KZVl2gbHpmRfhLtDXSerpEYeUM2O+sw==}
engines: {node: '>=18'}
hasBin: true
snapshots:
'@pnpm/exe.android-arm64@12.16.1':
'@pnpm/exe.android-arm64@12.18.0':
optional: true
'@pnpm/exe.android-x64@12.16.1':
'@pnpm/exe.android-x64@12.18.0':
optional: true
'@pnpm/exe.darwin-arm64@12.16.1':
'@pnpm/exe.darwin-arm64@12.18.0':
optional: true
'@pnpm/exe.darwin-x64@12.16.1':
'@pnpm/exe.darwin-x64@12.18.0':
optional: true
'@pnpm/exe.freebsd-x64@12.16.1':
'@pnpm/exe.freebsd-x64@12.18.0':
optional: true
'@pnpm/exe.linux-arm64-musl@12.16.1':
'@pnpm/exe.linux-arm64-musl@12.18.0':
optional: true
'@pnpm/exe.linux-arm64@12.16.1':
'@pnpm/exe.linux-arm64@12.18.0':
optional: true
'@pnpm/exe.linux-ppc64@12.16.1':
'@pnpm/exe.linux-ppc64@12.18.0':
optional: true
'@pnpm/exe.linux-riscv64@12.16.1':
'@pnpm/exe.linux-riscv64@12.18.0':
optional: true
'@pnpm/exe.linux-s390x@12.16.1':
'@pnpm/exe.linux-s390x@12.18.0':
optional: true
'@pnpm/exe.linux-x64-musl@12.16.1':
'@pnpm/exe.linux-x64-musl@12.18.0':
optional: true
'@pnpm/exe.linux-x64@12.16.1':
'@pnpm/exe.linux-x64@12.18.0':
optional: true
'@pnpm/exe.win32-arm64@12.16.1':
'@pnpm/exe.win32-arm64@12.18.0':
optional: true
'@pnpm/exe.win32-x64@12.16.1':
'@pnpm/exe.win32-x64@12.18.0':
optional: true
pnpm@12.16.1:
pnpm@12.18.0:
optionalDependencies:
'@pnpm/exe.android-arm64': 12.16.1
'@pnpm/exe.android-x64': 12.16.1
'@pnpm/exe.darwin-arm64': 12.16.1
'@pnpm/exe.darwin-x64': 12.16.1
'@pnpm/exe.freebsd-x64': 12.16.1
'@pnpm/exe.linux-arm64': 12.16.1
'@pnpm/exe.linux-arm64-musl': 12.16.1
'@pnpm/exe.linux-ppc64': 12.16.1
'@pnpm/exe.linux-riscv64': 12.16.1
'@pnpm/exe.linux-s390x': 12.16.1
'@pnpm/exe.linux-x64': 12.16.1
'@pnpm/exe.linux-x64-musl': 12.16.1
'@pnpm/exe.win32-arm64': 12.16.1
'@pnpm/exe.win32-x64': 12.16.1
'@pnpm/exe.android-arm64': 12.18.0
'@pnpm/exe.android-x64': 12.18.0
'@pnpm/exe.darwin-arm64': 12.18.0
'@pnpm/exe.darwin-x64': 12.18.0
'@pnpm/exe.freebsd-x64': 12.18.0
'@pnpm/exe.linux-arm64': 12.18.0
'@pnpm/exe.linux-arm64-musl': 12.18.0
'@pnpm/exe.linux-ppc64': 12.18.0
'@pnpm/exe.linux-riscv64': 12.18.0
'@pnpm/exe.linux-s390x': 12.18.0
'@pnpm/exe.linux-x64': 12.18.0
'@pnpm/exe.linux-x64-musl': 12.18.0
'@pnpm/exe.win32-arm64': 12.18.0
'@pnpm/exe.win32-x64': 12.18.0
---
lockfileVersion: '9.0'

1
backend/scripts/_env
View File

@ -62,6 +62,7 @@ export PENPOT_FLAGS="\
disable-tiered-file-data-storage \
enable-file-validation \
enable-file-schema-validation \
enable-debug-save-faults \
enable-redis-cache \
enable-link-preview \
enable-subscriptions";

14
backend/src/app/auth/oidc.clj
View File

@ -1115,11 +1115,15 @@
:else
(let [sxf (session/create-fn cfg profile info)
token (or (:invitation-token info)
(tokens/generate cfg
{:iss :auth
:exp (ct/in-future "15m")
:profile-id (:id profile)}))
;; The session token is always an :auth one; the
;; invitation token, when present, travels inside its
;; claims so the frontend goes through the regular
;; token verification path and then `logged-in`.
token (tokens/generate cfg
{:iss :auth
:exp (ct/in-future "15m")
:profile-id (:id profile)
:invitation-token (:invitation-token info)})
;; If proceed, update profile on the database
profile (update-profile-with-info cfg profile info)

15
backend/src/app/binfile/cleaner.clj
View File

@ -9,7 +9,7 @@
for recently imported shapes."
(:require
[app.common.data :as d]
[app.common.svg :as csvg]
[app.common.geom.matrix :as gmt]
[app.common.types.shape :as cts]
[app.common.uuid :as uuid]))
@ -106,14 +106,11 @@
:reverse-column :column-reverse
dir))))
(defn- fix-svg-attrs
"The json reader of the binfile rewrites every key of every entry to
kebab-case, but `:svg-attrs` keys are react prop names and are stored
in camelCase. `attrs->props` is the transform the svg import path
already applies to them, so running it again restores the names; it
is idempotent, so shapes that come in correct are left untouched."
(defn- fix-svg-transform
"The json reader decodes `:svg-transform` as a plain map; the svg defs
renderer multiplies it as a matrix, so turn it back into one."
[shape]
(d/update-when shape :svg-attrs csvg/attrs->props))
(d/update-when shape :svg-transform gmt/coerce))
(defn clean-shape-post-decode
"A shape procesor that expected to be executed after schema decoding
@ -123,7 +120,7 @@
(fix-shape-shadow-color)
(fix-root-shape)
(fix-legacy-flex-dir)
(fix-svg-attrs)))
(fix-svg-transform)))
(defn- fix-container
[container]

26
backend/src/app/binfile/common.clj
View File

@ -478,6 +478,18 @@
{:team-id team-id
:deleted-at nil}))
(def ^:private sql:get-files-rels
"SELECT flr.*,
fls.synced_at
FROM file_library_rel AS flr
JOIN file AS l
ON flr.library_file_id = l.id
LEFT JOIN file_library_sync AS fls
ON fls.file_id = flr.file_id
AND fls.library_file_id = flr.library_file_id
WHERE flr.file_id = ANY(?)
AND l.deleted_at IS NULL;")
(defn get-files-rels
"Given a set of file-id's, return all matching relations with the libraries"
[cfg ids]
@ -486,18 +498,8 @@
(assert (every? uuid? ids) "expected a set of uuids")
(db/run! cfg (fn [{:keys [::db/conn]}]
(let [ids (db/create-array conn "uuid" ids)
sql (str "SELECT flr.*,"
" fls.synced_at"
" FROM file_library_rel AS flr"
" JOIN file AS l"
" ON flr.library_file_id = l.id"
" LEFT JOIN file_library_sync AS fls"
" ON fls.file_id = flr.file_id"
" AND fls.library_file_id = flr.library_file_id"
" WHERE flr.file_id = ANY(?)"
" AND l.deleted_at IS NULL;")]
(db/exec! conn [sql ids])))))
(let [ids (db/create-array conn "uuid" ids)]
(db/exec! conn [sql:get-files-rels ids])))))
(def ^:private sql:upsert-file-library-sync
"INSERT INTO file_library_sync (file_id, library_file_id, synced_at)

17
backend/src/app/binfile/v3.clj
View File

@ -1039,15 +1039,17 @@
(defn- import-file-relations
[{:keys [::db/conn ::manifest ::bfc/timestamp] :as cfg}]
(let [relations (:relations manifest)]
(let [relations (:relations manifest)
file-ids (into #{} (map :id) (:files manifest))]
(doseq [[index [file-id libr-id]] (d/enumerate relations)]
(tap-progress cfg {:section :relations
:current (inc index) :total (count relations)})
(let [file-id (bfc/lookup-index file-id)
libr-id (bfc/lookup-index libr-id)]
(when (and file-id libr-id)
(if (and (not= file-id libr-id)
(contains? file-ids file-id)
(contains? file-ids libr-id))
(let [file-id (bfc/lookup-index file-id)
libr-id (bfc/lookup-index libr-id)]
(l/dbg :hint "create file library link"
:file-id (str file-id)
:lib-id (str libr-id)
@ -1055,7 +1057,10 @@
(let [rel-params {:file-id file-id
:library-file-id libr-id}]
(db/insert! conn :file-library-rel rel-params)
(bfc/upsert-file-library-sync! conn (assoc rel-params :synced-at timestamp))))))))
(bfc/upsert-file-library-sync! conn (assoc rel-params :synced-at timestamp))))
(l/warn :hint "ignoring invalid file library relation"
:file-id (str file-id)
:lib-id (str libr-id))))))
(defn- import-storage-objects
[{:keys [::bfc/input ::entries-index ::bfc/timestamp] :as cfg}]

20
backend/src/app/config.clj
View File

@ -111,8 +111,14 @@
:ssrf-extra-blocked-cidrs #{}})
(def schema:tenant
"Tenant identifier: hostname-label-style (letters, digits and
hyphens). It is interpolated into Redis keys separated by dots (msgbus
"Tenant identifier: hostname-label-style, lowercase (letters, digits
and hyphens). The value is normalized to lowercase when the
configuration is read, so an operator writing `PENPOT_TENANT=ACME` and
one writing `acme` name the same tenant: the value is compared as an
exact string in the keys and rows it lands in, and one uppercase
letter would otherwise split one tenant into two.
It is interpolated into Redis keys separated by dots (msgbus
topics, rate-limit buckets, exporter job keys, which are also matched
with a glob pattern), so `.` and whitespace are rejected at startup
because one tenant would then be able to read or overwrite another's
@ -125,7 +131,14 @@
tenant asking for it, and a tenant that has one would have to be
renamed. Tenants using those characters must be renamed before
upgrading."
[:re #"^[A-Za-z0-9-]+$"])
[:re #"^[a-z0-9-]+$"])
(defn- normalize-tenant
"Lowercase the configured tenant before it is validated, so
`PENPOT_TENANT=ACME` and `acme` mean the same tenant everywhere."
[config]
(cond-> config
(string? (:tenant config)) (update :tenant str/lower)))
(def schema:config
(do #_sm/optional-keys
@ -401,6 +414,7 @@
[& {:keys [prefix default] :or {prefix "penpot"}}]
(->> (read-env prefix)
(merge default)
(normalize-tenant)
(decode-config)))
(def version

134
backend/src/app/email.clj
View File

@ -305,7 +305,14 @@
second `params` provides the email data and optionally `::factory`
and `::reuse-conn`. When `::reuse-conn` is true the submission
reuses the caller's existing `::db/conn` (from cfg) instead of
acquiring a fresh one from the pool."
acquiring a fresh one from the pool.
NOTE: this renders the email before submitting, so the full subject
and body are stored as job params. Prefer `send-template` for
templated emails: it stores only the template id plus the raw data
and renders inside the worker. `send` is kept for raw (non
templated) emails and for draining jobs submitted in the old
format."
[cfg {:keys [::reuse-conn ::factory] :as params}]
(when (and reuse-conn (nil? (::db/conn cfg)))
(ex/raise :type :validation
@ -394,7 +401,7 @@
[:feedback-subject ::sm/text]
[:feedback-type ::sm/text]
[:feedback-content ::sm/text]
[:profile :map]])
[:profile [:map [:id ::sm/uuid]]]])
(def user-feedback
"A profile feedback email."
@ -468,7 +475,7 @@
(def ^:private schema:invite-to-organization
[:map
[:invited-by ::sm/text]
[:user-name [:maybe ::sm/text]]
[:user-name {:optional true} [:maybe ::sm/text]]
[:token ::sm/text]
[:organization schema:organization-data]])
@ -490,7 +497,7 @@
(def ^:private schema:renewal-notice
[:map
[:user-name [:maybe :string]]
[:user-name {:optional true} [:maybe :string]]
[:renewal-date ::sm/text]
[:estimated-amount ::sm/text]
[:organizations [:vector schema:organization-data]]])
@ -593,6 +600,125 @@
:id ::comment-notification
:schema schema:comment-notification))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; TEMPLATED SEND (render inside the worker)
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defn- template-entry
"Registry entry for a templated email: the factory that renders it and
the JSON encoder/decoder of its data schema. The job `params` column
is plain JSON, so values without a JSON form (e.g. the URIs of an
organization) are encoded to strings before submit and decoded back
to their types before the factory validates and renders them."
[factory schema]
{::factory factory
::encode (sm/encoder schema sm/json-transformer)
::decode (sm/decoder schema sm/json-transformer)})
;; Registry of template id to entry. This map is the single source of
;; truth for templated mail: `send-template` validates membership and
;; encodes the data here before submitting (so a job that could never
;; be rendered is rejected up front), and the worker decodes the data
;; and renders it with the entry factory. Callers pass the id string
;; itself (e.g. `"invite-to-team"`); keys stay `::eml/`-namespaced and
;; values stay plain data, following the params convention.
(def templates
{"feedback" (template-entry user-feedback schema:feedback)
"register" (template-entry register schema:register)
"password-recovery" (template-entry password-recovery schema:password-recovery)
"password-changed" (template-entry password-changed schema:password-changed)
"change-email" (template-entry change-email schema:change-email)
"invite-to-team" (template-entry invite-to-team schema:invite-to-team)
"invite-to-organization" (template-entry invite-to-organization schema:invite-to-organization)
"organization-setup-sso" (template-entry organization-setup-sso schema:organization-setup-sso)
"renewal-notice" (template-entry renewal-notice schema:renewal-notice)
"join-team" (template-entry join-team schema:join-team)
"request-file-access" (template-entry request-file-access schema:request-file-access)
"request-file-access-yourpenpot" (template-entry request-file-access-yourpenpot schema:request-file-access)
"request-file-access-yourpenpot-view" (template-entry request-file-access-yourpenpot-view schema:request-file-access)
"request-team-access" (template-entry request-team-access schema:request-team-access)
"comment-mention" (template-entry comment-mention schema:comment-mention)
"comment-thread" (template-entry comment-thread schema:comment-thread)
"comment-notification" (template-entry comment-notification schema:comment-notification)})
(defn render-template
"Render the templated email `template-id` from its JSON-decoded data,
as the worker does before sending it."
[template-id params]
(let [{:keys [::factory ::decode]} (get templates template-id)]
(factory (decode params))))
;; Envelope of the job params only: the template data travels encoded
;; by the template schema and is decoded by `render-template`.
(def ^:private schema:templated-params
[:map {:title "Templated Email Params"}
[:template (into [:enum] (sort (keys templates)))]
[:to [:or ::sm/email [::sm/vec ::sm/email]]]
[:reply-to {:optional true} ::sm/email]
[:from {:optional true} ::sm/email]
[:lang {:optional true} ::sm/text]
[:priority {:optional true} [:enum :high :low]]
[:extra-data {:optional true} ::sm/text]
[:attachments {:optional true}
[:map-of :string :string]]])
(defn send-template
"Schedule a templated email to be sent asynchronously using the
unified jobs machinery. Unlike `send`, this does NOT render the
email: it stores only the template id plus the raw template data as
`:send-templated-mail` job params, and the worker renders and sends
the email. The `::template` param is the template id string (one of
the keys of `templates`, e.g. `invite-to-team`);
`::reuse-conn` behaves like in `send`."
[cfg {:keys [::template ::reuse-conn] :as params}]
(when (and reuse-conn (nil? (::db/conn cfg)))
(ex/raise :type :validation
:code :missing-connection
:hint "reuse-conn requires the caller cfg to carry ::db/conn (call inside a transaction)"))
(when (nil? template)
(ex/raise :type :validation
:code :missing-template
:hint "template is mandatory for templated emails"))
(when-not (contains? templates template)
(ex/raise :type :validation
:code :unknown-email-template
:hint (str/ffmt "unknown email template: %", template)))
(let [encode (get-in templates [template ::encode])
recipe (-> params
(dissoc ::reuse-conn ::template)
;; Present-but-nil extras (e.g. `:file-id` when there is
;; no file) would fail the job schema, while the old
;; inline render simply ignored them.
(d/without-nils)
(encode))]
(jobs/submit (cond-> (dissoc cfg ::db/conn)
reuse-conn (assoc ::db/conn (::db/conn cfg)))
{::jobs/name :send-templated-mail
::jobs/delay 0
::jobs/max-retries 4
::jobs/priority 200
::jobs/params (cond-> (assoc recipe :template template)
;; `:public-uri` arrives as a URI record
;; from config, which plain JSON cannot
;; roundtrip; the templates only need its
;; string form.
(some? (:public-uri recipe))
(update :public-uri str))})))
(defmethod ig/init-key ::templated-job-def
[_ {sendmail ::sendmail}]
{::jobs/name :send-templated-mail
::jobs/schema schema:templated-params
::jobs/handler (fn [context params]
(let [template-id (:template params)]
(when-not (contains? templates template-id)
(ex/raise :type :internal
:code :unknown-email-template
:template-id template-id))
(sendmail context (render-template template-id (dissoc params :template)))))
::jobs/decoder (sm/decoder schema:templated-params sm/json-transformer)
::jobs/validator (sm/validator schema:templated-params)})
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; BOUNCE/COMPLAINS HELPERS
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

53
backend/src/app/http/errors.clj
View File

@ -17,6 +17,7 @@
[app.http.session :as-alias session]
[app.util.inet :as inet]
[clojure.spec.alpha :as s]
[cuerdas.core :as str]
[yetti.request :as yreq]
[yetti.response :as yres]))
@ -223,6 +224,25 @@
"25P03" "The transaction was idle too long and was cancelled"
"A database error occurred"))
(defn- transient-sql-state?
"True for SQLSTATE codes of failures that pass on their own: lost or
refused connections, lack of resources, a server shutting down, and
serialization failures or deadlocks of concurrent transactions."
[state]
(and (string? state)
(or (str/starts-with? state "08")
(str/starts-with? state "53")
(contains? #{"57P01" "57P02" "57P03" "40001" "40P01"} state))))
(defn- unavailable
"Response for a failure that passes on its own: the request can be sent
again."
[code hint]
{::yres/status 503
::yres/body {:type :server-error
:code code
:hint hint}})
(defmethod handle-exception org.postgresql.util.PSQLException
[error request parent-cause]
(let [state (.getSQLState ^java.sql.SQLException error)
@ -231,6 +251,9 @@
(l/error :hint "postgresql error"
:cause cause)
(cond
(transient-sql-state? state)
(unavailable :database-unavailable "The database is not available right now")
(= state "57014")
{::yres/status 504
::yres/body {:type :server-error
@ -271,13 +294,37 @@
(assoc :type :server-error)
(update :code #(or % :unhandled)))}))))
(defmethod handle-exception java.sql.SQLTransientException
[cause request _]
(binding [l/*context* (request->context request)]
(l/wrn :hint "transient database error" :cause cause)
(unavailable :database-unavailable "The database is not available right now")))
(defn- transient-io-exception?
"True for I/O failures that pass on their own: a lost, refused or timed
out connection, or a host name that did not resolve. A parse error or
data that cannot be read fails again on every attempt."
[cause]
(or (instance? java.net.SocketException cause)
(instance? java.io.InterruptedIOException cause)
(instance? java.net.UnknownHostException cause)
(instance? java.net.http.HttpTimeoutException cause)))
(defmethod handle-exception java.io.IOException
[cause request _]
(binding [l/*context* (request->context request)]
(l/wrn :hint "io exception" :cause cause)
{::yres/status 500
::yres/body {:type :server-error
:code :io-exception}}))
(if (transient-io-exception? cause)
(unavailable :io-exception "An input/output operation failed")
{::yres/status 500
::yres/body {:type :server-error
:code :io-exception}})))
(defmethod handle-exception InterruptedException
[cause request _]
(binding [l/*context* (request->context request)]
(l/wrn :hint "interrupted request" :cause cause)
(unavailable :interrupted "The request was interrupted")))
(defmethod handle-exception java.util.concurrent.CompletionException
[cause request _]

60
backend/src/app/loggers/audit.clj
View File

@ -48,6 +48,46 @@
"incomplete-user-info"
"unexpected-error"})
(def ^:private subscription-creation-events
#{"create-subscription" "create-trial-subscription"})
(def ^:private subscription-creation-telemetry-props
#{:type
:deployment
:quantity
:subscription-mode})
(def ^:private subscription-update-telemetry-props
#{:type
:deployment
:previous-status
:status
:previous-cancel-at-period-end
:cancel-at-period-end
:previous-cancel-at
:cancel-at
:changed-fields})
(def ^:private subscription-summary-telemetry-props
#{:type
:deployment
:instance-id
:snapshot-date
:trialing-count
:active-count
:active-cancel-scheduled-count
:past-due-count
:canceled-count
:unpaid-count
:incomplete-count
:incomplete-expired-count
:paused-count
:other-count
:total-count
:sync-succeeded
:sync-failed
:snapshot-complete})
(def ^:private safe-backend-context-keys
#{:version
:initiator
@ -348,6 +388,24 @@
(defn filter-telemetry-props
[{:keys [source name props type] :as params}]
(cond
(and (= type "action")
(contains? subscription-creation-events name)
(or (= source "backend")
(= source "frontend")))
(let [props' (into {} xf:filter-telemetry-props props)
props' (merge props' (select-keys props subscription-creation-telemetry-props))]
(assoc params :props props'))
(and (= source "backend")
(= type "action")
(= name "update-subscription"))
(assoc params :props (select-keys props subscription-update-telemetry-props))
(and (= source "backend")
(= type "action")
(= name "subscription-summary"))
(assoc params :props (select-keys props subscription-summary-telemetry-props))
(and (= source "backend")
(= name "organization-sso-auth-failed"))
(let [props' (into {} xf:filter-telemetry-props props)
@ -365,6 +423,8 @@
props' (-> props'
(assoc :lang (:lang props))
(assoc :auth-backend (:auth-backend props))
(assoc :source (:source props))
(assoc :activation (:activation props))
(assoc :email-domain (email/get-domain (:email props)))
(d/without-nils))]
(assoc params :props props'))

2
backend/src/app/loggers/audit/consumer_task.clj
View File

@ -231,6 +231,7 @@
"delete-file"
"delete-font"
"delete-organization"
"delete-organization-attribute"
"delete-organization-member"
"delete-project"
"delete-team"
@ -255,6 +256,7 @@
"success-activate-organization-sso"
"success-apply-organization-sso-changes"
"unassign-organization-attribute"
"update-organization-attribute"
"update-organization-invitation"
"update-organization-permissions"
"update-team-invitation"

41
backend/src/app/main.clj
View File

@ -656,24 +656,25 @@
::setup/shared-keys (ig/ref ::setup/shared-keys)}
::jobs/defs
{:sendmail (ig/ref ::email/job-def)
:delete-object (ig/ref :app.tasks.delete-object/job-def)
:demo-purge (ig/ref :app.tasks.demo-purge/job-def)
:run-webhook (ig/ref ::webhooks/run-webhook-job-def)
:process-webhook-event (ig/ref ::webhooks/process-webhook-event-job-def)
:file-gc (ig/ref :app.tasks.file-gc/job-def)
:offload-file-data (ig/ref :app.tasks.offload-file-data/job-def)
:objects-gc (ig/ref :app.tasks.objects-gc/job-def)
:export-binfile (ig/ref :app.tasks.export-binfile/job-def)
:import-binfile (ig/ref :app.tasks.import-binfile/job-def)
:storage-gc-deleted (ig/ref ::sto.gc-deleted/job-def)
:storage-gc-touched (ig/ref ::sto.gc-touched/job-def)
:storage-pending-gc (ig/ref ::sto.pending-gc/job-def)
:jobs-gc (ig/ref :app.jobs.gc/job-def)
:telemetry (ig/ref :app.tasks.telemetry/job-def)
:session-gc (ig/ref ::session/job-def)
:file-gc-scheduler (ig/ref :app.tasks.file-gc-scheduler/job-def)
:audit-log-consumer-nexus (ig/ref :app.loggers.audit.consumer-task/nexus-job-def)
{:sendmail (ig/ref ::email/job-def)
:send-templated-mail (ig/ref ::email/templated-job-def)
:delete-object (ig/ref :app.tasks.delete-object/job-def)
:demo-purge (ig/ref :app.tasks.demo-purge/job-def)
:run-webhook (ig/ref ::webhooks/run-webhook-job-def)
:process-webhook-event (ig/ref ::webhooks/process-webhook-event-job-def)
:file-gc (ig/ref :app.tasks.file-gc/job-def)
:offload-file-data (ig/ref :app.tasks.offload-file-data/job-def)
:objects-gc (ig/ref :app.tasks.objects-gc/job-def)
:export-binfile (ig/ref :app.tasks.export-binfile/job-def)
:import-binfile (ig/ref :app.tasks.import-binfile/job-def)
:storage-gc-deleted (ig/ref ::sto.gc-deleted/job-def)
:storage-gc-touched (ig/ref ::sto.gc-touched/job-def)
:storage-pending-gc (ig/ref ::sto.pending-gc/job-def)
:jobs-gc (ig/ref :app.jobs.gc/job-def)
:telemetry (ig/ref :app.tasks.telemetry/job-def)
:session-gc (ig/ref ::session/job-def)
:file-gc-scheduler (ig/ref :app.tasks.file-gc-scheduler/job-def)
:audit-log-consumer-nexus (ig/ref :app.loggers.audit.consumer-task/nexus-job-def)
:audit-log-consumer-admin-console (ig/ref :app.loggers.audit.consumer-task/admin-console-job-def)}
::jobs/request-pool
@ -684,6 +685,10 @@
{::email/sendmail (ig/ref ::email/sendmail)
::mtx/metrics (ig/ref ::mtx/metrics)}
:app.email/templated-job-def
{::email/sendmail (ig/ref ::email/sendmail)
::mtx/metrics (ig/ref ::mtx/metrics)}
:app.tasks.delete-object/job-def
{::db/pool (ig/ref ::db/pool)
::sto/storage (ig/ref ::sto/storage)

26
backend/src/app/media.clj
View File

@ -19,6 +19,7 @@
(:require
[app.common.data :as d]
[app.common.exceptions :as ex]
[app.common.media :as cm]
[app.config :as cf]
[app.db :as-alias db]
[app.http.client :as http]
@ -58,9 +59,22 @@
:code :unknown-size
:hint "seems like the url points to resource with unknown size"))
(-> {:size size :mtype mtype}
(validation/validate-media-type!)
(validation/validate-media-size!))))]
;; Fast fail on the declared header type and size; the
;; downloaded bytes are strictly verified after the write
;; (there is no file on disk yet to measure or sniff).
(when-not (contains? cm/image-types mtype)
(ex/raise :type :validation
:code :media-type-not-allowed
:hint "seems like you are uploading an invalid media object"))
(when (> size (cf/get :media-max-file-size))
(ex/raise :type :restriction
:code :media-max-file-size-reached
:hint (str/ffmt "the uploaded file size % is greater than the maximum %"
size
(cf/get :media-max-file-size))))
{:size size :mtype mtype}))]
(let [{:keys [body] :as response}
(try
@ -102,7 +116,11 @@
:hint "unexpected state: unable to write to file"))
;; Sanitize: strip trailing data after image EOF markers
(let [new-size (sanitize/truncate-after-eof path mtype)]
(let [new-size (sanitize/truncate-after-eof path mtype)
upload {:path path :mtype mtype :size new-size}]
;; Strict: the downloaded bytes must be what the remote
;; header claimed, same rule as direct uploads.
(validation/check-media-type upload)
{:path path
:mtype mtype
:size new-size})))

233
backend/src/app/media/svg.clj
View File

@ -12,8 +12,10 @@
[app.common.data.macros :as dm]
[app.common.exceptions :as ex]
[app.common.logging :as l]
[app.storage.tmp :as tmp]
[clojure.xml :as xml]
[cuerdas.core :as str])
[cuerdas.core :as str]
[datoteka.fs :as fs])
(:import
clojure.lang.XMLHandler
java.io.InputStream
@ -46,15 +48,85 @@
(xml/parse istream secure-parser-factory))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; SVG SERIALIZATION
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(def ^:private xml-declaration
"Declaration written at the top of every sanitized document."
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>")
(defn- escape-text
"Escape the characters that are special inside a XML text node."
[s]
(-> s
(str/replace "&" "&amp;")
(str/replace "<" "&lt;")
(str/replace ">" "&gt;")))
(defn- escape-attr
"Escape the characters that are special inside a XML attribute value.
The value is written between double quotes, so the quote itself is
escaped too.
Tabs and line breaks become spaces: that is what the XML parser does
with them on the next read, and it keeps the output stable."
[s]
(-> (str s)
(str/replace #"[\t\r\n]" " ")
(str/replace "&" "&amp;")
(str/replace "<" "&lt;")
(str/replace ">" "&gt;")
(str/replace "\"" "&quot;")
(str/replace "'" "&apos;")))
(defn- emit-element
"Serialize a parsed element, escaping every text and attribute value.
The tag and attribute names come from the parser, so they are valid
XML names; only their values can carry markup.
The attributes are sorted by name: the parser hands them over in the
reverse order of the document, so writing them as they come would
make the output change on every sanitization."
[{:keys [tag attrs content]}]
(let [attrs-str (apply str
(for [[k v] (sort-by (comp name key) attrs)]
(str " " (name k) "=\"" (escape-attr (str v)) "\"")))
content-str (apply str
(for [child content]
(if (map? child)
(emit-element child)
(escape-text child))))]
(if (seq content)
(str "<" (name tag) attrs-str ">" content-str "</" (name tag) ">")
(str "<" (name tag) attrs-str "/>"))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; SVG SANITIZATION
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(def ^:private dangerous-attrs-pattern #"(?i)^on\w+$")
(def ^:private javascript-href-pattern #"(?i)^javascript:")
(def ^:private url-ignored-chars
"Characters that the URL parser discards before it interprets the
scheme of an URL."
#"[\u0000-\u0020\u007f]")
(defn- javascript-url?
"True when the value is a javascript: URL.
The comparison drops the characters the URL parser drops (tab, CR, LF
and spaces), so a scheme split with one of them (`java&#x09;script:`)
cannot slip through."
[v]
(let [v (str/replace (str v) url-ignored-chars "")]
(boolean (re-find javascript-href-pattern (str/trim v)))))
(def ^:private dangerous-tags
#{"script" "foreignObject" "set"
"animate" "animateTransform" "animateColor" "animateMotion"})
"animate" "animateTransform" "animateColor" "animateMotion"
"iframe" "meta" "embed" "link" "base"})
(defn- local-name
"Return the local name of a parsed tag or attribute keyword.
@ -70,24 +142,82 @@
(last (str/split n ":"))
n)))
(def ^:private svg-namespace "http://www.w3.org/2000/svg")
(defn- namespace-bindings
"Return the namespace bindings in scope for an element: `::default`
plus the keyword prefixes declared in `attrs`, merged over the
inherited ones. The default binding is stored under a namespaced
keyword because a prefix cannot contain `/`, so no prefixed binding
can produce the same key.
The XML parser does not process namespaces, so the declarations are
plain attributes (`:xmlns`, `:xmlns:svg`) and the prefix is part of
the tag name (`:svg:svg`)."
[attrs inherited]
(let [own (reduce-kv (fn [acc k v]
(let [n (name k)]
(cond
(= n "xmlns") (assoc acc ::default v)
(str/starts-with? n "xmlns:") (assoc acc (keyword (subs n 6)) v)
:else acc)))
{}
attrs)]
(merge inherited own)))
(defn- element-namespace
"Return the namespace URI of `tag` under `bindings`: the default
namespace when the tag has no prefix, the prefixed binding otherwise.
The prefix is part of the tag name (`:svg:svg`), so the binding key
is the prefix."
[tag bindings]
(let [n (name tag)
idx (str/last-index-of n ":")]
(if (nil? idx)
(::default bindings)
(get bindings (keyword (subs n 0 idx))))))
(defn- svg-root?
"True when the element is an `svg` element in the SVG namespace.
The namespace is resolved with the same model as the recursion
(`namespace-bindings` + `element-namespace`), so the check does not
introduce a second rule."
[{:keys [tag attrs]}]
(let [bindings (namespace-bindings attrs {})]
(and (= "svg" (local-name tag))
(= svg-namespace (element-namespace tag bindings)))))
(defn- sanitize-svg-element
"Recursively sanitize an SVG element by removing dangerous tags and attributes."
[{:keys [tag attrs content] :as element}]
(when (and (map? element) tag)
(when-not (contains? dangerous-tags (local-name tag))
(let [clean-attrs (->> attrs
(remove (fn [[k v]]
(or (re-matches dangerous-attrs-pattern (local-name k))
(and (= "href" (local-name k))
(string? v)
(re-find javascript-href-pattern (str/trim v))))))
(into {}))
"Recursively sanitize an SVG element by removing dangerous tags and
attributes.
The namespace bindings are tracked through the recursion: an element
that is not in the SVG namespace is dropped with its whole subtree,
because the filter only knows about SVG elements and a child that
rebinds the namespace to XHTML can carry live HTML."
[element bindings]
(let [{:keys [tag attrs content]} element
bindings (namespace-bindings attrs bindings)]
(when (and (map? element) tag
(not (contains? dangerous-tags (local-name tag)))
(= svg-namespace (element-namespace tag bindings)))
(let [clean-attrs (->> attrs
(remove (fn [[k v]]
(let [n (name k)]
(or (and (not (str/starts-with? n "xmlns"))
(re-matches dangerous-attrs-pattern (local-name k)))
(and (= "href" (local-name k))
(string? v)
(javascript-url? v))))))
(into {}))
clean-content (when content
(->> content
(filter #(or (string? %) (map? %)))
(map (fn [child]
(if (map? child)
(sanitize-svg-element child)
(sanitize-svg-element child bindings)
child)))
(filter some?)
vec))]
@ -97,22 +227,65 @@
(defn sanitize-svg
"Sanitize SVG content by removing dangerous elements and attributes.
Removes <script> tags, <foreignObject> elements, event handlers (on*),
and javascript: URLs from href attributes."
and javascript: URLs from href attributes.
The document must be an SVG document: any element that is not in the
SVG namespace is dropped with its subtree, so a root that is not an
`svg` element in the SVG namespace is rejected, because the filter
only knows about SVG elements and attributes.
The filtered tree is written back with an escaping serializer: a
payload hidden in an attribute value or in a text node would become
live markup again if the values were written verbatim."
[svg-text]
(try
(let [parsed (parse-svg svg-text)
sanitized (sanitize-svg-element parsed)]
(if sanitized
(with-out-str (xml/emit sanitized))
(ex/raise :type :validation
:code :invalid-svg-file
:hint "SVG sanitization produced no output")))
(catch Exception e
(l/warn :hint "SVG sanitization failed, rejecting upload" :cause e)
(let [parsed (try
(parse-svg svg-text)
(catch Exception e
(l/warn :hint "SVG parsing failed, rejecting upload" :cause e)
(ex/raise :type :validation
:code :invalid-svg-file
:hint "SVG parsing failed during sanitization"
:cause e)))
_ (when-not (svg-root? parsed)
(l/warn :hint "SVG sanitization rejected a document that is not an svg document")
(ex/raise :type :validation
:code :invalid-svg-file
:hint "the uploaded document is not an svg document"))
sanitized (sanitize-svg-element parsed {})]
(when-not sanitized
(l/warn :hint "SVG sanitization produced no output, rejecting upload")
(ex/raise :type :validation
:code :invalid-svg-file
:hint "SVG parsing failed during sanitization"
:cause e))))
:hint "SVG sanitization produced no output"))
(str xml-declaration (emit-element sanitized))))
(defn sanitize-svg-upload
"Return the upload untouched when its mtype is not `image/svg+xml`.
When it is, write a sanitized copy to a temporary file and return the
upload pointing to it, with the size of the sanitized content.
The caller must check the size of the upload before calling this: the
whole file is read into memory to sanitize it. The size limit applies
to the upload, not to the sanitized copy, which can be up to six times
bigger (a single quote becomes `&apos;`).
Every path that stores an uploaded SVG as its final object must go
through this function, or the stored file can carry live markup. Two
paths store raw bytes on purpose and do not: `upload-tempfile` (a
channel between processes, not publicly accessible) and
`upload-chunk` (the chunks of an upload; the assembled object goes
through this function). The profile picture is not an SVG either:
`import-profile-picture` ignores an `image/svg+xml` download."
[{:keys [mtype path] :as upload}]
(if (= "image/svg+xml" mtype)
(let [content (slurp (str path) :encoding "UTF-8")
clean (sanitize-svg content)
tmp-path (tmp/tempfile :prefix "penpot-svg-" :suffix ".svg" :min-age "1h")]
(spit (str tmp-path) clean :encoding "UTF-8")
(assoc upload :path tmp-path :size (fs/size tmp-path)))
upload))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; SVG INFO EXTRACTION
@ -120,7 +293,11 @@
(defn get-basic-info-from-svg
[{:keys [tag attrs] :as data}]
(when (not= tag :svg)
;; The local name is enough here: this function only reads the
;; dimensions. Whether the document is a valid SVG is decided by
;; `sanitize-svg`, which rejects anything that is not an `svg` element
;; in the SVG namespace.
(when (not= "svg" (local-name tag))
(ex/raise :type :validation
:code :unable-to-parse-svg
:hint "uploaded svg has invalid content"))

292
backend/src/app/media/validation.clj
View File

@ -13,7 +13,11 @@
[app.common.schema :as sm]
[app.config :as cf]
[cuerdas.core :as str]
[datoteka.fs :as fs]))
[datoteka.fs :as fs])
(:import
com.j256.simplemagic.ContentInfo
com.j256.simplemagic.ContentInfoUtil
java.nio.file.Path))
(def schema:upload
[:map {:title "Upload"}
@ -32,37 +36,289 @@
(def check-input
(sm/check-fn schema:input))
(defn validate-media-type!
([upload] (validate-media-type! upload cm/image-types))
(def allowed-upload-prefixes
;; Upload temp files are created server-side by the multipart parser
;; (yetti `default-temp-dir`, `/tmp/undertow/`) plus the tmp dir used
;; by tests and storage tmp files (`/tmp/penpot/`). Anything else
;; (e.g. `/etc/passwd` smuggled via Transit `~#path`) is rejected.
["/tmp/undertow/" "/tmp/penpot/"])
(defn check-upload-path
[upload]
(let [real (try (str (fs/real (:path upload)))
(catch Exception _ nil))]
(when (or (nil? real)
(not (some #(str/starts-with? real %) allowed-upload-prefixes)))
(ex/raise :type :validation
:code :params-validation
:hint "invalid upload path"))
upload))
(def ^:private ^ContentInfoUtil magic-util
"Shared magic-byte detector (thread-safe for matching, read-only
after construction). Built once: parsing the embedded magic database
on every upload would be wasteful."
(ContentInfoUtil.))
(defn- normalize-mtype
"Lowercase a mime string and drop any `;` parameters. Returns nil
for missing or blank values."
[mtype]
(when (string? mtype)
(let [clean (-> (str/split mtype #";" 2)
(first)
(str/trim)
(str/lower))]
(when-not (str/empty? clean)
clean))))
(def ^:private simplemagic-aliases
"Spellings reported by `ContentInfoUtil` that differ from the Penpot
vocabulary in `app.common.media`. Single place to extend when the
magic database gains new spellings."
{"image/x-png" "image/png"
"image/x-jpeg" "image/jpeg"
"image/jpg" "image/jpeg"
"application/x-font-ttf" "font/ttf"
"application/x-font-otf" "font/otf"
"application/font-sfnt" "font/ttf"
"application/x-font-woff" "font/woff"
"application/font-woff" "font/woff"
"application/x-font-woff2" "font/woff2"
"application/font-woff2" "font/woff2"})
(def ^:private magic-name->mtype
"The magic database knows TrueType and OpenType by name but reports
an empty mime for them. Map those names to the Penpot vocabulary."
{"TrueType" "font/ttf"
"OpenType" "font/otf"})
(defn- read-four-cc
"Reads the FourCC signature of the file at `path` — its first four
bytes as an ASCII string. Used only for the font containers the
magic database does not know (WOFF/WOFF2). Returns nil on any read
failure."
[^Path path]
(try
(with-open [in (java.io.FileInputStream. (str path))]
(let [buf (byte-array 4)
n (.read in buf)]
(when (= n 4)
(String. ^bytes buf "US-ASCII"))))
(catch Exception _ nil)))
(def ^:private four-cc->mtype
{"wOFF" "font/woff"
"wOF2" "font/woff2"})
(def ^:private svg-probe-bytes
"How many leading bytes the SVG root probe reads. The root element
of a real document sits at the top; 64KB leaves room for a long
prolog or comments while keeping memory constant."
65536)
(defn- starts-at?
"True when `prefix` occurs in `s` exactly at `i`."
[^String s ^long i ^String prefix]
(let [plen (.length prefix)]
(and (<= (+ i plen) (.length s))
(.regionMatches s (int i) prefix (int 0) (int plen)))))
(defn- skip-ws
"Index of the first non-whitespace char in `s` at or after `i`."
[^String s ^long i]
(let [n (.length s)]
(loop [i i]
(if (and (< i n) (Character/isWhitespace (.charAt s i)))
(recur (inc i))
i))))
(defn- index-of-from
"Index of the first occurrence of `end` in `s` at or after `i`, or
nil when absent."
[^String s ^long i ^String end]
(loop [i i]
(cond
(> i (- (.length s) (.length end))) nil
(starts-at? s i end) i
:else (recur (inc i)))))
(defn- skip-markup-decl
"Index just past a `<!...>` declaration opening at `i` (which points
at `<`), honouring `[...]` nesting so a doctype internal subset
cannot fool it. Returns nil when unterminated."
[^String s ^long i]
(let [n (.length s)]
(loop [i (+ i 2) depth 0]
(when (< i n)
(let [c (.charAt s i)]
(cond
(= c \[) (recur (inc i) (inc depth))
(= c \]) (recur (inc i) (dec depth))
(and (= c \>) (zero? depth)) (inc i)
:else (recur (inc i) depth)))))))
(defn- svg-root-at?
"True when the first markup in `s` is a lowercase `<svg` root
element, skipping the XML prolog, comments and doctype. Anything
else — including a truncated head — is false; full validity stays
downstream with the SVG parser."
[^String s]
(loop [i (skip-ws s 0)]
(when (< i (.length s))
(cond
(starts-at? s i "<?")
(when-some [j (index-of-from s (+ i 2) "?>")]
(recur (skip-ws s (+ j 2))))
(starts-at? s i "<!--")
(when-some [j (index-of-from s (+ i 4) "-->")]
(recur (skip-ws s (+ j 3))))
(starts-at? s i "<!")
(when-some [j (skip-markup-decl s i)]
(recur (skip-ws s j)))
(starts-at? s i "<svg")
(let [j (+ i 4)]
(boolean
(when (< j (.length s))
(contains? #{\space \tab \newline \return \/ \>} (.charAt s j)))))
:else false))))
(defn- read-head
"Read at most `svg-probe-bytes` leading bytes of the file at `path`,
decoded as UTF-8 like the rest of the pipeline. Malformed sequences
become replacement chars and can never match a tag. A leading U+FEFF
(UTF-8 BOM) is stripped: it is an encoding artifact, not content.
Returns nil on any read failure."
[^Path path]
(try
(with-open [in (java.io.FileInputStream. (str path))]
(let [buf (byte-array svg-probe-bytes)
n (.read in buf)]
(when (pos? n)
(let [s (String. ^bytes buf (int 0) (int n) "UTF-8")]
(if (and (pos? (.length s)) (= \uFEFF (.charAt s 0)))
(subs s 1)
s)))))
(catch Exception _ nil)))
(defn- svg-file?
"True when the file at `path` opens with an `<svg` root element.
This is the arbiter for SVG when magic returns nothing or a generic
XML type. Reads at most `svg-probe-bytes` leading bytes, so memory
stays constant no matter the file size. Full validity stays
downstream (`parse-svg` + sanitize): this only tells SVG-looking
text apart from other magic-less files."
[^Path path]
(boolean
(when (and (some? path) (fs/exists? path))
(some-> (read-head path) (svg-root-at?)))))
(defn detect-mtype
"Sniff the real mime type of the file at `path` using magic bytes.
Returns a normalized Penpot mime string, or nil when the file cannot
be read or nothing matches."
[^Path path]
(try
(when (and (some? path) (fs/exists? path))
(let [^ContentInfo info (.findMatch magic-util (str path))]
(if (some? info)
(let [raw (normalize-mtype (.getMimeType info))]
(if (some? raw)
(get simplemagic-aliases raw raw)
(get magic-name->mtype (.getName info))))
(get four-cc->mtype (read-four-cc path)))))
(catch Exception _ nil)))
(defn- svg-declared?
[declared]
(= declared "image/svg+xml"))
(defn- sniff-match?
"Strict comparison of the declared client mime against the sniffed
file content. SVG is the only exception: magic may return
`image/svg+xml` (an SVG with an XML prolog) or a generic XML type;
when sniffed is nil or a generic XML type, a declared `image/svg+xml`
is accepted only when the root-tag probe confirms an `<svg` root.
Everything else must be exactly equal — including
`application/octet-stream`, which never matches."
[declared sniffed path]
(and (some? declared)
(or (= declared sniffed)
(and (svg-declared? declared)
(or (nil? sniffed)
(contains? #{"application/xml" "text/xml"} sniffed))
(svg-file? path)))))
(defn check-media-type
([upload] (check-media-type upload cm/image-types))
([upload allowed]
(when-not (contains? allowed (:mtype upload))
(ex/raise :type :validation
:code :media-type-not-allowed
:hint "Seems like you are uploading an invalid media object"))
(let [declared (normalize-mtype (:mtype upload))
path (:path upload)
sniffed (detect-mtype path)]
;; Declared type outside the allowlist is rejected on its face,
;; with the established code.
(when-not (contains? allowed declared)
(ex/raise :type :validation
:code :media-type-not-allowed
:hint "seems like you are uploading an invalid media object"))
;; Declared type allowed but file bytes differ: the client lied.
(when-not (sniff-match? declared sniffed path)
(ex/raise :type :validation
:code :media-type-mismatch
:hint (str/ffmt "the uploaded file content (%) does not match the declared type (%)"
(or sniffed "unknown")
(or (:mtype upload) "unknown")))))
upload))
(defn validate-media-size!
(defn- real-size
"Bytes on disk for the file at `path`. Returns nil when the file
cannot be read — callers run `check-upload-path` first, so a nil
here fails closed with the path code instead of a limit code."
[path]
(try
(when (some? path)
(fs/size path))
(catch Exception _ nil)))
(defn check-media-size
"Rejects when the file bytes on disk exceed the configured
`:media-max-file-size` limit. The declared `:size` is ignored —
only the file at `:path` counts."
[upload]
(let [max-size (cf/get :media-max-file-size)]
(when (> (:size upload) max-size)
(let [max-size (cf/get :media-max-file-size)
disk-size (real-size (:path upload))]
(when (nil? disk-size)
(ex/raise :type :validation
:code :params-validation
:hint "invalid upload path"))
(when (> disk-size max-size)
(ex/raise :type :restriction
:code :media-max-file-size-reached
:hint (str/ffmt "the uploaded file size % is greater than the maximum %"
(:size upload)
disk-size
max-size)))
upload))
(defn validate-font-size!
"Validates that the font file `upload` does not exceed the configured
`:font-max-file-size` limit. Accepts the same map shape as
`validate-media-size!` — requires a `:size` key in bytes."
(defn check-font-size
"Rejects when the font file bytes on disk exceed the configured
`:font-max-file-size` limit. The declared `:size` is ignored —
only the file at `:path` counts."
[upload]
(let [max-size (cf/get :font-max-file-size)]
(when (> (:size upload) max-size)
(let [max-size (cf/get :font-max-file-size)
disk-size (real-size (:path upload))]
(when (nil? disk-size)
(ex/raise :type :validation
:code :params-validation
:hint "invalid upload path"))
(when (> disk-size max-size)
(ex/raise :type :restriction
:code :font-max-file-size-reached
:hint (str/ffmt "the uploaded font size % is greater than the maximum %"
(:size upload)
disk-size
max-size)))
upload))

8
backend/src/app/migrations.clj
View File

@ -532,7 +532,13 @@
:fn (mg/resource "app/migrations/sql/0156-reset-pg-storage-defaults.sql")}
{:name "0157-add-audit-log-pending-table"
:fn (mg/resource "app/migrations/sql/0157-add-audit-log-pending-table.sql")}])
:fn (mg/resource "app/migrations/sql/0157-add-audit-log-pending-table.sql")}
{:name "0157-migrate-legacy-file-data"
:fn (mg/resource "app/migrations/sql/0157-migrate-legacy-file-data.sql")}
{:name "0158-del-forbidden-file-library-rels"
:fn (mg/resource "app/migrations/sql/0158-del-forbidden-file-library-rels.sql")}])
(defn apply-migrations!
[pool name migrations]

55
backend/src/app/migrations/sql/0157-migrate-legacy-file-data.sql Normal file
View File

@ -0,0 +1,55 @@
-- Migrate legacy payloads to file_data: labelled file_change.data as
-- snapshots, file.data as mains (backend 'db', deleted_at copied so the
-- GC keeps collecting them), then delete orphan file_change rows with
-- data but no label (no read path reaches them).
--
-- Idempotent: INSERTs ignore conflicts, UPDATEs only null the legacy
-- column where the file_data row exists. Big instances pre-migrate by
-- hand (operations/2.19.0-migrate-legacy-file-data.md) and fake this
-- step instead:
--
-- INSERT INTO migrations (module, step)
-- VALUES ('main', '0157-migrate-legacy-file-data');
INSERT INTO file_data (file_id, id, created_at, modified_at, deleted_at,
type, backend, metadata, data)
SELECT file_id, id, created_at, updated_at, deleted_at,
'snapshot', 'db', NULL, data
FROM file_change
WHERE data IS NOT NULL
AND label IS NOT NULL
ON CONFLICT (file_id, id) DO NOTHING;
UPDATE file_change AS fc
SET data = NULL
WHERE fc.data IS NOT NULL
AND fc.label IS NOT NULL
AND EXISTS (SELECT 1
FROM file_data AS fd
WHERE fd.file_id = fc.file_id
AND fd.id = fc.id
AND fd.type = 'snapshot');
INSERT INTO file_data (file_id, id, created_at, modified_at, deleted_at,
type, backend, metadata, data)
SELECT id, id, created_at, modified_at, deleted_at,
'main', 'db', NULL, data
FROM file
WHERE data IS NOT NULL
ON CONFLICT (file_id, id) DO NOTHING;
UPDATE file AS f
SET data = NULL
WHERE f.data IS NOT NULL
AND EXISTS (SELECT 1
FROM file_data AS fd
WHERE fd.file_id = f.id
AND fd.id = f.id
AND fd.type = 'main');
-- Orphan revisions: legacy payloads without label. No file_data row
-- corresponds to them (snapshots are told apart by label), so there is
-- nothing to copy; the row itself is garbage. Delete it.
DELETE FROM file_change
WHERE data IS NOT NULL
AND label IS NULL;

38
backend/src/app/migrations/sql/0158-del-forbidden-file-library-rels.sql Normal file
View File

@ -0,0 +1,38 @@
-- Remove file-library relations that violate the same-team invariant:
-- relations across teams and self-relations. Leftover from before the
-- binfile export, the view-only bundle, link-file-to-library and the
-- manifest writers enforced same-team libraries. Relations with both
-- endpoints in the same team are kept.
--
-- The sync rows go first: the second statement reads file_library_rel
-- while it still contains the forbidden rows.
WITH forbidden AS (
SELECT flr.file_id, flr.library_file_id
FROM file_library_rel flr
JOIN file f ON f.id = flr.file_id
JOIN project fp ON fp.id = f.project_id
JOIN file l ON l.id = flr.library_file_id
JOIN project lp ON lp.id = l.project_id
WHERE fp.team_id <> lp.team_id
OR flr.file_id = flr.library_file_id
)
DELETE FROM file_library_sync fls
USING forbidden
WHERE fls.file_id = forbidden.file_id
AND fls.library_file_id = forbidden.library_file_id;
WITH forbidden AS (
SELECT flr.file_id, flr.library_file_id
FROM file_library_rel flr
JOIN file f ON f.id = flr.file_id
JOIN project fp ON fp.id = f.project_id
JOIN file l ON l.id = flr.library_file_id
JOIN project lp ON lp.id = l.project_id
WHERE fp.team_id <> lp.team_id
OR flr.file_id = flr.library_file_id
)
DELETE FROM file_library_rel flr
USING forbidden
WHERE flr.file_id = forbidden.file_id
AND flr.library_file_id = forbidden.library_file_id;

32
backend/src/app/nitrate.clj
View File

@ -444,7 +444,8 @@
[:map
[:organization-id ::sm/uuid]
[:owner-id ::sm/uuid]
[:permissions [:map-of :keyword :string]]]
[:permissions [:map-of :keyword :string]]
[:restrictions-met {:optional true} [:maybe [:map-of :keyword :boolean]]]]
params))
(defn- get-organization-sso-api
@ -652,6 +653,35 @@
:cause cause)
team)))))
(defn add-organization-info-to-team-strict
"Like `add-organization-info-to-team`, but fails closed for authorization
checks: the team is returned unchanged only when Nitrate answers 404 (the
team belongs to no organization). Any other failure, including a 4xx or an
invalid response body, raises instead of looking like a team without an
organization."
[cfg team params]
(let [params (assoc (or params {})
:team-id (:id team)
:throw-on-error? true)
team-with-organization
(try
(call cfg :get-team-organization params)
(catch Throwable cause
(let [{:keys [type status]} (ex-data cause)]
(if (and (= :nitrate-http-error type) (= 404 status))
::no-organization
(throw cause)))))]
(cond
(= ::no-organization team-with-organization)
team
(nil? team-with-organization)
(ex/raise :type :nitrate-unavailable
:hint "nitrate did not return a valid team organization response")
:else
(apply-organization-info-to-team team team-with-organization))))
(defn add-organization-info-to-teams
"Enriches teams with organization information using one batched Nitrate request.
Teams absent from the Nitrate response are returned unchanged.

27
backend/src/app/rpc.clj
View File

@ -366,6 +366,28 @@
(wrap-params-validation cfg $ mdata)
(wrap-authentication cfg $ mdata)))
(defn- wrap-shared-key-authorization
"Authorizes a shared-key caller against the method's `::allowed-shared-keys`.
A management method declares the shared-key ids allowed to call it with
`::allowed-shared-keys`. When the metadata is absent any configured shared key is
accepted, preserving the previous behavior. When it is present and empty no
key is accepted."
[_ f mdata]
(if (contains? mdata ::allowed-shared-keys)
(let [allowed-keys (::allowed-shared-keys mdata)]
(fn [cfg params]
(if (contains? allowed-keys (::auth-key-id params))
(f cfg params)
(do
(l/wrn :hint "shared key is not allowed for this method"
:handler-name (::handler-name params)
:key-id (::auth-key-id params))
(ex/raise :type :authorization
:code :shared-key-not-allowed
:hint "shared key is not allowed for this method")))))
f))
(defn- wrap-management
[cfg f mdata]
(as-> f $
@ -376,7 +398,10 @@
(wrap-audit cfg $ mdata)
(wrap-nitrate-sso cfg $ mdata)
(wrap-params-validation cfg $ mdata)
(wrap-authentication cfg $ mdata)))
(wrap-authentication cfg $ mdata)
;; Outermost on purpose: a denied key never opens a transaction,
;; validates params or runs the method body.
(wrap-shared-key-authorization cfg $ mdata)))
(defn- process-method
[cfg wrap-fn [f mdata]]

88
backend/src/app/rpc/commands/auth.clj
View File

@ -202,11 +202,11 @@
(update-password conn))]
(when profile
(login-lockout/clear-attempts! cfg (:id profile))
(eml/send cfg {::eml/reuse-conn true
::eml/factory eml/password-changed
:public-uri (cf/get :public-uri)
:to (:email profile)
:name (:fullname profile)})))
(eml/send-template cfg {::eml/reuse-conn true
::eml/template "password-changed"
:public-uri (cf/get :public-uri)
:to (:email profile)
:name (:fullname profile)})))
nil))
@ -373,28 +373,40 @@
[cfg uri]
(try
(let [storage (sto/resolve cfg)
input (media/download-image cfg uri)
input (media/run cfg {:cmd :info :input input})
hash (sto/calculate-hash (:path input))
content (-> (sto/content (:path input) (:size input))
(sto/wrap-with-hash hash))
sobject (sto/put-object! storage {::sto/content content
::sto/deduplicate? true
:bucket "profile"
:content-type (:mtype input)})]
(:id sobject))
input (media/download-image cfg uri)]
(if (= "image/svg+xml" (:mtype input))
;; The avatar is always rendered as an image, so an SVG would only
;; add the risk of storing live markup: it is not allowed here.
(do
(l/wrn :hint "ignoring svg profile picture" :uri uri)
nil)
(let [input (media/run cfg {:cmd :info :input input})
hash (sto/calculate-hash (:path input))
content (-> (sto/content (:path input) (:size input))
(sto/wrap-with-hash hash))
sobject (sto/put-object! storage {::sto/content content
::sto/deduplicate? true
:bucket "profile"
:content-type (:mtype input)})]
(:id sobject))))
(catch Throwable cause
(l/wrn :hint "unable to import profile picture"
:uri uri
:cause cause)
nil)))
(def ^:private default-renderer-new-profiles
"Renderer stored on new profiles. Profiles without one fall back to SVG."
:wasm)
(defn create-profile
"Create the profile entry on the database with limited set of input
attrs (all the other attrs are filled with default values)."
[{:keys [::db/conn] :as cfg} {:keys [email] :as params}]
(let [id (or (:id params) (uuid/next))
props (-> (:props params)
(update :renderer d/nilv default-renderer-new-profiles)
(merge {:viewed-tutorial? false
:viewed-walkthrough? false
:nudge {:big 10 :small 1}
@ -485,13 +497,13 @@
{:iss :profile-identity
:profile-id (:id profile)
:exp (ct/in-future {:days 30})})]
(eml/send cfg {::eml/reuse-conn true
::eml/factory eml/register
:public-uri (cf/get :public-uri)
:to (:email profile)
:name (:fullname profile)
:token vtoken
:extra-data ptoken}))))
(eml/send-template cfg {::eml/reuse-conn true
::eml/template "register"
:public-uri (cf/get :public-uri)
:to (:email profile)
:name (:fullname profile)
:token vtoken
:extra-data ptoken}))))
(defn register-profile
[{:keys [::db/conn] :as cfg} {:keys [token] :as params}]
@ -516,8 +528,9 @@
created? (-> profile meta :created true?)
props (-> (audit/profile->props profile)
(assoc :from-invitation (some? invitation)))]
props (cond-> (audit/profile->props profile)
:always (assoc :from-invitation (some? invitation))
created? (assoc :renderer-default (-> profile :props :renderer)))]
(cond
;; When profile is blocked, we just ignore it and return plain data
@ -543,8 +556,8 @@
;; return :invitation-token. The frontend redirects to
;; :auth-verify-token, which immediately accepts the
;; invitation.
;; - active + no/mismatched invitation → mint session
;; ("login" action). New profile, no further action.
;; - active + no/mismatched invitation → mint session.
;; New profile, no further action.
;; - not-active + matching invitation → send the
;; verify-email mail with the invitation token EMBEDDED
;; into the verify-email JWE. No session yet. When the
@ -557,7 +570,11 @@
created?
(let [accept-invitation? (and (some? invitation)
(= (:email profile)
(:member-email invitation)))]
(:member-email invitation)))
props (assoc props :activation
(if (:is-active profile)
"immediate"
"email-verification"))]
(cond
(and (:is-active profile) accept-invitation?)
(let [invitation (assoc invitation :member-id (:id profile))
@ -567,7 +584,6 @@
:invitation-token token}
(rph/with-transform (session/create-fn cfg profile claims))
(rph/with-meta {::audit/replace-props props
::audit/context {:action "accept-invitation"}
::audit/profile-id (:id profile)})))
(:is-active profile)
@ -575,7 +591,6 @@
(rph/with-transform (session/create-fn cfg profile claims))
(rph/with-meta
{::audit/replace-props props
::audit/context {:action "login"}
::audit/profile-id (:id profile)}))
:else
@ -589,7 +604,6 @@
:email (:email profile)}
(rph/with-meta
{::audit/replace-props props
::audit/context {:action "email-verification"}
::audit/profile-id (:id profile)})))))
;; When email verification is disabled and an inactive profile already
@ -654,13 +668,13 @@
{:iss :profile-identity
:profile-id (:id profile)
:exp (ct/in-future {:days 30})})]
(eml/send cfg {::eml/reuse-conn true
::eml/factory eml/password-recovery
:public-uri (cf/get :public-uri)
:to (:email profile)
:token (:token profile)
:name (:fullname profile)
:extra-data ptoken})
(eml/send-template cfg {::eml/reuse-conn true
::eml/template "password-recovery"
:public-uri (cf/get :public-uri)
:to (:email profile)
:token (:token profile)
:name (:fullname profile)
:extra-data ptoken})
nil))]
(let [profile (->> (profile/clean-email email)

2
backend/src/app/rpc/commands/binfile.clj
View File

@ -194,6 +194,8 @@
(assoc params :file file))
params)
_ (media.v/check-upload-path (:file params))
version (or version
(case (bfc/parse-file-format (-> params :file :path))
:binfile-v1 1

336
backend/src/app/rpc/commands/comments.clj
View File

@ -40,6 +40,23 @@
(def comment-max-length 750)
(defn- share-link-pages
"The set of pages granted by a share-link, or nil when the access to the
file is not restricted to a page scope."
[perms]
(when (= :share-link (:type perms))
(or (:pages perms) #{})))
(defn- check-page-scope
"Raises a :not-found error when `page-id` is outside the page scope granted
by a share-link."
[perms page-id]
(when-let [pages (share-link-pages perms)]
(when-not (contains? pages page-id)
(ex/raise :type :not-found
:code :object-not-found
:hint "not found"))))
(defn- format-comment
[{:keys [content]}]
(->> (d/interleave-all
@ -112,10 +129,10 @@
(doseq [mention comment-mentions]
(let [{:keys [fullname email props]} (get team-users mention)]
(when (mention-email? props)
(eml/send
(eml/send-template
cfg
{::eml/reuse-conn true
::eml/factory eml/comment-mention
::eml/template "comment-mention"
:public-uri (cf/get :public-uri)
:to email
:name fullname
@ -128,10 +145,10 @@
(doseq [mention thread-mentions]
(let [{:keys [fullname email props]} (get team-users mention)]
(when (mention-email? props)
(eml/send
(eml/send-template
cfg
{::eml/reuse-conn true
::eml/factory eml/comment-thread
::eml/template "comment-thread"
:public-uri (cf/get :public-uri)
:to email
:name fullname
@ -144,10 +161,10 @@
(doseq [user-id notificate-users-ids]
(let [{:keys [id fullname email props]} (get team-users user-id)]
(when (notification-email? id (:owner-id thread) props)
(eml/send
(eml/send-template
cfg
{::eml/reuse-conn true
::eml/factory eml/comment-notification
::eml/template "comment-notification"
:public-uri (cf/get :public-uri)
:to email
:name fullname
@ -235,8 +252,8 @@
(db/run! cfg (fn [{:keys [::db/conn] :as cfg}]
(let [perms (files/check-comment-permissions! cfg profile-id file-id share-id)
threads (get-comment-threads conn profile-id file-id)]
(if (= :share-link (:type perms))
(filterv #(contains? (:pages perms) (:page-id %)) threads)
(if-let [pages (share-link-pages perms)]
(filterv #(contains? pages (:page-id %)) threads)
threads)))))
(defn- get-comment-threads-sql
@ -337,11 +354,8 @@
(let [perms (files/check-comment-permissions! cfg profile-id file-id share-id)
thread (some-> (db/exec-one! conn [sql:get-comment-thread profile-id file-id id])
(decode-row))]
(when (and thread (= :share-link (:type perms)))
(when-not (contains? (:pages perms) (:page-id thread))
(ex/raise :type :not-found
:code :object-not-found
:hint "not found")))
(when thread
(check-page-scope perms (:page-id thread)))
thread))))
;; --- COMMAND: Retrieve Comments
@ -361,11 +375,7 @@
(db/run! cfg (fn [{:keys [::db/conn] :as cfg}]
(let [{:keys [file-id page-id]} (get-comment-thread conn thread-id)
perms (files/check-comment-permissions! cfg profile-id file-id share-id)]
(when (and (= :share-link (:type perms))
(not (contains? (:pages perms) page-id)))
(ex/raise :type :not-found
:code :object-not-found
:hint "not found"))
(check-page-scope perms page-id)
(get-comments conn thread-id)))))
(def sql:get-comments
@ -390,32 +400,49 @@
;; All the profiles that had comment the file, plus the current
;; profile.
(defn- file-comment-users-sql
[where]
(str/ffmt
"WITH available_profiles AS (
SELECT DISTINCT c.owner_id AS id
FROM comment c
JOIN comment_thread ct
ON ct.id = c.thread_id
WHERE ct.file_id = ?::uuid
%1
),
profile_ids AS (
SELECT id FROM available_profiles
UNION
SELECT ?::uuid
)
SELECT p.id,
p.email,
p.fullname AS name,
p.fullname,
p.photo_id,
p.is_active
FROM profile p
JOIN profile_ids AS x
ON x.id = p.id;"
where))
(def ^:private sql:file-comment-users
"WITH available_profiles AS (
SELECT DISTINCT c.owner_id AS id
FROM comment c
JOIN comment_thread ct
ON ct.id = c.thread_id
WHERE ct.file_id = ?::uuid
),
profile_ids AS (
SELECT id FROM available_profiles
UNION
SELECT ?::uuid
)
SELECT p.id,
p.email,
p.fullname AS name,
p.fullname,
p.photo_id,
p.is_active
FROM profile p
JOIN profile_ids AS x
ON x.id = p.id;")
(file-comment-users-sql ""))
(def ^:private sql:file-comment-users-by-page
(file-comment-users-sql "AND ct.page_id = ANY(?::uuid[])"))
(defn get-file-comments-users
[conn file-id profile-id]
(db/exec! conn [sql:file-comment-users file-id profile-id]))
"Returns the profiles of the users that commented the file. When `pages` is
not nil, only the commenters of the threads on those pages are returned."
[conn file-id profile-id pages]
(if (some? pages)
(db/exec! conn [sql:file-comment-users-by-page
file-id
(db/create-array conn "uuid" pages)
profile-id])
(db/exec! conn [sql:file-comment-users file-id profile-id])))
(def ^:private
schema:get-profiles-for-file-comments
@ -425,14 +452,17 @@
(sv/defmethod ::get-profiles-for-file-comments
"Retrieves a list of profiles with limited set of properties of all
participants on comment threads of the file."
participants on comment threads of the file. When the request is done with
a share link, only the participants on the pages granted by the link are
returned."
{::doc/added "1.15"
::doc/changes ["1.15" "Imported from queries and renamed."]
::doc/changes ["1.15" "Imported from queries and renamed."
"2.19" "With a share link, only the participants on the pages granted by the link are returned."]
::sm/params schema:get-profiles-for-file-comments}
[cfg {:keys [::rpc/profile-id file-id share-id]}]
(db/run! cfg (fn [{:keys [::db/conn] :as cfg}]
(files/check-comment-permissions! cfg profile-id file-id share-id)
(get-file-comments-users conn file-id profile-id))))
(let [perms (files/check-comment-permissions! cfg profile-id file-id share-id)]
(get-file-comments-users conn file-id profile-id (share-link-pages perms))))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; MUTATION COMMANDS
@ -474,21 +504,22 @@
::rtry/when rtry/conflict-exception?
::sm/params schema:create-comment-thread}
[cfg {:keys [::rpc/profile-id file-id page-id share-id] :as params}]
(files/check-comment-permissions! cfg profile-id file-id share-id)
(let [perms (files/check-comment-permissions! cfg profile-id file-id share-id)]
(check-page-scope perms page-id)
(let [{:keys [team-id project-id] :as file} (get-file cfg file-id page-id)]
(-> cfg
(assoc ::quotes/profile-id profile-id)
(assoc ::quotes/team-id team-id)
(assoc ::quotes/project-id project-id)
(assoc ::quotes/file-id file-id)
(quotes/check! {::quotes/id ::quotes/comment-threads-per-file}
{::quotes/id ::quotes/comments-per-file}))
(let [{:keys [team-id project-id] :as file} (get-file cfg file-id page-id)]
(-> cfg
(assoc ::quotes/profile-id profile-id)
(assoc ::quotes/team-id team-id)
(assoc ::quotes/project-id project-id)
(assoc ::quotes/file-id file-id)
(quotes/check! {::quotes/id ::quotes/comment-threads-per-file}
{::quotes/id ::quotes/comments-per-file}))
(let [params (assoc params ::file file)
thread (db/tx-run! cfg create-comment-thread params)]
(let [params (assoc params ::file file)
thread (db/tx-run! cfg create-comment-thread params)]
(vary-meta thread assoc ::audit/props (assoc thread :team-id team-id)))))
(vary-meta thread assoc ::audit/props (assoc thread :team-id team-id))))))
(defn- create-comment-thread
[{:keys [::db/conn] :as cfg}
@ -559,8 +590,9 @@
::sm/params schema:update-comment-thread-status
::db/transaction true}
[{:keys [::db/conn] :as cfg} {:keys [::rpc/profile-id id share-id]}]
(let [{:keys [file-id]} (get-comment-thread conn id ::sql/for-update true)]
(files/check-comment-permissions! cfg profile-id file-id share-id)
(let [thread (get-comment-thread conn id ::sql/for-update true)
perms (files/check-comment-permissions! cfg profile-id (:file-id thread) share-id)]
(check-page-scope perms (:page-id thread))
(upsert-comment-thread-status! conn profile-id id)))
;; --- COMMAND: Update Comment Thread
@ -577,8 +609,9 @@
::sm/params schema:update-comment-thread
::db/transaction true}
[{:keys [::db/conn] :as cfg} {:keys [::rpc/profile-id id is-resolved share-id]}]
(let [{:keys [file-id]} (get-comment-thread conn id ::sql/for-update true)]
(files/check-comment-permissions! cfg profile-id file-id share-id)
(let [thread (get-comment-thread conn id ::sql/for-update true)
perms (files/check-comment-permissions! cfg profile-id (:file-id thread) share-id)]
(check-page-scope perms (:page-id thread))
(db/update! conn :comment-thread
{:is-resolved is-resolved}
{:id id})
@ -603,56 +636,58 @@
(let [{:keys [file-id page-id] :as thread}
(get-comment-thread conn thread-id ::sql/for-update true)
{:keys [team-id project-id] :as file}
(get-file cfg file-id page-id)]
perms (files/check-comment-permissions! cfg profile-id file-id share-id)]
(files/check-comment-permissions! cfg profile-id file-id share-id)
(check-page-scope perms page-id)
(quotes/check! cfg {::quotes/id ::quotes/comments-per-file
::quotes/profile-id profile-id
::quotes/team-id team-id
::quotes/project-id project-id
::quotes/file-id file-id})
(let [{:keys [team-id project-id] :as file}
(get-file cfg file-id page-id)]
(let [profile (profile/get-profile conn profile-id)
mentions (into #{} mentions)
params {:id (uuid/next)
:created-at request-at
:modified-at request-at
:thread-id thread-id
:owner-id profile-id
:content content
:mentions (db/encode-pgarray mentions conn "uuid")}
(quotes/check! cfg {::quotes/id ::quotes/comments-per-file
::quotes/profile-id profile-id
::quotes/team-id team-id
::quotes/project-id project-id
::quotes/file-id file-id})
comment (-> (db/insert! conn :comment params)
(decode-row)
(assoc :file-id file-id)
(add-owner profile))]
(let [profile (profile/get-profile conn profile-id)
mentions (into #{} mentions)
params {:id (uuid/next)
:created-at request-at
:modified-at request-at
:thread-id thread-id
:owner-id profile-id
:content content
:mentions (db/encode-pgarray mentions conn "uuid")}
;; Update thread modified-at attribute and assoc the current
;; profile to the participant set.
(let [mentions (into (:mentions thread) mentions)
participants (-> (:participants thread #{})
(conj profile-id))
params {:modified-at request-at
:participants (db/tjson participants)
:mentions (db/encode-pgarray mentions conn "uuid")}
comment (-> (db/insert! conn :comment params)
(decode-row)
(assoc :file-id file-id)
(add-owner profile))]
;; Update the page-name cached attribute on comment thread table.
params (cond-> params
(not= (:page-name file) (:page-name thread))
(assoc :page-name (:page-name file)))]
;; Update thread modified-at attribute and assoc the current
;; profile to the participant set.
(let [mentions (into (:mentions thread) mentions)
participants (-> (:participants thread #{})
(conj profile-id))
params {:modified-at request-at
:participants (db/tjson participants)
:mentions (db/encode-pgarray mentions conn "uuid")}
(db/update! conn :comment-thread params
{:id thread-id}
{::db/return-keys false}))
;; Update the page-name cached attribute on comment thread table.
params (cond-> params
(not= (:page-name file) (:page-name thread))
(assoc :page-name (:page-name file)))]
;; Update the current profile status in relation to the current thread
(upsert-comment-thread-status! conn profile-id thread-id)
(db/update! conn :comment-thread params
{:id thread-id}
{::db/return-keys false}))
(send-comment-emails! cfg profile comment thread file)
;; Update the current profile status in relation to the current thread
(upsert-comment-thread-status! conn profile-id thread-id)
(vary-meta comment assoc ::audit/props (assoc comment :team-id team-id)))))
(send-comment-emails! cfg profile comment thread file)
(vary-meta comment assoc ::audit/props (assoc comment :team-id team-id))))))
;; --- COMMAND: Update Comment
@ -675,9 +710,11 @@
(get-comment conn id ::sql/for-update true)
{:keys [file-id page-id] :as thread}
(get-comment-thread conn thread-id ::sql/for-update true)]
(get-comment-thread conn thread-id ::sql/for-update true)
(files/check-comment-permissions! cfg profile-id file-id share-id)
perms (files/check-comment-permissions! cfg profile-id file-id share-id)]
(check-page-scope perms page-id)
;; Don't allow edit comments to not owners
(when-not (= owner-id profile-id)
@ -715,8 +752,13 @@
::sm/params schema:delete-comment-thread
::db/transaction true}
[{:keys [::db/conn] :as cfg} {:keys [::rpc/profile-id id share-id]}]
(let [{:keys [owner-id file-id] :as thread} (get-comment-thread conn id ::sql/for-update true)]
(files/check-comment-permissions! cfg profile-id file-id share-id)
(let [{:keys [owner-id file-id page-id] :as thread}
(get-comment-thread conn id ::sql/for-update true)
perms (files/check-comment-permissions! cfg profile-id file-id share-id)]
(check-page-scope perms page-id)
(when-not (= owner-id profile-id)
(ex/raise :type :validation
:code :not-allowed))
@ -741,10 +783,12 @@
(let [{:keys [owner-id thread-id] :as comment}
(get-comment conn id ::sql/for-update true)
{:keys [file-id]}
(get-comment-thread conn thread-id)]
{:keys [file-id page-id]}
(get-comment-thread conn thread-id)
(files/check-comment-permissions! cfg profile-id file-id share-id)
perms (files/check-comment-permissions! cfg profile-id file-id share-id)]
(check-page-scope perms page-id)
(when-not (= owner-id profile-id)
(ex/raise :type :validation
:code :not-allowed))
@ -768,8 +812,9 @@
::sm/params schema:update-comment-thread-position
::db/transaction true}
[{:keys [::db/conn] :as cfg} {:keys [::rpc/profile-id ::rpc/request-at id position frame-id share-id]}]
(let [{:keys [file-id]} (get-comment-thread conn id ::sql/for-update true)]
(files/check-comment-permissions! cfg profile-id file-id share-id)
(let [thread (get-comment-thread conn id ::sql/for-update true)
perms (files/check-comment-permissions! cfg profile-id (:file-id thread) share-id)]
(check-page-scope perms (:page-id thread))
(db/update! conn :comment-thread
{:modified-at request-at
:position (db/pgpoint position)
@ -792,8 +837,9 @@
::sm/params schema:update-comment-thread-frame
::db/transaction true}
[{:keys [::db/conn] :as cfg} {:keys [::rpc/profile-id ::rpc/request-at id frame-id share-id]}]
(let [{:keys [file-id]} (get-comment-thread conn id ::sql/for-update true)]
(files/check-comment-permissions! cfg profile-id file-id share-id)
(let [thread (get-comment-thread conn id ::sql/for-update true)
perms (files/check-comment-permissions! cfg profile-id (:file-id thread) share-id)]
(check-page-scope perms (:page-id thread))
(db/update! conn :comment-thread
{:modified-at request-at
:frame-id frame-id}
@ -801,17 +847,69 @@
{::db/return-keys false})
nil))
;; --- COMMAND: Mark all comment threads as read
;; The dashboard sends the thread ids in batches of this same size
;; (frontend/src/app/main/data/comments.cljs), so the batch has to fit in
;; this limit.
(def ^:private max-mark-all-threads 500)
(def ^:private
schema:mark-all-threads-as-read
[:map {:title "mark-all-threads-as-read"}
[:threads [:vector ::sm/uuid]]])
[:threads [:vector {:max max-mark-all-threads} ::sm/uuid]]])
(def ^:private sql:comment-threads-files
"SELECT ct.id, ct.file_id
FROM comment_thread ct
WHERE ct.id = ANY(?::uuid[])")
(def ^:private sql:upsert-comment-thread-status-batch
"insert into comment_thread_status (thread_id, profile_id, modified_at)
select t.id, ?::uuid, ?::timestamptz
from unnest(?::uuid[]) as t(id)
on conflict (thread_id, profile_id)
do update set modified_at = excluded.modified_at;")
(defn- get-comment-threads-files
[conn thread-ids]
(db/exec! conn [sql:comment-threads-files
(db/create-array conn "uuid" thread-ids)]))
(defn- check-threads-access
"Raises a :not-found error when any of the threads does not exist or the
caller has no access to the file that owns it."
[cfg profile-id thread-ids]
(let [rows (get-comment-threads-files (::db/conn cfg) thread-ids)
found (into #{} (map :id) rows)]
(when-not (= (count thread-ids) (count found))
(ex/raise :type :not-found
:code :object-not-found
:hint "not found"))
(doseq [file-id (distinct (map :file-id rows))]
(files/check-comment-permissions! cfg profile-id file-id nil))))
(defn- upsert-comment-thread-status-batch
[conn profile-id thread-ids]
(db/exec! conn [sql:upsert-comment-thread-status-batch
profile-id
(ct/in-future "1s")
(db/create-array conn "uuid" thread-ids)]))
(sv/defmethod ::mark-all-threads-as-read
"Marks the given comment threads as read for the current profile. The
caller needs to have access to the file that owns every thread."
{::doc/added "1.15"
::doc/changes [["2.19" "The caller needs access to the file that owns every thread and the number of threads is limited to 500"]]
::sm/params schema:mark-all-threads-as-read}
[cfg {:keys [::rpc/profile-id threads] :as params}]
(db/tx-run!
cfg
(fn [{:keys [::db/conn]}]
(doseq [thread-id threads]
(upsert-comment-thread-status! conn profile-id thread-id)))))
[cfg {:keys [::rpc/profile-id threads]}]
(let [thread-ids (distinct threads)]
;; Nothing to check and nothing to write: an empty list must not open a
;; transaction nor run the queries with an empty array.
(when (seq thread-ids)
(db/tx-run!
cfg
(fn [{:keys [::db/conn] :as cfg}]
(check-threads-access cfg profile-id thread-ids)
(upsert-comment-thread-status-batch conn profile-id thread-ids)
nil)))))

3
backend/src/app/rpc/commands/demo.clj
View File

@ -66,7 +66,8 @@
::doc/changes [["1.15" "This method is migrated from mutations to commands."]
["2.18" "Add optional `skip-onboarding` param. When true, the profile is created with `onboarding-viewed` and `release-notes-viewed` (current version) set, skipping the onboarding flow."]
["2.18" "Add optional `expires-in` param. When set, the demo purge is scheduled that long after creation instead of the global deletion delay. Only values between 5 minutes and the global delay are accepted."]
["2.18.1" "Add optional `renderer` param. When set to `wasm` or `svg`, the profile is created with that `renderer` in props. When omitted, no renderer is stored and the Penpot default applies."]]
["2.18.1" "Add optional `renderer` param. When set to `wasm` or `svg`, the profile is created with that `renderer` in props. When omitted, no renderer is stored and the Penpot default applies."]
["2.20" "When `renderer` is omitted, the profile is created with the default renderer for new accounts (`wasm`)."]]
::sm/params schema:create-demo-profile}
[cfg {:keys [skip-onboarding expires-in renderer]}]

22
backend/src/app/rpc/commands/feedback.clj
View File

@ -103,16 +103,16 @@
(d/without-nils
{"error-report.txt" (:error-report params)})]
(eml/send cfg {::eml/reuse-conn true
::eml/factory eml/user-feedback
:to destination
:reply-to (:email profile)
:email (:email profile)
:attachments attachments
(eml/send-template cfg {::eml/reuse-conn true
::eml/template "feedback"
:to destination
:reply-to (:email profile)
:email (:email profile)
:attachments attachments
:feedback-subject (:subject params)
:feedback-type (:type params "not-specified")
:feedback-content (:content params)
:feedback-error-href (:error-href params)
:profile profile})
:feedback-subject (:subject params)
:feedback-type (:type params "not-specified")
:feedback-content (:content params)
:feedback-error-href (:error-href params)
:profile (select-keys profile [:id :fullname :email])})
nil))

16
backend/src/app/rpc/commands/files.clj
View File

@ -1394,16 +1394,16 @@
WHERE file_id = ANY(?::uuid[])")
(def ^:private sql:restore-file-changes
;; Only durable history comes back: a snapshot (carries `label`) or a legacy
;; payload row (carries `data`, the column the new model replaces with
;; `file_data`). Revisions have neither and are deliberately left alone: they
;; are born already scheduled for deletion one hour later, and the collector
;; only looks at rows that carry a date, so unmarking one here would keep it
;; in the table for ever. The `data` half is a guard for the rows still
;; pending migration, not a dependency: drop it with the last one.
;; Only durable history comes back: a snapshot, the rows that carry `label`.
;; Everything else is deliberately left alone. Revisions are born already
;; scheduled for deletion one hour later, and the collector only looks at
;; rows that carry a date, so unmarking one here would keep it in the table
;; for ever. Rows that still carry a legacy payload without a label are
;; garbage that no read path reaches; 0157-migrate-legacy-file-data deletes
;; them, so they must keep their date and go with the collector.
"UPDATE file_change SET deleted_at = null
WHERE file_id = ANY(?::uuid[])
AND (label IS NOT NULL OR data IS NOT NULL)")
AND label IS NOT NULL")
(def ^:private sql:restore-file-data
"UPDATE file_data SET deleted_at = null

17
backend/src/app/rpc/commands/files_snapshot.clj
View File

@ -196,13 +196,20 @@
(ex/raise :type :conflict
:code :file-modified
:hint "the file was modified during the restore process, please retry")))
(let [vern (fsnap/restore! cfg file-id id)]
(let [snapshot (fsnap/get-minimal-snapshot cfg id)
vern (fsnap/restore! cfg file-id id)]
;; Peers name the restored version by its label when a
;; user named it, else by its date
(mbus/pub! msgbus
:topic file-id
:message {:type :file-restored
:session-id session-id
:file-id file-id
:vern vern})
:message (cond-> {:type :file-restored
:session-id session-id
:profile-id profile-id
:file-id file-id
:vern vern
:created-at (:created-at snapshot)}
(= "user" (:created-by snapshot))
(assoc :label (:label snapshot))))
nil)))))
(def ^:private schema:update-file-snapshot

16
backend/src/app/rpc/commands/files_thumbnails.clj
View File

@ -23,6 +23,7 @@
[app.features.fdata :as fdata]
[app.loggers.audit :as-alias audit]
[app.loggers.webhooks :as-alias webhooks]
[app.media.svg :as svg]
[app.media.validation :as media.v]
[app.rpc :as-alias rpc]
[app.rpc.climit :as-alias climit]
@ -247,8 +248,7 @@
(defn- persist-thumbnail!
[storage media created-at]
(let [path (:path media)
mtype (:mtype media)
(let [{:keys [path mtype]} (svg/sanitize-svg-upload media)
hash (sto/calculate-hash path)
data (-> (sto/content path)
(sto/wrap-with-hash hash))]
@ -302,8 +302,9 @@
::rpc/multipart-only-params [:media]}
[cfg {:keys [::rpc/profile-id file-id object-id media tag]}]
(media.v/validate-media-type! media)
(media.v/validate-media-size! media)
(media.v/check-upload-path media)
(media.v/check-media-type media)
(media.v/check-media-size media)
(db/run! cfg files/check-edition-permissions! profile-id file-id)
(when-let [file (files/get-minimal-file cfg file-id {::db/check-deleted false})]
@ -404,8 +405,9 @@
::rpc/multipart-only-params [:media]}
[cfg {:keys [::rpc/profile-id file-id] :as params}]
(media.v/validate-media-type! (:media params))
(media.v/validate-media-size! (:media params))
(media.v/check-upload-path (:media params))
(media.v/check-media-type (:media params))
(media.v/check-media-size (:media params))
(db/run! cfg files/check-edition-permissions! profile-id file-id)
@ -413,7 +415,7 @@
(let [storage (::sto/storage cfg)
file (bfc/get-file cfg file-id :include-deleted? true :load-data? false)
props (db/tjson (or (:props params) {}))
{:keys [path mtype]} (:media params)
{:keys [path mtype]} (svg/sanitize-svg-upload (:media params))
hash (sto/calculate-hash path)
data (-> (sto/content path)
(sto/wrap-with-hash hash))

587
backend/src/app/rpc/commands/files_update.clj
View File

@ -38,7 +38,8 @@
[app.util.blob :as blob]
[app.util.pointer-map :as pmap]
[app.util.services :as sv]
[clojure.set :as set]))
[clojure.set :as set]
[cuerdas.core :as str]))
(declare ^:private get-lagged-changes)
(declare ^:private send-notifications!)
@ -69,19 +70,38 @@
[:changes-with-metadata {:optional true}
[:vector [:map
[:changes [:vector cpc/schema:change]]
[:commit-id {:optional true} ::sm/uuid]
[:hint-origin {:optional true} :keyword]
[:hint-events {:optional true} [:vector [:string {:max 250}]]]]]]
[:skip-validate {:optional true} ::sm/boolean]])
(def ^:private
schema:update-file-result
[:vector {:title "update-file-result"}
[:map
[:changes [:vector cpc/schema:change]]
[:file-id ::sm/uuid]
[:id ::sm/uuid]
[:revn {:min 0} ::sm/int]
[:session-id ::sm/uuid]]])
[:map {:title "update-file-result"}
;; The revision the file had before this update
[:revn {:min 0} ::sm/int]
;; Every change stored after the incoming revision, this one included
[:lagged
[:vector
[:map
[:changes [:vector cpc/schema:change]]
[:file-id ::sm/uuid]
[:id ::sm/uuid]
[:revn {:min 0} ::sm/int]
[:session-id ::sm/uuid]]]]
;; Commits not saved: the first one that failed and every later one
[:rejected {:optional true}
[:map
[:commit-ids [:vector ::sm/uuid]]
[:failed-commit-id ::sm/uuid]
;; The data the failed commit touches was invalid before it
[:preexisting-damage {:optional true} :boolean]
[:error
[:map
[:type :keyword]
[:code {:optional true} [:maybe :keyword]]
[:hint {:optional true} [:maybe :string]]
[:details {:optional true} [:vector :any]]]]]]])
;; --- HELPERS
@ -184,12 +204,18 @@
(->> changes-with-metadata (mapcat :changes) vec)
(vec changes))
;; Saved commit by commit only when every group names its commit
commits (when (and (seq changes-with-metadata)
(every? :commit-id changes-with-metadata))
changes-with-metadata)
params (-> params
(assoc :profile-id profile-id)
(assoc :features (set/difference features cfeat/frontend-only-features))
(assoc :team team)
(assoc :file file)
(assoc :changes changes))
(assoc :changes changes)
(assoc :commits commits))
cfg (assoc cfg ::timestamp (ct/now)
::validation-timings (atom {}))
@ -280,7 +306,7 @@
Only intended for internal use on this module."
[{:keys [::db/conn ::timestamp] :as cfg}
{:keys [profile-id file team features changes session-id skip-validate] :as params}]
{:keys [profile-id file team features changes commits session-id skip-validate] :as params}]
(fdata/with-tracked-pointers cfg (:id file)
@ -298,7 +324,17 @@
cfeat/*previous* (:features file)]
(update-file-data! cfg file
process-changes-and-validate
changes skip-validate))
changes commits skip-validate))
;; Only the accepted changes are stored and broadcast
changes (::changes file)
rejected (::rejected file)
params (assoc params :changes changes)
file (dissoc file ::changes ::rejected)
;; A request whose first commit is rejected leaves the file as is
saved? (or (nil? rejected) (seq changes))
snapshot (when saved? (::snapshot file))
deleted-at
(ct/plus timestamp (ct/duration {:hours 1}))
@ -315,7 +351,7 @@
(ct/plus (ct/duration {:millis 1})))]
(if repair
(let [snapshot-file (or (::snapshot file)
(let [snapshot-file (or snapshot
(fdata/realize cfg (:original repair)))]
(fsnap/create! cfg snapshot-file
{:label (str "internal/repair/" revn)
@ -323,62 +359,65 @@
:deleted-at snapshot-deleted-at
:profile-id profile-id
:session-id repair-session-id}))
(when-let [file (::snapshot file)]
(fsnap/create! cfg file
(when snapshot
(fsnap/create! cfg snapshot
{:label (str "internal/snapshot/" revn)
:created-by "system"
:deleted-at snapshot-deleted-at
:profile-id profile-id
:session-id session-id})))
(when repair
(when saved?
(when repair
(db/insert! conn :file-change
{:id (uuid/next)
:session-id repair-session-id
:profile-id profile-id
:created-at timestamp
:updated-at timestamp
:deleted-at deleted-at
:file-id (:id file)
:revn (:revn repair)
:version (:version file)
:features (into-array (:features file))
:changes (blob/encode (:changes repair))}
{::db/return-keys false})
(l/wrn :hint "file auto-repaired"
:file-id (str (:id file))
:error-codes (error-codes (:errors repair))
:error-count (count (:errors repair))
:change-count (count (:changes repair)))
(record-repair-outcome cfg :repaired))
;; Insert change (xlog) with deleted_at in a future data for
;; make them automatically eleggible for GC once they expires
(db/insert! conn :file-change
{:id (uuid/next)
:session-id repair-session-id
:session-id session-id
:profile-id profile-id
:created-at timestamp
:updated-at timestamp
:created-at change-created-at
:updated-at change-created-at
:deleted-at deleted-at
:file-id (:id file)
:revn (:revn repair)
:revn (:revn file)
:version (:version file)
:features (into-array (:features file))
:changes (blob/encode (:changes repair))}
:changes (blob/encode changes)}
{::db/return-keys false})
(l/wrn :hint "file auto-repaired"
:file-id (str (:id file))
:error-codes (error-codes (:errors repair))
:error-count (count (:errors repair))
:change-count (count (:changes repair)))
(record-repair-outcome cfg :repaired))
;; Insert change (xlog) with deleted_at in a future data for
;; make them automatically eleggible for GC once they expires
(db/insert! conn :file-change
{:id (uuid/next)
:session-id session-id
:profile-id profile-id
:created-at change-created-at
:updated-at change-created-at
:deleted-at deleted-at
:file-id (:id file)
:revn (:revn file)
:version (:version file)
:features (into-array (:features file))
:changes (blob/encode changes)}
{::db/return-keys false})
(persist-file! cfg (dissoc file ::repair))
(persist-file! cfg (dissoc file ::repair))
(when (contains? cf/flags :redis-cache)
(invalidate-caches! cfg file))
(when (contains? cf/flags :redis-cache)
(invalidate-caches! cfg file))
;; Send asynchronous notifications
(when repair
(send-repair-notifications cfg params file repair))
(send-notifications! cfg params file))
;; Send asynchronous notifications
(when repair
(send-repair-notifications cfg params file repair))
(send-notifications! cfg params file)
(with-meta {:revn revn :lagged (get-lagged-changes conn params)}
(with-meta (cond-> {:revn revn :lagged (get-lagged-changes conn params)}
(some? rejected)
(assoc :rejected rejected))
{::audit/replace-props
{:id (:id file)
:name (:name file)
@ -390,7 +429,7 @@
;; `modified-at` mirrors `persist-file!`: same `::timestamp`
;; fallback, so both agree to the millisecond when the
;; request timestamp exists.
:revn (:revn file)
:revn (if saved? (:revn file) revn)
:modified-at (or timestamp (ct/now))}}))))
(defn get-file
@ -484,9 +523,9 @@
(apply update-fn cfg file args)))
(defn- soft-validate-file-schema!
[file]
[file changes]
(try
(val/validate-file-schema! file)
(val/validate-file-schema-affected! file changes)
(catch Throwable cause
(l/error :hint "file schema validation error" :cause cause))))
@ -534,16 +573,236 @@
(mapv (fn [id] {:type :add-page :id id})
page-ids))))
(defn- soft-validate-added-pages!
"Logs the errors of `validate-added-pages` instead of raising them."
[file libs changes]
(try
(when-let [errors (validate-added-pages file libs changes)]
(l/error :hint "added page validation error"
:file-id (str (:id file))
:page-ids (->> errors (map :page-id) distinct (mapv str))
:codes (->> errors (map :code) distinct (mapv name))))
(catch Throwable cause
(l/error :hint "added page validation error"
:file-id (str (:id file))
:cause cause))))
(defmacro ^:private with-validation-timing
"Runs `body` and records its time as the validation step `step-key`
(`step-label` in the metric), also when it throws."
[cfg step-key step-label & body]
`(let [t# (ct/tpoint)]
(try
~@body
(finally
(record-validation-timing ~cfg ~step-key ~step-label (inst-ms (t#)))))))
(defn- validate-file
"Validates the entities `changes` affect and the schema of the data they
modify."
[cfg file libs changes skip-validate]
(binding [pmap/*tracked* nil]
;; Pages added through the `{:page ...}` form have no top-level `:id`,
;; so the affected checks below do not see them. Check them here in
;; soft mode, whenever libraries are resolved for a referential
;; check, until the hard check learns to collect them.
(when (and (or (contains? cf/flags :file-validation)
(contains? cf/flags :soft-file-validation))
(not skip-validate))
(soft-validate-added-pages! file libs changes))
(when (contains? cf/flags :soft-file-validation)
(with-validation-timing cfg :soft-validate-affected-ms "soft-validate-affected"
(soft-validate-file! file libs changes)))
(when (contains? cf/flags :soft-file-schema-validation)
(with-validation-timing cfg :soft-validate-schema-ms "soft-validate-schema"
(soft-validate-file-schema! file changes)))
(when (and (contains? cf/flags :file-validation)
(not skip-validate))
(with-validation-timing cfg :validate-affected-ms "validate-affected"
(val/validate-file-affected! file libs changes)))
(when (and (contains? cf/flags :file-schema-validation)
(not skip-validate))
(with-validation-timing cfg :validate-schema-ms "validate-schema"
(val/validate-file-schema-affected! file changes)))))
(def ^:private debug-fault-prefix
"Start of the plugin data key that asks for a debug save fault."
"save-fault:")
(defn- get-debug-fault
"Name of the fault a debug marker in `changes` asks for, or nil. A marker
is a `:set-plugin-data` of the file in the `:penpot-debug` namespace with
no value, so applying it changes nothing."
[changes]
(some (fn [{:keys [type namespace key]}]
(when (and (= :set-plugin-data type)
(= :penpot-debug namespace)
(str/starts-with? key debug-fault-prefix))
(subs key (count debug-fault-prefix))))
changes))
(defn- raise-debug-fault
"Throws the error named `fault`, standing in for a real failure while the
changes are applied."
[fault]
(let [message "debug save fault"]
(case fault
"validation" (ex/raise :type :validation :code :debug-fault :hint message)
"runtime" (throw (RuntimeException. message))
"ex-info" (throw (ex-info message {}))
"assertion" (throw (AssertionError. message))
"sql" (throw (org.postgresql.util.PSQLException.
message org.postgresql.util.PSQLState/CONNECTION_FAILURE))
"io" (throw (java.net.SocketTimeoutException. message))
"eof" (throw (java.io.EOFException. message))
"interrupted" (throw (InterruptedException. message))
"not-found" (ex/raise :type :not-found :code :object-not-found :hint message)
"stack-overflow" (throw (StackOverflowError. message))
(ex/raise :type :validation :code :unknown-debug-fault :hint fault))))
(defn- apply-changes
[cfg file changes]
(let [state (atom {})
file (binding [cpc/*state* state]
(-> file
(update :data cpc/process-changes changes)
(update :data d/without-nils)))]
(if-let [media-refs (-> @state :media-refs not-empty)]
(bfc/update-media-references! cfg file media-refs)
file)))
"Applies and validates `changes`. With the `debug-save-faults` flag, a
debug marker in them makes this fail with the error it names."
[cfg file libs changes skip-validate]
(when (contains? cf/flags :debug-save-faults)
(some-> (get-debug-fault changes) (raise-debug-fault)))
(let [file (with-validation-timing cfg :process-changes-ms "process-changes"
(update file :data (fn [data]
(-> data
(cpc/process-changes changes)
(d/without-nils)))))]
(validate-file cfg file libs changes skip-validate)
file))
(defn- decode-objects
"Decodes the objects of the pages `changes` touch. Every file derived from
`file` shares them, so each attempt does not decode them again."
[file changes]
(doseq [page-id (into #{} (keep :page-id) changes)]
(let [objects (get-in file [:data :pages-index page-id :objects])]
(doseq [id (keys objects)]
(get objects id)))))
(defn- infrastructure-error?
[cause]
(some (fn [cause]
(or (instance? java.sql.SQLException cause)
(instance? java.io.IOException cause)
(instance? InterruptedException cause)
;; A file fragment that cannot be loaded
(= :not-found (:type (ex-data cause)))))
(take-while some? (iterate ex-cause cause))))
(defn- change-failure?
"True when `cause` comes from applying or validating changes. Only these
reject a commit; any other error fails the whole request."
[cause]
(and (or (instance? Exception cause)
(instance? AssertionError cause))
(not (infrastructure-error? cause))))
(defn- preexisting-damage?
"True when `file` already fails validation on the pages and components
`changes` touch, before they are applied."
[file libs changes skip-validate]
(try
(binding [pmap/*tracked* nil]
(when (and (contains? cf/flags :file-validation)
(not skip-validate))
(val/validate-file-affected! file libs changes))
(when (and (contains? cf/flags :file-schema-validation)
(not skip-validate))
(val/validate-file-schema-affected! file changes)))
false
(catch Throwable cause
(if (change-failure? cause)
true
(throw cause)))))
(defn- rejection
"Describes `commits` as rejected, the first one failing with `cause`.
With `damaged`, the data that commit touches was invalid before it."
[commits cause damaged]
(let [data (ex-data cause)]
(cond-> {:commit-ids (mapv :commit-id commits)
:failed-commit-id (:commit-id (first commits))
;; An ex-info without a type reports like any unexpected exception
:error (if (:type data)
(cond-> {:type (:type data)
:code (:code data)
:hint (or (:hint data) (ex-message cause))}
(:details data)
(assoc :details (mapv #(dissoc % :shape) (:details data))))
{:type :internal
:code :unexpected-error
:hint (or (ex-message cause) (.getName (class cause)))})}
damaged
(assoc :preexisting-damage true))))
(defn- apply-prefix
"Applies and validates the first `n` commits of `attempt`, starting from
its `:file` and its `:tracked` pointers. Returns the resulting `:file`,
with the pointers to persist in `:tracked` and the media refs in `:refs`,
or the cause of the failure."
[{:keys [cfg file libs commits skip-validate state tracked]} n]
(reset! pmap/*tracked* tracked)
(reset! state {})
(let [changes (into [] (comp (take n) (mapcat :changes)) commits)]
(try
(let [file (apply-changes cfg file libs changes skip-validate)]
{:file (assoc file ::changes changes)
:tracked @pmap/*tracked*
:refs @state})
(catch Throwable cause
(when-not (change-failure? cause)
(throw cause))
cause))))
(defn- apply-commits
"Keeps the longest prefix of the `:commits` of `attempt` that applies and
validates, found by bisection, and rejects the rest. `cause` is the
failure of all of them together. Takes about log2 of the commits
attempts; a commit that repairs an earlier one can keep both. Leaves the
pointers and media refs of the kept prefix in place and returns its file,
with the accepted changes in `::changes` and the rejection in
`::rejected`, which tells whether the data the first rejected commit
touches was already invalid. The whole search is timed as the `retry`
step of `cfg`."
[cfg {:keys [file libs commits skip-validate state tracked] :as attempt} cause]
(let [tpoint (ct/tpoint)]
(with-validation-timing cfg :retry-ms "retry"
(loop [lo 0
hi (count commits)
cause cause
attempts 0
accepted nil]
(if (> (- hi lo) 1)
(let [mid (quot (+ lo hi) 2)
result (apply-prefix attempt mid)]
(if (instance? Throwable result)
(recur lo mid result (inc attempts) accepted)
(recur mid hi cause (inc attempts) result)))
(let [rejected (subvec commits lo)
base (:file accepted file)
damaged (preexisting-damage? base libs (:changes (first rejected))
skip-validate)]
(reset! pmap/*tracked* (:tracked accepted tracked))
(reset! state (:refs accepted {}))
(l/error :hint "update-file commit rejected"
:file-id (str (:id file))
:commit-id (str (:commit-id (first rejected)))
:rejected (count rejected)
:commits (count commits)
:attempts attempts
:preexisting-damage damaged
:elapsed (ct/format-duration (tpoint))
:cause cause)
(assoc (:file accepted (assoc file ::changes []))
::rejected (rejection rejected cause damaged))))))))
(defn- fail-repair
[cfg file errors cause]
@ -556,10 +815,11 @@
(defn- repair-preexisting-damage
"Repair damage that was already in `original` on the pages/components
the client's `changes` touch, then re-apply those changes. Throws
`cause` when the client's changes are at fault or the repair cannot
fix the file."
[cfg original libs changes tracked tracked-0 cause]
the client's `changes` touch, then re-apply those changes. Starts from
the `tracked` pointers and leaves the media refs of the changes in
`cpc/*state*`. Throws `cause` when the client's changes are at fault
or the repair cannot fix the file."
[cfg original libs changes tracked cause]
(let [libs (or libs {})]
(try
(let [pre-errors
@ -569,49 +829,42 @@
(when (empty? pre-errors)
(throw cause))
(when tracked
(reset! tracked tracked-0))
(reset! pmap/*tracked* tracked)
(let [trepair (ct/tpoint)
result
(try
(binding [pmap/*tracked* tracked]
(cfr/repair-file-until-valid
original libs
(fn [file applied]
(binding [pmap/*tracked* nil]
(val/validate-file-affected file libs (into applied changes))))
:initial-errors pre-errors))
(finally
(record-validation-timing cfg :repair-ms "repair" (inst-ms (trepair)))))
(let [result
(with-validation-timing cfg :repair-ms "repair"
(cfr/repair-file-until-valid
original libs
(fn [file applied]
(binding [pmap/*tracked* nil]
(val/validate-file-affected file libs (into applied changes))))
:initial-errors pre-errors))
repair-changes (:changes result)]
(when (seq (:errors result))
(fail-repair cfg original (:errors result) cause))
(let [treprocess (ct/tpoint)
repaired (try
(binding [pmap/*tracked* tracked]
(apply-changes cfg (:file result) changes))
(finally
(record-validation-timing cfg :reprocess-changes-ms "reprocess-changes"
(inst-ms (treprocess)))))
scope (into repair-changes changes)]
(reset! cpc/*state* {})
(let [repaired (with-validation-timing cfg :reprocess-changes-ms "reprocess-changes"
(update (:file result) :data
(fn [data]
(-> data
(cpc/process-changes changes)
(d/without-nils)))))
scope (into repair-changes changes)]
(binding [pmap/*tracked* nil]
(val/validate-file-affected! repaired libs scope)
(when (contains? cf/flags :file-schema-validation)
(let [t (ct/tpoint)]
(try
(val/validate-file-schema! repaired)
(finally
(record-validation-timing cfg :validate-schema-ms "validate-schema" (inst-ms (t))))))))
(with-validation-timing cfg :validate-schema-ms "validate-schema"
(val/validate-file-schema! repaired))))
(-> repaired
(assoc :revn (+ (:revn original) 2))
(assoc ::changes changes)
(assoc ::repair {:changes repair-changes
:errors pre-errors
:revn (inc (:revn original))
@ -626,96 +879,94 @@
(record-repair-outcome cfg :unrepaired)
(throw cause)))))))
(defn- repair-or-reject
"Handles the failure `cause` of applying all `changes` at once. Repairs
damage `original` already had when `cause` is a referential integrity
error. If that does not save every change and there are `commits`,
keeps the longest prefix of them that validates."
[cfg original file libs changes commits skip-validate tracked cause]
(let [repaired
(when (and (referential-integrity-error? cause)
(contains? cf/flags :file-validation)
(not skip-validate))
(try
(repair-preexisting-damage cfg original libs changes tracked cause)
(catch Throwable inner
(when-not (identical? inner cause)
(throw inner)))))]
(cond
(some? repaired)
repaired
(and (some? commits)
(change-failure? cause))
(apply-commits cfg
{;; The retries record no step timings
:cfg (dissoc cfg ::mtx/metrics ::validation-timings)
:file file
:libs libs
:commits (vec commits)
:skip-validate skip-validate
:state cpc/*state*
:tracked tracked}
cause)
:else
(throw cause))))
(defn- process-changes-and-validate
[cfg file changes skip-validate]
(let [original file
tracked pmap/*tracked*
tracked-0 (some-> tracked deref)
"Applies and validates all `changes` at once. If that fails, repairs
damage the file already had or keeps the longest prefix of the
`commits` that validates."
[cfg file changes commits skip-validate]
(let [original file
should-resolve-libs?
(and (or (contains? cf/flags :file-validation)
(contains? cf/flags :soft-file-validation))
(not skip-validate))
tlibs
(ct/tpoint)
libs
(try
(when should-resolve-libs?
(bfc/get-resolved-file-libraries cfg file))
(finally
(when should-resolve-libs?
(record-validation-timing cfg :resolve-libs-ms "resolve-libs" (inst-ms (tlibs))))))
(when should-resolve-libs?
(with-validation-timing cfg :resolve-libs-ms "resolve-libs"
(bfc/get-resolved-file-libraries cfg file)))
tproc
(ct/tpoint)
;; The main purpose of this atom is provide a contextual state
;; for the changes subsystem where optionally some hints can
;; be provided for the changes processing. Right now we are
;; using it for notify about the existence of media refs when
;; a new shape is added.
state
(atom {})
;; True when validation reads every object of the touched pages
schema-check?
(or (contains? cf/flags :soft-file-schema-validation)
(and (contains? cf/flags :file-schema-validation)
(not skip-validate)))
file
(try
(apply-changes cfg
(-> (files/check-version! file)
(update :revn inc))
changes)
(finally
(record-validation-timing cfg :process-changes-ms "process-changes" (inst-ms (tproc)))))
(-> (files/check-version! file)
(update :revn inc))
pending-repair
(atom nil)]
tracked
@pmap/*tracked*
(fdata/with-untracked
;; Pages added through the `{:page ...}` form have no top-level `:id`,
;; so the affected check below does not see them. Check them here in
;; soft mode until the hard check learns to collect them.
(when should-resolve-libs?
(try
(when-let [errors (validate-added-pages file libs changes)]
(l/error :hint "added page validation error"
:file-id (str (:id file))
:page-ids (->> errors (map :page-id) distinct (mapv str))
:codes (->> errors (map :code) distinct (mapv name))))
(catch Throwable cause
(l/error :hint "added page validation error"
:file-id (str (:id file))
:cause cause))))
(when (contains? cf/flags :soft-file-validation)
(let [t (ct/tpoint)]
file
(binding [cpc/*state* state]
;; Decoded on `file`, the objects the schema check reads are
;; shared by every retry
(when (and (next commits) schema-check?)
(decode-objects file changes))
(try
(soft-validate-file! file libs changes)
(finally
(record-validation-timing cfg :soft-validate-affected-ms "soft-validate-affected" (inst-ms (t)))))))
(when (contains? cf/flags :soft-file-schema-validation)
(let [t (ct/tpoint)]
(try
(soft-validate-file-schema! file)
(finally
(record-validation-timing cfg :soft-validate-schema-ms "soft-validate-schema" (inst-ms (t)))))))
(when (and (contains? cf/flags :file-validation)
(not skip-validate))
(let [t (ct/tpoint)]
(try
(val/validate-file-affected! file libs changes)
(-> (apply-changes cfg file libs changes skip-validate)
(assoc ::changes changes))
(catch Throwable cause
(reset! pending-repair cause)
(when-not (referential-integrity-error? cause)
(throw cause)))
(finally
(record-validation-timing cfg :validate-affected-ms "validate-affected" (inst-ms (t)))))))
(repair-or-reject cfg original file libs changes commits
skip-validate tracked cause))))]
(when (and (contains? cf/flags :file-schema-validation)
(not skip-validate)
(nil? @pending-repair))
(let [t (ct/tpoint)]
(try
(val/validate-file-schema! file)
(finally
(record-validation-timing cfg :validate-schema-ms "validate-schema" (inst-ms (t))))))))
(if-let [cause @pending-repair]
(repair-preexisting-damage cfg original libs changes tracked tracked-0 cause)
(if-let [media-refs (-> @state :media-refs not-empty)]
(bfc/update-media-references! cfg file media-refs)
file)))
(defn- take-snapshot?

6
backend/src/app/rpc/commands/fonts.clj
View File

@ -122,9 +122,9 @@
(let [data (reduce-kv
(fn [acc mtype session-id]
(let [assembled (assemble-chunks cfg profile-id session-id)]
(-> {:mtype mtype :size (:size assembled)}
(media.v/validate-media-type! cm/font-types)
(media.v/validate-font-size!))
(-> {:mtype mtype :size (:size assembled) :path (:path assembled)}
(media.v/check-media-type cm/font-types)
(media.v/check-font-size))
(assoc acc mtype (:path assembled))))
{}
uploads)]

3
backend/src/app/rpc/commands/ldap.clj
View File

@ -89,7 +89,8 @@
:member-id (:id profile)
:member-email (:email profile))
token (tokens/generate cfg claims)]
(-> {:invitation-token token}
(-> (profile/strip-private-attrs profile)
(assoc :invitation-token token)
(rph/with-transform (session/create-fn cfg profile))
(rph/with-meta {::audit/props (:props profile)
::audit/profile-id (:id profile)})))

10
backend/src/app/rpc/commands/management.clj
View File

@ -106,10 +106,12 @@
(db/exec-one! conn ["SET CONSTRAINTS ALL DEFERRED"])
(binding [bfc/*state* (volatile! {:index {file-id (uuid/next)}})]
(duplicate-file (assoc cfg ::bfc/timestamp (ct/now))
(-> params
(assoc :profile-id profile-id)
(assoc :reset-shared-flag true)))))))
(-> (duplicate-file (assoc cfg ::bfc/timestamp (ct/now))
(-> params
(assoc :profile-id profile-id)
(assoc :reset-shared-flag true)))
;; Clients only need the metadata, don't send the full file contents.
(dissoc :data))))))
;; --- COMMAND: Duplicate Project

20
backend/src/app/rpc/commands/media.clj
View File

@ -65,8 +65,9 @@
[:process-image/global]]}
[{:keys [::db/pool] :as cfg} {:keys [::rpc/profile-id file-id content] :as params}]
(files/check-edition-permissions! pool profile-id file-id)
(media.v/validate-media-type! content)
(media.v/validate-media-size! content)
(media.v/check-upload-path content)
(media.v/check-media-type content)
(media.v/check-media-size content)
(let [team-id (:team-id (db/exec-one! pool [sql:get-team-id-for-file file-id]))]
;; TEMPORARILY DISABLED (2026-10-19): media-storage-bytes-per-team
@ -136,15 +137,7 @@
(defn- process-main-image
[info]
(let [path (:path info)
mtype (:mtype info)
path (if (= mtype "image/svg+xml")
(let [content (slurp path)
sanitized (svg/sanitize-svg content)
temp-path (tmp/tempfile :prefix "penpot-svg-" :suffix ".svg" :min-age "5m")]
(spit (str temp-path) sanitized)
temp-path)
path)
(let [{:keys [path mtype]} (svg/sanitize-svg-upload info)
hash (sto/calculate-hash path)
data (-> (sto/content path)
(sto/wrap-with-hash hash))]
@ -376,6 +369,7 @@
::sm/result schema:upload-chunk-result}
[{:keys [::db/pool] :as cfg}
{:keys [::rpc/profile-id session-id index content] :as _params}]
(media.v/check-upload-path content)
(let [session (db/tx-run! cfg check-upload-chunk-slot session-id profile-id index content)]
(l/trc :hint "upload-chunk"
:session-id session-id
@ -576,8 +570,8 @@
content (-> content
(assoc :filename (str "upload:" name))
(assoc :mtype mtype)
(media.v/validate-media-type!)
(media.v/validate-media-size!))
(media.v/check-media-type)
(media.v/check-media-size))
mobj (create-file-media-object cfg (assoc params
:from-chunks? true
:content content))]

22
backend/src/app/rpc/commands/nitrate.clj
View File

@ -445,7 +445,8 @@
;; Check moveTeams permission on the source organization
(when (contains? cf/flags :admin-console)
(let [organization-perms (nitrate/call cfg :get-organization-permissions
{:organization-id organization-id})]
{:organization-id organization-id
::rpc/profile-id profile-id})]
(if (nil? organization-perms)
(ex/raise :type :validation
:code :not-allowed
@ -480,8 +481,10 @@
"Returns info about external (non-organization-member) invitations pending for a team.
External invitations are those sent to users who are not members of the given organization.
Returns {:allows-anybody bool :external-emails [...]}"
[{:keys [::db/conn] :as cfg} team-id organization-id]
(let [organization-perms (nitrate/call cfg :get-organization-permissions {:organization-id organization-id})
[{:keys [::db/conn] :as cfg} team-id organization-id profile-id]
(let [organization-perms (nitrate/call cfg :get-organization-permissions
{:organization-id organization-id
::rpc/profile-id profile-id})
allows-anybody (cto/allowed? :add-anybody-to-team {:organization-perms organization-perms})]
(if allows-anybody
{:allows-anybody true :external-emails []}
@ -516,13 +519,16 @@
(when (contains? cf/flags :admin-console)
(let [organization-member-ids-before (into #{} (nitrate/call cfg :get-organization-members {:organization-id organization-id}))
team-with-organization (nitrate/call cfg :get-team-organization {:team-id team-id})
team-with-organization (nitrate/call cfg :get-team-organization {:team-id team-id
::rpc/profile-id profile-id})
source-organization-id (get-in team-with-organization [:organization :id])
source-organization-perms (when source-organization-id
(nitrate/call cfg :get-organization-permissions
{:organization-id source-organization-id}))
{:organization-id source-organization-id
::rpc/profile-id profile-id}))
target-organization-perms (nitrate/call cfg :get-organization-permissions
{:organization-id organization-id})
{:organization-id organization-id
::rpc/profile-id profile-id})
target-organization-same-owner? (and (some? source-organization-perms)
(some? target-organization-perms)
(= (:owner-id source-organization-perms)
@ -571,7 +577,7 @@
(notifications/notify-team-change cfg team "dashboard.team-belong-organization"))
;; Delete pending invitations for users who are not members of the target organization
(let [{:keys [allows-anybody external-emails]} (get-external-invitation-info cfg team-id organization-id)]
(let [{:keys [allows-anybody external-emails]} (get-external-invitation-info cfg team-id organization-id profile-id)]
(when (and (not allows-anybody) (seq external-emails))
(let [conn (::db/conn cfg)
emails-array (db/create-array conn "text" external-emails)]
@ -686,7 +692,7 @@
(ex/raise :type :validation
:code :insufficient-permissions))
(nitrate/assert-membership cfg profile-id organization-id)
(let [{:keys [allows-anybody external-emails]} (get-external-invitation-info cfg team-id organization-id)]
(let [{:keys [allows-anybody external-emails]} (get-external-invitation-info cfg team-id organization-id profile-id)]
{:has-external-invitations (boolean (seq external-emails))
:allows-anybody allows-anybody}))
{:has-external-invitations false

31
backend/src/app/rpc/commands/profile.clj
View File

@ -225,11 +225,11 @@
(update-profile-password! cfg (assoc profile :password password))
(eml/send cfg {::eml/reuse-conn true
::eml/factory eml/password-changed
:public-uri (cf/get :public-uri)
:to (:email profile)
:name (:fullname profile)})
(eml/send-template cfg {::eml/reuse-conn true
::eml/template "password-changed"
:public-uri (cf/get :public-uri)
:to (:email profile)
:name (:fullname profile)})
(->> (rph/get-request params)
(session/get-session)
@ -313,8 +313,9 @@
::sm/result :nil}
[cfg {:keys [::rpc/profile-id file] :as params}]
;; Validate incoming mime type
(media.v/validate-media-type! file #{"image/jpeg" "image/png" "image/webp"})
(media.v/validate-media-size! file)
(media.v/check-upload-path file)
(media.v/check-media-type file #{"image/jpeg" "image/png" "image/webp"})
(media.v/check-media-size file)
(update-profile-photo cfg (assoc params :profile-id profile-id)))
(defn update-profile-photo
@ -462,14 +463,14 @@
:email (:email profile)
:hint "looks like the email has spam complaint reports"))
(eml/send cfg {::eml/reuse-conn true
::eml/factory eml/change-email
:public-uri (cf/get :public-uri)
:to (:email profile)
:name (:fullname profile)
:pending-email email
:token token
:extra-data ptoken})
(eml/send-template cfg {::eml/reuse-conn true
::eml/template "change-email"
:public-uri (cf/get :public-uri)
:to (:email profile)
:name (:fullname profile)
:pending-email email
:token token
:extra-data ptoken})
nil))
;; --- MUTATION: Update Profile Props

25
backend/src/app/rpc/commands/teams.clj
View File

@ -85,6 +85,17 @@
(def check-read-permissions!
(perms/make-check-fn has-read-permissions?))
(defn check-invitation-permissions
"Returns the team permissions of `profile-id`, raising the error of the
invitation family when they are not the admin ones that every invitation
command requires. Accepts a connection, a pool or a cfg."
[conn profile-id team-id]
(let [perms (get-permissions conn profile-id team-id)]
(when-not (:is-admin perms)
(ex/raise :type :validation
:code :insufficient-permissions))
perms))
(defn decode-row
[{:keys [features subscription] :as row}]
(when row
@ -489,8 +500,9 @@
{::doc/added "1.17"
::sm/params schema:get-team-invitations}
[{:keys [::db/pool] :as cfg} {:keys [::rpc/profile-id team-id]}]
(check-invitation-permissions cfg profile-id team-id)
(dm/with-open [conn (db/open pool)]
(check-read-permissions! cfg profile-id team-id)
(get-team-invitations conn team-id)))
@ -544,7 +556,8 @@
(nitrate/assert-membership cfg profile-id organization-id)
(let [organization-perms (nitrate/call cfg :get-organization-permissions
{:organization-id organization-id})]
{:organization-id organization-id
::rpc/profile-id profile-id})]
(if (nil? organization-perms)
(ex/raise :type :validation
:code :not-allowed
@ -892,8 +905,7 @@
can-delete?
(if in-organization?
(cto/allowed? :delete-team
{:organization-perms {:owner-id (dm/get-in team [:organization :owner-id])
:permissions (dm/get-in team [:organization :permissions])}
{:organization-perms organization
:profile-id profile-id
:team-perms perms})
(boolean (:is-owner perms)))]
@ -1102,8 +1114,9 @@
[cfg {:keys [::rpc/profile-id file] :as params}]
;; Validate incoming mime type
(media.v/validate-media-type! file #{"image/jpeg" "image/png" "image/webp"})
(media.v/validate-media-size! file)
(media.v/check-upload-path file)
(media.v/check-media-type file #{"image/jpeg" "image/png" "image/webp"})
(media.v/check-media-size file)
(update-team-photo cfg (assoc params :profile-id profile-id)))
(defn update-team-photo

189
backend/src/app/rpc/commands/teams_invitations.clj
View File

@ -291,24 +291,24 @@
(not recent?))
(if organization
(when (contains? cf/flags :admin-console)
(eml/send cfg {::eml/reuse-conn true
::eml/factory eml/invite-to-organization
:public-uri (cf/get :public-uri)
:to email
:invited-by (:fullname profile)
:user-name (:fullname member)
:organization organization
:token itoken
:extra-data ptoken}))
(eml/send cfg {::eml/reuse-conn true
::eml/factory eml/invite-to-team
:public-uri (cf/get :public-uri)
:to email
:invited-by (:fullname profile)
:team (:name team)
:organization (:organization team)
:token itoken
:extra-data ptoken})))
(eml/send-template cfg {::eml/reuse-conn true
::eml/template "invite-to-organization"
:public-uri (cf/get :public-uri)
:to email
:invited-by (:fullname profile)
:user-name (:fullname member)
:organization organization
:token itoken
:extra-data ptoken}))
(eml/send-template cfg {::eml/reuse-conn true
::eml/template "invite-to-team"
:public-uri (cf/get :public-uri)
:to email
:invited-by (:fullname profile)
:team (:name team)
:organization (:organization team)
:token itoken
:extra-data ptoken})))
itoken)))))
@ -355,13 +355,13 @@
(db/delete! conn :team-invitation
{:team-id team-id :email-to (:email member)})
(eml/send cfg {::eml/reuse-conn true
::eml/factory eml/join-team
:public-uri (cf/get :public-uri)
:to (:email member)
:invited-by (:fullname profile)
:team (:name team)
:team-id (:id team)})))
(eml/send-template cfg {::eml/reuse-conn true
::eml/template "join-team"
:public-uri (cf/get :public-uri)
:to (:email member)
:invited-by (:fullname profile)
:team (:name team)
:team-id (:id team)})))
(def ^:private sql:valid-access-request-profiles
"SELECT p.id, p.email, p.is_blocked
@ -382,13 +382,12 @@
"Unified function to handle both create and resend team invitations.
Accepts either:
- emails (set) + role (single role for all emails)
- invitations (vector of {:email :role} maps)"
- invitations (vector of {:email :role} maps)
`team` must already carry its organization info when nitrate is
active (see `nitrate/add-organization-info-to-team-strict`)."
[{:keys [::db/conn] :as cfg} {:keys [profile team role emails invitations] :as params}]
(let [;; Enrich team with organization info once for all invitations when nitrate is active
team (if (contains? cf/flags :admin-console)
(nitrate/add-organization-info-to-team cfg team {})
team)
organization (:organization team)
(let [organization (:organization team)
organization-id (:id organization)
restricted? (and organization-id (not (cto/allowed? :add-anybody-to-team {:organization-perms organization})))
all-organization-member-ids
@ -485,11 +484,13 @@
::doc/module :teams
::sm/params schema:create-team-invitations}
[cfg {:keys [::rpc/profile-id team-id role emails] :as params}]
(let [perms (teams/get-permissions cfg profile-id team-id)
(let [perms (teams/check-invitation-permissions cfg profile-id team-id)
profile (db/get-by-id cfg :profile profile-id)
team (db/get-by-id cfg :team team-id)
;; Strict lookup: a failed Nitrate call must not skip the
;; organization policy below by looking like a team without one.
team-with-org (when (contains? cf/flags :admin-console)
(nitrate/add-organization-info-to-team cfg team {}))
(nitrate/add-organization-info-to-team-strict cfg team {::rpc/profile-id profile-id}))
organization (:organization team-with-org)
;; Determine which format is being used
using-emails-format? (and emails role)
@ -502,15 +503,10 @@
(count emails)
(count (:invitations params)))]
(when-not (:is-admin perms)
(ex/raise :type :validation
:code :insufficient-permissions))
(when (and (contains? cf/flags :admin-console)
organization
(not (cto/allowed? :send-invitations
{:organization-perms {:owner-id (:owner-id organization)
:permissions (:permissions organization)}
{:organization-perms organization
:profile-id profile-id
:team-perms perms})))
(ex/raise :type :validation
@ -540,7 +536,9 @@
;; Check if the current profile is allowed to send emails
(teams/check-profile-muted cfg profile)
(let [team (db/get-by-id cfg :team team-id)
(let [;; Reuse the strict organization lookup made for the policy
;; check above, so every rule sees the same Nitrate answer.
team (or team-with-org team)
;; NOTE: Is important pass RPC method params down to the
;; `create-team-invitations` because it uses the implicit
;; RPC properties from params for fill necessary data on
@ -631,26 +629,43 @@
::doc/module :teams
::sm/params schema:get-team-invitation-token}
[{:keys [::db/pool] :as cfg} {:keys [::rpc/profile-id team-id email] :as params}]
(teams/check-edition-permissions! cfg profile-id team-id)
(let [email (profile/clean-email email)
;; The token is a bearer credential that accepts the invitation with the
;; role of the invitation row, so it takes the same permissions as the
;; rest of the invitation commands.
(let [perms (teams/check-invitation-permissions cfg profile-id team-id)
email (profile/clean-email email)
invit (-> (db/get pool :team-invitation
{:team-id team-id
:email-to email})
(update :role keyword))
(update :role keyword))]
member (profile/get-profile-by-email pool (:email-to invit))
token (create-invitation-token cfg {:team-id (:team-id invit)
;; The inviter is the creator of
;; the invitation row, not the
;; profile requesting the token.
:profile-id (or (:created-by invit)
profile-id)
:valid-until (:valid-until invit)
:role (:role invit)
:member-id (:id member)
:member-email (or (:email member)
(profile/clean-email (:email-to invit)))})]
{:token token}))
;; An admin cannot create nor promote an owner invitation, so it
;; cannot mint its token either: only the team owner can.
(when (and (= :owner (:role invit))
(not (:is-owner perms)))
(ex/raise :type :validation
:code :cant-promote-to-owner
:hint "only the team owner can get an owner invitation token"))
(let [member (profile/get-profile-by-email pool (:email-to invit))
token (create-invitation-token cfg {:team-id (:team-id invit)
;; The inviter is the creator of
;; the invitation row, not the
;; profile requesting the token.
:profile-id (or (:created-by invit)
profile-id)
:valid-until (:valid-until invit)
:role (:role invit)
:member-id (:id member)
:member-email (or (:email member)
(profile/clean-email (:email-to invit)))})]
;; The event is named as an action and carries the granted role, so
;; the minting of the token is traceable in the audit log.
(rph/with-meta
{:token token}
{::audit/name "create-team-invitation-token"
::audit/props {:role (:role invit)
:invitation-id (:id invit)}}))))
;; --- Mutation: Update invitation role
@ -666,11 +681,7 @@
::sm/params schema:update-team-invitation-role
::db/transaction true}
[{:keys [::db/conn]} {:keys [::rpc/profile-id team-id email role] :as params}]
(let [perms (teams/get-permissions conn profile-id team-id)]
(when-not (:is-admin perms)
(ex/raise :type :validation
:code :insufficient-permissions))
(let [perms (teams/check-invitation-permissions conn profile-id team-id)]
;; Don't allow promote to owner to admin users.
(when (and (not (:is-owner perms)) (= role :owner))
@ -695,17 +706,13 @@
::sm/params schema:delete-team-invition
::db/transaction true}
[{:keys [::db/conn]} {:keys [::rpc/profile-id team-id email] :as params}]
(let [perms (teams/get-permissions conn profile-id team-id)]
(teams/check-invitation-permissions conn profile-id team-id)
(when-not (:is-admin perms)
(ex/raise :type :validation
:code :insufficient-permissions))
(let [invitation (db/delete! conn :team-invitation
{:team-id team-id
:email-to (profile/clean-email email)}
{::db/return-keys true})]
(rph/wrap nil {::audit/props {:invitation-id (:id invitation)}}))))
(let [invitation (db/delete! conn :team-invitation
{:team-id team-id
:email-to (profile/clean-email email)}
{::db/return-keys true})]
(rph/wrap nil {::audit/props {:invitation-id (:id invitation)}})))
;; --- Mutation: Request Team Invitation
@ -809,30 +816,30 @@
(teams/check-email-spam conn (:email team-owner) true)
(let [request (upsert-team-access-request cfg team-id profile-id)
factory (cond
(and (some? file) (:is-default team) is-viewer)
eml/request-file-access-yourpenpot-view
template (cond
(and (some? file) (:is-default team) is-viewer)
"request-file-access-yourpenpot-view"
(and (some? file) (:is-default team))
eml/request-file-access-yourpenpot
(and (some? file) (:is-default team))
"request-file-access-yourpenpot"
(some? file)
eml/request-file-access
(some? file)
"request-file-access"
:else
eml/request-team-access)]
:else
"request-team-access")]
(eml/send cfg {::eml/reuse-conn true
::eml/factory factory
:public-uri (cf/get :public-uri)
:to (:email team-owner)
:requested-by (:fullname requester)
:requested-by-email (:email requester)
:team-name (:name team)
:team-id team-id
:file-name (:name file)
:file-id file-id
:page-id (:page-id file)})
(eml/send-template cfg {::eml/reuse-conn true
::eml/template template
:public-uri (cf/get :public-uri)
:to (:email team-owner)
:requested-by (:fullname requester)
:requested-by-email (:email requester)
:team-name (:name team)
:team-id team-id
:file-name (:name file)
:file-id file-id
:page-id (:page-id file)})
(with-meta {:request request}
{::audit/props {:request 1}}))))

10
backend/src/app/rpc/management/exporter.clj
View File

@ -28,13 +28,19 @@
schema:upload-tempfile-result
[:map {:title "upload-templfile-result"}])
(def ^:private allowed-shared-key-ids
"The exporter uploads its artifacts; the media worker uploads its results."
#{:exporter :media-processor})
(sv/defmethod ::upload-tempfile
{::doc/added "2.12"
::sm/params schema:upload-tempfile-params
::rpc/multipart-only-params [:content]
::sm/result schema:upload-tempfile-result}
::sm/result schema:upload-tempfile-result
::rpc/allowed-shared-keys allowed-shared-key-ids}
[cfg {:keys [::rpc/profile-id content]}]
(media.v/validate-media-type! content cm/tempfile-types)
(media.v/check-upload-path content)
(media.v/check-media-type content cm/tempfile-types)
(let [storage (sto/resolve cfg)
hash (sto/calculate-hash (:path content))
data (-> (sto/content (:path content))

7
backend/src/app/rpc/management/files.clj
View File

@ -28,6 +28,10 @@
(def projection-version 1)
(def ^:private allowed-shared-key-ids
"Only the nexus file-security pipeline reads file projections."
#{:nexus})
;; Bounds for the data that crosses to nexus on every sweep. The
;; projection stays small by construction; anything past the caps sets
;; `:truncated?` instead of growing the payload.
@ -220,7 +224,8 @@
{::doc/added "2.20"
::sm/params schema:get-file-for-analysis-params
::sm/result schema:get-file-for-analysis-result
::rpc/auth false}
::rpc/auth false
::rpc/allowed-shared-keys allowed-shared-key-ids}
[cfg {:keys [file-id]}]
;; `realize?` loads the whole object tree eagerly: the projection
;; walks every shape, so lazy pointers (no `load-fn` bound here)

27
backend/src/app/rpc/management/jobs.clj
View File

@ -10,11 +10,12 @@
the ledger. The methods are family-agnostic (they operate on any row
of the `job` table) and use the plain JSON schemas of `app.jobs`.
Auth: every method declares ::rpc/auth false because these endpoints
do not require a profile. Actual authentication is enforced by the
management route resolver (resolve-management-methods in app.rpc),
which mandates a valid shared-key before dispatching to these methods.
If the route resolver changes, these methods must be updated accordingly."
Auth: these endpoints take no profile, so every method declares
::rpc/auth false. The route applies mw/shared-key-auth
(app.http.middleware) to authenticate the `<id> <secret>` header, and
wrap-management applies wrap-shared-key-authorization (app.rpc) to
check the key id against ::rpc/allowed-shared-keys. If the route
middleware or wrap-management changes, revisit these methods."
(:require
[app.common.exceptions :as ex]
[app.common.schema :as sm]
@ -24,6 +25,10 @@
[app.rpc.doc :as doc]
[app.util.services :as sv]))
(def ^:private allowed-shared-key-ids
"Only the exporter worker claims and settles durable jobs today."
#{:exporter})
;; ---- RPC METHOD: CLAIM-JOB
(def ^:private schema:claim-job-params
@ -42,7 +47,8 @@
{::doc/added "2.20"
::sm/params schema:claim-job-params
::sm/result schema:claim-job-result
::rpc/auth false} ;; shared-key enforced by route resolver
::rpc/auth false
::rpc/allowed-shared-keys allowed-shared-key-ids}
[cfg {:keys [job-id scheduled-at]}]
(let [row (jobs/get-job cfg job-id)]
;; Race condition: if the job is claimed between get-job and claim! by
@ -74,7 +80,8 @@
{::doc/added "2.20"
::sm/params schema:report-job-progress-params
::sm/result schema:report-job-progress-result
::rpc/auth false} ;; shared-key enforced by route resolver
::rpc/auth false
::rpc/allowed-shared-keys allowed-shared-key-ids}
[cfg {:keys [job-id progress]}]
;; A lost race (row already terminal) reports :skip so the worker
;; stops retrying a report that can never land, mirroring claim-job.
@ -130,7 +137,8 @@
{::doc/added "2.20"
::sm/params schema:complete-job-params
::sm/result schema:complete-job-result
::rpc/auth false} ;; shared-key enforced by route resolver
::rpc/auth false
::rpc/allowed-shared-keys allowed-shared-key-ids}
[cfg {:keys [job-id result resource-id]}]
(if (pos? (jobs/complete cfg
:job-id job-id
@ -156,7 +164,8 @@
{::doc/added "2.20"
::sm/params schema:fail-job-params
::sm/result schema:fail-job-result
::rpc/auth false} ;; shared-key enforced by route resolver
::rpc/auth false
::rpc/allowed-shared-keys allowed-shared-key-ids}
[cfg {:keys [job-id error]}]
(if (pos? (jobs/fail cfg job-id error))
{:action :run}

132
backend/src/app/rpc/management/nitrate.clj
View File

@ -27,6 +27,7 @@
[app.http.session :as session]
[app.jobs :as jobs]
[app.loggers.audit :as audit]
[app.media.svg :as svg]
[app.media.validation :as media.v]
[app.nitrate :as nitrate]
[app.rpc :as rpc]
@ -46,6 +47,10 @@
[cuerdas.core :as str]))
(def ^:private allowed-shared-key-ids
"The Nitrate admin-console is the only caller of this API."
#{:admin-console})
(defn- profile-to-map [profile]
{:id (:id profile)
:name (:fullname profile)
@ -58,6 +63,7 @@
(sv/defmethod ::authenticate
"Authenticate the current user"
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params [:map]
::sm/result schema:profile
::nitrate/sso false}
@ -105,6 +111,7 @@
(sv/defmethod ::get-penpot-version
"Get the current Penpot version"
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params [:map]
::sm/result schema:get-penpot-version-result
::rpc/auth false}
@ -124,6 +131,7 @@
(sv/defmethod ::get-teams
"List teams for which current user is owner"
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params [:map]
::sm/result schema:get-teams-result
::nitrate/sso false}
@ -135,6 +143,7 @@
(sv/defmethod ::get-member-teams
"List all active teams where the current user is a member"
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params [:map]
::sm/result schema:get-member-teams-result
::nitrate/sso false}
@ -151,6 +160,7 @@
(sv/defmethod ::update-profile-theme
"Update the current user's theme"
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params schema:update-profile-theme
::db/transaction true
::nitrate/sso false}
@ -171,6 +181,7 @@
(sv/defmethod ::update-profile-props
"Merge writable properties into the current user's profile"
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params schema:update-profile-props
::db/transaction true
::nitrate/sso false}
@ -193,21 +204,25 @@
Accepts an optional previous-id to mark the old logo for garbage
collection when replacing an existing one."
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params schema:upload-organization-logo
::rpc/multipart-only-params [:content]
::sm/result schema:upload-organization-logo-result
::nitrate/sso false}
[{:keys [::sto/storage]} {:keys [content organization-id previous-id]}]
(media.v/validate-media-type! content cm/image-types)
(media.v/check-upload-path content)
(media.v/check-media-type content cm/image-types)
(media.v/check-media-size content)
(when previous-id
(sto/touch-object! storage previous-id))
(let [hash (sto/calculate-hash (:path content))
data (-> (sto/content (:path content))
(let [{:keys [path mtype]} (svg/sanitize-svg-upload content)
hash (sto/calculate-hash path)
data (-> (sto/content path)
(sto/wrap-with-hash hash))
obj (sto/put-object! storage {::sto/content data
::sto/deduplicate? true
:bucket "organization"
:content-type (:mtype content)
:content-type mtype
:organization-id organization-id})]
{:id (:id obj)}))
@ -225,6 +240,7 @@
(sv/defmethod ::notify-team-change
"Notify to Penpot a team change from nitrate"
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params schema:notify-team-change
::rpc/auth false}
[cfg team]
@ -245,6 +261,7 @@
(sv/defmethod ::notify-user-added-to-organization
"Notify to Penpot that an user has joined an organization from nitrate"
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params schema:notify-user-added-to-organization
::rpc/auth false}
[cfg {:keys [profile-id organization-id]}]
@ -276,6 +293,7 @@
(sv/defmethod ::get-managed-profiles
"List profiles that belong to teams for which current user is owner"
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params [:map]
::sm/result schema:managed-profile-result
::nitrate/sso false}
@ -316,6 +334,7 @@
(sv/defmethod ::get-teams-summary
"Get summary information for a list of teams"
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params schema:get-teams-summary-params
::sm/result schema:get-teams-summary-result
::nitrate/sso false}
@ -437,6 +456,7 @@ RETURNING id, deleted_at;")
"For a deleted organization, preserve organization teams and only prefix or delete
imported Personal Projects before notifying connected users."
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params schema:notify-organization-deletion
::rpc/auth false}
[cfg {:keys [organization-id]}]
@ -457,6 +477,7 @@ RETURNING id, deleted_at;")
"For a given user, find all owned organizations and apply the deleted-organization
transfer rules to their imported Personal Projects teams."
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params schema:notify-user-organizations-deletion
::nitrate/sso false}
[cfg {:keys [profile-id]}]
@ -483,6 +504,7 @@ RETURNING id, deleted_at;")
(sv/defmethod ::get-profile-by-email
"Get profile by email"
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params [:map [:email ::sm/email]]
::sm/result schema:profile
::nitrate/sso false}
@ -507,6 +529,7 @@ RETURNING id, deleted_at;")
(sv/defmethod ::get-profile-by-id
"Get profile by email"
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params [:map [:id ::sm/uuid]]
::sm/result schema:profile
::nitrate/sso false}
@ -520,6 +543,33 @@ RETURNING id, deleted_at;")
(profile-to-map profile)))
;; ---- API: get-profiles-by-ids
(def ^:private sql:get-profiles-by-ids
"SELECT DISTINCT id, fullname, email, photo_id
FROM profile
WHERE id = ANY(?)
AND deleted_at IS NULL;")
(def ^:private schema:get-profiles-by-ids-params
[:map [:ids [:vector ::sm/uuid]]])
(sv/defmethod ::get-profiles-by-ids
"Get multiple profiles by id. Unknown ids are simply absent from the result."
{::doc/added "2.20"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params schema:get-profiles-by-ids-params
::sm/result [:vector schema:profile]
::nitrate/sso false}
[cfg {:keys [ids]}]
(if (empty? ids)
[]
(db/run! cfg (fn [{:keys [::db/conn]}]
(let [ids-array (db/create-array conn "uuid" ids)]
(->> (db/exec! conn [sql:get-profiles-by-ids ids-array])
(mapv profile-to-map)))))))
;; ---- API: get-organization-member-team-counts
(def ^:private sql:get-organization-member-team-counts
@ -542,6 +592,7 @@ RETURNING id, deleted_at;")
(sv/defmethod ::get-organization-member-team-counts
"Get the number of non-default teams each profile belongs to within a set of teams."
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params schema:get-organization-member-team-counts-params
::sm/result schema:get-organization-member-team-counts-result
::rpc/auth false}
@ -562,6 +613,47 @@ RETURNING id, deleted_at;")
(db/exec! conn [sql:get-organization-member-team-counts ids-array])))))))
;; ---- API: get-organization-team-roles
(def ^:private sql:get-organization-team-roles
"SELECT t.id AS team_id, t.name AS team_name, tpr.profile_id,
CASE WHEN tpr.is_owner THEN 'owner' ELSE 'admin' END AS role
FROM team_profile_rel AS tpr
JOIN team AS t ON t.id = tpr.team_id
JOIN profile AS p ON p.id = tpr.profile_id
WHERE t.id = ANY(?)
AND t.deleted_at IS NULL
AND t.is_default IS FALSE
AND p.deleted_at IS NULL
AND (tpr.is_owner OR tpr.is_admin);")
(def ^:private schema:get-organization-team-roles-params
[:map [:team-ids [:vector ::sm/uuid]]])
(def ^:private schema:get-organization-team-roles-result
[:vector [:map
[:team-id ::sm/uuid]
[:team-name ::sm/text]
[:profile-id ::sm/uuid]
[:role [:enum :owner :admin]]]])
(sv/defmethod ::get-organization-team-roles
"Get the owner and admin profiles of each non-default team within a set of teams."
{::doc/added "2.20"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params schema:get-organization-team-roles-params
::sm/result schema:get-organization-team-roles-result
::rpc/auth false}
[cfg {:keys [team-ids]}]
(if (empty? team-ids)
[]
(db/run! cfg (fn [{:keys [::db/conn]}]
(let [ids-array (db/create-array conn "uuid" team-ids)]
(->> (db/exec! conn [sql:get-organization-team-roles ids-array])
(mapv (fn [{:keys [role] :as row}]
(assoc row :role (keyword role))))))))))
;; ---- API: get-organization-member-teams
(def ^:private sql:get-organization-member-teams
@ -594,6 +686,7 @@ RETURNING id, deleted_at;")
"Get the non-default teams a profile belongs to within a set of teams,
with the profile's role in each one."
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params schema:get-organization-member-teams-params
::sm/result schema:get-organization-member-teams-result
::nitrate/sso false}
@ -631,6 +724,7 @@ RETURNING id, deleted_at;")
(sv/defmethod ::invite-to-organization
"Invite to organization"
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params [:map
[:email ::sm/email]
[:organization cto/schema:organization-with-avatar]]
@ -663,6 +757,7 @@ RETURNING id, deleted_at;")
(sv/defmethod ::get-organization-invitations
"Get valid invitations for an organization, returning at most one invitation per email."
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params schema:get-organization-invitations-params
::sm/result schema:get-organization-invitations-result
::nitrate/sso false}
@ -691,6 +786,7 @@ RETURNING id, deleted_at;")
(sv/defmethod ::delete-organization-invitations
"Delete all invitations for one email in an organization scope (organization + organization teams)."
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params schema:delete-organization-invitations-params
::nitrate/sso false}
[cfg {:keys [organization-id email]}]
@ -718,6 +814,7 @@ RETURNING id, deleted_at;")
Called from Nitrate when an organization is about to be deleted, so users that click
their invitation token hit the existing invalid-token landing page."
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params schema:delete-all-organization-invitations-params
::rpc/auth false}
[cfg {:keys [organization-id]}]
@ -756,6 +853,7 @@ RETURNING id, deleted_at;")
(sv/defmethod ::remove-from-organization
"Remove an user from an organization"
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params [:map
[:profile-id ::sm/uuid]
[:organization-id ::sm/uuid]
@ -801,6 +899,7 @@ RETURNING id, deleted_at;")
"Get a summary of the teams that would be deleted, transferred, or exited
if the user were removed from the organization"
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params [:map
[:profile-id ::sm/uuid]
[:organization-id ::sm/uuid]
@ -836,6 +935,7 @@ RETURNING id, deleted_at;")
(sv/defmethod ::send-renewal-email
"Send an Enterprise subscription renewal notice email to a user."
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params schema:send-renewal-email-params
::rpc/auth false}
[cfg {:keys [profile-id user-email user-name renewal-date estimated-amount organizations]}]
@ -849,14 +949,14 @@ RETURNING id, deleted_at;")
(:fullname (profile/get-profile cfg profile-id))
(str/trim user-name))]
(db/tx-run! cfg (fn [tx-cfg]
(eml/send tx-cfg {::eml/reuse-conn true
::eml/factory eml/renewal-notice
:public-uri (cf/get :public-uri)
:to user-email
:user-name user-name
:renewal-date renewal-date
:estimated-amount amount-str
:organizations organizations}))))
(eml/send-template tx-cfg {::eml/reuse-conn true
::eml/template "renewal-notice"
:public-uri (cf/get :public-uri)
:to user-email
:user-name user-name
:renewal-date renewal-date
:estimated-amount amount-str
:organizations organizations}))))
nil)
;; API: exists-organization-team-invitations-for-non-members /
@ -912,6 +1012,7 @@ RETURNING id, deleted_at;")
(sv/defmethod ::exists-organization-team-invitations-for-non-members
"Return if there are any team invitations for emails that are not organization members."
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params schema:organization-team-invitations-for-non-members-params
::sm/result schema:exists-organization-team-invitations-for-non-members-result
::nitrate/sso false}
@ -922,6 +1023,7 @@ RETURNING id, deleted_at;")
(sv/defmethod ::delete-organization-team-invitations-for-non-members
"Delete team invitations for emails that are not organization members."
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params schema:organization-team-invitations-for-non-members-params
::db/transaction true
::nitrate/sso false}
@ -954,6 +1056,7 @@ RETURNING id, deleted_at;")
events)"
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::audit/skip true
::sm/params schema:push-audit-events-params
::rpc/auth false}
@ -1061,6 +1164,7 @@ RETURNING id, deleted_at;")
"Get detailed information for all non-deleted teams in an organization,
including owner info and project/file/member counts."
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params schema:get-teams-detail-params
::sm/result schema:get-teams-detail-result
::nitrate/sso false}
@ -1091,6 +1195,7 @@ RETURNING id, deleted_at;")
(customer-configured), so it is checked against the SSRF blocklist before
any outbound request is attempted."
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params cto/schema:nitrate-sso
::sm/result schema:check-organization-sso-result
::rpc/auth false}
@ -1104,6 +1209,7 @@ RETURNING id, deleted_at;")
(sv/defmethod ::notify-organization-sso-change
"Nitrate notifies that an organization sso values have changed"
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params [:map
[:organization-id ::sm/uuid]
[:updated-props ::sm/boolean]
@ -1163,6 +1269,7 @@ RETURNING id, deleted_at;")
email allow-list. Requires the `admin-console-bulk-create-profiles` flag, disabled
by default so it is only available on test environments."
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params schema:bulk-create-profiles-params
::sm/result schema:bulk-create-profiles-result
::rpc/auth false}
@ -1197,6 +1304,7 @@ RETURNING id, deleted_at;")
(sv/defmethod ::get-air-gapped
"Returns whether this Penpot instance runs in air-gapped mode."
{::doc/added "2.18"
::rpc/allowed-shared-keys allowed-shared-key-ids
::sm/params [:map]
::sm/result schema:get-air-gapped-result
::rpc/auth false}

10
backend/src/app/rpc/nitrate/emails_helper.clj
View File

@ -55,11 +55,11 @@
[{:keys [::db/conn] :as cfg} organization-name {:keys [email profile]}]
(when (or (nil? profile)
(eml/allow-send-emails? conn profile))
(eml/send cfg {::eml/reuse-conn true
::eml/factory eml/organization-setup-sso
:public-uri (cf/get :public-uri)
:to email
:organization-name organization-name})))
(eml/send-template cfg {::eml/reuse-conn true
::eml/template "organization-setup-sso"
:public-uri (cf/get :public-uri)
:to email
:organization-name organization-name})))
(defn- get-organization-sso-notify-recipients
"Unique organization members and pending organization/team invitees for SSO activation emails."

141
backend/src/app/srepl/procs/fdata_storage.clj
View File

@ -1,141 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.srepl.procs.fdata-storage
(:require
[app.common.logging :as l]
[app.db :as db]))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; SNAPSHOTS
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(def sql:get-unmigrated-snapshots
"SELECT fc.id, fc.file_id
FROM file_change AS fc
WHERE fc.data IS NOT NULL
AND fc.label IS NOT NULL
ORDER BY fc.id ASC")
(def sql:get-migrated-snapshots
"SELECT f.id, f.file_id
FROM file_data AS f
WHERE f.data IS NOT NULL
AND f.type = 'snapshot'
AND f.id != f.file_id
ORDER BY f.id ASC")
(defn migrate-snapshot-to-storage
"Migrate the current existing files to store data in new storage
tables."
{:query sql:get-unmigrated-snapshots}
[{:keys [::db/conn]} {:keys [id file-id]} & {:as options}]
(let [{:keys [id file-id data created-at updated-at]}
(db/get* conn :file-change {:id id :file-id file-id}
::db/for-update true
::db/remove-deleted false)]
(when data
(l/inf :hint "migrating snapshot" :file-id (str file-id) :id (str id))
(db/update! conn :file-change
{:data nil}
{:id id :file-id file-id}
{::db/return-keys false})
(db/insert! conn :file-data
{:backend "db"
:metadata nil
:type "snapshot"
:data data
:created-at created-at
:modified-at updated-at
:file-id file-id
:id id}
{::db/return-keys false}))))
(defn rollback-snapshot-from-storage
"Migrate back to the file table storage."
{:query sql:get-unmigrated-snapshots}
[{:keys [::db/conn]} {:keys [id file-id]} & {:as opts}]
(when-let [{:keys [id file-id data]}
(db/get* conn :file-data {:id id :file-id file-id :type "snapshot"}
::db/for-update true
::db/remove-deleted false)]
(l/inf :hint "rollback snapshot" :file-id (str file-id) :id (str id))
(db/update! conn :file-change
{:data data}
{:id id :file-id file-id}
{::db/return-keys false})
(db/delete! conn :file-data
{:id id :file-id file-id :type "snapshot"}
{::db/return-keys false})))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; FILES
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(def sql:get-unmigrated-files
"SELECT f.id
FROM file AS f
WHERE f.data IS NOT NULL
ORDER BY f.modified_at ASC")
(def sql:get-migrated-files
"SELECT f.id, f.file_id
FROM file_data AS f
WHERE f.data IS NOT NULL
AND f.id = f.file_id
ORDER BY f.id ASC")
(defn migrate-file-to-storage
"Migrate the current existing files to store data in new storage
tables."
{:query sql:get-unmigrated-files}
[{:keys [::db/conn] :as cfg} {:keys [id]} & {:as opts}]
(let [{:keys [id data created-at modified-at]}
(db/get* conn :file {:id id}
::db/for-update true
::db/remove-deleted false)]
(when data
(l/inf :hint "migrating file" :file-id (str id))
(db/update! conn :file {:data nil} {:id id} ::db/return-keys false)
(db/insert! conn :file-data
{:backend "db"
:metadata nil
:type "main"
:data data
:created-at created-at
:modified-at modified-at
:file-id id
:id id}
{::db/return-keys false}))
(let [snapshots-sql
(str "WITH snapshots AS (" sql:get-unmigrated-snapshots ") "
"SELECT s.* FROM snapshots AS s WHERE s.file_id = ?")]
(run! (fn [params]
(migrate-snapshot-to-storage cfg params opts))
(db/plan cfg [snapshots-sql id])))))
(defn rollback-file-from-storage
"Migrate back to the file table storage."
{:query sql:get-migrated-files}
[{:keys [::db/conn] :as cfg} {:keys [id]} & {:as opts}]
(when-let [{:keys [id data]}
(db/get* conn :file-data {:id id :file-id id :type "main"}
::db/for-update true
::db/remove-deleted false)]
(l/inf :hint "rollback file" :file-id (str id))
(db/update! conn :file {:data data} {:id id} ::db/return-keys false)
(db/delete! conn :file-data {:file-id id :id id :type "main"} ::db/return-keys false)
(let [snapshots-sql
(str "WITH snapshots AS (" sql:get-migrated-snapshots ") "
"SELECT s.* FROM snapshots AS s WHERE s.file_id = ?")]
(run! (fn [params]
(rollback-snapshot-from-storage cfg params opts))
(db/plan cfg [snapshots-sql id])))))

44
backend/test/backend_tests/auth_ldap_test.clj
View File

@ -7,9 +7,53 @@
(ns backend-tests.auth-ldap-test
(:require
[app.auth.ldap :as ldap-auth]
[app.common.time :as ct]
[app.rpc.commands.ldap :as ldap-rpc]
[app.tokens :as tokens]
[backend-tests.helpers :as th]
[clj-ldap.client :as ldap]
[clojure.test :as t]))
(t/use-fixtures :once th/state-init)
(t/use-fixtures :each th/database-reset)
;; --- login-with-ldap: the invitation branch returns the whole profile
(defn- mock-authenticate
[email fullname]
(fn [_cfg _params]
{:email email :fullname fullname :backend "ldap"}))
(t/deftest login-with-ldap-invitation-returns-full-profile
(t/testing "the invitation branch returns the profile, not only the token"
(let [email "ldap-invitation@example.com"
system (assoc th/*system* ::ldap-auth/provider {:host "localhost"})
token (tokens/generate system {:iss :team-invitation
:exp (ct/in-future {:hours 1})})]
(with-redefs [ldap-auth/authenticate (mock-authenticate email "Invited Person")]
(let [result (ldap-rpc/sm$login-with-ldap
system
{:email email
:password "secret"
:invitation-token token})]
(t/is (uuid? (:id result)))
(t/is (= email (:email result)))
(t/is (= "ldap" (:auth-backend result)))
(t/is (string? (:invitation-token result)))
(t/is (not= token (:invitation-token result))))))))
(t/deftest login-with-ldap-without-invitation-returns-profile
(t/testing "the regular branch keeps returning the profile without a token"
(let [email "ldap-plain@example.com"
system (assoc th/*system* ::ldap-auth/provider {:host "localhost"})]
(with-redefs [ldap-auth/authenticate (mock-authenticate email "Plain Person")]
(let [result (ldap-rpc/sm$login-with-ldap system {:email email
:password "secret"})]
(t/is (uuid? (:id result)))
(t/is (= email (:email result)))
(t/is (= "ldap" (:auth-backend result)))
(t/is (nil? (:invitation-token result))))))))
;; --- search-user: filter must be escaped (RED: currently not escaped)
(t/deftest search-user-escapes-email-in-filter

51
backend/test/backend_tests/auth_oidc_test.clj
View File

@ -418,6 +418,11 @@
[result]
(get-in result [::yres/headers "location"]))
(defn- location-token
"Extract the `token` query param from a redirect location."
[location]
(second (re-find #"[?&]token=([^&]+)" location)))
(t/deftest callback-param-error-redirects-to-login
(let [cfg (dissoc base-cfg :app.email/blacklist :app.email/whitelist)
request {:params {:error "access_denied"}}]
@ -507,6 +512,52 @@
(t/is (.contains loc "http://localhost:3449?screen=auth-verify-token&"))
(t/is (.contains loc "token=")))))))
(t/deftest callback-with-invitation-mints-auth-token-with-invitation-claim
(let [cfg (dissoc base-cfg :app.email/blacklist :app.email/whitelist)
invitation (tokens/generate cfg {:iss :team-invitation
:exp (ct/in-future {:hours 1})
:member-id test-profile-id
:member-email "u@e.com"})
state (make-state-token cfg {:invitation-token invitation})
request (default-request cfg :state state)]
(binding [cf/config {:public-uri "http://localhost:3449"}
cf/flags #{:registration}]
(with-redefs [app.auth.oidc/resolve-provider (constantly {:type "oidc" :id "oidc"})
app.auth.oidc/get-info (constantly {:email "u@e.com" :fullname "U"
:backend "oidc" :email-verified false
:invitation-token invitation
:props {}})
app.auth.oidc/get-profile (constantly test-profile)
app.auth.oidc/update-profile-with-info (fn [cfg profile info] profile)
app.loggers.audit/submit (constantly nil)]
(let [result (#'oidc/callback-handler cfg request)
loc (redirect-location result)
claims (tokens/decode cfg (location-token loc))]
(t/is (= :auth (:iss claims))
"the session token must be an :auth token, not the invitation one")
(t/is (= invitation (:invitation-token claims))
"the invitation token must travel inside the :auth claims")
(t/is (= test-profile-id (:profile-id claims))))))))
(t/deftest callback-without-invitation-mints-plain-auth-token
(let [cfg (dissoc base-cfg :app.email/blacklist :app.email/whitelist)
state (make-state-token cfg {})
request (default-request cfg :state state)]
(binding [cf/config {:public-uri "http://localhost:3449"}
cf/flags #{:registration}]
(with-redefs [app.auth.oidc/resolve-provider (constantly {:type "oidc" :id "oidc"})
app.auth.oidc/get-info (constantly {:email "u@e.com" :fullname "U"
:backend "oidc" :email-verified false
:props {}})
app.auth.oidc/get-profile (constantly test-profile)
app.auth.oidc/update-profile-with-info (fn [cfg profile info] profile)
app.loggers.audit/submit (constantly nil)]
(let [result (#'oidc/callback-handler cfg request)
loc (redirect-location result)
claims (tokens/decode cfg (location-token loc))]
(t/is (= :auth (:iss claims)))
(t/is (not (contains? claims :invitation-token))))))))
(t/deftest callback-gracefully-handles-unable-to-retrieve-user-info
(let [cfg (dissoc base-cfg :app.email/blacklist :app.email/whitelist)
state (make-state-token cfg {})

2
backend/test/backend_tests/binfile_svg_test.clj
View File

@ -146,7 +146,7 @@
(defn- write-temp-svg
"Write `text` to a tempfile and return `[path size]`."
[text]
(let [path (fs/create-tempfile :prefix "penpot-svg-import-test-" :suffix ".svg")
(let [path (fs/create-tempfile :dir "/tmp/penpot" :prefix "penpot-svg-import-test-" :suffix ".svg")
data (utf8bytes text)]
(spit (str path) text :encoding "UTF-8")
[path (alength data)]))

388
backend/test/backend_tests/binfile_test.clj
View File

@ -15,6 +15,7 @@
[app.common.files.changes-builder :as pcb]
[app.common.files.tokens :as cfo]
[app.common.files.validate :as cfv]
[app.common.geom.matrix :as gmt]
[app.common.thumbnails :as thc]
[app.common.time :as ct]
[app.common.types.shape :as cts]
@ -30,13 +31,16 @@
[app.storage.tmp :as tmp]
[backend-tests.helpers :as th]
[backend-tests.storage-test :as stt]
[clojure.java.io :as jio]
[clojure.data.json :as json]
[clojure.test :as t]
[cuerdas.core :as str]
[datoteka.fs :as fs]
[datoteka.io :as io])
(:import
java.io.ByteArrayInputStream
java.io.ByteArrayOutputStream
java.io.DataInputStream
java.io.DataOutputStream
java.io.OutputStreamWriter
java.io.Writer
java.util.zip.Deflater
@ -216,6 +220,63 @@
:token-radius token-radius
:token-opacity token-opacity}))
(defn- export-file
[file & {:keys [include-libraries]}]
(let [output (tmp/tempfile :suffix ".zip")]
(v3/export-files!
(cond-> (-> th/*system*
(assoc ::bfc/file-ids #{(:id file)})
(assoc ::bfc/output (io/output-stream output)))
include-libraries
(assoc ::bfc/export-type :include-libraries)))
output))
(defn- import-bundle
[profile path]
(-> th/*system*
(assoc ::bfc/project-id (:default-project-id profile))
(assoc ::bfc/team-id (:default-team-id profile))
(assoc ::bfc/profile-id (:id profile))
(assoc ::bfc/input path)
(v3/import-files!)
(:file-ids)))
(defn- read-manifest
[path]
(with-open [zip (ZipFile. (.toFile ^java.nio.file.Path path))]
(let [entry (.getEntry zip "manifest.json")]
(with-open [input (.getInputStream zip entry)]
(json/read-str (slurp input :encoding "UTF-8"))))))
(defn- tamper-manifest-relations
[src dst relations]
(with-open [zin (ZipFile. (.toFile ^java.nio.file.Path src))
zout (ZipOutputStream. (io/output-stream dst))]
(doseq [entry (enumeration-seq (.entries zin))]
(let [name (.getName ^ZipEntry entry)]
(if (= name "manifest.json")
(let [manifest (-> (slurp (.getInputStream zin entry) :encoding "UTF-8")
(json/read-str)
(assoc "relations" relations))]
(.putNextEntry zout (ZipEntry. name))
(.write zout (.getBytes (json/write-str manifest) "UTF-8"))
(.closeEntry zout))
(do
(.putNextEntry zout (ZipEntry. name))
(with-open [input (.getInputStream zin entry)]
(io/copy input zout))
(.closeEntry zout)))))))
(defn- prepare-victim-file
[n]
(let [profile (th/create-profile* n)
team (th/create-team* n {:profile-id (:id profile)})
project (th/create-project* n {:team-id (:id team)
:profile-id (:id profile)})
file (th/create-file* n {:profile-id (:id profile)
:project-id (:id project)})]
[profile file]))
(def ^:private svg-raw-page-id (uuid/custom 1 1))
(def ^:private svg-raw-root-id (uuid/custom 3 1))
(def ^:private svg-raw-child-id (uuid/custom 3 2))
@ -316,12 +377,12 @@
(defn- import-svg-attrs-asset
"Imports the `svg-attrs-camel-case.penpot` asset, a real penpot export
whose shapes carry `:svg-attrs` keys in camelCase (the format the
binary export writes), and returns the imported file."
whose shapes carry `:svg-attrs` keys in camelCase (the spelling the
binary export wrote at the time), and returns the imported file. The
v3 reader kebab-izes every key, so the shapes come back with
kebab-case keys."
[profile]
(let [input (-> "backend_tests/test_files/svg-attrs-camel-case.penpot"
io/resource
jio/file)
(let [input (th/tempfile "backend_tests/test_files/svg-attrs-camel-case.penpot")
result (-> th/*system*
(assoc ::bfc/project-id (:default-project-id profile))
(assoc ::bfc/profile-id (:id profile))
@ -330,13 +391,11 @@
(v3/import-files!))]
(bfc/get-file th/*system* (first (:file-ids result)))))
(t/deftest import-binfile-v3-preserves-camel-case-svg-attrs
(t/deftest import-binfile-v3-normalizes-svg-attrs-to-kebab
;; The json reader used by the v3 import rewrites every key of every
;; zip entry to kebab-case, and `:svg-attrs` is the one shape map
;; whose keys are camelCase react prop names. A shape exported with
;; `fillRule: "evenodd"` must not come back as `:fill-rule`, or the
;; renderer falls back to the default fill rule and the shape is
;; painted without its hole.
;; zip entry to kebab-case, and birth stores `:svg-attrs` in kebab-case
;; too, so a shape exported with `fillRule: "evenodd"` stays kebab and
;; the renderer finds the fill rule it expects.
(let [profile (th/create-profile* 1)
file (import-svg-attrs-asset profile)
shape (get-in file [:data :pages-index
@ -344,10 +403,11 @@
:objects
(uuid/uuid "ce3641bd-48c8-804c-8008-b54d35cc6f80")])]
(t/is (= {:fillRule "evenodd"}
(t/is (some? shape) "the fixture shape was found")
(t/is (= {:fill-rule "evenodd"}
(:svg-attrs shape)))))
(t/deftest import-binfile-v3-preserves-camel-case-svg-attrs-on-components
(t/deftest import-binfile-v3-normalizes-svg-attrs-to-kebab-on-components
;; Same guarantee for shapes stored inside a component: the v3 import
;; cleans those in a different code path than page shapes.
(let [profile (th/create-profile* 1)
@ -358,14 +418,132 @@
(get (uuid/uuid shape-id))
:svg-attrs))]
(t/is (= {:fillRule "evenodd"}
(t/is (= {:fill-rule "evenodd"}
(shape "fae4bc76-0cc2-8057-8008-b540912cdf78"
"fae4bc76-0cc2-8057-8008-b540912774cb")))
(t/is (= {:fillRule "nonzero"}
(t/is (= {:fill-rule "nonzero"}
(shape "fae4bc76-0cc2-8057-8008-b540912c917b"
"fae4bc76-0cc2-8057-8008-b540912774c8")))))
(def ^:private svg-defs-page-id (uuid/custom 4 1))
(def ^:private svg-defs-shape-id (uuid/custom 4 2))
(defn- prepare-svg-defs-file
"A file with an svg-raw shape carrying a filter def, the stored
kebab form birth produces (what an SVG import leaves behind)."
([profile] (prepare-svg-defs-file profile 1))
([profile idx]
(let [page-id svg-defs-page-id
shape-id svg-defs-shape-id
file (th/create-file* idx {:profile-id (:id profile)
:project-id (:default-project-id profile)
:is-shared false})]
(update-file!
:file-id (:id file)
:profile-id (:id profile)
:revn 0
:vern 0
:changes
[{:type :add-page
:name "page 1"
:id page-id}])
(update-file!
:file-id (:id file)
:profile-id (:id profile)
:revn 0
:vern 0
:changes
[{:type :add-obj
:page-id page-id
:id shape-id
:parent-id uuid/zero
:frame-id uuid/zero
:components-v2 true
:obj (cts/setup-shape
{:id shape-id
:name "svg-filter"
:frame-id uuid/zero
:parent-id uuid/zero
:type :svg-raw
:content {:tag :rect
:attrs {:x "10" :fill "red"}
:content []}
:svg-attrs {:fill-rule "evenodd"}
:svg-transform (gmt/matrix 1 0 0 1 10 20)
:svg-defs {"f1" {:tag :filter
:attrs {:id "f1"}
:content [{:tag :feGaussianBlur
:attrs {:std-deviation "2"}
:content []}]}}})}])
(dissoc file :data))))
(defn- export-import-file
"Exports `file` to a zip and imports it back, returning a map with
the export output and the imported file."
[profile file]
(let [output (tmp/tempfile :suffix ".zip")]
(v3/export-files!
(-> th/*system*
(assoc ::bfc/file-ids #{(:id file)})
(assoc ::bfc/export-type :detach-libraries)
(assoc ::bfc/output (io/output-stream output))))
(let [result (-> th/*system*
(assoc ::bfc/project-id (:default-project-id profile))
(assoc ::bfc/profile-id (:id profile))
(assoc ::bfc/team-id (:default-team-id profile))
(assoc ::bfc/input output)
(v3/import-files!))]
{:output output
:imported (bfc/get-file th/*system* (first (:file-ids result)))})))
(defn- svg-defs-shape
[file]
(get-in file [:data :pages-index svg-defs-page-id :objects svg-defs-shape-id]))
(t/deftest import-binfile-v3-normalizes-svg-defs-to-kebab
;; Def node attrs converge to the stored kebab convention through an
;; export/import cycle: tags come back as keywords (the schema decoder
;; restores them from the stored strings), attr keys stay kebab, outer
;; def keys come back as strings (the decoder prefers the string
;; branch), and the string ids inside survive byte-identical.
(let [profile (th/create-profile* 1)
file (prepare-svg-defs-file profile)
{:keys [imported]} (export-import-file profile file)
shape (svg-defs-shape imported)
node (get (:svg-defs shape) "f1")]
(t/is (some? shape) "the svg shape was found")
(t/is (= {:fill-rule "evenodd"} (:svg-attrs shape)))
(t/is (some? node) "the filter def was found")
(t/is (= :filter (:tag node)) "tags decode back to keywords")
(t/is (= "f1" (get-in node [:attrs :id])) "def ids survive byte-identical")
(t/is (= "2" (get-in node [:content 0 :attrs :std-deviation])))
(t/is (gmt/matrix? (:svg-transform shape))
"svg-transform comes back as a matrix, not a plain map")
(t/is (= (gmt/matrix 1 0 0 1 10 20) (:svg-transform shape)))))
(t/deftest export-import-svg-roundtrip-converges
;; The determinism proof: export, import, export, import. The first
;; import lands on the kebab end state and the second trip leaves it
;; exactly there.
(let [profile (th/create-profile* 1)
file (prepare-svg-defs-file profile)
trip-1 (:imported (export-import-file profile file))
shape-1 (svg-defs-shape trip-1)
trip-2 (:imported (export-import-file profile trip-1))
shape-2 (svg-defs-shape trip-2)]
(t/is (some? shape-1) "the svg shape survived the first trip")
(t/is (some? shape-2) "the svg shape survived the second trip")
(t/is (= {:fill-rule "evenodd"} (:svg-attrs shape-1))
"first trip lands on the kebab end state")
(t/is (= (:svg-attrs shape-1) (:svg-attrs shape-2))
"svg-attrs are a fixed point")
(t/is (= (:svg-defs shape-1) (:svg-defs shape-2))
"svg-defs are a fixed point")))
(t/deftest export-binfile-v3
(let [profile (th/create-profile* 1)
file (prepare-simple-file profile)
@ -2915,3 +3093,181 @@
(t/is (thrown? IllegalArgumentException
(bfc/get-file th/*system* (uuid/random)
:load-data? false opt true)))))
;; --- Library relations security tests
(t/deftest import-v3-ignores-relations-outside-package
(let [profile (th/create-profile* 1)
file (prepare-simple-file profile)
[_ victim] (prepare-victim-file 3)
bundle (export-file file)
evil (tmp/tempfile :suffix ".zip")
_ (tamper-manifest-relations bundle evil
[[(str (:id file)) (str (:id victim))]])
[file-id :as result] (import-bundle profile evil)]
(t/is (= 1 (count result)))
(t/is (empty? (th/db-query :file-library-rel {:file-id file-id})))
(t/is (empty? (th/db-query :file-library-sync {:file-id file-id})))))
(defn- prepare-other-team
[n]
(let [profile (th/create-profile* n)
team (th/create-team* n {:profile-id (:id profile)})
project (th/create-project* n {:team-id (:id team)
:profile-id (:id profile)})
file (th/create-file* n {:profile-id (:id profile)
:project-id (:id project)})
library (th/create-file* (inc n) {:profile-id (:id profile)
:project-id (:id project)})]
[profile file library]))
(t/deftest import-v3-ignores-foreign-file-id
(let [profile (th/create-profile* 1)
file (prepare-simple-file profile)
[_ victim] (prepare-victim-file 3)
bundle (export-file file)
evil (tmp/tempfile :suffix ".zip")
_ (tamper-manifest-relations bundle evil
[[(str (:id victim)) (str (:id file))]])
result (import-bundle profile evil)]
(t/is (= 1 (count result)))
(t/is (empty? (th/db-query :file-library-rel {:file-id (:id victim)})))
(t/is (empty? (th/db-query :file-library-sync {:file-id (:id victim)})))))
(t/deftest import-v3-ignores-unknown-library
(let [profile (th/create-profile* 1)
file (prepare-simple-file profile)
bundle (export-file file)
evil (tmp/tempfile :suffix ".zip")
_ (tamper-manifest-relations bundle evil
[[(str (:id file)) (str (uuid/random))]])
[file-id :as result] (import-bundle profile evil)]
(t/is (= 1 (count result)))
(t/is (empty? (th/db-query :file-library-rel {:file-id file-id})))))
(t/deftest import-v3-ignores-self-library-relation
(let [profile (th/create-profile* 1)
file (prepare-simple-file profile)
bundle (export-file file)
evil (tmp/tempfile :suffix ".zip")
_ (tamper-manifest-relations bundle evil
[[(str (:id file)) (str (:id file))]])
[file-id :as result] (import-bundle profile evil)]
(t/is (= 1 (count result)))
(t/is (empty? (th/db-query :file-library-rel {:file-id file-id})))))
(t/deftest import-v3-keeps-valid-library-relation
(let [profile (th/create-profile* 1)
file (prepare-simple-file profile)
library (th/create-file* 2 {:profile-id (:id profile)
:project-id (:default-project-id profile)})]
(th/link-file-to-library* {:file-id (:id file)
:library-id (:id library)})
(let [bundle (export-file file :include-libraries true)
result (import-bundle profile bundle)
new-file (->> result
(map #(bfc/get-file th/*system* %))
(some #(when (= (:name %) (:name file)) %)))]
(t/is (= 2 (count result)))
(t/is (some? new-file))
(t/is (= 1 (count (th/db-query :file-library-rel
{:file-id (:id new-file)})))))))
(t/deftest export-multi-file-includes-package-relations
(let [profile (th/create-profile* 1)
file (prepare-simple-file profile)
library (th/create-file* 2 {:profile-id (:id profile)
:project-id (:default-project-id profile)})
[_ other-file other-library] (prepare-other-team 3)
_ (th/link-file-to-library* {:file-id (:id file)
:library-id (:id library)})
_ (th/link-file-to-library* {:file-id (:id other-file)
:library-id (:id other-library)})
_ (th/db-insert! :file-library-rel
{:file-id (:id file)
:library-file-id (:id other-library)})
output (tmp/tempfile :suffix ".zip")]
(v3/export-files!
(-> th/*system*
(assoc ::bfc/file-ids #{(:id file) (:id other-file)})
(assoc ::bfc/export-type :include-libraries)
(assoc ::bfc/output (io/output-stream output))))
(let [rels (set (map vec (get (read-manifest output) "relations")))]
(t/is (contains? rels [(str (:id file)) (str (:id library))]))
(t/is (contains? rels [(str (:id other-file)) (str (:id other-library))]))
(t/is (contains? rels [(str (:id file)) (str (:id other-library))])))))
;; --- Forbidden library relations cleanup (migration candidate)
;;
;; The two statements below are the exact cleanup proposed for a future
;; migration (see tmp/CHECK-LIBRARY-RELATIONS.sql, section 6). The sync
;; rows must go first: the second statement reads file_library_rel
;; while it still contains the forbidden rows.
(def ^:private sql:delete-forbidden-library-sync
"WITH forbidden AS (
SELECT flr.file_id, flr.library_file_id
FROM file_library_rel flr
JOIN file f ON f.id = flr.file_id
JOIN project fp ON fp.id = f.project_id
JOIN file l ON l.id = flr.library_file_id
JOIN project lp ON lp.id = l.project_id
WHERE fp.team_id <> lp.team_id
OR flr.file_id = flr.library_file_id
)
DELETE FROM file_library_sync fls
USING forbidden
WHERE fls.file_id = forbidden.file_id
AND fls.library_file_id = forbidden.library_file_id")
(def ^:private sql:delete-forbidden-library-rel
"WITH forbidden AS (
SELECT flr.file_id, flr.library_file_id
FROM file_library_rel flr
JOIN file f ON f.id = flr.file_id
JOIN project fp ON fp.id = f.project_id
JOIN file l ON l.id = flr.library_file_id
JOIN project lp ON lp.id = l.project_id
WHERE fp.team_id <> lp.team_id
OR flr.file_id = flr.library_file_id
)
DELETE FROM file_library_rel flr
USING forbidden
WHERE flr.file_id = forbidden.file_id
AND flr.library_file_id = forbidden.library_file_id")
(t/deftest cleanup-deletes-forbidden-library-relations
(let [profile (th/create-profile* 1)
file (prepare-simple-file profile)
library (th/create-file* 2 {:profile-id (:id profile)
:project-id (:default-project-id profile)})
[_ victim] (prepare-victim-file 3)
_ (th/link-file-to-library* {:file-id (:id file)
:library-id (:id library)})
_ (th/db-insert! :file-library-rel
{:file-id (:id file)
:library-file-id (:id victim)})
_ (th/db-insert! :file-library-sync
{:file-id (:id file)
:library-file-id (:id victim)})
_ (th/db-insert! :file-library-rel
{:file-id (:id file)
:library-file-id (:id file)})
_ (th/db-insert! :file-library-sync
{:file-id (:id file)
:library-file-id (:id file)})]
(t/is (= 3 (count (th/db-query :file-library-rel {:file-id (:id file)}))))
(th/db-exec! [sql:delete-forbidden-library-sync])
(th/db-exec! [sql:delete-forbidden-library-rel])
(t/testing "same-team relation and its sync row survive"
(t/is (= [{:file-id (:id file) :library-file-id (:id library)}]
(map #(select-keys % [:file-id :library-file-id])
(th/db-query :file-library-rel {:file-id (:id file)}))))
(t/is (= [{:file-id (:id file) :library-file-id (:id library)}]
(map #(select-keys % [:file-id :library-file-id])
(th/db-query :file-library-sync {:file-id (:id file)})))))
(t/testing "cross-team and self relations and their sync rows are gone"
(t/is (empty? (th/db-query :file-library-rel {:library-file-id (:id victim)})))
(t/is (empty? (th/db-query :file-library-sync {:library-file-id (:id victim)})))
(t/is (empty? (th/db-query :file-library-rel {:library-file-id (:id file)})))
(t/is (empty? (th/db-query :file-library-sync {:library-file-id (:id file)}))))))

11
backend/test/backend_tests/config_test.clj
View File

@ -6,10 +6,21 @@
(ns backend-tests.config-test
(:require
[app.common.schema :as sm]
[app.common.time :as ct]
[app.config :as cf]
[clojure.test :as t]))
(t/deftest tenant-is-normalized-to-lowercase
(t/testing "an uppercase tenant names the same tenant"
(t/is (= "acme" (:tenant (#'cf/normalize-tenant {:tenant "ACME"}))))
(t/is (= "acme" (:tenant (#'cf/normalize-tenant {:tenant "acme"})))))
(t/testing "a config without a tenant is left untouched"
(t/is (= {:x 1} (#'cf/normalize-tenant {:x 1}))))
(t/testing "the validated form is lowercase-only"
(t/is (not (sm/validate cf/schema:tenant "ACME")))
(t/is (sm/validate cf/schema:tenant "acme"))))
(t/deftest get-public-uri-normalizes-base
(t/testing "trailing slash is ensured with and without subpath"
(doseq [[base expected] [["http://localhost:3449" "http://localhost:3449/"]

193
backend/test/backend_tests/email_sending_test.clj
View File

@ -15,6 +15,7 @@
[clojure.test :as t]
[cuerdas.core :as str]
[integrant.core :as ig]
[lambdaisland.uri :as u]
[promesa.core :as p]))
(t/use-fixtures :once th/state-init)
@ -142,3 +143,195 @@
(assoc email ::emails/reuse-conn true))))]
(t/testing "the email job is submitted through the caller conn"
(t/is (= "sendmail" (:name (th/db-get :job {:id job-id})))))))
(defn- templated-job-def
[sendmail]
(ig/init-key ::emails/templated-job-def
{::emails/sendmail sendmail}))
(defn- submit-templated
[job-def params]
(db/tx-run! {::db/pool th/*pool*}
(fn [cfg]
(emails/send-template (assoc cfg
::jobs/defs {:send-templated-mail job-def}
::mtx/metrics (get th/*system* :app.metrics/metrics))
(assoc params ::emails/reuse-conn true)))))
(def ^:private invite-to-team-recipe
{:to "invitee@example.com"
:public-uri (cf/get :public-uri)
:invited-by "Owner User"
:team "Design Team"
:token "test-token"
:organization {:name "Acme Inc"
:sso-active false}})
(t/deftest templated-registry-covers-all-factories
(t/is (= #{"feedback" "register" "password-recovery" "password-changed"
"change-email" "invite-to-team" "invite-to-organization"
"organization-setup-sso" "renewal-notice" "join-team"
"request-file-access" "request-file-access-yourpenpot"
"request-file-access-yourpenpot-view" "request-team-access"
"comment-mention" "comment-thread" "comment-notification"}
(set (keys emails/templates)))))
(t/deftest send-template-validation
(t/testing "missing template fails fast"
(t/is (thrown-with-msg? clojure.lang.ExceptionInfo #"template"
(emails/send-template {} {:to "a@example.com"}))))
(t/testing "unknown template fails fast"
(t/is (thrown-with-msg? clojure.lang.ExceptionInfo #"unknown email template"
(emails/send-template {} {:to "a@example.com"
::emails/template "nope"}))))
(t/testing "template ids are strings, never keywords"
(t/is (thrown-with-msg? clojure.lang.ExceptionInfo #"unknown email template"
(emails/send-template {} {:to "a@example.com"
::emails/template :app.email/register}))))
(t/testing "reuse-conn without a caller connection fails fast"
(t/is (thrown-with-msg? clojure.lang.ExceptionInfo #"reuse-conn"
(emails/send-template {} {:to "a@example.com"
::emails/reuse-conn true
::emails/template "register"})))))
(t/deftest send-template-stores-recipe-not-rendered-mail
(let [job-def (templated-job-def (fn [_] nil))
job-id (submit-templated job-def (assoc invite-to-team-recipe
::emails/template "invite-to-team"))
row (th/db-get :job {:id job-id})
params (jobs/decode-params job-def (:params row))
rendered (emails/render emails/invite-to-team invite-to-team-recipe)]
(t/testing "the job lands on the new queue"
(t/is (= "send-templated-mail" (:name row))))
(t/testing "stored params carry the template id and no rendered content"
(t/is (= "invite-to-team" (:template params)))
(t/is (not (contains? params :subject)))
(t/is (not (contains? params :body))))
(t/testing "stored params are smaller than the rendered html alone"
(let [stored-size (count (str (:params row)))
rendered-html (count (get-in rendered [:body "text/html"]))]
(t/is (< stored-size rendered-html))))
(t/testing "the worker renders the same mail from the stored recipe"
(t/is (= rendered (emails/render-template (:template params)
(dissoc params :template)))))))
(defn- decode-template-data
[params]
(let [decode (get-in emails/templates [(:template params) ::emails/decode])]
(decode (dissoc params :template))))
(t/deftest send-template-roundtrips-uuids-and-profile
(let [sent (atom nil)
job-def (templated-job-def (fn [_ email] (reset! sent email)))
team-id (random-uuid)
recipe {:to "member@example.com"
:public-uri (cf/get :public-uri)
:invited-by "Owner User"
:team "Design Team"
:team-id team-id}
job-id (submit-templated job-def (assoc recipe
::emails/template "join-team"))
params (jobs/decode-params job-def (:params (th/db-get :job {:id job-id})))]
(t/testing "uuids survive the plain-JSON queue roundtrip typed"
(let [data (decode-template-data params)]
(t/is (uuid? (:team-id data)))
(t/is (= team-id (:team-id data)))))
(t/testing "the handler renders and delivers the mail"
((::jobs/handler job-def) nil params)
(t/is (= (:subject (emails/render emails/join-team recipe))
(:subject @sent)))
(t/is (= "member@example.com" (:to @sent)))))
(let [sent (atom nil)
job-def (templated-job-def (fn [_ email] (reset! sent email)))
recipe {:to "dest@example.com"
:reply-to "user@example.com"
:email "user@example.com"
:attachments {"error-report.txt" "trace"}
:feedback-subject "subj"
:feedback-type "bug"
:feedback-content "content"
:profile {:id (random-uuid) :fullname "User" :email "user@example.com"}}
job-id (submit-templated job-def (assoc recipe
::emails/template "feedback"))
params (jobs/decode-params job-def (:params (th/db-get :job {:id job-id})))]
(t/testing "nested profile id survives the roundtrip typed"
(t/is (uuid? (get-in (decode-template-data params) [:profile :id]))))
(t/testing "the handler renders the feedback mail"
((::jobs/handler job-def) nil params)
(t/is (= (:subject (emails/render emails/user-feedback recipe))
(:subject @sent))))))
(def ^:private organization-with-uris
{:id (random-uuid)
:name "Acme Inc"
:initials "AI"
:logo (u/uri "https://example.com/assets/by-id/logo")
:avatar-bg-url (u/uri "https://example.com/images/avatar-bg.jpg")
:sso-active false})
(t/deftest send-template-roundtrips-organization-uris
(doseq [[template factory recipe]
[["invite-to-team" emails/invite-to-team
{:to "invitee@example.com"
:public-uri (cf/get :public-uri)
:invited-by "Owner User"
:team "Design Team"
:organization organization-with-uris
:token "test-token"}]
["invite-to-organization" emails/invite-to-organization
{:to "invitee@example.com"
:public-uri (cf/get :public-uri)
:invited-by "Owner User"
:user-name "Invitee"
:organization organization-with-uris
:token "test-token"}]
["renewal-notice" emails/renewal-notice
{:to "billing@example.com"
:public-uri (cf/get :public-uri)
:user-name "Billing"
:renewal-date "2026-12-01"
:estimated-amount "$10.00"
:organizations [organization-with-uris]}]]]
(t/testing template
(let [sent (atom nil)
job-def (templated-job-def (fn [_ email] (reset! sent email)))
job-id (submit-templated job-def (assoc recipe ::emails/template template))
params (jobs/decode-params job-def (:params (th/db-get :job {:id job-id})))]
((::jobs/handler job-def) nil params)
(t/is (= (emails/render factory recipe) @sent))))))
;; `send-template` drops nil values before enqueuing, so a nil
;; `:user-name` (e.g. inviting an unregistered email) must still render.
(t/deftest send-template-renders-with-nil-user-name
(doseq [[template factory recipe]
[["invite-to-organization" emails/invite-to-organization
{:to "invitee@example.com"
:public-uri (cf/get :public-uri)
:invited-by "Owner User"
:user-name nil
:organization {:name "Acme Inc"}
:token "test-token"}]
["renewal-notice" emails/renewal-notice
{:to "billing@example.com"
:public-uri (cf/get :public-uri)
:user-name nil
:renewal-date "2026-12-01"
:estimated-amount "$10.00"
:organizations [{:name "Acme"}]}]]]
(t/testing template
(let [sent (atom nil)
job-def (templated-job-def (fn [_ email] (reset! sent email)))
job-id (submit-templated job-def (assoc recipe ::emails/template template))
params (jobs/decode-params job-def (:params (th/db-get :job {:id job-id})))]
((::jobs/handler job-def) nil params)
(t/is (= (emails/render factory recipe) @sent))))))
(t/deftest send-templated-mail-handler-rejects-unknown-template
(let [job-def (templated-job-def (fn [_] nil))
code (try
((::jobs/handler job-def) nil {:template "nope"
:to "a@example.com"})
nil
(catch clojure.lang.ExceptionInfo e
(-> e ex-data :code)))]
(t/is (= :unknown-email-template code))))

4
backend/test/backend_tests/file_change_retention_test.clj
View File

@ -118,8 +118,8 @@
(t/testing "the snapshot comes back"
(t/is (nil? (:deleted-at (find-change snapshot-id)))))
(t/testing "the legacy payload row comes back"
(t/is (nil? (:deleted-at (find-change legacy-id))))))))
(t/testing "the legacy payload row keeps its date"
(t/is (= trash-at (:deleted-at (find-change legacy-id))))))))
(t/deftest deleting-a-file-does-not-push-the-revision-schedule-out
(binding [ct/*clock* (ct/fixed-clock test-now)]

52
backend/test/backend_tests/files_update_validation_timing_test.clj
View File

@ -14,6 +14,7 @@
[app.metrics :as mtx]
[app.metrics.definition :as-alias mdef]
[app.rpc.commands.files-update :as files-update]
[app.util.pointer-map :as pmap]
[clojure.test :as t]
[integrant.core :as ig])
(:import
@ -61,11 +62,12 @@
(t/deftest process-changes-and-validate-records-all-steps
(let [metrics (ig/init-key :app.metrics/metrics {:default main/default-metrics})
cfg (test-cfg metrics)]
(binding [cf/flags all-flags]
(binding [cf/flags all-flags
pmap/*tracked* (pmap/create-tracked)]
(with-redefs [bfc/get-resolved-file-libraries (fn [_ _] {})
val/validate-file-affected! (fn [_ _ _] nil)
val/validate-file-schema! (fn [_] nil)]
(let [file (@#'files-update/process-changes-and-validate cfg (test-file) [] nil)
val/validate-file-schema-affected! (fn [_ _] nil)]
(let [file (@#'files-update/process-changes-and-validate cfg (test-file) [] nil nil)
timings @(::files-update/validation-timings cfg)]
(t/is (map? file))
(t/is (number? (:resolve-libs-ms timings)))
@ -84,15 +86,16 @@
(t/deftest process-changes-and-validate-skips-when-skip-validate
(let [metrics (ig/init-key :app.metrics/metrics {:default main/default-metrics})
cfg (test-cfg metrics)]
(binding [cf/flags all-flags]
(binding [cf/flags all-flags
pmap/*tracked* (pmap/create-tracked)]
(with-redefs [bfc/get-resolved-file-libraries (fn [_ _]
(throw (ex-info "must not resolve libs" {})))
val/validate-file-affected! (fn [_ _ _]
(throw (ex-info "must not validate" {})))
val/validate-file-schema! (fn [_]
(throw (ex-info "must not validate schema" {})))]
val/validate-file-schema-affected! (fn [_ _]
(throw (ex-info "must not validate schema" {})))]
;; soft validations still run: they only check flags, not skip-validate
(let [file (@#'files-update/process-changes-and-validate cfg (test-file) [] true)
(let [file (@#'files-update/process-changes-and-validate cfg (test-file) [] nil true)
timings @(::files-update/validation-timings cfg)]
(t/is (map? file))
(t/is (nil? (:resolve-libs-ms timings)))
@ -107,14 +110,15 @@
(t/deftest process-changes-and-validate-records-timing-on-error
(let [metrics (ig/init-key :app.metrics/metrics {:default main/default-metrics})
cfg (test-cfg metrics)]
(binding [cf/flags all-flags]
(binding [cf/flags all-flags
pmap/*tracked* (pmap/create-tracked)]
(with-redefs [bfc/get-resolved-file-libraries (fn [_ _] {})
val/validate-file-affected! (fn [_ _ _]
(throw (ex-info "bad file" {:code :referential-integrity})))
val/validate-file-schema! (fn [_] nil)]
val/validate-file-schema-affected! (fn [_ _] nil)]
(let [timings-atom (::files-update/validation-timings cfg)]
(t/is (thrown? clojure.lang.ExceptionInfo
(@#'files-update/process-changes-and-validate cfg (test-file) [] nil)))
(@#'files-update/process-changes-and-validate cfg (test-file) [] nil nil)))
;; the failing step still recorded its timing via finally
(t/is (number? (:validate-affected-ms @timings-atom)))
(t/is (number? (:resolve-libs-ms @timings-atom)))
@ -126,3 +130,31 @@
(let [record @#'files-update/record-validation-timing]
(t/is (nil? (record {} :resolve-libs-ms "resolve-libs" 12)))
(t/is (nil? (record {::mtx/metrics nil} :resolve-libs-ms "resolve-libs" 12)))))
(t/deftest retries-record-one-sample-per-step-and-one-retry-sample
(let [metrics (ig/init-key :app.metrics/metrics {:default main/default-metrics})
cfg (test-cfg metrics)
commit (fn [name]
{:commit-id (uuid/next)
:changes [{:type :mod-page :id (uuid/next) :name name}]})
commits [(commit "a") (commit "b") (commit "bad") (commit "c")]]
(binding [cf/flags all-flags
pmap/*tracked* (pmap/create-tracked)]
(with-redefs [bfc/get-resolved-file-libraries (fn [_ _] {})
val/validate-file-affected! (fn [_ _ changes]
(when (some #(= "bad" (:name %)) changes)
(throw (ex-info "bad file" {:type :validation
:code :referential-integrity}))))
val/validate-file-schema-affected! (fn [_ _] nil)]
(let [file (@#'files-update/process-changes-and-validate
cfg (test-file) (into [] (mapcat :changes) commits) commits nil)
timings @(::files-update/validation-timings cfg)]
(t/is (= (mapv :commit-id (subvec commits 2))
(:commit-ids (::files-update/rejected file))))
(t/is (number? (:retry-ms timings)))
;; The first pass fails at validate-affected, before validate-schema
(doseq [step ["process-changes" "soft-validate-affected" "soft-validate-schema"
"validate-affected" "retry"]]
(t/is (= 1.0 (bucket-value metrics "penpot_rpc_update_file_validation_timing"
["step"] [step] "60000.0"))
(str "expected one sample for step " step))))))))

63
backend/test/backend_tests/helpers.clj
View File

@ -17,6 +17,7 @@
[app.common.spec :as us]
[app.common.time :as ct]
[app.common.transit :as tr]
[app.common.types.shape :as cts]
[app.common.uuid :as uuid]
[app.config :as cf]
[app.db :as db]
@ -86,7 +87,8 @@
:enable-quotes
:enable-rpc-climit
:enable-auto-file-snapshot
:disable-file-validation])
:disable-file-validation
:disable-admin-console])
(defn init-config
([next]
@ -378,6 +380,21 @@
:session-id session-id
:profile-id profile-id})))))))
(defn add-rect-change
"A change that adds a 10x10 rect `id` to the root of `page-id`."
[page-id id]
{:type :add-obj
:page-id page-id
:id id
:parent-id uuid/zero
:frame-id uuid/zero
:obj (cts/setup-shape {:id id
:type :rect
:name "rect"
:x 0 :y 0 :width 10 :height 10
:parent-id uuid/zero
:frame-id uuid/zero})})
(declare command!)
(defn update-file! [& {:keys [profile-id file-id changes revn] :or {revn 0}}]
@ -547,13 +564,38 @@
(method-fn params))))
(defn management-command!
"Invokes a management RPC method.
Resolves the shared-key id from an explicit `::rpc/auth-key-id`, from
`:app.http/auth-key-id` on the data or on the supplied request, and falls
back to the first id the method declares. The fallback lets the existing
tests exercise the allowed path without repeating the key; a test that
checks a rejection declares the key explicitly, including `nil`."
[{:keys [::type] :as data}]
(let [[_ method-fn] (get-in *system* [:app.rpc/management-methods type])]
(let [[mdata method-fn] (get-in *system* [:app.rpc/management-methods type])]
(when-not method-fn
(ex/raise :type :assertion
:code :rpc-method-not-found
:hint (str/ffmt "management rpc method '%' not found" (name type))))
(let [params (prepare-rpc-params data)]
(let [request (-> data meta :app.http/request)
key-id (cond
(contains? data ::rpc/auth-key-id)
(::rpc/auth-key-id data)
(contains? data :app.http/auth-key-id)
(:app.http/auth-key-id data)
(and (map? request)
(contains? request :app.http/auth-key-id))
(:app.http/auth-key-id request)
:else
(first (::rpc/allowed-shared-keys mdata)))
data (if (or (contains? data ::rpc/auth-key-id)
(nil? key-id))
data
(assoc data ::rpc/auth-key-id key-id))
params (prepare-rpc-params data)]
(try-on! (method-fn params)))))
(def multipart-headers
@ -735,6 +777,21 @@
(io/file tmp))
tmp))
(defmacro with-temp-file
"Evaluate `body` with `sym` bound to the path of a temporary file that
holds `content`. The file is created under /tmp/penpot, one of the two
places the upload validation accepts (/tmp/undertow is the other one),
and it is deleted when the body ends, also when it throws."
[[sym content] & body]
`(let [path# (fs/create-tempfile :dir "/tmp/penpot" :prefix "penpot-test-")]
(try
(spit (str path#) ~content :encoding "UTF-8")
(let [~sym path#]
~@body)
(finally
(when (fs/exists? path#)
(fs/delete path#))))))
(defn pause
[]
(let [^java.io.Console cnsl (System/console)]

43
backend/test/backend_tests/http_errors_test.clj
View File

@ -68,3 +68,46 @@
headers (::yres/headers response)]
(t/is (= 429 (::yres/status response)))
(t/is (nil? (get headers "retry-after")))))
(defn- psql-exception
[state]
(org.postgresql.util.PSQLException. "database failure" state))
(t/deftest transient-database-failures-answer-503
(doseq [state [org.postgresql.util.PSQLState/CONNECTION_FAILURE
org.postgresql.util.PSQLState/CONNECTION_UNABLE_TO_CONNECT
org.postgresql.util.PSQLState/SERIALIZATION_FAILURE
org.postgresql.util.PSQLState/DEADLOCK_DETECTED]]
(let [response (http-errors/handle (psql-exception state) {})]
(t/is (= 503 (::yres/status response)) (str state))
(t/is (= :server-error (-> response ::yres/body :type))))))
(t/deftest other-database-failures-answer-500
(let [response (http-errors/handle (psql-exception org.postgresql.util.PSQLState/UNIQUE_VIOLATION) {})]
(t/is (= 500 (::yres/status response)))
(t/is (= :database-error (-> response ::yres/body :code)))))
(t/deftest database-timeouts-keep-answering-504
(let [response (http-errors/handle (psql-exception org.postgresql.util.PSQLState/QUERY_CANCELED) {})]
(t/is (= 504 (::yres/status response)))
(t/is (= :statement-timeout (-> response ::yres/body :code)))))
(t/deftest transient-connection-failures-answer-503
(doseq [cause [(java.sql.SQLTransientConnectionException. "pool exhausted")
(java.net.SocketException. "broken pipe")
(java.net.ConnectException. "connection refused")
(java.net.SocketTimeoutException. "read timed out")
(java.net.UnknownHostException. "storage.example.com")
(InterruptedException. "shutting down")]]
(let [response (http-errors/handle cause {})]
(t/is (= 503 (::yres/status response)) (str (class cause)))
(t/is (= :server-error (-> response ::yres/body :type))))))
(t/deftest permanent-io-failures-answer-500
(doseq [cause [(java.io.IOException. "generic")
(java.io.EOFException. "truncated blob")
(java.nio.file.NoSuchFileException. "/data/missing")
(com.fasterxml.jackson.core.JsonParseException. nil "bad json")]]
(let [response (http-errors/handle cause {})]
(t/is (= 500 (::yres/status response)) (str (class cause)))
(t/is (= :io-exception (-> response ::yres/body :code))))))

8
backend/test/backend_tests/loggers_audit_consumer_task_test.clj
View File

@ -117,13 +117,19 @@
http-mock {:target 'app.http.client/req :return {:status 204}}]
(insert-audit-row! (:id prof) "create-project")
(insert-audit-row! (:id prof) "create-organization-attribute")
(insert-audit-row! (:id prof) "delete-organization-attribute")
(insert-audit-row! (:id prof) "update-organization-attribute")
(insert-audit-row! (:id prof) "auditlog-exported")
(insert-audit-row! (:id prof) "unrelated-event")
(run-both! {:uri archive-uri})
(t/is (:called? @admin-console-mock))
(t/is (:called? @http-mock))
(let [[_ _ params] (:call-args @admin-console-mock)]
(t/is (= #{"create-project" "create-organization-attribute" "auditlog-exported"}
(t/is (= #{"create-project"
"create-organization-attribute"
"delete-organization-attribute"
"update-organization-attribute"
"auditlog-exported"}
(into #{} (map :name) (:events params)))))
(let [rows (th/db-exec! ["select * from audit_log"])]
(t/is (= 0 (count rows))))))))

83
backend/test/backend_tests/loggers_audit_test.clj Normal file
View File

@ -0,0 +1,83 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns backend-tests.loggers-audit-test
"Tests for the anonymisation filter applied to the telemetry shadow rows."
(:require
[app.common.uuid :as uuid]
[app.loggers.audit :as audit]
[clojure.test :as t]))
(def ^:private profile-id (uuid/next))
(defn- shadow-props
[event]
(:props (audit/filter-telemetry-props event)))
(t/deftest identify-keeps-the-session-origin-fields
(let [props (shadow-props {:source "frontend"
:type "identify"
:name "signin"
:props {:id profile-id
:auth-backend "penpot"
:source "register"
:activation "immediate"}})]
(t/is (= "register" (:source props)))
(t/is (= "immediate" (:activation props)))
(t/is (= "penpot" (:auth-backend props)))
(t/is (= profile-id (:id props)))))
(t/deftest identify-drops-personal-data-and-derives-the-domain
(let [props (shadow-props {:source "frontend"
:type "identify"
:name "signin"
:props {:id profile-id
:email "someone@example.com"
:fullname "Some One"}})]
(t/is (not (contains? props :email)))
(t/is (not (contains? props :fullname)))
(t/is (= "example.com" (:email-domain props)))))
(t/deftest backend-auth-event-keeps-activation-and-drops-action
(let [props (shadow-props {:source "backend"
:type "action"
:name "register-profile"
:props {:id profile-id
:activation "email-verification"
:action "register"}})]
(t/is (= "email-verification" (:activation props)))
(t/is (not (contains? props :action)))))
(t/deftest absent-origin-fields-are-not-emitted-as-nil
(let [props (shadow-props {:source "frontend"
:type "identify"
:name "signin"
:props {:id profile-id}})]
(t/is (not (contains? props :source)))
(t/is (not (contains? props :activation)))
(t/is (not (contains? props :auth-backend)))
(t/is (not (contains? props :email-domain)))))
(t/deftest strings-outside-the-allowlist-are-dropped
(let [props (shadow-props {:source "frontend"
:type "identify"
:name "signin"
:props {:id profile-id
:fullname "Some One"
:anything "else"}})]
(t/is (not (contains? props :fullname)))
(t/is (not (contains? props :anything)))))
(t/deftest events-outside-the-auth-branch-keep-the-default-filter
(let [props (shadow-props {:source "backend"
:type "action"
:name "create-file"
:props {:id profile-id
:team-id profile-id
:activation "immediate"}})]
(t/is (= profile-id (:id props)))
(t/is (= profile-id (:team-id props)))
(t/is (not (contains? props :activation)))))

394
backend/test/backend_tests/media_test.clj
View File

@ -11,6 +11,7 @@
[app.media.svg :as svg]
[backend-tests.helpers :as th]
[clojure.test :as t]
[cuerdas.core :as str]
[datoteka.fs :as fs]))
(t/use-fixtures :once th/state-init)
@ -60,45 +61,43 @@
(t/testing "sanitize-svg removes script tags"
(let [svg "<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"100\" height=\"100\"><script>alert('xss')</script><rect width=\"50\" height=\"50\"/></svg>"
result (svg/sanitize-svg svg)]
(t/is (not (clojure.string/includes? result "<script>")))
(t/is (not (clojure.string/includes? result "alert")))
(t/is (clojure.string/includes? result "<rect")))))
(t/is (not (str/includes? result "<script>")))
(t/is (not (str/includes? result "alert")))
(t/is (str/includes? result "<rect")))))
(t/deftest sanitize-svg-event-handlers
(t/testing "sanitize-svg removes event handler attributes"
(let [svg "<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"100\" height=\"100\" onload=\"alert('xss')\"><rect width=\"50\" height=\"50\" onmouseover=\"alert('xss')\"/></svg>"
result (svg/sanitize-svg svg)]
(t/is (not (clojure.string/includes? result "onload")))
(t/is (not (clojure.string/includes? result "onmouseover")))
(t/is (not (clojure.string/includes? result "alert")))
(t/is (clojure.string/includes? result "<rect")))))
(t/is (not (str/includes? result "onload")))
(t/is (not (str/includes? result "onmouseover")))
(t/is (not (str/includes? result "alert")))
(t/is (str/includes? result "<rect")))))
(t/deftest sanitize-svg-javascript-href
(t/testing "sanitize-svg removes javascript: URLs from href attributes"
(let [svg "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"100\" height=\"100\"><a xlink:href=\"javascript:alert('xss')\"><rect width=\"50\" height=\"50\"/></a></svg>"
result (svg/sanitize-svg svg)]
(t/is (not (clojure.string/includes? result "javascript:")))
(t/is (not (clojure.string/includes? result "alert")))
(t/is (clojure.string/includes? result "<a")))))
(t/is (not (str/includes? result "javascript:")))
(t/is (not (str/includes? result "alert")))
(t/is (str/includes? result "<a")))))
(t/deftest sanitize-svg-foreign-object
(t/testing "sanitize-svg removes foreignObject elements"
(let [svg "<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"100\" height=\"100\"><foreignObject width=\"100\" height=\"100\"><body xmlns=\"http://www.w3.org/1999/xhtml\"><script>alert('xss')</script></body></foreignObject><rect width=\"50\" height=\"50\"/></svg>"
result (svg/sanitize-svg svg)]
(t/is (not (clojure.string/includes? result "foreignObject")))
(t/is (not (clojure.string/includes? result "<script>")))
(t/is (clojure.string/includes? result "<rect")))))
(t/is (not (str/includes? result "foreignObject")))
(t/is (not (str/includes? result "<script>")))
(t/is (str/includes? result "<rect")))))
(t/deftest sanitize-svg-clean-content
(t/testing "sanitize-svg preserves clean SVG content"
(let [svg "<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"100\" height=\"100\"><rect width=\"50\" height=\"50\" fill=\"red\"/><circle cx=\"75\" cy=\"75\" r=\"20\" fill=\"blue\"/></svg>"
result (svg/sanitize-svg svg)]
(t/is (clojure.string/includes? result "<rect"))
(t/is (clojure.string/includes? result "<circle"))
(t/is (or (clojure.string/includes? result "fill=\"red\"")
(clojure.string/includes? result "fill='red'")))
(t/is (or (clojure.string/includes? result "fill=\"blue\"")
(clojure.string/includes? result "fill='blue'"))))))
(t/is (str/includes? result "<rect"))
(t/is (str/includes? result "<circle"))
(t/is (str/includes? result "fill=\"red\""))
(t/is (str/includes? result "fill=\"blue\"")))))
(t/deftest sanitize-svg-invalid-svg-rejected
(t/testing "sanitize-svg rejects malformed SVG input"
@ -110,98 +109,381 @@
(t/testing "sanitize-svg preserves legitimate xlink:href attributes"
(let [svg "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"100\" height=\"100\"><use xlink:href=\"#icon\"/></svg>"
result (svg/sanitize-svg svg)]
(t/is (clojure.string/includes? result "xlink:href"))
(t/is (clojure.string/includes? result "#icon")))))
(t/is (str/includes? result "xlink:href"))
(t/is (str/includes? result "#icon")))))
(t/deftest sanitize-svg-javascript-href-whitespace
(t/testing "sanitize-svg catches javascript: URLs with leading whitespace"
(let [svg "<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"100\" height=\"100\"><a href=\" javascript:alert('xss')\"><rect width=\"50\" height=\"50\"/></a></svg>"
result (svg/sanitize-svg svg)]
(t/is (not (clojure.string/includes? result "javascript:")))
(t/is (not (clojure.string/includes? result "alert")))
(t/is (clojure.string/includes? result "<a")))))
(t/is (not (str/includes? result "javascript:")))
(t/is (not (str/includes? result "alert")))
(t/is (str/includes? result "<a")))))
(t/deftest sanitize-svg-nested-script
(t/testing "sanitize-svg removes script tags from nested elements"
(let [svg "<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"100\" height=\"100\"><g><script>alert('xss')</script></g></svg>"
result (svg/sanitize-svg svg)]
(t/is (not (clojure.string/includes? result "<script")))
(t/is (not (clojure.string/includes? result "alert")))
(t/is (clojure.string/includes? result "<g")))))
(t/is (not (str/includes? result "<script")))
(t/is (not (str/includes? result "alert")))
(t/is (str/includes? result "<g")))))
(t/deftest sanitize-svg-smil-bypass
(t/testing "sanitize-svg removes SMIL animation elements that can set on* attrs"
(let [svg "<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"100\" height=\"100\"><rect width=\"100\" height=\"100\" id=\"r\"/><set attributeName=\"onmouseover\" to=\"alert('xss')\" xlink:href=\"#r\" begin=\"0s\"/></svg>"
result (svg/sanitize-svg svg)]
(t/is (not (clojure.string/includes? result "<set")))
(t/is (not (clojure.string/includes? result "onmouseover")))
(t/is (clojure.string/includes? result "<rect")))))
(t/is (not (str/includes? result "<set")))
(t/is (not (str/includes? result "onmouseover")))
(t/is (str/includes? result "<rect")))))
(t/deftest sanitize-svg-namespace-prefixed-script
(t/testing "sanitize-svg removes script tags written with a namespace prefix bound to the SVG namespace"
(let [svg "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:x=\"http://www.w3.org/2000/svg\" width=\"100\" height=\"100\"><x:script>alert('xss')</x:script><rect width=\"50\" height=\"50\"/></svg>"
result (svg/sanitize-svg svg)]
(t/is (not (clojure.string/includes? result "<x:script")))
(t/is (not (clojure.string/includes? result "alert")))
(t/is (clojure.string/includes? result "<rect")))))
(t/is (not (str/includes? result "<x:script")))
(t/is (not (str/includes? result "alert")))
(t/is (str/includes? result "<rect")))))
(t/deftest sanitize-svg-arbitrary-prefixed-script
(t/testing "sanitize-svg removes script tags written with an arbitrary namespace prefix"
(let [svg "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:p=\"http://www.w3.org/2000/svg\" width=\"100\" height=\"100\"><p:script>alert('xss')</p:script><rect width=\"50\" height=\"50\"/></svg>"
result (svg/sanitize-svg svg)]
(t/is (not (clojure.string/includes? result "<p:script")))
(t/is (not (clojure.string/includes? result "alert")))
(t/is (clojure.string/includes? result "<rect")))))
(t/is (not (str/includes? result "<p:script")))
(t/is (not (str/includes? result "alert")))
(t/is (str/includes? result "<rect")))))
(t/deftest sanitize-svg-nested-prefixed-script
(t/testing "sanitize-svg removes namespace-prefixed script tags from nested elements"
(let [svg "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:x=\"http://www.w3.org/2000/svg\" width=\"100\" height=\"100\"><g><x:script>alert('xss')</x:script></g></svg>"
result (svg/sanitize-svg svg)]
(t/is (not (clojure.string/includes? result "<x:script")))
(t/is (not (clojure.string/includes? result "alert")))
(t/is (clojure.string/includes? result "<g")))))
(t/is (not (str/includes? result "<x:script")))
(t/is (not (str/includes? result "alert")))
(t/is (str/includes? result "<g")))))
(t/deftest sanitize-svg-prefixed-dangerous-tags
(t/testing "sanitize-svg removes dangerous animation elements written with a namespace prefix"
(let [svg "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:x=\"http://www.w3.org/2000/svg\" width=\"100\" height=\"100\"><x:foreignObject width=\"10\" height=\"10\"/><x:set attributeName=\"onmouseover\" to=\"alert('xss')\"/><x:animate attributeName=\"fill\" to=\"red\"/><x:animateTransform attributeName=\"transform\"/><rect width=\"50\" height=\"50\"/></svg>"
result (svg/sanitize-svg svg)]
(t/is (not (clojure.string/includes? result "foreignObject")))
(t/is (not (clojure.string/includes? result "<x:set")))
(t/is (not (clojure.string/includes? result "<x:animate")))
(t/is (not (clojure.string/includes? result "alert")))
(t/is (clojure.string/includes? result "<rect")))))
(t/is (not (str/includes? result "foreignObject")))
(t/is (not (str/includes? result "<x:set")))
(t/is (not (str/includes? result "<x:animate")))
(t/is (not (str/includes? result "alert")))
(t/is (str/includes? result "<rect")))))
(t/deftest sanitize-svg-prefixed-javascript-href
(t/testing "sanitize-svg removes javascript: URLs from href attributes written with a namespace prefix"
(let [svg "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:x=\"http://www.w3.org/2000/svg\" width=\"100\" height=\"100\"><a x:href=\"javascript:alert('xss')\"><rect width=\"50\" height=\"50\"/></a></svg>"
result (svg/sanitize-svg svg)]
(t/is (not (clojure.string/includes? result "javascript:")))
(t/is (not (clojure.string/includes? result "alert")))
(t/is (clojure.string/includes? result "<a")))))
(t/is (not (str/includes? result "javascript:")))
(t/is (not (str/includes? result "alert")))
(t/is (str/includes? result "<a")))))
(t/deftest sanitize-svg-prefixed-event-handlers
(t/testing "sanitize-svg removes event handler attributes written with a namespace prefix"
(let [svg "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:x=\"http://www.w3.org/2000/svg\" width=\"100\" height=\"100\" x:onload=\"alert('xss')\"><rect width=\"50\" height=\"50\" x:onclick=\"alert('xss')\"/></svg>"
result (svg/sanitize-svg svg)]
(t/is (not (clojure.string/includes? result "onload")))
(t/is (not (clojure.string/includes? result "onclick")))
(t/is (not (clojure.string/includes? result "alert")))
(t/is (clojure.string/includes? result "<rect")))))
(t/is (not (str/includes? result "onload")))
(t/is (not (str/includes? result "onclick")))
(t/is (not (str/includes? result "alert")))
(t/is (str/includes? result "<rect")))))
(t/deftest sanitize-svg-prefixed-xlink-namespace-href
(t/testing "sanitize-svg removes javascript: URLs from href written with a prefix bound to the xlink namespace"
(let [svg "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:y=\"http://www.w3.org/1999/xlink\" width=\"100\" height=\"100\"><a y:href=\"javascript:alert('xss')\"><rect width=\"50\" height=\"50\"/></a></svg>"
result (svg/sanitize-svg svg)]
(t/is (not (clojure.string/includes? result "javascript:")))
(t/is (not (clojure.string/includes? result "alert")))
(t/is (clojure.string/includes? result "<a")))))
(t/is (not (str/includes? result "javascript:")))
(t/is (not (str/includes? result "alert")))
(t/is (str/includes? result "<a")))))
(t/deftest sanitize-svg-preserved-prefixed-attrs
(t/testing "sanitize-svg preserves legitimate attributes written with a namespace prefix"
(let [svg "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xml=\"http://www.w3.org/XML/1998/namespace\"><text xml:space=\"preserve\">hola</text></svg>"
result (svg/sanitize-svg svg)]
(t/is (clojure.string/includes? result "xml:space"))
(t/is (clojure.string/includes? result "hola")))))
(t/is (str/includes? result "xml:space"))
(t/is (str/includes? result "hola")))))
;; --- SVG sanitization: escaping on serialization
;;
;; The sanitizer filters dangerous elements and attributes by name, so the
;; escaping happens when the tree is written back. A payload hidden in an
;; attribute value or in a text node must not become live markup again.
(def ^:private clean-svg
"<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"10\" height=\"10\"><circle r=\"4\"/></svg>")
(def ^:private evil-attribute-svg
"<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"100\" height=\"100\" id=\"x' onload='alert(1)\"><rect width=\"10\" height=\"10\"/></svg>")
(def ^:private evil-text-svg
"<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"100\" height=\"100\"><text>&lt;/text&gt;&lt;script&gt;alert(1)&lt;/script&gt;&lt;text&gt;</text></svg>")
(def ^:private evil-cdata-svg
"<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"100\" height=\"100\"><text><![CDATA[</text><script>alert(1)</script><text>]]></text></svg>")
(def ^:private evil-javascript-svg
"An SVG with a javascript: URL obfuscated with a tab inside the scheme."
"<svg xmlns=\"http://www.w3.org/2000/svg\"><a href=\"java&#x09;script:alert(1)\">x</a></svg>")
(def ^:private xhtml-document
"A document that is not an SVG at all, with HTML only elements."
"<html xmlns=\"http://www.w3.org/1999/xhtml\"><body><iframe srcdoc=\"&lt;script&gt;alert(1)&lt;/script&gt;\"/><object data=\"data:text/html,x\"/><meta http-equiv=\"refresh\" content=\"0;url=javascript:alert(1)\"/></body></html>")
(t/deftest sanitize-svg-escapes-attribute-values
(t/testing "an event handler smuggled in an attribute value is escaped"
(let [result (svg/sanitize-svg evil-attribute-svg)
parsed (svg/parse-svg result)]
(t/is (not (contains? (:attrs parsed) :onload)))
(t/is (= "x' onload='alert(1)"
(get (:attrs parsed) :id)))
(t/is (str/includes? result "&apos;")))))
(t/deftest sanitize-svg-escapes-text-content
(t/testing "entity encoded markup in a text node is escaped"
(let [result (svg/sanitize-svg evil-text-svg)]
(t/is (not (str/includes? result "<script")))
(t/is (str/includes? result "&lt;script&gt;")))))
(t/deftest sanitize-svg-escapes-cdata-content
(t/testing "markup hidden in a CDATA node is escaped"
(let [result (svg/sanitize-svg evil-cdata-svg)]
(t/is (not (str/includes? result "<script")))
(t/is (str/includes? result "&lt;script&gt;")))))
(t/deftest sanitize-svg-escapes-ampersands
(t/testing "ampersands in text and attribute values are escaped"
(let [result (svg/sanitize-svg "<svg xmlns=\"http://www.w3.org/2000/svg\" id=\"a&amp;b\"><text>a &amp; b</text></svg>")]
(t/is (str/includes? result "a &amp; b"))
(t/is (= "a&b"
(get-in (svg/parse-svg result) [:attrs :id]))))))
(t/deftest sanitize-svg-output-is-valid-xml
(t/testing "the sanitized output can be parsed back"
(doseq [payload [clean-svg
evil-attribute-svg
evil-text-svg
evil-cdata-svg]]
(t/is (some? (svg/parse-svg (svg/sanitize-svg payload)))))))
(t/deftest sanitize-svg-rejects-non-svg-documents
(t/testing "a document that is not an svg document is rejected"
(doseq [payload [xhtml-document
;; an svg element declared in the xhtml namespace
"<svg xmlns=\"http://www.w3.org/1999/xhtml\"><rect/></svg>"
;; an svg element without any namespace
"<svg><rect/></svg>"]]
(let [error (try (svg/sanitize-svg payload) nil
(catch clojure.lang.ExceptionInfo e e))]
(t/is (some? error))
(t/is (th/ex-of-type? error :validation))
(t/is (th/ex-of-code? error :invalid-svg-file))))))
(t/deftest sanitize-svg-rejects-non-svg-root-local-name
(t/testing "a root in the svg namespace whose local name is not svg is rejected"
(doseq [payload ["<rect xmlns=\"http://www.w3.org/2000/svg\" width=\"10\" height=\"10\"/>"
"<html xmlns=\"http://www.w3.org/2000/svg\"></html>"]]
(let [error (try (svg/sanitize-svg payload) nil
(catch clojure.lang.ExceptionInfo e e))]
(t/is (some? error))
(t/is (th/ex-of-code? error :invalid-svg-file))))))
(t/deftest sanitize-svg-reports-non-svg-root
(t/testing "a root without the svg namespace reports it, not a serializer failure"
(let [error (try (svg/sanitize-svg "<svg><rect/></svg>") nil
(catch clojure.lang.ExceptionInfo e e))]
(t/is (some? error))
(t/is (th/ex-of-code? error :invalid-svg-file))
(t/is (str/includes? (ex-message error) "not an svg document")))))
(t/deftest sanitize-svg-accepts-svg-roots
(t/testing "an svg root in the svg namespace is accepted"
(t/is (some? (svg/sanitize-svg "<svg xmlns=\"http://www.w3.org/2000/svg\"><rect/></svg>"))))
(t/testing "an svg root with a namespace prefix is accepted"
(t/is (some? (svg/sanitize-svg "<svg:svg xmlns:svg=\"http://www.w3.org/2000/svg\"><svg:rect/></svg:svg>")))))
(t/deftest sanitize-svg-upload-rejects-invalid-svg
(t/testing "the helper rejects an upload that does not hold a valid svg"
(doseq [content ["<svg xmlns=\"http://www.w3.org/2000/svg\"><rect>"
xhtml-document]]
(th/with-temp-file [path content]
(let [upload {:filename "evil.svg"
:path path
:mtype "image/svg+xml"
:size (fs/size path)}
error (try (svg/sanitize-svg-upload upload) nil
(catch clojure.lang.ExceptionInfo e e))]
(t/is (some? error))
(t/is (th/ex-of-code? error :invalid-svg-file)))))))
(t/deftest sanitize-svg-rejects-doctype
(t/testing "a document with a doctype and an internal subset is rejected"
(let [error (try (svg/sanitize-svg
"<!DOCTYPE svg [<!ENTITY x \"y\">]><svg xmlns=\"http://www.w3.org/2000/svg\"><text>&x;</text></svg>")
nil
(catch clojure.lang.ExceptionInfo e e))]
(t/is (some? error))
(t/is (th/ex-of-code? error :invalid-svg-file)))))
(t/deftest sanitize-svg-escapes-double-quote
(t/testing "a double quote in an attribute value is escaped"
(let [result (svg/sanitize-svg "<svg xmlns=\"http://www.w3.org/2000/svg\" id=\"a&quot;b\"/>")
parsed (svg/parse-svg result)]
(t/is (str/includes? result "&quot;"))
(t/is (= "a\"b" (get (:attrs parsed) :id))))))
(t/deftest sanitize-svg-filters-prefixed-root-children
(t/testing "dangerous elements and attributes are removed inside a prefixed root"
(let [result (svg/sanitize-svg "<svg:svg xmlns:svg=\"http://www.w3.org/2000/svg\"><svg:script>alert(1)</svg:script><svg:rect onload=\"alert(1)\"/></svg:svg>")
parsed (svg/parse-svg result)]
(t/is (not (str/includes? result "script")))
(t/is (not (str/includes? result "onload")))
(t/is (= 1 (count (:content parsed)))))))
(t/deftest sanitize-svg-rejects-wrong-namespace
(t/testing "a root that is not in the svg namespace is rejected"
(doseq [payload ["<svg xmlns:foo=\"http://www.w3.org/2000/svg\"><rect/></svg>"
"<svg xmlns=\"\"><rect/></svg>"
"<svg xmlns=\" http://www.w3.org/2000/svg \"><rect/></svg>"
"<svg xmlns=\"http://www.w3.org/2000/svg&#9;\"><rect/></svg>"]]
(let [error (try (svg/sanitize-svg payload) nil
(catch clojure.lang.ExceptionInfo e e))]
(t/is (some? error))
(t/is (th/ex-of-code? error :invalid-svg-file))))))
;; --- SVG sanitization: namespace in scope
;;
;; The XML parser does not process namespaces, so a child can rebind the
;; default namespace to XHTML and smuggle live HTML (iframe/srcdoc/meta).
;; The sanitizer drops any element that is not in the SVG namespace.
(def ^:private xhtml-child-svg
"An SVG whose root is in the SVG namespace but whose child rebinds the
default namespace to XHTML and embeds live HTML."
"<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"100\" height=\"100\"><g xmlns=\"http://www.w3.org/1999/xhtml\"><iframe srcdoc=\"&lt;script&gt;alert(1)&lt;/script&gt;\"/><meta http-equiv=\"refresh\" content=\"0;url=javascript:alert(1)\"/></g><rect width=\"10\" height=\"10\"/></svg>")
(t/deftest sanitize-svg-drops-non-svg-namespace-subtrees
(t/testing "a child that rebinds the default namespace to XHTML is dropped"
(let [result (svg/sanitize-svg xhtml-child-svg)]
(t/is (not (str/includes? result "iframe")))
(t/is (not (str/includes? result "meta")))
(t/is (not (str/includes? result "srcdoc")))
(t/is (not (str/includes? result "http-equiv")))
(t/is (str/includes? result "<rect")))))
(t/deftest sanitize-svg-drops-prefixed-non-svg-namespace-subtrees
(t/testing "a child whose prefix is bound to XHTML is dropped"
(let [result (svg/sanitize-svg
"<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:h=\"http://www.w3.org/1999/xhtml\"><h:iframe srcdoc=\"&lt;script&gt;alert(1)&lt;/script&gt;\"/></svg>")]
(t/is (not (str/includes? result "iframe")))
(t/is (not (str/includes? result "srcdoc"))))))
(t/deftest sanitize-svg-keeps-prefixed-svg-namespace-subtrees
(t/testing "a child whose prefix is bound to the SVG namespace is kept"
(let [result (svg/sanitize-svg
"<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:s=\"http://www.w3.org/2000/svg\"><s:rect width=\"10\" height=\"10\"/></svg>")]
(t/is (str/includes? result "<s:rect")))))
(def ^:private default-prefix-xhtml-svg
"An SVG whose root declares a prefix named `default` bound to XHTML."
"<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:default=\"http://www.w3.org/1999/xhtml\"><default:iframe srcdoc=\"&lt;script&gt;alert(1)&lt;/script&gt;\"/><default:meta http-equiv=\"refresh\" content=\"0;url=javascript:alert(1)\"/></svg>")
(t/deftest sanitize-svg-drops-default-prefix-xhtml-subtrees
(t/testing "a prefix named default bound to XHTML is not the default namespace"
(let [result (svg/sanitize-svg default-prefix-xhtml-svg)]
(t/is (not (str/includes? result "default:iframe")))
(t/is (not (str/includes? result "default:meta")))
(t/is (not (str/includes? result "srcdoc")))
(t/is (not (str/includes? result "http-equiv"))))))
(def ^:private xmlns-onload-svg
"An SVG that declares a prefix named like an event handler."
"<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:onload=\"http://www.w3.org/2000/svg\"><onload:rect width=\"10\" height=\"10\"/></svg>")
(t/deftest sanitize-svg-keeps-xmlns-declarations
(t/testing "a namespace declaration whose local name looks like an event handler is kept"
(let [result (svg/sanitize-svg xmlns-onload-svg)]
(t/is (str/includes? result "xmlns:onload"))
(t/is (str/includes? result "<onload:rect")))))
(t/deftest sanitize-svg-drops-html-only-tags
(t/testing "HTML-only elements written in the svg namespace are dropped"
(doseq [tag ["iframe" "meta" "embed" "link" "base"]]
(let [result (svg/sanitize-svg
(str "<svg xmlns=\"http://www.w3.org/2000/svg\"><" tag " src=\"https://evil\"/></svg>"))]
(t/is (not (str/includes? result (str "<" tag))))))))
(t/deftest sanitize-svg-javascript-href-control-char
(t/testing "a javascript: URL with a control char inside the scheme is removed"
(let [result (svg/sanitize-svg evil-javascript-svg)
parsed (svg/parse-svg result)]
(t/is (not (contains? (get-in parsed [:content 0 :attrs]) :href)))
(t/is (not (str/includes? result "script:alert")))))
(t/testing "a javascript: URL with a line break inside the scheme is removed"
(let [result (svg/sanitize-svg "<svg xmlns=\"http://www.w3.org/2000/svg\"><a href=\"java&#x0a;script:alert(1)\">x</a></svg>")]
(t/is (not (str/includes? result "script:alert"))))))
(t/deftest sanitize-svg-preserves-safe-href
(t/testing "a safe href is preserved"
(let [parsed (svg/parse-svg (svg/sanitize-svg "<svg xmlns=\"http://www.w3.org/2000/svg\"><a href=\"https://example.com/\">x</a></svg>"))]
(t/is (= "https://example.com/"
(get-in parsed [:content 0 :attrs :href]))))))
(t/deftest sanitize-svg-rejects-script-root
(t/testing "a document whose root is a script is rejected as not an svg document"
(let [error (try (svg/sanitize-svg "<script>alert(1)</script>") nil
(catch clojure.lang.ExceptionInfo e e))]
(t/is (th/ex-of-type? error :validation))
(t/is (th/ex-of-code? error :invalid-svg-file))
(t/is (str/includes? (ex-message error) "not an svg document")))))
(t/deftest sanitize-svg-is-idempotent
(t/testing "sanitizing an already sanitized document does not change it"
(doseq [payload [clean-svg
evil-attribute-svg
evil-text-svg
evil-cdata-svg
evil-javascript-svg]]
(let [once (svg/sanitize-svg payload)]
(t/is (= once (svg/sanitize-svg once)))))))
(t/deftest get-basic-info-from-svg-accepts-prefixed-root
(t/testing "an svg root with a namespace prefix provides its dimensions"
(let [info (svg/get-basic-info-from-svg
(svg/parse-svg "<svg:svg xmlns:svg=\"http://www.w3.org/2000/svg\" width=\"120\" height=\"80\"/>"))]
(t/is (= 120 (:width info)))
(t/is (= 80 (:height info))))))
(t/deftest get-basic-info-from-svg-rejects-non-svg
(t/testing "a document whose root is not an svg element is rejected"
(let [error (try (svg/get-basic-info-from-svg
(svg/parse-svg "<html xmlns=\"http://www.w3.org/1999/xhtml\"/>"))
nil
(catch clojure.lang.ExceptionInfo e e))]
(t/is (some? error))
(t/is (th/ex-of-code? error :unable-to-parse-svg)))))
(t/deftest sanitize-svg-upload-sanitizes-svg
(t/testing "an SVG upload is replaced by a sanitized copy"
(th/with-temp-file [path evil-attribute-svg]
(let [upload {:filename "evil.svg"
:path path
:mtype "image/svg+xml"
:size (fs/size path)}
result (svg/sanitize-svg-upload upload)
parsed (svg/parse-svg (slurp (str (:path result))))]
(t/is (not= path (:path result)))
(t/is (not (contains? (:attrs parsed) :onload)))
(t/is (= "x' onload='alert(1)"
(get (:attrs parsed) :id)))
(t/is (= (fs/size (:path result)) (:size result)))))))
(t/deftest sanitize-svg-upload-passes-non-svg
(t/testing "an upload that is not an SVG is returned untouched"
(let [path (th/tempfile "backend_tests/test_files/sample.jpg")
upload {:filename "sample.jpg"
:path path
:mtype "image/jpeg"
:size 312043}]
(t/is (= upload (svg/sanitize-svg-upload upload))))))
(t/deftest info-invalid-image
(t/testing "info on invalid image raises error"

200
backend/test/backend_tests/media_validation_test.clj Normal file
View File

@ -0,0 +1,200 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
(ns backend-tests.media-validation-test
(:require
[app.common.media :as cm]
[app.media.validation :as media.v]
[backend-tests.helpers :as th]
[clojure.test :as t]
[datoteka.fs :as fs]
[datoteka.io :as io]))
(t/deftest check-media-type
(t/testing "matching png passes through"
(let [path (th/tempfile "backend_tests/test_files/sample.png")
upload {:filename "sample.png" :size 1234 :path path :mtype "image/png"}]
(t/is (= upload (media.v/check-media-type upload)))))
(t/testing "matching jpeg passes through"
(let [path (th/tempfile "backend_tests/test_files/sample.jpg")
upload {:filename "sample.jpg" :size 7923 :path path :mtype "image/jpeg"}]
(t/is (= upload (media.v/check-media-type upload)))))
(t/testing "svg with prolog passes through"
(let [path (th/tempfile "backend_tests/test_files/sample2.svg")
upload {:filename "sample.svg" :size 1234 :path path :mtype "image/svg+xml"}]
(t/is (= upload (media.v/check-media-type upload)))))
(t/testing "svg with comment and doctype passes through"
(let [path (fs/create-tempfile :dir "/tmp/penpot" :prefix "penpot-test-svg-")
_ (spit (str path) (str "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n"
"<!-- a comment -->\n"
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\" "
"\"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n"
(slurp (io/resource "backend_tests/test_files/sample1.svg"))))
upload {:filename "sample.svg" :size 1234 :path path :mtype "image/svg+xml"}]
(t/is (= upload (media.v/check-media-type upload)))))
(t/testing "svg with utf-8 comment passes through"
(let [path (fs/create-tempfile :dir "/tmp/penpot" :prefix "penpot-test-usvg-")
_ (spit (str path) (str "<!-- café con ñ y emojis \uD83D\uDE00 -->\n"
(slurp (io/resource "backend_tests/test_files/sample1.svg"))))
upload {:filename "sample.svg" :size 1234 :path path :mtype "image/svg+xml"}]
(t/is (= upload (media.v/check-media-type upload)))))
(t/testing "matching bom svg with prolog passes through"
(let [path (fs/create-tempfile :dir "/tmp/penpot" :prefix "penpot-test-bom-svg-")
_ (spit (str path) (str "\uFEFF<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n"
"<svg xmlns=\"http://www.w3.org/2000/svg\"><rect width=\"10\" height=\"10\"/></svg>"))
upload {:filename "bom.svg" :size 1234 :path path :mtype "image/svg+xml"}]
(t/is (= upload (media.v/check-media-type upload)))))
(t/testing "matching bom svg without prolog passes through"
(let [path (fs/create-tempfile :dir "/tmp/penpot" :prefix "penpot-test-bom-svg2-")
_ (spit (str path) (str "\uFEFF<svg xmlns=\"http://www.w3.org/2000/svg\"><rect width=\"10\" height=\"10\"/></svg>"))
upload {:filename "bom2.svg" :size 1234 :path path :mtype "image/svg+xml"}]
(t/is (= upload (media.v/check-media-type upload)))))
(t/testing "bom plain text declared as svg is rejected"
(let [path (fs/create-tempfile :dir "/tmp/penpot" :prefix "penpot-test-bom-txt-")
_ (spit (str path) "\uFEFFjust some text, no markup at all")
upload {:filename "evil.svg" :size 1234 :path path :mtype "image/svg+xml"}
error (try (media.v/check-media-type upload) nil
(catch clojure.lang.ExceptionInfo e e))]
(t/is (some? error))
(t/is (th/ex-of-type? error :validation))
(t/is (th/ex-of-code? error :media-type-mismatch))))
(t/testing "plain text declared as svg is rejected"
(let [path (fs/create-tempfile :dir "/tmp/penpot" :prefix "penpot-test-txt-")
_ (spit (str path) "just some text, no markup at all")
upload {:filename "evil.svg" :size 1234 :path path :mtype "image/svg+xml"}
error (try (media.v/check-media-type upload) nil
(catch clojure.lang.ExceptionInfo e e))]
(t/is (some? error))
(t/is (th/ex-of-type? error :validation))
(t/is (th/ex-of-code? error :media-type-mismatch))))
(t/testing "uppercase svg root is rejected"
(let [path (fs/create-tempfile :dir "/tmp/penpot" :prefix "penpot-test-usvg-")
_ (spit (str path) "<SVG xmlns=\"http://www.w3.org/2000/svg\"></SVG>")
upload {:filename "upper.svg" :size 1234 :path path :mtype "image/svg+xml"}
error (try (media.v/check-media-type upload) nil
(catch clojure.lang.ExceptionInfo e e))]
(t/is (some? error))
(t/is (th/ex-of-type? error :validation))
(t/is (th/ex-of-code? error :media-type-mismatch))))
(t/testing "matching font passes through"
(let [path (th/tempfile "backend_tests/test_files/font-1.ttf")
upload {:filename "font.ttf" :size 1234 :path path :mtype "font/ttf"}]
(t/is (= upload (media.v/check-media-type upload cm/font-types)))))
(t/testing "matching woff passes through"
(let [path (th/tempfile "backend_tests/test_files/font-1.woff")
upload {:filename "font.woff" :size 1234 :path path :mtype "font/woff"}]
(t/is (= upload (media.v/check-media-type upload cm/font-types)))))
(t/testing "matching woff2 passes through"
(let [path (th/tempfile "backend_tests/test_files/font-1.woff2")
upload {:filename "font.woff2" :size 1234 :path path :mtype "font/woff2"}]
(t/is (= upload (media.v/check-media-type upload cm/font-types)))))
(t/testing "matching otf passes through"
(let [path (th/tempfile "backend_tests/test_files/font-1.otf")
upload {:filename "font.otf" :size 1234 :path path :mtype "font/otf"}]
(t/is (= upload (media.v/check-media-type upload cm/font-types)))))
(t/testing "matching webp passes through"
(let [path (th/tempfile "backend_tests/test_files/sample.webp")
upload {:filename "sample.webp" :size 1234 :path path :mtype "image/webp"}]
(t/is (= upload (media.v/check-media-type upload)))))
(t/testing "png bytes declared as jpeg are rejected"
(let [path (th/tempfile "backend_tests/test_files/sample.png")
upload {:filename "sample.png" :size 1234 :path path :mtype "image/jpeg"}
error (try (media.v/check-media-type upload) nil
(catch clojure.lang.ExceptionInfo e e))]
(t/is (some? error))
(t/is (th/ex-of-type? error :validation))
(t/is (th/ex-of-code? error :media-type-mismatch))))
(t/testing "octet-stream declared for png is rejected"
(let [path (th/tempfile "backend_tests/test_files/sample.png")
upload {:filename "sample.png" :size 1234 :path path :mtype "application/octet-stream"}
error (try (media.v/check-media-type upload) nil
(catch clojure.lang.ExceptionInfo e e))]
(t/is (some? error))
(t/is (th/ex-of-type? error :validation))
(t/is (th/ex-of-code? error :media-type-not-allowed))))
(t/testing "png bytes declared as svg are rejected"
(let [path (th/tempfile "backend_tests/test_files/sample.png")
upload {:filename "sample.png" :size 1234 :path path :mtype "image/svg+xml"}
error (try (media.v/check-media-type upload) nil
(catch clojure.lang.ExceptionInfo e e))]
(t/is (some? error))
(t/is (th/ex-of-type? error :validation))
(t/is (th/ex-of-code? error :media-type-mismatch))))
(t/testing "disallowed real type is rejected"
(let [path (th/tempfile "backend_tests/test_files/file-with-library.penpot")
upload {:filename "bundle.penpot" :size 1234 :path path :mtype "application/zip"}
error (try (media.v/check-media-type upload) nil
(catch clojure.lang.ExceptionInfo e e))]
(t/is (some? error))
(t/is (th/ex-of-type? error :validation))
(t/is (th/ex-of-code? error :media-type-not-allowed)))))
(defn- sparse-file!
"Grow the file at `path` to `size` bytes without writing them. The
file stays sparse so no disk is used, while `fs/size` still reports
the full length."
[path size]
(with-open [raf (java.io.RandomAccessFile. (str path) "rw")]
(.setLength raf size))
path)
(t/deftest check-media-size
(t/testing "small file passes even when the declared size lies upwards"
(let [path (th/tempfile "backend_tests/test_files/sample.jpg")
upload {:filename "sample.jpg" :size (* 1024 1024 31) :path path :mtype "image/jpeg"}]
(t/is (= upload (media.v/check-media-size upload)))))
(t/testing "file over the limit is rejected even when declared size lies downwards"
(let [path (sparse-file! (fs/create-tempfile :dir "/tmp/penpot" :prefix "penpot-test-big-")
(* 1024 1024 31))
upload {:filename "big.jpg" :size 1 :path path :mtype "image/jpeg"}
error (try (media.v/check-media-size upload) nil
(catch clojure.lang.ExceptionInfo e e))]
(t/is (some? error))
(t/is (th/ex-of-type? error :restriction))
(t/is (th/ex-of-code? error :media-max-file-size-reached))))
(t/testing "missing file fails closed with the path code"
(let [upload {:filename "gone.jpg" :size 1 :path (fs/path "/tmp/penpot/no-such-file-12345.jpg")
:mtype "image/jpeg"}
error (try (media.v/check-media-size upload) nil
(catch clojure.lang.ExceptionInfo e e))]
(t/is (some? error))
(t/is (th/ex-of-type? error :validation))
(t/is (th/ex-of-code? error :params-validation)))))
(t/deftest check-font-size
(t/testing "small font passes even when the declared size lies upwards"
(let [path (th/tempfile "backend_tests/test_files/font-1.ttf")
upload {:filename "font.ttf" :size (* 1024 1024 31) :path path :mtype "font/ttf"}]
(t/is (= upload (media.v/check-font-size upload)))))
(t/testing "font over the limit is rejected even when declared size lies downwards"
(let [path (sparse-file! (fs/create-tempfile :dir "/tmp/penpot" :prefix "penpot-test-big-font-")
(* 1024 1024 31))
upload {:filename "big.ttf" :size 1 :path path :mtype "font/ttf"}
error (try (media.v/check-font-size upload) nil
(catch clojure.lang.ExceptionInfo e e))]
(t/is (some? error))
(t/is (th/ex-of-type? error :restriction))
(t/is (th/ex-of-code? error :font-max-file-size-reached)))))
(t/deftest check-upload-path
(t/testing "tempfile under /tmp/penpot passes through"
(let [path (th/tempfile "backend_tests/test_files/sample.jpg")
upload {:filename "sample.jpg" :size 7923 :path path :mtype "image/jpeg"}]
(t/is (= upload (media.v/check-upload-path upload)))))
(t/testing "system file outside temp is rejected"
(let [upload {:filename "x.png" :size 1 :path (fs/path "/etc/passwd") :mtype "image/png"}
error (try (media.v/check-upload-path upload) nil
(catch clojure.lang.ExceptionInfo e e))]
(t/is (some? error))
(t/is (th/ex-of-type? error :validation))
(t/is (th/ex-of-code? error :params-validation))))
(t/testing "missing file is rejected"
(let [upload {:filename "x.png" :size 1 :path (fs/path "/tmp/penpot/no-such-file-12345.png") :mtype "image/png"}
error (try (media.v/check-upload-path upload) nil
(catch clojure.lang.ExceptionInfo e e))]
(t/is (some? error))
(t/is (th/ex-of-type? error :validation))
(t/is (th/ex-of-code? error :params-validation)))))

180
backend/test/backend_tests/migrations_legacy_file_data_test.clj Normal file
View File

@ -0,0 +1,180 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns backend-tests.migrations-legacy-file-data-test
"Covers the 0157 data migration: legacy file/file_change payloads move
to file_data, keeping timestamps and deleted_at, nulling the legacy
column; unlabelled file_change payloads (orphan revisions) are
deleted."
(:require
[app.common.time :as ct]
[app.common.uuid :as uuid]
[app.db :as db]
[app.util.blob :as blob]
[backend-tests.helpers :as th]
[clojure.java.io :as io]
[clojure.test :as t]
[next.jdbc :as jdbc]))
(t/use-fixtures :once th/state-init)
(t/use-fixtures :each th/database-reset)
(defn- run-migration
[]
(let [sql (slurp (io/resource "app/migrations/sql/0157-migrate-legacy-file-data.sql"))]
(jdbc/execute! th/*pool* [sql])))
(defn- get-file-data
[file-id id]
(th/db-get :file-data {:file-id file-id :id id} {::db/remove-deleted false}))
(t/deftest migrates-legacy-file-and-snapshot
(let [profile (th/create-profile* 1)
file (th/create-file* 1 {:profile-id (:id profile)
:project-id (:default-project-id profile)})
fid (:id file)
;; Simulate an unmigrated main: move the payload back to
;; file.data and drop the file_data row.
main-data (:data (get-file-data fid fid))
_ (t/is (some? main-data) "expected file_data main row from create-file*")
file-row (th/db-get :file {:id fid} {::db/remove-deleted false})
_ (th/db-delete! :file-data {:file-id fid :id fid})
_ (th/db-update! :file {:data main-data} {:id fid})
;; Simulate an unmigrated snapshot (labelled) with its own
;; timestamps and a deletion date.
snap-id (uuid/next)
snap-data (blob/encode {:snapshot 1})
created-at (ct/in-past "2h")
updated-at (ct/in-past "1h")
deleted-at (ct/in-past "30m")
_ (th/db-insert! :file-change
{:id snap-id
:file-id fid
:revn 1
:data snap-data
:label "v1"
:created-at created-at
:updated-at updated-at
:deleted-at deleted-at})
;; Orphan revision (data, no label): deleted by the migration.
rev-id (uuid/next)
rev-data (blob/encode {:rev 1})
_ (th/db-insert! :file-change
{:id rev-id
:file-id fid
:revn 2
:data rev-data})
;; Live revision (changes, no data, no label): untouched.
live-id (uuid/next)
_ (th/db-insert! :file-change
{:id live-id
:file-id fid
:revn 3
:changes (blob/encode [{:type :dummy}])})]
(run-migration)
(t/testing "main payload moves to file_data as type main"
(let [fd (get-file-data fid fid)]
(t/is (some? fd))
(t/is (= "main" (:type fd)))
(t/is (= "db" (:backend fd)))
(t/is (nil? (:metadata fd)))
(t/is (= (seq ^bytes main-data) (seq ^bytes (:data fd))))
(t/is (= (:created-at file-row) (:created-at fd)))
(t/is (= (:modified-at file-row) (:modified-at fd)))
(t/is (= (:deleted-at file-row) (:deleted-at fd)))))
(t/testing "snapshot payload moves to file_data as type snapshot"
(let [fd (get-file-data fid snap-id)]
(t/is (some? fd))
(t/is (= "snapshot" (:type fd)))
(t/is (= "db" (:backend fd)))
(t/is (nil? (:metadata fd)))
(t/is (= (seq ^bytes snap-data) (seq ^bytes (:data fd))))
;; NOTE: timestamptz stores microsecond precision while the
;; test instants carry nanoseconds, so compare at millis.
(t/is (= (inst-ms created-at) (inst-ms (:created-at fd))))
(t/is (= (inst-ms updated-at) (inst-ms (:modified-at fd))))
(t/is (= (inst-ms deleted-at) (inst-ms (:deleted-at fd))))))
(t/testing "legacy columns are nulled"
(t/is (nil? (:data (th/db-get :file {:id fid} {::db/remove-deleted false}))))
(t/is (nil? (:data (th/db-get :file-change {:id snap-id} {::db/remove-deleted false})))))
(t/testing "unlabelled payload row is deleted, live revision stays"
(t/is (nil? (th/db-get :file-change {:id rev-id} {::db/remove-deleted false})))
(let [row (th/db-get :file-change {:id live-id} {::db/remove-deleted false})]
(t/is (some? row))
(t/is (some? (:changes row)))))
(t/testing "second run is a no-op"
(run-migration)
(t/is (some? (get-file-data fid fid)))
(t/is (some? (get-file-data fid snap-id)))
(t/is (nil? (:data (th/db-get :file {:id fid} {::db/remove-deleted false}))))
(t/is (nil? (:data (th/db-get :file-change {:id snap-id} {::db/remove-deleted false}))))
(t/is (nil? (th/db-get :file-change {:id rev-id} {::db/remove-deleted false}))))))
(t/deftest keeps-preexisting-file-data-row
(let [profile (th/create-profile* 1)
file (th/create-file* 1 {:profile-id (:id profile)
:project-id (:default-project-id profile)})
fid (:id file)
;; file_data already holds the current payload; the legacy
;; column keeps a stale copy that must not overwrite it.
current (:data (get-file-data fid fid))
stale (blob/encode {:stale true})
_ (t/is (some? current))
_ (t/is (not= (seq ^bytes current) (seq ^bytes stale)))
_ (th/db-update! :file {:data stale} {:id fid})]
(run-migration)
(t/testing "file_data keeps its data and legacy is nulled"
(let [fd (get-file-data fid fid)]
(t/is (= (seq ^bytes current) (seq ^bytes (:data fd)))))
(t/is (nil? (:data (th/db-get :file {:id fid} {::db/remove-deleted false})))))))
(t/deftest keeps-preexisting-snapshot-row
(let [profile (th/create-profile* 1)
file (th/create-file* 1 {:profile-id (:id profile)
:project-id (:default-project-id profile)})
fid (:id file)
;; file_data already holds the snapshot payload; the legacy column
;; keeps a stale copy that must not overwrite it.
snap-id (uuid/next)
current (blob/encode {:snapshot :current})
stale (blob/encode {:snapshot :stale})
_ (t/is (not= (seq ^bytes current) (seq ^bytes stale)))
_ (th/db-insert! :file-data
{:file-id fid
:id snap-id
:type "snapshot"
:backend "db"
:metadata nil
:data current})
_ (th/db-insert! :file-change
{:id snap-id
:file-id fid
:revn 1
:label "v1"
:data stale})]
(run-migration)
(t/testing "file_data keeps its data and legacy is nulled"
(let [fd (get-file-data fid snap-id)]
(t/is (= (seq ^bytes current) (seq ^bytes (:data fd)))))
(t/is (nil? (:data (th/db-get :file-change {:id snap-id}
{::db/remove-deleted false})))))))

44
backend/test/backend_tests/rpc_auth_test.clj
View File

@ -14,9 +14,13 @@
[app.http.session :as session]
[app.loggers.audit :as-alias audit]
[app.redis :as rds]
[app.rpc.commands.auth :as auth]
[app.rpc.commands.ldap :as ldap-cmd]
[app.storage :as sto]
[backend-tests.helpers :as th]
[clojure.test :as t]
[datoteka.io :as io]
[mockery.core :refer [with-mocks]]
[yetti.response :as yres])
(:import
java.lang.AutoCloseable))
@ -122,7 +126,7 @@
(.close ^AutoCloseable conn)))))
(t/deftest login-with-password-locks-after-max-attempts
(with-redefs [app.config/flags (flags/parse flags/default [:enable-account-lockout])]
(with-redefs [app.config/flags (flags/parse flags/default [:enable-account-lockout :disable-admin-console])]
(let [profile (th/create-profile* 200 {:is-active true})]
(cleanup-redis (:id profile))
;; 4 failed attempts — not yet locked
@ -141,7 +145,7 @@
(t/is (th/ex-of-code? (:error out) :account-locked))))))
(t/deftest login-with-password-returns-rate-limit-when-locked
(with-redefs [app.config/flags (flags/parse flags/default [:enable-account-lockout])]
(with-redefs [app.config/flags (flags/parse flags/default [:enable-account-lockout :disable-admin-console])]
(let [profile (th/create-profile* 201 {:is-active true})]
(cleanup-redis (:id profile))
;; Trigger lockout
@ -161,7 +165,7 @@
(t/is (pos? ttl) "ttl should be a positive number"))))))
(t/deftest successful-login-clears-failed-attempts
(with-redefs [app.config/flags (flags/parse flags/default [:enable-account-lockout])]
(with-redefs [app.config/flags (flags/parse flags/default [:enable-account-lockout :disable-admin-console])]
(let [profile (th/create-profile* 202 {:is-active true})]
(cleanup-redis (:id profile))
;; 3 failed attempts
@ -184,7 +188,7 @@
(t/is (th/ex-of-code? (:error out) :wrong-credentials)))))))
(t/deftest lockout-does-not-apply-when-flag-disabled
(with-redefs [app.config/flags (flags/parse flags/default [])]
(with-redefs [app.config/flags (flags/parse flags/default [:disable-admin-console])]
(let [profile (th/create-profile* 203 {:is-active true})]
(cleanup-redis (:id profile))
;; 10 failed attempts — no lockout without the flag
@ -199,7 +203,7 @@
"Verify that login-with-ldap short-circuits when account is locked,
without attempting LDAP authentication. Calls the actual command
function with a mock provider to test the full flow."
(with-redefs [app.config/flags (flags/parse flags/default [:enable-account-lockout :login-with-ldap])
(with-redefs [app.config/flags (flags/parse flags/default [:enable-account-lockout :login-with-ldap :disable-admin-console])
ldap/authenticate (constantly nil)]
(let [profile (th/create-profile* 204 {:is-active true})
cfg (assoc th/*system* ::ldap/provider {})]
@ -217,7 +221,7 @@
(t/deftest recover-profile-clears-lockout
"Verify that clearing attempts (e.g. after password reset) unlocks the account."
(with-redefs [app.config/flags (flags/parse flags/default [:enable-account-lockout])]
(with-redefs [app.config/flags (flags/parse flags/default [:enable-account-lockout :disable-admin-console])]
(let [profile (th/create-profile* 205 {:is-active true})
cfg th/*system*]
(cleanup-redis (:id profile))
@ -253,3 +257,31 @@
"session must not be bound to the typed-email profile")))))
;; --- Profile picture import
(def ^:private avatar-svg
"An SVG that smuggles a script element."
"<svg xmlns=\"http://www.w3.org/2000/svg\"><script>alert(1)</script></svg>")
(defn- mock-download
"Mock the http client so that any url answers with `body` and `mtype`."
[mtype body]
{:target 'app.http.client/req-with-redirects
:return {:status 200
:headers {"content-type" mtype
"content-length" (str (alength ^bytes body))}
:body (java.io.ByteArrayInputStream. ^bytes body)}})
(t/deftest import-profile-picture-rejects-svg
(t/testing "an svg profile picture is not imported"
(with-mocks [http-mock (mock-download "image/svg+xml"
(.getBytes ^String avatar-svg "UTF-8"))]
(let [result (auth/import-profile-picture th/*system* "https://example.com/avatar.svg")]
(t/is (nil? result)))))
(t/testing "a png profile picture is imported"
(let [body (io/read* (io/resource "backend_tests/test_files/sample.png"))]
(with-mocks [http-mock (mock-download "image/png" body)]
(let [result (auth/import-profile-picture th/*system* "https://example.com/avatar.png")]
(t/is (some? result))
(t/is (some? (sto/get-object (:app.storage/storage th/*system*) result))))))))

565
backend/test/backend_tests/rpc_comment_test.clj
View File

@ -323,6 +323,14 @@
:frame-id uuid/zero
:share-id share-id})]
(t/is (not (th/success? out)))
(t/is (= :not-found (th/ex-type (:error out))))))
(t/testing "outsider with who-comment=team share-link cannot get-profiles-for-file-comments"
(let [out (th/command! {::th/type :get-profiles-for-file-comments
::rpc/profile-id (:id outsider)
:file-id (:id file)
:share-id share-id})]
(t/is (not (th/success? out)))
(t/is (= :not-found (th/ex-type (:error out))))))))
(t/deftest share-link-who-comment-all-can-comment
@ -449,6 +457,292 @@
(t/is (not (th/success? out)))
(t/is (= :not-found (th/ex-type (:error out))))))))
(t/deftest share-link-page-scope-enforced-for-comment-profiles
(let [owner (th/create-profile* 1 {:is-active true})
outsider (th/create-profile* 2 {:is-active true})
member (th/create-profile* 3 {:is-active true})
team (th/create-team* 1 {:profile-id (:id owner)})
_ (th/create-team-role* {:team-id (:id team)
:profile-id (:id member)
:role :editor})
project (th/create-project* 1 {:team-id (:id team)
:profile-id (:id owner)})
file (th/create-file* 1 {:profile-id (:id owner)
:project-id (:id project)})
page-a (get-in file [:data :pages 0])
page-b (uuid/random)
_ (th/command! {::th/type :update-file
::rpc/profile-id (:id owner)
:id (:id file)
:session-id (uuid/random)
:revn 0
:vern 0
:changes [{:type :add-page
:id page-b
:page {:id page-b
:name "Page B"
:options {}
:objects {}}}]})
thread-a (th/command! {::th/type :create-comment-thread
::rpc/profile-id (:id owner)
:file-id (:id file)
:page-id page-a
:position (gpt/point 0)
:content "comment on page A"
:frame-id uuid/zero})
thread-b (th/command! {::th/type :create-comment-thread
::rpc/profile-id (:id member)
:file-id (:id file)
:page-id page-b
:position (gpt/point 0)
:content "comment on page B"
:frame-id uuid/zero})
thread-a-id (get-in thread-a [:result :id])
thread-b-id (get-in thread-b [:result :id])
thread-b-comment-id (get-in thread-b [:result :comment-id])
share (th/command! {::th/type :create-share-link
::rpc/profile-id (:id owner)
:file-id (:id file)
:pages #{page-a}
:who-comment "all"
:who-inspect "all"})
share-id (get-in share [:result :id])
;; A link without pages grants access to nothing.
empty-share (th/command! {::th/type :create-share-link
::rpc/profile-id (:id owner)
:file-id (:id file)
:pages #{}
:who-comment "all"
:who-inspect "all"})
empty-share-id (get-in empty-share [:result :id])]
(t/testing "the comment threads are created"
(t/is (th/success? thread-a))
(t/is (th/success? thread-b)))
(t/testing "share-link holder only gets the profiles of the shared page commenters"
(let [out (th/command! {::th/type :get-profiles-for-file-comments
::rpc/profile-id (:id outsider)
:file-id (:id file)
:share-id share-id})
emails (set (map :email (:result out)))]
(t/is (th/success? out))
(t/is (contains? emails (:email owner)))
(t/is (contains? emails (:email outsider)))
(t/is (not (contains? emails (:email member))))))
(t/testing "share-link holder cannot update the status of a thread of an unshared page"
(let [out (th/command! {::th/type :update-comment-thread-status
::rpc/profile-id (:id outsider)
:id thread-b-id
:share-id share-id})]
(t/is (not (th/success? out)))
(t/is (= :not-found (th/ex-type (:error out))))))
(t/testing "share-link holder cannot update a comment of an unshared page"
(let [out (th/command! {::th/type :update-comment
::rpc/profile-id (:id outsider)
:id thread-b-comment-id
:content "outsider edit on page B"
:share-id share-id})]
(t/is (not (th/success? out)))
(t/is (= :not-found (th/ex-type (:error out))))))
(t/testing "share-link holder cannot delete a comment of an unshared page"
(let [out (th/command! {::th/type :delete-comment
::rpc/profile-id (:id outsider)
:id thread-b-comment-id
:share-id share-id})]
(t/is (not (th/success? out)))
(t/is (= :not-found (th/ex-type (:error out))))))
(t/testing "share-link holder cannot delete a thread of an unshared page"
(let [out (th/command! {::th/type :delete-comment-thread
::rpc/profile-id (:id outsider)
:id thread-b-id
:share-id share-id})]
(t/is (not (th/success? out)))
(t/is (= :not-found (th/ex-type (:error out))))))
(t/testing "share-link holder cannot create a thread on an unshared page"
(let [out (th/command! {::th/type :create-comment-thread
::rpc/profile-id (:id outsider)
:file-id (:id file)
:page-id page-b
:position (gpt/point 0)
:content "comment on page B"
:frame-id uuid/zero
:share-id share-id})]
(t/is (not (th/success? out)))
(t/is (= :not-found (th/ex-type (:error out))))))
(t/testing "share-link holder cannot move a thread of an unshared page"
(let [out (th/command! {::th/type :update-comment-thread-position
::rpc/profile-id (:id outsider)
:id thread-b-id
:position (gpt/point 10 10)
:frame-id uuid/zero
:share-id share-id})]
(t/is (not (th/success? out)))
(t/is (= :not-found (th/ex-type (:error out))))))
(t/testing "share-link holder cannot resize a thread of an unshared page"
(let [out (th/command! {::th/type :update-comment-thread-frame
::rpc/profile-id (:id outsider)
:id thread-b-id
:frame-id uuid/zero
:share-id share-id})]
(t/is (not (th/success? out)))
(t/is (= :not-found (th/ex-type (:error out))))))
(t/testing "share-link holder can move a thread of the shared page"
(let [out (th/command! {::th/type :update-comment-thread-position
::rpc/profile-id (:id outsider)
:id thread-a-id
:position (gpt/point 10 10)
:frame-id uuid/zero
:share-id share-id})]
(t/is (th/success? out))))
(t/testing "share-link holder can resize a thread of the shared page"
(let [out (th/command! {::th/type :update-comment-thread-frame
::rpc/profile-id (:id outsider)
:id thread-a-id
:frame-id uuid/zero
:share-id share-id})]
(t/is (th/success? out))))
(t/testing "share-link holder cannot reply to a thread of an unshared page"
(let [out (th/command! {::th/type :create-comment
::rpc/profile-id (:id outsider)
:thread-id thread-b-id
:content "outsider reply on page B"
:share-id share-id})]
(t/is (not (th/success? out)))
(t/is (= :not-found (th/ex-type (:error out))))))
(t/testing "share-link holder can reply to a thread of the shared page"
(let [out (th/command! {::th/type :create-comment
::rpc/profile-id (:id outsider)
:thread-id thread-a-id
:content "outsider reply on page A"
:share-id share-id})]
(t/is (th/success? out))))
(t/testing "share-link holder cannot resolve a thread of an unshared page"
(let [out (th/command! {::th/type :update-comment-thread
::rpc/profile-id (:id outsider)
:id thread-b-id
:is-resolved true
:share-id share-id})]
(t/is (not (th/success? out)))
(t/is (= :not-found (th/ex-type (:error out))))))
(t/testing "share-link holder can resolve a thread of the shared page"
(let [out (th/command! {::th/type :update-comment-thread
::rpc/profile-id (:id outsider)
:id thread-a-id
:is-resolved true
:share-id share-id})]
(t/is (th/success? out))))
(t/testing "share-link holder can write on the shared page"
(let [thread (th/command! {::th/type :create-comment-thread
::rpc/profile-id (:id outsider)
:file-id (:id file)
:page-id page-a
:position (gpt/point 0)
:content "outsider comment on page A"
:frame-id uuid/zero
:share-id share-id})
thread-id (get-in thread [:result :id])
comment-id (get-in thread [:result :comment-id])]
(t/is (th/success? thread))
(t/testing "can mark a thread as read"
(let [out (th/command! {::th/type :update-comment-thread-status
::rpc/profile-id (:id outsider)
:id thread-id
:share-id share-id})]
(t/is (th/success? out))))
(t/testing "can edit its comment"
(let [out (th/command! {::th/type :update-comment
::rpc/profile-id (:id outsider)
:id comment-id
:content "outsider edit on page A"
:share-id share-id})]
(t/is (th/success? out))))
(t/testing "can delete its comment"
(let [out (th/command! {::th/type :delete-comment
::rpc/profile-id (:id outsider)
:id comment-id
:share-id share-id})]
(t/is (th/success? out))))
(t/testing "can delete its thread"
(let [out (th/command! {::th/type :delete-comment-thread
::rpc/profile-id (:id outsider)
:id thread-id
:share-id share-id})]
(t/is (th/success? out))))))
(t/testing "share-link without pages does not leak the commenter profiles"
(let [out (th/command! {::th/type :get-profiles-for-file-comments
::rpc/profile-id (:id outsider)
:file-id (:id file)
:share-id empty-share-id})
emails (set (map :email (:result out)))]
(t/is (th/success? out))
(t/is (= #{(:email outsider)} emails))))
(t/testing "share-link without pages cannot create a thread"
(let [out (th/command! {::th/type :create-comment-thread
::rpc/profile-id (:id outsider)
:file-id (:id file)
:page-id page-a
:position (gpt/point 0)
:content "outsider comment on page A"
:frame-id uuid/zero
:share-id empty-share-id})]
(t/is (not (th/success? out)))
(t/is (= :not-found (th/ex-type (:error out))))))
(t/testing "share-link without pages lists no comment threads"
(let [out (th/command! {::th/type :get-comment-threads
::rpc/profile-id (:id outsider)
:file-id (:id file)
:share-id empty-share-id})]
(t/is (th/success? out))
(t/is (= [] (:result out)))))
(t/testing "share-link without pages cannot get a comment thread"
(let [out (th/command! {::th/type :get-comment-thread
::rpc/profile-id (:id outsider)
:file-id (:id file)
:id thread-a-id
:share-id empty-share-id})]
(t/is (not (th/success? out)))
(t/is (= :not-found (th/ex-type (:error out))))))
(t/testing "share-link without pages cannot get the comments of a thread"
(let [out (th/command! {::th/type :get-comments
::rpc/profile-id (:id outsider)
:thread-id thread-a-id
:share-id empty-share-id})]
(t/is (not (th/success? out)))
(t/is (= :not-found (th/ex-type (:error out))))))))
(t/deftest membership-can-still-comment
(let [owner (th/create-profile* 1 {:is-active true})
member (th/create-profile* 2 {:is-active true})
@ -461,7 +755,23 @@
:profile-id (:id owner)})
file (th/create-file* 1 {:profile-id (:id owner)
:project-id (:id project)})
page-id (get-in file [:data :pages 0])]
page-id (get-in file [:data :pages 0])
_ (th/command! {::th/type :create-comment-thread
::rpc/profile-id (:id owner)
:file-id (:id file)
:page-id page-id
:position (gpt/point 0)
:content "owner comment"
:frame-id uuid/zero})
thread (th/command! {::th/type :create-comment-thread
::rpc/profile-id (:id member)
:file-id (:id file)
:page-id page-id
:position (gpt/point 0)
:content "member comment"
:frame-id uuid/zero})]
(t/testing "team member can get-comment-threads without share-id"
(let [out (th/command! {::th/type :get-comment-threads
@ -470,14 +780,45 @@
(t/is (th/success? out))))
(t/testing "team member can create-comment-thread without share-id"
(let [out (th/command! {::th/type :create-comment-thread
(t/is (th/success? thread)))
(t/testing "team member can move a thread of the file"
(let [out (th/command! {::th/type :update-comment-thread-position
::rpc/profile-id (:id member)
:file-id (:id file)
:page-id page-id
:position (gpt/point 0)
:content "member comment"
:id (get-in thread [:result :id])
:position (gpt/point 10 10)
:frame-id uuid/zero})]
(t/is (th/success? out))))))
(t/is (th/success? out))))
(t/testing "team member can resize a thread of the file"
(let [out (th/command! {::th/type :update-comment-thread-frame
::rpc/profile-id (:id member)
:id (get-in thread [:result :id])
:frame-id uuid/zero})]
(t/is (th/success? out))))
(t/testing "team member can reply to a thread of the file"
(let [out (th/command! {::th/type :create-comment
::rpc/profile-id (:id member)
:thread-id (get-in thread [:result :id])
:content "member reply"})]
(t/is (th/success? out))))
(t/testing "team member can resolve a thread of the file"
(let [out (th/command! {::th/type :update-comment-thread
::rpc/profile-id (:id member)
:id (get-in thread [:result :id])
:is-resolved true})]
(t/is (th/success? out))))
(t/testing "team member gets the profiles of all the file commenters"
(let [out (th/command! {::th/type :get-profiles-for-file-comments
::rpc/profile-id (:id member)
:file-id (:id file)})
emails (set (map :email (:result out)))]
(t/is (th/success? out))
(t/is (contains? emails (:email member)))
(t/is (contains? emails (:email owner)))))))
(t/deftest comment-audit-events-carry-team-id
;; Comment events carry the file id, so the resolver could find the team; the
@ -507,3 +848,213 @@
(get-in (meta (:result thread)) [:app.loggers.audit/props :team-id])))
(t/is (= (:id team)
(get-in (meta (:result comment)) [:app.loggers.audit/props :team-id]))))))
(t/deftest mark-all-threads-as-read-requires-access-to-the-threads
(let [owner (th/create-profile* 1 {:is-active true})
stranger (th/create-profile* 2 {:is-active true})
team (th/create-team* 1 {:profile-id (:id owner)})
project (th/create-project* 1 {:team-id (:id team)
:profile-id (:id owner)})
file (th/create-file* 1 {:profile-id (:id owner)
:project-id (:id project)})
page-id (get-in file [:data :pages 0])
thread (th/command! {::th/type :create-comment-thread
::rpc/profile-id (:id owner)
:file-id (:id file)
:page-id page-id
:position (gpt/point 0)
:content "private thread"
:frame-id uuid/zero})
thread-id (get-in thread [:result :id])]
(t/testing "the sibling rpc already rejects a profile without access"
(let [out (th/command! {::th/type :update-comment-thread-status
::rpc/profile-id (:id stranger)
:id thread-id})]
(t/is (not (th/success? out)))
(t/is (= :not-found (th/ex-type (:error out))))))
(t/testing "mark-all-threads-as-read rejects a profile without access"
(let [out (th/command! {::th/type :mark-all-threads-as-read
::rpc/profile-id (:id stranger)
:threads [thread-id]})]
(t/is (not (th/success? out)))
(t/is (= :not-found (th/ex-type (:error out)))))
(t/is (nil? (th/db-get :comment-thread-status
{:thread-id thread-id
:profile-id (:id stranger)}))))))
(t/deftest mark-all-threads-as-read-behaviour
(let [owner (th/create-profile* 1 {:is-active true})
member (th/create-profile* 2 {:is-active true})
viewer (th/create-profile* 3 {:is-active true})
outsider (th/create-profile* 4 {:is-active true})
team (th/create-team* 1 {:profile-id (:id owner)})
_ (th/create-team-role* {:team-id (:id team)
:profile-id (:id member)
:role :editor})
_ (th/create-team-role* {:team-id (:id team)
:profile-id (:id viewer)
:role :viewer})
project (th/create-project* 1 {:team-id (:id team)
:profile-id (:id owner)})
file (th/create-file* 1 {:profile-id (:id owner)
:project-id (:id project)})
page-id (get-in file [:data :pages 0])
thread (th/command! {::th/type :create-comment-thread
::rpc/profile-id (:id owner)
:file-id (:id file)
:page-id page-id
:position (gpt/point 0)
:content "hello"
:frame-id uuid/zero})
thread-id (get-in thread [:result :id])]
(t/testing "the thread is unread for a team member"
(let [out (th/command! {::th/type :get-unread-comment-threads
::rpc/profile-id (:id member)
:team-id (:id team)})]
(t/is (th/success? out))
(t/is (= 1 (count (:result out))))))
(t/testing "a team member can mark the threads as read"
(let [out (th/command! {::th/type :mark-all-threads-as-read
::rpc/profile-id (:id member)
:threads [thread-id]})]
(t/is (th/success? out)))
(let [out (th/command! {::th/type :get-unread-comment-threads
::rpc/profile-id (:id member)
:team-id (:id team)})]
(t/is (th/success? out))
(t/is (= 0 (count (:result out))))))
(t/testing "a viewer of the team can mark the threads as read"
(let [out (th/command! {::th/type :mark-all-threads-as-read
::rpc/profile-id (:id viewer)
:threads [thread-id]})]
(t/is (th/success? out))))
(t/testing "an unknown thread is rejected"
(let [out (th/command! {::th/type :mark-all-threads-as-read
::rpc/profile-id (:id member)
:threads [(uuid/random)]})]
(t/is (not (th/success? out)))
(t/is (= :not-found (th/ex-type (:error out))))))
(t/testing "more threads than the allowed maximum are rejected"
(let [out (th/command! {::th/type :mark-all-threads-as-read
::rpc/profile-id (:id member)
:threads (vec (repeatedly 501 uuid/random))})]
(t/is (not (th/success? out)))
(t/is (= :validation (th/ex-type (:error out))))))
(t/testing "a list with an inaccessible thread writes nothing"
(let [other-team (th/create-team* 2 {:profile-id (:id owner)})
other-project (th/create-project* 2 {:team-id (:id other-team)
:profile-id (:id owner)})
other-file (th/create-file* 2 {:profile-id (:id owner)
:project-id (:id other-project)})
other-thread (th/command! {::th/type :create-comment-thread
::rpc/profile-id (:id owner)
:file-id (:id other-file)
:page-id (get-in other-file [:data :pages 0])
:position (gpt/point 0)
:content "other team"
:frame-id uuid/zero})
other-thread-id (get-in other-thread [:result :id])
accessible-thread (th/command! {::th/type :create-comment-thread
::rpc/profile-id (:id owner)
:file-id (:id file)
:page-id page-id
:position (gpt/point 0)
:content "accessible"
:frame-id uuid/zero})
accessible-thread-id (get-in accessible-thread [:result :id])
out (th/command! {::th/type :mark-all-threads-as-read
::rpc/profile-id (:id member)
:threads [accessible-thread-id other-thread-id]})]
(t/is (not (th/success? out)))
(t/is (= :not-found (th/ex-type (:error out))))
(t/is (nil? (th/db-get :comment-thread-status
{:thread-id accessible-thread-id
:profile-id (:id member)})))
(t/is (nil? (th/db-get :comment-thread-status
{:thread-id other-thread-id
:profile-id (:id member)})))))
(t/testing "a profile without any relation with the team is rejected"
(let [out (th/command! {::th/type :mark-all-threads-as-read
::rpc/profile-id (:id outsider)
:threads [thread-id]})]
(t/is (not (th/success? out)))
(t/is (= :not-found (th/ex-type (:error out))))))))
(t/deftest mark-all-threads-as-read-edge-cases
(let [owner (th/create-profile* 1 {:is-active true})
member (th/create-profile* 2 {:is-active true})
viewer (th/create-profile* 3 {:is-active true})
team (th/create-team* 1 {:profile-id (:id owner)})
_ (th/create-team-role* {:team-id (:id team)
:profile-id (:id member)
:role :editor})
_ (th/create-team-role* {:team-id (:id team)
:profile-id (:id viewer)
:role :viewer})
project (th/create-project* 1 {:team-id (:id team)
:profile-id (:id owner)})
file (th/create-file* 1 {:profile-id (:id owner)
:project-id (:id project)})
page-id (get-in file [:data :pages 0])
thread (th/command! {::th/type :create-comment-thread
::rpc/profile-id (:id owner)
:file-id (:id file)
:page-id page-id
:position (gpt/point 0)
:content "hello"
:frame-id uuid/zero})
thread-id (get-in thread [:result :id])]
(t/testing "repeated thread ids are accepted and written once"
(let [out (th/command! {::th/type :mark-all-threads-as-read
::rpc/profile-id (:id member)
:threads [thread-id thread-id]})]
(t/is (th/success? out))
(t/is (= 1 (count (th/db-query :comment-thread-status
{:thread-id thread-id
:profile-id (:id member)}))))))
(t/testing "an empty list of threads is accepted and writes nothing"
(let [out (th/command! {::th/type :mark-all-threads-as-read
::rpc/profile-id (:id viewer)
:threads []})]
(t/is (th/success? out))
(t/is (= 0 (count (th/db-query :comment-thread-status
{:thread-id thread-id
:profile-id (:id viewer)}))))))
(t/testing "marking a thread as read twice keeps a single status row"
(let [out (th/command! {::th/type :mark-all-threads-as-read
::rpc/profile-id (:id member)
:threads [thread-id]})]
(t/is (th/success? out)))
(t/is (= 1 (count (th/db-query :comment-thread-status
{:thread-id thread-id
:profile-id (:id member)}))))
(let [out (th/command! {::th/type :get-unread-comment-threads
::rpc/profile-id (:id member)
:team-id (:id team)})]
(t/is (th/success? out))
(t/is (= 0 (count (:result out))))))))

Some files were not shown because too many files have changed in this diff Show More