mirror of
https://github.com/penpot/penpot.git
synced 2026-11-18 16:12:24 +00:00
Compare commits
No commits in common. "develop" and "2.19.0-RC4" have entirely different histories.
develop
...
2.19.0-RC4
@ -13,17 +13,19 @@ Multi-dimensional code review with quality gates. Every change gets reviewed bef
|
||||
|
||||
## When to Use
|
||||
|
||||
- The reviewer subagent of the `review-code` flow loads this skill to perform the review of a code change.
|
||||
- To review code, always go through the `review-code` flow — never load this skill directly for that. This is the criteria reference, not the flow.
|
||||
- The reviewer subagent of the `review-code` flow loads this skill to perform
|
||||
the review of a code change.
|
||||
- To review code, always go through the `review-code` flow — never load this
|
||||
skill directly for that. This is the criteria reference, not the flow.
|
||||
|
||||
## Core Principles
|
||||
|
||||
These principles underpin every axis. When in doubt, default to them.
|
||||
|
||||
- **DRY (Don't Repeat Yourself):** Every piece of knowledge has one authoritative representation. Flag duplicated logic as a required change — it is not "just similar", it is drift that will diverge.
|
||||
- **KISS (Keep It Simple, Stupid):** The simplest solution that works is the best solution. Complexity must earn its place: if it needs more than a sentence to explain, push for simplification before merge.
|
||||
- **YAGNI (You Aren't Gonna Need It):** No abstractions, hooks or generalizations for hypothetical future use cases. Generalize on the third occurrence, not the first.
|
||||
- **Don't invent problems:** Every finding must be a real risk, a real readability barrier, or a real architectural concern — not a hypothetical or a stylistic preference disguised as a problem.
|
||||
- **DRY (Don't Repeat Yourself):** Every piece of knowledge has one authoritative representation. If the same logic appears in two places, extract it into a shared helper, model, or type. Reviewers: flag duplicated logic as a required change — it's not "just similar," it's drift that will diverge.
|
||||
- **KISS (Keep It Simple, Stupid):** The simplest solution that works is the best solution. Complexity must earn its place. Reviewers: if you need more than one sentence to explain what a piece of code does, it's too complex — push for simplification before merge.
|
||||
- **YAGNI (You Aren't Gonna Need It):** Don't add abstractions, hooks, or generalizations for hypothetical future use cases. Generalize on the third occurrence, not the first. Reviewers: delete speculative generality.
|
||||
- **Don't invent problems:** Do not manufacture issues to produce more feedback. Every finding must be a real risk, a real readability barrier, or a real architectural concern — not a hypothetical or a stylistic preference disguised as a problem.
|
||||
|
||||
## The Five-Axis Review
|
||||
|
||||
@ -36,7 +38,7 @@ Does the code do what it claims to do?
|
||||
- Does it match the spec or task requirements?
|
||||
- Are edge cases handled (null, empty, boundary values)?
|
||||
- Are error paths handled (not just the happy path)?
|
||||
- Do the tests cover the changed behavior, including its edge cases?
|
||||
- Does it pass all tests? Are the tests actually testing the right things?
|
||||
- Are there off-by-one errors, race conditions, or state inconsistencies?
|
||||
|
||||
### 2. Readability & Simplicity
|
||||
@ -45,7 +47,10 @@ Can another engineer (or agent) understand this code without the author explaini
|
||||
|
||||
- Are names descriptive and consistent with project conventions? (No `temp`, `data`, `result` without context)
|
||||
- Is the control flow straightforward (avoid nested ternaries, deep callbacks)?
|
||||
- Are there "clever" tricks that should be simplified? Could this be done in fewer lines? (1000 lines where 100 suffice is a failure)
|
||||
- Are there any "clever" tricks that should be simplified?
|
||||
- **KISS check:** Is this the simplest approach that solves the problem? A 20-line straightforward function beats a 5-line clever one that requires a comment to explain.
|
||||
- Could this be done in fewer lines? (1000 lines where 100 suffice is a failure)
|
||||
- Are abstractions earning their complexity? (Don't generalize until the third use case)
|
||||
- Is a new conditional bolted onto an unrelated flow? Push the logic into its own helper, state, or policy.
|
||||
- Do repeated conditionals on the same shape appear? They signal a missing model or dispatcher.
|
||||
- Are there dead code artifacts: no-op variables, backwards-compat shims, or `// removed` comments?
|
||||
@ -55,13 +60,14 @@ Can another engineer (or agent) understand this code without the author explaini
|
||||
Does the change fit the system's design?
|
||||
|
||||
- Does it follow existing patterns or introduce a new one? If new, is it justified?
|
||||
- Does it keep clean module boundaries, with dependencies flowing the right way (no cycles)?
|
||||
- Does it maintain clean module boundaries?
|
||||
- **DRY check:** Is there existing code that does the same thing? Reuse the canonical helper instead of writing a near-duplicate. If two branches do nearly the same thing, collapse them.
|
||||
- Are dependencies flowing in the right direction (no circular dependencies)?
|
||||
- Is the abstraction level appropriate (not over-engineered, not too coupled)?
|
||||
- Is there existing code that already does this? Reuse the canonical helper instead of a near-duplicate, and collapse branches that do nearly the same thing.
|
||||
- Does this refactor reduce complexity or just relocate it? Count the concepts a reader must hold. Prefer the restructuring that makes whole branches disappear over one that re-centralizes the same logic. Prefer deleting an abstraction to polishing it.
|
||||
- Is feature-specific logic leaking into a shared or general-purpose module?
|
||||
- Are type boundaries explicit? Question gratuitous `any`/`unknown`/optional/casts and silent fallbacks.
|
||||
- Does the change reduce complexity or relocate it? Count the concepts a reader must hold: prefer the restructuring that makes whole branches disappear, and deleting an abstraction over polishing it.
|
||||
- **Give the remedy, not only the problem:** replace conditionals with dispatchers, separate orchestration from business logic, extract helpers, split large files. Prefer the remedy that removes moving pieces over one that spreads the same complexity around.
|
||||
- **Structural remedies:** When you flag a problem, propose the move — not just the problem. Replace conditionals with dispatchers, collapse duplicate branches, separate orchestration from business logic, extract helpers, split large files. Prefer the remedy that removes moving pieces over one that spreads the same complexity around.
|
||||
|
||||
### 4. Security
|
||||
|
||||
@ -105,17 +111,7 @@ For each finding, describe the circumstances under which it could fail: specific
|
||||
|
||||
Lead with what matters: correctness and security first, then structural issues, then everything else. A few high-conviction comments beat a long list.
|
||||
|
||||
5. **Check the evidence, do not re-run it** — Which tests cover the change? What did the author run, and what did CI report? Never re-run the full suite or a build yourself: the author's run and CI already cover that, and repeating it only lengthens the review. Run at most the single test file that settles a specific doubt. When no checks have been run at all (a change you did not see tested), run the module's checks before the verdict.
|
||||
|
||||
## Change Sizing
|
||||
|
||||
Small, focused changes are easier to review, faster to merge, and safer to deploy. Flag the size when a diff exceeds ~300 lines, or when it grows a file past ~1000 total lines, and name the remedy: split it, and separate refactoring from feature work. This is a finding about the change, not work to do inside the review.
|
||||
|
||||
## Dependencies
|
||||
|
||||
Before accepting a new dependency: does the existing stack already solve it, how large is the bundle impact, is it maintained, does `npm audit` report anything, is the license compatible. Prefer the standard library and existing utilities — every dependency is a liability.
|
||||
|
||||
For upgrades: read the changelog, not just the version number; one dependency per change; a green suite before and after; review the lockfile diff and never hand-edit it. Supply-chain triage follows `security-and-hardening`.
|
||||
5. **Verify the verification** — What tests were run? Did the build pass? Was the change tested manually? Screenshots for UI changes?
|
||||
|
||||
## Review Output
|
||||
|
||||
@ -155,18 +151,96 @@ Choose one:
|
||||
|
||||
List the finding IDs the verdict depends on (e.g. "Request changes: R1, R2").
|
||||
|
||||
## Change Sizing
|
||||
|
||||
Small, focused changes are easier to review, faster to merge, and safer to deploy.
|
||||
|
||||
```
|
||||
~100 lines changed → Good. Reviewable in one sitting.
|
||||
~300 lines changed → Acceptable if it's a single logical change.
|
||||
~1000 lines changed → Too large. Split it.
|
||||
```
|
||||
|
||||
**Watch file size, not just diff size.** Around 1000 *total* lines in a single file is a common inspection signal. When a change materially grows an already-large file, decompose first.
|
||||
|
||||
**Splitting strategies:**
|
||||
|
||||
| Strategy | How | When |
|
||||
|----------|-----|------|
|
||||
| **Stack** | Submit a small change, start the next one based on it | Sequential dependencies |
|
||||
| **By file group** | Separate changes for groups needing different reviewers | Cross-cutting concerns |
|
||||
| **Horizontal** | Create shared code/stubs first, then consumers | Layered architecture |
|
||||
| **Vertical** | Break into smaller full-stack slices of the feature | Feature work |
|
||||
|
||||
**Separate refactoring from feature work.** A change that refactors and adds new behavior is two changes — submit them separately.
|
||||
|
||||
## Change Descriptions
|
||||
|
||||
- **First line:** Short, imperative, standalone. "Delete the FizzBuzz RPC" not "Deleting the FizzBuzz RPC."
|
||||
- **Body:** What is changing and why. Include context and reasoning not visible in the code itself.
|
||||
- **Anti-patterns:** "Fix bug," "Fix build," "Add patch," "Phase 1."
|
||||
|
||||
## Dependencies
|
||||
|
||||
Before adding any dependency:
|
||||
|
||||
1. Does the existing stack solve this? (Often it does.)
|
||||
2. How large is the dependency? (Check bundle impact.)
|
||||
3. Is it actively maintained? (Check last commit, open issues.)
|
||||
4. Does it have known vulnerabilities? (`npm audit`)
|
||||
5. What's the license? (Must be compatible with the project.)
|
||||
|
||||
**Rule:** Prefer standard library and existing utilities over new dependencies. Every dependency is a liability.
|
||||
|
||||
**Upgrading dependencies:**
|
||||
|
||||
- Read the changelog, not just the version number. Semver is a promise the maintainer may not have kept.
|
||||
- One dependency per change. When a bulk bump breaks the build, you've lost which package did it.
|
||||
- Let the tests decide — a green suite before *and* after, not just "it installed."
|
||||
- Review the lockfile diff, not just `package.json`. Commit it and never hand-edit it.
|
||||
|
||||
For supply-chain risk triage, follow the `security-and-hardening` skill.
|
||||
|
||||
## Common Rationalizations
|
||||
|
||||
| Rationalization | Reality |
|
||||
|---|---|
|
||||
| "It works, that's good enough" | Working code that's unreadable, insecure, or architecturally wrong creates debt that compounds. |
|
||||
| "I wrote it, so I know it's correct" | Authors are blind to their own assumptions. Every change benefits from another set of eyes. |
|
||||
| "We'll clean it up later" | Later never comes. The review is the quality gate — use it. |
|
||||
| "AI-generated code is probably fine" | AI code needs more scrutiny, not less. It's confident and plausible, even when wrong. |
|
||||
| "The tests pass, so it's good" | Tests are necessary but not sufficient. They don't catch architecture, security, or readability problems. |
|
||||
| "The refactor makes it cleaner" | Relocating complexity isn't reducing it. If the reader still holds the same number of concepts, the structure didn't improve. |
|
||||
| "It's only a small addition to this file" | Small diffs still push files past healthy size and bolt branches onto unrelated flows. |
|
||||
| "It's just a version bump" | A bump is a behavior change you didn't write. Read the changelog. |
|
||||
| "I'll upgrade everything in one PR" | A bulk bump hides which package broke the build. One per change. |
|
||||
| "It's duplicated but it's only two places" | Two becomes three becomes five. Extract now, before the copies diverge. |
|
||||
| "The abstraction is future-proof" | YAGNI. Delete speculative generality — generalize on the third occurrence, not the first. |
|
||||
| "It's clever but efficient" | Cleverness is a readability tax. If it needs a comment to understand, simplify it. |
|
||||
|
||||
## Red Flags
|
||||
|
||||
- A change merged without any review.
|
||||
- A review that only checks whether the tests pass.
|
||||
- Security-sensitive changes without a security-focused review.
|
||||
- A change too large to review properly: split it.
|
||||
- PRs merged without any review
|
||||
- Review that only checks if tests pass (ignoring other axes)
|
||||
- "LGTM" without evidence of actual review
|
||||
- Security-sensitive changes without security-focused review
|
||||
- Large PRs that are "too big to review properly" (split them)
|
||||
- No regression tests with bug fix PRs
|
||||
- Accepting "I'll fix it later" — it never happens
|
||||
- A refactor that moves code around without reducing the number of concepts a reader must hold
|
||||
- New conditionals scattered into unrelated code paths (a missing abstraction)
|
||||
- A bespoke helper that duplicates an existing canonical one
|
||||
- A bulk "bump dependencies" PR with no changelog review
|
||||
|
||||
## Verification
|
||||
|
||||
Before emitting the verdict, confirm the evidence the review rests on. This covers the state of the code at review time, not the later fixing of findings:
|
||||
Before emitting the verdict, verify the change as it stands. This is the reviewer's own due diligence — it covers the state of the code at review time, not the later resolution of findings (fixing findings is the author's job; confirming them is a new review):
|
||||
|
||||
- [ ] The suite is green, and the evidence is the author's run or CI — not your own re-run. Say which one you relied on.
|
||||
- [ ] New behavior has tests that cover the edge cases in the diff.
|
||||
- [ ] The verification story is stated: what changed and how it was verified.
|
||||
- [ ] Dependency changes were reviewed against the changelog and the lockfile diff.
|
||||
- [ ] Tests pass — run them yourself, don't trust the claim
|
||||
- [ ] Build succeeds
|
||||
- [ ] The verification story is documented (what changed, how it was verified)
|
||||
- [ ] Dependency upgrades reviewed against changelog, isolated per package, verified by green suite
|
||||
|
||||
## See Also
|
||||
|
||||
- For detailed security review guidance, see `security-and-hardening`
|
||||
|
||||
@ -63,8 +63,6 @@ subject/body limits, and the `AI-assisted-by` trailer. Follow it exactly.
|
||||
- Do not pass `--author`. Author identity comes from the local git config.
|
||||
- Do not amend a commit you did not create in this session, unless explicitly asked.
|
||||
- Do not bypass pre-commit hooks (`--no-verify`) unless explicitly asked.
|
||||
- Add `Co-authored-by:` only for a human co-author. AI providers, agents and
|
||||
bots are rejected by `./scripts/check-commit`; for AI participation use the
|
||||
`AI-assisted-by:` trailer.
|
||||
- Never add a `Co-authored-by:` trailer. It is forbidden and fails `./scripts/check-commit`.
|
||||
- Do not add untracked files that were not created in this session.
|
||||
- Do not skip the `scripts/check-commit` verification step (Iron Rule 4).
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
---
|
||||
name: local-ci
|
||||
description: Run local CI-style checks with ./scripts/ci (lint, tests, format, and on request production builds) per monorepo module. Use when verifying changes before declaring work done, running lint or tests locally, fixing formatting, repairing Clojure delimiter errors, or checking that a module still compiles.
|
||||
description: Run local CI-style checks with ./scripts/ci (lint, tests, format) per monorepo module. Use when verifying changes before declaring work done, running lint or tests locally, fixing formatting, or repairing Clojure delimiter errors.
|
||||
---
|
||||
|
||||
# Local CI
|
||||
@ -29,12 +29,10 @@ Run from the repo root:
|
||||
```
|
||||
|
||||
Modules: `frontend` `backend` `common` `render-wasm` `exporter` `mcp`
|
||||
`plugins` `library` `media-processor`, or `--all` for every module.
|
||||
`plugins` `library`, or `--all` for every module.
|
||||
|
||||
With no task flags it runs three tasks per module, in order: **lint**,
|
||||
**test**, **fmt** (format check; `--fix` formats files instead). **build**
|
||||
is opt-in: it compiles the module the way production does and is the slow
|
||||
one, so it never runs unless you ask for it.
|
||||
**test**, **fmt** (format check; `--fix` formats files instead).
|
||||
|
||||
| Flag | Effect |
|
||||
|------|--------|
|
||||
@ -43,7 +41,6 @@ one, so it never runs unless you ask for it.
|
||||
| `--lint` / `--no-lint` | Run only lint / drop lint |
|
||||
| `--test` / `--no-test` | Run only tests / drop tests |
|
||||
| `--fmt` / `--no-fmt` | Run only format check / drop it |
|
||||
| `--build` / `--no-build` | Run only the production build / drop it |
|
||||
| `--fix` | Format files instead of checking (other tasks unaffected) |
|
||||
| `--paren-repair` | Fix delimiter errors in Clojure/CLJS files |
|
||||
| `--fail-fast` | Stop at the first failure |
|
||||
@ -75,12 +72,6 @@ log path.
|
||||
# Format the module without running the test suite
|
||||
./scripts/ci --fix --no-test frontend
|
||||
|
||||
# Check that the module still compiles (slow: minutes)
|
||||
./scripts/ci --build frontend
|
||||
|
||||
# Compile every module (8 builds, ~4 min with warm caches)
|
||||
./scripts/ci --build --all
|
||||
|
||||
# Broke delimiters in Clojure/CLJS files: repair first, then lint
|
||||
./scripts/ci --paren-repair frontend
|
||||
./scripts/ci --lint frontend
|
||||
@ -98,12 +89,7 @@ log path.
|
||||
- Test tasks are long-running (backend runs `clojure -M:dev:test`); give
|
||||
the bash call a generous timeout (10–20 minutes) instead of letting it
|
||||
time out mid-run.
|
||||
- `mcp` and `media-processor` have no lint task — they show as skipped,
|
||||
not failed. `common` has no build task for the same reason.
|
||||
- Build runs are the slow ones (frontend ~2 min even with warm caches;
|
||||
cold caches roughly double it). Give the shell a generous timeout and
|
||||
do not run two builds at once in the same checkout: `render-wasm` takes
|
||||
a build lock.
|
||||
- `mcp` has no lint task — it shows as skipped, not failed.
|
||||
- `--paren-repair` only fixes delimiters; run lint afterwards to catch
|
||||
what remains. See `mem:scripts/paren-repair`.
|
||||
- What to run and how to read test results: `mem:testing`.
|
||||
|
||||
@ -1,330 +0,0 @@
|
||||
---
|
||||
name: migrate-ds-area
|
||||
description: Migrate one Penpot UI area to the Design System components for one group of the DS migration epic (#12375): buttons and icons (icon-button*, button*, icon*), selects (select*) and later groups. Moves the area's SCSS off the deprecated styles, proves every change with a zero-tolerance visual test, then opens the draft PR and updates the group issue (PRs table row, QA comment). Use when the user asks to migrate an area or module to the DS, continue a DS migration group, or names an area of a group issue.
|
||||
---
|
||||
|
||||
# Skill: migrate-ds-area
|
||||
|
||||
Move one UI area (one `.cljs` + `.scss` pair, or a few related modules)
|
||||
from legacy components and raw HTML to the Design System components,
|
||||
prove with pixels what changed, and ship it as a draft PR that QA can
|
||||
test.
|
||||
|
||||
The epic is #12375. Each control type is a group with its own issue
|
||||
(buttons and icons #12376, selects #12377, radios #12378, …). The group
|
||||
issue holds the plan (order, waiting areas), the changes QA should
|
||||
expect, and the PRs table where people claim areas. Read it first.
|
||||
|
||||
## Input
|
||||
|
||||
- The group issue (e.g. `#12376`) and the area: a module path (e.g.
|
||||
`dashboard/fonts`) or an area named in the group issue.
|
||||
- No area given: take the next unclaimed area in the group issue's plan
|
||||
order, and say which one.
|
||||
|
||||
## Ground rules
|
||||
|
||||
- Never `git push`. The user pushes; then you open the PR.
|
||||
- PRs go to `develop`; branches start from `origin/develop`.
|
||||
- **Stay in the group.** Only the controls of this group change. In the
|
||||
buttons/icons group, selects, radios, numeric/text inputs, checkboxes,
|
||||
menus (`%menu-*`, `%dropdown-*`) and modal structure (`%modal-*`) stay;
|
||||
a checkbox tick or an input label icon stays legacy.
|
||||
- **Accept the DS look.** Do not add CSS to imitate legacy sizes or
|
||||
colors. Do keep positions: icons and labels must not move unless the
|
||||
DS size forces it, and then say so.
|
||||
- No negative margins to compensate; use positive DS spacing or
|
||||
`px2rem()`.
|
||||
- **Gaps in the DS wait.** An area that needs an icon or a feature the
|
||||
DS lacks keeps its legacy control; say so in the PR.
|
||||
- Skip legacy shared components that another group will replace
|
||||
(`context_menu_a11y`, `forms`, `select`, `dropdown`, `radio_buttons`,
|
||||
`search_bar`, `title_bar`, …): they wait for that group's cleanup,
|
||||
where they are updated if still used or deleted if not.
|
||||
- Debug-only UI gets no QA comment and no compare image.
|
||||
|
||||
## Devenv
|
||||
|
||||
Run every test, lint and `pnpm` command inside the workspace container,
|
||||
never on the host (a host `pnpm` relinks `node_modules` and breaks the
|
||||
container's watcher). The container is `penpot-devenv-ws<N>-main`, where
|
||||
`<N>` is the workspace number (`ws0` is the main repo; `wsN` lives in
|
||||
`~/.penpot/penpot_workspaces/wsN`). Write output to a file, then read it:
|
||||
|
||||
```bash
|
||||
C=penpot-devenv-ws1-main # your workspace
|
||||
docker exec -u penpot -w /home/penpot/penpot/frontend $C \
|
||||
bash -lc '<command>' > <scratch>/out.txt 2>&1
|
||||
```
|
||||
|
||||
When the app does not behave as the code says, check the build first:
|
||||
|
||||
- **Stale build after a branch switch** ("is not a function", or a page
|
||||
with empty `stl/css` classes): `touch` the stale `.cljs`/`.scss`, wait
|
||||
for the compiled file under `frontend/resources/public/` to update,
|
||||
hard-reload.
|
||||
- **A whole area unstyled:** a new `.scss` added after the watcher
|
||||
started is missing from `main.css` (grep `main.css` for the module
|
||||
prefix), or `main.css` was written corrupted after many `.scss`
|
||||
changed at once. Touch a `.scss`; if the module is still missing,
|
||||
restart the watcher in the workspace's tmux session (`penpot:0`,
|
||||
`./scripts/watch app`).
|
||||
- **Blank app after switching between `develop` and `staging`:**
|
||||
dependencies differ. Run `pnpm install --frozen-lockfile` in the
|
||||
container and restart the watcher.
|
||||
|
||||
## Workflow
|
||||
|
||||
### 1. Claim the area and branch
|
||||
|
||||
1. Read the group issue: plan, waiting areas, expected changes, PRs table.
|
||||
2. Add a row to the PRs table with no PR number, the user's @login as
|
||||
Owner and the state `Claimed` (`gh issue view/edit <n> --body-file`).
|
||||
Read the body right before editing, and read it again after saving:
|
||||
when two people edit at the same time, GitHub keeps only the last
|
||||
edit, so a row can vanish. Put yours back if it is gone.
|
||||
Skip an area someone else already claimed, unless the claim has had
|
||||
no PR for 7 days and its owner did not answer a comment asking.
|
||||
3. `git switch -c <user>-ds-<group>-<area> origin/develop`.
|
||||
4. Survey the area: every control of the group in the `.cljs`, the
|
||||
legacy icons, the placeholders and mixins the `.scss` uses, the
|
||||
`div`/`span` with `:on-click`.
|
||||
|
||||
### 2. Visual baseline
|
||||
|
||||
Add a `test.describe("<Area> controls", …)` block to the matching spec
|
||||
in `frontend/playwright/ui/visual-specs/` (project `ds`, dark theme).
|
||||
Put it next to the blocks of the same screen, not always at the end of
|
||||
the file: two PRs that both append at the end conflict.
|
||||
|
||||
- Every shot: `toHaveScreenshot({ maxDiffPixelRatio: 0, threshold: 0 })`.
|
||||
- Show every changed control at rest, on hover and open or pressed.
|
||||
Capture the element or the smallest stable container.
|
||||
- Locators must work before and after the change: text, role, testid,
|
||||
`[class*='…']`, or `a.or(b)` when the element type changes (raw
|
||||
`[class*='menu-btn']` before, `getByRole("button", { name: "Options" })`
|
||||
after). Scope `.or()` to the area so it cannot match elsewhere.
|
||||
- Missing data? Add a fixture under `frontend/playwright/data/` and mock
|
||||
it (`dashboardPage.mockRPC(...)`, `DashboardPage.mockConfigFlags` for
|
||||
flags). Watch for expired or inconsistent fixtures (dates in the past,
|
||||
wrong file for a mock).
|
||||
- Capture with `--update-snapshots=all`:
|
||||
```bash
|
||||
npx playwright test --project=ds <spec> -g "<describe>" --update-snapshots=all
|
||||
```
|
||||
Plain `--update-snapshots` keeps a snapshot within the project's 0.5%
|
||||
tolerance, so a real diff looks like 0.
|
||||
- Copy the snapshots (`<spec>.spec.js-snapshots/<Describe>-*`) to
|
||||
`<scratch>/diffs/<area>/before/` and look at them: the state you
|
||||
wanted must really be on screen.
|
||||
|
||||
### 3. Controls commit
|
||||
|
||||
#### Buttons and icons
|
||||
|
||||
| Legacy | DS |
|
||||
|---|---|
|
||||
| icon-only raw button | `[:> icon-button* {:variant "ghost" :aria-label (tr …) :icon i/x :on-click f}]` (always pass `:variant`; `"action"` for 24px icon toggles) |
|
||||
| pressed/selected | `:aria-pressed (boolean x)` |
|
||||
| text button | `[:> button* {:variant "primary"/"secondary"/"ghost"/"destructive" :type "button"} label]` |
|
||||
| `[:input {:type "button"}]` | `button*` |
|
||||
| modal close | `icon-button*` ghost, `:aria-label (tr "labels.close")`, `:tooltip-class` carrying the corner position |
|
||||
| legacy icon outside a button | `[:> icon* {:icon-id i/x :class …}]` |
|
||||
|
||||
- Not every raw button changes: rows, tabs, swatches, avatar widgets,
|
||||
full-height segments, round custom buttons and link-style text buttons
|
||||
stay raw (their legacy icon still changes). List the kept ones in the
|
||||
PR.
|
||||
- A `div`/`span` with `:on-click` in the area becomes a `[:button]`
|
||||
(with `aria-expanded`/`aria-haspopup` when it opens something). An
|
||||
`[:a]` without `href` stays; list it in the PR.
|
||||
- **Tooltip wrapper.** `icon-button*` renders inside a tooltip trigger
|
||||
wrapper. Position (`position: absolute`, grid placement, margins
|
||||
against siblings) goes on `:tooltip-class`; look (`--button-height`,
|
||||
colors) goes on `:class`.
|
||||
- **Tooltip at the right edge.** Small icon buttons at the right end of
|
||||
a sidebar row or title bar get `:tooltip-placement "top-left"`; list
|
||||
it as an expected change.
|
||||
- **Sizes through custom properties:** `--button-height: #{px2rem(64)}` /
|
||||
`--button-width`. Sass does not evaluate functions inside custom
|
||||
properties without `#{}`.
|
||||
- **A grown `button*` keeps its label at the start**: add
|
||||
`justify-content: center` when it is wider than its content.
|
||||
- **Icon colors:** DS icons use `stroke: currentColor`. Set `color` on
|
||||
the icon class, not `stroke`/`fill`. A legacy `color: transparent` left
|
||||
on the button or wrapper makes the DS icon invisible: drop it. A class
|
||||
that redefines the color variable on the icon breaks the hover that
|
||||
changes it on the button: define the variable on the button.
|
||||
- **Every DS icon needs a color of its own** (on its class or a parent):
|
||||
one that inherits none shows in the devenv debug yellow.
|
||||
- **Icon sizes:** `"s"` draws 12px inside a 16px box (compensate the
|
||||
margin by 2px per side to keep the drawing in place); `"m"` 16px; `"l"`
|
||||
24px; `"xl"` 32px.
|
||||
- **Stroke width:** DS icons keep the DS stroke width. Never set
|
||||
`stroke-width` on a DS icon to match the legacy one; a thinner or
|
||||
thicker line is part of the DS look.
|
||||
- **Legacy placeholders that style `svg` descendants** (e.g.
|
||||
`%dropdown-element-base` → `span svg { 12px box; stroke }`) override a
|
||||
DS icon placed inside them: give the icon class its own box
|
||||
(`inline-size`/`block-size` 16px) and `stroke: currentcolor` with more
|
||||
specificity.
|
||||
- **Flex shrink:** a DS svg inside a small flex button can shrink and be
|
||||
cut; add `flex-shrink: 0` to the icon class.
|
||||
- **Rotated legacy arrows:** prefer `i/arrow-left`/`-right`/`-up`/`-down`
|
||||
over rotating `i/arrow` (same path), unless a rotation class drives the
|
||||
open/closed state.
|
||||
- **Menus anchored by the button size** move when the button grows; open
|
||||
them from the top (`inset-block-start: calc(100% + var(--sp-xs))`) or
|
||||
fix the context menu offset, and say so.
|
||||
|
||||
#### Selects
|
||||
|
||||
`[:& select {:default-value v :options [{:value :label}] :on-change f}]`
|
||||
→ `[:> select* {:default-selected <id> :options <memoized vector of {:id :label}> :on-change f}]`
|
||||
(`app.main.ui.ds.controls.select`).
|
||||
|
||||
- **Ids are non-blank strings.** A blank id shows a dimmed "--", and an
|
||||
id that matches no option falls back to the first one. Give mixed,
|
||||
none and auto their own ids (`"multiple"`, `"none"`, `"auto"`) and map
|
||||
them back in the handler. Use `""` only where "--" is the wanted look.
|
||||
- **Convert the value.** The legacy select ran `d/read-string`, so
|
||||
handlers got keywords, numbers or uuids; `select*` passes the `:id`
|
||||
string. Convert with `keyword`, `d/parse-integer` or `uuid/parse`.
|
||||
- **Memoized options, live selection.** `select*` resets its selection
|
||||
when `:default-selected` or the `:options` identity changes. Build
|
||||
options with `mf/with-memo` and `mapv` (the schema wants a non-empty
|
||||
vector) and pass `:default-selected` from the live value, never a
|
||||
constant.
|
||||
- **Never pass nil.** In dev builds a present prop with value nil
|
||||
(`:class`, `:wrapper-class`, `:disabled`, `:default-selected`) throws
|
||||
`invalid props on component select*`. Use `(boolean x)`, `d/nilv`, or
|
||||
add the prop only when it has a value.
|
||||
- **Props the legacy select ignored stay out**: `select*` passes rest
|
||||
props to its button, so they would reach the DOM.
|
||||
- **Classes:** `:wrapper-class` for layout, `:class` for look.
|
||||
- **Portal:** add `:has-portal true` only where the visual test shows the
|
||||
list cut at the bottom of a panel, and say so in the QA comment.
|
||||
- **Locators:** the button is a `combobox` with no accessible name; use
|
||||
`getByRole("combobox").filter({ hasText: "Linear" })` (it matches the
|
||||
legacy select too) and `getByRole("option", { name: … })`.
|
||||
- **Keyboard:** the arrows no longer change the value while the list is
|
||||
closed. It is an expected change, already listed in #12377.
|
||||
|
||||
#### All groups
|
||||
|
||||
- New labels: add the key to `en.po` and `es.po` by hand, in sorted
|
||||
position, with a `#:` reference to the cljs line. Never run the
|
||||
translations sync.
|
||||
- The `.scss` only gets what the new controls need in this commit.
|
||||
- Some e2e specs are not Prettier-clean on develop: never run
|
||||
`prettier --write` on a whole spec; format only your block.
|
||||
|
||||
Then wait for the watcher (grep the compiled JS/CSS under
|
||||
`frontend/resources/public/` for the change), recapture with `=all`,
|
||||
copy to `after/`, diff:
|
||||
|
||||
```bash
|
||||
SK=.agents/skills/migrate-ds-area/scripts
|
||||
python3 -I $SK/diff.py <before> <after> # per-file pixel count + bbox
|
||||
python3 -I $SK/trip.py <b> <a> <out.png> <scale> # before | after | red diff
|
||||
```
|
||||
|
||||
`trip.py` pads images of different sizes, so a taller "after" still
|
||||
renders.
|
||||
Explain every diff. Fix what is not a DS-look change (moved icons, cut
|
||||
icons, lost hover color). Rerun a shot that changes on unchanged code
|
||||
before trusting it; flaky spots seen so far: the invite modal textarea
|
||||
border, the search input caret, the user account page, the path handler
|
||||
menu trigger.
|
||||
|
||||
Lint, then commit (read `mem:workflow/creating-commits` first):
|
||||
|
||||
```bash
|
||||
npx prettier --check <spec> <fixtures>; npx stylelint <scss>; clj-kondo --lint <cljs>; cljfmt check <cljs>
|
||||
git add -f <cljs> <scss> <po> <spec> <fixtures> # -f: visual-specs/ is gitignored
|
||||
./scripts/check-commit
|
||||
```
|
||||
|
||||
A plain `git add` with an ignored path in the list adds nothing; always
|
||||
`-f`, and check `git show --stat`. Never commit the screenshots.
|
||||
Subject: `:recycle: Use DS <controls> in <area>`. A pre-existing bug
|
||||
found and fixed on the way goes in its own `:bug:` commit.
|
||||
|
||||
### 4. SCSS commit
|
||||
|
||||
Run the `refactor-scss` skill on every `.scss` the PR touched (only
|
||||
those), in its own commit. Keep `refactor/` only for component-family
|
||||
placeholders (`%modal-*`, `%menu-*`, `%input-*`, `%checkbox-*`) and say
|
||||
why in a comment above the `@use`. Unused selectors are deleted, not
|
||||
commented out (list them in the commit body). Typography without an
|
||||
exact match uses the closest DS typography. Run the skill's
|
||||
**Self-review** before committing: no bare element selectors (add
|
||||
classes in the `.cljs`), no copied properties that do nothing, z-index
|
||||
tokens by purpose. Compare against the controls commit: it should be
|
||||
0 px; explain anything else (typically a typography or border change).
|
||||
For files that several modules share or that are huge, stop and propose
|
||||
a separate PR instead.
|
||||
|
||||
### 5. Hand off, PR and tracking
|
||||
|
||||
1. Report to the user: commits, visible changes, kept raw controls,
|
||||
pre-existing bugs, and the compare image path
|
||||
(`<scratch>/diffs/<area>/compare.png`, before | after | diff). Give
|
||||
the push command: `git push -u origin <branch>` (the branch tracks
|
||||
`origin/develop`, so a bare `git push` is refused).
|
||||
2. After the user pushes, check `git rev-parse HEAD origin/<branch>`,
|
||||
then (read `mem:workflow/creating-prs` first):
|
||||
```bash
|
||||
gh pr create --repo penpot/penpot --draft --base develop --head <branch> \
|
||||
--title "<commit subject>" --body-file <body> \
|
||||
--label "WG adoption" --label "no issue required" \
|
||||
--milestone <group issue milestone> --project Main
|
||||
gh pr edit <n> --repo penpot/penpot --remove-label "needs triage"
|
||||
```
|
||||
Never add `no changelog` to a PR: that label goes on issues.
|
||||
Body: AI note, `Part of #<group> (DS migration epic #12375).`, a
|
||||
short What / How, the kept raw controls, an empty `## Screenshots`
|
||||
when something is visible (the user attaches the image). Never
|
||||
`Closes #<group>`. Keep it short: reviewers find long PRs hard to read.
|
||||
3. In the group issue's PRs table, fill your row:
|
||||
`| #<n> | <area>: <controls> (<visible changes>) | @<owner> | Draft |`.
|
||||
4. Post a QA comment on the group issue: `**#<n>: <area>**`, *Where* (how
|
||||
to reach it), *Expected changes* (every visible change; QA treats
|
||||
anything not listed as a bug), *No other change* (what still works
|
||||
the same).
|
||||
|
||||
Do not mark the draft ready or ask for review: the user decides when.
|
||||
|
||||
### 6. Review fixes
|
||||
|
||||
When the reviewer comments on a PR:
|
||||
|
||||
1. Fix each comment in a new commit on the branch (never amend pushed
|
||||
commits). A comment you do not apply gets a short reason the user can
|
||||
post, with an image when it is visual.
|
||||
2. Rerun the area's visual spec before and after the fix with `=all`.
|
||||
Build a compare image for every visible change and give the user its
|
||||
full path, saying which PR it goes to.
|
||||
3. Update everything the fix makes stale, in the same turn: the QA
|
||||
comment (*Expected changes* must list the new visible changes), the
|
||||
PRs table row, and the PR body.
|
||||
4. When working on several PRs in one go, report per PR: branch, new
|
||||
commit, what changed, image path. Never mix two branches in one list.
|
||||
5. Give the push command per branch. If a rebase is coming soon, say
|
||||
that pushing now costs one extra CI run; the user decides.
|
||||
6. After a round of review fixes, check whether the same issue exists in
|
||||
the user's other open PRs of the group, and fix it there too.
|
||||
|
||||
### 7. After merges
|
||||
|
||||
Whenever PRs merge (or go from draft to ready), sync the state column of
|
||||
the group's PRs table with GitHub for **every** row, not only your own:
|
||||
read each row's PR (`gh pr view <n> --json state,isDraft`) and set
|
||||
Merged / In review / Draft. The table drifts when other people merge.
|
||||
|
||||
## Output
|
||||
|
||||
- Branch with 2 commits (controls, SCSS) plus any `:bug:` commit.
|
||||
- Visual spec block with zero-tolerance shots.
|
||||
- Compare image in the scratch folder.
|
||||
- After the push: draft PR, row and QA comment in the group issue.
|
||||
@ -1,25 +0,0 @@
|
||||
"""Pixel diff of two folders of PNG snapshots with the same file names.
|
||||
|
||||
Usage: python3 -I diff.py <before-dir> <after-dir>
|
||||
Prints, per file, the number of differing pixels and their bounding box,
|
||||
or the two sizes when they differ.
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
|
||||
from PIL import Image, ImageChops
|
||||
|
||||
before, after = sys.argv[1], sys.argv[2]
|
||||
for name in sorted(os.listdir(before)):
|
||||
other = os.path.join(after, name)
|
||||
if not os.path.exists(other):
|
||||
print(name, "missing in", after)
|
||||
continue
|
||||
a = Image.open(os.path.join(before, name)).convert("RGB")
|
||||
b = Image.open(other).convert("RGB")
|
||||
if a.size != b.size:
|
||||
print(name, "size", a.size, b.size)
|
||||
continue
|
||||
diff = ImageChops.difference(a, b)
|
||||
count = sum(1 for p in diff.getdata() if p != (0, 0, 0))
|
||||
print(name, count, diff.getbbox())
|
||||
@ -1,33 +0,0 @@
|
||||
"""Before | after | red diff image, one row per snapshot.
|
||||
|
||||
Usage: python3 -I trip.py <before-dir> <after-dir> <out.png> <scale>
|
||||
Both folders hold PNGs with the same names (crop them first if needed).
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
|
||||
from PIL import Image, ImageChops
|
||||
|
||||
before, after, out, scale = sys.argv[1], sys.argv[2], sys.argv[3], int(sys.argv[4])
|
||||
rows = []
|
||||
for name in sorted(os.listdir(before)):
|
||||
a = Image.open(os.path.join(before, name)).convert("RGB")
|
||||
b = Image.open(os.path.join(after, name)).convert("RGB")
|
||||
w, h = max(a.width, b.width), max(a.height, b.height)
|
||||
pad_a = Image.new("RGB", (w, h), (200, 200, 200)); pad_a.paste(a, (0, 0))
|
||||
pad_b = Image.new("RGB", (w, h), (200, 200, 200)); pad_b.paste(b, (0, 0))
|
||||
mask = ImageChops.difference(pad_a, pad_b).convert("L").point(lambda v: 255 if v else 0)
|
||||
red = Image.new("RGB", (w, h), (255, 255, 255))
|
||||
red.paste((255, 0, 0), mask=mask)
|
||||
row = Image.new("RGB", (w * 3 + 20, h), (200, 200, 200))
|
||||
for i, im in enumerate([pad_a, pad_b, red]):
|
||||
row.paste(im, (i * (w + 10), 0))
|
||||
rows.append(row.resize((row.width * scale, row.height * scale), Image.NEAREST))
|
||||
width = max(r.width for r in rows)
|
||||
height = sum(r.height + 10 for r in rows)
|
||||
canvas = Image.new("RGB", (width, height), (255, 255, 255))
|
||||
y = 0
|
||||
for r in rows:
|
||||
canvas.paste(r, (0, y))
|
||||
y += r.height + 10
|
||||
canvas.save(out)
|
||||
@ -1,421 +0,0 @@
|
||||
---
|
||||
name: refactor-scss
|
||||
description: Migrate one Penpot SCSS module off the deprecated styles under frontend/resources/styles/common/refactor/ to the current Design System (ds/ tokens, mixins and px2rem). Pure refactor, no visual change. Use when the user asks to refactor, migrate or clean up a .scss file to the DS.
|
||||
---
|
||||
|
||||
# Skill: refactor-scss
|
||||
|
||||
Refactor a single SCSS file so that it no longer depends on the deprecated
|
||||
styling system, using the current Design System instead.
|
||||
|
||||
This is a **pure refactor**. Do NOT introduce visual or behavioral changes.
|
||||
Preserve the generated CSS behavior exactly unless a rule below requires
|
||||
otherwise. The rules that allow a small visible change (closest DS
|
||||
typography, DS border width) want it shown in the visual check and listed
|
||||
in the report and the commit body.
|
||||
|
||||
Reviewers read every line of the result. A line that is
|
||||
copied but does nothing, a bare element selector or a raw z-index comes
|
||||
back as a review comment, so the rules below are strict about them.
|
||||
|
||||
## Input
|
||||
|
||||
- The path to one `.scss` file (e.g. `frontend/src/app/main/ui/foo/bar.scss`).
|
||||
- If no path is given, ask for it. Do not pick a file yourself.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Read the target `.scss` file and the `.cljs` file with the same name next
|
||||
to it.
|
||||
2. Read every deprecated source the file uses, so you inline real values,
|
||||
never remembered ones:
|
||||
- `frontend/resources/styles/common/refactor/mixins.scss`
|
||||
- `frontend/resources/styles/common/refactor/design-tokens.scss`
|
||||
- `frontend/resources/styles/common/refactor/spacing.scss`
|
||||
- any other file under `frontend/resources/styles/common/refactor/` that
|
||||
the stylesheet pulls in (placeholders, borders, z-index, fonts...).
|
||||
3. Read the DS modules you will map to:
|
||||
- `frontend/src/app/main/ui/ds/_utils.scss` (`px2rem()`)
|
||||
- `frontend/src/app/main/ui/ds/spacing.scss`
|
||||
- `frontend/src/app/main/ui/ds/_sizes.scss`
|
||||
- `frontend/src/app/main/ui/ds/typography.scss` (`use-typography()`)
|
||||
- `frontend/src/app/main/ui/ds/_borders.scss`
|
||||
- `frontend/src/app/main/ui/ds/z-index.scss`
|
||||
4. Apply the rules below and edit the file in place.
|
||||
5. Verify (see **Verification** and **Visual check**).
|
||||
6. Report (see **Output**). Do not commit. When the refactor goes with other
|
||||
changes, it belongs in its own commit, after the task's commits.
|
||||
|
||||
## Primary goal
|
||||
|
||||
After the refactor, the SCSS file must not depend on anything under
|
||||
`frontend/resources/styles/common/refactor/`. This includes imports, mixins,
|
||||
placeholder selectors, functions, deprecated variables and design-token
|
||||
aliases.
|
||||
|
||||
### Partial refactor
|
||||
|
||||
Some placeholders stand for a whole component family that will move to the DS
|
||||
in its own PR: `%input-element`, `%input-with-label`, `%input-checkbox`,
|
||||
`%modal-*`, `%menu-dropdown`, `%menu-item-base`, `%shortcut-*` and similar.
|
||||
Inlining them in a small task produces big, risky diffs. In that case:
|
||||
|
||||
- migrate everything else (tokens, sizes, spacing, small mixins);
|
||||
- keep the `refactor/` import only for those placeholders;
|
||||
- list each remaining dependency and the selectors that use it in the report,
|
||||
so the user can plan it as a separate PR.
|
||||
|
||||
## General rules
|
||||
|
||||
- Keep the resulting SCSS as small and readable as possible.
|
||||
- Avoid duplicated declarations and duplicated selectors.
|
||||
- Do not change class names.
|
||||
- Do not change DOM structure.
|
||||
- Do not change specificity unless required.
|
||||
- Follow the project's SCSS formatting (Prettier, see **Verification**).
|
||||
|
||||
## Imports
|
||||
|
||||
Use the DS modules. Import them with `as *` when they expose mixins,
|
||||
variables or functions:
|
||||
|
||||
```scss
|
||||
@use "ds/_utils.scss" as *;
|
||||
```
|
||||
|
||||
not
|
||||
|
||||
```scss
|
||||
@use "ds/_utils.scss" as utils;
|
||||
```
|
||||
|
||||
Do not introduce namespaces such as `utils.`, `typography.`, `borders.`,
|
||||
`spacing.` or `sizes.` unless a module explicitly requires one.
|
||||
|
||||
Remove every unused import.
|
||||
|
||||
## Deprecated mixins
|
||||
|
||||
If the stylesheet uses mixins from
|
||||
`frontend/resources/styles/common/refactor/mixins.scss`:
|
||||
|
||||
- locate the mixin;
|
||||
- inline its generated CSS (then apply the rest of these rules to it);
|
||||
- remove the `@include`;
|
||||
- remove the import.
|
||||
|
||||
Do not leave deprecated mixins.
|
||||
|
||||
## Placeholders
|
||||
|
||||
Replace every `@extend %placeholder` with the generated CSS. Remove every
|
||||
deprecated placeholder, except the component-family ones described in
|
||||
**Partial refactor**.
|
||||
|
||||
Inline only what does something on that element. Do not paste a
|
||||
placeholder (or mixin) body as it is. For each property, check:
|
||||
|
||||
- **It applies to the element.** `fill`, `stroke` and `stroke-width` only
|
||||
act on SVG shapes: drop them from a `span`, `div` or `button`.
|
||||
`color: transparent` on a wrapper that only holds an icon does nothing
|
||||
useful: drop it and set the real color.
|
||||
- **It is not the default or already given.** `stroke-width: 1px` is the
|
||||
SVG default; `fill: none` and `stroke: currentColor` are already set by
|
||||
the DS `icon*` (`.icon`). Drop them.
|
||||
- **A later rule does not override it.**
|
||||
- **It is not a stroke width on a DS icon.** DS icons keep the DS
|
||||
stroke width; drop any `stroke-width` that only imitates the legacy
|
||||
icon (1.33px, 2.25px...).
|
||||
|
||||
`%button-icon` is the usual case: on a wrapper it leaves `color:
|
||||
transparent; fill: none; stroke-width: 1px`, and none of the three is
|
||||
needed.
|
||||
|
||||
## Element selectors
|
||||
|
||||
Do not style HTML elements directly (`svg {}`, `h3 {}`, `button {}`,
|
||||
`input.x {}`, or `.component { svg {} }`).
|
||||
|
||||
When no class exists, **add one in the `.cljs`**: a DS component takes
|
||||
`:class (stl/css :x)` (`icon*`, `heading*`, `button*`…), and a plain
|
||||
element takes `{:class (stl/css :x)}`. Touching the `.cljs` for this is
|
||||
part of the refactor. A comment like "the icon has no class" is not an
|
||||
answer.
|
||||
|
||||
Prefer the DS component when one fits: `heading*` for `h1`–`h6`
|
||||
(`:level`, `:typography`), `text*` for text, `icon*` for icons.
|
||||
|
||||
`element.class` (e.g. `input.element-title`) is not needed to beat the
|
||||
input defaults: try the class alone and check it in the visual check.
|
||||
|
||||
An element selector stays only for markup you cannot reach (injected
|
||||
HTML, third-party widgets), with a comment saying so.
|
||||
|
||||
**Icon colors.** A DS icon draws with `stroke: currentColor`. Set
|
||||
`color` on the icon class or its parent, never `stroke`/`fill` on an
|
||||
`svg` selector.
|
||||
|
||||
## Color variables
|
||||
|
||||
Whenever a color changes across states, introduce a component-local CSS
|
||||
variable.
|
||||
|
||||
Instead of
|
||||
|
||||
```scss
|
||||
.button {
|
||||
color: var(--color-foreground-primary);
|
||||
|
||||
&:hover {
|
||||
color: var(--color-accent-primary);
|
||||
}
|
||||
|
||||
&:disabled {
|
||||
color: var(--color-foreground-disabled);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
write
|
||||
|
||||
```scss
|
||||
.button {
|
||||
--button-foreground-color: var(--color-foreground-primary);
|
||||
|
||||
color: var(--button-foreground-color);
|
||||
|
||||
&:hover {
|
||||
--button-foreground-color: var(--color-accent-primary);
|
||||
}
|
||||
|
||||
&:disabled {
|
||||
--button-foreground-color: var(--color-foreground-disabled);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Use this pattern for `color`, `background-color`, `border-color`,
|
||||
`outline-color`, `fill`, `stroke` and any similar property.
|
||||
|
||||
## Design token aliases
|
||||
|
||||
Remove every alias defined in
|
||||
`frontend/resources/styles/common/refactor/design-tokens.scss`. Resolve each
|
||||
alias to its final DS token. Do not keep intermediate aliases.
|
||||
|
||||
Instead of `color: var(--button-foreground-hover);` write
|
||||
`color: var(--color-accent-primary);`.
|
||||
|
||||
## Spacing
|
||||
|
||||
Replace deprecated spacing variables using
|
||||
`frontend/src/app/main/ui/ds/spacing.scss` for `margin`, `padding`, `gap` and
|
||||
`inset`.
|
||||
|
||||
If no exact spacing token exists, use `px2rem()`. Do not approximate with the
|
||||
closest token.
|
||||
|
||||
## Sizes
|
||||
|
||||
Replace deprecated size variables using
|
||||
`frontend/src/app/main/ui/ds/_sizes.scss` for `inline-size`, `block-size`,
|
||||
`width`, `height` and min/max sizes.
|
||||
|
||||
If no exact size exists, use `px2rem()` from
|
||||
`frontend/src/app/main/ui/ds/_utils.scss`. Do not leave hardcoded pixel values
|
||||
unless absolutely necessary.
|
||||
|
||||
## Typography
|
||||
|
||||
When any of `font-size`, `font-weight`, `line-height`, `font-family` or
|
||||
`letter-spacing` appear, check whether together they match one of the
|
||||
definitions in `frontend/src/app/main/ui/ds/typography.scss`.
|
||||
|
||||
If they match, replace all the individual declarations with:
|
||||
|
||||
```scss
|
||||
@include use-typography("<typography-name>");
|
||||
```
|
||||
|
||||
If there is no exact match, use the **closest DS typography** (same role:
|
||||
a panel title in uppercase → `headline-small`, body text → `body-*`),
|
||||
even when more than one detail differs. Show the change in the visual
|
||||
check and state it in the report and in the commit body (e.g. "11px
|
||||
regular → 12px medium").
|
||||
|
||||
Keep the original declarations only when no DS typography fits the role
|
||||
(e.g. a 32px display number in a 14px-only area); then leave a TODO
|
||||
naming the closest one and ask the user.
|
||||
|
||||
## Borders
|
||||
|
||||
Replace deprecated border radius and border width values using
|
||||
`frontend/src/app/main/ui/ds/_borders.scss`. A raw width close to a token
|
||||
(`1.75px`) takes the token (`$b-2`); say so in the report. If nothing is
|
||||
close, use `px2rem()`.
|
||||
|
||||
## Logical properties
|
||||
|
||||
Prefer logical properties whenever they preserve the existing behavior:
|
||||
|
||||
- `width` → `inline-size`
|
||||
- `height` → `block-size`
|
||||
- `max-width` → `max-inline-size`
|
||||
- `margin-left` → `margin-inline-start`
|
||||
- `padding-right` → `padding-inline-end`
|
||||
- `top` → `inset-block-start`
|
||||
- `left` → `inset-inline-start`
|
||||
|
||||
## Z-index
|
||||
|
||||
Never leave magic z-index values. Use tokens from
|
||||
`frontend/src/app/main/ui/ds/z-index.scss`, and pick the token **by what
|
||||
the element is, not by its number**: a dropdown, menu or select →
|
||||
`--z-index-dropdown`; a toolbar, palette, sidebar or panel →
|
||||
`--z-index-panels`; and so on (read the comments in that file). A raw `4`
|
||||
on a dropdown becomes `var(--z-index-dropdown)` even though the number
|
||||
changes. Check the visual check for overlap.
|
||||
|
||||
Leave a TODO only when the element fits no token's purpose.
|
||||
|
||||
Legacy z-index values between sibling elements that do not overlap can be
|
||||
removed, but only when the **Visual check** shows no pixel change in every
|
||||
state that matters (open menus, collapsed, hidden). Say so in the report.
|
||||
|
||||
## Duplicated properties
|
||||
|
||||
Within the same selector, avoid duplicated properties, duplicated CSS
|
||||
variables and duplicated declarations introduced by the refactor. Allow
|
||||
duplicates only when they are intentional browser fallbacks.
|
||||
|
||||
## Unused classes
|
||||
|
||||
Open the `.cljs` file with the same name as the SCSS file and check whether
|
||||
every CSS class is referenced (as `(stl/css :class-name)`,
|
||||
`(stl/css-case :class-name ...)` or similar).
|
||||
|
||||
CSS modules prefix every class with its module
|
||||
(`main_ui_<path>__<class>`), so only the `.cljs` of the same module can use
|
||||
it. Before calling a selector unused, also search the whole `frontend/src`
|
||||
for the class name as a string, a `:global` use, or a dynamic class built
|
||||
at runtime.
|
||||
|
||||
If a selector is unused after that check:
|
||||
|
||||
- delete it (do not leave it commented out, do not add a TODO);
|
||||
- list every deleted selector in the report and in the commit body.
|
||||
|
||||
A rule that matches nothing because the markup changed (e.g. `.icon svg`
|
||||
when the DS icon is the `svg` itself) is deleted the same way. Before
|
||||
deleting a descendant rule, check every element it matched, not only
|
||||
the one you changed: `.placeholder .icon svg` can also color a second
|
||||
icon, which then has no color (in the devenv it shows in the debug
|
||||
yellow).
|
||||
|
||||
Every DS icon needs a color of its own (`color` on its class or on a
|
||||
parent with a defined color): an icon that inherits nothing shows in
|
||||
the debug yellow in the devenv.
|
||||
|
||||
Classes used by markup injected from outside (e.g. an external widget)
|
||||
stay, with a comment saying so.
|
||||
|
||||
Also check the other direction: every class the `.cljs` passes to
|
||||
`stl/css` / `stl/css-case` must exist in the `.scss`. A missing one silently
|
||||
does nothing; report it.
|
||||
|
||||
## Sass and CSS pitfalls
|
||||
|
||||
- Sass does not evaluate functions inside CSS custom properties. Write
|
||||
`--button-height: calc(var(--height) - #{px2rem(16)});`, not
|
||||
`... - px2rem(16))`, or the browser gets the literal `px2rem(16)`.
|
||||
- A `/` between Sass size variables is read as a division
|
||||
(`grid-template: $sz-32 / auto` breaks). Split it into
|
||||
`grid-template-rows` / `grid-template-columns` (or areas).
|
||||
- A CSS-wide keyword cannot be part of a shorthand: `inset-inline: unset 0`
|
||||
is invalid and stylelint does not catch it. Use the longhands.
|
||||
- After editing, check the compiled CSS for any of these
|
||||
(`frontend/resources/public/css/main.css`, search the module class).
|
||||
|
||||
## Uncertainty
|
||||
|
||||
If any transformation is uncertain, never guess. Leave a TODO in the code
|
||||
and explain why in the report.
|
||||
|
||||
## Self-review
|
||||
|
||||
Before the report, read the final diff as the reviewer would and search
|
||||
the file for these; each hit must be fixed or have a real reason:
|
||||
|
||||
```bash
|
||||
rg -n 'svg \{|^\s*(h[1-6]|p|hr|input|button|span|a)\b[^-_]|[a-z]\.[a-z-]+ \{' <file>
|
||||
rg -n 'stroke-width|color: transparent|fill: none|z-index: [0-9]|TODO' <file>
|
||||
rg -n '[0-9]\.[0-9]+px|[^(]\b[0-9]+px' <file>
|
||||
```
|
||||
|
||||
- an element selector → add a class (see **Element selectors**);
|
||||
- a property that does not apply or is a default → drop it (see
|
||||
**Placeholders**);
|
||||
- a raw z-index → token by purpose;
|
||||
- a typography TODO → closest DS typography;
|
||||
- a raw px value → token or `px2rem()`.
|
||||
|
||||
## Verification
|
||||
|
||||
Run the project scripts from `frontend/` inside the devenv container
|
||||
(`penpot-devenv-ws<N>-main` for workspace `wsN`; never run `pnpm` on the
|
||||
host), writing output to a file first (never pipe lint output into
|
||||
filters):
|
||||
|
||||
```bash
|
||||
docker exec -u penpot -w /home/penpot/penpot/frontend penpot-devenv-ws<N>-main \
|
||||
bash -lc 'pnpm run lint:scss; echo LINT=$?; pnpm run check-fmt:scss; echo FMT=$?' \
|
||||
> <scratch>/scss-check.txt 2>&1
|
||||
```
|
||||
|
||||
If the format check fails, run `pnpm run fmt:scss` and check again.
|
||||
|
||||
**Dependency check.** A grep for `refactor/` is not enough. For the final
|
||||
file, check:
|
||||
|
||||
- `@use` / `@import` sources: all must be `ds/...` (or the sidebar common
|
||||
module, which is DS-only);
|
||||
- every `@include`, `@extend` and `%placeholder`;
|
||||
- every `$variable`: which imported module defines it (`$br-*` exists in both
|
||||
`ds/_borders.scss` and `refactor/borders.scss`; the `@use` decides);
|
||||
- every `var(--token)`: defined in the DS (colors, spacing, sizes) or only as a
|
||||
`refactor/` alias.
|
||||
|
||||
Report any failure as it is; do not hide it.
|
||||
|
||||
## Visual check
|
||||
|
||||
"No visual change" must be shown, not assumed. When a visual spec covers the
|
||||
component (`frontend/playwright/ui/visual-specs/`, project `ds`), or one can be
|
||||
added:
|
||||
|
||||
1. Capture before (base branch styles) and after with
|
||||
`pnpm exec playwright test --project=ds -g "<test>" --update-snapshots=all`
|
||||
in the container. `=all` is required: plain `--update-snapshots` only
|
||||
rewrites past the 0.5% tolerance, so a real diff can look like 0.
|
||||
2. Before each capture, wait until the compiled CSS/JS shows the expected
|
||||
version (search `main.css` for the changed rule).
|
||||
3. Compare with zero tolerance. If pixels change, build the before | after |
|
||||
diff (red) image and explain every change in the report.
|
||||
|
||||
## Output
|
||||
|
||||
1. The diff of the file (`git diff -- <file>`).
|
||||
2. A summary with:
|
||||
- removed imports;
|
||||
- removed deprecated mixins;
|
||||
- removed placeholders;
|
||||
- typography replacements;
|
||||
- design-token aliases removed;
|
||||
- logical properties introduced;
|
||||
- deleted unused selectors;
|
||||
- TODOs requiring manual review, each with its reason;
|
||||
- remaining `refactor/` dependencies (partial refactor) and where they are
|
||||
used;
|
||||
- classes missing from the `.scss`, if any.
|
||||
3. The verification results, including the visual check (pixel count, or why
|
||||
it was not possible).
|
||||
@ -22,20 +22,8 @@ Act as a senior software engineer and perform a thorough code review.
|
||||
diff, a commit range, a branch, a PR (number or URL), or specific files. If
|
||||
the target is ambiguous, ask before reviewing.
|
||||
2. Delegate the review to the `general` subagent (via the task tool), unless the
|
||||
user specifies another agent. The prompt carries the **`code-review-criteria`**
|
||||
skill name, all user context, and these requirements:
|
||||
- Load `code-review-criteria` and follow its process and output format.
|
||||
- Read `AGENTS.md` (if present) and the module memories it points to for the
|
||||
code being reviewed.
|
||||
- Return the COMPLETE review, verbatim, exactly as the skill instructs it to
|
||||
be produced. Never summarize it.
|
||||
- Read-only: modify no file and create no commit. Reviewing never writes.
|
||||
- Do not invent problems: every finding must be real and actionable.
|
||||
- Missing tests are an issue, not a suggestion — a severity-tagged finding.
|
||||
- Prioritize by impact: one structural issue outweighs ten nits, and "this
|
||||
could be better" is not a finding — say why and how.
|
||||
- Skip generated files, lockfile-only changes and unrelated modifications
|
||||
unless they introduce security risks.
|
||||
user specifies another agent. Include in the prompt the
|
||||
**`code-review-criteria`** skill name and all user context.
|
||||
3. When the subagent returns, output the review to the user verbatim. Do not
|
||||
summarize it and do not act on its findings.
|
||||
4. Right after the review, suggest how to proceed based on the findings. These
|
||||
@ -56,18 +44,43 @@ the findings. Once the user explicitly states the review is done (or moves on to
|
||||
a different task), this rule no longer applies and you act as a normal build
|
||||
agent again.
|
||||
|
||||
## Instructions for the subagent
|
||||
|
||||
1. Load the **`code-review-criteria`** skill and follow its process and output
|
||||
format.
|
||||
2. Read `AGENTS.md` (if present) and follow its instructions for finding and
|
||||
reading all related testing documentation from memories before reviewing.
|
||||
3. Return in your final message the COMPLETE review, verbatim, exactly as the
|
||||
skill instructs it to be produced. Do not summarize it — include the full
|
||||
structured review.
|
||||
|
||||
### Strong rules for the subagent
|
||||
|
||||
1. Do not invent problems. Every finding must be real and actionable.
|
||||
2. Read-only: do not modify any file and do not create a commit — reviewing
|
||||
never writes.
|
||||
3. Be specific and constructive. "This could be better" is not helpful — explain
|
||||
why and how.
|
||||
4. Prioritize by impact. One structural issue outweighs ten nits.
|
||||
5. Missing tests are an issue, not a suggestion. Report as a severity-tagged
|
||||
finding — never as a recommendation.
|
||||
6. Skip generated files, lockfile-only changes, and unrelated modifications
|
||||
unless they introduce security risks.
|
||||
|
||||
## Posting as a GitHub comment
|
||||
|
||||
Post only when the user asks for it explicitly; otherwise keep the review in the
|
||||
session. PR comments go in English, with this minimal structure:
|
||||
Only post the review as a comment when the user asks for it explicitly. Keep
|
||||
it in the session otherwise.
|
||||
|
||||
When posting: PR comments go in English, with this minimal structure:
|
||||
|
||||
1. **Intro line** — one sentence: what was found and the overall tone. No
|
||||
greetings.
|
||||
2. **`## Main issues`** — one subsection per required finding: what breaks, the
|
||||
failure circumstances, and the minimal fix (code snippet when useful). A
|
||||
missing test belongs here, not under minor suggestions.
|
||||
3. **`## Minor suggestions`** — one bullet per minor finding: file/function,
|
||||
problem, concrete remedy.
|
||||
2. **`## Main issues`** — one subsection per required finding: what breaks,
|
||||
the failure circumstances, and the minimal fix (code snippet when useful).
|
||||
A missing test belongs here, not under minor suggestions.
|
||||
3. **`## Minor suggestions`** — one bullet per minor finding or suggestion:
|
||||
file/function, problem, concrete remedy.
|
||||
4. **Closing** — optional and short. Cut it when it adds nothing.
|
||||
|
||||
Reference every finding by its ID (`R1`, `M1`, `S1`, … from
|
||||
|
||||
@ -6,40 +6,64 @@ slash: true
|
||||
|
||||
# Review Plan
|
||||
|
||||
Act as a senior software engineer and perform a thorough review of an implementation plan.
|
||||
Act as a senior software engineer and perform a thorough review of an
|
||||
implementation plan.
|
||||
|
||||
## When to use
|
||||
|
||||
- The user asks to review a plan, in any phrasing: "review this plan", "does this plan look right?", "second opinion on the plan" — or runs `/review-plan`.
|
||||
- A plan was just produced (typically by `/make-a-plan`) and the user wants it evaluated before executing it.
|
||||
- The user asks to review a plan, in any phrasing: "review this plan",
|
||||
"does this plan look right?", "second opinion on the plan" — or runs
|
||||
`/review-plan`.
|
||||
- A plan was just produced (typically by `/make-a-plan`) and the user
|
||||
wants it evaluated before executing it.
|
||||
|
||||
## Instructions
|
||||
|
||||
1. **Determine the plan under review** from the session context (for example, a plan just produced by `/make-a-plan`) or from a plan file path given by the user (typically under `.agents/plans/`). If a file path is given, read the file first so the complete plan is in context.
|
||||
2. Delegate the review to the `general` subagent (via the task tool), unless the user specifies another agent. Include in the prompt the **`plan-review-criteria`** skill name and all user context.
|
||||
3. When the subagent returns, output the review to the user verbatim. Do not summarize it and do not act on its findings.
|
||||
4. Right after the review, suggest the next step based on the verdict. These are suggestions — the user decides, and any instruction overrides them:
|
||||
1. **Determine the plan under review** from the session context (for example, a
|
||||
plan just produced by `/make-a-plan`) or from a plan file path given by the
|
||||
user (typically under `.agents/plans/`). If a file path is given, read the
|
||||
file first so the complete plan is in context.
|
||||
2. Delegate the review to the `general` subagent (via the task tool), unless the
|
||||
user specifies another agent. Include in the prompt the
|
||||
**`plan-review-criteria`** skill name and all user context.
|
||||
3. When the subagent returns, output the review to the user verbatim. Do not
|
||||
summarize it and do not act on its findings.
|
||||
4. Right after the review, suggest the next step based on the verdict. These
|
||||
are suggestions — the user decides, and any instruction overrides them:
|
||||
- **Approve** → suggest `/implement-plan` to execute it.
|
||||
- **Request changes** → suggest `/make-a-plan` to make a plan to address the findings.
|
||||
- **Request changes** → suggest `/make-a-plan` to make a plan to address the
|
||||
findings.
|
||||
|
||||
### Hard rule — read-only while reviewing
|
||||
|
||||
This flow is read-only **for the duration of the review**: from the moment it starts until the user considers the review finished (including any feedback, questions, or clarifications about it). During that period, never fix, implement, edit files or create commits — not even "obvious" fixes derived from the findings. Once the user explicitly states the review is done (or moves on to a different task), this rule no longer applies and you act as a normal build agent again.
|
||||
This flow is read-only **for the duration of the review**: from the moment it
|
||||
starts until the user considers the review finished (including any feedback,
|
||||
questions, or clarifications about it). During that period, never fix,
|
||||
implement, edit files or create commits — not even "obvious" fixes derived from
|
||||
the findings. Once the user explicitly states the review is done (or moves on to
|
||||
a different task), this rule no longer applies and you act as a normal build
|
||||
agent again.
|
||||
|
||||
## Instructions for the subagent
|
||||
|
||||
1. Load the **`plan-review-criteria`** skill and follow its process and output format.
|
||||
2. Read `AGENTS.md` (if present) and follow its instructions for finding and reading all related documentation and testing memories before reviewing.
|
||||
3. Cada hallazgo debe tener un identificador único y estable, con el formato `F1`, `F2`, etc. El identificador debe aparecer en el título del hallazgo y no debe reutilizarse dentro de la misma revisión. Esto permite que el usuario pueda responder sobre un hallazgo concreto.
|
||||
4. Return in your final message the COMPLETE review, verbatim, exactly as the skill instructs it to be produced. Do not summarize it — include the full structured review.
|
||||
1. Load the **`plan-review-criteria`** skill and follow its process and output
|
||||
format.
|
||||
2. Read `AGENTS.md` (if present) and follow its instructions for finding and
|
||||
reading all related documentation and testing memories before reviewing.
|
||||
3. Return in your final message the COMPLETE review, verbatim, exactly as the
|
||||
skill instructs it to be produced. Do not summarize it — include the full
|
||||
structured review.
|
||||
|
||||
### Strong rules for the subagent
|
||||
|
||||
1. Do not invent problems. Every finding must be real and actionable.
|
||||
2. Read-only: do not modify any file and do not create a commit — reviewing never writes.
|
||||
3. Be specific and constructive. "This could be better" is not helpful — explain why and how.
|
||||
2. Read-only: do not modify any file and do not create a commit — reviewing
|
||||
never writes.
|
||||
3. Be specific and constructive. "This could be better" is not helpful — explain
|
||||
why and how.
|
||||
4. Prioritize by impact. One structural issue outweighs ten nits.
|
||||
5. Judge the plan as the implementer would: every task executable without guessing, ordering follows the dependency graph, risks named.
|
||||
5. Judge the plan as the implementer would: every task executable without
|
||||
guessing, ordering follows the dependency graph, risks named.
|
||||
|
||||
## User input, overrides and additional context
|
||||
|
||||
|
||||
@ -5,18 +5,16 @@ description: Update the project CHANGES.md with issues from a given GitHub miles
|
||||
|
||||
# Skill: update-changelog
|
||||
|
||||
Entry point for all changelog work. All rules — sections, exclusions, entry
|
||||
format, the `:boom:`, `:rocket:` and `:lock:` contracts, pre-flight, and the
|
||||
anomaly report — live in `mem:workflow/updating-changelog` (file:
|
||||
`.serena/memories/workflow/updating-changelog.md`). This skill routes to that
|
||||
memory; it does not repeat it.
|
||||
Update `CHANGES.md` with entries for all issues and PRs in a given GitHub
|
||||
milestone. Each entry references the user-facing issue (not the PR) as the
|
||||
primary link, with the fix PR inline on the same line.
|
||||
|
||||
## When to Use
|
||||
|
||||
- Before a new release, to populate the changelog with all fixed issues
|
||||
- When new issues are added to an existing milestone and the changelog needs
|
||||
to be refreshed
|
||||
- To audit or repair an existing version section
|
||||
- To ensure every entry follows the correct format for the changelog
|
||||
|
||||
## Prerequisites
|
||||
|
||||
@ -25,37 +23,289 @@ memory; it does not repeat it.
|
||||
- `scripts/gh.py` for GitHub queries, `scripts/changelog.py` for changelog
|
||||
checks (run `python3 scripts/changelog.py <command> --help` for usage)
|
||||
|
||||
## What to Read
|
||||
## Workflow
|
||||
|
||||
1. `mem:workflow/updating-changelog` — the canonical flow: sources of truth,
|
||||
layout, entry rules, the `## Breaking change note` comment contract, the
|
||||
advisory (`:lock:`) contract, pre-flight checks, anomaly types, and pitfalls.
|
||||
2. `mem:scripts/changelog` — `boom`, `security`, `check-merged`, `cross-ref`,
|
||||
`dev-only-bugs`, `report`.
|
||||
3. `mem:scripts/gh` — `issues`, `issue`, `comments`, `prs`, `advisories`.
|
||||
### 1. Determine the target version
|
||||
|
||||
## Commands
|
||||
The version is typically a semver string like `2.15.3`. Confirm with the user
|
||||
if not specified.
|
||||
|
||||
### 2. Fetch all issues in the milestone
|
||||
|
||||
```bash
|
||||
python3 scripts/gh.py issues "<VERSION>" --exclude "no changelog"
|
||||
python3 scripts/gh.py issues "<VERSION>" --label "release blocker" --exclude "no changelog" # closed non-bug residue to review by hand
|
||||
python3 scripts/gh.py advisories --patched-version "<VERSION>"
|
||||
python3 scripts/changelog.py dev-only-bugs "<VERSION>" # list bugs that never reached production
|
||||
python3 scripts/changelog.py dev-only-bugs "<VERSION>" --apply # label the sure ones `no changelog`
|
||||
python3 scripts/changelog.py check-merged <PR numbers...>
|
||||
python3 scripts/changelog.py boom "<VERSION>" # print the :boom: section
|
||||
python3 scripts/changelog.py boom "<VERSION>" --write # replace it in CHANGES.md
|
||||
python3 scripts/changelog.py security "<VERSION>" # print the :lock: section
|
||||
python3 scripts/changelog.py security "<VERSION>" --write # replace it in CHANGES.md
|
||||
python3 scripts/changelog.py cross-ref "<VERSION>"
|
||||
python3 scripts/changelog.py report "<VERSION>"
|
||||
# All closed issues (default)
|
||||
python3 scripts/gh.py issues "2.16.0"
|
||||
|
||||
# Include open issues too
|
||||
python3 scripts/gh.py issues "2.16.0" --state all
|
||||
|
||||
# Exclude entries that should not go in the changelog
|
||||
python3 scripts/gh.py issues "2.16.0" --exclude "release blocker,no changelog"
|
||||
```
|
||||
|
||||
## Hard Rules
|
||||
**Exclusion rules (issue-level):**
|
||||
- `no changelog` label — chore/refactor work, no entry needed
|
||||
- `release blocker` label — blocked issues not yet ready for changelog
|
||||
- `Task` issue type — internal chores, not user-facing; excluded by `gh.py`
|
||||
(use `--include-tasks` to override)
|
||||
- **Rejected project status** — issues with "Rejected" status on the "Main"
|
||||
project board are excluded by `gh.py` (use `--include-rejected` to override).
|
||||
This status is independent of the GitHub issue `state`.
|
||||
|
||||
- `CHANGES.md` is edited only on explicit request or during the release process.
|
||||
- Only closed, non-excluded milestone issues are included; the issue is the changelog unit and the PR is inline.
|
||||
- `:boom:` and `:rocket:` are label-generated and `:lock:` is advisory-generated: never hand-edit them; edit the label, the `## Breaking change note` comment or the advisory and re-run the command.
|
||||
- Verify PR merge status before writing; only merged PRs are referenced.
|
||||
- Bugs that never reached production stay out: `dev-only-bugs --apply` labels the sure ones; candidates are the user's call, never label them yourself.
|
||||
- Fill `patched_versions` in the advisories before running `security`; unpublished advisories are valid.
|
||||
**Exclusion rules (PR-level):** PRs with these labels stay out regardless of
|
||||
their linked issue's labels: `release blocker`, `no issue required`.
|
||||
|
||||
Each entry carries `number`, `title`, `state`, `issue_type`, `labels`,
|
||||
`closing_prs`, and `project_status`.
|
||||
|
||||
### 3. Identify missing entries (optional)
|
||||
|
||||
```bash
|
||||
python3 scripts/gh.py issues "2.16.0" --exclude "release blocker,no changelog" --compare CHANGES.md
|
||||
```
|
||||
|
||||
Returns only milestone issues not yet referenced in the changelog. Note: it
|
||||
compares **issues** only. For unreferenced merged **PRs**, use the
|
||||
cross-reference in step 8.
|
||||
|
||||
### 4. Fetch additional PR details when needed
|
||||
|
||||
```bash
|
||||
# One or more PR numbers (also: --file prs.txt, or --stdin)
|
||||
python3 scripts/gh.py prs 9179 9204 9311
|
||||
|
||||
# All merged PRs in a milestone (default); --state all/open/closed for others
|
||||
python3 scripts/gh.py prs --milestone "2.16.0"
|
||||
```
|
||||
|
||||
Returns `number`, `title`, `body`, `state`, `merged_at`, `author`, `labels`,
|
||||
and `closing_issues`. Milestone mode uses paginated GraphQL (100 per page).
|
||||
|
||||
### 5. Categorize entries — strictly by issue type, never by labels or emoji
|
||||
|
||||
Use the **Issue Type** field (`issue_type` in the `gh.py` output). No separate
|
||||
query is needed.
|
||||
|
||||
> **⚠️ CRITICAL: Never use labels or title emoji prefixes for categorization.**
|
||||
> Labels like `bug`/`enhancement` and prefixes like `:bug:`/`:sparkles:` are
|
||||
> often wrong or missing. `issue_type` is the single source of truth.
|
||||
|
||||
| `issue_type` value | Changelog section |
|
||||
|--------------------|-------------------|
|
||||
| `Bug` | `### :bug: Bugs fixed` |
|
||||
| `Feature` or `Enhancement` | `### :sparkles: New features & Enhancements` |
|
||||
| `Task` | **Exclude** — internal chores, not user-facing |
|
||||
| `null` (not set) | Fallback to labels: `bug` label → bugs, otherwise enhancements |
|
||||
|
||||
**Breaking changes override everything:** an issue with the `breaking change`
|
||||
label goes under `### :boom: Breaking changes & Deprecations`, no matter its
|
||||
issue type. This and `release highlight` below are the only label-based
|
||||
rules — explicit exceptions to the "never by labels" principle above. The
|
||||
`:boom:` subsection goes first in the version, before `:rocket:` (matching
|
||||
the existing precedent).
|
||||
|
||||
**Highlights come from the `release highlight` label:** an issue with that
|
||||
label also gets an entry under `### :rocket: Epics and highlights`, on top of
|
||||
its regular entry (usually `:sparkles:`). In the section being refreshed,
|
||||
`:rocket:` lists exactly the labelled issues: never pick highlights yourself
|
||||
and never keep an unlabelled entry there. Product decides the highlights by
|
||||
labelling issues on GitHub, so a regeneration cannot undo that choice. Older
|
||||
version sections are left as they are.
|
||||
|
||||
**Community attribution:** if the issue or its fix PR has the
|
||||
`community contribution` label, add `(by @<github_username>)` on the entry
|
||||
line, **before** the issue/PR references. Use the **PR author** (the `author`
|
||||
field from step 4), not the issue author:
|
||||
|
||||
```markdown
|
||||
- Fix description of the bug (by @username) [#<ISSUE>](...) (PR: [#<PR>](...))
|
||||
```
|
||||
|
||||
**Only closed issues are included**, even if open ones are tracked in the
|
||||
milestone.
|
||||
|
||||
**Pairing rules:**
|
||||
|
||||
| Pattern | Changelog format |
|
||||
|---------|-----------------|
|
||||
| Closed issue + one or more fix PRs | Primary link = issue, PRs inline comma-separated |
|
||||
| PR with no linked issue | Link the issue if a matching closed one exists in the milestone; otherwise skip (the issue is the changelog unit) |
|
||||
| Closed issue with no fix PR in milestone | Link the issue directly, no PR reference |
|
||||
|
||||
> **False-positive associations:** a PR may wrongly claim to close an issue
|
||||
> from another context (ancient PR, cross-project reference). If titles are
|
||||
> clearly unrelated or the PR predates the issue by years, treat it as a data
|
||||
> glitch and skip it.
|
||||
|
||||
### 5a. Verify PR merge status before writing
|
||||
|
||||
A closed issue may list closing PRs that were **closed without merging**
|
||||
(e.g. a superseded community PR). Only **merged** PRs go in the changelog:
|
||||
|
||||
```bash
|
||||
python3 scripts/changelog.py check-merged <ALL_PR_NUMBERS>
|
||||
# also accepts: --file prs.txt, or numbers via --stdin
|
||||
```
|
||||
|
||||
If a closing PR is closed-unmerged, find the merged PR that superseded it
|
||||
(other PRs in the issue's closing list, similar titles, or pointers in the
|
||||
closed PR's timeline) and reference that one instead.
|
||||
|
||||
### 5b. Security advisory (GHSA) entries
|
||||
|
||||
Advisories fixed in a release go in the changelog even though they are
|
||||
**neither milestone issues nor PRs**. The GHSA ID and description come from
|
||||
the user or the release notes — never from the milestone fetch.
|
||||
|
||||
```markdown
|
||||
- Fix <user-facing description> (https://github.com/penpot/penpot/security/advisories/GHSA-XXXX-XXXX-XXXX)
|
||||
```
|
||||
|
||||
Rules: place under `### :bug: Bugs fixed` with **no issue or PR link**; do
|
||||
**not** fetch or verify the URL (it may be draft/unpublished and 404); write
|
||||
the description in imperative mood from the advisory title. These entries are
|
||||
invisible to the automation — add them by hand, and in step 9 apply only the
|
||||
backport/duplicate check to them.
|
||||
|
||||
### 6. Read the current CHANGES.md and run pre-flight checks
|
||||
|
||||
Newest version goes at the top, right after the `# CHANGELOG` header:
|
||||
|
||||
```markdown
|
||||
## <VERSION>
|
||||
|
||||
### :boom: Breaking changes & Deprecations
|
||||
|
||||
- <breaking change or deprecation> [#<ISSUE>](https://github.com/penpot/penpot/issues/<ISSUE>) (PR: [#<PR>](https://github.com/penpot/penpot/pull/<PR>))
|
||||
|
||||
### :bug: Bugs fixed
|
||||
|
||||
- Fix description of the bug [#<ISSUE>](https://github.com/penpot/penpot/issues/<ISSUE>) (PR: [#<PR>](https://github.com/penpot/penpot/pull/<PR>))
|
||||
- Fix another bug (by @contributor) [#<ISSUE>](https://github.com/penpot/penpot/issues/<ISSUE>) (PR: [#<PR>](https://github.com/penpot/penpot/pull/<PR>))
|
||||
|
||||
### :sparkles: New features & Enhancements
|
||||
|
||||
- Add new feature description [#<ISSUE>](https://github.com/penpot/penpot/issues/<ISSUE>) (PR: [#<PR>](https://github.com/penpot/penpot/pull/<PR>))
|
||||
```
|
||||
|
||||
Format details: entries start with `- ` plus a short imperative description;
|
||||
PR refs stay inline (`(PR: [#<N>](<url>))`, comma-separated for several);
|
||||
`(by @<username>)` goes before the issue link; only include non-empty
|
||||
sections; blank line between a section's last entry and the next title; never
|
||||
duplicate an entry from an earlier version section (the earlier version wins
|
||||
for backports).
|
||||
|
||||
**Pre-flight checks — fix violations directly in `CHANGES.md` before writing
|
||||
the new section.** Reconcile every existing entry (any version section) and
|
||||
every milestone candidate against the current milestone state (re-fetch, do
|
||||
not trust cached data):
|
||||
|
||||
1. **Duplicate across versions** → remove from the current section.
|
||||
2. **Stale milestone assignment** (issue moved out of this milestone) →
|
||||
remove the entry (or drop it if the section does not exist yet).
|
||||
3. **Newly applied exclusion labels** (`no changelog`, `release blocker`) →
|
||||
remove the entry.
|
||||
4. **Issue no longer closed/deleted/Rejected** → remove the entry.
|
||||
5. **Unmerged or moved PR reference** → fix the reference or remove the
|
||||
entry (a PR merged in a *different* milestone is a step-9 anomaly, do not
|
||||
silently remove it).
|
||||
6. **Issue type changed to `Task`** → remove the entry.
|
||||
7. **Breaking change misplaced** (issue has the `breaking change` label but
|
||||
sits in another section) → move the entry to `:boom:`.
|
||||
8. **Highlight out of sync** (`:rocket:` entry without the `release
|
||||
highlight` label, or a labelled issue missing from `:rocket:`) → remove
|
||||
the `:rocket:` entry or add it (step 7b). The regular entry stays.
|
||||
9. **Missing valid issues** (closed, non-excluded, unreferenced anywhere) →
|
||||
add them to the current section per step 5.
|
||||
|
||||
### 7. Build the description text
|
||||
|
||||
Derive it from the **issue title**, not the PR title. Strip leading emoji
|
||||
prefixes (`:bug:`, `:sparkles:`, `:tada:`) and describe the user-facing
|
||||
behavior:
|
||||
|
||||
| Issue title | Changelog description |
|
||||
|-------------|----------------------|
|
||||
| `Plugin API token methods fail with schema validation error on PRO` | `Fix Plugin API token methods failing with schema validation error on PRO` |
|
||||
| `Comment content is not sanitized before rendering, enabling stored XSS` | `Sanitize comment content on rendering` |
|
||||
| `Custom uploaded font family names are not sanitized` | `Sanitize font family names on custom uploaded fonts` |
|
||||
|
||||
Insert the new version section right after the `# CHANGELOG` header with the
|
||||
`edit` tool and enough context for a unique match.
|
||||
|
||||
### 7b. Populate `:rocket: Epics and highlights` from the label
|
||||
|
||||
List the milestone issues that pass step 2 exclusions and carry the
|
||||
`release highlight` label:
|
||||
|
||||
```bash
|
||||
python3 scripts/gh.py issues "2.16.0" --exclude "release blocker,no changelog" \
|
||||
| python3 -c "import sys,json; [print(i['number'], i['title']) for i in json.load(sys.stdin) if 'release highlight' in i['labels']]"
|
||||
```
|
||||
|
||||
Create the subsection if missing (place it before `### :sparkles:`) with one
|
||||
entry per labelled issue, same text and references as its regular entry.
|
||||
Every `:rocket:` entry MUST carry issue AND PR references. If no issue is
|
||||
labelled, leave `:rocket:` out and tell the user: the highlights are
|
||||
product's call, not the agent's. To change them, add or remove the label on
|
||||
GitHub and re-run; never edit `:rocket:` by hand.
|
||||
|
||||
### 8. Cross-reference milestone PRs against the changelog
|
||||
|
||||
```bash
|
||||
python3 scripts/changelog.py cross-ref "<MILESTONE>" [--changes CHANGES.md]
|
||||
```
|
||||
|
||||
Lists merged milestone PRs missing from the changelog section (decide per PR:
|
||||
add it or confirm its exclusion labels) and warns about CLOSED (unmerged) PRs
|
||||
in the milestone. GHSA entries never appear here — that absence is expected.
|
||||
|
||||
### 9. Generate the anomaly report
|
||||
|
||||
```bash
|
||||
python3 scripts/changelog.py report "<MILESTONE>" [--changes CHANGES.md --output CHANGES-ISSUES.md]
|
||||
```
|
||||
|
||||
Overwrites `CHANGES-ISSUES.md` with the current state. Every number renders as
|
||||
a full `[#N](https://github.com/penpot/penpot/issues/N)` or
|
||||
`[#N](https://github.com/penpot/penpot/pull/N)` link.
|
||||
|
||||
**An anomaly is a milestone mismatch or a `:boom:` mislabel** (the changelog
|
||||
pairing is misleading and a human must judge intent):
|
||||
|
||||
1. Issue in this milestone, referenced PR in another milestone (or none).
|
||||
2. PR in this milestone, closed issue in another milestone — except an issue
|
||||
with *no* milestone, which belongs to another (probably private) project
|
||||
and is neither anomaly nor changelog candidate.
|
||||
3. `:boom:` entry whose issue lacks the `breaking change` label. These stay
|
||||
listed in the report **and** in the changelog: either label the issue or
|
||||
move the entry to its regular section.
|
||||
4. `:rocket:` out of sync with the `release highlight` label. Step 6
|
||||
pre-flight should have fixed it: if it shows up, re-run the workflow.
|
||||
|
||||
**Highlight gaps are warnings, not anomalies:** missing `:rocket:` on a
|
||||
released X.Y.0 (patches never carry highlights); `:rocket:` entry without
|
||||
issue AND PR references. Gaps never count toward the anomaly total.
|
||||
|
||||
**Anything else is a rule violation**, not a report item: fix it in step 6
|
||||
pre-flight. If one shows up in the report, re-run the workflow.
|
||||
|
||||
## Key Principles
|
||||
|
||||
- **Issue = changelog unit**, PR = implementation detail inline.
|
||||
- **Latest version first**, below the `# CHANGELOG` header.
|
||||
- **Issue Type decides the section — exclusively**, except `breaking change`
|
||||
label → `:boom:` first, and `release highlight` label → also `:rocket:`.
|
||||
- **User-facing descriptions** in imperative mood.
|
||||
- **Community attribution** uses the PR author, placed before the issue link.
|
||||
- **Only closed issues**; **Rejected** project status excludes.
|
||||
- **`no changelog` and `Task`** stay out.
|
||||
- **Multiple fix PRs** go comma-separated inline.
|
||||
- **Duplicates**: the earlier version section wins.
|
||||
- **Taiga references**: resolve to the GitHub issue via description text or
|
||||
PRs mentioning the Taiga URL, then link issue + PR.
|
||||
- **GHSA entries** live under `:bug:` with only the advisory URL (see 5b).
|
||||
- **Re-fetch before editing**; prefer `scripts/gh.py` over raw `gh api`.
|
||||
- **Verify PR merge status** (`check-merged`); PR-level exclusions apply.
|
||||
- **Cross-reference PRs, not just issues** (`cross-ref`); watch for
|
||||
false-positive PR-to-issue links.
|
||||
|
||||
@ -9,8 +9,7 @@
|
||||
app.common.data.macros/get-in clojure.core/get-in
|
||||
app.common.data.macros/with-open clojure.core/with-open
|
||||
app.common.data.macros/select-keys clojure.core/select-keys
|
||||
app.common.logging/with-context clojure.core/do
|
||||
benches.render-wasm.errors/attempt clojure.core/do}
|
||||
app.common.logging/with-context clojure.core/do}
|
||||
|
||||
:hooks
|
||||
{:analyze-call
|
||||
@ -25,11 +24,7 @@
|
||||
shadow.lazy/loadable hooks.export/rumext-lazycomponent
|
||||
app.util.i18n/tr hooks.i18n/tr-dynamic
|
||||
app.common.i18n/tr hooks.i18n/tr-dynamic
|
||||
benches.render-wasm.declarations/defscene hooks.benches/defscene
|
||||
benches.render-wasm.declarations/defcase hooks.benches/defcase
|
||||
benches.render-wasm.runtime.commands/defcommand hooks.benches/defcommand
|
||||
benches.render-wasm.runtime.protocol/resolve-or-reject hooks.benches/resolve-or-reject
|
||||
benches.render-wasm.runtime.protocol/resolve-or-reject-with hooks.benches/resolve-or-reject-with}}
|
||||
}}
|
||||
|
||||
:output
|
||||
{:exclude-files
|
||||
@ -41,7 +36,7 @@
|
||||
"^(?:backend|frontend|exporter|common)/deps.edn"
|
||||
"^(?:backend|frontend|exporter|common)/scripts/"
|
||||
"^(?:backend|frontend|exporter|common)/dev/"
|
||||
"^(?:backend|frontend|exporter|common)/test/(?!performance_tests/)"]
|
||||
"^(?:backend|frontend|exporter|common)/test/"]
|
||||
|
||||
:linter-name true}
|
||||
|
||||
@ -106,4 +101,6 @@
|
||||
|
||||
:unused-binding
|
||||
{:exclude-destructured-as true
|
||||
:exclude-destructured-keys-in-fn-args false}}}
|
||||
:exclude-destructured-keys-in-fn-args false
|
||||
}
|
||||
}}
|
||||
|
||||
@ -1,77 +0,0 @@
|
||||
;; This Source Code Form is subject to the terms of the Mozilla Public
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns hooks.benches
|
||||
(:require [clj-kondo.hooks-api :as api]))
|
||||
|
||||
(defn defscene
|
||||
"Analyzes scene options and its compiled builder body."
|
||||
[{:keys [node]}]
|
||||
(let [[_ _ opts argv & body] (:children node)]
|
||||
(if (and argv (api/vector-node? argv))
|
||||
{:node (api/list-node [(api/token-node (quote do)) opts
|
||||
(api/list-node (into [(api/token-node (quote fn)) argv] body))])}
|
||||
{:node node})))
|
||||
|
||||
(defn defcase
|
||||
"Analyzes case options and its compiled operation body."
|
||||
[{:keys [node]}]
|
||||
(let [[_ _ opts argv & body] (:children node)]
|
||||
(if (and (seq body) argv (api/vector-node? argv))
|
||||
{:node (api/list-node [(api/token-node (quote do)) opts
|
||||
(api/list-node (into [(api/token-node (quote fn)) argv] body))])}
|
||||
{:node node})))
|
||||
|
||||
(defn defcommand
|
||||
"Analyzes command options and both generated functions."
|
||||
[{:keys [node]}]
|
||||
(let [[_ command & forms] (:children node)
|
||||
[doc forms] (if (string? (api/sexpr (first forms)))
|
||||
[(first forms) (next forms)]
|
||||
[nil forms])
|
||||
[options forms] (if (api/map-node? (first forms))
|
||||
[(first forms) (next forms)]
|
||||
[nil forms])
|
||||
[argv & body] forms
|
||||
[conditions body] (if (and (api/map-node? (first body)) (next body))
|
||||
[(first body) (next body)]
|
||||
[nil body])
|
||||
handlers (:async (some-> options api/sexpr))
|
||||
core (api/list-node
|
||||
(into [(api/token-node 'fn)
|
||||
(api/vector-node (mapv api/token-node handlers))]
|
||||
body))
|
||||
starred (api/token-node (symbol (str (api/sexpr command) "*")))
|
||||
measured (get (some-> options api/sexpr) :measured-variant true)
|
||||
definition (api/list-node
|
||||
(into [(api/token-node 'defn) command]
|
||||
(concat (when doc [doc]) [argv]
|
||||
(when conditions [conditions]) [core])))]
|
||||
{:node (api/list-node
|
||||
(into [(api/token-node 'do)]
|
||||
(concat (when options [(api/coerce (dissoc (api/sexpr options) :async))])
|
||||
[definition]
|
||||
(when measured
|
||||
[(api/list-node [(api/token-node 'def)
|
||||
starred command])]))))}))
|
||||
|
||||
(defn resolve-or-reject
|
||||
"Analyzes the body of a Promise-returning form."
|
||||
[{:keys [node]}]
|
||||
(let [[_ & body] (:children node)]
|
||||
{:node (api/list-node (into [(api/token-node (quote do))] body))}))
|
||||
|
||||
(defn resolve-or-reject-with
|
||||
"Analyzes the cause handler and body. Binds `cause` in the handler only."
|
||||
[{:keys [node]}]
|
||||
(let [[_ cause on-error & body] (:children node)]
|
||||
{:node (api/list-node
|
||||
(into [(api/token-node (quote do))
|
||||
(api/list-node
|
||||
[(api/token-node (quote let))
|
||||
(api/vector-node [cause (api/token-node nil)])
|
||||
on-error])]
|
||||
body))}))
|
||||
@ -1,9 +1,3 @@
|
||||
;; This Source Code Form is subject to the terms of the Mozilla Public
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns hooks.export
|
||||
(:require [clj-kondo.hooks-api :as api]))
|
||||
|
||||
|
||||
@ -10,7 +10,5 @@
|
||||
:extra-indents {rumext.v2/fnc [[:inner 0]]
|
||||
cljs.test/async [[:inner 0]]
|
||||
promesa.exec/thread [[:inner 0]]
|
||||
specify! [[:inner 0] [:inner 1]]
|
||||
benches.render-wasm.runtime.commands/defcommand [[:inner 0]]
|
||||
benches.render-wasm.runtime.protocol/resolve-or-reject-with [[:inner 0]]}
|
||||
specify! [[:inner 0] [:inner 1]]}
|
||||
}
|
||||
|
||||
@ -85,10 +85,6 @@ target instance is running and its MCP config has been generated:
|
||||
# 'main', 'ws1' and similar spellings are rejected.
|
||||
./manage.sh start-coding-agent claude --ws 1
|
||||
./manage.sh start-coding-agent opencode --ws 2
|
||||
|
||||
# Run a different executable (e.g. a wrapper script) with the same MCP
|
||||
# wiring. --ws and --bin may come in any order.
|
||||
./manage.sh start-coding-agent claude --bin claude-alt
|
||||
```
|
||||
|
||||
Equivalents by hand (run from inside the workspace directory):
|
||||
|
||||
@ -1,7 +1,7 @@
|
||||
description: An unexpected problem or behavior
|
||||
labels: ["needs triage"]
|
||||
name: Bug
|
||||
description: Create a report to help us improve
|
||||
name: Bug report
|
||||
type: Bug
|
||||
labels: ["needs triage"]
|
||||
|
||||
body:
|
||||
- type: markdown
|
||||
@ -15,7 +15,7 @@ body:
|
||||
- type: textarea
|
||||
attributes:
|
||||
label: Steps To Reproduce
|
||||
description: Steps to reproduce the behavior
|
||||
description: Steps to reproduce the behavior.
|
||||
placeholder: |
|
||||
Steps to reproduce the behavior:
|
||||
1. Go to '...'
|
||||
@ -26,22 +26,22 @@ body:
|
||||
- type: textarea
|
||||
id: expected
|
||||
attributes:
|
||||
description: A clear and concise description of what you expected to happen.
|
||||
label: Expected behavior
|
||||
description: A clear and concise description of what you expected to happen
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: actual
|
||||
attributes:
|
||||
description: A clear and concise description of what happens instead; what the bug is.
|
||||
label: Actual behavior
|
||||
description: A clear and concise description of what happens instead; what the bug is
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: screenshots
|
||||
attributes:
|
||||
description: If applicable, add screenshots to help explain your problem.
|
||||
label: Screenshots or video
|
||||
description: If applicable, add screenshots to help explain your problem
|
||||
- type: textarea
|
||||
id: desktop
|
||||
attributes:
|
||||
@ -87,4 +87,4 @@ body:
|
||||
id: additional-context
|
||||
attributes:
|
||||
label: Additional context
|
||||
description: Add any other context or screenshots about the bug here
|
||||
description: Any other context about the problem.
|
||||
2
.github/ISSUE_TEMPLATE/config.yml
vendored
2
.github/ISSUE_TEMPLATE/config.yml
vendored
@ -1 +1 @@
|
||||
blank_issues_enabled: false
|
||||
blank_issues_enabled: true
|
||||
|
||||
38
.github/ISSUE_TEMPLATE/enhancement.yml
vendored
38
.github/ISSUE_TEMPLATE/enhancement.yml
vendored
@ -1,38 +0,0 @@
|
||||
description: Improvement to existing behavior
|
||||
labels: ["needs triage"]
|
||||
name: "Enhancement"
|
||||
type: Enhancement
|
||||
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
## Before you start
|
||||
|
||||
Please search our [existing issues](https://github.com/penpot/penpot/issues) and open [pull requests](https://github.com/penpot/penpot/pulls) to lessen the change of filing duplicate issues or feature requests. Thank you.
|
||||
|
||||
---
|
||||
- type: textarea
|
||||
id: problem
|
||||
attributes:
|
||||
label: Is your feature request related to a problem? Please describe
|
||||
description: A clear and concise description of what the problem is. Ex. I'm always frustrated when (...)
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: solution
|
||||
attributes:
|
||||
label: Describe the solution you'd like
|
||||
description: A clear and concise description of what you want to happen
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: alternatives
|
||||
attributes:
|
||||
label: Describe alternatives you've considered
|
||||
description: A clear and concise description of any alternative solutions or features you've considered
|
||||
- type: textarea
|
||||
id: additional-context
|
||||
attributes:
|
||||
label: Additional context
|
||||
description: Add any other context or screenshots about the enhancement here
|
||||
26
.github/ISSUE_TEMPLATE/epic.yml
vendored
26
.github/ISSUE_TEMPLATE/epic.yml
vendored
@ -1,26 +0,0 @@
|
||||
description: An epic is a large user story that cannot be delivered as defined within a single iteration
|
||||
labels: ["needs triage"]
|
||||
name: "EPIC"
|
||||
type: EPIC
|
||||
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
## Before you start
|
||||
|
||||
Please search our [existing issues](https://github.com/penpot/penpot/issues) and open [pull requests](https://github.com/penpot/penpot/pulls) to lessen the change of filing duplicate issues or feature requests. Thank you.
|
||||
|
||||
---
|
||||
- type: textarea
|
||||
id: description
|
||||
attributes:
|
||||
label: Epic description
|
||||
description: Tell us which epic feature do you want to discuss
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: additional-context
|
||||
attributes:
|
||||
label: Additional context
|
||||
description: Add any other context or screenshots about the feature request here
|
||||
@ -1,7 +1,7 @@
|
||||
description: Suggest an idea for this project
|
||||
description: Suggest an idea for this project.
|
||||
labels: ["needs triage"]
|
||||
name: "Feature request"
|
||||
type: Feature
|
||||
type: Enhancement
|
||||
|
||||
body:
|
||||
- type: markdown
|
||||
@ -15,24 +15,24 @@ body:
|
||||
- type: textarea
|
||||
id: problem
|
||||
attributes:
|
||||
label: Is your feature request related to a problem? Please describe
|
||||
description: A clear and concise description of what the problem is. Ex. I'm always frustrated when (...)
|
||||
label: Is your feature request related to a problem? Please describe.
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: solution
|
||||
attributes:
|
||||
label: Describe the solution you'd like
|
||||
description: A clear and concise description of what you want to happen
|
||||
description: A clear and concise description of what you want to happen.
|
||||
label: Describe the solution you'd like.
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: alternatives
|
||||
attributes:
|
||||
label: Describe alternatives you've considered
|
||||
description: A clear and concise description of any alternative solutions or features you've considered
|
||||
label: Describe alternatives you've considered.
|
||||
description: A clear and concise description of any alternative solutions or features you've considered.
|
||||
- type: textarea
|
||||
id: additional-context
|
||||
attributes:
|
||||
label: Additional context
|
||||
description: Add any other context or screenshots about the feature request here
|
||||
description: Add any other context or screenshots about the feature request here.
|
||||
26
.github/ISSUE_TEMPLATE/task.yml
vendored
26
.github/ISSUE_TEMPLATE/task.yml
vendored
@ -1,26 +0,0 @@
|
||||
description: A specific piece of work
|
||||
labels: []
|
||||
name: "Task"
|
||||
type: Task
|
||||
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
## Before you start
|
||||
|
||||
Please search our [existing issues](https://github.com/penpot/penpot/issues) and open [pull requests](https://github.com/penpot/penpot/pulls) to lessen the change of filing duplicate issues or feature requests. Thank you.
|
||||
|
||||
---
|
||||
- type: textarea
|
||||
id: task
|
||||
attributes:
|
||||
description: A clear and concise description of the task
|
||||
label: Please describe
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: additional-context
|
||||
attributes:
|
||||
label: Additional context
|
||||
description: Add any other context or screenshots about the task here
|
||||
20
.github/dependabot.yml
vendored
Normal file
20
.github/dependabot.yml
vendored
Normal file
@ -0,0 +1,20 @@
|
||||
# https://docs.github.com/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file
|
||||
version: 2
|
||||
updates:
|
||||
# Actions are pinned to a commit SHA with a `# vX.Y.Z` comment; Dependabot
|
||||
# bumps both. All updates are grouped into a single monthly PR.
|
||||
- package-ecosystem: "github-actions"
|
||||
directory: "/"
|
||||
schedule:
|
||||
interval: "monthly"
|
||||
open-pull-requests-limit: 5
|
||||
labels:
|
||||
- "dependencies"
|
||||
- "team infra"
|
||||
- "no issue required"
|
||||
commit-message:
|
||||
prefix: ":arrow_up:"
|
||||
groups:
|
||||
github-actions:
|
||||
patterns:
|
||||
- "*"
|
||||
160
.github/workflows/auto-label.yml
vendored
160
.github/workflows/auto-label.yml
vendored
@ -5,7 +5,7 @@ name: Auto Label and Add to Project
|
||||
# code, it only reads ids from the event payload.
|
||||
on:
|
||||
issues:
|
||||
types: [opened, closed]
|
||||
types: [opened]
|
||||
pull_request_target:
|
||||
types: [opened]
|
||||
|
||||
@ -43,9 +43,6 @@ jobs:
|
||||
// === 1. CONFIGURATION ===
|
||||
const PROJECT_NUMBER = 8; // <--- Replace with your project board number
|
||||
const IS_ORG = true; // <--- Set to false if this is a personal project, true if an organization
|
||||
const STATUS_FIELD_ID = 'PVTSSF_lADOAcyBPM4BV2pqzhRPY48';
|
||||
const STATUS_BACKLOG = 'f75ad846';
|
||||
const STATUS_TECH_REVIEW = 'b0f45db9';
|
||||
const OWNER = context.repo.owner;
|
||||
const REPO = context.repo.repo;
|
||||
|
||||
@ -53,100 +50,6 @@ jobs:
|
||||
const isPR = !!context.payload.pull_request;
|
||||
const contentId = isPR ? context.payload.pull_request.node_id : context.payload.issue.node_id;
|
||||
|
||||
// A completed issue is already closed by GitHub. Route its Main
|
||||
// project status according to whether QA needs to verify it.
|
||||
if (context.eventName === 'issues' && context.payload.action === 'closed') {
|
||||
const issue = context.payload.issue;
|
||||
if (issue.state_reason !== 'completed') {
|
||||
console.log(`Issue #${issue.number} was closed as ${issue.state_reason}; leaving project status unchanged.`);
|
||||
return;
|
||||
}
|
||||
|
||||
const targetStatus = issue.labels.some(({ name }) => name === 'team-qa')
|
||||
? 'Ready for QA'
|
||||
: 'Product Review';
|
||||
|
||||
const projectQuery = `
|
||||
query($owner: String!, $number: Int!) {
|
||||
organization(login: $owner) {
|
||||
projectV2(number: $number) {
|
||||
id
|
||||
fields(first: 100) {
|
||||
nodes {
|
||||
... on ProjectV2SingleSelectField {
|
||||
id
|
||||
name
|
||||
options { id name }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
`;
|
||||
const projectRes = await github.graphql(projectQuery, {
|
||||
owner: OWNER,
|
||||
number: PROJECT_NUMBER
|
||||
});
|
||||
const project = projectRes.organization.projectV2;
|
||||
if (!project) {
|
||||
throw new Error(`Could not find Main project #${PROJECT_NUMBER}.`);
|
||||
}
|
||||
|
||||
const statusField = project.fields.nodes.find(({ name }) => name === 'Status');
|
||||
const statusOption = statusField?.options.find(({ name }) => name === targetStatus);
|
||||
if (!statusField || !statusOption) {
|
||||
throw new Error(`Could not find Status option "${targetStatus}" in Main.`);
|
||||
}
|
||||
|
||||
const issueProjectItemsQuery = `
|
||||
query($issueId: ID!) {
|
||||
node(id: $issueId) {
|
||||
... on Issue {
|
||||
projectItems(first: 100) {
|
||||
nodes {
|
||||
id
|
||||
project { id }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
`;
|
||||
const issueProjectItemsRes = await github.graphql(issueProjectItemsQuery, {
|
||||
issueId: contentId
|
||||
});
|
||||
const projectItems = issueProjectItemsRes.node?.projectItems?.nodes || [];
|
||||
const projectItem = projectItems.find(
|
||||
({ project: itemProject }) => itemProject.id === project.id
|
||||
);
|
||||
if (!projectItem) {
|
||||
console.log(`Issue #${issue.number} is not in Main project #${PROJECT_NUMBER}; skipping status update.`);
|
||||
return;
|
||||
}
|
||||
|
||||
const updateStatusMutation = `
|
||||
mutation($projectId: ID!, $itemId: ID!, $fieldId: ID!, $optionId: String!) {
|
||||
updateProjectV2ItemFieldValue(input: {
|
||||
projectId: $projectId
|
||||
itemId: $itemId
|
||||
fieldId: $fieldId
|
||||
value: { singleSelectOptionId: $optionId }
|
||||
}) {
|
||||
projectV2Item { id }
|
||||
}
|
||||
}
|
||||
`;
|
||||
await github.graphql(updateStatusMutation, {
|
||||
projectId: project.id,
|
||||
itemId: projectItem.id,
|
||||
fieldId: statusField.id,
|
||||
optionId: statusOption.id
|
||||
});
|
||||
console.log(`Issue #${issue.number} closed; Main status set to "${targetStatus}".`);
|
||||
return;
|
||||
}
|
||||
|
||||
// Define your labels here
|
||||
const labelToApply = 'needs triage';
|
||||
|
||||
@ -185,65 +88,6 @@ jobs:
|
||||
}
|
||||
`;
|
||||
|
||||
let itemId;
|
||||
try {
|
||||
const addRes = await github.graphql(addToProjectMutation, { projectId, contentId });
|
||||
itemId = addRes.addProjectV2ItemById.item.id;
|
||||
} catch (error) {
|
||||
const messages = [
|
||||
error && error.message,
|
||||
...(Array.isArray(error && error.errors)
|
||||
? error.errors.map(e => e && e.message)
|
||||
: [])
|
||||
].filter(Boolean).join('\n');
|
||||
if (!messages.includes('Content already exists in this project')) {
|
||||
throw error;
|
||||
}
|
||||
console.log('Item already in project, fetching existing item...');
|
||||
const existing = await github.graphql(
|
||||
`query($contentId: ID!) {
|
||||
node(id: $contentId) {
|
||||
... on Issue {
|
||||
projectItems(first: 50) {
|
||||
nodes { id project { id } }
|
||||
}
|
||||
}
|
||||
... on PullRequest {
|
||||
projectItems(first: 50) {
|
||||
nodes { id project { id } }
|
||||
}
|
||||
}
|
||||
}
|
||||
}`,
|
||||
{ contentId }
|
||||
);
|
||||
const nodes = (existing.node && existing.node.projectItems && existing.node.projectItems.nodes) || [];
|
||||
const match = nodes.find(n => n.project.id === projectId);
|
||||
if (!match) {
|
||||
throw new Error(`Item already in a project, but not found in project ${projectId}`);
|
||||
}
|
||||
itemId = match.id;
|
||||
}
|
||||
|
||||
// === 4. SET PROJECT STATUS ===
|
||||
const statusOptionId = isPR ? STATUS_TECH_REVIEW : STATUS_BACKLOG;
|
||||
const statusName = isPR ? 'Tech Review' : 'Backlog';
|
||||
console.log(`Setting status to "${statusName}" for ${isPR ? 'PR' : 'Issue'} #${issueNumber}...`);
|
||||
await github.graphql(
|
||||
`mutation($projectId: ID!, $itemId: ID!, $fieldId: ID!, $optionId: String!) {
|
||||
updateProjectV2ItemFieldValue(input: {
|
||||
projectId: $projectId,
|
||||
itemId: $itemId,
|
||||
fieldId: $fieldId,
|
||||
value: { singleSelectOptionId: $optionId }
|
||||
}) {
|
||||
projectV2Item {
|
||||
id
|
||||
}
|
||||
}
|
||||
}`,
|
||||
{ projectId, itemId, fieldId: STATUS_FIELD_ID, optionId: statusOptionId }
|
||||
);
|
||||
|
||||
await github.graphql(addToProjectMutation, { projectId, contentId });
|
||||
console.log("Automation successfully completed!");
|
||||
|
||||
|
||||
4
.github/workflows/build-docker-devenv.yml
vendored
4
.github/workflows/build-docker-devenv.yml
vendored
@ -38,13 +38,13 @@ jobs:
|
||||
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
|
||||
|
||||
- name: Login to Docker Registry (push destination)
|
||||
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
|
||||
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
|
||||
with:
|
||||
username: ${{ secrets.PUB_DOCKER_USERNAME }}
|
||||
password: ${{ secrets.PUB_DOCKER_PASSWORD }}
|
||||
|
||||
- name: Login to Docker Hardened Images registry (base image pull)
|
||||
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
|
||||
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
|
||||
with:
|
||||
registry: dhi.io
|
||||
username: ${{ secrets.PUB_DOCKER_USERNAME }}
|
||||
|
||||
10
.github/workflows/build-docker.yml
vendored
10
.github/workflows/build-docker.yml
vendored
@ -200,7 +200,7 @@ jobs:
|
||||
ref: ${{ inputs.sha != '' && inputs.sha || inputs.gh_ref }}
|
||||
|
||||
- name: Login to Docker Registry
|
||||
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
|
||||
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
|
||||
with:
|
||||
registry: ${{ secrets.DOCKER_REGISTRY }}
|
||||
username: ${{ secrets.DOCKER_USERNAME }}
|
||||
@ -210,7 +210,7 @@ jobs:
|
||||
# images from DockerHub for unregistered users.
|
||||
# https://docs.docker.com/docker-hub/usage/
|
||||
- name: Login to DockerHub Registry
|
||||
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
|
||||
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
|
||||
with:
|
||||
username: ${{ secrets.PUB_DOCKER_USERNAME }}
|
||||
password: ${{ secrets.PUB_DOCKER_PASSWORD }}
|
||||
@ -222,7 +222,7 @@ jobs:
|
||||
# reuses the same PUB_DOCKER_* credentials as the DockerHub
|
||||
# login above.
|
||||
- name: Login to Docker Hardened Images registry (base image pull)
|
||||
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
|
||||
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
|
||||
with:
|
||||
registry: dhi.io
|
||||
username: ${{ secrets.PUB_DOCKER_USERNAME }}
|
||||
@ -301,7 +301,7 @@ jobs:
|
||||
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
|
||||
|
||||
- name: Login to Docker Registry
|
||||
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
|
||||
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
|
||||
with:
|
||||
registry: ${{ secrets.DOCKER_REGISTRY }}
|
||||
username: ${{ secrets.DOCKER_USERNAME }}
|
||||
@ -371,7 +371,7 @@ jobs:
|
||||
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
|
||||
|
||||
- name: Login to Docker Registry
|
||||
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
|
||||
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
|
||||
with:
|
||||
registry: ${{ secrets.DOCKER_REGISTRY }}
|
||||
username: ${{ secrets.DOCKER_USERNAME }}
|
||||
|
||||
12
.github/workflows/build-renovate.yml
vendored
12
.github/workflows/build-renovate.yml
vendored
@ -114,20 +114,20 @@ jobs:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Login to Docker Registry
|
||||
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
|
||||
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
|
||||
with:
|
||||
registry: ${{ secrets.DOCKER_REGISTRY }}
|
||||
username: ${{ secrets.DOCKER_USERNAME }}
|
||||
password: ${{ secrets.DOCKER_PASSWORD }}
|
||||
|
||||
- name: Login to DockerHub Registry
|
||||
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
|
||||
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
|
||||
with:
|
||||
username: ${{ secrets.PUB_DOCKER_USERNAME }}
|
||||
password: ${{ secrets.PUB_DOCKER_PASSWORD }}
|
||||
|
||||
- name: Login to Docker Hardened Images registry
|
||||
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
|
||||
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
|
||||
with:
|
||||
registry: dhi.io
|
||||
username: ${{ secrets.PUB_DOCKER_USERNAME }}
|
||||
@ -191,13 +191,13 @@ jobs:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Login to DockerHub Registry
|
||||
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
|
||||
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
|
||||
with:
|
||||
username: ${{ secrets.PUB_DOCKER_USERNAME }}
|
||||
password: ${{ secrets.PUB_DOCKER_PASSWORD }}
|
||||
|
||||
- name: Login to Docker Hardened Images registry
|
||||
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
|
||||
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
|
||||
with:
|
||||
registry: dhi.io
|
||||
username: ${{ secrets.PUB_DOCKER_USERNAME }}
|
||||
@ -242,7 +242,7 @@ jobs:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Login to Docker Hardened Images registry
|
||||
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
|
||||
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
|
||||
with:
|
||||
registry: dhi.io
|
||||
username: ${{ secrets.PUB_DOCKER_USERNAME }}
|
||||
|
||||
2
.github/workflows/lint-workflows.yml
vendored
2
.github/workflows/lint-workflows.yml
vendored
@ -34,7 +34,7 @@ permissions: {}
|
||||
|
||||
env:
|
||||
ACTIONLINT_VERSION: '1.7.12'
|
||||
ZIZMOR_VERSION: '1.32.0'
|
||||
ZIZMOR_VERSION: '1.31.1'
|
||||
|
||||
jobs:
|
||||
lint:
|
||||
|
||||
4
.github/workflows/plugins-deploy-api-doc.yml
vendored
4
.github/workflows/plugins-deploy-api-doc.yml
vendored
@ -48,10 +48,10 @@ jobs:
|
||||
# No dependency cache here: a deploy must not restore a cache that
|
||||
# other workflows can write to.
|
||||
- name: Setup pnpm + Node.js
|
||||
uses: pnpm/setup@731dfb6463bbd11c11fd0ee11fe00ed0e36d48a1 # v3.2.0
|
||||
uses: pnpm/setup@a0881977cd8ea73b1ed212236c99ca10dcc80cf4 # v3.1.1
|
||||
with:
|
||||
working-directory: plugins
|
||||
runtime: node@24.22.3
|
||||
runtime: node@24.22.1
|
||||
install: false
|
||||
|
||||
- name: Install deps
|
||||
|
||||
@ -46,10 +46,10 @@ jobs:
|
||||
# No dependency cache here: a deploy must not restore a cache that
|
||||
# other workflows can write to.
|
||||
- name: Setup pnpm + Node.js
|
||||
uses: pnpm/setup@731dfb6463bbd11c11fd0ee11fe00ed0e36d48a1 # v3.2.0
|
||||
uses: pnpm/setup@a0881977cd8ea73b1ed212236c99ca10dcc80cf4 # v3.1.1
|
||||
with:
|
||||
working-directory: plugins
|
||||
runtime: node@24.22.3
|
||||
runtime: node@24.22.1
|
||||
install: false
|
||||
|
||||
- name: Install deps
|
||||
|
||||
@ -1,76 +0,0 @@
|
||||
name: Require Milestone on Linked Issues
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types:
|
||||
- opened
|
||||
- reopened
|
||||
- synchronize
|
||||
- edited
|
||||
- ready_for_review
|
||||
- converted_to_draft
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: read
|
||||
issues: read
|
||||
|
||||
jobs:
|
||||
check-linked-issues:
|
||||
name: Require Milestones on Linked Issues
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- name: Check linked issues
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
PR_NUMBER: ${{ github.event.pull_request.number }}
|
||||
IS_DRAFT: ${{ github.event.pull_request.draft }}
|
||||
REPOSITORY: ${{ github.repository }}
|
||||
run: |
|
||||
if [ "$IS_DRAFT" = "true" ]; then
|
||||
echo "Draft PR: milestone check skipped."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
OWNER="${REPOSITORY%%/*}"
|
||||
REPO="${REPOSITORY#*/}"
|
||||
|
||||
# shellcheck disable=SC2016 # GraphQL variables, not shell expansion.
|
||||
ISSUES=$(gh api graphql \
|
||||
-F owner="$OWNER" \
|
||||
-F repo="$REPO" \
|
||||
-F number="$PR_NUMBER" \
|
||||
-f query='
|
||||
query($owner: String!, $repo: String!, $number: Int!) {
|
||||
repository(owner: $owner, name: $repo) {
|
||||
pullRequest(number: $number) {
|
||||
closingIssuesReferences(first: 100) {
|
||||
nodes {
|
||||
number
|
||||
title
|
||||
url
|
||||
milestone {
|
||||
title
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
')
|
||||
|
||||
MISSING=$(echo "$ISSUES" | jq -r '
|
||||
.data.repository.pullRequest.closingIssuesReferences.nodes[]
|
||||
| select(.milestone == null)
|
||||
| "- #\(.number): \(.title) (\(.url))"
|
||||
')
|
||||
|
||||
if [ -n "$MISSING" ]; then
|
||||
echo "::error::The following linked issues have no milestone:"
|
||||
echo "$MISSING"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "All linked issues have a milestone."
|
||||
45
.github/workflows/require-milestone.yml
vendored
45
.github/workflows/require-milestone.yml
vendored
@ -1,45 +0,0 @@
|
||||
name: Require Milestone
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types:
|
||||
- opened
|
||||
- reopened
|
||||
- synchronize
|
||||
- edited
|
||||
- milestoned
|
||||
- demilestoned
|
||||
- ready_for_review
|
||||
- converted_to_draft
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: read
|
||||
|
||||
jobs:
|
||||
check-milestone:
|
||||
name: Require Milestone
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- name: Check milestone
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
PR_NUMBER: ${{ github.event.pull_request.number }}
|
||||
run: |
|
||||
pr_api="repos/${GITHUB_REPOSITORY}/pulls/${PR_NUMBER}"
|
||||
|
||||
is_draft=$(gh api "$pr_api" --jq '.draft')
|
||||
milestone=$(gh api "$pr_api" --jq '.milestone.title // ""')
|
||||
|
||||
if [ "$is_draft" = "true" ]; then
|
||||
echo "Draft PR: milestone check skipped."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [ -z "$milestone" ]; then
|
||||
echo "::error::This pull request must have a milestone."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Milestone assigned: $milestone"
|
||||
2
.github/workflows/tests-wasm.yml
vendored
2
.github/workflows/tests-wasm.yml
vendored
@ -49,7 +49,7 @@ jobs:
|
||||
- name: Format
|
||||
working-directory: ./render-wasm
|
||||
run: |
|
||||
./fmt --check
|
||||
cargo fmt --check
|
||||
|
||||
- name: Lint
|
||||
working-directory: ./render-wasm
|
||||
|
||||
2
.gitignore
vendored
2
.gitignore
vendored
@ -121,7 +121,7 @@ opencode.json
|
||||
/.opencode/reports
|
||||
/.opencode/prompts
|
||||
/.ci-logs
|
||||
/tmp
|
||||
/tmp/
|
||||
/.codex/
|
||||
/scripts/__pycache__/
|
||||
/tools/__pycache__
|
||||
|
||||
@ -12,15 +12,14 @@ The system has:
|
||||
|
||||
- **Three producers** (they create events): the Penpot backend (RPC layer), the Penpot frontend (browser app), and the Nitrate admin-console, which sends both server-side and client-side events.
|
||||
- **One central buffer**: the Postgres `audit_log` table in Penpot. It is temporary: it holds events for minutes/hours until they are archived, then they are deleted.
|
||||
- **Four consumers** (they read events): webhooks, error reporters, telemetry, and the external archives in Nexus and in the admin console.
|
||||
- **Two long-term archives**: **Nexus**, a separate service that stores events for the long term, and the **admin console** (Nitrate), which ingests its own allowlisted events through a second archive job (see section 12).
|
||||
- **Four consumers** (they read events): webhooks, error reporters, telemetry, and the external archive in Nexus.
|
||||
- **One long-term archive**: **Nexus**, a separate service that stores events for the long term. Archiving to **Nitrate** is in development and **does not exist on the current branch** (see section 11).
|
||||
|
||||
```
|
||||
Penpot frontend ─ POST push-audit-events (public) ─────────▶┐
|
||||
Nitrate admin-console (browser) ─ POST push-audit-events ──▶│
|
||||
Nitrate admin-console (server) ─ POST push-audit-events ──▶│ ─▶ audit_log table (buffer) ──┬─ cron 0s every 5m ──▶ Nexus (archive)
|
||||
Penpot backend (RPC wrap-audit) ─ submit ───────────────────┘ │ └─ cron 30s every 5m ─▶ admin console (archive)
|
||||
│ ├──▶ webhooks
|
||||
Nitrate admin-console (server) ─ POST push-audit-events ──▶│ ─▶ audit_log table (buffer) ── cron every 5m ──▶ Nexus (archive)
|
||||
Penpot backend (RPC wrap-audit) ─ submit ───────────────────┘ │ ├──▶ webhooks
|
||||
│ ├──▶ error reporters
|
||||
└──▶ telemetry (shadow rows)
|
||||
```
|
||||
@ -45,7 +44,7 @@ Rule of thumb: if you need to know for sure "did this user do X?", look at `sour
|
||||
|
||||
## 3. The `audit_log` table: a buffer, not a store
|
||||
|
||||
The Penpot `audit_log` table **is not the final destination**. It is a waiting tray: events sit here until every active consumer (Nexus, admin console) confirms them, and whoever finishes last deletes the row.
|
||||
The Penpot `audit_log` table **is not the final destination**. It is a waiting tray: events sit here until the archive cron sends them to Nexus and the cleanup cron (GC) deletes them.
|
||||
|
||||
Main columns (migration `backend/src/app/migrations/sql/0054-add-audit-log-table.sql` and later ones):
|
||||
|
||||
@ -57,16 +56,15 @@ Main columns (migration `backend/src/app/migrations/sql/0054-add-audit-log-table
|
||||
- `source` text: `backend` / `frontend` for full rows, `telemetry:backend` / `telemetry:frontend` for anonymized copies (see section 10).
|
||||
- `ip_addr` inet: real IP (or `0.0.0.0` on telemetry rows).
|
||||
- `props` / `context` jsonb holding transit-encoded maps: action data and metadata (versions, browser, request-id...).
|
||||
- No `archived_at`: the `audit_log_pending` queue (migration `0157`, consumer `1` = Nexus, `2` = admin console, ids never reused) tracks what is left. Every record is born pending only for the consumers enabled at insert time (never a fixed both): cheaper on single-consumer hosts and true to the design doc. Consequence, accepted on purpose: rows born with no send flag on get no pendings, so enabling archiving later does not pick them up — the sweep deletes them unarchived.
|
||||
- `archived_at` timestamptz: NULL until Nexus confirms receipt; only then the row can be deleted.
|
||||
|
||||
Indexes (each consumer has its own, so a slow consumer never blocks the others):
|
||||
|
||||
- `audit_log_pending` PK `(consumer, audit_log_id)`: each consumer owns its queue at constant cost, even with millions pending.
|
||||
- `audit_log (created_at, id)` (migration `0157`): the chronological drain order (`ORDER BY created_at ASC, id ASC`; same total order for both consumers, so concurrent drains cannot deadlock).
|
||||
- `audit_log_pending (audit_log_id)`: serves the "is anything still pending?" check of the final delete.
|
||||
- Partial `created_at WHERE archived_at IS NULL`: used by the archive scan.
|
||||
- Partial `archived_at WHERE archived_at IS NOT NULL`: used by the GC.
|
||||
- `(source, created_at)`: used by the telemetry scan.
|
||||
|
||||
Life of a row: `INSERT` (row plus one pending per enabled consumer, same transaction; a rollback takes both) → (every 5 min, Nexus at second 0 and admin console at second 30) each archive job sends its own 128-row batches oldest-first and deletes its pendings in the same transaction → whoever deletes the last pending deletes the row. A batch the destination refuses rolls back and stays for the next tick. Pendings of consumers that are off are reaped at the start of each job run (covers later deactivations), and orphan rows are swept at the end of each run, so there is no separate GC task. The FK is plain `DEFERRABLE` (never `INITIALLY DEFERRED`): every deleter removes pendings before rows, including the telemetry task's own deletes.
|
||||
Life of a row: `INSERT` → (every 5 min) archive sets `archived_at = now()` **only if Nexus answers 204**, in the same transaction → (every 5 min) GC runs `DELETE WHERE archived_at IS NOT NULL` **with no age filter**. So archive must run before GC, or data is deleted without being sent.
|
||||
|
||||
## 4. Anatomy of an event
|
||||
|
||||
@ -182,7 +180,7 @@ Who uses it:
|
||||
|
||||
Failure policy: every send is best-effort. A failure never rolls back the business action; it is only recorded for debugging.
|
||||
|
||||
Backend side (`backend/src/app/rpc/management/nitrate.clj` `::push-audit-events`): `::audit/skip`, `::rpc/auth false` (trust comes from the shared key, not from a user session; only the `admin-console` key id is allowed). It builds context with `audit/prepare-context-from-request` + `request-id`, takes the given `profile-id` as-is (no coercion here, must already be a uuid), sets `tracked-at = request-at`, defaults `type` to `"action"`, and calls `audit/submit` (which defaults `source` to `"backend"`). So these rows land as **`source = backend`**: they are audited like backend events because a trusted service set the author.
|
||||
Backend side (`backend/src/app/rpc/management/nitrate.clj` `::push-audit-events`): `::audit/skip`, `::rpc/auth false` (trust comes from the shared key, not from a user session). It builds context with `audit/prepare-context-from-request` + `request-id`, takes the given `profile-id` as-is (no coercion here, must already be a uuid), sets `tracked-at = request-at`, defaults `type` to `"action"`, and calls `audit/submit` (which defaults `source` to `"backend"`). So these rows land as **`source = backend`**: they are audited like backend events because a trusted service set the author.
|
||||
|
||||
### 7b. Browser channel (best-effort, like frontend events)
|
||||
|
||||
@ -252,9 +250,9 @@ Result: crash reports only exist if the frontend collector runs and `push-audit-
|
||||
|
||||
## 11. Consumers III: telemetry (shadow rows)
|
||||
|
||||
Telemetry reuses the same table with **anonymized shadow rows** (`source LIKE 'telemetry:%'`): day-truncated times, `0.0.0.0` IPs, props reduced to uuid/boolean/number values plus a few allowed fields (`source`, `lang`, `auth-backend`, `activation`, derived `email-domain` — never raw emails), and a minimal context allowlist. Full + shadow rows can exist together per event when both flags are on; that duplication is on purpose.
|
||||
Telemetry reuses the same table with **anonymized shadow rows** (`source LIKE 'telemetry:%'`): day-truncated times, `0.0.0.0` IPs, props reduced to uuid/boolean/number values plus a few allowed fields (`lang`, `auth-backend`, derived `email-domain` — never raw emails), and a minimal context allowlist. Full + shadow rows can exist together per event when both flags are on; that duplication is on purpose.
|
||||
|
||||
Filters (`filter-telemetry-props` / `filter-telemetry-context`): by default only uuid/boolean/number with a simple key; exceptions: login/register/update-profile and frontend `identify` keep `lang`/`auth-backend`/`source`/`activation`/`email-domain`; `organization-sso-auth-failed` keeps `failure-reason` if it is a known one; `instance-start` passes whole; frontend `navigate` keeps `[:route :file-id :team-id :page-id]`. Subscription actions have event-specific contracts: backend and frontend `create-subscription` / `create-trial-subscription` preserve the default uuid/boolean/number props plus `type`, `deployment`, `quantity` and `subscription-mode`; backend `update-subscription` preserves only `type`, `deployment`, `previous-status`, `status`, `previous-cancel-at-period-end`, `cancel-at-period-end`, `previous-cancel-at`, `cancel-at` and `changed-fields`; backend `subscription-summary` preserves only `type`, `deployment`, `instance-id`, `snapshot-date`, the per-status `*-count` values, `total-count`, `sync-succeeded`, `sync-failed` and `snapshot-complete`. These subscription exceptions apply only to action events. Other free text drops from telemetry but **stays in the full row and in Nexus**. In `register-profile`, `activation` is a prop (it is what reaches PostHog). `action` exists only in the `context` of the full `register-profile-retry` rows; `filter-telemetry-context` drops it from the anonymised copy, which keeps only `:version`, `:initiator`, `:client-version` and `:client-user-agent`.
|
||||
Filters (`filter-telemetry-props` / `filter-telemetry-context`): by default only uuid/boolean/number with a simple key; exceptions: login/register/update-profile and frontend `identify` keep `lang`/`auth-backend`/`email-domain`; `organization-sso-auth-failed` keeps `failure-reason` if it is a known one; `instance-start` passes whole; frontend `navigate` keeps `[:route :file-id :team-id :page-id]`. Free text drops from telemetry but **stays in the full row and in Nexus**.
|
||||
|
||||
Shipping (`backend/src/app/tasks/telemetry.clj`): the cron sends shadows to `:telemetry-uri` as JSON in 10k batches and **deletes** them on success; it purges leftovers older than 7 days. It also sends a legacy report (counters, teams, email domains...). On official hosts nothing is collected or sent (`telemetry-excluded?` covers `penpot.app`/`penpot.dev`).
|
||||
|
||||
@ -273,19 +271,17 @@ Nexus dispatches on `:type`: `"telemetry-events"` batches are coerced and insert
|
||||
- **No personal data is expected.** Profile ids are random UUIDs with no PII, so the telemetry consumer needs no login guard; grouping is by `instance`, not by person (see posthog-telemetry below).
|
||||
- **The endpoint always answers 200.** Failures are swallowed and only counted (`nexus_telemetry_errors_total`), so a bad batch never breaks the sending instance; successes are counted (`nexus_telemetry_events_total`, `nexus_telemetry_reports_total`). A body the parser cannot read never reaches the handler: it answers 400 and counts in `nexus_http_malformed_requests_total`. Old instances may send a fressian+zstd base64 blob instead of a plain vector; Nexus decodes both (50 MiB cap against decompression bombs).
|
||||
|
||||
## 12. Archive to Nexus and to the admin console
|
||||
## 12. Archive to Nexus (today) and to Nitrate (future)
|
||||
|
||||
### How it works today
|
||||
### Nexus: how it works today
|
||||
|
||||
Every 5 min each archive job drains its own queue (`:audit-log-consumer-nexus` on flag `:nexus`, cron `0 */5 * * * ?`; `:audit-log-consumer-admin-console` on flag `:admin-console`, cron `30 */5 * * * ?`, 30s offset so the two drains never start on the same second), in 128-row chunks taken oldest-first (`ORDER BY created_at ASC, id ASC`) with `FOR UPDATE OF audit_log` (`SET LOCAL lock_timeout = '30s'`) inside the same transaction that sends and completes them, with 100 ms pauses between chunks. Batches arrive at Nexus in chronological order (it numbers by arrival for order-sensitive consumers). Nexus receives transit `{:events [...]}` with `x-shared-key: "nexus <key>"` (`:nexus-shared-key`, or derived from the instance secret) and `origin` = public-uri; flag on but no URI raises `:task-not-configured`. The admin console receives only its allowlisted event names (enriched with team names, props emails, and optional top-level `:profile-email` / `:profile-name` from the actor `:profile-id`); anything filtered out counts as done for the admin-console queue. A chunk whose destination refuses it (Nexus non-204 raises `:consumer-chunk-refused`) or throws (transport failure, admin-console error) rolls back and stays for the next tick, and the job aborts as failed: it archives what it can, and the cron retries the rest. Committed chunks stay sent. The send fns return counts or throw carrying the attempted count as `:events`; `run-single-consumer` records the `failed` metric from it, outside every chunk transaction (a local DB failure carries no `:events` and stays uncounted — it is not a destination failure). No `warn` before the throw: the runner reports the aborted job. Each job touches only its own queue, so a fallen Nexus never starves the admin console (nor vice versa).
|
||||
Nexus is the long-term store. Every 5 min the `:audit-log-archive` cron takes 128-row chunks of unarchived rows (`FOR UPDATE SKIP LOCKED`, ordered by `created_at`), POSTs them as transit `{:events [...]}` with `x-shared-key: "nexus <key>"` (`:nexus-shared-key`, or derived from the instance secret) and `origin` = public-uri, and marks `archived_at = now()` **only on HTTP 204**, in the same transaction. Anything else is retried next run; flag on but no URI raises `:task-not-configured`. It loops with 100 ms pauses until empty.
|
||||
|
||||
A chunk is **one POST** and the task advances only on 204, so a single oversized event wedges the whole queue: a crash report whose message embeds a whole payload (a serialized file, for example) can push the request past the proxy body limit and the rejection blocks every later event until it is pruned by hand. Frontend crash reports are therefore size-bounded at the source (`mem:frontend/routing-app-shell-subtleties`): `app.common.exceptions/get-hint` and the CLJS `format-throwable` clamp exception messages (`max-message-length`).
|
||||
|
||||
On the other side, Nexus checks `origin` against its allowlist (403 if not allowed), coerces the events, and inserts them into its own `audit_log` (partitioned by `received_at`, with an incrementing sequence number and `source`). Watch the rename: Penpot's `created-at` is stored as **`received-at`** in Nexus; Nexus's `created-at` is when it was created there. `tracked-at` keeps the origin time. Telemetry batches arriving at `telemetry.penpot.app` land in the same table through the telemetry endpoint above.
|
||||
|
||||
Concurrent jobs are safe: both lock rows in `created_at ASC, id ASC` order (one total order, deliberately blocking — no `SKIP LOCKED`, so archive arrival stays chronological), so the second waits for the first to commit and then sees whether it was last. Only acknowledged work deletes pendings; an admin-console failure never blocks the Nexus send of the same rows (each job owns its queue).
|
||||
|
||||
Observability (`backend/src/app/loggers/audit/consumer_task.clj`, definitions in `backend/src/app/main.clj:default-metrics`): one `dbg` summary per job run (`:consumer` label, sent/skipped/chunks, reaped disabled pendings, swept orphans, `:elapsed` as `"1.5s"` via `ct/format-duration`, frozen in a strict binding first because log props evaluate lazily in the logger thread), one `dbg` per admin-console send call (`ready` with totals, `sent`/`failed` with counts), plus `dbg` when the queue is empty or the job's send flag is off. Queue depth is Grafana's business: it queries the database directly, so the run performs no `GROUP BY` for the logs. No `warn` before a throw: the runner reports the aborted job (refusal details travel in the error data). Metrics, all recorded with bounded labels only: `penpot_audit_consumer_events_total{ destination = nexus|admin-console, outcome = archived|failed|skipped }` (`skipped` = filtered by the admin-console allowlist) and `penpot_audit_consumer_discarded_total{ reason = reaped|swept }` (entries discarded without archiving: `reaped` counts pendings dropped for disabled consumers, `swept` counts rows deleted while nobody waited for them — the silent R3 loss, now dashboard-visible instead of log-only). Timing needs no custom histogram: the jobs substrate already records execution time per job (`jobs-execution-timing`, split by job name), so each archive job is timed as itself. No queue-depth gauge: pending depth is read off the replica database directly. The bulk frontend ingest counts `penpot_audit_ingest_events_total{ kind = full|shadow, outcome = inserted|failed }` (both faces share one transaction, so a failure drops the batch on both). Archive success counts are recorded by `run-single-consumer` on the drain totals: a chunk that returns normally committed its transaction (a rollback always throws instead), so they cannot inflate. Logs never carry event data (`props`, `context`, `profile-id`, `ip-addr`, `team-name`, emails): only counts, statuses and the uri host.
|
||||
Concurrent runs are safe: rows are locked with `FOR UPDATE SKIP LOCKED` inside the same transaction that sends and marks them, so two archivers running at once each take different rows. Only rows Nexus acknowledges (HTTP 204) get marked; anything else is retried on the next run.
|
||||
|
||||
### How Nexus consumes the data (and forwards to PostHog)
|
||||
|
||||
@ -295,16 +291,20 @@ Stored rows are not the end: a consumer pipeline reads them back in sequence ord
|
||||
2. **`posthog`** (runs after `local`): forwards our own SaaS archived events to PostHog. It sends an allowlist of backend track events (teams, members, projects, files, comments, Nitrate/SSO, ...) plus frontend events minus a noise exclusion list, grouped by **team**, with person identify data from the local profile mirror and snake_case props. Both paths include the `initiator` property when the stored context has one, so PostHog can split Penpot vs admin-console traffic.
|
||||
3. **`posthog-telemetry`**: forwards events with `source = telemetry:frontend|telemetry:backend` to a separate PostHog project. Because these rows are already anonymous (no IPs, truncated times, filtered props), grouping is by **instance** (the self-hosted server, from `context.instance-id`), with one `$groupidentify` event per instance per chunk so PostHog learns per-instance properties (version, URI, flags). Each consumer has its own PostHog credentials.
|
||||
|
||||
So the full journey is: Penpot buffer → Nexus `audit_log` (via either archive cron or the telemetry endpoint) → consumers → PostHog (SaaS events grouped by team, telemetry grouped by instance) plus local analytics views and materialized KPIs.
|
||||
So the full journey is: Penpot buffer → Nexus `audit_log` (via archive cron or telemetry endpoint) → consumers → PostHog (SaaS events grouped by team, telemetry grouped by instance) plus local analytics views and materialized KPIs.
|
||||
|
||||
### Admin console: the second consumer
|
||||
### Nitrate: what exists and what is planned
|
||||
|
||||
Penpot → admin-console archiving lives in its own job (`:audit-log-consumer-admin-console`, consumer `2`, flag `:admin-console`, params closed empty): it drains its own pending queue and ingests allowlisted events through `nitrate/call :ingest-audit-log`, enriched as described above. What also exists (do not confuse with archiving):
|
||||
Today **only Nexus archives**. Archiving Penpot → Nitrate is in development and not on this branch.
|
||||
|
||||
- **Nitrate → Penpot**: `POST /api/management/methods/push-audit-events` (`backend/src/app/rpc/management/nitrate.clj`) lets the trusted Nitrate backend inject events into the Penpot audit log (no RPC auth, `::audit/skip`, request context + `submit`; only the `admin-console` shared-key id can call it). Section 7a is the main user.
|
||||
What already exists (do not confuse with archiving):
|
||||
|
||||
- **Nitrate → Penpot**: `POST /api/management/methods/push-audit-events` (`backend/src/app/rpc/management/nitrate.clj`) lets the trusted Nitrate backend inject events into the Penpot audit log (no RPC auth, `::audit/skip`, request context + `submit`). Section 7a is the main user.
|
||||
- **Admin-console browser → collector**: usage events from the admin-console UI traveling through the normal public `push-audit-events` (section 7b).
|
||||
- **Penpot frontend Nitrate bits**: membership/subscription events (`frontend/src/app/main/data/nitrate.cljs`, `nitrate-audit/*`) through the same public endpoint.
|
||||
|
||||
The future Penpot → Nitrate archive will follow the Nexus pattern (cron reading the buffer and POSTing batches), but there is no contract or flag on this branch yet. When it lands, document here: endpoint, auth, format, which `source` values travel, and how it shares the GC.
|
||||
|
||||
## 13. Flags and config
|
||||
|
||||
Flags (`common/src/app/common/flags.cljc`, varia; enabled with `PENPOT_FLAGS=enable-<name>`):
|
||||
@ -312,13 +312,13 @@ Flags (`common/src/app/common/flags.cljc`, varia; enabled with `PENPOT_FLAGS=ena
|
||||
| Flag | What it does |
|
||||
|---|---|
|
||||
| `:audit-log` | Stores full rows; turns on `wrap-audit` and frontend ingest |
|
||||
| `:nexus` | Send flag of the Nexus archive job `:audit-log-consumer-nexus` (cron `0 */5 * * * ?`; needs `:nexus-uri` (ingest path `api/audit` appended by the job)) |
|
||||
| `:admin-console` | Send flag of the admin-console archive job `:audit-log-consumer-admin-console` (cron `30 */5 * * * ?`, 30s after Nexus); also the module flag for the admin-console integration |
|
||||
| `:audit-log-archive` | Turns on the cron that sends to Nexus (needs `:audit-log-archive-uri`) |
|
||||
| `:audit-log-gc` | Turns on the cron that deletes archived rows |
|
||||
| `:audit-log-logger` | Structured `app.audit` log per event |
|
||||
| `:error-reporting` | Starts the error reporters (off by default); database reporter on flag alone, Mattermost on flag plus webhook URL |
|
||||
| `:telemetry` | Stores shadow rows and turns on shipping (auto with `:telemetry-enabled`) |
|
||||
|
||||
Config (`backend/src/app/config.clj`): `:nexus-uri`, `:telemetry-uri` (default `https://telemetry.penpot.app/`), `:nexus-shared-key` / instance secret, `:public-uri` (sent as `origin` to Nexus).
|
||||
Config (`backend/src/app/config.clj`): `:audit-log-archive-uri`, `:telemetry-uri` (default `https://telemetry.penpot.app/`), `:nexus-shared-key` / instance secret, `:public-uri` (sent as `origin` to Nexus).
|
||||
|
||||
## 14. Practical guide: add or change events
|
||||
|
||||
@ -337,6 +337,6 @@ Config (`backend/src/app/config.clj`): `:nexus-uri`, `:telemetry-uri` (default `
|
||||
- Penpot frontend producer: `frontend/src/app/main/data/event.cljs` (collector, buffer, send), `frontend/src/app/main/repo.cljs` + `frontend/src/app/util/http.cljs` (headers), `frontend/src/app/main/data/nitrate.cljs` (Nitrate membership events).
|
||||
- Admin-console producer (private repo, no paths cited here): server-side organization/subscription events through the management `push-audit-events`, browser-side UI events through the public `push-audit-events`.
|
||||
- Consumers: `backend/src/app/loggers/webhooks.clj`, `backend/src/app/loggers/database.clj`, `backend/src/app/loggers/mattermost.clj`, `backend/src/app/tasks/telemetry.clj`.
|
||||
- Archive (two jobs sharing one namespace + pending queue, no GC): `backend/src/app/loggers/audit/consumer_task.clj` (`:audit-log-consumer-nexus` / `:audit-log-consumer-admin-console` job-defs, shared drain/reap/sweep helpers), producer hook in `backend/src/app/loggers/audit.clj` (`append-audit-entry`, single `queue-pendings` helper) plus the bulk ingest in `backend/src/app/rpc/commands/audit.clj` (`handle-events`), metrics/cron wiring in `backend/src/app/main.clj` (one cron entry per job, Nexus at second 0 and admin console at second 30, each gated by its own send flag), single migration `0157` (queue, drop of `archived_at`, covering `(created_at, id)` index for the chronological scan; no backfill — pre-existing rows are swept unsent, an accepted loss).
|
||||
- Archive/GC: `backend/src/app/loggers/audit/archive_task.clj`, `backend/src/app/loggers/audit/gc_task.clj`, wiring and crons in `backend/src/app/main.clj`.
|
||||
- Nexus (private repo, no paths cited here): archive ingest, ordered consumers (local profile mirror, PostHog SaaS events, PostHog telemetry), telemetry endpoint, analytics views.
|
||||
- Tests: `backend_tests/rpc-audit-test.clj`, `rpc-team-test/accept-organization-invitation-audit-event`.
|
||||
|
||||
@ -11,14 +11,8 @@
|
||||
- Account lockout (flag `:account-lockout`, `app.auth.login-lockout`) is Redis-backed and keyed per profile id. Password and LDAP flows check/increment on the profile derived from the typed email and clear on the profile that actually logs in.
|
||||
- Lockout and RPC rate limits share the `app.http.errors/handle-error :rate-limit` HTTP path: status 429, body `{:type :rate-limit :code ... :hint ... :ttl ...}`, and any supplied `::http/headers` preserved. Account lockout raises `:code :account-locked` and, when it carries a non-nil `:ttl` (seconds), the handler adds `retry-after`. The RPC limiter (`app.rpc.rlimit`) raises `:code :request-blocked` and sets `retry-after` itself in `::http/headers` (seconds until the longest rejecting limit resets), alongside `x-rate-limit-remaining`/`x-rate-limit-reset`. CORS (`app.http.middleware/with-cors-headers`) exposes `content-type`, `retry-after`, and both `x-rate-limit-*` headers. External flags: `enable-account-lockout`, `enable-rpc-rlimit`.
|
||||
- Logout may return an OIDC provider redirect URI when the session claims include provider/session data and the provider has a logout URI.
|
||||
- Organization SSO (Nitrate): the RPC `check-nitrate-sso` takes the route's **query string proper** (`:query-string`, e.g. `screen=workspace&team-id=…`), never a URL and never with the `?` separator: the client strips it (`rt/token->query-string`) and the server puts it back. The server builds the destination with `app.auth.oidc/build-dest-url` (`public-uri` + `?` + the params, parsed and rendered back), so the redirect can never leave the public URI; a value whose parsed keys are not param names is not a route: a URL parses into a single key holding the whole URL, and a key carrying `:`, `/`, `?` or `#` is rejected with a `:warn` and the user lands on the public URI (the RPC never fails because of it). Do not reason from `u/map->query-string`: it keeps a valueless key as `key=`, so rendering is not what filters a URL out — the key shape is. A leading `?` is still trimmed server-side, so a client bundled before the trim keeps its destination. Never require `screen`: the root of the app carries its params instead of a screen (`?template=…`), and `check-nitrate-sso` is called from that branch too (`dtm/resolve-login-team-id`, which today reads only `:authorized` and never follows the redirect, so a wrong destination there is a silent `:warn` and not a visible break); the callback re-checks the destination before redirecting, which covers state tokens minted before that. The state token also carries `:profile-id`: only that session is granted the `props :sso` entry (`session-mismatch` otherwise, and the redirect still happens).
|
||||
- Organization SSO needs `app.nitrate/organization-access?` whenever the RPC receives an `:organization-id` (that id always comes from the client): the profile must be a member of the organization or its owner; otherwise the RPC answers `{:authorized true :reason :no-organization-access}`, the same short-circuit shape as `:no-team-access`.
|
||||
- Invitation tokens are verified through token issuers and only accepted when the token member id/email matches the authenticated profile; otherwise login proceeds without consuming the invitation.
|
||||
- When public registration is disabled, invitation-based password registration also requires a matching, non-expired `team_invitation` row; registration revalidates it under a row lock before creating the profile. SSO registration (OIDC and compatible providers) follows the OIDC callback gate: it requires `:registration` or `:oidc-registration`, or a team invitation token issued for the registering email (an invitation substitutes the flags only when `member-email` matches the OIDC/`prepared-register` email via `clean-email`, as in the password flow, which raises `:email-does-not-match-invitation` otherwise). The OIDC callback verifies the invitation JWT and the email before redirecting to register, and `register-profile` revalidates the invitation under a row lock when no SSO flag is set; it never requires password login. Only OIDC-typed backends (`oidc-provider-types` in `app.auth.oidc`) take the SSO registration gate; any other `:backend` in a prepared-register token is rejected as `:invalid-token`.
|
||||
- Access tokens (`app.http.access-token`): personal tokens have JWT issuer `access-token` and DB type other than `mcp`; MCP tokens have issuer `urn:penpot:mcp-token` (`actoken/mcp-token-issuer`) and DB type `mcp`. Authentication requires issuer and type to agree, and the claim `uid` to match the row's profile; other issuers and undecryptable tokens are treated as anonymous.
|
||||
- Management RPC (`/api/management/methods/:name`) is shared-key only. The middleware authenticates `<id> <secret>`; the RPC wrapper `wrap-shared-key-authorization` authorizes the key id against the method's `::rpc/allowed-shared-keys` set. Absent metadata accepts any configured key; all current methods declare one (`exporter`/`media-processor` for `upload-tempfile`, `nexus` for file analysis, `exporter` for jobs, `admin-console` for Nitrate). A valid key outside the set answers 403 `:shared-key-not-allowed`; denied calls do not reach the audit wrapper.
|
||||
- Personal tokens need flag `:access-tokens`. MCP tokens need `:mcp` or `:access-tokens`, and only on routes whose `actoken/authz` gets `::actoken/allow-mcp true` (the RPC routes); elsewhere (e.g. assets) they are anonymous.
|
||||
- On RPC, an MCP token reaches only commands with `::rpc/token-type "mcp"` (`wrap-authentication` in `app.rpc`); every other command answers 403 `:token-auth-required`. A command with that metadata also rejects sessions and personal tokens. Currently only `send-mcp-feedback`. Blocking a profile invalidates sessions, not tokens, so such commands check `:is-blocked`/`:is-active` themselves.
|
||||
- When public registration is disabled, invitation-based password registration also requires a matching, non-expired `team_invitation` row; registration revalidates it under a row lock before creating the profile.
|
||||
- HTTP/session parsing details such as cookie/header precedence, JWT session token versions, and SameSite behavior are in `mem:backend/subtleties`.
|
||||
|
||||
## Permission model
|
||||
@ -26,16 +20,13 @@
|
||||
- `app.rpc.permissions` provides predicate/check factories. Failed permission checks intentionally raise `:not-found` / `:object-not-found`, not an authorization-specific error, to avoid leaking object existence.
|
||||
- Team role flags are normalized as owner > admin > editor > viewer. Owner/admin imply edit; any membership row implies read.
|
||||
- File/project role rows only count with a live team membership: the file/project permission queries require a matching `team_profile_rel` row, and leaving or removing a member deletes their team-scoped `file`/`project`/`team-project-profile-rel` rows (GHSA-v9r9-h77c-55m2).
|
||||
- The organization-owner fallback (`get-organization-owner-permissions` via Nitrate over HTTP) grants read-only access for a non-member organization owner. It resolves the owning team through `resolve-team-id`, which refuses soft-deleted file/project/team: the file and project lookups filter `deleted_at`, and a final `db/get*` check covers the team row itself — `nil` means no access. `db/get*` only filters soft-deleted rows when `deleted_at` is in the projection. The re-checks of open websocket subscriptions rely on this (`mem:backend/websocket-subscriptions`).
|
||||
- File/project/comment checks are implemented in the owning command namespaces, often via helpers imported from `files`, `teams`, or `projects`; do not bypass those helpers with direct DB lookups unless preserving their not-found semantics.
|
||||
- Comment permission includes both logged-in state and the file/team comment policy. Shared viewer paths may pass `share-id`; preserve that path when changing comment queries. RPCs that accept `share-id` enforce the link's page scope before mutating (`share-link-pages` + `check-page-scope`): a thread or page outside the link raises `:not-found`, and that check runs before the ownership checks. `get-profiles-for-file-comments` filters its query by the same pages. `mark-all-threads-as-read` does not take `share-id`.
|
||||
- Comment permission includes both logged-in state and the file/team comment policy. Shared viewer paths may pass `share-id`; preserve that path when changing comment queries.
|
||||
|
||||
## Teams, projects, and invitations
|
||||
|
||||
- Team/project commands mix DB changes, email, message bus notifications, media/storage cleanup, feature flags, quotas, and audit metadata. Keep mutations transactional when the existing command does so.
|
||||
- Invitation flows validate muted/bounced emails before sending and use tokenized invitation state. Accepting an invitation is tied to the invited member identity, not just possession of a token.
|
||||
- The whole invitation family is team-admin only: creating, updating and deleting invitations, minting the invitation token (`get-team-invitation-token`) and listing the pending ones (`get-team-invitations`, which carries the invited email and the offered role). The token of an invitation that grants the `owner` role is restricted to the team owner, because an admin cannot create nor promote an owner invitation (`cant-promote-to-owner`). This family is the exception to the rule above: it answers `:validation` / `:insufficient-permissions` instead of `:not-found`, so the five commands speak one language.
|
||||
- With the `:admin-console` flag, the organization `send-invitations` policy (role value plus Nitrate `restrictions-met`, through `cto/allowed?`) gates every invitation command that hands out or manages access: create (and resend, which reuses it), `update-team-invitation-role`, `delete-team-invitation` and `get-team-invitation-token`. Only `get-team-invitations` stays on team admin alone, so a blocked admin still sees the list. The shared helper `check-send-invitations-policy` in `teams_invitations.clj` does a strict Nitrate lookup (fails closed; 404 means no organization) and runs outside a DB transaction, which is why the role and delete commands carry no `::db/transaction`.
|
||||
- Logical deletion is used for many product objects; prefer existing logical-deletion helpers over hard deletes unless the command already performs permanent cleanup.
|
||||
- Bounced/spam-complaint emails can mute/block a profile for login/registration and email sending. Devenv Mailpit is the normal local path for registration/email-flow testing.
|
||||
|
||||
|
||||
@ -9,7 +9,6 @@ Backend: JVM Clojure; Integrant; PostgreSQL; Redis/Valkey; RPC; HTTP; storage; m
|
||||
- Embedded Ladybug graph experiment, projection, incremental sync, console, and risks: `mem:backend/graph-experiment`
|
||||
- Session lifetime config, token `:exp`, and idle/absolute GC: `mem:backend/session-expiration`.
|
||||
- Auth flows, permission model, teams, projects, invitations, comments, webhooks, audit: `mem:backend/auth-permissions-product-domains`
|
||||
- WebSocket subscriptions, the connection registry (local per backend, unlike the bus), and how a lost permission gets revoked: `mem:backend/websocket-subscriptions`.
|
||||
- Audit-log event collection (RPC wrapper, frontend ingestion), telemetry duality, webhook fan-out, error reporters, Nexus archival and retention: `mem:audit-log` (guía completa, también para perfiles junior; el stub `mem:backend/audit-log` solo redirige aquí)
|
||||
- Services, task-queue/Pub-Sub topology constraints -> `mem:prod-infra/core`.
|
||||
|
||||
@ -33,65 +32,29 @@ Backend: JVM Clojure; Integrant; PostgreSQL; Redis/Valkey; RPC; HTTP; storage; m
|
||||
|
||||
RPC commands are defined with `app.util.services/defmethod` and schemas. Use `get-` prefixes for read operations. Command metadata usually includes auth, docs version, params schema, and result schema. Return plain maps/vectors or raise structured exceptions from `app.common.exceptions`.
|
||||
|
||||
Backend RPC command areas without focused memories include access tokens (auth rules, incl. MCP tokens, in `mem:backend/auth-permissions-product-domains`), binfile, demo, feedback, file snapshots, fonts, management, Nitrate, and webhooks beyond the notes in `mem:backend/auth-permissions-product-domains`; inspect nearby command tests and command metadata before changing them.
|
||||
Backend RPC command areas without focused memories include access tokens, binfile, demo, feedback, file snapshots, fonts, management, Nitrate, and webhooks beyond the notes in `mem:backend/auth-permissions-product-domains`; inspect nearby command tests and command metadata before changing them.
|
||||
|
||||
## DB conventions
|
||||
|
||||
`app.db` helpers accept cfg, pool, or conn in most places and convert kebab-case to snake_case:
|
||||
- `db/get`, `db/get*`, `db/query`, `db/insert!`, `db/update!`, `db/delete!`.
|
||||
- `db/get*`/`db/query` signature is `(ds table where-params & {:as opts})`: trailing keywords become an opts MAP (ignored by the select builder — use `{::db/remove-deleted false}` map as the single vararg to also see rows with `deleted_at` set), while a trailing ODD number of keywords crashes with "Don't know how to create ISeq from: Keyword". Extra keywords are NOT a column filter: rows always come back with all columns.
|
||||
- Next.js/JDBC caveat: where-maps don't support value vectors (no `IN`); use `status = ANY(?)` with `db/create-array` inside `db/tx-run!`.
|
||||
- Use `db/run!` for multiple operations on one connection.
|
||||
- Use `db/tx-run!` for transactions.
|
||||
- `job`-substrate specifics (unified jobs): the only writers of a job status are `claim`, `retry-job`, `complete`, `fail` and `cancel` in `app.jobs`; each one is a conditional update plus its `job_event` row in the same transaction, so history can never contradict the row. A writer that affects no row writes no event and no metric. They never join the caller's transaction, because a lifecycle write reports on work that already happened: a rolled back caller cannot undo a transition, and an independent connection cannot see the caller's uncommitted job row, so a job must be committed before it can be claimed. GC deletes return resource ids via `RETURNING` inside one transaction.
|
||||
- Every public function of `app.jobs` takes a cfg map. A bare pool or connection is a caller bug and is not coerced: the substrate has no conversion of its own, and the ones in `app.db` are the db layer's business.
|
||||
- Every new foreign key of the substrate is `DEFERRABLE`, `job_event.job_id` included.
|
||||
- Some tables carry a SQL deletion guard (`raise_deletion_protection` + a `deletion_protection__tgr` statement trigger): `profile`, `team`, `team_font_variant`, `file_media_object`, `file_thumbnail`, `file_tagged_object_thumbnail` and `job`. A `DELETE` raises unless the transaction runs `SET LOCAL rules.deletion_protection TO off` (the guard is on when the GUC is unset, empty or `on`); see `app.tasks.objects-gc`, `app.jobs.gc` and `th/db-force-delete`.
|
||||
|
||||
Database migrations live in `backend/src/app/migrations/`; pure SQL migrations are under `backend/src/app/migrations/sql/`. SQL filenames conventionally start with a sequence and verb/table description, e.g. `0026-mod-profile-table-add-is-active-field`. Applied migrations are tracked in the `migrations` table.
|
||||
|
||||
Migrations must stay within PostgreSQL 15 syntax: the official docker images (`docker/images/docker-compose.yaml`) pin `postgres:15`, so anything newer breaks those deployments (`SET STORAGE DEFAULT` is PostgreSQL 16, for instance). The documented floor is >=15.
|
||||
|
||||
For interactive PostgreSQL access with correct dev defaults, use `scripts/psql`; to dump the current DDL schema, use `scripts/db-schema` (see `mem:scripts/psql`).
|
||||
For interactive PostgreSQL access with correct dev defaults, use `scripts/psql`; to dump
|
||||
the current DDL schema, use `scripts/db-schema` (see `mem:scripts/psql`).
|
||||
|
||||
For deeper details on transaction semantics, advisory locks, Transit vs JSON helpers, and dev/test DB URLs: `mem:backend/subtleties`.
|
||||
|
||||
## Background tasks (unified jobs)
|
||||
## Background tasks
|
||||
|
||||
Every background job is a job-def: a plain `(defn execute-X [cfg params] ...)` in its namespace + a malli params schema + an `ig/init-key` that returns the job-def map `{::jobs/name, ::jobs/schema, ::jobs/handler, ::jobs/decoder, ::jobs/validator}` (decoder/validator precompiled at init), optionally with `::jobs/result-schema` when the result carries types plain JSON does not keep (a uuid, whether a value or a map key): the substrate compiles the decoder from that schema on registry init and checks a result against the schema before storing it. The registry is the `::jobs/defs` wiring in `app.main`, which also populates a module-level registry used by `jobs/submit` as fallback — job-def components cannot ig/ref `::jobs/defs` (wiring cycle). Submit from RPC code passes its RPC cfg (it carries `::jobs/defs` via ig/ref).
|
||||
|
||||
- `::jobs/handler` is `(fn [context params] ...)`. It is NOT given the runner cfg: a job-def closes over its own dependencies, and handing it the runner cfg would let a handler reach the pool or any other component. A wrapper passes the context to its implementation only when the implementation takes it; most do not, and their implementation stays `[cfg params]`.
|
||||
- The context is built by `jobs/make-context` and has exactly five keys: `id`, `name`, `label`, `resource-id`, `profile-id`, behind a closed schema. `profile-id` is the owner of a user-facing job and nil for an internal one, and it is there so a handler can revalidate permissions and check who owns its resource without a second query. `queue` is absent on purpose (a routing detail of the dispatcher, not a property of the job) and so are `retry-num`/`max-retries`/`attempt` (retry policy belongs to the runner, and an attempt number cannot be durable because the `noop` strategy re-runs without incrementing the counter).
|
||||
- `jobs/invoke` is the in-process escape hatch: it delivers a nil context without a row, validates and delivers `::jobs/context` when given, and keeps `::jobs/job-id` independent. `::jobs/context` and `::jobs/job-id` are cfg options, not params.
|
||||
- Business params live in the `params` column (plain JSON, decoded with the job-def decoder). `submit` takes optional `::jobs/profile-id`, `::jobs/resource-id` (both validated as UUIDs) and `::jobs/expires-at` (an instant: the retention the jobs GC sweeps by), all stored as columns; nothing is inferred from `params`. The binfile creation commands are the callers that pass a profile-id and a resource.
|
||||
- A job-def may carry optional metadata keys read from the registry by its consumers, never by the substrate: `::jobs/family` (e.g. `:export`) groups the jobs of a family for quotes, `::jobs/resource-role` (`:input` or `:output`) says whether the job's `resource_id` is what it consumes or what it produces, and `::jobs/queue-name` names the queue the creation command routes the job to (`submit-job` falls back to `:binfile` when the def does not set it).
|
||||
- Job resources (`app.jobs.storage`) are the objects a job owns through `job.resource_id`: they live in the `job-resource` bucket, are written with the profile of the job as owner (only that profile can read them back from `/assets/by-id`) and are never deduplicated. They carry no expiry of their own, because `expired-at` is stored as `deleted_at` and an object with it is unreadable from birth: the retention of the artifact is the retention of its job row. The writer marks the object as touched right away, so an artifact that no row references (a crash between the upload and the completion) is reclaimed by `storage-gc-touched`, and a row that does reference it freezes it until the jobs GC deletes the row and touches the object.
|
||||
- The binfile jobs (`:export-binfile` and `:import-binfile`) run the core of `bf.v1`/`bf.v3` through `app.binfile.jobs`, which turns the `:progress` taps of the core into `progress` events of the job with `jobs/heartbeat` (the `modified_at` touch at most once per second, progress events at up to ~200ms). Each event is a self-contained milestone: a `:stage` keyword of one vocabulary shared by the two formats (the same work has a different name in v1 and v3) and an optional `:counters` map, with the counter of the activity named after its own stage and the outer scopes in course as context, which the tap carries itself (the reader puts `::outer-counters` in every tap inside them — today only `:files`, extendable to `:teams`/`:projects` for multi-scope exports — so the adapter remembers nothing between taps and a tap that happens outside any scope has no context); the ids and names the taps carry never reach the event. Every tap is reported, so a tap with no units to count still keeps the job alive, and a milestone the throttle drops loses nothing because the next one carries the same state. The adapter checks that the job is still active before the run, and the beat raises an `:interrupt` right there on the run thread when the job is no longer active, so the work stops at its next event point inside the transaction of its handler, which rolls back. A run whose last tap predates the interruption is discarded by the check once it returns. The adapter passes the job id from the handler context. Their job-def components carry pool, storage, metrics and msgbus, because a job with a `profile_id` cannot store an event without a msgbus on its cfg. Both readers answer with the same shape (`:file-ids` and, in version 3, the resolution of the libraries), because the version 1 reader returns a bare set of file ids.
|
||||
- Both binfile handlers revalidate their permission at run time, not only when the job is created, because it can be revoked while the job waits: the export checks the read permission of every file before reading anything and stores the package in `job-resource` as the resource of the job (answering with the completion envelope), and the import checks the edition permission of the destination and runs the core inside a transaction, which is what makes a cancelled import roll back instead of leaving half of a file. The import releases its package (marks it as deleted and drops the reference from the job row in the same transaction) and deletes its temporary copy once the job is terminal, whatever the outcome. Both handlers check their own result against a schema before returning it (the import job-def declares that same schema as its result schema, which is what types the resolution when the row is read).
|
||||
- The v3 and v1 imports only materialize a library relation when both ids belong to the package and the link would neither be a self link nor close a dependency cycle (the same `bfc/library-link-status` check that `link-file-to-library` uses to raise); relations are processed in sorted `(file_id, library_file_id)` order, which `get-files-rels` also returns, so the first edge wins deterministically, and every edge it ignores is logged with a warning. The v2 import stays out of that guard on purpose: it is only reachable from the developer REPL (`app.srepl.binfile`) and keeps inserting the relations as they come. Migration `0158-del-forbidden-file-library-rels` deletes the legacy cross-team and self rows from `file_library_rel` and their `file_library_sync` rows, sync rows first.
|
||||
- A version 1 package lists its files in the metadata and the importer creates one file per entry. The v1 export (gone from the product; only tests and the developer REPL use it) unioned the requested files with their libraries without deduplicating, so a package exported from a file inside a library cycle lists that file twice. The importer reads the pair of objects of every entry to keep the stream in sync, imports the first occurrence only, warns about the repeats, and answers with one id per imported file.
|
||||
- The domain cfg of a binfile request lives in `app.binfile.common` (`export-cfg`, `import-cfg`), so the legacy RPC and the jobs freeze the same ids, resolve the same team features and apply the same anti-zip-bomb limits. `resolve-export-type` is the translation the legacy RPC needs for its deprecated boolean flags; the creation command asks the caller for the type instead of guessing a default.
|
||||
- Two retention knobs, easy to confuse: `:jobs-retention` sweeps terminal internal rows (the ones without a profile), while `:jobs-user-ttl` is the default expiry a user-facing job gets when it is created; the jobs GC deletes the row once `expires_at` passes and touches its resource. The heavy work of a user job runs on its own `:binfile` queue (`:worker-binfile-parallelism`, default 1) so it does not block the default runner.
|
||||
- The quote of a family of jobs (`::export-jobs-per-profile`, `::import-jobs-per-profile`) counts the jobs of the profile that are still on their way (`new`/`scheduled`/`running`/`retry`) whose name the registry gives for that family, so a new job of the family is covered without touching the quote.
|
||||
- A user-facing job is created through `app.rpc.commands.jobs`. One command per job type: `create-export-binfile-job` and `create-import-binfile-job` name the job they create and their body carries only `:params`, whose schema the command imports from the job namespace (`::sm/params`), so the RPC validates the business params up front, before the command runs; the job-def is still the place that owns them and validates them again at run time. The commands check the permission and the quote of the family before the expensive part, submit with `max-retries 0`, an expiry from `:jobs-user-ttl` and the queue the job-def routes to through `::jobs/queue-name` (`:binfile` when the def says nothing; read by the command, never by the substrate), and answer with a summary of the row. The import command has its own creation schema (`import-binfile/schema:create-params`, version optional) because the request is not the stored params: the command fills the version (from the caller or the package header) and the manifest metadata kept for audit. It assembles the upload, reads the manifest of a version 3 package for the audit (never its content) and stores the package in `job-resource` as the resource of the job, releasing it when the job never comes to own it.
|
||||
- `get-job` is the read surface a profile has over its own jobs: it answers with the same summary the creation commands return plus the `result` of a finished job or its public error (through `decode-job-error`, so the keywords come back), and a job of another profile is not distinguishable from one that does not exist: both answer `not-found`. `decode-row` decodes `params`, `result` and `error`, and the result is then decoded with the decoder the substrate compiled from the result schema the job-def the row names declares (`decode-result`; a def without one, or a row whose job-def is gone, keeps the plain JSON the row holds: a uuid as the text it is on disk, a uuid used as a map key as a keyword). The event log still has no read surface: a client that follows a job takes its progress from the websocket.
|
||||
- `heartbeat` uses named options `[cfg & {:keys [job-id progress] :as options}]` and returns the number of durable writes, so 0 means there was nothing to write: no job context, or the throttle did not allow it. A write that finds the job no longer active (terminal or gone) raises an `:interrupt` (`code :job-interrupted`, carrying the job id and the status it found) through the shared `jobs/check-active` instead of counting zero, so a task in course stops at its next beat; the interrupt is internal and never reaches HTTP (the management progress report answers `:skip` for it). The job id falls back to `::jobs/job-id` on the cfg and then to the runner-bound `*job-id*`. The `modified_at` touch is throttled to ~1s and the progress event to ~200ms, so the check fires at most once per second with no extra query (the read of the row happens only when a write already found nothing). A progress report is a milestone (`schema:progress`): a required `:stage` keyword plus an optional `:counters` map of `scope -> {:current, :total?}`, and no other key. Counters are non-negative, a `total` is positive and never lower than its `current`, and stage and counter keys are names of at most 64 characters. A milestone without counters is valid (a point of the run with no units to count), the stage vocabulary belongs to the worker and not to the substrate, and a client must degrade for a stage it does not know. Read a stored payload back with `jobs/decode-progress`: the database holds the keywords as strings. The 1s beat (down from 60s, so a cancelled heavy job is seen as gone at its next beat) costs one `modified_at` touch per beating job per second: cheap while the only heavy beater runs alone on its queue, revisit if concurrent beating jobs ever grow.
|
||||
- `job.error` conforms to `app.jobs/schema:job-error` (`type` and `code` keywords, `hint` text, extra details allowed). Read it back with `jobs/decode-job-error`; a plain `db/decode-json-pgobject` leaves the keywords as strings. `complete` takes named options and an optional `resource-id`, which is only ever set, never replaced: it reads the row locked inside its own transaction and applies the rules in a fixed order, skipping a job that is no longer completable before refusing one for its resource, so the answer is a function of the row and not of the order two callers arrive in. It checks the result against the schema the job-def of the row declares before it writes (a violation raises `:invalid-result` and nothing is stored), and only for a transition that is going to write: a terminal job answers `:skip` without validating. `complete` is the single door of a result, so the check covers the runner and the external worker that completes a job through the management RPC (`::complete-job`), which stores a result the substrate would otherwise take blind.
|
||||
|
||||
For worker dispatch, cron, retry semantics (`ex/raise :type ::wrk/retry` with `:delay`/`:strategy`), deduplication, job events, and queue internals: `mem:backend/subtleties`.
|
||||
|
||||
## Jobs metrics
|
||||
|
||||
- `app.jobs.metrics` owns the jobs metric names, bounded labels and the periodic backlog sampler.
|
||||
- Metrics use only `name`, `queue`, `outcome`, `reason`, `stage` and `kind` labels. Job IDs, profile IDs, params, reply bodies and exception text must never be labels. Every label goes as-is (lowercased where case varies): values come from fixed call sites, so a new value in Prometheus means new code, not noise. Only a missing value falls back (`other`, or `failed` for outcomes).
|
||||
- The canonical job name is a keyword (`app.jobs/schema:job-name`): `submit` and `invoke` reject strings. The row stores text, so `jobs/get-job-def` still coerces (it is the runner's boundary), cron entries carry the job under `:job` with `d/name` only at the SQL/log boundary, and test helpers keyword-ize row names before invoking.
|
||||
- The durable lifecycle metrics are submitted, dispatched, completed, retries, orphaned and rescheduled counters; queue-wait, execution and total-time histograms; dispatcher and backlog state; GC, cron and ephemeral request metrics.
|
||||
- `app.metrics/run!` is safe for recording failures, but metrics components still require a valid metrics instance. Durable submit call sites carry `::mtx/metrics` through RPC or job-def configuration. The jobs substrate takes the **cfg**, not the instance, and resolves it with `app.metrics/instance`, which raises `:missing-metrics` when absent: `app.jobs`, `app.jobs.metrics`, the runner, the dispatcher, cron and the jobs GC all follow that. `app.http`, `app.rpc.climit` and `app.storage.s3.metrics` still take the instance and were left out of scope on purpose, so do not read the codebase as uniform. The six job writers resolve at the top of their body, before writing, because their metric lands in a `db/after-commit` callback that swallows exceptions. The GC, the dispatcher and cron resolve inside that callback instead, so their guarantee rests on the component schema instead. A job that calls `heartbeat` from a component cfg must carry `::mtx/metrics` on it, and the event helper records the counter from the cfg the transaction hands it.
|
||||
- SQL-backed job metrics are registered with `app.db/after-commit` from INSIDE the transaction that changed the row, so a rollback cannot inflate submitted, terminal, retry, event or dispatcher counters. Every lifecycle writer owns that transaction (`app.jobs/own-transaction` drops `::db/conn`), and `db/transact!` gives a transaction that joins no other its own callback context, so the write and its metric commit or vanish together. `submit` is the exception and joins the caller on purpose: creating a job is part of the caller's business work.
|
||||
- `penpot_jobs_events_total` counts stored job events with no labels: the interesting split is `kind`, and grouping a growing append-only table to build a metric is not worth the query on the write path.
|
||||
- The backlog sampler is `::jobs-metrics/sampler`; it runs every 30 seconds on worker-enabled, writable systems, groups by status and publishes the oldest pending age, and does not start on a read-only database.
|
||||
- `penpot_tasks_timing` remains exported for compatibility while the jobs-specific histograms are adopted.
|
||||
A task handler is an Integrant component with `ig/assert-key`, `ig/expand-key`, and `ig/init-key`, returning the function run by the worker. New tasks also need wiring in `app.main`: handler config, worker registry entry, and cron entry if scheduled.
|
||||
|
||||
For worker dispatch, cron, retry semantics, deduplication, and queue internals: `mem:backend/subtleties`.
|
||||
|
||||
## REPL
|
||||
|
||||
@ -138,7 +101,9 @@ IMPORTANT: all CLI commands must be executed from the `backend/` subdirectory.
|
||||
* **Linting:** `pnpm run lint:clj`.
|
||||
* **Formatting:** `pnpm run check-fmt:clj` to check, `pnpm run fmt:clj` to fix. After running `fmt:clj`, `check-fmt:clj` is redundant. Avoid unrelated whitespace diffs.
|
||||
|
||||
**Before linting:** if delimiter errors are suspected (after LLM edits), run `scripts/paren-repair` on the affected files first. Delimiter errors produce misleading linter/compiler output. See `mem:scripts/paren-repair`.
|
||||
**Before linting:** if delimiter errors are suspected (after LLM edits), run
|
||||
`scripts/paren-repair` on the affected files first. Delimiter errors produce
|
||||
misleading linter/compiler output. See `mem:scripts/paren-repair`.
|
||||
|
||||
## Testing
|
||||
|
||||
|
||||
@ -19,12 +19,6 @@
|
||||
- Renewal fires when `modified-at` is older than 6h (`default-renewal-max-age`, not configurable). It `UPDATE`s the same row (`modified-at` only) and issues a new token that keeps the original `:exp`.
|
||||
- `read-session` does not check age. Idle expiration is enforced on every request by `wrap-authz` (sessions past the idle window authenticate like missing ones and are never renewed); the GC only deletes the rows afterwards.
|
||||
|
||||
## Revocation paths
|
||||
|
||||
- Profile deletion → `session/invalidate-all` (all sessions of the profile die).
|
||||
- Authenticated password change → `session/invalidate-others` (every session except the caller's dies).
|
||||
- Password recovery (`recover-profile`, GHSA-3j2p-qr45-q365) → `session/invalidate-all` inside the same transaction as the password `UPDATE`; there is no current session to preserve, and a captured session must not survive recovery. Sibling of GHSA-mj9f-5cwq-7p3q (logout deletes the current session row).
|
||||
|
||||
## GC (`::tasks/gc`, cron `session-gc`, daily)
|
||||
|
||||
- Deletes from `http_session_v2` where `modified_at < now - :auth-token-cookie-max-age` **or** `created_at < now - :auth-token-cookie-max-age-absolute`.
|
||||
|
||||
@ -25,19 +25,12 @@
|
||||
- Existing object values can remain readable until physical deletion.
|
||||
- `:expired-at` blocks reads after the expiration time.
|
||||
- `del-object!` sets `deleted_at` on live rows only (`deleted_at IS NULL`): a repeated call returns `false`. It does not remove backend content.
|
||||
- `storage-gc-deleted` removes the database row and backend content after the deletion delay, processing the pending objects in chunks of 25 with keyset pagination (`(deleted_at, id)` cursor, matching the `storage_object(deleted_at, id)` index). Only `status = 'valid'` objects are read: pending ones (still being written, see the status lifecycle) stay untouched even when already marked as deleted. Each chunk is one short read-write transaction: `SELECT ... FOR UPDATE SKIP LOCKED` (so two concurrent runs never take the same chunk; no separate read transaction exists and the run does not block the table vacuum) followed by the database delete and the backend delete grouped by backend. The cursor advances to the `(deleted_at, id)` of the last row read regardless of the deletion result, so a chunk that fails is skipped and not visited again in the same run: the error aborts the transaction (nothing of the chunk is deleted, locks are released), the handler catches it inside the transaction function and rolls it back explicitly — the commit that `tx-run!` performs afterwards is a JDBC-driver no-op on the already-rolled-back connection — and the loop continues with the returned rows. Accepted quirks, both benign and self-healing: (1) with rows of several backends in one chunk, a failure after one group's physical delete leaves those rows pending with already-removed content — pending rows are excluded from reads and the next run finishes them, since the physical delete is idempotent (S3 ignores missing keys, FS uses `deleteIfExists`); (2) rows locked by a concurrent run are skipped via `SKIP LOCKED` and deferred to the next scheduled run, and the `deleted_at <= now` cutoff is sampled per chunk. It emits the `penpot_storage_gc_deleted_objects_total{backend}` counter (one increment per deleted backend group, reported only after the chunk commits, so rolled-back chunks are never counted twice); run duration is covered by the generic `penpot_tasks_timing{name="storage-gc-deleted"}` histogram and chunk failures by the log-derived `penpot_log_events_total`. Every 5000 deleted objects the run pauses 5 seconds (throttle between chunk transactions, never holding one open). It runs as the `storage-gc-deleted` job (`::job-def`) and beats `jobs/heartbeat` after each committed chunk, so a long sweep keeps its lease and never holds it during a chunk transaction.
|
||||
- `storage-gc-deleted` removes the database row and backend content after the deletion delay, processing the pending objects in chunks of 25 with keyset pagination (`(deleted_at, id)` cursor, matching the `storage_object(deleted_at, id)` index). Only `status = 'valid'` objects are read: pending ones (still being written, see the status lifecycle) stay untouched even when already marked as deleted. Each chunk is one short read-write transaction: `SELECT ... FOR UPDATE SKIP LOCKED` (so two concurrent runs never take the same chunk; no separate read transaction exists and the run does not block the table vacuum) followed by the database delete and the backend delete grouped by backend. The cursor advances to the `(deleted_at, id)` of the last row read regardless of the deletion result, so a chunk that fails is skipped and not visited again in the same run: the error aborts the transaction (nothing of the chunk is deleted, locks are released), the handler catches it inside the transaction function and rolls it back explicitly — the commit that `tx-run!` performs afterwards is a JDBC-driver no-op on the already-rolled-back connection — and the loop continues with the returned rows. Accepted quirks, both benign and self-healing: (1) with rows of several backends in one chunk, a failure after one group's physical delete leaves those rows pending with already-removed content — pending rows are excluded from reads and the next run finishes them, since the physical delete is idempotent (S3 ignores missing keys, FS uses `deleteIfExists`); (2) rows locked by a concurrent run are skipped via `SKIP LOCKED` and deferred to the next scheduled run, and the `deleted_at <= now` cutoff is sampled per chunk. It emits the `penpot_storage_gc_deleted_objects_total{backend}` counter (one increment per deleted backend group, reported only after the chunk commits, so rolled-back chunks are never counted twice); run duration is covered by the generic `penpot_tasks_timing{name="storage-gc-deleted"}` histogram and chunk failures by the log-derived `penpot_log_events_total`. Every 5000 deleted objects the run pauses 5 seconds (throttle between chunk transactions, never holding one open).
|
||||
- `storage-gc-touched` finds references before it sets `deleted_at`.
|
||||
- Orphan watch: `SELECT COALESCE(metadata->>'bucket', metadata->>'~:bucket') AS bucket, count(*) FROM storage_object so WHERE touched_at IS NULL AND deleted_at IS NULL AND status = 'valid' AND NOT EXISTS (SELECT 1 FROM upload_session_chunk usc WHERE usc.object_id = so.id) GROUP BY 1;` Only the `upload-session` row of that output is a leak signal (its rows always have a mapping or a pending touch/deleted mark); other buckets use domain-table references, not this mapping.
|
||||
- `objects-gc` removes deleted domain rows and touches their storage object IDs. Each proc streams its pending keys with `db/plan` inside a read-only transaction and deletes them in chunks, each chunk in its own transaction with `FOR UPDATE SKIP LOCKED`: a chunk that fails is logged and skipped instead of aborting the run.
|
||||
- Use `::db/reuse-conn true` with `sto/resolve` inside a database transaction.
|
||||
|
||||
## GC Cost Model
|
||||
|
||||
- `sto/touch-object!` is one `UPDATE storage_object SET touched_at`: DB-only, never touches S3/FS backend content.
|
||||
- Physical FK cascades are avoided by convention; domain deletion is logical (`deleted_at`) everywhere. GC removes small objects first, then parents, then grandparents (see the call order in `app.tasks.objects-gc/execute-objects-gc`). Per-row delete cost is small and bounded.
|
||||
- `objects-gc` sweeps in chunks of 100 (`::chunk-size`, overridable per job params); `storage-gc-deleted` in chunks of 25. `objects-gc` (and the other `db/plan` sweeps) stream the pending keys in a read-only transaction of their own and delete every chunk in a transaction of its own: the cursor fetch and the deletes are never the same transaction. `storage-gc-deleted` has no separate cursor transaction: it reads, locks and deletes each chunk in the same short transaction (see the keyset pagination above).
|
||||
- Consequence: per-chunk cost is DB-only and bounded, so these sweeps cannot outrun the 30-minute jobs lease in practice (would need millions of pending rows in one proc). Per-chunk `heartbeat` where present is belt-and-braces, not load-bearing.
|
||||
|
||||
## Connection Reuse Details
|
||||
|
||||
### `app.storage/resolve` patterns:
|
||||
@ -78,7 +71,6 @@ Since `put-object!` uses backend-specific operations (`impl/resolve-backend` + `
|
||||
- Objects can therefore share content across users and files within one bucket.
|
||||
- Deleted objects are not reused.
|
||||
- `tempfile` objects never use deduplication, even when the caller requests it.
|
||||
- `job-resource` objects never use deduplication either, whatever the caller asks: the object belongs to one profile, and a shared blob would serve the artifact of one user to the job of another. `put-object!` excludes the bucket from the dedupable set, so the invariant does not depend on call sites.
|
||||
- Use `sto/wrap-with-hash` when the caller already calculated the content hash.
|
||||
|
||||
## Bucket Rules
|
||||
@ -93,16 +85,15 @@ Since `put-object!` uses backend-specific operations (`impl/resolve-backend` + `
|
||||
| `organization` | Organization logos uploaded by the Nitrate management API. | Yes | Public | No reference scan. A touched object is deleted. |
|
||||
| `tempfile` | Export files and temporary font downloads. | No | Authentication required | No reference scan. A touched object uses a two-hour deletion delay. |
|
||||
| `upload-session` | Chunked-upload chunks. References: `upload_session_chunk.object_id` and `upload_session_chunk.session_id` (both NO ACTION DEFERRABLE: restrict semantics, procedural deletion). | No | Authentication required | Reference scan on `upload_session_chunk.object_id`; a referenced chunk freezes, an orphan is deleted after the delay. Deferred rows keep their mappings until give-up. |
|
||||
| `file-data` | Encoded file data when `file-data-backend` is `storage`. Reference: the `file_data.storage_object_id` column (FK `NO ACTION DEFERRABLE INITIALLY IMMEDIATE`); the object metadata keeps `file-id` and the `file_data` row ID. | Yes | Authentication required | Reference scan. |
|
||||
| `file-data` | Encoded file data when `file-data-backend` is `storage`. Reference metadata has `storage-ref-id`, `file-id`, and the `file_data` row ID. | Yes | Authentication required | Reference scan. |
|
||||
| `file-data-fragment` | Compatibility value for file-data fragments. The current backend has no dedicated producer for this bucket. | No current write semantics | Public | No touched-object collector case. |
|
||||
| `file-change` | Compatibility value for file changes. Current snapshots store data in `file_data`, not this bucket. | No current write semantics | Authentication required | No touched-object collector case. |
|
||||
| `job-resource` | Storage objects owned by job rows (`job.resource_id`), written by `app.jobs.storage`. | No | Authentication required, owner-scoped | jobs-GC touch → reference scan. |
|
||||
|
||||
- The valid bucket set is `app.storage.schema/metadata-buckets` (derived from `bucket-requirements`, where a new bucket and its required metadata keys are one entry); `app.storage/valid-buckets` aliases it.
|
||||
- The valid bucket set lives in `app.storage/valid-buckets`.
|
||||
- `file-media-object` is the default bucket for old rows without bucket metadata.
|
||||
- Do not assign a new bucket without adding its access and cleanup behavior.
|
||||
- The touched-object collector raises an internal error for an unknown bucket.
|
||||
- It supports `file-media-object`, `team-font-variant`, `file-object-thumbnail`, `file-thumbnail`, `profile`, `file-data`, `tempfile`, `upload-session`, `job-resource`, and `organization`.
|
||||
- It supports `file-media-object`, `team-font-variant`, `file-object-thumbnail`, `file-thumbnail`, `profile`, `file-data`, `tempfile`, `upload-session`, and `organization`.
|
||||
- It does not support `file-data-fragment` or `file-change`.
|
||||
|
||||
## Access Rules
|
||||
@ -110,7 +101,6 @@ Since `put-object!` uses backend-specific operations (`impl/resolve-backend` + `
|
||||
- `app.http.assets` decides direct object authentication from the bucket.
|
||||
- Public buckets are `file-media-object`, `file-object-thumbnail`, `team-font-variant`, `file-data-fragment`, and `organization`.
|
||||
- Other valid buckets require a session or access-token profile ID.
|
||||
- `tempfile` and `job-resource` are owner-scoped: the request must come from the profile stored in the object metadata, and a mismatch answers 404 (counted as `unauthorized`). A missing owner is tolerated only in `tempfile`, where legacy objects predate the metadata; `job-resource` always carries one, so an object without it is never served.
|
||||
- File-media routes also require file read permission.
|
||||
- Non-public direct responses set `content-disposition: attachment`.
|
||||
- FS responses use `x-accel-redirect` for the configured asset path.
|
||||
@ -118,40 +108,13 @@ Since `put-object!` uses backend-specific operations (`impl/resolve-backend` + `
|
||||
|
||||
## File Data
|
||||
|
||||
- `file-data-backend` accepts `legacy-db`, `db`, or `storage`; `legacy-db` is read-only and the default is `db`. The configuration reads `legacy-db` as `db` at startup, with a warning (`normalize-file-data-backend`), so a deployment that kept the variable saves instead of failing on every write.
|
||||
- `legacy-db` reads main data from `file.data` and snapshots from `file_change.data`; it no longer writes them (`fdata/upsert!` rejects it).
|
||||
- `file-data-backend` accepts `legacy-db`, `db`, or `storage`.
|
||||
- `legacy-db` stores main data in `file.data` and snapshots in `file_change.data`.
|
||||
- `db` stores encoded data in `file_data.data`.
|
||||
- `storage` stores encoded data in storage subsystem with `file-data` bucket, keeps `data` nil in `file_data` table, and points at the object with `file_data.storage_object_id`.
|
||||
- `file_data.storage_object_id` is a `NO ACTION DEFERRABLE INITIALLY IMMEDIATE` foreign key to `storage_object(id)`: the database refuses to physically delete a referenced object. The `IMMEDIATE` is load-bearing: the RI violation fires inside the `storage-gc-deleted` chunk transaction, so the chunk rolls back, is logged at error level with a generic hint plus the `constraint` and `table` the database reports (any `NO ACTION` foreign key on `storage_object` can block the delete, not only `file_data`'s) and is retried on the next run; with `INITIALLY DEFERRED` the check would fire at commit, outside the catch, and the sweep would never move past the blocked chunk.
|
||||
- The column is the only source of truth. The release is a coordinated restart (no rolling deploy) when the instance uses the storage backend (`PENPOT_FILE_DATA_BACKEND=storage`) or still has rows carrying `metadata.storage-ref-id`; with the default `db` backend and no such rows the previous and the new code read the same `file_data.data` column, so a rolling deploy is safe. `metadata.storage-ref-id` is no longer written or read, and the `file-data` metadata schema does not declare it. The migration backfills the rows it can resolve (valid uuid with an existing `storage_object`) and removes the key from those rows; a corrupt or orphan pointer keeps the key, which nothing reads. A large instance runs `operations/2.20.0-file-data-storage-object-id.md`: column, index and FK online, then the backfill (only if a row still carries `metadata.storage-ref-id`) with every instance stopped, verifies, and marks the migration as applied (`INSERT INTO migrations`). The backfill must never run while the previous code serves when the storage backend is in use (or rows with the pointer remain), because that code writes and scans the metadata pointer. A rollback to the previous code cannot read the rows written by this release when the storage backend is in use.
|
||||
- `fdata/upsert!` touches the `storage_object_id` it receives in the params (the pointer the persisted row had) before replacing it, so the replaced blob is reclaimed. Callers pass the column from the file map (main/restore), `nil` for a new row (fragments, snapshots, new files) or the row itself (offload); there is no query in the upsert path, which is why every caller reads the file under the row lock first (the migration-on-read path included, see `mem:backend/subtleties`). It returns `{:written <bool> :storage-object-id <uuid or nil>}`, the pointer of the object the write created (nil for `db`/`legacy-db`), and `binfile.common/update-file!` propagates it so a migrating read returns the map with the pointer of the row it persisted instead of the pre-write one. The write also compares the expected pointer with `IS NOT DISTINCT FROM` (the caller's value, not the new one) and raises `:stale-file-data-pointer` when the row changed under it, so a stale map fails instead of leaking the replaced blob; the check lives in `upsert-in-database` because the `legacy-db` branches do not use the upsert SQL.
|
||||
- `binfile.common/update-file!` takes the file map read from the database plus the fields to change (`[cfg file & params]`), merges them and builds the `file_data` params from the merged map, so the pointer travels in the read map and a caller cannot lose it by building a partial map; the params cannot override `:storage-object-id`. `insert-file!` always drops it (an INSERT never replaces a persisted pointer; `duplicate-file` hands the source map there). `bfc/get-file` keeps the pointer for the write paths, and every response goes through `bfc/file->response`, which strips it and any residual `storage-ref-id` left in the metadata of the rows the migration does not backfill: each `file-data` object is authentication-only and any logged-in profile could otherwise fetch it from `/assets/by-id`.
|
||||
- The in-place binfile import was removed: `import-files!` raises `:not-supported` when the cfg carries `::bfc/file-id`, so no caller can overwrite a storage-backed row without carrying its previous pointer.
|
||||
- `storage` stores encoded data in storage subsystem with `file-data` bucket and keeps `data` nil in `file_data` table.
|
||||
- The `file_data.metadata.storage-ref-id` value points to the storage object.
|
||||
- `fdata/upsert!` touches a storage object from incoming metadata before it stores the new row.
|
||||
- File snapshots use `file_data` for snapshot data and `file_change` for snapshot metadata.
|
||||
- Tiered offload (`:tiered-file-data-storage` flag + `db` backend): `file-gc` submits an `:offload-file-data` job (priority 10, delay 1000, same params) only when it actually processed the file; the handler takes the `file` row with `FOR UPDATE SKIP LOCKED` and skips (`{:skipped 1}`, no error) when the row is locked or gone, then moves every live `file_data` row of that file with `backend = 'db'` to `backend = 'storage'` via `fdata/upsert!` (one `put-object!` in bucket `file-data` per row, `data` becomes nil and `storage_object_id` points at the new object) inside a single all-or-nothing transaction per file. The skip only serializes because every `file_data` writer locks the `file` row first (the update RPC and the snapshot paths with `FOR UPDATE` through `lock-for-update?`, the migration-on-read path with `FOR UPDATE`). A storage failure aborts only that file and bubbles for a runner retry; orphans go to `storage-gc-touched`. A second run is a no-op because the query only matches `backend = 'db'`.
|
||||
- The offload normalizes each `SELECT fd.*` row before `upsert!`: the row builder omits NULL columns, jsonb metadata arrives as a `PGobject`, and a JSON `null` decodes to nil and must disappear instead of travelling as `:metadata nil`, which the update schema rejects.
|
||||
- Offload pacing: none by deliberate decision. Sleeps only make sense when batching multiple transactions; the per-file transaction is short by construction and the scheduler run takes minutes per production metrics, so neither loop sleeps. `jobs/heartbeat` (pool-based liveness, ~1 write/s) stays on both paths.
|
||||
- Offload metrics (bounded labels only, never ids or error text): `penpot_file_data_offload_rows_total{type}` and `penpot_file_data_offload_bytes_total{type}` (`type`: `main`/`snapshot`/`fragment`, successful runs only) plus `penpot_file_data_offload_timing_ms{result}` (`result`: `ok`/`skip`/`error`). Helpers live in `app.jobs.metrics`; the job-def requires `::mtx/metrics`.
|
||||
- Writes always resolve the backend through `fdata/default-backend` (`cf/get :file-data-backend`), so post-offload edits land back in hot (`db`) storage without extra code; reads stay transparent through `fdata/resolve-file-data` + `decode-file-data` + `realize`.
|
||||
- Deleting an offloaded row touches its storage object first (`objects-gc/delete-file-data-chunk` via `fdata/touch-storage-object` with the row's `storage_object_id`), so `storage-gc-touched` reclaims the blob with the usual delay; restoring a file unmarks the referenced objects.
|
||||
- `storage-gc-touched` resolves the `file-data` reference with `SELECT EXISTS (SELECT 1 FROM file_data WHERE storage_object_id = ?)`, an indexed lookup that no longer parses the object metadata.
|
||||
- A row with `backend='storage'` and no readable object makes `resolve-file-data` raise `:missing-storage-object` with the file and row ids instead of failing later in the storage backend.
|
||||
- The `file-id` and row id kept in the storage object metadata are informational for diagnostics; nothing parses them to resolve the reference (the reference is the column).
|
||||
|
||||
## File Change
|
||||
|
||||
- `file_change` is partitioned by `HASH (file_id)` in 16 partitions, with the same bounds as `file_data`, so a join between them is resolved partition by partition (`enable_partitionwise_join` is set on every pooled connection).
|
||||
- The automatic migration (`0158-partition-file-change`) refuses to run when `file_change` is above 10 GiB total size or 2M rows (`pg_total_relation_size` + `pg_class.reltuples`, catalog reads). It analyzes the table first, because `reltuples` is -1 without statistics. It raises with the path of `operations/2.20.0-file-change-partition.md`, because the copy happens inside the migration transaction and a big instance would restart in a loop. Below the guard it raises `statement_timeout` to 30 minutes for the copy and the index builds, and analyzes the new partitions at the end, after the copy and the indexes, so the planner does not start with empty statistics.
|
||||
- Its primary key is `(file_id, id)`: a partitioned table needs the partition key in the primary key, so every access filters by `file_id`. From 2.20 the four snapshot commands (`update-file-snapshot`, `delete-file-snapshot`, `lock-file-snapshot`, `unlock-file-snapshot`) require the `file-id` param, because they use it to prune the partition; without it they answer `:params-validation`.
|
||||
- It has a `kind` column: `1` is a snapshot (it has a label and its data lives in `file_data`; the read-only `legacy-db` backend keeps old payloads in `data`) and `2` is a revision (it has no label and it holds the changes). The values are in `app.features.file-changes`.
|
||||
- `kind` is a deliberate reserve, not a leftover: today no query filters by it, the queries still tell a snapshot from a revision by its label (`label IS NOT NULL`), and `kind` takes over when they are rewritten. No constraint ties `kind` and `label` together on purpose: the day `kind` is the source of truth, that constraint would have to go. It stays nullable on purpose: the previous version inserts without it, and a rollback to it has to keep writing; the `NOT NULL` comes in a later migration.
|
||||
- Every write of the application to a row of `file_change` moves `updated_at` to the moment of the write (`lock-by!`, `unlock!`, the snapshot rename and delete, `delete-object`, the restore of a file). The actions of the foreign keys (`ON DELETE SET NULL` on `locked_by` and `profile_id`) change the row without touching it; during the partition both tables carry the same keys, so both receive them. The runbook of the partition uses it as the watermark of its delta passes: a write of the application that leaves it behind is lost at the cutover.
|
||||
- `file_change_legacy` is a frozen copy of the previous layout of the partition: it keeps the rows and the indexes of the old table. A new instance (empty table) does not keep it. On an upgraded instance the table stays until the operator drops it by hand days later (`DROP TABLE file_change_legacy;`, see `operations/2.20.0-file-change-partition.md`); nothing drops it automatically.
|
||||
- A revision carries only the delta of a save (the changes). The features of the file are not repeated on it: nobody read them from there.
|
||||
- A snapshot carries the features and the migrations of the file.
|
||||
- It keeps the `data` column for reading only: migration `0157-migrate-legacy-file-data` moved the existing blobs to `file_data`, and `legacy-db` remains as a read-only backend for the payloads still in `data`. `fdata/upsert!` rejects `legacy-db` and the `file-data-backend` default is `db`. It has no `data_ref_id` or `data_backend` columns: nothing read them since the file storage refactor.
|
||||
- It has no index for the system snapshots. The old one filtered them by `data IS NOT NULL`, and with the payload living in `file_data` the column no longer tells snapshots apart. The new one could only filter them by `deleted_at IS NULL`, but the system snapshots always carry a deletion date in the future, so it would never match a row. The query that would use it filters that date with an "or", which no partial index can serve.
|
||||
- The `objects-gc` purge groups each chunk by `file_id` and locks it with `file_id = ?` and `id = ANY(?::uuid[])`: a tuple `IN ((file_id, id), ...)` gives the planner no constant partition key, so it scans all 16 partitions in every chunk. Each group is one `FOR UPDATE SKIP LOCKED` query inside the chunk transaction.
|
||||
|
||||
## Metrics
|
||||
|
||||
|
||||
@ -9,7 +9,6 @@
|
||||
- Params with `::sm/params` are decoded/conformed through the JSON transformer and successful IObj results get `:encode/json` metadata. Legacy spec conforming only applies when no Malli params schema exists. Client params are stripped of qualified keys (`d/without-qualified`) before merging with the server auth context, so request bodies cannot override `::profile-id`, `::auth-type`, or `::token-perms`.
|
||||
- Params schemas are open by default, so undeclared client keys reach the handler unless the map is `:closed true`. Creation commands (`create-file`, `create-project`, `create-team`, `create-team-with-invitations`, `upload-file-media-object`, `create-file-media-object-from-url`, `assemble-file-media-object`) use closed schemas: a client-provided `:id` fails with `:params-validation`. Their internal creation functions still accept an optional explicit `:id` for imports, duplicates and deterministic test fixtures.
|
||||
- Nil RPC bodies become HTTP 204 unless explicit status metadata is present. Stream bodies default to `application/octet-stream` when no content type is set.
|
||||
- The upload schema (`media.v/schema:upload`) takes a `:path` from the caller, and a Transit body can tag any string as a `path`, so a command that reads such an upload reads a file the client named. `create-import-binfile-job` avoids that boundary by accepting only a chunked upload (`:upload-id`); the commands that still take a plain file (the legacy `import-binfile`, the media and font uploads) trust it, and the guard belongs where uploads are accepted, not in each command.
|
||||
|
||||
## Concurrency limiter (CLIMIT)
|
||||
|
||||
@ -27,39 +26,27 @@
|
||||
|
||||
- Most `app.db` helpers accept a pool, connection, or map containing `::db/pool` / `::db/conn`; preserve that convention in shared code.
|
||||
- `db/tx-run!` uses `next.jdbc.transaction/*nested-tx* :ignore`: nested transaction calls reuse the outer transaction, not a savepoint. Use explicit savepoints when nested rollback semantics matter.
|
||||
- Whether a call is nested is decided by the CONNECTION, not by an ambient flag: `db/transact!` binds `*after-commit-conn*` around the body, and a transactable that is not identical to that connection is an independent unit of work that owns its own after-commit context. A `tx-run!` that passes a pool while a transaction is open therefore commits on its own and drains its own callbacks, instead of leaving them in a context that may roll back.
|
||||
- `after-commit` callbacks run on commit only. A transaction marked `::db/rollback` (used by `srepl` and `app.tasks/offload-file-data`) commits nothing, so it runs none of them.
|
||||
- `db/run!` opens/reuses one connection but does not create a transaction.
|
||||
- `db/tjson` is Transit JSON for jsonb storage; `db/json` is plain JSON. Job params (`job.params`) are plain JSON, decoded with the job-def decoder (`app.jobs/decode-params`); only the legacy `task` props were Transit (`decode-transit-pgobject`). Job results (`job.result`) are plain JSON too: they are read with the result decoder of the job-def (`app.jobs/decode-result`), and a result map keyed by uuid must declare its schema (`[:map-of ::sm/uuid ...]`, the import resolution does) or every key comes back as a keyword and every uuid as the text it is on disk.
|
||||
- `db/tjson` is Transit JSON for jsonb storage; `db/json` is plain JSON. Worker task props use Transit and are decoded with `decode-transit-pgobject`.
|
||||
- Advisory transaction locks accept UUIDs or ints. UUID locks are hashed using a zero-UUID seeded siphash.
|
||||
|
||||
## Workers and cron
|
||||
|
||||
- `job` carries `tenant` (the instance the job belongs to) and a bare `queue`. Several instances share one database and the tenant is what keeps an instance from claiming another environment's jobs, so every query that acts on one instance's work filters on `tenant`: dispatch claim, orphan sweep, lost-job reschedule, cron no-overlap count, submit dedupe. The `job__dispatcher__idx` leads with `tenant` (equality) then `(priority DESC, scheduled_at)`; `queue` does not lead because the dispatcher claims every queue of its tenant. The `<tenant>:<queue>` prefix survives only in the Redis key, composed once by `app.worker/queue-key`.
|
||||
- Submit dedupe only removes not-yet-due `new` jobs of the same tenant with the same name/queue/label; it does not dedupe due, scheduled, retry, running, or completed work. The `::jobs/skip-if-active` submit option goes further: it skips the insert (returning `nil`, recording `penpot_jobs_skipped_total`) when a live job (`new`/`scheduled`/`running`/`retry`) already exists for the same tenant/name/queue/label, requires an explicit non-blank label, and carries the same best-effort cross-backend race caveat as dedupe.
|
||||
- Jobs metrics rule (one event, one family): counters exist only for events without latency; when an event has a histogram, its `_count` is the counter (no twin counter). `dispatched` (pushes) vs `queue-wait` (claims) are deliberately different events — do not merge them.
|
||||
- The dispatcher selects `new`/`retry` jobs with `FOR UPDATE SKIP LOCKED`, marks them `scheduled`, and publishes the JSON payload `[id scheduled-at]` to the `penpot.worker.queue:<tenant>:<queue>` Redis list. The runner skips Redis messages whose scheduled timestamp no longer matches DB state.
|
||||
- Lost `scheduled` jobs are rescheduled after 5 minutes; `running` jobs untouched longer than `:jobs-lease` (default 30 min) are marked `aborted` with `app.jobs/orphan-error` (`type :internal`, `code :orphan`). `aborted` is a system-side terminal state like `cancelled` (never retried, swept by retention like other terminals, reported with an error log); long jobs must `heartbeat` per chunk so a healthy backlog is never mistaken for a dead process. The `:progress` report is optional and only feeds the durable log. A beat whose write finds the job no longer active raises an `:interrupt` (`code :job-interrupted`) instead of counting zero: it is the internal stop-the-work signal, not an error, so neither the listener nor the runner logs it as one.
|
||||
- An orphan gets its `end` event (`outcome "aborted"`) from the sweep itself: one bulk `INSERT ... SELECT` in the same transaction as the bulk `UPDATE`, never one insert per orphan. No msgbus notification goes out for it; orphans alert through the error log and the orphaned counter.
|
||||
- `job_event` is append-only and deleted by cascade with the job. `progress` is a row there, not a column: no Redis and no mutable column to keep in sync.
|
||||
- `job` is deletion-protected at the SQL level (`deletion_protection__tgr`): a stray `DELETE FROM job` raises instead of dropping the row and its `job_event` cascade without touching the referenced `storage_object`. The only intended deleters, the jobs GC and the submit dedupe, disable the guard with `SET LOCAL rules.deletion_protection TO off` inside their transaction; tests use `th/db-force-delete`.
|
||||
- A missing job-def raises (`:no-job-definition`) instead of completing. Throwing with `ex-data :type ::retry` still controls retry behavior; `:strategy ::noop` retries without incrementing retry count.
|
||||
- Cron entries claim their `scheduled_task` row with `FOR UPDATE SKIP LOCKED`, disable statement/idle-in-transaction timeouts locally, submit one `job` row per entry when no active instance exists (no-overlap), and reschedule themselves in `finally` unless interrupted. Worker, dispatcher, and cron components do not start when the DB pool is read-only.
|
||||
- Integrant params schemas name injected components as bare `::ns/key` entries (pool, metrics, msgbus, job defs), and a bare entry is a malli registry lookup, not `any?`: the owning namespace must `sm/register!` it at load, or the backend crashes at boot with `:malli.core/invalid-schema` instead of asserting. Unit tests never build those keys, so `backend-tests.worker-test` compiles the three worker schemas and `backend-tests.http-websocket-test` compiles the revocation-watcher one to catch it. The assertion only runs with `PENPOT_FLAGS=enable-backend-asserts`, so without it a broken schema reaches production and only crashes at boot.
|
||||
- Whether a bare entry works has nothing to do with its position or with how many of them there are: it is a registry lookup, so it works only if that exact keyword was `sm/register!`'d. `[:map ::db/pool ::metrics/metrics]` is fine, `[:map :app.aaa/unregistered :app.ccc/unregistered]` throws. When a component is **not** registered as a malli type, spell it as an explicit pair instead of registering a type just to satisfy the lookup — `[:app.nitrate/client {:optional true} [:maybe :map]]` is how `auth.oidc/schema:routes-params` and `loggers.audit.archive-task/schema:audit-log-archive-params` declare it. `{:optional true}` matters because `init-key` yields `nil` when the owning feature flag (`:admin-console`) is off, and a required key would then fail the assert on a perfectly normal boot.
|
||||
- `sm/valid?` catches every `Throwable` and returns `false`, so an uncompilable schema and a genuinely invalid value both surface as a bare `Assert failed:` with no cause. When an `assert-key` fires with no message, compile the schema by hand to see the real exception.
|
||||
- Task queues are tenant-prefixed. Submit dedupe only removes not-yet-due `new` tasks with the same name/queue/label; it does not dedupe due, scheduled, retry, running, or completed work.
|
||||
- The dispatcher selects `new`/`retry` tasks with `FOR UPDATE SKIP LOCKED`, marks them `scheduled`, and publishes Redis payload `[id scheduled-at]`. The runner skips Redis messages whose scheduled timestamp no longer matches DB state.
|
||||
- Lost `scheduled` tasks are rescheduled after about 5 minutes; `running` tasks older than about 24 hours are marked failed as orphans.
|
||||
- A task handler that is missing or returns an invalid result currently defaults to completed after warning. Throwing with `ex-data :type ::retry` controls retry behavior; `:strategy ::noop` retries without incrementing retry count.
|
||||
- Cron jobs lock their `scheduled_task` row with `FOR UPDATE SKIP LOCKED`, disable statement/idle-in-transaction timeouts locally, and reschedule themselves in `finally` unless interrupted. Worker, dispatcher, and cron components do not start when the DB pool is read-only.
|
||||
|
||||
## Config and HTTP/session middleware
|
||||
|
||||
- `app.config/config` and `flags` are dynamic `defonce` vars populated from `PENPOT_*` env vars through the shared schema string transformer. Tests and tooling can bind them.
|
||||
- Duration config values are parsed as ISO-8601 with the `PT` prefix, so a day is written in hours (`:jobs-retention` and `:jobs-user-ttl` default to a week, and an operator writes `168h`): `7d` does not parse, and the value fails validation at boot instead of being silently ignored.
|
||||
- `app.main` wires `PENPOT_DATABASE_*` into `::db/pool`, and `app.db/create-datasource-config` forwards pgjdbc's own knobs as `dataSourceProperties`, which `Properties.getProperty` silently drops unless the value is a String (hence the `str`). A query parameter of the same name in `PENPOT_DATABASE_URI` (for example `?prepareThreshold=0`) wins over the property: pgjdbc ignores the Properties value when the URL carries it, so the URI is the place to look when a variable seems to do nothing. The prepared statement cache is three optional `PENPOT_DATABASE_PREPARED_STATEMENT_CACHE_*` keys: the threshold is the executions before a statement is server-prepared and accepts `-1` or non-negatives (`{:min -1}`; the driver clamps any other negative to `-1`, so rejecting it only turns a typo into a boot error), its `0` stops server-side prepared statements while the driver's client-side cache of parsed queries stays in use, `..._QUERIES` and `..._SIZE` reject negatives and `0` in either disables that cache, and the size is bytes rounded up to the whole MiB the driver accepts. Do not describe the threshold `0` as disabling the cache: it only stops the server-side prepares.
|
||||
- `parse-flags` automatically adds `:disable-secure-session-cookies` when `public-uri` is plain HTTP and not localhost. This changes cookie defaults without an explicit env flag.
|
||||
- The backend sets Clojure `*assert*` globally from the `:backend-asserts` feature flag. Assertion-dependent checks can therefore differ by runtime flags.
|
||||
- Request body parsing is mostly POST-oriented and supports Transit JSON plus plain JSON. Plain JSON request keys are kebab-decoded before being merged into `:params`. A body the parser cannot read is a client error, classified at the parse site and never by exception class (`clojure.data.json` signals its syntax errors with a bare `java.lang.Exception`): 400 `:validation`/`:malformed-json`, one `warn` line (`malformed request body`) carrying path, peer address and content-type instead of a stack trace, and `penpot_http_malformed_requests_total`. `RequestTooBigException` keeps its 413.
|
||||
- Response formatting negotiates with `Accept` or `_fmt=json`. Transit is the default for collection/boolean bodies; JSON encoding has special pointer-map handling.
|
||||
- Auth prefers the session cookie token before the `Authorization` header. Headers may be `Token` or `Bearer`. Only `kid=1`/`ver=1` tokens are decoded as session tokens; anything else is left unauthenticated (legacy v1 tokens were removed).
|
||||
- Shared-key auth requires `x-shared-key` as `<key-id> <key>` and stores the lowercased key id on the request. If no shared keys are configured it always rejects. Management methods declare the key ids they accept with `::rpc/allowed-shared-keys`; a valid key outside that set answers 403 `:shared-key-not-allowed`, and a method without the metadata accepts any configured key.
|
||||
- Shared-key auth requires `x-shared-key` as `<key-id> <key>` and stores the lowercased key id on the request. If no shared keys are configured it always rejects.
|
||||
- Session management uses DB storage unless the DB pool is read-only, then falls back to the in-memory manager. Sessions use only the v2 UUID model (`http_session_v2`); legacy string ids were removed.
|
||||
- Session cookies are renewed when `modified-at` is older than the 6h renewal interval. SameSite is `none` for CORS, otherwise strict/lax based on config. Session lifetime config and GC: `mem:backend/session-expiration`.
|
||||
|
||||
@ -73,9 +60,7 @@
|
||||
|
||||
## Metrics recording
|
||||
|
||||
- `app.metrics/run!` is safe by default: a recording failure never throws (a metrics bug must not change the behavior of the measured operation). The first failure per metric id logs at `warn`, later ones at `debug`. The `instance` precondition is a plain assert (the backend enables `:backend-asserts`), and the collector lookup sits outside the recording guard, so a missing instance fails hard even when asserts are disabled. `::mtx/metrics` is required by the storage, s3-backend, db-pool and jobs APIs; jobs helpers never silently skip a missing metrics instance. The resolver is `app.metrics/instance` and it raises, unlike `run!`.
|
||||
- `app.msgbus` (`pub!`, `sub!`, `purge!`) keeps taking the **instance** on purpose: msgbus is what produces it, and `app.rpc.notifications` already shows the caller passing a cfg. `app.jobs` resolves `::mbus/msgbus` from the cfg internally, and `insert-event` keeps its explicit pre-insert check so a profile job never loses its notification.
|
||||
- `app.db/after-commit` is the boundary for SQL-backed metrics: callbacks are drained by the commit of the transaction that owns the connection they were registered on, and discarded when it rolls back. Because nesting follows the connection, that transaction is the outermost one only for a real nested call.
|
||||
- `app.metrics/run!` is safe by default: a recording failure never throws (a metrics bug must not change the behavior of the measured operation). The first failure per metric id logs at `warn`, later ones at `debug`. The `instance` precondition is a plain assert (the backend enables `:backend-asserts`), and the collector lookup sits outside the recording guard, so a missing instance fails hard even when asserts are disabled. `::mtx/metrics` is required by the storage, s3-backend, and db-pool schemas; `app.db` wires the prometheus `MetricsTrackerFactory` unconditionally. Storage-specific metric contracts: `mem:backend/storage`.
|
||||
- `penpot_http_malformed_requests_total` (counter, no labels) counts requests whose body the parser could not read. `app.http.middleware/wrap-parse-request` emits it where it classifies the error, so the instance travels as a middleware argument (`[mw/parse-request (::mtx/metrics cfg)]`) and each service declares the definition with its own prefix (nexus: `nexus_http_malformed_requests_total`). The per-endpoint request counters of nexus do not count these requests: they sit inside the parse middleware, so the failure happens before them.
|
||||
|
||||
## Log-derived metrics
|
||||
@ -87,32 +72,21 @@
|
||||
- The `media-storage-bytes-per-team` quota check on `upload-file-media-object` is TEMPORARILY DISABLED (2026-10-19, kept commented in `app.rpc.commands.media` with its `sql:get-team-id-for-file` lookup still active for the audit props): the quote performs a full `SUM` over `storage_object` per upload and was too slow in production. The `upload-sessions-per-profile` quota is unaffected and stays active. Re-enable only with a cheaper approximation.
|
||||
- Storage abstraction, backend configuration, logical buckets, object lifecycle, deduplication, access rules, and garbage collection: `mem:backend/storage`.
|
||||
- SVG validation strips DOCTYPE and uses secure SAX parsing. Basic SVG info falls back to 100x100 dimensions when width/height/viewBox are missing.
|
||||
- The SVG sanitizer (`app.media.svg`) is a **blocklist**, not an allowlist: it drops dangerous elements and attributes by name and escapes every text and attribute value when it writes the tree back, so a payload hidden in a value cannot become live markup again. The allowlist rework is [#12190](https://github.com/penpot/penpot/issues/12190) and lands in media-processor, not here. `sanitize-svg` also tracks the namespace bindings through the recursion and drops any element that is not in the SVG namespace, so a document whose root is not an `svg` element in the SVG namespace is rejected and a child that rebinds `xmlns` to XHTML (`iframe`/`srcdoc`/`meta`) is dropped with its subtree; it compares the scheme of an URL the way the browser does (dropping tab/CR/LF/spaces), so `java	script:` does not slip through the `javascript:` filter. The paths that store an uploaded SVG as its final object go through `app.media.svg/sanitize-svg-upload` (media upload, the URL download, both thumbnail writers and the organization logo); the three binfile imports go through `app.binfile.common/sanitize-imported-svg`, which calls `sanitize-svg` directly. The profile picture is not an SVG: `import-profile-picture` ignores an `image/svg+xml` download, because the avatar is always rendered as an image. The caller checks the size before calling it: the helper reads the whole file, and the size limit applies to the upload, not to the sanitized copy, which can be up to six times bigger.
|
||||
- Upload handlers run the `media.v` checks in a fixed order: `check-upload-path`, then `check-media-type`, then `check-media-size` (or `check-font-size` for fonts). The path check gates every later read: `check-media-type` sniffs the bytes at `:path` and `check-media-size` reads `fs/size`, so a path the client named outside `allowed-upload-prefixes` must be rejected before either touches the disk. `check-media-size` leans on that and answers a file it cannot read with the path code (`:params-validation`), not `:media-max-file-size-reached`. Keep the order when touching or adding a handler; two branches that reorder these lines conflict.
|
||||
- Raster metadata is shell-derived with ImageMagick `identify`, verifies detected MIME against the supplied MIME, and swaps dimensions for EXIF orientations 6/8.
|
||||
- Remote image download requires 2xx status, `content-length`, a known MIME, and size under the configured maximum before writing the temp file; mismatched byte count is an internal error.
|
||||
- Font processing shells out to FontForge and WOFF conversion tools and can derive TTF/OTF/WOFF variants from uploaded fonts.
|
||||
|
||||
## File data persistence
|
||||
|
||||
- File data backends for writing are `db` and `storage`; `legacy-db` is a read-only alias for old payloads. The storage backend keeps encoded file data in storage bucket `file-data`; the DB row stores nil data and points at the object with the `storage_object_id` column (FK `NO ACTION DEFERRABLE`).
|
||||
- `fdata/upsert!` touches the storage object the persisted row pointed at before replacing it; the pointer travels in the params (`file->file-data-params` from the read file map, `nil` for a new row, the row for the offload) and the params of `update-file!` cannot override it. `binfile.common/update-file!` merges the read file map with the fields to change, so the pointer survives; `insert-file!` drops it because an INSERT never replaces a persisted object. The column is the only source of truth and the metadata no longer carries `storage-ref-id` (the migration removes it from the backfilled rows and the write path never stores it); the release is a coordinated restart when the storage backend is in use (the previous code cannot read the new rows), while the default `db` backend reads the same data column and tolerates a rolling deploy. Responses built from a file map go through `bfc/file->response`, which strips the pointer and a residual `storage-ref-id` from the metadata.
|
||||
- Pointer-map fragments are persisted separately as type `fragment`, and only modified pointer maps are written. A pointer keeps its id while nothing changes; content (`assoc`/`without`) and metadata (`with-meta`) changes produce a fresh id, so a fragment row is never rewritten after it is written. Re-assoc with an identical value returns the same map in array and hash maps alike, so a save keeps the fragment id of every page it does not change.
|
||||
- `enable-objects-map` skips unloaded pointer pages: their fragment was already wrapped when it was written, and loading it only to wrap it again costs one page read per untouched page on every save. A page with a plain fragment is wrapped when it is first loaded or edited; a data migration that enables `fdata/objects-map` must rewrite the data itself.
|
||||
- Snapshots share fragment rows with the live file (they are stored with pointers, not materialized). `file-gc` marks every fragment referenced neither by the current main file nor by an unpurged snapshot; the file is read unrealized and unchanged pages keep their ids via reattach, so a pass over an unchanged file writes no new fragments.
|
||||
- A save attaches its automatic snapshot before applying pending file migrations: the snapshot keeps the pre-migration pointers, data and migration set, and restoring it leaves the next read to migrate the file. This also holds for files without the feature. The preview (`get-file-snapshot`) applies the pending migrations in memory with the live file's libraries and never persists them: the stored version keeps its data, the payload carries the snapshot's `:migrations` (an ordered set) and `fdata/objects-map` is realized for clients that do not declare it. The historical feature set is deliberately not checked against the enabled one: a version may predate a feature change. The window is narrow: the `get-file` RPC reads with `migrate?` on and its ETag hashes `available-migrations`, so a post-deploy read cannot be a 304 from the pre-deploy state and migrates before any save in the normal flow; the pre-migration snapshot only appears when the save is the first request to touch the file after the deploy (an open tab that saves without re-reading, or a non-UI writer). `create-file-snapshot` and the restore backup read with `migrate?` on, so those versions are stored already migrated.
|
||||
- `:migrations` is always a `d/ordered-set`: `file-migrations/resolve-applied-migrations` builds one, `fmg/migrate` re-wraps the union after applying, and `file-snapshots/decode-snapshot` converts the `file_change` pgarray (as a plain vector, `set/difference` and `contains?` would compare indexes instead of names). A snapshot with NULL `:migrations` and an old `:version` predates the `file_change.migrations` column (SQL migration 0140): the preview regenerates the set from the stored version and applies that full chain, while `restore!` uses a fixed 67 baseline, so preview and restore can disagree for very old snapshots.
|
||||
- The migration read path (`binfile.common/get-file*`) reads the `file` row once (`sql:get-file`, with its data columns; with the storage backend the blob lives in the object, not in the column) plus an explicit `resolve-applied-migrations`, because a row read does not resolve them. A read with no pending migration is completed by `attach-file-data` on that map: one row read and one migrations resolution. Only when a migration is pending is the row re-read under `FOR UPDATE` on the `file` row, where `need-migration?` is re-checked on the fresh map and the migrated file persisted (`binfile.common/migrate-file`); that re-read resolves the migrations again, and `migrate-file` once more after persisting, to return the applied set. The migration path holds two raw copies of the blob (the first read and the locked re-read) but a single decoded one: the alternative, a row-only probe before the read, avoided the first raw copy at the cost of an extra row read on every open, and was removed. `get-file` runs the read inside `db/tx-run!` (joining the caller's transaction when the cfg carries a connection) so the persist commits with the read; a read-only read (`:read-only?`) takes no lock and writes nothing, so it runs on a plain connection (`db/run!`) instead of opening a transaction of its own, and with a cfg that already carries a connection it still joins the caller's. It accepts a cfg map only and throws an `IllegalArgumentException` otherwise (`assert` is elided without `:backend-asserts`); a data read rejects `:lock-for-share?` (only row-only reads accept it), because the migration branch re-reads with `FOR UPDATE`, and a lock option needs a read-write transaction from the caller because PostgreSQL rejects row locks in a read-only transaction. A row-only read (`load-data?` false) resolves the applied migrations by default too (`:load-migrations?`), because the row can be completed later and the caller may need them; a caller that does not want them passes `:load-migrations? false`. Persisting is an optimization of the read (the read already returns migrated data): a row that changed between the first read and the locked read is not overwritten, and a pending migration discovered on the unlocked read is applied in memory only (the next read persists it under the lock). This was the only writer of the file row outside the lock protocol (save, library absorption, restore, `file-gc` and offload already take it); without it, a save that committed in between was overwritten by the stale map (lost content, `revn` moving back so the next save fails with `:revn-conflict`, and the object the save left as a frozen orphan). `migrate-file` captures `fdata/original-pointers` before realizing, reattaches unchanged positions after applying the migrations, and persists the reattached file; it returns that reattached map when it persists (the pages the migration does not change keep their pointers and fragment ids) and the realized map when it does not, so a read is plain only when the caller asks with `:realize?`, and a migration that adds `fdata/objects-map` skips the reattach so the pages get wrapped.
|
||||
- `fdata/load-pointer` reads rows that a previous `file-gc` pass marked on purpose (explicit `::db/remove-deleted false`): the fragment is only gone once `objects-gc` purges it. The `get-file-fragment` RPC keeps rejecting marked rows.
|
||||
- Bind the fragment loader through `fdata/with-pointers` (reads), `fdata/with-tracked-pointers` (writes with a loader) or `fdata/with-tracked` (writes with nothing to load yet, e.g. file creation), and `fdata/with-untracked` (validation passes that must track nothing) instead of hand-rolled `binding` forms. Two direct bindings remain on purpose: the fine-grained tracked toggles of `app.rpc.commands.files-update` (validation/repair turn tracking off, restore the existing atom and read it before repairing), and `binfile.common/encode-file` (a fresh tracked set inheriting the current one). Converge them only if atom-scoped macros (`with-tracking`, `with-inherited-tracking`) are added; today the extra indirection buys nothing.
|
||||
- Snapshot creation takes `FOR UPDATE` on the file row: both `create-file-snapshot` and the backup read of `restore-file-snapshot` hold it from the read to the insert, so they serialize with a `file-gc` pass (`FOR UPDATE ... SKIP LOCKED`) and with a concurrent save, and never pin fragments the pass is about to mark. The lock is exclusive and not `FOR SHARE` because the migration-on-read path may re-read the same row with `FOR UPDATE`: a shared lock that asks for the exclusive one deadlocks two concurrent readers (one creating a snapshot, the other saving, both seeing a pending migration). `get-file-snapshot` realizes the snapshot pointers with the live file id (fragment rows are keyed by file, not by snapshot).
|
||||
- File data backends are `legacy-db`, `db`, and `storage`. The storage backend keeps encoded file data in storage bucket `file-data`; the DB row stores metadata with `storage-ref-id` and nil data.
|
||||
- `fdata/upsert!` touches any storage object referenced by incoming metadata before storing the new row/blob.
|
||||
- Pointer-map fragments are persisted separately as type `fragment`, and only modified pointer maps are written.
|
||||
- `fdata/realize` combines pointer realization and object-map realization. Use it before operations that need complete in-memory file data instead of pointer placeholders.
|
||||
- A `file_data` row is one of three classes, told apart by `type`: `main` is the file's current state and carries `id = file_id`; `snapshot` is a point-in-time copy whose `id` is the one its `file_change` row shares; `fragment` is a piece of the pointer-map, referenced from the file's data.
|
||||
- Their lifecycles differ: `main` and `snapshot` live with the file (dated when it is deleted, undated when it is restored), while a `fragment` carries no date and is collected by `file_gc` once the file stops referencing it. Neither is a session buffer, so do not read a `fragment` as an ephemeral revision.
|
||||
- File snapshots (versions) keep their metadata row in `file_change` and their payload in `file_data` (`type = 'snapshot'`). With the `db` backend `file_change.data` stays NULL: that column is only read by the `legacy-db` backend (it is no longer written), plus the rows written before the 2025-07 file storage refactor.
|
||||
- Tell a snapshot apart by `label IS NOT NULL` (with `created_by` / `deleted_at` as needed), never by `file_change.data IS NOT NULL`: the label holds with every file data backend, and the payload column stopped meaning "this is a snapshot" when the payload moved out. Queries that identify versions must use the label. `0157-migrate-legacy-file-data` moved the labelled leftovers to `file_data` and deleted the unlabelled payload rows, so from 2.19 on the label is the only marker there is.
|
||||
- File snapshots (versions) keep their metadata row in `file_change` and their payload in `file_data` (`type = 'snapshot'`). With the `db` backend `file_change.data` stays NULL: that column is only written by the `legacy-db` backend, plus the rows written before the 2025-07 file storage refactor.
|
||||
- Tell a snapshot apart by `label IS NOT NULL` (with `created_by` / `deleted_at` as needed), never by `file_change.data IS NOT NULL`: the label holds with every file data backend, and the payload column stopped meaning "this is a snapshot" when the payload moved out. Queries that identify versions must use the label.
|
||||
- The version quota counts of `app.rpc.quotes` (`sql:get-snapshots-per-file`, `sql:get-snapshots-per-team`) are served by the partial index `file_change__user_snapshots__idx`, whose predicate mirrors their where clause; changing one without the other drops the index only scan and the count walks the whole table again.
|
||||
- Revisions live only in `file_change` and have no `file_data` counterpart. They are born already scheduled for deletion one hour later (`app.rpc.commands.files-update`) and the snapshot paths cannot reach one, because they resolve their row through a `label IS NOT NULL` filter.
|
||||
- A deletion date is assigned once and never moved: `restore-files` (`app.rpc.commands.files`) unmarks only durable history, which is `label IS NOT NULL` (the unlabelled rows that still carry a legacy payload are removed by `0157-migrate-legacy-file-data`, so restore leaves them dated), the `:file` method of `app.tasks.delete-object` dates only rows that have no date yet, and `objects-gc` collects only rows that carry one. Unmarking a revision leaves it invisible to `objects-gc` for ever, because nothing sets that date again.
|
||||
- A deletion date is assigned once and never moved: `restore-files` (`app.rpc.commands.files`) unmarks only durable history, the `:file` method of `app.tasks.delete-object` dates only rows that have no date yet, and `objects-gc` collects only rows that carry one. Unmarking a revision leaves it invisible to `objects-gc` for ever, because nothing sets that date again.
|
||||
- Binfile v1/v3 import and the v3 export `get-file` step run inside one long transaction. They raise `idle_in_transaction_session_timeout` to 30 min (`app.binfile.common/transaction-idle-timeout-ms`) instead of disabling it, so a connection left idle while the ZIP is decoded or the file data is realized is bounded; `statement_timeout` stays at 300 s.
|
||||
- `update-file` with `:changes-with-metadata` groups that all carry `:commit-id` is partial: fast path applies and validates everything once; on a change failure it bisects the commits for the longest prefix that applies and validates (each attempt applies a prefix from the original data and runs referential integrity + file schema; ~log2 K attempts, none for a single commit, the first attempt's error reused), keeping that prefix (a commit that repairs an earlier one can keep both) and answering 200 with `:rejected {:commit-ids :failed-commit-id :error}`, plus `:preexisting-damage true` when the data the first rejected commit touches already fails validation without it (`preexisting-damage?`: one more referential + schema check of its pages/components on the base file, the accepted prefix or the original; the frontend then says the page data is damaged). Only accepted changes reach `file_change`, persistence and the msgbus; a fully rejected request writes and broadcasts nothing (revn unchanged). Flat `:changes` or groups without ids stay all-or-nothing. Before each attempt (`apply-prefix`, which returns `{:file :tracked :refs}` or the cause) `pmap/*tracked*` and the `cpc/*state*` atom are reset to their pre-attempt values (at the end `apply-commits` sets those of the kept prefix): a pointer map already `modified?` keeps its id when changed again, so the failed version would be persisted. DB/IO/interrupt errors and `:not-found` fragments (`change-failure?`) fail the whole request, never reject a commit. A rejection whose ex-info has no `:type` reports `:internal :unexpected-error`. `app.http.errors` answers 503 (`:server-error` + code) for transient SQLSTATEs (classes 08/53, 57P01–3, 40001, 40P01), `SQLTransientException`, `InterruptedException` and only the `IOException`s that pass on their own (`transient-io-exception?`: `SocketException`, `InterruptedIOException` incl. socket timeouts, `UnknownHostException`, `HttpTimeoutException`), which the client's save retries (and GETs retry by themselves); any other `IOException` (EOF from a corrupt blob, Jackson parse errors, missing files) stays 500 `:io-exception`, as do other SQL errors. With the `:debug-save-faults` flag, a `:set-plugin-data` file change in namespace `:penpot-debug` keyed `save-fault:<kind>` raises that fault in `apply-changes` (used by `debug.failNextSave`/`commitInvalidChange`). With a schema check running (hard schema validation without skip-validate, or soft) and 2+ commits, the touched pages' objects are decoded on the original file first (`decode-objects`; referential validation alone reads few objects, so no pre-decode then): objects-map caches decoded values per instance and a derived map copies the cache, so otherwise every attempt re-decodes the page (~260 ms at 10k shapes). Validation step timings (`with-validation-timing`, upstream `penpot_rpc_update_file_validation_timing` + the `update-file` debug log) come from the first attempt only (retries run with a cfg without metrics and timings); the whole bisection is one `retry` step sample, and the rejection log carries `:attempts`/`:elapsed`.
|
||||
|
||||
@ -3,34 +3,12 @@
|
||||
JVM `clojure.test` (kaocha runner) under `backend/test/backend_tests/`.
|
||||
|
||||
- READ `mem:testing` FIRST — it defines the execution discipline (no piping, tee to file, preferred commands) that applies to all JVM test runs.
|
||||
- Test DB/Redis come from the `PENPOT_TEST_*` env vars already set in the shell (`PENPOT_TEST_DATABASE_URI`, `PENPOT_TEST_REDIS_URI`, ...); per instance they point at `penpot_test` / `penpot_test_wsN` and valkey DB `6+N` (`mem:devenv/core`). Do not override them with values copied from CI logs or other sources unless the user asks (e.g. "use a temporary database"); the CI env (`redis://redis/1`, `postgresql://postgres/penpot_test`) only works inside the CI docker network.
|
||||
- The test database may have migrations applied from newer versions (it is shared dev infra). Reinitializing it — `dropdb`/`createdb` of the `PENPOT_TEST_*` database only — is explicitly permitted when that blocks a test run; never drop the main `penpot` database or another instance's test database.
|
||||
- All CLI commands must be executed from the `backend/` subdirectory.
|
||||
- Tests are invoked directly via `clojure -M:dev:test` (kaocha) — there is no pnpm wrapper. Kaocha auto-discovers test namespaces, so no runner registration is needed.
|
||||
- **Preferred way to run them: `scripts/ci <module> --test`**, which runs lint, tests and format per module and writes a log per task under `.ci-logs/`. It exports `PENPOT_FLAGS` with `enable-backend-asserts` appended, so use it rather than calling `clojure -M:dev:test` by hand.
|
||||
- **Why the flag matters:** `clojure.core/assert` is elided at compile time when `*assert*` is false, and `app.config` only sets that var from the flags when it loads. The test fixture redefines the flags after that, so they never reach it: without the flag the suite runs with every assertion compiled out, and a test that depends on an assert only means something with it set. `scripts/ci` and `tests-backend.yml` export the flag (GitHub Actions sets it too). When running the suite directly — including a `--focus` run — the command is `PENPOT_FLAGS="enable-backend-asserts" clojure -M:dev:test`.
|
||||
- `clojure.core/assert` is elided at compile time unless `PENPOT_FLAGS` contains `enable-backend-asserts` (`app.config` sets `*assert*` from it). `scripts/ci` and `tests-backend.yml` export the flag; a bare `clojure -M:dev:test` (or `--focus` run) without it silently skips every precondition assert.
|
||||
- Coverage: if code is added or modified in `src/`, corresponding tests in `test/backend_tests/` must be added or updated.
|
||||
- Isolated run: `clojure -M:dev:test --focus backend-tests.my-ns-test` for a specific test namespace, or `clojure -M:dev:test --focus backend-tests.my-ns-test/my-test-var` for a specific test var. Keep the `PENPOT_FLAGS="enable-backend-asserts"` prefix on these too; a focused run without it hides assert-dependent failures exactly like a full one.
|
||||
- A **whole-suite** crash reported as `Execution error ... Assert failed: ...` with no `FAIL in` line is not a test failure: an `ig/assert-key` ran while the test system was being built, which happens outside `clojure.test`, so kaocha reports it as a load/build error and the summary has no counts. Fix the offending schema (see `mem:backend/subtleties`) rather than hunting for a test.
|
||||
- Multiple focus symbols are supported by REPEATING the flag (kaocha accumulates them, last-wins does not apply): `clojure -M:dev:test --focus backend-tests.ns-one --focus backend-tests.ns-two` — preferred over a shell loop of isolated runs.
|
||||
- Isolated run: `clojure -M:dev:test --focus backend-tests.my-ns-test` for a specific test namespace, or `clojure -M:dev:test --focus backend-tests.my-ns-test/my-test-var` for a specific test var.
|
||||
- Regression run: `clojure -M:dev:test` to ensure no regressions in related functional areas.
|
||||
- If you need to filter output, tee to a temp file first: `clojure -M:dev:test 2>&1 | tee /tmp/penpot-test-output.txt`.
|
||||
- RPC test helpers `command!`/`management-command!` split the data map: qualified keys become server params, unqualified keys become request body params. To inject request-level context (headers, `:app.http/auth-key-id`, ip), pass a map under `:app.http/request` metadata; non-map `IRequest` stubs fall back to a dummy request.
|
||||
- The test system is `main/system-config` + `main/worker-config` with the
|
||||
components that must not run in the background dropped: HTTP, the srepl,
|
||||
cron, the dispatcher and **every** `[<profile> :app.worker/runner]`
|
||||
(`helpers/remove-job-runners`). A live runner of any queue races the tests
|
||||
that drive the loop by hand (`wdisp/run-batch` + `@#'wrkr/run-worker-loop`):
|
||||
it takes the job the test just dispatched and completes it behind the test's
|
||||
back, so the test's own assertions read a state it did not produce. Do not
|
||||
re-add a runner by editing that list by hand; the removal matches by key
|
||||
shape on purpose.
|
||||
- `system-config` and `worker-config` are `def`s: their `(cf/get ...)` calls
|
||||
are evaluated when the namespace loads, against the real environment, while
|
||||
test code calling `cf/get` at run time sees the `penpot-test`-prefixed
|
||||
config. The two can disagree — in the devenv `:tenant` is `devenv-ws0` for
|
||||
the defs and `default` inside a test — so a component built from
|
||||
`worker-config` can talk to a different Redis key prefix than a cfg the test
|
||||
builds itself. The symptom is a component that silently does nothing
|
||||
locally and races in CI (which sets no `PENPOT_TENANT`).
|
||||
- `backend-tests.helpers` has `metrics-stub` (a metrics instance with no collectors, where every `run!` is a no-op), `init-climit` (a concurrency limiter built from an inline config map, with an optional second argument for the metrics instance to publish to) and `with-climit`, the macro that builds one, runs the body and halts it. `init-climit` starts the republisher daemon thread, so a test that builds a limiter without halting it leaks that thread for the rest of the JVM run: use `with-climit`. Log lines are capturable with `add-watch` on `app.common.logging/log-record` (it holds the last record; the watcher sees every one, but only of the levels the logger has enabled: a line the app writes at TRACE never gets there). The logger swaps the record on an executor thread, after the call that logged it returns: keep the watch installed until the record arrives (poll for it) and only then remove it, or the record lands on nobody under load.
|
||||
- `backend-tests.helpers` has `metrics-stub` (a metrics instance with no collectors, where every `run!` is a no-op), `init-climit` (a concurrency limiter built from an inline config map, with an optional second argument for the metrics instance to publish to) and `with-climit`, the macro that builds one, runs the body and halts it. `init-climit` starts the republisher daemon thread, so a test that builds a limiter without halting it leaks that thread for the rest of the JVM run: use `with-climit`. Log lines are capturable with `add-watch` on `app.common.logging/log-record` (it holds the last record; the watcher sees every one, but only of the levels the logger has enabled: a line the app writes at TRACE never gets there).
|
||||
|
||||
@ -1,44 +0,0 @@
|
||||
# Backend WebSocket Subscriptions
|
||||
|
||||
## Registry
|
||||
|
||||
- `app.http.websocket/state` holds `{:connections {conn-id → wsp} :by-profile {profile-id → #{conn-id}}}` in **one** atom, so a single `swap!` keeps both consistent; `register-connection`/`unregister-connection` maintain them together and drop a profile from the index once its last connection leaves.
|
||||
- Each connection's own `::ws/state` atom holds its subscriptions (`::file-subscription`, `::team-subscription`). `state` is shadowed by `::ws/state` in every `handle-message` destructuring, so only `on-connect`, `on-disconnect` and the `repl-*` helpers touch the registry — reading the registry from a handler is a bug.
|
||||
- `::profile-id` and `::session-id` on a connection live in the `app.http.websocket` namespace, while `::ws/id`, `::ws/state`, `::ws/output-ch`, `::ws/session-id` live in `app.util.websocket`. Mixing them up yields silently `nil` values (a `nil` `session-id` makes the channel's `:xf` drop **every** message).
|
||||
- The registry is local to one backend instance. The message bus is shared (`mem:prod-infra/core`). Anything that must reach connections owned by another instance has to go through the bus — closing a connection from an RPC handler only ever works in a single-backend deployment.
|
||||
|
||||
## Revocation model
|
||||
|
||||
A subscription is authorized once, at `:subscribe-file`/`:subscribe-team`, and that decision used to be trusted for the whole connection life (GHSA-m53j-2766-6jqw, medium). Two independent mechanisms now bound it:
|
||||
|
||||
- **Announcement (primary).** A mutation that can change access publishes on `app.rpc.notifications/internal-revocation-topic` (`notify-permissions-changed` for one profile, `notify-team-permissions-changed` for one team id — the watcher resolves the members itself, so a large team is one event, not N). The `::revocation-watcher` Integrant component in `app.http.websocket` subscribes and calls `revalidate-profile-subscriptions`, which looks the profile up in `:by-profile` and re-checks every subscription with `file-readable?`/`team-readable?`: existence (the file gate is the same `sql:get-minimal-file` as the `:subscribe-file` gate, made non-throwing via `{:db/throw-if-not-exists false}`; the team gate is `db/get*` with `::sql/columns [:id :deleted-at]` — `deleted_at` must be in the projection or the soft-delete filter is a no-op) AND the `has-read-permissions?` predicates — and `sql:file-permissions` additionally filters `deleted_at` on file, project and team, as do `sql:get-team-id-for-file`/`-project` plus a final team-row check in `resolve-team-id`, so the organization-owner fallback grants nothing through soft-deleted file/project/team. The existence gate closes that fallback (expelled org owner, deleted resource). The event is only the trigger; the fresh check is the decision, which is what keeps a downgrade to `viewer` (still readable) and a non-member organization owner on a live resource from being mistaken for a revocation. Both publishers defer through `db/after-commit`, because the watcher checks on another connection and would otherwise see the pre-commit state. `:subscribe-team` deliberately has no existence gate of its own; only the re-checks add one, matching the file side (user decision, PR #12189 review).
|
||||
- **Interval (safety net).** `start-relay` runs the relay as a `go-loop` that waits on the subscription channel and a one-shot tick channel together via `sp/alts!` with `:priority true` on the tick: when both hold something, the tick wins, so a steady stream of messages can never push the re-check away. The cadence is `subscription-revalidation-interval` (default `default-revalidation-interval`, 5 min, a code constant in `app.http.websocket` and **not** in the `config.clj` default map, same rule as session lifetimes). The deadline is fixed per cadence: a single tick channel covers a whole cadence and message turns reuse it (minting a fresh timeout per turn would both churn timers and starve under load exactly like an in-loop timer does); a new one is minted only after a tick fires. It covers what no announcement can: Nitrate transferring organization ownership (an HTTP call to another service) and rows deleted straight over SQL. Closing the channel ends the loop either way, because a take on a closed channel is always ready with nil — every way a subscription can end closes it, so a finished relay never keeps polling. Cost is three or four queries per subscription per interval (existence row + predicates), plus one Nitrate call per team per interval when the `:admin-console` flag is on (cached 30 s); one `timeout-chan` per subscription per cadence on promesa's default scheduler (one or two threads shared with every other scheduled task) — do not shorten the interval without sizing that. A tick abandoned at teardown (the subscription closed while it was pending) still fires its scheduled close harmlessly. Two places fail open, deliberately: `still-authorized?` (a transient DB error logs and keeps the subscription; `InterruptedException` is rethrown) and the `catch Throwable` in `watch-revocations`.
|
||||
|
||||
Which mutations announce: `delete-team-member`, `leave-team`, `update-team-member-role` (member only), `delete-team`, `delete-file`, `delete-project`, `move-files`, `move-project` (one team event each). Anything not listed here is covered only by the interval. `delete-share-link` deliberately does not: a share-link visitor never passes the websocket subscribe check, because `check-read-permissions!` is called without a `share-id` and that arity only considers team membership.
|
||||
|
||||
`close-file-subscription`/`close-team-subscription` are idempotent: no-op on a mismatched id, and they `dissoc` the subscription so a repeat call and `:pointer-update` do not publish to a dead topic. Closing the channel is what ends the relay (`take!` on a closed channel returns `nil`).
|
||||
|
||||
## Topic and layering
|
||||
|
||||
- `internal-revocation-topic` is a fixed uuid that is never a real resource id, so it cannot collide with a file, team or profile topic — and deliberately not the `uuid/zero` system topic, which is `sp/pipe`d straight into the client output and anything published there reaches every client. No client can subscribe to the internal topic: `:subscribe-file`/`:subscribe-team` only accept ids that already passed a permission check, and `:broadcast`/`:pointer-update` only republish to ids the server already holds.
|
||||
- The topic and the publish helpers live in `app.rpc.notifications`, not in `app.http.websocket`: the command namespaces cannot require the watcher, because `app.http.websocket` already requires `app.rpc.commands.files`/`teams`. The watcher keeps the registry, which is the part it owns.
|
||||
|
||||
## Time and channel traps in this namespace
|
||||
|
||||
- The relay waits with `sp/alts!` over `[tick-ch channel]` with `:priority true`, following the `msgbus.clj` pattern (`start-io-loop` destructures the same `[val port]` shape and compares ports with `identical?`). A take on a closed channel is always ready with nil, which is what ends the loop — and what makes closing the channel from any teardown path prompt even while parked in `alts!`. `sp/timeout-chan` is one-shot: it closes once after the delay, and one abandoned at teardown still fires its scheduled close harmlessly.
|
||||
- `ct/diff a b` returns **b − a** (`Duration/between`), so "time until a future instant" is `(inst-ms (ct/diff (ct/now) future))`.
|
||||
- `ct/duration` unit keywords are singular: `{:millis 150}`, `{:minutes 5}`, never `:milliseconds`/`:days`.
|
||||
- `still-authorized?` takes the check itself: `(boolean (check-access))`. Calling `(boolean pred)` on the closure is always true and silently disables the interval — this exact bug shipped once and no test caught it until the resubscribe-counting test existed. Naming rule here: no `?` for the action that queries and can fail (`check-access`), `?` only for the safe yes/no answer (`still-authorized?`).
|
||||
|
||||
## Tests
|
||||
|
||||
`backend/test/backend_tests/http_websocket_test.clj` drives the real Redis-backed msgbus, so the revocation tests are end-to-end: register the connection in the real registry, run the real command, wait for the subscription to drop, then assert no content arrives. A connection built without `register-connection` is invisible to the watcher, so the watcher-based tests must go through `open-registered-connection`. Interval tests shorten the config via `with-redefs` on `cf/get` rather than waiting minutes. `with-clean-registry` runs as an `:each` fixture, so no test can leak a live relay loop into the next one. `captured-events` forwards the keyword args through to the real `mbus/pub!` — rebuilding them positionally publishes to the topic `:message` with a `nil` payload and silently untests every other notice. Interval tests take access away with a direct row delete, never with a command: a command announces, so the test would pass even with the interval removed. The same rule covers the two counting/fan-out tests: `revocation-closes-subscriptions-that-are-no-longer-authorized` takes access away with `delete-team-membership!` (raw row delete, publishes nothing) because a command announcement lets the live watcher close both subscriptions before the test's own `revalidate-profile-subscriptions` call, and its `(= 2 …)` count then reads 0; `team-event-closes-every-member-subscription` soft-deletes the team row (`sql:team-members` joins `team_profile_rel` and `profile` only, so the watcher still resolves every member) and asserts nothing closed before one synthetic `notify-team-permissions-changed`, leaving the command path to `deleting-a-team-announces-once` and `deleting-a-team-closes-open-subscriptions`. The resubscribe-counting test is the one that proves finished relays stop polling. The soft-delete + org-owner re-check tests pair a direct row soft-delete with `with-redefs` on `nitrate/organization-owner-of-team?`: before the existence gate, the org-owner fallback kept the subscription open on a deleted resource. The soft-deleted-project/team-closes-org-owner-file-subscription tests pin the root fix in `app.rpc.permissions` (`resolve-team-id` refuses dead file/project/team for every caller of the fallback). `watcher-consumes-revocation-events-from-the-bus` takes access away with a direct row delete and drives the watcher with a synthetic bus event — never a command and never a second `notify-*` — so the "still subscribed" assertion cannot race an after-commit hook and a broken command announcement cannot mask a broken watcher. `moving-a-project-closes-source-team-subscriptions` pins the move-project asymmetry: the file subscription drops, the source-team subscription survives (the member is still a member).
|
||||
|
||||
`revocation-watcher-schema-resolves-every-ref` covers the `::revocation-watcher` params schema, which no other test compiles. It existed broken for a while: the three injected keys were written as bare keywords, which is a valid malli *property* position and not a key, so the schema raised `:malli.core/invalid-schema` — invisible unless `PENPOT_FLAGS=enable-backend-asserts` was set, and a crash of the whole system build rather than one test (see `mem:backend/testing` on the flag).
|
||||
|
||||
## See also
|
||||
|
||||
- `mem:backend/auth-permissions-product-domains` for the permission predicates and the not-found semantics that the re-checks reuse.
|
||||
- `mem:prod-infra/core` for why the bus crosses backends and the registry does not.
|
||||
- `mem:frontend/routing-app-shell-subtleties` for the client side: the frontend already reacts to `team-membership-change` by navigating away, which is why revocation needs no new client message.
|
||||
- `.agents/plans/2026-10-07-websocket-subscription-revocation.md` for the original plan, and the sibling `.agents/plans/2026-10-05-leave-team-file-access-cleanup.md` whose hardened permission queries this re-verification depends on (without live team membership required in `sql:file-permissions`, the re-check would keep reporting access for a member who already left).
|
||||
@ -8,7 +8,6 @@ How to shape a function in this codebase. Each rule came from a review round; do
|
||||
- **Functions that obtain data start with `get-` or `fetch-`.** A reader must see at a glance that the function reads (`get-file`, `get-snapshot`, `get-file-json`); a bare noun hides it (`file-json` says nothing about reading). New helpers that fetch by id follow this, never a `current-` prefix: the id parameter already says which one it is.
|
||||
- **`?` marks a predicate, not a data prop.** A var or function that answers a question takes `?` (`export-error?`, `(defn editable? …)`); a map key that names a property does not (`:export-error`, never `:export-error?`), even when its value is a boolean. Existing `?` keys are legacy to remove gradually; do not add a new one.
|
||||
- **Do not shape production code for tests.** No return values added just to assert them; assert on the observable effect. When preconditions move, move the tests with them.
|
||||
- **All code-facing text MUST be in English** — identifiers, comments, docstrings, test descriptions. No exceptions; see the hard rule in `AGENTS.md`.
|
||||
|
||||
Reference implementation: the http metrics samplers in `app.http` (`sample-worker-metrics` / `sample-connector-metrics` are guard-free; `sample-http-metrics` guards at the boundary).
|
||||
|
||||
|
||||
@ -8,8 +8,5 @@ Behaviors confirmed against the language/stdlib — do not re-derive from assump
|
||||
- The analyzer reads `:async` only from the fn name meta and the `fn` operator meta; list-level meta is ignored (would make a MetaFn). `(fn ^:async [] …)` puts the meta on argv (pre/post only, NOT async).
|
||||
- Valid: `(defn ^:async f)`, `(defn- ^:async f)`, `(^:async fn [] …)` (the latter is what stock `t/async` generates itself).
|
||||
- `^:async` fns never throw synchronously (rejected promises instead); `try/catch/finally` supported; continuations are microtasks, timers and RxJS schedulers are macrotasks (FIFO).
|
||||
- Inside an `^:async` fn, a nested call that needs a temp binding (e.g. calling a local fn `(g (f x))`, whose arg gets a `G__` binding) compiles to `(await (async function(){…})())`: `await` unwraps a returned promise, so `(settled (upload (make-ctx)))` blocks until the upload promise settles instead of passing the promise on (a test that pushes the answer later hangs, idle). Bind each step in its own `let` binding.
|
||||
- `!` suffix allowed cases (only exceptions to the no-new-`!` rule in `mem:clojure/design-rules`): mutating public `app.db` API (`insert!`, `update!`, `delete!`, `run!`, `tx-run!`, `exec!`, ...) and fns whose contract is to work over transients (stdlib `conj!`, `assoc!`, `dissoc!`, `disj!`, `persistent!` and custom helpers over them); any further case must be registered here with justification before use.
|
||||
- Penpot extends `clojure.core/Inst` in `app.common.time` ( JVM-only section), so `inst-ms` also accepts `Duration` (returns `.toMillis`, NOT epoch millis), `FileTime` and `OffsetDateTime`. It only works where that namespace is loaded; on a bare `Duration`, prefer explicit `.toMillis`, which reads as what it is (an amount, not an instant).
|
||||
- The threading macros insert the threaded value at DIFFERENT positions: `->` and `some->` put it FIRST (`(-> x (f a))` → `(f x a)`), `->>` and `some->>` put it LAST (`(->> x (f a))` → `(f a x)`). `some->`/`some->>` also short-circuit on nil. With unary calls both nest the same way (`(-> x (f) (g))` and `(->> x (f) (g))` are both `(g (f x))`): what changes between them is the position inside each call, not the order of the steps. A pipeline that calls a fn whose first argument is a config/context (`app.features.fdata/resolve-file-data` is `(cfg, file)`) must use `->>`/`some->>`; with `->`/`some->` the arguments are swapped and the call silently returns the wrong value (it cost a broken `get-snapshot` in `app.features.file-snapshots`).
|
||||
- `cuerdas.core/index-of` returns `nil` (not `-1`) when the substring is absent, like `clojure.string/index-of`. `(pos? nil)` **throws on the JVM** but evaluates to `false` in CLJS (JS coercion), so `(if (pos? (str/index-of s "\n")) …)` works in the browser and crashes the backend. Guard with `some?` before `pos?` (this was `app.common.exceptions/first-line`).
|
||||
|
||||
@ -1,7 +0,0 @@
|
||||
# Clojure Logging Conventions
|
||||
|
||||
Structured logging through `app.common.logging` (`l/trc`/`l/dbg`/`l/info`/`l/wrn`/`l/error`/`l/fatal`): `:hint` carries the human-readable message, every other datum goes in its own keyword field (`:id`, `:container`, `:removed`, ...).
|
||||
|
||||
- `:hint` values start lowercase: `"fix dangling children"`, not `"Fix dangling children"`. Uppercase-first hints exist as legacy (~57 occurrences, the densest in `backend/src`; also `l/trc` in migration `legacy-28` and `log/error` in `app.common.files.repair`) — do not copy them, and do not mass-rewrite them unless the surrounding code needs the change.
|
||||
- A name that requires uppercase (an env var like `PENPOT_MEDIA_PROCESSING_SERVICE_URI`, an acronym like `MCP` or `LDAP`) stays uppercase inside an otherwise lowercase sentence.
|
||||
- Repair work done by a file migration logs at `l/trc`, only when it changes something, and with the ids it touched: the house style of `migrations.cljc` (`legacy-28` precedes; the file has no `warn`/`info` lines). The shipped `log4j2.xml` pins the `app` logger to `info` (static file, `monitorInterval="60"`), so those trace lines appear only after the operator raises the level; the durable record of a repaired file is the migration id in its `:migrations`.
|
||||
@ -1,9 +0,0 @@
|
||||
# Promesa
|
||||
|
||||
Promise library of the monorepo (frontend CLJS and common, `funcool/promesa` 12.0.1 in `common/deps.edn`). Its API has two call styles and mixing them in one expression is the trap: `fmap`/`mcat`/`merr` are **fn first, promise last** (`->>` friendly) and `then`/`then'`/`catch`/`handle` are **promise first, fn last** (`->` friendly). Convention: `fmap`/`mcat`/`merr` inside `->>` pipelines, `then`/`catch` with `->`. A `then` written inside a `->>` reads backwards and hides whether the step carries a value or a promise.
|
||||
- `fmap` (alias `map`, both `(f p)`) is functor map: `f` returns a plain value.
|
||||
- `mcat` (alias `mapcat`, both `(f p)`) is monadic bind: `f` must return a promise, and the result mirrors it.
|
||||
- `then` also unwraps a promise returned by `f`, which is why it covers both cases at the cost of not naming which one it is; `then'` maps without unwrapping.
|
||||
- `merr` (fn first) and `catch` (promise first) are the error side.
|
||||
- `all` takes a collection of promises (a lazy seq works); `let` chains sequential bindings.
|
||||
- Seam with the rx pipelines of the frontend: `beicon.v2.core/from` builds an observable from "arrays, collections, and promise instance" (`external/beicon`), so `(->> p (p/mcat f) (rx/from) (rx/map g))` is the usual bridge (`app.main.data.imports`, `app.util.zip`).
|
||||
@ -26,12 +26,6 @@ Respect the intended abstraction direction in new/refactored code:
|
||||
|
||||
Some legacy code violates this layering; do not copy those violations into new code when a focused refactor is practical.
|
||||
|
||||
## Build-mode dependent laziness
|
||||
|
||||
`app.common.data.macros/lazy-delay` returns a delay whose expression is evaluated eagerly by default. Only the builds that set the `app.common.data.macros/lazy-delay?` closure define to `true` defer it to the first deref; every production build does (frontend `main` and `worker`, exporter, library), so the Closure compiler replaces the constant and drops the eager branch. On the JVM it is always eager.
|
||||
|
||||
`app.common.schema/check-fn` resolves its schema through it: an invalid schema definition fails when the namespace loads in development and in tests, and only on the first check call in production. The other lazily built parts of `check-fn` (validator and explainer) are always deferred.
|
||||
|
||||
## Focused memory routing
|
||||
|
||||
Model, schema, and persistence shape:
|
||||
|
||||
@ -6,7 +6,7 @@
|
||||
- Do not treat nil as a distinct persisted state from absence. Import/export and cleanup paths may filter nil attrs away.
|
||||
- Avoid Clojure-special naming in exported object attrs, especially boolean names ending in `?`; exported/imported data must survive JSON/SVG/Transit and external tooling.
|
||||
- Any new shape attr that participates in component sync must be listed in `app.common.types.component/sync-attrs` with the correct touched group. Attrs absent from `sync-attrs` are ignored by component synchronization.
|
||||
- Shape `:svg-attrs` keys are kebab-case (`fill-rule`, not `fillRule`): birth (`app.common.svg/attrs->kebab-props`, applied everywhere data lands on a shape) normalizes any outside spelling through the `svg-props` whitelist, and render boundaries (`get-svg-props`, `render-xml*`, `svg-node*`) translate back to camelCase for React. `:svg-defs` node attrs follow the same rule recursively; svg-raw `:content` attrs are kebab-ized spelling-only, never filtered — the sidebar reads `:stroke-style`, which is outside the whitelist. Non-spec attrs Penpot keeps live in `penpot-extra-attrs` (seeded with `:stroke-style`) and travel in source spelling. Excluded, each with its reason: `:tag` values (React and tag sets need them verbatim; the schema decoder restores keywords from stored strings), outer `:svg-defs` keys (id strings at birth, keywords or strings after JSON decode — accept both), all values (the kebab reader never touches values). Repair sites: birth plus migration `0035-fix-shape-svg-attrs-and-defs` (replaces the withdrawn `0021`); the binfile cleaner carries no svg repair.
|
||||
- Shape `:svg-attrs` is the one map whose keys are camelCase React prop names (`fillRule`, not `fill-rule`), because the SVG import path runs them through `app.common.svg/attrs->props`. The `.penpot` reader (`app.common.json/read-kebab-key`) rewrites every json key of every entry to kebab-case, nested maps included, so any code that decodes a binary file must run `:svg-attrs` back through `attrs->props` or the names are lost. `:svg-defs` and the `:content` tree of `svg-raw` shapes keep their source names and round-trip on their own.
|
||||
|
||||
## Cross-module update checklist
|
||||
|
||||
|
||||
@ -52,8 +52,3 @@ its own drift. Comparing `:selrect.width` from shape A to
|
||||
`:selrect.width` from shape B is comparing two values that
|
||||
"semantically" should be equal but were computed via different
|
||||
operation chains — exact equality will often be false.
|
||||
|
||||
## Library sync comparison
|
||||
|
||||
- `update-attrs` (`app.common.logic.libraries`) compares main vs copy attrs exactly first, then with tolerance for: `drifting-geometry-attrs` (`:x :y :width :height :selrect :points`, via `gsh/close-attrs?`); path and bool `:content` (`segment/content-close?`; bool shapes keep their path in `:content`); and text `:position-data` (`position-data-close?` over `mth/close-map?`: every numeric field of every entry within `mth/close?`, other fields such as fills, fonts and text exact). All other attrs (`:opacity`, radii, `:transform`, `:rotation`, text `:content`) stay exact: tolerance there hides real changes.
|
||||
- `close-attrs?` pairs lists with `d/zip`, which truncates to the shorter list and treats nil vs vector as equal for `:points`/`:position-data`. Do not use it for variable-length data. That is why `:position-data` has its own helper, and why `attr-equal?` checks the `:points` count before calling it.
|
||||
|
||||
@ -20,10 +20,8 @@
|
||||
|
||||
- Full referential/semantic validation currently runs only when file features contain `"components/v2"`.
|
||||
- Validation starts at root plus orphan shapes, then validates component records. `validate-file!` raises `:validation :referential-integrity` with collected details.
|
||||
- `update-file` validates per save with the change-scoped variants: `validate-file-affected!` (referential, whole touched pages/components) and `validate-file-schema-affected!` (file schema with `:pages-index` reduced to the touched pages; non-page data checked whole). The schema variant maps change types to pages itself (`touched-page-ids`): a new change type that writes a page must be added there, or to `file-level-change-types` if it touches no page; an unknown type makes it check the whole file. The whole-file schema check loads and decodes every page, ~75–80% of a save on multi-page files.
|
||||
- `repair-file` does not mutate data directly; it reduces validation errors into redo changes using `changes-builder`. Callers must apply or persist those changes.
|
||||
- `repair-file-until-valid` stops on: no errors; unchanged `:data` after apply (backend page/`PointerMap` `=` is pointer identity); stable error fingerprint `(frequencies (map #(dissoc % :shape) errors))` vs previous pass (check fingerprint before `repair-file`/`process-changes`); or `max-iterations`. Each `repair-error` builds from the original `file-data`, so N `:set`-based errors on one shape take N passes.
|
||||
- `:ref-shape-not-found` repair: when the copy root's local component has a `:main-instance-id` that resolves to no shape but a `:main-instance` shape of that component sits on its `:main-instance-page`, it emits `mod-component` pointing the id back to that main (keeps the copy linked; a repair scoped to the copy's page never sees the main's `:invalid-main-instance-id`). Otherwise near-main reassign, else detach the copy subtree.
|
||||
- `:missing-slot` fires only for a REAL swap: a copy sub-head whose `shape-ref` is no longer a child of the near main parent. A pure positional mismatch (ref still a sibling elsewhere) is a reorder — valid, realigned by the async component sync; do not "repair" it by assigning swap slots (a slot freezes the child out of normal sync). `fix-missing-swap-slots` (migration 0019) follows the same membership rule.
|
||||
|
||||
## Migrations
|
||||
@ -32,6 +30,4 @@
|
||||
- Migrations are an ordered set mixing legacy version-derived ids and newer named ids. Keep append order stable; `migrate` applies the set difference between available migrations and file migrations.
|
||||
- `migrate-file` synthesizes legacy migration ids from old numeric versions when `:migrations` is absent, migrates legacy features, and records feature flags created through `cfeat/*new*`.
|
||||
- When a file had no previous `:migrations`, `migrate-file` marks all migrations as migrated in metadata so callers persist the complete migration set, not only transformations that changed data.
|
||||
- Closed schemas (`schema:stroke-attrs` and others) reject a key that was dropped from the declaration, so removing a declared attr needs a migration in the same change, for data already written with it.
|
||||
- Rebase collision on a migration number: upstream keeps the id. Renumber the branch migration to the next free one, keep it last in `available-migrations`, and rename everything that spells the number — its test file and namespace (`files_migrations_00NN_test.cljc`), the test names, and comments in production code or memories that name it.
|
||||
- Withdrawing a migration id: remove both the `defmethod` and the id from `available-migrations`; the replacement migration subsumes it and files that recorded the old id re-migrate through the replacement (persisted `:migrations` keep the old id via union — do not rewrite them, including playwright fixture JSONs). Idempotent/pure-cleanup migrations only; never withdraw one whose transform must not run again unless the replacement is safe for those files. Precedents: "0021-fix-shape-svg-attrs" -> "0035-fix-shape-svg-attrs-and-defs"; "legacy-36" -> "0036-clear-nil-keys-in-objects".
|
||||
- Closed schemas (`schema:stroke-attrs` and others) reject a key that was dropped from the declaration, so removing a declared attr needs a migration in the same change, for data already written with it.
|
||||
@ -38,10 +38,6 @@ Useful builders:
|
||||
|
||||
`add-variant-with-copy` does not accept position params for children; use `gsh/absolute-move` after creation if positions matter.
|
||||
|
||||
## Generative sampling caution
|
||||
|
||||
- `sg/sample` is unseeded per run. If a test filters samples by type (e.g. keep only `:svg-raw`) then `first` can yield nil on unlucky seeds; `assoc` onto nil builds a typeless map that fails `:multi` dispatch on `:type`. Pin `:seed` in `sg/sample` opts whenever a test depends on a filtered subtype being present (verified seed 42 deals svg-raw shapes on JVM and JS).
|
||||
|
||||
## Driving production paths
|
||||
|
||||
For shape mutations, prefer production-path helpers such as `cls/generate-update-shapes` plus `thf/apply-changes`. For component swaps with keep-touched behavior, use `tho/swap-component-in-shape` with `{:keep-touched? true}`.
|
||||
|
||||
@ -6,10 +6,8 @@ You are working on the GitHub project `penpot/penpot`, a monorepo.
|
||||
- A section's top-level memory is `<section>/core`. When a section is relevant, read the core memory before focused memories.
|
||||
- Edits/stale refs/duplication cleanup: `mem:memory-maintenance`.
|
||||
- Cross-cutting testing principles, TDD workflow, and anti-patterns: `mem:testing`.
|
||||
- Structured logging conventions (`:hint` casing and fields, log levels): `mem:clojure/logging` — read before adding or changing any log call.
|
||||
- Verified Clojure language behaviors that contradict common assumptions (e.g. `int?` covers `Long`; `integer?` is the general predicate): `mem:clojure/idioms` — read before assuming stdlib predicate semantics.
|
||||
- Read `mem:clojure/design-rules` (function shape, boundary checks, naming) before writing or modifying Clojure/ClojureScript code — not only when designing a solution or an API.
|
||||
- Promises: which call style to use where (`fmap`/`mcat`/`merr` vs `then`/`catch`) and how they bridge to rx: `mem:clojure/promesa` — read before chaining promises or mixing them with observables.
|
||||
|
||||
# Development workflow
|
||||
|
||||
@ -17,12 +15,10 @@ You are working on the GitHub project `penpot/penpot`, a monorepo.
|
||||
- Before `git commit` → `mem:workflow/creating-commits` (subject/body format, 80-char body wrapping enforced by `scripts/check-commit`, `AI-assisted-by: model-name` trailer)
|
||||
- Before `gh issue create` → `mem:workflow/creating-issues` (title derivation, body template, labels, Issue Type)
|
||||
- Before `gh pr create` / `gh pr edit` → `mem:workflow/creating-prs` (title format, body structure, "Note:" line)
|
||||
- Before updating `CHANGES.md` → `mem:workflow/updating-changelog` (sources of truth, entry rules, the `:boom:`/`:rocket:` label contracts, pre-flight checks, anomaly report)
|
||||
- Before a repo-wide pnpm version update → `mem:workflow/updating-pnpm` (workspace layout, `scripts/sync-pnpm-version` flow, the stamp-missing-field and ignored-builds gotchas, verification steps)
|
||||
- Before a Node version update → `mem:workflow/updating-node` (the six pin sites in `.nvmrc`, the devenv and image Dockerfiles and the plugins-deploy workflows, the DHI tag and digest rules, what to verify)
|
||||
- Before bumping a non-JS dependency pinned in `docker/images/` (JDK, ImageMagick) → `mem:workflow/docker-images`
|
||||
- Before a Node version update → `mem:workflow/updating-node` (the six pin sites in `.nvmrc`, the devenv and image Dockerfiles and the plugins-deploy workflows, the DHI tag gotcha, what to verify)
|
||||
- **Never `git push`, force-push, or modify `git origin`** (or any other remote). The user pushes from their own shell; if a push is required, say so and wait. Never amend a commit that the user has already pushed unless explicitly asked.
|
||||
- **Never edit `CHANGES.md` by hand.** The changelog is generated from GitHub milestones during the release process; update it only via `mem:workflow/updating-changelog` (the `update-changelog` skill flow) or on explicit user request.
|
||||
- **Never edit `CHANGES.md` by hand.** The changelog is generated from GitHub milestones during the release process; update it only via the `update-changelog` skill flow or on explicit user request.
|
||||
- You have access to the GitHub CLI `gh` or corresponding MCP tools.
|
||||
- Issues are also managed on Taiga. Read issues using the `read_taiga_issue` tool.
|
||||
- Before writing code, analyze the task in depth and describe your plan. If the task is complex, break it down into atomic steps. After making changes, run the applicable lint and format checks for the affected module before considering the work done (per example `mem:backend/core` or `mem:frontend/core`).
|
||||
@ -62,11 +58,11 @@ The memory is structured in a way that you can get the critical information abou
|
||||
- `scripts/taiga.py` — Fetch public issues, user stories, and tasks from the Penpot Taiga project without authentication. See `mem:scripts/taiga`.
|
||||
- `scripts/gh.py` — GitHub operations helper: list milestone issues, fetch PR details, compare against CHANGES.md. Requires `gh` CLI. See `mem:scripts/gh`.
|
||||
- `scripts/project-anomalies.py` — Main-board anomaly check for a milestone (`check` writes `tmp/<MILESTONE>-ANOMALIES.md`). Bulk resolution via `scripts/gh.py`. See `mem:scripts/project-anomalies`.
|
||||
- `scripts/changelog.py` — Changelog checks for `mem:workflow/updating-changelog`: `boom` (generate the `:boom:` section from `breaking change` issues), `check-merged` (PR merge status), `cross-ref` (milestone PRs vs changelog section), `report` (anomaly report to CHANGES-ISSUES.md). Calls `scripts/gh.py` via subprocess. See `mem:scripts/changelog`.
|
||||
- `scripts/changelog.py` — Changelog checks for the `update-changelog` skill: `check-merged` (PR merge status), `cross-ref` (milestone PRs vs changelog section), `report` (anomaly report to CHANGES-ISSUES.md). Calls `scripts/gh.py` via subprocess. See `mem:scripts/changelog`.
|
||||
- `scripts/error-reports.mjs` — Query error reports via RPC API with token authentication. Supports list/get operations with filtering and pagination. See `mem:scripts/error-reports`.
|
||||
- `scripts/clean-node-modules` — Remove stale `node_modules` from all pnpm workspaces (root, modules, member packages). Keeps the shared pnpm store at `<repo>/.pnpm-store` unless `--store`; ignores `external/` and `.opencode/`. Usage and reinstall steps: `mem:workflow/updating-pnpm`.
|
||||
- `scripts/setup-opencode` — Install the opencode CLI in-container as a global pnpm package in `$PNPM_HOME`. Not part of the devenv image; see `mem:devenv/core`.
|
||||
- `scripts/ci` — CI orchestration script: runs lint, tests, and format checks per module, plus the production build on request (`--build`). Modules: `frontend backend common render-wasm exporter mcp plugins library media-processor`. Logs go to `.ci-logs/`; read the log file on failure. It is the preferred way to run the backend tests, and it exports `PENPOT_FLAGS` with `enable-backend-asserts`, which is what makes `clojure.core/assert` exist in the compiled test code. See `mem:scripts/ci`.
|
||||
- `scripts/ci` — CI orchestration script: runs lint, tests, and format checks per module (`frontend backend common render-wasm exporter mcp plugins library`). Logs go to `.ci-logs/`; read the log file on failure. See `mem:scripts/ci`.
|
||||
|
||||
# Dependency graph
|
||||
|
||||
|
||||
@ -5,7 +5,7 @@ Compose-based dev environment under `docker/devenv/`, driven by `manage.sh`. Par
|
||||
## Compose project layout
|
||||
|
||||
- `penpotdev-infra`: shared `postgres`, `rustfs`, `valkey`, `mailer`, `ldap`. File: `docker-compose.infra.yml`.
|
||||
- `penpotdev-wsN` (N=0,1,…): per-instance `main` + `redis` (Valkey). File: `docker-compose.main.yml`. ws0 (a.k.a. `main`) binds `$PWD`; ws1+ bind clones at `${PENPOT_WORKSPACES_DIR}/wsN/` (default `${XDG_DATA_HOME:-~/.local/share}/penpot-devenv/workspaces/`, or the former `~/.penpot/penpot_workspaces/` when it already exists), maintained by the developer.
|
||||
- `penpotdev-wsN` (N=0,1,…): per-instance `main` + `redis` (Valkey). File: `docker-compose.main.yml`. ws0 (a.k.a. `main`) binds `$PWD`; ws1+ bind clones at `${PENPOT_WORKSPACES_DIR}/wsN/` (default `~/.penpot/penpot_workspaces/`), maintained by the developer.
|
||||
- Optional overlay `docker-compose.opencode.yml`: added by `instance-compose` as an extra `-f` only when `PENPOT_OPENCODE_CONFIG_DIR` is set (i.e. `run-devenv --opencode-config-dir DIR` ran in this process). Bind-mounts the host dir at `/home/penpot/.config/opencode` (`:z`). Flag-only, per-call; not read from ambient env. Parser `parse-opencode-config-dir` absolutizes (`~`, realpath) because compose resolves relative bind sources against the compose file's dir. Only instances brought up with the flag get the mount.
|
||||
- All projects join external network `penpot_shared`. Created idempotently by `ensure-devenv-network`, never removed by lifecycle commands.
|
||||
|
||||
@ -15,7 +15,6 @@ Compose-based dev environment under `docker/devenv/`, driven by `manage.sh`. Par
|
||||
- For ws1+, `instance-env-overrides` computes the per-instance overrides (container/volume names, host ports offset `10000·N`, `PENPOT_PUBLIC_URI`, `PENPOT_REDIS_URI`, `PENPOT_TEST_DATABASE_URI`, `PENPOT_TEST_REDIS_URI`) and `instance-compose` injects them as env vars at compose time — never written to disk, recomputed each call so they can't drift. ws0 uses `defaults.env` as-is.
|
||||
- Per-instance test databases are created by `ensure-instance-test-database` (called from `start-instance`): `CREATE DATABASE` guarded by a `pg_database` existence check, so it is idempotent. Needed because `postgresql_init.sql` only runs on first volume creation. The backend applies migrations itself on first use, so an empty database is enough.
|
||||
- `backend/scripts/_env`: backend-internal only — secret keys, `PENPOT_FLAGS` (with `enable-backend-worker` gated on `PENPOT_BACKEND_WORKER`), `JAVA_OPTS`, `setup_s3_bucket()`. Never duplicates `defaults.env`.
|
||||
- Frontend admin-console is off by default in devenv: `docker-compose.main.yml` sets `PENPOT_DEVENV=true`, which makes `app.config` (`goog-define devenv-mode`) prepend `disable-admin-console` ahead of the user `config.js` flags; flags parse in order and user flags win, so `enable-admin-console` in `frontend/resources/public/js/config.js` re-enables it. Outside devenv (prod builds) the default is unset and the admin-console stays on, matching `flags/default`.
|
||||
- Compose files use pure `${VAR}` substitution; missing var = compose fails.
|
||||
|
||||
## Invariants
|
||||
@ -30,7 +29,7 @@ Compose-based dev environment under `docker/devenv/`, driven by `manage.sh`. Par
|
||||
|
||||
## Worker policy
|
||||
|
||||
Backend workers run only on ws0. `_env` gates `enable-backend-worker` on `PENPOT_BACKEND_WORKER`; ws1+ inject it as false. Workers are pure fire-and-forget: `jobs/submit` inserts a row into the shared Postgres `job` table and returns (the legacy `task` table stays dormant); RPC handlers never wait on completion and workers never publish to msgbus. The reason for "ws0 only" is avoiding multi-instance worker races (cron dedup is best-effort across instances, `jobs/submit` `dedupe` is racy across submitters); details in `mem:prod-infra/core`.
|
||||
Backend workers run only on ws0. `_env` gates `enable-backend-worker` on `PENPOT_BACKEND_WORKER`; ws1+ inject it as false. Workers are pure fire-and-forget: `wrk/submit!` inserts a row into the shared Postgres `task` table and returns; RPC handlers never wait on completion and workers never publish to msgbus. The reason for "ws0 only" is avoiding multi-instance worker races (cron dedup is best-effort across instances, `wrk/submit!` `dedupe` is racy across submitters); details in `mem:prod-infra/core`.
|
||||
|
||||
Each workspace is independent and can be started/stopped in any order. Shared infra (Postgres, RustFS, etc.) is shut down only when no instances remain running.
|
||||
|
||||
@ -78,42 +77,3 @@ No `--delete` on the working-tree pass: gitignored caches in the workspace survi
|
||||
- `start-devenv` / `log-devenv` / `drop-devenv`: legacy paths around ws0 + shared infra. `drop-devenv` never removes volumes.
|
||||
|
||||
`exporter/scripts/run` and `wait-and-start.sh` source `backend/scripts/_env` then `_env.local` if present.
|
||||
|
||||
## Image and dependency pins
|
||||
|
||||
Two Dockerfile trees, both multi-arch (`linux/amd64,linux/arm64`):
|
||||
`docker/devenv` (dev) and `docker/images` + `docker/imagemagick` (prod).
|
||||
|
||||
- Hand-pinned versions, one source each:
|
||||
- `docker/devenv/Dockerfile`: `NODE_VERSION` (+ per-arch `NODE_SHA256`),
|
||||
`PNPM_VERSION` (+ `PNPM_SHA256`), `CADDY_VERSION`, `CLOJURE_VERSION`,
|
||||
Zulu JDK URL + `ESUM`, `RUSTUP_VERSION`/`RUST_VERSION`/
|
||||
`EMSCRIPTEN_VERSION`, `CLJKONDO_VERSION`, `BABASHKA_VERSION`,
|
||||
`CLJFMT_VERSION`, `PIXI_VERSION`, `GITHUB_CLI_VERSION`,
|
||||
`AWS_CLI_VERSION` (+ `BINARY_SHA256`), `UV_VERSION`, `SERENA_VERSION`,
|
||||
`postgresql-client-18`, and the `penpotapp/imagemagick:<tag>`
|
||||
`COPY --from`.
|
||||
- `docker/images/Dockerfile.backend`: Zulu JDK URL + `ESUM`. The `jlink`
|
||||
`--add-modules` list must keep `jdk.management`
|
||||
(`mem:backend/subtleties`).
|
||||
- `docker/images/Dockerfile.{exporter,mcp,media-processor}`: `ARG
|
||||
PNPM_VERSION` (+ arch SHAs, same values as devenv).
|
||||
- Node and pnpm must agree in five places: `.nvmrc`, the devenv tarball pin,
|
||||
the two `plugins-deploy-*` workflows that set `runtime: node@<version>`, and
|
||||
the four Dockerfile pins. pnpm procedure: `mem:workflow/updating-pnpm`.
|
||||
- `dhi.io` base images are pinned `tag@sha256:<index digest>`; Renovate moves
|
||||
them (`penpot-renovate[bot]`, commits titled `:arrow_up: Update DHI base
|
||||
images`). Tag and digest move together: a new tag with the old digest still
|
||||
pulls the old image, and the digest must be the multi-arch index, not a
|
||||
per-platform one.
|
||||
- Every hand-pinned checksum is per architecture; one wrong value breaks half
|
||||
of the multi-arch build.
|
||||
- `SERENA_UPDATE_VERSION` in `docker-compose.main.yml` must never be lower than the Dockerfile's `SERENA_VERSION`. Any value makes `start-tmux.sh` install that Serena release on start, overriding the version baked into the image; a stale value silently downgrades Serena (it once replaced 1.7.0 with 1.6.1). If the image already has that version the install is an offline no-op; otherwise it downloads on every start (`run-devenv` recreates the container) and, offline, fails after ~25s and keeps the image version. Bump both together; empty means use the image version.
|
||||
- Infra images live in `docker-compose.infra.yml`: `rustfs/rustfs` (owns the
|
||||
S3 data volume), `axllent/mailpit` (SQLite in a volume), `valkey/valkey`,
|
||||
`postgres:18.6`, `rroemhild/test-openldap`. Keep the volume names stable
|
||||
when bumping them; replacement services get new volumes and leave the old
|
||||
data untouched.
|
||||
- Build locally: `docker build -f docker/devenv/Dockerfile docker/devenv/`
|
||||
and `cd docker/images && ./build.sh <image> --local`. CI:
|
||||
`build-docker-devenv.yml`, `build-docker.yml`.
|
||||
|
||||
@ -1,106 +0,0 @@
|
||||
# Auth Events and the Session Origin
|
||||
|
||||
`signin` is the frontend `identify` event emitted once per session from
|
||||
`logged-in` (`app.main.data.auth`). It carries the profile plus a closed
|
||||
vocabulary that says why the session started.
|
||||
|
||||
## Vocabulary
|
||||
|
||||
- `source`: `login` | `register`. Always present.
|
||||
- `auth-backend`: `penpot` | `ldap` | `google` | `github` | `gitlab` | `oidc`.
|
||||
Same key the profile already uses. Always present.
|
||||
- `activation`: `immediate` | `email-verification`. Only on signups.
|
||||
- `from-invitation`: boolean. Always present in `signin`
|
||||
(`(some? (:invitation-token profile))`); `auth-origin` omits it when `nil`.
|
||||
|
||||
`auth-origin` (public, validated against `schema:auth-origin`) drops the nil
|
||||
fields and checks with `(assert (sm/check ...))`: it throws in development and
|
||||
tests, and `:elide-asserts true` removes it from the release build, so a bad
|
||||
value can never break a real login. Emitters never call it directly.
|
||||
|
||||
## Where each value comes from
|
||||
|
||||
- `source` and `activation` travel as profile metadata (`::ev/source`,
|
||||
`::ev/activation`), written by `origin-meta`. `login-from-token` derives both
|
||||
from the token issuer with `iss->origin`: `:auth` → `login`; `:verify-email` →
|
||||
`register` + `email-verification`. An unknown `:iss` produces no metadata, so
|
||||
the assertion fires in development and the event goes without `:source` in
|
||||
production.
|
||||
- `auth-backend` and `from-invitation` are read from the profile.
|
||||
- `profile-origin` composes the four and `logged-in` merges them into the event;
|
||||
the emitters only declare the fact.
|
||||
|
||||
| Flow | `source` | `activation` | `from-invitation` |
|
||||
|---|---|---|---|
|
||||
| Password login | login | | false |
|
||||
| Password login from an invitation link | login | | true |
|
||||
| LDAP login | login | | false |
|
||||
| Social login, existing active profile | login | | false |
|
||||
| Signup, email verification disabled | register | immediate | false |
|
||||
| Signup, email verification enabled | register | email-verification | false |
|
||||
| Social signup, active profile | register | immediate | false |
|
||||
| Social signup, inactive profile | register | email-verification | false |
|
||||
| Signup from an invitation link, email verification disabled | register | immediate | true |
|
||||
| Signup from an invitation link, email verification enabled | register | email-verification | true |
|
||||
| Social login from an invitation link | login | | true |
|
||||
| Social signup from an invitation link, email verification enabled | register | email-verification | true |
|
||||
| LDAP login from an invitation link | login | | true |
|
||||
|
||||
LDAP is the explicit exception: the directory provisions the account on the
|
||||
first login, but there is no user signup, so the flow is always `login` and
|
||||
never carries `activation`. Do not "fix" it by inferring a signup.
|
||||
|
||||
## Invitations go through `logged-in`
|
||||
|
||||
- `login-from-register` accepts an optional `:invitation-token`;
|
||||
`with-register-origin` decorates the profile and `logged-in` does the rest.
|
||||
The four places that used to navigate by hand to `:auth-verify-token`
|
||||
(`ui/auth/register.cljs` twice, `ui/static.cljs`, `ui/auth/login_modal.cljs`)
|
||||
emit `login-from-register` with the token. The modal used to live in
|
||||
`ui/viewer/login.cljs`; it was extracted to the shared
|
||||
`ui/auth/login_modal.cljs` (`login-register-modal*`, also used by
|
||||
`ui/mcp_authorize.cljs`).
|
||||
- The invitation navigates through `get-invitation-nav`, which does not wait for
|
||||
`::dtm/teams-fetched`: `fetch-teams` has no error handling, so depending on it
|
||||
would leave the invitation unaccepted when `get-teams` fails. For invited
|
||||
profiles `get-redirect-events` emits `rx/empty`, so the teams path cannot move
|
||||
the user away from the token verification screen.
|
||||
- The OIDC callback with an invitation mints an `:auth` token that carries the
|
||||
invitation JWE in its claims (`backend/src/app/auth/oidc.clj`), so the
|
||||
frontend follows the regular path (`login-from-token` → `logged-in`). That
|
||||
flow visits the token verification screen twice.
|
||||
- `ui/auth/login_modal.cljs` (the register modal: `:is-active` branch calls
|
||||
`on-close`, or `on-register` when the caller passes one, as in the MCP
|
||||
authorize flow) and `ui/static.cljs` (`rt/reload true`) `:is-active`
|
||||
branches still bypass `logged-in`; they are a separate decision, not an
|
||||
oversight.
|
||||
|
||||
## Backend side
|
||||
|
||||
- The `register-profile` success branches
|
||||
(`backend/src/app/rpc/commands/auth.clj`) put `activation` in the event props
|
||||
and emit **no** `action`: the name already says it is a signup, and a constant
|
||||
action would only add noise. The retry branches
|
||||
(`register-profile-retry`) keep `action` in `context`
|
||||
(`ignore-because-blocked`, `ignore`, `resend-email-verification`,
|
||||
`ignore-because-complaints`), where it distinguishes outcomes.
|
||||
- The LDAP invitation branch of `login-with-ldap` returns the full profile plus
|
||||
`:invitation-token`; a partial map would break the `auth-origin` assertion.
|
||||
- `register-profile` does not accept the invitation: it mints the session and
|
||||
returns the token. The acceptance is `accept-team-invitation`, emitted from
|
||||
`verify-token`.
|
||||
- `filter-telemetry-props` keeps `source` and `activation` in the anonymised
|
||||
row; `:action` is not on the allowlist.
|
||||
|
||||
## Analytics
|
||||
|
||||
- `register-profile` is a track event, so the props adapter forwards
|
||||
`activation` with no extra work.
|
||||
- `signin` is an identify event built from a closed list of profile keys. It
|
||||
only reaches analytics if the identify branch of Nexus forwards `source`,
|
||||
`activation` and `from_invitation` (`external/nexus`, another repository).
|
||||
|
||||
## `::ev/source` is no longer dead metadata
|
||||
|
||||
It is read by `profile-origin`. Do not remove it as unused, and keep the
|
||||
vocabulary in sync with `schema:auth-origin` and the table above.
|
||||
@ -49,23 +49,16 @@ Get `team-id` from `(:current-team-id @app.main.store/state)`. Get file ids from
|
||||
|
||||
## Runtime patching with `set!`
|
||||
|
||||
Some frontend vars are deliberately mutable escape hatches for runtime instrumentation or circular-dependency patching. From `cljs_repl`, use `set!` for temporary debugging of CLJS vars such as `app.main.errors/reload-file`, `app.main.errors/is-plugin-error?`, `app.main.errors/last-report`, or `app.main.errors/last-exception`. These patches affect only the live browser runtime and disappear on reload or recompilation.
|
||||
|
||||
`set!` on `app.main.store/on-event` does nothing at runtime: the store captured the function when it was created. To watch events, subscribe to the store stream and dispose the subscription after:
|
||||
Some frontend vars are deliberately mutable escape hatches for runtime instrumentation or circular-dependency patching. From `cljs_repl`, use `set!` for temporary debugging of CLJS vars such as `app.main.store/on-event`, `app.main.errors/reload-file`, `app.main.errors/is-plugin-error?`, `app.main.errors/last-report`, or `app.main.errors/last-exception`. These patches affect only the live browser runtime and disappear on reload or recompilation.
|
||||
|
||||
```clojure
|
||||
;; Log non-noisy Potok events temporarily.
|
||||
(def sub
|
||||
(beicon.v2.core/subs!
|
||||
(fn [event]
|
||||
(when (potok.v2.core/event? event)
|
||||
(.log js/console (potok.v2.core/repr-event event))))
|
||||
app.main.store/stream))
|
||||
;; (beicon.v2.core/dispose! sub)
|
||||
(set! app.main.store/on-event
|
||||
(fn [event]
|
||||
(when (potok.v2.core/event? event)
|
||||
(.log js/console (potok.v2.core/repr-event event)))))
|
||||
```
|
||||
|
||||
`cljs_repl` starts every call in `cljs.user`; begin a call with `(in-ns 'my-ns)` to reuse helpers defined in another namespace.
|
||||
|
||||
Restore mutable hooks after debugging, or reload the frontend. Use JVM `alter-var-root` only for JVM Clojure; it is not the normal way to patch live CLJS browser vars.
|
||||
|
||||
## Browser-console debug namespace
|
||||
|
||||
@ -150,5 +150,5 @@ The frontend bundle is built once per SHA and restored by this job (`mem:fronten
|
||||
Details: README, "Running in CI".
|
||||
|
||||
## Substrate
|
||||
`mem:common/testing`, `mem:common/component-data-model`, `mem:common/component-swap-pipeline`,
|
||||
`mem:common/test-setup`, `mem:common/component-data-model`, `mem:common/component-swap-pipeline`,
|
||||
`mem:frontend/testing`.
|
||||
|
||||
@ -12,7 +12,7 @@ Frontend: CLJS SPA; React/Rumext; Potok; RxJS; okulary refs; SCSS modules; share
|
||||
- `app.render_wasm.*`: frontend bridge to Rust/WASM renderer.
|
||||
- `app.util.*`: DOM, HTTP, i18n, keyboard, codegen, and general frontend utilities.
|
||||
- `frontend/packages/*` and `frontend/text-editor`: JS/TS workspace packages consumed by the app.
|
||||
- Nitrate subscription/organization UI and flows live under `app.main.data.nitrate` and `app.main.ui.nitrate*`; backend/API behavior is covered by backend memories, and shared organization permission rules (`allowed?`, `can-send-invitations?`) are in `common/src/app/common/types/organization.cljc`.
|
||||
- Nitrate subscription/organization UI and flows live under `app.main.data.nitrate` and `app.main.ui.nitrate*`; backend/API behavior is covered by backend memories, and shared permission rules are in `common/src/app/common/types/nitrate_permissions.cljc`.
|
||||
- `app.config` external bridges (`external-feature-flag`, `external-session-id`, `external-context-info`, `external-notify-register-success`, `initialize-external-context-info`) delegate to `globalThis` hooks injected by the SaaS host. Never delete `external-feature-flag`, even with zero call sites: it is the seam for future A/B tests.
|
||||
|
||||
|
||||
@ -46,7 +46,6 @@ Workspace behavior:
|
||||
|
||||
App shell and product flows:
|
||||
- Routing, root app shell, websocket, and global errors: `mem:frontend/routing-app-shell-subtleties`.
|
||||
- Authentication events, the session origin vocabulary, and the invitation flow: `mem:frontend/auth-events`.
|
||||
- Dashboard and viewer flows: `mem:frontend/dashboard-viewer-subtleties`.
|
||||
- Plugin JS API runtime inside the frontend app: `mem:frontend/plugin-api-to-cljs-binding`.
|
||||
|
||||
|
||||
@ -5,17 +5,13 @@
|
||||
- Dashboard initialization fetches projects and fonts for the team, then listens to websocket messages only for global topic `uuid/zero` or the current profile id.
|
||||
- Project fetch replaces each project map completely instead of merging, so fields such as `deleted-at` can disappear cleanly.
|
||||
- Dashboard file/project mutations are often optimistic local updates with fire-and-forget RPC watchers. Bulk permanent delete/restore paths use SSE progress and progress notifications.
|
||||
- The v3 format carries no size per file: the size of one is the sum of its own entries plus the `objects/` its media and thumbnail metadata reference, the only link between a file and that shared storage.
|
||||
- Those metadata entries are read one at a time on purpose: a container holds thousands and every read opens its own decompression stream, which makes Chromium fail the analysis — zip.js then masks the cause behind a `Cannot close a ERRORED writable stream` — while Firefox tolerates it. Do not turn this back into a parallel read. The read is best-effort: a reference that cannot be read leaves that file without size, never the whole package.
|
||||
- The `:error` of an import entry is user-facing text, never an i18n key. The legacy SSE import stays in the backend for other clients.
|
||||
- File creation/duplication strips file `:data` before putting file summaries into dashboard state.
|
||||
- The pending invitations are a team-admin only list (backend `:is-admin` already includes the owner). The dashboard hides the invitations entry from the header tabs and the team-options dropdown for non-admins, `invitations-section-content` keys the section on `:is-admin`, and `fetch-invitations` answers `insufficient-permissions` by emitting `invitations-fetched` with `[]`, so a demoted admin does not keep the cached emails and the header does not paint the invite button from a stale list. The organization `send-invitations` policy (`can-invite?`) does not gate the list, but it gates every action on it: inviting (header and empty-state buttons render disabled with the `dashboard.invite-profile-disabled` tooltip), the role selector (read-only label), row selection (so resend and delete never show) and copy link (`invitation-link-copyable?`). The backend enforces the same rule.
|
||||
|
||||
## Viewer
|
||||
|
||||
- `app.main.ui.ui` parses the viewer's `file-id` query param with `uuid/parse*`, so a missing or malformed id arrives as nil. `viewer-page*` calls `check-file-id!` (same namespace) during render, before mounting `viewer*`: it raises `ex/raise :type :not-found :code :missing-file-id`, which the error boundary turns into the not-found page. Without the guard the `dv/initialize` assertion fails on the nil file-id and the user gets an internal-error page. Test: `frontend-tests.ui.viewer-test/check-file-id-raises-not-found-for-invalid-file-id`.
|
||||
- Viewer initialization sets `:current-file-id`, `:current-share-id`, and `:viewer-local`, then fetches the view-only bundle. Comment threads are fetched only for logged-in users.
|
||||
- Viewer bundle fetch sends the full supported feature set because anonymous shared viewers may not know team-enabled features.
|
||||
- View-only bundles can contain pointer values in `:pages-index` and file data. Viewer resolves those fragments with `:get-file-fragment` before storing the bundle, through `app.main.features.pointer-map/resolve-pointer` (share-link scoped; the workspace and library loaders use `resolve-file`, which resolves the same two levels). Both fail loudly: a fragment that cannot be resolved rejects the stream and is logged with file id and fragment id instead of silently dropping the page.
|
||||
- View-only bundles can contain pointer values in `:pages-index` and file data. Viewer resolves those fragments with `:get-file-fragment` before storing the bundle.
|
||||
- `bundle-fetched` indexes pages and precomputes viewer frames/all-frames, stores libraries/users/thumbnails/permissions under `:viewer`, then navigates to frame id, query index, or auto-selected frame.
|
||||
- Viewer zoom and interaction mode changes update both `:viewer-local` and the `:viewer` route query params. `update-zoom-querystring` guards with a query-param comparison (`not= current expected`) and uses `::rt/replace`; without that guard the load sequence (`bundle-fetched` → `zoom-to-fill` → `update-zoom-querystring` → `rt/nav`) re-runs on every navigation and crashes with React "maximum update depth exceeded" (2.18.0-RC5 regression, fixed in `31b73460c3`; regression test: `bundle-fetched-with-zoom-fill-url-does-not-navigate`).
|
||||
@ -15,9 +15,6 @@
|
||||
## Runtime initialization and permissions
|
||||
|
||||
- The frontend initializes `@penpot/plugins-runtime` only after `features/initialize` and only when feature `plugins/runtime` is active. It also installs the runtime `isPluginError` predicate into frontend error handling.
|
||||
- `app.plugins/init-plugins-runtime` returns whether `ɵloadPlugin` was installed. It calls `preg/signal-runtime-ready` even on failure: `workspace.cljs` waits on that promise before `mcp/init`, so skipping it hangs MCP initialization forever.
|
||||
- When SES bootstrap fails, `app.plugins` emits one `plugins-runtime-unavailable` telemetry event (with the SES error message), and `app.main.data.plugins/load-plugin!` shows the lazy `plugins.runtime-unavailable` toast when the user opens a plugin. `start-plugin!` (MCP path) stays silent by design.
|
||||
- Runtime availability is detected by the existing `(fn? (unchecked-get global "ɵloadPlugin"))` check; do not add a second mechanism.
|
||||
- Manifest parsing expands write permissions to read permissions (`content:write` => `content:read`, etc.). Permission checks also allow the all-zero plugin id and the hard-coded MCP plugin id.
|
||||
- Manifest URL origin differs by manifest version: v1 clears the path; v2 joins `.` to the plugin URL. Existing plugin ids are reused by matching manifest name and host.
|
||||
- The MCP plugin id is defined in `app.plugins.register` to avoid a circular dependency with workspace MCP code.
|
||||
@ -26,9 +23,6 @@
|
||||
|
||||
- Public Plugin API objects are lightweight handles, not durable snapshots. Most getters locate fresh state from `app.main.store/state` using hidden `$id`, `$file`, `$page`, etc.
|
||||
- `not-valid` logs by default but throws when the plugin flag `throwValidationErrors` is enabled. The MCP execute-code handler deliberately enables that flag while running code.
|
||||
- A write to a removed shape always throws `Shape not found`, whatever that flag: setters through the shape proxy's `:on-set u/check-shape-writable` (runs outside the error handling), and methods that must not vanish silently (`resize`, `rotate`) call `u/check-shape-located` first. Methods run inside `:on-error (u/handle-error ...)`, which logs instead of throwing when the flag is off, so it rethrows errors whose `ex-data :type` is `::u/shape-not-found` or `::r/read-only`.
|
||||
- A write while the workspace cannot take it (`r/read-only-reason`: save-failure lock `[:workspace-global :persistence-lock]`, `[:permissions :can-edit]` false, version preview `:preview-id`, lost render context `[:render-state :lost]`) always throws, whatever that flag. The `[:workspace-global :read-only?]` flag alone does not block: the Inspect tab sets it and plugins may still write. `r/check-permission` with `content:write`/`library:write` calls `r/check-writable` first, which covers most writers; writers without a permission check (most top-level context methods in `api.cljs`: `create*`, `group`, `align*`, …; z-order methods; text range `applyTypography`; ruler guide `remove`) call `r/check-writable` themselves, and shape setters get it through `u/check-shape-writable`. A new writer without a write-permission check must call it too. An async writer calls it again after each server await, before the next server write or commit (`rx/tap` on the response stream; `dwm/create-svg-shape-with-images` takes it as `:check-writable`), so a lock that lands mid-way rejects the promise with the read-only error. `r/has-permission?` is the check without it, used by the undo blocks so a block opened before a lock still closes. Comments (`comment:write`) stay writable: they do not go through file saves.
|
||||
- `penpotMgmt` (`app.plugins.management`) checks permissions with `r/has-permission?` (`granted?`), never `r/check-permission`: its calls act on files through the backend, often from the dashboard with no file open, so the workspace read-only check (`check-writable`) must not block them.
|
||||
- `naturalChildOrdering` and `throwValidationErrors` are stored per plugin under `[:plugins :flags plugin-id ...]`; changing default behavior affects automation and MCP diagnostics.
|
||||
- Plugin data is stored under keyword namespaces: private data uses `(keyword "plugin" plugin-id)`, shared data uses `(keyword "shared" namespace)`.
|
||||
|
||||
@ -37,7 +31,4 @@
|
||||
- Plugin listeners are watches on the global store and callbacks are debounced about 10ms. Callback exceptions are caught and logged so plugin code does not crash the app.
|
||||
- `selectionchange` callbacks receive arrays of shape id strings, while `filechange`, `pagechange`, and `shapechange` return proxies.
|
||||
- `contentsave` fires only when persistence status transitions to `:saved`; it calls the callback with no value.
|
||||
- `penpot.waitForSave(timeout?)` (`app.plugins.persist/wait-for-save`; runtime forwarder checks `content:read`) waits on the **shared** persistence queue, not only the plugin's own commits. Flow: ~100ms intake race (buffer/queue non-empty vs deadline, capped to the timeout) → empty resolves; pending emits `::dps/force-persist` and races `dps/wait-persisted-or-error` against `::dps/rollback-rejected` / `::dps/rolled-back` (rollback rejects even if status later reads `:saved`). Default 60s. `:error` status at intake rejects at once (`earlier-failure`): `save-retrying` while `dps/save-retrying?` (slow retry cycle, recovers by itself), otherwise the stored `[:error :code]` with reopen advice (a terminal failure stays until the file is reopened, so it fails every call). Rejections are `Error`s with `code`: `save-timeout`, `rolled-back`, `save-retrying`, or the failure code (`save-failed` unless the backend gave one).
|
||||
- Plugin timeouts validate through `u/valid-timeout?` (positive finite number ≤ `u/max-timeout`, signed 32-bit) and reject through `u/reject-not-valid`, which rejects an `Error` (never a bare string) so callers can read `.message`.
|
||||
- `contentrollback` is not a state watch: it subscribes to `st/stream` for `::dps/rolled-back` (kept in `subscriptions`, disposed by `remove-listener`), not debounced, delivered async so the reverted state is in place. Payload `{fileIds pageIds shapeIds}` from `pcm/rollback-targets` (page shapes only, from the reverted commits' redo changes). When the rollback falls back to reloading the file, `::dps/rolled-back` waits for the reloaded data, so the callback never reads the rejected data. The runtime's `validEvents` must list every event name, or `penpot.on` rejects it; a stale runtime build rejects new events. The MCP plugin forwards these rollbacks with the next task response and the server puts a warning first in that tool response (`RollbackNotices`). Unsent ones survive a tab or plugin reload in `penpot.localStorage` (key `rollbacks/<file-id>`, `LocalStorageRollbackStore`; the MCP manifest in `app.main.data.workspace.mcp` and `mcp/packages/plugin/public/manifest.json` grant `allow:localstorage`, which the runtime checks on access); the entry goes once a response carried them.
|
||||
- Plugin history `undoBlockBegin` creates a workspace undo transaction with a JS `Symbol`; `undoBlockFinish` commits that symbol. Missing finish eventually relies on the workspace transaction timeout.
|
||||
@ -7,8 +7,6 @@
|
||||
- `set-token!` and `replace-token!` are deliberately asymmetric. `set-token!` is a total no-op when the token already is the current one (no URL write, no report), so navigating to the screen you are on cannot grow the history stack. `replace-token!` always writes and always reports, because it is how the app corrects the URL in place (legacy `#/…` translation, cleaned share links) and that correction must reach the router even when the token did not change. This matches `Html5History`, which did the same; do not "fix" the asymmetry without deciding what navigates to the current screen.
|
||||
- `app.util.dom/replace-history-state!` is the deliberate bypass used by `app.main.ui.dashboard`/`app.main.ui.static` to rewrite the URL **without** routing. Keep using it there instead of `replace-token!`, which would re-enter the router.
|
||||
- Route params live entirely in the query map under the reserved `screen` key; duplicate query params can become vectors, so use `rt/get-query-param` when a scalar is required.
|
||||
- The organization SSO flow sends the **query string** (`rt/get-current-query-string`), not the URL and not the token: the backend builds the destination URL from `public-uri` + `?` + those params, so the client never chooses the origin of the redirect. The history token (`rt/get-current-token`, `location.search`) keeps its leading `?` because every other consumer concatenates it over the base path (`str cf/public-uri token`, `dom/append-query-param`, `dom/remove-query-param`); `rt/token->query-string` is the only bridge between the two shapes, so do not trim at a call site by hand; `rt/token->url` is the single client builder of an absolute destination from either shape (used by the SSO retry and error flow), so do not assemble `public-uri + "?" + params` at call sites. The gate is not the only caller: `dtm/resolve-login-team-id` runs from the root branch of `on-navigate` too, and there the URL carries its params instead of a screen (`?template=…`) — that call only reads `:authorized`, so it never follows the destination, which is exactly why a wrong destination there goes unnoticed. Do not add a rule that requires `screen`. See `mem:backend/auth-permissions-product-domains`.
|
||||
- The router events expose their payload through `@event` (`IDeref`): `rt/navigate` returns `{:id :params :options}` and `rt/nav-raw` returns its `:href`/`:uri`. `rt/nav-raw` is the way out of the router (raw URL, outside the token scheme) and nothing in the app listens to `::rt/nav-raw`; the destination travels in the effect, so `@event` is the only way to tell where the user is being sent (tests included).
|
||||
- Legacy `#/…` hash URLs translate client-side to the query format (one-version compat; see `legacy-routes` in `app.main.ui.routes`, TODO(next-version) to delete).
|
||||
- Unknown/empty routes trigger an extra `get-profile`/`get-teams` check before redirecting. This avoids invitation and root-route race conditions.
|
||||
- The root app has two error layers, and they reset differently. `:exception` in the store renders `static/exception-page*` directly and is cleared by `rt/navigated`; a React render error is caught by `app.main.ui.error-boundary` (`react-error-boundary`), whose error lives in internal React state that only `reset-error-boundary` clears. Because every exit from the error page (the logo, the no-permission dialogs, the SSO dialog, the feedback link) only *navigates*, the boundary is given `:resetKeys [(eb/reset-key route)]` — the screen name — so a navigation counts as a fresh attempt at rendering. Three traps in that `resetKeys` value, all learned the hard way: it must be a real JS array (`#js [...]`), never a ClojureScript vector, because `react-error-boundary` reads `.length` and `.some` and a `PersistentVector` has neither, so `componentDidUpdate` throws `some is not a function`; it must hold the screen name as a *string*, because the library compares entries with `Object.is` and ClojureScript does not intern keywords, so two equal keywords are two objects and a keyword key would read as new on every route rebuilt, resetting the boundary on a param change inside the same screen; and it must be the screen rather than the params, so a param change does not replay the screen that just crashed. Navigating to the *same* screen therefore does not reset, and a reload is the only way out of that case.
|
||||
@ -24,9 +22,6 @@
|
||||
- `flash-persistence` only delegates to another handler for `:authentication`, `:not-found` and `:code :vern-conflict`. Everything else it reports itself, and the report type is derived from the cause: `#{:validation :server-error :internal}` are application defects and report as `unhandled-exception` so they reach the internal reports and the alerts; the rest stay `handled-exception`. `:cause-type` wins over `:type` when classifying, because `persistence-failed` wraps the original cause under `:type :persistence`. Environment failures are deliberately kept handled: they land here after the retry budget is spent and the user's connection is not our bug. `:assertion` is also kept handled, matching its own handler.
|
||||
- The report governor is global and stateful across tests: two reports with the same fingerprint inside `report-window-ms` collapse. Call `errors/reset-report-governor!` between assertions that expect an emission.
|
||||
- Error messages are bounded at the source: `app.common.exceptions/get-hint` clamps to `max-message-length` (1000 chars) and the CLJS `format-throwable` clamps the message line of the stack trace and honors `:trace-length`. A message can embed a whole payload (a serialized file, for example) and reach megabytes; the bound keeps the audit `:hint` prop and the `:report` string small. The `Context: Hint:` header in `generate-report` prints `get-hint` (not the raw `:hint` from `ex-data`) for the same reason.
|
||||
- A team scoped navigation carries a team whenever any of the four tiers exists, and `app.main.data.common/resolve-team-id` decides it for every team scoped `go-to-*` helper of that namespace: the UUID the caller passes, `:current-team-id`, the team of the route already on screen (`?team-id`), then the profile `:default-team-id`. `go-to-feedback` and `go-to-viewer` are not team scoped and do not go through it, and `:team-id :default` keeps meaning the profile default, resolved before the rest. The route goes before the profile default because an emitter that runs from a team screen knows which team the user is looking at, and the personal team would open their file under another team. Two `rt/nav` emitters outside that namespace still decide their own id and can still emit nil, and they are the residual: `app.main.ui.nitrate.entry/nitrate-entry*` (the profile default) and the root branch of `app.main.ui.routes/on-query-navigate` (`dtm/resolve-login-team-id`, which answers nil when the profile has no default team). When none of the tiers exists the helper still fires the navigation with a nil id, so the not-found page appears instead of the click doing nothing.
|
||||
- `app.main.ui/team-container*` raises `:not-found :invalid-team-id` for a dashboard or workspace route without a valid `team-id`; the raise is what keeps a broken URL from painting a blank shell, so it stays after the hook fix of #12728. `exception-section*` skips `errors/submit-report` for `:not-found`, so this failure never reaches PostHog and cannot be measured from the error dashboard. The exit from that page is the `error-container*` header (logo plus `not-found.no-permission.go-dashboard`), which emits `go-to-dashboard-recent` with the profile default and falls back to `rt/nav-root`; every not-found and authentication page already carries it, so there is no button to add there.
|
||||
- A click cannot reach a team screen before its team is resolved: `team-container*` paints its children only when the route id matches the store team, and `refs/team` derives from `:current-team-id`, so with that key empty the dashboard and the workspace paint nothing clickable. The nil-id emissions come from events that run on their own (login, invitation acceptance, membership change, the sidebar logo, the workspace shortcuts) and from typed or bookmarked URLs.
|
||||
|
||||
## Store and websocket
|
||||
|
||||
@ -34,4 +29,3 @@ For general store mechanics such as `emit!`, `last-events`, persistence, and und
|
||||
|
||||
- Websocket initialization uses `cf/public-uri` joined with `ws/notifications`, converting `http/https` to `ws/wss`, and includes the current `session-id` as query param.
|
||||
- Reinitializing or finalizing websocket stops the previous receive stream. Incoming websocket payloads become Potok data events under `app.main.data.websocket/message`.
|
||||
- A job the client created is followed with `app.main.data.jobs/watch-job`: it emits the `:job-event` messages of that job (filtered by `:job-id`) and one final outcome read from the row with `get-job`, which is read when the stream starts and again on every `opened` of the websocket, so a job that ended before the client started following it, or while the socket was down, still answers. The stream ends after that outcome, and a read that fails reaches the caller: the repo retries the transient transport errors of a GET, and anything else is a real failure the flow has to show. The `:stage` of a progress event is a stable key of the vocabulary of the worker, never a display string: `app.main.ui.jobs.progress` resolves it with a `case` of literal `tr` calls, falls back to a generic label for a stage it does not know, and the counter of the activity is named after its own stage (the counter of `:files` is the outer context).
|
||||
|
||||
@ -17,8 +17,7 @@ Frontend testing is async-first: everything essentially asynchronous is modeled
|
||||
- `mock/with-mocks` (callback style, legacy compat): installs with `set!` so mocks survive async boundaries; bodies run deferred past the current tick via `asap`, so only done-chained (`t/async`) contexts are allowed; `done'` restores and completes exactly once (twice only warns; never calling it stalls the run and leaks the mocks). Prefer `mock/with-mocks*` for new tests.
|
||||
- `mock/with-mocks*` (direction): body forms wrapped in a generated `^:async` fn, evaluates to a promise — `await` it, `await` nested scopes too, no `done` in test code. Rejections and non-promise returns report as `:error` via `run-mocked`.
|
||||
- `mock/stub` wraps fns for arities 0-6 (the `:esm` test build dispatches multi-arity vars as `cljs$core$IFn$_invoke$arity$N`); when the mocked var is variadic-defined (or called with more than 6 args), the call compiles to variadic dispatch, so use a plain variadic `fn` — the stub does not forward variadic. A mock must not call the mocked var again (self-delegation inside a multi-arity function recurses).
|
||||
- Helpers in `frontend-tests.helpers.async`: `->promise` (single-value observable → promise; beicon has no `to-promise`), `await-response` (subscribe→push→await, atomic), `settle`, `wait-for`, `wait-for-store`, `observe` (stream → termination promise; asserts provided, timeout rejects).
|
||||
- `wait-for` waits on a **signal**, not on turns: `:signal` (an observable) re-checks the predicate on every emission and resolves on the event that changes the state; `:timeout-ms` (default 5000) only bounds failure then. Without a signal it polls against a real-time deadline — 50 turns were ~54 ms idle, and turns are not a unit of time. `wait-for-store` passes `ptk/input-stream` as the signal (the store notifies after applying the update, so the predicate sees the new state); the persistence fixture exposes `requests$` for `(= N (count @requests))`. A wait that resolves on an event consumes no turn. The waits with nothing to listen to stay on the poll: test atoms filled outside the store (`@cycles`, `@reports`, `@done`), `@failures`, and predicates over JS objects or closures.
|
||||
- Helpers in `frontend-tests.helpers.async`: `->promise` (single-value observable → promise; beicon has no `to-promise`), `await-response` (subscribe→push→await, atomic), `settle`, `wait-for` (immediate check + bounded poll, fails instead of hanging), `observe` (stream → termination promise; asserts provided, timeout rejects).
|
||||
- Fixtures return promises (`with-watchdog`, `with-persistence`); `await` them from `^:async` tests. `main_errors` and `fonts` are fully migrated (no legacy `with-mocks` left).
|
||||
- Valid `^:async` placements: `mem:clojure/idioms`.
|
||||
|
||||
@ -38,15 +37,8 @@ To assert over emitted event sequences, observe termination: subscribe through `
|
||||
- `st/emit!` doubles collect heterogeneous events: audit `DataEvent`s deref, toast reify-objects do not — discriminate with `ptk/type` (total, never throws), never blind `deref`.
|
||||
- Subscription order decides delivery order: never resolve settlement from a pre-subscribed branch racing the pipeline.
|
||||
- Auto-answering mocks lose deadline expressiveness (can't time answers), need teardown timer-cancellation, and post-teardown deliveries hit real implementations — explicit pushes + async delivery + `wait-for` won on every axis.
|
||||
- A wait that observes an effect the test asserts takes a signal. A wait that means "the side effect advanced before the next emit" has no event to listen to: keep the poll (or an explicit `settle`) and say so, because a signal would resolve inside the same `emit!` and the next step would overtake the pipeline (`persistence-rollback-test` drove its four emits this way).
|
||||
- With `rp/cmd!` mocked, a store-state wait can resolve before the flow that changes it ends, and the mock scope then closes in flight: the next call hits the real HTTP layer (malformed URL from a nil `cf/public-uri`) and the error is reported on a later test. `unlink-file-from-library` dissocs the library before its async chain, so the only proof the chain ran inside the mock is its second RPC call; wait for that, through the mock.
|
||||
- Teardown belongs to the terminal continuation, never to `finally`-around-triggers (it would dispose in-flight flows).
|
||||
- A body that awaits must be `(^:async fn …)` even if the rest is sync; sync sequences are atomic vs the event loop.
|
||||
- `app.main.errors` prints every handled cause with `ex/print-throwable` before notifying, and the app logger does not gate it. A test that runs a real handler end to end stubs that var (`main-errors-test/quiet-throwable`): without it, the 10 000-error governor test alone wrote 190 k of the 192 k lines of a run log (13,8 MB), and GitHub truncates a step log past ~4 MB, cutting the summary and any failing assertion. Same for a production `console.error` a test provokes on purpose: `code-gen-markup-svg-wasm-test` silences `(.-error js/console)` in a `try/finally` while two WASM panics go through `get-svg-markup` (2 blocks, 70 lines, and the exception dumps its own `data`/`arr`).
|
||||
- The `:esm` test build cannot load the CommonJS `sax` module behind `tubax`, so SVG parsing fails in tests (`dwm/valid-svg-string?` answers false, `dwm/svg->clj` errors): mock both with the parsed data.
|
||||
- A hook-order violation in a mounted component reaches neither the root's `onUncaughtError`/`onCaughtError`/`onRecoverableError` nor the markup: React logs an internal error to `console.error` and recovers, so the callbacks stay empty and the tree repaints. Assert on a captured `console.error` too. It only fires when a render *adds* hooks (nil → uuid); a render that calls none at all slips through React's bookkeeping, so the regression test needs the "hooks arrive" direction. `frontend-tests.ui.team-container-test` does both, with a jsdom document.
|
||||
- `beicon.v2.core` has no `never`. A stream that must **not emit** is `(rx/empty)`: it completes at once, which is enough for `merge`, `filter` and `take` where only the emission matters. A stream that must **stay open** is a `(rx/subject)` nobody pushes to; `rx/empty` would complete the flow behind it. An undeclared `rx/never` compiles to an undefined var and the test dies as a rejected body, which `cljs.test` reports as a test with zero assertions instead of a failure.
|
||||
- `set!` mocks are global and outlive the fixture, so a flow another module left running (a store's pending timer) can still reach the mocked transport mid-test; `with-persistence` records every command for that reason. Filter to the commands your layer issues — persistence only sends `update-file` — before counting or asserting emptiness, and never let a stray call release a `wait-for` on one.
|
||||
|
||||
From `frontend/`:
|
||||
- Full unit test run (always builds, suppressed output): `pnpm run test:quiet`.
|
||||
@ -62,7 +54,7 @@ New frontend test namespaces must be required/listed in `frontend_tests/runner.c
|
||||
|
||||
## Playwright integration tests
|
||||
|
||||
Do not add, modify, or run Playwright integration tests under `frontend/playwright` unless explicitly asked. When explicitly asked, run the whole suite with a worker count sized to the machine: `pnpm run test:e2e --workers=N` from `frontend/`, about one worker per two hardware threads, because each keeps two or three busy and holds about 2 GB of browser memory. The config's default of one worker leaves it slow; four take a 16-thread host from 48 minutes to 35 and fit a 64 GB machine beside a watching devenv, while a 32 GB machine takes two, or four with a slim devenv. `--grep "pattern"` narrows the run. Ensure dependencies are installed through `./scripts/setup` if the environment is not prepared.
|
||||
Do not add, modify, or run Playwright integration tests under `frontend/playwright` unless explicitly asked. When explicitly asked, use `pnpm run test:e2e` or `pnpm run test:e2e --grep "pattern"` from `frontend/`; ensure dependencies are installed through `./scripts/setup` if the environment is not prepared.
|
||||
|
||||
Integration tests fake backend behavior by intercepting network/websocket traffic, so every RPC or websocket the page needs must be mocked. Use existing Page Object Models:
|
||||
- `BasePage.mockRPC` intercepts RPC calls and already prefixes `/api/rpc/command/`; pass command names such as `get-profile`, not full URLs.
|
||||
|
||||
@ -7,11 +7,10 @@ high-coverage support reference, never the base.
|
||||
## Workflow
|
||||
|
||||
- Canonicalize with `node ./scripts/translations.js sync -l <locale>` from `frontend/`: sorts entries, syncs `#:` comments from `en`, deletes keys missing in `en`.
|
||||
- `sync` never ADDS a key missing in the locale: it only syncs comments (for keys that already exist) and drops keys absent from `en`. A key added to `en` alone stays missing in `es` and its text silently falls back to English. Every new key is born in `en` AND `es` in the same change; compare the `msgid` sets to check (`comm -23 <(grep '^msgid ' en.po | sort -u) <(grep '^msgid ' es.po | sort -u)` must be empty).
|
||||
- `sync` copies ALL comment flags from `en`, including `#, fuzzy`. Translated entries must NOT stay fuzzy: strip the flag after translating (mirror `es.po`, which keeps fuzzy only on genuinely untranslated entries). Fuzzy entries are excluded from `msgfmt --statistics` translated counts.
|
||||
- Canonical files carry NO `#~` obsolete blocks (`en`/`es` have zero). Drop them; `sync` does not resurrect them.
|
||||
- Definition of done: `msgfmt --check <locale>.po` exit 0, `msgfmt --statistics` shows 0 fuzzy, 0 untranslated, and the locale's `msgid` set is identical to `en.po`'s (no key missing, none extra).
|
||||
- `rehash` scans `frontend/src` AND `common/src` for `(tr "key"` occurrences (any line counts, including `;;` comments) and refreshes `#:` refs in `en`, marking unreferenced keys `#, unused`. The refs in `en` are broadly stale, so a repo-wide `rehash` rewrites hundreds of lines (measured: ~487 on `en` alone): it is a maintenance action on its own, never a step to run inside a change that only adds or edits a key.
|
||||
- Definition of done: `msgfmt --check <locale>.po` exit 0 and `msgfmt --statistics` shows 0 fuzzy, 0 untranslated.
|
||||
- `rehash` scans `frontend/src` AND `common/src` for `(tr "key"` occurrences (any line counts, including `;;` comments) and refreshes `#:` refs in `en`, marking unreferenced keys `#, unused`.
|
||||
- Dynamic keys are invisible to `rehash`: eliminate them instead of
|
||||
declaring them. Preference order: literal `(tr "key")` args only;
|
||||
never put a branch inside `tr`, hoist it out (`(if cond (tr "a") (tr "b"))`);
|
||||
@ -29,12 +28,6 @@ high-coverage support reference, never the base.
|
||||
|
||||
## Entry rules
|
||||
|
||||
- Placement: a key names the feature that renders it (`dashboard.import.*`,
|
||||
`files-export-modal.*`, `jobs.*`); `labels.*` is the bucket for the short
|
||||
generic words reused across screens (Accept, Cancel, Download…), never the
|
||||
home of a feature's own vocabulary. Keep a feature's strings together: the
|
||||
durable jobs write their progress stages as `jobs.progress.stage.*` and
|
||||
their row statuses as `jobs.*`, not half in `labels.*`.
|
||||
- Keys resolved dynamically from server-provided data carry a `#, backend`
|
||||
flag (set it in `en`, `sync` copies it to the locales): do not remove or
|
||||
rename these entries, and keep their `%s` placeholders (the backend only
|
||||
@ -47,6 +40,7 @@ high-coverage support reference, never the base.
|
||||
- Preserve verbatim: `%s`/`%d`, `{var}`/`{{...}}`, markdown `[text](%s)`, HTML tags, `\n` positions, brand names (Penpot), key names (Ctrl/Shift/Alt), technical terms (SVG, CSS, HSV, RGB).
|
||||
|
||||
## Catalan (ca) conventions
|
||||
|
||||
- Normative IEC/Termcat Catalan. Address the user in VOSALTRES (2nd person plural): "Deseu", "Creeu", "Ja teniu un compte?". Buttons/menus use short imperatives ("Crea", "Mou", "Restaura").
|
||||
- Established glossary (reuse exactly, do not re-coin): layer=capa, board=tauler, stroke=traç, fill=Emplenat, blur=Difuminat, shadow=Ombra, clipboard=porta-retalls, delete=Elimina, rename=Canvia el nom, shortcut=drecera, grid=graella (keep "grid" where the file already does, e.g. grid-layout editing), plugins/extensions UI=extensions, layout=Disposició, gradient=Degradat, wireframing kept as loanword.
|
||||
- Ela geminada uses the middle dot: Cancel·la, paral·lel, al·lega.
|
||||
@ -54,20 +48,6 @@ high-coverage support reference, never the base.
|
||||
- Shortcut/action names (`shortcuts.*`) are noun/infinitive labels, not sentences. Error strings are direct, no hedging.
|
||||
- Same English source in different contexts may legitimately differ (verb "Copia" vs noun "Còpia"; "Desactivat" vs "Deshabilitada" agreeing with "drecera"). Normalize only true duplicates.
|
||||
|
||||
## Russian (ru) conventions
|
||||
|
||||
- Formal address with capital ВЫ ("Вы уверены…", "Вы собираетесь…"). Buttons/menus use infinitives ("Создать", "Удалить", "Дублировать"), never imperatives.
|
||||
- Established glossary (reuse exactly, do not re-coin): layer=слой, board=доска (EN `board` is NEVER кадр — normalized in lote1: toolbar/viewer/shape-menu/header-menu/export-frames/undo/num-of-frames), path/draw tool=контур, stroke=обводка, fill=заливка, blur=размытие, shadow=тень, grid=сетка, clipboard=буфер обмена, shortcuts=сочетания клавиш (normalized in lote1: title/show-shortcuts/not-found), plugin=плагин, layout=макет (`labels.layout` fixed in lote1), gradient=градиент, token=токен, set=набор, theme=тема, file=файл, project=проект, team=команда, library=библиотека, component=компонент.
|
||||
- No external RU style guide applies verbatim (Mozilla ru guide is marketing-only, GNOME glossary unmaintained, Adobe uses монтажная область for artboard but `ru.po` majority already says доска — keep доска for consistency).
|
||||
- `words.ru.txt` `[function]` DELIBERATELY omits 1-letter words (в/с/к/о/у/а/и) and ending-colliding particles (на/по/не/ли/же/бы/его/их/те/или…): Russian inflects (проект→проекта, запросили=запрос+или), so listing them flags every inflected noun/verb. Never "complete" that list; put tripped valid words in `[ok]`.
|
||||
- Never abbreviate т. д. / т. е. in `ru.po`: the lone т/д tokens poison the glue detector's frequency map and flag short words (тон→т+он). Write и так далее out.
|
||||
- Tokens-domain glossary (lote2, reuse): alias=алиас, remap=переназначить, viewport=вьюпорт, set=набор, theme=тема, spread (shadow)=распространение, margins=внешние отступы, paddings=внутренние отступы, gaps=промежутки, type identifiers stay in English (Line Height, Font Size, innerShadow/dropShadow, none/Uppercase/Lowercase/Capitalize, thin/light/regular/bold/Italic).
|
||||
- Editor-domain glossary (lote3, reuse): variant=вариант, property=свойство, guide=направляющая, board guide=направляющая доски, MCP server=MCP-сервер, size preset=пресет размера, border-radius section=скругление, path handler=маркер, canvas=холст, render=рендер/рендеринг, plugin (never extensión)=плагин.
|
||||
- Avoid standalone те (those) in phrasing ("те же" → "такие же"): one lone те validates те+мы splits and flags every темы. Same poisoning class as т. д.
|
||||
- `words.ru.txt` `[function]` also omits пока (dropped in lote4): with standalone за anywhere in the file, пока+за flags показа. Rare-typo detection loss accepted.
|
||||
- Billing-domain glossary (lote4, reuse): plan names follow shipped `ru.po`, NOT es: Enterprise=Корпоративный, Professional=Профессиональный, Unlimited=Безлимитный, trial=пробный; subscription=подписка, storage=хранилище, seat editors=редакторы, viewer role=наблюдатель, amounts as `175 $/мес.` / `10 ГБ`; access token=токен доступа, MCP key=MCP-ключ (regenerate=перевыпустить), markdown `[text|target:self](%s)` kept verbatim.
|
||||
- Closing-domain glossary (lote5, reuse): shortcut noun everywhere=сочетание (shortcuts section titles are noun/infinitive labels, never ярлыки); node=узел, segment=сегмент, measurements=размеры; WebGL unavailable flow uses гайд/руководство по неполадкам + рендеринг WebGL выключен/включён; `ru.po` reached 2275/2275 keys (100%) with 0 fuzzy, 0 untranslated per `msgfmt --statistics`.
|
||||
|
||||
## QA before commit
|
||||
|
||||
- Run `node ./scripts/translations.js check -l <locale>` from `frontend/` (no default locale: pass `-l` explicitly): 0 errors required; review warnings by hand. Word lists live in `frontend/scripts/check-translations/words.<locale>.txt` (`[elision]` `[function]` `[common]` `[ok]` `[brands]`); new valid words that trip the gate go to `[ok]`; `check --self-test` covers the detector rules. Without a catalog only the universal checks run. `#, fuzzy` entries are skipped (known-pending, owned elsewhere).
|
||||
|
||||
@ -12,7 +12,7 @@
|
||||
|
||||
- Co-located SCSS modules are preferred. Use `app.main.style/stl` helpers from CLJS and design-system SCSS tokens/mixins instead of legacy global selectors or high-specificity nesting.
|
||||
- Keep CSS specificity low. Avoid nested selectors unless they target elements the component owns; CSS Modules already prevent class-name collisions.
|
||||
- Prefer CSS logical properties for directional spacing/layout (`padding-inline-start`, etc.). Physical `width`/`height` are still acceptable where they are clearer. Never mix: a stylesheet logical property (`inline-size`) and an inline physical `width` on the same element cascade separately, and the logical mapping wins — a dynamically driven width must be `width` on both sides (import modal progress bar hit this). And an inline style passed from CLJS must be a JS object (`#js {:width …}`): for plain `[:div …]` elements the Rumext macro does not convert the `:style` map at runtime — a plain CLJS map reaches React raw and React never finds the keys (static style literals are converted at compile time, which is why `:style` maps seem to work elsewhere).
|
||||
- Prefer CSS logical properties for directional spacing/layout (`padding-inline-start`, etc.). Physical `width`/`height` are still acceptable where they are clearer.
|
||||
- Use named design-system variables/tokens for spacing, borders, fixed dimensions, colors, and typography. Avoid hardcoded px/rem values and deprecated `resources/styles/common/refactor/spacing.scss` variables.
|
||||
- Use component-local CSS custom properties for variants and theming instead of one-off Sass variables when a component has multiple visual states.
|
||||
- Prefer DS typography components (`heading*`, `text*`) and typography mixins instead of plain text wrappers or deprecated typography mixins.
|
||||
|
||||
File diff suppressed because one or more lines are too long
@ -6,7 +6,6 @@
|
||||
- `apply-modifiers*` is the lower-level commit path once object/text modifiers are ready. It updates frame guides, frame comment threads, and then emits `update-shapes` with `:reg-objects? true`.
|
||||
- Transform commits restrict diff attrs to `transform-attrs` to avoid scanning unrelated shape attrs.
|
||||
- Text transforms may carry derived `:position-data`; `assoc-position-data` attaches it while preserving the original text shape context.
|
||||
- A drag commits when its `mse/drag-stopper` fires, `:interrupt` (Escape) included. The save-failure lock emits `::plk/cancel-gestures` first: drags built on `plk/gesture-stopper` (start-move, start-resize, start-rotate, start-move-line-point, box/curve/line drawing) see a `cancelled?` flag set before the stop and emit `dwt/cancel-transform` (`clear-local-transform` + `finish-transform`) or `cancel-drawing` instead. A new drag gesture must use it (or, in a component, drop the drag when read-only turns on): `commit-changes` refuses the commit anyway, but the preview (WASM modifiers/props) would stay.
|
||||
|
||||
## Component-copy touched suppression
|
||||
|
||||
|
||||
@ -13,26 +13,19 @@ enabling advanced AI-driven features in Penpot.
|
||||
- Build Tool: TypeScript Compiler (tsc) + esbuild
|
||||
- Package Manager: pnpm
|
||||
|
||||
## Principles & Code Style
|
||||
## General Principles
|
||||
|
||||
IMPORTANT: Use an idiomatic, object-oriented style.
|
||||
* You keep each concern in exactly one home.
|
||||
A mechanism whose parts are only correct in combination is implemented as a single component/class, and its parts are private to it;
|
||||
expose a minimal public surface. Java-like encapsulation: If helper functions and constants are only used by one abstraction,
|
||||
make them internal to the respective class.
|
||||
* For any non-trivial interfaces, you use functions/methods that expect explicitly typed abstractions
|
||||
rather than mere functions (i.e. use the strategy pattern, for example).
|
||||
* Prefer declaring and instantiating concrete classes over defining only interfaces and using object literals.
|
||||
In particular, this implies that, for any non-trivial interfaces, you use interfaces that expect explicitly typed abstractions
|
||||
rather than mere functions (i.e. use the strategy pattern, for example).
|
||||
|
||||
Comments:
|
||||
* When describing parameters, methods/functions and classes, you use a precise style, where the initial (elliptical) phrase
|
||||
clearly defines *what* it is. Any details then follow in subsequent sentences.
|
||||
* When describing what blocks of code do, you also use an elliptical style and start with a lower-case letter unless
|
||||
the comment is a lengthy explanation with at least two sentences (in which case you start with a capital letter, as is
|
||||
required for sentences).
|
||||
When describing parameters, methods/functions and classes, you use a precise style, where the initial (elliptical) phrase
|
||||
clearly defines *what* it is. Any details then follow in subsequent sentences.
|
||||
|
||||
Syntax:
|
||||
* Never use if blocks (or other blocks) without braces, even for single-line cases.
|
||||
When describing what blocks of code do, you also use an elliptical style and start with a lower-case letter unless
|
||||
the comment is a lengthy explanation with at least two sentences (in which case you start with a capital letter, as is
|
||||
required for sentences).
|
||||
|
||||
## Project Structure (Excerpt)
|
||||
|
||||
@ -57,7 +50,7 @@ mcp/
|
||||
│ ├── src/
|
||||
│ │ ├── main.ts # handles communication
|
||||
│ │ └── plugin.ts # plugin implementation
|
||||
│ └── package.json # Plugin deps: published plugin-styles, linked plugin-types
|
||||
│ └── package.json # Includes @penpot-mcp/common dependency
|
||||
└── prepare-api-docs # Python project for the generation of API docs
|
||||
```
|
||||
|
||||
@ -107,36 +100,8 @@ From the `mcp/` directory, run
|
||||
(node reports `Promise resolution is still pending but the event loop has
|
||||
already resolved` for the file's root test). Inside tests the logger writes to
|
||||
stdout directly. `NODE_TEST_CONTEXT` is the env var `node --test` sets.
|
||||
* `packages/plugin` takes `@penpot/plugin-styles` from the registry but links
|
||||
`@penpot/plugin-types` to `plugins/libs/plugin-types` (`link:` in the manifest
|
||||
and in `mcp/pnpm-lock.yaml`, with no registry entry, because a linked package
|
||||
has no resolution): the published types do not declare the `penpotMgmt`
|
||||
global the plugin needs. On a plugins libs version bump, only the styles
|
||||
version moves. After editing the manifest, `pnpm install --lockfile-only` in
|
||||
`mcp/` must leave the lock untouched.
|
||||
* Cross-cutting testing principles and anti-patterns: `mem:testing`.
|
||||
|
||||
## OAuth for remote MCP clients
|
||||
|
||||
- Authorization/discovery URLs, self-contained client IDs, consent and PKCE/code binding, backend Redis state, deployment wiring, and compatibility limits: `mem:mcp/oauth`.
|
||||
|
||||
## Host and Origin allowlist (`RequestGuard`)
|
||||
|
||||
- `RequestGuard` checks `Host` and `Origin` (hostname only, port ignored) on `/mcp`, the REPL (Express middleware, 403) and the plugin WebSocket (close `1008` before any message is read). It stops web pages on the user's machine from driving a local server (DNS rebinding, cross-site WebSocket).
|
||||
- Allowlist: `PENPOT_MCP_ALLOWED_HOSTS` (`*` turns the check off); unset → loopback only in single-user mode (local or remote: no other barrier, `execute_code` still runs in the user's session), any host in multi-user mode (relies on user tokens; behind nginx `/mcp/ws` the server sees `Host: penpot-mcp:4402`). A missing `Origin` passes (non-browser clients).
|
||||
- Entries must be `*`, a hostname or a bracketed IPv6 address; one with a scheme, port or path makes `fromEnvironment` throw, so the server does not start. Startup logs a `warn` when the check is off in single-user mode.
|
||||
- The plugin iframe has `allow-same-origin`, so its WebSocket carries the real plugin origin (`http://localhost:4400` locally, the Penpot origin when integrated).
|
||||
- Test mocks of `PenpotMcpServer` passed to `PluginBridge` must provide `requestGuard`.
|
||||
|
||||
## Backend tools (`submit_feedback`)
|
||||
|
||||
- Tools that call the Penpot backend directly (not through the plugin) register only when `PENPOT_MCP_BACKEND_URI` is nonblank (`PenpotMcpServer.initTools`). Currently only `SubmitFeedbackTool`.
|
||||
- `PenpotBackendClient` owns the HTTP call and the status-to-message mapping; it posts to `api/main/methods/send-mcp-feedback` with `Authorization: Token <MCP user token>`. The token comes from the session context (OAuth bearer or `?userToken=`); without one the tool fails before calling the backend.
|
||||
- Tool zod schemas mirror the backend malli schema (lengths, blank, control characters); keep both in sync. The backend is the authority.
|
||||
- Success means the mail job was queued, not delivered; the tool text says so. `formatArgs` hides the feedback from logs.
|
||||
- Backend side: `send-mcp-feedback` in `app.rpc.commands.feedback`, configured by `PENPOT_MCP_FEEDBACK_EMAIL`; MCP-token auth rules in `mem:backend/auth-permissions-product-domains`.
|
||||
- User docs: `mcp/README.md` (env tables), `docs/mcp/index.md` (tool list), `docs/technical-guide/configuration.md` (MCP feedback section).
|
||||
|
||||
## Devenv plugin/server wiring
|
||||
|
||||
In the normal Penpot devenv MCP path, the browser plugin does not discover or route through Postgres. The frontend provides the plugin extension API with `mcp.getServerUrl()`, currently derived from `frontend/src/app/config.cljs` as `penpotMcpServerURI` if set, otherwise `<public-uri>/mcp/ws`. The MCP plugin opens a direct WebSocket to that URL and appends the current MCP access token as a query parameter.
|
||||
@ -148,11 +113,5 @@ For parallel devenvs, prefer same-origin MCP routing: each Penpot instance shoul
|
||||
## Plugin reconnect policy
|
||||
|
||||
- The plugin treats WebSocket close code `1008` (policy violation) as terminal: it stops auto-reconnecting and stays disconnected until the user explicitly reconnects. Other close codes keep the capped-backoff retry. The decision lives in `ReconnectPolicy.ts` (`shouldReconnectAfterClose`), kept as a pure module so it is unit-testable without DOM/CSS.
|
||||
- The MCP server emits `1008` for a `Host`/`Origin` outside the `RequestGuard` allowlist, for a missing `userToken` in multi-user mode, for missing/invalid connection metadata, and to close a connection displaced by a newer one with the same user and session ID.
|
||||
- Session IDs are deterministic per tab and file, so a reconnect reuses its ID while the old socket may be half-open: the newest connection wins (`PluginBridge.displaceConnection`). Across instances, each new connection publishes a session claim on a per-user Redis channel; an instance displaces its local connection only on a foreign claim that arrives after its own claim (channel order, no clocks). Discovery deduplicates by session ID.
|
||||
- A tab rejected with `1008` never reaches `connected`, so the frontend's 60s reconnect watcher (`start-reconnect-watcher` in `app.main.data.workspace.mcp`, started only on `connected`) does not engage; recovery is manual via the MCP menu.
|
||||
|
||||
## Rollback warnings
|
||||
|
||||
- `ExecuteCodeTaskHandler` awaits `penpot.waitForSave()` after the code and outside its swallow catch, so a save failure, save timeout or rollback during the wait fails that `execute_code` call itself. It hard-fails as "missing Plugin API support" when `penpot.waitForSave` is absent (old Penpot or stale runtime build, or broken `api.cljs` wiring). Both waits run only when `penpotMgmt.workspace.status` is `"ready"` (or there is no `penpotMgmt`), checked at each wait: dashboard calls skip them, because persistence `:error` outlives the workspace and would fail successful `penpotMgmt` calls. Budget: layout wait 30s before the code + save wait 60s after it, under `PENPOT_MCP_TOOL_TIMEOUT_S` (120s). `RollbackNotices` below only cover rollbacks that land after the call returned (concurrent edits, a response after a timeout).
|
||||
- The plugin's `RollbackLog` (`packages/plugin/src/RollbackLog.ts`) holds the `contentrollback` events not yet sent with a task response; `Task.sendResponse` attaches them and acknowledges them once sent, and the server's `unclaimed-rollbacks` message (a response that came after its task timed out) puts them back. With a `RollbackStore` (`LocalStorageRollbackStore` over `penpot.localStorage`, one entry per file) every change is written through, so a reload of the tab or plugin keeps them; storage failures fall back to memory. Plugin tests run with `node --experimental-strip-types`, which rejects TypeScript parameter properties: declare fields explicitly.
|
||||
- The MCP server emits `1008` for a duplicate connection on the same user token (`PluginBridge`) and for a missing `userToken` in multi-user mode.
|
||||
- A tab rejected with `1008` never reaches `connected`, so the frontend's 60s reconnect watcher (`start-reconnect-watcher` in `app.main.data.workspace.mcp`, started only on `connected`) does not engage; recovery is manual via "Connect here".
|
||||
|
||||
@ -1,47 +0,0 @@
|
||||
# MCP OAuth
|
||||
|
||||
## Contract and scope
|
||||
|
||||
- MCP-specific authorization; existing Penpot login identifies the user on the approval page. No application vetting or persistent consent policy. Access requires user approval; registration alone grants nothing.
|
||||
- Return the existing fixed MCP user token. On approval, enable `:mcp-enabled` and create the token only if no usable one exists; preserve an existing usable token. A token is unusable, and replaced, when expired (settings can give MCP tokens an expiry), when its JWT issuer is not `urn:penpot:mcp-token` (tokens issued before MCP tokens had their own issuer), or when this instance cannot decrypt it (e.g. secret key changed); the backend rejects all three. No refresh token, `expires_in`, or new token lifetime.
|
||||
- No new client database table or schema migration. Client IDs must remain usable across separate authorization flows without server-side registration storage. Do not replace them with expiring Redis registrations.
|
||||
- MCP transport continues matching user tokens to connected Penpot plugins; no new per-request token introspection or independent semantic validation. The logged-in Penpot instance remains the content-access boundary. Do not silently add resource-server validation or change token semantics.
|
||||
- README scope: user/developer usage instructions; document `PENPOT_MCP_AUTH_URI` in its environment-variable table. Keep OAuth architecture and protocol details here and non-obvious invariants in source comments.
|
||||
|
||||
## Deployment and discovery
|
||||
|
||||
- MCP process: `PENPOT_MCP_AUTH_URI=https://penpot.example/api/mcp/oauth` with `--multi-user`. Unset/empty disables OAuth; configured OAuth requires multi-user. Absolute HTTPS, or HTTP on loopback for development; no userinfo, query, or fragment. Invalid configuration fails startup; startup logs OAuth status.
|
||||
- Devenv: `mcp/scripts/start-mcp-devenv-oauth` starts multi-user mode with `PENPOT_MCP_AUTH_URI` derived from `PENPOT_PUBLIC_URI`. Manual opt-in only: agentic tooling needs the single-user `start-mcp-devenv`.
|
||||
- Public MCP transport: `<Penpot public URI>/mcp/stream`, proxied to Node `/mcp`. Missing credentials with OAuth enabled: 401 plus `WWW-Authenticate` referencing `<auth base>/resource` and scope `mcp`.
|
||||
- Bearer header takes precedence over legacy `?userToken=`. Malformed headers do not fall back to the query token. Existing query-token clients remain supported.
|
||||
- Protected-resource metadata: `/api/mcp/oauth/resource`; issuer: `/api/mcp/oauth`; authorization-server metadata: `/.well-known/oauth-authorization-server/api/mcp/oauth`. All reach the backend. Bundled devenv/production nginx configs forward the well-known path; custom proxies must do likewise. Keep Penpot public URI and external URLs consistent.
|
||||
- Node implementation: `mcp/packages/server/src/McpAuthentication.ts`, integrated by `PenpotMcpServer.ts`; OAuth-enabled tools advertise `_meta.securitySchemes`.
|
||||
|
||||
## Client identification
|
||||
|
||||
- `backend/src/app/mcp/oauth_clients.clj`: dynamic registration at `/api/mcp/oauth/register`, HTTPS client metadata documents (CIMD), and registrations obtained in advance via `(app.mcp.oauth-clients/register! {} metadata)`.
|
||||
- DCR returns `penpot-mcp-v1.` plus unpadded base64url JSON: normalized callback URLs, optional name, authentication method, random nonce, optional secret hash. Fixed grant/response/scope values are reconstructed. Nonce distinguishes registrations with otherwise identical metadata.
|
||||
- IDs are public, unsigned, unencrypted, non-expiring metadata; validate on every resolution. Client secrets are generated randomly and returned separately; only the hash enters the ID. No signing/encryption key or client record in PostgreSQL/Redis. Old experimental database-backed UUID IDs require registration again.
|
||||
- An edited payload changes the ID. Authorization-code exchange compares the exact ID string against the approved request before resolving metadata. Do not replace this with comparison of decoded fields; code ownership and PKCE binding must survive metadata edits.
|
||||
- Supported authentication: `none`, `client_secret_basic`, `client_secret_post`; no `private_key_jwt`. CIMD selects public-client `none` from the document's supported methods; other listed methods do not imply support here. Scope `mcp`; grant `authorization_code`; response `code`. CIMD may advertise additional grants (e.g. Codex lists `refresh_token`); require `authorization_code` support and normalize to the grant Penpot implements, rather than reject broader client capabilities.
|
||||
- Entire ID capped at 2,048 ASCII characters. DCR returns `invalid_client_metadata` if encoded metadata exceeds this; shorter/fewer callback URLs reduce size. OAuth defines no fixed ID-length maximum (RFC 6749 §2.2); stateless IDs are explicitly described by RFC 7591 §A.5.2. Installed MCP client SDK 2.3.1 accepts a 2,048-character ID; this does not establish Codex/Claude compatibility.
|
||||
- Redirect matching is exact except loopback HTTP port changes for native clients (`127.0.0.1`, `[::1]`, exact `localhost`). Host, path and query must still match; reject fragments/userinfo and lookalike localhost domains. Allow HTTPS, loopback HTTP, and reverse-domain native schemes.
|
||||
- CIMD: fetch with existing SSRF-checked `app.http.client/req`, no redirect following, require matching document `client_id`, bounded 64-KiB body and request/body-read timeouts. Cache max 256 entries/one minute, further limited by HTTP max-age; no-store/no-cache/no max-age means no reuse.
|
||||
|
||||
## Approval and exchange
|
||||
|
||||
- `backend/src/app/http/mcp_oauth.clj`: separate router branch from ordinary RPC middleware; preserve OAuth snake_case JSON and form encoding. Session auth applies to approval request/decision only. Public metadata/register/token endpoints allow noncredentialed CORS; MCP transport CORS was not added.
|
||||
- Errors: every failure answers with an OAuth `{"error": ...}` body, never the shared Penpot error shape. Failures without `:oauth-error` are also passed to `app.http.errors/handle` for logging only; its response is discarded. `wrap-errors` also wraps the session middleware of `/request` and `/decision`. Safe to log because these routes never set request `:params`.
|
||||
- `/authorize`: validate client/callback, `code`, PKCE S256, scope/resource/state; store pending request for 600 seconds; redirect to `?screen=mcp-authorize&request=<opaque handle>`.
|
||||
- `frontend/src/app/main/ui/mcp_authorize.cljs`: existing login with return URL; show application and account details; submit explicit Allow/Cancel. No fixed MCP token in frontend state or browser redirect. Approval refreshes profile and broadcasts MCP enablement; does not connect a plugin automatically.
|
||||
- `/request`: first authenticated read binds request to profile plus hashed login-session ID and CSRF value, using compare-and-replace. Re-read after the attempt: concurrent GETs may win/lose binding; only the matching session may proceed.
|
||||
- `/decision`: check exact Origin, session binding and CSRF, then atomically consume the request. Denial creates no token/enables nothing. Approval calls transactional `access-token/ensure-mcp-access!`, then stores code for 120 seconds. Code state holds token row ID, never the fixed token. Callback echoes OAuth `state` and RFC 9207 `iss`.
|
||||
- `/token`: validate grant, exact client ID/callback/resource and PKCE; resolve current client metadata/authentication method and verify any secret; atomically consume validated code; fetch the unexpired token by original row ID/profile/type. Regeneration deletes the old token row, so a pending code cannot silently yield the replacement token. Return fixed `access_token`, `token_type: Bearer`, scope `mcp`.
|
||||
- `backend/src/app/mcp/oauth_state.clj` and Lua: tenant/kind/hash-prefixed keys; random handles; SET NX EX must succeed before returning handle. Reads retain raw JSON for atomic comparison. Consume/delete and bind/replace compare the validated raw record in Lua; replacement preserves TTL. Redis failures fail closed. Backend Redis is required independently of Node's optional Redis plugin bridge.
|
||||
- `backend/src/app/rpc/commands/access_token.clj`: `ensure-mcp-access!` decodes the stored token with `ex/ignoring` to apply the usable-token rule above. Lock profile row in both MCP generation and OAuth provisioning, including when no token exists. Serializes simultaneous approvals and settings regeneration. Provisioning must run within a transaction.
|
||||
|
||||
## Verification and limits
|
||||
|
||||
- Tests: `mcp_oauth_clients_test.clj`, `mcp_oauth_http_test.clj`, `mcp_oauth_state_test.clj`, OAuth provisioning in `rpc_access_tokens_test.clj`; MCP authentication/HTTP tests; frontend approval tests. Cover no-storage registration, size boundaries, edited-ID rejection, code replay/concurrency, session/CSRF binding, and fixed-token reuse.
|
||||
- Live OpenAI/Codex/Claude compatibility and marketplace acceptance remain unverified. In particular, verify token persistence without `expires_in` and acceptance of the existing matching-only resource boundary; do not claim protocol conformance from SDK tests alone.
|
||||
- Protocol references: https://www.rfc-editor.org/rfc/rfc6749.html#section-2.2 ; https://www.rfc-editor.org/rfc/rfc7591.html#appendix-A.5.2 ; https://www.rfc-editor.org/rfc/rfc7636.html ; https://www.rfc-editor.org/rfc/rfc9207.html .
|
||||
@ -26,7 +26,6 @@ No manual line wrapping: one line per bullet or paragraph, however long. Memorie
|
||||
## Add/update threshold
|
||||
|
||||
Add or update memories only with stable, non-obvious project conventions that avoid complex rediscovery in the future.
|
||||
Do not restate what the code already says (comments included): keep only what code cannot represent — decisions and their reasons, cross-file invariants, ordering constraints, environment and tool behavior, and traps that break builds.
|
||||
Do not add: quick-read facts; generic language/framework knowledge; one-off task notes; volatile line-level details; behavior likely to change soon.
|
||||
|
||||
## Maintenance Actions
|
||||
|
||||
@ -18,7 +18,6 @@
|
||||
|
||||
## Sandbox and global cleanup
|
||||
|
||||
- `src/index.ts` calls `bootstrapSES()` (in `src/lib/bootstrap.ts`) at module load. If `repairIntrinsics` throws (e.g. SES 2.2+ rejects a `Temporal` wrapped by a browser extension), the error is caught, `getPluginsRuntimeError()` exposes it, and `initPluginsRuntime` installs no `ɵ*` globals: the frontend app still boots, plugins and MCP are unavailable in that environment. On failure, `Function.prototype.toString` has already been replaced but no intrinsics are hardened and no compartment/sandbox is created — never run plugins on that path.
|
||||
- The runtime uses SES compartments. Public API return values are passed through `ses.safeReturn` before crossing back to plugin code.
|
||||
- Plugin `fetch` is sanitized: credentials are omitted and Authorization is blanked. The exposed response only includes ok/status/statusText/url/text/json.
|
||||
- Timer callbacks are wrapped to mark plugin-originated errors, and timeout/interval IDs are tracked so plugin close can clear them.
|
||||
|
||||
@ -4,36 +4,32 @@ Backend (`app.config`, `PENPOT_*` env vars) is parameterized; deployments choose
|
||||
|
||||
## Services
|
||||
|
||||
- **PostgreSQL**: durable store. Profiles, teams, files, sessions, audit, `storage_object` metadata, the `job` work queue (plus the dormant legacy `task` table), `scheduled_task` cron registry, migrations. File-data also lives here when the file-data backend is `legacy-db`/`db`. One shared DB across all backends.
|
||||
- **PostgreSQL**: durable store. Profiles, teams, files, sessions, audit, `storage_object` metadata, the `task` queue, `scheduled_task` cron registry, migrations. File-data also lives here when the file-data backend is `legacy-db`/`db`. One shared DB across all backends.
|
||||
- **Redis (Valkey-compatible)**: per-backend message bus and cache. Concrete uses: msgbus Pub/Sub for collaborative-editing broadcasts and team/profile-org notifications fired by RPC handlers (`app.rpc.notifications`, `files_update`, `teams`, `websocket`); file-summary cache gated by `enable-redis-cache`; rate-limit counters; and the dispatcher→runner work hand-off list `penpot.worker.queue:<tenant>:<queue>`. `PENPOT_REDIS_URI`. A burst of client-side `RedisCommandTimeoutException` (2s on `DEL`/`SET`, 10s on `BLPOP`) with the exporter healthy is the JVM paused or starved, not Redis: the timeout fires because nothing reads the reply.
|
||||
- **Object storage**: backends `:s3` and `:fs`. S3 in prod; devenv uses RustFS. Holds uploaded media, file-data when the file-data backend is `storage`, exports. Backend-side details (resolve, dedup, bucket set, object lifecycle, and file-data backends): `mem:backend/storage`.
|
||||
- **SMTP mailer**: invitations, password resets, email verification (sent via the `:sendmail` worker job).
|
||||
- **SMTP mailer**: invitations, password resets, email verification (sent via the `:sendmail` worker task).
|
||||
- **LDAP** (optional auth provider): helpers in `app.auth.*`, gated by `enable-login-with-ldap`.
|
||||
|
||||
## Task queue and worker model
|
||||
|
||||
Async jobs are enqueued via `jobs/submit` (`app.jobs`), which inserts a row into the shared Postgres `job` table tagged with `queue = "<tenant>:<queue-name>"`. Submission is **fire-and-forget** — RPC handlers never poll and never wait. The completion signal is the `job` row's `status` / `completed_at` columns, which the creation commands read back to answer with the state of the job they just created. A job submitted with `::jobs/profile-id` also publishes a `:job-event` message on that profile's msgbus topic after each transaction commits, and the binfile commands (`create-export-binfile-job`, `create-import-binfile-job`) are the callers that pass that owner, so the progress of their jobs does reach the msgbus. Soft-delete RPCs return immediately after marking the top-level row, leaving the cascade and reaping to workers.
|
||||
Async tasks are enqueued via `wrk/submit!` (`app.worker`), which inserts a row into the shared Postgres `task` table tagged with `queue = "<tenant>:<queue-name>"`. Submission is **fire-and-forget** — RPC handlers never poll, never wait, and workers never publish to msgbus. The only completion signal is the `task` row's `status` / `completed_at` columns, which nothing in `rpc/` reads. Soft-delete RPCs return immediately after marking the top-level row, leaving the cascade and reaping to workers.
|
||||
|
||||
Workers run on backends with `enable-backend-worker` in `PENPOT_FLAGS`. Each worker-enabled backend has a `dispatcher` (polls `job` with `FOR UPDATE SKIP LOCKED`, marks status='scheduled', RPUSHes the JSON `[id scheduled-at]` payloads into **its own** Redis list) and one or more `runner`s per queue (BLPOP from that same local list, execute, update the Postgres row). The Redis hand-off list is purely intra-backend — cross-backend coordination happens at the Postgres row level. The queues are `default`, `webhooks`, `cron` and `binfile`: the last one carries the heavy work of a user job (`:worker-binfile-parallelism`, default 1), so a long export or import does not block the default runner.
|
||||
Workers run on backends with `enable-backend-worker` in `PENPOT_FLAGS`. Each worker-enabled backend has a `dispatcher` (polls `task` with `FOR UPDATE SKIP LOCKED`, marks status='scheduled', RPUSHes claimed task IDs into **its own** Redis list) and one or more `runner`s per queue (BLPOP from that same local list, execute, update the Postgres row). The Redis hand-off list is purely intra-backend — cross-backend coordination happens at the Postgres row level.
|
||||
|
||||
## Cross-backend safety
|
||||
|
||||
Postgres row locking is the only correctness primitive: `job` claims via `FOR UPDATE SKIP LOCKED`, cron firing via `FOR UPDATE SKIP LOCKED` on the `scheduled_task` row, plus job-handler-internal locks (e.g. `file_gc_scheduler` locks candidate file rows). This makes the work-claim path safe across any number of worker-enabled backends.
|
||||
Postgres row locking is the only correctness primitive: `task` claims via `FOR UPDATE SKIP LOCKED`, cron firing via `FOR UPDATE SKIP LOCKED` on the `scheduled_task` row, plus task-handler-internal locks (e.g. `file_gc_scheduler` locks candidate file rows). This makes the work-claim path safe across any number of worker-enabled backends.
|
||||
|
||||
Two known race patterns survive multi-backend operation:
|
||||
|
||||
- **Cron dedup is best-effort.** The lock on `scheduled_task` is released when the task body finishes. If two backends' cron timers fire for the same scheduled instant with a gap larger than the task body's runtime, both execute it. Penpot's cron entries are idempotent (`session-gc`, `objects-gc`, `storage-gc-*`, `storage-pending-gc`, `jobs-gc`, `file-gc-scheduler`); the exceptions are `:telemetry` (would double-report) and `:audit-log-consumer-nexus` / `:audit-log-consumer-admin-console` (depend on archive target idempotency).
|
||||
- **`jobs/submit ::dedupe true`** does a non-atomic `DELETE` then `INSERT`. Concurrent cross-backend submits can both bypass the `DELETE` (each sees the other's uncommitted insert as absent) and end up with duplicate `'new'` rows. Each row claims and runs once independently, so the underlying work is fine; the "at most one pending" guarantee weakens.
|
||||
- **Cron dedup is best-effort.** The lock on `scheduled_task` is released when the task body finishes. If two backends' cron timers fire for the same scheduled instant with a gap larger than the task body's runtime, both execute it. Penpot's cron entries are idempotent (`session-gc`, `objects-gc`, `storage-gc-*`, `tasks-gc`, `upload-session-gc`, `file-gc-scheduler`); the exceptions are `:telemetry` (would double-report) and `:audit-log-archive` (depends on archive target idempotency).
|
||||
- **`wrk/submit! ::dedupe true`** does a non-atomic `DELETE` then `INSERT`. Concurrent cross-backend submits can both bypass the `DELETE` (each sees the other's uncommitted insert as absent) and end up with duplicate `'new'` rows. Each row claims and runs once independently, so the underlying work is fine; the "at most one pending" guarantee weakens.
|
||||
|
||||
Penpot in production lives with both: horizontal-scale deployments accept "exactly-once" as "essentially-once for idempotent operations." Devenv parallel instances handle it by running workers only on ws0 (see `mem:devenv/core`).
|
||||
|
||||
## Jobs observability
|
||||
|
||||
Worker-enabled, writable backends publish unified jobs metrics through the normal Prometheus registry. Counters describe submit, dispatch, terminal outcomes, retries, orphan recovery, rescheduling and total stored job events (unlabelled); histograms separate queue wait, handler execution and total job age. SQL-backed events are recorded after the outer PostgreSQL transaction commits. The `::jobs-metrics/sampler` component publishes backlog by status and the oldest pending-job age every 30 seconds. All labels are bounded operational values; job IDs, profile IDs, params and error text are excluded. Read-only backends do not start the sampler or the workers.
|
||||
|
||||
## Draining one instance
|
||||
|
||||
Each backend has a per-process availability flag (`app.health`, a global atom). While offline, `/readyz` answers `503 OFFLINE` without querying the database, so the load balancer removes that instance from rotation while requests in flight still finish. Operator surface: `app.srepl.main/set-offline` / `set-online`; a restart comes back online. In-flight worker jobs are safe: jobs claimed as `scheduled` return to `new` after 5 minutes via `sql:reschedule-lost` and another instance picks them up.
|
||||
Each backend has a per-process availability flag (`app.health`, a global atom). While offline, `/readyz` answers `503 OFFLINE` without querying the database, so the load balancer removes that instance from rotation while requests in flight still finish. Operator surface: `app.srepl.main/set-offline` / `set-online`; a restart comes back online. In-flight worker tasks are safe: tasks claimed as `scheduled` return to `new` after 5 minutes via `sql:reschedule-lost` and another instance picks them up.
|
||||
|
||||
## Touching production data
|
||||
|
||||
@ -43,17 +39,12 @@ Production runs the deployed toolbox, not the devenv one:
|
||||
- **Start from the ids, not from an inventory.** The ids come from the error or the logs. Unbounded queries over tables with tens of millions of rows (`storage_object`, `file_media_object`) do not finish in a reasonable time; bounded to an id list they are index lookups.
|
||||
- **One query per reference column.** Two columns in one `IN (col_a, col_b)` lose the index and scan the whole table; split them.
|
||||
- The read-only replica cuts long queries: see `mem:prod-infra/grafana`.
|
||||
- A migration that builds an index runs inside the migration transaction, so it cannot use `CONCURRENTLY`, and a plain `CREATE INDEX` locks the table against writes for the whole build. On a large table build the index out of band, before deploying, with `CREATE INDEX CONCURRENTLY` on its own (never inside a transaction block). A concurrent build interrupted midway leaves the index invalid: drop it and retry (check `pg_index` and `pg_index.indisvalid`).
|
||||
- A partitioned parent cannot take `CREATE INDEX CONCURRENTLY` (`cannot create index on partitioned table ... concurrently`). Build one index per partition with `CONCURRENTLY`, then `CREATE INDEX ... ON ONLY` on the parent and `ALTER INDEX ... ATTACH PARTITION` for each child; the parent index only becomes valid when every partition is attached.
|
||||
- Write the migration as plain, idempotent SQL: `ADD COLUMN IF NOT EXISTS`, `CREATE INDEX IF NOT EXISTS`, and `DROP CONSTRAINT IF EXISTS` followed by `ADD CONSTRAINT`. Do not encode shape checks as `pg_catalog` introspection: they are easy to get wrong (a missing `NOT condeferred` accepts an `INITIALLY DEFERRED` foreign key) and they duplicate what the runbook verifies by hand. `IF NOT EXISTS` matches the name, not the definition, so the out-of-band procedure verifies the index shape and completeness.
|
||||
- The runbook is the contract for a large instance: it builds and verifies the out-of-band work and then marks the migration step as applied (`INSERT INTO migrations (module, step) ...`), so the body does not run at startup. Verify the column, the pending rows, the index (valid, no predicate or expression, every partition attached) and the constraint shape before inserting the row; the marker is the only thing that skips the migration.
|
||||
- A reference that a GC task resolves must be `DEFERRABLE INITIALLY IMMEDIATE`, never `INITIALLY DEFERRED`: the immediate check fires inside the transaction that catches the violation and retries the chunk; the deferred one fires at commit, outside the catch. A validated foreign key belongs out of band: `ADD FOREIGN KEY` runs as an `ALTER TABLE` and it scans the whole referencing table; `NOT VALID` on a partitioned table only exists since PostgreSQL 18.
|
||||
- A backfill that removes a key an old reader uses (for example `metadata.storage-ref-id`, which the previous storage GC scans) must run with the old code stopped. Otherwise the old reader loses the reference and can delete a live object, and the rows it writes in the meantime miss the backfill.
|
||||
- `ADD COLUMN` takes a brief `ACCESS EXCLUSIVE` even when `IF NOT EXISTS` skips it, and `ADD FOREIGN KEY` takes `ACCESS EXCLUSIVE` on the referencing table and `SHARE ROW EXCLUSIVE` on the referenced one. A migration accepts those locks; a deployment that cannot take them runs the runbook and marks the step as applied.
|
||||
- A migration that builds an index runs inside the migration transaction, so it cannot use `CONCURRENTLY`, and a plain `CREATE INDEX` locks the table against writes for the whole build. On a large table (file saves write to `file_change` on every save) build the index out of band, before deploying, with `CREATE INDEX CONCURRENTLY` on its own (never inside a transaction block); the shipped migration uses `IF NOT EXISTS`, finds it and does nothing. A concurrent build interrupted midway leaves the index invalid: drop it and retry (check `pg_indexes` and `pg_index.indisvalid`).
|
||||
|
||||
## See also
|
||||
|
||||
- Devenv composition and the ws0-only worker placement: `mem:devenv/core`.
|
||||
- Storage backend resolution, dedup, bucket behavior, object lifecycle, and file-data lifecycle: `mem:backend/storage`.
|
||||
- Storage operation metrics (S3 API calls, logical ops, dedup, asset requests) and the logical-bucket vs physical-target distinction: `mem:backend/storage` (Metrics section).
|
||||
- Storage operation metrics (S3 API calls, logical ops, dedup, asset requests) and the logical-bucket vs physical-target distinction: `mem:backend/storage` (Metrics section).
|
||||
- Grafana (datasource ids and how to use them, main dashboard, panel conventions and gotchas): `mem:prod-infra/grafana`.
|
||||
|
||||
@ -36,7 +36,6 @@ From `render-wasm/`:
|
||||
- Rust tests: `./test` or `cargo test <name>`.
|
||||
- Cross-cutting testing principles and anti-patterns: `mem:testing`.
|
||||
- Lint: `./lint`.
|
||||
- Format: `./fmt`; check only: `./fmt --check`.
|
||||
- `./build`, `./test`, `./lint` and `./fmt` all run `setup`, which selects the Rust version pinned in `_build_env` (skip with `PENPOT_WASM_PREPARED=1`).
|
||||
- Format check: `cargo fmt --check`.
|
||||
|
||||
Do not change exported WASM function signatures without updating the corresponding frontend bridge and verifying the frontend renderer path.
|
||||
@ -1,15 +1,12 @@
|
||||
# Changelog helper
|
||||
|
||||
`scripts/changelog.py` holds the checks used by the changelog flow. The rules and the step-by-step live in `mem:workflow/updating-changelog`; this memory documents the subcommands. Calls `scripts/gh.py` via subprocess; never call the GitHub API directly from this script.
|
||||
`scripts/changelog.py` holds the checks used by the `update-changelog` skill (extracted from the skill file so the skill stays workflow-only). Calls `scripts/gh.py` via subprocess; never call the GitHub API directly from this script.
|
||||
|
||||
## Subcommands
|
||||
|
||||
- `check-merged <PR...>` (`--file`, `--stdin` accepted) — warn on any non-merged PR, exit 1 when found; run before writing changelog entries.
|
||||
- `boom <MILESTONE> [--changes CHANGES.md --write]` — generate `### :boom: Breaking changes & Deprecations` from the closed, non-excluded milestone issues labelled `breaking change`, sorted by issue number. Each issue becomes a `#### <issue title>` block: content = body of the issue's last `## Breaking change note` comment (whole markdown, headings demoted to `#####` outside code fences; fallback = heading plus refs with a stderr warning), refs = issue link plus merged closing PRs on their own line, with `(by @author)` for `community contribution`. `--write` replaces only that subsection of the version (inserts it after the version header when missing, keeps the heading when there are no entries; idempotent). Print-only by default. Uses `gh.py comments`.
|
||||
- `security <MILESTONE> [--changes CHANGES.md --write]` — generate `### :lock: Security fixes` from the advisories whose `patched_versions` include the version (via `gh.py advisories --patched-version`), sorted by GHSA id. Entry: `- Fix <description> (<advisory URL>) (PR: [#N](...))`; description = `Fix` + advisory summary, first word decapitalized unless it starts with an acronym; PRs = merged milestone PRs labelled `pr pt` whose body contains the GHSA id, comma-separated, and no PR is legal. Warns on stderr when a `pr pt` PR references an advisory with no matching `patched_versions`. `--write` replaces only that subsection, placed after `:rocket:` (or `:boom:`, or the version header when neither exists); empty entries remove the subsection; idempotent. Print-only by default.
|
||||
- `cross-ref <MILESTONE> [--changes CHANGES.md]` — fetch milestone PRs with `--state all` once, report merged PRs missing from the version section plus CLOSED (unmerged) warnings. PR links inside `:lock:` entries count as referenced; advisory URLs never appear here by design.
|
||||
- `dev-only-bugs <MILESTONE> [--apply]` — list closed milestone bugs that never reached production, without `no changelog` (bugs carrying `release blocker` are analyzed too). Sure: parent issue is a `Feature`/`Enhancement` of the same milestone; `--apply` adds `no changelog` to these only. Candidates (never labelled automatically): parent is an epic of the milestone, `regression` label, or the "Affected versions" / "Environment" body field names only hourly, develop, staging, PRE or the milestone version itself (`only_unreleased_envs`; any production hint or other version keeps the bug in — versions compare as full semver with missing patch = 0, so for milestone 2.19.2 the released 2.19.0 keeps the bug in). Reads `body` and `parent` from one batched `gh.py issue` lookup.
|
||||
- `report <MILESTONE> [--changes CHANGES.md --output CHANGES-ISSUES.md]` — overwrite the anomaly report: A/B milestone mismatches, E `:boom:` entry without the label, F `:rocket:` sync, G `:boom:` sync, plus the C/D `:rocket:` gaps and the `dev-only-bugs` lists (warnings). `:boom:`/`:rocket:` rule violations are fixed by regenerating or relabelling, not by editing entries in place.
|
||||
- `cross-ref <MILESTONE> [--changes CHANGES.md]` — fetch milestone PRs with `--state all` once, report merged PRs missing from the version section plus CLOSED (unmerged) warnings; GHSA entries never appear here by design.
|
||||
- `report <MILESTONE> [--changes CHANGES.md --output CHANGES-ISSUES.md]` — overwrite the anomaly report (milestone mismatches type A/B plus `:rocket:` gaps C/D); rule violations never belong in the report, they are fixed in CHANGES.md first.
|
||||
|
||||
## Conventions
|
||||
|
||||
@ -18,5 +15,4 @@
|
||||
- Version-section lookup accepts an optional `(suffix)` after the version header (e.g. `(Unreleased)`); a missing section is an error for `cross-ref` and an empty section for `report`.
|
||||
- PR/issue reference parsing covers standard `[#N](.../pull/N)` / `[#N](.../issues/N)` links plus legacy `PR:[N]` / `[Github #N]` forms.
|
||||
- Shared parsing helpers (`extract_version_section`, `extract_subsection`, `collect_changelog_prs/issues`) live at module top so future subcommands reuse them instead of duplicating regexes.
|
||||
- Report anomaly types: A/B milestone mismatches, E `:boom:` entry whose issue lacks the `breaking change` label (counted, entries preserved — human labels the issue or moves the entry), G `breaking change` issue missing from `:boom:` (counted, same contract as F); C/D are `:rocket:` gaps (warnings, never counted).
|
||||
- `issue_excluded` (closed, non-Task, non-Rejected, no `no changelog` label) is module-level and shared by `report` and `boom` so the exclusions cannot drift.
|
||||
- Report anomaly types: A/B milestone mismatches, E `:boom:` entry whose issue lacks the `breaking change` label (counted, entries preserved — human labels the issue or moves the entry); C/D are `:rocket:` gaps (warnings, never counted).
|
||||
|
||||
@ -1,15 +1,8 @@
|
||||
# CI (scripts/ci)
|
||||
|
||||
`scripts/ci` runs CI-style checks — lint, tests, format — and, on request,
|
||||
the production build of each module, for one or more monorepo modules, and
|
||||
prints a per-task summary. It is the local equivalent of CI; use it to
|
||||
verify changes before declaring work done.
|
||||
|
||||
It also exports `PENPOT_FLAGS` with `enable-backend-asserts` appended (keeping
|
||||
whatever the caller already set), which is what puts `clojure.core/assert`
|
||||
into the compiled test code. Prefer it over calling `clojure -M:dev:test` by
|
||||
hand for that reason alone: without the flag the suite passes while every
|
||||
assertion is compiled out. GitHub Actions sets the same flag.
|
||||
`scripts/ci` runs CI-style checks — lint, tests, format — for one or more
|
||||
monorepo modules and prints a per-task summary. It is the local equivalent
|
||||
of CI; use it to verify changes before declaring work done.
|
||||
|
||||
## When to use
|
||||
|
||||
@ -26,7 +19,6 @@ Run from the repo root:
|
||||
|
||||
```bash
|
||||
./scripts/ci MODULE... # lint + test + fmt per module
|
||||
./scripts/ci --build MODULE # production build only
|
||||
./scripts/ci --all --no-test # lint + fmt on all modules
|
||||
./scripts/ci --lint frontend # lint only
|
||||
./scripts/ci --fix --no-test frontend # format files, skip tests
|
||||
@ -34,15 +26,14 @@ Run from the repo root:
|
||||
./scripts/ci --dry-run --all # preview what would run
|
||||
```
|
||||
|
||||
Modules: `frontend backend common render-wasm exporter mcp plugins library
|
||||
media-processor`.
|
||||
Modules: `frontend backend common render-wasm exporter mcp plugins library`.
|
||||
|
||||
Flags:
|
||||
|
||||
- Default tasks: `lint`, `test`, `fmt` (format check; `--fix` formats
|
||||
instead). Build is **not** in the default set.
|
||||
- `--lint` / `--test` / `--fmt` / `--build` run one task only; `--no-lint` /
|
||||
`--no-test` / `--no-fmt` / `--no-build` drop one task from the default set.
|
||||
instead).
|
||||
- `--lint` / `--test` / `--fmt` run one task only; `--no-lint` /
|
||||
`--no-test` / `--no-fmt` drop one task from the default set.
|
||||
- `--paren-repair` runs only the delimiter repair — it wraps
|
||||
`scripts/paren-repair` over each module's Clojure/CLJS sources; see
|
||||
`mem:scripts/paren-repair`.
|
||||
@ -51,49 +42,6 @@ Flags:
|
||||
output; `--dry-run` prints commands without running; `--clean` removes
|
||||
the log directory.
|
||||
|
||||
## Build task
|
||||
|
||||
`--build` compiles each module the way production does, reusing caches
|
||||
(`.shadow-cljs`, `target/`, `node_modules`) instead of clearing them. It
|
||||
does not package bundles: `manage.sh build-bundle` remains the way to
|
||||
produce deployable artifacts.
|
||||
|
||||
| Module | Command | Output |
|
||||
|---|---|---|
|
||||
| `frontend` | `NODE_ENV=production pnpm run build:app:main && pnpm run build:app:libs && pnpm run build:app:assets` | `frontend/resources/public/`, `frontend/target/` |
|
||||
| `backend` | `clojure -T:build jar` | `backend/target/penpot.jar` |
|
||||
| `common` | *(none; skipped)* | — |
|
||||
| `render-wasm` | `./build` (targets `frontend` and `export`) | `render-wasm/target/` |
|
||||
| `exporter` | `NODE_ENV=production pnpm run build:app` | `exporter/target/` |
|
||||
| `mcp` | `pnpm run build` | `mcp/*/dist` |
|
||||
| `plugins` | `pnpm run build:runtime && pnpm run build:plugins && pnpm run build:styles-example && pnpm run build:doc` | `plugins/dist/` |
|
||||
| `library` | `NODE_ENV=production clojure -M:dev:shadow-cljs release library` | `library/target/library/` |
|
||||
| `media-processor` | `pnpm run types:check && pnpm run build` | `media-processor/dist/` |
|
||||
|
||||
- Every output path above is gitignored, so a build leaves `git status`
|
||||
clean.
|
||||
- The module `build` scripts in each `scripts/build` are **not** used: they
|
||||
delete `node_modules`, wipe `resources/public` and run `pnpm install`,
|
||||
which belongs to the bundle pipeline, not to a local check.
|
||||
- `frontend` and `library` have a `build` script that clears the shadow-cljs
|
||||
cache first; the task calls the inner commands to stay incremental.
|
||||
`exporter` uses `build:app` for the same reason.
|
||||
- `common` has no artifact of its own: its compilation is exercised by its
|
||||
`test` task and by the builds of `frontend`, `backend` and `exporter`.
|
||||
- `media-processor` is the ninth module. It has no lint, so that task shows
|
||||
as skipped; its build runs `tsc --noEmit` before esbuild because esbuild
|
||||
does not type-check.
|
||||
- Cold-cache cost measured on the devenv: `library` ~120 s, `frontend` ~120 s,
|
||||
`exporter` ~80 s, `backend` ~35 s. With warm caches the whole set (8
|
||||
builds) takes ~4 min. The first run of `clojure -T:build jar` downloads
|
||||
`tools.build` if it is not in `~/.m2`.
|
||||
- `pnpm run` may trigger a dependency check/install when `node_modules`
|
||||
drifts from the lockfile (`verify-deps-before-run`), so the first run
|
||||
after a dependency change can need network.
|
||||
- `render-wasm/build` takes a lock (`_build_lock`): never run two builds at
|
||||
once in the same checkout.
|
||||
|
||||
|
||||
## Logs and exit codes
|
||||
|
||||
- Full output of every task: `.ci-logs/<module>-<task>.log`.
|
||||
@ -105,16 +53,12 @@ produce deployable artifacts.
|
||||
|
||||
## Notes
|
||||
|
||||
- `mcp` has no lint task and `media-processor` has no lint task; both show
|
||||
as skipped. `render-wasm` uses `./lint`, `./test`, and `./fmt`.
|
||||
- `scripts/test_ci.py` (unittest, `python3 scripts/test_ci.py`) covers the
|
||||
flag wiring, one build command per module and the help text through
|
||||
`--dry-run`; it executes no real build.
|
||||
- `mcp` has no lint task (shows as skipped). `render-wasm` uses `./lint`,
|
||||
`./test`, and `cargo fmt`.
|
||||
- Exports `PENPOT_FLAGS` with `enable-backend-asserts` appended to any
|
||||
existing value, so backend test runs compile in and exercise
|
||||
precondition asserts; see `mem:backend/testing`.
|
||||
- Test and build tasks are long-running (backend: `clojure -M:dev:test`;
|
||||
frontend build: ~2 min warm); use a generous timeout when calling it from
|
||||
an agent shell.
|
||||
- Test tasks are long-running (backend: `clojure -M:dev:test`); use a
|
||||
generous timeout when calling it from an agent shell.
|
||||
- Skill entry point: `.agents/skills/local-ci/SKILL.md`.
|
||||
- Testing principles and output discipline: `mem:testing`.
|
||||
|
||||
@ -11,7 +11,6 @@ repository via GraphQL and REST APIs through the authenticated `gh` CLI.
|
||||
- Comparing milestone issues against CHANGES.md to find missing entries.
|
||||
- Explicitly linking a GitHub issue to a pull request.
|
||||
- Listing or inspecting GitHub Security Advisories (GHSA).
|
||||
- Fetching issue comments (the `## Breaking change note` markdown for the changelog).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
@ -40,7 +39,7 @@ python3 scripts/gh.py issues "2.16.0" --label "bug"
|
||||
python3 scripts/gh.py issues "2.16.0" --label "bug,regression"
|
||||
|
||||
# Exclude by label
|
||||
python3 scripts/gh.py issues "2.16.0" --exclude "no changelog"
|
||||
python3 scripts/gh.py issues "2.16.0" --exclude "release blocker,no changelog"
|
||||
|
||||
# Show only issues NOT yet in CHANGES.md
|
||||
python3 scripts/gh.py issues "2.16.0" --compare CHANGES.md
|
||||
@ -81,17 +80,7 @@ python3 scripts/gh.py prs --milestone "2.16.0" --state all
|
||||
|
||||
Fetch issue details by number in batches of 50 (same input styles as `prs`: numbers, `--file`, `--stdin`). No filters. A dead number fails its whole batch (unlike PRs, issues error instead of resolving to null), so the batch falls back to one-by-one lookups and reports misses as `{"number", "error": "not_found"}`. Any other failure aborts.
|
||||
|
||||
**Output**: JSON array to stdout; progress to stderr. Each issue carries the shared issue fields plus `body` and `parent` (`number`, `issue_type`, `milestone`, or null); `changelog.py dev-only-bugs` reads both.
|
||||
|
||||
### `comments`
|
||||
|
||||
Fetch every comment of one or more issues via REST (paginated, 100 per page), oldest first. Same input styles as `issue`: numbers, `--file`, `--stdin`.
|
||||
|
||||
```bash
|
||||
python3 scripts/gh.py comments 11235 11236
|
||||
```
|
||||
|
||||
**Output fields per comment**: `issue`, `id`, `author`, `created_at`, `body`. JSON array to stdout; progress to stderr. `changelog.py boom` reads the `## Breaking change note` comment from this output.
|
||||
**Output**: JSON array to stdout; progress to stderr.
|
||||
|
||||
### `link-issue`
|
||||
|
||||
@ -102,7 +91,7 @@ python3 scripts/gh.py link-issue <ISSUE_NUMBER> <PR_NUMBER>
|
||||
# Short alias: python3 scripts/gh.py link <ISSUE_NUMBER> <PR_NUMBER>
|
||||
```
|
||||
|
||||
The command resolves both node IDs and calls `addCloseIssueReferences`. It trusts the successful mutation instead of re-querying, because GitHub does not reliably report mutation-created links through `closedByPullRequestsReferences(userLinkedOnly: true)`. It is safe to rerun and works for merged PRs. It does not guarantee the issue stays open: adding `Closes #N` to the body of an already-merged PR closed the issue immediately when tried (#12896 / PR #12067, 2026-11-09). JSON goes to stdout; progress and errors go to stderr; a missing issue/PR or a failed mutation exits non-zero.
|
||||
The command resolves both node IDs and calls `addCloseIssueReferences`. It trusts the successful mutation instead of re-querying, because GitHub does not reliably report mutation-created links through `closedByPullRequestsReferences(userLinkedOnly: true)`. It is safe to rerun, works for merged PRs, and does not close an issue retroactively. JSON goes to stdout; progress and errors go to stderr; a missing issue/PR or a failed mutation exits non-zero.
|
||||
|
||||
### `advisories`
|
||||
|
||||
@ -118,9 +107,6 @@ python3 scripts/gh.py advisories --severity critical
|
||||
# Filter by state
|
||||
python3 scripts/gh.py advisories --state triage
|
||||
|
||||
# Filter by patched version declared in the advisory
|
||||
python3 scripts/gh.py advisories --patched-version 2.19.1
|
||||
|
||||
# Get full detail for a single advisory
|
||||
python3 scripts/gh.py advisories GHSA-xvj6-fh9w-gjw7
|
||||
```
|
||||
@ -129,8 +115,6 @@ python3 scripts/gh.py advisories GHSA-xvj6-fh9w-gjw7
|
||||
|
||||
**Detail output** (single advisory) adds: description, vulnerabilities (package, version ranges), credits, timestamps.
|
||||
|
||||
`--patched-version` filters the raw advisories before formatting: the match is an exact token of `vulnerabilities[*].patched_versions` (split on `,`/`;`), so `2.19.1` never matches `2.19.10`. Every state is included (`triage`, `draft`, `published`, `closed`, `withdrawn`). The raw `vulnerabilities` stay internal to the filter: the list output keeps the summary schema.
|
||||
|
||||
**Output**: JSON to stdout; progress to stderr.
|
||||
|
||||
## Key principles
|
||||
|
||||
@ -73,8 +73,6 @@ a test that reproduces it:
|
||||
- **One Assertion Per Concept** — each test verifies one behavior; split
|
||||
compound assertions
|
||||
- **Descriptive Test Names** — names read like specifications
|
||||
- **English Only** — test names, `testing` strings, comments, and every
|
||||
code-facing text MUST be in English; see the hard rule in `AGENTS.md`.
|
||||
|
||||
## Prefer Real Implementations Over Mocks
|
||||
|
||||
|
||||
@ -43,9 +43,10 @@ that breaks them is wrong, even if the code is right.
|
||||
- For another commit: `./scripts/check-commit -c <ref>`
|
||||
- **NEVER** hand-wave the body as "one long line". If a line exceeds 80,
|
||||
break it at a space.
|
||||
- **`Co-authored-by:` is for human co-authors only** (`Co-authored-by: Name <email>`). AI providers, agents and bots must not be credited that way; use `AI-assisted-by:` instead. `scripts/check-commit` rejects a trailer value that names an AI provider or agent (`Claude`, `Copilot`, `Cursor`, `Codex`, `Gemini`, `Aider`, `Cline`, `Windsurf`, `DeepSeek`, `OpenCode`, `devin-ai`, `amazon q`, and similar) or carries a bot identity (`[bot]`, `bot@`). `NON_HUMAN_COAUTHOR_PATTERN` is the list; extend it when a new tool starts adding trailers. A person whose own name or email matches one of those tokens (a surname like `Cline`, an address at a provider) is rejected too; amend the trailer or credit them with `Reviewed-by:`.
|
||||
- **NEVER add a `Co-authored-by:` trailer.** It is forbidden; `scripts/check-commit` fails the commit when any line matches it (case-insensitive).
|
||||
- CI runs `scripts/check-commit` on every new commit (`mem:workflow/creating-commits` is enforced by `.github/workflows/commit-checker.yml`), so a forbidden trailer fails the build.
|
||||
- Exceptions inside the body (do not wrap these): `Signed-off-by:`, `Co-authored-by:`, `AI-assisted-by:` trailers, and lines carrying a URL.
|
||||
- Exceptions inside the body (do not wrap these): `Signed-off-by:`,
|
||||
`AI-assisted-by:` trailers, and lines carrying a URL.
|
||||
|
||||
**AI-assisted-by trailer rules:**
|
||||
- Use only the model name, e.g. `mimo-v2.5`, `deepseek-v4-flash`
|
||||
@ -55,14 +56,6 @@ that breaks them is wrong, even if the code is right.
|
||||
|
||||
`:bug:` bug fix · `:sparkles:` enhancement · `:tada:` new feature · `:recycle:` refactor · `:lipstick:` cosmetic · `:ambulance:` critical fix · `:books:` docs · `:construction:` WIP · `:boom:` breaking · `:wrench:` config · `:zap:` perf · `:whale:` docker · `:paperclip:` other · `:arrow_up:` dep upgrade · `:arrow_down:` dep downgrade · `:fire:` removal · `:globe_with_meridians:` translations · `:rocket:` epic/highlight
|
||||
|
||||
## Atomic Commits
|
||||
|
||||
A PR squash-merges into one commit by default, so the split inside the branch
|
||||
does not reach `main` and nothing depends on it. When the PR asks for a
|
||||
regular merge (`mem:workflow/creating-prs` → Merge Strategy), every commit
|
||||
must stand alone: it builds, passes the linter, and needs no later commit on
|
||||
the same branch to make sense.
|
||||
|
||||
## Referencing Issues
|
||||
|
||||
Use `Closes #NNNN` (not `Fixes #NNNN`) in commit and PR body text for readable context. The explicit issue-to-PR link is assigned with `python3 scripts/gh.py link-issue <ISSUE_NUMBER> <PR_NUMBER>` as described in `mem:workflow/creating-prs`; do not rely on the keyword to create that link.
|
||||
|
||||
@ -35,36 +35,11 @@ Command what should be built. Format: `[Imperative verb] [what] in/on [where]`.
|
||||
|
||||
| Field | Rule |
|
||||
|-------|------|
|
||||
| **Labels** | `community contribution` (PRs from non-core) · `breaking change` when the change breaks compatibility (see below) · skip workflow labels (`backport candidate`, `team-qa`) · do **not** add `bug` or `enhancement` labels (use Issue Type instead) |
|
||||
| **Labels** | `community contribution` (PRs from non-core) · skip workflow labels (`backport candidate`, `team-qa`) · do **not** add `bug` or `enhancement` labels (use Issue Type instead) |
|
||||
| **Milestone** | Use the current or next planned milestone. Fetch available milestones: `gh api repos/penpot/penpot/milestones --jq '.[].title'`. If unsure, omit. |
|
||||
| **Project** | Always `Main` (project number 8). Use `--project "Main"` flag. |
|
||||
| **Issue Type** | See Issue Type section below. Cannot be set via `gh issue create` — use GraphQL after creation. |
|
||||
|
||||
## Breaking changes
|
||||
|
||||
When the issue introduces a breaking change or a deprecation, it is the unit
|
||||
`mem:workflow/updating-changelog` turns into a block under
|
||||
`### :boom: Breaking changes & Deprecations`. The section is generated from
|
||||
the issue (`scripts/changelog.py boom`), so the issue must carry both:
|
||||
|
||||
1. The `breaking change` label.
|
||||
2. A `## Breaking change note` **comment** (not a body section). Everything
|
||||
after that marker line is copied to the changelog: paragraphs, bullet
|
||||
lists, code blocks and headings are kept (headings are demoted to level 5
|
||||
in `CHANGES.md`). Write it for self-hosters: what changes, what they must
|
||||
do, and what breaks if they do nothing. Keep it short: explain the
|
||||
implications and link the runbook or docs for the detail, never paste the
|
||||
runbook itself. Links must point to a long-lived branch (`staging` or
|
||||
`main`), never to a feature branch.
|
||||
|
||||
Post the note as a new comment. Do not add a breaking-change section to the
|
||||
issue body and do not hand-write the changelog block. When the issue comes
|
||||
from a PR, reuse the text of the PR's breaking-change section.
|
||||
Do not hard-wrap the note: one line per paragraph or bullet.
|
||||
|
||||
The full changelog rules (sections, exclusions, `:rocket:`, pre-flight and
|
||||
anomaly report) live in `mem:workflow/updating-changelog`.
|
||||
|
||||
## Issue Body Template
|
||||
|
||||
Write the body to a temp file to avoid shell quoting issues:
|
||||
@ -262,9 +237,6 @@ Identify:
|
||||
description). Omit internal details. Use the templates above.
|
||||
- **Issue Type:** use the mapping table above (also handles `:bug:` /
|
||||
`:sparkles:` / `:tada:` title prefixes).
|
||||
- **Breaking changes:** when the PR body has a breaking-change section, post
|
||||
its user-facing text as a `## Breaking change note` comment on the issue and
|
||||
add the `breaking change` label (see **Breaking changes** above).
|
||||
|
||||
### Create the issue
|
||||
|
||||
@ -315,7 +287,7 @@ Add `Closes #<ISSUE_NUMBER>` to the PR body for readable context, then run the e
|
||||
python3 scripts/gh.py link-issue <ISSUE_NUMBER> <PR_NUMBER>
|
||||
```
|
||||
|
||||
The command creates the GitHub Development link by calling `addCloseIssueReferences` and trusts the successful mutation (GitHub does not reliably report mutation-created links back through the API). It is safe to rerun. Do not rely on it to keep the issue open: adding `Closes #N` to the body of an already-merged PR closed the issue immediately when tried (#12896 / PR #12067, 2026-11-09; the timeline shows a cross-reference plus a ClosedEvent with no closer). For changelog units that outcome is fine — a merged fix means a closed issue — but be aware when linking old merged PRs. Do not rely on the body keyword as the assignment operation.
|
||||
The command creates the GitHub Development link by calling `addCloseIssueReferences` and trusts the successful mutation (GitHub does not reliably report mutation-created links back through the API). It is safe to rerun and does not close an issue retroactively when the PR is already merged. Do not rely on the body keyword as the assignment operation.
|
||||
|
||||
### Clean up
|
||||
|
||||
@ -331,8 +303,6 @@ rm -f /tmp/issue-body.md /tmp/pr-body.md
|
||||
label or the author is not a core team member, add the label to the issue.
|
||||
- **Don't put implementation details in the issue body** — the issue is for
|
||||
users, QA, and changelog readers.
|
||||
- **Drop the PR's `no issue required` label** once the issue exists: it only
|
||||
marks PRs that need no changelog unit, and it is stale from then on.
|
||||
|
||||
## Creating Issues from Draft Body
|
||||
|
||||
|
||||
@ -32,19 +32,6 @@ PR titles follow commit title conventions:
|
||||
|
||||
See `mem:workflow/creating-commits` for emoji codes. Squash merge uses the PR title as the final commit subject, so title format matters.
|
||||
|
||||
## Merge Strategy
|
||||
|
||||
Policy: `CONTRIBUTING.md` → "Merging", under "Pull Requests".
|
||||
|
||||
- Squash merge is the default, so the PR's own commits never reach `main`
|
||||
and the PR title becomes the landed subject.
|
||||
- A regular merge is possible on request only. The request is the **first
|
||||
line** of the PR body, above the `**Note:**` line:
|
||||
`> **Merge:** do not squash; each commit is self-contained.`
|
||||
- Ask for it only when every commit on the branch is self-contained (builds
|
||||
and passes lint on its own). Default to squash unless the user asks for
|
||||
the opposite.
|
||||
|
||||
## Description Body
|
||||
|
||||
Include concise sections covering:
|
||||
@ -73,15 +60,13 @@ PR descriptions follow this structure:
|
||||
```
|
||||
|
||||
The "Note:" line is required at the top. Adjust if this is a manual (non-AI) PR.
|
||||
When the PR requests a regular merge instead of a squash merge, its
|
||||
`**Merge:**` line goes above the `**Note:**` line (see Merge Strategy).
|
||||
|
||||
## Writing Principles
|
||||
|
||||
- **Write for humans.** The diff shows what changed. The description explains why.
|
||||
- **Be concise.** Focus on reasoning: What was the problem? Why did it happen? How did you solve it?
|
||||
- **Prefer bullets over paragraphs.** Short bullet items, grouped by area with bold lead-ins where helpful, are far easier to digest than prose; keep any remaining paragraph to a few sentences.
|
||||
- **No manual line wraps.** Markdown renders adapting to the viewport; hard-wrapped lines degrade rendering. One line per paragraph or bullet, however long. This applies to PR descriptions and to PR/issue comments and review replies alike.
|
||||
- **No manual line wraps.** Markdown renders adapting to the viewport; hard-wrapped lines degrade rendering. One line per paragraph or bullet, however long.
|
||||
- **Skip the obvious.** Don't explain what `git diff` already shows.
|
||||
|
||||
### What NOT to Include
|
||||
@ -95,7 +80,7 @@ When the PR requests a regular merge instead of a squash merge, its
|
||||
## Explicit Issue Assignment
|
||||
|
||||
- For each GitHub issue that a PR resolves, run `python3 scripts/gh.py link-issue <ISSUE_NUMBER> <PR_NUMBER>` after creating or editing the PR. Do not rely on `Closes #NNNN` in the body; it is only human-readable context.
|
||||
- The command calls `addCloseIssueReferences` and trusts the successful mutation: GitHub does not reliably report mutation-created links back through `closedByPullRequestsReferences(userLinkedOnly: true)`, so the command exits non-zero only when a link target is missing or the mutation fails. It is safe to rerun and also works for an already merged PR. Do not rely on it to keep the issue open: adding `Closes #N` to the body of an already-merged PR closed the issue immediately when tried (#12896 / PR #12067, 2026-11-09).
|
||||
- The command calls `addCloseIssueReferences` and trusts the successful mutation: GitHub does not reliably report mutation-created links back through `closedByPullRequestsReferences(userLinkedOnly: true)`, so the command exits non-zero only when a link target is missing or the mutation fails. It is safe to rerun and also works for an already merged PR; it does not close an issue retroactively.
|
||||
- Skip this process for `Relates to #NNNN` and Taiga references, which do not represent a closing relationship.
|
||||
|
||||
## Before Opening
|
||||
|
||||
@ -1,7 +0,0 @@
|
||||
# Non-JS dependencies in the production images
|
||||
|
||||
Node and pnpm pins: `mem:workflow/updating-node` and `mem:workflow/updating-pnpm`. Only the parts the Dockerfiles cannot show:
|
||||
|
||||
- JDK: two pins that move together — `docker/devenv/Dockerfile` (`setup-jvm` stage, full JDK) and `docker/images/Dockerfile.backend` (JDK + `jlink` runtime). Same build, same sums in both. They compile and run the same backend: `manage.sh` builds the bundle inside the devenv image (`build-backend-bundle` → `build "backend"`) and `backend/scripts/run.template.sh` starts the runtime with `--enable-preview`, which requires the runtime major to match the compiler major. A runtime line behind the compiler line is a silent trap.
|
||||
- JDK line: tracks a non-LTS line, so there is no minor to pick — the last build of the line is the ceiling; moving line is a major and needs a real build to validate the `jlink` module list (`--add-modules` in `Dockerfile.backend`). JDK 27 keeps all of them; check with `bin/jlink --add-modules <list>` against the downloaded JDK before bumping. Source of the URL and both sums: the Azul metadata API (`api.azul.com/metadata/v1/zulu/packages/?java_version=<n>&os=linux&arch=<x64|aarch64>&java_package_type=jdk&archive_type=tar.gz&latest=true&release_status=ga&availability_types=CA`), then confirm the sums by downloading the tarball and running `sha256sum`.
|
||||
- ImageMagick: publish `penpotapp/imagemagick:<ver>` **before** bumping the `COPY --from` refs in the backend, exporter and devenv Dockerfiles; otherwise both builds break on the missing tag.
|
||||
@ -1,221 +0,0 @@
|
||||
# Updating the changelog
|
||||
|
||||
> Note about this memory: unlike other Serena memories (dense notes written for LLM agents, see `mem:memory-maintenance`), this document is also written for human release managers who need full context. That is why it uses plain language and examples. It is the canonical home of the changelog flow; the `update-changelog` skill is only an entry point that routes here.
|
||||
|
||||
`CHANGES.md` (repository root) is the release notes file. Every entry comes from GitHub: a milestone issue, its merged PR, or a security advisory whose `patched_versions` include the version. It is never written from scratch by hand; the only manual parts are the wording of a description when the title is not enough and the advisory summaries (polished on GitHub, the source of the generated security entries).
|
||||
|
||||
The commands live in `scripts/gh.py` (GitHub queries) and `scripts/changelog.py` (changelog checks). Their flags and internals are documented in `mem:scripts/gh` and `mem:scripts/changelog`.
|
||||
|
||||
## 1. When to run it
|
||||
|
||||
- Before a release, to fill the section of the version being released.
|
||||
- When new issues are added to an existing milestone.
|
||||
- When asked to audit or refresh `CHANGES.md`.
|
||||
|
||||
Editing `CHANGES.md` happens only on explicit request or as part of the release process; see `mem:critical-info`.
|
||||
|
||||
## 2. Sources of truth
|
||||
|
||||
| Question | Source |
|
||||
|---|---|
|
||||
| Which issues belong to the version? | GitHub milestone |
|
||||
| Which security fixes? | Advisories whose `patched_versions` include the version |
|
||||
| Which section? | Issue Type |
|
||||
| Is it a breaking change? | `breaking change` label |
|
||||
| Is it a highlight? | `release highlight` label |
|
||||
|
||||
- **Issue Type decides the section**: `Feature` → `:tada:`, `Enhancement` → `:sparkles:`, `Bug` → `:bug:`, `Task` is excluded (internal chores). Labels `bug`/`enhancement` and title emoji prefixes are often wrong or missing; do not use them.
|
||||
- **Security fixes come from advisories**: an advisory whose `patched_versions` include the version becomes a `:lock:` entry (section 7). `triage`, `draft` and `published` are all valid states; the advisory may still be unpublished when the changelog is written. A security fix may also carry a regular bug issue; both entries are legal and independent (the issue lives in `:bug:` as any other bug).
|
||||
- **Two labels override the Issue Type**: `breaking change` → an entry only under `:boom:`, never in its regular section; `release highlight` → an extra entry under `:rocket:` on top of its regular entry.
|
||||
- **Only closed issues**. An open issue in the milestone is not generated, even if someone added a provisional entry by hand before the merge.
|
||||
- **Excluded issues**: `no changelog`, `Task` issue type, `Rejected` project status. `release blocker` is a workflow label (the issue blocks the release), not a changelog exclusion: bugs carrying it are analyzed by `dev-only-bugs` (section 8, step 3), and closed non-bug issues carrying it are reviewed by hand (section 8, step 3).
|
||||
- **Bugs that never reached production** stay out: a bug found and fixed while building a feature of the same, unreleased version never affected users. The team rarely labels them, so the flow finds them with `dev-only-bugs` and labels them `no changelog` (section 8, step 3).
|
||||
- **Excluded PRs**: a PR with `release blocker` or `no issue required` does not bring an entry of its own.
|
||||
- **An issue with no milestone** belongs to another (probably private) project: it is neither changelog candidate nor anomaly.
|
||||
|
||||
## 3. Layout of `CHANGES.md`
|
||||
|
||||
The newest version goes at the top, right after the `# CHANGELOG` header. Inside a version, the section order is fixed:
|
||||
|
||||
```markdown
|
||||
# CHANGELOG
|
||||
|
||||
## 2.20.0 (Unreleased)
|
||||
|
||||
### :boom: Breaking changes & Deprecations
|
||||
|
||||
#### <Breaking change title>
|
||||
|
||||
<content>
|
||||
|
||||
[#ISSUE](https://github.com/penpot/penpot/issues/ISSUE) (PR: [#PR](https://github.com/penpot/penpot/pull/PR))
|
||||
|
||||
### :rocket: Epics and highlights
|
||||
|
||||
- Add configurable keyboard shortcuts [#9924](https://github.com/penpot/penpot/issues/9924) (PR: [#10237](https://github.com/penpot/penpot/pull/10237))
|
||||
|
||||
### :lock: Security fixes
|
||||
|
||||
- Fix <user-facing description of the vulnerability> (<advisory URL>) (PR: [#PR](https://github.com/penpot/penpot/pull/PR))
|
||||
|
||||
### :tada: Features
|
||||
|
||||
- Add <user-facing description of the feature> [#ISSUE](https://github.com/penpot/penpot/issues/ISSUE) (PR: [#PR](https://github.com/penpot/penpot/pull/PR))
|
||||
|
||||
### :sparkles: Enhancements
|
||||
|
||||
- Improve <user-facing description of the existing behaviour> [#ISSUE](https://github.com/penpot/penpot/issues/ISSUE) (PR: [#PR](https://github.com/penpot/penpot/pull/PR))
|
||||
|
||||
### :bug: Bugs
|
||||
|
||||
- Fix <user-facing description of the bug> [#ISSUE](https://github.com/penpot/penpot/issues/ISSUE) (PR: [#PR](https://github.com/penpot/penpot/pull/PR))
|
||||
```
|
||||
|
||||
- `:boom:` first, then `:rocket:`, `:lock:`, `:tada:`, `:sparkles:`, `:bug:`. Features come before enhancements so they stand out to readers deciding whether to update.
|
||||
- **This layout applies from 2.20.0.** Versions up to 2.19.x use the old one: `:bug: Bugs fixed` before `:sparkles: New features & Enhancements`, which mixes features and enhancements. Keep the old layout in those sections, including new 2.19.x patches, so a minor line reads the same. The in-app update dialog only reads `:rocket:`, so the change does not affect it.
|
||||
- Other headings exist in old versions (`:heart:` community contributions); do not add them to new versions.
|
||||
- Include a section only when it has entries. Historical versions may have an empty `:boom:` or `:rocket:` heading; `boom --write` keeps the `:boom:` heading even when there are no entries. `:lock:` is omitted (and removed on `--write`) when the version has no security advisories.
|
||||
- One blank line between the last entry of a section and the next heading.
|
||||
- No manual line wrapping: one line per bullet or paragraph, however long.
|
||||
|
||||
## 4. Regular entries (`:tada:`, `:sparkles:`, `:bug:`)
|
||||
|
||||
The entry shape is:
|
||||
|
||||
```markdown
|
||||
- <description> (by @github_username) [#ISSUE](https://github.com/penpot/penpot/issues/ISSUE) (PR: [#PR](https://github.com/penpot/penpot/pull/PR), [#PR2](https://github.com/penpot/penpot/pull/PR2))
|
||||
```
|
||||
|
||||
- **Description**: derived from the issue title, not the PR title. Strip emoji prefixes (`:bug:`, `:sparkles:`, `:tada:`) and write the user-facing behaviour in imperative mood. Examples: `Plugin API token methods fail with schema validation error` → `Fix Plugin API token methods failing with schema validation error`; `Custom uploaded font family names are not sanitized` → `Sanitize font family names on custom uploaded fonts`.
|
||||
- **Issue is the primary link**: the issue is the changelog unit, the PR is the implementation. Multiple fix PRs go comma-separated inline.
|
||||
- **Community attribution**: when the issue or its PR carries `community contribution`, add `(by @username)` before the issue link. Use the PR author, not the issue author.
|
||||
- **Closed issue with no PR** in the milestone: link the issue only, no PR reference.
|
||||
- **PR with no linked issue**: link a matching closed issue in the milestone when one exists; otherwise skip it, because the issue is the changelog unit.
|
||||
- **False-positive PR links**: a PR may claim to close an unrelated issue (ancient PR, cross-project reference). If the titles are clearly unrelated or the PR predates the issue by years, treat it as a data glitch and ignore it.
|
||||
|
||||
## 5. `:boom:`: breaking changes and deprecations
|
||||
|
||||
Each breaking change is a `####` block, not a bullet. The title is the issue title without its emoji prefix; the content comes from a comment on the issue; the references go on their own line at the end.
|
||||
|
||||
The **issue** must carry:
|
||||
|
||||
1. The `breaking change` label.
|
||||
2. A comment whose first line is `## Breaking change note`. Everything after that marker line is the changelog content: paragraphs, bullet lists, code blocks, tables, headings (they are demoted to `#####` on write), and so on. The marker itself is case-insensitive and accepts `#`, `##` or `###`; use `##`.
|
||||
|
||||
Example comment:
|
||||
|
||||
```markdown
|
||||
## Breaking change note
|
||||
|
||||
The Admin Console is now enabled by default and the `enable-admin-console` flag is no longer needed.
|
||||
|
||||
Self-hosted instances must deploy the `penpot-admin-console` service and set `PENPOT_ADMIN_CONSOLE_URI` in frontend and backend (check the [upgrade guide](https://help.penpot.app/technical-guide/getting-started/docker/#upgrade-to-2.19)).
|
||||
```
|
||||
|
||||
Generated block:
|
||||
|
||||
```markdown
|
||||
#### Enable the Admin Console by default
|
||||
|
||||
The Admin Console is now enabled by default and the `enable-admin-console` flag is no longer needed.
|
||||
|
||||
Self-hosted instances must deploy the `penpot-admin-console` service and set `PENPOT_ADMIN_CONSOLE_URI` in frontend and backend (check the [upgrade guide](https://help.penpot.app/technical-guide/getting-started/docker/#upgrade-to-2.19)).
|
||||
|
||||
[#12807](https://github.com/penpot/penpot/issues/12807) (PR: [#12475](https://github.com/penpot/penpot/pull/12475))
|
||||
```
|
||||
|
||||
Rules for the note:
|
||||
|
||||
- **Write it for admins**: what changes, what they must do, and what breaks if they do nothing. Do not paste the full runbook; explain the implications and link the runbook or docs.
|
||||
- **Link a long-lived branch**, `staging` or `main`, never a feature branch (`issue-1234`). The link may 404 until the PR merges; that is expected.
|
||||
- **Do not add issue or PR links**: the generator appends them.
|
||||
- **Do not hard-wrap the text**: one line per paragraph or bullet. The generator copies lines verbatim, so wrapped lines end up wrapped in `CHANGES.md`.
|
||||
- **One note per issue**. If several comments carry the marker, the generator uses the last one and warns.
|
||||
- **No note**: the generator warns on stderr and emits only the heading and references; fix the issue, not `CHANGES.md`.
|
||||
- **`####`, not `###`**: `###` is the level of the changelog sections. An entry at `###` would render as a sibling section and the scripts that read and rewrite `:boom:` would cut the section there. The note author never writes the `####`: the generator adds it.
|
||||
- **Only merged closing PRs** are referenced; unmerged or superseded PRs are skipped.
|
||||
- **Never hand-edit the section**. Generate it with `boom` (section 8). The note lives on GitHub; edit it there.
|
||||
|
||||
## 6. `:rocket:`: epics and highlights
|
||||
|
||||
- Lists exactly the milestone issues with the `release highlight` label, and nothing else. Product decides highlights by labelling issues; a regeneration cannot undo that choice.
|
||||
- Every entry copies the text and references of the issue's regular entry and MUST carry both an issue link and a PR reference.
|
||||
- If no issue is labelled, leave the section out; do not pick highlights yourself.
|
||||
- Never edit `:rocket:` by hand: add or remove the label and regenerate.
|
||||
|
||||
## 7. `:lock:`: security fixes
|
||||
|
||||
Generated from the advisories that declare the version in `patched_versions`; never written by hand.
|
||||
|
||||
- **Source and filter**: `python3 scripts/gh.py advisories --patched-version <VERSION>` keeps the advisories whose `vulnerabilities[*].patched_versions` contain the version as an exact token (so `2.19.1` never matches `2.19.10`). Any state is valid — `triage`, `draft`, `published` — because the advisory may still be unpublished when the changelog is written.
|
||||
- **Fill the advisories first**: `patched_versions` must be filled before generating the changelog. The release process publishes the advisories with (or after) the release; the changelog does not wait for publication.
|
||||
- **PR reference**: a merged PR in the milestone labelled `pr pt` whose body contains the GHSA id contributes its number inline; several PRs go comma-separated. An advisory with no PR is legal: the entry links only the advisory.
|
||||
- **Missing `patched_versions`**: when a `pr pt` PR references an advisory that declares no matching `patched_versions`, `security` prints a warning on stderr and leaves it out. Fix the advisory, not `CHANGES.md`.
|
||||
- **Entry format**: `- Fix <description> (<advisory URL>) (PR: [#PR](...))`. The description is `Fix` plus the advisory summary, with the first word decapitalized unless it starts with an acronym (`SVG`, `CVE-...`).
|
||||
- **Wording**: the flow may polish the description. Re-running `security --write` rewrites it from the summary, so the durable place for wording is the advisory summary on GitHub; keep it a problem statement, because the generator adds `Fix`.
|
||||
- **Empty version**: no matching advisories → the subsection is not inserted (and is removed by `--write`).
|
||||
- **Never hand-edit the section**: generate it with `security` (section 8).
|
||||
|
||||
## 8. Flow
|
||||
|
||||
1. **Confirm the version** (a semver string such as `2.19.0`) with the user when it is not obvious.
|
||||
2. **Fetch the milestone issues**: `python3 scripts/gh.py issues "<VERSION>" --exclude "no changelog"` (closed by default; add `--state all` only to inspect open issues). `gh.py` already drops `Task` and `Rejected`; pass `--include-tasks` / `--include-rejected` only to inspect them. To find what is missing from an existing section, add `--compare CHANGES.md`.
|
||||
3. **Keep out bugs that never reached production**: `python3 scripts/changelog.py dev-only-bugs "<VERSION>"` lists them; add `--apply` to label the sure ones `no changelog` on GitHub, then re-fetch step 2. The label puts the decision on the issue, so it survives a regeneration. Bugs carrying `release blocker` are analyzed like any other bug.
|
||||
- **Sure**: a bug whose parent issue is a `Feature` or `Enhancement` of the same milestone. `--apply` labels these.
|
||||
- **Candidates**: a bug whose parent is an epic of the same milestone, that carries the `regression` label, or whose "Affected versions" / "Environment" field names only hourly, develop, staging, PRE or the milestone version itself. `regression` also marks regressions users hit in production, and the field says where the bug was seen, not since when it exists, so never label these yourself: show them to the user, who labels the ones they confirm. The anomaly report lists them too (section 10).
|
||||
- **Closed non-bug issues carrying `release blocker`** have no automatic check: `python3 scripts/gh.py issues "<VERSION>" --label "release blocker" --exclude "no changelog"` lists them (closed, non-Task). Show them to the user: the internal ones get the `no changelog` label, the rest keep their entry.
|
||||
4. **Fetch PR details when needed**: `python3 scripts/gh.py prs <numbers...>` (also `--milestone`). Use it for the PR author, labels and merge state.
|
||||
5. **Categorize** by Issue Type and the two labels (section 2), and **build the descriptions** (section 4).
|
||||
6. **Verify PR merge status** before writing: `python3 scripts/changelog.py check-merged <PR numbers...>`. Only merged PRs go in the changelog; if a closing PR was closed unmerged, look for the PR that superseded it.
|
||||
7. **Reconcile the existing text** (section 9).
|
||||
8. **Generate `:boom:`**: `python3 scripts/changelog.py boom "<VERSION>"` prints the section; add `--write` to replace it in `CHANGES.md`. The write replaces only that subsection, inserts it after the version header when missing, keeps the heading when there are no entries, and is idempotent.
|
||||
9. **Generate `:rocket:`** from the `release highlight` label (section 6).
|
||||
10. **Generate `:lock:`**: `python3 scripts/changelog.py security "<VERSION>"` prints the section; add `--write` to replace it in `CHANGES.md`. The contract lives in section 7.
|
||||
11. **Cross-check PRs**: `python3 scripts/changelog.py cross-ref "<VERSION>"`. It lists merged milestone PRs missing from the section and warns about closed (unmerged) PRs. PRs linked from `:lock:` count as referenced; an advisory with no PR is invisible here by design.
|
||||
12. **Generate the anomaly report**: `python3 scripts/changelog.py report "<VERSION>"`. It overwrites `CHANGES-ISSUES.md` with the current state (section 10).
|
||||
|
||||
## 9. Pre-flight: reconcile before writing
|
||||
|
||||
Re-fetch the milestone (do not trust cached data) and fix these in `CHANGES.md` before adding the new section:
|
||||
|
||||
1. **Duplicate across versions** → remove it from the current section; the earlier version wins for backports. The same holds after merging a patch branch into `staging`: the fix keeps its released patch entry and leaves the newer, unreleased section (the merge conflict in `CHANGES.md` is resolved by keeping both sections, newest first, minus the duplicate).
|
||||
2. **Stale milestone assignment** (the issue moved out of this milestone) → remove the entry.
|
||||
3. **Newly applied exclusion labels** (`no changelog`) → remove the entry. (`release blocker` is not an exclusion label; it never removes an entry by itself.)
|
||||
4. **Issue no longer closed, deleted, or now `Rejected`** → remove the entry.
|
||||
5. **Unmerged or moved PR reference** → fix the reference or remove the entry. A PR merged in a different milestone is an anomaly (section 10); do not remove it silently.
|
||||
6. **Issue Type changed** → to `Task`: remove the entry; to another type: move the entry to that type's section.
|
||||
7. **Breaking change out of sync** (a `breaking change` issue outside `:boom:`, or a `:boom:` entry without the label) → regenerate the section; a labelled issue has no regular entry.
|
||||
8. **Highlight out of sync** (`:rocket:` entry without the label, or a labelled issue missing) → remove it or add it.
|
||||
9. **Missing valid issues** (closed, non-excluded, referenced nowhere) → add them to their section.
|
||||
|
||||
## 10. Anomaly report
|
||||
|
||||
`report` classifies problems it cannot decide alone. The total counts A, B, E, F and G; C and D are warnings.
|
||||
|
||||
- **A — issue in this milestone, referenced PR in another milestone (or none).** The claim "this issue is fixed by this PR in this milestone" is inconsistent. Fix the milestone on the issue or the PR, or remove the entry.
|
||||
- **B — PR in this milestone whose closing issue is in another milestone.** Same fix. An issue with no milestone is not an anomaly (another, probably private, project).
|
||||
- **E — `:boom:` entry whose issue lacks the `breaking change` label.** The entry stays in the changelog: either label the issue or move the entry to its regular section.
|
||||
- **F — `:rocket:` out of sync with the `release highlight` label.** A rule violation: `:rocket:` must list exactly the labelled issues. Fix the label or re-run the flow.
|
||||
- **G — `breaking change` issue missing from `:boom:`.** A rule violation: `:boom:` must list exactly the labelled issues. Regenerate the section or relabel.
|
||||
- **C — released X.Y.0 version without a `:rocket:` section** (gap; patches never carry highlights).
|
||||
- **D — `:rocket:` entry without issue AND PR references** (gap).
|
||||
|
||||
**Bugs that may never have reached production** are warnings too (section 8, step 3): sub-issues of a milestone feature still unlabelled (run `dev-only-bugs --apply`) and candidates for the user to confirm. They never count toward the total.
|
||||
|
||||
Anything else is a rule violation, not a report item: fix it in the pre-flight and re-run.
|
||||
|
||||
## 11. Pitfalls
|
||||
|
||||
- **Open issues with provisional entries**: a PR author may add the entry before the merge. `boom` will not generate it until the issue closes; keep the hand-written block identical to what the generator would produce.
|
||||
- **Labels, not emoji**: title prefixes are cosmetic and often missing; the section comes from Issue Type and the two labels.
|
||||
- **Taiga references**: old entries link to Taiga. Resolve them to the GitHub issue via the description text or the PRs that mention the Taiga URL, then link issue and PR.
|
||||
- **Long-lived links only**: runbooks and docs referenced from a note must point to `staging` or `main`, never to a feature branch that will be deleted.
|
||||
- **Generated sections**: `:boom:`, `:rocket:` and `:lock:` are generated. Never patch them by hand; edit the label, the note comment or the advisory and re-run.
|
||||
- **Deleted or transferred issues**: an issue referenced in the changelog that returns 410/404 (deleted or moved out of the repo) must lose its entry in the pre-flight. Its security fixes, if any, stay in `:lock:` through the advisory.
|
||||
|
||||
## 12. Where the code is
|
||||
|
||||
- `scripts/gh.py`: `issues`, `issue`, `comments`, `prs`, `link-issue`, `advisories` (with `--patched-version`). Details: `mem:scripts/gh`.
|
||||
- `scripts/changelog.py`: `check-merged`, `boom`, `security`, `cross-ref`, `dev-only-bugs`, `report`. Details: `mem:scripts/changelog`.
|
||||
- Tests: `scripts/test_gh.py`, `scripts/test_changelog.py`.
|
||||
- The note contract for issue authors: `mem:workflow/creating-issues`, section **Breaking changes**.
|
||||
@ -1,16 +1,15 @@
|
||||
# Updating Node Across the Toolchain
|
||||
|
||||
No single source of truth: nothing reads `.nvmrc` at build time, and the version is repeated in a different shape per file. Six sites move together, and the image ones carry a digest beside the tag.
|
||||
No single source of truth: nothing reads `.nvmrc` at build time, and the version is repeated in a different shape per file. Six sites move together.
|
||||
|
||||
## Pin sites
|
||||
|
||||
- `.nvmrc` → `v<X.Y.Z>`, with the leading `v`.
|
||||
- `docker/devenv/Dockerfile`, `setup-node` stage → `ENV NODE_VERSION=v<X.Y.Z>`; downloads `https://nodejs.org/dist/${NODE_VERSION}/node-${NODE_VERSION}-linux-{arm64,x64}.tar.gz`, one tarball per arch, and verifies each against a `NODE_SHA256` right below its `BINARY_URL`. Both hashes come from `SHASUMS256.txt` in that same dist directory, so a bump must recompute the two; only the arch SHAs change, the structure stays.
|
||||
- `docker/devenv/Dockerfile`, `setup-node` stage → `ENV NODE_VERSION=v<X.Y.Z>`; downloads `https://nodejs.org/dist/${NODE_VERSION}/node-${NODE_VERSION}-linux-{arm64,x64}.tar.gz`. Node is the only download in that file with **no** checksum verification (pnpm, JDK and AWS CLI do verify), so bumping Node has no hash to recompute.
|
||||
- `docker/images/Dockerfile.mcp` → two stages: `FROM dhi.io/node:<X.Y.Z>-debian13-dev AS build` and `FROM dhi.io/node:<X.Y.Z>-debian13 AS image`. The release stage carries no `-dev`.
|
||||
- `docker/images/Dockerfile.media-processor` and `docker/images/Dockerfile.exporter` → single `FROM dhi.io/node:<X.Y.Z>-debian13-dev`.
|
||||
- `.github/workflows/plugins-deploy-api-doc.yml` and `.github/workflows/plugins-deploy-styles-doc.yml` → `runtime: node@<X.Y.Z>` in the `pnpm/setup` step. Its `working-directory: plugins` makes pnpm's own version come from the manifest, so this pin is the Node one; keep it equal to `.nvmrc` minus the `v`.
|
||||
- DHI tags take the bare version; only the devenv `NODE_VERSION` carries the `v`.
|
||||
- Every `dhi.io` base image in the repo (the node ones above, plus `debian-base` and `nginx`) is pinned by digest: `FROM dhi.io/<image>:<tag>@sha256:<digest>`. Tag and digest are a pair; Renovate's `Update DHI base images` commits rewrite both in one go, so a hand-made tag bump that leaves the digest behind keeps the old image.
|
||||
|
||||
## Gotchas
|
||||
|
||||
@ -18,11 +17,9 @@ No single source of truth: nothing reads `.nvmrc` at build time, and the version
|
||||
- Node 24.x is the Krypton LTS line: take the newest patch of that major unless a major bump is the point.
|
||||
- Renovate also opens these bumps, on `renovate-*` branches and with its config living outside this repo; a manual bump and Renovate's must not diverge on any of the six sites.
|
||||
- A backport merge can deliver a newer Node into `docker/` and the plugins-deploy workflows while main's `.nvmrc` stays behind, because main receives partial backports of `.github/` and `docker/` from develop. Merging main into staging therefore conflicts on the pin lines: take main's set and finish the bump (`.nvmrc` plus the `packageManager` fields via `mem:workflow/updating-pnpm`), or the six sites drift.
|
||||
- The newer version wins in both directions. A maintenance→main merge (`main-2.18.x` into `main`) can carry the newer Node while `.nvmrc` and the `packageManager` fields stay on the older pair; then adopt the newer set and finish the bump on all six sites plus the 35 `packageManager` fields.
|
||||
- Node stays on the 24.x line even when the rest of `.github/` moves on: `99db795966` (`update GitHub Actions`) took the two `plugins-deploy-*` `runtime:` pins to `node@26.13.1` on develop while `.nvmrc` stayed on `v24.22.3`. The project keeps every site on the `.nvmrc` value; resolve that conflict back to the 24.x pin instead of taking develop's.
|
||||
- Both workflows must stay valid YAML. The `python3` in the devenv has no `yaml` module, so validate with `uv run --with pyyaml python3 -c ...` instead of a bare import.
|
||||
|
||||
## Verification
|
||||
|
||||
- `rg '<old version>'` finds nothing outside `CHANGES.md` and untracked scratch files (a repo-wide grep without path filters hits multi-megabyte ClojureScript analyzer caches: always scope it). Grep `dhi\.io/node` as well: a line whose tag moved but whose digest did not still pulls the old image.
|
||||
- `rg '<old version>'` finds nothing outside `CHANGES.md` and untracked scratch files (a repo-wide grep without path filters hits multi-megabyte ClojureScript analyzer caches: always scope it).
|
||||
- After a devenv rebuild, `node --version` matches `.nvmrc` and `pnpm --version` matches `PNPM_VERSION` in `docker/devenv/Dockerfile`.
|
||||
|
||||
@ -23,12 +23,12 @@ file (never pipe tool output through filters).
|
||||
- The store survives `node_modules` cleans. It is content-addressed and
|
||||
integrity-verified, so it cannot go stale; staleness lives in
|
||||
node_modules. Only `scripts/clean-node-modules --store` removes it.
|
||||
- Every first-party `package.json` (34 files: workspace roots plus
|
||||
- Every first-party `package.json` (35 files: workspace roots plus
|
||||
members) must carry a `packageManager` field with the identical
|
||||
`pnpm@<version>+sha512.<hash>` value. Do not let them drift.
|
||||
`external/` (vendored trees with their own lifecycles), `.opencode/`,
|
||||
`.pnpm-store/`, `bundles/`, `target/` build output and
|
||||
`docker/images/bundle-*` copies are never stamped.
|
||||
`.pnpm-store/`, `bundles/` and `docker/images/bundle-*` copies are
|
||||
never stamped.
|
||||
- pnpm is a system binary everywhere (devenv image, CI runners, Docker
|
||||
images). Nothing may call corepack: it is gone from Node 25+. pnpm
|
||||
auto-downloads the `packageManager` version on mismatch
|
||||
@ -60,7 +60,7 @@ file (never pipe tool output through filters).
|
||||
3. From the repo root, on a host whose system pnpm is the target version
|
||||
(rebuilt devenv), run `scripts/sync-pnpm-version`. It resolves the
|
||||
integrity hash via `pnpm view` + node (no npm, no corepack, no
|
||||
python3) and stamps the identical field into all 34 files, replacing
|
||||
python3) and stamps the identical field into all 35 files, replacing
|
||||
only the value line (each file keeps its own indent) and inserting
|
||||
the key after `"name"`/`"version"` where missing. Explicit version
|
||||
instead: `scripts/sync-pnpm-version <version>`. The explicit form works
|
||||
@ -71,7 +71,6 @@ file (never pipe tool output through filters).
|
||||
4. Run `scripts/sync-pnpm-version --install` (plain `pnpm install` in
|
||||
each of the 11 workspace roots) to refresh lockfile metadata, then
|
||||
fix any failing workspace (see gotchas) and re-run that directory.
|
||||
For a version-only bump, `pnpm install --lockfile-only` in each root is enough: it rewrites only the pnpm self-reference entries and needs no node_modules. The repo-root `pnpm-lock.yaml` is rewritten by itself on the first `pnpm` call that self-installs the target version.
|
||||
|
||||
## Gotchas
|
||||
|
||||
@ -89,17 +88,14 @@ file (never pipe tool output through filters).
|
||||
so a nested one silently forks install and lockfile behavior. Do not
|
||||
reintroduce it.
|
||||
- `media-processor/target/package.json` is a generated, gitignored copy that
|
||||
the script also stamps, so stamping and `--check` report 35 files while
|
||||
only 34 are tracked. A stale copy there makes `--check` fail with nothing
|
||||
the script also stamps, so stamping and `--check` report 36 files while
|
||||
only 35 are tracked. A stale copy there makes `--check` fail with nothing
|
||||
wrong in the repo: delete `media-processor/target` and re-run.
|
||||
- Expect metadata-only lockfile diffs when only the pnpm version moves:
|
||||
the pnpm self-reference entries, plus a new `packageManagerDependencies`
|
||||
section in lockfiles last written by older pnpm. Large diffs mean
|
||||
re-resolution; inspect them before accepting.
|
||||
- A backport merge of main into staging can bring a newer `PNPM_VERSION` in `docker/` while every `packageManager` field stays behind, since main receives partial `.github/`+`docker/` backports from develop. Resolve those conflicts with main's version and run `scripts/sync-pnpm-version <version>` in the same merge.
|
||||
- Direction does not decide the winner, the version does. A maintenance→main merge (`main-2.18.x` into `main`) can carry the newer Node and pnpm pins in `docker/` and the `plugins-deploy-*` workflows while main's `.nvmrc` and 35 `packageManager` fields stay on the older pair. Take the newer pair on every pin site (`docker/`, workflows, `.nvmrc`, `packageManager`) and run `scripts/sync-pnpm-version <version>` in the same merge; check with `--check` plus `pnpm install --lockfile-only --frozen-lockfile` per workspace.
|
||||
- Those merges conflict on the pnpm self-reference block in all 11 lockfiles (importers `packageManagerDependencies`, `packages/@pnpm/exe.*@<ver>`, `pnpm@<ver>` and the matching `snapshots` entries). Resolve each workspace's lockfile keeping its own importer/package resolution and swapping only that block to the newer version. Do not start from the older side and regenerate with `pnpm install --lockfile-only`: pnpm re-resolves every changed range upward (this merge took root `@playwright/test` from 1.63.0 to 1.64.1 and `@modelcontextprotocol/server` from 2.3.1 to 2.4.0), which pollutes the merge with unrelated bumps. If a workspace later needs a real re-resolution after a manifest bump, `pnpm install --lockfile-only --fix-lockfile` accepts the hand-merged file and only fills what the manifests demand.
|
||||
- `pnpm install` cannot run in `frontend` or `exporter` when `github.com` does not resolve: the `@penpot/svgo` git dependency triggers a `git fetch`. Frontend's lockfile entry carries no integrity, so pnpm always refetches; exporter's has one and still works. For frontend, resolve the self-reference hunks only.
|
||||
|
||||
## Verification
|
||||
|
||||
|
||||
@ -1,54 +1,42 @@
|
||||
# the name by which the project can be referenced within Serena/when chatting with the LLM.
|
||||
project_name: "penpot"
|
||||
|
||||
# list of language servers to start when using the LSP backend; choose from:
|
||||
# ada al angular ansible astro
|
||||
# bash bsl clojure cpp cpp_ccls
|
||||
# crystal csharp csharp_omnisharp cue dart
|
||||
# deno elixir elm erlang fortran
|
||||
# fsharp gdscript gleam go groovy
|
||||
# haskell haxe hlsl html java
|
||||
# json julia julia_fatou kotlin latex
|
||||
# lean4 lua lua_emmylua luau markdown
|
||||
# matlab msl nextflow nix ocaml
|
||||
# pascal perl php php_devsense php_phpactor
|
||||
# php_phpantom powershell python python_basedpyright python_jedi
|
||||
# python_pyrefly python_ty qml r rego
|
||||
# ruby ruby_solargraph rust scala scss
|
||||
# solidity svelte swift systemverilog terraform
|
||||
# toml typescript typescript_vts vue wolfram
|
||||
# yaml zig
|
||||
# list of languages for which language servers are started (LSP backend only); choose from:
|
||||
# ada al angular ansible bash
|
||||
# bsl clojure cpp cpp_ccls crystal
|
||||
# csharp csharp_omnisharp cue dart elixir
|
||||
# elm erlang fortran fsharp gdscript
|
||||
# go groovy haskell haxe hlsl
|
||||
# html java json julia kotlin
|
||||
# latex lean4 lua luau markdown
|
||||
# matlab msl nix ocaml pascal
|
||||
# perl php php_phpactor php_phpantom powershell
|
||||
# python python_jedi python_pyrefly python_ty r
|
||||
# rego ruby ruby_solargraph rust scala
|
||||
# scss solidity svelte swift systemverilog
|
||||
# terraform toml typescript typescript_vts vue
|
||||
# yaml zig
|
||||
# (This list may be outdated; generated with scripts/print_language_list.py;
|
||||
# For the current list, see values of the LanguageServerId enum here:
|
||||
# For the current list, see values of Language enum here:
|
||||
# https://github.com/oraios/serena/blob/main/src/solidlsp/ls_config.py)
|
||||
# For some languages, there are several alternative language servers, e.g. csharp_omnisharp, ruby_solargraph.)
|
||||
# For some languages, there are alternative language servers, e.g. csharp_omnisharp, ruby_solargraph.)
|
||||
# Note:
|
||||
# - For C, use cpp
|
||||
# - For JavaScript, use typescript
|
||||
# - For Angular projects, use angular (subsumes typescript+html; requires `npm install` in the project root)
|
||||
# - For Astro projects, use astro (subsumes typescript/javascript for .astro projects; requires npm)
|
||||
# - For Svelte projects, use svelte (subsumes typescript/javascript for .svelte projects; requires npm)
|
||||
# - For Deno projects, use deno (serves the same .ts/.js files as typescript; requires the deno CLI on PATH)
|
||||
# - For SCSS / Sass / plain CSS, use scss (some-sass-language-server handles all three)
|
||||
# - For Free Pascal/Lazarus, use pascal
|
||||
# - External Python adapters may add further registered IDs; install the adapter package first
|
||||
# and then use its ID here, for example: example
|
||||
# Special requirements:
|
||||
# Some language servers require additional setup/installations.
|
||||
# Some languages require additional setup/installations.
|
||||
# See here for details: https://oraios.github.io/serena/01-about/020_programming-languages.html#language-servers
|
||||
# When using multiple language servers, the first language server that supports a given file will be used for that file.
|
||||
# The first language server is the default language and the respective language server will be used as a fallback.
|
||||
# When using multiple languages, the first language server that supports a given file will be used for that file.
|
||||
# The first language is the default language and the respective language server will be used as a fallback.
|
||||
# Note that when using the JetBrains backend, language servers are not used and this list is correspondingly ignored.
|
||||
language_servers:
|
||||
- clojure
|
||||
- typescript
|
||||
- rust
|
||||
|
||||
# same list for Serena 1.x, which only reads this key; keep both lists equal
|
||||
languages:
|
||||
- clojure
|
||||
- typescript
|
||||
- rust
|
||||
- clojure
|
||||
- typescript
|
||||
- rust
|
||||
|
||||
# the encoding used by text files in the project
|
||||
# For a list of possible encodings, see https://docs.python.org/3.11/library/codecs.html#standard-encodings
|
||||
@ -189,18 +177,3 @@ activation_command:
|
||||
# maximum time in seconds to wait for activation_command to complete before killing it (default 180s).
|
||||
# must be a positive number.
|
||||
activation_command_timeout: 180.0
|
||||
|
||||
# list of APIs (facades or facade methods, e.g. "lsp" or "lsp.get_diagnostics_for_symbol") to include in the REPL
|
||||
# that would otherwise be disabled (particularly optional methods, which are disabled by default).
|
||||
# This extends the existing inclusions (e.g. from the global configuration).
|
||||
included_apis: []
|
||||
|
||||
# list of APIs (facades or facade methods, e.g. "lsp" or "lsp.find_symbol") to exclude from the REPL.
|
||||
# This extends the existing exclusions (e.g. from the global configuration).
|
||||
excluded_apis: []
|
||||
|
||||
# The interface through which the agent (LLM) accesses Serena's functionality (overrides the global setting).
|
||||
# Valid values: tools, REPL (see the global configuration for details); leave empty to use the global setting.
|
||||
# Note: the interface is fixed at startup. If a project is activated post-init, its setting is not applied.
|
||||
agent_interface: REPL
|
||||
|
||||
|
||||
22
AGENTS.md
22
AGENTS.md
@ -10,8 +10,7 @@
|
||||
If the user pushes, treat that commit as final from the agent's side.
|
||||
- **Never edit `CHANGES.md` by hand** in commits or PRs. The changelog is
|
||||
generated from GitHub milestones during the release process; update it only
|
||||
via `mem:workflow/updating-changelog` (the `update-changelog` skill flow) or
|
||||
on explicit user request.
|
||||
via the `update-changelog` skill flow or on explicit user request.
|
||||
- **Never pipe test output directly to filters** (`| head`, `| tail`, `| grep`, etc.).
|
||||
Always redirect to a file first: `command > /tmp/output.txt 2>&1`, then read/grep the file.
|
||||
This prevents hiding test failures. See `mem:testing` for details.
|
||||
@ -28,18 +27,12 @@
|
||||
- Before `gh pr create` / `gh pr edit` → `mem:workflow/creating-prs` (title format, body structure, AI note)
|
||||
Don't infer format from the title of a previous commit/issue/PR — the memory
|
||||
is the source of truth.
|
||||
- **Everything in code MUST be in English.** Identifiers (functions,
|
||||
variables, types, namespaces), code comments, docstrings, test
|
||||
descriptions, log hints — always English, regardless of the language the
|
||||
author works in. Tests are not an exception. The only place another
|
||||
language may appear in the codebase is product copy rendered through the
|
||||
i18n system.
|
||||
|
||||
## CRITICAL: Read module memories BEFORE writing any code
|
||||
|
||||
Do this **before planning, before coding, before touching any file**:
|
||||
|
||||
1. Read `critical-info` through Serena (see "Accessing memories" below) or read `.serena/memories/critical-info.md`.
|
||||
1. Read `critical-info` (use `serena_read_memory critical-info` or read `.serena/memories/critical-info.md`).
|
||||
It describes the project structure and tells you which modules exist.
|
||||
2. From `critical-info`, identify which modules your task affects.
|
||||
3. Read each affected module's **core memory** — the name is `<module>/core`
|
||||
@ -110,12 +103,9 @@ When working on a task:
|
||||
|
||||
## Accessing memories
|
||||
|
||||
- **If Serena's `serena_repl` tool is available** (Serena 2.x): call `s.mem.read_memory(name)`
|
||||
and `s.mem.list_memories()` through it.
|
||||
- **If Serena's `read_memory` / `list_memories` tools are available** (Serena 1.x; some clients
|
||||
show them as `serena_read_memory` / `serena_list_memories`): use them.
|
||||
- Either way, `read_memory` takes a memory name (e.g. `critical-info`, `frontend/core`).
|
||||
- **If Serena is NOT available**: read the filesystem directly.
|
||||
- **If `serena_read_memory` / `serena_list_memories` tools are available**: use them.
|
||||
`serena_read_memory` takes a memory name (e.g. `critical-info`, `frontend/core`).
|
||||
- **If tools are NOT available**: read the filesystem directly.
|
||||
Memory name `mem:foo/bar` maps to file `.serena/memories/foo/bar.md`.
|
||||
|
||||
## Cross-reference convention
|
||||
@ -162,6 +152,6 @@ precision while maintaining a strong focus on maintainability and performance.
|
||||
- `scripts/nrepl-eval.mjs` — Evaluate Clojure code via nREPL (backend + frontend).
|
||||
- `scripts/check-commit` — Validate commit messages against Penpot's commit guidelines.
|
||||
- `scripts/check-fmt-clj` — Check Clojure formatting without modifying files.
|
||||
- `scripts/ci` — CI orchestration script for running lint, tests, format checks and, on request (`--build`), production builds across modules. See `mem:scripts/ci`.
|
||||
- `scripts/ci` — CI orchestration script for running lint, tests, and format checks across modules. See `mem:scripts/ci`.
|
||||
- `scripts/gh.py` — Multi-purpose GitHub CLI helper. Subcommands: `issues` (list issues in a milestone), `prs` (fetch PR details), `advisories` (list/inspect security advisories). See `python3 scripts/gh.py --help`.
|
||||
|
||||
|
||||
239
CHANGES.md
239
CHANGES.md
@ -1,184 +1,6 @@
|
||||
# CHANGELOG
|
||||
|
||||
## 2.21.0
|
||||
|
||||
### :boom: Breaking changes & Deprecations
|
||||
|
||||
### :lock: Security fixes
|
||||
|
||||
- Fix real-time collaboration WebSocket never re-validates file access after `:subscribe-file`, letting revoked users keep receiving live edit content (https://github.com/penpot/penpot/security/advisories/GHSA-m53j-2766-6jqw) (PR: [#12189](https://github.com/penpot/penpot/pull/12189))
|
||||
|
||||
### :tada: Features
|
||||
|
||||
- Auto-link libraries during import based on slugified name [#9263](https://github.com/penpot/penpot/issues/9263) (PR: [#9958](https://github.com/penpot/penpot/pull/9958))
|
||||
- Implement basic bullet lists [#12630](https://github.com/penpot/penpot/issues/12630) (PR: [#12720](https://github.com/penpot/penpot/pull/12720))
|
||||
|
||||
### :sparkles: Enhancements
|
||||
|
||||
- Improve performance of payload decoding in the workspace (by @lukel97) [#12916](https://github.com/penpot/penpot/issues/12916) (PR: [#12704](https://github.com/penpot/penpot/pull/12704))
|
||||
- Keep the search when changing library in the Swap component menu (by @desty) [#12335](https://github.com/penpot/penpot/issues/12335) (PR: [#12873](https://github.com/penpot/penpot/pull/12873))
|
||||
|
||||
### :bug: Bugs
|
||||
|
||||
- Fix workspace stuttering when switching pages in a large file (by @lukel97) [#12917](https://github.com/penpot/penpot/issues/12917) (PR: [#12811](https://github.com/penpot/penpot/pull/12811))
|
||||
- Fix workspace freezes when undoing or redoing a move of a board with many components (by @lukel97) [#12913](https://github.com/penpot/penpot/issues/12913) (PR: [#12870](https://github.com/penpot/penpot/pull/12870))
|
||||
- Fix dashboard freezes for about a second after duplicating a large file (by @lukel97) [#12914](https://github.com/penpot/penpot/issues/12914) (PR: [#12911](https://github.com/penpot/penpot/pull/12911))
|
||||
- Fix not being able to collapse a token set group after reordering it [#12625](https://github.com/penpot/penpot/issues/12625) (PR: [#12671](https://github.com/penpot/penpot/pull/12671))
|
||||
- Fix slow variant switch with the WebGL renderer [#12968](https://github.com/penpot/penpot/issues/12968) (PR: [#12985](https://github.com/penpot/penpot/pull/12985))
|
||||
- Fix login page GitHub logo showing with low contrast in the light theme [#12938](https://github.com/penpot/penpot/issues/12938) (PR: [#12943](https://github.com/penpot/penpot/pull/12943))
|
||||
- Fix leak and black screen on wasm-critical errors [#12977](https://github.com/penpot/penpot/issues/12977) (PR: [#12978](https://github.com/penpot/penpot/pull/12978))
|
||||
|
||||
## 2.20.0
|
||||
|
||||
### :boom: Breaking changes & Deprecations
|
||||
|
||||
#### Add table partitioning for file-change
|
||||
|
||||
The file history table (`file_change`) is now partitioned by file, with the same 16 partitions as the file payload table (`file_data`). Queries that filter by file, and the join between the two tables, resolve against a single partition, so they stay fast as the table grows. The rewrite also removes the dead columns and stores the table in its final layout; the space the old table held comes back when it is dropped. A save no longer writes the file features on every revision, and the team snapshot quota is a single lookup.
|
||||
|
||||
The four snapshot RPC methods (`update-file-snapshot`, `delete-file-snapshot`, `lock-file-snapshot` and `unlock-file-snapshot`) now require `file-id`. The frontend already sends it; API clients that call them must add it.
|
||||
|
||||
On small and medium instances the table is rewritten automatically on boot; there is nothing to do. On large instances (hundreds of GB in `file_change`) it does not fit in the startup transaction: run it by hand **before upgrading to 2.20.0** following the [large-instance runbook](https://github.com/penpot/penpot/blob/staging/operations/2.20.0-file-change-partition.md). The copy runs while the instance is online, followed by a cutover of a few minutes with the backend and the workers stopped. The old table is kept for rollback; check the disk headroom before starting, and drop the old table days later to get the space back.
|
||||
|
||||
[#12416](https://github.com/penpot/penpot/issues/12416) (PR: [#12417](https://github.com/penpot/penpot/pull/12417))
|
||||
|
||||
#### Protect the file data storage objects with a foreign key in the backend
|
||||
|
||||
This release moves the file data storage pointer out of `file_data.metadata->>'storage-ref-id'` and into a new `file_data.storage_object_id` column with a foreign key to `storage_object(id)`. The metadata key is gone: nothing writes or reads it.
|
||||
|
||||
What it means for self-hosted instances:
|
||||
|
||||
- The release is a coordinated restart. The new code reads the column; the migration drops `storage-ref-id` from the rows it backfills. Rolling back to the previous binary is not supported after this release runs.
|
||||
- A fresh database or a small instance needs nothing special: the startup migration does the whole job.
|
||||
- A large instance should not let the startup migration run. Build the column, the index and the foreign key online beforehand, stop the instances, run the backfill only if a row still carries `metadata.storage-ref-id`, verify the end state, and mark the migration as applied. The full procedure is documented in [`operations/2.20.0-file-data-storage-object-id.md`](https://github.com/penpot/penpot/blob/main/operations/2.20.0-file-data-storage-object-id.md).
|
||||
- If the storage backend was never enabled, there is nothing to backfill and the stopped window is just the restart.
|
||||
|
||||
If you upgrade blindly: on fresh databases and small instances the startup migration completes the change. On a large instance, the foreign key validation scan can block writes for a long time; follow the runbook instead.
|
||||
|
||||
[#12768](https://github.com/penpot/penpot/issues/12768) (PR: [#12769](https://github.com/penpot/penpot/pull/12769))
|
||||
|
||||
### :rocket: Epics and highlights
|
||||
|
||||
- Allow MCP to work with multiple open Penpot files [#11759](https://github.com/penpot/penpot/issues/11759)
|
||||
- Allow MCP to access and manage Penpot Dashboard resources [#11761](https://github.com/penpot/penpot/issues/11761)
|
||||
|
||||
### :tada: Features
|
||||
|
||||
- Add a feedback channel to the MCP server [#12490](https://github.com/penpot/penpot/issues/12490) (PR: [#12551](https://github.com/penpot/penpot/pull/12551))
|
||||
- Allow MCP to work with multiple open Penpot files [#11759](https://github.com/penpot/penpot/issues/11759)
|
||||
- Implement full text format paste within Penpot [#12047](https://github.com/penpot/penpot/issues/12047) (PR: [#12137](https://github.com/penpot/penpot/pull/12137))
|
||||
- Add team, project and file management to the Plugin API for global plugins [#12170](https://github.com/penpot/penpot/issues/12170) (PR: [#12169](https://github.com/penpot/penpot/pull/12169))
|
||||
- Serve Penpot under a subpath [#3194](https://github.com/penpot/penpot/issues/3194) (PR: [#12358](https://github.com/penpot/penpot/pull/12358))
|
||||
- Allow MCP to access and manage Penpot Dashboard resources [#11761](https://github.com/penpot/penpot/issues/11761)
|
||||
- Add OAuth support for MCP client connections [#12034](https://github.com/penpot/penpot/issues/12034) (PR: [#12480](https://github.com/penpot/penpot/pull/12480))
|
||||
|
||||
### :sparkles: Enhancements
|
||||
|
||||
- Auto-scroll sortable lists when dragging an item near their top or bottom edge (by @ShreyashAgare26) [#10478](https://github.com/penpot/penpot/issues/10478) (PR: [#11595](https://github.com/penpot/penpot/pull/11595))
|
||||
- Make WebGL the default renderer for new accounts [#11855](https://github.com/penpot/penpot/issues/11855) (PR: [#12880](https://github.com/penpot/penpot/pull/12880))
|
||||
- Prevent the hosted MCP server from running execute_code twice when the plugin is open in two tabs [#12008](https://github.com/penpot/penpot/issues/12008) (PR: [#11927](https://github.com/penpot/penpot/pull/11927))
|
||||
- Allow detaching typography tokens [#11887](https://github.com/penpot/penpot/issues/11887) (PR: [#11969](https://github.com/penpot/penpot/pull/11969))
|
||||
- Complete the Russian translations [#11955](https://github.com/penpot/penpot/issues/11955) (PR: [#11956](https://github.com/penpot/penpot/pull/11956))
|
||||
- Run .penpot file import and export as durable background jobs [#12194](https://github.com/penpot/penpot/issues/12194) (PR: [#12056](https://github.com/penpot/penpot/pull/12056), [#12192](https://github.com/penpot/penpot/pull/12192))
|
||||
- Open files and projects in a new tab with Ctrl/Cmd+click in the dashboard [#12228](https://github.com/penpot/penpot/issues/12228) (PR: [#12230](https://github.com/penpot/penpot/pull/12230))
|
||||
- Add an RPC method to read file-level plugin data [#12258](https://github.com/penpot/penpot/issues/12258) (PR: [#10171](https://github.com/penpot/penpot/pull/10171))
|
||||
- Add a get-file-tokens RPC method to read a file's design tokens [#12260](https://github.com/penpot/penpot/issues/12260) (PR: [#12172](https://github.com/penpot/penpot/pull/12172))
|
||||
- Show the progress and the size of each file in the import and export dialogs [#12340](https://github.com/penpot/penpot/issues/12340) (PR: [#12192](https://github.com/penpot/penpot/pull/12192))
|
||||
- Focus the search field when opening the color tokens tab of the color picker [#12344](https://github.com/penpot/penpot/issues/12344) (PR: [#12345](https://github.com/penpot/penpot/pull/12345))
|
||||
- Harden the file data offload job with a row lock, per-file transactions, throttle and metrics [#12398](https://github.com/penpot/penpot/issues/12398) (PR: [#12403](https://github.com/penpot/penpot/pull/12403))
|
||||
- Render email templates inside the mail job in the backend [#12445](https://github.com/penpot/penpot/issues/12445) (PR: [#12448](https://github.com/penpot/penpot/pull/12448))
|
||||
- Run local WASM exports for shapes and boards without freezing the UI [#12512](https://github.com/penpot/penpot/issues/12512) (PR: [#12515](https://github.com/penpot/penpot/pull/12515))
|
||||
- Use WASM to generate SVG markup in the code inspector when render-wasm is active [#12567](https://github.com/penpot/penpot/issues/12567) (PR: [#12608](https://github.com/penpot/penpot/pull/12608))
|
||||
- Close the workspace search with the Escape key (by @filipsajdak) [#9540](https://github.com/penpot/penpot/issues/9540) (PR: [#12374](https://github.com/penpot/penpot/pull/12374))
|
||||
- Restrict management RPC methods to their declared shared keys [#12730](https://github.com/penpot/penpot/issues/12730) (PR: [#12736](https://github.com/penpot/penpot/pull/12736))
|
||||
- Reduce memory used by duplicate shape ids in the workspace [#12791](https://github.com/penpot/penpot/issues/12791) (PR: [#12792](https://github.com/penpot/penpot/pull/12792))
|
||||
- Reduce memory consumption on shape decoding [#12789](https://github.com/penpot/penpot/issues/12789) (PR: [#12790](https://github.com/penpot/penpot/pull/12790))
|
||||
- Notify other users when a snapshot is restored [#12425](https://github.com/penpot/penpot/issues/12425) (PR: [#12555](https://github.com/penpot/penpot/pull/12555))
|
||||
- Normalize the configured tenant to lowercase [#12842](https://github.com/penpot/penpot/issues/12842) (PR: [#12843](https://github.com/penpot/penpot/pull/12843))
|
||||
- Speed up saves on large files by checking only the pages a save changes [#12602](https://github.com/penpot/penpot/issues/12602) (PR: [#12555](https://github.com/penpot/penpot/pull/12555))
|
||||
- Retry saves after short server failures and keep the workspace open when a save fails [#12601](https://github.com/penpot/penpot/issues/12601) (PR: [#12555](https://github.com/penpot/penpot/pull/12555))
|
||||
- Switch the workspace to read-only mode when the file can't be saved [#12507](https://github.com/penpot/penpot/issues/12507) (PR: [#12555](https://github.com/penpot/penpot/pull/12555))
|
||||
- Validate upload content against real file bytes in media uploads [#12796](https://github.com/penpot/penpot/issues/12796) (PR: [#12801](https://github.com/penpot/penpot/pull/12801))
|
||||
- Add the file and MCP session to audit events sent from plugins and the MCP [#12902](https://github.com/penpot/penpot/issues/12902) (PR: [#12972](https://github.com/penpot/penpot/pull/12972))
|
||||
- Add skip-if-active option to job submission in the backend jobs substrate [#12265](https://github.com/penpot/penpot/issues/12265) (PR: [#12266](https://github.com/penpot/penpot/pull/12266))
|
||||
- Add HTML paste from clipboard in text editor v3 [#10465](https://github.com/penpot/penpot/issues/10465)
|
||||
- Skip loading the thumbnail rasterizer in WebGL renderer sessions to cut workspace memory [#13029](https://github.com/penpot/penpot/issues/13029) (PR: [#12835](https://github.com/penpot/penpot/pull/12835))
|
||||
- Speed up uploading page shapes to WASM [#13030](https://github.com/penpot/penpot/issues/13030) (PR: [#12854](https://github.com/penpot/penpot/pull/12854))
|
||||
|
||||
### :bug: Bugs
|
||||
|
||||
- Fix font upload failing with a regex error [#10436](https://github.com/penpot/penpot/issues/10436) (PR: [#11951](https://github.com/penpot/penpot/pull/11951))
|
||||
- Fix dashboard file thumbnails loading slow for large multi-page files [#12053](https://github.com/penpot/penpot/issues/12053) (PR: [#12054](https://github.com/penpot/penpot/pull/12054))
|
||||
- Fix Plugin API resolving a font family token to its separate words when the family name has several words [#11748](https://github.com/penpot/penpot/issues/11748) (PR: [#11750](https://github.com/penpot/penpot/pull/11750))
|
||||
- Fix file import recreating circular library references when the exported libraries use each other [#12052](https://github.com/penpot/penpot/issues/12052) (PR: [#12062](https://github.com/penpot/penpot/pull/12062))
|
||||
- Fix asset redirect responses advertising a cache lifetime of about 1000 days instead of 24 hours (by @ners) [#12761](https://github.com/penpot/penpot/issues/12761) (PR: [#12758](https://github.com/penpot/penpot/pull/12758))
|
||||
- Fix an internal error on typography tokens due to circular dependencies on other token types [#9580](https://github.com/penpot/penpot/issues/9580)
|
||||
- Fix the MCP server rejecting request bodies over 100kb (express.json default) with PayloadTooLargeError (by @GitNimay) [#12080](https://github.com/penpot/penpot/issues/12080) (PR: [#12450](https://github.com/penpot/penpot/pull/12450))
|
||||
- Fix library component, color and typography counters showing zero when opening files from the dashboard (by @lukel97) [#12826](https://github.com/penpot/penpot/issues/12826) (PR: [#12787](https://github.com/penpot/penpot/pull/12787))
|
||||
- Fix workspace crash with Maximum call stack size exceeded when applying a library update [#12576](https://github.com/penpot/penpot/issues/12576) (PR: [#12661](https://github.com/penpot/penpot/pull/12661))
|
||||
- Fix canvas rendering an imported shape without its hole when the SVG fill rule is evenodd [#11947](https://github.com/penpot/penpot/issues/11947) (PR: [#12006](https://github.com/penpot/penpot/pull/12006))
|
||||
- Fix Plugin API component.instance() and createShapeFromSvg() nesting the new shape into whatever board is at (0, 0) of the current page (by @makesomethingshit) [#12086](https://github.com/penpot/penpot/issues/12086) (PR: [#12499](https://github.com/penpot/penpot/pull/12499))
|
||||
- Fix view mode breaking zoom, board navigation, and comments when the URL has no page id [#12684](https://github.com/penpot/penpot/issues/12684) (PR: [#12687](https://github.com/penpot/penpot/pull/12687))
|
||||
- Fix autosave stopping and valid edits being lost when part of a save fails validation [#12553](https://github.com/penpot/penpot/issues/12553) (PR: [#12555](https://github.com/penpot/penpot/pull/12555))
|
||||
- Fix dashboard duplicate copying a file without the edits that are still saving [#12603](https://github.com/penpot/penpot/issues/12603) (PR: [#12555](https://github.com/penpot/penpot/pull/12555))
|
||||
- Fix Alt measurements still showing a shape that was deleted [#12605](https://github.com/penpot/penpot/issues/12605) (PR: [#12555](https://github.com/penpot/penpot/pull/12555))
|
||||
- Fix Plugin API ignoring changes to a deleted shape without an error [#12606](https://github.com/penpot/penpot/issues/12606) (PR: [#12555](https://github.com/penpot/penpot/pull/12555))
|
||||
- Fix typography sample overflowing its row in locales with a wider sample (ko, ar) (by @desty) [#11486](https://github.com/penpot/penpot/issues/11486) (PR: [#11487](https://github.com/penpot/penpot/pull/11487))
|
||||
- Fix token not showing when selecting two text layers (by @ShreyashAgare26) [#11924](https://github.com/penpot/penpot/issues/11924) (PR: [#11944](https://github.com/penpot/penpot/pull/11944))
|
||||
- Fix inner shadow fill not working in the SVG render while it works in WebGL [#11967](https://github.com/penpot/penpot/issues/11967) (PR: [#12141](https://github.com/penpot/penpot/pull/12141))
|
||||
- Fix @penpot/plugin-styles loading Work Sans from a third-party CDN (by @ump45nose) [#12023](https://github.com/penpot/penpot/issues/12023) (PR: [#12087](https://github.com/penpot/penpot/pull/12087))
|
||||
- Fix schema error on swatch component for image fills [#12135](https://github.com/penpot/penpot/issues/12135) (PR: [#12136](https://github.com/penpot/penpot/pull/12136))
|
||||
- Fix worker storing more shape data than needed [#12740](https://github.com/penpot/penpot/issues/12740) (PR: [#12840](https://github.com/penpot/penpot/pull/12840))
|
||||
- Fix worker not freeing pages [#12738](https://github.com/penpot/penpot/issues/12738) (PR: [#12751](https://github.com/penpot/penpot/pull/12751))
|
||||
- Fix editing the L value in HSL mode resetting H and S to 0 (by @makesomethingshit) [#12615](https://github.com/penpot/penpot/issues/12615) (PR: [#12629](https://github.com/penpot/penpot/pull/12629))
|
||||
- Restrict dashboard plugins to the MCP plugin and localhost plugins [#12948](https://github.com/penpot/penpot/issues/12948) (PR: [#12949](https://github.com/penpot/penpot/pull/12949))
|
||||
- Fix workspace memory growing on every page switch with the WebGL renderer [#13028](https://github.com/penpot/penpot/issues/13028) (PR: [#12877](https://github.com/penpot/penpot/pull/12877))
|
||||
- Fix workspace showing React error #310 when the team container renders without a team id (by @filipsajdak) [#12728](https://github.com/penpot/penpot/issues/12728) (PR: [#12920](https://github.com/penpot/penpot/pull/12920))
|
||||
|
||||
## 2.19.2
|
||||
|
||||
### :bug: Bugs fixed
|
||||
|
||||
- Fix referential integrity error when copying component variants with nested components across libraries and teams [#10704](https://github.com/penpot/penpot/issues/10704) (PR: [#12928](https://github.com/penpot/penpot/pull/12928))
|
||||
- Fix migration 0033 inserting nil shape entries for dangling child references that blocked file opening [#12936](https://github.com/penpot/penpot/issues/12936) (PR: [#12979](https://github.com/penpot/penpot/pull/12979))
|
||||
- Fix switching a variant and adding a component instance taking over 10 seconds in files with many token-styled shapes [#13019](https://github.com/penpot/penpot/issues/13019) (PR: [#13021](https://github.com/penpot/penpot/pull/13021))
|
||||
- Fix autosave failing after uploading an SVG as a component in the assets panel, and on other mains named unlike their component [#12773](https://github.com/penpot/penpot/issues/12773) (PR: [#12794](https://github.com/penpot/penpot/pull/12794))
|
||||
- Fix `enable-oidc-registration` not working when password login is disabled [#12964](https://github.com/penpot/penpot/issues/12964) (PR: [#12974](https://github.com/penpot/penpot/pull/12974))
|
||||
- Fix self-hosted backend opening 60 database connections at startup and keeping them open while idle (by @UsamaSadiq) [#12622](https://github.com/penpot/penpot/issues/12622) (PR: [#13052](https://github.com/penpot/penpot/pull/13052))
|
||||
|
||||
## 2.19.1
|
||||
|
||||
### :lock: Security fixes
|
||||
|
||||
- Fix editor-role team members retrieving the signed invitation token of any pending invitation (https://github.com/penpot/penpot/security/advisories/GHSA-4mmc-j8rh-74pf) (PR: [#12892](https://github.com/penpot/penpot/pull/12892))
|
||||
|
||||
- Fix share-link page-scope escape in `get-profiles-for-file-comments` leaking the identity of comment authors outside the link's scope (https://github.com/penpot/penpot/security/advisories/GHSA-89cq-3wv7-987r) (PR: [#12888](https://github.com/penpot/penpot/pull/12888))
|
||||
|
||||
- Fix missing authorization and unbounded array in mark-all-threads-as-read that let any authenticated user write the status of inaccessible threads (https://github.com/penpot/penpot/security/advisories/GHSA-j4vc-cmf6-jhr7) (PR: [#12894](https://github.com/penpot/penpot/pull/12894))
|
||||
|
||||
- Fix incomplete SVG sanitizer escaping that let a "sanitized" SVG carry live script (CVE-2026-105694) (https://github.com/penpot/penpot/security/advisories/GHSA-qwhj-rchj-58mg) (PR: [#12887](https://github.com/penpot/penpot/pull/12887))
|
||||
|
||||
### :bug: Bugs fixed
|
||||
|
||||
- Fix component instance (variant) inside a layout resetting to its original size after a shared library update of an unrelated component [#12670](https://github.com/penpot/penpot/issues/12670) (PR: [#12851](https://github.com/penpot/penpot/pull/12851))
|
||||
|
||||
## 2.19.0
|
||||
|
||||
### :boom: Breaking changes & Deprecations
|
||||
|
||||
#### Migrate legacy file payloads to file_data and delete orphan revisions
|
||||
|
||||
File payloads now live in one place: every file state and snapshot is kept in `file_data`, and the orphan `file_change` rows that kept old payloads alive forever are removed. No read path reached those rows and nothing ever collected them, so the upgrade also reclaims the database and TOAST space they held. There is no API or UI change.
|
||||
|
||||
On small and medium instances this happens automatically on boot and there is nothing to do. On large instances (millions of snapshots, hundreds of GB in TOAST) the work does not fit in the startup transaction: run it by hand **before upgrading to 2.19.0** following the [large-instance runbook](https://github.com/penpot/penpot/blob/staging/operations/2.19.0-migrate-legacy-file-data.md). The runbook tells you whether your instance needs it, keeps the instance online while the data moves, and verifies the result before you deploy.
|
||||
|
||||
[#12784](https://github.com/penpot/penpot/issues/12784) (PR: [#12785](https://github.com/penpot/penpot/pull/12785))
|
||||
|
||||
#### Enable the Admin Console by default
|
||||
|
||||
The Admin Console is now enabled by default and the `enable-admin-console` flag is no longer needed. Self-hosted instances must deploy the `penpot-admin-console` service and set `PENPOT_ADMIN_CONSOLE_URI` in frontend and backend (check the [upgrade guide](https://help.penpot.app/technical-guide/getting-started/docker/#upgrade-to-2.19)).
|
||||
|
||||
[#12807](https://github.com/penpot/penpot/issues/12807) (PR: [#12475](https://github.com/penpot/penpot/pull/12475))
|
||||
## 2.19.0 (Unreleased)
|
||||
|
||||
### :rocket: Epics and highlights
|
||||
|
||||
@ -252,38 +74,6 @@ The Admin Console is now enabled by default and the `enable-admin-console` flag
|
||||
- Fix thumbnails failing on older Chromium browsers and each thumbnail rendering several times [#11878](https://github.com/penpot/penpot/issues/11878) (PR: [#11879](https://github.com/penpot/penpot/pull/11879))
|
||||
- Fix internal error when opening the libraries modal [#11943](https://github.com/penpot/penpot/issues/11943) (PR: [#11946](https://github.com/penpot/penpot/pull/11946))
|
||||
- Fix Plugin API `connectLibrary` linking unpublished files as libraries [#12001](https://github.com/penpot/penpot/issues/12001) (PR: [#12002](https://github.com/penpot/penpot/pull/12002))
|
||||
- Fix incorrect padding at the top of the tokens and themes section [#9585](https://github.com/penpot/penpot/issues/9585) (PR: [#12422](https://github.com/penpot/penpot/pull/12422))
|
||||
- Fix token tree path handling for repeated and nested segments (by @makesomethingshit) [#11588](https://github.com/penpot/penpot/issues/11588) (PR: [#11591](https://github.com/penpot/penpot/pull/11591))
|
||||
- Fix comment input duplicating text when confirming a Japanese IME composition with Enter [#11757](https://github.com/penpot/penpot/issues/11757) (PR: [#11768](https://github.com/penpot/penpot/pull/11768))
|
||||
- Fix file not saving after deleting a variant [#11758](https://github.com/penpot/penpot/issues/11758) (PR: [#12216](https://github.com/penpot/penpot/pull/12216))
|
||||
- Fix component copies not syncing when a shape is moved out of a main component and another change follows at once [#11978](https://github.com/penpot/penpot/issues/11978) (PR: [#11981](https://github.com/penpot/penpot/pull/11981))
|
||||
- Fix MCP plugin showing a version mismatch warning on every patch release (by @robertkodra) [#12007](https://github.com/penpot/penpot/issues/12007) (PR: [#12009](https://github.com/penpot/penpot/pull/12009))
|
||||
- Fix workspace crash when switching to the path tool while drawing a shape [#12100](https://github.com/penpot/penpot/issues/12100) (PR: [#12101](https://github.com/penpot/penpot/pull/12101))
|
||||
- Fix referential integrity error when swapping a component [#12144](https://github.com/penpot/penpot/issues/12144) (PR: [#12333](https://github.com/penpot/penpot/pull/12333))
|
||||
- Fix workspace saving text shapes on every page open when a custom font is missing [#12151](https://github.com/penpot/penpot/issues/12151) (PR: [#12164](https://github.com/penpot/penpot/pull/12164))
|
||||
- Fix swapped nested components losing their absolute position after a library update [#12176](https://github.com/penpot/penpot/issues/12176) (PR: [#12556](https://github.com/penpot/penpot/pull/12556))
|
||||
- Fix theme selector showing the editor message when the tokens source is a library [#12177](https://github.com/penpot/penpot/issues/12177) (PR: [#12203](https://github.com/penpot/penpot/pull/12203))
|
||||
- Fix untranslated texts in the This file tab of the libraries modal [#12178](https://github.com/penpot/penpot/issues/12178) (PR: [#12204](https://github.com/penpot/penpot/pull/12204))
|
||||
- Fix workspace staying in the saving state and losing unsaved edits when a save request gets no answer [#12222](https://github.com/penpot/penpot/issues/12222) (PR: [#12223](https://github.com/penpot/penpot/pull/12223))
|
||||
- Fix scrub cursor showing on numeric inputs with mixed values in the sidebar [#12227](https://github.com/penpot/penpot/issues/12227) (PR: [#12229](https://github.com/penpot/penpot/pull/12229))
|
||||
- Fix internal error when copying a swapped nested component [#12289](https://github.com/penpot/penpot/issues/12289) (PR: [#12434](https://github.com/penpot/penpot/pull/12434))
|
||||
- Show an error when uploading a custom font fails [#12304](https://github.com/penpot/penpot/issues/12304) (PR: [#12315](https://github.com/penpot/penpot/pull/12315))
|
||||
- Fix React warning about the unknown `tooltipClass` prop in layout-item size inputs [#12311](https://github.com/penpot/penpot/issues/12311) (PR: [#12329](https://github.com/penpot/penpot/pull/12329))
|
||||
- Fix guides inside a board still being interactive [#12322](https://github.com/penpot/penpot/issues/12322) (PR: [#12460](https://github.com/penpot/penpot/pull/12460))
|
||||
- Fix pasted text not being visible in an empty text box until it is resized [#12438](https://github.com/penpot/penpot/issues/12438) (PR: [#12859](https://github.com/penpot/penpot/pull/12859))
|
||||
- Fix files failing referential integrity validation staying unsaveable in every session [#12464](https://github.com/penpot/penpot/issues/12464) (PR: [#12524](https://github.com/penpot/penpot/pull/12524), [#12662](https://github.com/penpot/penpot/pull/12662))
|
||||
- Fix unicode characters not rendering in text layers with the new render engine [#12469](https://github.com/penpot/penpot/issues/12469) (PR: [#12471](https://github.com/penpot/penpot/pull/12471))
|
||||
- Fix text image fills staying blank when media loads after text layout [#12563](https://github.com/penpot/penpot/issues/12563) (PR: [#12564](https://github.com/penpot/penpot/pull/12564))
|
||||
- Fix swapped nested icons moving outside the layout after a library update [#12657](https://github.com/penpot/penpot/issues/12657) (PR: [#12716](https://github.com/penpot/penpot/pull/12716))
|
||||
- Fix garbled bullets before the element counts in the libraries modal [#12677](https://github.com/penpot/penpot/issues/12677) (PR: [#12678](https://github.com/penpot/penpot/pull/12678))
|
||||
- Fix token dropdown of numeric inputs showing the scrub cursor [#12679](https://github.com/penpot/penpot/issues/12679) (PR: [#12680](https://github.com/penpot/penpot/pull/12680))
|
||||
- Fix variant property names and values accepting `,` and `=` [#12683](https://github.com/penpot/penpot/issues/12683) (PR: [#12688](https://github.com/penpot/penpot/pull/12688))
|
||||
- Fix changing a variant property from another page leaving the main component label out of date [#12692](https://github.com/penpot/penpot/issues/12692) (PR: [#12693](https://github.com/penpot/penpot/pull/12693))
|
||||
- Fix `:referential-integrity` errors when moving variant mains in or out of a set [#12696](https://github.com/penpot/penpot/issues/12696) (PR: [#12706](https://github.com/penpot/penpot/pull/12706))
|
||||
- Fix `:referential-integrity` errors when creating, combining or renaming variant sets [#12734](https://github.com/penpot/penpot/issues/12734) (PR: [#12779](https://github.com/penpot/penpot/pull/12779))
|
||||
- Fix save errors after masking a variant with Ctrl+M [#12748](https://github.com/penpot/penpot/issues/12748) (PR: [#12749](https://github.com/penpot/penpot/pull/12749))
|
||||
- Fix icons in component instances not being shown on the canvas [#12752](https://github.com/penpot/penpot/issues/12752)
|
||||
- Fix upload endpoints accepting non-multipart upload paths and validation error bodies being truncated [#12896](https://github.com/penpot/penpot/issues/12896) (PR: [#12067](https://github.com/penpot/penpot/pull/12067))
|
||||
|
||||
### :sparkles: New features & Enhancements
|
||||
|
||||
@ -299,7 +89,7 @@ The Admin Console is now enabled by default and the `enable-admin-console` flag
|
||||
- Apply tokens to stroke per side [#9842](https://github.com/penpot/penpot/issues/9842) (PR: [#10913](https://github.com/penpot/penpot/pull/10913))
|
||||
- Add individual stroke widths per side (top, right, bottom, left) [#9854](https://github.com/penpot/penpot/issues/9854) (PR: [#10648](https://github.com/penpot/penpot/pull/10648))
|
||||
- Expand the token tree right after importing it (by @AKnassa) [#9819](https://github.com/penpot/penpot/issues/9819) (PR: [#10664](https://github.com/penpot/penpot/pull/10664))
|
||||
- Update the selection size badge live during resize, replacing the resize tooltip [#10259](https://github.com/penpot/penpot/issues/10259) (PR: [#11884](https://github.com/penpot/penpot/pull/11884))
|
||||
- Update the selection size badge live during resize, replacing the resize tooltip [#10259](https://github.com/penpot/penpot/issues/10259)
|
||||
- Add color variants, rotation correction, viewport flip and hide-when-too-small to the selection size badge [#10264](https://github.com/penpot/penpot/issues/10264) (PR: [#11884](https://github.com/penpot/penpot/pull/11884))
|
||||
- Support single-file SVG export with the WebGL render [#10546](https://github.com/penpot/penpot/issues/10546)
|
||||
- Persist the hide resolved comments preference and separate the filter options in the comments menu [#10686](https://github.com/penpot/penpot/issues/10686) (PR: [#10694](https://github.com/penpot/penpot/pull/10694))
|
||||
@ -323,36 +113,13 @@ The Admin Console is now enabled by default and the `enable-admin-console` flag
|
||||
- Make "Fix when scrolling" work in the WebGL render [#11684](https://github.com/penpot/penpot/issues/11684) (PR: [#11752](https://github.com/penpot/penpot/pull/11752))
|
||||
- Complete Catalan translations [#11739](https://github.com/penpot/penpot/issues/11739) (PR: [#11741](https://github.com/penpot/penpot/pull/11741))
|
||||
- Remove the optional client-provided id from creation RPC endpoints [#11783](https://github.com/penpot/penpot/issues/11783) (PR: [#11784](https://github.com/penpot/penpot/pull/11784))
|
||||
- Allow color tokens for the canvas background [#11791](https://github.com/penpot/penpot/issues/11791) (PR: [#11923](https://github.com/penpot/penpot/pull/11923), [#11950](https://github.com/penpot/penpot/pull/11950))
|
||||
- Allow color tokens for the canvas background [#11791](https://github.com/penpot/penpot/issues/11791) (PR: [#11923](https://github.com/penpot/penpot/pull/11923))
|
||||
- Show a "no assets yet" message for empty files in the libraries modal [#11811](https://github.com/penpot/penpot/issues/11811) (PR: [#11818](https://github.com/penpot/penpot/pull/11818))
|
||||
- Upgrade the MCP SDK to v2 and adopt the new protocol version [#11827](https://github.com/penpot/penpot/issues/11827) (PR: [#11841](https://github.com/penpot/penpot/pull/11841))
|
||||
- Remove legacy SSE support from the MCP server [#11846](https://github.com/penpot/penpot/issues/11846) (PR: [#11841](https://github.com/penpot/penpot/pull/11841))
|
||||
- Delete selected path handles with Backspace or Delete [#11858](https://github.com/penpot/penpot/issues/11858) (PR: [#11896](https://github.com/penpot/penpot/pull/11896))
|
||||
- Use the tooltip component in the token pill info [#11886](https://github.com/penpot/penpot/issues/11886) (PR: [#11900](https://github.com/penpot/penpot/pull/11900))
|
||||
- Hide the duplicate cursor when alt-dragging a shape that cannot be duplicated (by @0xTHAC0) [#11905](https://github.com/penpot/penpot/issues/11905) (PR: [#11195](https://github.com/penpot/penpot/pull/11195))
|
||||
- Document the team auto-add behavior with `disable-email-verification` [#11376](https://github.com/penpot/penpot/issues/11376) (PR: [#11425](https://github.com/penpot/penpot/pull/11425))
|
||||
- Make the workspace recover from interrupted saves and show a proper error instead of the internal error page [#11799](https://github.com/penpot/penpot/issues/11799)
|
||||
- Add component management operations to the Plugin API [#12301](https://github.com/penpot/penpot/issues/12301) (PR: [#12303](https://github.com/penpot/penpot/pull/12303))
|
||||
- Record the imported file ids in the import-binfile audit event [#12485](https://github.com/penpot/penpot/issues/12485) (PR: [#12491](https://github.com/penpot/penpot/pull/12491))
|
||||
|
||||
## 2.18.7
|
||||
|
||||
### :bug: Bugs fixed
|
||||
|
||||
- Fix binfile import accepting library relations outside the package and self-relations [#12895](https://github.com/penpot/penpot/issues/12895) (PR: [#12862](https://github.com/penpot/penpot/pull/12862))
|
||||
|
||||
## 2.18.6
|
||||
|
||||
### :bug: Bugs fixed
|
||||
|
||||
- Fix upload endpoints storing server files when the path points outside the temp dir [#12771](https://github.com/penpot/penpot/issues/12771) (PR: [#12772](https://github.com/penpot/penpot/pull/12772))
|
||||
|
||||
## 2.18.5
|
||||
|
||||
### :bug: Bugs fixed
|
||||
|
||||
- Fix the Admin Console failing on every URL on Docker deployments by reverting the subpath support [#12358](https://github.com/penpot/penpot/pull/12358)
|
||||
|
||||
## 2.18.4
|
||||
|
||||
### :bug: Bugs fixed
|
||||
|
||||
@ -20,7 +20,6 @@ Center](https://help.penpot.app/).
|
||||
- [Title format](#title)
|
||||
- [Description](#description)
|
||||
- [Review process](#review-process)
|
||||
- [Merging](#merging)
|
||||
- [What we won't accept](#what-we-wont-accept)
|
||||
- [Good first issues](#good-first-issues)
|
||||
- [Commit Guidelines](#commit-guidelines)
|
||||
@ -112,8 +111,8 @@ Branch names are not enforced, but we recommend the following:
|
||||
- Otherwise, use a short, descriptive name with words separated by hyphens
|
||||
and no slashes (e.g. `fix-ellipse-icon-typo`, `feat-auto-link-libraries`).
|
||||
|
||||
Since PRs are squash-merged by default, the branch name does not survive
|
||||
into the commit history — what matters is the [PR title](#title).
|
||||
Since PRs are squash-merged, the branch name does not survive into the
|
||||
commit history — what matters is the [PR title](#title).
|
||||
|
||||
### Format
|
||||
|
||||
@ -157,27 +156,9 @@ for more concrete information.
|
||||
- Address review feedback by **pushing new commits** — do not
|
||||
force-push during review, as it breaks comment threads.
|
||||
- PRs require at least **one approval** before merge.
|
||||
- We **squash-merge** by default; see [Merging](#merging) for the reason
|
||||
and for how to ask for a regular merge. The PR title becomes the final
|
||||
- We use **squash-merge** by default. The PR title becomes the final
|
||||
commit message, so follow the [title format](#title) above.
|
||||
|
||||
### Merging
|
||||
|
||||
PRs are **squash-merged** by default. One PR lands as one commit, which keeps
|
||||
the change atomic: the history stays readable, and `git bisect` never stops
|
||||
on an intermediate commit that fails the tests or the linter.
|
||||
|
||||
This is the default, not a hard rule. If you keep your commits atomic and
|
||||
self-contained — each one builds, passes the linter and makes sense on its
|
||||
own — you may ask for a regular merge instead. Put the request in the first
|
||||
line of the PR description, above everything else:
|
||||
|
||||
> **Merge:** do not squash; each commit is self-contained.
|
||||
|
||||
Maintainers honor the request when the commits hold up, and squash-merge
|
||||
otherwise. After a squash merge the PR title becomes the commit subject on
|
||||
the main branch; see [Title format](#title).
|
||||
|
||||
### What we won't accept
|
||||
|
||||
To save time on both sides, please avoid submitting PRs that:
|
||||
@ -244,11 +225,8 @@ Commit messages must follow this format:
|
||||
those prefixes to an existing subject)
|
||||
- **Wrap body lines at 80 characters or fewer** (trailers and URLs excepted)
|
||||
- Separate the subject from the body with a **blank line**
|
||||
- Use `Co-authored-by:` only for a human co-author
|
||||
(`Co-authored-by: Name <email>`). AI providers, agents and bots are not
|
||||
allowed there; for AI participation add an `AI-assisted-by: <model-name>`
|
||||
trailer instead. `./scripts/check-commit` rejects the trailer when its value
|
||||
names one of them.
|
||||
- Never add a `Co-authored-by` trailer — it is forbidden and fails
|
||||
`./scripts/check-commit`
|
||||
|
||||
You can check a commit against these rules with `./scripts/check-commit`.
|
||||
CI runs the same script on every new commit in
|
||||
|
||||
@ -13,6 +13,9 @@
|
||||
<a href="https://tree.taiga.io/project/penpot/" rel="nofollow">
|
||||
<img alt="Managed with Taiga.io" src="https://img.shields.io/badge/managed%20with-TAIGA.io-709f14.svg">
|
||||
</a>
|
||||
<a href="https://gitpod.io/#https://github.com/penpot/penpot" rel="nofollow">
|
||||
<img alt="Gitpod ready-to-code" src="https://img.shields.io/badge/Gitpod-ready--to--code-blue?logo=gitpod">
|
||||
</a>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
|
||||
@ -40,6 +40,7 @@
|
||||
nrepl/nrepl {:mvn/version "1.8.0"}
|
||||
|
||||
org.postgresql/postgresql {:mvn/version "42.7.13"}
|
||||
org.xerial/sqlite-jdbc {:mvn/version "3.53.4.0"}
|
||||
|
||||
com.zaxxer/HikariCP {:mvn/version "7.1.0"}
|
||||
|
||||
@ -53,8 +54,6 @@
|
||||
|
||||
org.jsoup/jsoup {:mvn/version "1.23.2"}
|
||||
|
||||
com.j256.simplemagic/simplemagic {:mvn/version "1.17"}
|
||||
|
||||
at.yawk.lz4/lz4-java
|
||||
{:mvn/version "1.12.0"}
|
||||
|
||||
|
||||
@ -4,7 +4,7 @@
|
||||
"license": "MPL-2.0",
|
||||
"author": "Kaleidos INC Sucursal en España SL",
|
||||
"private": true,
|
||||
"packageManager": "pnpm@12.21.0+sha512.66fd8278ee28935ce951792d34367ff3af5c432a2756445823029533835b8da529bbe0be5130afd56ed0713fb8789e47bad82d93bd5ae2fe54b15562d6276ef0",
|
||||
"packageManager": "pnpm@12.15.0+sha512.b3951d6d6ff4a9f13dc7cf56cd28d2b8eb37dbcc9c68e52e1e1191873ea9f82a017287ab50baadc5ccc494f036c67c0e47eb13608c2fa36f47711b61ea128e61",
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "https://github.com/penpot/penpot"
|
||||
|
||||
131
backend/pnpm-lock.yaml
generated
131
backend/pnpm-lock.yaml
generated
@ -7,162 +7,153 @@ importers:
|
||||
configDependencies: {}
|
||||
packageManagerDependencies:
|
||||
pnpm:
|
||||
specifier: 12.21.0
|
||||
version: 12.21.0
|
||||
specifier: 12.15.0
|
||||
version: 12.15.0
|
||||
|
||||
packages:
|
||||
|
||||
'@pnpm/exe.android-arm64@12.21.0':
|
||||
resolution: {integrity: sha512-iaWidxR5B9NUBpwMznCQu1yUpKB6BjTY5BYX2lORj76VsfINdj8Z8XWSzSNrGI8qlxa28Jzdp5fiaspzOaJZlA==}
|
||||
'@pnpm/exe.android-arm64@12.15.0':
|
||||
resolution: {integrity: sha512-eVCqmscY70LGvgRzu3mi8IBWBFGix+a6mNZoWmXxESsBdeiKEF9xQSuaeel10yn+zI7TXy2jlI63vFLzETk+aQ==}
|
||||
cpu: [arm64]
|
||||
os: [android]
|
||||
|
||||
'@pnpm/exe.android-x64@12.21.0':
|
||||
resolution: {integrity: sha512-35e5v3wlwjIYxDywq+R6/z8CE4gLOEgRxt18yqz1tMRLmx2mnW+CFJI2Q8+srTNjn7gykXUC0x1k195JNikC5A==}
|
||||
'@pnpm/exe.android-x64@12.15.0':
|
||||
resolution: {integrity: sha512-Gx/WbkFcdUFUg8g8AShezcS9EiQxfYvdAm2hfKLO8sfJmJvKOCt5fzlgHWDzmGhoclibgW+tUGZ/KF4F4wOTCA==}
|
||||
cpu: [x64]
|
||||
os: [android]
|
||||
|
||||
'@pnpm/exe.darwin-arm64@12.21.0':
|
||||
resolution: {integrity: sha512-F5OyMX/6zLSowpTZEQRojbCmPS1Pq+fyRaIinslsIe6Glh3qbyQ52u+Fe0eEzSaRuIbWZkXmAAXSWh6Q5Wrlgg==}
|
||||
'@pnpm/exe.darwin-arm64@12.15.0':
|
||||
resolution: {integrity: sha512-N0ZykiDWCvqSBJvt71j7cSoBR993FqvYZFtmpCtqvW3pguiI+NAgraLPA5OTwu5mV+/d8hYpVcPktJA3xFDLtA==}
|
||||
cpu: [arm64]
|
||||
os: [darwin]
|
||||
|
||||
'@pnpm/exe.darwin-x64@12.21.0':
|
||||
resolution: {integrity: sha512-EzpvcsaE6hrrgYeEiAFYd2rSgFybYyxXhkAY7K3y6Hg3loVfcyJ81hHczpFzi8lW5RXtUlFECRPZ8ZAMRFQYYg==}
|
||||
'@pnpm/exe.darwin-x64@12.15.0':
|
||||
resolution: {integrity: sha512-7Ntr2HlBUxzwWOg7NwDxCbRW/3vXuqKO+hcXm+w3SBi9rrHY53li1wLqWVT7Gjxd+lVEHM1QJZo6WJEGFU8C6A==}
|
||||
cpu: [x64]
|
||||
os: [darwin]
|
||||
|
||||
'@pnpm/exe.freebsd-x64@12.21.0':
|
||||
resolution: {integrity: sha512-wNjuEnmdLks4aayPu/QZWQ5tQM9Jn+odUp/vwQM8BXRYoulkTZhtZTHoD60N6mUdLWef1hgUeUh8llNGEUsimw==}
|
||||
'@pnpm/exe.freebsd-x64@12.15.0':
|
||||
resolution: {integrity: sha512-btOiyL7fL7W4uQGH0aqFai2TNC3IoKcZ7gNHBVbEFZt2dWrnrD0SufL3iGs3fLlmb4OtuU35uf4Zpr9V9Jh9mA==}
|
||||
cpu: [x64]
|
||||
os: [freebsd]
|
||||
|
||||
'@pnpm/exe.linux-arm64-musl@12.21.0':
|
||||
resolution: {integrity: sha512-Ceit9zn4g5ZqCJtx1fV8eJE7p8aeXMrGZ+Dc01vRqZLIUNgNeI+Ur0ruyqxtsnJx6rGADyXzohumxPLHjaJ5AQ==}
|
||||
'@pnpm/exe.linux-arm64-musl@12.15.0':
|
||||
resolution: {integrity: sha512-pNC38k+O+90FOjbPNSnw0E7apq1PKTmz3fFe1amn+lys/U5kL++uzir8bRiho8ONcyA3d9EU7wdqcY0DKkj+JA==}
|
||||
cpu: [arm64]
|
||||
os: [linux]
|
||||
libc: [musl]
|
||||
|
||||
'@pnpm/exe.linux-arm64@12.21.0':
|
||||
resolution: {integrity: sha512-4Oj/iF2TDCe2lcH0n4TupPR5sSdrNLclJB6xBElLMPCUOAvBhdf2rRlSaO5M4erJ6g4Fyy7YTXTpcmfFsKyxEA==}
|
||||
'@pnpm/exe.linux-arm64@12.15.0':
|
||||
resolution: {integrity: sha512-qML/8aCXslh8td/w4U+HJNqUcZ2+m6EfTbWScIzZRmyPsRFN+xB4LP1U/hymxc1iSJiKtG56HJx2fvUErWFIGw==}
|
||||
cpu: [arm64]
|
||||
os: [linux]
|
||||
libc: [glibc]
|
||||
|
||||
'@pnpm/exe.linux-ppc64@12.21.0':
|
||||
resolution: {integrity: sha512-s2o7rdSDeUTBFsNqxptpZXlTgT6cojmMVUOvTQvFNSv6Easian02ha4tR32fwwD97Au4YzIIjMCwywCF4D82Sw==}
|
||||
'@pnpm/exe.linux-ppc64@12.15.0':
|
||||
resolution: {integrity: sha512-tjRDUUl7RCJ6T/6eoNPxOqkwU4Mzi5RtTXmSAxujyz37OTwwsMU7Ps087Zk4gHcUvRSeyF40ZFsew3vtE+nmUA==}
|
||||
cpu: [ppc64]
|
||||
os: [linux]
|
||||
libc: [glibc]
|
||||
|
||||
'@pnpm/exe.linux-riscv64@12.21.0':
|
||||
resolution: {integrity: sha512-EYv3GzxeBTITCuM3yGHHbpRFpUvyulkXpQ6LsjKAQNL3iaba1E6h9iDmlouLb3Io4AMMsfdk0/uG6poQ41uewA==}
|
||||
'@pnpm/exe.linux-riscv64@12.15.0':
|
||||
resolution: {integrity: sha512-bF1uBWEfbwaeryz6lIiaJ8AWD082otZfvzGX5l+6LZRpRwUEpJHUsJrORaYP5/cNMv5ZR2p6TvvcJcQzwtDzmw==}
|
||||
cpu: [riscv64]
|
||||
os: [linux]
|
||||
libc: [glibc]
|
||||
|
||||
'@pnpm/exe.linux-s390x@12.21.0':
|
||||
resolution: {integrity: sha512-8VXGNGVNEfs7pL9GTZOzFxjFEnMt4WL3rLOADR5/kPQ7ROTV9GfpVI2DPO/zArXFSkvfc9Zad/WypHLbvzoaTQ==}
|
||||
'@pnpm/exe.linux-s390x@12.15.0':
|
||||
resolution: {integrity: sha512-zKMhIzsexYMToeLmqV5jPsvV/f9vwsJb2kUH2NFYzalbxImeBrlA4kL7GwF8gD4pagV6jYYcJb8RjbZIdjHptg==}
|
||||
cpu: [s390x]
|
||||
os: [linux]
|
||||
libc: [glibc]
|
||||
|
||||
'@pnpm/exe.linux-x64-musl@12.21.0':
|
||||
resolution: {integrity: sha512-MnC5vpIXYubhDZCgCwlfJMg7Cg6HupN2IuouT9gD/tUmOfVdQYqeyYccWbthyo/Jeto8de1oQlxeRN1+1Hl2DQ==}
|
||||
'@pnpm/exe.linux-x64-musl@12.15.0':
|
||||
resolution: {integrity: sha512-A9J0Jg0Rdx+qQU4qXtwyST5tto9IOu1zi4fDk35nbc6DWahggbcKtFvZPKFwL2qgUXGXOX//hXv6z/sp6emO8A==}
|
||||
cpu: [x64]
|
||||
os: [linux]
|
||||
libc: [musl]
|
||||
|
||||
'@pnpm/exe.linux-x64@12.21.0':
|
||||
resolution: {integrity: sha512-8eufCjHXspKbNS/kh8q4Q0gqJoOuvC4Q5pgos6XS8By/nAMtP+jf8F1Cw9er840kfAX+PJ/ts+A9qnLE91rvXQ==}
|
||||
'@pnpm/exe.linux-x64@12.15.0':
|
||||
resolution: {integrity: sha512-yc+QlZR+9+nr29J4fRzhMoapgxTBDndq/+peyoq1XUufnQFnfbXUMFudDIKzHi545linGqmOmXfn0NoufNgcOg==}
|
||||
cpu: [x64]
|
||||
os: [linux]
|
||||
libc: [glibc]
|
||||
|
||||
'@pnpm/exe.netbsd-x64@12.21.0':
|
||||
resolution: {integrity: sha512-eyh1M8hQZCkLWxuva7S/fc1SbFbgqg0/5di0dgWmyiuc2Io3dbWVihDun8X+XM8zsR5uhIDvvZpbS9aZQbCovg==}
|
||||
cpu: [x64]
|
||||
os: [netbsd]
|
||||
|
||||
'@pnpm/exe.win32-arm64@12.21.0':
|
||||
resolution: {integrity: sha512-7xTxxUXo/grRLivSeIl6B21gnsz7ooe5EMPcJmQqwxc8WLJVReN8MzrlHru8bqXfVr4X3+6ZGZscntKenlCBqQ==}
|
||||
'@pnpm/exe.win32-arm64@12.15.0':
|
||||
resolution: {integrity: sha512-fPXZxuY1VclkOga9bc53qNCXcegfeoBR8+Ue+AospDwKzuGGSHll0oDxkLfkmpjyRqpe7bH5DyQPYp/xSMaIVA==}
|
||||
cpu: [arm64]
|
||||
os: [win32]
|
||||
|
||||
'@pnpm/exe.win32-x64@12.21.0':
|
||||
resolution: {integrity: sha512-9TpkJemVApxJ1Z1w0ZFGN+AuSx4nJOqXSJUqMqDvaXTYwGOzvBiiWdFP9GN0ketiWCQS1Q0/pza2E7I3pP0OBQ==}
|
||||
'@pnpm/exe.win32-x64@12.15.0':
|
||||
resolution: {integrity: sha512-nrd+Tp1We1DGj4DGG5Bea9lZNh9y1PjlLBdEwdInS4D0h0UIDtM7JaGGGDJhNzIwcHASyihWI7HLV+dtG+fhYg==}
|
||||
cpu: [x64]
|
||||
os: [win32]
|
||||
|
||||
pnpm@12.21.0:
|
||||
resolution: {integrity: sha512-Zv2CeO4ok1zpUXktNDZ/869cQyonVkRYIwKVM4NbjaUpu+C+UTCv1W7QcT+4eJ5Hutgtk71a4v5UsVVi1idu8A==}
|
||||
pnpm@12.15.0:
|
||||
resolution: {integrity: sha512-s5UdbW/0qfE9x89WzSjSuOs328ycaOUuHhGRhz6p+CoBcoerULqtxczElPA2xnwOR+sTYIwvo29HcRth6hKOYQ==}
|
||||
engines: {node: '>=18'}
|
||||
hasBin: true
|
||||
|
||||
snapshots:
|
||||
|
||||
'@pnpm/exe.android-arm64@12.21.0':
|
||||
'@pnpm/exe.android-arm64@12.15.0':
|
||||
optional: true
|
||||
|
||||
'@pnpm/exe.android-x64@12.21.0':
|
||||
'@pnpm/exe.android-x64@12.15.0':
|
||||
optional: true
|
||||
|
||||
'@pnpm/exe.darwin-arm64@12.21.0':
|
||||
'@pnpm/exe.darwin-arm64@12.15.0':
|
||||
optional: true
|
||||
|
||||
'@pnpm/exe.darwin-x64@12.21.0':
|
||||
'@pnpm/exe.darwin-x64@12.15.0':
|
||||
optional: true
|
||||
|
||||
'@pnpm/exe.freebsd-x64@12.21.0':
|
||||
'@pnpm/exe.freebsd-x64@12.15.0':
|
||||
optional: true
|
||||
|
||||
'@pnpm/exe.linux-arm64-musl@12.21.0':
|
||||
'@pnpm/exe.linux-arm64-musl@12.15.0':
|
||||
optional: true
|
||||
|
||||
'@pnpm/exe.linux-arm64@12.21.0':
|
||||
'@pnpm/exe.linux-arm64@12.15.0':
|
||||
optional: true
|
||||
|
||||
'@pnpm/exe.linux-ppc64@12.21.0':
|
||||
'@pnpm/exe.linux-ppc64@12.15.0':
|
||||
optional: true
|
||||
|
||||
'@pnpm/exe.linux-riscv64@12.21.0':
|
||||
'@pnpm/exe.linux-riscv64@12.15.0':
|
||||
optional: true
|
||||
|
||||
'@pnpm/exe.linux-s390x@12.21.0':
|
||||
'@pnpm/exe.linux-s390x@12.15.0':
|
||||
optional: true
|
||||
|
||||
'@pnpm/exe.linux-x64-musl@12.21.0':
|
||||
'@pnpm/exe.linux-x64-musl@12.15.0':
|
||||
optional: true
|
||||
|
||||
'@pnpm/exe.linux-x64@12.21.0':
|
||||
'@pnpm/exe.linux-x64@12.15.0':
|
||||
optional: true
|
||||
|
||||
'@pnpm/exe.netbsd-x64@12.21.0':
|
||||
'@pnpm/exe.win32-arm64@12.15.0':
|
||||
optional: true
|
||||
|
||||
'@pnpm/exe.win32-arm64@12.21.0':
|
||||
'@pnpm/exe.win32-x64@12.15.0':
|
||||
optional: true
|
||||
|
||||
'@pnpm/exe.win32-x64@12.21.0':
|
||||
optional: true
|
||||
|
||||
pnpm@12.21.0:
|
||||
pnpm@12.15.0:
|
||||
optionalDependencies:
|
||||
'@pnpm/exe.android-arm64': 12.21.0
|
||||
'@pnpm/exe.android-x64': 12.21.0
|
||||
'@pnpm/exe.darwin-arm64': 12.21.0
|
||||
'@pnpm/exe.darwin-x64': 12.21.0
|
||||
'@pnpm/exe.freebsd-x64': 12.21.0
|
||||
'@pnpm/exe.linux-arm64': 12.21.0
|
||||
'@pnpm/exe.linux-arm64-musl': 12.21.0
|
||||
'@pnpm/exe.linux-ppc64': 12.21.0
|
||||
'@pnpm/exe.linux-riscv64': 12.21.0
|
||||
'@pnpm/exe.linux-s390x': 12.21.0
|
||||
'@pnpm/exe.linux-x64': 12.21.0
|
||||
'@pnpm/exe.linux-x64-musl': 12.21.0
|
||||
'@pnpm/exe.netbsd-x64': 12.21.0
|
||||
'@pnpm/exe.win32-arm64': 12.21.0
|
||||
'@pnpm/exe.win32-x64': 12.21.0
|
||||
'@pnpm/exe.android-arm64': 12.15.0
|
||||
'@pnpm/exe.android-x64': 12.15.0
|
||||
'@pnpm/exe.darwin-arm64': 12.15.0
|
||||
'@pnpm/exe.darwin-x64': 12.15.0
|
||||
'@pnpm/exe.freebsd-x64': 12.15.0
|
||||
'@pnpm/exe.linux-arm64': 12.15.0
|
||||
'@pnpm/exe.linux-arm64-musl': 12.15.0
|
||||
'@pnpm/exe.linux-ppc64': 12.15.0
|
||||
'@pnpm/exe.linux-riscv64': 12.15.0
|
||||
'@pnpm/exe.linux-s390x': 12.15.0
|
||||
'@pnpm/exe.linux-x64': 12.15.0
|
||||
'@pnpm/exe.linux-x64-musl': 12.15.0
|
||||
'@pnpm/exe.win32-arm64': 12.15.0
|
||||
'@pnpm/exe.win32-x64': 12.15.0
|
||||
|
||||
---
|
||||
lockfileVersion: '9.0'
|
||||
|
||||
@ -69,11 +69,5 @@
|
||||
:import-binfile/by-profile
|
||||
{:permits 1 :queue 2 :timeout 60000}
|
||||
|
||||
:create-import-binfile-job/global
|
||||
{:permits 4 :queue 10 :timeout 60000}
|
||||
|
||||
:create-import-binfile-job/by-profile
|
||||
{:permits 1 :queue 2 :timeout 60000}
|
||||
|
||||
:profile-plugin-ops/by-profile
|
||||
{:permits 1 :queue 2 :timeout 30000}}
|
||||
|
||||
@ -62,14 +62,13 @@ export PENPOT_FLAGS="\
|
||||
disable-tiered-file-data-storage \
|
||||
enable-file-validation \
|
||||
enable-file-schema-validation \
|
||||
enable-debug-save-faults \
|
||||
enable-redis-cache \
|
||||
enable-link-preview \
|
||||
enable-subscriptions";
|
||||
|
||||
# Uncomment for nexus integration testing
|
||||
# export PENPOT_FLAGS="$PENPOT_FLAGS enable-nexus";
|
||||
# export PENPOT_NEXUS_URI="http://localhost:6070/";
|
||||
# export PENPOT_FLAGS="$PENPOT_FLAGS enable-audit-log-archive";
|
||||
# export PENPOT_AUDIT_LOG_ARCHIVE_URI="http://localhost:6070/api/audit";
|
||||
|
||||
# Default deletion delay for devenv
|
||||
export PENPOT_DELETION_DELAY="24h"
|
||||
@ -93,7 +92,6 @@ export JAVA_OPTS="\
|
||||
-XX:+UseShenandoahGC \
|
||||
-XX:+UseCompactObjectHeaders \
|
||||
-XX:ShenandoahGCMode=generational \
|
||||
-XX:NativeMemoryTracking=summary \
|
||||
-XX:-OmitStackTraceInFastThrow \
|
||||
--sun-misc-unsafe-memory-access=allow \
|
||||
--enable-preview \
|
||||
|
||||
@ -629,7 +629,7 @@
|
||||
(l/warn :hint "received incomplete user info object (please set correct scopes)" :info info)
|
||||
(ex/raise :type :internal
|
||||
:code :incomplete-user-info
|
||||
:hint "incomplete user info"
|
||||
:hint "inconmplete user info"
|
||||
:info info)))
|
||||
|
||||
;; If the provider is OIDC, we can proceed to check
|
||||
@ -690,113 +690,15 @@
|
||||
(assoc :query (u/map->query-string params)))]
|
||||
(redirect-response uri))))
|
||||
|
||||
(defn- public-uri
|
||||
"The configured public URI, parsed, with the app base path normalized with
|
||||
a trailing slash."
|
||||
[]
|
||||
(u/ensure-path-slash (u/uri (cf/get :public-uri))))
|
||||
|
||||
(defn- public-uri-base
|
||||
"The public URI as a string, with the app base path normalized with a
|
||||
trailing slash. Every URL the server builds starts here, so the app base
|
||||
path is never lost."
|
||||
[]
|
||||
(str (public-uri)))
|
||||
|
||||
(defn- redirect-with-organization-sso-error
|
||||
[{:keys [dest-url organization-id organization-name]}]
|
||||
(-> (str (or dest-url (public-uri-base)))
|
||||
(-> (str (or dest-url (cf/get :public-uri)))
|
||||
(u/append-query-param :sso-error true)
|
||||
(u/append-query-param :organization-id organization-id)
|
||||
(cond-> organization-name
|
||||
(u/append-query-param :organization-name organization-name))
|
||||
(redirect-response)))
|
||||
|
||||
(defn- routable-params?
|
||||
"True when `params` are params of a route: at least one, and no key carries
|
||||
a scheme, a path or a fragment. `screen=workspace&team-id=…` and the root
|
||||
params of the application (`template=…`) qualify; `https://evil.example/`
|
||||
does not."
|
||||
[params]
|
||||
(and (seq params)
|
||||
(every? #(nil? (re-find #"[/:?#]" (name %))) (keys params))))
|
||||
|
||||
(defn build-dest-url
|
||||
"Return the absolute URL for `query-string` under the current public URI.
|
||||
|
||||
The query string is the route of the client (`screen=…¶ms`, and the root
|
||||
of the application may carry only its params, like `template=…`): it does not
|
||||
carry the `?` that separates it from the path, which is put back here, so the
|
||||
destination is always built by the server and the client only picks the route.
|
||||
A value that still comes with the separator is trimmed, so a client bundled
|
||||
before that change keeps its destination.
|
||||
|
||||
The params are parsed and rendered back, so a value that is not a route never
|
||||
becomes a destination: it is ignored with a warning and the user lands on the
|
||||
public URI, which is also where a blank one lands."
|
||||
[query-string]
|
||||
(try
|
||||
(let [base (public-uri-base)
|
||||
query (str query-string)
|
||||
route (if (str/starts-with? query "?")
|
||||
(subs query 1)
|
||||
query)
|
||||
params (u/query-string->map route)]
|
||||
(cond
|
||||
(str/blank? query)
|
||||
base
|
||||
|
||||
(not (routable-params? params))
|
||||
(do (l/warn :hint "ignoring a redirect target that is not a route"
|
||||
:query-string (subs route 0 (min 200 (count route))))
|
||||
base)
|
||||
|
||||
:else
|
||||
(str base "?" (u/map->query-string params))))
|
||||
(catch Exception e
|
||||
;; The destination must never break the SSO flow: a query string the
|
||||
;; server cannot parse lands on the public URI instead.
|
||||
(l/warn :hint "could not build the redirect target" :cause e)
|
||||
(public-uri-base))))
|
||||
|
||||
(defn- same-origin?
|
||||
"True when `uri` has the same origin as `base`: scheme, host (case
|
||||
insensitive) and port, with the default port made explicit."
|
||||
[uri base]
|
||||
(let [default-port (fn [scheme] (if (= "https" (str/lower (str scheme))) 443 80))
|
||||
port (fn [u] (or (:port u) (default-port (:scheme u))))]
|
||||
(and (= (str/lower (str (:scheme uri)))
|
||||
(str/lower (str (:scheme base))))
|
||||
(= (str/lower (str (:host uri)))
|
||||
(str/lower (str (:host base))))
|
||||
(= (port uri) (port base)))))
|
||||
|
||||
(defn- under-base-path?
|
||||
"True when the path of `uri` is under the base path of `base`."
|
||||
[uri base]
|
||||
(str/starts-with? (str (:path uri)) (:path base)))
|
||||
|
||||
(defn- trusted-dest-url
|
||||
"Return `dest-url` when it points inside the current public URI, nil
|
||||
otherwise.
|
||||
|
||||
The destination is built by the server (see `build-dest-url`), but state
|
||||
tokens minted before that was the case may carry any URL, so the callback
|
||||
checks the destination before redirecting to it. The check compares the
|
||||
parsed origin, so neither a userinfo prefix
|
||||
(`https://penpot.example.com@evil.example/`) nor a look-alike host
|
||||
(`https://penpot.example.com.evil.example/`) can pass, and it requires
|
||||
the path to be under the base path of the public URI, so the destination
|
||||
stays inside the application."
|
||||
[dest-url]
|
||||
(when (some? dest-url)
|
||||
(let [dest (try (u/uri (str dest-url)) (catch Exception _ nil))
|
||||
base (public-uri)]
|
||||
(when (and (some? dest)
|
||||
(same-origin? dest base)
|
||||
(under-base-path? dest base))
|
||||
(str dest-url)))))
|
||||
|
||||
(defn- redirect-to-register
|
||||
[cfg info provider]
|
||||
(let [info (assoc info
|
||||
@ -1013,18 +915,11 @@
|
||||
|
||||
(defn build-organization-sso-auth-redirect-uri
|
||||
"Build the OIDC authorization redirect URI for an organization SSO config.
|
||||
Raises if the config is incomplete or OIDC discovery fails.
|
||||
|
||||
`query-string` is the route of the client (`screen=…¶ms`, the query
|
||||
string proper, without its separator): the destination is built here, so it
|
||||
can never point outside the public URI.
|
||||
`profile-id` is the profile that started the flow; it travels in the
|
||||
state token so the callback only grants the organization SSO entry to that
|
||||
very session."
|
||||
[cfg sso & {:keys [query-string organization-id provider profile-id]}]
|
||||
Raises if the config is incomplete or OIDC discovery fails."
|
||||
[cfg sso & {:keys [dest-url organization-id provider]}]
|
||||
(let [organization-id (or organization-id (:organization-id sso))
|
||||
issuer (organization-sso-discovery-uri sso)
|
||||
dest-url (build-dest-url query-string)]
|
||||
dest-url (or dest-url (str (cf/get :public-uri)))]
|
||||
(when-not issuer
|
||||
(raise-invalid-sso-config
|
||||
:hint "missing issuer"
|
||||
@ -1034,7 +929,6 @@
|
||||
state-token (tokens/generate cfg {:iss "oidc"
|
||||
:dest-url dest-url
|
||||
:organization-id organization-id
|
||||
:profile-id profile-id
|
||||
:issuer issuer
|
||||
:exp (ct/in-future "4h")})]
|
||||
(build-auth-redirect-uri oidc-provider state-token))
|
||||
@ -1130,39 +1024,23 @@
|
||||
(defn- organization-sso-callback-handler
|
||||
"Handle the organization-SSO branch of the OIDC callback: state carries
|
||||
:dest-url — exchange the authorization code with the OIDC provider to
|
||||
verify authentication actually occurred, then redirect back to dest-url.
|
||||
|
||||
The state also carries the :profile-id that started the flow: only that
|
||||
session is granted the organization SSO entry, so a state prepared by
|
||||
someone else cannot mark the session of the one who completes it."
|
||||
verify authentication actually occurred, then redirect back to dest-url."
|
||||
[cfg request state code]
|
||||
(let [dest-url (trusted-dest-url (:dest-url state))]
|
||||
(let [dest-url (:dest-url state)]
|
||||
(try
|
||||
(let [organization-id (:organization-id state)
|
||||
sso (nitrate/call cfg :get-organization-sso {:organization-id organization-id})
|
||||
provider (prepare-organization-sso-provider cfg sso)
|
||||
_info (get-info cfg provider state code)
|
||||
session (session/get-session request)
|
||||
exp (ct/in-future {:hours 4})
|
||||
;; A state minted before the profile claim existed never
|
||||
;; matches, and neither does the session of another profile.
|
||||
same-profile? (and (some? session)
|
||||
(some? (:profile-id state))
|
||||
(= (:profile-id state) (:profile-id session)))]
|
||||
(if (and (some? session) (not same-profile?))
|
||||
(submit-organization-sso-auth-event
|
||||
cfg request (:profile-id session) organization-id "organization-sso-auth-failed"
|
||||
:failure-reason "session-mismatch")
|
||||
(do
|
||||
(when (and session organization-id)
|
||||
(let [props (-> (or (:props session) {})
|
||||
(update :sso assoc organization-id exp))]
|
||||
(session/update-session (::session/manager cfg) (assoc session :props props))
|
||||
(submit-organization-sso-auth-event
|
||||
cfg request (:profile-id session) organization-id "organization-sso-token-issued")))
|
||||
(submit-organization-sso-auth-event
|
||||
cfg request (:profile-id session) organization-id "organization-sso-auth-succeeded")))
|
||||
(redirect-response (or dest-url (public-uri-base))))
|
||||
exp (ct/in-future {:hours 4})]
|
||||
(when (and session organization-id)
|
||||
(let [props (-> (or (:props session) {})
|
||||
(update :sso assoc organization-id exp))]
|
||||
(session/update-session (::session/manager cfg) (assoc session :props props))))
|
||||
(submit-organization-sso-auth-event
|
||||
cfg request (:profile-id session) organization-id "organization-sso-auth-succeeded")
|
||||
(redirect-response dest-url))
|
||||
(catch Throwable cause
|
||||
(let [{:keys [code]} (ex-data cause)]
|
||||
(binding [l/*context* (errors/request->context request)]
|
||||
@ -1180,36 +1058,10 @@
|
||||
(let [organization-id (:organization-id state)
|
||||
organization-name (:name (nitrate/call cfg :get-organization-summary {:organization-id organization-id}))]
|
||||
(redirect-with-organization-sso-error
|
||||
{:dest-url (or dest-url (public-uri-base))
|
||||
{:dest-url dest-url
|
||||
:organization-id organization-id
|
||||
:organization-name organization-name}))))))
|
||||
|
||||
(def oidc-provider-types
|
||||
"Backends authenticating through an OIDC-compatible provider.
|
||||
Only these backends take the SSO registration gate."
|
||||
#{"oidc" "google" "github" "gitlab"})
|
||||
|
||||
(defn sso-registration-allowed?
|
||||
"SSO registration is allowed when public registration or explicit
|
||||
OIDC registration is enabled. Shared by the callback gate and the
|
||||
`register-profile` token gate — keep one definition."
|
||||
[]
|
||||
(or (contains? cf/flags :registration)
|
||||
(contains? cf/flags :oidc-registration)))
|
||||
|
||||
(defn- valid-invitation-for?
|
||||
"Returns true when `invitation-token` is a verifiable
|
||||
`:team-invitation` JWT issued for `email`. Never throws: a garbage
|
||||
token means there is no valid invitation."
|
||||
[cfg invitation-token email]
|
||||
(try
|
||||
(boolean
|
||||
(when (and (some? invitation-token) (some? email))
|
||||
(let [invitation (tokens/verify cfg {:token invitation-token :iss :team-invitation})]
|
||||
(= (profile/clean-email (:member-email invitation))
|
||||
(profile/clean-email email)))))
|
||||
(catch Throwable _ false)))
|
||||
|
||||
(defn- callback-handler
|
||||
[cfg {:keys [params] :as request}]
|
||||
(if-let [error (get params :error)]
|
||||
@ -1242,8 +1094,8 @@
|
||||
(redirect-with-error "email-domain-not-allowed")
|
||||
|
||||
:else
|
||||
(if (or (sso-registration-allowed?)
|
||||
(valid-invitation-for? cfg (:invitation-token info) (:email info)))
|
||||
(if (or (contains? cf/flags :registration)
|
||||
(contains? cf/flags :oidc-registration))
|
||||
(redirect-to-register cfg info provider)
|
||||
(redirect-with-error "registration-disabled")))
|
||||
|
||||
@ -1261,15 +1113,11 @@
|
||||
|
||||
:else
|
||||
(let [sxf (session/create-fn cfg profile info)
|
||||
;; The session token is always an :auth one; the
|
||||
;; invitation token, when present, travels inside its
|
||||
;; claims so the frontend goes through the regular
|
||||
;; token verification path and then `logged-in`.
|
||||
token (tokens/generate cfg
|
||||
{:iss :auth
|
||||
:exp (ct/in-future "15m")
|
||||
:profile-id (:id profile)
|
||||
:invitation-token (:invitation-token info)})
|
||||
token (or (:invitation-token info)
|
||||
(tokens/generate cfg
|
||||
{:iss :auth
|
||||
:exp (ct/in-future "15m")
|
||||
:profile-id (:id profile)}))
|
||||
|
||||
;; If proceed, update profile on the database
|
||||
profile (update-profile-with-info cfg profile info)
|
||||
|
||||
@ -9,7 +9,7 @@
|
||||
for recently imported shapes."
|
||||
(:require
|
||||
[app.common.data :as d]
|
||||
[app.common.geom.matrix :as gmt]
|
||||
[app.common.svg :as csvg]
|
||||
[app.common.types.shape :as cts]
|
||||
[app.common.uuid :as uuid]))
|
||||
|
||||
@ -106,11 +106,14 @@
|
||||
:reverse-column :column-reverse
|
||||
dir))))
|
||||
|
||||
(defn- fix-svg-transform
|
||||
"The json reader decodes `:svg-transform` as a plain map; the svg defs
|
||||
renderer multiplies it as a matrix, so turn it back into one."
|
||||
(defn- fix-svg-attrs
|
||||
"The json reader of the binfile rewrites every key of every entry to
|
||||
kebab-case, but `:svg-attrs` keys are react prop names and are stored
|
||||
in camelCase. `attrs->props` is the transform the svg import path
|
||||
already applies to them, so running it again restores the names; it
|
||||
is idempotent, so shapes that come in correct are left untouched."
|
||||
[shape]
|
||||
(d/update-when shape :svg-transform gmt/coerce))
|
||||
(d/update-when shape :svg-attrs csvg/attrs->props))
|
||||
|
||||
(defn clean-shape-post-decode
|
||||
"A shape procesor that expected to be executed after schema decoding
|
||||
@ -120,7 +123,7 @@
|
||||
(fix-shape-shadow-color)
|
||||
(fix-root-shape)
|
||||
(fix-legacy-flex-dir)
|
||||
(fix-svg-transform)))
|
||||
(fix-svg-attrs)))
|
||||
|
||||
(defn- fix-container
|
||||
[container]
|
||||
|
||||
@ -24,6 +24,7 @@
|
||||
[app.common.weak :as weak]
|
||||
[app.config :as cf]
|
||||
[app.db :as db]
|
||||
[app.db.sql :as sql]
|
||||
[app.features.fdata :as fdata]
|
||||
[app.features.file-migrations :as fmigr]
|
||||
[app.loggers.audit :as-alias audit]
|
||||
@ -42,6 +43,7 @@
|
||||
|
||||
(def ^:dynamic *state* nil)
|
||||
(def ^:dynamic *options* nil)
|
||||
(def ^:dynamic *reference-file* nil)
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; DEFAULTS
|
||||
@ -72,11 +74,11 @@
|
||||
;; Maximum total decompressed size allowed for all JSON/text zip entries
|
||||
;; combined within a single import job: 500MiB. Bounds the case where many
|
||||
;; entries, each individually under default-max-text-entry-size, still sum
|
||||
;; to an unreasonable total. Set this high because legitimate giant
|
||||
;; exports exist: hundreds of thousands of KB-sized entries (pages,
|
||||
;; shapes) that never trip the per-entry cap but sum past 400MiB. The
|
||||
;; per-entry cap still bounds single-entry DEFLATE amplification, so this
|
||||
;; only moves the cumulative ceiling.
|
||||
;; to an unreasonable total. Set well above the 200MiB it replaces because
|
||||
;; legitimate giant exports exist: hundreds of thousands of KB-sized
|
||||
;; entries (pages, shapes) that never trip the per-entry cap but sum past
|
||||
;; 400MiB. The per-entry cap still bounds single-entry DEFLATE
|
||||
;; amplification, so this only moves the cumulative ceiling.
|
||||
(def ^:const default-max-text-total-size
|
||||
(* 1024 1024 500))
|
||||
|
||||
@ -128,12 +130,18 @@
|
||||
(defn initial-state
|
||||
[]
|
||||
{:storage-objects #{}
|
||||
:files #{}})
|
||||
:files #{}
|
||||
:teams #{}
|
||||
:projects #{}})
|
||||
|
||||
(defn collect-storage-objects
|
||||
[state items]
|
||||
(update state :storage-objects into xf-map-media-id items))
|
||||
|
||||
(defn collect-summary
|
||||
[state key items]
|
||||
(update state key into xf-map-media-id items))
|
||||
|
||||
(defn lookup-index
|
||||
[id]
|
||||
(when id
|
||||
@ -151,7 +159,7 @@
|
||||
[index obj & attrs]
|
||||
(reduce (fn [index attr-fn]
|
||||
(let [old-id (attr-fn obj)
|
||||
new-id (uuid/next)]
|
||||
new-id (if (::overwrite *options*) old-id (uuid/next))]
|
||||
(assoc index old-id new-id)))
|
||||
index
|
||||
attrs))
|
||||
@ -201,7 +209,6 @@
|
||||
p.team_id,
|
||||
coalesce(fd.backend, 'legacy-db') AS backend,
|
||||
fd.metadata AS metadata,
|
||||
fd.storage_object_id AS storage_object_id,
|
||||
fd.data AS data
|
||||
FROM file AS f
|
||||
LEFT JOIN file_data AS fd ON (fd.file_id = f.id AND fd.id = f.id)
|
||||
@ -211,230 +218,139 @@
|
||||
(str sql:files-with-data " WHERE f.id = ?"))
|
||||
|
||||
(def sql:get-file-without-data
|
||||
"SELECT f.id,
|
||||
f.project_id,
|
||||
f.created_at,
|
||||
f.modified_at,
|
||||
f.deleted_at,
|
||||
f.name,
|
||||
f.is_shared,
|
||||
f.has_media_trimmed,
|
||||
f.revn,
|
||||
f.ignore_sync_until,
|
||||
f.comment_thread_seqn,
|
||||
f.features,
|
||||
f.version,
|
||||
f.vern,
|
||||
p.team_id
|
||||
FROM file AS f
|
||||
INNER JOIN project AS p ON (p.id = f.project_id)
|
||||
WHERE f.id = ?")
|
||||
(str "WITH files AS (" sql:files-with-data ")"
|
||||
"SELECT f.id,
|
||||
f.project_id,
|
||||
f.created_at,
|
||||
f.modified_at,
|
||||
f.deleted_at,
|
||||
f.name,
|
||||
f.is_shared,
|
||||
f.has_media_trimmed,
|
||||
f.revn,
|
||||
f.ignore_sync_until,
|
||||
f.comment_thread_seqn,
|
||||
f.features,
|
||||
f.version,
|
||||
f.vern,
|
||||
f.team_id
|
||||
FROM files AS f
|
||||
WHERE f.id = ?"))
|
||||
|
||||
(defn- migrate-file
|
||||
"Apply the pending migrations to the file.
|
||||
[{:keys [::db/conn] :as cfg} {:keys [read-only?]} {:keys [id] :as file}]
|
||||
(binding [pmap/*load-fn* (partial fdata/load-pointer cfg id)
|
||||
pmap/*tracked* (pmap/create-tracked)]
|
||||
(let [libs (delay (get-resolved-file-libraries cfg file))
|
||||
;; For avoid unnecesary overhead of creating multiple
|
||||
;; pointers and handly internally with objects map in their
|
||||
;; worst case (when probably all shapes and all pointers
|
||||
;; will be readed in any case), we just realize/resolve them
|
||||
;; before applying the migration to the file.
|
||||
file (-> (fdata/realize cfg file)
|
||||
(fmg/migrate-file libs))]
|
||||
|
||||
With `persist-after-migration?` the migrated map is also persisted, which is
|
||||
only safe from a map read under the `file` row lock (see `get-file*`). The
|
||||
returned map is then the reattached one: decoded data where the pages the
|
||||
migration did not change keep their pointers and their fragment ids, plus
|
||||
the `:storage-object-id` of the row the write left, so a caller that
|
||||
persists again touches the current object. Without `persist-after-migration?`
|
||||
the realized map is returned, which is what a read-only context can do.
|
||||
`get-file*` realizes the result when the caller asks for it."
|
||||
[cfg {:keys [id] :as file} {:keys [persist-after-migration?]}]
|
||||
(fdata/with-tracked-pointers cfg id
|
||||
(let [libs (delay (get-resolved-file-libraries cfg file))
|
||||
;; Capture the pointer layout before realizing: the migration
|
||||
;; runs on plain data, and unchanged positions must keep their
|
||||
;; original fragment ids instead of being re-wrapped with new
|
||||
;; ones on persist.
|
||||
original (fdata/original-pointers (:data file))
|
||||
objects-map-before? (contains? (:features file) "fdata/objects-map")
|
||||
realized (-> (fdata/realize cfg file)
|
||||
(fmg/migrate-file libs))
|
||||
objects-map-added? (and (not objects-map-before?)
|
||||
(contains? (:features realized) "fdata/objects-map"))]
|
||||
|
||||
(if persist-after-migration?
|
||||
(let [file (if objects-map-added?
|
||||
realized
|
||||
(fdata/reattach-unchanged original realized))
|
||||
written (update-file! cfg file)]
|
||||
;; When the file is migrated, we break the rule of not
|
||||
;; performing mutations on get operations and update the file
|
||||
;; with all migrations applied. The reattached map is also what
|
||||
;; the read returns: the caller realizes it only when it asks.
|
||||
(assoc (fmigr/resolve-applied-migrations cfg file)
|
||||
:storage-object-id (:storage-object-id written)))
|
||||
realized))))
|
||||
|
||||
(defn- fetch-row
|
||||
"Read the file row. With `load-data?` the file data columns come along, but
|
||||
they are not resolved nor decoded (see `attach-file-data`). With
|
||||
`load-migrations?` the applied migrations are resolved, which is what
|
||||
`need-migration?` expects. A skip-locked read answers nil instead of raising
|
||||
when the row is gone."
|
||||
[{:keys [::db/conn] :as cfg} id
|
||||
{:keys [include-deleted?
|
||||
skip-locked?
|
||||
lock-for-update?
|
||||
lock-for-share?
|
||||
throw-if-not-exists?
|
||||
load-migrations?
|
||||
load-data?]}]
|
||||
|
||||
(let [sql (if load-data?
|
||||
sql:get-file
|
||||
sql:get-file-without-data)
|
||||
sql (cond
|
||||
lock-for-update?
|
||||
(str sql " FOR UPDATE of f")
|
||||
|
||||
lock-for-share?
|
||||
(str sql " FOR SHARE of f")
|
||||
|
||||
:else
|
||||
sql)
|
||||
|
||||
sql (if (and skip-locked? (or lock-for-share? lock-for-update?))
|
||||
(str sql " SKIP LOCKED")
|
||||
sql)
|
||||
|
||||
file (db/get-with-sql conn [sql id] {::db/throw-if-not-exists false
|
||||
::db/remove-deleted (not include-deleted?)})
|
||||
file (some-> file
|
||||
(d/update-when :features db/decode-pgarray #{})
|
||||
(d/update-when :metadata fdata/decode-metadata))
|
||||
file (if (and file load-migrations?)
|
||||
(fmigr/resolve-applied-migrations cfg file)
|
||||
file)]
|
||||
|
||||
(or file
|
||||
(when-not (or skip-locked? (not throw-if-not-exists?))
|
||||
(ex/raise :type :not-found
|
||||
:code :object-not-found
|
||||
:hint "database object not found"
|
||||
:table :file
|
||||
:file-id id)))))
|
||||
|
||||
|
||||
(defn- attach-file-data
|
||||
"Resolve the data of an already read row and decode it when `decode?`. It
|
||||
completes the map it receives instead of reading the row again. It does not
|
||||
realize: `get-file*` owns that step, once per path."
|
||||
[cfg file {:keys [decode?]}]
|
||||
(let [file (fdata/resolve-file-data cfg file)]
|
||||
(if decode?
|
||||
(fdata/decode-file-data cfg file)
|
||||
file)))
|
||||
|
||||
(defn- fetch-file
|
||||
"Read the row with its data resolved and decoded, under the requested lock."
|
||||
[cfg id options]
|
||||
(when-let [file (fetch-row cfg id options)]
|
||||
(attach-file-data cfg file options)))
|
||||
|
||||
(def ^:private
|
||||
default-get-file-options
|
||||
{:load-migrations? true
|
||||
:throw-if-not-exists? true
|
||||
:persist-after-migration? true
|
||||
:load-data? true
|
||||
:migrate? true
|
||||
:decode? true})
|
||||
(if (or read-only? (db/read-only? conn))
|
||||
file
|
||||
(do ;; When file is migrated, we break the rule of no
|
||||
;; perform mutations on get operations and update the
|
||||
;; file with all migrations applied
|
||||
(update-file! cfg file)
|
||||
(fmigr/resolve-applied-migrations cfg file))))))
|
||||
|
||||
(defn- get-file*
|
||||
[{:keys [::db/conn] :as cfg} id options]
|
||||
[{:keys [::db/conn] :as cfg} id
|
||||
{:keys [migrate?
|
||||
realize?
|
||||
decode?
|
||||
skip-locked?
|
||||
include-deleted?
|
||||
load-data?
|
||||
throw-if-not-exists?
|
||||
lock-for-update?
|
||||
lock-for-share?]
|
||||
:or {lock-for-update? false
|
||||
lock-for-share? false
|
||||
load-data? true
|
||||
migrate? true
|
||||
decode? true
|
||||
include-deleted? false
|
||||
throw-if-not-exists? true
|
||||
realize? false}
|
||||
:as options}]
|
||||
|
||||
(assert (db/connection? conn) "expected cfg with valid connection")
|
||||
(when (and (not load-data?)
|
||||
(or lock-for-share? lock-for-share? skip-locked?))
|
||||
(throw (IllegalArgumentException. "locking is incompatible when `load-data?` is false")))
|
||||
|
||||
(let [{:keys [migrate?
|
||||
realize?
|
||||
decode?
|
||||
load-data?
|
||||
lock-for-share?
|
||||
read-only?]
|
||||
:as options}
|
||||
(merge default-get-file-options options)
|
||||
(let [sql
|
||||
(if load-data?
|
||||
sql:get-file
|
||||
sql:get-file-without-data)
|
||||
|
||||
read-only?
|
||||
(or read-only? (db/read-only? conn))]
|
||||
sql
|
||||
(cond
|
||||
lock-for-update?
|
||||
(str sql " FOR UPDATE of f")
|
||||
|
||||
(when (and realize? (or (not load-data?) (not decode?)))
|
||||
(throw (IllegalArgumentException. "invalid options, you only can realize when loading data and decoding")))
|
||||
lock-for-share?
|
||||
(str sql " FOR SHARE of f")
|
||||
|
||||
(when (and lock-for-share? load-data?)
|
||||
(throw (IllegalArgumentException. "invalid options, `:lock-for-share?` is only allowed on row-only reads")))
|
||||
:else
|
||||
sql)
|
||||
|
||||
(cond
|
||||
;; The caller only wants the row: that read is the answer.
|
||||
(not load-data?)
|
||||
(fetch-row cfg id options)
|
||||
sql
|
||||
(if skip-locked?
|
||||
(str sql " SKIP LOCKED")
|
||||
sql)
|
||||
|
||||
(and migrate? decode? (not read-only?))
|
||||
(when-let [file (fetch-row cfg id options)]
|
||||
(if (fmg/need-migration? file)
|
||||
(when-let [file (fetch-file cfg id (assoc options :lock-for-update? true))]
|
||||
(cond-> file
|
||||
(fmg/need-migration? file)
|
||||
(as-> $ (migrate-file cfg $ options))
|
||||
file
|
||||
(db/get-with-sql conn [sql id]
|
||||
{::db/throw-if-not-exists false
|
||||
::db/remove-deleted (not include-deleted?)})
|
||||
|
||||
realize?
|
||||
(->> (fdata/realize cfg))))
|
||||
file
|
||||
(-> file
|
||||
(d/update-when :features db/decode-pgarray #{})
|
||||
(d/update-when :metadata fdata/decode-metadata))]
|
||||
|
||||
(let [file (attach-file-data cfg file options)]
|
||||
(cond-> file
|
||||
realize?
|
||||
(->> (fdata/realize cfg))))))
|
||||
(if file
|
||||
(if load-data?
|
||||
(let [file
|
||||
(->> file
|
||||
(fmigr/resolve-applied-migrations cfg)
|
||||
(fdata/resolve-file-data cfg))
|
||||
|
||||
:else
|
||||
(when-let [file (fetch-file cfg id options)]
|
||||
(cond-> file
|
||||
(and migrate? decode? (fmg/need-migration? file))
|
||||
(as-> $ (migrate-file cfg $ (assoc options :persist-after-migration? false)))
|
||||
will-migrate?
|
||||
(and migrate? (fmg/need-migration? file))]
|
||||
|
||||
realize?
|
||||
(->> (fdata/realize cfg)))))))
|
||||
(if decode?
|
||||
(cond->> (fdata/decode-file-data cfg file)
|
||||
(and realize? (not will-migrate?))
|
||||
(fdata/realize cfg)
|
||||
|
||||
will-migrate?
|
||||
(migrate-file cfg options))
|
||||
|
||||
file))
|
||||
file)
|
||||
|
||||
(when-not (or skip-locked? (not throw-if-not-exists?))
|
||||
(ex/raise :type :not-found
|
||||
:code :object-not-found
|
||||
:hint "database object not found"
|
||||
:table :file
|
||||
:file-id id)))))
|
||||
|
||||
(defn get-file
|
||||
"Get file, resolve all features and apply migrations.
|
||||
|
||||
Runs inside a transaction: it joins the caller's when the cfg already
|
||||
carries a connection and opens one otherwise, so the migration write
|
||||
commits with the read. A read-only read (`:read-only?`) takes no lock and
|
||||
writes nothing, so it runs on a plain connection instead of opening a
|
||||
transaction of its own; with a cfg that already carries a connection it
|
||||
still joins the caller's. Accepts a cfg map only: a bare pool or a
|
||||
connection raises `IllegalArgumentException`.
|
||||
|
||||
Useful when you have plan to apply massive or not surgical
|
||||
operations on file, because it removes the overhead of lazy fetching
|
||||
and decoding.
|
||||
|
||||
Options:
|
||||
- `:realize?` returns plain data instead of fragment pointers; it
|
||||
requires `:load-data?` and `:decode?`.
|
||||
- `:decode?` decodes the stored blob (default true).
|
||||
- `:load-data?` reads the data columns (default true). A row-only read
|
||||
(`false`) is the only mode that accepts `:lock-for-share?`.
|
||||
- `:migrate?` applies the pending migrations on read (default true).
|
||||
- `:load-migrations?` resolves the applied migrations on a row read
|
||||
(default true).
|
||||
- `:read-only?` applies the pending migrations in memory only and skips
|
||||
the transaction. It does not take a lock.
|
||||
- `:lock-for-update?`, `:skip-locked?`, `:include-deleted?` and
|
||||
`:throw-if-not-exists?` are the usual read knobs. A lock needs a
|
||||
read-write transaction from the caller: PostgreSQL rejects row locks in
|
||||
a read-only transaction.
|
||||
- `:persist-after-migration?` is internal: `get-file*` turns it off
|
||||
for the in-memory migration of a read-only context. Do not pass it."
|
||||
and decoding."
|
||||
[cfg file-id & {:as opts}]
|
||||
(when-not (map? cfg)
|
||||
(throw (IllegalArgumentException.
|
||||
"expected cfg map, not a pool or a connection")))
|
||||
(if (:read-only? opts)
|
||||
(db/run! cfg get-file* file-id opts)
|
||||
(db/tx-run! cfg get-file* file-id opts)))
|
||||
(db/run! cfg get-file* file-id opts))
|
||||
|
||||
(defn clean-file-features
|
||||
[file]
|
||||
@ -462,27 +378,19 @@
|
||||
inner join project as p on (p.id = f.project_id)
|
||||
where fpr.file_id = ?
|
||||
and fpr.profile_id = ?
|
||||
and f.deleted_at is null
|
||||
and p.deleted_at is null
|
||||
and exists (select 1
|
||||
from team_profile_rel as tpr
|
||||
inner join team as t on (t.id = tpr.team_id)
|
||||
where tpr.team_id = p.team_id
|
||||
and tpr.profile_id = fpr.profile_id
|
||||
and t.deleted_at is null)
|
||||
and tpr.profile_id = fpr.profile_id)
|
||||
union all
|
||||
select tpr.is_owner,
|
||||
tpr.is_admin,
|
||||
tpr.can_edit
|
||||
from team_profile_rel as tpr
|
||||
inner join team as t on (t.id = tpr.team_id)
|
||||
inner join project as p on (p.team_id = tpr.team_id)
|
||||
inner join file as f on (p.id = f.project_id)
|
||||
where f.id = ?
|
||||
and tpr.profile_id = ?
|
||||
and t.deleted_at is null
|
||||
and p.deleted_at is null
|
||||
and f.deleted_at is null
|
||||
union all
|
||||
select ppr.is_owner,
|
||||
ppr.is_admin,
|
||||
@ -492,14 +400,10 @@
|
||||
inner join project as p on (p.id = ppr.project_id)
|
||||
where f.id = ?
|
||||
and ppr.profile_id = ?
|
||||
and f.deleted_at is null
|
||||
and p.deleted_at is null
|
||||
and exists (select 1
|
||||
from team_profile_rel as tpr
|
||||
inner join team as t on (t.id = tpr.team_id)
|
||||
where tpr.team_id = p.team_id
|
||||
and tpr.profile_id = ppr.profile_id
|
||||
and t.deleted_at is null)")
|
||||
and tpr.profile_id = ppr.profile_id)")
|
||||
|
||||
(defn- get-file-permissions*
|
||||
[conn profile-id file-id]
|
||||
@ -547,31 +451,47 @@
|
||||
[cfg project-id]
|
||||
(db/get cfg :project {:id project-id}))
|
||||
|
||||
(def ^:private sql:get-files-rels
|
||||
"SELECT flr.*,
|
||||
fls.synced_at
|
||||
FROM file_library_rel AS flr
|
||||
JOIN file AS l
|
||||
ON flr.library_file_id = l.id
|
||||
LEFT JOIN file_library_sync AS fls
|
||||
ON fls.file_id = flr.file_id
|
||||
AND fls.library_file_id = flr.library_file_id
|
||||
WHERE flr.file_id = ANY(?)
|
||||
AND l.deleted_at IS NULL
|
||||
ORDER BY flr.file_id, flr.library_file_id;")
|
||||
(def ^:private sql:get-teams
|
||||
"SELECT t.* FROM team WHERE id = ANY(?)")
|
||||
|
||||
(defn get-teams
|
||||
[cfg ids]
|
||||
(let [conn (db/get-connection cfg)
|
||||
ids (db/create-array conn "uuid" ids)]
|
||||
(->> (db/exec! conn [sql:get-teams ids])
|
||||
(map decode-row-features))))
|
||||
|
||||
(defn get-team
|
||||
[cfg team-id]
|
||||
(-> (db/get cfg :team {:id team-id})
|
||||
(decode-row-features)))
|
||||
|
||||
(defn get-fonts
|
||||
[cfg team-id]
|
||||
(db/query cfg :team-font-variant
|
||||
{:team-id team-id
|
||||
:deleted-at nil}))
|
||||
|
||||
(defn get-files-rels
|
||||
"Given a set of file-id's, return all matching relations with the libraries.
|
||||
Rows come back in stable `(file-id, library-file-id)` order so export
|
||||
and import resolve cycles deterministically."
|
||||
"Given a set of file-id's, return all matching relations with the libraries"
|
||||
[cfg ids]
|
||||
|
||||
(assert (set? ids) "expected a set of uuids")
|
||||
(assert (every? uuid? ids) "expected a set of uuids")
|
||||
|
||||
(db/run! cfg (fn [{:keys [::db/conn]}]
|
||||
(let [ids (db/create-array conn "uuid" ids)]
|
||||
(db/exec! conn [sql:get-files-rels ids])))))
|
||||
(let [ids (db/create-array conn "uuid" ids)
|
||||
sql (str "SELECT flr.*,"
|
||||
" fls.synced_at"
|
||||
" FROM file_library_rel AS flr"
|
||||
" JOIN file AS l"
|
||||
" ON flr.library_file_id = l.id"
|
||||
" LEFT JOIN file_library_sync AS fls"
|
||||
" ON fls.file_id = flr.file_id"
|
||||
" AND fls.library_file_id = flr.library_file_id"
|
||||
" WHERE flr.file_id = ANY(?)"
|
||||
" AND l.deleted_at IS NULL;")]
|
||||
(db/exec! conn [sql ids])))))
|
||||
|
||||
(def ^:private sql:upsert-file-library-sync
|
||||
"INSERT INTO file_library_sync (file_id, library_file_id, synced_at)
|
||||
@ -606,20 +526,6 @@
|
||||
(->> (db/exec! conn [sql:get-libraries ids'])
|
||||
(into #{} xf-map-id))))))
|
||||
|
||||
(defn library-link-status
|
||||
"Check whether linking `file-id` to `library-id` is safe. Returns
|
||||
`:ok` when the link can be created, `:self-link` when both ids are
|
||||
the same, and `:circular` when the link would close a dependency
|
||||
cycle (i.e. `file-id` is already among the transitive libraries of
|
||||
`library-id`). Shared by the UI command (`link-file-to-library`,
|
||||
which raises) and the v3 importer (`import-file-relations`, which
|
||||
skips and logs), so both stay consistent."
|
||||
[cfg file-id library-id]
|
||||
(cond
|
||||
(= file-id library-id) :self-link
|
||||
(contains? (get-libraries cfg [library-id]) file-id) :circular
|
||||
:else :ok))
|
||||
|
||||
(defn get-file-object-thumbnails
|
||||
"Return all file object thumbnails for a given file."
|
||||
[cfg file-id]
|
||||
@ -628,6 +534,15 @@
|
||||
:deleted-at nil})
|
||||
(not-empty)))
|
||||
|
||||
(defn get-file-thumbnail
|
||||
"Return the thumbnail for the specified file-id"
|
||||
[cfg {:keys [id revn]}]
|
||||
(db/get* cfg :file-thumbnail
|
||||
{:file-id id
|
||||
:revn revn
|
||||
:data nil}
|
||||
{::sql/columns [:media-id :file-id :revn]}))
|
||||
|
||||
(def ^:private sql:get-missing-media-references
|
||||
"SELECT fmo.*
|
||||
FROM file_media_object AS fmo
|
||||
@ -786,12 +701,7 @@
|
||||
(let [library-ids (get-libraries cfg [file-id])
|
||||
data (reduce (fn [data library-id]
|
||||
(if-let [library (get-file cfg library-id :include-deleted? true)]
|
||||
;; The library is read without `:realize?`, so its
|
||||
;; `:components` may be a pointer map. Bind the
|
||||
;; library's own loader: the absorb walks the
|
||||
;; components and would raise without one.
|
||||
(cond-> (fdata/with-pointers cfg library-id
|
||||
(ctf/absorb-assets data (:data library)))
|
||||
(cond-> (ctf/absorb-assets data (:data library))
|
||||
(= library-id (cfo/get-tokens-source data))
|
||||
(cfo/absorb-tokens-lib (:data library)))
|
||||
data))
|
||||
@ -874,33 +784,18 @@
|
||||
(dissoc :metadata)
|
||||
(dissoc :migrations)))
|
||||
|
||||
(defn file->response
|
||||
"Strip the internal keys of a file map before it leaves the backend in a
|
||||
response. The file data storage pointer is bookkeeping for the write path:
|
||||
a `file-data` object is authentication-only, so a leaked id would let any
|
||||
logged-in profile fetch the blob from `/assets/by-id`. The metadata key of
|
||||
the old pointer is stripped too: a row the migration does not backfill keeps
|
||||
it, and the metadata schema is open, so it would travel in the response."
|
||||
[file]
|
||||
(-> file
|
||||
(dissoc :storage-object-id)
|
||||
(d/update-when :metadata #(dissoc % :storage-ref-id))))
|
||||
|
||||
(defn- file->file-data-params
|
||||
[{:keys [id] :as file}]
|
||||
[{:keys [id] :as file} & {:as opts}]
|
||||
(let [created-at (or (:created-at file) (ct/now))
|
||||
modified-at (or (:modified-at file) created-at)]
|
||||
(-> (d/without-nils
|
||||
{:id id
|
||||
:type "main"
|
||||
:file-id id
|
||||
:data (:data file)
|
||||
:metadata (:metadata file)
|
||||
:created-at created-at
|
||||
:modified-at modified-at})
|
||||
;; The storage object the persisted row pointed at (nil for a new or
|
||||
;; imported file). `upsert!` touches it before replacing the row.
|
||||
(assoc :storage-object-id (:storage-object-id file)))))
|
||||
(d/without-nils
|
||||
{:id id
|
||||
:type "main"
|
||||
:file-id id
|
||||
:data (:data file)
|
||||
:metadata (:metadata file)
|
||||
:created-at created-at
|
||||
:modified-at modified-at})))
|
||||
|
||||
(defn insert-file!
|
||||
"Insert a new file into the database table. Expectes a not-encoded file.
|
||||
@ -924,63 +819,34 @@
|
||||
:cause cause)
|
||||
(throw cause))))
|
||||
|
||||
;; An INSERT never replaces a persisted pointer: `duplicate-file` hands
|
||||
;; the source file map here, and touching the source object would be a
|
||||
;; write on a live object.
|
||||
(->> (file->file-data-params (dissoc file :storage-object-id))
|
||||
(->> (file->file-data-params file)
|
||||
(fdata/upsert! cfg))
|
||||
|
||||
nil))
|
||||
|
||||
(defn update-file!
|
||||
"Update an existing file on the database. Expects a not-encoded file map,
|
||||
as returned by `get-file`, and optionally the fields to change as keyword
|
||||
args; they are merged into the file map before persisting.
|
||||
"Update an existing file on the database. Expects not encoded file."
|
||||
[{:keys [::db/conn] :as cfg} {:keys [id] :as file} & {:as opts}]
|
||||
|
||||
The storage object the persisted row pointed at travels in the map
|
||||
(`:storage-object-id`), and `upsert!` touches it before replacing the row.
|
||||
That is why the caller must pass the map it read instead of building a
|
||||
partial one: nothing is queried here. The params only change mutable fields
|
||||
of the file: the pointer and the identity (`:id`, `:file-id`) always come
|
||||
from the read map.
|
||||
(if (::reset-migrations? opts false)
|
||||
(fmigr/reset-migrations! conn file)
|
||||
(fmigr/upsert-migrations! conn file))
|
||||
|
||||
Returns the map `fdata/upsert!` answers with: `:written` and the
|
||||
`:storage-object-id` of the object this write created, so a caller that
|
||||
persists again touches the current object instead of the replaced one."
|
||||
[{:keys [::db/conn] :as cfg} file & {:as params}]
|
||||
(let [file
|
||||
(encode-file cfg file)
|
||||
|
||||
;; A map read with `:load-data? false` has no `:data`; persisting it would
|
||||
;; wipe the file content. `ex/raise` instead of `assert`: the guard must
|
||||
;; exist in the deployed binary, where `clojure.core/assert` is compiled
|
||||
;; out.
|
||||
(when-not (contains? file :data)
|
||||
(ex/raise :type :internal
|
||||
:code :missing-file-data
|
||||
:hint "expected a file map read with data"
|
||||
:file-id (:id file)))
|
||||
file-params
|
||||
(file->params (dissoc file :id))
|
||||
|
||||
(let [reset? (::reset-migrations? params false)
|
||||
;; The pointer and the identity are bookkeeping of the persisted row,
|
||||
;; not editable fields: they can only come from the read map. A pointer
|
||||
;; in the params would skip the touch and leak the replaced blob; an
|
||||
;; `:id` would retarget the UPDATE.
|
||||
file (merge file (dissoc params ::reset-migrations?
|
||||
:storage-object-id
|
||||
:id
|
||||
:file-id))]
|
||||
file-data-params
|
||||
(file->file-data-params file)]
|
||||
|
||||
(if reset?
|
||||
(fmigr/reset-migrations! conn file)
|
||||
(fmigr/upsert-migrations! conn file))
|
||||
(db/update! conn :file file-params
|
||||
{:id id}
|
||||
{::db/return-keys false})
|
||||
|
||||
(let [file (encode-file cfg file)
|
||||
file-params (file->params (dissoc file :id))]
|
||||
|
||||
(db/update! conn :file file-params
|
||||
{:id (:id file)}
|
||||
{::db/return-keys false})
|
||||
|
||||
(fdata/upsert! cfg (file->file-data-params file)))))
|
||||
(fdata/upsert! cfg file-data-params)
|
||||
nil))
|
||||
|
||||
(defn save-file!
|
||||
"Applies all the final validations and perist the file, binfile
|
||||
@ -993,7 +859,8 @@
|
||||
(let [file (-> file
|
||||
(assoc :created-at timestamp)
|
||||
(assoc :modified-at timestamp)
|
||||
(assoc :ignore-sync-until (ct/plus timestamp (ct/duration {:seconds 5})))
|
||||
(cond-> (not (::overwrite cfg))
|
||||
(assoc :ignore-sync-until (ct/plus timestamp (ct/duration {:seconds 5}))))
|
||||
(update :features
|
||||
(fn [features]
|
||||
(-> (::features cfg #{})
|
||||
@ -1010,7 +877,9 @@
|
||||
(when (ex/exception? result)
|
||||
(l/error :hint "file schema validation error" :cause result))))
|
||||
|
||||
(insert-file! cfg file opts)))
|
||||
(if (::overwrite cfg)
|
||||
(update-file! cfg file (assoc opts ::reset-migrations? true))
|
||||
(insert-file! cfg file opts))))
|
||||
|
||||
(def ^:private sql:get-file-libraries
|
||||
"WITH RECURSIVE libs AS (
|
||||
@ -1075,12 +944,7 @@
|
||||
(let [library-ids (->> (get-file-libraries conn (:id file))
|
||||
(map :id)
|
||||
(cons (:id file)))
|
||||
;; Fragment rows are file-local and callers bind the loader of
|
||||
;; the file they work on, so each library realizes its pointers
|
||||
;; here, with its own id, before anything walks its data.
|
||||
load-fn (fn [library-id]
|
||||
(->> (get-file cfg library-id :migrate? false)
|
||||
(fdata/realize-pointers cfg)))]
|
||||
load-fn #(get-file cfg % :migrate? false)]
|
||||
(weak/loadable-weak-value-map library-ids load-fn {id file})))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
@ -1198,46 +1062,3 @@
|
||||
{:bytes bytes
|
||||
:size (alength ^bytes bytes)
|
||||
:hash (sto/calculate-hash (java.io.ByteArrayInputStream. bytes))})))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; REQUEST CFG
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(defn resolve-export-type
|
||||
"The canonical export type of a request on the legacy RPC: an explicit
|
||||
`type` wins, and the deprecated boolean flags are translated. The
|
||||
creation command of a job does not need it, because it asks the caller
|
||||
for the type instead of guessing it."
|
||||
[{:keys [type include-libraries embed-assets]}]
|
||||
(cond
|
||||
(some? type) type
|
||||
(true? include-libraries) :include-libraries
|
||||
(true? embed-assets) :merge-libraries
|
||||
:else :detach-libraries))
|
||||
|
||||
(defn export-cfg
|
||||
"The domain cfg of an export: the frozen set of files, how their
|
||||
libraries are handled, and where the package is written."
|
||||
[cfg {:keys [file-ids export-type output]}]
|
||||
(-> cfg
|
||||
(assoc ::file-ids file-ids)
|
||||
(assoc ::export-type export-type)
|
||||
(assoc ::output output)))
|
||||
|
||||
(defn import-cfg
|
||||
"The domain cfg of an import: the destination team/profile/project, the
|
||||
name, the input and the anti-zip-bomb limits. The caller resolves the
|
||||
team, because what allows it to import there differs between the RPC and
|
||||
a job; what both must share is fixed here."
|
||||
[cfg {:keys [profile-id project-id team name input]}]
|
||||
(-> cfg
|
||||
(assoc ::features (cfeat/get-team-enabled-features cf/flags team))
|
||||
(assoc ::project-id project-id)
|
||||
(assoc ::profile-id profile-id)
|
||||
(assoc ::team-id (:id team))
|
||||
(assoc ::name name)
|
||||
(assoc ::input input)
|
||||
(assoc ::import-max-binary-entry-size (cf/get :binfile-import-max-binary-entry-size))
|
||||
(assoc ::import-max-text-entry-size (cf/get :binfile-import-max-text-entry-size))
|
||||
(assoc ::import-max-text-total-size (cf/get :binfile-import-max-text-total-size))
|
||||
(assoc ::import-max-zip-entries (cf/get :binfile-import-max-zip-entries))))
|
||||
|
||||
@ -1,120 +0,0 @@
|
||||
;; This Source Code Form is subject to the terms of the Mozilla Public
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.binfile.jobs
|
||||
"The progress bridge of the binfile jobs.
|
||||
|
||||
A job has no SSE channel, so the `:progress` points that `bf.v1` and
|
||||
`bf.v3` emit become `progress` events of the job through
|
||||
`jobs/heartbeat`, which throttles them and drops the ones that do not
|
||||
fit. The adapter turns a tap into a self-contained milestone: a `:stage`
|
||||
of a single vocabulary (the two formats name the same work differently)
|
||||
and the counters the tap carries, with the outer scopes in course as
|
||||
context. The reader puts the outer counters in every tap inside them,
|
||||
so the adapter holds no state between taps. The ids the taps carry are
|
||||
not part of the job event contract.
|
||||
|
||||
The handlers in `app.tasks.export-binfile` and
|
||||
`app.tasks.import-binfile` run the core through `run` and
|
||||
guard it with `check-active`: a job that was cancelled, or that
|
||||
already reached a terminal state, raises an `:interrupt` right there on
|
||||
the run thread, so the work stops at its next event point and the
|
||||
transaction of its handler rolls back with it. A run whose last tap
|
||||
happened before the interruption is discarded once it returns, which is
|
||||
what makes that transaction roll back too."
|
||||
(:require
|
||||
[app.jobs :as jobs]
|
||||
[app.util.events :as events]))
|
||||
|
||||
(defn check-active
|
||||
"Raise an `:interrupt` when the job is no longer active.
|
||||
|
||||
The substrate owns both the check and the error; this is the adapter's
|
||||
entry point to it, used by the two handlers before their run and by the
|
||||
tests. Answers with the cfg, so a handler can thread it."
|
||||
[cfg context]
|
||||
(jobs/check-active cfg (:id context)))
|
||||
|
||||
(def ^:private stage-of
|
||||
"The stage a tap section becomes: one vocabulary for the two formats, so
|
||||
a client resolves each stage once. The counter of the activity is named
|
||||
after its own stage, which is what lets a client render
|
||||
`counters[stage]` without a table of equivalences."
|
||||
{:file :files
|
||||
:page :pages
|
||||
:media :media
|
||||
:thumbnail :thumbnails
|
||||
:thumbnails :thumbnails
|
||||
:component :components
|
||||
:color :colors
|
||||
:typography :typographies
|
||||
:tokens-lib :tokens
|
||||
:tokens-status :tokens
|
||||
:storage-object :storage-objects
|
||||
:storage-objects :storage-objects
|
||||
:relations :relations
|
||||
:manifest :manifest
|
||||
:v1/metadata :metadata
|
||||
:v1/files :files
|
||||
:v1/rels :relations
|
||||
:v1/sobjects :storage-objects})
|
||||
|
||||
(defn- stage
|
||||
"The stage of a tap section; a section without a name of its own keeps
|
||||
it, so a new tap is never dropped from the stream."
|
||||
[section]
|
||||
(get stage-of section section))
|
||||
|
||||
(defn- counter
|
||||
"The counter a tap reports, when it reports one: a tap that only marks a
|
||||
point of the run has no units to count."
|
||||
[{:keys [current total]}]
|
||||
(when (some? current)
|
||||
(cond-> {:current current}
|
||||
(some? total)
|
||||
(assoc :total total))))
|
||||
|
||||
(defn- milestone
|
||||
"The milestone a tap becomes: what the job is doing now and the counters
|
||||
the tap carries, with the outer scopes in course as context.
|
||||
|
||||
The tap of a scope carries its outer counters, so the milestone is
|
||||
self-contained and the adapter remembers nothing between taps: a client
|
||||
that misses one (the substrate throttles the events) still renders the
|
||||
truth from the next."
|
||||
[{:keys [section outer-counters] :as tap}]
|
||||
(let [stage (stage section)
|
||||
own (counter tap)]
|
||||
(cond-> {:stage stage}
|
||||
(or (some? outer-counters) (some? own))
|
||||
(assoc :counters (cond-> (or outer-counters {})
|
||||
(some? own) (assoc stage own))))))
|
||||
|
||||
(defn run
|
||||
"Run `f` with the progress taps of the core turned into milestones of the
|
||||
job.
|
||||
|
||||
`f` is called with the cfg, so the adapter forwards what it receives
|
||||
and a caller can pass the core directly.
|
||||
|
||||
Every tap is reported: the substrate throttles the durable events, and
|
||||
a milestone that does not fit loses no information because the next one
|
||||
carries the same state.
|
||||
|
||||
The taps run on this same thread: the sink calls the heartbeat inline,
|
||||
so an `:interrupt` it raises aborts the core right there, inside the
|
||||
transaction of its handler, which rolls back."
|
||||
[cfg context f]
|
||||
(let [job-id (:id context)
|
||||
sink (fn [[type data]]
|
||||
(when (= :progress type)
|
||||
(jobs/heartbeat cfg
|
||||
:job-id job-id
|
||||
:progress (milestone data))))]
|
||||
(binding [events/*sink* sink]
|
||||
(let [result (f cfg)]
|
||||
(check-active cfg context)
|
||||
result))))
|
||||
@ -329,13 +329,13 @@
|
||||
[:v1/metadata :v1/files :v1/rels :v1/sobjects]))))
|
||||
|
||||
(defmethod write-section :v1/metadata
|
||||
[{:keys [::output ::bfc/file-ids ::bfc/include-libraries] :as cfg}]
|
||||
(if-let [fids (get-files cfg file-ids)]
|
||||
[{:keys [::output ::bfc/ids ::bfc/include-libraries] :as cfg}]
|
||||
(if-let [fids (get-files cfg ids)]
|
||||
(let [lids (when include-libraries
|
||||
(bfc/get-libraries cfg file-ids))
|
||||
file-ids (into fids lids)]
|
||||
(write-obj! output {:version cf/version :files file-ids})
|
||||
(vswap! bfc/*state* assoc :files file-ids))
|
||||
(bfc/get-libraries cfg ids))
|
||||
ids (into fids lids)]
|
||||
(write-obj! output {:version cf/version :files ids})
|
||||
(vswap! bfc/*state* assoc :files ids))
|
||||
(ex/raise :type :not-found
|
||||
:code :files-not-found
|
||||
:hint "unable to retrieve files for export")))
|
||||
@ -482,11 +482,9 @@
|
||||
(bfm/apply-pending-migrations! cfg)
|
||||
|
||||
;; Knowing that the ids of the created files are in index,
|
||||
;; just lookup them and return them in the same shape as v3. The
|
||||
;; metadata of an old package may repeat a file id, so the answer
|
||||
;; carries one id per imported file.
|
||||
;; just lookup them and return them in the same shape as v3.
|
||||
(let [files (-> bfc/*state* deref :files)]
|
||||
{:file-ids (into [] (comp (keep #(get-in @bfc/*state* [:index %])) (distinct)) files)})))))
|
||||
{:file-ids (into [] (keep #(get-in @bfc/*state* [:index %])) files)})))))
|
||||
|
||||
(defmethod read-import :v1
|
||||
[options]
|
||||
@ -512,114 +510,89 @@
|
||||
|
||||
(defmethod read-section :v1/files
|
||||
[{:keys [::bfc/input ::bfc/project-id ::bfc/name] :as system}]
|
||||
(let [imported (volatile! #{})]
|
||||
(doseq [[idx expected-file-id] (d/enumerate (-> bfc/*state* deref :files))]
|
||||
(let [file (read-obj! input)
|
||||
media (read-obj! input)
|
||||
file-id (:id file)]
|
||||
(doseq [[idx expected-file-id] (d/enumerate (-> bfc/*state* deref :files))]
|
||||
(let [file (read-obj! input)
|
||||
media (read-obj! input)
|
||||
|
||||
(when (not= file-id expected-file-id)
|
||||
(ex/raise :type :validation
|
||||
:code :inconsistent-penpot-file
|
||||
:found-id file-id
|
||||
:expected-id expected-file-id
|
||||
:hint "the penpot file seems corrupt, found unexpected uuid (file-id)"))
|
||||
file-id (:id file)
|
||||
file-id' (bfc/lookup-index file-id)
|
||||
|
||||
(if (contains? @imported file-id)
|
||||
;; Old exports could repeat a file id in the metadata (the file was
|
||||
;; also one of its own libraries). The entry is read above to keep
|
||||
;; the stream in sync, but the file is imported only once.
|
||||
(l/warn :hint "ignoring repeated file entry"
|
||||
:file-id (str file-id)
|
||||
::l/sync? true)
|
||||
file (bfc/clean-file-features file)
|
||||
thumbnails (:thumbnails file)]
|
||||
|
||||
(let [file-id' (bfc/lookup-index file-id)
|
||||
file (bfc/clean-file-features file)
|
||||
thumbnails (:thumbnails file)]
|
||||
(when (not= file-id expected-file-id)
|
||||
(ex/raise :type :validation
|
||||
:code :inconsistent-penpot-file
|
||||
:found-id file-id
|
||||
:expected-id expected-file-id
|
||||
:hint "the penpot file seems corrupt, found unexpected uuid (file-id)"))
|
||||
|
||||
(vswap! imported conj file-id)
|
||||
(l/dbg :hint "processing file"
|
||||
:id (str file-id)
|
||||
:features (:features file)
|
||||
:version (-> file :data :version)
|
||||
:media (count media)
|
||||
:thumbnails (count thumbnails)
|
||||
::l/sync? true)
|
||||
|
||||
(l/dbg :hint "processing file"
|
||||
:id (str file-id)
|
||||
:features (:features file)
|
||||
:version (-> file :data :version)
|
||||
:media (count media)
|
||||
:thumbnails (count thumbnails)
|
||||
::l/sync? true)
|
||||
(when (seq thumbnails)
|
||||
(let [thumbnails (remap-thumbnails thumbnails file-id')]
|
||||
(l/dbg :hint "updated index with thumbnails"
|
||||
:total (count thumbnails)
|
||||
::l/sync? true)
|
||||
(vswap! bfc/*state* update :thumbnails bfc/into-vec thumbnails)))
|
||||
|
||||
(when (seq thumbnails)
|
||||
(let [thumbnails (remap-thumbnails thumbnails file-id')]
|
||||
(l/dbg :hint "updated index with thumbnails"
|
||||
:total (count thumbnails)
|
||||
::l/sync? true)
|
||||
(vswap! bfc/*state* update :thumbnails bfc/into-vec thumbnails)))
|
||||
(when (seq media)
|
||||
;; Update index with media
|
||||
(l/dbg :hint "update index with media" :total (count media) ::l/sync? true)
|
||||
(vswap! bfc/*state* update :index bfc/update-index (map :id media))
|
||||
|
||||
(when (seq media)
|
||||
;; Update index with media
|
||||
(l/dbg :hint "update index with media" :total (count media) ::l/sync? true)
|
||||
(vswap! bfc/*state* update :index bfc/update-index (map :id media))
|
||||
;; Store file media for later insertion
|
||||
(l/dbg :hint "update media references" ::l/sync? true)
|
||||
(vswap! bfc/*state* update :media into (map #(update % :id bfc/lookup-index)) media))
|
||||
|
||||
;; Store file media for later insertion
|
||||
(l/dbg :hint "update media references" ::l/sync? true)
|
||||
(vswap! bfc/*state* update :media into (map #(update % :id bfc/lookup-index)) media))
|
||||
(let [file (-> file
|
||||
(assoc :id file-id')
|
||||
(cond-> (and (= idx 0) (some? name))
|
||||
(assoc :name name))
|
||||
(assoc :project-id project-id)
|
||||
(dissoc :thumbnails))
|
||||
file (bfc/process-file system file)]
|
||||
|
||||
(let [file (-> file
|
||||
(assoc :id file-id')
|
||||
(cond-> (and (= idx 0) (some? name))
|
||||
(assoc :name name))
|
||||
(assoc :project-id project-id)
|
||||
(dissoc :thumbnails))
|
||||
file (bfc/process-file system file)]
|
||||
;; All features that are enabled and requires explicit migration are
|
||||
;; added to the state for a posterior migration step.
|
||||
(doseq [feature (-> (::bfc/features system)
|
||||
(set/difference cfeat/no-migration-features)
|
||||
(set/difference (:features file)))]
|
||||
(vswap! bfc/*state* update :pending-to-migrate (fnil conj []) [feature file-id']))
|
||||
|
||||
;; All features that are enabled and requires explicit migration
|
||||
;; are added to the state for a posterior migration step.
|
||||
(doseq [feature (-> (::bfc/features system)
|
||||
(set/difference cfeat/no-migration-features)
|
||||
(set/difference (:features file)))]
|
||||
(vswap! bfc/*state* update :pending-to-migrate (fnil conj []) [feature file-id']))
|
||||
(l/dbg :hint "create file" :id (str file-id') ::l/sync? true)
|
||||
(bfc/save-file! system file ::db/return-keys false)
|
||||
|
||||
(l/dbg :hint "create file" :id (str file-id') ::l/sync? true)
|
||||
(bfc/save-file! system file ::db/return-keys false)
|
||||
|
||||
file-id')))))))
|
||||
file-id'))))
|
||||
|
||||
(defmethod read-section :v1/rels
|
||||
[{:keys [::db/conn ::bfc/input ::bfc/timestamp] :as cfg}]
|
||||
(let [rels (->> (read-obj! input)
|
||||
(sort-by (fn [{:keys [file-id library-file-id]}]
|
||||
[(str file-id) (str library-file-id)])))
|
||||
[{:keys [::db/conn ::bfc/input ::bfc/timestamp]}]
|
||||
(let [rels (read-obj! input)
|
||||
ids (into #{} (-> bfc/*state* deref :files))]
|
||||
;; Insert all file relations
|
||||
(doseq [{:keys [library-file-id] :as rel} rels]
|
||||
(let [rel (-> rel
|
||||
(update :file-id bfc/lookup-index)
|
||||
(update :library-file-id bfc/lookup-index))]
|
||||
|
||||
(if (contains? ids library-file-id)
|
||||
(case (bfc/library-link-status cfg (:file-id rel) (:library-file-id rel))
|
||||
:self-link
|
||||
(l/warn :hint "ignoring self-referencing library link"
|
||||
:file-id (:file-id rel)
|
||||
:lib-id (:library-file-id rel)
|
||||
::l/sync? true)
|
||||
|
||||
:circular
|
||||
(l/warn :hint "ignoring circular library link"
|
||||
:file-id (:file-id rel)
|
||||
:lib-id (:library-file-id rel)
|
||||
::l/sync? true)
|
||||
|
||||
:ok
|
||||
(do
|
||||
(l/dbg :hint "create file library link"
|
||||
:file-id (:file-id rel)
|
||||
:lib-id (:library-file-id rel)
|
||||
::l/sync? true)
|
||||
(let [rel-params (dissoc rel :synced-at)]
|
||||
(db/insert! conn :file-library-rel rel-params)
|
||||
(bfc/upsert-file-library-sync! conn {:file-id (:file-id rel-params)
|
||||
:library-file-id (:library-file-id rel-params)
|
||||
:synced-at (or (:synced-at rel)
|
||||
timestamp)}))))
|
||||
(do
|
||||
(l/dbg :hint "create file library link"
|
||||
:file-id (:file-id rel)
|
||||
:lib-id (:library-file-id rel)
|
||||
::l/sync? true)
|
||||
(let [rel-params (dissoc rel :synced-at)]
|
||||
(db/insert! conn :file-library-rel rel-params)
|
||||
(bfc/upsert-file-library-sync! conn {:file-id (:file-id rel-params)
|
||||
:library-file-id (:library-file-id rel-params)
|
||||
:synced-at (or (:synced-at rel)
|
||||
timestamp)})))
|
||||
|
||||
(l/warn :hint "ignoring file library link"
|
||||
:file-id (:file-id rel)
|
||||
@ -716,12 +689,12 @@
|
||||
`::bfc/embed-assets`: instead of including the libraries, embed in the
|
||||
same file library all assets used from external libraries."
|
||||
|
||||
[{:keys [::bfc/file-ids] :as cfg} output]
|
||||
[{:keys [::bfc/ids] :as cfg} output]
|
||||
|
||||
(dm/assert!
|
||||
"expected a set of uuid's for `::bfc/file-ids` parameter"
|
||||
(and (set? file-ids)
|
||||
(every? uuid? file-ids)))
|
||||
"expected a set of uuid's for `::bfc/ids` parameter"
|
||||
(and (set? ids)
|
||||
(every? uuid? ids)))
|
||||
|
||||
(dm/assert!
|
||||
"expected instance of jio/IOFactory for `input`"
|
||||
|
||||
445
backend/src/app/binfile/v2.clj
Normal file
445
backend/src/app/binfile/v2.clj
Normal file
@ -0,0 +1,445 @@
|
||||
;; This Source Code Form is subject to the terms of the Mozilla Public
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.binfile.v2
|
||||
"A sqlite3 based binary file exportation with support for exportation
|
||||
of entire team (or multiple teams) at once."
|
||||
(:refer-clojure :exclude [read])
|
||||
(:require
|
||||
[app.binfile.common :as bfc]
|
||||
[app.common.data :as d]
|
||||
[app.common.features :as cfeat]
|
||||
[app.common.logging :as l]
|
||||
[app.common.time :as ct]
|
||||
[app.common.transit :as t]
|
||||
[app.common.uuid :as uuid]
|
||||
[app.config :as cf]
|
||||
[app.db :as db]
|
||||
[app.db.sql :as sql]
|
||||
[app.loggers.audit :as-alias audit]
|
||||
[app.loggers.webhooks :as-alias webhooks]
|
||||
[app.storage :as sto]
|
||||
[app.storage.tmp :as tmp]
|
||||
[app.util.events :as events]
|
||||
[app.worker :as-alias wrk]
|
||||
[clojure.set :as set]
|
||||
[cuerdas.core :as str]
|
||||
[datoteka.io :as io]
|
||||
[promesa.util :as pu])
|
||||
(:import
|
||||
java.sql.DriverManager))
|
||||
|
||||
(set! *warn-on-reflection* true)
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; LOW LEVEL API
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(defn- create-database
|
||||
([cfg]
|
||||
(let [path (tmp/tempfile :prefix "penpot.binfile." :suffix ".sqlite")]
|
||||
(create-database cfg path)))
|
||||
([cfg path]
|
||||
(let [db (DriverManager/getConnection (str "jdbc:sqlite:" path))]
|
||||
(assoc cfg ::db db ::path path))))
|
||||
|
||||
(def ^:private
|
||||
sql:create-kvdata-table
|
||||
"CREATE TABLE kvdata (
|
||||
tag text NOT NULL,
|
||||
key text NOT NULL,
|
||||
val text NOT NULL,
|
||||
dat blob NULL
|
||||
)")
|
||||
|
||||
(def ^:private
|
||||
sql:create-kvdata-index
|
||||
"CREATE INDEX kvdata__tag_key__idx
|
||||
ON kvdata (tag, key)")
|
||||
|
||||
(defn- setup-schema!
|
||||
[{:keys [::db]}]
|
||||
(db/exec-one! db [sql:create-kvdata-table])
|
||||
(db/exec-one! db [sql:create-kvdata-index]))
|
||||
|
||||
(defn- write!
|
||||
[{:keys [::db]} tag k v & [data]]
|
||||
(db/insert! db :kvdata
|
||||
{:tag (d/name tag)
|
||||
:key (str k)
|
||||
:val (t/encode-str v {:type :json-verbose})
|
||||
:dat data}
|
||||
{::db/return-keys false}))
|
||||
|
||||
(defn- read-blob
|
||||
[{:keys [::db]} tag k]
|
||||
(let [obj (db/get db :kvdata
|
||||
{:tag (d/name tag)
|
||||
:key (str k)}
|
||||
{::sql/columns [:dat]})]
|
||||
(:dat obj)))
|
||||
|
||||
(defn- read-seq
|
||||
([{:keys [::db]} tag]
|
||||
(->> (db/query db :kvdata
|
||||
{:tag (d/name tag)}
|
||||
{::sql/columns [::val]})
|
||||
(map :val)
|
||||
(map t/decode-str)))
|
||||
([{:keys [::db]} tag k]
|
||||
(->> (db/query db :kvdata
|
||||
{:tag (d/name tag)
|
||||
:key (str k)}
|
||||
{::sql/columns [::val]})
|
||||
(map :val)
|
||||
(map t/decode-str))))
|
||||
|
||||
(defn- read-obj
|
||||
[{:keys [::db]} tag k]
|
||||
(let [obj (db/get db :kvdata
|
||||
{:tag (d/name tag)
|
||||
:key (str k)}
|
||||
{::sql/columns [:val]})]
|
||||
(-> obj :val t/decode-str)))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; IMPORT/EXPORT IMPL
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(declare ^:private write-project!)
|
||||
(declare ^:private write-file!)
|
||||
|
||||
(defn- write-team!
|
||||
[cfg team-id]
|
||||
|
||||
(let [team (bfc/get-team cfg team-id)
|
||||
fonts (bfc/get-fonts cfg team-id)]
|
||||
|
||||
(events/tap :progress
|
||||
{:op :export
|
||||
:section :write-team
|
||||
:id team-id
|
||||
:name (:name team)})
|
||||
|
||||
(l/trc :hint "write" :obj "team"
|
||||
:id (str team-id)
|
||||
:fonts (count fonts))
|
||||
|
||||
(when-let [photo-id (:photo-id team)]
|
||||
(vswap! bfc/*state* update :storage-objects conj photo-id))
|
||||
|
||||
(vswap! bfc/*state* update :teams conj team-id)
|
||||
(vswap! bfc/*state* bfc/collect-storage-objects fonts)
|
||||
|
||||
(write! cfg :team team-id team)
|
||||
|
||||
(doseq [{:keys [id] :as font} fonts]
|
||||
(vswap! bfc/*state* update :team-font-variants conj id)
|
||||
(write! cfg :team-font-variant id font))))
|
||||
|
||||
(defn- write-project!
|
||||
[cfg project]
|
||||
(events/tap :progress
|
||||
{:op :export
|
||||
:section :write-project
|
||||
:id (:id project)
|
||||
:name (:name project)})
|
||||
(l/trc :hint "write" :obj "project" :id (str (:id project)))
|
||||
(write! cfg :project (str (:id project)) project)
|
||||
(vswap! bfc/*state* update :projects conj (:id project)))
|
||||
|
||||
(defn- write-file!
|
||||
[cfg file-id]
|
||||
(let [file (bfc/get-file cfg file-id :realize? true)
|
||||
thumbs (bfc/get-file-object-thumbnails cfg file-id)
|
||||
media (bfc/get-file-media cfg file)
|
||||
rels (bfc/get-files-rels cfg #{file-id})]
|
||||
|
||||
(events/tap :progress
|
||||
{:op :export
|
||||
:section :write-file
|
||||
:id file-id
|
||||
:name (:name file)})
|
||||
|
||||
(vswap! bfc/*state* (fn [state]
|
||||
(-> state
|
||||
(update :files conj file-id)
|
||||
(update :file-media-objects into bfc/xf-map-id media)
|
||||
(bfc/collect-storage-objects thumbs)
|
||||
(bfc/collect-storage-objects media))))
|
||||
|
||||
(write! cfg :file file-id file)
|
||||
(write! cfg :file-rels file-id rels)
|
||||
|
||||
(run! (partial write! cfg :file-media-object file-id) media)
|
||||
(run! (partial write! cfg :file-object-thumbnail file-id) thumbs)
|
||||
|
||||
(when-let [thumb (bfc/get-file-thumbnail cfg file)]
|
||||
(vswap! bfc/*state* bfc/collect-storage-objects [thumb])
|
||||
(write! cfg :file-thumbnail file-id thumb))
|
||||
|
||||
(l/trc :hint "write" :obj "file"
|
||||
:thumbnails (count thumbs)
|
||||
:rels (count rels)
|
||||
:media (count media))))
|
||||
|
||||
(defn- write-storage-object!
|
||||
[{:keys [::sto/storage] :as cfg} id]
|
||||
(let [sobj (sto/get-object storage id)
|
||||
data (with-open [input (sto/get-object-data storage sobj)]
|
||||
(io/read input))]
|
||||
|
||||
(l/trc :hint "write" :obj "storage-object" :id (str id) :size (:size sobj))
|
||||
(write! cfg :storage-object id (meta sobj) data)))
|
||||
|
||||
(defn- read-storage-object!
|
||||
[{:keys [::sto/storage ::bfc/timestamp] :as cfg} id]
|
||||
(let [mdata (d/update-when (read-obj cfg :storage-object id)
|
||||
:content-type bfc/normalize-content-type)
|
||||
data (read-blob cfg :storage-object id)
|
||||
;; NOTE: v2 bundles are legacy and no longer produced; the
|
||||
;; sanitizer call stays as defense in depth and is covered by
|
||||
;; the shared helper unit tests, with no dedicated v2
|
||||
;; import test.
|
||||
clean (when (bfc/svg-object? mdata)
|
||||
(bfc/sanitize-imported-svg mdata data))
|
||||
data (or (:bytes clean) data)
|
||||
hash (or (:hash clean) (sto/calculate-hash data))
|
||||
|
||||
content (-> (sto/content data)
|
||||
(sto/wrap-with-hash hash))
|
||||
|
||||
params (-> mdata
|
||||
(assoc ::sto/content content)
|
||||
(assoc ::sto/deduplicate? true)
|
||||
(assoc ::sto/touched-at timestamp))
|
||||
|
||||
sobject (sto/put-object! storage params)]
|
||||
|
||||
(vswap! bfc/*state* update :index assoc id (:id sobject))
|
||||
|
||||
(l/trc :hint "read" :obj "storage-object"
|
||||
:id (str id)
|
||||
:new-id (str (:id sobject))
|
||||
:size (:size sobject))))
|
||||
|
||||
(defn read-team!
|
||||
[{:keys [::db/conn ::bfc/timestamp] :as cfg} team-id]
|
||||
(l/trc :hint "read" :obj "team" :id (str team-id))
|
||||
|
||||
(let [team (read-obj cfg :team team-id)
|
||||
team (-> team
|
||||
(update :id bfc/lookup-index)
|
||||
(update :photo-id bfc/lookup-index)
|
||||
(assoc :created-at timestamp)
|
||||
(assoc :modified-at timestamp))]
|
||||
|
||||
(events/tap :progress
|
||||
{:op :import
|
||||
:section :read-team
|
||||
:id team-id
|
||||
:name (:name team)})
|
||||
|
||||
(db/insert! conn :team
|
||||
(update team :features db/encode-pgarray conn "text")
|
||||
::db/return-keys false)
|
||||
|
||||
(doseq [font (->> (read-seq cfg :team-font-variant)
|
||||
(filter #(= team-id (:team-id %))))]
|
||||
(let [font (-> font
|
||||
(update :id bfc/lookup-index)
|
||||
(update :team-id bfc/lookup-index)
|
||||
(update :woff1-file-id bfc/lookup-index)
|
||||
(update :woff2-file-id bfc/lookup-index)
|
||||
(update :ttf-file-id bfc/lookup-index)
|
||||
(update :otf-file-id bfc/lookup-index)
|
||||
(assoc :created-at timestamp)
|
||||
(assoc :modified-at timestamp))]
|
||||
(db/insert! conn :team-font-variant font
|
||||
::db/return-keys false)))
|
||||
|
||||
team))
|
||||
|
||||
(defn read-project!
|
||||
[{:keys [::db/conn ::bfc/timestamp] :as cfg} project-id]
|
||||
(l/trc :hint "read" :obj "project" :id (str project-id))
|
||||
|
||||
(let [project (read-obj cfg :project project-id)
|
||||
project (-> project
|
||||
(update :id bfc/lookup-index)
|
||||
(update :team-id bfc/lookup-index)
|
||||
(assoc :created-at timestamp)
|
||||
(assoc :modified-at timestamp))]
|
||||
|
||||
(events/tap :progress
|
||||
{:op :import
|
||||
:section :read-project
|
||||
:id project-id
|
||||
:name (:name project)})
|
||||
|
||||
(db/insert! conn :project project
|
||||
::db/return-keys false)))
|
||||
|
||||
(defn read-file!
|
||||
[{:keys [::db/conn ::bfc/timestamp] :as cfg} file-id]
|
||||
(l/trc :hint "read" :obj "file" :id (str file-id))
|
||||
|
||||
(let [file (-> (read-obj cfg :file file-id)
|
||||
(update :id bfc/lookup-index)
|
||||
(update :project-id bfc/lookup-index))
|
||||
file (bfc/process-file cfg file)]
|
||||
|
||||
(events/tap :progress
|
||||
{:op :import
|
||||
:section :read-file
|
||||
:id file-id
|
||||
:name (:name file)})
|
||||
|
||||
;; All features that are enabled and requires explicit migration are
|
||||
;; added to the state for a posterior migration step.
|
||||
(doseq [feature (-> (::bfc/features cfg)
|
||||
(set/difference cfeat/no-migration-features)
|
||||
(set/difference (:features file)))]
|
||||
(vswap! bfc/*state* update :pending-to-migrate (fnil conj []) [feature (:id file)]))
|
||||
|
||||
(bfc/save-file! cfg file ::db/return-keys false))
|
||||
|
||||
(doseq [thumbnail (read-seq cfg :file-object-thumbnail file-id)]
|
||||
(let [thumbnail (-> thumbnail
|
||||
(update :file-id bfc/lookup-index)
|
||||
(update :media-id bfc/lookup-index))
|
||||
file-id (:file-id thumbnail)
|
||||
|
||||
thumbnail (update thumbnail :object-id
|
||||
#(str/replace-first % #"^(.*?)/" (str file-id "/")))]
|
||||
|
||||
(db/insert! conn :file-tagged-object-thumbnail thumbnail
|
||||
{::db/return-keys false})))
|
||||
|
||||
(doseq [rel (read-obj cfg :file-rels file-id)]
|
||||
(let [rel (-> rel
|
||||
(update :file-id bfc/lookup-index)
|
||||
(update :library-file-id bfc/lookup-index))]
|
||||
(db/insert! conn :file-library-rel rel
|
||||
::db/return-keys false)
|
||||
(bfc/upsert-file-library-sync! conn (assoc rel :synced-at timestamp))))
|
||||
|
||||
(doseq [media (read-seq cfg :file-media-object file-id)]
|
||||
(let [media (-> media
|
||||
(update :id bfc/lookup-index)
|
||||
(update :file-id bfc/lookup-index)
|
||||
(update :media-id bfc/lookup-index)
|
||||
(update :thumbnail-id bfc/lookup-index))]
|
||||
(db/insert! conn :file-media-object media
|
||||
::db/return-keys false
|
||||
::sql/on-conflict-do-nothing true))))
|
||||
|
||||
(def ^:private empty-summary
|
||||
{:teams #{}
|
||||
:files #{}
|
||||
:projects #{}
|
||||
:file-media-objects #{}
|
||||
:team-font-variants #{}
|
||||
:storage-objects #{}})
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; PUBLIC API
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
(defn export-team!
|
||||
[cfg team-id]
|
||||
(let [id (uuid/next)
|
||||
tp (ct/tpoint)
|
||||
cfg (create-database cfg)]
|
||||
|
||||
(l/inf :hint "start"
|
||||
:operation "export"
|
||||
:id (str id)
|
||||
:path (str (::path cfg)))
|
||||
|
||||
(try
|
||||
(db/tx-run! cfg (fn [cfg]
|
||||
(setup-schema! cfg)
|
||||
(binding [bfc/*state* (volatile! empty-summary)]
|
||||
(write-team! cfg team-id)
|
||||
|
||||
(run! (partial write-project! cfg)
|
||||
(bfc/get-team-projects cfg team-id))
|
||||
|
||||
(run! (partial write-file! cfg)
|
||||
(bfc/get-team-files-ids cfg team-id))
|
||||
|
||||
(run! (partial write-storage-object! cfg)
|
||||
(-> bfc/*state* deref :storage-objects))
|
||||
|
||||
(write! cfg :manifest "team-id" team-id)
|
||||
(write! cfg :manifest "objects" (deref bfc/*state*))
|
||||
|
||||
(::path cfg))))
|
||||
(finally
|
||||
(pu/close! (::db cfg))
|
||||
|
||||
(let [elapsed (tp)]
|
||||
(l/inf :hint "end"
|
||||
:operation "export"
|
||||
:id (str id)
|
||||
:elapsed (ct/format-duration elapsed)))))))
|
||||
|
||||
(defn import-team!
|
||||
[cfg path]
|
||||
(let [id (uuid/next)
|
||||
tp (ct/tpoint)
|
||||
|
||||
cfg (-> (create-database cfg path)
|
||||
(assoc ::bfc/timestamp (ct/now)))]
|
||||
|
||||
(l/inf :hint "start"
|
||||
:operation "import"
|
||||
:id (str id)
|
||||
:path (str (::path cfg)))
|
||||
|
||||
(try
|
||||
(db/tx-run! cfg (fn [{:keys [::db/conn] :as cfg}]
|
||||
(db/exec-one! conn ["SET idle_in_transaction_session_timeout = 0"])
|
||||
(db/exec-one! conn ["SET CONSTRAINTS ALL DEFERRED"])
|
||||
|
||||
(binding [bfc/*state* (volatile! {:index {}})]
|
||||
(let [objects (read-obj cfg :manifest "objects")]
|
||||
|
||||
;; We first process all storage objects, they have
|
||||
;; deduplication so we can't rely on simple reindex. This
|
||||
;; operation populates the index for all storage objects.
|
||||
(run! (partial read-storage-object! cfg) (:storage-objects objects))
|
||||
|
||||
;; Populate index with all the incoming objects
|
||||
(vswap! bfc/*state* update :index
|
||||
(fn [index]
|
||||
(-> index
|
||||
(bfc/update-index (:teams objects))
|
||||
(bfc/update-index (:projects objects))
|
||||
(bfc/update-index (:files objects))
|
||||
(bfc/update-index (:file-media-objects objects))
|
||||
(bfc/update-index (:team-font-variants objects)))))
|
||||
|
||||
(let [team-id (read-obj cfg :manifest "team-id")
|
||||
team (read-team! cfg team-id)
|
||||
features (cfeat/get-team-enabled-features cf/flags team)
|
||||
cfg (assoc cfg ::bfc/features features)]
|
||||
|
||||
(run! (partial read-project! cfg) (:projects objects))
|
||||
(run! (partial read-file! cfg) (:files objects))
|
||||
|
||||
;; (run-pending-migrations! cfg)
|
||||
|
||||
team)))))
|
||||
(finally
|
||||
(pu/close! (::db cfg))
|
||||
|
||||
(let [elapsed (tp)]
|
||||
(l/inf :hint "end"
|
||||
:operation "import"
|
||||
:id (str id)
|
||||
:elapsed (ct/format-duration elapsed)))))))
|
||||
@ -231,32 +231,6 @@
|
||||
(def validate-file-thumbnail
|
||||
(sm/check-fn schema:file-thumbnail))
|
||||
|
||||
;; --- PROGRESS
|
||||
|
||||
(defn- tap-progress
|
||||
"Emit a progress tap of the core, with the counters of the outer scopes
|
||||
in course when the cfg has any.
|
||||
|
||||
The outer counters live in `::outer-counters` of the cfg while a scope
|
||||
is being processed (today only `:files`, tomorrow `:teams` and
|
||||
`:projects` too), so every tap inside carries them as `:outer-counters`:
|
||||
the stages inside do not repeat them, and whoever reads the taps (the
|
||||
jobs adapter) builds its report from one tap alone instead of
|
||||
remembering the last position it saw. A tap that happens outside any
|
||||
scope carries no outer counters, which is the truth about it."
|
||||
[cfg data]
|
||||
(events/tap :progress
|
||||
(cond-> data
|
||||
(some? (::outer-counters cfg))
|
||||
(assoc :outer-counters (::outer-counters cfg)))))
|
||||
|
||||
(defn- with-outer-counter
|
||||
"Return cfg with the counter of `scope` set to `position`, keeping the
|
||||
counters of the other outer scopes already in course, so entering a
|
||||
nested scope never wipes the ones above it."
|
||||
[cfg scope position]
|
||||
(assoc-in cfg [::outer-counters scope] position))
|
||||
|
||||
;; --- EXPORT IMPL
|
||||
|
||||
(defn- write-entry!
|
||||
@ -295,10 +269,8 @@
|
||||
|
||||
(defn- export-storage-objects
|
||||
[{:keys [::output] :as cfg}]
|
||||
(let [storage (sto/resolve cfg)
|
||||
ids (-> bfc/*state* deref :storage-objects not-empty)
|
||||
total (count ids)]
|
||||
(doseq [[index id] (d/enumerate ids)]
|
||||
(let [storage (sto/resolve cfg)]
|
||||
(doseq [id (-> bfc/*state* deref :storage-objects not-empty)]
|
||||
(let [sobject (sto/get-object storage id)
|
||||
smeta (meta sobject)
|
||||
ext (cmedia/mtype->extension (:content-type smeta))
|
||||
@ -310,8 +282,7 @@
|
||||
|
||||
(write-entry! output path params)
|
||||
|
||||
(tap-progress cfg {:section :storage-object :id id
|
||||
:current (inc index) :total total})
|
||||
(events/tap :progress {:section :storage-object :id id})
|
||||
|
||||
(with-open [input (sto/get-object-data storage sobject)]
|
||||
(.putNextEntry ^ZipOutputStream output (ZipEntry. (str "objects/" id ext)))
|
||||
@ -339,7 +310,7 @@
|
||||
|
||||
thumbnails (bfc/get-file-object-thumbnails cfg file-id)]
|
||||
|
||||
(tap-progress cfg {:section :file :id file-id :name (:name file)})
|
||||
(events/tap :progress {:section :file :id file-id :name (:name file)})
|
||||
|
||||
(vswap! bfc/*state* update :files assoc file-id
|
||||
{:id file-id
|
||||
@ -373,8 +344,7 @@
|
||||
|
||||
(write-entry! output path page)
|
||||
|
||||
(tap-progress cfg {:section :page :id page-id :name (:name page) :file-id file-id
|
||||
:current (inc index) :total (count pages)})
|
||||
(events/tap :progress {:section :page :id page-id :name (:name page) :file-id file-id})
|
||||
|
||||
(doseq [[shape-id shape] objects]
|
||||
(let [path (str "files/" file-id "/pages/" page-id "/" shape-id ".json")
|
||||
@ -385,33 +355,30 @@
|
||||
(vswap! bfc/*state* bfc/collect-storage-objects media)
|
||||
(vswap! bfc/*state* bfc/collect-storage-objects thumbnails)
|
||||
|
||||
(doseq [[index {:keys [id] :as item}] (d/enumerate media)]
|
||||
(let [path (str "files/" file-id "/media/" id ".json")
|
||||
encoded (encode-media item)]
|
||||
(doseq [{:keys [id] :as media} media]
|
||||
(let [path (str "files/" file-id "/media/" id ".json")
|
||||
media (encode-media media)]
|
||||
|
||||
(tap-progress cfg {:section :media :id id :file-id file-id
|
||||
:current (inc index) :total (count media)})
|
||||
(write-entry! output path encoded)))
|
||||
(events/tap :progress {:section :media :id id :file-id file-id})
|
||||
(write-entry! output path media)))
|
||||
|
||||
(doseq [[index thumbnail] (d/enumerate thumbnails)]
|
||||
(doseq [thumbnail thumbnails]
|
||||
(let [data (cth/parse-object-id (:object-id thumbnail))
|
||||
path (str "files/" file-id "/thumbnails/" (:tag data) "/" (:page-id data)
|
||||
"/" (:frame-id data) ".json")
|
||||
data (-> data
|
||||
(assoc :media-id (:media-id thumbnail))
|
||||
(encode-file-thumbnail))]
|
||||
(tap-progress cfg {:section :thumbnails :id (:object-id thumbnail) :file-id file-id
|
||||
:current (inc index) :total (count thumbnails)})
|
||||
(events/tap :progress {:section :thumbnails :id (:object-id thumbnail) :file-id file-id})
|
||||
(write-entry! output path data)))
|
||||
|
||||
(doseq [[index [id component]] (d/enumerate components)]
|
||||
(doseq [[id component] components]
|
||||
(let [path (str "files/" file-id "/components/" id ".json")
|
||||
component (encode-component component)]
|
||||
(tap-progress cfg {:section :component :id id :file-id file-id
|
||||
:current (inc index) :total (count components)})
|
||||
(events/tap :progress {:section :component :id id :file-id file-id})
|
||||
(write-entry! output path component)))
|
||||
|
||||
(doseq [[index [id color]] (d/enumerate colors)]
|
||||
(doseq [[id color] colors]
|
||||
(let [path (str "files/" file-id "/colors/" id ".json")
|
||||
color (-> (encode-color color)
|
||||
(dissoc :file-id))
|
||||
@ -419,22 +386,20 @@
|
||||
(and (contains? color :path)
|
||||
(str/empty? (:path color)))
|
||||
(dissoc :path))]
|
||||
(tap-progress cfg {:section :color :id id :file-id file-id
|
||||
:current (inc index) :total (count colors)})
|
||||
(events/tap :progress {:section :color :id id :file-id file-id})
|
||||
(write-entry! output path color)))
|
||||
|
||||
(doseq [[index [id object]] (d/enumerate typographies)]
|
||||
(doseq [[id object] typographies]
|
||||
(let [path (str "files/" file-id "/typographies/" id ".json")
|
||||
typography (encode-typography object)]
|
||||
(tap-progress cfg {:section :typography :id id :file-id file-id
|
||||
:current (inc index) :total (count typographies)})
|
||||
(events/tap :progress {:section :typography :id id :file-id file-id})
|
||||
(write-entry! output path typography)))
|
||||
|
||||
(when (and tokens-lib
|
||||
(not (ctob/empty-lib? tokens-lib)))
|
||||
(let [path (str "files/" file-id "/tokens.json")
|
||||
encoded-tokens (encode-tokens-lib tokens-lib)]
|
||||
(tap-progress cfg {:section :tokens-lib :file-id file-id})
|
||||
(events/tap :progress {:section :tokens-lib :file-id file-id})
|
||||
(write-entry! output path encoded-tokens)))
|
||||
|
||||
(when tokens-status
|
||||
@ -450,19 +415,18 @@
|
||||
encoded-status (-> (encode-tokens-status tokens-status)
|
||||
(cfo/ids->names effective-tokens-lib))]
|
||||
|
||||
(tap-progress cfg {:section :tokens-status :file-id file-id})
|
||||
(events/tap :progress {:section :tokens-status :file-id file-id})
|
||||
(write-entry! output path encoded-status)))))
|
||||
|
||||
(defn- export-files
|
||||
[{:keys [::bfc/file-ids ::bfc/export-type ::output] :as cfg}]
|
||||
[{:keys [::bfc/ids ::bfc/export-type ::output] :as cfg}]
|
||||
|
||||
(let [original-ids file-ids
|
||||
file-ids (into file-ids (when (= export-type :include-libraries)
|
||||
(bfc/get-libraries cfg file-ids)))
|
||||
rels (if (= export-type :include-libraries)
|
||||
(->> (bfc/get-files-rels cfg file-ids)
|
||||
(mapv (juxt :file-id :library-file-id)))
|
||||
[])
|
||||
(let [original-ids ids
|
||||
ids (into ids (when (= export-type :include-libraries) (bfc/get-libraries cfg ids)))
|
||||
rels (if (= export-type :include-libraries)
|
||||
(->> (bfc/get-files-rels cfg ids)
|
||||
(mapv (juxt :file-id :library-file-id)))
|
||||
[])
|
||||
|
||||
;; Compute external libraries: referenced by original files but
|
||||
;; not included in the export set. Only relevant for :link-later.
|
||||
@ -487,11 +451,10 @@
|
||||
(vswap! bfc/*state* assoc :files (d/ordered-map))
|
||||
|
||||
;; Write all the exporting files
|
||||
(doseq [[index file-id] (d/enumerate file-ids)]
|
||||
(doseq [[index file-id] (d/enumerate ids)]
|
||||
(-> cfg
|
||||
(assoc ::file-id file-id)
|
||||
(with-outer-counter :files {:current (inc index)
|
||||
:total (count file-ids)})
|
||||
(assoc ::file-seqn index)
|
||||
(export-file)))
|
||||
|
||||
;; Write manifest file
|
||||
@ -784,19 +747,17 @@
|
||||
|
||||
(defn- read-file-colors
|
||||
[{:keys [::bfc/input ::entries-index] :as cfg} file-id]
|
||||
(let [entries (get-in entries-index [:colors (str file-id)])]
|
||||
(->> (d/enumerate entries)
|
||||
(reduce (fn [result [index {:keys [id entry]}]]
|
||||
(let [object (->> (read-entry cfg input entry)
|
||||
(decode-color)
|
||||
(validate-color))]
|
||||
(tap-progress cfg {:section :color :id id :file-id file-id
|
||||
:current (inc index) :total (count entries)})
|
||||
(if (= id (:id object))
|
||||
(assoc result id object)
|
||||
result)))
|
||||
{})
|
||||
(not-empty))))
|
||||
(->> (get-in entries-index [:colors (str file-id)])
|
||||
(reduce (fn [result {:keys [id entry]}]
|
||||
(let [object (->> (read-entry cfg input entry)
|
||||
(decode-color)
|
||||
(validate-color))]
|
||||
(events/tap :progress {:section :color :id id :file-id file-id})
|
||||
(if (= id (:id object))
|
||||
(assoc result id object)
|
||||
result)))
|
||||
{})
|
||||
(not-empty)))
|
||||
|
||||
(defn- read-file-components
|
||||
[{:keys [::bfc/input ::entries-index] :as cfg} file-id]
|
||||
@ -815,18 +776,15 @@
|
||||
(reduce-kv (fn [objects id shape]
|
||||
(assoc objects id (bfl/clean-shape-pre-decode shape)))
|
||||
objects
|
||||
objects))))
|
||||
entries
|
||||
(get-in entries-index [:components (str file-id)])]
|
||||
objects))))]
|
||||
|
||||
(->> (d/enumerate entries)
|
||||
(reduce (fn [result [index {:keys [id entry]}]]
|
||||
(->> (get-in entries-index [:components (str file-id)])
|
||||
(reduce (fn [result {:keys [id entry]}]
|
||||
(let [object (->> (read-entry cfg input entry)
|
||||
(clean-component-pre-decode)
|
||||
(decode-component)
|
||||
(clean-component-post-decode))]
|
||||
(tap-progress cfg {:section :component :id id :file-id file-id
|
||||
:current (inc index) :total (count entries)})
|
||||
(events/tap :progress {:section :component :id id :file-id file-id})
|
||||
(if (= id (:id object))
|
||||
(assoc result id object)
|
||||
result)))
|
||||
@ -835,24 +793,22 @@
|
||||
|
||||
(defn- read-file-typographies
|
||||
[{:keys [::bfc/input ::entries-index] :as cfg} file-id]
|
||||
(let [entries (get-in entries-index [:typographies (str file-id)])]
|
||||
(->> (d/enumerate entries)
|
||||
(reduce (fn [result [index {:keys [id entry]}]]
|
||||
(let [object (->> (read-entry cfg input entry)
|
||||
(decode-typography)
|
||||
(validate-typography))]
|
||||
(tap-progress cfg {:section :typography :id id :file-id file-id
|
||||
:current (inc index) :total (count entries)})
|
||||
(if (= id (:id object))
|
||||
(assoc result id object)
|
||||
result)))
|
||||
{})
|
||||
(not-empty))))
|
||||
(->> (get-in entries-index [:typographies (str file-id)])
|
||||
(reduce (fn [result {:keys [id entry]}]
|
||||
(let [object (->> (read-entry cfg input entry)
|
||||
(decode-typography)
|
||||
(validate-typography))]
|
||||
(events/tap :progress {:section :typography :id id :file-id file-id})
|
||||
(if (= id (:id object))
|
||||
(assoc result id object)
|
||||
result)))
|
||||
{})
|
||||
(not-empty)))
|
||||
|
||||
(defn- read-file-tokens-lib*
|
||||
[{:keys [::bfc/input ::entries-index] :as cfg} file-id]
|
||||
(when-let [{:keys [entry]} (first (get-in entries-index [:tokens (str file-id)]))]
|
||||
(tap-progress cfg {:section :tokens-lib :file-id file-id})
|
||||
(events/tap :progress {:section :tokens-lib :file-id file-id})
|
||||
(->> (read-entry cfg input entry nil)
|
||||
(decode-tokens-lib)
|
||||
(validate-tokens-lib))))
|
||||
@ -873,7 +829,7 @@
|
||||
(defn- read-file-tokens-status
|
||||
[{:keys [::bfc/input ::entries-index] :as cfg} file-id tokens-lib]
|
||||
(when-let [{:keys [entry]} (first (get-in entries-index [:tokens-status (str file-id)]))]
|
||||
(tap-progress cfg {:section :tokens-status :file-id file-id})
|
||||
(events/tap :progress {:section :tokens-status :file-id file-id})
|
||||
(->> (read-entry cfg input entry nil)
|
||||
(cfo/names->ids tokens-lib)
|
||||
(decode-tokens-status)
|
||||
@ -895,21 +851,19 @@
|
||||
|
||||
(defn- read-file-pages
|
||||
[{:keys [::bfc/input ::entries-index] :as cfg} file-id]
|
||||
(let [entries (get-in entries-index [:pages (str file-id)])]
|
||||
(->> (d/enumerate entries)
|
||||
(keep (fn [[index {:keys [id entry]}]]
|
||||
(let [page (->> (read-entry cfg input entry)
|
||||
(decode-page))
|
||||
page (dissoc page :options)]
|
||||
(tap-progress cfg {:section :page :id id :file-id file-id
|
||||
:current (inc index) :total (count entries)})
|
||||
(when (= id (:id page))
|
||||
(let [objects (read-file-shapes cfg file-id id)]
|
||||
(assoc page :objects objects))))))
|
||||
(sort-by :index)
|
||||
(reduce (fn [result {:keys [id] :as page}]
|
||||
(assoc result id (dissoc page :index)))
|
||||
(d/ordered-map)))))
|
||||
(->> (get-in entries-index [:pages (str file-id)])
|
||||
(keep (fn [{:keys [id entry]}]
|
||||
(let [page (->> (read-entry cfg input entry)
|
||||
(decode-page))
|
||||
page (dissoc page :options)]
|
||||
(events/tap :progress {:section :page :id id :file-id file-id})
|
||||
(when (= id (:id page))
|
||||
(let [objects (read-file-shapes cfg file-id id)]
|
||||
(assoc page :objects objects))))))
|
||||
(sort-by :index)
|
||||
(reduce (fn [result {:keys [id] :as page}]
|
||||
(assoc result id (dissoc page :index)))
|
||||
(d/ordered-map))))
|
||||
|
||||
(defn- read-file-thumbnails
|
||||
[{:keys [::bfc/input ::entries-index] :as cfg} file-id]
|
||||
@ -950,11 +904,8 @@
|
||||
:plugin-data plugin-data}))
|
||||
|
||||
(defn- import-file
|
||||
[{:keys [::db/conn ::bfc/project-id ::manifest] :as cfg}
|
||||
{file-id :id file-name :name}
|
||||
position]
|
||||
(let [cfg (with-outer-counter cfg :files position)
|
||||
file-id' (bfc/lookup-index file-id)
|
||||
[{:keys [::db/conn ::bfc/project-id ::manifest] :as cfg} {file-id :id file-name :name}]
|
||||
(let [file-id' (bfc/lookup-index file-id)
|
||||
file (read-file cfg file-id)
|
||||
media (read-file-media cfg file-id)
|
||||
thumbnails (read-file-thumbnails cfg file-id)]
|
||||
@ -966,19 +917,16 @@
|
||||
:version (:version file)
|
||||
::l/sync? true)
|
||||
|
||||
(tap-progress cfg {:section :file :file-id file-id})
|
||||
|
||||
(vswap! bfc/*state* update :index bfc/update-index media :id)
|
||||
|
||||
(doseq [[index item] (d/enumerate media)]
|
||||
(doseq [item media]
|
||||
(let [params (-> item
|
||||
(update :id bfc/lookup-index)
|
||||
(assoc :file-id file-id')
|
||||
(d/update-when :media-id bfc/lookup-index)
|
||||
(d/update-when :thumbnail-id bfc/lookup-index))]
|
||||
|
||||
(tap-progress cfg {:section :media :id (:id params) :file-id file-id
|
||||
:current (inc index) :total (count media)})
|
||||
(events/tap :progress {:section :media :id (:id params) :file-id file-id})
|
||||
|
||||
(l/dbg :hint "inserting media object"
|
||||
:file-id (str file-id')
|
||||
@ -988,9 +936,10 @@
|
||||
:old-id (str (:id item))
|
||||
::l/sync? true)
|
||||
|
||||
(db/insert! conn :file-media-object params)))
|
||||
(db/insert! conn :file-media-object params
|
||||
::db/on-conflict-do-nothing? (::bfc/overwrite cfg))))
|
||||
|
||||
(doseq [[index item] (d/enumerate thumbnails)]
|
||||
(doseq [item thumbnails]
|
||||
(let [media-id (bfc/lookup-index (:media-id item))
|
||||
object-id (-> (assoc item :file-id file-id')
|
||||
(cth/fmt-object-id))
|
||||
@ -1004,12 +953,13 @@
|
||||
:media-id (str media-id)
|
||||
::l/sync? true)
|
||||
|
||||
(tap-progress cfg {:section :thumbnail :file-id file-id :object-id object-id
|
||||
:current (inc index) :total (count thumbnails)})
|
||||
(events/tap :progress {:section :thumbnail :file-id file-id :object-id object-id})
|
||||
|
||||
(db/insert! conn :file-tagged-object-thumbnail params
|
||||
::db/on-conflict-do-nothing? true)))
|
||||
|
||||
(events/tap :progress {:section :file :file-id file-id})
|
||||
|
||||
(let [data (-> (read-file-data cfg file-id (:tokens-source file))
|
||||
(d/without-nils)
|
||||
(assoc :id file-id')
|
||||
@ -1037,55 +987,27 @@
|
||||
file-id')))
|
||||
|
||||
(defn- import-file-relations
|
||||
"Import library relations from the manifest, ignoring relations whose
|
||||
endpoints are not both part of the package. Of the rest, skip edges that
|
||||
would close a dependency cycle or link a file to itself (same check as
|
||||
`link-file-to-library`, see `bfc/library-link-status`). Relations are
|
||||
processed in sorted manifest order and the first edge wins: an edge
|
||||
that would close a cycle with previously imported relations is skipped
|
||||
and logged."
|
||||
[{:keys [::db/conn ::manifest ::bfc/timestamp] :as cfg}]
|
||||
(let [relations (sort-by (fn [[file-id libr-id]] [(str file-id) (str libr-id)])
|
||||
(:relations manifest))
|
||||
file-ids (into #{} (map :id) (:files manifest))]
|
||||
(doseq [[index [file-id libr-id]] (d/enumerate relations)]
|
||||
(tap-progress cfg {:section :relations
|
||||
:current (inc index)
|
||||
:total (count relations)})
|
||||
(events/tap :progress {:section :relations})
|
||||
(doseq [[file-id libr-id] (:relations manifest)]
|
||||
|
||||
(if (and (contains? file-ids file-id)
|
||||
(contains? file-ids libr-id))
|
||||
(let [file-id (bfc/lookup-index file-id)
|
||||
libr-id (bfc/lookup-index libr-id)]
|
||||
(case (bfc/library-link-status cfg file-id libr-id)
|
||||
:self-link
|
||||
(l/warn :hint "ignoring self-referencing library link"
|
||||
:file-id (str file-id)
|
||||
:lib-id (str libr-id)
|
||||
::l/sync? true)
|
||||
(let [file-id (bfc/lookup-index file-id)
|
||||
libr-id (bfc/lookup-index libr-id)]
|
||||
|
||||
:circular
|
||||
(l/warn :hint "ignoring circular library link"
|
||||
:file-id (str file-id)
|
||||
:lib-id (str libr-id)
|
||||
::l/sync? true)
|
||||
|
||||
:ok
|
||||
(do
|
||||
(l/dbg :hint "create file library link"
|
||||
:file-id (str file-id)
|
||||
:lib-id (str libr-id)
|
||||
::l/sync? true)
|
||||
(let [rel-params {:file-id file-id
|
||||
:library-file-id libr-id}]
|
||||
(db/insert! conn :file-library-rel rel-params)
|
||||
(bfc/upsert-file-library-sync! conn (assoc rel-params :synced-at timestamp))))))
|
||||
(l/warn :hint "ignoring invalid file library relation"
|
||||
:file-id (str file-id)
|
||||
:lib-id (str libr-id))))))
|
||||
(when (and file-id libr-id)
|
||||
(l/dbg :hint "create file library link"
|
||||
:file-id (str file-id)
|
||||
:lib-id (str libr-id)
|
||||
::l/sync? true)
|
||||
(let [rel-params {:file-id file-id
|
||||
:library-file-id libr-id}]
|
||||
(db/insert! conn :file-library-rel rel-params)
|
||||
(bfc/upsert-file-library-sync! conn (assoc rel-params :synced-at timestamp)))))))
|
||||
|
||||
(defn- import-storage-objects
|
||||
[{:keys [::bfc/input ::entries-index ::bfc/timestamp] :as cfg}]
|
||||
(events/tap :progress {:section :storage-objects})
|
||||
|
||||
;; IMPORTANT: we strongly do not reuse the main connection that can
|
||||
;; run inside a transaction because the storage upload process can
|
||||
;; fail in the middle of uploading and leave garbage on the underlying
|
||||
@ -1094,12 +1016,9 @@
|
||||
;; what the storage subsystem registers in other parallel
|
||||
;; transaction
|
||||
(let [storage (sto/resolve cfg)
|
||||
entries (:objects entries-index)
|
||||
total (count entries)]
|
||||
entries (:objects entries-index)]
|
||||
|
||||
(doseq [[index {:keys [id entry]}] (d/enumerate entries)]
|
||||
(tap-progress cfg {:section :storage-objects
|
||||
:current (inc index) :total total})
|
||||
(doseq [{:keys [id entry]} entries]
|
||||
(let [object (-> (read-entry cfg input entry)
|
||||
(decode-storage-object)
|
||||
(d/update-when :content-type bfc/normalize-content-type)
|
||||
@ -1281,15 +1200,12 @@
|
||||
(vswap! bfc/*state* update :index assoc old-lib-id library-id)))
|
||||
|
||||
(let [files (get manifest :files)
|
||||
total (count files)
|
||||
file-ids (reduce (fn [result [index file]]
|
||||
file-ids (reduce (fn [result file]
|
||||
(let [name' (get file :name)
|
||||
file (assoc file :name name')]
|
||||
(conj result (import-file cfg file
|
||||
{:current (inc index)
|
||||
:total total}))))
|
||||
file (assoc file :name name')]
|
||||
(conj result (import-file cfg file))))
|
||||
[]
|
||||
(d/enumerate files))
|
||||
files)
|
||||
;; Build map of file-id to file-name for resolution
|
||||
files-info (into {} (map (fn [file-id manifest-file]
|
||||
[file-id (:name manifest-file)])
|
||||
@ -1304,6 +1220,56 @@
|
||||
{:file-ids file-ids
|
||||
:resolution resolution})))
|
||||
|
||||
(defn- invalidate-thumbnails
|
||||
[cfg file-id]
|
||||
(let [storage (sto/resolve cfg ::db/reuse-conn true)
|
||||
|
||||
sql-1
|
||||
(str "update file_tagged_object_thumbnail "
|
||||
" set deleted_at = now() "
|
||||
" where file_id=? returning media_id")
|
||||
|
||||
sql-2
|
||||
(str "update file_thumbnail "
|
||||
" set deleted_at = now() "
|
||||
" where file_id=? returning media_id")]
|
||||
|
||||
(run! #(sto/touch-object! storage %)
|
||||
(sequence
|
||||
(keep :media-id)
|
||||
(concat
|
||||
(db/exec! cfg [sql-1 file-id])
|
||||
(db/exec! cfg [sql-2 file-id]))))))
|
||||
|
||||
(defn- import-file-and-overwrite*
|
||||
[{:keys [::manifest ::bfc/file-id] :as cfg}]
|
||||
|
||||
(when (not= 1 (count (:files manifest)))
|
||||
(ex/raise :type :validation
|
||||
:code :invalid-condition
|
||||
:hint "unable to perform in-place update with binfile containing more than 1 file"
|
||||
:manifest manifest))
|
||||
|
||||
(bfc/configure-import-transaction cfg)
|
||||
|
||||
(let [ref-file (bfc/get-minimal-file cfg file-id ::db/for-update true)
|
||||
file (first (get manifest :files))
|
||||
cfg (assoc cfg ::bfc/overwrite true)]
|
||||
|
||||
(vswap! bfc/*state* update :index assoc (:id file) file-id)
|
||||
|
||||
(binding [bfc/*options* cfg
|
||||
bfc/*reference-file* ref-file]
|
||||
|
||||
(import-storage-objects cfg)
|
||||
(import-file cfg file)
|
||||
|
||||
(invalidate-thumbnails cfg file-id)
|
||||
(bfm/apply-pending-migrations! cfg)
|
||||
|
||||
{:file-ids [file-id]
|
||||
:resolution {}})))
|
||||
|
||||
(defn- import-files
|
||||
[{:keys [::bfc/timestamp ::bfc/input] :or {timestamp (ct/now)} :as cfg}]
|
||||
|
||||
@ -1351,34 +1317,35 @@
|
||||
|
||||
(cfeat/check-supported-features! features)))
|
||||
|
||||
(tap-progress cfg {:section :manifest})
|
||||
(events/tap :progress {:section :manifest})
|
||||
|
||||
(binding [bfc/*state* (volatile! {:media [] :index {}})]
|
||||
(db/tx-run! cfg import-files*))))
|
||||
(if (::bfc/file-id cfg)
|
||||
(db/tx-run! cfg import-file-and-overwrite*)
|
||||
(db/tx-run! cfg import-files*)))))
|
||||
|
||||
;; --- PUBLIC API
|
||||
|
||||
(defn export-files!
|
||||
"Do the exportation of a specified file in custom penpot binary
|
||||
format. The cfg carries what `bfc/export-cfg` froze: `::bfc/file-ids`
|
||||
(the files to export), `::bfc/export-type` (how linked libraries are
|
||||
handled) and `::bfc/output` (where the package is written).
|
||||
format. There are some options available for customize the output:
|
||||
|
||||
`::bfc/export-type` determines how linked libraries are handled.
|
||||
Valid values: `:include-libraries` (include linked libraries),
|
||||
`::bfc/export-type`: determines how linked libraries are handled.
|
||||
Valid values:
|
||||
`:include-libraries` (include linked libraries),
|
||||
`:merge-libraries` (embed library assets in the file),
|
||||
`:detach-libraries` (treat assets as basic objects),
|
||||
`:link-later` (preserve component metadata for relinking on import)."
|
||||
|
||||
[{:keys [::bfc/file-ids ::bfc/output] :as cfg}]
|
||||
[{:keys [::bfc/ids] :as cfg} output]
|
||||
|
||||
(assert
|
||||
(and (set? file-ids) (every? uuid? file-ids))
|
||||
"expected a set of uuid's for `::bfc/file-ids` parameter")
|
||||
(and (set? ids) (every? uuid? ids))
|
||||
"expected a set of uuid's for `::bfc/ids` parameter")
|
||||
|
||||
(assert
|
||||
(satisfies? jio/IOFactory output)
|
||||
"expected instance of jio/IOFactory for `::bfc/output`")
|
||||
"expected instance of jio/IOFactory for `input`")
|
||||
|
||||
(let [id (uuid/next)
|
||||
tp (ct/tpoint)
|
||||
@ -1404,11 +1371,6 @@
|
||||
(vreset! ab true)
|
||||
nil)
|
||||
|
||||
(catch clojure.lang.ExceptionInfo cause
|
||||
(if (= :interrupt (:type (ex-data cause)))
|
||||
(vreset! ab true)
|
||||
(throw cause)))
|
||||
|
||||
(catch Throwable cause
|
||||
(vreset! cs cause)
|
||||
(vreset! ab true)
|
||||
@ -1423,12 +1385,6 @@
|
||||
(defn import-files!
|
||||
[{:keys [::bfc/input] :as cfg}]
|
||||
|
||||
(when (::bfc/file-id cfg)
|
||||
(ex/raise :type :validation
|
||||
:code :not-supported
|
||||
:hint "in-place import is not supported"
|
||||
:file-id (::bfc/file-id cfg)))
|
||||
|
||||
(assert
|
||||
(and (uuid? (::bfc/profile-id cfg))
|
||||
(uuid? (::bfc/project-id cfg)))
|
||||
|
||||
@ -10,7 +10,6 @@
|
||||
[app.common.data :as d]
|
||||
[app.common.exceptions :as ex]
|
||||
[app.common.flags :as flags]
|
||||
[app.common.logging :as l]
|
||||
[app.common.schema :as sm]
|
||||
[app.common.time :as ct]
|
||||
[app.common.uri :as u]
|
||||
@ -78,11 +77,6 @@
|
||||
|
||||
:telemetry-uri "https://telemetry.penpot.app/"
|
||||
|
||||
:jobs-lease (ct/duration {:minutes 30})
|
||||
:jobs-retention (ct/duration {:days 7})
|
||||
:jobs-user-ttl (ct/duration {:days 7})
|
||||
:jobs-request-timeout (ct/duration {:minutes 2})
|
||||
|
||||
:media-max-file-size (* 1024 1024 30) ; 30MiB
|
||||
:font-max-file-size (* 1024 1024 30) ; 30MiB
|
||||
|
||||
@ -103,44 +97,12 @@
|
||||
|
||||
:quotes-upload-sessions-per-profile 5
|
||||
:quotes-upload-chunks-per-session 20
|
||||
:quotes-export-jobs-per-profile 10
|
||||
:quotes-import-jobs-per-profile 10
|
||||
:upload-max-chunk-size (* 1024 1024 30) ; 30MiB
|
||||
|
||||
;; SSRF protection
|
||||
:ssrf-allowed-hosts #{}
|
||||
:ssrf-extra-blocked-cidrs #{}})
|
||||
|
||||
(def schema:tenant
|
||||
"Tenant identifier: hostname-label-style, lowercase (letters, digits
|
||||
and hyphens). The value is normalized to lowercase when the
|
||||
configuration is read, so an operator writing `PENPOT_TENANT=ACME` and
|
||||
one writing `acme` name the same tenant: the value is compared as an
|
||||
exact string in the keys and rows it lands in, and one uppercase
|
||||
letter would otherwise split one tenant into two.
|
||||
|
||||
It is interpolated into Redis keys separated by dots (msgbus
|
||||
topics, rate-limit buckets, exporter job keys, which are also matched
|
||||
with a glob pattern), so `.` and whitespace are rejected at startup
|
||||
because one tenant would then be able to read or overwrite another's
|
||||
keys.
|
||||
|
||||
`%`, `_` and `:` are rejected too, and the reason they were added is
|
||||
gone: the job queue used to be filtered with a LIKE over a
|
||||
`<tenant>:<queue>` prefix and now has a `tenant` column. They are kept
|
||||
because narrowing the rule is a startup contract change with no real
|
||||
tenant asking for it, and a tenant that has one would have to be
|
||||
renamed. Tenants using those characters must be renamed before
|
||||
upgrading."
|
||||
[:re #"^[a-z0-9-]+$"])
|
||||
|
||||
(defn- normalize-tenant
|
||||
"Lowercase the configured tenant before it is validated, so
|
||||
`PENPOT_TENANT=ACME` and `acme` mean the same tenant everywhere."
|
||||
[config]
|
||||
(cond-> config
|
||||
(string? (:tenant config)) (update :tenant str/lower)))
|
||||
|
||||
(def schema:config
|
||||
(do #_sm/optional-keys
|
||||
[:map {:title "config"}
|
||||
@ -148,7 +110,7 @@
|
||||
[:admins {:optional true} [::sm/set ::sm/email]]
|
||||
[:secret-key {:optional true} :string]
|
||||
|
||||
[:tenant {:optional false} schema:tenant]
|
||||
[:tenant {:optional false} :string]
|
||||
[:is-saas ::sm/boolean]
|
||||
[:public-uri {:optional false} ::sm/uri]
|
||||
[:host {:optional false} :string]
|
||||
@ -159,10 +121,6 @@
|
||||
[:http-server-io-threads {:optional true} ::sm/int]
|
||||
[:http-server-max-worker-threads {:optional true} ::sm/int]
|
||||
|
||||
;; How often a websocket subscription re-checks that its access still
|
||||
;; holds. The default lives in `app.http.websocket`, not here.
|
||||
[:subscription-revalidation-interval {:optional true} ::ct/duration]
|
||||
|
||||
;; Explicit CORS allowlist used when the :cors flag is enabled.
|
||||
;; Configured via PENPOT_ALLOWED_ORIGINS as a comma/whitespace
|
||||
;; separated list of origins (e.g. "https://plugins.example.com").
|
||||
@ -215,29 +173,22 @@
|
||||
|
||||
[:deletion-delay {:optional true} ::ct/duration]
|
||||
[:file-clean-delay {:optional true} ::ct/duration]
|
||||
[:jobs-lease {:optional true} ::ct/duration]
|
||||
[:jobs-retention {:optional true} ::ct/duration]
|
||||
[:jobs-user-ttl {:optional true} ::ct/duration]
|
||||
[:jobs-request-timeout {:optional true} ::ct/duration]
|
||||
[:telemetry-enabled {:optional true} ::sm/boolean]
|
||||
[:default-blob-version {:optional true} ::sm/int]
|
||||
[:allow-demo-users {:optional true} ::sm/boolean]
|
||||
[:error-report-webhook {:optional true} :string]
|
||||
[:user-feedback-destination {:optional true} :string]
|
||||
[:mcp-feedback-email {:optional true} ::sm/email]
|
||||
|
||||
[:default-rpc-rlimit {:optional true} [::sm/vec :string]]
|
||||
[:rpc-rlimit-config {:optional true} ::fs/path]
|
||||
[:rpc-climit-config {:optional true} ::fs/path]
|
||||
|
||||
[:nexus-uri {:optional true} ::sm/uri]
|
||||
[:audit-log-archive-uri {:optional true} :string]
|
||||
[:audit-log-http-handler-concurrency {:optional true} ::sm/int]
|
||||
|
||||
[:default-executor-parallelism {:optional true} ::sm/int] ;; REVIEW
|
||||
[:worker-default-parallelism {:optional true} ::sm/int]
|
||||
[:worker-webhook-parallelism {:optional true} ::sm/int]
|
||||
[:worker-cron-parallelism {:optional true} ::sm/int]
|
||||
[:worker-binfile-parallelism {:optional true} ::sm/int]
|
||||
|
||||
[:database-password {:optional true} [:maybe :string]]
|
||||
[:database-uri {:optional true} ::sm/uri]
|
||||
@ -245,12 +196,6 @@
|
||||
[:database-readonly {:optional true} ::sm/boolean]
|
||||
[:database-min-pool-size {:optional true} ::sm/int]
|
||||
[:database-max-pool-size {:optional true} ::sm/int]
|
||||
;; pgjdbc prepared statement cache (see app.db). Size is in bytes and is
|
||||
;; converted to the whole MiB the driver accepts. Absent keys mean the
|
||||
;; driver defaults; see docs/technical-guide/configuration.md.
|
||||
[:database-prepared-statement-cache-threshold {:optional true} [::sm/int {:min -1}]]
|
||||
[:database-prepared-statement-cache-queries {:optional true} [::sm/int {:min 0}]]
|
||||
[:database-prepared-statement-cache-size {:optional true} [::sm/int {:min 0}]]
|
||||
|
||||
[:quotes-teams-per-profile {:optional true} ::sm/int]
|
||||
[:quotes-access-tokens-per-profile {:optional true} ::sm/int]
|
||||
@ -262,8 +207,6 @@
|
||||
[:quotes-font-variants-per-team {:optional true} ::sm/int]
|
||||
[:quotes-comment-threads-per-file {:optional true} ::sm/int]
|
||||
[:quotes-comments-per-file {:optional true} ::sm/int]
|
||||
[:quotes-export-jobs-per-profile {:optional true} ::sm/int]
|
||||
[:quotes-import-jobs-per-profile {:optional true} ::sm/int]
|
||||
[:quotes-snapshots-per-file {:optional true} ::sm/int]
|
||||
[:quotes-snapshots-per-team {:optional true} ::sm/int]
|
||||
[:quotes-team-access-requests-per-team {:optional true} ::sm/int]
|
||||
@ -419,31 +362,13 @@
|
||||
(def explain-config
|
||||
(sm/explainer schema:config))
|
||||
|
||||
(defn- normalize-file-data-backend
|
||||
"The file data moved to `file_data` (migration `0157-migrate-legacy-file-data`), so
|
||||
`file.data` and `file_change.data` are read-only leftovers and the `legacy-db`
|
||||
back end is read-only: a write against it fails on every save. Accepting it would
|
||||
start a healthy instance that cannot save a single file, so it reads as `db`,
|
||||
which serves the legacy rows too. Warn instead of aborting: the operator may
|
||||
still be halfway through the migration."
|
||||
[config]
|
||||
(if (= "legacy-db" (c/get config :file-data-backend))
|
||||
(do
|
||||
(l/warn :hint "file-data-backend=legacy-db is read-only, using db"
|
||||
:file-data-backend "legacy-db"
|
||||
:replacement "db")
|
||||
(assoc config :file-data-backend "db"))
|
||||
config))
|
||||
|
||||
(defn read-config
|
||||
"Reads the configuration from environment variables and decodes all
|
||||
known values."
|
||||
[& {:keys [prefix default] :or {prefix "penpot"}}]
|
||||
(->> (read-env prefix)
|
||||
(merge default)
|
||||
(normalize-tenant)
|
||||
(decode-config)
|
||||
(normalize-file-data-backend)))
|
||||
(decode-config)))
|
||||
|
||||
(def version
|
||||
(v/parse (or (some-> (io/resource "version.txt")
|
||||
@ -497,41 +422,6 @@
|
||||
[& segments]
|
||||
(apply join-uri (c/get config :public-uri) segments))
|
||||
|
||||
(defn get-jobs-lease
|
||||
"Max time a job can run without touching modified_at (heartbeat or
|
||||
progress) before the dispatcher marks it as `aborted` (system-side
|
||||
terminal, never retried, reported with an error log)."
|
||||
[]
|
||||
(or (c/get config :jobs-lease)
|
||||
(ct/duration {:minutes 30})))
|
||||
|
||||
(defn get-jobs-request-timeout
|
||||
"Default timeout for the ephemeral request! calls (waiting for the
|
||||
reply-key blpop); can be overridden per call. Any override is applied
|
||||
by raising the pooled connection command timeout for the duration of
|
||||
the call, which the pool restores on return."
|
||||
[]
|
||||
(or (c/get config :jobs-request-timeout)
|
||||
(ct/duration {:minutes 2})))
|
||||
|
||||
(defn get-jobs-retention
|
||||
"How long terminal (completed/failed/cancelled/aborted) internal job rows are
|
||||
kept before the jobs GC deletes them. The legacy `task` table is not
|
||||
touched by the jobs GC: while both versions run in parallel it is
|
||||
cleaned by the legacy `tasks-gc` of that version."
|
||||
[]
|
||||
(or (c/get config :jobs-retention)
|
||||
(ct/duration {:days 7})))
|
||||
|
||||
(defn get-jobs-user-ttl
|
||||
"How long a user-facing job, and the artifact it owns, is kept before
|
||||
the jobs GC deletes the row and marks the object for the storage GC.
|
||||
Not the same as `:jobs-retention`, which only sweeps terminal internal
|
||||
rows."
|
||||
[]
|
||||
(or (c/get config :jobs-user-ttl)
|
||||
(ct/duration {:days 7})))
|
||||
|
||||
(defn get
|
||||
"A configuration getter. Helps code be more testable."
|
||||
([key]
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
x
Reference in New Issue
Block a user