diff --git a/.agents/skills/code-review-criteria/SKILL.md b/.agents/skills/code-review-criteria/SKILL.md
index 398aa440ee..b95c563a57 100644
--- a/.agents/skills/code-review-criteria/SKILL.md
+++ b/.agents/skills/code-review-criteria/SKILL.md
@@ -13,19 +13,17 @@ Multi-dimensional code review with quality gates. Every change gets reviewed bef
## When to Use
-- The reviewer subagent of the `review-code` flow loads this skill to perform
- the review of a code change.
-- To review code, always go through the `review-code` flow — never load this
- skill directly for that. This is the criteria reference, not the flow.
+- The reviewer subagent of the `review-code` flow loads this skill to perform the review of a code change.
+- To review code, always go through the `review-code` flow — never load this skill directly for that. This is the criteria reference, not the flow.
## Core Principles
These principles underpin every axis. When in doubt, default to them.
-- **DRY (Don't Repeat Yourself):** Every piece of knowledge has one authoritative representation. If the same logic appears in two places, extract it into a shared helper, model, or type. Reviewers: flag duplicated logic as a required change — it's not "just similar," it's drift that will diverge.
-- **KISS (Keep It Simple, Stupid):** The simplest solution that works is the best solution. Complexity must earn its place. Reviewers: if you need more than one sentence to explain what a piece of code does, it's too complex — push for simplification before merge.
-- **YAGNI (You Aren't Gonna Need It):** Don't add abstractions, hooks, or generalizations for hypothetical future use cases. Generalize on the third occurrence, not the first. Reviewers: delete speculative generality.
-- **Don't invent problems:** Do not manufacture issues to produce more feedback. Every finding must be a real risk, a real readability barrier, or a real architectural concern — not a hypothetical or a stylistic preference disguised as a problem.
+- **DRY (Don't Repeat Yourself):** Every piece of knowledge has one authoritative representation. Flag duplicated logic as a required change — it is not "just similar", it is drift that will diverge.
+- **KISS (Keep It Simple, Stupid):** The simplest solution that works is the best solution. Complexity must earn its place: if it needs more than a sentence to explain, push for simplification before merge.
+- **YAGNI (You Aren't Gonna Need It):** No abstractions, hooks or generalizations for hypothetical future use cases. Generalize on the third occurrence, not the first.
+- **Don't invent problems:** Every finding must be a real risk, a real readability barrier, or a real architectural concern — not a hypothetical or a stylistic preference disguised as a problem.
## The Five-Axis Review
@@ -38,7 +36,7 @@ Does the code do what it claims to do?
- Does it match the spec or task requirements?
- Are edge cases handled (null, empty, boundary values)?
- Are error paths handled (not just the happy path)?
-- Does it pass all tests? Are the tests actually testing the right things?
+- Do the tests cover the changed behavior, including its edge cases?
- Are there off-by-one errors, race conditions, or state inconsistencies?
### 2. Readability & Simplicity
@@ -47,10 +45,7 @@ Can another engineer (or agent) understand this code without the author explaini
- Are names descriptive and consistent with project conventions? (No `temp`, `data`, `result` without context)
- Is the control flow straightforward (avoid nested ternaries, deep callbacks)?
-- Are there any "clever" tricks that should be simplified?
-- **KISS check:** Is this the simplest approach that solves the problem? A 20-line straightforward function beats a 5-line clever one that requires a comment to explain.
-- Could this be done in fewer lines? (1000 lines where 100 suffice is a failure)
-- Are abstractions earning their complexity? (Don't generalize until the third use case)
+- Are there "clever" tricks that should be simplified? Could this be done in fewer lines? (1000 lines where 100 suffice is a failure)
- Is a new conditional bolted onto an unrelated flow? Push the logic into its own helper, state, or policy.
- Do repeated conditionals on the same shape appear? They signal a missing model or dispatcher.
- Are there dead code artifacts: no-op variables, backwards-compat shims, or `// removed` comments?
@@ -60,14 +55,13 @@ Can another engineer (or agent) understand this code without the author explaini
Does the change fit the system's design?
- Does it follow existing patterns or introduce a new one? If new, is it justified?
-- Does it maintain clean module boundaries?
-- **DRY check:** Is there existing code that does the same thing? Reuse the canonical helper instead of writing a near-duplicate. If two branches do nearly the same thing, collapse them.
-- Are dependencies flowing in the right direction (no circular dependencies)?
+- Does it keep clean module boundaries, with dependencies flowing the right way (no cycles)?
- Is the abstraction level appropriate (not over-engineered, not too coupled)?
-- Does this refactor reduce complexity or just relocate it? Count the concepts a reader must hold. Prefer the restructuring that makes whole branches disappear over one that re-centralizes the same logic. Prefer deleting an abstraction to polishing it.
+- Is there existing code that already does this? Reuse the canonical helper instead of a near-duplicate, and collapse branches that do nearly the same thing.
- Is feature-specific logic leaking into a shared or general-purpose module?
- Are type boundaries explicit? Question gratuitous `any`/`unknown`/optional/casts and silent fallbacks.
-- **Structural remedies:** When you flag a problem, propose the move — not just the problem. Replace conditionals with dispatchers, collapse duplicate branches, separate orchestration from business logic, extract helpers, split large files. Prefer the remedy that removes moving pieces over one that spreads the same complexity around.
+- Does the change reduce complexity or relocate it? Count the concepts a reader must hold: prefer the restructuring that makes whole branches disappear, and deleting an abstraction over polishing it.
+- **Give the remedy, not only the problem:** replace conditionals with dispatchers, separate orchestration from business logic, extract helpers, split large files. Prefer the remedy that removes moving pieces over one that spreads the same complexity around.
### 4. Security
@@ -111,7 +105,17 @@ For each finding, describe the circumstances under which it could fail: specific
Lead with what matters: correctness and security first, then structural issues, then everything else. A few high-conviction comments beat a long list.
-5. **Verify the verification** — What tests were run? Did the build pass? Was the change tested manually? Screenshots for UI changes?
+5. **Check the evidence, do not re-run it** — Which tests cover the change? What did the author run, and what did CI report? Never re-run the full suite or a build yourself: the author's run and CI already cover that, and repeating it only lengthens the review. Run at most the single test file that settles a specific doubt. When no checks have been run at all (a change you did not see tested), run the module's checks before the verdict.
+
+## Change Sizing
+
+Small, focused changes are easier to review, faster to merge, and safer to deploy. Flag the size when a diff exceeds ~300 lines, or when it grows a file past ~1000 total lines, and name the remedy: split it, and separate refactoring from feature work. This is a finding about the change, not work to do inside the review.
+
+## Dependencies
+
+Before accepting a new dependency: does the existing stack already solve it, how large is the bundle impact, is it maintained, does `npm audit` report anything, is the license compatible. Prefer the standard library and existing utilities — every dependency is a liability.
+
+For upgrades: read the changelog, not just the version number; one dependency per change; a green suite before and after; review the lockfile diff and never hand-edit it. Supply-chain triage follows `security-and-hardening`.
## Review Output
@@ -151,96 +155,18 @@ Choose one:
List the finding IDs the verdict depends on (e.g. "Request changes: R1, R2").
-## Change Sizing
-
-Small, focused changes are easier to review, faster to merge, and safer to deploy.
-
-```
-~100 lines changed → Good. Reviewable in one sitting.
-~300 lines changed → Acceptable if it's a single logical change.
-~1000 lines changed → Too large. Split it.
-```
-
-**Watch file size, not just diff size.** Around 1000 *total* lines in a single file is a common inspection signal. When a change materially grows an already-large file, decompose first.
-
-**Splitting strategies:**
-
-| Strategy | How | When |
-|----------|-----|------|
-| **Stack** | Submit a small change, start the next one based on it | Sequential dependencies |
-| **By file group** | Separate changes for groups needing different reviewers | Cross-cutting concerns |
-| **Horizontal** | Create shared code/stubs first, then consumers | Layered architecture |
-| **Vertical** | Break into smaller full-stack slices of the feature | Feature work |
-
-**Separate refactoring from feature work.** A change that refactors and adds new behavior is two changes — submit them separately.
-
-## Change Descriptions
-
-- **First line:** Short, imperative, standalone. "Delete the FizzBuzz RPC" not "Deleting the FizzBuzz RPC."
-- **Body:** What is changing and why. Include context and reasoning not visible in the code itself.
-- **Anti-patterns:** "Fix bug," "Fix build," "Add patch," "Phase 1."
-
-## Dependencies
-
-Before adding any dependency:
-
-1. Does the existing stack solve this? (Often it does.)
-2. How large is the dependency? (Check bundle impact.)
-3. Is it actively maintained? (Check last commit, open issues.)
-4. Does it have known vulnerabilities? (`npm audit`)
-5. What's the license? (Must be compatible with the project.)
-
-**Rule:** Prefer standard library and existing utilities over new dependencies. Every dependency is a liability.
-
-**Upgrading dependencies:**
-
-- Read the changelog, not just the version number. Semver is a promise the maintainer may not have kept.
-- One dependency per change. When a bulk bump breaks the build, you've lost which package did it.
-- Let the tests decide — a green suite before *and* after, not just "it installed."
-- Review the lockfile diff, not just `package.json`. Commit it and never hand-edit it.
-
-For supply-chain risk triage, follow the `security-and-hardening` skill.
-
-## Common Rationalizations
-
-| Rationalization | Reality |
-|---|---|
-| "It works, that's good enough" | Working code that's unreadable, insecure, or architecturally wrong creates debt that compounds. |
-| "I wrote it, so I know it's correct" | Authors are blind to their own assumptions. Every change benefits from another set of eyes. |
-| "We'll clean it up later" | Later never comes. The review is the quality gate — use it. |
-| "AI-generated code is probably fine" | AI code needs more scrutiny, not less. It's confident and plausible, even when wrong. |
-| "The tests pass, so it's good" | Tests are necessary but not sufficient. They don't catch architecture, security, or readability problems. |
-| "The refactor makes it cleaner" | Relocating complexity isn't reducing it. If the reader still holds the same number of concepts, the structure didn't improve. |
-| "It's only a small addition to this file" | Small diffs still push files past healthy size and bolt branches onto unrelated flows. |
-| "It's just a version bump" | A bump is a behavior change you didn't write. Read the changelog. |
-| "I'll upgrade everything in one PR" | A bulk bump hides which package broke the build. One per change. |
-| "It's duplicated but it's only two places" | Two becomes three becomes five. Extract now, before the copies diverge. |
-| "The abstraction is future-proof" | YAGNI. Delete speculative generality — generalize on the third occurrence, not the first. |
-| "It's clever but efficient" | Cleverness is a readability tax. If it needs a comment to understand, simplify it. |
-
## Red Flags
-- PRs merged without any review
-- Review that only checks if tests pass (ignoring other axes)
-- "LGTM" without evidence of actual review
-- Security-sensitive changes without security-focused review
-- Large PRs that are "too big to review properly" (split them)
-- No regression tests with bug fix PRs
-- Accepting "I'll fix it later" — it never happens
-- A refactor that moves code around without reducing the number of concepts a reader must hold
-- New conditionals scattered into unrelated code paths (a missing abstraction)
-- A bespoke helper that duplicates an existing canonical one
-- A bulk "bump dependencies" PR with no changelog review
+- A change merged without any review.
+- A review that only checks whether the tests pass.
+- Security-sensitive changes without a security-focused review.
+- A change too large to review properly: split it.
## Verification
-Before emitting the verdict, verify the change as it stands. This is the reviewer's own due diligence — it covers the state of the code at review time, not the later resolution of findings (fixing findings is the author's job; confirming them is a new review):
+Before emitting the verdict, confirm the evidence the review rests on. This covers the state of the code at review time, not the later fixing of findings:
-- [ ] Tests pass — run them yourself, don't trust the claim
-- [ ] Build succeeds
-- [ ] The verification story is documented (what changed, how it was verified)
-- [ ] Dependency upgrades reviewed against changelog, isolated per package, verified by green suite
-
-## See Also
-
-- For detailed security review guidance, see `security-and-hardening`
+- [ ] The suite is green, and the evidence is the author's run or CI — not your own re-run. Say which one you relied on.
+- [ ] New behavior has tests that cover the edge cases in the diff.
+- [ ] The verification story is stated: what changed and how it was verified.
+- [ ] Dependency changes were reviewed against the changelog and the lockfile diff.
diff --git a/.agents/skills/create-commit/SKILL.md b/.agents/skills/create-commit/SKILL.md
index dd85191fd9..264f870e03 100644
--- a/.agents/skills/create-commit/SKILL.md
+++ b/.agents/skills/create-commit/SKILL.md
@@ -63,6 +63,8 @@ subject/body limits, and the `AI-assisted-by` trailer. Follow it exactly.
- Do not pass `--author`. Author identity comes from the local git config.
- Do not amend a commit you did not create in this session, unless explicitly asked.
- Do not bypass pre-commit hooks (`--no-verify`) unless explicitly asked.
-- Never add a `Co-authored-by:` trailer. It is forbidden and fails `./scripts/check-commit`.
+- Add `Co-authored-by:` only for a human co-author. AI providers, agents and
+ bots are rejected by `./scripts/check-commit`; for AI participation use the
+ `AI-assisted-by:` trailer.
- Do not add untracked files that were not created in this session.
- Do not skip the `scripts/check-commit` verification step (Iron Rule 4).
diff --git a/.agents/skills/review-code/SKILL.md b/.agents/skills/review-code/SKILL.md
index 21e7a7c7fd..65c811e71d 100644
--- a/.agents/skills/review-code/SKILL.md
+++ b/.agents/skills/review-code/SKILL.md
@@ -22,8 +22,20 @@ Act as a senior software engineer and perform a thorough code review.
diff, a commit range, a branch, a PR (number or URL), or specific files. If
the target is ambiguous, ask before reviewing.
2. Delegate the review to the `general` subagent (via the task tool), unless the
- user specifies another agent. Include in the prompt the
- **`code-review-criteria`** skill name and all user context.
+ user specifies another agent. The prompt carries the **`code-review-criteria`**
+ skill name, all user context, and these requirements:
+ - Load `code-review-criteria` and follow its process and output format.
+ - Read `AGENTS.md` (if present) and the module memories it points to for the
+ code being reviewed.
+ - Return the COMPLETE review, verbatim, exactly as the skill instructs it to
+ be produced. Never summarize it.
+ - Read-only: modify no file and create no commit. Reviewing never writes.
+ - Do not invent problems: every finding must be real and actionable.
+ - Missing tests are an issue, not a suggestion — a severity-tagged finding.
+ - Prioritize by impact: one structural issue outweighs ten nits, and "this
+ could be better" is not a finding — say why and how.
+ - Skip generated files, lockfile-only changes and unrelated modifications
+ unless they introduce security risks.
3. When the subagent returns, output the review to the user verbatim. Do not
summarize it and do not act on its findings.
4. Right after the review, suggest how to proceed based on the findings. These
@@ -44,43 +56,18 @@ the findings. Once the user explicitly states the review is done (or moves on to
a different task), this rule no longer applies and you act as a normal build
agent again.
-## Instructions for the subagent
-
-1. Load the **`code-review-criteria`** skill and follow its process and output
- format.
-2. Read `AGENTS.md` (if present) and follow its instructions for finding and
- reading all related testing documentation from memories before reviewing.
-3. Return in your final message the COMPLETE review, verbatim, exactly as the
- skill instructs it to be produced. Do not summarize it — include the full
- structured review.
-
-### Strong rules for the subagent
-
-1. Do not invent problems. Every finding must be real and actionable.
-2. Read-only: do not modify any file and do not create a commit — reviewing
- never writes.
-3. Be specific and constructive. "This could be better" is not helpful — explain
- why and how.
-4. Prioritize by impact. One structural issue outweighs ten nits.
-5. Missing tests are an issue, not a suggestion. Report as a severity-tagged
- finding — never as a recommendation.
-6. Skip generated files, lockfile-only changes, and unrelated modifications
- unless they introduce security risks.
-
## Posting as a GitHub comment
-Only post the review as a comment when the user asks for it explicitly. Keep
-it in the session otherwise.
-
-When posting: PR comments go in English, with this minimal structure:
+Post only when the user asks for it explicitly; otherwise keep the review in the
+session. PR comments go in English, with this minimal structure:
1. **Intro line** — one sentence: what was found and the overall tone. No
greetings.
-2. **`## Main issues`** — one subsection per required finding: what breaks,
- the failure circumstances, and the minimal fix (code snippet when useful).
- A missing test belongs here, not under minor suggestions.
-3. **`## Minor suggestions`** — one bullet per minor finding or suggestion:
- file/function, problem, concrete remedy.
+2. **`## Main issues`** — one subsection per required finding: what breaks, the
+ failure circumstances, and the minimal fix (code snippet when useful). A
+ missing test belongs here, not under minor suggestions.
+3. **`## Minor suggestions`** — one bullet per minor finding: file/function,
+ problem, concrete remedy.
4. **Closing** — optional and short. Cut it when it adds nothing.
Reference every finding by its ID (`R1`, `M1`, `S1`, … from
diff --git a/.github/ISSUE_TEMPLATE/bug-report.yml b/.github/ISSUE_TEMPLATE/bug.yml
similarity index 90%
rename from .github/ISSUE_TEMPLATE/bug-report.yml
rename to .github/ISSUE_TEMPLATE/bug.yml
index 9031969b8e..af5e0e3418 100644
--- a/.github/ISSUE_TEMPLATE/bug-report.yml
+++ b/.github/ISSUE_TEMPLATE/bug.yml
@@ -1,7 +1,7 @@
-description: Create a report to help us improve
-name: Bug report
-type: Bug
+description: An unexpected problem or behavior
labels: ["needs triage"]
+name: Bug
+type: Bug
body:
- type: markdown
@@ -15,7 +15,7 @@ body:
- type: textarea
attributes:
label: Steps To Reproduce
- description: Steps to reproduce the behavior.
+ description: Steps to reproduce the behavior
placeholder: |
Steps to reproduce the behavior:
1. Go to '...'
@@ -26,22 +26,22 @@ body:
- type: textarea
id: expected
attributes:
- description: A clear and concise description of what you expected to happen.
label: Expected behavior
+ description: A clear and concise description of what you expected to happen
validations:
required: true
- type: textarea
id: actual
attributes:
- description: A clear and concise description of what happens instead; what the bug is.
label: Actual behavior
+ description: A clear and concise description of what happens instead; what the bug is
validations:
required: true
- type: textarea
id: screenshots
attributes:
- description: If applicable, add screenshots to help explain your problem.
label: Screenshots or video
+ description: If applicable, add screenshots to help explain your problem
- type: textarea
id: desktop
attributes:
@@ -87,4 +87,4 @@ body:
id: additional-context
attributes:
label: Additional context
- description: Any other context about the problem.
+ description: Add any other context or screenshots about the bug here
diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml
index 0086358db1..3ba13e0cec 100644
--- a/.github/ISSUE_TEMPLATE/config.yml
+++ b/.github/ISSUE_TEMPLATE/config.yml
@@ -1 +1 @@
-blank_issues_enabled: true
+blank_issues_enabled: false
diff --git a/.github/ISSUE_TEMPLATE/enhancement.yml b/.github/ISSUE_TEMPLATE/enhancement.yml
new file mode 100644
index 0000000000..ee4b3249e0
--- /dev/null
+++ b/.github/ISSUE_TEMPLATE/enhancement.yml
@@ -0,0 +1,38 @@
+description: Improvement to existing behavior
+labels: ["needs triage"]
+name: "Enhancement"
+type: Enhancement
+
+body:
+ - type: markdown
+ attributes:
+ value: |
+ ## Before you start
+
+ Please search our [existing issues](https://github.com/penpot/penpot/issues) and open [pull requests](https://github.com/penpot/penpot/pulls) to lessen the change of filing duplicate issues or feature requests. Thank you.
+
+ ---
+ - type: textarea
+ id: problem
+ attributes:
+ label: Is your feature request related to a problem? Please describe
+ description: A clear and concise description of what the problem is. Ex. I'm always frustrated when (...)
+ validations:
+ required: true
+ - type: textarea
+ id: solution
+ attributes:
+ label: Describe the solution you'd like
+ description: A clear and concise description of what you want to happen
+ validations:
+ required: true
+ - type: textarea
+ id: alternatives
+ attributes:
+ label: Describe alternatives you've considered
+ description: A clear and concise description of any alternative solutions or features you've considered
+ - type: textarea
+ id: additional-context
+ attributes:
+ label: Additional context
+ description: Add any other context or screenshots about the enhancement here
diff --git a/.github/ISSUE_TEMPLATE/epic.yml b/.github/ISSUE_TEMPLATE/epic.yml
new file mode 100644
index 0000000000..37dd3cff17
--- /dev/null
+++ b/.github/ISSUE_TEMPLATE/epic.yml
@@ -0,0 +1,26 @@
+description: An epic is a large user story that cannot be delivered as defined within a single iteration
+labels: ["needs triage"]
+name: "EPIC"
+type: EPIC
+
+body:
+ - type: markdown
+ attributes:
+ value: |
+ ## Before you start
+
+ Please search our [existing issues](https://github.com/penpot/penpot/issues) and open [pull requests](https://github.com/penpot/penpot/pulls) to lessen the change of filing duplicate issues or feature requests. Thank you.
+
+ ---
+ - type: textarea
+ id: description
+ attributes:
+ label: Epic description
+ description: Tell us which epic feature do you want to discuss
+ validations:
+ required: true
+ - type: textarea
+ id: additional-context
+ attributes:
+ label: Additional context
+ description: Add any other context or screenshots about the feature request here
diff --git a/.github/ISSUE_TEMPLATE/feature-request.yml b/.github/ISSUE_TEMPLATE/feature.yml
similarity index 80%
rename from .github/ISSUE_TEMPLATE/feature-request.yml
rename to .github/ISSUE_TEMPLATE/feature.yml
index 17f451ae3f..2ba0eb4dbd 100644
--- a/.github/ISSUE_TEMPLATE/feature-request.yml
+++ b/.github/ISSUE_TEMPLATE/feature.yml
@@ -1,7 +1,7 @@
-description: Suggest an idea for this project.
+description: Suggest an idea for this project
labels: ["needs triage"]
name: "Feature request"
-type: Enhancement
+type: Feature
body:
- type: markdown
@@ -15,24 +15,24 @@ body:
- type: textarea
id: problem
attributes:
+ label: Is your feature request related to a problem? Please describe
description: A clear and concise description of what the problem is. Ex. I'm always frustrated when (...)
- label: Is your feature request related to a problem? Please describe.
validations:
required: true
- type: textarea
id: solution
attributes:
- description: A clear and concise description of what you want to happen.
- label: Describe the solution you'd like.
+ label: Describe the solution you'd like
+ description: A clear and concise description of what you want to happen
validations:
required: true
- type: textarea
id: alternatives
attributes:
- label: Describe alternatives you've considered.
- description: A clear and concise description of any alternative solutions or features you've considered.
+ label: Describe alternatives you've considered
+ description: A clear and concise description of any alternative solutions or features you've considered
- type: textarea
id: additional-context
attributes:
label: Additional context
- description: Add any other context or screenshots about the feature request here.
+ description: Add any other context or screenshots about the feature request here
diff --git a/.github/ISSUE_TEMPLATE/task.yml b/.github/ISSUE_TEMPLATE/task.yml
new file mode 100644
index 0000000000..c82e98a04a
--- /dev/null
+++ b/.github/ISSUE_TEMPLATE/task.yml
@@ -0,0 +1,26 @@
+description: A specific piece of work
+labels: []
+name: "Task"
+type: Task
+
+body:
+ - type: markdown
+ attributes:
+ value: |
+ ## Before you start
+
+ Please search our [existing issues](https://github.com/penpot/penpot/issues) and open [pull requests](https://github.com/penpot/penpot/pulls) to lessen the change of filing duplicate issues or feature requests. Thank you.
+
+ ---
+ - type: textarea
+ id: task
+ attributes:
+ description: A clear and concise description of the task
+ label: Please describe
+ validations:
+ required: true
+ - type: textarea
+ id: additional-context
+ attributes:
+ label: Additional context
+ description: Add any other context or screenshots about the task here
diff --git a/.github/workflows/auto-label.yml b/.github/workflows/auto-label.yml
index 65dcc51875..ff4f55f096 100644
--- a/.github/workflows/auto-label.yml
+++ b/.github/workflows/auto-label.yml
@@ -43,6 +43,9 @@ jobs:
// === 1. CONFIGURATION ===
const PROJECT_NUMBER = 8; // <--- Replace with your project board number
const IS_ORG = true; // <--- Set to false if this is a personal project, true if an organization
+ const STATUS_FIELD_ID = 'PVTSSF_lADOAcyBPM4BV2pqzhRPY48';
+ const STATUS_BACKLOG = 'f75ad846';
+ const STATUS_TECH_REVIEW = 'b0f45db9';
const OWNER = context.repo.owner;
const REPO = context.repo.repo;
@@ -88,6 +91,65 @@ jobs:
}
`;
- await github.graphql(addToProjectMutation, { projectId, contentId });
+ let itemId;
+ try {
+ const addRes = await github.graphql(addToProjectMutation, { projectId, contentId });
+ itemId = addRes.addProjectV2ItemById.item.id;
+ } catch (error) {
+ const messages = [
+ error && error.message,
+ ...(Array.isArray(error && error.errors)
+ ? error.errors.map(e => e && e.message)
+ : [])
+ ].filter(Boolean).join('\n');
+ if (!messages.includes('Content already exists in this project')) {
+ throw error;
+ }
+ console.log('Item already in project, fetching existing item...');
+ const existing = await github.graphql(
+ `query($contentId: ID!) {
+ node(id: $contentId) {
+ ... on Issue {
+ projectItems(first: 50) {
+ nodes { id project { id } }
+ }
+ }
+ ... on PullRequest {
+ projectItems(first: 50) {
+ nodes { id project { id } }
+ }
+ }
+ }
+ }`,
+ { contentId }
+ );
+ const nodes = (existing.node && existing.node.projectItems && existing.node.projectItems.nodes) || [];
+ const match = nodes.find(n => n.project.id === projectId);
+ if (!match) {
+ throw new Error(`Item already in a project, but not found in project ${projectId}`);
+ }
+ itemId = match.id;
+ }
+
+ // === 4. SET PROJECT STATUS ===
+ const statusOptionId = isPR ? STATUS_TECH_REVIEW : STATUS_BACKLOG;
+ const statusName = isPR ? 'Tech Review' : 'Backlog';
+ console.log(`Setting status to "${statusName}" for ${isPR ? 'PR' : 'Issue'} #${issueNumber}...`);
+ await github.graphql(
+ `mutation($projectId: ID!, $itemId: ID!, $fieldId: ID!, $optionId: String!) {
+ updateProjectV2ItemFieldValue(input: {
+ projectId: $projectId,
+ itemId: $itemId,
+ fieldId: $fieldId,
+ value: { singleSelectOptionId: $optionId }
+ }) {
+ projectV2Item {
+ id
+ }
+ }
+ }`,
+ { projectId, itemId, fieldId: STATUS_FIELD_ID, optionId: statusOptionId }
+ );
+
console.log("Automation successfully completed!");
diff --git a/.github/workflows/plugins-deploy-api-doc.yml b/.github/workflows/plugins-deploy-api-doc.yml
index 4d22311892..0d963cdfe6 100644
--- a/.github/workflows/plugins-deploy-api-doc.yml
+++ b/.github/workflows/plugins-deploy-api-doc.yml
@@ -51,7 +51,7 @@ jobs:
uses: pnpm/setup@731dfb6463bbd11c11fd0ee11fe00ed0e36d48a1 # v3.2.0
with:
working-directory: plugins
- runtime: node@26.13.1
+ runtime: node@24.22.3
install: false
- name: Install deps
diff --git a/.github/workflows/plugins-deploy-styles-doc.yml b/.github/workflows/plugins-deploy-styles-doc.yml
index 8fd4157a41..60ffe05038 100644
--- a/.github/workflows/plugins-deploy-styles-doc.yml
+++ b/.github/workflows/plugins-deploy-styles-doc.yml
@@ -49,7 +49,7 @@ jobs:
uses: pnpm/setup@731dfb6463bbd11c11fd0ee11fe00ed0e36d48a1 # v3.2.0
with:
working-directory: plugins
- runtime: node@26.13.1
+ runtime: node@24.22.3
install: false
- name: Install deps
diff --git a/.github/workflows/require-milestone-linked-issues.yml b/.github/workflows/require-milestone-linked-issues.yml
new file mode 100644
index 0000000000..d00dade45e
--- /dev/null
+++ b/.github/workflows/require-milestone-linked-issues.yml
@@ -0,0 +1,76 @@
+name: Require Milestone on Linked Issues
+
+on:
+ pull_request:
+ types:
+ - opened
+ - reopened
+ - synchronize
+ - edited
+ - ready_for_review
+ - converted_to_draft
+ workflow_dispatch:
+
+permissions:
+ contents: read
+ pull-requests: read
+ issues: read
+
+jobs:
+ check-linked-issues:
+ name: Require Milestones on Linked Issues
+ runs-on: ubuntu-latest
+
+ steps:
+ - name: Check linked issues
+ env:
+ GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ PR_NUMBER: ${{ github.event.pull_request.number }}
+ IS_DRAFT: ${{ github.event.pull_request.draft }}
+ REPOSITORY: ${{ github.repository }}
+ run: |
+ if [ "$IS_DRAFT" = "true" ]; then
+ echo "Draft PR: milestone check skipped."
+ exit 0
+ fi
+
+ OWNER="${REPOSITORY%%/*}"
+ REPO="${REPOSITORY#*/}"
+
+ # shellcheck disable=SC2016 # GraphQL variables, not shell expansion.
+ ISSUES=$(gh api graphql \
+ -F owner="$OWNER" \
+ -F repo="$REPO" \
+ -F number="$PR_NUMBER" \
+ -f query='
+ query($owner: String!, $repo: String!, $number: Int!) {
+ repository(owner: $owner, name: $repo) {
+ pullRequest(number: $number) {
+ closingIssuesReferences(first: 100) {
+ nodes {
+ number
+ title
+ url
+ milestone {
+ title
+ }
+ }
+ }
+ }
+ }
+ }
+ ')
+
+ MISSING=$(echo "$ISSUES" | jq -r '
+ .data.repository.pullRequest.closingIssuesReferences.nodes[]
+ | select(.milestone == null)
+ | "- #\(.number): \(.title) (\(.url))"
+ ')
+
+ if [ -n "$MISSING" ]; then
+ echo "::error::The following linked issues have no milestone:"
+ echo "$MISSING"
+ exit 1
+ fi
+
+ echo "All linked issues have a milestone."
diff --git a/.github/workflows/require-milestone.yml b/.github/workflows/require-milestone.yml
new file mode 100644
index 0000000000..641e1a98fa
--- /dev/null
+++ b/.github/workflows/require-milestone.yml
@@ -0,0 +1,45 @@
+name: Require Milestone
+
+on:
+ pull_request:
+ types:
+ - opened
+ - reopened
+ - synchronize
+ - edited
+ - milestoned
+ - demilestoned
+ - ready_for_review
+ - converted_to_draft
+
+permissions:
+ contents: read
+ pull-requests: read
+
+jobs:
+ check-milestone:
+ name: Require Milestone
+ runs-on: ubuntu-latest
+
+ steps:
+ - name: Check milestone
+ env:
+ GH_TOKEN: ${{ github.token }}
+ PR_NUMBER: ${{ github.event.pull_request.number }}
+ run: |
+ pr_api="repos/${GITHUB_REPOSITORY}/pulls/${PR_NUMBER}"
+
+ is_draft=$(gh api "$pr_api" --jq '.draft')
+ milestone=$(gh api "$pr_api" --jq '.milestone.title // ""')
+
+ if [ "$is_draft" = "true" ]; then
+ echo "Draft PR: milestone check skipped."
+ exit 0
+ fi
+
+ if [ -z "$milestone" ]; then
+ echo "::error::This pull request must have a milestone."
+ exit 1
+ fi
+
+ echo "Milestone assigned: $milestone"
diff --git a/.serena/memories/backend/subtleties.md b/.serena/memories/backend/subtleties.md
index d387fb3b34..8b09aa1ad0 100644
--- a/.serena/memories/backend/subtleties.md
+++ b/.serena/memories/backend/subtleties.md
@@ -53,6 +53,7 @@
- `app.config/config` and `flags` are dynamic `defonce` vars populated from `PENPOT_*` env vars through the shared schema string transformer. Tests and tooling can bind them.
- Duration config values are parsed as ISO-8601 with the `PT` prefix, so a day is written in hours (`:jobs-retention` and `:jobs-user-ttl` default to a week, and an operator writes `168h`): `7d` does not parse, and the value fails validation at boot instead of being silently ignored.
+- `app.main` wires `PENPOT_DATABASE_*` into `::db/pool`, and `app.db/create-datasource-config` forwards pgjdbc's own knobs as `dataSourceProperties`, which `Properties.getProperty` silently drops unless the value is a String (hence the `str`). A query parameter of the same name in `PENPOT_DATABASE_URI` (for example `?prepareThreshold=0`) wins over the property: pgjdbc ignores the Properties value when the URL carries it, so the URI is the place to look when a variable seems to do nothing. The prepared statement cache is three optional `PENPOT_DATABASE_PREPARED_STATEMENT_CACHE_*` keys: the threshold is the executions before a statement is server-prepared and accepts `-1` or non-negatives (`{:min -1}`; the driver clamps any other negative to `-1`, so rejecting it only turns a typo into a boot error), its `0` stops server-side prepared statements while the driver's client-side cache of parsed queries stays in use, `..._QUERIES` and `..._SIZE` reject negatives and `0` in either disables that cache, and the size is bytes rounded up to the whole MiB the driver accepts. Do not describe the threshold `0` as disabling the cache: it only stops the server-side prepares.
- `parse-flags` automatically adds `:disable-secure-session-cookies` when `public-uri` is plain HTTP and not localhost. This changes cookie defaults without an explicit env flag.
- The backend sets Clojure `*assert*` globally from the `:backend-asserts` feature flag. Assertion-dependent checks can therefore differ by runtime flags.
- Request body parsing is mostly POST-oriented and supports Transit JSON plus plain JSON. Plain JSON request keys are kebab-decoded before being merged into `:params`. A body the parser cannot read is a client error, classified at the parse site and never by exception class (`clojure.data.json` signals its syntax errors with a bare `java.lang.Exception`): 400 `:validation`/`:malformed-json`, one `warn` line (`malformed request body`) carrying path, peer address and content-type instead of a stack trace, and `penpot_http_malformed_requests_total`. `RequestTooBigException` keeps its 413.
diff --git a/.serena/memories/backend/testing.md b/.serena/memories/backend/testing.md
index 61c92d8fed..b761504a87 100644
--- a/.serena/memories/backend/testing.md
+++ b/.serena/memories/backend/testing.md
@@ -33,4 +33,4 @@ JVM `clojure.test` (kaocha runner) under `backend/test/backend_tests/`.
`worker-config` can talk to a different Redis key prefix than a cfg the test
builds itself. The symptom is a component that silently does nothing
locally and races in CI (which sets no `PENPOT_TENANT`).
-- `backend-tests.helpers` has `metrics-stub` (a metrics instance with no collectors, where every `run!` is a no-op), `init-climit` (a concurrency limiter built from an inline config map, with an optional second argument for the metrics instance to publish to) and `with-climit`, the macro that builds one, runs the body and halts it. `init-climit` starts the republisher daemon thread, so a test that builds a limiter without halting it leaks that thread for the rest of the JVM run: use `with-climit`. Log lines are capturable with `add-watch` on `app.common.logging/log-record` (it holds the last record; the watcher sees every one, but only of the levels the logger has enabled: a line the app writes at TRACE never gets there).
+- `backend-tests.helpers` has `metrics-stub` (a metrics instance with no collectors, where every `run!` is a no-op), `init-climit` (a concurrency limiter built from an inline config map, with an optional second argument for the metrics instance to publish to) and `with-climit`, the macro that builds one, runs the body and halts it. `init-climit` starts the republisher daemon thread, so a test that builds a limiter without halting it leaks that thread for the rest of the JVM run: use `with-climit`. Log lines are capturable with `add-watch` on `app.common.logging/log-record` (it holds the last record; the watcher sees every one, but only of the levels the logger has enabled: a line the app writes at TRACE never gets there). The logger swaps the record on an executor thread, after the call that logged it returns: keep the watch installed until the record arrives (poll for it) and only then remove it, or the record lands on nobody under load.
diff --git a/.serena/memories/common/core.md b/.serena/memories/common/core.md
index 816d131e14..1c969b5ba5 100644
--- a/.serena/memories/common/core.md
+++ b/.serena/memories/common/core.md
@@ -26,6 +26,12 @@ Respect the intended abstraction direction in new/refactored code:
Some legacy code violates this layering; do not copy those violations into new code when a focused refactor is practical.
+## Build-mode dependent laziness
+
+`app.common.data.macros/lazy-delay` returns a delay whose expression is evaluated eagerly by default. Only the builds that set the `app.common.data.macros/lazy-delay?` closure define to `true` defer it to the first deref; every production build does (frontend `main` and `worker`, exporter, library), so the Closure compiler replaces the constant and drops the eager branch. On the JVM it is always eager.
+
+`app.common.schema/check-fn` resolves its schema through it: an invalid schema definition fails when the namespace loads in development and in tests, and only on the first check call in production. The other lazily built parts of `check-fn` (validator and explainer) are always deferred.
+
## Focused memory routing
Model, schema, and persistence shape:
diff --git a/.serena/memories/frontend/plugin-api-to-cljs-binding.md b/.serena/memories/frontend/plugin-api-to-cljs-binding.md
index 31ceb4b10c..a55d857bbd 100644
--- a/.serena/memories/frontend/plugin-api-to-cljs-binding.md
+++ b/.serena/memories/frontend/plugin-api-to-cljs-binding.md
@@ -37,5 +37,7 @@
- Plugin listeners are watches on the global store and callbacks are debounced about 10ms. Callback exceptions are caught and logged so plugin code does not crash the app.
- `selectionchange` callbacks receive arrays of shape id strings, while `filechange`, `pagechange`, and `shapechange` return proxies.
- `contentsave` fires only when persistence status transitions to `:saved`; it calls the callback with no value.
+- `penpot.waitForSave(timeout?)` (`app.plugins.persist/wait-for-save`; runtime forwarder checks `content:read`) waits on the **shared** persistence queue, not only the plugin's own commits. Flow: ~100ms intake race (buffer/queue non-empty vs deadline, capped to the timeout) → empty resolves; pending emits `::dps/force-persist` and races `dps/wait-persisted-or-error` against `::dps/rollback-rejected` / `::dps/rolled-back` (rollback rejects even if status later reads `:saved`). Default 60s. `:error` status at intake rejects at once (`earlier-failure`): `save-retrying` while `dps/save-retrying?` (slow retry cycle, recovers by itself), otherwise the stored `[:error :code]` with reopen advice (a terminal failure stays until the file is reopened, so it fails every call). Rejections are `Error`s with `code`: `save-timeout`, `rolled-back`, `save-retrying`, or the failure code (`save-failed` unless the backend gave one).
+- Plugin timeouts validate through `u/valid-timeout?` (positive finite number ≤ `u/max-timeout`, signed 32-bit) and reject through `u/reject-not-valid`, which rejects an `Error` (never a bare string) so callers can read `.message`.
- `contentrollback` is not a state watch: it subscribes to `st/stream` for `::dps/rolled-back` (kept in `subscriptions`, disposed by `remove-listener`), not debounced, delivered async so the reverted state is in place. Payload `{fileIds pageIds shapeIds}` from `pcm/rollback-targets` (page shapes only, from the reverted commits' redo changes). When the rollback falls back to reloading the file, `::dps/rolled-back` waits for the reloaded data, so the callback never reads the rejected data. The runtime's `validEvents` must list every event name, or `penpot.on` rejects it; a stale runtime build rejects new events. The MCP plugin forwards these rollbacks with the next task response and the server puts a warning first in that tool response (`RollbackNotices`). Unsent ones survive a tab or plugin reload in `penpot.localStorage` (key `rollbacks/` against the downloaded JDK before bumping. Source of the URL and both sums: the Azul metadata API (`api.azul.com/metadata/v1/zulu/packages/?java_version=
-
-
-
diff --git a/backend/deps.edn b/backend/deps.edn
index 3a8a282638..03b38e7881 100644
--- a/backend/deps.edn
+++ b/backend/deps.edn
@@ -40,7 +40,6 @@
nrepl/nrepl {:mvn/version "1.8.0"}
org.postgresql/postgresql {:mvn/version "42.7.13"}
- org.xerial/sqlite-jdbc {:mvn/version "3.53.4.0"}
com.zaxxer/HikariCP {:mvn/version "7.1.0"}
diff --git a/backend/src/app/binfile/common.clj b/backend/src/app/binfile/common.clj
index 0bed77adc1..ed01b120f0 100644
--- a/backend/src/app/binfile/common.clj
+++ b/backend/src/app/binfile/common.clj
@@ -26,7 +26,6 @@
[app.common.weak :as weak]
[app.config :as cf]
[app.db :as db]
- [app.db.sql :as sql]
[app.features.fdata :as fdata]
[app.features.file-migrations :as fmigr]
[app.loggers.audit :as-alias audit]
@@ -131,18 +130,12 @@
(defn initial-state
[]
{:storage-objects #{}
- :files #{}
- :teams #{}
- :projects #{}})
+ :files #{}})
(defn collect-storage-objects
[state items]
(update state :storage-objects into xf-map-media-id items))
-(defn collect-summary
- [state key items]
- (update state key into xf-map-media-id items))
-
(defn lookup-index
[id]
(when id
@@ -876,27 +869,6 @@
[cfg project-id]
(db/get cfg :project {:id project-id}))
-(def ^:private sql:get-teams
- "SELECT t.* FROM team WHERE id = ANY(?)")
-
-(defn get-teams
- [cfg ids]
- (let [conn (db/get-connection cfg)
- ids (db/create-array conn "uuid" ids)]
- (->> (db/exec! conn [sql:get-teams ids])
- (map decode-row-features))))
-
-(defn get-team
- [cfg team-id]
- (-> (db/get cfg :team {:id team-id})
- (decode-row-features)))
-
-(defn get-fonts
- [cfg team-id]
- (db/query cfg :team-font-variant
- {:team-id team-id
- :deleted-at nil}))
-
(def ^:private sql:get-files-rels
"SELECT flr.*,
fls.synced_at
@@ -978,15 +950,6 @@
:deleted-at nil})
(not-empty)))
-(defn get-file-thumbnail
- "Return the thumbnail for the specified file-id"
- [cfg {:keys [id revn]}]
- (db/get* cfg :file-thumbnail
- {:file-id id
- :revn revn
- :data nil}
- {::sql/columns [:media-id :file-id :revn]}))
-
(def ^:private sql:get-missing-media-references
"SELECT fmo.*
FROM file_media_object AS fmo
diff --git a/backend/src/app/binfile/v2.clj b/backend/src/app/binfile/v2.clj
deleted file mode 100644
index 06bb22b7c8..0000000000
--- a/backend/src/app/binfile/v2.clj
+++ /dev/null
@@ -1,450 +0,0 @@
-;; This Source Code Form is subject to the terms of the Mozilla Public
-;; License, v. 2.0. If a copy of the MPL was not distributed with this
-;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
-;;
-;; Copyright (c) KALEIDOS SUBSIDIARY SL
-
-(ns app.binfile.v2
- "A sqlite3 based binary file exportation with support for exportation
- of entire team (or multiple teams) at once."
- (:refer-clojure :exclude [read])
- (:require
- [app.binfile.common :as bfc]
- [app.common.data :as d]
- [app.common.features :as cfeat]
- [app.common.logging :as l]
- [app.common.time :as ct]
- [app.common.transit :as t]
- [app.common.uuid :as uuid]
- [app.config :as cf]
- [app.db :as db]
- [app.db.sql :as sql]
- [app.loggers.audit :as-alias audit]
- [app.loggers.webhooks :as-alias webhooks]
- [app.storage :as sto]
- [app.storage.tmp :as tmp]
- [app.util.events :as events]
- [app.worker :as-alias wrk]
- [clojure.set :as set]
- [cuerdas.core :as str]
- [datoteka.io :as io]
- [promesa.util :as pu])
- (:import
- java.sql.DriverManager))
-
-(set! *warn-on-reflection* true)
-
-;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
-;; LOW LEVEL API
-;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
-
-(defn- create-database
- ([cfg]
- (let [path (tmp/tempfile :prefix "penpot.binfile." :suffix ".sqlite")]
- (create-database cfg path)))
- ([cfg path]
- (let [db (DriverManager/getConnection (str "jdbc:sqlite:" path))]
- (assoc cfg ::db db ::path path))))
-
-(def ^:private
- sql:create-kvdata-table
- "CREATE TABLE kvdata (
- tag text NOT NULL,
- key text NOT NULL,
- val text NOT NULL,
- dat blob NULL
- )")
-
-(def ^:private
- sql:create-kvdata-index
- "CREATE INDEX kvdata__tag_key__idx
- ON kvdata (tag, key)")
-
-(defn- setup-schema!
- [{:keys [::db]}]
- (db/exec-one! db [sql:create-kvdata-table])
- (db/exec-one! db [sql:create-kvdata-index]))
-
-(defn- write!
- [{:keys [::db]} tag k v & [data]]
- (db/insert! db :kvdata
- {:tag (d/name tag)
- :key (str k)
- :val (t/encode-str v {:type :json-verbose})
- :dat data}
- {::db/return-keys false}))
-
-(defn- read-blob
- [{:keys [::db]} tag k]
- (let [obj (db/get db :kvdata
- {:tag (d/name tag)
- :key (str k)}
- {::sql/columns [:dat]})]
- (:dat obj)))
-
-(defn- read-seq
- ([{:keys [::db]} tag]
- (->> (db/query db :kvdata
- {:tag (d/name tag)}
- {::sql/columns [::val]})
- (map :val)
- (map t/decode-str)))
- ([{:keys [::db]} tag k]
- (->> (db/query db :kvdata
- {:tag (d/name tag)
- :key (str k)}
- {::sql/columns [::val]})
- (map :val)
- (map t/decode-str))))
-
-(defn- read-obj
- [{:keys [::db]} tag k]
- (let [obj (db/get db :kvdata
- {:tag (d/name tag)
- :key (str k)}
- {::sql/columns [:val]})]
- (-> obj :val t/decode-str)))
-
-;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
-;; IMPORT/EXPORT IMPL
-;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
-
-(declare ^:private write-project!)
-(declare ^:private write-file!)
-
-(defn- write-team!
- [cfg team-id]
-
- (let [team (bfc/get-team cfg team-id)
- fonts (bfc/get-fonts cfg team-id)]
-
- (events/tap :progress
- {:op :export
- :section :write-team
- :id team-id
- :name (:name team)})
-
- (l/trc :hint "write" :obj "team"
- :id (str team-id)
- :fonts (count fonts))
-
- (when-let [photo-id (:photo-id team)]
- (vswap! bfc/*state* update :storage-objects conj photo-id))
-
- (vswap! bfc/*state* update :teams conj team-id)
- (vswap! bfc/*state* bfc/collect-storage-objects fonts)
-
- (write! cfg :team team-id team)
-
- (doseq [{:keys [id] :as font} fonts]
- (vswap! bfc/*state* update :team-font-variants conj id)
- (write! cfg :team-font-variant id font))))
-
-(defn- write-project!
- [cfg project]
- (events/tap :progress
- {:op :export
- :section :write-project
- :id (:id project)
- :name (:name project)})
- (l/trc :hint "write" :obj "project" :id (str (:id project)))
- (write! cfg :project (str (:id project)) project)
- (vswap! bfc/*state* update :projects conj (:id project)))
-
-(defn- write-file!
- [cfg file-id]
- (let [file (bfc/get-file cfg file-id :realize? true)
- thumbs (bfc/get-file-object-thumbnails cfg file-id)
- media (bfc/get-file-media cfg file)
- rels (bfc/get-files-rels cfg #{file-id})]
-
- (events/tap :progress
- {:op :export
- :section :write-file
- :id file-id
- :name (:name file)})
-
- (vswap! bfc/*state* (fn [state]
- (-> state
- (update :files conj file-id)
- (update :file-media-objects into bfc/xf-map-id media)
- (bfc/collect-storage-objects thumbs)
- (bfc/collect-storage-objects media))))
-
- (write! cfg :file file-id file)
- (write! cfg :file-rels file-id rels)
-
- (run! (partial write! cfg :file-media-object file-id) media)
- (run! (partial write! cfg :file-object-thumbnail file-id) thumbs)
-
- (when-let [thumb (bfc/get-file-thumbnail cfg file)]
- (vswap! bfc/*state* bfc/collect-storage-objects [thumb])
- (write! cfg :file-thumbnail file-id thumb))
-
- (l/trc :hint "write" :obj "file"
- :thumbnails (count thumbs)
- :rels (count rels)
- :media (count media))))
-
-(defn- write-storage-object!
- [{:keys [::sto/storage] :as cfg} id]
- (let [sobj (sto/get-object storage id)
- data (with-open [input (sto/get-object-data storage sobj)]
- (io/read input))]
-
- (l/trc :hint "write" :obj "storage-object" :id (str id) :size (:size sobj))
- (write! cfg :storage-object id (meta sobj) data)))
-
-(defn- read-storage-object!
- [{:keys [::sto/storage ::bfc/timestamp] :as cfg} id]
- (let [mdata (d/update-when (read-obj cfg :storage-object id)
- :content-type bfc/normalize-content-type)
- data (read-blob cfg :storage-object id)
- ;; NOTE: v2 bundles are legacy and no longer produced; the
- ;; sanitizer call stays as defense in depth and is covered by
- ;; the shared helper unit tests, with no dedicated v2
- ;; import test.
- clean (when (bfc/svg-object? mdata)
- (bfc/sanitize-imported-svg mdata data))
- data (or (:bytes clean) data)
- hash (or (:hash clean) (sto/calculate-hash data))
-
- content (-> (sto/content data)
- (sto/wrap-with-hash hash))
-
- params (-> mdata
- (assoc ::sto/content content)
- (assoc ::sto/deduplicate? true)
- (assoc ::sto/touched-at timestamp))
-
- sobject (sto/put-object! storage params)]
-
- (vswap! bfc/*state* update :index assoc id (:id sobject))
-
- (l/trc :hint "read" :obj "storage-object"
- :id (str id)
- :new-id (str (:id sobject))
- :size (:size sobject))))
-
-(defn read-team!
- [{:keys [::db/conn ::bfc/timestamp] :as cfg} team-id]
- (l/trc :hint "read" :obj "team" :id (str team-id))
-
- (let [team (read-obj cfg :team team-id)
- team (-> team
- (update :id bfc/lookup-index)
- (update :photo-id bfc/lookup-index)
- (assoc :created-at timestamp)
- (assoc :modified-at timestamp))]
-
- (events/tap :progress
- {:op :import
- :section :read-team
- :id team-id
- :name (:name team)})
-
- (db/insert! conn :team
- (update team :features db/encode-pgarray conn "text")
- ::db/return-keys false)
-
- (doseq [font (->> (read-seq cfg :team-font-variant)
- (filter #(= team-id (:team-id %))))]
- (let [font (-> font
- (update :id bfc/lookup-index)
- (update :team-id bfc/lookup-index)
- (update :woff1-file-id bfc/lookup-index)
- (update :woff2-file-id bfc/lookup-index)
- (update :ttf-file-id bfc/lookup-index)
- (update :otf-file-id bfc/lookup-index)
- (assoc :created-at timestamp)
- (assoc :modified-at timestamp))]
- (db/insert! conn :team-font-variant font
- ::db/return-keys false)))
-
- team))
-
-(defn read-project!
- [{:keys [::db/conn ::bfc/timestamp] :as cfg} project-id]
- (l/trc :hint "read" :obj "project" :id (str project-id))
-
- (let [project (read-obj cfg :project project-id)
- project (-> project
- (update :id bfc/lookup-index)
- (update :team-id bfc/lookup-index)
- (assoc :created-at timestamp)
- (assoc :modified-at timestamp))]
-
- (events/tap :progress
- {:op :import
- :section :read-project
- :id project-id
- :name (:name project)})
-
- (db/insert! conn :project project
- ::db/return-keys false)))
-
-(defn read-file!
- [{:keys [::db/conn ::bfc/timestamp] :as cfg} file-id]
- (l/trc :hint "read" :obj "file" :id (str file-id))
-
- (let [file (-> (read-obj cfg :file file-id)
- (update :id bfc/lookup-index)
- (update :project-id bfc/lookup-index))
- file (bfc/process-file cfg file)]
-
- (events/tap :progress
- {:op :import
- :section :read-file
- :id file-id
- :name (:name file)})
-
- ;; All features that are enabled and requires explicit migration are
- ;; added to the state for a posterior migration step.
- (doseq [feature (-> (::bfc/features cfg)
- (set/difference cfeat/no-migration-features)
- (set/difference (:features file)))]
- (vswap! bfc/*state* update :pending-to-migrate (fnil conj []) [feature (:id file)]))
-
- (bfc/save-file! cfg file ::db/return-keys false))
-
- (doseq [thumbnail (read-seq cfg :file-object-thumbnail file-id)]
- (let [thumbnail (-> thumbnail
- (update :file-id bfc/lookup-index)
- (update :media-id bfc/lookup-index))
- file-id (:file-id thumbnail)
-
- thumbnail (update thumbnail :object-id
- #(str/replace-first % #"^(.*?)/" (str file-id "/")))]
-
- (db/insert! conn :file-tagged-object-thumbnail thumbnail
- {::db/return-keys false})))
-
- ;; NOTE: v2 packages are only produced and imported through the
- ;; developer REPL (`app.srepl.binfile`), never through the product, so
- ;; this path keeps inserting the relations as they come: unlike v1 and
- ;; v3 it does not skip self-referencing or circular links (see
- ;; `bfc/library-link-status`).
- (doseq [rel (read-obj cfg :file-rels file-id)]
- (let [rel (-> rel
- (update :file-id bfc/lookup-index)
- (update :library-file-id bfc/lookup-index))]
- (db/insert! conn :file-library-rel rel
- ::db/return-keys false)
- (bfc/upsert-file-library-sync! conn (assoc rel :synced-at timestamp))))
-
- (doseq [media (read-seq cfg :file-media-object file-id)]
- (let [media (-> media
- (update :id bfc/lookup-index)
- (update :file-id bfc/lookup-index)
- (update :media-id bfc/lookup-index)
- (update :thumbnail-id bfc/lookup-index))]
- (db/insert! conn :file-media-object media
- ::db/return-keys false
- ::sql/on-conflict-do-nothing true))))
-
-(def ^:private empty-summary
- {:teams #{}
- :files #{}
- :projects #{}
- :file-media-objects #{}
- :team-font-variants #{}
- :storage-objects #{}})
-
-;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
-;; PUBLIC API
-;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
-
-(defn export-team!
- [cfg team-id]
- (let [id (uuid/next)
- tp (ct/tpoint)
- cfg (create-database cfg)]
-
- (l/inf :hint "start"
- :operation "export"
- :id (str id)
- :path (str (::path cfg)))
-
- (try
- (db/tx-run! cfg (fn [cfg]
- (setup-schema! cfg)
- (binding [bfc/*state* (volatile! empty-summary)]
- (write-team! cfg team-id)
-
- (run! (partial write-project! cfg)
- (bfc/get-team-projects cfg team-id))
-
- (run! (partial write-file! cfg)
- (bfc/get-team-files-ids cfg team-id))
-
- (run! (partial write-storage-object! cfg)
- (-> bfc/*state* deref :storage-objects))
-
- (write! cfg :manifest "team-id" team-id)
- (write! cfg :manifest "objects" (deref bfc/*state*))
-
- (::path cfg))))
- (finally
- (pu/close! (::db cfg))
-
- (let [elapsed (tp)]
- (l/inf :hint "end"
- :operation "export"
- :id (str id)
- :elapsed (ct/format-duration elapsed)))))))
-
-(defn import-team!
- [cfg path]
- (let [id (uuid/next)
- tp (ct/tpoint)
-
- cfg (-> (create-database cfg path)
- (assoc ::bfc/timestamp (ct/now)))]
-
- (l/inf :hint "start"
- :operation "import"
- :id (str id)
- :path (str (::path cfg)))
-
- (try
- (db/tx-run! cfg (fn [{:keys [::db/conn] :as cfg}]
- (db/exec-one! conn ["SET idle_in_transaction_session_timeout = 0"])
- (db/exec-one! conn ["SET CONSTRAINTS ALL DEFERRED"])
-
- (binding [bfc/*state* (volatile! {:index {}})]
- (let [objects (read-obj cfg :manifest "objects")]
-
- ;; We first process all storage objects, they have
- ;; deduplication so we can't rely on simple reindex. This
- ;; operation populates the index for all storage objects.
- (run! (partial read-storage-object! cfg) (:storage-objects objects))
-
- ;; Populate index with all the incoming objects
- (vswap! bfc/*state* update :index
- (fn [index]
- (-> index
- (bfc/update-index (:teams objects))
- (bfc/update-index (:projects objects))
- (bfc/update-index (:files objects))
- (bfc/update-index (:file-media-objects objects))
- (bfc/update-index (:team-font-variants objects)))))
-
- (let [team-id (read-obj cfg :manifest "team-id")
- team (read-team! cfg team-id)
- features (cfeat/get-team-enabled-features cf/flags team)
- cfg (assoc cfg ::bfc/features features)]
-
- (run! (partial read-project! cfg) (:projects objects))
- (run! (partial read-file! cfg) (:files objects))
-
- ;; (run-pending-migrations! cfg)
-
- team)))))
- (finally
- (pu/close! (::db cfg))
-
- (let [elapsed (tp)]
- (l/inf :hint "end"
- :operation "import"
- :id (str id)
- :elapsed (ct/format-duration elapsed)))))))
diff --git a/backend/src/app/config.clj b/backend/src/app/config.clj
index e1d00c902a..2d32ea02e9 100644
--- a/backend/src/app/config.clj
+++ b/backend/src/app/config.clj
@@ -265,6 +265,12 @@
[:database-readonly {:optional true} ::sm/boolean]
[:database-min-pool-size {:optional true} ::sm/int]
[:database-max-pool-size {:optional true} ::sm/int]
+ ;; pgjdbc prepared statement cache (see app.db). Size is in bytes and is
+ ;; converted to the whole MiB the driver accepts. Absent keys mean the
+ ;; driver defaults; see docs/technical-guide/configuration.md.
+ [:database-prepared-statement-cache-threshold {:optional true} [::sm/int {:min -1}]]
+ [:database-prepared-statement-cache-queries {:optional true} [::sm/int {:min 0}]]
+ [:database-prepared-statement-cache-size {:optional true} [::sm/int {:min 0}]]
[:quotes-teams-per-profile {:optional true} ::sm/int]
[:quotes-access-tokens-per-profile {:optional true} ::sm/int]
diff --git a/backend/src/app/db.clj b/backend/src/app/db.clj
index bfce3b2274..8bdeae67a4 100644
--- a/backend/src/app/db.clj
+++ b/backend/src/app/db.clj
@@ -57,7 +57,9 @@
(def ^:private schema:pool-options
[:map {:title "pool-options"}
- [::connect-timeout {:optional true} ::sm/int]
+ [::connection-timeout {:optional true} ::sm/int]
+ [::idle-timeout {:optional true} ::sm/int]
+ [::max-lifetime {:optional true} ::sm/int]
[::max-size {:optional true} ::sm/int]
[::min-size {:optional true} ::sm/int]
[::name {:optional true} :keyword]
@@ -66,6 +68,17 @@
[::username {:optional true} :string]
[::validation-timeout {:optional true} ::sm/int]
[::read-only {:optional true} ::sm/boolean]
+ ;; pgjdbc prepared statement cache. Absent keys mean the driver
+ ;; defaults; the size is in bytes and is converted to whole MiB for
+ ;; the driver (see create-datasource-config). The threshold is the
+ ;; executions before a statement is server-prepared: 0 stops
+ ;; server-side prepared statements (the cache of parsed queries stays
+ ;; in use) and -1 prepares from the first execution. Any other
+ ;; negative is a typo the driver would clamp to -1, so the schema
+ ;; rejects it.
+ [::prepared-statement-cache-threshold {:optional true} [::sm/int {:min -1}]]
+ [::prepared-statement-cache-queries {:optional true} [::sm/int {:min 0}]]
+ [::prepared-statement-cache-size {:optional true} [::sm/int {:min 0}]]
[::mtx/metrics ::mtx/metrics]])
(def defaults
@@ -94,7 +107,10 @@
:credentials (and (contains? cfg ::username)
(contains? cfg ::password))
:min-size (::min-size cfg)
- :max-size (::max-size cfg))
+ :max-size (::max-size cfg)
+ :prepared-statement-cache-threshold (::prepared-statement-cache-threshold cfg)
+ :prepared-statement-cache-queries (::prepared-statement-cache-queries cfg)
+ :prepared-statement-cache-size (::prepared-statement-cache-size cfg))
(create-pool cfg))))
(defmethod ig/halt-key! ::pool
@@ -113,6 +129,17 @@
;; pairwise, when both are partitioned with the same bounds
"SET enable_partitionwise_join = on;"))
+;; pgjdbc reads dataSourceProperties through Properties.getProperty, which
+;; drops non-String values silently, so everything is passed as a string.
+;; preparedStatementCacheSizeMiB only accepts a whole-MiB integer: the
+;; configured size (bytes) is converted up, so any positive value keeps the
+;; cache enabled and 0 disables the cache. A query parameter of the same name
+;; in the URI wins over these properties: pgjdbc ignores the Properties value
+;; when the URL carries it.
+(defn- bytes->mib-ceil
+ [bytes]
+ (Math/ceilDiv (long bytes) (* 1024 1024)))
+
(defn- create-datasource-config
[{:keys [::uri] :as cfg}]
@@ -132,6 +159,13 @@
(.setConnectionInitSql initsql)
(.setInitializationFailTimeout -1))
+ (some->> (::prepared-statement-cache-threshold cfg) (str)
+ (.addDataSourceProperty config "prepareThreshold"))
+ (some->> (::prepared-statement-cache-queries cfg) (str)
+ (.addDataSourceProperty config "preparedStatementCacheQueries"))
+ (some->> (::prepared-statement-cache-size cfg) (bytes->mib-ceil) (str)
+ (.addDataSourceProperty config "preparedStatementCacheSizeMiB"))
+
(->> (mtx/get-registry (::mtx/metrics cfg))
(PrometheusMetricsTrackerFactory.)
(.setMetricsTrackerFactory config))
diff --git a/backend/src/app/main.clj b/backend/src/app/main.clj
index 26b6552e92..cef4a21b03 100644
--- a/backend/src/app/main.clj
+++ b/backend/src/app/main.clj
@@ -425,6 +425,9 @@
::db/read-only (cf/get :database-readonly false)
::db/min-size (cf/get :database-min-pool-size)
::db/max-size (cf/get :database-max-pool-size)
+ ::db/prepared-statement-cache-threshold (cf/get :database-prepared-statement-cache-threshold)
+ ::db/prepared-statement-cache-queries (cf/get :database-prepared-statement-cache-queries)
+ ::db/prepared-statement-cache-size (cf/get :database-prepared-statement-cache-size)
::mtx/metrics (ig/ref ::mtx/metrics)}
;; Default netty IO pool (shared between several services)
diff --git a/backend/src/app/srepl/binfile.clj b/backend/src/app/srepl/binfile.clj
deleted file mode 100644
index 43cb69f8f3..0000000000
--- a/backend/src/app/srepl/binfile.clj
+++ /dev/null
@@ -1,41 +0,0 @@
-;; This Source Code Form is subject to the terms of the Mozilla Public
-;; License, v. 2.0. If a copy of the MPL was not distributed with this
-;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
-;;
-;; Copyright (c) KALEIDOS SUBSIDIARY SL
-
-(ns app.srepl.binfile
- (:require
- [app.binfile.v2 :as binfile.v2]
- [app.common.uuid :as uuid]
- [app.db :as db]
- [app.srepl.helpers :as h]
- [app.system :as sys]
- [cuerdas.core :as str]))
-
-(defn export-team!
- [team-id]
- (let [team-id (h/parse-uuid team-id)]
- (binfile.v2/export-team! sys/system team-id)))
-
-(defn import-team!
- [path & {:keys [owner rollback?] :or {rollback? true}}]
- (db/tx-run! (assoc sys/system ::db/rollback rollback?)
- (fn [cfg]
- (let [team (binfile.v2/import-team! cfg path)
- owner (cond
- (string? owner)
- (db/get* cfg :profile {:email (str/lower owner)})
- (uuid? owner)
- (db/get* cfg :profile {:id owner}))]
-
- (when owner
- (db/insert! cfg :team-profile-rel
- {:id (uuid/next)
- :team-id (:id team)
- :profile-id (:id owner)
- :is-admin true
- :is-owner true
- :can-edit true}))
-
- team))))
diff --git a/backend/test/backend_tests/config_test.clj b/backend/test/backend_tests/config_test.clj
index fa168708ef..5e0e2418b8 100644
--- a/backend/test/backend_tests/config_test.clj
+++ b/backend/test/backend_tests/config_test.clj
@@ -76,6 +76,23 @@
:worker-default-parallelism 4)]
(t/is (= 4 (cf/get :worker-default-parallelism 1))))))
+(t/deftest prepared-statement-cache-keys
+ (t/testing "negative sizes are rejected; the threshold takes -1 or non-negatives"
+ (t/is (not (cf/validate-config (assoc cf/config :database-prepared-statement-cache-queries -1))))
+ (t/is (not (cf/validate-config (assoc cf/config :database-prepared-statement-cache-size -1))))
+ (t/is (not (cf/validate-config (assoc cf/config :database-prepared-statement-cache-threshold -2))))
+ (t/is (cf/validate-config (assoc cf/config :database-prepared-statement-cache-threshold -1))))
+ (t/testing "a -1 threshold survives the environment decoding and validation"
+ (let [cfg (merge cf/config
+ (cf/decode-config {:database-prepared-statement-cache-threshold "-1"}))]
+ (t/is (= -1 (:database-prepared-statement-cache-threshold cfg)))
+ (t/is (cf/validate-config cfg))))
+ (t/testing "a decoded -2 threshold fails validation"
+ (let [cfg (merge cf/config
+ (cf/decode-config {:database-prepared-statement-cache-threshold "-2"}))]
+ (t/is (= -2 (:database-prepared-statement-cache-threshold cfg)))
+ (t/is (not (cf/validate-config cfg))))))
+
(t/deftest jobs-user-ttl
(t/testing "the env value is decoded as a duration"
(let [decoded (cf/decode-config {:jobs-user-ttl "72h"})]
diff --git a/backend/test/backend_tests/db_test.clj b/backend/test/backend_tests/db_test.clj
index b0f20cb256..71de5b7b24 100644
--- a/backend/test/backend_tests/db_test.clj
+++ b/backend/test/backend_tests/db_test.clj
@@ -36,6 +36,113 @@
(do (Thread/sleep 10)
(recur (dec attempt)))))))
+(t/deftest pool-passes-prepared-statement-cache-properties
+ ;; The driver reads dataSourceProperties through Properties.getProperty,
+ ;; which keeps only String values, so the wiring must set them as strings.
+ ;; Comparing the literals alone cannot catch a property the driver ignores,
+ ;; so the threshold cases also read it back from a real connection.
+ (let [metrics (ig/init-key :app.metrics/metrics {:default {}})
+ init-pool (fn [opts]
+ (ig/init-key :app.db/pool
+ (merge {:app.db/uri (java.net.URI. "postgresql://localhost/test")
+ :app.metrics/metrics metrics
+ :app.db/name :test-prepared-statement-cache}
+ opts)))
+ with-connection (fn [opts f]
+ (let [ds (init-pool (merge {:app.db/uri (:database-uri th/config)
+ :app.db/username (:database-username th/config)
+ :app.db/password (:database-password th/config)}
+ opts))]
+ (try
+ (with-open [conn (.getConnection ^HikariDataSource ds)]
+ (f conn))
+ (finally
+ (.close ^HikariDataSource ds)))))
+ prepare-threshold (fn [conn]
+ (.getPrepareThreshold
+ ^org.postgresql.PGConnection
+ (.unwrap conn org.postgresql.PGConnection)))]
+ (t/testing "absent keys stay absent"
+ (let [ds (init-pool {})]
+ (t/is (empty? (.getDataSourceProperties ^HikariDataSource ds)))
+ (.close ^HikariDataSource ds)))
+ (t/testing "the driver defaults apply when no option is set"
+ (with-connection {} (fn [conn] (t/is (= 5 (prepare-threshold conn))))))
+ (t/testing "configured keys reach the driver as strings"
+ (let [ds (init-pool {:app.db/prepared-statement-cache-threshold 1
+ :app.db/prepared-statement-cache-queries 512
+ :app.db/prepared-statement-cache-size 10485760})
+ props (.getDataSourceProperties ^HikariDataSource ds)]
+ (t/is (= "1" (.getProperty props "prepareThreshold")))
+ (t/is (= "512" (.getProperty props "preparedStatementCacheQueries")))
+ (t/is (= "10" (.getProperty props "preparedStatementCacheSizeMiB")))
+ (.close ^HikariDataSource ds)))
+ (t/testing "a configured threshold reaches the driver"
+ (with-connection {:app.db/prepared-statement-cache-threshold 1}
+ (fn [conn] (t/is (= 1 (prepare-threshold conn)))))
+ (with-connection {:app.db/prepared-statement-cache-threshold 0}
+ (fn [conn] (t/is (= 0 (prepare-threshold conn))))))
+ (t/testing "a -1 threshold reaches the driver (forceBinary)"
+ (let [ds (init-pool {:app.db/prepared-statement-cache-threshold -1})]
+ (t/is (= "-1" (.getProperty (.getDataSourceProperties ^HikariDataSource ds)
+ "prepareThreshold")))
+ (.close ^HikariDataSource ds)))
+ (t/testing "a zero threshold reaches the driver (server-side statements off)"
+ (let [ds (init-pool {:app.db/prepared-statement-cache-threshold 0})]
+ (t/is (= "0" (.getProperty (.getDataSourceProperties ^HikariDataSource ds)
+ "prepareThreshold")))
+ (.close ^HikariDataSource ds)))
+ (t/testing "a positive sub-MiB size rounds up so the cache stays enabled"
+ (let [ds (init-pool {:app.db/prepared-statement-cache-size 1})]
+ (t/is (= "1" (.getProperty (.getDataSourceProperties ^HikariDataSource ds)
+ "preparedStatementCacheSizeMiB")))
+ (.close ^HikariDataSource ds)))
+ (t/testing "the MiB ceiling rounds a byte over the boundary up"
+ (let [ds (init-pool {:app.db/prepared-statement-cache-size 1048577})]
+ (t/is (= "2" (.getProperty (.getDataSourceProperties ^HikariDataSource ds)
+ "preparedStatementCacheSizeMiB")))
+ (.close ^HikariDataSource ds)))
+ (t/testing "the cache limits hold a closed statement in the server"
+ ;; A closed statement stays prepared in the server only while the
+ ;; driver's cache holds it, so counting it in pg_prepared_statements is
+ ;; what tells an accepted property from an effective one.
+ (let [cached? (fn [opts]
+ (with-connection
+ (merge {:app.db/prepared-statement-cache-threshold 1} opts)
+ (fn [conn]
+ (dotimes [_ 2]
+ (with-open [stmt (.prepareStatement ^Connection conn "SELECT ?::int AS cache_probe")]
+ (.setInt stmt 1 1)
+ (with-open [rs (.executeQuery stmt)] (.next rs))))
+ (-> (db/exec-one! conn ["SELECT count(*) AS n FROM pg_prepared_statements
+ WHERE statement LIKE '%AS cache_probe'"])
+ :n
+ pos?))))]
+ (t/testing "a positive sub-MiB size keeps the cache enabled"
+ (t/is (cached? {:app.db/prepared-statement-cache-size 1})))
+ (t/testing "zero size disables the cache"
+ (t/is (not (cached? {:app.db/prepared-statement-cache-size 0}))))
+ (t/testing "zero queries disables the cache"
+ (t/is (not (cached? {:app.db/prepared-statement-cache-queries 0}))))))))
+
+(t/deftest pool-rejects-a-non-minus-one-negative-threshold
+ ;; The pool boundary mirrors app.config: -1 prepares from the first
+ ;; execution, 0 stops server-side prepared statements, and any other
+ ;; negative is a typo the driver would silently clamp to -1.
+ (let [options {:app.db/uri (:database-uri th/config)
+ :app.db/username (:database-username th/config)
+ :app.db/password (:database-password th/config)
+ :app.metrics/metrics (ig/init-key :app.metrics/metrics {:default {}})}]
+ (t/testing "a -2 threshold fails the pool schema"
+ (t/is (thrown? clojure.lang.ExceptionInfo
+ (ig/assert-key :app.db/pool
+ (assoc options :app.db/prepared-statement-cache-threshold -2)))))
+ (t/testing "-1 and 0 pass the pool schema"
+ (t/is (nil? (ig/assert-key :app.db/pool
+ (assoc options :app.db/prepared-statement-cache-threshold -1))))
+ (t/is (nil? (ig/assert-key :app.db/pool
+ (assoc options :app.db/prepared-statement-cache-threshold 0)))))))
+
(t/deftest pool-requires-metrics
;; Metrics is required by the pool schema: a pool wired without it
;; fails at the boundary (AssertionError when asserts are enabled) or
diff --git a/backend/test/backend_tests/rpc_file_test.clj b/backend/test/backend_tests/rpc_file_test.clj
index e45794a739..6019839a59 100644
--- a/backend/test/backend_tests/rpc_file_test.clj
+++ b/backend/test/backend_tests/rpc_file_test.clj
@@ -13,6 +13,7 @@
[app.common.pprint :as pp]
[app.common.thumbnails :as thc]
[app.common.time :as ct]
+ [app.common.types.page :as ctp]
[app.common.types.shape :as cts]
[app.common.types.tokens-lib :as ctob]
[app.common.types.tokens-status :as ctos]
@@ -3433,3 +3434,35 @@
(t/is (= (into #{} (map :id) (:result miss))
(into #{} (map :id) (:result hit))))
(t/is (every? #(not (contains? % :migrations)) (:result hit))))))))
+
+(t/deftest add-page-rejects-an-invalid-shape
+ ;; Duplicate page and the undo of a page deletion send the whole page as
+ ;; the client holds it. A tab that loaded the file before a migration
+ ;; ran holds the old format, here camelCase svg attrs (#13107), and the
+ ;; page must not reach the file: the next migrating read would reject
+ ;; the whole file.
+ (let [profile (th/create-profile* 1 {:is-active true})
+ file (th/create-file* 1 {:profile-id (:id profile)
+ :project-id (:default-project-id profile)
+ :is-shared false})
+ page-id (uuid/next)
+ shape-id (uuid/next)
+ shape (cts/setup-shape {:id shape-id :type :rect :name "rect"
+ :x 0 :y 0 :width 10 :height 10
+ :parent-id uuid/zero :frame-id uuid/zero
+ :svg-attrs {:fillRule "nonzero"}})
+ page (-> (ctp/make-empty-page {:id page-id :name "Copy"})
+ (assoc-in [:objects shape-id] shape)
+ (update-in [:objects uuid/zero :shapes] (fnil conj []) shape-id))
+ out (th/command! {::th/type :update-file
+ ::rpc/profile-id (:id profile)
+ :id (:id file)
+ :session-id (uuid/random)
+ :revn 0
+ :vern 0
+ :features cfeat/supported-features
+ :changes [{:type :add-page :id page-id :page page}]})
+ stored (bfc/get-file th/*system* (:id file) :migrate? false :realize? true)]
+ (t/is (th/ex-of-type? (:error out) :assertion))
+ (t/is (th/ex-of-code? (:error out) :data-validation))
+ (t/is (not (contains? (get-in stored [:data :pages-index]) page-id)))))
diff --git a/backend/test/backend_tests/rpc_file_update_test.clj b/backend/test/backend_tests/rpc_file_update_test.clj
index c5a8ba9b6d..34e4fe8b00 100644
--- a/backend/test/backend_tests/rpc_file_update_test.clj
+++ b/backend/test/backend_tests/rpc_file_update_test.clj
@@ -282,18 +282,20 @@
(defn- update-file-logged
"Updates the file and returns the response in `:out` and, in `:rejected`,
- the props of the log record of the rejection, if any."
+ the props of the log record of the rejection, if any. Keeps the watch
+ until the record arrives: the logger writes it on another thread, so it
+ can land after the update returns."
[ctx params]
- (let [records (atom [])
- out (try
- (add-watch l/log-record ::capture
- (fn [_ _ _ record] (swap! records conj record)))
- (update-file ctx params)
- (finally
- (remove-watch l/log-record ::capture)))]
- {:out out
- :rejected (some-> (wait-for-record records "update-file commit rejected")
- ::l/props)}))
+ (let [records (atom [])]
+ (try
+ (add-watch l/log-record ::capture
+ (fn [_ _ _ record] (swap! records conj record)))
+ (let [out (update-file ctx params)]
+ {:out out
+ :rejected (some-> (wait-for-record records "update-file commit rejected")
+ ::l/props)})
+ (finally
+ (remove-watch l/log-record ::capture)))))
(t/deftest bisects-the-commits-to-find-the-invalid-one
(let [{:keys [page-id] :as ctx} (create-file)
diff --git a/common/src/app/common/data/macros.cljc b/common/src/app/common/data/macros.cljc
index 4a6bd5b2a9..39bba7db72 100644
--- a/common/src/app/common/data/macros.cljc
+++ b/common/src/app/common/data/macros.cljc
@@ -123,6 +123,27 @@
(list 'js* (c/str "(~{}?." (str/snake prop) "?? ~{})") obj (list 'cljs.core/get obj prop))
(list `c/get obj prop)))
+#?(:cljs (goog-define lazy-delay? false))
+
+(defmacro lazy-delay
+ "Like `delay`, but only lazy when the build asks for it.
+
+ With the default value of `lazy-delay?` the expression is evaluated
+ immediately, so errors raised while computing it (an invalid schema
+ definition, for instance) surface when the namespace is loaded instead of
+ on the first deref. Production builds set `lazy-delay?` to `true` through
+ `:closure-defines`; then the expression is fully deferred and the compiler
+ drops the eager branch. On the JVM the expression is always evaluated
+ immediately."
+ [expr]
+ (if (:ns &env)
+ `(if lazy-delay?
+ (delay ~expr)
+ (let [value# ~expr]
+ (delay value#)))
+ `(let [value# ~expr]
+ (delay value#))))
+
(defn runtime-assert
[hint f]
(try
diff --git a/common/src/app/common/files/changes.cljc b/common/src/app/common/files/changes.cljc
index b2b0e6de60..b81764e580 100644
--- a/common/src/app/common/files/changes.cljc
+++ b/common/src/app/common/files/changes.cljc
@@ -930,6 +930,13 @@
(when (and id name page)
(ex/raise :type :conflict
:hint "id+name or page should be provided, never both"))
+
+ ;; NOTE: we only perform hard validation on backend. The page arrives
+ ;; whole (duplicate page, undo of a page deletion), as the client holds
+ ;; it, so each of its shapes is validated as `:add-obj` validates one.
+ #?(:clj (doseq [shape (vals (:objects page))]
+ (validate-shape shape (:id page))))
+
(let [page (if (and (string? name) (uuid? id))
(ctp/make-empty-page {:id id :name name})
page)]
diff --git a/common/src/app/common/files/migrations.cljc b/common/src/app/common/files/migrations.cljc
index 1e364407f1..bc55f4b88d 100644
--- a/common/src/app/common/files/migrations.cljc
+++ b/common/src/app/common/files/migrations.cljc
@@ -2422,19 +2422,22 @@
(update :pages-index d/update-vals update-container)
(d/update-when :components d/update-vals update-container))))
-(defmethod migrate-data "0035-fix-shape-svg-attrs-and-defs"
+(defmethod migrate-data "0037-fix-shape-svg-attrs-and-defs"
;; Stored svg attribute keys are kebab-case (issue #11947): the v3 reader
;; already outputs kebab and birth normalizes any outside spelling, so
;; export/import converges to the same state. Old birth and the cleaner
;; era stored camelCase React prop names instead, which the reader
- ;; kebab-ized on import beyond repair. Convert once, with the same
- ;; birth transforms: `:svg-attrs` and `:svg-defs` node attrs go through
+ ;; kebab-ized on import beyond repair. Convert with the same birth
+ ;; transforms: `:svg-attrs` and `:svg-defs` node attrs go through
;; the whitelist+kebab transform (unknown attrs drop, same as birth),
;; while svg-raw `:content` attrs are kebab-ized spelling-only (unknown
;; keys kept: the sidebar reads keys outside the whitelist). Kebab
;; input passes through unchanged, so re-running is safe. This replaces
- ;; "0021-fix-shape-svg-attrs" (id withdrawn from `available-migrations`
- ;; below): files that recorded it migrate again through this one.
+ ;; "0021-fix-shape-svg-attrs" and "0035-fix-shape-svg-attrs-and-defs"
+ ;; (ids withdrawn from `available-migrations` below): files that
+ ;; recorded either migrate again through this one. 0035 was withdrawn
+ ;; because a client that loaded a file before 0035 ran could still save
+ ;; camelCase keys into it afterwards (#13107).
;; Node tags in defs and svg-raw content are stored as keywords; files
;; that hold them as strings get them converted here, since the closed
;; node schemas reject strings. `:svg-transform` held as a plain map
@@ -2579,5 +2582,5 @@
"0033-normalize-main-instance-names"
"0034-fix-path-content-offset"
"0035-fix-dangling-children"
- "0035-fix-shape-svg-attrs-and-defs"
- "0036-clear-nil-keys-in-objects"]))
+ "0036-clear-nil-keys-in-objects"
+ "0037-fix-shape-svg-attrs-and-defs"]))
diff --git a/common/src/app/common/flags.cljc b/common/src/app/common/flags.cljc
index 4439a5b316..f7be0be0da 100644
--- a/common/src/app/common/flags.cljc
+++ b/common/src/app/common/flags.cljc
@@ -238,6 +238,7 @@
:enable-inspect-styles
:enable-feature-fdata-objects-map
:enable-feature-render-wasm
+ :enable-feature-text-editor-wasm-html-paste
:enable-token-import-from-library
:enable-render-switch
:enable-render-wasm-info
diff --git a/common/src/app/common/render_wasm/helpers.cljc b/common/src/app/common/render_wasm/helpers.cljc
index 7a73d6f94e..64ee812a7a 100644
--- a/common/src/app/common/render_wasm/helpers.cljc
+++ b/common/src/app/common/render_wasm/helpers.cljc
@@ -26,12 +26,10 @@
(throw (ex-info hint {:type :wasm-error :code code :hint hint :fn fn-name} cause)))))
(defmacro call
- "A helper for calling a wasm function.
- Catches any exception thrown by the WASM function, reads the error code from
- WASM when available, and routes it based on the error type:
- - :wasm-non-blocking: call app.main.errors/on-error (eventually, shows a toast and logs the error)
- - :wasm-critical or unknown: throws an exception to be handled by the global error handler (eventually, shows the internal error page)
- Restores the stack pointer on failure, since a trap leaves its shadow-stack frames behind."
+ "Calls the WASM function `name`. On failure, throws a `:wasm-error` whose
+ `:code` comes from `error-code`, or `:wasm-critical` when WASM set none.
+ Restores the stack pointer on failure, since a trap leaves its shadow-stack
+ frames behind."
[module name & params]
(let [fn-sym (with-meta (gensym "fn-") {:tag 'function})
module-sym (gensym "module")
diff --git a/common/src/app/common/schema.cljc b/common/src/app/common/schema.cljc
index f4fcab85da..14fa04f919 100644
--- a/common/src/app/common/schema.cljc
+++ b/common/src/app/common/schema.cljc
@@ -11,6 +11,7 @@
#?(:clj [malli.dev.pretty :as mdp])
#?(:clj [malli.dev.virhe :as v])
[app.common.data :as d]
+ [app.common.data.macros :as dm]
[app.common.i18n :as i18n :refer [tr]]
[app.common.json :as json]
[app.common.math :as mth]
@@ -320,25 +321,32 @@
([s] (lookup sr/default-registry s))
([registry s] (schema (mr/schema registry s))))
-(defn check-fn
- "Create a predefined check function"
- [s & {:keys [hint type code]}]
- (let [s #?(:clj
- (schema s)
- :cljs
- (try
- (schema s)
- (catch :default cause
- (let [data (ex-data cause)]
- (if (= :malli.core/invalid-schema (:type data))
- (throw (ex-info
- (str "Invalid schema\n"
- (pp/pprint-str (:data data)))
- {}))
- (throw cause))))))
+(defn- resolve-schema
+ "Resolves the schema definition. On ClojureScript it rewrites the malli
+ invalid schema error with a human readable message."
+ [s]
+ #?(:clj (schema s)
+ :cljs (try
+ (schema s)
+ (catch :default cause
+ (let [data (ex-data cause)]
+ (if (= :malli.core/invalid-schema (:type data))
+ (throw (ex-info
+ (str "Invalid schema\n"
+ (pp/pprint-str (:data data)))
+ {}))
+ (throw cause)))))))
- validator* (delay (m/validator s))
- explainer* (delay (m/explainer s))
+(defn check-fn
+ "Create a predefined check function.
+
+ The schema definition is resolved lazily on production builds, and
+ eagerly on development ones, so invalid schemas are detected early
+ while developing. See `app.common.data.macros/lazy-delay`."
+ [s & {:keys [hint type code]}]
+ (let [s* (dm/lazy-delay (resolve-schema s))
+ validator* (delay (m/validator @s*))
+ explainer* (delay (m/explainer @s*))
hint (or ^boolean hint "check error")
type (or ^boolean type :assertion)
code (or ^boolean code :data-validation)]
diff --git a/common/src/app/common/svg.cljc b/common/src/app/common/svg.cljc
index 15a3bff12d..7349e7e1d8 100644
--- a/common/src/app/common/svg.cljc
+++ b/common/src/app/common/svg.cljc
@@ -673,15 +673,32 @@
{}
attrs)))
+(defn- drop-non-scalar-values
+ "Removes the entries whose value is not a string, a number or a
+ boolean: the stored schema admits nothing else."
+ [attrs]
+ (reduce-kv (fn [res k v]
+ (if (or (string? v) (number? v) (boolean? v))
+ res
+ (dissoc res k)))
+ attrs
+ attrs))
+
(defn attrs->kebab-props
"Like `attrs->props` (same whitelist, trim and `:style` parsing) but
with kebab-case keys, the stored convention for `:svg-attrs` and
- `:svg-defs` node attrs. Kebab is a fixed point: running it twice
- returns the same map."
+ `:svg-defs` node attrs. Values that are not scalars, at the top level
+ or inside `:style`, are dropped: files written by old versions hold
+ maps there, which no renderer can use and the closed schema rejects.
+ Kebab is a fixed point: running it twice returns the same map."
[attrs]
- (-> attrs
- (attrs->props)
- (d/kebab-keys)))
+ (let [props (attrs->props attrs)
+ style (some-> (:style props) (drop-non-scalar-values) (not-empty))]
+ (-> props
+ (dissoc :style)
+ (drop-non-scalar-values)
+ (cond-> (some? style) (assoc :style style))
+ (d/kebab-keys))))
(defn- normalize-node-tree
"Walks a svg node tree applying `attrs-fn` to every `:attrs` map. Tags
diff --git a/common/src/app/common/types/tokens_lib.cljc b/common/src/app/common/types/tokens_lib.cljc
index d045e93e55..950cde98d9 100644
--- a/common/src/app/common/types/tokens_lib.cljc
+++ b/common/src/app/common/types/tokens_lib.cljc
@@ -1678,7 +1678,7 @@
(every? empty? (vals sets))
(and (empty? sets) (empty? themes)))
(throw (ex-info "the file doesn't contain any tokens"
- {:error/code :error.import/invalid-json-data})))
+ {:error/code :error.import/no-token-files-found})))
(as-> (make-tokens-lib) library
(reduce-kv (fn [library name tokens]
@@ -1817,8 +1817,6 @@
A `:dtcg` file is not meant to import back unchanged: tokens come back
with their DTCG type and value, a spacing token as a dimension, `4` as
`4px` and `rgb(255, 0, 0)` as `#ff0000`."
- ([tokens-lib]
- (tsw/write-files (tokens-lib->document tokens-lib (legacy-activation tokens-lib))))
([tokens-lib tokens-status]
(export-dtcg-multi-file tokens-lib tokens-status nil))
([tokens-lib tokens-status opts]
diff --git a/common/test/common_tests/data_macros_test.cljc b/common/test/common_tests/data_macros_test.cljc
new file mode 100644
index 0000000000..8759cd7570
--- /dev/null
+++ b/common/test/common_tests/data_macros_test.cljc
@@ -0,0 +1,33 @@
+;; This Source Code Form is subject to the terms of the Mozilla Public
+;; License, v. 2.0. If a copy of the MPL was not distributed with this
+;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
+;;
+;; Copyright (c) KALEIDOS SUBSIDIARY SL
+
+(ns common-tests.data-macros-test
+ (:require
+ [app.common.data.macros :as dm]
+ [clojure.test :as t]))
+
+(defn- eager?
+ "Mirrors the build mode check performed by `dm/lazy-delay`."
+ []
+ #?(:clj true
+ :cljs (not dm/lazy-delay?)))
+
+(t/deftest lazy-delay-returns-a-delay-with-the-value
+ (let [value (dm/lazy-delay (str "foo" "bar"))]
+ (t/is (delay? value))
+ (t/is (= "foobar" @value))))
+
+(t/deftest lazy-delay-evaluates-immediately-by-default
+ (let [calls (atom 0)
+ _ (dm/lazy-delay (swap! calls inc))]
+ (t/is (= (if (eager?) 1 0) @calls))))
+
+(t/deftest lazy-delay-evaluates-the-expression-only-once
+ (let [calls (atom 0)
+ value (dm/lazy-delay (swap! calls inc))]
+ (t/is (= 1 @value))
+ (t/is (= 1 @value))
+ (t/is (= 1 @calls))))
diff --git a/common/test/common_tests/files_migrations_0035_test.cljc b/common/test/common_tests/files_migrations_0035_test.cljc
index 350b669fed..c64ce69527 100644
--- a/common/test/common_tests/files_migrations_0035_test.cljc
+++ b/common/test/common_tests/files_migrations_0035_test.cljc
@@ -7,8 +7,6 @@
(ns common-tests.files-migrations-0035-test
(:require
[app.common.files.migrations :as cfm]
- [app.common.geom.matrix :as gmt]
- [app.common.geom.rect :as grc]
[app.common.types.file :as ctf]
[app.common.types.objects-map :as omap]
[app.common.types.shape :as cts]
@@ -189,308 +187,3 @@
(t/is (some? (get objects' frame-id))
"the frame that exists is kept"))))
-;; ======================================================================
-;; 0035-fix-shape-svg-attrs-and-defs
-;; Stored svg attribute keys are kebab-case (issue #11947). Old birth and
-;; the cleaner era stored camelCase React prop names, which the v3 reader
-;; kebab-ized on import beyond repair. This migration converts once, with
-;; the same birth transforms: `:svg-attrs` and `:svg-defs` node attrs go
-;; through the whitelist+kebab transform, svg-raw `:content` attrs are
-;; kebab-ized spelling-only. It replaces "0021-fix-shape-svg-attrs" (id
-;; withdrawn): files that recorded it migrate again through this one.
-;; ======================================================================
-
-(def ^:private migration-id
- "0035-fix-shape-svg-attrs-and-defs")
-
-(defn- make-shape
- [shape-id shape]
- (assoc shape :id shape-id))
-
-(defn- camel-shape
- [shape-id]
- (make-shape shape-id
- {:type :rect
- :svg-attrs {:fillRule "evenodd"
- :stroke-width "2"
- :stroke-style "dotted"
- :data-foo "drop-me"}}))
-
-(defn- camel-defs
- []
- {"g1" {:tag :linearGradient
- :attrs {:id "g1"
- :gradientUnits "userSpaceOnUse"
- :xlink:href "#base"
- :data-x "drop-me"}
- :content [{:tag :stop
- :attrs {:offset "0"
- :stopColor "#ffffff"}
- :content []}]}})
-
-(t/deftest migration-0035-converts-camel-svg-attrs-to-kebab-in-pages
- (let [shape-id (uuid/next)
- page-id (uuid/next)
- data {:pages-index
- {page-id
- {:objects
- {shape-id (camel-shape shape-id)}}}}
- data' (cfm/migrate-data data migration-id)
- attrs (get-in data' [:pages-index page-id :objects shape-id :svg-attrs])]
- (t/is (= "evenodd" (:fill-rule attrs)))
- (t/is (= "2" (:stroke-width attrs)))
- (t/is (= "dotted" (:stroke-style attrs)) "penpot-extra-attrs survive")
- (t/is (not (contains? attrs :data-foo)) "unknown attrs are dropped")
- (t/is (not (contains? attrs :fillRule)) "no camel keys survive")))
-
-(t/deftest migration-0035-converts-camel-svg-attrs-to-kebab-in-components
- (let [shape-id (uuid/next)
- component-id (uuid/next)
- data {:components
- {component-id
- {:objects
- {shape-id (camel-shape shape-id)}}}}
- data' (cfm/migrate-data data migration-id)
- attrs (get-in data' [:components component-id :objects shape-id :svg-attrs])]
- (t/is (= "evenodd" (:fill-rule attrs)))
- (t/is (not (contains? attrs :fillRule)) "no camel keys survive")))
-
-(t/deftest migration-0035-normalizes-def-node-attrs
- (let [shape-id (uuid/next)
- page-id (uuid/next)
- data {:pages-index
- {page-id
- {:objects
- {shape-id (make-shape shape-id
- {:type :rect
- :svg-defs (camel-defs)})}}}}
- data' (cfm/migrate-data data migration-id)
- grad (get-in data' [:pages-index page-id :objects shape-id :svg-defs "g1"])]
- (t/is (= :linearGradient (:tag grad)) "tags are untouched")
- (t/is (= "g1" (get-in grad [:attrs :id])) ":id survives")
- (t/is (= "userSpaceOnUse" (get-in grad [:attrs :gradient-units])))
- (t/is (= "#base" (get-in grad [:attrs :xlink-href])))
- (t/is (not (contains? (:attrs grad) :data-x)) "unknown def attrs are dropped")
- (t/is (= "#ffffff" (get-in grad [:content 0 :attrs :stop-color]))
- "nested stops are normalized")))
-
-(t/deftest migration-0035-kebabizes-content-spelling-only
- (let [shape-id (uuid/next)
- page-id (uuid/next)
- raw-id (uuid/next)
- data {:pages-index
- {page-id
- {:objects
- {shape-id (make-shape shape-id
- {:type :path
- :content [{:marker "untouched"}]})
- raw-id (make-shape raw-id
- {:type :svg-raw
- :content {:tag :g
- :attrs {:strokeWidth "2"
- :data-foo "keep-me"}
- :content [{:tag :path
- :attrs {:strokeLinecap "round"}
- :content []}]}})}}}}
- data' (cfm/migrate-data data migration-id)
- path (get-in data' [:pages-index page-id :objects shape-id])
- raw (get-in data' [:pages-index page-id :objects raw-id])
- content (:content raw)]
- (t/is (= [{:marker "untouched"}] (:content path))
- "non-svg-raw content is untouched")
- (t/is (= "2" (get-in content [:attrs :stroke-width])))
- (t/is (= "keep-me" (get-in content [:attrs :data-foo]))
- "unknown content keys survive")
- (t/is (= "round" (get-in content [:content 0 :attrs :stroke-linecap])))))
-
-(defn- string-tag-defs
- []
- {:a {:tag "mask"
- :attrs {:id "a" :maskUnits "userSpaceOnUse"}
- :content [{:tag "rect"
- :attrs {:width "10" :height "10" :fill "#FFF"}
- :content []}]}})
-
-(defn- string-tag-content
- []
- {:tag "g"
- :attrs {}
- :content [{:tag "text"
- :attrs {:fontSize "12"}
- :content ["Hi"]}
- {:tag "style"
- :attrs nil
- :content [".a{fill:red}"]}]})
-
-(t/deftest migration-0035-keywordizes-string-tags
- (let [shape-id (uuid/next)
- raw-id (uuid/next)
- page-id (uuid/next)
- data {:pages-index
- {page-id
- {:objects
- {shape-id (make-shape shape-id
- {:type :rect
- :svg-defs (string-tag-defs)})
- raw-id (make-shape raw-id
- {:type :svg-raw
- :content (string-tag-content)})}}}}
- data' (cfm/migrate-data data migration-id)
- mask (get-in data' [:pages-index page-id :objects shape-id :svg-defs :a])
- content (get-in data' [:pages-index page-id :objects raw-id :content])]
- (t/is (= :mask (:tag mask)))
- (t/is (= :rect (get-in mask [:content 0 :tag])) "nested def tags too")
- (t/is (= "userSpaceOnUse" (get-in mask [:attrs :mask-units])))
- (t/is (= :g (:tag content)))
- (t/is (= :text (get-in content [:content 0 :tag])) "nested content tags too")
- (t/is (= ["Hi"] (get-in content [:content 0 :content])))
- (t/is (= {} (get-in content [:content 1 :attrs])) "nil attrs become an empty map")
- (t/is (= data' (cfm/migrate-data data' migration-id)) "re-running is stable")))
-
-(t/deftest migration-0035-string-tag-file-passes-file-validation
- ;; `migrate-file` validates the whole file data after migrating: a file
- ;; with string node tags or nil node attrs must come out valid, or it
- ;; cannot be opened.
- (let [rect-id (uuid/next)
- raw-id (uuid/next)
- file (ctf/make-file {:name "String svg tags"})
- page-id (first (get-in file [:data :pages]))
- rect (-> (cts/setup-shape {:id rect-id :type :rect :name "rect"})
- (assoc :svg-defs (string-tag-defs)))
- raw (-> (cts/setup-shape {:id raw-id :type :svg-raw :name "raw"})
- (assoc :content (string-tag-content)))
- file (-> file
- (assoc :migrations (disj cfm/available-migrations migration-id))
- (update-in [:data :pages-index page-id :objects] assoc rect-id rect raw-id raw)
- (update-in [:data :pages-index page-id :objects uuid/zero :shapes] conj rect-id raw-id))
- file' (cfm/migrate-file file {})]
- (t/is (= :mask (get-in file' [:data :pages-index page-id :objects rect-id :svg-defs :a :tag])))
- (t/is (= :g (get-in file' [:data :pages-index page-id :objects raw-id :content :tag])))))
-
-(t/deftest migration-0035-turns-plain-map-svg-transform-into-matrix
- ;; The svg defs renderer multiplies `:svg-transform` as a matrix; a
- ;; plain map there renders as `matrix(NaN, ...)`.
- (let [plain-id (uuid/next)
- matrix-id (uuid/next)
- page-id (uuid/next)
- data {:pages-index
- {page-id
- {:objects
- {plain-id (make-shape plain-id
- {:type :path
- :svg-transform {:a 1.0 :b 0.0 :c 0.0 :d 1.0 :e 10.0 :f 20.0}})
- matrix-id (make-shape matrix-id
- {:type :path
- :svg-transform (gmt/matrix 2 0 0 2 0 0)})}}}}
- data' (cfm/migrate-data data migration-id)
- plain' (get-in data' [:pages-index page-id :objects plain-id :svg-transform])
- matrix' (get-in data' [:pages-index page-id :objects matrix-id :svg-transform])]
- (t/is (gmt/matrix? plain'))
- (t/is (= (gmt/matrix 1 0 0 1 10 20) plain'))
- (t/is (= (gmt/matrix 2 0 0 2 0 0) matrix') "matrices are left as they are")
- (t/is (= data' (cfm/migrate-data data' migration-id)) "re-running is stable")))
-
-(t/deftest migration-0035-leaves-kebab-files-equal
- (let [shape-id (uuid/next)
- page-id (uuid/next)
- data {:pages-index
- {page-id
- {:objects
- {shape-id (make-shape shape-id
- {:type :rect
- :svg-attrs {:fill-rule "evenodd"
- :stroke-width "2"}
- :svg-defs {"g1" {:tag :linearGradient
- :attrs {:id "g1"
- :gradient-units "userSpaceOnUse"}
- :content []}}})}}}}
- data' (cfm/migrate-data data migration-id)]
- (t/is (= data data') "kebab is a fixed point")))
-
-(t/deftest migration-0035-is-idempotent
- (let [shape-id (uuid/next)
- page-id (uuid/next)
- data {:pages-index
- {page-id
- {:objects
- {shape-id (camel-shape shape-id)}}}}
- data' (cfm/migrate-data data migration-id)]
- (t/is (= data' (cfm/migrate-data data' migration-id)))))
-
-(t/deftest migration-0035-normalizes-flat-svg-viewbox-to-rect
- ;; Carried over from the withdrawn 0021: legacy files hold
- ;; `:svg-viewbox` as a plain map instead of a rect record.
- (let [shape-id (uuid/next)
- page-id (uuid/next)
- data {:pages-index
- {page-id
- {:objects
- {shape-id (make-shape shape-id
- {:type :rect
- :svg-viewbox {:x 1 :y 2 :width 3 :height 4}})}}}}
- data' (cfm/migrate-data data migration-id)
- viewbox (get-in data' [:pages-index page-id :objects shape-id :svg-viewbox])]
- (t/is (grc/rect? viewbox) "flat viewbox becomes a rect record")
- (t/is (= 1 (:x viewbox)))
- (t/is (= 2 (:y viewbox)))
- (t/is (= 3 (:width viewbox)))
- (t/is (= 4 (:height viewbox)))
- (t/is (= data' (cfm/migrate-data data' migration-id))
- "re-running is stable")))
-
-(t/deftest migration-0035-passes-non-maps-through
- (let [shape-id (uuid/next)
- page-id (uuid/next)
- data {:pages-index
- {page-id
- {:objects
- {shape-id (make-shape shape-id
- {:type :rect
- :svg-attrs "junk"
- :svg-defs "junk"})}}}}
- data' (cfm/migrate-data data migration-id)
- shape (get-in data' [:pages-index page-id :objects shape-id])]
- (t/is (= "junk" (:svg-attrs shape)) "non-maps reach validation untouched")
- (t/is (= "junk" (:svg-defs shape)))))
-
-(t/deftest migration-0035-junk-viewbox-becomes-default-rect
- ;; Unlike `:svg-attrs`/`:svg-defs` (passed through to validation),
- ;; `:svg-viewbox` goes through `make-rect`, which does not throw on
- ;; junk: it falls back to the default rect. Inherited from 0021.
- (let [shape-id (uuid/next)
- page-id (uuid/next)
- data {:pages-index
- {page-id
- {:objects
- {shape-id (make-shape shape-id
- {:type :rect
- :svg-viewbox "junk"})}}}}
- data' (cfm/migrate-data data migration-id)
- viewbox (get-in data' [:pages-index page-id :objects shape-id :svg-viewbox])]
- (t/is (grc/rect? viewbox) "junk viewbox becomes a rect, not an error")
- (t/is (= 0 (:x viewbox)))
- (t/is (= 0.01 (:width viewbox)))))
-
-(t/deftest migration-0035-runs-through-file-migration
- ;; A file that recorded the withdrawn "0021-fix-shape-svg-attrs" (and
- ;; everything else current) still needs migration: 0021 left
- ;; `available-migrations`, which re-arms it for 0035.
- (let [shape-id (uuid/next)
- file (ctf/make-file {:name "Legacy camel svg attrs"})
- page-id (first (get-in file [:data :pages]))
- shape (-> (cts/setup-shape {:id shape-id :type :rect})
- (assoc :svg-attrs {:fillRule "evenodd"}))
- file (-> file
- (assoc :migrations (-> cfm/available-migrations
- (disj migration-id)
- (conj "0021-fix-shape-svg-attrs")))
- (assoc-in [:data :pages-index page-id :objects shape-id] shape))
- file' (cfm/migrate-file file {})]
- (t/is (cfm/need-migration? file) "new migration detected")
- (t/is (not (cfm/need-migration? file')) "new migration recorded")
- (t/is (contains? (:migrations file') migration-id) "migration id persisted")
- (t/is (not (contains? cfm/available-migrations "0021-fix-shape-svg-attrs"))
- "the withdrawn 0021 id stays out: its camel-restoring transform must never run again")
- (t/is (= "evenodd"
- (get-in file' [:data :pages-index page-id :objects shape-id :svg-attrs :fill-rule]))
- "migration repaired file data before schema validation")))
diff --git a/common/test/common_tests/files_migrations_0037_test.cljc b/common/test/common_tests/files_migrations_0037_test.cljc
new file mode 100644
index 0000000000..eff0afb82a
--- /dev/null
+++ b/common/test/common_tests/files_migrations_0037_test.cljc
@@ -0,0 +1,371 @@
+;; This Source Code Form is subject to the terms of the Mozilla Public
+;; License, v. 2.0. If a copy of the MPL was not distributed with this
+;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
+;;
+;; Copyright (c) KALEIDOS SUBSIDIARY SL
+
+(ns common-tests.files-migrations-0037-test
+ (:require
+ [app.common.files.migrations :as cfm]
+ [app.common.geom.matrix :as gmt]
+ [app.common.geom.rect :as grc]
+ [app.common.types.file :as ctf]
+ [app.common.types.shape :as cts]
+ [app.common.uuid :as uuid]
+ [clojure.test :as t]))
+
+;; ======================================================================
+;; 0037-fix-shape-svg-attrs-and-defs
+;; Stored svg attribute keys are kebab-case (issue #11947). Old birth and
+;; the cleaner era stored camelCase React prop names, which the v3 reader
+;; kebab-ized on import beyond repair. This migration converts them with
+;; the same birth transforms: `:svg-attrs` and `:svg-defs` node attrs go
+;; through the whitelist+kebab transform, svg-raw `:content` attrs are
+;; kebab-ized spelling-only. It replaces "0021-fix-shape-svg-attrs" and
+;; "0035-fix-shape-svg-attrs-and-defs" (ids withdrawn): files that
+;; recorded either migrate again through this one.
+;; ======================================================================
+
+(def ^:private migration-id
+ "0037-fix-shape-svg-attrs-and-defs")
+
+(defn- make-shape
+ [shape-id shape]
+ (assoc shape :id shape-id))
+
+(defn- camel-shape
+ [shape-id]
+ (make-shape shape-id
+ {:type :rect
+ :svg-attrs {:fillRule "evenodd"
+ :stroke-width "2"
+ :stroke-style "dotted"
+ :data-foo "drop-me"}}))
+
+(defn- camel-defs
+ []
+ {"g1" {:tag :linearGradient
+ :attrs {:id "g1"
+ :gradientUnits "userSpaceOnUse"
+ :xlink:href "#base"
+ :data-x "drop-me"}
+ :content [{:tag :stop
+ :attrs {:offset "0"
+ :stopColor "#ffffff"}
+ :content []}]}})
+
+(t/deftest migration-0037-converts-camel-svg-attrs-to-kebab-in-pages
+ (let [shape-id (uuid/next)
+ page-id (uuid/next)
+ data {:pages-index
+ {page-id
+ {:objects
+ {shape-id (camel-shape shape-id)}}}}
+ data' (cfm/migrate-data data migration-id)
+ attrs (get-in data' [:pages-index page-id :objects shape-id :svg-attrs])]
+ (t/is (= "evenodd" (:fill-rule attrs)))
+ (t/is (= "2" (:stroke-width attrs)))
+ (t/is (= "dotted" (:stroke-style attrs)) "penpot-extra-attrs survive")
+ (t/is (not (contains? attrs :data-foo)) "unknown attrs are dropped")
+ (t/is (not (contains? attrs :fillRule)) "no camel keys survive")))
+
+(t/deftest migration-0037-converts-camel-svg-attrs-to-kebab-in-components
+ (let [shape-id (uuid/next)
+ component-id (uuid/next)
+ data {:components
+ {component-id
+ {:objects
+ {shape-id (camel-shape shape-id)}}}}
+ data' (cfm/migrate-data data migration-id)
+ attrs (get-in data' [:components component-id :objects shape-id :svg-attrs])]
+ (t/is (= "evenodd" (:fill-rule attrs)))
+ (t/is (not (contains? attrs :fillRule)) "no camel keys survive")))
+
+(t/deftest migration-0037-normalizes-def-node-attrs
+ (let [shape-id (uuid/next)
+ page-id (uuid/next)
+ data {:pages-index
+ {page-id
+ {:objects
+ {shape-id (make-shape shape-id
+ {:type :rect
+ :svg-defs (camel-defs)})}}}}
+ data' (cfm/migrate-data data migration-id)
+ grad (get-in data' [:pages-index page-id :objects shape-id :svg-defs "g1"])]
+ (t/is (= :linearGradient (:tag grad)) "tags are untouched")
+ (t/is (= "g1" (get-in grad [:attrs :id])) ":id survives")
+ (t/is (= "userSpaceOnUse" (get-in grad [:attrs :gradient-units])))
+ (t/is (= "#base" (get-in grad [:attrs :xlink-href])))
+ (t/is (not (contains? (:attrs grad) :data-x)) "unknown def attrs are dropped")
+ (t/is (= "#ffffff" (get-in grad [:content 0 :attrs :stop-color]))
+ "nested stops are normalized")))
+
+(t/deftest migration-0037-kebabizes-content-spelling-only
+ (let [shape-id (uuid/next)
+ page-id (uuid/next)
+ raw-id (uuid/next)
+ data {:pages-index
+ {page-id
+ {:objects
+ {shape-id (make-shape shape-id
+ {:type :path
+ :content [{:marker "untouched"}]})
+ raw-id (make-shape raw-id
+ {:type :svg-raw
+ :content {:tag :g
+ :attrs {:strokeWidth "2"
+ :data-foo "keep-me"}
+ :content [{:tag :path
+ :attrs {:strokeLinecap "round"}
+ :content []}]}})}}}}
+ data' (cfm/migrate-data data migration-id)
+ path (get-in data' [:pages-index page-id :objects shape-id])
+ raw (get-in data' [:pages-index page-id :objects raw-id])
+ content (:content raw)]
+ (t/is (= [{:marker "untouched"}] (:content path))
+ "non-svg-raw content is untouched")
+ (t/is (= "2" (get-in content [:attrs :stroke-width])))
+ (t/is (= "keep-me" (get-in content [:attrs :data-foo]))
+ "unknown content keys survive")
+ (t/is (= "round" (get-in content [:content 0 :attrs :stroke-linecap])))))
+
+(defn- string-tag-defs
+ []
+ {:a {:tag "mask"
+ :attrs {:id "a" :maskUnits "userSpaceOnUse"}
+ :content [{:tag "rect"
+ :attrs {:width "10" :height "10" :fill "#FFF"}
+ :content []}]}})
+
+(defn- string-tag-content
+ []
+ {:tag "g"
+ :attrs {}
+ :content [{:tag "text"
+ :attrs {:fontSize "12"}
+ :content ["Hi"]}
+ {:tag "style"
+ :attrs nil
+ :content [".a{fill:red}"]}]})
+
+(t/deftest migration-0037-keywordizes-string-tags
+ (let [shape-id (uuid/next)
+ raw-id (uuid/next)
+ page-id (uuid/next)
+ data {:pages-index
+ {page-id
+ {:objects
+ {shape-id (make-shape shape-id
+ {:type :rect
+ :svg-defs (string-tag-defs)})
+ raw-id (make-shape raw-id
+ {:type :svg-raw
+ :content (string-tag-content)})}}}}
+ data' (cfm/migrate-data data migration-id)
+ mask (get-in data' [:pages-index page-id :objects shape-id :svg-defs :a])
+ content (get-in data' [:pages-index page-id :objects raw-id :content])]
+ (t/is (= :mask (:tag mask)))
+ (t/is (= :rect (get-in mask [:content 0 :tag])) "nested def tags too")
+ (t/is (= "userSpaceOnUse" (get-in mask [:attrs :mask-units])))
+ (t/is (= :g (:tag content)))
+ (t/is (= :text (get-in content [:content 0 :tag])) "nested content tags too")
+ (t/is (= ["Hi"] (get-in content [:content 0 :content])))
+ (t/is (= {} (get-in content [:content 1 :attrs])) "nil attrs become an empty map")
+ (t/is (= data' (cfm/migrate-data data' migration-id)) "re-running is stable")))
+
+(t/deftest migration-0037-string-tag-file-passes-file-validation
+ ;; `migrate-file` validates the whole file data after migrating: a file
+ ;; with string node tags or nil node attrs must come out valid, or it
+ ;; cannot be opened.
+ (let [rect-id (uuid/next)
+ raw-id (uuid/next)
+ file (ctf/make-file {:name "String svg tags"})
+ page-id (first (get-in file [:data :pages]))
+ rect (-> (cts/setup-shape {:id rect-id :type :rect :name "rect"})
+ (assoc :svg-defs (string-tag-defs)))
+ raw (-> (cts/setup-shape {:id raw-id :type :svg-raw :name "raw"})
+ (assoc :content (string-tag-content)))
+ file (-> file
+ (assoc :migrations (disj cfm/available-migrations migration-id))
+ (update-in [:data :pages-index page-id :objects] assoc rect-id rect raw-id raw)
+ (update-in [:data :pages-index page-id :objects uuid/zero :shapes] conj rect-id raw-id))
+ file' (cfm/migrate-file file {})]
+ (t/is (= :mask (get-in file' [:data :pages-index page-id :objects rect-id :svg-defs :a :tag])))
+ (t/is (= :g (get-in file' [:data :pages-index page-id :objects raw-id :content :tag])))))
+
+(t/deftest migration-0037-drops-map-valued-svg-attrs
+ ;; Files exported by old Penpot versions hold maps where an svg attr
+ ;; value should be, such as a `:filter` that once read
+ ;; `url(#filter_2)`. The closed `:svg-attrs` schema admits scalars
+ ;; only, so `migrate-file` must drop them or the file cannot be
+ ;; imported or opened. The values below are copied from the
+ ;; "Avataaars" and "Tutorial for beginners" templates.
+ (let [shape-id (uuid/next)
+ file (ctf/make-file {:name "Map-valued svg attrs"})
+ page-id (first (get-in file [:data :pages]))
+ shape (-> (cts/setup-shape {:id shape-id :type :rect :name "rect"})
+ (assoc :svg-attrs {:fillRule "evenodd"
+ :filter {(keyword "{") "{ \"url(#filter_2) nil}\""}
+ :mask {(keyword "url(-filter0-d-16837-160)") nil}
+ :style {:fill "red"
+ :filter {:a "b"}}}))
+ file (-> file
+ (assoc :migrations (disj cfm/available-migrations migration-id))
+ (update-in [:data :pages-index page-id :objects] assoc shape-id shape)
+ (update-in [:data :pages-index page-id :objects uuid/zero :shapes] conj shape-id))
+ file' (cfm/migrate-file file {})
+ attrs (get-in file' [:data :pages-index page-id :objects shape-id :svg-attrs])]
+ (t/is (= {:fill-rule "evenodd" :style {:fill "red"}} attrs)
+ "map values are dropped, scalars are kept")))
+
+(t/deftest migration-0037-turns-plain-map-svg-transform-into-matrix
+ ;; The svg defs renderer multiplies `:svg-transform` as a matrix; a
+ ;; plain map there renders as `matrix(NaN, ...)`.
+ (let [plain-id (uuid/next)
+ matrix-id (uuid/next)
+ page-id (uuid/next)
+ data {:pages-index
+ {page-id
+ {:objects
+ {plain-id (make-shape plain-id
+ {:type :path
+ :svg-transform {:a 1.0 :b 0.0 :c 0.0 :d 1.0 :e 10.0 :f 20.0}})
+ matrix-id (make-shape matrix-id
+ {:type :path
+ :svg-transform (gmt/matrix 2 0 0 2 0 0)})}}}}
+ data' (cfm/migrate-data data migration-id)
+ plain' (get-in data' [:pages-index page-id :objects plain-id :svg-transform])
+ matrix' (get-in data' [:pages-index page-id :objects matrix-id :svg-transform])]
+ (t/is (gmt/matrix? plain'))
+ (t/is (= (gmt/matrix 1 0 0 1 10 20) plain'))
+ (t/is (= (gmt/matrix 2 0 0 2 0 0) matrix') "matrices are left as they are")
+ (t/is (= data' (cfm/migrate-data data' migration-id)) "re-running is stable")))
+
+(t/deftest migration-0037-leaves-kebab-files-equal
+ (let [shape-id (uuid/next)
+ page-id (uuid/next)
+ data {:pages-index
+ {page-id
+ {:objects
+ {shape-id (make-shape shape-id
+ {:type :rect
+ :svg-attrs {:fill-rule "evenodd"
+ :stroke-width "2"}
+ :svg-defs {"g1" {:tag :linearGradient
+ :attrs {:id "g1"
+ :gradient-units "userSpaceOnUse"}
+ :content []}}})}}}}
+ data' (cfm/migrate-data data migration-id)]
+ (t/is (= data data') "kebab is a fixed point")))
+
+(t/deftest migration-0037-is-idempotent
+ (let [shape-id (uuid/next)
+ page-id (uuid/next)
+ data {:pages-index
+ {page-id
+ {:objects
+ {shape-id (camel-shape shape-id)}}}}
+ data' (cfm/migrate-data data migration-id)]
+ (t/is (= data' (cfm/migrate-data data' migration-id)))))
+
+(t/deftest migration-0037-normalizes-flat-svg-viewbox-to-rect
+ ;; Carried over from the withdrawn 0021: legacy files hold
+ ;; `:svg-viewbox` as a plain map instead of a rect record.
+ (let [shape-id (uuid/next)
+ page-id (uuid/next)
+ data {:pages-index
+ {page-id
+ {:objects
+ {shape-id (make-shape shape-id
+ {:type :rect
+ :svg-viewbox {:x 1 :y 2 :width 3 :height 4}})}}}}
+ data' (cfm/migrate-data data migration-id)
+ viewbox (get-in data' [:pages-index page-id :objects shape-id :svg-viewbox])]
+ (t/is (grc/rect? viewbox) "flat viewbox becomes a rect record")
+ (t/is (= 1 (:x viewbox)))
+ (t/is (= 2 (:y viewbox)))
+ (t/is (= 3 (:width viewbox)))
+ (t/is (= 4 (:height viewbox)))
+ (t/is (= data' (cfm/migrate-data data' migration-id))
+ "re-running is stable")))
+
+(t/deftest migration-0037-passes-non-maps-through
+ (let [shape-id (uuid/next)
+ page-id (uuid/next)
+ data {:pages-index
+ {page-id
+ {:objects
+ {shape-id (make-shape shape-id
+ {:type :rect
+ :svg-attrs "junk"
+ :svg-defs "junk"})}}}}
+ data' (cfm/migrate-data data migration-id)
+ shape (get-in data' [:pages-index page-id :objects shape-id])]
+ (t/is (= "junk" (:svg-attrs shape)) "non-maps reach validation untouched")
+ (t/is (= "junk" (:svg-defs shape)))))
+
+(t/deftest migration-0037-junk-viewbox-becomes-default-rect
+ ;; Unlike `:svg-attrs`/`:svg-defs` (passed through to validation),
+ ;; `:svg-viewbox` goes through `make-rect`, which does not throw on
+ ;; junk: it falls back to the default rect. Inherited from 0021.
+ (let [shape-id (uuid/next)
+ page-id (uuid/next)
+ data {:pages-index
+ {page-id
+ {:objects
+ {shape-id (make-shape shape-id
+ {:type :rect
+ :svg-viewbox "junk"})}}}}
+ data' (cfm/migrate-data data migration-id)
+ viewbox (get-in data' [:pages-index page-id :objects shape-id :svg-viewbox])]
+ (t/is (grc/rect? viewbox) "junk viewbox becomes a rect, not an error")
+ (t/is (= 0 (:x viewbox)))
+ (t/is (= 0.01 (:width viewbox)))))
+
+(t/deftest migration-0037-reruns-for-files-that-recorded-withdrawn-0021
+ ;; A file that recorded the withdrawn "0021-fix-shape-svg-attrs" (and
+ ;; everything else current) still needs migration: 0021 left
+ ;; `available-migrations`, which re-arms it for 0037.
+ (let [shape-id (uuid/next)
+ file (ctf/make-file {:name "Legacy camel svg attrs"})
+ page-id (first (get-in file [:data :pages]))
+ shape (-> (cts/setup-shape {:id shape-id :type :rect})
+ (assoc :svg-attrs {:fillRule "evenodd"}))
+ file (-> file
+ (assoc :migrations (-> cfm/available-migrations
+ (disj migration-id)
+ (conj "0021-fix-shape-svg-attrs")))
+ (assoc-in [:data :pages-index page-id :objects shape-id] shape))
+ file' (cfm/migrate-file file {})]
+ (t/is (cfm/need-migration? file) "new migration detected")
+ (t/is (not (cfm/need-migration? file')) "new migration recorded")
+ (t/is (contains? (:migrations file') migration-id) "migration id persisted")
+ (t/is (not (contains? cfm/available-migrations "0021-fix-shape-svg-attrs"))
+ "the withdrawn 0021 id stays out of available-migrations")
+ (t/is (= "evenodd"
+ (get-in file' [:data :pages-index page-id :objects shape-id :svg-attrs :fill-rule]))
+ "migration repaired file data before schema validation")))
+
+(t/deftest migration-0037-reruns-for-files-that-recorded-withdrawn-0035
+ ;; A file that recorded 0035 holds camelCase keys when a client that
+ ;; loaded it before 0035 ran saved them afterwards (#13107). 0035 also
+ ;; left `available-migrations`, so 0037 must run on it again.
+ (let [shape-id (uuid/next)
+ file (ctf/make-file {:name "Legacy camel svg attrs"})
+ page-id (first (get-in file [:data :pages]))
+ shape (-> (cts/setup-shape {:id shape-id :type :rect})
+ (assoc :svg-attrs {:fillRule "evenodd"}))
+ file (-> file
+ (assoc :migrations (-> cfm/available-migrations
+ (disj migration-id)
+ (conj "0035-fix-shape-svg-attrs-and-defs")))
+ (assoc-in [:data :pages-index page-id :objects shape-id] shape))
+ file' (cfm/migrate-file file {})]
+ (t/is (cfm/need-migration? file) "new migration detected")
+ (t/is (not (cfm/need-migration? file')) "new migration recorded")
+ (t/is (contains? (:migrations file') migration-id) "migration id persisted")
+ (t/is (not (contains? cfm/available-migrations "0035-fix-shape-svg-attrs-and-defs"))
+ "the withdrawn 0035 id stays out of available-migrations")
+ (t/is (= "evenodd"
+ (get-in file' [:data :pages-index page-id :objects shape-id :svg-attrs :fill-rule]))
+ "migration repaired file data before schema validation")))
diff --git a/common/test/common_tests/runner.cljc b/common/test/common_tests/runner.cljc
index 5243c00c61..d28469ee1e 100644
--- a/common/test/common_tests/runner.cljc
+++ b/common/test/common_tests/runner.cljc
@@ -17,6 +17,7 @@
[common-tests.attrs-test]
[common-tests.buffer-test]
[common-tests.colors-test]
+ [common-tests.data-macros-test]
[common-tests.data-test]
[common-tests.exceptions-test]
[common-tests.files-branch-merge-test]
@@ -28,6 +29,7 @@
[common-tests.files-migrations-0034-test]
[common-tests.files-migrations-0035-test]
[common-tests.files-migrations-0036-test]
+ [common-tests.files-migrations-0037-test]
[common-tests.files-migrations-test]
[common-tests.files-ref-paths-test]
[common-tests.files.comp-processors-test]
@@ -138,6 +140,7 @@
'common-tests.attrs-test
'common-tests.buffer-test
'common-tests.colors-test
+ 'common-tests.data-macros-test
'common-tests.data-test
'common-tests.exceptions-test
'common-tests.files-branch-merge-test
@@ -151,6 +154,7 @@
'common-tests.files-migrations-0034-test
'common-tests.files-migrations-0035-test
'common-tests.files-migrations-0036-test
+ 'common-tests.files-migrations-0037-test
'common-tests.files-migrations-test
'common-tests.files-ref-paths-test
'common-tests.files.comp-processors-test
diff --git a/common/test/common_tests/schema_test.cljc b/common/test/common_tests/schema_test.cljc
index 08e26295a6..3e36900906 100644
--- a/common/test/common_tests/schema_test.cljc
+++ b/common/test/common_tests/schema_test.cljc
@@ -7,6 +7,7 @@
(ns common-tests.schema-test
(:require
[app.common.data :as d]
+ [app.common.data.macros :as dm]
[app.common.schema :as sm]
[app.common.schema.generators :as sg]
[app.common.uuid :as uuid]
@@ -283,3 +284,43 @@
(t/testing "generate"
(t/is (true? (sm/validate ::sm/user-provided-uuid (sg/generate ::sm/user-provided-uuid)))))))
+
+(defn- catch-error
+ [f]
+ (try
+ (f)
+ nil
+ (catch #?(:clj Throwable :cljs :default) cause
+ cause)))
+
+(defn- lazy-resolution?
+ "Mirrors the build mode check performed by `app.common.data.macros/lazy-delay`."
+ []
+ #?(:clj false
+ :cljs dm/lazy-delay?))
+
+(t/deftest test-check-fn
+ (t/testing "returns the value when it is valid"
+ (let [check (sm/check-fn ::sm/uuid)]
+ (t/is (= uuid/zero (check uuid/zero)))))
+
+ (t/testing "raises with the explain data when the value is invalid"
+ (let [check (sm/check-fn ::sm/uuid :hint "invalid uuid"
+ :type :custom :code :custom-code)
+ cause (catch-error #(check "not-a-uuid"))
+ data (ex-data cause)]
+ (t/is (some? cause))
+ (t/is (= "invalid uuid" (ex-message cause)))
+ (t/is (= :custom (:type data)))
+ (t/is (= :custom-code (:code data)))
+ (t/is (= "invalid uuid" (:hint data)))
+ (t/is (map? (::sm/explain data)))))
+
+ (t/testing "resolves the schema definition according to the build mode"
+ ;; A schema that does not exist raises only when it is resolved, so this
+ ;; checks when the resolution happens: on `check-fn` on non production
+ ;; builds, on the first call of the returned function on production ones.
+ (let [check (catch-error #(sm/check-fn ::unknown-schema))]
+ (if (lazy-resolution?)
+ (t/is (some? (catch-error #(check ::anything))))
+ (t/is (some? check))))))
diff --git a/docker/devenv/Dockerfile b/docker/devenv/Dockerfile
index 55540a934f..6cd9b8cdfe 100644
--- a/docker/devenv/Dockerfile
+++ b/docker/devenv/Dockerfile
@@ -150,12 +150,12 @@ RUN set -eux; \
ARCH="$(dpkg --print-architecture)"; \
case "${ARCH}" in \
aarch64|arm64) \
- ESUM='9fa5bf865783c43840101fcdc3a5222b60daf4fe11f26806247bf807c08aba38'; \
- BINARY_URL='https://cdn.azul.com/zulu/bin/zulu27.28.101-ca-jdk27.0.0-linux_aarch64.tar.gz'; \
+ ESUM='18875b92ad2535230a13175f718933f7cc0593b4affa2dab442815ec245dacbb'; \
+ BINARY_URL='https://cdn.azul.com/zulu/bin/zulu27.30.201-ca-jdk27.0.1.1-linux_aarch64.tar.gz'; \
;; \
amd64|x86_64) \
- ESUM='ccbc15c4edbedfdcc03c2d29a2aa2c6daf9e6ffb4cda7dd5ece0fcbb37350267'; \
- BINARY_URL='https://cdn.azul.com/zulu/bin/zulu27.28.101-ca-jdk27.0.0-linux_x64.tar.gz'; \
+ ESUM='c4d7bf4c80a21e935187ecdcb111e4b2f14e1fdd18364974fe7f7da54b12bea0'; \
+ BINARY_URL='https://cdn.azul.com/zulu/bin/zulu27.30.201-ca-jdk27.0.1.1-linux_x64.tar.gz'; \
;; \
*) \
echo "Unsupported arch: ${ARCH}"; \
diff --git a/docker/devenv/files/tmux.conf b/docker/devenv/files/tmux.conf
index 062c1622c7..349aa43fc7 100644
--- a/docker/devenv/files/tmux.conf
+++ b/docker/devenv/files/tmux.conf
@@ -6,5 +6,8 @@ set -g mouse off
set -g history-limit 50000
setw -g mode-keys emacs
+set -g set-clipboard on
+set -g allow-passthrough on
+
bind-key -r C-n resize-pane -U
bind-key -r C-m resize-pane -D
diff --git a/docker/gitpod/Dockerfile b/docker/gitpod/Dockerfile
deleted file mode 100644
index 30ec7da8fe..0000000000
--- a/docker/gitpod/Dockerfile
+++ /dev/null
@@ -1,122 +0,0 @@
-FROM gitpod/workspace-postgres
-
-# Install custom tools, runtimes, etc.
-# For example "bastet", a command-line tetris clone:
-# RUN brew install bastet
-#
-# More information: https://www.gitpod.io/docs/config-docker/
-
-RUN set -ex; \
- brew install redis; \
- brew install imagemagick; \
- brew install ghostscript; \
- brew install mailhog; \
- brew install openldap; \
- brew install poppler; \
- sudo mkdir -p /var/log/nginx; \
- sudo chown gitpod:gitpod /var/log/nginx
-
-COPY files/nginx.conf /etc/nginx/nginx.conf
-COPY files/nginx-mime.types /etc/nginx/mime.types
-
-USER root
-
-ENV NODE_VERSION=v16.17.0 \
- CLOJURE_VERSION=1.11.1.1165 \
- CLJKONDO_VERSION=2022.09.08 \
- BABASHKA_VERSION=0.9.162 \
- LANG=en_US.UTF-8 \
- LC_ALL=en_US.UTF-8
-
-RUN set -ex; \
- useradd -m -g users -s /bin/bash penpot; \
- passwd penpot -d; \
- echo "penpot ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
-
-RUN set -ex; \
- apt-get -qq update; \
- apt-get -qqy install \
- gconf-service \
- libasound2 \
- libatk1.0-0 \
- libatk-bridge2.0-0 \
- libcairo2 \
- libcups2 \
- libdbus-1-3 \
- libexpat1 \
- libfontconfig1 \
- libgcc1 \
- libgconf-2-4 \
- libgdk-pixbuf2.0-0 \
- libglib2.0-0 \
- libgtk-3-0 \
- libnspr4 \
- libpango-1.0-0 \
- libpangocairo-1.0-0 \
- libx11-6 \
- libx11-xcb1 \
- libxcb1 \
- libxcomposite1 \
- libxcursor1 \
- libxdamage1 \
- libxext6 \
- libxfixes3 \
- libxi6 \
- libxrandr2 \
- libxrender1 \
- libxshmfence1 \
- libxss1 \
- libxtst6 \
- fonts-liberation \
- libappindicator1 \
- libnss3 \
- libgbm1 \
- ; \
- rm -rf /var/lib/apt/lists/*;
-
-RUN set -ex; \
- curl -LfsSo /tmp/openjdk.tar.gz https://github.com/AdoptOpenJDK/openjdk16-binaries/releases/download/jdk-16%2B36/OpenJDK16-jdk_x64_linux_hotspot_16_36.tar.gz; \
- mkdir -p /usr/lib/jvm/openjdk16; \
- cd /usr/lib/jvm/openjdk16; \
- tar -xf /tmp/openjdk.tar.gz --strip-components=1; \
- rm -rf /tmp/openjdk.tar.gz;
-
-# Install clojure cli
-RUN set -ex; \
- curl -LfsSo /tmp/clojure.sh https://download.clojure.org/install/linux-install-$CLOJURE_VERSION.sh; \
- chmod +x /tmp/clojure.sh; \
- /tmp/clojure.sh; \
- rm -rf /tmp/clojure.sh;
-
-# Install clj-kondo
-RUN set -ex; \
- curl -LfsSo /tmp/clj-kondo.zip https://github.com/borkdude/clj-kondo/releases/download/v$CLJKONDO_VERSION/clj-kondo-$CLJKONDO_VERSION-linux-amd64.zip; \
- cd /usr/local/bin; \
- unzip /tmp/clj-kondo.zip; \
- rm /tmp/clj-kondo.zip;
-
-# Install babashka
-RUN set -ex; \
- cd /tmp; \
- curl -LfsSo /tmp/babashka.tar.gz https://github.com/babashka/babashka/releases/download/v$BABASHKA_VERSION/babashka-$BABASHKA_VERSION-linux-amd64.tar.gz; \
- cd /usr/local/bin; \
- tar -xf /tmp/babashka.tar.gz; \
- rm -rf /tmp/babashka.tar.gz;
-
-USER gitpod
-
-ENV PATH="/usr/lib/jvm/openjdk16/bin:/usr/local/nodejs/bin:$PATH" \
- JAVA_HOME=/usr/lib/jvm/openjdk16 \
- PENPOT_SMTP_ENABLED=true \
- PENPOT_SMTP_HOST=localhost \
- PENPOT_SMTP_PORT=1025 \
- PENPOT_SMTP_USER= \
- PENPOT_SMTP_PASSWORD= \
- PENPOT_SMTP_SSL=false \
- PENPOT_SMTP_TLS=false \
- PENPOT_SMTP_DEFAULT_REPLY_TO=no-reply@example.com \
- PENPOT_SMTP_DEFAULT_FROM=no-reply@example.com \
- PENPOT_DATABASE_URI="postgresql://localhost/penpot" \
- PENPOT_REDIS_URI="redis://localhost/0"
-
-# TODO Retrieve OpenLDAP from rroemhild/docker-test-openldap
diff --git a/docker/gitpod/files/nginx-mime.types b/docker/gitpod/files/nginx-mime.types
deleted file mode 100644
index 91f6aba0ac..0000000000
--- a/docker/gitpod/files/nginx-mime.types
+++ /dev/null
@@ -1,100 +0,0 @@
-
-types {
- text/html html htm shtml;
- text/css css;
- text/xml xml;
- image/gif gif;
- image/jpeg jpeg jpg;
- application/javascript js;
- application/javascript mjs;
- application/atom+xml atom;
- application/rss+xml rss;
-
- text/mathml mml;
- text/plain txt;
- text/vnd.sun.j2me.app-descriptor jad;
- text/vnd.wap.wml wml;
- text/x-component htc;
-
- image/avif avif;
- image/png png;
- image/svg+xml svg svgz;
- image/tiff tif tiff;
- image/vnd.wap.wbmp wbmp;
- image/webp webp;
- image/x-icon ico;
- image/x-jng jng;
- image/x-ms-bmp bmp;
-
- font/woff woff;
- font/woff2 woff2;
-
- application/java-archive jar war ear;
- application/json json;
- application/mac-binhex40 hqx;
- application/msword doc;
- application/pdf pdf;
- application/postscript ps eps ai;
- application/rtf rtf;
- application/vnd.apple.mpegurl m3u8;
- application/vnd.google-earth.kml+xml kml;
- application/vnd.google-earth.kmz kmz;
- application/vnd.ms-excel xls;
- application/vnd.ms-fontobject eot;
- application/vnd.ms-powerpoint ppt;
- application/vnd.oasis.opendocument.graphics odg;
- application/vnd.oasis.opendocument.presentation odp;
- application/vnd.oasis.opendocument.spreadsheet ods;
- application/vnd.oasis.opendocument.text odt;
- application/vnd.openxmlformats-officedocument.presentationml.presentation
- pptx;
- application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
- xlsx;
- application/vnd.openxmlformats-officedocument.wordprocessingml.document
- docx;
- application/vnd.wap.wmlc wmlc;
- application/wasm wasm;
- application/x-7z-compressed 7z;
- application/x-cocoa cco;
- application/x-java-archive-diff jardiff;
- application/x-java-jnlp-file jnlp;
- application/x-makeself run;
- application/x-perl pl pm;
- application/x-pilot prc pdb;
- application/x-rar-compressed rar;
- application/x-redhat-package-manager rpm;
- application/x-sea sea;
- application/x-shockwave-flash swf;
- application/x-stuffit sit;
- application/x-tcl tcl tk;
- application/x-x509-ca-cert der pem crt;
- application/x-xpinstall xpi;
- application/xhtml+xml xhtml;
- application/xspf+xml xspf;
- application/zip zip;
-
- application/octet-stream bin exe dll;
- application/octet-stream deb;
- application/octet-stream dmg;
- application/octet-stream iso img;
- application/octet-stream msi msp msm;
-
- audio/midi mid midi kar;
- audio/mpeg mp3;
- audio/ogg ogg;
- audio/x-m4a m4a;
- audio/x-realaudio ra;
-
- video/3gpp 3gpp 3gp;
- video/mp2t ts;
- video/mp4 mp4;
- video/mpeg mpeg mpg;
- video/quicktime mov;
- video/webm webm;
- video/x-flv flv;
- video/x-m4v m4v;
- video/x-mng mng;
- video/x-ms-asf asx asf;
- video/x-ms-wmv wmv;
- video/x-msvideo avi;
-}
diff --git a/docker/gitpod/files/nginx.conf b/docker/gitpod/files/nginx.conf
deleted file mode 100644
index ca403adfd2..0000000000
--- a/docker/gitpod/files/nginx.conf
+++ /dev/null
@@ -1,145 +0,0 @@
-# This Nginx config file is optional.
-# If you don't have this file, the Nginx config from the workspace-full docker image will be taken.
-
-worker_processes auto;
-pid /var/run/nginx/nginx.pid;
-include /etc/nginx/modules-enabled/*.conf;
-
-env GITPOD_REPO_ROOT;
-
-events {
- worker_connections 768;
- # multi_accept on;
-}
-
-http {
- sendfile on;
- tcp_nopush on;
- tcp_nodelay on;
- keepalive_timeout 65;
- types_hash_max_size 2048;
- # server_tokens off;
-
- # server_names_hash_bucket_size 64;
- # server_name_in_redirect off;
-
- include /etc/nginx/mime.types;
- default_type application/octet-stream;
-
- access_log /var/log/nginx/access.log;
- error_log /var/log/nginx/error.log;
-
- gzip on;
-
- gzip_vary on;
- gzip_proxied any;
- gzip_comp_level 3;
- gzip_buffers 16 8k;
- gzip_http_version 1.1;
-
- gzip_types text/plain text/css text/javascript application/javascript application/json application/transit+json;
-
- map $http_upgrade $connection_upgrade {
- default upgrade;
- '' close;
- }
-
- # include /etc/nginx/sites-enabled/*;
-
- server {
- set_by_lua $gitpod_repo_root 'return os.getenv("GITPOD_REPO_ROOT")';
-
- listen 0.0.0.0:3449 default_server;
- server_name _;
-
- client_max_body_size 5M;
- charset utf-8;
-
- proxy_http_version 1.1;
- proxy_set_header Host $http_host;
- proxy_set_header X-Real-IP $remote_addr;
- proxy_set_header X-Scheme $scheme;
- proxy_set_header X-Forwarded-Proto $scheme;
- proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-
- resolver 8.8.8.8;
-
- etag off;
-
- root $gitpod_repo_root/frontend/resources/public;
-
- location @handle_redirect {
- set $redirect_uri "$upstream_http_location";
- set $redirect_host "$upstream_http_x_host";
- set $redirect_cache_control "$upstream_http_cache_control";
-
- proxy_buffering off;
-
- proxy_set_header Host "$redirect_host";
- proxy_hide_header etag;
- proxy_hide_header x-amz-id-2;
- proxy_hide_header x-amz-request-id;
- proxy_hide_header x-amz-meta-server-side-encryption;
- proxy_hide_header x-amz-server-side-encryption;
- proxy_pass $redirect_uri;
-
- add_header x-internal-redirect "$redirect_uri";
- add_header x-cache-control "$redirect_cache_control";
- add_header cache-control "$redirect_cache_control";
- }
-
- location /assets {
- proxy_pass http://127.0.0.1:6060/assets;
- recursive_error_pages on;
- proxy_intercept_errors on;
- error_page 301 302 307 = @handle_redirect;
- }
-
- location /internal/assets {
- internal;
- alias $gitpod_repo_root/backend/assets;
- add_header x-internal-redirect "$upstream_http_x_accel_redirect";
- }
-
- location /api {
- proxy_pass http://127.0.0.1:6060/api;
- }
-
- location /webhooks {
- proxy_pass http://127.0.0.1:6060/webhooks;
- }
-
- location /dbg {
- proxy_pass http://127.0.0.1:6060/dbg;
- }
-
- location /export {
- proxy_pass http://127.0.0.1:6061;
- }
-
- location /playground {
- alias $gitpod_repo_root/experiments/;
- add_header Cache-Control "no-cache, max-age=0";
- autoindex on;
- }
-
- location /ws/notifications {
- proxy_set_header Upgrade $http_upgrade;
- proxy_set_header Connection 'upgrade';
- proxy_pass http://127.0.0.1:6060/ws/notifications;
- }
-
- location / {
- add_header Cache-Control "no-cache, max-age=0";
-
- location ~ ^/(/|css|fonts|images|js|wasm) {
- }
-
- location ~ ^/[^/]+/(.*)$ {
- return 301 " /404";
- }
-
- try_files $uri /index.html$is_args$args =404;
- }
- }
-}
diff --git a/docker/gitpod/files/postgresql_init.sql b/docker/gitpod/files/postgresql_init.sql
deleted file mode 100644
index 1766a643d2..0000000000
--- a/docker/gitpod/files/postgresql_init.sql
+++ /dev/null
@@ -1,3 +0,0 @@
-CREATE DATABASE penpot;
-CREATE USER penpot PASSWORD 'penpot';
-ALTER ROLE penpot SUPERUSER;
diff --git a/docker/images/Dockerfile.backend b/docker/images/Dockerfile.backend
index 1e287b7080..c2f4c5e6b7 100644
--- a/docker/images/Dockerfile.backend
+++ b/docker/images/Dockerfile.backend
@@ -27,12 +27,12 @@ RUN set -eux; \
ARCH="$(dpkg --print-architecture)"; \
case "${ARCH}" in \
aarch64|arm64) \
- ESUM='8ff07320636ef5b30f0d7e16687932df52255c3ef3ffd920a6aae3723511ce7f'; \
- BINARY_URL='https://cdn.azul.com/zulu/bin/zulu26.32.203-ca-jdk26.0.2.1-linux_aarch64.tar.gz'; \
+ ESUM='18875b92ad2535230a13175f718933f7cc0593b4affa2dab442815ec245dacbb'; \
+ BINARY_URL='https://cdn.azul.com/zulu/bin/zulu27.30.201-ca-jdk27.0.1.1-linux_aarch64.tar.gz'; \
;; \
amd64|x86_64) \
- ESUM='c6b810ca659e3293205444f47ac4af163396fd6d025c51fd5f767e628b2ceede'; \
- BINARY_URL='https://cdn.azul.com/zulu/bin/zulu26.32.203-ca-jdk26.0.2.1-linux_x64.tar.gz'; \
+ ESUM='c4d7bf4c80a21e935187ecdcb111e4b2f14e1fdd18364974fe7f7da54b12bea0'; \
+ BINARY_URL='https://cdn.azul.com/zulu/bin/zulu27.30.201-ca-jdk27.0.1.1-linux_x64.tar.gz'; \
;; \
*) \
echo "Unsupported arch: ${ARCH}"; \
diff --git a/docker/images/Dockerfile.frontend b/docker/images/Dockerfile.frontend
index 191f0ccda6..1be6ee9680 100644
--- a/docker/images/Dockerfile.frontend
+++ b/docker/images/Dockerfile.frontend
@@ -1,4 +1,4 @@
-FROM dhi.io/nginx:1.31.8-debian13-dev@sha256:2168f2f90d8dd61baf5ac0359d92f601f988ba7d7ef669dc778cd7a887130295
+FROM dhi.io/nginx:1.31.8-debian13-dev@sha256:5f75855fe078ae96027706da06716fbe1c3dbb618fc603e61894e1b41adc56d7
LABEL maintainer="Penpot