diff --git a/.agents/skills/code-review-criteria/SKILL.md b/.agents/skills/code-review-criteria/SKILL.md index 398aa440ee..b95c563a57 100644 --- a/.agents/skills/code-review-criteria/SKILL.md +++ b/.agents/skills/code-review-criteria/SKILL.md @@ -13,19 +13,17 @@ Multi-dimensional code review with quality gates. Every change gets reviewed bef ## When to Use -- The reviewer subagent of the `review-code` flow loads this skill to perform - the review of a code change. -- To review code, always go through the `review-code` flow — never load this - skill directly for that. This is the criteria reference, not the flow. +- The reviewer subagent of the `review-code` flow loads this skill to perform the review of a code change. +- To review code, always go through the `review-code` flow — never load this skill directly for that. This is the criteria reference, not the flow. ## Core Principles These principles underpin every axis. When in doubt, default to them. -- **DRY (Don't Repeat Yourself):** Every piece of knowledge has one authoritative representation. If the same logic appears in two places, extract it into a shared helper, model, or type. Reviewers: flag duplicated logic as a required change — it's not "just similar," it's drift that will diverge. -- **KISS (Keep It Simple, Stupid):** The simplest solution that works is the best solution. Complexity must earn its place. Reviewers: if you need more than one sentence to explain what a piece of code does, it's too complex — push for simplification before merge. -- **YAGNI (You Aren't Gonna Need It):** Don't add abstractions, hooks, or generalizations for hypothetical future use cases. Generalize on the third occurrence, not the first. Reviewers: delete speculative generality. -- **Don't invent problems:** Do not manufacture issues to produce more feedback. Every finding must be a real risk, a real readability barrier, or a real architectural concern — not a hypothetical or a stylistic preference disguised as a problem. +- **DRY (Don't Repeat Yourself):** Every piece of knowledge has one authoritative representation. Flag duplicated logic as a required change — it is not "just similar", it is drift that will diverge. +- **KISS (Keep It Simple, Stupid):** The simplest solution that works is the best solution. Complexity must earn its place: if it needs more than a sentence to explain, push for simplification before merge. +- **YAGNI (You Aren't Gonna Need It):** No abstractions, hooks or generalizations for hypothetical future use cases. Generalize on the third occurrence, not the first. +- **Don't invent problems:** Every finding must be a real risk, a real readability barrier, or a real architectural concern — not a hypothetical or a stylistic preference disguised as a problem. ## The Five-Axis Review @@ -38,7 +36,7 @@ Does the code do what it claims to do? - Does it match the spec or task requirements? - Are edge cases handled (null, empty, boundary values)? - Are error paths handled (not just the happy path)? -- Does it pass all tests? Are the tests actually testing the right things? +- Do the tests cover the changed behavior, including its edge cases? - Are there off-by-one errors, race conditions, or state inconsistencies? ### 2. Readability & Simplicity @@ -47,10 +45,7 @@ Can another engineer (or agent) understand this code without the author explaini - Are names descriptive and consistent with project conventions? (No `temp`, `data`, `result` without context) - Is the control flow straightforward (avoid nested ternaries, deep callbacks)? -- Are there any "clever" tricks that should be simplified? -- **KISS check:** Is this the simplest approach that solves the problem? A 20-line straightforward function beats a 5-line clever one that requires a comment to explain. -- Could this be done in fewer lines? (1000 lines where 100 suffice is a failure) -- Are abstractions earning their complexity? (Don't generalize until the third use case) +- Are there "clever" tricks that should be simplified? Could this be done in fewer lines? (1000 lines where 100 suffice is a failure) - Is a new conditional bolted onto an unrelated flow? Push the logic into its own helper, state, or policy. - Do repeated conditionals on the same shape appear? They signal a missing model or dispatcher. - Are there dead code artifacts: no-op variables, backwards-compat shims, or `// removed` comments? @@ -60,14 +55,13 @@ Can another engineer (or agent) understand this code without the author explaini Does the change fit the system's design? - Does it follow existing patterns or introduce a new one? If new, is it justified? -- Does it maintain clean module boundaries? -- **DRY check:** Is there existing code that does the same thing? Reuse the canonical helper instead of writing a near-duplicate. If two branches do nearly the same thing, collapse them. -- Are dependencies flowing in the right direction (no circular dependencies)? +- Does it keep clean module boundaries, with dependencies flowing the right way (no cycles)? - Is the abstraction level appropriate (not over-engineered, not too coupled)? -- Does this refactor reduce complexity or just relocate it? Count the concepts a reader must hold. Prefer the restructuring that makes whole branches disappear over one that re-centralizes the same logic. Prefer deleting an abstraction to polishing it. +- Is there existing code that already does this? Reuse the canonical helper instead of a near-duplicate, and collapse branches that do nearly the same thing. - Is feature-specific logic leaking into a shared or general-purpose module? - Are type boundaries explicit? Question gratuitous `any`/`unknown`/optional/casts and silent fallbacks. -- **Structural remedies:** When you flag a problem, propose the move — not just the problem. Replace conditionals with dispatchers, collapse duplicate branches, separate orchestration from business logic, extract helpers, split large files. Prefer the remedy that removes moving pieces over one that spreads the same complexity around. +- Does the change reduce complexity or relocate it? Count the concepts a reader must hold: prefer the restructuring that makes whole branches disappear, and deleting an abstraction over polishing it. +- **Give the remedy, not only the problem:** replace conditionals with dispatchers, separate orchestration from business logic, extract helpers, split large files. Prefer the remedy that removes moving pieces over one that spreads the same complexity around. ### 4. Security @@ -111,7 +105,17 @@ For each finding, describe the circumstances under which it could fail: specific Lead with what matters: correctness and security first, then structural issues, then everything else. A few high-conviction comments beat a long list. -5. **Verify the verification** — What tests were run? Did the build pass? Was the change tested manually? Screenshots for UI changes? +5. **Check the evidence, do not re-run it** — Which tests cover the change? What did the author run, and what did CI report? Never re-run the full suite or a build yourself: the author's run and CI already cover that, and repeating it only lengthens the review. Run at most the single test file that settles a specific doubt. When no checks have been run at all (a change you did not see tested), run the module's checks before the verdict. + +## Change Sizing + +Small, focused changes are easier to review, faster to merge, and safer to deploy. Flag the size when a diff exceeds ~300 lines, or when it grows a file past ~1000 total lines, and name the remedy: split it, and separate refactoring from feature work. This is a finding about the change, not work to do inside the review. + +## Dependencies + +Before accepting a new dependency: does the existing stack already solve it, how large is the bundle impact, is it maintained, does `npm audit` report anything, is the license compatible. Prefer the standard library and existing utilities — every dependency is a liability. + +For upgrades: read the changelog, not just the version number; one dependency per change; a green suite before and after; review the lockfile diff and never hand-edit it. Supply-chain triage follows `security-and-hardening`. ## Review Output @@ -151,96 +155,18 @@ Choose one: List the finding IDs the verdict depends on (e.g. "Request changes: R1, R2"). -## Change Sizing - -Small, focused changes are easier to review, faster to merge, and safer to deploy. - -``` -~100 lines changed → Good. Reviewable in one sitting. -~300 lines changed → Acceptable if it's a single logical change. -~1000 lines changed → Too large. Split it. -``` - -**Watch file size, not just diff size.** Around 1000 *total* lines in a single file is a common inspection signal. When a change materially grows an already-large file, decompose first. - -**Splitting strategies:** - -| Strategy | How | When | -|----------|-----|------| -| **Stack** | Submit a small change, start the next one based on it | Sequential dependencies | -| **By file group** | Separate changes for groups needing different reviewers | Cross-cutting concerns | -| **Horizontal** | Create shared code/stubs first, then consumers | Layered architecture | -| **Vertical** | Break into smaller full-stack slices of the feature | Feature work | - -**Separate refactoring from feature work.** A change that refactors and adds new behavior is two changes — submit them separately. - -## Change Descriptions - -- **First line:** Short, imperative, standalone. "Delete the FizzBuzz RPC" not "Deleting the FizzBuzz RPC." -- **Body:** What is changing and why. Include context and reasoning not visible in the code itself. -- **Anti-patterns:** "Fix bug," "Fix build," "Add patch," "Phase 1." - -## Dependencies - -Before adding any dependency: - -1. Does the existing stack solve this? (Often it does.) -2. How large is the dependency? (Check bundle impact.) -3. Is it actively maintained? (Check last commit, open issues.) -4. Does it have known vulnerabilities? (`npm audit`) -5. What's the license? (Must be compatible with the project.) - -**Rule:** Prefer standard library and existing utilities over new dependencies. Every dependency is a liability. - -**Upgrading dependencies:** - -- Read the changelog, not just the version number. Semver is a promise the maintainer may not have kept. -- One dependency per change. When a bulk bump breaks the build, you've lost which package did it. -- Let the tests decide — a green suite before *and* after, not just "it installed." -- Review the lockfile diff, not just `package.json`. Commit it and never hand-edit it. - -For supply-chain risk triage, follow the `security-and-hardening` skill. - -## Common Rationalizations - -| Rationalization | Reality | -|---|---| -| "It works, that's good enough" | Working code that's unreadable, insecure, or architecturally wrong creates debt that compounds. | -| "I wrote it, so I know it's correct" | Authors are blind to their own assumptions. Every change benefits from another set of eyes. | -| "We'll clean it up later" | Later never comes. The review is the quality gate — use it. | -| "AI-generated code is probably fine" | AI code needs more scrutiny, not less. It's confident and plausible, even when wrong. | -| "The tests pass, so it's good" | Tests are necessary but not sufficient. They don't catch architecture, security, or readability problems. | -| "The refactor makes it cleaner" | Relocating complexity isn't reducing it. If the reader still holds the same number of concepts, the structure didn't improve. | -| "It's only a small addition to this file" | Small diffs still push files past healthy size and bolt branches onto unrelated flows. | -| "It's just a version bump" | A bump is a behavior change you didn't write. Read the changelog. | -| "I'll upgrade everything in one PR" | A bulk bump hides which package broke the build. One per change. | -| "It's duplicated but it's only two places" | Two becomes three becomes five. Extract now, before the copies diverge. | -| "The abstraction is future-proof" | YAGNI. Delete speculative generality — generalize on the third occurrence, not the first. | -| "It's clever but efficient" | Cleverness is a readability tax. If it needs a comment to understand, simplify it. | - ## Red Flags -- PRs merged without any review -- Review that only checks if tests pass (ignoring other axes) -- "LGTM" without evidence of actual review -- Security-sensitive changes without security-focused review -- Large PRs that are "too big to review properly" (split them) -- No regression tests with bug fix PRs -- Accepting "I'll fix it later" — it never happens -- A refactor that moves code around without reducing the number of concepts a reader must hold -- New conditionals scattered into unrelated code paths (a missing abstraction) -- A bespoke helper that duplicates an existing canonical one -- A bulk "bump dependencies" PR with no changelog review +- A change merged without any review. +- A review that only checks whether the tests pass. +- Security-sensitive changes without a security-focused review. +- A change too large to review properly: split it. ## Verification -Before emitting the verdict, verify the change as it stands. This is the reviewer's own due diligence — it covers the state of the code at review time, not the later resolution of findings (fixing findings is the author's job; confirming them is a new review): +Before emitting the verdict, confirm the evidence the review rests on. This covers the state of the code at review time, not the later fixing of findings: -- [ ] Tests pass — run them yourself, don't trust the claim -- [ ] Build succeeds -- [ ] The verification story is documented (what changed, how it was verified) -- [ ] Dependency upgrades reviewed against changelog, isolated per package, verified by green suite - -## See Also - -- For detailed security review guidance, see `security-and-hardening` +- [ ] The suite is green, and the evidence is the author's run or CI — not your own re-run. Say which one you relied on. +- [ ] New behavior has tests that cover the edge cases in the diff. +- [ ] The verification story is stated: what changed and how it was verified. +- [ ] Dependency changes were reviewed against the changelog and the lockfile diff. diff --git a/.agents/skills/create-commit/SKILL.md b/.agents/skills/create-commit/SKILL.md index dd85191fd9..264f870e03 100644 --- a/.agents/skills/create-commit/SKILL.md +++ b/.agents/skills/create-commit/SKILL.md @@ -63,6 +63,8 @@ subject/body limits, and the `AI-assisted-by` trailer. Follow it exactly. - Do not pass `--author`. Author identity comes from the local git config. - Do not amend a commit you did not create in this session, unless explicitly asked. - Do not bypass pre-commit hooks (`--no-verify`) unless explicitly asked. -- Never add a `Co-authored-by:` trailer. It is forbidden and fails `./scripts/check-commit`. +- Add `Co-authored-by:` only for a human co-author. AI providers, agents and + bots are rejected by `./scripts/check-commit`; for AI participation use the + `AI-assisted-by:` trailer. - Do not add untracked files that were not created in this session. - Do not skip the `scripts/check-commit` verification step (Iron Rule 4). diff --git a/.agents/skills/review-code/SKILL.md b/.agents/skills/review-code/SKILL.md index 21e7a7c7fd..65c811e71d 100644 --- a/.agents/skills/review-code/SKILL.md +++ b/.agents/skills/review-code/SKILL.md @@ -22,8 +22,20 @@ Act as a senior software engineer and perform a thorough code review. diff, a commit range, a branch, a PR (number or URL), or specific files. If the target is ambiguous, ask before reviewing. 2. Delegate the review to the `general` subagent (via the task tool), unless the - user specifies another agent. Include in the prompt the - **`code-review-criteria`** skill name and all user context. + user specifies another agent. The prompt carries the **`code-review-criteria`** + skill name, all user context, and these requirements: + - Load `code-review-criteria` and follow its process and output format. + - Read `AGENTS.md` (if present) and the module memories it points to for the + code being reviewed. + - Return the COMPLETE review, verbatim, exactly as the skill instructs it to + be produced. Never summarize it. + - Read-only: modify no file and create no commit. Reviewing never writes. + - Do not invent problems: every finding must be real and actionable. + - Missing tests are an issue, not a suggestion — a severity-tagged finding. + - Prioritize by impact: one structural issue outweighs ten nits, and "this + could be better" is not a finding — say why and how. + - Skip generated files, lockfile-only changes and unrelated modifications + unless they introduce security risks. 3. When the subagent returns, output the review to the user verbatim. Do not summarize it and do not act on its findings. 4. Right after the review, suggest how to proceed based on the findings. These @@ -44,43 +56,18 @@ the findings. Once the user explicitly states the review is done (or moves on to a different task), this rule no longer applies and you act as a normal build agent again. -## Instructions for the subagent - -1. Load the **`code-review-criteria`** skill and follow its process and output - format. -2. Read `AGENTS.md` (if present) and follow its instructions for finding and - reading all related testing documentation from memories before reviewing. -3. Return in your final message the COMPLETE review, verbatim, exactly as the - skill instructs it to be produced. Do not summarize it — include the full - structured review. - -### Strong rules for the subagent - -1. Do not invent problems. Every finding must be real and actionable. -2. Read-only: do not modify any file and do not create a commit — reviewing - never writes. -3. Be specific and constructive. "This could be better" is not helpful — explain - why and how. -4. Prioritize by impact. One structural issue outweighs ten nits. -5. Missing tests are an issue, not a suggestion. Report as a severity-tagged - finding — never as a recommendation. -6. Skip generated files, lockfile-only changes, and unrelated modifications - unless they introduce security risks. - ## Posting as a GitHub comment -Only post the review as a comment when the user asks for it explicitly. Keep -it in the session otherwise. - -When posting: PR comments go in English, with this minimal structure: +Post only when the user asks for it explicitly; otherwise keep the review in the +session. PR comments go in English, with this minimal structure: 1. **Intro line** — one sentence: what was found and the overall tone. No greetings. -2. **`## Main issues`** — one subsection per required finding: what breaks, - the failure circumstances, and the minimal fix (code snippet when useful). - A missing test belongs here, not under minor suggestions. -3. **`## Minor suggestions`** — one bullet per minor finding or suggestion: - file/function, problem, concrete remedy. +2. **`## Main issues`** — one subsection per required finding: what breaks, the + failure circumstances, and the minimal fix (code snippet when useful). A + missing test belongs here, not under minor suggestions. +3. **`## Minor suggestions`** — one bullet per minor finding: file/function, + problem, concrete remedy. 4. **Closing** — optional and short. Cut it when it adds nothing. Reference every finding by its ID (`R1`, `M1`, `S1`, … from diff --git a/.github/ISSUE_TEMPLATE/bug-report.yml b/.github/ISSUE_TEMPLATE/bug.yml similarity index 90% rename from .github/ISSUE_TEMPLATE/bug-report.yml rename to .github/ISSUE_TEMPLATE/bug.yml index 9031969b8e..af5e0e3418 100644 --- a/.github/ISSUE_TEMPLATE/bug-report.yml +++ b/.github/ISSUE_TEMPLATE/bug.yml @@ -1,7 +1,7 @@ -description: Create a report to help us improve -name: Bug report -type: Bug +description: An unexpected problem or behavior labels: ["needs triage"] +name: Bug +type: Bug body: - type: markdown @@ -15,7 +15,7 @@ body: - type: textarea attributes: label: Steps To Reproduce - description: Steps to reproduce the behavior. + description: Steps to reproduce the behavior placeholder: | Steps to reproduce the behavior: 1. Go to '...' @@ -26,22 +26,22 @@ body: - type: textarea id: expected attributes: - description: A clear and concise description of what you expected to happen. label: Expected behavior + description: A clear and concise description of what you expected to happen validations: required: true - type: textarea id: actual attributes: - description: A clear and concise description of what happens instead; what the bug is. label: Actual behavior + description: A clear and concise description of what happens instead; what the bug is validations: required: true - type: textarea id: screenshots attributes: - description: If applicable, add screenshots to help explain your problem. label: Screenshots or video + description: If applicable, add screenshots to help explain your problem - type: textarea id: desktop attributes: @@ -87,4 +87,4 @@ body: id: additional-context attributes: label: Additional context - description: Any other context about the problem. + description: Add any other context or screenshots about the bug here diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml index 0086358db1..3ba13e0cec 100644 --- a/.github/ISSUE_TEMPLATE/config.yml +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -1 +1 @@ -blank_issues_enabled: true +blank_issues_enabled: false diff --git a/.github/ISSUE_TEMPLATE/enhancement.yml b/.github/ISSUE_TEMPLATE/enhancement.yml new file mode 100644 index 0000000000..ee4b3249e0 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/enhancement.yml @@ -0,0 +1,38 @@ +description: Improvement to existing behavior +labels: ["needs triage"] +name: "Enhancement" +type: Enhancement + +body: + - type: markdown + attributes: + value: | + ## Before you start + + Please search our [existing issues](https://github.com/penpot/penpot/issues) and open [pull requests](https://github.com/penpot/penpot/pulls) to lessen the change of filing duplicate issues or feature requests. Thank you. + + --- + - type: textarea + id: problem + attributes: + label: Is your feature request related to a problem? Please describe + description: A clear and concise description of what the problem is. Ex. I'm always frustrated when (...) + validations: + required: true + - type: textarea + id: solution + attributes: + label: Describe the solution you'd like + description: A clear and concise description of what you want to happen + validations: + required: true + - type: textarea + id: alternatives + attributes: + label: Describe alternatives you've considered + description: A clear and concise description of any alternative solutions or features you've considered + - type: textarea + id: additional-context + attributes: + label: Additional context + description: Add any other context or screenshots about the enhancement here diff --git a/.github/ISSUE_TEMPLATE/epic.yml b/.github/ISSUE_TEMPLATE/epic.yml new file mode 100644 index 0000000000..37dd3cff17 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/epic.yml @@ -0,0 +1,26 @@ +description: An epic is a large user story that cannot be delivered as defined within a single iteration +labels: ["needs triage"] +name: "EPIC" +type: EPIC + +body: + - type: markdown + attributes: + value: | + ## Before you start + + Please search our [existing issues](https://github.com/penpot/penpot/issues) and open [pull requests](https://github.com/penpot/penpot/pulls) to lessen the change of filing duplicate issues or feature requests. Thank you. + + --- + - type: textarea + id: description + attributes: + label: Epic description + description: Tell us which epic feature do you want to discuss + validations: + required: true + - type: textarea + id: additional-context + attributes: + label: Additional context + description: Add any other context or screenshots about the feature request here diff --git a/.github/ISSUE_TEMPLATE/feature-request.yml b/.github/ISSUE_TEMPLATE/feature.yml similarity index 80% rename from .github/ISSUE_TEMPLATE/feature-request.yml rename to .github/ISSUE_TEMPLATE/feature.yml index 17f451ae3f..2ba0eb4dbd 100644 --- a/.github/ISSUE_TEMPLATE/feature-request.yml +++ b/.github/ISSUE_TEMPLATE/feature.yml @@ -1,7 +1,7 @@ -description: Suggest an idea for this project. +description: Suggest an idea for this project labels: ["needs triage"] name: "Feature request" -type: Enhancement +type: Feature body: - type: markdown @@ -15,24 +15,24 @@ body: - type: textarea id: problem attributes: + label: Is your feature request related to a problem? Please describe description: A clear and concise description of what the problem is. Ex. I'm always frustrated when (...) - label: Is your feature request related to a problem? Please describe. validations: required: true - type: textarea id: solution attributes: - description: A clear and concise description of what you want to happen. - label: Describe the solution you'd like. + label: Describe the solution you'd like + description: A clear and concise description of what you want to happen validations: required: true - type: textarea id: alternatives attributes: - label: Describe alternatives you've considered. - description: A clear and concise description of any alternative solutions or features you've considered. + label: Describe alternatives you've considered + description: A clear and concise description of any alternative solutions or features you've considered - type: textarea id: additional-context attributes: label: Additional context - description: Add any other context or screenshots about the feature request here. + description: Add any other context or screenshots about the feature request here diff --git a/.github/ISSUE_TEMPLATE/task.yml b/.github/ISSUE_TEMPLATE/task.yml new file mode 100644 index 0000000000..c82e98a04a --- /dev/null +++ b/.github/ISSUE_TEMPLATE/task.yml @@ -0,0 +1,26 @@ +description: A specific piece of work +labels: [] +name: "Task" +type: Task + +body: + - type: markdown + attributes: + value: | + ## Before you start + + Please search our [existing issues](https://github.com/penpot/penpot/issues) and open [pull requests](https://github.com/penpot/penpot/pulls) to lessen the change of filing duplicate issues or feature requests. Thank you. + + --- + - type: textarea + id: task + attributes: + description: A clear and concise description of the task + label: Please describe + validations: + required: true + - type: textarea + id: additional-context + attributes: + label: Additional context + description: Add any other context or screenshots about the task here diff --git a/.github/workflows/auto-label.yml b/.github/workflows/auto-label.yml index 65dcc51875..ff4f55f096 100644 --- a/.github/workflows/auto-label.yml +++ b/.github/workflows/auto-label.yml @@ -43,6 +43,9 @@ jobs: // === 1. CONFIGURATION === const PROJECT_NUMBER = 8; // <--- Replace with your project board number const IS_ORG = true; // <--- Set to false if this is a personal project, true if an organization + const STATUS_FIELD_ID = 'PVTSSF_lADOAcyBPM4BV2pqzhRPY48'; + const STATUS_BACKLOG = 'f75ad846'; + const STATUS_TECH_REVIEW = 'b0f45db9'; const OWNER = context.repo.owner; const REPO = context.repo.repo; @@ -88,6 +91,65 @@ jobs: } `; - await github.graphql(addToProjectMutation, { projectId, contentId }); + let itemId; + try { + const addRes = await github.graphql(addToProjectMutation, { projectId, contentId }); + itemId = addRes.addProjectV2ItemById.item.id; + } catch (error) { + const messages = [ + error && error.message, + ...(Array.isArray(error && error.errors) + ? error.errors.map(e => e && e.message) + : []) + ].filter(Boolean).join('\n'); + if (!messages.includes('Content already exists in this project')) { + throw error; + } + console.log('Item already in project, fetching existing item...'); + const existing = await github.graphql( + `query($contentId: ID!) { + node(id: $contentId) { + ... on Issue { + projectItems(first: 50) { + nodes { id project { id } } + } + } + ... on PullRequest { + projectItems(first: 50) { + nodes { id project { id } } + } + } + } + }`, + { contentId } + ); + const nodes = (existing.node && existing.node.projectItems && existing.node.projectItems.nodes) || []; + const match = nodes.find(n => n.project.id === projectId); + if (!match) { + throw new Error(`Item already in a project, but not found in project ${projectId}`); + } + itemId = match.id; + } + + // === 4. SET PROJECT STATUS === + const statusOptionId = isPR ? STATUS_TECH_REVIEW : STATUS_BACKLOG; + const statusName = isPR ? 'Tech Review' : 'Backlog'; + console.log(`Setting status to "${statusName}" for ${isPR ? 'PR' : 'Issue'} #${issueNumber}...`); + await github.graphql( + `mutation($projectId: ID!, $itemId: ID!, $fieldId: ID!, $optionId: String!) { + updateProjectV2ItemFieldValue(input: { + projectId: $projectId, + itemId: $itemId, + fieldId: $fieldId, + value: { singleSelectOptionId: $optionId } + }) { + projectV2Item { + id + } + } + }`, + { projectId, itemId, fieldId: STATUS_FIELD_ID, optionId: statusOptionId } + ); + console.log("Automation successfully completed!"); diff --git a/.github/workflows/plugins-deploy-api-doc.yml b/.github/workflows/plugins-deploy-api-doc.yml index 4d22311892..0d963cdfe6 100644 --- a/.github/workflows/plugins-deploy-api-doc.yml +++ b/.github/workflows/plugins-deploy-api-doc.yml @@ -51,7 +51,7 @@ jobs: uses: pnpm/setup@731dfb6463bbd11c11fd0ee11fe00ed0e36d48a1 # v3.2.0 with: working-directory: plugins - runtime: node@26.13.1 + runtime: node@24.22.3 install: false - name: Install deps diff --git a/.github/workflows/plugins-deploy-styles-doc.yml b/.github/workflows/plugins-deploy-styles-doc.yml index 8fd4157a41..60ffe05038 100644 --- a/.github/workflows/plugins-deploy-styles-doc.yml +++ b/.github/workflows/plugins-deploy-styles-doc.yml @@ -49,7 +49,7 @@ jobs: uses: pnpm/setup@731dfb6463bbd11c11fd0ee11fe00ed0e36d48a1 # v3.2.0 with: working-directory: plugins - runtime: node@26.13.1 + runtime: node@24.22.3 install: false - name: Install deps diff --git a/.github/workflows/require-milestone-linked-issues.yml b/.github/workflows/require-milestone-linked-issues.yml new file mode 100644 index 0000000000..d00dade45e --- /dev/null +++ b/.github/workflows/require-milestone-linked-issues.yml @@ -0,0 +1,76 @@ +name: Require Milestone on Linked Issues + +on: + pull_request: + types: + - opened + - reopened + - synchronize + - edited + - ready_for_review + - converted_to_draft + workflow_dispatch: + +permissions: + contents: read + pull-requests: read + issues: read + +jobs: + check-linked-issues: + name: Require Milestones on Linked Issues + runs-on: ubuntu-latest + + steps: + - name: Check linked issues + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + PR_NUMBER: ${{ github.event.pull_request.number }} + IS_DRAFT: ${{ github.event.pull_request.draft }} + REPOSITORY: ${{ github.repository }} + run: | + if [ "$IS_DRAFT" = "true" ]; then + echo "Draft PR: milestone check skipped." + exit 0 + fi + + OWNER="${REPOSITORY%%/*}" + REPO="${REPOSITORY#*/}" + + # shellcheck disable=SC2016 # GraphQL variables, not shell expansion. + ISSUES=$(gh api graphql \ + -F owner="$OWNER" \ + -F repo="$REPO" \ + -F number="$PR_NUMBER" \ + -f query=' + query($owner: String!, $repo: String!, $number: Int!) { + repository(owner: $owner, name: $repo) { + pullRequest(number: $number) { + closingIssuesReferences(first: 100) { + nodes { + number + title + url + milestone { + title + } + } + } + } + } + } + ') + + MISSING=$(echo "$ISSUES" | jq -r ' + .data.repository.pullRequest.closingIssuesReferences.nodes[] + | select(.milestone == null) + | "- #\(.number): \(.title) (\(.url))" + ') + + if [ -n "$MISSING" ]; then + echo "::error::The following linked issues have no milestone:" + echo "$MISSING" + exit 1 + fi + + echo "All linked issues have a milestone." diff --git a/.github/workflows/require-milestone.yml b/.github/workflows/require-milestone.yml new file mode 100644 index 0000000000..641e1a98fa --- /dev/null +++ b/.github/workflows/require-milestone.yml @@ -0,0 +1,45 @@ +name: Require Milestone + +on: + pull_request: + types: + - opened + - reopened + - synchronize + - edited + - milestoned + - demilestoned + - ready_for_review + - converted_to_draft + +permissions: + contents: read + pull-requests: read + +jobs: + check-milestone: + name: Require Milestone + runs-on: ubuntu-latest + + steps: + - name: Check milestone + env: + GH_TOKEN: ${{ github.token }} + PR_NUMBER: ${{ github.event.pull_request.number }} + run: | + pr_api="repos/${GITHUB_REPOSITORY}/pulls/${PR_NUMBER}" + + is_draft=$(gh api "$pr_api" --jq '.draft') + milestone=$(gh api "$pr_api" --jq '.milestone.title // ""') + + if [ "$is_draft" = "true" ]; then + echo "Draft PR: milestone check skipped." + exit 0 + fi + + if [ -z "$milestone" ]; then + echo "::error::This pull request must have a milestone." + exit 1 + fi + + echo "Milestone assigned: $milestone" diff --git a/.serena/memories/backend/subtleties.md b/.serena/memories/backend/subtleties.md index d387fb3b34..8b09aa1ad0 100644 --- a/.serena/memories/backend/subtleties.md +++ b/.serena/memories/backend/subtleties.md @@ -53,6 +53,7 @@ - `app.config/config` and `flags` are dynamic `defonce` vars populated from `PENPOT_*` env vars through the shared schema string transformer. Tests and tooling can bind them. - Duration config values are parsed as ISO-8601 with the `PT` prefix, so a day is written in hours (`:jobs-retention` and `:jobs-user-ttl` default to a week, and an operator writes `168h`): `7d` does not parse, and the value fails validation at boot instead of being silently ignored. +- `app.main` wires `PENPOT_DATABASE_*` into `::db/pool`, and `app.db/create-datasource-config` forwards pgjdbc's own knobs as `dataSourceProperties`, which `Properties.getProperty` silently drops unless the value is a String (hence the `str`). A query parameter of the same name in `PENPOT_DATABASE_URI` (for example `?prepareThreshold=0`) wins over the property: pgjdbc ignores the Properties value when the URL carries it, so the URI is the place to look when a variable seems to do nothing. The prepared statement cache is three optional `PENPOT_DATABASE_PREPARED_STATEMENT_CACHE_*` keys: the threshold is the executions before a statement is server-prepared and accepts `-1` or non-negatives (`{:min -1}`; the driver clamps any other negative to `-1`, so rejecting it only turns a typo into a boot error), its `0` stops server-side prepared statements while the driver's client-side cache of parsed queries stays in use, `..._QUERIES` and `..._SIZE` reject negatives and `0` in either disables that cache, and the size is bytes rounded up to the whole MiB the driver accepts. Do not describe the threshold `0` as disabling the cache: it only stops the server-side prepares. - `parse-flags` automatically adds `:disable-secure-session-cookies` when `public-uri` is plain HTTP and not localhost. This changes cookie defaults without an explicit env flag. - The backend sets Clojure `*assert*` globally from the `:backend-asserts` feature flag. Assertion-dependent checks can therefore differ by runtime flags. - Request body parsing is mostly POST-oriented and supports Transit JSON plus plain JSON. Plain JSON request keys are kebab-decoded before being merged into `:params`. A body the parser cannot read is a client error, classified at the parse site and never by exception class (`clojure.data.json` signals its syntax errors with a bare `java.lang.Exception`): 400 `:validation`/`:malformed-json`, one `warn` line (`malformed request body`) carrying path, peer address and content-type instead of a stack trace, and `penpot_http_malformed_requests_total`. `RequestTooBigException` keeps its 413. diff --git a/.serena/memories/backend/testing.md b/.serena/memories/backend/testing.md index 61c92d8fed..b761504a87 100644 --- a/.serena/memories/backend/testing.md +++ b/.serena/memories/backend/testing.md @@ -33,4 +33,4 @@ JVM `clojure.test` (kaocha runner) under `backend/test/backend_tests/`. `worker-config` can talk to a different Redis key prefix than a cfg the test builds itself. The symptom is a component that silently does nothing locally and races in CI (which sets no `PENPOT_TENANT`). -- `backend-tests.helpers` has `metrics-stub` (a metrics instance with no collectors, where every `run!` is a no-op), `init-climit` (a concurrency limiter built from an inline config map, with an optional second argument for the metrics instance to publish to) and `with-climit`, the macro that builds one, runs the body and halts it. `init-climit` starts the republisher daemon thread, so a test that builds a limiter without halting it leaks that thread for the rest of the JVM run: use `with-climit`. Log lines are capturable with `add-watch` on `app.common.logging/log-record` (it holds the last record; the watcher sees every one, but only of the levels the logger has enabled: a line the app writes at TRACE never gets there). +- `backend-tests.helpers` has `metrics-stub` (a metrics instance with no collectors, where every `run!` is a no-op), `init-climit` (a concurrency limiter built from an inline config map, with an optional second argument for the metrics instance to publish to) and `with-climit`, the macro that builds one, runs the body and halts it. `init-climit` starts the republisher daemon thread, so a test that builds a limiter without halting it leaks that thread for the rest of the JVM run: use `with-climit`. Log lines are capturable with `add-watch` on `app.common.logging/log-record` (it holds the last record; the watcher sees every one, but only of the levels the logger has enabled: a line the app writes at TRACE never gets there). The logger swaps the record on an executor thread, after the call that logged it returns: keep the watch installed until the record arrives (poll for it) and only then remove it, or the record lands on nobody under load. diff --git a/.serena/memories/common/core.md b/.serena/memories/common/core.md index 816d131e14..1c969b5ba5 100644 --- a/.serena/memories/common/core.md +++ b/.serena/memories/common/core.md @@ -26,6 +26,12 @@ Respect the intended abstraction direction in new/refactored code: Some legacy code violates this layering; do not copy those violations into new code when a focused refactor is practical. +## Build-mode dependent laziness + +`app.common.data.macros/lazy-delay` returns a delay whose expression is evaluated eagerly by default. Only the builds that set the `app.common.data.macros/lazy-delay?` closure define to `true` defer it to the first deref; every production build does (frontend `main` and `worker`, exporter, library), so the Closure compiler replaces the constant and drops the eager branch. On the JVM it is always eager. + +`app.common.schema/check-fn` resolves its schema through it: an invalid schema definition fails when the namespace loads in development and in tests, and only on the first check call in production. The other lazily built parts of `check-fn` (validator and explainer) are always deferred. + ## Focused memory routing Model, schema, and persistence shape: diff --git a/.serena/memories/frontend/plugin-api-to-cljs-binding.md b/.serena/memories/frontend/plugin-api-to-cljs-binding.md index 31ceb4b10c..a55d857bbd 100644 --- a/.serena/memories/frontend/plugin-api-to-cljs-binding.md +++ b/.serena/memories/frontend/plugin-api-to-cljs-binding.md @@ -37,5 +37,7 @@ - Plugin listeners are watches on the global store and callbacks are debounced about 10ms. Callback exceptions are caught and logged so plugin code does not crash the app. - `selectionchange` callbacks receive arrays of shape id strings, while `filechange`, `pagechange`, and `shapechange` return proxies. - `contentsave` fires only when persistence status transitions to `:saved`; it calls the callback with no value. +- `penpot.waitForSave(timeout?)` (`app.plugins.persist/wait-for-save`; runtime forwarder checks `content:read`) waits on the **shared** persistence queue, not only the plugin's own commits. Flow: ~100ms intake race (buffer/queue non-empty vs deadline, capped to the timeout) → empty resolves; pending emits `::dps/force-persist` and races `dps/wait-persisted-or-error` against `::dps/rollback-rejected` / `::dps/rolled-back` (rollback rejects even if status later reads `:saved`). Default 60s. `:error` status at intake rejects at once (`earlier-failure`): `save-retrying` while `dps/save-retrying?` (slow retry cycle, recovers by itself), otherwise the stored `[:error :code]` with reopen advice (a terminal failure stays until the file is reopened, so it fails every call). Rejections are `Error`s with `code`: `save-timeout`, `rolled-back`, `save-retrying`, or the failure code (`save-failed` unless the backend gave one). +- Plugin timeouts validate through `u/valid-timeout?` (positive finite number ≤ `u/max-timeout`, signed 32-bit) and reject through `u/reject-not-valid`, which rejects an `Error` (never a bare string) so callers can read `.message`. - `contentrollback` is not a state watch: it subscribes to `st/stream` for `::dps/rolled-back` (kept in `subscriptions`, disposed by `remove-listener`), not debounced, delivered async so the reverted state is in place. Payload `{fileIds pageIds shapeIds}` from `pcm/rollback-targets` (page shapes only, from the reverted commits' redo changes). When the rollback falls back to reloading the file, `::dps/rolled-back` waits for the reloaded data, so the callback never reads the rejected data. The runtime's `validEvents` must list every event name, or `penpot.on` rejects it; a stale runtime build rejects new events. The MCP plugin forwards these rollbacks with the next task response and the server puts a warning first in that tool response (`RollbackNotices`). Unsent ones survive a tab or plugin reload in `penpot.localStorage` (key `rollbacks/`, `LocalStorageRollbackStore`; the MCP manifest in `app.main.data.workspace.mcp` and `mcp/packages/plugin/public/manifest.json` grant `allow:localstorage`, which the runtime checks on access); the entry goes once a response carried them. - Plugin history `undoBlockBegin` creates a workspace undo transaction with a JS `Symbol`; `undoBlockFinish` commits that symbol. Missing finish eventually relies on the workspace transaction timeout. \ No newline at end of file diff --git a/.serena/memories/frontend/testing.md b/.serena/memories/frontend/testing.md index 45e6bff9e4..e568292436 100644 --- a/.serena/memories/frontend/testing.md +++ b/.serena/memories/frontend/testing.md @@ -46,6 +46,7 @@ To assert over emitted event sequences, observe termination: subscribe through ` - The `:esm` test build cannot load the CommonJS `sax` module behind `tubax`, so SVG parsing fails in tests (`dwm/valid-svg-string?` answers false, `dwm/svg->clj` errors): mock both with the parsed data. - A hook-order violation in a mounted component reaches neither the root's `onUncaughtError`/`onCaughtError`/`onRecoverableError` nor the markup: React logs an internal error to `console.error` and recovers, so the callbacks stay empty and the tree repaints. Assert on a captured `console.error` too. It only fires when a render *adds* hooks (nil → uuid); a render that calls none at all slips through React's bookkeeping, so the regression test needs the "hooks arrive" direction. `frontend-tests.ui.team-container-test` does both, with a jsdom document. - `beicon.v2.core` has no `never`. A stream that must **not emit** is `(rx/empty)`: it completes at once, which is enough for `merge`, `filter` and `take` where only the emission matters. A stream that must **stay open** is a `(rx/subject)` nobody pushes to; `rx/empty` would complete the flow behind it. An undeclared `rx/never` compiles to an undefined var and the test dies as a rejected body, which `cljs.test` reports as a test with zero assertions instead of a failure. +- `set!` mocks are global and outlive the fixture, so a flow another module left running (a store's pending timer) can still reach the mocked transport mid-test; `with-persistence` records every command for that reason. Filter to the commands your layer issues — persistence only sends `update-file` — before counting or asserting emptiness, and never let a stray call release a `wait-for` on one. From `frontend/`: - Full unit test run (always builds, suppressed output): `pnpm run test:quiet`. diff --git a/.serena/memories/mcp/core.md b/.serena/memories/mcp/core.md index 8ca829c32f..255b8fb9b2 100644 --- a/.serena/memories/mcp/core.md +++ b/.serena/memories/mcp/core.md @@ -120,6 +120,14 @@ From the `mcp/` directory, run - Authorization/discovery URLs, self-contained client IDs, consent and PKCE/code binding, backend Redis state, deployment wiring, and compatibility limits: `mem:mcp/oauth`. +## Host and Origin allowlist (`RequestGuard`) + +- `RequestGuard` checks `Host` and `Origin` (hostname only, port ignored) on `/mcp`, the REPL (Express middleware, 403) and the plugin WebSocket (close `1008` before any message is read). It stops web pages on the user's machine from driving a local server (DNS rebinding, cross-site WebSocket). +- Allowlist: `PENPOT_MCP_ALLOWED_HOSTS` (`*` turns the check off); unset → loopback only in single-user mode (local or remote: no other barrier, `execute_code` still runs in the user's session), any host in multi-user mode (relies on user tokens; behind nginx `/mcp/ws` the server sees `Host: penpot-mcp:4402`). A missing `Origin` passes (non-browser clients). +- Entries must be `*`, a hostname or a bracketed IPv6 address; one with a scheme, port or path makes `fromEnvironment` throw, so the server does not start. Startup logs a `warn` when the check is off in single-user mode. +- The plugin iframe has `allow-same-origin`, so its WebSocket carries the real plugin origin (`http://localhost:4400` locally, the Penpot origin when integrated). +- Test mocks of `PenpotMcpServer` passed to `PluginBridge` must provide `requestGuard`. + ## Backend tools (`submit_feedback`) - Tools that call the Penpot backend directly (not through the plugin) register only when `PENPOT_MCP_BACKEND_URI` is nonblank (`PenpotMcpServer.initTools`). Currently only `SubmitFeedbackTool`. @@ -140,10 +148,11 @@ For parallel devenvs, prefer same-origin MCP routing: each Penpot instance shoul ## Plugin reconnect policy - The plugin treats WebSocket close code `1008` (policy violation) as terminal: it stops auto-reconnecting and stays disconnected until the user explicitly reconnects. Other close codes keep the capped-backoff retry. The decision lives in `ReconnectPolicy.ts` (`shouldReconnectAfterClose`), kept as a pure module so it is unit-testable without DOM/CSS. -- The MCP server emits `1008` for a missing `userToken` in multi-user mode, for missing/invalid connection metadata, and to close a connection displaced by a newer one with the same user and session ID. +- The MCP server emits `1008` for a `Host`/`Origin` outside the `RequestGuard` allowlist, for a missing `userToken` in multi-user mode, for missing/invalid connection metadata, and to close a connection displaced by a newer one with the same user and session ID. - Session IDs are deterministic per tab and file, so a reconnect reuses its ID while the old socket may be half-open: the newest connection wins (`PluginBridge.displaceConnection`). Across instances, each new connection publishes a session claim on a per-user Redis channel; an instance displaces its local connection only on a foreign claim that arrives after its own claim (channel order, no clocks). Discovery deduplicates by session ID. - A tab rejected with `1008` never reaches `connected`, so the frontend's 60s reconnect watcher (`start-reconnect-watcher` in `app.main.data.workspace.mcp`, started only on `connected`) does not engage; recovery is manual via the MCP menu. ## Rollback warnings +- `ExecuteCodeTaskHandler` awaits `penpot.waitForSave()` after the code and outside its swallow catch, so a save failure, save timeout or rollback during the wait fails that `execute_code` call itself. It hard-fails as "missing Plugin API support" when `penpot.waitForSave` is absent (old Penpot or stale runtime build, or broken `api.cljs` wiring). Both waits run only when `penpotMgmt.workspace.status` is `"ready"` (or there is no `penpotMgmt`), checked at each wait: dashboard calls skip them, because persistence `:error` outlives the workspace and would fail successful `penpotMgmt` calls. Budget: layout wait 30s before the code + save wait 60s after it, under `PENPOT_MCP_TOOL_TIMEOUT_S` (120s). `RollbackNotices` below only cover rollbacks that land after the call returned (concurrent edits, a response after a timeout). - The plugin's `RollbackLog` (`packages/plugin/src/RollbackLog.ts`) holds the `contentrollback` events not yet sent with a task response; `Task.sendResponse` attaches them and acknowledges them once sent, and the server's `unclaimed-rollbacks` message (a response that came after its task timed out) puts them back. With a `RollbackStore` (`LocalStorageRollbackStore` over `penpot.localStorage`, one entry per file) every change is written through, so a reload of the tab or plugin keeps them; storage failures fall back to memory. Plugin tests run with `node --experimental-strip-types`, which rejects TypeScript parameter properties: declare fields explicitly. diff --git a/.serena/memories/workflow/creating-commits.md b/.serena/memories/workflow/creating-commits.md index d225217867..6c25b175a8 100644 --- a/.serena/memories/workflow/creating-commits.md +++ b/.serena/memories/workflow/creating-commits.md @@ -43,10 +43,9 @@ that breaks them is wrong, even if the code is right. - For another commit: `./scripts/check-commit -c ` - **NEVER** hand-wave the body as "one long line". If a line exceeds 80, break it at a space. -- **NEVER add a `Co-authored-by:` trailer.** It is forbidden; `scripts/check-commit` fails the commit when any line matches it (case-insensitive). +- **`Co-authored-by:` is for human co-authors only** (`Co-authored-by: Name `). AI providers, agents and bots must not be credited that way; use `AI-assisted-by:` instead. `scripts/check-commit` rejects a trailer value that names an AI provider or agent (`Claude`, `Copilot`, `Cursor`, `Codex`, `Gemini`, `Aider`, `Cline`, `Windsurf`, `DeepSeek`, `OpenCode`, `devin-ai`, `amazon q`, and similar) or carries a bot identity (`[bot]`, `bot@`). `NON_HUMAN_COAUTHOR_PATTERN` is the list; extend it when a new tool starts adding trailers. A person whose own name or email matches one of those tokens (a surname like `Cline`, an address at a provider) is rejected too; amend the trailer or credit them with `Reviewed-by:`. - CI runs `scripts/check-commit` on every new commit (`mem:workflow/creating-commits` is enforced by `.github/workflows/commit-checker.yml`), so a forbidden trailer fails the build. -- Exceptions inside the body (do not wrap these): `Signed-off-by:`, - `AI-assisted-by:` trailers, and lines carrying a URL. +- Exceptions inside the body (do not wrap these): `Signed-off-by:`, `Co-authored-by:`, `AI-assisted-by:` trailers, and lines carrying a URL. **AI-assisted-by trailer rules:** - Use only the model name, e.g. `mimo-v2.5`, `deepseek-v4-flash` diff --git a/.serena/memories/workflow/docker-images.md b/.serena/memories/workflow/docker-images.md index 28d42bc9e7..3415c8247d 100644 --- a/.serena/memories/workflow/docker-images.md +++ b/.serena/memories/workflow/docker-images.md @@ -2,5 +2,6 @@ Node and pnpm pins: `mem:workflow/updating-node` and `mem:workflow/updating-pnpm`. Only the parts the Dockerfiles cannot show: -- JDK: the image tracks a non-LTS line, so there is no minor to pick — the last build of the line is the ceiling; moving line is a major and needs a real build to validate the `jlink` module list. +- JDK: two pins that move together — `docker/devenv/Dockerfile` (`setup-jvm` stage, full JDK) and `docker/images/Dockerfile.backend` (JDK + `jlink` runtime). Same build, same sums in both. They compile and run the same backend: `manage.sh` builds the bundle inside the devenv image (`build-backend-bundle` → `build "backend"`) and `backend/scripts/run.template.sh` starts the runtime with `--enable-preview`, which requires the runtime major to match the compiler major. A runtime line behind the compiler line is a silent trap. +- JDK line: tracks a non-LTS line, so there is no minor to pick — the last build of the line is the ceiling; moving line is a major and needs a real build to validate the `jlink` module list (`--add-modules` in `Dockerfile.backend`). JDK 27 keeps all of them; check with `bin/jlink --add-modules ` against the downloaded JDK before bumping. Source of the URL and both sums: the Azul metadata API (`api.azul.com/metadata/v1/zulu/packages/?java_version=&os=linux&arch=&java_package_type=jdk&archive_type=tar.gz&latest=true&release_status=ga&availability_types=CA`), then confirm the sums by downloading the tarball and running `sha256sum`. - ImageMagick: publish `penpotapp/imagemagick:` **before** bumping the `COPY --from` refs in the backend, exporter and devenv Dockerfiles; otherwise both builds break on the missing tag. diff --git a/.serena/memories/workflow/updating-node.md b/.serena/memories/workflow/updating-node.md index f8b803d9b5..b50aba5539 100644 --- a/.serena/memories/workflow/updating-node.md +++ b/.serena/memories/workflow/updating-node.md @@ -19,6 +19,7 @@ No single source of truth: nothing reads `.nvmrc` at build time, and the version - Renovate also opens these bumps, on `renovate-*` branches and with its config living outside this repo; a manual bump and Renovate's must not diverge on any of the six sites. - A backport merge can deliver a newer Node into `docker/` and the plugins-deploy workflows while main's `.nvmrc` stays behind, because main receives partial backports of `.github/` and `docker/` from develop. Merging main into staging therefore conflicts on the pin lines: take main's set and finish the bump (`.nvmrc` plus the `packageManager` fields via `mem:workflow/updating-pnpm`), or the six sites drift. - The newer version wins in both directions. A maintenance→main merge (`main-2.18.x` into `main`) can carry the newer Node while `.nvmrc` and the `packageManager` fields stay on the older pair; then adopt the newer set and finish the bump on all six sites plus the 35 `packageManager` fields. +- Node stays on the 24.x line even when the rest of `.github/` moves on: `99db795966` (`update GitHub Actions`) took the two `plugins-deploy-*` `runtime:` pins to `node@26.13.1` on develop while `.nvmrc` stayed on `v24.22.3`. The project keeps every site on the `.nvmrc` value; resolve that conflict back to the 24.x pin instead of taking develop's. - Both workflows must stay valid YAML. The `python3` in the devenv has no `yaml` module, so validate with `uv run --with pyyaml python3 -c ...` instead of a bare import. ## Verification diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 27ec89998b..66e393892c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -244,8 +244,11 @@ Commit messages must follow this format: those prefixes to an existing subject) - **Wrap body lines at 80 characters or fewer** (trailers and URLs excepted) - Separate the subject from the body with a **blank line** -- Never add a `Co-authored-by` trailer — it is forbidden and fails - `./scripts/check-commit` +- Use `Co-authored-by:` only for a human co-author + (`Co-authored-by: Name `). AI providers, agents and bots are not + allowed there; for AI participation add an `AI-assisted-by: ` + trailer instead. `./scripts/check-commit` rejects the trailer when its value + names one of them. You can check a commit against these rules with `./scripts/check-commit`. CI runs the same script on every new commit in diff --git a/README.md b/README.md index ffd702aa31..831eaf8d8a 100644 --- a/README.md +++ b/README.md @@ -13,9 +13,6 @@ Managed with Taiga.io - - Gitpod ready-to-code -

diff --git a/backend/deps.edn b/backend/deps.edn index 3a8a282638..03b38e7881 100644 --- a/backend/deps.edn +++ b/backend/deps.edn @@ -40,7 +40,6 @@ nrepl/nrepl {:mvn/version "1.8.0"} org.postgresql/postgresql {:mvn/version "42.7.13"} - org.xerial/sqlite-jdbc {:mvn/version "3.53.4.0"} com.zaxxer/HikariCP {:mvn/version "7.1.0"} diff --git a/backend/src/app/binfile/common.clj b/backend/src/app/binfile/common.clj index 0bed77adc1..ed01b120f0 100644 --- a/backend/src/app/binfile/common.clj +++ b/backend/src/app/binfile/common.clj @@ -26,7 +26,6 @@ [app.common.weak :as weak] [app.config :as cf] [app.db :as db] - [app.db.sql :as sql] [app.features.fdata :as fdata] [app.features.file-migrations :as fmigr] [app.loggers.audit :as-alias audit] @@ -131,18 +130,12 @@ (defn initial-state [] {:storage-objects #{} - :files #{} - :teams #{} - :projects #{}}) + :files #{}}) (defn collect-storage-objects [state items] (update state :storage-objects into xf-map-media-id items)) -(defn collect-summary - [state key items] - (update state key into xf-map-media-id items)) - (defn lookup-index [id] (when id @@ -876,27 +869,6 @@ [cfg project-id] (db/get cfg :project {:id project-id})) -(def ^:private sql:get-teams - "SELECT t.* FROM team WHERE id = ANY(?)") - -(defn get-teams - [cfg ids] - (let [conn (db/get-connection cfg) - ids (db/create-array conn "uuid" ids)] - (->> (db/exec! conn [sql:get-teams ids]) - (map decode-row-features)))) - -(defn get-team - [cfg team-id] - (-> (db/get cfg :team {:id team-id}) - (decode-row-features))) - -(defn get-fonts - [cfg team-id] - (db/query cfg :team-font-variant - {:team-id team-id - :deleted-at nil})) - (def ^:private sql:get-files-rels "SELECT flr.*, fls.synced_at @@ -978,15 +950,6 @@ :deleted-at nil}) (not-empty))) -(defn get-file-thumbnail - "Return the thumbnail for the specified file-id" - [cfg {:keys [id revn]}] - (db/get* cfg :file-thumbnail - {:file-id id - :revn revn - :data nil} - {::sql/columns [:media-id :file-id :revn]})) - (def ^:private sql:get-missing-media-references "SELECT fmo.* FROM file_media_object AS fmo diff --git a/backend/src/app/binfile/v2.clj b/backend/src/app/binfile/v2.clj deleted file mode 100644 index 06bb22b7c8..0000000000 --- a/backend/src/app/binfile/v2.clj +++ /dev/null @@ -1,450 +0,0 @@ -;; This Source Code Form is subject to the terms of the Mozilla Public -;; License, v. 2.0. If a copy of the MPL was not distributed with this -;; file, You can obtain one at http://mozilla.org/MPL/2.0/. -;; -;; Copyright (c) KALEIDOS SUBSIDIARY SL - -(ns app.binfile.v2 - "A sqlite3 based binary file exportation with support for exportation - of entire team (or multiple teams) at once." - (:refer-clojure :exclude [read]) - (:require - [app.binfile.common :as bfc] - [app.common.data :as d] - [app.common.features :as cfeat] - [app.common.logging :as l] - [app.common.time :as ct] - [app.common.transit :as t] - [app.common.uuid :as uuid] - [app.config :as cf] - [app.db :as db] - [app.db.sql :as sql] - [app.loggers.audit :as-alias audit] - [app.loggers.webhooks :as-alias webhooks] - [app.storage :as sto] - [app.storage.tmp :as tmp] - [app.util.events :as events] - [app.worker :as-alias wrk] - [clojure.set :as set] - [cuerdas.core :as str] - [datoteka.io :as io] - [promesa.util :as pu]) - (:import - java.sql.DriverManager)) - -(set! *warn-on-reflection* true) - -;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; -;; LOW LEVEL API -;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; - -(defn- create-database - ([cfg] - (let [path (tmp/tempfile :prefix "penpot.binfile." :suffix ".sqlite")] - (create-database cfg path))) - ([cfg path] - (let [db (DriverManager/getConnection (str "jdbc:sqlite:" path))] - (assoc cfg ::db db ::path path)))) - -(def ^:private - sql:create-kvdata-table - "CREATE TABLE kvdata ( - tag text NOT NULL, - key text NOT NULL, - val text NOT NULL, - dat blob NULL - )") - -(def ^:private - sql:create-kvdata-index - "CREATE INDEX kvdata__tag_key__idx - ON kvdata (tag, key)") - -(defn- setup-schema! - [{:keys [::db]}] - (db/exec-one! db [sql:create-kvdata-table]) - (db/exec-one! db [sql:create-kvdata-index])) - -(defn- write! - [{:keys [::db]} tag k v & [data]] - (db/insert! db :kvdata - {:tag (d/name tag) - :key (str k) - :val (t/encode-str v {:type :json-verbose}) - :dat data} - {::db/return-keys false})) - -(defn- read-blob - [{:keys [::db]} tag k] - (let [obj (db/get db :kvdata - {:tag (d/name tag) - :key (str k)} - {::sql/columns [:dat]})] - (:dat obj))) - -(defn- read-seq - ([{:keys [::db]} tag] - (->> (db/query db :kvdata - {:tag (d/name tag)} - {::sql/columns [::val]}) - (map :val) - (map t/decode-str))) - ([{:keys [::db]} tag k] - (->> (db/query db :kvdata - {:tag (d/name tag) - :key (str k)} - {::sql/columns [::val]}) - (map :val) - (map t/decode-str)))) - -(defn- read-obj - [{:keys [::db]} tag k] - (let [obj (db/get db :kvdata - {:tag (d/name tag) - :key (str k)} - {::sql/columns [:val]})] - (-> obj :val t/decode-str))) - -;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; -;; IMPORT/EXPORT IMPL -;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; - -(declare ^:private write-project!) -(declare ^:private write-file!) - -(defn- write-team! - [cfg team-id] - - (let [team (bfc/get-team cfg team-id) - fonts (bfc/get-fonts cfg team-id)] - - (events/tap :progress - {:op :export - :section :write-team - :id team-id - :name (:name team)}) - - (l/trc :hint "write" :obj "team" - :id (str team-id) - :fonts (count fonts)) - - (when-let [photo-id (:photo-id team)] - (vswap! bfc/*state* update :storage-objects conj photo-id)) - - (vswap! bfc/*state* update :teams conj team-id) - (vswap! bfc/*state* bfc/collect-storage-objects fonts) - - (write! cfg :team team-id team) - - (doseq [{:keys [id] :as font} fonts] - (vswap! bfc/*state* update :team-font-variants conj id) - (write! cfg :team-font-variant id font)))) - -(defn- write-project! - [cfg project] - (events/tap :progress - {:op :export - :section :write-project - :id (:id project) - :name (:name project)}) - (l/trc :hint "write" :obj "project" :id (str (:id project))) - (write! cfg :project (str (:id project)) project) - (vswap! bfc/*state* update :projects conj (:id project))) - -(defn- write-file! - [cfg file-id] - (let [file (bfc/get-file cfg file-id :realize? true) - thumbs (bfc/get-file-object-thumbnails cfg file-id) - media (bfc/get-file-media cfg file) - rels (bfc/get-files-rels cfg #{file-id})] - - (events/tap :progress - {:op :export - :section :write-file - :id file-id - :name (:name file)}) - - (vswap! bfc/*state* (fn [state] - (-> state - (update :files conj file-id) - (update :file-media-objects into bfc/xf-map-id media) - (bfc/collect-storage-objects thumbs) - (bfc/collect-storage-objects media)))) - - (write! cfg :file file-id file) - (write! cfg :file-rels file-id rels) - - (run! (partial write! cfg :file-media-object file-id) media) - (run! (partial write! cfg :file-object-thumbnail file-id) thumbs) - - (when-let [thumb (bfc/get-file-thumbnail cfg file)] - (vswap! bfc/*state* bfc/collect-storage-objects [thumb]) - (write! cfg :file-thumbnail file-id thumb)) - - (l/trc :hint "write" :obj "file" - :thumbnails (count thumbs) - :rels (count rels) - :media (count media)))) - -(defn- write-storage-object! - [{:keys [::sto/storage] :as cfg} id] - (let [sobj (sto/get-object storage id) - data (with-open [input (sto/get-object-data storage sobj)] - (io/read input))] - - (l/trc :hint "write" :obj "storage-object" :id (str id) :size (:size sobj)) - (write! cfg :storage-object id (meta sobj) data))) - -(defn- read-storage-object! - [{:keys [::sto/storage ::bfc/timestamp] :as cfg} id] - (let [mdata (d/update-when (read-obj cfg :storage-object id) - :content-type bfc/normalize-content-type) - data (read-blob cfg :storage-object id) - ;; NOTE: v2 bundles are legacy and no longer produced; the - ;; sanitizer call stays as defense in depth and is covered by - ;; the shared helper unit tests, with no dedicated v2 - ;; import test. - clean (when (bfc/svg-object? mdata) - (bfc/sanitize-imported-svg mdata data)) - data (or (:bytes clean) data) - hash (or (:hash clean) (sto/calculate-hash data)) - - content (-> (sto/content data) - (sto/wrap-with-hash hash)) - - params (-> mdata - (assoc ::sto/content content) - (assoc ::sto/deduplicate? true) - (assoc ::sto/touched-at timestamp)) - - sobject (sto/put-object! storage params)] - - (vswap! bfc/*state* update :index assoc id (:id sobject)) - - (l/trc :hint "read" :obj "storage-object" - :id (str id) - :new-id (str (:id sobject)) - :size (:size sobject)))) - -(defn read-team! - [{:keys [::db/conn ::bfc/timestamp] :as cfg} team-id] - (l/trc :hint "read" :obj "team" :id (str team-id)) - - (let [team (read-obj cfg :team team-id) - team (-> team - (update :id bfc/lookup-index) - (update :photo-id bfc/lookup-index) - (assoc :created-at timestamp) - (assoc :modified-at timestamp))] - - (events/tap :progress - {:op :import - :section :read-team - :id team-id - :name (:name team)}) - - (db/insert! conn :team - (update team :features db/encode-pgarray conn "text") - ::db/return-keys false) - - (doseq [font (->> (read-seq cfg :team-font-variant) - (filter #(= team-id (:team-id %))))] - (let [font (-> font - (update :id bfc/lookup-index) - (update :team-id bfc/lookup-index) - (update :woff1-file-id bfc/lookup-index) - (update :woff2-file-id bfc/lookup-index) - (update :ttf-file-id bfc/lookup-index) - (update :otf-file-id bfc/lookup-index) - (assoc :created-at timestamp) - (assoc :modified-at timestamp))] - (db/insert! conn :team-font-variant font - ::db/return-keys false))) - - team)) - -(defn read-project! - [{:keys [::db/conn ::bfc/timestamp] :as cfg} project-id] - (l/trc :hint "read" :obj "project" :id (str project-id)) - - (let [project (read-obj cfg :project project-id) - project (-> project - (update :id bfc/lookup-index) - (update :team-id bfc/lookup-index) - (assoc :created-at timestamp) - (assoc :modified-at timestamp))] - - (events/tap :progress - {:op :import - :section :read-project - :id project-id - :name (:name project)}) - - (db/insert! conn :project project - ::db/return-keys false))) - -(defn read-file! - [{:keys [::db/conn ::bfc/timestamp] :as cfg} file-id] - (l/trc :hint "read" :obj "file" :id (str file-id)) - - (let [file (-> (read-obj cfg :file file-id) - (update :id bfc/lookup-index) - (update :project-id bfc/lookup-index)) - file (bfc/process-file cfg file)] - - (events/tap :progress - {:op :import - :section :read-file - :id file-id - :name (:name file)}) - - ;; All features that are enabled and requires explicit migration are - ;; added to the state for a posterior migration step. - (doseq [feature (-> (::bfc/features cfg) - (set/difference cfeat/no-migration-features) - (set/difference (:features file)))] - (vswap! bfc/*state* update :pending-to-migrate (fnil conj []) [feature (:id file)])) - - (bfc/save-file! cfg file ::db/return-keys false)) - - (doseq [thumbnail (read-seq cfg :file-object-thumbnail file-id)] - (let [thumbnail (-> thumbnail - (update :file-id bfc/lookup-index) - (update :media-id bfc/lookup-index)) - file-id (:file-id thumbnail) - - thumbnail (update thumbnail :object-id - #(str/replace-first % #"^(.*?)/" (str file-id "/")))] - - (db/insert! conn :file-tagged-object-thumbnail thumbnail - {::db/return-keys false}))) - - ;; NOTE: v2 packages are only produced and imported through the - ;; developer REPL (`app.srepl.binfile`), never through the product, so - ;; this path keeps inserting the relations as they come: unlike v1 and - ;; v3 it does not skip self-referencing or circular links (see - ;; `bfc/library-link-status`). - (doseq [rel (read-obj cfg :file-rels file-id)] - (let [rel (-> rel - (update :file-id bfc/lookup-index) - (update :library-file-id bfc/lookup-index))] - (db/insert! conn :file-library-rel rel - ::db/return-keys false) - (bfc/upsert-file-library-sync! conn (assoc rel :synced-at timestamp)))) - - (doseq [media (read-seq cfg :file-media-object file-id)] - (let [media (-> media - (update :id bfc/lookup-index) - (update :file-id bfc/lookup-index) - (update :media-id bfc/lookup-index) - (update :thumbnail-id bfc/lookup-index))] - (db/insert! conn :file-media-object media - ::db/return-keys false - ::sql/on-conflict-do-nothing true)))) - -(def ^:private empty-summary - {:teams #{} - :files #{} - :projects #{} - :file-media-objects #{} - :team-font-variants #{} - :storage-objects #{}}) - -;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; -;; PUBLIC API -;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; - -(defn export-team! - [cfg team-id] - (let [id (uuid/next) - tp (ct/tpoint) - cfg (create-database cfg)] - - (l/inf :hint "start" - :operation "export" - :id (str id) - :path (str (::path cfg))) - - (try - (db/tx-run! cfg (fn [cfg] - (setup-schema! cfg) - (binding [bfc/*state* (volatile! empty-summary)] - (write-team! cfg team-id) - - (run! (partial write-project! cfg) - (bfc/get-team-projects cfg team-id)) - - (run! (partial write-file! cfg) - (bfc/get-team-files-ids cfg team-id)) - - (run! (partial write-storage-object! cfg) - (-> bfc/*state* deref :storage-objects)) - - (write! cfg :manifest "team-id" team-id) - (write! cfg :manifest "objects" (deref bfc/*state*)) - - (::path cfg)))) - (finally - (pu/close! (::db cfg)) - - (let [elapsed (tp)] - (l/inf :hint "end" - :operation "export" - :id (str id) - :elapsed (ct/format-duration elapsed))))))) - -(defn import-team! - [cfg path] - (let [id (uuid/next) - tp (ct/tpoint) - - cfg (-> (create-database cfg path) - (assoc ::bfc/timestamp (ct/now)))] - - (l/inf :hint "start" - :operation "import" - :id (str id) - :path (str (::path cfg))) - - (try - (db/tx-run! cfg (fn [{:keys [::db/conn] :as cfg}] - (db/exec-one! conn ["SET idle_in_transaction_session_timeout = 0"]) - (db/exec-one! conn ["SET CONSTRAINTS ALL DEFERRED"]) - - (binding [bfc/*state* (volatile! {:index {}})] - (let [objects (read-obj cfg :manifest "objects")] - - ;; We first process all storage objects, they have - ;; deduplication so we can't rely on simple reindex. This - ;; operation populates the index for all storage objects. - (run! (partial read-storage-object! cfg) (:storage-objects objects)) - - ;; Populate index with all the incoming objects - (vswap! bfc/*state* update :index - (fn [index] - (-> index - (bfc/update-index (:teams objects)) - (bfc/update-index (:projects objects)) - (bfc/update-index (:files objects)) - (bfc/update-index (:file-media-objects objects)) - (bfc/update-index (:team-font-variants objects))))) - - (let [team-id (read-obj cfg :manifest "team-id") - team (read-team! cfg team-id) - features (cfeat/get-team-enabled-features cf/flags team) - cfg (assoc cfg ::bfc/features features)] - - (run! (partial read-project! cfg) (:projects objects)) - (run! (partial read-file! cfg) (:files objects)) - - ;; (run-pending-migrations! cfg) - - team))))) - (finally - (pu/close! (::db cfg)) - - (let [elapsed (tp)] - (l/inf :hint "end" - :operation "import" - :id (str id) - :elapsed (ct/format-duration elapsed))))))) diff --git a/backend/src/app/config.clj b/backend/src/app/config.clj index e1d00c902a..2d32ea02e9 100644 --- a/backend/src/app/config.clj +++ b/backend/src/app/config.clj @@ -265,6 +265,12 @@ [:database-readonly {:optional true} ::sm/boolean] [:database-min-pool-size {:optional true} ::sm/int] [:database-max-pool-size {:optional true} ::sm/int] + ;; pgjdbc prepared statement cache (see app.db). Size is in bytes and is + ;; converted to the whole MiB the driver accepts. Absent keys mean the + ;; driver defaults; see docs/technical-guide/configuration.md. + [:database-prepared-statement-cache-threshold {:optional true} [::sm/int {:min -1}]] + [:database-prepared-statement-cache-queries {:optional true} [::sm/int {:min 0}]] + [:database-prepared-statement-cache-size {:optional true} [::sm/int {:min 0}]] [:quotes-teams-per-profile {:optional true} ::sm/int] [:quotes-access-tokens-per-profile {:optional true} ::sm/int] diff --git a/backend/src/app/db.clj b/backend/src/app/db.clj index bfce3b2274..8bdeae67a4 100644 --- a/backend/src/app/db.clj +++ b/backend/src/app/db.clj @@ -57,7 +57,9 @@ (def ^:private schema:pool-options [:map {:title "pool-options"} - [::connect-timeout {:optional true} ::sm/int] + [::connection-timeout {:optional true} ::sm/int] + [::idle-timeout {:optional true} ::sm/int] + [::max-lifetime {:optional true} ::sm/int] [::max-size {:optional true} ::sm/int] [::min-size {:optional true} ::sm/int] [::name {:optional true} :keyword] @@ -66,6 +68,17 @@ [::username {:optional true} :string] [::validation-timeout {:optional true} ::sm/int] [::read-only {:optional true} ::sm/boolean] + ;; pgjdbc prepared statement cache. Absent keys mean the driver + ;; defaults; the size is in bytes and is converted to whole MiB for + ;; the driver (see create-datasource-config). The threshold is the + ;; executions before a statement is server-prepared: 0 stops + ;; server-side prepared statements (the cache of parsed queries stays + ;; in use) and -1 prepares from the first execution. Any other + ;; negative is a typo the driver would clamp to -1, so the schema + ;; rejects it. + [::prepared-statement-cache-threshold {:optional true} [::sm/int {:min -1}]] + [::prepared-statement-cache-queries {:optional true} [::sm/int {:min 0}]] + [::prepared-statement-cache-size {:optional true} [::sm/int {:min 0}]] [::mtx/metrics ::mtx/metrics]]) (def defaults @@ -94,7 +107,10 @@ :credentials (and (contains? cfg ::username) (contains? cfg ::password)) :min-size (::min-size cfg) - :max-size (::max-size cfg)) + :max-size (::max-size cfg) + :prepared-statement-cache-threshold (::prepared-statement-cache-threshold cfg) + :prepared-statement-cache-queries (::prepared-statement-cache-queries cfg) + :prepared-statement-cache-size (::prepared-statement-cache-size cfg)) (create-pool cfg)))) (defmethod ig/halt-key! ::pool @@ -113,6 +129,17 @@ ;; pairwise, when both are partitioned with the same bounds "SET enable_partitionwise_join = on;")) +;; pgjdbc reads dataSourceProperties through Properties.getProperty, which +;; drops non-String values silently, so everything is passed as a string. +;; preparedStatementCacheSizeMiB only accepts a whole-MiB integer: the +;; configured size (bytes) is converted up, so any positive value keeps the +;; cache enabled and 0 disables the cache. A query parameter of the same name +;; in the URI wins over these properties: pgjdbc ignores the Properties value +;; when the URL carries it. +(defn- bytes->mib-ceil + [bytes] + (Math/ceilDiv (long bytes) (* 1024 1024))) + (defn- create-datasource-config [{:keys [::uri] :as cfg}] @@ -132,6 +159,13 @@ (.setConnectionInitSql initsql) (.setInitializationFailTimeout -1)) + (some->> (::prepared-statement-cache-threshold cfg) (str) + (.addDataSourceProperty config "prepareThreshold")) + (some->> (::prepared-statement-cache-queries cfg) (str) + (.addDataSourceProperty config "preparedStatementCacheQueries")) + (some->> (::prepared-statement-cache-size cfg) (bytes->mib-ceil) (str) + (.addDataSourceProperty config "preparedStatementCacheSizeMiB")) + (->> (mtx/get-registry (::mtx/metrics cfg)) (PrometheusMetricsTrackerFactory.) (.setMetricsTrackerFactory config)) diff --git a/backend/src/app/main.clj b/backend/src/app/main.clj index 26b6552e92..cef4a21b03 100644 --- a/backend/src/app/main.clj +++ b/backend/src/app/main.clj @@ -425,6 +425,9 @@ ::db/read-only (cf/get :database-readonly false) ::db/min-size (cf/get :database-min-pool-size) ::db/max-size (cf/get :database-max-pool-size) + ::db/prepared-statement-cache-threshold (cf/get :database-prepared-statement-cache-threshold) + ::db/prepared-statement-cache-queries (cf/get :database-prepared-statement-cache-queries) + ::db/prepared-statement-cache-size (cf/get :database-prepared-statement-cache-size) ::mtx/metrics (ig/ref ::mtx/metrics)} ;; Default netty IO pool (shared between several services) diff --git a/backend/src/app/srepl/binfile.clj b/backend/src/app/srepl/binfile.clj deleted file mode 100644 index 43cb69f8f3..0000000000 --- a/backend/src/app/srepl/binfile.clj +++ /dev/null @@ -1,41 +0,0 @@ -;; This Source Code Form is subject to the terms of the Mozilla Public -;; License, v. 2.0. If a copy of the MPL was not distributed with this -;; file, You can obtain one at http://mozilla.org/MPL/2.0/. -;; -;; Copyright (c) KALEIDOS SUBSIDIARY SL - -(ns app.srepl.binfile - (:require - [app.binfile.v2 :as binfile.v2] - [app.common.uuid :as uuid] - [app.db :as db] - [app.srepl.helpers :as h] - [app.system :as sys] - [cuerdas.core :as str])) - -(defn export-team! - [team-id] - (let [team-id (h/parse-uuid team-id)] - (binfile.v2/export-team! sys/system team-id))) - -(defn import-team! - [path & {:keys [owner rollback?] :or {rollback? true}}] - (db/tx-run! (assoc sys/system ::db/rollback rollback?) - (fn [cfg] - (let [team (binfile.v2/import-team! cfg path) - owner (cond - (string? owner) - (db/get* cfg :profile {:email (str/lower owner)}) - (uuid? owner) - (db/get* cfg :profile {:id owner}))] - - (when owner - (db/insert! cfg :team-profile-rel - {:id (uuid/next) - :team-id (:id team) - :profile-id (:id owner) - :is-admin true - :is-owner true - :can-edit true})) - - team)))) diff --git a/backend/test/backend_tests/config_test.clj b/backend/test/backend_tests/config_test.clj index fa168708ef..5e0e2418b8 100644 --- a/backend/test/backend_tests/config_test.clj +++ b/backend/test/backend_tests/config_test.clj @@ -76,6 +76,23 @@ :worker-default-parallelism 4)] (t/is (= 4 (cf/get :worker-default-parallelism 1)))))) +(t/deftest prepared-statement-cache-keys + (t/testing "negative sizes are rejected; the threshold takes -1 or non-negatives" + (t/is (not (cf/validate-config (assoc cf/config :database-prepared-statement-cache-queries -1)))) + (t/is (not (cf/validate-config (assoc cf/config :database-prepared-statement-cache-size -1)))) + (t/is (not (cf/validate-config (assoc cf/config :database-prepared-statement-cache-threshold -2)))) + (t/is (cf/validate-config (assoc cf/config :database-prepared-statement-cache-threshold -1)))) + (t/testing "a -1 threshold survives the environment decoding and validation" + (let [cfg (merge cf/config + (cf/decode-config {:database-prepared-statement-cache-threshold "-1"}))] + (t/is (= -1 (:database-prepared-statement-cache-threshold cfg))) + (t/is (cf/validate-config cfg)))) + (t/testing "a decoded -2 threshold fails validation" + (let [cfg (merge cf/config + (cf/decode-config {:database-prepared-statement-cache-threshold "-2"}))] + (t/is (= -2 (:database-prepared-statement-cache-threshold cfg))) + (t/is (not (cf/validate-config cfg)))))) + (t/deftest jobs-user-ttl (t/testing "the env value is decoded as a duration" (let [decoded (cf/decode-config {:jobs-user-ttl "72h"})] diff --git a/backend/test/backend_tests/db_test.clj b/backend/test/backend_tests/db_test.clj index b0f20cb256..71de5b7b24 100644 --- a/backend/test/backend_tests/db_test.clj +++ b/backend/test/backend_tests/db_test.clj @@ -36,6 +36,113 @@ (do (Thread/sleep 10) (recur (dec attempt))))))) +(t/deftest pool-passes-prepared-statement-cache-properties + ;; The driver reads dataSourceProperties through Properties.getProperty, + ;; which keeps only String values, so the wiring must set them as strings. + ;; Comparing the literals alone cannot catch a property the driver ignores, + ;; so the threshold cases also read it back from a real connection. + (let [metrics (ig/init-key :app.metrics/metrics {:default {}}) + init-pool (fn [opts] + (ig/init-key :app.db/pool + (merge {:app.db/uri (java.net.URI. "postgresql://localhost/test") + :app.metrics/metrics metrics + :app.db/name :test-prepared-statement-cache} + opts))) + with-connection (fn [opts f] + (let [ds (init-pool (merge {:app.db/uri (:database-uri th/config) + :app.db/username (:database-username th/config) + :app.db/password (:database-password th/config)} + opts))] + (try + (with-open [conn (.getConnection ^HikariDataSource ds)] + (f conn)) + (finally + (.close ^HikariDataSource ds))))) + prepare-threshold (fn [conn] + (.getPrepareThreshold + ^org.postgresql.PGConnection + (.unwrap conn org.postgresql.PGConnection)))] + (t/testing "absent keys stay absent" + (let [ds (init-pool {})] + (t/is (empty? (.getDataSourceProperties ^HikariDataSource ds))) + (.close ^HikariDataSource ds))) + (t/testing "the driver defaults apply when no option is set" + (with-connection {} (fn [conn] (t/is (= 5 (prepare-threshold conn)))))) + (t/testing "configured keys reach the driver as strings" + (let [ds (init-pool {:app.db/prepared-statement-cache-threshold 1 + :app.db/prepared-statement-cache-queries 512 + :app.db/prepared-statement-cache-size 10485760}) + props (.getDataSourceProperties ^HikariDataSource ds)] + (t/is (= "1" (.getProperty props "prepareThreshold"))) + (t/is (= "512" (.getProperty props "preparedStatementCacheQueries"))) + (t/is (= "10" (.getProperty props "preparedStatementCacheSizeMiB"))) + (.close ^HikariDataSource ds))) + (t/testing "a configured threshold reaches the driver" + (with-connection {:app.db/prepared-statement-cache-threshold 1} + (fn [conn] (t/is (= 1 (prepare-threshold conn))))) + (with-connection {:app.db/prepared-statement-cache-threshold 0} + (fn [conn] (t/is (= 0 (prepare-threshold conn)))))) + (t/testing "a -1 threshold reaches the driver (forceBinary)" + (let [ds (init-pool {:app.db/prepared-statement-cache-threshold -1})] + (t/is (= "-1" (.getProperty (.getDataSourceProperties ^HikariDataSource ds) + "prepareThreshold"))) + (.close ^HikariDataSource ds))) + (t/testing "a zero threshold reaches the driver (server-side statements off)" + (let [ds (init-pool {:app.db/prepared-statement-cache-threshold 0})] + (t/is (= "0" (.getProperty (.getDataSourceProperties ^HikariDataSource ds) + "prepareThreshold"))) + (.close ^HikariDataSource ds))) + (t/testing "a positive sub-MiB size rounds up so the cache stays enabled" + (let [ds (init-pool {:app.db/prepared-statement-cache-size 1})] + (t/is (= "1" (.getProperty (.getDataSourceProperties ^HikariDataSource ds) + "preparedStatementCacheSizeMiB"))) + (.close ^HikariDataSource ds))) + (t/testing "the MiB ceiling rounds a byte over the boundary up" + (let [ds (init-pool {:app.db/prepared-statement-cache-size 1048577})] + (t/is (= "2" (.getProperty (.getDataSourceProperties ^HikariDataSource ds) + "preparedStatementCacheSizeMiB"))) + (.close ^HikariDataSource ds))) + (t/testing "the cache limits hold a closed statement in the server" + ;; A closed statement stays prepared in the server only while the + ;; driver's cache holds it, so counting it in pg_prepared_statements is + ;; what tells an accepted property from an effective one. + (let [cached? (fn [opts] + (with-connection + (merge {:app.db/prepared-statement-cache-threshold 1} opts) + (fn [conn] + (dotimes [_ 2] + (with-open [stmt (.prepareStatement ^Connection conn "SELECT ?::int AS cache_probe")] + (.setInt stmt 1 1) + (with-open [rs (.executeQuery stmt)] (.next rs)))) + (-> (db/exec-one! conn ["SELECT count(*) AS n FROM pg_prepared_statements + WHERE statement LIKE '%AS cache_probe'"]) + :n + pos?))))] + (t/testing "a positive sub-MiB size keeps the cache enabled" + (t/is (cached? {:app.db/prepared-statement-cache-size 1}))) + (t/testing "zero size disables the cache" + (t/is (not (cached? {:app.db/prepared-statement-cache-size 0})))) + (t/testing "zero queries disables the cache" + (t/is (not (cached? {:app.db/prepared-statement-cache-queries 0})))))))) + +(t/deftest pool-rejects-a-non-minus-one-negative-threshold + ;; The pool boundary mirrors app.config: -1 prepares from the first + ;; execution, 0 stops server-side prepared statements, and any other + ;; negative is a typo the driver would silently clamp to -1. + (let [options {:app.db/uri (:database-uri th/config) + :app.db/username (:database-username th/config) + :app.db/password (:database-password th/config) + :app.metrics/metrics (ig/init-key :app.metrics/metrics {:default {}})}] + (t/testing "a -2 threshold fails the pool schema" + (t/is (thrown? clojure.lang.ExceptionInfo + (ig/assert-key :app.db/pool + (assoc options :app.db/prepared-statement-cache-threshold -2))))) + (t/testing "-1 and 0 pass the pool schema" + (t/is (nil? (ig/assert-key :app.db/pool + (assoc options :app.db/prepared-statement-cache-threshold -1)))) + (t/is (nil? (ig/assert-key :app.db/pool + (assoc options :app.db/prepared-statement-cache-threshold 0))))))) + (t/deftest pool-requires-metrics ;; Metrics is required by the pool schema: a pool wired without it ;; fails at the boundary (AssertionError when asserts are enabled) or diff --git a/backend/test/backend_tests/rpc_file_test.clj b/backend/test/backend_tests/rpc_file_test.clj index e45794a739..6019839a59 100644 --- a/backend/test/backend_tests/rpc_file_test.clj +++ b/backend/test/backend_tests/rpc_file_test.clj @@ -13,6 +13,7 @@ [app.common.pprint :as pp] [app.common.thumbnails :as thc] [app.common.time :as ct] + [app.common.types.page :as ctp] [app.common.types.shape :as cts] [app.common.types.tokens-lib :as ctob] [app.common.types.tokens-status :as ctos] @@ -3433,3 +3434,35 @@ (t/is (= (into #{} (map :id) (:result miss)) (into #{} (map :id) (:result hit)))) (t/is (every? #(not (contains? % :migrations)) (:result hit)))))))) + +(t/deftest add-page-rejects-an-invalid-shape + ;; Duplicate page and the undo of a page deletion send the whole page as + ;; the client holds it. A tab that loaded the file before a migration + ;; ran holds the old format, here camelCase svg attrs (#13107), and the + ;; page must not reach the file: the next migrating read would reject + ;; the whole file. + (let [profile (th/create-profile* 1 {:is-active true}) + file (th/create-file* 1 {:profile-id (:id profile) + :project-id (:default-project-id profile) + :is-shared false}) + page-id (uuid/next) + shape-id (uuid/next) + shape (cts/setup-shape {:id shape-id :type :rect :name "rect" + :x 0 :y 0 :width 10 :height 10 + :parent-id uuid/zero :frame-id uuid/zero + :svg-attrs {:fillRule "nonzero"}}) + page (-> (ctp/make-empty-page {:id page-id :name "Copy"}) + (assoc-in [:objects shape-id] shape) + (update-in [:objects uuid/zero :shapes] (fnil conj []) shape-id)) + out (th/command! {::th/type :update-file + ::rpc/profile-id (:id profile) + :id (:id file) + :session-id (uuid/random) + :revn 0 + :vern 0 + :features cfeat/supported-features + :changes [{:type :add-page :id page-id :page page}]}) + stored (bfc/get-file th/*system* (:id file) :migrate? false :realize? true)] + (t/is (th/ex-of-type? (:error out) :assertion)) + (t/is (th/ex-of-code? (:error out) :data-validation)) + (t/is (not (contains? (get-in stored [:data :pages-index]) page-id))))) diff --git a/backend/test/backend_tests/rpc_file_update_test.clj b/backend/test/backend_tests/rpc_file_update_test.clj index c5a8ba9b6d..34e4fe8b00 100644 --- a/backend/test/backend_tests/rpc_file_update_test.clj +++ b/backend/test/backend_tests/rpc_file_update_test.clj @@ -282,18 +282,20 @@ (defn- update-file-logged "Updates the file and returns the response in `:out` and, in `:rejected`, - the props of the log record of the rejection, if any." + the props of the log record of the rejection, if any. Keeps the watch + until the record arrives: the logger writes it on another thread, so it + can land after the update returns." [ctx params] - (let [records (atom []) - out (try - (add-watch l/log-record ::capture - (fn [_ _ _ record] (swap! records conj record))) - (update-file ctx params) - (finally - (remove-watch l/log-record ::capture)))] - {:out out - :rejected (some-> (wait-for-record records "update-file commit rejected") - ::l/props)})) + (let [records (atom [])] + (try + (add-watch l/log-record ::capture + (fn [_ _ _ record] (swap! records conj record))) + (let [out (update-file ctx params)] + {:out out + :rejected (some-> (wait-for-record records "update-file commit rejected") + ::l/props)}) + (finally + (remove-watch l/log-record ::capture))))) (t/deftest bisects-the-commits-to-find-the-invalid-one (let [{:keys [page-id] :as ctx} (create-file) diff --git a/common/src/app/common/data/macros.cljc b/common/src/app/common/data/macros.cljc index 4a6bd5b2a9..39bba7db72 100644 --- a/common/src/app/common/data/macros.cljc +++ b/common/src/app/common/data/macros.cljc @@ -123,6 +123,27 @@ (list 'js* (c/str "(~{}?." (str/snake prop) "?? ~{})") obj (list 'cljs.core/get obj prop)) (list `c/get obj prop))) +#?(:cljs (goog-define lazy-delay? false)) + +(defmacro lazy-delay + "Like `delay`, but only lazy when the build asks for it. + + With the default value of `lazy-delay?` the expression is evaluated + immediately, so errors raised while computing it (an invalid schema + definition, for instance) surface when the namespace is loaded instead of + on the first deref. Production builds set `lazy-delay?` to `true` through + `:closure-defines`; then the expression is fully deferred and the compiler + drops the eager branch. On the JVM the expression is always evaluated + immediately." + [expr] + (if (:ns &env) + `(if lazy-delay? + (delay ~expr) + (let [value# ~expr] + (delay value#))) + `(let [value# ~expr] + (delay value#)))) + (defn runtime-assert [hint f] (try diff --git a/common/src/app/common/files/changes.cljc b/common/src/app/common/files/changes.cljc index b2b0e6de60..b81764e580 100644 --- a/common/src/app/common/files/changes.cljc +++ b/common/src/app/common/files/changes.cljc @@ -930,6 +930,13 @@ (when (and id name page) (ex/raise :type :conflict :hint "id+name or page should be provided, never both")) + + ;; NOTE: we only perform hard validation on backend. The page arrives + ;; whole (duplicate page, undo of a page deletion), as the client holds + ;; it, so each of its shapes is validated as `:add-obj` validates one. + #?(:clj (doseq [shape (vals (:objects page))] + (validate-shape shape (:id page)))) + (let [page (if (and (string? name) (uuid? id)) (ctp/make-empty-page {:id id :name name}) page)] diff --git a/common/src/app/common/files/migrations.cljc b/common/src/app/common/files/migrations.cljc index 1e364407f1..bc55f4b88d 100644 --- a/common/src/app/common/files/migrations.cljc +++ b/common/src/app/common/files/migrations.cljc @@ -2422,19 +2422,22 @@ (update :pages-index d/update-vals update-container) (d/update-when :components d/update-vals update-container)))) -(defmethod migrate-data "0035-fix-shape-svg-attrs-and-defs" +(defmethod migrate-data "0037-fix-shape-svg-attrs-and-defs" ;; Stored svg attribute keys are kebab-case (issue #11947): the v3 reader ;; already outputs kebab and birth normalizes any outside spelling, so ;; export/import converges to the same state. Old birth and the cleaner ;; era stored camelCase React prop names instead, which the reader - ;; kebab-ized on import beyond repair. Convert once, with the same - ;; birth transforms: `:svg-attrs` and `:svg-defs` node attrs go through + ;; kebab-ized on import beyond repair. Convert with the same birth + ;; transforms: `:svg-attrs` and `:svg-defs` node attrs go through ;; the whitelist+kebab transform (unknown attrs drop, same as birth), ;; while svg-raw `:content` attrs are kebab-ized spelling-only (unknown ;; keys kept: the sidebar reads keys outside the whitelist). Kebab ;; input passes through unchanged, so re-running is safe. This replaces - ;; "0021-fix-shape-svg-attrs" (id withdrawn from `available-migrations` - ;; below): files that recorded it migrate again through this one. + ;; "0021-fix-shape-svg-attrs" and "0035-fix-shape-svg-attrs-and-defs" + ;; (ids withdrawn from `available-migrations` below): files that + ;; recorded either migrate again through this one. 0035 was withdrawn + ;; because a client that loaded a file before 0035 ran could still save + ;; camelCase keys into it afterwards (#13107). ;; Node tags in defs and svg-raw content are stored as keywords; files ;; that hold them as strings get them converted here, since the closed ;; node schemas reject strings. `:svg-transform` held as a plain map @@ -2579,5 +2582,5 @@ "0033-normalize-main-instance-names" "0034-fix-path-content-offset" "0035-fix-dangling-children" - "0035-fix-shape-svg-attrs-and-defs" - "0036-clear-nil-keys-in-objects"])) + "0036-clear-nil-keys-in-objects" + "0037-fix-shape-svg-attrs-and-defs"])) diff --git a/common/src/app/common/flags.cljc b/common/src/app/common/flags.cljc index 4439a5b316..f7be0be0da 100644 --- a/common/src/app/common/flags.cljc +++ b/common/src/app/common/flags.cljc @@ -238,6 +238,7 @@ :enable-inspect-styles :enable-feature-fdata-objects-map :enable-feature-render-wasm + :enable-feature-text-editor-wasm-html-paste :enable-token-import-from-library :enable-render-switch :enable-render-wasm-info diff --git a/common/src/app/common/render_wasm/helpers.cljc b/common/src/app/common/render_wasm/helpers.cljc index 7a73d6f94e..64ee812a7a 100644 --- a/common/src/app/common/render_wasm/helpers.cljc +++ b/common/src/app/common/render_wasm/helpers.cljc @@ -26,12 +26,10 @@ (throw (ex-info hint {:type :wasm-error :code code :hint hint :fn fn-name} cause))))) (defmacro call - "A helper for calling a wasm function. - Catches any exception thrown by the WASM function, reads the error code from - WASM when available, and routes it based on the error type: - - :wasm-non-blocking: call app.main.errors/on-error (eventually, shows a toast and logs the error) - - :wasm-critical or unknown: throws an exception to be handled by the global error handler (eventually, shows the internal error page) - Restores the stack pointer on failure, since a trap leaves its shadow-stack frames behind." + "Calls the WASM function `name`. On failure, throws a `:wasm-error` whose + `:code` comes from `error-code`, or `:wasm-critical` when WASM set none. + Restores the stack pointer on failure, since a trap leaves its shadow-stack + frames behind." [module name & params] (let [fn-sym (with-meta (gensym "fn-") {:tag 'function}) module-sym (gensym "module") diff --git a/common/src/app/common/schema.cljc b/common/src/app/common/schema.cljc index f4fcab85da..14fa04f919 100644 --- a/common/src/app/common/schema.cljc +++ b/common/src/app/common/schema.cljc @@ -11,6 +11,7 @@ #?(:clj [malli.dev.pretty :as mdp]) #?(:clj [malli.dev.virhe :as v]) [app.common.data :as d] + [app.common.data.macros :as dm] [app.common.i18n :as i18n :refer [tr]] [app.common.json :as json] [app.common.math :as mth] @@ -320,25 +321,32 @@ ([s] (lookup sr/default-registry s)) ([registry s] (schema (mr/schema registry s)))) -(defn check-fn - "Create a predefined check function" - [s & {:keys [hint type code]}] - (let [s #?(:clj - (schema s) - :cljs - (try - (schema s) - (catch :default cause - (let [data (ex-data cause)] - (if (= :malli.core/invalid-schema (:type data)) - (throw (ex-info - (str "Invalid schema\n" - (pp/pprint-str (:data data))) - {})) - (throw cause)))))) +(defn- resolve-schema + "Resolves the schema definition. On ClojureScript it rewrites the malli + invalid schema error with a human readable message." + [s] + #?(:clj (schema s) + :cljs (try + (schema s) + (catch :default cause + (let [data (ex-data cause)] + (if (= :malli.core/invalid-schema (:type data)) + (throw (ex-info + (str "Invalid schema\n" + (pp/pprint-str (:data data))) + {})) + (throw cause))))))) - validator* (delay (m/validator s)) - explainer* (delay (m/explainer s)) +(defn check-fn + "Create a predefined check function. + + The schema definition is resolved lazily on production builds, and + eagerly on development ones, so invalid schemas are detected early + while developing. See `app.common.data.macros/lazy-delay`." + [s & {:keys [hint type code]}] + (let [s* (dm/lazy-delay (resolve-schema s)) + validator* (delay (m/validator @s*)) + explainer* (delay (m/explainer @s*)) hint (or ^boolean hint "check error") type (or ^boolean type :assertion) code (or ^boolean code :data-validation)] diff --git a/common/src/app/common/svg.cljc b/common/src/app/common/svg.cljc index 15a3bff12d..7349e7e1d8 100644 --- a/common/src/app/common/svg.cljc +++ b/common/src/app/common/svg.cljc @@ -673,15 +673,32 @@ {} attrs))) +(defn- drop-non-scalar-values + "Removes the entries whose value is not a string, a number or a + boolean: the stored schema admits nothing else." + [attrs] + (reduce-kv (fn [res k v] + (if (or (string? v) (number? v) (boolean? v)) + res + (dissoc res k))) + attrs + attrs)) + (defn attrs->kebab-props "Like `attrs->props` (same whitelist, trim and `:style` parsing) but with kebab-case keys, the stored convention for `:svg-attrs` and - `:svg-defs` node attrs. Kebab is a fixed point: running it twice - returns the same map." + `:svg-defs` node attrs. Values that are not scalars, at the top level + or inside `:style`, are dropped: files written by old versions hold + maps there, which no renderer can use and the closed schema rejects. + Kebab is a fixed point: running it twice returns the same map." [attrs] - (-> attrs - (attrs->props) - (d/kebab-keys))) + (let [props (attrs->props attrs) + style (some-> (:style props) (drop-non-scalar-values) (not-empty))] + (-> props + (dissoc :style) + (drop-non-scalar-values) + (cond-> (some? style) (assoc :style style)) + (d/kebab-keys)))) (defn- normalize-node-tree "Walks a svg node tree applying `attrs-fn` to every `:attrs` map. Tags diff --git a/common/src/app/common/types/tokens_lib.cljc b/common/src/app/common/types/tokens_lib.cljc index d045e93e55..950cde98d9 100644 --- a/common/src/app/common/types/tokens_lib.cljc +++ b/common/src/app/common/types/tokens_lib.cljc @@ -1678,7 +1678,7 @@ (every? empty? (vals sets)) (and (empty? sets) (empty? themes))) (throw (ex-info "the file doesn't contain any tokens" - {:error/code :error.import/invalid-json-data}))) + {:error/code :error.import/no-token-files-found}))) (as-> (make-tokens-lib) library (reduce-kv (fn [library name tokens] @@ -1817,8 +1817,6 @@ A `:dtcg` file is not meant to import back unchanged: tokens come back with their DTCG type and value, a spacing token as a dimension, `4` as `4px` and `rgb(255, 0, 0)` as `#ff0000`." - ([tokens-lib] - (tsw/write-files (tokens-lib->document tokens-lib (legacy-activation tokens-lib)))) ([tokens-lib tokens-status] (export-dtcg-multi-file tokens-lib tokens-status nil)) ([tokens-lib tokens-status opts] diff --git a/common/test/common_tests/data_macros_test.cljc b/common/test/common_tests/data_macros_test.cljc new file mode 100644 index 0000000000..8759cd7570 --- /dev/null +++ b/common/test/common_tests/data_macros_test.cljc @@ -0,0 +1,33 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns common-tests.data-macros-test + (:require + [app.common.data.macros :as dm] + [clojure.test :as t])) + +(defn- eager? + "Mirrors the build mode check performed by `dm/lazy-delay`." + [] + #?(:clj true + :cljs (not dm/lazy-delay?))) + +(t/deftest lazy-delay-returns-a-delay-with-the-value + (let [value (dm/lazy-delay (str "foo" "bar"))] + (t/is (delay? value)) + (t/is (= "foobar" @value)))) + +(t/deftest lazy-delay-evaluates-immediately-by-default + (let [calls (atom 0) + _ (dm/lazy-delay (swap! calls inc))] + (t/is (= (if (eager?) 1 0) @calls)))) + +(t/deftest lazy-delay-evaluates-the-expression-only-once + (let [calls (atom 0) + value (dm/lazy-delay (swap! calls inc))] + (t/is (= 1 @value)) + (t/is (= 1 @value)) + (t/is (= 1 @calls)))) diff --git a/common/test/common_tests/files_migrations_0035_test.cljc b/common/test/common_tests/files_migrations_0035_test.cljc index 350b669fed..c64ce69527 100644 --- a/common/test/common_tests/files_migrations_0035_test.cljc +++ b/common/test/common_tests/files_migrations_0035_test.cljc @@ -7,8 +7,6 @@ (ns common-tests.files-migrations-0035-test (:require [app.common.files.migrations :as cfm] - [app.common.geom.matrix :as gmt] - [app.common.geom.rect :as grc] [app.common.types.file :as ctf] [app.common.types.objects-map :as omap] [app.common.types.shape :as cts] @@ -189,308 +187,3 @@ (t/is (some? (get objects' frame-id)) "the frame that exists is kept")))) -;; ====================================================================== -;; 0035-fix-shape-svg-attrs-and-defs -;; Stored svg attribute keys are kebab-case (issue #11947). Old birth and -;; the cleaner era stored camelCase React prop names, which the v3 reader -;; kebab-ized on import beyond repair. This migration converts once, with -;; the same birth transforms: `:svg-attrs` and `:svg-defs` node attrs go -;; through the whitelist+kebab transform, svg-raw `:content` attrs are -;; kebab-ized spelling-only. It replaces "0021-fix-shape-svg-attrs" (id -;; withdrawn): files that recorded it migrate again through this one. -;; ====================================================================== - -(def ^:private migration-id - "0035-fix-shape-svg-attrs-and-defs") - -(defn- make-shape - [shape-id shape] - (assoc shape :id shape-id)) - -(defn- camel-shape - [shape-id] - (make-shape shape-id - {:type :rect - :svg-attrs {:fillRule "evenodd" - :stroke-width "2" - :stroke-style "dotted" - :data-foo "drop-me"}})) - -(defn- camel-defs - [] - {"g1" {:tag :linearGradient - :attrs {:id "g1" - :gradientUnits "userSpaceOnUse" - :xlink:href "#base" - :data-x "drop-me"} - :content [{:tag :stop - :attrs {:offset "0" - :stopColor "#ffffff"} - :content []}]}}) - -(t/deftest migration-0035-converts-camel-svg-attrs-to-kebab-in-pages - (let [shape-id (uuid/next) - page-id (uuid/next) - data {:pages-index - {page-id - {:objects - {shape-id (camel-shape shape-id)}}}} - data' (cfm/migrate-data data migration-id) - attrs (get-in data' [:pages-index page-id :objects shape-id :svg-attrs])] - (t/is (= "evenodd" (:fill-rule attrs))) - (t/is (= "2" (:stroke-width attrs))) - (t/is (= "dotted" (:stroke-style attrs)) "penpot-extra-attrs survive") - (t/is (not (contains? attrs :data-foo)) "unknown attrs are dropped") - (t/is (not (contains? attrs :fillRule)) "no camel keys survive"))) - -(t/deftest migration-0035-converts-camel-svg-attrs-to-kebab-in-components - (let [shape-id (uuid/next) - component-id (uuid/next) - data {:components - {component-id - {:objects - {shape-id (camel-shape shape-id)}}}} - data' (cfm/migrate-data data migration-id) - attrs (get-in data' [:components component-id :objects shape-id :svg-attrs])] - (t/is (= "evenodd" (:fill-rule attrs))) - (t/is (not (contains? attrs :fillRule)) "no camel keys survive"))) - -(t/deftest migration-0035-normalizes-def-node-attrs - (let [shape-id (uuid/next) - page-id (uuid/next) - data {:pages-index - {page-id - {:objects - {shape-id (make-shape shape-id - {:type :rect - :svg-defs (camel-defs)})}}}} - data' (cfm/migrate-data data migration-id) - grad (get-in data' [:pages-index page-id :objects shape-id :svg-defs "g1"])] - (t/is (= :linearGradient (:tag grad)) "tags are untouched") - (t/is (= "g1" (get-in grad [:attrs :id])) ":id survives") - (t/is (= "userSpaceOnUse" (get-in grad [:attrs :gradient-units]))) - (t/is (= "#base" (get-in grad [:attrs :xlink-href]))) - (t/is (not (contains? (:attrs grad) :data-x)) "unknown def attrs are dropped") - (t/is (= "#ffffff" (get-in grad [:content 0 :attrs :stop-color])) - "nested stops are normalized"))) - -(t/deftest migration-0035-kebabizes-content-spelling-only - (let [shape-id (uuid/next) - page-id (uuid/next) - raw-id (uuid/next) - data {:pages-index - {page-id - {:objects - {shape-id (make-shape shape-id - {:type :path - :content [{:marker "untouched"}]}) - raw-id (make-shape raw-id - {:type :svg-raw - :content {:tag :g - :attrs {:strokeWidth "2" - :data-foo "keep-me"} - :content [{:tag :path - :attrs {:strokeLinecap "round"} - :content []}]}})}}}} - data' (cfm/migrate-data data migration-id) - path (get-in data' [:pages-index page-id :objects shape-id]) - raw (get-in data' [:pages-index page-id :objects raw-id]) - content (:content raw)] - (t/is (= [{:marker "untouched"}] (:content path)) - "non-svg-raw content is untouched") - (t/is (= "2" (get-in content [:attrs :stroke-width]))) - (t/is (= "keep-me" (get-in content [:attrs :data-foo])) - "unknown content keys survive") - (t/is (= "round" (get-in content [:content 0 :attrs :stroke-linecap]))))) - -(defn- string-tag-defs - [] - {:a {:tag "mask" - :attrs {:id "a" :maskUnits "userSpaceOnUse"} - :content [{:tag "rect" - :attrs {:width "10" :height "10" :fill "#FFF"} - :content []}]}}) - -(defn- string-tag-content - [] - {:tag "g" - :attrs {} - :content [{:tag "text" - :attrs {:fontSize "12"} - :content ["Hi"]} - {:tag "style" - :attrs nil - :content [".a{fill:red}"]}]}) - -(t/deftest migration-0035-keywordizes-string-tags - (let [shape-id (uuid/next) - raw-id (uuid/next) - page-id (uuid/next) - data {:pages-index - {page-id - {:objects - {shape-id (make-shape shape-id - {:type :rect - :svg-defs (string-tag-defs)}) - raw-id (make-shape raw-id - {:type :svg-raw - :content (string-tag-content)})}}}} - data' (cfm/migrate-data data migration-id) - mask (get-in data' [:pages-index page-id :objects shape-id :svg-defs :a]) - content (get-in data' [:pages-index page-id :objects raw-id :content])] - (t/is (= :mask (:tag mask))) - (t/is (= :rect (get-in mask [:content 0 :tag])) "nested def tags too") - (t/is (= "userSpaceOnUse" (get-in mask [:attrs :mask-units]))) - (t/is (= :g (:tag content))) - (t/is (= :text (get-in content [:content 0 :tag])) "nested content tags too") - (t/is (= ["Hi"] (get-in content [:content 0 :content]))) - (t/is (= {} (get-in content [:content 1 :attrs])) "nil attrs become an empty map") - (t/is (= data' (cfm/migrate-data data' migration-id)) "re-running is stable"))) - -(t/deftest migration-0035-string-tag-file-passes-file-validation - ;; `migrate-file` validates the whole file data after migrating: a file - ;; with string node tags or nil node attrs must come out valid, or it - ;; cannot be opened. - (let [rect-id (uuid/next) - raw-id (uuid/next) - file (ctf/make-file {:name "String svg tags"}) - page-id (first (get-in file [:data :pages])) - rect (-> (cts/setup-shape {:id rect-id :type :rect :name "rect"}) - (assoc :svg-defs (string-tag-defs))) - raw (-> (cts/setup-shape {:id raw-id :type :svg-raw :name "raw"}) - (assoc :content (string-tag-content))) - file (-> file - (assoc :migrations (disj cfm/available-migrations migration-id)) - (update-in [:data :pages-index page-id :objects] assoc rect-id rect raw-id raw) - (update-in [:data :pages-index page-id :objects uuid/zero :shapes] conj rect-id raw-id)) - file' (cfm/migrate-file file {})] - (t/is (= :mask (get-in file' [:data :pages-index page-id :objects rect-id :svg-defs :a :tag]))) - (t/is (= :g (get-in file' [:data :pages-index page-id :objects raw-id :content :tag]))))) - -(t/deftest migration-0035-turns-plain-map-svg-transform-into-matrix - ;; The svg defs renderer multiplies `:svg-transform` as a matrix; a - ;; plain map there renders as `matrix(NaN, ...)`. - (let [plain-id (uuid/next) - matrix-id (uuid/next) - page-id (uuid/next) - data {:pages-index - {page-id - {:objects - {plain-id (make-shape plain-id - {:type :path - :svg-transform {:a 1.0 :b 0.0 :c 0.0 :d 1.0 :e 10.0 :f 20.0}}) - matrix-id (make-shape matrix-id - {:type :path - :svg-transform (gmt/matrix 2 0 0 2 0 0)})}}}} - data' (cfm/migrate-data data migration-id) - plain' (get-in data' [:pages-index page-id :objects plain-id :svg-transform]) - matrix' (get-in data' [:pages-index page-id :objects matrix-id :svg-transform])] - (t/is (gmt/matrix? plain')) - (t/is (= (gmt/matrix 1 0 0 1 10 20) plain')) - (t/is (= (gmt/matrix 2 0 0 2 0 0) matrix') "matrices are left as they are") - (t/is (= data' (cfm/migrate-data data' migration-id)) "re-running is stable"))) - -(t/deftest migration-0035-leaves-kebab-files-equal - (let [shape-id (uuid/next) - page-id (uuid/next) - data {:pages-index - {page-id - {:objects - {shape-id (make-shape shape-id - {:type :rect - :svg-attrs {:fill-rule "evenodd" - :stroke-width "2"} - :svg-defs {"g1" {:tag :linearGradient - :attrs {:id "g1" - :gradient-units "userSpaceOnUse"} - :content []}}})}}}} - data' (cfm/migrate-data data migration-id)] - (t/is (= data data') "kebab is a fixed point"))) - -(t/deftest migration-0035-is-idempotent - (let [shape-id (uuid/next) - page-id (uuid/next) - data {:pages-index - {page-id - {:objects - {shape-id (camel-shape shape-id)}}}} - data' (cfm/migrate-data data migration-id)] - (t/is (= data' (cfm/migrate-data data' migration-id))))) - -(t/deftest migration-0035-normalizes-flat-svg-viewbox-to-rect - ;; Carried over from the withdrawn 0021: legacy files hold - ;; `:svg-viewbox` as a plain map instead of a rect record. - (let [shape-id (uuid/next) - page-id (uuid/next) - data {:pages-index - {page-id - {:objects - {shape-id (make-shape shape-id - {:type :rect - :svg-viewbox {:x 1 :y 2 :width 3 :height 4}})}}}} - data' (cfm/migrate-data data migration-id) - viewbox (get-in data' [:pages-index page-id :objects shape-id :svg-viewbox])] - (t/is (grc/rect? viewbox) "flat viewbox becomes a rect record") - (t/is (= 1 (:x viewbox))) - (t/is (= 2 (:y viewbox))) - (t/is (= 3 (:width viewbox))) - (t/is (= 4 (:height viewbox))) - (t/is (= data' (cfm/migrate-data data' migration-id)) - "re-running is stable"))) - -(t/deftest migration-0035-passes-non-maps-through - (let [shape-id (uuid/next) - page-id (uuid/next) - data {:pages-index - {page-id - {:objects - {shape-id (make-shape shape-id - {:type :rect - :svg-attrs "junk" - :svg-defs "junk"})}}}} - data' (cfm/migrate-data data migration-id) - shape (get-in data' [:pages-index page-id :objects shape-id])] - (t/is (= "junk" (:svg-attrs shape)) "non-maps reach validation untouched") - (t/is (= "junk" (:svg-defs shape))))) - -(t/deftest migration-0035-junk-viewbox-becomes-default-rect - ;; Unlike `:svg-attrs`/`:svg-defs` (passed through to validation), - ;; `:svg-viewbox` goes through `make-rect`, which does not throw on - ;; junk: it falls back to the default rect. Inherited from 0021. - (let [shape-id (uuid/next) - page-id (uuid/next) - data {:pages-index - {page-id - {:objects - {shape-id (make-shape shape-id - {:type :rect - :svg-viewbox "junk"})}}}} - data' (cfm/migrate-data data migration-id) - viewbox (get-in data' [:pages-index page-id :objects shape-id :svg-viewbox])] - (t/is (grc/rect? viewbox) "junk viewbox becomes a rect, not an error") - (t/is (= 0 (:x viewbox))) - (t/is (= 0.01 (:width viewbox))))) - -(t/deftest migration-0035-runs-through-file-migration - ;; A file that recorded the withdrawn "0021-fix-shape-svg-attrs" (and - ;; everything else current) still needs migration: 0021 left - ;; `available-migrations`, which re-arms it for 0035. - (let [shape-id (uuid/next) - file (ctf/make-file {:name "Legacy camel svg attrs"}) - page-id (first (get-in file [:data :pages])) - shape (-> (cts/setup-shape {:id shape-id :type :rect}) - (assoc :svg-attrs {:fillRule "evenodd"})) - file (-> file - (assoc :migrations (-> cfm/available-migrations - (disj migration-id) - (conj "0021-fix-shape-svg-attrs"))) - (assoc-in [:data :pages-index page-id :objects shape-id] shape)) - file' (cfm/migrate-file file {})] - (t/is (cfm/need-migration? file) "new migration detected") - (t/is (not (cfm/need-migration? file')) "new migration recorded") - (t/is (contains? (:migrations file') migration-id) "migration id persisted") - (t/is (not (contains? cfm/available-migrations "0021-fix-shape-svg-attrs")) - "the withdrawn 0021 id stays out: its camel-restoring transform must never run again") - (t/is (= "evenodd" - (get-in file' [:data :pages-index page-id :objects shape-id :svg-attrs :fill-rule])) - "migration repaired file data before schema validation"))) diff --git a/common/test/common_tests/files_migrations_0037_test.cljc b/common/test/common_tests/files_migrations_0037_test.cljc new file mode 100644 index 0000000000..eff0afb82a --- /dev/null +++ b/common/test/common_tests/files_migrations_0037_test.cljc @@ -0,0 +1,371 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns common-tests.files-migrations-0037-test + (:require + [app.common.files.migrations :as cfm] + [app.common.geom.matrix :as gmt] + [app.common.geom.rect :as grc] + [app.common.types.file :as ctf] + [app.common.types.shape :as cts] + [app.common.uuid :as uuid] + [clojure.test :as t])) + +;; ====================================================================== +;; 0037-fix-shape-svg-attrs-and-defs +;; Stored svg attribute keys are kebab-case (issue #11947). Old birth and +;; the cleaner era stored camelCase React prop names, which the v3 reader +;; kebab-ized on import beyond repair. This migration converts them with +;; the same birth transforms: `:svg-attrs` and `:svg-defs` node attrs go +;; through the whitelist+kebab transform, svg-raw `:content` attrs are +;; kebab-ized spelling-only. It replaces "0021-fix-shape-svg-attrs" and +;; "0035-fix-shape-svg-attrs-and-defs" (ids withdrawn): files that +;; recorded either migrate again through this one. +;; ====================================================================== + +(def ^:private migration-id + "0037-fix-shape-svg-attrs-and-defs") + +(defn- make-shape + [shape-id shape] + (assoc shape :id shape-id)) + +(defn- camel-shape + [shape-id] + (make-shape shape-id + {:type :rect + :svg-attrs {:fillRule "evenodd" + :stroke-width "2" + :stroke-style "dotted" + :data-foo "drop-me"}})) + +(defn- camel-defs + [] + {"g1" {:tag :linearGradient + :attrs {:id "g1" + :gradientUnits "userSpaceOnUse" + :xlink:href "#base" + :data-x "drop-me"} + :content [{:tag :stop + :attrs {:offset "0" + :stopColor "#ffffff"} + :content []}]}}) + +(t/deftest migration-0037-converts-camel-svg-attrs-to-kebab-in-pages + (let [shape-id (uuid/next) + page-id (uuid/next) + data {:pages-index + {page-id + {:objects + {shape-id (camel-shape shape-id)}}}} + data' (cfm/migrate-data data migration-id) + attrs (get-in data' [:pages-index page-id :objects shape-id :svg-attrs])] + (t/is (= "evenodd" (:fill-rule attrs))) + (t/is (= "2" (:stroke-width attrs))) + (t/is (= "dotted" (:stroke-style attrs)) "penpot-extra-attrs survive") + (t/is (not (contains? attrs :data-foo)) "unknown attrs are dropped") + (t/is (not (contains? attrs :fillRule)) "no camel keys survive"))) + +(t/deftest migration-0037-converts-camel-svg-attrs-to-kebab-in-components + (let [shape-id (uuid/next) + component-id (uuid/next) + data {:components + {component-id + {:objects + {shape-id (camel-shape shape-id)}}}} + data' (cfm/migrate-data data migration-id) + attrs (get-in data' [:components component-id :objects shape-id :svg-attrs])] + (t/is (= "evenodd" (:fill-rule attrs))) + (t/is (not (contains? attrs :fillRule)) "no camel keys survive"))) + +(t/deftest migration-0037-normalizes-def-node-attrs + (let [shape-id (uuid/next) + page-id (uuid/next) + data {:pages-index + {page-id + {:objects + {shape-id (make-shape shape-id + {:type :rect + :svg-defs (camel-defs)})}}}} + data' (cfm/migrate-data data migration-id) + grad (get-in data' [:pages-index page-id :objects shape-id :svg-defs "g1"])] + (t/is (= :linearGradient (:tag grad)) "tags are untouched") + (t/is (= "g1" (get-in grad [:attrs :id])) ":id survives") + (t/is (= "userSpaceOnUse" (get-in grad [:attrs :gradient-units]))) + (t/is (= "#base" (get-in grad [:attrs :xlink-href]))) + (t/is (not (contains? (:attrs grad) :data-x)) "unknown def attrs are dropped") + (t/is (= "#ffffff" (get-in grad [:content 0 :attrs :stop-color])) + "nested stops are normalized"))) + +(t/deftest migration-0037-kebabizes-content-spelling-only + (let [shape-id (uuid/next) + page-id (uuid/next) + raw-id (uuid/next) + data {:pages-index + {page-id + {:objects + {shape-id (make-shape shape-id + {:type :path + :content [{:marker "untouched"}]}) + raw-id (make-shape raw-id + {:type :svg-raw + :content {:tag :g + :attrs {:strokeWidth "2" + :data-foo "keep-me"} + :content [{:tag :path + :attrs {:strokeLinecap "round"} + :content []}]}})}}}} + data' (cfm/migrate-data data migration-id) + path (get-in data' [:pages-index page-id :objects shape-id]) + raw (get-in data' [:pages-index page-id :objects raw-id]) + content (:content raw)] + (t/is (= [{:marker "untouched"}] (:content path)) + "non-svg-raw content is untouched") + (t/is (= "2" (get-in content [:attrs :stroke-width]))) + (t/is (= "keep-me" (get-in content [:attrs :data-foo])) + "unknown content keys survive") + (t/is (= "round" (get-in content [:content 0 :attrs :stroke-linecap]))))) + +(defn- string-tag-defs + [] + {:a {:tag "mask" + :attrs {:id "a" :maskUnits "userSpaceOnUse"} + :content [{:tag "rect" + :attrs {:width "10" :height "10" :fill "#FFF"} + :content []}]}}) + +(defn- string-tag-content + [] + {:tag "g" + :attrs {} + :content [{:tag "text" + :attrs {:fontSize "12"} + :content ["Hi"]} + {:tag "style" + :attrs nil + :content [".a{fill:red}"]}]}) + +(t/deftest migration-0037-keywordizes-string-tags + (let [shape-id (uuid/next) + raw-id (uuid/next) + page-id (uuid/next) + data {:pages-index + {page-id + {:objects + {shape-id (make-shape shape-id + {:type :rect + :svg-defs (string-tag-defs)}) + raw-id (make-shape raw-id + {:type :svg-raw + :content (string-tag-content)})}}}} + data' (cfm/migrate-data data migration-id) + mask (get-in data' [:pages-index page-id :objects shape-id :svg-defs :a]) + content (get-in data' [:pages-index page-id :objects raw-id :content])] + (t/is (= :mask (:tag mask))) + (t/is (= :rect (get-in mask [:content 0 :tag])) "nested def tags too") + (t/is (= "userSpaceOnUse" (get-in mask [:attrs :mask-units]))) + (t/is (= :g (:tag content))) + (t/is (= :text (get-in content [:content 0 :tag])) "nested content tags too") + (t/is (= ["Hi"] (get-in content [:content 0 :content]))) + (t/is (= {} (get-in content [:content 1 :attrs])) "nil attrs become an empty map") + (t/is (= data' (cfm/migrate-data data' migration-id)) "re-running is stable"))) + +(t/deftest migration-0037-string-tag-file-passes-file-validation + ;; `migrate-file` validates the whole file data after migrating: a file + ;; with string node tags or nil node attrs must come out valid, or it + ;; cannot be opened. + (let [rect-id (uuid/next) + raw-id (uuid/next) + file (ctf/make-file {:name "String svg tags"}) + page-id (first (get-in file [:data :pages])) + rect (-> (cts/setup-shape {:id rect-id :type :rect :name "rect"}) + (assoc :svg-defs (string-tag-defs))) + raw (-> (cts/setup-shape {:id raw-id :type :svg-raw :name "raw"}) + (assoc :content (string-tag-content))) + file (-> file + (assoc :migrations (disj cfm/available-migrations migration-id)) + (update-in [:data :pages-index page-id :objects] assoc rect-id rect raw-id raw) + (update-in [:data :pages-index page-id :objects uuid/zero :shapes] conj rect-id raw-id)) + file' (cfm/migrate-file file {})] + (t/is (= :mask (get-in file' [:data :pages-index page-id :objects rect-id :svg-defs :a :tag]))) + (t/is (= :g (get-in file' [:data :pages-index page-id :objects raw-id :content :tag]))))) + +(t/deftest migration-0037-drops-map-valued-svg-attrs + ;; Files exported by old Penpot versions hold maps where an svg attr + ;; value should be, such as a `:filter` that once read + ;; `url(#filter_2)`. The closed `:svg-attrs` schema admits scalars + ;; only, so `migrate-file` must drop them or the file cannot be + ;; imported or opened. The values below are copied from the + ;; "Avataaars" and "Tutorial for beginners" templates. + (let [shape-id (uuid/next) + file (ctf/make-file {:name "Map-valued svg attrs"}) + page-id (first (get-in file [:data :pages])) + shape (-> (cts/setup-shape {:id shape-id :type :rect :name "rect"}) + (assoc :svg-attrs {:fillRule "evenodd" + :filter {(keyword "{") "{ \"url(#filter_2) nil}\""} + :mask {(keyword "url(-filter0-d-16837-160)") nil} + :style {:fill "red" + :filter {:a "b"}}})) + file (-> file + (assoc :migrations (disj cfm/available-migrations migration-id)) + (update-in [:data :pages-index page-id :objects] assoc shape-id shape) + (update-in [:data :pages-index page-id :objects uuid/zero :shapes] conj shape-id)) + file' (cfm/migrate-file file {}) + attrs (get-in file' [:data :pages-index page-id :objects shape-id :svg-attrs])] + (t/is (= {:fill-rule "evenodd" :style {:fill "red"}} attrs) + "map values are dropped, scalars are kept"))) + +(t/deftest migration-0037-turns-plain-map-svg-transform-into-matrix + ;; The svg defs renderer multiplies `:svg-transform` as a matrix; a + ;; plain map there renders as `matrix(NaN, ...)`. + (let [plain-id (uuid/next) + matrix-id (uuid/next) + page-id (uuid/next) + data {:pages-index + {page-id + {:objects + {plain-id (make-shape plain-id + {:type :path + :svg-transform {:a 1.0 :b 0.0 :c 0.0 :d 1.0 :e 10.0 :f 20.0}}) + matrix-id (make-shape matrix-id + {:type :path + :svg-transform (gmt/matrix 2 0 0 2 0 0)})}}}} + data' (cfm/migrate-data data migration-id) + plain' (get-in data' [:pages-index page-id :objects plain-id :svg-transform]) + matrix' (get-in data' [:pages-index page-id :objects matrix-id :svg-transform])] + (t/is (gmt/matrix? plain')) + (t/is (= (gmt/matrix 1 0 0 1 10 20) plain')) + (t/is (= (gmt/matrix 2 0 0 2 0 0) matrix') "matrices are left as they are") + (t/is (= data' (cfm/migrate-data data' migration-id)) "re-running is stable"))) + +(t/deftest migration-0037-leaves-kebab-files-equal + (let [shape-id (uuid/next) + page-id (uuid/next) + data {:pages-index + {page-id + {:objects + {shape-id (make-shape shape-id + {:type :rect + :svg-attrs {:fill-rule "evenodd" + :stroke-width "2"} + :svg-defs {"g1" {:tag :linearGradient + :attrs {:id "g1" + :gradient-units "userSpaceOnUse"} + :content []}}})}}}} + data' (cfm/migrate-data data migration-id)] + (t/is (= data data') "kebab is a fixed point"))) + +(t/deftest migration-0037-is-idempotent + (let [shape-id (uuid/next) + page-id (uuid/next) + data {:pages-index + {page-id + {:objects + {shape-id (camel-shape shape-id)}}}} + data' (cfm/migrate-data data migration-id)] + (t/is (= data' (cfm/migrate-data data' migration-id))))) + +(t/deftest migration-0037-normalizes-flat-svg-viewbox-to-rect + ;; Carried over from the withdrawn 0021: legacy files hold + ;; `:svg-viewbox` as a plain map instead of a rect record. + (let [shape-id (uuid/next) + page-id (uuid/next) + data {:pages-index + {page-id + {:objects + {shape-id (make-shape shape-id + {:type :rect + :svg-viewbox {:x 1 :y 2 :width 3 :height 4}})}}}} + data' (cfm/migrate-data data migration-id) + viewbox (get-in data' [:pages-index page-id :objects shape-id :svg-viewbox])] + (t/is (grc/rect? viewbox) "flat viewbox becomes a rect record") + (t/is (= 1 (:x viewbox))) + (t/is (= 2 (:y viewbox))) + (t/is (= 3 (:width viewbox))) + (t/is (= 4 (:height viewbox))) + (t/is (= data' (cfm/migrate-data data' migration-id)) + "re-running is stable"))) + +(t/deftest migration-0037-passes-non-maps-through + (let [shape-id (uuid/next) + page-id (uuid/next) + data {:pages-index + {page-id + {:objects + {shape-id (make-shape shape-id + {:type :rect + :svg-attrs "junk" + :svg-defs "junk"})}}}} + data' (cfm/migrate-data data migration-id) + shape (get-in data' [:pages-index page-id :objects shape-id])] + (t/is (= "junk" (:svg-attrs shape)) "non-maps reach validation untouched") + (t/is (= "junk" (:svg-defs shape))))) + +(t/deftest migration-0037-junk-viewbox-becomes-default-rect + ;; Unlike `:svg-attrs`/`:svg-defs` (passed through to validation), + ;; `:svg-viewbox` goes through `make-rect`, which does not throw on + ;; junk: it falls back to the default rect. Inherited from 0021. + (let [shape-id (uuid/next) + page-id (uuid/next) + data {:pages-index + {page-id + {:objects + {shape-id (make-shape shape-id + {:type :rect + :svg-viewbox "junk"})}}}} + data' (cfm/migrate-data data migration-id) + viewbox (get-in data' [:pages-index page-id :objects shape-id :svg-viewbox])] + (t/is (grc/rect? viewbox) "junk viewbox becomes a rect, not an error") + (t/is (= 0 (:x viewbox))) + (t/is (= 0.01 (:width viewbox))))) + +(t/deftest migration-0037-reruns-for-files-that-recorded-withdrawn-0021 + ;; A file that recorded the withdrawn "0021-fix-shape-svg-attrs" (and + ;; everything else current) still needs migration: 0021 left + ;; `available-migrations`, which re-arms it for 0037. + (let [shape-id (uuid/next) + file (ctf/make-file {:name "Legacy camel svg attrs"}) + page-id (first (get-in file [:data :pages])) + shape (-> (cts/setup-shape {:id shape-id :type :rect}) + (assoc :svg-attrs {:fillRule "evenodd"})) + file (-> file + (assoc :migrations (-> cfm/available-migrations + (disj migration-id) + (conj "0021-fix-shape-svg-attrs"))) + (assoc-in [:data :pages-index page-id :objects shape-id] shape)) + file' (cfm/migrate-file file {})] + (t/is (cfm/need-migration? file) "new migration detected") + (t/is (not (cfm/need-migration? file')) "new migration recorded") + (t/is (contains? (:migrations file') migration-id) "migration id persisted") + (t/is (not (contains? cfm/available-migrations "0021-fix-shape-svg-attrs")) + "the withdrawn 0021 id stays out of available-migrations") + (t/is (= "evenodd" + (get-in file' [:data :pages-index page-id :objects shape-id :svg-attrs :fill-rule])) + "migration repaired file data before schema validation"))) + +(t/deftest migration-0037-reruns-for-files-that-recorded-withdrawn-0035 + ;; A file that recorded 0035 holds camelCase keys when a client that + ;; loaded it before 0035 ran saved them afterwards (#13107). 0035 also + ;; left `available-migrations`, so 0037 must run on it again. + (let [shape-id (uuid/next) + file (ctf/make-file {:name "Legacy camel svg attrs"}) + page-id (first (get-in file [:data :pages])) + shape (-> (cts/setup-shape {:id shape-id :type :rect}) + (assoc :svg-attrs {:fillRule "evenodd"})) + file (-> file + (assoc :migrations (-> cfm/available-migrations + (disj migration-id) + (conj "0035-fix-shape-svg-attrs-and-defs"))) + (assoc-in [:data :pages-index page-id :objects shape-id] shape)) + file' (cfm/migrate-file file {})] + (t/is (cfm/need-migration? file) "new migration detected") + (t/is (not (cfm/need-migration? file')) "new migration recorded") + (t/is (contains? (:migrations file') migration-id) "migration id persisted") + (t/is (not (contains? cfm/available-migrations "0035-fix-shape-svg-attrs-and-defs")) + "the withdrawn 0035 id stays out of available-migrations") + (t/is (= "evenodd" + (get-in file' [:data :pages-index page-id :objects shape-id :svg-attrs :fill-rule])) + "migration repaired file data before schema validation"))) diff --git a/common/test/common_tests/runner.cljc b/common/test/common_tests/runner.cljc index 5243c00c61..d28469ee1e 100644 --- a/common/test/common_tests/runner.cljc +++ b/common/test/common_tests/runner.cljc @@ -17,6 +17,7 @@ [common-tests.attrs-test] [common-tests.buffer-test] [common-tests.colors-test] + [common-tests.data-macros-test] [common-tests.data-test] [common-tests.exceptions-test] [common-tests.files-branch-merge-test] @@ -28,6 +29,7 @@ [common-tests.files-migrations-0034-test] [common-tests.files-migrations-0035-test] [common-tests.files-migrations-0036-test] + [common-tests.files-migrations-0037-test] [common-tests.files-migrations-test] [common-tests.files-ref-paths-test] [common-tests.files.comp-processors-test] @@ -138,6 +140,7 @@ 'common-tests.attrs-test 'common-tests.buffer-test 'common-tests.colors-test + 'common-tests.data-macros-test 'common-tests.data-test 'common-tests.exceptions-test 'common-tests.files-branch-merge-test @@ -151,6 +154,7 @@ 'common-tests.files-migrations-0034-test 'common-tests.files-migrations-0035-test 'common-tests.files-migrations-0036-test + 'common-tests.files-migrations-0037-test 'common-tests.files-migrations-test 'common-tests.files-ref-paths-test 'common-tests.files.comp-processors-test diff --git a/common/test/common_tests/schema_test.cljc b/common/test/common_tests/schema_test.cljc index 08e26295a6..3e36900906 100644 --- a/common/test/common_tests/schema_test.cljc +++ b/common/test/common_tests/schema_test.cljc @@ -7,6 +7,7 @@ (ns common-tests.schema-test (:require [app.common.data :as d] + [app.common.data.macros :as dm] [app.common.schema :as sm] [app.common.schema.generators :as sg] [app.common.uuid :as uuid] @@ -283,3 +284,43 @@ (t/testing "generate" (t/is (true? (sm/validate ::sm/user-provided-uuid (sg/generate ::sm/user-provided-uuid))))))) + +(defn- catch-error + [f] + (try + (f) + nil + (catch #?(:clj Throwable :cljs :default) cause + cause))) + +(defn- lazy-resolution? + "Mirrors the build mode check performed by `app.common.data.macros/lazy-delay`." + [] + #?(:clj false + :cljs dm/lazy-delay?)) + +(t/deftest test-check-fn + (t/testing "returns the value when it is valid" + (let [check (sm/check-fn ::sm/uuid)] + (t/is (= uuid/zero (check uuid/zero))))) + + (t/testing "raises with the explain data when the value is invalid" + (let [check (sm/check-fn ::sm/uuid :hint "invalid uuid" + :type :custom :code :custom-code) + cause (catch-error #(check "not-a-uuid")) + data (ex-data cause)] + (t/is (some? cause)) + (t/is (= "invalid uuid" (ex-message cause))) + (t/is (= :custom (:type data))) + (t/is (= :custom-code (:code data))) + (t/is (= "invalid uuid" (:hint data))) + (t/is (map? (::sm/explain data))))) + + (t/testing "resolves the schema definition according to the build mode" + ;; A schema that does not exist raises only when it is resolved, so this + ;; checks when the resolution happens: on `check-fn` on non production + ;; builds, on the first call of the returned function on production ones. + (let [check (catch-error #(sm/check-fn ::unknown-schema))] + (if (lazy-resolution?) + (t/is (some? (catch-error #(check ::anything)))) + (t/is (some? check)))))) diff --git a/docker/devenv/Dockerfile b/docker/devenv/Dockerfile index 55540a934f..6cd9b8cdfe 100644 --- a/docker/devenv/Dockerfile +++ b/docker/devenv/Dockerfile @@ -150,12 +150,12 @@ RUN set -eux; \ ARCH="$(dpkg --print-architecture)"; \ case "${ARCH}" in \ aarch64|arm64) \ - ESUM='9fa5bf865783c43840101fcdc3a5222b60daf4fe11f26806247bf807c08aba38'; \ - BINARY_URL='https://cdn.azul.com/zulu/bin/zulu27.28.101-ca-jdk27.0.0-linux_aarch64.tar.gz'; \ + ESUM='18875b92ad2535230a13175f718933f7cc0593b4affa2dab442815ec245dacbb'; \ + BINARY_URL='https://cdn.azul.com/zulu/bin/zulu27.30.201-ca-jdk27.0.1.1-linux_aarch64.tar.gz'; \ ;; \ amd64|x86_64) \ - ESUM='ccbc15c4edbedfdcc03c2d29a2aa2c6daf9e6ffb4cda7dd5ece0fcbb37350267'; \ - BINARY_URL='https://cdn.azul.com/zulu/bin/zulu27.28.101-ca-jdk27.0.0-linux_x64.tar.gz'; \ + ESUM='c4d7bf4c80a21e935187ecdcb111e4b2f14e1fdd18364974fe7f7da54b12bea0'; \ + BINARY_URL='https://cdn.azul.com/zulu/bin/zulu27.30.201-ca-jdk27.0.1.1-linux_x64.tar.gz'; \ ;; \ *) \ echo "Unsupported arch: ${ARCH}"; \ diff --git a/docker/devenv/files/tmux.conf b/docker/devenv/files/tmux.conf index 062c1622c7..349aa43fc7 100644 --- a/docker/devenv/files/tmux.conf +++ b/docker/devenv/files/tmux.conf @@ -6,5 +6,8 @@ set -g mouse off set -g history-limit 50000 setw -g mode-keys emacs +set -g set-clipboard on +set -g allow-passthrough on + bind-key -r C-n resize-pane -U bind-key -r C-m resize-pane -D diff --git a/docker/gitpod/Dockerfile b/docker/gitpod/Dockerfile deleted file mode 100644 index 30ec7da8fe..0000000000 --- a/docker/gitpod/Dockerfile +++ /dev/null @@ -1,122 +0,0 @@ -FROM gitpod/workspace-postgres - -# Install custom tools, runtimes, etc. -# For example "bastet", a command-line tetris clone: -# RUN brew install bastet -# -# More information: https://www.gitpod.io/docs/config-docker/ - -RUN set -ex; \ - brew install redis; \ - brew install imagemagick; \ - brew install ghostscript; \ - brew install mailhog; \ - brew install openldap; \ - brew install poppler; \ - sudo mkdir -p /var/log/nginx; \ - sudo chown gitpod:gitpod /var/log/nginx - -COPY files/nginx.conf /etc/nginx/nginx.conf -COPY files/nginx-mime.types /etc/nginx/mime.types - -USER root - -ENV NODE_VERSION=v16.17.0 \ - CLOJURE_VERSION=1.11.1.1165 \ - CLJKONDO_VERSION=2022.09.08 \ - BABASHKA_VERSION=0.9.162 \ - LANG=en_US.UTF-8 \ - LC_ALL=en_US.UTF-8 - -RUN set -ex; \ - useradd -m -g users -s /bin/bash penpot; \ - passwd penpot -d; \ - echo "penpot ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers - -RUN set -ex; \ - apt-get -qq update; \ - apt-get -qqy install \ - gconf-service \ - libasound2 \ - libatk1.0-0 \ - libatk-bridge2.0-0 \ - libcairo2 \ - libcups2 \ - libdbus-1-3 \ - libexpat1 \ - libfontconfig1 \ - libgcc1 \ - libgconf-2-4 \ - libgdk-pixbuf2.0-0 \ - libglib2.0-0 \ - libgtk-3-0 \ - libnspr4 \ - libpango-1.0-0 \ - libpangocairo-1.0-0 \ - libx11-6 \ - libx11-xcb1 \ - libxcb1 \ - libxcomposite1 \ - libxcursor1 \ - libxdamage1 \ - libxext6 \ - libxfixes3 \ - libxi6 \ - libxrandr2 \ - libxrender1 \ - libxshmfence1 \ - libxss1 \ - libxtst6 \ - fonts-liberation \ - libappindicator1 \ - libnss3 \ - libgbm1 \ - ; \ - rm -rf /var/lib/apt/lists/*; - -RUN set -ex; \ - curl -LfsSo /tmp/openjdk.tar.gz https://github.com/AdoptOpenJDK/openjdk16-binaries/releases/download/jdk-16%2B36/OpenJDK16-jdk_x64_linux_hotspot_16_36.tar.gz; \ - mkdir -p /usr/lib/jvm/openjdk16; \ - cd /usr/lib/jvm/openjdk16; \ - tar -xf /tmp/openjdk.tar.gz --strip-components=1; \ - rm -rf /tmp/openjdk.tar.gz; - -# Install clojure cli -RUN set -ex; \ - curl -LfsSo /tmp/clojure.sh https://download.clojure.org/install/linux-install-$CLOJURE_VERSION.sh; \ - chmod +x /tmp/clojure.sh; \ - /tmp/clojure.sh; \ - rm -rf /tmp/clojure.sh; - -# Install clj-kondo -RUN set -ex; \ - curl -LfsSo /tmp/clj-kondo.zip https://github.com/borkdude/clj-kondo/releases/download/v$CLJKONDO_VERSION/clj-kondo-$CLJKONDO_VERSION-linux-amd64.zip; \ - cd /usr/local/bin; \ - unzip /tmp/clj-kondo.zip; \ - rm /tmp/clj-kondo.zip; - -# Install babashka -RUN set -ex; \ - cd /tmp; \ - curl -LfsSo /tmp/babashka.tar.gz https://github.com/babashka/babashka/releases/download/v$BABASHKA_VERSION/babashka-$BABASHKA_VERSION-linux-amd64.tar.gz; \ - cd /usr/local/bin; \ - tar -xf /tmp/babashka.tar.gz; \ - rm -rf /tmp/babashka.tar.gz; - -USER gitpod - -ENV PATH="/usr/lib/jvm/openjdk16/bin:/usr/local/nodejs/bin:$PATH" \ - JAVA_HOME=/usr/lib/jvm/openjdk16 \ - PENPOT_SMTP_ENABLED=true \ - PENPOT_SMTP_HOST=localhost \ - PENPOT_SMTP_PORT=1025 \ - PENPOT_SMTP_USER= \ - PENPOT_SMTP_PASSWORD= \ - PENPOT_SMTP_SSL=false \ - PENPOT_SMTP_TLS=false \ - PENPOT_SMTP_DEFAULT_REPLY_TO=no-reply@example.com \ - PENPOT_SMTP_DEFAULT_FROM=no-reply@example.com \ - PENPOT_DATABASE_URI="postgresql://localhost/penpot" \ - PENPOT_REDIS_URI="redis://localhost/0" - -# TODO Retrieve OpenLDAP from rroemhild/docker-test-openldap diff --git a/docker/gitpod/files/nginx-mime.types b/docker/gitpod/files/nginx-mime.types deleted file mode 100644 index 91f6aba0ac..0000000000 --- a/docker/gitpod/files/nginx-mime.types +++ /dev/null @@ -1,100 +0,0 @@ - -types { - text/html html htm shtml; - text/css css; - text/xml xml; - image/gif gif; - image/jpeg jpeg jpg; - application/javascript js; - application/javascript mjs; - application/atom+xml atom; - application/rss+xml rss; - - text/mathml mml; - text/plain txt; - text/vnd.sun.j2me.app-descriptor jad; - text/vnd.wap.wml wml; - text/x-component htc; - - image/avif avif; - image/png png; - image/svg+xml svg svgz; - image/tiff tif tiff; - image/vnd.wap.wbmp wbmp; - image/webp webp; - image/x-icon ico; - image/x-jng jng; - image/x-ms-bmp bmp; - - font/woff woff; - font/woff2 woff2; - - application/java-archive jar war ear; - application/json json; - application/mac-binhex40 hqx; - application/msword doc; - application/pdf pdf; - application/postscript ps eps ai; - application/rtf rtf; - application/vnd.apple.mpegurl m3u8; - application/vnd.google-earth.kml+xml kml; - application/vnd.google-earth.kmz kmz; - application/vnd.ms-excel xls; - application/vnd.ms-fontobject eot; - application/vnd.ms-powerpoint ppt; - application/vnd.oasis.opendocument.graphics odg; - application/vnd.oasis.opendocument.presentation odp; - application/vnd.oasis.opendocument.spreadsheet ods; - application/vnd.oasis.opendocument.text odt; - application/vnd.openxmlformats-officedocument.presentationml.presentation - pptx; - application/vnd.openxmlformats-officedocument.spreadsheetml.sheet - xlsx; - application/vnd.openxmlformats-officedocument.wordprocessingml.document - docx; - application/vnd.wap.wmlc wmlc; - application/wasm wasm; - application/x-7z-compressed 7z; - application/x-cocoa cco; - application/x-java-archive-diff jardiff; - application/x-java-jnlp-file jnlp; - application/x-makeself run; - application/x-perl pl pm; - application/x-pilot prc pdb; - application/x-rar-compressed rar; - application/x-redhat-package-manager rpm; - application/x-sea sea; - application/x-shockwave-flash swf; - application/x-stuffit sit; - application/x-tcl tcl tk; - application/x-x509-ca-cert der pem crt; - application/x-xpinstall xpi; - application/xhtml+xml xhtml; - application/xspf+xml xspf; - application/zip zip; - - application/octet-stream bin exe dll; - application/octet-stream deb; - application/octet-stream dmg; - application/octet-stream iso img; - application/octet-stream msi msp msm; - - audio/midi mid midi kar; - audio/mpeg mp3; - audio/ogg ogg; - audio/x-m4a m4a; - audio/x-realaudio ra; - - video/3gpp 3gpp 3gp; - video/mp2t ts; - video/mp4 mp4; - video/mpeg mpeg mpg; - video/quicktime mov; - video/webm webm; - video/x-flv flv; - video/x-m4v m4v; - video/x-mng mng; - video/x-ms-asf asx asf; - video/x-ms-wmv wmv; - video/x-msvideo avi; -} diff --git a/docker/gitpod/files/nginx.conf b/docker/gitpod/files/nginx.conf deleted file mode 100644 index ca403adfd2..0000000000 --- a/docker/gitpod/files/nginx.conf +++ /dev/null @@ -1,145 +0,0 @@ -# This Nginx config file is optional. -# If you don't have this file, the Nginx config from the workspace-full docker image will be taken. - -worker_processes auto; -pid /var/run/nginx/nginx.pid; -include /etc/nginx/modules-enabled/*.conf; - -env GITPOD_REPO_ROOT; - -events { - worker_connections 768; - # multi_accept on; -} - -http { - sendfile on; - tcp_nopush on; - tcp_nodelay on; - keepalive_timeout 65; - types_hash_max_size 2048; - # server_tokens off; - - # server_names_hash_bucket_size 64; - # server_name_in_redirect off; - - include /etc/nginx/mime.types; - default_type application/octet-stream; - - access_log /var/log/nginx/access.log; - error_log /var/log/nginx/error.log; - - gzip on; - - gzip_vary on; - gzip_proxied any; - gzip_comp_level 3; - gzip_buffers 16 8k; - gzip_http_version 1.1; - - gzip_types text/plain text/css text/javascript application/javascript application/json application/transit+json; - - map $http_upgrade $connection_upgrade { - default upgrade; - '' close; - } - - # include /etc/nginx/sites-enabled/*; - - server { - set_by_lua $gitpod_repo_root 'return os.getenv("GITPOD_REPO_ROOT")'; - - listen 0.0.0.0:3449 default_server; - server_name _; - - client_max_body_size 5M; - charset utf-8; - - proxy_http_version 1.1; - proxy_set_header Host $http_host; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header X-Scheme $scheme; - proxy_set_header X-Forwarded-Proto $scheme; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - - resolver 8.8.8.8; - - etag off; - - root $gitpod_repo_root/frontend/resources/public; - - location @handle_redirect { - set $redirect_uri "$upstream_http_location"; - set $redirect_host "$upstream_http_x_host"; - set $redirect_cache_control "$upstream_http_cache_control"; - - proxy_buffering off; - - proxy_set_header Host "$redirect_host"; - proxy_hide_header etag; - proxy_hide_header x-amz-id-2; - proxy_hide_header x-amz-request-id; - proxy_hide_header x-amz-meta-server-side-encryption; - proxy_hide_header x-amz-server-side-encryption; - proxy_pass $redirect_uri; - - add_header x-internal-redirect "$redirect_uri"; - add_header x-cache-control "$redirect_cache_control"; - add_header cache-control "$redirect_cache_control"; - } - - location /assets { - proxy_pass http://127.0.0.1:6060/assets; - recursive_error_pages on; - proxy_intercept_errors on; - error_page 301 302 307 = @handle_redirect; - } - - location /internal/assets { - internal; - alias $gitpod_repo_root/backend/assets; - add_header x-internal-redirect "$upstream_http_x_accel_redirect"; - } - - location /api { - proxy_pass http://127.0.0.1:6060/api; - } - - location /webhooks { - proxy_pass http://127.0.0.1:6060/webhooks; - } - - location /dbg { - proxy_pass http://127.0.0.1:6060/dbg; - } - - location /export { - proxy_pass http://127.0.0.1:6061; - } - - location /playground { - alias $gitpod_repo_root/experiments/; - add_header Cache-Control "no-cache, max-age=0"; - autoindex on; - } - - location /ws/notifications { - proxy_set_header Upgrade $http_upgrade; - proxy_set_header Connection 'upgrade'; - proxy_pass http://127.0.0.1:6060/ws/notifications; - } - - location / { - add_header Cache-Control "no-cache, max-age=0"; - - location ~ ^/(/|css|fonts|images|js|wasm) { - } - - location ~ ^/[^/]+/(.*)$ { - return 301 " /404"; - } - - try_files $uri /index.html$is_args$args =404; - } - } -} diff --git a/docker/gitpod/files/postgresql_init.sql b/docker/gitpod/files/postgresql_init.sql deleted file mode 100644 index 1766a643d2..0000000000 --- a/docker/gitpod/files/postgresql_init.sql +++ /dev/null @@ -1,3 +0,0 @@ -CREATE DATABASE penpot; -CREATE USER penpot PASSWORD 'penpot'; -ALTER ROLE penpot SUPERUSER; diff --git a/docker/images/Dockerfile.backend b/docker/images/Dockerfile.backend index 1e287b7080..c2f4c5e6b7 100644 --- a/docker/images/Dockerfile.backend +++ b/docker/images/Dockerfile.backend @@ -27,12 +27,12 @@ RUN set -eux; \ ARCH="$(dpkg --print-architecture)"; \ case "${ARCH}" in \ aarch64|arm64) \ - ESUM='8ff07320636ef5b30f0d7e16687932df52255c3ef3ffd920a6aae3723511ce7f'; \ - BINARY_URL='https://cdn.azul.com/zulu/bin/zulu26.32.203-ca-jdk26.0.2.1-linux_aarch64.tar.gz'; \ + ESUM='18875b92ad2535230a13175f718933f7cc0593b4affa2dab442815ec245dacbb'; \ + BINARY_URL='https://cdn.azul.com/zulu/bin/zulu27.30.201-ca-jdk27.0.1.1-linux_aarch64.tar.gz'; \ ;; \ amd64|x86_64) \ - ESUM='c6b810ca659e3293205444f47ac4af163396fd6d025c51fd5f767e628b2ceede'; \ - BINARY_URL='https://cdn.azul.com/zulu/bin/zulu26.32.203-ca-jdk26.0.2.1-linux_x64.tar.gz'; \ + ESUM='c4d7bf4c80a21e935187ecdcb111e4b2f14e1fdd18364974fe7f7da54b12bea0'; \ + BINARY_URL='https://cdn.azul.com/zulu/bin/zulu27.30.201-ca-jdk27.0.1.1-linux_x64.tar.gz'; \ ;; \ *) \ echo "Unsupported arch: ${ARCH}"; \ diff --git a/docker/images/Dockerfile.frontend b/docker/images/Dockerfile.frontend index 191f0ccda6..1be6ee9680 100644 --- a/docker/images/Dockerfile.frontend +++ b/docker/images/Dockerfile.frontend @@ -1,4 +1,4 @@ -FROM dhi.io/nginx:1.31.8-debian13-dev@sha256:2168f2f90d8dd61baf5ac0359d92f601f988ba7d7ef669dc778cd7a887130295 +FROM dhi.io/nginx:1.31.8-debian13-dev@sha256:5f75855fe078ae96027706da06716fbe1c3dbb618fc603e61894e1b41adc56d7 LABEL maintainer="Penpot " ENV DEBIAN_FRONTEND=noninteractive diff --git a/docker/images/files/nginx-entrypoint.sh b/docker/images/files/nginx-entrypoint.sh index f809591827..c6ad751c00 100644 --- a/docker/images/files/nginx-entrypoint.sh +++ b/docker/images/files/nginx-entrypoint.sh @@ -68,6 +68,42 @@ else export PENPOT_PUBLIC_PATH_DIRECTIVES="" fi +# nginx's `auto` starts one worker per CPU of the host, not of the container, +# and each worker reserves the memory for all its connections at startup. A +# container limited to a few CPUs on a large node can run out of memory before +# serving a request, so size the workers from the CPUs the container can use +# (its cpuset, and its CPU quota rounded up) when they are fewer than the host's. +container_cpu_limit() { + local quota="" period="" cpus + cpus="$(nproc 2>/dev/null)" || return 1 + if [ -r /sys/fs/cgroup/cpu.max ]; then + read -r quota period < /sys/fs/cgroup/cpu.max + elif [ -r /sys/fs/cgroup/cpu/cpu.cfs_quota_us ]; then + quota="$(< /sys/fs/cgroup/cpu/cpu.cfs_quota_us)" + period="$(< /sys/fs/cgroup/cpu/cpu.cfs_period_us)" + fi + # "max" (cgroup v2) and -1 (cgroup v1) mean no quota, and 0 is not one. + if [[ "$quota" =~ ^[1-9][0-9]*$ && "$period" =~ ^[1-9][0-9]*$ ]]; then + quota=$(( (quota + period - 1) / period )) + (( quota < cpus )) && cpus=$quota + fi + (( cpus < $(nproc --all) )) || return 1 + echo "$cpus" +} + +export PENPOT_HTTP_SERVER_WORKER_PROCESSES=${PENPOT_HTTP_SERVER_WORKER_PROCESSES:-$(container_cpu_limit || echo auto)} +export PENPOT_HTTP_SERVER_WORKER_CONNECTIONS=${PENPOT_HTTP_SERVER_WORKER_CONNECTIONS:-65535} + +if [[ ! "$PENPOT_HTTP_SERVER_WORKER_PROCESSES" =~ ^(auto|[1-9][0-9]*)$ ]]; then + echo "penpot: invalid PENPOT_HTTP_SERVER_WORKER_PROCESSES '${PENPOT_HTTP_SERVER_WORKER_PROCESSES}'; expected auto or a positive integer" >&2 + exit 1 +fi + +if [[ ! "$PENPOT_HTTP_SERVER_WORKER_CONNECTIONS" =~ ^[1-9][0-9]*$ ]]; then + echo "penpot: invalid PENPOT_HTTP_SERVER_WORKER_CONNECTIONS '${PENPOT_HTTP_SERVER_WORKER_CONNECTIONS}'; expected a positive integer" >&2 + exit 1 +fi + export PENPOT_BACKEND_URI=${PENPOT_BACKEND_URI:-http://penpot-backend:6060} export PENPOT_EXPORTER_URI=${PENPOT_EXPORTER_URI:-http://penpot-exporter:6061} export PENPOT_HTTP_SERVER_MAX_BODY_SIZE=${PENPOT_HTTP_SERVER_MAX_BODY_SIZE:-367001600} # Default to 350MiB @@ -75,7 +111,7 @@ export PENPOT_IPV6_LISTEN_DIRECTIVE=${PENPOT_IPV6_LISTEN_DIRECTIVE:-"listen [::] if is_truthy "${PENPOT_DISABLE_IPV6_LISTEN:-}"; then export PENPOT_IPV6_LISTEN_DIRECTIVE="" fi -envsubst "\$PENPOT_BACKEND_URI,\$PENPOT_EXPORTER_URI,\$PENPOT_HTTP_SERVER_MAX_BODY_SIZE,\$PENPOT_IPV6_LISTEN_DIRECTIVE,\$PENPOT_PUBLIC_PATH,\$PENPOT_PUBLIC_PATH_DIRECTIVES" \ +envsubst "\$PENPOT_HTTP_SERVER_WORKER_PROCESSES,\$PENPOT_HTTP_SERVER_WORKER_CONNECTIONS,\$PENPOT_BACKEND_URI,\$PENPOT_EXPORTER_URI,\$PENPOT_HTTP_SERVER_MAX_BODY_SIZE,\$PENPOT_IPV6_LISTEN_DIRECTIVE,\$PENPOT_PUBLIC_PATH,\$PENPOT_PUBLIC_PATH_DIRECTIVES" \ < /tmp/nginx.conf.template > /etc/nginx/nginx.conf if [[ " $PENPOT_FLAGS " != *" disable-admin-console "* ]]; then diff --git a/docker/images/files/nginx.conf.template b/docker/images/files/nginx.conf.template index 54d62e18eb..544d0fcdfd 100644 --- a/docker/images/files/nginx.conf.template +++ b/docker/images/files/nginx.conf.template @@ -1,9 +1,9 @@ -worker_processes auto; +worker_processes ${PENPOT_HTTP_SERVER_WORKER_PROCESSES}; pid /tmp/nginx.pid; include /etc/nginx/overrides/main.d/*.conf; events { - worker_connections 65535; + worker_connections ${PENPOT_HTTP_SERVER_WORKER_CONNECTIONS}; multi_accept on; } diff --git a/docs/technical-guide/configuration.md b/docs/technical-guide/configuration.md index 4cda6efef4..5e9731badd 100644 --- a/docs/technical-guide/configuration.md +++ b/docs/technical-guide/configuration.md @@ -780,6 +780,33 @@ If the PostgreSQL server is shared with other applications, make sure `PENPOT_DATABASE_MAX_POOL_SIZE` plus the connections used by everything else stays below the server's `max_connections` (100 by default). +#### Prepared statement cache + +__Since version 2.20.0__ + +The driver keeps two things per connection: a cache of parsed queries and, once a query +runs enough times, a statement prepared in the server whose plan the server can reuse. +Both are on by default; you only need these settings to tune them or to turn them off: + +```bash +# Backend +PENPOT_DATABASE_PREPARED_STATEMENT_CACHE_THRESHOLD: 5 +PENPOT_DATABASE_PREPARED_STATEMENT_CACHE_QUERIES: 256 +PENPOT_DATABASE_PREPARED_STATEMENT_CACHE_SIZE: 5242880 +``` + +The values above are the driver defaults. `PENPOT_DATABASE_PREPARED_STATEMENT_CACHE_THRESHOLD` +is how many times a query runs on a connection before the driver prepares it server-side: +`-1` prepares it from the first execution and forces binary transfer, and `0` stops the +driver from preparing statements in the server. The cache of parsed queries stays in use +with the threshold at `0`: the connection caches up to +`PENPOT_DATABASE_PREPARED_STATEMENT_CACHE_QUERIES` queries, using at most +`PENPOT_DATABASE_PREPARED_STATEMENT_CACHE_SIZE` bytes per connection (the size is rounded +up to whole MiB for the driver), and setting either of those two to `0` disables the +cache. With 60 connections, the ceiling is 60 × the size. Leave a variable unset and the +driver default applies. A parameter of the same name in `PENPOT_DATABASE_URI`, such as +`?prepareThreshold=0`, takes precedence over these variables. + ### Storage Storage refers to storing the user uploaded different objects in Penpot (assets, file data,...). @@ -1049,6 +1076,27 @@ PENPOT_INTERNAL_RESOLVER: 127.0.0.11 names. Defaults to the nameservers found in `/etc/resolv.conf`. `127.0.0.11` is the embedded Docker DNS server; use the address of your own resolver on other setups. +### Nginx workers + +The frontend container sizes its nginx workers from the CPUs it can use, its CPU quota or +its cpuset, when they are fewer than the CPUs of the host. Otherwise it starts one worker +per CPU of the host. Each worker reserves the memory for all its connections when it +starts, so tune these if the frontend uses more memory than you expect: + +```bash +# Frontend +PENPOT_HTTP_SERVER_WORKER_PROCESSES: 2 +PENPOT_HTTP_SERVER_WORKER_CONNECTIONS: 2048 +``` + +- `PENPOT_HTTP_SERVER_WORKER_PROCESSES`: Number of nginx worker processes, a positive + integer or `auto` for one per host CPU. Defaults to the CPUs the container can use, with + the quota rounded up, or `auto` when it has no limit. +- `PENPOT_HTTP_SERVER_WORKER_CONNECTIONS`: Maximum simultaneous connections per worker, + a positive integer. Defaults to `65535`, which takes about 30 MB of memory per worker. + +The container refuses to start if either value is not valid. + ## Other flags There are other flags that are useful for a more customized Penpot experience. This section has the list of the flags meant diff --git a/docs/technical-guide/developer/data-model/penpot-file-format.md b/docs/technical-guide/developer/data-model/penpot-file-format.md index 431dbffaa2..970e9276c8 100644 --- a/docs/technical-guide/developer/data-model/penpot-file-format.md +++ b/docs/technical-guide/developer/data-model/penpot-file-format.md @@ -21,7 +21,7 @@ The `.penpot` format (version 3) uses a ZIP container with JSON files for metada | Version | Description | Status | |---------|-------------|--------| | v1 | Custom binary format | Deprecated | -| v2 | SQLite-based format | Never released (internal only) | +| v2 | SQLite-based format | Removed (never released) | | v3 | ZIP + JSON format | **Current** | ## File Structure diff --git a/exporter/shadow-cljs.edn b/exporter/shadow-cljs.edn index 076e6a10f6..9bb46f47f6 100644 --- a/exporter/shadow-cljs.edn +++ b/exporter/shadow-cljs.edn @@ -24,7 +24,8 @@ :output-wrapper false} :release - {:compiler-options + {:closure-defines {app.common.data.macros/lazy-delay? true} + :compiler-options {:fn-invoke-direct true :source-map true :optimizations #shadow/env ["PENPOT_BUILD_OPTIMIZATIONS" :as :keyword :default :simple] diff --git a/frontend/playwright/ui/visual-specs/visual-dashboard.spec.js b/frontend/playwright/ui/visual-specs/visual-dashboard.spec.js index 94985dfeee..522363eacb 100644 --- a/frontend/playwright/ui/visual-specs/visual-dashboard.spec.js +++ b/frontend/playwright/ui/visual-specs/visual-dashboard.spec.js @@ -1103,3 +1103,24 @@ test.describe("Team settings organization", () => { await expect(add).toHaveScreenshot(shot); }); }); + +test.describe("Team hero link", () => { + test("Shows the Team management link", async ({ page }) => { + const shot = { maxDiffPixelRatio: 0, threshold: 0 }; + const dashboardPage = new DashboardPage(page); + await dashboardPage.setupDashboardFull(); + await dashboardPage.mockRPC( + `get-projects?team-id=${DashboardPage.secondTeamId}`, + "dashboard/get-projects-second-team.json", + ); + await dashboardPage.goToSecondTeamDashboard(); + await expect(page.getByText("Team Up")).toBeVisible(); + + const link = page.getByText("Team management", { exact: true }); + await page.mouse.move(10, 700); + await expect(link).toHaveScreenshot(shot); + + await link.hover(); + await expect(link).toHaveScreenshot(shot); + }); +}); diff --git a/frontend/playwright/ui/visual-specs/visual-login.spec.js b/frontend/playwright/ui/visual-specs/visual-login.spec.js index 7c9650d656..797d215ea6 100644 --- a/frontend/playwright/ui/visual-specs/visual-login.spec.js +++ b/frontend/playwright/ui/visual-specs/visual-login.spec.js @@ -108,3 +108,18 @@ test.describe("Login dialog section links", () => { await expect(loginHere).toHaveScreenshot(shot); }); }); + +test.describe("Password recovery page", () => { + test("Shows the Go to login link", async ({ page }) => { + const shot = { maxDiffPixelRatio: 0, threshold: 0 }; + await page.goto("/?screen=auth-recovery&token=fake-token"); + + const link = page.getByText("Go to login", { exact: true }); + await expect(link).toBeVisible(); + await page.mouse.move(10, 10); + await expect(link).toHaveScreenshot(shot); + + await link.hover(); + await expect(link).toHaveScreenshot(shot); + }); +}); diff --git a/frontend/shadow-cljs.edn b/frontend/shadow-cljs.edn index c18338fbf4..1b63abb96a 100644 --- a/frontend/shadow-cljs.edn +++ b/frontend/shadow-cljs.edn @@ -76,6 +76,7 @@ :release {:closure-defines {goog.DEBUG false goog.debug.LOGGING_ENABLED true + app.common.data.macros/lazy-delay? true app.config/compiled-version-tag #shadow/env ["VERSION_TAG" :default "develop"]} :compiler-options {:fn-invoke-direct true @@ -115,7 +116,8 @@ :warnings {:fn-deprecated false}} :release - {:compiler-options + {:closure-defines {app.common.data.macros/lazy-delay? true} + :compiler-options {:fn-invoke-direct true :optimizations #shadow/env ["PENPOT_BUILD_OPTIMIZATIONS" :as :keyword :default :advanced] :output-wrapper true diff --git a/frontend/src/app/main/data/team.cljs b/frontend/src/app/main/data/team.cljs index 6659fa8aae..b4b2845de6 100644 --- a/frontend/src/app/main/data/team.cljs +++ b/frontend/src/app/main/data/team.cljs @@ -143,11 +143,15 @@ (rx/of team-form-modal) :else - (let [pending-id (str (uuid/next)) - callback-url (dom/append-query-param (rt/get-current-href) - :pending-action-id pending-id)] - (->> (rp/cmd! :check-nitrate-sso {:team-id team-id - :url callback-url}) + (let [pending-id (str (uuid/next)) + ;; The callback travels as a query string; the token is + ;; kept in its own shape because `append-query-param` + ;; works on URLs. + callback-query-string (-> (rt/get-current-token) + (dom/append-query-param :pending-action-id pending-id) + rt/token->query-string)] + (->> (rp/cmd! :check-nitrate-sso {:team-id team-id + :query-string callback-query-string}) (rx/mapcat (fn [{:keys [authorized redirect-uri]}] (cond diff --git a/frontend/src/app/main/data/workspace/notifications.cljs b/frontend/src/app/main/data/workspace/notifications.cljs index 5abddee2d4..d7728ad41b 100644 --- a/frontend/src/app/main/data/workspace/notifications.cljs +++ b/frontend/src/app/main/data/workspace/notifications.cljs @@ -375,7 +375,7 @@ (modal/show :file-restored {:fullname (restorer-fullname state profile-id session-id) :label label - :created-at created-at + :date created-at ;; Made on the replaced version, so the reload drops them :unsaved (boolean (seq (dps/pending-file-changes state file)))})) (rx/empty)))))) diff --git a/frontend/src/app/main/errors.cljs b/frontend/src/app/main/errors.cljs index 753467a9a3..7b58b2e539 100644 --- a/frontend/src/app/main/errors.cljs +++ b/frontend/src/app/main/errors.cljs @@ -746,17 +746,41 @@ (ex/print-throwable cause :prefix "Unexpected Error") (flash :cause cause :type :unhandled)))) +(def recoverable-wasm-codes + "WASM error codes that show a toast; any other code shows the error page." + #{:non-blocking :wasm-reload-context-failure :viewer-canvas-failed}) + (defmethod ptk/handle-error :wasm-error [error] (when-let [cause (::instance error)] (ex/print-throwable cause) - (let [code (get error :code)] - (cond - (= code :panic) - (st/emit! (rt/assign-exception error)) + (if (contains? recoverable-wasm-codes (:code error)) + (flash :type :handled :cause cause) + (st/emit! (rt/assign-exception error))))) - :else - (flash :type :handled :cause cause))))) +(defn- handle-wasm-effect-error + [cause] + (let [{:keys [type code]} (ex-data cause)] + (if (and (= type :wasm-error) (contains? recoverable-wasm-codes code)) + (on-error cause) + (throw cause)))) + +(defn wrap-wasm-effect + "Runs the React effect fn `f` and guards the cleanup fn it returns: a + recoverable WASM error goes to `on-error`, any other error is rethrown." + [f] + (let [result (try + (f) + (catch :default cause + (handle-wasm-effect-error cause) + nil))] + (if (fn? result) + (fn [] + (try + (result) + (catch :default cause + (handle-wasm-effect-error cause)))) + result))) ;; We receive a explicit authentication error; If the uri is for ;; workspace, dashboard, viewer or settings, then assign the exception diff --git a/frontend/src/app/main/ui/auth/recovery.cljs b/frontend/src/app/main/ui/auth/recovery.cljs index 3a772b79af..6ca6f6b2d0 100644 --- a/frontend/src/app/main/ui/auth/recovery.cljs +++ b/frontend/src/app/main/ui/auth/recovery.cljs @@ -10,10 +10,13 @@ [app.common.schema :as sm] [app.main.data.notifications :as ntf] [app.main.data.profile :as du] + [app.main.refs :as refs] [app.main.router :as rt] [app.main.store :as st] [app.main.ui.components.forms :as fm] + [app.util.dom :as dom] [app.util.i18n :as i18n :refer [tr]] + [app.util.keyboard :as kbd] [rumext.v2 :as mf])) (def ^:private schema:recovery-form @@ -88,16 +91,25 @@ ;; --- Recovery Request Page +(defn- on-go-to-login + [event] + ;; Ctrl/Cmd-click opens the href in a new tab, as any link. + (when-not (kbd/mod? event) + (dom/prevent-default event) + (st/emit! (rt/nav :auth-login)))) + (mf/defc recovery-page* [{:keys [params]}] - [:div {:class (stl/css :wrapper)} - [:h1 {:class (stl/css :title)} (tr "auth.recovery-request-title")] - [:div {:class (stl/css :subtitle)} (tr "auth.recovery-request-subtitle")] - [:hr {:class (stl/css :separator)}] - [:> recovery-form* {:params params}] + (let [router (mf/deref refs/router)] + [:div {:class (stl/css :wrapper)} + [:h1 {:class (stl/css :title)} (tr "auth.recovery-request-title")] + [:div {:class (stl/css :subtitle)} (tr "auth.recovery-request-subtitle")] + [:hr {:class (stl/css :separator)}] + [:> recovery-form* {:params params}] - [:div {:class (stl/css :links)} - [:div {:class (stl/css :go-back-row)} - [:a {:on-click #(st/emit! (rt/nav :auth-login)) - :class (stl/css :go-back-link)} - (tr "profile.recovery.go-to-login")]]]]) + [:div {:class (stl/css :links)} + [:div {:class (stl/css :go-back-row)} + [:a {:href (rt/resolve-uri router :auth-login {}) + :on-click on-go-to-login + :class (stl/css :go-back-link)} + (tr "profile.recovery.go-to-login")]]]])) diff --git a/frontend/src/app/main/ui/dashboard/projects.cljs b/frontend/src/app/main/ui/dashboard/projects.cljs index 7953332aac..a6dd98560e 100644 --- a/frontend/src/app/main/ui/dashboard/projects.cljs +++ b/frontend/src/app/main/ui/dashboard/projects.cljs @@ -69,9 +69,16 @@ (mf/defc team-hero* {::mf/wrap [mf/memo]} [{:keys [team on-close]}] - (let [on-nav-members-click + (let [router (mf/deref refs/router) + members-href (rt/resolve-uri router :dashboard-members {:team-id (:id team)}) + + on-nav-members-click (mf/use-fn - #(st/emit! (dcm/go-to-dashboard-members))) + (fn [event] + ;; Ctrl/Cmd-click opens the href in a new tab, as any link. + (when-not (kbd/mod? event) + (dom/prevent-default event) + (st/emit! (dcm/go-to-dashboard-members))))) on-invite (mf/use-fn @@ -95,7 +102,9 @@ [:div {:class (stl/css :title)} (tr "dasboard.team-hero.title")] [:div {:class (stl/css :info)} [:span (tr "dasboard.team-hero.text")] - [:a {:on-click on-nav-members-click} (tr "dasboard.team-hero.management")]] + [:a {:href members-href + :on-click on-nav-members-click} + (tr "dasboard.team-hero.management")]] [:> button* {:variant "primary" :on-click on-invite} (tr "onboarding.choice.team-up.invite-members")]] diff --git a/frontend/src/app/main/ui/dashboard/team.cljs b/frontend/src/app/main/ui/dashboard/team.cljs index b126559cac..25c10850fb 100644 --- a/frontend/src/app/main/ui/dashboard/team.cljs +++ b/frontend/src/app/main/ui/dashboard/team.cljs @@ -48,22 +48,33 @@ [app.util.dom :as dom] [app.util.forms :as uforms] [app.util.i18n :as i18n :refer [tr]] + [app.util.keyboard :as kbd] [app.util.timers :as tm] [beicon.v2.core :as rx] [cuerdas.core :as str] [rumext.v2 :as mf])) +(defn- nav-in-app + "Click handler for a link with href: a plain click navigates inside the + app; Ctrl/Cmd-click leaves the browser to open the href in a new tab." + [event nav-event] + (when-not (kbd/mod? event) + (dom/prevent-default event) + (st/emit! nav-event))) + (mf/defc header* {::mf/wrap [mf/memo]} [{:keys [section team profile]}] - (let [on-nav-members (mf/use-fn #(st/emit! (dcm/go-to-dashboard-members))) - on-nav-settings (mf/use-fn #(st/emit! (dcm/go-to-dashboard-settings))) - on-nav-invitations (mf/use-fn #(st/emit! (dcm/go-to-dashboard-invitations))) - on-nav-webhooks (mf/use-fn #(st/emit! (dcm/go-to-dashboard-webhooks))) + (let [on-nav-members (mf/use-fn #(nav-in-app % (dcm/go-to-dashboard-members))) + on-nav-settings (mf/use-fn #(nav-in-app % (dcm/go-to-dashboard-settings))) + on-nav-invitations (mf/use-fn #(nav-in-app % (dcm/go-to-dashboard-invitations))) + on-nav-webhooks (mf/use-fn #(nav-in-app % (dcm/go-to-dashboard-webhooks))) route (mf/deref refs/route) + router (mf/deref refs/router) invite-email (-> route :query-params :invite-email) team-id (:id team) + section-href #(rt/resolve-uri router % {:team-id team-id}) members-section? (= section :dashboard-team-members) settings-section? (= section :dashboard-team-settings) @@ -110,20 +121,28 @@ [:li [:a {:class (stl/css-case :dashboard-header-link true :active members-section?) + :href (section-href :dashboard-members) + :aria-current (when members-section? "page") :on-click on-nav-members} (tr "labels.members")]] (when is-admin? [:li [:a {:class (stl/css-case :dashboard-header-link true :active invitations-section?) + :href (section-href :dashboard-invitations) + :aria-current (when invitations-section? "page") :on-click on-nav-invitations} (tr "labels.invitations")]]) (when (contains? cfg/flags :webhooks) [:li [:a {:class (stl/css-case :dashboard-header-link true :active webhooks-section?) + :href (section-href :dashboard-webhooks) + :aria-current (when webhooks-section? "page") :on-click on-nav-webhooks} (tr "labels.webhooks")]]) [:li [:a {:class (stl/css-case :dashboard-header-link true :active settings-section?) + :href (section-href :dashboard-settings) + :aria-current (when settings-section? "page") :on-click on-nav-settings} (tr "labels.settings")]]]] [:div {:class (stl/css :dashboard-buttons)} (when (and is-admin? diff --git a/frontend/src/app/main/ui/hooks/wasm_effect.clj b/frontend/src/app/main/ui/hooks/wasm_effect.clj new file mode 100644 index 0000000000..c7c7f11aea --- /dev/null +++ b/frontend/src/app/main/ui/hooks/wasm_effect.clj @@ -0,0 +1,16 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns app.main.ui.hooks.wasm-effect + (:require + [rumext.v2 :as mf])) + +(defmacro with-wasm-effect + "Like `mf/with-effect`, but a recoverable WASM error shows a toast + instead of reaching the React error boundary." + [deps & body] + `(mf/with-effect ~deps + (app.main.errors/wrap-wasm-effect (fn [] ~@body)))) diff --git a/frontend/src/app/main/ui/hooks/wasm_effect.cljs b/frontend/src/app/main/ui/hooks/wasm_effect.cljs new file mode 100644 index 0000000000..f4d624869f --- /dev/null +++ b/frontend/src/app/main/ui/hooks/wasm_effect.cljs @@ -0,0 +1,12 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns app.main.ui.hooks.wasm-effect + "React effects that call WASM. Kept apart from `app.main.ui.hooks`: + the renderer loads that namespace, so it cannot require `app.main.errors`." + (:require-macros [app.main.ui.hooks.wasm-effect]) + (:require + [app.main.errors])) diff --git a/frontend/src/app/main/ui/workspace/file_restored_modal.cljs b/frontend/src/app/main/ui/workspace/file_restored_modal.cljs index badadb4957..c4aee1305f 100644 --- a/frontend/src/app/main/ui/workspace/file_restored_modal.cljs +++ b/frontend/src/app/main/ui/workspace/file_restored_modal.cljs @@ -36,7 +36,9 @@ which was reloaded, and whether their edits not saved yet were lost." {::mf/register modal/components ::mf/register-as :file-restored} - [{:keys [fullname label created-at unsaved]}] + ;; The modal wrapper passes the props map without camelCasing its keys, + ;; so every prop of this component must be a one-word name. + [{:keys [fullname label date unsaved]}] (let [on-close (mf/use-fn #(st/emit! (modal/hide)))] [:div {:class (stl/css :modal-overlay)} [:div {:class (stl/css :modal-dialog) @@ -49,7 +51,7 @@ :class (stl/css :modal-title)} (tr "workspace.file-restored.title")] [:> text* {:as "p" :typography t/body-large} - (restored-message fullname label created-at)] + (restored-message fullname label date)] (when unsaved [:> text* {:as "p" :typography t/body-large} (tr "workspace.file-restored.unsaved-changes")]) diff --git a/frontend/src/app/main/ui/workspace/shapes/text/v3_editor.cljs b/frontend/src/app/main/ui/workspace/shapes/text/v3_editor.cljs index 403f6c3030..21b6a23d1a 100644 --- a/frontend/src/app/main/ui/workspace/shapes/text/v3_editor.cljs +++ b/frontend/src/app/main/ui/workspace/shapes/text/v3_editor.cljs @@ -14,9 +14,11 @@ [app.main.data.workspace :as dw] [app.main.data.workspace.texts :as dwt] [app.main.data.workspace.undo :as dwu] + [app.main.errors :as errors] [app.main.refs :as refs] [app.main.store :as st] [app.main.ui.css-cursors :as cur] + [app.main.ui.hooks.wasm-effect :refer [with-wasm-effect]] [app.render-wasm.api :as wasm.api] [app.render-wasm.text-editor :as text-editor] [app.util.dom :as dom] @@ -607,48 +609,48 @@ (.removeEventListener node "beforeinput" on-before-input))))) ;; Focus contenteditable on mount - (mf/use-effect - (mf/deps contenteditable-ref) - (fn [] - ;; Group the whole editing session (edits, reflow resizes, finalize) into a single - ;; undo entry. Nested transactions (e.g. style shortcuts) are ref-counted and fold in. - (st/emit! (dwu/start-undo-transaction shape-id :timeout nil)) - (when-let [node (mf/ref-val contenteditable-ref)] - ;; Focus and select all text on mount (this will trigger on-focus) - (.focus node) - (text-editor/text-editor-select-all) - (wasm.api/request-render-preserving-target "text-editor-select-all-on-mount")) - ;; On unmount, finalize the editor content and then dispose the WASM editor. - ;; We finalize on unmount instead of relying on the browser blur event, because - ;; it was not being reliable (timing issues, Firefox issues…) - (fn [] - (on-blur) - (st/emit! dw/hide-context-menu - (dwt/v3-stop-text-layout-preview shape-id) - (dwu/commit-undo-transaction shape-id)) - (text-editor/text-editor-dispose) - (wasm.api/request-render-preserving-target "text-editor-dispose")))) - - (mf/use-effect - (mf/deps) - (fn [] - (let [timeout-id (atom nil) - schedule-blink (fn schedule-blink [] - ;; The caret only blinks for a collapsed cursor. With an active - ;; selection there is nothing to animate, so skip the repaint: - ;; re-compositing every interval would otherwise redraw the - ;; selection over and over (a visible flicker at high zoom). - (when (and (text-editor/text-editor-has-focus?) - (not (text-editor/text-editor-has-selection?))) - ;; Redraw only the caret (cached frame + overlay) instead of a - ;; full `request-render`, which flashes on zoomed-in views by - ;; kicking off a progressive tile-by-tile shape re-render. - (wasm.api/render-text-editor-overlay!)) - (reset! timeout-id (js/setTimeout schedule-blink caret-blink-interval-ms)))] - (schedule-blink) + (with-wasm-effect [contenteditable-ref] + ;; Group the whole editing session (edits, reflow resizes, finalize) into a single + ;; undo entry. Nested transactions (e.g. style shortcuts) are ref-counted and fold in. + (st/emit! (dwu/start-undo-transaction shape-id :timeout nil)) + (when-let [node (mf/ref-val contenteditable-ref)] + ;; Focus and select all text on mount (this will trigger on-focus) + (.focus node) + ;; Guarded apart so a recoverable error still returns the cleanup below, + ;; which commits the undo transaction opened above. + (errors/wrap-wasm-effect (fn [] - (when @timeout-id - (js/clearTimeout @timeout-id)))))) + (text-editor/text-editor-select-all) + (wasm.api/request-render-preserving-target "text-editor-select-all-on-mount")))) + ;; On unmount, finalize the editor content and then dispose the WASM editor. + ;; We finalize on unmount instead of relying on the browser blur event, because + ;; it was not being reliable (timing issues, Firefox issues…) + (fn [] + (on-blur) + (st/emit! dw/hide-context-menu + (dwt/v3-stop-text-layout-preview shape-id) + (dwu/commit-undo-transaction shape-id)) + (text-editor/text-editor-dispose) + (wasm.api/request-render-preserving-target "text-editor-dispose"))) + + (with-wasm-effect [] + (let [timeout-id (atom nil) + schedule-blink (fn schedule-blink [] + ;; The caret only blinks for a collapsed cursor. With an active + ;; selection there is nothing to animate, so skip the repaint: + ;; re-compositing every interval would otherwise redraw the + ;; selection over and over (a visible flicker at high zoom). + (when (and (text-editor/text-editor-has-focus?) + (not (text-editor/text-editor-has-selection?))) + ;; Redraw only the caret (cached frame + overlay) instead of a + ;; full `request-render`, which flashes on zoomed-in views by + ;; kicking off a progressive tile-by-tile shape re-render. + (wasm.api/render-text-editor-overlay!)) + (reset! timeout-id (js/setTimeout schedule-blink caret-blink-interval-ms)))] + (schedule-blink) + (fn [] + (when @timeout-id + (js/clearTimeout @timeout-id))))) ;; Composition and input events [:g.text-editor {:clip-path (dm/fmt "url(#%)" clip-id) diff --git a/frontend/src/app/main/ui/workspace/shapes/text/viewport_texts_html.cljs b/frontend/src/app/main/ui/workspace/shapes/text/viewport_texts_html.cljs index 3869f62a03..87b65864a3 100644 --- a/frontend/src/app/main/ui/workspace/shapes/text/viewport_texts_html.cljs +++ b/frontend/src/app/main/ui/workspace/shapes/text/viewport_texts_html.cljs @@ -97,12 +97,14 @@ (st/emit! (dwt/resize-text id width height))))) (st/emit! (dwt/clean-text-modifier id)))) - ;; Always clear the task and log measurement errors. - (p/catch (fn [cause] - (log/error :hint "Could not measure text shape" - :shape-id id - :cause cause) - nil)))) + ;; Log measurement errors and resolve to nil. The returned promise + ;; stays pending until the measurement settles, so callers keep the + ;; text mounted while it is measured. + (p/merr (fn [cause] + (log/error :hint "Could not measure text shape" + :shape-id id + :cause cause) + (p/resolved nil))))) (defn- update-text-modifier [{:keys [grow-type id] :as shape} node] diff --git a/frontend/src/app/main/ui/workspace/tokens/export/modal.cljs b/frontend/src/app/main/ui/workspace/tokens/export/modal.cljs index b1c9ee0f09..1d5db32d98 100644 --- a/frontend/src/app/main/ui/workspace/tokens/export/modal.cljs +++ b/frontend/src/app/main/ui/workspace/tokens/export/modal.cljs @@ -47,14 +47,27 @@ :on-click on-export} (tr "workspace.tokens.export")]]]) +(defn- single-file-json + "The tokens of the file as one JSON file. Its active themes and sets are + the ones activated in this file, not the ones stored in the library." + [tokens-lib tokens-status] + (when (and tokens-lib tokens-status) + (some-> (ctob/export-dtcg-json tokens-lib tokens-status) + (json/encode :key-fn identity :indent 2)))) + +(defn- multi-file-entries + "The tokens of the file as one JSON file per set, plus the themes and + the metadata, with the active themes and sets of `single-file-json`." + [tokens-lib tokens-status] + (when (and tokens-lib tokens-status) + (ctob/export-dtcg-multi-file tokens-lib tokens-status))) + (mf/defc single-file-tab* {::mf/private true} [] - (let [tokens-data (some-> (deref refs/tokens-lib) - (ctob/export-dtcg-json)) - tokens-json (some-> tokens-data - (json/encode :key-fn identity :indent 2)) - is-disabled (empty? tokens-data) + (let [tokens-json (single-file-json (deref refs/tokens-lib) + (deref refs/tokens-status)) + is-disabled (nil? tokens-json) on-export (mf/use-fn (mf/deps tokens-json) @@ -79,8 +92,8 @@ (mf/defc multi-file-tab* {::mf/private true} [] - (let [files (some->> (deref refs/tokens-lib) - (ctob/export-dtcg-multi-file)) + (let [files (multi-file-entries (deref refs/tokens-lib) + (deref refs/tokens-status)) is-disabled (or (empty? files) (every? (fn [[_ v]] (empty? v)) files)) on-export diff --git a/frontend/src/app/main/ui/workspace/viewport/widgets.cljs b/frontend/src/app/main/ui/workspace/viewport/widgets.cljs index 2c5374ced8..766f4987cd 100644 --- a/frontend/src/app/main/ui/workspace/viewport/widgets.cljs +++ b/frontend/src/app/main/ui/workspace/viewport/widgets.cljs @@ -9,6 +9,7 @@ (:require [app.common.data :as d] [app.common.data.macros :as dm] + [app.common.files.helpers :as cfh] [app.common.geom.point :as gpt] [app.common.types.component :as ctk] [app.common.types.container :as ctn] @@ -262,34 +263,39 @@ [{:keys [objects base-objects zoom selected focus is-show-artboard-names on-frame-enter on-frame-leave on-frame-select]}] (let [selected (or selected #{}) - frame-ids (mf/with-memo [base-objects] - (vec (ctt/get-frames-ids base-objects {:skip-copies? true :ignore-index? true}))) - shapes (into [] (keep (d/getf objects)) frame-ids) - shapes (if (dbg/enabled? :shape-titles) - (into (set shapes) - (map (d/getf objects)) - selected) - shapes) + show-id? (dbg/enabled? :shape-titles) + root-frames (into [] + (comp (keep (d/getf objects)) + (filter cfh/frame-shape?) + (filter #(= uuid/zero (:parent-id %))) + (remove #(and (ctk/instance-head? %) + (not (ctk/main-instance? %))))) + (dm/get-in objects [uuid/zero :shapes])) + shapes (if show-id? + (into [] + (comp (keep (d/getf objects)) + (filter (fn [{:keys [id]}] + (and (not= id uuid/zero) + (or (empty? focus) (contains? focus id)))))) + (into (set (ctt/get-frames-ids base-objects {:skip-copies? true :ignore-index? true})) + selected)) + root-frames) edition (mf/deref refs/selected-edition) grid-edition? (ctl/grid-layout? objects edition)] [:g.frame-titles.blurrable - (for [{:keys [id parent-id] :as shape} shapes] - (when (and - (not= id uuid/zero) - (or (dbg/enabled? :shape-titles) (= parent-id uuid/zero)) - (or (empty? focus) (contains? focus id))) - [:> frame-title* {:key (dm/str "frame-title-" id) - :frame shape - :zoom zoom - :is-selected (contains? selected id) - :is-show-artboard-names is-show-artboard-names - :is-show-id (dbg/enabled? :shape-titles) - :is-grid-edition (and (= id edition) grid-edition?) - :on-frame-enter on-frame-enter - :on-frame-leave on-frame-leave - :on-frame-select on-frame-select}]))])) + (for [{:keys [id] :as shape} shapes] + [:> frame-title* {:key (dm/str "frame-title-" id) + :frame shape + :zoom zoom + :is-selected (contains? selected id) + :is-show-artboard-names is-show-artboard-names + :is-show-id show-id? + :is-grid-edition (and (= id edition) grid-edition?) + :on-frame-enter on-frame-enter + :on-frame-leave on-frame-leave + :on-frame-select on-frame-select}])])) (mf/defc frame-flow* [{:keys [flow frame is-selected zoom on-frame-enter on-frame-leave on-frame-select]}] diff --git a/frontend/src/app/main/ui/workspace/viewport_wasm.cljs b/frontend/src/app/main/ui/workspace/viewport_wasm.cljs index 23a319f991..75dcfe7c45 100644 --- a/frontend/src/app/main/ui/workspace/viewport_wasm.cljs +++ b/frontend/src/app/main/ui/workspace/viewport_wasm.cljs @@ -27,6 +27,7 @@ [app.main.ui.context :as ctx] [app.main.ui.flex-controls :as mfc] [app.main.ui.hooks :as ui-hooks] + [app.main.ui.hooks.wasm-effect :refer [with-wasm-effect]] [app.main.ui.measurements :as msr] [app.main.ui.workspace.review.overlays :refer [review-overlays*]] [app.main.ui.workspace.shapes.path.editor :refer [path-editor*]] @@ -467,7 +468,7 @@ ;; canvas, even though we are not using `page-id` inside the hook. ;; We think moving this out to a handler will make the render code ;; harder to follow through. - (mf/with-effect [page-id] + (with-wasm-effect [page-id] (when-let [canvas (mf/ref-val canvas-ref)] (let [timeout-id-ref (volatile! nil) unmounted? (volatile! false) @@ -516,7 +517,7 @@ (js/clearTimeout timeout-id)) (wasm.api/clear-canvas))))) - (mf/with-effect [show-text-editor? workspace-editor-state edition @canvas-init? @initialized?] + (with-wasm-effect [show-text-editor? workspace-editor-state edition @canvas-init? @initialized?] (let [active-editor-state (get workspace-editor-state edition)] (when (and show-text-editor? active-editor-state @canvas-init? @initialized?) (let [content (-> active-editor-state @@ -538,16 +539,16 @@ (ts/raf (fn [_] (wasm.api/render-sync)))))] cancel))) - (mf/with-effect [vport] + (with-wasm-effect [vport] (when (and @canvas-init? @initialized?) (wasm.api/resize-canvas! (mf/ref-val canvas-ref)) (wasm.api/set-view-box zoom vbox))) - (mf/with-effect [@canvas-init? preview-blend] + (with-wasm-effect [@canvas-init? preview-blend] (when (and @canvas-init? preview-blend) (wasm.api/request-render "with-effect"))) - (mf/with-effect [@canvas-init? vern zoom vbox background] + (with-wasm-effect [@canvas-init? vern zoom vbox background] (when @canvas-init? (if (not @initialized?) (do @@ -580,27 +581,27 @@ (wasm.api/clear-focus-mode) (wasm.api/set-focus-mode focus))))) - (mf/with-effect [background] + (with-wasm-effect [background] (when (and @canvas-init? @initialized?) (wasm.api/set-canvas-background background))) ;; Grid overlay: `clear-grid` must run only when the overlay was shown and is now off ;; (e.g. leave grid frame, or `page-transition?`). Do not call it on every ;; `hover-top-frame-id` change while not hovering a grid frame. - (mf/with-effect [@canvas-init? hover-grid?] + (with-wasm-effect [@canvas-init? hover-grid?] (when @canvas-init? (when (and (not hover-grid?) (mf/ref-val prev-hover-grid-shown?-ref)) (wasm.api/clear-grid)) (mf/set-ref-val! prev-hover-grid-shown?-ref hover-grid?))) - (mf/with-effect [@canvas-init? has-grid? hover-grid? - (if (and has-grid? hover-grid?) @hover-top-frame-id ::no-grid-hover-id)] + (with-wasm-effect [@canvas-init? has-grid? hover-grid? + (if (and has-grid? hover-grid?) @hover-top-frame-id ::no-grid-hover-id)] (when (and @canvas-init? hover-grid?) (wasm.api/show-grid @hover-top-frame-id))) ;; Rulers-wasm: push visibility / offsets / selection band into the ;; render-wasm overlay (always active, no feature flag). - (mf/with-effect [@canvas-init?] + (with-wasm-effect [@canvas-init?] (when @canvas-init? (wasm.api/push-ruler-theme-colors!) (theme/add-color-scheme-listener! @@ -611,7 +612,7 @@ ;; Text-editor-wasm: push the theme colors (selection background, caret) ;; into the WASM text editor so the selection follows the design tokens per ;; theme (purple on light, teal on dark) instead of a hardcoded default. - (mf/with-effect [@canvas-init?] + (with-wasm-effect [@canvas-init?] (when @canvas-init? (wasm.api/text-editor-apply-theme) (theme/add-color-scheme-listener! @@ -626,37 +627,37 @@ ;; off a progressive tile-by-tile re-render that flashes) or a cached-atlas ;; blit (whose scale-capped atlas flashes crisp->blurry on zoomed-in views, ;; e.g. when the text editor opens at high zoom). - (mf/with-effect [@canvas-init? frame-visible?] + (with-wasm-effect [@canvas-init? frame-visible?] (when @canvas-init? (wasm.api/set-rulers-frame-visible! frame-visible?) (wasm.api/render-from-backbuffer!))) - (mf/with-effect [@canvas-init? show-rulers?] + (with-wasm-effect [@canvas-init? show-rulers?] (when @canvas-init? (wasm.api/set-rulers-visible! show-rulers?) (wasm.api/render-from-backbuffer!))) - (mf/with-effect [@canvas-init? show-rulers? offset-x offset-y] + (with-wasm-effect [@canvas-init? show-rulers? offset-x offset-y] (when (and @canvas-init? show-rulers?) (wasm.api/set-rulers-offsets! offset-x offset-y))) - (mf/with-effect [@canvas-init? show-rulers? - (some-> ruler-selection :x) (some-> ruler-selection :y) - (some-> ruler-selection :width) (some-> ruler-selection :height)] + (with-wasm-effect [@canvas-init? show-rulers? + (some-> ruler-selection :x) (some-> ruler-selection :y) + (some-> ruler-selection :width) (some-> ruler-selection :height)] (when (and @canvas-init? show-rulers?) (wasm.api/set-rulers-selection! ruler-selection) (wasm.api/render-from-backbuffer!))) ;; Paint background + rulers instantly, before shapes finish loading. Runs ;; after the ruler push effects so the WASM ruler state is already set. - (mf/with-effect [@canvas-init? page-id] + (with-wasm-effect [@canvas-init? page-id] (when @canvas-init? (wasm.api/render-ui-only))) ;; Ruler guides: push the WASM-visible guide set to the render engine. ;; `wasm-guides` is also passed to the SVG overlay for index-based hit ;; testing — it must stay in sync with what we serialize here. - (mf/with-effect [@canvas-init? wasm-guides objects] + (with-wasm-effect [@canvas-init? wasm-guides objects] (when @canvas-init? (wasm.api/set-guides wasm-guides objects))) @@ -669,7 +670,7 @@ (hooks/setup-shortcuts path-editing? path-drawing? text-editing? grid-editing?) (hooks/setup-active-frames base-objects hover-ids selected active-frames zoom transform vbox) - (mf/with-effect [path-editing? edition @initialized?] + (with-wasm-effect [path-editing? edition @initialized?] (when (and path-editing? edition @initialized?) (wselect/use-shape edition) (wasm.api/set-shape-hidden true) diff --git a/frontend/src/app/plugins/api.cljs b/frontend/src/app/plugins/api.cljs index 92ca262da5..3043ff510b 100644 --- a/frontend/src/app/plugins/api.cljs +++ b/frontend/src/app/plugins/api.cljs @@ -46,6 +46,7 @@ [app.plugins.management :as management] [app.plugins.page :as page] [app.plugins.parser :as parser] + [app.plugins.persist :as wpsp] [app.plugins.reflow :as wrfp] [app.plugins.register :as r] [app.plugins.shape :as shape] @@ -765,4 +766,9 @@ :waitForLayoutUpdate (fn [timeout] ;; Resolves once every shape with reflow work in flight has settled. - (wrfp/wait-for-layout-update timeout)))) + (wrfp/wait-for-layout-update timeout)) + + :waitForSave + (fn [timeout] + ;; Resolves once pending edits are saved (or there is nothing to save). + (wpsp/wait-for-save timeout)))) diff --git a/frontend/src/app/plugins/persist.cljs b/frontend/src/app/plugins/persist.cljs new file mode 100644 index 0000000000..0d3c81eb59 --- /dev/null +++ b/frontend/src/app/plugins/persist.cljs @@ -0,0 +1,130 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns app.plugins.persist + "Promise adapter for the plugin `waitForSave` method. Owns timeout + validation, the short commit→buffer intake race, force-flush, and the + save-vs-rollback race; persistence itself lives in + `app.main.data.persistence`." + (:require + [app.common.data.macros :as dm] + [app.main.data.persistence :as dps] + [app.main.refs :as refs] + [app.main.store :as st] + [app.plugins.utils :as u] + [app.util.object :as obj] + [beicon.v2.core :as rx] + [potok.v2.core :as ptk])) + +;; Leaves room for the 30s layout wait and the code itself under the +;; default MCP tool timeout (120s). +(def ^:private default-timeout 60000) + +;; Primary quiescence: how long to wait for commits to enter buffer/queue +;; before treating empty persistence as "nothing to save". Not the 200ms +;; pending-status debounce. +(def ^:private intake-ms 100) + +(defn- pending-work? + "True when persistence still holds buffered or queued commits." + [{:keys [buffer queue]}] + (boolean (or (seq buffer) (seq queue)))) + +(defn- ->js-error + "Turns a cljs/JS failure into a JS Error for Promise rejection, keeping + the `ex-data` `:code` as the Error `code` property." + [cause] + (let [err (if (instance? js/Error cause) + cause + (js/Error. (or (ex-message cause) (str cause)))) + code (:code (ex-data cause))] + (when (keyword? code) + (obj/set! err "code" (name code))) + err)) + +(def ^:private earlier-failure-message + (dm/str "Changes could not be saved: the file has a failed save that " + "stays until the file is reopened, so no change can be confirmed " + "as saved. Reopen the file to resume saving.")) + +(def ^:private retrying-message + (dm/str "Changes could not be saved yet: saves are failing and Penpot " + "keeps retrying them. Try again later.")) + +(defn- earlier-failure + "Rejection for a wait that starts while persistence is `:error`." + [pstate] + (if (dps/save-retrying? pstate) + (ex-info retrying-message {:type :persistence :code :save-retrying}) + (ex-info earlier-failure-message + {:type :persistence + :code (get-in pstate [:error :code] :save-failed)}))) + +(defn- wait-intake + "Races buffer/queue becoming non-empty against an intake deadline of + `delay-ms` (capped by the caller to the wait timeout)." + [delay-ms] + (->> (rx/race (->> (rx/from-atom refs/persistence {:emit-current-value? true}) + (rx/filter pending-work?) + (rx/take 1) + (rx/map (constantly :pending))) + (->> (rx/of :empty) + (rx/delay delay-ms))) + (rx/take 1))) + +(defn- wait-save-or-rollback + "Races a clean persist settle against rollback start or completion. + Soft rejection emits `::rollback-rejected` before settle can publish + empty `:saved`; `::rolled-back` may arrive later after a reload." + [timeout-ms] + (rx/race (->> (dps/wait-persisted-or-error timeout-ms) + (rx/map (constantly :saved))) + (->> st/stream + (rx/filter (some-fn (ptk/type? ::dps/rollback-rejected) + (ptk/type? ::dps/rolled-back))) + (rx/take 1) + (rx/mapcat (fn [_] + (rx/throw (ex-info "Changes were rolled back" + {:type :persistence :code :rolled-back}))))))) + +(defn wait-for-save + "Returns a JS Promise that resolves once pending file changes are saved, + or there is nothing to save after a short intake window. + + Rejects when `timeout` is invalid, the save fails (also a failure from + before the call: a terminal one stays until the file is reopened), + changes are rolled back during the wait, or the deadline elapses. The + Error `code` is `save-timeout`, `rolled-back`, `save-retrying` (an + earlier failure Penpot keeps retrying), or the failure code + (`save-failed` unless the backend gave a more specific one). A nil + `timeout` uses `default-timeout`." + ([] (wait-for-save nil)) + ([timeout] + (js/Promise. + (fn [resolve reject] + (if-not (u/valid-timeout? timeout) + (u/reject-not-valid reject :waitForSave timeout) + (let [timeout-ms (or timeout default-timeout) + delay-ms (js/Math.min intake-ms timeout-ms)] + (->> (wait-intake delay-ms) + (rx/mapcat + (fn [_] + (let [pstate @refs/persistence] + (cond + (= :error (:status pstate)) + (rx/throw (earlier-failure pstate)) + + (pending-work? pstate) + (do + (st/emit! ::dps/force-persist) + (wait-save-or-rollback timeout-ms)) + + :else + (rx/of :empty))))) + (rx/take 1) + (rx/subs! + (fn [_] (resolve)) + (fn [cause] (reject (->js-error cause))))))))))) diff --git a/frontend/src/app/plugins/reflow.cljs b/frontend/src/app/plugins/reflow.cljs index da92d7c1a2..c2cf60625d 100644 --- a/frontend/src/app/plugins/reflow.cljs +++ b/frontend/src/app/plugins/reflow.cljs @@ -9,35 +9,16 @@ argument validation, the default deadline and the rejection shape; the workspace only reports when its pending work has drained." (:require - [app.common.data.macros :as dm] [app.common.files.helpers :as cfh] [app.common.uuid :as uuid] [app.main.data.workspace.reflow :as wrf] + [app.plugins.utils :as u] [beicon.v2.core :as rx])) ;; Ceiling for callers that pass no timeout, so a pipeline that never drains ;; its marks rejects the promise rather than leaving it unsettled. (def ^:private default-timeout 30000) -;; Largest value a signed 32-bit timer accepts. -(def ^:private max-timeout 2147483647) - -(defn- valid-timeout? - "Checks that a plugin timeout fits a signed 32-bit timer." - [value] - (or (nil? value) - (and (number? value) - (pos? value) - (<= value max-timeout) - (js/Number.isFinite value)))) - -(defn- reject-invalid! - [reject value] - (let [msg (dm/str "[PENPOT PLUGIN] Value not valid: " value - ". Code: " :waitForLayoutUpdate)] - (.error js/console msg) - (reject (js/Error. msg)))) - (defn shape-wait-ids "Ids a per-shape wait covers: the shape subtree, its ancestors, and the file its components sync from." @@ -59,8 +40,8 @@ ([ids timeout] (js/Promise. (fn [resolve reject] - (if-not (valid-timeout? timeout) - (reject-invalid! reject timeout) + (if-not (u/valid-timeout? timeout) + (u/reject-not-valid reject :waitForLayoutUpdate timeout) ;; Race the settle signal against the deadline; the loser is ;; unsubscribed. `settled` replays on subscribe, so an already drained ;; map wins even against a 1ms deadline. diff --git a/frontend/src/app/plugins/utils.cljs b/frontend/src/app/plugins/utils.cljs index a9a6479de4..296b7f8465 100644 --- a/frontend/src/app/plugins/utils.cljs +++ b/frontend/src/app/plugins/utils.cljs @@ -341,10 +341,24 @@ (display-not-valid code value))) (defn reject-not-valid + "Rejects a promise with the standard invalid-value Error and logs it." [reject code value] (let [msg (dm/str "[PENPOT PLUGIN] Value not valid: " value ". Code: " code)] (.error js/console msg) - (reject msg))) + (reject (js/Error. msg)))) + +(def max-timeout + "Largest value a signed 32-bit timer accepts." + 2147483647) + +(defn valid-timeout? + "Checks that a plugin timeout fits a signed 32-bit timer." + [value] + (or (nil? value) + (and (number? value) + (pos? value) + (<= value max-timeout) + (js/Number.isFinite value)))) (defn mixed-value [values] diff --git a/frontend/src/app/worker/exports.cljs b/frontend/src/app/worker/exports.cljs index fa83b4ef29..5513ea029f 100644 --- a/frontend/src/app/worker/exports.cljs +++ b/frontend/src/app/worker/exports.cljs @@ -128,6 +128,14 @@ :name name :suffix suffix}) +(defn local-export-filename + "A single export's `:name` already carries the suffix (added by the UI)." + [kind entry export-name] + (case kind + :pdf-frames (str (or export-name "export") ".pdf") + :single (wasm.exports/sanitize-filename (dissoc entry :suffix) #{}) + (str (or export-name "export") ".zip"))) + (defn- run-local-export [{:keys [job-id kind exports objects background] export-name :name}] (let [exports (mapv export-entry exports)] @@ -162,14 +170,7 @@ :else (let [entry (first exports) - filename (case kind - :pdf-frames - (str (or export-name "export") ".pdf") - - :single - (wasm.exports/sanitize-filename entry #{}) - - (str (or export-name "export") ".zip")) + filename (local-export-filename kind entry export-name) mtype (case kind :pdf-frames "application/pdf" :single (export-mtype (:type entry)) diff --git a/frontend/test/frontend_tests/data/team_test.cljs b/frontend/test/frontend_tests/data/team_test.cljs index d05041c0db..3b6abffc4f 100644 --- a/frontend/test/frontend_tests/data/team_test.cljs +++ b/frontend/test/frontend_tests/data/team_test.cljs @@ -10,7 +10,6 @@ [app.main.repo :as rp] [app.main.router :as rt] [app.main.ui.dashboard.team :as dtui] - [app.util.dom :as dom] [app.util.session-state :as ss] [beicon.v2.core :as rx] [cljs.test :as t :include-macros true] @@ -35,8 +34,8 @@ (let [team-id "00000000-0000-4000-8000-000000000001" organization-id "00000000-0000-4000-8000-000000000002" profile-id "00000000-0000-4000-8000-000000000003" - dashboard-url "https://penpot.example/#/dashboard-recent?team-id=current-team" - callback-url (str dashboard-url "&pending-action-id=pending-id") + route-token "?screen=dashboard-recent&team-id=current-team" + callback-query "screen=dashboard-recent&team-id=current-team&pending-action-id=pending-id" team {:id team-id :organization {:id organization-id :name "Product" @@ -54,8 +53,7 @@ :check-nitrate-sso (rx/of {:authorized false :redirect-uri "https://sso.example.test/login"})))) - rt/get-current-href (mock/stub (constantly dashboard-url)) - dom/append-query-param (mock/stub (fn [_url _key _value] callback-url)) + rt/get-current-token (mock/stub (constantly route-token)) uuid/next (mock/stub (constantly "pending-id")) ss/save-pending-action! (mock/stub (fn [& args] (swap! saved-actions conj args))) @@ -68,7 +66,7 @@ :on-next #(swap! emitted conj %))))) (t/is (= [[:get-teams nil] - [:check-nitrate-sso {:team-id team-id :url callback-url}]] + [:check-nitrate-sso {:team-id team-id :query-string callback-query}]] @requests)) (t/is (= [["pending-id" {:type :create-team-in-organization :team-id team-id}]] @@ -81,8 +79,8 @@ (let [team-id "00000000-0000-4000-8000-000000000001" organization-id "00000000-0000-4000-8000-000000000002" profile-id "00000000-0000-4000-8000-000000000003" - dashboard-url "https://penpot.example/#/dashboard-recent?team-id=current-team" - callback-url (str dashboard-url "&pending-action-id=pending-id") + route-token "?screen=dashboard-recent&team-id=current-team" + callback-query "screen=dashboard-recent&team-id=current-team&pending-action-id=pending-id" team {:id team-id :organization {:id organization-id :name "Product" @@ -97,8 +95,7 @@ (case command :get-teams (rx/of [team]) :check-nitrate-sso (rx/of {:authorized true})))) - rt/get-current-href (mock/stub (constantly dashboard-url)) - dom/append-query-param (mock/stub (fn [_url _key _value] callback-url)) + rt/get-current-token (mock/stub (constantly route-token)) uuid/next (mock/stub (constantly "pending-id")) modal/show (mock/stub (fn [& args] {:modal-show args}))} (await (async/observe @@ -107,7 +104,7 @@ nil) :on-next #(swap! emitted conj %))))) - (t/is (= [:check-nitrate-sso {:team-id team-id :url callback-url}] + (t/is (= [:check-nitrate-sso {:team-id team-id :query-string callback-query}] (last @requests))) (t/is (= {:modal-show [:team-form {:organization-id organization-id :organization-name "Product"}]} diff --git a/frontend/test/frontend_tests/data/workspace_notifications_test.cljs b/frontend/test/frontend_tests/data/workspace_notifications_test.cljs index 0ad8a12697..c6b4d721b4 100644 --- a/frontend/test/frontend_tests/data/workspace_notifications_test.cljs +++ b/frontend/test/frontend_tests/data/workspace_notifications_test.cljs @@ -140,7 +140,7 @@ props (:props (shown-modal events))] (t/is (= "Ada" (:fullname props))) (t/is (= "Hand-off v2" (:label props))) - (t/is (= created-at (:created-at props))) + (t/is (= created-at (:date props))) (t/is (true? (:unsaved props))))) ;; Scenario: the restoring session left before its message arrived, so it diff --git a/frontend/test/frontend_tests/data/workspace_shortcuts_test.cljs b/frontend/test/frontend_tests/data/workspace_shortcuts_test.cljs index 3d873a5260..6d498762a2 100644 --- a/frontend/test/frontend_tests/data/workspace_shortcuts_test.cljs +++ b/frontend/test/frontend_tests/data/workspace_shortcuts_test.cljs @@ -7,7 +7,12 @@ (ns frontend-tests.data.workspace-shortcuts-test (:require [app.config :as cf] + [app.main.data.shortcuts :as ds] + [app.main.data.workspace.path :as drp] + [app.main.data.workspace.path.shortcuts :as path-shortcuts] [app.main.data.workspace.shortcuts :as shortcuts] + [app.main.data.workspace.transforms :as dwt] + [app.main.refs :as refs] [app.main.store :as st] [cljs.test :as t :include-macros true])) @@ -110,3 +115,50 @@ (keyboard-event {:ctrl? true :code "Backslash" :prevented? prevented?}))) (t/is (false? @prevented?)) (t/is (empty? @emitted))))) + +(def ^:private arrows + {:up ds/up-arrow + :down ds/down-arrow + :left ds/left-arrow + :right ds/right-arrow}) + +(defn- move-shortcuts + "The move shortcuts of `shortcuts-map` as `[id direction fast?]`." + [shortcuts-map] + (for [dir (keys arrows) + fast? [false true] + :let [id (keyword (str (if fast? "move-fast-" "move-unit-") (name dir)))] + :when (contains? shortcuts-map id)] + [id dir fast?])) + +(defn- run-move-shortcut + "Runs the :fn of shortcut `id` in `shortcuts-map` with the moves stubbed + and returns the moves it emitted, as `[direction fast?]`." + [shortcuts-map id] + (let [emitted (atom [])] + (with-redefs [dwt/move-selected (fn [dir fast?] [::move dir fast?]) + drp/move-selected (fn [dir fast?] [::move dir fast?]) + refs/permissions (atom {:can-edit true}) + refs/workspace-read-only? (atom false) + st/emit! (mock-emit! emitted)] + ((:fn (get shortcuts-map id)))) + (into [] (keep (fn [[tag dir fast?]] (when (= ::move tag) [dir fast?]))) @emitted))) + +(t/deftest move-shortcuts-move-in-their-direction + (doseq [[map-name shortcuts-map] [["workspace" shortcuts/shortcuts] + ["path editor" path-shortcuts/shortcuts]] + [id dir fast?] (move-shortcuts shortcuts-map)] + (t/is (= [[dir fast?]] (run-move-shortcut shortcuts-map id)) + (str map-name " " id " moves " (name dir) (when fast? " fast"))))) + +(t/deftest move-shortcuts-show-the-arrow-of-their-direction + (doseq [[map-name shortcuts-map] [["workspace" shortcuts/shortcuts] + ["path editor" path-shortcuts/shortcuts]] + [id dir fast?] (move-shortcuts shortcuts-map)] + (t/is (= (cond-> (get arrows dir) fast? ds/shift) + (:tooltip (get shortcuts-map id))) + (str map-name " " id " shows the " (name dir) " arrow")))) + +(t/deftest move-shortcuts-cover-every-direction + (t/is (= 8 (count (move-shortcuts shortcuts/shortcuts)))) + (t/is (= 8 (count (move-shortcuts path-shortcuts/shortcuts))))) diff --git a/frontend/test/frontend_tests/main_errors_test.cljs b/frontend/test/frontend_tests/main_errors_test.cljs index 89b9b6bf73..c71db79a44 100644 --- a/frontend/test/frontend_tests/main_errors_test.cljs +++ b/frontend/test/frontend_tests/main_errors_test.cljs @@ -1198,3 +1198,115 @@ "errors.save-refused-merged-branch" "errors.save-refused-archived-branch")) (get-in state [:notification :content]))))))))) + +;; --------------------------------------------------------------------------- +;; WASM errors +;; --------------------------------------------------------------------------- + +(defn- ^:async handle-wasm-error + "Sends a `:wasm-error` with `code` through `on-error`. Resolves to the + errors assigned to the error page and the events emitted." + [code] + (let [events (atom []) + assigned (atom []) + cause (ex-info "WASM Error" {:type :wasm-error :code code :hint "WASM Error"})] + (await + (mock/with-mocks* + {st/emit! (mock/stub (fn [& emitted] (swap! events into emitted))) + rt/get-current-href (constantly "https://penpot.example.com/#/workspace") + rt/assign-exception (fn [error] (swap! assigned conj error) error) + tm/schedule (mock/stub (fn [f] (f))) + st/format-last-events (mock/stub (fn [& _] "(stub last events)"))} + (errors/on-error cause) + (await (async/settle)))) + {:assigned @assigned :events @events})) + +(t/deftest ^:async non-recoverable-wasm-errors-show-the-error-page + ;; Scenario: a panic, a trap with no code and an unknown code. + ;; Proves: each shows the error page and no toast. + (doseq [code [:panic :wasm-critical :unknown-code]] + (errors/reset-report-governor!) + (let [{:keys [assigned events]} (await (handle-wasm-error code))] + (t/is (= [code] (map :code assigned)) + (str code " must show the error page")) + (t/is (empty? (report-events events)) + (str code " must not show a toast"))))) + +(t/deftest ^:async recoverable-wasm-errors-show-a-toast + ;; Scenario: WASM errors with a recoverable code. Proves: each shows a + ;; handled report and a toast, and keeps the page. + (doseq [code [:non-blocking :wasm-reload-context-failure :viewer-canvas-failed]] + (errors/reset-report-governor!) + (let [{:keys [assigned events]} (await (handle-wasm-error code))] + (t/is (empty? assigned) + (str code " must not show the error page")) + (t/is (= ["handled-exception"] + (map (comp ::ev/name deref) (report-events events))) + (str code " must show a toast"))))) + +;; --------------------------------------------------------------------------- +;; wrap-wasm-effect +;; --------------------------------------------------------------------------- + +(defn- wasm-error + [code] + (ex-info "WASM Error" {:type :wasm-error :code code})) + +(defn- run-wasm-effect + "Runs `f` through `wrap-wasm-effect` with `on-error` captured. Returns the + effect result, or the thrown error under `:thrown`, and the errors handled." + [f] + (let [handled (atom [])] + (with-redefs [errors/on-error (mock/stub #(swap! handled conj %))] + (let [result (try + {:value (errors/wrap-wasm-effect f)} + (catch :default cause + {:thrown cause}))] + (assoc result :handled @handled))))) + +(t/deftest wrap-wasm-effect-returns-the-effect-value + (t/testing "an effect that does not throw keeps its return value" + (let [{:keys [value handled]} (run-wasm-effect (constantly :done))] + (t/is (= :done value)) + (t/is (empty? handled))))) + +(t/deftest wrap-wasm-effect-keeps-the-cleanup + (t/testing "a returned cleanup fn still runs the original cleanup" + (let [cleaned (atom false) + {:keys [value]} (run-wasm-effect (fn [] #(reset! cleaned true)))] + (t/is (fn? value)) + (value) + (t/is (true? @cleaned))))) + +(t/deftest wrap-wasm-effect-handles-recoverable-errors + (t/testing "a recoverable WASM error goes to on-error and is not rethrown" + (doseq [code errors/recoverable-wasm-codes] + (let [cause (wasm-error code) + {:keys [value thrown handled]} (run-wasm-effect #(throw cause))] + (t/is (nil? value) (str code)) + (t/is (nil? thrown) (str code " must not be rethrown")) + (t/is (= [cause] handled) (str code " must go to on-error")))))) + +(t/deftest wrap-wasm-effect-rethrows-other-errors + (t/testing "critical WASM errors and other errors are rethrown unchanged" + (doseq [cause [(wasm-error :panic) + (wasm-error :wasm-critical) + (ex-info "other" {:type :validation}) + (js/Error. "plain")]] + (let [{:keys [thrown handled]} (run-wasm-effect #(throw cause))] + (t/is (identical? cause thrown) (ex-message cause)) + (t/is (empty? handled) (str (ex-message cause) " must not go to on-error")))))) + +(t/deftest wrap-wasm-effect-guards-the-cleanup + (t/testing "a recoverable error in the cleanup goes to on-error" + (let [cause (wasm-error :non-blocking) + handled (atom []) + cleanup (:value (run-wasm-effect (fn [] #(throw cause))))] + (with-redefs [errors/on-error (mock/stub #(swap! handled conj %))] + (cleanup)) + (t/is (= [cause] @handled)))) + + (t/testing "a critical error in the cleanup is rethrown" + (let [cause (wasm-error :wasm-critical) + cleanup (:value (run-wasm-effect (fn [] #(throw cause))))] + (t/is (thrown? js/Error (cleanup)))))) diff --git a/frontend/test/frontend_tests/plugins/utils_test.cljs b/frontend/test/frontend_tests/plugins/utils_test.cljs index 7238732279..ac3096dd46 100644 --- a/frontend/test/frontend_tests/plugins/utils_test.cljs +++ b/frontend/test/frontend_tests/plugins/utils_test.cljs @@ -289,3 +289,9 @@ message (plugins.utils/error-messages (sm/explain schema {:value ""}))] (t/is (= "Field value is invalid: Empty input" message)))) +(t/deftest test-reject-not-valid-rejects-an-error + ;; Plugins read `.message` off the rejection, so it must be an Error. + (let [rejected (atom nil)] + (plugins.utils/reject-not-valid #(reset! rejected %) :someMethod "bad") + (t/is (instance? js/Error @rejected)) + (t/is (str/includes? (.-message @rejected) "Code: :someMethod")))) diff --git a/frontend/test/frontend_tests/plugins/wait_for_save_test.cljs b/frontend/test/frontend_tests/plugins/wait_for_save_test.cljs new file mode 100644 index 0000000000..c9951aa657 --- /dev/null +++ b/frontend/test/frontend_tests/plugins/wait_for_save_test.cljs @@ -0,0 +1,304 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns frontend-tests.plugins.wait-for-save-test + "Contract tests for app.plugins.persist/wait-for-save (plugin waitForSave)." + (:require + [app.main.data.persistence :as dps] + [app.main.refs :as refs] + [app.main.store :as st] + [app.plugins.api :as api] + [app.plugins.persist :as persist] + [beicon.v2.core :as rx] + [cljs.test :as t :include-macros true] + [frontend-tests.helpers.async :as async] + [frontend-tests.helpers.mock :as mock] + [frontend-tests.helpers.persistence :as thp :refer [local-commit with-persistence]] + [okulary.core :as l] + [potok.v2.core :as ptk])) + +(defn- with-save-mocks + "Redirects persistence refs and st/emit! onto the fixture store." + [store body] + (mock/with-mocks* + {refs/persistence (l/derived :persistence store) + st/state store + st/stream (ptk/input-stream store)} + (body))) + +(defn- await-outcome + "Resolves to {:ok value} or {:error err} for a JS promise." + [p] + (-> p + (.then (fn [v] {:ok v})) + (.catch (fn [e] {:error e})))) + +(defn- error-code + [outcome] + (some-> (:error outcome) .-code)) + +(defn- saves + "The `update-file` commands the transport received. Every flow of the + editor shares the mocked transport and a lingering one can still reach + it, so a stray call must neither pass for a save nor release a wait on + one." + [requests] + (filterv #(= :update-file (first %)) @requests)) + +(t/deftest ^:async empty-persistence-resolves-after-intake + (await + (with-persistence + (^:async fn [{:keys [requests store]}] + (await + (with-save-mocks store + (^:async fn [] + (let [outcome (await (await-outcome (persist/wait-for-save 2000)))] + (t/is (contains? outcome :ok) + "empty buffer and queue resolve after intake") + (t/is (empty? (saves requests)) + "nothing to save sends no update-file"))))))))) + +(t/deftest ^:async context-wait-for-save-resolves + ;; Goes through the plugin context proxy, the only way plugins reach it. + (await + (with-persistence + (^:async fn [{:keys [store]}] + (await + (with-save-mocks store + (^:async fn [] + (let [^js ctx (api/create-context "00000000-0000-0000-0000-000000000000") + outcome (await (await-outcome (.waitForSave ctx 2000)))] + (t/is (contains? outcome :ok) + "context waitForSave resolves with nothing to save"))))))))) + +(t/deftest ^:async invalid-timeout-rejects + (await + (with-persistence + (^:async fn [{:keys [store]}] + (await + (with-save-mocks store + (^:async fn [] + (doseq [value [-1 0 ##NaN ##Inf 2147483648]] + (let [outcome (await (await-outcome (persist/wait-for-save value)))] + (t/is (instance? js/Error (:error outcome)) + (str value " rejects with an Error")) + (t/is (re-find #"not valid.*waitForSave" + (or (some-> (:error outcome) .-message) "")))))))))))) + +(t/deftest ^:async terminal-error-rejects + (await + (with-persistence + (^:async fn [{:keys [store]}] + (await + (with-save-mocks store + (^:async fn [] + ;; Empty queue+buffer with :error is a terminal waiter release + ;; (see dps/terminal-status?). + (swap! store assoc-in [:persistence :status] :error) + (swap! store assoc-in [:persistence :error] + {:type :persistence :code :save-permission-denied}) + (swap! store assoc-in [:persistence :queue] #queue []) + (swap! store assoc-in [:persistence :buffer] []) + (let [outcome (await (await-outcome (persist/wait-for-save 2000)))] + (t/is (contains? outcome :error) + "terminal :error rejects waitForSave") + (t/is (re-find #"(?i)could not be saved.*reopen" + (or (some-> (:error outcome) .-message) ""))) + (t/is (= "save-permission-denied" (error-code outcome)) + "the stored failure code is kept"))))))))) + +(t/deftest ^:async retrying-error-rejects-without-reopen + (await + (with-persistence + (^:async fn [{:keys [store]}] + (await + (with-save-mocks store + (^:async fn [] + ;; A transport failure the slow retry cycle keeps resending + ;; (dps/save-retrying?) recovers by itself: no reopen advice. + (swap! store update :persistence assoc + :status :error + :error {:type :persistence :cause-type :network} + :failing-since (js/Date.now)) + (let [outcome (await (await-outcome (persist/wait-for-save 2000))) + message (or (some-> (:error outcome) .-message) "")] + (t/is (contains? outcome :error)) + (t/is (= "save-retrying" (error-code outcome))) + (t/is (re-find #"(?i)retrying" message)) + (t/is (not (re-find #"(?i)reopen" message))))))))))) + +(t/deftest ^:async save-failure-during-wait-rejects + (await + (with-persistence + (^:async fn [{:keys [file-id response requests store]}] + (await + (with-save-mocks store + (^:async fn [] + (ptk/emit! store (local-commit file-id)) + (await (async/wait-for #(seq (get-in @store [:persistence :buffer])) + "commit is buffered")) + (let [p (persist/wait-for-save 5000)] + (await (async/wait-for #(seq (saves requests)) + "force-persist flushed the buffer")) + (rx/error! response (ex-info "rejected" {:type :validation :code :invalid-data})) + (let [outcome (await (await-outcome p))] + (t/is (contains? outcome :error) + "a save that fails during the wait rejects") + (t/is (re-find #"(?i)could not be saved" + (or (some-> (:error outcome) .-message) ""))) + (t/is (= "invalid-data" (error-code outcome)) + "the backend failure code reaches err.code")))))))))) + +(t/deftest ^:async rolled-back-during-wait-rejects + (await + (with-persistence + (^:async fn [{:keys [file-id response store]}] + (await + (with-save-mocks store + (^:async fn [] + (ptk/emit! store (local-commit file-id)) + (await (async/wait-for #(seq (get-in @store [:persistence :buffer])) + "commit is buffered")) + (let [p (persist/wait-for-save 5000) + _ (js/setTimeout + (fn [] + (ptk/emit! store + (ptk/data-event ::dps/rolled-back + {:file-ids #{file-id} + :page-ids #{} + :shape-ids #{}}))) + 20) + outcome (await (await-outcome p))] + (t/is (contains? outcome :error) + "rollback during wait rejects even if status later looks saved") + (t/is (re-find #"(?i)roll" + (or (some-> (:error outcome) .-message) ""))) + (t/is (= "rolled-back" (error-code outcome))) + ;; Keep the save path from hanging the fixture + (when (seq (get-in @store [:persistence :queue])) + (rx/push! response {:revn 1})))))))))) + +(t/deftest ^:async buffer-race-waits-for-commit-not-stale-saved + (await + (with-persistence + (^:async fn [{:keys [file-id response requests store]}] + (await + (with-save-mocks store + (^:async fn [] + (t/is (empty? (get-in @store [:persistence :buffer]))) + (t/is (or (nil? (get-in @store [:persistence :status])) + (= :saved (get-in @store [:persistence :status])))) + ;; Emit after starting the wait so a naive immediate empty + ;; :saved resolve would succeed before the commit buffers. + (let [p (persist/wait-for-save 5000)] + (js/setTimeout + (fn [] (ptk/emit! store (local-commit file-id))) + 0) + (await (async/wait-for #(seq (saves requests)) + "force-persist sent the edit")) + (rx/push! response {:revn 1}) + (let [outcome (await (await-outcome p))] + (t/is (contains? outcome :ok) + "wait covers the late-buffered commit instead of stale empty :saved")))))))))) + +(t/deftest ^:async pending-work-emits-force-persist + (await + (with-persistence + (^:async fn [{:keys [file-id response requests store]}] + (await + (with-save-mocks store + (^:async fn [] + (ptk/emit! store (local-commit file-id)) + (await (async/wait-for #(seq (get-in @store [:persistence :buffer])) + "commit is buffered")) + (let [p (persist/wait-for-save 5000)] + (await (async/wait-for #(seq (saves requests)) + "force-persist flushed the buffer")) + (rx/push! response {:revn 1}) + (let [outcome (await (await-outcome p))] + (t/is (contains? outcome :ok) + "pending work resolves once the save lands")))))))))) + +(t/deftest ^:async pending-work-short-timeout-rejects + (await + (with-persistence + (^:async fn [{:keys [file-id store]}] + (await + (with-save-mocks store + (^:async fn [] + (ptk/emit! store (local-commit file-id)) + (await (async/wait-for #(seq (get-in @store [:persistence :buffer])) + "commit is buffered")) + ;; Never answer the save; the wait must time out. + (let [outcome (await (await-outcome (persist/wait-for-save 30)))] + (t/is (contains? outcome :error) + "pending work with no save answer rejects on timeout") + (t/is (re-find #"(?i)timed out|waiting" + (or (some-> (:error outcome) .-message) ""))) + (t/is (= "save-timeout" (error-code outcome))))))))))) + +(t/deftest ^:async empty-short-timeout-resolves + (await + (with-persistence + (^:async fn [{:keys [store]}] + (await + (with-save-mocks store + (^:async fn [] + ;; Timeout shorter than the default 100ms intake; must still + ;; resolve (capped intake), not reject as a timeout. + (let [outcome (await (await-outcome (persist/wait-for-save 5)))] + (t/is (contains? outcome :ok) + "empty short timeout resolves (nothing to save)"))))))))) + +(t/deftest ^:async soft-rejection-during-wait-rejects + (await + (with-persistence + (^:async fn [{:keys [file-id response requests store]}] + (await + (with-save-mocks store + (^:async fn [] + (let [commit (local-commit file-id)] + (ptk/emit! store commit) + (await (async/wait-for #(seq (get-in @store [:persistence :buffer])) + "commit is buffered")) + (let [p (persist/wait-for-save 5000)] + (await (async/wait-for #(seq (saves requests)) + "force-persist flushed the buffer")) + (rx/push! response (thp/rejection (:id @commit))) + (let [outcome (await (await-outcome p))] + (t/is (contains? outcome :error) + "soft rejection rejects wait-for-save, not success on later :saved") + (t/is (re-find #"(?i)roll" + (or (some-> (:error outcome) .-message) ""))) + (t/is (= "rolled-back" (error-code outcome))))))))))))) + +(t/deftest ^:async rollback-rejected-beats-empty-saved + (await + (with-persistence + (^:async fn [{:keys [file-id response requests store]}] + (await + (with-save-mocks store + (^:async fn [] + (ptk/emit! store (local-commit file-id)) + (await (async/wait-for #(seq (get-in @store [:persistence :buffer])) + "commit is buffered")) + (let [p (persist/wait-for-save 5000)] + (await (async/wait-for #(seq (saves requests)) + "force-persist flushed the buffer")) + ;; Rollback starts before settle can publish empty :saved; + ;; never emit ::rolled-back (reload-delay case). + (ptk/emit! store (ptk/data-event ::dps/rollback-rejected {})) + (swap! store assoc-in [:persistence :status] :saved) + (swap! store assoc-in [:persistence :buffer] []) + (swap! store assoc-in [:persistence :queue] #queue []) + (let [outcome (await (await-outcome p))] + (t/is (contains? outcome :error) + "rollback-rejected rejects even when :saved would win the race") + (t/is (re-find #"(?i)roll" + (or (some-> (:error outcome) .-message) ""))) + (t/is (= "rolled-back" (error-code outcome))) + (when (seq (get-in @store [:persistence :queue])) + (rx/push! response {:revn 1}))))))))))) diff --git a/frontend/test/frontend_tests/runner.cljs b/frontend/test/frontend_tests/runner.cljs index 5ff1e7db87..30a0e2c7da 100644 --- a/frontend/test/frontend_tests/runner.cljs +++ b/frontend/test/frontend_tests/runner.cljs @@ -119,6 +119,7 @@ [frontend-tests.plugins.user-test] [frontend-tests.plugins.utils-test] [frontend-tests.plugins.value-objects-test] + [frontend-tests.plugins.wait-for-save-test] [frontend-tests.render-dimensions-test] [frontend-tests.render-wasm.call-test] [frontend-tests.render-wasm.modifiers-test] @@ -136,6 +137,7 @@ [frontend-tests.text-editor-paste-guard-test] [frontend-tests.tokens.context-menu-test] [frontend-tests.tokens.copy-paste-props-test] + [frontend-tests.tokens.export-modal-test] [frontend-tests.tokens.import-export-test] [frontend-tests.tokens.logic.token-actions-test] [frontend-tests.tokens.logic.token-data-test] @@ -196,6 +198,7 @@ [frontend-tests.ui.subscription-test] [frontend-tests.ui.team-container-test] [frontend-tests.ui.text-attrs-multiple-test] + [frontend-tests.ui.viewport-texts-html-test] [frontend-tests.ui.viewport-top-bar-test] [frontend-tests.ui.workspace-libraries-test] [frontend-tests.util-clipboard-test] @@ -352,6 +355,7 @@ 'frontend-tests.plugins.user-test 'frontend-tests.plugins.utils-test 'frontend-tests.plugins.value-objects-test + 'frontend-tests.plugins.wait-for-save-test 'frontend-tests.render-wasm.call-test 'frontend-tests.render-wasm.modifiers-test 'frontend-tests.render-wasm.process-objects-test @@ -368,6 +372,7 @@ 'frontend-tests.svg-filters-test 'frontend-tests.tokens.copy-paste-props-test 'frontend-tests.tokens.context-menu-test + 'frontend-tests.tokens.export-modal-test 'frontend-tests.tokens.import-export-test 'frontend-tests.tokens.logic.token-actions-test 'frontend-tests.tokens.logic.token-data-test @@ -430,6 +435,7 @@ 'frontend-tests.ui.subscription-test 'frontend-tests.ui.team-container-test 'frontend-tests.ui.text-attrs-multiple-test + 'frontend-tests.ui.viewport-texts-html-test 'frontend-tests.ui.viewport-top-bar-test 'frontend-tests.ui.workspace-libraries-test 'frontend-tests.util-clipboard-test diff --git a/frontend/test/frontend_tests/tokens/export_modal_test.cljs b/frontend/test/frontend_tests/tokens/export_modal_test.cljs new file mode 100644 index 0000000000..834ea88736 --- /dev/null +++ b/frontend/test/frontend_tests/tokens/export_modal_test.cljs @@ -0,0 +1,65 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns frontend-tests.tokens.export-modal-test + "The active themes and sets the export writes into the files." + (:require + [app.common.files.tokens :as cfo] + [app.common.json :as json] + [app.common.types.tokens-lib :as ctob] + [app.main.ui.workspace.tokens.export.modal :as export-modal] + [cljs.test :as t :include-macros true])) + +(def ^:private single-file-json @#'export-modal/single-file-json) +(def ^:private multi-file-entries @#'export-modal/multi-file-entries) + +(defn- imported-lib + "A library imported from a file with the themes `mode/light` and + `mode/dark`, where the file marked `mode/dark` as active." + [] + (ctob/parse-multi-set-dtcg-json + {"mode/light" {"surface" {"$value" "#ffffff" "$type" "color"}} + "mode/dark" {"surface" {"$value" "#111111" "$type" "color"}} + "$themes" [{"name" "light" "group" "mode" "selectedTokenSets" {"mode/light" "enabled"}} + {"name" "dark" "group" "mode" "selectedTokenSets" {"mode/dark" "enabled"}}] + "$metadata" {"tokenSetOrder" ["mode/light" "mode/dark"] + "activeThemes" ["mode/dark"] + "activeSets" ["mode/dark"]}})) + +(defn- status-with-light-active + "The file's status after the user picked `mode/light` in the theme + selector." + [lib] + (let [light (ctob/get-theme-by-name lib "mode" "light")] + (cfo/activate-theme (cfo/make-tokens-status-from-lib lib) lib (ctob/get-id light)))) + +(defn- active-in + [metadata] + {:themes (set (get metadata "activeThemes")) + :sets (set (get metadata "activeSets"))}) + +(t/deftest export-writes-the-themes-active-in-the-file + (let [lib (imported-lib) + status (status-with-light-active lib)] + (t/testing "the library still holds the themes active in the imported file" + (t/is (contains? (ctob/get-legacy-active-themes lib) "mode/dark"))) + + (t/testing "single file" + (let [exported (json/decode (single-file-json lib status) :key-fn identity)] + (t/is (= {:themes #{"mode/light"} :sets #{"mode/light"}} + (active-in (get exported "$metadata")))))) + + (t/testing "multiple files" + (t/is (= {:themes #{"mode/light"} :sets #{"mode/light"}} + (active-in (get (multi-file-entries lib status) "$metadata.json"))))))) + +(t/deftest export-without-tokens + (t/testing "an empty library has nothing to export" + (let [lib (ctob/make-tokens-lib)] + (t/is (nil? (single-file-json lib (cfo/make-tokens-status-from-lib lib)))))) + (t/testing "without a status there is nothing to export" + (t/is (nil? (single-file-json (imported-lib) nil))) + (t/is (nil? (multi-file-entries (imported-lib) nil))))) diff --git a/frontend/test/frontend_tests/tokens/import_export_test.cljs b/frontend/test/frontend_tests/tokens/import_export_test.cljs index 315cd1c851..a38110d19f 100644 --- a/frontend/test/frontend_tests/tokens/import_export_test.cljs +++ b/frontend/test/frontend_tests/tokens/import_export_test.cljs @@ -112,7 +112,7 @@ color.value tries to reference missing, which is not defined."))) (fn [_] (throw (js/Error. "Should be an error"))) (fn [err] - (t/is (= :error.import/invalid-json-data (:error/code (ex-data err)))) + (t/is (= :error.import/no-token-files-found (:error/code (ex-data err)))) (done))))))) (t/deftest import-invalid-json-stream-test @@ -138,7 +138,7 @@ color.value tries to reference missing, which is not defined."))) (fn [] (throw (js/Error. "Should be an error"))) (fn [err] - (t/is (= :error.import/invalid-json-data (:error/code (ex-data err)))) + (t/is (= :error.import/no-token-files-found (:error/code (ex-data err)))) (done))))))) (t/deftest import-missing-references-json-test diff --git a/frontend/test/frontend_tests/ui/file_restored_modal_test.cljs b/frontend/test/frontend_tests/ui/file_restored_modal_test.cljs index 169baa4573..983f88ab39 100644 --- a/frontend/test/frontend_tests/ui/file_restored_modal_test.cljs +++ b/frontend/test/frontend_tests/ui/file_restored_modal_test.cljs @@ -6,10 +6,14 @@ (ns frontend-tests.ui.file-restored-modal-test (:require + ["react-dom/server" :as rds] [app.common.time :as ct] + [app.main.data.modal :as modal] [app.main.ui.workspace.file-restored-modal :as frm] [app.util.i18n :as i18n] - [cljs.test :as t :include-macros true])) + [cljs.test :as t :include-macros true] + [cuerdas.core :as str] + [rumext.v2 :as mf])) (defn- message "The translation key and arguments of the restore message." @@ -32,3 +36,17 @@ (message "Ada" nil nil))) (t/is (= ["workspace.file-restored.message-anonymous"] (message nil nil nil))))) + +(defn- render-modal + "Markup of the registered `:file-restored` modal for the `props` map, + rendered the way the modal wrapper does it (`mf/element` with the map)." + [props] + (with-redefs [i18n/tr (fn ([key] key) ([key & args] (str/join "|" (into [key] args))))] + (rds/renderToStaticMarkup + (mf/element (get @modal/components :file-restored) props)))) + +(t/deftest the-modal-shows-the-date-of-an-unnamed-version + (let [created-at (ct/now) + markup (render-modal {:fullname "Ada" :date created-at :unsaved false})] + (t/is (str/includes? markup (str "workspace.file-restored.message-date|Ada|" + (ct/format-inst created-at "PPp")))))) diff --git a/frontend/test/frontend_tests/ui/viewport_texts_html_test.cljs b/frontend/test/frontend_tests/ui/viewport_texts_html_test.cljs new file mode 100644 index 0000000000..d6369e5882 --- /dev/null +++ b/frontend/test/frontend_tests/ui/viewport_texts_html_test.cljs @@ -0,0 +1,71 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns frontend-tests.ui.viewport-texts-html-test + "How the SVG renderer measures text shapes and commits the result." + (:require + [app.common.uuid :as uuid] + [app.main.data.workspace.texts :as dwt] + [app.main.store :as st] + [app.main.ui.workspace.shapes.text.viewport-texts-html :as vth] + [app.util.text-svg-position :as tsp] + [cljs.test :as t :include-macros true] + [frontend-tests.helpers.async :as async] + [frontend-tests.helpers.mock :as mock] + [potok.v2.core :as ptk] + [promesa.core :as p])) + +(def ^:private fixed-text + {:id (uuid/next) :type :text :grow-type :fixed}) + +(defn- measure-text + "Measure `shape` as the SVG renderer does and return the measurement + promise. Kept out of the `^:async` test bodies: there, a call through the + var compiles to an awaited wrapper that would wait for the measurement." + [shape] + (#'vth/update-text-shape shape nil)) + +(defn- with-measurement + "Run `f` with the text measurement mocked to `measurement`, a promise the + test settles. Events emitted to the store are recorded in `emitted`." + [measurement emitted f] + (mock/with-mocks* + {tsp/calc-position-data (mock/stub (constantly measurement)) + st/emit! (mock/stub (fn [& events] (swap! emitted into events) nil))} + (await (f)))) + +(t/deftest ^:async measurement-promise-settles-with-the-measurement + (let [measurement (p/deferred) + emitted (atom []) + settled? (atom false)] + (await + (with-measurement + measurement + emitted + (^:async fn [] + (let [result (p/finally (measure-text fixed-text) + #(reset! settled? true))] + (await (async/settle)) + (t/is (false? @settled?) + "the promise stays pending while the text is measured") + (p/resolve measurement []) + (await result) + (t/is (true? @settled?) + "the promise settles once the measurement settles") + (t/is (some #(= ::dwt/clean-text-modifier (ptk/type %)) @emitted) + "the measurement completes and cleans the text modifier"))))))) + +(t/deftest ^:async failed-measurement-resolves-to-nil + (let [measurement (p/deferred)] + (await + (with-measurement + measurement + (atom []) + (^:async fn [] + (let [result (measure-text fixed-text)] + (p/reject measurement (ex-info "measure failed" {})) + (t/is (nil? (await result)) + "a failed measurement resolves to nil"))))))) diff --git a/frontend/test/frontend_tests/worker/exports_test.cljs b/frontend/test/frontend_tests/worker/exports_test.cljs index 5f1a9f6953..02f0f42f60 100644 --- a/frontend/test/frontend_tests/worker/exports_test.cljs +++ b/frontend/test/frontend_tests/worker/exports_test.cljs @@ -7,6 +7,7 @@ (ns frontend-tests.worker.exports-test (:require [app.common.uuid :as uuid] + [app.worker.exports :as wexports] [app.worker.wasm :as wwasm] [beicon.v2.core :as rx] [cljs.test :as t :include-macros true] @@ -48,6 +49,14 @@ @order) "job B starts only after job A has finished"))) +(t/deftest single-export-filename-does-not-repeat-suffix + (t/is (= "Ellipseabc.svg" + (wexports/local-export-filename + :single {:name "Ellipseabc" :suffix "abc" :type :svg} nil))) + (t/is (= "Ellipse.png" + (wexports/local-export-filename + :single {:name "Ellipse" :suffix nil :type :png} nil)))) + (t/deftest cancel-job-is-visible-to-cancelled (let [job-id (uuid/next)] (t/is (false? (wwasm/cancelled? job-id))) diff --git a/library/shadow-cljs.edn b/library/shadow-cljs.edn index 772c09c854..a4516d2efb 100644 --- a/library/shadow-cljs.edn +++ b/library/shadow-cljs.edn @@ -35,7 +35,8 @@ :warnings {:fn-deprecated false}} :release - {:compiler-options + {:closure-defines {app.common.data.macros/lazy-delay? true} + :compiler-options {:fn-invoke-direct true :optimizations #shadow/env ["PENPOT_BUILD_OPTIMIZATIONS" :as :keyword :default :advanced] :pretty-print false diff --git a/mcp/README.md b/mcp/README.md index 2f1636d24a..095fe64214 100644 --- a/mcp/README.md +++ b/mcp/README.md @@ -295,9 +295,10 @@ The Penpot MCP server can be configured using environment variables. | `PENPOT_MCP_REPL_PORT` | Port for the REPL server (development/debugging) | `4403` | | `PENPOT_MCP_REPL_HOST` | Address on which the REPL server listens (binds to) | `localhost` | | `PENPOT_MCP_REPL_ENABLE` | Explicitly enable/disable the REPL server. Set to `true` to enable. When unset, defaults to the value of `PENPOT_MCP_DEVENV`. The REPL server never starts in multi-user mode. | (unset) | -| `PENPOT_MCP_REMOTE_MODE` | Enable remote mode (disables file system access). Set to `true` to enable. | `false` | +| `PENPOT_MCP_REMOTE_MODE` | Enable remote mode (disables file system access). Set to `true` to enable. In single-user mode it does not open the server to other hosts; see `PENPOT_MCP_ALLOWED_HOSTS`. | `false` | +| `PENPOT_MCP_ALLOWED_HOSTS` | Comma-separated hostnames (no ports; IPv6 in brackets) that requests to the MCP endpoint, the plugin WebSocket and the REPL server may carry in their `Host` and `Origin` headers. Requests with any other hostname are rejected, which stops web pages from driving the server. Entries with a scheme, port or path stop the server from starting. When unset, single-user mode allows only `localhost`, `127.0.0.1` and `[::1]`, with or without remote mode, and multi-user mode allows any hostname. Set it when clients, the plugin or a proxy reach a single-user server under another name. Set it to `*` to turn the check off (not recommended in single-user mode: any web page you open can then drive the server). | (unset) | | `PENPOT_MCP_DEVENV` | Enable Penpot development environment tools in local single-user mode. Set to `true` to enable. | `false` | -| `PENPOT_MCP_TOOL_TIMEOUT_S` | Timeout, in seconds, for tool calls dispatched to the Penpot plugin | `120` | +| `PENPOT_MCP_TOOL_TIMEOUT_S` | Timeout, in seconds, for tool calls dispatched to the Penpot plugin. `execute_code` waits up to 30s for layout and 60s for the save, so keep it above 90s plus the code's run time | `120` | | `PENPOT_MCP_MAX_BODY_SIZE` | Maximum size of a JSON request body accepted by the MCP endpoint, in any format supported by Express (e.g. `10mb`) | `10mb` | | `PENPOT_MCP_EXPORT_SHAPE_MAX_PARALLEL_REQUESTS` | Maximum number of parallel export shape requests (multi-user mode only). | `0` (no limit) | | `PENPOT_MCP_AUTH_URI` | Public base URL of the MCP authorization service, e.g. `https://penpot.example/api/mcp/oauth`. Set to enable OAuth; requires `--multi-user`. Use HTTPS (HTTP is allowed on loopback for development), without credentials, query parameters, or a fragment. | (unset; OAuth disabled) | diff --git a/mcp/packages/plugin/src/ExecuteCodeTaskHandler.test.ts b/mcp/packages/plugin/src/ExecuteCodeTaskHandler.test.ts new file mode 100644 index 0000000000..cae3eaec25 --- /dev/null +++ b/mcp/packages/plugin/src/ExecuteCodeTaskHandler.test.ts @@ -0,0 +1,193 @@ +import assert from "node:assert/strict"; +import { readFileSync } from "node:fs"; +import test from "node:test"; +import { runInNewContext } from "node:vm"; +import ts from "typescript"; +import * as PenpotUtils from "./PenpotUtils.ts"; +import { ExecuteCodeTaskConsole } from "./ExecuteCodeTaskConsole.ts"; + +type TaskResponse = { + type: string; + response: { + id: string; + success: boolean; + data?: unknown; + error?: string; + }; +}; + +/** Compiles a TypeScript source file for a simulated plugin context. */ +function compile(name: string) { + return ts.transpileModule(readFileSync(new URL(name, import.meta.url), "utf8"), { + compilerOptions: { module: ts.ModuleKind.CommonJS, target: ts.ScriptTarget.ES2020 }, + }).outputText; +} + +/** + * Loads Task + ExecuteCodeTaskHandler with `penpot` / `penpotMgmt` resolved from + * the outer globalThis (so installPenpot can configure them before construction). + */ +function loadHandler(): { ExecuteCodeTaskHandler: any; Task: any } { + const taskExports: Record = {}; + const handlerExports: Record = {}; + + const context: Record = { + console, + exports: taskExports, + require: (id: string) => { + if (id === "../TaskHandler" || id.endsWith("/TaskHandler")) { + return taskExports; + } + if (id.includes("common/src")) { + return {}; + } + if (id.includes("PenpotUtils")) { + return PenpotUtils; + } + if (id.includes("ExecuteCodeTaskConsole")) { + return { ExecuteCodeTaskConsole }; + } + throw new Error(`Unexpected import: ${id}`); + }, + }; + Object.defineProperty(context, "penpot", { + enumerable: true, + get: () => (globalThis as any).penpot, + }); + Object.defineProperty(context, "penpotMgmt", { + enumerable: true, + get: () => (globalThis as any).penpotMgmt, + }); + + runInNewContext(compile("TaskHandler.ts"), context); + + context.exports = handlerExports; + runInNewContext(compile("task-handlers/ExecuteCodeTaskHandler.ts"), context); + + return { + ExecuteCodeTaskHandler: handlerExports.ExecuteCodeTaskHandler, + Task: taskExports.Task, + }; +} + +function installPenpot(options: { waitForSave?: (() => Promise) | null; workspaceStatus?: string }): { + messages: TaskResponse[]; +} { + const messages: TaskResponse[] = []; + const penpot: Record = { + flags: { + naturalChildOrdering: false, + throwValidationErrors: false, + }, + waitForLayoutUpdate: async () => undefined, + ui: { + sendMessage: (message: TaskResponse) => { + messages.push(message); + }, + }, + }; + if (options.waitForSave !== null && options.waitForSave !== undefined) { + penpot.waitForSave = options.waitForSave; + } + (globalThis as any).penpot = penpot; + (globalThis as any).penpotMgmt = + options.workspaceStatus === undefined ? undefined : { workspace: { status: options.workspaceStatus } }; + return { messages }; +} + +function lastResponse(messages: TaskResponse[]) { + assert.ok(messages.length > 0, "expected a task response"); + return messages[messages.length - 1].response; +} + +test("awaits waitForSave before reporting success", async () => { + let resolveSave!: () => void; + let waitForSaveCalls = 0; + const saveGate = new Promise((resolve) => { + resolveSave = resolve; + }); + const { messages } = installPenpot({ + waitForSave: async () => { + waitForSaveCalls += 1; + return saveGate; + }, + }); + + const { ExecuteCodeTaskHandler, Task } = loadHandler(); + const handler = new ExecuteCodeTaskHandler(); + const task = new Task("req-1", "executeCode", { code: "return 42;" }); + const done = handler.handle(task); + + await new Promise((r) => setTimeout(r, 20)); + assert.equal(messages.length, 0, "must not succeed before save settles"); + assert.equal(waitForSaveCalls, 1, "must call waitForSave after the code runs"); + + resolveSave(); + await done; + + const response = lastResponse(messages); + assert.equal(response.success, true); + assert.equal((response.data as any).result, 42); +}); + +test("reports an error when waitForSave rejects", async () => { + const { messages } = installPenpot({ + waitForSave: async () => { + throw new Error("Changes could not be saved"); + }, + }); + + const { ExecuteCodeTaskHandler, Task } = loadHandler(); + const handler = new ExecuteCodeTaskHandler(); + const task = new Task("req-2", "executeCode", { code: "return 1;" }); + await assert.rejects(handler.handle(task), /could not be saved/i); + assert.equal(messages.length, 0, "must not report success"); +}); + +test("hard-fails when waitForSave is missing", async () => { + const { messages } = installPenpot({ waitForSave: null }); + + const { ExecuteCodeTaskHandler, Task } = loadHandler(); + const handler = new ExecuteCodeTaskHandler(); + const task = new Task("req-3", "executeCode", { code: "return 1;" }); + await assert.rejects(handler.handle(task), /waitForSave/); + assert.equal(messages.length, 0, "must not report success"); +}); + +for (const workspaceStatus of ["none", "loading"]) { + test(`skips waitForSave when the workspace is ${workspaceStatus}`, async () => { + let waitForSaveCalls = 0; + const { messages } = installPenpot({ + workspaceStatus, + waitForSave: async () => { + waitForSaveCalls += 1; + throw new Error("Changes could not be saved"); + }, + }); + + const { ExecuteCodeTaskHandler, Task } = loadHandler(); + const handler = new ExecuteCodeTaskHandler(); + const task = new Task("req-4", "executeCode", { code: "return 7;" }); + await handler.handle(task); + + assert.equal(waitForSaveCalls, 0, "no workspace edit can be pending"); + const response = lastResponse(messages); + assert.equal(response.success, true); + assert.equal((response.data as any).result, 7); + }); +} + +test("awaits waitForSave when the workspace is ready", async () => { + const { messages } = installPenpot({ + workspaceStatus: "ready", + waitForSave: async () => { + throw new Error("Changes could not be saved"); + }, + }); + + const { ExecuteCodeTaskHandler, Task } = loadHandler(); + const handler = new ExecuteCodeTaskHandler(); + const task = new Task("req-5", "executeCode", { code: "return 1;" }); + await assert.rejects(handler.handle(task), /could not be saved/i); + assert.equal(messages.length, 0, "must not report success"); +}); diff --git a/mcp/packages/plugin/src/ReconnectPolicy.ts b/mcp/packages/plugin/src/ReconnectPolicy.ts index 9d2947894d..5767977906 100644 --- a/mcp/packages/plugin/src/ReconnectPolicy.ts +++ b/mcp/packages/plugin/src/ReconnectPolicy.ts @@ -1,8 +1,9 @@ /** * WebSocket close code the MCP server uses for policy violations. * - * The server rejects a second plugin connection for the same user token, and - * connections missing a token in multi-user mode, with this code. + * The server rejects a second plugin connection for the same user token, + * connections missing a token in multi-user mode, and connections whose `Host` + * or `Origin` is not allowed, with this code. */ export const WS_CLOSE_POLICY_VIOLATION = 1008; diff --git a/mcp/packages/plugin/src/task-handlers/ExecuteCodeTaskHandler.ts b/mcp/packages/plugin/src/task-handlers/ExecuteCodeTaskHandler.ts index 43bfd37e35..1e2fd5a7c8 100644 --- a/mcp/packages/plugin/src/task-handlers/ExecuteCodeTaskHandler.ts +++ b/mcp/packages/plugin/src/task-handlers/ExecuteCodeTaskHandler.ts @@ -44,28 +44,34 @@ export class ExecuteCodeTaskHandler extends TaskHandler { // set the flags naturalChildOrdering and throwValidationErrors to true during code execution. let originalNaturalChildOrdering: any, originalThrowValidationErrors: any; - if (penpot.flags) { - originalNaturalChildOrdering = penpot.flags.naturalChildOrdering; - penpot.flags.naturalChildOrdering = true; - originalThrowValidationErrors = penpot.flags.throwValidationErrors; - penpot.flags.throwValidationErrors = true; - } else { + if (!penpot.flags || typeof penpot.waitForSave !== "function") { throw new Error( - "You are using a version of the Penpot MCP server which is incompatible " + - "with the connected Penpot version. " + + "The connected Penpot build is missing Plugin API support that this MCP " + + "plugin needs (for example `penpot.waitForSave`), or the MCP plugin is " + + "too new for that Penpot. Upgrade Penpot (and its plugins-runtime) so " + + "they match this MCP version. " + "Read the documentation for instructions on which version to use: " + "https://github.com/penpot/penpot/tree/develop/mcp\n" + "If you are an LLM, tell the user about this!" ); } + originalNaturalChildOrdering = penpot.flags.naturalChildOrdering; + penpot.flags.naturalChildOrdering = true; + originalThrowValidationErrors = penpot.flags.throwValidationErrors; + penpot.flags.throwValidationErrors = true; + + // Layout and save waits only apply with a ready workspace: on the dashboard (or while a file + // loads) no edit of this call can be pending. Checked at each wait, since the code may navigate. + const management = this.context.penpotMgmt; + const workspaceReady = () => !management || management.workspace.status === "ready"; + let result: any; try { // wait for layout updates prior to executing the supplied code (if method is available) try { - const management = this.context.penpotMgmt; // @ts-ignore - TODO Penpot.waitForLayoutUpdate is not yet in the released types - if ((!management || management.workspace.status === "ready") && penpot.waitForLayoutUpdate) { + if (workspaceReady() && penpot.waitForLayoutUpdate) { // @ts-ignore await penpot.waitForLayoutUpdate(); } @@ -93,6 +99,11 @@ export class ExecuteCodeTaskHandler extends TaskHandler { result = ExecuteCodeTaskHandler.encodeBytesAsBase64Envelope(result); } + // Wait for the save outside any swallow catch: a failed or rolled-back save must fail the task. + if (workspaceReady()) { + await penpot.waitForSave(); + } + // return result and captured log let resultData: ExecuteCodeTaskResultData = { result: result, diff --git a/mcp/packages/server/data/api_types.yml b/mcp/packages/server/data/api_types.yml index ecb78c628d..d86d6e1d94 100644 --- a/mcp/packages/server/data/api_types.yml +++ b/mcp/packages/server/data/api_types.yml @@ -87,6 +87,7 @@ Penpot: flatten(shapes: Shape[]): Path[]; createVariantFromComponents(shapes: Board[]): VariantContainer; waitForLayoutUpdate(timeout?: number): Promise; + waitForSave(timeout?: number): Promise; } ``` @@ -354,6 +355,7 @@ Penpot: * themechange: event emitted when the user changes its theme. The callback will receive the new theme (currently: either `dark` or `light`) * filechange: event emitted when a different file is opened. The callback will receive the new file. * contentsave: event emitted after the file content is saved in the backend. The callback receives no arguments. + * contentrollback: event emitted after the backend rejected some changes and they were undone in the workspace. The callback receives a `ContentRollback` with the files, pages and shapes affected. * finish: event emitted when the current file is closed. The callback will receive the id of the closed file. Type Parameters @@ -1041,6 +1043,33 @@ Penpot: The promise to be resolved when the layout is updated. It is rejected with an Error, both on timeout and on an invalid timeout value. + waitForSave: |- + ``` + waitForSave(timeout?: number): Promise + ``` + + This method returns a promise that will be resolved when pending file + changes have been saved to the backend, or when there is nothing to save. + It force-flushes the persistence buffer first when work is pending. + + Parameters + + * timeout: number + + Maximum time to wait, in milliseconds. If the timeout + elapses before the save settles, the promise is rejected. Defaults to + 60000; the promise never waits indefinitely. + + Returns Promise + + The promise to be resolved when the save has settled. It is + rejected with an Error on save failure, when changes are rolled back + during the wait, on timeout, or on an invalid timeout value. A save + that failed before the call also rejects: with `save-retrying` while + Penpot keeps retrying it, otherwise with the failure code until the + file is reopened. The Error `code` property tells the cases apart: + `save-timeout`, `rolled-back`, `save-retrying`, or the failure code + (`save-failed` unless the backend gave a more specific one). ActiveUser: overview: |- Interface ActiveUser @@ -5365,7 +5394,8 @@ Team: Example ``` - const [team] = await penpotMgmt.listTeams();const projects = await team.listProjects(); + const [team] = await penpotMgmt.listTeams(); + const projects = await team.listProjects(); ``` listDeletedProjects: |- ``` @@ -5519,7 +5549,8 @@ Project: Example ``` - const file = await project.createFile({ name: 'Login' });await file.open(); + const file = await project.createFile({ name: 'Login' }); + await file.open(); ``` duplicate: |- ``` @@ -5585,7 +5616,7 @@ ProjectFile: } ``` - Referenced by: PenpotMgmt, Project, ProjectFile + Referenced by: PenpotMgmt, Project, ProjectFile, Team members: Properties: id: |- @@ -5814,7 +5845,8 @@ PenpotMgmt: Example ``` - const file = await penpotMgmt.getFile(penpotMgmt.workspace.fileId!);file.name = 'Checkout'; + const file = await penpotMgmt.getFile(penpotMgmt.workspace.fileId!); + file.name = 'Checkout'; ``` openFile: |- ``` @@ -5940,6 +5972,7 @@ Context: flatten(shapes: Shape[]): Path[]; createVariantFromComponents(shapes: Board[]): VariantContainer; waitForLayoutUpdate(timeout?: number): Promise; + waitForSave(timeout?: number): Promise; } ``` members: @@ -6793,6 +6826,33 @@ Context: The promise to be resolved when the layout is updated. It is rejected with an Error, both on timeout and on an invalid timeout value. + waitForSave: |- + ``` + waitForSave(timeout?: number): Promise + ``` + + This method returns a promise that will be resolved when pending file + changes have been saved to the backend, or when there is nothing to save. + It force-flushes the persistence buffer first when work is pending. + + Parameters + + * timeout: number + + Maximum time to wait, in milliseconds. If the timeout + elapses before the save settles, the promise is rejected. Defaults to + 60000; the promise never waits indefinitely. + + Returns Promise + + The promise to be resolved when the save has settled. It is + rejected with an Error on save failure, when changes are rolled back + during the wait, on timeout, or on an invalid timeout value. A save + that failed before the call also rejects: with `save-retrying` while + Penpot keeps retrying it, otherwise with the failure code until the + file is reopened. The Error `code` property tells the cases apart: + `save-timeout`, `rolled-back`, `save-retrying`, or the failure code + (`save-failed` unless the backend gave a more specific one). ContextGeometryUtils: overview: |- Interface ContextGeometryUtils @@ -8162,6 +8222,7 @@ EventsMap: finish: string; shapechange: Shape; contentsave: void; + contentrollback: ContentRollback; } ``` @@ -8222,6 +8283,56 @@ EventsMap: ``` The `contentsave` event will trigger after the file content is saved in the backend. + contentrollback: |- + ``` + readonly contentrollback: ContentRollback + ``` + + The `contentrollback` event will trigger after the backend rejected some changes + and the workspace undid them, together with the rest of their undo steps and any + later change to the same file. Changes made by a plugin can be undone this way + after the call that made them returned, so a plugin that relies on them should + read the affected shapes again. The event fires once the undone data is in the + workspace. When the workspace could not undo the changes in place, it reloads the + file from the server first; the whole file then holds the server data, so shapes + outside the listed ones may have changed too. +ContentRollback: + overview: |- + Interface ContentRollback + ========================= + + What a rollback of rejected changes undid, as given to the `contentrollback` event. + + ``` + interface ContentRollback { + fileIds: string[]; + pageIds: string[]; + shapeIds: string[]; + } + ``` + + Referenced by: EventsMap + members: + Properties: + fileIds: |- + ``` + readonly fileIds: string[] + ``` + + Ids of the files that lost changes. + pageIds: |- + ``` + readonly pageIds: string[] + ``` + + Ids of the pages the undone changes touched. + shapeIds: |- + ``` + readonly shapeIds: string[] + ``` + + Ids of the page shapes the undone changes touched. A shape the undone changes + created no longer exists; one they removed exists again. Export: overview: |- Interface Export diff --git a/mcp/packages/server/data/initial_instructions.md b/mcp/packages/server/data/initial_instructions.md index d126811742..dc49df83e7 100644 --- a/mcp/packages/server/data/initial_instructions.md +++ b/mcp/packages/server/data/initial_instructions.md @@ -25,13 +25,21 @@ before trying to observe changes. To wait only for a single shape (and its children), use `await shape.waitForLayoutUpdate();`. Both accept an optional timeout in milliseconds (default 30000) and reject when it elapses. Every `execute_code` call waits for updates before applying your code, so you never need to call `waitForLayoutUpdate` at the beginning of your code. +Every `execute_code` call also waits for pending changes to be saved before it returns success, so you should not call +`await penpot.waitForSave()` at the end of your code (that would wait twice). +That automatic save wait defaults to 60s and runs after the automatic layout wait (default 30s); both, plus your code, +must fit inside the MCP tool timeout (`PENPOT_MCP_TOOL_TIMEOUT_S`, default 120s). If layout or saves are slow and tools +time out before a save error appears, raise that env var. ## Rejected Changes and Read-Only Mode -Penpot saves your changes to the backend after your `execute_code` call returns. -When the backend rejects a save, Penpot undoes the rejected changes, together with the rest of their undo steps -and any later change to the same file. This can happen a few seconds after the call that made the changes succeeded. -The next tool response then starts with a warning that names the files, pages and shapes involved. +`execute_code` only reports success after Penpot has saved the changes from that call (or confirmed there was nothing +to save). If the save fails, times out, or the backend rejects the changes and the workspace rolls them back, the tool +call fails with an error — do not treat a failed call as if the design was updated. +A timeout while saves are still retrying means the save was not confirmed; it does not mean the file is fine. + +When a rollback is reported only on a later tool response (rare: concurrent edits, or a race after a timed-out response), +that response starts with a warning that names the files, pages and shapes involved. When you get this warning: * Do not trust what you know about the design from before: shapes you created may be gone, and removed ones may be back. * Read the affected pages and shapes again (e.g. `penpotUtils.shapeStructure(page.root, 3)`) and check that they hold what you expect. @@ -39,6 +47,11 @@ When you get this warning: While saves are failing, the workspace becomes read-only: every write through the API throws an error saying that the workspace is read-only because changes cannot be saved. Do not retry in a loop; wait and try again later, or tell the user. +Once a save has failed for good, every `execute_code` call in that file fails with "Changes could not be saved", reads +included, until the file is reopened. Tell the user to reopen the file. If the error says Penpot keeps retrying the +saves, wait and try again later instead. Calls made from the dashboard do not wait for saves. +A save failure does not undo `penpotMgmt` calls (projects, files, teams) made in the same code: read that state back +before retrying them. Writes also throw when the user cannot edit the file, a version is being previewed or the canvas stopped rendering; the error says which. These do not pass by waiting: tell the user. diff --git a/mcp/packages/server/src/PenpotMcpServer.http.test.ts b/mcp/packages/server/src/PenpotMcpServer.http.test.ts index d5534de879..0698bbe1e9 100644 --- a/mcp/packages/server/src/PenpotMcpServer.http.test.ts +++ b/mcp/packages/server/src/PenpotMcpServer.http.test.ts @@ -1,4 +1,5 @@ import assert from "node:assert/strict"; +import { request as httpRequest } from "node:http"; import { afterEach, beforeEach, test } from "node:test"; import { type CallToolResult, Client, StreamableHTTPClientTransport } from "@modelcontextprotocol/client"; import { PenpotMcpServer } from "./PenpotMcpServer"; @@ -173,6 +174,72 @@ test("OAuth challenges missing credentials and accepts bearer and query tokens", } }); +/** + * Posts an MCP initialize request with the given extra headers and returns the response status. + * + * Uses `node:http`, since `fetch` does not let the caller set the `Host` header. + */ +function initializeStatus(headers: Record): Promise { + const body = JSON.stringify({ + jsonrpc: "2.0", + id: 1, + method: "initialize", + params: { + protocolVersion: "2025-06-18", + capabilities: {}, + clientInfo: { name: "guard-test", version: "1" }, + }, + }); + return new Promise((resolve, reject) => { + const req = httpRequest( + `${baseUrl}/mcp`, + { + method: "POST", + headers: { + "Content-Type": "application/json", + Accept: "application/json, text/event-stream", + ...headers, + }, + }, + (res) => { + res.resume(); + resolve(res.statusCode!); + } + ); + req.on("error", reject); + req.end(body); + }); +} + +for (const remoteMode of [undefined, "true"]) { + test(`single-user mode rejects requests from foreign hosts and origins (remote mode: ${remoteMode})`, async () => { + await server.stop(); + if (remoteMode === undefined) { + delete process.env.PENPOT_MCP_REMOTE_MODE; + } else { + process.env.PENPOT_MCP_REMOTE_MODE = remoteMode; + } + delete process.env.PENPOT_MCP_ALLOWED_HOSTS; + server = new PenpotMcpServer(false); + await server.start(); + + assert.equal(await initializeStatus({ Host: `attacker.example:${server.port}` }), 403); + assert.equal( + await initializeStatus({ Host: `localhost:${server.port}`, Origin: "https://attacker.example" }), + 403 + ); + assert.equal(await initializeStatus({ Host: `localhost:${server.port}` }), 200); + assert.equal( + await initializeStatus({ Host: `127.0.0.1:${server.port}`, Origin: "http://localhost:4400" }), + 200 + ); + }); +} + +test("multi-user mode accepts requests from any host by default", async () => { + assert.equal(await initializeStatus({ Host: `penpot.example.com`, Origin: "https://penpot.example.com" }), 200); +}); + /** * Makes the plugin bridge answer each dispatched task at once with the given responses, in order. */ diff --git a/mcp/packages/server/src/PenpotMcpServer.ts b/mcp/packages/server/src/PenpotMcpServer.ts index 56f7ac5b2c..d9a1668f89 100644 --- a/mcp/packages/server/src/PenpotMcpServer.ts +++ b/mcp/packages/server/src/PenpotMcpServer.ts @@ -25,6 +25,7 @@ import { NreplClient } from "./NreplClient"; import { ReplServer } from "./ReplServer"; import { ApiDocs } from "./ApiDocs"; import { McpAuthentication } from "./McpAuthentication"; +import { RequestGuard } from "./RequestGuard"; /** * Session context for request-scoped data. @@ -115,6 +116,11 @@ export class PenpotMcpServer { private httpServer?: HttpServer; private readonly mcpHandler: McpHttpHandler; private readonly authentication: McpAuthentication; + + /** + * Check of the `Host` and `Origin` headers of HTTP and plugin WebSocket requests. + */ + public readonly requestGuard: RequestGuard; public readonly pluginBridge: PluginBridge; private readonly replServer: ReplServer | null; private apiDocs: ApiDocs; @@ -167,6 +173,8 @@ export class PenpotMcpServer { this.maxBodySize = process.env.PENPOT_MCP_MAX_BODY_SIZE ?? "10mb"; const toolTimeoutSecs = parseInt(process.env.PENPOT_MCP_TOOL_TIMEOUT_S ?? "120", 10); + this.requestGuard = RequestGuard.fromEnvironment(process.env, this.isMultiUserMode()); + this.configLoader = new ConfigurationLoader(process.cwd()); this.apiDocs = new ApiDocs(); @@ -190,7 +198,7 @@ export class PenpotMcpServer { this.pluginBridge = new PluginBridge(this, this.webSocketPort, toolTimeoutSecs, this.redisBridge); if (shouldStartReplServer(PenpotMcpServer.isReplEnabled(process.env), this.isMultiUserMode())) { - this.replServer = new ReplServer(this.pluginBridge, this.replPort, this.replHost); + this.replServer = new ReplServer(this.pluginBridge, this.requestGuard, this.replPort, this.replHost); } else { this.replServer = null; } @@ -332,6 +340,7 @@ export class PenpotMcpServer { async start(): Promise { const { default: express } = await import("express"); this.app = express(); + this.app.use(this.requestGuard.httpMiddleware); this.app.use(express.json({ limit: this.maxBodySize })); this.setupHttpEndpoints(); @@ -345,6 +354,13 @@ export class PenpotMcpServer { ); this.logger.info(`Remote mode: ${this.isRemoteMode()}`); this.logger.info(`DevEnv mode: ${this.isDevEnv()}`); + this.logger.info(`Allowed hosts and origins: ${this.requestGuard.describe()}`); + if (this.requestGuard.allowsAny() && !this.isMultiUserMode()) { + this.logger.warn( + "Host and Origin check is off and no user token is required: " + + "any web page opened in a browser that reaches this server can drive it" + ); + } this.logger.info(`Modern Streamable HTTP endpoint: http://${this.host}:${this.port}/mcp`); this.logger.info(`WebSocket server URL: ws://${this.host}:${this.webSocketPort}`); diff --git a/mcp/packages/server/src/PluginBridge.sessions.test.ts b/mcp/packages/server/src/PluginBridge.sessions.test.ts index 3e676725f2..428e3d736f 100644 --- a/mcp/packages/server/src/PluginBridge.sessions.test.ts +++ b/mcp/packages/server/src/PluginBridge.sessions.test.ts @@ -5,6 +5,7 @@ import { WebSocket } from "ws"; import { HEARTBEAT_REQUEST_INTERVAL_MS, PluginBridge } from "./PluginBridge"; import { PluginTask } from "./PluginTask"; import type { PenpotMcpServer } from "./PenpotMcpServer"; +import { RequestGuard } from "./RequestGuard"; let bridge: PluginBridge; let port = 17_600; @@ -22,6 +23,7 @@ beforeEach(() => { host: "127.0.0.1", isMultiUserMode: () => multiUser, getSessionContext: () => ({ userToken }), + requestGuard: RequestGuard.allowing(["localhost", "127.0.0.1"]), } as PenpotMcpServer, port, 1 @@ -203,3 +205,34 @@ test("a task fails at once when its plugin connection closes before answering", // well within the 1s task timeout of these tests assert.ok(Date.now() - started < 500); }); + +test("closes a WebSocket opened by a page on a foreign site with a policy violation", async () => { + const socket = new WebSocket(`ws://127.0.0.1:${port}?userToken=alice`, { + headers: { Origin: "http://attacker.example" }, + }); + sockets.push(socket); + const [code] = await once(socket, "close", { signal: AbortSignal.timeout(1500) }); + assert.equal(code, 1008); + assert.equal(bridge.getUserConnections("alice"), undefined); +}); + +test("closes a WebSocket that reaches the server under a foreign host name", async () => { + const socket = new WebSocket(`ws://127.0.0.1:${port}?userToken=alice`, { + headers: { Host: `attacker.example:${port}` }, + }); + sockets.push(socket); + const [code] = await once(socket, "close", { signal: AbortSignal.timeout(1500) }); + assert.equal(code, 1008); +}); + +test("accepts a WebSocket opened by a plugin page served from an allowed host", async () => { + const socket = new WebSocket(`ws://127.0.0.1:${port}?userToken=alice`, { + headers: { Origin: "http://localhost:4400" }, + }); + sockets.push(socket); + await once(socket, "open"); + const initialized = once(socket, "message", { signal: AbortSignal.timeout(1500) }); + socket.send(JSON.stringify({ type: "initialize", session: { sessionId: "tab", fileId: null, fileName: null } })); + const [message] = await initialized; + assert.equal(JSON.parse(message.toString()).type, "initialized"); +}); diff --git a/mcp/packages/server/src/PluginBridge.ts b/mcp/packages/server/src/PluginBridge.ts index 8bf426e9a8..50fe62fa78 100644 --- a/mcp/packages/server/src/PluginBridge.ts +++ b/mcp/packages/server/src/PluginBridge.ts @@ -127,6 +127,14 @@ export class PluginBridge implements TaskDispatchHost { */ private setupWebSocketHandlers(): void { this.wsServer.on("connection", (ws: WebSocket, request: http.IncomingMessage) => { + // reject pages on foreign sites before reading any message + const guardResult = this.mcpServer.requestGuard.check(request.headers); + if (!guardResult.isAllowed) { + this.logger.warn("Rejected WebSocket connection: %s", guardResult.reason); + ws.close(1008, "Origin or host not allowed; see PENPOT_MCP_ALLOWED_HOSTS"); + return; + } + // extract userToken from query parameters const url = new URL(request.url!, `ws://${request.headers.host}`); const userToken = url.searchParams.get("userToken"); diff --git a/mcp/packages/server/src/RedisBridge.test.ts b/mcp/packages/server/src/RedisBridge.test.ts index c64650ef1e..37daac69a6 100644 --- a/mcp/packages/server/src/RedisBridge.test.ts +++ b/mcp/packages/server/src/RedisBridge.test.ts @@ -5,6 +5,7 @@ import Redis, { type Command } from "ioredis"; import { WebSocket } from "ws"; import { RedisBridge } from "./RedisBridge"; import { PluginBridge } from "./PluginBridge"; +import { RequestGuard } from "./RequestGuard"; import { PluginTask } from "./PluginTask"; import type { PenpotMcpServer } from "./PenpotMcpServer"; import type { PenpotSession } from "@penpot/mcp-common"; @@ -163,6 +164,7 @@ function instance(t: TestContext) { host: "127.0.0.1", isMultiUserMode: () => true, getSessionContext: () => ({ userToken: "alice" }), + requestGuard: RequestGuard.allowingAny(), } as PenpotMcpServer, wsPort, 0.1, diff --git a/mcp/packages/server/src/ReplServer.ts b/mcp/packages/server/src/ReplServer.ts index 1894ade79a..51db327c81 100644 --- a/mcp/packages/server/src/ReplServer.ts +++ b/mcp/packages/server/src/ReplServer.ts @@ -4,6 +4,7 @@ import { fileURLToPath } from "url"; import { PluginBridge } from "./PluginBridge"; import { ExecuteCodePluginTask } from "./tasks/ExecuteCodePluginTask"; import { createLogger } from "./logger"; +import type { RequestGuard } from "./RequestGuard"; /** * Web-based REPL server for executing code through the PluginBridge. @@ -22,6 +23,7 @@ export class ReplServer { constructor( private readonly pluginBridge: PluginBridge, + private readonly requestGuard: RequestGuard, port: number = 4403, host: string = "localhost" ) { @@ -36,6 +38,7 @@ export class ReplServer { * Sets up Express middleware for request parsing and static content. */ private setupMiddleware(): void { + this.app.use(this.requestGuard.httpMiddleware); this.app.use(express.json()); } diff --git a/mcp/packages/server/src/RequestGuard.test.ts b/mcp/packages/server/src/RequestGuard.test.ts new file mode 100644 index 0000000000..ea60180b0b --- /dev/null +++ b/mcp/packages/server/src/RequestGuard.test.ts @@ -0,0 +1,60 @@ +import assert from "node:assert/strict"; +import { test } from "node:test"; +import { RequestGuard } from "./RequestGuard"; + +test("single-user mode allows only loopback hosts and origins by default", () => { + const guard = RequestGuard.fromEnvironment({}, false); + assert.equal(guard.allowsAny(), false); + assert.equal(guard.check({ host: "localhost:4401" }).isAllowed, true); + assert.equal(guard.check({ host: "127.0.0.1:4401", origin: "http://localhost:4400" }).isAllowed, true); + assert.equal(guard.check({ host: "[::1]:4401" }).isAllowed, true); + assert.equal(guard.check({ host: "attacker.example:4401" }).isAllowed, false); + assert.equal(guard.check({ host: "localhost:4401", origin: "https://attacker.example" }).isAllowed, false); + assert.equal(guard.check({ host: "localhost:4401", origin: "null" }).isAllowed, false); + assert.equal(guard.check({}).isAllowed, false); +}); + +test("multi-user mode allows any host and origin by default", () => { + const guard = RequestGuard.fromEnvironment({ PENPOT_MCP_ALLOWED_HOSTS: " , " }, true); + assert.equal(guard.allowsAny(), true); + assert.equal(guard.check({ host: "penpot.example.com", origin: "https://other.example" }).isAllowed, true); + assert.equal(guard.describe(), ""); +}); + +test("the configured hosts replace the default in either mode", () => { + for (const isMultiUserMode of [false, true]) { + const guard = RequestGuard.fromEnvironment( + { PENPOT_MCP_ALLOWED_HOSTS: " Penpot.Example.com, localhost " }, + isMultiUserMode + ); + assert.equal(guard.describe(), "penpot.example.com, localhost"); + assert.equal(guard.check({ host: "penpot.example.com", origin: "https://penpot.example.com" }).isAllowed, true); + assert.equal(guard.check({ host: "localhost:4401" }).isAllowed, true); + assert.equal(guard.check({ host: "127.0.0.1:4401" }).isAllowed, false); + assert.equal(guard.check({ host: "penpot.example.com", origin: "https://attacker.example" }).isAllowed, false); + } +}); + +test("the wildcard turns the check off in single-user mode", () => { + const guard = RequestGuard.fromEnvironment({ PENPOT_MCP_ALLOWED_HOSTS: "localhost, *" }, false); + assert.equal(guard.describe(), ""); + assert.equal(guard.check({ host: "attacker.example", origin: "https://attacker.example" }).isAllowed, true); +}); + +test("hostnames, bracketed IPv6 addresses and the wildcard are valid entries", () => { + const guard = RequestGuard.fromEnvironment( + { PENPOT_MCP_ALLOWED_HOSTS: "localhost, 127.0.0.1, [::1], penpot-mcp, penpot_mcp, penpot.example.com" }, + false + ); + assert.equal(guard.check({ host: "penpot-mcp:4402" }).isAllowed, true); + assert.equal(guard.check({ host: "[::1]:4401" }).isAllowed, true); +}); + +test("entries with a scheme, port or path stop the server from starting", () => { + for (const entry of ["http://localhost", "localhost:4401", "localhost/", "[::1]:4401", "local host"]) { + assert.throws( + () => RequestGuard.fromEnvironment({ PENPOT_MCP_ALLOWED_HOSTS: `localhost, ${entry}` }, false), + (error: Error) => error.message.includes(`"${entry}"`) + ); + } +}); diff --git a/mcp/packages/server/src/RequestGuard.ts b/mcp/packages/server/src/RequestGuard.ts new file mode 100644 index 0000000000..d4a17c0adc --- /dev/null +++ b/mcp/packages/server/src/RequestGuard.ts @@ -0,0 +1,142 @@ +import type { IncomingHttpHeaders } from "node:http"; +import type { NextFunction, Request, Response } from "express"; +import { localhostAllowedHostnames, validateHostHeader, validateOriginHeader } from "@modelcontextprotocol/server"; +import { createLogger } from "./logger"; + +/** + * Outcome of checking the headers of one request against a {@link RequestGuard}. + */ +export class RequestGuardResult { + /** + * @param isAllowed whether the request may proceed + * @param reason why the request was rejected; empty when it is allowed + */ + constructor( + public readonly isAllowed: boolean, + public readonly reason: string = "" + ) {} +} + +/** + * Allowlist check of the `Host` and `Origin` headers of requests reaching the MCP server. + * + * Any web page the user opens can send requests to servers on the user's machine. The `Host` + * check stops DNS rebinding, where a page on a foreign domain whose name resolves to the + * loopback address reaches the server as a same-origin request. The `Origin` check stops + * pages on foreign sites from sending requests or opening the plugin WebSocket directly. + * Both headers are compared by hostname only, ignoring the port. A request without an + * `Origin` header comes from a client other than a browser and passes the `Origin` check. + * + * The allowed hostnames come from `PENPOT_MCP_ALLOWED_HOSTS`, where `*` allows any hostname. + * When it is unset, single-user mode allows only loopback hostnames, in local and remote mode + * alike, since it has no other barrier. Multi-user mode allows any hostname, since its server + * is reached under names it cannot know and relies on user tokens instead. + */ +export class RequestGuard { + /** + * Pattern of a valid allowlist entry: a hostname or a bracketed IPv6 address, without scheme or port. + */ + private static readonly HOSTNAME_PATTERN = /^(?:[a-z0-9_-]+(?:\.[a-z0-9_-]+)*|\[[0-9a-f:.]+\])$/; + + private readonly logger = createLogger("RequestGuard"); + + /** + * @param allowedHostnames the hostnames allowed in `Host` and `Origin`, or null to allow any + */ + private constructor(private readonly allowedHostnames: string[] | null) {} + + /** + * Creates the guard configured by `PENPOT_MCP_ALLOWED_HOSTS`. + * + * The variable holds a comma-separated list of hostnames, without ports; IPv6 addresses + * go in brackets (e.g. `[::1]`). The entry `*` allows any hostname, which turns the check off. + * + * @param env the environment variables + * @param isMultiUserMode whether the server runs in multi-user mode + * @throws Error if an entry is neither `*` nor a hostname, e.g. when it has a scheme or port + */ + public static fromEnvironment(env: Record, isMultiUserMode: boolean): RequestGuard { + const configured = (env.PENPOT_MCP_ALLOWED_HOSTS ?? "") + .split(",") + .map((hostname) => hostname.trim().toLowerCase()) + .filter((hostname) => hostname.length > 0); + for (const hostname of configured) { + if (hostname !== "*" && !RequestGuard.HOSTNAME_PATTERN.test(hostname)) { + throw new Error( + `PENPOT_MCP_ALLOWED_HOSTS entry "${hostname}" is not a hostname; ` + + "give hostnames without scheme, port or path (e.g. localhost, [::1])" + ); + } + } + if (configured.includes("*")) { + return RequestGuard.allowingAny(); + } + if (configured.length > 0) { + return new RequestGuard(configured); + } + return isMultiUserMode ? RequestGuard.allowingAny() : RequestGuard.allowing(localhostAllowedHostnames()); + } + + /** + * Creates a guard that allows only the given hostnames. + */ + public static allowing(hostnames: string[]): RequestGuard { + return new RequestGuard([...hostnames]); + } + + /** + * Creates a guard that allows every request. + */ + public static allowingAny(): RequestGuard { + return new RequestGuard(null); + } + + /** + * Indicates whether the guard allows every request. + */ + public allowsAny(): boolean { + return this.allowedHostnames === null; + } + + /** + * Describes the allowed hostnames, for logging. + */ + public describe(): string { + return this.allowedHostnames === null ? "" : this.allowedHostnames.join(", "); + } + + /** + * Checks the `Host` and `Origin` headers of a request. + */ + public check(headers: IncomingHttpHeaders): RequestGuardResult { + if (this.allowedHostnames === null) { + return new RequestGuardResult(true); + } + const host = validateHostHeader(headers.host, this.allowedHostnames); + if (!host.ok) { + return new RequestGuardResult(false, host.message); + } + const origin = validateOriginHeader(headers.origin, this.allowedHostnames); + if (!origin.ok) { + return new RequestGuardResult(false, origin.message); + } + return new RequestGuardResult(true); + } + + /** + * Express middleware that answers a rejected request with HTTP 403. + */ + public readonly httpMiddleware = (req: Request, res: Response, next: NextFunction): void => { + const result = this.check(req.headers); + if (result.isAllowed) { + next(); + return; + } + this.logger.warn("Rejected HTTP request to %s: %s", req.path, result.reason); + res.status(403).json({ + jsonrpc: "2.0", + error: { code: -32000, message: `${result.reason}; see PENPOT_MCP_ALLOWED_HOSTS` }, + id: null, + }); + }; +} diff --git a/plugins/CHANGELOG.md b/plugins/CHANGELOG.md index b86307e031..db2c6f861a 100644 --- a/plugins/CHANGELOG.md +++ b/plugins/CHANGELOG.md @@ -2,12 +2,14 @@ ### 🚀 Features +- **plugin-types, plugins-runtime:** Added `penpot.waitForSave(timeout?)` to wait until pending file changes are saved (or there is nothing to save). Rejects on save failure, rollback during the wait, timeout (default 60000 ms), or an invalid timeout. The Error `code` property is `save-timeout`, `rolled-back`, `save-retrying` (an earlier failed save Penpot keeps retrying) or the save failure code. - **plugin-types, plugins-runtime:** `penpotMgmt` returns `Team`, `Project` and `ProjectFile` objects. Assigning `name` renames teams, projects and files, `project.pinned` pins a project and `file.shared` shares a file as a team library; `createProject`, `createFile`, `moveTo`, `duplicate`, `remove` and `restore` live on the objects. Added `penpotMgmt.listTeams`, `penpotMgmt.createTeam`, `penpotMgmt.getFile`, `listDeletedProjects` and `listDeletedFiles`, and the `manage:projects`, `manage:teams` and `manage:delete` permissions. Restoration uses the dashboard's Trash operation; projects need recoverable files and empty projects cannot be restored; the API exposes no permanent deletion method. Team deletion is not exposed. - **plugins-runtime, plugin-types**: Added global plugins (`scope: "global"`) that remain active across dashboard and workspace navigation until closed or logged out, with `penpotMgmt` for project and file discovery, file navigation and workspace events. Global scope adds the `allow:global` permission, which users must accept when installing the plugin. - **plugin-types:** Added the `contentrollback` event and the `ContentRollback` type. The event fires after the backend rejected some changes and the workspace undid them, with the ids of the files, pages and shapes affected. It fires once the undone data is in the workspace, also when the workspace has to reload the file to undo the changes. ### 🩹 Fixes +- **plugins-runtime:** Methods that rejected an invalid value with a plain string (`connectLibrary`, file `restore`, `remove`, `pin`, `export`, `findVersions`, `saveVersion`) now reject with an `Error` carrying the same message. - **plugin-runtime:** `LibraryComponent.instance()`, `createShapeFromSvg()` and `createShapeFromSvgWithImages()` now create shapes at the page root instead of inside a board at the page origin. Closes #12086. - **plugin-types, plugin-runtime:** Dashboard property setters now update values after backend success and log failures without interrupting the workspace. File moves reject projects in other teams, backend rejections include their type and code, and `project.fileCount` stays at its listing snapshot. Duplicated projects include their file count and pinned state. - **plugin-runtime:** Cancel plugin loads before they execute after unloading or logout, and allow reopening plugins that close during startup. diff --git a/plugins/apps/plugin-api-test-suite/src/generated/api-surface.json b/plugins/apps/plugin-api-test-suite/src/generated/api-surface.json index 46d748bfc4..d8886bec66 100644 --- a/plugins/apps/plugin-api-test-suite/src/generated/api-surface.json +++ b/plugins/apps/plugin-api-test-suite/src/generated/api-surface.json @@ -74,6 +74,7 @@ "verticalPadding", "verticalSizing" ], + "ContentRollback": ["fileIds", "pageIds", "shapeIds"], "Context": [ "activeUsers", "alignHorizontal", @@ -115,7 +116,8 @@ "uploadMediaUrl", "version", "viewport", - "waitForLayoutUpdate" + "waitForLayoutUpdate", + "waitForSave" ], "ContextGeometryUtils": ["center"], "ContextTypesUtils": [ @@ -135,6 +137,7 @@ "Dissolve": ["duration", "easing", "type"], "Ellipse": ["fills", "type"], "EventsMap": [ + "contentrollback", "contentsave", "filechange", "finish", @@ -376,6 +379,8 @@ "PenpotMgmt": [ "createTeam", "getFile", + "listDeletedFiles", + "listDeletedProjects", "listProjects", "listTeams", "off", @@ -399,11 +404,13 @@ "fileCount", "id", "isDefault", + "listDeletedFiles", "listFiles", "moveTo", "name", "pinned", "remove", + "restore", "teamId" ], "ProjectFile": [ @@ -415,6 +422,7 @@ "open", "projectId", "remove", + "restore", "shared", "teamId" ], @@ -528,9 +536,10 @@ "createProject", "id", "isDefault", + "listDeletedFiles", + "listDeletedProjects", "listProjects", - "name", - "remove" + "name" ], "Text": [ "align", @@ -2145,6 +2154,26 @@ "array": false } }, + "ContentRollback": { + "fileIds": { + "decl": "ContentRollback", + "kind": "get", + "type": null, + "array": true + }, + "pageIds": { + "decl": "ContentRollback", + "kind": "get", + "type": null, + "array": true + }, + "shapeIds": { + "decl": "ContentRollback", + "kind": "get", + "type": null, + "array": true + } + }, "Context": { "management": { "decl": "Context", @@ -2409,6 +2438,12 @@ "kind": "method", "type": null, "array": false + }, + "waitForSave": { + "decl": "Context", + "kind": "method", + "type": null, + "array": false } }, "ContextGeometryUtils": { @@ -3045,6 +3080,12 @@ "kind": "get", "type": null, "array": false + }, + "contentrollback": { + "decl": "EventsMap", + "kind": "get", + "type": "ContentRollback", + "array": false } }, "Export": { @@ -6843,6 +6884,12 @@ "kind": "method", "type": null, "array": false + }, + "waitForSave": { + "decl": "Context", + "kind": "method", + "type": null, + "array": false } }, "PenpotMgmt": { @@ -6870,6 +6917,18 @@ "type": "Project", "array": true }, + "listDeletedProjects": { + "decl": "PenpotMgmt", + "kind": "method", + "type": "Project", + "array": true + }, + "listDeletedFiles": { + "decl": "PenpotMgmt", + "kind": "method", + "type": "ProjectFile", + "array": true + }, "getFile": { "decl": "PenpotMgmt", "kind": "method", @@ -6998,6 +7057,18 @@ "type": "ProjectFile", "array": true }, + "listDeletedFiles": { + "decl": "Project", + "kind": "method", + "type": "ProjectFile", + "array": true + }, + "restore": { + "decl": "Project", + "kind": "method", + "type": null, + "array": false + }, "createFile": { "decl": "Project", "kind": "method", @@ -7066,6 +7137,12 @@ "type": null, "array": false }, + "restore": { + "decl": "ProjectFile", + "kind": "method", + "type": null, + "array": false + }, "moveTo": { "decl": "ProjectFile", "kind": "method", @@ -8752,17 +8829,23 @@ "type": "Project", "array": true }, + "listDeletedProjects": { + "decl": "Team", + "kind": "method", + "type": "Project", + "array": true + }, + "listDeletedFiles": { + "decl": "Team", + "kind": "method", + "type": "ProjectFile", + "array": true + }, "createProject": { "decl": "Team", "kind": "method", "type": "Project", "array": false - }, - "remove": { - "decl": "Team", - "kind": "method", - "type": null, - "array": false } }, "Text": { diff --git a/plugins/apps/plugin-api-test-suite/src/tests/wait-save.test.ts b/plugins/apps/plugin-api-test-suite/src/tests/wait-save.test.ts new file mode 100644 index 0000000000..fa9e674727 --- /dev/null +++ b/plugins/apps/plugin-api-test-suite/src/tests/wait-save.test.ts @@ -0,0 +1,40 @@ +import { expect, expectReject } from '../framework/expect'; +import { describe, test } from '../framework/registry'; + +// waitForSave: resolves once the pending file changes are saved, or when +// there is nothing to save. It waits on the shared persistence queue, so the +// edits of earlier tests may be in flight too; each test only asserts what +// holds whatever else is queued. + +describe('WaitForSave', () => { + test('resolves after an edit is saved', async (ctx) => { + const r = ctx.penpot.createRectangle(); + ctx.board.appendChild(r); + r.name = 'wait-save-probe'; + + const value = await ctx.penpot.waitForSave(); + expect(value).toBeUndefined(); + expect(r.name).toBe('wait-save-probe'); + }); + + test('resolves when there is nothing to save', async (ctx) => { + await ctx.penpot.waitForSave(); + await ctx.penpot.waitForSave(5000); + }); + + test('rejects an invalid timeout with an Error', async (ctx) => { + for (const value of [ + -1, + 0, + Number.NaN, + Number.POSITIVE_INFINITY, + 2 ** 31, + ]) { + const p = ctx.penpot.waitForSave(value); + expect(p instanceof Promise).toBe(true); + await expectReject(p, /not valid/); + const error = await p.catch((e: unknown) => e); + expect(error instanceof Error).toBe(true); + } + }); +}); diff --git a/plugins/libs/plugin-types/index.d.ts b/plugins/libs/plugin-types/index.d.ts index a21eb02a64..fade26e1f2 100644 --- a/plugins/libs/plugin-types/index.d.ts +++ b/plugins/libs/plugin-types/index.d.ts @@ -1630,6 +1630,24 @@ export interface Context { * rejected with an Error, both on timeout and on an invalid timeout value. */ waitForLayoutUpdate(timeout?: number): Promise; + + /** + * This method returns a promise that will be resolved when pending file + * changes have been saved to the backend, or when there is nothing to save. + * It force-flushes the persistence buffer first when work is pending. + * @param timeout Maximum time to wait, in milliseconds. If the timeout + * elapses before the save settles, the promise is rejected. Defaults to + * 60000; the promise never waits indefinitely. + * @return The promise to be resolved when the save has settled. It is + * rejected with an Error on save failure, when changes are rolled back + * during the wait, on timeout, or on an invalid timeout value. A save + * that failed before the call also rejects: with `save-retrying` while + * Penpot keeps retrying it, otherwise with the failure code until the + * file is reopened. The Error `code` property tells the cases apart: + * `save-timeout`, `rolled-back`, `save-retrying`, or the failure code + * (`save-failed` unless the backend gave a more specific one). + */ + waitForSave(timeout?: number): Promise; } /** diff --git a/plugins/libs/plugins-runtime/src/lib/api/index.ts b/plugins/libs/plugins-runtime/src/lib/api/index.ts index b7fd1a0c02..5fa70a68c7 100644 --- a/plugins/libs/plugins-runtime/src/lib/api/index.ts +++ b/plugins/libs/plugins-runtime/src/lib/api/index.ts @@ -413,6 +413,11 @@ export function createApi( checkPermission('content:read'); return plugin.context.waitForLayoutUpdate(timeout); }, + + waitForSave(timeout?: number): Promise { + checkPermission('content:read'); + return plugin.context.waitForSave(timeout); + }, }; const penpotMgmt: PenpotMgmt = { diff --git a/plugins/libs/plugins-runtime/src/lib/api/plugin-api.spec.ts b/plugins/libs/plugins-runtime/src/lib/api/plugin-api.spec.ts index b90ec91f63..d177d02f9c 100644 --- a/plugins/libs/plugins-runtime/src/lib/api/plugin-api.spec.ts +++ b/plugins/libs/plugins-runtime/src/lib/api/plugin-api.spec.ts @@ -37,6 +37,8 @@ describe('Plugin api', () => { theme: 'dark', addListener: vi.fn().mockReturnValueOnce(Symbol()), removeListener: vi.fn(), + waitForSave: vi.fn().mockResolvedValue(undefined), + waitForLayoutUpdate: vi.fn().mockResolvedValue(undefined), }, }; } @@ -186,6 +188,30 @@ describe('Plugin api', () => { }); }); + describe('waitForSave', () => { + it('forwards to plugin.context.waitForSave', async () => { + await expect(api.penpot.waitForSave(1234)).resolves.toBeUndefined(); + expect(pluginManager.context.waitForSave).toHaveBeenCalledWith(1234); + }); + + it('requires content:read', () => { + const waitForSave = vi.fn().mockResolvedValue(undefined); + const { penpot } = createApi({ + ...pluginManager, + manifest: { + ...pluginManager.manifest, + permissions: [], + }, + context: { ...pluginManager.context, waitForSave }, + } as any); + + expect(() => penpot.waitForSave(100)).toThrow( + 'Permission content:read is not granted', + ); + expect(waitForSave).not.toHaveBeenCalled(); + }); + }); + describe('ui', () => { describe.concurrent('permissions', () => { const api = createApi({ diff --git a/render-wasm/src/render.rs b/render-wasm/src/render.rs index 14da5d4de2..ed1ab3f680 100644 --- a/render-wasm/src/render.rs +++ b/render-wasm/src/render.rs @@ -439,9 +439,12 @@ pub(crate) struct RenderState { pub nested_fills: Vec>, pub nested_blurs: Vec>, // FIXME: why is this an option? pub nested_shadows: Vec>, - /// Cumulative [`Shape::masked_group_filter_reach`] of the masked groups - /// being walked: their children cast into tiles they don't touch. - masked_group_reach: Vec, + /// How far past the current tile shapes can paint into it, in document + /// units: the walker culls against the tile grown by the last entry. Each + /// walked frame or group adds its [`Shape::children_reach`]. In a tile + /// with a background blur the bottom entry is how far its kernel samples + /// the backdrop, capped at the surface margin. + group_reach: Vec, pub show_grid: Option, pub focus_mode: FocusMode, /// Viewer-only whitelist for fixed-scroll layer passes. @@ -669,7 +672,7 @@ impl RenderState { nested_fills: vec![], nested_blurs: vec![], nested_shadows: vec![], - masked_group_reach: vec![], + group_reach: vec![], show_grid: None, focus_mode: FocusMode::new(), include_filter: None, @@ -813,17 +816,12 @@ impl RenderState { }; let scale = self.get_scale(); - let scaled_sigma = radius_to_sigma(blur.value * scale); - // Cap sigma so the blur kernel (≈3σ) stays within the tile margin. - // This prevents visible seams at tile boundaries when zoomed in. - // During export there's no tiling, so skip the cap. - let sigma = if self.export_context.is_some() { - scaled_sigma - } else { - let margin = self.surfaces.margins().width as f32; - let max_sigma = margin / 3.0; - scaled_sigma.min(max_sigma) - }; + // During export there's no tiling, so skip the margin cap. + let margin = self + .export_context + .is_none() + .then(|| self.surfaces.margins().width as f32); + let sigma = Self::background_blur_sigma(blur, scale, margin); let blur_filter = match skia::image_filters::blur((sigma, sigma), skia::TileMode::Clamp, None, None) { @@ -2200,6 +2198,53 @@ impl RenderState { self.surfaces.update_render_context(self.render_area, scale); } + /// Device-space sigma of a background blur at `scale`. In a tile it is + /// capped so the blur kernel (≈3σ) stays within the tile `margin`, in + /// device pixels, which prevents visible seams at tile boundaries when + /// zoomed in. + fn background_blur_sigma(blur: Blur, scale: f32, margin: Option) -> f32 { + let sigma = radius_to_sigma(blur.value * scale); + margin.map_or(sigma, |margin| sigma.min(margin / 3.0)) + } + + /// How far past `tile_rect` the shapes in `roots` with a visible + /// background blur that paint into it sample the backdrop, in document + /// units: the widest blur kernel (about 3σ, plus a device pixel), up to + /// the tile `margin` in device pixels. `None` when no such shape paints + /// into the tile. + fn background_blur_in_tile( + tree: ShapesPoolRef, + roots: impl IntoIterator, + tile_rect: Rect, + scale: f32, + margin: f32, + ) -> Option { + let mut reach: Option = None; + let mut pending: Vec = roots.into_iter().collect(); + while let Some(id) = pending.pop() { + let Some(shape) = tree.get(&id) else { + continue; + }; + if shape.hidden || !shape.extrect(tree, scale).intersects(tile_rect) { + continue; + } + if let Some(blur) = shape.visible_background_blur() { + let sigma = Self::background_blur_sigma(blur, scale, Some(margin)); + let device_reach = (3.0 * sigma + 1.0).min(margin); + reach = Some(reach.unwrap_or(0.0).max(device_reach / scale)); + if device_reach >= margin { + // No blur in the tile can sample further. + return reach; + } + } + pending.extend(shape.children_ids_iter(false).copied()); + if shape.is_masked_group() { + pending.extend(shape.mask_id().copied()); + } + } + reach + } + fn rebuild_backbuffer_crop_cache(&mut self, tree: ShapesPoolRef) { self.backbuffer_crop_cache.clear(); @@ -2455,7 +2500,7 @@ impl RenderState { self.nested_fills.clear(); self.nested_blurs.clear(); self.nested_shadows.clear(); - self.masked_group_reach.clear(); + self.group_reach.clear(); // reorder by distance to the center. self.current_tile = None; @@ -2752,7 +2797,7 @@ impl RenderState { let saved_nested_fills = std::mem::take(&mut self.nested_fills); let saved_nested_blurs = std::mem::take(&mut self.nested_blurs); let saved_nested_shadows = std::mem::take(&mut self.nested_shadows); - let saved_masked_group_reach = std::mem::take(&mut self.masked_group_reach); + let saved_group_reach = std::mem::take(&mut self.group_reach); let saved_ignore_nested_blurs = self.ignore_nested_blurs; let saved_preview_mode = self.preview_mode; @@ -2825,7 +2870,7 @@ impl RenderState { self.nested_fills = saved_nested_fills; self.nested_blurs = saved_nested_blurs; self.nested_shadows = saved_nested_shadows; - self.masked_group_reach = saved_masked_group_reach; + self.group_reach = saved_group_reach; self.ignore_nested_blurs = saved_ignore_nested_blurs; self.preview_mode = saved_preview_mode; @@ -2961,6 +3006,11 @@ impl RenderState { clip_bounds: Option<&ClipStack>, target_surface: SurfaceId, ) { + if matches!(element.shape_type, Type::Frame(_) | Type::Group(_)) { + let reach = self.group_reach.last().copied().unwrap_or(0.0) + element.children_reach(); + self.group_reach.push(reach); + } + // Masked groups needs two rendering passes, the first one rendering // the content and the second one rendering the mask so we need to do // an extra save_layer to keep all the masked group separate from @@ -2978,10 +3028,6 @@ impl RenderState { } if group.masked { - let reach = self.masked_group_reach.last().copied().unwrap_or(0.0) - + element.masked_group_filter_reach(); - self.masked_group_reach.push(reach); - // A masked group's blur and shadows are applied as a single // image filter over the whole masked result. let scale = self.get_scale(); @@ -3069,6 +3115,9 @@ impl RenderState { if let Type::Group(group) = element.shape_type { if group.masked { self.surfaces.canvas(target_surface).restore(); + // Kept through the mask pass: the group's filter also + // carries the mask's children into this tile. + self.group_reach.pop(); } } } else { @@ -3080,7 +3129,6 @@ impl RenderState { // the blend mode 'destination-in') the content // of the group and the mask. if group.masked { - self.masked_group_reach.pop(); self.pending_nodes.push(NodeRenderState { id: element.id, visited_children: true, @@ -3109,6 +3157,10 @@ impl RenderState { self.nested_fills.pop(); self.nested_blurs.pop(); self.nested_shadows.pop(); + // A masked group pops it after the mask pass instead. + if !element.is_masked_group() { + self.group_reach.pop(); + } } _ => {} } @@ -4005,9 +4057,9 @@ impl RenderState { ); let has_effects = transformed_element.has_effects_that_extend_bounds(); - let area = match self.masked_group_reach.last() { - Some(&reach) => self.render_area_with_margins.with_outset((reach, reach)), - None => self.render_area_with_margins, + let area = match self.group_reach.last() { + Some(&reach) => self.render_area.with_outset((reach, reach)), + None => self.render_area, }; let is_visible = export @@ -4406,6 +4458,7 @@ impl RenderState { // let's check if there are more pending nodes if let Some(next_tile) = self.pending_tiles.pop() { self.update_render_context(next_tile); + self.group_reach.clear(); // Reset for the new tile. We'll flip it to true if the // tile has shapes, so a later "is_empty=true" reflects // a resumed-from-yield case rather than a genuinely @@ -4440,10 +4493,21 @@ impl RenderState { // If so, we need ALL root shapes rendered (not just those // assigned to this tile) because the blur snapshots Current // which must contain the shapes behind it. - let tile_has_bg_blur = ids.iter().any(|id| { - tree.get(id) - .is_some_and(|s| s.visible_background_blur().is_some()) - }); + let bg_blur_reach = if self.options.is_fast_mode() { + None + } else { + Self::background_blur_in_tile( + tree, + root_ids.iter().filter(|id| ids.contains(*id)).copied(), + self.render_area, + current_scale, + self.surfaces.margins().width as f32, + ) + }; + let tile_has_bg_blur = bg_blur_reach.is_some(); + if let Some(reach) = bg_blur_reach { + self.group_reach.push(reach); + } // We only need first level shapes, in the same order as the parent node. // @@ -4960,6 +5024,7 @@ impl RenderState { mod tests { use super::*; use crate::shapes::{Frame, Group, Rect as RectType}; + use crate::state::ShapesPool; fn frame(clip: bool) -> Shape { let mut shape = Shape::new(Uuid::new_v4()); @@ -5183,4 +5248,59 @@ mod tests { assert_eq!(stack[0].0, Rect::from_ltrb(2.0, 2.0, 98.0, 98.0)); } + + #[test] + fn finds_a_visible_nested_background_blur_in_the_tiles_it_touches() { + // A board holding a rect with a background blur at (300, 300)-(400, 400). + let mut pool = ShapesPool::new(); + pool.initialize(2); + let board = Uuid::new_v4(); + let glass = Uuid::new_v4(); + { + let shape = pool.add_shape(board); + shape.set_shape_type(Type::Frame(Frame::default())); + shape.set_selrect(0.0, 0.0, 1000.0, 1000.0); + shape.add_child(glass); + } + { + let shape = pool.add_shape(glass); + shape.set_shape_type(Type::Rect(RectType::default())); + shape.set_selrect(300.0, 300.0, 400.0, 400.0); + shape.set_parent(board); + } + let blur = |value| Some(Blur::new(BlurType::BackgroundBlur, false, value)); + let touching = Rect::from_xywh(256.0, 256.0, 256.0, 256.0); + let away = Rect::from_xywh(512.0, 512.0, 256.0, 256.0); + let reach = |pool: &ShapesPool, tile| { + RenderState::background_blur_in_tile(pool, [board], tile, 2.0, 256.0) + }; + let found = |pool: &ShapesPool, tile| reach(pool, tile).is_some(); + let set_hidden = |pool: &mut ShapesPool, id, hidden| { + pool.get_mut(id).expect("shape").set_hidden(hidden); + }; + let set_blur = |pool: &mut ShapesPool, blur| { + pool.get_mut(&glass) + .expect("glass") + .set_background_blur(blur); + }; + + set_blur(&mut pool, blur(10.0)); + assert!(found(&pool, touching)); + assert!(!found(&pool, away)); + + set_hidden(&mut pool, &glass, true); + assert!(!found(&pool, touching), "hidden shape"); + set_hidden(&mut pool, &glass, false); + + set_hidden(&mut pool, &board, true); + assert!(!found(&pool, touching), "hidden board"); + set_hidden(&mut pool, &board, false); + + // The reach grows with the blur, up to the margin (256 / 2). + let small = reach(&pool, touching).expect("small blur"); + set_blur(&mut pool, blur(40.0)); + assert!(reach(&pool, touching).expect("larger blur") > small); + set_blur(&mut pool, blur(1000.0)); + assert_eq!(reach(&pool, touching), Some(128.0)); + } } diff --git a/render-wasm/src/shapes.rs b/render-wasm/src/shapes.rs index 111745cbae..b8c2591ab8 100644 --- a/render-wasm/src/shapes.rs +++ b/render-wasm/src/shapes.rs @@ -1968,20 +1968,26 @@ impl Shape { } } - /// How far the masked-group layer filter reaches past the content, in - /// document units: the widest drop shadow plus the layer blur. - pub fn masked_group_filter_reach(&self) -> f32 { + /// How far this frame or group's effects carry its descendants' pixels + /// past their own bounds, in document units: a group's widest drop shadow + /// (a masked group's layer filter casts it, other groups' texts repeat + /// it) plus the layer blur. + pub fn children_reach(&self) -> f32 { let reach = |filter: Option| { filter.map_or(0.0, |f| { let r = f.compute_fast_bounds(math::Rect::default()); (-r.left).max(-r.top).max(r.right).max(r.bottom) }) }; - let shadows = self - .drop_shadows_visible() - .map(|shadow| reach(shadow.get_drop_shadow_filter())) - .fold(0.0, f32::max); - let blur = self.masked_group_layer_blur().map_or(0.0, |blur| { + let shadows = match self.shape_type { + Type::Group(_) => self + .drop_shadows_visible() + .map(|shadow| reach(shadow.get_drop_shadow_filter())) + .fold(0.0, f32::max), + Type::Frame(_) => 0.0, + _ => return 0.0, + }; + let blur = self.visible_layer_blur().map_or(0.0, |blur| { let sigma = radius_to_sigma(blur.value); reach(skia::image_filters::blur((sigma, sigma), None, None, None)) }); @@ -2404,6 +2410,50 @@ mod tests { } } + #[test] + fn children_reach_covers_a_visible_layer_blur_or_group_drop_shadow() { + let shape_of = |shape_type| { + let mut shape = any_shape(); + shape.set_shape_type(shape_type); + shape + }; + let group = |masked| shape_of(Type::Group(Group { masked })); + let frame = |clip| { + let mut shape = shape_of(Type::Frame(Frame::default())); + shape.clip_content = clip; + shape + }; + let rect = || shape_of(Type::Rect(Rect::default())); + let blur = |hidden| Some(Blur::new(BlurType::LayerBlur, hidden, 10.0)); + // (shape, layer blur, drop shadow, reaches) + let cases = [ + (group(false), None, false, false), + (group(false), blur(false), false, true), + (frame(false), blur(false), false, true), + (frame(true), blur(false), false, true), + (group(false), blur(true), false, false), + (group(false), None, true, true), + // A board's shadow is cast by the board, not repeated by children. + (frame(false), None, true, false), + // A masked group applies both through its layer filter. + (group(true), blur(false), true, true), + // Only containers have descendants to carry. + (rect(), blur(false), true, false), + ]; + for (i, (mut shape, blur, has_shadow, reaches)) in cases.into_iter().enumerate() { + shape.set_blur(blur); + if has_shadow { + shape.add_shadow(drop_shadow(4.0, 0.0, (30.0, 0.0))); + } + assert_eq!(shape.children_reach() > 0.0, reaches, "case {i}"); + } + + // A shadow reaches past its offset. + let mut shadowed = group(false); + shadowed.add_shadow(drop_shadow(4.0, 0.0, (30.0, 0.0))); + assert!(shadowed.children_reach() > 30.0); + } + fn masked_group_with(shadows: Vec, blur: Option) -> Shape { let mut shape = any_shape(); shape.set_shape_type(Type::Group(Group { masked: true })); diff --git a/scripts/check-commit b/scripts/check-commit index bc2fa0b6fc..c434e458b8 100755 --- a/scripts/check-commit +++ b/scripts/check-commit @@ -4,7 +4,7 @@ Check commit messages against Penpot's commit guidelines. Validates commit messages using the rules defined in: - .github/workflows/commit-checker.yml (regex pattern) - - CONTRIBUTING.md (formatting rules, subject length, DCO) + - CONTRIBUTING.md (formatting rules, subject length) - .serena/memories/workflow/creating-commits.md (body wrapped at 80 chars) By default, checks HEAD. Use --commit to specify a different commit. @@ -53,11 +53,26 @@ MAX_SUBJECT_LINE = 80 MAX_BODY_LINE = 80 TRAILER_PATTERN = re.compile( - r"^(Signed-off-by|Co-developed-by|Reviewed-by|" + r"^(Signed-off-by|Co-authored-by|Co-developed-by|Reviewed-by|" r"Acked-by|Tested-by|Reported-by|Suggested-by|AI-assisted-by):" ) -FORBIDDEN_TRAILER_PATTERN = re.compile(r"^\s*Co-authored-by\s*:", re.IGNORECASE) +# ── Co-authored-by trailer ─────────────────────────────────────────────────── +# A human co-author may be credited with Co-authored-by. The trailer is +# forbidden when its value names an AI provider, an AI agent, or a bot: for AI +# participation the message uses "AI-assisted-by: " instead. +# Extend this list when a new tool starts adding trailers. +NON_HUMAN_COAUTHOR_PATTERN = re.compile( + r"^\s*co-authored-by\s*:.*(" + r"\b(?:chatgpt|openai|anthropic|claude|copilot|codex|gemini|bard|" + r"deepseek|mistral|qwen|kimi|llama|grok|cursor|" + r"windsurf|cline|aider|codeium|tabnine|amazon\s+q|sourcegraph\s+cody|" + r"roo\s+code|kilo\s+code|opencode)\b" + r"|\bdevin-ai|@devin\.ai" + r"|\[bot\]|\bbot@" + r")", + re.IGNORECASE, +) URL_PATTERN = re.compile(r"https?://\S+") @@ -249,23 +264,12 @@ def check_body_line_length(message): ) -def check_signed_off_by(message): - """Check for the DCO Signed-off-by line (required for code changes).""" - if "Signed-off-by:" not in message: - return False, ( - "Missing 'Signed-off-by:' line in the commit footer.\n" - " Add it with 'git commit -s' or append it manually:\n" - " Signed-off-by: Your Real Name " - ) - return True, None - - -def check_no_co_authored_by(message): - """Check that the forbidden Co-authored-by trailer is absent.""" +def check_co_authored_by(message): + """Co-authored-by is allowed for humans, not for AI providers or bots.""" offenders = [ (line_number, line) for line_number, line in enumerate(message.split("\n"), start=1) - if FORBIDDEN_TRAILER_PATTERN.match(line) + if NON_HUMAN_COAUTHOR_PATTERN.match(line) ] if not offenders: return True, None @@ -274,8 +278,10 @@ def check_no_co_authored_by(message): for line_number, line in offenders ) return False, ( - "The 'Co-authored-by:' trailer is forbidden and must not appear " - "in the commit message. Offending line(s):\n" + details + "The 'Co-authored-by:' trailer is only for human co-authors. This one " + "names an AI provider, an AI agent, or a bot. Offending line(s):\n" + + details + + "\n For AI participation use: AI-assisted-by: " ) @@ -313,7 +319,7 @@ def main(): ("Subject capitalized", check_subject_capitalized), ("Blank line after subject", check_body_blank_line), ("Body wrapped at 80 chars", check_body_line_length), - ("No Co-authored-by trailer", check_no_co_authored_by), + ("Co-authored-by is human only", check_co_authored_by), ] all_ok = True diff --git a/scripts/test_check_commit.py b/scripts/test_check_commit.py index fca40ccb93..2be60ba9ac 100644 --- a/scripts/test_check_commit.py +++ b/scripts/test_check_commit.py @@ -6,7 +6,9 @@ Run with: python3 scripts/test_check_commit.py Covers the commit message rules enforced by scripts/check-commit and documented -in .serena/memories/workflow/creating-commits.md. +in .serena/memories/workflow/creating-commits.md: the body line-wrapping +validator, the Co-authored-by validator (which allows human co-authors and +rejects AI providers and bots), and message checking from files. """ import contextlib @@ -92,6 +94,14 @@ class BodyLineLengthTests(unittest.TestCase): line = "Signed-off-by: Someone With A Long Name " self.assert_ok(":bug: Fix crash\n\nBody.\n\n" + line + "\n") + def test_human_co_author_trailer_passes(self): + line = ( + "Co-authored-by: Someone With A Very Long Name " + "" + ) + self.assertGreater(len(line), 80) + self.assert_ok(":bug: Fix crash\n\nBody.\n\n" + line + "\n") + def test_unbreakable_token_passes(self): line = "a" * 100 # no whitespace to wrap at self.assert_ok(":bug: Fix crash\n\n" + line + "\n") @@ -111,14 +121,16 @@ class BodyLineLengthTests(unittest.TestCase): self.assertIn("line 4", error) -class ForbiddenTrailerTests(unittest.TestCase): +class CoAuthorTrailerTests(unittest.TestCase): + """Co-authored-by is allowed for humans, forbidden for AI providers/bots.""" + def assert_ok(self, message): - ok, error = check_commit.check_no_co_authored_by(message) + ok, error = check_commit.check_co_authored_by(message) self.assertTrue(ok, error) self.assertIsNone(error) def assert_fail(self, message): - ok, error = check_commit.check_no_co_authored_by(message) + ok, error = check_commit.check_co_authored_by(message) self.assertFalse(ok) self.assertIsNotNone(error) return error @@ -126,15 +138,57 @@ class ForbiddenTrailerTests(unittest.TestCase): def test_body_without_trailer_passes(self): self.assert_ok(":bug: Fix crash\n\nBody.\n\nSigned-off-by: Someone \n") - def test_co_authored_by_fails(self): - error = self.assert_fail(":bug: Fix crash\n\nBody.\n\nCo-authored-by: Someone \n") - self.assertIn("Co-authored-by", error) + def test_human_co_author_passes(self): + self.assert_ok(":bug: Fix crash\n\nBody.\n\nCo-authored-by: Jane Doe \n") - def test_co_authored_by_case_insensitive_fails(self): - self.assert_fail(":bug: Fix crash\n\nBody.\n\nco-authored-by: Someone \n") + def test_human_with_github_noreply_email_passes(self): + self.assert_ok( + ":bug: Fix crash\n\nBody.\n\n" + "Co-authored-by: Jane Doe <1234+jane@users.noreply.github.com>\n" + ) + + def test_human_whose_email_ends_with_bot_passes(self): + # "abbot@" must not count as a "bot@" identity. + self.assert_ok( + ":bug: Fix crash\n\nBody.\n\n" + "Co-authored-by: Jane Abbot \n" + ) + + def test_claude_fails(self): + error = self.assert_fail( + ":bug: Fix crash\n\nBody.\n\n" + "Co-authored-by: Claude \n" + ) + self.assertIn("Co-authored-by", error) + self.assertIn("AI-assisted-by", error) + + def test_cursor_fails(self): + self.assert_fail( + ":bug: Fix crash\n\nBody.\n\n" + "Co-authored-by: Cursor Agent \n" + ) + + def test_copilot_fails(self): + self.assert_fail( + ":bug: Fix crash\n\nBody.\n\n" + "Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>\n" + ) + + def test_case_insensitive_fails(self): + self.assert_fail( + ":bug: Fix crash\n\nBody.\n\nco-authored-by: CHATGPT \n" + ) + + def test_bot_identity_fails(self): + self.assert_fail( + ":bug: Fix crash\n\nBody.\n\n" + "Co-authored-by: dependabot[bot] \n" + ) def test_co_authored_by_in_body_fails(self): - self.assert_fail(":bug: Fix crash\n\nCo-authored-by: Someone \nBody.\n") + self.assert_fail( + ":bug: Fix crash\n\nCo-authored-by: Gemini \nBody.\n" + ) def test_similar_trailer_passes(self): self.assert_ok(":bug: Fix crash\n\nBody.\n\nCo-developed-by: Someone \n")