diff --git a/.agents/skills/update-changelog/SKILL.md b/.agents/skills/update-changelog/SKILL.md index c83ab8d1f7..87c493e7c4 100644 --- a/.agents/skills/update-changelog/SKILL.md +++ b/.agents/skills/update-changelog/SKILL.md @@ -31,7 +31,7 @@ memory; it does not repeat it. layout, entry rules, the `## Breaking change note` comment contract, the advisory (`:lock:`) contract, pre-flight checks, anomaly types, and pitfalls. 2. `mem:scripts/changelog` — `boom`, `security`, `check-merged`, `cross-ref`, - `report`. + `dev-only-bugs`, `report`. 3. `mem:scripts/gh` — `issues`, `issue`, `comments`, `prs`, `advisories`. ## Commands @@ -39,6 +39,8 @@ memory; it does not repeat it. ```bash python3 scripts/gh.py issues "" --exclude "release blocker,no changelog" python3 scripts/gh.py advisories --patched-version "" +python3 scripts/changelog.py dev-only-bugs "" # list bugs that never reached production +python3 scripts/changelog.py dev-only-bugs "" --apply # label the sure ones `no changelog` python3 scripts/changelog.py check-merged python3 scripts/changelog.py boom "" # print the :boom: section python3 scripts/changelog.py boom "" --write # replace it in CHANGES.md @@ -54,4 +56,5 @@ python3 scripts/changelog.py report "" - Only closed, non-excluded milestone issues are included; the issue is the changelog unit and the PR is inline. - `:boom:` and `:rocket:` are label-generated and `:lock:` is advisory-generated: never hand-edit them; edit the label, the `## Breaking change note` comment or the advisory and re-run the command. - Verify PR merge status before writing; only merged PRs are referenced. +- Bugs that never reached production stay out: `dev-only-bugs --apply` labels the sure ones; candidates are the user's call, never label them yourself. - Fill `patched_versions` in the advisories before running `security`; unpublished advisories are valid. diff --git a/.gitignore b/.gitignore index 5e38fb2361..974b11f495 100644 --- a/.gitignore +++ b/.gitignore @@ -121,7 +121,7 @@ opencode.json /.opencode/reports /.opencode/prompts /.ci-logs -/tmp/ +/tmp /.codex/ /scripts/__pycache__/ /tools/__pycache__ diff --git a/.serena/memories/backend/testing.md b/.serena/memories/backend/testing.md index e06f9ace32..62bd36173d 100644 --- a/.serena/memories/backend/testing.md +++ b/.serena/memories/backend/testing.md @@ -3,6 +3,8 @@ JVM `clojure.test` (kaocha runner) under `backend/test/backend_tests/`. - READ `mem:testing` FIRST — it defines the execution discipline (no piping, tee to file, preferred commands) that applies to all JVM test runs. +- Test DB/Redis come from the `PENPOT_TEST_*` env vars already set in the shell (`PENPOT_TEST_DATABASE_URI`, `PENPOT_TEST_REDIS_URI`, ...); per instance they point at `penpot_test` / `penpot_test_wsN` and valkey DB `6+N` (`mem:devenv/core`). Do not override them with values copied from CI logs or other sources unless the user asks (e.g. "use a temporary database"); the CI env (`redis://redis/1`, `postgresql://postgres/penpot_test`) only works inside the CI docker network. +- The test database may have migrations applied from newer versions (it is shared dev infra). Reinitializing it — `dropdb`/`createdb` of the `PENPOT_TEST_*` database only — is explicitly permitted when that blocks a test run; never drop the main `penpot` database or another instance's test database. - All CLI commands must be executed from the `backend/` subdirectory. - Tests are invoked directly via `clojure -M:dev:test` (kaocha) — there is no pnpm wrapper. Kaocha auto-discovers test namespaces, so no runner registration is needed. - **Preferred way to run them: `scripts/ci --test`**, which runs lint, tests and format per module and writes a log per task under `.ci-logs/`. It exports `PENPOT_FLAGS` with `enable-backend-asserts` appended, so use it rather than calling `clojure -M:dev:test` by hand. diff --git a/.serena/memories/clojure/logging.md b/.serena/memories/clojure/logging.md new file mode 100644 index 0000000000..56fba6a1b3 --- /dev/null +++ b/.serena/memories/clojure/logging.md @@ -0,0 +1,7 @@ +# Clojure Logging Conventions + +Structured logging through `app.common.logging` (`l/trc`/`l/dbg`/`l/info`/`l/wrn`/`l/error`/`l/fatal`): `:hint` carries the human-readable message, every other datum goes in its own keyword field (`:id`, `:container`, `:removed`, ...). + +- `:hint` values start lowercase: `"fix dangling children"`, not `"Fix dangling children"`. Uppercase-first hints exist as legacy (~57 occurrences, the densest in `backend/src`; also `l/trc` in migration `legacy-28` and `log/error` in `app.common.files.repair`) — do not copy them, and do not mass-rewrite them unless the surrounding code needs the change. +- A name that requires uppercase (an env var like `PENPOT_MEDIA_PROCESSING_SERVICE_URI`, an acronym like `MCP` or `LDAP`) stays uppercase inside an otherwise lowercase sentence. +- Repair work done by a file migration logs at `l/trc`, only when it changes something, and with the ids it touched: the house style of `migrations.cljc` (`legacy-28` precedes; the file has no `warn`/`info` lines). The shipped `log4j2.xml` pins the `app` logger to `info` (static file, `monitorInterval="60"`), so those trace lines appear only after the operator raises the level; the durable record of a repaired file is the migration id in its `:migrations`. diff --git a/.serena/memories/critical-info.md b/.serena/memories/critical-info.md index f1c5a2ba4a..ca15de08db 100644 --- a/.serena/memories/critical-info.md +++ b/.serena/memories/critical-info.md @@ -6,6 +6,7 @@ You are working on the GitHub project `penpot/penpot`, a monorepo. - A section's top-level memory is `
/core`. When a section is relevant, read the core memory before focused memories. - Edits/stale refs/duplication cleanup: `mem:memory-maintenance`. - Cross-cutting testing principles, TDD workflow, and anti-patterns: `mem:testing`. +- Structured logging conventions (`:hint` casing and fields, log levels): `mem:clojure/logging` — read before adding or changing any log call. - Verified Clojure language behaviors that contradict common assumptions (e.g. `int?` covers `Long`; `integer?` is the general predicate): `mem:clojure/idioms` — read before assuming stdlib predicate semantics. - Read `mem:clojure/design-rules` (function shape, boundary checks, naming) before writing or modifying Clojure/ClojureScript code — not only when designing a solution or an API. - Promises: which call style to use where (`fmap`/`mcat`/`merr` vs `then`/`catch`) and how they bridge to rx: `mem:clojure/promesa` — read before chaining promises or mixing them with observables. diff --git a/.serena/memories/scripts/changelog.md b/.serena/memories/scripts/changelog.md index 7134377fa5..6785692d82 100644 --- a/.serena/memories/scripts/changelog.md +++ b/.serena/memories/scripts/changelog.md @@ -8,7 +8,8 @@ - `boom [--changes CHANGES.md --write]` — generate `### :boom: Breaking changes & Deprecations` from the closed, non-excluded milestone issues labelled `breaking change`, sorted by issue number. Each issue becomes a `#### ` block: content = body of the issue's last `## Breaking change note` comment (whole markdown, headings demoted to `#####` outside code fences; fallback = heading plus refs with a stderr warning), refs = issue link plus merged closing PRs on their own line, with `(by @author)` for `community contribution`. `--write` replaces only that subsection of the version (inserts it after the version header when missing, keeps the heading when there are no entries; idempotent). Print-only by default. Uses `gh.py comments`. - `security [--changes CHANGES.md --write]` — generate `### :lock: Security fixes` from the advisories whose `patched_versions` include the version (via `gh.py advisories --patched-version`), sorted by GHSA id. Entry: `- Fix () (PR: [#N](...))`; description = `Fix` + advisory summary, first word decapitalized unless it starts with an acronym; PRs = merged milestone PRs labelled `pr pt` whose body contains the GHSA id, comma-separated, and no PR is legal. Warns on stderr when a `pr pt` PR references an advisory with no matching `patched_versions`. `--write` replaces only that subsection, placed after `:rocket:` (or `:boom:`, or the version header when neither exists); empty entries remove the subsection; idempotent. Print-only by default. - `cross-ref [--changes CHANGES.md]` — fetch milestone PRs with `--state all` once, report merged PRs missing from the version section plus CLOSED (unmerged) warnings. PR links inside `:lock:` entries count as referenced; advisory URLs never appear here by design. -- `report [--changes CHANGES.md --output CHANGES-ISSUES.md]` — overwrite the anomaly report: A/B milestone mismatches, E `:boom:` entry without the label, F `:rocket:` sync, G `:boom:` sync, plus the C/D `:rocket:` gaps (warnings). `:boom:`/`:rocket:` rule violations are fixed by regenerating or relabelling, not by editing entries in place. +- `dev-only-bugs [--apply]` — list closed milestone bugs that never reached production, without `no changelog`. Sure: parent issue is a `Feature`/`Enhancement` of the same milestone; `--apply` adds `no changelog` to these only. Candidates (never labelled automatically): parent is an epic of the milestone, `regression` label, or the "Affected versions" / "Environment" body field names only hourly, develop, staging, PRE or this milestone (`only_unreleased_envs`; any production hint or other version keeps the bug in). Reads `body` and `parent` from one batched `gh.py issue` lookup. +- `report [--changes CHANGES.md --output CHANGES-ISSUES.md]` — overwrite the anomaly report: A/B milestone mismatches, E `:boom:` entry without the label, F `:rocket:` sync, G `:boom:` sync, plus the C/D `:rocket:` gaps and the `dev-only-bugs` lists (warnings). `:boom:`/`:rocket:` rule violations are fixed by regenerating or relabelling, not by editing entries in place. ## Conventions diff --git a/.serena/memories/scripts/gh.md b/.serena/memories/scripts/gh.md index c2a10d8f6b..6b546e578c 100644 --- a/.serena/memories/scripts/gh.md +++ b/.serena/memories/scripts/gh.md @@ -81,7 +81,7 @@ python3 scripts/gh.py prs --milestone "2.16.0" --state all Fetch issue details by number in batches of 50 (same input styles as `prs`: numbers, `--file`, `--stdin`). No filters. A dead number fails its whole batch (unlike PRs, issues error instead of resolving to null), so the batch falls back to one-by-one lookups and reports misses as `{"number", "error": "not_found"}`. Any other failure aborts. -**Output**: JSON array to stdout; progress to stderr. +**Output**: JSON array to stdout; progress to stderr. Each issue carries the shared issue fields plus `body` and `parent` (`number`, `issue_type`, `milestone`, or null); `changelog.py dev-only-bugs` reads both. ### `comments` diff --git a/.serena/memories/workflow/updating-changelog.md b/.serena/memories/workflow/updating-changelog.md index 54ca604790..6cf4134d1a 100644 --- a/.serena/memories/workflow/updating-changelog.md +++ b/.serena/memories/workflow/updating-changelog.md @@ -24,11 +24,12 @@ Editing `CHANGES.md` happens only on explicit request or as part of the release | Is it a breaking change? | `breaking change` label | | Is it a highlight? | `release highlight` label | -- **Issue Type decides the section**: `Bug` → `:bug:`, `Feature`/`Enhancement` → `:sparkles:`, `Task` is excluded (internal chores). Labels `bug`/`enhancement` and title emoji prefixes are often wrong or missing; do not use them. +- **Issue Type decides the section**: `Feature` → `:tada:`, `Enhancement` → `:sparkles:`, `Bug` → `:bug:`, `Task` is excluded (internal chores). Labels `bug`/`enhancement` and title emoji prefixes are often wrong or missing; do not use them. - **Security fixes come from advisories**: an advisory whose `patched_versions` include the version becomes a `:lock:` entry (section 7). `triage`, `draft` and `published` are all valid states; the advisory may still be unpublished when the changelog is written. A security fix may also carry a regular bug issue; both entries are legal and independent (the issue lives in `:bug:` as any other bug). - **Two labels override the Issue Type**: `breaking change` → an entry only under `:boom:`, never in its regular section; `release highlight` → an extra entry under `:rocket:` on top of its regular entry. - **Only closed issues**. An open issue in the milestone is not generated, even if someone added a provisional entry by hand before the merge. - **Excluded issues**: `no changelog`, `release blocker`, `Task` issue type, `Rejected` project status. +- **Bugs that never reached production** stay out: a bug found and fixed while building a feature of the same, unreleased version never affected users. The team rarely labels them, so the flow finds them with `dev-only-bugs` and labels them `no changelog` (section 8, step 3). - **Excluded PRs**: a PR with `release blocker` or `no issue required` does not bring an entry of its own. - **An issue with no milestone** belongs to another (probably private) project: it is neither changelog candidate nor anomaly. @@ -39,7 +40,7 @@ The newest version goes at the top, right after the `# CHANGELOG` header. Inside ```markdown # CHANGELOG -## 2.19.0 (Unreleased) +## 2.20.0 (Unreleased) ### :boom: Breaking changes & Deprecations @@ -57,21 +58,27 @@ The newest version goes at the top, right after the `# CHANGELOG` header. Inside - Fix () (PR: [#PR](https://github.com/penpot/penpot/pull/PR)) -### :bug: Bugs fixed - -- Fix [#ISSUE](https://github.com/penpot/penpot/issues/ISSUE) (PR: [#PR](https://github.com/penpot/penpot/pull/PR)) - -### :sparkles: New features & Enhancements +### :tada: Features - Add [#ISSUE](https://github.com/penpot/penpot/issues/ISSUE) (PR: [#PR](https://github.com/penpot/penpot/pull/PR)) + +### :sparkles: Enhancements + +- Improve [#ISSUE](https://github.com/penpot/penpot/issues/ISSUE) (PR: [#PR](https://github.com/penpot/penpot/pull/PR)) + +### :bug: Bugs + +- Fix [#ISSUE](https://github.com/penpot/penpot/issues/ISSUE) (PR: [#PR](https://github.com/penpot/penpot/pull/PR)) ``` -- `:boom:` first, then `:rocket:`, `:lock:`, `:bug:`, `:sparkles:`. Other headings exist in old versions (`:heart:` community contributions); do not add them to new versions. +- `:boom:` first, then `:rocket:`, `:lock:`, `:tada:`, `:sparkles:`, `:bug:`. Features come before enhancements so they stand out to readers deciding whether to update. +- **This layout applies from 2.20.0.** Versions up to 2.19.x use the old one: `:bug: Bugs fixed` before `:sparkles: New features & Enhancements`, which mixes features and enhancements. Keep the old layout in those sections, including new 2.19.x patches, so a minor line reads the same. The in-app update dialog only reads `:rocket:`, so the change does not affect it. +- Other headings exist in old versions (`:heart:` community contributions); do not add them to new versions. - Include a section only when it has entries. Historical versions may have an empty `:boom:` or `:rocket:` heading; `boom --write` keeps the `:boom:` heading even when there are no entries. `:lock:` is omitted (and removed on `--write`) when the version has no security advisories. - One blank line between the last entry of a section and the next heading. - No manual line wrapping: one line per bullet or paragraph, however long. -## 4. Regular entries (`:bug:`, `:sparkles:`) +## 4. Regular entries (`:tada:`, `:sparkles:`, `:bug:`) The entry shape is: @@ -153,15 +160,18 @@ Generated from the advisories that declare the version in `patched_versions`; ne 1. **Confirm the version** (a semver string such as `2.19.0`) with the user when it is not obvious. 2. **Fetch the milestone issues**: `python3 scripts/gh.py issues "" --exclude "release blocker,no changelog"` (closed by default; add `--state all` only to inspect open issues). `gh.py` already drops `Task` and `Rejected`; pass `--include-tasks` / `--include-rejected` only to inspect them. To find what is missing from an existing section, add `--compare CHANGES.md`. -3. **Fetch PR details when needed**: `python3 scripts/gh.py prs ` (also `--milestone`). Use it for the PR author, labels and merge state. -4. **Categorize** by Issue Type and the two labels (section 2), and **build the descriptions** (section 4). -5. **Verify PR merge status** before writing: `python3 scripts/changelog.py check-merged `. Only merged PRs go in the changelog; if a closing PR was closed unmerged, look for the PR that superseded it. -6. **Reconcile the existing text** (section 9). -7. **Generate `:boom:`**: `python3 scripts/changelog.py boom ""` prints the section; add `--write` to replace it in `CHANGES.md`. The write replaces only that subsection, inserts it after the version header when missing, keeps the heading when there are no entries, and is idempotent. -8. **Generate `:rocket:`** from the `release highlight` label (section 6). -9. **Generate `:lock:`**: `python3 scripts/changelog.py security ""` prints the section; add `--write` to replace it in `CHANGES.md`. The contract lives in section 7. -10. **Cross-check PRs**: `python3 scripts/changelog.py cross-ref ""`. It lists merged milestone PRs missing from the section and warns about closed (unmerged) PRs. PRs linked from `:lock:` count as referenced; an advisory with no PR is invisible here by design. -11. **Generate the anomaly report**: `python3 scripts/changelog.py report ""`. It overwrites `CHANGES-ISSUES.md` with the current state (section 10). +3. **Keep out bugs that never reached production**: `python3 scripts/changelog.py dev-only-bugs ""` lists them; add `--apply` to label the sure ones `no changelog` on GitHub, then re-fetch step 2. The label puts the decision on the issue, so it survives a regeneration. + - **Sure**: a bug whose parent issue is a `Feature` or `Enhancement` of the same milestone. `--apply` labels these. + - **Candidates**: a bug whose parent is an epic of the same milestone, that carries the `regression` label, or whose "Affected versions" / "Environment" field names only hourly, develop, staging, PRE or a version of this milestone. `regression` also marks regressions users hit in production, and the field says where the bug was seen, not since when it exists, so never label these yourself: show them to the user, who labels the ones they confirm. The anomaly report lists them too (section 10). +4. **Fetch PR details when needed**: `python3 scripts/gh.py prs ` (also `--milestone`). Use it for the PR author, labels and merge state. +5. **Categorize** by Issue Type and the two labels (section 2), and **build the descriptions** (section 4). +6. **Verify PR merge status** before writing: `python3 scripts/changelog.py check-merged `. Only merged PRs go in the changelog; if a closing PR was closed unmerged, look for the PR that superseded it. +7. **Reconcile the existing text** (section 9). +8. **Generate `:boom:`**: `python3 scripts/changelog.py boom ""` prints the section; add `--write` to replace it in `CHANGES.md`. The write replaces only that subsection, inserts it after the version header when missing, keeps the heading when there are no entries, and is idempotent. +9. **Generate `:rocket:`** from the `release highlight` label (section 6). +10. **Generate `:lock:`**: `python3 scripts/changelog.py security ""` prints the section; add `--write` to replace it in `CHANGES.md`. The contract lives in section 7. +11. **Cross-check PRs**: `python3 scripts/changelog.py cross-ref ""`. It lists merged milestone PRs missing from the section and warns about closed (unmerged) PRs. PRs linked from `:lock:` count as referenced; an advisory with no PR is invisible here by design. +12. **Generate the anomaly report**: `python3 scripts/changelog.py report ""`. It overwrites `CHANGES-ISSUES.md` with the current state (section 10). ## 9. Pre-flight: reconcile before writing @@ -172,7 +182,7 @@ Re-fetch the milestone (do not trust cached data) and fix these in `CHANGES.md` 3. **Newly applied exclusion labels** (`no changelog`, `release blocker`) → remove the entry. 4. **Issue no longer closed, deleted, or now `Rejected`** → remove the entry. 5. **Unmerged or moved PR reference** → fix the reference or remove the entry. A PR merged in a different milestone is an anomaly (section 10); do not remove it silently. -6. **Issue Type changed to `Task`** → remove the entry. +6. **Issue Type changed** → to `Task`: remove the entry; to another type: move the entry to that type's section. 7. **Breaking change out of sync** (a `breaking change` issue outside `:boom:`, or a `:boom:` entry without the label) → regenerate the section; a labelled issue has no regular entry. 8. **Highlight out of sync** (`:rocket:` entry without the label, or a labelled issue missing) → remove it or add it. 9. **Missing valid issues** (closed, non-excluded, referenced nowhere) → add them to their section. @@ -189,6 +199,8 @@ Re-fetch the milestone (do not trust cached data) and fix these in `CHANGES.md` - **C — released X.Y.0 version without a `:rocket:` section** (gap; patches never carry highlights). - **D — `:rocket:` entry without issue AND PR references** (gap). +**Bugs that may never have reached production** are warnings too (section 8, step 3): sub-issues of a milestone feature still unlabelled (run `dev-only-bugs --apply`) and candidates for the user to confirm. They never count toward the total. + Anything else is a rule violation, not a report item: fix it in the pre-flight and re-run. ## 11. Pitfalls @@ -203,6 +215,6 @@ Anything else is a rule violation, not a report item: fix it in the pre-flight a ## 12. Where the code is - `scripts/gh.py`: `issues`, `issue`, `comments`, `prs`, `link-issue`, `advisories` (with `--patched-version`). Details: `mem:scripts/gh`. -- `scripts/changelog.py`: `check-merged`, `boom`, `security`, `cross-ref`, `report`. Details: `mem:scripts/changelog`. +- `scripts/changelog.py`: `check-merged`, `boom`, `security`, `cross-ref`, `dev-only-bugs`, `report`. Details: `mem:scripts/changelog`. - Tests: `scripts/test_gh.py`, `scripts/test_changelog.py`. - The note contract for issue authors: `mem:workflow/creating-issues`, section **Breaking changes**. diff --git a/backend/test/backend_tests/climit_test.clj b/backend/test/backend_tests/climit_test.clj index 090ae4d6b8..d65b2a500a 100644 --- a/backend/test/backend_tests/climit_test.clj +++ b/backend/test/backend_tests/climit_test.clj @@ -445,10 +445,9 @@ (t/is (thrown? clojure.lang.ExceptionInfo (invoke! cfg (fn [_ _] :never) {})))) - ;; the count came back down instead of staying up for good. Only the - ;; last publication is asserted, not the exact list: the sampler - ;; republishes the same count when its first tick lands inside the - ;; window. + ;; the count came back down instead of staying up for good. The + ;; republisher daemon thread can add publications inside the window, + ;; so what is asserted is the last value, not the whole sequence. (t/is (= 0 (waiting-count climit name))) (t/is (= 0 (waiting-last name))) diff --git a/common/src/app/common/files/migrations.cljc b/common/src/app/common/files/migrations.cljc index 7033bfcb7d..1d935272d8 100644 --- a/common/src/app/common/files/migrations.cljc +++ b/common/src/app/common/files/migrations.cljc @@ -2233,44 +2233,49 @@ ;; been added to enforce this from now on. ;; Main instances in variants are a special case because they need to match the variant name. (letfn [(update-shape [container shape-id] - (let [shape (ctn/get-shape container shape-id) + (let [shape (ctn/get-shape container shape-id)] + ;; A shape may reference a child that does not exist in the + ;; container (a dangling reference). There is nothing to update + ;; on it, and storing it would add a nil entry to the objects + ;; map, making the whole file invalid. + (if (nil? shape) + container + (let [shape (cond + ;; Variant container must have a normalized name + (ctk/is-variant-container? shape) + (let [name (:name shape) + clean-name (cpn/clean-path name) + clean-name (if (str/blank? clean-name) + "(empty)" ;; this may occur if the name contains only separators + clean-name)] + (if (= name clean-name) + shape + (assoc shape :name clean-name))) - shape (cond - ;; Variant container must have a normalized name - (ctk/is-variant-container? shape) - (let [name (:name shape) - clean-name (cpn/clean-path name) - clean-name (if (str/blank? clean-name) - "(empty)" ;; this may occur if the name contains only separators - clean-name)] - (if (= name clean-name) - shape - (assoc shape :name clean-name))) + ;; Main instance inside a variant must have the same name as the container + (ctk/main-instance? shape) + (if (ctk/is-variant? shape) + (let [variant-container (ctn/get-shape container (:parent-id shape))] + (if (or (= (:name shape) (:name variant-container)) + (nil? variant-container)) + shape + (assoc shape :name (:name variant-container)))) - ;; Main instance inside a variant must have the same name as the container - (ctk/main-instance? shape) - (if (ctk/is-variant? shape) - (let [variant-container (ctn/get-shape container (:parent-id shape))] - (if (or (= (:name shape) (:name variant-container)) - (nil? variant-container)) - shape - (assoc shape :name (:name variant-container)))) + ;; Other main instances must have the same name as the path+name of the component + (let [component (ctkl/get-component data (:component-id shape) true) + expected-name (when component + (cpn/merge-path-item (:path component) (:name component)))] + (if (or (= expected-name (:name shape)) + (nil? expected-name)) + shape + (assoc shape :name expected-name)))) - ;; Other main instances must have the same name as the path+name of the component - (let [component (ctkl/get-component data (:component-id shape) true) - expected-name (when component - (cpn/merge-path-item (:path component) (:name component)))] - (if (or (= expected-name (:name shape)) - (nil? expected-name)) - shape - (assoc shape :name expected-name)))) + ;; Any other shape does not need processing + :else shape)] - ;; Any other shape does not need processing - :else shape)] - - (reduce update-shape - (ctt/set-shape container shape) - (:shapes shape)))) + (reduce update-shape + (ctt/set-shape container shape) + (:shapes shape)))))) (update-container [container] @@ -2322,6 +2327,35 @@ (update :pages-index d/update-vals update-container) (d/update-when :components d/update-vals update-container))))) +(defmethod migrate-data "0035-fix-dangling-children" + [data _] + ;; A shape may list a child id that does not exist in the objects map of + ;; the page or component that owns it (a dangling reference). Such a + ;; reference is invalid: any operation that walks the shape tree through + ;; the children lists trips on it, and the file cannot be migrated nor + ;; opened. Here we remove the dangling ids from the children lists, + ;; keeping every shape that really exists. + (letfn [(fix-shape [container-id objects shape] + (d/update-when shape :shapes + (fn [shapes] + (let [shapes' (filterv #(contains? objects %) shapes)] + (when (not= shapes shapes') + (l/trc :hint "fix dangling children" + :container container-id + :shape-id (:id shape) + :removed (remove #(contains? objects %) shapes))) + shapes')))) + + (fix-objects [container-id objects] + (update-vals objects (partial fix-shape container-id objects))) + + (update-container [container] + (d/update-when container :objects #(fix-objects (:id container) %)))] + + (-> data + (update :pages-index d/update-vals update-container) + (d/update-when :components d/update-vals update-container)))) + (defmethod migrate-data "0035-fix-shape-svg-attrs-and-defs" ;; Stored svg attribute keys are kebab-case (issue #11947): the v3 reader ;; already outputs kebab and birth normalizes any outside spelling, so @@ -2459,4 +2493,5 @@ "0032-fix-children-not-in-parent" "0033-normalize-main-instance-names" "0034-fix-path-content-offset" + "0035-fix-dangling-children" "0035-fix-shape-svg-attrs-and-defs"])) diff --git a/common/src/app/common/files/variant.cljc b/common/src/app/common/files/variant.cljc index 989a80a58b..802c90fd1c 100644 --- a/common/src/app/common/files/variant.cljc +++ b/common/src/app/common/files/variant.cljc @@ -28,6 +28,25 @@ (map #(ctcl/get-component data % true)) reverse))) +(defn collect-variant-properties + "Map of component-id → variant-properties for clipboard copy. + Variant container: every related component. Variant main: that + component only. Anything else: empty map." + [data objects shape] + (cond + (ctc/is-variant-container? shape) + (into {} + (map (juxt :id :variant-properties)) + (find-variant-components data objects (:id shape))) + + (ctc/is-variant-main? shape) + (if-let [component (ctcl/get-component data (:component-id shape) true)] + {(:id component) (:variant-properties component)} + {}) + + :else + {})) + (defn extract-properties-names [shape data] (->> shape diff --git a/common/src/app/common/logic/libraries.cljc b/common/src/app/common/logic/libraries.cljc index 1b928d407b..1554de71e5 100644 --- a/common/src/app/common/logic/libraries.cljc +++ b/common/src/app/common/logic/libraries.cljc @@ -2766,7 +2766,8 @@ page-id nil variant-id - props))] + props + {:apply-changes-local-library? true}))] [root-shape changes])) (defn generate-add-component @@ -3282,6 +3283,25 @@ (and is-component-root? is-component-main?) (regenerate-component new-obj)) + ;; Unknown-component paste into a variant set: align props/names + ;; after regenerate. Reload the shape so we use the new component-id. + ;; Prefit the component name/path to the container so + ;; create-new-properties-from-variant does not treat the component + ;; full name as a former set name and inflate properties (add-name?). + changes (cond-> changes + (and is-component-root? + is-component-main? + (ctk/is-variant-container? parent)) + (as-> $ + (let [shape (get (pcb/get-objects $) (:id new-obj)) + [cpath cname] (cpn/split-group-name (:name parent)) + changes (pcb/update-component + $ + (:component-id shape) + #(assoc % :name cname :path cpath) + {:apply-changes-local-library? true})] + (clvp/generate-make-shapes-variant changes [shape] parent)))) + ;; This is needed for the recursive call to find the new object as parent page' (ctst/add-shape (:id new-obj) new-obj diff --git a/common/src/app/common/logic/variant_properties.cljc b/common/src/app/common/logic/variant_properties.cljc index 02490a3c81..141cbd2279 100644 --- a/common/src/app/common/logic/variant_properties.cljc +++ b/common/src/app/common/logic/variant_properties.cljc @@ -266,10 +266,20 @@ (first (reduce (fn [[changes used-values] shape] - (let [component (ctcl/get-component data (:component-id shape) true)] - (if (or (zero? num-shapes) ;; do nothing if there are no shapes - (and (= variant-id (:variant-id shape)) ;; or we are only moving the shape inside its parent (it is - (not (:deleted component)))) ;; the same parent and the component isn't deleted) + (let [component (ctcl/get-component data (:component-id shape) true) + already-here? (= variant-id (:variant-id shape)) + prop-names (cfv/extract-properties-names shape data) + base-names (map :name base-props) + props-ok? (= prop-names base-names) + names-ok? (= (cpn/merge-path-item (:path component) (:name component)) + container-name) + variant-ok? (= (:variant-name shape) + (ctv/properties-to-name (:variant-properties component))) + aligned? (and props-ok? names-ok? variant-ok?)] + (if (or (zero? num-shapes) ;; do nothing if there are no shapes + (and already-here? ;; or already a fully aligned child of this container + aligned? + (not (:deleted component)))) [changes used-values] (let [props (-> (create-new-properties shape total-props) (fill-blank-properties used-values)) diff --git a/common/test/common_tests/files/variant_test.cljc b/common/test/common_tests/files/variant_test.cljc new file mode 100644 index 0000000000..fda97d565e --- /dev/null +++ b/common/test/common_tests/files/variant_test.cljc @@ -0,0 +1,49 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns common-tests.files.variant-test + (:require + [app.common.files.variant :as cfv] + [app.common.test-helpers.components :as thc] + [app.common.test-helpers.files :as thf] + [app.common.test-helpers.ids-map :as thi] + [app.common.test-helpers.shapes :as ths] + [app.common.test-helpers.variants :as thv] + [clojure.test :as t])) + +(t/use-fixtures :each thi/test-fixture) + +(t/deftest test-collect-variant-properties-for-main + (let [file (-> (thf/sample-file :file1) + (thv/add-variant :v01 :c01 :m01 :c02 :m02)) + page (thf/current-page file) + main (ths/get-shape file :m01) + comp (thc/get-component file :c01) + collected (cfv/collect-variant-properties (:data file) (:objects page) main)] + + (t/is (= {(:id comp) (:variant-properties comp)} collected)))) + +(t/deftest test-collect-variant-properties-for-container + (let [file (-> (thf/sample-file :file1) + (thv/add-variant :v01 :c01 :m01 :c02 :m02)) + page (thf/current-page file) + container (ths/get-shape file :v01) + c01 (thc/get-component file :c01) + c02 (thc/get-component file :c02) + collected (cfv/collect-variant-properties (:data file) (:objects page) container)] + + (t/is (= {(:id c01) (:variant-properties c01) + (:id c02) (:variant-properties c02)} + collected)))) + +(t/deftest test-collect-variant-properties-for-plain-shape + (let [file (-> (thf/sample-file :file1) + (ths/add-sample-shape :rect1 :type :rect)) + page (thf/current-page file) + shape (ths/get-shape file :rect1) + collected (cfv/collect-variant-properties (:data file) (:objects page) shape)] + + (t/is (= {} collected)))) diff --git a/common/test/common_tests/files_migrations_0033_test.cljc b/common/test/common_tests/files_migrations_0033_test.cljc new file mode 100644 index 0000000000..6f56602b77 --- /dev/null +++ b/common/test/common_tests/files_migrations_0033_test.cljc @@ -0,0 +1,138 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns common-tests.files-migrations-0033-test + "Regression coverage for the shape tree traversal of the + `0033-normalize-main-instance-names` migration. + + The migration walks a container from its root down, following the + `:shapes` vectors. A file can hold a `:shapes` entry pointing to a + shape that is not in the `:objects` map (a dangling reference). The + traversal must skip it instead of storing a nil shape into the + `:objects` map, and must keep normalizing the names of the shapes + that do exist." + (:require + [app.common.files.migrations :as cfm] + [app.common.types.shape :as cts] + [app.common.uuid :as uuid] + [clojure.test :as t])) + +(def ^:private migration "0033-normalize-main-instance-names") + +(defn- make-root + [children] + (cts/setup-shape {:id uuid/zero + :type :frame + :name "Root" + :shapes children})) + +(defn- make-main-instance + "A frame declared as the main instance of a component." + [main-id file-id component-id name] + (-> (cts/setup-shape {:id main-id :type :frame :name name}) + (assoc :main-instance true + :component-id component-id + :component-file file-id))) + +(defn- page-data + "File data with a single page owning `objects`." + [page-id objects] + {:pages-index {page-id {:id page-id :objects objects}}}) + +(defn- component-data + "File data with a single component owning `objects`." + [component-id objects] + {:components {component-id {:id component-id :objects objects}}}) + +;; --- The guard: a missing child must not become a nil shape + +(t/deftest missing-child-of-a-page-does-not-create-a-nil-shape + (let [page-id (uuid/next) + child-id (uuid/next) + root (make-root [child-id]) + data (page-data page-id {uuid/zero root}) + migrated (cfm/migrate-data data migration) + objects' (get-in migrated [:pages-index page-id :objects])] + + (t/is (not (contains? objects' nil)) + "no nil key is added to the objects map") + (t/is (= 1 (count objects')) + "the objects map keeps only the shapes that exist") + (t/is (= root (get objects' uuid/zero)) + "the root is left untouched"))) + +(t/deftest missing-child-of-a-component-does-not-create-a-nil-shape + (let [component-id (uuid/next) + child-id (uuid/next) + root (make-root [child-id]) + data (component-data component-id {uuid/zero root}) + migrated (cfm/migrate-data data migration) + objects' (get-in migrated [:components component-id :objects])] + + (t/is (not (contains? objects' nil)) + "no nil key is added to the objects map") + (t/is (= 1 (count objects')) + "the objects map keeps only the shapes that exist") + (t/is (= root (get objects' uuid/zero)) + "the component root is left untouched"))) + +(t/deftest missing-child-deep-in-the-tree-does-not-create-a-nil-shape + (let [page-id (uuid/next) + frame-id (uuid/next) + missing-id (uuid/next) + frame (-> (cts/setup-shape {:id frame-id :type :frame :name "Frame"}) + (assoc :shapes [missing-id])) + data (page-data page-id {uuid/zero (make-root [frame-id]) + frame-id frame}) + migrated (cfm/migrate-data data migration) + objects' (get-in migrated [:pages-index page-id :objects])] + + (t/is (not (contains? objects' nil)) + "no nil key is added to the objects map") + (t/is (= frame (get objects' frame-id)) + "the parent of the missing child is left untouched"))) + +;; --- The behavior: names keep being normalized next to a missing child + +(t/deftest main-instance-of-a-page-is-renamed-next-to-a-missing-child + (let [file-id (uuid/next) + page-id (uuid/next) + component-id (uuid/next) + main-id (uuid/next) + missing-id (uuid/next) + main-shape (make-main-instance main-id file-id component-id "Group/Name") + data (assoc (page-data page-id {uuid/zero (make-root [missing-id main-id]) + main-id main-shape}) + :components {component-id {:id component-id + :name "Name" + :path "Group" + :main-instance-id main-id}}) + migrated (cfm/migrate-data data migration) + objects' (get-in migrated [:pages-index page-id :objects])] + + (t/is (not (contains? objects' nil)) + "no nil key is added to the objects map") + (t/is (= "Group / Name" (get-in objects' [main-id :name])) + "the main instance takes the path and name of its component"))) + +(t/deftest main-instance-of-a-component-is-renamed-next-to-a-missing-child + (let [file-id (uuid/next) + component-id (uuid/next) + main-id (uuid/next) + missing-id (uuid/next) + main-shape (make-main-instance main-id file-id component-id "Group/Name") + data (component-data component-id {uuid/zero (make-root [missing-id main-id]) + main-id main-shape}) + data (assoc-in data [:components component-id :name] "Name") + data (assoc-in data [:components component-id :path] "Group") + data (assoc-in data [:components component-id :main-instance-id] main-id) + migrated (cfm/migrate-data data migration) + objects' (get-in migrated [:components component-id :objects])] + + (t/is (not (contains? objects' nil)) + "no nil key is added to the objects map") + (t/is (= "Group / Name" (get-in objects' [main-id :name])) + "the main instance takes the path and name of its component"))) diff --git a/common/test/common_tests/files_migrations_0035_test.cljc b/common/test/common_tests/files_migrations_0035_test.cljc index 8dd421d614..350b669fed 100644 --- a/common/test/common_tests/files_migrations_0035_test.cljc +++ b/common/test/common_tests/files_migrations_0035_test.cljc @@ -10,10 +10,186 @@ [app.common.geom.matrix :as gmt] [app.common.geom.rect :as grc] [app.common.types.file :as ctf] + [app.common.types.objects-map :as omap] [app.common.types.shape :as cts] [app.common.uuid :as uuid] [clojure.test :as t])) +;; ====================================================================== +;; 0035-fix-dangling-children +;; A shape can list a child id that is not present in the `:objects` map +;; of the page or component owning it (a dangling reference). Any +;; traversal of the shape tree trips on it, and the file becomes +;; unopenable. The migration drops those ids and keeps every shape that +;; really exists. +;; ====================================================================== + +(def ^:private migration "0035-fix-dangling-children") + +(defn- make-root + [children] + (cts/setup-shape {:id uuid/zero + :type :frame + :name "Root" + :shapes children})) + +(defn- make-frame + [frame-id child-ids] + (-> (cts/setup-shape {:id frame-id :type :frame :name "Frame"}) + (assoc :shapes child-ids))) + +(defn- page-data + [page-id objects] + {:pages-index {page-id {:id page-id :objects objects}}}) + +(defn- component-data + [component-id objects] + {:components {component-id {:id component-id :objects objects}}}) + +(t/deftest removes-the-dangling-children-of-a-page + (let [page-id (uuid/next) + frame-id (uuid/next) + child-id (uuid/next) + frame (make-frame frame-id [child-id]) + data (page-data page-id {uuid/zero (make-root [frame-id child-id]) + frame-id frame}) + migrated (cfm/migrate-data data migration) + objects' (get-in migrated [:pages-index page-id :objects])] + + (t/is (= [frame-id] (get-in objects' [uuid/zero :shapes])) + "the dangling id is removed from the root children list") + (t/is (= [] (get-in objects' [frame-id :shapes])) + "the dangling id is removed from the frame children list") + (t/is (= 2 (count objects')) + "every shape that exists is kept") + (t/is (some? (get objects' frame-id)) + "the frame itself is kept"))) + +(t/deftest removes-the-dangling-children-of-a-component + (let [component-id (uuid/next) + frame-id (uuid/next) + child-id (uuid/next) + frame (cts/setup-shape {:id frame-id :type :frame :name "Frame"}) + data (component-data component-id {uuid/zero (make-root [frame-id child-id]) + frame-id frame}) + migrated (cfm/migrate-data data migration) + objects' (get-in migrated [:components component-id :objects])] + + (t/is (= [frame-id] (get-in objects' [uuid/zero :shapes])) + "the dangling id is removed from the children list") + (t/is (= 2 (count objects')) + "every shape that exists is kept") + (t/is (= frame (get objects' frame-id)) + "the existing shape is left untouched"))) + +(t/deftest migration-works-with-an-objects-map + ;; The backend stores the page :objects as an ObjectsMap (the lazy + ;; representation enabled by the fdata/objects-map feature). It is an + ;; IPersistentMap that does not implement IFn on the JVM: a migration + ;; must consult it with contains?/get, never by calling it as a + ;; function. + (let [page-id (uuid/next) + frame-id (uuid/next) + child-id (uuid/next) + frame (make-frame frame-id [child-id]) + objects (omap/wrap {uuid/zero (make-root [frame-id child-id]) + frame-id frame}) + data (page-data page-id objects) + migrated (cfm/migrate-data data migration) + objects' (get-in migrated [:pages-index page-id :objects])] + + (t/is (= [frame-id] (get-in objects' [uuid/zero :shapes])) + "the dangling id is removed with the ObjectsMap representation") + (t/is (= [] (get-in objects' [frame-id :shapes])) + "the frame children list is pruned the same way"))) + +(t/deftest keeps-the-order-and-the-shapes-of-a-valid-tree + (let [page-id (uuid/next) + frame-id (uuid/next) + rect-id (uuid/next) + frame (cts/setup-shape {:id frame-id :type :frame :name "Frame"}) + rect (cts/setup-shape {:id rect-id :type :rect :name "Rect"}) + data (page-data page-id {uuid/zero (make-root [frame-id rect-id]) + frame-id frame + rect-id rect}) + migrated (cfm/migrate-data data migration) + objects' (get-in migrated [:pages-index page-id :objects])] + + (t/is (= [frame-id rect-id] (get-in objects' [uuid/zero :shapes])) + "the children list is kept as it was") + (t/is (= data migrated) + "a tree without dangling references is not modified"))) + +(t/deftest leaves-shapes-with-empty-children-alone + (let [page-id (uuid/next) + frame (cts/setup-shape {:id (uuid/next) :type :frame :name "Frame"}) + data (page-data page-id {uuid/zero (make-root []) + (:id frame) frame}) + migrated (cfm/migrate-data data migration)] + + (t/is (= data migrated) + "empty children vectors are left alone"))) + +(t/deftest leaves-shapes-without-children-key-alone + (let [page-id (uuid/next) + rect-id (uuid/next) + rect (cts/setup-shape {:id rect-id :type :rect :name "Rect"}) + data (page-data page-id {uuid/zero (make-root []) + rect-id rect}) + migrated (cfm/migrate-data data migration)] + + (t/is (= data migrated) + "shapes without a children key are left alone") + (t/is (not (contains? (get-in migrated [:pages-index page-id :objects rect-id]) :shapes)) + "no children key is created on the shape"))) + +(t/deftest migration-is-idempotent + (let [page-id (uuid/next) + child-id (uuid/next) + data (page-data page-id {uuid/zero (make-root [child-id])}) + migrated (cfm/migrate-data data migration)] + + (t/is (= migrated (cfm/migrate-data migrated migration)) + "applying the migration twice does not change the result"))) + +(t/deftest file-with-dangling-children-passes-the-whole-migration + (let [file-id (uuid/next) + page-id (uuid/next) + frame-id (uuid/next) + missing-id (uuid/next) + frame (make-frame frame-id [missing-id]) + base (ctf/make-file-data file-id page-id) + data (-> base + (assoc-in [:pages-index page-id :objects uuid/zero] + (make-root [frame-id missing-id])) + (assoc-in [:pages-index page-id :objects frame-id] frame)) + ;; The file reached the state before 0033 ran, so 0033, 0034 and + ;; 0035 are all pending, exactly as in the reported failure. + file {:id file-id + :data data + :migrations (disj cfm/available-migrations + "0033-normalize-main-instance-names" + "0034-fix-path-content-offset" + migration)}] + + (t/is (cfm/need-migration? file) + "the file is scheduled for migration") + + (let [migrated (cfm/migrate-file file {}) + objects' (get-in migrated [:data :pages-index page-id :objects])] + + (t/is (contains? (:migrations migrated) migration) + "the migration is recorded as applied") + (t/is (not (contains? objects' nil)) + "the migrated data has no nil shape entry") + (t/is (= [frame-id] (get-in objects' [uuid/zero :shapes])) + "the dangling reference of the root is removed") + (t/is (= [] (get-in objects' [frame-id :shapes])) + "the dangling reference of the frame is removed") + (t/is (some? (get objects' frame-id)) + "the frame that exists is kept")))) + +;; ====================================================================== ;; 0035-fix-shape-svg-attrs-and-defs ;; Stored svg attribute keys are kebab-case (issue #11947). Old birth and ;; the cleaner era stored camelCase React prop names, which the v3 reader @@ -22,6 +198,7 @@ ;; through the whitelist+kebab transform, svg-raw `:content` attrs are ;; kebab-ized spelling-only. It replaces "0021-fix-shape-svg-attrs" (id ;; withdrawn): files that recorded it migrate again through this one. +;; ====================================================================== (def ^:private migration-id "0035-fix-shape-svg-attrs-and-defs") diff --git a/common/test/common_tests/logic/variants_paste_test.cljc b/common/test/common_tests/logic/variants_paste_test.cljc new file mode 100644 index 0000000000..020b8dcd7e --- /dev/null +++ b/common/test/common_tests/logic/variants_paste_test.cljc @@ -0,0 +1,202 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns common-tests.logic.variants-paste-test + (:require + [app.common.data :as d] + [app.common.files.changes-builder :as pcb] + [app.common.files.helpers :as cfh] + [app.common.files.validate :as cfv] + [app.common.geom.point :as gpt] + [app.common.logic.libraries :as cll] + [app.common.logic.variant-properties :as clvp] + [app.common.test-helpers.components :as thc] + [app.common.test-helpers.compositions :as tho] + [app.common.test-helpers.files :as thf] + [app.common.test-helpers.ids-map :as thi] + [app.common.test-helpers.shapes :as ths] + [app.common.test-helpers.variants :as thv] + [app.common.types.component :as ctk] + [app.common.types.variant :as ctv] + [clojure.test :as t])) + +(t/use-fixtures :each thi/test-fixture) + +(defn- paste-variants-into-container + "Paste individual variants from file-a into file-b's variant container. + Omit file-a from `libraries` to hit the unknown-component path; include + it to hit `duplicate-variant`. Optional `variant-props` mirrors clipboard + collection (Task 2)." + [file-a file-b source-labels target-container-label + & {:keys [libraries variant-props] + :or {libraries {(:id file-b) file-b}}}] + (let [page-a (thf/current-page file-a) + page-b (thf/current-page file-b) + container-b (ths/get-shape file-b target-container-label) + parent-id (:id container-b) + + source-ids (mapv thi/id source-labels) + tree-ids (into [] + (mapcat #(cfh/get-children-ids-with-self (:objects page-a) %)) + source-ids) + + pasted (->> tree-ids + (map (d/getf (:objects page-a))) + (map (fn [shape] + (cond-> shape + (contains? (set source-ids) (:id shape)) + (assoc :parent-id parent-id + :frame-id parent-id)))) + (d/index-by :id)) + + all-objects (merge (:objects page-b) pasted)] + (-> (pcb/empty-changes nil) + (pcb/with-page-id (:id page-b)) + (pcb/with-library-data (:data file-b)) + (pcb/with-objects all-objects) + (cll/generate-duplicate-changes all-objects + page-b + (set source-ids) + (gpt/point 0 0) + libraries + (:data file-b) + (:id file-b) + {:variant-props variant-props})))) + +(defn- property-names + [component] + (mapv :name (:variant-properties component))) + +(defn- variant-mains-in + [file container-label] + (let [container (ths/get-shape file container-label)] + (->> (vals (:objects (thf/current-page file))) + (filter ctk/main-instance?) + (filter #(= (:id container) (:variant-id %)))))) + +(defn- other-set-file + [] + (-> (thf/sample-file :file-b) + (thv/add-variant-two-properties :vb :cb1 :mb1 :cb2 :mb2) + (ths/update-shape :vb :name "Other") + (ths/update-shape :mb1 :name "Other") + (ths/update-shape :mb2 :name "Other") + (thc/update-component :cb1 {:name "Other"}) + (thc/update-component :cb2 {:name "Other"}))) + +(t/deftest test-paste-unknown-variant-into-container-validates + ;; #10704: paste individual variants from file A into file B's set + ;; when A is not in libraries must leave a valid file (desired behavior). + (let [file-a (-> (thf/sample-file :file-a) + (thv/add-variant :va :ca1 :ma1 :ca2 :ma2)) + file-b (other-set-file) + changes (paste-variants-into-container file-a file-b [:ma1 :ma2] :vb) + file' (thf/apply-changes file-b changes :validate? false) + errors (cfv/validate-file file' {}) + pasted (->> (variant-mains-in file' :vb) + (remove #(#{(thi/id :mb1) (thi/id :mb2)} (:id %))))] + + (t/is (nil? errors)) + (doseq [shape pasted] + (let [comp (thc/get-component-by-id file' (:component-id shape))] + (t/is (= ["Property 1" "Property 2"] (property-names comp))) + (t/is (= "Other" (:name shape))) + (t/is (= (:variant-name shape) + (ctv/properties-to-name (:variant-properties comp)))))))) + +(t/deftest test-duplicate-variant-in-own-container-bumps-last-value + ;; Same-container duplicate via duplicate-variant: originals keep their + ;; props; the new main gets a bumped last-property value. Does not cover + ;; generate-make-shapes-variant's already-aligned early-exit (new mains + ;; enter with a nil variant-id). + (let [file (-> (thf/sample-file :file1) + (thv/add-variant :v01 :c01 :m01 :c02 :m02)) + c01-props (:variant-properties (thc/get-component file :c01)) + c02-props (:variant-properties (thc/get-component file :c02)) + file' (tho/duplicate-shape file :m01) + mains (variant-mains-in file' :v01) + new-main (->> mains + (remove #(#{(thi/id :m01) (thi/id :m02)} (:id %))) + first) + new-comp (thc/get-component-by-id file' (:component-id new-main))] + + (thf/validate-file! file') + (t/is (= 3 (count mains))) + (t/is (= c01-props (:variant-properties (thc/get-component file' :c01)))) + (t/is (= c02-props (:variant-properties (thc/get-component file' :c02)))) + (t/is (= ["Property 1"] (property-names new-comp))) + (t/is (= (str ctv/value-prefix 3) + (-> new-comp :variant-properties last :value))))) + +(t/deftest test-make-shapes-variant-skips-when-already-aligned + ;; Locks the narrowed early-exit: a fully aligned child must not have its + ;; props or variant-name rewritten. + (let [file (-> (thf/sample-file :file1) + (thv/add-variant :v01 :c01 :m01 :c02 :m02)) + page (thf/current-page file) + container (ths/get-shape file :v01) + shape (ths/get-shape file :m01) + before (:variant-properties (thc/get-component file :c01)) + before-vn (:variant-name shape) + changes (-> (pcb/empty-changes nil) + (pcb/with-page page) + (pcb/with-objects (:objects page)) + (pcb/with-library-data (:data file)) + (clvp/generate-make-shapes-variant [shape] container)) + file' (thf/apply-changes file changes) + shape' (ths/get-shape-by-id file' (:id shape))] + + (t/is (= before (:variant-properties (thc/get-component file' :c01)))) + (t/is (= before-vn (:variant-name shape'))))) + +(t/deftest test-paste-known-variant-into-other-container-aligns + ;; file A present in libraries → duplicate-variant path; props must match + ;; the target set and the file must validate. + (let [file-a (-> (thf/sample-file :file-a) + (thv/add-variant :va :ca1 :ma1 :ca2 :ma2)) + file-b (other-set-file) + libraries {(:id file-a) file-a + (:id file-b) file-b} + changes (paste-variants-into-container file-a file-b [:ma1 :ma2] :vb + :libraries libraries) + file' (thf/apply-changes file-b changes :validate? false) + errors (cfv/validate-file file' {}) + pasted (->> (variant-mains-in file' :vb) + (remove #(#{(thi/id :mb1) (thi/id :mb2)} (:id %))))] + + (t/is (nil? errors)) + (t/is (= 2 (count pasted))) + (doseq [shape pasted] + (let [comp (thc/get-component-by-id file' (:component-id shape))] + (t/is (= ["Property 1" "Property 2"] (property-names comp))) + (t/is (= "Other" (:name shape))) + (t/is (= (:variant-name shape) + (ctv/properties-to-name (:variant-properties comp)))))))) + +(t/deftest test-paste-unknown-preserves-matching-property-values + ;; Clipboard props (Task 2) whose names exist on the target survive regenerate. + (let [file-a (-> (thf/sample-file :file-a) + (thv/add-variant :va :ca1 :ma1 :ca2 :ma2)) + file-b (other-set-file) + props {(:id (thc/get-component file-a :ca1)) + (:variant-properties (thc/get-component file-a :ca1)) + (:id (thc/get-component file-a :ca2)) + (:variant-properties (thc/get-component file-a :ca2))} + changes (paste-variants-into-container file-a file-b [:ma1 :ma2] :vb + :variant-props props) + file' (thf/apply-changes file-b changes :validate? false) + pasted (->> (variant-mains-in file' :vb) + (remove #(#{(thi/id :mb1) (thi/id :mb2)} (:id %))) + (map #(thc/get-component-by-id file' (:component-id %)))) + values (->> pasted + (mapcat :variant-properties) + (filter #(= "Property 1" (:name %))) + (map :value) + set)] + + (thf/validate-file! file') + (t/is (contains? values "Value1")) + (t/is (contains? values "Value2")))) diff --git a/common/test/common_tests/runner.cljc b/common/test/common_tests/runner.cljc index f650081321..df2662df08 100644 --- a/common/test/common_tests/runner.cljc +++ b/common/test/common_tests/runner.cljc @@ -22,6 +22,7 @@ [common-tests.files-changes-test] [common-tests.files-migrations-0025-test] [common-tests.files-migrations-0026-test] + [common-tests.files-migrations-0033-test] [common-tests.files-migrations-0034-test] [common-tests.files-migrations-0035-test] [common-tests.files-migrations-test] @@ -32,6 +33,7 @@ [common-tests.files.shapes-builder-test] [common-tests.files.tokens-test] [common-tests.files.validate-test] + [common-tests.files.variant-test] [common-tests.geom-align-test] [common-tests.geom-bounds-layout-nil-test] [common-tests.geom-bounds-map-test] @@ -81,6 +83,7 @@ [common-tests.logic.text-touched-test] [common-tests.logic.token-apply-test] [common-tests.logic.token-test] + [common-tests.logic.variants-paste-test] [common-tests.logic.variants-reparent-test] [common-tests.logic.variants-switch-test] [common-tests.logic.variants-test] @@ -135,6 +138,7 @@ 'common-tests.files-changes-test 'common-tests.files-migrations-0025-test 'common-tests.files-migrations-0026-test + 'common-tests.files-migrations-0033-test 'common-tests.files-migrations-0034-test 'common-tests.files-migrations-0035-test 'common-tests.files-migrations-test @@ -145,6 +149,7 @@ 'common-tests.files.shapes-builder-test 'common-tests.files.tokens-test 'common-tests.files.validate-test + 'common-tests.files.variant-test 'common-tests.geom-align-test 'common-tests.geom-bounds-layout-nil-test 'common-tests.geom-bounds-map-test @@ -194,6 +199,7 @@ 'common-tests.logic.text-touched-test 'common-tests.logic.token-apply-test 'common-tests.logic.token-test + 'common-tests.logic.variants-paste-test 'common-tests.logic.variants-reparent-test 'common-tests.logic.variants-switch-test 'common-tests.logic.variants-test diff --git a/frontend/src/app/main/data/workspace/clipboard.cljs b/frontend/src/app/main/data/workspace/clipboard.cljs index 877e535f75..4569501922 100644 --- a/frontend/src/app/main/data/workspace/clipboard.cljs +++ b/frontend/src/app/main/data/workspace/clipboard.cljs @@ -160,10 +160,8 @@ (let [page-id (:current-page-id state) data (dsh/lookup-file-data state) objects (-> (dsh/get-page data page-id) - (get :objects)) - - components (cfv/find-variant-components data objects (:id shape))] - (into {} (map (juxt :id :variant-properties) components)))) + (get :objects))] + (cfv/collect-variant-properties data objects shape))) ;; Collects all the items together and split images into a @@ -179,7 +177,8 @@ (some? images) (update :images into images) - (ctk/is-variant-container? item) + (or (ctk/is-variant-container? item) + (ctk/is-variant-main? item)) (update :variant-properties merge (collect-variants state item)))) (maybe-translate [shape objects parent-frame-id] diff --git a/scripts/changelog.py b/scripts/changelog.py index 280ffa1f85..e09cfa1218 100644 --- a/scripts/changelog.py +++ b/scripts/changelog.py @@ -10,6 +10,7 @@ Subcommands: boom Generate the :boom: section from `breaking change` issues security Generate the :lock: section from advisories fixed in a version cross-ref Compare merged milestone PRs against the changelog section + dev-only-bugs List milestone bugs that never reached production report Generate the anomaly report (CHANGES-ISSUES.md) Usage: @@ -20,6 +21,7 @@ Usage: python3 scripts/changelog.py security "2.19.1" python3 scripts/changelog.py security "2.19.1" --write python3 scripts/changelog.py cross-ref "2.16.0" + python3 scripts/changelog.py dev-only-bugs "2.16.0" [--apply] python3 scripts/changelog.py report "2.16.0" python3 scripts/changelog.py report "2.16.0" --changes CHANGES.md --output CHANGES-ISSUES.md @@ -615,6 +617,103 @@ def cmd_cross_ref(args: argparse.Namespace) -> None: print(f" #{pr['number']} {pr['title'][:80]}") +# ───────────────────────────────────────────── +# Subcommand: dev-only-bugs +# ───────────────────────────────────────────── + +# Bugs found and fixed while building a feature of the same, unreleased +# milestone never reached end users and stay out of the changelog. +AUTO_PARENT_TYPES = {"Feature", "Enhancement"} + +VERSION_FIELD_RE = re.compile( + r"^#{2,4}\s*(?:affected versions?|versions?|environment)[^\n]*\n(.*?)(?=^#{2,4}\s|\Z)", + re.IGNORECASE | re.MULTILINE | re.DOTALL, +) +DEV_ENV_RE = re.compile(r"\b(?:hourly|develop|staging|pre|unreleased)\b", re.IGNORECASE) +PROD_ENV_RE = re.compile( + r"\b(?:production|pro|main|saas|design\.penpot\.app|self[- ]?hosted?)\b", + re.IGNORECASE, +) +VERSION_RE = re.compile(r"\b(\d+)\.(\d+)(?:\.\d+)?") + + +def version_field(body: str | None) -> str: + """Return the "Affected versions" / "Environment" text of an issue body.""" + match = VERSION_FIELD_RE.search(body or "") + return " ".join(match.group(1).split()) if match else "" + + +def only_unreleased_envs(text: str, milestone: str) -> bool: + """True when the version text names unreleased environments only: + hourly, develop, staging, PRE or a version of this milestone. Any + production hint or other version number keeps the bug in the changelog.""" + current = tuple(milestone.split(".")[:2]) + versions = [m for m in VERSION_RE.findall(text)] + dev = bool(DEV_ENV_RE.search(text)) or current in versions + prod = bool(PROD_ENV_RE.search(text)) or any(v != current for v in versions) + return dev and not prod + + +def find_dev_only_bugs(milestone: str, issues: list[dict]) -> tuple[list[dict], list[dict]]: + """Split the milestone's changelog bugs that never reached production. + + ``issues`` come from ``gh.py issue`` (they carry ``body`` and + ``parent``). Returns ``(auto, suggested)``: ``auto`` bugs are + sub-issues of a Feature or Enhancement of this milestone and get the + `no changelog` label; ``suggested`` ones need a human to confirm.""" + auto, suggested = [], [] + for issue in issues: + if issue.get("issue_type") != "Bug" or issue.get("state") != "CLOSED": + continue + if {"no changelog", "release blocker"} & set(issue.get("labels", [])): + continue + parent = issue.get("parent") or {} + entry = {"issue": issue["number"], "issue_title": issue.get("title", "")} + if parent.get("milestone") == milestone: + entry["reason"] = f"sub-issue of {parent.get('issue_type')} #{parent['number']} in {milestone}" + (auto if parent.get("issue_type") in AUTO_PARENT_TYPES else suggested).append(entry) + continue + # `regression` says something that worked broke, not where: it + # also marks regressions users hit in production. + if "regression" in issue.get("labels", []): + entry["reason"] = "labelled `regression`" + suggested.append(entry) + continue + text = version_field(issue.get("body")) + if only_unreleased_envs(text, milestone): + entry["reason"] = f'only seen on "{text[:80]}"' + suggested.append(entry) + return auto, suggested + + +def fetch_milestone_bugs(milestone: str) -> list[dict]: + """Closed milestone bugs with body and parent (two batched lookups).""" + issues = json.loads(run_gh_py("issues", milestone)) + numbers = [str(i["number"]) for i in issues if i.get("issue_type") == "Bug"] + return json.loads(run_gh_py("issue", *numbers)) if numbers else [] + + +def cmd_dev_only_bugs(args: argparse.Namespace) -> None: + """List bugs that never reached production; --apply labels the sure ones.""" + auto, suggested = find_dev_only_bugs(args.milestone, fetch_milestone_bugs(args.milestone)) + + print(f"Label `no changelog` ({len(auto)}):") + for e in auto: + print(f" #{e['issue']} {e['issue_title'][:70]} ({e['reason']})") + print(f"Candidates to confirm ({len(suggested)}):") + for e in suggested: + print(f" #{e['issue']} {e['issue_title'][:70]} ({e['reason']})") + + if args.apply: + for e in auto: + subprocess.run( + ["gh", "issue", "edit", str(e["issue"]), "--repo", REPO, + "--add-label", "no changelog"], + check=True, capture_output=True, + ) + print(f"Labelled {len(auto)} issues `no changelog`", file=sys.stderr) + + # ───────────────────────────────────────────── # Subcommand: report # ───────────────────────────────────────────── @@ -872,6 +971,9 @@ def cmd_report(args: argparse.Namespace) -> None: ): anomalies_d.append({"version": ver, "line": line[:100]}) + # Bugs that may never have reached production: warnings, not anomalies. + dev_auto, dev_suggested = find_dev_only_bugs(milestone, fetch_milestone_bugs(milestone)) + def fmt_ms(ms: str | None) -> str: return ms if ms else "_none_" @@ -1033,6 +1135,26 @@ def cmd_report(args: argparse.Namespace) -> None: "✅ No highlight gaps found. All released X.Y.0 versions have properly referenced :rocket: entries.\n\n" ) + if dev_auto or dev_suggested: + f.write("## Bugs that may never have reached production\n\n") + f.write( + "These are warnings, not anomalies. Bugs found while building " + "a feature of this release never affected end users. Add the " + "`no changelog` label to the ones you confirm and re-run.\n\n" + ) + if dev_auto: + f.write( + "### Sub-issues of a feature of this milestone (run `dev-only-bugs --apply`)\n\n" + ) + for e in dev_auto: + f.write(f"- {issue_link_title(e['issue'], e['issue_title'][:80])}: {e['reason']}\n") + f.write("\n") + if dev_suggested: + f.write("### Candidates to confirm\n\n") + for e in dev_suggested: + f.write(f"- {issue_link_title(e['issue'], e['issue_title'][:80])}: {e['reason']}\n") + f.write("\n") + f.write("---\n\n") f.write("## Context\n\n") f.write(f"- Milestone: **{milestone}**\n") @@ -1100,6 +1222,15 @@ def main() -> None: help="Path to the changelog file (default: CHANGES.md)") p_cross.set_defaults(func=cmd_cross_ref) + p_dev = sub.add_parser( + "dev-only-bugs", + help="List milestone bugs that never reached production", + ) + p_dev.add_argument("milestone", help="Milestone title (e.g. '2.16.0')") + p_dev.add_argument("--apply", action="store_true", + help="Label the sub-issues of a milestone feature `no changelog`") + p_dev.set_defaults(func=cmd_dev_only_bugs) + p_report = sub.add_parser( "report", help="Generate the anomaly report (CHANGES-ISSUES.md)" ) diff --git a/scripts/gh.py b/scripts/gh.py index eebd5155fd..30c12e58f2 100755 --- a/scripts/gh.py +++ b/scripts/gh.py @@ -752,8 +752,10 @@ GQL_ISSUE_BY_NUMBER_QUERY_ITEM = """\ number title state + body milestone {{ title }} issueType {{ name }} + parent {{ number issueType {{ name }} milestone {{ title }} }} labels(first: 20) {{ nodes {{ name }} }} assignees(first: 10) {{ nodes {{ login }} }} closedByPullRequestsReferences(first: 5) {{ nodes {{ number }} }} @@ -775,12 +777,19 @@ def issue_node_to_dict(node: dict) -> dict: """Convert a GraphQL issue node to the shared issue dict shape.""" issue_type = node.get("issueType") ms = node.get("milestone") + parent = node.get("parent") return { "number": node["number"], "title": node["title"], "state": node["state"], + "body": node.get("body"), "milestone": ms["title"] if ms else None, "issue_type": issue_type["name"] if issue_type else None, + "parent": { + "number": parent["number"], + "issue_type": (parent.get("issueType") or {}).get("name"), + "milestone": (parent.get("milestone") or {}).get("title"), + } if parent else None, "labels": [lbl["name"] for lbl in node["labels"]["nodes"]], "assignees": node_assignees(node), "closing_prs": [pr["number"] for pr in node["closedByPullRequestsReferences"]["nodes"]], diff --git a/scripts/test_changelog.py b/scripts/test_changelog.py index e27d3de1d2..8eb0311c9f 100644 --- a/scripts/test_changelog.py +++ b/scripts/test_changelog.py @@ -657,5 +657,61 @@ class CmdSecurityTests(unittest.TestCase): self.assertIn("will not be listed", err) +def bug(number, body="", parent=None, labels=()): + return { + "number": number, + "title": f"Bug {number}", + "state": "CLOSED", + "issue_type": "Bug", + "labels": list(labels), + "body": body, + "parent": parent, + } + + +class OnlyUnreleasedEnvsTests(unittest.TestCase): + def check(self, text): + return ch.only_unreleased_envs(text, "2.19.0") + + def test_dev_environments(self): + for text in ["HOURLY", "https://hourly.penpot.dev/", "PRE & hourly", + "staging (current)", "develop (2.19.0 unreleased)", + "2.19.0-RC1 (develop)", "2.19.0"]: + with self.subTest(text=text): + self.assertTrue(self.check(text)) + + def test_production_hints(self): + for text in ["PRO", "pre/pro/hourly", "develop / staging / main", + "2.18.0-RC5", "2.18.1, 2.19.0-RC1", "self-hosted: 2.17.2", + "design.penpot.app and self-host"]: + with self.subTest(text=text): + self.assertFalse(self.check(text)) + + def test_no_signal(self): + for text in ["", "_No response_", "-", "development build"]: + with self.subTest(text=text): + self.assertFalse(self.check(text)) + + +class FindDevOnlyBugsTests(unittest.TestCase): + def test_splits_auto_and_suggested(self): + issues = [ + bug(1, parent={"number": 10, "issue_type": "Feature", "milestone": "2.19.0"}), + bug(2, parent={"number": 11, "issue_type": "Enhancement", "milestone": "2.19.0"}), + bug(3, parent={"number": 12, "issue_type": "EPIC", "milestone": "2.19.0"}), + bug(4, parent={"number": 13, "issue_type": "Feature", "milestone": "2.18.0"}), + bug(5, body="### Affected versions\n\nhourly\n"), + bug(6, body="### Affected versions\n\n2.18.2\n"), + bug(7, parent={"number": 10, "issue_type": "Feature", "milestone": "2.19.0"}, + labels=["no changelog"]), + {**bug(8, parent={"number": 10, "issue_type": "Feature", "milestone": "2.19.0"}), + "issue_type": "Enhancement"}, + bug(9, body="### Affected versions\n\n2.18.2\n", labels=["regression"]), + ] + auto, suggested = ch.find_dev_only_bugs("2.19.0", issues) + self.assertEqual([e["issue"] for e in auto], [1, 2]) + self.assertEqual([e["issue"] for e in suggested], [3, 5, 9]) + + if __name__ == "__main__": unittest.main(verbosity=2)