Merge remote-tracking branch 'origin/merge-staging' into staging

This commit is contained in:
Andrey Antukh 2026-11-12 15:32:33 +01:00
commit 92c478b0e2
22 changed files with 948 additions and 72 deletions

View File

@ -31,7 +31,7 @@ memory; it does not repeat it.
layout, entry rules, the `## Breaking change note` comment contract, the
advisory (`:lock:`) contract, pre-flight checks, anomaly types, and pitfalls.
2. `mem:scripts/changelog` — `boom`, `security`, `check-merged`, `cross-ref`,
`report`.
`dev-only-bugs`, `report`.
3. `mem:scripts/gh` — `issues`, `issue`, `comments`, `prs`, `advisories`.
## Commands
@ -39,6 +39,8 @@ memory; it does not repeat it.
```bash
python3 scripts/gh.py issues "<VERSION>" --exclude "release blocker,no changelog"
python3 scripts/gh.py advisories --patched-version "<VERSION>"
python3 scripts/changelog.py dev-only-bugs "<VERSION>" # list bugs that never reached production
python3 scripts/changelog.py dev-only-bugs "<VERSION>" --apply # label the sure ones `no changelog`
python3 scripts/changelog.py check-merged <PR numbers...>
python3 scripts/changelog.py boom "<VERSION>" # print the :boom: section
python3 scripts/changelog.py boom "<VERSION>" --write # replace it in CHANGES.md
@ -54,4 +56,5 @@ python3 scripts/changelog.py report "<VERSION>"
- Only closed, non-excluded milestone issues are included; the issue is the changelog unit and the PR is inline.
- `:boom:` and `:rocket:` are label-generated and `:lock:` is advisory-generated: never hand-edit them; edit the label, the `## Breaking change note` comment or the advisory and re-run the command.
- Verify PR merge status before writing; only merged PRs are referenced.
- Bugs that never reached production stay out: `dev-only-bugs --apply` labels the sure ones; candidates are the user's call, never label them yourself.
- Fill `patched_versions` in the advisories before running `security`; unpublished advisories are valid.

2
.gitignore vendored
View File

@ -121,7 +121,7 @@ opencode.json
/.opencode/reports
/.opencode/prompts
/.ci-logs
/tmp/
/tmp
/.codex/
/scripts/__pycache__/
/tools/__pycache__

View File

@ -3,6 +3,8 @@
JVM `clojure.test` (kaocha runner) under `backend/test/backend_tests/`.
- READ `mem:testing` FIRST — it defines the execution discipline (no piping, tee to file, preferred commands) that applies to all JVM test runs.
- Test DB/Redis come from the `PENPOT_TEST_*` env vars already set in the shell (`PENPOT_TEST_DATABASE_URI`, `PENPOT_TEST_REDIS_URI`, ...); per instance they point at `penpot_test` / `penpot_test_wsN` and valkey DB `6+N` (`mem:devenv/core`). Do not override them with values copied from CI logs or other sources unless the user asks (e.g. "use a temporary database"); the CI env (`redis://redis/1`, `postgresql://postgres/penpot_test`) only works inside the CI docker network.
- The test database may have migrations applied from newer versions (it is shared dev infra). Reinitializing it — `dropdb`/`createdb` of the `PENPOT_TEST_*` database only — is explicitly permitted when that blocks a test run; never drop the main `penpot` database or another instance's test database.
- All CLI commands must be executed from the `backend/` subdirectory.
- Tests are invoked directly via `clojure -M:dev:test` (kaocha) — there is no pnpm wrapper. Kaocha auto-discovers test namespaces, so no runner registration is needed.
- **Preferred way to run them: `scripts/ci <module> --test`**, which runs lint, tests and format per module and writes a log per task under `.ci-logs/`. It exports `PENPOT_FLAGS` with `enable-backend-asserts` appended, so use it rather than calling `clojure -M:dev:test` by hand.

View File

@ -0,0 +1,7 @@
# Clojure Logging Conventions
Structured logging through `app.common.logging` (`l/trc`/`l/dbg`/`l/info`/`l/wrn`/`l/error`/`l/fatal`): `:hint` carries the human-readable message, every other datum goes in its own keyword field (`:id`, `:container`, `:removed`, ...).
- `:hint` values start lowercase: `"fix dangling children"`, not `"Fix dangling children"`. Uppercase-first hints exist as legacy (~57 occurrences, the densest in `backend/src`; also `l/trc` in migration `legacy-28` and `log/error` in `app.common.files.repair`) — do not copy them, and do not mass-rewrite them unless the surrounding code needs the change.
- A name that requires uppercase (an env var like `PENPOT_MEDIA_PROCESSING_SERVICE_URI`, an acronym like `MCP` or `LDAP`) stays uppercase inside an otherwise lowercase sentence.
- Repair work done by a file migration logs at `l/trc`, only when it changes something, and with the ids it touched: the house style of `migrations.cljc` (`legacy-28` precedes; the file has no `warn`/`info` lines). The shipped `log4j2.xml` pins the `app` logger to `info` (static file, `monitorInterval="60"`), so those trace lines appear only after the operator raises the level; the durable record of a repaired file is the migration id in its `:migrations`.

View File

@ -6,6 +6,7 @@ You are working on the GitHub project `penpot/penpot`, a monorepo.
- A section's top-level memory is `<section>/core`. When a section is relevant, read the core memory before focused memories.
- Edits/stale refs/duplication cleanup: `mem:memory-maintenance`.
- Cross-cutting testing principles, TDD workflow, and anti-patterns: `mem:testing`.
- Structured logging conventions (`:hint` casing and fields, log levels): `mem:clojure/logging` — read before adding or changing any log call.
- Verified Clojure language behaviors that contradict common assumptions (e.g. `int?` covers `Long`; `integer?` is the general predicate): `mem:clojure/idioms` — read before assuming stdlib predicate semantics.
- Read `mem:clojure/design-rules` (function shape, boundary checks, naming) before writing or modifying Clojure/ClojureScript code — not only when designing a solution or an API.
- Promises: which call style to use where (`fmap`/`mcat`/`merr` vs `then`/`catch`) and how they bridge to rx: `mem:clojure/promesa` — read before chaining promises or mixing them with observables.

View File

@ -8,7 +8,8 @@
- `boom <MILESTONE> [--changes CHANGES.md --write]` — generate `### :boom: Breaking changes & Deprecations` from the closed, non-excluded milestone issues labelled `breaking change`, sorted by issue number. Each issue becomes a `#### <issue title>` block: content = body of the issue's last `## Breaking change note` comment (whole markdown, headings demoted to `#####` outside code fences; fallback = heading plus refs with a stderr warning), refs = issue link plus merged closing PRs on their own line, with `(by @author)` for `community contribution`. `--write` replaces only that subsection of the version (inserts it after the version header when missing, keeps the heading when there are no entries; idempotent). Print-only by default. Uses `gh.py comments`.
- `security <MILESTONE> [--changes CHANGES.md --write]` — generate `### :lock: Security fixes` from the advisories whose `patched_versions` include the version (via `gh.py advisories --patched-version`), sorted by GHSA id. Entry: `- Fix <description> (<advisory URL>) (PR: [#N](...))`; description = `Fix` + advisory summary, first word decapitalized unless it starts with an acronym; PRs = merged milestone PRs labelled `pr pt` whose body contains the GHSA id, comma-separated, and no PR is legal. Warns on stderr when a `pr pt` PR references an advisory with no matching `patched_versions`. `--write` replaces only that subsection, placed after `:rocket:` (or `:boom:`, or the version header when neither exists); empty entries remove the subsection; idempotent. Print-only by default.
- `cross-ref <MILESTONE> [--changes CHANGES.md]` — fetch milestone PRs with `--state all` once, report merged PRs missing from the version section plus CLOSED (unmerged) warnings. PR links inside `:lock:` entries count as referenced; advisory URLs never appear here by design.
- `report <MILESTONE> [--changes CHANGES.md --output CHANGES-ISSUES.md]` — overwrite the anomaly report: A/B milestone mismatches, E `:boom:` entry without the label, F `:rocket:` sync, G `:boom:` sync, plus the C/D `:rocket:` gaps (warnings). `:boom:`/`:rocket:` rule violations are fixed by regenerating or relabelling, not by editing entries in place.
- `dev-only-bugs <MILESTONE> [--apply]` — list closed milestone bugs that never reached production, without `no changelog`. Sure: parent issue is a `Feature`/`Enhancement` of the same milestone; `--apply` adds `no changelog` to these only. Candidates (never labelled automatically): parent is an epic of the milestone, `regression` label, or the "Affected versions" / "Environment" body field names only hourly, develop, staging, PRE or this milestone (`only_unreleased_envs`; any production hint or other version keeps the bug in). Reads `body` and `parent` from one batched `gh.py issue` lookup.
- `report <MILESTONE> [--changes CHANGES.md --output CHANGES-ISSUES.md]` — overwrite the anomaly report: A/B milestone mismatches, E `:boom:` entry without the label, F `:rocket:` sync, G `:boom:` sync, plus the C/D `:rocket:` gaps and the `dev-only-bugs` lists (warnings). `:boom:`/`:rocket:` rule violations are fixed by regenerating or relabelling, not by editing entries in place.
## Conventions

View File

@ -81,7 +81,7 @@ python3 scripts/gh.py prs --milestone "2.16.0" --state all
Fetch issue details by number in batches of 50 (same input styles as `prs`: numbers, `--file`, `--stdin`). No filters. A dead number fails its whole batch (unlike PRs, issues error instead of resolving to null), so the batch falls back to one-by-one lookups and reports misses as `{"number", "error": "not_found"}`. Any other failure aborts.
**Output**: JSON array to stdout; progress to stderr.
**Output**: JSON array to stdout; progress to stderr. Each issue carries the shared issue fields plus `body` and `parent` (`number`, `issue_type`, `milestone`, or null); `changelog.py dev-only-bugs` reads both.
### `comments`

View File

@ -24,11 +24,12 @@ Editing `CHANGES.md` happens only on explicit request or as part of the release
| Is it a breaking change? | `breaking change` label |
| Is it a highlight? | `release highlight` label |
- **Issue Type decides the section**: `Bug` → `:bug:`, `Feature`/`Enhancement` → `:sparkles:`, `Task` is excluded (internal chores). Labels `bug`/`enhancement` and title emoji prefixes are often wrong or missing; do not use them.
- **Issue Type decides the section**: `Feature` → `:tada:`, `Enhancement` → `:sparkles:`, `Bug` → `:bug:`, `Task` is excluded (internal chores). Labels `bug`/`enhancement` and title emoji prefixes are often wrong or missing; do not use them.
- **Security fixes come from advisories**: an advisory whose `patched_versions` include the version becomes a `:lock:` entry (section 7). `triage`, `draft` and `published` are all valid states; the advisory may still be unpublished when the changelog is written. A security fix may also carry a regular bug issue; both entries are legal and independent (the issue lives in `:bug:` as any other bug).
- **Two labels override the Issue Type**: `breaking change` → an entry only under `:boom:`, never in its regular section; `release highlight` → an extra entry under `:rocket:` on top of its regular entry.
- **Only closed issues**. An open issue in the milestone is not generated, even if someone added a provisional entry by hand before the merge.
- **Excluded issues**: `no changelog`, `release blocker`, `Task` issue type, `Rejected` project status.
- **Bugs that never reached production** stay out: a bug found and fixed while building a feature of the same, unreleased version never affected users. The team rarely labels them, so the flow finds them with `dev-only-bugs` and labels them `no changelog` (section 8, step 3).
- **Excluded PRs**: a PR with `release blocker` or `no issue required` does not bring an entry of its own.
- **An issue with no milestone** belongs to another (probably private) project: it is neither changelog candidate nor anomaly.
@ -39,7 +40,7 @@ The newest version goes at the top, right after the `# CHANGELOG` header. Inside
```markdown
# CHANGELOG
## 2.19.0 (Unreleased)
## 2.20.0 (Unreleased)
### :boom: Breaking changes & Deprecations
@ -57,21 +58,27 @@ The newest version goes at the top, right after the `# CHANGELOG` header. Inside
- Fix <user-facing description of the vulnerability> (<advisory URL>) (PR: [#PR](https://github.com/penpot/penpot/pull/PR))
### :bug: Bugs fixed
- Fix <user-facing description of the bug> [#ISSUE](https://github.com/penpot/penpot/issues/ISSUE) (PR: [#PR](https://github.com/penpot/penpot/pull/PR))
### :sparkles: New features & Enhancements
### :tada: Features
- Add <user-facing description of the feature> [#ISSUE](https://github.com/penpot/penpot/issues/ISSUE) (PR: [#PR](https://github.com/penpot/penpot/pull/PR))
### :sparkles: Enhancements
- Improve <user-facing description of the existing behaviour> [#ISSUE](https://github.com/penpot/penpot/issues/ISSUE) (PR: [#PR](https://github.com/penpot/penpot/pull/PR))
### :bug: Bugs
- Fix <user-facing description of the bug> [#ISSUE](https://github.com/penpot/penpot/issues/ISSUE) (PR: [#PR](https://github.com/penpot/penpot/pull/PR))
```
- `:boom:` first, then `:rocket:`, `:lock:`, `:bug:`, `:sparkles:`. Other headings exist in old versions (`:heart:` community contributions); do not add them to new versions.
- `:boom:` first, then `:rocket:`, `:lock:`, `:tada:`, `:sparkles:`, `:bug:`. Features come before enhancements so they stand out to readers deciding whether to update.
- **This layout applies from 2.20.0.** Versions up to 2.19.x use the old one: `:bug: Bugs fixed` before `:sparkles: New features & Enhancements`, which mixes features and enhancements. Keep the old layout in those sections, including new 2.19.x patches, so a minor line reads the same. The in-app update dialog only reads `:rocket:`, so the change does not affect it.
- Other headings exist in old versions (`:heart:` community contributions); do not add them to new versions.
- Include a section only when it has entries. Historical versions may have an empty `:boom:` or `:rocket:` heading; `boom --write` keeps the `:boom:` heading even when there are no entries. `:lock:` is omitted (and removed on `--write`) when the version has no security advisories.
- One blank line between the last entry of a section and the next heading.
- No manual line wrapping: one line per bullet or paragraph, however long.
## 4. Regular entries (`:bug:`, `:sparkles:`)
## 4. Regular entries (`:tada:`, `:sparkles:`, `:bug:`)
The entry shape is:
@ -153,15 +160,18 @@ Generated from the advisories that declare the version in `patched_versions`; ne
1. **Confirm the version** (a semver string such as `2.19.0`) with the user when it is not obvious.
2. **Fetch the milestone issues**: `python3 scripts/gh.py issues "<VERSION>" --exclude "release blocker,no changelog"` (closed by default; add `--state all` only to inspect open issues). `gh.py` already drops `Task` and `Rejected`; pass `--include-tasks` / `--include-rejected` only to inspect them. To find what is missing from an existing section, add `--compare CHANGES.md`.
3. **Fetch PR details when needed**: `python3 scripts/gh.py prs <numbers...>` (also `--milestone`). Use it for the PR author, labels and merge state.
4. **Categorize** by Issue Type and the two labels (section 2), and **build the descriptions** (section 4).
5. **Verify PR merge status** before writing: `python3 scripts/changelog.py check-merged <PR numbers...>`. Only merged PRs go in the changelog; if a closing PR was closed unmerged, look for the PR that superseded it.
6. **Reconcile the existing text** (section 9).
7. **Generate `:boom:`**: `python3 scripts/changelog.py boom "<VERSION>"` prints the section; add `--write` to replace it in `CHANGES.md`. The write replaces only that subsection, inserts it after the version header when missing, keeps the heading when there are no entries, and is idempotent.
8. **Generate `:rocket:`** from the `release highlight` label (section 6).
9. **Generate `:lock:`**: `python3 scripts/changelog.py security "<VERSION>"` prints the section; add `--write` to replace it in `CHANGES.md`. The contract lives in section 7.
10. **Cross-check PRs**: `python3 scripts/changelog.py cross-ref "<VERSION>"`. It lists merged milestone PRs missing from the section and warns about closed (unmerged) PRs. PRs linked from `:lock:` count as referenced; an advisory with no PR is invisible here by design.
11. **Generate the anomaly report**: `python3 scripts/changelog.py report "<VERSION>"`. It overwrites `CHANGES-ISSUES.md` with the current state (section 10).
3. **Keep out bugs that never reached production**: `python3 scripts/changelog.py dev-only-bugs "<VERSION>"` lists them; add `--apply` to label the sure ones `no changelog` on GitHub, then re-fetch step 2. The label puts the decision on the issue, so it survives a regeneration.
- **Sure**: a bug whose parent issue is a `Feature` or `Enhancement` of the same milestone. `--apply` labels these.
- **Candidates**: a bug whose parent is an epic of the same milestone, that carries the `regression` label, or whose "Affected versions" / "Environment" field names only hourly, develop, staging, PRE or a version of this milestone. `regression` also marks regressions users hit in production, and the field says where the bug was seen, not since when it exists, so never label these yourself: show them to the user, who labels the ones they confirm. The anomaly report lists them too (section 10).
4. **Fetch PR details when needed**: `python3 scripts/gh.py prs <numbers...>` (also `--milestone`). Use it for the PR author, labels and merge state.
5. **Categorize** by Issue Type and the two labels (section 2), and **build the descriptions** (section 4).
6. **Verify PR merge status** before writing: `python3 scripts/changelog.py check-merged <PR numbers...>`. Only merged PRs go in the changelog; if a closing PR was closed unmerged, look for the PR that superseded it.
7. **Reconcile the existing text** (section 9).
8. **Generate `:boom:`**: `python3 scripts/changelog.py boom "<VERSION>"` prints the section; add `--write` to replace it in `CHANGES.md`. The write replaces only that subsection, inserts it after the version header when missing, keeps the heading when there are no entries, and is idempotent.
9. **Generate `:rocket:`** from the `release highlight` label (section 6).
10. **Generate `:lock:`**: `python3 scripts/changelog.py security "<VERSION>"` prints the section; add `--write` to replace it in `CHANGES.md`. The contract lives in section 7.
11. **Cross-check PRs**: `python3 scripts/changelog.py cross-ref "<VERSION>"`. It lists merged milestone PRs missing from the section and warns about closed (unmerged) PRs. PRs linked from `:lock:` count as referenced; an advisory with no PR is invisible here by design.
12. **Generate the anomaly report**: `python3 scripts/changelog.py report "<VERSION>"`. It overwrites `CHANGES-ISSUES.md` with the current state (section 10).
## 9. Pre-flight: reconcile before writing
@ -172,7 +182,7 @@ Re-fetch the milestone (do not trust cached data) and fix these in `CHANGES.md`
3. **Newly applied exclusion labels** (`no changelog`, `release blocker`) → remove the entry.
4. **Issue no longer closed, deleted, or now `Rejected`** → remove the entry.
5. **Unmerged or moved PR reference** → fix the reference or remove the entry. A PR merged in a different milestone is an anomaly (section 10); do not remove it silently.
6. **Issue Type changed to `Task`** → remove the entry.
6. **Issue Type changed** → to `Task`: remove the entry; to another type: move the entry to that type's section.
7. **Breaking change out of sync** (a `breaking change` issue outside `:boom:`, or a `:boom:` entry without the label) → regenerate the section; a labelled issue has no regular entry.
8. **Highlight out of sync** (`:rocket:` entry without the label, or a labelled issue missing) → remove it or add it.
9. **Missing valid issues** (closed, non-excluded, referenced nowhere) → add them to their section.
@ -189,6 +199,8 @@ Re-fetch the milestone (do not trust cached data) and fix these in `CHANGES.md`
- **C — released X.Y.0 version without a `:rocket:` section** (gap; patches never carry highlights).
- **D — `:rocket:` entry without issue AND PR references** (gap).
**Bugs that may never have reached production** are warnings too (section 8, step 3): sub-issues of a milestone feature still unlabelled (run `dev-only-bugs --apply`) and candidates for the user to confirm. They never count toward the total.
Anything else is a rule violation, not a report item: fix it in the pre-flight and re-run.
## 11. Pitfalls
@ -203,6 +215,6 @@ Anything else is a rule violation, not a report item: fix it in the pre-flight a
## 12. Where the code is
- `scripts/gh.py`: `issues`, `issue`, `comments`, `prs`, `link-issue`, `advisories` (with `--patched-version`). Details: `mem:scripts/gh`.
- `scripts/changelog.py`: `check-merged`, `boom`, `security`, `cross-ref`, `report`. Details: `mem:scripts/changelog`.
- `scripts/changelog.py`: `check-merged`, `boom`, `security`, `cross-ref`, `dev-only-bugs`, `report`. Details: `mem:scripts/changelog`.
- Tests: `scripts/test_gh.py`, `scripts/test_changelog.py`.
- The note contract for issue authors: `mem:workflow/creating-issues`, section **Breaking changes**.

View File

@ -445,10 +445,9 @@
(t/is (thrown? clojure.lang.ExceptionInfo
(invoke! cfg (fn [_ _] :never) {}))))
;; the count came back down instead of staying up for good. Only the
;; last publication is asserted, not the exact list: the sampler
;; republishes the same count when its first tick lands inside the
;; window.
;; the count came back down instead of staying up for good. The
;; republisher daemon thread can add publications inside the window,
;; so what is asserted is the last value, not the whole sequence.
(t/is (= 0 (waiting-count climit name)))
(t/is (= 0 (waiting-last name)))

View File

@ -2233,44 +2233,49 @@
;; been added to enforce this from now on.
;; Main instances in variants are a special case because they need to match the variant name.
(letfn [(update-shape [container shape-id]
(let [shape (ctn/get-shape container shape-id)
(let [shape (ctn/get-shape container shape-id)]
;; A shape may reference a child that does not exist in the
;; container (a dangling reference). There is nothing to update
;; on it, and storing it would add a nil entry to the objects
;; map, making the whole file invalid.
(if (nil? shape)
container
(let [shape (cond
;; Variant container must have a normalized name
(ctk/is-variant-container? shape)
(let [name (:name shape)
clean-name (cpn/clean-path name)
clean-name (if (str/blank? clean-name)
"(empty)" ;; this may occur if the name contains only separators
clean-name)]
(if (= name clean-name)
shape
(assoc shape :name clean-name)))
shape (cond
;; Variant container must have a normalized name
(ctk/is-variant-container? shape)
(let [name (:name shape)
clean-name (cpn/clean-path name)
clean-name (if (str/blank? clean-name)
"(empty)" ;; this may occur if the name contains only separators
clean-name)]
(if (= name clean-name)
shape
(assoc shape :name clean-name)))
;; Main instance inside a variant must have the same name as the container
(ctk/main-instance? shape)
(if (ctk/is-variant? shape)
(let [variant-container (ctn/get-shape container (:parent-id shape))]
(if (or (= (:name shape) (:name variant-container))
(nil? variant-container))
shape
(assoc shape :name (:name variant-container))))
;; Main instance inside a variant must have the same name as the container
(ctk/main-instance? shape)
(if (ctk/is-variant? shape)
(let [variant-container (ctn/get-shape container (:parent-id shape))]
(if (or (= (:name shape) (:name variant-container))
(nil? variant-container))
shape
(assoc shape :name (:name variant-container))))
;; Other main instances must have the same name as the path+name of the component
(let [component (ctkl/get-component data (:component-id shape) true)
expected-name (when component
(cpn/merge-path-item (:path component) (:name component)))]
(if (or (= expected-name (:name shape))
(nil? expected-name))
shape
(assoc shape :name expected-name))))
;; Other main instances must have the same name as the path+name of the component
(let [component (ctkl/get-component data (:component-id shape) true)
expected-name (when component
(cpn/merge-path-item (:path component) (:name component)))]
(if (or (= expected-name (:name shape))
(nil? expected-name))
shape
(assoc shape :name expected-name))))
;; Any other shape does not need processing
:else shape)]
;; Any other shape does not need processing
:else shape)]
(reduce update-shape
(ctt/set-shape container shape)
(:shapes shape))))
(reduce update-shape
(ctt/set-shape container shape)
(:shapes shape))))))
(update-container
[container]
@ -2322,6 +2327,35 @@
(update :pages-index d/update-vals update-container)
(d/update-when :components d/update-vals update-container)))))
(defmethod migrate-data "0035-fix-dangling-children"
[data _]
;; A shape may list a child id that does not exist in the objects map of
;; the page or component that owns it (a dangling reference). Such a
;; reference is invalid: any operation that walks the shape tree through
;; the children lists trips on it, and the file cannot be migrated nor
;; opened. Here we remove the dangling ids from the children lists,
;; keeping every shape that really exists.
(letfn [(fix-shape [container-id objects shape]
(d/update-when shape :shapes
(fn [shapes]
(let [shapes' (filterv #(contains? objects %) shapes)]
(when (not= shapes shapes')
(l/trc :hint "fix dangling children"
:container container-id
:shape-id (:id shape)
:removed (remove #(contains? objects %) shapes)))
shapes'))))
(fix-objects [container-id objects]
(update-vals objects (partial fix-shape container-id objects)))
(update-container [container]
(d/update-when container :objects #(fix-objects (:id container) %)))]
(-> data
(update :pages-index d/update-vals update-container)
(d/update-when :components d/update-vals update-container))))
(defmethod migrate-data "0035-fix-shape-svg-attrs-and-defs"
;; Stored svg attribute keys are kebab-case (issue #11947): the v3 reader
;; already outputs kebab and birth normalizes any outside spelling, so
@ -2459,4 +2493,5 @@
"0032-fix-children-not-in-parent"
"0033-normalize-main-instance-names"
"0034-fix-path-content-offset"
"0035-fix-dangling-children"
"0035-fix-shape-svg-attrs-and-defs"]))

View File

@ -28,6 +28,25 @@
(map #(ctcl/get-component data % true))
reverse)))
(defn collect-variant-properties
"Map of component-id → variant-properties for clipboard copy.
Variant container: every related component. Variant main: that
component only. Anything else: empty map."
[data objects shape]
(cond
(ctc/is-variant-container? shape)
(into {}
(map (juxt :id :variant-properties))
(find-variant-components data objects (:id shape)))
(ctc/is-variant-main? shape)
(if-let [component (ctcl/get-component data (:component-id shape) true)]
{(:id component) (:variant-properties component)}
{})
:else
{}))
(defn extract-properties-names
[shape data]
(->> shape

View File

@ -2766,7 +2766,8 @@
page-id
nil
variant-id
props))]
props
{:apply-changes-local-library? true}))]
[root-shape changes]))
(defn generate-add-component
@ -3282,6 +3283,25 @@
(and is-component-root? is-component-main?)
(regenerate-component new-obj))
;; Unknown-component paste into a variant set: align props/names
;; after regenerate. Reload the shape so we use the new component-id.
;; Prefit the component name/path to the container so
;; create-new-properties-from-variant does not treat the component
;; full name as a former set name and inflate properties (add-name?).
changes (cond-> changes
(and is-component-root?
is-component-main?
(ctk/is-variant-container? parent))
(as-> $
(let [shape (get (pcb/get-objects $) (:id new-obj))
[cpath cname] (cpn/split-group-name (:name parent))
changes (pcb/update-component
$
(:component-id shape)
#(assoc % :name cname :path cpath)
{:apply-changes-local-library? true})]
(clvp/generate-make-shapes-variant changes [shape] parent))))
;; This is needed for the recursive call to find the new object as parent
page' (ctst/add-shape (:id new-obj)
new-obj

View File

@ -266,10 +266,20 @@
(first
(reduce
(fn [[changes used-values] shape]
(let [component (ctcl/get-component data (:component-id shape) true)]
(if (or (zero? num-shapes) ;; do nothing if there are no shapes
(and (= variant-id (:variant-id shape)) ;; or we are only moving the shape inside its parent (it is
(not (:deleted component)))) ;; the same parent and the component isn't deleted)
(let [component (ctcl/get-component data (:component-id shape) true)
already-here? (= variant-id (:variant-id shape))
prop-names (cfv/extract-properties-names shape data)
base-names (map :name base-props)
props-ok? (= prop-names base-names)
names-ok? (= (cpn/merge-path-item (:path component) (:name component))
container-name)
variant-ok? (= (:variant-name shape)
(ctv/properties-to-name (:variant-properties component)))
aligned? (and props-ok? names-ok? variant-ok?)]
(if (or (zero? num-shapes) ;; do nothing if there are no shapes
(and already-here? ;; or already a fully aligned child of this container
aligned?
(not (:deleted component))))
[changes used-values]
(let [props (-> (create-new-properties shape total-props)
(fill-blank-properties used-values))

View File

@ -0,0 +1,49 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns common-tests.files.variant-test
(:require
[app.common.files.variant :as cfv]
[app.common.test-helpers.components :as thc]
[app.common.test-helpers.files :as thf]
[app.common.test-helpers.ids-map :as thi]
[app.common.test-helpers.shapes :as ths]
[app.common.test-helpers.variants :as thv]
[clojure.test :as t]))
(t/use-fixtures :each thi/test-fixture)
(t/deftest test-collect-variant-properties-for-main
(let [file (-> (thf/sample-file :file1)
(thv/add-variant :v01 :c01 :m01 :c02 :m02))
page (thf/current-page file)
main (ths/get-shape file :m01)
comp (thc/get-component file :c01)
collected (cfv/collect-variant-properties (:data file) (:objects page) main)]
(t/is (= {(:id comp) (:variant-properties comp)} collected))))
(t/deftest test-collect-variant-properties-for-container
(let [file (-> (thf/sample-file :file1)
(thv/add-variant :v01 :c01 :m01 :c02 :m02))
page (thf/current-page file)
container (ths/get-shape file :v01)
c01 (thc/get-component file :c01)
c02 (thc/get-component file :c02)
collected (cfv/collect-variant-properties (:data file) (:objects page) container)]
(t/is (= {(:id c01) (:variant-properties c01)
(:id c02) (:variant-properties c02)}
collected))))
(t/deftest test-collect-variant-properties-for-plain-shape
(let [file (-> (thf/sample-file :file1)
(ths/add-sample-shape :rect1 :type :rect))
page (thf/current-page file)
shape (ths/get-shape file :rect1)
collected (cfv/collect-variant-properties (:data file) (:objects page) shape)]
(t/is (= {} collected))))

View File

@ -0,0 +1,138 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns common-tests.files-migrations-0033-test
"Regression coverage for the shape tree traversal of the
`0033-normalize-main-instance-names` migration.
The migration walks a container from its root down, following the
`:shapes` vectors. A file can hold a `:shapes` entry pointing to a
shape that is not in the `:objects` map (a dangling reference). The
traversal must skip it instead of storing a nil shape into the
`:objects` map, and must keep normalizing the names of the shapes
that do exist."
(:require
[app.common.files.migrations :as cfm]
[app.common.types.shape :as cts]
[app.common.uuid :as uuid]
[clojure.test :as t]))
(def ^:private migration "0033-normalize-main-instance-names")
(defn- make-root
[children]
(cts/setup-shape {:id uuid/zero
:type :frame
:name "Root"
:shapes children}))
(defn- make-main-instance
"A frame declared as the main instance of a component."
[main-id file-id component-id name]
(-> (cts/setup-shape {:id main-id :type :frame :name name})
(assoc :main-instance true
:component-id component-id
:component-file file-id)))
(defn- page-data
"File data with a single page owning `objects`."
[page-id objects]
{:pages-index {page-id {:id page-id :objects objects}}})
(defn- component-data
"File data with a single component owning `objects`."
[component-id objects]
{:components {component-id {:id component-id :objects objects}}})
;; --- The guard: a missing child must not become a nil shape
(t/deftest missing-child-of-a-page-does-not-create-a-nil-shape
(let [page-id (uuid/next)
child-id (uuid/next)
root (make-root [child-id])
data (page-data page-id {uuid/zero root})
migrated (cfm/migrate-data data migration)
objects' (get-in migrated [:pages-index page-id :objects])]
(t/is (not (contains? objects' nil))
"no nil key is added to the objects map")
(t/is (= 1 (count objects'))
"the objects map keeps only the shapes that exist")
(t/is (= root (get objects' uuid/zero))
"the root is left untouched")))
(t/deftest missing-child-of-a-component-does-not-create-a-nil-shape
(let [component-id (uuid/next)
child-id (uuid/next)
root (make-root [child-id])
data (component-data component-id {uuid/zero root})
migrated (cfm/migrate-data data migration)
objects' (get-in migrated [:components component-id :objects])]
(t/is (not (contains? objects' nil))
"no nil key is added to the objects map")
(t/is (= 1 (count objects'))
"the objects map keeps only the shapes that exist")
(t/is (= root (get objects' uuid/zero))
"the component root is left untouched")))
(t/deftest missing-child-deep-in-the-tree-does-not-create-a-nil-shape
(let [page-id (uuid/next)
frame-id (uuid/next)
missing-id (uuid/next)
frame (-> (cts/setup-shape {:id frame-id :type :frame :name "Frame"})
(assoc :shapes [missing-id]))
data (page-data page-id {uuid/zero (make-root [frame-id])
frame-id frame})
migrated (cfm/migrate-data data migration)
objects' (get-in migrated [:pages-index page-id :objects])]
(t/is (not (contains? objects' nil))
"no nil key is added to the objects map")
(t/is (= frame (get objects' frame-id))
"the parent of the missing child is left untouched")))
;; --- The behavior: names keep being normalized next to a missing child
(t/deftest main-instance-of-a-page-is-renamed-next-to-a-missing-child
(let [file-id (uuid/next)
page-id (uuid/next)
component-id (uuid/next)
main-id (uuid/next)
missing-id (uuid/next)
main-shape (make-main-instance main-id file-id component-id "Group/Name")
data (assoc (page-data page-id {uuid/zero (make-root [missing-id main-id])
main-id main-shape})
:components {component-id {:id component-id
:name "Name"
:path "Group"
:main-instance-id main-id}})
migrated (cfm/migrate-data data migration)
objects' (get-in migrated [:pages-index page-id :objects])]
(t/is (not (contains? objects' nil))
"no nil key is added to the objects map")
(t/is (= "Group / Name" (get-in objects' [main-id :name]))
"the main instance takes the path and name of its component")))
(t/deftest main-instance-of-a-component-is-renamed-next-to-a-missing-child
(let [file-id (uuid/next)
component-id (uuid/next)
main-id (uuid/next)
missing-id (uuid/next)
main-shape (make-main-instance main-id file-id component-id "Group/Name")
data (component-data component-id {uuid/zero (make-root [missing-id main-id])
main-id main-shape})
data (assoc-in data [:components component-id :name] "Name")
data (assoc-in data [:components component-id :path] "Group")
data (assoc-in data [:components component-id :main-instance-id] main-id)
migrated (cfm/migrate-data data migration)
objects' (get-in migrated [:components component-id :objects])]
(t/is (not (contains? objects' nil))
"no nil key is added to the objects map")
(t/is (= "Group / Name" (get-in objects' [main-id :name]))
"the main instance takes the path and name of its component")))

View File

@ -10,10 +10,186 @@
[app.common.geom.matrix :as gmt]
[app.common.geom.rect :as grc]
[app.common.types.file :as ctf]
[app.common.types.objects-map :as omap]
[app.common.types.shape :as cts]
[app.common.uuid :as uuid]
[clojure.test :as t]))
;; ======================================================================
;; 0035-fix-dangling-children
;; A shape can list a child id that is not present in the `:objects` map
;; of the page or component owning it (a dangling reference). Any
;; traversal of the shape tree trips on it, and the file becomes
;; unopenable. The migration drops those ids and keeps every shape that
;; really exists.
;; ======================================================================
(def ^:private migration "0035-fix-dangling-children")
(defn- make-root
[children]
(cts/setup-shape {:id uuid/zero
:type :frame
:name "Root"
:shapes children}))
(defn- make-frame
[frame-id child-ids]
(-> (cts/setup-shape {:id frame-id :type :frame :name "Frame"})
(assoc :shapes child-ids)))
(defn- page-data
[page-id objects]
{:pages-index {page-id {:id page-id :objects objects}}})
(defn- component-data
[component-id objects]
{:components {component-id {:id component-id :objects objects}}})
(t/deftest removes-the-dangling-children-of-a-page
(let [page-id (uuid/next)
frame-id (uuid/next)
child-id (uuid/next)
frame (make-frame frame-id [child-id])
data (page-data page-id {uuid/zero (make-root [frame-id child-id])
frame-id frame})
migrated (cfm/migrate-data data migration)
objects' (get-in migrated [:pages-index page-id :objects])]
(t/is (= [frame-id] (get-in objects' [uuid/zero :shapes]))
"the dangling id is removed from the root children list")
(t/is (= [] (get-in objects' [frame-id :shapes]))
"the dangling id is removed from the frame children list")
(t/is (= 2 (count objects'))
"every shape that exists is kept")
(t/is (some? (get objects' frame-id))
"the frame itself is kept")))
(t/deftest removes-the-dangling-children-of-a-component
(let [component-id (uuid/next)
frame-id (uuid/next)
child-id (uuid/next)
frame (cts/setup-shape {:id frame-id :type :frame :name "Frame"})
data (component-data component-id {uuid/zero (make-root [frame-id child-id])
frame-id frame})
migrated (cfm/migrate-data data migration)
objects' (get-in migrated [:components component-id :objects])]
(t/is (= [frame-id] (get-in objects' [uuid/zero :shapes]))
"the dangling id is removed from the children list")
(t/is (= 2 (count objects'))
"every shape that exists is kept")
(t/is (= frame (get objects' frame-id))
"the existing shape is left untouched")))
(t/deftest migration-works-with-an-objects-map
;; The backend stores the page :objects as an ObjectsMap (the lazy
;; representation enabled by the fdata/objects-map feature). It is an
;; IPersistentMap that does not implement IFn on the JVM: a migration
;; must consult it with contains?/get, never by calling it as a
;; function.
(let [page-id (uuid/next)
frame-id (uuid/next)
child-id (uuid/next)
frame (make-frame frame-id [child-id])
objects (omap/wrap {uuid/zero (make-root [frame-id child-id])
frame-id frame})
data (page-data page-id objects)
migrated (cfm/migrate-data data migration)
objects' (get-in migrated [:pages-index page-id :objects])]
(t/is (= [frame-id] (get-in objects' [uuid/zero :shapes]))
"the dangling id is removed with the ObjectsMap representation")
(t/is (= [] (get-in objects' [frame-id :shapes]))
"the frame children list is pruned the same way")))
(t/deftest keeps-the-order-and-the-shapes-of-a-valid-tree
(let [page-id (uuid/next)
frame-id (uuid/next)
rect-id (uuid/next)
frame (cts/setup-shape {:id frame-id :type :frame :name "Frame"})
rect (cts/setup-shape {:id rect-id :type :rect :name "Rect"})
data (page-data page-id {uuid/zero (make-root [frame-id rect-id])
frame-id frame
rect-id rect})
migrated (cfm/migrate-data data migration)
objects' (get-in migrated [:pages-index page-id :objects])]
(t/is (= [frame-id rect-id] (get-in objects' [uuid/zero :shapes]))
"the children list is kept as it was")
(t/is (= data migrated)
"a tree without dangling references is not modified")))
(t/deftest leaves-shapes-with-empty-children-alone
(let [page-id (uuid/next)
frame (cts/setup-shape {:id (uuid/next) :type :frame :name "Frame"})
data (page-data page-id {uuid/zero (make-root [])
(:id frame) frame})
migrated (cfm/migrate-data data migration)]
(t/is (= data migrated)
"empty children vectors are left alone")))
(t/deftest leaves-shapes-without-children-key-alone
(let [page-id (uuid/next)
rect-id (uuid/next)
rect (cts/setup-shape {:id rect-id :type :rect :name "Rect"})
data (page-data page-id {uuid/zero (make-root [])
rect-id rect})
migrated (cfm/migrate-data data migration)]
(t/is (= data migrated)
"shapes without a children key are left alone")
(t/is (not (contains? (get-in migrated [:pages-index page-id :objects rect-id]) :shapes))
"no children key is created on the shape")))
(t/deftest migration-is-idempotent
(let [page-id (uuid/next)
child-id (uuid/next)
data (page-data page-id {uuid/zero (make-root [child-id])})
migrated (cfm/migrate-data data migration)]
(t/is (= migrated (cfm/migrate-data migrated migration))
"applying the migration twice does not change the result")))
(t/deftest file-with-dangling-children-passes-the-whole-migration
(let [file-id (uuid/next)
page-id (uuid/next)
frame-id (uuid/next)
missing-id (uuid/next)
frame (make-frame frame-id [missing-id])
base (ctf/make-file-data file-id page-id)
data (-> base
(assoc-in [:pages-index page-id :objects uuid/zero]
(make-root [frame-id missing-id]))
(assoc-in [:pages-index page-id :objects frame-id] frame))
;; The file reached the state before 0033 ran, so 0033, 0034 and
;; 0035 are all pending, exactly as in the reported failure.
file {:id file-id
:data data
:migrations (disj cfm/available-migrations
"0033-normalize-main-instance-names"
"0034-fix-path-content-offset"
migration)}]
(t/is (cfm/need-migration? file)
"the file is scheduled for migration")
(let [migrated (cfm/migrate-file file {})
objects' (get-in migrated [:data :pages-index page-id :objects])]
(t/is (contains? (:migrations migrated) migration)
"the migration is recorded as applied")
(t/is (not (contains? objects' nil))
"the migrated data has no nil shape entry")
(t/is (= [frame-id] (get-in objects' [uuid/zero :shapes]))
"the dangling reference of the root is removed")
(t/is (= [] (get-in objects' [frame-id :shapes]))
"the dangling reference of the frame is removed")
(t/is (some? (get objects' frame-id))
"the frame that exists is kept"))))
;; ======================================================================
;; 0035-fix-shape-svg-attrs-and-defs
;; Stored svg attribute keys are kebab-case (issue #11947). Old birth and
;; the cleaner era stored camelCase React prop names, which the v3 reader
@ -22,6 +198,7 @@
;; through the whitelist+kebab transform, svg-raw `:content` attrs are
;; kebab-ized spelling-only. It replaces "0021-fix-shape-svg-attrs" (id
;; withdrawn): files that recorded it migrate again through this one.
;; ======================================================================
(def ^:private migration-id
"0035-fix-shape-svg-attrs-and-defs")

View File

@ -0,0 +1,202 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns common-tests.logic.variants-paste-test
(:require
[app.common.data :as d]
[app.common.files.changes-builder :as pcb]
[app.common.files.helpers :as cfh]
[app.common.files.validate :as cfv]
[app.common.geom.point :as gpt]
[app.common.logic.libraries :as cll]
[app.common.logic.variant-properties :as clvp]
[app.common.test-helpers.components :as thc]
[app.common.test-helpers.compositions :as tho]
[app.common.test-helpers.files :as thf]
[app.common.test-helpers.ids-map :as thi]
[app.common.test-helpers.shapes :as ths]
[app.common.test-helpers.variants :as thv]
[app.common.types.component :as ctk]
[app.common.types.variant :as ctv]
[clojure.test :as t]))
(t/use-fixtures :each thi/test-fixture)
(defn- paste-variants-into-container
"Paste individual variants from file-a into file-b's variant container.
Omit file-a from `libraries` to hit the unknown-component path; include
it to hit `duplicate-variant`. Optional `variant-props` mirrors clipboard
collection (Task 2)."
[file-a file-b source-labels target-container-label
& {:keys [libraries variant-props]
:or {libraries {(:id file-b) file-b}}}]
(let [page-a (thf/current-page file-a)
page-b (thf/current-page file-b)
container-b (ths/get-shape file-b target-container-label)
parent-id (:id container-b)
source-ids (mapv thi/id source-labels)
tree-ids (into []
(mapcat #(cfh/get-children-ids-with-self (:objects page-a) %))
source-ids)
pasted (->> tree-ids
(map (d/getf (:objects page-a)))
(map (fn [shape]
(cond-> shape
(contains? (set source-ids) (:id shape))
(assoc :parent-id parent-id
:frame-id parent-id))))
(d/index-by :id))
all-objects (merge (:objects page-b) pasted)]
(-> (pcb/empty-changes nil)
(pcb/with-page-id (:id page-b))
(pcb/with-library-data (:data file-b))
(pcb/with-objects all-objects)
(cll/generate-duplicate-changes all-objects
page-b
(set source-ids)
(gpt/point 0 0)
libraries
(:data file-b)
(:id file-b)
{:variant-props variant-props}))))
(defn- property-names
[component]
(mapv :name (:variant-properties component)))
(defn- variant-mains-in
[file container-label]
(let [container (ths/get-shape file container-label)]
(->> (vals (:objects (thf/current-page file)))
(filter ctk/main-instance?)
(filter #(= (:id container) (:variant-id %))))))
(defn- other-set-file
[]
(-> (thf/sample-file :file-b)
(thv/add-variant-two-properties :vb :cb1 :mb1 :cb2 :mb2)
(ths/update-shape :vb :name "Other")
(ths/update-shape :mb1 :name "Other")
(ths/update-shape :mb2 :name "Other")
(thc/update-component :cb1 {:name "Other"})
(thc/update-component :cb2 {:name "Other"})))
(t/deftest test-paste-unknown-variant-into-container-validates
;; #10704: paste individual variants from file A into file B's set
;; when A is not in libraries must leave a valid file (desired behavior).
(let [file-a (-> (thf/sample-file :file-a)
(thv/add-variant :va :ca1 :ma1 :ca2 :ma2))
file-b (other-set-file)
changes (paste-variants-into-container file-a file-b [:ma1 :ma2] :vb)
file' (thf/apply-changes file-b changes :validate? false)
errors (cfv/validate-file file' {})
pasted (->> (variant-mains-in file' :vb)
(remove #(#{(thi/id :mb1) (thi/id :mb2)} (:id %))))]
(t/is (nil? errors))
(doseq [shape pasted]
(let [comp (thc/get-component-by-id file' (:component-id shape))]
(t/is (= ["Property 1" "Property 2"] (property-names comp)))
(t/is (= "Other" (:name shape)))
(t/is (= (:variant-name shape)
(ctv/properties-to-name (:variant-properties comp))))))))
(t/deftest test-duplicate-variant-in-own-container-bumps-last-value
;; Same-container duplicate via duplicate-variant: originals keep their
;; props; the new main gets a bumped last-property value. Does not cover
;; generate-make-shapes-variant's already-aligned early-exit (new mains
;; enter with a nil variant-id).
(let [file (-> (thf/sample-file :file1)
(thv/add-variant :v01 :c01 :m01 :c02 :m02))
c01-props (:variant-properties (thc/get-component file :c01))
c02-props (:variant-properties (thc/get-component file :c02))
file' (tho/duplicate-shape file :m01)
mains (variant-mains-in file' :v01)
new-main (->> mains
(remove #(#{(thi/id :m01) (thi/id :m02)} (:id %)))
first)
new-comp (thc/get-component-by-id file' (:component-id new-main))]
(thf/validate-file! file')
(t/is (= 3 (count mains)))
(t/is (= c01-props (:variant-properties (thc/get-component file' :c01))))
(t/is (= c02-props (:variant-properties (thc/get-component file' :c02))))
(t/is (= ["Property 1"] (property-names new-comp)))
(t/is (= (str ctv/value-prefix 3)
(-> new-comp :variant-properties last :value)))))
(t/deftest test-make-shapes-variant-skips-when-already-aligned
;; Locks the narrowed early-exit: a fully aligned child must not have its
;; props or variant-name rewritten.
(let [file (-> (thf/sample-file :file1)
(thv/add-variant :v01 :c01 :m01 :c02 :m02))
page (thf/current-page file)
container (ths/get-shape file :v01)
shape (ths/get-shape file :m01)
before (:variant-properties (thc/get-component file :c01))
before-vn (:variant-name shape)
changes (-> (pcb/empty-changes nil)
(pcb/with-page page)
(pcb/with-objects (:objects page))
(pcb/with-library-data (:data file))
(clvp/generate-make-shapes-variant [shape] container))
file' (thf/apply-changes file changes)
shape' (ths/get-shape-by-id file' (:id shape))]
(t/is (= before (:variant-properties (thc/get-component file' :c01))))
(t/is (= before-vn (:variant-name shape')))))
(t/deftest test-paste-known-variant-into-other-container-aligns
;; file A present in libraries → duplicate-variant path; props must match
;; the target set and the file must validate.
(let [file-a (-> (thf/sample-file :file-a)
(thv/add-variant :va :ca1 :ma1 :ca2 :ma2))
file-b (other-set-file)
libraries {(:id file-a) file-a
(:id file-b) file-b}
changes (paste-variants-into-container file-a file-b [:ma1 :ma2] :vb
:libraries libraries)
file' (thf/apply-changes file-b changes :validate? false)
errors (cfv/validate-file file' {})
pasted (->> (variant-mains-in file' :vb)
(remove #(#{(thi/id :mb1) (thi/id :mb2)} (:id %))))]
(t/is (nil? errors))
(t/is (= 2 (count pasted)))
(doseq [shape pasted]
(let [comp (thc/get-component-by-id file' (:component-id shape))]
(t/is (= ["Property 1" "Property 2"] (property-names comp)))
(t/is (= "Other" (:name shape)))
(t/is (= (:variant-name shape)
(ctv/properties-to-name (:variant-properties comp))))))))
(t/deftest test-paste-unknown-preserves-matching-property-values
;; Clipboard props (Task 2) whose names exist on the target survive regenerate.
(let [file-a (-> (thf/sample-file :file-a)
(thv/add-variant :va :ca1 :ma1 :ca2 :ma2))
file-b (other-set-file)
props {(:id (thc/get-component file-a :ca1))
(:variant-properties (thc/get-component file-a :ca1))
(:id (thc/get-component file-a :ca2))
(:variant-properties (thc/get-component file-a :ca2))}
changes (paste-variants-into-container file-a file-b [:ma1 :ma2] :vb
:variant-props props)
file' (thf/apply-changes file-b changes :validate? false)
pasted (->> (variant-mains-in file' :vb)
(remove #(#{(thi/id :mb1) (thi/id :mb2)} (:id %)))
(map #(thc/get-component-by-id file' (:component-id %))))
values (->> pasted
(mapcat :variant-properties)
(filter #(= "Property 1" (:name %)))
(map :value)
set)]
(thf/validate-file! file')
(t/is (contains? values "Value1"))
(t/is (contains? values "Value2"))))

View File

@ -22,6 +22,7 @@
[common-tests.files-changes-test]
[common-tests.files-migrations-0025-test]
[common-tests.files-migrations-0026-test]
[common-tests.files-migrations-0033-test]
[common-tests.files-migrations-0034-test]
[common-tests.files-migrations-0035-test]
[common-tests.files-migrations-test]
@ -32,6 +33,7 @@
[common-tests.files.shapes-builder-test]
[common-tests.files.tokens-test]
[common-tests.files.validate-test]
[common-tests.files.variant-test]
[common-tests.geom-align-test]
[common-tests.geom-bounds-layout-nil-test]
[common-tests.geom-bounds-map-test]
@ -81,6 +83,7 @@
[common-tests.logic.text-touched-test]
[common-tests.logic.token-apply-test]
[common-tests.logic.token-test]
[common-tests.logic.variants-paste-test]
[common-tests.logic.variants-reparent-test]
[common-tests.logic.variants-switch-test]
[common-tests.logic.variants-test]
@ -135,6 +138,7 @@
'common-tests.files-changes-test
'common-tests.files-migrations-0025-test
'common-tests.files-migrations-0026-test
'common-tests.files-migrations-0033-test
'common-tests.files-migrations-0034-test
'common-tests.files-migrations-0035-test
'common-tests.files-migrations-test
@ -145,6 +149,7 @@
'common-tests.files.shapes-builder-test
'common-tests.files.tokens-test
'common-tests.files.validate-test
'common-tests.files.variant-test
'common-tests.geom-align-test
'common-tests.geom-bounds-layout-nil-test
'common-tests.geom-bounds-map-test
@ -194,6 +199,7 @@
'common-tests.logic.text-touched-test
'common-tests.logic.token-apply-test
'common-tests.logic.token-test
'common-tests.logic.variants-paste-test
'common-tests.logic.variants-reparent-test
'common-tests.logic.variants-switch-test
'common-tests.logic.variants-test

View File

@ -160,10 +160,8 @@
(let [page-id (:current-page-id state)
data (dsh/lookup-file-data state)
objects (-> (dsh/get-page data page-id)
(get :objects))
components (cfv/find-variant-components data objects (:id shape))]
(into {} (map (juxt :id :variant-properties) components))))
(get :objects))]
(cfv/collect-variant-properties data objects shape)))
;; Collects all the items together and split images into a
@ -179,7 +177,8 @@
(some? images)
(update :images into images)
(ctk/is-variant-container? item)
(or (ctk/is-variant-container? item)
(ctk/is-variant-main? item))
(update :variant-properties merge (collect-variants state item))))
(maybe-translate [shape objects parent-frame-id]

View File

@ -10,6 +10,7 @@ Subcommands:
boom Generate the :boom: section from `breaking change` issues
security Generate the :lock: section from advisories fixed in a version
cross-ref Compare merged milestone PRs against the changelog section
dev-only-bugs List milestone bugs that never reached production
report Generate the anomaly report (CHANGES-ISSUES.md)
Usage:
@ -20,6 +21,7 @@ Usage:
python3 scripts/changelog.py security "2.19.1"
python3 scripts/changelog.py security "2.19.1" --write
python3 scripts/changelog.py cross-ref "2.16.0"
python3 scripts/changelog.py dev-only-bugs "2.16.0" [--apply]
python3 scripts/changelog.py report "2.16.0"
python3 scripts/changelog.py report "2.16.0" --changes CHANGES.md --output CHANGES-ISSUES.md
@ -615,6 +617,103 @@ def cmd_cross_ref(args: argparse.Namespace) -> None:
print(f" #{pr['number']} {pr['title'][:80]}")
# ─────────────────────────────────────────────
# Subcommand: dev-only-bugs
# ─────────────────────────────────────────────
# Bugs found and fixed while building a feature of the same, unreleased
# milestone never reached end users and stay out of the changelog.
AUTO_PARENT_TYPES = {"Feature", "Enhancement"}
VERSION_FIELD_RE = re.compile(
r"^#{2,4}\s*(?:affected versions?|versions?|environment)[^\n]*\n(.*?)(?=^#{2,4}\s|\Z)",
re.IGNORECASE | re.MULTILINE | re.DOTALL,
)
DEV_ENV_RE = re.compile(r"\b(?:hourly|develop|staging|pre|unreleased)\b", re.IGNORECASE)
PROD_ENV_RE = re.compile(
r"\b(?:production|pro|main|saas|design\.penpot\.app|self[- ]?hosted?)\b",
re.IGNORECASE,
)
VERSION_RE = re.compile(r"\b(\d+)\.(\d+)(?:\.\d+)?")
def version_field(body: str | None) -> str:
"""Return the "Affected versions" / "Environment" text of an issue body."""
match = VERSION_FIELD_RE.search(body or "")
return " ".join(match.group(1).split()) if match else ""
def only_unreleased_envs(text: str, milestone: str) -> bool:
"""True when the version text names unreleased environments only:
hourly, develop, staging, PRE or a version of this milestone. Any
production hint or other version number keeps the bug in the changelog."""
current = tuple(milestone.split(".")[:2])
versions = [m for m in VERSION_RE.findall(text)]
dev = bool(DEV_ENV_RE.search(text)) or current in versions
prod = bool(PROD_ENV_RE.search(text)) or any(v != current for v in versions)
return dev and not prod
def find_dev_only_bugs(milestone: str, issues: list[dict]) -> tuple[list[dict], list[dict]]:
"""Split the milestone's changelog bugs that never reached production.
``issues`` come from ``gh.py issue`` (they carry ``body`` and
``parent``). Returns ``(auto, suggested)``: ``auto`` bugs are
sub-issues of a Feature or Enhancement of this milestone and get the
`no changelog` label; ``suggested`` ones need a human to confirm."""
auto, suggested = [], []
for issue in issues:
if issue.get("issue_type") != "Bug" or issue.get("state") != "CLOSED":
continue
if {"no changelog", "release blocker"} & set(issue.get("labels", [])):
continue
parent = issue.get("parent") or {}
entry = {"issue": issue["number"], "issue_title": issue.get("title", "")}
if parent.get("milestone") == milestone:
entry["reason"] = f"sub-issue of {parent.get('issue_type')} #{parent['number']} in {milestone}"
(auto if parent.get("issue_type") in AUTO_PARENT_TYPES else suggested).append(entry)
continue
# `regression` says something that worked broke, not where: it
# also marks regressions users hit in production.
if "regression" in issue.get("labels", []):
entry["reason"] = "labelled `regression`"
suggested.append(entry)
continue
text = version_field(issue.get("body"))
if only_unreleased_envs(text, milestone):
entry["reason"] = f'only seen on "{text[:80]}"'
suggested.append(entry)
return auto, suggested
def fetch_milestone_bugs(milestone: str) -> list[dict]:
"""Closed milestone bugs with body and parent (two batched lookups)."""
issues = json.loads(run_gh_py("issues", milestone))
numbers = [str(i["number"]) for i in issues if i.get("issue_type") == "Bug"]
return json.loads(run_gh_py("issue", *numbers)) if numbers else []
def cmd_dev_only_bugs(args: argparse.Namespace) -> None:
"""List bugs that never reached production; --apply labels the sure ones."""
auto, suggested = find_dev_only_bugs(args.milestone, fetch_milestone_bugs(args.milestone))
print(f"Label `no changelog` ({len(auto)}):")
for e in auto:
print(f" #{e['issue']} {e['issue_title'][:70]} ({e['reason']})")
print(f"Candidates to confirm ({len(suggested)}):")
for e in suggested:
print(f" #{e['issue']} {e['issue_title'][:70]} ({e['reason']})")
if args.apply:
for e in auto:
subprocess.run(
["gh", "issue", "edit", str(e["issue"]), "--repo", REPO,
"--add-label", "no changelog"],
check=True, capture_output=True,
)
print(f"Labelled {len(auto)} issues `no changelog`", file=sys.stderr)
# ─────────────────────────────────────────────
# Subcommand: report
# ─────────────────────────────────────────────
@ -872,6 +971,9 @@ def cmd_report(args: argparse.Namespace) -> None:
):
anomalies_d.append({"version": ver, "line": line[:100]})
# Bugs that may never have reached production: warnings, not anomalies.
dev_auto, dev_suggested = find_dev_only_bugs(milestone, fetch_milestone_bugs(milestone))
def fmt_ms(ms: str | None) -> str:
return ms if ms else "_none_"
@ -1033,6 +1135,26 @@ def cmd_report(args: argparse.Namespace) -> None:
"✅ No highlight gaps found. All released X.Y.0 versions have properly referenced :rocket: entries.\n\n"
)
if dev_auto or dev_suggested:
f.write("## Bugs that may never have reached production\n\n")
f.write(
"These are warnings, not anomalies. Bugs found while building "
"a feature of this release never affected end users. Add the "
"`no changelog` label to the ones you confirm and re-run.\n\n"
)
if dev_auto:
f.write(
"### Sub-issues of a feature of this milestone (run `dev-only-bugs --apply`)\n\n"
)
for e in dev_auto:
f.write(f"- {issue_link_title(e['issue'], e['issue_title'][:80])}: {e['reason']}\n")
f.write("\n")
if dev_suggested:
f.write("### Candidates to confirm\n\n")
for e in dev_suggested:
f.write(f"- {issue_link_title(e['issue'], e['issue_title'][:80])}: {e['reason']}\n")
f.write("\n")
f.write("---\n\n")
f.write("## Context\n\n")
f.write(f"- Milestone: **{milestone}**\n")
@ -1100,6 +1222,15 @@ def main() -> None:
help="Path to the changelog file (default: CHANGES.md)")
p_cross.set_defaults(func=cmd_cross_ref)
p_dev = sub.add_parser(
"dev-only-bugs",
help="List milestone bugs that never reached production",
)
p_dev.add_argument("milestone", help="Milestone title (e.g. '2.16.0')")
p_dev.add_argument("--apply", action="store_true",
help="Label the sub-issues of a milestone feature `no changelog`")
p_dev.set_defaults(func=cmd_dev_only_bugs)
p_report = sub.add_parser(
"report", help="Generate the anomaly report (CHANGES-ISSUES.md)"
)

View File

@ -752,8 +752,10 @@ GQL_ISSUE_BY_NUMBER_QUERY_ITEM = """\
number
title
state
body
milestone {{ title }}
issueType {{ name }}
parent {{ number issueType {{ name }} milestone {{ title }} }}
labels(first: 20) {{ nodes {{ name }} }}
assignees(first: 10) {{ nodes {{ login }} }}
closedByPullRequestsReferences(first: 5) {{ nodes {{ number }} }}
@ -775,12 +777,19 @@ def issue_node_to_dict(node: dict) -> dict:
"""Convert a GraphQL issue node to the shared issue dict shape."""
issue_type = node.get("issueType")
ms = node.get("milestone")
parent = node.get("parent")
return {
"number": node["number"],
"title": node["title"],
"state": node["state"],
"body": node.get("body"),
"milestone": ms["title"] if ms else None,
"issue_type": issue_type["name"] if issue_type else None,
"parent": {
"number": parent["number"],
"issue_type": (parent.get("issueType") or {}).get("name"),
"milestone": (parent.get("milestone") or {}).get("title"),
} if parent else None,
"labels": [lbl["name"] for lbl in node["labels"]["nodes"]],
"assignees": node_assignees(node),
"closing_prs": [pr["number"] for pr in node["closedByPullRequestsReferences"]["nodes"]],

View File

@ -657,5 +657,61 @@ class CmdSecurityTests(unittest.TestCase):
self.assertIn("will not be listed", err)
def bug(number, body="", parent=None, labels=()):
return {
"number": number,
"title": f"Bug {number}",
"state": "CLOSED",
"issue_type": "Bug",
"labels": list(labels),
"body": body,
"parent": parent,
}
class OnlyUnreleasedEnvsTests(unittest.TestCase):
def check(self, text):
return ch.only_unreleased_envs(text, "2.19.0")
def test_dev_environments(self):
for text in ["HOURLY", "https://hourly.penpot.dev/", "PRE & hourly",
"staging (current)", "develop (2.19.0 unreleased)",
"2.19.0-RC1 (develop)", "2.19.0"]:
with self.subTest(text=text):
self.assertTrue(self.check(text))
def test_production_hints(self):
for text in ["PRO", "pre/pro/hourly", "develop / staging / main",
"2.18.0-RC5", "2.18.1, 2.19.0-RC1", "self-hosted: 2.17.2",
"design.penpot.app and self-host"]:
with self.subTest(text=text):
self.assertFalse(self.check(text))
def test_no_signal(self):
for text in ["", "_No response_", "-", "development build"]:
with self.subTest(text=text):
self.assertFalse(self.check(text))
class FindDevOnlyBugsTests(unittest.TestCase):
def test_splits_auto_and_suggested(self):
issues = [
bug(1, parent={"number": 10, "issue_type": "Feature", "milestone": "2.19.0"}),
bug(2, parent={"number": 11, "issue_type": "Enhancement", "milestone": "2.19.0"}),
bug(3, parent={"number": 12, "issue_type": "EPIC", "milestone": "2.19.0"}),
bug(4, parent={"number": 13, "issue_type": "Feature", "milestone": "2.18.0"}),
bug(5, body="### Affected versions\n\nhourly\n"),
bug(6, body="### Affected versions\n\n2.18.2\n"),
bug(7, parent={"number": 10, "issue_type": "Feature", "milestone": "2.19.0"},
labels=["no changelog"]),
{**bug(8, parent={"number": 10, "issue_type": "Feature", "milestone": "2.19.0"}),
"issue_type": "Enhancement"},
bug(9, body="### Affected versions\n\n2.18.2\n", labels=["regression"]),
]
auto, suggested = ch.find_dev_only_bugs("2.19.0", issues)
self.assertEqual([e["issue"] for e in auto], [1, 2])
self.assertEqual([e["issue"] for e in suggested], [3, 5, 9])
if __name__ == "__main__":
unittest.main(verbosity=2)