mirror of
https://github.com/penpot/penpot.git
synced 2026-11-18 16:12:24 +00:00
Compare commits
No commits in common. "develop" and "2.18.0-RC5" have entirely different histories.
develop
...
2.18.0-RC5
@ -1,461 +1,19 @@
|
|||||||
# Agentic development with opencode inside devenv
|
# Agent skills
|
||||||
|
|
||||||
This doc shows how to run AI-assisted development for Penpot inside the
|
This folder is the single home for the skills our coding agents use.
|
||||||
devenv with [opencode](https://opencode.ai). It covers the setup once,
|
Each skill is a folder with a `SKILL.md` inside — a short instruction
|
||||||
then points to the skills that drive daily work.
|
manual that an agent loads only when it needs it.
|
||||||
|
|
||||||
Full reference lives in the technical guide:
|
One copy serves every tool:
|
||||||
|
|
||||||
- [Dev environment](../docs/technical-guide/developer/devenv.md)
|
- **opencode** reads this folder directly.
|
||||||
|
- **Claude Code** reads it through the `.claude/skills` symlink.
|
||||||
|
- **Codex** reads it directly.
|
||||||
|
|
||||||
This file does not repeat those guides. It gives the short path and
|
To change how the agents behave, edit the `SKILL.md` here. There is no
|
||||||
leaves room for notes we add step by step.
|
second copy to keep in sync.
|
||||||
|
|
||||||
## TL;DR
|
## How the skills are organized
|
||||||
|
|
||||||
```bash
|
|
||||||
./manage.sh pull-devenv
|
|
||||||
./manage.sh run-devenv --ws 0 --attach
|
|
||||||
```
|
|
||||||
|
|
||||||
Then open a shell in the container tmux session, run `opencode` inside
|
|
||||||
~/penpot directory.
|
|
||||||
|
|
||||||
## 1. Introduction
|
|
||||||
|
|
||||||
The LLM client — opencode, Claude Code, or Codex — runs in a shell
|
|
||||||
inside the plain devenv container, with the repo mounted and the
|
|
||||||
skills in this folder driving the work. One client session per
|
|
||||||
workspace (`ws0` is the live repo, `ws1+` are sibling clones).
|
|
||||||
|
|
||||||
This doc is written around opencode, but Claude Code runs the same
|
|
||||||
way inside devenv and follows exactly the same flows. This is not
|
|
||||||
the "agentic devenv" (`--agentic`) from the technical guide, which
|
|
||||||
runs the client outside devenv and wires it in over MCP — here the
|
|
||||||
client lives inside the sandboxed devenv docker.
|
|
||||||
|
|
||||||
The content lives at the paths the ecosystem agreed on: `AGENTS.md` at the
|
|
||||||
repository root and the skills in this folder, `.agents/skills/`. Codex,
|
|
||||||
opencode, Cursor, Zed, Amp, omp and pi read both without any setup.
|
|
||||||
|
|
||||||
Claude Code reads `AGENTS.md` from 2.1.277, through the built-in
|
|
||||||
`agents-md` plugin, but only in a project that has no instruction file of
|
|
||||||
its own. A `CLAUDE.md`, `.claude/CLAUDE.md` or `CLAUDE.local.md` anywhere
|
|
||||||
from the repository root down to your working directory hands the whole
|
|
||||||
project back to `CLAUDE.md`, and this file's guidance stops arriving. Keep
|
|
||||||
none of them here. Personal steering goes in `AGENTS.local.md`, or in
|
|
||||||
`.claude/rules/*.md` and `~/.claude/CLAUDE.md`, which the plugin does not
|
|
||||||
count and which therefore leave the project guidance in place.
|
|
||||||
|
|
||||||
Skills are untouched by that plugin and still load from `.claude/skills`
|
|
||||||
alone, which is not committed, so link it once per clone:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mkdir -p .claude && ln -s ../.agents/skills .claude/skills
|
|
||||||
# or, with Node:
|
|
||||||
npx skills add ./.agents/skills --agent claude-code
|
|
||||||
```
|
|
||||||
|
|
||||||
Below 2.1.277, and on Bedrock, Vertex and Foundry where the fallback has
|
|
||||||
not arrived, add `ln -s AGENTS.md CLAUDE.md` and drop it once your client
|
|
||||||
has the feature. The auto-update channel decides when that is: `latest`,
|
|
||||||
the default, has it; `stable` runs about a week behind; Homebrew and the
|
|
||||||
Linux packages update by hand. Personal skills go in
|
|
||||||
`.agents/local/skills/<name>/`.
|
|
||||||
|
|
||||||
Every one of those paths is gitignored, so your own files keep working and
|
|
||||||
no checkout overwrites them. Inside the devenv, `ws0` sees the links from
|
|
||||||
the live checkout, and `ws1` and above are seeded on their first sync.
|
|
||||||
|
|
||||||
Unlike the agentic devenv, running the client inside the devenv
|
|
||||||
docker gives it full access to the live environment: every
|
|
||||||
dependency already resolved by the image, so the agent can write
|
|
||||||
and run tests directly, query the running PostgreSQL, and reach
|
|
||||||
the backend and frontend through nREPL — no proxies, no round
|
|
||||||
trips outside the container.
|
|
||||||
|
|
||||||
And if you later want vision, it is one MCP entry away — a
|
|
||||||
headless Playwright server in your `opencode.json`:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"$schema": "https://opencode.ai/config.json",
|
|
||||||
"mcp": {
|
|
||||||
"playwright": {
|
|
||||||
"type": "local",
|
|
||||||
"command": ["npx", "-y", "@playwright/mcp", "--headless"],
|
|
||||||
"enabled": true,
|
|
||||||
"env": {}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## 2. Quickstart: bring up devenv and run opencode inside
|
|
||||||
|
|
||||||
Pull once, then bring up the workspace you want (add `--ws 1`,
|
|
||||||
`--ws 2`, … for more):
|
|
||||||
|
|
||||||
```bash
|
|
||||||
./manage.sh pull-devenv
|
|
||||||
./manage.sh run-devenv --ws 0 --attach # ws0 (the live repo)
|
|
||||||
```
|
|
||||||
|
|
||||||
This attaches to the container tmux session. Open a new shell there
|
|
||||||
(`Ctrl+b c`), `cd` to the repo, and run opencode directly:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cd penpot;
|
|
||||||
opencode
|
|
||||||
```
|
|
||||||
|
|
||||||
One session drives exactly one workspace — for N parallel workspaces,
|
|
||||||
open one container shell and one opencode session per workspace.
|
|
||||||
|
|
||||||
Stop with `./manage.sh stop-devenv [--ws N | --all]`. Shared infra
|
|
||||||
stops when the last workspace stops.
|
|
||||||
|
|
||||||
|
|
||||||
## 3. Connecting providers
|
|
||||||
|
|
||||||
### Starting point: Zen free models, no login needed
|
|
||||||
|
|
||||||
The easiest way to try the setup is Zen's free models — they need no
|
|
||||||
login and no key. Just run opencode, pick a free model, and work.
|
|
||||||
|
|
||||||
That said, creating a Zen account and connecting with your API key is
|
|
||||||
still worth it from day one: it unlocks the full model list, spend
|
|
||||||
limits, auto-reload, and the Go-overflow fallback below. Connect
|
|
||||||
through the TUI:
|
|
||||||
|
|
||||||
- Run `/connect`, pick a provider, paste the key.
|
|
||||||
- Run `/models` to see what that provider offers.
|
|
||||||
|
|
||||||
### OpenCode Go (subscription, best value for daily use)
|
|
||||||
|
|
||||||
$10/month subscription with generous included usage — up to $60/month
|
|
||||||
of model consumption at published per-token rates, depending on the
|
|
||||||
model. Best value if you work mostly with open coding models
|
|
||||||
(GLM, Kimi, Qwen, DeepSeek, MiniMax, LongCat…).
|
|
||||||
|
|
||||||
The strong part: when you hit a model's monthly limit, Go can fall
|
|
||||||
back to your Zen balance instead of blocking (enable *Use balance* in
|
|
||||||
the console). So the setup many of us use is Go + Zen credit on top —
|
|
||||||
subscription first, pay-as-you-go overflow after.
|
|
||||||
|
|
||||||
### OpenCode Zen (pay per use)
|
|
||||||
|
|
||||||
Zen is the opencode team's gateway: curated models tested for coding
|
|
||||||
agents, fair prices, no markups, stable latency. You top up credit
|
|
||||||
and pay per request, with monthly spend limits and auto-reload.
|
|
||||||
|
|
||||||
Free usage is generous: the free models carry limits good enough for
|
|
||||||
real work, not just a quick taste. Worth knowing: brand-new,
|
|
||||||
unannounced models often show up on Zen first with a very generous
|
|
||||||
free quota so people try them — e.g. `0x Alpha`, which later turned
|
|
||||||
out to be GLM-5.3-Flash. Keep an eye on the free list; the newest
|
|
||||||
entry is often the best deal.
|
|
||||||
|
|
||||||
Two reasons to have a Zen account even with Go:
|
|
||||||
|
|
||||||
- It absorbs Go overflow (see above).
|
|
||||||
- Its free models let you try the whole setup before paying.
|
|
||||||
|
|
||||||
### OpenRouter (widest catalog)
|
|
||||||
|
|
||||||
If you already have an account, connect it: the widest model range in
|
|
||||||
one place. Trade-off is latency and occasional instability versus Zen,
|
|
||||||
which is tuned for coding agents.
|
|
||||||
|
|
||||||
Beyond code: OpenRouter also serves image, video, and audio models.
|
|
||||||
opencode itself cannot call those directly — it is built for code —
|
|
||||||
but a cheap model can quickly build you a small tool or script that
|
|
||||||
talks to them through the OpenRouter API. So if you also generate
|
|
||||||
content other than code and text, having OpenRouter connected is
|
|
||||||
worth it: the agent wires the plumbing for you.
|
|
||||||
|
|
||||||
### OpenAI (subscription or API key)
|
|
||||||
|
|
||||||
If you have an OpenAI subscription or API access, connect it — it
|
|
||||||
works very well as a daily driver alongside (or instead of) Go/Zen.
|
|
||||||
|
|
||||||
### Suggested combos
|
|
||||||
|
|
||||||
| Profile | Connect |
|
|
||||||
|---|---|
|
|
||||||
| Try it out | Nothing (Zen free models, no login) |
|
|
||||||
| Try it out, properly | Zen account + API key (free models + limits) |
|
|
||||||
| Daily use, best value | Go + Zen credit (overflow) |
|
|
||||||
| Widest model choice | Add OpenRouter |
|
|
||||||
| Already pay OpenAI | Add OpenAI account |
|
|
||||||
|
|
||||||
## 4. Recommended models
|
|
||||||
|
|
||||||
Personal picks from Andrey, current as of September 2026. Models come
|
|
||||||
and go, so treat this as a snapshot — the shape (one cheap solver,
|
|
||||||
one reviewer/planner, one explorer) matters more than the names.
|
|
||||||
|
|
||||||
| Model | Role | How often |
|
|
||||||
|---|---|---|
|
|
||||||
| Muse Spark 1.3 (`high`) | Main solver: plan, review, develop. Sharp and cheap — covers ~70% of coding tasks. | Daily |
|
|
||||||
| GLM-5.3-Flash | Reasoning all-rounder, now mostly code/plan reviewer and planner. | Daily |
|
|
||||||
| DeepSeek V4.1 Flash (`high`) | Explorer: code and idea exploration, sometimes development. Especially good at small bash/node utilities for repo chores and changelog updates. | Daily |
|
|
||||||
| LongCat 2.0 | Backup solver, occasional stand-in for Muse Spark 1.3. | Weekly |
|
|
||||||
| GPT-5.6 Luna | Alternative to DeepSeek Flash; pricier, unclear the extra cost pays off. | Rarely |
|
|
||||||
| Qwen3.8 Flash | As strong as the top three; used in rotation to avoid hammering one model. Less Go subsidy than the top picks, so mostly in overflow mode. | Overflow |
|
|
||||||
| MiMo-V2.5-Pro | Former main model; slightly pricier now next to Muse Spark / GLM-Flash / LongCat, and less Go subsidy — used in overflow. | Overflow |
|
|
||||||
| Kimi K3 | Heavy reasoning for hard reviews and plans. Expensive, ~1% of tasks. | Rarely |
|
|
||||||
| GLM-5.3 | Same slot as Kimi K3: hard reviews and plans only. | Rarely |
|
|
||||||
|
|
||||||
**TL;DR:** the first three (Muse Spark 1.3, GLM-5.3-Flash, DeepSeek
|
|
||||||
V4.1 Flash) are a good starting point.
|
|
||||||
|
|
||||||
## 5. Customizing your `opencode.json`
|
|
||||||
|
|
||||||
opencode merges config in this order (later wins):
|
|
||||||
|
|
||||||
1. Global: `~/.config/opencode/opencode.json` (on host, or the dir
|
|
||||||
mounted with `--opencode-config-dir` inside devenv —
|
|
||||||
see §9 Advanced usage).
|
|
||||||
2. Project: `opencode.json` at the repo root (gitignored on purpose —
|
|
||||||
use it to override the global entries for one workspace).
|
|
||||||
|
|
||||||
Below is a full working example of my personal config at the date of
|
|
||||||
writing this. It is only an example: define whatever subagents you
|
|
||||||
need, with whatever models you like or work with.
|
|
||||||
|
|
||||||
Copy it to `opencode.json` on the root of the repo:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"$schema": "https://opencode.ai/config.json",
|
|
||||||
"disabled_providers": ["amazon-bedrock"],
|
|
||||||
"subagent_depth": 2,
|
|
||||||
"agent": {
|
|
||||||
"compaction": {
|
|
||||||
"model": "opencode-go/deepseek-flash",
|
|
||||||
"variant": "high"
|
|
||||||
},
|
|
||||||
"title": {
|
|
||||||
"model": "opencode-go/deepseek-flash",
|
|
||||||
"variant": "low"
|
|
||||||
},
|
|
||||||
"explore": {
|
|
||||||
"model": "opencode-go/deepseek-flash",
|
|
||||||
"variant": "high"
|
|
||||||
},
|
|
||||||
"build": {
|
|
||||||
"prompt": "{file:.agents/prompts/engineer-agent-prompt.md}",
|
|
||||||
"permission": {
|
|
||||||
"external_directory": {
|
|
||||||
"/tmp/**": "allow"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"general": {
|
|
||||||
"prompt": "{file:.agents/prompts/engineer-agent-prompt.md}",
|
|
||||||
"permission": {
|
|
||||||
"external_directory": {
|
|
||||||
"/tmp/**": "allow"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"engineer-glm": {
|
|
||||||
"mode": "subagent",
|
|
||||||
"model": "opencode-go/glm-5.3-flash",
|
|
||||||
"variant": "high",
|
|
||||||
"prompt": "{file:.agents/prompts/engineer-agent-prompt.md}",
|
|
||||||
"permission": {
|
|
||||||
"*": "allow",
|
|
||||||
"task": {
|
|
||||||
"*": "allow"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"engineer-kimi": {
|
|
||||||
"mode": "subagent",
|
|
||||||
"model": "opencode/kimi-k3",
|
|
||||||
"variant": "high",
|
|
||||||
"prompt": "{file:.agents/prompts/engineer-agent-prompt.md}",
|
|
||||||
"permission": {
|
|
||||||
"*": "allow",
|
|
||||||
"task": {
|
|
||||||
"*": "allow"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
},
|
|
||||||
|
|
||||||
"engineer-qwen": {
|
|
||||||
"mode": "subagent",
|
|
||||||
"model": "opencode-go/qwen3.7-plus",
|
|
||||||
"variant": "high",
|
|
||||||
"prompt": "{file:.agents/prompts/engineer-agent-prompt.md}",
|
|
||||||
"permission": {
|
|
||||||
"*": "allow",
|
|
||||||
"task": {
|
|
||||||
"*": "allow"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
What the blocks mean:
|
|
||||||
|
|
||||||
- `compaction` / `title` / `explore`: cheap background agents. Keep
|
|
||||||
them on a fast model; `title` uses the `low` variant on purpose.
|
|
||||||
- `build` / `general`: the main agents. They load the shared prompt
|
|
||||||
`{file:.agents/prompts/engineer-agent-prompt.md}` and may only touch
|
|
||||||
`/tmp/**` outside the repo without asking for explicit permision.
|
|
||||||
- `engineer-*`: one subagent per model family, all with the same
|
|
||||||
prompt and full permissions (`"*": "allow"`). They purpose are
|
|
||||||
specially for delegate work to them because are defined to be used
|
|
||||||
only as subagents.
|
|
||||||
- `disabled_providers` / `subagent_depth`: global guards. Keep
|
|
||||||
`"$schema"` — opencode refuses to start if any field is wrong.
|
|
||||||
|
|
||||||
How the `engineer-*` subagents are actually used — delegating work to
|
|
||||||
them to keep the main context clean — is covered in §6 Common agentic
|
|
||||||
flows.
|
|
||||||
|
|
||||||
Note this is opencode-only: other clients have their own way of
|
|
||||||
defining subagents or helpers — or none at all.
|
|
||||||
|
|
||||||
## 6. Common agentic flows
|
|
||||||
|
|
||||||
Work happens two ways: directly in your session, or delegated to a
|
|
||||||
subagent. Besides the `engineer-*` subagents from §5 there is a
|
|
||||||
builtin `general` subagent. Delegating planning and review to a
|
|
||||||
subagent starts a fresh, clean context with a clean prompt instead
|
|
||||||
of growing the main session — the main lever for keeping context
|
|
||||||
small. To delegate without switching models, delegate to `general`.
|
|
||||||
|
|
||||||
### Issue / error report flow
|
|
||||||
|
|
||||||
1. **Frame the problem.** Enter Plan mode (TAB in opencode) and paste the
|
|
||||||
report with your intent: "investigate this and find the possible cause",
|
|
||||||
"investigate and tell me where this points", or "does this still apply?".
|
|
||||||
Explore until you and the agent roughly agree on the problem.
|
|
||||||
2. **Write the plan.** Run `/make-a-plan` — it executes in Build mode.
|
|
||||||
If you need to step in and answer something yourself, press TAB to
|
|
||||||
leave Build mode. Use Plan mode only when you want a hard guarantee
|
|
||||||
that the agent modifies no file under any circumstance. If you
|
|
||||||
explored with a weaker model but want a stronger one to write the
|
|
||||||
plan, switch models first or delegate:
|
|
||||||
`/make-a-plan delegate to @engineer-glm`.
|
|
||||||
3. **Iterate on the plan.** The plan is saved to `.agents/plans/`, so you
|
|
||||||
never depend on LLM memory: read the file directly, or run `/review-plan`
|
|
||||||
for a second opinion (delegation works here too). Complex plans deserve a
|
|
||||||
review; simple ones can skip it.
|
|
||||||
4. **Execute.** Run `/implement-plan`. It first prints the full picture —
|
|
||||||
whether it will create an issue and a branch, the execution style, and a
|
|
||||||
task checklist — and waits for your go-ahead. Say "step by step" to stop
|
|
||||||
after each task (one commit per task) so you can verify as it goes;
|
|
||||||
the default runs all tasks with one final commit.
|
|
||||||
5. **Land the work.** When it finishes, either push yourself and run
|
|
||||||
`/create-pr`, or loop `/review-code` → `/make-a-plan` →
|
|
||||||
`/implement-plan` until the findings are addressed, then push and
|
|
||||||
`/create-pr`. Nothing pushes for you — you always push from your shell.
|
|
||||||
|
|
||||||
> Note: `/implement-plan` checks the current branch. On a base branch
|
|
||||||
> (`main`, `develop`, `staging`) it creates a GitHub issue and a branch
|
|
||||||
> `issue-NNNN`; on an existing feature branch it continues there and
|
|
||||||
> creates nothing. The pre-run summary tells you which applies. Read the
|
|
||||||
> skill at `.agents/skills/implement-plan/SKILL.md` — it is
|
|
||||||
> self-explanatory.
|
|
||||||
|
|
||||||
### Big feature with multiple plans
|
|
||||||
|
|
||||||
When the work is too large for a single plan, tell `/make-a-plan`
|
|
||||||
up front: produce a high-level roadmap where each task will get its
|
|
||||||
own execution plan, and the roadmap doubles as the progress tracker.
|
|
||||||
|
|
||||||
From there the flow mirrors the issue flow above, one level down:
|
|
||||||
take each roadmap task in turn, write its own plan (`/make-a-plan`,
|
|
||||||
delegating when it helps), review it when the task is complex
|
|
||||||
(`/review-plan`), implement it (`/implement-plan`), and mark progress
|
|
||||||
on the roadmap as you land each piece.
|
|
||||||
|
|
||||||
### Plan file naming (base + derivatives)
|
|
||||||
|
|
||||||
Base plans live in `.agents/plans/` as `YYYY-MM-DD-<slug>.md`.
|
|
||||||
Derived plans reuse the parent basename verbatim and append one
|
|
||||||
suffix per level with `--`, with no new date — the parent prefix
|
|
||||||
keeps everything adjacent in `ls`:
|
|
||||||
|
|
||||||
- Review followup on implemented work:
|
|
||||||
`2026-09-14-paste-before-init-crash.md` →
|
|
||||||
`2026-09-14-paste-before-init-crash--review-01.md`
|
|
||||||
- Roadmap sub-plan (task number from the roadmap):
|
|
||||||
`2026-09-20-upload-pipeline-roadmap.md` →
|
|
||||||
`2026-09-20-upload-pipeline-roadmap--task-01-chunk-upload.md`
|
|
||||||
- Chained: `...--task-02-gc--review-01.md`
|
|
||||||
|
|
||||||
While a plan is still unimplemented, `/make-a-plan` revises it in
|
|
||||||
place. Once implemented and reviewed, it writes a new followup file.
|
|
||||||
Full rules live in the `planner` skill.
|
|
||||||
|
|
||||||
### Plan status lifecycle
|
|
||||||
|
|
||||||
Every plan carries `Status: draft | reviewed | done` plus an
|
|
||||||
append-only `Review Log` (UTC ISO 8601, one line per entry):
|
|
||||||
|
|
||||||
- `draft` — fresh from `/make-a-plan`.
|
|
||||||
- `reviewed` — review feedback incorporated. `/review-plan` stays
|
|
||||||
read-only; only your explicit "apply" makes `/make-a-plan` apply
|
|
||||||
the changes, flip the status, and log one line.
|
|
||||||
- `done` — `/implement-plan` closes it on completion, logging the
|
|
||||||
issue URL when one exists (never commit hashes).
|
|
||||||
|
|
||||||
A plan approved with no changes goes `draft` → `done` directly.
|
|
||||||
|
|
||||||
A plan is never staged or committed: the `done` status and its
|
|
||||||
`Review Log` are session bookkeeping, not part of the change. Their
|
|
||||||
absence from the commit is expected.
|
|
||||||
|
|
||||||
## 7. Connecting `gh` CLI with a token
|
|
||||||
|
|
||||||
The `create-issue` and `create-pr` flows need an authenticated `gh`
|
|
||||||
so they can run on their own. Create a fine-grained token with the
|
|
||||||
minimum scopes:
|
|
||||||
|
|
||||||
1. GitHub → Settings → Developer settings → Personal access tokens →
|
|
||||||
Fine-grained tokens → Generate new token.
|
|
||||||
2. Under Organization permissions, grant access to **Projects**.
|
|
||||||
3. Under Repository permissions, grant at least **Issues** and
|
|
||||||
**Pull requests**.
|
|
||||||
|
|
||||||
Then authenticate the CLI and follow the prompts:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
gh auth login
|
|
||||||
```
|
|
||||||
|
|
||||||
Verify with `gh auth status` (token lives in
|
|
||||||
`~/.config/gh/hosts.yml`). You still push from your own shell — the
|
|
||||||
agents only read and open issues and PRs.
|
|
||||||
|
|
||||||
## 8. Troubleshooting / FAQ
|
|
||||||
|
|
||||||
> TBD — filled in step by step as issues come up.
|
|
||||||
|
|
||||||
## 9. Advanced usage
|
|
||||||
|
|
||||||
### Personal agents and prompts without committing them here
|
|
||||||
|
|
||||||
Bind-mount a host dir over the container's `~/.config/opencode`:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
./manage.sh run-devenv --ws 0 --opencode-config-dir ../penpot-opencode
|
|
||||||
```
|
|
||||||
|
|
||||||
It applies at container creation, so changing it needs a stop + rerun
|
|
||||||
of that instance.
|
|
||||||
|
|
||||||
## Summary of available skills
|
|
||||||
|
|
||||||
### How the skills are organized
|
|
||||||
|
|
||||||
**Flows** are the six skills you invoke by name. Each one covers one step
|
**Flows** are the six skills you invoke by name. Each one covers one step
|
||||||
in the life of a change: plan it, review the plan, implement it, review
|
in the life of a change: plan it, review the plan, implement it, review
|
||||||
@ -470,7 +28,7 @@ issue, a commit. Flows call them, but they also work on their own.
|
|||||||
**Utilities** are small helpers for everyday work: search, file lookup,
|
**Utilities** are small helpers for everyday work: search, file lookup,
|
||||||
JSON, REPL access, and so on.
|
JSON, REPL access, and so on.
|
||||||
|
|
||||||
### Flows
|
## Flows
|
||||||
|
|
||||||
| Skill | What it does | When you would say |
|
| Skill | What it does | When you would say |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
@ -481,14 +39,14 @@ JSON, REPL access, and so on.
|
|||||||
| [`create-pr`](skills/create-pr/SKILL.md) | Opens a pull request for the current branch — with checks on base branch, commits, issue, and push state — or updates an existing PR's title and description. | "open a PR for this branch" |
|
| [`create-pr`](skills/create-pr/SKILL.md) | Opens a pull request for the current branch — with checks on base branch, commits, issue, and push state — or updates an existing PR's title and description. | "open a PR for this branch" |
|
||||||
| [`resolve-git-conflicts`](skills/resolve-git-conflicts/SKILL.md) | Untangles merge or rebase conflicts: explains both sides, proposes a resolution, applies it after you approve. Never runs `git rebase --continue`. | "resolve these conflicts" |
|
| [`resolve-git-conflicts`](skills/resolve-git-conflicts/SKILL.md) | Untangles merge or rebase conflicts: explains both sides, proposes a resolution, applies it after you approve. Never runs `git rebase --continue`. | "resolve these conflicts" |
|
||||||
|
|
||||||
### References
|
## References
|
||||||
|
|
||||||
| Skill | What it holds |
|
| Skill | What it holds |
|
||||||
|---|---|
|
|---|---|
|
||||||
| [`plan-review-criteria`](skills/plan-review-criteria/SKILL.md) | The plan review rubric: six axes, severity levels, approval standard, output format. The `review-plan` reviewer loads it. |
|
| [`plan-review-criteria`](skills/plan-review-criteria/SKILL.md) | The plan review rubric: six axes, severity levels, approval standard, output format. The `review-plan` reviewer loads it. |
|
||||||
| [`code-review-criteria`](skills/code-review-criteria/SKILL.md) | The code review rubric: five axes, core principles (DRY, KISS, YAGNI), severity format, verdict. The `review-code` reviewer loads it. |
|
| [`code-review-criteria`](skills/code-review-criteria/SKILL.md) | The code review rubric: five axes, core principles (DRY, KISS, YAGNI), severity format, verdict. The `review-code` reviewer loads it. |
|
||||||
|
|
||||||
### Procedures
|
## Procedures
|
||||||
|
|
||||||
| Skill | What it does |
|
| Skill | What it does |
|
||||||
|---|---|
|
|---|---|
|
||||||
@ -496,7 +54,7 @@ JSON, REPL access, and so on.
|
|||||||
| [`create-issue`](skills/create-issue/SKILL.md) | Creates a GitHub issue that follows Penpot conventions. Used by `implement-plan`; also works on its own. |
|
| [`create-issue`](skills/create-issue/SKILL.md) | Creates a GitHub issue that follows Penpot conventions. Used by `implement-plan`; also works on its own. |
|
||||||
| [`create-commit`](skills/create-commit/SKILL.md) | Makes a commit the Penpot way: emoji subject, clear body, `AI-assisted-by` trailer. Used by `implement-plan`; also works alone when you say "commit this". |
|
| [`create-commit`](skills/create-commit/SKILL.md) | Makes a commit the Penpot way: emoji subject, clear body, `AI-assisted-by` trailer. Used by `implement-plan`; also works alone when you say "commit this". |
|
||||||
|
|
||||||
### Utilities
|
## Utilities
|
||||||
|
|
||||||
| Skill | What it does |
|
| Skill | What it does |
|
||||||
|---|---|
|
|---|---|
|
||||||
@ -512,9 +70,8 @@ JSON, REPL access, and so on.
|
|||||||
| [`ste`](skills/ste/SKILL.md) | Rewrites prose in Simplified Technical English. Loads only when you name it. |
|
| [`ste`](skills/ste/SKILL.md) | Rewrites prose in Simplified Technical English. Loads only when you name it. |
|
||||||
| [`refine-prompt`](skills/refine-prompt/SKILL.md) | Rewrites a rough prompt into a clearer one. Never runs the prompt. |
|
| [`refine-prompt`](skills/refine-prompt/SKILL.md) | Rewrites a rough prompt into a clearer one. Never runs the prompt. |
|
||||||
| [`update-changelog`](skills/update-changelog/SKILL.md) | Regenerates `CHANGES.md` from a GitHub milestone. |
|
| [`update-changelog`](skills/update-changelog/SKILL.md) | Regenerates `CHANGES.md` from a GitHub milestone. |
|
||||||
| [`find-project-anomalies`](skills/find-project-anomalies/SKILL.md) | Checks a milestone against the Main project board and reports anomalies to `tmp/<MILESTONE>-ANOMALIES.md`. |
|
|
||||||
|
|
||||||
### A typical round
|
## A typical round
|
||||||
|
|
||||||
1. `/make-a-plan` — you get a plan and a saved file in `.agents/plans/`.
|
1. `/make-a-plan` — you get a plan and a saved file in `.agents/plans/`.
|
||||||
2. `/review-plan` — a second opinion; approve or request changes.
|
2. `/review-plan` — a second opinion; approve or request changes.
|
||||||
@ -525,7 +82,7 @@ JSON, REPL access, and so on.
|
|||||||
Every step also works on its own, and you can always say what you want
|
Every step also works on its own, and you can always say what you want
|
||||||
in plain words — the agents pick the right skill from what you say.
|
in plain words — the agents pick the right skill from what you say.
|
||||||
|
|
||||||
### Adding or changing a skill
|
## Adding or changing a skill
|
||||||
|
|
||||||
Create a folder here with a `SKILL.md` inside. The file needs `name` and
|
Create a folder here with a `SKILL.md` inside. The file needs `name` and
|
||||||
`description` in its frontmatter, and a clear "When to use" section so
|
`description` in its frontmatter, and a clear "When to use" section so
|
||||||
|
|||||||
@ -1,15 +0,0 @@
|
|||||||
Act as a senior full-stack software engineer for this project.
|
|
||||||
|
|
||||||
## Instructions
|
|
||||||
|
|
||||||
1. Read `AGENTS.md` first and follow its memory-reading rules: read `mem:critical-info`, then the core memory of every module your work touches, plus any deeper memories they reference.
|
|
||||||
2. Work autonomously: explore the codebase first, follow existing patterns and conventions, apply DRY/KISS.
|
|
||||||
3. Verify before finishing: run tests, lint and fm, fix anything you broke. Never report done with failing checks.
|
|
||||||
4. Before finishing, review the affected memories and documentation against the implementation. If the change introduces behavior, contracts, decisions, or constraints that are not documented, or makes existing documentation inaccurate, update the relevant memories and docs in the same change.
|
|
||||||
|
|
||||||
## Strong Rules
|
|
||||||
|
|
||||||
1. All new functionality ships with tests. No exceptions.
|
|
||||||
2. Do not touch unrelated modules.
|
|
||||||
3. Never `git push`, force-push, or modify remotes. Only create commits when the
|
|
||||||
command or the user explicitly instructs it.
|
|
||||||
@ -13,17 +13,19 @@ Multi-dimensional code review with quality gates. Every change gets reviewed bef
|
|||||||
|
|
||||||
## When to Use
|
## When to Use
|
||||||
|
|
||||||
- The reviewer subagent of the `review-code` flow loads this skill to perform the review of a code change.
|
- The reviewer subagent of the `review-code` flow loads this skill to perform
|
||||||
- To review code, always go through the `review-code` flow — never load this skill directly for that. This is the criteria reference, not the flow.
|
the review of a code change.
|
||||||
|
- To review code, always go through the `review-code` flow — never load this
|
||||||
|
skill directly for that. This is the criteria reference, not the flow.
|
||||||
|
|
||||||
## Core Principles
|
## Core Principles
|
||||||
|
|
||||||
These principles underpin every axis. When in doubt, default to them.
|
These principles underpin every axis. When in doubt, default to them.
|
||||||
|
|
||||||
- **DRY (Don't Repeat Yourself):** Every piece of knowledge has one authoritative representation. Flag duplicated logic as a required change — it is not "just similar", it is drift that will diverge.
|
- **DRY (Don't Repeat Yourself):** Every piece of knowledge has one authoritative representation. If the same logic appears in two places, extract it into a shared helper, model, or type. Reviewers: flag duplicated logic as a required change — it's not "just similar," it's drift that will diverge.
|
||||||
- **KISS (Keep It Simple, Stupid):** The simplest solution that works is the best solution. Complexity must earn its place: if it needs more than a sentence to explain, push for simplification before merge.
|
- **KISS (Keep It Simple, Stupid):** The simplest solution that works is the best solution. Complexity must earn its place. Reviewers: if you need more than one sentence to explain what a piece of code does, it's too complex — push for simplification before merge.
|
||||||
- **YAGNI (You Aren't Gonna Need It):** No abstractions, hooks or generalizations for hypothetical future use cases. Generalize on the third occurrence, not the first.
|
- **YAGNI (You Aren't Gonna Need It):** Don't add abstractions, hooks, or generalizations for hypothetical future use cases. Generalize on the third occurrence, not the first. Reviewers: delete speculative generality.
|
||||||
- **Don't invent problems:** Every finding must be a real risk, a real readability barrier, or a real architectural concern — not a hypothetical or a stylistic preference disguised as a problem.
|
- **Don't invent problems:** Do not manufacture issues to produce more feedback. Every finding must be a real risk, a real readability barrier, or a real architectural concern — not a hypothetical or a stylistic preference disguised as a problem.
|
||||||
|
|
||||||
## The Five-Axis Review
|
## The Five-Axis Review
|
||||||
|
|
||||||
@ -36,7 +38,7 @@ Does the code do what it claims to do?
|
|||||||
- Does it match the spec or task requirements?
|
- Does it match the spec or task requirements?
|
||||||
- Are edge cases handled (null, empty, boundary values)?
|
- Are edge cases handled (null, empty, boundary values)?
|
||||||
- Are error paths handled (not just the happy path)?
|
- Are error paths handled (not just the happy path)?
|
||||||
- Do the tests cover the changed behavior, including its edge cases?
|
- Does it pass all tests? Are the tests actually testing the right things?
|
||||||
- Are there off-by-one errors, race conditions, or state inconsistencies?
|
- Are there off-by-one errors, race conditions, or state inconsistencies?
|
||||||
|
|
||||||
### 2. Readability & Simplicity
|
### 2. Readability & Simplicity
|
||||||
@ -45,7 +47,10 @@ Can another engineer (or agent) understand this code without the author explaini
|
|||||||
|
|
||||||
- Are names descriptive and consistent with project conventions? (No `temp`, `data`, `result` without context)
|
- Are names descriptive and consistent with project conventions? (No `temp`, `data`, `result` without context)
|
||||||
- Is the control flow straightforward (avoid nested ternaries, deep callbacks)?
|
- Is the control flow straightforward (avoid nested ternaries, deep callbacks)?
|
||||||
- Are there "clever" tricks that should be simplified? Could this be done in fewer lines? (1000 lines where 100 suffice is a failure)
|
- Are there any "clever" tricks that should be simplified?
|
||||||
|
- **KISS check:** Is this the simplest approach that solves the problem? A 20-line straightforward function beats a 5-line clever one that requires a comment to explain.
|
||||||
|
- Could this be done in fewer lines? (1000 lines where 100 suffice is a failure)
|
||||||
|
- Are abstractions earning their complexity? (Don't generalize until the third use case)
|
||||||
- Is a new conditional bolted onto an unrelated flow? Push the logic into its own helper, state, or policy.
|
- Is a new conditional bolted onto an unrelated flow? Push the logic into its own helper, state, or policy.
|
||||||
- Do repeated conditionals on the same shape appear? They signal a missing model or dispatcher.
|
- Do repeated conditionals on the same shape appear? They signal a missing model or dispatcher.
|
||||||
- Are there dead code artifacts: no-op variables, backwards-compat shims, or `// removed` comments?
|
- Are there dead code artifacts: no-op variables, backwards-compat shims, or `// removed` comments?
|
||||||
@ -55,13 +60,14 @@ Can another engineer (or agent) understand this code without the author explaini
|
|||||||
Does the change fit the system's design?
|
Does the change fit the system's design?
|
||||||
|
|
||||||
- Does it follow existing patterns or introduce a new one? If new, is it justified?
|
- Does it follow existing patterns or introduce a new one? If new, is it justified?
|
||||||
- Does it keep clean module boundaries, with dependencies flowing the right way (no cycles)?
|
- Does it maintain clean module boundaries?
|
||||||
|
- **DRY check:** Is there existing code that does the same thing? Reuse the canonical helper instead of writing a near-duplicate. If two branches do nearly the same thing, collapse them.
|
||||||
|
- Are dependencies flowing in the right direction (no circular dependencies)?
|
||||||
- Is the abstraction level appropriate (not over-engineered, not too coupled)?
|
- Is the abstraction level appropriate (not over-engineered, not too coupled)?
|
||||||
- Is there existing code that already does this? Reuse the canonical helper instead of a near-duplicate, and collapse branches that do nearly the same thing.
|
- Does this refactor reduce complexity or just relocate it? Count the concepts a reader must hold. Prefer the restructuring that makes whole branches disappear over one that re-centralizes the same logic. Prefer deleting an abstraction to polishing it.
|
||||||
- Is feature-specific logic leaking into a shared or general-purpose module?
|
- Is feature-specific logic leaking into a shared or general-purpose module?
|
||||||
- Are type boundaries explicit? Question gratuitous `any`/`unknown`/optional/casts and silent fallbacks.
|
- Are type boundaries explicit? Question gratuitous `any`/`unknown`/optional/casts and silent fallbacks.
|
||||||
- Does the change reduce complexity or relocate it? Count the concepts a reader must hold: prefer the restructuring that makes whole branches disappear, and deleting an abstraction over polishing it.
|
- **Structural remedies:** When you flag a problem, propose the move — not just the problem. Replace conditionals with dispatchers, collapse duplicate branches, separate orchestration from business logic, extract helpers, split large files. Prefer the remedy that removes moving pieces over one that spreads the same complexity around.
|
||||||
- **Give the remedy, not only the problem:** replace conditionals with dispatchers, separate orchestration from business logic, extract helpers, split large files. Prefer the remedy that removes moving pieces over one that spreads the same complexity around.
|
|
||||||
|
|
||||||
### 4. Security
|
### 4. Security
|
||||||
|
|
||||||
@ -99,23 +105,13 @@ For detailed security guidance, see `security-and-hardening`.
|
|||||||
| **Low:** | Minor, optional | Author may ignore — formatting, style preferences |
|
| **Low:** | Minor, optional | Author may ignore — formatting, style preferences |
|
||||||
| **Suggestion:** | Worth considering | Not required, but improves the code |
|
| **Suggestion:** | Worth considering | Not required, but improves the code |
|
||||||
|
|
||||||
**Finding IDs.** Assign every finding a stable identifier by severity class: `R1`, `R2`, … for required findings (Critical, High), `M1`, `M2`, … for minor findings (Medium, Low), and `S1`, `S2`, … for suggestions. Number within each class in review order. Use the ID everywhere the finding is mentioned — in section headers, in the verdict, in follow-up discussion. Never renumber within a review. Example: `**R2** — `app/validate.cljs:42` — duplicate branch logic…`.
|
**Unique finding IDs.** Assign every finding a stable identifier: `F1`, `F2`, `F3`, … numbered in order of severity (Critical first, then High, Medium, Low, Suggestion). Use the ID everywhere the finding is mentioned — in section headers, in the verdict, in follow-up discussion. Never renumber within a review. Example: `**F3 (High)** — `app/validate.cljs:42` — duplicate branch logic…`.
|
||||||
|
|
||||||
For each finding, describe the circumstances under which it could fail: specific inputs, load conditions, timing, or user actions that trigger the problem. "This crashes when input is null" is actionable; "this might crash" is not.
|
For each finding, describe the circumstances under which it could fail: specific inputs, load conditions, timing, or user actions that trigger the problem. "This crashes when input is null" is actionable; "this might crash" is not.
|
||||||
|
|
||||||
Lead with what matters: correctness and security first, then structural issues, then everything else. A few high-conviction comments beat a long list.
|
Lead with what matters: correctness and security first, then structural issues, then everything else. A few high-conviction comments beat a long list.
|
||||||
|
|
||||||
5. **Check the evidence, do not re-run it** — Which tests cover the change? What did the author run, and what did CI report? Never re-run the full suite or a build yourself: the author's run and CI already cover that, and repeating it only lengthens the review. Run at most the single test file that settles a specific doubt. When no checks have been run at all (a change you did not see tested), run the module's checks before the verdict.
|
5. **Verify the verification** — What tests were run? Did the build pass? Was the change tested manually? Screenshots for UI changes?
|
||||||
|
|
||||||
## Change Sizing
|
|
||||||
|
|
||||||
Small, focused changes are easier to review, faster to merge, and safer to deploy. Flag the size when a diff exceeds ~300 lines, or when it grows a file past ~1000 total lines, and name the remedy: split it, and separate refactoring from feature work. This is a finding about the change, not work to do inside the review.
|
|
||||||
|
|
||||||
## Dependencies
|
|
||||||
|
|
||||||
Before accepting a new dependency: does the existing stack already solve it, how large is the bundle impact, is it maintained, does `npm audit` report anything, is the license compatible. Prefer the standard library and existing utilities — every dependency is a liability.
|
|
||||||
|
|
||||||
For upgrades: read the changelog, not just the version number; one dependency per change; a green suite before and after; review the lockfile diff and never hand-edit it. Supply-chain triage follows `security-and-hardening`.
|
|
||||||
|
|
||||||
## Review Output
|
## Review Output
|
||||||
|
|
||||||
@ -127,11 +123,11 @@ Briefly explain what the code does and give an overall assessment.
|
|||||||
|
|
||||||
### Critical and High-Priority Issues
|
### Critical and High-Priority Issues
|
||||||
|
|
||||||
List problems that could cause security incidents, data loss, crashes, incorrect behavior, or major performance degradation. Each finding gets its unique ID (`R1`, `R2`, …). For each: state the severity, identify the file/function/code section, explain why it's a problem, describe failure circumstances, and provide a concrete improvement with corrected code when useful.
|
List problems that could cause security incidents, data loss, crashes, incorrect behavior, or major performance degradation. Each finding gets its unique ID (`F1`, `F2`, …). For each: state the severity, identify the file/function/code section, explain why it's a problem, describe failure circumstances, and provide a concrete improvement with corrected code when useful.
|
||||||
|
|
||||||
### Other Findings
|
### Other Findings
|
||||||
|
|
||||||
List medium- and low-priority issues, including maintainability and design concerns. Use `M1`, `M2`, … for Medium/Low and `S1`, `S2`, … for Suggestions.
|
List medium- and low-priority issues, including maintainability and design concerns. Continue the ID sequence started above (`F3`, `F4`, …).
|
||||||
|
|
||||||
### Suggested Refactoring
|
### Suggested Refactoring
|
||||||
|
|
||||||
@ -153,20 +149,98 @@ Choose one:
|
|||||||
- **Approve with minor changes** — Good to merge after addressing low/medium issues
|
- **Approve with minor changes** — Good to merge after addressing low/medium issues
|
||||||
- **Request changes** — Critical or high issues must be resolved before merge
|
- **Request changes** — Critical or high issues must be resolved before merge
|
||||||
|
|
||||||
List the finding IDs the verdict depends on (e.g. "Request changes: R1, R2").
|
List the finding IDs the verdict depends on (e.g. "Request changes: F1, F4").
|
||||||
|
|
||||||
|
## Change Sizing
|
||||||
|
|
||||||
|
Small, focused changes are easier to review, faster to merge, and safer to deploy.
|
||||||
|
|
||||||
|
```
|
||||||
|
~100 lines changed → Good. Reviewable in one sitting.
|
||||||
|
~300 lines changed → Acceptable if it's a single logical change.
|
||||||
|
~1000 lines changed → Too large. Split it.
|
||||||
|
```
|
||||||
|
|
||||||
|
**Watch file size, not just diff size.** Around 1000 *total* lines in a single file is a common inspection signal. When a change materially grows an already-large file, decompose first.
|
||||||
|
|
||||||
|
**Splitting strategies:**
|
||||||
|
|
||||||
|
| Strategy | How | When |
|
||||||
|
|----------|-----|------|
|
||||||
|
| **Stack** | Submit a small change, start the next one based on it | Sequential dependencies |
|
||||||
|
| **By file group** | Separate changes for groups needing different reviewers | Cross-cutting concerns |
|
||||||
|
| **Horizontal** | Create shared code/stubs first, then consumers | Layered architecture |
|
||||||
|
| **Vertical** | Break into smaller full-stack slices of the feature | Feature work |
|
||||||
|
|
||||||
|
**Separate refactoring from feature work.** A change that refactors and adds new behavior is two changes — submit them separately.
|
||||||
|
|
||||||
|
## Change Descriptions
|
||||||
|
|
||||||
|
- **First line:** Short, imperative, standalone. "Delete the FizzBuzz RPC" not "Deleting the FizzBuzz RPC."
|
||||||
|
- **Body:** What is changing and why. Include context and reasoning not visible in the code itself.
|
||||||
|
- **Anti-patterns:** "Fix bug," "Fix build," "Add patch," "Phase 1."
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
|
||||||
|
Before adding any dependency:
|
||||||
|
|
||||||
|
1. Does the existing stack solve this? (Often it does.)
|
||||||
|
2. How large is the dependency? (Check bundle impact.)
|
||||||
|
3. Is it actively maintained? (Check last commit, open issues.)
|
||||||
|
4. Does it have known vulnerabilities? (`npm audit`)
|
||||||
|
5. What's the license? (Must be compatible with the project.)
|
||||||
|
|
||||||
|
**Rule:** Prefer standard library and existing utilities over new dependencies. Every dependency is a liability.
|
||||||
|
|
||||||
|
**Upgrading dependencies:**
|
||||||
|
|
||||||
|
- Read the changelog, not just the version number. Semver is a promise the maintainer may not have kept.
|
||||||
|
- One dependency per change. When a bulk bump breaks the build, you've lost which package did it.
|
||||||
|
- Let the tests decide — a green suite before *and* after, not just "it installed."
|
||||||
|
- Review the lockfile diff, not just `package.json`. Commit it and never hand-edit it.
|
||||||
|
|
||||||
|
For supply-chain risk triage, follow the `security-and-hardening` skill.
|
||||||
|
|
||||||
|
## Common Rationalizations
|
||||||
|
|
||||||
|
| Rationalization | Reality |
|
||||||
|
|---|---|
|
||||||
|
| "It works, that's good enough" | Working code that's unreadable, insecure, or architecturally wrong creates debt that compounds. |
|
||||||
|
| "I wrote it, so I know it's correct" | Authors are blind to their own assumptions. Every change benefits from another set of eyes. |
|
||||||
|
| "We'll clean it up later" | Later never comes. The review is the quality gate — use it. |
|
||||||
|
| "AI-generated code is probably fine" | AI code needs more scrutiny, not less. It's confident and plausible, even when wrong. |
|
||||||
|
| "The tests pass, so it's good" | Tests are necessary but not sufficient. They don't catch architecture, security, or readability problems. |
|
||||||
|
| "The refactor makes it cleaner" | Relocating complexity isn't reducing it. If the reader still holds the same number of concepts, the structure didn't improve. |
|
||||||
|
| "It's only a small addition to this file" | Small diffs still push files past healthy size and bolt branches onto unrelated flows. |
|
||||||
|
| "It's just a version bump" | A bump is a behavior change you didn't write. Read the changelog. |
|
||||||
|
| "I'll upgrade everything in one PR" | A bulk bump hides which package broke the build. One per change. |
|
||||||
|
| "It's duplicated but it's only two places" | Two becomes three becomes five. Extract now, before the copies diverge. |
|
||||||
|
| "The abstraction is future-proof" | YAGNI. Delete speculative generality — generalize on the third occurrence, not the first. |
|
||||||
|
| "It's clever but efficient" | Cleverness is a readability tax. If it needs a comment to understand, simplify it. |
|
||||||
|
|
||||||
## Red Flags
|
## Red Flags
|
||||||
|
|
||||||
- A change merged without any review.
|
- PRs merged without any review
|
||||||
- A review that only checks whether the tests pass.
|
- Review that only checks if tests pass (ignoring other axes)
|
||||||
- Security-sensitive changes without a security-focused review.
|
- "LGTM" without evidence of actual review
|
||||||
- A change too large to review properly: split it.
|
- Security-sensitive changes without security-focused review
|
||||||
|
- Large PRs that are "too big to review properly" (split them)
|
||||||
|
- No regression tests with bug fix PRs
|
||||||
|
- Accepting "I'll fix it later" — it never happens
|
||||||
|
- A refactor that moves code around without reducing the number of concepts a reader must hold
|
||||||
|
- New conditionals scattered into unrelated code paths (a missing abstraction)
|
||||||
|
- A bespoke helper that duplicates an existing canonical one
|
||||||
|
- A bulk "bump dependencies" PR with no changelog review
|
||||||
|
|
||||||
## Verification
|
## Verification
|
||||||
|
|
||||||
Before emitting the verdict, confirm the evidence the review rests on. This covers the state of the code at review time, not the later fixing of findings:
|
Before emitting the verdict, verify the change as it stands. This is the reviewer's own due diligence — it covers the state of the code at review time, not the later resolution of findings (fixing findings is the author's job; confirming them is a new review):
|
||||||
|
|
||||||
- [ ] The suite is green, and the evidence is the author's run or CI — not your own re-run. Say which one you relied on.
|
- [ ] Tests pass — run them yourself, don't trust the claim
|
||||||
- [ ] New behavior has tests that cover the edge cases in the diff.
|
- [ ] Build succeeds
|
||||||
- [ ] The verification story is stated: what changed and how it was verified.
|
- [ ] The verification story is documented (what changed, how it was verified)
|
||||||
- [ ] Dependency changes were reviewed against the changelog and the lockfile diff.
|
- [ ] Dependency upgrades reviewed against changelog, isolated per package, verified by green suite
|
||||||
|
|
||||||
|
## See Also
|
||||||
|
|
||||||
|
- For detailed security review guidance, see `security-and-hardening`
|
||||||
|
|||||||
@ -1,7 +1,6 @@
|
|||||||
---
|
---
|
||||||
name: create-commit
|
name: create-commit
|
||||||
description: Stage, review, and commit files following Penpot commit conventions.
|
description: Stage, review, and commit files following Penpot commit conventions.
|
||||||
slash: true
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# Skill: create-commit
|
# Skill: create-commit
|
||||||
@ -21,18 +20,6 @@ Before drafting any commit, read `mem:workflow/creating-commits` end-to-end. It
|
|||||||
is the authoritative source for the commit message format, the emoji menu,
|
is the authoritative source for the commit message format, the emoji menu,
|
||||||
subject/body limits, and the `AI-assisted-by` trailer. Follow it exactly.
|
subject/body limits, and the `AI-assisted-by` trailer. Follow it exactly.
|
||||||
|
|
||||||
## Iron Rules (non-negotiable)
|
|
||||||
|
|
||||||
1. **Wrap every body line at 80 characters or fewer.** Count characters, do
|
|
||||||
not eyeball. Exceptions: `Signed-off-by:` / `AI-assisted-by:` trailers and
|
|
||||||
lines carrying a URL. This is the rule agents skip most often.
|
|
||||||
2. **Subject ≤80 chars**, imperative, capitalized, no trailing period.
|
|
||||||
`Merge`/`Revert`/`Reapply` commits skip the length check.
|
|
||||||
3. **Blank line between subject and body.**
|
|
||||||
4. **Run `./scripts/check-commit` and require exit code 0.** It mechanically
|
|
||||||
checks rules 1–3. A non-zero exit is a hard blocker: fix the message and
|
|
||||||
re-commit. Never report the commit as done with a failing checker.
|
|
||||||
|
|
||||||
## Workflow
|
## Workflow
|
||||||
|
|
||||||
1. **Stage the files** specified by the calling context. Do not ask for
|
1. **Stage the files** specified by the calling context. Do not ask for
|
||||||
@ -42,18 +29,12 @@ subject/body limits, and the `AI-assisted-by` trailer. Follow it exactly.
|
|||||||
that does not match the stated intent, **STOP** and tell the user before
|
that does not match the stated intent, **STOP** and tell the user before
|
||||||
committing.
|
committing.
|
||||||
3. Draft the message following the format in the memory doc, wrapping the body
|
3. Draft the message following the format in the memory doc, wrapping the body
|
||||||
at 80 characters per line, and run:
|
at 72 characters per line, and run:
|
||||||
```bash
|
```bash
|
||||||
git commit -m "<subject>" -m "<body>"
|
git commit -m "<subject>" -m "<body>"
|
||||||
```
|
```
|
||||||
(or `git commit -F -` if the body has unusual characters).
|
(or `git commit -F -` if the body has unusual characters).
|
||||||
4. **Verify the message with the checker**:
|
4. The `AI-assisted-by` trailer value is provided by the calling context — use
|
||||||
```bash
|
|
||||||
./scripts/check-commit
|
|
||||||
```
|
|
||||||
If it fails, amend the message (`git commit --amend`) until it passes. Do
|
|
||||||
not finish with a failing checker.
|
|
||||||
5. The `AI-assisted-by` trailer value is provided by the calling context — use
|
|
||||||
it verbatim.
|
it verbatim.
|
||||||
|
|
||||||
## Constraints
|
## Constraints
|
||||||
@ -63,8 +44,4 @@ subject/body limits, and the `AI-assisted-by` trailer. Follow it exactly.
|
|||||||
- Do not pass `--author`. Author identity comes from the local git config.
|
- Do not pass `--author`. Author identity comes from the local git config.
|
||||||
- Do not amend a commit you did not create in this session, unless explicitly asked.
|
- Do not amend a commit you did not create in this session, unless explicitly asked.
|
||||||
- Do not bypass pre-commit hooks (`--no-verify`) unless explicitly asked.
|
- Do not bypass pre-commit hooks (`--no-verify`) unless explicitly asked.
|
||||||
- Add `Co-authored-by:` only for a human co-author. AI providers, agents and
|
|
||||||
bots are rejected by `./scripts/check-commit`; for AI participation use the
|
|
||||||
`AI-assisted-by:` trailer.
|
|
||||||
- Do not add untracked files that were not created in this session.
|
- Do not add untracked files that were not created in this session.
|
||||||
- Do not skip the `scripts/check-commit` verification step (Iron Rule 4).
|
|
||||||
|
|||||||
@ -1,7 +1,6 @@
|
|||||||
---
|
---
|
||||||
name: create-issue
|
name: create-issue
|
||||||
description: Create or update GitHub issues (from PR, from draft body, retitle existing). Routes to the canonical flow in `mem:workflow/creating-issues`.
|
description: Create or update GitHub issues (from PR, from draft body, retitle existing). Routes to the canonical flow in `mem:workflow/creating-issues`.
|
||||||
slash: true
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# Skill: create-issue
|
# Skill: create-issue
|
||||||
@ -20,9 +19,6 @@ right flow.
|
|||||||
- **Create from draft body** — Taiga story, user report, discussion; no PR
|
- **Create from draft body** — Taiga story, user report, discussion; no PR
|
||||||
yet.
|
yet.
|
||||||
→ memory section **Creating Issues from Draft Body**
|
→ memory section **Creating Issues from Draft Body**
|
||||||
- **Create as sub-issue** — the issue must be grouped under an umbrella/EPIC
|
|
||||||
issue; create it first, then link it to its parent.
|
|
||||||
→ memory section **Adding an Issue as a Sub-issue**
|
|
||||||
- **Retitle existing issue** — current title is vague, prefixed, or stale.
|
- **Retitle existing issue** — current title is vague, prefixed, or stale.
|
||||||
→ memory section **Retitling an Existing Issue**
|
→ memory section **Retitling an Existing Issue**
|
||||||
|
|
||||||
|
|||||||
@ -73,31 +73,10 @@ one. Title or description fixes belong to Update mode.
|
|||||||
Write the title and body following `mem:workflow/creating-prs` (title format,
|
Write the title and body following `mem:workflow/creating-prs` (title format,
|
||||||
description structure, writing principles) and `mem:workflow/creating-commits`
|
description structure, writing principles) and `mem:workflow/creating-commits`
|
||||||
(commit type emojis). Derive the title and body from the commits and, when
|
(commit type emojis). Derive the title and body from the commits and, when
|
||||||
there is one, from the issue body. Include `Closes #NNNN` for readable context,
|
there is one, from the issue body. Reference the issue with `Closes #NNNN`.
|
||||||
but do not treat it as the link; use `python3 scripts/gh.py link-issue <ISSUE_NUMBER> <PR_NUMBER>` after creating the PR.
|
|
||||||
|
|
||||||
Repeat the `AI-assisted-by:` trailer in the body, once per model that worked
|
|
||||||
on the branch, so the PR states the assistance where a reviewer reads it. The
|
|
||||||
branch commits keep their own trailers, and a squash merge carries every one
|
|
||||||
of them into the landed message.
|
|
||||||
|
|
||||||
Before offering or accepting a draft PR, warn that CI doesn't run on them. Add
|
|
||||||
`--draft` only when the user agrees to that.
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
gh pr create --repo penpot/penpot --base "<BASE>" --title "<TITLE>" \
|
gh pr create --repo penpot/penpot --title "<TITLE>" --body-file /tmp/pr-body.md
|
||||||
--project "Main" --body-file /tmp/pr-body.md
|
|
||||||
```
|
|
||||||
|
|
||||||
`--base` is the branch resolved in step 1: without it the PR opens against the
|
|
||||||
repository default, which is wrong for a branch cut from `staging`. `--project
|
|
||||||
"Main"` is required by `mem:workflow/creating-prs`.
|
|
||||||
|
|
||||||
If an issue is present, run the explicit assignment command from
|
|
||||||
`mem:workflow/creating-prs` before reporting success:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python3 scripts/gh.py link-issue <ISSUE_NUMBER> <PR_NUMBER>
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### 5. Report
|
### 5. Report
|
||||||
@ -116,13 +95,6 @@ gh pr edit <NUMBER> --repo penpot/penpot --title "<TITLE>" --body-file /tmp/pr-b
|
|||||||
gh pr view <NUMBER> --repo penpot/penpot --json title,body
|
gh pr view <NUMBER> --repo penpot/penpot --json title,body
|
||||||
```
|
```
|
||||||
|
|
||||||
If the updated body contains `Closes #NNNN`, run the explicit assignment
|
|
||||||
command from `mem:workflow/creating-prs`:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python3 scripts/gh.py link-issue <ISSUE_NUMBER> <PR_NUMBER>
|
|
||||||
```
|
|
||||||
|
|
||||||
4. Report and stop.
|
4. Report and stop.
|
||||||
|
|
||||||
## User context
|
## User context
|
||||||
|
|||||||
@ -1,116 +0,0 @@
|
|||||||
---
|
|
||||||
name: find-project-anomalies
|
|
||||||
description: Check a GitHub milestone against the Main project board, report the five anomaly types to tmp/<MILESTONE>-ANOMALIES.md, and fix missing milestone assignments on request.
|
|
||||||
---
|
|
||||||
|
|
||||||
# Skill: find-project-anomalies
|
|
||||||
|
|
||||||
Check every issue and PR in a given GitHub milestone against the Main
|
|
||||||
project board. Report the five anomaly types below, then fix the ones a
|
|
||||||
human confirms.
|
|
||||||
|
|
||||||
## Getting the version ($ARGUMENTS)
|
|
||||||
|
|
||||||
The milestone version comes from the skill arguments (`$ARGUMENTS`), the
|
|
||||||
same way other skills receive theirs. Accept `2.17.0`, `v2.17.0`,
|
|
||||||
`milestone 2.17.0`, or any text clearly containing one `X.Y.Z` version.
|
|
||||||
|
|
||||||
If no version is given and none can be deduced from the arguments or the
|
|
||||||
conversation context, **stop and ask the user** which milestone to check.
|
|
||||||
Do not guess, do not default to the latest release.
|
|
||||||
|
|
||||||
## When to Use
|
|
||||||
|
|
||||||
- Before a release, to catch board/milestone drift while there is still time
|
|
||||||
to fix it
|
|
||||||
- After triage sessions, to verify every milestone item is tracked, owned,
|
|
||||||
and correctly assigned
|
|
||||||
- Whenever someone asks "is milestone X.Y.Z clean on Main?"
|
|
||||||
|
|
||||||
## Prerequisites
|
|
||||||
|
|
||||||
- `gh` CLI authenticated (`gh auth status`)
|
|
||||||
- Python 3.8+
|
|
||||||
- `scripts/gh.py` for GitHub queries, `scripts/project-anomalies.py` for
|
|
||||||
detection and reporting (`python3 scripts/project-anomalies.py check --help`)
|
|
||||||
|
|
||||||
## Workflow
|
|
||||||
|
|
||||||
### 1. Resolve the version (see above), then run the check
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python3 scripts/project-anomalies.py check "2.17.0"
|
|
||||||
# explicit path:
|
|
||||||
python3 scripts/project-anomalies.py check "2.17.0" --output tmp/2.17.0-ANOMALIES.md
|
|
||||||
```
|
|
||||||
|
|
||||||
This fetches milestone issues and PRs in bulk (one GraphQL call per 50
|
|
||||||
items), resolves outsiders the same way, and overwrites
|
|
||||||
`tmp/<MILESTONE>-ANOMALIES.md`. Every number renders as a full
|
|
||||||
`[#N](https://github.com/penpot/penpot/issues/N)` or
|
|
||||||
`[#N](https://github.com/penpot/penpot/pull/N)` link.
|
|
||||||
|
|
||||||
### 2. Read the report and judge each anomaly
|
|
||||||
|
|
||||||
The five types, in report order:
|
|
||||||
|
|
||||||
1. **OPEN issue with a MERGED PR** — the fix landed but the issue never
|
|
||||||
closed (or it reopened). Fix: close the issue, or move it out of the
|
|
||||||
milestone if the fix did not actually land here.
|
|
||||||
2. **Milestone issue, PR elsewhere or issue off Main** — a *merged* closing
|
|
||||||
PR in another milestone (or none), or the issue itself not on the Main
|
|
||||||
board. An unmerged PR's milestone means nothing (it landed nowhere), so
|
|
||||||
it never triggers this type. Fix: align the milestones (see step 3), or
|
|
||||||
add the issue to Main.
|
|
||||||
3. **Milestone PR, issue elsewhere or off Main** — the mirror view over
|
|
||||||
*merged* PRs only: a PR released here closes an issue tracked elsewhere
|
|
||||||
(or nowhere on Main). Unmerged PRs never trigger this type — their
|
|
||||||
milestone means nothing yet. Fix: align the milestones (see step 3), or
|
|
||||||
add the issue to Main.
|
|
||||||
4. **MERGED PR on a `needs triage` issue** — landed without triage. Only a
|
|
||||||
human can triage; never auto-remove the label.
|
|
||||||
5. **MERGED PR on an unassigned, non-community issue** — no owner and no
|
|
||||||
`community contribution` label (checked on both issue and PR). Either
|
|
||||||
assign an owner or confirm it is community work. Never invent an owner.
|
|
||||||
|
|
||||||
Scope notes (deliberate, not bugs):
|
|
||||||
- Project membership is checked **on the issue side only**. PRs are not the
|
|
||||||
unit tracked on Main, so an unprojected PR alone is never an anomaly.
|
|
||||||
- An issue with *no* milestone closed by a milestone PR **is** reported
|
|
||||||
here (type 3): unlike the changelog flow, this pairing always needs a
|
|
||||||
human look.
|
|
||||||
|
|
||||||
### 3. Fix missing milestone assignments (only these, only on confirmation)
|
|
||||||
|
|
||||||
The only mechanical fix in this skill is assigning a missing milestone.
|
|
||||||
The report already prints the exact command next to each such case:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
gh pr edit <PR> --milestone "<MILESTONE>"
|
|
||||||
gh issue edit <ISSUE> --milestone "<MILESTONE>"
|
|
||||||
```
|
|
||||||
|
|
||||||
Rules:
|
|
||||||
- Apply **only** when one side lacks a milestone. When both sides have
|
|
||||||
*different* milestones, a human decides which one moves — never pick a
|
|
||||||
side yourself.
|
|
||||||
- Confirm **each fix (or each small batch)** with the user before running
|
|
||||||
it. Read back what the command will change.
|
|
||||||
- Everything else (closing issues, triaging, assigning owners, adding
|
|
||||||
items to the Main board) is human work: point at it, do not do it.
|
|
||||||
|
|
||||||
### 4. Re-run until clean
|
|
||||||
|
|
||||||
After fixes land, re-run step 1 and confirm the report shows zero
|
|
||||||
anomalies. The `tmp/` report is scratch output (gitignored): quote or
|
|
||||||
paste entries into chat when reporting back, do not commit it.
|
|
||||||
|
|
||||||
## Key Principles
|
|
||||||
|
|
||||||
- **No version, no run.** Stop and ask when the milestone is unknown.
|
|
||||||
- **Report first, touch nothing.** No board or milestone changes without
|
|
||||||
explicit per-item confirmation.
|
|
||||||
- **Milestone gaps auto-fix; judgment calls do not.** Only a missing
|
|
||||||
milestone is mechanical. triage, ownership, and close/reopen decisions
|
|
||||||
belong to humans.
|
|
||||||
- **Every number clickable.** The report is useless without full GitHub links.
|
|
||||||
@ -1,7 +1,6 @@
|
|||||||
---
|
---
|
||||||
name: implement-plan
|
name: implement-plan
|
||||||
description: Implementation flow — execute a ready plan from the session context
|
description: Implementation flow — execute a ready plan from the session context: read the plan, detect the flow, then present the full picture (issue and branch to create or the branch to continue on, execution style, task checklist) and wait for confirmation. Default is every task with one final commit; on request ("step by step"), one task and one commit at a time with a pause after each; on request ("direct"), no issue and no branch — the commit lands on the current branch. Use it when the user asks to implement or execute a plan, in any phrasing.
|
||||||
slash: true
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# Implement Plan
|
# Implement Plan
|
||||||
@ -14,14 +13,6 @@ By default it ends with exactly one commit. When the user asks for it
|
|||||||
("step by step"), it commits once per task instead and waits for the
|
("step by step"), it commits once per task instead and waits for the
|
||||||
user's confirmation after each one (see *Execution modes*).
|
user's confirmation after each one (see *Execution modes*).
|
||||||
|
|
||||||
## Plan file handling
|
|
||||||
|
|
||||||
Plan files are local workflow artifacts. They must never be staged or
|
|
||||||
committed, even when `.gitignore` already excludes them. Never use
|
|
||||||
`git add -f`, change ignore rules, or otherwise force a plan file into a
|
|
||||||
commit. Close and update the plan when this flow requires it, but leave it
|
|
||||||
in the local working tree.
|
|
||||||
|
|
||||||
## When to use
|
## When to use
|
||||||
|
|
||||||
- The user asks to implement or execute a plan, in any phrasing:
|
- The user asks to implement or execute a plan, in any phrasing:
|
||||||
@ -111,18 +102,11 @@ Implement every task in one go. Work methodically, keeping changes
|
|||||||
focused on what the issue requires. Respect the plan's proposed
|
focused on what the issue requires. Respect the plan's proposed
|
||||||
parallelization when it applies.
|
parallelization when it applies.
|
||||||
|
|
||||||
When the implementation is complete, close the plan file first: flip its
|
When the implementation is complete, load the **`create-commit`** skill
|
||||||
`Status` to `done` and append one UTC ISO 8601 line to its `Review Log`
|
and follow its workflow to commit the changes. Provide a brief summary
|
||||||
with the issue URL when one exists (standalone mode, e.g.
|
of what was implemented and why, the issue reference (`issue-NNNN`) when
|
||||||
`https://github.com/penpot/penpot/issues/NNNN`); in continue/direct
|
there is one, and the model name you are running as so the
|
||||||
mode with no issue, just `done` with no invented identifier. Never
|
`AI-assisted-by` trailer is set correctly.
|
||||||
record commit hashes. Leave the closed plan file local and uncommitted.
|
|
||||||
Then load the **`create-commit`** skill and follow its workflow to commit
|
|
||||||
only the implementation, tests, memory, and documentation changes. Before
|
|
||||||
committing, verify that the staged file list does not contain the plan
|
|
||||||
file. Provide a brief summary of what was implemented and why, the issue
|
|
||||||
reference (`issue-NNNN`) when there is one, and the model name you are
|
|
||||||
running as so the `AI-assisted-by` trailer is set correctly.
|
|
||||||
|
|
||||||
### Step-by-step mode (on request)
|
### Step-by-step mode (on request)
|
||||||
|
|
||||||
@ -133,9 +117,6 @@ per task" — loop one task at a time:
|
|||||||
- Commit it now: load the **`create-commit`** skill and follow it —
|
- Commit it now: load the **`create-commit`** skill and follow it —
|
||||||
one commit per task, never two tasks in one commit. Same inputs as
|
one commit per task, never two tasks in one commit. Same inputs as
|
||||||
always: what and why, the issue reference, your model name.
|
always: what and why, the issue reference, your model name.
|
||||||
- After the final task commit, close the plan file (`Status: done`, one
|
|
||||||
`Review Log` line with the issue URL when one exists). Leave it local
|
|
||||||
and uncommitted; never amend the task commit to include it.
|
|
||||||
- Show the user the result (what changed, files touched, how it was
|
- Show the user the result (what changed, files touched, how it was
|
||||||
verified).
|
verified).
|
||||||
- WAIT for the user's confirmation before starting the next task.
|
- WAIT for the user's confirmation before starting the next task.
|
||||||
@ -153,6 +134,11 @@ instruction from me overrides them):
|
|||||||
`/make-a-plan` by itself if the findings need one.
|
`/make-a-plan` by itself if the findings need one.
|
||||||
- `/create-pr` — when the task is done and the branch is ready to merge.
|
- `/create-pr` — when the task is done and the branch is ready to merge.
|
||||||
|
|
||||||
## User input, overrides and additional context
|
## User context
|
||||||
|
|
||||||
$ARGUMENTS
|
Extra context in the user's invocation (the message that triggered this
|
||||||
|
skill) plays the role command arguments play elsewhere: `standalone`,
|
||||||
|
`continue`, `direct` (`no branch` / `direct commit`), `no issue` /
|
||||||
|
`without issue`, an explicit base such as `from origin/develop`, or
|
||||||
|
`step by step` / `one commit per task` for the step-by-step execution
|
||||||
|
mode. Modes combine freely, for example "standalone step by step".
|
||||||
|
|||||||
@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
name: local-ci
|
name: local-ci
|
||||||
description: Run local CI-style checks with ./scripts/ci (lint, tests, format, and on request production builds) per monorepo module. Use when verifying changes before declaring work done, running lint or tests locally, fixing formatting, repairing Clojure delimiter errors, or checking that a module still compiles.
|
description: Run local CI-style checks with ./scripts/ci (lint, tests, format) per monorepo module. Use when verifying changes before declaring work done, running lint or tests locally, fixing formatting, or repairing Clojure delimiter errors.
|
||||||
---
|
---
|
||||||
|
|
||||||
# Local CI
|
# Local CI
|
||||||
@ -29,12 +29,10 @@ Run from the repo root:
|
|||||||
```
|
```
|
||||||
|
|
||||||
Modules: `frontend` `backend` `common` `render-wasm` `exporter` `mcp`
|
Modules: `frontend` `backend` `common` `render-wasm` `exporter` `mcp`
|
||||||
`plugins` `library` `media-processor`, or `--all` for every module.
|
`plugins` `library`, or `--all` for every module.
|
||||||
|
|
||||||
With no task flags it runs three tasks per module, in order: **lint**,
|
With no task flags it runs three tasks per module, in order: **lint**,
|
||||||
**test**, **fmt** (format check; `--fix` formats files instead). **build**
|
**test**, **fmt** (format check; `--fix` formats files instead).
|
||||||
is opt-in: it compiles the module the way production does and is the slow
|
|
||||||
one, so it never runs unless you ask for it.
|
|
||||||
|
|
||||||
| Flag | Effect |
|
| Flag | Effect |
|
||||||
|------|--------|
|
|------|--------|
|
||||||
@ -43,7 +41,6 @@ one, so it never runs unless you ask for it.
|
|||||||
| `--lint` / `--no-lint` | Run only lint / drop lint |
|
| `--lint` / `--no-lint` | Run only lint / drop lint |
|
||||||
| `--test` / `--no-test` | Run only tests / drop tests |
|
| `--test` / `--no-test` | Run only tests / drop tests |
|
||||||
| `--fmt` / `--no-fmt` | Run only format check / drop it |
|
| `--fmt` / `--no-fmt` | Run only format check / drop it |
|
||||||
| `--build` / `--no-build` | Run only the production build / drop it |
|
|
||||||
| `--fix` | Format files instead of checking (other tasks unaffected) |
|
| `--fix` | Format files instead of checking (other tasks unaffected) |
|
||||||
| `--paren-repair` | Fix delimiter errors in Clojure/CLJS files |
|
| `--paren-repair` | Fix delimiter errors in Clojure/CLJS files |
|
||||||
| `--fail-fast` | Stop at the first failure |
|
| `--fail-fast` | Stop at the first failure |
|
||||||
@ -75,12 +72,6 @@ log path.
|
|||||||
# Format the module without running the test suite
|
# Format the module without running the test suite
|
||||||
./scripts/ci --fix --no-test frontend
|
./scripts/ci --fix --no-test frontend
|
||||||
|
|
||||||
# Check that the module still compiles (slow: minutes)
|
|
||||||
./scripts/ci --build frontend
|
|
||||||
|
|
||||||
# Compile every module (8 builds, ~4 min with warm caches)
|
|
||||||
./scripts/ci --build --all
|
|
||||||
|
|
||||||
# Broke delimiters in Clojure/CLJS files: repair first, then lint
|
# Broke delimiters in Clojure/CLJS files: repair first, then lint
|
||||||
./scripts/ci --paren-repair frontend
|
./scripts/ci --paren-repair frontend
|
||||||
./scripts/ci --lint frontend
|
./scripts/ci --lint frontend
|
||||||
@ -98,12 +89,7 @@ log path.
|
|||||||
- Test tasks are long-running (backend runs `clojure -M:dev:test`); give
|
- Test tasks are long-running (backend runs `clojure -M:dev:test`); give
|
||||||
the bash call a generous timeout (10–20 minutes) instead of letting it
|
the bash call a generous timeout (10–20 minutes) instead of letting it
|
||||||
time out mid-run.
|
time out mid-run.
|
||||||
- `mcp` and `media-processor` have no lint task — they show as skipped,
|
- `mcp` has no lint task — it shows as skipped, not failed.
|
||||||
not failed. `common` has no build task for the same reason.
|
|
||||||
- Build runs are the slow ones (frontend ~2 min even with warm caches;
|
|
||||||
cold caches roughly double it). Give the shell a generous timeout and
|
|
||||||
do not run two builds at once in the same checkout: `render-wasm` takes
|
|
||||||
a build lock.
|
|
||||||
- `--paren-repair` only fixes delimiters; run lint afterwards to catch
|
- `--paren-repair` only fixes delimiters; run lint afterwards to catch
|
||||||
what remains. See `mem:scripts/paren-repair`.
|
what remains. See `mem:scripts/paren-repair`.
|
||||||
- What to run and how to read test results: `mem:testing`.
|
- What to run and how to read test results: `mem:testing`.
|
||||||
|
|||||||
@ -1,7 +1,6 @@
|
|||||||
---
|
---
|
||||||
name: make-a-plan
|
name: make-a-plan
|
||||||
description: Planning flow — research the subject of this session, produce an implementation plan with the planner skill, resolve open questions with the user in plain language, and save the final plan to .agents/plans/.
|
description: Planning flow — research the subject of this session, produce an implementation plan with the planner skill, resolve open questions with the user in plain language, and save the final plan to .agents/plans/. Use it when the user asks to plan, design, or break down a task, in any phrasing.
|
||||||
slash: true
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# Make a Plan
|
# Make a Plan
|
||||||
@ -17,18 +16,8 @@ stop — this skill needs the build agent to save the plan.
|
|||||||
- The user asks to plan, design, or break down a task, in any phrasing:
|
- The user asks to plan, design, or break down a task, in any phrasing:
|
||||||
"make a plan", "how would we build X", "design an approach for Y" —
|
"make a plan", "how would we build X", "design an approach for Y" —
|
||||||
or runs `/make-a-plan`.
|
or runs `/make-a-plan`.
|
||||||
- The user asks to rework or extend an existing plan. While the plan is
|
- The user asks to rework or extend an existing plan (for example, after
|
||||||
still unimplemented (pre-`implement-plan` iteration, e.g. after
|
review findings) — revise the saved plan file in place.
|
||||||
`/review-plan` findings or user feedback), revise the saved plan file
|
|
||||||
in place. Once the plan has been implemented and reviewed (post
|
|
||||||
`/review-code` findings on committed work), write a new derived plan
|
|
||||||
file instead — never rewrite the executed plan. The `planner` skill
|
|
||||||
defines the derived naming (`--review-NN`, `--task-NN`).
|
|
||||||
- The user says to apply `review-plan` findings to a still-unimplemented
|
|
||||||
plan: apply the agreed changes, flip `Status` to `reviewed`, and
|
|
||||||
append one UTC ISO 8601 line to `Review Log` describing what changed.
|
|
||||||
This apply step is the only pre-implementation write to those fields —
|
|
||||||
`review-plan` itself stays read-only and never writes.
|
|
||||||
|
|
||||||
Do not use it to execute a plan — that is the `implement-plan` flow.
|
Do not use it to execute a plan — that is the `implement-plan` flow.
|
||||||
|
|
||||||
@ -45,12 +34,8 @@ Do not use it to execute a plan — that is the `implement-plan` flow.
|
|||||||
3. Once all decisions are answered and the plan is final, save it verbatim to the
|
3. Once all decisions are answered and the plan is final, save it verbatim to the
|
||||||
announced path under `.agents/plans/` (create the directory if it does not
|
announced path under `.agents/plans/` (create the directory if it does not
|
||||||
exist). This step is the flow's explicit authorization to write the plan
|
exist). This step is the flow's explicit authorization to write the plan
|
||||||
file — the only write allowed here. A fresh plan uses
|
file — the only write allowed here. If I later ask for changes, update the
|
||||||
`.agents/plans/YYYY-MM-DD-<slug>.md` with `Status: draft` and an
|
saved file directly.
|
||||||
empty `Review Log`; a derived plan uses the parent
|
|
||||||
basename plus the `planner` suffix (`--review-NN`, `--task-NN`) in its
|
|
||||||
own new file, also starting as `draft`. If I later ask for changes to a still-unimplemented plan,
|
|
||||||
update the saved file directly.
|
|
||||||
4. Present me with a clear, self-contained summary of the plan's most relevant points
|
4. Present me with a clear, self-contained summary of the plan's most relevant points
|
||||||
only after all required decisions have been answered. Write it for someone who knows
|
only after all required decisions have been answered. Write it for someone who knows
|
||||||
only the project's high-level goal and may not know the plan's low-level context.
|
only the project's high-level goal and may not know the plan's low-level context.
|
||||||
@ -107,6 +92,9 @@ the final response by suggesting the next steps, in this order:
|
|||||||
These are suggestions, not a required pipeline — any instruction from me
|
These are suggestions, not a required pipeline — any instruction from me
|
||||||
overrides them (for example, asking you to implement the plan directly).
|
overrides them (for example, asking you to implement the plan directly).
|
||||||
|
|
||||||
## User input, overrides and additional context
|
## User context
|
||||||
|
|
||||||
$ARGUMENTS
|
Extra context in the user's invocation (the message that triggered this skill)
|
||||||
|
plays the role command arguments play elsewhere: for example, `delegated` to
|
||||||
|
hand the research and drafting to the `general` subagent, or corrections and
|
||||||
|
feedback about a previous plan.
|
||||||
|
|||||||
@ -1,330 +0,0 @@
|
|||||||
---
|
|
||||||
name: migrate-ds-area
|
|
||||||
description: Migrate one Penpot UI area to the Design System components for one group of the DS migration epic (#12375): buttons and icons (icon-button*, button*, icon*), selects (select*) and later groups. Moves the area's SCSS off the deprecated styles, proves every change with a zero-tolerance visual test, then opens the draft PR and updates the group issue (PRs table row, QA comment). Use when the user asks to migrate an area or module to the DS, continue a DS migration group, or names an area of a group issue.
|
|
||||||
---
|
|
||||||
|
|
||||||
# Skill: migrate-ds-area
|
|
||||||
|
|
||||||
Move one UI area (one `.cljs` + `.scss` pair, or a few related modules)
|
|
||||||
from legacy components and raw HTML to the Design System components,
|
|
||||||
prove with pixels what changed, and ship it as a draft PR that QA can
|
|
||||||
test.
|
|
||||||
|
|
||||||
The epic is #12375. Each control type is a group with its own issue
|
|
||||||
(buttons and icons #12376, selects #12377, radios #12378, …). The group
|
|
||||||
issue holds the plan (order, waiting areas), the changes QA should
|
|
||||||
expect, and the PRs table where people claim areas. Read it first.
|
|
||||||
|
|
||||||
## Input
|
|
||||||
|
|
||||||
- The group issue (e.g. `#12376`) and the area: a module path (e.g.
|
|
||||||
`dashboard/fonts`) or an area named in the group issue.
|
|
||||||
- No area given: take the next unclaimed area in the group issue's plan
|
|
||||||
order, and say which one.
|
|
||||||
|
|
||||||
## Ground rules
|
|
||||||
|
|
||||||
- Never `git push`. The user pushes; then you open the PR.
|
|
||||||
- PRs go to `develop`; branches start from `origin/develop`.
|
|
||||||
- **Stay in the group.** Only the controls of this group change. In the
|
|
||||||
buttons/icons group, selects, radios, numeric/text inputs, checkboxes,
|
|
||||||
menus (`%menu-*`, `%dropdown-*`) and modal structure (`%modal-*`) stay;
|
|
||||||
a checkbox tick or an input label icon stays legacy.
|
|
||||||
- **Accept the DS look.** Do not add CSS to imitate legacy sizes or
|
|
||||||
colors. Do keep positions: icons and labels must not move unless the
|
|
||||||
DS size forces it, and then say so.
|
|
||||||
- No negative margins to compensate; use positive DS spacing or
|
|
||||||
`px2rem()`.
|
|
||||||
- **Gaps in the DS wait.** An area that needs an icon or a feature the
|
|
||||||
DS lacks keeps its legacy control; say so in the PR.
|
|
||||||
- Skip legacy shared components that another group will replace
|
|
||||||
(`context_menu_a11y`, `forms`, `select`, `dropdown`, `radio_buttons`,
|
|
||||||
`search_bar`, `title_bar`, …): they wait for that group's cleanup,
|
|
||||||
where they are updated if still used or deleted if not.
|
|
||||||
- Debug-only UI gets no QA comment and no compare image.
|
|
||||||
|
|
||||||
## Devenv
|
|
||||||
|
|
||||||
Run every test, lint and `pnpm` command inside the workspace container,
|
|
||||||
never on the host (a host `pnpm` relinks `node_modules` and breaks the
|
|
||||||
container's watcher). The container is `penpot-devenv-ws<N>-main`, where
|
|
||||||
`<N>` is the workspace number (`ws0` is the main repo; `wsN` lives in
|
|
||||||
`~/.penpot/penpot_workspaces/wsN`). Write output to a file, then read it:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
C=penpot-devenv-ws1-main # your workspace
|
|
||||||
docker exec -u penpot -w /home/penpot/penpot/frontend $C \
|
|
||||||
bash -lc '<command>' > <scratch>/out.txt 2>&1
|
|
||||||
```
|
|
||||||
|
|
||||||
When the app does not behave as the code says, check the build first:
|
|
||||||
|
|
||||||
- **Stale build after a branch switch** ("is not a function", or a page
|
|
||||||
with empty `stl/css` classes): `touch` the stale `.cljs`/`.scss`, wait
|
|
||||||
for the compiled file under `frontend/resources/public/` to update,
|
|
||||||
hard-reload.
|
|
||||||
- **A whole area unstyled:** a new `.scss` added after the watcher
|
|
||||||
started is missing from `main.css` (grep `main.css` for the module
|
|
||||||
prefix), or `main.css` was written corrupted after many `.scss`
|
|
||||||
changed at once. Touch a `.scss`; if the module is still missing,
|
|
||||||
restart the watcher in the workspace's tmux session (`penpot:0`,
|
|
||||||
`./scripts/watch app`).
|
|
||||||
- **Blank app after switching between `develop` and `staging`:**
|
|
||||||
dependencies differ. Run `pnpm install --frozen-lockfile` in the
|
|
||||||
container and restart the watcher.
|
|
||||||
|
|
||||||
## Workflow
|
|
||||||
|
|
||||||
### 1. Claim the area and branch
|
|
||||||
|
|
||||||
1. Read the group issue: plan, waiting areas, expected changes, PRs table.
|
|
||||||
2. Add a row to the PRs table with no PR number, the user's @login as
|
|
||||||
Owner and the state `Claimed` (`gh issue view/edit <n> --body-file`).
|
|
||||||
Read the body right before editing, and read it again after saving:
|
|
||||||
when two people edit at the same time, GitHub keeps only the last
|
|
||||||
edit, so a row can vanish. Put yours back if it is gone.
|
|
||||||
Skip an area someone else already claimed, unless the claim has had
|
|
||||||
no PR for 7 days and its owner did not answer a comment asking.
|
|
||||||
3. `git switch -c <user>-ds-<group>-<area> origin/develop`.
|
|
||||||
4. Survey the area: every control of the group in the `.cljs`, the
|
|
||||||
legacy icons, the placeholders and mixins the `.scss` uses, the
|
|
||||||
`div`/`span` with `:on-click`.
|
|
||||||
|
|
||||||
### 2. Visual baseline
|
|
||||||
|
|
||||||
Add a `test.describe("<Area> controls", …)` block to the matching spec
|
|
||||||
in `frontend/playwright/ui/visual-specs/` (project `ds`, dark theme).
|
|
||||||
Put it next to the blocks of the same screen, not always at the end of
|
|
||||||
the file: two PRs that both append at the end conflict.
|
|
||||||
|
|
||||||
- Every shot: `toHaveScreenshot({ maxDiffPixelRatio: 0, threshold: 0 })`.
|
|
||||||
- Show every changed control at rest, on hover and open or pressed.
|
|
||||||
Capture the element or the smallest stable container.
|
|
||||||
- Locators must work before and after the change: text, role, testid,
|
|
||||||
`[class*='…']`, or `a.or(b)` when the element type changes (raw
|
|
||||||
`[class*='menu-btn']` before, `getByRole("button", { name: "Options" })`
|
|
||||||
after). Scope `.or()` to the area so it cannot match elsewhere.
|
|
||||||
- Missing data? Add a fixture under `frontend/playwright/data/` and mock
|
|
||||||
it (`dashboardPage.mockRPC(...)`, `DashboardPage.mockConfigFlags` for
|
|
||||||
flags). Watch for expired or inconsistent fixtures (dates in the past,
|
|
||||||
wrong file for a mock).
|
|
||||||
- Capture with `--update-snapshots=all`:
|
|
||||||
```bash
|
|
||||||
npx playwright test --project=ds <spec> -g "<describe>" --update-snapshots=all
|
|
||||||
```
|
|
||||||
Plain `--update-snapshots` keeps a snapshot within the project's 0.5%
|
|
||||||
tolerance, so a real diff looks like 0.
|
|
||||||
- Copy the snapshots (`<spec>.spec.js-snapshots/<Describe>-*`) to
|
|
||||||
`<scratch>/diffs/<area>/before/` and look at them: the state you
|
|
||||||
wanted must really be on screen.
|
|
||||||
|
|
||||||
### 3. Controls commit
|
|
||||||
|
|
||||||
#### Buttons and icons
|
|
||||||
|
|
||||||
| Legacy | DS |
|
|
||||||
|---|---|
|
|
||||||
| icon-only raw button | `[:> icon-button* {:variant "ghost" :aria-label (tr …) :icon i/x :on-click f}]` (always pass `:variant`; `"action"` for 24px icon toggles) |
|
|
||||||
| pressed/selected | `:aria-pressed (boolean x)` |
|
|
||||||
| text button | `[:> button* {:variant "primary"/"secondary"/"ghost"/"destructive" :type "button"} label]` |
|
|
||||||
| `[:input {:type "button"}]` | `button*` |
|
|
||||||
| modal close | `icon-button*` ghost, `:aria-label (tr "labels.close")`, `:tooltip-class` carrying the corner position |
|
|
||||||
| legacy icon outside a button | `[:> icon* {:icon-id i/x :class …}]` |
|
|
||||||
|
|
||||||
- Not every raw button changes: rows, tabs, swatches, avatar widgets,
|
|
||||||
full-height segments, round custom buttons and link-style text buttons
|
|
||||||
stay raw (their legacy icon still changes). List the kept ones in the
|
|
||||||
PR.
|
|
||||||
- A `div`/`span` with `:on-click` in the area becomes a `[:button]`
|
|
||||||
(with `aria-expanded`/`aria-haspopup` when it opens something). An
|
|
||||||
`[:a]` without `href` stays; list it in the PR.
|
|
||||||
- **Tooltip wrapper.** `icon-button*` renders inside a tooltip trigger
|
|
||||||
wrapper. Position (`position: absolute`, grid placement, margins
|
|
||||||
against siblings) goes on `:tooltip-class`; look (`--button-height`,
|
|
||||||
colors) goes on `:class`.
|
|
||||||
- **Tooltip at the right edge.** Small icon buttons at the right end of
|
|
||||||
a sidebar row or title bar get `:tooltip-placement "top-left"`; list
|
|
||||||
it as an expected change.
|
|
||||||
- **Sizes through custom properties:** `--button-height: #{px2rem(64)}` /
|
|
||||||
`--button-width`. Sass does not evaluate functions inside custom
|
|
||||||
properties without `#{}`.
|
|
||||||
- **A grown `button*` keeps its label at the start**: add
|
|
||||||
`justify-content: center` when it is wider than its content.
|
|
||||||
- **Icon colors:** DS icons use `stroke: currentColor`. Set `color` on
|
|
||||||
the icon class, not `stroke`/`fill`. A legacy `color: transparent` left
|
|
||||||
on the button or wrapper makes the DS icon invisible: drop it. A class
|
|
||||||
that redefines the color variable on the icon breaks the hover that
|
|
||||||
changes it on the button: define the variable on the button.
|
|
||||||
- **Every DS icon needs a color of its own** (on its class or a parent):
|
|
||||||
one that inherits none shows in the devenv debug yellow.
|
|
||||||
- **Icon sizes:** `"s"` draws 12px inside a 16px box (compensate the
|
|
||||||
margin by 2px per side to keep the drawing in place); `"m"` 16px; `"l"`
|
|
||||||
24px; `"xl"` 32px.
|
|
||||||
- **Stroke width:** DS icons keep the DS stroke width. Never set
|
|
||||||
`stroke-width` on a DS icon to match the legacy one; a thinner or
|
|
||||||
thicker line is part of the DS look.
|
|
||||||
- **Legacy placeholders that style `svg` descendants** (e.g.
|
|
||||||
`%dropdown-element-base` → `span svg { 12px box; stroke }`) override a
|
|
||||||
DS icon placed inside them: give the icon class its own box
|
|
||||||
(`inline-size`/`block-size` 16px) and `stroke: currentcolor` with more
|
|
||||||
specificity.
|
|
||||||
- **Flex shrink:** a DS svg inside a small flex button can shrink and be
|
|
||||||
cut; add `flex-shrink: 0` to the icon class.
|
|
||||||
- **Rotated legacy arrows:** prefer `i/arrow-left`/`-right`/`-up`/`-down`
|
|
||||||
over rotating `i/arrow` (same path), unless a rotation class drives the
|
|
||||||
open/closed state.
|
|
||||||
- **Menus anchored by the button size** move when the button grows; open
|
|
||||||
them from the top (`inset-block-start: calc(100% + var(--sp-xs))`) or
|
|
||||||
fix the context menu offset, and say so.
|
|
||||||
|
|
||||||
#### Selects
|
|
||||||
|
|
||||||
`[:& select {:default-value v :options [{:value :label}] :on-change f}]`
|
|
||||||
→ `[:> select* {:default-selected <id> :options <memoized vector of {:id :label}> :on-change f}]`
|
|
||||||
(`app.main.ui.ds.controls.select`).
|
|
||||||
|
|
||||||
- **Ids are non-blank strings.** A blank id shows a dimmed "--", and an
|
|
||||||
id that matches no option falls back to the first one. Give mixed,
|
|
||||||
none and auto their own ids (`"multiple"`, `"none"`, `"auto"`) and map
|
|
||||||
them back in the handler. Use `""` only where "--" is the wanted look.
|
|
||||||
- **Convert the value.** The legacy select ran `d/read-string`, so
|
|
||||||
handlers got keywords, numbers or uuids; `select*` passes the `:id`
|
|
||||||
string. Convert with `keyword`, `d/parse-integer` or `uuid/parse`.
|
|
||||||
- **Memoized options, live selection.** `select*` resets its selection
|
|
||||||
when `:default-selected` or the `:options` identity changes. Build
|
|
||||||
options with `mf/with-memo` and `mapv` (the schema wants a non-empty
|
|
||||||
vector) and pass `:default-selected` from the live value, never a
|
|
||||||
constant.
|
|
||||||
- **Never pass nil.** In dev builds a present prop with value nil
|
|
||||||
(`:class`, `:wrapper-class`, `:disabled`, `:default-selected`) throws
|
|
||||||
`invalid props on component select*`. Use `(boolean x)`, `d/nilv`, or
|
|
||||||
add the prop only when it has a value.
|
|
||||||
- **Props the legacy select ignored stay out**: `select*` passes rest
|
|
||||||
props to its button, so they would reach the DOM.
|
|
||||||
- **Classes:** `:wrapper-class` for layout, `:class` for look.
|
|
||||||
- **Portal:** add `:has-portal true` only where the visual test shows the
|
|
||||||
list cut at the bottom of a panel, and say so in the QA comment.
|
|
||||||
- **Locators:** the button is a `combobox` with no accessible name; use
|
|
||||||
`getByRole("combobox").filter({ hasText: "Linear" })` (it matches the
|
|
||||||
legacy select too) and `getByRole("option", { name: … })`.
|
|
||||||
- **Keyboard:** the arrows no longer change the value while the list is
|
|
||||||
closed. It is an expected change, already listed in #12377.
|
|
||||||
|
|
||||||
#### All groups
|
|
||||||
|
|
||||||
- New labels: add the key to `en.po` and `es.po` by hand, in sorted
|
|
||||||
position, with a `#:` reference to the cljs line. Never run the
|
|
||||||
translations sync.
|
|
||||||
- The `.scss` only gets what the new controls need in this commit.
|
|
||||||
- Some e2e specs are not Prettier-clean on develop: never run
|
|
||||||
`prettier --write` on a whole spec; format only your block.
|
|
||||||
|
|
||||||
Then wait for the watcher (grep the compiled JS/CSS under
|
|
||||||
`frontend/resources/public/` for the change), recapture with `=all`,
|
|
||||||
copy to `after/`, diff:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
SK=.agents/skills/migrate-ds-area/scripts
|
|
||||||
python3 -I $SK/diff.py <before> <after> # per-file pixel count + bbox
|
|
||||||
python3 -I $SK/trip.py <b> <a> <out.png> <scale> # before | after | red diff
|
|
||||||
```
|
|
||||||
|
|
||||||
`trip.py` pads images of different sizes, so a taller "after" still
|
|
||||||
renders.
|
|
||||||
Explain every diff. Fix what is not a DS-look change (moved icons, cut
|
|
||||||
icons, lost hover color). Rerun a shot that changes on unchanged code
|
|
||||||
before trusting it; flaky spots seen so far: the invite modal textarea
|
|
||||||
border, the search input caret, the user account page, the path handler
|
|
||||||
menu trigger.
|
|
||||||
|
|
||||||
Lint, then commit (read `mem:workflow/creating-commits` first):
|
|
||||||
|
|
||||||
```bash
|
|
||||||
npx prettier --check <spec> <fixtures>; npx stylelint <scss>; clj-kondo --lint <cljs>; cljfmt check <cljs>
|
|
||||||
git add -f <cljs> <scss> <po> <spec> <fixtures> # -f: visual-specs/ is gitignored
|
|
||||||
./scripts/check-commit
|
|
||||||
```
|
|
||||||
|
|
||||||
A plain `git add` with an ignored path in the list adds nothing; always
|
|
||||||
`-f`, and check `git show --stat`. Never commit the screenshots.
|
|
||||||
Subject: `:recycle: Use DS <controls> in <area>`. A pre-existing bug
|
|
||||||
found and fixed on the way goes in its own `:bug:` commit.
|
|
||||||
|
|
||||||
### 4. SCSS commit
|
|
||||||
|
|
||||||
Run the `refactor-scss` skill on every `.scss` the PR touched (only
|
|
||||||
those), in its own commit. Keep `refactor/` only for component-family
|
|
||||||
placeholders (`%modal-*`, `%menu-*`, `%input-*`, `%checkbox-*`) and say
|
|
||||||
why in a comment above the `@use`. Unused selectors are deleted, not
|
|
||||||
commented out (list them in the commit body). Typography without an
|
|
||||||
exact match uses the closest DS typography. Run the skill's
|
|
||||||
**Self-review** before committing: no bare element selectors (add
|
|
||||||
classes in the `.cljs`), no copied properties that do nothing, z-index
|
|
||||||
tokens by purpose. Compare against the controls commit: it should be
|
|
||||||
0 px; explain anything else (typically a typography or border change).
|
|
||||||
For files that several modules share or that are huge, stop and propose
|
|
||||||
a separate PR instead.
|
|
||||||
|
|
||||||
### 5. Hand off, PR and tracking
|
|
||||||
|
|
||||||
1. Report to the user: commits, visible changes, kept raw controls,
|
|
||||||
pre-existing bugs, and the compare image path
|
|
||||||
(`<scratch>/diffs/<area>/compare.png`, before | after | diff). Give
|
|
||||||
the push command: `git push -u origin <branch>` (the branch tracks
|
|
||||||
`origin/develop`, so a bare `git push` is refused).
|
|
||||||
2. After the user pushes, check `git rev-parse HEAD origin/<branch>`,
|
|
||||||
then (read `mem:workflow/creating-prs` first):
|
|
||||||
```bash
|
|
||||||
gh pr create --repo penpot/penpot --draft --base develop --head <branch> \
|
|
||||||
--title "<commit subject>" --body-file <body> \
|
|
||||||
--label "WG adoption" --label "no issue required" \
|
|
||||||
--milestone <group issue milestone> --project Main
|
|
||||||
gh pr edit <n> --repo penpot/penpot --remove-label "needs triage"
|
|
||||||
```
|
|
||||||
Never add `no changelog` to a PR: that label goes on issues.
|
|
||||||
Body: AI note, `Part of #<group> (DS migration epic #12375).`, a
|
|
||||||
short What / How, the kept raw controls, an empty `## Screenshots`
|
|
||||||
when something is visible (the user attaches the image). Never
|
|
||||||
`Closes #<group>`. Keep it short: reviewers find long PRs hard to read.
|
|
||||||
3. In the group issue's PRs table, fill your row:
|
|
||||||
`| #<n> | <area>: <controls> (<visible changes>) | @<owner> | Draft |`.
|
|
||||||
4. Post a QA comment on the group issue: `**#<n>: <area>**`, *Where* (how
|
|
||||||
to reach it), *Expected changes* (every visible change; QA treats
|
|
||||||
anything not listed as a bug), *No other change* (what still works
|
|
||||||
the same).
|
|
||||||
|
|
||||||
Do not mark the draft ready or ask for review: the user decides when.
|
|
||||||
|
|
||||||
### 6. Review fixes
|
|
||||||
|
|
||||||
When the reviewer comments on a PR:
|
|
||||||
|
|
||||||
1. Fix each comment in a new commit on the branch (never amend pushed
|
|
||||||
commits). A comment you do not apply gets a short reason the user can
|
|
||||||
post, with an image when it is visual.
|
|
||||||
2. Rerun the area's visual spec before and after the fix with `=all`.
|
|
||||||
Build a compare image for every visible change and give the user its
|
|
||||||
full path, saying which PR it goes to.
|
|
||||||
3. Update everything the fix makes stale, in the same turn: the QA
|
|
||||||
comment (*Expected changes* must list the new visible changes), the
|
|
||||||
PRs table row, and the PR body.
|
|
||||||
4. When working on several PRs in one go, report per PR: branch, new
|
|
||||||
commit, what changed, image path. Never mix two branches in one list.
|
|
||||||
5. Give the push command per branch. If a rebase is coming soon, say
|
|
||||||
that pushing now costs one extra CI run; the user decides.
|
|
||||||
6. After a round of review fixes, check whether the same issue exists in
|
|
||||||
the user's other open PRs of the group, and fix it there too.
|
|
||||||
|
|
||||||
### 7. After merges
|
|
||||||
|
|
||||||
Whenever PRs merge (or go from draft to ready), sync the state column of
|
|
||||||
the group's PRs table with GitHub for **every** row, not only your own:
|
|
||||||
read each row's PR (`gh pr view <n> --json state,isDraft`) and set
|
|
||||||
Merged / In review / Draft. The table drifts when other people merge.
|
|
||||||
|
|
||||||
## Output
|
|
||||||
|
|
||||||
- Branch with 2 commits (controls, SCSS) plus any `:bug:` commit.
|
|
||||||
- Visual spec block with zero-tolerance shots.
|
|
||||||
- Compare image in the scratch folder.
|
|
||||||
- After the push: draft PR, row and QA comment in the group issue.
|
|
||||||
@ -1,25 +0,0 @@
|
|||||||
"""Pixel diff of two folders of PNG snapshots with the same file names.
|
|
||||||
|
|
||||||
Usage: python3 -I diff.py <before-dir> <after-dir>
|
|
||||||
Prints, per file, the number of differing pixels and their bounding box,
|
|
||||||
or the two sizes when they differ.
|
|
||||||
"""
|
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
|
|
||||||
from PIL import Image, ImageChops
|
|
||||||
|
|
||||||
before, after = sys.argv[1], sys.argv[2]
|
|
||||||
for name in sorted(os.listdir(before)):
|
|
||||||
other = os.path.join(after, name)
|
|
||||||
if not os.path.exists(other):
|
|
||||||
print(name, "missing in", after)
|
|
||||||
continue
|
|
||||||
a = Image.open(os.path.join(before, name)).convert("RGB")
|
|
||||||
b = Image.open(other).convert("RGB")
|
|
||||||
if a.size != b.size:
|
|
||||||
print(name, "size", a.size, b.size)
|
|
||||||
continue
|
|
||||||
diff = ImageChops.difference(a, b)
|
|
||||||
count = sum(1 for p in diff.getdata() if p != (0, 0, 0))
|
|
||||||
print(name, count, diff.getbbox())
|
|
||||||
@ -1,33 +0,0 @@
|
|||||||
"""Before | after | red diff image, one row per snapshot.
|
|
||||||
|
|
||||||
Usage: python3 -I trip.py <before-dir> <after-dir> <out.png> <scale>
|
|
||||||
Both folders hold PNGs with the same names (crop them first if needed).
|
|
||||||
"""
|
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
|
|
||||||
from PIL import Image, ImageChops
|
|
||||||
|
|
||||||
before, after, out, scale = sys.argv[1], sys.argv[2], sys.argv[3], int(sys.argv[4])
|
|
||||||
rows = []
|
|
||||||
for name in sorted(os.listdir(before)):
|
|
||||||
a = Image.open(os.path.join(before, name)).convert("RGB")
|
|
||||||
b = Image.open(os.path.join(after, name)).convert("RGB")
|
|
||||||
w, h = max(a.width, b.width), max(a.height, b.height)
|
|
||||||
pad_a = Image.new("RGB", (w, h), (200, 200, 200)); pad_a.paste(a, (0, 0))
|
|
||||||
pad_b = Image.new("RGB", (w, h), (200, 200, 200)); pad_b.paste(b, (0, 0))
|
|
||||||
mask = ImageChops.difference(pad_a, pad_b).convert("L").point(lambda v: 255 if v else 0)
|
|
||||||
red = Image.new("RGB", (w, h), (255, 255, 255))
|
|
||||||
red.paste((255, 0, 0), mask=mask)
|
|
||||||
row = Image.new("RGB", (w * 3 + 20, h), (200, 200, 200))
|
|
||||||
for i, im in enumerate([pad_a, pad_b, red]):
|
|
||||||
row.paste(im, (i * (w + 10), 0))
|
|
||||||
rows.append(row.resize((row.width * scale, row.height * scale), Image.NEAREST))
|
|
||||||
width = max(r.width for r in rows)
|
|
||||||
height = sum(r.height + 10 for r in rows)
|
|
||||||
canvas = Image.new("RGB", (width, height), (255, 255, 255))
|
|
||||||
y = 0
|
|
||||||
for r in rows:
|
|
||||||
canvas.paste(r, (0, y))
|
|
||||||
y += r.height + 10
|
|
||||||
canvas.save(out)
|
|
||||||
@ -70,7 +70,7 @@ Skipping this step is the #1 cause of incorrect or incomplete plans.
|
|||||||
Each task follows this structure:
|
Each task follows this structure:
|
||||||
|
|
||||||
```markdown
|
```markdown
|
||||||
### Task [N]: [Short descriptive title]
|
## Task [N]: [Short descriptive title]
|
||||||
|
|
||||||
**Description:** One or two paragraphs explaining what this task accomplishes.
|
**Description:** One or two paragraphs explaining what this task accomplishes.
|
||||||
Should be clear and concise.
|
Should be clear and concise.
|
||||||
@ -153,45 +153,6 @@ Announce the save path `.agents/plans/YYYY-MM-DD-<slug>.md` (today's date,
|
|||||||
lowercase hyphen-separated slug, e.g. `2026-09-10-add-batch-get-profiles`;
|
lowercase hyphen-separated slug, e.g. `2026-09-10-add-batch-get-profiles`;
|
||||||
an explicit user path wins).
|
an explicit user path wins).
|
||||||
|
|
||||||
### Derived plans
|
|
||||||
|
|
||||||
Never invent a fresh slug when the plan derives from an existing one.
|
|
||||||
The derived name is `<parent-basename>` plus one suffix per level,
|
|
||||||
joined with `--` (double hyphen; single hyphens already separate
|
|
||||||
slug words, so `--` marks where the derivation starts). The parent
|
|
||||||
name is never edited, and no new date is added — the parent prefix
|
|
||||||
already carries its date, which keeps parent and derivatives adjacent
|
|
||||||
in `ls`. Record the real creation date inside the plan (`Created:`).
|
|
||||||
|
|
||||||
Valid names match:
|
|
||||||
|
|
||||||
```
|
|
||||||
^\d{4}-\d{2}-\d{2}-[a-z0-9-]+(--(review-\d{2}|task-\d{2})(-[a-z0-9-]+)?)*\.md$
|
|
||||||
```
|
|
||||||
|
|
||||||
- `review-NN` — a new plan addressing findings of a `review-code` or
|
|
||||||
`review-plan` on already-implemented work. `NN` counts reviews of
|
|
||||||
that parent from `01`. Example: parent
|
|
||||||
`2026-09-14-paste-before-init-crash.md` →
|
|
||||||
`2026-09-14-paste-before-init-crash--review-01.md`,
|
|
||||||
then `--review-02.md`.
|
|
||||||
- `task-NN-<short-slug>` — sub-plan for task `NN` of a high-level
|
|
||||||
roadmap plan. `NN` is the roadmap task number. Example: parent
|
|
||||||
`2026-09-20-upload-pipeline-roadmap.md` →
|
|
||||||
`2026-09-20-upload-pipeline-roadmap--task-01-chunk-upload.md`.
|
|
||||||
Levels chain: `...--task-02-gc--review-01.md`.
|
|
||||||
|
|
||||||
Never use `v2`, `final`, `new`, or `fix2` as suffixes. Keep the
|
|
||||||
optional short slug to 3-4 lowercase hyphen-separated words.
|
|
||||||
|
|
||||||
Every derived plan opens its `Context` with:
|
|
||||||
|
|
||||||
```markdown
|
|
||||||
Parent: `<parent-basename>.md`
|
|
||||||
Source: review-code over `<commit>` (branch `<branch>`) | task `NN` of roadmap `<parent-basename>.md`
|
|
||||||
Created: YYYY-MM-DD
|
|
||||||
```
|
|
||||||
|
|
||||||
End the response by suggesting the next steps: `/review-plan` to get a second
|
End the response by suggesting the next steps: `/review-plan` to get a second
|
||||||
opinion on the plan and `/implement-plan` to execute it.
|
opinion on the plan and `/implement-plan` to execute it.
|
||||||
|
|
||||||
@ -202,10 +163,6 @@ Use this document shape:
|
|||||||
```markdown
|
```markdown
|
||||||
# Plan: Title
|
# Plan: Title
|
||||||
|
|
||||||
Status: draft | reviewed | done
|
|
||||||
Review Log:
|
|
||||||
- YYYY-MM-DDTHH:MM:SSZ — <what changed and why, one line per entry>
|
|
||||||
|
|
||||||
## Context
|
## Context
|
||||||
## Affected Modules
|
## Affected Modules
|
||||||
## Architecture Decisions
|
## Architecture Decisions
|
||||||
@ -217,23 +174,6 @@ Review Log:
|
|||||||
## Open Questions
|
## Open Questions
|
||||||
```
|
```
|
||||||
|
|
||||||
`Status` and `Review Log` are write-restricted metadata, not free text:
|
|
||||||
|
|
||||||
- `make-a-plan` creates every plan with `Status: draft` and an empty
|
|
||||||
log. `reviewed` never means "a review was emitted" — it means
|
|
||||||
"review feedback was incorporated". A plan approved with no changes
|
|
||||||
goes `draft` → `done` without passing through `reviewed`.
|
|
||||||
- Only a user-triggered apply step writes them before implementation:
|
|
||||||
when the user says to apply `review-plan` findings, `make-a-plan`
|
|
||||||
applies the changes, flips to `reviewed`, and appends one UTC
|
|
||||||
ISO 8601 line describing what changed.
|
|
||||||
- `implement-plan` flips to `done` on completion and appends one line
|
|
||||||
with the issue URL when one exists (standalone mode); otherwise
|
|
||||||
just `done`. Never record commit hashes — they rot on amend/rebase
|
|
||||||
and git already links the commit.
|
|
||||||
- The log is append-only: never rewrite or delete lines.
|
|
||||||
- `review-plan` and `review-code` never write these fields.
|
|
||||||
|
|
||||||
Omit empty sections only when they do not apply. Every implementation task
|
Omit empty sections only when they do not apply. Every implementation task
|
||||||
still requires acceptance criteria, verification, dependencies, likely files,
|
still requires acceptance criteria, verification, dependencies, likely files,
|
||||||
and scope.
|
and scope.
|
||||||
|
|||||||
@ -1,421 +0,0 @@
|
|||||||
---
|
|
||||||
name: refactor-scss
|
|
||||||
description: Migrate one Penpot SCSS module off the deprecated styles under frontend/resources/styles/common/refactor/ to the current Design System (ds/ tokens, mixins and px2rem). Pure refactor, no visual change. Use when the user asks to refactor, migrate or clean up a .scss file to the DS.
|
|
||||||
---
|
|
||||||
|
|
||||||
# Skill: refactor-scss
|
|
||||||
|
|
||||||
Refactor a single SCSS file so that it no longer depends on the deprecated
|
|
||||||
styling system, using the current Design System instead.
|
|
||||||
|
|
||||||
This is a **pure refactor**. Do NOT introduce visual or behavioral changes.
|
|
||||||
Preserve the generated CSS behavior exactly unless a rule below requires
|
|
||||||
otherwise. The rules that allow a small visible change (closest DS
|
|
||||||
typography, DS border width) want it shown in the visual check and listed
|
|
||||||
in the report and the commit body.
|
|
||||||
|
|
||||||
Reviewers read every line of the result. A line that is
|
|
||||||
copied but does nothing, a bare element selector or a raw z-index comes
|
|
||||||
back as a review comment, so the rules below are strict about them.
|
|
||||||
|
|
||||||
## Input
|
|
||||||
|
|
||||||
- The path to one `.scss` file (e.g. `frontend/src/app/main/ui/foo/bar.scss`).
|
|
||||||
- If no path is given, ask for it. Do not pick a file yourself.
|
|
||||||
|
|
||||||
## Workflow
|
|
||||||
|
|
||||||
1. Read the target `.scss` file and the `.cljs` file with the same name next
|
|
||||||
to it.
|
|
||||||
2. Read every deprecated source the file uses, so you inline real values,
|
|
||||||
never remembered ones:
|
|
||||||
- `frontend/resources/styles/common/refactor/mixins.scss`
|
|
||||||
- `frontend/resources/styles/common/refactor/design-tokens.scss`
|
|
||||||
- `frontend/resources/styles/common/refactor/spacing.scss`
|
|
||||||
- any other file under `frontend/resources/styles/common/refactor/` that
|
|
||||||
the stylesheet pulls in (placeholders, borders, z-index, fonts...).
|
|
||||||
3. Read the DS modules you will map to:
|
|
||||||
- `frontend/src/app/main/ui/ds/_utils.scss` (`px2rem()`)
|
|
||||||
- `frontend/src/app/main/ui/ds/spacing.scss`
|
|
||||||
- `frontend/src/app/main/ui/ds/_sizes.scss`
|
|
||||||
- `frontend/src/app/main/ui/ds/typography.scss` (`use-typography()`)
|
|
||||||
- `frontend/src/app/main/ui/ds/_borders.scss`
|
|
||||||
- `frontend/src/app/main/ui/ds/z-index.scss`
|
|
||||||
4. Apply the rules below and edit the file in place.
|
|
||||||
5. Verify (see **Verification** and **Visual check**).
|
|
||||||
6. Report (see **Output**). Do not commit. When the refactor goes with other
|
|
||||||
changes, it belongs in its own commit, after the task's commits.
|
|
||||||
|
|
||||||
## Primary goal
|
|
||||||
|
|
||||||
After the refactor, the SCSS file must not depend on anything under
|
|
||||||
`frontend/resources/styles/common/refactor/`. This includes imports, mixins,
|
|
||||||
placeholder selectors, functions, deprecated variables and design-token
|
|
||||||
aliases.
|
|
||||||
|
|
||||||
### Partial refactor
|
|
||||||
|
|
||||||
Some placeholders stand for a whole component family that will move to the DS
|
|
||||||
in its own PR: `%input-element`, `%input-with-label`, `%input-checkbox`,
|
|
||||||
`%modal-*`, `%menu-dropdown`, `%menu-item-base`, `%shortcut-*` and similar.
|
|
||||||
Inlining them in a small task produces big, risky diffs. In that case:
|
|
||||||
|
|
||||||
- migrate everything else (tokens, sizes, spacing, small mixins);
|
|
||||||
- keep the `refactor/` import only for those placeholders;
|
|
||||||
- list each remaining dependency and the selectors that use it in the report,
|
|
||||||
so the user can plan it as a separate PR.
|
|
||||||
|
|
||||||
## General rules
|
|
||||||
|
|
||||||
- Keep the resulting SCSS as small and readable as possible.
|
|
||||||
- Avoid duplicated declarations and duplicated selectors.
|
|
||||||
- Do not change class names.
|
|
||||||
- Do not change DOM structure.
|
|
||||||
- Do not change specificity unless required.
|
|
||||||
- Follow the project's SCSS formatting (Prettier, see **Verification**).
|
|
||||||
|
|
||||||
## Imports
|
|
||||||
|
|
||||||
Use the DS modules. Import them with `as *` when they expose mixins,
|
|
||||||
variables or functions:
|
|
||||||
|
|
||||||
```scss
|
|
||||||
@use "ds/_utils.scss" as *;
|
|
||||||
```
|
|
||||||
|
|
||||||
not
|
|
||||||
|
|
||||||
```scss
|
|
||||||
@use "ds/_utils.scss" as utils;
|
|
||||||
```
|
|
||||||
|
|
||||||
Do not introduce namespaces such as `utils.`, `typography.`, `borders.`,
|
|
||||||
`spacing.` or `sizes.` unless a module explicitly requires one.
|
|
||||||
|
|
||||||
Remove every unused import.
|
|
||||||
|
|
||||||
## Deprecated mixins
|
|
||||||
|
|
||||||
If the stylesheet uses mixins from
|
|
||||||
`frontend/resources/styles/common/refactor/mixins.scss`:
|
|
||||||
|
|
||||||
- locate the mixin;
|
|
||||||
- inline its generated CSS (then apply the rest of these rules to it);
|
|
||||||
- remove the `@include`;
|
|
||||||
- remove the import.
|
|
||||||
|
|
||||||
Do not leave deprecated mixins.
|
|
||||||
|
|
||||||
## Placeholders
|
|
||||||
|
|
||||||
Replace every `@extend %placeholder` with the generated CSS. Remove every
|
|
||||||
deprecated placeholder, except the component-family ones described in
|
|
||||||
**Partial refactor**.
|
|
||||||
|
|
||||||
Inline only what does something on that element. Do not paste a
|
|
||||||
placeholder (or mixin) body as it is. For each property, check:
|
|
||||||
|
|
||||||
- **It applies to the element.** `fill`, `stroke` and `stroke-width` only
|
|
||||||
act on SVG shapes: drop them from a `span`, `div` or `button`.
|
|
||||||
`color: transparent` on a wrapper that only holds an icon does nothing
|
|
||||||
useful: drop it and set the real color.
|
|
||||||
- **It is not the default or already given.** `stroke-width: 1px` is the
|
|
||||||
SVG default; `fill: none` and `stroke: currentColor` are already set by
|
|
||||||
the DS `icon*` (`.icon`). Drop them.
|
|
||||||
- **A later rule does not override it.**
|
|
||||||
- **It is not a stroke width on a DS icon.** DS icons keep the DS
|
|
||||||
stroke width; drop any `stroke-width` that only imitates the legacy
|
|
||||||
icon (1.33px, 2.25px...).
|
|
||||||
|
|
||||||
`%button-icon` is the usual case: on a wrapper it leaves `color:
|
|
||||||
transparent; fill: none; stroke-width: 1px`, and none of the three is
|
|
||||||
needed.
|
|
||||||
|
|
||||||
## Element selectors
|
|
||||||
|
|
||||||
Do not style HTML elements directly (`svg {}`, `h3 {}`, `button {}`,
|
|
||||||
`input.x {}`, or `.component { svg {} }`).
|
|
||||||
|
|
||||||
When no class exists, **add one in the `.cljs`**: a DS component takes
|
|
||||||
`:class (stl/css :x)` (`icon*`, `heading*`, `button*`…), and a plain
|
|
||||||
element takes `{:class (stl/css :x)}`. Touching the `.cljs` for this is
|
|
||||||
part of the refactor. A comment like "the icon has no class" is not an
|
|
||||||
answer.
|
|
||||||
|
|
||||||
Prefer the DS component when one fits: `heading*` for `h1`–`h6`
|
|
||||||
(`:level`, `:typography`), `text*` for text, `icon*` for icons.
|
|
||||||
|
|
||||||
`element.class` (e.g. `input.element-title`) is not needed to beat the
|
|
||||||
input defaults: try the class alone and check it in the visual check.
|
|
||||||
|
|
||||||
An element selector stays only for markup you cannot reach (injected
|
|
||||||
HTML, third-party widgets), with a comment saying so.
|
|
||||||
|
|
||||||
**Icon colors.** A DS icon draws with `stroke: currentColor`. Set
|
|
||||||
`color` on the icon class or its parent, never `stroke`/`fill` on an
|
|
||||||
`svg` selector.
|
|
||||||
|
|
||||||
## Color variables
|
|
||||||
|
|
||||||
Whenever a color changes across states, introduce a component-local CSS
|
|
||||||
variable.
|
|
||||||
|
|
||||||
Instead of
|
|
||||||
|
|
||||||
```scss
|
|
||||||
.button {
|
|
||||||
color: var(--color-foreground-primary);
|
|
||||||
|
|
||||||
&:hover {
|
|
||||||
color: var(--color-accent-primary);
|
|
||||||
}
|
|
||||||
|
|
||||||
&:disabled {
|
|
||||||
color: var(--color-foreground-disabled);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
write
|
|
||||||
|
|
||||||
```scss
|
|
||||||
.button {
|
|
||||||
--button-foreground-color: var(--color-foreground-primary);
|
|
||||||
|
|
||||||
color: var(--button-foreground-color);
|
|
||||||
|
|
||||||
&:hover {
|
|
||||||
--button-foreground-color: var(--color-accent-primary);
|
|
||||||
}
|
|
||||||
|
|
||||||
&:disabled {
|
|
||||||
--button-foreground-color: var(--color-foreground-disabled);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Use this pattern for `color`, `background-color`, `border-color`,
|
|
||||||
`outline-color`, `fill`, `stroke` and any similar property.
|
|
||||||
|
|
||||||
## Design token aliases
|
|
||||||
|
|
||||||
Remove every alias defined in
|
|
||||||
`frontend/resources/styles/common/refactor/design-tokens.scss`. Resolve each
|
|
||||||
alias to its final DS token. Do not keep intermediate aliases.
|
|
||||||
|
|
||||||
Instead of `color: var(--button-foreground-hover);` write
|
|
||||||
`color: var(--color-accent-primary);`.
|
|
||||||
|
|
||||||
## Spacing
|
|
||||||
|
|
||||||
Replace deprecated spacing variables using
|
|
||||||
`frontend/src/app/main/ui/ds/spacing.scss` for `margin`, `padding`, `gap` and
|
|
||||||
`inset`.
|
|
||||||
|
|
||||||
If no exact spacing token exists, use `px2rem()`. Do not approximate with the
|
|
||||||
closest token.
|
|
||||||
|
|
||||||
## Sizes
|
|
||||||
|
|
||||||
Replace deprecated size variables using
|
|
||||||
`frontend/src/app/main/ui/ds/_sizes.scss` for `inline-size`, `block-size`,
|
|
||||||
`width`, `height` and min/max sizes.
|
|
||||||
|
|
||||||
If no exact size exists, use `px2rem()` from
|
|
||||||
`frontend/src/app/main/ui/ds/_utils.scss`. Do not leave hardcoded pixel values
|
|
||||||
unless absolutely necessary.
|
|
||||||
|
|
||||||
## Typography
|
|
||||||
|
|
||||||
When any of `font-size`, `font-weight`, `line-height`, `font-family` or
|
|
||||||
`letter-spacing` appear, check whether together they match one of the
|
|
||||||
definitions in `frontend/src/app/main/ui/ds/typography.scss`.
|
|
||||||
|
|
||||||
If they match, replace all the individual declarations with:
|
|
||||||
|
|
||||||
```scss
|
|
||||||
@include use-typography("<typography-name>");
|
|
||||||
```
|
|
||||||
|
|
||||||
If there is no exact match, use the **closest DS typography** (same role:
|
|
||||||
a panel title in uppercase → `headline-small`, body text → `body-*`),
|
|
||||||
even when more than one detail differs. Show the change in the visual
|
|
||||||
check and state it in the report and in the commit body (e.g. "11px
|
|
||||||
regular → 12px medium").
|
|
||||||
|
|
||||||
Keep the original declarations only when no DS typography fits the role
|
|
||||||
(e.g. a 32px display number in a 14px-only area); then leave a TODO
|
|
||||||
naming the closest one and ask the user.
|
|
||||||
|
|
||||||
## Borders
|
|
||||||
|
|
||||||
Replace deprecated border radius and border width values using
|
|
||||||
`frontend/src/app/main/ui/ds/_borders.scss`. A raw width close to a token
|
|
||||||
(`1.75px`) takes the token (`$b-2`); say so in the report. If nothing is
|
|
||||||
close, use `px2rem()`.
|
|
||||||
|
|
||||||
## Logical properties
|
|
||||||
|
|
||||||
Prefer logical properties whenever they preserve the existing behavior:
|
|
||||||
|
|
||||||
- `width` → `inline-size`
|
|
||||||
- `height` → `block-size`
|
|
||||||
- `max-width` → `max-inline-size`
|
|
||||||
- `margin-left` → `margin-inline-start`
|
|
||||||
- `padding-right` → `padding-inline-end`
|
|
||||||
- `top` → `inset-block-start`
|
|
||||||
- `left` → `inset-inline-start`
|
|
||||||
|
|
||||||
## Z-index
|
|
||||||
|
|
||||||
Never leave magic z-index values. Use tokens from
|
|
||||||
`frontend/src/app/main/ui/ds/z-index.scss`, and pick the token **by what
|
|
||||||
the element is, not by its number**: a dropdown, menu or select →
|
|
||||||
`--z-index-dropdown`; a toolbar, palette, sidebar or panel →
|
|
||||||
`--z-index-panels`; and so on (read the comments in that file). A raw `4`
|
|
||||||
on a dropdown becomes `var(--z-index-dropdown)` even though the number
|
|
||||||
changes. Check the visual check for overlap.
|
|
||||||
|
|
||||||
Leave a TODO only when the element fits no token's purpose.
|
|
||||||
|
|
||||||
Legacy z-index values between sibling elements that do not overlap can be
|
|
||||||
removed, but only when the **Visual check** shows no pixel change in every
|
|
||||||
state that matters (open menus, collapsed, hidden). Say so in the report.
|
|
||||||
|
|
||||||
## Duplicated properties
|
|
||||||
|
|
||||||
Within the same selector, avoid duplicated properties, duplicated CSS
|
|
||||||
variables and duplicated declarations introduced by the refactor. Allow
|
|
||||||
duplicates only when they are intentional browser fallbacks.
|
|
||||||
|
|
||||||
## Unused classes
|
|
||||||
|
|
||||||
Open the `.cljs` file with the same name as the SCSS file and check whether
|
|
||||||
every CSS class is referenced (as `(stl/css :class-name)`,
|
|
||||||
`(stl/css-case :class-name ...)` or similar).
|
|
||||||
|
|
||||||
CSS modules prefix every class with its module
|
|
||||||
(`main_ui_<path>__<class>`), so only the `.cljs` of the same module can use
|
|
||||||
it. Before calling a selector unused, also search the whole `frontend/src`
|
|
||||||
for the class name as a string, a `:global` use, or a dynamic class built
|
|
||||||
at runtime.
|
|
||||||
|
|
||||||
If a selector is unused after that check:
|
|
||||||
|
|
||||||
- delete it (do not leave it commented out, do not add a TODO);
|
|
||||||
- list every deleted selector in the report and in the commit body.
|
|
||||||
|
|
||||||
A rule that matches nothing because the markup changed (e.g. `.icon svg`
|
|
||||||
when the DS icon is the `svg` itself) is deleted the same way. Before
|
|
||||||
deleting a descendant rule, check every element it matched, not only
|
|
||||||
the one you changed: `.placeholder .icon svg` can also color a second
|
|
||||||
icon, which then has no color (in the devenv it shows in the debug
|
|
||||||
yellow).
|
|
||||||
|
|
||||||
Every DS icon needs a color of its own (`color` on its class or on a
|
|
||||||
parent with a defined color): an icon that inherits nothing shows in
|
|
||||||
the debug yellow in the devenv.
|
|
||||||
|
|
||||||
Classes used by markup injected from outside (e.g. an external widget)
|
|
||||||
stay, with a comment saying so.
|
|
||||||
|
|
||||||
Also check the other direction: every class the `.cljs` passes to
|
|
||||||
`stl/css` / `stl/css-case` must exist in the `.scss`. A missing one silently
|
|
||||||
does nothing; report it.
|
|
||||||
|
|
||||||
## Sass and CSS pitfalls
|
|
||||||
|
|
||||||
- Sass does not evaluate functions inside CSS custom properties. Write
|
|
||||||
`--button-height: calc(var(--height) - #{px2rem(16)});`, not
|
|
||||||
`... - px2rem(16))`, or the browser gets the literal `px2rem(16)`.
|
|
||||||
- A `/` between Sass size variables is read as a division
|
|
||||||
(`grid-template: $sz-32 / auto` breaks). Split it into
|
|
||||||
`grid-template-rows` / `grid-template-columns` (or areas).
|
|
||||||
- A CSS-wide keyword cannot be part of a shorthand: `inset-inline: unset 0`
|
|
||||||
is invalid and stylelint does not catch it. Use the longhands.
|
|
||||||
- After editing, check the compiled CSS for any of these
|
|
||||||
(`frontend/resources/public/css/main.css`, search the module class).
|
|
||||||
|
|
||||||
## Uncertainty
|
|
||||||
|
|
||||||
If any transformation is uncertain, never guess. Leave a TODO in the code
|
|
||||||
and explain why in the report.
|
|
||||||
|
|
||||||
## Self-review
|
|
||||||
|
|
||||||
Before the report, read the final diff as the reviewer would and search
|
|
||||||
the file for these; each hit must be fixed or have a real reason:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
rg -n 'svg \{|^\s*(h[1-6]|p|hr|input|button|span|a)\b[^-_]|[a-z]\.[a-z-]+ \{' <file>
|
|
||||||
rg -n 'stroke-width|color: transparent|fill: none|z-index: [0-9]|TODO' <file>
|
|
||||||
rg -n '[0-9]\.[0-9]+px|[^(]\b[0-9]+px' <file>
|
|
||||||
```
|
|
||||||
|
|
||||||
- an element selector → add a class (see **Element selectors**);
|
|
||||||
- a property that does not apply or is a default → drop it (see
|
|
||||||
**Placeholders**);
|
|
||||||
- a raw z-index → token by purpose;
|
|
||||||
- a typography TODO → closest DS typography;
|
|
||||||
- a raw px value → token or `px2rem()`.
|
|
||||||
|
|
||||||
## Verification
|
|
||||||
|
|
||||||
Run the project scripts from `frontend/` inside the devenv container
|
|
||||||
(`penpot-devenv-ws<N>-main` for workspace `wsN`; never run `pnpm` on the
|
|
||||||
host), writing output to a file first (never pipe lint output into
|
|
||||||
filters):
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docker exec -u penpot -w /home/penpot/penpot/frontend penpot-devenv-ws<N>-main \
|
|
||||||
bash -lc 'pnpm run lint:scss; echo LINT=$?; pnpm run check-fmt:scss; echo FMT=$?' \
|
|
||||||
> <scratch>/scss-check.txt 2>&1
|
|
||||||
```
|
|
||||||
|
|
||||||
If the format check fails, run `pnpm run fmt:scss` and check again.
|
|
||||||
|
|
||||||
**Dependency check.** A grep for `refactor/` is not enough. For the final
|
|
||||||
file, check:
|
|
||||||
|
|
||||||
- `@use` / `@import` sources: all must be `ds/...` (or the sidebar common
|
|
||||||
module, which is DS-only);
|
|
||||||
- every `@include`, `@extend` and `%placeholder`;
|
|
||||||
- every `$variable`: which imported module defines it (`$br-*` exists in both
|
|
||||||
`ds/_borders.scss` and `refactor/borders.scss`; the `@use` decides);
|
|
||||||
- every `var(--token)`: defined in the DS (colors, spacing, sizes) or only as a
|
|
||||||
`refactor/` alias.
|
|
||||||
|
|
||||||
Report any failure as it is; do not hide it.
|
|
||||||
|
|
||||||
## Visual check
|
|
||||||
|
|
||||||
"No visual change" must be shown, not assumed. When a visual spec covers the
|
|
||||||
component (`frontend/playwright/ui/visual-specs/`, project `ds`), or one can be
|
|
||||||
added:
|
|
||||||
|
|
||||||
1. Capture before (base branch styles) and after with
|
|
||||||
`pnpm exec playwright test --project=ds -g "<test>" --update-snapshots=all`
|
|
||||||
in the container. `=all` is required: plain `--update-snapshots` only
|
|
||||||
rewrites past the 0.5% tolerance, so a real diff can look like 0.
|
|
||||||
2. Before each capture, wait until the compiled CSS/JS shows the expected
|
|
||||||
version (search `main.css` for the changed rule).
|
|
||||||
3. Compare with zero tolerance. If pixels change, build the before | after |
|
|
||||||
diff (red) image and explain every change in the report.
|
|
||||||
|
|
||||||
## Output
|
|
||||||
|
|
||||||
1. The diff of the file (`git diff -- <file>`).
|
|
||||||
2. A summary with:
|
|
||||||
- removed imports;
|
|
||||||
- removed deprecated mixins;
|
|
||||||
- removed placeholders;
|
|
||||||
- typography replacements;
|
|
||||||
- design-token aliases removed;
|
|
||||||
- logical properties introduced;
|
|
||||||
- deleted unused selectors;
|
|
||||||
- TODOs requiring manual review, each with its reason;
|
|
||||||
- remaining `refactor/` dependencies (partial refactor) and where they are
|
|
||||||
used;
|
|
||||||
- classes missing from the `.scss`, if any.
|
|
||||||
3. The verification results, including the visual check (pixel count, or why
|
|
||||||
it was not possible).
|
|
||||||
@ -1,7 +1,6 @@
|
|||||||
---
|
---
|
||||||
name: review-code
|
name: review-code
|
||||||
description: Code review flow — review a diff, PR, or code change, delegating the review to a subagent that follows the code-review-criteria skill. Use it when the user asks to review code or a PR, in any phrasing.
|
description: Code review flow — review a diff, PR, or code change, delegating the review to a subagent that follows the code-review-criteria skill. Use it when the user asks to review code or a PR, in any phrasing.
|
||||||
slash: true
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# Review Code
|
# Review Code
|
||||||
@ -22,20 +21,8 @@ Act as a senior software engineer and perform a thorough code review.
|
|||||||
diff, a commit range, a branch, a PR (number or URL), or specific files. If
|
diff, a commit range, a branch, a PR (number or URL), or specific files. If
|
||||||
the target is ambiguous, ask before reviewing.
|
the target is ambiguous, ask before reviewing.
|
||||||
2. Delegate the review to the `general` subagent (via the task tool), unless the
|
2. Delegate the review to the `general` subagent (via the task tool), unless the
|
||||||
user specifies another agent. The prompt carries the **`code-review-criteria`**
|
user specifies another agent. Include in the prompt the
|
||||||
skill name, all user context, and these requirements:
|
**`code-review-criteria`** skill name and all user context.
|
||||||
- Load `code-review-criteria` and follow its process and output format.
|
|
||||||
- Read `AGENTS.md` (if present) and the module memories it points to for the
|
|
||||||
code being reviewed.
|
|
||||||
- Return the COMPLETE review, verbatim, exactly as the skill instructs it to
|
|
||||||
be produced. Never summarize it.
|
|
||||||
- Read-only: modify no file and create no commit. Reviewing never writes.
|
|
||||||
- Do not invent problems: every finding must be real and actionable.
|
|
||||||
- Missing tests are an issue, not a suggestion — a severity-tagged finding.
|
|
||||||
- Prioritize by impact: one structural issue outweighs ten nits, and "this
|
|
||||||
could be better" is not a finding — say why and how.
|
|
||||||
- Skip generated files, lockfile-only changes and unrelated modifications
|
|
||||||
unless they introduce security risks.
|
|
||||||
3. When the subagent returns, output the review to the user verbatim. Do not
|
3. When the subagent returns, output the review to the user verbatim. Do not
|
||||||
summarize it and do not act on its findings.
|
summarize it and do not act on its findings.
|
||||||
4. Right after the review, suggest how to proceed based on the findings. These
|
4. Right after the review, suggest how to proceed based on the findings. These
|
||||||
@ -56,24 +43,31 @@ the findings. Once the user explicitly states the review is done (or moves on to
|
|||||||
a different task), this rule no longer applies and you act as a normal build
|
a different task), this rule no longer applies and you act as a normal build
|
||||||
agent again.
|
agent again.
|
||||||
|
|
||||||
## Posting as a GitHub comment
|
## Instructions for the subagent
|
||||||
|
|
||||||
Post only when the user asks for it explicitly; otherwise keep the review in the
|
1. Load the **`code-review-criteria`** skill and follow its process and output
|
||||||
session. PR comments go in English, with this minimal structure:
|
format.
|
||||||
|
2. Read `AGENTS.md` (if present) and follow its instructions for finding and
|
||||||
|
reading all related testing documentation from memories before reviewing.
|
||||||
|
3. Return in your final message the COMPLETE review, verbatim, exactly as the
|
||||||
|
skill instructs it to be produced. Do not summarize it — include the full
|
||||||
|
structured review.
|
||||||
|
|
||||||
1. **Intro line** — one sentence: what was found and the overall tone. No
|
### Strong rules for the subagent
|
||||||
greetings.
|
|
||||||
2. **`## Main issues`** — one subsection per required finding: what breaks, the
|
|
||||||
failure circumstances, and the minimal fix (code snippet when useful). A
|
|
||||||
missing test belongs here, not under minor suggestions.
|
|
||||||
3. **`## Minor suggestions`** — one bullet per minor finding: file/function,
|
|
||||||
problem, concrete remedy.
|
|
||||||
4. **Closing** — optional and short. Cut it when it adds nothing.
|
|
||||||
|
|
||||||
Reference every finding by its ID (`R1`, `M1`, `S1`, … from
|
1. Do not invent problems. Every finding must be real and actionable.
|
||||||
`code-review-criteria`) so follow-up replies can point to findings
|
2. Read-only: do not modify any file and do not create a commit — reviewing
|
||||||
unambiguously.
|
never writes.
|
||||||
|
3. Be specific and constructive. "This could be better" is not helpful — explain
|
||||||
|
why and how.
|
||||||
|
4. Prioritize by impact. One structural issue outweighs ten nits.
|
||||||
|
5. Missing tests are an issue, not a suggestion. Report as a severity-tagged
|
||||||
|
finding — never as a recommendation.
|
||||||
|
6. Skip generated files, lockfile-only changes, and unrelated modifications
|
||||||
|
unless they introduce security risks.
|
||||||
|
|
||||||
## User input, overrides and additional context
|
## User context
|
||||||
|
|
||||||
$ARGUMENTS
|
Extra context in the user's invocation (the message that triggered this skill)
|
||||||
|
plays the role command arguments play elsewhere: for example, a PR number or
|
||||||
|
URL, a commit range, specific files, or a different agent to run the review.
|
||||||
|
|||||||
@ -1,46 +1,71 @@
|
|||||||
---
|
---
|
||||||
name: review-plan
|
name: review-plan
|
||||||
description: Plan review flow — evaluate an implementation plan before it is executed, delegating the review to a subagent that follows the plan-review-criteria skill. Use it when the user asks to review a plan, in any phrasing.
|
description: Plan review flow — evaluate an implementation plan before it is executed, delegating the review to a subagent that follows the plan-review-criteria skill. Use it when the user asks to review a plan, in any phrasing.
|
||||||
slash: true
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# Review Plan
|
# Review Plan
|
||||||
|
|
||||||
Act as a senior software engineer and perform a thorough review of an implementation plan.
|
Act as a senior software engineer and perform a thorough review of an
|
||||||
|
implementation plan.
|
||||||
|
|
||||||
## When to use
|
## When to use
|
||||||
|
|
||||||
- The user asks to review a plan, in any phrasing: "review this plan", "does this plan look right?", "second opinion on the plan" — or runs `/review-plan`.
|
- The user asks to review a plan, in any phrasing: "review this plan",
|
||||||
- A plan was just produced (typically by `/make-a-plan`) and the user wants it evaluated before executing it.
|
"does this plan look right?", "second opinion on the plan" — or runs
|
||||||
|
`/review-plan`.
|
||||||
|
- A plan was just produced (typically by `/make-a-plan`) and the user
|
||||||
|
wants it evaluated before executing it.
|
||||||
|
|
||||||
## Instructions
|
## Instructions
|
||||||
|
|
||||||
1. **Determine the plan under review** from the session context (for example, a plan just produced by `/make-a-plan`) or from a plan file path given by the user (typically under `.agents/plans/`). If a file path is given, read the file first so the complete plan is in context.
|
1. **Determine the plan under review** from the session context (for example, a
|
||||||
2. Delegate the review to the `general` subagent (via the task tool), unless the user specifies another agent. Include in the prompt the **`plan-review-criteria`** skill name and all user context.
|
plan just produced by `/make-a-plan`) or from a plan file path given by the
|
||||||
3. When the subagent returns, output the review to the user verbatim. Do not summarize it and do not act on its findings.
|
user (typically under `.agents/plans/`). If a file path is given, read the
|
||||||
4. Right after the review, suggest the next step based on the verdict. These are suggestions — the user decides, and any instruction overrides them:
|
file first so the complete plan is in context.
|
||||||
|
2. Delegate the review to the `general` subagent (via the task tool), unless the
|
||||||
|
user specifies another agent. Include in the prompt the
|
||||||
|
**`plan-review-criteria`** skill name and all user context.
|
||||||
|
3. When the subagent returns, output the review to the user verbatim. Do not
|
||||||
|
summarize it and do not act on its findings.
|
||||||
|
4. Right after the review, suggest the next step based on the verdict. These
|
||||||
|
are suggestions — the user decides, and any instruction overrides them:
|
||||||
- **Approve** → suggest `/implement-plan` to execute it.
|
- **Approve** → suggest `/implement-plan` to execute it.
|
||||||
- **Request changes** → suggest `/make-a-plan` to make a plan to address the findings.
|
- **Request changes** → suggest `/make-a-plan` to make a plan to address the
|
||||||
|
findings.
|
||||||
|
|
||||||
### Hard rule — read-only while reviewing
|
### Hard rule — read-only while reviewing
|
||||||
|
|
||||||
This flow is read-only **for the duration of the review**: from the moment it starts until the user considers the review finished (including any feedback, questions, or clarifications about it). During that period, never fix, implement, edit files or create commits — not even "obvious" fixes derived from the findings. Once the user explicitly states the review is done (or moves on to a different task), this rule no longer applies and you act as a normal build agent again.
|
This flow is read-only **for the duration of the review**: from the moment it
|
||||||
|
starts until the user considers the review finished (including any feedback,
|
||||||
|
questions, or clarifications about it). During that period, never fix,
|
||||||
|
implement, edit files or create commits — not even "obvious" fixes derived from
|
||||||
|
the findings. Once the user explicitly states the review is done (or moves on to
|
||||||
|
a different task), this rule no longer applies and you act as a normal build
|
||||||
|
agent again.
|
||||||
|
|
||||||
## Instructions for the subagent
|
## Instructions for the subagent
|
||||||
|
|
||||||
1. Load the **`plan-review-criteria`** skill and follow its process and output format.
|
1. Load the **`plan-review-criteria`** skill and follow its process and output
|
||||||
2. Read `AGENTS.md` (if present) and follow its instructions for finding and reading all related documentation and testing memories before reviewing.
|
format.
|
||||||
3. Cada hallazgo debe tener un identificador único y estable, con el formato `F1`, `F2`, etc. El identificador debe aparecer en el título del hallazgo y no debe reutilizarse dentro de la misma revisión. Esto permite que el usuario pueda responder sobre un hallazgo concreto.
|
2. Read `AGENTS.md` (if present) and follow its instructions for finding and
|
||||||
4. Return in your final message the COMPLETE review, verbatim, exactly as the skill instructs it to be produced. Do not summarize it — include the full structured review.
|
reading all related documentation and testing memories before reviewing.
|
||||||
|
3. Return in your final message the COMPLETE review, verbatim, exactly as the
|
||||||
|
skill instructs it to be produced. Do not summarize it — include the full
|
||||||
|
structured review.
|
||||||
|
|
||||||
### Strong rules for the subagent
|
### Strong rules for the subagent
|
||||||
|
|
||||||
1. Do not invent problems. Every finding must be real and actionable.
|
1. Do not invent problems. Every finding must be real and actionable.
|
||||||
2. Read-only: do not modify any file and do not create a commit — reviewing never writes.
|
2. Read-only: do not modify any file and do not create a commit — reviewing
|
||||||
3. Be specific and constructive. "This could be better" is not helpful — explain why and how.
|
never writes.
|
||||||
|
3. Be specific and constructive. "This could be better" is not helpful — explain
|
||||||
|
why and how.
|
||||||
4. Prioritize by impact. One structural issue outweighs ten nits.
|
4. Prioritize by impact. One structural issue outweighs ten nits.
|
||||||
5. Judge the plan as the implementer would: every task executable without guessing, ordering follows the dependency graph, risks named.
|
5. Judge the plan as the implementer would: every task executable without
|
||||||
|
guessing, ordering follows the dependency graph, risks named.
|
||||||
|
|
||||||
## User input, overrides and additional context
|
## User context
|
||||||
|
|
||||||
$ARGUMENTS
|
Extra context in the user's invocation (the message that triggered this skill)
|
||||||
|
plays the role command arguments play elsewhere: for example, a plan file path
|
||||||
|
to review, or a different agent to run the review.
|
||||||
|
|||||||
@ -1,38 +0,0 @@
|
|||||||
---
|
|
||||||
name: rewrite-commit-message
|
|
||||||
description: Rewrite HEAD's commit message (typically a squash that concatenated old messages) into one proper title plus description following Penpot commit conventions, preserving every distinct AI-assisted-by attribution. Amends in place, never pushes.
|
|
||||||
slash: true
|
|
||||||
---
|
|
||||||
|
|
||||||
# Skill: rewrite-commit-message
|
|
||||||
|
|
||||||
HEAD's message is a concatenation of older messages (typical after a
|
|
||||||
squash): several subjects, bodies and repeated trailers stacked in a
|
|
||||||
single message. Replace it with one proper message and amend HEAD.
|
|
||||||
|
|
||||||
## When to Use
|
|
||||||
|
|
||||||
- The user asks to rewrite, retitle or fix the last commit message
|
|
||||||
- After a squash left HEAD with stacked messages and trailer repeats
|
|
||||||
|
|
||||||
## Steps
|
|
||||||
|
|
||||||
1. Read the full current message (`git log -1 --format=%B HEAD`) and
|
|
||||||
the branch diff (`git diff develop..HEAD --stat`, plus the code
|
|
||||||
when the summary needs grounding).
|
|
||||||
2. Collect every DISTINCT `AI-assisted-by:` value in the current
|
|
||||||
message. All of them survive into the new message, one line each:
|
|
||||||
attributions are never dropped in a rewrite, even when most of
|
|
||||||
the old text is.
|
|
||||||
3. Draft one message per `mem:workflow/creating-commits` (read it
|
|
||||||
end-to-end first): single subject (imperative, capitalized, no
|
|
||||||
trailing period, ≤80 chars), blank line, body wrapped at ≤80
|
|
||||||
chars covering the whole diff (not the old per-commit slices),
|
|
||||||
then the preserved trailers.
|
|
||||||
4. Write it to a file (a heredoc keeps real newlines; inline `-m`
|
|
||||||
with `\n` does not) and `git commit --amend -F <file>`.
|
|
||||||
5. Run `./scripts/check-commit` and require exit code 0. A non-zero
|
|
||||||
exit is a hard blocker: fix the message, amend again.
|
|
||||||
6. Report the new hash. Never push: the user pushes from their own
|
|
||||||
shell. Warn when local history was rewritten over pushed commits
|
|
||||||
(the push will need force — their call, not yours).
|
|
||||||
@ -5,57 +5,821 @@ description: Update the project CHANGES.md with issues from a given GitHub miles
|
|||||||
|
|
||||||
# Skill: update-changelog
|
# Skill: update-changelog
|
||||||
|
|
||||||
Entry point for all changelog work. All rules — sections, exclusions, entry
|
Update `CHANGES.md` with entries for all issues and PRs in a given GitHub
|
||||||
format, the `:boom:`, `:rocket:` and `:lock:` contracts, pre-flight, and the
|
milestone. Each entry references the user-facing issue (not the PR) as the
|
||||||
anomaly report — live in `mem:workflow/updating-changelog` (file:
|
primary link, with the fix PR inline on the same line.
|
||||||
`.serena/memories/workflow/updating-changelog.md`). This skill routes to that
|
|
||||||
memory; it does not repeat it.
|
|
||||||
|
|
||||||
## When to Use
|
## When to Use
|
||||||
|
|
||||||
- Before a new release, to populate the changelog with all fixed issues
|
- Before a new release, to populate the changelog with all fixed issues
|
||||||
- When new issues are added to an existing milestone and the changelog needs
|
- When new issues are added to an existing milestone and the changelog needs
|
||||||
to be refreshed
|
to be refreshed
|
||||||
- To audit or repair an existing version section
|
- To ensure every entry follows the correct format for the changelog
|
||||||
|
|
||||||
## Prerequisites
|
## Prerequisites
|
||||||
|
|
||||||
- `gh` CLI authenticated (`gh auth status`)
|
- `gh` CLI authenticated (`gh auth status`)
|
||||||
- Python 3.8+
|
- Python 3.8+
|
||||||
- `scripts/gh.py` for GitHub queries, `scripts/changelog.py` for changelog
|
- `scripts/gh.py` helper script available
|
||||||
checks (run `python3 scripts/changelog.py <command> --help` for usage)
|
|
||||||
|
|
||||||
## What to Read
|
## Workflow
|
||||||
|
|
||||||
1. `mem:workflow/updating-changelog` — the canonical flow: sources of truth,
|
### 1. Determine the target version
|
||||||
layout, entry rules, the `## Breaking change note` comment contract, the
|
|
||||||
advisory (`:lock:`) contract, pre-flight checks, anomaly types, and pitfalls.
|
|
||||||
2. `mem:scripts/changelog` — `boom`, `security`, `check-merged`, `cross-ref`,
|
|
||||||
`dev-only-bugs`, `report`.
|
|
||||||
3. `mem:scripts/gh` — `issues`, `issue`, `comments`, `prs`, `advisories`.
|
|
||||||
|
|
||||||
## Commands
|
The version is typically a semver string like `2.15.3`. Confirm with the user
|
||||||
|
if not specified.
|
||||||
|
|
||||||
|
### 2. Fetch all issues in the milestone
|
||||||
|
|
||||||
|
Use the helper script. It uses GraphQL for efficient single-pass fetching
|
||||||
|
(closing PRs are included in the same query — no N+1):
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python3 scripts/gh.py issues "<VERSION>" --exclude "no changelog"
|
# All closed issues (default)
|
||||||
python3 scripts/gh.py issues "<VERSION>" --label "release blocker" --exclude "no changelog" # closed non-bug residue to review by hand
|
python3 scripts/gh.py issues "2.16.0"
|
||||||
python3 scripts/gh.py advisories --patched-version "<VERSION>"
|
|
||||||
python3 scripts/changelog.py dev-only-bugs "<VERSION>" # list bugs that never reached production
|
# Include open issues too
|
||||||
python3 scripts/changelog.py dev-only-bugs "<VERSION>" --apply # label the sure ones `no changelog`
|
python3 scripts/gh.py issues "2.16.0" --state all
|
||||||
python3 scripts/changelog.py check-merged <PR numbers...>
|
|
||||||
python3 scripts/changelog.py boom "<VERSION>" # print the :boom: section
|
# Exclude entries that should not go in the changelog
|
||||||
python3 scripts/changelog.py boom "<VERSION>" --write # replace it in CHANGES.md
|
python3 scripts/gh.py issues "2.16.0" --exclude "release blocker,no changelog"
|
||||||
python3 scripts/changelog.py security "<VERSION>" # print the :lock: section
|
|
||||||
python3 scripts/changelog.py security "<VERSION>" --write # replace it in CHANGES.md
|
|
||||||
python3 scripts/changelog.py cross-ref "<VERSION>"
|
|
||||||
python3 scripts/changelog.py report "<VERSION>"
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Hard Rules
|
**Exclusion rules (issue-level):**
|
||||||
|
- `no changelog` label — Chore/refactor work that doesn't need a changelog entry
|
||||||
|
- `release blocker` label — Blocked issues not yet ready for changelog
|
||||||
|
- `Task` issue type — Internal chores are not user-facing; automatically excluded by `gh.py`. Use `--include-tasks` to override.
|
||||||
|
- **Rejected project status** — Issues with a "Rejected" status in the "Main" project board are automatically excluded by `gh.py`. This project-level status (independent of the GitHub issue `state`) indicates the issue was rejected from the release. Use `--include-rejected` to override.
|
||||||
|
|
||||||
- `CHANGES.md` is edited only on explicit request or during the release process.
|
**Exclusion rules (PR-level):**
|
||||||
- Only closed, non-excluded milestone issues are included; the issue is the changelog unit and the PR is inline.
|
In addition to issue-level exclusions, PRs with these labels should be
|
||||||
- `:boom:` and `:rocket:` are label-generated and `:lock:` is advisory-generated: never hand-edit them; edit the label, the `## Breaking change note` comment or the advisory and re-run the command.
|
excluded regardless of their linked issue's labels:
|
||||||
- Verify PR merge status before writing; only merged PRs are referenced.
|
- `release blocker` — PR is part of a pending release blocker batch
|
||||||
- Bugs that never reached production stay out: `dev-only-bugs --apply` labels the sure ones; candidates are the user's call, never label them yourself.
|
- `no issue required` — Trivial fix not tracked as an issue
|
||||||
- Fill `patched_versions` in the advisories before running `security`; unpublished advisories are valid.
|
|
||||||
|
The script outputs JSON with each entry containing `number`, `title`, `state`,
|
||||||
|
`issue_type`, `labels`, `closing_prs` (the PRs that fix each issue), and
|
||||||
|
`project_status` (the "Main" project board status, e.g. "Done", "Rejected",
|
||||||
|
or `null` if not tracked in a project).
|
||||||
|
|
||||||
|
### 3. Identify missing entries (optional)
|
||||||
|
|
||||||
|
If updating from an existing `CHANGES.md`, find issues in the milestone that
|
||||||
|
are NOT yet referenced in the changelog:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 scripts/gh.py issues "2.16.0" --exclude "release blocker,no changelog" --compare CHANGES.md
|
||||||
|
```
|
||||||
|
|
||||||
|
This returns a filtered JSON array with only the missing issues.
|
||||||
|
|
||||||
|
> **Note:** The `--compare` flag checks **issues** only (via issue number
|
||||||
|
> references in the changelog). To find merged **PRs** not yet referenced,
|
||||||
|
> use the milestone PR cross-reference described in step 10 below.
|
||||||
|
|
||||||
|
### 4. Fetch additional PR details when needed
|
||||||
|
|
||||||
|
When you need more context for specific PRs (e.g. to find the PR author for
|
||||||
|
community contribution attribution, or to read the PR body for
|
||||||
|
"Fixes/Closes #NNN" patterns):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# One or more PR numbers
|
||||||
|
python3 scripts/gh.py prs 9179 9204 9311
|
||||||
|
|
||||||
|
# From a file
|
||||||
|
python3 scripts/gh.py prs --file prs.txt
|
||||||
|
|
||||||
|
# From stdin
|
||||||
|
cat prs.txt | python3 scripts/gh.py prs --stdin
|
||||||
|
```
|
||||||
|
|
||||||
|
The `prs` command also supports listing all PRs in a milestone in one call:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# All merged PRs in a milestone (default)
|
||||||
|
python3 scripts/gh.py prs --milestone "2.16.0"
|
||||||
|
|
||||||
|
# All states (merged, open, closed)
|
||||||
|
python3 scripts/gh.py prs --milestone "2.16.0" --state all
|
||||||
|
```
|
||||||
|
|
||||||
|
The `prs` command returns JSON with `number`, `title`, `body`, `state`,
|
||||||
|
`merged_at`, `author`, `labels`, and `closing_issues`. PRs are fetched in
|
||||||
|
batches of 50 via GraphQL to stay within API limits (milestone mode uses
|
||||||
|
paginated GraphQL on the milestone's `pullRequests` connection).
|
||||||
|
|
||||||
|
You can also list all PRs in a milestone in a single call:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# All merged PRs in a milestone (default)
|
||||||
|
python3 scripts/gh.py prs --milestone "2.16.0"
|
||||||
|
|
||||||
|
# All states (merged, open, closed)
|
||||||
|
python3 scripts/gh.py prs --milestone "2.16.0" --state all
|
||||||
|
|
||||||
|
# Open PRs only
|
||||||
|
python3 scripts/gh.py prs --milestone "2.16.0" --state open
|
||||||
|
```
|
||||||
|
|
||||||
|
The milestone path uses paginated GraphQL on the milestone's `pullRequests`
|
||||||
|
connection (100 per page), avoiding one-by-one fetches.
|
||||||
|
|
||||||
|
### 5. Categorize entries — strictly by issue type, never by labels or emoji
|
||||||
|
|
||||||
|
Use the **Issue Type** field (GitHub's native issue type, exposed as
|
||||||
|
`issue_type` in the `gh.py` JSON output) to determine which section an entry
|
||||||
|
belongs to.
|
||||||
|
|
||||||
|
> **⚠️ CRITICAL: Never use labels or title emoji prefixes for categorization.**
|
||||||
|
> Labels like `bug` and `enhancement`, as well as title prefixes like `:bug:`
|
||||||
|
> and `:sparkles:`, are frequently inaccurate, missing, or contradictory to the
|
||||||
|
> actual issue type. The `issue_type` field from `gh.py` is the single source
|
||||||
|
> of truth.
|
||||||
|
|
||||||
|
| `issue_type` value | Changelog section |
|
||||||
|
|--------------------|-------------------|
|
||||||
|
| `Bug` | `### :bug: Bugs fixed` |
|
||||||
|
| `Feature` or `Enhancement` | `### :sparkles: New features & Enhancements` |
|
||||||
|
| `Task` | **Exclude** — internal chores are not user-facing |
|
||||||
|
| `null` (not set) | Check labels as a fallback: `bug` label → bugs, otherwise enhancements |
|
||||||
|
|
||||||
|
The `gh.py` issues command already includes `issue_type` in every entry's
|
||||||
|
output. **No separate GraphQL query is needed.**
|
||||||
|
|
||||||
|
**Preserve highlighted entries:** If an entry is already featured in
|
||||||
|
`### :rocket: Epics and highlights`, keep it in that section when refreshing a
|
||||||
|
changelog version. Do not remove a highlighted entry just because issue type
|
||||||
|
categorization would otherwise place it under `### :sparkles: New features &
|
||||||
|
Enhancements`.
|
||||||
|
|
||||||
|
**Community contribution attribution:** If the issue or its fix PR has the
|
||||||
|
`community contribution` label, add an attribution `(by @<github_username>)`
|
||||||
|
on the changelog entry line, **before** the GitHub issue/PR references.
|
||||||
|
|
||||||
|
The attribution should reference the **PR author**, not the issue author.
|
||||||
|
The `prs` subcommand includes the `author` field — use that:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 scripts/gh.py prs <PR_NUMBER> | python3 -c "import sys,json; print(json.load(sys.stdin)[0]['author'])"
|
||||||
|
```
|
||||||
|
|
||||||
|
Placement in the entry line:
|
||||||
|
```markdown
|
||||||
|
- Fix description of the bug (by @username) [#<ISSUE>](...) (PR: [#<PR>](...))
|
||||||
|
```
|
||||||
|
|
||||||
|
**Only closed issues are included.** An issue must have `state: "closed"` to
|
||||||
|
appear in the changelog. Open/unresolved issues are omitted, even if they are
|
||||||
|
tracked in the milestone.
|
||||||
|
|
||||||
|
**Pairing rules:**
|
||||||
|
|
||||||
|
| Pattern | Changelog format |
|
||||||
|
|---------|-----------------|
|
||||||
|
| Closed issue + one or more PRs fix it | Primary link = issue, PR inline comma-separated |
|
||||||
|
| PR exists with no linked issue | If a corresponding closed issue exists in the same milestone, link the issue. Otherwise, skip the entry (the issue must be the changelog unit). |
|
||||||
|
| Closed issue with no fix PR in milestone | Link the issue directly, without a PR reference. |
|
||||||
|
|
||||||
|
> **False-positive associations:** A PR may incorrectly claim to close an issue
|
||||||
|
> from a different context (e.g., a very old PR referencing a modern issue, or a
|
||||||
|
> cross-project reference). If the PR title and issue title are clearly unrelated,
|
||||||
|
> or the PR was created years before the issue, treat it as a data glitch and
|
||||||
|
> skip it. PR [#3](https://github.com/penpot/penpot/pull/3) (ancient License PR
|
||||||
|
> claiming to close a plugin API issue) is a known example.
|
||||||
|
|
||||||
|
### 5a. ⚠️ Verify PR merge status before writing
|
||||||
|
|
||||||
|
A closed issue may list closing PRs that were **closed without merging**
|
||||||
|
(e.g., a community PR that was superseded by another). The changelog must
|
||||||
|
only reference **merged** PRs. Verify before writing:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Collect all PR numbers from the candidate entries and check them
|
||||||
|
python3 scripts/gh.py prs <ALL_PR_NUMBERS> | python3 -c "
|
||||||
|
import json, sys
|
||||||
|
for pr in json.load(sys.stdin):
|
||||||
|
if pr['state'] != 'MERGED':
|
||||||
|
print(f'WARNING: #{pr[\"number\"]} is {pr[\"state\"]} (not merged)')
|
||||||
|
"
|
||||||
|
```
|
||||||
|
|
||||||
|
If a closing PR is closed-unmerged, find the actual merged PR that
|
||||||
|
superseded it:
|
||||||
|
1. Check the issue's closing PRs list for other PRs (there may be multiple)
|
||||||
|
2. Look for other PRs with similar titles or descriptions referencing the same issue
|
||||||
|
3. Inspect the closed PR's conversation timeline for a pointer to the replacement
|
||||||
|
|
||||||
|
Replace the reference in the changelog entry with the correct merged PR number.
|
||||||
|
|
||||||
|
### 5b. Security advisory (GHSA) entries
|
||||||
|
|
||||||
|
Security advisories fixed in a release are documented in the changelog even
|
||||||
|
though they are **neither milestone issues nor PRs**. The GHSA ID and its
|
||||||
|
description are supplied by the user or the release notes — they never come
|
||||||
|
from the milestone fetch in step 2.
|
||||||
|
|
||||||
|
**Format** (matches the existing precedent in `CHANGES.md`, e.g. the
|
||||||
|
`create-font-variant` arbitrary file read advisory):
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
- Fix <user-facing description> (https://github.com/penpot/penpot/security/advisories/GHSA-XXXX-XXXX-XXXX)
|
||||||
|
```
|
||||||
|
|
||||||
|
Rules:
|
||||||
|
- Place the entry under `### :bug: Bugs fixed`, with **no issue or PR link** —
|
||||||
|
only the advisory URL.
|
||||||
|
- The advisory may be **draft/unpublished** at changelog time (the URL 404s
|
||||||
|
publicly). Do **not** web-fetch or verify the URL, and do **not** drop the
|
||||||
|
entry because of that. Rely on the GHSA ID provided by the user.
|
||||||
|
- Derive the description from the supplied advisory title, imperative mood and
|
||||||
|
user-facing (e.g. `Fix command injection in SVG exporter via legacy fill-color`).
|
||||||
|
- These entries are **invisible to the automation**: they are not returned by
|
||||||
|
`gh.py issues`, not matched by `--compare` (step 3), not part of the PR
|
||||||
|
cross-reference (step 10), and not scanned by the anomaly-report regexes
|
||||||
|
(step 11, which only match `issues/` and `pull/` links). Add them manually.
|
||||||
|
- During pre-flight checks (step 6a) apply only the **backport/duplicate**
|
||||||
|
check: if the same GHSA already appears in an earlier version section, remove
|
||||||
|
it from the current section. Their absence from milestone cross-references
|
||||||
|
is expected, not an anomaly.
|
||||||
|
|
||||||
|
### 6. Read the current CHANGES.md
|
||||||
|
|
||||||
|
Read the top of `CHANGES.md` to understand the existing format and find the
|
||||||
|
insertion point (newest version goes at the top, after the `# CHANGELOG`
|
||||||
|
header).
|
||||||
|
|
||||||
|
Key format rules from the existing file:
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
## <VERSION>
|
||||||
|
|
||||||
|
### :bug: Bugs fixed
|
||||||
|
|
||||||
|
- Fix description of the bug [#<ISSUE>](https://github.com/penpot/penpot/issues/<ISSUE>) (PR: [#<PR>](https://github.com/penpot/penpot/pull/<PR>))
|
||||||
|
- Fix another bug (by @contributor) [#<ISSUE>](https://github.com/penpot/penpot/issues/<ISSUE>) (PR: [#<PR>](https://github.com/penpot/penpot/pull/<PR>))
|
||||||
|
|
||||||
|
### :sparkles: New features & Enhancements
|
||||||
|
|
||||||
|
- Add new feature description [#<ISSUE>](https://github.com/penpot/penpot/issues/<ISSUE>) (PR: [#<PR>](https://github.com/penpot/penpot/pull/<PR>))
|
||||||
|
```
|
||||||
|
|
||||||
|
Format details:
|
||||||
|
- Entries start with `- ` followed by a short description in imperative mood
|
||||||
|
- Primary link is **always the issue** (user-facing artifact)
|
||||||
|
- PR references are inline on the same line: `(PR: [#<N>](<url>))`
|
||||||
|
If an issue has multiple fix PRs, they are comma-separated:
|
||||||
|
`(PR: [#<N>](<url>), [#<M>](<url>))`
|
||||||
|
- The description should describe the fix/feature from the user's perspective
|
||||||
|
- Community contributions get `(by @<username>)` **before** the issue link
|
||||||
|
- Sections are separated by a blank line between the last entry and the next
|
||||||
|
section title
|
||||||
|
- Only include a section if there are entries for it
|
||||||
|
- When an entry already exists in an earlier version section, it must be removed
|
||||||
|
from the current version to avoid duplicates
|
||||||
|
|
||||||
|
### 6a. Pre-flight checks — fix rule violations in the changelog
|
||||||
|
|
||||||
|
**The LLM must apply these checks during the workflow and fix any
|
||||||
|
violations directly in `CHANGES.md`. They are not anomalies — they are
|
||||||
|
process errors that should be corrected before writing the new section.**
|
||||||
|
|
||||||
|
The changelog is a *snapshot* of the milestone at a point in time, but
|
||||||
|
milestones and changelog entries can drift. The LLM must reconcile the
|
||||||
|
existing changelog against the current state of the milestone and the
|
||||||
|
existing changelog entries.
|
||||||
|
|
||||||
|
For each entry that already exists in `CHANGES.md` (in any version
|
||||||
|
section) or in the candidate set for the current milestone, check:
|
||||||
|
|
||||||
|
1. **Duplicate across versions.** Is the same issue already documented
|
||||||
|
in another (older) version section? If yes, this is a *backport*:
|
||||||
|
- The user-facing fix was already released. Remove the duplicate
|
||||||
|
from the current section. The earlier version is the canonical
|
||||||
|
reference.
|
||||||
|
|
||||||
|
2. **Stale milestone assignment.** Has the issue been moved out of the
|
||||||
|
current milestone since the changelog was last updated (e.g., a fix
|
||||||
|
arrived late and the issue was reassigned to a future milestone)?
|
||||||
|
- Verify the issue is still in the current milestone via
|
||||||
|
`python3 scripts/gh.py issues <MILESTONE> --state all`. If it's no
|
||||||
|
longer there, remove the entry from the current section. (If the
|
||||||
|
target section doesn't exist yet, the entry is simply dropped.)
|
||||||
|
|
||||||
|
3. **Exclusion labels newly applied.** Did the issue acquire a
|
||||||
|
`no changelog` or `release blocker` label since the changelog was
|
||||||
|
last updated? If yes, remove the entry from the current section.
|
||||||
|
|
||||||
|
4. **Issue state changed.** Is the issue still closed? Has it been
|
||||||
|
reopened, deleted, or moved to a `Rejected` project status? If yes,
|
||||||
|
remove the entry.
|
||||||
|
|
||||||
|
5. **Unmerged or removed PR references.** For every PR referenced in
|
||||||
|
the entry, is the PR still merged? Was the PR closed without
|
||||||
|
merging (superseded)? Was the PR moved to a different milestone?
|
||||||
|
If the only referenced PR is no longer merged, fix the reference
|
||||||
|
(find the actual merged fix PR) or remove the entry. A PR that is
|
||||||
|
merged in a *different* milestone is reported as an anomaly in
|
||||||
|
step 11 — do not silently remove it.
|
||||||
|
|
||||||
|
6. **Issue type changed.** Did the issue type change (e.g., from Bug to
|
||||||
|
Task)? If the new type is `Task`, the issue is internal and should
|
||||||
|
be removed.
|
||||||
|
|
||||||
|
7. **Cross-section completeness.** For every closed, non-excluded
|
||||||
|
milestone issue that is *not* referenced in any version section of
|
||||||
|
the changelog, add it to the current section (per the categorization
|
||||||
|
rules in step 5).
|
||||||
|
|
||||||
|
After these checks, the changelog should be internally consistent with
|
||||||
|
the milestone. **Do not defer these fixes to step 11 — they are
|
||||||
|
workflow errors, not anomalies.** Step 11 only reports milestone
|
||||||
|
mismatches that require human judgment about the team's release
|
||||||
|
intent.
|
||||||
|
|
||||||
|
### 7. Build the description text
|
||||||
|
|
||||||
|
Derive the description from the issue title, not the PR title. Strip leading
|
||||||
|
emoji prefixes (`:bug:`, `:sparkles:`, `:tada:`) and focus on the
|
||||||
|
user-facing behavior.
|
||||||
|
|
||||||
|
Examples:
|
||||||
|
|
||||||
|
| Issue title | Changelog description |
|
||||||
|
|-------------|----------------------|
|
||||||
|
| `Plugin API token methods fail with schema validation error on PRO` | `Fix Plugin API token methods failing with schema validation error on PRO` |
|
||||||
|
| `Comment content is not sanitized before rendering, enabling stored XSS` | `Sanitize comment content on rendering` |
|
||||||
|
| `Custom uploaded font family names are not sanitized` | `Sanitize font family names on custom uploaded fonts` |
|
||||||
|
|
||||||
|
### 8. Insert the section into CHANGES.md
|
||||||
|
|
||||||
|
Insert the new version section right after the `# CHANGELOG` header (before
|
||||||
|
the previous version entry). Use the `edit` tool with enough context to make
|
||||||
|
a unique match.
|
||||||
|
|
||||||
|
### 9. Verify
|
||||||
|
|
||||||
|
Read the top of `CHANGES.md` and confirm:
|
||||||
|
- The version header is correct
|
||||||
|
- Every entry has a GitHub link
|
||||||
|
- Entries with a fix PR have the PR sub-line
|
||||||
|
- The section ordering is correct (newest first)
|
||||||
|
- Formatting matches the surrounding entries
|
||||||
|
|
||||||
|
### 10. Cross-reference milestone PRs against the changelog
|
||||||
|
|
||||||
|
Issues can be fixed by PRs that aren't in the milestone, and merged PRs in
|
||||||
|
the milestone may not close any tracked issue. After writing, run a full
|
||||||
|
cross-reference to catch gaps:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# List all merged PRs in the milestone
|
||||||
|
python3 scripts/gh.py prs --milestone "<MILESTONE>" --state merged > /tmp/milestone-prs.json
|
||||||
|
|
||||||
|
# Extract PR numbers from the changelog section
|
||||||
|
python3 -c "
|
||||||
|
import json, re
|
||||||
|
|
||||||
|
with open('CHANGES.md') as f:
|
||||||
|
content = f.read()
|
||||||
|
|
||||||
|
# Extract the version section (adjust regex to match the actual version)
|
||||||
|
match = re.search(r'## <MILESTONE> \(Unreleased\)\n(.*?)(?:\n## |\Z)', content, re.DOTALL)
|
||||||
|
section = match.group(1)
|
||||||
|
|
||||||
|
# Collect all PR numbers referenced
|
||||||
|
changelog_prs = set()
|
||||||
|
for m in re.findall(r'\[#(\d+)\]\(https://github\.com/penpot/penpot/pull/\d+\)', section):
|
||||||
|
changelog_prs.add(int(m))
|
||||||
|
|
||||||
|
# Collect all milestone PRs (filtered)
|
||||||
|
with open('/tmp/milestone-prs.json') as f:
|
||||||
|
milestone_prs = json.load(f)
|
||||||
|
|
||||||
|
milestone_merged = {pr['number'] for pr in milestone_prs}
|
||||||
|
|
||||||
|
# PRs in milestone but not in changelog
|
||||||
|
missing = sorted(milestone_merged - changelog_prs)
|
||||||
|
print(f'Milestone merged PRs: {len(milestone_merged)}')
|
||||||
|
print(f'Changelog referenced PRs: {len(changelog_prs)}')
|
||||||
|
print(f'PRs in milestone but NOT in changelog: {len(missing)}')
|
||||||
|
for num in missing:
|
||||||
|
pr = next(p for p in milestone_prs if p['number'] == num)
|
||||||
|
print(f' #{num} {pr[\"title\"][:80]}')
|
||||||
|
"
|
||||||
|
```
|
||||||
|
|
||||||
|
For each missing PR found, decide whether it should be added to the
|
||||||
|
changelog or is legitimately excluded (check its labels).
|
||||||
|
|
||||||
|
Also verify that no closed-unmerged PRs remain in the changelog:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 scripts/gh.py prs --milestone "<MILESTONE>" --state all | python3 -c "
|
||||||
|
import json, sys
|
||||||
|
data = json.load(sys.stdin)
|
||||||
|
closed = [p for p in data if p['state'] == 'CLOSED']
|
||||||
|
if closed:
|
||||||
|
print('WARNING: CLOSED (unmerged) PRs in milestone:')
|
||||||
|
for p in closed:
|
||||||
|
print(f' #{p[\"number\"]} {p[\"title\"][:80]}')
|
||||||
|
"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Post-edit audit checklist:**
|
||||||
|
- ✅ All referenced PRs are merged (no closed-unmerged artifacts)
|
||||||
|
- ✅ Every merged milestone PR is either in the changelog or excluded by label
|
||||||
|
- ✅ PR and issue counts are internally consistent
|
||||||
|
- ✅ No false-positive PR-to-issue associations
|
||||||
|
- ✅ Advisory (GHSA) entries are not milestone PRs — their absence from the
|
||||||
|
cross-reference is intentional (see step 5b)
|
||||||
|
|
||||||
|
## Version section template
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
## <VERSION>
|
||||||
|
|
||||||
|
### :bug: Bugs fixed
|
||||||
|
|
||||||
|
- <fix description> [#<ISSUE>](https://github.com/penpot/penpot/issues/<ISSUE>) (PR: [#<PR>](https://github.com/penpot/penpot/pull/<PR>))
|
||||||
|
- <fix description> (by @contributor) [#<ISSUE>](https://github.com/penpot/penpot/issues/<ISSUE>) (PR: [#<PR>](https://github.com/penpot/penpot/pull/<PR>))
|
||||||
|
- <fix description> (https://github.com/penpot/penpot/security/advisories/GHSA-XXXX-XXXX-XXXX)
|
||||||
|
```
|
||||||
|
|
||||||
|
Advisory (GHSA) entries have no issue or PR link — just the advisory URL. See
|
||||||
|
step 5b.
|
||||||
|
|
||||||
|
### 11. Generate anomaly report and save to CHANGES-ISSUES.md
|
||||||
|
|
||||||
|
After all edits and cross-referencing are complete, generate a structured
|
||||||
|
report and save it to `CHANGES-ISSUES.md` (overwriting if exists).
|
||||||
|
This provides a persistent record of any discrepancies between the milestone
|
||||||
|
and the changelog.
|
||||||
|
|
||||||
|
**Every issue and PR number in the report must be rendered as a full GitHub
|
||||||
|
Markdown link** using the same URL format as `CHANGES.md`:
|
||||||
|
- Issue N → `[#N](https://github.com/penpot/penpot/issues/N)`
|
||||||
|
- PR N → `[#N](https://github.com/penpot/penpot/pull/N)`
|
||||||
|
|
||||||
|
The titles and notes should also link to the corresponding issue/PR page
|
||||||
|
where applicable, so the report is self-contained and clickable from any
|
||||||
|
Markdown viewer.
|
||||||
|
|
||||||
|
## What is an anomaly
|
||||||
|
|
||||||
|
**An anomaly is a milestone-mismatch between an issue and its referenced
|
||||||
|
PR.** It indicates that the changelog claim "this issue is fixed by this PR,
|
||||||
|
all in milestone M" is inconsistent with the actual milestone assignments.
|
||||||
|
There are exactly two types:
|
||||||
|
|
||||||
|
1. **Issue is in the milestone, but its referenced PR is in a different
|
||||||
|
milestone (or has no milestone).** The changelog claims a fix in this
|
||||||
|
release, but the PR is being released elsewhere — the fix may not
|
||||||
|
actually ship here.
|
||||||
|
2. **PR is in the milestone, but the issue it closes is in a different
|
||||||
|
milestone.** The PR is being released here, but the issue it fixes is
|
||||||
|
being released in a different version — the changelog pairing is
|
||||||
|
misleading.
|
||||||
|
|
||||||
|
**Exception — issue with no milestone is NOT an anomaly.** Milestones
|
||||||
|
are only required for issues tracked in the "Main" project. A milestone
|
||||||
|
PR that closes an issue with no milestone references an issue from
|
||||||
|
another (probably private) project; that is expected and the issue is
|
||||||
|
not part of this changelog. Do not report it.
|
||||||
|
|
||||||
|
**Anything else is not an anomaly.** Other discrepancies (exclusion
|
||||||
|
labels on in-changelog issues, missing valid issues, unmerged PR
|
||||||
|
references, duplicates across versions, stale milestone assignments)
|
||||||
|
are **rule violations** that the LLM must fix directly in `CHANGES.md`
|
||||||
|
during step 6a (pre-flight checks). They should not appear in this
|
||||||
|
report — if they do, the LLM has skipped the pre-flight step and
|
||||||
|
needs to re-run the workflow.
|
||||||
|
|
||||||
|
The changelog's primary unit is the **issue**, not the PR, so a missing or
|
||||||
|
mismatched PR only matters when its issue is part of this milestone.
|
||||||
|
|
||||||
|
Run this self-contained script:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 << 'PYEOF'
|
||||||
|
import json, re, subprocess, sys
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
|
||||||
|
MILESTONE = "<MILESTONE>"
|
||||||
|
CHANGES_MD = "CHANGES.md"
|
||||||
|
OUTPUT = "CHANGES-ISSUES.md"
|
||||||
|
REPO = "penpot/penpot"
|
||||||
|
|
||||||
|
# --- URL helpers (match CHANGES.md format exactly) ---
|
||||||
|
def issue_url(n): return f"https://github.com/{REPO}/issues/{n}"
|
||||||
|
def pr_url(n): return f"https://github.com/{REPO}/pull/{n}"
|
||||||
|
def issue_link(n): return f"[#{n}]({issue_url(n)})"
|
||||||
|
def pr_link(n): return f"[#{n}]({pr_url(n)})"
|
||||||
|
def issue_link_title(n, title):
|
||||||
|
url = issue_url(n)
|
||||||
|
if title:
|
||||||
|
return f"[#{n}]({url}) — [{title}]({url})"
|
||||||
|
return f"[#{n}]({url})"
|
||||||
|
def pr_link_title(n, title):
|
||||||
|
url = pr_url(n)
|
||||||
|
if title:
|
||||||
|
return f"[#{n}]({url}) — [{title}]({url})"
|
||||||
|
return f"[#{n}]({url})"
|
||||||
|
def fmt_pr_list(nums):
|
||||||
|
return ", ".join(pr_link(n) for n in nums)
|
||||||
|
def fmt_issue_list(nums):
|
||||||
|
return ", ".join(issue_link(n) for n in nums)
|
||||||
|
|
||||||
|
# --- Fetch milestone data ---
|
||||||
|
result = subprocess.run(
|
||||||
|
["python3", "scripts/gh.py", "issues", MILESTONE, "--state", "all"],
|
||||||
|
capture_output=True, text=True)
|
||||||
|
all_issues = json.loads(result.stdout)
|
||||||
|
issue_by_num = {i['number']: i for i in all_issues}
|
||||||
|
|
||||||
|
result = subprocess.run(
|
||||||
|
["python3", "scripts/gh.py", "prs", "--milestone", MILESTONE, "--state", "all"],
|
||||||
|
capture_output=True, text=True)
|
||||||
|
all_prs = json.loads(result.stdout)
|
||||||
|
pr_by_num = {p['number']: p for p in all_prs}
|
||||||
|
|
||||||
|
# --- Read changelog section ---
|
||||||
|
with open(CHANGES_MD) as f:
|
||||||
|
content = f.read()
|
||||||
|
|
||||||
|
m = re.search(rf'## {re.escape(MILESTONE)}(?:\s*\([^)]*\))?\n(.*?)(?:\n## |\Z)', content, re.DOTALL)
|
||||||
|
section = m.group(1) if m else ""
|
||||||
|
|
||||||
|
changelog_issues = set()
|
||||||
|
for num in re.findall(r'\[#(\d+)\]\(https://github\.com/penpot/penpot/issues/\d+\)', section):
|
||||||
|
changelog_issues.add(int(num))
|
||||||
|
for num in re.findall(r'\[Github #(\d+)\]', section):
|
||||||
|
changelog_issues.add(int(num))
|
||||||
|
|
||||||
|
changelog_prs = set()
|
||||||
|
for num in re.findall(r'\[#(\d+)\]\(https://github\.com/penpot/penpot/pull/\d+\)', section):
|
||||||
|
changelog_prs.add(int(num))
|
||||||
|
for num in re.findall(r'PR:\[(\d+)\]', section):
|
||||||
|
changelog_prs.add(int(num))
|
||||||
|
|
||||||
|
# --- Milestone lookup caches ---
|
||||||
|
# PRs and issues returned by milestone queries are KNOWN to be in MILESTONE.
|
||||||
|
# For everything else, fall back to `gh` per-item lookups.
|
||||||
|
pr_milestone_cache = {p['number']: MILESTONE for p in all_prs}
|
||||||
|
issue_milestone_cache = {i['number']: MILESTONE for i in all_issues}
|
||||||
|
|
||||||
|
def get_pr_milestone(pr_num):
|
||||||
|
"""Return the milestone title for a PR, or None if unassigned / unknown."""
|
||||||
|
if pr_num in pr_milestone_cache:
|
||||||
|
return pr_milestone_cache[pr_num]
|
||||||
|
try:
|
||||||
|
r = subprocess.run(
|
||||||
|
["gh", "pr", "view", str(pr_num), "--json", "milestone"],
|
||||||
|
capture_output=True, text=True, check=True)
|
||||||
|
data = json.loads(r.stdout)
|
||||||
|
ms = data.get('milestone')
|
||||||
|
pr_milestone_cache[pr_num] = (ms or {}).get('title')
|
||||||
|
except (subprocess.CalledProcessError, json.JSONDecodeError):
|
||||||
|
pr_milestone_cache[pr_num] = None
|
||||||
|
return pr_milestone_cache[pr_num]
|
||||||
|
|
||||||
|
def get_issue_milestone(issue_num):
|
||||||
|
"""Return the milestone title for an issue, or None if unassigned / unknown."""
|
||||||
|
if issue_num in issue_milestone_cache:
|
||||||
|
return issue_milestone_cache[issue_num]
|
||||||
|
try:
|
||||||
|
r = subprocess.run(
|
||||||
|
["gh", "issue", "view", str(issue_num), "--json", "milestone"],
|
||||||
|
capture_output=True, text=True, check=True)
|
||||||
|
data = json.loads(r.stdout)
|
||||||
|
ms = data.get('milestone')
|
||||||
|
issue_milestone_cache[issue_num] = (ms or {}).get('title')
|
||||||
|
except (subprocess.CalledProcessError, json.JSONDecodeError):
|
||||||
|
issue_milestone_cache[issue_num] = None
|
||||||
|
return issue_milestone_cache[issue_num]
|
||||||
|
|
||||||
|
# --- Exclusion rules (shared) ---
|
||||||
|
EXCLUDED_LABELS = {'release blocker', 'no changelog'}
|
||||||
|
EXCLUDED_ISSUE_TYPES = {'Task'}
|
||||||
|
EXCLUDED_PROJECT_STATUS = {'Rejected'}
|
||||||
|
|
||||||
|
def issue_excluded(issue):
|
||||||
|
if not issue: return True
|
||||||
|
if issue.get('state') != 'CLOSED': return True
|
||||||
|
if issue.get('issue_type') in EXCLUDED_ISSUE_TYPES: return True
|
||||||
|
if issue.get('project_status') in EXCLUDED_PROJECT_STATUS: return True
|
||||||
|
if EXCLUDED_LABELS & set(issue.get('labels', [])): return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
# --- ANOMALIES: milestone mismatches between issues and their referenced PRs ---
|
||||||
|
# These are the ONLY items that should appear in the report. All other
|
||||||
|
# discrepancies (exclusion labels, missing valid issues, unmerged PRs,
|
||||||
|
# duplicates, stale milestone assignments) are workflow errors that the
|
||||||
|
# LLM must fix in step 6a (pre-flight checks) — they are not anomalies.
|
||||||
|
|
||||||
|
# Type A: issue in MILESTONE, referenced PR in different milestone or no milestone
|
||||||
|
anomalies_a = [] # list of dicts: {issue, issue_title, pr, pr_milestone}
|
||||||
|
for issue_num in sorted(changelog_issues):
|
||||||
|
issue = issue_by_num.get(issue_num)
|
||||||
|
if not issue: continue
|
||||||
|
if get_issue_milestone(issue_num) != MILESTONE: continue
|
||||||
|
for pr_num in issue.get('closing_prs', []):
|
||||||
|
pr_ms = get_pr_milestone(pr_num)
|
||||||
|
if pr_ms != MILESTONE:
|
||||||
|
anomalies_a.append({
|
||||||
|
'issue': issue_num,
|
||||||
|
'issue_title': issue.get('title', ''),
|
||||||
|
'pr': pr_num,
|
||||||
|
'pr_milestone': pr_ms, # may be None
|
||||||
|
})
|
||||||
|
|
||||||
|
# Type B: PR in MILESTONE, the issue it closes is in different milestone or no milestone
|
||||||
|
anomalies_b = [] # list of dicts: {pr, pr_title, issue, issue_milestone}
|
||||||
|
for pr_num in sorted(changelog_prs):
|
||||||
|
pr = pr_by_num.get(pr_num)
|
||||||
|
if not pr: continue
|
||||||
|
if get_pr_milestone(pr_num) != MILESTONE: continue
|
||||||
|
for issue_num in pr.get('closing_issues', []):
|
||||||
|
issue_ms = get_issue_milestone(issue_num)
|
||||||
|
# No milestone = issue from another (probably private) project —
|
||||||
|
# milestones are only required for the "Main" project. Not an
|
||||||
|
# anomaly, and the issue never belongs in this changelog.
|
||||||
|
if issue_ms is None: continue
|
||||||
|
if issue_ms != MILESTONE:
|
||||||
|
anomalies_b.append({
|
||||||
|
'pr': pr_num,
|
||||||
|
'pr_title': pr.get('title', ''),
|
||||||
|
'issue': issue_num,
|
||||||
|
'issue_milestone': issue_ms, # may be None
|
||||||
|
})
|
||||||
|
|
||||||
|
# --- Write report ---
|
||||||
|
def fmt_ms(ms):
|
||||||
|
return ms if ms else "_none_"
|
||||||
|
|
||||||
|
with open(OUTPUT, 'w') as f:
|
||||||
|
f.write(f'# Changelog Anomaly Report — {MILESTONE}\n\n')
|
||||||
|
f.write(f'Generated: {datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")}\n\n')
|
||||||
|
f.write('---\n\n')
|
||||||
|
|
||||||
|
n_a = len(anomalies_a)
|
||||||
|
n_b = len(anomalies_b)
|
||||||
|
|
||||||
|
f.write('## Summary\n\n')
|
||||||
|
f.write(f'- **Issue in {MILESTONE}, referenced PR in different milestone or no milestone:** {n_a}\n')
|
||||||
|
f.write(f'- **PR in {MILESTONE}, closing issue in a different milestone:** {n_b}\n')
|
||||||
|
f.write(f'- **Total anomalies:** {n_a + n_b}\n\n')
|
||||||
|
|
||||||
|
# --- Anomalies section ---
|
||||||
|
if n_a or n_b:
|
||||||
|
f.write('## Anomalies\n\n')
|
||||||
|
f.write('These are milestone mismatches between an issue in the changelog '
|
||||||
|
'and its referenced PR (or vice-versa). The changelog claim '
|
||||||
|
'"this issue is fixed by this PR, all in this milestone" is '
|
||||||
|
'inconsistent with the actual milestone assignments. '
|
||||||
|
'Resolve by either updating the milestone on the issue/PR or '
|
||||||
|
'removing the misleading entry from the changelog.\n\n')
|
||||||
|
|
||||||
|
if n_a:
|
||||||
|
f.write(f'### Issue in {MILESTONE}, PR in different milestone or no milestone\n\n')
|
||||||
|
by_issue = {}
|
||||||
|
for a in anomalies_a:
|
||||||
|
by_issue.setdefault(a['issue'], []).append(a)
|
||||||
|
for issue_num in sorted(by_issue):
|
||||||
|
entries = by_issue[issue_num]
|
||||||
|
title = entries[0]['issue_title']
|
||||||
|
f.write(f'- {issue_link_title(issue_num, title[:80])}\n')
|
||||||
|
for e in entries:
|
||||||
|
ms_label = fmt_ms(e['pr_milestone'])
|
||||||
|
badge = '🔴' if e['pr_milestone'] is None else '⚠️'
|
||||||
|
f.write(f' - {badge} Referenced {pr_link(e["pr"])} is in milestone **{ms_label}** (expected: {MILESTONE})\n')
|
||||||
|
f.write('\n')
|
||||||
|
|
||||||
|
if n_b:
|
||||||
|
f.write(f'\n### PR in {MILESTONE}, closing issue in a different milestone\n\n')
|
||||||
|
by_pr = {}
|
||||||
|
for b in anomalies_b:
|
||||||
|
by_pr.setdefault(b['pr'], []).append(b)
|
||||||
|
for pr_num in sorted(by_pr):
|
||||||
|
entries = by_pr[pr_num]
|
||||||
|
title = entries[0]['pr_title']
|
||||||
|
f.write(f'- {pr_link_title(pr_num, title[:80])}\n')
|
||||||
|
for e in entries:
|
||||||
|
ms_label = fmt_ms(e['issue_milestone'])
|
||||||
|
badge = '🔴' if e['issue_milestone'] is None else '⚠️'
|
||||||
|
f.write(f' - {badge} Closing {issue_link(e["issue"])} is in milestone **{ms_label}** (expected: {MILESTONE})\n')
|
||||||
|
f.write('\n')
|
||||||
|
else:
|
||||||
|
f.write('✅ No anomalies found. All (issue, PR) pairs in the changelog have aligned milestone assignments.\n\n')
|
||||||
|
|
||||||
|
# --- Context ---
|
||||||
|
f.write('---\n\n')
|
||||||
|
f.write('## Context\n\n')
|
||||||
|
f.write(f'- Milestone: **{MILESTONE}**\n')
|
||||||
|
f.write(f'- Milestone total issues (all states): {len(all_issues)}\n')
|
||||||
|
f.write(f'- Closed issues in milestone: {sum(1 for i in all_issues if i.get("state") == "CLOSED")}\n')
|
||||||
|
f.write(f'- Valid issues after exclusions (after step 5/6a): {len([i for i in all_issues if not issue_excluded(i)])}\n')
|
||||||
|
f.write(f'- Issues referenced in changelog: {len(changelog_issues)}\n')
|
||||||
|
f.write(f'- PRs referenced in changelog: {len(changelog_prs)}\n')
|
||||||
|
|
||||||
|
print(f"Anomaly report written to {OUTPUT}")
|
||||||
|
PYEOF
|
||||||
|
```
|
||||||
|
|
||||||
|
This generates `CHANGES-ISSUES.md` containing **only the anomalies** —
|
||||||
|
milestone mismatches between issues and their referenced PRs:
|
||||||
|
|
||||||
|
1. **Issue in milestone, referenced PR in different milestone or no milestone** —
|
||||||
|
the changelog claims a fix here, but the PR is released elsewhere.
|
||||||
|
2. **PR in milestone, closing issue in a different milestone** —
|
||||||
|
the PR is released here, but the issue it fixes belongs to another version.
|
||||||
|
(An issue with *no* milestone belongs to another, probably private,
|
||||||
|
project — milestones are only required on the "Main" project — so it is
|
||||||
|
neither an anomaly nor a changelog candidate.)
|
||||||
|
|
||||||
|
**Rule violations are not in the report** — they are workflow errors the
|
||||||
|
LLM must fix directly in `CHANGES.md` during step 6a (pre-flight checks).
|
||||||
|
If the report contains a rule violation, the LLM has skipped the pre-flight
|
||||||
|
step and needs to re-run the workflow before re-generating the report.
|
||||||
|
|
||||||
|
The report is overwritten each time it's generated, reflecting the current
|
||||||
|
state of the milestone and changelog. Every number is rendered as a full
|
||||||
|
`[#N](https://github.com/penpot/penpot/issues/N)` or
|
||||||
|
`[#N](https://github.com/penpot/penpot/pull/N)` link so the report is
|
||||||
|
self-contained and clickable in any Markdown viewer.
|
||||||
|
|
||||||
|
## Key Principles
|
||||||
|
|
||||||
|
- **Issue = changelog unit.** The primary link always points to the
|
||||||
|
user-facing issue, not the implementation PR.
|
||||||
|
- **PR = implementation detail.** Reference the PR inline so readers
|
||||||
|
can find the code changes.
|
||||||
|
- **Latest version first.** New sections are inserted at the top of the
|
||||||
|
changelog, below the `# CHANGELOG` header.
|
||||||
|
- **Issue Type determines section — exclusively.** Use the `issue_type` field from `gh.py` output (Bug → `:bug:`, Feature/Enhancement → `:sparkles:`). **Do not** use labels (`bug`, `enhancement`) or title emoji prefixes (`:bug:`, `:sparkles:`) — they are frequently wrong or contradictory. The `issue_type` is the single source of truth.
|
||||||
|
- **User-facing descriptions.** Write from the user's perspective — describe
|
||||||
|
what broke and what was fixed, not internal implementation details.
|
||||||
|
- **Community attribution.** When the issue or fix PR has the
|
||||||
|
`community contribution` label, add `(by @<username>)` on the entry line
|
||||||
|
between the description and the issue link. Use the **PR author** (not the
|
||||||
|
issue author) for the attribution.
|
||||||
|
- **Only closed issues.** An issue must have `state: "closed"` to appear in
|
||||||
|
the changelog. Open/unresolved issues are omitted.
|
||||||
|
- **Rejected project status.** Issues marked as "Rejected" in the "Main"
|
||||||
|
project board are automatically excluded by `gh.py`, even if they are
|
||||||
|
closed. The project status is distinct from the GitHub issue state.
|
||||||
|
Use `--include-rejected` to override this behavior.
|
||||||
|
- **Excluded issues.** Issues with `no changelog` label must be excluded.
|
||||||
|
Issues with `issue_type: "Task"` must also be excluded — they are internal
|
||||||
|
chores, not user-facing changes.
|
||||||
|
- **Multiple PRs per issue.** If multiple PRs fix the same issue, list them
|
||||||
|
comma-separated inline: `(PR: [#A](url), [#B](url))`.
|
||||||
|
- **Duplicate removal.** If an entry already exists in a prior version section,
|
||||||
|
remove it from the current version. Check for text-level duplicates (after
|
||||||
|
stripping links and attributions) across version sections.
|
||||||
|
- **Taiga references.** If a changelog entry references a Taiga URL
|
||||||
|
(`tree.taiga.io`), attempt to find a corresponding GitHub issue via the
|
||||||
|
Taiga description text or by searching GitHub PRs that reference the Taiga
|
||||||
|
URL. Replace the Taiga reference with the GitHub issue link and add the PR
|
||||||
|
reference if applicable.
|
||||||
|
- **Security advisory (GHSA) entries.** Advisories fixed in the release are
|
||||||
|
listed under `### :bug: Bugs fixed` with the advisory URL and **no issue or
|
||||||
|
PR link**, even though they are not in the milestone. The GHSA ID and
|
||||||
|
description come from the user — do **not** fetch or verify the URL, and do
|
||||||
|
not drop a draft (unpublished) advisory. Precedent:
|
||||||
|
`- Fix arbitrary file read security issue on create-font-variant rpc method
|
||||||
|
(https://github.com/penpot/penpot/security/advisories/GHSA-xp3f-g8rq-9px2)`.
|
||||||
|
See step 5b.
|
||||||
|
- **Re-fetch before editing.** Milestones can change — always re-fetch issues
|
||||||
|
before making edits, don't rely on cached data.
|
||||||
|
- **Use `scripts/gh.py`.** Prefer the helper script over raw `gh api` calls for
|
||||||
|
milestone issue listing and PR detail fetching. It handles GraphQL
|
||||||
|
pagination, batching, and label filtering automatically.
|
||||||
|
- **Verify PR merge status.** Not all closing PRs are merged — community PRs
|
||||||
|
can be superseded and closed without merging. Always check that every PR
|
||||||
|
referenced in the changelog has `state: MERGED`.
|
||||||
|
- **PR-level exclusions apply.** A PR can carry its own exclusion labels
|
||||||
|
(`release blocker`, `no issue required`) independent of its linked issue's
|
||||||
|
labels. Check both.
|
||||||
|
- **Cross-reference milestone PRs, not just issues.** The `--compare` flag on
|
||||||
|
the `issues` command only compares issue numbers. Merged PRs not linked to
|
||||||
|
any milestone issue can be missed. Use `python3 scripts/gh.py prs --milestone`
|
||||||
|
for a full PR cross-reference.
|
||||||
|
- **False-positive PR-to-issue associations.** A PR may claim to close an
|
||||||
|
issue from a different project or context. If the PR title and issue title
|
||||||
|
are clearly unrelated, or the PR predates the issue by years, treat it as a
|
||||||
|
data glitch and skip it.
|
||||||
|
- **Anomaly = milestone mismatch only.** The report contains only milestone
|
||||||
|
mismatches: (1) the issue is in this milestone but the referenced PR is
|
||||||
|
in a different milestone (or unassigned), and (2) the PR is in this
|
||||||
|
milestone but the issue it closes is in a different milestone. An
|
||||||
|
*unassigned* (milestone-less) issue closed by a milestone PR is **not**
|
||||||
|
an anomaly: milestones are required only for the "Main" project, so such
|
||||||
|
issues come from another (probably private) project and are not changelog
|
||||||
|
candidates. These anomalies are reported because the changelog pairing is
|
||||||
|
*misleading* — the human needs to decide whether the milestone or the
|
||||||
|
changelog is wrong. All other discrepancies (exclusion labels, missing
|
||||||
|
valid issues, unmerged PR references, duplicates, stale milestone
|
||||||
|
assignments) are **rule violations** that the LLM must fix directly in
|
||||||
|
`CHANGES.md` during step 6a (pre-flight checks). They never appear in
|
||||||
|
the report — if they do, the pre-flight step was skipped.
|
||||||
|
|||||||
1
.claude/skills
Symbolic link
1
.claude/skills
Symbolic link
@ -0,0 +1 @@
|
|||||||
|
../.agents/skills
|
||||||
@ -9,8 +9,7 @@
|
|||||||
app.common.data.macros/get-in clojure.core/get-in
|
app.common.data.macros/get-in clojure.core/get-in
|
||||||
app.common.data.macros/with-open clojure.core/with-open
|
app.common.data.macros/with-open clojure.core/with-open
|
||||||
app.common.data.macros/select-keys clojure.core/select-keys
|
app.common.data.macros/select-keys clojure.core/select-keys
|
||||||
app.common.logging/with-context clojure.core/do
|
app.common.logging/with-context clojure.core/do}
|
||||||
benches.render-wasm.errors/attempt clojure.core/do}
|
|
||||||
|
|
||||||
:hooks
|
:hooks
|
||||||
{:analyze-call
|
{:analyze-call
|
||||||
@ -23,13 +22,7 @@
|
|||||||
rumext.v2/defc hooks.export/rumext-defc
|
rumext.v2/defc hooks.export/rumext-defc
|
||||||
rumext.v2/lazy-component hooks.export/rumext-lazycomponent
|
rumext.v2/lazy-component hooks.export/rumext-lazycomponent
|
||||||
shadow.lazy/loadable hooks.export/rumext-lazycomponent
|
shadow.lazy/loadable hooks.export/rumext-lazycomponent
|
||||||
app.util.i18n/tr hooks.i18n/tr-dynamic
|
}}
|
||||||
app.common.i18n/tr hooks.i18n/tr-dynamic
|
|
||||||
benches.render-wasm.declarations/defscene hooks.benches/defscene
|
|
||||||
benches.render-wasm.declarations/defcase hooks.benches/defcase
|
|
||||||
benches.render-wasm.runtime.commands/defcommand hooks.benches/defcommand
|
|
||||||
benches.render-wasm.runtime.protocol/resolve-or-reject hooks.benches/resolve-or-reject
|
|
||||||
benches.render-wasm.runtime.protocol/resolve-or-reject-with hooks.benches/resolve-or-reject-with}}
|
|
||||||
|
|
||||||
:output
|
:output
|
||||||
{:exclude-files
|
{:exclude-files
|
||||||
@ -41,7 +34,7 @@
|
|||||||
"^(?:backend|frontend|exporter|common)/deps.edn"
|
"^(?:backend|frontend|exporter|common)/deps.edn"
|
||||||
"^(?:backend|frontend|exporter|common)/scripts/"
|
"^(?:backend|frontend|exporter|common)/scripts/"
|
||||||
"^(?:backend|frontend|exporter|common)/dev/"
|
"^(?:backend|frontend|exporter|common)/dev/"
|
||||||
"^(?:backend|frontend|exporter|common)/test/(?!performance_tests/)"]
|
"^(?:backend|frontend|exporter|common)/test/"]
|
||||||
|
|
||||||
:linter-name true}
|
:linter-name true}
|
||||||
|
|
||||||
@ -52,9 +45,6 @@
|
|||||||
:potok/reify-type
|
:potok/reify-type
|
||||||
{:level :error}
|
{:level :error}
|
||||||
|
|
||||||
:penpot/tr-dynamic
|
|
||||||
{:level :warning}
|
|
||||||
|
|
||||||
:redundant-primitive-coercion
|
:redundant-primitive-coercion
|
||||||
{:level :off}
|
{:level :off}
|
||||||
|
|
||||||
@ -106,4 +96,6 @@
|
|||||||
|
|
||||||
:unused-binding
|
:unused-binding
|
||||||
{:exclude-destructured-as true
|
{:exclude-destructured-as true
|
||||||
:exclude-destructured-keys-in-fn-args false}}}
|
:exclude-destructured-keys-in-fn-args false
|
||||||
|
}
|
||||||
|
}}
|
||||||
|
|||||||
@ -1,77 +0,0 @@
|
|||||||
;; This Source Code Form is subject to the terms of the Mozilla Public
|
|
||||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
|
||||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
|
||||||
;;
|
|
||||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
|
||||||
|
|
||||||
(ns hooks.benches
|
|
||||||
(:require [clj-kondo.hooks-api :as api]))
|
|
||||||
|
|
||||||
(defn defscene
|
|
||||||
"Analyzes scene options and its compiled builder body."
|
|
||||||
[{:keys [node]}]
|
|
||||||
(let [[_ _ opts argv & body] (:children node)]
|
|
||||||
(if (and argv (api/vector-node? argv))
|
|
||||||
{:node (api/list-node [(api/token-node (quote do)) opts
|
|
||||||
(api/list-node (into [(api/token-node (quote fn)) argv] body))])}
|
|
||||||
{:node node})))
|
|
||||||
|
|
||||||
(defn defcase
|
|
||||||
"Analyzes case options and its compiled operation body."
|
|
||||||
[{:keys [node]}]
|
|
||||||
(let [[_ _ opts argv & body] (:children node)]
|
|
||||||
(if (and (seq body) argv (api/vector-node? argv))
|
|
||||||
{:node (api/list-node [(api/token-node (quote do)) opts
|
|
||||||
(api/list-node (into [(api/token-node (quote fn)) argv] body))])}
|
|
||||||
{:node node})))
|
|
||||||
|
|
||||||
(defn defcommand
|
|
||||||
"Analyzes command options and both generated functions."
|
|
||||||
[{:keys [node]}]
|
|
||||||
(let [[_ command & forms] (:children node)
|
|
||||||
[doc forms] (if (string? (api/sexpr (first forms)))
|
|
||||||
[(first forms) (next forms)]
|
|
||||||
[nil forms])
|
|
||||||
[options forms] (if (api/map-node? (first forms))
|
|
||||||
[(first forms) (next forms)]
|
|
||||||
[nil forms])
|
|
||||||
[argv & body] forms
|
|
||||||
[conditions body] (if (and (api/map-node? (first body)) (next body))
|
|
||||||
[(first body) (next body)]
|
|
||||||
[nil body])
|
|
||||||
handlers (:async (some-> options api/sexpr))
|
|
||||||
core (api/list-node
|
|
||||||
(into [(api/token-node 'fn)
|
|
||||||
(api/vector-node (mapv api/token-node handlers))]
|
|
||||||
body))
|
|
||||||
starred (api/token-node (symbol (str (api/sexpr command) "*")))
|
|
||||||
measured (get (some-> options api/sexpr) :measured-variant true)
|
|
||||||
definition (api/list-node
|
|
||||||
(into [(api/token-node 'defn) command]
|
|
||||||
(concat (when doc [doc]) [argv]
|
|
||||||
(when conditions [conditions]) [core])))]
|
|
||||||
{:node (api/list-node
|
|
||||||
(into [(api/token-node 'do)]
|
|
||||||
(concat (when options [(api/coerce (dissoc (api/sexpr options) :async))])
|
|
||||||
[definition]
|
|
||||||
(when measured
|
|
||||||
[(api/list-node [(api/token-node 'def)
|
|
||||||
starred command])]))))}))
|
|
||||||
|
|
||||||
(defn resolve-or-reject
|
|
||||||
"Analyzes the body of a Promise-returning form."
|
|
||||||
[{:keys [node]}]
|
|
||||||
(let [[_ & body] (:children node)]
|
|
||||||
{:node (api/list-node (into [(api/token-node (quote do))] body))}))
|
|
||||||
|
|
||||||
(defn resolve-or-reject-with
|
|
||||||
"Analyzes the cause handler and body. Binds `cause` in the handler only."
|
|
||||||
[{:keys [node]}]
|
|
||||||
(let [[_ cause on-error & body] (:children node)]
|
|
||||||
{:node (api/list-node
|
|
||||||
(into [(api/token-node (quote do))
|
|
||||||
(api/list-node
|
|
||||||
[(api/token-node (quote let))
|
|
||||||
(api/vector-node [cause (api/token-node nil)])
|
|
||||||
on-error])]
|
|
||||||
body))}))
|
|
||||||
@ -1,9 +1,3 @@
|
|||||||
;; This Source Code Form is subject to the terms of the Mozilla Public
|
|
||||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
|
||||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
|
||||||
;;
|
|
||||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
|
||||||
|
|
||||||
(ns hooks.export
|
(ns hooks.export
|
||||||
(:require [clj-kondo.hooks-api :as api]))
|
(:require [clj-kondo.hooks-api :as api]))
|
||||||
|
|
||||||
|
|||||||
@ -1,17 +0,0 @@
|
|||||||
(ns hooks.i18n
|
|
||||||
(:require [clj-kondo.hooks-api :as api]))
|
|
||||||
|
|
||||||
(defn tr-dynamic
|
|
||||||
[{:keys [:node]}]
|
|
||||||
(let [[_ code & _] (:children node)]
|
|
||||||
(when (and (some? code)
|
|
||||||
(not (api/string-node? code)))
|
|
||||||
(let [{:keys [:row :col :end-row :end-col]} (meta code)]
|
|
||||||
(api/reg-finding! {:message "dynamic key in (tr ...) is invisible to rehash; use a string literal or declare it with ;; (tr \"key\")"
|
|
||||||
:type :penpot/tr-dynamic
|
|
||||||
:row row
|
|
||||||
:col col
|
|
||||||
;; end positions are required for #_:clj-kondo/ignore to match
|
|
||||||
:end-row end-row
|
|
||||||
:end-col end-col}))))
|
|
||||||
{:node node})
|
|
||||||
@ -10,7 +10,5 @@
|
|||||||
:extra-indents {rumext.v2/fnc [[:inner 0]]
|
:extra-indents {rumext.v2/fnc [[:inner 0]]
|
||||||
cljs.test/async [[:inner 0]]
|
cljs.test/async [[:inner 0]]
|
||||||
promesa.exec/thread [[:inner 0]]
|
promesa.exec/thread [[:inner 0]]
|
||||||
specify! [[:inner 0] [:inner 1]]
|
specify! [[:inner 0] [:inner 1]]}
|
||||||
benches.render-wasm.runtime.commands/defcommand [[:inner 0]]
|
|
||||||
benches.render-wasm.runtime.protocol/resolve-or-reject-with [[:inner 0]]}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@ -85,10 +85,6 @@ target instance is running and its MCP config has been generated:
|
|||||||
# 'main', 'ws1' and similar spellings are rejected.
|
# 'main', 'ws1' and similar spellings are rejected.
|
||||||
./manage.sh start-coding-agent claude --ws 1
|
./manage.sh start-coding-agent claude --ws 1
|
||||||
./manage.sh start-coding-agent opencode --ws 2
|
./manage.sh start-coding-agent opencode --ws 2
|
||||||
|
|
||||||
# Run a different executable (e.g. a wrapper script) with the same MCP
|
|
||||||
# wiring. --ws and --bin may come in any order.
|
|
||||||
./manage.sh start-coding-agent claude --bin claude-alt
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Equivalents by hand (run from inside the workspace directory):
|
Equivalents by hand (run from inside the workspace directory):
|
||||||
|
|||||||
9
.github/CODEOWNERS
vendored
9
.github/CODEOWNERS
vendored
@ -1,9 +0,0 @@
|
|||||||
# Changes to workflows, CI configuration and repository automation need
|
|
||||||
# review from the ci-maintainers team. Issue and PR templates do not.
|
|
||||||
/.github/workflows/ @penpot/ci-maintainers
|
|
||||||
/.github/actions/ @penpot/ci-maintainers
|
|
||||||
/.github/scripts/ @penpot/ci-maintainers
|
|
||||||
/.github/dependabot.yml @penpot/ci-maintainers
|
|
||||||
/.github/zizmor.yml @penpot/ci-maintainers
|
|
||||||
/.github/actionlint.yaml @penpot/ci-maintainers
|
|
||||||
/.github/CODEOWNERS @penpot/ci-maintainers
|
|
||||||
@ -1,7 +1,7 @@
|
|||||||
description: An unexpected problem or behavior
|
description: Create a report to help us improve
|
||||||
labels: ["needs triage"]
|
name: Bug report
|
||||||
name: Bug
|
|
||||||
type: Bug
|
type: Bug
|
||||||
|
labels: ["needs triage"]
|
||||||
|
|
||||||
body:
|
body:
|
||||||
- type: markdown
|
- type: markdown
|
||||||
@ -15,7 +15,7 @@ body:
|
|||||||
- type: textarea
|
- type: textarea
|
||||||
attributes:
|
attributes:
|
||||||
label: Steps To Reproduce
|
label: Steps To Reproduce
|
||||||
description: Steps to reproduce the behavior
|
description: Steps to reproduce the behavior.
|
||||||
placeholder: |
|
placeholder: |
|
||||||
Steps to reproduce the behavior:
|
Steps to reproduce the behavior:
|
||||||
1. Go to '...'
|
1. Go to '...'
|
||||||
@ -26,22 +26,22 @@ body:
|
|||||||
- type: textarea
|
- type: textarea
|
||||||
id: expected
|
id: expected
|
||||||
attributes:
|
attributes:
|
||||||
|
description: A clear and concise description of what you expected to happen.
|
||||||
label: Expected behavior
|
label: Expected behavior
|
||||||
description: A clear and concise description of what you expected to happen
|
|
||||||
validations:
|
validations:
|
||||||
required: true
|
required: true
|
||||||
- type: textarea
|
- type: textarea
|
||||||
id: actual
|
id: actual
|
||||||
attributes:
|
attributes:
|
||||||
|
description: A clear and concise description of what happens instead; what the bug is.
|
||||||
label: Actual behavior
|
label: Actual behavior
|
||||||
description: A clear and concise description of what happens instead; what the bug is
|
|
||||||
validations:
|
validations:
|
||||||
required: true
|
required: true
|
||||||
- type: textarea
|
- type: textarea
|
||||||
id: screenshots
|
id: screenshots
|
||||||
attributes:
|
attributes:
|
||||||
|
description: If applicable, add screenshots to help explain your problem.
|
||||||
label: Screenshots or video
|
label: Screenshots or video
|
||||||
description: If applicable, add screenshots to help explain your problem
|
|
||||||
- type: textarea
|
- type: textarea
|
||||||
id: desktop
|
id: desktop
|
||||||
attributes:
|
attributes:
|
||||||
@ -87,4 +87,4 @@ body:
|
|||||||
id: additional-context
|
id: additional-context
|
||||||
attributes:
|
attributes:
|
||||||
label: Additional context
|
label: Additional context
|
||||||
description: Add any other context or screenshots about the bug here
|
description: Any other context about the problem.
|
||||||
2
.github/ISSUE_TEMPLATE/config.yml
vendored
2
.github/ISSUE_TEMPLATE/config.yml
vendored
@ -1 +1 @@
|
|||||||
blank_issues_enabled: false
|
blank_issues_enabled: true
|
||||||
|
|||||||
38
.github/ISSUE_TEMPLATE/enhancement.yml
vendored
38
.github/ISSUE_TEMPLATE/enhancement.yml
vendored
@ -1,38 +0,0 @@
|
|||||||
description: Improvement to existing behavior
|
|
||||||
labels: ["needs triage"]
|
|
||||||
name: "Enhancement"
|
|
||||||
type: Enhancement
|
|
||||||
|
|
||||||
body:
|
|
||||||
- type: markdown
|
|
||||||
attributes:
|
|
||||||
value: |
|
|
||||||
## Before you start
|
|
||||||
|
|
||||||
Please search our [existing issues](https://github.com/penpot/penpot/issues) and open [pull requests](https://github.com/penpot/penpot/pulls) to lessen the change of filing duplicate issues or feature requests. Thank you.
|
|
||||||
|
|
||||||
---
|
|
||||||
- type: textarea
|
|
||||||
id: problem
|
|
||||||
attributes:
|
|
||||||
label: Is your feature request related to a problem? Please describe
|
|
||||||
description: A clear and concise description of what the problem is. Ex. I'm always frustrated when (...)
|
|
||||||
validations:
|
|
||||||
required: true
|
|
||||||
- type: textarea
|
|
||||||
id: solution
|
|
||||||
attributes:
|
|
||||||
label: Describe the solution you'd like
|
|
||||||
description: A clear and concise description of what you want to happen
|
|
||||||
validations:
|
|
||||||
required: true
|
|
||||||
- type: textarea
|
|
||||||
id: alternatives
|
|
||||||
attributes:
|
|
||||||
label: Describe alternatives you've considered
|
|
||||||
description: A clear and concise description of any alternative solutions or features you've considered
|
|
||||||
- type: textarea
|
|
||||||
id: additional-context
|
|
||||||
attributes:
|
|
||||||
label: Additional context
|
|
||||||
description: Add any other context or screenshots about the enhancement here
|
|
||||||
26
.github/ISSUE_TEMPLATE/epic.yml
vendored
26
.github/ISSUE_TEMPLATE/epic.yml
vendored
@ -1,26 +0,0 @@
|
|||||||
description: An epic is a large user story that cannot be delivered as defined within a single iteration
|
|
||||||
labels: ["needs triage"]
|
|
||||||
name: "EPIC"
|
|
||||||
type: EPIC
|
|
||||||
|
|
||||||
body:
|
|
||||||
- type: markdown
|
|
||||||
attributes:
|
|
||||||
value: |
|
|
||||||
## Before you start
|
|
||||||
|
|
||||||
Please search our [existing issues](https://github.com/penpot/penpot/issues) and open [pull requests](https://github.com/penpot/penpot/pulls) to lessen the change of filing duplicate issues or feature requests. Thank you.
|
|
||||||
|
|
||||||
---
|
|
||||||
- type: textarea
|
|
||||||
id: description
|
|
||||||
attributes:
|
|
||||||
label: Epic description
|
|
||||||
description: Tell us which epic feature do you want to discuss
|
|
||||||
validations:
|
|
||||||
required: true
|
|
||||||
- type: textarea
|
|
||||||
id: additional-context
|
|
||||||
attributes:
|
|
||||||
label: Additional context
|
|
||||||
description: Add any other context or screenshots about the feature request here
|
|
||||||
@ -1,7 +1,7 @@
|
|||||||
description: Suggest an idea for this project
|
description: Suggest an idea for this project.
|
||||||
labels: ["needs triage"]
|
labels: ["needs triage"]
|
||||||
name: "Feature request"
|
name: "Feature request"
|
||||||
type: Feature
|
type: Enhancement
|
||||||
|
|
||||||
body:
|
body:
|
||||||
- type: markdown
|
- type: markdown
|
||||||
@ -15,24 +15,24 @@ body:
|
|||||||
- type: textarea
|
- type: textarea
|
||||||
id: problem
|
id: problem
|
||||||
attributes:
|
attributes:
|
||||||
label: Is your feature request related to a problem? Please describe
|
|
||||||
description: A clear and concise description of what the problem is. Ex. I'm always frustrated when (...)
|
description: A clear and concise description of what the problem is. Ex. I'm always frustrated when (...)
|
||||||
|
label: Is your feature request related to a problem? Please describe.
|
||||||
validations:
|
validations:
|
||||||
required: true
|
required: true
|
||||||
- type: textarea
|
- type: textarea
|
||||||
id: solution
|
id: solution
|
||||||
attributes:
|
attributes:
|
||||||
label: Describe the solution you'd like
|
description: A clear and concise description of what you want to happen.
|
||||||
description: A clear and concise description of what you want to happen
|
label: Describe the solution you'd like.
|
||||||
validations:
|
validations:
|
||||||
required: true
|
required: true
|
||||||
- type: textarea
|
- type: textarea
|
||||||
id: alternatives
|
id: alternatives
|
||||||
attributes:
|
attributes:
|
||||||
label: Describe alternatives you've considered
|
label: Describe alternatives you've considered.
|
||||||
description: A clear and concise description of any alternative solutions or features you've considered
|
description: A clear and concise description of any alternative solutions or features you've considered.
|
||||||
- type: textarea
|
- type: textarea
|
||||||
id: additional-context
|
id: additional-context
|
||||||
attributes:
|
attributes:
|
||||||
label: Additional context
|
label: Additional context
|
||||||
description: Add any other context or screenshots about the feature request here
|
description: Add any other context or screenshots about the feature request here.
|
||||||
26
.github/ISSUE_TEMPLATE/task.yml
vendored
26
.github/ISSUE_TEMPLATE/task.yml
vendored
@ -1,26 +0,0 @@
|
|||||||
description: A specific piece of work
|
|
||||||
labels: []
|
|
||||||
name: "Task"
|
|
||||||
type: Task
|
|
||||||
|
|
||||||
body:
|
|
||||||
- type: markdown
|
|
||||||
attributes:
|
|
||||||
value: |
|
|
||||||
## Before you start
|
|
||||||
|
|
||||||
Please search our [existing issues](https://github.com/penpot/penpot/issues) and open [pull requests](https://github.com/penpot/penpot/pulls) to lessen the change of filing duplicate issues or feature requests. Thank you.
|
|
||||||
|
|
||||||
---
|
|
||||||
- type: textarea
|
|
||||||
id: task
|
|
||||||
attributes:
|
|
||||||
description: A clear and concise description of the task
|
|
||||||
label: Please describe
|
|
||||||
validations:
|
|
||||||
required: true
|
|
||||||
- type: textarea
|
|
||||||
id: additional-context
|
|
||||||
attributes:
|
|
||||||
label: Additional context
|
|
||||||
description: Add any other context or screenshots about the task here
|
|
||||||
14
.github/actionlint.yaml
vendored
14
.github/actionlint.yaml
vendored
@ -1,14 +0,0 @@
|
|||||||
# actionlint configuration — https://github.com/rhysd/actionlint/blob/main/docs/config.md
|
|
||||||
self-hosted-runner:
|
|
||||||
labels:
|
|
||||||
- penpot-standar-runner
|
|
||||||
- penpot-extended-runner
|
|
||||||
|
|
||||||
paths:
|
|
||||||
.github/workflows/**/*.{yml,yaml}:
|
|
||||||
ignore:
|
|
||||||
# shellcheck style/info notices that are pervasive in the existing
|
|
||||||
# scripts (unquoted `>> $GITHUB_OUTPUT`, repeated redirects). Real
|
|
||||||
# shellcheck warnings and errors are still reported.
|
|
||||||
- 'SC2086:info'
|
|
||||||
- 'SC2129:style'
|
|
||||||
41
.github/scripts/playwright-summary.jq
vendored
41
.github/scripts/playwright-summary.jq
vendored
@ -1,41 +0,0 @@
|
|||||||
def specs: [.. | objects | select(has("tests") and has("file"))];
|
|
||||||
def dur: [.tests[].results[]?.duration // 0] | add;
|
|
||||||
|
|
||||||
specs as $s
|
|
||||||
| ($s | map(select(any(.tests[]; .status == "unexpected")))) as $failed
|
|
||||||
| ($s | map(select(any(.tests[]; .status == "flaky")))) as $flaky
|
|
||||||
| ($s | map(select(any(.tests[]; .status == "skipped")))) as $skipped
|
|
||||||
| ($s | length) as $total
|
|
||||||
| ($s | map(dur) | add // 0 | . / 1000 | floor) as $cpu
|
|
||||||
| (if ($failed | length) > 0 then "❌"
|
|
||||||
elif ($flaky | length) > 0 then "⚠️"
|
|
||||||
else "✅" end) as $icon
|
|
||||||
|
|
||||||
| "## \($icon) Integration tests\n\n"
|
|
||||||
+ "| Total | Passed | Flaky | Failed | Skipped | Test time |\n"
|
|
||||||
+ "|---|---|---|---|---|---|\n"
|
|
||||||
+ "| \($total) | \($total - ($failed|length) - ($flaky|length) - ($skipped|length)) "
|
|
||||||
+ "| \($flaky|length) | \($failed|length) | \($skipped|length) | \($cpu / 60 | floor)m |\n"
|
|
||||||
|
|
||||||
+ (if ($failed | length) > 0 then
|
|
||||||
"\n### Failed\n\n"
|
|
||||||
+ ($failed | map("- `\(.file):\(.line)` — \(.title)") | join("\n")) + "\n"
|
|
||||||
else "" end)
|
|
||||||
|
|
||||||
+ (if ($flaky | length) > 0 then
|
|
||||||
"\n### Flaky (passed on retry)\n\n"
|
|
||||||
+ ($flaky
|
|
||||||
| map({ t: "`\(.file):\(.line)` — \(.title)",
|
|
||||||
r: ([.tests[].results[]? | select(.status == "failed")] | length) })
|
|
||||||
| sort_by(-.r)
|
|
||||||
| map("- \(.t) _(\(.r) \(if .r == 1 then "retry" else "retries" end))_")
|
|
||||||
| join("\n")) + "\n"
|
|
||||||
else "" end)
|
|
||||||
|
|
||||||
+ (if $total > 0 then
|
|
||||||
"\n<details><summary>Slowest specs</summary>\n\n"
|
|
||||||
+ ($s | map({ t: "`\(.file)` — \(.title)", d: (dur / 1000 | floor) })
|
|
||||||
| sort_by(-.d) | .[0:5]
|
|
||||||
| map("- \(.t) — \(.d)s") | join("\n"))
|
|
||||||
+ "\n\n</details>\n"
|
|
||||||
else "" end)
|
|
||||||
185
.github/workflows/auto-label.yml
vendored
185
.github/workflows/auto-label.yml
vendored
@ -1,51 +1,31 @@
|
|||||||
name: Auto Label and Add to Project
|
name: Auto Label and Add to Project
|
||||||
|
|
||||||
# pull_request_target is required so the GitHub App secrets are available
|
|
||||||
# for PRs from forks. It is safe here: the job never checks out or runs PR
|
|
||||||
# code, it only reads ids from the event payload.
|
|
||||||
on:
|
on:
|
||||||
issues:
|
issues:
|
||||||
types: [opened, closed]
|
types: [opened]
|
||||||
pull_request_target:
|
pull_request_target:
|
||||||
types: [opened]
|
types: [opened]
|
||||||
|
|
||||||
permissions: {}
|
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
triage:
|
triage:
|
||||||
# The Renovate Dependency Dashboard and Renovate's GitHub Actions PRs do
|
|
||||||
# not need triage. Other Renovate PRs (base images, Node.js, pnpm) do.
|
|
||||||
if: >-
|
|
||||||
!(github.event_name == 'issues'
|
|
||||||
&& github.event.issue.user.login == 'penpot-renovate[bot]')
|
|
||||||
&& !(github.event_name == 'pull_request_target'
|
|
||||||
&& github.event.pull_request.user.login == 'penpot-renovate[bot]'
|
|
||||||
&& github.event.pull_request.head.ref == 'renovate-github-actions')
|
|
||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
steps:
|
steps:
|
||||||
- name: Generate GitHub App token
|
- name: Generate GitHub App token
|
||||||
id: triage-app-token
|
id: triage-app-token
|
||||||
uses: actions/create-github-app-token@64ab729a6294000f5aa4a4bc0b73d1f32bc45510 # v3.3.0
|
uses: actions/create-github-app-token@v1
|
||||||
with:
|
with:
|
||||||
client-id: ${{ secrets.TRIAGE_APP_CLIENT_ID }}
|
app-id: ${{ secrets.TRIAGE_APP_ID }}
|
||||||
private-key: ${{ secrets.TRIAGE_APP_PRIVATE_KEY }}
|
private-key: ${{ secrets.TRIAGE_APP_PRIVATE_KEY }}
|
||||||
# No owner/repositories: the token is scoped to this repository.
|
owner: penpot
|
||||||
# Organization permissions (the project board) still apply.
|
|
||||||
permission-issues: write
|
|
||||||
permission-pull-requests: write
|
|
||||||
permission-organization-projects: write
|
|
||||||
|
|
||||||
- name: Process Issue or PR
|
- name: Process Issue or PR
|
||||||
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
|
uses: actions/github-script@v7
|
||||||
with:
|
with:
|
||||||
github-token: ${{ steps.triage-app-token.outputs.token }}
|
github-token: ${{ steps.triage-app-token.outputs.token }}
|
||||||
script: |
|
script: |
|
||||||
// === 1. CONFIGURATION ===
|
// === 1. CONFIGURATION ===
|
||||||
const PROJECT_NUMBER = 8; // <--- Replace with your project board number
|
const PROJECT_NUMBER = 8; // <--- Replace with your project board number
|
||||||
const IS_ORG = true; // <--- Set to false if this is a personal project, true if an organization
|
const IS_ORG = true; // <--- Set to false if this is a personal project, true if an organization
|
||||||
const STATUS_FIELD_ID = 'PVTSSF_lADOAcyBPM4BV2pqzhRPY48';
|
|
||||||
const STATUS_BACKLOG = 'f75ad846';
|
|
||||||
const STATUS_TECH_REVIEW = 'b0f45db9';
|
|
||||||
const OWNER = context.repo.owner;
|
const OWNER = context.repo.owner;
|
||||||
const REPO = context.repo.repo;
|
const REPO = context.repo.repo;
|
||||||
|
|
||||||
@ -53,100 +33,6 @@ jobs:
|
|||||||
const isPR = !!context.payload.pull_request;
|
const isPR = !!context.payload.pull_request;
|
||||||
const contentId = isPR ? context.payload.pull_request.node_id : context.payload.issue.node_id;
|
const contentId = isPR ? context.payload.pull_request.node_id : context.payload.issue.node_id;
|
||||||
|
|
||||||
// A completed issue is already closed by GitHub. Route its Main
|
|
||||||
// project status according to whether QA needs to verify it.
|
|
||||||
if (context.eventName === 'issues' && context.payload.action === 'closed') {
|
|
||||||
const issue = context.payload.issue;
|
|
||||||
if (issue.state_reason !== 'completed') {
|
|
||||||
console.log(`Issue #${issue.number} was closed as ${issue.state_reason}; leaving project status unchanged.`);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
const targetStatus = issue.labels.some(({ name }) => name === 'team-qa')
|
|
||||||
? 'Ready for QA'
|
|
||||||
: 'Product Review';
|
|
||||||
|
|
||||||
const projectQuery = `
|
|
||||||
query($owner: String!, $number: Int!) {
|
|
||||||
organization(login: $owner) {
|
|
||||||
projectV2(number: $number) {
|
|
||||||
id
|
|
||||||
fields(first: 100) {
|
|
||||||
nodes {
|
|
||||||
... on ProjectV2SingleSelectField {
|
|
||||||
id
|
|
||||||
name
|
|
||||||
options { id name }
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
`;
|
|
||||||
const projectRes = await github.graphql(projectQuery, {
|
|
||||||
owner: OWNER,
|
|
||||||
number: PROJECT_NUMBER
|
|
||||||
});
|
|
||||||
const project = projectRes.organization.projectV2;
|
|
||||||
if (!project) {
|
|
||||||
throw new Error(`Could not find Main project #${PROJECT_NUMBER}.`);
|
|
||||||
}
|
|
||||||
|
|
||||||
const statusField = project.fields.nodes.find(({ name }) => name === 'Status');
|
|
||||||
const statusOption = statusField?.options.find(({ name }) => name === targetStatus);
|
|
||||||
if (!statusField || !statusOption) {
|
|
||||||
throw new Error(`Could not find Status option "${targetStatus}" in Main.`);
|
|
||||||
}
|
|
||||||
|
|
||||||
const issueProjectItemsQuery = `
|
|
||||||
query($issueId: ID!) {
|
|
||||||
node(id: $issueId) {
|
|
||||||
... on Issue {
|
|
||||||
projectItems(first: 100) {
|
|
||||||
nodes {
|
|
||||||
id
|
|
||||||
project { id }
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
`;
|
|
||||||
const issueProjectItemsRes = await github.graphql(issueProjectItemsQuery, {
|
|
||||||
issueId: contentId
|
|
||||||
});
|
|
||||||
const projectItems = issueProjectItemsRes.node?.projectItems?.nodes || [];
|
|
||||||
const projectItem = projectItems.find(
|
|
||||||
({ project: itemProject }) => itemProject.id === project.id
|
|
||||||
);
|
|
||||||
if (!projectItem) {
|
|
||||||
console.log(`Issue #${issue.number} is not in Main project #${PROJECT_NUMBER}; skipping status update.`);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
const updateStatusMutation = `
|
|
||||||
mutation($projectId: ID!, $itemId: ID!, $fieldId: ID!, $optionId: String!) {
|
|
||||||
updateProjectV2ItemFieldValue(input: {
|
|
||||||
projectId: $projectId
|
|
||||||
itemId: $itemId
|
|
||||||
fieldId: $fieldId
|
|
||||||
value: { singleSelectOptionId: $optionId }
|
|
||||||
}) {
|
|
||||||
projectV2Item { id }
|
|
||||||
}
|
|
||||||
}
|
|
||||||
`;
|
|
||||||
await github.graphql(updateStatusMutation, {
|
|
||||||
projectId: project.id,
|
|
||||||
itemId: projectItem.id,
|
|
||||||
fieldId: statusField.id,
|
|
||||||
optionId: statusOption.id
|
|
||||||
});
|
|
||||||
console.log(`Issue #${issue.number} closed; Main status set to "${targetStatus}".`);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Define your labels here
|
// Define your labels here
|
||||||
const labelToApply = 'needs triage';
|
const labelToApply = 'needs triage';
|
||||||
|
|
||||||
@ -185,65 +71,6 @@ jobs:
|
|||||||
}
|
}
|
||||||
`;
|
`;
|
||||||
|
|
||||||
let itemId;
|
await github.graphql(addToProjectMutation, { projectId, contentId });
|
||||||
try {
|
|
||||||
const addRes = await github.graphql(addToProjectMutation, { projectId, contentId });
|
|
||||||
itemId = addRes.addProjectV2ItemById.item.id;
|
|
||||||
} catch (error) {
|
|
||||||
const messages = [
|
|
||||||
error && error.message,
|
|
||||||
...(Array.isArray(error && error.errors)
|
|
||||||
? error.errors.map(e => e && e.message)
|
|
||||||
: [])
|
|
||||||
].filter(Boolean).join('\n');
|
|
||||||
if (!messages.includes('Content already exists in this project')) {
|
|
||||||
throw error;
|
|
||||||
}
|
|
||||||
console.log('Item already in project, fetching existing item...');
|
|
||||||
const existing = await github.graphql(
|
|
||||||
`query($contentId: ID!) {
|
|
||||||
node(id: $contentId) {
|
|
||||||
... on Issue {
|
|
||||||
projectItems(first: 50) {
|
|
||||||
nodes { id project { id } }
|
|
||||||
}
|
|
||||||
}
|
|
||||||
... on PullRequest {
|
|
||||||
projectItems(first: 50) {
|
|
||||||
nodes { id project { id } }
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}`,
|
|
||||||
{ contentId }
|
|
||||||
);
|
|
||||||
const nodes = (existing.node && existing.node.projectItems && existing.node.projectItems.nodes) || [];
|
|
||||||
const match = nodes.find(n => n.project.id === projectId);
|
|
||||||
if (!match) {
|
|
||||||
throw new Error(`Item already in a project, but not found in project ${projectId}`);
|
|
||||||
}
|
|
||||||
itemId = match.id;
|
|
||||||
}
|
|
||||||
|
|
||||||
// === 4. SET PROJECT STATUS ===
|
|
||||||
const statusOptionId = isPR ? STATUS_TECH_REVIEW : STATUS_BACKLOG;
|
|
||||||
const statusName = isPR ? 'Tech Review' : 'Backlog';
|
|
||||||
console.log(`Setting status to "${statusName}" for ${isPR ? 'PR' : 'Issue'} #${issueNumber}...`);
|
|
||||||
await github.graphql(
|
|
||||||
`mutation($projectId: ID!, $itemId: ID!, $fieldId: ID!, $optionId: String!) {
|
|
||||||
updateProjectV2ItemFieldValue(input: {
|
|
||||||
projectId: $projectId,
|
|
||||||
itemId: $itemId,
|
|
||||||
fieldId: $fieldId,
|
|
||||||
value: { singleSelectOptionId: $optionId }
|
|
||||||
}) {
|
|
||||||
projectV2Item {
|
|
||||||
id
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}`,
|
|
||||||
{ projectId, itemId, fieldId: STATUS_FIELD_ID, optionId: statusOptionId }
|
|
||||||
);
|
|
||||||
|
|
||||||
console.log("Automation successfully completed!");
|
console.log("Automation successfully completed!");
|
||||||
|
|
||||||
|
|||||||
31
.github/workflows/build-adhoc.yml
vendored
31
.github/workflows/build-adhoc.yml
vendored
@ -20,50 +20,25 @@ on:
|
|||||||
required: false
|
required: false
|
||||||
default: false
|
default: false
|
||||||
|
|
||||||
permissions: {}
|
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
build-bundle:
|
build-bundle:
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
uses: ./.github/workflows/build-bundle.yml
|
uses: ./.github/workflows/build-bundle.yml
|
||||||
secrets:
|
secrets: inherit
|
||||||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
|
||||||
AWS_REGION: ${{ secrets.AWS_REGION }}
|
|
||||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
|
||||||
MATTERMOST_WEBHOOK: ${{ secrets.MATTERMOST_WEBHOOK }}
|
|
||||||
S3_BUCKET: ${{ secrets.S3_BUCKET }}
|
|
||||||
with:
|
with:
|
||||||
gh_ref: ${{ inputs.gh_ref }}
|
gh_ref: ${{ inputs.gh_ref }}
|
||||||
force: ${{ inputs.force }}
|
force: ${{ inputs.force }}
|
||||||
|
|
||||||
build-docker:
|
build-docker:
|
||||||
needs: build-bundle
|
needs: build-bundle
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
uses: ./.github/workflows/build-docker.yml
|
uses: ./.github/workflows/build-docker.yml
|
||||||
secrets:
|
secrets: inherit
|
||||||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
|
||||||
AWS_REGION: ${{ secrets.AWS_REGION }}
|
|
||||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
|
||||||
DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
|
|
||||||
DOCKER_REGISTRY: ${{ secrets.DOCKER_REGISTRY }}
|
|
||||||
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
|
|
||||||
MATTERMOST_WEBHOOK: ${{ secrets.MATTERMOST_WEBHOOK }}
|
|
||||||
PUB_DOCKER_PASSWORD: ${{ secrets.PUB_DOCKER_PASSWORD }}
|
|
||||||
PUB_DOCKER_USERNAME: ${{ secrets.PUB_DOCKER_USERNAME }}
|
|
||||||
S3_BUCKET: ${{ secrets.S3_BUCKET }}
|
|
||||||
with:
|
with:
|
||||||
gh_ref: ${{ inputs.gh_ref }}
|
gh_ref: ${{ inputs.gh_ref }}
|
||||||
# Pin build-docker to the commit build-bundle actually bundled.
|
|
||||||
sha: ${{ needs.build-bundle.outputs.sha }}
|
|
||||||
force: ${{ inputs.force }}
|
force: ${{ inputs.force }}
|
||||||
|
|
||||||
build-docker-admin-console:
|
build-docker-admin-console:
|
||||||
permissions: {}
|
|
||||||
uses: ./.github/workflows/build-docker-admin-console.yml
|
uses: ./.github/workflows/build-docker-admin-console.yml
|
||||||
secrets:
|
secrets: inherit
|
||||||
ORG_WORKFLOW_TOKEN: ${{ secrets.ORG_WORKFLOW_TOKEN }}
|
|
||||||
with:
|
with:
|
||||||
gh_ref: ${{ inputs.nitrate_ref || inputs.gh_ref }}
|
gh_ref: ${{ inputs.nitrate_ref || inputs.gh_ref }}
|
||||||
force: ${{ inputs.force }}
|
force: ${{ inputs.force }}
|
||||||
|
|||||||
88
.github/workflows/build-bundle.yml
vendored
88
.github/workflows/build-bundle.yml
vendored
@ -20,26 +20,12 @@ on:
|
|||||||
description: 'Name of the branch or ref'
|
description: 'Name of the branch or ref'
|
||||||
type: string
|
type: string
|
||||||
required: true
|
required: true
|
||||||
|
default: 'develop'
|
||||||
force:
|
force:
|
||||||
description: 'Rebuild and overwrite even if this version already exists in S3'
|
description: 'Rebuild and overwrite even if this version already exists in S3'
|
||||||
type: boolean
|
type: boolean
|
||||||
required: false
|
required: false
|
||||||
default: false
|
default: false
|
||||||
outputs:
|
|
||||||
sha:
|
|
||||||
description: 'Bundled commit, full 40-char SHA.'
|
|
||||||
value: ${{ jobs.check.outputs.sha }}
|
|
||||||
secrets:
|
|
||||||
AWS_ACCESS_KEY_ID:
|
|
||||||
required: false
|
|
||||||
AWS_REGION:
|
|
||||||
required: false
|
|
||||||
AWS_SECRET_ACCESS_KEY:
|
|
||||||
required: false
|
|
||||||
MATTERMOST_WEBHOOK:
|
|
||||||
required: false
|
|
||||||
S3_BUCKET:
|
|
||||||
required: false
|
|
||||||
|
|
||||||
# Literal group name: under `workflow_call`, `github.workflow` resolves to the
|
# Literal group name: under `workflow_call`, `github.workflow` resolves to the
|
||||||
# caller's workflow, which put this workflow and the other reusable one called
|
# caller's workflow, which put this workflow and the other reusable one called
|
||||||
@ -49,9 +35,6 @@ concurrency:
|
|||||||
group: build-bundle-${{ inputs.gh_ref }}
|
group: build-bundle-${{ inputs.gh_ref }}
|
||||||
cancel-in-progress: true
|
cancel-in-progress: true
|
||||||
|
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
# ── 1. Decide whether there is anything to build ───────────────────────
|
# ── 1. Decide whether there is anything to build ───────────────────────
|
||||||
check:
|
check:
|
||||||
@ -61,76 +44,57 @@ jobs:
|
|||||||
outputs:
|
outputs:
|
||||||
gh_ref: ${{ steps.vars.outputs.gh_ref }}
|
gh_ref: ${{ steps.vars.outputs.gh_ref }}
|
||||||
bundle_version: ${{ steps.vars.outputs.bundle_version }}
|
bundle_version: ${{ steps.vars.outputs.bundle_version }}
|
||||||
build_key: ${{ steps.vars.outputs.build_key }}
|
|
||||||
short_sha: ${{ steps.vars.outputs.short_sha }}
|
|
||||||
sha: ${{ steps.vars.outputs.sha }}
|
sha: ${{ steps.vars.outputs.sha }}
|
||||||
commit_title: ${{ steps.vars.outputs.commit_title }}
|
commit_title: ${{ steps.vars.outputs.commit_title }}
|
||||||
exists: ${{ steps.check.outputs.exists }}
|
exists: ${{ steps.check.outputs.exists }}
|
||||||
|
|
||||||
steps:
|
steps:
|
||||||
- name: Checkout repository
|
- name: Checkout repository
|
||||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
uses: actions/checkout@v6
|
||||||
with:
|
with:
|
||||||
persist-credentials: false
|
|
||||||
fetch-depth: 0
|
fetch-depth: 0
|
||||||
ref: ${{ inputs.gh_ref }}
|
ref: ${{ inputs.gh_ref }}
|
||||||
|
|
||||||
- name: Extract some useful variables
|
- name: Extract some useful variables
|
||||||
id: vars
|
id: vars
|
||||||
env:
|
|
||||||
GH_REF: ${{ inputs.gh_ref || github.ref_name }}
|
|
||||||
run: |
|
run: |
|
||||||
# Same command the build uses to stamp version.txt.
|
echo "gh_ref=${{ inputs.gh_ref || github.ref_name }}" >> $GITHUB_OUTPUT
|
||||||
BUNDLE_VERSION=$(./manage.sh version)
|
echo "bundle_version=$(git describe --tags --always)" >> $GITHUB_OUTPUT
|
||||||
SHORT_SHA=$(git rev-parse --short=12 HEAD)
|
echo "sha=$(git rev-parse --short=12 HEAD)" >> $GITHUB_OUTPUT
|
||||||
# The key takes only the closest tag (--abbrev=0): the sha already
|
|
||||||
# pins the commit, and the full describe output is not stable
|
|
||||||
# across checkouts (its -g<sha> suffix length depends on the
|
|
||||||
# local object count). Must match build-docker.yml.
|
|
||||||
VERSION_TAG=$(git describe --tags --match '*.*.*' --abbrev=0)
|
|
||||||
echo "gh_ref=$GH_REF" >> $GITHUB_OUTPUT
|
|
||||||
echo "bundle_version=$BUNDLE_VERSION" >> $GITHUB_OUTPUT
|
|
||||||
echo "build_key=sha-${SHORT_SHA}-${VERSION_TAG}" >> $GITHUB_OUTPUT
|
|
||||||
echo "short_sha=$SHORT_SHA" >> $GITHUB_OUTPUT
|
|
||||||
echo "sha=$(git rev-parse HEAD)" >> $GITHUB_OUTPUT
|
|
||||||
echo "commit_title=$(git log -1 --pretty=%s)" >> $GITHUB_OUTPUT
|
echo "commit_title=$(git log -1 --pretty=%s)" >> $GITHUB_OUTPUT
|
||||||
|
|
||||||
# Keyed by commit sha AND version, same as build-docker.yml's marker
|
# The uploaded zip carries its version as S3 metadata. If the
|
||||||
# check, so both pipelines agree on what "already built" means. The
|
# existing object was already built from this same commit, the
|
||||||
# version is stamped into the bundle, so a new tag on an already
|
# whole build job is skipped. `force` bypasses this check entirely.
|
||||||
# bundled commit (2.18.1-RC1 -> 2.18.1) must produce a new bundle.
|
|
||||||
# `force` bypasses this check entirely.
|
|
||||||
- name: Check if this bundle is already built
|
- name: Check if this bundle is already built
|
||||||
id: check
|
id: check
|
||||||
env:
|
env:
|
||||||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
||||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
||||||
AWS_DEFAULT_REGION: ${{ secrets.AWS_REGION }}
|
AWS_DEFAULT_REGION: ${{ secrets.AWS_REGION }}
|
||||||
S3_BUCKET: ${{ secrets.S3_BUCKET }}
|
|
||||||
FORCE: ${{ inputs.force }}
|
|
||||||
BUILD_KEY: ${{ steps.vars.outputs.build_key }}
|
|
||||||
run: |
|
run: |
|
||||||
if [ "$FORCE" = "true" ]; then
|
if [ "${{ inputs.force }}" = "true" ]; then
|
||||||
echo "exists=false" >> $GITHUB_OUTPUT
|
echo "exists=false" >> $GITHUB_OUTPUT
|
||||||
{
|
{
|
||||||
echo "### 🔁 Bundle build forced"
|
echo "### 🔁 Bundle build forced"
|
||||||
echo ""
|
echo ""
|
||||||
echo "\`force: true\` — skipping the S3 sha check."
|
echo "\`force: true\` — skipping the S3 version check."
|
||||||
} >> "$GITHUB_STEP_SUMMARY"
|
} >> "$GITHUB_STEP_SUMMARY"
|
||||||
exit 0
|
exit 0
|
||||||
fi
|
fi
|
||||||
|
|
||||||
BUNDLE_EXISTS=$(aws s3api head-object \
|
EXISTING_VERSION=$(aws s3api head-object \
|
||||||
--bucket "$S3_BUCKET" \
|
--bucket ${{ secrets.S3_BUCKET }} \
|
||||||
--key "penpot-${BUILD_KEY}.zip" \
|
--key "penpot-${{ steps.vars.outputs.gh_ref }}.zip" \
|
||||||
> /dev/null 2>&1 && echo "true" || echo "false")
|
--query 'Metadata."bundle-version"' \
|
||||||
|
--output text 2>/dev/null || echo "none")
|
||||||
|
|
||||||
if [ "$BUNDLE_EXISTS" = "true" ]; then
|
if [ "$EXISTING_VERSION" = "${{ steps.vars.outputs.bundle_version }}" ]; then
|
||||||
echo "exists=true" >> $GITHUB_OUTPUT
|
echo "exists=true" >> $GITHUB_OUTPUT
|
||||||
{
|
{
|
||||||
echo "### ⏭️ Bundle build skipped"
|
echo "### ⏭️ Bundle build skipped"
|
||||||
echo ""
|
echo ""
|
||||||
echo "The bundle in S3 was already built for \`${BUILD_KEY}\`."
|
echo "The bundle in S3 was already built from \`${{ steps.vars.outputs.bundle_version }}\`."
|
||||||
} >> "$GITHUB_STEP_SUMMARY"
|
} >> "$GITHUB_STEP_SUMMARY"
|
||||||
else
|
else
|
||||||
echo "exists=false" >> $GITHUB_OUTPUT
|
echo "exists=false" >> $GITHUB_OUTPUT
|
||||||
@ -146,9 +110,8 @@ jobs:
|
|||||||
|
|
||||||
steps:
|
steps:
|
||||||
- name: Checkout repository
|
- name: Checkout repository
|
||||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
uses: actions/checkout@v6
|
||||||
with:
|
with:
|
||||||
persist-credentials: false
|
|
||||||
fetch-depth: 0
|
fetch-depth: 0
|
||||||
ref: ${{ inputs.gh_ref }}
|
ref: ${{ inputs.gh_ref }}
|
||||||
|
|
||||||
@ -173,25 +136,18 @@ jobs:
|
|||||||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
||||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
||||||
AWS_DEFAULT_REGION: ${{ secrets.AWS_REGION }}
|
AWS_DEFAULT_REGION: ${{ secrets.AWS_REGION }}
|
||||||
S3_BUCKET: ${{ secrets.S3_BUCKET }}
|
|
||||||
BUILD_KEY: ${{ needs.check.outputs.build_key }}
|
|
||||||
BUNDLE_VERSION: ${{ needs.check.outputs.bundle_version }}
|
|
||||||
run: |
|
run: |
|
||||||
aws s3 cp zips/penpot.zip \
|
aws s3 cp zips/penpot.zip \
|
||||||
"s3://${S3_BUCKET}/penpot-${BUILD_KEY}.zip" \
|
s3://${{ secrets.S3_BUCKET }}/penpot-${{ needs.check.outputs.gh_ref }}.zip \
|
||||||
--metadata "bundle-version=${BUNDLE_VERSION}"
|
--metadata bundle-version=${{ needs.check.outputs.bundle_version }}
|
||||||
|
|
||||||
- name: Write step summary
|
- name: Write step summary
|
||||||
env:
|
|
||||||
SHORT_SHA: ${{ needs.check.outputs.short_sha }}
|
|
||||||
BUNDLE_VERSION: ${{ needs.check.outputs.bundle_version }}
|
|
||||||
COMMIT_TITLE: ${{ needs.check.outputs.commit_title }}
|
|
||||||
run: |
|
run: |
|
||||||
{
|
{
|
||||||
echo "### ✅ Bundle built"
|
echo "### ✅ Bundle built"
|
||||||
echo ""
|
echo ""
|
||||||
echo "- Version: \`${BUNDLE_VERSION}\` (\`./manage.sh version\`)"
|
echo "- Version: \`${{ needs.check.outputs.bundle_version }}\` (\`git describe --tags --always\`)"
|
||||||
echo "- Commit: [\`${SHORT_SHA}\`](${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/commit/${SHORT_SHA}) — ${COMMIT_TITLE}"
|
echo "- Commit: [\`${{ needs.check.outputs.sha }}\`](https://github.com/${{ github.repository }}/commit/${{ needs.check.outputs.sha }}) — ${{ needs.check.outputs.commit_title }}"
|
||||||
echo "- Built at: $(date -u +'%Y-%m-%d %H:%M:%S UTC')"
|
echo "- Built at: $(date -u +'%Y-%m-%d %H:%M:%S UTC')"
|
||||||
} >> "$GITHUB_STEP_SUMMARY"
|
} >> "$GITHUB_STEP_SUMMARY"
|
||||||
|
|
||||||
|
|||||||
31
.github/workflows/build-develop.yml
vendored
31
.github/workflows/build-develop.yml
vendored
@ -18,50 +18,25 @@ concurrency:
|
|||||||
group: ${{ github.workflow }}
|
group: ${{ github.workflow }}
|
||||||
cancel-in-progress: true
|
cancel-in-progress: true
|
||||||
|
|
||||||
permissions: {}
|
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
build-bundle:
|
build-bundle:
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
uses: ./.github/workflows/build-bundle.yml
|
uses: ./.github/workflows/build-bundle.yml
|
||||||
secrets:
|
secrets: inherit
|
||||||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
|
||||||
AWS_REGION: ${{ secrets.AWS_REGION }}
|
|
||||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
|
||||||
MATTERMOST_WEBHOOK: ${{ secrets.MATTERMOST_WEBHOOK }}
|
|
||||||
S3_BUCKET: ${{ secrets.S3_BUCKET }}
|
|
||||||
with:
|
with:
|
||||||
gh_ref: "develop"
|
gh_ref: "develop"
|
||||||
force: ${{ inputs.force || false }}
|
force: ${{ inputs.force || false }}
|
||||||
|
|
||||||
build-docker:
|
build-docker:
|
||||||
needs: build-bundle
|
needs: build-bundle
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
uses: ./.github/workflows/build-docker.yml
|
uses: ./.github/workflows/build-docker.yml
|
||||||
secrets:
|
secrets: inherit
|
||||||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
|
||||||
AWS_REGION: ${{ secrets.AWS_REGION }}
|
|
||||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
|
||||||
DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
|
|
||||||
DOCKER_REGISTRY: ${{ secrets.DOCKER_REGISTRY }}
|
|
||||||
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
|
|
||||||
MATTERMOST_WEBHOOK: ${{ secrets.MATTERMOST_WEBHOOK }}
|
|
||||||
PUB_DOCKER_PASSWORD: ${{ secrets.PUB_DOCKER_PASSWORD }}
|
|
||||||
PUB_DOCKER_USERNAME: ${{ secrets.PUB_DOCKER_USERNAME }}
|
|
||||||
S3_BUCKET: ${{ secrets.S3_BUCKET }}
|
|
||||||
with:
|
with:
|
||||||
gh_ref: "develop"
|
gh_ref: "develop"
|
||||||
# Pin build-docker to the commit build-bundle actually bundled.
|
|
||||||
sha: ${{ needs.build-bundle.outputs.sha }}
|
|
||||||
force: ${{ inputs.force || false }}
|
force: ${{ inputs.force || false }}
|
||||||
|
|
||||||
build-docker-admin-console:
|
build-docker-admin-console:
|
||||||
permissions: {}
|
|
||||||
uses: ./.github/workflows/build-docker-admin-console.yml
|
uses: ./.github/workflows/build-docker-admin-console.yml
|
||||||
secrets:
|
secrets: inherit
|
||||||
ORG_WORKFLOW_TOKEN: ${{ secrets.ORG_WORKFLOW_TOKEN }}
|
|
||||||
with:
|
with:
|
||||||
gh_ref: "develop"
|
gh_ref: "develop"
|
||||||
force: ${{ inputs.force || false }}
|
force: ${{ inputs.force || false }}
|
||||||
|
|||||||
22
.github/workflows/build-docker-admin-console.yml
vendored
22
.github/workflows/build-docker-admin-console.yml
vendored
@ -39,8 +39,6 @@ on:
|
|||||||
description: 'Token with Actions write access on penpot-nitrate'
|
description: 'Token with Actions write access on penpot-nitrate'
|
||||||
required: true
|
required: true
|
||||||
|
|
||||||
permissions: {}
|
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
build-nitrate-docker:
|
build-nitrate-docker:
|
||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
@ -50,23 +48,22 @@ jobs:
|
|||||||
WORKFLOW: build-docker-admin-console.yml
|
WORKFLOW: build-docker-admin-console.yml
|
||||||
GH_REF: ${{ inputs.gh_ref }}
|
GH_REF: ${{ inputs.gh_ref }}
|
||||||
DISPATCH_REF: ${{ inputs.dispatch_ref }}
|
DISPATCH_REF: ${{ inputs.dispatch_ref }}
|
||||||
FORCE: ${{ inputs.force }}
|
|
||||||
steps:
|
steps:
|
||||||
- name: Trigger nitrate docker build
|
- name: Trigger nitrate docker build
|
||||||
id: dispatch
|
id: dispatch
|
||||||
run: |
|
run: |
|
||||||
DISTINCT_ID="${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
|
DISTINCT_ID="${{ github.run_id }}-${{ github.run_attempt }}"
|
||||||
CALLER_URL="${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
|
CALLER_URL="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
|
||||||
|
|
||||||
gh workflow run "$WORKFLOW" --repo "$REPO" --ref "$DISPATCH_REF" \
|
gh workflow run "$WORKFLOW" --repo "$REPO" --ref "$DISPATCH_REF" \
|
||||||
-f gh_ref="$GH_REF" \
|
-f gh_ref="$GH_REF" \
|
||||||
-f force="$FORCE" \
|
-f force="${{ inputs.force }}" \
|
||||||
-f caller_run_id="$DISTINCT_ID" \
|
-f caller_run_id="$DISTINCT_ID" \
|
||||||
-f caller_run_url="$CALLER_URL"
|
-f caller_run_url="$CALLER_URL"
|
||||||
|
|
||||||
# Locate the dispatched run using the correlation id embedded in its run-name
|
# Locate the dispatched run using the correlation id embedded in its run-name
|
||||||
RUN_ID=""
|
RUN_ID=""
|
||||||
for _ in $(seq 1 24); do
|
for i in $(seq 1 24); do
|
||||||
sleep 5
|
sleep 5
|
||||||
RUN_ID=$(gh run list --repo "$REPO" --workflow "$WORKFLOW" \
|
RUN_ID=$(gh run list --repo "$REPO" --workflow "$WORKFLOW" \
|
||||||
--limit 10 --json databaseId,displayTitle \
|
--limit 10 --json databaseId,displayTitle \
|
||||||
@ -86,25 +83,20 @@ jobs:
|
|||||||
echo "::notice title=Nitrate docker build::$RUN_URL"
|
echo "::notice title=Nitrate docker build::$RUN_URL"
|
||||||
|
|
||||||
- name: Wait for nitrate docker build
|
- name: Wait for nitrate docker build
|
||||||
env:
|
|
||||||
RUN_ID: ${{ steps.dispatch.outputs.run_id }}
|
|
||||||
run: |
|
run: |
|
||||||
gh run watch "$RUN_ID" \
|
gh run watch "${{ steps.dispatch.outputs.run_id }}" \
|
||||||
--repo "$REPO" \
|
--repo "$REPO" \
|
||||||
--interval 30 \
|
--interval 30 \
|
||||||
--exit-status
|
--exit-status
|
||||||
|
|
||||||
- name: Report result
|
- name: Report result
|
||||||
if: always() && steps.dispatch.outputs.run_id != ''
|
if: always() && steps.dispatch.outputs.run_id != ''
|
||||||
env:
|
|
||||||
RUN_ID: ${{ steps.dispatch.outputs.run_id }}
|
|
||||||
RUN_URL: ${{ steps.dispatch.outputs.run_url }}
|
|
||||||
run: |
|
run: |
|
||||||
CONCLUSION=$(gh run view "$RUN_ID" \
|
CONCLUSION=$(gh run view "${{ steps.dispatch.outputs.run_id }}" \
|
||||||
--repo "$REPO" --json conclusion --jq '.conclusion')
|
--repo "$REPO" --json conclusion --jq '.conclusion')
|
||||||
{
|
{
|
||||||
echo "### 🐳 Nitrate docker build"
|
echo "### 🐳 Nitrate docker build"
|
||||||
echo ""
|
echo ""
|
||||||
echo "- Result: \`${CONCLUSION:-in_progress}\`"
|
echo "- Result: \`${CONCLUSION:-in_progress}\`"
|
||||||
echo "- Run: ${RUN_URL}"
|
echo "- Run: ${{ steps.dispatch.outputs.run_url }}"
|
||||||
} >> "$GITHUB_STEP_SUMMARY"
|
} >> "$GITHUB_STEP_SUMMARY"
|
||||||
|
|||||||
44
.github/workflows/build-docker-devenv.yml
vendored
44
.github/workflows/build-docker-devenv.yml
vendored
@ -3,55 +3,40 @@ name: DevEnv Docker Image Builder
|
|||||||
on:
|
on:
|
||||||
workflow_dispatch:
|
workflow_dispatch:
|
||||||
|
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
|
|
||||||
concurrency:
|
|
||||||
group: build-docker-devenv
|
|
||||||
cancel-in-progress: false
|
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
build-and-push:
|
build-and-push:
|
||||||
name: Build and push DevEnv Docker image
|
name: Build and push DevEnv Docker image
|
||||||
runs-on: penpot-standar-runner
|
runs-on: penpot-extended-runner
|
||||||
timeout-minutes: 90
|
|
||||||
env:
|
|
||||||
DOCKER_IMAGE: 'penpotapp/devenv'
|
|
||||||
|
|
||||||
steps:
|
steps:
|
||||||
- name: Set common environment variables
|
- name: Set common environment variables
|
||||||
run: |
|
run: |
|
||||||
# Each job execution will use its own docker configuration.
|
# Each job execution will use its own docker configuration.
|
||||||
echo "DOCKER_CONFIG=${RUNNER_TEMP}/.docker-${GITHUB_RUN_ID}-${GITHUB_JOB}" >> "$GITHUB_ENV"
|
echo "DOCKER_CONFIG=${{ runner.temp }}/.docker-${{ github.run_id }}-${{ github.job }}" >> $GITHUB_ENV
|
||||||
|
|
||||||
- name: Checkout code
|
- name: Checkout code
|
||||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
uses: actions/checkout@v6
|
||||||
with:
|
|
||||||
persist-credentials: false
|
|
||||||
|
|
||||||
- name: Set up QEMU
|
|
||||||
uses: docker/setup-qemu-action@99012661954931238ded8c8b007157a8430204e1 # v4.4.0
|
|
||||||
with:
|
|
||||||
platforms: linux/amd64,linux/arm64
|
|
||||||
|
|
||||||
- name: Set up Docker Buildx
|
- name: Set up Docker Buildx
|
||||||
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
|
uses: docker/setup-buildx-action@v4
|
||||||
|
|
||||||
- name: Login to Docker Registry (push destination)
|
- name: Login to Docker Registry (push destination)
|
||||||
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
|
uses: docker/login-action@v4
|
||||||
with:
|
with:
|
||||||
username: ${{ secrets.PUB_DOCKER_USERNAME }}
|
username: ${{ secrets.PUB_DOCKER_USERNAME }}
|
||||||
password: ${{ secrets.PUB_DOCKER_PASSWORD }}
|
password: ${{ secrets.PUB_DOCKER_PASSWORD }}
|
||||||
|
|
||||||
- name: Login to Docker Hardened Images registry (base image pull)
|
- name: Login to Docker Hardened Images registry (base image pull)
|
||||||
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
|
uses: docker/login-action@v4
|
||||||
with:
|
with:
|
||||||
registry: dhi.io
|
registry: dhi.io
|
||||||
username: ${{ secrets.PUB_DOCKER_USERNAME }}
|
username: ${{ secrets.PUB_DOCKER_USERNAME }}
|
||||||
password: ${{ secrets.PUB_DOCKER_PASSWORD }}
|
password: ${{ secrets.PUB_DOCKER_PASSWORD }}
|
||||||
|
|
||||||
- name: Build and push DevEnv Docker image
|
- name: Build and push DevEnv Docker image
|
||||||
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
|
uses: docker/build-push-action@v7
|
||||||
|
env:
|
||||||
|
DOCKER_IMAGE: 'penpotapp/devenv'
|
||||||
with:
|
with:
|
||||||
context: ./docker/devenv/
|
context: ./docker/devenv/
|
||||||
file: ./docker/devenv/Dockerfile
|
file: ./docker/devenv/Dockerfile
|
||||||
@ -71,13 +56,4 @@ jobs:
|
|||||||
TEXT: |
|
TEXT: |
|
||||||
🚀 *[PENPOT] New devenv available*
|
🚀 *[PENPOT] New devenv available*
|
||||||
📄 You may want to update your devenv.
|
📄 You may want to update your devenv.
|
||||||
|
@alvaro
|
||||||
- name: Notify Mattermost (developers)
|
|
||||||
uses: mattermost/action-mattermost-notify@ae31bb6f9e26a54336e79696f108a2c91cf55b4e # v2.1.0
|
|
||||||
with:
|
|
||||||
MATTERMOST_WEBHOOK_URL: ${{ secrets.MATTERMOST_WEBHOOK }}
|
|
||||||
MATTERMOST_CHANNEL: penpotdev
|
|
||||||
TEXT: |
|
|
||||||
🚀 *[PENPOT] New devenv available*
|
|
||||||
📄 You may want to update your devenv.
|
|
||||||
(\cc @alvaro)
|
|
||||||
|
|||||||
217
.github/workflows/build-docker.yml
vendored
217
.github/workflows/build-docker.yml
vendored
@ -19,40 +19,12 @@ on:
|
|||||||
description: 'Name of the branch or ref'
|
description: 'Name of the branch or ref'
|
||||||
type: string
|
type: string
|
||||||
required: true
|
required: true
|
||||||
sha:
|
default: 'develop'
|
||||||
description: >-
|
|
||||||
Exact commit to check out (full 40-char SHA, e.g. from
|
|
||||||
build-bundle.yml). Falls back to gh_ref when empty; gh_ref
|
|
||||||
always still names the branch tag to move.
|
|
||||||
type: string
|
|
||||||
required: false
|
|
||||||
default: ''
|
|
||||||
force:
|
force:
|
||||||
description: 'Rebuild and overwrite even if this sha is already promoted'
|
description: 'Rebuild and overwrite even if this sha is already promoted'
|
||||||
type: boolean
|
type: boolean
|
||||||
required: false
|
required: false
|
||||||
default: false
|
default: false
|
||||||
secrets:
|
|
||||||
AWS_ACCESS_KEY_ID:
|
|
||||||
required: false
|
|
||||||
AWS_REGION:
|
|
||||||
required: false
|
|
||||||
AWS_SECRET_ACCESS_KEY:
|
|
||||||
required: false
|
|
||||||
DOCKER_PASSWORD:
|
|
||||||
required: false
|
|
||||||
DOCKER_REGISTRY:
|
|
||||||
required: false
|
|
||||||
DOCKER_USERNAME:
|
|
||||||
required: false
|
|
||||||
MATTERMOST_WEBHOOK:
|
|
||||||
required: false
|
|
||||||
PUB_DOCKER_PASSWORD:
|
|
||||||
required: false
|
|
||||||
PUB_DOCKER_USERNAME:
|
|
||||||
required: false
|
|
||||||
S3_BUCKET:
|
|
||||||
required: false
|
|
||||||
|
|
||||||
# Literal group name: under `workflow_call`, `github.workflow` resolves to the
|
# Literal group name: under `workflow_call`, `github.workflow` resolves to the
|
||||||
# caller's workflow, which put this workflow and the other reusable one called
|
# caller's workflow, which put this workflow and the other reusable one called
|
||||||
@ -70,51 +42,43 @@ env:
|
|||||||
# missing (e.g. if runners ever move to separate machines).
|
# missing (e.g. if runners ever move to separate machines).
|
||||||
BUNDLE_CACHE: /var/tmp/penpot-bundle-cache
|
BUNDLE_CACHE: /var/tmp/penpot-bundle-cache
|
||||||
|
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
# ── 1. Resolve the build key and check the whole set at once ───────────
|
# ── 1. Resolve the build key and check the whole set at once ───────────
|
||||||
prepare:
|
prepare:
|
||||||
name: Prepare
|
name: Prepare
|
||||||
runs-on: penpot-standar-runner
|
runs-on: penpot-extended-runner
|
||||||
timeout-minutes: 15
|
timeout-minutes: 15
|
||||||
outputs:
|
outputs:
|
||||||
gh_ref: ${{ steps.vars.outputs.gh_ref }}
|
gh_ref: ${{ steps.vars.outputs.gh_ref }}
|
||||||
ref_slug: ${{ steps.vars.outputs.ref_slug }}
|
|
||||||
bundle_version: ${{ steps.vars.outputs.bundle_version }}
|
bundle_version: ${{ steps.vars.outputs.bundle_version }}
|
||||||
sha: ${{ steps.vars.outputs.sha }}
|
sha: ${{ steps.vars.outputs.sha }}
|
||||||
short_sha: ${{ steps.vars.outputs.short_sha }}
|
|
||||||
build_key: ${{ steps.vars.outputs.build_key }}
|
|
||||||
commit_title: ${{ steps.vars.outputs.commit_title }}
|
commit_title: ${{ steps.vars.outputs.commit_title }}
|
||||||
exists: ${{ steps.check.outputs.exists }}
|
exists: ${{ steps.check.outputs.exists }}
|
||||||
|
|
||||||
steps:
|
steps:
|
||||||
- name: Checkout code
|
- name: Checkout code
|
||||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
uses: actions/checkout@v6
|
||||||
with:
|
with:
|
||||||
persist-credentials: false
|
ref: ${{ inputs.gh_ref }}
|
||||||
fetch-depth: 0
|
|
||||||
ref: ${{ inputs.sha != '' && inputs.sha || inputs.gh_ref }}
|
|
||||||
|
|
||||||
- name: Extract some useful variables
|
- name: Extract some useful variables
|
||||||
id: vars
|
id: vars
|
||||||
env:
|
env:
|
||||||
GH_REF: ${{ inputs.gh_ref || github.ref_name }}
|
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
||||||
|
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
||||||
|
AWS_DEFAULT_REGION: ${{ secrets.AWS_REGION }}
|
||||||
run: |
|
run: |
|
||||||
# A git ref accepts characters that a Docker tag does not.
|
GH_REF="${{ inputs.gh_ref || github.ref_name }}"
|
||||||
REF_SLUG=$(printf '%s' "$GH_REF" | tr -c 'A-Za-z0-9._-' '-' | cut -c1-100)
|
|
||||||
echo "gh_ref=$GH_REF" >> $GITHUB_OUTPUT
|
echo "gh_ref=$GH_REF" >> $GITHUB_OUTPUT
|
||||||
echo "ref_slug=$REF_SLUG" >> $GITHUB_OUTPUT
|
echo "sha=$(git rev-parse --short=12 HEAD)" >> $GITHUB_OUTPUT
|
||||||
# Must match the key computed by build-bundle.yml.
|
|
||||||
BUNDLE_VERSION=$(./manage.sh version)
|
|
||||||
SHORT_SHA=$(git rev-parse --short=12 HEAD)
|
|
||||||
VERSION_TAG=$(git describe --tags --match '*.*.*' --abbrev=0)
|
|
||||||
echo "sha=$(git rev-parse HEAD)" >> $GITHUB_OUTPUT
|
|
||||||
echo "short_sha=$SHORT_SHA" >> $GITHUB_OUTPUT
|
|
||||||
echo "commit_title=$(git log -1 --pretty=%s)" >> $GITHUB_OUTPUT
|
echo "commit_title=$(git log -1 --pretty=%s)" >> $GITHUB_OUTPUT
|
||||||
|
|
||||||
|
BUNDLE_VERSION=$(aws s3api head-object \
|
||||||
|
--bucket ${{ secrets.S3_BUCKET }} \
|
||||||
|
--key "penpot-$GH_REF.zip" \
|
||||||
|
--query 'Metadata."bundle-version"' \
|
||||||
|
--output text)
|
||||||
echo "bundle_version=$BUNDLE_VERSION" >> $GITHUB_OUTPUT
|
echo "bundle_version=$BUNDLE_VERSION" >> $GITHUB_OUTPUT
|
||||||
echo "build_key=sha-${SHORT_SHA}-${VERSION_TAG}" >> $GITHUB_OUTPUT
|
|
||||||
|
|
||||||
# The image set is a single block, so a single set-level check is
|
# The image set is a single block, so a single set-level check is
|
||||||
# enough: `promote` drops a marker object in S3 only after every
|
# enough: `promote` drops a marker object in S3 only after every
|
||||||
@ -127,16 +91,13 @@ jobs:
|
|||||||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
||||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
||||||
AWS_DEFAULT_REGION: ${{ secrets.AWS_REGION }}
|
AWS_DEFAULT_REGION: ${{ secrets.AWS_REGION }}
|
||||||
S3_BUCKET: ${{ secrets.S3_BUCKET }}
|
|
||||||
FORCE: ${{ inputs.force }}
|
|
||||||
BUILD_KEY: ${{ steps.vars.outputs.build_key }}
|
|
||||||
run: |
|
run: |
|
||||||
if [ "$FORCE" = "true" ]; then
|
if [ "${{ inputs.force }}" = "true" ]; then
|
||||||
echo "exists=false" >> $GITHUB_OUTPUT
|
echo "exists=false" >> $GITHUB_OUTPUT
|
||||||
mkdir -p "$BUNDLE_CACHE"
|
mkdir -p "$BUNDLE_CACHE"
|
||||||
find "$BUNDLE_CACHE" -type f -mtime +1 -delete || true
|
find "$BUNDLE_CACHE" -type f -mtime +1 -delete || true
|
||||||
ZIP="$BUNDLE_CACHE/penpot-${BUILD_KEY}.zip"
|
ZIP="$BUNDLE_CACHE/penpot-${{ steps.vars.outputs.bundle_version }}.zip"
|
||||||
aws s3 cp "s3://${S3_BUCKET}/penpot-${BUILD_KEY}.zip" "$ZIP.$$.tmp"
|
aws s3 cp "s3://${{ secrets.S3_BUCKET }}/penpot-${{ steps.vars.outputs.gh_ref }}.zip" "$ZIP.$$.tmp"
|
||||||
mv "$ZIP.$$.tmp" "$ZIP"
|
mv "$ZIP.$$.tmp" "$ZIP"
|
||||||
{
|
{
|
||||||
echo "### 🔁 Image set build forced"
|
echo "### 🔁 Image set build forced"
|
||||||
@ -147,14 +108,14 @@ jobs:
|
|||||||
fi
|
fi
|
||||||
|
|
||||||
if aws s3api head-object \
|
if aws s3api head-object \
|
||||||
--bucket "$S3_BUCKET" \
|
--bucket ${{ secrets.S3_BUCKET }} \
|
||||||
--key "markers/images-${BUILD_KEY}" \
|
--key "markers/images-sha-${{ steps.vars.outputs.sha }}" \
|
||||||
> /dev/null 2>&1; then
|
> /dev/null 2>&1; then
|
||||||
echo "exists=true" >> $GITHUB_OUTPUT
|
echo "exists=true" >> $GITHUB_OUTPUT
|
||||||
{
|
{
|
||||||
echo "### ⏭️ Image set build skipped"
|
echo "### ⏭️ Image set build skipped"
|
||||||
echo ""
|
echo ""
|
||||||
echo "The whole set was already built and promoted for \`${BUILD_KEY}\`."
|
echo "The whole set was already built and promoted for \`sha-${{ steps.vars.outputs.sha }}\`."
|
||||||
} >> "$GITHUB_STEP_SUMMARY"
|
} >> "$GITHUB_STEP_SUMMARY"
|
||||||
else
|
else
|
||||||
echo "exists=false" >> $GITHUB_OUTPUT
|
echo "exists=false" >> $GITHUB_OUTPUT
|
||||||
@ -164,9 +125,9 @@ jobs:
|
|||||||
# prune stale bundles while at it.
|
# prune stale bundles while at it.
|
||||||
mkdir -p "$BUNDLE_CACHE"
|
mkdir -p "$BUNDLE_CACHE"
|
||||||
find "$BUNDLE_CACHE" -type f -mtime +1 -delete || true
|
find "$BUNDLE_CACHE" -type f -mtime +1 -delete || true
|
||||||
ZIP="$BUNDLE_CACHE/penpot-${BUILD_KEY}.zip"
|
ZIP="$BUNDLE_CACHE/penpot-${{ steps.vars.outputs.bundle_version }}.zip"
|
||||||
if [ ! -f "$ZIP" ]; then
|
if [ ! -f "$ZIP" ]; then
|
||||||
aws s3 cp "s3://${S3_BUCKET}/penpot-${BUILD_KEY}.zip" "$ZIP.$$.tmp"
|
aws s3 cp "s3://${{ secrets.S3_BUCKET }}/penpot-${{ steps.vars.outputs.gh_ref }}.zip" "$ZIP.$$.tmp"
|
||||||
mv "$ZIP.$$.tmp" "$ZIP"
|
mv "$ZIP.$$.tmp" "$ZIP"
|
||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
@ -174,7 +135,7 @@ jobs:
|
|||||||
# ── 2. One build per image, in parallel, only when needed ──────────────
|
# ── 2. One build per image, in parallel, only when needed ──────────────
|
||||||
build:
|
build:
|
||||||
name: Build ${{ matrix.image }}
|
name: Build ${{ matrix.image }}
|
||||||
runs-on: penpot-standar-runner
|
runs-on: penpot-extended-runner
|
||||||
timeout-minutes: 60
|
timeout-minutes: 60
|
||||||
needs: prepare
|
needs: prepare
|
||||||
if: needs.prepare.outputs.exists == 'false'
|
if: needs.prepare.outputs.exists == 'false'
|
||||||
@ -194,13 +155,12 @@ jobs:
|
|||||||
echo "DOCKER_CONFIG=${{ runner.temp }}/.docker-${{ github.run_id }}-${{ github.job }}-${{ matrix.image }}" >> $GITHUB_ENV
|
echo "DOCKER_CONFIG=${{ runner.temp }}/.docker-${{ github.run_id }}-${{ github.job }}-${{ matrix.image }}" >> $GITHUB_ENV
|
||||||
|
|
||||||
- name: Checkout code
|
- name: Checkout code
|
||||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
uses: actions/checkout@v6
|
||||||
with:
|
with:
|
||||||
persist-credentials: false
|
ref: ${{ inputs.gh_ref }}
|
||||||
ref: ${{ inputs.sha != '' && inputs.sha || inputs.gh_ref }}
|
|
||||||
|
|
||||||
- name: Login to Docker Registry
|
- name: Login to Docker Registry
|
||||||
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
|
uses: docker/login-action@v4
|
||||||
with:
|
with:
|
||||||
registry: ${{ secrets.DOCKER_REGISTRY }}
|
registry: ${{ secrets.DOCKER_REGISTRY }}
|
||||||
username: ${{ secrets.DOCKER_USERNAME }}
|
username: ${{ secrets.DOCKER_USERNAME }}
|
||||||
@ -210,7 +170,7 @@ jobs:
|
|||||||
# images from DockerHub for unregistered users.
|
# images from DockerHub for unregistered users.
|
||||||
# https://docs.docker.com/docker-hub/usage/
|
# https://docs.docker.com/docker-hub/usage/
|
||||||
- name: Login to DockerHub Registry
|
- name: Login to DockerHub Registry
|
||||||
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
|
uses: docker/login-action@v4
|
||||||
with:
|
with:
|
||||||
username: ${{ secrets.PUB_DOCKER_USERNAME }}
|
username: ${{ secrets.PUB_DOCKER_USERNAME }}
|
||||||
password: ${{ secrets.PUB_DOCKER_PASSWORD }}
|
password: ${{ secrets.PUB_DOCKER_PASSWORD }}
|
||||||
@ -222,7 +182,7 @@ jobs:
|
|||||||
# reuses the same PUB_DOCKER_* credentials as the DockerHub
|
# reuses the same PUB_DOCKER_* credentials as the DockerHub
|
||||||
# login above.
|
# login above.
|
||||||
- name: Login to Docker Hardened Images registry (base image pull)
|
- name: Login to Docker Hardened Images registry (base image pull)
|
||||||
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
|
uses: docker/login-action@v4
|
||||||
with:
|
with:
|
||||||
registry: dhi.io
|
registry: dhi.io
|
||||||
username: ${{ secrets.PUB_DOCKER_USERNAME }}
|
username: ${{ secrets.PUB_DOCKER_USERNAME }}
|
||||||
@ -236,41 +196,38 @@ jobs:
|
|||||||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
||||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
||||||
AWS_DEFAULT_REGION: ${{ secrets.AWS_REGION }}
|
AWS_DEFAULT_REGION: ${{ secrets.AWS_REGION }}
|
||||||
S3_BUCKET: ${{ secrets.S3_BUCKET }}
|
|
||||||
BUILD_KEY: ${{ needs.prepare.outputs.build_key }}
|
|
||||||
IMAGE: ${{ matrix.image }}
|
|
||||||
run: |
|
run: |
|
||||||
ZIP="$BUNDLE_CACHE/penpot-${BUILD_KEY}.zip"
|
ZIP="$BUNDLE_CACHE/penpot-${{ needs.prepare.outputs.bundle_version }}.zip"
|
||||||
if [ ! -f "$ZIP" ]; then
|
if [ ! -f "$ZIP" ]; then
|
||||||
echo "Bundle not found in host cache; falling back to S3."
|
echo "Bundle not found in host cache; falling back to S3."
|
||||||
mkdir -p "$BUNDLE_CACHE"
|
mkdir -p "$BUNDLE_CACHE"
|
||||||
aws s3 cp "s3://${S3_BUCKET}/penpot-${BUILD_KEY}.zip" "$ZIP.$$.tmp"
|
aws s3 cp "s3://${{ secrets.S3_BUCKET }}/penpot-${{ needs.prepare.outputs.gh_ref }}.zip" "$ZIP.$$.tmp"
|
||||||
mv "$ZIP.$$.tmp" "$ZIP"
|
mv "$ZIP.$$.tmp" "$ZIP"
|
||||||
fi
|
fi
|
||||||
# Extract only the bundle this job needs.
|
# Extract only the bundle this job needs.
|
||||||
pushd docker/images
|
pushd docker/images
|
||||||
unzip -q "$ZIP" "penpot/${IMAGE}/*"
|
unzip -q "$ZIP" "penpot/${{ matrix.image }}/*"
|
||||||
mv "penpot/${IMAGE}" "bundle-${IMAGE}"
|
mv "penpot/${{ matrix.image }}" "bundle-${{ matrix.image }}"
|
||||||
popd
|
popd
|
||||||
|
|
||||||
- name: Set up QEMU (stable)
|
- name: Set up QEMU (stable)
|
||||||
uses: docker/setup-qemu-action@99012661954931238ded8c8b007157a8430204e1 # v4.4.0
|
uses: docker/setup-qemu-action@v4
|
||||||
with:
|
with:
|
||||||
platforms: linux/amd64,linux/arm64
|
platforms: linux/amd64,linux/arm64
|
||||||
|
|
||||||
- name: Set up Docker Buildx
|
- name: Set up Docker Buildx
|
||||||
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
|
uses: docker/setup-buildx-action@v4
|
||||||
|
|
||||||
- name: Extract metadata (tags, labels)
|
- name: Extract metadata (tags, labels)
|
||||||
id: meta
|
id: meta
|
||||||
uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6.2.0
|
uses: docker/metadata-action@v6
|
||||||
with:
|
with:
|
||||||
images: ${{ matrix.image }}
|
images: ${{ matrix.image }}
|
||||||
labels: |
|
labels: |
|
||||||
bundle_version=${{ needs.prepare.outputs.bundle_version }}
|
bundle_version=${{ needs.prepare.outputs.bundle_version }}
|
||||||
|
|
||||||
- name: Build and push Docker image
|
- name: Build and push Docker image
|
||||||
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
|
uses: docker/build-push-action@v7
|
||||||
with:
|
with:
|
||||||
context: ./docker/images/
|
context: ./docker/images/
|
||||||
file: ./docker/images/Dockerfile.${{ matrix.image }}
|
file: ./docker/images/Dockerfile.${{ matrix.image }}
|
||||||
@ -280,15 +237,18 @@ jobs:
|
|||||||
sbom: true
|
sbom: true
|
||||||
# Immutable tag only; branch tags are moved atomically for the
|
# Immutable tag only; branch tags are moved atomically for the
|
||||||
# whole image set by the `promote` job.
|
# whole image set by the `promote` job.
|
||||||
tags: ${{ secrets.DOCKER_REGISTRY }}/${{ matrix.image }}:${{ needs.prepare.outputs.build_key }}
|
tags: ${{ secrets.DOCKER_REGISTRY }}/${{ matrix.image }}:sha-${{ needs.prepare.outputs.sha }}
|
||||||
labels: ${{ steps.meta.outputs.labels }}
|
labels: ${{ steps.meta.outputs.labels }}
|
||||||
cache-from: type=registry,ref=${{ secrets.DOCKER_REGISTRY }}/${{ matrix.image }}:buildcache
|
cache-from: type=registry,ref=${{ secrets.DOCKER_REGISTRY }}/${{ matrix.image }}:buildcache
|
||||||
cache-to: type=registry,ref=${{ secrets.DOCKER_REGISTRY }}/${{ matrix.image }}:buildcache,mode=max
|
cache-to: type=registry,ref=${{ secrets.DOCKER_REGISTRY }}/${{ matrix.image }}:buildcache,mode=max
|
||||||
|
|
||||||
# ── 3a. Move the branch tags of ALL images together (fresh build) ──────
|
# ── 3. Move the branch tags of ALL images together ─────────────────────
|
||||||
|
# Runs only when every build succeeded (default `needs` semantics); if
|
||||||
|
# the set was already complete, `build` is skipped and so is this job —
|
||||||
|
# the S3 marker guarantees the branch tags were already moved.
|
||||||
promote:
|
promote:
|
||||||
name: Promote image set
|
name: Promote image set
|
||||||
runs-on: penpot-standar-runner
|
runs-on: penpot-extended-runner
|
||||||
timeout-minutes: 10
|
timeout-minutes: 10
|
||||||
needs: [prepare, build]
|
needs: [prepare, build]
|
||||||
|
|
||||||
@ -298,26 +258,22 @@ jobs:
|
|||||||
echo "DOCKER_CONFIG=${{ runner.temp }}/.docker-${{ github.run_id }}-${{ github.job }}" >> $GITHUB_ENV
|
echo "DOCKER_CONFIG=${{ runner.temp }}/.docker-${{ github.run_id }}-${{ github.job }}" >> $GITHUB_ENV
|
||||||
|
|
||||||
- name: Set up Docker Buildx
|
- name: Set up Docker Buildx
|
||||||
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
|
uses: docker/setup-buildx-action@v4
|
||||||
|
|
||||||
- name: Login to Docker Registry
|
- name: Login to Docker Registry
|
||||||
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
|
uses: docker/login-action@v4
|
||||||
with:
|
with:
|
||||||
registry: ${{ secrets.DOCKER_REGISTRY }}
|
registry: ${{ secrets.DOCKER_REGISTRY }}
|
||||||
username: ${{ secrets.DOCKER_USERNAME }}
|
username: ${{ secrets.DOCKER_USERNAME }}
|
||||||
password: ${{ secrets.DOCKER_PASSWORD }}
|
password: ${{ secrets.DOCKER_PASSWORD }}
|
||||||
|
|
||||||
- name: Point branch tags to the new build key
|
- name: Point branch tags to the new build key
|
||||||
env:
|
|
||||||
DOCKER_REGISTRY: ${{ secrets.DOCKER_REGISTRY }}
|
|
||||||
REF_SLUG: ${{ needs.prepare.outputs.ref_slug }}
|
|
||||||
BUILD_KEY: ${{ needs.prepare.outputs.build_key }}
|
|
||||||
run: |
|
run: |
|
||||||
set -e
|
set -e
|
||||||
for image in $ALL_IMAGES; do
|
for image in $ALL_IMAGES; do
|
||||||
docker buildx imagetools create \
|
docker buildx imagetools create \
|
||||||
-t "${DOCKER_REGISTRY}/${image}:${REF_SLUG}" \
|
-t "${{ secrets.DOCKER_REGISTRY }}/$image:${{ needs.prepare.outputs.gh_ref }}" \
|
||||||
"${DOCKER_REGISTRY}/${image}:${BUILD_KEY}"
|
"${{ secrets.DOCKER_REGISTRY }}/$image:sha-${{ needs.prepare.outputs.sha }}"
|
||||||
done
|
done
|
||||||
|
|
||||||
# The marker is written LAST: its presence certifies that all five
|
# The marker is written LAST: its presence certifies that all five
|
||||||
@ -327,93 +283,28 @@ jobs:
|
|||||||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
||||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
||||||
AWS_DEFAULT_REGION: ${{ secrets.AWS_REGION }}
|
AWS_DEFAULT_REGION: ${{ secrets.AWS_REGION }}
|
||||||
S3_BUCKET: ${{ secrets.S3_BUCKET }}
|
|
||||||
BUILD_KEY: ${{ needs.prepare.outputs.build_key }}
|
|
||||||
run: |
|
run: |
|
||||||
echo "$GITHUB_RUN_ID" | aws s3 cp - \
|
echo "${{ github.run_id }}" | aws s3 cp - \
|
||||||
"s3://${S3_BUCKET}/markers/images-${BUILD_KEY}"
|
"s3://${{ secrets.S3_BUCKET }}/markers/images-sha-${{ needs.prepare.outputs.sha }}"
|
||||||
|
|
||||||
- name: Write step summary
|
- name: Write step summary
|
||||||
env:
|
|
||||||
REF_SLUG: ${{ needs.prepare.outputs.ref_slug }}
|
|
||||||
SHORT_SHA: ${{ needs.prepare.outputs.short_sha }}
|
|
||||||
BUILD_KEY: ${{ needs.prepare.outputs.build_key }}
|
|
||||||
BUNDLE_VERSION: ${{ needs.prepare.outputs.bundle_version }}
|
|
||||||
COMMIT_TITLE: ${{ needs.prepare.outputs.commit_title }}
|
|
||||||
run: |
|
run: |
|
||||||
{
|
{
|
||||||
echo "### ✅ Image set promoted"
|
echo "### ✅ Image set promoted"
|
||||||
echo ""
|
echo ""
|
||||||
echo "- Version: \`${BUNDLE_VERSION}\` (\`./manage.sh version\`)"
|
echo "- Version: \`${{ needs.prepare.outputs.bundle_version }}\` (\`git describe --tags --always\`)"
|
||||||
echo "- Commit: [\`${SHORT_SHA}\`](${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/commit/${SHORT_SHA}) — ${COMMIT_TITLE}"
|
echo "- Commit: [\`${{ needs.prepare.outputs.sha }}\`](https://github.com/${{ github.repository }}/commit/${{ needs.prepare.outputs.sha }}) — ${{ needs.prepare.outputs.commit_title }}"
|
||||||
echo "- Built at: $(date -u +'%Y-%m-%d %H:%M:%S UTC')"
|
echo "- Built at: $(date -u +'%Y-%m-%d %H:%M:%S UTC')"
|
||||||
echo ""
|
echo ""
|
||||||
echo "All \`:${REF_SLUG}\` tags now point to \`${BUILD_KEY}\`."
|
echo "All \`:${{ needs.prepare.outputs.gh_ref }}\` tags now point to \`sha-${{ needs.prepare.outputs.sha }}\`."
|
||||||
} >> "$GITHUB_STEP_SUMMARY"
|
|
||||||
|
|
||||||
# ── 3b. Skip path: make sure THIS ref's tags point to the existing
|
|
||||||
# images (another ref may have built and promoted the same commit
|
|
||||||
# first — the sha marker says the images exist, not that this ref's
|
|
||||||
# tags were ever created).
|
|
||||||
retag:
|
|
||||||
name: Ensure branch tags (already-built commit)
|
|
||||||
runs-on: penpot-standar-runner
|
|
||||||
timeout-minutes: 10
|
|
||||||
needs: prepare
|
|
||||||
if: needs.prepare.outputs.exists == 'true'
|
|
||||||
|
|
||||||
steps:
|
|
||||||
- name: Set common environment variables
|
|
||||||
run: |
|
|
||||||
echo "DOCKER_CONFIG=${{ runner.temp }}/.docker-${{ github.run_id }}-${{ github.job }}" >> $GITHUB_ENV
|
|
||||||
|
|
||||||
- name: Set up Docker Buildx
|
|
||||||
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
|
|
||||||
|
|
||||||
- name: Login to Docker Registry
|
|
||||||
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
|
|
||||||
with:
|
|
||||||
registry: ${{ secrets.DOCKER_REGISTRY }}
|
|
||||||
username: ${{ secrets.DOCKER_USERNAME }}
|
|
||||||
password: ${{ secrets.DOCKER_PASSWORD }}
|
|
||||||
|
|
||||||
- name: Point branch tags to the existing build key
|
|
||||||
env:
|
|
||||||
DOCKER_REGISTRY: ${{ secrets.DOCKER_REGISTRY }}
|
|
||||||
REF_SLUG: ${{ needs.prepare.outputs.ref_slug }}
|
|
||||||
BUILD_KEY: ${{ needs.prepare.outputs.build_key }}
|
|
||||||
run: |
|
|
||||||
set -e
|
|
||||||
for image in $ALL_IMAGES; do
|
|
||||||
docker buildx imagetools create \
|
|
||||||
-t "${DOCKER_REGISTRY}/${image}:${REF_SLUG}" \
|
|
||||||
"${DOCKER_REGISTRY}/${image}:${BUILD_KEY}"
|
|
||||||
done
|
|
||||||
|
|
||||||
- name: Write step summary
|
|
||||||
env:
|
|
||||||
REF_SLUG: ${{ needs.prepare.outputs.ref_slug }}
|
|
||||||
SHORT_SHA: ${{ needs.prepare.outputs.short_sha }}
|
|
||||||
BUILD_KEY: ${{ needs.prepare.outputs.build_key }}
|
|
||||||
BUNDLE_VERSION: ${{ needs.prepare.outputs.bundle_version }}
|
|
||||||
COMMIT_TITLE: ${{ needs.prepare.outputs.commit_title }}
|
|
||||||
run: |
|
|
||||||
{
|
|
||||||
echo "### ✅ Image set already built (branch tags ensured)"
|
|
||||||
echo ""
|
|
||||||
echo "- Version: \`${BUNDLE_VERSION}\` (\`./manage.sh version\`)"
|
|
||||||
echo "- Commit: [\`${SHORT_SHA}\`](${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/commit/${SHORT_SHA}) — ${COMMIT_TITLE}"
|
|
||||||
echo "- Checked at: $(date -u +'%Y-%m-%d %H:%M:%S UTC')"
|
|
||||||
echo ""
|
|
||||||
echo "All \`:${REF_SLUG}\` tags now point to \`${BUILD_KEY}\`."
|
|
||||||
} >> "$GITHUB_STEP_SUMMARY"
|
} >> "$GITHUB_STEP_SUMMARY"
|
||||||
|
|
||||||
# ── 4. Single failure notification for the whole workflow ─────────────
|
# ── 4. Single failure notification for the whole workflow ─────────────
|
||||||
notify:
|
notify:
|
||||||
name: Notify failure
|
name: Notify failure
|
||||||
runs-on: penpot-standar-runner
|
runs-on: penpot-extended-runner
|
||||||
timeout-minutes: 5
|
timeout-minutes: 5
|
||||||
needs: [prepare, build, promote, retag]
|
needs: [prepare, build, promote]
|
||||||
if: failure()
|
if: failure()
|
||||||
|
|
||||||
steps:
|
steps:
|
||||||
|
|||||||
260
.github/workflows/build-renovate.yml
vendored
260
.github/workflows/build-renovate.yml
vendored
@ -1,260 +0,0 @@
|
|||||||
name: Renovate Docker Image Check
|
|
||||||
|
|
||||||
# Triggered by pushes to Renovate branches, not by pull_request: this is a
|
|
||||||
# public repository and these jobs run on the self-hosted runners, so a PR
|
|
||||||
# from a fork must never be able to start them. Only users with write
|
|
||||||
# access can push here.
|
|
||||||
on:
|
|
||||||
push:
|
|
||||||
branches:
|
|
||||||
- 'renovate-*'
|
|
||||||
|
|
||||||
permissions: {}
|
|
||||||
|
|
||||||
concurrency:
|
|
||||||
group: renovate-image-check-${{ github.ref_name }}
|
|
||||||
cancel-in-progress: true
|
|
||||||
|
|
||||||
env:
|
|
||||||
BUNDLE_CACHE: /var/tmp/penpot-bundle-cache
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
# ── 1. Pick an existing develop bundle and detect what changed ────────
|
|
||||||
prepare:
|
|
||||||
name: Prepare
|
|
||||||
runs-on: penpot-standar-runner
|
|
||||||
timeout-minutes: 10
|
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
outputs:
|
|
||||||
bundle_key: ${{ steps.bundle.outputs.bundle_key }}
|
|
||||||
images: ${{ steps.changes.outputs.images }}
|
|
||||||
devenv: ${{ steps.changes.outputs.devenv }}
|
|
||||||
imagemagick: ${{ steps.changes.outputs.imagemagick }}
|
|
||||||
|
|
||||||
steps:
|
|
||||||
- name: Checkout code
|
|
||||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
||||||
with:
|
|
||||||
persist-credentials: false
|
|
||||||
fetch-depth: 0
|
|
||||||
|
|
||||||
- name: Find the develop commit this branch is based on
|
|
||||||
id: base
|
|
||||||
run: |
|
|
||||||
echo "sha=$(git merge-base HEAD origin/develop)" >> "$GITHUB_OUTPUT"
|
|
||||||
|
|
||||||
- name: Detect changed images
|
|
||||||
id: changes
|
|
||||||
env:
|
|
||||||
BASE_SHA: ${{ steps.base.outputs.sha }}
|
|
||||||
run: |
|
|
||||||
changed() {
|
|
||||||
git diff --quiet "$BASE_SHA" HEAD -- "$@" && echo false || echo true
|
|
||||||
}
|
|
||||||
{
|
|
||||||
# Anything outside .github can affect the production images.
|
|
||||||
echo "images=$(changed . ':!.github')"
|
|
||||||
echo "devenv=$(changed docker/devenv)"
|
|
||||||
echo "imagemagick=$(changed docker/imagemagick)"
|
|
||||||
} >> "$GITHUB_OUTPUT"
|
|
||||||
|
|
||||||
# Renovate only changes Dockerfiles and package manager versions, so
|
|
||||||
# the most recent develop bundle is enough to build the images. The
|
|
||||||
# version part of the key does not matter here: take any bundle of
|
|
||||||
# the commit.
|
|
||||||
- name: Find the latest develop bundle
|
|
||||||
id: bundle
|
|
||||||
env:
|
|
||||||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
|
||||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
|
||||||
AWS_DEFAULT_REGION: ${{ secrets.AWS_REGION }}
|
|
||||||
S3_BUCKET: ${{ secrets.S3_BUCKET }}
|
|
||||||
BASE_SHA: ${{ steps.base.outputs.sha }}
|
|
||||||
run: |
|
|
||||||
for SHA in $(git rev-list --first-parent --max-count=50 "$BASE_SHA"); do
|
|
||||||
SHORT_SHA=$(git rev-parse --short=12 "$SHA")
|
|
||||||
BUNDLE_KEY=$(aws s3api list-objects-v2 --bucket "$S3_BUCKET" \
|
|
||||||
--prefix "penpot-sha-${SHORT_SHA}-" \
|
|
||||||
--query 'Contents[0].Key' --output text)
|
|
||||||
if [ -n "$BUNDLE_KEY" ] && [ "$BUNDLE_KEY" != "None" ]; then
|
|
||||||
echo "bundle_key=$BUNDLE_KEY" >> "$GITHUB_OUTPUT"
|
|
||||||
echo "Using bundle \`${BUNDLE_KEY}\`" >> "$GITHUB_STEP_SUMMARY"
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
echo "::error::No bundle found for the last 50 develop commits"
|
|
||||||
exit 1
|
|
||||||
|
|
||||||
# ── 2. Production images, built from the PR Dockerfiles ───────────────
|
|
||||||
images:
|
|
||||||
name: Build ${{ matrix.image }}
|
|
||||||
runs-on: penpot-standar-runner
|
|
||||||
timeout-minutes: 60
|
|
||||||
needs: prepare
|
|
||||||
if: needs.prepare.outputs.images == 'true'
|
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
strategy:
|
|
||||||
fail-fast: false
|
|
||||||
max-parallel: 3
|
|
||||||
matrix:
|
|
||||||
image: [backend, frontend, exporter, storybook, mcp]
|
|
||||||
|
|
||||||
steps:
|
|
||||||
- name: Set common environment variables
|
|
||||||
env:
|
|
||||||
MATRIX_IMAGE: ${{ matrix.image }}
|
|
||||||
run: |
|
|
||||||
echo "DOCKER_CONFIG=${RUNNER_TEMP}/.docker-${GITHUB_RUN_ID}-${GITHUB_JOB}-${MATRIX_IMAGE}" >> "$GITHUB_ENV"
|
|
||||||
|
|
||||||
- name: Checkout code
|
|
||||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
||||||
with:
|
|
||||||
persist-credentials: false
|
|
||||||
|
|
||||||
- name: Login to Docker Registry
|
|
||||||
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
|
|
||||||
with:
|
|
||||||
registry: ${{ secrets.DOCKER_REGISTRY }}
|
|
||||||
username: ${{ secrets.DOCKER_USERNAME }}
|
|
||||||
password: ${{ secrets.DOCKER_PASSWORD }}
|
|
||||||
|
|
||||||
- name: Login to DockerHub Registry
|
|
||||||
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
|
|
||||||
with:
|
|
||||||
username: ${{ secrets.PUB_DOCKER_USERNAME }}
|
|
||||||
password: ${{ secrets.PUB_DOCKER_PASSWORD }}
|
|
||||||
|
|
||||||
- name: Login to Docker Hardened Images registry
|
|
||||||
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
|
|
||||||
with:
|
|
||||||
registry: dhi.io
|
|
||||||
username: ${{ secrets.PUB_DOCKER_USERNAME }}
|
|
||||||
password: ${{ secrets.PUB_DOCKER_PASSWORD }}
|
|
||||||
|
|
||||||
- name: Prepare Penpot bundle
|
|
||||||
env:
|
|
||||||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
|
||||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
|
||||||
AWS_DEFAULT_REGION: ${{ secrets.AWS_REGION }}
|
|
||||||
S3_BUCKET: ${{ secrets.S3_BUCKET }}
|
|
||||||
BUNDLE_KEY: ${{ needs.prepare.outputs.bundle_key }}
|
|
||||||
IMAGE: ${{ matrix.image }}
|
|
||||||
run: |
|
|
||||||
ZIP="$BUNDLE_CACHE/${BUNDLE_KEY}"
|
|
||||||
if [ ! -f "$ZIP" ]; then
|
|
||||||
mkdir -p "$BUNDLE_CACHE"
|
|
||||||
aws s3 cp "s3://${S3_BUCKET}/${BUNDLE_KEY}" "$ZIP.$$.tmp"
|
|
||||||
mv "$ZIP.$$.tmp" "$ZIP"
|
|
||||||
fi
|
|
||||||
pushd docker/images
|
|
||||||
unzip -q "$ZIP" "penpot/${IMAGE}/*"
|
|
||||||
mv "penpot/${IMAGE}" "bundle-${IMAGE}"
|
|
||||||
popd
|
|
||||||
|
|
||||||
- name: Set up QEMU
|
|
||||||
uses: docker/setup-qemu-action@99012661954931238ded8c8b007157a8430204e1 # v4.4.0
|
|
||||||
with:
|
|
||||||
platforms: linux/amd64,linux/arm64
|
|
||||||
|
|
||||||
- name: Set up Docker Buildx
|
|
||||||
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
|
|
||||||
|
|
||||||
- name: Build Docker image
|
|
||||||
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
|
|
||||||
with:
|
|
||||||
context: ./docker/images/
|
|
||||||
file: ./docker/images/Dockerfile.${{ matrix.image }}
|
|
||||||
platforms: linux/amd64,linux/arm64
|
|
||||||
push: false
|
|
||||||
cache-from: type=registry,ref=${{ secrets.DOCKER_REGISTRY }}/${{ matrix.image }}:buildcache
|
|
||||||
|
|
||||||
# ── 3. Devenv, only when the PR touches it ────────────────────────────
|
|
||||||
devenv:
|
|
||||||
name: Build devenv
|
|
||||||
runs-on: penpot-standar-runner
|
|
||||||
timeout-minutes: 90
|
|
||||||
needs: prepare
|
|
||||||
if: needs.prepare.outputs.devenv == 'true'
|
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
|
|
||||||
steps:
|
|
||||||
- name: Set common environment variables
|
|
||||||
run: |
|
|
||||||
echo "DOCKER_CONFIG=${RUNNER_TEMP}/.docker-${GITHUB_RUN_ID}-${GITHUB_JOB}" >> "$GITHUB_ENV"
|
|
||||||
|
|
||||||
- name: Checkout code
|
|
||||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
||||||
with:
|
|
||||||
persist-credentials: false
|
|
||||||
|
|
||||||
- name: Login to DockerHub Registry
|
|
||||||
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
|
|
||||||
with:
|
|
||||||
username: ${{ secrets.PUB_DOCKER_USERNAME }}
|
|
||||||
password: ${{ secrets.PUB_DOCKER_PASSWORD }}
|
|
||||||
|
|
||||||
- name: Login to Docker Hardened Images registry
|
|
||||||
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
|
|
||||||
with:
|
|
||||||
registry: dhi.io
|
|
||||||
username: ${{ secrets.PUB_DOCKER_USERNAME }}
|
|
||||||
password: ${{ secrets.PUB_DOCKER_PASSWORD }}
|
|
||||||
|
|
||||||
- name: Set up QEMU
|
|
||||||
uses: docker/setup-qemu-action@99012661954931238ded8c8b007157a8430204e1 # v4.4.0
|
|
||||||
with:
|
|
||||||
platforms: linux/amd64,linux/arm64
|
|
||||||
|
|
||||||
- name: Set up Docker Buildx
|
|
||||||
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
|
|
||||||
|
|
||||||
- name: Build devenv image
|
|
||||||
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
|
|
||||||
with:
|
|
||||||
context: ./docker/devenv/
|
|
||||||
file: ./docker/devenv/Dockerfile
|
|
||||||
platforms: linux/amd64,linux/arm64
|
|
||||||
push: false
|
|
||||||
cache-from: type=registry,ref=penpotapp/devenv:buildcache
|
|
||||||
|
|
||||||
# ── 4. ImageMagick, only when the PR touches it ───────────────────────
|
|
||||||
# amd64 only: compiling ImageMagick under QEMU takes too long for a check.
|
|
||||||
imagemagick:
|
|
||||||
name: Build imagemagick
|
|
||||||
runs-on: penpot-standar-runner
|
|
||||||
timeout-minutes: 60
|
|
||||||
needs: prepare
|
|
||||||
if: needs.prepare.outputs.imagemagick == 'true'
|
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
|
|
||||||
steps:
|
|
||||||
- name: Set common environment variables
|
|
||||||
run: |
|
|
||||||
echo "DOCKER_CONFIG=${RUNNER_TEMP}/.docker-${GITHUB_RUN_ID}-${GITHUB_JOB}" >> "$GITHUB_ENV"
|
|
||||||
|
|
||||||
- name: Checkout code
|
|
||||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
||||||
with:
|
|
||||||
persist-credentials: false
|
|
||||||
|
|
||||||
- name: Login to Docker Hardened Images registry
|
|
||||||
uses: docker/login-action@8aa5831fb8d48d8f76594c7ddde61ae58019f550 # v4.8.0
|
|
||||||
with:
|
|
||||||
registry: dhi.io
|
|
||||||
username: ${{ secrets.PUB_DOCKER_USERNAME }}
|
|
||||||
password: ${{ secrets.PUB_DOCKER_PASSWORD }}
|
|
||||||
|
|
||||||
- name: Set up Docker Buildx
|
|
||||||
uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
|
|
||||||
|
|
||||||
- name: Build imagemagick image
|
|
||||||
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
|
|
||||||
with:
|
|
||||||
context: ./docker/imagemagick/
|
|
||||||
file: ./docker/imagemagick/Dockerfile
|
|
||||||
platforms: linux/amd64
|
|
||||||
push: false
|
|
||||||
31
.github/workflows/build-staging.yml
vendored
31
.github/workflows/build-staging.yml
vendored
@ -18,50 +18,25 @@ concurrency:
|
|||||||
group: ${{ github.workflow }}
|
group: ${{ github.workflow }}
|
||||||
cancel-in-progress: true
|
cancel-in-progress: true
|
||||||
|
|
||||||
permissions: {}
|
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
build-bundle:
|
build-bundle:
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
uses: ./.github/workflows/build-bundle.yml
|
uses: ./.github/workflows/build-bundle.yml
|
||||||
secrets:
|
secrets: inherit
|
||||||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
|
||||||
AWS_REGION: ${{ secrets.AWS_REGION }}
|
|
||||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
|
||||||
MATTERMOST_WEBHOOK: ${{ secrets.MATTERMOST_WEBHOOK }}
|
|
||||||
S3_BUCKET: ${{ secrets.S3_BUCKET }}
|
|
||||||
with:
|
with:
|
||||||
gh_ref: "staging"
|
gh_ref: "staging"
|
||||||
force: ${{ inputs.force || false }}
|
force: ${{ inputs.force || false }}
|
||||||
|
|
||||||
build-docker:
|
build-docker:
|
||||||
needs: build-bundle
|
needs: build-bundle
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
uses: ./.github/workflows/build-docker.yml
|
uses: ./.github/workflows/build-docker.yml
|
||||||
secrets:
|
secrets: inherit
|
||||||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
|
||||||
AWS_REGION: ${{ secrets.AWS_REGION }}
|
|
||||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
|
||||||
DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
|
|
||||||
DOCKER_REGISTRY: ${{ secrets.DOCKER_REGISTRY }}
|
|
||||||
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
|
|
||||||
MATTERMOST_WEBHOOK: ${{ secrets.MATTERMOST_WEBHOOK }}
|
|
||||||
PUB_DOCKER_PASSWORD: ${{ secrets.PUB_DOCKER_PASSWORD }}
|
|
||||||
PUB_DOCKER_USERNAME: ${{ secrets.PUB_DOCKER_USERNAME }}
|
|
||||||
S3_BUCKET: ${{ secrets.S3_BUCKET }}
|
|
||||||
with:
|
with:
|
||||||
gh_ref: "staging"
|
gh_ref: "staging"
|
||||||
# Pin build-docker to the commit build-bundle actually bundled.
|
|
||||||
sha: ${{ needs.build-bundle.outputs.sha }}
|
|
||||||
force: ${{ inputs.force || false }}
|
force: ${{ inputs.force || false }}
|
||||||
|
|
||||||
build-docker-admin-console:
|
build-docker-admin-console:
|
||||||
permissions: {}
|
|
||||||
uses: ./.github/workflows/build-docker-admin-console.yml
|
uses: ./.github/workflows/build-docker-admin-console.yml
|
||||||
secrets:
|
secrets: inherit
|
||||||
ORG_WORKFLOW_TOKEN: ${{ secrets.ORG_WORKFLOW_TOKEN }}
|
|
||||||
with:
|
with:
|
||||||
gh_ref: "staging"
|
gh_ref: "staging"
|
||||||
force: ${{ inputs.force || false }}
|
force: ${{ inputs.force || false }}
|
||||||
|
|||||||
43
.github/workflows/build-tag.yml
vendored
43
.github/workflows/build-tag.yml
vendored
@ -21,56 +21,32 @@ concurrency:
|
|||||||
group: ${{ github.workflow }}-${{ github.ref_name }}
|
group: ${{ github.workflow }}-${{ github.ref_name }}
|
||||||
cancel-in-progress: false
|
cancel-in-progress: false
|
||||||
|
|
||||||
permissions: {}
|
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
build-bundle:
|
build-bundle:
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
uses: ./.github/workflows/build-bundle.yml
|
uses: ./.github/workflows/build-bundle.yml
|
||||||
secrets:
|
secrets: inherit
|
||||||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
|
||||||
AWS_REGION: ${{ secrets.AWS_REGION }}
|
|
||||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
|
||||||
MATTERMOST_WEBHOOK: ${{ secrets.MATTERMOST_WEBHOOK }}
|
|
||||||
S3_BUCKET: ${{ secrets.S3_BUCKET }}
|
|
||||||
with:
|
with:
|
||||||
gh_ref: ${{ github.ref_name }}
|
gh_ref: ${{ github.ref_name }}
|
||||||
force: ${{ inputs.force || false }}
|
force: ${{ inputs.force || false }}
|
||||||
|
|
||||||
build-docker:
|
build-docker:
|
||||||
needs: build-bundle
|
needs: build-bundle
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
uses: ./.github/workflows/build-docker.yml
|
uses: ./.github/workflows/build-docker.yml
|
||||||
secrets:
|
secrets: inherit
|
||||||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
|
||||||
AWS_REGION: ${{ secrets.AWS_REGION }}
|
|
||||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
|
||||||
DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
|
|
||||||
DOCKER_REGISTRY: ${{ secrets.DOCKER_REGISTRY }}
|
|
||||||
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
|
|
||||||
MATTERMOST_WEBHOOK: ${{ secrets.MATTERMOST_WEBHOOK }}
|
|
||||||
PUB_DOCKER_PASSWORD: ${{ secrets.PUB_DOCKER_PASSWORD }}
|
|
||||||
PUB_DOCKER_USERNAME: ${{ secrets.PUB_DOCKER_USERNAME }}
|
|
||||||
S3_BUCKET: ${{ secrets.S3_BUCKET }}
|
|
||||||
with:
|
with:
|
||||||
gh_ref: ${{ github.ref_name }}
|
gh_ref: ${{ github.ref_name }}
|
||||||
force: ${{ inputs.force || false }}
|
force: ${{ inputs.force || false }}
|
||||||
|
|
||||||
build-docker-admin-console:
|
build-docker-admin-console:
|
||||||
permissions: {}
|
|
||||||
uses: ./.github/workflows/build-docker-admin-console.yml
|
uses: ./.github/workflows/build-docker-admin-console.yml
|
||||||
secrets:
|
secrets: inherit
|
||||||
ORG_WORKFLOW_TOKEN: ${{ secrets.ORG_WORKFLOW_TOKEN }}
|
|
||||||
with:
|
with:
|
||||||
gh_ref: ${{ github.ref_name }}
|
gh_ref: ${{ github.ref_name }}
|
||||||
force: ${{ inputs.force || false }}
|
force: ${{ inputs.force || false }}
|
||||||
|
|
||||||
notify:
|
notify:
|
||||||
name: Notifications
|
name: Notifications
|
||||||
permissions: {}
|
runs-on: ubuntu-24.04
|
||||||
runs-on: ubuntu-latest
|
|
||||||
needs:
|
needs:
|
||||||
- build-docker
|
- build-docker
|
||||||
- build-docker-admin-console
|
- build-docker-admin-console
|
||||||
@ -90,16 +66,7 @@ jobs:
|
|||||||
needs:
|
needs:
|
||||||
- build-docker
|
- build-docker
|
||||||
- build-docker-admin-console
|
- build-docker-admin-console
|
||||||
permissions:
|
|
||||||
contents: write
|
|
||||||
uses: ./.github/workflows/release.yml
|
uses: ./.github/workflows/release.yml
|
||||||
secrets:
|
secrets: inherit
|
||||||
AWS_REGION: ${{ secrets.AWS_REGION }}
|
|
||||||
DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
|
|
||||||
DOCKER_REGISTRY: ${{ secrets.DOCKER_REGISTRY }}
|
|
||||||
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
|
|
||||||
MATTERMOST_WEBHOOK: ${{ secrets.MATTERMOST_WEBHOOK }}
|
|
||||||
PUB_DOCKER_PASSWORD: ${{ secrets.PUB_DOCKER_PASSWORD }}
|
|
||||||
PUB_DOCKER_USERNAME: ${{ secrets.PUB_DOCKER_USERNAME }}
|
|
||||||
with:
|
with:
|
||||||
gh_ref: ${{ github.ref_name }}
|
gh_ref: ${{ github.ref_name }}
|
||||||
|
|||||||
29
.github/workflows/build-tmp-tokens.yml
vendored
29
.github/workflows/build-tmp-tokens.yml
vendored
@ -2,44 +2,23 @@ name: _TMP TOKENS
|
|||||||
|
|
||||||
on:
|
on:
|
||||||
workflow_dispatch:
|
workflow_dispatch:
|
||||||
# schedule:
|
schedule:
|
||||||
# - cron: '46 5-20 * * 1-5'
|
- cron: '46 5-20 * * 1-5'
|
||||||
|
|
||||||
concurrency:
|
concurrency:
|
||||||
group: ${{ github.workflow }}
|
group: ${{ github.workflow }}
|
||||||
cancel-in-progress: true
|
cancel-in-progress: true
|
||||||
|
|
||||||
permissions: {}
|
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
build-bundle:
|
build-bundle:
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
uses: ./.github/workflows/build-bundle.yml
|
uses: ./.github/workflows/build-bundle.yml
|
||||||
secrets:
|
secrets: inherit
|
||||||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
|
||||||
AWS_REGION: ${{ secrets.AWS_REGION }}
|
|
||||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
|
||||||
MATTERMOST_WEBHOOK: ${{ secrets.MATTERMOST_WEBHOOK }}
|
|
||||||
S3_BUCKET: ${{ secrets.S3_BUCKET }}
|
|
||||||
with:
|
with:
|
||||||
gh_ref: "hiru-tokens-in-libs"
|
gh_ref: "hiru-tokens-in-libs"
|
||||||
|
|
||||||
build-docker:
|
build-docker:
|
||||||
needs: build-bundle
|
needs: build-bundle
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
uses: ./.github/workflows/build-docker.yml
|
uses: ./.github/workflows/build-docker.yml
|
||||||
secrets:
|
secrets: inherit
|
||||||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
|
||||||
AWS_REGION: ${{ secrets.AWS_REGION }}
|
|
||||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
|
||||||
DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
|
|
||||||
DOCKER_REGISTRY: ${{ secrets.DOCKER_REGISTRY }}
|
|
||||||
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
|
|
||||||
MATTERMOST_WEBHOOK: ${{ secrets.MATTERMOST_WEBHOOK }}
|
|
||||||
PUB_DOCKER_PASSWORD: ${{ secrets.PUB_DOCKER_PASSWORD }}
|
|
||||||
PUB_DOCKER_USERNAME: ${{ secrets.PUB_DOCKER_USERNAME }}
|
|
||||||
S3_BUCKET: ${{ secrets.S3_BUCKET }}
|
|
||||||
with:
|
with:
|
||||||
gh_ref: "hiru-tokens-in-libs"
|
gh_ref: "hiru-tokens-in-libs"
|
||||||
|
|||||||
55
.github/workflows/commit-checker.yml
vendored
55
.github/workflows/commit-checker.yml
vendored
@ -7,16 +7,19 @@ on:
|
|||||||
- reopened
|
- reopened
|
||||||
- synchronize
|
- synchronize
|
||||||
- ready_for_review
|
- ready_for_review
|
||||||
|
pull_request_target:
|
||||||
|
types:
|
||||||
|
- opened
|
||||||
|
- edited
|
||||||
|
- reopened
|
||||||
|
- synchronize
|
||||||
|
- ready_for_review
|
||||||
push:
|
push:
|
||||||
branches:
|
branches:
|
||||||
- main
|
- main
|
||||||
- develop
|
- develop
|
||||||
- staging
|
- staging
|
||||||
|
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
pull-requests: read
|
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
check-commit-message:
|
check-commit-message:
|
||||||
if: ${{ !github.event.pull_request.draft }}
|
if: ${{ !github.event.pull_request.draft }}
|
||||||
@ -24,7 +27,7 @@ jobs:
|
|||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
steps:
|
steps:
|
||||||
- name: Check Commit Type
|
- name: Check Commit Type
|
||||||
uses: gsactions/commit-message-checker@16fa2d5de096ae0d35626443bcd24f1e756cafee # v2.0.0
|
uses: gsactions/commit-message-checker@v2
|
||||||
with:
|
with:
|
||||||
pattern: '^(((:(lipstick|globe_with_meridians|wrench|books|arrow_up|arrow_down|zap|ambulance|construction|boom|fire|whale|bug|sparkles|paperclip|tada|recycle|rewind|construction_worker):)\s[A-Z].*[^.])|(Merge|Revert|Reapply).+[^.])$'
|
pattern: '^(((:(lipstick|globe_with_meridians|wrench|books|arrow_up|arrow_down|zap|ambulance|construction|boom|fire|whale|bug|sparkles|paperclip|tada|recycle|rewind|construction_worker):)\s[A-Z].*[^.])|(Merge|Revert|Reapply).+[^.])$'
|
||||||
flags: 'gm'
|
flags: 'gm'
|
||||||
@ -47,46 +50,4 @@ jobs:
|
|||||||
# pattern: '^.+(Resolves|Fixes): \#[0-9]+$'
|
# pattern: '^.+(Resolves|Fixes): \#[0-9]+$'
|
||||||
# error: 'You need at least one "Resolves|Fixes: #<issue number>" line.'
|
# error: 'You need at least one "Resolves|Fixes: #<issue number>" line.'
|
||||||
|
|
||||||
check-commit-guidelines:
|
|
||||||
if: ${{ !github.event.pull_request.draft }}
|
|
||||||
name: Check Commit Guidelines
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
steps:
|
|
||||||
# On PR events check out the base ref, so the script below is
|
|
||||||
# always the trusted version: PR commits are only read, never
|
|
||||||
# executed. On push events this resolves to the pushed sha.
|
|
||||||
- name: Checkout trusted code
|
|
||||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
|
||||||
with:
|
|
||||||
persist-credentials: false
|
|
||||||
fetch-depth: 0
|
|
||||||
ref: ${{ github.event.pull_request.base.sha || github.sha }}
|
|
||||||
|
|
||||||
- name: Fetch PR commits
|
|
||||||
if: ${{ github.event_name == 'pull_request' || github.event_name == 'pull_request_target' }}
|
|
||||||
run: git fetch --no-tags origin "pull/${{ github.event.pull_request.number }}/head:pr-head"
|
|
||||||
|
|
||||||
- name: Run scripts/check-commit on every new commit
|
|
||||||
run: |
|
|
||||||
set -euo pipefail
|
|
||||||
if [ "${{ github.event_name }}" = "push" ]; then
|
|
||||||
if [ "${{ github.event.before }}" = "0000000000000000000000000000000000000000" ]; then
|
|
||||||
RANGE="${{ github.sha }}"
|
|
||||||
else
|
|
||||||
RANGE="${{ github.event.before }}..${{ github.sha }}"
|
|
||||||
fi
|
|
||||||
else
|
|
||||||
RANGE="${{ github.event.pull_request.base.sha }}..pr-head"
|
|
||||||
fi
|
|
||||||
COMMITS="$(git rev-list --reverse "$RANGE")"
|
|
||||||
if [ -z "$COMMITS" ]; then
|
|
||||||
echo "No new commits to check."
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
FAILED=0
|
|
||||||
for COMMIT in $COMMITS; do
|
|
||||||
./scripts/check-commit -c "$COMMIT" || FAILED=1
|
|
||||||
done
|
|
||||||
exit "$FAILED"
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
78
.github/workflows/lint-workflows.yml
vendored
78
.github/workflows/lint-workflows.yml
vendored
@ -1,78 +0,0 @@
|
|||||||
name: "CI: Workflows"
|
|
||||||
|
|
||||||
# Static analysis of the GitHub Actions workflows themselves:
|
|
||||||
# - actionlint: syntax, expressions, runner labels, job/step references,
|
|
||||||
# and shellcheck on every `run:` block.
|
|
||||||
# - zizmor: security audits (template injection, dangerous triggers,
|
|
||||||
# credential persistence, ...). Configured in .github/zizmor.yml.
|
|
||||||
|
|
||||||
on:
|
|
||||||
pull_request:
|
|
||||||
types: [opened, synchronize, reopened, ready_for_review]
|
|
||||||
paths:
|
|
||||||
- '.github/workflows/**'
|
|
||||||
- '.github/actions/**'
|
|
||||||
- '.github/actionlint.yaml'
|
|
||||||
- '.github/zizmor.yml'
|
|
||||||
push:
|
|
||||||
branches:
|
|
||||||
- develop
|
|
||||||
- staging
|
|
||||||
- main
|
|
||||||
paths:
|
|
||||||
- '.github/workflows/**'
|
|
||||||
- '.github/actions/**'
|
|
||||||
- '.github/actionlint.yaml'
|
|
||||||
- '.github/zizmor.yml'
|
|
||||||
workflow_dispatch:
|
|
||||||
|
|
||||||
concurrency:
|
|
||||||
group: lint-workflows-${{ github.ref }}
|
|
||||||
cancel-in-progress: true
|
|
||||||
|
|
||||||
permissions: {}
|
|
||||||
|
|
||||||
env:
|
|
||||||
ACTIONLINT_VERSION: '1.7.12'
|
|
||||||
ZIZMOR_VERSION: '1.32.0'
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
lint:
|
|
||||||
name: Lint workflows
|
|
||||||
if: ${{ !github.event.pull_request.draft }}
|
|
||||||
# PR code is untrusted: never run this on the self-hosted runners.
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
timeout-minutes: 10
|
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
|
|
||||||
steps:
|
|
||||||
- name: Checkout
|
|
||||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
||||||
with:
|
|
||||||
persist-credentials: false
|
|
||||||
|
|
||||||
- name: Install actionlint
|
|
||||||
run: |
|
|
||||||
curl -fsSL --retry 3 \
|
|
||||||
"https://github.com/rhysd/actionlint/releases/download/v${ACTIONLINT_VERSION}/actionlint_${ACTIONLINT_VERSION}_linux_amd64.tar.gz" \
|
|
||||||
| tar -xz -C "$RUNNER_TEMP" actionlint
|
|
||||||
echo "$RUNNER_TEMP" >> "$GITHUB_PATH"
|
|
||||||
|
|
||||||
# Each finding is emitted as a `::error` annotation on the PR diff.
|
|
||||||
- name: Run actionlint
|
|
||||||
run: |
|
|
||||||
# shellcheck disable=SC2016 # Go template, not shell expansion.
|
|
||||||
actionlint -format '{{range $err := .}}::error file={{$err.Filepath}},line={{$err.Line}},col={{$err.Column}},title=actionlint ({{$err.Kind}})::{{$err.Message}}\n{{end}}'
|
|
||||||
|
|
||||||
- name: Run zizmor
|
|
||||||
if: ${{ !cancelled() }}
|
|
||||||
env:
|
|
||||||
# Enables the online audits (known-vulnerable-actions,
|
|
||||||
# impostor-commit, ref-confusion, ...).
|
|
||||||
GH_TOKEN: ${{ github.token }}
|
|
||||||
run: |
|
|
||||||
pipx run "zizmor==${ZIZMOR_VERSION}" \
|
|
||||||
--config .github/zizmor.yml \
|
|
||||||
--format github \
|
|
||||||
.github/workflows
|
|
||||||
49
.github/workflows/plugins-deploy-api-doc.yml
vendored
49
.github/workflows/plugins-deploy-api-doc.yml
vendored
@ -33,32 +33,48 @@ jobs:
|
|||||||
steps:
|
steps:
|
||||||
- name: Extract some useful variables
|
- name: Extract some useful variables
|
||||||
id: vars
|
id: vars
|
||||||
env:
|
|
||||||
GH_REF: ${{ inputs.gh_ref || github.ref_name }}
|
|
||||||
run: |
|
run: |
|
||||||
echo "gh_ref=$GH_REF" >> $GITHUB_OUTPUT
|
echo "gh_ref=${{ inputs.gh_ref || github.ref_name }}" >> $GITHUB_OUTPUT
|
||||||
|
|
||||||
- name: Checkout
|
- name: Checkout
|
||||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
uses: actions/checkout@v6
|
||||||
with:
|
with:
|
||||||
persist-credentials: false
|
|
||||||
fetch-depth: 0
|
fetch-depth: 0
|
||||||
ref: ${{ steps.vars.outputs.gh_ref }}
|
ref: ${{ steps.vars.outputs.gh_ref }}
|
||||||
|
|
||||||
# No dependency cache here: a deploy must not restore a cache that
|
# START: Setup Node and PNPM enabling cache
|
||||||
# other workflows can write to.
|
- name: Setup Node.js
|
||||||
- name: Setup pnpm + Node.js
|
uses: actions/setup-node@v6
|
||||||
uses: pnpm/setup@731dfb6463bbd11c11fd0ee11fe00ed0e36d48a1 # v3.2.0
|
|
||||||
with:
|
with:
|
||||||
working-directory: plugins
|
node-version-file: .nvmrc
|
||||||
runtime: node@24.22.3
|
|
||||||
install: false
|
- name: Enable PNPM
|
||||||
|
working-directory: ./plugins
|
||||||
|
shell: bash
|
||||||
|
run: |
|
||||||
|
corepack enable;
|
||||||
|
corepack install;
|
||||||
|
|
||||||
|
- name: Get pnpm store path
|
||||||
|
id: pnpm-store
|
||||||
|
working-directory: ./plugins
|
||||||
|
shell: bash
|
||||||
|
run: echo "STORE_PATH=$(pnpm store path --silent)" >> $GITHUB_OUTPUT
|
||||||
|
|
||||||
|
- name: Cache pnpm store
|
||||||
|
uses: actions/cache@v5
|
||||||
|
with:
|
||||||
|
path: ${{ steps.pnpm-store.outputs.STORE_PATH }}
|
||||||
|
key: ${{ runner.os }}-pnpm-${{ hashFiles('plugins/pnpm-lock.yaml') }}
|
||||||
|
restore-keys: |
|
||||||
|
${{ runner.os }}-pnpm-
|
||||||
|
# END: Setup Node and PNPM enabling cache
|
||||||
|
|
||||||
- name: Install deps
|
- name: Install deps
|
||||||
working-directory: ./plugins
|
working-directory: ./plugins
|
||||||
shell: bash
|
shell: bash
|
||||||
run: |
|
run: |
|
||||||
pnpm install --frozen-lockfile;
|
pnpm install --no-frozen-lockfile;
|
||||||
pnpm add -D -w wrangler@latest;
|
pnpm add -D -w wrangler@latest;
|
||||||
|
|
||||||
- name: Build docs
|
- name: Build docs
|
||||||
@ -67,9 +83,8 @@ jobs:
|
|||||||
run: pnpm run build:doc
|
run: pnpm run build:doc
|
||||||
|
|
||||||
- name: Select Worker name
|
- name: Select Worker name
|
||||||
env:
|
|
||||||
REF: ${{ steps.vars.outputs.gh_ref }}
|
|
||||||
run: |
|
run: |
|
||||||
|
REF="${{ steps.vars.outputs.gh_ref }}"
|
||||||
case "$REF" in
|
case "$REF" in
|
||||||
main)
|
main)
|
||||||
echo "WORKER_NAME=penpot-plugins-api-doc-pro" >> $GITHUB_ENV
|
echo "WORKER_NAME=penpot-plugins-api-doc-pro" >> $GITHUB_ENV
|
||||||
@ -87,7 +102,7 @@ jobs:
|
|||||||
working-directory: plugins
|
working-directory: plugins
|
||||||
shell: bash
|
shell: bash
|
||||||
run: |
|
run: |
|
||||||
sed -i "s/WORKER_URI/${WORKER_URI}/g" wrangler-penpot-plugins-api-doc.toml
|
sed -i "s/WORKER_URI/${{ env.WORKER_URI }}/g" wrangler-penpot-plugins-api-doc.toml
|
||||||
|
|
||||||
- name: Add noindex header and robots.txt files for non-production environments
|
- name: Add noindex header and robots.txt files for non-production environments
|
||||||
if: ${{ steps.vars.outputs.gh_ref != 'main' }}
|
if: ${{ steps.vars.outputs.gh_ref != 'main' }}
|
||||||
@ -107,7 +122,7 @@ jobs:
|
|||||||
EOF
|
EOF
|
||||||
|
|
||||||
- name: Deploy to Cloudflare Workers
|
- name: Deploy to Cloudflare Workers
|
||||||
uses: cloudflare/wrangler-action@953926a2e2182532811c01a25e53647d93bf07c0 # v4.1.3
|
uses: cloudflare/wrangler-action@v3
|
||||||
with:
|
with:
|
||||||
workingDirectory: plugins
|
workingDirectory: plugins
|
||||||
apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
|
apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
|
||||||
|
|||||||
86
.github/workflows/plugins-deploy-package.yml
vendored
86
.github/workflows/plugins-deploy-package.yml
vendored
@ -23,17 +23,11 @@ on:
|
|||||||
description: 'Name of the branch'
|
description: 'Name of the branch'
|
||||||
type: string
|
type: string
|
||||||
required: true
|
required: true
|
||||||
|
default: 'develop'
|
||||||
plugin_name:
|
plugin_name:
|
||||||
description: 'Publig name (from plugins/apps/<plugin_name>-plugin)'
|
description: 'Publig name (from plugins/apps/<plugin_name>-plugin)'
|
||||||
type: string
|
type: string
|
||||||
required: true
|
required: true
|
||||||
secrets:
|
|
||||||
CLOUDFLARE_ACCOUNT_ID:
|
|
||||||
required: false
|
|
||||||
CLOUDFLARE_API_TOKEN:
|
|
||||||
required: false
|
|
||||||
MATTERMOST_WEBHOOK:
|
|
||||||
required: false
|
|
||||||
|
|
||||||
permissions:
|
permissions:
|
||||||
contents: read
|
contents: read
|
||||||
@ -41,71 +35,77 @@ permissions:
|
|||||||
jobs:
|
jobs:
|
||||||
deploy:
|
deploy:
|
||||||
runs-on: penpot-standar-runner
|
runs-on: penpot-standar-runner
|
||||||
container:
|
|
||||||
image: penpotapp/devenv:latest
|
|
||||||
volumes:
|
|
||||||
# Persistent pnpm store shared by all runner instances on the host.
|
|
||||||
# pnpm's content-addressable store is safe for concurrent access.
|
|
||||||
- /var/cache/github-runner/pnpm-store:/root/.local/share/pnpm/store
|
|
||||||
defaults:
|
|
||||||
run:
|
|
||||||
shell: bash
|
|
||||||
env:
|
|
||||||
GH_REF: ${{ inputs.gh_ref }}
|
|
||||||
PLUGIN_NAME: ${{ inputs.plugin_name }}
|
|
||||||
steps:
|
steps:
|
||||||
# plugin_name is a free-form string that ends up in paths, worker names
|
|
||||||
# and GITHUB_ENV; reject anything that is not a plain slug before it is
|
|
||||||
# used anywhere.
|
|
||||||
- name: Validate inputs
|
|
||||||
run: |
|
|
||||||
if ! [[ "$PLUGIN_NAME" =~ ^[a-z0-9][a-z0-9-]*$ ]]; then
|
|
||||||
echo "::error::Invalid plugin_name: must match ^[a-z0-9][a-z0-9-]*$"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
- name: Checkout
|
- name: Checkout
|
||||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
uses: actions/checkout@v6
|
||||||
with:
|
with:
|
||||||
persist-credentials: false
|
|
||||||
fetch-depth: 0
|
fetch-depth: 0
|
||||||
ref: ${{ inputs.gh_ref }}
|
ref: ${{ inputs.gh_ref }}
|
||||||
|
|
||||||
|
# START: Setup Node and PNPM enabling cache
|
||||||
|
- name: Setup Node.js
|
||||||
|
uses: actions/setup-node@v6
|
||||||
|
with:
|
||||||
|
node-version-file: .nvmrc
|
||||||
|
|
||||||
|
- name: Enable PNPM
|
||||||
|
working-directory: ./plugins
|
||||||
|
shell: bash
|
||||||
|
run: |
|
||||||
|
corepack enable;
|
||||||
|
corepack install;
|
||||||
|
|
||||||
|
- name: Get pnpm store path
|
||||||
|
id: pnpm-store
|
||||||
|
working-directory: ./plugins
|
||||||
|
shell: bash
|
||||||
|
run: echo "STORE_PATH=$(pnpm store path --silent)" >> $GITHUB_OUTPUT
|
||||||
|
|
||||||
|
- name: Cache pnpm store
|
||||||
|
uses: actions/cache@v5
|
||||||
|
with:
|
||||||
|
path: ${{ steps.pnpm-store.outputs.STORE_PATH }}
|
||||||
|
key: ${{ runner.os }}-pnpm-${{ hashFiles('plugins/pnpm-lock.yaml') }}
|
||||||
|
restore-keys: |
|
||||||
|
${{ runner.os }}-pnpm-
|
||||||
|
# END: Setup Node and PNPM enabling cache
|
||||||
|
|
||||||
- name: Install deps
|
- name: Install deps
|
||||||
working-directory: ./plugins
|
working-directory: ./plugins
|
||||||
shell: bash
|
shell: bash
|
||||||
run: |
|
run: |
|
||||||
pnpm install --frozen-lockfile;
|
pnpm install --no-frozen-lockfile;
|
||||||
pnpm add -D -w wrangler@latest;
|
pnpm add -D -w wrangler@latest;
|
||||||
|
|
||||||
- name: "Build package for ${{ inputs.plugin_name }}-plugin"
|
- name: "Build package for ${{ inputs.plugin_name }}-plugin"
|
||||||
working-directory: plugins
|
working-directory: plugins
|
||||||
shell: bash
|
shell: bash
|
||||||
run: pnpm --filter "${PLUGIN_NAME}-plugin" build
|
run: pnpm --filter ${{ inputs.plugin_name }}-plugin build
|
||||||
|
|
||||||
- name: Select Worker name
|
- name: Select Worker name
|
||||||
run: |
|
run: |
|
||||||
case "$GH_REF" in
|
REF="${{ inputs.gh_ref }}"
|
||||||
|
case "$REF" in
|
||||||
main)
|
main)
|
||||||
echo "WORKER_NAME=${PLUGIN_NAME}-plugin-pro" >> $GITHUB_ENV
|
echo "WORKER_NAME=${{ inputs.plugin_name }}-plugin-pro" >> $GITHUB_ENV
|
||||||
echo "WORKER_URI=${PLUGIN_NAME}.plugins.penpot.app" >> $GITHUB_ENV ;;
|
echo "WORKER_URI=${{ inputs.plugin_name }}.plugins.penpot.app" >> $GITHUB_ENV ;;
|
||||||
staging)
|
staging)
|
||||||
echo "WORKER_NAME=${PLUGIN_NAME}-plugin-pre" >> $GITHUB_ENV
|
echo "WORKER_NAME=${{ inputs.plugin_name }}-plugin-pre" >> $GITHUB_ENV
|
||||||
echo "WORKER_URI=${PLUGIN_NAME}.plugins.penpot.dev" >> $GITHUB_ENV ;;
|
echo "WORKER_URI=${{ inputs.plugin_name }}.plugins.penpot.dev" >> $GITHUB_ENV ;;
|
||||||
develop)
|
develop)
|
||||||
echo "WORKER_NAME=${PLUGIN_NAME}-plugin-hourly" >> $GITHUB_ENV
|
echo "WORKER_NAME=${{ inputs.plugin_name }}-plugin-hourly" >> $GITHUB_ENV
|
||||||
echo "WORKER_URI=${PLUGIN_NAME}.plugins.hourly.penpot.dev" >> $GITHUB_ENV ;;
|
echo "WORKER_URI=${{ inputs.plugin_name }}.plugins.hourly.penpot.dev" >> $GITHUB_ENV ;;
|
||||||
*) echo "Unsupported branch ${GH_REF}" && exit 1 ;;
|
*) echo "Unsupported branch ${REF}" && exit 1 ;;
|
||||||
esac
|
esac
|
||||||
|
|
||||||
- name: Set the custom url
|
- name: Set the custom url
|
||||||
working-directory: plugins
|
working-directory: plugins
|
||||||
shell: bash
|
shell: bash
|
||||||
run: |
|
run: |
|
||||||
sed -i "s/WORKER_URI/${WORKER_URI}/g" "apps/${PLUGIN_NAME}-plugin/wrangler.toml"
|
sed -i "s/WORKER_URI/${{ env.WORKER_URI }}/g" apps/${{ inputs.plugin_name }}-plugin/wrangler.toml
|
||||||
|
|
||||||
- name: Deploy to Cloudflare Workers
|
- name: Deploy to Cloudflare Workers
|
||||||
uses: cloudflare/wrangler-action@953926a2e2182532811c01a25e53647d93bf07c0 # v4.1.3
|
uses: cloudflare/wrangler-action@v3
|
||||||
with:
|
with:
|
||||||
workingDirectory: plugins
|
workingDirectory: plugins
|
||||||
apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
|
apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
|
||||||
|
|||||||
44
.github/workflows/plugins-deploy-packages.yml
vendored
44
.github/workflows/plugins-deploy-packages.yml
vendored
@ -21,9 +21,6 @@ on:
|
|||||||
- staging
|
- staging
|
||||||
- main
|
- main
|
||||||
|
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
detect-changes:
|
detect-changes:
|
||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
@ -39,11 +36,9 @@ jobs:
|
|||||||
# [For new plugins]
|
# [For new plugins]
|
||||||
# Add more outputs here
|
# Add more outputs here
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
- uses: actions/checkout@v6
|
||||||
with:
|
|
||||||
persist-credentials: false
|
|
||||||
- id: filter
|
- id: filter
|
||||||
uses: dorny/paths-filter@ceb8a2b8f2d89434be7ff52d3de7ec3738c5cc9d # v4.0.3
|
uses: dorny/paths-filter@v4
|
||||||
with:
|
with:
|
||||||
filters: |
|
filters: |
|
||||||
colors_to_tokens:
|
colors_to_tokens:
|
||||||
@ -77,10 +72,7 @@ jobs:
|
|||||||
needs: detect-changes
|
needs: detect-changes
|
||||||
if: github.event_name == 'workflow_dispatch' || needs.detect-changes.outputs.colors_to_tokens == 'true'
|
if: github.event_name == 'workflow_dispatch' || needs.detect-changes.outputs.colors_to_tokens == 'true'
|
||||||
uses: ./.github/workflows/plugins-deploy-package.yml
|
uses: ./.github/workflows/plugins-deploy-package.yml
|
||||||
secrets:
|
secrets: inherit
|
||||||
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
|
|
||||||
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
|
|
||||||
MATTERMOST_WEBHOOK: ${{ secrets.MATTERMOST_WEBHOOK }}
|
|
||||||
with:
|
with:
|
||||||
gh_ref: "${{ inputs.gh_ref || github.ref_name }}"
|
gh_ref: "${{ inputs.gh_ref || github.ref_name }}"
|
||||||
plugin_name: colors-to-tokens
|
plugin_name: colors-to-tokens
|
||||||
@ -89,10 +81,7 @@ jobs:
|
|||||||
needs: detect-changes
|
needs: detect-changes
|
||||||
if: github.event_name == 'workflow_dispatch' || needs.detect-changes.outputs.contrast == 'true'
|
if: github.event_name == 'workflow_dispatch' || needs.detect-changes.outputs.contrast == 'true'
|
||||||
uses: ./.github/workflows/plugins-deploy-package.yml
|
uses: ./.github/workflows/plugins-deploy-package.yml
|
||||||
secrets:
|
secrets: inherit
|
||||||
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
|
|
||||||
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
|
|
||||||
MATTERMOST_WEBHOOK: ${{ secrets.MATTERMOST_WEBHOOK }}
|
|
||||||
with:
|
with:
|
||||||
gh_ref: "${{ inputs.gh_ref || github.ref_name }}"
|
gh_ref: "${{ inputs.gh_ref || github.ref_name }}"
|
||||||
plugin_name: contrast
|
plugin_name: contrast
|
||||||
@ -101,10 +90,7 @@ jobs:
|
|||||||
needs: detect-changes
|
needs: detect-changes
|
||||||
if: github.event_name == 'workflow_dispatch' || needs.detect-changes.outputs.create_palette == 'true'
|
if: github.event_name == 'workflow_dispatch' || needs.detect-changes.outputs.create_palette == 'true'
|
||||||
uses: ./.github/workflows/plugins-deploy-package.yml
|
uses: ./.github/workflows/plugins-deploy-package.yml
|
||||||
secrets:
|
secrets: inherit
|
||||||
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
|
|
||||||
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
|
|
||||||
MATTERMOST_WEBHOOK: ${{ secrets.MATTERMOST_WEBHOOK }}
|
|
||||||
with:
|
with:
|
||||||
gh_ref: "${{ inputs.gh_ref || github.ref_name }}"
|
gh_ref: "${{ inputs.gh_ref || github.ref_name }}"
|
||||||
plugin_name: create-palette
|
plugin_name: create-palette
|
||||||
@ -113,10 +99,7 @@ jobs:
|
|||||||
needs: detect-changes
|
needs: detect-changes
|
||||||
if: github.event_name == 'workflow_dispatch' || needs.detect-changes.outputs.icons == 'true'
|
if: github.event_name == 'workflow_dispatch' || needs.detect-changes.outputs.icons == 'true'
|
||||||
uses: ./.github/workflows/plugins-deploy-package.yml
|
uses: ./.github/workflows/plugins-deploy-package.yml
|
||||||
secrets:
|
secrets: inherit
|
||||||
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
|
|
||||||
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
|
|
||||||
MATTERMOST_WEBHOOK: ${{ secrets.MATTERMOST_WEBHOOK }}
|
|
||||||
with:
|
with:
|
||||||
gh_ref: "${{ inputs.gh_ref || github.ref_name }}"
|
gh_ref: "${{ inputs.gh_ref || github.ref_name }}"
|
||||||
plugin_name: icons
|
plugin_name: icons
|
||||||
@ -125,10 +108,7 @@ jobs:
|
|||||||
needs: detect-changes
|
needs: detect-changes
|
||||||
if: github.event_name == 'workflow_dispatch' || needs.detect-changes.outputs.lorem_ipsum == 'true'
|
if: github.event_name == 'workflow_dispatch' || needs.detect-changes.outputs.lorem_ipsum == 'true'
|
||||||
uses: ./.github/workflows/plugins-deploy-package.yml
|
uses: ./.github/workflows/plugins-deploy-package.yml
|
||||||
secrets:
|
secrets: inherit
|
||||||
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
|
|
||||||
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
|
|
||||||
MATTERMOST_WEBHOOK: ${{ secrets.MATTERMOST_WEBHOOK }}
|
|
||||||
with:
|
with:
|
||||||
gh_ref: "${{ inputs.gh_ref || github.ref_name }}"
|
gh_ref: "${{ inputs.gh_ref || github.ref_name }}"
|
||||||
plugin_name: lorem-ipsum
|
plugin_name: lorem-ipsum
|
||||||
@ -137,10 +117,7 @@ jobs:
|
|||||||
needs: detect-changes
|
needs: detect-changes
|
||||||
if: github.event_name == 'workflow_dispatch' || needs.detect-changes.outputs.rename_layers == 'true'
|
if: github.event_name == 'workflow_dispatch' || needs.detect-changes.outputs.rename_layers == 'true'
|
||||||
uses: ./.github/workflows/plugins-deploy-package.yml
|
uses: ./.github/workflows/plugins-deploy-package.yml
|
||||||
secrets:
|
secrets: inherit
|
||||||
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
|
|
||||||
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
|
|
||||||
MATTERMOST_WEBHOOK: ${{ secrets.MATTERMOST_WEBHOOK }}
|
|
||||||
with:
|
with:
|
||||||
gh_ref: "${{ inputs.gh_ref || github.ref_name }}"
|
gh_ref: "${{ inputs.gh_ref || github.ref_name }}"
|
||||||
plugin_name: rename-layers
|
plugin_name: rename-layers
|
||||||
@ -149,10 +126,7 @@ jobs:
|
|||||||
needs: detect-changes
|
needs: detect-changes
|
||||||
if: github.event_name == 'workflow_dispatch' || needs.detect-changes.outputs.table == 'true'
|
if: github.event_name == 'workflow_dispatch' || needs.detect-changes.outputs.table == 'true'
|
||||||
uses: ./.github/workflows/plugins-deploy-package.yml
|
uses: ./.github/workflows/plugins-deploy-package.yml
|
||||||
secrets:
|
secrets: inherit
|
||||||
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
|
|
||||||
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
|
|
||||||
MATTERMOST_WEBHOOK: ${{ secrets.MATTERMOST_WEBHOOK }}
|
|
||||||
with:
|
with:
|
||||||
gh_ref: "${{ inputs.gh_ref || github.ref_name }}"
|
gh_ref: "${{ inputs.gh_ref || github.ref_name }}"
|
||||||
plugin_name: table
|
plugin_name: table
|
||||||
|
|||||||
49
.github/workflows/plugins-deploy-styles-doc.yml
vendored
49
.github/workflows/plugins-deploy-styles-doc.yml
vendored
@ -31,32 +31,48 @@ jobs:
|
|||||||
steps:
|
steps:
|
||||||
- name: Extract some useful variables
|
- name: Extract some useful variables
|
||||||
id: vars
|
id: vars
|
||||||
env:
|
|
||||||
GH_REF: ${{ inputs.gh_ref || github.ref_name }}
|
|
||||||
run: |
|
run: |
|
||||||
echo "gh_ref=$GH_REF" >> $GITHUB_OUTPUT
|
echo "gh_ref=${{ inputs.gh_ref || github.ref_name }}" >> $GITHUB_OUTPUT
|
||||||
|
|
||||||
- name: Checkout
|
- name: Checkout
|
||||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
uses: actions/checkout@v6
|
||||||
with:
|
with:
|
||||||
persist-credentials: false
|
|
||||||
fetch-depth: 0
|
fetch-depth: 0
|
||||||
ref: ${{ steps.vars.outputs.gh_ref }}
|
ref: ${{ steps.vars.outputs.gh_ref }}
|
||||||
|
|
||||||
# No dependency cache here: a deploy must not restore a cache that
|
# START: Setup Node and PNPM enabling cache
|
||||||
# other workflows can write to.
|
- name: Setup Node.js
|
||||||
- name: Setup pnpm + Node.js
|
uses: actions/setup-node@v6
|
||||||
uses: pnpm/setup@731dfb6463bbd11c11fd0ee11fe00ed0e36d48a1 # v3.2.0
|
|
||||||
with:
|
with:
|
||||||
working-directory: plugins
|
node-version-file: .nvmrc
|
||||||
runtime: node@24.22.3
|
|
||||||
install: false
|
- name: Enable PNPM
|
||||||
|
working-directory: ./plugins
|
||||||
|
shell: bash
|
||||||
|
run: |
|
||||||
|
corepack enable;
|
||||||
|
corepack install;
|
||||||
|
|
||||||
|
- name: Get pnpm store path
|
||||||
|
id: pnpm-store
|
||||||
|
working-directory: ./plugins
|
||||||
|
shell: bash
|
||||||
|
run: echo "STORE_PATH=$(pnpm store path --silent)" >> $GITHUB_OUTPUT
|
||||||
|
|
||||||
|
- name: Cache pnpm store
|
||||||
|
uses: actions/cache@v5
|
||||||
|
with:
|
||||||
|
path: ${{ steps.pnpm-store.outputs.STORE_PATH }}
|
||||||
|
key: ${{ runner.os }}-pnpm-${{ hashFiles('plugins/pnpm-lock.yaml') }}
|
||||||
|
restore-keys: |
|
||||||
|
${{ runner.os }}-pnpm-
|
||||||
|
# END: Setup Node and PNPM enabling cache
|
||||||
|
|
||||||
- name: Install deps
|
- name: Install deps
|
||||||
working-directory: ./plugins
|
working-directory: ./plugins
|
||||||
shell: bash
|
shell: bash
|
||||||
run: |
|
run: |
|
||||||
pnpm install --frozen-lockfile;
|
pnpm install --no-frozen-lockfile;
|
||||||
pnpm add -D -w wrangler@latest;
|
pnpm add -D -w wrangler@latest;
|
||||||
|
|
||||||
- name: Build styles
|
- name: Build styles
|
||||||
@ -65,9 +81,8 @@ jobs:
|
|||||||
run: pnpm run build:styles-example
|
run: pnpm run build:styles-example
|
||||||
|
|
||||||
- name: Select Worker name
|
- name: Select Worker name
|
||||||
env:
|
|
||||||
REF: ${{ steps.vars.outputs.gh_ref }}
|
|
||||||
run: |
|
run: |
|
||||||
|
REF="${{ steps.vars.outputs.gh_ref }}"
|
||||||
case "$REF" in
|
case "$REF" in
|
||||||
main)
|
main)
|
||||||
echo "WORKER_NAME=penpot-plugins-styles-doc-pro" >> $GITHUB_ENV
|
echo "WORKER_NAME=penpot-plugins-styles-doc-pro" >> $GITHUB_ENV
|
||||||
@ -85,7 +100,7 @@ jobs:
|
|||||||
working-directory: plugins
|
working-directory: plugins
|
||||||
shell: bash
|
shell: bash
|
||||||
run: |
|
run: |
|
||||||
sed -i "s/WORKER_URI/${WORKER_URI}/g" wrangler-penpot-plugins-styles-doc.toml
|
sed -i "s/WORKER_URI/${{ env.WORKER_URI }}/g" wrangler-penpot-plugins-styles-doc.toml
|
||||||
|
|
||||||
- name: Add noindex header and robots.txt files for non-production environments
|
- name: Add noindex header and robots.txt files for non-production environments
|
||||||
if: ${{ steps.vars.outputs.gh_ref != 'main' }}
|
if: ${{ steps.vars.outputs.gh_ref != 'main' }}
|
||||||
@ -105,7 +120,7 @@ jobs:
|
|||||||
EOF
|
EOF
|
||||||
|
|
||||||
- name: Deploy to Cloudflare Workers
|
- name: Deploy to Cloudflare Workers
|
||||||
uses: cloudflare/wrangler-action@953926a2e2182532811c01a25e53647d93bf07c0 # v4.1.3
|
uses: cloudflare/wrangler-action@v3
|
||||||
with:
|
with:
|
||||||
workingDirectory: plugins
|
workingDirectory: plugins
|
||||||
apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
|
apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
|
||||||
|
|||||||
45
.github/workflows/release.yml
vendored
45
.github/workflows/release.yml
vendored
@ -13,49 +13,31 @@ on:
|
|||||||
description: 'Tag to release'
|
description: 'Tag to release'
|
||||||
type: string
|
type: string
|
||||||
required: true
|
required: true
|
||||||
secrets:
|
|
||||||
AWS_REGION:
|
|
||||||
required: false
|
|
||||||
DOCKER_PASSWORD:
|
|
||||||
required: false
|
|
||||||
DOCKER_REGISTRY:
|
|
||||||
required: false
|
|
||||||
DOCKER_USERNAME:
|
|
||||||
required: false
|
|
||||||
MATTERMOST_WEBHOOK:
|
|
||||||
required: false
|
|
||||||
PUB_DOCKER_PASSWORD:
|
|
||||||
required: false
|
|
||||||
PUB_DOCKER_USERNAME:
|
|
||||||
required: false
|
|
||||||
|
|
||||||
permissions:
|
permissions:
|
||||||
contents: write
|
contents: write
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
release:
|
release:
|
||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-24.04
|
||||||
outputs:
|
outputs:
|
||||||
version: ${{ steps.vars.outputs.gh_ref }}
|
version: ${{ steps.vars.outputs.gh_ref }}
|
||||||
release_notes: ${{ steps.extract_release_notes.outputs.release_notes }}
|
release_notes: ${{ steps.extract_release_notes.outputs.release_notes }}
|
||||||
steps:
|
steps:
|
||||||
- name: Extract some useful variables
|
- name: Extract some useful variables
|
||||||
id: vars
|
id: vars
|
||||||
env:
|
|
||||||
GH_REF: ${{ inputs.gh_ref || github.ref_name }}
|
|
||||||
run: |
|
run: |
|
||||||
echo "gh_ref=$GH_REF" >> $GITHUB_OUTPUT
|
echo "gh_ref=${{ inputs.gh_ref || github.ref_name }}" >> $GITHUB_OUTPUT
|
||||||
|
|
||||||
- name: Checkout code
|
- name: Checkout code
|
||||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
uses: actions/checkout@v6
|
||||||
with:
|
with:
|
||||||
persist-credentials: false
|
|
||||||
fetch-depth: 0
|
fetch-depth: 0
|
||||||
ref: ${{ steps.vars.outputs.gh_ref }}
|
ref: ${{ steps.vars.outputs.gh_ref }}
|
||||||
|
|
||||||
# --- Publicly release the docker images ---
|
# --- Publicly release the docker images ---
|
||||||
- name: Configure ECR credentials
|
- name: Configure ECR credentials
|
||||||
uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6.3.0
|
uses: aws-actions/configure-aws-credentials@v4
|
||||||
with:
|
with:
|
||||||
aws-access-key-id: ${{ secrets.DOCKER_USERNAME }}
|
aws-access-key-id: ${{ secrets.DOCKER_USERNAME }}
|
||||||
aws-secret-access-key: ${{ secrets.DOCKER_PASSWORD }}
|
aws-secret-access-key: ${{ secrets.DOCKER_PASSWORD }}
|
||||||
@ -80,7 +62,7 @@ jobs:
|
|||||||
|
|
||||||
echo "$PUB_DOCKER_PASSWORD" | skopeo login --username "$PUB_DOCKER_USERNAME" --password-stdin docker.io
|
echo "$PUB_DOCKER_PASSWORD" | skopeo login --username "$PUB_DOCKER_USERNAME" --password-stdin docker.io
|
||||||
|
|
||||||
IMAGES=("frontend" "backend" "exporter" "admin-console" "mcp" "storybook")
|
IMAGES=("frontend" "backend" "exporter" "mcp" "storybook")
|
||||||
SHORT_TAG=${TAG%.*}
|
SHORT_TAG=${TAG%.*}
|
||||||
|
|
||||||
for image in "${IMAGES[@]}"; do
|
for image in "${IMAGES[@]}"; do
|
||||||
@ -110,19 +92,14 @@ jobs:
|
|||||||
echo "EOF" >> $GITHUB_OUTPUT
|
echo "EOF" >> $GITHUB_OUTPUT
|
||||||
|
|
||||||
# --- Create GitHub release ---
|
# --- Create GitHub release ---
|
||||||
# Updates the release if it already exists, so re-running is safe.
|
|
||||||
- name: Create GitHub release
|
- name: Create GitHub release
|
||||||
|
uses: softprops/action-gh-release@v2
|
||||||
env:
|
env:
|
||||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||||
GH_REPO: ${{ github.repository }}
|
with:
|
||||||
TAG: ${{ steps.vars.outputs.gh_ref }}
|
tag_name: ${{ steps.vars.outputs.gh_ref }}
|
||||||
NOTES: ${{ steps.extract_release_notes.outputs.release_notes }}
|
name: ${{ steps.vars.outputs.gh_ref }}
|
||||||
run: |
|
body: ${{ steps.extract_release_notes.outputs.release_notes }}
|
||||||
if gh release view "$TAG" > /dev/null 2>&1; then
|
|
||||||
gh release edit "$TAG" --title "$TAG" --notes "$NOTES"
|
|
||||||
else
|
|
||||||
gh release create "$TAG" --verify-tag --title "$TAG" --notes "$NOTES"
|
|
||||||
fi
|
|
||||||
|
|
||||||
- name: Notify Mattermost
|
- name: Notify Mattermost
|
||||||
if: failure()
|
if: failure()
|
||||||
|
|||||||
@ -1,76 +0,0 @@
|
|||||||
name: Require Milestone on Linked Issues
|
|
||||||
|
|
||||||
on:
|
|
||||||
pull_request:
|
|
||||||
types:
|
|
||||||
- opened
|
|
||||||
- reopened
|
|
||||||
- synchronize
|
|
||||||
- edited
|
|
||||||
- ready_for_review
|
|
||||||
- converted_to_draft
|
|
||||||
workflow_dispatch:
|
|
||||||
|
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
pull-requests: read
|
|
||||||
issues: read
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
check-linked-issues:
|
|
||||||
name: Require Milestones on Linked Issues
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
|
|
||||||
steps:
|
|
||||||
- name: Check linked issues
|
|
||||||
env:
|
|
||||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
||||||
PR_NUMBER: ${{ github.event.pull_request.number }}
|
|
||||||
IS_DRAFT: ${{ github.event.pull_request.draft }}
|
|
||||||
REPOSITORY: ${{ github.repository }}
|
|
||||||
run: |
|
|
||||||
if [ "$IS_DRAFT" = "true" ]; then
|
|
||||||
echo "Draft PR: milestone check skipped."
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
OWNER="${REPOSITORY%%/*}"
|
|
||||||
REPO="${REPOSITORY#*/}"
|
|
||||||
|
|
||||||
# shellcheck disable=SC2016 # GraphQL variables, not shell expansion.
|
|
||||||
ISSUES=$(gh api graphql \
|
|
||||||
-F owner="$OWNER" \
|
|
||||||
-F repo="$REPO" \
|
|
||||||
-F number="$PR_NUMBER" \
|
|
||||||
-f query='
|
|
||||||
query($owner: String!, $repo: String!, $number: Int!) {
|
|
||||||
repository(owner: $owner, name: $repo) {
|
|
||||||
pullRequest(number: $number) {
|
|
||||||
closingIssuesReferences(first: 100) {
|
|
||||||
nodes {
|
|
||||||
number
|
|
||||||
title
|
|
||||||
url
|
|
||||||
milestone {
|
|
||||||
title
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
')
|
|
||||||
|
|
||||||
MISSING=$(echo "$ISSUES" | jq -r '
|
|
||||||
.data.repository.pullRequest.closingIssuesReferences.nodes[]
|
|
||||||
| select(.milestone == null)
|
|
||||||
| "- #\(.number): \(.title) (\(.url))"
|
|
||||||
')
|
|
||||||
|
|
||||||
if [ -n "$MISSING" ]; then
|
|
||||||
echo "::error::The following linked issues have no milestone:"
|
|
||||||
echo "$MISSING"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo "All linked issues have a milestone."
|
|
||||||
45
.github/workflows/require-milestone.yml
vendored
45
.github/workflows/require-milestone.yml
vendored
@ -1,45 +0,0 @@
|
|||||||
name: Require Milestone
|
|
||||||
|
|
||||||
on:
|
|
||||||
pull_request:
|
|
||||||
types:
|
|
||||||
- opened
|
|
||||||
- reopened
|
|
||||||
- synchronize
|
|
||||||
- edited
|
|
||||||
- milestoned
|
|
||||||
- demilestoned
|
|
||||||
- ready_for_review
|
|
||||||
- converted_to_draft
|
|
||||||
|
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
pull-requests: read
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
check-milestone:
|
|
||||||
name: Require Milestone
|
|
||||||
runs-on: ubuntu-latest
|
|
||||||
|
|
||||||
steps:
|
|
||||||
- name: Check milestone
|
|
||||||
env:
|
|
||||||
GH_TOKEN: ${{ github.token }}
|
|
||||||
PR_NUMBER: ${{ github.event.pull_request.number }}
|
|
||||||
run: |
|
|
||||||
pr_api="repos/${GITHUB_REPOSITORY}/pulls/${PR_NUMBER}"
|
|
||||||
|
|
||||||
is_draft=$(gh api "$pr_api" --jq '.draft')
|
|
||||||
milestone=$(gh api "$pr_api" --jq '.milestone.title // ""')
|
|
||||||
|
|
||||||
if [ "$is_draft" = "true" ]; then
|
|
||||||
echo "Draft PR: milestone check skipped."
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [ -z "$milestone" ]; then
|
|
||||||
echo "::error::This pull request must have a milestone."
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo "Milestone assigned: $milestone"
|
|
||||||
11
.github/workflows/tests-backend.yml
vendored
11
.github/workflows/tests-backend.yml
vendored
@ -28,9 +28,6 @@ concurrency:
|
|||||||
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
||||||
cancel-in-progress: true
|
cancel-in-progress: true
|
||||||
|
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
test-backend:
|
test-backend:
|
||||||
if: ${{ !github.event.pull_request.draft }}
|
if: ${{ !github.event.pull_request.draft }}
|
||||||
@ -63,9 +60,7 @@ jobs:
|
|||||||
|
|
||||||
steps:
|
steps:
|
||||||
- name: Checkout repository
|
- name: Checkout repository
|
||||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
uses: actions/checkout@v6
|
||||||
with:
|
|
||||||
persist-credentials: false
|
|
||||||
|
|
||||||
- name: Fmt
|
- name: Fmt
|
||||||
working-directory: ./backend
|
working-directory: ./backend
|
||||||
@ -80,10 +75,6 @@ jobs:
|
|||||||
- name: Tests
|
- name: Tests
|
||||||
working-directory: ./backend
|
working-directory: ./backend
|
||||||
env:
|
env:
|
||||||
# `clojure.core/assert` is elided at compile time, so without this
|
|
||||||
# flag the precondition asserts do not exist in the compiled code
|
|
||||||
# and the suite silently stops checking them.
|
|
||||||
PENPOT_FLAGS: "enable-backend-asserts"
|
|
||||||
PENPOT_TEST_DATABASE_URI: "postgresql://postgres/penpot_test"
|
PENPOT_TEST_DATABASE_URI: "postgresql://postgres/penpot_test"
|
||||||
PENPOT_TEST_DATABASE_USERNAME: penpot_test
|
PENPOT_TEST_DATABASE_USERNAME: penpot_test
|
||||||
PENPOT_TEST_DATABASE_PASSWORD: penpot_test
|
PENPOT_TEST_DATABASE_PASSWORD: penpot_test
|
||||||
|
|||||||
9
.github/workflows/tests-common.yml
vendored
9
.github/workflows/tests-common.yml
vendored
@ -26,9 +26,6 @@ concurrency:
|
|||||||
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
||||||
cancel-in-progress: true
|
cancel-in-progress: true
|
||||||
|
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
test-common:
|
test-common:
|
||||||
if: ${{ !github.event.pull_request.draft }}
|
if: ${{ !github.event.pull_request.draft }}
|
||||||
@ -42,13 +39,13 @@ jobs:
|
|||||||
|
|
||||||
steps:
|
steps:
|
||||||
- name: Checkout repository
|
- name: Checkout repository
|
||||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
uses: actions/checkout@v6
|
||||||
with:
|
|
||||||
persist-credentials: false
|
|
||||||
|
|
||||||
- name: Lint
|
- name: Lint
|
||||||
working-directory: ./common
|
working-directory: ./common
|
||||||
run: |
|
run: |
|
||||||
|
corepack enable;
|
||||||
|
corepack install;
|
||||||
pnpm install;
|
pnpm install;
|
||||||
pnpm run check-fmt:clj
|
pnpm run check-fmt:clj
|
||||||
pnpm run check-fmt:js
|
pnpm run check-fmt:js
|
||||||
|
|||||||
69
.github/workflows/tests-composable-suite.yml
vendored
Normal file
69
.github/workflows/tests-composable-suite.yml
vendored
Normal file
@ -0,0 +1,69 @@
|
|||||||
|
name: "CI: Composable Test Suite"
|
||||||
|
|
||||||
|
# Runs the composable component test suite (it exercises component semantics
|
||||||
|
# through the real Plugin API against the full frontend, so it needs the
|
||||||
|
# frontend bundle + the plugin runtime, but no backend): the driver serves the
|
||||||
|
# prebuilt frontend bundle and intercepts every backend RPC with Playwright
|
||||||
|
# fixtures. See plugins/apps/composable-test-suite/README.md ("Running in CI").
|
||||||
|
|
||||||
|
defaults:
|
||||||
|
run:
|
||||||
|
shell: bash
|
||||||
|
|
||||||
|
on:
|
||||||
|
pull_request:
|
||||||
|
paths:
|
||||||
|
- 'plugins/**'
|
||||||
|
- 'frontend/**'
|
||||||
|
- 'common/**'
|
||||||
|
types:
|
||||||
|
- opened
|
||||||
|
- synchronize
|
||||||
|
- ready_for_review
|
||||||
|
|
||||||
|
push:
|
||||||
|
branches:
|
||||||
|
- develop
|
||||||
|
- staging
|
||||||
|
paths:
|
||||||
|
- 'plugins/**'
|
||||||
|
- 'frontend/**'
|
||||||
|
- 'common/**'
|
||||||
|
|
||||||
|
concurrency:
|
||||||
|
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
||||||
|
cancel-in-progress: true
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
composable-test-suite:
|
||||||
|
if: ${{ !github.event.pull_request.draft }}
|
||||||
|
name: "Run composable test suite (mocked backend)"
|
||||||
|
runs-on: penpot-extended-runner
|
||||||
|
container:
|
||||||
|
image: penpotapp/devenv:latest
|
||||||
|
volumes:
|
||||||
|
- /var/cache/github-runner/m2:/root/.m2
|
||||||
|
- /var/cache/github-runner/gitlib:/root/.gitlibs
|
||||||
|
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v6
|
||||||
|
|
||||||
|
# The driver serves the prebuilt bundle from frontend/resources/public.
|
||||||
|
- name: Build frontend bundle
|
||||||
|
working-directory: ./frontend
|
||||||
|
run: ./scripts/build
|
||||||
|
|
||||||
|
- name: Install deps
|
||||||
|
working-directory: ./plugins
|
||||||
|
run: |
|
||||||
|
corepack enable;
|
||||||
|
corepack install;
|
||||||
|
pnpm install;
|
||||||
|
|
||||||
|
- name: Install Playwright Chromium
|
||||||
|
working-directory: ./plugins
|
||||||
|
run: pnpm --filter composable-test-suite exec playwright install --with-deps chromium
|
||||||
|
|
||||||
|
- name: Run composable test suite (mocked)
|
||||||
|
working-directory: ./plugins
|
||||||
|
run: pnpm --filter composable-test-suite run test:ci
|
||||||
430
.github/workflows/tests-e2e.yml
vendored
430
.github/workflows/tests-e2e.yml
vendored
@ -1,430 +0,0 @@
|
|||||||
name: "CI: E2E"
|
|
||||||
|
|
||||||
# Single entry point for every suite that drives a real frontend bundle.
|
|
||||||
# The bundle is built ONCE by `build-bundle` and restored by every consumer,
|
|
||||||
# so adding a suite no longer adds another `frontend/scripts/build` run.
|
|
||||||
#
|
|
||||||
# Suites and what they need from the bundle:
|
|
||||||
#
|
|
||||||
# - Integration Tests: Playwright specs, backend faked with frontend mocks.
|
|
||||||
# - Composable test suite: Plugin runtime, backend faked with Playwright RPC
|
|
||||||
# fixtures. See plugins/apps/composable-test-suite/README.md ("Running in CI").
|
|
||||||
# - Plugin API Test Suite (mocked): Plugin runtime, backend faked with
|
|
||||||
# Playwright RPC fixtures and MOCK_BACKEND=1.
|
|
||||||
# See plugins/apps/plugin-api-test-suite/README.md.
|
|
||||||
#
|
|
||||||
# Hand-off: `build-bundle` restores `actions/cache` keyed by the checked-out
|
|
||||||
# SHA, builds only on a miss, and saves it; every consumer restores the same
|
|
||||||
# key after `needs: build-bundle` completes. A re-run of the same SHA reuses
|
|
||||||
# the cached bundle instead of rebuilding it.
|
|
||||||
#
|
|
||||||
# NOTE: the job `name:` values are the GitHub check contexts, so they are kept
|
|
||||||
# stable on purpose (branch protection may require them by name). Only the
|
|
||||||
# workflow file/name changed.
|
|
||||||
|
|
||||||
defaults:
|
|
||||||
run:
|
|
||||||
shell: bash
|
|
||||||
|
|
||||||
on:
|
|
||||||
workflow_dispatch:
|
|
||||||
inputs:
|
|
||||||
gh_ref:
|
|
||||||
description: 'Name of the branch or ref'
|
|
||||||
type: string
|
|
||||||
required: true
|
|
||||||
default: 'develop'
|
|
||||||
|
|
||||||
shards:
|
|
||||||
description: 'Shard layout (JSON array)'
|
|
||||||
type: choice
|
|
||||||
required: true
|
|
||||||
default: '[1, 2, 3, 4]'
|
|
||||||
options:
|
|
||||||
- '[1, 2, 3, 4]'
|
|
||||||
- '[1, 2, 3, 4, 5, 6]'
|
|
||||||
- '[1, 2]'
|
|
||||||
- '[1]'
|
|
||||||
|
|
||||||
workers:
|
|
||||||
description: 'Playwright workers per shard'
|
|
||||||
type: string
|
|
||||||
required: true
|
|
||||||
default: '2'
|
|
||||||
|
|
||||||
base_url:
|
|
||||||
description: 'Penpot base URL for the (disabled) live Plugin API suite'
|
|
||||||
required: false
|
|
||||||
default: 'https://localhost:3449'
|
|
||||||
|
|
||||||
pull_request:
|
|
||||||
paths:
|
|
||||||
- 'frontend/**'
|
|
||||||
- 'common/**'
|
|
||||||
- 'render-wasm/**'
|
|
||||||
- 'plugins/**'
|
|
||||||
- '.github/workflows/tests-e2e.yml'
|
|
||||||
|
|
||||||
types:
|
|
||||||
- opened
|
|
||||||
- synchronize
|
|
||||||
- ready_for_review
|
|
||||||
|
|
||||||
push:
|
|
||||||
branches:
|
|
||||||
- develop
|
|
||||||
- staging
|
|
||||||
|
|
||||||
paths:
|
|
||||||
- 'frontend/**'
|
|
||||||
- 'common/**'
|
|
||||||
- 'render-wasm/**'
|
|
||||||
- 'plugins/**'
|
|
||||||
- '.github/workflows/tests-e2e.yml'
|
|
||||||
|
|
||||||
concurrency:
|
|
||||||
group: ${{ github.workflow }}-${{ github.event.pull_request.number || inputs.gh_ref || github.ref }}
|
|
||||||
cancel-in-progress: true
|
|
||||||
|
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
# ── 1. Build the frontend bundle once ──────────────────────────────────
|
|
||||||
build-bundle:
|
|
||||||
if: ${{ !github.event.pull_request.draft }}
|
|
||||||
name: "Build Integration Bundle"
|
|
||||||
runs-on: penpot-extended-runner
|
|
||||||
timeout-minutes: 30
|
|
||||||
|
|
||||||
container:
|
|
||||||
image: penpotapp/devenv:latest
|
|
||||||
volumes:
|
|
||||||
- /var/cache/github-runner/m2:/root/.m2
|
|
||||||
- /var/cache/github-runner/gitlib:/root/.gitlibs
|
|
||||||
|
|
||||||
outputs:
|
|
||||||
bundle_key: ${{ steps.vars.outputs.bundle_key }}
|
|
||||||
|
|
||||||
steps:
|
|
||||||
# An empty `ref` makes checkout fall back to its default (the PR merge
|
|
||||||
# ref on pull_request, the pushed ref on push).
|
|
||||||
- name: Checkout repository
|
|
||||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
||||||
with:
|
|
||||||
persist-credentials: false
|
|
||||||
ref: ${{ inputs.gh_ref }}
|
|
||||||
|
|
||||||
# The cache key must come from the SHA actually checked out: on a manual
|
|
||||||
# run `github.sha` points at the dispatching ref, not at `gh_ref`.
|
|
||||||
- name: Extract cache key
|
|
||||||
id: vars
|
|
||||||
run: |
|
|
||||||
echo "bundle_key=frontend-bundle-$(git rev-parse HEAD)" >> $GITHUB_OUTPUT
|
|
||||||
|
|
||||||
- name: Restore Bundle Cache
|
|
||||||
id: restore
|
|
||||||
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
|
||||||
with:
|
|
||||||
key: ${{ steps.vars.outputs.bundle_key }}
|
|
||||||
path: frontend/resources/public
|
|
||||||
|
|
||||||
- name: Build Bundle
|
|
||||||
if: steps.restore.outputs.cache-hit != 'true'
|
|
||||||
working-directory: ./frontend
|
|
||||||
run: |
|
|
||||||
./scripts/build
|
|
||||||
|
|
||||||
- name: Store Bundle Cache
|
|
||||||
if: steps.restore.outputs.cache-hit != 'true'
|
|
||||||
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
|
||||||
with:
|
|
||||||
key: ${{ steps.vars.outputs.bundle_key }}
|
|
||||||
path: frontend/resources/public
|
|
||||||
|
|
||||||
# ── 2. Consumers: restore the bundle, never rebuild it ─────────────────
|
|
||||||
test-integration:
|
|
||||||
if: ${{ !github.event.pull_request.draft }}
|
|
||||||
name: "Integration Tests (${{ matrix.shard }})"
|
|
||||||
runs-on: penpot-extended-runner
|
|
||||||
timeout-minutes: 40
|
|
||||||
|
|
||||||
needs: build-bundle
|
|
||||||
|
|
||||||
strategy:
|
|
||||||
fail-fast: false
|
|
||||||
matrix:
|
|
||||||
shard: ${{ fromJSON(inputs.shards || '[1, 2, 3, 4]') }}
|
|
||||||
|
|
||||||
container:
|
|
||||||
image: penpotapp/devenv:latest
|
|
||||||
volumes:
|
|
||||||
- /var/cache/github-runner/m2:/root/.m2
|
|
||||||
- /var/cache/github-runner/gitlib:/root/.gitlibs
|
|
||||||
- /var/cache/github-runner/ms-playwright:/ms-playwright
|
|
||||||
env:
|
|
||||||
PLAYWRIGHT_BROWSERS_PATH: /ms-playwright
|
|
||||||
|
|
||||||
steps:
|
|
||||||
- name: Checkout Repository
|
|
||||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
||||||
with:
|
|
||||||
persist-credentials: false
|
|
||||||
ref: ${{ inputs.gh_ref }}
|
|
||||||
|
|
||||||
- name: Restore Cache
|
|
||||||
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
|
||||||
with:
|
|
||||||
key: ${{ needs.build-bundle.outputs.bundle_key }}
|
|
||||||
path: frontend/resources/public
|
|
||||||
fail-on-cache-miss: true
|
|
||||||
|
|
||||||
- name: Install deps
|
|
||||||
working-directory: ./frontend
|
|
||||||
run: |
|
|
||||||
pnpm install --frozen-lockfile;
|
|
||||||
|
|
||||||
# No-op once the shared volume is warm; keeps the first run working.
|
|
||||||
- name: Install Playwright Chromium
|
|
||||||
working-directory: ./frontend
|
|
||||||
run: pnpm exec playwright install chromium
|
|
||||||
|
|
||||||
# `strategy.job-total` is the matrix size, so the shard denominator
|
|
||||||
# follows the `shards` input without being hardcoded.
|
|
||||||
- name: Run Tests
|
|
||||||
working-directory: ./frontend
|
|
||||||
env:
|
|
||||||
WORKERS: ${{ inputs.workers }}
|
|
||||||
SHARD: ${{ matrix.shard }}
|
|
||||||
SHARD_TOTAL: ${{ strategy.job-total }}
|
|
||||||
run: |
|
|
||||||
WORKERS=${WORKERS:-2}
|
|
||||||
echo "Running shard ${SHARD}/${SHARD_TOTAL} with $WORKERS workers"
|
|
||||||
pnpm exec playwright test --project default \
|
|
||||||
--workers="$WORKERS" \
|
|
||||||
--shard="${SHARD}/${SHARD_TOTAL}" \
|
|
||||||
--reporter=blob
|
|
||||||
|
|
||||||
- name: Upload blob report
|
|
||||||
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
|
|
||||||
if: always()
|
|
||||||
with:
|
|
||||||
name: integration-blob-report-${{ matrix.shard }}
|
|
||||||
path: frontend/blob-report/
|
|
||||||
overwrite: true
|
|
||||||
retention-days: 3
|
|
||||||
|
|
||||||
- name: Upload test result
|
|
||||||
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
|
|
||||||
if: always()
|
|
||||||
with:
|
|
||||||
name: integration-tests-result-${{ matrix.shard }}
|
|
||||||
path: frontend/test-results/
|
|
||||||
overwrite: true
|
|
||||||
if-no-files-found: ignore
|
|
||||||
retention-days: 3
|
|
||||||
|
|
||||||
composable-test-suite:
|
|
||||||
if: ${{ github.event_name != 'workflow_dispatch' && !github.event.pull_request.draft }}
|
|
||||||
name: "Run composable test suite (mocked backend)"
|
|
||||||
runs-on: penpot-extended-runner
|
|
||||||
timeout-minutes: 30
|
|
||||||
|
|
||||||
needs: build-bundle
|
|
||||||
|
|
||||||
container:
|
|
||||||
image: penpotapp/devenv:latest
|
|
||||||
volumes:
|
|
||||||
- /var/cache/github-runner/m2:/root/.m2
|
|
||||||
- /var/cache/github-runner/gitlib:/root/.gitlibs
|
|
||||||
|
|
||||||
steps:
|
|
||||||
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
||||||
with:
|
|
||||||
persist-credentials: false
|
|
||||||
|
|
||||||
# The driver serves the prebuilt bundle from frontend/resources/public.
|
|
||||||
- name: Restore Cache
|
|
||||||
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
|
||||||
with:
|
|
||||||
key: ${{ needs.build-bundle.outputs.bundle_key }}
|
|
||||||
path: frontend/resources/public
|
|
||||||
fail-on-cache-miss: true
|
|
||||||
|
|
||||||
- name: Install deps
|
|
||||||
working-directory: ./plugins
|
|
||||||
run: |
|
|
||||||
pnpm install;
|
|
||||||
|
|
||||||
- name: Install Playwright Chromium
|
|
||||||
working-directory: ./plugins
|
|
||||||
run: pnpm --filter composable-test-suite exec playwright install --with-deps chromium
|
|
||||||
|
|
||||||
- name: Run composable test suite (mocked)
|
|
||||||
working-directory: ./plugins
|
|
||||||
run: pnpm --filter composable-test-suite run test:ci
|
|
||||||
|
|
||||||
api-test-suite-mocked:
|
|
||||||
if: ${{ github.event_name != 'workflow_dispatch' && !github.event.pull_request.draft }}
|
|
||||||
name: "Run Plugin API Test Suite (mocked)"
|
|
||||||
runs-on: penpot-extended-runner
|
|
||||||
timeout-minutes: 30
|
|
||||||
|
|
||||||
needs: build-bundle
|
|
||||||
|
|
||||||
container:
|
|
||||||
image: penpotapp/devenv:latest
|
|
||||||
volumes:
|
|
||||||
- /var/cache/github-runner/m2:/root/.m2
|
|
||||||
- /var/cache/github-runner/gitlib:/root/.gitlibs
|
|
||||||
|
|
||||||
steps:
|
|
||||||
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
||||||
with:
|
|
||||||
persist-credentials: false
|
|
||||||
|
|
||||||
# Mocked mode serves the prebuilt bundle from frontend/resources/public.
|
|
||||||
- name: Restore Cache
|
|
||||||
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
|
||||||
with:
|
|
||||||
key: ${{ needs.build-bundle.outputs.bundle_key }}
|
|
||||||
path: frontend/resources/public
|
|
||||||
fail-on-cache-miss: true
|
|
||||||
|
|
||||||
- name: Install deps
|
|
||||||
working-directory: ./plugins
|
|
||||||
run: |
|
|
||||||
pnpm install;
|
|
||||||
|
|
||||||
- name: Install Playwright Chromium
|
|
||||||
working-directory: ./plugins
|
|
||||||
run: pnpm --filter plugin-api-test-suite exec playwright install --with-deps chromium
|
|
||||||
|
|
||||||
- name: Generate API surface
|
|
||||||
working-directory: ./plugins
|
|
||||||
run: pnpm --filter plugin-api-test-suite run gen:api
|
|
||||||
|
|
||||||
- name: Run API test suite (mocked)
|
|
||||||
working-directory: ./plugins
|
|
||||||
env:
|
|
||||||
MOCK_BACKEND: "1"
|
|
||||||
run: pnpm --filter plugin-api-test-suite run test:ci
|
|
||||||
|
|
||||||
# ── 3. Merge the per-shard integration reports ─────────────────────────
|
|
||||||
merge-reports:
|
|
||||||
if: ${{ !cancelled() && !github.event.pull_request.draft && needs.test-integration.result != 'skipped' }}
|
|
||||||
name: "Merge Integration Reports"
|
|
||||||
runs-on: penpot-extended-runner
|
|
||||||
timeout-minutes: 15
|
|
||||||
|
|
||||||
needs: test-integration
|
|
||||||
|
|
||||||
container:
|
|
||||||
image: penpotapp/devenv:latest
|
|
||||||
volumes:
|
|
||||||
- /var/cache/github-runner/m2:/root/.m2
|
|
||||||
- /var/cache/github-runner/gitlib:/root/.gitlibs
|
|
||||||
|
|
||||||
steps:
|
|
||||||
- name: Checkout Repository
|
|
||||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
||||||
with:
|
|
||||||
persist-credentials: false
|
|
||||||
ref: ${{ inputs.gh_ref }}
|
|
||||||
|
|
||||||
- name: Install deps
|
|
||||||
working-directory: ./frontend
|
|
||||||
run: |
|
|
||||||
pnpm install --frozen-lockfile;
|
|
||||||
|
|
||||||
- name: Download blob reports
|
|
||||||
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
|
|
||||||
with:
|
|
||||||
path: frontend/all-blob-reports
|
|
||||||
pattern: integration-blob-report-*
|
|
||||||
merge-multiple: true
|
|
||||||
|
|
||||||
- name: Merge into HTML report
|
|
||||||
working-directory: ./frontend
|
|
||||||
env:
|
|
||||||
PLAYWRIGHT_JSON_OUTPUT_NAME: report.json
|
|
||||||
run: |
|
|
||||||
pnpm exec playwright merge-reports \
|
|
||||||
--reporter=html,json,list ./all-blob-reports
|
|
||||||
|
|
||||||
- name: Test summary
|
|
||||||
if: always()
|
|
||||||
continue-on-error: true
|
|
||||||
working-directory: ./frontend
|
|
||||||
run: |
|
|
||||||
if [ ! -f report.json ]; then
|
|
||||||
echo "No report produced (all shards failed early)." >> "$GITHUB_STEP_SUMMARY"
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
jq -r -f ../.github/scripts/playwright-summary.jq report.json \
|
|
||||||
>> "$GITHUB_STEP_SUMMARY" \
|
|
||||||
|| echo "Summary generation failed; see the HTML report artifact." \
|
|
||||||
>> "$GITHUB_STEP_SUMMARY"
|
|
||||||
|
|
||||||
- name: Upload JSON report
|
|
||||||
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
|
|
||||||
if: always()
|
|
||||||
with:
|
|
||||||
name: integration-json-report
|
|
||||||
path: frontend/report.json
|
|
||||||
overwrite: true
|
|
||||||
if-no-files-found: ignore
|
|
||||||
retention-days: 7
|
|
||||||
|
|
||||||
- name: Upload HTML report
|
|
||||||
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
|
|
||||||
if: always()
|
|
||||||
with:
|
|
||||||
name: integration-html-report
|
|
||||||
path: frontend/playwright-report/
|
|
||||||
overwrite: true
|
|
||||||
retention-days: 7
|
|
||||||
|
|
||||||
# ── 4. Live Plugin API suite (disabled) ────────────────────────────────
|
|
||||||
#
|
|
||||||
# True end-to-end against a LIVE instance: point PENPOT_BASE_URL at a
|
|
||||||
# reachable instance and provide login credentials via repo secrets. Manual
|
|
||||||
# because the CI runner has no Docker to stand up a full stack.
|
|
||||||
#
|
|
||||||
# api-test-suite-live:
|
|
||||||
# if: ${{ github.event_name == 'workflow_dispatch' }}
|
|
||||||
# name: Run Plugin API Test Suite (live)
|
|
||||||
# runs-on: penpot-extended-runner
|
|
||||||
# container:
|
|
||||||
# image: penpotapp/devenv:latest
|
|
||||||
#
|
|
||||||
# env:
|
|
||||||
# PENPOT_BASE_URL: ${{ inputs.base_url }}
|
|
||||||
# E2E_LOGIN_EMAIL: ${{ secrets.E2E_LOGIN_EMAIL }}
|
|
||||||
# E2E_LOGIN_PASSWORD: ${{ secrets.E2E_LOGIN_PASSWORD }}
|
|
||||||
#
|
|
||||||
# steps:
|
|
||||||
# - uses: actions/checkout@v6
|
|
||||||
#
|
|
||||||
# - name: Setup Node
|
|
||||||
# uses: actions/setup-node@v6
|
|
||||||
# with:
|
|
||||||
# node-version-file: .nvmrc
|
|
||||||
#
|
|
||||||
# - name: Install deps
|
|
||||||
# working-directory: ./plugins
|
|
||||||
# run: |
|
|
||||||
# pnpm install;
|
|
||||||
#
|
|
||||||
# - name: Install Playwright Chromium
|
|
||||||
# working-directory: ./plugins
|
|
||||||
# run: pnpm --filter plugin-api-test-suite exec playwright install --with-deps chromium
|
|
||||||
#
|
|
||||||
# - name: Generate API surface
|
|
||||||
# working-directory: ./plugins
|
|
||||||
# run: pnpm --filter plugin-api-test-suite run gen:api
|
|
||||||
#
|
|
||||||
# # Note: requires a running Penpot instance reachable at PENPOT_BASE_URL.
|
|
||||||
# - name: Run API test suite
|
|
||||||
# working-directory: ./plugins
|
|
||||||
# run: pnpm --filter plugin-api-test-suite run test:ci
|
|
||||||
13
.github/workflows/tests-exporter.yml
vendored
13
.github/workflows/tests-exporter.yml
vendored
@ -9,7 +9,6 @@ on:
|
|||||||
paths:
|
paths:
|
||||||
- 'exporter/**'
|
- 'exporter/**'
|
||||||
- 'common/**'
|
- 'common/**'
|
||||||
- 'render-wasm/**'
|
|
||||||
|
|
||||||
types:
|
types:
|
||||||
- opened
|
- opened
|
||||||
@ -24,20 +23,16 @@ on:
|
|||||||
paths:
|
paths:
|
||||||
- 'exporter/**'
|
- 'exporter/**'
|
||||||
- 'common/**'
|
- 'common/**'
|
||||||
- 'render-wasm/**'
|
|
||||||
|
|
||||||
concurrency:
|
concurrency:
|
||||||
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
||||||
cancel-in-progress: true
|
cancel-in-progress: true
|
||||||
|
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
test-exporter:
|
test-exporter:
|
||||||
if: ${{ !github.event.pull_request.draft }}
|
if: ${{ !github.event.pull_request.draft }}
|
||||||
name: "Exporter Tests"
|
name: "Exporter Tests"
|
||||||
runs-on: penpot-extended-runner
|
runs-on: penpot-runner-02
|
||||||
container:
|
container:
|
||||||
image: penpotapp/devenv:latest
|
image: penpotapp/devenv:latest
|
||||||
volumes:
|
volumes:
|
||||||
@ -46,13 +41,13 @@ jobs:
|
|||||||
|
|
||||||
steps:
|
steps:
|
||||||
- name: Checkout repository
|
- name: Checkout repository
|
||||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
uses: actions/checkout@v6
|
||||||
with:
|
|
||||||
persist-credentials: false
|
|
||||||
|
|
||||||
- name: Lint
|
- name: Lint
|
||||||
working-directory: ./exporter
|
working-directory: ./exporter
|
||||||
run: |
|
run: |
|
||||||
|
corepack enable;
|
||||||
|
corepack install;
|
||||||
pnpm install;
|
pnpm install;
|
||||||
pnpm run check-fmt:clj
|
pnpm run check-fmt:clj
|
||||||
pnpm run lint:clj
|
pnpm run lint:clj
|
||||||
|
|||||||
9
.github/workflows/tests-frontend.yml
vendored
9
.github/workflows/tests-frontend.yml
vendored
@ -30,9 +30,6 @@ concurrency:
|
|||||||
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
||||||
cancel-in-progress: true
|
cancel-in-progress: true
|
||||||
|
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
test-frontend:
|
test-frontend:
|
||||||
if: ${{ !github.event.pull_request.draft }}
|
if: ${{ !github.event.pull_request.draft }}
|
||||||
@ -46,13 +43,13 @@ jobs:
|
|||||||
|
|
||||||
steps:
|
steps:
|
||||||
- name: Checkout repository
|
- name: Checkout repository
|
||||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
uses: actions/checkout@v6
|
||||||
with:
|
|
||||||
persist-credentials: false
|
|
||||||
|
|
||||||
- name: Lint
|
- name: Lint
|
||||||
working-directory: ./frontend
|
working-directory: ./frontend
|
||||||
run: |
|
run: |
|
||||||
|
corepack enable;
|
||||||
|
corepack install;
|
||||||
pnpm install;
|
pnpm install;
|
||||||
pnpm run check-fmt:js
|
pnpm run check-fmt:js
|
||||||
pnpm run check-fmt:clj
|
pnpm run check-fmt:clj
|
||||||
|
|||||||
250
.github/workflows/tests-integration.yml
vendored
Normal file
250
.github/workflows/tests-integration.yml
vendored
Normal file
@ -0,0 +1,250 @@
|
|||||||
|
name: "CI: Integration"
|
||||||
|
|
||||||
|
defaults:
|
||||||
|
run:
|
||||||
|
shell: bash
|
||||||
|
|
||||||
|
on:
|
||||||
|
workflow_dispatch:
|
||||||
|
inputs:
|
||||||
|
gh_ref:
|
||||||
|
description: 'Name of the branch or ref'
|
||||||
|
type: string
|
||||||
|
required: true
|
||||||
|
default: 'develop'
|
||||||
|
|
||||||
|
shards:
|
||||||
|
description: 'Shard layout (JSON array)'
|
||||||
|
type: choice
|
||||||
|
required: true
|
||||||
|
default: '[1, 2, 3, 4]'
|
||||||
|
options:
|
||||||
|
- '[1, 2, 3, 4]'
|
||||||
|
- '[1, 2, 3, 4, 5, 6]'
|
||||||
|
- '[1, 2]'
|
||||||
|
- '[1]'
|
||||||
|
|
||||||
|
workers:
|
||||||
|
description: 'Playwright workers per shard'
|
||||||
|
type: string
|
||||||
|
required: true
|
||||||
|
default: '2'
|
||||||
|
|
||||||
|
pull_request:
|
||||||
|
paths:
|
||||||
|
- 'frontend/**'
|
||||||
|
- 'common/**'
|
||||||
|
- 'render-wasm/**'
|
||||||
|
- '.github/workflows/tests-integration.yml'
|
||||||
|
|
||||||
|
types:
|
||||||
|
- opened
|
||||||
|
- synchronize
|
||||||
|
- ready_for_review
|
||||||
|
|
||||||
|
push:
|
||||||
|
branches:
|
||||||
|
- develop
|
||||||
|
- staging
|
||||||
|
|
||||||
|
paths:
|
||||||
|
- 'frontend/**'
|
||||||
|
- 'common/**'
|
||||||
|
- 'render-wasm/**'
|
||||||
|
- '.github/workflows/tests-integration.yml'
|
||||||
|
|
||||||
|
concurrency:
|
||||||
|
group: ${{ github.workflow }}-${{ github.event.pull_request.number || inputs.gh_ref || github.ref }}
|
||||||
|
cancel-in-progress: true
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
build-integration:
|
||||||
|
if: ${{ !github.event.pull_request.draft }}
|
||||||
|
name: "Build Integration Bundle"
|
||||||
|
runs-on: penpot-extended-runner
|
||||||
|
timeout-minutes: 30
|
||||||
|
container:
|
||||||
|
image: penpotapp/devenv:latest
|
||||||
|
volumes:
|
||||||
|
- /var/cache/github-runner/m2:/root/.m2
|
||||||
|
- /var/cache/github-runner/gitlib:/root/.gitlibs
|
||||||
|
|
||||||
|
outputs:
|
||||||
|
bundle_key: ${{ steps.vars.outputs.bundle_key }}
|
||||||
|
|
||||||
|
steps:
|
||||||
|
# An empty `ref` makes checkout fall back to its default (the PR merge
|
||||||
|
# ref on pull_request, the pushed ref on push).
|
||||||
|
- name: Checkout repository
|
||||||
|
uses: actions/checkout@v6
|
||||||
|
with:
|
||||||
|
ref: ${{ inputs.gh_ref }}
|
||||||
|
|
||||||
|
# The cache key must come from the SHA actually checked out: on a manual
|
||||||
|
# run `github.sha` points at the dispatching ref, not at `gh_ref`.
|
||||||
|
- name: Extract cache key
|
||||||
|
id: vars
|
||||||
|
run: |
|
||||||
|
echo "bundle_key=integration-bundle-$(git rev-parse HEAD)" >> $GITHUB_OUTPUT
|
||||||
|
|
||||||
|
- name: Build Bundle
|
||||||
|
working-directory: ./frontend
|
||||||
|
run: |
|
||||||
|
./scripts/build
|
||||||
|
|
||||||
|
- name: Store Bundle Cache
|
||||||
|
uses: actions/cache@v5
|
||||||
|
with:
|
||||||
|
key: ${{ steps.vars.outputs.bundle_key }}
|
||||||
|
path: frontend/resources/public
|
||||||
|
|
||||||
|
test-integration:
|
||||||
|
if: ${{ !github.event.pull_request.draft }}
|
||||||
|
name: "Integration Tests (${{ matrix.shard }})"
|
||||||
|
runs-on: penpot-extended-runner
|
||||||
|
timeout-minutes: 40
|
||||||
|
|
||||||
|
needs: build-integration
|
||||||
|
|
||||||
|
strategy:
|
||||||
|
fail-fast: false
|
||||||
|
matrix:
|
||||||
|
shard: ${{ fromJSON(inputs.shards || '[1, 2, 3, 4]') }}
|
||||||
|
|
||||||
|
container:
|
||||||
|
image: penpotapp/devenv:latest
|
||||||
|
volumes:
|
||||||
|
- /var/cache/github-runner/m2:/root/.m2
|
||||||
|
- /var/cache/github-runner/gitlib:/root/.gitlibs
|
||||||
|
- /var/cache/github-runner/ms-playwright:/ms-playwright
|
||||||
|
env:
|
||||||
|
PLAYWRIGHT_BROWSERS_PATH: /ms-playwright
|
||||||
|
|
||||||
|
steps:
|
||||||
|
- name: Checkout Repository
|
||||||
|
uses: actions/checkout@v6
|
||||||
|
with:
|
||||||
|
ref: ${{ inputs.gh_ref }}
|
||||||
|
|
||||||
|
- name: Restore Cache
|
||||||
|
uses: actions/cache/restore@v5
|
||||||
|
with:
|
||||||
|
key: ${{ needs.build-integration.outputs.bundle_key }}
|
||||||
|
path: frontend/resources/public
|
||||||
|
|
||||||
|
- name: Install deps
|
||||||
|
working-directory: ./frontend
|
||||||
|
run: |
|
||||||
|
corepack enable;
|
||||||
|
corepack install;
|
||||||
|
pnpm install --frozen-lockfile;
|
||||||
|
|
||||||
|
# No-op once the shared volume is warm; keeps the first run working.
|
||||||
|
- name: Install Playwright Chromium
|
||||||
|
working-directory: ./frontend
|
||||||
|
run: pnpm exec playwright install chromium
|
||||||
|
|
||||||
|
# `strategy.job-total` is the matrix size, so the shard denominator
|
||||||
|
# follows the `shards` input without being hardcoded.
|
||||||
|
- name: Run Tests
|
||||||
|
working-directory: ./frontend
|
||||||
|
env:
|
||||||
|
WORKERS: ${{ inputs.workers }}
|
||||||
|
run: |
|
||||||
|
WORKERS=${WORKERS:-2}
|
||||||
|
echo "Running shard ${{ matrix.shard }}/${{ strategy.job-total }} with $WORKERS workers"
|
||||||
|
pnpm exec playwright test --project default \
|
||||||
|
--workers="$WORKERS" \
|
||||||
|
--shard=${{ matrix.shard }}/${{ strategy.job-total }} \
|
||||||
|
--reporter=blob
|
||||||
|
|
||||||
|
- name: Upload blob report
|
||||||
|
uses: actions/upload-artifact@v7
|
||||||
|
if: always()
|
||||||
|
with:
|
||||||
|
name: integration-blob-report-${{ matrix.shard }}
|
||||||
|
path: frontend/blob-report/
|
||||||
|
overwrite: true
|
||||||
|
retention-days: 3
|
||||||
|
|
||||||
|
- name: Upload test result
|
||||||
|
uses: actions/upload-artifact@v7
|
||||||
|
if: always()
|
||||||
|
with:
|
||||||
|
name: integration-tests-result-${{ matrix.shard }}
|
||||||
|
path: frontend/test-results/
|
||||||
|
overwrite: true
|
||||||
|
if-no-files-found: ignore
|
||||||
|
retention-days: 3
|
||||||
|
|
||||||
|
merge-reports:
|
||||||
|
if: ${{ always() && !github.event.pull_request.draft && needs.test-integration.result != 'skipped' }}
|
||||||
|
name: "Merge Integration Reports"
|
||||||
|
runs-on: penpot-extended-runner
|
||||||
|
timeout-minutes: 15
|
||||||
|
|
||||||
|
needs: test-integration
|
||||||
|
|
||||||
|
container:
|
||||||
|
image: penpotapp/devenv:latest
|
||||||
|
volumes:
|
||||||
|
- /var/cache/github-runner/m2:/root/.m2
|
||||||
|
- /var/cache/github-runner/gitlib:/root/.gitlibs
|
||||||
|
|
||||||
|
steps:
|
||||||
|
- name: Checkout Repository
|
||||||
|
uses: actions/checkout@v6
|
||||||
|
with:
|
||||||
|
ref: ${{ inputs.gh_ref }}
|
||||||
|
|
||||||
|
- name: Install deps
|
||||||
|
working-directory: ./frontend
|
||||||
|
run: |
|
||||||
|
corepack enable;
|
||||||
|
corepack install;
|
||||||
|
pnpm install --frozen-lockfile;
|
||||||
|
|
||||||
|
- name: Download blob reports
|
||||||
|
uses: actions/download-artifact@v7
|
||||||
|
with:
|
||||||
|
path: frontend/all-blob-reports
|
||||||
|
pattern: integration-blob-report-*
|
||||||
|
merge-multiple: true
|
||||||
|
|
||||||
|
- name: Merge into HTML report
|
||||||
|
working-directory: ./frontend
|
||||||
|
env:
|
||||||
|
PLAYWRIGHT_JSON_OUTPUT_NAME: report.json
|
||||||
|
run: |
|
||||||
|
pnpm exec playwright merge-reports \
|
||||||
|
--reporter=html,json,list ./all-blob-reports
|
||||||
|
|
||||||
|
- name: Test summary
|
||||||
|
if: always()
|
||||||
|
working-directory: ./frontend
|
||||||
|
run: |
|
||||||
|
if [ ! -f report.json ]; then
|
||||||
|
echo "No report produced (all shards failed early)." >> "$GITHUB_STEP_SUMMARY"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
jq -r -f ../.github/scripts/playwright-summary.jq report.json >> "$GITHUB_STEP_SUMMARY"
|
||||||
|
|
||||||
|
# Kept for 30 days so flakiness rates can be aggregated across runs
|
||||||
|
# without scraping job logs.
|
||||||
|
- name: Upload JSON report
|
||||||
|
uses: actions/upload-artifact@v7
|
||||||
|
if: always()
|
||||||
|
with:
|
||||||
|
name: integration-json-report
|
||||||
|
path: frontend/report.json
|
||||||
|
overwrite: true
|
||||||
|
if-no-files-found: ignore
|
||||||
|
retention-days: 30
|
||||||
|
|
||||||
|
- name: Upload HTML report
|
||||||
|
uses: actions/upload-artifact@v7
|
||||||
|
with:
|
||||||
|
name: integration-html-report
|
||||||
|
path: frontend/playwright-report/
|
||||||
|
overwrite: true
|
||||||
|
retention-days: 7
|
||||||
9
.github/workflows/tests-library.yml
vendored
9
.github/workflows/tests-library.yml
vendored
@ -28,9 +28,6 @@ concurrency:
|
|||||||
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
||||||
cancel-in-progress: true
|
cancel-in-progress: true
|
||||||
|
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
test-library:
|
test-library:
|
||||||
if: ${{ !github.event.pull_request.draft }}
|
if: ${{ !github.event.pull_request.draft }}
|
||||||
@ -44,13 +41,13 @@ jobs:
|
|||||||
|
|
||||||
steps:
|
steps:
|
||||||
- name: Checkout repository
|
- name: Checkout repository
|
||||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
uses: actions/checkout@v6
|
||||||
with:
|
|
||||||
persist-credentials: false
|
|
||||||
|
|
||||||
- name: Lint
|
- name: Lint
|
||||||
working-directory: ./library
|
working-directory: ./library
|
||||||
run: |
|
run: |
|
||||||
|
corepack enable;
|
||||||
|
corepack install;
|
||||||
pnpm install;
|
pnpm install;
|
||||||
pnpm run check-fmt
|
pnpm run check-fmt
|
||||||
pnpm run lint
|
pnpm run lint
|
||||||
|
|||||||
7
.github/workflows/tests-mcp.yml
vendored
7
.github/workflows/tests-mcp.yml
vendored
@ -24,9 +24,6 @@ on:
|
|||||||
paths:
|
paths:
|
||||||
- 'mcp/**'
|
- 'mcp/**'
|
||||||
|
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
test-mcp:
|
test-mcp:
|
||||||
if: ${{ !github.event.pull_request.draft }}
|
if: ${{ !github.event.pull_request.draft }}
|
||||||
@ -36,9 +33,7 @@ jobs:
|
|||||||
|
|
||||||
steps:
|
steps:
|
||||||
- name: Checkout code
|
- name: Checkout code
|
||||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
uses: actions/checkout@v6
|
||||||
with:
|
|
||||||
persist-credentials: false
|
|
||||||
|
|
||||||
- name: Setup
|
- name: Setup
|
||||||
working-directory: ./mcp
|
working-directory: ./mcp
|
||||||
|
|||||||
133
.github/workflows/tests-plugin-api-suite.yml
vendored
Normal file
133
.github/workflows/tests-plugin-api-suite.yml
vendored
Normal file
@ -0,0 +1,133 @@
|
|||||||
|
name: "CI: Plugin API Test Suite"
|
||||||
|
|
||||||
|
# Runs the Plugin API Test Suite (it exercises the real Penpot Plugin API, so it
|
||||||
|
# needs a running frontend + the plugin runtime). Two jobs:
|
||||||
|
#
|
||||||
|
# - api-test-suite-mocked (pull_request / push): the per-PR gate. Serves the
|
||||||
|
# prebuilt frontend bundle and intercepts every backend RPC with Playwright
|
||||||
|
# (MOCK_BACKEND=1). No backend / no login. Validates the frontend Plugin API
|
||||||
|
# binding + in-memory store; backend-result-dependent tests are skipped via the
|
||||||
|
# `skipIfMocked` tag. See plugins/apps/plugin-api-test-suite/README.md.
|
||||||
|
#
|
||||||
|
# - api-test-suite-live (workflow_dispatch): true end-to-end against a LIVE
|
||||||
|
# instance. Point PENPOT_BASE_URL at a reachable instance and provide login
|
||||||
|
# credentials via repo secrets. Manual because the CI runner has no Docker to
|
||||||
|
# stand up a full stack.
|
||||||
|
|
||||||
|
defaults:
|
||||||
|
run:
|
||||||
|
shell: bash
|
||||||
|
|
||||||
|
on:
|
||||||
|
workflow_dispatch:
|
||||||
|
inputs:
|
||||||
|
base_url:
|
||||||
|
description: "Penpot base URL (e.g. https://localhost:3449)"
|
||||||
|
required: false
|
||||||
|
default: "https://localhost:3449"
|
||||||
|
|
||||||
|
pull_request:
|
||||||
|
paths:
|
||||||
|
- 'plugins/**'
|
||||||
|
- 'frontend/**'
|
||||||
|
- 'common/**'
|
||||||
|
types:
|
||||||
|
- opened
|
||||||
|
- synchronize
|
||||||
|
- ready_for_review
|
||||||
|
|
||||||
|
push:
|
||||||
|
branches:
|
||||||
|
- develop
|
||||||
|
- staging
|
||||||
|
paths:
|
||||||
|
- 'plugins/**'
|
||||||
|
- 'frontend/src/app/plugins/**'
|
||||||
|
- 'common/**'
|
||||||
|
|
||||||
|
concurrency:
|
||||||
|
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
||||||
|
cancel-in-progress: true
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
api-test-suite-mocked:
|
||||||
|
if: ${{ github.event_name != 'workflow_dispatch' && !github.event.pull_request.draft }}
|
||||||
|
name: "Run Plugin API Test Suite (mocked)"
|
||||||
|
runs-on: penpot-extended-runner
|
||||||
|
container:
|
||||||
|
image: penpotapp/devenv:latest
|
||||||
|
volumes:
|
||||||
|
- /var/cache/github-runner/m2:/root/.m2
|
||||||
|
- /var/cache/github-runner/gitlib:/root/.gitlibs
|
||||||
|
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v6
|
||||||
|
|
||||||
|
# Mocked mode serves the prebuilt bundle from frontend/resources/public.
|
||||||
|
- name: Build frontend bundle
|
||||||
|
working-directory: ./frontend
|
||||||
|
run: ./scripts/build
|
||||||
|
|
||||||
|
- name: Install deps
|
||||||
|
working-directory: ./plugins
|
||||||
|
run: |
|
||||||
|
corepack enable;
|
||||||
|
corepack install;
|
||||||
|
pnpm install;
|
||||||
|
|
||||||
|
- name: Install Playwright Chromium
|
||||||
|
working-directory: ./plugins
|
||||||
|
run: pnpm --filter plugin-api-test-suite exec playwright install --with-deps chromium
|
||||||
|
|
||||||
|
- name: Generate API surface
|
||||||
|
working-directory: ./plugins
|
||||||
|
run: pnpm --filter plugin-api-test-suite run gen:api
|
||||||
|
|
||||||
|
- name: Run API test suite (mocked)
|
||||||
|
working-directory: ./plugins
|
||||||
|
env:
|
||||||
|
MOCK_BACKEND: "1"
|
||||||
|
run: pnpm --filter plugin-api-test-suite run test:ci
|
||||||
|
|
||||||
|
## The following job will launch the whole suite of tests but we need
|
||||||
|
## to have a full environment in the CI for this to work.
|
||||||
|
|
||||||
|
# api-test-suite-live:
|
||||||
|
# if: ${{ github.event_name == 'workflow_dispatch' }}
|
||||||
|
# name: Run Plugin API Test Suite (live)
|
||||||
|
# runs-on: penpot-extended-runner
|
||||||
|
# container:
|
||||||
|
# image: penpotapp/devenv:latest
|
||||||
|
#
|
||||||
|
# env:
|
||||||
|
# PENPOT_BASE_URL: ${{ github.event.inputs.base_url }}
|
||||||
|
# E2E_LOGIN_EMAIL: ${{ secrets.E2E_LOGIN_EMAIL }}
|
||||||
|
# E2E_LOGIN_PASSWORD: ${{ secrets.E2E_LOGIN_PASSWORD }}
|
||||||
|
#
|
||||||
|
# steps:
|
||||||
|
# - uses: actions/checkout@v6
|
||||||
|
#
|
||||||
|
# - name: Setup Node
|
||||||
|
# uses: actions/setup-node@v6
|
||||||
|
# with:
|
||||||
|
# node-version-file: .nvmrc
|
||||||
|
#
|
||||||
|
# - name: Install deps
|
||||||
|
# working-directory: ./plugins
|
||||||
|
# run: |
|
||||||
|
# corepack enable;
|
||||||
|
# corepack install;
|
||||||
|
# pnpm install;
|
||||||
|
#
|
||||||
|
# - name: Install Playwright Chromium
|
||||||
|
# working-directory: ./plugins
|
||||||
|
# run: pnpm --filter plugin-api-test-suite exec playwright install --with-deps chromium
|
||||||
|
#
|
||||||
|
# - name: Generate API surface
|
||||||
|
# working-directory: ./plugins
|
||||||
|
# run: pnpm --filter plugin-api-test-suite run gen:api
|
||||||
|
#
|
||||||
|
# # Note: requires a running Penpot instance reachable at PENPOT_BASE_URL.
|
||||||
|
# - name: Run API test suite
|
||||||
|
# working-directory: ./plugins
|
||||||
|
# run: pnpm --filter plugin-api-test-suite run test:ci
|
||||||
9
.github/workflows/tests-plugins.yml
vendored
9
.github/workflows/tests-plugins.yml
vendored
@ -26,9 +26,6 @@ concurrency:
|
|||||||
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
||||||
cancel-in-progress: true
|
cancel-in-progress: true
|
||||||
|
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
test-plugins:
|
test-plugins:
|
||||||
if: ${{ !github.event.pull_request.draft }}
|
if: ${{ !github.event.pull_request.draft }}
|
||||||
@ -41,14 +38,14 @@ jobs:
|
|||||||
- /var/cache/github-runner/gitlib:/root/.gitlibs
|
- /var/cache/github-runner/gitlib:/root/.gitlibs
|
||||||
|
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
- uses: actions/checkout@v6
|
||||||
with:
|
|
||||||
persist-credentials: false
|
|
||||||
|
|
||||||
- name: Install deps
|
- name: Install deps
|
||||||
working-directory: ./plugins
|
working-directory: ./plugins
|
||||||
shell: bash
|
shell: bash
|
||||||
run: |
|
run: |
|
||||||
|
corepack enable;
|
||||||
|
corepack install;
|
||||||
pnpm install -r;
|
pnpm install -r;
|
||||||
|
|
||||||
- name: Run Lint
|
- name: Run Lint
|
||||||
|
|||||||
9
.github/workflows/tests-wasm.yml
vendored
9
.github/workflows/tests-wasm.yml
vendored
@ -26,9 +26,6 @@ concurrency:
|
|||||||
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
||||||
cancel-in-progress: true
|
cancel-in-progress: true
|
||||||
|
|
||||||
permissions:
|
|
||||||
contents: read
|
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
test-render-wasm:
|
test-render-wasm:
|
||||||
if: ${{ !github.event.pull_request.draft }}
|
if: ${{ !github.event.pull_request.draft }}
|
||||||
@ -42,14 +39,12 @@ jobs:
|
|||||||
|
|
||||||
steps:
|
steps:
|
||||||
- name: Checkout repository
|
- name: Checkout repository
|
||||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
uses: actions/checkout@v6
|
||||||
with:
|
|
||||||
persist-credentials: false
|
|
||||||
|
|
||||||
- name: Format
|
- name: Format
|
||||||
working-directory: ./render-wasm
|
working-directory: ./render-wasm
|
||||||
run: |
|
run: |
|
||||||
./fmt --check
|
cargo fmt --check
|
||||||
|
|
||||||
- name: Lint
|
- name: Lint
|
||||||
working-directory: ./render-wasm
|
working-directory: ./render-wasm
|
||||||
|
|||||||
39
.github/zizmor.yml
vendored
39
.github/zizmor.yml
vendored
@ -1,39 +0,0 @@
|
|||||||
# zizmor configuration — https://docs.zizmor.sh/configuration/
|
|
||||||
#
|
|
||||||
# Every audit is enforced for every workflow. New workflows are checked with
|
|
||||||
# no exceptions. The entries below are the only remaining ones, and each is
|
|
||||||
# marked as either a pending clean-up or an accepted risk with its reason.
|
|
||||||
# Remove a file from a list in the same PR that fixes it, and the whole rule
|
|
||||||
# entry once its list is empty.
|
|
||||||
rules:
|
|
||||||
# Accepted: wrangler is added with `pnpm add wrangler@latest` before each
|
|
||||||
# deploy instead of being a dependency of the plugins workspace. Only used
|
|
||||||
# to publish to Cloudflare; not worth adding to the workspace.
|
|
||||||
adhoc-packages:
|
|
||||||
ignore:
|
|
||||||
- plugins-deploy-api-doc.yml
|
|
||||||
- plugins-deploy-package.yml
|
|
||||||
- plugins-deploy-styles-doc.yml
|
|
||||||
# Accepted: auto-label.yml needs pull_request_target so the GitHub App
|
|
||||||
# secrets are available for PRs from forks. It never checks out or runs
|
|
||||||
# PR code, it only reads ids from the event payload.
|
|
||||||
dangerous-triggers:
|
|
||||||
ignore:
|
|
||||||
- auto-label.yml
|
|
||||||
# Style nudge towards the `$/...` syntax; not worth enforcing.
|
|
||||||
self-repository:
|
|
||||||
disable: true
|
|
||||||
# Accepted: penpotapp/devenv is our own image, built and pushed by
|
|
||||||
# build-docker-devenv.yml, so the floating tag is intentional.
|
|
||||||
unpinned-images:
|
|
||||||
ignore:
|
|
||||||
- plugins-deploy-package.yml
|
|
||||||
- tests-backend.yml
|
|
||||||
- tests-common.yml
|
|
||||||
- tests-e2e.yml
|
|
||||||
- tests-exporter.yml
|
|
||||||
- tests-frontend.yml
|
|
||||||
- tests-library.yml
|
|
||||||
- tests-mcp.yml
|
|
||||||
- tests-plugins.yml
|
|
||||||
- tests-wasm.yml
|
|
||||||
21
.gitignore
vendored
21
.gitignore
vendored
@ -16,10 +16,7 @@
|
|||||||
.repl
|
.repl
|
||||||
opencode.json
|
opencode.json
|
||||||
.opencode/package-lock.json
|
.opencode/package-lock.json
|
||||||
/*.html
|
|
||||||
/*.jpg
|
/*.jpg
|
||||||
/*.json
|
|
||||||
!/package.json
|
|
||||||
/*.md
|
/*.md
|
||||||
!CHANGES.md
|
!CHANGES.md
|
||||||
!CONTRIBUTING.md
|
!CONTRIBUTING.md
|
||||||
@ -27,6 +24,7 @@ opencode.json
|
|||||||
!AGENTS.md
|
!AGENTS.md
|
||||||
!CODE_OF_CONDUCT.md
|
!CODE_OF_CONDUCT.md
|
||||||
!SECURITY.md
|
!SECURITY.md
|
||||||
|
!HIGHLIGHTS.md
|
||||||
/*.png
|
/*.png
|
||||||
/*.svg
|
/*.svg
|
||||||
/*.sql
|
/*.sql
|
||||||
@ -61,8 +59,6 @@ opencode.json
|
|||||||
/docker/images/bundle*
|
/docker/images/bundle*
|
||||||
/exporter/target
|
/exporter/target
|
||||||
/exporter/.shadow-cljs
|
/exporter/.shadow-cljs
|
||||||
/exporter/resources/wasm/
|
|
||||||
/exporter/src/app/wasm/shared.js
|
|
||||||
/frontend/.storybook/preview-body.html
|
/frontend/.storybook/preview-body.html
|
||||||
/frontend/.storybook/preview-head.html
|
/frontend/.storybook/preview-head.html
|
||||||
/frontend/playwright-report/
|
/frontend/playwright-report/
|
||||||
@ -86,7 +82,6 @@ opencode.json
|
|||||||
/web
|
/web
|
||||||
/library/target/
|
/library/target/
|
||||||
/library/*.zip
|
/library/*.zip
|
||||||
/library/.shadow-cljs
|
|
||||||
/external
|
/external
|
||||||
/penpot-nitrate
|
/penpot-nitrate
|
||||||
/test-results/
|
/test-results/
|
||||||
@ -94,15 +89,7 @@ opencode.json
|
|||||||
/blob-report/
|
/blob-report/
|
||||||
/playwright/.cache/
|
/playwright/.cache/
|
||||||
/render-wasm/target/
|
/render-wasm/target/
|
||||||
/render-wasm/.render-wasm-build.lock
|
|
||||||
|
|
||||||
# render-wasm build artifacts copied into the exporter (autogenerated)
|
|
||||||
/exporter/resources/wasm/render-wasm.js
|
|
||||||
/exporter/resources/wasm/render-wasm.wasm
|
|
||||||
/exporter/src/app/wasm/shared.js
|
|
||||||
|
|
||||||
/media-processor/dist/
|
/media-processor/dist/
|
||||||
/media-processor/target/
|
|
||||||
/**/node_modules
|
/**/node_modules
|
||||||
/**/.yarn/*
|
/**/.yarn/*
|
||||||
/.pnpm-store
|
/.pnpm-store
|
||||||
@ -111,18 +98,12 @@ opencode.json
|
|||||||
*.iml
|
*.iml
|
||||||
/.claude
|
/.claude
|
||||||
/CLAUDE.md
|
/CLAUDE.md
|
||||||
/CLAUDE.local.md
|
|
||||||
/.playwright-mcp
|
/.playwright-mcp
|
||||||
/.devenv/mcp/
|
/.devenv/mcp/
|
||||||
/opencode.json
|
/opencode.json
|
||||||
/.agents/plans
|
/.agents/plans
|
||||||
/.agents/local/
|
|
||||||
/AGENTS.local.md
|
|
||||||
/.opencode/reports
|
/.opencode/reports
|
||||||
/.opencode/prompts
|
/.opencode/prompts
|
||||||
/.ci-logs
|
/.ci-logs
|
||||||
/tmp
|
|
||||||
/.codex/
|
/.codex/
|
||||||
/scripts/__pycache__/
|
|
||||||
/tools/__pycache__
|
/tools/__pycache__
|
||||||
/scripts/__pycache__
|
|
||||||
|
|||||||
@ -1,169 +1,299 @@
|
|||||||
// Penpot OpenCode V2 plugin: custom tools for Penpot development.
|
import { tool } from "@opencode-ai/plugin"
|
||||||
//
|
|
||||||
// Tools are registered with `ctx.tool.transform()` and JSON Schema inputs.
|
|
||||||
// Keep this plugin dependency-free so the auto-discovered local plugin loads
|
|
||||||
// without project npm dependencies.
|
|
||||||
|
|
||||||
import path from "path"
|
import path from "path"
|
||||||
import { spawn } from "child_process"
|
import { spawn } from "child_process"
|
||||||
|
|
||||||
function runCommand(command, args, options = {}) {
|
const penpotPsqlTool = tool({
|
||||||
const {
|
description:
|
||||||
cwd,
|
"Execute a SQL command against the Penpot database. Uses the defaults from scripts/psql.",
|
||||||
env,
|
|
||||||
stdin,
|
|
||||||
closeStdin = false,
|
|
||||||
successMessage = "Command executed successfully",
|
|
||||||
} = options
|
|
||||||
|
|
||||||
return new Promise((resolve) => {
|
args: {
|
||||||
let stdout = ""
|
sql: tool.schema
|
||||||
let stderr = ""
|
.string()
|
||||||
|
.describe("SQL command to execute"),
|
||||||
|
|
||||||
const proc = spawn(command, args, { cwd, env })
|
test: tool.schema
|
||||||
|
.boolean()
|
||||||
proc.stdout.on("data", (data) => {
|
.describe("Use the penpot_test database")
|
||||||
stdout += data.toString()
|
.optional(),
|
||||||
})
|
|
||||||
|
|
||||||
proc.stderr.on("data", (data) => {
|
|
||||||
stderr += data.toString()
|
|
||||||
})
|
|
||||||
|
|
||||||
proc.on("error", (error) => {
|
|
||||||
resolve(`Error: ${error.message}`)
|
|
||||||
})
|
|
||||||
|
|
||||||
proc.on("close", (exitCode) => {
|
|
||||||
const output =
|
|
||||||
exitCode === 0
|
|
||||||
? stdout.trim() || successMessage
|
|
||||||
: `Error (exit ${exitCode}): ${
|
|
||||||
(stderr || stdout).trim() || "No error output"
|
|
||||||
}`
|
|
||||||
resolve(output)
|
|
||||||
})
|
|
||||||
|
|
||||||
// Close stdin so the child cannot wait on it indefinitely. `psql -c`
|
|
||||||
// never reads stdin, so only the paren-repair pipe mode needs this, but
|
|
||||||
// closing it unconditionally is harmless there.
|
|
||||||
if (stdin !== undefined) {
|
|
||||||
proc.stdin.end(stdin)
|
|
||||||
} else if (closeStdin) {
|
|
||||||
proc.stdin.end()
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
function executePsql(sql, useTestDb, cwd) {
|
|
||||||
const host = process.env.PENPOT_DB_HOST || "postgres"
|
|
||||||
const user = process.env.PENPOT_DB_USER || "penpot"
|
|
||||||
const db = useTestDb
|
|
||||||
? "penpot_test"
|
|
||||||
: process.env.PENPOT_DB_NAME || "penpot"
|
|
||||||
const password = process.env.PENPOT_DB_PASSWORD || "penpot"
|
|
||||||
|
|
||||||
const psqlArgs = ["-h", host, "-U", user, "-d", db, "-c", sql]
|
|
||||||
|
|
||||||
return runCommand("psql", psqlArgs, {
|
|
||||||
cwd,
|
|
||||||
env: { ...process.env, PGPASSWORD: password },
|
|
||||||
successMessage: "Query executed successfully",
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
function executeParenRepair({ files, code }, directory) {
|
|
||||||
const script = path.join(directory, "scripts/paren-repair")
|
|
||||||
|
|
||||||
const fileList = files
|
|
||||||
? files
|
|
||||||
.split(",")
|
|
||||||
.map((file) => file.trim())
|
|
||||||
.filter(Boolean)
|
|
||||||
: []
|
|
||||||
|
|
||||||
const childArgs =
|
|
||||||
fileList.length > 0 ? [script, ...fileList] : [script]
|
|
||||||
|
|
||||||
return runCommand("bb", childArgs, {
|
|
||||||
cwd: directory,
|
|
||||||
stdin: code,
|
|
||||||
closeStdin: true,
|
|
||||||
successMessage: "No changes needed",
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
const penpotPsqlInputSchema = {
|
|
||||||
type: "object",
|
|
||||||
properties: {
|
|
||||||
sql: {
|
|
||||||
type: "string",
|
|
||||||
description: "SQL command to execute",
|
|
||||||
},
|
|
||||||
test: {
|
|
||||||
type: "boolean",
|
|
||||||
description: "Use the penpot_test database",
|
|
||||||
},
|
|
||||||
},
|
},
|
||||||
required: ["sql"],
|
|
||||||
additionalProperties: false,
|
|
||||||
}
|
|
||||||
|
|
||||||
const parenRepairInputSchema = {
|
async execute(args, context) {
|
||||||
type: "object",
|
const host = process.env.PENPOT_DB_HOST || "postgres"
|
||||||
properties: {
|
const user = process.env.PENPOT_DB_USER || "penpot"
|
||||||
|
const db = args.test
|
||||||
|
? "penpot_test"
|
||||||
|
: process.env.PENPOT_DB_NAME || "penpot"
|
||||||
|
const password = process.env.PENPOT_DB_PASSWORD || "penpot"
|
||||||
|
|
||||||
|
const psqlArgs = ["-h", host, "-U", user, "-d", db, "-c", args.sql]
|
||||||
|
|
||||||
|
return new Promise((resolve) => {
|
||||||
|
let stdout = ""
|
||||||
|
let stderr = ""
|
||||||
|
|
||||||
|
const proc = spawn("psql", psqlArgs, {
|
||||||
|
cwd: context.worktree,
|
||||||
|
env: { ...process.env, PGPASSWORD: password },
|
||||||
|
})
|
||||||
|
|
||||||
|
proc.stdout.on("data", (data) => {
|
||||||
|
stdout += data.toString()
|
||||||
|
})
|
||||||
|
|
||||||
|
proc.stderr.on("data", (data) => {
|
||||||
|
stderr += data.toString()
|
||||||
|
})
|
||||||
|
|
||||||
|
proc.on("error", (error) => {
|
||||||
|
resolve(`Error: ${error.message}`)
|
||||||
|
})
|
||||||
|
|
||||||
|
proc.on("close", (exitCode) => {
|
||||||
|
const output =
|
||||||
|
exitCode === 0
|
||||||
|
? stdout.trim() || "Query executed successfully"
|
||||||
|
: `Error (exit ${exitCode}): ${
|
||||||
|
(stderr || stdout).trim() || "No error output"
|
||||||
|
}`
|
||||||
|
resolve(output)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
},
|
||||||
|
})
|
||||||
|
|
||||||
|
const parenRepairTool = tool({
|
||||||
|
description:
|
||||||
|
"Fix mismatched parentheses/braces in Clojure files (.clj, .cljs, .cljc) then reformat with cljfmt.",
|
||||||
|
|
||||||
|
args: {
|
||||||
// A string is used instead of an array so OpenCode displays it
|
// A string is used instead of an array so OpenCode displays it
|
||||||
// in the generic tool invocation.
|
// in the generic tool invocation.
|
||||||
files: {
|
files: tool.schema
|
||||||
type: "string",
|
.string()
|
||||||
description:
|
.describe(
|
||||||
"Comma-separated file paths to fix, for example: frontend/src/app/config.cljs, backend/src/core.clj",
|
"Comma-separated file paths to fix, for example: frontend/src/app/config.cljs, backend/src/core.clj",
|
||||||
},
|
)
|
||||||
code: {
|
.optional(),
|
||||||
type: "string",
|
|
||||||
description: "Code string to fix via stdin",
|
code: tool.schema
|
||||||
},
|
.string()
|
||||||
|
.describe("Code string to fix via stdin")
|
||||||
|
.optional(),
|
||||||
},
|
},
|
||||||
additionalProperties: false,
|
|
||||||
}
|
|
||||||
|
|
||||||
async function setup(ctx) {
|
async execute(args, context) {
|
||||||
// Use the plugin instance location as the working directory for both tools.
|
const script = path.join(context.worktree, "scripts/paren-repair")
|
||||||
const directory =
|
|
||||||
ctx.location.directory ?? ctx.location.project?.canonical
|
|
||||||
|
|
||||||
// Keep this callback synchronous: transforms are replayable state edits.
|
const files = args.files
|
||||||
// The async work happens later, inside each tool's `execute`.
|
? args.files
|
||||||
await ctx.tool.transform((editor) => {
|
.split(",")
|
||||||
editor.add({
|
.map((file) => file.trim())
|
||||||
name: "penpot-psql",
|
.filter(Boolean)
|
||||||
description:
|
: []
|
||||||
"Execute a SQL command against the Penpot database. Uses the defaults from scripts/psql.",
|
|
||||||
input: penpotPsqlInputSchema,
|
const paramInfo =
|
||||||
async execute(input) {
|
files.length > 0
|
||||||
const content = await executePsql(
|
? `files=[${files.join(", ")}]`
|
||||||
input.sql,
|
: args.code !== undefined
|
||||||
input.test === true,
|
? `code=(${args.code.length} chars)`
|
||||||
directory,
|
: "none"
|
||||||
)
|
|
||||||
return { content }
|
return new Promise((resolve) => {
|
||||||
},
|
const childArgs =
|
||||||
|
files.length > 0
|
||||||
|
? [script, ...files]
|
||||||
|
: [script]
|
||||||
|
|
||||||
|
const proc = spawn("bb", childArgs, {
|
||||||
|
cwd: context.worktree,
|
||||||
|
})
|
||||||
|
|
||||||
|
let stdout = ""
|
||||||
|
let stderr = ""
|
||||||
|
|
||||||
|
proc.stdout.on("data", (data) => {
|
||||||
|
stdout += data.toString()
|
||||||
|
})
|
||||||
|
|
||||||
|
proc.stderr.on("data", (data) => {
|
||||||
|
stderr += data.toString()
|
||||||
|
})
|
||||||
|
|
||||||
|
proc.on("error", (error) => {
|
||||||
|
resolve(`Error: ${error.message}`)
|
||||||
|
})
|
||||||
|
|
||||||
|
proc.on("close", (exitCode) => {
|
||||||
|
const output =
|
||||||
|
exitCode === 0
|
||||||
|
? stdout.trim() || "No changes needed"
|
||||||
|
: `Error (exit ${exitCode}): ${
|
||||||
|
(stderr || stdout).trim() || "No error output"
|
||||||
|
}`
|
||||||
|
|
||||||
|
resolve(output)
|
||||||
|
})
|
||||||
|
|
||||||
|
// Close stdin in all cases so the process cannot wait indefinitely.
|
||||||
|
if (args.code !== undefined) {
|
||||||
|
proc.stdin.end(args.code)
|
||||||
|
} else {
|
||||||
|
proc.stdin.end()
|
||||||
|
}
|
||||||
})
|
})
|
||||||
|
},
|
||||||
|
})
|
||||||
|
|
||||||
editor.add({
|
export default async function plugin() {
|
||||||
name: "paren-repair",
|
return {
|
||||||
description:
|
tool: {
|
||||||
"Fix mismatched parentheses/braces in Clojure files (.clj, .cljs, .cljc) then reformat with cljfmt.",
|
"paren-repair": parenRepairTool,
|
||||||
input: parenRepairInputSchema,
|
"penpot-psql": penpotPsqlTool,
|
||||||
async execute(input) {
|
},
|
||||||
const content = await executeParenRepair(input, directory)
|
}
|
||||||
return { content }
|
|
||||||
},
|
|
||||||
})
|
|
||||||
})
|
|
||||||
}
|
}
|
||||||
|
|
||||||
export default {
|
|
||||||
id: "penpot",
|
|
||||||
setup,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
// import { tool } from "@opencode-ai/plugin"
|
||||||
|
// import path from "path"
|
||||||
|
// import { spawn } from "child_process"
|
||||||
|
|
||||||
|
// function formatFiles(files) {
|
||||||
|
// if (files.length === 0) return "stdin"
|
||||||
|
|
||||||
|
// // Keep the visible tool title reasonably short.
|
||||||
|
// if (files.length <= 3) return files.join(", ")
|
||||||
|
|
||||||
|
// return `${files.slice(0, 3).join(", ")} (+${files.length - 3} more)`
|
||||||
|
// }
|
||||||
|
|
||||||
|
// const parenRepairTool = tool({
|
||||||
|
// description:
|
||||||
|
// "Fix mismatched parentheses/braces in Clojure files, then reformat with cljfmt.",
|
||||||
|
|
||||||
|
// args: {
|
||||||
|
// files: tool.schema
|
||||||
|
// .array(tool.schema.string())
|
||||||
|
// .describe("Array of file paths to fix")
|
||||||
|
// .optional(),
|
||||||
|
|
||||||
|
// code: tool.schema
|
||||||
|
// .string()
|
||||||
|
// .describe("Code string to fix via stdin")
|
||||||
|
// .optional(),
|
||||||
|
// },
|
||||||
|
|
||||||
|
// async execute(args, context) {
|
||||||
|
// const script = path.join(context.worktree, "scripts/paren-repair")
|
||||||
|
|
||||||
|
// const files = (args.files ?? []).map((file) => {
|
||||||
|
// const absolute = path.isAbsolute(file)
|
||||||
|
// ? file
|
||||||
|
// : path.resolve(context.worktree, file)
|
||||||
|
|
||||||
|
// return path.relative(context.worktree, absolute)
|
||||||
|
// })
|
||||||
|
|
||||||
|
// const targetSummary =
|
||||||
|
// files.length > 0
|
||||||
|
// ? formatFiles(files)
|
||||||
|
// : args.code !== undefined
|
||||||
|
// ? `stdin (${args.code.length} chars)`
|
||||||
|
// : "no input"
|
||||||
|
|
||||||
|
// // This updates the tool-call title immediately, while it is running.
|
||||||
|
// await context.metadata({
|
||||||
|
// title: `Paren repair: ${targetSummary}`,
|
||||||
|
// metadata: {
|
||||||
|
// files,
|
||||||
|
// codeChars: args.code?.length,
|
||||||
|
// },
|
||||||
|
// })
|
||||||
|
|
||||||
|
// const childArgs =
|
||||||
|
// args.files && args.files.length > 0
|
||||||
|
// ? [script, ...args.files]
|
||||||
|
// : [script]
|
||||||
|
|
||||||
|
// return new Promise((resolve) => {
|
||||||
|
// const proc = spawn("bb", childArgs, {
|
||||||
|
// cwd: context.worktree,
|
||||||
|
// })
|
||||||
|
|
||||||
|
// let stdout = ""
|
||||||
|
// let stderr = ""
|
||||||
|
|
||||||
|
// if (args.code !== undefined) {
|
||||||
|
// proc.stdin.end(args.code)
|
||||||
|
// }
|
||||||
|
|
||||||
|
// proc.stdout.on("data", (data) => {
|
||||||
|
// stdout += data.toString()
|
||||||
|
// })
|
||||||
|
|
||||||
|
// proc.stderr.on("data", (data) => {
|
||||||
|
// stderr += data.toString()
|
||||||
|
// })
|
||||||
|
|
||||||
|
// proc.on("close", (exitCode) => {
|
||||||
|
// const successful = exitCode === 0
|
||||||
|
|
||||||
|
// const commandOutput = successful
|
||||||
|
// ? stdout.trim() || "No changes needed"
|
||||||
|
// : `Error (exit ${exitCode}): ${(stderr || stdout).trim()}`
|
||||||
|
|
||||||
|
// const parameterOutput =
|
||||||
|
// files.length > 0
|
||||||
|
// ? `Files passed:\n${files.map((file) => `- ${file}`).join("\n")}`
|
||||||
|
// : args.code !== undefined
|
||||||
|
// ? `Input passed through stdin: ${args.code.length} characters`
|
||||||
|
// : "No files or stdin input were passed"
|
||||||
|
|
||||||
|
// resolve({
|
||||||
|
// title: `Paren repair: ${targetSummary}`,
|
||||||
|
// output: `${parameterOutput}\n\n${commandOutput}`,
|
||||||
|
// metadata: {
|
||||||
|
// files,
|
||||||
|
// codeChars: args.code?.length,
|
||||||
|
// exitCode,
|
||||||
|
// successful,
|
||||||
|
// },
|
||||||
|
// })
|
||||||
|
// })
|
||||||
|
|
||||||
|
// proc.on("error", (error) => {
|
||||||
|
// resolve({
|
||||||
|
// title: `Paren repair failed: ${targetSummary}`,
|
||||||
|
// output: [
|
||||||
|
// files.length > 0
|
||||||
|
// ? `Files passed:\n${files.map((file) => `- ${file}`).join("\n")}`
|
||||||
|
// : `Input: ${targetSummary}`,
|
||||||
|
// `Failed to start bb: ${error.message}`,
|
||||||
|
// ].join("\n\n"),
|
||||||
|
// metadata: {
|
||||||
|
// files,
|
||||||
|
// codeChars: args.code?.length,
|
||||||
|
// successful: false,
|
||||||
|
// },
|
||||||
|
// })
|
||||||
|
// })
|
||||||
|
// })
|
||||||
|
// },
|
||||||
|
// })
|
||||||
|
|
||||||
|
// export default async function plugin() {
|
||||||
|
// return {
|
||||||
|
// tool: {
|
||||||
|
// "paren-repair": parenRepairTool,
|
||||||
|
// },
|
||||||
|
// }
|
||||||
|
// }
|
||||||
|
|||||||
@ -1,32 +0,0 @@
|
|||||||
import assert from "node:assert/strict"
|
|
||||||
import test from "node:test"
|
|
||||||
|
|
||||||
import plugin from "../plugins/penpot.js"
|
|
||||||
|
|
||||||
test("exports only the OpenCode V2 plugin contract", () => {
|
|
||||||
assert.equal(plugin.id, "penpot")
|
|
||||||
assert.equal("server" in plugin, false)
|
|
||||||
})
|
|
||||||
|
|
||||||
test("registers the Penpot tools during setup", async () => {
|
|
||||||
const tools = []
|
|
||||||
const context = {
|
|
||||||
location: { directory: "/tmp/opencode/penpot-plugin-test" },
|
|
||||||
tool: {
|
|
||||||
async transform(apply) {
|
|
||||||
apply({
|
|
||||||
add(tool) {
|
|
||||||
tools.push(tool)
|
|
||||||
},
|
|
||||||
})
|
|
||||||
},
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
await plugin.setup(context)
|
|
||||||
|
|
||||||
assert.deepEqual(
|
|
||||||
tools.map((tool) => tool.name).sort(),
|
|
||||||
["paren-repair", "penpot-psql"],
|
|
||||||
)
|
|
||||||
})
|
|
||||||
@ -1,342 +0,0 @@
|
|||||||
# Audit Log and event processing
|
|
||||||
|
|
||||||
Complete guide to the Penpot event system: what gets recorded, where each piece lives, and how events travel from the moment they happen until they are archived. Start with the overview (sections 1-3); the deeper technical detail comes later (section 8 on). If you only want the idea, the first three sections are enough.
|
|
||||||
|
|
||||||
> Note about this memory: unlike other Serena memories (dense notes written for LLM agents, see `mem:memory-maintenance`), this document is also written for human developers who need full context. That is why it uses plain language and examples. It replaces the old `mem:backend/audit-log`, which is now just a redirect stub.
|
|
||||||
|
|
||||||
## 1. What it is (2-minute overview)
|
|
||||||
|
|
||||||
The *audit log* answers "who did what, when, and from where". Every important user action becomes an **event**: a small object with a name, a type, who did it, from which IP, plus action data (`props`) and metadata (`context`).
|
|
||||||
|
|
||||||
The system has:
|
|
||||||
|
|
||||||
- **Three producers** (they create events): the Penpot backend (RPC layer), the Penpot frontend (browser app), and the Nitrate admin-console, which sends both server-side and client-side events.
|
|
||||||
- **One central buffer**: the Postgres `audit_log` table in Penpot. It is temporary: it holds events for minutes/hours until they are archived, then they are deleted.
|
|
||||||
- **Four consumers** (they read events): webhooks, error reporters, telemetry, and the external archives in Nexus and in the admin console.
|
|
||||||
- **Two long-term archives**: **Nexus**, a separate service that stores events for the long term, and the **admin console** (Nitrate), which ingests its own allowlisted events through a second archive job (see section 12).
|
|
||||||
|
|
||||||
```
|
|
||||||
Penpot frontend ─ POST push-audit-events (public) ─────────▶┐
|
|
||||||
Nitrate admin-console (browser) ─ POST push-audit-events ──▶│
|
|
||||||
Nitrate admin-console (server) ─ POST push-audit-events ──▶│ ─▶ audit_log table (buffer) ──┬─ cron 0s every 5m ──▶ Nexus (archive)
|
|
||||||
Penpot backend (RPC wrap-audit) ─ submit ───────────────────┘ │ └─ cron 30s every 5m ─▶ admin console (archive)
|
|
||||||
│ ├──▶ webhooks
|
|
||||||
│ ├──▶ error reporters
|
|
||||||
└──▶ telemetry (shadow rows)
|
|
||||||
```
|
|
||||||
|
|
||||||
The Penpot backend is the **center**: it checks, completes, and stores every event, no matter where it comes from. Nobody writes to the database directly; everything goes through the backend.
|
|
||||||
|
|
||||||
This document is the reference for all flows. One thing to keep in mind while reading: the Nitrate admin-console (both its server and its browser UI) is a **private** application, not a public client. Its events describe private SaaS management actions (organizations, subscriptions, SSO), and its server channel uses a trusted shared key instead of a user session.
|
|
||||||
|
|
||||||
## 2. The three kinds of events
|
|
||||||
|
|
||||||
Not all events are worth the same. This is the most important distinction in this document:
|
|
||||||
|
|
||||||
| Aspect | Backend events | Penpot frontend events | Nitrate admin-console events |
|
|
||||||
|---|---|---|---|
|
|
||||||
| What they describe | Real actions run on the server (create file, login, invite to team...) | Things that happen in the Penpot browser app (navigate, edit, JS errors, performance) | Organization and subscription actions in the admin-console (create/rename/delete organization, members, SSO settings, checkout, onboarding) |
|
|
||||||
| Trust level | **High**: created inside the same request that runs the action | **Best-effort**: some loss is expected (tab closed, offline, full buffer); they show **usage**, they are not strict proof | **Mixed**: server-side events are high trust (like backend); browser events are best-effort (like frontend) |
|
|
||||||
| Who sets `profile-id` | The server, from the logged-in session (never trusts the client) | The server too: it ignores what the client says and uses the session | Server channel: the trusted admin-console server sends it explicitly. Browser channel: the Penpot server uses the session |
|
|
||||||
| Who sets the time | Server time (`created-at`); the source of truth | Client proposes `tracked-at`, but the server fixes it if it is in the future or more than 1h late | Same rule: server stamps `created-at`; client `tracked-at` is corrected |
|
|
||||||
| Used for | Audit, webhooks, compliance, abuse checks | Product analytics, usage counters, crash reports | Organization audit trail (admin-console e2e spec), subscription metrics |
|
|
||||||
|
|
||||||
Rule of thumb: if you need to know for sure "did this user do X?", look at `source = backend`. If you want to know "how do people use the app?", also look at `source = frontend`.
|
|
||||||
|
|
||||||
## 3. The `audit_log` table: a buffer, not a store
|
|
||||||
|
|
||||||
The Penpot `audit_log` table **is not the final destination**. It is a waiting tray: events sit here until every active consumer (Nexus, admin console) confirms them, and whoever finishes last deletes the row.
|
|
||||||
|
|
||||||
Main columns (migration `backend/src/app/migrations/sql/0054-add-audit-log-table.sql` and later ones):
|
|
||||||
|
|
||||||
- `id` uuid PK (`gen_random_uuid()` today; the PK used to be `(created_at, profile_id)`).
|
|
||||||
- `name` / `type` text NOT NULL (for example `name = "create-file"`, `type = "action"`).
|
|
||||||
- `created_at` timestamptz NOT NULL default `now()`: **server time**, the source of truth.
|
|
||||||
- `tracked_at` timestamptz default `now()`: what the client claimed; fixed on ingest.
|
|
||||||
- `profile_id` uuid NOT NULL: who did it.
|
|
||||||
- `source` text: `backend` / `frontend` for full rows, `telemetry:backend` / `telemetry:frontend` for anonymized copies (see section 10).
|
|
||||||
- `ip_addr` inet: real IP (or `0.0.0.0` on telemetry rows).
|
|
||||||
- `props` / `context` jsonb holding transit-encoded maps: action data and metadata (versions, browser, request-id...).
|
|
||||||
- No `archived_at`: the `audit_log_pending` queue (migration `0157`, consumer `1` = Nexus, `2` = admin console, ids never reused) tracks what is left. Every record is born pending only for the consumers enabled at insert time (never a fixed both): cheaper on single-consumer hosts and true to the design doc. Consequence, accepted on purpose: rows born with no send flag on get no pendings, so enabling archiving later does not pick them up — the sweep deletes them unarchived.
|
|
||||||
|
|
||||||
Indexes (each consumer has its own, so a slow consumer never blocks the others):
|
|
||||||
|
|
||||||
- `audit_log_pending` PK `(consumer, audit_log_id)`: each consumer owns its queue at constant cost, even with millions pending.
|
|
||||||
- `audit_log (created_at, id)` (migration `0157`): the chronological drain order (`ORDER BY created_at ASC, id ASC`; same total order for both consumers, so concurrent drains cannot deadlock).
|
|
||||||
- `audit_log_pending (audit_log_id)`: serves the "is anything still pending?" check of the final delete.
|
|
||||||
- `(source, created_at)`: used by the telemetry scan.
|
|
||||||
|
|
||||||
Life of a row: `INSERT` (row plus one pending per enabled consumer, same transaction; a rollback takes both) → (every 5 min, Nexus at second 0 and admin console at second 30) each archive job sends its own 128-row batches oldest-first and deletes its pendings in the same transaction → whoever deletes the last pending deletes the row. A batch the destination refuses rolls back and stays for the next tick. Pendings of consumers that are off are reaped at the start of each job run (covers later deactivations), and orphan rows are swept at the end of each run, so there is no separate GC task. The FK is plain `DEFERRABLE` (never `INITIALLY DEFERRED`): every deleter removes pendings before rows, including the telemetry task's own deletes.
|
|
||||||
|
|
||||||
## 4. Anatomy of an event
|
|
||||||
|
|
||||||
Every event ends up with this shape (schema `schema:event` in `backend/src/app/loggers/audit.clj`):
|
|
||||||
|
|
||||||
- `name`: what happened (`"create-file"`, `"navigate"`, `"create-organization"`...). For backend, it defaults to the RPC command name (prefixed with `<module>-` outside `main`).
|
|
||||||
- `type`: category (`"action"`, `"identify"`, `"trigger"`). Most are `"action"`.
|
|
||||||
- `profile-id`: **who** did it (uuid). See the invariant in section 5.
|
|
||||||
- `ip-addr`: request IP (or `"0.0.0.0"` on telemetry).
|
|
||||||
- `props`: action data (team/project/file ids, visited route...). **Never secrets** (see `clean-props`).
|
|
||||||
- `context`: metadata about how/when it was captured (backend/frontend version, user-agent, locale, browser, `request-id`, `initiator`, `event-origin`...). See section 8 for the full field guide, including `initiator` (which app started it: `"app"` means Penpot itself).
|
|
||||||
- `source`: `backend`, `frontend`, `telemetry:backend`, `telemetry:frontend`. Nitrate admin-console server events land as `backend`; its browser events land as `frontend` (see section 7).
|
|
||||||
- `source`: `backend`, `frontend`, `telemetry:backend`, `telemetry:frontend`.
|
|
||||||
- `tracked-at` / `created-at`: claimed time and server time.
|
|
||||||
- Memory-only (never stored): `::webhooks/event?`, `::webhooks/batch-key`, `::webhooks/batch-timeout` for the webhook fan-out.
|
|
||||||
|
|
||||||
Backend `create-file` example (simplified):
|
|
||||||
|
|
||||||
```clojure
|
|
||||||
{:name "create-file" :type "action" :source "backend"
|
|
||||||
:profile-id #uuid "…" :ip-addr "203.0.113.7"
|
|
||||||
:props {:project-id #uuid "…" :name "My design"}
|
|
||||||
:context {:version "2.x" :initiator "app" :request-id "…"
|
|
||||||
:client-version "2.x" :client-user-agent "Mozilla/…"}
|
|
||||||
:created-at #inst "…" :tracked-at #inst "…"}
|
|
||||||
```
|
|
||||||
|
|
||||||
Frontend `navigate` example:
|
|
||||||
|
|
||||||
```clojure
|
|
||||||
{:name "navigate" :type "action" :source "frontend"
|
|
||||||
:profile-id #uuid "…" :ip-addr "203.0.113.7"
|
|
||||||
:props {:route "dashboard" :team-id #uuid "…"}
|
|
||||||
:context {:version "2.x" :locale "en" :browser "Chrome" :os "Linux"
|
|
||||||
:event-origin "dashboard" :session-id "…" :session #inst "…"}
|
|
||||||
:created-at #inst "…" :tracked-at #inst "…"}
|
|
||||||
```
|
|
||||||
|
|
||||||
## 5. Backend events: the happy path
|
|
||||||
|
|
||||||
Most backend events need **no manual code**. The `wrap-audit` wrapper in `backend/src/app/rpc.clj` runs after every RPC command when `:webhooks`, `:audit-log`, or `:telemetry` is on (unless the command sets `::audit/skip`). It builds the event with `prepare-rpc-event` and passes it to `audit/submit`.
|
|
||||||
|
|
||||||
Each command can customize through result metadata (`rph/with-meta`):
|
|
||||||
|
|
||||||
- `::audit/replace-props`: replaces props fully (auth commands use `profile->props` so a register event carries the profile, not the password).
|
|
||||||
- `::audit/props`: merges extras over the request params.
|
|
||||||
- `::audit/context`, `::audit/profile-id`, `::audit/name`, `::audit/type`: override the defaults.
|
|
||||||
- `clean-props` always strips nils, namespaced keys, and `:session-id/:password/:old-password/:token/:client-secret` as a last line of defense. **Never put secrets in props.**
|
|
||||||
|
|
||||||
Critical `profile-id` INVARIANT: the event belongs to **whoever made the request**. It resolves as `::audit/profile-id` metadata → `::rpc/profile-id` → `uuid/zero`. It is **never** taken from the response: some results carry someone else's `:profile-id` (for example `get-error-report` returns the report with its content merged; `verify-token` on an invitation returns the inviter), and using it would blame the wrong person. Commands with `::rpc/auth false` (`login-*`, `register-profile`, `create-demo-profile`, `verify-token`, `prepare-register-profile`) have no caller to fall back on, so they **must** set `::audit/profile-id` in the metadata or the event lands on `uuid/zero`.
|
|
||||||
|
|
||||||
The `::audit/profile-id` override goes through `coerce-profile-id`: a string is parsed, anything that cannot become a uuid is dropped (with a warning) and the event falls back to the caller. Do not pass the value raw: `schema:event` needs a uuid and `submit*` swallows the validation error (it only logs, the RPC still succeeds), so a non-uuid override **loses the row silently**. Hand-built events with `event-from-rpc-params` + `submit` (clone template, accept team invitation, `management/nitrate` push-audit-events) do **not** go through that coercion: they must already carry a uuid.
|
|
||||||
|
|
||||||
### Team id and entity ids (projection data)
|
|
||||||
|
|
||||||
The event stream is meant to let a consumer (Nexus) rebuild what it needs **without joining production tables**, so events carry the ids a projection keys on:
|
|
||||||
|
|
||||||
- **Team id enrichment.** After `clean-props`, `prepare-rpc-event` calls `enrich-team-id`: when props have no `:team-id` but carry a `:project-id` or `:file-id`, it resolves the team with a local query and adds `:team-id`. It never resolves from `:id` (read commands put the entity id there, and that would add a query to every read), and it skips read events entirely (`get-*` names plus `has-file-libraries`: their events carry no consumer state). Commands that already loaded the team add `:team-id` themselves in `::audit/props` (`delete-file`, comments, media upload, `delete/update-webhook`, `clone-template`, ...), so the resolver does not run.
|
|
||||||
- **Membership.** `add-profile-to-team!` emits `add-team-member` (props `:team-id`, `:member-id`, `:role`, `:is-owner`, `:is-admin`, `:can-edit`) whenever it actually inserts a row, only with `:audit-log` on, inside the caller's transaction. It covers the flows that add a member with no event of their own (`create-team` owner, invitee added directly when `:email-verification` is off, approved join-request, duplicated team). With `on-conflict-do-nothing?` a conflict returns nil and emits nothing, so an existing membership is reported once. The event belongs to the actor: the two flows where the owner adds somebody else (`create-invitation`, `add-member-to-team`) pass `:actor-id` through a separate `event-opts` arg (never in `params`, which is inserted as-is); everywhere else actor and member coincide. `:role` is derived in the emitter from the flags (inverse of `permissions-for-role`: owner > admin > editor > viewer) because the table has no role column; request context travels in `event-opts` as `:rpc-params`.
|
|
||||||
- **Created entity ids.** `create-project` and `create-file` add the new row's `:id` (and `:project-id`/`:team-id`) to `::audit/props`, so a projection can key it.
|
|
||||||
- **Bulk operations.** `clone-template`, `import-binfile` and `duplicate-project` create entities through internal calls that emit nothing on their own; they emit a `create-file` (and `create-project`) per entity (only with `:audit-log`). The import paths go through `audit/submit-file-created-event`, which reads the light columns (`id`, `name`, `project-id`, timestamps, `revn`) back from the row — the file must really exist, a phantom id emits nothing. `duplicate-project` already holds the rows and uses `audit/submit-create-event` directly. Both binfile versions return the same shape from `import-files!` (`{:file-ids [...]}`).
|
|
||||||
- **Deletes.** `delete-team` still returns nil but carries the deleted row (`:id`, `:name`, `:created-at`, `:modified-at`, `:deleted-at`) in `::audit/props`, like `delete-project`. `delete-profile` carries `:id`, `:email`, `:fullname`, `:created-at`, `:modified-at` and `:deleted-at` (without this its props are empty: every param is namespaced and `clean-props` drops them). `create-team` adds `:created-at` and `:modified-at`. `delete-team-member` needs nothing: the delete is physical and the params already carry `:team-id`/`:member-id`.
|
|
||||||
- **Hard deletes.** `objects-gc` emits one `hard-delete-objects` event per purged chunk, carrying only that chunk's entity key (`:team-ids`, `:project-ids` or `:file-ids`; one chunk transaction purges a single entity, so one event can never span sections) with context `{:triggered-by "objects-gc"}`, only with `:audit-log` on. No other GC path emits (profiles, fonts, thumbnails); those purges stay silent.
|
|
||||||
- **SREPL deletes.** `delete-profile!` carries the subject id in props (it used to emit nothing usable); the event profile stays zero because SREPL has no actor. `delete-team!` props already carried the id. Both travel as actions with a `triggered-by: srepl` context.
|
|
||||||
- **SREPL restores.** `restore-profile!` carries the full row (with id) in props with a zero event profile; `restore-team!` and `restore-project!` already carried their ids. The consumer resolves the subject from props by a uniform rule, with no SREPL special case.
|
|
||||||
- **Cascades.** The `:delete-object` worker task emits one `delete-object-with-cascade` event per execution, type `action`, with `:object`/`:id`/`:deleted-at` plus the affected ids nested under `:cascade` (`:team-ids`, `:project-ids`, `:file-ids`), in the same transaction. The handler submits unconditionally (`submit` fans out per flag, so no `:audit-log` re-check); the `audit_log` row itself is still only written with `:audit-log` on. Direct calls (objects-gc) and snapshot cascades stay silent on purpose.
|
|
||||||
- **Ownership.** `create-team-with-invitations` carries the new team `:id` like `create-team`; `update-team-member-role` names the demoted member in `:prev-owner-id` on owner transfers.
|
|
||||||
|
|
||||||
Entry points:
|
|
||||||
|
|
||||||
- `submit`: the normal one (fills defaults, validates, runs inside `tx-run!`, never breaks the RPC).
|
|
||||||
- `insert`: low level, for CLI helpers and the webhook subsystem; direct write, no webhook/telemetry fan-out, silent unless `:audit-log` is on.
|
|
||||||
- Boot emits `trigger/instance-start` from `setup/props` so every restart stays visible.
|
|
||||||
|
|
||||||
Cases already covered by tests, do not break: `accept-organization-invitation` tells its two flows apart **with props, not with origin**: `:organization-member-add-source` is `"direct-organization-invitation"` or `"team-invitation"`, and `:belongs-to-team-on-add` repeats it as a boolean. Do not add a third prop. The `context` `:event-origin` belongs to the browser (`make-data-event` sets it and it is on the frontend allowlist; `safe-backend-context-keys` has no `:event-origin`, so a backend event has nowhere to put one).
|
|
||||||
|
|
||||||
## 6. Penpot frontend events: what is collected, with what guarantees
|
|
||||||
|
|
||||||
The browser **never writes to the table**; it POSTs transit batches to `push-audit-events` (`backend/src/app/rpc/commands/audit.clj`), which stamps server `id`, session `profile-id`, request IP, server `created-at`, and server-side `:initiator` (from the `x-client` header, see section 8; anything the client sends as `initiator` is overwritten), and distrusts the client clock (future or >1h-late `tracked-at` is reset, the original kept in `context.original-tracked-at`). Without `:audit-log`/`:telemetry`, or on a read-only pool, the endpoint is a no-op.
|
|
||||||
|
|
||||||
Valid types (`schema:frontend-event`): `"action"`, `"identify"`, `"trigger"`. Names must match `#"[\d\w-]{1,50}"` (max 250). `props` is a free map with no schema: **tests are the only guard of the contract** (see section 13).
|
|
||||||
|
|
||||||
What they are for: usage metrics (navigation, editing), workspace file stats, Nitrate membership changes, crash reports (`unhandled-exception` / `exception-page`). **Not proof for audit**: collection is best-effort and includes `PerformanceObserver` noise (`performance-*`, `user-input`, `long-task`). So backend tests must never assert exact frontend event counts.
|
|
||||||
|
|
||||||
How the browser collects (`frontend/src/app/main/data/event.cljs`), most to least important:
|
|
||||||
|
|
||||||
1. **On-demand start**: `initialize` asks the backend with `get-environment-data` (old `get-enabled-flags` is deprecated) and only starts on `:audit-log` or `:telemetry`. On RPC failure it still starts in telemetry mode (the backend will reject if needed).
|
|
||||||
2. **Potok translation**: `make-proto-event` (events with the `Event` protocol) and `make-data-event` (data-events with `::name`) turn the action into `{type, name, props, context}`. `simplify-props` flattens complex values (maps → `:placeholder/map`, and so on) and drops namespaced keys and nils. Context gains `:event-origin`, `:event-namespace`, `:event-symbol`.
|
|
||||||
3. **Capped buffer**: queue of max 1024 events (`append-to-buffer` drops beyond that); every 2s debounce (or on logout, or `::force-persist`) a chunk of max 100 is taken, filtered to the current profile, and sent fire-and-forget to `api/main/methods/push-audit-events` as transit. On confirm, the chunk leaves the buffer (`::chunk-persisted`).
|
|
||||||
4. **Performance noise**: `PerformanceObserver` (`event`/`longtask`) plus blocking measurement with `scheduler.postTask` create `trigger` events (`performance-blocking-event`, `user-input`, `long-task`) with 1s debounce and 1s threshold. Info only, not audit.
|
|
||||||
5. **`skip-audit?`**: UI flag (for example resuming a pending dashboard action after Nitrate SSO, or moving a team into an organization) so one user gesture is not counted twice.
|
|
||||||
|
|
||||||
## 7. Nitrate admin-console events: the third producer
|
|
||||||
|
|
||||||
The admin-console is the third producer. It sends organization and subscription events through **two channels** that look the same once stored, but have different trust levels. All the sending code (server and browser helpers) lives in the private Nitrate repo: it is not Penpot code and no file paths are cited here. What this section documents is only the Penpot-visible side: endpoints, payloads, and which event names arrive.
|
|
||||||
|
|
||||||
### 7a. Server-side channel (high trust, like backend events)
|
|
||||||
|
|
||||||
Code: the admin-console server has a helper that POSTs to the Penpot **management** API:
|
|
||||||
|
|
||||||
```
|
|
||||||
POST api/management/methods/push-audit-events
|
|
||||||
headers: x-shared-key: "admin-console <key>", Cookie: <user cookies>
|
|
||||||
body: {events: [{name, props, profileId, context?}]}
|
|
||||||
```
|
|
||||||
|
|
||||||
Auth happens at the HTTP layer (`backend/src/app/http/middleware.clj` `wrap-shared-key-auth`): the `x-shared-key` header carries `"<key-id> <secret>"` (`admin-console` here), checked in constant time against the configured keys; the key-id lands in the event context as `:initiator`. The call goes server-to-server (ky client, internal URI, user cookies forwarded).
|
|
||||||
|
|
||||||
Who uses it:
|
|
||||||
|
|
||||||
- Organization management (best-effort wrapper): `create-organization`, `rename-organization`, `delete-organization`, member add/remove, and more. It optionally passes the UI origin in the context.
|
|
||||||
- Subscriptions: `create-subscription`, renewals, cancellations, and more, including deployment lookup (saas/selfhost).
|
|
||||||
|
|
||||||
Failure policy: every send is best-effort. A failure never rolls back the business action; it is only recorded for debugging.
|
|
||||||
|
|
||||||
Backend side (`backend/src/app/rpc/management/nitrate.clj` `::push-audit-events`): `::audit/skip`, `::rpc/auth false` (trust comes from the shared key, not from a user session; only the `admin-console` key id is allowed). It builds context with `audit/prepare-context-from-request` + `request-id`, takes the given `profile-id` as-is (no coercion here, must already be a uuid), sets `tracked-at = request-at`, defaults `type` to `"action"`, and calls `audit/submit` (which defaults `source` to `"backend"`). So these rows land as **`source = backend`**: they are audited like backend events because a trusted service set the author.
|
|
||||||
|
|
||||||
### 7b. Browser channel (best-effort, like frontend events)
|
|
||||||
|
|
||||||
Code: the admin-console browser has a helper that POSTs to the Penpot **public** API:
|
|
||||||
|
|
||||||
```
|
|
||||||
POST api/main/methods/push-audit-events
|
|
||||||
credentials: include, keepalive: true
|
|
||||||
headers: x-client: "penpot-admin-console/<version>"
|
|
||||||
body: {events: [{name, type: "action", timestamp, props, context: {eventOrigin?}}]}
|
|
||||||
```
|
|
||||||
|
|
||||||
Dozens of helpers cover the admin-console UI: `create-organization`, `delete-organization-member`, `create-subscription` / `create-trial-subscription`, `open-subscription-modal`, `start-nitrate-checkout`, `open-current-subscription`, SSO settings flows (`open/apply/success/fail/discard-organization-sso-changes`, activate/deactivate modals), `change-organization-advanced-permission`, `onboarding-step` / `onboarding-finish`, `close-subscription-modal`. These helpers live in the private Nitrate repo: they are not Penpot code, you will not find them in this repository. Only the event names above (what arrives at Penpot) are listed here on purpose.
|
|
||||||
|
|
||||||
These rows land as **`source = frontend`** with the session `profile-id`: same guarantees as Penpot frontend events (best-effort, usage analytics). Every browser helper sends an `eventOrigin` in the context: plain `"admin-console"`, with a suffix only when the same event is emitted from two places in the app (for example the SSO discard event distinguishes the config form from the unsaved-changes modal).
|
|
||||||
|
|
||||||
The Nitrate repo also has a shared, generic push client for the management endpoint (not tied to the admin-console server): any other Nitrate-side service that needs to record something in the Penpot audit log can reuse it instead of writing its own HTTP call. Same endpoint, same payload shape as the server channel above.
|
|
||||||
|
|
||||||
## 8. How the frontend helps the backend audit: HTTP headers
|
|
||||||
|
|
||||||
Besides explicit events, **every RPC request from the frontend carries headers** so the backend can fill its own event `context`. This is how the backend learns "which browser/version/screen" made the request:
|
|
||||||
|
|
||||||
- `x-frontend-version` and `x-client: penpot-frontend/<version>`: added by `default-headers` in `frontend/src/app/util/http.cljs` to **all** fetch requests (unless `omit-default-headers`). The admin-console browser channel sends `penpot-admin-console/<version>` instead (`x-frontend-version` is ignored on the event-ingest path, only `x-client` matters there).
|
|
||||||
- `user-agent`: set by the browser alone; the backend trims it to 500 chars (`get-client-user-agent`).
|
|
||||||
- `x-external-session-id`: external SaaS session via `cf/external-session-id`; added by `repo.cljs` (`send!`, special `cmd!`, `multipart-upload`, export...) and also stored by `persist-events` in the frontend event context. The backend checks it (max 256, ignores `"null"`/empty).
|
|
||||||
- `x-session-id`: `cf/session-id` (tab session), only in the main `send!`.
|
|
||||||
- `x-event-origin`: UI origin (`::ev/origin` from param metadata, for example `"dashboard"`, `"workspace"`); sent by `send!` and the OIDC/export/upload `cmd!`. The backend trims it to 200 chars and stores it as `:client-event-origin` (different from the `:event-origin` the frontend puts in its own context!).
|
|
||||||
|
|
||||||
On the backend, `prepare-context-from-request` (`backend/src/app/loggers/audit.clj`) joins everything into the event context: `:external-session-id`, `:initiator` (`auth-key-id` or `"app"`), `:access-token-id/type`, `:client-event-origin`, `:client-user-agent`, `:client-version`, `:version` (backend), plus `:request-id`. Only `safe-backend-context-keys` (`:version`, `:initiator`, `:client-version`, `:client-user-agent`) survive in the telemetry shadow copy; the rest (`:client-event-origin`, `:external-session-id`, `:access-token-id/type`, `:request-id`...) is stored fully in the normal row and sent to Nexus, but dropped from telemetry.
|
|
||||||
|
|
||||||
### Context field guide: `initiator`
|
|
||||||
|
|
||||||
`initiator` labels the application that started a request. The server sets it on both channels, but only the shared-key channel authenticates the sending application:
|
|
||||||
|
|
||||||
- **Backend channel** (RPC + management API): it comes from the shared-key auth layer (`wrap-shared-key-auth` in `backend/src/app/http/middleware.clj`): when a trusted service calls the management API with `x-shared-key: "<key-id> <secret>"`, the key-id (lowercased) is stored on the request as `::http/auth-key-id`, and `prepare-context-from-request` copies it into `:initiator`. When there is no shared key (a normal user RPC), it falls back to `"app"`, which means Penpot itself.
|
|
||||||
- **Frontend channel** (`push-audit-events`): `get-client-initiator` maps the `x-client` header product to an initiator: `penpot-frontend` → `"app"`, `penpot-admin-console` → `"admin-console"`, legacy `penpot-nitrate` (old admin-console releases, remove once redeployed) → `"admin-console"`, missing or unknown → `"app"`. It overwrites any `initiator` in the event context, but the caller controls the header. Any authenticated caller can claim `"admin-console"` by sending that product in `x-client`. Use this label for usage analytics; it does not prove which application sent the event or grant permissions. The session still determines `profile-id`.
|
|
||||||
|
|
||||||
Values you will see:
|
|
||||||
|
|
||||||
- `"app"`: normal Penpot traffic (browser → public RPC or event ingest, or backend internal work). This is the vast majority.
|
|
||||||
- `"admin-console"`: the authenticated Nitrate admin-console on management API calls (section 7a), or a caller claiming that product on browser events (section 7b, via the `x-client` mapping).
|
|
||||||
- `"nexus"`, `"exporter"`, `"media-processor"`: other first-party services using their own shared keys (key ids come from `::setup/shared-keys` in `backend/src/app/setup.clj`, derived from the instance secret unless overridden).
|
|
||||||
|
|
||||||
Why it matters: it supports per-application usage counts, with the trust level described above. It survives into telemetry on both channels (one of the four backend keys, and on the frontend allowlist too), so per-initiator counts stay available even on anonymized rows.
|
|
||||||
|
|
||||||
More metadata the Penpot frontend adds inside each event (not as headers): `collect-context` with `ua-parser` (browser, engine, OS, device, screen, CPU arch), `locale` (updated on language change), `:session` / `:session-id` / `:external-session-id`, and SaaS host extras (`add-external-context-info`).
|
|
||||||
|
|
||||||
## 9. Consumers I: webhooks
|
|
||||||
|
|
||||||
Webhooks let an external URL get notified when something happens in a team (for example "a file was created"). The flow has three steps:
|
|
||||||
|
|
||||||
1. **Match.** Some audit events are marked as webhook-relevant when they are created. For each of those, Penpot looks up the active webhooks of the affected team: it reads the team straight from the event, or finds it through the project or the file in the event. Each matching webhook gets its own delivery job. If several quick events belong together, they are grouped into one delivery (the grouping rule travels with the event itself).
|
|
||||||
2. **Deliver.** Penpot POSTs the event to the webhook URL in the format that webhook asked for: JSON (with camelCase keys), transit, or form-encoded. Every attempt is written to a delivery log with the request, the response, and the error if any, so failures can be inspected later.
|
|
||||||
3. **Protect.** Each webhook counts its consecutive failures. After 3 in a row, Penpot switches the webhook off automatically, so a dead URL stops receiving traffic until someone fixes it. A successful delivery resets the counter to zero.
|
|
||||||
|
|
||||||
One guarantee: sending webhooks never changes the audit event itself. If delivery fails, only the webhook (and its delivery log) is affected; the audit row stays untouched and still gets archived normally.
|
|
||||||
|
|
||||||
Every match also writes a `webhook` / `trigger` row to `audit_log` (even with zero matches): its props are `{:name <origin event name>, :event-id <origin row id>, :total-affected <matched webhook count>}`. `process-event` assigns the origin row id up-front so the trigger can reference it; the telemetry shadow row always gets its own id (reusing it would collide on the PK).
|
|
||||||
|
|
||||||
## 10. Consumers II: error reporters
|
|
||||||
|
|
||||||
Both `app.loggers.database` and `app.loggers.mattermost` listen to backend `:error` logs **and** frontend crash events (found by the `::audit/event` marker), through a sliding-buffer channel so a flood cannot stall the app. The database one stores them in `server-error-report` (source 3 = backend log, 4 = audit-log origin, 5 = rate-limit); the Mattermost one forwards a short note to `:error-report-webhook` when set.
|
|
||||||
|
|
||||||
Both reporters only start when the `:error-reporting` flag is on (off by default). The database reporter starts on the flag alone; the Mattermost reporter needs the flag **plus** its webhook URL configured. Without the flag, nothing is stored and `emit` calls from `push-audit-events` are silent no-ops. Each reporter also keeps a runtime on/off switch for emergencies via REPL.
|
|
||||||
|
|
||||||
Result: crash reports only exist if the frontend collector runs and `push-audit-events` accepts `unhandled-exception`/`exception-page`. Turning off the whole pipeline also blinds frontend error reporting.
|
|
||||||
|
|
||||||
## 11. Consumers III: telemetry (shadow rows)
|
|
||||||
|
|
||||||
Telemetry reuses the same table with **anonymized shadow rows** (`source LIKE 'telemetry:%'`): day-truncated times, `0.0.0.0` IPs, props reduced to uuid/boolean/number values plus a few allowed fields (`source`, `lang`, `auth-backend`, `activation`, derived `email-domain` — never raw emails), and a minimal context allowlist. Full + shadow rows can exist together per event when both flags are on; that duplication is on purpose.
|
|
||||||
|
|
||||||
Filters (`filter-telemetry-props` / `filter-telemetry-context`): by default only uuid/boolean/number with a simple key; exceptions: login/register/update-profile and frontend `identify` keep `lang`/`auth-backend`/`source`/`activation`/`email-domain`; `organization-sso-auth-failed` keeps `failure-reason` if it is a known one; `instance-start` passes whole; frontend `navigate` keeps `[:route :file-id :team-id :page-id]`. Subscription actions have event-specific contracts: backend and frontend `create-subscription` / `create-trial-subscription` preserve the default uuid/boolean/number props plus `type`, `deployment`, `quantity` and `subscription-mode`; backend `update-subscription` preserves only `type`, `deployment`, `previous-status`, `status`, `previous-cancel-at-period-end`, `cancel-at-period-end`, `previous-cancel-at`, `cancel-at` and `changed-fields`; backend `subscription-summary` preserves only `type`, `deployment`, `instance-id`, `snapshot-date`, the per-status `*-count` values, `total-count`, `sync-succeeded`, `sync-failed` and `snapshot-complete`. These subscription exceptions apply only to action events. Other free text drops from telemetry but **stays in the full row and in Nexus**. In `register-profile`, `activation` is a prop (it is what reaches PostHog). `action` exists only in the `context` of the full `register-profile-retry` rows; `filter-telemetry-context` drops it from the anonymised copy, which keeps only `:version`, `:initiator`, `:client-version` and `:client-user-agent`.
|
|
||||||
|
|
||||||
Shipping (`backend/src/app/tasks/telemetry.clj`): the cron sends shadows to `:telemetry-uri` as JSON in 10k batches and **deletes** them on success; it purges leftovers older than 7 days. It also sends a legacy report (counters, teams, email domains...). On official hosts nothing is collected or sent (`telemetry-excluded?` covers `penpot.app`/`penpot.dev`).
|
|
||||||
|
|
||||||
### Where telemetry events go: `telemetry.penpot.app` is Nexus
|
|
||||||
|
|
||||||
The default `:telemetry-uri` (`https://telemetry.penpot.app/`) points at the **Nexus telemetry endpoint**, so telemetry is a second road into Nexus, separate from the archive cron. The payload is plain JSON:
|
|
||||||
|
|
||||||
```
|
|
||||||
POST https://telemetry.penpot.app/
|
|
||||||
{type: "telemetry-events", version, instance-id, events: [...]}
|
|
||||||
```
|
|
||||||
|
|
||||||
Nexus dispatches on `:type`: `"telemetry-events"` batches are coerced and inserted into its own `audit_log`; anything else (old flat payloads, `"telemetry-legacy-report"`) is upserted into `telemetry_report` per instance + day. Three details matter:
|
|
||||||
|
|
||||||
- **Anonymization happens in Penpot, before sending.** By the time events leave, times are day-truncated, IPs are already `0.0.0.0`, props/context are filtered (see above), and emails are gone (only derived `email-domain` survives). Nexus additionally forces the outcome: it prefixes any non-`telemetry:` source, sets `received-at = tracked-at` (already truncated), pins IP to `0.0.0.0`, and stamps the sending `instance-id` into every event context.
|
|
||||||
- **No personal data is expected.** Profile ids are random UUIDs with no PII, so the telemetry consumer needs no login guard; grouping is by `instance`, not by person (see posthog-telemetry below).
|
|
||||||
- **The endpoint always answers 200.** Failures are swallowed and only counted (`nexus_telemetry_errors_total`), so a bad batch never breaks the sending instance; successes are counted (`nexus_telemetry_events_total`, `nexus_telemetry_reports_total`). A body the parser cannot read never reaches the handler: it answers 400 and counts in `nexus_http_malformed_requests_total`. Old instances may send a fressian+zstd base64 blob instead of a plain vector; Nexus decodes both (50 MiB cap against decompression bombs).
|
|
||||||
|
|
||||||
## 12. Archive to Nexus and to the admin console
|
|
||||||
|
|
||||||
### How it works today
|
|
||||||
|
|
||||||
Every 5 min each archive job drains its own queue (`:audit-log-consumer-nexus` on flag `:nexus`, cron `0 */5 * * * ?`; `:audit-log-consumer-admin-console` on flag `:admin-console`, cron `30 */5 * * * ?`, 30s offset so the two drains never start on the same second), in 128-row chunks taken oldest-first (`ORDER BY created_at ASC, id ASC`) with `FOR UPDATE OF audit_log` (`SET LOCAL lock_timeout = '30s'`) inside the same transaction that sends and completes them, with 100 ms pauses between chunks. Batches arrive at Nexus in chronological order (it numbers by arrival for order-sensitive consumers). Nexus receives transit `{:events [...]}` with `x-shared-key: "nexus <key>"` (`:nexus-shared-key`, or derived from the instance secret) and `origin` = public-uri; flag on but no URI raises `:task-not-configured`. The admin console receives only its allowlisted event names (enriched with team names, props emails, and optional top-level `:profile-email` / `:profile-name` from the actor `:profile-id`); anything filtered out counts as done for the admin-console queue. A chunk whose destination refuses it (Nexus non-204 raises `:consumer-chunk-refused`) or throws (transport failure, admin-console error) rolls back and stays for the next tick, and the job aborts as failed: it archives what it can, and the cron retries the rest. Committed chunks stay sent. The send fns return counts or throw carrying the attempted count as `:events`; `run-single-consumer` records the `failed` metric from it, outside every chunk transaction (a local DB failure carries no `:events` and stays uncounted — it is not a destination failure). No `warn` before the throw: the runner reports the aborted job. Each job touches only its own queue, so a fallen Nexus never starves the admin console (nor vice versa).
|
|
||||||
|
|
||||||
A chunk is **one POST** and the task advances only on 204, so a single oversized event wedges the whole queue: a crash report whose message embeds a whole payload (a serialized file, for example) can push the request past the proxy body limit and the rejection blocks every later event until it is pruned by hand. Frontend crash reports are therefore size-bounded at the source (`mem:frontend/routing-app-shell-subtleties`): `app.common.exceptions/get-hint` and the CLJS `format-throwable` clamp exception messages (`max-message-length`).
|
|
||||||
|
|
||||||
On the other side, Nexus checks `origin` against its allowlist (403 if not allowed), coerces the events, and inserts them into its own `audit_log` (partitioned by `received_at`, with an incrementing sequence number and `source`). Watch the rename: Penpot's `created-at` is stored as **`received-at`** in Nexus; Nexus's `created-at` is when it was created there. `tracked-at` keeps the origin time. Telemetry batches arriving at `telemetry.penpot.app` land in the same table through the telemetry endpoint above.
|
|
||||||
|
|
||||||
Concurrent jobs are safe: both lock rows in `created_at ASC, id ASC` order (one total order, deliberately blocking — no `SKIP LOCKED`, so archive arrival stays chronological), so the second waits for the first to commit and then sees whether it was last. Only acknowledged work deletes pendings; an admin-console failure never blocks the Nexus send of the same rows (each job owns its queue).
|
|
||||||
|
|
||||||
Observability (`backend/src/app/loggers/audit/consumer_task.clj`, definitions in `backend/src/app/main.clj:default-metrics`): one `dbg` summary per job run (`:consumer` label, sent/skipped/chunks, reaped disabled pendings, swept orphans, `:elapsed` as `"1.5s"` via `ct/format-duration`, frozen in a strict binding first because log props evaluate lazily in the logger thread), one `dbg` per admin-console send call (`ready` with totals, `sent`/`failed` with counts), plus `dbg` when the queue is empty or the job's send flag is off. Queue depth is Grafana's business: it queries the database directly, so the run performs no `GROUP BY` for the logs. No `warn` before a throw: the runner reports the aborted job (refusal details travel in the error data). Metrics, all recorded with bounded labels only: `penpot_audit_consumer_events_total{ destination = nexus|admin-console, outcome = archived|failed|skipped }` (`skipped` = filtered by the admin-console allowlist) and `penpot_audit_consumer_discarded_total{ reason = reaped|swept }` (entries discarded without archiving: `reaped` counts pendings dropped for disabled consumers, `swept` counts rows deleted while nobody waited for them — the silent R3 loss, now dashboard-visible instead of log-only). Timing needs no custom histogram: the jobs substrate already records execution time per job (`jobs-execution-timing`, split by job name), so each archive job is timed as itself. No queue-depth gauge: pending depth is read off the replica database directly. The bulk frontend ingest counts `penpot_audit_ingest_events_total{ kind = full|shadow, outcome = inserted|failed }` (both faces share one transaction, so a failure drops the batch on both). Archive success counts are recorded by `run-single-consumer` on the drain totals: a chunk that returns normally committed its transaction (a rollback always throws instead), so they cannot inflate. Logs never carry event data (`props`, `context`, `profile-id`, `ip-addr`, `team-name`, emails): only counts, statuses and the uri host.
|
|
||||||
|
|
||||||
### How Nexus consumes the data (and forwards to PostHog)
|
|
||||||
|
|
||||||
Stored rows are not the end: a consumer pipeline reads them back in sequence order and processes them. Each consumer remembers its place (sequence number), fetches the next chunk, runs its handler, then saves the new position. Three consumers run in order:
|
|
||||||
|
|
||||||
1. **`local`**: keeps a mirror of user profiles. It listens to profile events (`register-profile`, `update-profile`, `login-with-*`, `delete-profile`, ...) and upserts email, name, props, and delete markers. It runs first because the next consumer needs profile data.
|
|
||||||
2. **`posthog`** (runs after `local`): forwards our own SaaS archived events to PostHog. It sends an allowlist of backend track events (teams, members, projects, files, comments, Nitrate/SSO, ...) plus frontend events minus a noise exclusion list, grouped by **team**, with person identify data from the local profile mirror and snake_case props. Both paths include the `initiator` property when the stored context has one, so PostHog can split Penpot vs admin-console traffic.
|
|
||||||
3. **`posthog-telemetry`**: forwards events with `source = telemetry:frontend|telemetry:backend` to a separate PostHog project. Because these rows are already anonymous (no IPs, truncated times, filtered props), grouping is by **instance** (the self-hosted server, from `context.instance-id`), with one `$groupidentify` event per instance per chunk so PostHog learns per-instance properties (version, URI, flags). Each consumer has its own PostHog credentials.
|
|
||||||
|
|
||||||
So the full journey is: Penpot buffer → Nexus `audit_log` (via either archive cron or the telemetry endpoint) → consumers → PostHog (SaaS events grouped by team, telemetry grouped by instance) plus local analytics views and materialized KPIs.
|
|
||||||
|
|
||||||
### Admin console: the second consumer
|
|
||||||
|
|
||||||
Penpot → admin-console archiving lives in its own job (`:audit-log-consumer-admin-console`, consumer `2`, flag `:admin-console`, params closed empty): it drains its own pending queue and ingests allowlisted events through `nitrate/call :ingest-audit-log`, enriched as described above. What also exists (do not confuse with archiving):
|
|
||||||
|
|
||||||
- **Nitrate → Penpot**: `POST /api/management/methods/push-audit-events` (`backend/src/app/rpc/management/nitrate.clj`) lets the trusted Nitrate backend inject events into the Penpot audit log (no RPC auth, `::audit/skip`, request context + `submit`; only the `admin-console` shared-key id can call it). Section 7a is the main user.
|
|
||||||
- **Admin-console browser → collector**: usage events from the admin-console UI traveling through the normal public `push-audit-events` (section 7b).
|
|
||||||
- **Penpot frontend Nitrate bits**: membership/subscription events (`frontend/src/app/main/data/nitrate.cljs`, `nitrate-audit/*`) through the same public endpoint.
|
|
||||||
|
|
||||||
## 13. Flags and config
|
|
||||||
|
|
||||||
Flags (`common/src/app/common/flags.cljc`, varia; enabled with `PENPOT_FLAGS=enable-<name>`):
|
|
||||||
|
|
||||||
| Flag | What it does |
|
|
||||||
|---|---|
|
|
||||||
| `:audit-log` | Stores full rows; turns on `wrap-audit` and frontend ingest |
|
|
||||||
| `:nexus` | Send flag of the Nexus archive job `:audit-log-consumer-nexus` (cron `0 */5 * * * ?`; needs `:nexus-uri` (ingest path `api/audit` appended by the job)) |
|
|
||||||
| `:admin-console` | Send flag of the admin-console archive job `:audit-log-consumer-admin-console` (cron `30 */5 * * * ?`, 30s after Nexus); also the module flag for the admin-console integration |
|
|
||||||
| `:audit-log-logger` | Structured `app.audit` log per event |
|
|
||||||
| `:error-reporting` | Starts the error reporters (off by default); database reporter on flag alone, Mattermost on flag plus webhook URL |
|
|
||||||
| `:telemetry` | Stores shadow rows and turns on shipping (auto with `:telemetry-enabled`) |
|
|
||||||
|
|
||||||
Config (`backend/src/app/config.clj`): `:nexus-uri`, `:telemetry-uri` (default `https://telemetry.penpot.app/`), `:nexus-shared-key` / instance secret, `:public-uri` (sent as `origin` to Nexus).
|
|
||||||
|
|
||||||
## 14. Practical guide: add or change events
|
|
||||||
|
|
||||||
- **New backend command that must (or must not) emit**: `wrap-audit` does it alone; to exclude it set `::audit/skip`. To customize props/name/type use result metadata. Add a test mocking `app.loggers.audit/submit` (nil; `helpers.clj` stubs it globally) and assert `:call-args-list` — see `backend_tests/rpc-audit-test.clj` (full-row, telemetry-only, dual, no-op without flags, `insert`, `prepare-rpc-event`) and `rpc-team-test/accept-organization-invitation-audit-event` (three cases, exact props; counts matter: asserting with `first` passes even when a command emits the same name twice).
|
|
||||||
- **New Penpot frontend event**: create the data-event or implement `Event`/`PerformanceEvent` in `app.main.data.event`, with `::name`/`::type`/`::origin`. Honor `skip-audit?` if the action can resume. Remember: with no props schema, **your asserts are the contract**.
|
|
||||||
- **New admin-console event**: server action → add the backend-log call next to the mutation, best-effort so it never breaks the action; browser action → add a new event helper plus its test.
|
|
||||||
- **Command that creates an entity or already knows its team**: put the new row's `:id` in `::audit/props` (`create-project`/`create-file`) and set `:team-id` when the team is loaded, so the consumer never has to join production tables. For entities created inside an internal call (`clone-template`, `import-binfile`, `duplicate-project`), emit one event per entity: `audit/submit-file-created-event` when only the id is known (it reads the row back), `audit/submit-create-event` when the row is already in hand.
|
|
||||||
- **Never** put secrets in `props`; `clean-props` is a last defense, not an excuse.
|
|
||||||
- **Do not duplicate events**: if the backend already emits an event for an action, do not add a frontend (or admin-console browser) event for the same action — the backend one wins because it is reliable. A second event for the same action is only acceptable with a strong reason (for example the backend cannot see it at all), and then it must be clearly justified.
|
|
||||||
- **Follow the naming policy**: use kebab-case names (`"create-organization"`, not `"createOrganization"`), and before inventing prop keys, look at similar existing events and reuse their key names (`organization-id`, `team-id`, `file-id`, `event-origin`...). Same concept, same key spelling — that is what keeps the data queryable later.
|
|
||||||
|
|
||||||
## 15. Code map
|
|
||||||
|
|
||||||
- Backend producer: `backend/src/app/loggers/audit.clj` (`submit`, `submit*`, `insert`, `prepare-rpc-event`, `enrich-team-id`, `submit-create-event`, `event-from-rpc-params`, telemetry filters), `backend/src/app/rpc.clj` (`wrap-audit`), `backend/src/app/rpc/commands/audit.clj` (`push-audit-events`, `get-environment-data`, deprecated `get-enabled-flags`), `backend/src/app/rpc/management/nitrate.clj` (`push-audit-events` Nitrate→Penpot), `backend/src/app/http/middleware.clj` + `backend/src/app/http/management.clj` (shared-key auth).
|
|
||||||
- Membership emitter: `backend/src/app/rpc/commands/teams.clj` (`add-profile-to-team!` → `add-team-member`).
|
|
||||||
- Penpot frontend producer: `frontend/src/app/main/data/event.cljs` (collector, buffer, send), `frontend/src/app/main/repo.cljs` + `frontend/src/app/util/http.cljs` (headers), `frontend/src/app/main/data/nitrate.cljs` (Nitrate membership events).
|
|
||||||
- Admin-console producer (private repo, no paths cited here): server-side organization/subscription events through the management `push-audit-events`, browser-side UI events through the public `push-audit-events`.
|
|
||||||
- Consumers: `backend/src/app/loggers/webhooks.clj`, `backend/src/app/loggers/database.clj`, `backend/src/app/loggers/mattermost.clj`, `backend/src/app/tasks/telemetry.clj`.
|
|
||||||
- Archive (two jobs sharing one namespace + pending queue, no GC): `backend/src/app/loggers/audit/consumer_task.clj` (`:audit-log-consumer-nexus` / `:audit-log-consumer-admin-console` job-defs, shared drain/reap/sweep helpers), producer hook in `backend/src/app/loggers/audit.clj` (`append-audit-entry`, single `queue-pendings` helper) plus the bulk ingest in `backend/src/app/rpc/commands/audit.clj` (`handle-events`), metrics/cron wiring in `backend/src/app/main.clj` (one cron entry per job, Nexus at second 0 and admin console at second 30, each gated by its own send flag), single migration `0157` (queue, drop of `archived_at`, covering `(created_at, id)` index for the chronological scan; no backfill — pre-existing rows are swept unsent, an accepted loss).
|
|
||||||
- Nexus (private repo, no paths cited here): archive ingest, ordered consumers (local profile mirror, PostHog SaaS events, PostHog telemetry), telemetry endpoint, analytics views.
|
|
||||||
- Tests: `backend_tests/rpc-audit-test.clj`, `rpc-team-test/accept-organization-invitation-audit-event`.
|
|
||||||
@ -1,3 +0,0 @@
|
|||||||
# Backend Audit Log
|
|
||||||
|
|
||||||
Superseded by `mem:audit-log`: the full audit-log and event-processing guide (backend center, frontend collection, headers, webhooks, error reporters, telemetry, Nexus archive and planned Nitrate archive, table-as-buffer lifecycle, event anatomy, flags, tests). Read that memory instead; keep this stub so old references do not rot.
|
|
||||||
@ -7,37 +7,23 @@
|
|||||||
- Basic Penpot registration is token staged: prepare/register creates or verifies temporary tokens, then profile creation/session setup is reused by other auth backends. The frontend `/auth/verify-token` flow is a hub for registration confirmation, email change, and invitation tokens.
|
- Basic Penpot registration is token staged: prepare/register creates or verifies temporary tokens, then profile creation/session setup is reused by other auth backends. The frontend `/auth/verify-token` flow is a hub for registration confirmation, email change, and invitation tokens.
|
||||||
- OIDC-compatible providers share a generic flow: redirect to provider, validate callback/request token, fetch identity data, then login an existing profile or register a new one. Known providers may have hardcoded endpoints; generic OIDC can use discovery/configured endpoints.
|
- OIDC-compatible providers share a generic flow: redirect to provider, validate callback/request token, fetch identity data, then login an existing profile or register a new one. Known providers may have hardcoded endpoints; generic OIDC can use discovery/configured endpoints.
|
||||||
- LDAP login validates credentials against the external directory, fetches identity data, then logs in or registers a matching Penpot profile. LDAP registration is not a separate Penpot signup flow.
|
- LDAP login validates credentials against the external directory, fetches identity data, then logs in or registers a matching Penpot profile. LDAP registration is not a separate Penpot signup flow.
|
||||||
- LDAP session identity MUST come from the directory-returned email (`info.email`): the profile matching the typed email can differ (aliases, UPNs, multi-valued `mail` attributes) and is used only for lockout checks, never to bind the session.
|
|
||||||
- Account lockout (flag `:account-lockout`, `app.auth.login-lockout`) is Redis-backed and keyed per profile id. Password and LDAP flows check/increment on the profile derived from the typed email and clear on the profile that actually logs in.
|
|
||||||
- Lockout and RPC rate limits share the `app.http.errors/handle-error :rate-limit` HTTP path: status 429, body `{:type :rate-limit :code ... :hint ... :ttl ...}`, and any supplied `::http/headers` preserved. Account lockout raises `:code :account-locked` and, when it carries a non-nil `:ttl` (seconds), the handler adds `retry-after`. The RPC limiter (`app.rpc.rlimit`) raises `:code :request-blocked` and sets `retry-after` itself in `::http/headers` (seconds until the longest rejecting limit resets), alongside `x-rate-limit-remaining`/`x-rate-limit-reset`. CORS (`app.http.middleware/with-cors-headers`) exposes `content-type`, `retry-after`, and both `x-rate-limit-*` headers. External flags: `enable-account-lockout`, `enable-rpc-rlimit`.
|
|
||||||
- Logout may return an OIDC provider redirect URI when the session claims include provider/session data and the provider has a logout URI.
|
- Logout may return an OIDC provider redirect URI when the session claims include provider/session data and the provider has a logout URI.
|
||||||
- Organization SSO (Nitrate): the RPC `check-nitrate-sso` takes the route's **query string proper** (`:query-string`, e.g. `screen=workspace&team-id=…`), never a URL and never with the `?` separator: the client strips it (`rt/token->query-string`) and the server puts it back. The server builds the destination with `app.auth.oidc/build-dest-url` (`public-uri` + `?` + the params, parsed and rendered back), so the redirect can never leave the public URI; a value whose parsed keys are not param names is not a route: a URL parses into a single key holding the whole URL, and a key carrying `:`, `/`, `?` or `#` is rejected with a `:warn` and the user lands on the public URI (the RPC never fails because of it). Do not reason from `u/map->query-string`: it keeps a valueless key as `key=`, so rendering is not what filters a URL out — the key shape is. A leading `?` is still trimmed server-side, so a client bundled before the trim keeps its destination. Never require `screen`: the root of the app carries its params instead of a screen (`?template=…`), and `check-nitrate-sso` is called from that branch too (`dtm/resolve-login-team-id`, which today reads only `:authorized` and never follows the redirect, so a wrong destination there is a silent `:warn` and not a visible break); the callback re-checks the destination before redirecting, which covers state tokens minted before that. The state token also carries `:profile-id`: only that session is granted the `props :sso` entry (`session-mismatch` otherwise, and the redirect still happens).
|
|
||||||
- Organization SSO needs `app.nitrate/organization-access?` whenever the RPC receives an `:organization-id` (that id always comes from the client): the profile must be a member of the organization or its owner; otherwise the RPC answers `{:authorized true :reason :no-organization-access}`, the same short-circuit shape as `:no-team-access`.
|
|
||||||
- Invitation tokens are verified through token issuers and only accepted when the token member id/email matches the authenticated profile; otherwise login proceeds without consuming the invitation.
|
- Invitation tokens are verified through token issuers and only accepted when the token member id/email matches the authenticated profile; otherwise login proceeds without consuming the invitation.
|
||||||
- When public registration is disabled, invitation-based password registration also requires a matching, non-expired `team_invitation` row; registration revalidates it under a row lock before creating the profile. SSO registration (OIDC and compatible providers) follows the OIDC callback gate: it requires `:registration` or `:oidc-registration`, or a team invitation token issued for the registering email (an invitation substitutes the flags only when `member-email` matches the OIDC/`prepared-register` email via `clean-email`, as in the password flow, which raises `:email-does-not-match-invitation` otherwise). The OIDC callback verifies the invitation JWT and the email before redirecting to register, and `register-profile` revalidates the invitation under a row lock when no SSO flag is set; it never requires password login. Only OIDC-typed backends (`oidc-provider-types` in `app.auth.oidc`) take the SSO registration gate; any other `:backend` in a prepared-register token is rejected as `:invalid-token`.
|
- HTTP/session parsing details such as cookie/header precedence, JWT session token versions, and SameSite behavior are in `mem:backend/http-storage-filedata-subtleties`.
|
||||||
- Access tokens (`app.http.access-token`): personal tokens have JWT issuer `access-token` and DB type other than `mcp`; MCP tokens have issuer `urn:penpot:mcp-token` (`actoken/mcp-token-issuer`) and DB type `mcp`. Authentication requires issuer and type to agree, and the claim `uid` to match the row's profile; other issuers and undecryptable tokens are treated as anonymous.
|
|
||||||
- Management RPC (`/api/management/methods/:name`) is shared-key only. The middleware authenticates `<id> <secret>`; the RPC wrapper `wrap-shared-key-authorization` authorizes the key id against the method's `::rpc/allowed-shared-keys` set. Absent metadata accepts any configured key; all current methods declare one (`exporter`/`media-processor` for `upload-tempfile`, `nexus` for file analysis, `exporter` for jobs, `admin-console` for Nitrate). A valid key outside the set answers 403 `:shared-key-not-allowed`; denied calls do not reach the audit wrapper.
|
|
||||||
- Personal tokens need flag `:access-tokens`. MCP tokens need `:mcp` or `:access-tokens`, and only on routes whose `actoken/authz` gets `::actoken/allow-mcp true` (the RPC routes); elsewhere (e.g. assets) they are anonymous.
|
|
||||||
- On RPC, an MCP token reaches only commands with `::rpc/token-type "mcp"` (`wrap-authentication` in `app.rpc`); every other command answers 403 `:token-auth-required`. A command with that metadata also rejects sessions and personal tokens. Currently only `send-mcp-feedback`. Blocking a profile invalidates sessions, not tokens, so such commands check `:is-blocked`/`:is-active` themselves.
|
|
||||||
- HTTP/session parsing details such as cookie/header precedence, JWT session token versions, and SameSite behavior are in `mem:backend/subtleties`.
|
|
||||||
|
|
||||||
## Permission model
|
## Permission model
|
||||||
|
|
||||||
- `app.rpc.permissions` provides predicate/check factories. Failed permission checks intentionally raise `:not-found` / `:object-not-found`, not an authorization-specific error, to avoid leaking object existence.
|
- `app.rpc.permissions` provides predicate/check factories. Failed permission checks intentionally raise `:not-found` / `:object-not-found`, not an authorization-specific error, to avoid leaking object existence.
|
||||||
- Team role flags are normalized as owner > admin > editor > viewer. Owner/admin imply edit; any membership row implies read.
|
- Team role flags are normalized as owner > admin > editor > viewer. Owner/admin imply edit; any membership row implies read.
|
||||||
- File/project role rows only count with a live team membership: the file/project permission queries require a matching `team_profile_rel` row, and leaving or removing a member deletes their team-scoped `file`/`project`/`team-project-profile-rel` rows (GHSA-v9r9-h77c-55m2).
|
|
||||||
- The organization-owner fallback (`get-organization-owner-permissions` via Nitrate over HTTP) grants read-only access for a non-member organization owner. It resolves the owning team through `resolve-team-id`, which refuses soft-deleted file/project/team: the file and project lookups filter `deleted_at`, and a final `db/get*` check covers the team row itself — `nil` means no access. `db/get*` only filters soft-deleted rows when `deleted_at` is in the projection. The re-checks of open websocket subscriptions rely on this (`mem:backend/websocket-subscriptions`).
|
|
||||||
- File/project/comment checks are implemented in the owning command namespaces, often via helpers imported from `files`, `teams`, or `projects`; do not bypass those helpers with direct DB lookups unless preserving their not-found semantics.
|
- File/project/comment checks are implemented in the owning command namespaces, often via helpers imported from `files`, `teams`, or `projects`; do not bypass those helpers with direct DB lookups unless preserving their not-found semantics.
|
||||||
- Comment permission includes both logged-in state and the file/team comment policy. Shared viewer paths may pass `share-id`; preserve that path when changing comment queries. RPCs that accept `share-id` enforce the link's page scope before mutating (`share-link-pages` + `check-page-scope`): a thread or page outside the link raises `:not-found`, and that check runs before the ownership checks. `get-profiles-for-file-comments` filters its query by the same pages. `mark-all-threads-as-read` does not take `share-id`.
|
- Comment permission includes both logged-in state and the file/team comment policy. Shared viewer paths may pass `share-id`; preserve that path when changing comment queries.
|
||||||
|
|
||||||
## Teams, projects, and invitations
|
## Teams, projects, and invitations
|
||||||
|
|
||||||
- Team/project commands mix DB changes, email, message bus notifications, media/storage cleanup, feature flags, quotas, and audit metadata. Keep mutations transactional when the existing command does so.
|
- Team/project commands mix DB changes, email, message bus notifications, media/storage cleanup, feature flags, quotas, and audit metadata. Keep mutations transactional when the existing command does so.
|
||||||
- Invitation flows validate muted/bounced emails before sending and use tokenized invitation state. Accepting an invitation is tied to the invited member identity, not just possession of a token.
|
- Invitation flows validate muted/bounced emails before sending and use tokenized invitation state. Accepting an invitation is tied to the invited member identity, not just possession of a token.
|
||||||
- The whole invitation family is team-admin only: creating, updating and deleting invitations, minting the invitation token (`get-team-invitation-token`) and listing the pending ones (`get-team-invitations`, which carries the invited email and the offered role). The token of an invitation that grants the `owner` role is restricted to the team owner, because an admin cannot create nor promote an owner invitation (`cant-promote-to-owner`). This family is the exception to the rule above: it answers `:validation` / `:insufficient-permissions` instead of `:not-found`, so the five commands speak one language.
|
|
||||||
- With the `:admin-console` flag, the organization `send-invitations` policy (role value plus Nitrate `restrictions-met`, through `cto/allowed?`) gates every invitation command that hands out or manages access: create (and resend, which reuses it), `update-team-invitation-role`, `delete-team-invitation` and `get-team-invitation-token`. Only `get-team-invitations` stays on team admin alone, so a blocked admin still sees the list. The shared helper `check-send-invitations-policy` in `teams_invitations.clj` does a strict Nitrate lookup (fails closed; 404 means no organization) and runs outside a DB transaction, which is why the role and delete commands carry no `::db/transaction`.
|
|
||||||
- Logical deletion is used for many product objects; prefer existing logical-deletion helpers over hard deletes unless the command already performs permanent cleanup.
|
- Logical deletion is used for many product objects; prefer existing logical-deletion helpers over hard deletes unless the command already performs permanent cleanup.
|
||||||
- Bounced/spam-complaint emails can mute/block a profile for login/registration and email sending. Devenv Mailpit is the normal local path for registration/email-flow testing.
|
- Bounced/spam-complaint emails can mute/block a profile for login/registration and email sending. Devenv MailCatcher is the normal local path for registration/email-flow testing.
|
||||||
|
|
||||||
## Comments, webhooks, and audit
|
## Comments, webhooks, and audit
|
||||||
|
|
||||||
@ -51,4 +37,4 @@
|
|||||||
- Enable LDAP login locally with frontend flag `enable-login-with-ldap`; the devenv includes a configured test LDAP service.
|
- Enable LDAP login locally with frontend flag `enable-login-with-ldap`; the devenv includes a configured test LDAP service.
|
||||||
- OIDC testing requires external provider app credentials plus matching backend/frontend config.
|
- OIDC testing requires external provider app credentials plus matching backend/frontend config.
|
||||||
- Backend domain tests usually live under `backend/test/backend_tests/rpc/commands/*_test.clj` or nearby backend test namespaces. Use focused `clojure -M:dev:test --focus ...` from `backend/` when possible.
|
- Backend domain tests usually live under `backend/test/backend_tests/rpc/commands/*_test.clj` or nearby backend test namespaces. Use focused `clojure -M:dev:test --focus ...` from `backend/` when possible.
|
||||||
- For auth/session or HTTP behavior, combine backend tests with the HTTP/session notes in `mem:backend/subtleties` because RPC-level tests may not exercise cookie/header transforms.
|
- For auth/session or HTTP behavior, combine backend tests with the HTTP/session notes in `mem:backend/http-storage-filedata-subtleties` because RPC-level tests may not exercise cookie/header transforms.
|
||||||
@ -4,13 +4,10 @@ Backend: JVM Clojure; Integrant; PostgreSQL; Redis/Valkey; RPC; HTTP; storage; m
|
|||||||
|
|
||||||
## Focused memories
|
## Focused memories
|
||||||
|
|
||||||
- Cross-cutting backend subtleties (RPC, DB helpers, workers, cron, CLIMIT, config and HTTP/session middleware, http self-metrics, media processing, and file data persistence): `mem:backend/subtleties`
|
- RPC, DB helpers, workers, cron: `mem:backend/rpc-db-worker-subtleties`
|
||||||
- Storage abstraction, logical buckets, object lifecycle, deduplication, access, and garbage collection: `mem:backend/storage`.
|
- Storage abstraction, logical buckets, object lifecycle, deduplication, access, and garbage collection: `mem:backend/storage`.
|
||||||
- Embedded Ladybug graph experiment, projection, incremental sync, console, and risks: `mem:backend/graph-experiment`
|
- HTTP sessions, config, media processing, and file data persistence: `mem:backend/http-storage-filedata-subtleties`.
|
||||||
- Session lifetime config, token `:exp`, and idle/absolute GC: `mem:backend/session-expiration`.
|
|
||||||
- Auth flows, permission model, teams, projects, invitations, comments, webhooks, audit: `mem:backend/auth-permissions-product-domains`
|
- Auth flows, permission model, teams, projects, invitations, comments, webhooks, audit: `mem:backend/auth-permissions-product-domains`
|
||||||
- WebSocket subscriptions, the connection registry (local per backend, unlike the bus), and how a lost permission gets revoked: `mem:backend/websocket-subscriptions`.
|
|
||||||
- Audit-log event collection (RPC wrapper, frontend ingestion), telemetry duality, webhook fan-out, error reporters, Nexus archival and retention: `mem:audit-log` (guía completa, también para perfiles junior; el stub `mem:backend/audit-log` solo redirige aquí)
|
|
||||||
- Services, task-queue/Pub-Sub topology constraints -> `mem:prod-infra/core`.
|
- Services, task-queue/Pub-Sub topology constraints -> `mem:prod-infra/core`.
|
||||||
|
|
||||||
## Stable namespace map
|
## Stable namespace map
|
||||||
@ -25,7 +22,6 @@ Backend: JVM Clojure; Integrant; PostgreSQL; Redis/Valkey; RPC; HTTP; storage; m
|
|||||||
- `app.worker`: task execution/cron plumbing.
|
- `app.worker`: task execution/cron plumbing.
|
||||||
- `app.main`: Integrant system map and component wiring.
|
- `app.main`: Integrant system map and component wiring.
|
||||||
- `app.config`: `PENPOT_*` env config and feature flags.
|
- `app.config`: `PENPOT_*` env config and feature flags.
|
||||||
- `app.health`: per-process availability flag; `/readyz` answers 503 while offline. Flipped from `app.srepl.main` (`set-offline` / `set-online`).
|
|
||||||
- `app.srepl.*`: development REPL helpers for manual backend operations (data inspection, migration helpers, one-off admin tasks).
|
- `app.srepl.*`: development REPL helpers for manual backend operations (data inspection, migration helpers, one-off admin tasks).
|
||||||
- `app.nitrate`, `app.rpc.commands.nitrate`, and `app.rpc.management.nitrate`: external Nitrate subscription/organization integration, gated by the `:nitrate` feature flag and shared-key HTTP calls.
|
- `app.nitrate`, `app.rpc.commands.nitrate`, and `app.rpc.management.nitrate`: external Nitrate subscription/organization integration, gated by the `:nitrate` feature flag and shared-key HTTP calls.
|
||||||
|
|
||||||
@ -33,65 +29,27 @@ Backend: JVM Clojure; Integrant; PostgreSQL; Redis/Valkey; RPC; HTTP; storage; m
|
|||||||
|
|
||||||
RPC commands are defined with `app.util.services/defmethod` and schemas. Use `get-` prefixes for read operations. Command metadata usually includes auth, docs version, params schema, and result schema. Return plain maps/vectors or raise structured exceptions from `app.common.exceptions`.
|
RPC commands are defined with `app.util.services/defmethod` and schemas. Use `get-` prefixes for read operations. Command metadata usually includes auth, docs version, params schema, and result schema. Return plain maps/vectors or raise structured exceptions from `app.common.exceptions`.
|
||||||
|
|
||||||
Backend RPC command areas without focused memories include access tokens (auth rules, incl. MCP tokens, in `mem:backend/auth-permissions-product-domains`), binfile, demo, feedback, file snapshots, fonts, management, Nitrate, and webhooks beyond the notes in `mem:backend/auth-permissions-product-domains`; inspect nearby command tests and command metadata before changing them.
|
Backend RPC command areas without focused memories include access tokens, binfile, demo, feedback, file snapshots, fonts, management, Nitrate, and webhooks beyond the notes in `mem:backend/auth-permissions-product-domains`; inspect nearby command tests and command metadata before changing them.
|
||||||
|
|
||||||
## DB conventions
|
## DB conventions
|
||||||
|
|
||||||
`app.db` helpers accept cfg, pool, or conn in most places and convert kebab-case to snake_case:
|
`app.db` helpers accept cfg, pool, or conn in most places and convert kebab-case to snake_case:
|
||||||
- `db/get`, `db/get*`, `db/query`, `db/insert!`, `db/update!`, `db/delete!`.
|
- `db/get`, `db/get*`, `db/query`, `db/insert!`, `db/update!`, `db/delete!`.
|
||||||
- `db/get*`/`db/query` signature is `(ds table where-params & {:as opts})`: trailing keywords become an opts MAP (ignored by the select builder — use `{::db/remove-deleted false}` map as the single vararg to also see rows with `deleted_at` set), while a trailing ODD number of keywords crashes with "Don't know how to create ISeq from: Keyword". Extra keywords are NOT a column filter: rows always come back with all columns.
|
|
||||||
- Next.js/JDBC caveat: where-maps don't support value vectors (no `IN`); use `status = ANY(?)` with `db/create-array` inside `db/tx-run!`.
|
|
||||||
- Use `db/run!` for multiple operations on one connection.
|
- Use `db/run!` for multiple operations on one connection.
|
||||||
- Use `db/tx-run!` for transactions.
|
- Use `db/tx-run!` for transactions.
|
||||||
- `job`-substrate specifics (unified jobs): the only writers of a job status are `claim`, `retry-job`, `complete`, `fail` and `cancel` in `app.jobs`; each one is a conditional update plus its `job_event` row in the same transaction, so history can never contradict the row. A writer that affects no row writes no event and no metric. They never join the caller's transaction, because a lifecycle write reports on work that already happened: a rolled back caller cannot undo a transition, and an independent connection cannot see the caller's uncommitted job row, so a job must be committed before it can be claimed. GC deletes return resource ids via `RETURNING` inside one transaction.
|
|
||||||
- Every public function of `app.jobs` takes a cfg map. A bare pool or connection is a caller bug and is not coerced: the substrate has no conversion of its own, and the ones in `app.db` are the db layer's business.
|
|
||||||
- Every new foreign key of the substrate is `DEFERRABLE`, `job_event.job_id` included.
|
|
||||||
- Some tables carry a SQL deletion guard (`raise_deletion_protection` + a `deletion_protection__tgr` statement trigger): `profile`, `team`, `team_font_variant`, `file_media_object`, `file_thumbnail`, `file_tagged_object_thumbnail` and `job`. A `DELETE` raises unless the transaction runs `SET LOCAL rules.deletion_protection TO off` (the guard is on when the GUC is unset, empty or `on`); see `app.tasks.objects-gc`, `app.jobs.gc` and `th/db-force-delete`.
|
|
||||||
|
|
||||||
Database migrations live in `backend/src/app/migrations/`; pure SQL migrations are under `backend/src/app/migrations/sql/`. SQL filenames conventionally start with a sequence and verb/table description, e.g. `0026-mod-profile-table-add-is-active-field`. Applied migrations are tracked in the `migrations` table.
|
Database migrations live in `backend/src/app/migrations/`; pure SQL migrations are under `backend/src/app/migrations/sql/`. SQL filenames conventionally start with a sequence and verb/table description, e.g. `0026-mod-profile-table-add-is-active-field`. Applied migrations are tracked in the `migrations` table.
|
||||||
|
|
||||||
Migrations must stay within PostgreSQL 15 syntax: the official docker images (`docker/images/docker-compose.yaml`) pin `postgres:15`, so anything newer breaks those deployments (`SET STORAGE DEFAULT` is PostgreSQL 16, for instance). The documented floor is >=15.
|
For interactive PostgreSQL access with correct dev defaults, use `scripts/psql`; to dump
|
||||||
|
the current DDL schema, use `scripts/db-schema` (see `mem:scripts/psql`).
|
||||||
|
|
||||||
For interactive PostgreSQL access with correct dev defaults, use `scripts/psql`; to dump the current DDL schema, use `scripts/db-schema` (see `mem:scripts/psql`).
|
For deeper details on transaction semantics, advisory locks, Transit vs JSON helpers, and dev/test DB URLs: `mem:backend/rpc-db-worker-subtleties`.
|
||||||
|
|
||||||
For deeper details on transaction semantics, advisory locks, Transit vs JSON helpers, and dev/test DB URLs: `mem:backend/subtleties`.
|
## Background tasks
|
||||||
|
|
||||||
## Background tasks (unified jobs)
|
A task handler is an Integrant component with `ig/assert-key`, `ig/expand-key`, and `ig/init-key`, returning the function run by the worker. New tasks also need wiring in `app.main`: handler config, worker registry entry, and cron entry if scheduled.
|
||||||
|
|
||||||
Every background job is a job-def: a plain `(defn execute-X [cfg params] ...)` in its namespace + a malli params schema + an `ig/init-key` that returns the job-def map `{::jobs/name, ::jobs/schema, ::jobs/handler, ::jobs/decoder, ::jobs/validator}` (decoder/validator precompiled at init), optionally with `::jobs/result-schema` when the result carries types plain JSON does not keep (a uuid, whether a value or a map key): the substrate compiles the decoder from that schema on registry init and checks a result against the schema before storing it. The registry is the `::jobs/defs` wiring in `app.main`, which also populates a module-level registry used by `jobs/submit` as fallback — job-def components cannot ig/ref `::jobs/defs` (wiring cycle). Submit from RPC code passes its RPC cfg (it carries `::jobs/defs` via ig/ref).
|
|
||||||
|
|
||||||
- `::jobs/handler` is `(fn [context params] ...)`. It is NOT given the runner cfg: a job-def closes over its own dependencies, and handing it the runner cfg would let a handler reach the pool or any other component. A wrapper passes the context to its implementation only when the implementation takes it; most do not, and their implementation stays `[cfg params]`.
|
|
||||||
- The context is built by `jobs/make-context` and has exactly five keys: `id`, `name`, `label`, `resource-id`, `profile-id`, behind a closed schema. `profile-id` is the owner of a user-facing job and nil for an internal one, and it is there so a handler can revalidate permissions and check who owns its resource without a second query. `queue` is absent on purpose (a routing detail of the dispatcher, not a property of the job) and so are `retry-num`/`max-retries`/`attempt` (retry policy belongs to the runner, and an attempt number cannot be durable because the `noop` strategy re-runs without incrementing the counter).
|
|
||||||
- `jobs/invoke` is the in-process escape hatch: it delivers a nil context without a row, validates and delivers `::jobs/context` when given, and keeps `::jobs/job-id` independent. `::jobs/context` and `::jobs/job-id` are cfg options, not params.
|
|
||||||
- Business params live in the `params` column (plain JSON, decoded with the job-def decoder). `submit` takes optional `::jobs/profile-id`, `::jobs/resource-id` (both validated as UUIDs) and `::jobs/expires-at` (an instant: the retention the jobs GC sweeps by), all stored as columns; nothing is inferred from `params`. The binfile creation commands are the callers that pass a profile-id and a resource.
|
|
||||||
- A job-def may carry optional metadata keys read from the registry by its consumers, never by the substrate: `::jobs/family` (e.g. `:export`) groups the jobs of a family for quotes, `::jobs/resource-role` (`:input` or `:output`) says whether the job's `resource_id` is what it consumes or what it produces, and `::jobs/queue-name` names the queue the creation command routes the job to (`submit-job` falls back to `:binfile` when the def does not set it).
|
|
||||||
- Job resources (`app.jobs.storage`) are the objects a job owns through `job.resource_id`: they live in the `job-resource` bucket, are written with the profile of the job as owner (only that profile can read them back from `/assets/by-id`) and are never deduplicated. They carry no expiry of their own, because `expired-at` is stored as `deleted_at` and an object with it is unreadable from birth: the retention of the artifact is the retention of its job row. The writer marks the object as touched right away, so an artifact that no row references (a crash between the upload and the completion) is reclaimed by `storage-gc-touched`, and a row that does reference it freezes it until the jobs GC deletes the row and touches the object.
|
|
||||||
- The binfile jobs (`:export-binfile` and `:import-binfile`) run the core of `bf.v1`/`bf.v3` through `app.binfile.jobs`, which turns the `:progress` taps of the core into `progress` events of the job with `jobs/heartbeat` (the `modified_at` touch at most once per second, progress events at up to ~200ms). Each event is a self-contained milestone: a `:stage` keyword of one vocabulary shared by the two formats (the same work has a different name in v1 and v3) and an optional `:counters` map, with the counter of the activity named after its own stage and the outer scopes in course as context, which the tap carries itself (the reader puts `::outer-counters` in every tap inside them — today only `:files`, extendable to `:teams`/`:projects` for multi-scope exports — so the adapter remembers nothing between taps and a tap that happens outside any scope has no context); the ids and names the taps carry never reach the event. Every tap is reported, so a tap with no units to count still keeps the job alive, and a milestone the throttle drops loses nothing because the next one carries the same state. The adapter checks that the job is still active before the run, and the beat raises an `:interrupt` right there on the run thread when the job is no longer active, so the work stops at its next event point inside the transaction of its handler, which rolls back. A run whose last tap predates the interruption is discarded by the check once it returns. The adapter passes the job id from the handler context. Their job-def components carry pool, storage, metrics and msgbus, because a job with a `profile_id` cannot store an event without a msgbus on its cfg. Both readers answer with the same shape (`:file-ids` and, in version 3, the resolution of the libraries), because the version 1 reader returns a bare set of file ids.
|
|
||||||
- Both binfile handlers revalidate their permission at run time, not only when the job is created, because it can be revoked while the job waits: the export checks the read permission of every file before reading anything and stores the package in `job-resource` as the resource of the job (answering with the completion envelope), and the import checks the edition permission of the destination and runs the core inside a transaction, which is what makes a cancelled import roll back instead of leaving half of a file. The import releases its package (marks it as deleted and drops the reference from the job row in the same transaction) and deletes its temporary copy once the job is terminal, whatever the outcome. Both handlers check their own result against a schema before returning it (the import job-def declares that same schema as its result schema, which is what types the resolution when the row is read).
|
|
||||||
- The v3 and v1 imports only materialize a library relation when both ids belong to the package and the link would neither be a self link nor close a dependency cycle (the same `bfc/library-link-status` check that `link-file-to-library` uses to raise); relations are processed in sorted `(file_id, library_file_id)` order, which `get-files-rels` also returns, so the first edge wins deterministically, and every edge it ignores is logged with a warning. The v2 import stays out of that guard on purpose: it is only reachable from the developer REPL (`app.srepl.binfile`) and keeps inserting the relations as they come. Migration `0158-del-forbidden-file-library-rels` deletes the legacy cross-team and self rows from `file_library_rel` and their `file_library_sync` rows, sync rows first.
|
|
||||||
- A version 1 package lists its files in the metadata and the importer creates one file per entry. The v1 export (gone from the product; only tests and the developer REPL use it) unioned the requested files with their libraries without deduplicating, so a package exported from a file inside a library cycle lists that file twice. The importer reads the pair of objects of every entry to keep the stream in sync, imports the first occurrence only, warns about the repeats, and answers with one id per imported file.
|
|
||||||
- The domain cfg of a binfile request lives in `app.binfile.common` (`export-cfg`, `import-cfg`), so the legacy RPC and the jobs freeze the same ids, resolve the same team features and apply the same anti-zip-bomb limits. `resolve-export-type` is the translation the legacy RPC needs for its deprecated boolean flags; the creation command asks the caller for the type instead of guessing a default.
|
|
||||||
- Two retention knobs, easy to confuse: `:jobs-retention` sweeps terminal internal rows (the ones without a profile), while `:jobs-user-ttl` is the default expiry a user-facing job gets when it is created; the jobs GC deletes the row once `expires_at` passes and touches its resource. The heavy work of a user job runs on its own `:binfile` queue (`:worker-binfile-parallelism`, default 1) so it does not block the default runner.
|
|
||||||
- The quote of a family of jobs (`::export-jobs-per-profile`, `::import-jobs-per-profile`) counts the jobs of the profile that are still on their way (`new`/`scheduled`/`running`/`retry`) whose name the registry gives for that family, so a new job of the family is covered without touching the quote.
|
|
||||||
- A user-facing job is created through `app.rpc.commands.jobs`. One command per job type: `create-export-binfile-job` and `create-import-binfile-job` name the job they create and their body carries only `:params`, whose schema the command imports from the job namespace (`::sm/params`), so the RPC validates the business params up front, before the command runs; the job-def is still the place that owns them and validates them again at run time. The commands check the permission and the quote of the family before the expensive part, submit with `max-retries 0`, an expiry from `:jobs-user-ttl` and the queue the job-def routes to through `::jobs/queue-name` (`:binfile` when the def says nothing; read by the command, never by the substrate), and answer with a summary of the row. The import command has its own creation schema (`import-binfile/schema:create-params`, version optional) because the request is not the stored params: the command fills the version (from the caller or the package header) and the manifest metadata kept for audit. It assembles the upload, reads the manifest of a version 3 package for the audit (never its content) and stores the package in `job-resource` as the resource of the job, releasing it when the job never comes to own it.
|
|
||||||
- `get-job` is the read surface a profile has over its own jobs: it answers with the same summary the creation commands return plus the `result` of a finished job or its public error (through `decode-job-error`, so the keywords come back), and a job of another profile is not distinguishable from one that does not exist: both answer `not-found`. `decode-row` decodes `params`, `result` and `error`, and the result is then decoded with the decoder the substrate compiled from the result schema the job-def the row names declares (`decode-result`; a def without one, or a row whose job-def is gone, keeps the plain JSON the row holds: a uuid as the text it is on disk, a uuid used as a map key as a keyword). The event log still has no read surface: a client that follows a job takes its progress from the websocket.
|
|
||||||
- `heartbeat` uses named options `[cfg & {:keys [job-id progress] :as options}]` and returns the number of durable writes, so 0 means there was nothing to write: no job context, or the throttle did not allow it. A write that finds the job no longer active (terminal or gone) raises an `:interrupt` (`code :job-interrupted`, carrying the job id and the status it found) through the shared `jobs/check-active` instead of counting zero, so a task in course stops at its next beat; the interrupt is internal and never reaches HTTP (the management progress report answers `:skip` for it). The job id falls back to `::jobs/job-id` on the cfg and then to the runner-bound `*job-id*`. The `modified_at` touch is throttled to ~1s and the progress event to ~200ms, so the check fires at most once per second with no extra query (the read of the row happens only when a write already found nothing). A progress report is a milestone (`schema:progress`): a required `:stage` keyword plus an optional `:counters` map of `scope -> {:current, :total?}`, and no other key. Counters are non-negative, a `total` is positive and never lower than its `current`, and stage and counter keys are names of at most 64 characters. A milestone without counters is valid (a point of the run with no units to count), the stage vocabulary belongs to the worker and not to the substrate, and a client must degrade for a stage it does not know. Read a stored payload back with `jobs/decode-progress`: the database holds the keywords as strings. The 1s beat (down from 60s, so a cancelled heavy job is seen as gone at its next beat) costs one `modified_at` touch per beating job per second: cheap while the only heavy beater runs alone on its queue, revisit if concurrent beating jobs ever grow.
|
|
||||||
- `job.error` conforms to `app.jobs/schema:job-error` (`type` and `code` keywords, `hint` text, extra details allowed). Read it back with `jobs/decode-job-error`; a plain `db/decode-json-pgobject` leaves the keywords as strings. `complete` takes named options and an optional `resource-id`, which is only ever set, never replaced: it reads the row locked inside its own transaction and applies the rules in a fixed order, skipping a job that is no longer completable before refusing one for its resource, so the answer is a function of the row and not of the order two callers arrive in. It checks the result against the schema the job-def of the row declares before it writes (a violation raises `:invalid-result` and nothing is stored), and only for a transition that is going to write: a terminal job answers `:skip` without validating. `complete` is the single door of a result, so the check covers the runner and the external worker that completes a job through the management RPC (`::complete-job`), which stores a result the substrate would otherwise take blind.
|
|
||||||
|
|
||||||
For worker dispatch, cron, retry semantics (`ex/raise :type ::wrk/retry` with `:delay`/`:strategy`), deduplication, job events, and queue internals: `mem:backend/subtleties`.
|
|
||||||
|
|
||||||
## Jobs metrics
|
|
||||||
|
|
||||||
- `app.jobs.metrics` owns the jobs metric names, bounded labels and the periodic backlog sampler.
|
|
||||||
- Metrics use only `name`, `queue`, `outcome`, `reason`, `stage` and `kind` labels. Job IDs, profile IDs, params, reply bodies and exception text must never be labels. Every label goes as-is (lowercased where case varies): values come from fixed call sites, so a new value in Prometheus means new code, not noise. Only a missing value falls back (`other`, or `failed` for outcomes).
|
|
||||||
- The canonical job name is a keyword (`app.jobs/schema:job-name`): `submit` and `invoke` reject strings. The row stores text, so `jobs/get-job-def` still coerces (it is the runner's boundary), cron entries carry the job under `:job` with `d/name` only at the SQL/log boundary, and test helpers keyword-ize row names before invoking.
|
|
||||||
- The durable lifecycle metrics are submitted, dispatched, completed, retries, orphaned and rescheduled counters; queue-wait, execution and total-time histograms; dispatcher and backlog state; GC, cron and ephemeral request metrics.
|
|
||||||
- `app.metrics/run!` is safe for recording failures, but metrics components still require a valid metrics instance. Durable submit call sites carry `::mtx/metrics` through RPC or job-def configuration. The jobs substrate takes the **cfg**, not the instance, and resolves it with `app.metrics/instance`, which raises `:missing-metrics` when absent: `app.jobs`, `app.jobs.metrics`, the runner, the dispatcher, cron and the jobs GC all follow that. `app.http`, `app.rpc.climit` and `app.storage.s3.metrics` still take the instance and were left out of scope on purpose, so do not read the codebase as uniform. The six job writers resolve at the top of their body, before writing, because their metric lands in a `db/after-commit` callback that swallows exceptions. The GC, the dispatcher and cron resolve inside that callback instead, so their guarantee rests on the component schema instead. A job that calls `heartbeat` from a component cfg must carry `::mtx/metrics` on it, and the event helper records the counter from the cfg the transaction hands it.
|
|
||||||
- SQL-backed job metrics are registered with `app.db/after-commit` from INSIDE the transaction that changed the row, so a rollback cannot inflate submitted, terminal, retry, event or dispatcher counters. Every lifecycle writer owns that transaction (`app.jobs/own-transaction` drops `::db/conn`), and `db/transact!` gives a transaction that joins no other its own callback context, so the write and its metric commit or vanish together. `submit` is the exception and joins the caller on purpose: creating a job is part of the caller's business work.
|
|
||||||
- `penpot_jobs_events_total` counts stored job events with no labels: the interesting split is `kind`, and grouping a growing append-only table to build a metric is not worth the query on the write path.
|
|
||||||
- The backlog sampler is `::jobs-metrics/sampler`; it runs every 30 seconds on worker-enabled, writable systems, groups by status and publishes the oldest pending age, and does not start on a read-only database.
|
|
||||||
- `penpot_tasks_timing` remains exported for compatibility while the jobs-specific histograms are adopted.
|
|
||||||
|
|
||||||
|
For worker dispatch, cron, retry semantics, deduplication, and queue internals: `mem:backend/rpc-db-worker-subtleties`.
|
||||||
|
|
||||||
## REPL
|
## REPL
|
||||||
|
|
||||||
@ -128,7 +86,7 @@ Fixtures can populate local data for manual testing/perf work. From the backend
|
|||||||
|
|
||||||
* **Type Hinting:** Use explicit JVM type hints (e.g. `^String`, `^long`) in performance-critical paths to avoid reflection overhead.
|
* **Type Hinting:** Use explicit JVM type hints (e.g. `^String`, `^long`) in performance-critical paths to avoid reflection overhead.
|
||||||
* **Batch inserts:** Use `db/insert-many!` for bulk row inserts — generates a single SQL with multiple parameter tuples. Avoid on very large datasets (SQL length / parameter count limits).
|
* **Batch inserts:** Use `db/insert-many!` for bulk row inserts — generates a single SQL with multiple parameter tuples. Avoid on very large datasets (SQL length / parameter count limits).
|
||||||
* **Server-side cursors:** Use `db/plan` (fetch-size 1000, forward-only, read-only) for large result sets. Never fetch large collections into memory at once. `db/plan` is an `IReduceInit`: `reduce`/`transduce` over it (`sequence` does not work on it), and with `partition-all` the transducer's completing arity is what flushes the last partial chunk. A plan row is a **view over the open result set**, valid only inside the reducing function: read the columns you need right there — buffering rows to read them later (a `partition-all` buffer, a lazy seq) silently yields nil columns. `db/cursor` (manual `DECLARE`/`FETCH`) is a leftover with no callers left.
|
* **Server-side cursors:** Use `db/plan` (fetch-size 1000, forward-only, read-only) or `db/cursor` for large result sets. Never fetch large collections into memory at once.
|
||||||
* **Transaction discipline:** Use `tx-run!` for writes (opens a transaction), `run!` for reads (single connection, no transaction). Set `:read-only` on `tx-run!` when applicable to let PostgreSQL optimize.
|
* **Transaction discipline:** Use `tx-run!` for writes (opens a transaction), `run!` for reads (single connection, no transaction). Set `:read-only` on `tx-run!` when applicable to let PostgreSQL optimize.
|
||||||
|
|
||||||
## Lint and Format
|
## Lint and Format
|
||||||
@ -138,7 +96,9 @@ IMPORTANT: all CLI commands must be executed from the `backend/` subdirectory.
|
|||||||
* **Linting:** `pnpm run lint:clj`.
|
* **Linting:** `pnpm run lint:clj`.
|
||||||
* **Formatting:** `pnpm run check-fmt:clj` to check, `pnpm run fmt:clj` to fix. After running `fmt:clj`, `check-fmt:clj` is redundant. Avoid unrelated whitespace diffs.
|
* **Formatting:** `pnpm run check-fmt:clj` to check, `pnpm run fmt:clj` to fix. After running `fmt:clj`, `check-fmt:clj` is redundant. Avoid unrelated whitespace diffs.
|
||||||
|
|
||||||
**Before linting:** if delimiter errors are suspected (after LLM edits), run `scripts/paren-repair` on the affected files first. Delimiter errors produce misleading linter/compiler output. See `mem:scripts/paren-repair`.
|
**Before linting:** if delimiter errors are suspected (after LLM edits), run
|
||||||
|
`scripts/paren-repair` on the affected files first. Delimiter errors produce
|
||||||
|
misleading linter/compiler output. See `mem:scripts/paren-repair`.
|
||||||
|
|
||||||
## Testing
|
## Testing
|
||||||
|
|
||||||
|
|||||||
@ -1,633 +0,0 @@
|
|||||||
# Graph Experiment
|
|
||||||
|
|
||||||
## Scope
|
|
||||||
|
|
||||||
- Purpose: project Penpot file data into an embedded Ladybug graph database.
|
|
||||||
- Purpose: keep the graph current with Penpot file changes.
|
|
||||||
- Purpose: expose a read-only graph console for backend debugging.
|
|
||||||
- This is an experiment, not a replacement for PostgreSQL file storage.
|
|
||||||
- The graph subsystem is off unless `:graph` is in the backend flags.
|
|
||||||
- The main Penpot frontend has no graph feature code for this subsystem.
|
|
||||||
- The graph console is a backend-served HTML template with JavaScript.
|
|
||||||
|
|
||||||
## Memory Links
|
|
||||||
|
|
||||||
- Read `mem:backend/core` for backend architecture, HTTP routes, DB rules, and test commands.
|
|
||||||
- Read `mem:backend/subtleties` for RPC and message bus behavior, and for file data loading and realization.
|
|
||||||
- Read `mem:common/changes-architecture` for the change record vocabulary.
|
|
||||||
- Read `mem:frontend/routing-app-shell-subtleties` for the existing notification WebSocket.
|
|
||||||
- Read `mem:prod-infra/core` for Redis or Valkey message bus topology.
|
|
||||||
|
|
||||||
## Branch Surface
|
|
||||||
|
|
||||||
- The graph experiment adds about 6,336 lines and changes about 27 files.
|
|
||||||
- The graph implementation lives under `backend/src/app/graph/`.
|
|
||||||
- The graph console lives at `backend/resources/app/templates/graph-console.tmpl`.
|
|
||||||
- The existing debug page gains graph links in `backend/resources/app/templates/debug.tmpl`.
|
|
||||||
- The existing debug HTTP routes gain graph handlers in `backend/src/app/http/debug.clj`.
|
|
||||||
- The backend system passes the message bus to the debug route component in `backend/src/app/main.clj`.
|
|
||||||
- The backend adds Ladybug and Arrow dependencies in `backend/deps.edn`.
|
|
||||||
- The backend adds JVM options for Ladybug and Arrow native access.
|
|
||||||
- The common flag registry adds `:graph` in `common/src/app/common/flags.cljc`.
|
|
||||||
- The graph experiment adds `graph_sync_parity_test.clj` and `graph_binder_gate_test.clj`.
|
|
||||||
|
|
||||||
## System Model
|
|
||||||
|
|
||||||
### Storage layers
|
|
||||||
|
|
||||||
- PostgreSQL remains the source of truth for Penpot files.
|
|
||||||
- The graph database stores a projection of one file.
|
|
||||||
- A persistent graph uses a `.lbug` path under `PENPOT_GRAPH_DIR`.
|
|
||||||
- The default graph directory is `/tmp/penpot-graph`.
|
|
||||||
- A debug session uses a Ladybug `:memory:` database.
|
|
||||||
- A debug session database lives inside the backend JVM process.
|
|
||||||
- A debug session does not survive a backend restart.
|
|
||||||
- A debug session does not store file data back to PostgreSQL.
|
|
||||||
|
|
||||||
### Two graph update paths
|
|
||||||
|
|
||||||
- Cold projection reads the complete file and rebuilds the graph.
|
|
||||||
- Incremental sync reads file change records and updates the open graph.
|
|
||||||
- Both paths must produce the same graph for the same file state.
|
|
||||||
- The parity test treats cold projection as the reference path.
|
|
||||||
- A reload discards the session graph and uses cold projection again.
|
|
||||||
|
|
||||||
## Main Namespaces
|
|
||||||
|
|
||||||
### `app.graph.ladybug`
|
|
||||||
|
|
||||||
- Opens and closes Ladybug `Database` and `Connection` objects.
|
|
||||||
- Installs and loads the Ladybug JSON extension.
|
|
||||||
- Executes Cypher statements.
|
|
||||||
- Executes prepared statements.
|
|
||||||
- Binds scalar parameters.
|
|
||||||
- Formats UUID, string, integer, number, JSON, and timestamp values.
|
|
||||||
- Formats compound values such as arrays, maps, and structs.
|
|
||||||
- Converts Ladybug values back to Clojure values.
|
|
||||||
- Limits normal query results to 200 rows by default.
|
|
||||||
- Detects result truncation with `:truncated?`.
|
|
||||||
- Uses query timeout `0` by default.
|
|
||||||
- Query timeout `0` disables the timeout.
|
|
||||||
- Provides `validate-on-connection!` for parse, bind, and read-only checks.
|
|
||||||
- `exec-prepared-on-connection!` prepares every statement before the first execution.
|
|
||||||
- A prepare failure stops the batch before a mutation runs.
|
|
||||||
|
|
||||||
### `app.graph.schema`
|
|
||||||
|
|
||||||
- Provides the public schema facade.
|
|
||||||
- Exposes schema version `penpot-graph-slice-4`.
|
|
||||||
- Delegates node and relationship definitions to `app.graph.schema.nodes`.
|
|
||||||
|
|
||||||
### `app.graph.schema.nodes`
|
|
||||||
|
|
||||||
- Holds the single registry for graph node tables.
|
|
||||||
- Generates node DDL.
|
|
||||||
- Generates relationship DDL.
|
|
||||||
- Maps Penpot shape types to graph tables.
|
|
||||||
- Projects source attributes into graph attributes.
|
|
||||||
- Formats graph column values.
|
|
||||||
- Quotes reserved graph labels such as `Group` and `Boolean`.
|
|
||||||
- Defines container tables and shape tables.
|
|
||||||
- Defines `IsChildOf`, `IsInstanceOf`, `RefersTo`, and `FillsSwapSlot`.
|
|
||||||
|
|
||||||
### `app.graph.schema.contract`
|
|
||||||
|
|
||||||
- Records deliberate graph contract decisions.
|
|
||||||
- Renames graph columns such as `:revn` to `revision`.
|
|
||||||
- Drops attributes that do not belong in this graph slice.
|
|
||||||
- Records attributes that the graph does not project.
|
|
||||||
- Applies per-table dropped attributes.
|
|
||||||
- Defines type overrides for vectors, transforms, colors, maps, and JSON arrays.
|
|
||||||
- Maps selected map keys to the frontend JSON naming convention.
|
|
||||||
- `:background-blur` remains a declared unprojected attribute.
|
|
||||||
|
|
||||||
### `app.graph.schema.projection`
|
|
||||||
|
|
||||||
- Derives projected schemas from canonical Malli schemas.
|
|
||||||
- Builds the projected document schema.
|
|
||||||
- Builds projected shape schemas.
|
|
||||||
- Selects the schema for each shape type.
|
|
||||||
|
|
||||||
### `app.graph.schema.types`
|
|
||||||
|
|
||||||
- Maps Malli types to Ladybug types.
|
|
||||||
- Maps matrices to `DOUBLE[6]`.
|
|
||||||
- Maps points to `DOUBLE[2]`.
|
|
||||||
- Maps rectangles to `DOUBLE[4]`.
|
|
||||||
- Maps colors to `UINT32`.
|
|
||||||
- Maps collections to Ladybug arrays.
|
|
||||||
- Maps `:map-of` schemas to `MAP`.
|
|
||||||
- Maps closed scalar maps to `STRUCT`.
|
|
||||||
- Maps other complex values to `JSON`.
|
|
||||||
|
|
||||||
### `app.graph.schema.values`
|
|
||||||
|
|
||||||
- Coerces source values to graph column values.
|
|
||||||
- Writes fixed vectors with deterministic order.
|
|
||||||
- Packs colors into the graph color representation.
|
|
||||||
- Sorts set values when deterministic output is needed.
|
|
||||||
|
|
||||||
### `app.graph.arrow`
|
|
||||||
|
|
||||||
- Loads projection rows with Apache Arrow.
|
|
||||||
- Creates temporary staged node and relationship tables.
|
|
||||||
- Uses `COPY ... FROM (MATCH ...)` for bulk loading.
|
|
||||||
- Groups relationship loads by source and target table pair.
|
|
||||||
- Resolves relationship endpoints with joins.
|
|
||||||
- Does not use `createArrowRelTable` for UUID relationship endpoints.
|
|
||||||
- The Arrow `RootAllocator` nests outside the Ladybug connection and closes after the connection and database close sequence; Ladybug keeps staged buffers until connection close, so an inner allocator leaks.
|
|
||||||
- `with-allocator!` keeps the body exception when the allocator close also throws, attaching the close failure as suppressed.
|
|
||||||
|
|
||||||
### `app.graph.ingest`
|
|
||||||
|
|
||||||
- Fetches a complete file with `bfc/get-file` and `:realize? true`.
|
|
||||||
- Rejects missing files.
|
|
||||||
- Rejects files without file data.
|
|
||||||
- Can run file data validation before projection.
|
|
||||||
- Creates the DDL.
|
|
||||||
- Loads nodes and edges through Arrow.
|
|
||||||
- Executes post-load transforms.
|
|
||||||
- Writes graph metadata last.
|
|
||||||
- Treats the final metadata write as the complete-build marker.
|
|
||||||
- Supports a persistent database path and an open connection.
|
|
||||||
- `ingest-on-connection!` requires a caller-owned `:arrow-alloc` that outlives the connection; it raises `:missing-arrow-allocator` without one.
|
|
||||||
|
|
||||||
### `app.graph.projection.document`
|
|
||||||
|
|
||||||
- Projects `Document`, `Page`, `Component`, and supported shape nodes.
|
|
||||||
- Skips the page root frame.
|
|
||||||
- Creates `IsChildOf` edges from shapes to parents.
|
|
||||||
- Creates page edges to the document.
|
|
||||||
- Creates component edges to the document.
|
|
||||||
- Stores page order in `Page.index` and edge `position`.
|
|
||||||
- Reverses the stored `:shapes` list for Penpot z-order.
|
|
||||||
- Adds `page-id` to every projected shape.
|
|
||||||
- Propagates an instance head `component-id` to descendants.
|
|
||||||
- Stops component inheritance at a non-Frame shape with its own component ID.
|
|
||||||
- Skips deleted components during cold projection.
|
|
||||||
- Logs unsupported shape types and missing shape records.
|
|
||||||
|
|
||||||
### `app.graph.projection.transforms`
|
|
||||||
|
|
||||||
- Runs after the base nodes and edges load.
|
|
||||||
- `link-component-instances` creates `IsInstanceOf` edges.
|
|
||||||
- A Frame needs `component-file` to qualify as an instance head.
|
|
||||||
- `link-shape-refs` creates `RefersTo` edges from `shape-ref`.
|
|
||||||
- Ladybug limits multi-label relationship `MERGE` statements.
|
|
||||||
- The transform emits one statement for each shape-table pair.
|
|
||||||
- `link-swap-slots` creates `FillsSwapSlot` edges.
|
|
||||||
- Swap slot IDs come from `swap-slot-<uuid>` entries in `touched`.
|
|
||||||
- The transform removes swap slot entries from `touched` after edge creation.
|
|
||||||
- The transform order matters because it reads and then changes `touched`.
|
|
||||||
|
|
||||||
### `app.graph.meta`
|
|
||||||
|
|
||||||
- Stores graph provenance in `GraphMeta`.
|
|
||||||
- Stores schema version, source revision, producer, and build time.
|
|
||||||
- The source revision identifies the file revision used for cold projection.
|
|
||||||
|
|
||||||
### `app.graph.stats` and `app.graph.report`
|
|
||||||
|
|
||||||
- `app.graph.stats` counts graph nodes and relationships from the live catalog.
|
|
||||||
- `app.graph.report` prints ingest information for REPL use.
|
|
||||||
|
|
||||||
## Cold Projection Flow
|
|
||||||
|
|
||||||
1. Get the file row and realized file data from PostgreSQL.
|
|
||||||
2. Read the file revision from the file row.
|
|
||||||
3. Build the node and edge projection.
|
|
||||||
4. Create all graph tables from the graph schema.
|
|
||||||
5. Load node rows with Arrow.
|
|
||||||
6. Load relationship rows with Arrow.
|
|
||||||
7. Run `CHECKPOINT;`.
|
|
||||||
8. Run the registered derived transforms.
|
|
||||||
9. Write `GraphMeta` as the final build step.
|
|
||||||
10. Return file ID, file revision, database path, projection stats, and transform stats.
|
|
||||||
|
|
||||||
### Projection node groups
|
|
||||||
|
|
||||||
- `Document` contains file-level attributes without the file data blob.
|
|
||||||
- `Document.options` receives file-level options from the data blob.
|
|
||||||
- `Page` contains page attributes without the page object map.
|
|
||||||
- `Component` contains component attributes without component object maps.
|
|
||||||
- Shape tables contain the supported shape attributes.
|
|
||||||
- The graph stores selected derived attributes such as `page-id`.
|
|
||||||
|
|
||||||
### Projection relationship groups
|
|
||||||
|
|
||||||
- Structural edges use `IsChildOf`.
|
|
||||||
- Page and component edges point to `Document`.
|
|
||||||
- Derived edges come from the post-load transform registry.
|
|
||||||
|
|
||||||
## Incremental Sync
|
|
||||||
|
|
||||||
### Change source
|
|
||||||
|
|
||||||
- `app.rpc.commands.files-update` persists the file update first.
|
|
||||||
- The same command publishes a `:file-change` message to the file topic.
|
|
||||||
- The topic key is the file UUID.
|
|
||||||
- The message contains the file ID, profile ID, session ID, revision, version, and changes.
|
|
||||||
- Library changes also publish a team-topic message.
|
|
||||||
- The graph session only consumes the file-topic `:file-change` messages.
|
|
||||||
|
|
||||||
### Session subscription
|
|
||||||
|
|
||||||
- `app.graph.debug/start-sync-loop!` creates a channel with a dropping buffer of 64.
|
|
||||||
- The session subscribes the channel to the file UUID topic.
|
|
||||||
- The loop reads one message at a time.
|
|
||||||
- The loop ignores message types other than `:file-change`.
|
|
||||||
- The loop stops when the channel closes.
|
|
||||||
- `destroy-session!` closes the channel and purges its message bus subscription.
|
|
||||||
|
|
||||||
### Session state
|
|
||||||
|
|
||||||
- Sessions are stored in a global `defonce` atom.
|
|
||||||
- The map key is the string form of `profile-id`.
|
|
||||||
- One profile has one graph session.
|
|
||||||
- Loading another file first destroys the old session.
|
|
||||||
- A session stores the Ladybug database, connection, and Arrow allocator; the allocator closes after the connection and database.
|
|
||||||
- A session stores a shared lock for graph access.
|
|
||||||
- A session stores file metadata.
|
|
||||||
- A session stores the incremental sync index.
|
|
||||||
- A session stores the message bus channel.
|
|
||||||
- A session stores load time and profile ID.
|
|
||||||
- The session keeps projection statistics but drops full projection rows after index creation.
|
|
||||||
|
|
||||||
### Sync index
|
|
||||||
|
|
||||||
- `build-index` starts from the complete cold projection.
|
|
||||||
- The index stores the graph file ID and document ID.
|
|
||||||
- The index stores the current graph revision.
|
|
||||||
- The index stores page IDs, names, and positions.
|
|
||||||
- The index stores component IDs, names, and deleted state.
|
|
||||||
- The index stores shape table, parent, position, frame, page, and component context.
|
|
||||||
- The index stores child IDs by parent ID.
|
|
||||||
- The index supports later change application without another PostgreSQL file read.
|
|
||||||
|
|
||||||
### Change application
|
|
||||||
|
|
||||||
- `apply-changes!` processes the change list in source order.
|
|
||||||
- Each supported change returns a new index and a list of Cypher statements.
|
|
||||||
- Unsupported changes enter the `:skipped` result.
|
|
||||||
- Supported changes enter the `:applied` result.
|
|
||||||
- The function collects all statements before it executes them.
|
|
||||||
- The function appends a document revision statement when at least one change applies.
|
|
||||||
- The index revision advances only when at least one change applies.
|
|
||||||
- A larger incoming revision than the index revision creates a warning.
|
|
||||||
- A revision gap does not trigger catch-up.
|
|
||||||
|
|
||||||
### Shape change rules
|
|
||||||
|
|
||||||
- `:add-obj` reuses `projection.document/denormalized-shape`.
|
|
||||||
- `:add-obj` creates the shape node and its parent edge.
|
|
||||||
- `:mod-obj` applies supported `:set` operations to graph columns.
|
|
||||||
- `:mod-obj` keeps false and zero values as values.
|
|
||||||
- `:del-obj` deletes shapes in deep post-order.
|
|
||||||
- `:mov-objects` detaches shapes from the old parent.
|
|
||||||
- `:mov-objects` closes the old sibling position gap.
|
|
||||||
- `:mov-objects` inserts shapes at the new position.
|
|
||||||
- `:mov-objects` updates `parent_id` and `frame_id`.
|
|
||||||
- `:mov-objects` rewrites container `shapes` values.
|
|
||||||
- The parent columns and child lists must match a cold projection.
|
|
||||||
|
|
||||||
### Page and component change rules
|
|
||||||
|
|
||||||
- Page add creates a projected page node and a document edge.
|
|
||||||
- Page delete removes the page subtree.
|
|
||||||
- Page modification updates supported page attributes.
|
|
||||||
- Component add creates a component node and document edge.
|
|
||||||
- Component modification updates supported component attributes.
|
|
||||||
- Component delete uses a soft-delete state.
|
|
||||||
- Component restore removes the soft-delete state.
|
|
||||||
- Component purge removes the component node and document edge.
|
|
||||||
- Component sync paths need more parity coverage than the current tests provide.
|
|
||||||
|
|
||||||
## Session Locking
|
|
||||||
|
|
||||||
- The sync loop and HTTP handlers share one lock per session.
|
|
||||||
- The lock protects one Ladybug connection from concurrent access.
|
|
||||||
- Queries acquire the lock before binder validation and execution.
|
|
||||||
- Graph data export acquires the lock before catalog reads.
|
|
||||||
- Session export acquires the lock before `EXPORT DATABASE`.
|
|
||||||
- A long query blocks sync for the same session.
|
|
||||||
- A sync batch blocks queries for the same session.
|
|
||||||
- Ladybug connection thread safety is not assumed.
|
|
||||||
|
|
||||||
## Graph Query Rules
|
|
||||||
|
|
||||||
- The console accepts Cypher text.
|
|
||||||
- Blank query text raises a validation error.
|
|
||||||
- The query first passes Ladybug prepare and bind checks.
|
|
||||||
- The query must pass the engine read-only analysis.
|
|
||||||
- A mutating query is rejected.
|
|
||||||
- The graph console does not provide a write path.
|
|
||||||
- A session graph is rebuilt from the file by Reload.
|
|
||||||
- Normal query results have a 200-row limit.
|
|
||||||
- Query results use string values for the HTML console representation.
|
|
||||||
- JSON requests receive a Transit JSON response with the query and result.
|
|
||||||
- HTML requests receive the rendered console with the result.
|
|
||||||
|
|
||||||
## Graph Data Export
|
|
||||||
|
|
||||||
### G6 data
|
|
||||||
|
|
||||||
- `/dbg/actions/graph-data` reads the live Ladybug database.
|
|
||||||
- It does not read the sync index for nodes and edges.
|
|
||||||
- It therefore shows database drift if a batch fails after index update.
|
|
||||||
- Node export covers all registered node tables.
|
|
||||||
- Relationship export reads the Ladybug relationship catalog.
|
|
||||||
- Relationship export includes source, target, relationship name, and position.
|
|
||||||
- Node and relationship export uses a 100,000-row limit.
|
|
||||||
- The response reports `truncated` when a limit cuts the result.
|
|
||||||
- The response reports buffer-manager memory usage.
|
|
||||||
|
|
||||||
### `.lbug` export
|
|
||||||
|
|
||||||
- `source=file` rebuilds the persistent graph from PostgreSQL file data.
|
|
||||||
- `source=file` runs a synchronous full ingest for each request.
|
|
||||||
- `source=session` exports the caller profile's live in-memory graph.
|
|
||||||
- Session export uses Ladybug `EXPORT DATABASE` to Parquet files.
|
|
||||||
- Session export creates a new `.lbug` database with `IMPORT DATABASE`.
|
|
||||||
- The temporary Parquet staging directory is deleted after import.
|
|
||||||
- The final session `.lbug` file remains in the system temporary directory.
|
|
||||||
- The HTTP response streams the database file to the caller.
|
|
||||||
|
|
||||||
## HTTP Routes and Access
|
|
||||||
|
|
||||||
- The graph routes live in `backend/src/app/http/debug.clj`.
|
|
||||||
- The graph route list is added only when `:graph` is enabled.
|
|
||||||
- `/dbg/graph` serves the graph console page.
|
|
||||||
- `/dbg/actions/graph-files` returns the profile file tree.
|
|
||||||
- `/dbg/actions/graph-load` loads a file into the profile session.
|
|
||||||
- `/dbg/actions/graph-unload` closes the profile session.
|
|
||||||
- `/dbg/actions/graph-reload` rebuilds the loaded file graph.
|
|
||||||
- `/dbg/actions/graph-query` runs a read-only Cypher query.
|
|
||||||
- `/dbg/actions/graph-sync-status` returns the sync state.
|
|
||||||
- `/dbg/actions/graph-data` returns nodes and edges for G6.
|
|
||||||
- `/dbg/actions/graph-export` streams a `.lbug` database.
|
|
||||||
- The `/dbg` session middleware remains active.
|
|
||||||
- The `/dbg` admin middleware remains active.
|
|
||||||
- A devenv host with a profile ID passes the debug authorization rule.
|
|
||||||
- Other hosts need a profile email in the configured admin set.
|
|
||||||
- `/dbg/actions/graph-files` lists reachable teams, projects, and files.
|
|
||||||
- The file tree query has a 500-file limit.
|
|
||||||
- The graph handlers resolve graph namespaces at call time.
|
|
||||||
- The backend requires `app.graph.debug` and `app.graph.ingest` when the flag is on.
|
|
||||||
- Ladybug native loading then fails during route initialization instead of first use.
|
|
||||||
|
|
||||||
## Console Frontend
|
|
||||||
|
|
||||||
### Page type
|
|
||||||
|
|
||||||
- `graph-console.tmpl` is a backend resource template.
|
|
||||||
- It is not a Rumext component.
|
|
||||||
- It is not part of the main frontend route table.
|
|
||||||
- The page uses browser `fetch` calls and a browser WebSocket.
|
|
||||||
- The page loads G6 version `5.1.1` from jsDelivr.
|
|
||||||
|
|
||||||
### File tree
|
|
||||||
|
|
||||||
- The page fetches `/dbg/actions/graph-files`.
|
|
||||||
- The response contains team, project, and file groups.
|
|
||||||
- The page creates the tree with DOM APIs.
|
|
||||||
- A file click submits the graph load form.
|
|
||||||
- The page shows a message when no file exists.
|
|
||||||
|
|
||||||
### Graph rendering
|
|
||||||
|
|
||||||
- The page fetches `/dbg/actions/graph-data`.
|
|
||||||
- The page converts graph nodes and edges to G6 data.
|
|
||||||
- The page skips repaint when the node and edge signature does not change.
|
|
||||||
- The page marks added, removed, and changed graph entities.
|
|
||||||
- The page supports tree, dagre, circular, force, and combo layouts.
|
|
||||||
- The page supports collapsed container combos.
|
|
||||||
- The page has render guards at 4,000 nodes and 8,000 edges.
|
|
||||||
- The `?safe` query option bypasses the render guard.
|
|
||||||
- The page shows graph size by node count and relationship count.
|
|
||||||
- The page shows buffer-manager memory in MiB.
|
|
||||||
- The page reports a CDN failure when G6 is undefined.
|
|
||||||
|
|
||||||
### Query result filtering
|
|
||||||
|
|
||||||
- A query can return `filter_*` columns with node IDs.
|
|
||||||
- The HTML result table hides columns with the `filter_` prefix.
|
|
||||||
- The JSON result keeps the full result.
|
|
||||||
- The graph view uses the hidden IDs to select matching nodes.
|
|
||||||
- The graph view re-runs the query after graph refresh.
|
|
||||||
- This keeps the query filter aligned with the current graph.
|
|
||||||
- A user column named `filter_*` follows the same hiding rule.
|
|
||||||
|
|
||||||
### Node inspector
|
|
||||||
|
|
||||||
- A node click creates a query for that node.
|
|
||||||
- The inspector calls `/dbg/actions/graph-query` with JSON negotiation.
|
|
||||||
- The inspector displays the full projected row.
|
|
||||||
- The inspector uses table and ID values from the graph data.
|
|
||||||
|
|
||||||
## WebSocket Data Flow
|
|
||||||
|
|
||||||
1. The page opens `/ws/notifications` with a random `session-id` query value.
|
|
||||||
2. The page sends `:subscribe-file` with a Transit UUID value.
|
|
||||||
3. The server makes sure that the file exists and that the profile has read permission.
|
|
||||||
4. The server subscribes the connection to the file topic.
|
|
||||||
5. `files_update` publishes `:file-change` to the same topic.
|
|
||||||
6. The graph session consumes the message from its message bus subscription.
|
|
||||||
7. The WebSocket server sends the message to the browser connection.
|
|
||||||
8. The browser adds the change to the changelog.
|
|
||||||
9. The browser fetches sync status after 150 milliseconds.
|
|
||||||
10. The browser fetches graph data after a 400-millisecond debounce.
|
|
||||||
11. The browser repaints the G6 graph when the graph data changes.
|
|
||||||
|
|
||||||
### WebSocket reconnect behavior
|
|
||||||
|
|
||||||
- The page reconnects after three seconds when the socket closes.
|
|
||||||
- The page resubscribes to the file after the socket opens.
|
|
||||||
- The page refreshes sync status after reconnect.
|
|
||||||
- The page refreshes graph data after reconnect.
|
|
||||||
- Reconnect does not recover dropped message-bus changes.
|
|
||||||
- The page shows the sync error or skipped-change state when the status reports it.
|
|
||||||
|
|
||||||
## Feature Flag and Runtime Dependencies
|
|
||||||
|
|
||||||
- `:graph` is defined in `common/src/app/common/flags.cljc`.
|
|
||||||
- The flag is off by default.
|
|
||||||
- `com.ladybugdb/lbug` version `0.20.4` is a backend dependency.
|
|
||||||
- `org.apache.arrow/arrow-memory-netty` version `19.0.0` supports Arrow `RootAllocator`.
|
|
||||||
- The JVM uses `--enable-native-access=ALL-UNNAMED`.
|
|
||||||
- The JVM uses `--add-opens=java.base/java.nio=ALL-UNNAMED`.
|
|
||||||
- The JVM uses `--sun-misc-unsafe-memory-access=allow`.
|
|
||||||
- The JVM options appear in the development alias and backend launch scripts.
|
|
||||||
- A Ladybug version change needs new binder and parity tests.
|
|
||||||
- A JDK version change needs a startup test with the graph flag enabled.
|
|
||||||
|
|
||||||
## Tests
|
|
||||||
|
|
||||||
### `backend-tests.graph-sync-parity-test`
|
|
||||||
|
|
||||||
- Uses two Ladybug `:memory:` databases under one Arrow allocator outer to both; an allocator closed before its connections leaks.
|
|
||||||
- Does not use PostgreSQL or a live graph session.
|
|
||||||
- Projects initial file data into database A.
|
|
||||||
- Applies changes to database A through incremental sync.
|
|
||||||
- Applies the same changes to file data.
|
|
||||||
- Projects the changed file data into database B.
|
|
||||||
- Compares every node row and relationship row.
|
|
||||||
- Reports differences by table, row key, and column.
|
|
||||||
- Covers shape add, shape modification, shape deletion, movement, and page changes.
|
|
||||||
- Contains a test that injects a sync defect and expects a graph difference.
|
|
||||||
- Contains a test that `ingest-on-connection!` without `:arrow-alloc` raises `:missing-arrow-allocator`.
|
|
||||||
- Does not cover all component change variants.
|
|
||||||
- Does not cover every movement insertion mode.
|
|
||||||
|
|
||||||
### `backend-tests.graph-binder-gate-test`
|
|
||||||
|
|
||||||
- Creates the live graph DDL in a Ladybug `:memory:` database.
|
|
||||||
- Prepares each sync statement template without executing it.
|
|
||||||
- Detects parse errors and missing tables.
|
|
||||||
- Detects missing columns and bad label quoting.
|
|
||||||
- Reports the expected read-only classification.
|
|
||||||
- Covers reserved node labels across the node registry.
|
|
||||||
- Reports an error result for an invalid statement.
|
|
||||||
|
|
||||||
### Test gaps
|
|
||||||
|
|
||||||
- No automated HTTP handler tests cover graph routes.
|
|
||||||
- No automated session lifecycle tests cover load and unload.
|
|
||||||
- No automated WebSocket tests cover graph subscription.
|
|
||||||
- No automated export tests cover persistent and session sources.
|
|
||||||
- Component add, modify, delete, restore, and purge need parity tests.
|
|
||||||
- Page delete needs parity coverage.
|
|
||||||
- Movement with `:after-shape` needs parity coverage.
|
|
||||||
- Buffer overflow and revision gap behavior need tests.
|
|
||||||
- Partial batch failure and recovery need tests.
|
|
||||||
- Query timeout and long-query behavior need tests.
|
|
||||||
|
|
||||||
## Known Risks and Limits
|
|
||||||
|
|
||||||
### Dropped changes
|
|
||||||
|
|
||||||
- The sync channel uses a dropping buffer of 64.
|
|
||||||
- A burst can discard file-change messages.
|
|
||||||
- The sync loop logs a revision gap when it sees a larger revision.
|
|
||||||
- The sync loop does not fetch missing rows from `file_change`.
|
|
||||||
- Reload is the only built-in recovery path.
|
|
||||||
|
|
||||||
### Partial batch state
|
|
||||||
|
|
||||||
- `apply-changes!` does not provide Ladybug transaction atomicity.
|
|
||||||
- A statement failure can leave a partly changed graph.
|
|
||||||
- The in-memory index can advance before the database state is complete.
|
|
||||||
- `/dbg/actions/graph-data` reads the database and exposes this drift.
|
|
||||||
- Reload rebuilds the graph from PostgreSQL file data.
|
|
||||||
|
|
||||||
### Query resource use
|
|
||||||
|
|
||||||
- The default session query timeout is zero.
|
|
||||||
- A costly query can hold the session lock for a long time.
|
|
||||||
- The same lock blocks incremental sync.
|
|
||||||
- The graph export also holds the same lock during catalog reads.
|
|
||||||
- The graph schema has a high memory floor.
|
|
||||||
- The console reports about 115 MiB for the wide slice before file data.
|
|
||||||
|
|
||||||
### Session lifecycle
|
|
||||||
|
|
||||||
- Sessions have no TTL.
|
|
||||||
- Sessions remain until unload, replacement, or process shutdown.
|
|
||||||
- Each session owns native Ladybug memory and its Arrow allocator.
|
|
||||||
- Many profiles can create many native databases.
|
|
||||||
- A profile load replaces its previous session.
|
|
||||||
- Two browser tabs for one profile share one graph session.
|
|
||||||
|
|
||||||
### Temporary files
|
|
||||||
|
|
||||||
- Session export leaves the final `.lbug` file in the system temporary directory.
|
|
||||||
- Long-lived servers can accumulate exported session databases.
|
|
||||||
- The staging directory is deleted after import.
|
|
||||||
|
|
||||||
### Browser dependency
|
|
||||||
|
|
||||||
- The graph view depends on a runtime CDN request.
|
|
||||||
- A network restriction can remove the G6 view.
|
|
||||||
- Queries and session status still use backend endpoints without G6.
|
|
||||||
|
|
||||||
### Data exposure
|
|
||||||
|
|
||||||
- The graph console can list many files available to the profile.
|
|
||||||
- The console can load complete projected file data.
|
|
||||||
- The console can export a graph database.
|
|
||||||
- The console can inspect all projected node attributes.
|
|
||||||
- The console is safe only when the `/dbg` access boundary is correct.
|
|
||||||
- The graph flag must remain off for deployments that do not need this tool.
|
|
||||||
|
|
||||||
### Contract drift
|
|
||||||
|
|
||||||
- The graph schema is a deliberate slice of the Penpot file model.
|
|
||||||
- New source attributes do not enter the graph automatically in all cases.
|
|
||||||
- Dropped and unprojected attributes need an explicit contract decision.
|
|
||||||
- `applied_tokens` key mapping depends on the JSON naming function.
|
|
||||||
- `filter_*` is a frontend convention, not a graph schema guarantee.
|
|
||||||
|
|
||||||
### Ladybug dialect coupling
|
|
||||||
|
|
||||||
- Cypher strings contain Ladybug-specific syntax.
|
|
||||||
- Label quoting handles reserved labels explicitly.
|
|
||||||
- Relationship transforms depend on Ladybug relationship limits.
|
|
||||||
- Arrow loading depends on Ladybug `COPY FROM (MATCH ...)` behavior.
|
|
||||||
- A dependency upgrade needs schema, binder, Arrow, and parity checks.
|
|
||||||
|
|
||||||
## REPL Helpers
|
|
||||||
|
|
||||||
- `app.srepl.main` resolves graph functions only when a helper runs.
|
|
||||||
- `graph-smoke-test!` runs a basic Ladybug operation.
|
|
||||||
- `graph-query-test!` runs a graph query test.
|
|
||||||
- `ingest-file-to-graph!` projects a file into a graph database.
|
|
||||||
- These helpers use `requiring-resolve` to keep the graph dependency lazy.
|
|
||||||
|
|
||||||
## Operational Invariants
|
|
||||||
|
|
||||||
- PostgreSQL file data remains authoritative.
|
|
||||||
- Cold projection and incremental sync must produce equal graph state.
|
|
||||||
- The graph revision must identify the last applied file revision.
|
|
||||||
- The document revision must update when a sync batch applies.
|
|
||||||
- A missing or skipped change must remain visible in sync status.
|
|
||||||
- A graph query from the console must be read-only.
|
|
||||||
- A graph session must serialize connection access.
|
|
||||||
- Graph routes must remain behind the `:graph` flag and `/dbg` access control.
|
|
||||||
- The Arrow allocator nests outside the Ladybug connection and must outlive all Ladybug operations that use its buffers; a session owns its allocator until unload.
|
|
||||||
- `GraphMeta` must be written after the full ingest and transforms finish.
|
|
||||||
|
|
||||||
## Key Files
|
|
||||||
|
|
||||||
- `backend/src/app/graph/ladybug.clj`: Ladybug API and query gates.
|
|
||||||
- `backend/src/app/graph/arrow.clj`: Arrow bulk load.
|
|
||||||
- `backend/src/app/graph/ingest.clj`: Complete file ingest.
|
|
||||||
- `backend/src/app/graph/debug.clj`: Session lifecycle, sync loop, query, and export.
|
|
||||||
- `backend/src/app/graph/sync.clj`: Incremental change application.
|
|
||||||
- `backend/src/app/graph/meta.clj`: Graph provenance.
|
|
||||||
- `backend/src/app/graph/stats.clj`: Graph counts.
|
|
||||||
- `backend/src/app/graph/report.clj`: REPL ingest report.
|
|
||||||
- `backend/src/app/graph/projection/document.clj`: Base document projection.
|
|
||||||
- `backend/src/app/graph/projection/transforms.clj`: Derived relationship transforms.
|
|
||||||
- `backend/src/app/graph/schema/nodes.clj`: Node and relationship registry.
|
|
||||||
- `backend/src/app/graph/schema/contract.clj`: Projection contract decisions.
|
|
||||||
- `backend/src/app/graph/schema/projection.clj`: Malli projection schemas.
|
|
||||||
- `backend/src/app/graph/schema/types.clj`: Malli-to-Ladybug type mapping.
|
|
||||||
- `backend/src/app/graph/schema/values.clj`: Value coercion.
|
|
||||||
- `backend/src/app/http/debug.clj`: Graph route registration and handlers.
|
|
||||||
- `backend/src/app/http/websocket.clj`: File WebSocket subscription handlers.
|
|
||||||
- `backend/src/app/rpc/commands/files_update.clj`: File-change publication.
|
|
||||||
- `backend/src/app/main.clj`: Integrant message bus wiring.
|
|
||||||
- `backend/resources/app/templates/graph-console.tmpl`: Graph console browser code.
|
|
||||||
- `backend/resources/app/templates/debug.tmpl`: Debug page graph links.
|
|
||||||
- `common/src/app/common/flags.cljc`: `:graph` feature flag.
|
|
||||||
- `backend/test/backend_tests/graph_sync_parity_test.clj`: Cold versus sync parity.
|
|
||||||
- `backend/test/backend_tests/graph_binder_gate_test.clj`: Cypher binder gate.
|
|
||||||
|
|
||||||
## Development Commands
|
|
||||||
|
|
||||||
- Run backend commands from the `backend/` directory.
|
|
||||||
- Run focused parity tests with `clojure -M:dev:test --focus backend-tests.graph-sync-parity-test`.
|
|
||||||
- Run focused binder tests with `clojure -M:dev:test --focus backend-tests.graph-binder-gate-test`.
|
|
||||||
- Run the backend test suite with `clojure -M:dev:test`.
|
|
||||||
- Examine Clojure formatting with `pnpm run check-fmt:clj`.
|
|
||||||
- Run backend Clojure lint with `pnpm run lint:clj`.
|
|
||||||
- Write test output to a file before reading or filtering it.
|
|
||||||
28
.serena/memories/backend/http-storage-filedata-subtleties.md
Normal file
28
.serena/memories/backend/http-storage-filedata-subtleties.md
Normal file
@ -0,0 +1,28 @@
|
|||||||
|
# Backend HTTP, Storage, Media, and File Data Subtleties
|
||||||
|
|
||||||
|
## Config and HTTP/session middleware
|
||||||
|
|
||||||
|
- `app.config/config` and `flags` are dynamic `defonce` vars populated from `PENPOT_*` env vars through the shared schema string transformer. Tests and tooling can bind them.
|
||||||
|
- `parse-flags` automatically adds `:disable-secure-session-cookies` when `public-uri` is plain HTTP and not localhost. This changes cookie defaults without an explicit env flag.
|
||||||
|
- The backend sets Clojure `*assert*` globally from the `:backend-asserts` feature flag. Assertion-dependent checks can therefore differ by runtime flags.
|
||||||
|
- Request body parsing is mostly POST-oriented and supports Transit JSON plus plain JSON. Plain JSON request keys are kebab-decoded before being merged into `:params`.
|
||||||
|
- Response formatting negotiates with `Accept` or `_fmt=json`. Transit is the default for collection/boolean bodies; JSON encoding has special pointer-map handling.
|
||||||
|
- Auth prefers the session cookie token before the `Authorization` header. Headers may be `Token` or `Bearer`; JWTs with `kid=1` and `ver=1` are decoded as v1 session tokens, otherwise they are treated as legacy tokens.
|
||||||
|
- Shared-key auth requires `x-shared-key` as `<key-id> <key>` and stores the lowercased key id on the request. If no shared keys are configured it always rejects.
|
||||||
|
- Session management uses DB storage unless the DB pool is read-only, then falls back to the in-memory manager. DB sessions support both legacy string ids and v2 UUID session ids.
|
||||||
|
- Session cookies are renewed when using a legacy string id or when `modified-at` is older than the renewal interval. SameSite is `none` for CORS, otherwise strict/lax based on config.
|
||||||
|
|
||||||
|
## Storage and media
|
||||||
|
|
||||||
|
- Storage abstraction, backend configuration, logical buckets, object lifecycle, deduplication, access rules, and garbage collection: `mem:backend/storage`.
|
||||||
|
- SVG validation strips DOCTYPE and uses secure SAX parsing. Basic SVG info falls back to 100x100 dimensions when width/height/viewBox are missing.
|
||||||
|
- Raster metadata is shell-derived with ImageMagick `identify`, verifies detected MIME against the supplied MIME, and swaps dimensions for EXIF orientations 6/8.
|
||||||
|
- Remote image download requires 2xx status, `content-length`, a known MIME, and size under the configured maximum before writing the temp file; mismatched byte count is an internal error.
|
||||||
|
- Font processing shells out to FontForge and WOFF conversion tools and can derive TTF/OTF/WOFF variants from uploaded fonts.
|
||||||
|
|
||||||
|
## File data persistence
|
||||||
|
|
||||||
|
- File data backends are `legacy-db`, `db`, and `storage`. The storage backend keeps encoded file data in storage bucket `file-data`; the DB row stores metadata with `storage-ref-id` and nil data.
|
||||||
|
- `fdata/upsert!` touches any storage object referenced by incoming metadata before storing the new row/blob.
|
||||||
|
- Pointer-map fragments are persisted separately as type `fragment`, and only modified pointer maps are written.
|
||||||
|
- `fdata/realize` combines pointer realization and object-map realization. Use it before operations that need complete in-memory file data instead of pointer placeholders.
|
||||||
26
.serena/memories/backend/rpc-db-worker-subtleties.md
Normal file
26
.serena/memories/backend/rpc-db-worker-subtleties.md
Normal file
@ -0,0 +1,26 @@
|
|||||||
|
# Backend RPC/DB/Worker Subtleties
|
||||||
|
|
||||||
|
## RPC exposure and wrappers
|
||||||
|
|
||||||
|
- RPC commands are discovered from vars created by `app.util.services/defmethod`; adding a command namespace is not enough unless `backend/src/app/rpc.clj` includes it in `resolve-methods`.
|
||||||
|
- `GET`/`HEAD` RPC calls are only allowed for method names starting with `get-`. Other methods are method-not-allowed even if they are read-only internally.
|
||||||
|
- RPC auth defaults to enabled. Public endpoints must set `::auth false` metadata explicitly.
|
||||||
|
- The wrapper stack does auth before params validation, then auditing/rate/concurrency/metrics/retry/condition handling, with DB transaction handling inside that stack. `::db/transaction` metadata controls transaction wrapping.
|
||||||
|
- Params with `::sm/params` are decoded/conformed through the JSON transformer and successful IObj results get `:encode/json` metadata. Legacy spec conforming only applies when no Malli params schema exists.
|
||||||
|
- Nil RPC bodies become HTTP 204 unless explicit status metadata is present. Stream bodies default to `application/octet-stream` when no content type is set.
|
||||||
|
|
||||||
|
## DB helpers
|
||||||
|
|
||||||
|
- Most `app.db` helpers accept a pool, connection, or map containing `::db/pool` / `::db/conn`; preserve that convention in shared code.
|
||||||
|
- `db/tx-run!` uses `next.jdbc.transaction/*nested-tx* :ignore`: nested transaction calls reuse the outer transaction, not a savepoint. Use explicit savepoints when nested rollback semantics matter.
|
||||||
|
- `db/run!` opens/reuses one connection but does not create a transaction.
|
||||||
|
- `db/tjson` is Transit JSON for jsonb storage; `db/json` is plain JSON. Worker task props use Transit and are decoded with `decode-transit-pgobject`.
|
||||||
|
- Advisory transaction locks accept UUIDs or ints. UUID locks are hashed using a zero-UUID seeded siphash.
|
||||||
|
|
||||||
|
## Workers and cron
|
||||||
|
|
||||||
|
- Task queues are tenant-prefixed. Submit dedupe only removes not-yet-due `new` tasks with the same name/queue/label; it does not dedupe due, scheduled, retry, running, or completed work.
|
||||||
|
- The dispatcher selects `new`/`retry` tasks with `FOR UPDATE SKIP LOCKED`, marks them `scheduled`, and publishes Redis payload `[id scheduled-at]`. The runner skips Redis messages whose scheduled timestamp no longer matches DB state.
|
||||||
|
- Lost `scheduled` tasks are rescheduled after about 5 minutes; `running` tasks older than about 24 hours are marked failed as orphans.
|
||||||
|
- A task handler that is missing or returns an invalid result currently defaults to completed after warning. Throwing with `ex-data :type ::retry` controls retry behavior; `:strategy ::noop` retries without incrementing retry count.
|
||||||
|
- Cron jobs lock their `scheduled_task` row with `FOR UPDATE SKIP LOCKED`, disable statement/idle-in-transaction timeouts locally, and reschedule themselves in `finally` unless interrupted. Worker, dispatcher, and cron components do not start when the DB pool is read-only.
|
||||||
@ -1,41 +0,0 @@
|
|||||||
# Backend Session Expiration
|
|
||||||
|
|
||||||
## Config
|
|
||||||
|
|
||||||
- `:auth-token-cookie-name` / `PENPOT_AUTH_TOKEN_COOKIE_NAME` (default `auth-token`): cookie name.
|
|
||||||
- `:auth-token-cookie-max-age` / `PENPOT_AUTH_TOKEN_COOKIE_MAX_AGE` (default 7d): idle window; drives the sliding cookie `Expires` and the GC idle threshold.
|
|
||||||
- `:auth-token-cookie-max-age-absolute` / `PENPOT_AUTH_TOKEN_COOKIE_MAX_AGE_ABSOLUTE` (default 30d): hard cap from `created-at`; drives the token `:exp` and the GC absolute threshold.
|
|
||||||
- Durations decode as `<n><unit>` with hour/minute/second units (`168h`, `30m`); day units like `7d` are rejected by the `Duration/parse` string path, use hours (`168h`, `720h`).
|
|
||||||
- Session lifetime defaults live only as code constants in `session.clj` (`default-cookie-max-age`, `default-cookie-max-age-absolute`); do not duplicate them in the `config.clj` default map. Call sites always pass the constant as the `cf/get` fallback.
|
|
||||||
|
|
||||||
## Token and session model
|
|
||||||
|
|
||||||
- Sessions live only in `http_session_v2`. Legacy v1 support (`http_session`, string ids, `:ver 0` tokens) was removed and the table dropped (`0155-drop-http-session-table`).
|
|
||||||
- `assign-token` emits header `{:kid 1 :ver 1}` with claims `:sid`, `:iat` (= `modified-at`) and `:exp` = `created-at + absolute-max-age` (omitted only when `created-at` is nil, which neither manager produces today; the branch is defensive).
|
|
||||||
- `:exp` is anchored to `created-at`, never `modified-at`, so renewal cannot extend the absolute maximum.
|
|
||||||
- Tokens issued before `:exp` existed carry no `:exp`; they are still bounded by the GC's `created_at` condition and acquire `:exp` on their next renewal (self-healing, no operator action).
|
|
||||||
- `wrap-authz` resolves the session by `(:sid claims)` only; there is no fallback to reading a session by the raw token string.
|
|
||||||
- `middleware/wrap-auth` attaches `::http/auth-data` only for `kid=1`/`ver=1` tokens with a configured decoder; anything else stays unauthenticated.
|
|
||||||
- Renewal fires when `modified-at` is older than 6h (`default-renewal-max-age`, not configurable). It `UPDATE`s the same row (`modified-at` only) and issues a new token that keeps the original `:exp`.
|
|
||||||
- `read-session` does not check age. Idle expiration is enforced on every request by `wrap-authz` (sessions past the idle window authenticate like missing ones and are never renewed); the GC only deletes the rows afterwards.
|
|
||||||
|
|
||||||
## Revocation paths
|
|
||||||
|
|
||||||
- Profile deletion → `session/invalidate-all` (all sessions of the profile die).
|
|
||||||
- Authenticated password change → `session/invalidate-others` (every session except the caller's dies).
|
|
||||||
- Password recovery (`recover-profile`, GHSA-3j2p-qr45-q365) → `session/invalidate-all` inside the same transaction as the password `UPDATE`; there is no current session to preserve, and a captured session must not survive recovery. Sibling of GHSA-mj9f-5cwq-7p3q (logout deletes the current session row).
|
|
||||||
|
|
||||||
## GC (`::tasks/gc`, cron `session-gc`, daily)
|
|
||||||
|
|
||||||
- Deletes from `http_session_v2` where `modified_at < now - :auth-token-cookie-max-age` **or** `created_at < now - :auth-token-cookie-max-age-absolute`.
|
|
||||||
- The two thresholds are separate task params (`::tasks/max-age`, `::tasks/max-age-absolute`) built in `ig/expand-key`; both must be durations.
|
|
||||||
- Do not collapse the two conditions: `modified_at` alone never collects active sessions; `created_at` alone logs out active users at the idle window.
|
|
||||||
|
|
||||||
## Related lifetimes
|
|
||||||
|
|
||||||
- Organization SSO entries in session `:props` (`:sso {org-id exp}`) last 4h, hardcoded in `app.auth.oidc`.
|
|
||||||
- Access tokens (`app.rpc.commands.access-token`) have a user-chosen `:expires-at` (Never/30/60/90/180d), independent from HTTP sessions.
|
|
||||||
- Cookie `SameSite`/`secure` depend on `:cors`, `:strict-session-cookies`, `:secure-session-cookies`; `parse-flags` auto-adds `:disable-secure-session-cookies` for non-localhost HTTP `public-uri`.
|
|
||||||
- Without `PENPOT_SECRET_KEY`, derived subsystem keys change on restart and all sessions/invitations are invalidated.
|
|
||||||
- Read-only DB pool: sessions fall back to `inmemory-manager`, lost on restart, no GC.
|
|
||||||
- HTTP middleware and cookie/header precedence details: `mem:backend/subtleties`.
|
|
||||||
@ -24,61 +24,20 @@
|
|||||||
- `get-object` excludes rows with `deleted_at`.
|
- `get-object` excludes rows with `deleted_at`.
|
||||||
- Existing object values can remain readable until physical deletion.
|
- Existing object values can remain readable until physical deletion.
|
||||||
- `:expired-at` blocks reads after the expiration time.
|
- `:expired-at` blocks reads after the expiration time.
|
||||||
- `del-object!` sets `deleted_at` on live rows only (`deleted_at IS NULL`): a repeated call returns `false`. It does not remove backend content.
|
- `del-object!` sets `deleted_at`. It does not remove backend content.
|
||||||
- `storage-gc-deleted` removes the database row and backend content after the deletion delay, processing the pending objects in chunks of 25 with keyset pagination (`(deleted_at, id)` cursor, matching the `storage_object(deleted_at, id)` index). Only `status = 'valid'` objects are read: pending ones (still being written, see the status lifecycle) stay untouched even when already marked as deleted. Each chunk is one short read-write transaction: `SELECT ... FOR UPDATE SKIP LOCKED` (so two concurrent runs never take the same chunk; no separate read transaction exists and the run does not block the table vacuum) followed by the database delete and the backend delete grouped by backend. The cursor advances to the `(deleted_at, id)` of the last row read regardless of the deletion result, so a chunk that fails is skipped and not visited again in the same run: the error aborts the transaction (nothing of the chunk is deleted, locks are released), the handler catches it inside the transaction function and rolls it back explicitly — the commit that `tx-run!` performs afterwards is a JDBC-driver no-op on the already-rolled-back connection — and the loop continues with the returned rows. Accepted quirks, both benign and self-healing: (1) with rows of several backends in one chunk, a failure after one group's physical delete leaves those rows pending with already-removed content — pending rows are excluded from reads and the next run finishes them, since the physical delete is idempotent (S3 ignores missing keys, FS uses `deleteIfExists`); (2) rows locked by a concurrent run are skipped via `SKIP LOCKED` and deferred to the next scheduled run, and the `deleted_at <= now` cutoff is sampled per chunk. It emits the `penpot_storage_gc_deleted_objects_total{backend}` counter (one increment per deleted backend group, reported only after the chunk commits, so rolled-back chunks are never counted twice); run duration is covered by the generic `penpot_tasks_timing{name="storage-gc-deleted"}` histogram and chunk failures by the log-derived `penpot_log_events_total`. Every 5000 deleted objects the run pauses 5 seconds (throttle between chunk transactions, never holding one open). It runs as the `storage-gc-deleted` job (`::job-def`) and beats `jobs/heartbeat` after each committed chunk, so a long sweep keeps its lease and never holds it during a chunk transaction.
|
- `storage-gc-deleted` removes the database row and backend content after the deletion delay.
|
||||||
- `storage-gc-touched` finds references before it sets `deleted_at`.
|
- `storage-gc-touched` finds references before it sets `deleted_at`.
|
||||||
- Orphan watch: `SELECT COALESCE(metadata->>'bucket', metadata->>'~:bucket') AS bucket, count(*) FROM storage_object so WHERE touched_at IS NULL AND deleted_at IS NULL AND status = 'valid' AND NOT EXISTS (SELECT 1 FROM upload_session_chunk usc WHERE usc.object_id = so.id) GROUP BY 1;` Only the `upload-session` row of that output is a leak signal (its rows always have a mapping or a pending touch/deleted mark); other buckets use domain-table references, not this mapping.
|
- `objects-gc` removes deleted domain rows and touches their storage object IDs.
|
||||||
- `objects-gc` removes deleted domain rows and touches their storage object IDs. Each proc streams its pending keys with `db/plan` inside a read-only transaction and deletes them in chunks, each chunk in its own transaction with `FOR UPDATE SKIP LOCKED`: a chunk that fails is logged and skipped instead of aborting the run.
|
|
||||||
- Use `::db/reuse-conn true` with `sto/resolve` inside a database transaction.
|
- Use `::db/reuse-conn true` with `sto/resolve` inside a database transaction.
|
||||||
|
|
||||||
## GC Cost Model
|
|
||||||
|
|
||||||
- `sto/touch-object!` is one `UPDATE storage_object SET touched_at`: DB-only, never touches S3/FS backend content.
|
|
||||||
- Physical FK cascades are avoided by convention; domain deletion is logical (`deleted_at`) everywhere. GC removes small objects first, then parents, then grandparents (see the call order in `app.tasks.objects-gc/execute-objects-gc`). Per-row delete cost is small and bounded.
|
|
||||||
- `objects-gc` sweeps in chunks of 100 (`::chunk-size`, overridable per job params); `storage-gc-deleted` in chunks of 25. `objects-gc` (and the other `db/plan` sweeps) stream the pending keys in a read-only transaction of their own and delete every chunk in a transaction of its own: the cursor fetch and the deletes are never the same transaction. `storage-gc-deleted` has no separate cursor transaction: it reads, locks and deletes each chunk in the same short transaction (see the keyset pagination above).
|
|
||||||
- Consequence: per-chunk cost is DB-only and bounded, so these sweeps cannot outrun the 30-minute jobs lease in practice (would need millions of pending rows in one proc). Per-chunk `heartbeat` where present is belt-and-braces, not load-bearing.
|
|
||||||
|
|
||||||
## Connection Reuse Details
|
|
||||||
|
|
||||||
### `app.storage/resolve` patterns:
|
|
||||||
|
|
||||||
**1. Pool mode (default)** - `(sto/resolve cfg)`
|
|
||||||
- Returns storage abstraction from config
|
|
||||||
- Uses whatever database pool is available
|
|
||||||
- **Safe to call outside transaction context**
|
|
||||||
- Used in: `rpc/commands/media.clj:363`, `rpc/commands/auth.clj:327`, `rpc/commands/profile.clj:362`
|
|
||||||
|
|
||||||
**2. Connection reuse mode** - `(sto/resolve cfg ::db/reuse-conn true)`
|
|
||||||
- Internally calls `db/get-connection cfg` to obtain connectable
|
|
||||||
- Configures storage with the specific connection from config
|
|
||||||
- **Must be paired with transaction that owns this connection**
|
|
||||||
- Used in: `features/fdata.clj:100`, `rpc/commands/media.clj:425`, `rpc/commands/files_thumbnails.clj:307,319`, `binfile/v3.clj:722`
|
|
||||||
|
|
||||||
**3. Explicit configuration** - `(sto/configure storage conn)`
|
|
||||||
- Sets `::db/conn` on storage map directly
|
|
||||||
- Asserts `db/conn? connection` (storage.clj:349)
|
|
||||||
- Used inside `db/tx-run!` blocks where `conn` is already available
|
|
||||||
- Used in: `tasks/file_gc.clj:256`, `rpc/commands/files_thumbnails.clj:347,371`
|
|
||||||
|
|
||||||
### Key Warning (from function notes):
|
|
||||||
|
|
||||||
The improved note in `import-storage-objects` and `handle-persistence` warns: **Do not reuse the main database connection for storage operations within a transaction.** The storage upload process can fail mid-operation, leaving orphaned objects on the backend. If the outer transaction aborts, pending storage objects become unreconciliable because the storage subsystem registers its pending state in separate transactions.
|
|
||||||
|
|
||||||
### Rule of Thumb for `sto/put-object!`:
|
|
||||||
|
|
||||||
Since `put-object!` uses backend-specific operations (`impl/resolve-backend` + `impl/put-object`) and does not directly use `::db/conn` or `::db/pool`, **all usage of `put-object!` will never run inside a common transaction** (if configured at all). The storage backend operations are independent of the database transaction boundary.
|
|
||||||
|
|
||||||
## Deduplication
|
## Deduplication
|
||||||
|
|
||||||
- Deduplication requires `::sto/deduplicate?`, a content hash, and bucket metadata.
|
- Deduplication requires `::sto/deduplicate?`, a content hash, and bucket metadata.
|
||||||
- The lookup matches hash, bucket, backend, and `deleted_at IS NULL`.
|
- The lookup matches hash, bucket, backend, and `deleted_at IS NULL`.
|
||||||
- The lookup only considers rows with `status='valid'`; pending rows are invisible.
|
|
||||||
- A hit whose blob is missing is repaired in place: the same row/id is kept, and `put-object!` rewrites the blob under that id. This heals all existing references to the object. If the rewrite fails, the row is left live and valid for a later retry.
|
|
||||||
- The lookup does not include file ID, profile ID, team ID, or organization ID.
|
- The lookup does not include file ID, profile ID, team ID, or organization ID.
|
||||||
- Objects can therefore share content across users and files within one bucket.
|
- Objects can therefore share content across users and files within one bucket.
|
||||||
- Deleted objects are not reused.
|
- Deleted objects are not reused.
|
||||||
- `tempfile` objects never use deduplication, even when the caller requests it.
|
- `tempfile` objects never use deduplication, even when the caller requests it.
|
||||||
- `job-resource` objects never use deduplication either, whatever the caller asks: the object belongs to one profile, and a shared blob would serve the artifact of one user to the job of another. `put-object!` excludes the bucket from the dedupable set, so the invariant does not depend on call sites.
|
|
||||||
- Use `sto/wrap-with-hash` when the caller already calculated the content hash.
|
- Use `sto/wrap-with-hash` when the caller already calculated the content hash.
|
||||||
|
|
||||||
## Bucket Rules
|
## Bucket Rules
|
||||||
@ -91,18 +50,16 @@ Since `put-object!` uses backend-specific operations (`impl/resolve-backend` + `
|
|||||||
| `file-thumbnail` | File grid thumbnails in `file_thumbnail.media_id`. | Yes | Authentication required | Reference scan. |
|
| `file-thumbnail` | File grid thumbnails in `file_thumbnail.media_id`. | Yes | Authentication required | Reference scan. |
|
||||||
| `profile` | User and team profile photos. References: `profile.photo_id` and `team.photo_id`. | Yes | Authentication required | Reference scan. |
|
| `profile` | User and team profile photos. References: `profile.photo_id` and `team.photo_id`. | Yes | Authentication required | Reference scan. |
|
||||||
| `organization` | Organization logos uploaded by the Nitrate management API. | Yes | Public | No reference scan. A touched object is deleted. |
|
| `organization` | Organization logos uploaded by the Nitrate management API. | Yes | Public | No reference scan. A touched object is deleted. |
|
||||||
| `tempfile` | Export files and temporary font downloads. | No | Authentication required | No reference scan. A touched object uses a two-hour deletion delay. |
|
| `tempfile` | Export files, chunked-upload chunks, and temporary font downloads. | No | Authentication required | No reference scan. A touched object uses a two-hour deletion delay. |
|
||||||
| `upload-session` | Chunked-upload chunks. References: `upload_session_chunk.object_id` and `upload_session_chunk.session_id` (both NO ACTION DEFERRABLE: restrict semantics, procedural deletion). | No | Authentication required | Reference scan on `upload_session_chunk.object_id`; a referenced chunk freezes, an orphan is deleted after the delay. Deferred rows keep their mappings until give-up. |
|
| `file-data` | Encoded file data when `file-data-backend` is `storage`. Reference metadata has `storage-ref-id`, `file-id`, and the `file_data` row ID. | Yes | Authentication required | Reference scan. |
|
||||||
| `file-data` | Encoded file data when `file-data-backend` is `storage`. Reference: the `file_data.storage_object_id` column (FK `NO ACTION DEFERRABLE INITIALLY IMMEDIATE`); the object metadata keeps `file-id` and the `file_data` row ID. | Yes | Authentication required | Reference scan. |
|
|
||||||
| `file-data-fragment` | Compatibility value for file-data fragments. The current backend has no dedicated producer for this bucket. | No current write semantics | Public | No touched-object collector case. |
|
| `file-data-fragment` | Compatibility value for file-data fragments. The current backend has no dedicated producer for this bucket. | No current write semantics | Public | No touched-object collector case. |
|
||||||
| `file-change` | Compatibility value for file changes. Current snapshots store data in `file_data`, not this bucket. | No current write semantics | Authentication required | No touched-object collector case. |
|
| `file-change` | Compatibility value for file changes. Current snapshots store data in `file_data`, not this bucket. | No current write semantics | Authentication required | No touched-object collector case. |
|
||||||
| `job-resource` | Storage objects owned by job rows (`job.resource_id`), written by `app.jobs.storage`. | No | Authentication required, owner-scoped | jobs-GC touch → reference scan. |
|
|
||||||
|
|
||||||
- The valid bucket set is `app.storage.schema/metadata-buckets` (derived from `bucket-requirements`, where a new bucket and its required metadata keys are one entry); `app.storage/valid-buckets` aliases it.
|
- The valid bucket set lives in `app.storage/valid-buckets`.
|
||||||
- `file-media-object` is the default bucket for old rows without bucket metadata.
|
- `file-media-object` is the default bucket for old rows without bucket metadata.
|
||||||
- Do not assign a new bucket without adding its access and cleanup behavior.
|
- Do not assign a new bucket without adding its access and cleanup behavior.
|
||||||
- The touched-object collector raises an internal error for an unknown bucket.
|
- The touched-object collector raises an internal error for an unknown bucket.
|
||||||
- It supports `file-media-object`, `team-font-variant`, `file-object-thumbnail`, `file-thumbnail`, `profile`, `file-data`, `tempfile`, `upload-session`, `job-resource`, and `organization`.
|
- It supports `file-media-object`, `team-font-variant`, `file-object-thumbnail`, `file-thumbnail`, `profile`, `file-data`, `tempfile`, and `organization`.
|
||||||
- It does not support `file-data-fragment` or `file-change`.
|
- It does not support `file-data-fragment` or `file-change`.
|
||||||
|
|
||||||
## Access Rules
|
## Access Rules
|
||||||
@ -110,7 +67,6 @@ Since `put-object!` uses backend-specific operations (`impl/resolve-backend` + `
|
|||||||
- `app.http.assets` decides direct object authentication from the bucket.
|
- `app.http.assets` decides direct object authentication from the bucket.
|
||||||
- Public buckets are `file-media-object`, `file-object-thumbnail`, `team-font-variant`, `file-data-fragment`, and `organization`.
|
- Public buckets are `file-media-object`, `file-object-thumbnail`, `team-font-variant`, `file-data-fragment`, and `organization`.
|
||||||
- Other valid buckets require a session or access-token profile ID.
|
- Other valid buckets require a session or access-token profile ID.
|
||||||
- `tempfile` and `job-resource` are owner-scoped: the request must come from the profile stored in the object metadata, and a mismatch answers 404 (counted as `unauthorized`). A missing owner is tolerated only in `tempfile`, where legacy objects predate the metadata; `job-resource` always carries one, so an object without it is never served.
|
|
||||||
- File-media routes also require file read permission.
|
- File-media routes also require file read permission.
|
||||||
- Non-public direct responses set `content-disposition: attachment`.
|
- Non-public direct responses set `content-disposition: attachment`.
|
||||||
- FS responses use `x-accel-redirect` for the configured asset path.
|
- FS responses use `x-accel-redirect` for the configured asset path.
|
||||||
@ -118,56 +74,10 @@ Since `put-object!` uses backend-specific operations (`impl/resolve-backend` + `
|
|||||||
|
|
||||||
## File Data
|
## File Data
|
||||||
|
|
||||||
- `file-data-backend` accepts `legacy-db`, `db`, or `storage`; `legacy-db` is read-only and the default is `db`. The configuration reads `legacy-db` as `db` at startup, with a warning (`normalize-file-data-backend`), so a deployment that kept the variable saves instead of failing on every write.
|
- `file-data-backend` accepts `legacy-db`, `db`, or `storage`.
|
||||||
- `legacy-db` reads main data from `file.data` and snapshots from `file_change.data`; it no longer writes them (`fdata/upsert!` rejects it).
|
- `legacy-db` stores main data in `file.data` and snapshots in `file_change.data`.
|
||||||
- `db` stores encoded data in `file_data.data`.
|
- `db` stores encoded data in `file_data.data`.
|
||||||
- `storage` stores encoded data in storage subsystem with `file-data` bucket, keeps `data` nil in `file_data` table, and points at the object with `file_data.storage_object_id`.
|
- `storage` stores encoded data in storage subsystem with `file-data` bucket and keeps `data` nil in `file_data` table.
|
||||||
- `file_data.storage_object_id` is a `NO ACTION DEFERRABLE INITIALLY IMMEDIATE` foreign key to `storage_object(id)`: the database refuses to physically delete a referenced object. The `IMMEDIATE` is load-bearing: the RI violation fires inside the `storage-gc-deleted` chunk transaction, so the chunk rolls back, is logged at error level with a generic hint plus the `constraint` and `table` the database reports (any `NO ACTION` foreign key on `storage_object` can block the delete, not only `file_data`'s) and is retried on the next run; with `INITIALLY DEFERRED` the check would fire at commit, outside the catch, and the sweep would never move past the blocked chunk.
|
- The `file_data.metadata.storage-ref-id` value points to the storage object.
|
||||||
- The column is the only source of truth. The release is a coordinated restart (no rolling deploy) when the instance uses the storage backend (`PENPOT_FILE_DATA_BACKEND=storage`) or still has rows carrying `metadata.storage-ref-id`; with the default `db` backend and no such rows the previous and the new code read the same `file_data.data` column, so a rolling deploy is safe. `metadata.storage-ref-id` is no longer written or read, and the `file-data` metadata schema does not declare it. The migration backfills the rows it can resolve (valid uuid with an existing `storage_object`) and removes the key from those rows; a corrupt or orphan pointer keeps the key, which nothing reads. A large instance runs `operations/2.20.0-file-data-storage-object-id.md`: column, index and FK online, then the backfill (only if a row still carries `metadata.storage-ref-id`) with every instance stopped, verifies, and marks the migration as applied (`INSERT INTO migrations`). The backfill must never run while the previous code serves when the storage backend is in use (or rows with the pointer remain), because that code writes and scans the metadata pointer. A rollback to the previous code cannot read the rows written by this release when the storage backend is in use.
|
- `fdata/upsert!` touches a storage object from incoming metadata before it stores the new row.
|
||||||
- `fdata/upsert!` touches the `storage_object_id` it receives in the params (the pointer the persisted row had) before replacing it, so the replaced blob is reclaimed. Callers pass the column from the file map (main/restore), `nil` for a new row (fragments, snapshots, new files) or the row itself (offload); there is no query in the upsert path, which is why every caller reads the file under the row lock first (the migration-on-read path included, see `mem:backend/subtleties`). It returns `{:written <bool> :storage-object-id <uuid or nil>}`, the pointer of the object the write created (nil for `db`/`legacy-db`), and `binfile.common/update-file!` propagates it so a migrating read returns the map with the pointer of the row it persisted instead of the pre-write one. The write also compares the expected pointer with `IS NOT DISTINCT FROM` (the caller's value, not the new one) and raises `:stale-file-data-pointer` when the row changed under it, so a stale map fails instead of leaking the replaced blob; the check lives in `upsert-in-database` because the `legacy-db` branches do not use the upsert SQL.
|
|
||||||
- `binfile.common/update-file!` takes the file map read from the database plus the fields to change (`[cfg file & params]`), merges them and builds the `file_data` params from the merged map, so the pointer travels in the read map and a caller cannot lose it by building a partial map; the params cannot override `:storage-object-id`. `insert-file!` always drops it (an INSERT never replaces a persisted pointer; `duplicate-file` hands the source map there). `bfc/get-file` keeps the pointer for the write paths, and every response goes through `bfc/file->response`, which strips it and any residual `storage-ref-id` left in the metadata of the rows the migration does not backfill: each `file-data` object is authentication-only and any logged-in profile could otherwise fetch it from `/assets/by-id`.
|
|
||||||
- The in-place binfile import was removed: `import-files!` raises `:not-supported` when the cfg carries `::bfc/file-id`, so no caller can overwrite a storage-backed row without carrying its previous pointer.
|
|
||||||
- File snapshots use `file_data` for snapshot data and `file_change` for snapshot metadata.
|
- File snapshots use `file_data` for snapshot data and `file_change` for snapshot metadata.
|
||||||
- Tiered offload (`:tiered-file-data-storage` flag + `db` backend): `file-gc` submits an `:offload-file-data` job (priority 10, delay 1000, same params) only when it actually processed the file; the handler takes the `file` row with `FOR UPDATE SKIP LOCKED` and skips (`{:skipped 1}`, no error) when the row is locked or gone, then moves every live `file_data` row of that file with `backend = 'db'` to `backend = 'storage'` via `fdata/upsert!` (one `put-object!` in bucket `file-data` per row, `data` becomes nil and `storage_object_id` points at the new object) inside a single all-or-nothing transaction per file. The skip only serializes because every `file_data` writer locks the `file` row first (the update RPC and the snapshot paths with `FOR UPDATE` through `lock-for-update?`, the migration-on-read path with `FOR UPDATE`). A storage failure aborts only that file and bubbles for a runner retry; orphans go to `storage-gc-touched`. A second run is a no-op because the query only matches `backend = 'db'`.
|
|
||||||
- The offload normalizes each `SELECT fd.*` row before `upsert!`: the row builder omits NULL columns, jsonb metadata arrives as a `PGobject`, and a JSON `null` decodes to nil and must disappear instead of travelling as `:metadata nil`, which the update schema rejects.
|
|
||||||
- Offload pacing: none by deliberate decision. Sleeps only make sense when batching multiple transactions; the per-file transaction is short by construction and the scheduler run takes minutes per production metrics, so neither loop sleeps. `jobs/heartbeat` (pool-based liveness, ~1 write/s) stays on both paths.
|
|
||||||
- Offload metrics (bounded labels only, never ids or error text): `penpot_file_data_offload_rows_total{type}` and `penpot_file_data_offload_bytes_total{type}` (`type`: `main`/`snapshot`/`fragment`, successful runs only) plus `penpot_file_data_offload_timing_ms{result}` (`result`: `ok`/`skip`/`error`). Helpers live in `app.jobs.metrics`; the job-def requires `::mtx/metrics`.
|
|
||||||
- Writes always resolve the backend through `fdata/default-backend` (`cf/get :file-data-backend`), so post-offload edits land back in hot (`db`) storage without extra code; reads stay transparent through `fdata/resolve-file-data` + `decode-file-data` + `realize`.
|
|
||||||
- Deleting an offloaded row touches its storage object first (`objects-gc/delete-file-data-chunk` via `fdata/touch-storage-object` with the row's `storage_object_id`), so `storage-gc-touched` reclaims the blob with the usual delay; restoring a file unmarks the referenced objects.
|
|
||||||
- `storage-gc-touched` resolves the `file-data` reference with `SELECT EXISTS (SELECT 1 FROM file_data WHERE storage_object_id = ?)`, an indexed lookup that no longer parses the object metadata.
|
|
||||||
- A row with `backend='storage'` and no readable object makes `resolve-file-data` raise `:missing-storage-object` with the file and row ids instead of failing later in the storage backend.
|
|
||||||
- The `file-id` and row id kept in the storage object metadata are informational for diagnostics; nothing parses them to resolve the reference (the reference is the column).
|
|
||||||
|
|
||||||
## File Change
|
|
||||||
|
|
||||||
- `file_change` is partitioned by `HASH (file_id)` in 16 partitions, with the same bounds as `file_data`, so a join between them is resolved partition by partition (`enable_partitionwise_join` is set on every pooled connection).
|
|
||||||
- The automatic migration (`0158-partition-file-change`) refuses to run when `file_change` is above 10 GiB total size or 2M rows (`pg_total_relation_size` + `pg_class.reltuples`, catalog reads). It analyzes the table first, because `reltuples` is -1 without statistics. It raises with the path of `operations/2.20.0-file-change-partition.md`, because the copy happens inside the migration transaction and a big instance would restart in a loop. Below the guard it raises `statement_timeout` to 30 minutes for the copy and the index builds, and analyzes the new partitions at the end, after the copy and the indexes, so the planner does not start with empty statistics.
|
|
||||||
- Its primary key is `(file_id, id)`: a partitioned table needs the partition key in the primary key, so every access filters by `file_id`. From 2.20 the four snapshot commands (`update-file-snapshot`, `delete-file-snapshot`, `lock-file-snapshot`, `unlock-file-snapshot`) require the `file-id` param, because they use it to prune the partition; without it they answer `:params-validation`.
|
|
||||||
- It has a `kind` column: `1` is a snapshot (it has a label and its data lives in `file_data`; the read-only `legacy-db` backend keeps old payloads in `data`) and `2` is a revision (it has no label and it holds the changes). The values are in `app.features.file-changes`.
|
|
||||||
- `kind` is a deliberate reserve, not a leftover: today no query filters by it, the queries still tell a snapshot from a revision by its label (`label IS NOT NULL`), and `kind` takes over when they are rewritten. No constraint ties `kind` and `label` together on purpose: the day `kind` is the source of truth, that constraint would have to go. It stays nullable on purpose: the previous version inserts without it, and a rollback to it has to keep writing; the `NOT NULL` comes in a later migration.
|
|
||||||
- Every write of the application to a row of `file_change` moves `updated_at` to the moment of the write (`lock-by!`, `unlock!`, the snapshot rename and delete, `delete-object`, the restore of a file). The actions of the foreign keys (`ON DELETE SET NULL` on `locked_by` and `profile_id`) change the row without touching it; during the partition both tables carry the same keys, so both receive them. The runbook of the partition uses it as the watermark of its delta passes: a write of the application that leaves it behind is lost at the cutover.
|
|
||||||
- `file_change_legacy` is a frozen copy of the previous layout of the partition: it keeps the rows and the indexes of the old table. A new instance (empty table) does not keep it. On an upgraded instance the table stays until the operator drops it by hand days later (`DROP TABLE file_change_legacy;`, see `operations/2.20.0-file-change-partition.md`); nothing drops it automatically.
|
|
||||||
- A revision carries only the delta of a save (the changes). The features of the file are not repeated on it: nobody read them from there.
|
|
||||||
- A snapshot carries the features and the migrations of the file.
|
|
||||||
- It keeps the `data` column for reading only: migration `0157-migrate-legacy-file-data` moved the existing blobs to `file_data`, and `legacy-db` remains as a read-only backend for the payloads still in `data`. `fdata/upsert!` rejects `legacy-db` and the `file-data-backend` default is `db`. It has no `data_ref_id` or `data_backend` columns: nothing read them since the file storage refactor.
|
|
||||||
- It has no index for the system snapshots. The old one filtered them by `data IS NOT NULL`, and with the payload living in `file_data` the column no longer tells snapshots apart. The new one could only filter them by `deleted_at IS NULL`, but the system snapshots always carry a deletion date in the future, so it would never match a row. The query that would use it filters that date with an "or", which no partial index can serve.
|
|
||||||
- The `objects-gc` purge groups each chunk by `file_id` and locks it with `file_id = ?` and `id = ANY(?::uuid[])`: a tuple `IN ((file_id, id), ...)` gives the planner no constant partition key, so it scans all 16 partitions in every chunk. Each group is one `FOR UPDATE SKIP LOCKED` query inside the chunk transaction.
|
|
||||||
|
|
||||||
## Metrics
|
|
||||||
|
|
||||||
- `bucket` is always the Penpot logical bucket (object metadata), never an S3 bucket. Unknown/absent buckets are labeled `"unknown"`.
|
|
||||||
- `target` is the physical S3 destination id. Today it is always `"default"` (hardcoded in `app.storage.s3/build-s3-client`; the `::target-id` config key was removed as unused until the per-bucket routing plan lands).
|
|
||||||
- Physical S3 API calls (AWS SDK `MetricPublisher`, `app.storage.s3.metrics`):
|
|
||||||
- `penpot_storage_s3_requests_total{operation,target,result}` — one count per logical SDK call (the published `ApiCall` collection, not per attempt); retries are counted apart in `retries_total`, so total attempts = `requests + retries`. `result` is `"ok"` only when the SDK reports success as exactly `true`; a missing success flag counts as `error`.
|
|
||||||
- `penpot_storage_s3_retries_total{operation,target}` — SDK retry count.
|
|
||||||
- `penpot_storage_s3_timing{operation,target}` — call latency histogram (ms); explicit buckets up to 60000 ms (S3 slow calls exceed the default 7500 ms cap).
|
|
||||||
- Logical storage operations (`app.storage`, `::mtx/metrics` required by the schema):
|
|
||||||
- `penpot_storage_operations_total{op,bucket,backend}` — `put`, `repair`, `get-data`, `get-bytes`, `del`, `touch`, `exists`. All ops are success-only: `put`/`repair` emit after the backend write, `get-*` after the backend fetch opens, `touch`/`del` only when a row actually changed. `touch-object!`/`del-object!` take the object id (UUID) only — no object overload. Labels come from the updated row itself via `UPDATE ... RETURNING id, backend, metadata` (no extra `SELECT`); with no row matched they emit nothing. `del-object!` only matches live rows (`deleted_at IS NULL`): a repeated del returns `false` and emits nothing. Post-open stream read errors stay counted as attempts. `del` only marks `deleted_at`; physical deletion is a GC concern. `exists` is emitted per deduplication-hit probe, always paired with a `hit`/`repair` outcome (never on probe failure), not per user-facing existence check.
|
|
||||||
- `penpot_storage_dedup_total{result,bucket}` — `hit`, `miss`, `repair`, `skip`.
|
|
||||||
- Storage GC poison (`app.storage.gc-touched`, `::mtx/metrics` required in the handler cfg):
|
|
||||||
- `penpot_storage_gc_poison_total` — touched rows whose metadata fails to decode. Each one is logged at error level and deferred exactly one day instead of aborting the chunk. No labels; incremented once per row per scan.
|
|
||||||
- Asset serving (`app.http.assets`, `::mtx/metrics` required in the handler cfg):
|
|
||||||
- `penpot_storage_asset_requests_total{route,backend,bucket,result}` — `route` is `by-id`, `by-file-media-id`, or `thumbnail`; `result` is `served` (<400), `not-found` (404), `unauthorized` (401/403), or `error` (everything else, including a nil/non-number status: every serve path must set `::yres/status`). Serve-path exceptions are counted by `serve-object-measured` and then rethrown. Permission-denied file-media requests and tempfile ownership mismatches both answer HTTP 404 (to avoid leaking existence) but are counted as `unauthorized`. Malformed UUIDs raise before any emission point and are never counted. Counts backend requests that trigger a browser GET to the object store (one per cache miss), so it is a proxy for object GETs, not an exact count.
|
|
||||||
- The physical and logical counters intentionally overlap in coverage but differ in meaning; do not sum them.
|
|
||||||
- Metrics is not optional: `::mtx/metrics` is required by the storage and s3-backend schemas, and the assets handler cfg always carries it. Wiring a component without metrics is a bug, not a supported mode.
|
|
||||||
- Recording never fails: `app.metrics/run!` is safe by default at every emit point (`emit-op!`, `emit-dedup!`, `emit-asset!`, and the three S3 publisher emissions). The first recording failure per metric id logs at `warn`, later ones at `debug` (no log flood). The `instance` precondition is a plain assert and the collector lookup is outside the recording guard, so a missing instance fails hard (see `mem:backend/subtleties`). The `publish` outer try/catch stays: it is an SDK `MetricPublisher` contract boundary, not a metrics guard.
|
|
||||||
|
|||||||
@ -1,118 +0,0 @@
|
|||||||
# Backend Subtleties
|
|
||||||
|
|
||||||
## RPC exposure and wrappers
|
|
||||||
|
|
||||||
- RPC commands are discovered from vars created by `app.util.services/defmethod`; adding a command namespace is not enough unless `backend/src/app/rpc.clj` includes it in `resolve-methods`.
|
|
||||||
- `GET`/`HEAD` RPC calls are only allowed for method names starting with `get-`. Other methods are method-not-allowed even if they are read-only internally.
|
|
||||||
- RPC auth defaults to enabled. Public endpoints must set `::auth false` metadata explicitly.
|
|
||||||
- The wrapper stack does auth before params validation, then auditing/rate/concurrency/metrics/retry/condition handling, with DB transaction handling inside that stack. `::db/transaction` metadata controls transaction wrapping.
|
|
||||||
- Params with `::sm/params` are decoded/conformed through the JSON transformer and successful IObj results get `:encode/json` metadata. Legacy spec conforming only applies when no Malli params schema exists. Client params are stripped of qualified keys (`d/without-qualified`) before merging with the server auth context, so request bodies cannot override `::profile-id`, `::auth-type`, or `::token-perms`.
|
|
||||||
- Params schemas are open by default, so undeclared client keys reach the handler unless the map is `:closed true`. Creation commands (`create-file`, `create-project`, `create-team`, `create-team-with-invitations`, `upload-file-media-object`, `create-file-media-object-from-url`, `assemble-file-media-object`) use closed schemas: a client-provided `:id` fails with `:params-validation`. Their internal creation functions still accept an optional explicit `:id` for imports, duplicates and deterministic test fixtures.
|
|
||||||
- Nil RPC bodies become HTTP 204 unless explicit status metadata is present. Stream bodies default to `application/octet-stream` when no content type is set.
|
|
||||||
- The upload schema (`media.v/schema:upload`) takes a `:path` from the caller, and a Transit body can tag any string as a `path`, so a command that reads such an upload reads a file the client named. `create-import-binfile-job` avoids that boundary by accepting only a chunked upload (`:upload-id`); the commands that still take a plain file (the legacy `import-binfile`, the media and font uploads) trust it, and the guard belongs where uploads are accepted, not in each command.
|
|
||||||
|
|
||||||
## Concurrency limiter (CLIMIT)
|
|
||||||
|
|
||||||
- `::climit/id` is either a plain keyword, which gives one shared limiter per process, or the vector form `[[id key-fn]]`, which gives one limiter per key. `climit/wrap` takes the key only from that positional `key-fn` (`climit/get-limits` maps a plain keyword to a single-element vector), so `::climit/key-fn` is metadata that nothing reads: a command declaring `::climit/id :x/by-profile` next to it ends up with a single shared limiter, and the missing key is visible in the `app.rpc.climit` log lines, whose `id` carries no key suffix. That is what happened to `push-audit-events`, the last user of the dead metadata, from `a5c6d78ee5` (2024-01-27, "fix some fundamental bugs on climit module", which moved the key to the vector form without updating it) until it was fixed; nothing declares `::climit/key-fn` any more, so keep using the vector form for anything that must be per profile.
|
|
||||||
- A gauge written by whichever limiter of a name ran last cannot show a queue: every per-profile limiter of the name collapses into one series, so a healthy profile writing 0 overwrites the 15 of the one that is queueing (measured in production: 4 of 1441 samples). `penpot_rpc_climit_waiting` replaces it and is accumulated: one counter per name and instance, up when a call starts waiting for a permit, down when it stops, so a call that enters and never leaves keeps the value above zero for ever. `penpot_rpc_climit_rejected_total` counts rejections by name and `reason`: `queue-full` when the error carries `:size`, `timeout` when it carries `:timeout` (both carry `:code :capacity-limit-reached`, so the timeout field is the discriminator). The two gauges that could not tell the truth (`penpot_rpc_climit_queue`, `penpot_rpc_climit_permits`) were deleted, and so was `penpot_http_worker_busy_threads`, which reads 0 on an instance with 72 blocked threads: `penpot_http_worker_pool_size` is the signal for retained threads. A stuck limiter also shows up in `penpot_rpc_climit_timing_count` (the acquired path), which is a counter: compare the same limiter name across instances and the stuck one reads 0 acquisitions while its peer reads thousands. In thread dumps it reads as threads parked in `psm/acquire` whose CPU time does not grow between two dumps, on an instance that is otherwise idle.
|
|
||||||
- The waiting counters are a field of the `ConcurrencyLimiter` component (a `ConcurrentHashMap` of name to `AtomicLong`), so they start at zero, die with the component, and the sampler that closes over that map republishes only its own names every 15 s; the sampler stops with the component in `halt-key!`. `impl-invoke` pairs one decrement with one increment through a per-call flag: the flag turns a second call into a no-op and the `finally` guarantees that there is one. The increment sits inside the `try` and the decrement lives only in the acquired callback and the `finally`, so a handler that throws, a rejection and a throwable from the publishing path all leave the count where it was and the thread without the limiter marked as held (a leftover mark would skip that limiter for that key for ever). Concurrent `inc`/`dec` publications of two threads can land out of order and the value can read low for a moment; the republish corrects it.
|
|
||||||
- The `app.rpc.climit` log lines carry `queue`, `queue-max`, `permits`, `elapsed`, `code` and `reason`, and their `id` carries the key (`name/<profile or key>`). That is the only place where the profile behind a queue is visible: a pile-up is chased there, on demand, not in the metrics. The `queue` of the line is the calls inside the limiter minus its permits, so it reads 0 for a lone caller that times out: that line goes out at TRACE instead of WARN, and the rejection count has to come from the metric.
|
|
||||||
- Nothing in the limiter path has a timeout unless the config sets one, and `psm/acquire` then calls `Semaphore.acquire`, which parks the calling thread forever. A parked thread still counts as in-system for every outer limiter of its chain, so the outermost one (the method's own by-profile limiter) stops granting permits and answers 429 to everything. A limiter without `:queue` gets an unlimited queue and therefore never rejects: its pile-up is silent, with no 429 and no log line except the WARN `enqueued`/`acquired` lines of the keyed id.
|
|
||||||
- Deadlock shape between a command's own limits and the root ones (fixed): a command declaring its own limits held `process-image/by-profile` before `root/by-profile`, while a command with no limits of its own takes the roots first and then re-enters with the same keyed limits from inside its handler (`create-file-media-object-from-url`, `upload-photo`) and waits for them holding a root permit. Two calls of one profile then held the same two limiters inverted and parked for good: 69 threads at 0 CPU, an audit limiter with 0 acquisitions in 24h and 100% 429s, on an instance that never served a 204. Both commands declare their limits now, so the shape cannot happen any more: the root limits are taken last and every path takes the limiters in the same order.
|
|
||||||
- `climit/wrap` takes the command's own limits before the root ones: the root limiters are appended to the sequence first and the last element of the sequence is the outermost one, so `root/global`/`root/by-profile` end up innermost. A request waiting for its own limits no longer holds the root permits of its profile, which is what kept a pile-up of heavy requests from blocking the other commands of that profile. Every acquisition goes through that wrapper, so every path takes the limiters in the same order: there is no way to enter the limiter from inside a handler, and a command that calls another command internally would take the limiters twice (the second call waits for a permit the first one holds and answers 429 when the timeout runs out). The key of a limit is a function of the params (a keyword is looked up in them, a function is applied to them, any other value is the key itself). Commands declare the by-profile limit before the global one, which makes it the outermost of the pair, and a test fails when the relation between all the limits the commands declare cannot be sorted: two commands that take the same limits in opposite order, or a ring of three, leave the paths of one profile holding them in opposite order.
|
|
||||||
- The two commands that used to enter climit from inside their handler declare their limits now, so the whole command runs inside them: the download of an external image takes a `process-image` permit, and that limit has one permit per profile, so a slow download holds the image processing of that profile for as long as it takes. It takes the global permit of that limit too (8 permits, queue 20), so eight slow downloads of eight different profiles leave the image processing of every profile waiting. The copy of the body has no timeout (the 30 s of the request covers the connection and the headers), so a server that stops sending keeps the permits until the operating system gives up. The storage write of the profile photo runs inside its limit too.
|
|
||||||
- The shipped `backend/resources/climit.edn` sets a timeout and a queue on every limiter, 60s for operations that can be slow and 30s otherwise, with a test that fails when an entry ships without one, and `init-key` logs a WARN for every entry without a timeout. The queue matters because the root limits no longer bound the waiters: a command takes its own limits first, so a pile-up waits in the queue of that limit. The file and the declarations have to list the same ids: a `::climit/id` without an entry makes `wrap` log `no config found for specified queue` and run the handler with no limiter at all, and a test fails when a command declares an id the file does not list. Only key a limit by profile when the work contends on something shared (`update-file`, images, fonts, thumbnails, the profile row): a global cap on a cheap command adds a rejection path nobody needs, which is why `profile-plugin-ops` and `submit-audit-events` declare only `by-profile`. The production file lives outside the repository: mirror it there, and remember that a parked thread only goes away with a restart or a deploy.
|
|
||||||
|
|
||||||
## DB helpers
|
|
||||||
|
|
||||||
- Most `app.db` helpers accept a pool, connection, or map containing `::db/pool` / `::db/conn`; preserve that convention in shared code.
|
|
||||||
- `db/tx-run!` uses `next.jdbc.transaction/*nested-tx* :ignore`: nested transaction calls reuse the outer transaction, not a savepoint. Use explicit savepoints when nested rollback semantics matter.
|
|
||||||
- Whether a call is nested is decided by the CONNECTION, not by an ambient flag: `db/transact!` binds `*after-commit-conn*` around the body, and a transactable that is not identical to that connection is an independent unit of work that owns its own after-commit context. A `tx-run!` that passes a pool while a transaction is open therefore commits on its own and drains its own callbacks, instead of leaving them in a context that may roll back.
|
|
||||||
- `after-commit` callbacks run on commit only. A transaction marked `::db/rollback` (used by `srepl` and `app.tasks/offload-file-data`) commits nothing, so it runs none of them.
|
|
||||||
- `db/run!` opens/reuses one connection but does not create a transaction.
|
|
||||||
- `db/tjson` is Transit JSON for jsonb storage; `db/json` is plain JSON. Job params (`job.params`) are plain JSON, decoded with the job-def decoder (`app.jobs/decode-params`); only the legacy `task` props were Transit (`decode-transit-pgobject`). Job results (`job.result`) are plain JSON too: they are read with the result decoder of the job-def (`app.jobs/decode-result`), and a result map keyed by uuid must declare its schema (`[:map-of ::sm/uuid ...]`, the import resolution does) or every key comes back as a keyword and every uuid as the text it is on disk.
|
|
||||||
- Advisory transaction locks accept UUIDs or ints. UUID locks are hashed using a zero-UUID seeded siphash.
|
|
||||||
|
|
||||||
## Workers and cron
|
|
||||||
|
|
||||||
- `job` carries `tenant` (the instance the job belongs to) and a bare `queue`. Several instances share one database and the tenant is what keeps an instance from claiming another environment's jobs, so every query that acts on one instance's work filters on `tenant`: dispatch claim, orphan sweep, lost-job reschedule, cron no-overlap count, submit dedupe. The `job__dispatcher__idx` leads with `tenant` (equality) then `(priority DESC, scheduled_at)`; `queue` does not lead because the dispatcher claims every queue of its tenant. The `<tenant>:<queue>` prefix survives only in the Redis key, composed once by `app.worker/queue-key`.
|
|
||||||
- Submit dedupe only removes not-yet-due `new` jobs of the same tenant with the same name/queue/label; it does not dedupe due, scheduled, retry, running, or completed work. The `::jobs/skip-if-active` submit option goes further: it skips the insert (returning `nil`, recording `penpot_jobs_skipped_total`) when a live job (`new`/`scheduled`/`running`/`retry`) already exists for the same tenant/name/queue/label, requires an explicit non-blank label, and carries the same best-effort cross-backend race caveat as dedupe.
|
|
||||||
- Jobs metrics rule (one event, one family): counters exist only for events without latency; when an event has a histogram, its `_count` is the counter (no twin counter). `dispatched` (pushes) vs `queue-wait` (claims) are deliberately different events — do not merge them.
|
|
||||||
- The dispatcher selects `new`/`retry` jobs with `FOR UPDATE SKIP LOCKED`, marks them `scheduled`, and publishes the JSON payload `[id scheduled-at]` to the `penpot.worker.queue:<tenant>:<queue>` Redis list. The runner skips Redis messages whose scheduled timestamp no longer matches DB state.
|
|
||||||
- Lost `scheduled` jobs are rescheduled after 5 minutes; `running` jobs untouched longer than `:jobs-lease` (default 30 min) are marked `aborted` with `app.jobs/orphan-error` (`type :internal`, `code :orphan`). `aborted` is a system-side terminal state like `cancelled` (never retried, swept by retention like other terminals, reported with an error log); long jobs must `heartbeat` per chunk so a healthy backlog is never mistaken for a dead process. The `:progress` report is optional and only feeds the durable log. A beat whose write finds the job no longer active raises an `:interrupt` (`code :job-interrupted`) instead of counting zero: it is the internal stop-the-work signal, not an error, so neither the listener nor the runner logs it as one.
|
|
||||||
- An orphan gets its `end` event (`outcome "aborted"`) from the sweep itself: one bulk `INSERT ... SELECT` in the same transaction as the bulk `UPDATE`, never one insert per orphan. No msgbus notification goes out for it; orphans alert through the error log and the orphaned counter.
|
|
||||||
- `job_event` is append-only and deleted by cascade with the job. `progress` is a row there, not a column: no Redis and no mutable column to keep in sync.
|
|
||||||
- `job` is deletion-protected at the SQL level (`deletion_protection__tgr`): a stray `DELETE FROM job` raises instead of dropping the row and its `job_event` cascade without touching the referenced `storage_object`. The only intended deleters, the jobs GC and the submit dedupe, disable the guard with `SET LOCAL rules.deletion_protection TO off` inside their transaction; tests use `th/db-force-delete`.
|
|
||||||
- A missing job-def raises (`:no-job-definition`) instead of completing. Throwing with `ex-data :type ::retry` still controls retry behavior; `:strategy ::noop` retries without incrementing retry count.
|
|
||||||
- Cron entries claim their `scheduled_task` row with `FOR UPDATE SKIP LOCKED`, disable statement/idle-in-transaction timeouts locally, submit one `job` row per entry when no active instance exists (no-overlap), and reschedule themselves in `finally` unless interrupted. Worker, dispatcher, and cron components do not start when the DB pool is read-only.
|
|
||||||
- Integrant params schemas name injected components as bare `::ns/key` entries (pool, metrics, msgbus, job defs), and a bare entry is a malli registry lookup, not `any?`: the owning namespace must `sm/register!` it at load, or the backend crashes at boot with `:malli.core/invalid-schema` instead of asserting. Unit tests never build those keys, so `backend-tests.worker-test` compiles the three worker schemas and `backend-tests.http-websocket-test` compiles the revocation-watcher one to catch it. The assertion only runs with `PENPOT_FLAGS=enable-backend-asserts`, so without it a broken schema reaches production and only crashes at boot.
|
|
||||||
- Whether a bare entry works has nothing to do with its position or with how many of them there are: it is a registry lookup, so it works only if that exact keyword was `sm/register!`'d. `[:map ::db/pool ::metrics/metrics]` is fine, `[:map :app.aaa/unregistered :app.ccc/unregistered]` throws. When a component is **not** registered as a malli type, spell it as an explicit pair instead of registering a type just to satisfy the lookup — `[:app.nitrate/client {:optional true} [:maybe :map]]` is how `auth.oidc/schema:routes-params` and `loggers.audit.archive-task/schema:audit-log-archive-params` declare it. `{:optional true}` matters because `init-key` yields `nil` when the owning feature flag (`:admin-console`) is off, and a required key would then fail the assert on a perfectly normal boot.
|
|
||||||
- `sm/valid?` catches every `Throwable` and returns `false`, so an uncompilable schema and a genuinely invalid value both surface as a bare `Assert failed:` with no cause. When an `assert-key` fires with no message, compile the schema by hand to see the real exception.
|
|
||||||
|
|
||||||
## Config and HTTP/session middleware
|
|
||||||
|
|
||||||
- `app.config/config` and `flags` are dynamic `defonce` vars populated from `PENPOT_*` env vars through the shared schema string transformer. Tests and tooling can bind them.
|
|
||||||
- Duration config values are parsed as ISO-8601 with the `PT` prefix, so a day is written in hours (`:jobs-retention` and `:jobs-user-ttl` default to a week, and an operator writes `168h`): `7d` does not parse, and the value fails validation at boot instead of being silently ignored.
|
|
||||||
- `app.main` wires `PENPOT_DATABASE_*` into `::db/pool`, and `app.db/create-datasource-config` forwards pgjdbc's own knobs as `dataSourceProperties`, which `Properties.getProperty` silently drops unless the value is a String (hence the `str`). A query parameter of the same name in `PENPOT_DATABASE_URI` (for example `?prepareThreshold=0`) wins over the property: pgjdbc ignores the Properties value when the URL carries it, so the URI is the place to look when a variable seems to do nothing. The prepared statement cache is three optional `PENPOT_DATABASE_PREPARED_STATEMENT_CACHE_*` keys: the threshold is the executions before a statement is server-prepared and accepts `-1` or non-negatives (`{:min -1}`; the driver clamps any other negative to `-1`, so rejecting it only turns a typo into a boot error), its `0` stops server-side prepared statements while the driver's client-side cache of parsed queries stays in use, `..._QUERIES` and `..._SIZE` reject negatives and `0` in either disables that cache, and the size is bytes rounded up to the whole MiB the driver accepts. Do not describe the threshold `0` as disabling the cache: it only stops the server-side prepares.
|
|
||||||
- `parse-flags` automatically adds `:disable-secure-session-cookies` when `public-uri` is plain HTTP and not localhost. This changes cookie defaults without an explicit env flag.
|
|
||||||
- The backend sets Clojure `*assert*` globally from the `:backend-asserts` feature flag. Assertion-dependent checks can therefore differ by runtime flags.
|
|
||||||
- Request body parsing is mostly POST-oriented and supports Transit JSON plus plain JSON. Plain JSON request keys are kebab-decoded before being merged into `:params`. A body the parser cannot read is a client error, classified at the parse site and never by exception class (`clojure.data.json` signals its syntax errors with a bare `java.lang.Exception`): 400 `:validation`/`:malformed-json`, one `warn` line (`malformed request body`) carrying path, peer address and content-type instead of a stack trace, and `penpot_http_malformed_requests_total`. `RequestTooBigException` keeps its 413.
|
|
||||||
- Response formatting negotiates with `Accept` or `_fmt=json`. Transit is the default for collection/boolean bodies; JSON encoding has special pointer-map handling.
|
|
||||||
- Auth prefers the session cookie token before the `Authorization` header. Headers may be `Token` or `Bearer`. Only `kid=1`/`ver=1` tokens are decoded as session tokens; anything else is left unauthenticated (legacy v1 tokens were removed).
|
|
||||||
- Shared-key auth requires `x-shared-key` as `<key-id> <key>` and stores the lowercased key id on the request. If no shared keys are configured it always rejects. Management methods declare the key ids they accept with `::rpc/allowed-shared-keys`; a valid key outside that set answers 403 `:shared-key-not-allowed`, and a method without the metadata accepts any configured key.
|
|
||||||
- Session management uses DB storage unless the DB pool is read-only, then falls back to the in-memory manager. Sessions use only the v2 UUID model (`http_session_v2`); legacy string ids were removed.
|
|
||||||
- Session cookies are renewed when `modified-at` is older than the 6h renewal interval. SameSite is `none` for CORS, otherwise strict/lax based on config. Session lifetime config and GC: `mem:backend/session-expiration`.
|
|
||||||
|
|
||||||
## HTTP server self-metrics
|
|
||||||
|
|
||||||
- `app.http` enables Undertow connection statistics via the yetti option `:server/statistics` (requires yetti ≥ v11.11, which exposes it; before the patch `ListenerInfo#getConnectorStatistics` returned `nil`).
|
|
||||||
- `app.metrics/create-sampler` holds the sampler shape: a single daemon `px/scheduled-executor` thread with a self-rescheduling `px/schedule` chain, so a failing sample never cancels the next one; the owner starts it in `ig/init-key` and stops it with `px/shutdown-now` in its `halt-key!`. Http and CLIMIT both sample every 15 s.
|
|
||||||
- `app.http` uses it to publish worker and listener state: `penpot_http_worker_queue_size`, `penpot_http_worker_pool_size`, `penpot_http_worker_max_pool_size`, `penpot_http_connector_active_connections`, `penpot_http_connector_requests_total`, `penpot_http_connector_errors_total`. Definitions live in `app.main/default-metrics`. The xnio busy-thread count is not published: it reads 0 with threads blocked inside a limiter, which is why it was deleted.
|
|
||||||
- The xnio worker MXBean can return transient `-1` (e.g. on the queue or the pool size); negative samples are discarded (gauge keeps its previous value). Undertow exposes absolute request/error totals, so the sampler keeps a watermark atom and publishes deltas; a counter reset (decreasing totals) skips the negative delta and moves the watermark forward.
|
|
||||||
- The `process_*` families (`process_open_fds`, `process_max_fds`, `process_cpu_seconds_total`, …) come from the prometheus client `StandardExports`, registered by `app.metrics/create-registry`. They read the OS MXBean reflectively and need the `jdk.management` module: on a pruned `jlink` JRE the MXBean is `sun.management.BaseOperatingSystemImpl`, the getters throw `NoSuchMethodException` and `StandardExports#collect` swallows it, so those families silently vanish from `/metrics`. `docker/images/Dockerfile.backend` keeps `jdk.management` in the `--add-modules` list, and `backend-tests.metrics-test` pins the contract.
|
|
||||||
|
|
||||||
## Metrics recording
|
|
||||||
|
|
||||||
- `app.metrics/run!` is safe by default: a recording failure never throws (a metrics bug must not change the behavior of the measured operation). The first failure per metric id logs at `warn`, later ones at `debug`. The `instance` precondition is a plain assert (the backend enables `:backend-asserts`), and the collector lookup sits outside the recording guard, so a missing instance fails hard even when asserts are disabled. `::mtx/metrics` is required by the storage, s3-backend, db-pool and jobs APIs; jobs helpers never silently skip a missing metrics instance. The resolver is `app.metrics/instance` and it raises, unlike `run!`.
|
|
||||||
- `app.msgbus` (`pub!`, `sub!`, `purge!`) keeps taking the **instance** on purpose: msgbus is what produces it, and `app.rpc.notifications` already shows the caller passing a cfg. `app.jobs` resolves `::mbus/msgbus` from the cfg internally, and `insert-event` keeps its explicit pre-insert check so a profile job never loses its notification.
|
|
||||||
- `app.db/after-commit` is the boundary for SQL-backed metrics: callbacks are drained by the commit of the transaction that owns the connection they were registered on, and discarded when it rolls back. Because nesting follows the connection, that transaction is the outermost one only for a real nested call.
|
|
||||||
- `penpot_http_malformed_requests_total` (counter, no labels) counts requests whose body the parser could not read. `app.http.middleware/wrap-parse-request` emits it where it classifies the error, so the instance travels as a middleware argument (`[mw/parse-request (::mtx/metrics cfg)]`) and each service declares the definition with its own prefix (nexus: `nexus_http_malformed_requests_total`). The per-endpoint request counters of nexus do not count these requests: they sit inside the parse middleware, so the failure happens before them.
|
|
||||||
|
|
||||||
## Log-derived metrics
|
|
||||||
|
|
||||||
- `app.loggers.metrics/reporter` counts the log records the application emits and publishes them as `penpot_log_events_total{logger, level}` (counter), so the error panels do not have to count text in Loki. It is a component that hooks `add-watch` on `app.common.logging/log-record`, like `app.loggers.database`, but with no thread and no channel: the increment is in memory, so it goes straight in the watcher. Only `warn`, `error` and `fatal` are counted, and only the records that go through `app.common.logging` (the application's own logs; a library writing straight to log4j2 is not counted, and today that is one line in a thousand). The body of the watcher is wrapped in a `try/catch` that swallows the error on purpose: the watcher fires from the `swap!` of the log record, inside the logging path. With `::l/sync?` (`app.binfile.v1`, `app.msgbus`) an exception there would reach whoever was logging; on the async path it escapes to the logging executor worker, whose trace goes to the thread's uncaught handler -- outside the application's logging -- and the pool replaces the dead worker, so logging continues and that line is lost. The catch also makes a publication that fails every time invisible: `penpot_log_events_total` stays flat and nothing is written about it. The published `level` values are lowercase (`warn`), unlike the `level` label of the Loki logs (`ERROR`), so a panel that moves from one source to the other has to change it. Cardinality is `logger` (fixed names taken from the code) times `level`; a logger built from a dynamic value would break that.
|
|
||||||
|
|
||||||
## Storage and media
|
|
||||||
|
|
||||||
- The `media-storage-bytes-per-team` quota check on `upload-file-media-object` is TEMPORARILY DISABLED (2026-10-19, kept commented in `app.rpc.commands.media` with its `sql:get-team-id-for-file` lookup still active for the audit props): the quote performs a full `SUM` over `storage_object` per upload and was too slow in production. The `upload-sessions-per-profile` quota is unaffected and stays active. Re-enable only with a cheaper approximation.
|
|
||||||
- Storage abstraction, backend configuration, logical buckets, object lifecycle, deduplication, access rules, and garbage collection: `mem:backend/storage`.
|
|
||||||
- SVG validation strips DOCTYPE and uses secure SAX parsing. Basic SVG info falls back to 100x100 dimensions when width/height/viewBox are missing.
|
|
||||||
- The SVG sanitizer (`app.media.svg`) is a **blocklist**, not an allowlist: it drops dangerous elements and attributes by name and escapes every text and attribute value when it writes the tree back, so a payload hidden in a value cannot become live markup again. The allowlist rework is [#12190](https://github.com/penpot/penpot/issues/12190) and lands in media-processor, not here. `sanitize-svg` also tracks the namespace bindings through the recursion and drops any element that is not in the SVG namespace, so a document whose root is not an `svg` element in the SVG namespace is rejected and a child that rebinds `xmlns` to XHTML (`iframe`/`srcdoc`/`meta`) is dropped with its subtree; it compares the scheme of an URL the way the browser does (dropping tab/CR/LF/spaces), so `java	script:` does not slip through the `javascript:` filter. The paths that store an uploaded SVG as its final object go through `app.media.svg/sanitize-svg-upload` (media upload, the URL download, both thumbnail writers and the organization logo); the three binfile imports go through `app.binfile.common/sanitize-imported-svg`, which calls `sanitize-svg` directly. The profile picture is not an SVG: `import-profile-picture` ignores an `image/svg+xml` download, because the avatar is always rendered as an image. The caller checks the size before calling it: the helper reads the whole file, and the size limit applies to the upload, not to the sanitized copy, which can be up to six times bigger.
|
|
||||||
- Upload handlers run the `media.v` checks in a fixed order: `check-upload-path`, then `check-media-type`, then `check-media-size` (or `check-font-size` for fonts). The path check gates every later read: `check-media-type` sniffs the bytes at `:path` and `check-media-size` reads `fs/size`, so a path the client named outside `allowed-upload-prefixes` must be rejected before either touches the disk. `check-media-size` leans on that and answers a file it cannot read with the path code (`:params-validation`), not `:media-max-file-size-reached`. Keep the order when touching or adding a handler; two branches that reorder these lines conflict.
|
|
||||||
- Raster metadata is shell-derived with ImageMagick `identify`, verifies detected MIME against the supplied MIME, and swaps dimensions for EXIF orientations 6/8.
|
|
||||||
- Remote image download requires 2xx status, `content-length`, a known MIME, and size under the configured maximum before writing the temp file; mismatched byte count is an internal error.
|
|
||||||
- Font processing shells out to FontForge and WOFF conversion tools and can derive TTF/OTF/WOFF variants from uploaded fonts.
|
|
||||||
|
|
||||||
## File data persistence
|
|
||||||
|
|
||||||
- File data backends for writing are `db` and `storage`; `legacy-db` is a read-only alias for old payloads. The storage backend keeps encoded file data in storage bucket `file-data`; the DB row stores nil data and points at the object with the `storage_object_id` column (FK `NO ACTION DEFERRABLE`).
|
|
||||||
- `fdata/upsert!` touches the storage object the persisted row pointed at before replacing it; the pointer travels in the params (`file->file-data-params` from the read file map, `nil` for a new row, the row for the offload) and the params of `update-file!` cannot override it. `binfile.common/update-file!` merges the read file map with the fields to change, so the pointer survives; `insert-file!` drops it because an INSERT never replaces a persisted object. The column is the only source of truth and the metadata no longer carries `storage-ref-id` (the migration removes it from the backfilled rows and the write path never stores it); the release is a coordinated restart when the storage backend is in use (the previous code cannot read the new rows), while the default `db` backend reads the same data column and tolerates a rolling deploy. Responses built from a file map go through `bfc/file->response`, which strips the pointer and a residual `storage-ref-id` from the metadata.
|
|
||||||
- Pointer-map fragments are persisted separately as type `fragment`, and only modified pointer maps are written. A pointer keeps its id while nothing changes; content (`assoc`/`without`) and metadata (`with-meta`) changes produce a fresh id, so a fragment row is never rewritten after it is written. Re-assoc with an identical value returns the same map in array and hash maps alike, so a save keeps the fragment id of every page it does not change.
|
|
||||||
- `enable-objects-map` skips unloaded pointer pages: their fragment was already wrapped when it was written, and loading it only to wrap it again costs one page read per untouched page on every save. A page with a plain fragment is wrapped when it is first loaded or edited; a data migration that enables `fdata/objects-map` must rewrite the data itself.
|
|
||||||
- Snapshots share fragment rows with the live file (they are stored with pointers, not materialized). `file-gc` marks every fragment referenced neither by the current main file nor by an unpurged snapshot; the file is read unrealized and unchanged pages keep their ids via reattach, so a pass over an unchanged file writes no new fragments.
|
|
||||||
- A save attaches its automatic snapshot before applying pending file migrations: the snapshot keeps the pre-migration pointers, data and migration set, and restoring it leaves the next read to migrate the file. This also holds for files without the feature. The preview (`get-file-snapshot`) applies the pending migrations in memory with the live file's libraries and never persists them: the stored version keeps its data, the payload carries the snapshot's `:migrations` (an ordered set) and `fdata/objects-map` is realized for clients that do not declare it. The historical feature set is deliberately not checked against the enabled one: a version may predate a feature change. The window is narrow: the `get-file` RPC reads with `migrate?` on and its ETag hashes `available-migrations`, so a post-deploy read cannot be a 304 from the pre-deploy state and migrates before any save in the normal flow; the pre-migration snapshot only appears when the save is the first request to touch the file after the deploy (an open tab that saves without re-reading, or a non-UI writer). `create-file-snapshot` and the restore backup read with `migrate?` on, so those versions are stored already migrated.
|
|
||||||
- `:migrations` is always a `d/ordered-set`: `file-migrations/resolve-applied-migrations` builds one, `fmg/migrate` re-wraps the union after applying, and `file-snapshots/decode-snapshot` converts the `file_change` pgarray (as a plain vector, `set/difference` and `contains?` would compare indexes instead of names). A snapshot with NULL `:migrations` and an old `:version` predates the `file_change.migrations` column (SQL migration 0140): the preview regenerates the set from the stored version and applies that full chain, while `restore!` uses a fixed 67 baseline, so preview and restore can disagree for very old snapshots.
|
|
||||||
- The migration read path (`binfile.common/get-file*`) reads the `file` row once (`sql:get-file`, with its data columns; with the storage backend the blob lives in the object, not in the column) plus an explicit `resolve-applied-migrations`, because a row read does not resolve them. A read with no pending migration is completed by `attach-file-data` on that map: one row read and one migrations resolution. Only when a migration is pending is the row re-read under `FOR UPDATE` on the `file` row, where `need-migration?` is re-checked on the fresh map and the migrated file persisted (`binfile.common/migrate-file`); that re-read resolves the migrations again, and `migrate-file` once more after persisting, to return the applied set. The migration path holds two raw copies of the blob (the first read and the locked re-read) but a single decoded one: the alternative, a row-only probe before the read, avoided the first raw copy at the cost of an extra row read on every open, and was removed. `get-file` runs the read inside `db/tx-run!` (joining the caller's transaction when the cfg carries a connection) so the persist commits with the read; a read-only read (`:read-only?`) takes no lock and writes nothing, so it runs on a plain connection (`db/run!`) instead of opening a transaction of its own, and with a cfg that already carries a connection it still joins the caller's. It accepts a cfg map only and throws an `IllegalArgumentException` otherwise (`assert` is elided without `:backend-asserts`); a data read rejects `:lock-for-share?` (only row-only reads accept it), because the migration branch re-reads with `FOR UPDATE`, and a lock option needs a read-write transaction from the caller because PostgreSQL rejects row locks in a read-only transaction. A row-only read (`load-data?` false) resolves the applied migrations by default too (`:load-migrations?`), because the row can be completed later and the caller may need them; a caller that does not want them passes `:load-migrations? false`. Persisting is an optimization of the read (the read already returns migrated data): a row that changed between the first read and the locked read is not overwritten, and a pending migration discovered on the unlocked read is applied in memory only (the next read persists it under the lock). This was the only writer of the file row outside the lock protocol (save, library absorption, restore, `file-gc` and offload already take it); without it, a save that committed in between was overwritten by the stale map (lost content, `revn` moving back so the next save fails with `:revn-conflict`, and the object the save left as a frozen orphan). `migrate-file` captures `fdata/original-pointers` before realizing, reattaches unchanged positions after applying the migrations, and persists the reattached file; it returns that reattached map when it persists (the pages the migration does not change keep their pointers and fragment ids) and the realized map when it does not, so a read is plain only when the caller asks with `:realize?`, and a migration that adds `fdata/objects-map` skips the reattach so the pages get wrapped.
|
|
||||||
- `fdata/load-pointer` reads rows that a previous `file-gc` pass marked on purpose (explicit `::db/remove-deleted false`): the fragment is only gone once `objects-gc` purges it. The `get-file-fragment` RPC keeps rejecting marked rows.
|
|
||||||
- Bind the fragment loader through `fdata/with-pointers` (reads), `fdata/with-tracked-pointers` (writes with a loader) or `fdata/with-tracked` (writes with nothing to load yet, e.g. file creation), and `fdata/with-untracked` (validation passes that must track nothing) instead of hand-rolled `binding` forms. Two direct bindings remain on purpose: the fine-grained tracked toggles of `app.rpc.commands.files-update` (validation/repair turn tracking off, restore the existing atom and read it before repairing), and `binfile.common/encode-file` (a fresh tracked set inheriting the current one). Converge them only if atom-scoped macros (`with-tracking`, `with-inherited-tracking`) are added; today the extra indirection buys nothing.
|
|
||||||
- Snapshot creation takes `FOR UPDATE` on the file row: both `create-file-snapshot` and the backup read of `restore-file-snapshot` hold it from the read to the insert, so they serialize with a `file-gc` pass (`FOR UPDATE ... SKIP LOCKED`) and with a concurrent save, and never pin fragments the pass is about to mark. The lock is exclusive and not `FOR SHARE` because the migration-on-read path may re-read the same row with `FOR UPDATE`: a shared lock that asks for the exclusive one deadlocks two concurrent readers (one creating a snapshot, the other saving, both seeing a pending migration). `get-file-snapshot` realizes the snapshot pointers with the live file id (fragment rows are keyed by file, not by snapshot).
|
|
||||||
- `fdata/realize` combines pointer realization and object-map realization. Use it before operations that need complete in-memory file data instead of pointer placeholders.
|
|
||||||
- A `file_data` row is one of three classes, told apart by `type`: `main` is the file's current state and carries `id = file_id`; `snapshot` is a point-in-time copy whose `id` is the one its `file_change` row shares; `fragment` is a piece of the pointer-map, referenced from the file's data.
|
|
||||||
- Their lifecycles differ: `main` and `snapshot` live with the file (dated when it is deleted, undated when it is restored), while a `fragment` carries no date and is collected by `file_gc` once the file stops referencing it. Neither is a session buffer, so do not read a `fragment` as an ephemeral revision.
|
|
||||||
- File snapshots (versions) keep their metadata row in `file_change` and their payload in `file_data` (`type = 'snapshot'`). With the `db` backend `file_change.data` stays NULL: that column is only read by the `legacy-db` backend (it is no longer written), plus the rows written before the 2025-07 file storage refactor.
|
|
||||||
- Tell a snapshot apart by `label IS NOT NULL` (with `created_by` / `deleted_at` as needed), never by `file_change.data IS NOT NULL`: the label holds with every file data backend, and the payload column stopped meaning "this is a snapshot" when the payload moved out. Queries that identify versions must use the label. `0157-migrate-legacy-file-data` moved the labelled leftovers to `file_data` and deleted the unlabelled payload rows, so from 2.19 on the label is the only marker there is.
|
|
||||||
- The version quota counts of `app.rpc.quotes` (`sql:get-snapshots-per-file`, `sql:get-snapshots-per-team`) are served by the partial index `file_change__user_snapshots__idx`, whose predicate mirrors their where clause; changing one without the other drops the index only scan and the count walks the whole table again.
|
|
||||||
- Revisions live only in `file_change` and have no `file_data` counterpart. They are born already scheduled for deletion one hour later (`app.rpc.commands.files-update`) and the snapshot paths cannot reach one, because they resolve their row through a `label IS NOT NULL` filter.
|
|
||||||
- A deletion date is assigned once and never moved: `restore-files` (`app.rpc.commands.files`) unmarks only durable history, which is `label IS NOT NULL` (the unlabelled rows that still carry a legacy payload are removed by `0157-migrate-legacy-file-data`, so restore leaves them dated), the `:file` method of `app.tasks.delete-object` dates only rows that have no date yet, and `objects-gc` collects only rows that carry one. Unmarking a revision leaves it invisible to `objects-gc` for ever, because nothing sets that date again.
|
|
||||||
- Binfile v1/v3 import and the v3 export `get-file` step run inside one long transaction. They raise `idle_in_transaction_session_timeout` to 30 min (`app.binfile.common/transaction-idle-timeout-ms`) instead of disabling it, so a connection left idle while the ZIP is decoded or the file data is realized is bounded; `statement_timeout` stays at 300 s.
|
|
||||||
- `update-file` with `:changes-with-metadata` groups that all carry `:commit-id` is partial: fast path applies and validates everything once; on a change failure it bisects the commits for the longest prefix that applies and validates (each attempt applies a prefix from the original data and runs referential integrity + file schema; ~log2 K attempts, none for a single commit, the first attempt's error reused), keeping that prefix (a commit that repairs an earlier one can keep both) and answering 200 with `:rejected {:commit-ids :failed-commit-id :error}`, plus `:preexisting-damage true` when the data the first rejected commit touches already fails validation without it (`preexisting-damage?`: one more referential + schema check of its pages/components on the base file, the accepted prefix or the original; the frontend then says the page data is damaged). Only accepted changes reach `file_change`, persistence and the msgbus; a fully rejected request writes and broadcasts nothing (revn unchanged). Flat `:changes` or groups without ids stay all-or-nothing. Before each attempt (`apply-prefix`, which returns `{:file :tracked :refs}` or the cause) `pmap/*tracked*` and the `cpc/*state*` atom are reset to their pre-attempt values (at the end `apply-commits` sets those of the kept prefix): a pointer map already `modified?` keeps its id when changed again, so the failed version would be persisted. DB/IO/interrupt errors and `:not-found` fragments (`change-failure?`) fail the whole request, never reject a commit. A rejection whose ex-info has no `:type` reports `:internal :unexpected-error`. `app.http.errors` answers 503 (`:server-error` + code) for transient SQLSTATEs (classes 08/53, 57P01–3, 40001, 40P01), `SQLTransientException`, `InterruptedException` and only the `IOException`s that pass on their own (`transient-io-exception?`: `SocketException`, `InterruptedIOException` incl. socket timeouts, `UnknownHostException`, `HttpTimeoutException`), which the client's save retries (and GETs retry by themselves); any other `IOException` (EOF from a corrupt blob, Jackson parse errors, missing files) stays 500 `:io-exception`, as do other SQL errors. With the `:debug-save-faults` flag, a `:set-plugin-data` file change in namespace `:penpot-debug` keyed `save-fault:<kind>` raises that fault in `apply-changes` (used by `debug.failNextSave`/`commitInvalidChange`). With a schema check running (hard schema validation without skip-validate, or soft) and 2+ commits, the touched pages' objects are decoded on the original file first (`decode-objects`; referential validation alone reads few objects, so no pre-decode then): objects-map caches decoded values per instance and a derived map copies the cache, so otherwise every attempt re-decodes the page (~260 ms at 10k shapes). Validation step timings (`with-validation-timing`, upstream `penpot_rpc_update_file_validation_timing` + the `update-file` debug log) come from the first attempt only (retries run with a cfg without metrics and timings); the whole bisection is one `retry` step sample, and the rejection log carries `:attempts`/`:elapsed`.
|
|
||||||
@ -3,34 +3,9 @@
|
|||||||
JVM `clojure.test` (kaocha runner) under `backend/test/backend_tests/`.
|
JVM `clojure.test` (kaocha runner) under `backend/test/backend_tests/`.
|
||||||
|
|
||||||
- READ `mem:testing` FIRST — it defines the execution discipline (no piping, tee to file, preferred commands) that applies to all JVM test runs.
|
- READ `mem:testing` FIRST — it defines the execution discipline (no piping, tee to file, preferred commands) that applies to all JVM test runs.
|
||||||
- Test DB/Redis come from the `PENPOT_TEST_*` env vars already set in the shell (`PENPOT_TEST_DATABASE_URI`, `PENPOT_TEST_REDIS_URI`, ...); per instance they point at `penpot_test` / `penpot_test_wsN` and valkey DB `6+N` (`mem:devenv/core`). Do not override them with values copied from CI logs or other sources unless the user asks (e.g. "use a temporary database"); the CI env (`redis://redis/1`, `postgresql://postgres/penpot_test`) only works inside the CI docker network.
|
|
||||||
- The test database may have migrations applied from newer versions (it is shared dev infra). Reinitializing it — `dropdb`/`createdb` of the `PENPOT_TEST_*` database only — is explicitly permitted when that blocks a test run; never drop the main `penpot` database or another instance's test database.
|
|
||||||
- All CLI commands must be executed from the `backend/` subdirectory.
|
- All CLI commands must be executed from the `backend/` subdirectory.
|
||||||
- Tests are invoked directly via `clojure -M:dev:test` (kaocha) — there is no pnpm wrapper. Kaocha auto-discovers test namespaces, so no runner registration is needed.
|
- Tests are invoked directly via `clojure -M:dev:test` (kaocha) — there is no pnpm wrapper. Kaocha auto-discovers test namespaces, so no runner registration is needed.
|
||||||
- **Preferred way to run them: `scripts/ci <module> --test`**, which runs lint, tests and format per module and writes a log per task under `.ci-logs/`. It exports `PENPOT_FLAGS` with `enable-backend-asserts` appended, so use it rather than calling `clojure -M:dev:test` by hand.
|
|
||||||
- **Why the flag matters:** `clojure.core/assert` is elided at compile time when `*assert*` is false, and `app.config` only sets that var from the flags when it loads. The test fixture redefines the flags after that, so they never reach it: without the flag the suite runs with every assertion compiled out, and a test that depends on an assert only means something with it set. `scripts/ci` and `tests-backend.yml` export the flag (GitHub Actions sets it too). When running the suite directly — including a `--focus` run — the command is `PENPOT_FLAGS="enable-backend-asserts" clojure -M:dev:test`.
|
|
||||||
- Coverage: if code is added or modified in `src/`, corresponding tests in `test/backend_tests/` must be added or updated.
|
- Coverage: if code is added or modified in `src/`, corresponding tests in `test/backend_tests/` must be added or updated.
|
||||||
- Isolated run: `clojure -M:dev:test --focus backend-tests.my-ns-test` for a specific test namespace, or `clojure -M:dev:test --focus backend-tests.my-ns-test/my-test-var` for a specific test var. Keep the `PENPOT_FLAGS="enable-backend-asserts"` prefix on these too; a focused run without it hides assert-dependent failures exactly like a full one.
|
- Isolated run: `clojure -M:dev:test --focus backend-tests.my-ns-test` for a specific test namespace, or `clojure -M:dev:test --focus backend-tests.my-ns-test/my-test-var` for a specific test var.
|
||||||
- A **whole-suite** crash reported as `Execution error ... Assert failed: ...` with no `FAIL in` line is not a test failure: an `ig/assert-key` ran while the test system was being built, which happens outside `clojure.test`, so kaocha reports it as a load/build error and the summary has no counts. Fix the offending schema (see `mem:backend/subtleties`) rather than hunting for a test.
|
|
||||||
- Multiple focus symbols are supported by REPEATING the flag (kaocha accumulates them, last-wins does not apply): `clojure -M:dev:test --focus backend-tests.ns-one --focus backend-tests.ns-two` — preferred over a shell loop of isolated runs.
|
|
||||||
- Regression run: `clojure -M:dev:test` to ensure no regressions in related functional areas.
|
- Regression run: `clojure -M:dev:test` to ensure no regressions in related functional areas.
|
||||||
- If you need to filter output, tee to a temp file first: `clojure -M:dev:test 2>&1 | tee /tmp/penpot-test-output.txt`.
|
- If you need to filter output, tee to a temp file first: `clojure -M:dev:test 2>&1 | tee /tmp/penpot-test-output.txt`.
|
||||||
- RPC test helpers `command!`/`management-command!` split the data map: qualified keys become server params, unqualified keys become request body params. To inject request-level context (headers, `:app.http/auth-key-id`, ip), pass a map under `:app.http/request` metadata; non-map `IRequest` stubs fall back to a dummy request.
|
|
||||||
- The test system is `main/system-config` + `main/worker-config` with the
|
|
||||||
components that must not run in the background dropped: HTTP, the srepl,
|
|
||||||
cron, the dispatcher and **every** `[<profile> :app.worker/runner]`
|
|
||||||
(`helpers/remove-job-runners`). A live runner of any queue races the tests
|
|
||||||
that drive the loop by hand (`wdisp/run-batch` + `@#'wrkr/run-worker-loop`):
|
|
||||||
it takes the job the test just dispatched and completes it behind the test's
|
|
||||||
back, so the test's own assertions read a state it did not produce. Do not
|
|
||||||
re-add a runner by editing that list by hand; the removal matches by key
|
|
||||||
shape on purpose.
|
|
||||||
- `system-config` and `worker-config` are `def`s: their `(cf/get ...)` calls
|
|
||||||
are evaluated when the namespace loads, against the real environment, while
|
|
||||||
test code calling `cf/get` at run time sees the `penpot-test`-prefixed
|
|
||||||
config. The two can disagree — in the devenv `:tenant` is `devenv-ws0` for
|
|
||||||
the defs and `default` inside a test — so a component built from
|
|
||||||
`worker-config` can talk to a different Redis key prefix than a cfg the test
|
|
||||||
builds itself. The symptom is a component that silently does nothing
|
|
||||||
locally and races in CI (which sets no `PENPOT_TENANT`).
|
|
||||||
- `backend-tests.helpers` has `metrics-stub` (a metrics instance with no collectors, where every `run!` is a no-op), `init-climit` (a concurrency limiter built from an inline config map, with an optional second argument for the metrics instance to publish to) and `with-climit`, the macro that builds one, runs the body and halts it. `init-climit` starts the republisher daemon thread, so a test that builds a limiter without halting it leaks that thread for the rest of the JVM run: use `with-climit`. Log lines are capturable with `add-watch` on `app.common.logging/log-record` (it holds the last record; the watcher sees every one, but only of the levels the logger has enabled: a line the app writes at TRACE never gets there). The logger swaps the record on an executor thread, after the call that logged it returns: keep the watch installed until the record arrives (poll for it) and only then remove it, or the record lands on nobody under load.
|
|
||||||
|
|||||||
@ -1,44 +0,0 @@
|
|||||||
# Backend WebSocket Subscriptions
|
|
||||||
|
|
||||||
## Registry
|
|
||||||
|
|
||||||
- `app.http.websocket/state` holds `{:connections {conn-id → wsp} :by-profile {profile-id → #{conn-id}}}` in **one** atom, so a single `swap!` keeps both consistent; `register-connection`/`unregister-connection` maintain them together and drop a profile from the index once its last connection leaves.
|
|
||||||
- Each connection's own `::ws/state` atom holds its subscriptions (`::file-subscription`, `::team-subscription`). `state` is shadowed by `::ws/state` in every `handle-message` destructuring, so only `on-connect`, `on-disconnect` and the `repl-*` helpers touch the registry — reading the registry from a handler is a bug.
|
|
||||||
- `::profile-id` and `::session-id` on a connection live in the `app.http.websocket` namespace, while `::ws/id`, `::ws/state`, `::ws/output-ch`, `::ws/session-id` live in `app.util.websocket`. Mixing them up yields silently `nil` values (a `nil` `session-id` makes the channel's `:xf` drop **every** message).
|
|
||||||
- The registry is local to one backend instance. The message bus is shared (`mem:prod-infra/core`). Anything that must reach connections owned by another instance has to go through the bus — closing a connection from an RPC handler only ever works in a single-backend deployment.
|
|
||||||
|
|
||||||
## Revocation model
|
|
||||||
|
|
||||||
A subscription is authorized once, at `:subscribe-file`/`:subscribe-team`, and that decision used to be trusted for the whole connection life (GHSA-m53j-2766-6jqw, medium). Two independent mechanisms now bound it:
|
|
||||||
|
|
||||||
- **Announcement (primary).** A mutation that can change access publishes on `app.rpc.notifications/internal-revocation-topic` (`notify-permissions-changed` for one profile, `notify-team-permissions-changed` for one team id — the watcher resolves the members itself, so a large team is one event, not N). The `::revocation-watcher` Integrant component in `app.http.websocket` subscribes and calls `revalidate-profile-subscriptions`, which looks the profile up in `:by-profile` and re-checks every subscription with `file-readable?`/`team-readable?`: existence (the file gate is the same `sql:get-minimal-file` as the `:subscribe-file` gate, made non-throwing via `{:db/throw-if-not-exists false}`; the team gate is `db/get*` with `::sql/columns [:id :deleted-at]` — `deleted_at` must be in the projection or the soft-delete filter is a no-op) AND the `has-read-permissions?` predicates — and `sql:file-permissions` additionally filters `deleted_at` on file, project and team, as do `sql:get-team-id-for-file`/`-project` plus a final team-row check in `resolve-team-id`, so the organization-owner fallback grants nothing through soft-deleted file/project/team. The existence gate closes that fallback (expelled org owner, deleted resource). The event is only the trigger; the fresh check is the decision, which is what keeps a downgrade to `viewer` (still readable) and a non-member organization owner on a live resource from being mistaken for a revocation. Both publishers defer through `db/after-commit`, because the watcher checks on another connection and would otherwise see the pre-commit state. `:subscribe-team` deliberately has no existence gate of its own; only the re-checks add one, matching the file side (user decision, PR #12189 review).
|
|
||||||
- **Interval (safety net).** `start-relay` runs the relay as a `go-loop` that waits on the subscription channel and a one-shot tick channel together via `sp/alts!` with `:priority true` on the tick: when both hold something, the tick wins, so a steady stream of messages can never push the re-check away. The cadence is `subscription-revalidation-interval` (default `default-revalidation-interval`, 5 min, a code constant in `app.http.websocket` and **not** in the `config.clj` default map, same rule as session lifetimes). The deadline is fixed per cadence: a single tick channel covers a whole cadence and message turns reuse it (minting a fresh timeout per turn would both churn timers and starve under load exactly like an in-loop timer does); a new one is minted only after a tick fires. It covers what no announcement can: Nitrate transferring organization ownership (an HTTP call to another service) and rows deleted straight over SQL. Closing the channel ends the loop either way, because a take on a closed channel is always ready with nil — every way a subscription can end closes it, so a finished relay never keeps polling. Cost is three or four queries per subscription per interval (existence row + predicates), plus one Nitrate call per team per interval when the `:admin-console` flag is on (cached 30 s); one `timeout-chan` per subscription per cadence on promesa's default scheduler (one or two threads shared with every other scheduled task) — do not shorten the interval without sizing that. A tick abandoned at teardown (the subscription closed while it was pending) still fires its scheduled close harmlessly. Two places fail open, deliberately: `still-authorized?` (a transient DB error logs and keeps the subscription; `InterruptedException` is rethrown) and the `catch Throwable` in `watch-revocations`.
|
|
||||||
|
|
||||||
Which mutations announce: `delete-team-member`, `leave-team`, `update-team-member-role` (member only), `delete-team`, `delete-file`, `delete-project`, `move-files`, `move-project` (one team event each). Anything not listed here is covered only by the interval. `delete-share-link` deliberately does not: a share-link visitor never passes the websocket subscribe check, because `check-read-permissions!` is called without a `share-id` and that arity only considers team membership.
|
|
||||||
|
|
||||||
`close-file-subscription`/`close-team-subscription` are idempotent: no-op on a mismatched id, and they `dissoc` the subscription so a repeat call and `:pointer-update` do not publish to a dead topic. Closing the channel is what ends the relay (`take!` on a closed channel returns `nil`).
|
|
||||||
|
|
||||||
## Topic and layering
|
|
||||||
|
|
||||||
- `internal-revocation-topic` is a fixed uuid that is never a real resource id, so it cannot collide with a file, team or profile topic — and deliberately not the `uuid/zero` system topic, which is `sp/pipe`d straight into the client output and anything published there reaches every client. No client can subscribe to the internal topic: `:subscribe-file`/`:subscribe-team` only accept ids that already passed a permission check, and `:broadcast`/`:pointer-update` only republish to ids the server already holds.
|
|
||||||
- The topic and the publish helpers live in `app.rpc.notifications`, not in `app.http.websocket`: the command namespaces cannot require the watcher, because `app.http.websocket` already requires `app.rpc.commands.files`/`teams`. The watcher keeps the registry, which is the part it owns.
|
|
||||||
|
|
||||||
## Time and channel traps in this namespace
|
|
||||||
|
|
||||||
- The relay waits with `sp/alts!` over `[tick-ch channel]` with `:priority true`, following the `msgbus.clj` pattern (`start-io-loop` destructures the same `[val port]` shape and compares ports with `identical?`). A take on a closed channel is always ready with nil, which is what ends the loop — and what makes closing the channel from any teardown path prompt even while parked in `alts!`. `sp/timeout-chan` is one-shot: it closes once after the delay, and one abandoned at teardown still fires its scheduled close harmlessly.
|
|
||||||
- `ct/diff a b` returns **b − a** (`Duration/between`), so "time until a future instant" is `(inst-ms (ct/diff (ct/now) future))`.
|
|
||||||
- `ct/duration` unit keywords are singular: `{:millis 150}`, `{:minutes 5}`, never `:milliseconds`/`:days`.
|
|
||||||
- `still-authorized?` takes the check itself: `(boolean (check-access))`. Calling `(boolean pred)` on the closure is always true and silently disables the interval — this exact bug shipped once and no test caught it until the resubscribe-counting test existed. Naming rule here: no `?` for the action that queries and can fail (`check-access`), `?` only for the safe yes/no answer (`still-authorized?`).
|
|
||||||
|
|
||||||
## Tests
|
|
||||||
|
|
||||||
`backend/test/backend_tests/http_websocket_test.clj` drives the real Redis-backed msgbus, so the revocation tests are end-to-end: register the connection in the real registry, run the real command, wait for the subscription to drop, then assert no content arrives. A connection built without `register-connection` is invisible to the watcher, so the watcher-based tests must go through `open-registered-connection`. Interval tests shorten the config via `with-redefs` on `cf/get` rather than waiting minutes. `with-clean-registry` runs as an `:each` fixture, so no test can leak a live relay loop into the next one. `captured-events` forwards the keyword args through to the real `mbus/pub!` — rebuilding them positionally publishes to the topic `:message` with a `nil` payload and silently untests every other notice. Interval tests take access away with a direct row delete, never with a command: a command announces, so the test would pass even with the interval removed. The same rule covers the two counting/fan-out tests: `revocation-closes-subscriptions-that-are-no-longer-authorized` takes access away with `delete-team-membership!` (raw row delete, publishes nothing) because a command announcement lets the live watcher close both subscriptions before the test's own `revalidate-profile-subscriptions` call, and its `(= 2 …)` count then reads 0; `team-event-closes-every-member-subscription` soft-deletes the team row (`sql:team-members` joins `team_profile_rel` and `profile` only, so the watcher still resolves every member) and asserts nothing closed before one synthetic `notify-team-permissions-changed`, leaving the command path to `deleting-a-team-announces-once` and `deleting-a-team-closes-open-subscriptions`. The resubscribe-counting test is the one that proves finished relays stop polling. The soft-delete + org-owner re-check tests pair a direct row soft-delete with `with-redefs` on `nitrate/organization-owner-of-team?`: before the existence gate, the org-owner fallback kept the subscription open on a deleted resource. The soft-deleted-project/team-closes-org-owner-file-subscription tests pin the root fix in `app.rpc.permissions` (`resolve-team-id` refuses dead file/project/team for every caller of the fallback). `watcher-consumes-revocation-events-from-the-bus` takes access away with a direct row delete and drives the watcher with a synthetic bus event — never a command and never a second `notify-*` — so the "still subscribed" assertion cannot race an after-commit hook and a broken command announcement cannot mask a broken watcher. `moving-a-project-closes-source-team-subscriptions` pins the move-project asymmetry: the file subscription drops, the source-team subscription survives (the member is still a member).
|
|
||||||
|
|
||||||
`revocation-watcher-schema-resolves-every-ref` covers the `::revocation-watcher` params schema, which no other test compiles. It existed broken for a while: the three injected keys were written as bare keywords, which is a valid malli *property* position and not a key, so the schema raised `:malli.core/invalid-schema` — invisible unless `PENPOT_FLAGS=enable-backend-asserts` was set, and a crash of the whole system build rather than one test (see `mem:backend/testing` on the flag).
|
|
||||||
|
|
||||||
## See also
|
|
||||||
|
|
||||||
- `mem:backend/auth-permissions-product-domains` for the permission predicates and the not-found semantics that the re-checks reuse.
|
|
||||||
- `mem:prod-infra/core` for why the bus crosses backends and the registry does not.
|
|
||||||
- `mem:frontend/routing-app-shell-subtleties` for the client side: the frontend already reacts to `team-membership-change` by navigating away, which is why revocation needs no new client message.
|
|
||||||
- `.agents/plans/2026-10-07-websocket-subscription-revocation.md` for the original plan, and the sibling `.agents/plans/2026-10-05-leave-team-file-access-cleanup.md` whose hardened permission queries this re-verification depends on (without live team membership required in `sql:file-permissions`, the re-check would keep reporting access for a member who already left).
|
|
||||||
@ -1,15 +0,0 @@
|
|||||||
# Clojure Design Rules
|
|
||||||
|
|
||||||
How to shape a function in this codebase. Each rule came from a review round; do not re-derive from taste.
|
|
||||||
|
|
||||||
- **Preconditions belong to the boundary, not to the core.** A function documents what it needs (hints, docstring) and assumes it. Absence checks live where the value enters the system: the caller that resolves optional objects, `ig/init-key`, the request handler — including `nil` that is optional-by-design (`when-let [cs (some-> server ...)]` at the entry point, as with `ConnectorStatistics` when the server option is off). A `(when (and (some? a) (some? b)) ...)` inside a leaf function means the guard is in the wrong place — if you cannot get `a` or `b`, you should not be calling it.
|
|
||||||
- **A static set of operations is written statically.** With a fixed, small set (four metrics) write the four calls. Do not build a collection and iterate it (`doseq` over a literal vector): it allocates, adds indirection and hides each operation. Use a collection when the set is dynamic (config, registry, input).
|
|
||||||
- **Default to no `!`; existing `!` is legacy being removed gradually.** Do not introduce new `!` names except the allowed cases documented in `mem:clojure/idioms` (add and justify the special case there first). Allowed: mutating public fns in `app.db` (`insert!`, `update!`, `delete!`, `run!`, `tx-run!`, ...) and fns whose contract is to work over transients (stdlib `conj!`/`assoc!`/`dissoc!`/`persistent!` and custom helpers over them). Existing `!` elsewhere (e.g. `emit!`, `send!`, `submit!`, `validate-file!`) stays until its refactor; keep a family consistent in the meantime. Reporting helpers (log, sample-and-publish) never take `!`: the mutation happens in the `!` API they call.
|
|
||||||
- **Functions that obtain data start with `get-` or `fetch-`.** A reader must see at a glance that the function reads (`get-file`, `get-snapshot`, `get-file-json`); a bare noun hides it (`file-json` says nothing about reading). New helpers that fetch by id follow this, never a `current-` prefix: the id parameter already says which one it is.
|
|
||||||
- **`?` marks a predicate, not a data prop.** A var or function that answers a question takes `?` (`export-error?`, `(defn editable? …)`); a map key that names a property does not (`:export-error`, never `:export-error?`), even when its value is a boolean. Existing `?` keys are legacy to remove gradually; do not add a new one.
|
|
||||||
- **Do not shape production code for tests.** No return values added just to assert them; assert on the observable effect. When preconditions move, move the tests with them.
|
|
||||||
- **All code-facing text MUST be in English** — identifiers, comments, docstrings, test descriptions. No exceptions; see the hard rule in `AGENTS.md`.
|
|
||||||
|
|
||||||
Reference implementation: the http metrics samplers in `app.http` (`sample-worker-metrics` / `sample-connector-metrics` are guard-free; `sample-http-metrics` guards at the boundary).
|
|
||||||
|
|
||||||
See also: `mem:clojure/idioms` (language behaviors), `mem:testing` (TDD and test conventions).
|
|
||||||
@ -1,15 +0,0 @@
|
|||||||
# Clojure Idioms (verified)
|
|
||||||
|
|
||||||
Behaviors confirmed against the language/stdlib — do not re-derive from assumption; a wrong assumption here already cost a review round.
|
|
||||||
|
|
||||||
- `int?` is NOT 32-bit-only: true for `Long`, `Integer`, `Short`, `Byte` (fixed-precision integers). Clojure integer literals are `Long`, so `(int? 5000)` is true.
|
|
||||||
- `integer?` is the general integer predicate; prefer it when any integer kind must match, `int?` only when fixed precision is meant.
|
|
||||||
- `await` is a `cljs.core` macro asserting `(:async &env)`: it fails at compile time outside an `^:async` context, never silently.
|
|
||||||
- The analyzer reads `:async` only from the fn name meta and the `fn` operator meta; list-level meta is ignored (would make a MetaFn). `(fn ^:async [] …)` puts the meta on argv (pre/post only, NOT async).
|
|
||||||
- Valid: `(defn ^:async f)`, `(defn- ^:async f)`, `(^:async fn [] …)` (the latter is what stock `t/async` generates itself).
|
|
||||||
- `^:async` fns never throw synchronously (rejected promises instead); `try/catch/finally` supported; continuations are microtasks, timers and RxJS schedulers are macrotasks (FIFO).
|
|
||||||
- Inside an `^:async` fn, a nested call that needs a temp binding (e.g. calling a local fn `(g (f x))`, whose arg gets a `G__` binding) compiles to `(await (async function(){…})())`: `await` unwraps a returned promise, so `(settled (upload (make-ctx)))` blocks until the upload promise settles instead of passing the promise on (a test that pushes the answer later hangs, idle). Bind each step in its own `let` binding.
|
|
||||||
- `!` suffix allowed cases (only exceptions to the no-new-`!` rule in `mem:clojure/design-rules`): mutating public `app.db` API (`insert!`, `update!`, `delete!`, `run!`, `tx-run!`, `exec!`, ...) and fns whose contract is to work over transients (stdlib `conj!`, `assoc!`, `dissoc!`, `disj!`, `persistent!` and custom helpers over them); any further case must be registered here with justification before use.
|
|
||||||
- Penpot extends `clojure.core/Inst` in `app.common.time` ( JVM-only section), so `inst-ms` also accepts `Duration` (returns `.toMillis`, NOT epoch millis), `FileTime` and `OffsetDateTime`. It only works where that namespace is loaded; on a bare `Duration`, prefer explicit `.toMillis`, which reads as what it is (an amount, not an instant).
|
|
||||||
- The threading macros insert the threaded value at DIFFERENT positions: `->` and `some->` put it FIRST (`(-> x (f a))` → `(f x a)`), `->>` and `some->>` put it LAST (`(->> x (f a))` → `(f a x)`). `some->`/`some->>` also short-circuit on nil. With unary calls both nest the same way (`(-> x (f) (g))` and `(->> x (f) (g))` are both `(g (f x))`): what changes between them is the position inside each call, not the order of the steps. A pipeline that calls a fn whose first argument is a config/context (`app.features.fdata/resolve-file-data` is `(cfg, file)`) must use `->>`/`some->>`; with `->`/`some->` the arguments are swapped and the call silently returns the wrong value (it cost a broken `get-snapshot` in `app.features.file-snapshots`).
|
|
||||||
- `cuerdas.core/index-of` returns `nil` (not `-1`) when the substring is absent, like `clojure.string/index-of`. `(pos? nil)` **throws on the JVM** but evaluates to `false` in CLJS (JS coercion), so `(if (pos? (str/index-of s "\n")) …)` works in the browser and crashes the backend. Guard with `some?` before `pos?` (this was `app.common.exceptions/first-line`).
|
|
||||||
@ -1,7 +0,0 @@
|
|||||||
# Clojure Logging Conventions
|
|
||||||
|
|
||||||
Structured logging through `app.common.logging` (`l/trc`/`l/dbg`/`l/info`/`l/wrn`/`l/error`/`l/fatal`): `:hint` carries the human-readable message, every other datum goes in its own keyword field (`:id`, `:container`, `:removed`, ...).
|
|
||||||
|
|
||||||
- `:hint` values start lowercase: `"fix dangling children"`, not `"Fix dangling children"`. Uppercase-first hints exist as legacy (~57 occurrences, the densest in `backend/src`; also `l/trc` in migration `legacy-28` and `log/error` in `app.common.files.repair`) — do not copy them, and do not mass-rewrite them unless the surrounding code needs the change.
|
|
||||||
- A name that requires uppercase (an env var like `PENPOT_MEDIA_PROCESSING_SERVICE_URI`, an acronym like `MCP` or `LDAP`) stays uppercase inside an otherwise lowercase sentence.
|
|
||||||
- Repair work done by a file migration logs at `l/trc`, only when it changes something, and with the ids it touched: the house style of `migrations.cljc` (`legacy-28` precedes; the file has no `warn`/`info` lines). The shipped `log4j2.xml` pins the `app` logger to `info` (static file, `monitorInterval="60"`), so those trace lines appear only after the operator raises the level; the durable record of a repaired file is the migration id in its `:migrations`.
|
|
||||||
@ -1,9 +0,0 @@
|
|||||||
# Promesa
|
|
||||||
|
|
||||||
Promise library of the monorepo (frontend CLJS and common, `funcool/promesa` 12.0.1 in `common/deps.edn`). Its API has two call styles and mixing them in one expression is the trap: `fmap`/`mcat`/`merr` are **fn first, promise last** (`->>` friendly) and `then`/`then'`/`catch`/`handle` are **promise first, fn last** (`->` friendly). Convention: `fmap`/`mcat`/`merr` inside `->>` pipelines, `then`/`catch` with `->`. A `then` written inside a `->>` reads backwards and hides whether the step carries a value or a promise.
|
|
||||||
- `fmap` (alias `map`, both `(f p)`) is functor map: `f` returns a plain value.
|
|
||||||
- `mcat` (alias `mapcat`, both `(f p)`) is monadic bind: `f` must return a promise, and the result mirrors it.
|
|
||||||
- `then` also unwraps a promise returned by `f`, which is why it covers both cases at the cost of not naming which one it is; `then'` maps without unwrapping.
|
|
||||||
- `merr` (fn first) and `catch` (promise first) are the error side.
|
|
||||||
- `all` takes a collection of promises (a lazy seq works); `let` chains sequential bindings.
|
|
||||||
- Seam with the rx pipelines of the frontend: `beicon.v2.core/from` builds an observable from "arrays, collections, and promise instance" (`external/beicon`), so `(->> p (p/mcat f) (rx/from) (rx/map g))` is the usual bridge (`app.main.data.imports`, `app.util.zip`).
|
|
||||||
@ -26,12 +26,6 @@ Respect the intended abstraction direction in new/refactored code:
|
|||||||
|
|
||||||
Some legacy code violates this layering; do not copy those violations into new code when a focused refactor is practical.
|
Some legacy code violates this layering; do not copy those violations into new code when a focused refactor is practical.
|
||||||
|
|
||||||
## Build-mode dependent laziness
|
|
||||||
|
|
||||||
`app.common.data.macros/lazy-delay` returns a delay whose expression is evaluated eagerly by default. Only the builds that set the `app.common.data.macros/lazy-delay?` closure define to `true` defer it to the first deref; every production build does (frontend `main` and `worker`, exporter, library), so the Closure compiler replaces the constant and drops the eager branch. On the JVM it is always eager.
|
|
||||||
|
|
||||||
`app.common.schema/check-fn` resolves its schema through it: an invalid schema definition fails when the namespace loads in development and in tests, and only on the first check call in production. The other lazily built parts of `check-fn` (validator and explainer) are always deferred.
|
|
||||||
|
|
||||||
## Focused memory routing
|
## Focused memory routing
|
||||||
|
|
||||||
Model, schema, and persistence shape:
|
Model, schema, and persistence shape:
|
||||||
|
|||||||
@ -6,7 +6,6 @@
|
|||||||
- Do not treat nil as a distinct persisted state from absence. Import/export and cleanup paths may filter nil attrs away.
|
- Do not treat nil as a distinct persisted state from absence. Import/export and cleanup paths may filter nil attrs away.
|
||||||
- Avoid Clojure-special naming in exported object attrs, especially boolean names ending in `?`; exported/imported data must survive JSON/SVG/Transit and external tooling.
|
- Avoid Clojure-special naming in exported object attrs, especially boolean names ending in `?`; exported/imported data must survive JSON/SVG/Transit and external tooling.
|
||||||
- Any new shape attr that participates in component sync must be listed in `app.common.types.component/sync-attrs` with the correct touched group. Attrs absent from `sync-attrs` are ignored by component synchronization.
|
- Any new shape attr that participates in component sync must be listed in `app.common.types.component/sync-attrs` with the correct touched group. Attrs absent from `sync-attrs` are ignored by component synchronization.
|
||||||
- Shape `:svg-attrs` keys are kebab-case (`fill-rule`, not `fillRule`): birth (`app.common.svg/attrs->kebab-props`, applied everywhere data lands on a shape) normalizes any outside spelling through the `svg-props` whitelist, and render boundaries (`get-svg-props`, `render-xml*`, `svg-node*`) translate back to camelCase for React. `:svg-defs` node attrs follow the same rule recursively; svg-raw `:content` attrs are kebab-ized spelling-only, never filtered — the sidebar reads `:stroke-style`, which is outside the whitelist. Non-spec attrs Penpot keeps live in `penpot-extra-attrs` (seeded with `:stroke-style`) and travel in source spelling. Excluded, each with its reason: `:tag` values (React and tag sets need them verbatim; the schema decoder restores keywords from stored strings), outer `:svg-defs` keys (id strings at birth, keywords or strings after JSON decode — accept both), all values (the kebab reader never touches values). Repair sites: birth plus migration `0035-fix-shape-svg-attrs-and-defs` (replaces the withdrawn `0021`); the binfile cleaner carries no svg repair.
|
|
||||||
|
|
||||||
## Cross-module update checklist
|
## Cross-module update checklist
|
||||||
|
|
||||||
|
|||||||
@ -52,8 +52,3 @@ its own drift. Comparing `:selrect.width` from shape A to
|
|||||||
`:selrect.width` from shape B is comparing two values that
|
`:selrect.width` from shape B is comparing two values that
|
||||||
"semantically" should be equal but were computed via different
|
"semantically" should be equal but were computed via different
|
||||||
operation chains — exact equality will often be false.
|
operation chains — exact equality will often be false.
|
||||||
|
|
||||||
## Library sync comparison
|
|
||||||
|
|
||||||
- `update-attrs` (`app.common.logic.libraries`) compares main vs copy attrs exactly first, then with tolerance for: `drifting-geometry-attrs` (`:x :y :width :height :selrect :points`, via `gsh/close-attrs?`); path and bool `:content` (`segment/content-close?`; bool shapes keep their path in `:content`); and text `:position-data` (`position-data-close?` over `mth/close-map?`: every numeric field of every entry within `mth/close?`, other fields such as fills, fonts and text exact). All other attrs (`:opacity`, radii, `:transform`, `:rotation`, text `:content`) stay exact: tolerance there hides real changes.
|
|
||||||
- `close-attrs?` pairs lists with `d/zip`, which truncates to the shorter list and treats nil vs vector as equal for `:points`/`:position-data`. Do not use it for variable-length data. That is why `:position-data` has its own helper, and why `attr-equal?` checks the `:points` count before calling it.
|
|
||||||
|
|||||||
@ -20,10 +20,7 @@
|
|||||||
|
|
||||||
- Full referential/semantic validation currently runs only when file features contain `"components/v2"`.
|
- Full referential/semantic validation currently runs only when file features contain `"components/v2"`.
|
||||||
- Validation starts at root plus orphan shapes, then validates component records. `validate-file!` raises `:validation :referential-integrity` with collected details.
|
- Validation starts at root plus orphan shapes, then validates component records. `validate-file!` raises `:validation :referential-integrity` with collected details.
|
||||||
- `update-file` validates per save with the change-scoped variants: `validate-file-affected!` (referential, whole touched pages/components) and `validate-file-schema-affected!` (file schema with `:pages-index` reduced to the touched pages; non-page data checked whole). The schema variant maps change types to pages itself (`touched-page-ids`): a new change type that writes a page must be added there, or to `file-level-change-types` if it touches no page; an unknown type makes it check the whole file. The whole-file schema check loads and decodes every page, ~75–80% of a save on multi-page files.
|
|
||||||
- `repair-file` does not mutate data directly; it reduces validation errors into redo changes using `changes-builder`. Callers must apply or persist those changes.
|
- `repair-file` does not mutate data directly; it reduces validation errors into redo changes using `changes-builder`. Callers must apply or persist those changes.
|
||||||
- `repair-file-until-valid` stops on: no errors; unchanged `:data` after apply (backend page/`PointerMap` `=` is pointer identity); stable error fingerprint `(frequencies (map #(dissoc % :shape) errors))` vs previous pass (check fingerprint before `repair-file`/`process-changes`); or `max-iterations`. Each `repair-error` builds from the original `file-data`, so N `:set`-based errors on one shape take N passes.
|
|
||||||
- `:ref-shape-not-found` repair: when the copy root's local component has a `:main-instance-id` that resolves to no shape but a `:main-instance` shape of that component sits on its `:main-instance-page`, it emits `mod-component` pointing the id back to that main (keeps the copy linked; a repair scoped to the copy's page never sees the main's `:invalid-main-instance-id`). Otherwise near-main reassign, else detach the copy subtree.
|
|
||||||
- `:missing-slot` fires only for a REAL swap: a copy sub-head whose `shape-ref` is no longer a child of the near main parent. A pure positional mismatch (ref still a sibling elsewhere) is a reorder — valid, realigned by the async component sync; do not "repair" it by assigning swap slots (a slot freezes the child out of normal sync). `fix-missing-swap-slots` (migration 0019) follows the same membership rule.
|
- `:missing-slot` fires only for a REAL swap: a copy sub-head whose `shape-ref` is no longer a child of the near main parent. A pure positional mismatch (ref still a sibling elsewhere) is a reorder — valid, realigned by the async component sync; do not "repair" it by assigning swap slots (a slot freezes the child out of normal sync). `fix-missing-swap-slots` (migration 0019) follows the same membership rule.
|
||||||
|
|
||||||
## Migrations
|
## Migrations
|
||||||
@ -32,6 +29,3 @@
|
|||||||
- Migrations are an ordered set mixing legacy version-derived ids and newer named ids. Keep append order stable; `migrate` applies the set difference between available migrations and file migrations.
|
- Migrations are an ordered set mixing legacy version-derived ids and newer named ids. Keep append order stable; `migrate` applies the set difference between available migrations and file migrations.
|
||||||
- `migrate-file` synthesizes legacy migration ids from old numeric versions when `:migrations` is absent, migrates legacy features, and records feature flags created through `cfeat/*new*`.
|
- `migrate-file` synthesizes legacy migration ids from old numeric versions when `:migrations` is absent, migrates legacy features, and records feature flags created through `cfeat/*new*`.
|
||||||
- When a file had no previous `:migrations`, `migrate-file` marks all migrations as migrated in metadata so callers persist the complete migration set, not only transformations that changed data.
|
- When a file had no previous `:migrations`, `migrate-file` marks all migrations as migrated in metadata so callers persist the complete migration set, not only transformations that changed data.
|
||||||
- Closed schemas (`schema:stroke-attrs` and others) reject a key that was dropped from the declaration, so removing a declared attr needs a migration in the same change, for data already written with it.
|
|
||||||
- Rebase collision on a migration number: upstream keeps the id. Renumber the branch migration to the next free one, keep it last in `available-migrations`, and rename everything that spells the number — its test file and namespace (`files_migrations_00NN_test.cljc`), the test names, and comments in production code or memories that name it.
|
|
||||||
- Withdrawing a migration id: remove both the `defmethod` and the id from `available-migrations`; the replacement migration subsumes it and files that recorded the old id re-migrate through the replacement (persisted `:migrations` keep the old id via union — do not rewrite them, including playwright fixture JSONs). Idempotent/pure-cleanup migrations only; never withdraw one whose transform must not run again unless the replacement is safe for those files. Precedents: "0021-fix-shape-svg-attrs" -> "0035-fix-shape-svg-attrs-and-defs"; "legacy-36" -> "0036-clear-nil-keys-in-objects".
|
|
||||||
|
|||||||
@ -38,10 +38,6 @@ Useful builders:
|
|||||||
|
|
||||||
`add-variant-with-copy` does not accept position params for children; use `gsh/absolute-move` after creation if positions matter.
|
`add-variant-with-copy` does not accept position params for children; use `gsh/absolute-move` after creation if positions matter.
|
||||||
|
|
||||||
## Generative sampling caution
|
|
||||||
|
|
||||||
- `sg/sample` is unseeded per run. If a test filters samples by type (e.g. keep only `:svg-raw`) then `first` can yield nil on unlucky seeds; `assoc` onto nil builds a typeless map that fails `:multi` dispatch on `:type`. Pin `:seed` in `sg/sample` opts whenever a test depends on a filtered subtype being present (verified seed 42 deals svg-raw shapes on JVM and JS).
|
|
||||||
|
|
||||||
## Driving production paths
|
## Driving production paths
|
||||||
|
|
||||||
For shape mutations, prefer production-path helpers such as `cls/generate-update-shapes` plus `thf/apply-changes`. For component swaps with keep-touched behavior, use `tho/swap-component-in-shape` with `{:keep-touched? true}`.
|
For shape mutations, prefer production-path helpers such as `cls/generate-update-shapes` plus `thf/apply-changes`. For component swaps with keep-touched behavior, use `tho/swap-component-in-shape` with `{:keep-touched? true}`.
|
||||||
|
|||||||
@ -1,13 +0,0 @@
|
|||||||
# Common URI helpers
|
|
||||||
|
|
||||||
`app.common.uri` wraps `lambdaisland.uri` and re-exports `join`, `parse`, `query-encode`, `percent-encode` and `uri?` with `dm/export`.
|
|
||||||
|
|
||||||
## `join` resolves references, it does NOT concatenate paths
|
|
||||||
|
|
||||||
- `lambdaisland.uri/join` implements RFC 3986 reference resolution, not path concatenation: a relative reference that does not start with `/` replaces everything after the last `/` of the path it is resolved against. Joining several segments resolves them successively, so each segment replaces the last segment of the previous result.
|
|
||||||
- `(join "https://h/" "api/teams/" "<id>" "users/" "<pid>")` returns `https://h/api/teams/users/<pid>`, not `.../api/teams/<id>/users/<pid>`: `"users/"` replaced `"<id>"`. `(apply join base segments)` behaves the same.
|
|
||||||
- `ensure-path-slash` normalizes only the BASE to end with `/`; it does not touch the segments, so a base ending in `/` is not enough for a multi-segment append.
|
|
||||||
- To append several segments: either join them into a single relative path first (dropping empty segments and collapsing slashes), or make every non-final segment end with `/`, including the dynamic ids. Keep that normalization in the shared URL builder, not in each call site.
|
|
||||||
- A segment starting with `/` resolves against the host root and silently drops the base subpath; reject it.
|
|
||||||
- Frontend `app.config` normalizes the base (`normalize-uri`) and relies on callers passing trailing-slash segments by convention (`resolve-file-media` joins `"assets/by-file-media-id/"` then `"<id>/thumbnail"`); it does not normalize segments.
|
|
||||||
- Backend shared builders are `app.config/join-uri` and `app.config/get-public-uri`. A past refactor moved Nitrate's per-request helper onto them without the segment normalization and broke every multi-segment Nitrate URL that passes `"api/teams/" team-id "users/" profile-id`.
|
|
||||||
@ -3,33 +3,32 @@ You are working on the GitHub project `penpot/penpot`, a monorepo.
|
|||||||
# Memory system
|
# Memory system
|
||||||
|
|
||||||
- Memories are the primary project guidance (not docs or other readme files).
|
- Memories are the primary project guidance (not docs or other readme files).
|
||||||
- A section's top-level memory is `<section>/core`. When a section is relevant, read the core memory before focused memories.
|
- A section's top-level memory is `<section>/core`. When a section is relevant, read the core memory
|
||||||
|
before focused memories.
|
||||||
- Edits/stale refs/duplication cleanup: `mem:memory-maintenance`.
|
- Edits/stale refs/duplication cleanup: `mem:memory-maintenance`.
|
||||||
- Cross-cutting testing principles, TDD workflow, and anti-patterns: `mem:testing`.
|
- Cross-cutting testing principles, TDD workflow, and anti-patterns: `mem:testing`.
|
||||||
- Structured logging conventions (`:hint` casing and fields, log levels): `mem:clojure/logging` — read before adding or changing any log call.
|
|
||||||
- Verified Clojure language behaviors that contradict common assumptions (e.g. `int?` covers `Long`; `integer?` is the general predicate): `mem:clojure/idioms` — read before assuming stdlib predicate semantics.
|
|
||||||
- Read `mem:clojure/design-rules` (function shape, boundary checks, naming) before writing or modifying Clojure/ClojureScript code — not only when designing a solution or an API.
|
|
||||||
- Promises: which call style to use where (`fmap`/`mcat`/`merr` vs `then`/`catch`) and how they bridge to rx: `mem:clojure/promesa` — read before chaining promises or mixing them with observables.
|
|
||||||
|
|
||||||
# Development workflow
|
# Development workflow
|
||||||
|
|
||||||
- Commit/PR/issue creation is **on explicit request only**. Before any of these actions, read the relevant memory — don't infer format from prior examples:
|
- Commit/PR/issue creation is **on explicit request only**. Before any of these actions, read the relevant memory — don't infer format from prior examples:
|
||||||
- Before `git commit` → `mem:workflow/creating-commits` (subject/body format, 80-char body wrapping enforced by `scripts/check-commit`, `AI-assisted-by: model-name` trailer)
|
- Before `git commit` → `mem:workflow/creating-commits` (subject format, body, `AI-assisted-by: model-name` trailer)
|
||||||
- Before `gh issue create` → `mem:workflow/creating-issues` (title derivation, body template, labels, Issue Type)
|
- Before `gh issue create` → `mem:workflow/creating-issues` (title derivation, body template, labels, Issue Type)
|
||||||
- Before `gh pr create` / `gh pr edit` → `mem:workflow/creating-prs` (title format, body structure, "Note:" line)
|
- Before `gh pr create` / `gh pr edit` → `mem:workflow/creating-prs` (title format, body structure, "Note:" line)
|
||||||
- Before updating `CHANGES.md` → `mem:workflow/updating-changelog` (sources of truth, entry rules, the `:boom:`/`:rocket:` label contracts, pre-flight checks, anomaly report)
|
- Before a repo-wide pnpm version update → `mem:workflow/updating-pnpm` (workspace
|
||||||
- Before a repo-wide pnpm version update → `mem:workflow/updating-pnpm` (workspace layout, `scripts/sync-pnpm-version` flow, the stamp-missing-field and ignored-builds gotchas, verification steps)
|
layout, `corepack use` sweep order, the stamp-missing-field and
|
||||||
- Before a Node version update → `mem:workflow/updating-node` (the six pin sites in `.nvmrc`, the devenv and image Dockerfiles and the plugins-deploy workflows, the DHI tag and digest rules, what to verify)
|
ignored-builds gotchas, verification steps)
|
||||||
- Before bumping a non-JS dependency pinned in `docker/images/` (JDK, ImageMagick) → `mem:workflow/docker-images`
|
|
||||||
- **Never `git push`, force-push, or modify `git origin`** (or any other remote). The user pushes from their own shell; if a push is required, say so and wait. Never amend a commit that the user has already pushed unless explicitly asked.
|
- **Never `git push`, force-push, or modify `git origin`** (or any other remote). The user pushes from their own shell; if a push is required, say so and wait. Never amend a commit that the user has already pushed unless explicitly asked.
|
||||||
- **Never edit `CHANGES.md` by hand.** The changelog is generated from GitHub milestones during the release process; update it only via `mem:workflow/updating-changelog` (the `update-changelog` skill flow) or on explicit user request.
|
- **Never edit `CHANGES.md` by hand.** The changelog is generated from GitHub milestones during the release process; update it only via the `update-changelog` skill flow or on explicit user request.
|
||||||
- You have access to the GitHub CLI `gh` or corresponding MCP tools.
|
- You have access to the GitHub CLI `gh` or corresponding MCP tools.
|
||||||
- Issues are also managed on Taiga. Read issues using the `read_taiga_issue` tool.
|
- Issues are also managed on Taiga. Read issues using the `read_taiga_issue` tool.
|
||||||
- Before writing code, analyze the task in depth and describe your plan. If the task is complex, break it down into atomic steps. After making changes, run the applicable lint and format checks for the affected module before considering the work done (per example `mem:backend/core` or `mem:frontend/core`).
|
- Before writing code, analyze the task in depth and describe your plan. If the task is complex, break it down into atomic steps.
|
||||||
- Align `let` binding values: when a `let` form has multiple bindings spanning several lines, align the value forms to the same column with spaces.
|
*After making changes, run the applicable lint and format checks for the affected module before considering the work done (per example `mem:backend/core` or `mem:frontend/core`).
|
||||||
- If you introduce delimiter errors (mismatched parens/brackets) in Clojure/CLJS files, fix them with `scripts/paren-repair` BEFORE running lint/format checks. See `mem:scripts/paren-repair` for usage.
|
- Align `let` binding values: when a `let` form has multiple bindings spanning
|
||||||
|
several lines, align the value forms to the same column with spaces.
|
||||||
|
- If you introduce delimiter errors (mismatched parens/brackets) in Clojure/CLJS files,
|
||||||
|
fix them with `scripts/paren-repair` BEFORE running lint/format checks.
|
||||||
|
See `mem:scripts/paren-repair` for usage.
|
||||||
- Never run anything that destroys data without explicit permission, including `drop-devenv`, `docker compose down -v`, `docker volume rm ...`. The user's real work lives in the volumes of the shared infra.
|
- Never run anything that destroys data without explicit permission, including `drop-devenv`, `docker compose down -v`, `docker volume rm ...`. The user's real work lives in the volumes of the shared infra.
|
||||||
- **Never wait for background work with a sleep.** A long command moved to the background reports its own completion: do not poll it with `sleep 150` (or any other wait) inside a foreground command, and do not re-read its output file to check whether it finished. Keep working on something else, or end the response and wait for the notification.
|
|
||||||
|
|
||||||
# Project modules
|
# Project modules
|
||||||
|
|
||||||
@ -46,31 +45,49 @@ This is a monorepo. Principles that apply to one module do *not* generally apply
|
|||||||
- `docs/`: documentation site; core workflow and conventions: `mem:docs/core`.
|
- `docs/`: documentation site; core workflow and conventions: `mem:docs/core`.
|
||||||
- `media-processor/`: TypeScript/Node.js HTTP service for image (sharp) and font (FontForge) processing; core conventions: `mem:media-processor/core`.
|
- `media-processor/`: TypeScript/Node.js HTTP service for image (sharp) and font (FontForge) processing; core conventions: `mem:media-processor/core`.
|
||||||
|
|
||||||
The memory is structured in a way that you can get the critical information about the module. You can read it from `mem:<MODULE>/core`
|
The memory is structured in a way that you can get the critical information about the
|
||||||
|
module. You can read it from `mem:<MODULE>/core`
|
||||||
|
|
||||||
# Low-centrality project paths
|
# Low-centrality project paths
|
||||||
|
|
||||||
- `docker/` contains devenv related code, not needed unless specifically instructed. When working on devenv startup, compose layout, instance config (`defaults.env`), tmux session lifecycle, RustFS provisioning, or anything in `manage.sh`'s `*-devenv` commands, read `mem:devenv/core`.
|
- `docker/` contains devenv related code, not needed unless specifically instructed.
|
||||||
|
When working on devenv startup, compose layout, instance config (`defaults.env`),
|
||||||
|
tmux session lifecycle, MinIO provisioning, or anything in `manage.sh`'s
|
||||||
|
`*-devenv` commands, read `mem:devenv/core`.
|
||||||
- `experiments/` contains standalone experimental HTML/JS/scripts; treat it as non-core unless the user explicitly asks about it.
|
- `experiments/` contains standalone experimental HTML/JS/scripts; treat it as non-core unless the user explicitly asks about it.
|
||||||
- `sample_media/` contains sample image/icon media and config used as fixtures/demo material; do not infer app behavior from it.
|
- `sample_media/` contains sample image/icon media and config used as fixtures/demo material; do not infer app behavior from it.
|
||||||
|
|
||||||
# Dev Scripts (scripts/)
|
# Dev Scripts (scripts/)
|
||||||
|
|
||||||
- `scripts/nrepl-eval.mjs` — Evaluate Clojure/ClojureScript code via nREPL. Supports `--backend` (port 6064) and `--frontend` (port 3447) aliases. See `mem:scripts/nrepl-eval`.
|
- `scripts/nrepl-eval.mjs` — Evaluate Clojure/ClojureScript code via nREPL.
|
||||||
- `scripts/paren-repair` — Fix mismatched delimiters in Clojure/CLJS files and reformat with cljfmt. Run before lint checks when LLM edits break parens. See `mem:scripts/paren-repair`.
|
Supports `--backend` (port 6064) and `--frontend` (port 3447) aliases.
|
||||||
- `scripts/psql` — PostgreSQL client wrapper with devenv defaults. Companion: `scripts/db-schema` for DDL dumps. See `mem:scripts/psql`.
|
See `mem:scripts/nrepl-eval`.
|
||||||
- `scripts/taiga.py` — Fetch public issues, user stories, and tasks from the Penpot Taiga project without authentication. See `mem:scripts/taiga`.
|
- `scripts/paren-repair` — Fix mismatched delimiters in Clojure/CLJS files
|
||||||
- `scripts/gh.py` — GitHub operations helper: list milestone issues, fetch PR details, compare against CHANGES.md. Requires `gh` CLI. See `mem:scripts/gh`.
|
and reformat with cljfmt. Run before lint checks when LLM edits break parens.
|
||||||
- `scripts/project-anomalies.py` — Main-board anomaly check for a milestone (`check` writes `tmp/<MILESTONE>-ANOMALIES.md`). Bulk resolution via `scripts/gh.py`. See `mem:scripts/project-anomalies`.
|
See `mem:scripts/paren-repair`.
|
||||||
- `scripts/changelog.py` — Changelog checks for `mem:workflow/updating-changelog`: `boom` (generate the `:boom:` section from `breaking change` issues), `check-merged` (PR merge status), `cross-ref` (milestone PRs vs changelog section), `report` (anomaly report to CHANGES-ISSUES.md). Calls `scripts/gh.py` via subprocess. See `mem:scripts/changelog`.
|
- `scripts/psql` — PostgreSQL client wrapper with devenv defaults.
|
||||||
- `scripts/error-reports.mjs` — Query error reports via RPC API with token authentication. Supports list/get operations with filtering and pagination. See `mem:scripts/error-reports`.
|
Companion: `scripts/db-schema` for DDL dumps. See `mem:scripts/psql`.
|
||||||
- `scripts/clean-node-modules` — Remove stale `node_modules` from all pnpm workspaces (root, modules, member packages). Keeps the shared pnpm store at `<repo>/.pnpm-store` unless `--store`; ignores `external/` and `.opencode/`. Usage and reinstall steps: `mem:workflow/updating-pnpm`.
|
- `scripts/taiga.py` — Fetch public issues, user stories, and tasks from the
|
||||||
- `scripts/setup-opencode` — Install the opencode CLI in-container as a global pnpm package in `$PNPM_HOME`. Not part of the devenv image; see `mem:devenv/core`.
|
Penpot Taiga project without authentication. See `mem:scripts/taiga`.
|
||||||
- `scripts/ci` — CI orchestration script: runs lint, tests, and format checks per module, plus the production build on request (`--build`). Modules: `frontend backend common render-wasm exporter mcp plugins library media-processor`. Logs go to `.ci-logs/`; read the log file on failure. It is the preferred way to run the backend tests, and it exports `PENPOT_FLAGS` with `enable-backend-asserts`, which is what makes `clojure.core/assert` exist in the compiled test code. See `mem:scripts/ci`.
|
- `scripts/gh.py` — GitHub operations helper: list milestone issues, fetch PR
|
||||||
|
details, compare against CHANGES.md. Requires `gh` CLI. See `mem:scripts/gh`.
|
||||||
|
- `scripts/error-reports.mjs` — Query error reports via RPC API with token
|
||||||
|
authentication. Supports list/get operations with filtering and pagination.
|
||||||
|
See `mem:scripts/error-reports`.
|
||||||
|
- `scripts/clean-node-modules` — Remove stale `node_modules` from all pnpm
|
||||||
|
workspaces (root, modules, member packages). Keeps the shared pnpm store
|
||||||
|
at `<repo>/.pnpm-store` unless `--store`; ignores `external/` and
|
||||||
|
`.opencode/`. Usage and reinstall steps: `mem:workflow/updating-pnpm`.
|
||||||
|
- `scripts/ci` — CI orchestration script: runs lint, tests, and format
|
||||||
|
checks per module (`frontend backend common render-wasm exporter mcp
|
||||||
|
plugins library`). Logs go to `.ci-logs/`; read the log file on failure.
|
||||||
|
See `mem:scripts/ci`.
|
||||||
|
|
||||||
# Dependency graph
|
# Dependency graph
|
||||||
|
|
||||||
`frontend -> common`, `backend -> common`, `exporter -> common`, and `frontend -> render-wasm`. Changes in `common` can affect frontend, backend, exporter, file migrations, and design-library behavior; validate across consumers when semantics change.
|
`frontend -> common`, `backend -> common`, `exporter -> common`, and `frontend -> render-wasm`. Changes in `common` can
|
||||||
|
affect frontend, backend, exporter, file migrations, and design-library behavior; validate across consumers when
|
||||||
|
semantics change.
|
||||||
|
|
||||||
# Working with Penpot designs
|
# Working with Penpot designs
|
||||||
|
|
||||||
@ -81,4 +98,6 @@ The memory is structured in a way that you can get the critical information abou
|
|||||||
|
|
||||||
## Detecting Crashes
|
## Detecting Crashes
|
||||||
|
|
||||||
The Penpot frontend can crash silently from the JS API's perspective: `execute_code` calls return successfully, but 1-2s later the workspace becomes unusable (Internal Error page). The `execute_code` tool then stops working, but `cljs_repl` still works. Use it to detect a crash via `(some? (:exception @app.main.store/state))`. For details on handling crashes, read memory `mem:frontend/handling-crashes`.
|
The Penpot frontend can crash silently from the JS API's perspective: `execute_code` calls return successfully, but 1-2s later the workspace becomes unusable (Internal Error page).
|
||||||
|
The `execute_code` tool then stops working, but `cljs_repl` still works. Use it to detect a crash via `(some? (:exception @app.main.store/state))`.
|
||||||
|
For details on handling crashes, read memory `mem:frontend/handling-crashes`.
|
||||||
|
|||||||
@ -1,38 +1,34 @@
|
|||||||
# Devenv startup and configuration
|
# Devenv startup and configuration
|
||||||
|
|
||||||
Compose-based dev environment under `docker/devenv/`, driven by `manage.sh`. Parallel instances share infra + Postgres + RustFS; each instance has its own `main` container, Valkey, source checkout, tmux session. The main database (`penpot`) and the S3 bucket (`penpot`) are shared by all instances on purpose; backend-test databases are isolated per instance (`penpot_test` on ws0, `penpot_test_wsN` on wsN) with per-instance Valkey DB numbers (`6+N`, since runtime backends occupy `0..5`).
|
Compose-based dev environment under `docker/devenv/`, driven by `manage.sh`. Parallel instances share infra + Postgres + MinIO; each instance has its own `main` container, Valkey, source checkout, tmux session.
|
||||||
|
|
||||||
## Compose project layout
|
## Compose project layout
|
||||||
|
|
||||||
- `penpotdev-infra`: shared `postgres`, `rustfs`, `valkey`, `mailer`, `ldap`. File: `docker-compose.infra.yml`.
|
- `penpotdev-infra`: shared `postgres`, `minio`, `minio-setup`, `mailer`, `ldap`. File: `docker-compose.infra.yml`.
|
||||||
- `penpotdev-wsN` (N=0,1,…): per-instance `main` + `redis` (Valkey). File: `docker-compose.main.yml`. ws0 (a.k.a. `main`) binds `$PWD`; ws1+ bind clones at `${PENPOT_WORKSPACES_DIR}/wsN/` (default `${XDG_DATA_HOME:-~/.local/share}/penpot-devenv/workspaces/`, or the former `~/.penpot/penpot_workspaces/` when it already exists), maintained by the developer.
|
- `penpotdev-wsN` (N=0,1,…): per-instance `main` + `redis` (Valkey). File: `docker-compose.main.yml`. ws0 (a.k.a. `main`) binds `$PWD`; ws1+ bind clones at `${PENPOT_WORKSPACES_DIR}/wsN/` (default `~/.penpot/penpot_workspaces/`), maintained by the developer.
|
||||||
- Optional overlay `docker-compose.opencode.yml`: added by `instance-compose` as an extra `-f` only when `PENPOT_OPENCODE_CONFIG_DIR` is set (i.e. `run-devenv --opencode-config-dir DIR` ran in this process). Bind-mounts the host dir at `/home/penpot/.config/opencode` (`:z`). Flag-only, per-call; not read from ambient env. Parser `parse-opencode-config-dir` absolutizes (`~`, realpath) because compose resolves relative bind sources against the compose file's dir. Only instances brought up with the flag get the mount.
|
- Optional overlay `docker-compose.opencode.yml`: added by `instance-compose` as an extra `-f` only when `PENPOT_OPENCODE_CONFIG_DIR` is set (i.e. `run-devenv --opencode-config-dir DIR` ran in this process). Bind-mounts the host dir at `/home/penpot/.config/opencode` (`:z`). Flag-only, per-call; not read from ambient env. Parser `parse-opencode-config-dir` absolutizes (`~`, realpath) because compose resolves relative bind sources against the compose file's dir. Only instances brought up with the flag get the mount.
|
||||||
- All projects join external network `penpot_shared`. Created idempotently by `ensure-devenv-network`, never removed by lifecycle commands.
|
- All projects join external network `penpot_shared`. Created idempotently by `ensure-devenv-network`, never removed by lifecycle commands.
|
||||||
|
|
||||||
## Source-of-truth files
|
## Source-of-truth files
|
||||||
|
|
||||||
- `docker/devenv/defaults.env`: ws0 baseline — container/volume names, runtime env, published host ports, tmux defaults. `manage.sh` aborts if unreadable.
|
- `docker/devenv/defaults.env`: ws0 baseline — container/volume names, runtime env, published host ports, tmux defaults. `manage.sh` aborts if unreadable.
|
||||||
- For ws1+, `instance-env-overrides` computes the per-instance overrides (container/volume names, host ports offset `10000·N`, `PENPOT_PUBLIC_URI`, `PENPOT_REDIS_URI`, `PENPOT_TEST_DATABASE_URI`, `PENPOT_TEST_REDIS_URI`) and `instance-compose` injects them as env vars at compose time — never written to disk, recomputed each call so they can't drift. ws0 uses `defaults.env` as-is.
|
- For ws1+, `instance-env-overrides` computes the per-instance overrides (container/volume names, host ports offset `10000·N`, `PENPOT_PUBLIC_URI`, `PENPOT_REDIS_URI`, `PENPOT_BACKEND_WORKER=false`) and `instance-compose` injects them as env vars at compose time — never written to disk, recomputed each call so they can't drift. ws0 uses `defaults.env` as-is.
|
||||||
- Per-instance test databases are created by `ensure-instance-test-database` (called from `start-instance`): `CREATE DATABASE` guarded by a `pg_database` existence check, so it is idempotent. Needed because `postgresql_init.sql` only runs on first volume creation. The backend applies migrations itself on first use, so an empty database is enough.
|
- `backend/scripts/_env`: backend-internal only — secret keys, `PENPOT_FLAGS` (with `enable-backend-worker` gated on `PENPOT_BACKEND_WORKER`), `JAVA_OPTS`, `setup_minio()`. Never duplicates `defaults.env`.
|
||||||
- `backend/scripts/_env`: backend-internal only — secret keys, `PENPOT_FLAGS` (with `enable-backend-worker` gated on `PENPOT_BACKEND_WORKER`), `JAVA_OPTS`, `setup_s3_bucket()`. Never duplicates `defaults.env`.
|
|
||||||
- Frontend admin-console is off by default in devenv: `docker-compose.main.yml` sets `PENPOT_DEVENV=true`, which makes `app.config` (`goog-define devenv-mode`) prepend `disable-admin-console` ahead of the user `config.js` flags; flags parse in order and user flags win, so `enable-admin-console` in `frontend/resources/public/js/config.js` re-enables it. Outside devenv (prod builds) the default is unset and the admin-console stays on, matching `flags/default`.
|
|
||||||
- Compose files use pure `${VAR}` substitution; missing var = compose fails.
|
- Compose files use pure `${VAR}` substitution; missing var = compose fails.
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
|
|
||||||
- `infra-compose` / `instance-compose` wrap `docker compose` with `env -i`, then re-inject what compose needs. Stripping is required because `defaults.env` is sourced into manage.sh's shell at startup (stale values would leak); the ws1+ overrides are deliberately re-injected as shell env vars precisely because Compose gives shell precedence over `--env-file`, so they override the `defaults.env` baseline.
|
- `infra-compose` / `instance-compose` wrap `docker compose` with `env -i`, then re-inject what compose needs. Stripping is required because `defaults.env` is sourced into manage.sh's shell at startup (stale values would leak); the ws1+ overrides are deliberately re-injected as shell env vars precisely because Compose gives shell precedence over `--env-file`, so they override the `defaults.env` baseline.
|
||||||
- Volume names pinned via `name:` (PENPOT_*_VOLUME), decoupled from the compose project name. Replacement services use new volumes and leave old volumes untouched. PostgreSQL 18 mounts `penpotdev_postgres_data_pg18` at `/var/lib/postgresql`; the old PG16 volume is not migrated automatically.
|
- Volume names pinned via `name:` (PENPOT_*_VOLUME), decoupled from the compose project name. ws1+ inject distinct per-instance volume names; ws0 keeps the historical `penpotdev_*` physical names so project renames never require data migration.
|
||||||
- Network aliases (`- main`, `- redis`) are not declared in main.yml. Compose's auto-service-alias still registers `redis` on the shared network, so DNS for `redis` is non-deterministic with multiple instances. Backend uses `PENPOT_REDIS_URI=redis://penpot-devenv-wsN-valkey/0` (container_name) instead.
|
- Network aliases (`- main`, `- redis`) are not declared in main.yml. Compose's auto-service-alias still registers `redis` on the shared network, so DNS for `redis` is non-deterministic with multiple instances. Backend uses `PENPOT_REDIS_URI=redis://penpot-devenv-wsN-valkey/0` (container_name) instead.
|
||||||
- No cross-project `depends_on`. `manage.sh ensure-infra-up` uses Compose `--wait`; PostgreSQL, RustFS, and Mailpit expose healthchecks. PostgreSQL checks TCP so its temporary init server cannot report ready.
|
- No cross-project `depends_on`. `manage.sh ensure-infra-up` `docker wait`s on the `minio-setup` one-shot.
|
||||||
- `JAVA_OPTS` in `manage.sh` is shadowed inside the container by `_env`. The `-e JAVA_OPTS=...` flag only matters for processes that don't source `_env`.
|
- `JAVA_OPTS` in `manage.sh` is shadowed inside the container by `_env`. The `-e JAVA_OPTS=...` flag only matters for processes that don't source `_env`.
|
||||||
- opencode is not part of the devenv image. `scripts/setup-opencode` installs the `@opencode/cli` global pnpm package into the penpot user's `$PNPM_HOME` (`/home/penpot/.local/share/pnpm`, inside the user-data volume, so it survives a container re-creation); nothing pins the version in the repo, it tracks the `latest` dist-tag at install time. pnpm 12 puts the global packages in `$PNPM_HOME/global` and the bin dir in `$PNPM_HOME/bin`, and aborts with `ERR_PNPM_GLOBAL_BIN_DIR_NOT_IN_PATH` unless that bin dir is on `PATH`; `files/bashrc` and the script both add it. The script exports `PNPM_HOME` itself because non-interactive shells never read `~/.bashrc`.
|
|
||||||
- `files/entrypoint.sh` chowns penpot's whole `~/.local` and `~/.cache` on every start, because `EXTERNAL_UID` remaps penpot's UID to the host user and build-time ownership then stops matching. That covers pnpm's global dir, store, `package-manager-store` and metadata cache, plus opencode's own state; without it, global installs and pnpm's `packageManager` auto-download fail on permissions. The dotfiles copied from `/root` (`.bashrc`, `.vimrc`, `.tmux.conf`) are chowned in the same pass. `~/.config` stays out on purpose (the opencode overlay bind-mounts a host directory inside it) and so does `/home/penpot` itself, which holds the repo bind mount.
|
|
||||||
|
|
||||||
## Worker policy
|
## Worker policy
|
||||||
|
|
||||||
Backend workers run only on ws0. `_env` gates `enable-backend-worker` on `PENPOT_BACKEND_WORKER`; ws1+ inject it as false. Workers are pure fire-and-forget: `jobs/submit` inserts a row into the shared Postgres `job` table and returns (the legacy `task` table stays dormant); RPC handlers never wait on completion and workers never publish to msgbus. The reason for "ws0 only" is avoiding multi-instance worker races (cron dedup is best-effort across instances, `jobs/submit` `dedupe` is racy across submitters); details in `mem:prod-infra/core`.
|
Backend workers run only on ws0. `_env` gates `enable-backend-worker` on `PENPOT_BACKEND_WORKER`; ws1+ inject it as false. Workers are pure fire-and-forget: `wrk/submit!` inserts a row into the shared Postgres `task` table and returns; RPC handlers never wait on completion and workers never publish to msgbus. The reason for "ws0 only" is avoiding multi-instance worker races (cron dedup is best-effort across instances, `wrk/submit!` `dedupe` is racy across submitters); details in `mem:prod-infra/core`.
|
||||||
|
|
||||||
Each workspace is independent and can be started/stopped in any order. Shared infra (Postgres, RustFS, etc.) is shut down only when no instances remain running.
|
Each workspace is independent and can be started/stopped in any order. Shared infra (postgres, minio, etc.) is shut down only when no instances remain running.
|
||||||
|
|
||||||
## Port layout
|
## Port layout
|
||||||
|
|
||||||
@ -47,7 +43,7 @@ Container-internal ports fixed; host side offset `10000·N`.
|
|||||||
| 14181 | 24181 | … | 14281 | Serena MCP |
|
| 14181 | 24181 | … | 14281 | Serena MCP |
|
||||||
| 14182 | 24182 | … | 24282 | Serena dashboard |
|
| 14182 | 24182 | … | 24282 | Serena dashboard |
|
||||||
|
|
||||||
Everything else (frontend dev, backend API, exporter, storybook, REPLs, plugin dev, MCP inspector/WebSocket) is in-process or same-origin via Caddy/nginx. Infra publishes RustFS S3 API 9000, RustFS console 9001, and Mailpit UI 1080 on loopback; ldap 10389/10636 remains a singleton without offsets. Mailpit stores its SQLite database in `penpotdev_mailpit_data`.
|
Everything else (frontend dev, backend API, exporter, storybook, REPLs, plugin dev, MCP inspector/WebSocket) is in-process or same-origin via Caddy/nginx. Infra publishes: mailer 1080, ldap 10389/10636 (singletons, not offset).
|
||||||
|
|
||||||
## Tmux + MCP routing
|
## Tmux + MCP routing
|
||||||
|
|
||||||
@ -78,42 +74,3 @@ No `--delete` on the working-tree pass: gitignored caches in the workspace survi
|
|||||||
- `start-devenv` / `log-devenv` / `drop-devenv`: legacy paths around ws0 + shared infra. `drop-devenv` never removes volumes.
|
- `start-devenv` / `log-devenv` / `drop-devenv`: legacy paths around ws0 + shared infra. `drop-devenv` never removes volumes.
|
||||||
|
|
||||||
`exporter/scripts/run` and `wait-and-start.sh` source `backend/scripts/_env` then `_env.local` if present.
|
`exporter/scripts/run` and `wait-and-start.sh` source `backend/scripts/_env` then `_env.local` if present.
|
||||||
|
|
||||||
## Image and dependency pins
|
|
||||||
|
|
||||||
Two Dockerfile trees, both multi-arch (`linux/amd64,linux/arm64`):
|
|
||||||
`docker/devenv` (dev) and `docker/images` + `docker/imagemagick` (prod).
|
|
||||||
|
|
||||||
- Hand-pinned versions, one source each:
|
|
||||||
- `docker/devenv/Dockerfile`: `NODE_VERSION` (+ per-arch `NODE_SHA256`),
|
|
||||||
`PNPM_VERSION` (+ `PNPM_SHA256`), `CADDY_VERSION`, `CLOJURE_VERSION`,
|
|
||||||
Zulu JDK URL + `ESUM`, `RUSTUP_VERSION`/`RUST_VERSION`/
|
|
||||||
`EMSCRIPTEN_VERSION`, `CLJKONDO_VERSION`, `BABASHKA_VERSION`,
|
|
||||||
`CLJFMT_VERSION`, `PIXI_VERSION`, `GITHUB_CLI_VERSION`,
|
|
||||||
`AWS_CLI_VERSION` (+ `BINARY_SHA256`), `UV_VERSION`, `SERENA_VERSION`,
|
|
||||||
`postgresql-client-18`, and the `penpotapp/imagemagick:<tag>`
|
|
||||||
`COPY --from`.
|
|
||||||
- `docker/images/Dockerfile.backend`: Zulu JDK URL + `ESUM`. The `jlink`
|
|
||||||
`--add-modules` list must keep `jdk.management`
|
|
||||||
(`mem:backend/subtleties`).
|
|
||||||
- `docker/images/Dockerfile.{exporter,mcp,media-processor}`: `ARG
|
|
||||||
PNPM_VERSION` (+ arch SHAs, same values as devenv).
|
|
||||||
- Node and pnpm must agree in five places: `.nvmrc`, the devenv tarball pin,
|
|
||||||
the two `plugins-deploy-*` workflows that set `runtime: node@<version>`, and
|
|
||||||
the four Dockerfile pins. pnpm procedure: `mem:workflow/updating-pnpm`.
|
|
||||||
- `dhi.io` base images are pinned `tag@sha256:<index digest>`; Renovate moves
|
|
||||||
them (`penpot-renovate[bot]`, commits titled `:arrow_up: Update DHI base
|
|
||||||
images`). Tag and digest move together: a new tag with the old digest still
|
|
||||||
pulls the old image, and the digest must be the multi-arch index, not a
|
|
||||||
per-platform one.
|
|
||||||
- Every hand-pinned checksum is per architecture; one wrong value breaks half
|
|
||||||
of the multi-arch build.
|
|
||||||
- `SERENA_UPDATE_VERSION` in `docker-compose.main.yml` must never be lower than the Dockerfile's `SERENA_VERSION`. Any value makes `start-tmux.sh` install that Serena release on start, overriding the version baked into the image; a stale value silently downgrades Serena (it once replaced 1.7.0 with 1.6.1). If the image already has that version the install is an offline no-op; otherwise it downloads on every start (`run-devenv` recreates the container) and, offline, fails after ~25s and keeps the image version. Bump both together; empty means use the image version.
|
|
||||||
- Infra images live in `docker-compose.infra.yml`: `rustfs/rustfs` (owns the
|
|
||||||
S3 data volume), `axllent/mailpit` (SQLite in a volume), `valkey/valkey`,
|
|
||||||
`postgres:18.6`, `rroemhild/test-openldap`. Keep the volume names stable
|
|
||||||
when bumping them; replacement services get new volumes and leave the old
|
|
||||||
data untouched.
|
|
||||||
- Build locally: `docker build -f docker/devenv/Dockerfile docker/devenv/`
|
|
||||||
and `cd docker/images && ./build.sh <image> --local`. CI:
|
|
||||||
`build-docker-devenv.yml`, `build-docker.yml`.
|
|
||||||
|
|||||||
@ -5,7 +5,6 @@
|
|||||||
## Layout and commands
|
## Layout and commands
|
||||||
|
|
||||||
- Source: `exporter/src/`; config: `deps.edn`, `shadow-cljs.edn`, `package.json`; runtime helpers/assets: `vendor/`, `scripts/`.
|
- Source: `exporter/src/`; config: `deps.edn`, `shadow-cljs.edn`, `package.json`; runtime helpers/assets: `vendor/`, `scripts/`.
|
||||||
- `exporter/resources/wasm/render-wasm.{js,wasm}` and `exporter/src/app/wasm/shared.js` are generated render-wasm build output, not source: they are gitignored. Never hand-edit or commit them; regenerate by rebuilding render-wasm.
|
|
||||||
- From `exporter/`: setup `./scripts/setup`; watch `pnpm run watch` or `pnpm run watch:app`; production build `pnpm run build`; test bundle `pnpm run build:test`; tests `pnpm run test` or `pnpm run test:quiet`; lint `pnpm run lint:clj`; format check/fix `pnpm run check-fmt:clj` / `pnpm run fmt:clj`.
|
- From `exporter/`: setup `./scripts/setup`; watch `pnpm run watch` or `pnpm run watch:app`; production build `pnpm run build`; test bundle `pnpm run build:test`; tests `pnpm run test` or `pnpm run test:quiet`; lint `pnpm run lint:clj`; format check/fix `pnpm run check-fmt:clj` / `pnpm run fmt:clj`.
|
||||||
- Because exporter consumes `common/`, shared file/shape/model changes may need exporter verification even when the immediate change is not under `exporter/`.
|
- Because exporter consumes `common/`, shared file/shape/model changes may need exporter verification even when the immediate change is not under `exporter/`.
|
||||||
- Cross-cutting testing principles and anti-patterns: `mem:testing`.
|
- Cross-cutting testing principles and anti-patterns: `mem:testing`.
|
||||||
|
|||||||
@ -1,106 +0,0 @@
|
|||||||
# Auth Events and the Session Origin
|
|
||||||
|
|
||||||
`signin` is the frontend `identify` event emitted once per session from
|
|
||||||
`logged-in` (`app.main.data.auth`). It carries the profile plus a closed
|
|
||||||
vocabulary that says why the session started.
|
|
||||||
|
|
||||||
## Vocabulary
|
|
||||||
|
|
||||||
- `source`: `login` | `register`. Always present.
|
|
||||||
- `auth-backend`: `penpot` | `ldap` | `google` | `github` | `gitlab` | `oidc`.
|
|
||||||
Same key the profile already uses. Always present.
|
|
||||||
- `activation`: `immediate` | `email-verification`. Only on signups.
|
|
||||||
- `from-invitation`: boolean. Always present in `signin`
|
|
||||||
(`(some? (:invitation-token profile))`); `auth-origin` omits it when `nil`.
|
|
||||||
|
|
||||||
`auth-origin` (public, validated against `schema:auth-origin`) drops the nil
|
|
||||||
fields and checks with `(assert (sm/check ...))`: it throws in development and
|
|
||||||
tests, and `:elide-asserts true` removes it from the release build, so a bad
|
|
||||||
value can never break a real login. Emitters never call it directly.
|
|
||||||
|
|
||||||
## Where each value comes from
|
|
||||||
|
|
||||||
- `source` and `activation` travel as profile metadata (`::ev/source`,
|
|
||||||
`::ev/activation`), written by `origin-meta`. `login-from-token` derives both
|
|
||||||
from the token issuer with `iss->origin`: `:auth` → `login`; `:verify-email` →
|
|
||||||
`register` + `email-verification`. An unknown `:iss` produces no metadata, so
|
|
||||||
the assertion fires in development and the event goes without `:source` in
|
|
||||||
production.
|
|
||||||
- `auth-backend` and `from-invitation` are read from the profile.
|
|
||||||
- `profile-origin` composes the four and `logged-in` merges them into the event;
|
|
||||||
the emitters only declare the fact.
|
|
||||||
|
|
||||||
| Flow | `source` | `activation` | `from-invitation` |
|
|
||||||
|---|---|---|---|
|
|
||||||
| Password login | login | | false |
|
|
||||||
| Password login from an invitation link | login | | true |
|
|
||||||
| LDAP login | login | | false |
|
|
||||||
| Social login, existing active profile | login | | false |
|
|
||||||
| Signup, email verification disabled | register | immediate | false |
|
|
||||||
| Signup, email verification enabled | register | email-verification | false |
|
|
||||||
| Social signup, active profile | register | immediate | false |
|
|
||||||
| Social signup, inactive profile | register | email-verification | false |
|
|
||||||
| Signup from an invitation link, email verification disabled | register | immediate | true |
|
|
||||||
| Signup from an invitation link, email verification enabled | register | email-verification | true |
|
|
||||||
| Social login from an invitation link | login | | true |
|
|
||||||
| Social signup from an invitation link, email verification enabled | register | email-verification | true |
|
|
||||||
| LDAP login from an invitation link | login | | true |
|
|
||||||
|
|
||||||
LDAP is the explicit exception: the directory provisions the account on the
|
|
||||||
first login, but there is no user signup, so the flow is always `login` and
|
|
||||||
never carries `activation`. Do not "fix" it by inferring a signup.
|
|
||||||
|
|
||||||
## Invitations go through `logged-in`
|
|
||||||
|
|
||||||
- `login-from-register` accepts an optional `:invitation-token`;
|
|
||||||
`with-register-origin` decorates the profile and `logged-in` does the rest.
|
|
||||||
The four places that used to navigate by hand to `:auth-verify-token`
|
|
||||||
(`ui/auth/register.cljs` twice, `ui/static.cljs`, `ui/auth/login_modal.cljs`)
|
|
||||||
emit `login-from-register` with the token. The modal used to live in
|
|
||||||
`ui/viewer/login.cljs`; it was extracted to the shared
|
|
||||||
`ui/auth/login_modal.cljs` (`login-register-modal*`, also used by
|
|
||||||
`ui/mcp_authorize.cljs`).
|
|
||||||
- The invitation navigates through `get-invitation-nav`, which does not wait for
|
|
||||||
`::dtm/teams-fetched`: `fetch-teams` has no error handling, so depending on it
|
|
||||||
would leave the invitation unaccepted when `get-teams` fails. For invited
|
|
||||||
profiles `get-redirect-events` emits `rx/empty`, so the teams path cannot move
|
|
||||||
the user away from the token verification screen.
|
|
||||||
- The OIDC callback with an invitation mints an `:auth` token that carries the
|
|
||||||
invitation JWE in its claims (`backend/src/app/auth/oidc.clj`), so the
|
|
||||||
frontend follows the regular path (`login-from-token` → `logged-in`). That
|
|
||||||
flow visits the token verification screen twice.
|
|
||||||
- `ui/auth/login_modal.cljs` (the register modal: `:is-active` branch calls
|
|
||||||
`on-close`, or `on-register` when the caller passes one, as in the MCP
|
|
||||||
authorize flow) and `ui/static.cljs` (`rt/reload true`) `:is-active`
|
|
||||||
branches still bypass `logged-in`; they are a separate decision, not an
|
|
||||||
oversight.
|
|
||||||
|
|
||||||
## Backend side
|
|
||||||
|
|
||||||
- The `register-profile` success branches
|
|
||||||
(`backend/src/app/rpc/commands/auth.clj`) put `activation` in the event props
|
|
||||||
and emit **no** `action`: the name already says it is a signup, and a constant
|
|
||||||
action would only add noise. The retry branches
|
|
||||||
(`register-profile-retry`) keep `action` in `context`
|
|
||||||
(`ignore-because-blocked`, `ignore`, `resend-email-verification`,
|
|
||||||
`ignore-because-complaints`), where it distinguishes outcomes.
|
|
||||||
- The LDAP invitation branch of `login-with-ldap` returns the full profile plus
|
|
||||||
`:invitation-token`; a partial map would break the `auth-origin` assertion.
|
|
||||||
- `register-profile` does not accept the invitation: it mints the session and
|
|
||||||
returns the token. The acceptance is `accept-team-invitation`, emitted from
|
|
||||||
`verify-token`.
|
|
||||||
- `filter-telemetry-props` keeps `source` and `activation` in the anonymised
|
|
||||||
row; `:action` is not on the allowlist.
|
|
||||||
|
|
||||||
## Analytics
|
|
||||||
|
|
||||||
- `register-profile` is a track event, so the props adapter forwards
|
|
||||||
`activation` with no extra work.
|
|
||||||
- `signin` is an identify event built from a closed list of profile keys. It
|
|
||||||
only reaches analytics if the identify branch of Nexus forwards `source`,
|
|
||||||
`activation` and `from_invitation` (`external/nexus`, another repository).
|
|
||||||
|
|
||||||
## `::ev/source` is no longer dead metadata
|
|
||||||
|
|
||||||
It is read by `profile-origin`. Do not remove it as unused, and keep the
|
|
||||||
vocabulary in sync with `schema:auth-origin` and the table above.
|
|
||||||
@ -49,23 +49,16 @@ Get `team-id` from `(:current-team-id @app.main.store/state)`. Get file ids from
|
|||||||
|
|
||||||
## Runtime patching with `set!`
|
## Runtime patching with `set!`
|
||||||
|
|
||||||
Some frontend vars are deliberately mutable escape hatches for runtime instrumentation or circular-dependency patching. From `cljs_repl`, use `set!` for temporary debugging of CLJS vars such as `app.main.errors/reload-file`, `app.main.errors/is-plugin-error?`, `app.main.errors/last-report`, or `app.main.errors/last-exception`. These patches affect only the live browser runtime and disappear on reload or recompilation.
|
Some frontend vars are deliberately mutable escape hatches for runtime instrumentation or circular-dependency patching. From `cljs_repl`, use `set!` for temporary debugging of CLJS vars such as `app.main.store/on-event`, `app.main.errors/reload-file`, `app.main.errors/is-plugin-error?`, `app.main.errors/last-report`, or `app.main.errors/last-exception`. These patches affect only the live browser runtime and disappear on reload or recompilation.
|
||||||
|
|
||||||
`set!` on `app.main.store/on-event` does nothing at runtime: the store captured the function when it was created. To watch events, subscribe to the store stream and dispose the subscription after:
|
|
||||||
|
|
||||||
```clojure
|
```clojure
|
||||||
;; Log non-noisy Potok events temporarily.
|
;; Log non-noisy Potok events temporarily.
|
||||||
(def sub
|
(set! app.main.store/on-event
|
||||||
(beicon.v2.core/subs!
|
(fn [event]
|
||||||
(fn [event]
|
(when (potok.v2.core/event? event)
|
||||||
(when (potok.v2.core/event? event)
|
(.log js/console (potok.v2.core/repr-event event)))))
|
||||||
(.log js/console (potok.v2.core/repr-event event))))
|
|
||||||
app.main.store/stream))
|
|
||||||
;; (beicon.v2.core/dispose! sub)
|
|
||||||
```
|
```
|
||||||
|
|
||||||
`cljs_repl` starts every call in `cljs.user`; begin a call with `(in-ns 'my-ns)` to reuse helpers defined in another namespace.
|
|
||||||
|
|
||||||
Restore mutable hooks after debugging, or reload the frontend. Use JVM `alter-var-root` only for JVM Clojure; it is not the normal way to patch live CLJS browser vars.
|
Restore mutable hooks after debugging, or reload the frontend. Use JVM `alter-var-root` only for JVM Clojure; it is not the normal way to patch live CLJS browser vars.
|
||||||
|
|
||||||
## Browser-console debug namespace
|
## Browser-console debug namespace
|
||||||
|
|||||||
@ -138,17 +138,15 @@ Distinguishing abstractions (the OOP articulation of the shared principles):
|
|||||||
control via Playwright; recipe in the README.
|
control via Playwright; recipe in the README.
|
||||||
|
|
||||||
## CI
|
## CI
|
||||||
Headless per-PR gate: the `composable-test-suite` job in
|
Headless per-PR gate: `.github/workflows/tests-composable-suite.yml` runs
|
||||||
`.github/workflows/tests-e2e.yml` runs `pnpm --filter composable-test-suite run test:ci` — mocked
|
`pnpm --filter composable-test-suite run test:ci` — mocked backend (frontend e2e static server +
|
||||||
backend (frontend e2e static server +
|
|
||||||
Playwright RPC fixtures, no backend/login), the in-sandbox bundle injected via `ɵloadPlugin`,
|
Playwright RPC fixtures, no backend/login), the in-sandbox bundle injected via `ɵloadPlugin`,
|
||||||
results streamed via console markers, `TEST_FILTER` by identifier substring. The mocked backend is
|
results streamed via console markers, `TEST_FILTER` by identifier substring. The mocked backend is
|
||||||
NOT a limitation for this suite (everything asserted is frontend store logic; empirically
|
NOT a limitation for this suite (everything asserted is frontend store logic; empirically
|
||||||
confirmed against the interactive runs). Architecture mirrors `plugin-api-test-suite`'s CI driver;
|
confirmed against the interactive runs). Architecture mirrors `plugin-api-test-suite`'s CI driver;
|
||||||
the mock harness exists in THREE places that must stay in sync (provenance note in `ci/run-ci.ts`).
|
the mock harness exists in THREE places that must stay in sync (provenance note in `ci/run-ci.ts`).
|
||||||
The frontend bundle is built once per SHA and restored by this job (`mem:frontend/e2e-ci-workflow`).
|
|
||||||
Details: README, "Running in CI".
|
Details: README, "Running in CI".
|
||||||
|
|
||||||
## Substrate
|
## Substrate
|
||||||
`mem:common/testing`, `mem:common/component-data-model`, `mem:common/component-swap-pipeline`,
|
`mem:common/test-setup`, `mem:common/component-data-model`, `mem:common/component-swap-pipeline`,
|
||||||
`mem:frontend/testing`.
|
`mem:frontend/testing`.
|
||||||
|
|||||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
x
Reference in New Issue
Block a user