diff --git a/.serena/memories/common/uri.md b/.serena/memories/common/uri.md new file mode 100644 index 0000000000..9648499409 --- /dev/null +++ b/.serena/memories/common/uri.md @@ -0,0 +1,13 @@ +# Common URI helpers + +`app.common.uri` wraps `lambdaisland.uri` and re-exports `join`, `parse`, `query-encode`, `percent-encode` and `uri?` with `dm/export`. + +## `join` resolves references, it does NOT concatenate paths + +- `lambdaisland.uri/join` implements RFC 3986 reference resolution, not path concatenation: a relative reference that does not start with `/` replaces everything after the last `/` of the path it is resolved against. Joining several segments resolves them successively, so each segment replaces the last segment of the previous result. +- `(join "https://h/" "api/teams/" "" "users/" "")` returns `https://h/api/teams/users/`, not `.../api/teams//users/`: `"users/"` replaced `""`. `(apply join base segments)` behaves the same. +- `ensure-path-slash` normalizes only the BASE to end with `/`; it does not touch the segments, so a base ending in `/` is not enough for a multi-segment append. +- To append several segments: either join them into a single relative path first (dropping empty segments and collapsing slashes), or make every non-final segment end with `/`, including the dynamic ids. Keep that normalization in the shared URL builder, not in each call site. +- A segment starting with `/` resolves against the host root and silently drops the base subpath; reject it. +- Frontend `app.config` normalizes the base (`normalize-uri`) and relies on callers passing trailing-slash segments by convention (`resolve-file-media` joins `"assets/by-file-media-id/"` then `"/thumbnail"`); it does not normalize segments. +- Backend shared builders are `app.config/join-uri` and `app.config/get-public-uri`. A past refactor moved Nitrate's per-request helper onto them without the segment normalization and broke every multi-segment Nitrate URL that passes `"api/teams/" team-id "users/" profile-id`. diff --git a/.serena/memories/prod-infra/core.md b/.serena/memories/prod-infra/core.md index ee7dc542b0..a3a9f60de4 100644 --- a/.serena/memories/prod-infra/core.md +++ b/.serena/memories/prod-infra/core.md @@ -31,3 +31,5 @@ Penpot in production lives with both: horizontal-scale deployments accept "exact - Devenv composition and the ws0-only worker placement: `mem:devenv/core`. - Storage backend resolution, dedup, bucket behavior, object lifecycle, and file-data lifecycle: `mem:backend/storage`. +- Storage operation metrics (S3 API calls, logical ops, dedup, asset requests) and the logical-bucket vs physical-target distinction: `mem:backend/storage` (Metrics section). +- Grafana (datasource ids and how to use them, main dashboard, panel conventions and gotchas): `mem:prod-infra/grafana`. diff --git a/.serena/memories/prod-infra/grafana.md b/.serena/memories/prod-infra/grafana.md new file mode 100644 index 0000000000..73aa61320e --- /dev/null +++ b/.serena/memories/prod-infra/grafana.md @@ -0,0 +1,42 @@ +# Grafana + +Private instance with Penpot metrics and logs. Dashboards are edited through the Grafana API (MCP tools); there is no dashboards-as-code for them. + +Keep this memory free of internal names: no host names, no deployed versions, no topology. The datasource ids and the label conventions are the useful technical part. + +## Datasources + +- Metrics — **Prometheus**, uid `xkRmyFBGk`. PromQL. Metric families: `penpot_*` (RPC, task timings, CLIMIT, HTTP, websocket, storage), `hikaricp_*`, `jvm_*` and `process_*` on the backend (`job="app"`); `pg_*` from the Postgres exporter (`job="postgres"`); `redis_*` from the Redis exporter (`job="redis"`); `node_*` and `container_*` from node-exporter and cAdvisor. Backend histograms are in **milliseconds**. +- Logs — **Loki**, uid `h3CGKhEMk`. LogQL, and filter by the structured metadata `logger`/`level` (`{job="penpot"} | logger="..." | level="ERROR"`). Some signals exist only as logs: CLIMIT rejections have no metric (`logger="app.rpc.climit"`, `hint="rejected"`). +- SQL — **Postgres**, uid `y8ea9tfMk`. Panels need `rawQuery: true`, `editorMode: "code"` and `format: "table"`. The `task` table is the only truth about the queue depth. +- **Postgres (telemetry)**, uid `9YAUuM_Gz`. +- `$__rate_interval` and `$__interval` are substituted by Grafana; use them instead of fixed windows in graphs. + +## Main dashboard + +- `Production status`, uid `penpot-production-status`. Sections: general state and deploy, RPC and CLIMIT, PostgreSQL and pool, Redis/msgbus, tasks, JVM/process. Variables `$env` and `$host` (custom), no auto refresh. +- Per-host investigation dashboards are separate, and the alert rules deeplink into them. + +## Conventions + +- Filter with `instance=~"$host.$env..*"`. To know which instances serve, count the ones reporting JVM metrics; `up{job="app"}` is not reliable (it can report 1 while the endpoint answers an empty body). +- Layout: a band of number-only panels at the top of each section, wider data panels below, tables tall enough to show every row, legend on the **right** sorted by last value descending (`sortBy: "Last *"`, `sortDesc: true`). +- Panels comparing used against total need `defaults.min = 0`, else the automatic axis exaggerates the difference. +- Counters per method or task name read better as `increase(...[1h])` than `rate(...)`, with `decimals: 0`. +- Cron "did it run" comes from the task metrics, not from the cron registry table (it keeps rows from old versions). +- Dashboard thresholds must match the alert rule thresholds. + +## Gotchas + +- PromQL literal dots need `\\.`; simpler to use `[.]` or a plain dot. +- The panel-level `datasource` must match its targets'. A SQL table declared as Prometheus renders empty even though the query returns rows. +- Shortening a host in a legend needs a short label in the query: `renameByRegex` renames the field name, but with `legendFormat` set the legend text comes from the datasource. + - Prometheus: `label_replace(, "host", "$1", "instance", "([^.]+)[.][^.]+.*")` plus `legendFormat: "... {{host}}"`. + - Loki metric query: `| label_format host="{{ regexReplaceAll "[.][^.]*[.][^.]*[.][^.]*$" .instance "" }}"` plus `legendFormat: "{{host}}"`. Quotes inside the template must be escaped as `\"`, or LogQL closes the string early (`not a valid duration string`). +- Tables from a Prometheus raw selector carry a `__name__` column; hide it with `organize`. +- Saving through the API normalizes the dashboard (rows may be flattened, `y` recomputed): submit the native form (rows with `panels: []`, panels top-level, contiguous `y`). +- Panel images cannot be rendered (no image renderer installed) and there is no browser here: verify a dashboard by running its queries with the variables substituted. +- Signals that exist only as logs are counted with Loki, and the count is then cross-checked against the nginx access log for the same window and the same status code (`job="frontend"`, e.g. a `429`): two independent sources must agree before a number is trusted. +- The app writes its own logs at WARN only when the CLIMIT queue is not empty and at TRACE otherwise, so production Loki holds just the under-pressure lines: a count of `hint="enqueued"` or `hint="acquired"` is not the number of calls, while rejections always land at WARN and their count is the real one. +- The MCP Prometheus tool can answer empty for every query, `up` included, while the datasource itself is fine: an empty result from it is not proof that a panel has no data, and blaming the datasource for it is wrong. Run PromQL by hand with `grafana_api_request` against `/api/datasources/uid//resources/api/v1/query?query=...` (or `POST /api/ds/query`, the path dashboards use), and check when a series is expected but the tool says nothing. +- **Check `list_dashboard_versions` before writing.** The user edits dashboards by hand and a full-JSON overwrite silently reverts that work; prefer `operations` patches per panel.