From fb2a0ac7fcf76a7e0e530504f4089b5366f9f9da Mon Sep 17 00:00:00 2001 From: Andrey Antukh Date: Mon, 12 Oct 2026 23:58:19 +0000 Subject: [PATCH] :books: Record the rules for production data Repair scripts run against production with a plain SQL client, not the devenv psql: no `\set`, `\if` or `\copy` there. The SQL datasource the agent can query is a read-only replica that rejects writes (a temporary table included) and cuts long queries, so its queries are plain SELECTs bounded by a list of ids: an unbounded scan over tens of millions of rows does not come back. AI-assisted-by: deepseek-v4.1-flash --- .serena/memories/prod-infra/core.md | 9 +++++++++ .serena/memories/prod-infra/grafana.md | 2 +- 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/.serena/memories/prod-infra/core.md b/.serena/memories/prod-infra/core.md index 2af6055cf7..e90f353b7e 100644 --- a/.serena/memories/prod-infra/core.md +++ b/.serena/memories/prod-infra/core.md @@ -31,6 +31,15 @@ Penpot in production lives with both: horizontal-scale deployments accept "exact Each backend has a per-process availability flag (`app.health`, a global atom). While offline, `/readyz` answers `503 OFFLINE` without querying the database, so the load balancer removes that instance from rotation while requests in flight still finish. Operator surface: `app.srepl.main/set-offline` / `set-online`; a restart comes back online. In-flight worker tasks are safe: tasks claimed as `scheduled` return to `new` after 5 minutes via `sql:reschedule-lost` and another instance picks them up. +## Touching production data + +Production runs the deployed toolbox, not the devenv one: + +- **No psql meta-commands there.** `scripts/psql` and `\set`, `\if`, `\copy` do not exist in production, so a data script must be **plain SQL**, copy-pasteable into any client. +- **Start from the ids, not from an inventory.** The ids come from the error or the logs. Unbounded queries over tables with tens of millions of rows (`storage_object`, `file_media_object`) do not finish in a reasonable time; bounded to an id list they are index lookups. +- **One query per reference column.** Two columns in one `IN (col_a, col_b)` lose the index and scan the whole table; split them. +- The read-only replica cuts long queries: see `mem:prod-infra/grafana`. + ## See also - Devenv composition and the ws0-only worker placement: `mem:devenv/core`. diff --git a/.serena/memories/prod-infra/grafana.md b/.serena/memories/prod-infra/grafana.md index fded5d8695..fbc40ec6e5 100644 --- a/.serena/memories/prod-infra/grafana.md +++ b/.serena/memories/prod-infra/grafana.md @@ -8,7 +8,7 @@ Keep this memory free of internal names: no host names, no deployed versions, no - Metrics — **Prometheus**, uid `xkRmyFBGk`. PromQL. Metric families: `penpot_*` (RPC, task timings, CLIMIT, HTTP, websocket, storage), `hikaricp_*`, `jvm_*` and `process_*` on the backend (`job="app"`); `pg_*` from the Postgres exporter (`job="postgres"`); `redis_*` from the Redis exporter (`job="redis"`); `node_*` and `container_*` from node-exporter and cAdvisor. Backend histograms are in **milliseconds**. - Logs — **Loki**, uid `h3CGKhEMk`. LogQL, and filter by the structured metadata `logger`/`level` (`{job="penpot"} | logger="..." | level="ERROR"`). Signals that only exist as logs: the profile behind a CLIMIT queue (`logger="app.rpc.climit"`, whose `id` carries the key). CLIMIT rejections have their own counters (`penpot_rpc_climit_rejected_total`, by `name` and `reason`), and their `hint="rejected"` line goes out at WARN only when something stays queued, so counting the lines is not counting the rejections. -- SQL — **Postgres**, uid `y8ea9tfMk`. Panels need `rawQuery: true`, `editorMode: "code"` and `format: "table"`. The `task` table is the only truth about the queue depth. +- SQL — **Postgres**, uid `y8ea9tfMk`. Panels need `rawQuery: true`, `editorMode: "code"` and `format: "table"`. The `task` table is the only truth about the queue depth. It is a **read-only replica**: it rejects any write, `CREATE TEMP TABLE` included, so queries through it must be plain `SELECT`s. It also cuts long-running queries (the MCP path has a short timeout), so bound them (e.g. a list of ids); an empty answer there is not proof that there is no data. - **Postgres (telemetry)**, uid `9YAUuM_Gz`. - `$__rate_interval` and `$__interval` are substituted by Grafana; use them instead of fixed windows in graphs.