📚 Record the rules for production data

Repair scripts run against production with a plain SQL client, not the
devenv psql: no `\set`, `\if` or `\copy` there. The SQL datasource the
agent can query is a read-only replica that rejects writes (a temporary
table included) and cuts long queries, so its queries are plain SELECTs
bounded by a list of ids: an unbounded scan over tens of millions of rows
does not come back.

AI-assisted-by: deepseek-v4.1-flash

.serena/memories/prod-infra/core.md

@@ -31,6 +31,15 @@ Penpot in production lives with both: horizontal-scale deployments accept "exact
 
 Each backend has a per-process availability flag (`app.health`, a global atom). While offline, `/readyz` answers `503 OFFLINE` without querying the database, so the load balancer removes that instance from rotation while requests in flight still finish. Operator surface: `app.srepl.main/set-offline` / `set-online`; a restart comes back online. In-flight worker tasks are safe: tasks claimed as `scheduled` return to `new` after 5 minutes via `sql:reschedule-lost` and another instance picks them up.
 
+## Touching production data
+
+Production runs the deployed toolbox, not the devenv one:
+
+- **No psql meta-commands there.** `scripts/psql` and `\set`, `\if`, `\copy` do not exist in production, so a data script must be **plain SQL**, copy-pasteable into any client.
+- **Start from the ids, not from an inventory.** The ids come from the error or the logs. Unbounded queries over tables with tens of millions of rows (`storage_object`, `file_media_object`) do not finish in a reasonable time; bounded to an id list they are index lookups.
+- **One query per reference column.** Two columns in one `IN (col_a, col_b)` lose the index and scan the whole table; split them.
+- The read-only replica cuts long queries: see `mem:prod-infra/grafana`.
+
 ## See also
 
 - Devenv composition and the ws0-only worker placement: `mem:devenv/core`.

.serena/memories/prod-infra/grafana.md

@@ -8,7 +8,7 @@ Keep this memory free of internal names: no host names, no deployed versions, no
 
 - Metrics — **Prometheus**, uid `xkRmyFBGk`. PromQL. Metric families: `penpot_*` (RPC, task timings, CLIMIT, HTTP, websocket, storage), `hikaricp_*`, `jvm_*` and `process_*` on the backend (`job="app"`); `pg_*` from the Postgres exporter (`job="postgres"`); `redis_*` from the Redis exporter (`job="redis"`); `node_*` and `container_*` from node-exporter and cAdvisor. Backend histograms are in **milliseconds**.
 - Logs — **Loki**, uid `h3CGKhEMk`. LogQL, and filter by the structured metadata `logger`/`level` (`{job="penpot"} | logger="..." | level="ERROR"`). Signals that only exist as logs: the profile behind a CLIMIT queue (`logger="app.rpc.climit"`, whose `id` carries the key). CLIMIT rejections have their own counters (`penpot_rpc_climit_rejected_total`, by `name` and `reason`), and their `hint="rejected"` line goes out at WARN only when something stays queued, so counting the lines is not counting the rejections.
-- SQL — **Postgres**, uid `y8ea9tfMk`. Panels need `rawQuery: true`, `editorMode: "code"` and `format: "table"`. The `task` table is the only truth about the queue depth.
+- SQL — **Postgres**, uid `y8ea9tfMk`. Panels need `rawQuery: true`, `editorMode: "code"` and `format: "table"`. The `task` table is the only truth about the queue depth. It is a **read-only replica**: it rejects any write, `CREATE TEMP TABLE` included, so queries through it must be plain `SELECT`s. It also cuts long-running queries (the MCP path has a short timeout), so bound them (e.g. a list of ids); an empty answer there is not proof that there is no data.
 - **Postgres (telemetry)**, uid `9YAUuM_Gz`.
 - `$__rate_interval` and `$__interval` are substituted by Grafana; use them instead of fixed windows in graphs.