github/penpot
1
0
Fork 0
mirror of https://github.com/penpot/penpot.git synced 2026-09-23 20:36:15 +00:00
Code Issues Packages Projects Releases Wiki Activity

✨ Add observability improvements (#11854)

Browse Source 
* 🐳 Add upstream diagnostics to nginx access log

Enrich every access-log line with the internal journey of the request:
the status the backend answered (us), the time spent connecting to it
(uct), the time spent waiting for its answer (urt) and the internal
address that served the request (ua).

A plain 502 line used to say nothing about where the request died. With
this format, the tail of the line classifies the failure: connection
rejected, backend accepted and hung (uct + urt under 1s), or backend
stuck until read timeout. This was the missing witness in the Sep 20
incident, where nginx received connection resets with zero timeouts and
zero rejections.

Applied both to the production image template and the devenv config.
With proxy_pass on variables there is no upstream keepalive, so uct
measures one real TCP connection per request.

Parsing the new fields (us, uct, urt, ua) on the log shipper is left to
ops, so they can be filtered in Loki.

AI-assisted-by: glm-5.3-flash

* 🐳 Add stub_status endpoint for nginx metrics

Add a dedicated localhost-only server (listen 127.0.0.1:8082) exposing
/stub_status next to every other location of the public server. Ops can
run the official nginx-prometheus-exporter as a sidecar against
http://127.0.0.1:8082/stub_status and get nginx_connections_active,
accepted vs handled, reading/writing/waiting and request rates in
Prometheus.

Binding it to localhost and its own server keeps it unreachable from
outside the host and out of the public surface, and access_log off
avoids polluting Loki with one line per Prometheus scrape. The base
image already ships stub_status compiled in, so no image rebuild is
needed.

Applied both to the production image template and the devenv config.

AI-assisted-by: glm-5.3-flash

* ✨ Expose http server gate metrics (worker and connector)

The backend already measured dispatch latency but nothing reported the
state of the "house door": the xnio worker queue and threads, and the
monitor-level listener counters. This was the exact blind spot of the
Sep 20 incident, where the server kept answering health checks while it
accepted connections and dropped them without response.

Add a periodic metrics sampler that lives and dies with the http
server (single daemon thread, 15s interval, each sample guarded so an
unexpected error does not cancel subsequent runs) and publishes:

- worker (xnio MXBean gauges): penpot_http_worker_queue_size,
  busy_threads, pool_size and max_pool_size. Negative samples are
  discarded: the MXBean transiently reports -1 on the busy thread
  count (verified live), and a stale negative would read as zero.
- listener (Undertow connector statistics, enabled via the new
  :server/statistics yetti option): penpot_http_connector_active*
  _connections gauge and requests_total / errors_total counters.
  Undertow exposes absolute totals, so the sampler keeps a watermark
  atom and publishes deltas, skipping (and moving forward past) a
  counter reset.

The connector-level part depends on yetti v11.11, which now accepts
a :server/statistics server option (patch authored and released
upstream; before it, ListenerInfo#getConnectorStatistics always
returned nil).

New tests cover the samplers with fake MXBean/collector statistics
against real prometheus collectors, including the negative-sample
filter, the delta/watermark logic and the sampler lifecycle.

AI-assisted-by: glm-5.3-flash

* 🐛 Include jdk.management in the backend runtime JRE

The production image builds a trimmed JRE with jlink and omitted
jdk.management. Without that module the OS MXBean is
sun.management.BaseOperatingSystemImpl, which has no
getProcessCpuTime, getOpenFileDescriptorCount nor
getMaxFileDescriptorCount. The prometheus client StandardExports
reads those getters reflectively and collect() swallows the
NoSuchMethodException, so process_open_fds, process_max_fds and
process_cpu_seconds_total silently disappeared from /metrics while
the other process_* families kept flowing.

Verified against Prometheus: the app job only ever exposed
process_start_time_seconds, process_virtual_memory_bytes and
process_resident_memory_bytes; the fd and cpu families were absent.
Reproduced locally by running the backend metrics registry on a JRE
built with the same jlink module list (false/false/false) and on one
with jdk.management added (true/true/true).

Add the module to --add-modules and pin the metric contract with
backend-tests.metrics-test.

AI-assisted-by: deepseek-v4.1-flash

* ♻️ Build the http metrics sampler on promesa.exec

Replace the hand-rolled ScheduledThreadPoolExecutor and ThreadFactory
with promesa.exec primitives: px/scheduled-executor with a daemon
thread factory, and a px/schedule chain that reschedules the next
sample when the current one finishes.

Beyond fitting the existing periodic-task pattern (worker/cron,
rpc/rlimit), the chained schedule makes the docstring promise real:
with scheduleAtFixedRate an exception escaping the runnable cancelled
the following executions, while the reschedule now happens in a
finally block.

The sampler shutdown uses px/shutdown-now (shutdown! is deprecated in
promesa 12.0.0) to cancel the pending sample, keeping the previous
halt semantics.

The lifecycle test moves to the promesa predicates and a new test
covers the error-resilience promise: the first sample runs, throws,
and the next one is still scheduled.

AI-assisted-by: deepseek-v4.1-flash

* ♻️ Tighten the http metrics samplers

The samplers are leaf functions: they receive what they need and
publish it. Drop the internal nil guards (if there is no metrics
instance or no mxbean there is nothing to call them for) and move the
checks to the boundary, where the optional data is resolved:
sample-http-metrics now short-circuits with some-> and when-let.

Write the four worker gauges as four static operations instead of a
vector of pairs walked by doseq: the set is fixed, so the collection
only adds an allocation and hides each operation.

Drop the ! suffix from the sample-*-metrics family: ! marks a function
whose contract is to mutate state, while these report, and the mutation
happens in the mtx/run! they call. The constant true return, which only
existed so the removed guard tests could assert it, goes away too.

Tests follow the move: the internal-guard tests are replaced by one
boundary test (a nil server publishes nothing).

AI-assisted-by: deepseek-v4.1-flash

* 📚 Add the function design rules memory

Document the rules that came out of the http metrics sampler review:
preconditions are checked at the boundary instead of re-checked in the
core, optional-by-design data is guarded where the optionality is born,
a fixed set of operations is written statically, ! marks mutation and
not reporting, and production code is not shaped for tests.

Also state in the memory maintenance guide that memories must not use
manual line wrapping.

Linked from critical-info so it is read when designing a solution or
an API, not only when touching the samplers.

AI-assisted-by: deepseek-v4.1-flash

* 📚 Unwrap the critical-info memory lines

The memory maintenance guide forbids manual line wrapping, so rewrite
critical-info with one line per bullet and paragraph. A stray `*` at
the start of one continuation line is dropped.

AI-assisted-by: deepseek-v4.1-flash

* ♻️ Drop the redundant guard in the http server halt

create-metrics-sampler always returns the scheduler, so the sampler is
always present when integrant calls halt-key!; the nil check was dead
code, same as the yt/stop! call next to it.

AI-assisted-by: deepseek-v4.1-flash

* ✨ Add srepl helper to delete profiles by email

Add `delete-profiles-by-email!` to app.srepl.main. It accepts a
single email, a comma separated list of emails or a coll of emails,
resolves each profile, logs it to audit and enqueues the
delete-object task. The deleted-at is backdated with the configured
deletion-delay so profiles and their owned teams are purged on the
next gc pass.

Extract the per-email deletion logic into a private fn and reuse it
from `delete-profiles-in-bulk!`. Add tests for the new
`parse-emails` helper.

AI-assisted-by: glm-5.3-flash
This commit is contained in:
Andrey Antukh 2026-09-23 12:50:18 +02:00
parent 45b8320ac7
commit 5b3844c37a
14 changed files with 706 additions and 99 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

61
.serena/memories/backend/subtleties.md Normal file
View File

@ -0,0 +1,61 @@
# Backend Subtleties
## RPC exposure and wrappers
- RPC commands are discovered from vars created by `app.util.services/defmethod`; adding a command namespace is not enough unless `backend/src/app/rpc.clj` includes it in `resolve-methods`.
- `GET`/`HEAD` RPC calls are only allowed for method names starting with `get-`. Other methods are method-not-allowed even if they are read-only internally.
- RPC auth defaults to enabled. Public endpoints must set `::auth false` metadata explicitly.
- The wrapper stack does auth before params validation, then auditing/rate/concurrency/metrics/retry/condition handling, with DB transaction handling inside that stack. `::db/transaction` metadata controls transaction wrapping.
- Params with `::sm/params` are decoded/conformed through the JSON transformer and successful IObj results get `:encode/json` metadata. Legacy spec conforming only applies when no Malli params schema exists. Client params are stripped of qualified keys (`d/without-qualified`) before merging with the server auth context, so request bodies cannot override `::profile-id`, `::auth-type`, or `::token-perms`.
- Params schemas are open by default, so undeclared client keys reach the handler unless the map is `:closed true`. Creation commands (`create-file`, `create-project`, `create-team`, `create-team-with-invitations`, `upload-file-media-object`, `create-file-media-object-from-url`, `assemble-file-media-object`) use closed schemas: a client-provided `:id` fails with `:params-validation`. Their internal creation functions still accept an optional explicit `:id` for imports, duplicates and deterministic test fixtures.
- Nil RPC bodies become HTTP 204 unless explicit status metadata is present. Stream bodies default to `application/octet-stream` when no content type is set.
## DB helpers
- Most `app.db` helpers accept a pool, connection, or map containing `::db/pool` / `::db/conn`; preserve that convention in shared code.
- `db/tx-run!` uses `next.jdbc.transaction/*nested-tx* :ignore`: nested transaction calls reuse the outer transaction, not a savepoint. Use explicit savepoints when nested rollback semantics matter.
- `db/run!` opens/reuses one connection but does not create a transaction.
- `db/tjson` is Transit JSON for jsonb storage; `db/json` is plain JSON. Worker task props use Transit and are decoded with `decode-transit-pgobject`.
- Advisory transaction locks accept UUIDs or ints. UUID locks are hashed using a zero-UUID seeded siphash.
## Workers and cron
- Task queues are tenant-prefixed. Submit dedupe only removes not-yet-due `new` tasks with the same name/queue/label; it does not dedupe due, scheduled, retry, running, or completed work.
- The dispatcher selects `new`/`retry` tasks with `FOR UPDATE SKIP LOCKED`, marks them `scheduled`, and publishes Redis payload `[id scheduled-at]`. The runner skips Redis messages whose scheduled timestamp no longer matches DB state.
- Lost `scheduled` tasks are rescheduled after about 5 minutes; `running` tasks older than about 24 hours are marked failed as orphans.
- A task handler that is missing or returns an invalid result currently defaults to completed after warning. Throwing with `ex-data :type ::retry` controls retry behavior; `:strategy ::noop` retries without incrementing retry count.
- Cron jobs lock their `scheduled_task` row with `FOR UPDATE SKIP LOCKED`, disable statement/idle-in-transaction timeouts locally, and reschedule themselves in `finally` unless interrupted. Worker, dispatcher, and cron components do not start when the DB pool is read-only.
## Config and HTTP/session middleware
- `app.config/config` and `flags` are dynamic `defonce` vars populated from `PENPOT_*` env vars through the shared schema string transformer. Tests and tooling can bind them.
- `parse-flags` automatically adds `:disable-secure-session-cookies` when `public-uri` is plain HTTP and not localhost. This changes cookie defaults without an explicit env flag.
- The backend sets Clojure `*assert*` globally from the `:backend-asserts` feature flag. Assertion-dependent checks can therefore differ by runtime flags.
- Request body parsing is mostly POST-oriented and supports Transit JSON plus plain JSON. Plain JSON request keys are kebab-decoded before being merged into `:params`.
- Response formatting negotiates with `Accept` or `_fmt=json`. Transit is the default for collection/boolean bodies; JSON encoding has special pointer-map handling.
- Auth prefers the session cookie token before the `Authorization` header. Headers may be `Token` or `Bearer`; JWTs with `kid=1` and `ver=1` are decoded as v1 session tokens, otherwise they are treated as legacy tokens.
- Shared-key auth requires `x-shared-key` as `<key-id> <key>` and stores the lowercased key id on the request. If no shared keys are configured it always rejects.
- Session management uses DB storage unless the DB pool is read-only, then falls back to the in-memory manager. DB sessions support both legacy string ids and v2 UUID session ids.
- Session cookies are renewed when using a legacy string id or when `modified-at` is older than the renewal interval. SameSite is `none` for CORS, otherwise strict/lax based on config.
## HTTP server self-metrics
- `app.http` enables Undertow connection statistics via the yetti option `:server/statistics` (requires yetti ≥ v11.11, which exposes it; before the patch `ListenerInfo#getConnectorStatistics` returned `nil`).
- A daemon sampler built on `promesa.exec` (`px/scheduled-executor` plus a self-rescheduling `px/schedule` chain, so a failing sample never cancels the next one; 15 s, started with the server in `ig/init-key` and stopped with `px/shutdown-now` in `halt-key!`) publishes worker and listener state: `penpot_http_worker_queue_size`, `busy_threads`, `pool_size`, `max_pool_size`, `penpot_http_connector_active_connections`, `requests_total`, `errors_total`. Definitions live in `app.main/default-metrics`.
- The xnio worker MXBean can return transient `-1` (e.g. busy-thread count); negative samples are discarded (gauge keeps its previous value). Undertow exposes absolute request/error totals, so the sampler keeps a watermark atom and publishes deltas; a counter reset (decreasing totals) skips the negative delta and moves the watermark forward.
- The `process_*` families (`process_open_fds`, `process_max_fds`, `process_cpu_seconds_total`, …) come from the prometheus client `StandardExports`, registered by `app.metrics/create-registry`. They read the OS MXBean reflectively and need the `jdk.management` module: on a pruned `jlink` JRE the MXBean is `sun.management.BaseOperatingSystemImpl`, the getters throw `NoSuchMethodException` and `StandardExports#collect` swallows it, so those families silently vanish from `/metrics`. `docker/images/Dockerfile.backend` keeps `jdk.management` in the `--add-modules` list, and `backend-tests.metrics-test` pins the contract.
## Storage and media
- Storage abstraction, backend configuration, logical buckets, object lifecycle, deduplication, access rules, and garbage collection: `mem:backend/storage`.
- SVG validation strips DOCTYPE and uses secure SAX parsing. Basic SVG info falls back to 100x100 dimensions when width/height/viewBox are missing.
- Raster metadata is shell-derived with ImageMagick `identify`, verifies detected MIME against the supplied MIME, and swaps dimensions for EXIF orientations 6/8.
- Remote image download requires 2xx status, `content-length`, a known MIME, and size under the configured maximum before writing the temp file; mismatched byte count is an internal error.
- Font processing shells out to FontForge and WOFF conversion tools and can derive TTF/OTF/WOFF variants from uploaded fonts.
## File data persistence
- File data backends are `legacy-db`, `db`, and `storage`. The storage backend keeps encoded file data in storage bucket `file-data`; the DB row stores metadata with `storage-ref-id` and nil data.
- `fdata/upsert!` touches any storage object referenced by incoming metadata before storing the new row/blob.
- Pointer-map fragments are persisted separately as type `fragment`, and only modified pointer maps are written.
- `fdata/realize` combines pointer realization and object-map realization. Use it before operations that need complete in-memory file data instead of pointer placeholders.

13
.serena/memories/clojure/design-rules.md Normal file
View File

@ -0,0 +1,13 @@
# Clojure Design Rules
How to shape a function in this codebase. Each rule came from a review round; do not re-derive from taste.
- **Preconditions belong to the boundary, not to the core.** A function documents what it needs (hints, docstring) and assumes it. Absence checks live where the value enters the system: the caller that resolves optional objects, `ig/init-key`, the request handler. A `(when (and (some? a) (some? b)) ...)` inside a leaf function means the guard is in the wrong place — if you cannot get `a` or `b`, you should not be calling it.
- **Optional-by-design data is guarded where the optionality is born.** Example: `ConnectorStatistics` is nil when the server option is off, so the guard is `when-let [cs (some-> server ...)]` at the entry point and the consumer assumes `cs`.
- **A static set of operations is written statically.** With a fixed, small set (four metrics) write the four calls. Do not build a collection and iterate it (`doseq` over a literal vector): it allocates, adds indirection and hides each operation. Use a collection when the set is dynamic (config, registry, input).
- **The name says what the function is; `!` says it mutates.** `!` marks functions whose contract is to change state or run a command (`run!`, `submit!`, `swap!`, `shutdown-now`). Reporting helpers (log, sample-and-publish) do not take it: the mutation happens in the `!` API they call. Keep a family consistent.
- **Do not shape production code for tests.** No return values added just to assert them; assert on the observable effect. When preconditions move, move the tests with them.
Reference implementation: the http metrics samplers in `app.http` (`sample-worker-metrics` / `sample-connector-metrics` are guard-free; `sample-http-metrics` guards at the boundary).
See also: `mem:clojure/idioms` (language behaviors), `mem:testing` (TDD and test conventions).

69
.serena/memories/critical-info.md
View File

@ -3,13 +3,11 @@ You are working on the GitHub project `penpot/penpot`, a monorepo.
# Memory system
- Memories are the primary project guidance (not docs or other readme files).
- A section's top-level memory is `<section>/core`. When a section is relevant, read the core memory
before focused memories.
- A section's top-level memory is `<section>/core`. When a section is relevant, read the core memory before focused memories.
- Edits/stale refs/duplication cleanup: `mem:memory-maintenance`.
- Cross-cutting testing principles, TDD workflow, and anti-patterns: `mem:testing`.
- Verified Clojure language behaviors that contradict common assumptions
(e.g. `int?` covers `Long`; `integer?` is the general predicate):
`mem:clojure/idioms` — read before assuming stdlib predicate semantics.
- Verified Clojure language behaviors that contradict common assumptions (e.g. `int?` covers `Long`; `integer?` is the general predicate): `mem:clojure/idioms` — read before assuming stdlib predicate semantics.
- When designing a solution or an API, read `mem:clojure/design-rules` (function shape, boundary checks, naming).
# Development workflow
@ -17,20 +15,14 @@ You are working on the GitHub project `penpot/penpot`, a monorepo.
- Before `git commit` → `mem:workflow/creating-commits` (subject/body format, 76-char body wrapping enforced by `scripts/check-commit`, `AI-assisted-by: model-name` trailer)
- Before `gh issue create` → `mem:workflow/creating-issues` (title derivation, body template, labels, Issue Type)
- Before `gh pr create` / `gh pr edit` → `mem:workflow/creating-prs` (title format, body structure, "Note:" line)
- Before a repo-wide pnpm version update → `mem:workflow/updating-pnpm` (workspace
layout, `scripts/sync-pnpm-version` flow, the stamp-missing-field and
ignored-builds gotchas, verification steps)
- Before a repo-wide pnpm version update → `mem:workflow/updating-pnpm` (workspace layout, `scripts/sync-pnpm-version` flow, the stamp-missing-field and ignored-builds gotchas, verification steps)
- **Never `git push`, force-push, or modify `git origin`** (or any other remote). The user pushes from their own shell; if a push is required, say so and wait. Never amend a commit that the user has already pushed unless explicitly asked.
- **Never edit `CHANGES.md` by hand.** The changelog is generated from GitHub milestones during the release process; update it only via the `update-changelog` skill flow or on explicit user request.
- You have access to the GitHub CLI `gh` or corresponding MCP tools.
- Issues are also managed on Taiga. Read issues using the `read_taiga_issue` tool.
- Before writing code, analyze the task in depth and describe your plan. If the task is complex, break it down into atomic steps.
*After making changes, run the applicable lint and format checks for the affected module before considering the work done (per example `mem:backend/core` or `mem:frontend/core`).
- Align `let` binding values: when a `let` form has multiple bindings spanning
several lines, align the value forms to the same column with spaces.
- If you introduce delimiter errors (mismatched parens/brackets) in Clojure/CLJS files,
fix them with `scripts/paren-repair` BEFORE running lint/format checks.
See `mem:scripts/paren-repair` for usage.
- Before writing code, analyze the task in depth and describe your plan. If the task is complex, break it down into atomic steps. After making changes, run the applicable lint and format checks for the affected module before considering the work done (per example `mem:backend/core` or `mem:frontend/core`).
- Align `let` binding values: when a `let` form has multiple bindings spanning several lines, align the value forms to the same column with spaces.
- If you introduce delimiter errors (mismatched parens/brackets) in Clojure/CLJS files, fix them with `scripts/paren-repair` BEFORE running lint/format checks. See `mem:scripts/paren-repair` for usage.
- Never run anything that destroys data without explicit permission, including `drop-devenv`, `docker compose down -v`, `docker volume rm ...`. The user's real work lives in the volumes of the shared infra.
# Project modules
@ -48,49 +40,28 @@ This is a monorepo. Principles that apply to one module do *not* generally apply
- `docs/`: documentation site; core workflow and conventions: `mem:docs/core`.
- `media-processor/`: TypeScript/Node.js HTTP service for image (sharp) and font (FontForge) processing; core conventions: `mem:media-processor/core`.
The memory is structured in a way that you can get the critical information about the
module. You can read it from `mem:<MODULE>/core`
The memory is structured in a way that you can get the critical information about the module. You can read it from `mem:<MODULE>/core`
# Low-centrality project paths
- `docker/` contains devenv related code, not needed unless specifically instructed.
When working on devenv startup, compose layout, instance config (`defaults.env`),
tmux session lifecycle, RustFS provisioning, or anything in `manage.sh`'s
`*-devenv` commands, read `mem:devenv/core`.
- `docker/` contains devenv related code, not needed unless specifically instructed. When working on devenv startup, compose layout, instance config (`defaults.env`), tmux session lifecycle, RustFS provisioning, or anything in `manage.sh`'s `*-devenv` commands, read `mem:devenv/core`.
- `experiments/` contains standalone experimental HTML/JS/scripts; treat it as non-core unless the user explicitly asks about it.
- `sample_media/` contains sample image/icon media and config used as fixtures/demo material; do not infer app behavior from it.
# Dev Scripts (scripts/)
- `scripts/nrepl-eval.mjs` — Evaluate Clojure/ClojureScript code via nREPL.
Supports `--backend` (port 6064) and `--frontend` (port 3447) aliases.
See `mem:scripts/nrepl-eval`.
- `scripts/paren-repair` — Fix mismatched delimiters in Clojure/CLJS files
and reformat with cljfmt. Run before lint checks when LLM edits break parens.
See `mem:scripts/paren-repair`.
- `scripts/psql` — PostgreSQL client wrapper with devenv defaults.
Companion: `scripts/db-schema` for DDL dumps. See `mem:scripts/psql`.
- `scripts/taiga.py` — Fetch public issues, user stories, and tasks from the
Penpot Taiga project without authentication. See `mem:scripts/taiga`.
- `scripts/gh.py` — GitHub operations helper: list milestone issues, fetch PR
details, compare against CHANGES.md. Requires `gh` CLI. See `mem:scripts/gh`.
- `scripts/error-reports.mjs` — Query error reports via RPC API with token
authentication. Supports list/get operations with filtering and pagination.
See `mem:scripts/error-reports`.
- `scripts/clean-node-modules` — Remove stale `node_modules` from all pnpm
workspaces (root, modules, member packages). Keeps the shared pnpm store
at `<repo>/.pnpm-store` unless `--store`; ignores `external/` and
`.opencode/`. Usage and reinstall steps: `mem:workflow/updating-pnpm`.
- `scripts/ci` — CI orchestration script: runs lint, tests, and format
checks per module (`frontend backend common render-wasm exporter mcp
plugins library`). Logs go to `.ci-logs/`; read the log file on failure.
See `mem:scripts/ci`.
- `scripts/nrepl-eval.mjs` — Evaluate Clojure/ClojureScript code via nREPL. Supports `--backend` (port 6064) and `--frontend` (port 3447) aliases. See `mem:scripts/nrepl-eval`.
- `scripts/paren-repair` — Fix mismatched delimiters in Clojure/CLJS files and reformat with cljfmt. Run before lint checks when LLM edits break parens. See `mem:scripts/paren-repair`.
- `scripts/psql` — PostgreSQL client wrapper with devenv defaults. Companion: `scripts/db-schema` for DDL dumps. See `mem:scripts/psql`.
- `scripts/taiga.py` — Fetch public issues, user stories, and tasks from the Penpot Taiga project without authentication. See `mem:scripts/taiga`.
- `scripts/gh.py` — GitHub operations helper: list milestone issues, fetch PR details, compare against CHANGES.md. Requires `gh` CLI. See `mem:scripts/gh`.
- `scripts/error-reports.mjs` — Query error reports via RPC API with token authentication. Supports list/get operations with filtering and pagination. See `mem:scripts/error-reports`.
- `scripts/clean-node-modules` — Remove stale `node_modules` from all pnpm workspaces (root, modules, member packages). Keeps the shared pnpm store at `<repo>/.pnpm-store` unless `--store`; ignores `external/` and `.opencode/`. Usage and reinstall steps: `mem:workflow/updating-pnpm`.
- `scripts/ci` — CI orchestration script: runs lint, tests, and format checks per module (`frontend backend common render-wasm exporter mcp plugins library`). Logs go to `.ci-logs/`; read the log file on failure. See `mem:scripts/ci`.
# Dependency graph
`frontend -> common`, `backend -> common`, `exporter -> common`, and `frontend -> render-wasm`. Changes in `common` can
affect frontend, backend, exporter, file migrations, and design-library behavior; validate across consumers when
semantics change.
`frontend -> common`, `backend -> common`, `exporter -> common`, and `frontend -> render-wasm`. Changes in `common` can affect frontend, backend, exporter, file migrations, and design-library behavior; validate across consumers when semantics change.
# Working with Penpot designs
@ -101,6 +72,4 @@ semantics change.
## Detecting Crashes
The Penpot frontend can crash silently from the JS API's perspective: `execute_code` calls return successfully, but 1-2s later the workspace becomes unusable (Internal Error page).
The `execute_code` tool then stops working, but `cljs_repl` still works. Use it to detect a crash via `(some? (:exception @app.main.store/state))`.
For details on handling crashes, read memory `mem:frontend/handling-crashes`.
The Penpot frontend can crash silently from the JS API's perspective: `execute_code` calls return successfully, but 1-2s later the workspace becomes unusable (Internal Error page). The `execute_code` tool then stops working, but `cljs_repl` still works. Use it to detect a crash via `(some? (:exception @app.main.store/state))`. For details on handling crashes, read memory `mem:frontend/handling-crashes`.

1
.serena/memories/memory-maintenance.md
View File

@ -21,6 +21,7 @@
Dense agent notes, not prose docs. Prefer invariants, terse bullets.
Avoid obvious context, rationale, and examples unless they prevent likely mistakes.
Keep guidance durable and generalizable, not task-local.
No manual line wrapping: one line per bullet or paragraph, however long. Memories render adapting to the viewport; hard-wrapped lines degrade rendering and diffs.
## Add/update threshold

4
backend/deps.edn
View File

@ -28,8 +28,8 @@
com.google.guava/guava {:mvn/version "33.7.1-jre"}
funcool/yetti
{:git/tag "v11.10"
:git/sha "88701f4"
{:git/tag "v11.11"
:git/sha "e810f87"
:git/url "https://github.com/funcool/yetti.git"
:exclusions [org.slf4j/slf4j-api]}

109
backend/src/app/http.clj
View File

@ -27,11 +27,16 @@
[app.rpc :as-alias rpc]
[app.setup :as-alias setup]
[integrant.core :as ig]
[promesa.exec :as px]
[reitit.core :as r]
[reitit.middleware :as rr]
[yetti.adapter :as yt]
[yetti.request :as yreq]
[yetti.response :as-alias yres]))
[yetti.response :as-alias yres])
(:import
io.undertow.server.ConnectorStatistics
io.undertow.Undertow
org.xnio.management.XnioWorkerMXBean))
(declare router-handler)
@ -45,6 +50,97 @@
::max-body-size 367001600 ; default 350 MiB
})
(def ^:private metrics-sample-interval-ms 15000)
(defn sample-worker-metrics
"Publishes the current state of the xnio worker thread pool (the
request dispatch queue and its threads) as gauges."
[metrics ^XnioWorkerMXBean mxbean]
(let [queue-size (.getWorkerQueueSize mxbean)
busy-count (.getBusyWorkerThreadCount mxbean)
pool-size (.getWorkerPoolSize mxbean)
max-size (.getMaxWorkerPoolSize mxbean)]
;; negative values are missing measurements, not zeros: the xnio
;; MXBean may transiently report -1 on the busy thread count.
(when (>= queue-size 0)
(mtx/run! metrics :id :http-worker-queue-size :val queue-size))
(when (>= busy-count 0)
(mtx/run! metrics :id :http-worker-busy-threads :val busy-count))
(when (>= pool-size 0)
(mtx/run! metrics :id :http-worker-pool-size :val pool-size))
(when (>= max-size 0)
(mtx/run! metrics :id :http-worker-max-pool-size :val max-size))))
(defn sample-connector-metrics
"Publishes the current state of the http listener connection
statistics. Undertow exposes absolute totals, so counters are
published as deltas of the last seen values (the atom state holds the
last observed totals). When a delta comes back negative (mainly
because the underlying counters were reset) the counter is skipped
and the reference updated."
[metrics state ^ConnectorStatistics cs]
(let [{:keys [last-requests last-errors]} (deref state)
total-requests (.getRequestCount cs)
total-errors (.getErrorCount cs)
delta-requests (max 0 (- total-requests last-requests))
delta-errors (max 0 (- total-errors last-errors))]
(when (pos? delta-requests)
(mtx/run! metrics :id :http-connector-requests-total :inc delta-requests))
(when (pos? delta-errors)
(mtx/run! metrics :id :http-connector-errors-total :inc delta-errors))
(mtx/run! metrics
:id :http-connector-active-connections
:val (.getActiveConnections cs))
(swap! state merge {:last-requests total-requests
:last-errors total-errors})))
(defn sample-http-metrics
"Samples the current state of the http server: worker thread pool
state and listener connection statistics. Called periodically by a
sampler that starts together with the server."
[metrics state ^Undertow server]
(try
(when-let [mxbean (some-> server (.getWorker) (.getMXBean))]
(sample-worker-metrics metrics mxbean))
(when-let [cs (some-> server (.getListenerInfo) (first) (.getConnectorStatistics))]
(sample-connector-metrics metrics state cs))
(catch Exception cause
(l/warn :msg "unexpected error on http metrics sampling"
:cause cause))))
(defn create-metrics-sampler
"Creates a daemon scheduler that periodically samples the state of
the http server and publishes it as metrics. A single thread is used,
and an unexpected error on a single sample does NOT cancel the
subsequent runs."
[^Undertow server metrics]
(let [state (atom {:last-requests 0 :last-errors 0})
scheduler (px/scheduled-executor
:parallelism 1
:factory (px/thread-factory :prefix "penpot/http-metrics/"
:daemon true))
sample (fn sample []
(try
(sample-http-metrics metrics state server)
(finally
;; reschedule even if a single sample fails, so
;; an unexpected error does not cancel the
;; following runs.
(px/schedule scheduler metrics-sample-interval-ms sample))))]
(px/schedule scheduler 0 sample)
scheduler))
(defmethod ig/expand-key ::server
[k v]
{k (merge default-params (d/without-nils v))})
@ -83,6 +179,7 @@
:xnio/io-threads (::io-threads cfg)
:xnio/max-worker-threads (::max-worker-threads cfg)
:ring/compat :ring2
:server/statistics true
:events/on-dispatch on-dispatch
:socket/backlog 4069}
@ -98,13 +195,17 @@
(throw (UnsupportedOperationException. "handler or router are required")))
server
(yt/server handler (d/without-nils options))]
(yt/start! (yt/server handler (d/without-nils options)))
(assoc cfg ::server (yt/start! server))))
sampler
(create-metrics-sampler server metrics)]
(assoc cfg ::server server ::metrics-sampler sampler)))
(defmethod ig/halt-key! ::server
[_ {:keys [::server ::port] :as cfg}]
[_ {:keys [::metrics-sampler ::server ::port] :as cfg}]
(l/info :msg "stopping http server" :port port)
(px/shutdown-now metrics-sampler)
(yt/stop! server))
(defn- not-found-handler

37
backend/src/app/main.clj
View File

@ -145,7 +145,42 @@
{::mdef/name "penpot_http_server_dispatch_timing"
::mdef/help "Histogram of dispatch handler"
::mdef/labels []
::mdef/type :histogram}})
::mdef/type :histogram}
:http-worker-queue-size
{::mdef/name "penpot_http_worker_queue_size"
::mdef/help "Current number of queued tasks in the http server xnio worker."
::mdef/type :gauge}
:http-worker-busy-threads
{::mdef/name "penpot_http_worker_busy_threads"
::mdef/help "Current number of busy threads in the http server xnio worker."
::mdef/type :gauge}
:http-worker-pool-size
{::mdef/name "penpot_http_worker_pool_size"
::mdef/help "Current number of threads in the http server xnio worker pool."
::mdef/type :gauge}
:http-worker-max-pool-size
{::mdef/name "penpot_http_worker_max_pool_size"
::mdef/help "Maximum number of threads of the http server xnio worker pool."
::mdef/type :gauge}
:http-connector-active-connections
{::mdef/name "penpot_http_connector_active_connections"
::mdef/help "Current number of active connections in the http listener."
::mdef/type :gauge}
:http-connector-requests-total
{::mdef/name "penpot_http_connector_requests_total"
::mdef/help "Total number of requests handled by the http listener."
::mdef/type :counter}
:http-connector-errors-total
{::mdef/name "penpot_http_connector_errors_total"
::mdef/help "Total number of handler errors in the http listener."
::mdef/type :counter}})
(def system-config
{::db/pool

115
backend/src/app/srepl/main.clj
View File

@ -144,6 +144,66 @@
(db/get-update-count)
(pos?)))))))
(defn parse-emails
"Parse the emails into a seq of cleaned emails. Accepts a single
email, a comma separated list of emails or a coll of emails.
Blank entries are skipped."
[emails]
(->> (cond
(string? emails)
(str/split emails #",")
(sequential? emails)
emails
:else
(throw (ex-info "expected email or comma separated list of emails"
{:emails emails})))
(map str/trim)
(remove str/empty?)))
(defn- delete-profile-by-email*
[system email deleted-at cause]
(when-let [profile (some-> (db/get* system :profile
{:email (str/lower email)}
{::db/remove-deleted false})
(profile/decode-row))]
(audit/insert system
{:name "delete-profile"
:type "action"
:profile-id (:id profile)
:tracked-at deleted-at
:props (audit/profile->props profile)
:context {:triggered-by "srepl"
:cause cause}})
(wrk/invoke! (-> system
(assoc ::wrk/task :delete-object)
(assoc ::wrk/params {:object :profile
:deleted-at deleted-at
:id (:id profile)})))
(:id profile)))
(defn delete-profiles-by-email!
"Mark profiles for deletion by email. Accepts a single email or a
comma separated list of emails (or a coll of emails).
The deletion is immediate: the deleted-at is backdated with the
configured deletion-delay so the profiles and their owned teams are
purged on the next gc pass."
[emails]
(let [emails (parse-emails emails)
deleted-at (ct/minus (ct/now) (cf/get-deletion-delay))
cause "explicit call to delete-profiles-by-email!"]
(db/tx-run! sys/system
(fn [system]
(reduce (fn [acc email]
(if-let [id (delete-profile-by-email* system email deleted-at cause)]
(update acc :deleted conj id)
(update acc :not-found conj email)))
{:total (count emails) :deleted [] :not-found []}
emails)))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; FEATURES
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
@ -745,47 +805,24 @@
(defn delete-profiles-in-bulk!
[system path]
(letfn [(process-data! [system deleted-at emails]
(loop [emails emails
deleted 0
total 0]
(if-let [email (first emails)]
(if-let [profile (some-> (db/get* system :profile
{:email (str/lower email)}
{::db/remove-deleted false})
(profile/decode-row))]
(do
(audit/insert system
{:name "delete-profile"
:type "action"
:profile-id (:id profile)
:tracked-at deleted-at
:props (audit/profile->props profile)
:context {:triggered-by "srepl"
:cause "explicit call to delete-profiles-in-bulk!"}})
(wrk/invoke! (-> system
(assoc ::wrk/task :delete-object)
(assoc ::wrk/params {:object :profile
:deleted-at deleted-at
:id (:id profile)})))
(recur (rest emails)
(inc deleted)
(inc total)))
(recur (rest emails)
deleted
(inc total)))
{:deleted deleted :total total})))]
(let [path (fs/path path)
deleted-at (ct/minus (ct/now) (cf/get-deletion-delay))
cause "explicit call to delete-profiles-in-bulk!"]
(let [path (fs/path path)
deleted-at (ct/minus (ct/now) (cf/get-deletion-delay))]
(when-not (fs/exists? path)
(throw (ex-info "path does not exists" {:path path})))
(when-not (fs/exists? path)
(throw (ex-info "path does not exists" {:path path})))
(db/tx-run! system
(fn [system]
(with-open [reader (io/reader path)]
(process-data! system deleted-at (line-seq reader))))))))
(db/tx-run! system
(fn [system]
(with-open [reader (io/reader path)]
(loop [emails (line-seq reader)
deleted 0
total 0]
(if-let [email (first emails)]
(if (delete-profile-by-email* system email deleted-at cause)
(recur (rest emails) (inc deleted) (inc total))
(recur (rest emails) deleted (inc total)))
{:deleted deleted :total total})))))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; CASCADE FIXING

257
backend/test/backend_tests/http_metrics_test.clj Normal file
View File

@ -0,0 +1,257 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns backend-tests.http-metrics-test
(:require
[app.http :as http]
[app.metrics :as mtx]
[backend-tests.helpers :as th]
[clojure.test :as t]
[promesa.exec :as px])
(:import
io.prometheus.client.CollectorRegistry
io.prometheus.client.Counter
io.prometheus.client.Gauge
io.undertow.server.ConnectorStatistics
java.util.concurrent.ScheduledThreadPoolExecutor
org.xnio.management.XnioWorkerMXBean))
(t/use-fixtures :once th/state-init)
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Helpers
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(def metric-definitions
{:http-worker-queue-size {:name "penpot_http_worker_queue_size"
:help "test"
:type :gauge}
:http-worker-busy-threads {:name "penpot_http_worker_busy_threads"
:help "test"
:type :gauge}
:http-worker-pool-size {:name "penpot_http_worker_pool_size"
:help "test"
:type :gauge}
:http-worker-max-pool-size {:name "penpot_http_worker_max_pool_size"
:help "test"
:type :gauge}
:http-connector-active-connections
{:name "penpot_http_connector_active_connections"
:help "test"
:type :gauge}
:http-connector-requests-total
{:name "penpot_http_connector_requests_total"
:help "test"
:type :counter}
:http-connector-errors-total
{:name "penpot_http_connector_errors_total"
:help "test"
:type :counter}})
(defn- fake-metrics
"Builds a minimal IMetrics instance backed by real prometheus
collectors on a private registry."
[]
(let [registry (CollectorRegistry.)
collectors
(into {}
(map (fn [[id {:keys [name help type]}]]
(let [builder (case type
:gauge (Gauge/build)
:counter (Counter/build))]
(doto builder
(.name name)
(.help help))
[id {:app.metrics.definition/type type
:app.metrics.definition/instance
(.register builder registry)}])))
metric-definitions)]
(reify app.metrics.IMetrics
(get-registry [_] registry)
(get-collector [_ id] (get collectors id))
(get-handler [_] nil))))
(defn- gauge-value
[^Gauge collector]
(.get (.labels collector (make-array String 0))))
(defn- counter-value
[^Counter collector]
(.get (.labels collector (make-array String 0))))
(defn- fake-mxbean
[{:keys [queue busy pool max]
:or {queue 0 busy 0 pool 4 max 512}}]
(reify XnioWorkerMXBean
(getProviderName [_] "test")
(getName [_] "test")
(isShutdownRequested [_] false)
(getCoreWorkerPoolSize [_] 32)
(getMaxWorkerPoolSize [_] max)
(getWorkerPoolSize [_] pool)
(getBusyWorkerThreadCount [_] busy)
(getIoThreadCount [_] 16)
(getWorkerQueueSize [_] queue)
(getServerMXBeans [_] #{})))
(defn- fake-connector-statistics
[{:keys [requests errors active]
:or {requests 0 errors 0 active 0}}]
(reify ConnectorStatistics
(getRequestCount [_] requests)
(getBytesSent [_] 0)
(getBytesReceived [_] 0)
(getErrorCount [_] errors)
(getProcessingTime [_] 0)
(getMaxProcessingTime [_] 0)
(getActiveConnections [_] active)
(getMaxActiveConnections [_] active)
(getActiveRequests [_] 0)
(getMaxActiveRequests [_] 0)
(reset [_] nil)))
(defn- collector-instance
[metrics id]
(:app.metrics.definition/instance (mtx/get-collector metrics id)))
(defn- make-state []
(atom {:last-requests 0 :last-errors 0}))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Test: worker metrics
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(t/deftest sample-worker-metrics-publish-all-gauges
(let [metrics (fake-metrics)
mxbean (fake-mxbean {:queue 2 :busy 10 :pool 40 :max 512})]
(http/sample-worker-metrics metrics mxbean)
(t/is (= 2.0 (gauge-value (collector-instance metrics :http-worker-queue-size))))
(t/is (= 10.0 (gauge-value (collector-instance metrics :http-worker-busy-threads))))
(t/is (= 40.0 (gauge-value (collector-instance metrics :http-worker-pool-size))))
(t/is (= 512.0 (gauge-value (collector-instance metrics :http-worker-max-pool-size))))))
(t/deftest sample-worker-metrics-skips-negative-samples
;; the xnio MXBean occasionally returns -1 on the busy thread count;
;; a negative value is a missing measurement, not a zero.
(let [metrics (fake-metrics)
mxbean (fake-mxbean {:queue 0 :busy -1 :pool 4 :max 512})]
(http/sample-worker-metrics metrics mxbean)
(t/is (= 0.0 (gauge-value (collector-instance metrics :http-worker-queue-size))))
(t/is (= 4.0 (gauge-value (collector-instance metrics :http-worker-pool-size))))
(t/is (= 512.0 (gauge-value (collector-instance metrics :http-worker-max-pool-size))))
(t/is (= 0.0
(gauge-value (collector-instance metrics :http-worker-busy-threads))))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Test: connector metrics
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(t/deftest sample-connector-metrics-publishes-gauges-and-counters
(let [metrics (fake-metrics)
state (make-state)
cs (fake-connector-statistics {:requests 10 :errors 2 :active 5})]
(http/sample-connector-metrics metrics state cs)
(t/is (= 5.0 (gauge-value (collector-instance metrics :http-connector-active-connections))))
(t/is (= 10.0 (counter-value (collector-instance metrics :http-connector-requests-total))))
(t/is (= 2.0 (counter-value (collector-instance metrics :http-connector-errors-total))))))
(t/deftest sample-connector-metrics-accumulates-delta
(let [metrics (fake-metrics)
state (make-state)]
(http/sample-connector-metrics metrics state (fake-connector-statistics {:requests 10 :errors 0 :active 1}))
(http/sample-connector-metrics metrics state (fake-connector-statistics {:requests 25 :errors 0 :active 1}))
(t/is (= 25.0 (counter-value (collector-instance metrics :http-connector-requests-total))))
(t/is (= 0.0 (counter-value (collector-instance metrics :http-connector-errors-total))))))
(t/deftest sample-connector-metrics-skips-negative-delta
;; when the undertow counters are reset, the computed delta can go
;; negative: the counter must not decrease, and the reference must be
;; updated so the next sampling continues from the new watermark.
(let [metrics (fake-metrics)
state (make-state)]
(http/sample-connector-metrics metrics state (fake-connector-statistics {:requests 20 :errors 5 :active 0}))
(http/sample-connector-metrics metrics state (fake-connector-statistics {:requests 10 :errors 3 :active 0}))
(http/sample-connector-metrics metrics state (fake-connector-statistics {:requests 15 :errors 6 :active 0}))
(t/is (= 25.0 (counter-value (collector-instance metrics :http-connector-requests-total))))
(t/is (= 8.0 (counter-value (collector-instance metrics :http-connector-errors-total))))))
(t/deftest sample-connector-metrics-state-advances-with-reset
;; after a reset (total decreased) followed by more requests, the
;; next delta must be computed from the new watermark and count only
;; the requests after the reset.
(let [metrics (fake-metrics)
state (make-state)]
(http/sample-connector-metrics metrics state (fake-connector-statistics {:requests 10 :errors 0 :active 0}))
(http/sample-connector-metrics metrics state (fake-connector-statistics {:requests 5 :errors 0 :active 0})) ; reset to 5
(http/sample-connector-metrics metrics state (fake-connector-statistics {:requests 8 :errors 0 :active 0}))
(t/is (= 13.0 (counter-value (collector-instance metrics :http-connector-requests-total))))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Test: sampler lifecycle
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(t/deftest sample-http-metrics-on-nil-server-does-nothing
;; the guards live at the boundary: without a server there is nothing
;; to sample and nothing must be published.
(let [metrics (fake-metrics)
state (make-state)]
(http/sample-http-metrics metrics state nil)
(t/is (= 0.0 (gauge-value (collector-instance metrics :http-worker-queue-size))))
(t/is (= 0.0
(counter-value (collector-instance metrics :http-connector-requests-total))))))
(t/deftest create-metrics-sampler-lifecycle
;; a smoke test of the lifecycle wiring: the sampler is created with
;; a running scheduler and ends up shut down.
(let [metrics (fake-metrics)
sampler (http/create-metrics-sampler nil metrics)]
(try
(t/is (some? sampler))
(t/is (px/executor? sampler))
(t/is (not (px/shutdown? sampler)))
(finally
(px/shutdown-now sampler)
(t/is (px/shutdown? sampler))))))
(t/deftest create-metrics-sampler-reschedules-after-error
;; the docstring promise: an unexpected error on a single sample must
;; not cancel the following runs. The first sample runs immediately
;; and throws; the next one must still be scheduled afterwards.
(let [calls (atom 0)]
(with-redefs [http/sample-http-metrics (fn [_ _ _]
(swap! calls inc)
(throw (ex-info "boom" {})))]
(let [sampler (http/create-metrics-sampler nil (fake-metrics))
queue (.getQueue ^ScheduledThreadPoolExecutor sampler)]
(try
(t/is (loop [i 0]
(cond (pos? @calls) true
(> i 200) false
:else (do (Thread/sleep 10) (recur (inc i)))))
"the first sample must run immediately")
(t/is (loop [i 0]
(cond (= 1 (.size queue)) true
(> i 200) false
:else (do (Thread/sleep 10) (recur (inc i)))))
"the next sample must be scheduled after the error")
(finally
(px/shutdown-now sampler)))))))

49
backend/test/backend_tests/metrics_test.clj Normal file
View File

@ -0,0 +1,49 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns backend-tests.metrics-test
(:require
[app.metrics :as mtx]
[clojure.test :as t]
[integrant.core :as ig])
(:import
io.prometheus.client.Collector$MetricFamilySamples
io.prometheus.client.Collector$MetricFamilySamples$Sample))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Helpers
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defn- sample-names
[metrics]
(->> (mtx/get-registry metrics)
(.metricFamilySamples)
(enumeration-seq)
(mapcat (fn [^Collector$MetricFamilySamples family]
(map (fn [^Collector$MetricFamilySamples$Sample sample]
(.-name sample))
(.samples family))))
(set)))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Tests
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(t/deftest process-metrics-are-exported
;; the process cpu and file descriptor families come from the
;; prometheus client `StandardExports`, registered by `app.metrics`.
;; They are read reflectively from the OS MXBean and depend on the
;; `jdk.management` module at runtime: a pruned jlink JRE turns the
;; MXBean into `sun.management.BaseOperatingSystemImpl`, the reflective
;; getters fail and the families are silently dropped (that is how the
;; production backend lost `process_open_fds`). This test pins the
;; contract the fd alert relies on.
(let [metrics (ig/init-key :app.metrics/metrics {:default {}})
names (sample-names metrics)]
(t/is (contains? names "process_open_fds"))
(t/is (contains? names "process_max_fds"))
(t/is (contains? names "process_cpu_seconds_total"))))

28
backend/test/backend_tests/srepl_main_test.clj Normal file
View File

@ -0,0 +1,28 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns backend-tests.srepl-main-test
(:require
[app.srepl.main :as srepl]
[clojure.test :as t]))
(t/deftest parse-emails
(t/is (= ["some@example.com"]
(srepl/parse-emails "some@example.com")))
(t/is (= ["some@example.com" "other@example.com"]
(srepl/parse-emails "some@example.com,other@example.com")))
(t/is (= ["some@example.com" "other@example.com"]
(srepl/parse-emails " some@example.com , other@example.com ,")))
(t/is (= ["some@example.com" "other@example.com"]
(srepl/parse-emails ["some@example.com" "other@example.com"])))
(t/is (= [] (srepl/parse-emails ",")))
(t/is (thrown? clojure.lang.ExceptionInfo
(srepl/parse-emails 42))))

27
docker/devenv/files/nginx.conf
View File

@ -31,7 +31,16 @@ http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
access_log /var/log/nginx/access.log;
# Access log enriched with upstream diagnostics: what the backend
# answered (us), how long it took to connect to it (uct), how long it
# took to answer (urt) and which internal address served the request
# (ua). With proxy_pass on variables there is no upstream keepalive,
# so uct measures one real TCP connection per request.
log_format penpot_upstream '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" "$http_user_agent" '
'us=$upstream_status uct=$upstream_connect_time '
'urt=$upstream_response_time ua=$upstream_addr';
access_log /var/log/nginx/access.log penpot_upstream;
error_log /var/log/nginx/error.log;
gzip on;
@ -297,4 +306,20 @@ http {
try_files $uri /index.html$is_args$args /index.html =404;
}
}
# Dedicated health endpoint for the optional nginx-prometheus-exporter
# sidecar (scraping http://127.0.0.1:8082/stub_status). Bound to
# localhost only and out of the public server, so it can not be
# reached from outside the host. Counts client-side connections only;
# it says nothing about the upstream pools.
server {
listen 127.0.0.1:8082;
server_name _;
access_log off;
location = /stub_status {
stub_status;
}
}
}

7
docker/images/Dockerfile.backend
View File

@ -18,6 +18,11 @@ RUN set -ex; \
apt-get clean; \
rm -rf /var/lib/apt/lists/*
# NOTE: jdk.management is required by the prometheus client hotspot
# exports. Without it the OS MXBean is sun.management.BaseOperatingSystemImpl,
# which has no getOpenFileDescriptorCount, getMaxFileDescriptorCount nor
# getProcessCpuTime, so process_open_fds, process_max_fds and
# process_cpu_seconds_total silently disappear from /metrics.
RUN set -eux; \
ARCH="$(dpkg --print-architecture)"; \
case "${ARCH}" in \
@ -44,7 +49,7 @@ RUN set -eux; \
--no-header-files \
--no-man-pages \
--strip-debug \
--add-modules java.base,jdk.net,jdk.management.agent,java.se,jdk.compiler,jdk.javadoc,jdk.attach,jdk.unsupported,jdk.jfr,jdk.jcmd \
--add-modules java.base,jdk.net,jdk.management,jdk.management.agent,java.se,jdk.compiler,jdk.javadoc,jdk.attach,jdk.unsupported,jdk.jfr,jdk.jcmd \
--output /opt/jre;

28
docker/images/files/nginx.conf.template
View File

@ -31,7 +31,17 @@ http {
default_type application/octet-stream;
error_log /dev/stderr;
access_log /dev/stdout;
# Access log enriched with upstream diagnostics: what the backend
# answered (us), how long it took to connect to it (uct), how long it
# took to answer (urt) and which internal address served the request
# (ua). With proxy_pass on variables there is no upstream keepalive,
# so uct measures one real TCP connection per request.
log_format penpot_upstream '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" "$http_user_agent" '
'us=$upstream_status uct=$upstream_connect_time '
'urt=$upstream_response_time ua=$upstream_addr';
access_log /dev/stdout penpot_upstream;
proxy_connect_timeout 300s;
proxy_send_timeout 300s;
@ -186,4 +196,20 @@ http {
}
}
# Dedicated health endpoint consumed by the nginx-prometheus-exporter
# sidecar (deployed by ops, scraping http://127.0.0.1:8082/stub_status).
# Bound to localhost only and out of the public server, so it can not
# be reached from outside the host. Counts client-side connections
# only; it says nothing about the upstream pools.
server {
listen 127.0.0.1:8082;
server_name _;
access_log off;
location = /stub_status {
stub_status;
}
}
}