# Backend Architecture and Workflow Backend: JVM Clojure; Integrant; PostgreSQL; Redis/Valkey; RPC; HTTP; storage; mail; audit/logging; workers. ## Focused memories - Cross-cutting backend subtleties (RPC, DB, workers, cron, HTTP/sessions, storage, file data): `mem:backend/subtleties` - Storage abstraction, logical buckets, object lifecycle, deduplication, access, and garbage collection: `mem:backend/storage`. - Embedded Ladybug graph experiment, projection, incremental sync, console, and risks: `mem:backend/graph-experiment` - Auth flows, permission model, teams, projects, invitations, comments, webhooks, audit: `mem:backend/auth-permissions-product-domains` - Audit-log event collection (RPC wrapper, frontend ingestion), telemetry duality, webhook fan-out, error reporters, Nexus archival and retention: `mem:backend/audit-log` - Services, task-queue/Pub-Sub topology constraints -> `mem:prod-infra/core`. ## Stable namespace map - `app.rpc.commands.*`: RPC command implementations exposed under `/api/rpc/command/`. - `app.rpc.permissions`: permission predicate/check helper factories. - `app.http.*`: HTTP routes and middleware. - `app.auth.*`: provider-specific authentication helpers such as LDAP/OIDC. - `app.loggers.*`: audit, webhook, database, and external log integrations. - `app.db.*` / `app.db`: next.jdbc wrapper and SQL helpers. - `app.tasks.*`: background task handlers. - `app.worker`: task execution/cron plumbing. - `app.main`: Integrant system map and component wiring. - `app.config`: `PENPOT_*` env config and feature flags. - `app.srepl.*`: development REPL helpers for manual backend operations (data inspection, migration helpers, one-off admin tasks). - `app.nitrate`, `app.rpc.commands.nitrate`, and `app.rpc.management.nitrate`: external Nitrate subscription/organization integration, gated by the `:nitrate` feature flag and shared-key HTTP calls. ## RPC conventions RPC commands are defined with `app.util.services/defmethod` and schemas. Use `get-` prefixes for read operations. Command metadata usually includes auth, docs version, params schema, and result schema. Return plain maps/vectors or raise structured exceptions from `app.common.exceptions`. Backend RPC command areas without focused memories include access tokens, binfile, demo, feedback, file snapshots, fonts, management, Nitrate, and webhooks beyond the notes in `mem:backend/auth-permissions-product-domains`; inspect nearby command tests and command metadata before changing them. ## DB conventions `app.db` helpers accept cfg, pool, or conn in most places and convert kebab-case to snake_case: - `db/get`, `db/get*`, `db/query`, `db/insert!`, `db/update!`, `db/delete!`. - Use `db/run!` for multiple operations on one connection. - Use `db/tx-run!` for transactions. Database migrations live in `backend/src/app/migrations/`; pure SQL migrations are under `backend/src/app/migrations/sql/`. SQL filenames conventionally start with a sequence and verb/table description, e.g. `0026-mod-profile-table-add-is-active-field`. Applied migrations are tracked in the `migrations` table. For interactive PostgreSQL access with correct dev defaults, use `scripts/psql`; to dump the current DDL schema, use `scripts/db-schema` (see `mem:scripts/psql`). For deeper details on transaction semantics, advisory locks, Transit vs JSON helpers, and dev/test DB URLs: `mem:backend/subtleties`. ## Background tasks A task handler is an Integrant component with `ig/assert-key`, `ig/expand-key`, and `ig/init-key`, returning the function run by the worker. New tasks also need wiring in `app.main`: handler config, worker registry entry, and cron entry if scheduled. For worker dispatch, cron, retry semantics, deduplication, and queue internals: `mem:backend/subtleties`. ## REPL In devenv, backend nREPL is exposed on port 6064. ### Non-interactive eval (preferred for agents) `./scripts/nrepl-eval.mjs` connects to an already-running nREPL server and evaluates code. Session state (defs, `in-ns`) persists across invocations via a stored session ID in `/tmp/penpot-nrepl-session--`. ```bash ./scripts/nrepl-eval.mjs '(+ 1 2)' # single expression ./scripts/nrepl-eval.mjs "(require '[my.ns :as ns] :reload)" # reload after edits ./scripts/nrepl-eval.mjs -e # inspect last exception (*e) ./scripts/nrepl-eval.mjs --reset-session '(def x 0)' # discard session, start fresh ./scripts/nrepl-eval.mjs <<'EOF' # multi-expression heredoc (def x 10) (+ x 20) EOF ``` Default port is 6064. Use `-p ` for a different port. Use `-t ` to override the 120s timeout. Do not start the nREPL server — assume it is already running. ### Interactive REPL `backend/scripts/nrepl` starts a REPLy client connected to the running nREPL. For an in-process backend REPL (where you control the JVM lifecycle), stop the running backend first so port 9090 is free, then run `backend/scripts/repl`. Useful top-level helpers include `(start)`, `(stop)`, `(restart)`, `(run-tests)`, and `(repl/refresh-all)`. Many `app.srepl.main` helpers accept the global `system` var, e.g. manual email or maintenance operations. ## Fixtures Fixtures can populate local data for manual testing/perf work. From the backend REPL, run `(app.cli.fixtures/run {:preset :small})`; fixture users conventionally look like `profileN@example.com` with password `123123`. Standalone fixture aliases may exist, but check current `backend/deps.edn` before relying on old command names. ## Performance * **Type Hinting:** Use explicit JVM type hints (e.g. `^String`, `^long`) in performance-critical paths to avoid reflection overhead. * **Batch inserts:** Use `db/insert-many!` for bulk row inserts — generates a single SQL with multiple parameter tuples. Avoid on very large datasets (SQL length / parameter count limits). * **Server-side cursors:** Use `db/plan` (fetch-size 1000, forward-only, read-only) or `db/cursor` for large result sets. Never fetch large collections into memory at once. * **Transaction discipline:** Use `tx-run!` for writes (opens a transaction), `run!` for reads (single connection, no transaction). Set `:read-only` on `tx-run!` when applicable to let PostgreSQL optimize. ## Lint and Format IMPORTANT: all CLI commands must be executed from the `backend/` subdirectory. * **Linting:** `pnpm run lint:clj`. * **Formatting:** `pnpm run check-fmt:clj` to check, `pnpm run fmt:clj` to fix. After running `fmt:clj`, `check-fmt:clj` is redundant. Avoid unrelated whitespace diffs. **Before linting:** if delimiter errors are suspected (after LLM edits), run `scripts/paren-repair` on the affected files first. Delimiter errors produce misleading linter/compiler output. See `mem:scripts/paren-repair`. ## Testing Backend test commands, coverage rules, and conventions: `mem:backend/testing`. Cross-cutting testing principles, anti-patterns, and verification checklist: `mem:testing`.