Andrey Antukh 9c5260314a
🐛 Revoke websocket subscriptions when access is revoked (#12189)
* 🐛 Revoke websocket subscriptions when access is revoked

GHSA-m53j-2766-6jqw. A websocket subscription was authorized once,
when it opened, and that decision was trusted for the rest of the
connection's life, so a member whose access was revoked kept receiving
live file content as long as the tab stayed open.

The revocation travels over the message bus instead of closing the
socket from the mutation: the connection registry is local to one
backend instance, while the RPC that revokes access runs on whichever
instance received the request. Every instance re-verifies the
subscriptions it owns with the same `has-read-permissions?` predicates
that `:subscribe-file` and `:subscribe-team` authorize with, so a
downgrade to viewer keeps its subscription and a non-member
organization owner keeps read-only access.

Two mechanisms cover each other:

- Access-changing mutations announce after the write. delete-team-
  member, leave-team, update-team-member-role, delete-file,
  delete-project, move-files, move-project and delete-team announce.
  delete-share-link is deliberately excluded: a share-link visitor
  never passes the subscribe permission check, so there is no
  subscription to revoke. Team-wide changes publish one event per
  team; the watcher resolves the members itself, and both publishers
  defer through `db/after-commit` so the watcher never decides from
  pre-commit state.
- Every relay re-checks access every 5 minutes by default
  (`subscription-revalidation-interval`), driven from the relay loop
  with `alts!` over the subscription channel and a one-shot tick
  channel, so a steady stream of edits cannot push the check away
  forever.

Supporting changes:

- The registry holds `:connections` and `:by-profile` in one atom,
  kept consistent by register/unregister in a single swap!; a profile
  lookup is one map access, and a profile whose last connection
  leaves is dropped from the index.
- `close-file-subscription` and `close-team-subscription` announce
  the :leave-* message, close the relay channel and purge the bus
  subscription, and are idempotent. Closing the channel ends the
  relay loop and its re-check task.
- The permission query the re-check uses filters soft-deleted files,
  projects and teams, matching its sibling queries.
- A transient database error fails open: it logs and keeps the
  subscription; the next tick retries.
- The revocation watcher's params schema declared the nitrate client
  as a bare keyword inside [:map ...], which malli resolved as a
  registry lookup and which took the whole system build down under
  enable-backend-asserts. It is now an explicit pair, optional
  because init-key yields nil with the admin-console flag off.

The tests reproduce the leak against the real message bus, including
the negative control for the interval re-check. The written records
live in mem:backend/websocket-subscriptions.

Closes #12155

AI-assisted-by: space-bunny-free
AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Mirror the subscribe gates in the revocation re-checks

Addresses the second review round on PR #12189 (pabloalba).

The re-checks only asked whether the profile could read. The membership
legs of sql:file-permissions filter soft-deleted rows since the F2 fix,
but the organization-owner fallback grants read-only access over HTTP
without looking at deleted_at, so a soft-deleted file or team kept an
expelled org owner subscribed indefinitely. The re-checks now gate on
existence as well: file-readable? and team-readable? require a live row
(db/get*, non-throwing, soft-deleted rows excluded) plus the read
permission, mirroring the :subscribe-file gates and extending them to
teams. Both relays and revalidate-profile-subscriptions use them; the
subscribe handlers keep their exact behavior.

Also from the same review:

- watcher-consumes-revocation-events-from-the-bus no longer races the
  command's after-commit announcement: it takes access away with a
  direct row delete and drives the watcher with a synthetic bus event,
  instead of deleting through the command and publishing a second
  notify-* that would mask a broken command announcement.
- moving-a-project-closes-source-team-subscriptions pins the
  move-project e2e: the file subscription drops, the source-team
  subscription survives (membership unchanged).
- soft-deleted-file/team-closes-org-owner-subscription cover the new
  gate with red-before/green-after tests.

mem:backend/websocket-subscriptions updated.

Closes #12155

AI-assisted-by: inkling-2-small-free

* 🐛 Refuse dead teams in resolve-team-id (PR #12189 review)

Second review round on PR #12189 (pabloalba).

The file re-check added in 87ca7aca53 stops at the file row, but the
organization-owner fallback grants read-only access to non-member org
owners via Nitrate over HTTP, which knows nothing about Penpot
soft-deletion. resolve-team-id resolved the team from unfiltered SQL,
so an expelled org owner kept the file subscription open for an
indefinitely soft-deleted file, project or team. Now both SQL lookups
filter deleted_at, and resolve-team-id checks the team row itself
(db/get*, which filters soft-deleted rows) for every caller — teams,
projects and files alike.

Also from the same review:

- file-readable? now uses the same sql:get-minimal-file as the
  subscribe gate (non-throwing via throw-if-not-exists false), and
  team-readable? projects id and deleted-at, so db/get* can filter.
- The test suite extracts the thrice-duplicated raw delete into
  delete-team-membership!; a sibling of the fail-open pin now redefs
  the existence probe to throw; memory and docstring aligned with the
  code.
- docs/technical-guide/developer/architecture/backend.md attributed
  the websocket to notifications.clj; it now names app.http.websocket
  as the owner and notifications.clj as the revocation publisher.
- mem:backend/auth-permissions-product-domains documents the fallback
  mechanism and the resolve-team-id contract.

Closes #12155

AI-assisted-by: inkling-2-small-free

* 🐛 Test resolve-team-id refusal and tidy review findings

Third pass over PR #12189 after review-code on 225eb98de2.

resolve-team-id now has a direct test: all three branches resolve
while rows are live, and file, project and team soft-deletes each
refuse their branch — the two new SQL predicates and the final team-row
check are all pinned by it.

From the same review:

- failing-existence-check-keeps-the-subscription and the sibling
  a-failing-check-keeps-the-subscription are deleted: they could not
  distinguish a live relay from a dead one. The fail-open policy
  remains documented in the docstring and the memory; the memory no
  longer claims the pins exist.
- team-readable? projects ::sql/columns like the rest of the backend;
  the fully-qualified keyword went away with the new alias.
- The dead when-not guard in get-organization-owner-viewer-team is
  removed (resolve-team-id guarantees the row live); the let binding
  in resolve-team-id no longer shadows the destructured team-id.
- docs/technical-guide names rpc/notifications.clj as the publisher
  and the tree lists the real top-level modules.

Closes #12155

AI-assisted-by: inkling-2-small-free

* 🐛 Make the revocation tests deterministic

Third review round on PR #12189 (pabloalba).

Both tests took access away through a command that announces, so the
live watcher could close the subscriptions before the test looked:

- revocation-closes-subscriptions-that-are-no-longer-authorized
  asserted an exact count of 2 closes after delete-team-member, and
  that count reads 0 whenever the watcher got there first. It now
  removes the member with delete-team-membership!, the raw row delete
  that publishes nothing, and asserts both subscriptions are still
  open before calling revalidate-profile-subscriptions.
- team-event-closes-every-member-subscription ran delete-team and then
  published a second notify-team-permissions-changed, so a missing
  announcement in delete-team still passed. It now soft-deletes the
  team row (sql:team-members joins team_profile_rel and profile only,
  so the watcher still resolves every member), asserts nothing closed,
  and drives one synthetic team event.

The command paths stay covered by the announce and end-to-end tests
for delete-team-member and delete-team. The file split the reviewer
suggested stays the optional follow-up, tracked outside this PR and
waiting for the merge.

Closes #12155

AI-assisted-by: deepseek-v4.1-flash
2026-11-13 18:27:30 +01:00

143 lines
6.6 KiB
Markdown

---
title: Backend app
desc: Dive into self-hosting, configuration, developer insights (architecture, data model), integration, and troubleshooting. See Penpot's Technical Guide.
---
# Backend app
This app is in charge of CRUD of data, integrity validation and persistence
into a database and also into a file system for media attachments.
To handle deletions it uses a garbage collector mechanism: no object in the
database is deleted instantly. Instead, a field <code class="language-bash">deleted_at</code> is set with the
date and time of the deletion, and every query ignores db rows that have this
field set. Then, an async task that runs periodically, locates rows whose
deletion date is older than a given threshold and permanently deletes them.
For this, and other possibly slow tasks, there is an internal async tasks
worker, that may be used to queue tasks to be scheduled and executed when the
backend is idle. Other tasks are email sending, collecting data for telemetry
and detecting unused media attachment, for removing them from the file storage.
## Backend structure
Penpot backend app code resides under <code class="language-text">backend/src/app</code> path in the main repository.
@startuml BackendGeneral
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Container.puml
!define DEVICONS https://raw.githubusercontent.com/tupadr3/plantuml-icon-font-sprites/master/devicons
!include DEVICONS/react.puml
!include DEVICONS/java.puml
!include DEVICONS/clojure.puml
!include DEVICONS/postgresql.puml
!include DEVICONS/redis.puml
!include DEVICONS/chrome.puml
HIDE_STEREOTYPE()
Container(frontend_app, "Frontend app", "React / ClojureScript", "", "react")
System_Boundary(backend, "Backend") {
Container(backend_app, "Backend app", "Clojure / JVM", "", "clojure")
ContainerDb(db, "Database", "PostgreSQL", "", "postgresql")
ContainerDb(redis, "Broker", "Redis", "", "redis")
}
BiRel(frontend_app, backend_app, "Open", "websocket")
Rel(frontend_app, backend_app, "Uses", "RPC API")
Rel(backend_app, db, "Uses", "SQL")
Rel(redis, backend_app, "Subscribes", "pub/sub")
Rel(backend_app, redis, "Notifies", "pub/sub")
@enduml
```
▾ backend/src/app/
▸ cli/
▸ http/
▸ migrations/
▸ rpc/
▸ setup/
▸ srepl/
▸ util/
▸ tasks/
main.clj
config.clj
health.clj
http.clj
jobs.clj
metrics.clj
migrations.clj
nitrate.clj
rpc.clj
setup.clj
srepl.clj
system.clj
worker.clj
...
```
* <code class="language-text">main.clj</code> defines the app global settings and the main entry point of the
application, served by a JVM.
* <code class="language-text">config.clj</code> defines of the configuration options read from linux
environment.
* <code class="language-text">http</code> contains the HTTP server and the backend routes list.
* <code class="language-text">migrations</code> contains the SQL scripts that define the database schema, in
the form of a sequence of migrations.
* <code class="language-text">rpc</code> is the main module to handle the RPC API calls.
* <code class="language-text">http/websocket.clj</code> manages the websocket endpoint: clients subscribe
to open files or teams and the server keeps the subscriptions; revoked access
is re-checked and dropped. <code class="language-text">rpc/notifications.clj</code> publishes
access-change events on the bus so the websocket re-checks the subscriptions
it owns.
* <code class="language-text">setup</code> initializes the environment (loads config variables, sets up the
database, executes migrations, loads initial data, etc).
* <code class="language-text">srepl</code> sets up an interactive REPL shell, with some useful commands to be
used to debug a running instance.
* <code class="language-text">cli</code> sets a command-line interface, with some more maintenance commands.
* <code class="language-text">metrics.clj</code> has some interceptors that watches RPC calls, calculate
statistics and other metrics, and send them to external systems to store and
analyze.
* <code class="language-text">worker.clj</code> and <code class="language-text">tasks</code> define some async tasks that are executed in
parallel to the main http server (using java threads), and scheduled in a
cron-like table. They are useful to do some garbage collection, data packing
and similar periodic maintenance tasks.
* <code class="language-text">db.clj</code>, <code class="language-text">emails.clj</code>, <code class="language-text">media.clj</code>, <code class="language-text">msgbus.clj</code>, <code class="language-text">storage.clj</code>,
<code class="language-text">rlimits.clj</code> are general libraries to use I/O resources (SQL database,
send emails, handle multimedia objects, use REDIS messages, external file
storage and semaphores).
* <code class="language-text">util/</code> has a collection of generic utility functions.
### RPC calls
The RPC (Remote Procedure Call) subsystem consists of a mechanism that allows
to expose clojure functions as an HTTP endpoint. We take advantage of being
using Clojure at both front and back ends, to avoid needing complex data
conversions.
1. Frontend initiates a "query" or "mutation" call to <code class="language-text">:xxx</code> method, and
passes a Clojure object as params.
2. Params are string-encoded using
[transit](https://github.com/cognitect/transit-clj), a format similar to
JSON but more powerful.
3. The call is mapped to <code class="language-text"><backend-host>/api/rpc/query/xxx</code> or
<code class="language-text"><backend-host>/api/rpc/mutation/xxx</code>.
4. The <code class="language-text">rpc</code> module receives the call, decode the parameters and executes the
corresponding method inside <code class="language-text">src/app/rpc/queries/</code> or <code class="language-text">src/app/rpc/mutations/</code>.
We have created a <code class="language-text">defmethod</code> macro to declare an RPC method and its
parameter specs.
5. The result value is also transit-encoded and returned to the frontend.
This way, frontend can execute backend calls like it was calling an async function,
with all the power of Clojure data structures.
### PubSub
To manage subscriptions to a file, to be notified of changes, we use a redis
server as a pub/sub broker. Whenever a user visits a file and opens a
websocket, the backend creates a subscription in redis, with a topic that has
the id of the file. If the user sends any change to the file, backend sends a
notification to this topic, that is received by all subscribers. Then the
notification is retrieved and sent to the user via the websocket.