mirror of
https://github.com/penpot/penpot.git
synced 2026-11-16 23:20:37 +00:00
* 🐛 Revoke websocket subscriptions when access is revoked GHSA-m53j-2766-6jqw. A websocket subscription was authorized once, when it opened, and that decision was trusted for the rest of the connection's life, so a member whose access was revoked kept receiving live file content as long as the tab stayed open. The revocation travels over the message bus instead of closing the socket from the mutation: the connection registry is local to one backend instance, while the RPC that revokes access runs on whichever instance received the request. Every instance re-verifies the subscriptions it owns with the same `has-read-permissions?` predicates that `:subscribe-file` and `:subscribe-team` authorize with, so a downgrade to viewer keeps its subscription and a non-member organization owner keeps read-only access. Two mechanisms cover each other: - Access-changing mutations announce after the write. delete-team- member, leave-team, update-team-member-role, delete-file, delete-project, move-files, move-project and delete-team announce. delete-share-link is deliberately excluded: a share-link visitor never passes the subscribe permission check, so there is no subscription to revoke. Team-wide changes publish one event per team; the watcher resolves the members itself, and both publishers defer through `db/after-commit` so the watcher never decides from pre-commit state. - Every relay re-checks access every 5 minutes by default (`subscription-revalidation-interval`), driven from the relay loop with `alts!` over the subscription channel and a one-shot tick channel, so a steady stream of edits cannot push the check away forever. Supporting changes: - The registry holds `:connections` and `:by-profile` in one atom, kept consistent by register/unregister in a single swap!; a profile lookup is one map access, and a profile whose last connection leaves is dropped from the index. - `close-file-subscription` and `close-team-subscription` announce the :leave-* message, close the relay channel and purge the bus subscription, and are idempotent. Closing the channel ends the relay loop and its re-check task. - The permission query the re-check uses filters soft-deleted files, projects and teams, matching its sibling queries. - A transient database error fails open: it logs and keeps the subscription; the next tick retries. - The revocation watcher's params schema declared the nitrate client as a bare keyword inside [:map ...], which malli resolved as a registry lookup and which took the whole system build down under enable-backend-asserts. It is now an explicit pair, optional because init-key yields nil with the admin-console flag off. The tests reproduce the leak against the real message bus, including the negative control for the interval re-check. The written records live in mem:backend/websocket-subscriptions. Closes #12155 AI-assisted-by: space-bunny-free AI-assisted-by: muse-spark-1.3-contributor * 🐛 Mirror the subscribe gates in the revocation re-checks Addresses the second review round on PR #12189 (pabloalba). The re-checks only asked whether the profile could read. The membership legs of sql:file-permissions filter soft-deleted rows since the F2 fix, but the organization-owner fallback grants read-only access over HTTP without looking at deleted_at, so a soft-deleted file or team kept an expelled org owner subscribed indefinitely. The re-checks now gate on existence as well: file-readable? and team-readable? require a live row (db/get*, non-throwing, soft-deleted rows excluded) plus the read permission, mirroring the :subscribe-file gates and extending them to teams. Both relays and revalidate-profile-subscriptions use them; the subscribe handlers keep their exact behavior. Also from the same review: - watcher-consumes-revocation-events-from-the-bus no longer races the command's after-commit announcement: it takes access away with a direct row delete and drives the watcher with a synthetic bus event, instead of deleting through the command and publishing a second notify-* that would mask a broken command announcement. - moving-a-project-closes-source-team-subscriptions pins the move-project e2e: the file subscription drops, the source-team subscription survives (membership unchanged). - soft-deleted-file/team-closes-org-owner-subscription cover the new gate with red-before/green-after tests. mem:backend/websocket-subscriptions updated. Closes #12155 AI-assisted-by: inkling-2-small-free * 🐛 Refuse dead teams in resolve-team-id (PR #12189 review) Second review round on PR #12189 (pabloalba). The file re-check added in 87ca7aca53 stops at the file row, but the organization-owner fallback grants read-only access to non-member org owners via Nitrate over HTTP, which knows nothing about Penpot soft-deletion. resolve-team-id resolved the team from unfiltered SQL, so an expelled org owner kept the file subscription open for an indefinitely soft-deleted file, project or team. Now both SQL lookups filter deleted_at, and resolve-team-id checks the team row itself (db/get*, which filters soft-deleted rows) for every caller — teams, projects and files alike. Also from the same review: - file-readable? now uses the same sql:get-minimal-file as the subscribe gate (non-throwing via throw-if-not-exists false), and team-readable? projects id and deleted-at, so db/get* can filter. - The test suite extracts the thrice-duplicated raw delete into delete-team-membership!; a sibling of the fail-open pin now redefs the existence probe to throw; memory and docstring aligned with the code. - docs/technical-guide/developer/architecture/backend.md attributed the websocket to notifications.clj; it now names app.http.websocket as the owner and notifications.clj as the revocation publisher. - mem:backend/auth-permissions-product-domains documents the fallback mechanism and the resolve-team-id contract. Closes #12155 AI-assisted-by: inkling-2-small-free * 🐛 Test resolve-team-id refusal and tidy review findings Third pass over PR #12189 after review-code on 225eb98de2. resolve-team-id now has a direct test: all three branches resolve while rows are live, and file, project and team soft-deletes each refuse their branch — the two new SQL predicates and the final team-row check are all pinned by it. From the same review: - failing-existence-check-keeps-the-subscription and the sibling a-failing-check-keeps-the-subscription are deleted: they could not distinguish a live relay from a dead one. The fail-open policy remains documented in the docstring and the memory; the memory no longer claims the pins exist. - team-readable? projects ::sql/columns like the rest of the backend; the fully-qualified keyword went away with the new alias. - The dead when-not guard in get-organization-owner-viewer-team is removed (resolve-team-id guarantees the row live); the let binding in resolve-team-id no longer shadows the destructured team-id. - docs/technical-guide names rpc/notifications.clj as the publisher and the tree lists the real top-level modules. Closes #12155 AI-assisted-by: inkling-2-small-free * 🐛 Make the revocation tests deterministic Third review round on PR #12189 (pabloalba). Both tests took access away through a command that announces, so the live watcher could close the subscriptions before the test looked: - revocation-closes-subscriptions-that-are-no-longer-authorized asserted an exact count of 2 closes after delete-team-member, and that count reads 0 whenever the watcher got there first. It now removes the member with delete-team-membership!, the raw row delete that publishes nothing, and asserts both subscriptions are still open before calling revalidate-profile-subscriptions. - team-event-closes-every-member-subscription ran delete-team and then published a second notify-team-permissions-changed, so a missing announcement in delete-team still passed. It now soft-deletes the team row (sql:team-members joins team_profile_rel and profile only, so the watcher still resolves every member), asserts nothing closed, and drives one synthetic team event. The command paths stay covered by the announce and end-to-end tests for delete-team-member and delete-team. The file split the reviewer suggested stays the optional follow-up, tracked outside this PR and waiting for the merge. Closes #12155 AI-assisted-by: deepseek-v4.1-flash
143 lines
6.6 KiB
Markdown
143 lines
6.6 KiB
Markdown
---
|
|
title: Backend app
|
|
desc: Dive into self-hosting, configuration, developer insights (architecture, data model), integration, and troubleshooting. See Penpot's Technical Guide.
|
|
---
|
|
|
|
# Backend app
|
|
|
|
This app is in charge of CRUD of data, integrity validation and persistence
|
|
into a database and also into a file system for media attachments.
|
|
|
|
To handle deletions it uses a garbage collector mechanism: no object in the
|
|
database is deleted instantly. Instead, a field <code class="language-bash">deleted_at</code> is set with the
|
|
date and time of the deletion, and every query ignores db rows that have this
|
|
field set. Then, an async task that runs periodically, locates rows whose
|
|
deletion date is older than a given threshold and permanently deletes them.
|
|
|
|
For this, and other possibly slow tasks, there is an internal async tasks
|
|
worker, that may be used to queue tasks to be scheduled and executed when the
|
|
backend is idle. Other tasks are email sending, collecting data for telemetry
|
|
and detecting unused media attachment, for removing them from the file storage.
|
|
|
|
## Backend structure
|
|
|
|
Penpot backend app code resides under <code class="language-text">backend/src/app</code> path in the main repository.
|
|
|
|
@startuml BackendGeneral
|
|
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Container.puml
|
|
!define DEVICONS https://raw.githubusercontent.com/tupadr3/plantuml-icon-font-sprites/master/devicons
|
|
!include DEVICONS/react.puml
|
|
!include DEVICONS/java.puml
|
|
!include DEVICONS/clojure.puml
|
|
!include DEVICONS/postgresql.puml
|
|
!include DEVICONS/redis.puml
|
|
!include DEVICONS/chrome.puml
|
|
|
|
HIDE_STEREOTYPE()
|
|
|
|
Container(frontend_app, "Frontend app", "React / ClojureScript", "", "react")
|
|
|
|
System_Boundary(backend, "Backend") {
|
|
Container(backend_app, "Backend app", "Clojure / JVM", "", "clojure")
|
|
ContainerDb(db, "Database", "PostgreSQL", "", "postgresql")
|
|
ContainerDb(redis, "Broker", "Redis", "", "redis")
|
|
}
|
|
|
|
BiRel(frontend_app, backend_app, "Open", "websocket")
|
|
Rel(frontend_app, backend_app, "Uses", "RPC API")
|
|
Rel(backend_app, db, "Uses", "SQL")
|
|
Rel(redis, backend_app, "Subscribes", "pub/sub")
|
|
Rel(backend_app, redis, "Notifies", "pub/sub")
|
|
|
|
@enduml
|
|
|
|
```
|
|
▾ backend/src/app/
|
|
▸ cli/
|
|
▸ http/
|
|
▸ migrations/
|
|
▸ rpc/
|
|
▸ setup/
|
|
▸ srepl/
|
|
▸ util/
|
|
▸ tasks/
|
|
main.clj
|
|
config.clj
|
|
health.clj
|
|
http.clj
|
|
jobs.clj
|
|
metrics.clj
|
|
migrations.clj
|
|
nitrate.clj
|
|
rpc.clj
|
|
setup.clj
|
|
srepl.clj
|
|
system.clj
|
|
worker.clj
|
|
...
|
|
```
|
|
|
|
* <code class="language-text">main.clj</code> defines the app global settings and the main entry point of the
|
|
application, served by a JVM.
|
|
* <code class="language-text">config.clj</code> defines of the configuration options read from linux
|
|
environment.
|
|
* <code class="language-text">http</code> contains the HTTP server and the backend routes list.
|
|
* <code class="language-text">migrations</code> contains the SQL scripts that define the database schema, in
|
|
the form of a sequence of migrations.
|
|
* <code class="language-text">rpc</code> is the main module to handle the RPC API calls.
|
|
* <code class="language-text">http/websocket.clj</code> manages the websocket endpoint: clients subscribe
|
|
to open files or teams and the server keeps the subscriptions; revoked access
|
|
is re-checked and dropped. <code class="language-text">rpc/notifications.clj</code> publishes
|
|
access-change events on the bus so the websocket re-checks the subscriptions
|
|
it owns.
|
|
* <code class="language-text">setup</code> initializes the environment (loads config variables, sets up the
|
|
database, executes migrations, loads initial data, etc).
|
|
* <code class="language-text">srepl</code> sets up an interactive REPL shell, with some useful commands to be
|
|
used to debug a running instance.
|
|
* <code class="language-text">cli</code> sets a command-line interface, with some more maintenance commands.
|
|
* <code class="language-text">metrics.clj</code> has some interceptors that watches RPC calls, calculate
|
|
statistics and other metrics, and send them to external systems to store and
|
|
analyze.
|
|
* <code class="language-text">worker.clj</code> and <code class="language-text">tasks</code> define some async tasks that are executed in
|
|
parallel to the main http server (using java threads), and scheduled in a
|
|
cron-like table. They are useful to do some garbage collection, data packing
|
|
and similar periodic maintenance tasks.
|
|
* <code class="language-text">db.clj</code>, <code class="language-text">emails.clj</code>, <code class="language-text">media.clj</code>, <code class="language-text">msgbus.clj</code>, <code class="language-text">storage.clj</code>,
|
|
<code class="language-text">rlimits.clj</code> are general libraries to use I/O resources (SQL database,
|
|
send emails, handle multimedia objects, use REDIS messages, external file
|
|
storage and semaphores).
|
|
* <code class="language-text">util/</code> has a collection of generic utility functions.
|
|
|
|
### RPC calls
|
|
|
|
The RPC (Remote Procedure Call) subsystem consists of a mechanism that allows
|
|
to expose clojure functions as an HTTP endpoint. We take advantage of being
|
|
using Clojure at both front and back ends, to avoid needing complex data
|
|
conversions.
|
|
|
|
1. Frontend initiates a "query" or "mutation" call to <code class="language-text">:xxx</code> method, and
|
|
passes a Clojure object as params.
|
|
2. Params are string-encoded using
|
|
[transit](https://github.com/cognitect/transit-clj), a format similar to
|
|
JSON but more powerful.
|
|
3. The call is mapped to <code class="language-text"><backend-host>/api/rpc/query/xxx</code> or
|
|
<code class="language-text"><backend-host>/api/rpc/mutation/xxx</code>.
|
|
4. The <code class="language-text">rpc</code> module receives the call, decode the parameters and executes the
|
|
corresponding method inside <code class="language-text">src/app/rpc/queries/</code> or <code class="language-text">src/app/rpc/mutations/</code>.
|
|
We have created a <code class="language-text">defmethod</code> macro to declare an RPC method and its
|
|
parameter specs.
|
|
5. The result value is also transit-encoded and returned to the frontend.
|
|
|
|
This way, frontend can execute backend calls like it was calling an async function,
|
|
with all the power of Clojure data structures.
|
|
|
|
### PubSub
|
|
|
|
To manage subscriptions to a file, to be notified of changes, we use a redis
|
|
server as a pub/sub broker. Whenever a user visits a file and opens a
|
|
websocket, the backend creates a subscription in redis, with a topic that has
|
|
the id of the file. If the user sends any change to the file, backend sends a
|
|
notification to this topic, that is received by all subscribers. Then the
|
|
notification is retrieved and sent to the user via the websocket.
|
|
|