diff --git a/.serena/memories/backend/core.md b/.serena/memories/backend/core.md index 708d30fc5d..7b085856d1 100644 --- a/.serena/memories/backend/core.md +++ b/.serena/memories/backend/core.md @@ -5,7 +5,8 @@ Backend: JVM Clojure; Integrant; PostgreSQL; Redis/Valkey; RPC; HTTP; storage; m ## Focused memories - RPC, DB helpers, workers, cron: `mem:backend/rpc-db-worker-subtleties` -- HTTP sessions, config, storage, media, file data persistence: `mem:backend/http-storage-filedata-subtleties` +- Storage abstraction, logical buckets, object lifecycle, deduplication, access, and garbage collection: `mem:backend/storage`. +- HTTP sessions, config, media processing, and file data persistence: `mem:backend/http-storage-filedata-subtleties`. - Auth flows, permission model, teams, projects, invitations, comments, webhooks, audit: `mem:backend/auth-permissions-product-domains` - Services, task-queue/Pub-Sub topology constraints -> `mem:prod-infra/core`. @@ -107,4 +108,3 @@ IMPORTANT: all CLI commands must be executed from the `backend/` subdirectory. J * **Isolated run:** `clojure -M:dev:test --focus backend-tests.my-ns-test` for a specific test namespace. * **Regression run:** `clojure -M:dev:test` to ensure no regressions in related functional areas. * **Principles:** Cross-cutting testing principles, anti-patterns, and verification checklist: `mem:testing`. - diff --git a/.serena/memories/backend/http-storage-filedata-subtleties.md b/.serena/memories/backend/http-storage-filedata-subtleties.md index e9c0962371..188ece7277 100644 --- a/.serena/memories/backend/http-storage-filedata-subtleties.md +++ b/.serena/memories/backend/http-storage-filedata-subtleties.md @@ -14,10 +14,7 @@ ## Storage and media -- Storage has a fixed valid bucket set. Backends are `:fs` and `:s3`; default backend comes from deprecated `assets-storage-backend` only when present, otherwise `objects-storage-backend`, defaulting to `:fs`. -- `put-object!` creates the DB `storage_object` row before writing backend content. Backend writes happen only for newly created rows, so deduplication can skip object writes. -- Deduplication only applies when requested, when the content can provide a hash, and when bucket metadata is present. Reads exclude soft-deleted storage rows. -- `sto/resolve` can reuse the current DB connection via `::db/reuse-conn true`; preserve this in transaction-sensitive code. +- Storage abstraction, backend configuration, logical buckets, object lifecycle, deduplication, access rules, and garbage collection: `mem:backend/storage`. - SVG validation strips DOCTYPE and uses secure SAX parsing. Basic SVG info falls back to 100x100 dimensions when width/height/viewBox are missing. - Raster metadata is shell-derived with ImageMagick `identify`, verifies detected MIME against the supplied MIME, and swaps dimensions for EXIF orientations 6/8. - Remote image download requires 2xx status, `content-length`, a known MIME, and size under the configured maximum before writing the temp file; mismatched byte count is an internal error. @@ -28,4 +25,4 @@ - File data backends are `legacy-db`, `db`, and `storage`. The storage backend keeps encoded file data in storage bucket `file-data`; the DB row stores metadata with `storage-ref-id` and nil data. - `fdata/upsert!` touches any storage object referenced by incoming metadata before storing the new row/blob. - Pointer-map fragments are persisted separately as type `fragment`, and only modified pointer maps are written. -- `fdata/realize` combines pointer realization and object-map realization. Use it before operations that need complete in-memory file data instead of pointer placeholders. \ No newline at end of file +- `fdata/realize` combines pointer realization and object-map realization. Use it before operations that need complete in-memory file data instead of pointer placeholders. diff --git a/.serena/memories/backend/storage.md b/.serena/memories/backend/storage.md new file mode 100644 index 0000000000..b0d81330f6 --- /dev/null +++ b/.serena/memories/backend/storage.md @@ -0,0 +1,83 @@ +# Backend Storage + +## Abstraction + +- `app.storage` stores binary objects. +- Each object has a `storage_object` database row. +- The row stores the UUID, size, backend, timestamps, and Transit metadata. +- The backend stores the binary content. +- Supported backends are `:fs` and `:s3`. +- FS uses one root directory and a UUID-derived path. +- S3 uses one configured bucket and an optional prefix. +- A Penpot bucket is metadata. It is not an S3 bucket or a filesystem directory. +- FS and S3 use the same UUID-derived object path. The bucket does not change the path. +- `PENPOT_OBJECTS_STORAGE_*` configures the current object backend. +- Deprecated asset-storage config keys remain supported for migration. +- Database rows keep the backend name. Keep the legacy `:assets-fs` and `:assets-s3` aliases. + +## Object Lifecycle + +- `put-object!` creates the database row before it writes backend content. +- Backend content is written only when the row is new. +- A failed backend write can leave an unreferenced database row. +- Callers often set `:touched-at` so garbage collection can remove such rows. +- `get-object` excludes rows with `deleted_at`. +- Existing object values can remain readable until physical deletion. +- `:expired-at` blocks reads after the expiration time. +- `del-object!` sets `deleted_at`. It does not remove backend content. +- `storage-gc-deleted` removes the database row and backend content after the deletion delay. +- `storage-gc-touched` finds references before it sets `deleted_at`. +- `objects-gc` removes deleted domain rows and touches their storage object IDs. +- Use `::db/reuse-conn true` with `sto/resolve` inside a database transaction. + +## Deduplication + +- Deduplication requires `::sto/deduplicate?`, a content hash, and bucket metadata. +- The lookup matches hash, bucket, backend, and `deleted_at IS NULL`. +- The lookup does not include file ID, profile ID, team ID, or organization ID. +- Objects can therefore share content across users and files within one bucket. +- Deleted objects are not reused. +- `tempfile` objects never use deduplication, even when the caller requests it. +- Use `sto/wrap-with-hash` when the caller already calculated the content hash. + +## Bucket Rules + +| Bucket | Content and references | Dedup | Direct `/assets/by-id` access | Cleanup | +| --- | --- | --- | --- | --- | +| `file-media-object` | Original file images and generated media thumbnails. References: `file_media_object.media_id` and `thumbnail_id`. | Yes | Public | Reference scan. | +| `team-font-variant` | Font variants in `team_font_variant`. References: `woff1_file_id`, `woff2_file_id`, `otf_file_id`, and `ttf_file_id`. | Yes | Public | Reference scan. | +| `file-object-thumbnail` | Frame and component thumbnails in `file_tagged_object_thumbnail.media_id`. | Yes | Public | Reference scan. | +| `file-thumbnail` | File grid thumbnails in `file_thumbnail.media_id`. | Yes | Authentication required | Reference scan. | +| `profile` | User and team profile photos. References: `profile.photo_id` and `team.photo_id`. | Yes | Authentication required | Reference scan. | +| `organization` | Organization logos uploaded by the Nitrate management API. | Yes | Public | No reference scan. A touched object is deleted. | +| `tempfile` | Export files, chunked-upload chunks, and temporary font downloads. | No | Authentication required | No reference scan. A touched object uses a two-hour deletion delay. | +| `file-data` | Encoded file data when `file-data-backend` is `storage`. Reference metadata has `storage-ref-id`, `file-id`, and the `file_data` row ID. | Yes | Authentication required | Reference scan. | +| `file-data-fragment` | Compatibility value for file-data fragments. The current backend has no dedicated producer for this bucket. | No current write semantics | Public | No touched-object collector case. | +| `file-change` | Compatibility value for file changes. Current snapshots store data in `file_data`, not this bucket. | No current write semantics | Authentication required | No touched-object collector case. | + +- The valid bucket set lives in `app.storage/valid-buckets`. +- `file-media-object` is the default bucket for old rows without bucket metadata. +- Do not assign a new bucket without adding its access and cleanup behavior. +- The touched-object collector raises an internal error for an unknown bucket. +- It supports `file-media-object`, `team-font-variant`, `file-object-thumbnail`, `file-thumbnail`, `profile`, `file-data`, `tempfile`, and `organization`. +- It does not support `file-data-fragment` or `file-change`. + +## Access Rules + +- `app.http.assets` decides direct object authentication from the bucket. +- Public buckets are `file-media-object`, `file-object-thumbnail`, `team-font-variant`, `file-data-fragment`, and `organization`. +- Other valid buckets require a session or access-token profile ID. +- File-media routes also require file read permission. +- Non-public direct responses set `content-disposition: attachment`. +- FS responses use `x-accel-redirect` for the configured asset path. +- S3 responses use a presigned URL and an HTTP redirect. + +## File Data + +- `file-data-backend` accepts `legacy-db`, `db`, or `storage`. +- `legacy-db` stores main data in `file.data` and snapshots in `file_change.data`. +- `db` stores encoded data in `file_data.data`. +- `storage` stores encoded data in storage subsystem with `file-data` bucket and keeps `data` nil in `file_data` table. +- The `file_data.metadata.storage-ref-id` value points to the storage object. +- `fdata/upsert!` touches a storage object from incoming metadata before it stores the new row. +- File snapshots use `file_data` for snapshot data and `file_change` for snapshot metadata. diff --git a/.serena/memories/prod-infra/core.md b/.serena/memories/prod-infra/core.md index e86eb69a32..1ec5af0308 100644 --- a/.serena/memories/prod-infra/core.md +++ b/.serena/memories/prod-infra/core.md @@ -6,7 +6,7 @@ Backend (`app.config`, `PENPOT_*` env vars) is parameterized; deployments choose - **PostgreSQL**: durable store. Profiles, teams, files, sessions, audit, `storage_object` metadata, the `task` queue, `scheduled_task` cron registry, migrations. File-data also lives here when the file-data backend is `legacy-db`/`db`. One shared DB across all backends. - **Redis (Valkey-compatible)**: per-backend message bus and cache. Concrete uses: msgbus Pub/Sub for collaborative-editing broadcasts and team/profile-org notifications fired by RPC handlers (`app.rpc.notifications`, `files_update`, `teams`, `websocket`); file-summary cache gated by `enable-redis-cache`; rate-limit counters; and the dispatcher→runner work hand-off list `penpot.worker.queue::`. `PENPOT_REDIS_URI`. -- **Object storage**: backends `:s3` and `:fs`. S3 in prod; devenv uses MinIO. Holds uploaded media, file-data when the file-data backend is `storage`, exports. Backend-side details (resolve, dedup, bucket set, file-data backends): `mem:backend/http-storage-filedata-subtleties`. +- **Object storage**: backends `:s3` and `:fs`. S3 in prod; devenv uses MinIO. Holds uploaded media, file-data when the file-data backend is `storage`, exports. Backend-side details (resolve, dedup, bucket set, object lifecycle, and file-data backends): `mem:backend/storage`. - **SMTP mailer**: invitations, password resets, email verification (sent via the `:sendmail` worker task). - **LDAP** (optional auth provider): helpers in `app.auth.*`, gated by `enable-login-with-ldap`. @@ -30,4 +30,4 @@ Penpot in production lives with both: horizontal-scale deployments accept "exact ## See also - Devenv composition and the ws0-only worker placement: `mem:devenv/core`. -- Storage backend resolution, dedup, file-data lifecycle: `mem:backend/http-storage-filedata-subtleties`. +- Storage backend resolution, dedup, bucket behavior, object lifecycle, and file-data lifecycle: `mem:backend/storage`.