mirror of
https://github.com/penpot/penpot.git
synced 2026-10-31 23:10:31 +00:00
6eece49f69
An object whose :storage-target id is not in the S3 routes configuration can no longer be located. Reads and serving now fail with :invalid-storage-target instead of silently using the default bucket, and garbage collection refuses to delete the row: it logs an error and parks the row (deleted_at +1 day, no deletion attempts, no give-up) until the target is configured again. Legacy rows (no target) and the default target keep working and deleting normally. Also addresses review findings: per-target dedup isolation and stale-repair coverage, legacy-row GC coverage through the real task paths, a single declared-target schema, closing already-built S3 clients when target init fails, and loader edge pins. Docs and memory updated. Closes #11630 AI-assisted-by: deepseek-flash
133 lines
7.3 KiB
Markdown
133 lines
7.3 KiB
Markdown
---
|
|
title: Assets storage
|
|
desc: Learn about assets storage, API, object buckets, sharing, and garbage collection. See Penpot's technical guide for developers. Try Penpot - It's free.
|
|
---
|
|
|
|
# Assets storage
|
|
|
|
The [storage.clj](https://github.com/penpot/penpot/blob/develop/backend/src/app/storage.clj)
|
|
is a module that manages storage of binary objects. It's a generic utility
|
|
that may be used for any kind of user uploaded files. Currently:
|
|
|
|
* Image assets in Penpot files.
|
|
* Uploaded fonts.
|
|
* Profile photos of users and teams.
|
|
|
|
There is an abstract interface and several implementations (or **backends**),
|
|
depending on where the objects are actually stored:
|
|
|
|
* <code class="language-clojure">:fs</code> stores objects in the file system, under a given base path.
|
|
* <code class="language-clojure">:s3</code> stores them in any cloud storage with an AWS-S3 compatible
|
|
interface.
|
|
|
|
Legacy rows may still reference the deprecated <code class="language-clojure">:assets-fs</code> and
|
|
<code class="language-clojure">:assets-s3</code> names, which alias the current backends.
|
|
|
|
## Storage API
|
|
|
|
The **StorageObject** record represents one stored object. It contains the
|
|
metadata, that is always stored in the database (table <code class="language-clojure">storage_object</code>),
|
|
while the actual object data goes to the backend.
|
|
|
|
* <code class="language-clojure">:id</code> is the identifier you use to reference the object, may be stored
|
|
in other places to represent the relationship with other element.
|
|
* <code class="language-clojure">:backend</code> points to the backend where the object data resides.
|
|
* <code class="language-clojure">:created-at</code> is the date/time of object creation.
|
|
* <code class="language-clojure">:deleted-at</code> is the date/time of object marked for deletion (see below).
|
|
* <code class="language-clojure">:expired-at</code> allows to create objects that are automatically deleted
|
|
at some time (useful for temporary objects).
|
|
* <code class="language-clojure">:touched-at</code> is used to check objects that may need to be deleted (see
|
|
below).
|
|
|
|
Also more metadata may be attached to objects, such as the <code class="language-clojure">:content-type</code> or
|
|
the <code class="language-clojure">:bucket</code> (see below).
|
|
|
|
You can use the API functions to manipulate objects. For example <code class="language-clojure">put-object!</code>
|
|
to create a new one, <code class="language-clojure">get-object</code> to retrieve the StorageObject,
|
|
<code class="language-clojure">get-object-data</code> or <code class="language-clojure">get-object-bytes</code> to read the binary contents, etc.
|
|
|
|
For profile photos or fonts, the object id is stored in the related table,
|
|
without further ado. But for file images, one more indirection is used. The
|
|
**file-media-object** is an abstraction that represents one image uploaded
|
|
by the user (in the future we may support other multimedia types). It has its
|
|
own database table, and references two <code class="language-clojure">StorageObjects</code>, one for the original
|
|
file and another one for the thumbnail. Image shapes contains the id of the
|
|
<code class="language-clojure">file-media-object</code> with the <code class="language-clojure">:is-local</code> property as true. Image assets in the
|
|
file library also have a <code class="language-clojure">file-media-object</code> with <code class="language-clojure">:is-local</code> false,
|
|
representing that the object may be being used in other files.
|
|
|
|
## Serving objects
|
|
|
|
Stored objects are always served by Penpot (even if they have a public URL,
|
|
like when <code class="language-clojure">:s3</code> storage are used). We have an endpoint <code class="language-text">/assets</code> with three
|
|
variants:
|
|
|
|
```bash
|
|
/assets/by-id/<uuid>
|
|
/assets/by-file-media-id/<uuid>
|
|
/assets/by-file-media-id/<uuid>/thumbnail
|
|
```
|
|
|
|
They take an object and retrieve its data to the user. For <code class="language-clojure">:db</code> backend, the
|
|
data is extracted from the database and served by the app. For the other ones,
|
|
we calculate the real url of the object, and pass it to our **nginx** server,
|
|
via special HTTP headers, for it to retrieve the data and serve it to the user.
|
|
|
|
This is the same in all environments (devenv, production or on premise).
|
|
|
|
## Object buckets
|
|
|
|
Obects may be organized in **buckets**, that are a kind of "intelligent" folders
|
|
(not related to AWS-S3 buckets, this is a Penpot internal concept).
|
|
|
|
The storage module may use the bucket (hardcoded) to make special treatment to
|
|
object, such as storing in a different path, or guessing how to know if an object
|
|
is referenced from other place.
|
|
|
|
When the <code class="language-clojure">:s3</code> backend is used, a semantic bucket may also be routed to a
|
|
named S3 **target** (a different bucket and/or key prefix). The chosen target id
|
|
is stored in the object metadata (<code class="language-clojure">:storage-target</code>) and resolved again on
|
|
reads, URL signing, deduplication and garbage collection. Objects without a
|
|
stored target use the default target. See the configuration guide for the
|
|
routing file format.
|
|
|
|
If an object references a target id that is not configured anymore, it cannot be
|
|
located: reads and serving fail, and garbage collection refuses to delete the
|
|
object, keeping the database row until the target is declared again.
|
|
|
|
## Sharing and deleting objects
|
|
|
|
To save storage space, duplicated objects wre shared. So, if for example
|
|
several users upload the same image, or a library asset is instantiated many
|
|
times, even by different users, the object data is actuall stored only once.
|
|
|
|
To achieve this, when an object is uploaded, its content is hashed, and the
|
|
hash compared with other objects in the same bucket. If there is a match,
|
|
the <code class="language-clojure">StorabeObject</code> is reused. Thus, there may be different, unrelated, shapes
|
|
or library assets whose <code class="language-clojure">:object-id</code> is the same.
|
|
|
|
### Garbage collector and reference count
|
|
|
|
Of course, when objects are shared, we cannot delete it directly when the
|
|
associated item is removed or unlinked. Instead, we need some mechanism to
|
|
track the references, and a garbage collector that deletes any object that
|
|
is no longer referenced.
|
|
|
|
We don't use explicit reference counts or indexes. Instead, the storage system
|
|
is intelligent enough to search, depending on the bucket (one for profile
|
|
photos, other for file media objects, etc.) if there is any element that is
|
|
using the object. For example, in the first case we look for user or team
|
|
profiles where the <code class="language-clojure">:photo-id</code> field matches the object id.
|
|
|
|
When one item stops using one storage object (e. g. an image shape is deleted),
|
|
we mark the object as <code class="language-clojure">:touched</code>. A periodic task revises all touched objectsm
|
|
checking if they are still referenced in other places. If not, they are marked
|
|
as :deleted. They're preserved in this state for some time (to allow "undeletion"
|
|
if the user undoes the change), and eventually, another garbage collection task
|
|
definitively deletes it, both in the backend and in the database table.
|
|
|
|
For <code class="language-clojure">file-media-objects</code>, there is another collector, that periodically checks
|
|
if a media object is referenced by any shape or asset in its file. If not, it
|
|
marks the object as <code class="language-clojure">:touched</code> triggering the process described above.
|
|
|