26 Commits

Author SHA1 Message Date
Andrey Antukh
d67a00c1d5
✨ Normalize storage metadata with a closed schema (#11987)
* ✨ Add Malli schema for storage metadata with dual decode

Phase 1 of the storage_object.metadata migration: reads accept both
Transit and plain JSON (sniffed by the marker) and always return the
normalized shape; writes validate against a closed per-bucket Malli
schema and still serialize as Transit unless the new
:storage-metadata-as-json config flag is set.

The 0155 migration normalizes existing rows inside Transit (reference
to bucket, default bucket, drop of chunk leftovers) and is
idempotent; large instances should fake it and run the batched
script instead.

AI-assisted-by: muse-spark-1.3-contributor

* ♻️ Address review findings on storage metadata Phase 1

Collapse the dead :reference leg of the gc-touched bucket fallback
(the decode always sets :bucket on non-nil metadata, so it is only
reachable with a NULL column) and fix its comment.

Pin the write flag off in the transit-assuming metadata tests so the
suite proves the same with the flag set, and add coverage for the
flag rollback contract, JSON hash survival, NULL metadata in
gc-touched, and the 0155 normalization statements.

AI-assisted-by: muse-spark-1.3-contributor

* ♻️ Backfill NULLs, canonical buckets, comment fix

Backfill NULL metadata columns in 0155 via coalesce (the key-missing
rule already matches them), derive valid-buckets from the Malli schema
dispatch entries so the list lives in one place, and correct the
lookup-bucket fallback comment to NULL columns.

AI-assisted-by: muse-spark-1.3-contributor

* ♻️ Defer corrupt metadata rows in storage gc-touched

Decode touched rows individually so one non-map metadata value no
longer aborts the whole chunk: corrupt rows are logged and deferred
exactly one day in the same transaction, keeping their metadata
intact for a later repair, while healthy rows process normally.

AI-assisted-by: muse-spark-1.3-contributor

* ♻️ Address storage metadata phase 1 review findings

Address the review findings on the storage metadata phase 1 branch:

- Fix put-and-delete-object: it stored the object with
  ::sto/expired-at, so the row was already deleted and del-object!
  returned false. Add delete-expired-object-returns-false to keep
  the expired-delete case covered.
- Cache the Malli decoder and encoder per process. Building them
  compiles the closed multi-dispatch schema, and decode-metadata
  runs on every read path (get-object, dedup probes, GC batches).
- Catch Exception instead of Throwable in try-decode-row so JVM
  Errors are not deferred as corrupt metadata.
- Add penpot_storage_gc_poison_total, emitted from
  storage-gc-touched; wire ::mtx/metrics into its handler.
- Anchor the encoding sniff to the start of the document so a
  plain JSON value that begins with a Transit-looking prefix is
  not read as Transit.
- Cover every bucket on both encodings, a JSON roundtrip through
  the jsonb column, nil metadata, the canonical bucket set and a
  poison-only GC chunk.
- Rename private check-metadata! to check-metadata.

AI-assisted-by: deepseek-v4.1-flash

* ♻️ Simplify the storage metadata schema to a single map

Replace the per-bucket :multi dispatch with a single closed map: the
bucket is validated with ::sm/one-of over metadata-buckets (now a plain
set) and the remaining keys are typed optional fields. Per-bucket
enforcement shrinks to a one-line :fn guard requiring :file-id and :id
for file-data, whose ids the GC reads to resolve references.

- Drop the dead (sm/register! ::metadata ...): nothing references the
  schema by keyword.
- Define tempfile-bucket and upload-session-bucket in the schema and
  alias them from app.storage, removing duplicated literals.
- Keep content-type required and the map closed, so an unknown bucket
  or key still fails fast on write.

AI-assisted-by: deepseek-v4.1-flash

* ♻️ Drop input coercion from encode-metadata

encode-metadata no longer runs the json-transformer decoder before
validation. On the write path its only effect was coercing string
UUIDs to UUID, and every producer already passes native UUIDs (the
RPC profile-id, uuid/random, or binfile ids decoded as ::sm/uuid).
Reads keep decoding, so stored Transit or JSON values still come
back as native types.

- Replace encode-accepts-string-uuids with
  encode-rejects-string-uuids, pinning the stricter contract.
- Pass native UUIDs in encode-writes-plain-json-with-flag.

AI-assisted-by: deepseek-v4.1-flash

* 📚 Document each statement in the storage metadata migration

Move the per-statement rules out of the header and add a comment to each
UPDATE explaining what it does: drop chunk leftovers, promote the legacy
"~:reference" to "~:bucket", drop residual "~:reference", and backfill the
default bucket. The header keeps the scope, the encoding note, the `->`
vs `?` note and the large-instance warning.

AI-assisted-by: deepseek-v4.1-flash

* 📚 Unwrap wrapped lines in the backend storage memory

One line per bullet or paragraph, as mem:memory-maintenance requires.
Only formatting; no content change.

AI-assisted-by: deepseek-v4.1-flash

* ♻️ Defer storage GC poison rows in their own transaction

process-chunk! no longer takes poison-ids; it only processes the healthy
chunk. The deferral moves to defer-poison! and process-touched! runs it in
its own transaction, separate from the freeze/delete work. The loop still
drains while there is chunk or poison, so a batch made only of poison rows
does not leave healthy rows behind the LIMIT 10 waiting for the next run.

Add a regression test: ten poison rows plus one healthy row with a later
touched_at are all handled in the same run.

AI-assisted-by: deepseek-v4.1-flash

* ♻️ Declare per-bucket metadata requirements in one map

Replace the file-data-specific predicate with bucket-requirements, a map
from bucket to the extra keys it must carry. metadata-buckets is derived
from its keys and a single generic :fn enforces presence, so a new bucket
and its contract are one entry. organization now requires
:organization-id; file-data keeps requiring :file-id and :id.

Update the http-assets test helper to set organization-id for its
organization objects.

AI-assisted-by: deepseek-v4.1-flash

* ♻️ Drop the ! suffix from storage GC helpers

Rename the internal helpers in app.storage.gc-touched (process-chunk,
defer-poison, mark-freeze-in-bulk, ...) to drop the trailing !.

AI-assisted-by: deepseek-v4.1-flash

* ♻️ Drop the ! suffix from storage GC deleted helpers

Rename the internal helpers in app.storage.gc-deleted (clean-deleted,
delete-sobjects, delete-give-up, ...) to drop the trailing !.

AI-assisted-by: deepseek-v4.1-flash

* 🐛 Fix dedup lookup for JSON-encoded storage metadata

get-database-object-by-hash only matched the Transit keys, so once the
:storage-metadata-as-json flag wrote plain JSON rows the dedup stopped
finding them and duplicated blobs. Match both encodings with a UNION ALL
of two indexable branches.

- Add migration 0156 with the plain-key dedup index; the legacy 0068
  index stays until Transit support is removed.
- Cover it with a JSON dedup test and a Transit -> JSON cross test.

AI-assisted-by: deepseek-v4.1-flash
2026-10-01 07:19:21 +02:00
Andrey Antukh
0e388442a1
✨ Add storage object status lifecycle and verified dedup (#11345)
* ♻️ Simplify storage GC delays and add skip-delay task params

The touched GC no longer applies an extra deletion-delay when marking
storage objects as deleted. By the time a storage object is touched, its
referencing domain row has already passed its own deletion delay, and the
reference scan is the only safety check needed. Touched objects are now
marked with deleted_at = now, so the deleted GC removes them on the next
run.

For the tempfile bucket, upload chunks now set touched-at in the future
(1h, aligned with the upload-session-gc TTL) instead of relying on a
special-case deletion delay.

Task handlers now read their task props:
- storage-gc-touched accepts :skip-delay to process all touched objects
  immediately, bypassing the min-age threshold.
- objects-gc accepts :chunk-size and :skip-delay to process recently
  deleted rows without waiting for the deletion delay.

This allows running the deletion cascade immediately from the REPL via
run-task! with the skip-delay option.

AI-assisted-by: deepseek-v4-flash

* ✨ Add storage object status lifecycle, verified dedup, and deletion retry tracking

Storage object lifecycle hardening:

- Add status column ('valid' | 'pending') as write-ahead marker for
  object creation. put-object! inserts in 'pending' state, writes blob,
  then promotes to 'valid'. Failed writes remove the pending row.
- Add :storage-pending-gc task to reclaim orphaned pending rows (e.g.
  after crash between blob write and promotion).
- Verify blob existence on every dedup hit via exists-object? (fs stat /
  s3 headObject). Missing blobs mark the row as deleted and create fresh
  object.
- Add deletion_attempts column (migration 0154) to track physical blob
  deletion attempts. Restructure gc_deleted to use chunked processing
  with per-chunk transactions (short lock duration). Failed deletions
  are deferred to tomorrow (deleted_at = NOW() + 1 day) to prevent
  infinite loops. After 7 attempts, give up and accept orphan.
- Change del-objects-in-bulk contract to return #{fail-ids} for precise
  per-id tracking (fs and s3 backends updated).
- Use tmp/tempfile for fs atomic writes with cleanup queue registration
  (crashed-JVM temp files swept ~60min later). Document ATOMIC_MOVE
  POSIX-only assumption.
- Add linear backoff to s3 exists-object? retries (100ms/200ms/300ms).
- Wrap compensating delete in put-object! catch block to prevent
  masking original error when connection is aborted.
- Fix assert messages in pending_gc.clj and gc_deleted.clj (pool
  assertion said 'expected valid storage' instead of 'db pool').
- Add pending-objects-excluded-from-gc-deleted test. Use unique path in
  put-object-write-failure-leaves-no-row test to avoid collisions.

AI-assisted-by: qwen3.7-plus

* 🐛 Fix review comments on gc-deleted and storage

- Fix process-chunk! returning nil causing (+ acc nil) crash
- Add FOR UPDATE SKIP LOCKED to sql:get-deleted-chunk to prevent
  infinite loop when another worker holds locks
- Pass :cause to log messages in gc_deleted.clj and s3.clj
- Fix extra space in log hint string
- Remove unused ::blob-missing? reference from storage memory
- Rename test to match actual behavior (leaves pending row)
- Add test for gc-deleted giving up after max attempts

AI-assisted-by: qwen3.7-plus
2026-08-27 12:37:05 +02:00
Andrey Antukh
81c3b3cd56
📎 Update copyright name on file header (#11346) 2026-08-25 11:55:10 +02:00
Yamila Moreno
ddba2ffa75
📎 Update Kaleidos Copyright (#9929) 2026-05-29 11:24:58 +02:00
Andrey Antukh
88fb5e7ab5 ♻️ Update integrant to latest version
This upgrade also includes complete elimination of use spec
from the backend codebase, completing the long running migration
to fully use malli for validation and decoding.
2024-11-13 19:09:19 +01:00
Andrey Antukh
0e92bcc0de 🎉 Add file-data offload mechanism 2024-08-09 14:28:18 +02:00
Andrey Antukh
addb392ecc ✨ Add safety mechanism for direct object deletion
The main objective is prevent deletion of objects that can leave
unreachable orphan objects which we are unable to correctly track.

Additionally, this commit includes:

1. Properly implement safe cascade deletion of all participating
   tables on soft deletion in the objects-gc task;

2. Make the file thumbnail related tables also participate in the
   touch/refcount mechanism applyign to the same safety checks;

3. Add helper for db query lazy iteration using PostgreSQL support
   for server side cursors;

4. Fix efficiency issues on gc related task using server side
   cursors instead of custom chunked iteration for processing data.

   The problem resided when a large chunk of rows that has identical
   value on the deleted_at column and the chunk size is small (the
   default); when the custom chunked iteration only reads a first N
   items and skip the rest of the set to the next run.

   This has caused many objects to remain pending to be eliminated,
   taking up space for longer than expected. The server side cursor
   based iteration does not has this problem and iterates correctly
   over all objects.

5. Fix refcount issues on font variant deletion RPC methods
2024-01-03 10:56:57 +01:00
Andrey Antukh
c64e14859c ✨ Simplify internal executor module 2023-11-27 14:25:12 +01:00
Andrey Antukh
dfdc9c9fa5 ♻️ Refactor storage internal concurrency model 2023-03-14 12:30:27 +01:00
Andrey Antukh
ab3b9cba45 ♻️ Refactor storage and assets related modules
- improve internal error handling
- add more specs and more asserts
2023-02-07 18:16:55 +01:00
Andrey Antukh
41134f22e9 📎 Update license header 2022-09-20 23:23:22 +02:00
Andrey Antukh
e657c1bbfa ✨ Start use datoteka.io ns 2022-09-09 12:19:49 +02:00
Andrey Antukh
ebcb385593 ♻️ Minor refactor on storages
Fix many issues on FS & S3 backend; removes the unused and broken
DB backend. Normalize operations on bytes and byte streams on a
separated namespace: app.util.bytes
2022-06-22 11:37:45 +02:00
Andrey Antukh
7f3c67724e 🐛 Fix svg media asset upload internal server error 2022-03-20 13:04:12 +01:00
Andrey Antukh
9e4a50fb15 ♻️ Refactor backend to be more async friendly 2022-03-03 16:05:52 +01:00
Andrey Antukh
219f9c478d 🎉 Add version 4 of blob encoding.
The version 4 starts using the hight performance fressian
binary encoding with very lightweight compression layer.
2021-12-21 16:19:25 +01:00
Andrey Antukh
2728fa2b8d ✨ Add proper fdata objects deletion. 2021-06-15 09:25:37 +02:00
Andrey Antukh
0c97a44a2a 🎉 Add file offloading to external storage mechanism. 2021-06-14 15:41:27 +02:00
Andrey Antukh
e15a212b14 🎉 Add dashboard custom fonts management. 2021-05-04 14:21:31 +02:00
Andrey Antukh
f545e41d10 📎 Fix license header. 2021-04-12 16:49:43 +02:00
Andrey Antukh
784a4f8ecd ✨ Add some type hints and remove legacy code. 2021-02-18 09:35:37 +01:00
Andrey Antukh
b80295a21c ✨ Fix all linter issues on backend code. 2021-02-01 09:48:28 +01:00
Andrey Antukh
a69a35a0b6 ✨ Improve storage recheck task and add more specs. 2021-01-26 11:13:21 +01:00
Andrey Antukh
d494e44df3 🎉 Add builtin copy fast path operation for storage. 2021-01-19 15:01:33 +01:00
Andrey Antukh
ab944fb9ae ♻️ Integrate new storage subsystem. 2021-01-07 11:29:57 +01:00
Andrey Antukh
760eb926bf 🎉 Add plugable storages abstraction layer (with support for fs, s3 and db). 2021-01-07 11:29:57 +01:00