36 Commits

Author SHA1 Message Date
Andrey Antukh
6feccaebb3 🐛 Cascade profile deletion in objects-gc task
The objects-gc task was performing a hard delete on profiles
without cascading the soft-delete to owned teams, projects, and files.
This left orphaned objects that were never cleaned up.

Now the task invokes delete-object before the hard delete, ensuring
all owned resources are properly marked for deletion and cleaned up
in subsequent GC iterations.

AI-assisted-by: qwen3.7-plus
2026-08-27 12:33:52 +02:00
Yamila Moreno
ddba2ffa75
📎 Update Kaleidos Copyright (#9929) 2026-05-29 11:24:58 +02:00
Andrey Antukh
94719eebf8 ♻️ Make storage and other objects deletion task vclock aware
This simplifes the mental model on how it works and simplifies testing
of the related code.

This also normalizes storage object deletion in the same way as the
rest of objects in penpot (now future deletion date on storage object
also means storage object to be deleted).
2025-11-19 17:28:55 +01:00
Andrey Antukh
5717708b56 ♻️ Refactor file storage
Make it more scallable and make it easily extensible
2025-10-13 12:24:05 +02:00
Andrey Antukh
17ec360720 Add several missing type hints for avoid reflection and autoboxing 2025-08-18 13:03:10 +02:00
Andrey Antukh
283eb0419c ♻️ Refactor time related namespaces
Mainly removes the custom app.util.time namespace
from frontend and backend and normalize all to use
the app.common.time namespace
2025-08-01 11:20:01 +02:00
Andrey Antukh
a706907b26 ♻️ Normalize logical deletion to future dates
Instead of managing the ...
2025-05-20 22:25:57 +02:00
Andrey Antukh
624750ad16
🐛 Fix incorrect order of execution of internal procs on objects-gc (#5929)
*  Improve efficiency on objects-gc tasks

Replacing db/cursor with db/plan

* 🐛 Use correct order of subprocs on objects-gc

Mainly affects the file deletion

* 📎 Increase chunk-size to 100 on objects-gc
2025-02-21 13:01:20 +01:00
Andrey Antukh
88fb5e7ab5 ♻️ Update integrant to latest version
This upgrade also includes complete elimination of use spec
from the backend codebase, completing the long running migration
to fully use malli for validation and decoding.
2024-11-13 19:09:19 +01:00
Andrey Antukh
32126d1874 ♻️ Refactor file changes gc tasks
Make it more friendly with the current snapshoting mechanism
2024-10-30 13:39:38 +01:00
Andrey Antukh
ceaafdbb1c Add offload mechanism for file snapshots 2024-08-26 13:52:42 +02:00
Andrey Antukh
0e92bcc0de 🎉 Add file-data offload mechanism 2024-08-09 14:28:18 +02:00
Andrey Antukh
9174bb140b ♻️ Refactor configuration validation
Replace spec with schema
2024-07-10 15:16:28 +02:00
Andrey Antukh
f9af7f0f09 🐛 Make profile deletion follow the delete-object flow
This removes the need of the specific task for cleaning
orphan teams.
2024-06-27 14:00:52 +02:00
Andrey Antukh
ec4260830c ♻️ Add API consistency fixes for task calling
Also adds a helper for calling tasks synchronously
2024-06-25 13:24:17 +02:00
Andrey Antukh
584a0fdba1 Reduce locking on objects-gc task
The main issue was the long running gc operation that
affects storage objects with deduplication. The long running
transacion ends locking some storage object rows which collaterally
made operations like import-binfile become blocked indefinitelly
because of the same rows (because of deduplication).

The solution used in this commit is split operations on small
chunks so we no longer use long running transactions that holds
too many locks. With this approach we will make a window to work
concurrently all operarate the distinct operations that requires
locks on the same rows.
2024-05-23 16:35:54 +02:00
Andrey Antukh
9c25723ee3 📎 Add note about fragments on object-gc ns 2024-04-10 15:31:49 +02:00
Andrey Antukh
746d898245 Improve the db api efficiency
Mainly setup proper defaults and reduce unnecesary allocations
on every db api call.
2024-01-04 12:41:16 +01:00
Andrey Antukh
addb392ecc Add safety mechanism for direct object deletion
The main objective is prevent deletion of objects that can leave
unreachable orphan objects which we are unable to correctly track.

Additionally, this commit includes:

1. Properly implement safe cascade deletion of all participating
   tables on soft deletion in the objects-gc task;

2. Make the file thumbnail related tables also participate in the
   touch/refcount mechanism applyign to the same safety checks;

3. Add helper for db query lazy iteration using PostgreSQL support
   for server side cursors;

4. Fix efficiency issues on gc related task using server side
   cursors instead of custom chunked iteration for processing data.

   The problem resided when a large chunk of rows that has identical
   value on the deleted_at column and the chunk size is small (the
   default); when the custom chunked iteration only reads a first N
   items and skip the rest of the set to the next run.

   This has caused many objects to remain pending to be eliminated,
   taking up space for longer than expected. The server side cursor
   based iteration does not has this problem and iterates correctly
   over all objects.

5. Fix refcount issues on font variant deletion RPC methods
2024-01-03 10:56:57 +01:00
Andrey Antukh
87615ce221 💄 Fix format issues on backend module 2023-11-29 12:55:58 +01:00
Andrey Antukh
e96fc32cc1 ⬆️ Update devenv dockerfile 2023-05-17 15:47:21 +02:00
Andrey Antukh
0dc7f4e07e Add test for orphaned teams deletion 2023-05-17 15:47:21 +02:00
Andrey Antukh
dfdc9c9fa5 ♻️ Refactor storage internal concurrency model 2023-03-14 12:30:27 +01:00
Andrey Antukh
ab3b9cba45 ♻️ Refactor storage and assets related modules
- improve internal error handling
- add more specs and more asserts
2023-02-07 18:16:55 +01:00
Andrey Antukh
b929564fa7 ♻️ Add admin facilities on the code base
- Fix bugs related to orphan teams on profile deletion
- Separate session based profile-id param from api user provided
2022-12-22 16:42:45 +01:00
luz paz
e30bea0b6f 🔧 Fix typos in source code
Found via `codespell -q 3 -S *.po,./frontend/yarn.lock -L childs,clen,fpr,inflight,ody,ot,ro,te,trys,ue`
2022-10-04 10:40:34 +02:00
Andrey Antukh
278f6685b6 Improve object deletion process on profile deletion 2022-09-28 09:45:43 +02:00
Andrey Antukh
41134f22e9 📎 Update license header 2022-09-20 23:23:22 +02:00
Andrey Antukh
5867e64d36 Improve objects-gc task 2022-08-12 08:34:57 +02:00
Andrey Antukh
1b444a42f2 ♻️ Refactor http server layer
Make it fully asynchronous.
2022-03-09 17:18:06 +01:00
Andrey Antukh
9e4a50fb15 ♻️ Refactor backend to be more async friendly 2022-03-03 16:05:52 +01:00
Andrey Antukh
116fafd0e1 📎 Minor log param naming change 2022-02-15 13:25:46 +01:00
Josh Soref
cd2d3d5fa3 🐛 Fix typos in backend 2021-11-15 09:53:10 -05:00
Andrey Antukh
b64d5ef357 🎉 Add unified logging api. 2021-09-30 11:48:14 +02:00
Andrey Antukh
2728fa2b8d Add proper fdata objects deletion. 2021-06-15 09:25:37 +02:00
Andrey Antukh
860e0227af ♻️ Reimplement GC mechanism for penpot database objects. 2021-06-09 15:27:07 +02:00