31 Commits

Author SHA1 Message Date
Andrey Antukh
d3ca9a0b83
✨ Add unified jobs substrate replacing legacy one-shot task system (#11542)
* 🎉 Move backend jobs to the unified job table substrate

Replace the generic 'task' table with a unified 'job' table as the
durable substrate for all backend jobs, and migrate every producer
and consumer to it. The legacy 'task' table stops receiving writes
but stays dormant (still cleaned by tasks-gc); upload-session
chunking and the other post-branch upstream changes are preserved.

New 0154 migration creates the 'job' table: dispatch and lifecycle
columns (name, queue, priority, scheduled_at, retries, status,
timestamps, props) plus optional user-facing ledger columns
(profile_id, target, progress, error, result, resource_id,
expires_at). No modified_at trigger: the application updates it on
every write for lease and orphan detection. Partial indexes cover
the dispatcher claim query, orphan scanning, and per-profile
listing.

New app.jobs public API: submit!/invoke!, heartbeats and progress
reporting, a precompiled job-def registry (name, schema, handler,
decoder, validator), an ephemeral request/response mode, and a
management API for external workers. The dispatcher claims
new/retry rows (FOR UPDATE SKIP LOCKED) with a plain JSON redis
payload and marks lease-expired running rows as orphaned; the
runner executes per-queue with the same retry conventions.

New jobs GC (daily): deletes expired rows and retains internal
terminal rows per :jobs-retention, touching referenced storage
objects so storage-gc-touched reclaims them. Job resources live in
a dedicated job-resource bucket with a reclaim branch in
storage-gc-touched.

All production call-sites (sendmail, delete-object, demo-purge,
file-gc, offload, webhooks, quotes, nitrate bulk delete,
snapshots, telemetry) and the ten cron tasks now submit through
app.jobs; cron is a pure scheduler and the legacy worker registry,
task-table inserts, and per-namespace handlers are removed.

Includes migrations, dispatcher, runner, jobs, request/response,
jobs GC, management API, and cron test suites.

AI-assisted-by: glm-5.3-flash
AI-assisted-by: muse-spark-1.3-contributor
AI-assisted-by: kimi-k3
AI-assisted-by: mimo-v2.5

* ✨ Address review comments on jobs substrate

Resolves the four open review threads on the jobs PR. Orphaned jobs
are now marked aborted, a system-side terminal state with no retry:
the dispatcher sweep stores the end event with outcome aborted in a
single bulk insert in the same transaction, records the terminal
outcome next to the orphaned counter, and reports an error log;
orphans keep alerting through the log and the counter, with no
user notification.

The aborted state is terminal everywhere: the status CHECK, the
sweep, the lifecycle writers, the jobs GC retention, the backlog
gauges and the user status mapping, with tests covering each
transition. The claim comment, the 2.20 doc version and the legacy
task table notes are corrected, and objects-gc heartbeats per chunk.

AI-assisted-by: muse-spark-1.3-contributor
2026-10-06 07:33:12 +02:00
Yuito Akatsuki (Tani Yutaka)
aeeca59b1a 🐛 Fix shared library deletion with storage-backed file data
Inject the storage component into the delete-object task handler and
validate it on initialization, so library absorption no longer fails with
"expected valid storage to be provided" for storage-backed file data.

Also strengthen the regression test: it now creates a real component in
the shared library and an instance in the referencing file, and verifies
the component is absorbed after deletion instead of only checking the
revision bump. The test disables the fdata/pointer-map feature to stay
focused on the storage fix.

Signed-off-by: Yuito Akatsuki (Tani Yutaka) <yuito@yuito-it.jp>
AI-assisted-by: deepseek-v4.1-flash
2026-09-24 18:17:07 +02:00
Andrey Antukh
81c3b3cd56
📎 Update copyright name on file header (#11346) 2026-08-25 11:55:10 +02:00
Yamila Moreno
ddba2ffa75
📎 Update Kaleidos Copyright (#9929) 2026-05-29 11:24:58 +02:00
Andrey Antukh
7b5817f407
♻️ Make several adjustments to the dashboard deleted page (#7999)
* ♻️ Make several sustantial adjustments to the dashboard deleted page

* 📎 Add PR feedback changes
2025-12-30 09:52:29 +01:00
Andrey Antukh
5717708b56 ♻️ Refactor file storage
Make it more scallable and make it easily extensible
2025-10-13 12:24:05 +02:00
Andrey Antukh
283eb0419c ♻️ Refactor time related namespaces
Mainly removes the custom app.util.time namespace
from frontend and backend and normalize all to use
the app.common.time namespace
2025-08-01 11:20:01 +02:00
Andrey Antukh
d2c800fc0f 🐛 Add missing handling of file_change on delete-object task 2025-02-21 14:24:18 +01:00
Andrey Antukh
88fb5e7ab5 ♻️ Update integrant to latest version
This upgrade also includes complete elimination of use spec
from the backend codebase, completing the long running migration
to fully use malli for validation and decoding.
2024-11-13 19:09:19 +01:00
Andrey Antukh
f9af7f0f09 🐛 Make profile deletion follow the delete-object flow
This removes the need of the specific task for cleaning
orphan teams.
2024-06-27 14:00:52 +02:00
Andrey Antukh
ec4260830c ♻️ Add API consistency fixes for task calling
Also adds a helper for calling tasks synchronously
2024-06-25 13:24:17 +02:00
Andrey Antukh
aa1cf3e03a ✨ Add some redundancy to delete_object task 2024-06-25 13:06:44 +02:00
Andrey Antukh
574c8d1789 ✨ Move library-absorb operation to async task
And make it not mandatory in case of failure
2024-05-27 11:19:15 +02:00
Andrey Antukh
761bbb7334 ✨ Add srepl helpers for delete/restore teams, projects, and files 2024-05-27 11:17:00 +02:00
Andrey Antukh
584a0fdba1 ✨ Reduce locking on objects-gc task
The main issue was the long running gc operation that
affects storage objects with deduplication. The long running
transacion ends locking some storage object rows which collaterally
made operations like import-binfile become blocked indefinitelly
because of the same rows (because of deduplication).

The solution used in this commit is split operations on small
chunks so we no longer use long running transactions that holds
too many locks. With this approach we will make a window to work
concurrently all operarate the distinct operations that requires
locks on the same rows.
2024-05-23 16:35:54 +02:00
Andrey Antukh
f0cf3e6411 🔥 Remove deprecated and unused tasks. 2021-09-30 11:48:14 +02:00
Andrey Antukh
c8102f4bff 🎉 Share link & pages on viewer. 2021-08-18 16:54:56 +02:00
Andrey Antukh
860e0227af ♻️ Reimplement GC mechanism for penpot database objects. 2021-06-09 15:27:07 +02:00
Andrey Antukh
bae709df5b 🐛 Fix custom font deletion task. 2021-06-03 12:55:31 +02:00
Andrey Antukh
e15a212b14 🎉 Add dashboard custom fonts management. 2021-05-04 14:21:31 +02:00
Andrey Antukh
f545e41d10 📎 Fix license header. 2021-04-12 16:49:43 +02:00
Andrey Antukh
e12a6e65a6 ♻️ Refactor logging. 2021-04-09 15:28:18 +02:00
Andrey Antukh
fbe2e2a285 ✨ Improve tasks metrics. 2021-02-22 14:50:01 +01:00
Andrey Antukh
a63f28a2e5 ✨ Normalize logging messages on backend. 2021-02-22 14:50:01 +01:00
Andrey Antukh
2e438385f3 ✨ Increase default deletion delay. 2021-02-10 17:06:09 +01:00
Andrey Antukh
b80295a21c ✨ Fix all linter issues on backend code. 2021-02-01 09:48:28 +01:00
Andrey Antukh
ab944fb9ae ♻️ Integrate new storage subsystem. 2021-01-07 11:29:57 +01:00
Andrey Antukh
9f12456456 ♻️ Replace mount with integrant. 2021-01-07 11:29:57 +01:00
Andrey Antukh
6881e3d795 ✨ Fix linter issues on backend tasks ns directory. 2020-12-02 15:35:14 +01:00
Andrey Antukh
fca48e39ee ✨ Make file to library link idempotent.
And remove unused api calls.
2020-11-23 17:43:06 +01:00
Andrey Antukh
6c67c3c71b ♻️ Make the namespacing independent of the branding. 2020-08-18 19:32:11 +02:00