diff --git a/.serena/memories/backend/core.md b/.serena/memories/backend/core.md index 287cd56dd2..1c4dc27119 100644 --- a/.serena/memories/backend/core.md +++ b/.serena/memories/backend/core.md @@ -57,10 +57,19 @@ For deeper details on transaction semantics, advisory locks, Transit vs JSON hel Every background job is a job-def: a plain `(defn execute-X [cfg params] ...)` in its namespace + a malli params schema + an `ig/init-key` that returns the job-def map `{::jobs/name, ::jobs/schema, ::jobs/handler, ::jobs/decoder, ::jobs/validator}` (decoder/validator precompiled at init). The registry is the `::jobs/defs` wiring in `app.main`, which also populates a module-level registry used by `jobs/submit` as fallback — job-def components cannot ig/ref `::jobs/defs` (wiring cycle). Submit from RPC code passes its RPC cfg (it carries `::jobs/defs` via ig/ref). - `::jobs/handler` is `(fn [context params] ...)`. It is NOT given the runner cfg: a job-def closes over its own dependencies, and handing it the runner cfg would let a handler reach the pool or any other component. A wrapper passes the context to its implementation only when the implementation takes it; most do not, and their implementation stays `[cfg params]`. -- The context is built by `jobs/make-context` and has exactly four keys: `id`, `name`, `label`, `resource-id`, behind a closed schema. `queue` is absent on purpose (a routing detail of the dispatcher, not a property of the job) and so are `retry-num`/`max-retries`/`attempt` (retry policy belongs to the runner, and an attempt number cannot be durable because the `noop` strategy re-runs without incrementing the counter). +- The context is built by `jobs/make-context` and has exactly five keys: `id`, `name`, `label`, `resource-id`, `profile-id`, behind a closed schema. `profile-id` is the owner of a user-facing job and nil for an internal one, and it is there so a handler can revalidate permissions and check who owns its resource without a second query. `queue` is absent on purpose (a routing detail of the dispatcher, not a property of the job) and so are `retry-num`/`max-retries`/`attempt` (retry policy belongs to the runner, and an attempt number cannot be durable because the `noop` strategy re-runs without incrementing the counter). - `jobs/invoke` is the in-process escape hatch: it delivers a nil context without a row, validates and delivers `::jobs/context` when given, and keeps `::jobs/job-id` independent. `::jobs/context` and `::jobs/job-id` are cfg options, not params. -- Business params live in the `params` column (plain JSON, decoded with the job-def decoder). `submit` takes an optional `::jobs/profile-id` and `::jobs/resource-id`, both validated as UUIDs and stored as columns; nothing is inferred from `params`. No caller passes a profile-id yet. -- `heartbeat` uses named options `[cfg & {:keys [job-id progress] :as options}]` and returns the number of durable writes. The job id falls back to `::jobs/job-id` on the cfg and then to the runner-bound `*job-id*`. A progress report needs `current`, allows optional `total` and `stage` (max 250 chars) and takes no other key. +- Business params live in the `params` column (plain JSON, decoded with the job-def decoder). `submit` takes optional `::jobs/profile-id`, `::jobs/resource-id` (both validated as UUIDs) and `::jobs/expires-at` (an instant: the retention the jobs GC sweeps by), all stored as columns; nothing is inferred from `params`. The binfile creation commands are the callers that pass a profile-id and a resource. +- A job-def may carry optional metadata keys read from the registry by its consumers, never by the substrate: `::jobs/family` (e.g. `:export`) groups the jobs of a family for quotes, `::jobs/resource-role` (`:input` or `:output`) says whether the job's `resource_id` is what it consumes or what it produces, and `::jobs/queue-name` names the queue the creation command routes the job to (`submit-job` falls back to `:binfile` when the def does not set it). +- Job resources (`app.jobs.storage`) are the objects a job owns through `job.resource_id`: they live in the `job-resource` bucket, are written with the profile of the job as owner (only that profile can read them back from `/assets/by-id`) and are never deduplicated. They carry no expiry of their own, because `expired-at` is stored as `deleted_at` and an object with it is unreadable from birth: the retention of the artifact is the retention of its job row. The writer marks the object as touched right away, so an artifact that no row references (a crash between the upload and the completion) is reclaimed by `storage-gc-touched`, and a row that does reference it freezes it until the jobs GC deletes the row and touches the object. +- The binfile jobs (`:export-binfile` and `:import-binfile`) run the core of `bf.v1`/`bf.v3` through `app.binfile.jobs`, which turns the `:progress` taps of the core into `progress` events of the job with `jobs/heartbeat` (the `modified_at` touch at most once per second, progress events at up to ~200ms). Each event is a self-contained milestone: a `:stage` keyword of one vocabulary shared by the two formats (the same work has a different name in v1 and v3) and an optional `:counters` map, with the counter of the activity named after its own stage and the outer scopes in course as context, which the tap carries itself (the reader puts `::outer-counters` in every tap inside them — today only `:files`, extendable to `:teams`/`:projects` for multi-scope exports — so the adapter remembers nothing between taps and a tap that happens outside any scope has no context); the ids and names the taps carry never reach the event. Every tap is reported, so a tap with no units to count still keeps the job alive, and a milestone the throttle drops loses nothing because the next one carries the same state. The adapter checks that the job is still active before the run, and the beat raises an `:interrupt` right there on the run thread when the job is no longer active, so the work stops at its next event point inside the transaction of its handler, which rolls back. A run whose last tap predates the interruption is discarded by the check once it returns. The adapter passes the job id from the handler context. Their job-def components carry pool, storage, metrics and msgbus, because a job with a `profile_id` cannot store an event without a msgbus on its cfg. Both readers answer with the same shape (`:file-ids` and, in version 3, the resolution of the libraries), because the version 1 reader returns a bare set of file ids. +- Both binfile handlers revalidate their permission at run time, not only when the job is created, because it can be revoked while the job waits: the export checks the read permission of every file before reading anything and stores the package in `job-resource` as the resource of the job (answering with the completion envelope), and the import checks the edition permission of the destination and runs the core inside a transaction, which is what makes a cancelled import roll back instead of leaving half of a file. The import releases its package (marks it as deleted and drops the reference from the job row in the same transaction) and deletes its temporary copy once the job is terminal, whatever the outcome. Both handlers check their own result against a schema before returning it, because the job-def has no result schema of its own. +- The domain cfg of a binfile request lives in `app.binfile.common` (`export-cfg`, `import-cfg`), so the legacy RPC and the jobs freeze the same ids, resolve the same team features and apply the same anti-zip-bomb limits. `resolve-export-type` is the translation the legacy RPC needs for its deprecated boolean flags; the creation command asks the caller for the type instead of guessing a default. +- Two retention knobs, easy to confuse: `:jobs-retention` sweeps terminal internal rows (the ones without a profile), while `:jobs-user-ttl` is the default expiry a user-facing job gets when it is created; the jobs GC deletes the row once `expires_at` passes and touches its resource. The heavy work of a user job runs on its own `:binfile` queue (`:worker-binfile-parallelism`, default 1) so it does not block the default runner. +- The quote of a family of jobs (`::export-jobs-per-profile`, `::import-jobs-per-profile`) counts the jobs of the profile that are still on their way (`new`/`scheduled`/`running`/`retry`) whose name the registry gives for that family, so a new job of the family is covered without touching the quote. +- A user-facing job is created through `app.rpc.commands.jobs`. One command per job type: `create-export-binfile-job` and `create-import-binfile-job` name the job they create and their body carries only `:params`, whose schema the command imports from the job namespace (`::sm/params`), so the RPC validates the business params up front, before the command runs; the job-def is still the place that owns them and validates them again at run time. The commands check the permission and the quote of the family before the expensive part, submit with `max-retries 0`, an expiry from `:jobs-user-ttl` and the queue the job-def routes to through `::jobs/queue-name` (`:binfile` when the def says nothing; read by the command, never by the substrate), and answer with a summary of the row. The import command has its own creation schema (`import-binfile/schema:create-params`, version optional) because the request is not the stored params: the command fills the version (from the caller or the package header) and the manifest metadata kept for audit. It assembles the upload, reads the manifest of a version 3 package for the audit (never its content) and stores the package in `job-resource` as the resource of the job, releasing it when the job never comes to own it. +- `get-job` is the read surface a profile has over its own jobs: it answers with the same summary the creation commands return plus the `result` of a finished job or its public error (through `decode-job-error`, so the keywords come back), and a job of another profile is not distinguishable from one that does not exist: both answer `not-found`. `decode-row` decodes `params`, `result` and `error`, and the result is the plain JSON the row holds, so a uuid comes back as the text it is on disk (a result has no schema of its own like the params do). The event log still has no read surface: a client that follows a job takes its progress from the websocket. +- `heartbeat` uses named options `[cfg & {:keys [job-id progress] :as options}]` and returns the number of durable writes, so 0 means there was nothing to write: no job context, or the throttle did not allow it. A write that finds the job no longer active (terminal or gone) raises an `:interrupt` (`code :job-interrupted`, carrying the job id and the status it found) through the shared `jobs/check-active` instead of counting zero, so a task in course stops at its next beat; the interrupt is internal and never reaches HTTP (the management progress report answers `:skip` for it). The job id falls back to `::jobs/job-id` on the cfg and then to the runner-bound `*job-id*`. The `modified_at` touch is throttled to ~1s and the progress event to ~200ms, so the check fires at most once per second with no extra query (the read of the row happens only when a write already found nothing). A progress report is a milestone (`schema:progress`): a required `:stage` keyword plus an optional `:counters` map of `scope -> {:current, :total?}`, and no other key. Counters are non-negative, a `total` is positive and never lower than its `current`, and stage and counter keys are names of at most 64 characters. A milestone without counters is valid (a point of the run with no units to count), the stage vocabulary belongs to the worker and not to the substrate, and a client must degrade for a stage it does not know. Read a stored payload back with `jobs/decode-progress`: the database holds the keywords as strings. The 1s beat (down from 60s, so a cancelled heavy job is seen as gone at its next beat) costs one `modified_at` touch per beating job per second: cheap while the only heavy beater runs alone on its queue, revisit if concurrent beating jobs ever grow. - `job.error` conforms to `app.jobs/schema:job-error` (`type` and `code` keywords, `hint` text, extra details allowed). Read it back with `jobs/decode-job-error`; a plain `db/decode-json-pgobject` leaves the keywords as strings. `complete` takes named options and an optional `resource-id`, which is only ever set, never replaced: it reads the row locked inside its own transaction and applies the rules in a fixed order, skipping a job that is no longer completable before refusing one for its resource, so the answer is a function of the row and not of the order two callers arrive in. For worker dispatch, cron, retry semantics (`ex/raise :type ::wrk/retry` with `:delay`/`:strategy`), deduplication, job events, and queue internals: `mem:backend/subtleties`. diff --git a/.serena/memories/backend/storage.md b/.serena/memories/backend/storage.md index baeba165a2..58cb134930 100644 --- a/.serena/memories/backend/storage.md +++ b/.serena/memories/backend/storage.md @@ -77,6 +77,7 @@ Since `put-object!` uses backend-specific operations (`impl/resolve-backend` + ` - Objects can therefore share content across users and files within one bucket. - Deleted objects are not reused. - `tempfile` objects never use deduplication, even when the caller requests it. +- `job-resource` objects never use deduplication either, whatever the caller asks: the object belongs to one profile, and a shared blob would serve the artifact of one user to the job of another. `put-object!` excludes the bucket from the dedupable set, so the invariant does not depend on call sites. - Use `sto/wrap-with-hash` when the caller already calculated the content hash. ## Bucket Rules @@ -94,7 +95,7 @@ Since `put-object!` uses backend-specific operations (`impl/resolve-backend` + ` | `file-data` | Encoded file data when `file-data-backend` is `storage`. Reference metadata has `storage-ref-id`, `file-id`, and the `file_data` row ID. | Yes | Authentication required | Reference scan. | | `file-data-fragment` | Compatibility value for file-data fragments. The current backend has no dedicated producer for this bucket. | No current write semantics | Public | No touched-object collector case. | | `file-change` | Compatibility value for file changes. Current snapshots store data in `file_data`, not this bucket. | No current write semantics | Authentication required | No touched-object collector case. | -| `job-resource` | Storage objects owned by job rows (`job.resource_id`). | Yes | Authentication required | jobs-GC touch → reference scan. | +| `job-resource` | Storage objects owned by job rows (`job.resource_id`), written by `app.jobs.storage`. | No | Authentication required, owner-scoped | jobs-GC touch → reference scan. | - The valid bucket set is `app.storage.schema/metadata-buckets` (derived from `bucket-requirements`, where a new bucket and its required metadata keys are one entry); `app.storage/valid-buckets` aliases it. - `file-media-object` is the default bucket for old rows without bucket metadata. @@ -108,6 +109,7 @@ Since `put-object!` uses backend-specific operations (`impl/resolve-backend` + ` - `app.http.assets` decides direct object authentication from the bucket. - Public buckets are `file-media-object`, `file-object-thumbnail`, `team-font-variant`, `file-data-fragment`, and `organization`. - Other valid buckets require a session or access-token profile ID. +- `tempfile` and `job-resource` are owner-scoped: the request must come from the profile stored in the object metadata, and a mismatch answers 404 (counted as `unauthorized`). A missing owner is tolerated only in `tempfile`, where legacy objects predate the metadata; `job-resource` always carries one, so an object without it is never served. - File-media routes also require file read permission. - Non-public direct responses set `content-disposition: attachment`. - FS responses use `x-accel-redirect` for the configured asset path. diff --git a/.serena/memories/backend/subtleties.md b/.serena/memories/backend/subtleties.md index 41f781070b..3e1b73a482 100644 --- a/.serena/memories/backend/subtleties.md +++ b/.serena/memories/backend/subtleties.md @@ -9,6 +9,7 @@ - Params with `::sm/params` are decoded/conformed through the JSON transformer and successful IObj results get `:encode/json` metadata. Legacy spec conforming only applies when no Malli params schema exists. Client params are stripped of qualified keys (`d/without-qualified`) before merging with the server auth context, so request bodies cannot override `::profile-id`, `::auth-type`, or `::token-perms`. - Params schemas are open by default, so undeclared client keys reach the handler unless the map is `:closed true`. Creation commands (`create-file`, `create-project`, `create-team`, `create-team-with-invitations`, `upload-file-media-object`, `create-file-media-object-from-url`, `assemble-file-media-object`) use closed schemas: a client-provided `:id` fails with `:params-validation`. Their internal creation functions still accept an optional explicit `:id` for imports, duplicates and deterministic test fixtures. - Nil RPC bodies become HTTP 204 unless explicit status metadata is present. Stream bodies default to `application/octet-stream` when no content type is set. +- The upload schema (`media.v/schema:upload`) takes a `:path` from the caller, and a Transit body can tag any string as a `path`, so a command that reads such an upload reads a file the client named. `create-import-binfile-job` avoids that boundary by accepting only a chunked upload (`:upload-id`); the commands that still take a plain file (the legacy `import-binfile`, the media and font uploads) trust it, and the guard belongs where uploads are accepted, not in each command. ## DB helpers @@ -25,7 +26,7 @@ - `job` carries `tenant` (the instance the job belongs to) and a bare `queue`. Several instances share one database and the tenant is what keeps an instance from claiming another environment's jobs, so every query that acts on one instance's work filters on `tenant`: dispatch claim, orphan sweep, lost-job reschedule, cron no-overlap count, submit dedupe. The `job__dispatcher__idx` leads with `tenant` (equality) then `(priority DESC, scheduled_at)`; `queue` does not lead because the dispatcher claims every queue of its tenant. The `:` prefix survives only in the Redis key, composed once by `app.worker/queue-key`. - Submit dedupe only removes not-yet-due `new` jobs of the same tenant with the same name/queue/label; it does not dedupe due, scheduled, retry, running, or completed work. - The dispatcher selects `new`/`retry` jobs with `FOR UPDATE SKIP LOCKED`, marks them `scheduled`, and publishes the JSON payload `[id scheduled-at]` to the `penpot.worker.queue::` Redis list. The runner skips Redis messages whose scheduled timestamp no longer matches DB state. -- Lost `scheduled` jobs are rescheduled after 5 minutes; `running` jobs untouched longer than `:jobs-lease` (default 30 min) are marked `aborted` with `app.jobs/orphan-error` (`type :internal`, `code :orphan`). `aborted` is a system-side terminal state like `cancelled` (never retried, swept by retention like other terminals, reported with an error log); long jobs must `heartbeat` per chunk so a healthy backlog is never mistaken for a dead process. The `:progress` report is optional and only feeds the durable log. +- Lost `scheduled` jobs are rescheduled after 5 minutes; `running` jobs untouched longer than `:jobs-lease` (default 30 min) are marked `aborted` with `app.jobs/orphan-error` (`type :internal`, `code :orphan`). `aborted` is a system-side terminal state like `cancelled` (never retried, swept by retention like other terminals, reported with an error log); long jobs must `heartbeat` per chunk so a healthy backlog is never mistaken for a dead process. The `:progress` report is optional and only feeds the durable log. A beat whose write finds the job no longer active raises an `:interrupt` (`code :job-interrupted`) instead of counting zero: it is the internal stop-the-work signal, not an error, so neither the listener nor the runner logs it as one. - An orphan gets its `end` event (`outcome "aborted"`) from the sweep itself: one bulk `INSERT ... SELECT` in the same transaction as the bulk `UPDATE`, never one insert per orphan. No msgbus notification goes out for it; orphans alert through the error log and the orphaned counter. - `job_event` is append-only and deleted by cascade with the job. `progress` is a row there, not a column: no Redis and no mutable column to keep in sync. - `job` is deletion-protected at the SQL level (`deletion_protection__tgr`): a stray `DELETE FROM job` raises instead of dropping the row and its `job_event` cascade without touching the referenced `storage_object`. The only intended deleters, the jobs GC and the submit dedupe, disable the guard with `SET LOCAL rules.deletion_protection TO off` inside their transaction; tests use `th/db-force-delete`. @@ -36,6 +37,7 @@ ## Config and HTTP/session middleware - `app.config/config` and `flags` are dynamic `defonce` vars populated from `PENPOT_*` env vars through the shared schema string transformer. Tests and tooling can bind them. +- Duration config values are parsed as ISO-8601 with the `PT` prefix, so a day is written in hours (`:jobs-retention` and `:jobs-user-ttl` default to a week, and an operator writes `168h`): `7d` does not parse, and the value fails validation at boot instead of being silently ignored. - `parse-flags` automatically adds `:disable-secure-session-cookies` when `public-uri` is plain HTTP and not localhost. This changes cookie defaults without an explicit env flag. - The backend sets Clojure `*assert*` globally from the `:backend-asserts` feature flag. Assertion-dependent checks can therefore differ by runtime flags. - Request body parsing is mostly POST-oriented and supports Transit JSON plus plain JSON. Plain JSON request keys are kebab-decoded before being merged into `:params`. diff --git a/.serena/memories/backend/testing.md b/.serena/memories/backend/testing.md index cacfbadfca..75c049e475 100644 --- a/.serena/memories/backend/testing.md +++ b/.serena/memories/backend/testing.md @@ -12,4 +12,21 @@ JVM `clojure.test` (kaocha runner) under `backend/test/backend_tests/`. - Multiple focus symbols are supported by REPEATING the flag (kaocha accumulates them, last-wins does not apply): `clojure -M:dev:test --focus backend-tests.ns-one --focus backend-tests.ns-two` — preferred over a shell loop of isolated runs. - Regression run: `clojure -M:dev:test` to ensure no regressions in related functional areas. - If you need to filter output, tee to a temp file first: `clojure -M:dev:test 2>&1 | tee /tmp/penpot-test-output.txt`. -- RPC test helpers `command!`/`management-command!` split the data map: qualified keys become server params, unqualified keys become request body params. To inject request-level context (headers, `:app.http/auth-key-id`, ip), pass a map under `:app.http/request` metadata; non-map `IRequest` stubs fall back to a dummy request. \ No newline at end of file +- RPC test helpers `command!`/`management-command!` split the data map: qualified keys become server params, unqualified keys become request body params. To inject request-level context (headers, `:app.http/auth-key-id`, ip), pass a map under `:app.http/request` metadata; non-map `IRequest` stubs fall back to a dummy request. +- The test system is `main/system-config` + `main/worker-config` with the + components that must not run in the background dropped: HTTP, the srepl, + cron, the dispatcher and **every** `[ :app.worker/runner]` + (`helpers/remove-job-runners`). A live runner of any queue races the tests + that drive the loop by hand (`wdisp/run-batch` + `@#'wrkr/run-worker-loop`): + it takes the job the test just dispatched and completes it behind the test's + back, so the test's own assertions read a state it did not produce. Do not + re-add a runner by editing that list by hand; the removal matches by key + shape on purpose. +- `system-config` and `worker-config` are `def`s: their `(cf/get ...)` calls + are evaluated when the namespace loads, against the real environment, while + test code calling `cf/get` at run time sees the `penpot-test`-prefixed + config. The two can disagree — in the devenv `:tenant` is `devenv-ws0` for + the defs and `default` inside a test — so a component built from + `worker-config` can talk to a different Redis key prefix than a cfg the test + builds itself. The symptom is a component that silently does nothing + locally and races in CI (which sets no `PENPOT_TENANT`). \ No newline at end of file diff --git a/.serena/memories/clojure/design-rules.md b/.serena/memories/clojure/design-rules.md index edf273e152..cac863cc16 100644 --- a/.serena/memories/clojure/design-rules.md +++ b/.serena/memories/clojure/design-rules.md @@ -6,6 +6,7 @@ How to shape a function in this codebase. Each rule came from a review round; do - **Optional-by-design data is guarded where the optionality is born.** Example: `ConnectorStatistics` is nil when the server option is off, so the guard is `when-let [cs (some-> server ...)]` at the entry point and the consumer assumes `cs`. - **A static set of operations is written statically.** With a fixed, small set (four metrics) write the four calls. Do not build a collection and iterate it (`doseq` over a literal vector): it allocates, adds indirection and hides each operation. Use a collection when the set is dynamic (config, registry, input). - **Default to no `!`; existing `!` is legacy being removed gradually.** Do not introduce new `!` names except the allowed cases documented in `mem:clojure/idioms` (add and justify the special case there first). Allowed: mutating public fns in `app.db` (`insert!`, `update!`, `delete!`, `run!`, `tx-run!`, ...) and fns whose contract is to work over transients (stdlib `conj!`/`assoc!`/`dissoc!`/`persistent!` and custom helpers over them). Existing `!` elsewhere (e.g. `emit!`, `send!`, `submit!`, `validate-file!`) stays until its refactor; keep a family consistent in the meantime. Reporting helpers (log, sample-and-publish) never take `!`: the mutation happens in the `!` API they call. +- **`?` marks a predicate, not a data prop.** A var or function that answers a question takes `?` (`export-error?`, `(defn editable? …)`); a map key that names a property does not (`:export-error`, never `:export-error?`), even when its value is a boolean. Existing `?` keys are legacy to remove gradually; do not add a new one. - **Do not shape production code for tests.** No return values added just to assert them; assert on the observable effect. When preconditions move, move the tests with them. Reference implementation: the http metrics samplers in `app.http` (`sample-worker-metrics` / `sample-connector-metrics` are guard-free; `sample-http-metrics` guards at the boundary). diff --git a/.serena/memories/frontend/dashboard-viewer-subtleties.md b/.serena/memories/frontend/dashboard-viewer-subtleties.md index f4cfd45cc9..097d3648fb 100644 --- a/.serena/memories/frontend/dashboard-viewer-subtleties.md +++ b/.serena/memories/frontend/dashboard-viewer-subtleties.md @@ -5,6 +5,7 @@ - Dashboard initialization fetches projects and fonts for the team, then listens to websocket messages only for global topic `uuid/zero` or the current profile id. - Project fetch replaces each project map completely instead of merging, so fields such as `deleted-at` can disappear cleanly. - Dashboard file/project mutations are often optimistic local updates with fire-and-forget RPC watchers. Bulk permanent delete/restore paths use SSE progress and progress notifications. +- The import wizard runs the package as a job of the backend: it reads the manifest of the file in the main thread (`app.main.data.imports`, no web worker) and follows the job, so its rows paint the milestone of the job (the stage and its counter, with the file in course) instead of the messages the old SSE stream sent. Before the job exists, the rows paint a client-side `:upload` milestone built from the chunked upload (`uploads/upload-blob-chunked` `:on-progress`, chunks done/total), which never comes from the backend; the chunks go one at a time and the row writes them as a percentage of the file, not as a counter. Once created and still waiting for a worker, each entry reports `:queued` ("Queued"); the `:start`/`:retry` events move it to bare progress, and only `:progress` carries a milestone. The export modal follows the same one-job-per-file model, but every file is queued up front (even the ones whose job is created only after the previous one is over) and its queue ends only with a milestone: `:start`/`:retry` do not end it there. Neither modal paints a generic "…ing file…" line any more: each row carries its own status. The `:error` of an import entry is user-facing text (the backend hint, or a translated label), never an i18n key: the wizard renders it as-is, it does not translate it again. The legacy SSE import stays in the backend for other clients. - File creation/duplication strips file `:data` before putting file summaries into dashboard state. ## Viewer diff --git a/.serena/memories/frontend/routing-app-shell-subtleties.md b/.serena/memories/frontend/routing-app-shell-subtleties.md index ea3e98243d..5c99aae4da 100644 --- a/.serena/memories/frontend/routing-app-shell-subtleties.md +++ b/.serena/memories/frontend/routing-app-shell-subtleties.md @@ -19,4 +19,5 @@ For general store mechanics such as `emit!`, `last-events`, persistence, and undo, read `mem:frontend/workspace-state-persistence-subtleties`. - Websocket initialization uses `cf/public-uri` joined with `ws/notifications`, converting `http/https` to `ws/wss`, and includes the current `session-id` as query param. -- Reinitializing or finalizing websocket stops the previous receive stream. Incoming websocket payloads become Potok data events under `app.main.data.websocket/message`. \ No newline at end of file +- Reinitializing or finalizing websocket stops the previous receive stream. Incoming websocket payloads become Potok data events under `app.main.data.websocket/message`. +- A job the client created is followed with `app.main.data.jobs/watch-job`: it emits the `:job-event` messages of that job (filtered by `:job-id`) and one final outcome read from the row with `get-job`, which is read when the stream starts and again on every `opened` of the websocket, so a job that ended before the client started following it, or while the socket was down, still answers. The stream ends after that outcome, and a read that fails reaches the caller: the repo retries the transient transport errors of a GET, and anything else is a real failure the flow has to show. The `:stage` of a progress event is a stable key of the vocabulary of the worker, never a display string: `app.main.ui.jobs.progress` resolves it with a `case` of literal `tr` calls, falls back to a generic label for a stage it does not know, and the counter of the activity is named after its own stage (the counter of `:files` is the outer context). \ No newline at end of file diff --git a/.serena/memories/frontend/translations.md b/.serena/memories/frontend/translations.md index 7053af9d4b..806ac5d990 100644 --- a/.serena/memories/frontend/translations.md +++ b/.serena/memories/frontend/translations.md @@ -7,10 +7,11 @@ high-coverage support reference, never the base. ## Workflow - Canonicalize with `node ./scripts/translations.js sync -l ` from `frontend/`: sorts entries, syncs `#:` comments from `en`, deletes keys missing in `en`. +- `sync` never ADDS a key missing in the locale: it only syncs comments (for keys that already exist) and drops keys absent from `en`. A key added to `en` alone stays missing in `es` and its text silently falls back to English. Every new key is born in `en` AND `es` in the same change; compare the `msgid` sets to check (`comm -23 <(grep '^msgid ' en.po | sort -u) <(grep '^msgid ' es.po | sort -u)` must be empty). - `sync` copies ALL comment flags from `en`, including `#, fuzzy`. Translated entries must NOT stay fuzzy: strip the flag after translating (mirror `es.po`, which keeps fuzzy only on genuinely untranslated entries). Fuzzy entries are excluded from `msgfmt --statistics` translated counts. - Canonical files carry NO `#~` obsolete blocks (`en`/`es` have zero). Drop them; `sync` does not resurrect them. -- Definition of done: `msgfmt --check .po` exit 0 and `msgfmt --statistics` shows 0 fuzzy, 0 untranslated. -- `rehash` scans `frontend/src` AND `common/src` for `(tr "key"` occurrences (any line counts, including `;;` comments) and refreshes `#:` refs in `en`, marking unreferenced keys `#, unused`. +- Definition of done: `msgfmt --check .po` exit 0, `msgfmt --statistics` shows 0 fuzzy, 0 untranslated, and the locale's `msgid` set is identical to `en.po`'s (no key missing, none extra). +- `rehash` scans `frontend/src` AND `common/src` for `(tr "key"` occurrences (any line counts, including `;;` comments) and refreshes `#:` refs in `en`, marking unreferenced keys `#, unused`. The refs in `en` are broadly stale, so a repo-wide `rehash` rewrites hundreds of lines (measured: ~487 on `en` alone): it is a maintenance action on its own, never a step to run inside a change that only adds or edits a key. - Dynamic keys are invisible to `rehash`: eliminate them instead of declaring them. Preference order: literal `(tr "key")` args only; never put a branch inside `tr`, hoist it out (`(if cond (tr "a") (tr "b"))`); @@ -28,6 +29,12 @@ high-coverage support reference, never the base. ## Entry rules +- Placement: a key names the feature that renders it (`dashboard.import.*`, + `files-export-modal.*`, `jobs.*`); `labels.*` is the bucket for the short + generic words reused across screens (Accept, Cancel, Download…), never the + home of a feature's own vocabulary. Keep a feature's strings together: the + durable jobs write their progress stages as `jobs.progress.stage.*` and + their row statuses as `jobs.*`, not half in `labels.*`. - Keys resolved dynamically from server-provided data carry a `#, backend` flag (set it in `en`, `sync` copies it to the locales): do not remove or rename these entries, and keep their `%s` placeholders (the backend only diff --git a/.serena/memories/prod-infra/core.md b/.serena/memories/prod-infra/core.md index d4999d86b8..7872efa128 100644 --- a/.serena/memories/prod-infra/core.md +++ b/.serena/memories/prod-infra/core.md @@ -12,9 +12,9 @@ Backend (`app.config`, `PENPOT_*` env vars) is parameterized; deployments choose ## Task queue and worker model -Async jobs are enqueued via `jobs/submit` (`app.jobs`), which inserts a row into the shared Postgres `job` table tagged with `queue = ":"`. Submission is **fire-and-forget** — RPC handlers never poll and never wait. The completion signal is the `job` row's `status` / `completed_at` columns, which nothing in `rpc/` reads today. A job submitted with `::jobs/profile-id` also publishes a `:job-event` message on that profile's msgbus topic after each transaction commits, but no caller passes a profile-id yet, so nothing reaches msgbus in production. Soft-delete RPCs return immediately after marking the top-level row, leaving the cascade and reaping to workers. +Async jobs are enqueued via `jobs/submit` (`app.jobs`), which inserts a row into the shared Postgres `job` table tagged with `queue = ":"`. Submission is **fire-and-forget** — RPC handlers never poll and never wait. The completion signal is the `job` row's `status` / `completed_at` columns, which the creation commands read back to answer with the state of the job they just created. A job submitted with `::jobs/profile-id` also publishes a `:job-event` message on that profile's msgbus topic after each transaction commits, and the binfile commands (`create-export-binfile-job`, `create-import-binfile-job`) are the callers that pass that owner, so the progress of their jobs does reach the msgbus. Soft-delete RPCs return immediately after marking the top-level row, leaving the cascade and reaping to workers. -Workers run on backends with `enable-backend-worker` in `PENPOT_FLAGS`. Each worker-enabled backend has a `dispatcher` (polls `job` with `FOR UPDATE SKIP LOCKED`, marks status='scheduled', RPUSHes the JSON `[id scheduled-at]` payloads into **its own** Redis list) and one or more `runner`s per queue (BLPOP from that same local list, execute, update the Postgres row). The Redis hand-off list is purely intra-backend — cross-backend coordination happens at the Postgres row level. +Workers run on backends with `enable-backend-worker` in `PENPOT_FLAGS`. Each worker-enabled backend has a `dispatcher` (polls `job` with `FOR UPDATE SKIP LOCKED`, marks status='scheduled', RPUSHes the JSON `[id scheduled-at]` payloads into **its own** Redis list) and one or more `runner`s per queue (BLPOP from that same local list, execute, update the Postgres row). The Redis hand-off list is purely intra-backend — cross-backend coordination happens at the Postgres row level. The queues are `default`, `webhooks`, `cron` and `binfile`: the last one carries the heavy work of a user job (`:worker-binfile-parallelism`, default 1), so a long export or import does not block the default runner. ## Cross-backend safety diff --git a/backend/resources/climit.edn b/backend/resources/climit.edn index c109b2afdb..1f59a0d3d3 100644 --- a/backend/resources/climit.edn +++ b/backend/resources/climit.edn @@ -53,6 +53,12 @@ :import-binfile/by-profile {:permits 1 :queue 2} + :create-import-binfile-job/global + {:permits 4} + + :create-import-binfile-job/by-profile + {:permits 1 :queue 2} + :profile-plugin-ops/global {:permits 4} diff --git a/backend/src/app/binfile/common.clj b/backend/src/app/binfile/common.clj index 2275a2cc6d..42085f7647 100644 --- a/backend/src/app/binfile/common.clj +++ b/backend/src/app/binfile/common.clj @@ -73,11 +73,11 @@ ;; Maximum total decompressed size allowed for all JSON/text zip entries ;; combined within a single import job: 500MiB. Bounds the case where many ;; entries, each individually under default-max-text-entry-size, still sum -;; to an unreasonable total. Set well above the 200MiB it replaces because -;; legitimate giant exports exist: hundreds of thousands of KB-sized -;; entries (pages, shapes) that never trip the per-entry cap but sum past -;; 400MiB. The per-entry cap still bounds single-entry DEFLATE -;; amplification, so this only moves the cumulative ceiling. +;; to an unreasonable total. Set this high because legitimate giant +;; exports exist: hundreds of thousands of KB-sized entries (pages, +;; shapes) that never trip the per-entry cap but sum past 400MiB. The +;; per-entry cap still bounds single-entry DEFLATE amplification, so this +;; only moves the cumulative ceiling. (def ^:const default-max-text-total-size (* 1024 1024 500)) @@ -1029,3 +1029,45 @@ {:bytes bytes :size (alength ^bytes bytes) :hash (sto/calculate-hash (java.io.ByteArrayInputStream. bytes))}))) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; REQUEST CFG +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(defn resolve-export-type + "The canonical export type of a request on the legacy RPC: an explicit + `type` wins, and the deprecated boolean flags are translated. The + creation command of a job does not need it, because it asks the caller + for the type instead of guessing it." + [{:keys [type include-libraries embed-assets]}] + (cond + (some? type) type + (true? include-libraries) :include-libraries + (true? embed-assets) :merge-libraries + :else :detach-libraries)) + +(defn export-cfg + "The domain cfg of an export: the frozen set of files and how their + libraries are handled." + [cfg {:keys [ids export-type]}] + (-> cfg + (assoc ::ids ids) + (assoc ::export-type export-type))) + +(defn import-cfg + "The domain cfg of an import: the destination team/profile/project, the + name, the input and the anti-zip-bomb limits. The caller resolves the + team, because what allows it to import there differs between the RPC and + a job; what both must share is fixed here." + [cfg {:keys [profile-id project-id team name input]}] + (-> cfg + (assoc ::features (cfeat/get-team-enabled-features cf/flags team)) + (assoc ::project-id project-id) + (assoc ::profile-id profile-id) + (assoc ::team-id (:id team)) + (assoc ::name name) + (assoc ::input input) + (assoc ::import-max-binary-entry-size (cf/get :binfile-import-max-binary-entry-size)) + (assoc ::import-max-text-entry-size (cf/get :binfile-import-max-text-entry-size)) + (assoc ::import-max-text-total-size (cf/get :binfile-import-max-text-total-size)) + (assoc ::import-max-zip-entries (cf/get :binfile-import-max-zip-entries)))) diff --git a/backend/src/app/binfile/jobs.clj b/backend/src/app/binfile/jobs.clj new file mode 100644 index 0000000000..c947ae858a --- /dev/null +++ b/backend/src/app/binfile/jobs.clj @@ -0,0 +1,171 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns app.binfile.jobs + "The binfile work a job runs. + + A job has no SSE channel, so the `:progress` points that `bf.v1` and + `bf.v3` emit become `progress` events of the job through + `jobs/heartbeat`, which throttles them and drops the ones that do not + fit. The adapter turns a tap into a self-contained milestone: a `:stage` + of a single vocabulary (the two formats name the same work differently) + and the counters the tap carries, with the outer scopes in course as + context. The reader puts the outer counters in every tap inside them, + so the adapter holds no state between taps. The ids the taps carry are + not part of the job event contract. + + The adapter checks that the job is still active before it starts, and + the beat checks it on every write: a job that was cancelled, or that + already reached a terminal state, raises an `:interrupt` right there on + the run thread, so the work stops at its next event point and the + transaction of its handler rolls back with it. A run whose last tap + happened before the interruption is discarded once it returns, which is + what makes that transaction roll back too." + (:require + [app.binfile.common :as bfc] + [app.binfile.v1 :as bf.v1] + [app.binfile.v3 :as bf.v3] + [app.common.exceptions :as ex] + [app.common.time :as ct] + [app.db :as db] + [app.jobs :as jobs] + [app.util.events :as events])) + +(defn check-active + "Raise an `:interrupt` when the job is no longer active. + + The substrate owns both the check and the error; this is the adapter's + entry point to it, used by the two handlers before their run and by the + tests." + [cfg context] + (jobs/check-active cfg (:id context))) + +(def ^:private stage-of + "The stage a tap section becomes: one vocabulary for the two formats, so + a client resolves each stage once. The counter of the activity is named + after its own stage, which is what lets a client render + `counters[stage]` without a table of equivalences." + {:file :files + :page :pages + :media :media + :thumbnail :thumbnails + :thumbnails :thumbnails + :component :components + :color :colors + :typography :typographies + :tokens-lib :tokens + :tokens-status :tokens + :storage-object :storage-objects + :storage-objects :storage-objects + :relations :relations + :manifest :manifest + :v1/metadata :metadata + :v1/files :files + :v1/rels :relations + :v1/sobjects :storage-objects}) + +(defn- stage + "The stage of a tap section; a section without a name of its own keeps + it, so a new tap is never dropped from the stream." + [section] + (get stage-of section section)) + +(defn- counter + "The counter a tap reports, when it reports one: a tap that only marks a + point of the run has no units to count." + [{:keys [current total]}] + (when (some? current) + (cond-> {:current current} + (some? total) + (assoc :total total)))) + +(defn- milestone + "The milestone a tap becomes: what the job is doing now and the counters + the tap carries, with the outer scopes in course as context. + + The tap of a scope carries its outer counters, so the milestone is + self-contained and the adapter remembers nothing between taps: a client + that misses one (the substrate throttles the events) still renders the + truth from the next." + [{:keys [section outer-counters] :as tap}] + (let [stage (stage section) + own (counter tap)] + (cond-> {:stage stage} + (or (some? outer-counters) (some? own)) + (assoc :counters (cond-> (or outer-counters {}) + (some? own) (assoc stage own)))))) + +(defn- with-progress + "Run `f` with the progress taps of the core turned into milestones of the + job. + + Every tap is reported: the substrate throttles the durable events, and + a milestone that does not fit loses no information because the next one + carries the same state. + + The taps run on this same thread: the sink calls the heartbeat inline, + so an `:interrupt` it raises aborts the core right there, inside the + transaction of its handler, which rolls back." + [cfg context f] + (let [job-id (:id context) + sink (fn [[type data]] + (when (= :progress type) + (jobs/heartbeat cfg + :job-id job-id + :progress (milestone data))))] + (binding [events/*sink* sink] + (let [result (f)] + (check-active cfg context) + result)))) + +(defn export-files + "Export a set of files into `output` as part of a job, reporting + progress as job events." + [cfg context {:keys [ids export-type output]}] + (check-active cfg context) + (with-progress cfg context + #(bf.v3/export-files! (bfc/export-cfg cfg {:ids ids + :export-type export-type}) + output))) + +(defn- touch-project + "Mark the destination project as modified. + + An import adds files, and the triggers of the schema only touch the + project on an update, so a project that just received files keeps its + old date and stays where it was in the list." + [cfg project-id] + (db/update! cfg :project + {:modified-at (ct/now)} + {:id project-id} + {::db/return-keys false})) + +(defn import-files + "Import the package at `input` as part of a job, reporting progress as + job events. The package was read from the resource of the job, so the + caller resolves the destination team and the adapter only fixes what + the legacy RPC and a job must share. + + Both formats answer with the same shape: the version 1 reader returns + the set of files it created, and the version 3 one returns a map that + already carries them together with the resolution of the libraries." + [cfg context {:keys [profile-id project-id team name input version]}] + (check-active cfg context) + (let [cfg (bfc/import-cfg cfg {:profile-id profile-id + :project-id project-id + :team team + :name name + :input input})] + (with-progress cfg context + #(let [result (case (int version) + 1 {:file-ids (vec (bf.v1/import-files! cfg))} + 3 (bf.v3/import-files! cfg) + (ex/raise :type :validation + :code :unsupported-version + :hint "unsupported binfile version" + :version version))] + (touch-project cfg project-id) + result)))) diff --git a/backend/src/app/binfile/v3.clj b/backend/src/app/binfile/v3.clj index 1d411336f7..338a4a8a6a 100644 --- a/backend/src/app/binfile/v3.clj +++ b/backend/src/app/binfile/v3.clj @@ -231,6 +231,32 @@ (def validate-file-thumbnail (sm/check-fn schema:file-thumbnail)) +;; --- PROGRESS + +(defn- tap-progress + "Emit a progress tap of the core, with the counters of the outer scopes + in course when the cfg has any. + + The outer counters live in `::outer-counters` of the cfg while a scope + is being processed (today only `:files`, tomorrow `:teams` and + `:projects` too), so every tap inside carries them as `:outer-counters`: + the stages inside do not repeat them, and whoever reads the taps (the + jobs adapter) builds its report from one tap alone instead of + remembering the last position it saw. A tap that happens outside any + scope carries no outer counters, which is the truth about it." + [cfg data] + (events/tap :progress + (cond-> data + (some? (::outer-counters cfg)) + (assoc :outer-counters (::outer-counters cfg))))) + +(defn- with-outer-counter + "Return cfg with the counter of `scope` set to `position`, keeping the + counters of the other outer scopes already in course, so entering a + nested scope never wipes the ones above it." + [cfg scope position] + (assoc-in cfg [::outer-counters scope] position)) + ;; --- EXPORT IMPL (defn- write-entry! @@ -264,8 +290,10 @@ (defn- export-storage-objects [{:keys [::output] :as cfg}] - (let [storage (sto/resolve cfg)] - (doseq [id (-> bfc/*state* deref :storage-objects not-empty)] + (let [storage (sto/resolve cfg) + ids (-> bfc/*state* deref :storage-objects not-empty) + total (count ids)] + (doseq [[index id] (d/enumerate ids)] (let [sobject (sto/get-object storage id) smeta (meta sobject) ext (cmedia/mtype->extension (:content-type smeta)) @@ -277,7 +305,8 @@ (write-entry! output path params) - (events/tap :progress {:section :storage-object :id id}) + (tap-progress cfg {:section :storage-object :id id + :current (inc index) :total total}) (with-open [input (sto/get-object-data storage sobject)] (.putNextEntry ^ZipOutputStream output (ZipEntry. (str "objects/" id ext))) @@ -305,7 +334,7 @@ thumbnails (bfc/get-file-object-thumbnails cfg file-id)] - (events/tap :progress {:section :file :id file-id :name (:name file)}) + (tap-progress cfg {:section :file :id file-id :name (:name file)}) (vswap! bfc/*state* update :files assoc file-id {:id file-id @@ -339,7 +368,8 @@ (write-entry! output path page) - (events/tap :progress {:section :page :id page-id :name (:name page) :file-id file-id}) + (tap-progress cfg {:section :page :id page-id :name (:name page) :file-id file-id + :current (inc index) :total (count pages)}) (doseq [[shape-id shape] objects] (let [path (str "files/" file-id "/pages/" page-id "/" shape-id ".json") @@ -350,30 +380,33 @@ (vswap! bfc/*state* bfc/collect-storage-objects media) (vswap! bfc/*state* bfc/collect-storage-objects thumbnails) - (doseq [{:keys [id] :as media} media] - (let [path (str "files/" file-id "/media/" id ".json") - media (encode-media media)] + (doseq [[index {:keys [id] :as item}] (d/enumerate media)] + (let [path (str "files/" file-id "/media/" id ".json") + encoded (encode-media item)] - (events/tap :progress {:section :media :id id :file-id file-id}) - (write-entry! output path media))) + (tap-progress cfg {:section :media :id id :file-id file-id + :current (inc index) :total (count media)}) + (write-entry! output path encoded))) - (doseq [thumbnail thumbnails] + (doseq [[index thumbnail] (d/enumerate thumbnails)] (let [data (cth/parse-object-id (:object-id thumbnail)) path (str "files/" file-id "/thumbnails/" (:tag data) "/" (:page-id data) "/" (:frame-id data) ".json") data (-> data (assoc :media-id (:media-id thumbnail)) (encode-file-thumbnail))] - (events/tap :progress {:section :thumbnails :id (:object-id thumbnail) :file-id file-id}) + (tap-progress cfg {:section :thumbnails :id (:object-id thumbnail) :file-id file-id + :current (inc index) :total (count thumbnails)}) (write-entry! output path data))) - (doseq [[id component] components] + (doseq [[index [id component]] (d/enumerate components)] (let [path (str "files/" file-id "/components/" id ".json") component (encode-component component)] - (events/tap :progress {:section :component :id id :file-id file-id}) + (tap-progress cfg {:section :component :id id :file-id file-id + :current (inc index) :total (count components)}) (write-entry! output path component))) - (doseq [[id color] colors] + (doseq [[index [id color]] (d/enumerate colors)] (let [path (str "files/" file-id "/colors/" id ".json") color (-> (encode-color color) (dissoc :file-id)) @@ -381,20 +414,22 @@ (and (contains? color :path) (str/empty? (:path color))) (dissoc :path))] - (events/tap :progress {:section :color :id id :file-id file-id}) + (tap-progress cfg {:section :color :id id :file-id file-id + :current (inc index) :total (count colors)}) (write-entry! output path color))) - (doseq [[id object] typographies] + (doseq [[index [id object]] (d/enumerate typographies)] (let [path (str "files/" file-id "/typographies/" id ".json") typography (encode-typography object)] - (events/tap :progress {:section :typography :id id :file-id file-id}) + (tap-progress cfg {:section :typography :id id :file-id file-id + :current (inc index) :total (count typographies)}) (write-entry! output path typography))) (when (and tokens-lib (not (ctob/empty-lib? tokens-lib))) (let [path (str "files/" file-id "/tokens.json") encoded-tokens (encode-tokens-lib tokens-lib)] - (events/tap :progress {:section :tokens-lib :file-id file-id}) + (tap-progress cfg {:section :tokens-lib :file-id file-id}) (write-entry! output path encoded-tokens))) (when tokens-status @@ -410,7 +445,7 @@ encoded-status (-> (encode-tokens-status tokens-status) (cfo/ids->names effective-tokens-lib))] - (events/tap :progress {:section :tokens-status :file-id file-id}) + (tap-progress cfg {:section :tokens-status :file-id file-id}) (write-entry! output path encoded-status))))) (defn- export-files @@ -449,7 +484,8 @@ (doseq [[index file-id] (d/enumerate ids)] (-> cfg (assoc ::file-id file-id) - (assoc ::file-seqn index) + (with-outer-counter :files {:current (inc index) + :total (count ids)}) (export-file))) ;; Write manifest file @@ -742,17 +778,19 @@ (defn- read-file-colors [{:keys [::bfc/input ::entries-index] :as cfg} file-id] - (->> (get-in entries-index [:colors (str file-id)]) - (reduce (fn [result {:keys [id entry]}] - (let [object (->> (read-entry cfg input entry) - (decode-color) - (validate-color))] - (events/tap :progress {:section :color :id id :file-id file-id}) - (if (= id (:id object)) - (assoc result id object) - result))) - {}) - (not-empty))) + (let [entries (get-in entries-index [:colors (str file-id)])] + (->> (d/enumerate entries) + (reduce (fn [result [index {:keys [id entry]}]] + (let [object (->> (read-entry cfg input entry) + (decode-color) + (validate-color))] + (tap-progress cfg {:section :color :id id :file-id file-id + :current (inc index) :total (count entries)}) + (if (= id (:id object)) + (assoc result id object) + result))) + {}) + (not-empty)))) (defn- read-file-components [{:keys [::bfc/input ::entries-index] :as cfg} file-id] @@ -771,15 +809,18 @@ (reduce-kv (fn [objects id shape] (assoc objects id (bfl/clean-shape-pre-decode shape))) objects - objects))))] + objects)))) + entries + (get-in entries-index [:components (str file-id)])] - (->> (get-in entries-index [:components (str file-id)]) - (reduce (fn [result {:keys [id entry]}] + (->> (d/enumerate entries) + (reduce (fn [result [index {:keys [id entry]}]] (let [object (->> (read-entry cfg input entry) (clean-component-pre-decode) (decode-component) (clean-component-post-decode))] - (events/tap :progress {:section :component :id id :file-id file-id}) + (tap-progress cfg {:section :component :id id :file-id file-id + :current (inc index) :total (count entries)}) (if (= id (:id object)) (assoc result id object) result))) @@ -788,22 +829,24 @@ (defn- read-file-typographies [{:keys [::bfc/input ::entries-index] :as cfg} file-id] - (->> (get-in entries-index [:typographies (str file-id)]) - (reduce (fn [result {:keys [id entry]}] - (let [object (->> (read-entry cfg input entry) - (decode-typography) - (validate-typography))] - (events/tap :progress {:section :typography :id id :file-id file-id}) - (if (= id (:id object)) - (assoc result id object) - result))) - {}) - (not-empty))) + (let [entries (get-in entries-index [:typographies (str file-id)])] + (->> (d/enumerate entries) + (reduce (fn [result [index {:keys [id entry]}]] + (let [object (->> (read-entry cfg input entry) + (decode-typography) + (validate-typography))] + (tap-progress cfg {:section :typography :id id :file-id file-id + :current (inc index) :total (count entries)}) + (if (= id (:id object)) + (assoc result id object) + result))) + {}) + (not-empty)))) (defn- read-file-tokens-lib [{:keys [::bfc/input ::entries-index] :as cfg} file-id] (when-let [{:keys [entry]} (first (get-in entries-index [:tokens (str file-id)]))] - (events/tap :progress {:section :tokens-lib :file-id file-id}) + (tap-progress cfg {:section :tokens-lib :file-id file-id}) (->> (read-entry cfg input entry nil) (decode-tokens-lib) (validate-tokens-lib)))) @@ -811,7 +854,7 @@ (defn- read-file-tokens-status [{:keys [::bfc/input ::entries-index] :as cfg} file-id tokens-lib] (when-let [{:keys [entry]} (first (get-in entries-index [:tokens-status (str file-id)]))] - (events/tap :progress {:section :tokens-status :file-id file-id}) + (tap-progress cfg {:section :tokens-status :file-id file-id}) (->> (read-entry cfg input entry nil) (cfo/names->ids tokens-lib) (decode-tokens-status) @@ -833,19 +876,21 @@ (defn- read-file-pages [{:keys [::bfc/input ::entries-index] :as cfg} file-id] - (->> (get-in entries-index [:pages (str file-id)]) - (keep (fn [{:keys [id entry]}] - (let [page (->> (read-entry cfg input entry) - (decode-page)) - page (dissoc page :options)] - (events/tap :progress {:section :page :id id :file-id file-id}) - (when (= id (:id page)) - (let [objects (read-file-shapes cfg file-id id)] - (assoc page :objects objects)))))) - (sort-by :index) - (reduce (fn [result {:keys [id] :as page}] - (assoc result id (dissoc page :index))) - (d/ordered-map)))) + (let [entries (get-in entries-index [:pages (str file-id)])] + (->> (d/enumerate entries) + (keep (fn [[index {:keys [id entry]}]] + (let [page (->> (read-entry cfg input entry) + (decode-page)) + page (dissoc page :options)] + (tap-progress cfg {:section :page :id id :file-id file-id + :current (inc index) :total (count entries)}) + (when (= id (:id page)) + (let [objects (read-file-shapes cfg file-id id)] + (assoc page :objects objects)))))) + (sort-by :index) + (reduce (fn [result {:keys [id] :as page}] + (assoc result id (dissoc page :index))) + (d/ordered-map))))) (defn- read-file-thumbnails [{:keys [::bfc/input ::entries-index] :as cfg} file-id] @@ -886,8 +931,11 @@ :plugin-data plugin-data})) (defn- import-file - [{:keys [::db/conn ::bfc/project-id ::manifest] :as cfg} {file-id :id file-name :name}] - (let [file-id' (bfc/lookup-index file-id) + [{:keys [::db/conn ::bfc/project-id ::manifest] :as cfg} + {file-id :id file-name :name} + position] + (let [cfg (with-outer-counter cfg :files position) + file-id' (bfc/lookup-index file-id) file (read-file cfg file-id) media (read-file-media cfg file-id) thumbnails (read-file-thumbnails cfg file-id)] @@ -899,16 +947,19 @@ :version (:version file) ::l/sync? true) + (tap-progress cfg {:section :file :file-id file-id}) + (vswap! bfc/*state* update :index bfc/update-index media :id) - (doseq [item media] + (doseq [[index item] (d/enumerate media)] (let [params (-> item (update :id bfc/lookup-index) (assoc :file-id file-id') (d/update-when :media-id bfc/lookup-index) (d/update-when :thumbnail-id bfc/lookup-index))] - (events/tap :progress {:section :media :id (:id params) :file-id file-id}) + (tap-progress cfg {:section :media :id (:id params) :file-id file-id + :current (inc index) :total (count media)}) (l/dbg :hint "inserting media object" :file-id (str file-id') @@ -921,7 +972,7 @@ (db/insert! conn :file-media-object params ::db/on-conflict-do-nothing? (::bfc/overwrite cfg)))) - (doseq [item thumbnails] + (doseq [[index item] (d/enumerate thumbnails)] (let [media-id (bfc/lookup-index (:media-id item)) object-id (-> (assoc item :file-id file-id') (cth/fmt-object-id)) @@ -935,13 +986,12 @@ :media-id (str media-id) ::l/sync? true) - (events/tap :progress {:section :thumbnail :file-id file-id :object-id object-id}) + (tap-progress cfg {:section :thumbnail :file-id file-id :object-id object-id + :current (inc index) :total (count thumbnails)}) (db/insert! conn :file-tagged-object-thumbnail params ::db/on-conflict-do-nothing? true))) - (events/tap :progress {:section :file :file-id file-id}) - (let [data (-> (read-file-data cfg file-id (:tokens-source file)) (d/without-nils) (assoc :id file-id') @@ -970,26 +1020,26 @@ (defn- import-file-relations [{:keys [::db/conn ::manifest ::bfc/timestamp] :as cfg}] - (events/tap :progress {:section :relations}) - (doseq [[file-id libr-id] (:relations manifest)] + (let [relations (:relations manifest)] + (doseq [[index [file-id libr-id]] (d/enumerate relations)] + (tap-progress cfg {:section :relations + :current (inc index) :total (count relations)}) - (let [file-id (bfc/lookup-index file-id) - libr-id (bfc/lookup-index libr-id)] + (let [file-id (bfc/lookup-index file-id) + libr-id (bfc/lookup-index libr-id)] - (when (and file-id libr-id) - (l/dbg :hint "create file library link" - :file-id (str file-id) - :lib-id (str libr-id) - ::l/sync? true) - (let [rel-params {:file-id file-id - :library-file-id libr-id}] - (db/insert! conn :file-library-rel rel-params) - (bfc/upsert-file-library-sync! conn (assoc rel-params :synced-at timestamp))))))) + (when (and file-id libr-id) + (l/dbg :hint "create file library link" + :file-id (str file-id) + :lib-id (str libr-id) + ::l/sync? true) + (let [rel-params {:file-id file-id + :library-file-id libr-id}] + (db/insert! conn :file-library-rel rel-params) + (bfc/upsert-file-library-sync! conn (assoc rel-params :synced-at timestamp)))))))) (defn- import-storage-objects [{:keys [::bfc/input ::entries-index ::bfc/timestamp] :as cfg}] - (events/tap :progress {:section :storage-objects}) - ;; IMPORTANT: we strongly do not reuse the main connection that can ;; run inside a transaction because the storage upload process can ;; fail in the middle of uploading and leave garbage on the underlying @@ -998,9 +1048,12 @@ ;; what the storage subsystem registers in other parallel ;; transaction (let [storage (sto/resolve cfg) - entries (:objects entries-index)] + entries (:objects entries-index) + total (count entries)] - (doseq [{:keys [id entry]} entries] + (doseq [[index {:keys [id entry]}] (d/enumerate entries)] + (tap-progress cfg {:section :storage-objects + :current (inc index) :total total}) (let [object (-> (read-entry cfg input entry) (decode-storage-object) (d/update-when :content-type bfc/normalize-content-type) @@ -1182,12 +1235,15 @@ (vswap! bfc/*state* update :index assoc old-lib-id library-id))) (let [files (get manifest :files) - file-ids (reduce (fn [result file] + total (count files) + file-ids (reduce (fn [result [index file]] (let [name' (get file :name) - file (assoc file :name name')] - (conj result (import-file cfg file)))) + file (assoc file :name name')] + (conj result (import-file cfg file + {:current (inc index) + :total total})))) [] - files) + (d/enumerate files)) ;; Build map of file-id to file-name for resolution files-info (into {} (map (fn [file-id manifest-file] [file-id (:name manifest-file)]) @@ -1244,7 +1300,7 @@ bfc/*reference-file* ref-file] (import-storage-objects cfg) - (import-file cfg file) + (import-file cfg file {:current 1 :total 1}) (invalidate-thumbnails cfg file-id) (bfm/apply-pending-migrations! cfg) @@ -1299,7 +1355,7 @@ (cfeat/check-supported-features! features))) - (events/tap :progress {:section :manifest}) + (tap-progress cfg {:section :manifest}) (binding [bfc/*state* (volatile! {:media [] :index {}})] (if (::bfc/file-id cfg) @@ -1352,6 +1408,11 @@ (vreset! ab true) nil) + (catch clojure.lang.ExceptionInfo cause + (if (= :interrupt (:type (ex-data cause))) + (vreset! ab true) + (throw cause))) + (catch Throwable cause (vreset! cs cause) (vreset! ab true) diff --git a/backend/src/app/config.clj b/backend/src/app/config.clj index 10a97b7bee..a4375c2184 100644 --- a/backend/src/app/config.clj +++ b/backend/src/app/config.clj @@ -79,6 +79,7 @@ :jobs-lease (ct/duration {:minutes 30}) :jobs-retention (ct/duration {:days 7}) + :jobs-user-ttl (ct/duration {:days 7}) :jobs-request-timeout (ct/duration {:minutes 2}) :media-max-file-size (* 1024 1024 30) ; 30MiB @@ -101,6 +102,8 @@ :quotes-upload-sessions-per-profile 5 :quotes-upload-chunks-per-session 20 + :quotes-export-jobs-per-profile 10 + :quotes-import-jobs-per-profile 10 :upload-max-chunk-size (* 1024 1024 30) ; 30MiB ;; SSRF protection @@ -196,6 +199,7 @@ [:file-clean-delay {:optional true} ::ct/duration] [:jobs-lease {:optional true} ::ct/duration] [:jobs-retention {:optional true} ::ct/duration] + [:jobs-user-ttl {:optional true} ::ct/duration] [:jobs-request-timeout {:optional true} ::ct/duration] [:telemetry-enabled {:optional true} ::sm/boolean] [:default-blob-version {:optional true} ::sm/int] @@ -215,6 +219,7 @@ [:worker-default-parallelism {:optional true} ::sm/int] [:worker-webhook-parallelism {:optional true} ::sm/int] [:worker-cron-parallelism {:optional true} ::sm/int] + [:worker-binfile-parallelism {:optional true} ::sm/int] [:database-password {:optional true} [:maybe :string]] [:database-uri {:optional true} ::sm/uri] @@ -233,6 +238,8 @@ [:quotes-font-variants-per-team {:optional true} ::sm/int] [:quotes-comment-threads-per-file {:optional true} ::sm/int] [:quotes-comments-per-file {:optional true} ::sm/int] + [:quotes-export-jobs-per-profile {:optional true} ::sm/int] + [:quotes-import-jobs-per-profile {:optional true} ::sm/int] [:quotes-snapshots-per-file {:optional true} ::sm/int] [:quotes-snapshots-per-team {:optional true} ::sm/int] [:quotes-team-access-requests-per-team {:optional true} ::sm/int] @@ -474,6 +481,15 @@ (or (c/get config :jobs-retention) (ct/duration {:days 7}))) +(defn get-jobs-user-ttl + "How long a user-facing job, and the artifact it owns, is kept before + the jobs GC deletes the row and marks the object for the storage GC. + Not the same as `:jobs-retention`, which only sweeps terminal internal + rows." + [] + (or (c/get config :jobs-user-ttl) + (ct/duration {:days 7}))) + (defn get "A configuration getter. Helps code be more testable." ([key] diff --git a/backend/src/app/http/assets.clj b/backend/src/app/http/assets.clj index e2c41d0886..e75f599fdc 100644 --- a/backend/src/app/http/assets.clj +++ b/backend/src/app/http/assets.clj @@ -150,14 +150,22 @@ [request] (some? (request-profile-id request))) -(defn- tempfile-owner-match? - "Check if the request's profile-id matches the tempfile's stored owner. - Returns true if no profile-id was stored (legacy objects)." +(def ^:private owner-scoped-buckets + "Buckets whose objects belong to a profile: a request from anybody else + must not reach them." + #{sto/tempfile-bucket sto/job-resource-bucket}) + +(defn- owner-match? + "Check that the request belongs to the profile that owns the object. + A missing owner is only tolerated in `tempfile`, where objects predate + the metadata; the `job-resource` bucket always carries one, so an + object without it fails closed." [obj request] - (let [stored-profile-id (:profile-id (meta obj)) - request-profile-id (request-profile-id request)] - (or (nil? stored-profile-id) - (= stored-profile-id request-profile-id)))) + (let [stored (-> obj meta :profile-id) + owner (request-profile-id request) + legacy? (= sto/tempfile-bucket (-> obj meta :bucket))] + (or (and legacy? (nil? stored)) + (= stored owner)))) (defn- serve-object-measured "Serve `obj`, recording one asset metric per outcome. A failure is @@ -175,7 +183,9 @@ "Handler that serves storage objects by id. For non-public buckets (e.g. profile), requires authentication via session cookie or access token. - For tempfile bucket, also requires ownership (profile-id match)." + For the owner-scoped buckets (tempfile, job-resource), also requires + ownership: the request must come from the profile the object belongs + to." [{:keys [::sto/storage] :as cfg} request] (let [id (get-id request) obj (sto/get-object storage id)] @@ -193,8 +203,8 @@ ;; The response stays 404 to avoid leaking existence, but the ;; metric records the internal 401 outcome. - (and (= (-> obj meta :bucket) sto/tempfile-bucket) - (not (tempfile-owner-match? obj request))) + (and (contains? owner-scoped-buckets (-> obj meta :bucket)) + (not (owner-match? obj request))) (do (emit-asset! cfg "by-id" obj 401) {::yres/status 404}) diff --git a/backend/src/app/http/sse.clj b/backend/src/app/http/sse.clj index 0fc7c385bd..4c1420876e 100644 --- a/backend/src/app/http/sse.clj +++ b/backend/src/app/http/sse.clj @@ -60,14 +60,13 @@ (partial write! output) (partial pu/close! output))] (try - (binding [events/*channel* channel] - (let [result (handler)] - (events/tap :end result))) - - (catch Throwable cause - (let [result (errors/handle' cause request)] - (events/tap channel :error result))) - + (binding [events/*sink* channel] + (try + (let [result (handler)] + (events/tap :end result)) + (catch Throwable cause + (let [result (errors/handle' cause request)] + (events/tap :error result))))) (finally (sp/close! channel) (px/await! listener))))))})) diff --git a/backend/src/app/jobs.clj b/backend/src/app/jobs.clj index 849ecb3784..be36f25549 100644 --- a/backend/src/app/jobs.clj +++ b/backend/src/app/jobs.clj @@ -9,9 +9,11 @@ Every job is defined as a job-def map returned by the `ig/init-key` of its module: {::name, ::schema, ::handler, ::decoder, ::validator} — decoder and - validator are precompiled at init time. The registry of job-defs is plain - integrant wiring (::jobs/defs); submit, dispatch and management consume it - by reference. + validator are precompiled at init time. A job-def may also carry the + optional metadata {::family, ::resource-role}, which the consumers of the + job (quotes, storage helpers) read from the registry: the substrate never + reads them. The registry of job-defs is plain integrant wiring (::jobs/defs); + submit, dispatch and management consume it by reference. Two execution modes are provided: - `submit` (durable): validates + JSON-encodes params and inserts a row @@ -27,8 +29,10 @@ on its loop, otherwise the dispatcher marks it `aborted` (a system-side terminal state, like `cancelled` but set by the dispatcher instead of a user: never retried, kept for triage and reported with an error - log) while its side effects continue. `heartbeat` also accepts an - optional `progress` report, which is stored as a `progress` row of + log) while its side effects continue. A beat that finds the job no + longer active raises an `:interrupt`, an internal signal that the task + should stop the work in course, never an error. `heartbeat` also accepts + an optional `progress` report, which is stored as a `progress` row of `job_event`: progress is durable history, not a mutable column, and needs no Redis. @@ -37,7 +41,7 @@ follow the job without polling the database. Reserved ledger columns (`profile_id`, `error`, `result`, `resource_id`, - `expires_at`) are only written by `submit` (profile and resource + `expires_at`) are only written by `submit` (profile, resource and expiry references) and by the terminal writers; `submit` never infers them from `params`." (:require @@ -82,14 +86,20 @@ whether to schedule again. A handler has no use for the counter, and the attempt number would lie anyway: the `noop` retry strategy runs again without incrementing it, so a durable \"which execution is this\" - number would need a second counter that nothing bounds." + number would need a second counter that nothing bounds. + + `profile-id` is the owner of a user-facing job and nil for an internal + one. It travels here so a handler can revalidate permissions and check + who owns the resource it is working on without a second query." [:map {:closed true :title "job-context"} [:id ::sm/uuid] [:name ::sm/text] [:label [:maybe ::sm/text]] ;; technical reference kept for garbage collection - [:resource-id [:maybe ::sm/uuid]]]) + [:resource-id [:maybe ::sm/uuid]] + ;; owner of a user-facing job; nil for an internal job + [:profile-id [:maybe ::sm/uuid]]]) (def check-context "Validate a handler context; raises with the malli explanation when it @@ -97,7 +107,7 @@ (sm/check-fn schema:context)) (defn make-context - "Build the handler context from a job row: exactly the four keys a + "Build the handler context from a job row: exactly the five keys a handler may see, and nothing else. The result is a plain map, not the database row, and it is not modified afterwards. @@ -107,7 +117,8 @@ (check-context {:id (:id job) :name (:name job) :label (if (str/blank? (:label job)) nil (:label job)) - :resource-id (:resource-id job)})) + :resource-id (:resource-id job) + :profile-id (:profile-id job)})) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; JOB DEFINITIONS (registry) @@ -123,6 +134,12 @@ (def ^:private schema:job-def [:map {:title "job-def"} [::name schema:job-name] + ;; Optional metadata of the job family, read from the registry by the + ;; consumers that need it (quotes, storage helpers, a future job + ;; listing). `app.jobs` itself never reads them: they are not part of + ;; the execution contract. + [::family {:optional true} ::sm/keyword] + [::resource-role {:optional true} ::sm/keyword] [::schema any?] ;; every handler is [context params]. A job-def already closes over its ;; own dependencies, so nothing else is handed to it: a handler cannot @@ -211,9 +228,9 @@ (def ^:private sql:insert-new-job "insert into job (id, name, tenant, params, queue, label, priority, - max_retries, profile_id, resource_id, created_at, - modified_at, scheduled_at) - values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + max_retries, profile_id, resource_id, expires_at, + created_at, modified_at, scheduled_at) + values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) returning id") ;; `tenant` is part of the match, not an extra: without it this DELETE @@ -243,7 +260,11 @@ [::profile-id {:optional true} ::sm/uuid] ;; Technical reference (storage object) kept for garbage collection. It ;; is a column, never part of `params`, and it is never inferred from it. - [::resource-id {:optional true} ::sm/uuid]]) + [::resource-id {:optional true} ::sm/uuid] + ;; Retention of the job: when it is set, the jobs GC deletes the row + ;; (and reclaims its resource) once the instant has passed. User-facing + ;; jobs set it at creation; internal jobs leave it nil. + [::expires-at {:optional true} ::ct/inst]]) (def check-options (sm/check-fn schema:options)) @@ -264,13 +285,17 @@ encodes them as plain JSON and inserts a row into the `job` table. Fire-and-forget: returns the job id immediately. + The optional ledger columns (`::profile-id`, `::resource-id` and + `::expires-at`) are validated and stored as columns; nothing is inferred + from `params`. + NOTE: the dedupe DELETE and the INSERT run atomically: joined to the caller's transaction when the cfg provides `::db/conn`, wrapped in their own transaction otherwise. Concurrent cross-backend submissions can, in rare race conditions, produce duplicated 'new' rows (accepted risk, see prod-infra documentation)." [cfg {:keys [::params ::name ::delay ::queue ::priority ::max-retries - ::dedupe ::label ::profile-id ::resource-id] + ::dedupe ::label ::profile-id ::resource-id ::expires-at] :or {delay 0 queue :default priority 100 max-retries 3 label ""} :as options}] @@ -322,7 +347,7 @@ :replace (or deleted 0)) (db/exec-one! conn [sql:insert-new-job id job-name tenant payload queue label priority max-retries - profile-id resource-id + profile-id resource-id expires-at now now scheduled-at])))] ;; Both statements always run inside db/tx-run!: joined to the ;; caller's transaction when the cfg provides a connection, @@ -348,7 +373,10 @@ (defn- decode-row [row] - (decode-json-col row :params)) + (-> row + (decode-json-col :params) + (decode-json-col :result) + (decode-json-col :error))) (defn get-job "Retrieve the job row (with the raw JSON params decoded to a plain map)." @@ -387,52 +415,93 @@ (def ^:private known-retry-reasons #{"backoff" "noop"}) -(def ^:private max-stage-length 250) +(def ^:private max-progress-key-length + "Longest accepted `:stage` or counter key. They name a vocabulary, they + are not a place for data, so the cap only keeps a runaway key out of the + log." + 64) + +(def schema:progress-counter + "Counter of one scope of a job: `current` is how many units are done and + `total` how many there are, when the worker knows it." + [:map {:closed true + :title "job-progress-counter"} + [:current ::sm/int] + [:total {:optional true} ::sm/int]]) (def schema:progress - "Progress report of a job: `current` is mandatory, `total` and `stage` are - optional, and no other key is accepted. `stage` is a short human label, - not a place for exception messages or params." + "Progress milestone of a job: the activity in course and the counters of + the scopes around it, each one under its own key. + + `:stage` is a stable keyword the client resolves (a translation, an + icon...), never a display string, and `:counters` is open on purpose: a + worker reports the scopes it knows and names the counter of the activity + after its own stage, so a client can render `counters[stage]` without a + table of equivalences. Neither is a place for names, params or exception + text: the payload is keywords and integers only." [:map {:closed true :title "job-progress"} - [:current ::sm/int] - [:total {:optional true} ::sm/int] - [:stage {:optional true} ::sm/text]]) + [:stage ::sm/keyword] + [:counters {:optional true} [:map-of ::sm/keyword schema:progress-counter]]]) (def check-progress - "Validate a progress report against its schema." + "Validate the shape of a progress milestone." (sm/check-fn schema:progress)) +(defn- check-counter + "Check the range rules of one counter: a non-negative `current` and a + positive `total` never lower than its `current`." + [key counter] + (when (neg? (:current counter)) + (ex/raise :type :validation + :code :invalid-progress + :hint "progress counter current must not be negative" + :counter key + :progress counter)) + (when-let [total (:total counter)] + (when (or (not (pos? total)) + (> (:current counter) total)) + (ex/raise :type :validation + :code :invalid-progress + :hint "progress counter total must be positive and not lower than current" + :counter key + :progress counter)))) + (defn validate-progress - "Check the range rules a schema cannot express: non-negative `current`, - positive `total`, `current` never greater than `total` and a short - `stage`." + "Check what the schema cannot express: non-negative counters, a positive + total never lower than its `current`, and stage and counter keys short + enough to be a name and not a payload." [progress] (let [progress (check-progress progress)] - (when (neg? (:current progress)) - (ex/raise :type :validation - :code :invalid-progress - :hint "progress current must not be negative" - :progress progress)) - (when-let [total (:total progress)] - (when (or (not (pos? total)) - (> (:current progress) total)) + (doseq [key (cons (:stage progress) (keys (:counters progress)))] + (when (> (count (name key)) max-progress-key-length) (ex/raise :type :validation :code :invalid-progress - :hint "progress total must be positive and not lower than current" + :hint (str "progress key must not be longer than " + max-progress-key-length " characters") + :key key :progress progress))) - (when (and (:stage progress) - (> (count (:stage progress)) max-stage-length)) - (ex/raise :type :validation - :code :invalid-progress - :hint (str "progress stage must not be longer than " - max-stage-length " characters") - :progress progress)) + (doseq [[key counter] (:counters progress)] + (check-counter key counter)) progress)) +(defn decode-progress + "Read back a stored progress payload as a Clojure map: `:stage` and the + counter keys come back as the strings the database holds, so they are + keywordized here, and a payload that is already a map is returned as is." + [progress] + (when (some? progress) + (-> (cond-> progress + (db/pgobject? progress) + (db/decode-json-pgobject)) + (d/update-when :stage keyword) + (d/update-when :counters + (fn [counters] + (into {} (map (fn [[k v]] [(keyword k) v])) counters)))))) + (def ^:private event-payload-keys {"start" #{:attempt} - "progress" #{:current :total :stage} + "progress" #{:stage :counters} "retry" #{:attempt :reason} "end" #{:outcome}}) @@ -490,7 +559,7 @@ :payload payload :created-at created-at})) -(defn insert-event +(defn- insert-event "Insert a `job_event` row, then count it and notify after the commit. Must be called inside a transaction that owns the job row. When the job @@ -501,7 +570,11 @@ The counter also runs after the commit, so a rolled back transaction never inflates it." [{:keys [::mbus/msgbus] :as cfg} job-id kind payload] - (let [payload (validate-event-payload kind payload) + (assert (db/connection-map? cfg) + "expected cfg with valid connection: insert-event must run inside a transaction") + (assert (some? db/*after-commit-context*) + "expected transaction context: insert-event must run inside a transaction") + (let [payload (validate-event-payload kind payload) connectable (db/get-connectable cfg) profile-id (:profile-id (db/exec-one! connectable [sql:job-event-owner job-id]))] (when (and profile-id @@ -543,8 +616,14 @@ ;; HEARTBEAT / PROGRESS ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; -(def ^:private heartbeat-interval (ct/duration {:seconds 60})) -(def ^:private progress-interval (ct/duration {:seconds 1})) +(def ^:private heartbeat-interval + ;; one second, down from a minute: a heavy job that was cancelled must + ;; be seen as gone at its next beat. The price is one `modified_at` + ;; touch per beating job per second (60x the old rate): cheap while the + ;; only heavy beater runs alone on its queue, revisit if concurrent + ;; beating jobs ever grow. + (ct/duration {:seconds 1})) +(def ^:private progress-interval (ct/duration {:millis 200})) (def ^:private prune-threshold 10000) (def ^:private prune-window (ct/duration {:hours 1})) @@ -597,6 +676,12 @@ (swap! heartbeats dissoc job-id) (swap! progresses dissoc job-id)) +(def ^:private active-statuses + "The statuses of a job that is still on its way. It mirrors the filter + of the heartbeat and progress writes below: a row outside this set is no + longer active and nothing can report to it." + #{"new" "scheduled" "running" "retry"}) + (def ^:private sql:touch-heartbeat "UPDATE job SET modified_at = ? @@ -632,6 +717,27 @@ 1) 0)))) +(defn check-active + "Raise an `:interrupt` when the job is no longer active. + + A job is active while its status is one of `active-statuses`; a row + that already reached a terminal state, or that no longer exists, is + not. The status is read from the row, because a write can only tell + that it matched no row, not why. + + The interrupt is an internal signal, not an error: the caller is a task + that must stop the work in course. `heartbeat` calls this when one of + its writes finds the job gone, and a caller can also ask explicitly, + which is what the binfile adapter does before and after its run." + [cfg job-id] + (let [job (get-job cfg job-id)] + (when-not (and (some? job) (contains? active-statuses (:status job))) + (ex/raise :type :interrupt + :code :job-interrupted + :hint "the job is no longer active" + :job-id job-id + :status (or (:status job) "gone"))))) + (defn heartbeat "Keep a running job alive and, optionally, report its progress. @@ -642,14 +748,18 @@ - `:progress` optional progress report (see `schema:progress`). - `::force?` internal: skips only the progress throttle. - Returns the number of durable writes performed, so 0 means the job is - terminal, has no job context, or the throttle did not allow a write. + Returns the number of durable writes performed, so 0 means there was + nothing to write: no job context, or the throttle did not allow it. A + write that finds the job no longer active raises an `:interrupt` (see + `check-active`) instead of counting zero, so a task in course stops at + its next beat. - The `modified_at` touch is throttled to ~60s and the progress event to - ~1s: external workers report progress as sparse milestones, while - handlers may beat on every iteration. Both writes go through the - connection pool (`::db/pool` on the cfg), never the caller transaction, - so a beat survives a rollback of the surrounding work. + Both writes go through the connection pool (`::db/pool` on the cfg), + never the caller transaction, so a beat survives a rollback of the + surrounding work: the `modified_at` touch is throttled to ~1s while + progress events go out at up to ~200ms, so the user sees movement + without hammering the row. External workers report progress as sparse + milestones, while handlers may beat on every iteration. Like every other function here the cfg is a cfg: a bare pool or connection is a caller bug and is not accommodated. @@ -662,7 +772,7 @@ forced path used by the management API propagates it so the external worker can retry." [cfg & {:keys [job-id progress] :as options}] - (let [job-id (or job-id (get cfg ::job-id) *job-id*) + (let [job-id (or job-id (get cfg ::job-id) *job-id*) ;; a malformed report is a caller bug: never swallow it progress (some-> progress validate-progress)] (when (uuid? job-id) @@ -671,20 +781,31 @@ force? (boolean (::force? options)) writes (volatile! 0)] (when (should-write? heartbeats job-id now heartbeat-interval) - (vswap! writes + (touch-job connectable job-id now))) + (let [n (touch-job connectable job-id now)] + ;; the touch is conditional on the row still being active: a + ;; write that matched no row is what raises the interrupt + (when (zero? n) + (check-active cfg job-id)) + (vswap! writes + n))) (when (and (some? progress) (or force? (should-write? progresses job-id now progress-interval))) - (let [store #(report-progress cfg job-id progress)] - (if force? - (vswap! writes + (store)) - (try - (vswap! writes + (store)) - (catch Throwable cause - (l/err :hint "unable to persist job progress" - :tenant (cf/get :tenant) - :job-id (str job-id) - :cause cause)))))) + (let [store #(report-progress cfg job-id progress) + n (if force? + (store) + (try + (store) + (catch Throwable cause + (l/err :hint "unable to persist job progress" + :tenant (cf/get :tenant) + :job-id (str job-id) + :cause cause) + 0)))] + ;; the raise stays outside the catch above: an interrupt is + ;; not a transient failure and must not be swallowed with it + (when (zero? n) + (check-active cfg job-id)) + (vswap! writes + n))) (int @writes))))) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; @@ -976,14 +1097,17 @@ "UPDATE job SET status='cancelled', modified_at=? WHERE id=? - AND status IN ('new','scheduled','retry') + AND status IN ('new','scheduled','retry','running') RETURNING id") (defn cancel - "Cancel a pending job (new/scheduled/retry) and record the `end` event. + "Cancel a pending or running job (new/scheduled/retry/running) and + record the `end` event. - A running or terminal job is left untouched and writes no event. Returns - the number of affected rows." + A running job stops at its next heartbeat: the beat finds no active + row and raises an `:interrupt`, which the runner treats as a normal + stop while the handler rolls back. A terminal job is left untouched + and writes no event. Returns the number of affected rows." [cfg job-id] (assert (mtx/instance cfg) "missing ::mtx/metrics on the jobs cfg") (let [job (get-job cfg job-id) diff --git a/backend/src/app/jobs/storage.clj b/backend/src/app/jobs/storage.clj new file mode 100644 index 0000000000..dd33252cb7 --- /dev/null +++ b/backend/src/app/jobs/storage.clj @@ -0,0 +1,152 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns app.jobs.storage + "Storage helpers for the objects a job owns through `job.resource_id`. + + A job resource lives in the `job-resource` bucket and belongs to the + profile that owns the job, so only that profile can read it back (the + asset route checks the owner, not just the session). + + The job row is what keeps the object alive: `storage-gc-touched` + freezes it while a row references it and deletes it once no row does, + and the jobs GC marks it as touched when it removes the row. The object + carries no expiry of its own: `expired-at` is stored as `deleted_at`, + which makes the object unreadable from birth, so the retention of the + artifact is the retention of its job." + (:require + [app.common.exceptions :as ex] + [app.common.logging :as l] + [app.common.time :as ct] + [app.config :as cf] + [app.db :as db] + [app.storage :as sto] + [app.storage.tmp :as tmp])) + +(defn- resource-uri + "The url a user downloads the artifact from. A string, because the + descriptor ends up in the JSON result of the job." + [id] + (cf/get-public-uri "assets/by-id/" (str id))) + +(defn put-resource + "Store an artifact of a job in the `job-resource` bucket and return the + descriptor that identifies it. + + Named options: + + - `:content` the artifact, an `app.storage/content` value built from a + temporary file or from bytes in memory + - `:filename` the name the user will see + - `:mtype` the content type of the artifact + + `profile-id` is the owner of the object: the profile that owns the job, + for both the package a job consumes and the artifact it produces. + + The object is marked as touched right away, so an artifact that no job + row ever references (a crash between the upload and the completion) is + reclaimed by the storage GC instead of leaking; while a row references + it, `storage-gc-touched` freezes it." + [cfg profile-id {:keys [content filename mtype]}] + (assert (uuid? profile-id) "expected the profile that owns the resource") + (assert (string? mtype) "expected the content type of the resource") + (let [storage (sto/resolve cfg) + object (sto/put-object! storage + {::sto/content content + ::sto/touched-at (ct/now) + :bucket sto/job-resource-bucket + :profile-id profile-id + :content-type mtype})] + {:resource-id (:id object) + :resource-uri (resource-uri (:id object)) + :filename filename + :mtype mtype + :size (:size object)})) + +(defn- check-input + "The object a job consumes must exist, be a job resource and belong to + the profile that owns the job. Raises `:not-found` otherwise: a handler + never reads another profile's package." + [context object] + (when (or (nil? object) + (not= sto/job-resource-bucket (:bucket (meta object))) + (not= (:profile-id context) (:profile-id (meta object)))) + (ex/raise :type :not-found + :code :job-resource-not-found + :hint "the input of the job is not available" + :job-id (:id context) + :resource-id (:resource-id context))) + object) + +(defn load-input + "Download the package a job consumes to a local temporary file and + return its path, ready for the binfile reader. + + An object that was released or expired is already invisible to + `sto/get-object`, so it lands in the same `:not-found` as a missing + one." + [cfg context] + (let [storage (sto/resolve cfg) + resource-id (:resource-id context) + object (check-input context + (when (uuid? resource-id) + (sto/get-object storage resource-id)))] + (if-let [input (sto/get-object-data storage object)] + (tmp/tempfile-from input) + (ex/raise :type :not-found + :code :job-resource-not-found + :hint "the input of the job is not readable" + :job-id (:id context) + :resource-id resource-id)))) + +(defn release-resource + "Release a job resource. + + The object is marked as deleted and, when it belongs to a job, the + row stops pointing at it in the same transaction: a row never + references an object that is already gone, and the foreign key is not + what ends up clearing the reference. A resource no job owns (the + package of a creation that never submitted) is released with a nil + job id, and then no row is touched at all. + + Best effort: a failure is logged and never changes the terminal state + of the job, and releasing without a resource is a no-op. Returns true + when the object was marked as deleted or a stale reference was + cleared." + ([cfg resource-id] + (release-resource cfg resource-id nil)) + ([cfg resource-id job-id] + (when (uuid? resource-id) + (try + (db/tx-run! + cfg + (fn [tx-cfg] + (let [storage (sto/resolve tx-cfg ::db/reuse-conn true) + deleted? (sto/del-object! storage resource-id) + ;; the job no longer owns the package: the pointer goes with + ;; it, so clearing it does not depend on the object being + ;; physically deleted later + cleared? (when (some? job-id) + (db/update! tx-cfg :job + {:resource-id nil :modified-at (ct/now)} + ["id = ? AND resource_id = ?" + job-id resource-id] + {::db/return-keys [:id]}))] + (or deleted? (some? cleared?))))) + (catch Throwable cause + (l/wrn :hint "unable to release the input of a job" + :job-id (str job-id) + :resource-id (str resource-id) + :cause cause) + false))))) + +(defn release-input + "Release the package a job consumed. + + Delegates to `release-resource` with the id of the job: the object + and the pointer are dropped together. Best effort, like there." + [cfg context] + (release-resource cfg (:resource-id context) (:id context))) diff --git a/backend/src/app/main.clj b/backend/src/app/main.clj index 419dd88175..87d0b31f29 100644 --- a/backend/src/app/main.clj +++ b/backend/src/app/main.clj @@ -590,6 +590,8 @@ :file-gc (ig/ref :app.tasks.file-gc/job-def) :offload-file-data (ig/ref :app.tasks.offload-file-data/job-def) :objects-gc (ig/ref :app.tasks.objects-gc/job-def) + :export-binfile (ig/ref :app.tasks.export-binfile/job-def) + :import-binfile (ig/ref :app.tasks.import-binfile/job-def) :storage-gc-deleted (ig/ref ::sto.gc-deleted/job-def) :storage-gc-touched (ig/ref ::sto.gc-touched/job-def) :storage-pending-gc (ig/ref ::sto.pending-gc/job-def) @@ -642,6 +644,19 @@ ::sto/storage (ig/ref ::sto/storage) ::mtx/metrics (ig/ref ::mtx/metrics)} + :app.tasks.export-binfile/job-def + {::db/pool (ig/ref ::db/pool) + ::sto/storage (ig/ref ::sto/storage) + ::mtx/metrics (ig/ref ::mtx/metrics) + ;; the job belongs to a profile, so its events are published + ::mbus/msgbus (ig/ref ::mbus/msgbus)} + + :app.tasks.import-binfile/job-def + {::db/pool (ig/ref ::db/pool) + ::sto/storage (ig/ref ::sto/storage) + ::mtx/metrics (ig/ref ::mtx/metrics) + ::mbus/msgbus (ig/ref ::mbus/msgbus)} + :app.storage.gc-deleted/job-def {::db/pool (ig/ref ::db/pool) ::sto/storage (ig/ref ::sto/storage) @@ -840,6 +855,17 @@ ::db/pool (ig/ref ::db/pool) ::mbus/msgbus (ig/ref ::mbus/msgbus)} + ;; the heavy work of a user job does not share the default runner + [::binfile ::wrk/runner] + {::wrk/parallelism (cf/get :worker-binfile-parallelism 1) + ::wrk/queue :binfile + ::wrk/tenant (cf/get :tenant) + ::rds/client (ig/ref ::rds/client) + ::jobs/defs (ig/ref ::jobs/defs) + ::mtx/metrics (ig/ref ::mtx/metrics) + ::db/pool (ig/ref ::db/pool) + ::mbus/msgbus (ig/ref ::mbus/msgbus)} + [::cron ::wrk/runner] {::wrk/parallelism (cf/get :worker-cron-parallelism 2) ::wrk/queue :cron diff --git a/backend/src/app/rpc.clj b/backend/src/app/rpc.clj index cf80790b2e..d1b5247b95 100644 --- a/backend/src/app/rpc.clj +++ b/backend/src/app/rpc.clj @@ -379,6 +379,7 @@ 'app.rpc.commands.files-update 'app.rpc.commands.files-snapshot 'app.rpc.commands.files-thumbnails + 'app.rpc.commands.jobs 'app.rpc.commands.ldap 'app.rpc.commands.management 'app.rpc.commands.media diff --git a/backend/src/app/rpc/commands/binfile.clj b/backend/src/app/rpc/commands/binfile.clj index de0d3faadd..67163c1568 100644 --- a/backend/src/app/rpc/commands/binfile.clj +++ b/backend/src/app/rpc/commands/binfile.clj @@ -10,7 +10,6 @@ [app.binfile.common :as bfc] [app.binfile.v1 :as bf.v1] [app.binfile.v3 :as bf.v3] - [app.common.features :as cfeat] [app.common.schema :as sm] [app.common.time :as ct] [app.common.uri :as u] @@ -47,19 +46,12 @@ [:embed-assets {:optional true} ::sm/boolean]]) (defn- export-binfile - [{:keys [::sto/storage] :as cfg} {:keys [type file-id include-libraries embed-assets]}] - (let [output (tmp/tempfile*) - ;; Convert legacy boolean flags to unified export-type - export-type (cond - (some? type) type - (true? include-libraries) :include-libraries - (true? embed-assets) :merge-libraries - :else :detach-libraries)] + [{:keys [::sto/storage] :as cfg} {:keys [file-id] :as params}] + (let [output (tmp/tempfile*)] (try - (-> cfg - (assoc ::bfc/ids #{file-id}) - (assoc ::bfc/export-type export-type) + (-> (bfc/export-cfg cfg {:ids #{file-id} + :export-type (bfc/resolve-export-type params)}) (bf.v3/export-files! output)) (let [data (sto/content output) @@ -98,22 +90,13 @@ :project-id project-id) cfg - (-> cfg - (assoc ::bfc/features (cfeat/get-team-enabled-features cf/flags team)) - (assoc ::bfc/project-id project-id) - (assoc ::bfc/profile-id profile-id) - (assoc ::bfc/team-id (:id team)) - (assoc ::bfc/name name) - (assoc ::bfc/import-max-binary-entry-size (cf/get :binfile-import-max-binary-entry-size)) - (assoc ::bfc/import-max-text-entry-size (cf/get :binfile-import-max-text-entry-size)) - (assoc ::bfc/import-max-text-total-size (cf/get :binfile-import-max-text-total-size)) - (assoc ::bfc/import-max-zip-entries (cf/get :binfile-import-max-zip-entries))) + (bfc/import-cfg cfg {:profile-id profile-id + :project-id project-id + :team team + :name name + :input (:path file)}) - input-path (:path file) - owned? (some? upload-id) - - cfg - (assoc cfg ::bfc/input input-path) + owned? (some? upload-id) result (try @@ -126,7 +109,7 @@ :version version}))) (finally (when owned? - (fs/delete input-path))))] + (fs/delete (:path file)))))] (db/update! pool :project {:modified-at (ct/now)} diff --git a/backend/src/app/rpc/commands/jobs.clj b/backend/src/app/rpc/commands/jobs.clj new file mode 100644 index 0000000000..d1aa5b85cb --- /dev/null +++ b/backend/src/app/rpc/commands/jobs.clj @@ -0,0 +1,355 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns app.rpc.commands.jobs + "Creation of durable, user-facing jobs. + + One command per job type: the body carries the business params its + job-def declares and nothing else, and the job-def schema is the single + place that says what those params are. Creating a job never runs it: it + freezes what the job will do, and the work belongs to the runner." + (:require + [app.binfile.common :as bfc] + [app.binfile.v3 :as bf.v3] + [app.common.data :as d] + [app.common.exceptions :as ex] + [app.common.schema :as sm] + [app.common.time :as ct] + [app.config :as cf] + [app.db :as db] + [app.jobs :as jobs] + [app.jobs.storage :as js] + [app.loggers.audit :as-alias audit] + [app.loggers.webhooks :as-alias webhooks] + [app.rpc :as-alias rpc] + [app.rpc.climit :as-alias climit] + [app.rpc.commands.files :as files] + [app.rpc.commands.media :as media] + [app.rpc.commands.projects :as projects] + [app.rpc.doc :as-alias doc] + [app.rpc.quotes :as quotes] + [app.storage :as sto] + [app.tasks.export-binfile :as export-binfile] + [app.tasks.import-binfile :as import-binfile] + [app.util.services :as sv] + [datoteka.fs :as fs])) + +(def ^:private schema:create-export-binfile-job + "The `.penpot` package of the files frozen in the params, produced by + the backend runner; the business params are what the `:export-binfile` + job-def declares." + [:map {:title "create-export-binfile-job" :closed true} + [:params export-binfile/schema:params]]) + +(def ^:private schema:create-import-binfile-job + "The `.penpot` package the caller uploaded, imported into a project by + the backend runner; the business params are what the `:import-binfile` + job-def declares for creation." + [:map {:title "create-import-binfile-job" :closed true} + [:params import-binfile/schema:create-params] + ;; the package arrives as a chunked upload and never as a path the + ;; caller names: see `assemble-upload` + [:upload-id ::sm/uuid]]) + +(def ^:private schema:job-summary + "What the caller gets when a job is created: enough to follow it, and + nothing about the work it will do." + [:map {:title "job-summary"} + [:id ::sm/uuid] + [:status ::sm/text] + [:name ::sm/text] + [:created-at ::ct/inst] + [:expires-at [:maybe ::ct/inst]]]) + +(defn- resolve-job-def + "The job the caller named, refused when it is not a job of `family`: the + registry is the source of truth, and a name that is not one of its jobs + is rejected before anything is read or stored. + + The job-def is returned because the caller needs it right away: it is + what decodes and validates the business params." + [cfg family name] + (let [job-def (get (jobs/get-defs cfg) (keyword name))] + (when-not (= family (::jobs/family job-def)) + (ex/raise :type :validation + :code :not-a-job-of-the-family + :hint "the job name is not a registered job of that family" + :name (d/name name) + :family (d/name family))) + job-def)) + +(defn- get-job-summary + [cfg job-id] + (let [job (jobs/get-job cfg job-id)] + {:id (:id job) + :status (jobs/get-user-status (:status job)) + :name (:name job) + :created-at (:created-at job) + :expires-at (:expires-at job)})) + +(defn- submit-job + "Create the job row with what every user job shares: the queue its + job-def routes the heavy work to (`::jobs/queue-name`, `:binfile` when + the def says nothing), no automatic retry and the expiry of the ledger." + [cfg job-def name params profile-id & {:keys [resource-id]}] + (jobs/submit cfg (cond-> {::jobs/name name + ::jobs/params params + ::jobs/queue (or (::jobs/queue-name job-def) + :binfile) + ::jobs/profile-id profile-id + ::jobs/max-retries 0 + ::jobs/expires-at (ct/plus (ct/now) + (cf/get-jobs-user-ttl))} + (some? resource-id) + (assoc ::jobs/resource-id resource-id)))) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; EXPORT +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(sv/defmethod ::create-export-binfile-job + "Create a durable job that exports a set of files as a `.penpot` + package. + + Its params are `:file-ids`, the set of files to export, and + `:export-type`, how the libraries of those files are handled: what + the job-def declares, and nothing more. There is no default for the + type, because a caller that does not say it would get a package that + is not the one it asked for. + + The job is created, not run: the caller follows it by its id. The + read permission of every file is checked here so a refusal arrives + synchronously, and the handler checks it again before reading." + {::doc/added "2.20" + ::webhooks/event? true + ::sm/params schema:create-export-binfile-job + ::sm/result schema:job-summary} + [cfg {:keys [::rpc/profile-id] :as envelope}] + (let [job-def (resolve-job-def cfg :export :export-binfile) + ;; the RPC layer decoded the params against the job-def schema + ;; before the command ran + params (jobs/validate-params job-def (:params envelope)) + file-ids (:file-ids params)] + + (when (empty? file-ids) + (ex/raise :type :validation + :code :no-files-to-export + :hint "expected at least one file to export")) + + (doseq [file-id file-ids] + (files/check-read-permissions! cfg profile-id file-id)) + + (quotes/check! cfg {::quotes/id ::quotes/export-jobs-per-profile + ::quotes/profile-id profile-id}) + + (let [summary (get-job-summary cfg (submit-job cfg job-def :export-binfile + params profile-id))] + (with-meta summary + {::audit/props {:job-id (:id summary) + :files (count file-ids) + :export-type (d/name (:export-type params))}})))) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; IMPORT +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(defn- assemble-upload + "The package the caller uploaded, assembled into a local file that + belongs to this call. + + The command only takes a chunked upload: a plain file upload would mean + reading a path the caller named, and the server must never read a file + it did not write itself." + [cfg profile-id upload-id] + (db/tx-run! cfg media/assemble-chunks profile-id upload-id)) + +(defn- resolve-version + "The version of the package: what the caller says, or what the header + of the file says." + [path version] + (or version + (case (bfc/parse-file-format path) + :binfile-v3 3 + 1))) + +(sv/defmethod ::create-import-binfile-job + "Create a durable job that imports a `.penpot` package into a project. + + The package arrives as a chunked upload, which the server assembles + itself: the command never reads a path the caller names. The package is + then checked and stored as the resource of the job, so any worker can + consume it by `resource_id`. The job is created, not run, and the caller + follows it by its id." + {::doc/added "2.20" + ::webhooks/event? true + ::sm/params schema:create-import-binfile-job + ::sm/result schema:job-summary + ;; assembling the upload and storing the package happens here, in the + ;; request: the same limit the legacy import command declares + ::climit/id [[:create-import-binfile-job/by-profile ::rpc/profile-id] + [:create-import-binfile-job/global]]} + [cfg {:keys [::rpc/profile-id] :as envelope}] + (let [job-def (resolve-job-def cfg :import :import-binfile) + ;; the RPC layer decoded the params against the create schema + ;; before the command ran + given (:params envelope) + project-id (:project-id given)] + + ;; the edition permission is checked here so a refusal arrives + ;; synchronously and before the upload is assembled; the handler + ;; checks it again before writing + (projects/check-edition-permissions! cfg profile-id project-id) + + (quotes/check! cfg {::quotes/id ::quotes/import-jobs-per-profile + ::quotes/profile-id profile-id}) + + ;; the shape of what the caller sent is checked before the upload + ;; is assembled: the version is known good here (the create schema + ;; rejects one out of range, and the caller may omit it), so the + ;; merged map below only adds what the package says + (jobs/validate-params job-def (assoc given :version (or (:version given) 1))) + + (let [file (assemble-upload cfg profile-id (:upload-id envelope)) + path (:path file)] + + ;; everything that happens once the upload is on disk belongs to + ;; this try: a failure anywhere in it must not leave the temporary + ;; file behind + ;; + ;; the version and the manifest metadata are resolved here, in the + ;; command, and not in the job: only the creation path reads the + ;; package, and the job keeps what the audit will show + (try + (let [version (resolve-version path (:version given)) + manifest (when (= 3 version) + (bf.v3/get-manifest cfg path)) + ;; the manifest may name the tool that produced the package + ;; with either spelling: resolve it once, for the params and + ;; for the audit + referer (or (:referer manifest) (:refer manifest)) + params (cond-> (assoc given :version version) + manifest + (assoc :generated-by (:generated-by manifest) + :referer referer))] + + ;; the params are complete here: the version came from the + ;; caller or from the package, and the manifest filled the audit + (jobs/validate-params job-def params) + + (let [staged (js/put-resource cfg profile-id + {:content (sto/content path) + :filename "package.penpot" + :mtype "application/zip"})] + (try + (let [summary (get-job-summary cfg (submit-job cfg job-def :import-binfile params profile-id + :resource-id (:resource-id staged)))] + (with-meta summary + {::audit/props {:job-id (:id summary) + :generated-by (:generated-by manifest) + :referer referer}})) + + (catch Throwable cause + ;; the job never came to own the package, so it must not be + ;; left for the storage GC to find late + (js/release-resource cfg (:resource-id staged)) + (throw cause))))) + + (finally + ;; the assembled file belongs to this call: the package is + ;; already in storage by the time we get here, or the creation + ;; failed and it is of no use to anyone + (fs/delete path)))))) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; READ +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(def ^:private schema:get-job + [:map {:title "get-job" :closed true} + [:id ::sm/uuid]]) + +(def ^:private schema:job-detail + "The summary of a job plus what only exists once it is over: the result + it produced or, when it failed, its public error." + [:map {:title "job-detail"} + [:id ::sm/uuid] + [:status ::sm/text] + [:name ::sm/text] + [:created-at ::ct/inst] + [:expires-at [:maybe ::ct/inst]] + [:result {:optional true} :any] + [:error {:optional true} :any]]) + +(defn- get-job-detail + "The job the caller owns, as the answer of `get-job`. A job that does + not exist and a job of another profile get the same answer: the caller + cannot tell them apart." + [cfg profile-id job-id] + (let [job (jobs/get-job cfg job-id)] + (when-not (and (some? job) (= profile-id (:profile-id job))) + (ex/raise :type :not-found + :code :job-not-found + :hint "the job does not exist or belongs to another profile" + :job-id job-id)) + (cond-> {:id (:id job) + :status (jobs/get-user-status (:status job)) + :name (:name job) + :created-at (:created-at job) + :expires-at (:expires-at job)} + (some? (:result job)) + (assoc :result (:result job)) + + (some? (:error job)) + (assoc :error (jobs/decode-job-error (:error job)))))) + +(sv/defmethod ::get-job + "Read a job the caller created: its state and, once it is over, what it + produced or why it failed. + + The result of a finished job is the plain JSON the row holds, because a + result has no schema of its own like the params do: a uuid comes back as + the text it is on disk. The progress and the history of the job are not + part of the answer: a client that follows a job takes its progress from + the websocket, and the event log still has no read surface." + {::doc/added "2.20" + ::sm/params schema:get-job + ::sm/result schema:job-detail} + [cfg {:keys [::rpc/profile-id] :as params}] + (get-job-detail cfg profile-id (:id params))) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; CANCEL +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(def ^:private schema:cancel-job + [:map {:title "cancel-job" :closed true} + [:id ::sm/uuid]]) + +(sv/defmethod ::cancel-job + "Cancel a job the caller created, whether it is still waiting or + already running. + + A running job stops at its next beat and rolls its work back; a job + that already finished between the request and the update is answered + with the state it is in, never an error. Like every other command + here, a job of another profile is indistinguishable from one that does + not exist." + {::doc/added "2.20" + ::sm/params schema:cancel-job + ::sm/result schema:job-summary} + [cfg {:keys [::rpc/profile-id] :as params}] + (let [job-id (:id params) + job (jobs/get-job cfg job-id)] + (when-not (and (some? job) (= profile-id (:profile-id job))) + (ex/raise :type :not-found + :code :job-not-found + :hint "the job does not exist or belongs to another profile" + :job-id job-id)) + (jobs/cancel cfg job-id) + ;; the answer names the outcome of this call, not the collapsed + ;; user status: a job the caller just cancelled reads "cancelled" + (assoc (get-job-summary cfg job-id) + :status (:status (jobs/get-job cfg job-id))))) diff --git a/backend/src/app/rpc/management/jobs.clj b/backend/src/app/rpc/management/jobs.clj index ab1561e704..1e71acb101 100644 --- a/backend/src/app/rpc/management/jobs.clj +++ b/backend/src/app/rpc/management/jobs.clj @@ -16,6 +16,7 @@ which mandates a valid shared-key before dispatching to these methods. If the route resolver changes, these methods must be updated accordingly." (:require + [app.common.exceptions :as ex] [app.common.schema :as sm] [app.common.time :as ct] [app.jobs :as jobs] @@ -60,8 +61,8 @@ (def ^:private schema:report-job-progress-params [:map {:title "report-job-progress-params"} ;; The progress schema is the same one the jobs substrate validates: - ;; `current` is mandatory, `total` and `stage` are optional, and no - ;; other key is accepted. + ;; `:stage` is mandatory (a keyword of the worker vocabulary) and + ;; `:counters` is an optional map of `scope -> {:current :total?}`. [:job-id ::sm/uuid] [:progress jobs/schema:progress]]) @@ -78,13 +79,25 @@ ;; A lost race (row already terminal) reports :skip so the worker ;; stops retrying a report that can never land, mirroring claim-job. ;; The forced route propagates database errors: the external worker - ;; must be able to retry a report it was not sure about. - (if (pos? (jobs/heartbeat cfg - :job-id job-id - :progress progress - ::jobs/force? true)) - {:action :run} - {:action :skip})) + ;; must be able to retry a report it was not sure about. The interrupt + ;; the beat raises when the job is no longer active is caught here for + ;; the same reason, and it never reaches the HTTP error mapping: it is + ;; an internal signal, not a failure of the request. + (let [result (ex/try! (jobs/heartbeat cfg + :job-id job-id + :progress progress + ::jobs/force? true))] + (cond + (ex/exception? result) + (if (= :interrupt (:type (ex-data result))) + {:action :skip} + (throw result)) + + (pos? result) + {:action :run} + + :else + {:action :skip}))) ;; ---- RPC METHOD: COMPLETE-JOB diff --git a/backend/src/app/rpc/quotes.clj b/backend/src/app/rpc/quotes.clj index 2cbe336a8d..049728c3a0 100644 --- a/backend/src/app/rpc/quotes.clj +++ b/backend/src/app/rpc/quotes.clj @@ -7,6 +7,7 @@ (ns app.rpc.quotes "Penpot resource usage quotes." (:require + [app.common.data :as d] [app.common.exceptions :as ex] [app.common.logging :as l] [app.common.schema :as sm] @@ -617,6 +618,61 @@ team-id team-id team-id team-id team-id team-id]) (generic-check!))) +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; QUOTE: BINFILES JOBS PER PROFILE +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(def ^:private schema:jobs-per-profile + [:map [::profile-id ::sm/uuid]]) + +(def ^:private valid-jobs-per-profile-quote? + (sm/lazy-validator schema:jobs-per-profile)) + +;; Only the jobs that are still on their way count: a terminal job is not +;; pending work anymore, and the jobs GC removes it. The tenant filters +;; like every other query of the substrate: instances share one database +;; and one profile can have work in more than one of them. +(def ^:private + sql:get-jobs-per-profile + "SELECT count(*) AS total + FROM job + WHERE tenant = ? + AND profile_id = ? + AND status IN ('new', 'scheduled', 'running', 'retry') + AND name = ANY(?)") + +(defn- names-of-family + "The registered names of a family of jobs. The quote counts what the + registry knows, so a new job of the family is covered without touching + this file." + [cfg family] + (->> (jobs/get-defs cfg) + (vals) + (filter #(= family (::jobs/family %))) + (map #(d/name (::jobs/name %))) + (vec))) + +(defn- check-jobs-quote + [cfg family config-key] + (let [{:keys [::db/conn ::profile-id ::target]} cfg] + (assert (valid-jobs-per-profile-quote? cfg) "invalid quote parameters") + (-> cfg + (assoc ::default (cf/get config-key Integer/MAX_VALUE)) + (assoc ::quote-sql [sql:get-quotes-1 target profile-id]) + (assoc ::count-sql [sql:get-jobs-per-profile + (cf/get :tenant) + profile-id + (db/create-array conn "text" (names-of-family cfg family))]) + (generic-check!)))) + +(defmethod check-quote ::export-jobs-per-profile + [cfg] + (check-jobs-quote cfg :export :quotes-export-jobs-per-profile)) + +(defmethod check-quote ::import-jobs-per-profile + [cfg] + (check-jobs-quote cfg :import :quotes-import-jobs-per-profile)) + ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; QUOTE: DEFAULT ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; diff --git a/backend/src/app/storage.clj b/backend/src/app/storage.clj index 6c7a87762b..f73fd3557e 100644 --- a/backend/src/app/storage.clj +++ b/backend/src/app/storage.clj @@ -248,11 +248,15 @@ backend' (impl/resolve-backend storage backend) bucket (:bucket mdata) + ;; `job-resource` objects belong to one profile, so they are + ;; never shared: a dedup hit would hand the artifact of one + ;; profile to the job of another. dedupable? (and (::deduplicate? params) (:hash mdata) (some? bucket) (not= tempfile-bucket bucket) - (not= upload-session-bucket bucket)) + (not= upload-session-bucket bucket) + (not= job-resource-bucket bucket)) hit (when dedupable? (get-database-object-by-hash pool backend bucket (:hash mdata)))] diff --git a/backend/src/app/storage/schema.clj b/backend/src/app/storage/schema.clj index 17e858b972..50a354d12f 100644 --- a/backend/src/app/storage/schema.clj +++ b/backend/src/app/storage/schema.clj @@ -53,7 +53,8 @@ "organization" #{:organization-id} tempfile-bucket #{} upload-session-bucket #{} - job-resource-bucket #{} + ;; read back only by the profile that owns them + job-resource-bucket #{:profile-id} "file-data" #{:file-id :id} "file-data-fragment" #{} "file-change" #{}}) diff --git a/backend/src/app/tasks/export_binfile.clj b/backend/src/app/tasks/export_binfile.clj new file mode 100644 index 0000000000..eaa1ba0f55 --- /dev/null +++ b/backend/src/app/tasks/export_binfile.clj @@ -0,0 +1,94 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns app.tasks.export-binfile + "The `:export-binfile` job: it produces the `.penpot` artifact of the + set of files frozen in its params." + (:require + [app.binfile.jobs :as bfj] + [app.common.schema :as sm] + [app.db :as db] + [app.jobs :as jobs] + [app.jobs.storage :as js] + [app.metrics :as mtx] + [app.rpc.commands.files :as files] + [app.storage :as sto] + [app.storage.tmp :as tmp] + [datoteka.fs :as fs] + [integrant.core :as ig])) + +(def schema:params + "Business params of an export: the files to export, frozen when the job + was created, and how their libraries are handled." + [:map {:title "export-binfile-params" :closed true} + [:file-ids [::sm/set ::sm/uuid]] + [:export-type [::sm/one-of #{:include-libraries :merge-libraries + :detach-libraries :link-later}]]]) + +(def ^:private artifact + "What the user sees of the package an export produces. A single name + describes a set of files, so it does not come from any of them." + {:filename "export.penpot" + :mtype "application/zip"}) + +(def schema:result + "What the job stores when the export succeeds: where the artifact can be + downloaded from and what it is." + [:map {:title "export-binfile-result" :closed true} + [:resource-uri ::sm/text] + [:filename ::sm/text] + [:mtype ::sm/text] + [:size ::sm/int]]) + +(def ^:private check-result + (sm/check-fn schema:result + :hint "invalid result of an export job" + :type :validation + :code :invalid-result)) + +(defn execute-export + "Plain handler, importable and testable without integrant. + + The read permission is checked again here, and not only when the job was + created: it can be revoked while the job waits in its queue. The artifact + is stored as the resource of the job, and the completion envelope carries + its descriptor as the business result and its id for the runner to + associate." + [cfg context params] + (let [profile-id (:profile-id context) + file-ids (:file-ids params)] + + (doseq [file-id file-ids] + (files/check-read-permissions! cfg profile-id file-id)) + + (let [output (tmp/tempfile* :suffix ".penpot")] + (try + (bfj/export-files cfg context {:ids file-ids + :export-type (:export-type params) + :output output}) + (let [resource (js/put-resource cfg profile-id + (assoc artifact :content (sto/content output)))] + {:result (check-result (dissoc resource :resource-id)) + :resource-id (:resource-id resource)}) + (finally + (fs/delete output)))))) + +(defmethod ig/assert-key ::job-def + [_ params] + (assert (db/pool? (::db/pool params)) "expected a valid database pool") + (assert (sto/valid-storage? (::sto/storage params)) "expected valid storage to be provided") + (assert (mtx/metrics? (::mtx/metrics params)) "expected valid metrics")) + +(defmethod ig/init-key ::job-def + [_ cfg] + {::jobs/name :export-binfile + ::jobs/family :export + ::jobs/resource-role :output + ::jobs/schema schema:params + ::jobs/handler (fn [context params] + (execute-export cfg context params)) + ::jobs/decoder (sm/decoder schema:params sm/json-transformer) + ::jobs/validator (sm/validator schema:params)}) diff --git a/backend/src/app/tasks/import_binfile.clj b/backend/src/app/tasks/import_binfile.clj new file mode 100644 index 0000000000..5dc286db08 --- /dev/null +++ b/backend/src/app/tasks/import_binfile.clj @@ -0,0 +1,126 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns app.tasks.import-binfile + "The `:import-binfile` job: it imports the package the job owns as a + resource into the project frozen in its params." + (:require + [app.binfile.jobs :as bfj] + [app.common.exceptions :as ex] + [app.common.schema :as sm] + [app.db :as db] + [app.jobs :as jobs] + [app.jobs.storage :as js] + [app.metrics :as mtx] + [app.rpc.commands.projects :as projects] + [app.rpc.commands.teams :as teams] + [app.storage :as sto] + [datoteka.fs :as fs] + [integrant.core :as ig])) + +(def schema:params + "Business params of an import: where the package goes, its name, the + format version and the manifest metadata kept for audit." + [:map {:title "import-binfile-params" :closed true} + [:project-id ::sm/uuid] + [:name [:or [:string {:max 250}] + [:map-of ::sm/uuid [:string {:max 250}]]]] + [:version [:enum 1 3]] + [:generated-by {:optional true} [:maybe ::sm/text]] + [:referer {:optional true} [:maybe ::sm/text]]]) + +(def schema:create-params + "The business params of an import as the creation command receives + them: where the package goes and its name. The version is optional + because the command reads it from the package header when the caller + does not say it, and the manifest metadata kept for audit is filled by + the command, never sent by the caller." + [:map {:title "import-binfile-create-params" :closed true} + [:project-id ::sm/uuid] + [:name [:or [:string {:max 250}] + [:map-of ::sm/uuid [:string {:max 250}]]]] + [:version {:optional true} [:enum 1 3]]]) + +(def schema:result + "What the job stores when the import succeeds: the files it created and + how the libraries of the package were resolved." + [:map {:title "import-binfile-result" :closed true} + [:file-ids [:vector ::sm/uuid]] + [:resolution {:optional true} :map] + [:name [:or [:string {:max 250}] + [:map-of ::sm/uuid [:string {:max 250}]]]] + [:version [:enum 1 3]]]) + +(def ^:private check-result + (sm/check-fn schema:result + :hint "invalid result of an import job" + :type :validation + :code :invalid-result)) + +(defn execute-import + "Plain handler, importable and testable without integrant. + + The edition permission is checked again here, and not only when the job + was created: it can be revoked while the job waits in its queue. The + core runs inside a transaction, so a cancellation detected between units + rolls the whole import back. + + The package is the input of the job: it is released and its temporary + copy deleted once the job is terminal, whatever the outcome. A crash is + covered by the storage GC." + [cfg context params] + (let [profile-id (:profile-id context) + project-id (:project-id params) + input (js/load-input cfg context)] + + (try + (projects/check-edition-permissions! cfg profile-id project-id) + + (let [team (or (teams/get-team cfg + :profile-id profile-id + :project-id project-id) + (ex/raise :type :not-found + :code :project-not-found + :hint "the destination project is not available" + :profile-id profile-id + :project-id project-id)) + result (db/tx-run! cfg + (fn [tx-cfg] + ;; the core joins this transaction: it + ;; is the one that decides whether the + ;; whole import is kept or rolled back + (bfj/import-files tx-cfg context + {:profile-id profile-id + :project-id project-id + :team team + :name (:name params) + :input input + :version (:version params)})))] + + (check-result (assoc result + :name (:name params) + :version (:version params)))) + + (finally + (fs/delete input) + (js/release-input cfg context))))) + +(defmethod ig/assert-key ::job-def + [_ params] + (assert (db/pool? (::db/pool params)) "expected a valid database pool") + (assert (sto/valid-storage? (::sto/storage params)) "expected valid storage to be provided") + (assert (mtx/metrics? (::mtx/metrics params)) "expected valid metrics")) + +(defmethod ig/init-key ::job-def + [_ cfg] + {::jobs/name :import-binfile + ::jobs/family :import + ::jobs/resource-role :input + ::jobs/schema schema:params + ::jobs/handler (fn [context params] + (execute-import cfg context params)) + ::jobs/decoder (sm/decoder schema:params sm/json-transformer) + ::jobs/validator (sm/validator schema:params)}) diff --git a/backend/src/app/util/events.clj b/backend/src/app/util/events.clj index ffbd96a47c..1c82380943 100644 --- a/backend/src/app/util/events.clj +++ b/backend/src/app/util/events.clj @@ -5,9 +5,15 @@ ;; Copyright (c) KALEIDOS SUBSIDIARY SL (ns app.util.events - "A generic asynchronous events notifications subsystem; used mainly - for mark event points in functions and be able to attach listeners - to them. Mainly used in http.sse for progress reporting." + "Event points of a run: the core marks what it does with `tap`, and + whoever runs it decides where those marks go. + + The sink bound for the run decides how each event travels: a function + is called inline, on the same thread, while a channel receives the + event for a listener of its own. The jobs adapter binds a function + that turns taps into heartbeats, so an `:interrupt` it raises aborts + the run right there; the SSE stream binds a channel instead, and its + listener writes what arrives to the response." (:refer-clojure :exclude [run!]) (:require [app.common.exceptions :as ex] @@ -15,19 +21,29 @@ [promesa.exec :as px] [promesa.exec.csp :as sp])) -(def ^:dynamic *channel* nil) +(def ^:dynamic *sink* nil) (defn tap - ([type data] - (when-let [channel *channel*] - (sp/put! channel [type data]) - nil)) - ([channel type data] - (when channel - (sp/put! channel [type data]) - nil))) + "Mark an event point of the run in course with `[type data]`. + + With no sink bound nothing happens, so untracked runs pay a single + nil check per event point. A function sink is called inline and any + exception it raises propagates to the producer: that is how an + `:interrupt` from a heartbeat stops the run at its next event point. + A channel sink receives the event for its listener instead." + [type data] + (when-let [sink *sink*] + (if (fn? sink) + (sink [type data]) + (sp/put! sink [type data])) + nil)) (defn spawn-listener + "Consume the events of `channel` on a thread of its own, until it + closes. + + `on-close` runs once the listener stops. A failure of `on-event` is + logged and stops the listener." [channel on-event on-close] (assert (sp/chan? channel) "expected active events channel") @@ -39,22 +55,14 @@ (let [result (ex/try! (on-event event))] (if (ex/exception? result) (do - (l/wrn :hint "unexpected exception" :cause result) + (l/err :hint "unexpected exception on calling-on-event of spawn-listener" + :cause result) (sp/close! channel)) (recur))))) (finally - (on-close))))) - -(defn run-with! - "A high-level facility for to run a function in context of event - emiter." - [f on-event] - - (binding [*channel* (sp/chan :buf 32)] - (let [listener (spawn-listener *channel* on-event (constantly nil))] - (try - (f) - (finally - (sp/close! *channel*) - (px/await! listener)))))) + (try + (on-close) + (catch Exception cause + (l/err :hint "unexpected exception on calling on-close of spawn-listener" + :cause cause))))))) diff --git a/backend/src/app/worker/runner.clj b/backend/src/app/worker/runner.clj index f5506f0450..2202722c0d 100644 --- a/backend/src/app/worker/runner.clj +++ b/backend/src/app/worker/runner.clj @@ -163,9 +163,22 @@ (throw cause)) (catch Throwable cause (let [edata (ex-data cause)] - (if (and (< (:retry-num job) - (:max-retries job)) - (= ::wrk/retry (:type edata))) + (cond + ;; the interrupt is the signal that the job is no longer active: + ;; a normal procedure, so it is not reported as an error and it + ;; never fails or retries a row that is already terminal + (= :interrupt (:type edata)) + (binding [l/*context* (-> (cf/logging-context) + (assoc :params job))] + (l/inf :hint "interrupted" + :tenant tenant + :job-id (str (:job-id edata)) + :status (:status edata)) + {:status "interrupted"}) + + (and (< (:retry-num job) + (:max-retries job)) + (= ::wrk/retry (:type edata))) (cond-> {:status "retry" :error cause} (ct/duration? (:delay edata)) (assoc :delay-ms (inst-ms (:delay edata))) @@ -174,6 +187,8 @@ (= ::wrk/noop (:strategy edata)) (assoc :inc-by 0)) + + :else (do (l/err :hint "unhandled exception on job" ::l/context (assoc (cf/logging-context) :params job) @@ -288,9 +303,12 @@ (process-result [{:keys [status] :as result}] (ex/try! (case status - "retry" (handle-job-retry result) - "failed" (handle-job-failure result) - "completed" (handle-job-completion result) + "retry" (handle-job-retry result) + "failed" (handle-job-failure result) + "completed" (handle-job-completion result) + ;; the job is no longer active: there is nothing to write, + ;; the row is already terminal and whoever stopped it did + "interrupted" nil (throw (IllegalArgumentException. (str "invalid status received: '" status "'")))))) diff --git a/backend/test/backend_tests/binfile_jobs_test.clj b/backend/test/backend_tests/binfile_jobs_test.clj new file mode 100644 index 0000000000..7b4619c96c --- /dev/null +++ b/backend/test/backend_tests/binfile_jobs_test.clj @@ -0,0 +1,403 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns backend-tests.binfile-jobs-test + "The binfile work of a job: the shared adapter that runs it with its + progress reported as job events, and the two job-defs that expose it." + (:require + [app.binfile.common :as bfc] + [app.binfile.jobs :as bfj] + [app.binfile.v3 :as v3] + [app.common.time :as ct] + [app.common.uuid :as uuid] + [app.config :as cf] + [app.db :as db] + [app.jobs :as jobs] + [app.jobs.storage :as js] + [app.rpc.commands.teams :as teams] + [app.storage :as sto] + [app.storage.tmp :as tmp] + [app.util.events :as events] + [backend-tests.helpers :as th] + [clojure.java.io :as jio] + [clojure.test :as t] + [datoteka.fs :as fs] + [datoteka.io :as io])) + +(t/use-fixtures :once th/state-init) +(t/use-fixtures :each th/database-reset) + +(def ^:private fixture + "A real penpot export, the same fixture the binfile tests read." + "backend_tests/test_files/svg-attrs-camel-case.penpot") + +(defn- fixture-path + [] + (-> fixture io/resource jio/file)) + +(defn- import-fixture! + "A file in the default project of the profile, imported from the + fixture through the core." + [profile] + (-> th/*system* + (assoc ::bfc/project-id (:default-project-id profile)) + (assoc ::bfc/profile-id (:id profile)) + (assoc ::bfc/team-id (:default-team-id profile)) + (assoc ::bfc/input (fixture-path)) + (v3/import-files!))) + +(defn- make-job! + "A job row as the substrate writes it: running, owned by a profile and + holding a resource when the test needs one." + [profile-id resource-id] + (let [id (uuid/next)] + (th/db-insert! :job {:id id + :name "export-binfile" + :tenant (cf/get :tenant) + :queue "binfile" + :params (db/json {}) + :priority 100 + :max-retries 0 + :retry-num 0 + :status "running" + :profile-id profile-id + :resource-id resource-id + :scheduled-at (ct/now) + :created-at (ct/now) + :modified-at (ct/now)}) + id)) + +(defn- job-cfg + "The system cfg as the runner hands it to a handler. The job id is NOT + bound here: the adapter takes it from the context, because the progress + bridge runs on a thread of its own." + [] + th/*system*) + +(defn- job-context + [job-id] + (jobs/make-context (jobs/get-job th/*system* job-id))) + +(defn- progress-events + "The progress events of a job, oldest first, with decoded payloads." + [job-id] + (->> (th/db-exec! ["SELECT payload FROM job_event + WHERE job_id = ? AND kind = 'progress' + ORDER BY created_at ASC, id ASC" + job-id]) + (mapv #(jobs/decode-progress (:payload %))))) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; ADAPTER +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(t/deftest export-through-the-adapter-produces-the-artifact + (let [profile (th/create-profile* 1) + file-id (first (:file-ids (import-fixture! profile))) + job-id (make-job! (:id profile) nil) + output (tmp/tempfile* :suffix ".penpot")] + + (bfj/export-files (job-cfg) (job-context job-id) + {:ids #{file-id} + :export-type :detach-libraries + :output output}) + + (t/testing "the artifact is a penpot file with its manifest" + (t/is (fs/exists? output)) + (t/is (= [file-id] (mapv :id (:files (v3/get-manifest th/*system* output)))))) + + (t/testing "the progress of the core became milestones of the job" + (let [events (progress-events job-id)] + (t/is (pos? (count events))) + (t/is (every? keyword? (map :stage events))) + (t/testing "the first one is the file in course, with its position" + (let [event (first events)] + (t/is (= :files (:stage event))) + (t/is (= {:current 1 :total 1} (get-in event [:counters :files]))))) + (t/testing "and nothing else: the event is the stage and the counters" + (t/is (every? #(<= (count (keys %)) 2) events))))))) + +(t/deftest import-through-the-adapter-imports-the-package + (let [profile (th/create-profile* 1) + team (teams/get-team th/*system* + :profile-id (:id profile) + :project-id (:default-project-id profile)) + staged (js/put-resource th/*system* (:id profile) + {:content (sto/content (fixture-path)) + :filename "package.penpot" + :mtype "application/zip"}) + job-id (make-job! (:id profile) (:resource-id staged)) + context (job-context job-id) + result (bfj/import-files (job-cfg) context + {:profile-id (:id profile) + :project-id (:default-project-id profile) + :team team + :name "imported" + :input (js/load-input th/*system* context) + :version 3})] + + (t/testing "the package is imported in the destination project" + (t/is (= 1 (count (:file-ids result))))) + + (t/testing "the job reported its progress" + (t/is (pos? (count (progress-events job-id))))))) + +(t/deftest the-adapter-composes-a-milestone-per-tap + (let [profile (th/create-profile* 1) + file-id (first (:file-ids (import-fixture! profile))) + job-id (make-job! (:id profile) nil) + reports (atom [])] + + ;; the substrate throttles the durable events, so the whole stream of + ;; milestones is only visible with the writes intercepted + (with-redefs [jobs/heartbeat (fn [_cfg & {:as options}] + (swap! reports conj (:progress options)) + 1)] + (bfj/export-files (job-cfg) (job-context job-id) + {:ids #{file-id} + :export-type :detach-libraries + :output (tmp/tempfile* :suffix ".penpot")})) + + (t/testing "the file opens the run with its position" + (let [event (first @reports)] + (t/is (= :files (:stage event))) + (t/is (= {:current 1 :total 1} (get-in event [:counters :files]))))) + + (t/testing "the stages inside the file carry it as context, with their own counter" + (let [event (first (filter #(= :pages (:stage %)) @reports))] + (t/is (= {:current 1 :total 1} (get-in event [:counters :files]))) + (t/is (pos? (get-in event [:counters :pages :current]))) + (t/is (pos? (get-in event [:counters :pages :total]))))) + + (t/testing "a stage outside a file carries no file context" + (let [events (filter #(= :storage-objects (:stage %)) @reports)] + (t/is (pos? (count events))) + (t/is (every? #(nil? (get-in % [:counters :files])) events)))) + + (t/testing "and every milestone is a stage and its counters, and nothing else" + (t/is (every? #(<= (count (keys %)) 2) @reports))))) + +(t/deftest a-tap-carries-the-outer-counters-in-course + (let [profile (th/create-profile* 1) + file-id (first (:file-ids (import-fixture! profile))) + job-id (make-job! (:id profile) nil) + taps (atom [])] + + ;; the taps themselves, before the adapter turns them into events + (with-redefs [events/tap (fn [_type data] (swap! taps conj data))] + (bfj/export-files (job-cfg) (job-context job-id) + {:ids #{file-id} + :export-type :detach-libraries + :output (tmp/tempfile* :suffix ".penpot")})) + + (t/testing "the tap that opens the file carries the outer counters" + (let [tap (first (filter #(= :file (:section %)) @taps))] + (t/is (= {:files {:current 1 :total 1}} (:outer-counters tap))))) + + (t/testing "every tap inside the file carries them as context" + (let [inner (filter #(some? (:file-id %)) @taps)] + (t/is (pos? (count inner))) + (t/is (every? #(= {:files {:current 1 :total 1}} (:outer-counters %)) inner)))) + + (t/testing "a tap outside a file carries no outer counters" + (let [outside (filter #(= :storage-object (:section %)) @taps)] + (t/is (pos? (count outside))) + (t/is (every? #(nil? (:outer-counters %)) outside)))))) + +(t/deftest a-milestone-merges-outer-counters-with-its-own + (let [milestone @#'bfj/milestone] + (t/testing "outer scopes survive next to the counter of the stage" + (t/is (= {:stage :pages + :counters {:teams {:current 1 :total 1} + :projects {:current 2 :total 2} + :files {:current 1 :total 4} + :pages {:current 2 :total 5}}} + (milestone {:section :page + :current 2 :total 5 + :outer-counters {:teams {:current 1 :total 1} + :projects {:current 2 :total 2} + :files {:current 1 :total 4}}})))) + (t/testing "a file tap needs no own counter, the outer one is enough" + (t/is (= {:stage :files + :counters {:files {:current 1 :total 2}}} + (milestone {:section :file + :outer-counters {:files {:current 1 :total 2}}})))))) + +(t/deftest entering-a-file-keeps-the-counters-of-the-outer-scopes + (let [cfg {::v3/outer-counters {:teams {:current 1 :total 1}}} + cfg' (@#'v3/with-outer-counter cfg :files {:current 2 :total 4})] + (t/is (= {:teams {:current 1 :total 1} + :files {:current 2 :total 4}} + (::v3/outer-counters cfg'))))) + +(t/deftest the-import-adapter-composes-the-same-milestones + (let [profile (th/create-profile* 1) + team (teams/get-team th/*system* + :profile-id (:id profile) + :project-id (:default-project-id profile)) + staged (js/put-resource th/*system* (:id profile) + {:content (sto/content (fixture-path)) + :filename "package.penpot" + :mtype "application/zip"}) + job-id (make-job! (:id profile) (:resource-id staged)) + context (job-context job-id) + reports (atom [])] + + (with-redefs [jobs/heartbeat (fn [_cfg & {:as options}] + (swap! reports conj (:progress options)) + 1)] + (bfj/import-files (job-cfg) context + {:profile-id (:id profile) + :project-id (:default-project-id profile) + :team team + :name "imported" + :input (js/load-input th/*system* context) + :version 3})) + + (t/testing "the run opens with the manifest, which has no units to count" + (let [event (first @reports)] + (t/is (= :manifest (:stage event))) + (t/is (nil? (:counters event))))) + + (t/testing "the file carries its position" + (let [event (first (filter #(= :files (:stage %)) @reports))] + (t/is (= {:current 1 :total 1} (get-in event [:counters :files]))))) + + (t/testing "a stage inside the file carries it as context, with its own counter" + (let [event (first (filter #(= :pages (:stage %)) @reports))] + (t/is (= {:current 1 :total 1} (get-in event [:counters :files]))) + (t/is (pos? (get-in event [:counters :pages :current]))) + (t/is (pos? (get-in event [:counters :pages :total]))))) + + (t/testing "and every milestone is a stage and its counters, and nothing else" + (t/is (every? #(<= (count (keys %)) 2) @reports))))) + +(t/deftest a-job-that-is-no-longer-active-stops-the-work + (let [profile (th/create-profile* 1) + file-id (first (:file-ids (import-fixture! profile))) + job-id (make-job! (:id profile) nil) + context (job-context job-id)] + + (t/testing "a running job is active" + (t/is (nil? (bfj/check-active th/*system* context)))) + + (t/testing "a cancelled job is not" + (th/db-update! :job {:status "cancelled"} {:id job-id}) + (let [cause (try (bfj/check-active th/*system* context) nil + (catch Throwable raised raised))] + (t/is (= :interrupt (th/ex-type cause))) + (t/is (= "cancelled" (:status (ex-data cause)))))) + + (t/testing "so the export stops at its next progress point" + (let [output (tmp/tempfile* :suffix ".penpot") + cause (try + (bfj/export-files (job-cfg) context + {:ids #{file-id} + :export-type :detach-libraries + :output output}) + nil + (catch Throwable raised raised))] + (t/is (= :interrupt (th/ex-type cause))) + (t/is (zero? (count (progress-events job-id)))))) + + (t/testing "a job that already completed is not active either" + (th/db-update! :job {:status "completed"} {:id job-id}) + (let [cause (try (bfj/check-active th/*system* context) nil + (catch Throwable raised raised))] + (t/is (= :interrupt (th/ex-type cause))))) + + (t/testing "and neither is a job row that is gone" + (th/db-force-delete :job {:id job-id}) + (let [cause (try (bfj/check-active th/*system* context) nil + (catch Throwable raised raised))] + (t/is (= :interrupt (th/ex-type cause))) + (t/is (= "gone" (:status (ex-data cause)))))))) + +(t/deftest an-interrupted-beat-stops-the-core-at-its-tap + (let [steps (atom []) + context {:id (uuid/next)} + cause (ex-info "the job is no longer active" + {:type :interrupt :code :job-interrupted})] + (with-redefs [jobs/heartbeat (fn [& _] (throw cause))] + (let [raised (try (@#'bfj/with-progress th/*system* context + (fn [] + (swap! steps conj :before) + ;; the sink calls the beat + ;; inline, so its interrupt + ;; aborts the run here + (events/tap :progress {:section :file}) + (swap! steps conj :after))) + nil + (catch Throwable stopped stopped))] + (t/is (= :interrupt (:type (ex-data raised)))) + (t/is (= :job-interrupted (:code (ex-data raised)))) + (t/is (= [:before] @steps) + "the core stopped at its next tap, the step after never ran"))))) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; JOB-DEFS +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(t/deftest the-binfile-jobs-follow-the-substrate-contract + (let [defs (::jobs/defs th/*system*)] + + (t/testing "the export job produces the resource of a user job" + (let [def (jobs/get-job-def defs :export-binfile)] + (t/is (= :export (::jobs/family def))) + (t/is (= :output (::jobs/resource-role def))) + (t/is (fn? (::jobs/handler def))))) + + (t/testing "the import job consumes the resource of the job" + (let [def (jobs/get-job-def defs :import-binfile)] + (t/is (= :import (::jobs/family def))) + (t/is (= :input (::jobs/resource-role def))) + (t/is (fn? (::jobs/handler def))))) + + (t/testing "the params are a closed contract" + (let [def (jobs/get-job-def defs :export-binfile) + file-id (uuid/next)] + (t/is (true? ((::jobs/validator def) + {:file-ids #{file-id} :export-type :detach-libraries}))) + (t/testing "a job id never travels in params" + (t/is (false? ((::jobs/validator def) + {:file-ids #{file-id} :export-type :detach-libraries + :job-id (uuid/next)})))))))) + +(t/deftest the-export-handler-runs-through-the-registry + (let [profile (th/create-profile* 1) + file-id (first (:file-ids (import-fixture! profile))) + job-id (make-job! (:id profile) nil) + result (jobs/invoke (-> th/*system* + (assoc ::jobs/name :export-binfile) + (assoc ::jobs/params {:file-ids #{file-id} + :export-type :detach-libraries}) + (assoc ::jobs/context (job-context job-id))))] + + (t/testing "the handler answers with the completion envelope of the job" + (t/is (uuid? (:resource-id result))) + (t/is (some? (get-in result [:result :resource-uri])))) + + (t/testing "the handler reported progress without knowing the job id" + (t/is (pos? (count (progress-events job-id))))))) + +(t/deftest the-import-handler-runs-through-the-registry + (let [profile (th/create-profile* 1) + staged (js/put-resource th/*system* (:id profile) + {:content (sto/content (fixture-path)) + :filename "package.penpot" + :mtype "application/zip"}) + job-id (make-job! (:id profile) (:resource-id staged)) + result (jobs/invoke (-> th/*system* + (assoc ::jobs/name :import-binfile) + (assoc ::jobs/params {:project-id (:default-project-id profile) + :name "imported" + :version 3}) + (assoc ::jobs/context (job-context job-id))))] + + (t/testing "the package of the job was imported" + (t/is (pos? (count (:file-ids result))))) + (t/testing "the handler reported progress without knowing the job id" + (t/is (pos? (count (progress-events job-id))))))) diff --git a/backend/test/backend_tests/config_test.clj b/backend/test/backend_tests/config_test.clj index 863be00a5e..af06598962 100644 --- a/backend/test/backend_tests/config_test.clj +++ b/backend/test/backend_tests/config_test.clj @@ -6,6 +6,7 @@ (ns backend-tests.config-test (:require + [app.common.time :as ct] [app.config :as cf] [clojure.test :as t])) @@ -51,11 +52,26 @@ (t/testing "schema decodes all runner parallelism keys (env strings to int)" (let [decoded (cf/decode-config {:worker-default-parallelism "4" :worker-webhook-parallelism "2" - :worker-cron-parallelism "3"})] + :worker-cron-parallelism "3" + :worker-binfile-parallelism "5"})] (t/is (= 4 (:worker-default-parallelism decoded))) (t/is (= 2 (:worker-webhook-parallelism decoded))) - (t/is (= 3 (:worker-cron-parallelism decoded))))) + (t/is (= 3 (:worker-cron-parallelism decoded))) + (t/is (= 5 (:worker-binfile-parallelism decoded))))) (t/testing "lookups resolve configured values (unnamespaced keys, as main uses)" (with-redefs [cf/config (assoc cf/config :worker-default-parallelism 4)] (t/is (= 4 (cf/get :worker-default-parallelism 1)))))) + +(t/deftest jobs-user-ttl + (t/testing "the env value is decoded as a duration" + (let [decoded (cf/decode-config {:jobs-user-ttl "72h"})] + (t/is (= (ct/duration {:days 3}) (:jobs-user-ttl decoded))))) + + (t/testing "the getter resolves the configured value" + (with-redefs [cf/config (assoc cf/config :jobs-user-ttl (ct/duration {:days 1}))] + (t/is (= (ct/duration {:days 1}) (cf/get-jobs-user-ttl))))) + + (t/testing "and falls back to a week" + (with-redefs [cf/config (dissoc cf/config :jobs-user-ttl)] + (t/is (= (ct/duration {:days 7}) (cf/get-jobs-user-ttl)))))) diff --git a/backend/test/backend_tests/helpers.clj b/backend/test/backend_tests/helpers.clj index d89250eee3..d39e2a26a6 100644 --- a/backend/test/backend_tests/helpers.clj +++ b/backend/test/backend_tests/helpers.clj @@ -102,6 +102,22 @@ (fs/create-dir "/tmp/penpot") (next))))) +(defn- remove-job-runners + "Drops every `[ :app.worker/runner]` from the system. + + The tests drive the runner loop by hand (`wdisp/run-batch` plus + `run-worker-loop` over a system they build themselves), so a live runner + of any queue races them: it takes the job the test just dispatched and + completes it behind the test's back. Matching the key by shape, instead + of listing the known queues by hand, keeps a new queue from silently + bringing that race back (which is what happened to the `binfile` runner + this key list was missing)." + [system] + (apply dissoc system + (filter #(and (vector? %) + (= :app.worker/runner (second %))) + (keys system)))) + (defn init-system [next] (let [templates [{:id "test" @@ -132,10 +148,8 @@ :app.loggers.mattermost/reporter :app.loggers.database/reporter :app.worker/cron - :app.worker/dispatcher - [:app.main/default :app.worker/runner] - [:app.main/webhook :app.worker/runner] - [:app.main/cron :app.worker/runner])) + :app.worker/dispatcher) + (remove-job-runners)) _ (ig/load-namespaces system) system (-> (ig/expand system) (ig/init))] (try diff --git a/backend/test/backend_tests/http_assets_test.clj b/backend/test/backend_tests/http_assets_test.clj index a5e60e9965..1f6c473a51 100644 --- a/backend/test/backend_tests/http_assets_test.clj +++ b/backend/test/backend_tests/http_assets_test.clj @@ -887,6 +887,48 @@ response (assets/objects-handler cfg request)] (t/is (= 204 (::yres/status response))))) +;; ---------------------------------------------------------------- +;; Tests: objects-handler — job-resource bucket ownership +;; ---------------------------------------------------------------- + +(t/deftest objects-handler-job-resource-owner-can-access + (let [storage (-> (:app.storage/storage th/*system*) + (configure-storage-backend)) + cfg (make-handler-cfg storage) + owner (th/create-profile* 1) + object (create-storage-object! storage "job-resource" "artifact" {:profile-id (:id owner)}) + request {:path-params {:id (str (:id object))} + ::session/profile-id (:id owner)} + response (assets/objects-handler cfg request)] + (t/is (= 204 (::yres/status response))))) + +(t/deftest objects-handler-job-resource-non-owner-gets-404 + ;; The artifact of a job belongs to the profile that owns the job. + (let [storage (-> (:app.storage/storage th/*system*) + (configure-storage-backend)) + cfg (make-handler-cfg storage) + owner (th/create-profile* 1) + stranger (th/create-profile* 2) + object (create-storage-object! storage "job-resource" "artifact" {:profile-id (:id owner)}) + request {:path-params {:id (str (:id object))} + ::session/profile-id (:id stranger)} + response (assets/objects-handler cfg request)] + (t/is (= 404 (::yres/status response))))) + +(t/deftest objects-handler-job-resource-released-object-gets-404 + ;; Once the job is gone and the GC has reclaimed the artifact, the route + ;; has nothing to serve. + (let [storage (-> (:app.storage/storage th/*system*) + (configure-storage-backend)) + cfg (make-handler-cfg storage) + owner (th/create-profile* 1) + object (create-storage-object! storage "job-resource" "artifact" {:profile-id (:id owner)}) + request {:path-params {:id (str (:id object))} + ::session/profile-id (:id owner)}] + (t/is (= 204 (::yres/status (assets/objects-handler cfg request)))) + (sto/del-object! storage (:id object)) + (t/is (= 404 (::yres/status (assets/objects-handler cfg request)))))) + ;; ---------------------------------------------------------------- ;; Tests: asset request metrics ;; ---------------------------------------------------------------- diff --git a/backend/test/backend_tests/jobs_metrics_test.clj b/backend/test/backend_tests/jobs_metrics_test.clj index af43e856f3..45f4b59725 100644 --- a/backend/test/backend_tests/jobs_metrics_test.clj +++ b/backend/test/backend_tests/jobs_metrics_test.clj @@ -298,10 +298,29 @@ (fn [{:keys [::db/conn]}] (jobs/heartbeat (assoc cfg ::db/conn conn) :job-id job-id - :progress {:current 1} + :progress {:stage :work} ::jobs/force? true))) (t/testing "the report is stored and counted, the caller rollback drops neither" (t/is (= 1 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job_event WHERE job_id = ? AND kind = 'progress'" job-id])))) (t/is (= (inc before) (counter-value metrics :jobs-events-total []))))))) + +(t/deftest the-binfile-queue-is-labelled-like-any-other + (let [metrics (make-metrics) + defs {:export-binfile {::jobs/name :export-binfile + ::jobs/schema [:map] + ::jobs/handler (fn [_context params] params) + ::jobs/decoder identity + ::jobs/validator (constantly true)}} + cfg {::jobs/defs defs + ::db/pool th/*pool* + ::mtx/metrics metrics}] + + (jobs/submit cfg {::jobs/name :export-binfile + ::jobs/params {} + ::jobs/queue :binfile}) + + (t/testing "the queue of the job is its label, never other" + (t/is (= 1.0 (counter-value metrics :jobs-submitted ["export-binfile" "binfile"]))) + (t/is (= 0.0 (counter-value metrics :jobs-submitted ["export-binfile" "other"])))))) diff --git a/backend/test/backend_tests/jobs_storage_test.clj b/backend/test/backend_tests/jobs_storage_test.clj new file mode 100644 index 0000000000..3da1f29525 --- /dev/null +++ b/backend/test/backend_tests/jobs_storage_test.clj @@ -0,0 +1,205 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns backend-tests.jobs-storage-test + "The storage helpers of the unified jobs substrate: the objects a job + owns through `job.resource_id`." + (:require + [app.common.time :as ct] + [app.common.uuid :as uuid] + [app.config :as cf] + [app.db :as db] + [app.jobs.storage :as js] + [app.storage :as sto] + [backend-tests.helpers :as th] + [clojure.test :as t] + [cuerdas.core :as str] + [datoteka.fs :as fs])) + +(t/use-fixtures :once th/state-init) +(t/use-fixtures :each (th/serial th/database-reset th/clean-storage)) + +(defn- get-storage + [] + (assoc (:app.storage/storage th/*system*) ::sto/backend :fs)) + +(defn- make-cfg + [] + {::db/pool th/*pool* + ::sto/storage (get-storage)}) + +(defn- get-object-row + "The `storage_object` row, deleted rows included." + [id] + (th/db-get :storage-object {:id id} {::db/remove-deleted false})) + +(defn- put-artifact + [cfg profile-id content] + (js/put-resource cfg profile-id {:content (sto/content content) + :filename "artifact.penpot" + :mtype "application/zip"})) + +(t/deftest put-resource-stores-the-artifact-for-its-owner + (let [cfg (make-cfg) + profile-id (:id (th/create-profile* 1)) + result (put-artifact cfg profile-id "penpot-bytes") + object (sto/get-object (get-storage) (:resource-id result)) + row (get-object-row (:resource-id result))] + + (t/testing "the object lives in the job-resource bucket" + (t/is (= sto/job-resource-bucket (:bucket (meta object))))) + + (t/testing "the owner and the content type travel in the metadata" + (t/is (= profile-id (:profile-id (meta object)))) + (t/is (= "application/zip" (:content-type (meta object))))) + + (t/testing "the object is readable: retention comes from the job row" + (t/is (nil? (:deleted-at row)))) + + (t/testing "it is touched right away so a crash leaves it reclaimable" + (t/is (some? (:touched-at row)))) + + (t/testing "the descriptor identifies the artifact" + (t/is (uuid? (:resource-id result))) + (t/is (= 12 (:size result))) + (t/is (= "artifact.penpot" (:filename result))) + (t/is (= "application/zip" (:mtype result))) + (t/is (str/includes? (str (:resource-uri result)) + (str (:resource-id result))))))) + +(t/deftest put-resource-requires-an-owner + (let [cfg (make-cfg)] + (t/testing "an artifact without an owner is a caller bug" + (t/is (thrown? Throwable + (js/put-resource cfg nil {:content (sto/content "x") + :filename "x.penpot" + :mtype "application/zip"})))))) + +(t/deftest load-input-gives-the-handler-a-local-path + (let [cfg (make-cfg) + profile-id (:id (th/create-profile* 1)) + staged (put-artifact cfg profile-id "package-bytes") + context {:id (uuid/next) + :resource-id (:resource-id staged) + :profile-id profile-id} + path (js/load-input cfg context)] + + (t/is (some? path)) + (t/is (= "package-bytes" (slurp (fs/file path)))))) + +(t/deftest load-input-refuses-an-input-of-another-profile + (let [cfg (make-cfg) + owner (:id (th/create-profile* 1)) + stranger (:id (th/create-profile* 2)) + staged (put-artifact cfg owner "package-bytes") + context {:id (uuid/next) + :resource-id (:resource-id staged) + :profile-id stranger}] + + (t/testing "a handler never reads a package that is not its job's" + (t/is (thrown? Exception (js/load-input cfg context)))))) + +(t/deftest load-input-refuses-a-missing-or-released-input + (let [cfg (make-cfg) + profile-id (:id (th/create-profile* 1)) + staged (put-artifact cfg profile-id "package-bytes") + context {:id (uuid/next) + :resource-id (:resource-id staged) + :profile-id profile-id}] + + (t/testing "a job without a resource reference has nothing to load" + (t/is (thrown? Exception (js/load-input cfg (dissoc context :resource-id))))) + + (t/testing "an input that was already released cannot be read again" + (t/is (true? (js/release-input cfg context))) + (t/is (thrown? Exception (js/load-input cfg context)))))) + +(t/deftest release-input-is-idempotent-and-never-throws + (let [cfg (make-cfg) + profile-id (:id (th/create-profile* 1)) + staged (put-artifact cfg profile-id "package-bytes") + context {:id (uuid/next) + :resource-id (:resource-id staged) + :profile-id profile-id}] + + (t/is (true? (js/release-input cfg context))) + (t/is (some? (:deleted-at (get-object-row (:resource-id staged))))) + + (t/testing "releasing twice is a no-op, not an error" + (t/is (false? (js/release-input cfg context)))) + + (t/testing "releasing a job without a resource is a no-op too" + (t/is (nil? (js/release-input cfg (dissoc context :resource-id))))))) + +(t/deftest release-input-clears-the-reference-of-the-job + ;; a job that no longer owns its package must not keep pointing at a + ;; deleted object: the object and the pointer are dropped together + (let [cfg (make-cfg) + profile-id (:id (th/create-profile* 1)) + staged (put-artifact cfg profile-id "package-bytes") + job-id (uuid/next) + context {:id job-id + :resource-id (:resource-id staged) + :profile-id profile-id}] + + (th/db-insert! :job {:id job-id + :name "import-binfile" + :tenant (cf/get :tenant) + :queue "binfile" + :params (db/json {}) + :priority 100 + :max-retries 0 + :retry-num 0 + :status "running" + :profile-id profile-id + :resource-id (:resource-id staged) + :scheduled-at (ct/now) + :created-at (ct/now) + :modified-at (ct/now)}) + + (t/is (true? (js/release-input cfg context))) + + (t/testing "the object is marked as deleted" + (t/is (some? (:deleted-at (get-object-row (:resource-id staged)))))) + + (t/testing "and the job row no longer points at it" + (t/is (nil? (:resource-id (th/db-get :job {:id job-id}))))) + + (t/testing "releasing twice leaves the row alone" + (t/is (false? (js/release-input cfg context))) + (t/is (nil? (:resource-id (th/db-get :job {:id job-id}))))))) + +(t/deftest release-resource-releases-a-package-no-job-owns + ;; the creation path stages the package before the job exists: when the + ;; submit fails the package is released with no job id, so the object + ;; goes and no row may be touched by it + (let [cfg (make-cfg) + profile-id (:id (th/create-profile* 1)) + staged (put-artifact cfg profile-id "package-bytes") + job-id (uuid/next)] + (th/db-insert! :job {:id job-id + :name "import-binfile" + :tenant (cf/get :tenant) + :queue "binfile" + :params (db/json {}) + :priority 100 + :max-retries 0 + :retry-num 0 + :status "running" + :profile-id profile-id + :resource-id nil + :scheduled-at (ct/now) + :created-at (ct/now) + :modified-at (ct/now)}) + + (let [before (th/db-get :job {:id job-id})] + (t/is (true? (js/release-resource cfg (:resource-id staged)))) + + (t/testing "the object is marked as deleted" + (t/is (some? (:deleted-at (get-object-row (:resource-id staged)))))) + + (t/testing "and the unrelated job row is exactly as it was" + (t/is (= before (th/db-get :job {:id job-id}))))))) diff --git a/backend/test/backend_tests/jobs_test.clj b/backend/test/backend_tests/jobs_test.clj index 2aba405738..20675fed2b 100644 --- a/backend/test/backend_tests/jobs_test.clj +++ b/backend/test/backend_tests/jobs_test.clj @@ -27,6 +27,11 @@ (t/use-fixtures :each test-fixture) +(defn- raised + "Run `f` and answer with the exception it raised, or nil when it did not." + [f] + (try (f) nil (catch Throwable cause cause))) + ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; JOB-DEF (plain handler + precompiled init-key) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; @@ -39,7 +44,8 @@ [cfg params] (when (::jobs/job-id cfg) (jobs/heartbeat cfg) - (jobs/heartbeat cfg :progress {:current 1 :stage "half"})) + (jobs/heartbeat cfg :progress {:stage :half + :counters {:work {:current 1 :total 2}}})) params) (def schema:echo-params @@ -101,7 +107,24 @@ WHERE job_id = ? AND kind = 'progress' ORDER BY created_at ASC, id ASC" job-id]) - (mapv #(db/decode-json-pgobject (:payload %))))) + (mapv #(jobs/decode-progress (:payload %))))) + +(defn- get-events + "Every event of a job, in insertion order." + [job-id] + (->> (th/db-exec! ["SELECT kind, payload FROM job_event + WHERE job_id = ? ORDER BY id ASC" job-id]) + (mapv (fn [{:keys [kind payload]}] + {:kind kind :payload (db/decode-json-pgobject payload)})))) + +(defn- get-outcomes + "Outcomes of the end events of a job, oldest first." + [job-id] + (mapv (comp :outcome :payload) (filter #(= "end" (:kind %)) (get-events job-id)))) + +(defn- get-kinds + [job-id] + (mapv :kind (get-events job-id))) (defn- fake-msgbus "A minimal msgbus that records every publication on the atom." @@ -159,48 +182,101 @@ (t/testing "no job row was created by the rejected submits" (t/is (= 0 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job"]))))))) +(t/deftest submit-persists-an-expiry-date + (let [cfg (make-cfg (get-job-defs)) + expires-at (ct/plus (ct/now) (ct/duration {:days 7}))] + + (t/testing "an explicit expiry is persisted as the retention of the job" + (let [job-id (jobs/submit cfg {::jobs/name :echo + ::jobs/params (make-params) + ::jobs/expires-at expires-at}) + row (jobs/get-job cfg job-id)] + (t/is (= (inst-ms expires-at) (inst-ms (:expires-at row)))))) + + (t/testing "without the option the column stays nil, as before" + (let [job-id (jobs/submit cfg {::jobs/name :echo + ::jobs/params (make-params)}) + row (jobs/get-job cfg job-id)] + (t/is (nil? (:expires-at row))))))) + +(t/deftest submit-rejects-an-invalid-expiry-date + (let [cfg (make-cfg (get-job-defs))] + (t/testing "a value that is not an instant is rejected before the insert" + (t/is (thrown? Exception + (jobs/submit cfg {::jobs/name :echo + ::jobs/params (make-params) + ::jobs/expires-at "not-an-instant"})))) + (t/testing "no job row was created by the rejected submit" + (t/is (= 0 (:cnt (th/db-exec-one! ["SELECT count(*) AS cnt FROM job"]))))))) + (t/deftest progress-report-validates-its-payload (let [cfg (make-cfg (get-job-defs)) job-id (jobs/submit cfg {::jobs/name :echo ::jobs/params (make-params)})] - (t/testing "current is mandatory" - (t/is (thrown? Exception - (jobs/heartbeat cfg :job-id job-id :progress {:total 10})))) - - (t/testing "unknown keys are rejected" + (t/testing "stage is mandatory" (t/is (thrown? Exception (jobs/heartbeat cfg :job-id job-id - :progress {:current 1 :step 1})))) + :progress {:counters {:work {:current 1}}})))) - (t/testing "current must not be negative" + (t/testing "stage must be a keyword, not a label" + (t/is (thrown? Exception + (jobs/heartbeat cfg :job-id job-id + :progress {:stage "work"})))) + + (t/testing "unknown keys are rejected, at the top and inside a counter" + (t/is (thrown? Exception + (jobs/heartbeat cfg :job-id job-id + :progress {:stage :work :step 1}))) + (t/is (thrown? Exception + (jobs/heartbeat cfg :job-id job-id + :progress {:stage :work + :counters {:work {:current 1 + :step 1}}})))) + + (t/testing "a counter current must not be negative" (t/is (thrown-with-msg? Exception #"negative" (jobs/heartbeat cfg :job-id job-id - :progress {:current -1})))) + :progress {:stage :work + :counters {:work {:current -1}}})))) - (t/testing "total must be positive" + (t/testing "a counter total must be positive" (t/is (thrown-with-msg? Exception #"positive" (jobs/heartbeat cfg :job-id job-id - :progress {:current 0 :total 0})))) + :progress {:stage :work + :counters {:work {:current 0 + :total 0}}})))) - (t/testing "current must not be greater than total" + (t/testing "a counter current must not be greater than its total" (t/is (thrown-with-msg? Exception #"not lower than current" (jobs/heartbeat cfg :job-id job-id - :progress {:current 5 :total 4})))) + :progress {:stage :work + :counters {:work {:current 5 + :total 4}}})))) - (t/testing "stage is limited to 250 characters" - (t/is (thrown-with-msg? Exception #"250" + (t/testing "keys are limited to a name and not a payload" + (t/is (thrown-with-msg? Exception #"64" (jobs/heartbeat cfg :job-id job-id - :progress {:current 1 - :stage (apply str (repeat 251 "x"))}))) + :progress {:stage (keyword (apply str (repeat 65 "x")))})))) + + (t/testing "a milestone without counters is accepted" + (t/is (= 2 (jobs/heartbeat cfg :job-id job-id :progress {:stage :relations})))) + + (t/testing "a milestone with counters is accepted and stored as it is" + (swap! jobs/progresses dissoc job-id) + (swap! jobs/heartbeats dissoc job-id) (t/is (= 2 (jobs/heartbeat cfg :job-id job-id - :progress {:current 1 - :stage (apply str (repeat 250 "x"))}))) - (t/is (= [{:current 1 :stage (apply str (repeat 250 "x"))}] + :progress {:stage :pages + :counters {:files {:current 2 :total 5} + :pages {:current 3 :total 8}}}))) + (t/is (= [{:stage :relations} + {:stage :pages + :counters {:files {:current 2 :total 5} + :pages {:current 3 :total 8}}}] (get-progresss job-id)))) (t/testing "a rejected report never reaches the durable log" - (t/is (= 1 (count (get-progresss job-id))))))) + (t/is (= 2 (count (get-progresss job-id))))))) (t/deftest progress-event-publishes-msgbus-for-profile-jobs (let [cfg (make-cfg (get-job-defs)) @@ -211,7 +287,8 @@ ::jobs/params (make-params) ::jobs/profile-id profile-id})] (t/is (pos? (jobs/heartbeat job-cfg :job-id job-id - :progress {:current 3 :total 7}))) + :progress {:stage :pages + :counters {:files {:current 3 :total 7}}}))) (t/testing "the event is published on the topic of the job profile" (t/is (= 1 (count @messages))) (let [{:keys [topic message]} (first @messages)] @@ -220,7 +297,9 @@ (t/is (= job-id (:job-id message))) (t/is (= profile-id (:profile-id message))) (t/is (= "progress" (:kind message))) - (t/is (= {:current 3 :total 7} (:payload message))) + (t/is (= {:stage :pages + :counters {:files {:current 3 :total 7}}} + (:payload message))) (t/is (some? (:event-id message))) (t/is (some? (:created-at message))))))) @@ -230,7 +309,7 @@ job-cfg (assoc cfg ::mbus/msgbus (fake-msgbus messages)) job-id (jobs/submit job-cfg {::jobs/name :echo ::jobs/params (make-params)})] - (t/is (pos? (jobs/heartbeat job-cfg :job-id job-id :progress {:current 1}))) + (t/is (pos? (jobs/heartbeat job-cfg :job-id job-id :progress {:stage :work}))) (t/testing "the event is stored but nothing is published" (t/is (= 1 (count (get-progresss job-id)))) (t/is (= [] @messages))))) @@ -245,7 +324,7 @@ (t/is (thrown-with-msg? Exception #"require ::mbus/msgbus" (jobs/heartbeat cfg :job-id job-id - :progress {:current 1} + :progress {:stage :work} ::jobs/force? true))) (t/is (= [] (get-progresss job-id)) "the event must not be stored without its notification")) @@ -254,7 +333,7 @@ (th/db-update! :job {:modified-at (ct/in-past {:minutes 5})} {:id job-id}) (swap! jobs/heartbeats dissoc job-id) (t/is (= 1 (jobs/heartbeat cfg :job-id job-id - :progress {:current 1}))) + :progress {:stage :work}))) (t/is (= [] (get-progresss job-id)))))) (t/deftest submit-validates-params-with-job-schema @@ -530,34 +609,67 @@ (t/is (= (inst-ms (:modified-at row1)) (inst-ms (:modified-at row2)))))))) -(t/deftest heartbeat-skips-terminal-states +(t/deftest heartbeat-interrupts-on-terminal-states (let [cfg (make-cfg (get-job-defs)) job-id (jobs/submit cfg {::jobs/name :echo ::jobs/params (make-params)})] - (t/testing "terminal states are never touched by heartbeat" + (t/testing "a terminal state raises the interrupt instead of touching the row" (th/db-update! :job {:status "completed" :modified-at (ct/in-past {:days 10})} {:id job-id}) (swap! @#'jobs/heartbeats dissoc job-id) - (let [before (jobs/get-job cfg job-id)] - (jobs/heartbeat cfg :job-id job-id) + (let [before (jobs/get-job cfg job-id) + cause (raised #(jobs/heartbeat cfg :job-id job-id))] + (t/is (= :interrupt (th/ex-type cause))) + (t/is (= :job-interrupted (th/ex-code cause))) + (t/is (= job-id (:job-id (ex-data cause)))) + (t/is (= "completed" (:status (ex-data cause)))) (t/is (= (inst-ms (:modified-at before)) (inst-ms (:modified-at (jobs/get-job cfg job-id))))))))) -(t/deftest progress-events-respect-throttle-and-skip-terminal-states +(t/deftest heartbeat-interrupts-when-the-row-is-gone + (let [cfg (make-cfg (get-job-defs)) + job-id (jobs/submit cfg {::jobs/name :echo + ::jobs/params (make-params)})] + (th/db-force-delete :job {:id job-id}) + (let [cause (raised #(jobs/heartbeat cfg :job-id job-id))] + (t/is (= :interrupt (th/ex-type cause))) + (t/is (= :job-interrupted (th/ex-code cause))) + (t/is (= "gone" (:status (ex-data cause))))))) + +(t/deftest a-transient-progress-failure-is-not-an-interrupt + "The handler path logs and ignores a progress insert that fails, but an + interrupt must not be swallowed with it: only an interrupt means the job + is gone." + (let [cfg (make-cfg (get-job-defs)) + job-id (jobs/submit cfg {::jobs/name :echo + ::jobs/params (make-params)})] + (swap! jobs/progresses dissoc job-id) + (swap! jobs/heartbeats dissoc job-id) + ;; the insert is the boundary that fails: the job itself is active, so + ;; the beat answers with the touch it did write instead of raising + (with-redefs-fn {#'jobs/report-progress (fn [& _] (throw (ex-info "boom" {})))} + #(t/is (= 1 (jobs/heartbeat cfg :job-id job-id :progress {:stage :work})))) + (t/is (= [] (get-progresss job-id))))) + +(t/deftest progress-events-respect-throttle-and-interrupt-on-terminal-states (let [cfg (make-cfg (get-job-defs)) job-id (jobs/submit cfg {::jobs/name :echo ::jobs/params (make-params)})] (t/testing "first progress report appends a progress event" (t/is (= 2 (jobs/heartbeat cfg :job-id job-id - :progress {:current 1 :total 10}))) - (t/is (= [{:current 1 :total 10}] (get-progresss job-id)))) + :progress {:stage :pages + :counters {:pages {:current 1 :total 10}}}))) + (t/is (= [{:stage :pages :counters {:pages {:current 1 :total 10}}}] + (get-progresss job-id)))) (t/testing "immediate second progress report is throttled" (t/is (= 0 (jobs/heartbeat cfg :job-id job-id - :progress {:current 2 :total 10}))) - (t/is (= [{:current 1 :total 10}] (get-progresss job-id))) + :progress {:stage :pages + :counters {:pages {:current 2 :total 10}}}))) + (t/is (= [{:stage :pages :counters {:pages {:current 1 :total 10}}}] + (get-progresss job-id))) (t/testing "after the throttle window elapses it appends again" (swap! @#'jobs/progresses @@ -565,9 +677,11 @@ (update-in m [job-id] #(ct/minus % (ct/duration {:seconds 2}))))) - (jobs/heartbeat cfg :job-id job-id :progress {:current 3 :total 10}) - (t/is (= [{:current 1 :total 10} - {:current 3 :total 10}] + (jobs/heartbeat cfg :job-id job-id + :progress {:stage :pages + :counters {:pages {:current 3 :total 10}}}) + (t/is (= [{:stage :pages :counters {:pages {:current 1 :total 10}}} + {:stage :pages :counters {:pages {:current 3 :total 10}}}] (get-progresss job-id)))))) (let [cfg (make-cfg (get-job-defs)) @@ -576,9 +690,11 @@ (t/testing "terminal states never get a progress event" (th/db-update! :job {:status "completed"} {:id job-id}) (swap! @#'jobs/progresses dissoc job-id) - (t/is (= 0 (jobs/heartbeat cfg :job-id job-id - :progress {:current 9}))) - (t/is (= [] (get-progresss job-id)))))) + (swap! @#'jobs/heartbeats dissoc job-id) + (let [cause (raised #(jobs/heartbeat cfg :job-id job-id + :progress {:stage :work}))] + (t/is (= :interrupt (th/ex-type cause))) + (t/is (= [] (get-progresss job-id))))))) (t/deftest throttle-prune-removes-stale-entries-keeps-fresh "When the throttle map exceeds prune-threshold, stale entries (older than @@ -609,7 +725,7 @@ (t/is (contains? state job-id-2) "fresh entry kept") (t/is (contains? state job-id-3) "new entry added")))))) -(t/deftest cancel-skips-running-and-terminal-jobs +(t/deftest cancel-skips-terminal-jobs (let [cfg (make-cfg (get-job-defs)) job-id (jobs/submit cfg {::jobs/name :echo ::jobs/params (make-params)})] @@ -617,27 +733,30 @@ (t/testing "pending job can be cancelled" (t/is (= 1 (jobs/cancel cfg job-id))) (t/is (= "cancelled" (:status (jobs/get-job cfg job-id)))) - + (t/is (= ["cancelled"] (get-outcomes job-id))) (t/testing "already cancelled job is not affected again" - (t/is (zero? (jobs/cancel cfg job-id)))))) + (t/is (zero? (jobs/cancel cfg job-id))) + (t/is (= ["cancelled"] (get-outcomes job-id)))))) + (doseq [status ["completed" "failed"]] + (t/testing (str status " job cannot be cancelled") + (let [cfg (make-cfg (get-job-defs)) + job-id (jobs/submit cfg {::jobs/name :echo + ::jobs/params (make-params)})] + (th/db-update! :job {:status status} {:id job-id}) + (t/is (zero? (jobs/cancel cfg job-id))) + (t/is (= status (:status (jobs/get-job cfg job-id)))) + (t/is (= [] (get-outcomes job-id))))))) - (let [cfg (make-cfg (get-job-defs)) - job-id (jobs/submit cfg {::jobs/name :echo - ::jobs/params (make-params)})] - (t/testing "running job cannot be cancelled" - (th/db-update! :job {:status "running"} {:id job-id}) - (t/is (zero? (jobs/cancel cfg job-id))) - (t/is (= "running" (:status (jobs/get-job cfg job-id))))))) - -(t/deftest cancel-affects-scheduled-and-retry-jobs +(t/deftest cancel-affects-pending-and-running-jobs (let [cfg (make-cfg (get-job-defs))] - (doseq [status ["scheduled" "retry"]] + (doseq [status ["scheduled" "retry" "running"]] (t/testing (str status " job can be cancelled") (let [job-id (jobs/submit cfg {::jobs/name :echo ::jobs/params (make-params)})] (th/db-update! :job {:status status} {:id job-id}) (t/is (= 1 (jobs/cancel cfg job-id))) - (t/is (= "cancelled" (:status (jobs/get-job cfg job-id))))))))) + (t/is (= "cancelled" (:status (jobs/get-job cfg job-id)))) + (t/is (= ["cancelled"] (get-outcomes job-id)))))))) (t/deftest get-user-status-maps-internal-statuses (t/are [status expected] (= expected (jobs/get-user-status status)) @@ -673,6 +792,32 @@ ::jobs/handler (fn [_context params] params)}}}))))) +(t/deftest job-def-carries-optional-family-and-resource-role + (let [job-def {::jobs/name :echo + ::jobs/family :export + ::jobs/resource-role :output + ::jobs/schema schema:echo-params + ::jobs/handler (fn [_context params] params) + ::jobs/decoder (sm/decoder schema:echo-params sm/json-transformer) + ::jobs/validator (sm/validator schema:echo-params)}] + + (t/testing "the metadata is accepted and readable from the registry" + (let [defs (-> (ig/init {::jobs/defs {:echo job-def}}) + (get ::jobs/defs)) + echo (jobs/get-job-def defs :echo)] + (t/is (= :export (::jobs/family echo))) + (t/is (= :output (::jobs/resource-role echo))))) + + (t/testing "a family is a keyword, not a free string" + (t/is (thrown? Exception + (ig/init {::jobs/defs {:echo (assoc job-def + ::jobs/family "export")}})))) + + (t/testing "a resource role is a keyword too" + (t/is (thrown? Exception + (ig/init {::jobs/defs {:echo (assoc job-def + ::jobs/resource-role "output")}})))))) + (t/deftest generic-schema-round-trip-preserves-type-sensitive-fields "For each registered job-def, verify that type-sensitive fields (uuids, insts) survive the JSON round-trip (db/json → decode). This catches the F2 class of @@ -742,25 +887,28 @@ (t/testing "progress reporting is a no-op when *job-id* is nil" (binding [jobs/*job-id* nil] - (t/is (nil? (jobs/heartbeat cfg :progress {:current 1})))) + (t/is (nil? (jobs/heartbeat cfg :progress {:stage :work})))) (t/is (= [] (get-progresss job-id)) "no progress event should be stored")))) -(t/deftest progress-noop-on-terminal-states +(t/deftest progress-interrupts-on-terminal-states "A progress report never stores an event when the job is already in a terminal state (completed, failed, cancelled, aborted): the row is locked and checked before the insert, so a report can never race a terminal - transition." + transition, and the beat raises the interrupt instead." (let [cfg (make-cfg (get-job-defs))] (doseq [status ["completed" "failed" "cancelled" "aborted"]] - (t/testing (str "progress is a no-op on " status " status") + (t/testing (str "progress interrupts on " status " status") (let [job-id (jobs/submit cfg {::jobs/name :echo ::jobs/params (make-params)})] (th/db-update! :job {:status status} {:id job-id}) - ;; reset the throttle so should-write? would allow the write + ;; reset the throttles so should-write? would allow the writes (swap! jobs/progresses dissoc job-id) + (swap! jobs/heartbeats dissoc job-id) (binding [jobs/*job-id* job-id] - (t/is (= 0 (jobs/heartbeat cfg :progress {:current 1})))) + (let [cause (raised #(jobs/heartbeat cfg :progress {:stage :work}))] + (t/is (= :interrupt (th/ex-type cause))) + (t/is (= status (:status (ex-data cause)))))) (t/is (= [] (get-progresss job-id)))))))) (t/deftest defs-halt-clears-module-registry @@ -797,8 +945,8 @@ (fn [{:keys [::db/conn]}] (jobs/heartbeat (assoc cfg ::db/conn conn) :job-id job-id - :progress {:current 1}))) - (t/is (= [{:current 1}] (get-progresss job-id))))) + :progress {:stage :work}))) + (t/is (= [{:stage :work}] (get-progresss job-id))))) (t/deftest progress-notification-survives-a-rolled-back-caller (let [cfg (make-cfg (get-job-defs)) @@ -814,9 +962,9 @@ (fn [{:keys [::db/conn]}] (jobs/heartbeat (assoc job-cfg ::db/conn conn) :job-id job-id - :progress {:current 1}))) + :progress {:stage :work}))) (t/testing "the event is stored" - (t/is (= [{:current 1}] (get-progresss job-id)))) + (t/is (= [{:stage :work}] (get-progresss job-id)))) (t/testing "and the notification went out with its own commit" (t/is (= 1 (count @messages))) (t/is (= "progress" (:kind (:message (first @messages)))))))) @@ -829,7 +977,7 @@ (t/is (thrown? Exception (jobs/heartbeat th/*pool* :job-id job-id - :progress {:current 1})))))) + :progress {:stage :work})))))) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; @@ -844,6 +992,7 @@ :tenant (cf/get :tenant) :queue "default" :label nil + :profile-id nil :resource-id nil :retry-num 0 :max-retries 3} @@ -851,7 +1000,7 @@ (t/deftest make-context-selects-exactly-the-agreed-keys (let [context (jobs/make-context (mk-row {}))] - (t/is (= #{:id :name :label :resource-id} (set (keys context)))) + (t/is (= #{:id :name :label :resource-id :profile-id} (set (keys context)))) (t/testing "the queue is a routing detail of the dispatcher, not context" (t/is (not (contains? context :queue)))) (t/testing "nor is the tenant that owns the queue" @@ -880,11 +1029,23 @@ (t/is (= resource-id (:resource-id (jobs/make-context (mk-row {:resource-id resource-id})))))))) +(t/deftest make-context-carries-the-job-owner + (t/testing "the key exists even when the job has no owner" + (let [context (jobs/make-context (mk-row {}))] + (t/is (contains? context :profile-id)) + (t/is (nil? (:profile-id context))))) + (t/testing "the owner of a user job is carried as is" + (let [profile-id (uuid/next)] + (t/is (= profile-id + (:profile-id (jobs/make-context (mk-row {:profile-id profile-id})))))))) + (t/deftest make-context-rejects-invalid-rows (t/testing "a missing id is rejected" (t/is (thrown? Exception (jobs/make-context (mk-row {:id nil}))))) (t/testing "a missing name is rejected" (t/is (thrown? Exception (jobs/make-context (mk-row {:name nil}))))) + (t/testing "a non uuid owner is rejected" + (t/is (thrown? Exception (jobs/make-context (mk-row {:profile-id "nope"}))))) (t/testing "a non uuid resource reference is rejected" (t/is (thrown? Exception (jobs/make-context (mk-row {:resource-id "nope"})))))) @@ -955,29 +1116,13 @@ "a job-id without a context still delivers a nil context") (t/testing "the job-id is what makes the durable writes reach the row" (t/is (pos? (jobs/heartbeat cfg :job-id job-id - :progress {:current 1}))) - (t/is (= [{:current 1}] (get-progresss job-id)))))) + :progress {:stage :work}))) + (t/is (= [{:stage :work}] (get-progresss job-id)))))) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; LIFECYCLE EVENTS ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; -(defn- get-events - "Every event of a job, in insertion order." - [job-id] - (->> (th/db-exec! ["SELECT kind, payload FROM job_event - WHERE job_id = ? ORDER BY id ASC" job-id]) - (mapv (fn [{:keys [kind payload]}] - {:kind kind :payload (db/decode-json-pgobject payload)})))) - -(defn- get-kinds - [job-id] - (mapv :kind (get-events job-id))) - -(defn- get-outcomes - [job-id] - (mapv (comp :outcome :payload) (filter #(= "end" (:kind %)) (get-events job-id)))) - (defn- mk-running "A job already claimed by a worker, ready for a terminal transition." [cfg] @@ -1269,6 +1414,23 @@ (t/is (nil? (:resource-id row))) (t/is (= "running" (:status row)))))))) +(t/deftest insert-event-requires-a-transaction + ;; The event insert is only valid inside the transaction that owns the + ;; job row: outside one the assert fires instead of writing. Like every + ;; assert-based guard, this only holds with the backend asserts flag. + (t/is (true? *assert*) + "run the suite with PENPOT_FLAGS=\"enable-backend-asserts\"") + (let [cfg (make-cfg (get-job-defs)) + job-id (mk-running cfg)] + (t/testing "a pool cfg with no transaction raises" + (t/is (thrown? AssertionError + (@#'jobs/insert-event cfg job-id "progress" {:stage :half})))) + (t/testing "a bare connection with no transaction raises too" + (db/run! cfg (fn [conn-cfg] + (t/is (thrown? AssertionError + (@#'jobs/insert-event conn-cfg job-id "progress" + {:stage :half})))))))) + (t/deftest complete-associates-a-resource-id-once (let [cfg (make-cfg (get-job-defs)) resource-id (uuid/next)] @@ -1393,10 +1555,17 @@ (let [cfg (make-cfg (get-job-defs)) job-id (mk-running cfg)] (t/testing "a progress report and a completion of the same job, at once" - (let [[_writes completed] (race #(jobs/heartbeat cfg :job-id job-id - :progress {:current 1} - ::jobs/force? true) - #(jobs/complete cfg :job-id job-id :result {:v 1}))] + ;; the report either lands before the end or loses the race and gets + ;; the interrupt: a beat on a job that is no longer active raises + (let [report (fn [] + (if (some? (raised #(jobs/heartbeat cfg :job-id job-id + :progress {:stage :work} + ::jobs/force? true))) + :interrupted + :reported)) + [outcome completed] (race report #(jobs/complete cfg :job-id job-id :result {:v 1}))] + (t/is (contains? #{:interrupted :reported} outcome) + "the report lands or gets the interrupt, never anything else") (t/is (= 1 completed) "the completion always wins, it owns the row") (t/testing "the report either lost the race or was stored before the end" (let [kinds (get-kinds job-id)] @@ -1406,10 +1575,11 @@ (t/is (<= 2 (count kinds) 3) "at most the one report that made it in before the end")))) (t/testing "a job that ended is not resurrected by a late report" - (let [before (get-kinds job-id)] - (t/is (zero? (jobs/heartbeat cfg :job-id job-id - :progress {:current 2} - ::jobs/force? true))) + (let [before (get-kinds job-id) + cause (raised #(jobs/heartbeat cfg :job-id job-id + :progress {:stage :work} + ::jobs/force? true))] + (t/is (= :interrupt (th/ex-type cause))) (t/is (= before (get-kinds job-id)) "the late report stored nothing")))))) (t/deftest progress-event-reaches-the-cfg-of-its-own-callback @@ -1426,9 +1596,9 @@ ::jobs/params (make-params) ::jobs/profile-id profile-id})] (t/is (pos? (jobs/heartbeat job-cfg :job-id job-id - :progress {:current 1}))) + :progress {:stage :work}))) (t/testing "the event was stored" - (t/is (= [{:current 1}] (get-progresss job-id)))) + (t/is (= [{:stage :work}] (get-progresss job-id)))) (t/testing "and the msgbus on that cfg was reachable from the callback" (t/is (= 1 (count @messages))) (t/is (= "progress" (:kind (:message (first @messages)))))))) diff --git a/backend/test/backend_tests/rpc/commands/jobs_test.clj b/backend/test/backend_tests/rpc/commands/jobs_test.clj new file mode 100644 index 0000000000..a03b5a9ca4 --- /dev/null +++ b/backend/test/backend_tests/rpc/commands/jobs_test.clj @@ -0,0 +1,717 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns backend-tests.rpc.commands.jobs-test + "The creation of a user-facing job: what it answers, what it freezes and + what it refuses before anything is stored." + (:require + [app.binfile.common :as bfc] + [app.binfile.v3 :as v3] + [app.common.exceptions :as ex] + [app.common.time :as ct] + [app.common.uuid :as uuid] + [app.config :as cf] + [app.db :as db] + [app.jobs :as jobs] + [app.loggers.audit :as-alias audit] + [app.rpc :as-alias rpc] + [app.rpc.climit :as-alias climit] + [app.rpc.commands.jobs :as cmd-jobs] + [app.rpc.quotes :as-alias quotes] + [app.storage :as sto] + [app.storage.tmp :as tmp] + [backend-tests.helpers :as th] + [clojure.java.io :as jio] + [clojure.test :as t] + [cuerdas.core :as str] + [datoteka.fs :as fs] + [datoteka.io :as io] + [mockery.core :refer [with-mocks]]) + (:import + java.util.zip.ZipEntry + java.util.zip.ZipFile + java.util.zip.ZipOutputStream)) + +(t/use-fixtures :once th/state-init) +(t/use-fixtures :each (th/serial th/database-reset th/clean-storage)) + +(def ^:private fixture + "A real penpot export, the same fixture the binfile tests read." + "backend_tests/test_files/svg-attrs-camel-case.penpot") + +(defn- import-fixture! + "A file in the default project of the profile, imported from the + fixture through the core." + [profile] + (-> th/*system* + (assoc ::bfc/project-id (:default-project-id profile)) + (assoc ::bfc/profile-id (:id profile)) + (assoc ::bfc/team-id (:default-team-id profile)) + (assoc ::bfc/input (-> fixture io/resource jio/file)) + (v3/import-files!))) + +(defn- create-export-binfile-job + [profile-id file-ids & {:as extra}] + (th/command! (merge {::th/type :create-export-binfile-job + ::rpc/profile-id profile-id + :params {:file-ids file-ids + :export-type :detach-libraries}} + extra))) + +(defn- count-jobs + [] + (count (th/db-exec! ["SELECT id FROM job"]))) + +(defn- count-storage-objects + [] + (count (th/db-exec! ["SELECT id FROM storage_object"]))) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; TESTS +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(t/deftest create-export-binfile-job-answers-with-a-pending-job + (let [profile (th/create-profile* 1) + file-id (first (:file-ids (import-fixture! profile))) + before (count-storage-objects) + out (create-export-binfile-job (:id profile) #{file-id})] + + (t/is (th/success? out)) + + (let [result (:result out) + job (jobs/get-job th/*system* (:id result))] + + (t/testing "the answer is enough to follow the job" + (t/is (uuid? (:id result))) + (t/is (= "pending" (:status result))) + (t/is (= "export-binfile" (:name result))) + (t/is (ct/inst? (:created-at result))) + (t/is (pos? (- (inst-ms (:expires-at result)) (inst-ms (ct/now)))))) + + (t/testing "and the row freezes what the job will do" + (let [job-def (jobs/get-job-def (::jobs/defs th/*system*) :export-binfile) + params (jobs/decode-params job-def (:params job))] + (t/is (= "new" (:status job))) + (t/is (= "binfile" (:queue job))) + (t/is (= 0 (:max-retries job))) + (t/is (= (:id profile) (:profile-id job))) + (t/is (= #{file-id} (:file-ids params))) + (t/is (= :detach-libraries (:export-type params))))) + + (t/testing "creating a job does not run it" + ;; no runner is running here, so a call that waited for one would + ;; hang instead of answering: seeing the job pending and no + ;; artifact produced is what proves the call does not wait + (t/is (= before (count-storage-objects))))))) + +(t/deftest create-export-binfile-job-requires-the-export-type + ;; the type is part of the contract of the job: a caller that does not + ;; say how to handle the libraries gets an error, not a package it did + ;; not ask for + (let [profile (th/create-profile* 1) + file-id (first (:file-ids (import-fixture! profile))) + out (th/command! {::th/type :create-export-binfile-job + ::rpc/profile-id (:id profile) + :params {:file-ids #{file-id}}})] + + (t/is (not (th/success? out))) + (t/is (= :params-validation (th/ex-code (:error out)))) + (t/testing "and nothing was created" + (t/is (zero? (count-jobs)))))) + +(t/deftest create-export-binfile-job-requires-at-least-one-file + (let [profile (th/create-profile* 1) + out (create-export-binfile-job (:id profile) #{})] + + (t/is (not (th/success? out))) + (t/is (= :validation (th/ex-type (:error out)))) + (t/is (= :no-files-to-export (th/ex-code (:error out)))) + (t/testing "and nothing was created" + (t/is (zero? (count-jobs)))))) + +(t/deftest create-export-binfile-job-checks-the-read-permission + (let [owner (th/create-profile* 1) + other (th/create-profile* 2) + file-id (first (:file-ids (import-fixture! owner))) + out (create-export-binfile-job (:id other) #{file-id})] + + (t/testing "a file the caller cannot read is not exported" + (t/is (not (th/success? out))) + (t/is (= :not-found (th/ex-type (:error out))))) + + (t/testing "and no job was created for it" + (t/is (zero? (count-jobs)))))) + +(t/deftest create-export-binfile-job-enforces-the-quote + (with-mocks [mock {:target 'app.config/get + :return (th/config-get-mock + {:quotes-export-jobs-per-profile 1})}] + + (let [profile (th/create-profile* 1) + file-id (first (:file-ids (import-fixture! profile)))] + + (t/testing "the first job fits in the quote" + (t/is (th/success? (create-export-binfile-job (:id profile) #{file-id}))) + (t/is (= 1 (count-jobs)))) + + (t/testing "the one that would go over it is refused" + (let [out (create-export-binfile-job (:id profile) #{file-id})] + (t/is (not (th/success? out))) + (t/is (= :restriction (th/ex-type (:error out)))) + (t/is (= :max-quote-reached (th/ex-code (:error out)))) + (t/is (= "export-jobs-per-profile" (:target (ex-data (:error out))))))) + + (t/testing "so the quote is what stopped the second job" + (t/is (= 1 (count-jobs))))))) + +(t/deftest create-export-binfile-job-adds-the-job-id-to-the-audit-props + (let [profile (th/create-profile* 1) + file-id (first (:file-ids (import-fixture! profile))) + out (create-export-binfile-job (:id profile) #{file-id}) + props (::audit/props (meta (:result out)))] + + (t/testing "the audit of the call points at the job, not at its files" + (t/is (= (:id (:result out)) (:job-id props))) + (t/is (= 1 (:files props))) + (t/is (= "detach-libraries" (:export-type props))) + (t/is (= #{:job-id :files :export-type} (set (keys props))))))) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; IMPORT +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(defn- tempfile-with + "A temporary file holding `content`, as the http layer would leave one + for a plain upload." + [content] + (let [path (fs/create-tempfile :dir tmp/default-tmp-dir + :prefix "test-upload-")] + (with-open [out (io/output-stream path)] + (io/write* out (.getBytes ^String content "UTF-8"))) + path)) + +(defn- upload-chunk + "One chunk of the fixture, as the upload command takes it. `content` + replaces the fixture with something that is no package at all." + [& {:keys [content]}] + (if (some? content) + {:filename "package.penpot" + :path (tempfile-with content) + :mtype "application/zip" + :size (count content)} + (let [source (-> fixture io/resource jio/file)] + {:filename "package.penpot" + :path (th/tempfile fixture) + :mtype "application/zip" + :size (.length source)}))) +(defn- upload-chunked! + "Upload one chunk and return the session id. By default the chunk is the + fixture; `:path` uploads another file and `:content` uploads a string + that is no package at all." + [profile & {:keys [content path]}] + (let [session-id (-> (th/command! {::th/type :create-upload-session + ::rpc/profile-id (:id profile) + :total-chunks 1}) + :result :session-id) + out (th/command! {::th/type :upload-chunk + ::rpc/profile-id (:id profile) + :session-id session-id + :index 0 + :content (if (some? path) + (let [size (.length (jio/file path))] + {:filename "package.penpot" + :path path + :mtype "application/zip" + :size size}) + (upload-chunk :content content))})] + (t/is (nil? (:error out))) + session-id)) + +(defn- create-import-binfile-job + [profile-id project-id & {:keys [upload-id] :as extra}] + (th/command! (merge {::th/type :create-import-binfile-job + ::rpc/profile-id profile-id + :params {:project-id project-id + :name "imported"} + :upload-id upload-id} + extra))) + +(defn- session-row + [session-id] + (th/db-get :upload-session {:id session-id} {::db/remove-deleted false})) + +(defn- package-with-refer + "The fixture with the key of its manifest renamed to the old spelling, + `refer`, and keeping the value it already had." + [] + (let [source (-> fixture io/resource jio/file) + target (fs/create-tempfile :dir tmp/default-tmp-dir + :prefix "test-refer-")] + (with-open [zip (ZipFile. source) + out (ZipOutputStream. (io/output-stream target))] + (doseq [entry (enumeration-seq (.entries zip))] + (.putNextEntry out (ZipEntry. (.getName entry))) + (when-not (.isDirectory entry) + (let [content (slurp (.getInputStream zip entry)) + content (if (= "manifest.json" (.getName entry)) + (str/replace content #"\"referer\"(?=\s*:)" "\"refer\"") + content)] + (.write out (.getBytes content "UTF-8")))) + (.closeEntry out))) + target)) + +(defn- manifest-of + "The manifest of a package, as the reader of the import sees it." + [path] + (v3/get-manifest th/*system* path)) + +(defn- live-storage-objects + [] + (count (th/db-exec! ["SELECT id FROM storage_object WHERE deleted_at IS NULL"]))) + +(defn- session-row + [session-id] + (th/db-get :upload-session {:id session-id} {::db/remove-deleted false})) + +(defn- package-with-refer + "The fixture with the key of its manifest renamed to the old spelling, + `refer`, and keeping the value it already had." + [] + (let [source (-> fixture io/resource jio/file) + target (fs/create-tempfile :dir tmp/default-tmp-dir + :prefix "test-refer-")] + (with-open [zip (ZipFile. source) + out (ZipOutputStream. (io/output-stream target))] + (doseq [entry (enumeration-seq (.entries zip))] + (.putNextEntry out (ZipEntry. (.getName entry))) + (when-not (.isDirectory entry) + (let [content (slurp (.getInputStream zip entry)) + content (if (= "manifest.json" (.getName entry)) + (str/replace content #"\"referer\"(?=\s*:)" "\"refer\"") + content)] + (.write out (.getBytes content "UTF-8")))) + (.closeEntry out))) + target)) + +(defn- manifest-of + "The manifest of a package, as the reader of the import sees it." + [path] + (v3/get-manifest th/*system* path)) + +(defn- assembled-tempfiles + "The temporary files the chunk assembly leaves behind. The copies this + test uploads are its own business." + [] + (->> (.list (jio/file tmp/default-tmp-dir)) + (filter #(str/starts-with? % "penpot.chunked-upload.")) + (seq))) + +(t/deftest create-import-binfile-job-audits-the-old-spelling-of-the-manifest-tool + (let [profile (th/create-profile* 1) + package (package-with-refer) + expected (:referer (manifest-of (-> fixture io/resource jio/file))) + session-id (upload-chunked! profile :path package) + out (create-import-binfile-job (:id profile) (:default-project-id profile) + :upload-id session-id) + props (::audit/props (meta (:result out)))] + + (t/testing "the package names its tool with the old spelling only" + (t/is (some? expected)) + (t/is (nil? (:referer (manifest-of package)))) + (t/is (= expected (:refer (manifest-of package))))) + + (t/testing "and the audit carries that value anyway" + (t/is (th/success? out)) + (t/is (= expected (:referer props)))))) + +(t/deftest create-import-binfile-job-limits-its-own-concurrency + ;; the command assembles and stores the package inside the request, so + ;; it declares the same kind of limit the legacy import command has + (let [[mdata _] (get (::rpc/methods th/*system*) :create-import-binfile-job)] + (t/is (= [[:create-import-binfile-job/by-profile ::rpc/profile-id] + [:create-import-binfile-job/global]] + (::climit/id mdata))))) + +(t/deftest create-export-binfile-job-accepts-the-params-of-a-json-client + ;; a JSON body carries text and lists, not uuids or sets: the command + ;; has to read them with the decoder of the job-def, like the runner + ;; does when it reads a job from its row + (let [profile (th/create-profile* 1) + file-id (first (:file-ids (import-fixture! profile))) + out (th/command! {::th/type :create-export-binfile-job + ::rpc/profile-id (:id profile) + :params {:file-ids [(str file-id)] + :export-type "detach-libraries"}})] + + (t/is (th/success? out)) + + (let [job-def (jobs/get-job-def (::jobs/defs th/*system*) :export-binfile) + params (jobs/decode-params job-def + (:params (jobs/get-job th/*system* + (:id (:result out)))))] + (t/is (= #{file-id} (:file-ids params))) + (t/is (= :detach-libraries (:export-type params)))))) + +(t/deftest create-import-binfile-job-accepts-the-params-of-a-json-client + (let [profile (th/create-profile* 1) + session-id (upload-chunked! profile) + out (th/command! {::th/type :create-import-binfile-job + ::rpc/profile-id (:id profile) + :params {:project-id (str (:default-project-id profile)) + :name "imported"} + :upload-id session-id})] + + (t/is (th/success? out)))) + +(t/deftest create-import-binfile-job-never-takes-a-path-from-the-caller + (let [profile (th/create-profile* 1) + out (create-import-binfile-job (:id profile) (:default-project-id profile) + :file {:filename "package.penpot" + :path (jio/file "/etc/passwd") + :mtype "application/zip" + :size 0})] + + (t/testing "a plain upload is not a way to name a package" + (t/is (not (th/success? out))) + (t/is (= :params-validation (th/ex-code (:error out))))) + + (t/testing "and no job is created" + (t/is (zero? (count-jobs)))))) + +(t/deftest create-import-binfile-job-answers-with-a-pending-job + (let [profile (th/create-profile* 1) + out (create-import-binfile-job (:id profile) (:default-project-id profile) + :upload-id (upload-chunked! profile))] + + (t/is (th/success? out)) + + (let [result (:result out) + job (jobs/get-job th/*system* (:id result)) + staged (sto/get-object (:app.storage/storage th/*system*) (:resource-id job))] + + (t/testing "the answer is enough to follow the job" + (t/is (uuid? (:id result))) + (t/is (= "pending" (:status result))) + (t/is (= "import-binfile" (:name result)))) + + (t/testing "the row points at the package the job will consume" + (t/is (some? (:resource-id job))) + (t/is (= "binfile" (:queue job))) + (t/is (= sto/job-resource-bucket (:bucket (meta staged)))) + (t/is (= (:id profile) (:profile-id (meta staged))))) + + (t/testing "and freezes the destination and the manifest metadata" + (let [job-def (jobs/get-job-def (::jobs/defs th/*system*) :import-binfile) + params (jobs/decode-params job-def (:params job))] + (t/is (= (:default-project-id profile) (:project-id params))) + (t/is (= "imported" (:name params))) + (t/is (= 3 (:version params))) + (t/is (string? (:generated-by params))))) + + (t/testing "the audit points at the job and the manifest, not the package" + (let [props (::audit/props (meta (:result out)))] + (t/is (= (:id result) (:job-id props))) + (t/is (= #{:job-id :generated-by :referer} (set (keys props))))))))) + +(t/deftest create-import-binfile-job-assembles-the-chunks-of-its-upload + (let [profile (th/create-profile* 1) + session-id (upload-chunked! profile) + out (create-import-binfile-job (:id profile) (:default-project-id profile) + :upload-id session-id)] + + (t/is (th/success? out)) + + (t/testing "the assembled package is the resource of the job" + (let [job (jobs/get-job th/*system* (:id (:result out)))] + (t/is (some? (:resource-id job))))) + + (t/testing "the upload session is consumed" + (t/is (some? (:deleted-at (session-row session-id))))) + + (t/testing "and the temporary file of the assembly is gone" + (t/is (empty? (assembled-tempfiles)))))) + +(t/deftest create-import-binfile-job-rejects-an-incomplete-upload + (let [profile (th/create-profile* 1) + session-id (-> (th/command! {::th/type :create-upload-session + ::rpc/profile-id (:id profile) + :total-chunks 2}) + :result :session-id) + out (create-import-binfile-job (:id profile) (:default-project-id profile) + :upload-id session-id)] + + (t/is (not (th/success? out))) + (t/is (= :missing-chunks (th/ex-code (:error out)))) + (t/is (zero? (count-jobs))))) + +(t/deftest create-import-binfile-job-refuses-an-upload-of-another-profile + (let [owner (th/create-profile* 1) + other (th/create-profile* 2) + session-id (upload-chunked! owner) + out (create-import-binfile-job (:id other) (:default-project-id other) + :upload-id session-id)] + + (t/testing "the upload belongs to the profile that made it" + (t/is (not (th/success? out))) + (t/is (= :not-found (th/ex-type (:error out))))) + + (t/testing "and it is not consumed by the attempt" + (t/is (nil? (:deleted-at (session-row session-id))))) + + (t/testing "nor is any job created" + (t/is (zero? (count-jobs)))))) + +(t/deftest create-import-binfile-job-checks-the-edition-permission + (let [owner (th/create-profile* 1) + other (th/create-profile* 2) + session-id (upload-chunked! other) + out (create-import-binfile-job (:id other) (:default-project-id owner) + :upload-id session-id)] + + (t/testing "a job that cannot edit the destination is refused" + (t/is (not (th/success? out)))) + + (t/testing "before the upload is assembled" + (t/is (nil? (:deleted-at (session-row session-id))))) + + (t/testing "and no job is created" + (t/is (zero? (count-jobs)))))) + +(t/deftest create-import-binfile-job-refuses-a-version-that-does-not-exist + (let [profile (th/create-profile* 1) + session-id (upload-chunked! profile) + ;; the chunks of the upload are alive: the GC reclaims them later + before (live-storage-objects) + out (create-import-binfile-job (:id profile) (:default-project-id profile) + :params {:project-id (:default-project-id profile) + :name "imported" + :version 2} + :upload-id session-id)] + + (t/is (not (th/success? out))) + (t/is (= :params-validation (th/ex-code (:error out)))) + (t/is (zero? (count-jobs))) + + (t/testing "no package was staged for it" + (t/is (= before (live-storage-objects)))) + + (t/testing "and the upload is not consumed by the rejection" + (t/is (nil? (:deleted-at (session-row session-id))))) + + (t/testing "so nothing had to be assembled either" + (t/is (empty? (assembled-tempfiles)))))) + +(t/deftest create-import-binfile-job-rejects-malformed-params-before-assembling + ;; the shape of the caller params is checked before the chunks are + ;; assembled: a name over the limit never copies the package to disk + (let [profile (th/create-profile* 1) + session-id (upload-chunked! profile) + before (live-storage-objects) + out (with-redefs [cmd-jobs/assemble-upload + (fn [& _] + (throw (ex-info "assembled" {})))] + (create-import-binfile-job (:id profile) (:default-project-id profile) + :params {:project-id (:default-project-id profile) + :name (apply str (repeat 251 "x"))} + :upload-id session-id))] + + (t/is (not (th/success? out))) + (t/is (= :params-validation (th/ex-code (:error out)))) + (t/is (zero? (count-jobs))) + + (t/testing "no package was staged for it" + (t/is (= before (live-storage-objects)))) + + (t/testing "so nothing had to be assembled either" + (t/is (empty? (assembled-tempfiles)))))) + +(t/deftest create-import-binfile-job-leaves-no-temporary-when-the-package-is-corrupt + (let [profile (th/create-profile* 1) + ;; a header that says "version 3" (the zip magic) over bytes that + ;; are not a zip at all: what fails is reading the manifest, once + ;; the chunks have already been assembled + session-id (upload-chunked! profile + {:content (str "PK\u0003\u0004" "not a zip")}) + before (live-storage-objects) + out (create-import-binfile-job (:id profile) (:default-project-id profile) + :upload-id session-id)] + + (t/is (not (th/success? out))) + (t/is (zero? (count-jobs))) + + (t/testing "no package was staged" + (t/is (= before (live-storage-objects)))) + + (t/testing "and the file assembled from the chunks is gone" + (t/is (empty? (assembled-tempfiles)))))) + +(t/deftest create-import-binfile-job-releases-the-package-when-the-submit-fails + (let [profile (th/create-profile* 1) + session-id (upload-chunked! profile) + ;; the chunks of the upload are alive: the GC reclaims them later + before (live-storage-objects) + out (with-redefs [jobs/submit (fn [& _] + (ex/raise :type :internal + :code :boom + :hint "cannot create the job"))] + (create-import-binfile-job (:id profile) (:default-project-id profile) + :upload-id session-id))] + + (t/testing "the failure reaches the caller" + (t/is (not (th/success? out)))) + + (t/testing "the staged package does not stay alive" + (t/is (= before (live-storage-objects)))) + + (t/testing "and no job row was left behind" + (t/is (zero? (count-jobs)))))) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; READ +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(defn- get-job! + [profile-id job-id] + (th/command! {::th/type :get-job + ::rpc/profile-id profile-id + :id job-id})) + +(defn- claim-job! + "Drive a pending job to `running`, as the dispatcher would: the terminal + writers only accept a job that is being executed." + [job-id] + (jobs/claim th/*system* job-id + (:scheduled-at (th/db-get :job {:id job-id} :id :scheduled-at)))) + +(t/deftest get-job-answers-with-the-state-of-the-job + (let [profile (th/create-profile* 1) + file-id (first (:file-ids (import-fixture! profile))) + job-id (:id (:result (create-export-binfile-job (:id profile) #{file-id}))) + out (get-job! (:id profile) job-id)] + + (t/is (th/success? out)) + (let [result (:result out)] + (t/is (= job-id (:id result))) + (t/is (= "pending" (:status result))) + (t/is (= "export-binfile" (:name result))) + (t/is (ct/inst? (:created-at result))) + (t/is (ct/inst? (:expires-at result))) + + (t/testing "a job that has not finished has nothing to report yet" + (t/is (not (contains? result :result))) + (t/is (not (contains? result :error))))))) + +(t/deftest get-job-answers-with-the-result-of-a-finished-job + (let [profile (th/create-profile* 1) + file-id (first (:file-ids (import-fixture! profile))) + job-id (:id (:result (create-export-binfile-job (:id profile) #{file-id})))] + + (t/is (pos? (claim-job! job-id))) + (t/is (pos? (jobs/complete th/*system* :job-id job-id + :result {:value 42 :file-ids [file-id]}))) + + (let [result (:result (get-job! (:id profile) job-id))] + (t/is (= "completed" (:status result))) + (t/testing "the result comes back as the map the handler produced" + ;; the row holds plain json, so a uuid comes back as the text it is + (t/is (= {:value 42 :file-ids [(str file-id)]} (:result result)))) + (t/is (not (contains? result :error)))))) + +(t/deftest get-job-answers-with-the-error-of-a-failed-job + (let [profile (th/create-profile* 1) + file-id (first (:file-ids (import-fixture! profile))) + job-id (:id (:result (create-export-binfile-job (:id profile) #{file-id})))] + + (t/is (pos? (claim-job! job-id))) + (t/is (pos? (jobs/fail th/*system* job-id {:type :internal + :code :boom + :hint "boom"}))) + + (let [result (:result (get-job! (:id profile) job-id))] + (t/is (= "failed" (:status result))) + (t/testing "the error is the public summary, with its keywords back" + (t/is (= {:type :internal :code :boom :hint "boom"} (:error result)))) + (t/testing "and it carries no result" + (t/is (not (contains? result :result))))))) + +(t/deftest get-job-hides-a-job-of-another-profile + (let [owner (th/create-profile* 1) + other (th/create-profile* 2) + file-id (first (:file-ids (import-fixture! owner))) + job-id (:id (:result (create-export-binfile-job (:id owner) #{file-id})))] + + (t/testing "the owner reads it" + (t/is (th/success? (get-job! (:id owner) job-id)))) + + (t/testing "another profile gets the answer of a job that does not exist" + (let [out (get-job! (:id other) job-id)] + (t/is (not (th/success? out))) + (t/is (= :not-found (th/ex-type (:error out)))))) + + (t/testing "and so does an unknown id" + (let [out (get-job! (:id owner) (uuid/next))] + (t/is (not (th/success? out))) + (t/is (= :not-found (th/ex-type (:error out)))))))) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; CANCEL +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(defn- cancel-job + [profile-id job-id] + (th/command! {::th/type :cancel-job + ::rpc/profile-id profile-id + :id job-id})) + +(t/deftest cancel-job-cancels-a-running-job + (let [profile (th/create-profile* 1) + file-id (first (:file-ids (import-fixture! profile))) + job-id (:id (:result (create-export-binfile-job (:id profile) #{file-id})))] + + (t/is (pos? (claim-job! job-id))) + + (let [out (cancel-job (:id profile) job-id)] + (t/is (th/success? out)) + (t/testing "the answer names the cancelled job" + (t/is (= job-id (:id (:result out)))) + (t/is (= "cancelled" (:status (:result out))))) + (t/testing "and the row is cancelled" + (t/is (= "cancelled" (:status (jobs/get-job th/*system* job-id)))))))) + +(t/deftest cancel-job-hides-a-job-of-another-profile + (let [owner (th/create-profile* 1) + other (th/create-profile* 2) + file-id (first (:file-ids (import-fixture! owner))) + job-id (:id (:result (create-export-binfile-job (:id owner) #{file-id})))] + + (t/testing "another profile gets the answer of a job that does not exist" + (let [out (cancel-job (:id other) job-id)] + (t/is (not (th/success? out))) + (t/is (= :not-found (th/ex-type (:error out)))))) + + (t/testing "and the job keeps running untouched" + (t/is (pos? (claim-job! job-id))) + (t/is (= "running" (:status (jobs/get-job th/*system* job-id))))))) + +(t/deftest cancel-job-answers-a-finished-job-without-error + (let [profile (th/create-profile* 1) + file-id (first (:file-ids (import-fixture! profile))) + job-id (:id (:result (create-export-binfile-job (:id profile) #{file-id})))] + + (t/is (pos? (claim-job! job-id))) + (t/is (pos? (jobs/complete th/*system* :job-id job-id + :result {:value 42}))) + + (t/testing "cancelling what is already over answers its state" + (let [out (cancel-job (:id profile) job-id)] + (t/is (th/success? out)) + (t/is (= "completed" (:status (:result out)))))))) + +(t/deftest cancel-job-rejects-a-malformed-id + (let [profile (th/create-profile* 1) + out (cancel-job (:id profile) "nope")] + (t/is (not (th/success? out))) + (t/is (= :validation (th/ex-type (:error out)))))) diff --git a/backend/test/backend_tests/rpc_management_jobs_test.clj b/backend/test/backend_tests/rpc_management_jobs_test.clj index d129e9687c..9ff0882e68 100644 --- a/backend/test/backend_tests/rpc_management_jobs_test.clj +++ b/backend/test/backend_tests/rpc_management_jobs_test.clj @@ -74,7 +74,7 @@ WHERE job_id = ? AND kind = 'progress' ORDER BY created_at ASC, id ASC" id]) - (mapv #(db/decode-json-pgobject (:payload %))))) + (mapv #(jobs/decode-progress (:payload %))))) (defn- mgmt [type params] @@ -138,17 +138,20 @@ (let [job-id (mk-job {}) _ (jobs/claim (make-cfg) job-id (:scheduled-at (th/db-get :job {:id job-id} :id :scheduled-at))) out (mgmt :report-job-progress {:job-id job-id - :progress {:total 100 :current 50}})] + :progress {:stage :pages + :counters {:pages {:current 50 :total 100}}}})] (t/is (nil? (:error out))) (t/is (= {:action :run} (:result out))) - (t/is (= [{:total 100 :current 50}] (get-progresss job-id))) + (t/is (= [{:stage :pages :counters {:pages {:current 50 :total 100}}}] + (get-progresss job-id))) (t/testing "the job row has no progress column anymore" (t/is (not (contains? (get-row job-id) :progress)))))) (t/deftest report-job-progress-noop-on-terminal-row (let [job-id (mk-job {:status "completed"}) out (mgmt :report-job-progress {:job-id job-id - :progress {:total 100 :current 100}})] + :progress {:stage :pages + :counters {:pages {:current 100 :total 100}}}})] (t/is (nil? (:error out))) (t/is (= {:action :skip} (:result out))) (t/is (= [] (get-progresss job-id))))) @@ -157,25 +160,27 @@ (let [job-id (mk-job {}) _ (jobs/claim (make-cfg) job-id (:scheduled-at (th/db-get :job {:id job-id} :id :scheduled-at))) _ (mgmt :report-job-progress {:job-id job-id - :progress {:total 10 :current 3}}) + :progress {:stage :pages + :counters {:pages {:current 3 :total 10}}}}) _ (mgmt :report-job-progress {:job-id job-id - :progress {:total 10 :current 4}})] + :progress {:stage :pages + :counters {:pages {:current 4 :total 10}}}})] (t/testing "the second immediate report is not throttled away" - (t/is (= [{:total 10 :current 3} - {:total 10 :current 4}] + (t/is (= [{:stage :pages :counters {:pages {:current 3 :total 10}}} + {:stage :pages :counters {:pages {:current 4 :total 10}}}] (get-progresss job-id)))))) (t/deftest report-job-progress-validates-the-payload (let [job-id (mk-job {}) _ (jobs/claim (make-cfg) job-id (:scheduled-at (th/db-get :job {:id job-id} :id :scheduled-at)))] - (t/testing "current is mandatory" + (t/testing "stage is mandatory" (t/is (= :validation (th/ex-type (:error (mgmt :report-job-progress {:job-id job-id - :progress {:total 10}})))))) + :progress {:counters {:work {:current 1}}}})))))) (t/testing "no extra key is accepted" (t/is (= :validation (th/ex-type (:error (mgmt :report-job-progress {:job-id job-id - :progress {:current 1 + :progress {:stage :work :message "boom"}})))))) (t/is (= [] (get-progresss job-id))))) @@ -243,7 +248,7 @@ (jobs/claim cfg job-id1 (:scheduled-at (th/db-get :job {:id job-id1} :id :scheduled-at))) (jobs/claim cfg job-id2 (:scheduled-at (th/db-get :job {:id job-id2} :id :scheduled-at))) (jobs/heartbeat cfg :job-id job-id1) - (jobs/heartbeat cfg :job-id job-id2 :progress {:current 1}) + (jobs/heartbeat cfg :job-id job-id2 :progress {:stage :work}) (jobs/complete cfg :job-id job-id1) (jobs/fail cfg job-id2 test-error) (t/is (not (contains? @@#'jobs/heartbeats job-id1))) @@ -300,7 +305,7 @@ (t/is (= {:action :skip} (:result out))))) (t/testing "progress on a terminal row reports skip" (let [out (mgmt :report-job-progress {:job-id done-id - :progress {:total 10 :current 4}})] + :progress {:stage :pages :counters {:pages {:current 4 :total 10}}}})] (t/is (nil? (:error out))) (t/is (= {:action :skip} (:result out)))))) (let [cfg (make-cfg) @@ -309,7 +314,7 @@ (t/testing "live paths report run" (t/is (= {:action :run} (:result (mgmt :report-job-progress {:job-id job-id - :progress {:total 10 :current 4}})))) + :progress {:stage :pages :counters {:pages {:current 4 :total 10}}}})))) (t/is (= {:action :run} (:result (mgmt :complete-job {:job-id job-id :result {:x 1}}))))))) diff --git a/backend/test/backend_tests/rpc_quotes_test.clj b/backend/test/backend_tests/rpc_quotes_test.clj index c691ea678b..84493e1ed4 100644 --- a/backend/test/backend_tests/rpc_quotes_test.clj +++ b/backend/test/backend_tests/rpc_quotes_test.clj @@ -6,9 +6,12 @@ (ns backend-tests.rpc-quotes-test (:require + [app.common.time :as ct] [app.common.uuid :as uuid] + [app.config :as cf] [app.db :as db] [app.http :as http] + [app.jobs :as jobs] [app.rpc :as-alias rpc] [app.rpc.cond :as cond] [app.rpc.quotes :as-alias quotes] @@ -472,3 +475,78 @@ (t/is (= :restriction (th/ex-type error))) (t/is (= :max-quote-reached (th/ex-code error))) (t/is (= "media-storage-bytes-per-team" (:target (ex-data error)))))))) + +(t/deftest binfile-jobs-per-profile-quote + (with-mocks [mock {:target 'app.config/get + :return (th/config-get-mock + {:quotes-export-jobs-per-profile 2 + :quotes-import-jobs-per-profile 1})}] + + (let [profile (th/create-profile* 1) + other (th/create-profile* 2) + + insert! (fn [profile-id name status & {:keys [tenant]}] + (th/db-insert! :job {:id (uuid/next) + :name name + :tenant (or tenant (cf/get :tenant)) + :queue "binfile" + :params (db/json {}) + :priority 100 + :max-retries 0 + :retry-num 0 + :status status + :profile-id profile-id + :scheduled-at (ct/now) + :created-at (ct/now) + :modified-at (ct/now)})) + + check! (fn check! + ([id profile-id] (check! th/*system* id profile-id)) + ([cfg id profile-id] + (quotes/check! cfg {::quotes/id id + ::quotes/profile-id profile-id}))) + + error (fn [& args] + (try (apply check! args) nil (catch Throwable cause cause)))] + + (t/testing "a profile without jobs is under the quote" + (t/is (nil? (check! ::quotes/export-jobs-per-profile (:id profile))))) + + (t/testing "the jobs of another profile do not count" + (insert! (:id other) "export-binfile" "new") + (insert! (:id profile) "export-binfile" "new") + (t/is (nil? (check! ::quotes/export-jobs-per-profile (:id profile))))) + + (t/testing "the jobs of another tenant do not count either" + (insert! (:id profile) "export-binfile" "new" :tenant "another-tenant") + (t/is (nil? (check! ::quotes/export-jobs-per-profile (:id profile))))) + + (t/testing "a job that already reached a terminal state does not count" + (insert! (:id profile) "export-binfile" "completed") + (insert! (:id profile) "export-binfile" "failed") + (insert! (:id profile) "export-binfile" "cancelled") + (t/is (nil? (check! ::quotes/export-jobs-per-profile (:id profile))))) + + (t/testing "the job that would go over the limit is rejected" + (insert! (:id profile) "export-binfile" "running") + (let [cause (error ::quotes/export-jobs-per-profile (:id profile))] + (t/is (some? cause)) + (t/is (= :restriction (:type (ex-data cause)))) + (t/is (= :max-quote-reached (:code (ex-data cause)))) + (t/is (= "export-jobs-per-profile" (:target (ex-data cause)))))) + + (t/testing "the families are independent" + (t/is (nil? (check! ::quotes/import-jobs-per-profile (:id profile)))) + (insert! (:id profile) "import-binfile" "new") + (t/is (some? (error ::quotes/import-jobs-per-profile (:id profile))))) + + (t/testing "a new name of the family is covered by the registry" + (let [future (th/create-profile* 3) + defs (assoc (::jobs/defs th/*system*) + :export-assets {::jobs/name :export-assets + ::jobs/family :export}) + cfg (assoc th/*system* ::jobs/defs defs)] + (insert! (:id future) "export-assets" "new") + (t/is (nil? (check! cfg ::quotes/export-jobs-per-profile (:id future)))) + (insert! (:id future) "export-assets" "new") + (t/is (some? (error cfg ::quotes/export-jobs-per-profile (:id future))))))))) diff --git a/backend/test/backend_tests/storage_test.clj b/backend/test/backend_tests/storage_test.clj index a49d5b4717..930e059433 100644 --- a/backend/test/backend_tests/storage_test.clj +++ b/backend/test/backend_tests/storage_test.clj @@ -113,6 +113,44 @@ :content-type "text/plain"})] (t/is (not= (:id object1) (:id object2))))) +(t/deftest job-resource-objects-require-an-owner + (let [storage (-> (:app.storage/storage th/*system*) + (configure-storage-backend)) + content (sto/content "content")] + + (t/testing "an object of this bucket cannot be stored without its owner" + (t/is (thrown? Throwable + (sto/put-object! storage {::sto/content content + :bucket sto/job-resource-bucket + :content-type "text/plain"})))) + + (t/testing "and storing it with an owner still works" + (t/is (some? (sto/put-object! storage {::sto/content content + :bucket sto/job-resource-bucket + :profile-id (uuid/next) + :content-type "text/plain"})))))) + +(t/deftest job-resource-objects-are-not-deduplicated + ;; An object in this bucket belongs to one profile: sharing a blob + ;; between two owners would serve one user's artifact to another. + (let [storage (-> (:app.storage/storage th/*system*) + (configure-storage-backend)) + content (-> (sto/content "content") + (sto/wrap-with-hash "same-hash")) + object1 (sto/put-object! storage {::sto/content content + ::sto/deduplicate? true + ::sto/touched-at (ct/now) + :bucket sto/job-resource-bucket + :profile-id (uuid/next) + :content-type "text/plain"}) + object2 (sto/put-object! storage {::sto/content content + ::sto/deduplicate? true + ::sto/touched-at (ct/now) + :bucket sto/job-resource-bucket + :profile-id (uuid/next) + :content-type "text/plain"})] + (t/is (not= (:id object1) (:id object2))))) + (t/deftest put-and-retrieve-expired-object (let [storage (-> (:app.storage/storage th/*system*) (configure-storage-backend)) @@ -1271,6 +1309,7 @@ job-id (uuid/next) object (sto/put-object! storage {::sto/content (sto/content "content") :content-type "text/plain" + :profile-id (uuid/next) :bucket sto/job-resource-bucket})] ;; the object is created with the job-resource bucket in metadata @@ -1319,6 +1358,7 @@ ;; a touched object with no referencing job row at all is deleted too (let [object (sto/put-object! storage {::sto/content (sto/content "content") :content-type "text/plain" + :profile-id (uuid/next) :bucket sto/job-resource-bucket})] (th/db-update! :storage-object {:touched-at (ct/now)} {:id (:id object)}) (let [res (binding [ct/*clock* (ct/fixed-clock (ct/in-future {:hours 3}))] diff --git a/backend/test/backend_tests/tasks/binfile_jobs_integration_test.clj b/backend/test/backend_tests/tasks/binfile_jobs_integration_test.clj new file mode 100644 index 0000000000..54a6372163 --- /dev/null +++ b/backend/test/backend_tests/tasks/binfile_jobs_integration_test.clj @@ -0,0 +1,268 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns backend-tests.tasks.binfile-jobs-integration-test + "The whole chain of a binfile job, without a full end-to-end: the + command creates the job, the dispatcher sends it to its own queue, the + runner of that queue executes it and the artifact it produced (or + consumed) ends up where the ledger says." + (:require + [app.binfile.common :as bfc] + [app.binfile.jobs :as bfj] + [app.binfile.v3 :as v3] + [app.common.time :as ct] + [app.common.uuid :as uuid] + [app.config :as cf] + [app.db :as db] + [app.jobs :as jobs] + [app.metrics :as-alias mtx] + [app.msgbus :as-alias mbus] + [app.redis :as rds] + [app.rpc :as-alias rpc] + [app.storage :as sto] + [app.storage.tmp :as tmp] + [app.worker :as wrk] + [app.worker.dispatcher :as wdisp] + [app.worker.runner :as wrkr] + [backend-tests.helpers :as th] + [clojure.java.io :as jio] + [clojure.test :as t] + [datoteka.io :as io])) + +(t/use-fixtures :once th/state-init) +(t/use-fixtures :each (th/serial th/database-reset th/clean-storage)) + +(def ^:private fixture + "A real penpot export, the same fixture the binfile tests read." + "backend_tests/test_files/svg-attrs-camel-case.penpot") + +(def ^:private fast-timeout + "A runner of an empty queue must not make the test wait for the default + timeout of the production loop." + (ct/duration {:millis 200})) + +(defn- get-storage + [] + (assoc (:app.storage/storage th/*system*) ::sto/backend :fs)) + +(defn- import-fixture! + "A file in the default project of the profile, imported from the + fixture through the core." + [profile] + (-> th/*system* + (assoc ::bfc/project-id (:default-project-id profile)) + (assoc ::bfc/profile-id (:id profile)) + (assoc ::bfc/team-id (:default-team-id profile)) + (assoc ::bfc/input (-> fixture io/resource jio/file)) + (v3/import-files!))) + +(defn- upload-chunked! + "Upload the fixture as a single chunk and return the session id." + [profile] + (let [source (-> fixture io/resource jio/file) + session-id (-> (th/command! {::th/type :create-upload-session + ::rpc/profile-id (:id profile) + :total-chunks 1}) + :result :session-id) + out (th/command! {::th/type :upload-chunk + ::rpc/profile-id (:id profile) + :session-id session-id + :index 0 + :content {:filename "package.penpot" + :path (th/tempfile fixture) + :mtype "application/zip" + :size (.length source)}})] + (t/is (nil? (:error out))) + session-id)) + +(defn- dispatcher-cfg + [] + {::db/pool th/*pool* + ::rds/client (get th/*system* :app.redis/client) + ::mtx/metrics (get th/*system* :app.metrics/metrics) + ::wrk/tenant (cf/get :tenant) + ::wdisp/lease (cf/get-jobs-lease) + ::wdisp/timeout (ct/duration "10s")}) + +(defn- runner-cfg + "The cfg the runner loop needs, with the real job-defs of the system: + the handler closes over the component of its own job-def." + [queue] + {::db/pool th/*pool* + ::rds/conn (rds/connect {::rds/client (get th/*system* :app.redis/client) + ::mtx/metrics (get th/*system* :app.metrics/metrics)}) + ::jobs/defs (::jobs/defs th/*system*) + ::mtx/metrics (get th/*system* :app.metrics/metrics) + ::mbus/msgbus (get th/*system* :app.msgbus/msgbus) + ::wrkr/id "test-runner" + ::wrkr/queue queue + ::wrk/tenant (cf/get :tenant) + ::wrkr/timeout fast-timeout}) + +(defn- run-one + [cfg] + (@#'wrkr/run-worker-loop cfg)) + +(defn- job-row + [job-id] + (-> (th/db-get :job {:id job-id}) + (update :result #(cond-> % (db/pgobject? %) db/decode-json-pgobject)) + (update :error jobs/decode-job-error))) + +(defn- project-files + [project-id] + (th/db-exec! ["SELECT id FROM file WHERE project_id = ? AND deleted_at IS NULL" + project-id])) + +(defn- download + "The artifact as a local file, as a client would fetch it." + [resource-id] + (let [object (sto/get-object (get-storage) resource-id) + path (tmp/tempfile* :suffix ".penpot")] + (with-open [out (io/output-stream path)] + (io/copy (sto/get-object-data (get-storage) object) out)) + path)) + +(defn- create-export-binfile-job + [profile file-ids] + (th/command! {::th/type :create-export-binfile-job + ::rpc/profile-id (:id profile) + :params {:file-ids file-ids + :export-type :detach-libraries}})) + +(defn- create-import-binfile-job + [profile project-id] + (th/command! {::th/type :create-import-binfile-job + ::rpc/profile-id (:id profile) + :params {:project-id project-id + :name "imported"} + :upload-id (upload-chunked! profile)})) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; TESTS +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(t/deftest the-export-job-runs-to-completion-in-its-own-queue + (let [profile (th/create-profile* 1) + file-id (first (:file-ids (import-fixture! profile))) + out (create-export-binfile-job profile #{file-id}) + job-id (:id (:result out))] + + (t/is (th/success? out)) + + (wdisp/run-batch (dispatcher-cfg)) + (t/testing "the dispatcher schedules it, and it waits for its queue" + (t/is (= "scheduled" (:status (jobs/get-job th/*system* job-id))))) + + (t/testing "a runner of another queue does not take it" + (run-one (runner-cfg "default")) + (t/is (= "scheduled" (:status (jobs/get-job th/*system* job-id))))) + + (t/testing "the runner of its queue does" + (run-one (runner-cfg "binfile")) + (t/is (= "completed" (:status (jobs/get-job th/*system* job-id))))) + + (let [row (job-row job-id) + job (:result row)] + (t/testing "the runner associated the artifact of the completion envelope" + (t/is (some? (:resource-id row)))) + + (t/testing "the result of the job is the descriptor the handler built" + (t/is (= "export.penpot" (:filename job))) + (t/is (= "application/zip" (:mtype job))) + (t/is (pos? (:size job))) + (t/is (some? (:resource-uri job)))) + + (t/testing "and the artifact is the package of those files" + (t/is (= [file-id] + (mapv :id (:files (v3/get-manifest th/*system* + (download (:resource-id row)))))))) + + (t/testing "the artifact belongs to the profile of the job" + (let [object (sto/get-object (get-storage) (:resource-id row))] + (t/is (= sto/job-resource-bucket (:bucket (meta object)))) + (t/is (= (:id profile) (:profile-id (meta object)))))) + + (t/testing "the runner reached the terminal state through the row" + (t/is (= "completed" (:status row))) + (t/is (some? (:completed-at row))))))) + +(t/deftest the-import-job-runs-to-completion-in-its-own-queue + (let [profile (th/create-profile* 1) + project-id (:default-project-id profile) + out (create-import-binfile-job profile project-id) + job-id (:id (:result out)) + ;; read before the run: the handler drops the reference as soon as + ;; it releases the package + input-id (:resource-id (th/db-get :job {:id job-id}))] + + (t/is (th/success? out)) + + (wdisp/run-batch (dispatcher-cfg)) + (run-one (runner-cfg "binfile")) + + (let [row (job-row job-id) + job (:result row)] + (t/testing "the job ends completed" + (t/is (= "completed" (:status row)))) + + (t/testing "the files of the package are in the destination" + ;; the result is JSON: its ids come back as text, the rows as uuids + (t/is (= (set (map str (:file-ids job))) + (set (map (comp str :id) (project-files project-id)))))) + + (t/testing "the result carries the files and their resolution" + (t/is (pos? (count (:file-ids job)))) + (t/is (= "imported" (:name job)))) + + (t/testing "the package it consumed is released" + (t/is (some? (:deleted-at (th/db-get :storage-object + {:id input-id} + {::db/remove-deleted false}))))) + + (t/testing "and the job no longer points at it" + (t/is (nil? (:resource-id row))))))) + +(t/deftest the-adapter-reports-progress-through-heartbeat + (let [profile (th/create-profile* 1) + file-id (first (:file-ids (import-fixture! profile))) + job-id (uuid/next) + calls (atom [])] + + (th/db-insert! :job {:id job-id + :name "export-binfile" + :tenant (cf/get :tenant) + :queue "binfile" + :params (db/json {}) + :priority 100 + :max-retries 0 + :retry-num 0 + :status "running" + :profile-id (:id profile) + :scheduled-at (ct/now) + :created-at (ct/now) + :modified-at (ct/now)}) + + (with-redefs [jobs/heartbeat (fn [_cfg & {:as options}] + (swap! calls conj options) + 1)] + (bfj/export-files th/*system* + (jobs/make-context (jobs/get-job th/*system* job-id)) + {:ids #{file-id} + :export-type :detach-libraries + :output (tmp/tempfile* :suffix ".penpot")})) + + (let [reports (map :progress @calls)] + + (t/testing "the adapter reports progress through heartbeat" + (t/is (pos? (count reports)))) + + (t/testing "with the payload of the job contract, and nothing else" + (t/is (every? keyword? (map :stage reports))) + (t/is (every? #(<= (count (keys %)) 2) reports))) + + (t/testing "about the job of the context, not one the caller named" + (t/is (every? #(= job-id (:job-id %)) @calls)))))) diff --git a/backend/test/backend_tests/tasks/export_binfile_test.clj b/backend/test/backend_tests/tasks/export_binfile_test.clj new file mode 100644 index 0000000000..80fcdedd36 --- /dev/null +++ b/backend/test/backend_tests/tasks/export_binfile_test.clj @@ -0,0 +1,193 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns backend-tests.tasks.export-binfile-test + "The `:export-binfile` job: what it produces, who may ask for it and + what it leaves behind when it fails." + (:require + [app.binfile.common :as bfc] + [app.binfile.v3 :as v3] + [app.common.exceptions :as ex] + [app.common.time :as ct] + [app.common.uuid :as uuid] + [app.config :as cf] + [app.db :as db] + [app.jobs :as jobs] + [app.storage :as sto] + [app.storage.tmp :as tmp] + [backend-tests.helpers :as th] + [clojure.java.io :as jio] + [clojure.test :as t] + [cuerdas.core :as str] + [datoteka.io :as io])) + +(t/use-fixtures :once th/state-init) +(t/use-fixtures :each (th/serial th/database-reset th/clean-storage)) + +(def ^:private fixture + "A real penpot export, the same fixture the binfile tests read." + "backend_tests/test_files/svg-attrs-camel-case.penpot") + +(defn- get-storage + [] + (assoc (:app.storage/storage th/*system*) ::sto/backend :fs)) + +(defn- import-fixture! + "A file in the default project of the profile, imported from the + fixture through the core." + [profile] + (-> th/*system* + (assoc ::bfc/project-id (:default-project-id profile)) + (assoc ::bfc/profile-id (:id profile)) + (assoc ::bfc/team-id (:default-team-id profile)) + (assoc ::bfc/input (-> fixture io/resource jio/file)) + (v3/import-files!))) + +(defn- make-job! + "A job row as the substrate writes it: running and owned by a profile." + [profile-id] + (let [id (uuid/next)] + (th/db-insert! :job {:id id + :name "export-binfile" + :tenant (cf/get :tenant) + :queue "binfile" + :params (db/json {}) + :priority 100 + :max-retries 0 + :retry-num 0 + :status "running" + :profile-id profile-id + :scheduled-at (ct/now) + :created-at (ct/now) + :modified-at (ct/now)}) + id)) + +(defn- run-export + "Run the export handler of a job, as the runner does." + [job-id params] + (jobs/invoke (-> th/*system* + (assoc ::jobs/name :export-binfile) + (assoc ::jobs/params params) + (assoc ::jobs/context + (jobs/make-context (jobs/get-job th/*system* job-id)))))) + +(defn- count-storage-objects + [] + (count (th/db-exec! ["SELECT id FROM storage_object"]))) + +(defn- temp-files + "The temporary files the export leaves in the tmp directory." + [] + (seq (.list (jio/file tmp/default-tmp-dir)))) + +(defn- download + "The artifact as a local file, as a client would fetch it." + [resource-id] + (let [object (sto/get-object (get-storage) resource-id) + path (tmp/tempfile* :suffix ".penpot")] + (with-open [out (io/output-stream path)] + (io/copy (sto/get-object-data (get-storage) object) out)) + path)) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; TESTS +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(t/deftest the-export-handler-stores-the-artifact-of-its-job + (let [profile (th/create-profile* 1) + file-id (first (:file-ids (import-fixture! profile))) + job-id (make-job! (:id profile)) + result (run-export job-id {:file-ids #{file-id} + :export-type :detach-libraries}) + object (sto/get-object (get-storage) (:resource-id result))] + + (t/testing "the handler answers with the completion envelope" + (t/is (uuid? (:resource-id result))) + (t/is (= "export.penpot" (get-in result [:result :filename]))) + (t/is (= "application/zip" (get-in result [:result :mtype]))) + (t/is (pos? (get-in result [:result :size]))) + (t/is (str/includes? (get-in result [:result :resource-uri]) + (str (:resource-id result))))) + + (t/testing "the artifact is a job resource owned by the profile of the job" + (t/is (= sto/job-resource-bucket (:bucket (meta object)))) + (t/is (= (:id profile) (:profile-id (meta object))))) + + (t/testing "it is touched, so a crash before the completion leaves it reclaimable" + (t/is (some? (:touched-at (th/db-get :storage-object + {:id (:resource-id result)}))))) + + (t/testing "the package is the one the core produces for those files" + (let [manifest (v3/get-manifest th/*system* (download (:resource-id result)))] + (t/is (= [file-id] (mapv :id (:files manifest)))))) + + (t/testing "the completion of the job associates the artifact" + (jobs/complete th/*system* + :job-id job-id + :result (:result result) + :resource-id (:resource-id result)) + (t/is (= (:resource-id result) + (:resource-id (jobs/get-job th/*system* job-id))))))) + +(t/deftest the-export-handler-revalidates-the-read-permission + (let [owner (th/create-profile* 1) + other (th/create-profile* 2) + file-id (first (:file-ids (import-fixture! owner))) + job-id (make-job! (:id other)) + before (count-storage-objects)] + + (t/testing "a job whose owner cannot read a file does not export it" + (t/is (thrown? Throwable + (run-export job-id {:file-ids #{file-id} + :export-type :detach-libraries})))) + + (t/testing "and nothing was stored for it" + (t/is (= before (count-storage-objects))) + (t/is (nil? (:resource-id (jobs/get-job th/*system* job-id))))) + + (t/testing "the temporary file is not left behind either" + (t/is (empty? (temp-files)))))) + +(t/deftest the-export-handler-leaves-nothing-when-the-storage-fails + (let [profile (th/create-profile* 1) + file-id (first (:file-ids (import-fixture! profile))) + job-id (make-job! (:id profile)) + before (count-storage-objects)] + + (with-redefs [sto/put-object! (fn [& _] + (ex/raise :type :internal + :code :boom + :hint "storage is down"))] + (t/is (thrown? Throwable + (run-export job-id {:file-ids #{file-id} + :export-type :detach-libraries})))) + + (t/testing "the job never comes to own an artifact" + (t/is (nil? (:resource-id (jobs/get-job th/*system* job-id))))) + + (t/testing "no artifact was stored" + (t/is (= before (count-storage-objects)))) + + (t/testing "and the temporary file of the export is gone" + (t/is (empty? (temp-files)))))) + +(t/deftest the-export-handler-leaves-nothing-behind-when-the-job-is-cancelled + (let [profile (th/create-profile* 1) + file-id (first (:file-ids (import-fixture! profile))) + job-id (make-job! (:id profile)) + before (count-storage-objects)] + + (th/db-update! :job {:status "cancelled"} {:id job-id}) + + (t/is (thrown? Throwable + (run-export job-id {:file-ids #{file-id} + :export-type :detach-libraries}))) + + (t/testing "no artifact was stored" + (t/is (= before (count-storage-objects)))) + + (t/testing "and the temporary file of the export was cleaned up" + (t/is (empty? (temp-files)))))) diff --git a/backend/test/backend_tests/tasks/import_binfile_test.clj b/backend/test/backend_tests/tasks/import_binfile_test.clj new file mode 100644 index 0000000000..e2c619177f --- /dev/null +++ b/backend/test/backend_tests/tasks/import_binfile_test.clj @@ -0,0 +1,254 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns backend-tests.tasks.import-binfile-test + "The `:import-binfile` job: what it imports, who may ask for it and what + it leaves behind when it fails." + (:require + [app.binfile.common :as bfc] + [app.binfile.v1 :as v1] + [app.binfile.v3 :as v3] + [app.common.time :as ct] + [app.common.uuid :as uuid] + [app.config :as cf] + [app.db :as db] + [app.jobs :as jobs] + [app.jobs.storage :as js] + [app.storage :as sto] + [app.storage.tmp :as tmp] + [backend-tests.helpers :as th] + [clojure.java.io :as jio] + [clojure.test :as t] + [datoteka.io :as io])) + +(t/use-fixtures :once th/state-init) +(t/use-fixtures :each (th/serial th/database-reset th/clean-storage)) + +(def ^:private fixture + "A real penpot export, the same fixture the binfile tests read." + "backend_tests/test_files/svg-attrs-camel-case.penpot") + +(defn- fixture-path + [] + (-> fixture io/resource jio/file)) + +(defn- import-fixture! + "A file in the default project of the profile, imported from the + fixture through the core." + [profile] + (-> th/*system* + (assoc ::bfc/project-id (:default-project-id profile)) + (assoc ::bfc/profile-id (:id profile)) + (assoc ::bfc/team-id (:default-team-id profile)) + (assoc ::bfc/input (fixture-path)) + (v3/import-files!))) + +(defn- export-to-v1! + "A version 1 package of a file, the format the old exporter writes." + [file-id] + (let [output (tmp/tempfile* :suffix ".penpot")] + (v1/export-files! (-> th/*system* + (assoc ::bfc/ids #{file-id}) + (assoc ::bfc/include-libraries false) + (assoc ::bfc/embed-assets false)) + output) + output)) + +(defn- stage-package! + "The package a job consumes, as its resource." + [profile path] + (js/put-resource th/*system* (:id profile) + {:content (sto/content path) + :filename "package.penpot" + :mtype "application/zip"})) + +(defn- make-job! + "A job row as the substrate writes it: running, owned by a profile and + holding the package it consumes." + [profile-id resource-id] + (let [id (uuid/next)] + (th/db-insert! :job {:id id + :name "import-binfile" + :tenant (cf/get :tenant) + :queue "binfile" + :params (db/json {}) + :priority 100 + :max-retries 0 + :retry-num 0 + :status "running" + :profile-id profile-id + :resource-id resource-id + :scheduled-at (ct/now) + :created-at (ct/now) + :modified-at (ct/now)}) + id)) + +(defn- run-import + "Run the import handler of a job, as the runner does." + [job-id params] + (jobs/invoke (-> th/*system* + (assoc ::jobs/name :import-binfile) + (assoc ::jobs/params params) + (assoc ::jobs/context + (jobs/make-context (jobs/get-job th/*system* job-id)))))) + +(defn- project-files + [project-id] + (th/db-exec! ["SELECT id FROM file WHERE project_id = ? AND deleted_at IS NULL" + project-id])) + +(defn- input-row + [resource-id] + (th/db-get :storage-object {:id resource-id} {::db/remove-deleted false})) + +(defn- temp-files + [] + (seq (.list (jio/file tmp/default-tmp-dir)))) + +(defn- project-modified-at + [project-id] + (:modified-at (th/db-get :project {:id project-id}))) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; TESTS +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(t/deftest the-import-handler-imports-the-package-of-its-job + (let [profile (th/create-profile* 1) + staged (stage-package! profile (fixture-path)) + job-id (make-job! (:id profile) (:resource-id staged)) + result (run-import job-id {:project-id (:default-project-id profile) + :name "imported" + :version 3})] + + (t/testing "the handler answers with the files it created" + (t/is (= 1 (count (:file-ids result)))) + (t/is (= "imported" (:name result))) + (t/is (= 3 (:version result))) + (t/testing "and the file is in the destination project" + (t/is (= (set (:file-ids result)) + (set (map :id (project-files (:default-project-id profile)))))))) + + (t/testing "the package of the job is released once the job is terminal" + (t/is (some? (:deleted-at (input-row (:resource-id staged)))))) + + (t/testing "and its temporary copy is gone" + (t/is (empty? (temp-files)))))) + +(t/deftest the-import-marks-the-destination-project-as-modified + (let [profile (th/create-profile* 1) + project-id (:default-project-id profile) + staged (stage-package! profile (fixture-path)) + job-id (make-job! (:id profile) (:resource-id staged)) + before (project-modified-at project-id)] + + (run-import job-id {:project-id project-id + :name "imported" + :version 3}) + + (t/testing "the project the import wrote into is not left looking untouched" + (t/is (pos? (- (inst-ms (project-modified-at project-id)) + (inst-ms before))))))) + +(t/deftest the-import-handler-imports-a-version-1-package + (let [profile (th/create-profile* 1) + file-id (first (:file-ids (import-fixture! profile))) + staged (stage-package! profile (export-to-v1! file-id)) + job-id (make-job! (:id profile) (:resource-id staged)) + result (run-import job-id {:project-id (:default-project-id profile) + :name "imported-v1" + :version 1})] + + (t/is (= 1 (count (:file-ids result)))) + (t/is (= 1 (:version result))))) + +(t/deftest the-import-handler-revalidates-the-edit-permission + (let [owner (th/create-profile* 1) + other (th/create-profile* 2) + staged (stage-package! other (fixture-path)) + ;; the package belongs to `other`, but the destination project + ;; belongs to `owner`: an import there is not allowed + job-id (make-job! (:id other) (:resource-id staged)) + before (project-files (:default-project-id owner))] + + (t/testing "a job that cannot edit the destination does not import it" + (t/is (thrown? Throwable + (run-import job-id {:project-id (:default-project-id owner) + :name "imported" + :version 3})))) + + (t/testing "nothing was written in the destination" + (t/is (= before (project-files (:default-project-id owner))))) + + (t/testing "the package is released anyway: the job is terminal" + (t/is (some? (:deleted-at (input-row (:resource-id staged)))))))) + +(t/deftest a-job-that-was-already-cancelled-never-imports + (let [profile (th/create-profile* 1) + staged (stage-package! profile (fixture-path)) + job-id (make-job! (:id profile) (:resource-id staged)) + before (project-files (:default-project-id profile))] + + (th/db-update! :job {:status "cancelled"} {:id job-id}) + + (t/testing "a cancelled job does not import anything" + (t/is (thrown? Throwable + (run-import job-id {:project-id (:default-project-id profile) + :name "imported" + :version 3}))) + (t/is (= before (project-files (:default-project-id profile))))) + + (t/testing "and the package is released" + (t/is (some? (:deleted-at (input-row (:resource-id staged)))))))) + +(t/deftest an-import-cancelled-while-it-runs-leaves-nothing-behind + (let [profile (th/create-profile* 1) + staged (stage-package! profile (fixture-path)) + job-id (make-job! (:id profile) (:resource-id staged)) + before (project-files (:default-project-id profile))] + + ;; the user cancels while the import is running: the first progress + ;; report is a good moment for it, because the core is already + ;; writing by then + (with-redefs [jobs/heartbeat (fn [& _] + (th/db-update! :job {:status "cancelled"} {:id job-id}) + 1)] + (t/is (thrown? Throwable + (run-import job-id {:project-id (:default-project-id profile) + :name "imported" + :version 3})))) + + (t/testing "the import is rolled back: the destination keeps no files" + (t/is (= before (project-files (:default-project-id profile))))) + + (t/testing "and the package is released anyway" + (t/is (some? (:deleted-at (input-row (:resource-id staged)))))))) + +(t/deftest a-package-that-is-not-a-penpot-file-fails-without-writing + (let [profile (th/create-profile* 1) + staged (stage-package! profile (fixture-path)) + job-id (make-job! (:id profile) (:resource-id staged)) + before (project-files (:default-project-id profile))] + + ;; the bytes of the resource are replaced by something that is not a + ;; package at all: the core must fail instead of importing garbage + (sto/del-object! (:app.storage/storage th/*system*) (:resource-id staged)) + (let [broken (js/put-resource th/*system* (:id profile) + {:content (sto/content "not a penpot package") + :filename "package.penpot" + :mtype "application/zip"})] + (th/db-update! :job {:resource-id (:resource-id broken)} {:id job-id}) + + (t/is (thrown? Throwable + (run-import job-id {:project-id (:default-project-id profile) + :name "imported" + :version 3}))) + + (t/testing "nothing was imported" + (t/is (= before (project-files (:default-project-id profile))))) + + (t/testing "and the broken package is released too" + (t/is (some? (:deleted-at (input-row (:resource-id broken))))))))) diff --git a/backend/test/backend_tests/util_events_test.clj b/backend/test/backend_tests/util_events_test.clj new file mode 100644 index 0000000000..9c75d32699 --- /dev/null +++ b/backend/test/backend_tests/util_events_test.clj @@ -0,0 +1,77 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns backend-tests.util-events-test + "The events subsystem: a tap delivers to whatever sink is bound, a + function inline on the same thread or a channel for a listener of its + own, and untracked runs pay nothing." + (:require + [app.util.events :as events] + [clojure.test :as t] + [promesa.exec :as px] + [promesa.exec.csp :as sp])) + +(t/deftest tap-outside-a-run-is-a-no-op + (t/testing "a tap with no sink bound does nothing" + (t/is (nil? (events/tap :one 1))))) + +(t/deftest tap-calls-a-function-sink-inline + (let [seen (atom [])] + (binding [events/*sink* (fn [event] (swap! seen conj event))] + (t/testing "every tap reaches the function, in order" + (t/is (nil? (events/tap :one 1))) + (t/is (nil? (events/tap :two 2))) + (t/is (= [[:one 1] [:two 2]] @seen)))))) + +(t/deftest tap-puts-on-a-channel-sink + (let [channel (sp/chan :buf 4)] + (try + (binding [events/*sink* channel] + (t/testing "a tap puts the event on the channel" + (t/is (nil? (events/tap :one 1))) + (t/is (= [:one 1] (sp/take! channel))))) + (finally + (sp/close! channel))))) + +(t/deftest an-exception-from-a-function-sink-propagates + (let [cause (ex-info "the job is no longer active" + {:type :interrupt :code :job-interrupted})] + (binding [events/*sink* (fn [_event] (throw cause))] + (t/testing "the tap raises the very exception the sink threw" + (t/is (identical? cause + (try (events/tap :one 1) nil + (catch Throwable raised raised)))))))) + +(t/deftest spawn-listener-consumes-until-the-channel-closes + (let [seen (atom []) + ended (promise) + channel (sp/chan :buf 4) + listener (events/spawn-listener + channel + (fn [event] (swap! seen conj event)) + (fn [] (deliver ended true)))] + (sp/put! channel [:one 1]) + (sp/put! channel [:two 2]) + (sp/close! channel) + (px/await! listener) + (t/testing "the listener saw every event, in order" + (t/is (= [[:one 1] [:two 2]] @seen))) + (t/testing "and the close hook ran" + (t/is (true? (deref ended 1000 false)))))) + +(t/deftest spawn-listener-stops-on-a-consumer-failure + (let [seen (atom []) + channel (sp/chan :buf 4) + listener (events/spawn-listener + channel + (fn [_event] (throw (ex-info "boom" {}))) + (fn [] nil))] + (sp/put! channel [:one 1]) + (sp/put! channel [:two 2]) + (px/await! listener) + (t/testing "the failure stops the listener, so nothing is consumed" + (t/is (= [] @seen))) + (sp/close! channel))) diff --git a/backend/test/backend_tests/worker_runner_test.clj b/backend/test/backend_tests/worker_runner_test.clj index 415d444ab8..2df99415a3 100644 --- a/backend/test/backend_tests/worker_runner_test.clj +++ b/backend/test/backend_tests/worker_runner_test.clj @@ -221,14 +221,15 @@ (push-payload job-id scheduled-at) (run-one (mk-cfg {})) - (t/testing "the handler receives exactly the four context keys" + (t/testing "the handler receives exactly the five context keys" (t/is (= 1 (count @received-contexts))) (let [context (first @received-contexts)] - (t/is (= #{:id :name :label :resource-id} (set (keys context)))) + (t/is (= #{:id :name :label :resource-id :profile-id} (set (keys context)))) (t/is (= job-id (:id context))) (t/is (= "echo-runner" (:name context))) (t/is (= "runner-label" (:label context))) (t/is (nil? (:resource-id context))) + (t/is (nil? (:profile-id context)) "an internal job has no owner") (t/testing "and nothing of the row leaks into it" (t/is (not (contains? context :params))) diff --git a/backend/test/backend_tests/worker_test.clj b/backend/test/backend_tests/worker_test.clj index 75f87be0c4..02f525820d 100644 --- a/backend/test/backend_tests/worker_test.clj +++ b/backend/test/backend_tests/worker_test.clj @@ -7,6 +7,11 @@ (ns backend-tests.worker-test (:require [app.common.schema :as sm] + [app.config :as cf] + [app.db :as db] + [app.jobs :as jobs] + [app.main :as main] + [app.metrics :as-alias mtx] [app.msgbus :as mbus] [app.worker :as wrk] [app.worker.cron :as wcron] @@ -44,3 +49,20 @@ @#'wdisp/schema:dispatcher @#'wrkr/schema:params]] (t/is (sm/schema? (sm/schema s)))))) + +(t/deftest the-binfile-runner-is-wired + (let [cfg (get main/worker-config [:app.main/binfile ::wrk/runner])] + + (t/is (some? cfg)) + + (t/testing "the heavy work of a user job does not share the default runner" + (t/is (= :binfile (::wrk/queue cfg))) + (t/is (= (cf/get :worker-binfile-parallelism 1) (::wrk/parallelism cfg)))) + + (t/testing "a job of a profile cannot store its events without a msgbus" + (t/is (contains? cfg ::mbus/msgbus))) + + (t/testing "and the runner resolves jobs, metrics and the pool" + (t/is (contains? cfg ::jobs/defs)) + (t/is (contains? cfg ::mtx/metrics)) + (t/is (contains? cfg ::db/pool))))) diff --git a/frontend/src/app/main/data/exports/files.cljs b/frontend/src/app/main/data/exports/files.cljs index d50948e26e..5758f38ee4 100644 --- a/frontend/src/app/main/data/exports/files.cljs +++ b/frontend/src/app/main/data/exports/files.cljs @@ -10,9 +10,10 @@ [app.common.data :as d] [app.common.schema :as sm] [app.main.data.event :as ev] + [app.main.data.jobs :as dj] [app.main.data.modal :as modal] [app.main.repo :as rp] - [app.util.sse :as sse] + [app.main.store :as st] [beicon.v2.core :as rx] [potok.v2.core :as ptk])) @@ -59,28 +60,73 @@ :team-id team-id :files files})))))))))) +(defn- export-file + "One file, one job: the file is exported on its own, so a failure does + not touch the others, and the artifact is read from the result of the + job once it is over. + + Every step of the job becomes a message for the caller: the milestone + under `:progress`, the artifact as `:uri` when the job completed, and + the public error of the job when it failed. The queue of a file is + announced by `export-files` before any job exists and only ends with a + milestone: a worker picking the job up is not one. + + `on-job` is an optional callback invoked with `{:job-id ... :file-id + ...}` for the created job, so the caller can track it per file and + cancel it while it runs." + [ws-conn type file on-job] + (->> (rp/cmd! :create-export-binfile-job + {:params {:file-ids #{(:id file)} + :export-type type}}) + (rx/mapcat (fn [{job-id :id}] + (when (fn? on-job) + (on-job {:job-id job-id :file-id (:id file)})) + (->> (dj/watch-job ws-conn job-id) + (rx/mapcat (fn [{:keys [kind status result error] :as emission}] + (cond + (= "completed" status) + (rx/of {:file-id (:id file) + :uri (:resource-uri result) + :filename (:name file)}) + + (= "failed" status) + (rx/of {:file-id (:id file) + :error error}) + + (= "cancelled" status) + (rx/of {:file-id (:id file) + :cancelled true}) + + (= :progress kind) + (rx/of {:file-id (:id file) + :progress (:payload emission)}) + + :else + (rx/empty))))))) + (rx/catch (fn [cause] + (rx/of {:file-id (:id file) + :error (ex-data cause)}))))) + (defn export-files - "Start files exportation process" - [& {:keys [type files]}] + "Start files exportation process. + + Every file is queued up front, even the ones whose job is only created + once the previous one is over, so a multiple export shows all of them + waiting from its first moment. + + The optional `:on-job` callback is invoked with `{:job-id ... + :file-id ...}` for every file job created, so the caller can track + them per file and cancel them while they run." + [& {:keys [type files on-job]}] (assert (check-export-files files) "expected a sequence of files") (assert (valid-types type) "expected valid export type") - (->> (rx/from files) - (rx/mapcat - (fn [file] - (->> (rp/cmd! ::sse/export-binfile {:file-id (:id file) - :version 3 - :type type}) - (rx/filter sse/end-of-stream?) - (rx/map sse/get-payload) - (rx/map (fn [uri] - {:file-id (:id file) - :uri uri - :filename (:name file)})) - (rx/catch (fn [cause] - (let [error (ex-data cause)] - (rx/of {:file-id (:id file) - :error error}))))))))) + (let [ws-conn (:ws-conn @st/state)] + (rx/concat + (rx/from (map (fn [file] {:file-id (:id file) :queued true}) files)) + (->> (rx/from files) + (rx/mapcat (fn [file] + (export-file ws-conn type file on-job))))))) ;;;;;;;;;;;;;;;;;;;;;; ;; Team Request diff --git a/frontend/src/app/main/data/imports.cljs b/frontend/src/app/main/data/imports.cljs new file mode 100644 index 0000000000..44cdd681d3 --- /dev/null +++ b/frontend/src/app/main/data/imports.cljs @@ -0,0 +1,294 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns app.main.data.imports + "Reading a penpot package and importing it as a job. + + Both steps run in the main thread: reading the manifest of the package + is cheap (it scales with the number of entries, not with the size of the + file) and the work itself belongs to the server, so the only thing the + client has to be near is the websocket that reports the progress of the + job." + (:require + [app.common.exceptions :as ex] + [app.common.json :as json] + [app.common.schema :as sm] + [app.common.uuid :as uuid] + [app.main.data.jobs :as dj] + [app.main.data.uploads :as uploads] + [app.main.repo :as rp] + [app.main.store :as st] + [app.util.http :as http] + [app.util.i18n :as i18n :refer [tr]] + [app.util.zip :as uz] + [beicon.v2.core :as rx] + [cuerdas.core :as str])) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; MESSAGES OF A FAILURE +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(defn- resolve-hint + "The hint or the explanation of an error, trimmed, or nil." + [error] + (let [data (if (map? error) error (ex-data error))] + (or (some-> (:hint data) str/trim not-empty) + (some-> (:explain data) str/trim not-empty)))) + +(defn- cause-message + "The message of a failure of the transport: the hint the server sent, + then the explanation, then the message of the cause and finally the + generic one. The wrapper of an aborted stream says nothing useful, so it + is skipped." + [cause default] + (let [message (some-> (ex-message cause) str/trim)] + (or (resolve-hint cause) + (when-not (or (str/blank? message) (= message "stream exception")) + message) + default))) + +(defn- job-message + "The message of a job that failed." + [error] + (or (resolve-hint error) (tr "labels.error"))) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; READING A PACKAGE +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(defn- slurp-uri + [uri response-type] + (->> (http/send! {:uri uri :response-type response-type :method :get}) + (rx/map :body))) + +(defn- parse-mtype + "The mime type behind the first bytes of a file, guessed the same way the + browser would: only a zip can be a penpot package of version 3." + [ba] + (let [u8 (js/Uint8Array. ba 0 4) + sg (areduce u8 i ret "" (str ret (if (zero? i) "" " ") (.toString (aget u8 i) 8)))] + (case sg + "120 113 3 4" "application/zip" + "1 13 32 206" "application/octet-stream" + "other"))) + +(def ^:private schema:manifest + [:map {:title "Manifest"} + [:type :string] + [:files + [:vector + [:map + [:id ::sm/uuid] + [:name :string]]]]]) + +(def ^:private decode-manifest + (sm/decoder schema:manifest sm/json-transformer)) + +(defn- read-zip-manifest + [zip-reader] + (->> (rx/from (uz/get-entry zip-reader "manifest.json")) + (rx/mapcat (fn [entry] + (if (nil? entry) + (rx/throw (ex/error :type :validation + :code :invalid-penpot-file + :hint "Not a valid Penpot file: manifest.json is missing")) + (uz/read-as-text entry)))) + (rx/map json/decode))) + +(defn- analyze-file + "The entries of one file the user dropped: one per file of a version 3 + package, one for a version 1 file, and an error for anything else." + [{:keys [uri] :as file}] + (let [stream (->> (slurp-uri uri :buffer) + (rx/merge-map + (fn [body] + (let [mtype (parse-mtype body)] + (cond + (= "application/zip" mtype) + (let [zip-reader (uz/reader body)] + (->> (read-zip-manifest zip-reader) + (rx/map + (fn [manifest] + (if (= (:type manifest) "penpot/export-files") + (let [manifest (decode-manifest manifest)] + (assoc file :type :binfile-v3 :files (:files manifest))) + (assoc file :type :unknown)))) + (rx/finalize (partial uz/close zip-reader)))) + + (= "application/octet-stream" mtype) + (rx/of (assoc file :type :binfile-v1)) + + :else + (rx/of (assoc file :type :unknown)))))) + (rx/share))] + + (->> (rx/merge + (->> stream + (rx/filter (fn [entry] (= :binfile-v1 (:type entry)))) + (rx/map (fn [entry] + (let [file-id (uuid/next)] + (-> entry + (assoc :file-id file-id) + (assoc :name (:name file)) + (assoc :status :success)))))) + + (->> stream + (rx/filter (fn [entry] (= :binfile-v3 (:type entry)))) + (rx/merge-map (fn [{:keys [files] :as entry}] + (->> (rx/from files) + (rx/map (fn [file] + (-> entry + (dissoc :files) + (assoc :name (:name file)) + (assoc :file-id (:id file)) + (assoc :status :success)))))))) + + (->> stream + (rx/filter (fn [data] (= :unknown (:type data)))) + (rx/map (fn [_] + {:uri (:uri file) + :status :error + :error (tr "dashboard.import.analyze-error")})))) + + (rx/catch (fn [cause] + (let [error (cause-message cause (tr "dashboard.import.analyze-error"))] + (rx/of (assoc file :error error :status :error)))))))) + +(defn analyze + "Read the manifest of every file the user dropped and answer with the + entries the wizard shows: one per file of a package." + [files] + (->> (rx/from files) + (rx/merge-map analyze-file))) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; IMPORTING A PACKAGE +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(defn- packages + "The packages to import: a version 1 file is one package of one entry, + and the entries of a version 3 file share their upload, which the uri + of the dropped file identifies." + [entries] + (concat + (->> entries + (filter #(= :binfile-v1 (:type %))) + (map (fn [entry] + {:uri (:uri entry) + :version 1 + :name (str/replace (:name entry) #".penpot$" "") + :entries [entry]}))) + + (->> entries + (filter #(= :binfile-v3 (:type %))) + (group-by :uri) + (map (fn [[uri entries]] + {:uri uri + :version 3 + :name (:name (first entries)) + :entries entries}))))) + +(defn- entry-messages + "The message of every entry of a package: the job works on the package, + and each of its entries shows the same step." + [entries message] + (map (fn [entry] (assoc message :file-id (:file-id entry))) entries)) + +(defn- upload-milestone + "The milestone of the upload phase: the chunks done out of the total. + It is built on the client, never comes from the backend, and reuses + the milestone shape so the wizard shows it like any other step." + [{:keys [current total]}] + {:stage :upload :counters {:upload {:current current :total total}}}) + +(defn- import-package + "One package as a job: the blob is uploaded in chunks, the job is created + with the upload and followed, and its outcome becomes the messages of + the wizard. + + `on-job` is an optional callback invoked with the id of the created + job, so the caller can cancel it while it runs." + [ws-conn project-id resolutions {:keys [uri version name entries]} on-job] + (let [upload-events (rx/subject) + upload-ended? (atom false) + finish-upload! (fn [] + (when (compare-and-set! upload-ended? false true) + (rx/end! upload-events))) + report-upload! (fn [progress] + (doseq [message (entry-messages entries {:status :progress + :progress (upload-milestone progress)})] + (rx/push! upload-events message)))] + (rx/merge + upload-events + (->> (slurp-uri uri :blob) + (rx/mapcat (fn [blob] + (->> (uploads/upload-blob-chunked blob :on-progress report-upload!) + (rx/tap (fn [_] (finish-upload!)))))) + (rx/mapcat (fn [{:keys [session-id]}] + (rp/cmd! :create-import-binfile-job + {:params {:project-id project-id + :name name + :version version} + :upload-id session-id}))) + (rx/mapcat (fn [{job-id :id}] + (when (fn? on-job) + (on-job job-id)) + ;; the job exists but may wait for a worker: tell + ;; every entry it is queued before following it + (rx/concat + (rx/from (entry-messages entries {:status :queued})) + (->> (dj/watch-job ws-conn job-id) + (rx/mapcat (fn [{:keys [kind status result error] :as emission}] + (cond + (= "completed" status) + (do + (when-let [resolution (not-empty (:resolution result))] + (vswap! resolutions merge resolution)) + (rx/from (entry-messages entries {:status :finish}))) + + (= "failed" status) + (rx/from (entry-messages entries {:status :error + :error (job-message error)})) + + (= "cancelled" status) + (rx/from (entry-messages entries {:status :error + :error (tr "jobs.import-cancelled")})) + + (= :progress kind) + (rx/from (entry-messages entries {:status :progress + :progress (:payload emission)})) + + ;; a worker picked it up (or will + ;; retry it): no milestone yet + (contains? #{:start :retry} kind) + (rx/from (entry-messages entries {:status :started})) + + :else + (rx/empty)))))))) + (rx/catch (fn [cause] + (finish-upload!) + (rx/from (entry-messages entries {:status :error + :error (cause-message cause (tr "labels.error"))})))))))) + +(defn import-files + "Import the entries the wizard analyzed, one job per package. + + Answers with the messages the wizard already understands: the step of + the job for every entry (progress, finish or error) and, once every + package is over, the resolution of the libraries they brought. + + The optional `:on-job` callback is invoked with the id of every + package job created, so the caller can cancel them while they run." + [{:keys [project-id entries on-job]}] + (let [ws-conn (:ws-conn @st/state) + resolutions (volatile! {}) + importing (->> (rx/from (packages entries)) + (rx/merge-map (fn [package] + (import-package ws-conn project-id resolutions package on-job))))] + (rx/concat importing + (->> (rx/defer #(rx/of @resolutions)) + (rx/map (fn [resolution] + {:libraries-resolution resolution})))))) diff --git a/frontend/src/app/main/data/jobs.cljs b/frontend/src/app/main/data/jobs.cljs new file mode 100644 index 0000000000..9ffc801512 --- /dev/null +++ b/frontend/src/app/main/data/jobs.cljs @@ -0,0 +1,94 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns app.main.data.jobs + "Following a job the client created. + + A running job publishes its progress as `:job-event` messages on the + websocket of the profile and keeps its outcome in its own row. This + namespace joins both: `watch-job` emits every event of the job and one + final outcome, read from the row and not from the event, because the + closing event carries only the outcome." + (:require + [app.main.repo :as rp] + [app.util.websocket :as ws] + [beicon.v2.core :as rx])) + +(def ^:private terminal-statuses + #{"completed" "failed" "cancelled"}) + +(defn- events-of + "The events of one job, as they arrive on the websocket, with the + milestone of the event under `:payload`." + [ws-conn job-id] + (->> (ws/get-rcv-stream ws-conn) + (rx/filter ws/message-event?) + (rx/map :payload) + (rx/filter #(and (= :job-event (:type %)) + (= job-id (:job-id %)))) + (rx/map (fn [event] + {:kind (keyword (:kind event)) + :payload (:payload event)})))) + +(defn- outcome-of + "The row of a job as the outcome of the stream, nil while it is not + over. A job that failed answers with the public error of the row." + [job] + (when (contains? terminal-statuses (:status job)) + (cond-> {:status (:status job)} + (some? (:result job)) + (assoc :result (:result job)) + + (some? (:error job)) + (assoc :error (:error job))))) + +(defn- read-outcome + [job-id] + (->> (rp/cmd! :get-job {:id job-id}) + (rx/map outcome-of) + (rx/filter some?))) + +(defn watch-job + "The lifecycle of `job-id` as an observable: every event it publishes + and one final outcome, after which the stream ends. + + The outcome is read from the row and not from the closing event: it is + read once when the stream starts and again on every reconnect of the + websocket, so a job that ended before this stream existed, or while the + socket was down, still answers. A read that fails reaches the caller, + which is the only one that can tell a transient failure from a job that + is gone. + + Emissions: + + - `{:kind :progress :payload {:stage ... :counters {...}}}` per event, + with the kind of the event (`:progress`, `:start`, `:retry`) + - `{:status \"completed\" :result {...}}` or `{:status \"failed\" + :error {...}}` or `{:status \"cancelled\"}` once, and then the + stream completes" + [ws-conn job-id] + (let [events (events-of ws-conn job-id) + progress (->> events + (rx/filter #(not= :end (:kind %)))) + triggers (rx/merge (->> (ws/get-rcv-stream ws-conn) + (rx/filter ws/opened-event?)) + (->> events + (rx/filter #(= :end (:kind %))))) + outcomes (->> (rx/merge (rx/of ::start) triggers) + (rx/mapcat (fn [_] (read-outcome job-id))) + (rx/take 1) + (rx/share))] + (rx/merge (rx/take-until outcomes progress) + outcomes))) + +(defn cancel-job + "Ask the server to cancel a job the caller created. Fire and forget: + failures are ignored because the job may have just finished on its + own." + [job-id] + (->> (rp/cmd! :cancel-job {:id job-id}) + (rx/catch (fn [_] (rx/of nil))) + (rx/subs! (constantly nil) (constantly nil) (constantly nil)))) diff --git a/frontend/src/app/main/data/uploads.cljs b/frontend/src/app/main/data/uploads.cljs index 5ae47c477a..f18956bd7f 100644 --- a/frontend/src/app/main/data/uploads.cljs +++ b/frontend/src/app/main/data/uploads.cljs @@ -11,7 +11,7 @@ by any feature that needs to upload large binary blobs: 1. create-upload-session – obtain a session-id - 2. upload-chunk – upload each slice (max-parallel-chunk-uploads in-flight) + 2. upload-chunk – upload each slice, one at a time 3. caller-specific step – e.g. assemble-file-media-object or import-binfile `upload-blob-chunked` drives steps 1 and 2 and emits the completed @@ -24,18 +24,13 @@ [app.main.repo :as rp] [beicon.v2.core :as rx])) -(def ^:private max-parallel-chunk-uploads - "Maximum number of chunk upload requests that may be in-flight at the - same time within a single chunked upload session." - 2) - (defn upload-blob-chunked "Uploads `blob` via the three-step chunked session API. Steps performed: 1. Creates an upload session (`create-upload-session`). - 2. Slices `blob` and uploads every chunk (`upload-chunk`), - with at most `max-parallel-chunk-uploads` concurrent requests. + 2. Slices `blob` and uploads every chunk (`upload-chunk`), one at a + time so the progress of the file is a straight line. Returns an observable that emits exactly one map: `{:session-id }` @@ -43,8 +38,11 @@ The caller is responsible for the final step (assemble / import). The optional `opts` map accepts: - `:chunk-size` – size in bytes of each chunk (default: `cf/upload-chunk-size`, 25 MiB)." - [blob & {:keys [chunk-size] :or {chunk-size cf/upload-chunk-size}}] + `:chunk-size` – size in bytes of each chunk (default: `cf/upload-chunk-size`, 25 MiB). + `:on-progress` – a fn called after every uploaded chunk with + `{:current :total }`, so the + caller can show the upload while it runs." + [blob & {:keys [chunk-size on-progress] :or {chunk-size cf/upload-chunk-size}}] (let [total-size (.-size blob) total-chunks (js/Math.ceil (/ total-size chunk-size))] (->> (rp/cmd! :create-upload-session @@ -53,17 +51,22 @@ (fn [{raw-session-id :session-id}] (let [session-id (cond-> raw-session-id (string? raw-session-id) uuid/uuid) + uploaded (atom 0) chunk-uploads (->> (range total-chunks) (map (fn [idx] (let [start (* idx chunk-size) end (min (+ start chunk-size) total-size) chunk (.slice blob start end)] - (rp/cmd! :upload-chunk - {:session-id session-id - :index idx - :content (list chunk (dm/str "chunk-" idx))})))))] + (->> (rp/cmd! :upload-chunk + {:session-id session-id + :index idx + :content (list chunk (dm/str "chunk-" idx))}) + (rx/tap (fn [_] + (when (fn? on-progress) + (on-progress {:current (swap! uploaded inc) + :total total-chunks})))))))))] (->> (rx/from chunk-uploads) - (rx/merge-all max-parallel-chunk-uploads) + (rx/concat-all) (rx/last) (rx/map (fn [_] {:session-id session-id}))))))))) diff --git a/frontend/src/app/main/repo.cljs b/frontend/src/app/main/repo.cljs index 01ce9e4c01..938e6aaa16 100644 --- a/frontend/src/app/main/repo.cljs +++ b/frontend/src/app/main/repo.cljs @@ -159,6 +159,7 @@ {:stream? true} :export-binfile {:response-type :blob} + :retrieve-list-of-builtin-templates {:query-params :all}}) (defn- send! diff --git a/frontend/src/app/main/ui/dashboard/import.cljs b/frontend/src/app/main/ui/dashboard/import.cljs index dd459da077..e03d67dbe9 100644 --- a/frontend/src/app/main/ui/dashboard/import.cljs +++ b/frontend/src/app/main/ui/dashboard/import.cljs @@ -13,6 +13,8 @@ [app.common.logging :as log] [app.main.data.dashboard :as dd] [app.main.data.event :as ev] + [app.main.data.imports :as imp] + [app.main.data.jobs :as dj] [app.main.data.modal :as modal] [app.main.data.notifications :as ntf] [app.main.repo :as rp] @@ -29,7 +31,7 @@ [app.main.ui.ds.notifications.context-notification :refer [context-notification*]] [app.main.ui.ds.product.loader :refer [loader*]] [app.main.ui.icons :as deprecated-icon] - [app.main.worker :as mw] + [app.main.ui.jobs.progress :as jp] [app.util.dom :as dom] [app.util.i18n :as i18n :refer [tr]] [app.util.keyboard :as kbd] @@ -108,45 +110,21 @@ (if (= (:file-id entry) (:file-id message)) (let [status (case (:status message) :progress :import-progress - :finish :import-success - :error :import-error)] - (-> entry - (assoc :progress (:progress message)) - (assoc :status status) - (assoc :error (:error message)) - (d/without-nils))) + :started :import-progress + :queued :import-queued + :finish :import-success + :error :import-error)] + ;; a message only carries what it knows: anything it does + ;; not bring (a stale milestone, an old error) is dropped + ;; instead of lingering on the row + (cond-> (assoc entry :status status) + (contains? message :progress) (assoc :progress (:progress message)) + (not (contains? message :progress)) (dissoc :progress) + (contains? message :error) (assoc :error (:error message)) + (not (contains? message :error)) (dissoc :error))) entry)) entries)) -(defn- parse-progress-message - [message] - (case (:type message) - :upload-data - (tr "dashboard.import.progress.upload-data" (:current message) (:total message)) - - :upload-media - (tr "dashboard.import.progress.upload-media" (:file message)) - - :process-page - (tr "dashboard.import.progress.process-page" (:file message)) - - :process-colors - (tr "dashboard.import.progress.process-colors") - - :process-typographies - (tr "dashboard.import.progress.process-typographies") - - :process-media - (tr "dashboard.import.progress.process-media") - - :process-components - (tr "dashboard.import.progress.process-components") - - :process-deleted-components - (tr "dashboard.import.progress.process-components") - - "")) - (defn- has-status-analyze? [item] (= (:status item) :analyze)) @@ -180,40 +158,32 @@ (defn- analyze-entries [state entries] - (let [features (get @st/state :features)] - (->> (mw/ask-many! - {:cmd :analyze-import - :files entries - :features features}) - (rx/mapcat #(rx/delay emit-delay (rx/of %))) - (rx/filter some?) - (rx/subs! - (fn [message] - (when (some? (:error message)) - (st/emit! (ev/event {::ev/name "import-files-error" - :error (:error message)}))) - (swap! state update-with-analyze-result message)))))) + (->> (imp/analyze entries) + (rx/mapcat #(rx/delay emit-delay (rx/of %))) + (rx/filter some?) + (rx/subs! + (fn [message] + (when (some? (:error message)) + (st/emit! (ev/event {::ev/name "import-files-error" + :error (:error message)}))) + (swap! state update-with-analyze-result message))))) (defn- import-files - [state library-resolution-data* project-id entries] + [state library-resolution-data* project-id entries on-job] (st/emit! (ev/event {::ev/name "import-files" :num-files (count entries)})) - (let [features (get @st/state :features)] - (->> (mw/ask-many! - {:cmd :import-files - :project-id project-id - :files entries - :features features}) - (rx/filter some?) - (rx/subs! - (fn [message] - ;; Capture library-resolution data if present (same for all - ;; entries from the same zip, so first one wins) - (if-let [resolution (-> (:libraries-resolution message) - (not-empty))] - (reset! library-resolution-data* resolution) - (swap! state update-entry-status message))))))) + (->> (imp/import-files {:project-id project-id + :entries entries + :on-job on-job}) + (rx/filter some?) + (rx/subs! + (fn [message] + ;; the resolution of the libraries of every package arrives once, + ;; when all of them are over + (if-let [resolution (-> (:libraries-resolution message) (not-empty))] + (reset! library-resolution-data* resolution) + (swap! state update-entry-status message)))))) (mf/defc import-entry* {::mf/memo true @@ -224,6 +194,7 @@ format (:type entry) loading? (or (= :analyze status) + (= :import-queued status) (= :import-progress status) (and is-progress (= :import-ready status))) analyze-error? (= :analyze-error status) @@ -336,24 +307,27 @@ [:> text* {:class (stl/css :error-message) :as "span" :typography t/body-small} - ;; backend-provided error key, dynamic by design - #_{:clj-kondo/ignore [:penpot/tr-dynamic]} + ;; the message is user-facing text already (a hint or a translated + ;; label), never a key (if (some? (:error entry)) - (tr (:error entry)) + (:error entry) (tr "dashboard.import.analyze-error"))] import-error? [:> text* {:class (stl/css :error-message) :as "span" :typography t/body-small} - ;; backend-provided error key, dynamic by design - #_{:clj-kondo/ignore [:penpot/tr-dynamic]} (if (some? (:error entry)) - (tr (:error entry)) + (:error entry) (tr "labels.error"))] + (and (= :import-queued status) (not import-success?)) + [:div {:class (stl/css :progress-message)} + (tr "jobs.queued")] + (and (not import-success?) (some? progress)) - [:div {:class (stl/css :progress-message)} (parse-progress-message progress)]) + [:div {:class (stl/css :progress-message)} + (jp/milestone-text progress :file? true)]) ;; This is legacy code, will be removed when legacy-zip format is removed [:div {:class (stl/css :linked-libraries)} @@ -577,7 +551,7 @@ {::mf/private true} [{:keys [entries template status errors? import-success-total auto-linked-count edition on-edit on-change on-delete - on-cancel on-continue on-accept pending-analysis?]}] + on-cancel on-cancel-import on-continue on-accept pending-analysis?]}] [:* [:div {:class (stl/css :modal-content)} (when (and (= :analyze status) errors?) @@ -639,13 +613,7 @@ (when (some? template) [:> import-entry* {:entry (assoc template :status status) - :can-be-deleted false}]) - - (when (= :import-progress status) - [:div {:class (stl/css :status-message) - :role "status" - :aria-live "polite"} - (tr "labels.uploading-file")])] + :can-be-deleted false}])] [:div {:class (stl/css :modal-footer)} [:div {:class (stl/css :action-buttons)} @@ -664,11 +632,16 @@ (tr "labels.continue")] :import-progress - [:> button* {:class (stl/css :accept-btn) - :variant "primary" - :disabled true - :on-click on-accept} - (tr "labels.accept")] + [:* + [:> button* {:class (stl/css :cancel-button) + :variant "secondary" + :on-click on-cancel-import} + (tr "labels.cancel")] + [:> button* {:class (stl/css :accept-btn) + :variant "primary" + :disabled true + :on-click on-accept} + (tr "labels.accept")]] (:import-success :import-error) [:> button* {:class (stl/css :accept-btn) @@ -749,6 +722,13 @@ status* (mf/use-state :analyze) status (deref status*) + ;; Jobs created while importing, to cancel them on demand, and + ;; the subscription to their messages, to stop listening. + jobs* (mf/use-state #{}) + jobs (deref jobs*) + sub* (mf/use-state nil) + sub (deref sub*) + edition* (mf/use-state nil) edition (deref edition*) @@ -795,8 +775,10 @@ (mf/deps entries) (fn [] (let [entries (filterv has-status-ready? entries)] + (reset! jobs* #{}) (reset! status* :import-progress) - (import-files state* resolution* project-id entries)))) + (reset! sub* (import-files state* resolution* project-id entries + #(swap! jobs* conj %)))))) continue-template (mf/use-fn @@ -841,13 +823,33 @@ (fn [file-id] (swap! state* remove-entry file-id))) + on-cancel-import + (mf/use-fn + (mf/deps jobs sub) + (fn [event] + (dom/prevent-default event) + ;; stop listening first, so no late message repaints the + ;; entries of a dialog that is going away + (when (some? sub) + (rx/dispose! sub)) + ;; a job may have just finished on its own: failures are + ;; ignored and an empty run cancels nothing + (run! dj/cancel-job jobs) + (reset! jobs* #{}) + (reset! sub* nil) + (st/emit! (modal/hide)))) + on-cancel (mf/use-fn - (mf/deps edition) + (mf/deps edition status on-cancel-import) (fn [event] (when (nil? edition) - (dom/prevent-default event) - (st/emit! (modal/hide))))) + (if (= :import-progress status) + ;; closing mid-import stops the jobs, like Cancel + (on-cancel-import event) + (do + (dom/prevent-default event) + (st/emit! (modal/hide))))))) on-continue (mf/use-fn @@ -1013,6 +1015,7 @@ :on-change on-entry-change :on-delete on-entry-delete :on-cancel on-cancel + :on-cancel-import on-cancel-import :on-continue on-continue :on-accept on-accept :pending-analysis? pending-analysis?}] diff --git a/frontend/src/app/main/ui/dashboard/import.scss b/frontend/src/app/main/ui/dashboard/import.scss index f6765642b7..eafdc1ecc6 100644 --- a/frontend/src/app/main/ui/dashboard/import.scss +++ b/frontend/src/app/main/ui/dashboard/import.scss @@ -34,7 +34,7 @@ background-color: var(--color-background-primary); border: $b-2 solid var(--color-background-quaternary); min-block-size: $sz-192; - inline-size: $sz-512; + inline-size: px2rem(640); max-block-size: px2rem(800); } @@ -64,13 +64,6 @@ inset-inline-end: px2rem(6); } -.status-message { - @include use-typography("body-small"); - - color: var(--color-foreground-primary); - font-style: italic; -} - .action-buttons { display: flex; justify-content: flex-end; diff --git a/frontend/src/app/main/ui/exports/files.cljs b/frontend/src/app/main/ui/exports/files.cljs index 92a458e772..a36545fb95 100644 --- a/frontend/src/app/main/ui/exports/files.cljs +++ b/frontend/src/app/main/ui/exports/files.cljs @@ -11,6 +11,7 @@ [app.common.data.macros :as dm] [app.config :as cf] [app.main.data.exports.files :as fexp] + [app.main.data.jobs :as dj] [app.main.data.modal :as modal] [app.main.store :as st] [app.main.ui.ds.buttons.button :refer [button*]] @@ -20,6 +21,7 @@ [app.main.ui.ds.foundations.typography.heading :refer [heading*]] [app.main.ui.ds.foundations.typography.text :refer [text*]] [app.main.ui.ds.product.loader :refer [loader*]] + [app.main.ui.jobs.progress :as jp] [app.main.ui.notifications.context-notification :refer [context-notification]] [app.util.dom :as dom] [app.util.i18n :as i18n :refer [tr]] @@ -30,7 +32,17 @@ [files file-id] (mapv #(cond-> % (= file-id (:id %)) - (assoc :export-error? true + (assoc :export-error true + :loading false)) + files)) + +(defn- mark-file-cancelled + "The job of a file was cancelled elsewhere: a neutral terminal state, + neither success nor error." + [files file-id] + (mapv #(cond-> % + (= file-id (:id %)) + (assoc :export-cancelled true :loading false)) files)) @@ -38,10 +50,27 @@ [files file-id] (mapv #(cond-> % (= file-id (:id %)) - (assoc :export-success? true + (assoc :export-success true :loading false)) files)) +(defn- mark-file-progress + "The milestone the job of a file is in, to show it while it runs." + [files file-id progress] + (mapv #(cond-> % + (= file-id (:id %)) + (assoc :progress progress + :queued false)) + files)) + +(defn- mark-file-queued + "The job of a file exists but no worker picked it up yet." + [files file-id] + (mapv #(cond-> % + (= file-id (:id %)) + (assoc :queued true)) + files)) + (defn- initialize-state "Initialize export dialog state" [files] @@ -54,26 +83,52 @@ {::mf/private true} [{:keys [file]}] (let [level (cond - (:export-success? file) :success - (:export-error? file) :error - :else :info)] + (:export-success file) :success + (:export-error file) :error + (:export-cancelled file) :warning + :else :info)] [:div {:class (stl/css-case :file-entry true :loading (:loading file) - :success (:export-success? file) - :error (:export-error? file))} + :success (:export-success file) + :error (:export-error file))} (if (:loading file) - [:div {:class (stl/css :file-name)} - [:> loader* {:width 26 - :title (tr "labels.loading")}] - [:> text* {:class (stl/css :file-name-label) - :as "span" - :typography t/body-large} - (:name file)]] + [:* + [:div {:class (stl/css :file-name)} + [:> loader* {:width 26 + :title (tr "labels.loading")}] + [:> text* {:class (stl/css :file-name-label) + :as "span" + :typography t/body-large} + (:name file)]] - [:> context-notification {:level level - :content (:name file)}])])) + ;; while the file is being produced: waiting for a worker, then + ;; the milestone of the job + (when (or (some? (:progress file)) (:queued file)) + (let [progress (:progress file)] + [:> text* {:class (stl/css :status-message) + :as "span" + :typography (if (some? progress) t/body-large t/body-medium) + :role "status" + :aria-live "polite"} + (if (some? progress) + (jp/milestone-text progress) + (tr "jobs.queued"))]))] + + [:* + [:> context-notification {:level level + :content (:name file)}] + + ;; a cancelled file is a neutral terminal state, but the label is + ;; what tells the user why no artifact came out + (when (:export-cancelled file) + [:> text* {:class (stl/css :status-message) + :as "span" + :typography t/body-medium + :role "status" + :aria-live "polite"} + (tr "jobs.export-cancelled")])])])) (mf/defc export-dialog {::mf/register modal/components @@ -90,25 +145,75 @@ selected (:selected state) status (:status state) + ;; Jobs still running, as file-id -> job-id, to cancel them on + ;; demand; a file that reaches its outcome leaves the map, so + ;; closing a finished export cancels nothing. And the + ;; subscription to their messages, to stop listening. + jobs* (mf/use-state {}) + jobs (deref jobs*) + sub* (mf/use-state nil) + sub (deref sub*) + start-export (mf/use-fn (mf/deps team-id selected files) (fn [] (swap! state* assoc :status :exporting) - (->> (fexp/export-files :files files :type selected) - (rx/subs! - (fn [{:keys [file-id error filename uri] :as result}] - (if error - (swap! state* update :files mark-file-error file-id) - (do - (swap! state* update :files mark-file-success file-id) - (dom/trigger-download-uri filename "application/penpot" uri)))))))) + (reset! jobs* {}) + (reset! sub* (->> (fexp/export-files :files files :type selected + :on-job #(swap! jobs* assoc (:file-id %) (:job-id %))) + (rx/subs! + (fn [{:keys [file-id error filename uri progress queued + cancelled]}] + (cond + (some? progress) + (swap! state* update :files mark-file-progress file-id progress) + + (true? queued) + (swap! state* update :files mark-file-queued file-id) + + (some? error) + (swap! jobs* dissoc file-id) + (swap! state* update :files mark-file-error file-id) + + (true? cancelled) + (swap! jobs* dissoc file-id) + (swap! state* update :files mark-file-cancelled file-id) + + ;; only a message carrying the artifact + ;; downloads: anything else is ignored + (some? uri) + (do + (swap! jobs* dissoc file-id) + (swap! state* update :files mark-file-success file-id) + (dom/trigger-download-uri filename "application/penpot" uri))))))))) + + on-cancel-export + (mf/use-fn + (mf/deps jobs sub) + (fn [event] + (dom/prevent-default event) + ;; stop listening first, so no late message repaints the + ;; entries of a dialog that is going away + (when (some? sub) + (rx/dispose! sub)) + ;; only the jobs still in flight are cancelled: finished ones + ;; already left the map, and failures are ignored because a job may have just finished on its own + (run! dj/cancel-job (vals jobs)) + (reset! jobs* {}) + (reset! sub* nil) + (st/emit! (modal/hide)))) on-cancel (mf/use-fn + (mf/deps status on-cancel-export) (fn [event] - (dom/prevent-default event) - (st/emit! (modal/hide)))) + (if (= :exporting status) + ;; closing mid-export stops the jobs, like Cancel + (on-cancel-export event) + (do + (dom/prevent-default event) + (st/emit! (modal/hide)))))) on-accept (mf/use-fn @@ -123,7 +228,7 @@ (let [type (-> (dom/get-target event) (dom/get-data "type") (keyword))] - (prn "AAA" selected type) + (swap! state* assoc :selected type))))] (mf/with-effect [has-libs?] @@ -208,16 +313,15 @@ [:* [:div {:class (stl/css :modal-content)} (for [file (:files state)] - [:> export-entry* {:file file :key (dm/str (:id file))}]) - - (when in-progress? - [:> text* {:as "span" :typography t/body-large :class (stl/css :status-message) - :role "status" - :aria-live "polite"} - (tr "labels.downloading-file")])] + [:> export-entry* {:file file :key (dm/str (:id file))}])] [:div {:class (stl/css :modal-footer)} [:div {:class (stl/css :action-buttons)} + (when in-progress? + [:> button* {:variant "secondary" + :type "button" + :on-click on-cancel-export} + (tr "labels.cancel")]) [:> button* {:variant "primary" :type "button" :disabled in-progress? diff --git a/frontend/src/app/main/ui/jobs/progress.cljs b/frontend/src/app/main/ui/jobs/progress.cljs new file mode 100644 index 0000000000..83967aa978 --- /dev/null +++ b/frontend/src/app/main/ui/jobs/progress.cljs @@ -0,0 +1,69 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns app.main.ui.jobs.progress + "How a milestone of a job is written in a row: the stage in course and + its counter. + + The stage is a stable key of the vocabulary of the worker, never a + display string, so it is resolved here with a `case` of literal `tr` + calls: a stage this client does not know falls back to a generic label + instead of breaking the row or showing the raw key." + (:require + [app.util.i18n :as i18n :refer [tr]])) + +(defn- stage-label + [stage] + (case stage + :files (tr "jobs.progress.stage.files") + :pages (tr "jobs.progress.stage.pages") + :media (tr "jobs.progress.stage.media") + :thumbnails (tr "jobs.progress.stage.thumbnails") + :components (tr "jobs.progress.stage.components") + :colors (tr "jobs.progress.stage.colors") + :typographies (tr "jobs.progress.stage.typographies") + :tokens (tr "jobs.progress.stage.tokens") + :storage-objects (tr "jobs.progress.stage.storage-objects") + :relations (tr "jobs.progress.stage.relations") + :metadata (tr "jobs.progress.stage.metadata") + :manifest (tr "jobs.progress.stage.manifest") + :upload (tr "jobs.progress.stage.upload") + (tr "jobs.progress.stage.unknown"))) + +(defn- counter-label + "The counter of one scope, `3/8` when the worker knows the total and the + bare current when it does not." + [{:keys [current total]}] + (when (some? current) + (if (some? total) + (str current "/" total) + (str current)))) + +(defn- upload-label + "The upload of a file counts chunks, not units of work, so it is written + as the percentage of the file already sent instead of a counter." + [{:keys [current total]}] + (when (and (some? current) (some? total) (pos? total)) + (str "(" (js/Math.round (* 100 (/ current total))) "%)"))) + +(defn milestone-text + "The text of a milestone: the stage in course and its counter, and with + `:file?` the file in course behind it, when the milestone carries its + counter. The upload stage is the exception: it counts chunks, so it is + written as a percentage of the file." + [milestone & {:keys [file?]}] + (when-some [stage (:stage milestone)] + (let [counters (:counters milestone) + own (if (= :upload stage) + (upload-label (get counters :upload)) + (counter-label (get counters stage)))] + (str (stage-label stage) + (when (some? own) + (str " " own)) + (when-let [files (and file? + (not= :files stage) + (counter-label (get counters :files)))] + (str " · " (stage-label :files) " " files)))))) diff --git a/frontend/src/app/worker.cljs b/frontend/src/app/worker.cljs index da6f0c8a95..860abcff27 100644 --- a/frontend/src/app/worker.cljs +++ b/frontend/src/app/worker.cljs @@ -12,7 +12,6 @@ [app.common.types.objects-map] [app.util.object :as obj] [app.worker.impl :as impl] - [app.worker.import] [app.worker.index] [app.worker.messages :as wm] [app.worker.thumbnails] diff --git a/frontend/src/app/worker/import.cljs b/frontend/src/app/worker/import.cljs deleted file mode 100644 index e8310efa89..0000000000 --- a/frontend/src/app/worker/import.cljs +++ /dev/null @@ -1,253 +0,0 @@ -;; This Source Code Form is subject to the terms of the Mozilla Public -;; License, v. 2.0. If a copy of the MPL was not distributed with this -;; file, You can obtain one at http://mozilla.org/MPL/2.0/. -;; -;; Copyright (c) KALEIDOS SUBSIDIARY SL - -(ns app.worker.import - (:refer-clojure :exclude [resolve]) - (:require - [app.common.exceptions :as ex] - [app.common.json :as json] - [app.common.logging :as log] - [app.common.schema :as sm] - [app.common.uuid :as uuid] - [app.main.data.uploads :as uploads] - [app.main.repo :as rp] - [app.util.http :as http] - [app.util.i18n :as i18n :refer [tr]] - [app.util.sse :as sse] - [app.util.zip :as uz] - [app.worker.impl :as impl] - [beicon.v2.core :as rx] - [cuerdas.core :as str])) - -(log/set-level! :warn) - -(defn- import-cause-message - "Prefer the server `:hint` (full text, e.g. SSE error payload), then `:explain` - when present; avoid the generic `stream exception` wrapper when a payload exists." - [cause default-msg] - (let [data (ex-data cause) - hint (some-> data :hint str/trim) - explain (some-> data :explain str/trim)] - (cond - (not (str/blank? hint)) hint - (not (str/blank? explain)) explain - :else - (let [msg (some-> (ex-message cause) str/trim)] - (if (or (str/blank? msg) (= msg "stream exception")) - default-msg - msg))))) - -;; Upload changes batches size -(def ^:const change-batch-size 100) - -(def conjv (fnil conj [])) - -(defn read-zip-manifest - [zip-reader] - (->> (rx/from (uz/get-entry zip-reader "manifest.json")) - (rx/mapcat (fn [entry] - (if (nil? entry) - (rx/throw (ex/error :type :validation - :code :invalid-penpot-file - :hint "Not a valid Penpot file: manifest.json is missing")) - (uz/read-as-text entry)))) - (rx/map json/decode))) - -(defn slurp-uri - ([uri] (slurp-uri uri :text)) - ([uri response-type] - (->> (http/send! - {:uri uri - :response-type response-type - :method :get}) - (rx/map :body)))) - -(defn parse-mtype [ba] - (let [u8 (js/Uint8Array. ba 0 4) - sg (areduce u8 i ret "" (str ret (if (zero? i) "" " ") (.toString (aget u8 i) 8)))] - (case sg - "120 113 3 4" "application/zip" - "1 13 32 206" "application/octet-stream" - "other"))) - -;; NOTE: this is a limited subset schema for the manifest file of -;; binfile-v3 format; is used for partially parse it and read the -;; files referenced inside the exported file - -(def ^:private schema:manifest - [:map {:title "Manifest"} - [:type :string] - [:files - [:vector - [:map - [:id ::sm/uuid] - [:name :string]]]]]) - -(def ^:private decode-manifest - (sm/decoder schema:manifest sm/json-transformer)) - -(defn analyze-file - [{:keys [uri] :as file}] - (let [stream (->> (slurp-uri uri :buffer) - (rx/merge-map - (fn [body] - (let [mtype (parse-mtype body)] - (cond - (= "application/zip" mtype) - (let [zip-reader (uz/reader body)] - (->> (read-zip-manifest zip-reader) - (rx/map - (fn [manifest] - (if (= (:type manifest) "penpot/export-files") - (let [manifest (decode-manifest manifest)] - (assoc file :type :binfile-v3 :files (:files manifest))) - (assoc file :type :unknown)))) - (rx/finalize (partial uz/close zip-reader)))) - - (= "application/octet-stream" mtype) - (rx/of (assoc file :type :binfile-v1)) - - :else - (rx/of (assoc file :type :unknown)))))) - - (rx/share))] - - (->> (rx/merge - (->> stream - (rx/filter (fn [entry] (= :binfile-v1 (:type entry)))) - (rx/map (fn [entry] - (let [file-id (uuid/next)] - (-> entry - (assoc :file-id file-id) - (assoc :name (:name file)) - (assoc :status :success)))))) - - (->> stream - (rx/filter (fn [entry] (= :binfile-v3 (:type entry)))) - (rx/merge-map (fn [{:keys [files] :as entry}] - (->> (rx/from files) - (rx/map (fn [file] - (-> entry - (dissoc :files) - (assoc :name (:name file)) - (assoc :file-id (:id file)) - (assoc :status :success)))))))) - - (->> stream - (rx/filter (fn [data] (= :unknown (:type data)))) - (rx/map (fn [_] - {:uri (:uri file) - :status :error - :error (tr "dashboard.import.analyze-error")})))) - - (rx/catch (fn [cause] - (let [error (import-cause-message cause (tr "dashboard.import.analyze-error"))] - (rx/of (assoc file :error error :status :error)))))))) - -(defmethod impl/handler :analyze-import - [{:keys [files]}] - (->> (rx/from files) - (rx/merge-map analyze-file))) - -(defn- import-blob-via-upload - "Fetches `uri` as a Blob, uploads it using the generic chunked-upload - session API and calls `import-binfile` with the resulting upload-id. - Returns an observable of SSE events from the import stream." - [uri {:keys [name version project-id]}] - (->> (slurp-uri uri :blob) - (rx/mapcat - (fn [blob] - (->> (uploads/upload-blob-chunked blob) - (rx/mapcat - (fn [{:keys [session-id]}] - (rp/cmd! ::sse/import-binfile - {:name name - :upload-id session-id - :version version - :project-id project-id})))))))) - -(defmethod impl/handler :import-files - [{:keys [project-id files]}] - (let [binfile-v1 (filter #(= :binfile-v1 (:type %)) files) - binfile-v3 (filter #(= :binfile-v3 (:type %)) files) - resolutions (volatile! {})] - - (rx/merge - (->> (rx/from binfile-v1) - (rx/merge-map - (fn [data] - (->> (import-blob-via-upload (:uri data) - {:name (str/replace (:name data) #".penpot$" "") - :version 1 - :project-id project-id}) - (rx/tap (fn [event] - (let [payload (sse/get-payload event) - type (sse/get-type event)] - (if (= type "progress") - (log/dbg :hint "import-binfile: progress" - :section (:section payload) - :name (:name payload)) - (log/dbg :hint "import-binfile: end"))))) - (rx/filter sse/end-of-stream?) - (rx/map (fn [_] - {:status :finish - :file-id (:file-id data)})) - (rx/catch - (fn [cause] - (log/error :hint "unexpected error on import process" - :project-id project-id - :cause cause) - (rx/of {:status :error - :error (import-cause-message cause (tr "labels.error")) - :file-id (:file-id data)}))))))) - - - (rx/concat - (->> (rx/from binfile-v3) - (rx/reduce (fn [result file] - (update result (:uri file) (fnil conj []) file)) - {}) - (rx/mapcat identity) - (rx/merge-map - (fn [[uri entries]] - (->> (import-blob-via-upload uri - {:name (-> entries first :name) - :version 3 - :project-id project-id}) - (rx/tap (fn [event] - (let [payload (sse/get-payload event) - type (sse/get-type event)] - (cond - (= type "progress") - (log/dbg :hint "import-binfile: progress" - :section (:section payload) - :name (:name payload)) - - :else - (log/dbg :hint "import-binfile: end"))))) - (rx/filter sse/end-of-stream?) - (rx/mapcat (fn [message] - (let [{:keys [resolution]} (sse/get-payload message)] - (when (seq resolution) - (vswap! resolutions merge resolution)) - (->> (rx/from entries) - (rx/map (fn [entry] - {:status :finish - :file-id (:file-id entry)})))))) - (rx/catch (fn [cause] - (log/error :hint "import-binfile: unexpected error on importing" - :project-id project-id - ::log/sync? true - :cause cause) - (let [err (import-cause-message cause (tr "labels.error"))] - (->> (rx/from entries) - (rx/map (fn [entry] - {:status :error - :error err - :file-id (:file-id entry)})))))))))) - (->> (rx/defer #(rx/of @resolutions)) - (rx/map (fn [resolutions] - {:libraries-resolution resolutions}))))))) diff --git a/frontend/test/frontend_tests/data/exports_files_test.cljs b/frontend/test/frontend_tests/data/exports_files_test.cljs new file mode 100644 index 0000000000..a979ac34d7 --- /dev/null +++ b/frontend/test/frontend_tests/data/exports_files_test.cljs @@ -0,0 +1,286 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns frontend-tests.data.exports-files-test + "Exporting files as jobs: one file, one job and one answer, with the + progress of the job reported while it runs." + (:require + [app.common.uuid :as uuid] + [app.main.data.exports.files :as fexp] + [app.main.repo :as rp] + [app.main.store :as st] + [app.util.websocket :as ws] + [beicon.v2.core :as rx] + [cljs.test :as t :include-macros true] + [frontend-tests.helpers.async :as hva] + [frontend-tests.helpers.mock :as mock])) + +(defn- file + [name] + {:id (uuid/next) + :name name + :project-id (uuid/next) + :is-shared false}) + +(defn- job + [job-id status & {:as extra}] + (merge {:id job-id :status status :name "export-binfile"} extra)) + +(defn- event + [job-id kind payload] + {:type :job-event + :job-id job-id + :kind kind + :payload payload}) + +(defn- message + [payload] + {:type :message :payload payload}) + +(defn- fake-server + "The backend of the export: the creation answers with a fresh job the + test drives, and the read answers the row of that job." + [calls rows] + (mock/stub + (fn [id params] + (case id + :create-export-binfile-job + (let [job-id (uuid/next)] + (swap! calls conj {:cmd id :params params :job-id job-id}) + (swap! rows assoc job-id (job job-id "running")) + (rx/of {:id job-id})) + + :get-job + (do + (swap! calls conj {:cmd id :params params}) + (rx/of (get @rows (:id params)))) + + (rx/of nil))))) + +(def ^:private created? + (comp #{:create-export-binfile-job} :cmd)) + +(defn- answered? + "True when the file already has its answer: the artifact or an error." + [results file-id] + (some (fn [result] + (and (= file-id (:file-id result)) + (or (:uri result) (:error result)))) + results)) + +(t/deftest ^:async a-file-is-exported-as-its-own-job + (let [ws-stream (rx/subject) + calls (atom []) + rows (atom {}) + seen (atom []) + target (file "My file")] + + (await + (mock/with-mocks* + {st/state (atom {:ws-conn ws-stream}) + rp/cmd! (fake-server calls rows) + ws/get-rcv-stream (mock/stub (fn [_] ws-stream))} + + (let [done (hva/observe (fexp/export-files :files [target] :type :detach-libraries) + {:on-next #(swap! seen conj %)}) + job-id (:job-id (first @calls))] + + (t/testing "the file is exported by a job of its own" + (t/is (= :create-export-binfile-job (:cmd (first @calls)))) + (t/is (= {:params {:file-ids #{(:id target)} + :export-type :detach-libraries}} + (:params (first @calls))))) + + (t/testing "the milestone of the job is reported while it runs" + (rx/push! ws-stream + (message (event job-id :progress + {:stage :pages + :counters {:files {:current 1 :total 1} + :pages {:current 1 :total 2}}}))) + (await (hva/wait-for #(some :progress @seen) "the milestone")) + + (t/is (= {:file-id (:id target) + :progress {:stage :pages + :counters {:files {:current 1 :total 1} + :pages {:current 1 :total 2}}}} + (last @seen)))) + + (t/testing "the artifact of the job answers when it is over" + (swap! rows assoc job-id + (job job-id "completed" + :result {:resource-uri "http://assets/export" + :filename "export.penpot" + :mtype "application/zip" + :size 120})) + (rx/push! ws-stream (message (event job-id :end {:outcome "completed"}))) + (await done) + + (t/is (= {:file-id (:id target) + :uri "http://assets/export" + :filename "My file"} + (last @seen))))))))) + +(t/deftest ^:async a-failure-fails-only-its-own-file + (let [ws-stream (rx/subject) + calls (atom []) + rows (atom {}) + seen (atom []) + one (file "First") + other (file "Second")] + + (await + (mock/with-mocks* + {st/state (atom {:ws-conn ws-stream}) + rp/cmd! (fake-server calls rows) + ws/get-rcv-stream (mock/stub (fn [_] ws-stream))} + + (let [done (hva/observe (fexp/export-files :files [one other] :type :detach-libraries) + {:on-next #(swap! seen conj %)})] + + (t/testing "the first file answers with its artifact" + (let [job-id (:job-id (first (filter created? @calls)))] + (swap! rows assoc job-id + (job job-id "completed" :result {:resource-uri "http://assets/first"})) + (rx/push! ws-stream (message (event job-id :end {:outcome "completed"}))) + (await (hva/wait-for #(answered? @seen (:id one)) "the first answer")))) + + (t/testing "and the second one fails on its own" + (await (hva/wait-for #(= 2 (count (filter created? @calls))) "the second job")) + (let [job-id (:job-id (second (filter created? @calls)))] + (swap! rows assoc job-id + (job job-id "failed" :error {:type :internal + :code :boom + :hint "boom"})) + (rx/push! ws-stream (message (event job-id :end {:outcome "failed"}))) + (await done))) + + (t/testing "the failure does not touch the file that worked" + (t/is (= {:file-id (:id one) + :uri "http://assets/first" + :filename "First"} + (first (filter #(and (= (:id one) (:file-id %)) + (:uri %)) + @seen))))) + + (t/is (= {:file-id (:id other) + :error {:type :internal :code :boom :hint "boom"}} + (last @seen)))))))) + +(t/deftest ^:async the-created-jobs-reach-the-caller-tagged-with-their-file + (let [ws-stream (rx/subject) + calls (atom []) + rows (atom {}) + seen-jobs (atom []) + one (file "First") + other (file "Second")] + + (await + (mock/with-mocks* + {st/state (atom {:ws-conn ws-stream}) + rp/cmd! (fake-server calls rows) + ws/get-rcv-stream (mock/stub (fn [_] ws-stream))} + + (let [done (hva/observe (fexp/export-files :files [one other] + :type :detach-libraries + :on-job #(swap! seen-jobs conj %)) + {:on-next (constantly nil)})] + + (await (hva/wait-for #(= 1 (count (filter created? @calls))) "the first job")) + + (t/testing "the server recorded the creation with its job id" + (let [creates (filterv created? @calls)] + (t/is (= 1 (count creates))) + (t/is (= :create-export-binfile-job (:cmd (first creates)))) + (t/is (uuid? (:job-id (first creates)))))) + + (t/testing "the caller learns that same job tagged with its file" + (t/is (= [{:job-id (:job-id (first (filterv created? @calls))) + :file-id (:id one)}] + @seen-jobs))) + + ;; files export one after the other: the second job is only + ;; created once the first one is over + (let [{job-1 :job-id} (first @seen-jobs)] + (swap! rows assoc job-1 (job job-1 "completed" :result {:resource-uri "http://assets/first"})) + (rx/push! ws-stream (message (event job-1 :end {:outcome "completed"})))) + + (await (hva/wait-for #(= 2 (count (filter created? @calls))) "the second job")) + + (t/testing "the caller learns the second job too, tagged and in order" + (t/is (= 2 (count @seen-jobs))) + (t/is (= (:job-id (second (filterv created? @calls))) (:job-id (second @seen-jobs)))) + (t/is (= (:id other) (:file-id (second @seen-jobs))))) + + (let [{job-2 :job-id} (second @seen-jobs)] + (swap! rows assoc job-2 (job job-2 "completed" :result {:resource-uri "http://assets/second"})) + (rx/push! ws-stream (message (event job-2 :end {:outcome "completed"}))) + (await done))))))) + +(t/deftest ^:async every-file-is-queued-until-its-first-milestone + (let [ws-stream (rx/subject) + calls (atom []) + rows (atom {}) + seen (atom []) + one (file "First") + other (file "Second")] + + (await + (mock/with-mocks* + {st/state (atom {:ws-conn ws-stream}) + rp/cmd! (fake-server calls rows) + ws/get-rcv-stream (mock/stub (fn [_] ws-stream))} + + (let [done (hva/observe (fexp/export-files :files [one other] :type :detach-libraries) + {:on-next #(swap! seen conj %)})] + + (t/testing "every file queues up front, before any of its jobs exists" + (await (hva/wait-for #(= 2 (count (filter :queued @seen))) "the queued messages")) + (t/is (= [{:file-id (:id one) :queued true} + {:file-id (:id other) :queued true}] + (vec (filter :queued @seen))))) + + (t/testing "a worker picking a job up is not a milestone and ends nothing" + (let [job-id (:job-id (first (filter created? @calls)))] + (rx/push! ws-stream (message (event job-id :start {:attempt 1}))) + (t/is (not-any? :started @seen)))) + + ;; finish the first job so the second one is created and the run ends + (let [job-1 (:job-id (first (filter created? @calls)))] + (swap! rows assoc job-1 + (job job-1 "completed" :result {:resource-uri "http://assets/first"})) + (rx/push! ws-stream (message (event job-1 :end {:outcome "completed"})))) + + (await (hva/wait-for #(= 2 (count (filter created? @calls))) "the second job")) + + (let [job-2 (:job-id (second (filter created? @calls)))] + (swap! rows assoc job-2 + (job job-2 "completed" :result {:resource-uri "http://assets/second"})) + (rx/push! ws-stream (message (event job-2 :end {:outcome "completed"}))) + (await done))))))) + +(t/deftest ^:async a-cancelled-job-marks-the-file-cancelled-without-artifact + (let [ws-stream (rx/subject) + calls (atom []) + rows (atom {}) + seen (atom []) + target (file "My file")] + + (await + (mock/with-mocks* + {st/state (atom {:ws-conn ws-stream}) + rp/cmd! (fake-server calls rows) + ws/get-rcv-stream (mock/stub (fn [_] ws-stream))} + + (let [done (hva/observe (fexp/export-files :files [target] :type :detach-libraries) + {:on-next #(swap! seen conj %)}) + job-id (:job-id (first @calls))] + + (t/testing "the cancelled outcome is an explicit message, never an artifact" + (swap! rows assoc job-id (job job-id "cancelled")) + (rx/push! ws-stream (message (event job-id :end {:outcome "cancelled"}))) + (await done) + (t/is (= {:file-id (:id target) :cancelled true} (last @seen))) + (t/is (not-any? :uri @seen)))))))) diff --git a/frontend/test/frontend_tests/data/imports_test.cljs b/frontend/test/frontend_tests/data/imports_test.cljs new file mode 100644 index 0000000000..e73802fe38 --- /dev/null +++ b/frontend/test/frontend_tests/data/imports_test.cljs @@ -0,0 +1,533 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns frontend-tests.data.imports-test + "Reading a penpot package and importing it as a job." + (:require + [app.common.json :as json] + [app.common.uuid :as uuid] + [app.main.data.imports :as imp] + [app.main.data.jobs :as dj] + [app.main.data.uploads :as uploads] + [app.main.repo :as rp] + [app.main.store :as st] + [app.util.http :as http] + [app.util.i18n :refer [tr]] + [app.util.websocket :as ws] + [app.util.zip :as uz] + [beicon.v2.core :as rx] + [cljs.test :as t :include-macros true] + [frontend-tests.helpers.async :as hva] + [frontend-tests.helpers.mock :as mock] + [promesa.core :as p])) + +(def ^:private project-id + (uuid/next)) + +(def ^:private uri + "blob:package") + +(def ^:private zip-magic + (js/Uint8Array. #js [0x50 0x4b 0x03 0x04])) + +(defn- entry + "An entry of the wizard, as the analysis leaves it." + [& {:keys [id name type] + :or {type :binfile-v3}}] + {:file-id (or id (uuid/next)) + :uri uri + :name (or name "My file") + :type type}) + +(defn- job + [job-id status & {:as extra}] + (merge {:id job-id :status status :name "import-binfile"} extra)) + +(defn- event + [job-id kind payload] + {:type :job-event + :job-id job-id + :kind kind + :payload payload}) + +(defn- message + [payload] + {:type :message :payload payload}) + +(defn- slurp-stub + "The transport of a request that answers with `body`." + [body] + (mock/stub (fn [_] (rx/of {:body body})))) + +(defn- upload-stub + "The chunked upload of a package. It is variadic, so it is a plain fn." + [session-id] + (fn [_ & _] (rx/of {:session-id session-id}))) + +(defn- upload-with-progress-stub + "The chunked upload of a package that reports two chunks before + emitting the session, so the test sees the upload phase. It is + variadic, so it is a plain fn." + [session-id] + (fn [_ & {:keys [on-progress]}] + (when (fn? on-progress) + (on-progress {:current 1 :total 2}) + (on-progress {:current 2 :total 2})) + (rx/of {:session-id session-id}))) + +(defn- ws-stub + [stream] + (mock/stub (fn [_] stream))) + +(defn- fake-server + "The backend of the import: the creation answers with a fresh job the + test drives, and the read answers the row of that job." + [calls rows create-row] + (mock/stub + (fn [id params] + (case id + :create-import-binfile-job + (let [job-id (uuid/next)] + (swap! calls conj {:cmd id :params params :job-id job-id}) + (swap! rows assoc job-id (create-row job-id)) + (rx/of {:id job-id})) + + :get-job + (do + (swap! calls conj {:cmd id :params params}) + (rx/of (get @rows (:id params)))) + + (rx/of nil))))) + +(defn- manifest + [& files] + (json/encode {:type "penpot/export-files" + :files (mapv (fn [[id name]] {:id id :name name}) files)})) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; READING A PACKAGE +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(t/deftest ^:async a-package-without-a-manifest-is-rejected + (let [seen (atom [])] + + (await + (mock/with-mocks* + {http/send! (slurp-stub zip-magic) + uz/get-entry (mock/stub (fn [_ _] (p/resolved nil))) + uz/close (mock/stub (fn [_] nil))} + + (await (hva/observe (imp/analyze [{:uri uri :name "broken.penpot"}]) + {:on-next #(swap! seen conj %)})) + + (t/testing "the entry carries the error of the reader" + (t/is (= 1 (count @seen))) + (t/is (= :error (:status (first @seen)))) + (t/is (string? (:error (first @seen))))))))) + +(t/deftest ^:async a-version-3-package-answers-one-entry-per-file + (let [first-id (uuid/next) + second-id (uuid/next) + seen (atom [])] + + (await + (mock/with-mocks* + {http/send! (slurp-stub zip-magic) + uz/get-entry (mock/stub (fn [_ _] (p/resolved ::entry))) + uz/read-as-text (mock/stub (fn [_] (p/resolved (manifest [first-id "First"] + [second-id "Second"])))) + uz/close (mock/stub (fn [_] nil))} + + (await (hva/observe (imp/analyze [{:uri uri :name "package.penpot"}]) + {:on-next #(swap! seen conj %)})) + + (t/testing "every file of the manifest is an entry of the wizard" + (t/is (= [first-id second-id] (mapv :file-id @seen))) + (t/is (= ["First" "Second"] (mapv :name @seen))) + (t/is (every? #(= :binfile-v3 (:type %)) @seen)) + (t/is (every? #(= :success (:status %)) @seen))) + + (t/testing "all of them share the upload of the file that was dropped" + (t/is (every? #(= uri (:uri %)) @seen))))))) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; IMPORTING A PACKAGE +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(t/deftest ^:async a-package-is-imported-as-its-own-job + (let [ws-stream (rx/subject) + calls (atom []) + rows (atom {}) + seen (atom []) + session-id (uuid/next) + resolution {:done [{:id (str (uuid/next)) :name "Published"}]} + one (entry :name "First") + other (entry :name "Second")] + + (await + (mock/with-mocks* + {st/state (atom {:ws-conn ws-stream}) + http/send! (slurp-stub ::blob) + uploads/upload-blob-chunked (upload-stub session-id) + rp/cmd! (fake-server calls rows (fn [job-id] (job job-id "running"))) + ws/get-rcv-stream (ws-stub ws-stream)} + + (let [done (hva/observe (imp/import-files {:project-id project-id + :entries [one other]}) + {:on-next #(swap! seen conj %)}) + job-id (:job-id (first @calls))] + + (t/testing "the package is uploaded and imported as one job" + (t/is (= :create-import-binfile-job (:cmd (first @calls)))) + (t/is (= {:project-id project-id :name "First" :version 3} + (:params (:params (first @calls))))) + (t/is (= session-id (:upload-id (:params (first @calls)))))) + + (t/testing "the milestone of the job reaches every entry of the package" + (rx/push! ws-stream + (message (event job-id :progress + {:stage :pages + :counters {:files {:current 1 :total 2} + :pages {:current 1 :total 8}}}))) + (await (hva/wait-for #(= 2 (count (filter (comp #{:progress} :status) @seen))) + "the milestone")) + + (t/is (= #{(:file-id one) (:file-id other)} + (into #{} (comp (filter (comp #{:progress} :status)) (map :file-id)) @seen))) + + (t/is (every? (fn [message] + (= {:stage :pages + :counters {:files {:current 1 :total 2} + :pages {:current 1 :total 8}}} + (:progress message))) + (filter (comp #{:progress} :status) @seen)))) + + (t/testing "the entries finish with the resolution of the package" + (swap! rows assoc job-id + (job job-id "completed" + :result {:file-ids (mapv (comp str :file-id) [one other]) + :resolution resolution + :name "First" + :version 3})) + (rx/push! ws-stream (message (event job-id :end {:outcome "completed"}))) + (await done) + + (t/is (= #{(:file-id one) (:file-id other)} + (into #{} (comp (filter (comp #{:finish} :status)) (map :file-id)) @seen))) + + (t/is (= {:libraries-resolution resolution} (last @seen))))))))) + +(t/deftest ^:async the-job-reports-queued-then-started-before-progress + (let [ws-stream (rx/subject) + calls (atom []) + rows (atom {}) + seen (atom []) + session-id (uuid/next) + one (entry :name "First") + other (entry :name "Second")] + + (await + (mock/with-mocks* + {st/state (atom {:ws-conn ws-stream}) + http/send! (slurp-stub ::blob) + uploads/upload-blob-chunked (upload-stub session-id) + rp/cmd! (fake-server calls rows (fn [job-id] (job job-id "running"))) + ws/get-rcv-stream (ws-stub ws-stream)} + + (let [done (hva/observe (imp/import-files {:project-id project-id + :entries [one other]}) + {:on-next #(swap! seen conj %)}) + job-id (:job-id (first @calls))] + + (t/testing "every entry is queued while the job waits for a worker" + (await (hva/wait-for #(= 2 (count (filter (comp #{:queued} :status) @seen))) + "the queued messages")) + (t/is (= #{(:file-id one) (:file-id other)} + (into #{} (comp (filter (comp #{:queued} :status)) (map :file-id)) @seen)))) + + (t/testing "a started job moves the entries to progress without a milestone" + (rx/push! ws-stream (message (event job-id :start {:attempt 1}))) + (await (hva/wait-for #(= 2 (count (filter (comp #{:started} :status) @seen))) + "the started messages")) + (t/is (= #{(:file-id one) (:file-id other)} + (into #{} (comp (filter (comp #{:started} :status)) (map :file-id)) @seen)))) + + (t/testing "the milestone still arrives on top" + (rx/push! ws-stream + (message (event job-id :progress + {:stage :pages + :counters {:pages {:current 1 :total 8}}}))) + (await (hva/wait-for #(some :progress (filter (comp #{:progress} :status) @seen)) + "the milestone")) + (swap! rows assoc job-id + (job job-id "completed" + :result {:file-ids (mapv (comp str :file-id) [one other]) + :resolution {} + :name "First" + :version 3})) + (rx/push! ws-stream (message (event job-id :end {:outcome "completed"}))) + (await done))))))) + +(t/deftest ^:async a-failed-job-fails-its-entries + (let [ws-stream (rx/subject) + calls (atom []) + rows (atom {}) + seen (atom []) + target (entry :name "Only")] + + (await + (mock/with-mocks* + {st/state (atom {:ws-conn ws-stream}) + http/send! (slurp-stub ::blob) + uploads/upload-blob-chunked (upload-stub (uuid/next)) + rp/cmd! (fake-server calls rows (fn [job-id] (job job-id "running"))) + ws/get-rcv-stream (ws-stub ws-stream)} + + (let [done (hva/observe (imp/import-files {:project-id project-id + :entries [target]}) + {:on-next #(swap! seen conj %)}) + job-id (:job-id (first @calls))] + + (swap! rows assoc job-id + (job job-id "failed" :error {:type :internal + :code :boom + :hint "The package is broken"})) + (rx/push! ws-stream (message (event job-id :end {:outcome "failed"}))) + (await done) + + (t/testing "the entry fails with the message of the job" + (t/is (= [{:status :queued + :file-id (:file-id target)} + {:status :error + :file-id (:file-id target) + :error "The package is broken"} + {:libraries-resolution {}}] + @seen)))))))) + +(t/deftest ^:async a-package-that-cannot-be-created-fails-its-entries + (let [ws-stream (rx/subject) + seen (atom []) + target (entry :name "Only")] + + (await + (mock/with-mocks* + {st/state (atom {:ws-conn ws-stream}) + http/send! (slurp-stub ::blob) + uploads/upload-blob-chunked (upload-stub (uuid/next)) + rp/cmd! (mock/stub (fn [_ _] + (rx/throw (ex-info "cannot create the job" + {:type :validation + :code :boom + :hint "Not now"})))) + ws/get-rcv-stream (ws-stub ws-stream)} + + (await (hva/observe (imp/import-files {:project-id project-id + :entries [target]}) + {:on-next #(swap! seen conj %)})) + + (t/testing "the failure reaches every entry of the package" + (t/is (= [{:status :error + :file-id (:file-id target) + :error "Not now"} + {:libraries-resolution {}}] + @seen))))))) + +(t/deftest ^:async a-job-that-already-ended-does-not-leave-the-entries-spinning + (let [ws-stream (rx/subject) + calls (atom []) + rows (atom {}) + seen (atom []) + target (entry :name "Only")] + + (await + (mock/with-mocks* + {st/state (atom {:ws-conn ws-stream}) + http/send! (slurp-stub ::blob) + uploads/upload-blob-chunked (upload-stub (uuid/next)) + ;; the job is over before the client reads it for the first time + rp/cmd! (fake-server calls rows + (fn [job-id] (job job-id "completed" :result {:resolution {}}))) + ws/get-rcv-stream (ws-stub ws-stream)} + + (await (hva/observe (imp/import-files {:project-id project-id + :entries [target]}) + {:on-next #(swap! seen conj %)})) + + (t/testing "the entry finishes without waiting for an event" + (t/is (= [{:status :queued + :file-id (:file-id target)} + {:status :finish + :file-id (:file-id target)} + {:libraries-resolution {}}] + @seen))))))) + +(t/deftest ^:async the-upload-reports-chunks-before-the-job-runs + (let [ws-stream (rx/subject) + calls (atom []) + rows (atom {}) + seen (atom []) + session-id (uuid/next) + target (entry :name "Only")] + + (await + (mock/with-mocks* + {st/state (atom {:ws-conn ws-stream}) + http/send! (slurp-stub ::blob) + uploads/upload-blob-chunked (upload-with-progress-stub session-id) + rp/cmd! (fake-server calls rows (fn [job-id] (job job-id "running"))) + ws/get-rcv-stream (ws-stub ws-stream)} + + (let [done (hva/observe (imp/import-files {:project-id project-id + :entries [target]}) + {:on-next #(swap! seen conj %)}) + job-id (:job-id (first @calls))] + + (await (hva/wait-for #(>= (count (filter (comp #{:progress} :status) @seen)) 2) + "the upload progress")) + + (t/testing "every uploaded chunk reaches the entry as an upload milestone" + (t/is (= [{:status :progress + :file-id (:file-id target) + :progress {:stage :upload + :counters {:upload {:current 1 :total 2}}}} + {:status :progress + :file-id (:file-id target) + :progress {:stage :upload + :counters {:upload {:current 2 :total 2}}}}] + (filterv (comp #{:progress} :status) @seen)))) + + (t/testing "the entry still finishes with the job" + (swap! rows assoc job-id (job job-id "completed" :result {:resolution {}})) + (rx/push! ws-stream (message (event job-id :end {:outcome "completed"}))) + (await done) + + (t/is (= {:libraries-resolution {}} (last @seen))) + (t/is (= :finish (:status (nth @seen 3)))))))))) + +(t/deftest ^:async a-cancelled-job-closes-its-entries-as-cancelled + (let [ws-stream (rx/subject) + calls (atom []) + rows (atom {}) + seen (atom []) + target (entry :name "Only")] + + (await + (mock/with-mocks* + {st/state (atom {:ws-conn ws-stream}) + http/send! (slurp-stub ::blob) + uploads/upload-blob-chunked (upload-stub (uuid/next)) + rp/cmd! (fake-server calls rows (fn [job-id] (job job-id "running"))) + ws/get-rcv-stream (ws-stub ws-stream)} + + (let [done (hva/observe (imp/import-files {:project-id project-id + :entries [target]}) + {:on-next #(swap! seen conj %)}) + job-id (:job-id (first @calls))] + + (t/testing "the cancelled ending terminates the stream" + (swap! rows assoc job-id (job job-id "cancelled")) + (rx/push! ws-stream (message (event job-id :end {:outcome "cancelled"}))) + (await done) + + (t/testing "the cancelled ending closes the entries and terminates the stream" + (t/is (= [{:status :queued + :file-id (:file-id target)} + {:status :error + :file-id (:file-id target) + :error (tr "jobs.import-cancelled")} + {:libraries-resolution {}}] + @seen))))))))) + +(t/deftest ^:async a-cancelled-job-closes-in-progress-entries-as-cancelled + (let [ws-stream (rx/subject) + calls (atom []) + rows (atom {}) + seen (atom []) + target (entry :name "Only")] + + (await + (mock/with-mocks* + {st/state (atom {:ws-conn ws-stream}) + http/send! (slurp-stub ::blob) + uploads/upload-blob-chunked (upload-stub (uuid/next)) + rp/cmd! (fake-server calls rows (fn [job-id] (job job-id "running"))) + ws/get-rcv-stream (ws-stub ws-stream)} + + (let [done (hva/observe (imp/import-files {:project-id project-id + :entries [target]}) + {:on-next #(swap! seen conj %)}) + job-id (:job-id (first @calls))] + + ;; drive the entries to progress first, so the cancel + ;; interrupts a run in course instead of a queued job + (rx/push! ws-stream (message (event job-id :progress {:stage :manifest}))) + (await (hva/wait-for #(some (fn [message] (= :progress (:status message))) @seen) + "the progress")) + + (t/testing "an external cancel closes every entry as cancelled" + (swap! rows assoc job-id (job job-id "cancelled")) + (rx/push! ws-stream (message (event job-id :end {:outcome "cancelled"}))) + (await done) + (t/is (= [{:status :queued + :file-id (:file-id target)} + {:status :progress + :file-id (:file-id target) + :progress {:stage :manifest}} + {:status :error + :file-id (:file-id target) + :error (tr "jobs.import-cancelled")} + {:libraries-resolution {}}] + @seen)))))))) + +(t/deftest ^:async the-created-job-id-reaches-the-caller + (let [ws-stream (rx/subject) + calls (atom []) + rows (atom {}) + seen-jobs (atom []) + target (entry :name "Only")] + + (await + (mock/with-mocks* + {st/state (atom {:ws-conn ws-stream}) + http/send! (slurp-stub ::blob) + uploads/upload-blob-chunked (upload-stub (uuid/next)) + rp/cmd! (fake-server calls rows (fn [job-id] (job job-id "running"))) + ws/get-rcv-stream (ws-stub ws-stream)} + + (let [done (hva/observe (imp/import-files {:project-id project-id + :entries [target] + :on-job #(swap! seen-jobs conj %)}) + {:on-next (constantly nil)}) + job-id (:job-id (first @calls))] + + (t/testing "the caller learns the id of the package job" + (t/is (= [job-id] @seen-jobs))) + + (swap! rows assoc job-id (job job-id "completed" :result {:resolution {}})) + (rx/push! ws-stream (message (event job-id :end {:outcome "completed"}))) + (await done)))))) + +(t/deftest ^:async cancel-job-asks-the-server-and-ignores-failures + (let [calls (atom []) + job-id (uuid/next)] + + (t/testing "the cancel command reaches the server" + (await + (mock/with-mocks* + {rp/cmd! (mock/stub (fn [id params] + (swap! calls conj {:cmd id :params params}) + (rx/of {:id job-id :status "cancelled"})))} + (dj/cancel-job job-id) + (await (hva/settle)) + (t/is (= [{:cmd :cancel-job :params {:id job-id}}] @calls))))) + + (t/testing "a failure is swallowed: the job may have just finished" + (await + (mock/with-mocks* + {rp/cmd! (mock/stub (fn [_ _] + (rx/throw (ex-info "gone" {}))))} + (dj/cancel-job job-id) + (await (hva/settle))))))) diff --git a/frontend/test/frontend_tests/data/jobs_test.cljs b/frontend/test/frontend_tests/data/jobs_test.cljs new file mode 100644 index 0000000000..996d9d1f0b --- /dev/null +++ b/frontend/test/frontend_tests/data/jobs_test.cljs @@ -0,0 +1,176 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns frontend-tests.data.jobs-test + "Following a job: the progress the websocket publishes and the outcome + the row holds." + (:require + [app.common.uuid :as uuid] + [app.main.data.jobs :as dj] + [app.main.repo :as rp] + [app.util.http :as http] + [app.util.websocket :as ws] + [beicon.v2.core :as rx] + [cljs.test :as t :include-macros true] + [cuerdas.core :as str] + [frontend-tests.helpers.async :as hva] + [frontend-tests.helpers.mock :as mock])) + +(def ^:private job-id + (uuid/next)) + +(defn- job + "The row of a job, as `get-job` answers it." + [status & {:as extra}] + (merge {:id job-id :status status :name "export-binfile"} extra)) + +(defn- event + ([kind payload] (event job-id kind payload)) + ([id kind payload] + {:type :job-event + :job-id id + :kind kind + :payload payload})) + +(defn- message + [payload] + {:type :message :payload payload}) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; FOLLOWING A JOB +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(t/deftest ^:async progress-events-are-emitted-and-the-outcome-ends-the-stream + (let [ws-stream (rx/subject) + row (atom (job "running")) + seen (atom [])] + + (await + (mock/with-mocks* + {rp/cmd! (mock/stub (fn [_ _] (rx/of @row))) + ws/get-rcv-stream (mock/stub (fn [_] ws-stream))} + + (let [done (hva/observe (dj/watch-job nil job-id) + {:on-next #(swap! seen conj %)})] + + (rx/push! ws-stream + (message (event :progress {:stage :pages + :counters {:pages {:current 1 :total 2}}}))) + + (t/testing "an event of another job is not part of this stream" + (rx/push! ws-stream + (message (event (uuid/next) :progress {:stage :media}))) + (await (hva/settle)) + (t/is (= [{:kind :progress + :payload {:stage :pages + :counters {:pages {:current 1 :total 2}}}}] + @seen))) + + (t/testing "the closing event makes the stream read the row and end" + (reset! row (job "completed" :result {:resource-uri "uri"})) + (rx/push! ws-stream (message (event :end {:outcome "completed"}))) + (await done) + + (t/is (= [{:kind :progress + :payload {:stage :pages + :counters {:pages {:current 1 :total 2}}}} + {:status "completed" :result {:resource-uri "uri"}}] + @seen)))))))) + +(t/deftest ^:async a-job-that-already-ended-needs-no-event + (let [ws-stream (rx/subject) + seen (atom [])] + + (await + (mock/with-mocks* + {rp/cmd! (mock/stub (fn [_ _] + (rx/of (job "completed" :result {:resource-uri "uri"})))) + ws/get-rcv-stream (mock/stub (fn [_] ws-stream))} + + (let [done (hva/observe (dj/watch-job nil job-id) + {:on-next #(swap! seen conj %)})] + (await done) + + (t/testing "the first read of the row is the outcome" + (t/is (= [{:status "completed" :result {:resource-uri "uri"}}] + @seen)))))))) + +(t/deftest ^:async a-reconnect-reads-the-row-again + (let [ws-stream (rx/subject) + row (atom (job "running")) + seen (atom [])] + + (await + (mock/with-mocks* + {rp/cmd! (mock/stub (fn [_ _] (rx/of @row))) + ws/get-rcv-stream (mock/stub (fn [_] ws-stream))} + + (let [done (hva/observe (dj/watch-job nil job-id) + {:on-next #(swap! seen conj %)})] + (await (hva/settle)) + + (t/testing "a job that is still running reports nothing" + (t/is (= [] @seen))) + + (t/testing "a reconnect answers for a job that failed while away" + (reset! row (job "failed" :error {:type :internal + :code :boom + :hint "boom"})) + (rx/push! ws-stream {:type :opened}) + (await done) + + (t/is (= [{:status "failed" + :error {:type :internal :code :boom :hint "boom"}}] + @seen)))))))) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; THE COMMANDS THAT FEED IT +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(defn- capture-requests! + [requests] + (fn [request] + (swap! requests conj request) + (rx/of {:status 200 + :uri (:uri request) + :headers {"content-type" "application/json"} + :body "{}"}))) + +(t/deftest ^:async the-commands-point-at-the-jobs-api + (let [requests (atom [])] + + (await + (mock/with-mocks* + {http/fetch (mock/stub (capture-requests! requests)) + http/response->map (mock/stub identity) + http/process-response-type (mock/stub (fn [_ response] (rx/of response)))} + + (await (hva/->promise (rp/cmd! :create-export-binfile-job + {:file-ids #{job-id} :export-type :detach-libraries}))) + (await (hva/->promise (rp/cmd! :create-import-binfile-job {:project-id (str job-id)}))) + (await (hva/->promise (rp/cmd! :get-job {:id (str job-id)}))) + (await (hva/->promise (rp/cmd! :create-export-job {}))) + + (let [[export import read exporter] @requests] + (t/testing "the binfile export asks the backend, not the exporter service" + (t/is (= :post (:method export))) + (t/is (str/ends-with? (:path (:uri export)) + "/api/main/methods/create-export-binfile-job"))) + + (t/testing "the import job needs no override: its name is the method" + (t/is (= :post (:method import))) + (t/is (str/ends-with? (:path (:uri import)) + "/api/main/methods/create-import-binfile-job"))) + + (t/testing "the read is a get with the id of the job" + (t/is (= :get (:method read))) + (t/is (str/ends-with? (:path (:uri read)) + "/api/main/methods/get-job")) + (t/is (= (str job-id) (:id (:query read))))) + + (t/testing "and the exporter keeps its own name" + (t/is (= :post (:method exporter))) + (t/is (str/ends-with? (:path (:uri exporter)) "/api/export/jobs")))))))) diff --git a/frontend/test/frontend_tests/data/uploads_test.cljs b/frontend/test/frontend_tests/data/uploads_test.cljs index 0a7806b5cc..5dcec770e4 100644 --- a/frontend/test/frontend_tests/data/uploads_test.cljs +++ b/frontend/test/frontend_tests/data/uploads_test.cljs @@ -13,6 +13,7 @@ [app.main.data.uploads :as uploads] [beicon.v2.core :as rx] [cljs.test :as t :include-macros true] + [frontend-tests.helpers.async :as hva] [frontend-tests.helpers.http :as http])) ;; --------------------------------------------------------------------------- @@ -115,3 +116,42 @@ (http/restore-fetch! orig) (t/is (= 3 @chunk-calls)) (done)))))))) + +(t/deftest ^:async upload-blob-chunked-reports-progress-per-chunk + (let [session-id (uuid/next) + ;; Three chunks of 10 bytes: 2 full + 1 partial + blob (make-blob 25) + seen (atom []) + + fetch-mock + (fn [url _opts] + (let [cmd (http/url->cmd url)] + (js/Promise.resolve + (case cmd + :create-upload-session + (http/make-transit-response + {:session-id session-id}) + + :upload-chunk + (http/make-transit-response + {:session-id session-id :index 0}) + + (http/make-json-response + {:error (str "unexpected cmd: " cmd)}))))) + + orig (http/install-fetch-mock! fetch-mock)] + + (await + (hva/observe (uploads/upload-blob-chunked blob + :chunk-size 10 + :on-progress #(swap! seen conj %)) + {:on-next (fn [{:keys [session-id]}] + (t/is (uuid? session-id)))})) + (http/restore-fetch! orig) + + (t/testing "every uploaded chunk is reported with uploaded/total" + ;; chunks upload one at a time, so they are reported in order + (t/is (= [{:current 1 :total 3} + {:current 2 :total 3} + {:current 3 :total 3}] + @seen))))) diff --git a/frontend/test/frontend_tests/runner.cljs b/frontend/test/frontend_tests/runner.cljs index 501973b6e8..a12c63a205 100644 --- a/frontend/test/frontend_tests/runner.cljs +++ b/frontend/test/frontend_tests/runner.cljs @@ -12,6 +12,9 @@ [frontend-tests.data.dashboard-test] [frontend-tests.data.event-test] [frontend-tests.data.exports-assets-test] + [frontend-tests.data.exports-files-test] + [frontend-tests.data.imports-test] + [frontend-tests.data.jobs-test] [frontend-tests.data.nitrate-test] [frontend-tests.data.persistence-retry-test] [frontend-tests.data.persistence-test] @@ -112,6 +115,7 @@ [frontend-tests.ui.history-test] [frontend-tests.ui.inspect-geometry-test] [frontend-tests.ui.inspect-stroke-tokens-test] + [frontend-tests.ui.jobs-progress-test] [frontend-tests.ui.layout-container-multiple-test] [frontend-tests.ui.measurements-test] [frontend-tests.ui.measures-menu-props-test] @@ -162,6 +166,9 @@ 'frontend-tests.data.repo-test 'frontend-tests.data.store-test 'frontend-tests.data.exports-assets-test + 'frontend-tests.data.exports-files-test + 'frontend-tests.data.imports-test + 'frontend-tests.data.jobs-test 'frontend-tests.data.svg-upload-test 'frontend-tests.data.uploads-test 'frontend-tests.data.viewer-test @@ -254,6 +261,7 @@ 'frontend-tests.ui.history-test 'frontend-tests.ui.inspect-geometry-test 'frontend-tests.ui.inspect-stroke-tokens-test + 'frontend-tests.ui.jobs-progress-test 'frontend-tests.ui.layout-container-multiple-test 'frontend-tests.ui.measurements-test 'frontend-tests.ui.measures-menu-props-test diff --git a/frontend/test/frontend_tests/ui/jobs_progress_test.cljs b/frontend/test/frontend_tests/ui/jobs_progress_test.cljs new file mode 100644 index 0000000000..9d25f852af --- /dev/null +++ b/frontend/test/frontend_tests/ui/jobs_progress_test.cljs @@ -0,0 +1,44 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns frontend-tests.ui.jobs-progress-test + "How a job milestone is written: the stage and its counter, and the + upload stage as a percentage of the file." + (:require + [app.main.ui.jobs.progress :as jp] + [app.util.i18n :as i18n] + [cljs.test :as t :include-macros true] + [frontend-tests.helpers.mock :as mock])) + +(def ^:private stub-tr + (mock/stub (fn [k] (str "TR:" k)))) + +(t/deftest a-counter-milestone-writes-its-stage-and-counter + (with-redefs [i18n/tr stub-tr] + (t/is (= "TR:jobs.progress.stage.pages 3/8" + (jp/milestone-text {:stage :pages :counters {:pages {:current 3 :total 8}}}))) + (t/testing "a counter with no total writes the bare current" + (t/is (= "TR:jobs.progress.stage.pages 3" + (jp/milestone-text {:stage :pages :counters {:pages {:current 3}}})))))) + +(t/deftest a-file-milestone-appends-the-file-counter + (with-redefs [i18n/tr stub-tr] + (t/is (= "TR:jobs.progress.stage.pages 3/8 · TR:jobs.progress.stage.files 1/2" + (jp/milestone-text {:stage :pages + :counters {:pages {:current 3 :total 8} + :files {:current 1 :total 2}}} + :file? true))))) + +(t/deftest an-upload-milestone-writes-the-percentage-of-the-file + (with-redefs [i18n/tr stub-tr] + (t/testing "the chunks sent become a whole percentage" + (t/is (= "TR:jobs.progress.stage.upload (33%)" + (jp/milestone-text {:stage :upload :counters {:upload {:current 1 :total 3}}}))) + (t/is (= "TR:jobs.progress.stage.upload (100%)" + (jp/milestone-text {:stage :upload :counters {:upload {:current 9 :total 9}}})))) + (t/testing "an upload with no total keeps the bare label" + (t/is (= "TR:jobs.progress.stage.upload" + (jp/milestone-text {:stage :upload :counters {:upload {:current 1}}})))))) diff --git a/frontend/test/frontend_tests/util_zip_test.cljs b/frontend/test/frontend_tests/util_zip_test.cljs index a9fbc658f7..6621335a8d 100644 --- a/frontend/test/frontend_tests/util_zip_test.cljs +++ b/frontend/test/frontend_tests/util_zip_test.cljs @@ -7,38 +7,9 @@ (ns frontend-tests.util-zip-test (:require [app.util.zip :as-alias uz] - [app.worker.import :as worker.import] - [beicon.v2.core :as rx] - [cljs.test :as t :include-macros true] - [frontend-tests.helpers.mock :as mock] - [promesa.core :as p])) + [cljs.test :as t :include-macros true])) (t/deftest read-as-text-nil-entry-raises-typed-error (t/testing "read-as-text guards against nil entry" (t/is (thrown-with-msg? js/Error #"nil" (app.util.zip/read-as-text nil))))) - -(t/deftest read-zip-manifest-missing-throws-validation-error - (t/async - done - (t/testing "read-zip-manifest rejects ZIPs without manifest.json" - (mock/with-mocks - {app.util.zip/get-entry (mock/stub (fn [_ _] (p/resolved nil)))} - (fn [done'] - (->> (worker.import/read-zip-manifest #js {}) - (rx/subs! - (fn [_] - (t/is false "expected validation error to be thrown") - (done')) - (fn [err] - (let [data (ex-data err)] - (t/is (= :invalid-penpot-file (:code data)) - "missing manifest.json raises typed :invalid-penpot-file error") - (t/is (string? (:hint data)) - "missing manifest.json error carries a :hint") - (t/is (re-find #"manifest\.json" (:hint data)) - "missing manifest.json :hint mentions manifest.json") - (t/is (nil? (re-find #"getData" (:hint data))) - "missing manifest.json :hint does not leak the raw TypeError text") - (done')))))) - done)))) \ No newline at end of file diff --git a/frontend/translations/en.po b/frontend/translations/en.po index c38fce16ae..d71621ab02 100644 --- a/frontend/translations/en.po +++ b/frontend/translations/en.po @@ -715,34 +715,6 @@ msgstr "Next file" msgid "dashboard.import.no-perms" msgstr "You don’t have permission to import to this team" -#: src/app/main/ui/dashboard/import.cljs:134 -msgid "dashboard.import.progress.process-colors" -msgstr "Processing colors" - -#: src/app/main/ui/dashboard/import.cljs:143, src/app/main/ui/dashboard/import.cljs:146 -msgid "dashboard.import.progress.process-components" -msgstr "Processing components" - -#: src/app/main/ui/dashboard/import.cljs:140 -msgid "dashboard.import.progress.process-media" -msgstr "Processing media" - -#: src/app/main/ui/dashboard/import.cljs:131 -msgid "dashboard.import.progress.process-page" -msgstr "Processing page: %s" - -#: src/app/main/ui/dashboard/import.cljs:137 -msgid "dashboard.import.progress.process-typographies" -msgstr "Processing typographies" - -#: src/app/main/ui/dashboard/import.cljs:125 -msgid "dashboard.import.progress.upload-data" -msgstr "Uploading data to server (%s/%s)" - -#: src/app/main/ui/dashboard/import.cljs:128 -msgid "dashboard.import.progress.upload-media" -msgstr "Uploading file: %s" - #: src/app/main/ui/dashboard/import.cljs:443 msgid "dashboard.import.resolve-libraries" msgstr "" @@ -2193,18 +2165,6 @@ msgstr "Size and position" msgid "inspect.attributes.stroke" msgstr "Stroke" -#, permanent, unused -msgid "inspect.attributes.stroke.alignment.center" -msgstr "Center" - -#, permanent, unused -msgid "inspect.attributes.stroke.alignment.inner" -msgstr "Inside" - -#, permanent, unused -msgid "inspect.attributes.stroke.alignment.outer" -msgstr "Outside" - #: src/app/main/ui/inspect/attributes/text.cljs:53, src/app/main/ui/inspect/attributes/text.cljs:158 msgid "inspect.attributes.typography" msgstr "Typography" @@ -2614,6 +2574,74 @@ msgstr "The token will expire on %s" msgid "integrations.token.will-not-expire" msgstr "The token has no expiration date" +#: src/app/main/ui/exports/files.cljs:111 +msgid "jobs.export-cancelled" +msgstr "Export cancelled" + +#: src/app/main/data/imports.cljs:259 +msgid "jobs.import-cancelled" +msgstr "Import cancelled" + +#: src/app/main/ui/jobs/progress.cljs:26 +msgid "jobs.progress.stage.colors" +msgstr "Processing the colors" + +#: src/app/main/ui/jobs/progress.cljs:25 +msgid "jobs.progress.stage.components" +msgstr "Processing the components" + +#: src/app/main/ui/jobs/progress.cljs:21 +msgid "jobs.progress.stage.files" +msgstr "Processing the files" + +#: src/app/main/ui/jobs/progress.cljs:32 +msgid "jobs.progress.stage.manifest" +msgstr "Reading the package" + +#: src/app/main/ui/jobs/progress.cljs:23 +msgid "jobs.progress.stage.media" +msgstr "Processing the media" + +#: src/app/main/ui/jobs/progress.cljs:31 +msgid "jobs.progress.stage.metadata" +msgstr "Reading the metadata" + +#: src/app/main/ui/jobs/progress.cljs:22 +msgid "jobs.progress.stage.pages" +msgstr "Processing the pages" + +#: src/app/main/ui/jobs/progress.cljs:30 +msgid "jobs.progress.stage.relations" +msgstr "Processing the library relations" + +#: src/app/main/ui/jobs/progress.cljs:29 +msgid "jobs.progress.stage.storage-objects" +msgstr "Processing the storage objects" + +#: src/app/main/ui/jobs/progress.cljs:24 +msgid "jobs.progress.stage.thumbnails" +msgstr "Processing the thumbnails" + +#: src/app/main/ui/jobs/progress.cljs:28 +msgid "jobs.progress.stage.tokens" +msgstr "Processing the tokens" + +#: src/app/main/ui/jobs/progress.cljs:27 +msgid "jobs.progress.stage.typographies" +msgstr "Processing the typographies" + +#: src/app/main/ui/jobs/progress.cljs:33 +msgid "jobs.progress.stage.unknown" +msgstr "Working" + +#: src/app/main/ui/jobs/progress.cljs:33 +msgid "jobs.progress.stage.upload" +msgstr "Uploading file…" + +#: src/app/main/ui/dashboard/import.cljs:328, src/app/main/ui/exports/files.cljs:103 +msgid "jobs.queued" +msgstr "Queued" + #: src/app/main/ui/dashboard/comments.cljs:94 msgid "label.mark-all-as-read" msgstr "Mark all as read" @@ -2844,10 +2872,6 @@ msgstr "Download %s" msgid "labels.download-simple" msgstr "Download" -#: src/app/main/ui/exports/files.cljs:217 -msgid "labels.downloading-file" -msgstr "Downloading file…" - #: src/app/main/ui/dashboard/file_menu.cljs:28, src/app/main/ui/dashboard/files.cljs:86, src/app/main/ui/dashboard/files.cljs:207, src/app/main/ui/dashboard/projects.cljs:225, src/app/main/ui/dashboard/projects.cljs:228, src/app/main/ui/dashboard/sidebar.cljs:362 msgid "labels.drafts" msgstr "Drafts" @@ -3439,10 +3463,6 @@ msgstr "Upload custom fonts" msgid "labels.uploading" msgstr "Uploading…" -#: src/app/main/ui/dashboard/import.cljs:648 -msgid "labels.uploading-file" -msgstr "Uploading file…" - #: src/app/main/ui/inspect/right_sidebar.cljs:65, src/app/main/ui/workspace/sidebar/options/menus/component.cljs:1034 msgid "labels.variant" msgstr "Variant" @@ -6251,10 +6271,6 @@ msgstr "Typographies" msgid "workspace.assets.typography.add-typography" msgstr "Add typography" -#: src/app/main/ui/workspace/sidebar/assets/typographies.cljs:461 -msgid "workspace.assets.typography.multiple-texts-selected" -msgstr "Select one text to create a typography style" - #: src/app/main/ui/workspace/sidebar/options/menus/typography.cljs:855 msgid "workspace.assets.typography.font-size" msgstr "Size" @@ -6275,6 +6291,10 @@ msgstr "Letter Spacing" msgid "workspace.assets.typography.line-height" msgstr "Line Height" +#: src/app/main/ui/workspace/sidebar/assets/typographies.cljs:461 +msgid "workspace.assets.typography.multiple-texts-selected" +msgstr "Select one text to create a typography style" + #: src/app/main/ui/dashboard/grid.cljs:226, src/app/main/ui/workspace/libraries.cljs:1070, src/app/main/ui/workspace/sidebar/options/menus/typography.cljs:802, src/app/main/ui/workspace/sidebar/options/menus/typography.cljs:838, src/app/main/ui/workspace/sidebar/options/menus/typography.cljs:957, src/app/main/ui/workspace/sidebar/options/menus/typography.cljs:976 msgid "workspace.assets.typography.sample" msgstr "Ag" @@ -9214,10 +9234,6 @@ msgstr "Line (%s)" msgid "workspace.toolbar.mcp" msgstr "MCP" -#: src/app/main/ui/workspace/top_toolbar.cljs:345 -msgid "workspace.toolbar.mcp-connect-here" -msgstr "Connect here" - #: src/app/main/ui/workspace/top_toolbar.cljs:342 msgid "workspace.toolbar.mcp-connected" msgstr "MCP connected" diff --git a/frontend/translations/es.po b/frontend/translations/es.po index 00ea23e0b5..c1464fe980 100644 --- a/frontend/translations/es.po +++ b/frontend/translations/es.po @@ -726,34 +726,6 @@ msgstr "Siguiente archivo" msgid "dashboard.import.no-perms" msgstr "No tienes permisos para importar en este equipo" -#: src/app/main/ui/dashboard/import.cljs:134 -msgid "dashboard.import.progress.process-colors" -msgstr "Procesando colores" - -#: src/app/main/ui/dashboard/import.cljs:143, src/app/main/ui/dashboard/import.cljs:146 -msgid "dashboard.import.progress.process-components" -msgstr "Procesando componentes" - -#: src/app/main/ui/dashboard/import.cljs:140 -msgid "dashboard.import.progress.process-media" -msgstr "Procesando media" - -#: src/app/main/ui/dashboard/import.cljs:131 -msgid "dashboard.import.progress.process-page" -msgstr "Procesando página: %s" - -#: src/app/main/ui/dashboard/import.cljs:137 -msgid "dashboard.import.progress.process-typographies" -msgstr "Procesando tipografías" - -#: src/app/main/ui/dashboard/import.cljs:125 -msgid "dashboard.import.progress.upload-data" -msgstr "Enviando datos al servidor (%s/%s)" - -#: src/app/main/ui/dashboard/import.cljs:128 -msgid "dashboard.import.progress.upload-media" -msgstr "Enviando fichero: %s" - #: src/app/main/ui/dashboard/import.cljs:443 msgid "dashboard.import.resolve-libraries" msgstr "" @@ -1542,6 +1514,12 @@ msgstr "" msgid "errors.comment-error" msgstr "Ha habido un error con el comentario" +#: src/app/main/errors.cljs:499 +msgid "errors.connection-error" +msgstr "" +"No se puede conectar con el servidor. Comprueba tu conexión e inténtalo de " +"nuevo." + #: src/app/main/errors.cljs:851 msgid "errors.deprecated" msgstr "" @@ -1782,6 +1760,10 @@ msgstr "" "informar del error y recarga para continuar desde los últimos cambios " "guardados." +#: src/app/main/data/persistence.cljs:307 +msgid "errors.save-retrying" +msgstr "Se ha perdido la conexión. Reintentando guardar los cambios." + #: src/app/main/errors.cljs:773 msgid "errors.svg-parser.invalid-svg" msgstr "El SVG no es válido o está mal formado" @@ -2224,18 +2206,6 @@ msgstr "Tamaño y posición" msgid "inspect.attributes.stroke" msgstr "Borde" -#, permanent, unused -msgid "inspect.attributes.stroke.alignment.center" -msgstr "Centro" - -#, permanent, unused -msgid "inspect.attributes.stroke.alignment.inner" -msgstr "Interior" - -#, permanent, unused -msgid "inspect.attributes.stroke.alignment.outer" -msgstr "Exterior" - #: src/app/main/ui/inspect/attributes/text.cljs:53, src/app/main/ui/inspect/attributes/text.cljs:158 msgid "inspect.attributes.typography" msgstr "Tipografía" @@ -2647,6 +2617,74 @@ msgstr "El token expirará el %s" msgid "integrations.token.will-not-expire" msgstr "El token no tiene fecha de expiración" +#: src/app/main/ui/exports/files.cljs:111 +msgid "jobs.export-cancelled" +msgstr "Exportación cancelada" + +#: src/app/main/data/imports.cljs:259 +msgid "jobs.import-cancelled" +msgstr "Importación cancelada" + +#: src/app/main/ui/jobs/progress.cljs:26 +msgid "jobs.progress.stage.colors" +msgstr "Procesando colores" + +#: src/app/main/ui/jobs/progress.cljs:25 +msgid "jobs.progress.stage.components" +msgstr "Procesando componentes" + +#: src/app/main/ui/jobs/progress.cljs:21 +msgid "jobs.progress.stage.files" +msgstr "Procesando archivos" + +#: src/app/main/ui/jobs/progress.cljs:32 +msgid "jobs.progress.stage.manifest" +msgstr "Leyendo el paquete" + +#: src/app/main/ui/jobs/progress.cljs:23 +msgid "jobs.progress.stage.media" +msgstr "Procesando media" + +#: src/app/main/ui/jobs/progress.cljs:31 +msgid "jobs.progress.stage.metadata" +msgstr "Leyendo metadatos" + +#: src/app/main/ui/jobs/progress.cljs:22 +msgid "jobs.progress.stage.pages" +msgstr "Procesando páginas" + +#: src/app/main/ui/jobs/progress.cljs:30 +msgid "jobs.progress.stage.relations" +msgstr "Procesando relaciones de biblioteca" + +#: src/app/main/ui/jobs/progress.cljs:29 +msgid "jobs.progress.stage.storage-objects" +msgstr "Procesando objetos de almacenamiento" + +#: src/app/main/ui/jobs/progress.cljs:24 +msgid "jobs.progress.stage.thumbnails" +msgstr "Procesando miniaturas" + +#: src/app/main/ui/jobs/progress.cljs:28 +msgid "jobs.progress.stage.tokens" +msgstr "Procesando tokens" + +#: src/app/main/ui/jobs/progress.cljs:27 +msgid "jobs.progress.stage.typographies" +msgstr "Procesando tipografías" + +#: src/app/main/ui/jobs/progress.cljs:33 +msgid "jobs.progress.stage.unknown" +msgstr "Trabajando" + +#: src/app/main/ui/jobs/progress.cljs:33 +msgid "jobs.progress.stage.upload" +msgstr "Subiendo archivo…" + +#: src/app/main/ui/dashboard/import.cljs:328, src/app/main/ui/exports/files.cljs:103 +msgid "jobs.queued" +msgstr "En cola" + #: src/app/main/ui/dashboard/comments.cljs:94 msgid "label.mark-all-as-read" msgstr "Marcar todo como leído" @@ -2877,10 +2915,6 @@ msgstr "Descargar %s" msgid "labels.download-simple" msgstr "Descargar" -#: src/app/main/ui/exports/files.cljs:217 -msgid "labels.downloading-file" -msgstr "Descargando archivo…" - #: src/app/main/ui/dashboard/file_menu.cljs:28, src/app/main/ui/dashboard/files.cljs:86, src/app/main/ui/dashboard/files.cljs:207, src/app/main/ui/dashboard/projects.cljs:225, src/app/main/ui/dashboard/projects.cljs:228, src/app/main/ui/dashboard/sidebar.cljs:362 msgid "labels.drafts" msgstr "Borradores" @@ -3476,10 +3510,6 @@ msgstr "Cargar fuente" msgid "labels.uploading" msgstr "Subiendo…" -#: src/app/main/ui/dashboard/import.cljs:648 -msgid "labels.uploading-file" -msgstr "Subiendo archivo…" - #: src/app/main/ui/inspect/right_sidebar.cljs:65, src/app/main/ui/workspace/sidebar/options/menus/component.cljs:1034 msgid "labels.variant" msgstr "Variante" @@ -4986,6 +5016,14 @@ msgstr "Cancelar" msgid "shortcuts.export-shapes" msgstr "Exportar elementos" +#: src/app/main/data/workspace/shortcuts.cljs:184 +msgid "shortcuts.find" +msgstr "Buscar" + +#: src/app/main/data/workspace/shortcuts.cljs:188 +msgid "shortcuts.find-and-replace" +msgstr "Buscar y reemplazar" + #: src/app/main/data/workspace/grid_layout/shortcuts.cljs:75, src/app/main/data/workspace/path/shortcuts.cljs:226, src/app/main/data/workspace/shortcuts.cljs:711 msgid "shortcuts.fit-all" msgstr "Zoom abarcar todo" @@ -5222,6 +5260,10 @@ msgstr "Pegar" msgid "shortcuts.paste-props" msgstr "Pegar propiedades" +#: src/app/main/data/workspace/shortcuts.cljs:125 +msgid "shortcuts.paste-replace" +msgstr "Pegar y reemplazar" + #: src/app/main/ui/settings/shortcuts.cljs:296, src/app/main/ui/settings/shortcuts.cljs:523 msgid "shortcuts.personalized" msgstr "Personalizados" @@ -6320,10 +6362,6 @@ msgstr "Tipografías" msgid "workspace.assets.typography.add-typography" msgstr "Añadir tipografía" -#: src/app/main/ui/workspace/sidebar/assets/typographies.cljs:461 -msgid "workspace.assets.typography.multiple-texts-selected" -msgstr "Selecciona un solo texto para crear un estilo de tipografía" - #: src/app/main/ui/workspace/sidebar/options/menus/typography.cljs:855 msgid "workspace.assets.typography.font-size" msgstr "Tamaño" @@ -6344,6 +6382,10 @@ msgstr "Interletrado" msgid "workspace.assets.typography.line-height" msgstr "Interlineado" +#: src/app/main/ui/workspace/sidebar/assets/typographies.cljs:461 +msgid "workspace.assets.typography.multiple-texts-selected" +msgstr "Selecciona un solo texto para crear un estilo de tipografía" + #: src/app/main/ui/dashboard/grid.cljs:226, src/app/main/ui/workspace/libraries.cljs:1070, src/app/main/ui/workspace/sidebar/options/menus/typography.cljs:802, src/app/main/ui/workspace/sidebar/options/menus/typography.cljs:838, src/app/main/ui/workspace/sidebar/options/menus/typography.cljs:957, src/app/main/ui/workspace/sidebar/options/menus/typography.cljs:976 msgid "workspace.assets.typography.sample" msgstr "Ag" @@ -6670,6 +6712,10 @@ msgstr "Desbloquear guías" msgid "workspace.header.reset-zoom" msgstr "Restablecer" +#: src/app/main/ui/workspace/left_header.cljs:125 +msgid "workspace.header.retrying" +msgstr "Reintentando guardar" + #: src/app/main/ui/workspace/left_header.cljs:126 msgid "workspace.header.save-error" msgstr "Error al guardar" @@ -9161,6 +9207,10 @@ msgstr "Tamaño" msgid "workspace.tokens.source" msgstr "Fuente:" +#: src/app/main/ui/workspace/tokens/management/context_menu.cljs:311, src/app/main/ui/workspace/tokens/management/context_menu.cljs:335 +msgid "workspace.tokens.stroke-width" +msgstr "Ancho del trazo" + #: src/app/main/ui/workspace/tokens/management/forms/form_container.cljs:54, src/app/main/ui/workspace/tokens/management/forms/typography.cljs:161 msgid "workspace.tokens.text-case-value-enter" msgstr "none | uppercase | lowercase | capitalize o {alias}" @@ -9297,14 +9347,22 @@ msgstr "Línea (%s)" msgid "workspace.toolbar.mcp" msgstr "MCP" -#: src/app/main/ui/workspace/top_toolbar.cljs:345 -msgid "workspace.toolbar.mcp-connect-here" -msgstr "Conectar aquí" - #: src/app/main/ui/workspace/top_toolbar.cljs:342 msgid "workspace.toolbar.mcp-connected" msgstr "MCP conectado" +#: src/app/main/ui/workspace/top_toolbar.cljs +msgid "workspace.toolbar.mcp-copy-session-id" +msgstr "Copiar ID de sesión" + +#: src/app/main/ui/workspace/top_toolbar.cljs +msgid "workspace.toolbar.mcp-session-copied" +msgstr "ID de sesión copiado" + +#: src/app/main/ui/workspace/top_toolbar.cljs +msgid "workspace.toolbar.mcp-session-id" +msgstr "ID de sesión: %s" + #: src/app/main/ui/workspace/top_toolbar.cljs:65, src/app/main/ui/workspace/top_toolbar.cljs:425 msgid "workspace.toolbar.move" msgstr "Mover (%s)"