diff --git a/.serena/memories/backend/core.md b/.serena/memories/backend/core.md index 9431fe4d00..cabb7ab55d 100644 --- a/.serena/memories/backend/core.md +++ b/.serena/memories/backend/core.md @@ -70,7 +70,7 @@ Every background job is a job-def: a plain `(defn execute-X [cfg params] ...)` i - The domain cfg of a binfile request lives in `app.binfile.common` (`export-cfg`, `import-cfg`), so the legacy RPC and the jobs freeze the same ids, resolve the same team features and apply the same anti-zip-bomb limits. `resolve-export-type` is the translation the legacy RPC needs for its deprecated boolean flags; the creation command asks the caller for the type instead of guessing a default. - Two retention knobs, easy to confuse: `:jobs-retention` sweeps terminal internal rows (the ones without a profile), while `:jobs-user-ttl` is the default expiry a user-facing job gets when it is created; the jobs GC deletes the row once `expires_at` passes and touches its resource. The heavy work of a user job runs on its own `:binfile` queue (`:worker-binfile-parallelism`, default 1) so it does not block the default runner. - The quote of a family of jobs (`::export-jobs-per-profile`, `::import-jobs-per-profile`) counts the jobs of the profile that are still on their way (`new`/`scheduled`/`running`/`retry`) whose name the registry gives for that family, so a new job of the family is covered without touching the quote. -- The `:exporter` queue has no runner in the backend: its consumer is the exporter process (a separate Node worker; see `mem:exporter/core`). The backend serves it through `app.rpc.management.jobs`, shared-key authenticated (`X-Shared-Key: exporter[:] `, the key the exporter derives from the same secret with a per-tenant salt; the bare `exporter` id is the legacy form): `claim-job {job-id scheduled-at}` (exact `scheduled-at` match; a stale payload answers `{:action :skip :status ...}`; the row's params come back decoded with the job-def, so the worker gets typed values — a row whose def is gone answers the stored shapes), `report-job-progress {job-id progress {stage counters}}` (the worker owns the milestone words; the substrate validates the shape), `create-job-session {job-id}` (a real `http_session_v2` of the owner profile, so the assets fetch as the user; TTL `:job-session-ttl` default 1h), `complete-job` (the params form carries `result`/`resource-id`; the artifact form carries a multipart `content` part, the single source of `filename`/`mtype`, declared with `::rpc/multipart-only-params` so a transit body cannot name a server path — path confined by `media.v/check-upload-path`, allowlist `common.media/export-artifact-types`, cap `:exporter-max-result-size`; the checks run before any bytes are stored and a lost race stores nothing) and `fail-job {job-id error session-id}`. Every method answers `{:action :skip :status ...}` on a row it cannot write and reports the row status, and a settle named with `session-id` closes that render session whatever the write says — only when the session belongs to the job's `profile-id`. Every method also checks the row belongs to the calling exporter's tenant (suffix of a per-tenant key id, own tenant for the bare id): anything else answers a bare `{:action :skip}` (no status, no write, no session close), so no oracle leaks tenant rows. `report-job-progress` carries the tenant in the heartbeat write itself (`:tenant` on `jobs/heartbeat`, filtered in the touch and progress statements), so the hot path needs no row read; a filter that matches nothing raises `:interrupt` with `:reason :foreign-tenant`. The tenant is normalized to lowercase when the configuration is read (`PENPOT_TENANT=ACME` behaves as `acme`), and `app.setup.keys/derive-exporter-key` is the single definition of the exporter key on the backend (registry and middleware both call it). +- The `:exporter` queue has no runner in the backend: its consumer is the exporter process (a separate Node worker; see `mem:exporter/core`). The backend serves it through `app.rpc.management.jobs`, shared-key authenticated (`X-Shared-Key: exporter[:] `, the key the exporter derives from the same secret with a per-tenant salt; the bare `exporter` id is the legacy form): `claim-job {job-id scheduled-at}` (exact `scheduled-at` match; a stale payload answers `{:action :skip :status ...}`; the row's params come back decoded with the job-def, so the worker gets typed values — a row whose def is gone answers the stored shapes), `report-job-progress {job-id progress {stage counters}}` (the worker owns the milestone words; the substrate validates the shape), `create-job-session {job-id}` (a real `http_session_v2` of the owner profile, so the assets fetch as the user; TTL `:job-session-ttl` default 1h), `complete-job` (the params form carries `result`/`resource-id`; the artifact form carries a multipart `content` part, the single source of `filename`/`mtype`, declared with `::rpc/multipart-only-params` so a transit body cannot name a server path — path confined by `media.v/check-upload-path`, allowlist `common.media/export-artifact-types`, cap `:exporter-max-result-size`; the checks run before any bytes are stored and a lost race stores nothing) and `fail-job {job-id error session-id}`. Every method answers `{:action :skip :status ...}` on a row it cannot write and reports the row status, and a settle named with `session-id` closes that render session whatever the write says — only when the session belongs to the job's `profile-id`. Every method also checks the row belongs to the calling exporter's tenant (suffix of a per-tenant key id, own tenant for the bare id): anything else answers a bare `{:action :skip}` (no status, no write, no session close), so no oracle leaks tenant rows; `complete-job` reads the row once at entry and gates there, before the completable/owned branches, so a foreign row that already ended or has no owner leaks nothing either. `report-job-progress` carries the tenant in the heartbeat write itself (`:tenant` on `jobs/heartbeat`, filtered in the touch and progress statements), so the hot path needs no row read; a filter that matches nothing raises `:interrupt` with `:reason :foreign-tenant`. The tenant is normalized to lowercase when the configuration is read (`PENPOT_TENANT=ACME` behaves as `acme`), and `app.setup.keys/derive-exporter-key` is the single definition of the exporter key on the backend (registry and middleware both call it). - A user-facing job is created through `app.rpc.commands.jobs`. One command per job type: `create-export-binfile-job`, `create-export-assets-job` and `create-import-binfile-job` name the job they create and their body carries only `:params`, whose schema the command imports from the job namespace (`::sm/params`), so the RPC validates the business params up front, before the command runs; the job-def is still the place that owns them and validates them again at run time. The commands check the permission and the quote of the family before the expensive part — binfile checks the files it reads by id, and assets checks every item on its own: an item carrying `:share-id` validates through the share-link permissions the same way `get-file-fragment` does (a logged viewer of the share is enough; the command stays authenticated, so the anonymous visitor is refused, the same as the exporter cookie it replaces) —, submit with `max-retries 0`, an expiry from `:jobs-user-ttl` (or ten minutes when the creation says `:transient`, the single-use exports of the clipboard and the Plugin API) and the queue the job-def routes to through `::jobs/queue-name` (`:binfile` when the def says nothing; read by the command, never by the substrate), and answer with a summary of the row. Assets params carry an optional `:kind` (`:shapes`/`:frames`, declared by the frames dialog; absent means shapes): the worker routes on it instead of inferring the export from the items. The import command has its own creation schema (`import-binfile/schema:create-params`, version optional) because the request is not the stored params: the command fills the version (from the caller or the package header) and the manifest metadata kept for audit. It assembles the upload, reads the manifest of a version 3 package for the audit (never its content) and stores the package in `job-resource` as the resource of the job, releasing it when the job never comes to own it. - `get-job` is the read surface a profile has over its own jobs: it answers with the same summary the creation commands return plus the `result` of a finished job or its public error (through `decode-job-error`, so the keywords come back), and a job of another profile is not distinguishable from one that does not exist: both answer `not-found`. `decode-row` decodes `params`, `result` and `error`, and the result is then decoded with the decoder the substrate compiled from the result schema the job-def the row names declares (`decode-result`; a def without one, or a row whose job-def is gone, keeps the plain JSON the row holds: a uuid as the text it is on disk, a uuid used as a map key as a keyword). The event log still has no read surface: a client that follows a job takes its progress from the websocket. - `heartbeat` uses named options `[cfg & {:keys [job-id progress] :as options}]` and returns the number of durable writes, so 0 means there was nothing to write: no job context, or the throttle did not allow it. A write that finds the job no longer active (terminal or gone) raises an `:interrupt` (`code :job-interrupted`, carrying the job id and the status it found) through the shared `jobs/check-active` instead of counting zero, so a task in course stops at its next beat; the interrupt is internal and never reaches HTTP (the management progress report answers `:skip` for it). The job id falls back to `::jobs/job-id` on the cfg and then to the runner-bound `*job-id*`. The `modified_at` touch is throttled to ~1s and the progress event to ~200ms, so the check fires at most once per second with no extra query (the read of the row happens only when a write already found nothing). A progress report is a milestone (`schema:progress`): a required `:stage` keyword plus an optional `:counters` map of `scope -> {:current, :total?}`, and no other key. Counters are non-negative, a `total` is positive and never lower than its `current`, and stage and counter keys are names of at most 64 characters. A milestone without counters is valid (a point of the run with no units to count), the stage vocabulary belongs to the worker and not to the substrate, and a client must degrade for a stage it does not know. Read a stored payload back with `jobs/decode-progress`: the database holds the keywords as strings. The 1s beat (down from 60s, so a cancelled heavy job is seen as gone at its next beat) costs one `modified_at` touch per beating job per second: cheap while the only heavy beater runs alone on its queue, revisit if concurrent beating jobs ever grow. @@ -94,7 +94,10 @@ dispatcher's guts. Verified against `app.worker/queue-key`, - Dispatcher order: the rows are marked `scheduled` FIRST, pushed LAST (RPUSH). A crash between the two leaves `scheduled` rows that `reschedule-lost-jobs` re-queues; the reverse order would leave a - duplicate payload nothing deduplicates. + duplicate payload nothing deduplicates. That sweep is on its way out: + `niwinz-consumer-generalization-1` replaces it with an abort sweep + (grace 5 min, cap 24 h, `LLEN` only inside the grace), so a consumer + must never rely on a lost payload being re-dispatched. - The `scheduled-at` in the payload must be handed back AS-IS on `claim-job`: the claim demands an exact match and answers `{:action :skip :status ...}` for a stale payload. That is what makes a claim idempotent and what turns diff --git a/.serena/memories/common/core.md b/.serena/memories/common/core.md index 3e82207713..459efde348 100644 --- a/.serena/memories/common/core.md +++ b/.serena/memories/common/core.md @@ -10,7 +10,8 @@ - `app.common.logic.*`: higher-level workflows/algorithms over files, shapes, components, variants, libraries, tokens, etc. - `app.common.geom.*`: geometry helpers and transformations. - `app.common.schema` / `app.common.schema.*`: Malli abstraction layer. -- `app.common.export`: asset-export job item/params schemas, the single source of truth the backend job-def and the exporter worker validate and decode against. +- `app.common.export`: asset-export job item/params schemas, the single source of truth the backend job-def and the exporter worker validate and decode against. `:suffix` is required (the renderer builds the file name with it) and `schema:params` references `schema:export-items` instead of restating the vector. +- `app.common.types.job`: the wire shape of a queue payload (`schema:queue-payload`, `[job-id scheduled-at]` as two strings), validated by the backend dispatcher before it encodes and by the exporter before it decodes. - `app.common.math`, `app.common.time`, `app.common.uuid`, `app.common.json`, etc.: cross-runtime utilities. - `app.common.test_helpers.*`: test builders and production-path helpers. diff --git a/.serena/memories/exporter/core.md b/.serena/memories/exporter/core.md index 0d4b2fd7ba..131bb3ca6c 100644 --- a/.serena/memories/exporter/core.md +++ b/.serena/memories/exporter/core.md @@ -31,7 +31,7 @@ on this side: the row of the job, its cancel and its result are the backend's. - Cancellation: a `skip` answer on any beat means the row is terminal (cancelled, aborted, settled). The runner raises `:job-cancelled` between units of work; mid-Skia the per-lease interval fires the SharedArrayBuffer signal and terminates the leased worker thread. The runner owns the local registry entry via `exporter.jobs` and releases it on settle. - `exporter.jobs` is the **local cancel arm only**: one flag per claimed job (`register`/`mark-cancelled` returning true/nil/`cancelled?`/`release`) plus one zero-arg check constructor per job (`check-cancelled`, raising `:job-cancelled`). No terminate callbacks and no shared-array signal travel through here — the abort belongs to whoever holds the render (the per-lease interval of `exporter.wasm.scope` terminates its own worker). - Temp files: `exporter.tmpdir/track`/`release` per job id; the `:exporter/tmpdir` init cleans orphans (aged `:job-ttl`). No redis-side store, cancel topic or abandoned-job sweep: an unclaimed-in-time row goes `aborted` by the backend's own lease GC. -- Run artifacts: `exporter.resources` (`create [tmpdir type name]`, `create-zip`/`add-to-zip`/`close-zip`); no per-file deletion timers — the tmpdir registry owns every path. +- Run artifacts: `exporter.resources` (`create [tmpdir type name]`, `create-zip`/`add-to-zip`/`close-zip`/`abort-zip`); no per-file deletion timers — the tmpdir registry owns every path. The write stream of a zip lives on the archiver and `abort-zip` destroys it: a render that fails or is cancelled would otherwise leak one descriptor per job, invisible because releasing the temp path does not close the handle. `close-zip` and `abort-zip` are idempotent with each other. ## Running it (devenv and deploy) @@ -44,15 +44,16 @@ on this side: the row of the job, its cancel and its result are the backend's. - Lifecycle is `exporter.utils.system`: async-aware Integrant-style init/halt (`init-key`/`halt-key` multimethods, `ref`s, `weavejester.dependency` ordering, `^:async` + `cljs.core/await`, no promesa chains); tests are `^:async deftest` with `await` directly in the body. - `exporter.main` mirrors the backend's `app.main`: a `system-config` def reading env at the wiring layer, a public `system` atom with the running map, `start`/`stop`/`restart` (`^:async`, resolving `:started`/`:stopped`/`:restarted`), a thread guard in `start` (a render worker thread runs this same bundle and boots the `exporter.wasm.worker` engine instead of the system — without it a warmed pool worker would nest pools and pollers and never post `ready`), `start-custom` as the test seam, idempotent `install-process-handlers` for the `uncaughtException`/SIGTERM/SIGINT wiring (signals drain through `shutdown`, whose `exit` is a test seam; runs on `start`, never on namespace load), `^:dev/before-load-async`/`^:dev/after-load-async` hooks bridging the promise lifecycle to shadow's `done` callbacks (a hook failure logs, never sticks the reload), and `main` as the entry point: shadow boots `exporter.main/main`. The entry leaves info logging enabled for the new tree (`setup!` on the `exporter` logger); without it the default level filters everything below fatal. - Service config maps take plain (non-namespaced) param keys (`{:exporter.browser/pool {:max 2}}`): the top-level key already scopes the component, so namespacing the params only adds noise. Component keys and error reasons stay namespaced. -- `exporter.config` is the process configuration: `PENPOT_`-prefixed env over defaults, malli-decoded and validated; `get`/`get-internal-uri`/`management-key` (HKDF)/`flags`/`version`. `prepare-config` over an explicit env object is pure and throws `:invalid-config` (tests exercise it); only the wiring entry `load-config!` reads the live environment and exits on it. +- `exporter.config` is the process configuration: `PENPOT_`-prefixed env over defaults, malli-decoded and validated; `get`/`get-internal-uri`/`management-key` (HKDF)/`flags`/`version`. `prepare-config` over an explicit env object is pure and throws `:invalid-config` (tests exercise it); only the wiring entry `load-config!` reads the live environment and exits on it. The tenant is lowercased before validation (schema `^[a-z0-9-]+$`, the backend rule), so `PENPOT_TENANT=ACME` and `acme` derive the same management key and pop the same queue. - `exporter.browser` is the browser pool service: the running instance is the raw pool, the public API takes it first (`exec` fails fast on a nil pool), the browser factory is injected through config for hermetic tests. - `exporter.wasm.pool` follows the backend `db/run!` shape: `run [cfg f & args]` resolves the pool from `cfg` (the `:exporter.wasm/pool` service instance, or a raw pool), acquires a worker, applies `f` to the cfg carrying it under `:exporter.wasm.pool/worker` plus any extra args, and returns the worker — or destroys it when `f` fails, like a rollback. Pure pool mechanics only (`init`/`halt`, `terminate`, `await-ready`); the render protocol lives one level up. -- `exporter.wasm.scope` owns the lease: `run [cfg on-scope]` lends one worker for the whole batch (serializer, per-lease signal buffer, mid-render abort interval), and the public low-level `render-on [cfg params]` renders on the leased worker — asserting `:worker-not-leased`/`:timeout-ms-missing` (`:assertion`, never a silent default: a bare `::key` resolves to the CURRENT ns, never to an alias, so cross-ns keys are literals or `::alias/key`) — with the rest preconfigured and namespaced by setter (`:exporter.renderer/on-object` from the caller, `:exporter.wasm.scope/cancel-buffer` from the lease); the check is mandatory at the lease entry and runs here when configured. +- `exporter.wasm.scope` owns the lease: `run [cfg on-scope]` lends one worker for the whole batch (serializer, per-lease signal buffer, mid-render abort interval), and the public low-level `render-on [cfg params]` renders on the leased worker — asserting `:worker-not-leased`/`:timeout-ms-missing` (`:assertion`, never a silent default: a bare `::key` resolves to the CURRENT ns, never to an alias, so cross-ns keys are literals or `::alias/key`) — with the rest preconfigured and namespaced by setter (`:exporter.renderer/on-object` from the caller, `:exporter.wasm.scope/cancel-buffer` from the lease); the `object`/`done` messages of a render go through a serializer, so `done` settles only after the last `on-object` resolved (the worker posts every object and the settle back to back); the check is mandatory at the lease entry and runs here when configured. - `exporter.renderer.browser` merges the bitmap/pdf/svg flows into one `render [cfg params on-object check-cancelled]` (branching by type, `on-object` always awaited, unknown types raise); `cfg` carries what the job params do not freeze (`:exporter.browser/pool`, `:base-uri`, `:public-uri`, `:svgo?`). - `exporter.browser.scope` owns the browser batch: `run [cfg exports]` fires every export through its own checkout, all at once, bounded by the pool. No lease — a browser batch shares nothing but the pool, so fan-out only. The facade validates once, partitions by backend and joins both scopes. - The wasm driver is `exporter.wasm.scope/run`, called from the facade; there is no `exporter.renderer.wasm` wrapper. -- `exporter.renderer` is the malli-validated facade with one entry point, the `render-fn` closure over `cfg`: it validates the task map (`:exports :on-object :check-cancelled`) and dispatches it through `render [cfg task]`, which assumes a valid task and only partitions and joins — the browser exports through `exporter.browser.scope`, the wasm ones through one leased worker — resolving nil once every export landed (objects travel through `on-object`). The `:exporter/renderer` service builds the closure from pools plus tmpdir (refs) and static config (`:base-uri`, `:public-uri`, `:management-key`, `:image-cache-size`, `:svgo?`). The facade merges the render env (`::wrender/env`) into every wasm export's params before the lease: the worker thread reads no config and no globals. The consumer renders through the injected fn and never requires the facade. `cfg` is task-free infrastructure with process lifetime (pools, uris, key, flags, tmpdir); everything per-task rides in the validated task map. One closed schema validates data plus injections, so a bad task — or a misspelled key — never touches a pool; task validation raises `:data-validation`. Each driver takes the same `[cfg params on-object check-cancelled]` shape (`headless?` is now `wasm?`: the browser also renders headless, so the old name lied) and implements cancel internally: cooperative checkpoints on the browser, checkpoints plus a mid-render abort of its own leased worker on wasm. The lease lives one level down, in `exporter.wasm.scope/run`, which the facade calls with the task keys set once in `cfg`; the browser batch goes through `exporter.browser.scope/run [cfg exports]`. -- `exporter.consumer.worker` is the queue poller (`read-payload`, `process`, `poll-once`, `start`/`stop` with a running flag, no promesa): K connections over the `:exporter` queue, claim → `run-export` through the injected `:renderer` and `:exporter/tmpdir` → settle before the next pop. The worker config carries the queue knobs plus both by ref; no render statics nest inside it. `stop` quits every connection and ends every loop; the temp area boots clean through the tmpdir init. +- `exporter.renderer` is the malli-validated facade with one entry point, the `render-fn` closure over `cfg`: it validates the task map (`:exports :on-object :check-cancelled`) and dispatches it through `render [cfg task]`, which assumes a valid task and only partitions and joins — the browser exports through `exporter.browser.scope`, the wasm ones through one leased worker — resolving nil once every export landed (objects travel through `on-object`). The `:exporter/renderer` service builds the closure from pools plus tmpdir (refs) and static config (`:base-uri`, `:public-uri`, `:image-cache-size`, `:svgo?`; the render paths carry no shared key). The facade merges the render env (`::wrender/env`) into every wasm export's params before the lease: the worker thread reads no config and no globals. The consumer renders through the injected fn and never requires the facade. `cfg` is task-free infrastructure with process lifetime (pools, uris, key, flags, tmpdir); everything per-task rides in the validated task map. One closed schema validates data plus injections, so a bad task — or a misspelled key — never touches a pool; task validation raises `:data-validation`. Each driver takes the same `[cfg params on-object check-cancelled]` shape (`headless?` is now `wasm?`: the browser also renders headless, so the old name lied) and implements cancel internally: cooperative checkpoints on the browser, checkpoints plus a mid-render abort of its own leased worker on wasm. The lease lives one level down, in `exporter.wasm.scope/run`, which the facade calls with the task keys set once in `cfg`; the browser batch goes through `exporter.browser.scope/run [cfg exports]`. +- The queue payload (`[job-id scheduled-at]`, a JSON array of two strings) is defined once in `app.common.types.job` (`schema:queue-payload`): the backend dispatcher validates before encoding and the worker validates before decoding, so a drift fails at push time. `sm/decoder` returns an invalid value untouched, so the worker checks the shape first. +- `exporter.consumer.worker` is the queue poller (`read-payload`, `process`, `poll-once`, `start`/`stop` with a running flag, no promesa): K connections over the `:exporter` queue, claim → `run-export` through the injected `:renderer` and `:exporter/tmpdir` → settle before the next pop. The worker config carries the queue knobs plus both by ref; no render statics nest inside it. `stop` falls the running flag, lets every poller finish the turn it already started (bounded by the pop timeout), and only then quits the connections; a payload popped after the flag fell goes back to its queue with `LPUSH`, because the shutdown must not start an export it is about to cut. The temp area boots clean through the tmpdir init. Do not assume the dispatcher re-queues a lost `scheduled` row: the `reschedule-lost` sweep of develop is replaced by an abort sweep in `niwinz-consumer-generalization-1`, so the payload must come back to the queue at the consumer. - `exporter.consumer.config` holds the consumer knobs (`concurrency` floored at 1, `queue-key`, `queue-name`): the runner, the worker and `exporter.main` read it. - `exporter.consumer.api` is the backend management client (claim/progress/session/settle, transit bodies, multipart artifact), with the same `fetch` def seam for hermetic tests. - `exporter.wasm` is the headless render-wasm driver under Node (artifact load + headless boot, font provisioning, shape rendering); `exporter.wasm.enums` binds the generated `shared.js` the `render-wasm/build export` step emits next to it (not committed). `exporter.wasm.text`/`serialize` own the browser-free call sequencing over the shared serializers. diff --git a/backend/src/app/rpc/management/jobs.clj b/backend/src/app/rpc/management/jobs.clj index da1737a681..a4f0354ef9 100644 --- a/backend/src/app/rpc/management/jobs.clj +++ b/backend/src/app/rpc/management/jobs.clj @@ -277,22 +277,20 @@ content) (defn- completable-job - "The row a completion may land on: running (or retrying), whatever - the JSON form asks for." - [cfg job-id] - (let [job (jobs/get-job cfg job-id)] - (when (and (some? job) - (contains? #{"running" "retry"} (:status job))) - job))) + "The row a completion may land on: running (or retrying), whatever the + form asks for." + [job] + (when (contains? #{"running" "retry"} (:status job)) + job)) (defn- artifact-owner - "The job an artifact completion may land on: completable AND owned, + "The row an artifact completion may land on: completable AND owned, because the object needs an owner and a job without one owns nothing (it may only settle without an artifact). Read before anything is stored: a lost race with a cancel never turns the bytes of the worker into an object." - [cfg job-id] - (let [job (completable-job cfg job-id)] + [job] + (let [job (completable-job job)] (when (and (some? job) (uuid? (:profile-id job))) job))) @@ -368,43 +366,43 @@ ::rpc/multipart-only-params [:content] ::rpc/allowed-shared-keys exporter-key?} [cfg {:keys [job-id result resource-id content session-id] :as params}] - (cond - ;; the multipart form stores the artifact itself: only for a job - ;; that can still complete and has an owner, because the object - ;; needs one - (some? content) - (if-let [job (artifact-owner cfg job-id)] - (if (tenant-ok? (::rpc/auth-key-id params) job) - (do - (check-artifact! content) - (complete-with-artifact cfg job content session-id)) - ;; a job of another tenant answers like a missing row, and its - ;; session is not ours to close - {:action :skip}) - (do - ;; a completion that lost the race is not a session to keep either - (close-job-session! cfg job-id session-id) - (skip-with-status cfg job-id))) - - ;; the params form: nothing to store here, the resource (if any) - ;; already lives in storage - :else - (if-let [job (completable-job cfg job-id)] - (if (tenant-ok? (::rpc/auth-key-id params) job) - (if (pos? (jobs/complete cfg - :job-id job-id - :result result - :resource-id resource-id)) + ;; One read, one gate: a job of another tenant answers like a missing + ;; row before any storage, write or session close, so neither its + ;; existence nor its state leaks. + (let [row (jobs/get-job cfg job-id)] + (if-not (tenant-ok? (::rpc/auth-key-id params) row) + {:action :skip} + (cond + ;; the multipart form stores the artifact itself: only for a job + ;; that can still complete and has an owner, because the object + ;; needs one + (some? content) + (if-let [job (artifact-owner row)] (do - (close-job-session! cfg job-id session-id) - {:action :run}) + (check-artifact! content) + (complete-with-artifact cfg job content session-id)) (do + ;; a completion that lost the race is not a session to keep either (close-job-session! cfg job-id session-id) (skip-with-status cfg job-id))) - {:action :skip}) - (do - (close-job-session! cfg job-id session-id) - (skip-with-status cfg job-id))))) + + ;; the params form: nothing to store here, the resource (if any) + ;; already lives in storage + :else + (if (completable-job row) + (if (pos? (jobs/complete cfg + :job-id job-id + :result result + :resource-id resource-id)) + (do + (close-job-session! cfg job-id session-id) + {:action :run}) + (do + (close-job-session! cfg job-id session-id) + (skip-with-status cfg job-id))) + (do + (close-job-session! cfg job-id session-id) + (skip-with-status cfg job-id))))))) ;; ---- RPC METHOD: FAIL-JOB diff --git a/backend/src/app/worker/dispatcher.clj b/backend/src/app/worker/dispatcher.clj index b084966714..a5a6f19cd4 100644 --- a/backend/src/app/worker/dispatcher.clj +++ b/backend/src/app/worker/dispatcher.clj @@ -11,6 +11,7 @@ [app.common.logging :as l] [app.common.schema :as sm] [app.common.time :as ct] + [app.common.types.job :as ctj] [app.config :as cf] [app.db :as db] [app.jobs :as jobs] @@ -100,8 +101,17 @@ RETURNING job.id, job.queue, job.name, job.created_at") SELECT unnest(?::uuid[]), 'end', ?::jsonb") (defn- encode-payload + "The payload the queue carries: the job id and its `scheduled-at`, in + the shape the consumer decodes (`app.common.types.job`). A drift + fails here, on the producer, instead of at the claim." [{:keys [id scheduled-at]}] - (json/encode [(str id) (ct/format-inst scheduled-at)])) + (let [payload [(str id) (ct/format-inst scheduled-at)]] + (when-not (ctj/valid-queue-payload? payload) + (throw (ex-info "invalid queue payload" + {:type :assertion + :code :invalid-queue-payload + :payload payload}))) + (json/encode payload))) (defn- reschedule-lost-jobs [{:keys [::db/conn ::timestamp ::wrk/tenant] :as cfg}] diff --git a/backend/test/backend_tests/rpc/commands/jobs_test.clj b/backend/test/backend_tests/rpc/commands/jobs_test.clj index 3d41d7ec73..752d0ae21a 100644 --- a/backend/test/backend_tests/rpc/commands/jobs_test.clj +++ b/backend/test/backend_tests/rpc/commands/jobs_test.clj @@ -223,8 +223,8 @@ :object-id (or object-id (uuid/next)) :type (or type :png) :name (or name "shape") + :suffix (or suffix "") :scale (or scale 1)} - suffix (assoc :suffix suffix) share-id (assoc :share-id share-id))) (defn- create-export-assets-job diff --git a/backend/test/backend_tests/rpc_management_jobs_test.clj b/backend/test/backend_tests/rpc_management_jobs_test.clj index d5697597f1..99b91c985d 100644 --- a/backend/test/backend_tests/rpc_management_jobs_test.clj +++ b/backend/test/backend_tests/rpc_management_jobs_test.clj @@ -216,6 +216,18 @@ (t/is (= {:action :skip :status "completed"} (:result out))) (t/is (= [] (get-progresss job-id))))) +(t/deftest report-job-progress-skips-a-missing-row + ;; The worker sends the beat with the id it claimed: a row that is + ;; gone answers a bare skip, the same contract the claim pins. + (let [job-id (uuid/next) + out (mgmt :report-job-progress {:job-id job-id + :progress {:stage :pages + :counters {:pages {:current 1 + :total 2}}}})] + (t/is (nil? (:error out))) + (t/is (= {:action :skip} (:result out))) + (t/is (zero? (count-job-events job-id))))) + (t/deftest report-job-progress-persists-rapid-reports (let [job-id (mk-job {}) _ (jobs/claim (make-cfg) job-id (:scheduled-at (th/db-get :job {:id job-id} :id :scheduled-at))) @@ -688,6 +700,33 @@ (t/is (= "running" (:status (get-row job-id)))) (t/is (nil? (:completed-at (get-row job-id)))))) +(t/deftest complete-job-skips-a-terminal-job-of-another-tenant + ;; The row already ended, so the multipart form does not even reach + ;; the "completable" branch: the tenant gate still answers first, and + ;; the state of a foreign row never leaks. + (let [profile (mk-profile) + job-id (mk-job {:status "completed" + :profile-id (:id profile) + :tenant "other-tenant"}) + owner-sess (session/create-session (get th/*system* :app.http.session/manager) + {:profile-id (:id profile)}) + out (multipart-complete job-id (artifact-tempfile "penpot") + :session-id (:id owner-sess))] + (t/is (nil? (:error out))) + (t/is (= {:action :skip} (:result out))) + (t/testing "nothing was stored, written or closed" + (t/is (zero? (count (th/db-exec! ["SELECT id FROM storage_object"])))) + (t/is (= "completed" (:status (get-row job-id)))) + (t/is (some? (get-session-row (:id owner-sess))))))) + +(t/deftest complete-job-params-form-skips-a-terminal-job-of-another-tenant + (let [job-id (mk-job {:status "completed" :tenant "other-tenant"}) + out (mgmt :complete-job {:job-id job-id + :result {:x 1}})] + (t/is (nil? (:error out))) + (t/is (= {:action :skip} (:result out))) + (t/is (= "completed" (:status (get-row job-id)))))) + (t/deftest complete-job-refuses-an-upload-outside-multipart ;; The upload map names a server path: only a request the multipart ;; parser flagged may carry one. A transit body is refused before any diff --git a/common/src/app/common/export.cljc b/common/src/app/common/export.cljc index dad8e37644..e387bdf51e 100644 --- a/common/src/app/common/export.cljc +++ b/common/src/app/common/export.cljc @@ -31,7 +31,7 @@ ;; a plain string on purpose, empty included: the export form sends ;; `""` when the user types no suffix, and the legacy `/api/export` ;; surface took it that way; the worker concatenates it as-is - [:suffix {:optional true} :string] + [:suffix :string] [:scale ::sm/number] [:share-id {:optional true} ::sm/uuid]]) @@ -48,7 +48,7 @@ joins pages into one pdf); absent it is a shapes export. The worker reads it as declared instead of inferring it from the items." [:map {:title "export-assets-params" :closed true} - [:exports [:vector {:min 1} schema:export-item]] + [:exports schema:export-items] [:kind {:optional true} [::sm/one-of #{:shapes :frames}]] [:name {:optional true} ::sm/text] [:skip-children {:optional true} ::sm/boolean] diff --git a/common/src/app/common/types/job.cljc b/common/src/app/common/types/job.cljc new file mode 100644 index 0000000000..2b7c3640d4 --- /dev/null +++ b/common/src/app/common/types/job.cljc @@ -0,0 +1,27 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns app.common.types.job + "The wire shapes of the job queue, shared by the process that pushes a + payload (the backend dispatcher) and the process that pops it (the + exporter worker). One definition on purpose: a drift between the two + fails at push time instead of at claim time." + (:require + [app.common.schema :as sm])) + +(def schema:queue-payload + "The two-element JSON array the dispatcher pushes and the consumer + pops: the job id as text and the `scheduled-at` of the row when it was + pushed, in `ct/format-inst` shape. Fewer than two items, other types + or a different container is not a payload: the consumer drops it + instead of claiming an invalid job." + [:tuple :string :string]) + +(def valid-queue-payload? + (sm/validator schema:queue-payload)) + +(def decode-queue-payload + (sm/decoder schema:queue-payload sm/string-transformer)) diff --git a/common/test/common_tests/export_test.cljc b/common/test/common_tests/export_test.cljc index c35ca0edc2..1eee248908 100644 --- a/common/test/common_tests/export_test.cljc +++ b/common/test/common_tests/export_test.cljc @@ -52,6 +52,14 @@ :scale 2}] (t/is (= typed (decode typed))))) +(t/deftest an-item-without-suffix-is-not-a-valid-export + ;; The renderer needs the suffix always (it builds the file name), so + ;; the contract requires it: the job is refused at creation instead of + ;; failing at render time. + (let [item (-> stored-item decode (dissoc :suffix))] + (t/is (not (sm/valid? export/schema:params {:exports [item]}))) + (t/is (sm/valid? export/schema:params {:exports [(assoc item :suffix "")]})))) + (t/deftest params-schema-owns-kind-and-options (let [decode-params (sm/decoder export/schema:params sm/json-transformer) params (decode-params {:exports [stored-item] diff --git a/common/test/common_tests/runner.cljc b/common/test/common_tests/runner.cljc index af6d785561..51cee127c0 100644 --- a/common/test/common_tests/runner.cljc +++ b/common/test/common_tests/runner.cljc @@ -101,6 +101,7 @@ [common-tests.types.container-test] [common-tests.types.fill-test] [common-tests.types.font-test] + [common-tests.types.job-test] [common-tests.types.modifiers-test] [common-tests.types.objects-map-test] [common-tests.types.organization-test] @@ -211,6 +212,7 @@ 'common-tests.types.container-test 'common-tests.types.fill-test 'common-tests.types.font-test + 'common-tests.types.job-test 'common-tests.types.modifiers-test 'common-tests.types.objects-map-test 'common-tests.types.organization-test diff --git a/common/test/common_tests/types/job_test.cljc b/common/test/common_tests/types/job_test.cljc new file mode 100644 index 0000000000..d7cfdc352a --- /dev/null +++ b/common/test/common_tests/types/job_test.cljc @@ -0,0 +1,33 @@ +;; This Source Code Form is subject to the terms of the Mozilla Public +;; License, v. 2.0. If a copy of the MPL was not distributed with this +;; file, You can obtain one at http://mozilla.org/MPL/2.0/. +;; +;; Copyright (c) KALEIDOS SUBSIDIARY SL + +(ns common-tests.types.job-test + "The queue payload both processes share: a two-element array of the + job id and the instant it was pushed, and nothing else." + (:require + [app.common.types.job :as ctj] + [clojure.test :as t])) + +(t/deftest a-payload-is-two-strings + (t/testing "the dispatcher shape validates" + (t/is (ctj/valid-queue-payload? + ["03b7300f-11e6-8131-8008-e4b09b780a0e" "2026-10-07T09:00:00Z"]))) + (t/testing "the decoder keeps it a vector of two strings" + (t/is (= ["03b7300f-11e6-8131-8008-e4b09b780a0e" "2026-10-07T09:00:00Z"] + (ctj/decode-queue-payload + ["03b7300f-11e6-8131-8008-e4b09b780a0e" "2026-10-07T09:00:00Z"]))))) + +(t/deftest anything-else-is-not-a-payload + (t/testing "an empty object is not the array the dispatcher pushes" + (t/is (not (ctj/valid-queue-payload? {})))) + (t/testing "nil is not a payload" + (t/is (not (ctj/valid-queue-payload? nil)))) + (t/testing "an empty array has no job id" + (t/is (not (ctj/valid-queue-payload? [])))) + (t/testing "two items of another type are not a payload" + (t/is (not (ctj/valid-queue-payload? [1 2])))) + (t/testing "a third item is not part of the contract" + (t/is (not (ctj/valid-queue-payload? ["a" "b" "c"]))))) diff --git a/exporter/README.md b/exporter/README.md index ed6271b289..f72fc0072f 100644 --- a/exporter/README.md +++ b/exporter/README.md @@ -58,14 +58,14 @@ packs it for the image. | `PENPOT_PUBLIC_URI` | — | The frontend the assets render through | | `PENPOT_INTERNAL_URI` | `public-uri` | Same network's frontend; the management API too | | `PENPOT_REDIS_URI` | `redis://redis/0` | The queue and the rest of redis | -| `PENPOT_TENANT` | `default` | Queue prefix (workspace name in devenv) | +| `PENPOT_TENANT` | `default` | Queue prefix (workspace name in devenv); lowercased on read | | `PENPOT_TEMPDIR` | `/tmp/penpot` | Temp area owned by the tmpdir service | | `PENPOT_SECRET_KEY` | — | Required; the management key derives from it when no shared key is set | | `PENPOT_EXPORTER_SHARED_KEY` | — | The shared key; derived from the secret otherwise | | `PENPOT_EXPORTER_JOB_TTL` | `3600` | Lifetime of the temp files a job owns, in seconds | | `PENPOT_EXPORTER_WORKER_CONCURRENCY` | `2` | `K`: pollers, running exports, pool sizes | | `PENPOT_WASM_WORKER_POOL_MIN` | `1` | Render workers kept warm; clamped to the max | -| `PENPOT_WASM_WORKER_IDLE_TIMEOUT` | `300` | Silence before a worker is terminated, in seconds | +| `PENPOT_WASM_WORKER_IDLE_TIMEOUT` | `300` | How long an idle pooled worker is kept before eviction, in seconds | | `PENPOT_WASM_WORKER_IMAGE_CACHE_SIZE` | `134217728` | Per-worker image cache budget, in bytes | ## Inspecting the backend diff --git a/exporter/src/exporter/config.cljs b/exporter/src/exporter/config.cljs index c26ebef534..a027add42e 100644 --- a/exporter/src/exporter/config.cljs +++ b/exporter/src/exporter/config.cljs @@ -50,7 +50,7 @@ [:public-uri {:optional true} ::sm/uri] [:internal-uri {:optional true} ::sm/uri] [:exporter-shared-key {:optional true} :string] - [:tenant {:optional true} :string] + [:tenant {:optional true} [:re #"^[a-z0-9-]+$"]] [:flags {:optional true} [::sm/set :keyword]] [:redis-uri {:optional true} :string] [:tempdir {:optional true} :string] @@ -73,6 +73,15 @@ [config] (flags/parse (:flags config))) +(defn- normalize-tenant + "Lowercase the configured tenant before it is validated, so `ACME` and + `acme` name the same tenant: the key derivation, the key id and the + queue key all compare it as an exact string, and the backend + normalizes its side too." + [config] + (cond-> config + (string? (:tenant config)) (update :tenant str/lower))) + (defn read-env "The `PENPOT_`-prefixed variables of `env` as kebab-case keys, the rest ignored. Takes the env object explicitly so tests pass a fake." @@ -95,6 +104,7 @@ environment." [env] (let [data (-> (merge defaults (d/without-nils (read-env env))) + (normalize-tenant) (decode-config))] (when-not (valid-config? data) (throw (ex/error :type :internal diff --git a/exporter/src/exporter/consumer.cljs b/exporter/src/exporter/consumer.cljs index f036112d46..a4f0fb05b8 100644 --- a/exporter/src/exporter/consumer.cljs +++ b/exporter/src/exporter/consumer.cljs @@ -261,18 +261,23 @@ plan/sanitize-file-regex "_")) nil)] - (check-beat (await (beat :rendering (counter counter-kind 0 total) :force? true))) - (await (render {:exports (mapv #(cond-> (assoc % :job-id job-id) - (some? (:skip-children plan)) - (assoc :skip-children (:skip-children plan))) - prepared) - :on-object append - :check-cancelled check-cancelled})) - (when-let [cause @failure] - (throw cause)) - (await (rsc/close-zip zip)) - (check-beat (await (beat :packaging (counter counter-kind total total) :force? true))) - resource)) + (try + (check-beat (await (beat :rendering (counter counter-kind 0 total) :force? true))) + (await (render {:exports (mapv #(cond-> (assoc % :job-id job-id) + (some? (:skip-children plan)) + (assoc :skip-children (:skip-children plan))) + prepared) + :on-object append + :check-cancelled check-cancelled})) + (when-let [cause @failure] + (throw cause)) + (await (rsc/close-zip zip)) + (check-beat (await (beat :packaging (counter counter-kind total total) :force? true))) + resource + (finally + ;; A render that failed or was cancelled never reaches the close + ;; above: destroying the stream is what releases the descriptor. + (rsc/abort-zip zip))))) (defn- ^:async join-pdf "The pages render one file per page; `pdfunite` stitches one pdf out diff --git a/exporter/src/exporter/consumer/worker.cljs b/exporter/src/exporter/consumer/worker.cljs index 8272aa019b..9d8c390eb7 100644 --- a/exporter/src/exporter/consumer/worker.cljs +++ b/exporter/src/exporter/consumer/worker.cljs @@ -19,10 +19,11 @@ `penpot.worker.queue::exporter`. Corrupt payloads are logged and dropped; a Redis hiccup is a lost turn, not a lost job: the poller waits a bit and pops again (what was already delivered is - claimed, and the loop does not stop the process). Stopping quits - every connection and ends every loop: unlike the legacy worker, - whose loops kept failing turns after the stop, no poller survives - its handle. + claimed, and the loop does not stop the process). Stopping falls the + running flag, lets every poller finish the turn it already started, + and only then quits the connections: a payload that arrives in that + window goes back to its queue instead of starting an export the + shutdown is about to cut, and no poller survives its handle. The worker is a system component (`:exporter.consumer/worker`): its config carries the queue knobs (`:concurrency`, `:queue-key`, the @@ -32,6 +33,7 @@ (:require ["ioredis" :as redis] [app.common.logging :as l] + [app.common.types.job :as ctj] [exporter.config :as cf] [exporter.consumer :as consumer] [exporter.consumer.api :as api] @@ -49,12 +51,19 @@ (defn read-payload "The JSON payload of the dispatcher: `job-id` and the `scheduled-at` - of the row when it was pushed. Anything else is a corrupt payload: + of the row when it was pushed, decoded against the shape both sides + share (`app.common.types.job`). Anything else is a corrupt payload: none, and the caller drops it." [payload] (try - (let [[job-id scheduled-at] (js->clj (js/JSON.parse payload))] - {:job-id job-id :scheduled-at scheduled-at}) + (let [decoded (js->clj (js/JSON.parse payload))] + ;; `sm/decoder` hands an invalid value back untouched, so the shape + ;; is checked first: a payload that is not the pair is dropped, not + ;; turned into a claim with nil ids. + (if (ctj/valid-queue-payload? decoded) + (let [[job-id scheduled-at] (ctj/decode-queue-payload decoded)] + {:job-id job-id :scheduled-at scheduled-at}) + (throw (ex-info "unexpected queue payload" {:payload decoded})))) (catch :default cause (l/warn :hint "corrupt queue payload dropped" :payload (str payload) :cause cause) @@ -81,14 +90,21 @@ (defn ^:async poll-once "One turn: pop the queue, claim what came, and wait for the settle. An empty pop and a dropped payload both resolve; a claimed job is - awaited the whole run." - [conn key on-payload] + awaited the whole run. A payload popped after the stop flag fell goes + back to the queue: the shutdown must not start an export it is about + to cut, and the next worker (or the next start) claims it." + [handle conn key on-payload] (let [answer (await (.blpop ^js conn key poll-timeout-s)) - payload (second answer) - claim (when (some? payload) (read-payload payload))] - (when (some? claim) - (await (on-payload claim)) - nil))) + payload (second answer)] + (if (and (some? payload) (not @(:running handle))) + (do + (await (.lpush ^js conn key payload)) + (l/info :hint "payload handed back to the queue" :key key) + nil) + (let [claim (when (some? payload) (read-payload payload))] + (when (some? claim) + (await (on-payload claim)) + nil))))) (defn- ^:async poll-loop "The wait of one poller, turn after turn, until the handle stops. A @@ -101,7 +117,7 @@ (loop [] (when @(:running handle) (try - (await (poll-once conn key on-payload)) + (await (poll-once handle conn key on-payload)) (catch :default cause (l/error :hint "poller turn failed" :key key :cause cause) (await (wait reconnect-delay-ms)))) @@ -123,7 +139,8 @@ [{:keys [concurrency queue-key connect] :as cfg}] (let [running (atom true) inflight (atom 0) - handle {:running running :conns (atom []) :inflight inflight} + handle {:running running :conns (atom []) :loops (atom []) + :inflight inflight} on-payload (^:async fn [claim] (swap! inflight inc) (try @@ -136,7 +153,7 @@ (doseq [_ (range k)] (let [conn (open)] (swap! (:conns handle) conj conn) - (poll-loop handle conn queue-key on-payload))) + (swap! (:loops handle) conj (poll-loop handle conn queue-key on-payload)))) handle)) (defn- ^:async drain-inflight @@ -148,14 +165,25 @@ (defn ^:async stop "Ends the pollers, closing their connections and their loops: the - running flag falls first, so a turn that ends after the quit does - not pop again, then the in-flight runs drain within the grace + running flag falls first, so a turn that ends after the stop hands + its payload back instead of starting an export, then the pollers + finish the turn they already began (bounded by the pop timeout, so a + running export never holds the shutdown here) and only after that + the connections quit. The in-flight runs drain within the grace period before the pools halt behind them. Idempotent: a second stop quits nothing." ([handle] (stop handle nil)) - ([handle {:keys [grace-ms] :or {grace-ms default-shutdown-grace-ms}}] + ([handle {:keys [grace-ms turn-ms] + :or {grace-ms default-shutdown-grace-ms + turn-ms (* 1000 (inc poll-timeout-s))}}] (when (and (some? handle) (compare-and-set! (:running handle) true false)) + ;; The blocking pop cannot be cancelled, so the wait is bounded by + ;; its own timeout: after it, every poller is either idle or + ;; running an export, which the drain below handles. + (await (js/Promise.race + #js [(js/Promise.allSettled @(or (:loops handle) (atom []))) + (wait turn-ms)])) (await (js/Promise.allSettled (mapv (fn [conn] (.quit ^js conn)) @(:conns handle)))) (let [inflight (or (:inflight handle) (atom 0))] diff --git a/exporter/src/exporter/resources.cljs b/exporter/src/exporter/resources.cljs index 4b26d503b6..9c5a82c8a5 100644 --- a/exporter/src/exporter/resources.cljs +++ b/exporter/src/exporter/resources.cljs @@ -18,6 +18,7 @@ ["archiver" :as arc] ["node:fs" :as fs] ["node:path" :as path] + [app.common.exceptions :as ex] [app.common.uuid :as uuid] [cljs.core :as c] [cuerdas.core :as str] @@ -43,6 +44,8 @@ (let [^js zip (new arc/ZipArchive) ^js out (fs/createWriteStream (:path resource)) on-complete (or on-complete (constantly nil)) + on-progress (or on-progress (constantly nil)) + on-error (or on-error (constantly nil)) progress (atom 0)] (.on zip "error" on-error) (.on zip "end" on-complete) @@ -51,6 +54,9 @@ num (swap! progress inc)] (on-progress {:done num :filename name})))) (.pipe zip out) + ;; The stream belongs to the archiver: `abort-zip` needs it to close + ;; the descriptor of a run that never reaches the happy path. + (unchecked-set zip "__out" out) zip)) (defn add-to-zip @@ -73,4 +79,19 @@ (reject cause))] (.on ^js zip "close" (fn [] (done))) (.on ^js zip "error" (fn [cause] (fail cause))) - (.finalize ^js zip))))))) + (.finalize ^js zip))))) + (unchecked-set zip "__closed" true) + nil)) + +(defn abort-zip + "Abort `zip` and destroy the write stream behind it, so a render that + fails or is cancelled does not leave the descriptor open: releasing + the temp path removes the name, not the handle. Idempotent, and safe + after `close-zip` (the archiver is then already settled)." + [zip] + (when (and (some? zip) (not (unchecked-get zip "__closed"))) + (ex/ignoring (.abort ^js zip)) + (when-let [out (unchecked-get zip "__out")] + (ex/ignoring (.destroy ^js out))) + (unchecked-set zip "__closed" true)) + nil) diff --git a/exporter/src/exporter/wasm/pool.cljs b/exporter/src/exporter/wasm/pool.cljs index 71319da673..58ec2bfad5 100644 --- a/exporter/src/exporter/wasm/pool.cljs +++ b/exporter/src/exporter/wasm/pool.cljs @@ -12,9 +12,10 @@ behave the same way for both render backends. A worker is expensive to build (it boots its own render-wasm module), hence the pooling. - Acquisition is not capped: the admission scheduler is the - backpressure, and the idle watchdog guarantees a wedged worker gives - its slot back. + Acquisition is not capped beyond the pool maximum: the K pollers of + the worker are the real concurrency, and a worker that stops answering + is cut by the silence watchdog of `exporter.wasm.scope`, which + terminates it and frees the slot. Workers run the same bundle as the main thread. There is always at least one worker, since a headless render has nowhere else to go. diff --git a/exporter/src/exporter/wasm/scope.cljs b/exporter/src/exporter/wasm/scope.cljs index 235a67584c..52f192382f 100644 --- a/exporter/src/exporter/wasm/scope.cljs +++ b/exporter/src/exporter/wasm/scope.cljs @@ -38,7 +38,12 @@ [^js worker params cancel-buffer on-object timeout-ms] (js/Promise. (fn [resolve reject] - (let [timer (volatile! nil)] + (let [timer (volatile! nil) + ;; The worker posts every object and `done` back to back, so + ;; the message handler must wait for `on-object` before it + ;; settles the render: the callback may still be appending to + ;; the artifact. A rejected callback rejects the render. + chain (serializer)] (letfn [(disarm [] (when-let [t @timer] (js/clearTimeout t) @@ -82,12 +87,20 @@ ;; A failure while the main thread handles the object ;; (moving the file, appending to the zip) has to end ;; the render too, or nothing ever settles this promise. - "object" (try - (on-object (t/decode-str (unchecked-get data "payload"))) - (catch :default cause - (cleanup) - (reject cause))) - "done" (do (cleanup) (resolve nil)) + "object" (-> (chain (fn [] + (on-object (t/decode-str (unchecked-get data "payload"))))) + (.catch (fn [cause] + (cleanup) + (reject cause)))) + ;; `done` queues behind every object of this render, so + ;; the settle never overtakes a pending append. + "done" (-> (chain (fn [] nil)) + (.then (fn [_] + (cleanup) + (resolve nil))) + (.catch (fn [cause] + (cleanup) + (reject cause)))) "error" (do (cleanup) (reject (ex/error :type :internal :code (or (some-> (unchecked-get data "code") keyword) diff --git a/exporter/test/exporter_tests/consumer_test.cljs b/exporter/test/exporter_tests/consumer_test.cljs index 330c84cf0c..3489f8e25a 100644 --- a/exporter/test/exporter_tests/consumer_test.cljs +++ b/exporter/test/exporter_tests/consumer_test.cljs @@ -23,6 +23,7 @@ [exporter.consumer.api :as api] [exporter.jobs :as jobs] [exporter.renderer :as renderer] + [exporter.resources :as rsc] [exporter.shell :as shell])) ;; ---- THE MANAGEMENT FAKE @@ -361,6 +362,34 @@ (finally (restore-fetch))))) +(t/deftest ^:async a-failed-multi-run-aborts-the-zip + ;; A render that fails after an object landed must destroy the write + ;; stream: removing the temp path does not close the handle. + (let [restore-fetch (fake-fetch (answer-with (uuid/next) (volatile! []))) + original rsc/abort-zip + aborted (atom 0) + render (fn [{:keys [on-object]}] + (let [path (shell/tempfile (os/tmpdir) :prefix "penpot.abort." + :suffix ".png")] + (-> (.then (fsp/writeFile path "rendered!") + (fn [_] (on-object {:path path :filename "one.png"}))) + (.then (fn [_] (js/Promise.reject (ex-info "render boom" {}))))))) + job-id (uuid/next)] + (set! rsc/abort-zip (fn [zip] (swap! aborted inc) (original zip))) + (try + (await (consumer/run-export render (os/tmpdir) {:job-id job-id} + {:exports [(one-shape "one") (one-shape "two")] + :name "the export"})) + (t/testing "the run settled as failed" + (t/is (= "fail-job" (last (steps-of))))) + (t/testing "the zip was aborted, not left open" + (t/is (= 1 @aborted))) + (catch :default cause + (t/is false (str "unexpected failure: " (ex-message cause)))) + (finally + (set! rsc/abort-zip original) + (restore-fetch))))) + (t/deftest ^:async cancel-before-the-render-settles-cancelled (let [restore-fetch (fake-fetch (fn [method _body] diff --git a/exporter/test/exporter_tests/consumer_worker_test.cljs b/exporter/test/exporter_tests/consumer_worker_test.cljs index 831f58ebbd..86d7e1c46a 100644 --- a/exporter/test/exporter_tests/consumer_worker_test.cljs +++ b/exporter/test/exporter_tests/consumer_worker_test.cljs @@ -79,14 +79,19 @@ (worker/read-payload (encoded-claim job-id "2026-10-07T09:00:00Z")))))) (t/testing "a corrupt payload is dropped, not thrown at the loop" - (t/is (nil? (worker/read-payload "{not a payload"))))) + (t/is (nil? (worker/read-payload "{not a payload")))) + (t/testing "valid JSON of another shape is corrupt too" + (doseq [payload ["{}" "null" "[]" "\"foo\"" "[1,2]" "[\"a\",\"b\",\"c\"]"]] + (t/is (nil? (worker/read-payload payload)) payload)))) (t/deftest ^:async poll-once-awaits-what-it-claims (try (let [queue (atom [(encoded-claim (uuid/next) "2026-10-07T09:00:00Z")]) conn (fake-conn queue) + handle {:running (atom true)} handled (atom [])] - (await (worker/poll-once conn + (await (worker/poll-once handle + conn "the.queue.key" (fn [claim] (swap! handled conj claim) @@ -95,15 +100,42 @@ (catch :default cause (t/is false (str "unexpected failure: " (ex-message cause)))))) +(t/deftest ^:async poll-once-hands-back-a-payload-popped-after-the-stop + ;; The pop cannot be cancelled, so the payload it returns during the + ;; shutdown goes back to the queue: the export is not started by a + ;; process that is about to halt. + (try + (let [payload (encoded-claim (uuid/next) "2026-10-07T09:00:00Z") + queue (atom [payload]) + conn (fake-conn queue) + pushed (atom []) + handle {:running (atom false)}] + (set! (.-lpush conn) + (fn [key value] + (swap! pushed conj [key value]) + (js/Promise.resolve 1))) + (await (worker/poll-once handle conn "the.queue.key" + (fn [_claim] + (t/is false "a stopping poller must not claim")))) + (t/testing "the payload went back to the same queue, untouched" + (t/is (= [["the.queue.key" payload]] @pushed))) + (t/testing "and nothing was handled" + (t/is (empty? @queue)))) + (catch :default cause + (t/is false (str "unexpected failure: " (ex-message cause)))))) + (t/deftest ^:async poll-once-drops-empty-pops-and-corrupt-payloads (try - (let [queue (atom ["{corrupt"]) - conn (fake-conn queue)] - (await (worker/poll-once conn + (let [queue (atom ["{corrupt"]) + conn (fake-conn queue) + handle {:running (atom true)}] + (await (worker/poll-once handle + conn "the.queue.key" (fn [_claim] (t/is false "corrupt must not be handled")))) - (await (worker/poll-once conn + (await (worker/poll-once handle + conn "the.queue.key" (fn [_claim] (t/is false "a corrupt pop has nothing to handle")))) @@ -188,10 +220,10 @@ c)) handle (await (worker/start (test-cfg {:concurrency 2 :connect conn})))] (t/is (= 2 (count @opened))) - (await (worker/stop handle)) + (await (worker/stop handle {:turn-ms 20})) (t/is (= 2 (count @quit))) (t/testing "a second stop is a no-op" - (await (worker/stop handle)) + (await (worker/stop handle {:turn-ms 20})) (t/is (= 2 (count @quit))))) (catch :default cause (t/is false (str "unexpected failure: " (ex-message cause)))))) diff --git a/exporter/test/exporter_tests/exporter_config_test.cljs b/exporter/test/exporter_tests/exporter_config_test.cljs index 0d1181b38b..3d60868aeb 100644 --- a/exporter/test/exporter_tests/exporter_config_test.cljs +++ b/exporter/test/exporter_tests/exporter_config_test.cljs @@ -10,7 +10,8 @@ schema, with the management key derived the way the backend expects." (:require [cljs.test :as t :include-macros true] - [exporter.config :as cf])) + [exporter.config :as cf] + [exporter.consumer.config :as ccfg])) (t/deftest read-env-keeps-only-the-prefixed-keys (let [env (cf/read-env #js {"PENPOT_SECRET_KEY" "s" @@ -41,6 +42,29 @@ (t/is (some? cause)) (t/is (= :invalid-config (:code (ex-data cause)))))) +(t/deftest prepare-config-normalizes-the-tenant-to-lowercase + ;; The backend lowercases the tenant it reads; a worker configured with + ;; another spelling would derive another key and pop another queue. + (let [config (cf/prepare-config #js {"PENPOT_SECRET_KEY" "s" + "PENPOT_TENANT" "ACME"})] + (t/is (= "acme" (:tenant config))) + (t/testing "the key derived from the normalized value is the same" + (t/is (= (cf/derive-management-key "s" "acme") + (cf/derive-management-key "s" (:tenant config))))) + (t/testing "and the queue key lands where the dispatcher pushes" + (t/is (= "penpot.worker.queue:acme:exporter" + (ccfg/queue-key (:tenant config))))))) + +(t/deftest prepare-config-refuses-a-tenant-outside-the-label-form + (let [cause (try + (cf/prepare-config #js {"PENPOT_SECRET_KEY" "s" + "PENPOT_TENANT" "acme!"}) + nil + (catch :default cause + cause))] + (t/is (some? cause)) + (t/is (= :invalid-config (:code (ex-data cause)))))) + (t/deftest derive-management-key-matches-the-tenant-formula ;; the backend derives the same key from the same secret and tenant ;; (HKDF blake2b512 over "exporter:", 32 bytes, base64url): diff --git a/exporter/test/exporter_tests/exporter_resources_test.cljs b/exporter/test/exporter_tests/exporter_resources_test.cljs index 92ebb0dad7..e0644db8c6 100644 --- a/exporter/test/exporter_tests/exporter_resources_test.cljs +++ b/exporter/test/exporter_tests/exporter_resources_test.cljs @@ -54,6 +54,40 @@ (finally (await (fsp/rm dir #js {:recursive true :force true})))))) +(t/deftest ^:async abort-zip-destroys-the-write-stream + (let [dir (test-area) + _ (await (fsp/mkdir dir #js {:recursive true})) + resource (rsc/create dir :zip "the export") + zip (rsc/create-zip :resource resource) + ^js out (unchecked-get zip "__out")] + (try + (t/testing "the stream is open before the abort" + (t/is (false? (.-destroyed out)))) + (rsc/abort-zip zip) + (t/testing "the abort releases the descriptor" + (t/is (true? (.-destroyed out)))) + (t/testing "a second abort is harmless" + (t/is (nil? (rsc/abort-zip zip))) + (t/is (true? (.-destroyed out)))) + (finally + (await (fsp/rm dir #js {:recursive true :force true})))))) + +(t/deftest ^:async abort-zip-after-close-is-a-no-op + (let [dir (test-area) + _ (await (fsp/mkdir dir #js {:recursive true})) + resource (rsc/create dir :zip "the export") + member (path/join dir "member.txt") + _ (await (fsp/writeFile member "member!")) + zip (rsc/create-zip :resource resource)] + (try + (rsc/add-to-zip zip member "member.txt") + (await (rsc/close-zip zip {:timeout-ms 1000})) + (t/testing "an abort after the close does not disturb the artifact" + (t/is (nil? (rsc/abort-zip zip))) + (t/is (pos? (.-size (await (fsp/stat (:path resource))))))) + (finally + (await (fsp/rm dir #js {:recursive true :force true})))))) + (defn- fake-zip "An event-emitter stand-in for `close-zip`: records its listeners and runs `on-finalize` with them once finalized." diff --git a/exporter/test/exporter_tests/runner.cljs b/exporter/test/exporter_tests/runner.cljs index 03c893648e..50c0dcdfad 100644 --- a/exporter/test/exporter_tests/runner.cljs +++ b/exporter/test/exporter_tests/runner.cljs @@ -39,10 +39,21 @@ ;; The tests that ride the mock http must never leave a rejected promise ;; dangling: a dangling one kills the whole run with ERR_UNHANDLED_REJECTION -;; and without any way to point at the chain that broke. Log and keep going. +;; and without any way to point at the chain that broke. The run keeps +;; going so it can report every test, but the summary counts each one as +;; a failure: a broken chain is a regression, not a log line. +(def ^:private dangling-rejections (atom [])) + (js/process.on "unhandledRejection" (fn [reason _promise] - (println (str "UNHANDLED-REJECTION: " reason)))) + (swap! dangling-rejections conj reason))) + +(defn- rejection-text + [reason] + (let [stack (unchecked-get reason "stack")] + (if (string? stack) + (str reason "\n" stack) + (str reason)))) (def test-namespaces ['exporter-tests.browser-scope-test @@ -194,6 +205,11 @@ (partial merge-with +) (:report-counters (t/get-current-env)))) (fn [] + (let [dangling @dangling-rejections] + (when (seq dangling) + (doseq [reason dangling] + (println (str "UNHANDLED-REJECTION: " (rejection-text reason)))) + (vswap! summary update :fail + (count dangling)))) (t/report @summary) (t/report (assoc @summary :type :end-run-tests)))])))) diff --git a/exporter/test/exporter_tests/wasm_scope_test.cljs b/exporter/test/exporter_tests/wasm_scope_test.cljs index b8e56ffc23..285513251c 100644 --- a/exporter/test/exporter_tests/wasm_scope_test.cljs +++ b/exporter/test/exporter_tests/wasm_scope_test.cljs @@ -74,6 +74,18 @@ [calls] (some #(and (vector? %) (= :post (first %))) calls)) +(defn- multi-behavior + "Posts every object back to back and then the settle, the way the real + worker does: nothing waits for the main thread between messages." + [payloads] + (fn [^js emitter _msg] + (js/setTimeout + (fn [] + (doseq [payload payloads] + (.emit emitter "message" #js {:type "object" :payload (transit/encode-str payload)})) + (.emit emitter "message" #js {:type "done"})) + 0))) + (t/deftest ^:async run-shares-one-worker-across-renders (try (let [calls (atom []) @@ -155,6 +167,47 @@ (catch :default cause (t/is false (str "unexpected failure: " (ex-message cause)))))) +(t/deftest ^:async a-slow-on-object-settles-before-the-render + ;; The worker posts every object and `done` in one burst; the settle + ;; must wait for the callbacks, or a multi-object export can close its + ;; artifact with entries still pending. + (try + (let [calls (atom []) + worker (stub-worker calls (multi-behavior [{:id "o1"} {:id "o2"}])) + seen (atom []) + cfg (test-cfg calls worker + (fn [object] + ;; resolves on a later tick, like a beat does + (js/Promise. + (fn [resolve] + (js/setTimeout + (fn [] + (swap! seen conj (:id object)) + (resolve nil)) + 5)))) + (atom false))] + (await (scope/run cfg (fn [render-fn] (render-fn (test-params))))) + (t/testing "every object landed before the render resolved" + (t/is (= ["o1" "o2"] @seen)))) + (catch :default cause + (t/is false (str "unexpected failure: " (ex-message cause)))))) + +(t/deftest ^:async a-rejected-on-object-rejects-the-render + (try + (let [calls (atom []) + worker (stub-worker calls (multi-behavior [{:id "o1"}])) + cfg (test-cfg calls worker + (fn [_object] + (js/Promise.reject (ex-info "append broke" {}))) + (atom false))] + (try + (await (scope/run cfg (fn [render-fn] (render-fn (test-params))))) + (t/is false "a rejected on-object must reject the render") + (catch :default cause + (t/is (= "append broke" (ex-message cause)))))) + (catch :default cause + (t/is false (str "unexpected failure: " (ex-message cause)))))) + (t/deftest ^:async render-on-without-lease-raises (try (try diff --git a/frontend/src/app/main/data/exports/assets.cljs b/frontend/src/app/main/data/exports/assets.cljs index b10fb215d6..1b1cc30179 100644 --- a/frontend/src/app/main/data/exports/assets.cljs +++ b/frontend/src/app/main/data/exports/assets.cljs @@ -94,10 +94,15 @@ :hint (tr "errors.export-failed"))))) (defn normalize-export - [{:keys [object-id name] :as export}] - (assoc export :name (if (str/blank? name) - (str object-id) - name))) + "The export item of the shared contract: the name filled from the + object id when blank, and the suffix defaulted to empty for items + created before the contract required it." + [{:keys [object-id name suffix] :as export}] + (assoc export + :name (if (str/blank? name) + (str object-id) + name) + :suffix (or suffix ""))) (defn- normalize-exports [exports] @@ -534,11 +539,13 @@ (let [ws-conn (:ws-conn state) file-ids (export-file-ids [export])] (swap! st/ongoing-tasks conj :export) - (rx/concat - (dwp/force-persist-files-and-wait file-ids save-wait-timeout-ms) - (rx/of (warn-unsaved-changes file-ids)) - (export-stream! ws-conn [export] :export-shapes - {:is-wasm (wasm-export-enabled? state)})))))))) + (->> (rx/concat + (dwp/force-persist-files-and-wait file-ids save-wait-timeout-ms) + (rx/of (warn-unsaved-changes file-ids)) + (export-stream! ws-conn [export] :export-shapes + {:is-wasm (wasm-export-enabled? state)})) + (rx/finalize (fn [] + (swap! st/ongoing-tasks disj :export)))))))))) (defn- export-file-page-ids [state exports] @@ -575,7 +582,7 @@ :type type :scale (or scale 1) :name name - :suffix suffix}) + :suffix (or suffix "")}) exports)) (defn- request-local-export-wasm @@ -676,14 +683,16 @@ ;; export is a shapes one kind (if (= :export-frames cmd) :frames :shapes)] (swap! st/ongoing-tasks conj :export) - (rx/concat - (dwp/force-persist-files-and-wait file-ids save-wait-timeout-ms) - (rx/of (warn-unsaved-changes file-ids)) - (export-stream! ws-conn exports cmd - {:force-multiple true - :name name - :kind kind - :is-wasm (wasm-export-enabled? state)})))))) + (->> (rx/concat + (dwp/force-persist-files-and-wait file-ids save-wait-timeout-ms) + (rx/of (warn-unsaved-changes file-ids)) + (export-stream! ws-conn exports cmd + {:force-multiple true + :name name + :kind kind + :is-wasm (wasm-export-enabled? state)})) + (rx/finalize (fn [] + (swap! st/ongoing-tasks disj :export)))))))) (defn request-multiple-export [{:keys [exports cmd] diff --git a/frontend/test/frontend_tests/data/exports_assets_test.cljs b/frontend/test/frontend_tests/data/exports_assets_test.cljs index a30e5a796f..db7c9ac33e 100644 --- a/frontend/test/frontend_tests/data/exports_assets_test.cljs +++ b/frontend/test/frontend_tests/data/exports_assets_test.cljs @@ -427,6 +427,37 @@ ;; Scenario: an export starts while its file has a change not saved yet. ;; Proves: the user is warned that the export may miss it, and the export ;; runs anyway. +(t/deftest ^:async a-failed-persist-releases-the-export-flag + ;; The persist runs before the job exists: if it fails, the flag must + ;; not stay set, or the leave-page warning never turns off. + (let [tasks (atom #{}) + store-errors (atom []) + store (export-store (test-state) store-errors)] + (await + (mock/with-mocks* + {dwp/force-persist-files-and-wait + (mock/stub (fn [_ _] (rx/throw (ex-info "persist failed" {:type :internal})))) + st/ongoing-tasks tasks} + (ptk/emit! store (de/request-simple-export {:export export})) + (await (async/wait-for #(not (contains? @tasks :export)) + "the leave-page warning is released")) + (t/is (empty? @tasks)))))) + +(t/deftest ^:async a-failed-persist-releases-the-export-flag-of-a-multiple-run + (let [tasks (atom #{}) + store-errors (atom []) + store (export-store (test-state) store-errors)] + (await + (mock/with-mocks* + {dwp/force-persist-files-and-wait + (mock/stub (fn [_ _] (rx/throw (ex-info "persist failed" {:type :internal})))) + st/ongoing-tasks tasks} + (ptk/emit! store (de/request-multiple-export + {:exports [(assoc export :enabled true)]})) + (await (async/wait-for #(not (contains? @tasks :export)) + "the leave-page warning is released")) + (t/is (empty? @tasks)))))) + (t/deftest ^:async export-warns-when-the-file-has-unsaved-changes (let [file-id (uuid/next) downloaded (atom false)