github/penpot
1
0
Fork 0
mirror of https://github.com/penpot/penpot.git synced 2026-08-05 12:29:00 +00:00
Code Issues Packages Projects Releases Wiki Activity

✨ Add media-processor service for image and font processing (#10767)

Browse Source 
* ✨ Add media-processor service for image and font processing

Externalizes ImageMagick and FontForge subprocess invocations into a
separate Node.js HTTP service (media-processor/). Backend dispatches
via feature flag :use-remote-media-processing.

Key changes:
- media-processor module (TypeScript, Express 5, Sharp, FontForge/woff)
  - POST /api/image/info, /api/image/thumbnail, /api/font/generate
  - Resource limits: 128MP rejection, prlimit (512MB + 30s CPU)
  - Streaming multipart via SequenceInputStream
- app.media split into validation (leaf), local (shell impls), remote (HTTP)
- Schema enforcement: :upload and :input schemas in validation namespace
- Configurable timeout (PENPOT_MEDIA_PROCESSING_SERVICE_TIMEOUT)
- 78 tests across 4 files (image, font, middleware, config)
- FontForge path escaping for command injection prevention
- Parallel font variant conversions with Promise.all

AI-assisted-by: mimo-v2.5-pro

* 🐳 Revert docker-compose changes from media-processor commit

Remove docker-compose.yaml modifications that were part of the media-processor
service commit. The media-processor service definition, flags, and environment
variables are reverted to their previous state.

AI-assisted-by: qwen3.7-plus

* ⬆️ Update dependencies

* 🐛 Fix PR review issues in media-processor

- Font path bug: sfntToWoff and woff2ToSfnt now copy input to temp dir
  when input is a file path, ensuring output lands in expected location
- Error preservation: execCommand preserves killed/signal/code properties
  from child process errors for OOM detection
- Content-Length: service-multipart-request calculates and includes
  Content-Length header for streaming multipart requests

AI-assisted-by: qwen3.7-plus

* 🐛 Fix code review issues in media-processor

- Rename PENPOT_MEDIA_PROCESSOR_SECRET_KEY to PENPOT_MEDIA_PROCESSOR_SHARED_KEY
  in devenv to match backend config key
- Fix timeout middleware to destroy request AFTER response finishes,
  preventing truncated 504 responses
- Fix quality=0 parsing to preserve explicit zero (was silently overridden to 85)
- Replace require('fs') with proper ES module import in upload-storage.ts
- Refactor font conversion temp-dir boilerplate into withTempInput helper
- Document FontForge escaping limitations (single quotes only)
- Fix misleading comment in image.ts about sharp metadata decoding

AI-assisted-by: qwen3.7-plus

* 🐛 Fix code review issues in media-processor (round 2)

- Fix queue middleware to skip next() when response already ended,
  preventing orphaned work after timeout
- Fix hybrid storage to use disk when Content-Length is absent (chunked
  transfer), preventing unbounded memory allocation
- Add source image format validation in generateThumbnail to reject
  unsupported formats (TIFF, BMP, etc.) with 400 instead of 500
- Remove dead code in convertFont for unreachable woff→woff path
- Remove unused isEnabled() method from LokiLogTransport
- Fix sfntToWoff to use correct extension (.ttf/.otf) based on source type
- Extract queue middleware to separate file for testability
- Add comprehensive tests for queue middleware and upload storage

AI-assisted-by: qwen3.7-plus

* 🐛 Fix code review issues in media-processor (round 3)

- Fix disk-backed upload cleanup after successful requests by adding
  cleanup middleware that removes temp files on response finish/close
- Wrap sharp metadata/decoding errors as 400 validation errors instead
  of 500 internal errors
- Only apply flatten() for JPEG output to preserve alpha channel in
  PNG and WebP outputs

AI-assisted-by: qwen3.7-plus

* ✨ Add comprehensive tests for media-processor

Phase 1 - Cleanup verification:
- Add cleanup middleware unit tests (6 tests)
- Add HTTP upload cleanup integration tests (5 tests)

Phase 2 - Error handling & alpha preservation:
- Add sharp error wrapping tests (4 tests)
- Add HTTP malformed image tests (2 tests)
- Add alpha preservation tests (3 tests)

Phase 3 - Edge cases:
- Add upload storage edge case tests (3 tests)
- Add queue middleware edge case tests (4 tests)

Phase 4 - Backend mock verification:
- Fix backend mocks to include :mtype field in image info responses
- Verify all error codes match actual service behavior

Total: 27 new tests added (160 tests passing)

AI-assisted-by: qwen3.7-plus

* 🐛 Fix code review issues in media-processor (round 4)

- Add Zod validation constraints for config values (int, positive, min)
- Fix auth middleware to compare Buffer byte lengths instead of string lengths
- Validate requested output dimensions in generateThumbnail (crop mode)
- Change queue middleware to release slot via callback in finally block
- Add comprehensive tests for all fixes

AI-assisted-by: qwen3.7-plus

* 🐛 Close HTTP response streams in backend media remote

- Wrap stream consumption in try/finally with .close() calls
- Add tests to verify stream closure for info, font-convert, and thumbnail

AI-assisted-by: qwen3.7-plus

* 🐛 Fix queue slot leak on upload failures

Make releaseQueue idempotent and attach fallback listener to release
slot when response finishes. This covers Multer errors that bypass
the route handler's finally block, preventing permanent queue stall.

AI-assisted-by: qwen3.7-plus

* 🐛 Cancel processing on timeout

Create AbortController in timeout middleware and abort signal when
timeout fires. Pass signal to Sharp and FontForge to cancel ongoing
processing and release resources when request is cancelled.

AI-assisted-by: qwen3.7-plus

* 🐛 Fix code review issues in media-processor (round 6)

- Error handler: check headersSent before writing response to prevent
  ERR_HTTP_HEADERS_SENT when timeout already sent 504
- Timeout config: increase default requestTimeout from 60s to 180s to
  match font processing timeout (120s) and backend request timeout
- Image processing: check abort signal before starting Sharp operations
  to cancel processing when timeout fires
- Queue lifecycle: remove res.on('close', release) fallback to hold
  queue slot until processing completes, preventing concurrency limit
  violation when client disconnects

AI-assisted-by: qwen3.7-plus

* 🐛 Close HTTP response stream in download-image

Wrap response body in with-open to ensure stream is closed after
writing to temp file, preventing HTTP connection leaks on repeated
URL imports.

AI-assisted-by: qwen3.7-plus

* 🐛 Close HTTP response stream on validation errors in download-image

Move with-open to wrap the entire validation and processing block,
ensuring the response body stream is closed even when validation fails
(non-2xx status, missing size, invalid media type). This prevents
HTTP connection leaks on repeated failed downloads.

Add test to verify stream closure on validation errors.

AI-assisted-by: qwen3.7-plus

* 🐛 Pass abort signal to Sharp toBuffer for timeout cancellation

Wrap Sharp's toBuffer() with Promise.race to check abort signal during
processing. This ensures large thumbnails stop processing when the
request times out, preventing wasted CPU/memory and queue capacity.

Add test to verify abort during toBuffer operation.

AI-assisted-by: qwen3.7-plus

* 🐛 Hold queue slot until Sharp completes and handle client disconnect

- Remove Promise.race from generateThumbnail — Sharp processing now
  completes fully before queue slot is released, preventing concurrency
  limit violations under timeout conditions
- Remove res.on("finish", release) fallback from queue middleware —
  error handler now explicitly calls releaseQueue in all error paths
- Add res.on("close") handler in timeout middleware to abort signal
  when client disconnects, ensuring processing stops early
- Add tests for client disconnect handling and queue slot lifecycle

AI-assisted-by: qwen3.7-plus

* 🐛 Address round 9 review findings

- Document Sharp 0.35.3 cancellation limitation in image.ts
- Add integration test for timeout cleanup with large images
- Fix font tools (sfntToWoff, woffToSfnt, woff2ToSfnt) to throw
  ProcessingError on resource limit kills instead of returning null
- Validate font signatures for same-format conversions to prevent
  arbitrary files from being persisted as valid fonts
- Fix concurrent mkdtemp race in upload-storage by using shared
  initialization promise

AI-assisted-by: qwen3.7-plus

* 🐛 Address round 10 review findings

- Add tmpdir assertion in font.ts to prevent path injection
- Preserve original error in queue middleware catch handler
- Change auth middleware response type from "internal" to "authorization"
- Add cleanup flag to prevent double cleanup in cleanup middleware
- Move quality clamping into parseQuality function for consistency
- Add integration tests for quality parameter clamping at route level
- Update existing tests to match new auth response type

AI-assisted-by: qwen3.7-plus

* 🐛 Address round 11 review findings

- Extract releaseSlot helper in error-handler to reduce duplication
- Remove redundant try/catch in font.ts withTempDir cleanup
- Improve font path validation error message for clarity
- Move path validation before try/catch to prevent swallowing
- Add debug logging for cleanup failures in cleanup middleware
- Inline TransportTargetSpec type alias in logger.ts
- Extract logging middleware to separate file for consistency
- Remove duplicate MIME validation in image thumbnail route
- Add test for font path validation (outside tmpdir rejection)
- Add tests for error handler queue release across all branches

AI-assisted-by: qwen3.7-plus

* 🐛 Remove Content-Length header from multipart requests

The JDK's HttpClient rejects Content-Length as a restricted header,
causing IllegalArgumentException when sending multipart requests to the
media-processor. Remove the explicit Content-Length header and let the
JDK use chunked transfer encoding. The media-processor will use disk
storage for all multipart requests (safe default behavior).

Remove unused size computations (file-size, header-bytes, footer-bytes,
total-size) that were only used for Content-Length.

Update test to verify Content-Length is not present in request headers.

AI-assisted-by: qwen3.7-plus

* 🐛 Fix pino ESM bundling for media-processor

Mark pino and its transports (pino-pretty, pino-loki) as external to
avoid bundling issues with worker thread modules that reference
__dirname (not available in ES modules).

AI-assisted-by: qwen3.7-plus
This commit is contained in:
Andrey Antukh 2026-08-05 09:41:48 +02:00 committed by GitHub
parent 3e59754a25
commit aeedb96260
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
65 changed files with 8922 additions and 1103 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored
View File

@ -88,6 +88,7 @@ opencode.json
/blob-report/
/playwright/.cache/
/render-wasm/target/
/media-processor/dist/
/**/node_modules
/**/.yarn/*
/.pnpm-store

1
.serena/memories/critical-info.md
View File

@ -39,6 +39,7 @@ This is a monorepo. Principles that apply to one module do *not* generally apply
- `plugins/`: TypeScript plugin runtime/examples and Plugin API types; core conventions: `mem:plugins/core`.
- `library/`: design library workflows; core conventions: `mem:library/core`.
- `docs/`: documentation site; core workflow and conventions: `mem:docs/core`.
- `media-processor/`: TypeScript/Node.js HTTP service for image (sharp) and font (FontForge) processing; core conventions: `mem:media-processor/core`.
The memory is structured in a way that you can get the critical information about the
module. You can read it from `mem:<MODULE>/core`

100
.serena/memories/media-processor/core.md Normal file
View File

@ -0,0 +1,100 @@
# Media Processor
Stateless HTTP service for Penpot image and font processing. Handles image info extraction, thumbnail generation (sharp), and font conversion (FontForge, woff-tools).
## Tech Stack
- Language: TypeScript
- Runtime: Node.js
- Framework: Express
- Image processing: sharp (libvips)
- Font processing: FontForge (TTF/OTF), sfnt2woff, woff2_decompress
- Upload handling: multer (hybrid storage: memory for small, disk for large)
- Logging: pino (with optional Loki transport)
- Config validation: Zod
- Testing: Vitest
- Package Manager: pnpm
## Project Structure
```
media-processor/
├── src/
│ ├── index.ts # Express app setup, routes, middleware
│ ├── config.ts # Zod-validated env config, HKDF key derivation
│ ├── types.ts # TypeScript type definitions
│ ├── upload.ts # Multer configuration, getFileBuffer helper
│ ├── upload-storage.ts # Hybrid storage engine (memory < threshold, disk >= threshold)
│ ├── logger.ts # Pino logger setup
│ ├── middleware/
│ │ ├── auth.ts # Timing-safe shared key authentication
│ │ ├── error-handler.ts # ProcessingError class, centralized error handling
│ │ └── timeout.ts # Request timeout middleware
│ ├── routes/
│ │ ├── health.ts # GET /api/health
│ │ ├── image.ts # POST /api/image/info, /api/image/thumbnail
│ │ └── font.ts # POST /api/font/convert
│ └── services/
│ ├── image.ts # sharp-based image info/thumbnail generation
│ ├── font.ts # FontForge/woff-tools font conversion
│ └── errors.ts # throwValidation, throwRestriction, throwProcessing
├── test/ # Vitest test files
├── vitest.config.ts # Test configuration
├── tsconfig.json # TypeScript configuration
├── esbuild.config.mjs # Build configuration
└── package.json # Dependencies and scripts
```
## Key Conventions
### Auth
- Requests authenticated via `x-shared-key` header using timing-safe comparison
- When no key configured, all requests rejected with 403
- Key derived from `PENPOT_SECRET_KEY` via HKDF (blake2b512) or set directly via `PENPOT_MEDIA_PROCESSOR_SHARED_KEY`
### Resource Limits
- Image: max pixels, max width/height enforced before processing
- Font: prlimit wraps FontForge processes with memory (AS) and CPU time limits
- Concurrency: p-queue limits concurrent requests (default 10)
- Upload: hybrid storage — memory for files < 10MB, disk for larger; configurable via `PENPOT_MEDIA_PROCESSOR_MEMORY_THRESHOLD`
- Max file size: configurable (default 350MB)
### Error Handling
- `throwValidation(code, hint)` — 400 errors for invalid input
- `throwRestriction(code, hint)` — 413 errors for resource limits exceeded
- `throwProcessing(code, hint)` — 503 errors for processing failures (e.g., resource limit kills)
### Image Processing
- EXIF orientation applied before dimension validation and thumbnail generation
- sharp caching disabled to prevent unbounded memory growth
- `withoutEnlargement: true` prevents upscaling small images
### Font Conversion
- Supported formats: TTF, OTF, WOFF, WOFF2
- SFNT type detected via magic bytes (0x4f54544f = OTF, 0x00010000 = TTF)
- Temp files cleaned up in finally blocks (best-effort)
## Commands
All commands run from `media-processor/` directory:
- `pnpm run test` — Run Vitest test suite
- `pnpm run types:check` — TypeScript type checking (tsc --noEmit)
- `pnpm run fmt` — Format code with Prettier
- `pnpm run fmt:check` — Check formatting without modifying
- `pnpm run build` — Build for production (esbuild)
- `pnpm run start:dev` — Start development server (tsx)
## Docker
- Exposed port: 6065 (configurable via `PENPOT_MEDIA_PROCESSOR_PORT`)
- Must be deployed on internal Docker network only (not public-facing)
- Backend communicates via `PENPOT_MEDIA_PROCESSING_SERVICE_URI`
## Testing Principles
Cross-cutting testing principles and anti-patterns: `mem:testing`.
- Run `pnpm run test` after changes
- Run `pnpm run types:check` after TypeScript changes
- Run `pnpm run fmt:check` before commits

4
backend/scripts/_env
View File

@ -4,6 +4,7 @@ export PENPOT_ADMIN_CONSOLE_SHARED_KEY=super-secret-nitrate-api-key
export PENPOT_EXPORTER_SHARED_KEY=super-secret-exporter-api-key
export PENPOT_NEXUS_SHARED_KEY=super-secret-nexus-api-key
export PENPOT_SECRET_KEY=super-secret-devenv-key
export PENPOT_MEDIA_PROCESSOR_SHARED_KEY=super-secret-media-processor-key
# DEPRECATED: only used for subscriptions
export PENPOT_MANAGEMENT_API_KEY=super-secret-management-api-key
@ -21,6 +22,8 @@ if [[ "${PENPOT_BACKEND_WORKER:-true}" == "true" ]]; then
__worker_flag="enable-backend-worker"
fi
export PENPOT_MEDIA_PROCESSING_SERVICE_URI=http://localhost:6065
export PENPOT_FLAGS="\
$PENPOT_FLAGS \
enable-login-with-password \
@ -36,6 +39,7 @@ export PENPOT_FLAGS="\
enable-feature-fdata-objects-map \
enable-audit-log \
enable-transit-readable-response \
disable-remote-media-processing \
enable-demo-users \
enable-user-feedback \
disable-secure-session-cookies \

4
backend/src/app/config.clj
View File

@ -121,6 +121,7 @@
[:exporter-shared-key {:optional true} :string]
[:admin-console-shared-key {:optional true} :string]
[:nexus-shared-key {:optional true} :string]
[:media-processor-shared-key {:optional true} :string]
[:management-api-key {:optional true} :string]
[:telemetry-uri {:optional true} :string]
@ -147,6 +148,9 @@
[:imagemagick-width-limit {:optional true} :string]
[:imagemagick-height-limit {:optional true} :string]
[:media-processing-service-uri {:optional true} ::sm/uri]
[:media-processing-service-timeout {:optional true} ::sm/int]
[:deletion-delay {:optional true} ::ct/duration]
[:file-clean-delay {:optional true} ::ct/duration]
[:telemetry-enabled {:optional true} ::sm/boolean]

10
backend/src/app/main.clj
View File

@ -335,6 +335,7 @@
::rpc/rlimit (ig/ref ::rpc/rlimit)
::setup/templates (ig/ref ::setup/templates)
::setup/props (ig/ref ::setup/props)
::setup/shared-keys (ig/ref ::setup/shared-keys)
::email/blacklist (ig/ref ::email/blacklist)
::email/whitelist (ig/ref ::email/whitelist)
@ -467,10 +468,11 @@
::migrations (ig/ref :app.migrations/migrations)}
::setup/shared-keys
{::setup/props (ig/ref ::setup/props)
:nexus (cf/get :nexus-shared-key)
:admin-console (cf/get :admin-console-shared-key)
:exporter (cf/get :exporter-shared-key)}
{::setup/props (ig/ref ::setup/props)
:nexus (cf/get :nexus-shared-key)
:admin-console (cf/get :admin-console-shared-key)
:exporter (cf/get :exporter-shared-key)
:media-processor (cf/get :media-processor-shared-key)}
::setup/clock
{}

517
backend/src/app/media.clj
View File

@ -5,316 +5,37 @@
;; Copyright (c) KALEIDOS INC Sucursal en España SL
(ns app.media
"Media & Font postprocessing."
"Media & Font postprocessing.
This namespace is the dispatch layer only. Processing implementations
live in two separate namespaces, each owning their own defmulti:
app.media.local — shell/ImageMagick/FontForge implementations
app.media.remote — HTTP delegation to media-processor service
Validation and schemas live in app.media.validation (leaf namespace,
no circular dep). When adding a new :cmd type, add defmethods in
BOTH local and remote."
(:require
[app.common.data :as d]
[app.common.data.macros :as dm]
[app.common.exceptions :as ex]
[app.common.logging :as l]
[app.common.media :as cm]
[app.common.schema :as sm]
[app.common.schema.openapi :as-alias oapi]
[app.common.time :as ct]
[app.config :as cf]
[app.db :as-alias db]
[app.http.client :as http]
[app.media.local :as media.local]
[app.media.remote :as media.remote]
[app.media.sanitize :as sanitize]
[app.media.validation :as validation]
[app.storage :as-alias sto]
[app.storage.tmp :as tmp]
[app.util.shell :as shell]
[buddy.core.bytes :as bb]
[buddy.core.codecs :as bc]
[clojure.string]
[clojure.xml :as xml]
[cuerdas.core :as str]
[datoteka.fs :as fs]
[datoteka.io :as io])
(:import
clojure.lang.XMLHandler
java.io.InputStream
javax.xml.parsers.SAXParserFactory
javax.xml.XMLConstants
org.apache.commons.io.IOUtils))
(def schema:upload
[:map {:title "Upload"}
[:filename :string]
[:size ::sm/int]
[:path ::fs/path]
[:mtype {:optional true} :string]
[:headers {:optional true}
[:map-of :string :string]]])
(def ^:private schema:input
[:map {:title "Input"}
[:path ::fs/path]
[:mtype {:optional true} ::sm/text]])
(def check-input
(sm/check-fn schema:input))
(defn validate-media-type!
([upload] (validate-media-type! upload cm/image-types))
([upload allowed]
(when-not (contains? allowed (:mtype upload))
(ex/raise :type :validation
:code :media-type-not-allowed
:hint "Seems like you are uploading an invalid media object"))
upload))
(defn validate-media-size!
[upload]
(let [max-size (cf/get :media-max-file-size)]
(when (> (:size upload) max-size)
(ex/raise :type :restriction
:code :media-max-file-size-reached
:hint (str/ffmt "the uploaded file size % is greater than the maximum %"
(:size upload)
max-size)))
upload))
(defn validate-font-size!
"Validates that the font file `upload` does not exceed the configured
`:font-max-file-size` limit. Accepts the same map shape as
`validate-media-size!` — requires a `:size` key in bytes."
[upload]
(let [max-size (cf/get :font-max-file-size)]
(when (> (:size upload) max-size)
(ex/raise :type :restriction
:code :font-max-file-size-reached
:hint (str/ffmt "the uploaded font size % is greater than the maximum %"
(:size upload)
max-size)))
upload))
(defmulti process (fn [_system params] (:cmd params)))
(defmethod process :default
[_system {:keys [cmd] :as params}]
(ex/raise :type :internal
:code :not-implemented
:hint (str/fmt "No impl found for process cmd: %s" cmd)))
[datoteka.io :as io]))
(defn run
[system params]
(process system params))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; SVG PARSING
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defn- secure-parser-factory
[^InputStream input ^XMLHandler handler]
(.. (doto (SAXParserFactory/newInstance)
(.setFeature XMLConstants/FEATURE_SECURE_PROCESSING true)
(.setFeature "http://apache.org/xml/features/disallow-doctype-decl" true))
(newSAXParser)
(parse input handler)))
(defn- strip-doctype
[data]
(cond-> data
(str/includes? data "<!DOCTYPE")
(str/replace #"<\!DOCTYPE[^>]*>" "")))
(defn- parse-svg
[text]
(let [text (strip-doctype text)]
(dm/with-open [istream (IOUtils/toInputStream ^String text "UTF-8")]
(xml/parse istream secure-parser-factory))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; IMAGE THUMBNAILS
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(def ^:private schema:thumbnail-params
[:map {:title "ThumbnailParams"}
[:input schema:input]
[:format [:enum :jpeg :webp :png]]
[:quality [:int {:min 1 :max 100}]]
[:width :int]
[:height :int]])
(def ^:private check-thumbnail-params
(sm/check-fn schema:thumbnail-params))
;; Related info on how thumbnails generation
;; http://www.imagemagick.org/Usage/thumbnails/
(def ^:private imagemagick-default-env
"Default environment variables for ImageMagick resource limits.
These are the soft ceiling — policy.xml is the hard ceiling."
{"MAGICK_THREAD_LIMIT" "2"
"MAGICK_MEMORY_LIMIT" "256MiB"
"MAGICK_MAP_LIMIT" "512MiB"
"MAGICK_AREA_LIMIT" "128MP"
"MAGICK_DISK_LIMIT" "1GiB"
"MAGICK_TIME_LIMIT" "30"})
(defn- get-imagemagick-env
"Returns environment variables for ImageMagick commands.
Reads individual PENPOT_IMAGEMAGICK_* config values, falling back to defaults."
[]
(let [thread (cf/get :imagemagick-thread-limit)
memory (cf/get :imagemagick-memory-limit)
map-l (cf/get :imagemagick-map-limit)
area (cf/get :imagemagick-area-limit)
disk (cf/get :imagemagick-disk-limit)
time (cf/get :imagemagick-time-limit)
width (cf/get :imagemagick-width-limit)
height (cf/get :imagemagick-height-limit)]
(cond-> imagemagick-default-env
thread (assoc "MAGICK_THREAD_LIMIT" thread)
memory (assoc "MAGICK_MEMORY_LIMIT" memory)
map-l (assoc "MAGICK_MAP_LIMIT" map-l)
area (assoc "MAGICK_AREA_LIMIT" area)
disk (assoc "MAGICK_DISK_LIMIT" disk)
time (assoc "MAGICK_TIME_LIMIT" time)
width (assoc "MAGICK_WIDTH_LIMIT" width)
height (assoc "MAGICK_HEIGHT_LIMIT" height))))
(defn- exec-magick!
"Execute an ImageMagick command with resource limits.
`args` is a vector of string arguments to pass to `magick`."
[system args]
(let [cmd (into ["magick"] args)
result (shell/exec! system
:cmd cmd
:env (get-imagemagick-env)
:timeout 60)]
(when (not= 0 (:exit result))
(ex/raise :type :validation
:code :invalid-image
:hint (str "ImageMagick command failed: " (:err result))
:cmd cmd
:exit (:exit result)))
result))
(defn- generic-process
[system {:keys [input format convert-args] :as params}]
(let [{:keys [path mtype]} input
format (or format (cm/mtype->format mtype))
ext (cm/format->extension format)
tmp (tmp/tempfile :prefix "penpot.media." :suffix ext)
args (into [(str path)] (conj (vec convert-args) (str tmp)))]
(exec-magick! system args)
(assoc params
:format format
:mtype (cm/format->mtype format)
:size (fs/size tmp)
:data tmp)))
(defmethod process :generic-thumbnail
[system params]
(let [{:keys [quality width height] :as params}
(check-thumbnail-params params)]
(generic-process system
(assoc params
:convert-args ["-auto-orient" "-strip"
"-thumbnail" (str width "x" height ">")
"-quality" (str quality)]))))
(defmethod process :profile-thumbnail
[system params]
(let [{:keys [quality width height] :as params}
(check-thumbnail-params params)]
(generic-process system
(assoc params
:convert-args ["-auto-orient" "-strip"
"-thumbnail" (str width "x" height "^")
"-gravity" "center"
"-extent" (str width "x" height)
"-quality" (str quality)]))))
(defn get-basic-info-from-svg
[{:keys [tag attrs] :as data}]
(when (not= tag :svg)
(ex/raise :type :validation
:code :unable-to-parse-svg
:hint "uploaded svg has invalid content"))
(reduce (fn [default f]
(if-let [res (f attrs)]
(reduced res)
default))
{:width 100 :height 100}
[(fn parse-width-and-height
[{:keys [width height]}]
(when (and (string? width)
(string? height))
(let [width (d/parse-double width)
height (d/parse-double height)]
(when (and width height)
{:width (int width)
:height (int height)}))))
(fn parse-viewbox
[{:keys [viewBox]}]
(let [[x y width height] (->> (str/split viewBox #"\s+" 4)
(map d/parse-double))]
(when (and x y width height)
{:width (int width)
:height (int height)})))]))
(defn- get-dimensions-with-orientation [system ^String path]
;; Image magick doesn't give info about exif rotation so we use the identify command
;; If we are processing an animated gif we use the first frame with -scene 0
(let [dim-result (exec-magick! system ["identify" "-format" "%w %h\n" path])
orient-result (exec-magick! system ["identify" "-format" "%[EXIF:Orientation]\n" path])]
(when (= 0 (:exit dim-result))
(let [[w h] (-> (:out dim-result)
str/trim
(clojure.string/split #"\s+")
(->> (mapv #(Integer/parseInt %))))
orientation-exit (:exit orient-result)
orientation (-> orient-result :out str/trim)]
(if (= 0 orientation-exit)
(case orientation
("6" "8") {:width h :height w} ; Rotated 90 or 270 degrees
{:width w :height h}) ; Normal or unknown orientation
{:width w :height h}))))) ; If orientation can't be read, use dimensions as-is
(defmethod process :info
[system {:keys [input] :as params}]
(let [{:keys [path mtype] :as input} (check-input input)]
(if (= mtype "image/svg+xml")
(let [info (some-> path slurp parse-svg get-basic-info-from-svg)]
(when-not info
(ex/raise :type :validation
:code :invalid-svg-file
:hint "uploaded svg does not provides dimensions"))
(merge input info {:ts (ct/now) :size (fs/size path)}))
(let [path-str (str path)
identify-res (exec-magick! system ["identify" "-format" "image/%[magick]\n" path-str])
;; identify prints one line per frame (animated GIFs, etc.); we take the first one
mtype' (if (zero? (:exit identify-res))
(-> identify-res
:out
str/trim
(str/split #"\s+" 2)
first
str/lower)
(ex/raise :type :validation
:code :invalid-image
:hint "invalid image"))
{:keys [width height]}
(or (get-dimensions-with-orientation system path-str)
(do
(l/warn "Failed to read image dimensions with orientation" {:path path})
(ex/raise :type :validation
:code :invalid-image
:hint "invalid image")))]
(when (and (string? mtype)
(not= (str/lower mtype) mtype'))
(ex/raise :type :validation
:code :media-type-mismatch
:hint (str "Seems like you are uploading a file whose content does not match the extension."
"Expected: " mtype ". Got: " mtype')))
(assoc input
:width width
:height height
:size (fs/size path)
:ts (ct/now))))))
(if (contains? cf/flags :remote-media-processing)
(media.remote/process system params)
(media.local/process system params)))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; IMAGE HELPERS
@ -338,8 +59,8 @@
:hint "seems like the url points to resource with unknown size"))
(-> {:size size :mtype mtype}
(validate-media-type!)
(validate-media-size!))))]
(validation/validate-media-type!)
(validation/validate-media-size!))))]
(let [{:keys [body] :as response}
(try
@ -367,188 +88,24 @@
(ex/raise :type :validation
:code :unable-to-download-image
:hint (str/ffmt "unable to download image from '%': I/O error" uri)
:cause cause)))
:cause cause)))]
{:keys [size mtype]} (parse-and-validate response)
path (tmp/tempfile :prefix "penpot.media.download.")
written (io/write* path body :size size)]
(if body
(with-open [body body]
(let [{:keys [size mtype]} (parse-and-validate response)
path (tmp/tempfile :prefix "penpot.media.download.")
written (io/write* path body :size size)]
(when (not= written size)
(ex/raise :type :internal
:code :mismatch-write-size
:hint "unexpected state: unable to write to file"))
(when (not= written size)
(ex/raise :type :internal
:code :mismatch-write-size
:hint "unexpected state: unable to write to file"))
;; Sanitize: strip trailing data after image EOF markers
(let [new-size (sanitize/truncate-after-eof path mtype)]
{:path path
:mtype mtype
:size new-size}))))
;; Sanitize: strip trailing data after image EOF markers
(let [new-size (sanitize/truncate-after-eof path mtype)]
{:path path
:mtype mtype
:size new-size})))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; FONTS
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defn- get-font-prlimit
"Returns resource limits for font processing tools, read from config."
[]
{:mem (cf/get :font-process-mem)
:cpu (cf/get :font-process-cpu)})
(defn- get-font-timeout
"Returns the wall-clock timeout for font processing, read from config."
[]
(cf/get :font-process-timeout))
(defn- exec-font!
"Execute a font processing command with resource limits.
`args` is a vector of string arguments."
[system args]
(shell/exec! system
:cmd args
:prlimit (get-font-prlimit)
:timeout (get-font-timeout)))
(defmethod process :generate-fonts
[system {:keys [input] :as params}]
(letfn [(ttf->otf [data]
(let [finput (tmp/tempfile :prefix "penpot.font." :suffix "")
foutput (fs/path (str finput ".otf"))]
(try
(io/write* finput data)
(let [res (exec-font! system ["fontforge" "-lang=ff" "-c"
(str/fmt "Open('%s'); Generate('%s')"
(str finput)
(str foutput))])]
(when (zero? (:exit res))
foutput))
(finally
(fs/delete finput)))))
(otf->ttf [data]
(let [finput (tmp/tempfile :prefix "penpot.font." :suffix "")
foutput (fs/path (str finput ".ttf"))]
(try
(io/write* finput data)
(let [res (exec-font! system ["fontforge" "-lang=ff" "-c"
(str/fmt "Open('%s'); Generate('%s')"
(str finput)
(str foutput))])]
(when (zero? (:exit res))
foutput))
(finally
(fs/delete finput)))))
(ttf-or-otf->woff [data]
(let [finput (tmp/tempfile :prefix "penpot.font." :suffix "")
foutput (fs/path (str finput ".woff"))]
(try
(io/write* finput data)
(let [res (exec-font! system ["sfnt2woff" (str finput)])]
(when (zero? (:exit res))
foutput))
(finally
(fs/delete finput)))))
(woff->sfnt [data]
(let [finput (tmp/tempfile :prefix "penpot" :suffix "")]
(try
(io/write* finput data)
(let [res (shell/exec! system
:cmd ["woff2sfnt" (str finput)]
:out-enc :bytes
:prlimit (get-font-prlimit)
:timeout (get-font-timeout))]
(when (zero? (:exit res))
(:out res)))
(finally
(fs/delete finput)))))
(woff2->sfnt [data]
;; woff2_decompress outputs to same directory with .ttf extension
(let [finput (tmp/tempfile :prefix "penpot.font." :suffix ".woff2")
foutput (fs/path (str/replace (str finput) #"\.woff2$" ".ttf"))]
(try
(io/write* finput data)
(let [res (exec-font! system ["woff2_decompress" (str finput)])]
(if (zero? (:exit res))
foutput
(do
(when (fs/exists? foutput)
(fs/delete foutput))
nil)))
(finally
(fs/delete finput)))))
;; Documented here:
;; https://docs.microsoft.com/en-us/typography/opentype/spec/otff#table-directory
(get-sfnt-type [data]
(let [buff (bb/slice data 0 4)
type (bc/bytes->hex buff)]
(case type
"4f54544f" :otf
"00010000" :ttf
(ex/raise :type :internal
:code :unexpected-data
:hint "unexpected font data"))))
(gen-if-nil [val factory]
(if (nil? val)
(factory)
val))]
(let [current (into #{} (keys input))]
(cond
(contains? current "font/ttf")
(let [data (get input "font/ttf")]
(-> input
(update "font/otf" gen-if-nil #(ttf->otf data))
(update "font/woff" gen-if-nil #(ttf-or-otf->woff data))))
(contains? current "font/otf")
(let [data (get input "font/otf")]
(-> input
(update "font/woff" gen-if-nil #(ttf-or-otf->woff data))
(assoc "font/ttf" (otf->ttf data))))
(contains? current "font/woff")
(let [data (get input "font/woff")
sfnt (woff->sfnt data)]
(when-not sfnt
(ex/raise :type :validation
:code :invalid-woff-file
:hint "invalid woff file"))
(let [stype (get-sfnt-type sfnt)]
(cond-> input
true
(-> (assoc "font/woff" data))
(= stype :otf)
(-> (assoc "font/otf" sfnt)
(assoc "font/ttf" (otf->ttf sfnt)))
(= stype :ttf)
(-> (assoc "font/otf" (ttf->otf sfnt))
(assoc "font/ttf" sfnt)))))
(contains? current "font/woff2")
(let [data (get input "font/woff2")
foutput (woff2->sfnt data)]
(when-not foutput
(ex/raise :type :validation
:code :invalid-woff2-file
:hint "invalid woff2 file"))
(try
(let [sfnt (io/read* foutput)
type (get-sfnt-type sfnt)]
(cond-> input
(= type :otf)
(-> (assoc "font/otf" sfnt)
(assoc "font/ttf" (otf->ttf sfnt))
(update "font/woff" gen-if-nil #(ttf-or-otf->woff sfnt)))
(= type :ttf)
(-> (assoc "font/ttf" sfnt)
(assoc "font/otf" (ttf->otf sfnt))
(update "font/woff" gen-if-nil #(ttf-or-otf->woff sfnt)))))
(finally
(fs/delete foutput))))))))
;; No body - validation will raise appropriate error
(parse-and-validate response)))))

426
backend/src/app/media/local.clj Normal file
View File

@ -0,0 +1,426 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
(ns app.media.local
"Local media processing via ImageMagick and FontForge shell commands."
(:require
[app.common.data :as d]
[app.common.data.macros :as dm]
[app.common.exceptions :as ex]
[app.common.logging :as l]
[app.common.media :as cm]
[app.common.schema :as sm]
[app.common.time :as ct]
[app.config :as cf]
[app.media.validation :as validation]
[app.storage.tmp :as tmp]
[app.util.shell :as shell]
[buddy.core.bytes :as bb]
[buddy.core.codecs :as bc]
[clojure.string]
[clojure.xml :as xml]
[cuerdas.core :as str]
[datoteka.fs :as fs]
[datoteka.io :as io])
(:import
clojure.lang.XMLHandler
java.io.InputStream
javax.xml.parsers.SAXParserFactory
javax.xml.XMLConstants
org.apache.commons.io.IOUtils))
(defmulti process (fn [_system params] (:cmd params)))
(defmethod process :default
[_system {:keys [cmd] :as params}]
(ex/raise :type :internal
:code :not-implemented
:hint (str/fmt "No impl found for local process cmd: %s" cmd)))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; SVG PARSING
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defn- secure-parser-factory
[^InputStream input ^XMLHandler handler]
(.. (doto (SAXParserFactory/newInstance)
(.setFeature XMLConstants/FEATURE_SECURE_PROCESSING true)
(.setFeature "http://apache.org/xml/features/disallow-doctype-decl" true))
(newSAXParser)
(parse input handler)))
(defn- strip-doctype
[data]
(cond-> data
(str/includes? data "<!DOCTYPE")
(str/replace #"<\!DOCTYPE[^>]*>" "")))
(defn parse-svg
[text]
(let [text (strip-doctype text)]
(dm/with-open [istream (IOUtils/toInputStream ^String text "UTF-8")]
(xml/parse istream secure-parser-factory))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; IMAGE THUMBNAILS
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(def ^:private schema:thumbnail-params
[:map {:title "ThumbnailParams"}
[:input validation/schema:input]
[:format [:enum :jpeg :webp :png]]
[:quality [:int {:min 1 :max 100}]]
[:width :int]
[:height :int]])
(def ^:private check-thumbnail-params
(sm/check-fn schema:thumbnail-params))
;; Related info on how thumbnails generation
;; http://www.imagemagick.org/Usage/thumbnails/
(def ^:private imagemagick-default-env
"Default environment variables for ImageMagick resource limits.
These are the soft ceiling — policy.xml is the hard ceiling."
{"MAGICK_THREAD_LIMIT" "2"
"MAGICK_MEMORY_LIMIT" "256MiB"
"MAGICK_MAP_LIMIT" "512MiB"
"MAGICK_AREA_LIMIT" "128MP"
"MAGICK_DISK_LIMIT" "1GiB"
"MAGICK_TIME_LIMIT" "30"})
(defn- get-imagemagick-env
"Returns environment variables for ImageMagick commands.
Reads individual PENPOT_IMAGEMAGICK_* config values, falling back to defaults."
[]
(let [thread (cf/get :imagemagick-thread-limit)
memory (cf/get :imagemagick-memory-limit)
map-l (cf/get :imagemagick-map-limit)
area (cf/get :imagemagick-area-limit)
disk (cf/get :imagemagick-disk-limit)
time (cf/get :imagemagick-time-limit)
width (cf/get :imagemagick-width-limit)
height (cf/get :imagemagick-height-limit)]
(cond-> imagemagick-default-env
thread (assoc "MAGICK_THREAD_LIMIT" thread)
memory (assoc "MAGICK_MEMORY_LIMIT" memory)
map-l (assoc "MAGICK_MAP_LIMIT" map-l)
area (assoc "MAGICK_AREA_LIMIT" area)
disk (assoc "MAGICK_DISK_LIMIT" disk)
time (assoc "MAGICK_TIME_LIMIT" time)
width (assoc "MAGICK_WIDTH_LIMIT" width)
height (assoc "MAGICK_HEIGHT_LIMIT" height))))
(defn- exec-magick!
"Execute an ImageMagick command with resource limits.
`args` is a vector of string arguments to pass to `magick`."
[system args]
(let [cmd (into ["magick"] args)
result (shell/exec! system
:cmd cmd
:env (get-imagemagick-env)
:timeout 60)]
(when (not= 0 (:exit result))
(ex/raise :type :validation
:code :invalid-image
:hint (str "ImageMagick command failed: " (:err result))
:cmd cmd
:exit (:exit result)))
result))
(defn- generic-process
[system {:keys [input format convert-args] :as params}]
(let [{:keys [path mtype]} input
format (or format (cm/mtype->format mtype))
ext (cm/format->extension format)
tmp (tmp/tempfile :prefix "penpot.media." :suffix ext)
args (into [(str path)] (conj (vec convert-args) (str tmp)))]
(exec-magick! system args)
(assoc params
:format format
:mtype (cm/format->mtype format)
:size (fs/size tmp)
:data tmp)))
(defmethod process :generic-thumbnail
[system params]
(let [{:keys [quality width height] :as params}
(check-thumbnail-params params)]
(generic-process system
(assoc params
:convert-args ["-auto-orient" "-strip"
"-thumbnail" (str width "x" height ">")
"-quality" (str quality)]))))
(defmethod process :profile-thumbnail
[system params]
(let [{:keys [quality width height] :as params}
(check-thumbnail-params params)]
(generic-process system
(assoc params
:convert-args ["-auto-orient" "-strip"
"-thumbnail" (str width "x" height "^")
"-gravity" "center"
"-extent" (str width "x" height)
"-quality" (str quality)]))))
(defn get-basic-info-from-svg
[{:keys [tag attrs] :as data}]
(when (not= tag :svg)
(ex/raise :type :validation
:code :unable-to-parse-svg
:hint "uploaded svg has invalid content"))
(reduce (fn [default f]
(if-let [res (f attrs)]
(reduced res)
default))
{:width 100 :height 100}
[(fn parse-width-and-height
[{:keys [width height]}]
(when (and (string? width)
(string? height))
(let [width (d/parse-double width)
height (d/parse-double height)]
(when (and width height)
{:width (int width)
:height (int height)}))))
(fn parse-viewbox
[{:keys [viewBox]}]
(let [[x y width height] (->> (str/split viewBox #"\s+" 4)
(map d/parse-double))]
(when (and x y width height)
{:width (int width)
:height (int height)})))]))
(defn- get-dimensions-with-orientation [system ^String path]
;; Image magick doesn't give info about exif rotation so we use the identify command
;; If we are processing an animated gif we use the first frame with -scene 0
(let [dim-result (exec-magick! system ["identify" "-format" "%w %h\n" path])
orient-result (exec-magick! system ["identify" "-format" "%[EXIF:Orientation]\n" path])]
(when (= 0 (:exit dim-result))
(let [[w h] (-> (:out dim-result)
str/trim
(clojure.string/split #"\s+")
(->> (mapv #(Integer/parseInt %))))
orientation-exit (:exit orient-result)
orientation (-> orient-result :out str/trim)]
(if (= 0 orientation-exit)
(case orientation
("6" "8") {:width h :height w} ; Rotated 90 or 270 degrees
{:width w :height h}) ; Normal or unknown orientation
{:width w :height h}))))) ; If orientation can't be read, use dimensions as-is
(defmethod process :info
[system {:keys [input] :as params}]
(let [{:keys [path mtype] :as input} (validation/check-input input)]
(if (= mtype "image/svg+xml")
(let [info (some-> path slurp parse-svg get-basic-info-from-svg)]
(when-not info
(ex/raise :type :validation
:code :invalid-svg-file
:hint "uploaded svg does not provides dimensions"))
(merge input info {:ts (ct/now) :size (fs/size path)}))
(let [path-str (str path)
identify-res (exec-magick! system ["identify" "-format" "image/%[magick]\n" path-str])
;; identify prints one line per frame (animated GIFs, etc.); we take the first one
mtype' (if (zero? (:exit identify-res))
(-> identify-res
:out
str/trim
(str/split #"\s+" 2)
first
str/lower)
(ex/raise :type :validation
:code :invalid-image
:hint "invalid image"))
{:keys [width height]}
(or (get-dimensions-with-orientation system path-str)
(do
(l/warn "Failed to read image dimensions with orientation" {:path path})
(ex/raise :type :validation
:code :invalid-image
:hint "invalid image")))]
(when (and (string? mtype)
(not= (str/lower mtype) mtype'))
(ex/raise :type :validation
:code :media-type-mismatch
:hint (str "Seems like you are uploading a file whose content does not match the extension."
"Expected: " mtype ". Got: " mtype')))
(assoc input
:width width
:height height
:size (fs/size path)
:ts (ct/now))))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; FONTS
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defn- get-font-prlimit
"Returns resource limits for font processing tools, read from config."
[]
{:mem (cf/get :font-process-mem)
:cpu (cf/get :font-process-cpu)})
(defn- get-font-timeout
"Returns the wall-clock timeout for font processing, read from config."
[]
(cf/get :font-process-timeout))
(defn- exec-font!
"Execute a font processing command with resource limits.
`args` is a vector of string arguments."
[system args]
(shell/exec! system
:cmd args
:prlimit (get-font-prlimit)
:timeout (get-font-timeout)))
(defmethod process :generate-fonts
[system {:keys [input] :as params}]
(letfn [(ttf->otf [data]
(let [finput (tmp/tempfile :prefix "penpot.font." :suffix "")
foutput (fs/path (str finput ".otf"))]
(try
(io/write* finput data)
(let [res (exec-font! system ["fontforge" "-lang=ff" "-c"
(str/fmt "Open('%s'); Generate('%s')"
(str finput)
(str foutput))])]
(when (zero? (:exit res))
foutput))
(finally
(fs/delete finput)))))
(otf->ttf [data]
(let [finput (tmp/tempfile :prefix "penpot.font." :suffix "")
foutput (fs/path (str finput ".ttf"))]
(try
(io/write* finput data)
(let [res (exec-font! system ["fontforge" "-lang=ff" "-c"
(str/fmt "Open('%s'); Generate('%s')"
(str finput)
(str foutput))])]
(when (zero? (:exit res))
foutput))
(finally
(fs/delete finput)))))
(ttf-or-otf->woff [data]
(let [finput (tmp/tempfile :prefix "penpot.font." :suffix "")
foutput (fs/path (str finput ".woff"))]
(try
(io/write* finput data)
(let [res (exec-font! system ["sfnt2woff" (str finput)])]
(when (zero? (:exit res))
foutput))
(finally
(fs/delete finput)))))
(woff->sfnt [data]
(let [finput (tmp/tempfile :prefix "penpot" :suffix "")]
(try
(io/write* finput data)
(let [res (shell/exec! system
:cmd ["woff2sfnt" (str finput)]
:out-enc :bytes
:prlimit (get-font-prlimit)
:timeout (get-font-timeout))]
(when (zero? (:exit res))
(:out res)))
(finally
(fs/delete finput)))))
(woff2->sfnt [data]
;; woff2_decompress outputs to same directory with .ttf extension
(let [finput (tmp/tempfile :prefix "penpot.font." :suffix ".woff2")
foutput (fs/path (str/replace (str finput) #"\.woff2$" ".ttf"))]
(try
(io/write* finput data)
(let [res (exec-font! system ["woff2_decompress" (str finput)])]
(if (zero? (:exit res))
foutput
(do
(when (fs/exists? foutput)
(fs/delete foutput))
nil)))
(finally
(fs/delete finput)))))
;; Documented here:
;; https://docs.microsoft.com/en-us/typography/opentype/spec/otff#table-directory
(get-sfnt-type [data]
(let [buff (bb/slice data 0 4)
type (bc/bytes->hex buff)]
(case type
"4f54544f" :otf
"00010000" :ttf
(ex/raise :type :internal
:code :unexpected-data
:hint "unexpected font data"))))
(gen-if-nil [val factory]
(if (nil? val)
(factory)
val))]
(let [current (into #{} (keys input))]
(cond
(contains? current "font/ttf")
(let [data (get input "font/ttf")]
(-> input
(update "font/otf" gen-if-nil #(ttf->otf data))
(update "font/woff" gen-if-nil #(ttf-or-otf->woff data))))
(contains? current "font/otf")
(let [data (get input "font/otf")]
(-> input
(update "font/woff" gen-if-nil #(ttf-or-otf->woff data))
(assoc "font/ttf" (otf->ttf data))))
(contains? current "font/woff")
(let [data (get input "font/woff")
sfnt (woff->sfnt data)]
(when-not sfnt
(ex/raise :type :validation
:code :invalid-woff-file
:hint "invalid woff file"))
(let [stype (get-sfnt-type sfnt)]
(cond-> input
true
(-> (assoc "font/woff" data))
(= stype :otf)
(-> (assoc "font/otf" sfnt)
(assoc "font/ttf" (otf->ttf sfnt)))
(= stype :ttf)
(-> (assoc "font/otf" (ttf->otf sfnt))
(assoc "font/ttf" sfnt)))))
(contains? current "font/woff2")
(let [data (get input "font/woff2")
foutput (woff2->sfnt data)]
(when-not foutput
(ex/raise :type :validation
:code :invalid-woff2-file
:hint "invalid woff2 file"))
(try
(let [sfnt (io/read* foutput)
type (get-sfnt-type sfnt)]
(cond-> input
(= type :otf)
(-> (assoc "font/otf" sfnt)
(assoc "font/ttf" (otf->ttf sfnt))
(update "font/woff" gen-if-nil #(ttf-or-otf->woff sfnt)))
(= type :ttf)
(-> (assoc "font/ttf" sfnt)
(assoc "font/otf" (ttf->otf sfnt))
(update "font/woff" gen-if-nil #(ttf-or-otf->woff sfnt)))))
(finally
(fs/delete foutput))))))))

264
backend/src/app/media/remote.clj Normal file
View File

@ -0,0 +1,264 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
(ns app.media.remote
"Remote media processing via the media-processor HTTP service."
(:require
[app.common.exceptions :as ex]
[app.common.media :as cm]
[app.common.time :as ct]
[app.common.uri :as uri]
[app.config :as cf]
[app.http.client :as http]
[app.media.local :as local]
[app.media.validation :as validation]
[app.setup :as-alias setup]
[app.storage.tmp :as tmp]
[app.util.json :as json]
[cuerdas.core :as str]
[datoteka.fs :as fs]
[datoteka.io :as io])
(:import
java.io.ByteArrayInputStream
java.io.InputStream
java.io.SequenceInputStream
java.net.ConnectException
java.net.http.HttpTimeoutException
java.util.Collections))
(defn- service-base-url
"Returns the base URL of the media-processor service."
[]
(or (cf/get :media-processing-service-uri)
(ex/raise :type :internal
:code :media-processor-not-configured
:hint "PENPOT_MEDIA_PROCESSING_SERVICE_URI is not configured")))
(defn- service-timeout
"Returns the HTTP timeout (ms) for media-processor requests."
[]
(or (cf/get :media-processing-service-timeout)
120000))
(defn- get-shared-key
"Returns the shared key for authenticating with the media-processor."
[system]
(-> system ::setup/shared-keys :media-processor))
(defn- parse-json-response
"Parse a JSON response body."
[body]
(json/read! body))
(defn- translate-error
"Translate a media-processor error response into a Penpot exception."
[status body]
(let [code (or (:code body) "media-processor-error")
hint (or (:hint body) "media-processor request failed")]
(case status
400 {:type :validation :code (keyword code) :hint hint}
403 {:type :authorization :code :forbidden :hint hint}
413 {:type :restriction :code (keyword code) :hint hint}
504 {:type :internal :code :media-processor-timeout :hint hint}
{:type :internal :code (keyword code) :hint hint})))
(defn service-request
"Make an HTTP request to the media-processor service."
[system {:keys [method uri body headers timeout]}]
(let [client (::http/client system)
timeout (or timeout (service-timeout))]
(try
(let [resp (http/req client
{:method method
:uri uri
:body body
:headers headers}
{:response-type :input-stream
:skip-ssrf-check? true
:timeout timeout})
status (:status resp)]
(when (not (<= 200 status 299))
(let [body (:body resp)]
(try
(let [parsed (try (parse-json-response body) (catch Exception _ nil))
err (translate-error status parsed)]
(ex/raise :type (:type err) :code (:code err) :hint (:hint err)))
(finally
(.close body)))))
resp)
(catch ConnectException _cause
(ex/raise :type :internal
:code :media-processor-unavailable
:hint "Cannot connect to media-processor service"))
(catch HttpTimeoutException _cause
(ex/raise :type :internal
:code :media-processor-timeout
:hint "media-processor service request timed out")))))
(defn- multipart-boundary
[]
(str "----PenpotBoundary" (System/currentTimeMillis)))
(defn- build-multipart-stream
"Build a streaming multipart/form-data body with a single file field.
Returns an InputStream that lazily reads from the file on demand."
[^String boundary mtype ^InputStream file-stream]
(let [header (.getBytes (str "--" boundary "\r\n"
"Content-Disposition: form-data; name=\"file\"; filename=\"file\"\r\n"
"Content-Type: " mtype "\r\n"
"\r\n")
"UTF-8")
footer (.getBytes (str "\r\n--" boundary "--\r\n")
"UTF-8")
parts (Collections/enumeration
[(ByteArrayInputStream. header)
file-stream
(ByteArrayInputStream. footer)])]
(SequenceInputStream. parts)))
(defn- service-multipart-request
"Send a multipart request to the media-processor service.
Accepts a file from disk via :path. The file stream is closed
after the HTTP request completes (success or failure)."
[system {:keys [endpoint path mtype query timeout]}]
(let [shared-key (get-shared-key system)
boundary (multipart-boundary)
ctype (or mtype "application/octet-stream")
base-url (service-base-url)
request-uri (cond-> (uri/join base-url endpoint)
(seq query)
(str "?" (uri/map->query-string query)))]
(with-open [file-stream (io/input-stream path)]
(let [body (build-multipart-stream boundary ctype file-stream)]
(service-request system
{:method :post
:uri request-uri
:body body
:headers {"Content-Type" (str "multipart/form-data; boundary=" boundary)
"x-shared-key" shared-key}
:timeout timeout})))))
(def ^:private known-font-types
"Priority-ordered list of font mime-types the system knows how to convert.
Order matters: when a font upload contains multiple variants, the first
match becomes the conversion source (ttf preferred for best coverage)."
["font/ttf" "font/otf" "font/woff" "font/woff2"])
(defn- font-convert
"Convert a font to the given target mime-type via the media-processor service.
Accepts source font data as a filesystem Path. Returns a tempfile Path."
[system source-mtype target-mtype data]
(let [resp (service-multipart-request system {:endpoint "api/font/convert"
:path data
:mtype source-mtype
:query {:target-type target-mtype}
:timeout 180000})
ext (cm/mtype->extension target-mtype)
tmp (tmp/tempfile :prefix "penpot.font." :suffix ext)
body (:body resp)]
(try
(io/write* tmp body)
(finally
(.close body)))
tmp))
(defn- font-missing-variants
"Return the set of target mime-types that should be generated for the given
source mime-type (excluding font/woff2, which is never generated)."
[source-mtype]
(case source-mtype
"font/ttf" #{"font/otf" "font/woff"}
"font/otf" #{"font/ttf" "font/woff"}
"font/woff" #{"font/ttf" "font/otf"}
"font/woff2" #{"font/ttf" "font/otf" "font/woff"}))
(defmulti process (fn [_system params] (:cmd params)))
(defmethod process :info
[system {:keys [input]}]
(let [{:keys [path mtype]} (validation/check-input input)]
(if (= mtype "image/svg+xml")
;; SVG: parse locally (Sharp doesn't support SVG)
(let [info (some-> path slurp local/parse-svg local/get-basic-info-from-svg)]
(when-not info
(ex/raise :type :validation
:code :invalid-svg-file
:hint "uploaded svg does not provide dimensions"))
(merge input info {:ts (ct/now) :size (fs/size path)}))
;; Raster: delegate to media-processor
(let [resp (service-multipart-request system {:endpoint "api/image/info"
:path path
:mtype mtype})
body (:body resp)]
(try
(let [info (parse-json-response body)
detected-mtype (:mtype info)]
(when (and (string? mtype)
(string? detected-mtype)
(not= (str/lower mtype) (str/lower detected-mtype)))
(ex/raise :type :validation
:code :media-type-mismatch
:hint (str "File content does not match the declared type. "
"Expected: " mtype ". Got: " detected-mtype)))
(assoc input
:width (:width info)
:height (:height info)
:size (fs/size path)
:ts (ct/now)))
(finally
(.close body)))))))
(defn- thumbnail-request
"Shared implementation for generic-thumbnail and profile-thumbnail."
[system params mode]
(let [{:keys [input format quality width height]} params
{:keys [path mtype]} (validation/check-input input)
fmt (name (or format (cm/mtype->format mtype) :jpeg))
resp (service-multipart-request system {:endpoint "api/image/thumbnail"
:path path
:mtype mtype
:query {:width width
:height height
:quality quality
:format fmt
:mode mode}})
out-format (or format (cm/mtype->format mtype) :jpeg)
ext (cm/format->extension out-format)
tmp (tmp/tempfile :prefix "penpot.media." :suffix ext)
body (:body resp)]
(try
(io/write* tmp body)
(finally
(.close body)))
(assoc params
:format out-format
:mtype (cm/format->mtype out-format)
:size (fs/size tmp)
:data tmp)))
(defmethod process :generic-thumbnail
[system params]
(thumbnail-request system params "fit"))
(defmethod process :profile-thumbnail
[system params]
(thumbnail-request system params "crop"))
(defmethod process :generate-fonts
[system {:keys [input]}]
(let [source-mtype (or (some #(when (contains? input %) %) known-font-types)
(ex/raise :type :validation
:code :invalid-font
:hint "No recognized font variant in input"))
data (get input source-mtype)
present (set (keys input))
targets (remove present (font-missing-variants source-mtype))]
(reduce (fn [acc target-mtype]
(assoc acc target-mtype
(font-convert system source-mtype target-mtype data)))
input
targets)))

68
backend/src/app/media/validation.clj Normal file
View File

@ -0,0 +1,68 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
(ns app.media.validation
"Schemas and validation functions for media uploads.
Leaf namespace — depends on app.common.* and app.config only."
(:require
[app.common.exceptions :as ex]
[app.common.media :as cm]
[app.common.schema :as sm]
[app.config :as cf]
[cuerdas.core :as str]
[datoteka.fs :as fs]))
(def schema:upload
[:map {:title "Upload"}
[:filename :string]
[:size ::sm/int]
[:path ::fs/path]
[:mtype {:optional true} :string]
[:headers {:optional true}
[:map-of :string :string]]])
(def schema:input
[:map {:title "Input"}
[:path ::fs/path]
[:mtype {:optional true} ::sm/text]])
(def check-input
(sm/check-fn schema:input))
(defn validate-media-type!
([upload] (validate-media-type! upload cm/image-types))
([upload allowed]
(when-not (contains? allowed (:mtype upload))
(ex/raise :type :validation
:code :media-type-not-allowed
:hint "Seems like you are uploading an invalid media object"))
upload))
(defn validate-media-size!
[upload]
(let [max-size (cf/get :media-max-file-size)]
(when (> (:size upload) max-size)
(ex/raise :type :restriction
:code :media-max-file-size-reached
:hint (str/ffmt "the uploaded file size % is greater than the maximum %"
(:size upload)
max-size)))
upload))
(defn validate-font-size!
"Validates that the font file `upload` does not exceed the configured
`:font-max-file-size` limit. Accepts the same map shape as
`validate-media-size!` — requires a `:size` key in bytes."
[upload]
(let [max-size (cf/get :font-max-file-size)]
(when (> (:size upload) max-size)
(ex/raise :type :restriction
:code :font-max-file-size-reached
:hint (str/ffmt "the uploaded font size % is greater than the maximum %"
(:size upload)
max-size)))
upload))

4
backend/src/app/rpc/commands/binfile.clj
View File

@ -19,7 +19,7 @@
[app.http.sse :as sse]
[app.loggers.audit :as-alias audit]
[app.loggers.webhooks :as-alias webhooks]
[app.media :as media]
[app.media.validation :as media.v]
[app.rpc :as-alias rpc]
[app.rpc.commands.files :as files]
[app.rpc.commands.media :as media-cmd]
@ -124,7 +124,7 @@
[:project-id ::sm/uuid]
[:file-id {:optional true} ::sm/uuid]
[:version {:optional true} ::sm/int]
[:file {:optional true} media/schema:upload]
[:file {:optional true} media.v/schema:upload]
[:upload-id {:optional true} ::sm/uuid]]
[:fn {:error/message "one of :file or :upload-id is required"}
(fn [{:keys [file upload-id]}]

14
backend/src/app/rpc/commands/files_thumbnails.clj
View File

@ -21,7 +21,7 @@
[app.db.sql :as-alias sql]
[app.loggers.audit :as-alias audit]
[app.loggers.webhooks :as-alias webhooks]
[app.media :as media]
[app.media.validation :as media.v]
[app.rpc :as-alias rpc]
[app.rpc.climit :as-alias climit]
[app.rpc.commands.files :as files]
@ -275,7 +275,7 @@
[:map {:title "create-file-object-thumbnail"}
[:file-id ::sm/uuid]
[:object-id [:string {:max 250}]]
[:media media/schema:upload]
[:media media.v/schema:upload]
[:tag {:optional true} [:string {:max 50}]]])
(sv/defmethod ::create-file-object-thumbnail
@ -289,8 +289,8 @@
::sm/params schema:create-file-object-thumbnail}
[cfg {:keys [::rpc/profile-id file-id object-id media tag]}]
(media/validate-media-type! media)
(media/validate-media-size! media)
(media.v/validate-media-type! media)
(media.v/validate-media-size! media)
(db/run! cfg files/check-edition-permissions! profile-id file-id)
(when-let [file (files/get-minimal-file cfg file-id {::db/check-deleted false})]
@ -379,7 +379,7 @@
[:map {:title "create-file-thumbnail"}
[:file-id ::sm/uuid]
[:revn ::sm/int]
[:media media/schema:upload]])
[:media media.v/schema:upload]])
(sv/defmethod ::create-file-thumbnail
"Creates or updates the file thumbnail. Mainly used for paint the
@ -394,8 +394,8 @@
::sm/params schema:create-file-thumbnail}
[cfg {:keys [::rpc/profile-id file-id] :as params}]
(media/validate-media-type! (:media params))
(media/validate-media-size! (:media params))
(media.v/validate-media-type! (:media params))
(media.v/validate-media-size! (:media params))
(db/run! cfg files/check-edition-permissions! profile-id file-id)

78
backend/src/app/rpc/commands/fonts.clj
View File

@ -21,6 +21,7 @@
[app.loggers.audit :as-alias audit]
[app.loggers.webhooks :as-alias webhooks]
[app.media :as media]
[app.media.validation :as media.v]
[app.rpc :as-alias rpc]
[app.rpc.climit :as-alias climit]
[app.rpc.commands.files :as files]
@ -38,10 +39,7 @@
[datoteka.fs :as fs]
[datoteka.io :as io])
(:import
java.io.InputStream
java.io.OutputStream
java.io.SequenceInputStream
java.util.Collections
java.util.zip.ZipEntry
java.util.zip.ZipOutputStream))
@ -96,18 +94,13 @@
(declare create-font-variant)
(def ^:private schema:create-font-variant
[:and
[:map {:title "create-font-variant"}
[:team-id ::sm/uuid]
[:font-id ::sm/uuid]
[:font-family types.font/schema:font-family]
[:font-weight [::sm/one-of {:format "number"} valid-weight]]
[:font-style [::sm/one-of {:format "string"} valid-style]]
[:data {:optional true} [:map-of ::sm/text [:or ::sm/bytes [::sm/vec ::sm/bytes]]]]
[:uploads {:optional true} [:map-of ::sm/text ::sm/uuid]]]
[:fn {:error/message "one of :data or :uploads is required"}
(fn [{:keys [data uploads]}]
(or (seq data) (seq uploads)))]])
[:map {:title "create-font-variant"}
[:team-id ::sm/uuid]
[:font-id ::sm/uuid]
[:font-family types.font/schema:font-family]
[:font-weight [::sm/one-of {:format "number"} valid-weight]]
[:font-style [::sm/one-of {:format "string"} valid-style]]
[:uploads [:map-of ::sm/text ::sm/uuid]]])
(defn- prepare-font-data-from-uploads
"Assembles each chunked-upload session in `uploads` (a `{mtype →
@ -118,8 +111,8 @@
(fn [acc mtype session-id]
(let [assembled (assemble-chunks cfg session-id)]
(-> {:mtype mtype :size (:size assembled)}
(media/validate-media-type! cm/font-types)
(media/validate-font-size!))
(media.v/validate-media-type! cm/font-types)
(media.v/validate-font-size!))
(assoc acc mtype (:path assembled))))
{}
uploads)]
@ -128,54 +121,23 @@
(assoc :data data)
(dissoc :uploads))))
(defn- prepare-font-data-from-legacy
"Validates the media type and size of every entry in the legacy
`:data` map (a `{mtype → bytes | [bytes]}` map). Normalises every
entry to a tempfile. Returns params with a normalised
`{mtype → path}` data map."
[{:keys [data] :as params}]
(let [data (reduce-kv
(fn [acc mtype content]
(let [tmp (tmp/tempfile :prefix "penpot.tempfont." :suffix "")
chunks (if (vector? content) content [content])
streams (map io/input-stream chunks)
streams (Collections/enumeration streams)]
;; Generate the tempfile from all chunks
(with-open [^OutputStream output (io/output-stream tmp)
^InputStream input (SequenceInputStream. streams)]
(io/copy input output))
;; Validate
(-> {:mtype mtype :size (fs/size tmp)}
(media/validate-media-type! cm/font-types)
(media/validate-font-size!))
(assoc acc mtype tmp)))
{}
data)]
(assoc params :data data)))
(sv/defmethod ::create-font-variant
"Upload a font variant. Font data may be provided either as a
Transit-encoded `:data` map (keyed by mime-type) for small fonts, or
as an `:uploads` map (keyed by mime-type, values are upload-session
UUIDs from the chunked-upload API) for large fonts. Exactly one of
the two must be present."
"Upload a font variant. Font data must be provided as an `:uploads`
map (keyed by mime-type, values are upload-session UUIDs from the
chunked-upload API)."
{::doc/added "1.18"
::doc/changes ["2.16" "Add :uploads param for chunked upload support"]
::doc/changes [["2.16" "Add :uploads param for chunked upload support"]
["2.18" "Remove :data param, use :uploads exclusively"]]
::climit/id [[:process-font/by-profile ::rpc/profile-id]
[:process-font/global]]
::webhooks/event? true
::sm/params schema:create-font-variant}
[{:keys [::db/pool] :as cfg} {:keys [::rpc/profile-id team-id uploads] :as params}]
[{:keys [::db/pool] :as cfg} {:keys [::rpc/profile-id team-id] :as params}]
(teams/check-edition-permissions! pool profile-id team-id)
(quotes/check! cfg {::quotes/id ::quotes/font-variants-per-team
::quotes/profile-id profile-id
::quotes/team-id team-id})
(let [params (if (some? uploads)
(db/tx-run! cfg prepare-font-data-from-uploads params)
(prepare-font-data-from-legacy params))]
(let [params (db/tx-run! cfg prepare-font-data-from-uploads params)]
(create-font-variant cfg (assoc params :profile-id profile-id))))
(defn create-font-variant
@ -229,9 +191,7 @@
(let [tpoint (ct/tpoint)
mtypes (vec (keys data))
total-size (reduce-kv (fn [acc _ content]
(+ acc (if (bytes? content)
(alength ^bytes content)
(fs/size content))))
(+ acc (fs/size content)))
0
data)]
@ -370,7 +330,7 @@
(defn- make-temporal-storage-object
[cfg profile-id content]
(let [storage (sto/resolve cfg)
content (media/check-input content)
content (media.v/check-input content)
hash (sto/calculate-hash (:path content))
data (-> (sto/content (:path content))
(sto/wrap-with-hash hash))

15
backend/src/app/rpc/commands/media.clj
View File

@ -16,6 +16,7 @@
[app.db :as db]
[app.loggers.audit :as-alias audit]
[app.media :as media]
[app.media.validation :as media.v]
[app.rpc :as-alias rpc]
[app.rpc.climit :as climit]
[app.rpc.commands.files :as files]
@ -44,7 +45,7 @@
[:file-id ::sm/uuid]
[:is-local ::sm/boolean]
[:name [:string {:max 250}]]
[:content media/schema:upload]])
[:content media.v/schema:upload]])
(sv/defmethod ::upload-file-media-object
{::doc/added "1.17"
@ -53,8 +54,8 @@
[:process-image/global]]}
[{:keys [::db/pool] :as cfg} {:keys [::rpc/profile-id file-id content] :as params}]
(files/check-edition-permissions! pool profile-id file-id)
(media/validate-media-type! content)
(media/validate-media-size! content)
(media.v/validate-media-type! content)
(media.v/validate-media-size! content)
(db/run! cfg (fn [{:keys [::db/conn] :as cfg}]
;; We get the minimal file for proper checking if
@ -315,7 +316,7 @@
[:map {:title "upload-chunk"}
[:session-id ::sm/uuid]
[:index ::sm/int]
[:content media/schema:upload]])
[:content media.v/schema:upload]])
(def ^:private schema:upload-chunk-result
[:map {:title "upload-chunk-result"}
@ -386,7 +387,7 @@
(defn assemble-chunks
"Validates that all expected chunks are present for `session-id` and
concatenates them into a single temporary file. Returns a map
conforming to `media/schema:upload` with `:filename`, `:path` and
conforming to `media.v/schema:upload` with `:filename`, `:path` and
`:size`.
Raises a :validation/:missing-chunks error when the number of stored
@ -440,8 +441,8 @@
content (-> content
(assoc :filename (str "upload:" name))
(assoc :mtype mtype)
(media/validate-media-type!)
(media/validate-media-size!))
(media.v/validate-media-type!)
(media.v/validate-media-size!))
mobj (create-file-media-object cfg (assoc params
:id id
:from-chunks? true

7
backend/src/app/rpc/commands/profile.clj
View File

@ -21,6 +21,7 @@
[app.loggers.audit :as audit]
[app.main :as-alias main]
[app.media :as media]
[app.media.validation :as media.v]
[app.nitrate :as nitrate]
[app.rpc :as-alias rpc]
[app.rpc.climit :as climit]
@ -280,7 +281,7 @@
(def ^:private
schema:update-profile-photo
[:map {:title "update-profile-photo"}
[:file media/schema:upload]])
[:file media.v/schema:upload]])
(sv/defmethod ::update-profile-photo
{:doc/added "1.1"
@ -288,8 +289,8 @@
::sm/result :nil}
[cfg {:keys [::rpc/profile-id file] :as params}]
;; Validate incoming mime type
(media/validate-media-type! file #{"image/jpeg" "image/png" "image/webp"})
(media/validate-media-size! file)
(media.v/validate-media-type! file #{"image/jpeg" "image/png" "image/webp"})
(media.v/validate-media-size! file)
(update-profile-photo cfg (assoc params :profile-id profile-id)))
(defn update-profile-photo

8
backend/src/app/rpc/commands/teams.clj
View File

@ -22,7 +22,7 @@
[app.features.logical-deletion :as ldel]
[app.loggers.audit :as audit]
[app.main :as-alias main]
[app.media :as media]
[app.media.validation :as media.v]
[app.msgbus :as mbus]
[app.nitrate :as nitrate]
[app.rpc :as-alias rpc]
@ -979,7 +979,7 @@
(def ^:private schema:update-team-photo
[:map {:title "update-team-photo"}
[:team-id ::sm/uuid]
[:file media/schema:upload]])
[:file media.v/schema:upload]])
(sv/defmethod ::update-team-photo
{::doc/added "1.17"
@ -987,8 +987,8 @@
[cfg {:keys [::rpc/profile-id file] :as params}]
;; Validate incoming mime type
(media/validate-media-type! file #{"image/jpeg" "image/png" "image/webp"})
(media/validate-media-size! file)
(media.v/validate-media-type! file #{"image/jpeg" "image/png" "image/webp"})
(media.v/validate-media-size! file)
(update-team-photo cfg (assoc params :profile-id profile-id)))
(defn update-team-photo

2
backend/src/app/rpc/management/exporter.clj
View File

@ -10,7 +10,7 @@
[app.common.time :as ct]
[app.common.uri :as u]
[app.config :as cf]
[app.media :refer [schema:upload]]
[app.media.validation :refer [schema:upload]]
[app.rpc :as-alias rpc]
[app.rpc.doc :as doc]
[app.storage :as sto]

4
backend/src/app/rpc/management/nitrate.clj
View File

@ -24,7 +24,7 @@
[app.http :as-alias http]
[app.http.session :as session]
[app.loggers.audit :as audit]
[app.media :as media]
[app.media.validation :as media.v]
[app.nitrate :as nitrate]
[app.rpc :as rpc]
[app.rpc.commands.auth :as auth]
@ -119,7 +119,7 @@
(def ^:private schema:upload-organization-logo
[:map
[:content media/schema:upload]
[:content media.v/schema:upload]
[:organization-id ::sm/uuid]
[:previous-id {:optional true} ::sm/uuid]])

3
backend/src/app/setup.clj
View File

@ -116,7 +116,8 @@
{}
[:exporter
:admin-console
:nexus])))
:nexus
:media-processor])))
(sm/register! ::props [:map-of :keyword ::sm/any])
(sm/register! ::shared-keys [:map-of :keyword ::sm/text])

593
backend/test/backend_tests/media_remote_test.clj Normal file
View File

@ -0,0 +1,593 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
(ns backend-tests.media-remote-test
(:require
[app.common.exceptions :as ex]
[app.config :as cf]
[app.media.remote :as media.remote]
[app.setup :as-alias setup]
[app.util.json :as json]
[backend-tests.helpers :as th]
[clojure.test :as t]
[cuerdas.core :as str]
[datoteka.fs :as fs]
[datoteka.io :as io]
[mockery.core :refer [with-mocks]])
(:import
java.io.ByteArrayInputStream))
(defn- mk-system
"Minimal system map for media.remote/process tests."
[]
{::setup/shared-keys {:media-processor "test-shared-key"}})
(defn- json-stream
"Create an InputStream from a Clojure data structure (JSON-encoded)."
[data]
(ByteArrayInputStream.
(json/encode data)))
(def config-mock
"Standard config mock for media-processor service."
{:media-processing-service-uri "http://localhost:6065"
:media-processing-service-timeout 5000})
(defn- write-font-tmp
"Write font bytes to a tempfile and return the Path. Caller is responsible for cleanup."
[bytes suffix]
(let [tmp (fs/create-tempfile :prefix "penpot-test-font-" :suffix suffix)]
(io/write* tmp bytes)
tmp))
;; ---------------------------------------------------------------------------
;; :info
;; ---------------------------------------------------------------------------
(t/deftest info-happy-path
(t/testing "info returns dimensions and merges into input"
(with-mocks [mock {:target 'app.media.remote/service-request
:return {:status 200
:body (json-stream {:width 800 :height 600 :mtype "image/jpeg" :size 12345 :orientation 1})}}]
(with-redefs [cf/get (th/config-get-mock config-mock)]
(let [path (th/tempfile "backend_tests/test_files/sample.jpg")
result (media.remote/process (mk-system)
{:cmd :info
:input {:path path :mtype "image/jpeg"}})]
(t/is (= 800 (:width result)))
(t/is (= 600 (:height result)))
(t/is (= (fs/size path) (:size result)))
(t/is (some? (:ts result)))
(t/is (= path (:path result)))
(t/is (= "image/jpeg" (:mtype result)))
(t/is (= 1 (:call-count @mock))))))))
(t/deftest info-verifies-request-params
(t/testing "info sends correct endpoint, method, and x-shared-key header"
(with-mocks [mock {:target 'app.media.remote/service-request
:return {:status 200
:body (json-stream {:width 100 :height 100 :mtype "image/jpeg" :size 1 :orientation 1})}}]
(with-redefs [cf/get (th/config-get-mock config-mock)]
(let [path (th/tempfile "backend_tests/test_files/sample.jpg")]
(media.remote/process (mk-system)
{:cmd :info :input {:path path :mtype "image/jpeg"}})
(let [[system req-map] (:call-args @mock)]
;; System passed through
(t/is (some? (::setup/shared-keys system)))
;; Request structure
(t/is (= :post (:method req-map)))
(t/is (str/includes? (str (:uri req-map)) "api/image/info"))
(t/is (= "test-shared-key" (get-in req-map [:headers "x-shared-key"])))
(t/is (str/starts-with?
(get-in req-map [:headers "Content-Type"])
"multipart/form-data"))))))))
(t/deftest info-no-content-length-header
(t/testing "info does not send Content-Length header (JDK uses chunked encoding)"
(with-mocks [mock {:target 'app.media.remote/service-request
:return {:status 200
:body (json-stream {:width 100 :height 100 :mtype "image/jpeg" :size 1 :orientation 1})}}]
(with-redefs [cf/get (th/config-get-mock config-mock)]
(let [path (th/tempfile "backend_tests/test_files/sample.jpg")]
(media.remote/process (mk-system)
{:cmd :info :input {:path path :mtype "image/jpeg"}})
(let [[_ req-map] (:call-args @mock)]
(t/is (nil? (get-in req-map [:headers "Content-Length"])))))))))
(t/deftest info-service-uri-not-configured
(t/testing "info throws when service URI is not configured"
(with-redefs [cf/get (th/config-get-mock {})]
(let [path (th/tempfile "backend_tests/test_files/sample.jpg")
err (ex/try! (media.remote/process (mk-system)
{:cmd :info :input {:path path :mtype "image/jpeg"}}))]
(t/is (ex/error? err))
(t/is (= :internal (:type (ex-data err))))
(t/is (= :media-processor-not-configured (:code (ex-data err))))))))
(t/deftest info-service-unavailable
(t/testing "info throws when service-request raises unavailable"
(with-mocks [mock {:target 'app.media.remote/service-request
:throw (ex-info "Cannot connect to media-processor service"
{:type :internal
:code :media-processor-unavailable})}]
(with-redefs [cf/get (th/config-get-mock config-mock)]
(let [path (th/tempfile "backend_tests/test_files/sample.jpg")
err (ex/try! (media.remote/process (mk-system)
{:cmd :info :input {:path path :mtype "image/jpeg"}}))]
(t/is (ex/error? err))
(t/is (= :internal (:type (ex-data err))))
(t/is (= :media-processor-unavailable (:code (ex-data err)))))))))
(t/deftest info-service-timeout
(t/testing "info throws when service-request raises timeout"
(with-mocks [mock {:target 'app.media.remote/service-request
:throw (ex-info "media-processor service request timed out"
{:type :internal
:code :media-processor-timeout})}]
(with-redefs [cf/get (th/config-get-mock config-mock)]
(let [path (th/tempfile "backend_tests/test_files/sample.jpg")
err (ex/try! (media.remote/process (mk-system)
{:cmd :info :input {:path path :mtype "image/jpeg"}}))]
(t/is (ex/error? err))
(t/is (= :internal (:type (ex-data err))))
(t/is (= :media-processor-timeout (:code (ex-data err)))))))))
(t/deftest info-mtype-mismatch
(t/testing "info raises :media-type-mismatch when detected mtype differs from declared"
(with-mocks [mock {:target 'app.media.remote/service-request
:return {:status 200
:body (json-stream {:width 100 :height 100 :size 100
:mtype "image/png"})}}]
(with-redefs [cf/get (th/config-get-mock config-mock)]
(let [path (th/tempfile "backend_tests/test_files/sample.jpg")
err (ex/try! (media.remote/process (mk-system)
{:cmd :info
:input {:path path :mtype "image/jpeg"}}))]
(t/is (ex/error? err))
(t/is (= :validation (:type (ex-data err))))
(t/is (= :media-type-mismatch (:code (ex-data err)))))))))
;; ---------------------------------------------------------------------------
;; :generic-thumbnail
;; ---------------------------------------------------------------------------
(t/deftest generic-thumbnail-happy-path
(t/testing "generic-thumbnail returns tempfile with correct format"
(let [thumb-bytes (.getBytes "fake-jpeg-data" "UTF-8")]
(with-mocks [mock {:target 'app.media.remote/service-request
:return {:status 200
:body (ByteArrayInputStream. thumb-bytes)}}]
(with-redefs [cf/get (th/config-get-mock config-mock)]
(let [path (th/tempfile "backend_tests/test_files/sample.jpg")
result (media.remote/process (mk-system)
{:cmd :generic-thumbnail
:input {:path path :mtype "image/jpeg"}
:format :jpeg
:quality 80
:width 200
:height 200})]
(t/is (= :jpeg (:format result)))
(t/is (= "image/jpeg" (:mtype result)))
(t/is (pos? (:size result)))
(t/is (fs/exists? (:data result)))))))))
(t/deftest generic-thumbnail-verifies-query-params
(t/testing "generic-thumbnail sends correct query params with mode=fit"
(with-mocks [mock {:target 'app.media.remote/service-request
:return {:status 200
:body (ByteArrayInputStream. (.getBytes "data" "UTF-8"))}}]
(with-redefs [cf/get (th/config-get-mock config-mock)]
(let [path (th/tempfile "backend_tests/test_files/sample.jpg")]
(media.remote/process (mk-system)
{:cmd :generic-thumbnail
:input {:path path :mtype "image/jpeg"}
:format :jpeg
:quality 85
:width 300
:height 400})
(let [[_ req-map] (:call-args @mock)]
(t/is (str/includes? (str (:uri req-map)) "width=300"))
(t/is (str/includes? (str (:uri req-map)) "height=400"))
(t/is (str/includes? (str (:uri req-map)) "quality=85"))
(t/is (str/includes? (str (:uri req-map)) "format=jpeg"))
(t/is (str/includes? (str (:uri req-map)) "mode=fit"))))))))
(t/deftest generic-thumbnail-service-unavailable
(t/testing "generic-thumbnail throws on service error"
(with-mocks [mock {:target 'app.media.remote/service-request
:throw (ex-info "Cannot connect to media-processor service"
{:type :internal
:code :media-processor-unavailable})}]
(with-redefs [cf/get (th/config-get-mock config-mock)]
(let [path (th/tempfile "backend_tests/test_files/sample.jpg")
err (ex/try! (media.remote/process (mk-system)
{:cmd :generic-thumbnail
:input {:path path :mtype "image/jpeg"}
:format :jpeg
:quality 85
:width 200
:height 200}))]
(t/is (ex/error? err))
(t/is (= :media-processor-unavailable (:code (ex-data err)))))))))
;; ---------------------------------------------------------------------------
;; :profile-thumbnail
;; ---------------------------------------------------------------------------
(t/deftest profile-thumbnail-happy-path
(t/testing "profile-thumbnail returns tempfile and uses mode=crop"
(let [thumb-bytes (.getBytes "fake-png-data" "UTF-8")]
(with-mocks [mock {:target 'app.media.remote/service-request
:return {:status 200
:body (ByteArrayInputStream. thumb-bytes)}}]
(with-redefs [cf/get (th/config-get-mock config-mock)]
(let [path (th/tempfile "backend_tests/test_files/sample.jpg")
result (media.remote/process (mk-system)
{:cmd :profile-thumbnail
:input {:path path :mtype "image/jpeg"}
:format :jpeg
:quality 85
:width 128
:height 128})]
(t/is (some? (:data result)))
(t/is (fs/exists? (:data result)))
;; Verify mode=crop in URI
(let [[_ req-map] (:call-args @mock)]
(t/is (str/includes? (str (:uri req-map)) "mode=crop")))))))))
(t/deftest profile-thumbnail-service-unavailable
(t/testing "profile-thumbnail throws on service error"
(with-mocks [mock {:target 'app.media.remote/service-request
:throw (ex-info "Cannot connect to media-processor service"
{:type :internal
:code :media-processor-unavailable})}]
(with-redefs [cf/get (th/config-get-mock config-mock)]
(let [path (th/tempfile "backend_tests/test_files/sample.jpg")
err (ex/try! (media.remote/process (mk-system)
{:cmd :profile-thumbnail
:input {:path path :mtype "image/jpeg"}
:format :jpeg
:quality 85
:width 128
:height 128}))]
(t/is (ex/error? err))
(t/is (= :media-processor-unavailable (:code (ex-data err)))))))))
;; ---------------------------------------------------------------------------
;; :generate-fonts
;; ---------------------------------------------------------------------------
(t/deftest generate-fonts-ttf-happy-path
(t/testing "generate-fonts with TTF path makes per-variant calls"
(let [ttfbytes (io/read* (io/resource "backend_tests/test_files/font-1.ttf"))
ttfpath (write-font-tmp ttfbytes ".ttf")
fake-bytes (.getBytes "fake-font-data" "UTF-8")]
(with-mocks [mock {:target 'app.media.remote/service-request
:return {:status 200
:body (ByteArrayInputStream. fake-bytes)}}]
(with-redefs [cf/get (th/config-get-mock config-mock)]
(try
(let [result (media.remote/process (mk-system)
{:cmd :generate-fonts
:input {"font/ttf" ttfpath}})]
;; Original path preserved
(t/is (= ttfpath (get result "font/ttf")))
;; Variants written to tempfiles
(t/is (fs/exists? (get result "font/otf")))
(t/is (fs/exists? (get result "font/woff")))
;; Two calls: one for otf, one for woff
(t/is (= 2 (:call-count @mock))))
(finally
(fs/delete ttfpath))))))))
(t/deftest generate-fonts-ttf-as-path
(t/testing "generate-fonts with TTF as tempfile Path works"
(let [ttfbytes (io/read* (io/resource "backend_tests/test_files/font-1.ttf"))
tmp-path (write-font-tmp ttfbytes ".ttf")
fake-bytes (.getBytes "fake-font-data" "UTF-8")]
(with-mocks [mock {:target 'app.media.remote/service-request
:return {:status 200
:body (ByteArrayInputStream. fake-bytes)}}]
(with-redefs [cf/get (th/config-get-mock config-mock)]
(try
(let [result (media.remote/process (mk-system)
{:cmd :generate-fonts
:input {"font/ttf" tmp-path}})]
;; Path preserved
(t/is (= tmp-path (get result "font/ttf")))
;; Variant written
(t/is (fs/exists? (get result "font/otf"))))
(finally
(fs/delete tmp-path))))))))
(t/deftest generate-fonts-otf-happy-path
(t/testing "generate-fonts with OTF path"
(let [otfbytes (io/read* (io/resource "backend_tests/test_files/font-1.otf"))
otfpath (write-font-tmp otfbytes ".otf")
fake-bytes (.getBytes "fake-font-data" "UTF-8")]
(with-mocks [mock {:target 'app.media.remote/service-request
:return {:status 200
:body (ByteArrayInputStream. fake-bytes)}}]
(with-redefs [cf/get (th/config-get-mock config-mock)]
(try
(let [result (media.remote/process (mk-system)
{:cmd :generate-fonts
:input {"font/otf" otfpath}})]
(t/is (= otfpath (get result "font/otf")))
(t/is (fs/exists? (get result "font/ttf")))
(t/is (fs/exists? (get result "font/woff")))
;; Two calls: one for ttf, one for woff
(t/is (= 2 (:call-count @mock))))
(finally
(fs/delete otfpath))))))))
(t/deftest generate-fonts-woff-happy-path
(t/testing "generate-fonts with WOFF path"
(let [woffbytes (io/read* (io/resource "backend_tests/test_files/font-1.woff"))
woffpath (write-font-tmp woffbytes ".woff")
fake-bytes (.getBytes "fake-font-data" "UTF-8")]
(with-mocks [mock {:target 'app.media.remote/service-request
:return {:status 200
:body (ByteArrayInputStream. fake-bytes)}}]
(with-redefs [cf/get (th/config-get-mock config-mock)]
(try
(let [result (media.remote/process (mk-system)
{:cmd :generate-fonts
:input {"font/woff" woffpath}})]
(t/is (= woffpath (get result "font/woff")))
(t/is (fs/exists? (get result "font/ttf")))
(t/is (fs/exists? (get result "font/otf")))
;; Two calls: one for ttf, one for otf
(t/is (= 2 (:call-count @mock))))
(finally
(fs/delete woffpath)))))))
(t/deftest generate-fonts-woff2-happy-path
(t/testing "generate-fonts with WOFF2 path"
(let [woff2bytes (io/read* (io/resource "backend_tests/test_files/font-1.woff2"))
woff2path (write-font-tmp woff2bytes ".woff2")
fake-bytes (.getBytes "fake-font-data" "UTF-8")]
(with-mocks [mock {:target 'app.media.remote/service-request
:return {:status 200
:body (ByteArrayInputStream. fake-bytes)}}]
(with-redefs [cf/get (th/config-get-mock config-mock)]
(try
(let [result (media.remote/process (mk-system)
{:cmd :generate-fonts
:input {"font/woff2" woff2path}})]
(t/is (= woff2path (get result "font/woff2")))
(t/is (fs/exists? (get result "font/ttf")))
(t/is (fs/exists? (get result "font/otf")))
(t/is (fs/exists? (get result "font/woff")))
;; Three calls: one for ttf, one for otf, one for woff
(t/is (= 3 (:call-count @mock))))
(finally
(fs/delete woff2path)))))))))
(t/deftest generate-fonts-verifies-query-params
(t/testing "generate-fonts sends target-type query param with 180s timeout"
(let [ttfbytes (io/read* (io/resource "backend_tests/test_files/font-1.ttf"))
ttfpath (write-font-tmp ttfbytes ".ttf")
fake-bytes (.getBytes "fake-font-data" "UTF-8")]
(with-mocks [mock {:target 'app.media.remote/service-request
:return {:status 200
:body (ByteArrayInputStream. fake-bytes)}}]
(with-redefs [cf/get (th/config-get-mock config-mock)]
(try
(media.remote/process (mk-system)
{:cmd :generate-fonts :input {"font/ttf" ttfpath}})
(let [[_ req-map] (:call-args @mock)]
(t/is (str/includes? (str (:uri req-map)) "target-type="))
(t/is (= 180000 (:timeout req-map))))
(finally
(fs/delete ttfpath))))))))
(t/deftest generate-fonts-woff-verifies-target-types
(t/testing "generate-fonts with WOFF sends target-type query param"
(let [woffbytes (io/read* (io/resource "backend_tests/test_files/font-1.woff"))
woffpath (write-font-tmp woffbytes ".woff")
fake-bytes (.getBytes "fake-font-data" "UTF-8")]
(with-mocks [mock {:target 'app.media.remote/service-request
:return {:status 200
:body (ByteArrayInputStream. fake-bytes)}}]
(with-redefs [cf/get (th/config-get-mock config-mock)]
(try
(media.remote/process (mk-system)
{:cmd :generate-fonts :input {"font/woff" woffpath}})
(let [[_ req-map] (:call-args @mock)]
(t/is (str/includes? (str (:uri req-map)) "target-type=")))
(finally
(fs/delete woffpath))))))))
(t/deftest generate-fonts-no-recognized-variant
(t/testing "generate-fonts throws when no recognized font variant"
(with-redefs [cf/get (th/config-get-mock config-mock)]
(let [err (ex/try! (media.remote/process (mk-system)
{:cmd :generate-fonts
:input {"font/unknown" (.getBytes "data" "UTF-8")}}))]
(t/is (ex/error? err))
(t/is (= :validation (:type (ex-data err))))
(t/is (= :invalid-font (:code (ex-data err))))))))
(t/deftest generate-fonts-connection-error
(t/testing "generate-fonts throws on service error"
(let [ttfbytes (io/read* (io/resource "backend_tests/test_files/font-1.ttf"))
ttfpath (write-font-tmp ttfbytes ".ttf")]
(with-mocks [mock {:target 'app.media.remote/service-request
:throw (ex-info "Cannot connect to media-processor service"
{:type :internal
:code :media-processor-unavailable})}]
(with-redefs [cf/get (th/config-get-mock config-mock)]
(try
(let [err (ex/try! (media.remote/process (mk-system)
{:cmd :generate-fonts :input {"font/ttf" ttfpath}}))]
(t/is (ex/error? err))
(t/is (= :media-processor-unavailable (:code (ex-data err)))))
(finally
(fs/delete ttfpath))))))))
(t/deftest generate-fonts-timeout-error
(t/testing "generate-fonts throws on service timeout"
(let [ttfbytes (io/read* (io/resource "backend_tests/test_files/font-1.ttf"))
ttfpath (write-font-tmp ttfbytes ".ttf")]
(with-mocks [mock {:target 'app.media.remote/service-request
:throw (ex-info "media-processor service request timed out"
{:type :internal
:code :media-processor-timeout})}]
(with-redefs [cf/get (th/config-get-mock config-mock)]
(try
(let [err (ex/try! (media.remote/process (mk-system)
{:cmd :generate-fonts :input {"font/ttf" ttfpath}}))]
(t/is (ex/error? err))
(t/is (= :media-processor-timeout (:code (ex-data err)))))
(finally
(fs/delete ttfpath))))))))
;; ---------------------------------------------------------------------------
;; Status code handling (service-request)
;; ---------------------------------------------------------------------------
(t/deftest service-request-raises-on-400
(t/testing "service-request raises :validation on status 400"
(with-mocks [mock {:target 'app.http.client/req
:return {:status 400
:body (json-stream {:type "validation"
:code "invalid-image"
:hint "bad input"})}}]
(with-redefs [cf/get (th/config-get-mock config-mock)]
(let [err (ex/try! (media.remote/service-request
(mk-system)
{:method :post
:uri "http://localhost:6065/api/image/info"
:body nil
:headers {}}))]
(t/is (ex/error? err))
(t/is (= :validation (:type (ex-data err))))
(t/is (= :invalid-image (:code (ex-data err)))))))))
(t/deftest service-request-raises-on-500
(t/testing "service-request raises :internal on status 500"
(with-mocks [mock {:target 'app.http.client/req
:return {:status 500
:body (json-stream {:type "internal"
:code "processing-error"
:hint "Internal server error"})}}]
(with-redefs [cf/get (th/config-get-mock config-mock)]
(let [err (ex/try! (media.remote/service-request
(mk-system)
{:method :post
:uri "http://localhost:6065/api/image/info"
:body nil
:headers {}}))]
(t/is (ex/error? err))
(t/is (= :internal (:type (ex-data err))))
(t/is (= :processing-error (:code (ex-data err)))))))))
(t/deftest service-request-passes-on-200
(t/testing "service-request returns response on status 200"
(with-mocks [mock {:target 'app.http.client/req
:return {:status 200
:body (json-stream {:width 100 :height 100})}}]
(with-redefs [cf/get (th/config-get-mock config-mock)]
(let [resp (media.remote/service-request
(mk-system)
{:method :post
:uri "http://localhost:6065/api/image/info"
:body nil
:headers {}})]
(t/is (= 200 (:status resp))))))))
;; ---------------------------------------------------------------------------
;; Shared key
;; ---------------------------------------------------------------------------
(t/deftest shared-key-sent-correctly
(t/testing "x-shared-key header matches the system's shared key"
(with-mocks [mock {:target 'app.media.remote/service-request
:return {:status 200
:body (json-stream {:width 1 :height 1 :mtype "image/jpeg" :size 1 :orientation 1})}}]
(with-redefs [cf/get (th/config-get-mock config-mock)]
(let [system {::setup/shared-keys {:media-processor "my-secret-key-123"}}]
(media.remote/process system
{:cmd :info
:input {:path (th/tempfile "backend_tests/test_files/sample.jpg")
:mtype "image/jpeg"}})
(let [[system-arg _] (:call-args @mock)]
;; System passed through correctly
(t/is (= "my-secret-key-123"
(-> system-arg ::setup/shared-keys :media-processor)))))))))
;; ---------------------------------------------------------------------------
;; Stream closure
;; ---------------------------------------------------------------------------
(defn- tracking-stream
"Create an InputStream that tracks whether it was closed.
Returns a map with :stream (the InputStream) and :closed (an atom)."
[^bytes data]
(let [closed (atom false)
delegate (ByteArrayInputStream. data)
stream (proxy [java.io.InputStream] []
(read
([] (.read delegate))
([^bytes b] (.read delegate b))
([^bytes b off len] (.read delegate b off len)))
(close []
(reset! closed true)
(.close delegate)))]
{:stream stream :closed closed}))
(t/deftest info-closes-response-stream
(t/testing "info closes the response stream after parsing JSON"
(let [json-str "{\"width\":100,\"height\":100,\"mtype\":\"image/jpeg\",\"size\":1,\"orientation\":1}"
json-data (.getBytes json-str "UTF-8")
{:keys [stream closed]} (tracking-stream json-data)]
(with-mocks [mock {:target 'app.media.remote/service-request
:return {:status 200
:body stream}}]
(with-redefs [cf/get (th/config-get-mock config-mock)]
(let [path (th/tempfile "backend_tests/test_files/sample.jpg")]
(media.remote/process (mk-system)
{:cmd :info
:input {:path path :mtype "image/jpeg"}})
;; Stream should be closed after processing
(t/is @closed)))))))
(t/deftest font-convert-closes-response-stream
(t/testing "font-convert closes the response stream after writing"
(let [{:keys [stream closed]} (tracking-stream (.getBytes "fake-font-data" "UTF-8"))]
(with-mocks [mock {:target 'app.media.remote/service-request
:return {:status 200
:body stream}}]
(with-redefs [cf/get (th/config-get-mock config-mock)]
(let [ttfbytes (io/read* (io/resource "backend_tests/test_files/font-1.ttf"))
ttfpath (write-font-tmp ttfbytes ".ttf")]
(try
(media.remote/process (mk-system)
{:cmd :generate-fonts
:input {"font/ttf" ttfpath}})
;; Stream should be closed after processing
(t/is @closed)
(finally
(fs/delete ttfpath)))))))))
(t/deftest thumbnail-closes-response-stream
(t/testing "thumbnail closes the response stream after writing"
(let [{:keys [stream closed]} (tracking-stream (.getBytes "fake-thumbnail-data" "UTF-8"))]
(with-mocks [mock {:target 'app.media.remote/service-request
:return {:status 200
:body stream}}]
(with-redefs [cf/get (th/config-get-mock config-mock)]
(let [path (th/tempfile "backend_tests/test_files/sample.jpg")]
(media.remote/process (mk-system)
{:cmd :generic-thumbnail
:input {:path path :mtype "image/jpeg"}
:format :jpeg
:quality 85
:width 200
:height 200})
;; Stream should be closed after processing
(t/is @closed)))))))

738
backend/test/backend_tests/rpc_font_test.clj
View File

@ -24,312 +24,6 @@
(t/use-fixtures :once th/state-init)
(t/use-fixtures :each th/database-reset)
(t/deftest ttf-font-upload-1
(with-mocks [mock {:target 'app.rpc.quotes/check! :return nil}]
(let [prof (th/create-profile* 1 {:is-active true})
team-id (:default-team-id prof)
proj-id (:default-project-id prof)
font-id (uuid/custom 10 1)
ttfdata (-> (io/resource "backend_tests/test_files/font-1.ttf")
(io/read*))
params {::th/type :create-font-variant
::rpc/profile-id (:id prof)
:team-id team-id
:font-id font-id
:font-family "somefont"
:font-weight 400
:font-style "normal"
:data {"font/ttf" ttfdata}}
out (th/command! params)]
(t/is (= 1 (:call-count @mock)))
;; (th/print-result! out)
(t/is (nil? (:error out)))
(let [result (:result out)]
(t/is (uuid? (:id result)))
(t/is (uuid? (:ttf-file-id result)))
(t/is (uuid? (:otf-file-id result)))
(t/is (uuid? (:woff1-file-id result)))
(t/are [k] (= (get params k)
(get result k))
:team-id
:font-id
:font-family
:font-weight
:font-style)))))
(t/deftest ttf-font-upload-2
(let [prof (th/create-profile* 1 {:is-active true})
team-id (:default-team-id prof)
proj-id (:default-project-id prof)
font-id (uuid/custom 10 1)
data (-> (io/resource "backend_tests/test_files/font-1.woff")
(io/read*))
params {::th/type :create-font-variant
::rpc/profile-id (:id prof)
:team-id team-id
:font-id font-id
:font-family "somefont"
:font-weight 400
:font-style "normal"
:data {"font/woff" data}}
out (th/command! params)]
;; (th/print-result! out)
(t/is (nil? (:error out)))
(let [result (:result out)]
(t/is (uuid? (:id result)))
(t/is (uuid? (:ttf-file-id result)))
(t/is (uuid? (:otf-file-id result)))
(t/is (uuid? (:woff1-file-id result)))
(t/are [k] (= (get params k)
(get result k))
:team-id
:font-id
:font-family
:font-weight
:font-style))))
(t/deftest woff2-font-upload-1
(let [prof (th/create-profile* 1 {:is-active true})
team-id (:default-team-id prof)
proj-id (:default-project-id prof)
font-id (uuid/custom 10 1)
data (-> (io/resource "backend_tests/test_files/font-1.woff2")
(io/read*))
params {::th/type :create-font-variant
::rpc/profile-id (:id prof)
:team-id team-id
:font-id font-id
:font-family "somefont"
:font-weight 400
:font-style "normal"
:data {"font/woff2" data}}
out (th/command! params)]
;; (th/print-result! out)
(t/is (nil? (:error out)))
(let [result (:result out)]
(t/is (uuid? (:id result)))
(t/is (uuid? (:ttf-file-id result)))
(t/is (uuid? (:otf-file-id result)))
(t/is (uuid? (:woff1-file-id result)))
(t/is (uuid? (:woff2-file-id result)))
(t/are [k] (= (get params k)
(get result k))
:team-id
:font-id
:font-family
:font-weight
:font-style))))
(t/deftest font-deletion-1
(let [prof (th/create-profile* 1 {:is-active true})
team-id (:default-team-id prof)
proj-id (:default-project-id prof)
font-id (uuid/custom 10 1)
data1 (-> (io/resource "backend_tests/test_files/font-1.woff")
(io/read*))
data2 (-> (io/resource "backend_tests/test_files/font-2.woff")
(io/read*))]
;; Create front variant
(let [params {::th/type :create-font-variant
::rpc/profile-id (:id prof)
:team-id team-id
:font-id font-id
:font-family "somefont"
:font-weight 400
:font-style "normal"
:data {"font/woff" data1}}
out (th/command! params)]
;; (th/print-result! out)
(t/is (nil? (:error out))))
(let [params {::th/type :create-font-variant
::rpc/profile-id (:id prof)
:team-id team-id
:font-id font-id
:font-family "somefont"
:font-weight 500
:font-style "normal"
:data {"font/woff" data2}}
out (th/command! params)]
;; (th/print-result! out)
(t/is (nil? (:error out))))
(let [res (binding [ct/*clock* (ct/fixed-clock (ct/in-future {:hours 3}))]
(th/run-task! :storage-gc-touched {}))]
(t/is (= 6 (:freeze res))))
(let [params {::th/type :delete-font
::rpc/profile-id (:id prof)
:team-id team-id
:id font-id}
out (th/command! params)]
;; (th/print-result! out)
(t/is (nil? (:error out)))
(t/is (nil? (:result out))))
(let [res (binding [ct/*clock* (ct/fixed-clock (ct/in-future {:hours 3}))]
(th/run-task! :storage-gc-touched {}))]
(t/is (= 0 (:freeze res)))
(t/is (= 0 (:delete res))))
(binding [ct/*clock* (ct/fixed-clock (ct/in-future {:days 8}))]
(let [res (th/run-task! :objects-gc {})]
(t/is (= 2 (:processed res)))))
(binding [ct/*clock* (ct/fixed-clock (ct/in-future {:days 8 :hours 3}))]
(let [res (th/run-task! :storage-gc-touched {})]
(t/is (= 0 (:freeze res)))
(t/is (= 6 (:delete res)))))))
(t/deftest font-deletion-2
(let [prof (th/create-profile* 1 {:is-active true})
team-id (:default-team-id prof)
proj-id (:default-project-id prof)
font-id (uuid/custom 10 1)
data1 (-> (io/resource "backend_tests/test_files/font-1.woff")
(io/read*))
data2 (-> (io/resource "backend_tests/test_files/font-2.woff")
(io/read*))]
;; Create front variant
(let [params {::th/type :create-font-variant
::rpc/profile-id (:id prof)
:team-id team-id
:font-id font-id
:font-family "somefont"
:font-weight 400
:font-style "normal"
:data {"font/woff" data1}}
out (th/command! params)]
;; (th/print-result! out)
(t/is (nil? (:error out))))
(let [params {::th/type :create-font-variant
::rpc/profile-id (:id prof)
:team-id team-id
:font-id (uuid/custom 10 2)
:font-family "somefont"
:font-weight 400
:font-style "normal"
:data {"font/woff" data2}}
out (th/command! params)]
;; (th/print-result! out)
(t/is (nil? (:error out))))
(let [res (binding [ct/*clock* (ct/fixed-clock (ct/in-future {:hours 3}))]
(th/run-task! :storage-gc-touched {}))]
(t/is (= 6 (:freeze res))))
(let [params {::th/type :delete-font
::rpc/profile-id (:id prof)
:team-id team-id
:id font-id}
out (th/command! params)]
;; (th/print-result! out)
(t/is (nil? (:error out)))
(t/is (nil? (:result out))))
(let [res (binding [ct/*clock* (ct/fixed-clock (ct/in-future {:hours 3}))]
(th/run-task! :storage-gc-touched {}))]
(t/is (= 0 (:freeze res)))
(t/is (= 0 (:delete res))))
(binding [ct/*clock* (ct/fixed-clock (ct/in-future {:days 8}))]
(let [res (th/run-task! :objects-gc {})]
(t/is (= 1 (:processed res)))))
(binding [ct/*clock* (ct/fixed-clock (ct/in-future {:days 8 :hours 3}))]
(let [res (th/run-task! :storage-gc-touched {})]
(t/is (= 0 (:freeze res)))
(t/is (= 3 (:delete res)))))))
(t/deftest font-deletion-3
(let [prof (th/create-profile* 1 {:is-active true})
team-id (:default-team-id prof)
proj-id (:default-project-id prof)
font-id (uuid/custom 10 1)
data1 (-> (io/resource "backend_tests/test_files/font-1.woff") (io/read*))
data2 (-> (io/resource "backend_tests/test_files/font-2.woff") (io/read*))
params1 {::th/type :create-font-variant ::rpc/profile-id (:id prof)
:team-id team-id :font-id font-id :font-family "somefont"
:font-weight 400 :font-style "normal" :data {"font/woff" data1}}
params2 {::th/type :create-font-variant ::rpc/profile-id (:id prof)
:team-id team-id :font-id font-id :font-family "somefont"
:font-weight 500 :font-style "normal" :data {"font/woff" data2}}
out1 (th/command! params1)
out2 (th/command! params2)]
(t/is (nil? (:error out1)))
(t/is (nil? (:error out2)))
;; freeze with hours 3 clock
(let [res (binding [ct/*clock* (ct/fixed-clock (ct/in-future {:hours 3}))]
(th/run-task! :storage-gc-touched {}))]
(t/is (= 6 (:freeze res))))
(let [params {::th/type :delete-font-variant ::rpc/profile-id (:id prof)
:team-id team-id :id (-> out1 :result :id)}
out (th/command! params)]
(t/is (nil? (:error out)))
(t/is (nil? (:result out))))
;; no-op with hours 3 clock (nothing touched yet)
(let [res (binding [ct/*clock* (ct/fixed-clock (ct/in-future {:hours 3}))]
(th/run-task! :storage-gc-touched {}))]
(t/is (= 0 (:freeze res)))
(t/is (= 0 (:delete res))))
;; objects-gc at days 8, then storage-gc-touched at days 8 + 3h
(binding [ct/*clock* (ct/fixed-clock (ct/in-future {:days 8}))]
(let [res (th/run-task! :objects-gc {})]
(t/is (= 1 (:processed res)))))
(binding [ct/*clock* (ct/fixed-clock (ct/in-future {:days 8 :hours 3}))]
(let [res (th/run-task! :storage-gc-touched {})]
(t/is (= 0 (:freeze res)))
(t/is (= 3 (:delete res)))))))
(t/deftest input-sanitization-1
(with-mocks [mock {:target 'app.rpc.quotes/check! :return nil}]
(let [prof (th/create-profile* 1 {:is-active true})
team-id (:default-team-id prof)
proj-id (:default-project-id prof)
font-id (uuid/custom 10 1)
ttfdata (-> (io/resource "backend_tests/test_files/font-1.ttf")
(io/read*))
params {::th/type :create-font-variant
::rpc/profile-id (:id prof)
:team-id team-id
:font-id font-id
:font-family "somefont"
:font-weight 400
:font-style "normal"
:data {"font/ttf" "/etc/passwd"}}
out (th/command! params)]
(t/is (= 0 (:call-count @mock)))
;; (th/print-result! out)
(let [error (:error out)
error-data (ex-data error)]
(t/is (th/ex-info? error))))))
;; -----------------------------------------------------------------------
;; Helpers for chunked-upload font tests
;; -----------------------------------------------------------------------
@ -399,119 +93,211 @@
:font-weight
:font-style))
;; -----------------------------------------------------------------------
;; Path 1 – Normal (direct :data bytes)
;; -----------------------------------------------------------------------
(t/deftest font-deletion-1
(let [prof (th/create-profile* 1 {:is-active true})
team-id (:default-team-id prof)
proj-id (:default-project-id prof)
font-id (uuid/custom 10 1)
(t/deftest create-font-variant-normal-ttf
data1 (-> (io/resource "backend_tests/test_files/font-1.woff")
(io/read*))
data2 (-> (io/resource "backend_tests/test_files/font-2.woff")
(io/read*))]
;; Create font variant
(let [session-id (upload-font-chunked! prof data1 "font/woff" (* 4 1024 1024))
params {::th/type :create-font-variant
::rpc/profile-id (:id prof)
:team-id team-id
:font-id font-id
:font-family "somefont"
:font-weight 400
:font-style "normal"
:uploads {"font/woff" session-id}}
out (th/command! params)]
;; (th/print-result! out)
(t/is (nil? (:error out))))
(let [session-id (upload-font-chunked! prof data2 "font/woff" (* 4 1024 1024))
params {::th/type :create-font-variant
::rpc/profile-id (:id prof)
:team-id team-id
:font-id font-id
:font-family "somefont"
:font-weight 500
:font-style "normal"
:uploads {"font/woff" session-id}}
out (th/command! params)]
;; (th/print-result! out)
(t/is (nil? (:error out))))
(let [res (binding [ct/*clock* (ct/fixed-clock (ct/in-future {:hours 3}))]
(th/run-task! :storage-gc-touched {}))]
(t/is (= 6 (:freeze res))))
(let [params {::th/type :delete-font
::rpc/profile-id (:id prof)
:team-id team-id
:id font-id}
out (th/command! params)]
;; (th/print-result! out)
(t/is (nil? (:error out)))
(t/is (nil? (:result out))))
(let [res (binding [ct/*clock* (ct/fixed-clock (ct/in-future {:hours 3}))]
(th/run-task! :storage-gc-touched {}))]
(t/is (= 0 (:freeze res)))
(t/is (= 0 (:delete res))))
(binding [ct/*clock* (ct/fixed-clock (ct/in-future {:days 8}))]
(let [res (th/run-task! :objects-gc {})]
(t/is (= 2 (:processed res)))))
(binding [ct/*clock* (ct/fixed-clock (ct/in-future {:days 8 :hours 3}))]
(let [res (th/run-task! :storage-gc-touched {})]
(t/is (= 0 (:freeze res)))
(t/is (= 6 (:delete res)))))))
(t/deftest font-deletion-2
(let [prof (th/create-profile* 1 {:is-active true})
team-id (:default-team-id prof)
proj-id (:default-project-id prof)
font-id (uuid/custom 10 1)
data1 (-> (io/resource "backend_tests/test_files/font-1.woff")
(io/read*))
data2 (-> (io/resource "backend_tests/test_files/font-2.woff")
(io/read*))]
;; Create font variant
(let [session-id (upload-font-chunked! prof data1 "font/woff" (* 4 1024 1024))
params {::th/type :create-font-variant
::rpc/profile-id (:id prof)
:team-id team-id
:font-id font-id
:font-family "somefont"
:font-weight 400
:font-style "normal"
:uploads {"font/woff" session-id}}
out (th/command! params)]
;; (th/print-result! out)
(t/is (nil? (:error out))))
(let [session-id (upload-font-chunked! prof data2 "font/woff" (* 4 1024 1024))
params {::th/type :create-font-variant
::rpc/profile-id (:id prof)
:team-id team-id
:font-id (uuid/custom 10 2)
:font-family "somefont"
:font-weight 400
:font-style "normal"
:uploads {"font/woff" session-id}}
out (th/command! params)]
;; (th/print-result! out)
(t/is (nil? (:error out))))
(let [res (binding [ct/*clock* (ct/fixed-clock (ct/in-future {:hours 3}))]
(th/run-task! :storage-gc-touched {}))]
(t/is (= 6 (:freeze res))))
(let [params {::th/type :delete-font
::rpc/profile-id (:id prof)
:team-id team-id
:id font-id}
out (th/command! params)]
;; (th/print-result! out)
(t/is (nil? (:error out)))
(t/is (nil? (:result out))))
(let [res (binding [ct/*clock* (ct/fixed-clock (ct/in-future {:hours 3}))]
(th/run-task! :storage-gc-touched {}))]
(t/is (= 0 (:freeze res)))
(t/is (= 0 (:delete res))))
(binding [ct/*clock* (ct/fixed-clock (ct/in-future {:days 8}))]
(let [res (th/run-task! :objects-gc {})]
(t/is (= 1 (:processed res)))))
(binding [ct/*clock* (ct/fixed-clock (ct/in-future {:days 8 :hours 3}))]
(let [res (th/run-task! :storage-gc-touched {})]
(t/is (= 0 (:freeze res)))
(t/is (= 3 (:delete res)))))))
(t/deftest font-deletion-3
(let [prof (th/create-profile* 1 {:is-active true})
team-id (:default-team-id prof)
proj-id (:default-project-id prof)
font-id (uuid/custom 10 1)
data1 (-> (io/resource "backend_tests/test_files/font-1.woff") (io/read*))
data2 (-> (io/resource "backend_tests/test_files/font-2.woff") (io/read*))
sid1 (upload-font-chunked! prof data1 "font/woff" (* 4 1024 1024))
sid2 (upload-font-chunked! prof data2 "font/woff" (* 4 1024 1024))
params1 {::th/type :create-font-variant ::rpc/profile-id (:id prof)
:team-id team-id :font-id font-id :font-family "somefont"
:font-weight 400 :font-style "normal" :uploads {"font/woff" sid1}}
params2 {::th/type :create-font-variant ::rpc/profile-id (:id prof)
:team-id team-id :font-id font-id :font-family "somefont"
:font-weight 500 :font-style "normal" :uploads {"font/woff" sid2}}
out1 (th/command! params1)
out2 (th/command! params2)]
(t/is (nil? (:error out1)))
(t/is (nil? (:error out2)))
;; freeze with hours 3 clock
(let [res (binding [ct/*clock* (ct/fixed-clock (ct/in-future {:hours 3}))]
(th/run-task! :storage-gc-touched {}))]
(t/is (= 6 (:freeze res))))
(let [params {::th/type :delete-font-variant ::rpc/profile-id (:id prof)
:team-id team-id :id (-> out1 :result :id)}
out (th/command! params)]
(t/is (nil? (:error out)))
(t/is (nil? (:result out))))
;; no-op with hours 3 clock (nothing touched yet)
(let [res (binding [ct/*clock* (ct/fixed-clock (ct/in-future {:hours 3}))]
(th/run-task! :storage-gc-touched {}))]
(t/is (= 0 (:freeze res)))
(t/is (= 0 (:delete res))))
;; objects-gc at days 8, then storage-gc-touched at days 8 + 3h
(binding [ct/*clock* (ct/fixed-clock (ct/in-future {:days 8}))]
(let [res (th/run-task! :objects-gc {})]
(t/is (= 1 (:processed res)))))
(binding [ct/*clock* (ct/fixed-clock (ct/in-future {:days 8 :hours 3}))]
(let [res (th/run-task! :storage-gc-touched {})]
(t/is (= 0 (:freeze res)))
(t/is (= 3 (:delete res)))))))
(t/deftest input-sanitization-1
(with-mocks [mock {:target 'app.rpc.quotes/check! :return nil}]
(let [prof (th/create-profile* 1 {:is-active true})
team-id (:default-team-id prof)
font-id (uuid/custom 10 10)
data (-> (io/resource "backend_tests/test_files/font-1.ttf") (io/read*))
params {::th/type :create-font-variant
::rpc/profile-id (:id prof)
:team-id team-id
:font-id font-id
:font-family "chunked-test"
:font-weight 400
:font-style "normal"
:data {"font/ttf" data}}
out (th/command! params)]
(t/is (= 1 (:call-count @mock)))
(t/is (nil? (:error out)))
(assert-font-variant-result params (:result out)))))
proj-id (:default-project-id prof)
font-id (uuid/custom 10 1)
(t/deftest create-font-variant-normal-otf
(with-mocks [mock {:target 'app.rpc.quotes/check! :return nil}]
(let [prof (th/create-profile* 1 {:is-active true})
team-id (:default-team-id prof)
font-id (uuid/custom 10 11)
data (-> (io/resource "backend_tests/test_files/font-1.otf") (io/read*))
params {::th/type :create-font-variant
::rpc/profile-id (:id prof)
:team-id team-id
:font-id font-id
:font-family "chunked-test"
:font-weight 400
:font-style "normal"
:data {"font/otf" data}}
out (th/command! params)]
(t/is (= 1 (:call-count @mock)))
(t/is (nil? (:error out)))
(assert-font-variant-result params (:result out)))))
ttfdata (-> (io/resource "backend_tests/test_files/font-1.ttf")
(io/read*))
(t/deftest create-font-variant-normal-woff
(with-mocks [mock {:target 'app.rpc.quotes/check! :return nil}]
(let [prof (th/create-profile* 1 {:is-active true})
team-id (:default-team-id prof)
font-id (uuid/custom 10 12)
data (-> (io/resource "backend_tests/test_files/font-1.woff") (io/read*))
params {::th/type :create-font-variant
session-id (upload-font-chunked! prof ttfdata "font/ttf" (* 4 1024 1024))
params {::th/type :create-font-variant
::rpc/profile-id (:id prof)
:team-id team-id
:font-id font-id
:font-family "chunked-test"
:team-id team-id
:font-id font-id
:font-family "somefont"
:font-weight 400
:font-style "normal"
:data {"font/woff" data}}
:font-style "normal"
:uploads {"font/ttf" session-id}}
out (th/command! params)]
(t/is (= 1 (:call-count @mock)))
(t/is (nil? (:error out)))
(assert-font-variant-result params (:result out)))))
;; (th/print-result! out)
(t/is (nil? (:error out))))))
;; -----------------------------------------------------------------------
;; Path 2 – Legacy chunking (:data with vector of byte-arrays per mtype)
;; -----------------------------------------------------------------------
(t/deftest create-font-variant-legacy-chunked-ttf
"Upload a TTF via the legacy :data path where each mtype value is a
vector of byte-array chunks (4 MiB each) instead of a single byte-array."
(with-mocks [mock {:target 'app.rpc.quotes/check! :return nil}]
(let [prof (th/create-profile* 1 {:is-active true})
team-id (:default-team-id prof)
font-id (uuid/custom 10 20)
full-bytes (-> (io/resource "backend_tests/test_files/font-1.ttf") (io/read*))
;; Simulate 4 MiB legacy chunks – font is small so a single chunk suffices
chunks (split-bytes-into-chunks full-bytes (* 4 1024 1024))
params {::th/type :create-font-variant
::rpc/profile-id (:id prof)
:team-id team-id
:font-id font-id
:font-family "legacy-chunked"
:font-weight 700
:font-style "italic"
:data {"font/ttf" (vec chunks)}}
out (th/command! params)]
(t/is (= 1 (:call-count @mock)))
(t/is (nil? (:error out)))
(assert-font-variant-result params (:result out)))))
(t/deftest create-font-variant-legacy-chunked-woff
"Upload a WOFF via the legacy :data path with multiple sub-4 KiB chunks
to exercise the SequenceInputStream concatenation path."
(with-mocks [mock {:target 'app.rpc.quotes/check! :return nil}]
(let [prof (th/create-profile* 1 {:is-active true})
team-id (:default-team-id prof)
font-id (uuid/custom 10 21)
full-bytes (-> (io/resource "backend_tests/test_files/font-1.woff") (io/read*))
;; Split into small chunks to exercise the SequenceInputStream path
chunks (split-bytes-into-chunks full-bytes 512)
params {::th/type :create-font-variant
::rpc/profile-id (:id prof)
:team-id team-id
:font-id font-id
:font-family "legacy-chunked-woff"
:font-weight 400
:font-style "normal"
:data {"font/woff" (vec chunks)}}
out (th/command! params)]
(t/is (= 1 (:call-count @mock)))
(t/is (nil? (:error out)))
(assert-font-variant-result params (:result out)))))
;; -----------------------------------------------------------------------
;; Path 3 – New standardized chunked upload (:uploads map)
;; Chunked upload (:uploads map)
;; -----------------------------------------------------------------------
(t/deftest create-font-variant-chunked-upload-ttf
@ -606,8 +392,8 @@
;; Error cases
;; -----------------------------------------------------------------------
(t/deftest create-font-variant-missing-data-and-uploads
"Neither :data nor :uploads is present — schema validation must reject it."
(t/deftest create-font-variant-missing-uploads
"Missing :uploads — schema validation must reject it."
(let [prof (th/create-profile* 1 {:is-active true})
team-id (:default-team-id prof)
font-id (uuid/custom 10 40)
@ -674,49 +460,6 @@
;; Font size validation tests
;; -----------------------------------------------------------------------
(t/deftest create-font-variant-size-exceeded-normal
"Direct :data upload exceeding font-max-file-size must be rejected."
(with-mocks [_mock {:target 'app.rpc.quotes/check! :return nil}]
(with-redefs [app.config/config (assoc app.config/config :font-max-file-size 1)]
(let [prof (th/create-profile* 1 {:is-active true})
team-id (:default-team-id prof)
font-id (uuid/custom 10 50)
data (-> (io/resource "backend_tests/test_files/font-1.ttf") (io/read*))
params {::th/type :create-font-variant
::rpc/profile-id (:id prof)
:team-id team-id
:font-id font-id
:font-family "size-exceeded"
:font-weight 400
:font-style "normal"
:data {"font/ttf" data}}
out (th/command! params)]
(t/is (some? (:error out)))
(t/is (= :restriction (-> out :error ex-data :type)))
(t/is (= :font-max-file-size-reached (-> out :error ex-data :code)))))))
(t/deftest create-font-variant-size-exceeded-legacy-chunked
"Legacy :data chunk-vector upload exceeding font-max-file-size must be rejected."
(with-mocks [_mock {:target 'app.rpc.quotes/check! :return nil}]
(with-redefs [app.config/config (assoc app.config/config :font-max-file-size 1)]
(let [prof (th/create-profile* 1 {:is-active true})
team-id (:default-team-id prof)
font-id (uuid/custom 10 51)
full-bytes (-> (io/resource "backend_tests/test_files/font-1.woff") (io/read*))
chunks (split-bytes-into-chunks full-bytes (* 4 1024 1024))
params {::th/type :create-font-variant
::rpc/profile-id (:id prof)
:team-id team-id
:font-id font-id
:font-family "size-exceeded-legacy"
:font-weight 400
:font-style "normal"
:data {"font/woff" (vec chunks)}}
out (th/command! params)]
(t/is (some? (:error out)))
(t/is (= :restriction (-> out :error ex-data :type)))
(t/is (= :font-max-file-size-reached (-> out :error ex-data :code)))))))
(t/deftest create-font-variant-size-exceeded-chunked-upload
"New :uploads path exceeding font-max-file-size must be rejected after assembly."
(with-mocks [_mock {:target 'app.rpc.quotes/check! :return nil}]
@ -738,72 +481,10 @@
(t/is (= :restriction (-> out :error ex-data :type)))
(t/is (= :font-max-file-size-reached (-> out :error ex-data :code))))))))
(t/deftest create-font-variant-size-within-limit
"Upload exactly at the limit must succeed."
(with-mocks [_mock {:target 'app.rpc.quotes/check! :return nil}]
(let [prof (th/create-profile* 1 {:is-active true})
team-id (:default-team-id prof)
font-id (uuid/custom 10 53)
font-bytes (-> (io/resource "backend_tests/test_files/font-1.ttf") (io/read*))
font-size (alength ^bytes font-bytes)]
(with-redefs [app.config/config (assoc app.config/config :font-max-file-size font-size)]
(let [params {::th/type :create-font-variant
::rpc/profile-id (:id prof)
:team-id team-id
:font-id font-id
:font-family "size-at-limit"
:font-weight 400
:font-style "normal"
:data {"font/ttf" font-bytes}}
out (th/command! params)]
(t/is (nil? (:error out)))
(assert-font-variant-result params (:result out)))))))
;; -----------------------------------------------------------------------
;; Font media-type validation tests
;; Font media-type validation
;; -----------------------------------------------------------------------
(t/deftest create-font-variant-invalid-type-normal
"Direct :data upload with a disallowed mtype must be rejected."
(with-mocks [_mock {:target 'app.rpc.quotes/check! :return nil}]
(let [prof (th/create-profile* 1 {:is-active true})
team-id (:default-team-id prof)
font-id (uuid/custom 10 60)
data (-> (io/resource "backend_tests/test_files/font-1.ttf") (io/read*))
params {::th/type :create-font-variant
::rpc/profile-id (:id prof)
:team-id team-id
:font-id font-id
:font-family "invalid-type"
:font-weight 400
:font-style "normal"
:data {"application/octet-stream" data}}
out (th/command! params)]
(t/is (some? (:error out)))
(t/is (= :validation (-> out :error ex-data :type)))
(t/is (= :media-type-not-allowed (-> out :error ex-data :code))))))
(t/deftest create-font-variant-invalid-type-legacy-chunked
"Legacy :data chunk-vector upload with a disallowed mtype must be rejected."
(with-mocks [_mock {:target 'app.rpc.quotes/check! :return nil}]
(let [prof (th/create-profile* 1 {:is-active true})
team-id (:default-team-id prof)
font-id (uuid/custom 10 61)
full-bytes (-> (io/resource "backend_tests/test_files/font-1.woff") (io/read*))
chunks (split-bytes-into-chunks full-bytes (* 4 1024 1024))
params {::th/type :create-font-variant
::rpc/profile-id (:id prof)
:team-id team-id
:font-id font-id
:font-family "invalid-type-legacy"
:font-weight 400
:font-style "normal"
:data {"image/png" (vec chunks)}}
out (th/command! params)]
(t/is (some? (:error out)))
(t/is (= :validation (-> out :error ex-data :type)))
(t/is (= :media-type-not-allowed (-> out :error ex-data :code))))))
(t/deftest create-font-variant-invalid-type-chunked-upload
"New :uploads path with a disallowed mtype must be rejected after assembly."
(with-mocks [_mock {:target 'app.rpc.quotes/check! :return nil}]
@ -836,46 +517,50 @@
data (-> (io/resource "backend_tests/test_files/font-1.ttf") (io/read*))]
;; name with < should fail
(let [params {::th/type :create-font-variant
(let [session-id (upload-font-chunked! prof data "font/ttf" (* 4 1024 1024))
params {::th/type :create-font-variant
::rpc/profile-id (:id prof)
:team-id team-id :font-id font-id
:font-family "evil<script>alert(1)</script>"
:font-weight 400 :font-style "normal"
:data {"font/ttf" data}}
:uploads {"font/ttf" session-id}}
out (th/command! params)]
(t/is (not (th/success? out)))
(t/is (th/ex-of-type? (:error out) :validation))
(t/is (th/ex-of-code? (:error out) :params-validation)))
;; name with ' should fail
(let [params {::th/type :create-font-variant
(let [session-id (upload-font-chunked! prof data "font/ttf" (* 4 1024 1024))
params {::th/type :create-font-variant
::rpc/profile-id (:id prof)
:team-id team-id :font-id font-id
:font-family "evil'name"
:font-weight 400 :font-style "normal"
:data {"font/ttf" data}}
:uploads {"font/ttf" session-id}}
out (th/command! params)]
(t/is (not (th/success? out)))
(t/is (th/ex-of-type? (:error out) :validation)))
;; name with } should fail
(let [params {::th/type :create-font-variant
(let [session-id (upload-font-chunked! prof data "font/ttf" (* 4 1024 1024))
params {::th/type :create-font-variant
::rpc/profile-id (:id prof)
:team-id team-id :font-id font-id
:font-family "evil}name"
:font-weight 400 :font-style "normal"
:data {"font/ttf" data}}
:uploads {"font/ttf" session-id}}
out (th/command! params)]
(t/is (not (th/success? out)))
(t/is (th/ex-of-type? (:error out) :validation)))
;; valid name should succeed
(let [params {::th/type :create-font-variant
(let [session-id (upload-font-chunked! prof data "font/ttf" (* 4 1024 1024))
params {::th/type :create-font-variant
::rpc/profile-id (:id prof)
:team-id team-id :font-id (uuid/custom 10 101)
:font-family "Source Sans Pro"
:font-weight 400 :font-style "normal"
:data {"font/ttf" data}}
:uploads {"font/ttf" session-id}}
out (th/command! params)]
(t/is (th/success? out))))))
@ -887,12 +572,13 @@
data (-> (io/resource "backend_tests/test_files/font-1.ttf") (io/read*))]
;; Create a valid font first
(let [params {::th/type :create-font-variant
(let [session-id (upload-font-chunked! prof data "font/ttf" (* 4 1024 1024))
params {::th/type :create-font-variant
::rpc/profile-id (:id prof)
:team-id team-id :font-id font-id
:font-family "ValidFont"
:font-weight 400 :font-style "normal"
:data {"font/ttf" data}}
:uploads {"font/ttf" session-id}}
out (th/command! params)]
(t/is (th/success? out)))

37
backend/test/backend_tests/rpc_media_test.clj
View File

@ -380,8 +380,41 @@
(t/is (= :validation (:type (ex-data err))))
(t/is (= :unable-to-download-image (:code (ex-data err))))))))
;; --------------------------------------------------------------------
;; Helpers for chunked-upload tests
(t/deftest download-image-closes-stream
(t/testing "response body stream is closed on success"
(let [closed? (atom false)
;; Minimal valid PNG (1x1 pixel, red)
png-data (byte-array [0x89 0x50 0x4E 0x47 0x0D 0x0A 0x1A 0x0A 0x00 0x00 0x00 0x0D 0x49 0x48 0x44 0x52 0x00 0x00 0x00 0x01 0x00 0x00 0x00 0x01 0x08 0x02 0x00 0x00 0x00 0x90 0x77 0x53 0xDE 0x00 0x00 0x00 0x0C 0x49 0x44 0x41 0x54 0x08 0xD7 0x63 0xF8 0xCF 0xC0 0x00 0x00 0x00 0x02 0x00 0x01 0xE2 0x21 0xBC 0x33 0x00 0x00 0x00 0x00 0x49 0x45 0x4E 0x44 0xAE 0x42 0x60 0x82])
body (proxy [java.io.ByteArrayInputStream] [png-data]
(close [] (reset! closed? true)))]
(with-mocks [http-mock {:target 'app.http.client/req-with-redirects
:return {:status 200
:headers {"content-type" "image/png"
"content-length" (str (alength png-data))}
:body body}}]
(let [cfg {::http/client :mock-client}
result (media/download-image cfg "https://example.com/image.png")]
(t/is (some? result))
(t/is @closed? "body stream should be closed after successful download")))))
(t/testing "response body stream is closed on validation error"
(let [closed? (atom false)
body (proxy [java.io.ByteArrayInputStream] [(byte-array 100)]
(close [] (reset! closed? true)))]
(with-mocks [http-mock {:target 'app.http.client/req-with-redirects
:return {:status 404
:headers {"content-type" "text/html"
"content-length" "100"}
:body body}}]
(let [cfg {::http/client :mock-client}
err (try
(media/download-image cfg "https://example.com/not-found.png")
nil
(catch clojure.lang.ExceptionInfo e e))]
(t/is (some? err))
(t/is (= :unable-to-download-image (:code (ex-data err))))
(t/is @closed? "body stream should be closed even on validation error"))))))
;; --------------------------------------------------------------------
(defn- split-file-into-chunks

29
backend/test/backend_tests/storage_test.clj
View File

@ -199,6 +199,25 @@
(let [res (th/db-exec-one! ["select count(*) from storage_object where deleted_at is not null"])]
(t/is (= 0 (:count res)))))))
(defn- upload-font-chunked!
"Splits `font-bytes` into a single chunk, creates an upload session,
uploads the chunk, and returns the session-id UUID."
[prof ^bytes font-bytes mtype]
(let [tmp (fs/create-tempfile :dir "/tmp/penpot" :prefix "test-font-chunk-")
_ (io/write* tmp font-bytes)
mfile {:filename "chunk" :path tmp :mtype mtype :size (alength font-bytes)}
session-id (-> (th/command! {::th/type :create-upload-session
::rpc/profile-id (:id prof)
:total-chunks 1})
:result :session-id)
out (th/command! {::th/type :upload-chunk
::rpc/profile-id (:id prof)
:session-id session-id
:index 0
:content mfile})]
(assert (nil? (:error out)))
session-id))
(t/deftest touched-gc-task-2
(let [storage (-> (:app.storage/storage th/*system*)
(configure-storage-backend))
@ -229,6 +248,8 @@
:name "testfile"
:content mfile}
session-id (upload-font-chunked! prof ttfdata "font/ttf")
params2 {::th/type :create-font-variant
::rpc/profile-id (:id prof)
:team-id team-id
@ -236,7 +257,7 @@
:font-family "somefont"
:font-weight 400
:font-style "normal"
:data {"font/ttf" ttfdata}}
:uploads {"font/ttf" session-id}}
out1 (th/command! params1)
out2 (th/command! params2)]
@ -250,7 +271,7 @@
(let [res (binding [ct/*clock* (ct/fixed-clock (ct/in-future {:hours 3}))]
(th/run-task! :storage-gc-touched {}))]
(t/is (= 5 (:freeze res)))
(t/is (= 0 (:delete res)))
(t/is (= 1 (:delete res)))
(let [result-1 (:result out1)
result-2 (:result out2)]
@ -271,7 +292,7 @@
(let [res (binding [ct/*clock* (ct/fixed-clock (ct/in-future {:hours 3}))]
(th/run-task! :storage-gc-touched {}))]
(t/is (= 2 (:freeze res)))
(t/is (= 3 (:delete res))))
(t/is (= 4 (:delete res))))
;; now check that there are no touched objects
(let [res (th/db-exec-one! ["select count(*) from storage_object where touched_at is not null"])]
@ -279,7 +300,7 @@
;; now check that all objects are marked to be deleted
(let [res (th/db-exec-one! ["select count(*) from storage_object where deleted_at is not null"])]
(t/is (= 3 (:count res))))))))
(t/is (= 4 (:count res))))))))
(t/deftest touched-gc-task-3
(let [storage (-> (:app.storage/storage th/*system*)

3
common/src/app/common/flags.cljc
View File

@ -178,7 +178,8 @@
:stroke-path
:stroke-per-side
:custom-shortcuts})
:custom-shortcuts
:remote-media-processing})
(def all-flags
(set/union email login varia))

86
docker/images/Dockerfile.media-processor Normal file
View File

@ -0,0 +1,86 @@
FROM ubuntu:26.04
LABEL maintainer="Penpot <docker@penpot.app>"
ENV LANG=en_US.UTF-8 \
LC_ALL=en_US.UTF-8 \
NODE_VERSION=v24.18.0 \
DEBIAN_FRONTEND=noninteractive \
PATH=/opt/node/bin:$PATH
RUN set -ex; \
useradd -U -M -u 1001 -s /bin/false -d /opt/penpot penpot; \
mkdir -p /etc/resolvconf/resolv.conf.d; \
echo "nameserver 127.0.0.11" > /etc/resolvconf/resolv.conf.d/tail; \
apt-get -qq update; \
apt-get -qq dist-upgrade; \
apt-get -qqy --no-install-recommends install \
curl \
tzdata \
locales \
ca-certificates \
; \
apt-get clean; \
rm -rf /var/lib/apt/lists/*; \
echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen; \
locale-gen; \
find /usr/share/i18n/locales/ -type f ! -name "en_US" ! -name "POSIX" ! -name "C" -delete;
RUN set -ex; \
apt-get -qq update; \
apt-get -qqy --no-install-recommends install \
fontforge \
woff-tools \
woff2 \
\
libgomp1 \
libheif1 \
libjpeg-turbo8 \
liblcms2-2 \
libopenexr-3-1-30 \
libopenjp2-7 \
libpng16-16 \
librsvg2-2 \
libtiff6 \
libwebp7 \
libwebpdemux2 \
libwebpmux3 \
libxml2-16 \
libzip5 \
libzstd1 \
; \
apt-get clean; \
rm -rf /var/lib/apt/lists/*;
RUN set -eux; \
ARCH="$(dpkg --print-architecture)"; \
case "${ARCH}" in \
aarch64|arm64) \
BINARY_URL="https://nodejs.org/dist/${NODE_VERSION}/node-${NODE_VERSION}-linux-arm64.tar.gz"; \
;; \
amd64|x86_64) \
BINARY_URL="https://nodejs.org/dist/${NODE_VERSION}/node-${NODE_VERSION}-linux-x64.tar.gz"; \
;; \
*) \
echo "Unsupported arch: ${ARCH}"; \
exit 1; \
;; \
esac; \
curl -LfsSo /tmp/nodejs.tar.gz ${BINARY_URL}; \
mkdir -p /opt/node; \
cd /opt/node; \
tar -xf /tmp/nodejs.tar.gz --strip-components=1; \
chown -R root /opt/node; \
rm -rf /tmp/nodejs.tar.gz; \
corepack enable; \
mkdir -p /opt/penpot; \
chown -R penpot:penpot /opt/penpot;
ARG BUNDLE_PATH="./bundle-media-processor/"
COPY --chown=penpot:penpot $BUNDLE_PATH /opt/penpot/media-processor/
WORKDIR /opt/penpot/media-processor
USER penpot:penpot
RUN ./setup
CMD ["node", "dist/index.js"]

3
media-processor/.prettierignore Normal file
View File

@ -0,0 +1,3 @@
dist/
node_modules/
coverage/

9
media-processor/.prettierrc Normal file
View File

@ -0,0 +1,9 @@
{
"tabWidth": 2,
"useTabs": false,
"semi": true,
"singleQuote": false,
"trailingComma": "es5",
"printWidth": 120,
"endOfLine": "lf"
}

17
media-processor/esbuild.config.mjs Normal file
View File

@ -0,0 +1,17 @@
import { build } from "esbuild";
await build({
entryPoints: ["src/index.ts"],
bundle: true,
platform: "node",
target: "node24",
format: "esm",
outfile: "dist/index.js",
external: ["sharp", "pino", "pino-pretty", "pino-loki"],
banner: {
js: `
import { createRequire } from 'node:module';
const require = createRequire(import.meta.url);
`,
},
});

40
media-processor/package.json Normal file
View File

@ -0,0 +1,40 @@
{
"name": "media-processor",
"version": "1.0.0",
"description": "Stateless HTTP service for Penpot image and font processing",
"type": "module",
"main": "dist/index.js",
"scripts": {
"build": "node esbuild.config.mjs",
"start": "node dist/index.js",
"start:dev": "tsx --env-file=../backend/scripts/_env src/index.ts",
"types:check": "tsc --noEmit",
"test": "vitest run",
"fmt": "prettier --write src/ test/",
"fmt:check": "prettier --check src/ test/",
"clean": "rm -rf dist/"
},
"packageManager": "pnpm@11.18.0+sha512.33d83c77da82f49fba836925c6f1b841181ec3132b670639bd012f7075f5c7cf634c5f870147c19aae7478fac01df09d8892e880454896edd23ee9b33757563c",
"dependencies": {
"express": "^5.2.1",
"multer": "^2.2.0",
"p-queue": "^9.3.3",
"pino": "^10.3.1",
"pino-loki": "^3.0.0",
"pino-pretty": "^13.1.3",
"sharp": "^0.35.3",
"zod": "^4.4.3"
},
"devDependencies": {
"@types/express": "^5.0.6",
"@types/multer": "^2.0.0",
"@types/node": "^26.1.2",
"@types/supertest": "^7.2.1",
"esbuild": "^0.28.1",
"prettier": "^3.6.2",
"supertest": "^7.2.2",
"tsx": "^4.22.4",
"typescript": "^7.0.2",
"vitest": "^4.1.10"
}
}

2649
media-processor/pnpm-lock.yaml generated Normal file
View File

File diff suppressed because it is too large Load Diff

2
media-processor/pnpm-workspace.yaml Normal file
View File

@ -0,0 +1,2 @@
allowBuilds:
esbuild: true

4
media-processor/scripts/build Executable file
View File

@ -0,0 +1,4 @@
#!/bin/bash
set -e
cd "$(dirname "$0")/.."
pnpm run build

6
media-processor/scripts/setup Executable file
View File

@ -0,0 +1,6 @@
#!/bin/bash
set -e
cd "$(dirname "$0")/.."
corepack enable
corepack install
pnpm install

64
media-processor/src/config.ts Normal file
View File

@ -0,0 +1,64 @@
import { z } from "zod";
import { hkdfSync } from "node:crypto";
import type { AppConfig } from "./types.js";
const envSchema = z.object({
PENPOT_MEDIA_PROCESSOR_PORT: z.coerce.number().int().positive().default(6065),
PENPOT_MEDIA_PROCESSOR_HOST: z.string().default("0.0.0.0"),
PENPOT_MEDIA_PROCESSOR_MAX_CONCURRENT_REQUESTS: z.coerce.number().int().min(1).default(10),
PENPOT_MEDIA_PROCESSOR_REQUEST_TIMEOUT: z.coerce.number().int().nonnegative().default(180000),
PENPOT_MEDIA_PROCESSOR_MAX_FILE_SIZE: z.coerce.number().int().positive().default(367001600), // 350 MB
PENPOT_MEDIA_PROCESSOR_MEMORY_THRESHOLD: z.coerce.number().int().positive().default(10485760), // 10 MB — uploads below this use memory storage; above use disk storage
PENPOT_MEDIA_PROCESSOR_IMAGE_MAX_PIXELS: z.coerce.number().int().positive().default(128_000_000),
PENPOT_MEDIA_PROCESSOR_IMAGE_MAX_WIDTH: z.coerce.number().int().positive().default(16384),
PENPOT_MEDIA_PROCESSOR_IMAGE_MAX_HEIGHT: z.coerce.number().int().positive().default(16384),
PENPOT_MEDIA_PROCESSOR_FONT_PROCESS_MEM: z.coerce.number().int().positive().default(512),
PENPOT_MEDIA_PROCESSOR_FONT_PROCESS_CPU_TIME: z.coerce.number().int().positive().default(30),
PENPOT_MEDIA_PROCESSOR_FONT_TIMEOUT: z.coerce.number().int().positive().default(120000),
PENPOT_MEDIA_PROCESSOR_SHARED_KEY: z.string().optional(),
PENPOT_SECRET_KEY: z.string().optional(),
PENPOT_MEDIA_PROCESSOR_LOG_LEVEL: z
.enum(["fatal", "error", "warn", "info", "debug", "trace", "silent"])
.default("info"),
PENPOT_LOGGERS_LOKI_URI: z.string().optional(),
PENPOT_LOGGERS_LOKI_JOB: z.string().default("media-processor"),
PENPOT_LOGGERS_LOKI_ENVIRONMENT: z.string().optional(),
PENPOT_LOGGERS_LOKI_INSTANCE: z.string().optional(),
});
function deriveSharedKey(secret: string): string {
const key = hkdfSync("blake2b512", secret, Buffer.from("media-processor"), "", 32);
return Buffer.from(key).toString("base64url");
}
export function loadConfig(): AppConfig {
const parsed = envSchema.parse(process.env);
let sharedKey: string | null = null;
if (parsed.PENPOT_MEDIA_PROCESSOR_SHARED_KEY) {
sharedKey = parsed.PENPOT_MEDIA_PROCESSOR_SHARED_KEY;
} else if (parsed.PENPOT_SECRET_KEY) {
sharedKey = deriveSharedKey(parsed.PENPOT_SECRET_KEY);
}
return {
port: parsed.PENPOT_MEDIA_PROCESSOR_PORT,
host: parsed.PENPOT_MEDIA_PROCESSOR_HOST,
maxConcurrentRequests: parsed.PENPOT_MEDIA_PROCESSOR_MAX_CONCURRENT_REQUESTS,
requestTimeout: parsed.PENPOT_MEDIA_PROCESSOR_REQUEST_TIMEOUT,
maxFileSize: parsed.PENPOT_MEDIA_PROCESSOR_MAX_FILE_SIZE,
memoryThreshold: parsed.PENPOT_MEDIA_PROCESSOR_MEMORY_THRESHOLD,
imageMaxPixels: parsed.PENPOT_MEDIA_PROCESSOR_IMAGE_MAX_PIXELS,
imageMaxWidth: parsed.PENPOT_MEDIA_PROCESSOR_IMAGE_MAX_WIDTH,
imageMaxHeight: parsed.PENPOT_MEDIA_PROCESSOR_IMAGE_MAX_HEIGHT,
fontProcessMem: parsed.PENPOT_MEDIA_PROCESSOR_FONT_PROCESS_MEM,
fontProcessCpuTime: parsed.PENPOT_MEDIA_PROCESSOR_FONT_PROCESS_CPU_TIME,
fontTimeout: parsed.PENPOT_MEDIA_PROCESSOR_FONT_TIMEOUT,
sharedKey,
logLevel: parsed.PENPOT_MEDIA_PROCESSOR_LOG_LEVEL,
lokiUri: parsed.PENPOT_LOGGERS_LOKI_URI || null,
lokiJob: parsed.PENPOT_LOGGERS_LOKI_JOB,
lokiEnvironment: parsed.PENPOT_LOGGERS_LOKI_ENVIRONMENT || null,
lokiInstance: parsed.PENPOT_LOGGERS_LOKI_INSTANCE || null,
};
}

59
media-processor/src/index.ts Normal file
View File

@ -0,0 +1,59 @@
import express, { type Express } from "express";
import { loadConfig } from "./config.js";
import { initLogger, logger, logActiveTransports } from "./logger.js";
import { healthRoutes } from "./routes/health.js";
import { createImageRoutes } from "./routes/image.js";
import { createFontRoutes } from "./routes/font.js";
import { errorHandler } from "./middleware/error-handler.js";
import { timeoutMiddleware } from "./middleware/timeout.js";
import { sharedKeyAuth } from "./middleware/auth.js";
import { createQueueMiddleware } from "./middleware/queue.js";
import { loggingMiddleware } from "./middleware/logging.js";
import { configureImageLimits } from "./services/image.js";
import { configureFontLimits } from "./services/font.js";
import { configureUploadLimits } from "./upload.js";
import sharp from "sharp";
// Auth is enforced via x-shared-key header (sharedKeyAuth middleware).
// When no key is configured, all requests are rejected (403).
// This service MUST be deployed on an internal Docker network only
// — do NOT expose to the public internet.
// Disable sharp/libvips caching to prevent unbounded memory growth
sharp.cache(false);
const config = loadConfig();
initLogger(config);
const app: Express = express();
// Configure resource limits
configureImageLimits({
maxPixels: config.imageMaxPixels,
maxWidth: config.imageMaxWidth,
maxHeight: config.imageMaxHeight,
});
configureFontLimits({
mem: config.fontProcessMem,
cpuTime: config.fontProcessCpuTime,
timeout: config.fontTimeout,
});
configureUploadLimits({ maxFileSize: config.maxFileSize, memoryThreshold: config.memoryThreshold });
const queueMiddleware = createQueueMiddleware(config.maxConcurrentRequests);
app.use(timeoutMiddleware(config.requestTimeout));
app.use(loggingMiddleware);
app.get("/api/health", healthRoutes);
app.use("/api/image", sharedKeyAuth(config.sharedKey), queueMiddleware, createImageRoutes());
app.use("/api/font", sharedKeyAuth(config.sharedKey), queueMiddleware, createFontRoutes());
app.use(errorHandler);
app.listen(config.port, config.host, () => {
logActiveTransports(logger);
logger.info(`media-processor listening on ${config.host}:${config.port}`);
});
export { app };

135
media-processor/src/logger.ts Normal file
View File

@ -0,0 +1,135 @@
import pino, { type TransportTargetOptions } from "pino";
import { loadConfig } from "./config.js";
import type { AppConfig } from "./types.js";
interface LogTransportProvider {
getTarget(): TransportTargetOptions | null;
getStartupMessage(): string | null;
}
class ConsoleLogTransport implements LogTransportProvider {
public constructor(private readonly config: AppConfig) {}
public getTarget(): TransportTargetOptions {
return {
target: "pino-pretty",
level: this.config.logLevel,
options: {
colorize: true,
translateTime: "SYS:yyyy-mm-dd HH:MM:ss.l",
ignore: "pid,hostname",
messageFormat: "{msg}",
levelFirst: true,
},
};
}
public getStartupMessage(): string {
return "Logging to console";
}
}
class LokiLogTransport implements LogTransportProvider {
private readonly host: string | null;
public constructor(
private readonly config: AppConfig,
lokiUri: string | null
) {
this.host = lokiUri;
}
public getTarget(): TransportTargetOptions | null {
if (this.host === null) {
return null;
}
return {
target: "pino-loki",
level: this.config.logLevel,
options: {
host: this.host,
json: false,
batching: true,
interval: 5,
replaceTimestamp: true,
labels: this.buildLabels(),
messageFormat: "{msg}",
ignore: "pid,hostname",
},
};
}
private buildLabels(): Record<string, string> {
const labels: Record<string, string> = {
job: this.config.lokiJob,
};
if (this.config.lokiEnvironment) {
labels.environment = this.config.lokiEnvironment;
}
if (this.config.lokiInstance) {
labels.instance = this.config.lokiInstance;
}
return labels;
}
public getStartupMessage(): string | null {
return this.host !== null ? `Logging to Loki: ${this.host}` : null;
}
}
function buildLogger(config: AppConfig) {
const consoleTransport = new ConsoleLogTransport(config);
const lokiTransport = new LokiLogTransport(config, config.lokiUri);
const transports: LogTransportProvider[] = [consoleTransport, lokiTransport];
const instance = pino({
level: config.logLevel,
timestamp: pino.stdTimeFunctions.isoTime,
transport: {
targets: transports
.map((t) => t.getTarget())
.filter((target): target is TransportTargetOptions => target !== null),
},
});
return { instance, transports };
}
let _instance: pino.Logger | null = null;
let _transports: LogTransportProvider[] = [];
export function initLogger(config: AppConfig): pino.Logger {
const result = buildLogger(config);
_instance = result.instance;
_transports = result.transports;
return _instance;
}
function getInstance(): pino.Logger {
if (_instance === null) {
return initLogger(loadConfig());
}
return _instance;
}
// Export as a getter so consumers see the lazily-initialized instance.
export const logger: pino.Logger = new Proxy({} as pino.Logger, {
get(_, prop) {
const inst = getInstance();
const value = (inst as unknown as Record<string | symbol, unknown>)[prop];
return typeof value === "function" ? value.bind(inst) : value;
},
});
export function logActiveTransports(log: pino.Logger): void {
for (const t of _transports) {
const msg = t.getStartupMessage();
if (msg !== null) {
log.info(msg);
}
}
}
export function createLogger(name: string) {
return logger.child({ name });
}

27
media-processor/src/middleware/auth.ts Normal file
View File

@ -0,0 +1,27 @@
import { timingSafeEqual } from "node:crypto";
import type { Request, Response, NextFunction } from "express";
export function sharedKeyAuth(expectedKey: string | null) {
if (expectedKey === null) {
return (_req: Request, res: Response, _next: NextFunction): void => {
res.status(403).json({ type: "authorization", code: "forbidden", hint: "Shared key not configured" });
};
}
return (req: Request, res: Response, next: NextFunction): void => {
const provided = req.headers["x-shared-key"];
if (typeof provided !== "string") {
res.status(403).json({ type: "authorization", code: "forbidden" });
return;
}
const providedBuf = Buffer.from(provided);
const expectedBuf = Buffer.from(expectedKey);
if (providedBuf.length === expectedBuf.length && timingSafeEqual(providedBuf, expectedBuf)) {
next();
} else {
res.status(403).json({ type: "authorization", code: "forbidden" });
}
};
}

25
media-processor/src/middleware/cleanup.ts Normal file
View File

@ -0,0 +1,25 @@
import { rm } from "node:fs/promises";
import type { Request, Response, NextFunction } from "express";
import { createLogger } from "../logger.js";
const logger = createLogger("cleanup");
export function cleanupMiddleware(req: Request, _res: Response, next: NextFunction): void {
let cleaned = false;
_res.on("finish", cleanup);
_res.on("close", cleanup);
async function cleanup() {
if (cleaned) return;
cleaned = true;
const file = req.file as (Express.Multer.File & { path?: string }) | undefined;
if (file?.path) {
await rm(file.path, { force: true }).catch((err) => {
logger.debug({ err, path: file.path }, "Failed to cleanup uploaded file");
});
}
}
next();
}

55
media-processor/src/middleware/error-handler.ts Normal file
View File

@ -0,0 +1,55 @@
import type { Request, Response, NextFunction } from "express";
import type { AppError } from "../types.js";
import { createLogger } from "../logger.js";
import multer from "multer";
const logger = createLogger("error-handler");
export class ProcessingError extends Error {
public readonly statusCode: number;
public readonly errorBody: AppError;
constructor(statusCode: number, body: AppError) {
super(body.hint ?? body.code);
this.statusCode = statusCode;
this.errorBody = body;
}
}
function releaseSlot(res: Response): void {
const releaseQueue = (res as any).locals?.releaseQueue;
if (releaseQueue) releaseQueue();
}
export function errorHandler(err: Error, _req: Request, res: Response, _next: NextFunction): void {
if (res.headersSent) {
return;
}
if (err instanceof ProcessingError) {
logger.warn({ err, statusCode: err.statusCode }, "Processing error");
res.status(err.statusCode).json(err.errorBody);
releaseSlot(res);
return;
}
if (err instanceof multer.MulterError) {
if (err.code === "LIMIT_FILE_SIZE") {
logger.warn({ err }, "Upload size limit exceeded");
res.status(413).json({
type: "restriction",
code: "payload-too-large",
});
releaseSlot(res);
return;
}
}
logger.error({ err }, "Unhandled error");
res.status(500).json({
type: "internal",
code: "processing-error",
hint: "Internal server error",
});
releaseSlot(res);
}

21
media-processor/src/middleware/logging.ts Normal file
View File

@ -0,0 +1,21 @@
import type { Request, Response, NextFunction } from "express";
import { logger } from "../logger.js";
const OP_NAMES: Record<string, string> = {
"POST /api/image/info": "image/info",
"POST /api/image/thumbnail": "image/thumbnail",
"POST /api/font/convert": "font/convert",
};
export function loggingMiddleware(req: Request, res: Response, next: NextFunction): void {
const start = Date.now();
res.on("finish", () => {
const path = req.originalUrl?.split("?")[0];
const op = OP_NAMES[`${req.method} ${path}`];
if (op) {
const meta = res.locals.opMeta ? `, ${res.locals.opMeta}` : "";
logger.info(`op=${op}${meta}, status=${res.statusCode}, elapsed=${Date.now() - start}ms`);
}
});
next();
}

34
media-processor/src/middleware/queue.ts Normal file
View File

@ -0,0 +1,34 @@
import type { Request, Response, NextFunction } from "express";
import PQueue from "p-queue";
export function createQueueMiddleware(concurrency: number) {
const queue = new PQueue({ concurrency });
return function queueMiddleware(_req: Request, res: Response, next: NextFunction): void {
queue
.add(
() =>
new Promise<void>((resolve) => {
if (res.writableEnded) {
resolve();
return;
}
let released = false;
const release = () => {
if (!released) {
released = true;
resolve();
}
};
// Store releaseQueue callback on res.locals so route handlers and error handler can call it
(res as any).locals = (res as any).locals || {};
(res as any).locals.releaseQueue = release;
next();
})
)
.catch((err) => next(err instanceof Error ? err : new Error("Request processing failed")));
};
}

35
media-processor/src/middleware/timeout.ts Normal file
View File

@ -0,0 +1,35 @@
import type { Request, Response, NextFunction } from "express";
export function timeoutMiddleware(timeout: number) {
return (req: Request, res: Response, next: NextFunction): void => {
// Create AbortController for request cancellation
const abortController = new AbortController();
(req as any).abortController = abortController;
const timer = setTimeout(() => {
if (!res.headersSent) {
res.status(504).json({
type: "internal",
code: "processing-timeout",
hint: "Request timed out",
});
// Abort the signal to cancel ongoing processing
abortController.abort();
res.on("finish", () => req.destroy());
}
}, timeout);
// Clear timer on finish (successful completion)
res.on("finish", () => clearTimeout(timer));
// Clear timer and abort signal on close (client disconnect)
res.on("close", () => {
clearTimeout(timer);
if (!abortController.signal.aborted) {
abortController.abort();
}
});
next();
};
}

55
media-processor/src/routes/font.ts Normal file
View File

@ -0,0 +1,55 @@
import { Router, type IRouter, type Request, type Response, type NextFunction } from "express";
import { getUpload, getFileInput } from "../upload.js";
import { convertFont } from "../services/font.js";
import { throwValidation } from "../services/errors.js";
import { cleanupMiddleware } from "../middleware/cleanup.js";
const VALID_TARGET_MTYPES = new Set(["font/ttf", "font/otf", "font/woff"]);
const VALID_SOURCE_MTYPES = new Set(["font/ttf", "font/otf", "font/woff", "font/woff2"]);
export function createFontRoutes(): IRouter {
const router: IRouter = Router();
const upload = getUpload();
router.post(
"/convert",
upload.single("file"),
cleanupMiddleware,
async (req: Request, res: Response, next: NextFunction) => {
const releaseQueue = (res as any).locals?.releaseQueue;
const signal = (req as any).abortController?.signal;
try {
if (!req.file) {
throwValidation("invalid-font", "No file uploaded");
}
const input = getFileInput(req.file!);
const sourceMtype = req.file!.mimetype;
if (!VALID_SOURCE_MTYPES.has(sourceMtype)) {
throwValidation("invalid-font", `Unrecognized font mime-type: ${sourceMtype}`);
}
const targetMtype = req.query["target-type"] as string;
if (!targetMtype || !VALID_TARGET_MTYPES.has(targetMtype)) {
throwValidation("invalid-font", `Invalid target-type. Must be one of: font/ttf, font/otf, font/woff`);
}
res.locals.opMeta = `src=${sourceMtype}, dest=${targetMtype}`;
const result = await convertFont(input, sourceMtype, targetMtype, signal);
if (!result) {
throwValidation("invalid-font", `Conversion from ${sourceMtype} to ${targetMtype} is not supported`);
}
res.setHeader("Content-Type", targetMtype);
res.send(result);
} catch (err) {
next(err);
} finally {
if (releaseQueue) releaseQueue();
}
}
);
return router;
}

5
media-processor/src/routes/health.ts Normal file
View File

@ -0,0 +1,5 @@
import type { Request, Response } from "express";
export function healthRoutes(_req: Request, res: Response): void {
res.json({ status: "ok" });
}

95
media-processor/src/routes/image.ts Normal file
View File

@ -0,0 +1,95 @@
import { Router, type IRouter, type Request, type Response, type NextFunction } from "express";
import { getUpload, getFileInput } from "../upload.js";
import { getImageInfo, generateThumbnail } from "../services/image.js";
import { throwValidation } from "../services/errors.js";
import { cleanupMiddleware } from "../middleware/cleanup.js";
import type { ThumbnailParams } from "../types.js";
export function parseQuality(value: string | undefined, defaultValue = 85): number {
if (value === undefined) return defaultValue;
const parsed = parseInt(value, 10);
if (isNaN(parsed)) return defaultValue;
return Math.min(100, Math.max(1, parsed));
}
export function createImageRoutes(): IRouter {
const router: IRouter = Router();
const upload = getUpload();
router.post(
"/info",
upload.single("file"),
cleanupMiddleware,
async (req: Request, res: Response, next: NextFunction) => {
const releaseQueue = (res as any).locals?.releaseQueue;
const signal = (req as any).abortController?.signal;
try {
if (!req.file) {
throwValidation("invalid-image", "No file uploaded");
}
const input = getFileInput(req.file!);
const info = await getImageInfo(input, req.file!.size, signal);
res.locals.opMeta = `mtype=${info.mtype}, size=${info.width}x${info.height}`;
res.json(info);
} catch (err) {
next(err);
} finally {
if (releaseQueue) releaseQueue();
}
}
);
router.post(
"/thumbnail",
upload.single("file"),
cleanupMiddleware,
async (req: Request, res: Response, next: NextFunction) => {
const releaseQueue = (res as any).locals?.releaseQueue;
const signal = (req as any).abortController?.signal;
try {
if (!req.file) {
throwValidation("invalid-image", "No file uploaded");
}
const input = getFileInput(req.file!);
const width = parseInt(req.query.width as string, 10);
const height = parseInt(req.query.height as string, 10);
const quality = parseQuality(req.query.quality as string);
const format = (req.query.format as string) || "jpeg";
const mode = (req.query.mode as string) || "fit";
if (isNaN(width) || isNaN(height) || width <= 0 || height <= 0) {
throwValidation("invalid-image", "width and height must be positive integers");
}
if (!["jpeg", "webp", "png"].includes(format)) {
throwValidation("invalid-image", `Unsupported format: ${format}`);
}
if (!["fit", "crop"].includes(mode)) {
throwValidation("invalid-image", `Unsupported mode: ${mode}`);
}
const params: ThumbnailParams = {
width,
height,
quality,
format: format as "jpeg" | "webp" | "png",
mode: mode as "fit" | "crop",
};
res.locals.opMeta = `size=${width}x${height}, fmt=${format}, mode=${mode}, q=${params.quality}`;
const { data, mtype } = await generateThumbnail(input, params, signal);
res.setHeader("Content-Type", mtype);
res.send(data);
} catch (err) {
next(err);
} finally {
if (releaseQueue) releaseQueue();
}
}
);
return router;
}

26
media-processor/src/services/errors.ts Normal file
View File

@ -0,0 +1,26 @@
import { ProcessingError } from "../middleware/error-handler.js";
import type { AppError } from "../types.js";
export function throwValidation(code: string, hint?: string): never {
throw new ProcessingError(400, {
type: "validation",
code,
hint,
} satisfies AppError);
}
export function throwRestriction(code: string, hint?: string): never {
throw new ProcessingError(413, {
type: "restriction",
code,
hint,
} satisfies AppError);
}
export function throwProcessing(code: string, hint?: string): never {
throw new ProcessingError(503, {
type: "internal",
code,
hint,
} satisfies AppError);
}

313
media-processor/src/services/font.ts Normal file
View File

@ -0,0 +1,313 @@
import { execFile } from "node:child_process";
import { writeFile, readFile, mkdtemp, rm, copyFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { throwValidation, throwProcessing } from "./errors.js";
import { createLogger } from "../logger.js";
import type { FileInput } from "../types.js";
const logger = createLogger("font");
let fontProcessMem = 512;
let fontProcessCpuTime = 30;
let fontTimeout = 120000;
export function configureFontLimits(opts: { mem: number; cpuTime: number; timeout: number }): void {
fontProcessMem = opts.mem;
fontProcessCpuTime = opts.cpuTime;
fontTimeout = opts.timeout;
}
export function execCommand(
cmd: string,
args: string[],
timeout?: number,
options?: { encoding?: BufferEncoding | "buffer"; signal?: AbortSignal }
): Promise<{ stdout: string | Buffer; stderr: string | Buffer }> {
const effectiveTimeout = timeout ?? fontTimeout;
const encoding = options?.encoding ?? "utf8";
// Use prlimit on Linux for memory + CPU resource limits
// Matches backend's prlimit-based font processing protection
const isLinux = process.platform === "linux";
let finalCmd = cmd;
let finalArgs = args;
if (isLinux && cmd !== "prlimit") {
// Wrap with prlimit: address space ceiling + CPU time limit
const prlimitArgs = [
`--as=${fontProcessMem * 1024 * 1024}`, // address space (memory)
`--cpu=${fontProcessCpuTime}`, // CPU seconds
"--",
cmd,
...args,
];
finalCmd = "prlimit";
finalArgs = prlimitArgs;
}
return new Promise((resolve, reject) => {
execFile(
finalCmd,
finalArgs,
{
timeout: effectiveTimeout,
encoding: encoding === "buffer" ? null : encoding,
signal: options?.signal,
},
(err, stdout, stderr) => {
if (err) {
const error = new Error(`Command failed: ${finalCmd} ${finalArgs.join(" ")}\n${stderr}`);
if (err.killed) (error as any).killed = err.killed;
if (err.signal) (error as any).signal = err.signal;
if (err.code !== null && err.code !== undefined) (error as any).code = err.code;
reject(error);
} else {
resolve({ stdout, stderr });
}
}
);
});
}
async function withTempDir<T>(fn: (dir: string) => Promise<T>): Promise<T> {
const dir = await mkdtemp(join(tmpdir(), "penpot.font."));
try {
return await fn(dir);
} finally {
await rm(dir, { recursive: true, force: true }).catch(() => {});
}
}
async function withTempInput<T>(
ext: string,
input: FileInput,
fn: (dir: string, inputPath: string) => Promise<T>
): Promise<T> {
return withTempDir(async (dir) => {
const inputPath = join(dir, `input${ext}`);
if (typeof input === "string") {
await copyFile(input, inputPath);
} else {
await writeFile(inputPath, input);
}
return fn(dir, inputPath);
});
}
async function fontConvert(
inputExt: string,
outputExt: string,
input: FileInput,
signal?: AbortSignal
): Promise<Buffer | null> {
return withTempDir(async (dir) => {
let inputPath: string;
if (typeof input === "string") {
inputPath = input; // Use path directly — avoids reading file into heap
} else {
inputPath = join(dir, `input${inputExt}`);
await writeFile(inputPath, input); // Write buffer to temp file
}
// Ensure input path is from tmpdir to prevent injection
if (!inputPath.startsWith(tmpdir())) {
throw new Error("Font processing denied: input path is outside expected directory");
}
const outputPath = join(dir, `input${outputExt}`);
try {
// Escape single quotes for FontForge's string parser (not shell).
// execFile passes args as an array — no shell injection vector.
// FontForge's own lexer uses doubled single quotes for escaping.
const escInput = inputPath.replace(/'/g, "''");
const escOutput = outputPath.replace(/'/g, "''");
await execCommand("fontforge", ["-lang=ff", "-c", `Open('${escInput}'); Generate('${escOutput}')`], undefined, {
signal,
});
return await readFile(outputPath);
} catch (err: unknown) {
const error = err as NodeJS.ErrnoException & { killed?: boolean; signal?: string };
// Detect resource limit kills from prlimit (SIGKILL = OOM, SIGXCPU = CPU time exceeded)
if (error.killed || error.signal === "SIGKILL" || error.signal === "SIGXCPU") {
logger.warn({ err, inputExt, outputExt }, "FontForge killed by resource limits");
throwProcessing("resource-limit-exceeded", "Font processing exceeded resource limits");
}
logger.warn({ err, inputExt, outputExt }, "FontForge conversion failed");
return null;
}
});
}
async function ttfToOtf(input: FileInput, signal?: AbortSignal): Promise<Buffer | null> {
return fontConvert(".ttf", ".otf", input, signal);
}
async function otfToTtf(input: FileInput, signal?: AbortSignal): Promise<Buffer | null> {
return fontConvert(".otf", ".ttf", input, signal);
}
async function sfntToWoff(input: FileInput, ext: string = ".ttf", signal?: AbortSignal): Promise<Buffer | null> {
return withTempInput(ext, input, async (dir, inputPath) => {
try {
await execCommand("sfnt2woff", [inputPath], undefined, { signal });
const output = join(dir, "input.woff");
return await readFile(output);
} catch (err: unknown) {
const error = err as NodeJS.ErrnoException & { killed?: boolean; signal?: string };
if (error.killed || error.signal === "SIGKILL" || error.signal === "SIGXCPU") {
logger.warn({ err }, "sfnt2woff killed by resource limits");
throwProcessing("resource-limit-exceeded", "Font processing exceeded resource limits");
}
logger.warn({ err }, "sfnt2woff conversion failed");
return null;
}
});
}
async function woffToSfnt(input: FileInput, signal?: AbortSignal): Promise<Buffer | null> {
return withTempInput(".woff", input, async (_dir, inputPath) => {
try {
const { stdout } = await execCommand("woff2sfnt", [inputPath], undefined, { encoding: "buffer", signal });
return stdout as Buffer;
} catch (err: unknown) {
const error = err as NodeJS.ErrnoException & { killed?: boolean; signal?: string };
if (error.killed || error.signal === "SIGKILL" || error.signal === "SIGXCPU") {
logger.warn({ err }, "woff2sfnt killed by resource limits");
throwProcessing("resource-limit-exceeded", "Font processing exceeded resource limits");
}
logger.warn({ err }, "woff2sfnt conversion failed");
return null;
}
});
}
async function woff2ToSfnt(input: FileInput, signal?: AbortSignal): Promise<Buffer | null> {
return withTempInput(".woff2", input, async (dir, inputPath) => {
const output = join(dir, "input.ttf");
try {
await execCommand("woff2_decompress", [inputPath], undefined, { signal });
return await readFile(output);
} catch (err: unknown) {
const error = err as NodeJS.ErrnoException & { killed?: boolean; signal?: string };
if (error.killed || error.signal === "SIGKILL" || error.signal === "SIGXCPU") {
logger.warn({ err }, "woff2_decompress killed by resource limits");
throwProcessing("resource-limit-exceeded", "Font processing exceeded resource limits");
}
logger.warn({ err }, "woff2_decompress failed");
return null;
}
});
}
function getSfntType(data: Buffer): "ttf" | "otf" {
const magic = data.subarray(0, 4).toString("hex");
switch (magic) {
case "4f54544f":
return "otf";
case "00010000":
return "ttf";
default:
throwValidation("invalid-font", "Unrecognized font format");
}
}
async function convertFromSfnt(sfnt: Buffer, targetType: string, signal?: AbortSignal): Promise<Buffer | null> {
if (targetType === "ttf") {
const stype = getSfntType(sfnt);
if (stype === "ttf") return sfnt;
return otfToTtf(sfnt, signal);
}
if (targetType === "otf") {
const stype = getSfntType(sfnt);
if (stype === "otf") return sfnt;
return ttfToOtf(sfnt, signal);
}
if (targetType === "woff") {
return sfntToWoff(sfnt, ".ttf", signal);
}
return null;
}
function validateFontSignature(data: Buffer, expectedType: string): void {
if (data.length < 4) {
throwValidation("invalid-font", "Font data too short");
}
const magic = data.subarray(0, 4).toString("hex");
switch (expectedType) {
case "ttf":
if (magic !== "00010000") {
throwValidation("invalid-font", "Invalid TTF signature");
}
break;
case "otf":
if (magic !== "4f54544f") {
throwValidation("invalid-font", "Invalid OTF signature");
}
break;
case "woff":
if (magic !== "774f4646") {
throwValidation("invalid-font", "Invalid WOFF signature");
}
break;
case "woff2":
if (magic !== "774f4632") {
throwValidation("invalid-font", "Invalid WOFF2 signature");
}
break;
}
}
export async function convertFont(
input: FileInput,
sourceMtype: string,
targetMtype: string,
signal?: AbortSignal
): Promise<Buffer | null> {
const sourceType = sourceMtype.replace("font/", "");
const targetType = targetMtype.replace("font/", "");
// Same type: validate signature and return data as-is
if (sourceType === targetType) {
let data: Buffer;
if (typeof input === "string") {
data = await readFile(input);
} else {
data = input;
}
validateFontSignature(data, sourceType);
return data;
}
// Source is TTF
if (sourceType === "ttf") {
if (targetType === "otf") return ttfToOtf(input, signal);
if (targetType === "woff") return sfntToWoff(input, ".ttf", signal);
return null;
}
// Source is OTF
if (sourceType === "otf") {
if (targetType === "ttf") return otfToTtf(input, signal);
if (targetType === "woff") return sfntToWoff(input, ".otf", signal);
return null;
}
// Source is WOFF: extract sfnt first, then convert
if (sourceType === "woff") {
const sfnt = await woffToSfnt(input, signal);
if (!sfnt) {
throwValidation("invalid-font", "Could not extract SFNT from WOFF");
}
return convertFromSfnt(sfnt, targetType, signal);
}
// Source is WOFF2: decompress to sfnt, then convert
const sfnt = await woff2ToSfnt(input, signal);
if (!sfnt) {
throwValidation("invalid-font", "Could not decompress WOFF2");
}
return convertFromSfnt(sfnt, targetType, signal);
}

204
media-processor/src/services/image.ts Normal file
View File

@ -0,0 +1,204 @@
import sharp from "sharp";
import type { FileInput, ImageInfo, ThumbnailParams } from "../types.js";
import { throwValidation, throwRestriction } from "./errors.js";
import { createLogger } from "../logger.js";
const logger = createLogger("image");
const SUPPORTED_MIMES = new Set(["image/jpeg", "image/png", "image/webp", "image/gif"]);
function orientationSwapDimensions(
width: number,
height: number,
orientation: number
): { width: number; height: number } {
if (orientation === 6 || orientation === 8) {
return { width: height, height: width };
}
return { width, height };
}
let imageMaxPixels = 128_000_000;
let imageMaxWidth = 16384;
let imageMaxHeight = 16384;
export function configureImageLimits(opts: { maxPixels: number; maxWidth: number; maxHeight: number }): void {
imageMaxPixels = opts.maxPixels;
imageMaxWidth = opts.maxWidth;
imageMaxHeight = opts.maxHeight;
}
function validateImageDimensions(width: number, height: number): void {
if (width > imageMaxWidth || height > imageMaxHeight) {
throwRestriction(
"image-dimensions-exceeded",
`Image dimensions ${width}x${height} exceed maximum ${imageMaxWidth}x${imageMaxHeight}`
);
}
const pixels = width * height;
if (pixels > imageMaxPixels) {
throwRestriction("image-pixel-count-exceeded", `Image pixel count ${pixels} exceeds maximum ${imageMaxPixels}`);
}
}
function validateOutputDimensions(width: number, height: number): void {
if (width > imageMaxWidth || height > imageMaxHeight) {
throwRestriction(
"output-dimensions-exceeded",
`Requested output dimensions ${width}x${height} exceed maximum ${imageMaxWidth}x${imageMaxHeight}`
);
}
const pixels = width * height;
if (pixels > imageMaxPixels) {
throwRestriction(
"output-pixel-count-exceeded",
`Requested output pixel count ${pixels} exceeds maximum ${imageMaxPixels}`
);
}
}
export async function getImageInfo(input: FileInput, size: number, signal?: AbortSignal): Promise<ImageInfo> {
if (signal?.aborted) {
throw new Error("Request cancelled");
}
let metadata;
try {
metadata = await sharp(input).metadata();
} catch (err) {
throwValidation("invalid-image", `Failed to decode image: ${(err as Error).message}`);
}
if (!metadata.width || !metadata.height) {
throwValidation("invalid-image", "Could not read image dimensions");
}
const mtype = metadata.format ? `image/${metadata.format}` : undefined;
if (!mtype || !SUPPORTED_MIMES.has(mtype)) {
throwValidation("invalid-image", `Unsupported image format: ${metadata.format}`);
}
const orientation = metadata.orientation ?? 1;
const { width, height } = orientationSwapDimensions(metadata.width!, metadata.height!, orientation);
validateImageDimensions(width, height);
logger.debug({ width, height, mtype: mtype!, size }, "Image info extracted");
return {
width,
height,
mtype: mtype!,
size,
orientation,
};
}
const FORMAT_MIMES: Record<string, string> = {
jpeg: "image/jpeg",
webp: "image/webp",
png: "image/png",
};
export async function generateThumbnail(
input: FileInput,
params: ThumbnailParams,
signal?: AbortSignal
): Promise<{ data: Buffer; mtype: string }> {
// Check if request was cancelled before starting
if (signal?.aborted) {
throw new Error("Request cancelled");
}
// Pre-validate source image dimensions using the same sharp instance
// that will be used for the resize pipeline. Sharp reads metadata
// (dimensions, orientation) from the image header without fully decoding
// the pixel data, then reuses the instance for the resize operations.
const source = sharp(input);
let srcMeta;
try {
srcMeta = await source.metadata();
} catch (err) {
throwValidation("invalid-image", `Failed to decode image: ${(err as Error).message}`);
}
// Check again after metadata read
if (signal?.aborted) {
throw new Error("Request cancelled");
}
if (srcMeta.width == null || srcMeta.height == null) {
throwValidation("invalid-image", "Could not read source image dimensions");
}
// Validate source image format
if (srcMeta.format && !SUPPORTED_MIMES.has(`image/${srcMeta.format}`)) {
throwValidation("unsupported-image-format", `Unsupported image format: ${srcMeta.format}`);
}
const orientation = srcMeta.orientation ?? 1;
const { width: displayWidth, height: displayHeight } = orientationSwapDimensions(
srcMeta.width,
srcMeta.height,
orientation
);
validateImageDimensions(displayWidth, displayHeight);
// Validate requested output dimensions (important for crop mode which can enlarge)
validateOutputDimensions(params.width, params.height);
logger.debug(
{ width: params.width, height: params.height, format: params.format, mode: params.mode },
"Generating thumbnail"
);
let pipeline = source.rotate();
// Only flatten for JPEG output (which doesn't support transparency).
// PNG and WebP support alpha, so preserve it.
if (params.format === "jpeg") {
pipeline = pipeline.flatten({ background: { r: 255, g: 255, b: 255 } });
}
if (params.mode === "fit") {
pipeline = pipeline.resize(params.width, params.height, {
fit: "inside",
withoutEnlargement: true,
});
} else {
pipeline = pipeline.resize(params.width, params.height, {
fit: "cover",
position: "center",
});
}
switch (params.format) {
case "jpeg":
pipeline = pipeline.jpeg({ quality: params.quality });
break;
case "webp":
pipeline = pipeline.webp({ quality: params.quality });
break;
case "png":
pipeline = pipeline.png();
break;
}
let data: Buffer;
try {
// Sharp 0.35.3 does not support cancellation of native libvips operations.
// toBuffer() only accepts { resolveWithObject: boolean }, no AbortSignal.
// We hold the queue slot until Sharp completes fully, then check signal
// to throw if the request was cancelled during processing. This prevents
// concurrency limit violations and handles timeouts gracefully.
data = await pipeline.toBuffer();
} catch (err) {
throwValidation("invalid-image", `Failed to process image: ${(err as Error).message}`);
}
if (signal?.aborted) {
throw new Error("Request cancelled");
}
return { data, mtype: FORMAT_MIMES[params.format] };
}

44
media-processor/src/types.ts Normal file
View File

@ -0,0 +1,44 @@
export type FileInput = Buffer | string;
export interface AppConfig {
port: number;
host: string;
maxConcurrentRequests: number;
requestTimeout: number;
maxFileSize: number;
memoryThreshold: number;
imageMaxPixels: number;
imageMaxWidth: number;
imageMaxHeight: number;
fontProcessMem: number;
fontProcessCpuTime: number;
fontTimeout: number;
sharedKey: string | null;
logLevel: string;
lokiUri: string | null;
lokiJob: string;
lokiEnvironment: string | null;
lokiInstance: string | null;
}
export interface ImageInfo {
width: number;
height: number;
mtype: string;
size: number;
orientation: number;
}
export interface ThumbnailParams {
width: number;
height: number;
quality: number;
format: "jpeg" | "webp" | "png";
mode: "fit" | "crop";
}
export interface AppError {
type: "validation" | "restriction" | "internal";
code: string;
hint?: string;
}

89
media-processor/src/upload-storage.ts Normal file
View File

@ -0,0 +1,89 @@
import multer from "multer";
import { mkdtemp, rm } from "node:fs/promises";
import { createWriteStream } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { randomBytes } from "node:crypto";
import type { Request } from "express";
interface HybridStorageOptions {
memoryThreshold: number;
}
interface FileInfo {
destination: string;
filename: string;
path: string;
size: number;
}
function getContentLength(req: Request): number {
const cl = req.headers["content-length"];
if (!cl) return -1;
const parsed = parseInt(cl, 10);
return isNaN(parsed) ? -1 : parsed;
}
export function createHybridStorage(opts: HybridStorageOptions): multer.StorageEngine {
const memoryStorage = multer.memoryStorage();
let tempDirPromise: Promise<string> | null = null;
async function ensureTempDir(): Promise<string> {
if (!tempDirPromise) {
tempDirPromise = mkdtemp(join(tmpdir(), "penpot.upload."));
}
return tempDirPromise;
}
return {
_handleFile(req: Request, file: Express.Multer.File, cb: (error?: any, info?: Partial<FileInfo>) => void): void {
const contentLength = getContentLength(req);
const useDisk = contentLength < 0 || contentLength >= opts.memoryThreshold;
if (!useDisk) {
memoryStorage._handleFile(req, file, cb);
return;
}
ensureTempDir()
.then((dir) => {
const filename = `${randomBytes(16).toString("hex")}${getExt(file.originalname)}`;
const filepath = join(dir, filename);
const ws = createWriteStream(filepath);
file.stream.pipe(ws);
ws.on("error", (err: Error) => {
cb(err);
});
ws.on("finish", () => {
cb(null, {
destination: dir,
filename,
path: filepath,
size: ws.bytesWritten,
});
});
})
.catch(cb);
},
_removeFile(req: Request, file: Express.Multer.File & { path?: string }, cb: (error: Error | null) => void): void {
if (file.path) {
rm(file.path, { force: true })
.then(() => cb(null))
.catch(() => cb(null));
} else {
cb(null);
}
},
};
}
function getExt(filename: string): string {
const dot = filename.lastIndexOf(".");
return dot >= 0 ? filename.substring(dot) : "";
}

50
media-processor/src/upload.ts Normal file
View File

@ -0,0 +1,50 @@
import multer from "multer";
import { readFile } from "node:fs/promises";
import { createHybridStorage } from "./upload-storage.js";
import type { Request } from "express";
import type { FileInput } from "./types.js";
let _upload: multer.Multer | null = null;
// Hybrid storage: small uploads (< memoryThreshold) buffered in RAM for speed;
// large uploads streamed to disk to avoid heap pressure.
// Default threshold is 10MB. Disk files are cleaned up after response finishes.
export function configureUploadLimits(opts: { maxFileSize: number; memoryThreshold: number }): void {
const storage = createHybridStorage({ memoryThreshold: opts.memoryThreshold });
_upload = multer({
storage,
limits: { fileSize: opts.maxFileSize },
});
}
export function getUpload(): multer.Multer {
if (!_upload) {
throw new Error("Upload not configured — call configureUploadLimits first");
}
return _upload;
}
// Returns file input suitable for sharp and font processing.
// For disk-stored files, returns the file path (libvips uses mmap).
// For memory-stored files, returns the buffer.
export function getFileInput(file: Express.Multer.File): FileInput {
if (file.path) {
return file.path;
}
if (file.buffer) {
return file.buffer;
}
throw new Error("File has no buffer or path");
}
// Returns file contents as Buffer regardless of storage backend (memory or disk).
export async function getFileBuffer(file: Express.Multer.File): Promise<Buffer> {
if (file.buffer) {
return file.buffer;
}
if (file.path) {
return readFile(file.path);
}
throw new Error("File has no buffer or path");
}

130
media-processor/test/config.test.ts Normal file
View File

@ -0,0 +1,130 @@
import { describe, it, expect, beforeEach, afterEach } from "vitest";
import { loadConfig } from "../src/config.js";
describe("loadConfig", () => {
const originalEnv = process.env;
beforeEach(() => {
process.env = { ...originalEnv };
});
afterEach(() => {
process.env = originalEnv;
});
it("uses defaults when env vars not set", () => {
const config = loadConfig();
expect(config.port).toBe(6065);
expect(config.host).toBe("0.0.0.0");
expect(config.maxConcurrentRequests).toBe(10);
expect(config.requestTimeout).toBe(180000);
expect(config.maxFileSize).toBe(367001600);
expect(config.memoryThreshold).toBe(10485760);
});
it("accepts valid config with all fields set", () => {
process.env.PENPOT_MEDIA_PROCESSOR_PORT = "8080";
process.env.PENPOT_MEDIA_PROCESSOR_HOST = "127.0.0.1";
process.env.PENPOT_MEDIA_PROCESSOR_MAX_CONCURRENT_REQUESTS = "20";
process.env.PENPOT_MEDIA_PROCESSOR_REQUEST_TIMEOUT = "30000";
process.env.PENPOT_MEDIA_PROCESSOR_MAX_FILE_SIZE = "104857600";
process.env.PENPOT_MEDIA_PROCESSOR_MEMORY_THRESHOLD = "5242880";
process.env.PENPOT_MEDIA_PROCESSOR_SHARED_KEY = "test-key";
const config = loadConfig();
expect(config.port).toBe(8080);
expect(config.host).toBe("127.0.0.1");
expect(config.maxConcurrentRequests).toBe(20);
expect(config.requestTimeout).toBe(30000);
expect(config.maxFileSize).toBe(104857600);
expect(config.memoryThreshold).toBe(5242880);
expect(config.sharedKey).toBe("test-key");
});
it("rejects concurrency=0", () => {
process.env.PENPOT_MEDIA_PROCESSOR_MAX_CONCURRENT_REQUESTS = "0";
expect(() => loadConfig()).toThrow();
});
it("rejects negative concurrency", () => {
process.env.PENPOT_MEDIA_PROCESSOR_MAX_CONCURRENT_REQUESTS = "-5";
expect(() => loadConfig()).toThrow();
});
it("rejects fractional concurrency", () => {
process.env.PENPOT_MEDIA_PROCESSOR_MAX_CONCURRENT_REQUESTS = "2.5";
expect(() => loadConfig()).toThrow();
});
it("rejects negative timeout", () => {
process.env.PENPOT_MEDIA_PROCESSOR_REQUEST_TIMEOUT = "-1000";
expect(() => loadConfig()).toThrow();
});
it("rejects fractional timeout", () => {
process.env.PENPOT_MEDIA_PROCESSOR_REQUEST_TIMEOUT = "1000.5";
expect(() => loadConfig()).toThrow();
});
it("rejects fractional port", () => {
process.env.PENPOT_MEDIA_PROCESSOR_PORT = "8080.5";
expect(() => loadConfig()).toThrow();
});
it("rejects negative port", () => {
process.env.PENPOT_MEDIA_PROCESSOR_PORT = "-8080";
expect(() => loadConfig()).toThrow();
});
it("rejects negative max file size", () => {
process.env.PENPOT_MEDIA_PROCESSOR_MAX_FILE_SIZE = "-100";
expect(() => loadConfig()).toThrow();
});
it("rejects negative memory threshold", () => {
process.env.PENPOT_MEDIA_PROCESSOR_MEMORY_THRESHOLD = "-100";
expect(() => loadConfig()).toThrow();
});
it("rejects negative image max pixels", () => {
process.env.PENPOT_MEDIA_PROCESSOR_IMAGE_MAX_PIXELS = "-100";
expect(() => loadConfig()).toThrow();
});
it("rejects fractional image max width", () => {
process.env.PENPOT_MEDIA_PROCESSOR_IMAGE_MAX_WIDTH = "100.5";
expect(() => loadConfig()).toThrow();
});
it("rejects fractional image max height", () => {
process.env.PENPOT_MEDIA_PROCESSOR_IMAGE_MAX_HEIGHT = "100.5";
expect(() => loadConfig()).toThrow();
});
it("rejects negative font process mem", () => {
process.env.PENPOT_MEDIA_PROCESSOR_FONT_PROCESS_MEM = "-512";
expect(() => loadConfig()).toThrow();
});
it("rejects negative font process cpu time", () => {
process.env.PENPOT_MEDIA_PROCESSOR_FONT_PROCESS_CPU_TIME = "-30";
expect(() => loadConfig()).toThrow();
});
it("rejects negative font timeout", () => {
process.env.PENPOT_MEDIA_PROCESSOR_FONT_TIMEOUT = "-120000";
expect(() => loadConfig()).toThrow();
});
it("accepts concurrency=1 (minimum valid)", () => {
process.env.PENPOT_MEDIA_PROCESSOR_MAX_CONCURRENT_REQUESTS = "1";
const config = loadConfig();
expect(config.maxConcurrentRequests).toBe(1);
});
it("accepts timeout=0 (edge case, might be valid for testing)", () => {
process.env.PENPOT_MEDIA_PROCESSOR_REQUEST_TIMEOUT = "0";
const config = loadConfig();
expect(config.requestTimeout).toBe(0);
});
});

BIN
media-processor/test/fixtures/font-1.otf vendored Normal file
View File

Binary file not shown.

BIN
media-processor/test/fixtures/font-1.ttf vendored Normal file
View File

Binary file not shown.

BIN
media-processor/test/fixtures/font-1.woff vendored Normal file
View File

Binary file not shown.

BIN
media-processor/test/fixtures/font-1.woff2 vendored Normal file
View File

Binary file not shown.

310
media-processor/test/font.test.ts Normal file
View File

@ -0,0 +1,310 @@
import { describe, it, expect, beforeAll } from "vitest";
import { readFile, writeFile, rm } from "node:fs/promises";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { convertFont, execCommand } from "../src/services/font.js";
import { ProcessingError } from "../src/middleware/error-handler.js";
const FIXTURES = join(import.meta.dirname, "fixtures");
let ttfData: Buffer;
let otfData: Buffer;
let woffData: Buffer;
let woff2Data: Buffer;
beforeAll(async () => {
[ttfData, otfData, woffData, woff2Data] = await Promise.all([
readFile(join(FIXTURES, "font-1.ttf")),
readFile(join(FIXTURES, "font-1.otf")),
readFile(join(FIXTURES, "font-1.woff")),
readFile(join(FIXTURES, "font-1.woff2")),
]);
});
describe("convertFont", () => {
describe("sourceType=ttf", () => {
it("ttf→ttf returns the input buffer unchanged", async () => {
const result = await convertFont(ttfData, "font/ttf", "font/ttf");
expect(result).toBe(ttfData);
});
it("ttf→otf returns non-null Buffer", async () => {
const result = await convertFont(ttfData, "font/ttf", "font/otf");
expect(result).toBeInstanceOf(Buffer);
expect(result!.length).toBeGreaterThan(0);
});
it("ttf→woff returns non-null Buffer", async () => {
const result = await convertFont(ttfData, "font/ttf", "font/woff");
expect(result).toBeInstanceOf(Buffer);
expect(result!.length).toBeGreaterThan(0);
});
});
describe("sourceType=otf", () => {
it("otf→otf returns the input buffer unchanged", async () => {
const result = await convertFont(otfData, "font/otf", "font/otf");
expect(result).toBe(otfData);
});
it("otf→ttf returns non-null Buffer", async () => {
const result = await convertFont(otfData, "font/otf", "font/ttf");
expect(result).toBeInstanceOf(Buffer);
expect(result!.length).toBeGreaterThan(0);
});
it("otf→woff returns non-null Buffer", async () => {
const result = await convertFont(otfData, "font/otf", "font/woff");
expect(result).toBeInstanceOf(Buffer);
expect(result!.length).toBeGreaterThan(0);
});
});
describe("sourceType=woff", () => {
it("woff→woff returns the input buffer unchanged", async () => {
const result = await convertFont(woffData, "font/woff", "font/woff");
expect(result).toBe(woffData);
});
it("woff→ttf returns non-null Buffer", async () => {
const result = await convertFont(woffData, "font/woff", "font/ttf");
expect(result).toBeInstanceOf(Buffer);
expect(result!.length).toBeGreaterThan(0);
});
it("woff→otf returns Buffer or null (FontForge limitation)", async () => {
const result = await convertFont(woffData, "font/woff", "font/otf");
// FontForge may fail to convert TTF-based WOFF to OTF for some fonts.
// The backend handles null gracefully (variant is just absent).
if (result !== null) {
expect(result).toBeInstanceOf(Buffer);
expect(result.length).toBeGreaterThan(0);
}
});
});
describe("sourceType=woff2", () => {
it("woff2→ttf returns non-null Buffer", async () => {
const result = await convertFont(woff2Data, "font/woff2", "font/ttf");
expect(result).toBeInstanceOf(Buffer);
expect(result!.length).toBeGreaterThan(0);
});
it("woff2→otf returns non-null Buffer", async () => {
const result = await convertFont(woff2Data, "font/woff2", "font/otf");
expect(result).toBeInstanceOf(Buffer);
expect(result!.length).toBeGreaterThan(0);
});
it("woff2→woff returns non-null Buffer", async () => {
const result = await convertFont(woff2Data, "font/woff2", "font/woff");
expect(result).toBeInstanceOf(Buffer);
expect(result!.length).toBeGreaterThan(0);
});
});
describe("invalid input", () => {
it("woff with garbage data throws ProcessingError", async () => {
const garbage = Buffer.from("not a font at all");
await expect(convertFont(garbage, "font/woff", "font/ttf")).rejects.toThrow(ProcessingError);
});
it("woff2 with garbage data throws ProcessingError", async () => {
const garbage = Buffer.from("not a font at all");
await expect(convertFont(garbage, "font/woff2", "font/ttf")).rejects.toThrow(ProcessingError);
});
it("sfnt with garbage data throws validation error", async () => {
const garbage = Buffer.from("not a font at all");
try {
await convertFont(garbage, "font/woff", "font/ttf");
expect.fail("should have thrown");
} catch (err) {
expect(err).toBeInstanceOf(ProcessingError);
const pe = err as import("../src/middleware/error-handler.js").ProcessingError;
expect(pe.statusCode).toBe(400);
expect(pe.errorBody.code).toBe("invalid-font");
}
});
it("ttf→ttf with invalid magic bytes throws validation error", async () => {
const invalidTtf = Buffer.from("00000000", "hex");
try {
await convertFont(invalidTtf, "font/ttf", "font/ttf");
expect.fail("should have thrown");
} catch (err) {
expect(err).toBeInstanceOf(ProcessingError);
const pe = err as import("../src/middleware/error-handler.js").ProcessingError;
expect(pe.statusCode).toBe(400);
expect(pe.errorBody.code).toBe("invalid-font");
}
});
it("otf→otf with invalid magic bytes throws validation error", async () => {
const invalidOtf = Buffer.from("00000000", "hex");
try {
await convertFont(invalidOtf, "font/otf", "font/otf");
expect.fail("should have thrown");
} catch (err) {
expect(err).toBeInstanceOf(ProcessingError);
const pe = err as import("../src/middleware/error-handler.js").ProcessingError;
expect(pe.statusCode).toBe(400);
expect(pe.errorBody.code).toBe("invalid-font");
}
});
it("woff→woff with invalid magic bytes throws validation error", async () => {
const invalidWoff = Buffer.from("00000000", "hex");
try {
await convertFont(invalidWoff, "font/woff", "font/woff");
expect.fail("should have thrown");
} catch (err) {
expect(err).toBeInstanceOf(ProcessingError);
const pe = err as import("../src/middleware/error-handler.js").ProcessingError;
expect(pe.statusCode).toBe(400);
expect(pe.errorBody.code).toBe("invalid-font");
}
});
it("woff2→woff2 with invalid magic bytes throws validation error", async () => {
const invalidWoff2 = Buffer.from("00000000", "hex");
try {
await convertFont(invalidWoff2, "font/woff2", "font/woff2");
expect.fail("should have thrown");
} catch (err) {
expect(err).toBeInstanceOf(ProcessingError);
const pe = err as import("../src/middleware/error-handler.js").ProcessingError;
expect(pe.statusCode).toBe(400);
expect(pe.errorBody.code).toBe("invalid-font");
}
});
});
describe("data integrity", () => {
it("ttf→otf produces valid font buffer", async () => {
const result = await convertFont(ttfData, "font/ttf", "font/otf");
expect(result).toBeInstanceOf(Buffer);
expect(result!.length).toBeGreaterThan(0);
});
it("otf→ttf produces valid font buffer", async () => {
const result = await convertFont(otfData, "font/otf", "font/ttf");
expect(result).toBeInstanceOf(Buffer);
expect(result!.length).toBeGreaterThan(0);
});
it("woff→ttf produces valid SFNT with correct magic bytes", async () => {
const result = await convertFont(woffData, "font/woff", "font/ttf");
expect(result).toBeInstanceOf(Buffer);
expect(result!.length).toBeGreaterThan(0);
// SFNT magic: 00010000 (TTF) or 4f54544f (OTF/CFF)
const magic = result!.subarray(0, 4).toString("hex");
expect(["00010000", "4f54544f"]).toContain(magic);
});
it("woff2→ttf produces valid font buffer", async () => {
const result = await convertFont(woff2Data, "font/woff2", "font/ttf");
expect(result).toBeInstanceOf(Buffer);
expect(result!.length).toBeGreaterThan(0);
});
});
describe("file path input", () => {
it("ttf→otf with file path returns non-null Buffer", async () => {
const tempPath = join(tmpdir(), `test-font-${Date.now()}.ttf`);
try {
await writeFile(tempPath, ttfData);
const result = await convertFont(tempPath, "font/ttf", "font/otf");
expect(result).toBeInstanceOf(Buffer);
expect(result!.length).toBeGreaterThan(0);
} finally {
await rm(tempPath, { force: true });
}
});
it("ttf→ttf with file path returns file contents as Buffer", async () => {
const tempPath = join(tmpdir(), `test-font-${Date.now()}.ttf`);
try {
await writeFile(tempPath, ttfData);
const result = await convertFont(tempPath, "font/ttf", "font/ttf");
expect(result).toBeInstanceOf(Buffer);
expect(result!.length).toBe(ttfData.length);
} finally {
await rm(tempPath, { force: true });
}
});
it("woff→ttf with file path returns non-null Buffer", async () => {
const tempPath = join(tmpdir(), `test-font-${Date.now()}.woff`);
try {
await writeFile(tempPath, woffData);
const result = await convertFont(tempPath, "font/woff", "font/ttf");
expect(result).toBeInstanceOf(Buffer);
expect(result!.length).toBeGreaterThan(0);
} finally {
await rm(tempPath, { force: true });
}
});
it("ttf→woff with file path returns non-null Buffer", async () => {
const tempPath = join(tmpdir(), `test-font-${Date.now()}.ttf`);
try {
await writeFile(tempPath, ttfData);
const result = await convertFont(tempPath, "font/ttf", "font/woff");
expect(result).toBeInstanceOf(Buffer);
expect(result!.length).toBeGreaterThan(0);
} finally {
await rm(tempPath, { force: true });
}
});
it("woff2→ttf with file path returns non-null Buffer", async () => {
const tempPath = join(tmpdir(), `test-font-${Date.now()}.woff2`);
try {
await writeFile(tempPath, woff2Data);
const result = await convertFont(tempPath, "font/woff2", "font/ttf");
expect(result).toBeInstanceOf(Buffer);
expect(result!.length).toBeGreaterThan(0);
} finally {
await rm(tempPath, { force: true });
}
});
});
describe("path validation", () => {
it("rejects string input path outside tmpdir", async () => {
const outsidePath = "/etc/passwd";
try {
await convertFont(outsidePath, "font/ttf", "font/otf");
expect.fail("should have thrown");
} catch (err) {
const error = err as Error;
expect(error.message).toContain("Font processing denied: input path is outside expected directory");
}
});
});
});
describe("execCommand", () => {
it("preserves killed and signal properties from child process errors", async () => {
try {
await execCommand("false", []);
expect.fail("should have thrown");
} catch (err) {
const error = err as Error & { killed?: boolean; signal?: string; code?: number };
expect(error).toBeInstanceOf(Error);
expect(error.message).toContain("Command failed");
}
});
it("preserves error properties when command is killed by signal", async () => {
try {
await execCommand("sh", ["-c", "kill -KILL $$"], 5000);
expect.fail("should have thrown");
} catch (err) {
const error = err as Error & { killed?: boolean; signal?: string };
expect(error).toBeInstanceOf(Error);
expect(error.signal).toBe("SIGKILL");
}
});
});

909
media-processor/test/image.test.ts Normal file
View File

@ -0,0 +1,909 @@
import { describe, it, expect, afterEach } from "vitest";
import { getImageInfo, generateThumbnail, configureImageLimits } from "../src/services/image.js";
import { parseQuality } from "../src/routes/image.js";
import { ProcessingError } from "../src/middleware/error-handler.js";
import sharp from "sharp";
import { writeFile, rm } from "node:fs/promises";
import { join } from "node:path";
import { tmpdir } from "node:os";
const DEFAULT_LIMITS = {
maxPixels: 128_000_000,
maxWidth: 16384,
maxHeight: 16384,
};
afterEach(() => {
configureImageLimits(DEFAULT_LIMITS);
});
describe("getImageInfo", () => {
it("returns correct info for a PNG", async () => {
const buffer = await sharp({
create: { width: 100, height: 80, channels: 4, background: { r: 255, g: 0, b: 0, alpha: 1 } },
})
.png()
.toBuffer();
const info = await getImageInfo(buffer, buffer.length);
expect(info.width).toBe(100);
expect(info.height).toBe(80);
expect(info.mtype).toBe("image/png");
expect(info.size).toBe(buffer.length);
expect(info.orientation).toBe(1);
});
it("returns correct info for a JPEG", async () => {
const buffer = await sharp({
create: { width: 200, height: 150, channels: 3, background: { r: 0, g: 255, b: 0 } },
})
.jpeg()
.toBuffer();
const info = await getImageInfo(buffer, buffer.length);
expect(info.width).toBe(200);
expect(info.height).toBe(150);
expect(info.mtype).toBe("image/jpeg");
});
it("returns correct info for a WebP", async () => {
const buffer = await sharp({
create: { width: 300, height: 250, channels: 3, background: { r: 0, g: 0, b: 255 } },
})
.webp()
.toBuffer();
const info = await getImageInfo(buffer, buffer.length);
expect(info.width).toBe(300);
expect(info.height).toBe(250);
expect(info.mtype).toBe("image/webp");
});
it("throws on invalid image data", async () => {
const buffer = Buffer.from("not an image");
await expect(getImageInfo(buffer, buffer.length)).rejects.toThrow();
});
it("returns correct info for a GIF", async () => {
// sharp create doesn't support GIF directly, so create PNG then convert
const pngBuffer = await sharp({
create: { width: 120, height: 90, channels: 3, background: { r: 200, g: 100, b: 50 } },
})
.png()
.toBuffer();
const gifBuffer = await sharp(pngBuffer).gif().toBuffer();
const info = await getImageInfo(gifBuffer, gifBuffer.length);
expect(info.width).toBe(120);
expect(info.height).toBe(90);
expect(info.mtype).toBe("image/gif");
});
it("returns size equal to buffer length", async () => {
const buffer = await sharp({
create: { width: 50, height: 50, channels: 3, background: { r: 0, g: 0, b: 0 } },
})
.jpeg()
.toBuffer();
const info = await getImageInfo(buffer, buffer.length);
expect(info.size).toBe(buffer.length);
});
it("defaults orientation to 1 when no EXIF data", async () => {
const buffer = await sharp({
create: { width: 60, height: 40, channels: 3, background: { r: 0, g: 0, b: 0 } },
})
.png()
.toBuffer();
const info = await getImageInfo(buffer, buffer.length);
expect(info.orientation).toBe(1);
});
it("throws on garbage data (sharp unsupported format)", async () => {
const buffer = Buffer.alloc(100, 0xff);
await expect(getImageInfo(buffer, buffer.length)).rejects.toThrow();
});
it("accepts file path input and returns correct info", async () => {
const buffer = await sharp({
create: { width: 100, height: 80, channels: 3, background: { r: 0, g: 128, b: 255 } },
})
.jpeg()
.toBuffer();
const tempPath = join(tmpdir(), `test-image-${Date.now()}.jpg`);
try {
await writeFile(tempPath, buffer);
const info = await getImageInfo(tempPath, buffer.length);
expect(info.width).toBe(100);
expect(info.height).toBe(80);
expect(info.mtype).toBe("image/jpeg");
expect(info.size).toBe(buffer.length);
} finally {
await rm(tempPath, { force: true });
}
});
it("throws restriction when width exceeds limit", async () => {
configureImageLimits({ maxPixels: 128_000_000, maxWidth: 100, maxHeight: 16384 });
const buffer = await sharp({
create: { width: 200, height: 50, channels: 3, background: { r: 0, g: 0, b: 0 } },
})
.jpeg()
.toBuffer();
try {
await getImageInfo(buffer, buffer.length);
expect.fail("should have thrown");
} catch (err) {
expect(err).toBeInstanceOf(ProcessingError);
const pe = err as import("../src/middleware/error-handler.js").ProcessingError;
expect(pe.statusCode).toBe(413);
expect(pe.errorBody.code).toBe("image-dimensions-exceeded");
}
});
it("throws restriction when height exceeds limit", async () => {
configureImageLimits({ maxPixels: 128_000_000, maxWidth: 16384, maxHeight: 100 });
const buffer = await sharp({
create: { width: 50, height: 200, channels: 3, background: { r: 0, g: 0, b: 0 } },
})
.jpeg()
.toBuffer();
try {
await getImageInfo(buffer, buffer.length);
expect.fail("should have thrown");
} catch (err) {
expect(err).toBeInstanceOf(ProcessingError);
const pe = err as import("../src/middleware/error-handler.js").ProcessingError;
expect(pe.statusCode).toBe(413);
expect(pe.errorBody.code).toBe("image-dimensions-exceeded");
}
});
it("throws restriction when pixel count exceeds limit", async () => {
configureImageLimits({ maxPixels: 1000, maxWidth: 16384, maxHeight: 16384 });
const buffer = await sharp({
create: { width: 50, height: 50, channels: 3, background: { r: 0, g: 0, b: 0 } },
})
.jpeg()
.toBuffer();
try {
await getImageInfo(buffer, buffer.length);
expect.fail("should have thrown");
} catch (err) {
expect(err).toBeInstanceOf(ProcessingError);
const pe = err as import("../src/middleware/error-handler.js").ProcessingError;
expect(pe.statusCode).toBe(413);
expect(pe.errorBody.code).toBe("image-pixel-count-exceeded");
}
});
it("passes when dimensions are exactly at the limit", async () => {
configureImageLimits({ maxPixels: 10000, maxWidth: 100, maxHeight: 100 });
const buffer = await sharp({
create: { width: 100, height: 100, channels: 3, background: { r: 0, g: 0, b: 0 } },
})
.jpeg()
.toBuffer();
const info = await getImageInfo(buffer, buffer.length);
expect(info.width).toBe(100);
expect(info.height).toBe(100);
});
it("throws when pixel count is exactly 1 over limit", async () => {
configureImageLimits({ maxPixels: 9999, maxWidth: 16384, maxHeight: 16384 });
const buffer = await sharp({
create: { width: 100, height: 100, channels: 3, background: { r: 0, g: 0, b: 0 } },
})
.jpeg()
.toBuffer();
try {
await getImageInfo(buffer, buffer.length);
expect.fail("should have thrown");
} catch (err) {
expect(err).toBeInstanceOf(ProcessingError);
const pe = err as import("../src/middleware/error-handler.js").ProcessingError;
expect(pe.errorBody.code).toBe("image-pixel-count-exceeded");
}
});
it("throws when signal is aborted before processing", async () => {
const buffer = await sharp({
create: { width: 100, height: 80, channels: 3, background: { r: 255, g: 0, b: 0 } },
})
.png()
.toBuffer();
const controller = new AbortController();
controller.abort();
try {
await getImageInfo(buffer, buffer.length, controller.signal);
expect.fail("should have thrown");
} catch (err) {
expect(err).toBeInstanceOf(Error);
expect((err as Error).message).toBe("Request cancelled");
}
});
});
describe("generateThumbnail", () => {
const createImage = (w: number, h: number) =>
sharp({
create: { width: w, height: h, channels: 3, background: { r: 128, g: 128, b: 128 } },
})
.jpeg()
.toBuffer();
it("mode=fit produces thumbnail fitting within dimensions (no upscale)", async () => {
const buffer = await createImage(1000, 800);
const { data, mtype } = await generateThumbnail(buffer, {
width: 200,
height: 200,
quality: 85,
format: "jpeg",
mode: "fit",
});
const meta = await sharp(data).metadata();
expect(meta.width).toBeLessThanOrEqual(200);
expect(meta.height).toBeLessThanOrEqual(200);
expect(mtype).toBe("image/jpeg");
});
it("mode=crop produces center-cropped thumbnail at exact dimensions", async () => {
const buffer = await createImage(1000, 800);
const { data, mtype } = await generateThumbnail(buffer, {
width: 200,
height: 200,
quality: 85,
format: "jpeg",
mode: "crop",
});
const meta = await sharp(data).metadata();
expect(meta.width).toBe(200);
expect(meta.height).toBe(200);
expect(mtype).toBe("image/jpeg");
});
it("supports webp output", async () => {
const buffer = await createImage(500, 400);
const { data, mtype } = await generateThumbnail(buffer, {
width: 100,
height: 100,
quality: 80,
format: "webp",
mode: "fit",
});
expect(mtype).toBe("image/webp");
const meta = await sharp(data).metadata();
expect(meta.width).toBeLessThanOrEqual(100);
});
it("supports png output", async () => {
const buffer = await createImage(500, 400);
const { data, mtype } = await generateThumbnail(buffer, {
width: 100,
height: 100,
quality: 80,
format: "png",
mode: "fit",
});
expect(mtype).toBe("image/png");
});
it("fit mode does not upscale small source", async () => {
const buffer = await createImage(50, 40);
const { data } = await generateThumbnail(buffer, {
width: 200,
height: 200,
quality: 85,
format: "jpeg",
mode: "fit",
});
const meta = await sharp(data).metadata();
expect(meta.width).toBe(50);
expect(meta.height).toBe(40);
});
it("crop mode with non-square target", async () => {
const buffer = await createImage(1000, 500);
const { data, mtype } = await generateThumbnail(buffer, {
width: 200,
height: 100,
quality: 85,
format: "jpeg",
mode: "crop",
});
const meta = await sharp(data).metadata();
expect(meta.width).toBe(200);
expect(meta.height).toBe(100);
expect(mtype).toBe("image/jpeg");
});
it("png output with crop mode", async () => {
const buffer = await createImage(800, 600);
const { data, mtype } = await generateThumbnail(buffer, {
width: 150,
height: 150,
quality: 80,
format: "png",
mode: "crop",
});
const meta = await sharp(data).metadata();
expect(meta.width).toBe(150);
expect(meta.height).toBe(150);
expect(mtype).toBe("image/png");
});
it("webp output with crop mode", async () => {
const buffer = await createImage(800, 600);
const { data, mtype } = await generateThumbnail(buffer, {
width: 150,
height: 150,
quality: 80,
format: "webp",
mode: "crop",
});
const meta = await sharp(data).metadata();
expect(meta.width).toBe(150);
expect(meta.height).toBe(150);
expect(mtype).toBe("image/webp");
});
it("source at exact target dimensions (fit mode) returns same size", async () => {
const buffer = await createImage(200, 200);
const { data } = await generateThumbnail(buffer, {
width: 200,
height: 200,
quality: 85,
format: "jpeg",
mode: "fit",
});
const meta = await sharp(data).metadata();
expect(meta.width).toBe(200);
expect(meta.height).toBe(200);
});
it("source at exact target dimensions (crop mode) returns same size", async () => {
const buffer = await createImage(200, 200);
const { data } = await generateThumbnail(buffer, {
width: 200,
height: 200,
quality: 85,
format: "jpeg",
mode: "crop",
});
const meta = await sharp(data).metadata();
expect(meta.width).toBe(200);
expect(meta.height).toBe(200);
});
it("very small source (1x1) with fit mode returns 1x1", async () => {
const buffer = await createImage(1, 1);
const { data } = await generateThumbnail(buffer, {
width: 200,
height: 200,
quality: 85,
format: "jpeg",
mode: "fit",
});
const meta = await sharp(data).metadata();
expect(meta.width).toBe(1);
expect(meta.height).toBe(1);
});
it("throws restriction when source exceeds dimension limits", async () => {
configureImageLimits({ maxPixels: 1000, maxWidth: 50, maxHeight: 50 });
const buffer = await createImage(200, 200);
try {
await generateThumbnail(buffer, {
width: 100,
height: 100,
quality: 85,
format: "jpeg",
mode: "fit",
});
expect.fail("should have thrown");
} catch (err) {
expect(err).toBeInstanceOf(ProcessingError);
const pe = err as import("../src/middleware/error-handler.js").ProcessingError;
expect(pe.statusCode).toBe(413);
}
});
it("removes alpha channel from PNG source", async () => {
const pngBuffer = await sharp({
create: { width: 100, height: 100, channels: 4, background: { r: 255, g: 0, b: 0, alpha: 0.5 } },
})
.png()
.toBuffer();
const { data } = await generateThumbnail(pngBuffer, {
width: 50,
height: 50,
quality: 85,
format: "jpeg",
mode: "fit",
});
const meta = await sharp(data).metadata();
// JPEG output should not have alpha
expect(meta.channels).toBe(3);
});
it("composites transparent PNG onto white background for JPEG", async () => {
// Create a fully transparent PNG — removeAlpha() would produce black,
// but the local ImageMagick path composites onto white.
const pngBuffer = await sharp({
create: { width: 10, height: 10, channels: 4, background: { r: 0, g: 0, b: 0, alpha: 0 } },
})
.png()
.toBuffer();
const { data } = await generateThumbnail(pngBuffer, {
width: 10,
height: 10,
quality: 85,
format: "jpeg",
mode: "fit",
});
// Sample a pixel — should be white (255,255,255), not black (0,0,0)
const pixel = await sharp(data).raw().toBuffer();
const r = pixel[0];
const g = pixel[1];
const b = pixel[2];
expect(r).toBe(255);
expect(g).toBe(255);
expect(b).toBe(255);
});
it("GIF source works with thumbnail generation", async () => {
const pngBuffer = await sharp({
create: { width: 200, height: 200, channels: 3, background: { r: 100, g: 100, b: 100 } },
})
.png()
.toBuffer();
const gifBuffer = await sharp(pngBuffer).gif().toBuffer();
const { data, mtype } = await generateThumbnail(gifBuffer, {
width: 100,
height: 100,
quality: 85,
format: "jpeg",
mode: "fit",
});
const meta = await sharp(data).metadata();
expect(meta.width).toBeLessThanOrEqual(100);
expect(meta.height).toBeLessThanOrEqual(100);
expect(mtype).toBe("image/jpeg");
});
it("JPEG quality affects output file size", async () => {
// Create an image with actual detail (gradient) so quality matters
const width = 200;
const height = 200;
const channels = 3;
const rawBuffer = Buffer.alloc(width * height * channels);
for (let y = 0; y < height; y++) {
for (let x = 0; x < width; x++) {
const idx = (y * width + x) * channels;
rawBuffer[idx] = (x * 255) / width;
rawBuffer[idx + 1] = (y * 255) / height;
rawBuffer[idx + 2] = ((x + y) * 255) / (width + height);
}
}
const buffer = await sharp(rawBuffer, { raw: { width, height, channels } }).jpeg().toBuffer();
const low = await generateThumbnail(buffer, {
width: 100,
height: 100,
quality: 10,
format: "jpeg",
mode: "fit",
});
const high = await generateThumbnail(buffer, {
width: 100,
height: 100,
quality: 100,
format: "jpeg",
mode: "fit",
});
expect(low.data.length).toBeLessThan(high.data.length);
});
it("accepts quality=1 (minimum valid quality)", async () => {
const buffer = await sharp({
create: { width: 100, height: 80, channels: 3, background: { r: 128, g: 128, b: 128 } },
})
.jpeg()
.toBuffer();
const { data, mtype } = await generateThumbnail(buffer, {
width: 50,
height: 40,
quality: 1,
format: "jpeg",
mode: "fit",
});
expect(data).toBeInstanceOf(Buffer);
expect(data.length).toBeGreaterThan(0);
expect(mtype).toBe("image/jpeg");
});
it("accepts file path input for thumbnail generation", async () => {
const buffer = await sharp({
create: { width: 100, height: 80, channels: 3, background: { r: 128, g: 128, b: 128 } },
})
.jpeg()
.toBuffer();
const tempPath = join(tmpdir(), `test-image-${Date.now()}.jpg`);
try {
await writeFile(tempPath, buffer);
const { data, mtype } = await generateThumbnail(tempPath, {
width: 50,
height: 40,
quality: 85,
format: "jpeg",
mode: "fit",
});
const meta = await sharp(data).metadata();
expect(meta.width).toBeLessThanOrEqual(50);
expect(meta.height).toBeLessThanOrEqual(40);
expect(mtype).toBe("image/jpeg");
} finally {
await rm(tempPath, { force: true });
}
});
it("throws validation error for unsupported source format (TIFF)", async () => {
const pngBuffer = await sharp({
create: { width: 100, height: 80, channels: 3, background: { r: 128, g: 128, b: 128 } },
})
.png()
.toBuffer();
const tiffBuffer = await sharp(pngBuffer).tiff().toBuffer();
try {
await generateThumbnail(tiffBuffer, {
width: 50,
height: 40,
quality: 85,
format: "jpeg",
mode: "fit",
});
expect.fail("should have thrown");
} catch (err) {
expect(err).toBeInstanceOf(ProcessingError);
const pe = err as import("../src/middleware/error-handler.js").ProcessingError;
expect(pe.statusCode).toBe(400);
expect(pe.errorBody.code).toBe("unsupported-image-format");
}
});
it("throws invalid-image for corrupted image data", async () => {
// Create corrupted image by truncating valid image
const validBuffer = await sharp({
create: { width: 100, height: 100, channels: 3, background: { r: 255, g: 0, b: 0 } },
})
.jpeg()
.toBuffer();
// Truncate to create corrupted data
const corruptedBuffer = validBuffer.subarray(0, Math.floor(validBuffer.length / 2));
try {
await getImageInfo(corruptedBuffer, corruptedBuffer.length);
expect.fail("should have thrown");
} catch (err) {
expect(err).toBeInstanceOf(ProcessingError);
const pe = err as import("../src/middleware/error-handler.js").ProcessingError;
expect(pe.statusCode).toBe(400);
expect(pe.errorBody.code).toBe("invalid-image");
}
});
it("throws invalid-image for truncated image data in generateThumbnail", async () => {
const validBuffer = await sharp({
create: { width: 100, height: 100, channels: 3, background: { r: 255, g: 0, b: 0 } },
})
.jpeg()
.toBuffer();
// Truncate to create corrupted data
const corruptedBuffer = validBuffer.subarray(0, Math.floor(validBuffer.length / 2));
try {
await generateThumbnail(corruptedBuffer, {
width: 50,
height: 50,
quality: 85,
format: "jpeg",
mode: "fit",
});
expect.fail("should have thrown");
} catch (err) {
expect(err).toBeInstanceOf(ProcessingError);
const pe = err as import("../src/middleware/error-handler.js").ProcessingError;
expect(pe.statusCode).toBe(400);
expect(pe.errorBody.code).toBe("invalid-image");
}
});
it("preserves alpha channel in PNG output from transparent PNG", async () => {
// Create a transparent PNG with alpha < 1
const transparentPng = await sharp({
create: { width: 100, height: 100, channels: 4, background: { r: 255, g: 0, b: 0, alpha: 0.5 } },
})
.png()
.toBuffer();
const { data, mtype } = await generateThumbnail(transparentPng, {
width: 50,
height: 50,
quality: 85,
format: "png",
mode: "fit",
});
expect(mtype).toBe("image/png");
const meta = await sharp(data).metadata();
// PNG should preserve alpha channel (4 channels)
expect(meta.channels).toBe(4);
});
it("preserves alpha channel in WebP output from transparent PNG", async () => {
const transparentPng = await sharp({
create: { width: 100, height: 100, channels: 4, background: { r: 0, g: 255, b: 0, alpha: 0.5 } },
})
.png()
.toBuffer();
const { data, mtype } = await generateThumbnail(transparentPng, {
width: 50,
height: 50,
quality: 85,
format: "webp",
mode: "fit",
});
expect(mtype).toBe("image/webp");
const meta = await sharp(data).metadata();
// WebP should preserve alpha channel (4 channels)
expect(meta.channels).toBe(4);
});
it("preserves alpha channel in PNG output from transparent WebP", async () => {
// Create a transparent WebP
const transparentWebp = await sharp({
create: { width: 100, height: 100, channels: 4, background: { r: 0, g: 0, b: 255, alpha: 0.5 } },
})
.webp()
.toBuffer();
const { data, mtype } = await generateThumbnail(transparentWebp, {
width: 50,
height: 50,
quality: 85,
format: "png",
mode: "fit",
});
expect(mtype).toBe("image/png");
const meta = await sharp(data).metadata();
// PNG should preserve alpha channel (4 channels)
expect(meta.channels).toBe(4);
});
it("throws restriction when requested width exceeds limit (crop mode)", async () => {
configureImageLimits({ maxPixels: 128_000_000, maxWidth: 100, maxHeight: 16384 });
const buffer = await createImage(50, 50);
try {
await generateThumbnail(buffer, {
width: 200,
height: 50,
quality: 85,
format: "jpeg",
mode: "crop",
});
expect.fail("should have thrown");
} catch (err) {
expect(err).toBeInstanceOf(ProcessingError);
const pe = err as import("../src/middleware/error-handler.js").ProcessingError;
expect(pe.statusCode).toBe(413);
expect(pe.errorBody.code).toBe("output-dimensions-exceeded");
}
});
it("throws restriction when requested height exceeds limit (crop mode)", async () => {
configureImageLimits({ maxPixels: 128_000_000, maxWidth: 16384, maxHeight: 100 });
const buffer = await createImage(50, 50);
try {
await generateThumbnail(buffer, {
width: 50,
height: 200,
quality: 85,
format: "jpeg",
mode: "crop",
});
expect.fail("should have thrown");
} catch (err) {
expect(err).toBeInstanceOf(ProcessingError);
const pe = err as import("../src/middleware/error-handler.js").ProcessingError;
expect(pe.statusCode).toBe(413);
expect(pe.errorBody.code).toBe("output-dimensions-exceeded");
}
});
it("throws restriction when requested pixel count exceeds limit (crop mode)", async () => {
configureImageLimits({ maxPixels: 10000, maxWidth: 16384, maxHeight: 16384 });
const buffer = await createImage(50, 50);
try {
await generateThumbnail(buffer, {
width: 200,
height: 200,
quality: 85,
format: "jpeg",
mode: "crop",
});
expect.fail("should have thrown");
} catch (err) {
expect(err).toBeInstanceOf(ProcessingError);
const pe = err as import("../src/middleware/error-handler.js").ProcessingError;
expect(pe.statusCode).toBe(413);
expect(pe.errorBody.code).toBe("output-pixel-count-exceeded");
}
});
it("accepts requested dimensions at the limit (crop mode)", async () => {
configureImageLimits({ maxPixels: 10000, maxWidth: 100, maxHeight: 100 });
const buffer = await createImage(50, 50);
const { data, mtype } = await generateThumbnail(buffer, {
width: 100,
height: 100,
quality: 85,
format: "jpeg",
mode: "crop",
});
expect(mtype).toBe("image/jpeg");
const meta = await sharp(data).metadata();
expect(meta.width).toBe(100);
expect(meta.height).toBe(100);
});
it("throws when signal is aborted before processing", async () => {
const buffer = await createImage(100, 80);
const controller = new AbortController();
controller.abort();
try {
await generateThumbnail(
buffer,
{
width: 50,
height: 40,
quality: 85,
format: "jpeg",
mode: "fit",
},
controller.signal
);
expect.fail("should have thrown");
} catch (err) {
expect(err).toBeInstanceOf(Error);
expect((err as Error).message).toBe("Request cancelled");
}
});
it("aborts during toBuffer when signal fires", async () => {
const buffer = await createImage(2000, 2000);
const controller = new AbortController();
const signal = controller.signal;
const promise = generateThumbnail(
buffer,
{
width: 1000,
height: 1000,
quality: 85,
format: "jpeg",
mode: "fit",
},
signal
);
setTimeout(() => controller.abort(), 10);
try {
await promise;
expect.fail("should have thrown");
} catch (err) {
expect(err).toBeInstanceOf(Error);
expect((err as Error).message).toBe("Request cancelled");
}
});
it("waits for Sharp to complete before checking signal", async () => {
const buffer = await createImage(100, 80);
const controller = new AbortController();
const signal = controller.signal;
const promise = generateThumbnail(
buffer,
{
width: 50,
height: 40,
quality: 85,
format: "jpeg",
mode: "fit",
},
signal
);
controller.abort();
try {
await promise;
expect.fail("should have thrown");
} catch (err) {
expect(err).toBeInstanceOf(Error);
expect((err as Error).message).toBe("Request cancelled");
}
});
});
describe("parseQuality", () => {
it("returns default when value is undefined", () => {
expect(parseQuality(undefined)).toBe(85);
});
it("returns default when value is empty string", () => {
expect(parseQuality("")).toBe(85);
});
it("returns default when value is not a number", () => {
expect(parseQuality("abc")).toBe(85);
});
it("returns parsed value when valid", () => {
expect(parseQuality("50")).toBe(50);
});
it("clamps quality=0 to 1 (minimum valid)", () => {
expect(parseQuality("0")).toBe(1);
});
it("preserves quality=1 (minimum valid)", () => {
expect(parseQuality("1")).toBe(1);
});
it("clamps quality=101 to 100 (maximum valid)", () => {
expect(parseQuality("101")).toBe(100);
});
it("preserves quality=100 (maximum valid)", () => {
expect(parseQuality("100")).toBe(100);
});
it("uses custom default value", () => {
expect(parseQuality(undefined, 75)).toBe(75);
expect(parseQuality("abc", 75)).toBe(75);
});
});

681
media-processor/test/middleware.test.ts Normal file
View File

@ -0,0 +1,681 @@
import { describe, it, expect, vi, beforeEach } from "vitest";
import { ProcessingError, errorHandler } from "../src/middleware/error-handler.js";
import { sharedKeyAuth } from "../src/middleware/auth.js";
import { timeoutMiddleware } from "../src/middleware/timeout.js";
import { cleanupMiddleware } from "../src/middleware/cleanup.js";
import { throwValidation, throwRestriction } from "../src/services/errors.js";
import multer from "multer";
import type { Request, Response, NextFunction } from "express";
import { EventEmitter } from "node:events";
function mockRes() {
const res = {
status: vi.fn().mockReturnThis(),
json: vi.fn().mockReturnThis(),
send: vi.fn().mockReturnThis(),
headersSent: false,
};
return res as unknown as Response;
}
function mockReq() {
return {} as Request;
}
describe("ProcessingError", () => {
it("stores statusCode", () => {
const err = new ProcessingError(400, {
type: "validation",
code: "test-error",
});
expect(err.statusCode).toBe(400);
});
it("stores errorBody", () => {
const body = { type: "validation" as const, code: "test-error", hint: "details" };
const err = new ProcessingError(400, body);
expect(err.errorBody).toEqual(body);
});
it("message defaults to code when no hint", () => {
const err = new ProcessingError(400, {
type: "validation",
code: "test-error",
});
expect(err.message).toBe("test-error");
});
it("message uses hint when provided", () => {
const err = new ProcessingError(400, {
type: "validation",
code: "test-error",
hint: "something went wrong",
});
expect(err.message).toBe("something went wrong");
});
it("is an instance of Error", () => {
const err = new ProcessingError(500, {
type: "internal",
code: "internal-error",
});
expect(err).toBeInstanceOf(Error);
});
});
describe("throwValidation", () => {
it("throws ProcessingError with status 400", () => {
try {
throwValidation("bad-input", "invalid value");
expect.fail("should have thrown");
} catch (err) {
expect(err).toBeInstanceOf(ProcessingError);
const pe = err as ProcessingError;
expect(pe.statusCode).toBe(400);
expect(pe.errorBody.type).toBe("validation");
expect(pe.errorBody.code).toBe("bad-input");
expect(pe.errorBody.hint).toBe("invalid value");
}
});
it("works without hint", () => {
try {
throwValidation("bad-input");
expect.fail("should have thrown");
} catch (err) {
expect(err).toBeInstanceOf(ProcessingError);
const pe = err as ProcessingError;
expect(pe.errorBody.hint).toBeUndefined();
}
});
});
describe("throwRestriction", () => {
it("throws ProcessingError with status 413", () => {
try {
throwRestriction("too-large", "file exceeds limit");
expect.fail("should have thrown");
} catch (err) {
expect(err).toBeInstanceOf(ProcessingError);
const pe = err as ProcessingError;
expect(pe.statusCode).toBe(413);
expect(pe.errorBody.type).toBe("restriction");
expect(pe.errorBody.code).toBe("too-large");
expect(pe.errorBody.hint).toBe("file exceeds limit");
}
});
it("works without hint", () => {
try {
throwRestriction("too-large");
expect.fail("should have thrown");
} catch (err) {
expect(err).toBeInstanceOf(ProcessingError);
const pe = err as ProcessingError;
expect(pe.errorBody.hint).toBeUndefined();
}
});
});
describe("errorHandler", () => {
let res: ReturnType<typeof mockRes>;
let next: NextFunction;
beforeEach(() => {
res = mockRes();
next = vi.fn();
});
it("handles ProcessingError (400 validation)", () => {
const err = new ProcessingError(400, {
type: "validation",
code: "bad-input",
hint: "invalid value",
});
errorHandler(err, mockReq(), res, next);
expect(res.status).toHaveBeenCalledWith(400);
expect(res.json).toHaveBeenCalledWith({
type: "validation",
code: "bad-input",
hint: "invalid value",
});
});
it("handles ProcessingError (413 restriction)", () => {
const err = new ProcessingError(413, {
type: "restriction",
code: "payload-too-large",
});
errorHandler(err, mockReq(), res, next);
expect(res.status).toHaveBeenCalledWith(413);
expect(res.json).toHaveBeenCalledWith({
type: "restriction",
code: "payload-too-large",
});
});
it("handles MulterError LIMIT_FILE_SIZE as 413", () => {
const err = new multer.MulterError("LIMIT_FILE_SIZE");
errorHandler(err, mockReq(), res, next);
expect(res.status).toHaveBeenCalledWith(413);
expect(res.json).toHaveBeenCalledWith({
type: "restriction",
code: "payload-too-large",
});
});
it("handles generic Error as 500", () => {
const err = new Error("something broke");
errorHandler(err, mockReq(), res, next);
expect(res.status).toHaveBeenCalledWith(500);
expect(res.json).toHaveBeenCalledWith({
type: "internal",
code: "processing-error",
hint: "Internal server error",
});
});
it("handles Error with empty message", () => {
const err = new Error("");
errorHandler(err, mockReq(), res, next);
expect(res.status).toHaveBeenCalledWith(500);
expect(res.json).toHaveBeenCalledWith({
type: "internal",
code: "processing-error",
hint: "Internal server error",
});
});
it("does not write response if headers already sent", () => {
const err = new ProcessingError(400, {
type: "validation",
code: "bad-input",
hint: "invalid value",
});
const resWithHeadersSent = {
...res,
headersSent: true,
};
errorHandler(err, mockReq(), resWithHeadersSent, next);
expect(resWithHeadersSent.status).not.toHaveBeenCalled();
expect(resWithHeadersSent.json).not.toHaveBeenCalled();
});
it("calls releaseQueue for ProcessingError", () => {
const releaseQueue = vi.fn();
const resWithLocals = { ...res, locals: { releaseQueue } } as any;
const err = new ProcessingError(400, { type: "validation", code: "test" });
errorHandler(err, mockReq(), resWithLocals, next);
expect(releaseQueue).toHaveBeenCalled();
});
it("calls releaseQueue for MulterError LIMIT_FILE_SIZE", () => {
const releaseQueue = vi.fn();
const resWithLocals = { ...res, locals: { releaseQueue } } as any;
const err = new multer.MulterError("LIMIT_FILE_SIZE");
errorHandler(err, mockReq(), resWithLocals, next);
expect(releaseQueue).toHaveBeenCalled();
});
it("calls releaseQueue for generic Error", () => {
const releaseQueue = vi.fn();
const resWithLocals = { ...res, locals: { releaseQueue } } as any;
const err = new Error("something broke");
errorHandler(err, mockReq(), resWithLocals, next);
expect(releaseQueue).toHaveBeenCalled();
});
it("does not throw when releaseQueue is not set", () => {
const resWithNoLocals = { ...res, locals: {} } as any;
const err = new ProcessingError(400, { type: "validation", code: "test" });
expect(() => errorHandler(err, mockReq(), resWithNoLocals, next)).not.toThrow();
});
});
describe("sharedKeyAuth", () => {
let res: ReturnType<typeof mockRes>;
let next: NextFunction;
beforeEach(() => {
res = mockRes();
next = vi.fn();
});
it("returns 403 when expectedKey is null", () => {
const middleware = sharedKeyAuth(null);
const req = { headers: {} } as unknown as Request;
middleware(req, res, next);
expect(res.status).toHaveBeenCalledWith(403);
expect(res.json).toHaveBeenCalledWith({
type: "authorization",
code: "forbidden",
hint: "Shared key not configured",
});
expect(next).not.toHaveBeenCalled();
});
it("returns 403 when expectedKey is null regardless of NODE_ENV", () => {
const originalEnv = process.env.NODE_ENV;
delete process.env.NODE_ENV;
try {
const middleware = sharedKeyAuth(null);
const req = { headers: {} } as unknown as Request;
middleware(req, res, next);
expect(res.status).toHaveBeenCalledWith(403);
expect(next).not.toHaveBeenCalled();
} finally {
process.env.NODE_ENV = originalEnv;
}
});
it("passes through with correct key", () => {
const middleware = sharedKeyAuth("test-key");
const req = { headers: { "x-shared-key": "test-key" } } as unknown as Request;
middleware(req, res, next);
expect(next).toHaveBeenCalled();
expect(res.status).not.toHaveBeenCalled();
});
it("returns 403 with wrong key", () => {
const middleware = sharedKeyAuth("test-key");
const req = { headers: { "x-shared-key": "wrong-key" } } as unknown as Request;
middleware(req, res, next);
expect(res.status).toHaveBeenCalledWith(403);
expect(res.json).toHaveBeenCalledWith({ type: "authorization", code: "forbidden" });
expect(next).not.toHaveBeenCalled();
});
it("returns 403 with missing header", () => {
const middleware = sharedKeyAuth("test-key");
const req = { headers: {} } as unknown as Request;
middleware(req, res, next);
expect(res.status).toHaveBeenCalledWith(403);
expect(res.json).toHaveBeenCalledWith({ type: "authorization", code: "forbidden" });
expect(next).not.toHaveBeenCalled();
});
it("returns 403 with undefined header value", () => {
const middleware = sharedKeyAuth("test-key");
const req = { headers: { "x-shared-key": undefined } } as unknown as Request;
middleware(req, res, next);
expect(res.status).toHaveBeenCalledWith(403);
expect(next).not.toHaveBeenCalled();
});
it("returns 403 when key is null and NODE_ENV is production", () => {
const originalEnv = process.env.NODE_ENV;
process.env.NODE_ENV = "production";
try {
const middleware = sharedKeyAuth(null);
const req = { headers: {} } as unknown as Request;
middleware(req, res, next);
expect(res.status).toHaveBeenCalledWith(403);
expect(res.json).toHaveBeenCalledWith({
type: "authorization",
code: "forbidden",
hint: "Shared key not configured",
});
expect(next).not.toHaveBeenCalled();
} finally {
process.env.NODE_ENV = originalEnv;
}
});
it("returns 403 for multibyte Unicode with same string length but different byte length", () => {
const middleware = sharedKeyAuth("test-key");
const req = { headers: { "x-shared-key": "test-ké" } } as unknown as Request;
middleware(req, res, next);
expect(res.status).toHaveBeenCalledWith(403);
expect(next).not.toHaveBeenCalled();
});
it("returns 403 for emoji input (multibyte)", () => {
const middleware = sharedKeyAuth("test-key");
const req = { headers: { "x-shared-key": "test-k🔑" } } as unknown as Request;
middleware(req, res, next);
expect(res.status).toHaveBeenCalledWith(403);
expect(next).not.toHaveBeenCalled();
});
it("returns 403 for accented characters with same string length", () => {
const middleware = sharedKeyAuth("abcdefgh");
const req = { headers: { "x-shared-key": "ábcdefgh" } } as unknown as Request;
middleware(req, res, next);
expect(res.status).toHaveBeenCalledWith(403);
expect(next).not.toHaveBeenCalled();
});
});
describe("timeoutMiddleware", () => {
it("calls next() immediately", () => {
const middleware = timeoutMiddleware(1000);
const req = new EventEmitter() as unknown as Request;
const res = new EventEmitter() as unknown as Response;
const next = vi.fn();
middleware(req, res, next);
expect(next).toHaveBeenCalled();
});
it("clears timer when response finishes", async () => {
vi.useFakeTimers();
const middleware = timeoutMiddleware(1000);
const req = new EventEmitter() as unknown as Request;
const res = new EventEmitter() as unknown as Response;
const next = vi.fn();
middleware(req, res, next);
res.emit("finish");
// Advance past timeout - should not throw
vi.advanceTimersByTime(2000);
vi.useRealTimers();
});
it("clears timer when response closes", async () => {
vi.useFakeTimers();
const middleware = timeoutMiddleware(1000);
const req = new EventEmitter() as unknown as Request;
const res = new EventEmitter() as unknown as Response;
const next = vi.fn();
middleware(req, res, next);
res.emit("close");
// Advance past timeout - should not throw
vi.advanceTimersByTime(2000);
vi.useRealTimers();
});
it("sends 504 response when timeout expires before response", async () => {
vi.useFakeTimers();
const middleware = timeoutMiddleware(100);
const req = new EventEmitter() as unknown as Request;
(req as any).destroy = vi.fn();
const res = new EventEmitter() as unknown as Response;
(res as any).headersSent = false;
(res as any).status = vi.fn().mockReturnThis();
(res as any).json = vi.fn().mockReturnThis();
const next = vi.fn();
middleware(req, res, next);
vi.advanceTimersByTime(150);
expect(res.status).toHaveBeenCalledWith(504);
expect(res.json).toHaveBeenCalledWith({
type: "internal",
code: "processing-timeout",
hint: "Request timed out",
});
vi.useRealTimers();
});
it("does not send response if headers already sent", async () => {
vi.useFakeTimers();
const middleware = timeoutMiddleware(100);
const req = new EventEmitter() as unknown as Request;
(req as any).destroy = vi.fn();
const res = new EventEmitter() as unknown as Response;
(res as any).headersSent = true;
(res as any).status = vi.fn().mockReturnThis();
(res as any).json = vi.fn().mockReturnThis();
const next = vi.fn();
middleware(req, res, next);
vi.advanceTimersByTime(150);
expect(res.status).not.toHaveBeenCalled();
vi.useRealTimers();
});
it("destroys request AFTER response finishes (not before)", async () => {
vi.useFakeTimers();
const middleware = timeoutMiddleware(100);
const req = new EventEmitter() as unknown as Request;
(req as any).destroy = vi.fn();
const res = new EventEmitter() as unknown as Response;
(res as any).headersSent = false;
(res as any).status = vi.fn().mockReturnThis();
(res as any).json = vi.fn().mockReturnThis();
const next = vi.fn();
middleware(req, res, next);
// Advance to timeout - this triggers the 504 response
vi.advanceTimersByTime(100);
// req.destroy should NOT be called yet (response not finished)
expect(req.destroy).not.toHaveBeenCalled();
expect(res.status).toHaveBeenCalledWith(504);
// Now simulate response finishing
res.emit("finish");
// Now req.destroy should be called
expect(req.destroy).toHaveBeenCalled();
vi.useRealTimers();
});
it("creates AbortController and attaches to request", () => {
const middleware = timeoutMiddleware(1000);
const req = new EventEmitter() as unknown as Request;
const res = new EventEmitter() as unknown as Response;
const next = vi.fn();
middleware(req, res, next);
expect((req as any).abortController).toBeDefined();
expect((req as any).abortController.signal).toBeDefined();
expect((req as any).abortController.signal.aborted).toBe(false);
});
it("aborts signal when timeout fires", async () => {
vi.useFakeTimers();
const middleware = timeoutMiddleware(100);
const req = new EventEmitter() as unknown as Request;
(req as any).destroy = vi.fn();
const res = new EventEmitter() as unknown as Response;
(res as any).headersSent = false;
(res as any).status = vi.fn().mockReturnThis();
(res as any).json = vi.fn().mockReturnThis();
const next = vi.fn();
middleware(req, res, next);
// Signal should not be aborted yet
expect((req as any).abortController.signal.aborted).toBe(false);
// Advance to timeout
vi.advanceTimersByTime(150);
// Signal should now be aborted
expect((req as any).abortController.signal.aborted).toBe(true);
vi.useRealTimers();
});
it("does not abort signal when response finishes before timeout", async () => {
vi.useFakeTimers();
const middleware = timeoutMiddleware(1000);
const req = new EventEmitter() as unknown as Request;
const res = new EventEmitter() as unknown as Response;
const next = vi.fn();
middleware(req, res, next);
// Response finishes before timeout
res.emit("finish");
// Advance past timeout
vi.advanceTimersByTime(2000);
// Signal should NOT be aborted (timer was cleared)
expect((req as any).abortController.signal.aborted).toBe(false);
vi.useRealTimers();
});
it("aborts signal when response closes (client disconnect)", async () => {
vi.useFakeTimers();
const middleware = timeoutMiddleware(1000);
const req = new EventEmitter() as unknown as Request;
const res = new EventEmitter() as unknown as Response;
const next = vi.fn();
middleware(req, res, next);
// Signal should not be aborted initially
expect((req as any).abortController.signal.aborted).toBe(false);
// Simulate client disconnect (response closes)
res.emit("close");
// Signal should now be aborted
expect((req as any).abortController.signal.aborted).toBe(true);
vi.useRealTimers();
});
it("does not abort signal again if already aborted when response closes", async () => {
vi.useFakeTimers();
const middleware = timeoutMiddleware(100);
const req = new EventEmitter() as unknown as Request;
const res = new EventEmitter() as unknown as Response;
(res as any).headersSent = false;
(res as any).status = vi.fn().mockReturnValue({ json: vi.fn() });
const next = vi.fn();
middleware(req, res, next);
// Advance past timeout to trigger abort
vi.advanceTimersByTime(200);
// Signal should be aborted from timeout
expect((req as any).abortController.signal.aborted).toBe(true);
// Simulate client disconnect (response closes)
res.emit("close");
// Signal should still be aborted (no error thrown)
expect((req as any).abortController.signal.aborted).toBe(true);
vi.useRealTimers();
});
});
describe("cleanupMiddleware", () => {
it("calls next() immediately", () => {
const req = {} as Request;
const res = new EventEmitter() as unknown as Response;
const next = vi.fn();
cleanupMiddleware(req, res, next);
expect(next).toHaveBeenCalled();
});
it("removes file on response finish", async () => {
const req = {
file: {
path: "/tmp/test-file.jpg",
},
} as unknown as Request;
const res = new EventEmitter() as unknown as Response;
const next = vi.fn();
// Mock the rm function by spying on the cleanup behavior
// We'll verify the middleware registers the finish handler
cleanupMiddleware(req, res, next);
// Emit finish event - this should trigger cleanup
// The actual rm is mocked internally, so we just verify no errors
res.emit("finish");
// Wait for async cleanup
await new Promise((resolve) => setTimeout(resolve, 10));
});
it("removes file on response close", async () => {
const req = {
file: {
path: "/tmp/test-file.jpg",
},
} as unknown as Request;
const res = new EventEmitter() as unknown as Response;
const next = vi.fn();
cleanupMiddleware(req, res, next);
// Emit close event - this should trigger cleanup
res.emit("close");
// Wait for async cleanup
await new Promise((resolve) => setTimeout(resolve, 10));
});
it("does nothing when req.file is undefined", async () => {
const req = {} as Request;
const res = new EventEmitter() as unknown as Response;
const next = vi.fn();
cleanupMiddleware(req, res, next);
// Emit finish event - should not throw
res.emit("finish");
// Wait for async cleanup
await new Promise((resolve) => setTimeout(resolve, 10));
});
it("does nothing when req.file.path is undefined", async () => {
const req = {
file: {
buffer: Buffer.from("test"),
},
} as unknown as Request;
const res = new EventEmitter() as unknown as Response;
const next = vi.fn();
cleanupMiddleware(req, res, next);
// Emit finish event - should not throw
res.emit("finish");
// Wait for async cleanup
await new Promise((resolve) => setTimeout(resolve, 10));
});
it("doesn't throw when file doesn't exist", async () => {
const req = {
file: {
path: "/tmp/nonexistent-file.jpg",
},
} as unknown as Request;
const res = new EventEmitter() as unknown as Response;
const next = vi.fn();
cleanupMiddleware(req, res, next);
// Emit finish event - should not throw even if file doesn't exist
res.emit("finish");
// Wait for async cleanup
await new Promise((resolve) => setTimeout(resolve, 10));
});
});

377
media-processor/test/queue.test.ts Normal file
View File

@ -0,0 +1,377 @@
import { describe, it, expect, vi, beforeEach } from "vitest";
import { createQueueMiddleware } from "../src/middleware/queue.js";
import type { Request, Response, NextFunction } from "express";
import { EventEmitter } from "node:events";
function mockRes() {
const res = new EventEmitter() as any;
res.status = vi.fn().mockReturnThis();
res.json = vi.fn().mockReturnThis();
res.send = vi.fn().mockReturnThis();
res.headersSent = false;
res.writableEnded = false;
return res as Response;
}
function mockReq() {
return {} as Request;
}
describe("queueMiddleware", () => {
it("calls next() when queue has capacity", async () => {
const middleware = createQueueMiddleware(1);
const req = mockReq();
const res = mockRes();
const next = vi.fn();
middleware(req, res, next);
expect(next).toHaveBeenCalled();
});
it("skips next() when res.writableEnded is true (timeout already sent)", async () => {
const middleware = createQueueMiddleware(1);
const req = mockReq();
const res = mockRes();
(res as any).writableEnded = true;
const next = vi.fn();
middleware(req, res, next);
// next() should NOT be called because response already ended
expect(next).not.toHaveBeenCalled();
});
it("queues requests when concurrency limit reached", async () => {
const middleware = createQueueMiddleware(1);
const req1 = mockReq();
const res1 = mockRes();
const next1 = vi.fn();
const req2 = mockReq();
const res2 = mockRes();
const next2 = vi.fn();
// First request takes the slot
middleware(req1, res1, next1);
expect(next1).toHaveBeenCalled();
// Second request should queue
middleware(req2, res2, next2);
expect(next2).not.toHaveBeenCalled();
// Release first request's queue slot (simulating processing completion)
const releaseQueue = (res1 as any).locals?.releaseQueue;
expect(releaseQueue).toBeDefined();
releaseQueue();
// Now second request should proceed
await new Promise((resolve) => setTimeout(resolve, 10));
expect(next2).toHaveBeenCalled();
});
it("resolves promise when releaseQueue is called", async () => {
const middleware = createQueueMiddleware(1);
const req1 = mockReq();
const res1 = mockRes();
const next1 = vi.fn();
const req2 = mockReq();
const res2 = mockRes();
const next2 = vi.fn();
middleware(req1, res1, next1);
middleware(req2, res2, next2);
// Release first request's queue slot
const releaseQueue = (res1 as any).locals?.releaseQueue;
expect(releaseQueue).toBeDefined();
releaseQueue();
await new Promise((resolve) => setTimeout(resolve, 10));
expect(next2).toHaveBeenCalled();
});
it("processes requests sequentially with concurrency 1", async () => {
const middleware = createQueueMiddleware(1);
const order: number[] = [];
const req1 = mockReq();
const res1 = mockRes();
const next1 = vi.fn(() => order.push(1));
const req2 = mockReq();
const res2 = mockRes();
const next2 = vi.fn(() => order.push(2));
const req3 = mockReq();
const res3 = mockRes();
const next3 = vi.fn(() => order.push(3));
middleware(req1, res1, next1);
middleware(req2, res2, next2);
middleware(req3, res3, next3);
// Only first should be called immediately
expect(next1).toHaveBeenCalled();
expect(next2).not.toHaveBeenCalled();
expect(next3).not.toHaveBeenCalled();
// Release first request's queue slot
const releaseQueue1 = (res1 as any).locals?.releaseQueue;
expect(releaseQueue1).toBeDefined();
releaseQueue1();
await new Promise((resolve) => setTimeout(resolve, 10));
// Now second should be called
expect(next2).toHaveBeenCalled();
expect(next3).not.toHaveBeenCalled();
// Release second request's queue slot
const releaseQueue2 = (res2 as any).locals?.releaseQueue;
expect(releaseQueue2).toBeDefined();
releaseQueue2();
await new Promise((resolve) => setTimeout(resolve, 10));
// Now third should be called
expect(next3).toHaveBeenCalled();
// Verify sequential order
expect(order).toEqual([1, 2, 3]);
});
it("processes requests in parallel with concurrency 10", async () => {
const middleware = createQueueMiddleware(10);
const calls: number[] = [];
// Create 5 requests (less than concurrency limit)
const requests = Array.from({ length: 5 }, (_, i) => {
const req = mockReq();
const res = mockRes();
const next = vi.fn(() => calls.push(i));
return { req, res, next };
});
// All should be called immediately
requests.forEach(({ req, res, next }) => {
middleware(req, res, next);
});
// All 5 should be called immediately since concurrency is 10
expect(calls.length).toBe(5);
expect(calls).toEqual([0, 1, 2, 3, 4]);
});
it("handles request errors gracefully", async () => {
const middleware = createQueueMiddleware(1);
const req1 = mockReq();
const res1 = mockRes();
const next1 = vi.fn();
const req2 = mockReq();
const res2 = mockRes();
const next2 = vi.fn();
middleware(req1, res1, next1);
middleware(req2, res2, next2);
// Simulate error by releasing queue slot (as would happen in finally block)
const releaseQueue = (res1 as any).locals?.releaseQueue;
expect(releaseQueue).toBeDefined();
releaseQueue();
await new Promise((resolve) => setTimeout(resolve, 10));
// Second request should still proceed even after first "error"
expect(next2).toHaveBeenCalled();
});
it("doesn't block on slow requests within concurrency limit", async () => {
const middleware = createQueueMiddleware(2);
const calls: number[] = [];
const req1 = mockReq();
const res1 = mockRes();
const next1 = vi.fn(() => calls.push(1));
const req2 = mockReq();
const res2 = mockRes();
const next2 = vi.fn(() => calls.push(2));
const req3 = mockReq();
const res3 = mockRes();
const next3 = vi.fn(() => calls.push(3));
// Start first two requests (concurrency is 2)
middleware(req1, res1, next1);
middleware(req2, res2, next2);
// Both should be called immediately
expect(next1).toHaveBeenCalled();
expect(next2).toHaveBeenCalled();
expect(next3).not.toHaveBeenCalled();
// Third request should wait
middleware(req3, res3, next3);
expect(next3).not.toHaveBeenCalled();
// Release first request's queue slot
const releaseQueue1 = (res1 as any).locals?.releaseQueue;
expect(releaseQueue1).toBeDefined();
releaseQueue1();
await new Promise((resolve) => setTimeout(resolve, 10));
// Now third should proceed
expect(next3).toHaveBeenCalled();
});
it("releases slot when error handler calls releaseQueue (covers Multer error path)", async () => {
// This test verifies that the error handler releases the queue slot
// by calling releaseQueue from res.locals. This covers the Multer error
// case where the route handler never runs.
const middleware = createQueueMiddleware(1);
const req1 = mockReq();
const res1 = mockRes();
const next1 = vi.fn();
const req2 = mockReq();
const res2 = mockRes();
const next2 = vi.fn();
middleware(req1, res1, next1);
middleware(req2, res2, next2);
// First request is processing, second is queued
expect(next1).toHaveBeenCalled();
expect(next2).not.toHaveBeenCalled();
// Simulate error handler calling releaseQueue (e.g., Multer error)
const releaseQueue = (res1 as any).locals.releaseQueue;
releaseQueue();
await new Promise((resolve) => setTimeout(resolve, 10));
// Second request should proceed because slot was released
expect(next2).toHaveBeenCalled();
});
it("releases slot when releaseQueue callback is called", async () => {
const middleware = createQueueMiddleware(1);
const req1 = mockReq();
const res1 = mockRes();
const next1 = vi.fn();
const req2 = mockReq();
const res2 = mockRes();
const next2 = vi.fn();
middleware(req1, res1, next1);
middleware(req2, res2, next2);
// First request is processing, second is queued
expect(next1).toHaveBeenCalled();
expect(next2).not.toHaveBeenCalled();
// Simulate processing completing by calling releaseQueue
const releaseQueue = (res1 as any).locals?.releaseQueue;
expect(releaseQueue).toBeDefined();
releaseQueue();
await new Promise((resolve) => setTimeout(resolve, 10));
// Now second request should proceed
expect(next2).toHaveBeenCalled();
});
it("releases slot on processing error (via finally block)", async () => {
const middleware = createQueueMiddleware(1);
const req1 = mockReq();
const res1 = mockRes();
const next1 = vi.fn();
const req2 = mockReq();
const res2 = mockRes();
const next2 = vi.fn();
middleware(req1, res1, next1);
middleware(req2, res2, next2);
// First request is processing, second is queued
expect(next1).toHaveBeenCalled();
expect(next2).not.toHaveBeenCalled();
// Simulate processing error and release in finally block
const releaseQueue = (res1 as any).locals?.releaseQueue;
expect(releaseQueue).toBeDefined();
releaseQueue();
await new Promise((resolve) => setTimeout(resolve, 10));
// Second request should proceed even after error
expect(next2).toHaveBeenCalled();
});
it("releaseQueue is idempotent (can be called multiple times)", async () => {
const middleware = createQueueMiddleware(1);
const req1 = mockReq();
const res1 = mockRes();
const next1 = vi.fn();
const req2 = mockReq();
const res2 = mockRes();
const next2 = vi.fn();
middleware(req1, res1, next1);
middleware(req2, res2, next2);
// First request is processing, second is queued
expect(next1).toHaveBeenCalled();
expect(next2).not.toHaveBeenCalled();
// Call releaseQueue multiple times
const releaseQueue = (res1 as any).locals?.releaseQueue;
expect(releaseQueue).toBeDefined();
releaseQueue();
releaseQueue(); // Should not throw or cause issues
releaseQueue();
await new Promise((resolve) => setTimeout(resolve, 10));
// Second request should proceed
expect(next2).toHaveBeenCalled();
});
it("holds queue slot when client disconnects (close event)", async () => {
const middleware = createQueueMiddleware(1);
const req1 = mockReq();
const res1 = mockRes();
const next1 = vi.fn();
const req2 = mockReq();
const res2 = mockRes();
const next2 = vi.fn();
middleware(req1, res1, next1);
middleware(req2, res2, next2);
// First request is processing, second is queued
expect(next1).toHaveBeenCalled();
expect(next2).not.toHaveBeenCalled();
// Simulate client disconnect (close event)
res1.emit("close");
await new Promise((resolve) => setTimeout(resolve, 10));
// Second request should NOT proceed because slot is still held
expect(next2).not.toHaveBeenCalled();
// Now release the slot (simulating processing completion)
const releaseQueue = (res1 as any).locals?.releaseQueue;
expect(releaseQueue).toBeDefined();
releaseQueue();
await new Promise((resolve) => setTimeout(resolve, 10));
// Now second request should proceed
expect(next2).toHaveBeenCalled();
});
});

296
media-processor/test/routes-integration.test.ts Normal file
View File

@ -0,0 +1,296 @@
import { describe, it, expect, beforeAll, afterAll } from "vitest";
import request from "supertest";
import express from "express";
import { createImageRoutes } from "../src/routes/image.js";
import { createFontRoutes } from "../src/routes/font.js";
import { errorHandler } from "../src/middleware/error-handler.js";
import { timeoutMiddleware } from "../src/middleware/timeout.js";
import { sharedKeyAuth } from "../src/middleware/auth.js";
import { createQueueMiddleware } from "../src/middleware/queue.js";
import { configureImageLimits } from "../src/services/image.js";
import { configureFontLimits } from "../src/services/font.js";
import { configureUploadLimits } from "../src/upload.js";
import sharp from "sharp";
import { readdir, rm, stat } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
// Test app with low memoryThreshold to force disk storage
function createTestApp() {
const app = express();
// Configure with low threshold to force disk storage
configureImageLimits({
maxPixels: 128_000_000,
maxWidth: 16384,
maxHeight: 16384,
});
configureFontLimits({
mem: 1024 * 1024 * 512,
cpuTime: 30,
timeout: 30,
});
// Very low threshold to force disk storage for small files
configureUploadLimits({ maxFileSize: 10 * 1024 * 1024, memoryThreshold: 10 });
const queueMiddleware = createQueueMiddleware(10);
app.use(timeoutMiddleware(5000));
app.use("/api/image", sharedKeyAuth("test-key"), queueMiddleware, createImageRoutes());
app.use("/api/font", sharedKeyAuth("test-key"), queueMiddleware, createFontRoutes());
app.use(errorHandler);
return app;
}
describe("HTTP upload cleanup", () => {
let app: ReturnType<typeof createTestApp>;
beforeAll(() => {
app = createTestApp();
});
async function getTempFiles(): Promise<string[]> {
const tmp = tmpdir();
const files = await readdir(tmp);
const uploadDirs = files.filter((f) => f.startsWith("penpot.upload."));
// Get all files inside upload directories
const allFiles: string[] = [];
for (const dir of uploadDirs) {
try {
const dirPath = join(tmp, dir);
const dirFiles = await readdir(dirPath);
allFiles.push(...dirFiles.map((f) => join(dir, f)));
} catch {
// Directory might not exist or be inaccessible
}
}
return allFiles;
}
it("removes disk-backed file after successful image/info request", async () => {
const beforeFiles = await getTempFiles();
// Create a small image
const imageBuffer = await sharp({
create: { width: 100, height: 100, channels: 3, background: { r: 255, g: 0, b: 0 } },
})
.jpeg()
.toBuffer();
const response = await request(app)
.post("/api/image/info")
.set("x-shared-key", "test-key")
.attach("file", imageBuffer, { filename: "test.jpg", contentType: "image/jpeg" });
expect(response.status).toBe(200);
expect(response.body.width).toBe(100);
expect(response.body.height).toBe(100);
// Wait for cleanup
await new Promise((resolve) => setTimeout(resolve, 100));
const afterFiles = await getTempFiles();
// No new temp files should remain
const newFiles = afterFiles.filter((f) => !beforeFiles.includes(f));
expect(newFiles.length).toBe(0);
});
it("removes disk-backed file after successful image/thumbnail request", async () => {
const beforeFiles = await getTempFiles();
const imageBuffer = await sharp({
create: { width: 200, height: 200, channels: 3, background: { r: 0, g: 255, b: 0 } },
})
.png()
.toBuffer();
const response = await request(app)
.post("/api/image/thumbnail?width=100&height=100&format=jpeg&mode=fit")
.set("x-shared-key", "test-key")
.attach("file", imageBuffer, { filename: "test.png", contentType: "image/png" });
expect(response.status).toBe(200);
expect(response.headers["content-type"]).toMatch(/image\/jpeg/);
// Wait for cleanup
await new Promise((resolve) => setTimeout(resolve, 100));
const afterFiles = await getTempFiles();
const newFiles = afterFiles.filter((f) => !beforeFiles.includes(f));
expect(newFiles.length).toBe(0);
});
it("removes disk-backed file after successful font/convert request", async () => {
const beforeFiles = await getTempFiles();
// Create a minimal TTF font (this is a simplified test - in reality you'd use a real font)
// For this test, we'll just verify the cleanup happens even if the conversion fails
const fontBuffer = Buffer.from("not a real font");
const response = await request(app)
.post("/api/font/convert?target-type=font/woff")
.set("x-shared-key", "test-key")
.attach("file", fontBuffer, { filename: "test.ttf", contentType: "font/ttf" });
// The conversion will fail, but cleanup should still happen
// We expect either 400 (invalid font) or 500 (processing error)
expect([400, 500]).toContain(response.status);
// Wait for cleanup
await new Promise((resolve) => setTimeout(resolve, 100));
const afterFiles = await getTempFiles();
const newFiles = afterFiles.filter((f) => !beforeFiles.includes(f));
expect(newFiles.length).toBe(0);
});
it("removes disk-backed file after failed request", async () => {
const beforeFiles = await getTempFiles();
// Send invalid image data
const invalidBuffer = Buffer.from("not an image");
const response = await request(app)
.post("/api/image/info")
.set("x-shared-key", "test-key")
.attach("file", invalidBuffer, { filename: "invalid.jpg", contentType: "image/jpeg" });
expect(response.status).toBe(400);
// Wait for cleanup
await new Promise((resolve) => setTimeout(resolve, 100));
const afterFiles = await getTempFiles();
const newFiles = afterFiles.filter((f) => !beforeFiles.includes(f));
expect(newFiles.length).toBe(0);
});
it("removes disk-backed file after timeout", async () => {
// Create a test app with very short timeout
const timeoutApp = express();
configureImageLimits({ maxPixels: 128_000_000, maxWidth: 16384, maxHeight: 16384 });
configureUploadLimits({ maxFileSize: 10 * 1024 * 1024, memoryThreshold: 10 });
const queueMiddleware = createQueueMiddleware(10);
timeoutApp.use(timeoutMiddleware(10)); // 10ms timeout - very aggressive
timeoutApp.use("/api/image", sharedKeyAuth("test-key"), queueMiddleware, createImageRoutes());
timeoutApp.use(errorHandler);
const beforeFiles = await getTempFiles();
// Create a large image that will take time to process
const imageBuffer = await sharp({
create: { width: 4000, height: 4000, channels: 3, background: { r: 255, g: 0, b: 0 } },
})
.jpeg({ quality: 100 })
.toBuffer();
const response = await request(timeoutApp)
.post("/api/image/thumbnail?width=2000&height=2000&format=jpeg&mode=fit")
.set("x-shared-key", "test-key")
.attach("file", imageBuffer, { filename: "large.jpg", contentType: "image/jpeg" });
// Should timeout
expect(response.status).toBe(504);
expect(response.body.type).toBe("internal");
expect(response.body.code).toBe("processing-timeout");
// Wait for processing to settle (Sharp may still be working in background)
await new Promise((resolve) => setTimeout(resolve, 2000));
const afterFiles = await getTempFiles();
const newFiles = afterFiles.filter((f) => !beforeFiles.includes(f));
expect(newFiles.length).toBe(0);
});
});
describe("HTTP malformed image handling", () => {
let app: ReturnType<typeof createTestApp>;
beforeAll(() => {
app = createTestApp();
});
it("returns 400 for corrupted image in /api/image/info", async () => {
// Create a valid image then truncate it
const validBuffer = await sharp({
create: { width: 100, height: 100, channels: 3, background: { r: 255, g: 0, b: 0 } },
})
.jpeg()
.toBuffer();
const corruptedBuffer = validBuffer.subarray(0, Math.floor(validBuffer.length / 2));
const response = await request(app)
.post("/api/image/info")
.set("x-shared-key", "test-key")
.attach("file", corruptedBuffer, { filename: "corrupted.jpg", contentType: "image/jpeg" });
expect(response.status).toBe(400);
expect(response.body.type).toBe("validation");
expect(response.body.code).toBe("invalid-image");
});
it("returns 400 for corrupted image in /api/image/thumbnail", async () => {
const validBuffer = await sharp({
create: { width: 100, height: 100, channels: 3, background: { r: 255, g: 0, b: 0 } },
})
.jpeg()
.toBuffer();
const corruptedBuffer = validBuffer.subarray(0, Math.floor(validBuffer.length / 2));
const response = await request(app)
.post("/api/image/thumbnail?width=50&height=50&format=jpeg&mode=fit")
.set("x-shared-key", "test-key")
.attach("file", corruptedBuffer, { filename: "corrupted.jpg", contentType: "image/jpeg" });
expect(response.status).toBe(400);
expect(response.body.type).toBe("validation");
expect(response.body.code).toBe("invalid-image");
});
});
describe("HTTP quality parameter clamping", () => {
let app: ReturnType<typeof createTestApp>;
beforeAll(() => {
app = createTestApp();
});
it("clamps quality=0 to 1 at route level", async () => {
const imageBuffer = await sharp({
create: { width: 100, height: 100, channels: 3, background: { r: 255, g: 0, b: 0 } },
})
.jpeg()
.toBuffer();
const response = await request(app)
.post("/api/image/thumbnail?width=50&height=50&quality=0&format=jpeg&mode=fit")
.set("x-shared-key", "test-key")
.attach("file", imageBuffer, { filename: "test.jpg", contentType: "image/jpeg" });
expect(response.status).toBe(200);
expect(response.headers["content-type"]).toMatch(/image\/jpeg/);
});
it("clamps quality=101 to 100 at route level", async () => {
const imageBuffer = await sharp({
create: { width: 100, height: 100, channels: 3, background: { r: 255, g: 0, b: 0 } },
})
.jpeg()
.toBuffer();
const response = await request(app)
.post("/api/image/thumbnail?width=50&height=50&quality=101&format=jpeg&mode=fit")
.set("x-shared-key", "test-key")
.attach("file", imageBuffer, { filename: "test.jpg", contentType: "image/jpeg" });
expect(response.status).toBe(200);
expect(response.headers["content-type"]).toMatch(/image\/jpeg/);
});
});

1
media-processor/test/setup.ts Normal file
View File

@ -0,0 +1 @@
process.env.PENPOT_MEDIA_PROCESSOR_LOG_LEVEL = "silent";

212
media-processor/test/upload-storage.test.ts Normal file
View File

@ -0,0 +1,212 @@
import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
import { createHybridStorage } from "../src/upload-storage.js";
import type { Request } from "express";
import { Readable } from "node:stream";
import { rm } from "node:fs/promises";
function mockReq(contentLength?: string): Request {
const headers: Record<string, string> = {};
if (contentLength !== undefined) {
headers["content-length"] = contentLength;
}
return { headers } as Request;
}
function mockFile(content: string = "test content") {
const stream = Readable.from([content]);
return {
fieldname: "file",
originalname: "test.txt",
encoding: "7bit",
mimetype: "text/plain",
stream,
} as Express.Multer.File;
}
describe("createHybridStorage", () => {
let storage: ReturnType<typeof createHybridStorage>;
let tempDirs: string[] = [];
beforeEach(() => {
storage = createHybridStorage({ memoryThreshold: 1024 });
});
afterEach(async () => {
for (const dir of tempDirs) {
await rm(dir, { recursive: true, force: true });
}
tempDirs = [];
});
it("uses memory storage when Content-Length is below threshold", async () => {
const req = mockReq("100");
const file = mockFile("small content");
await new Promise<void>((resolve, reject) => {
storage._handleFile(req, file, (err, info) => {
if (err) reject(err);
else {
expect(info).toBeDefined();
expect((info as any).path).toBeUndefined();
expect((info as any).buffer).toBeDefined();
resolve();
}
});
});
});
it("uses disk storage when Content-Length is above threshold", async () => {
const req = mockReq("2048");
const file = mockFile("x".repeat(2048));
await new Promise<void>((resolve, reject) => {
storage._handleFile(req, file, (err, info) => {
if (err) reject(err);
else {
expect(info).toBeDefined();
expect((info as any).path).toBeDefined();
expect((info as any).destination).toBeDefined();
tempDirs.push((info as any).destination);
resolve();
}
});
});
});
it("uses disk storage when Content-Length is absent (chunked transfer)", async () => {
const req = mockReq();
const file = mockFile("chunked content");
await new Promise<void>((resolve, reject) => {
storage._handleFile(req, file, (err, info) => {
if (err) reject(err);
else {
expect(info).toBeDefined();
expect((info as any).path).toBeDefined();
expect((info as any).destination).toBeDefined();
tempDirs.push((info as any).destination);
resolve();
}
});
});
});
it("uses disk storage when Content-Length is invalid", async () => {
const req = mockReq("not-a-number");
const file = mockFile("content");
await new Promise<void>((resolve, reject) => {
storage._handleFile(req, file, (err, info) => {
if (err) reject(err);
else {
expect(info).toBeDefined();
expect((info as any).path).toBeDefined();
tempDirs.push((info as any).destination);
resolve();
}
});
});
});
it("removes file from disk", async () => {
const req = mockReq("2048");
const file = mockFile("x".repeat(2048));
const info = await new Promise<any>((resolve, reject) => {
storage._handleFile(req, file, (err, info) => {
if (err) reject(err);
else resolve(info);
});
});
tempDirs.push(info.destination);
await new Promise<void>((resolve, reject) => {
storage._removeFile(req, { ...file, path: info.path } as any, (err) => {
if (err) reject(err);
else resolve();
});
});
});
it("uses memory storage when Content-Length is 0", async () => {
const req = mockReq("0");
const file = mockFile("");
await new Promise<void>((resolve, reject) => {
storage._handleFile(req, file, (err, info) => {
if (err) reject(err);
else {
expect(info).toBeDefined();
expect((info as any).path).toBeUndefined();
expect((info as any).buffer).toBeDefined();
resolve();
}
});
});
});
it("uses disk storage when Content-Length equals threshold", async () => {
const req = mockReq("1024");
const file = mockFile("x".repeat(1024));
await new Promise<void>((resolve, reject) => {
storage._handleFile(req, file, (err, info) => {
if (err) reject(err);
else {
expect(info).toBeDefined();
expect((info as any).path).toBeDefined();
expect((info as any).destination).toBeDefined();
tempDirs.push((info as any).destination);
resolve();
}
});
});
});
it("uses disk storage when Content-Length is very large", async () => {
const req = mockReq("1073741824"); // 1GB
const file = mockFile("x"); // Small actual content, but large Content-Length
await new Promise<void>((resolve, reject) => {
storage._handleFile(req, file, (err, info) => {
if (err) reject(err);
else {
expect(info).toBeDefined();
expect((info as any).path).toBeDefined();
expect((info as any).destination).toBeDefined();
tempDirs.push((info as any).destination);
resolve();
}
});
});
});
it("reuses the same temp directory for concurrent uploads", async () => {
const req1 = mockReq("2048");
const file1 = mockFile("x".repeat(2048));
const req2 = mockReq("2048");
const file2 = mockFile("y".repeat(2048));
const [info1, info2] = await Promise.all([
new Promise<any>((resolve, reject) => {
storage._handleFile(req1, file1, (err, info) => {
if (err) reject(err);
else resolve(info);
});
}),
new Promise<any>((resolve, reject) => {
storage._handleFile(req2, file2, (err, info) => {
if (err) reject(err);
else resolve(info);
});
}),
]);
tempDirs.push(info1.destination);
tempDirs.push(info2.destination);
// Both uploads should use the same temp directory
expect(info1.destination).toBe(info2.destination);
});
});

19
media-processor/tsconfig.json Normal file
View File

@ -0,0 +1,19 @@
{
"compilerOptions": {
"target": "ES2022",
"module": "ESNext",
"moduleResolution": "bundler",
"lib": ["ES2022"],
"outDir": "./dist",
"rootDir": "./src",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true,
"declaration": true,
"sourceMap": true,
"resolveJsonModule": true
},
"include": ["src/**/*"],
"exclude": ["node_modules", "dist"]
}

8
media-processor/vitest.config.ts Normal file
View File

@ -0,0 +1,8 @@
import { defineConfig } from "vitest/config";
export default defineConfig({
envPrefix: [],
test: {
setupFiles: ["./test/setup.ts"],
},
});