github/penpot
1
0
Fork 0
mirror of https://github.com/penpot/penpot.git synced 2026-07-01 20:05:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

✨ Add configurable resource usage limits for imagemagick (#10240)

Browse Source 
* 🐳 Add ImageMagick policy.xml resource limits to backend Docker image

Add a restrictive policy.xml to the backend Docker image that caps
ImageMagick resource usage: 256MiB memory, 512MiB map, 128MP area,
30s time limit, 16KP max dimensions. Blocks PS/EPS/PDF/XPS coders
to prevent Ghostscript attack surface.

Co-authored-by: mimo-v2.5-pro <mimo-v2.5-pro@penpot.app>

* ✨ Add timeout support to shell/exec!

Add optional :timeout parameter (in seconds) that uses
Process.waitFor(long, TimeUnit). On timeout, the process is
destroyed forcibly and an :internal/:process-timeout exception
is raised. Stdout/stderr readers handle IOException from closed
streams when the process is killed.

Co-authored-by: mimo-v2.5-pro <mimo-v2.5-pro@penpot.app>

* ♻️ Rename ::wrk/netty-executor to ::wrk/executor with cached pool

Replace DefaultEventExecutorGroup (fixed Netty thread pool) with a
cached thread pool (px/cached-executor) for general async task
offloading. The cached pool creates threads on demand and reuses
idle ones, which is more appropriate for blocking I/O workloads
(shell commands, message bus, rate limiting, etc.).

Changes:
- Rename ::wrk/netty-executor to ::wrk/executor in worker/executor.clj
- Switch implementation from DefaultEventExecutorGroup to px/cached-executor
- Update all ig/ref wiring in main.clj (msgbus, tmp cleaner, climit, rlimit, rpc)
- Remove ::wrk/netty-executor from redis.clj (let lettuce create its own
  eventExecutorGroup instead of sharing a Netty executor)
- Assert executor is present in shell/exec! to prevent silent nil usage
- Remove executor-threads config (no longer needed for cached pool)

The ::wrk/netty-io-executor (NioEventLoopGroup) remains unchanged as it
handles actual non-blocking network I/O for Redis and S3.

Co-authored-by: mimo-v2.5-pro <mimo-v2.5-pro@penpot.app>

* 🔥 Remove im4java dependency and replace with direct ImageMagick CLI calls

- Replace im4java Java library with direct 'magick' CLI calls via shell/exec!
- Add PENPOT_IMAGEMAGICK_* config env vars for resource limits (thread, memory, map, area, disk, time, width, height)
- Use configurable ImageMagick environment with sensible defaults matching policy.xml
- Remove -Dim4java.useV7=true JVM flag from startup scripts
- Remove org.im4java/im4java from deps.edn
- All ImageMagick commands now use shell/exec! with 60s timeout and resource limits

Co-authored-by: mimo-v2.5-pro <mimo-v2.5-pro@penpot.app>

* 💄 Rename imagemagick env functions and optimize config reads

- Rename imagemagick-defaults -> imagemagick-default-env
- Rename imagemagick-env -> get-imagemagick-env
- Optimize to avoid double cf/get calls per config key

Co-authored-by: mimo-v2.5-pro <mimo-v2.5-pro@penpot.app>

* ✨ Add tests for shell/exec! timeout and media processing

- Add shell_test.clj: tests for exec! timeout, env vars, stdin, stderr
- Add media_test.clj: tests for info, generic-thumbnail, profile-thumbnail
- Fix generic-process to prefer explicit format over input mtype
- Fix shell/exec! to use cached executor when system has no executor
- Fix reduce-kv accumulator in set-env (must return penv)

Co-authored-by: mimo-v2.5-pro <mimo-v2.5-pro@penpot.app>

* ♻️ Refactor media/process to take system as first argument

- Change (defmulti process :cmd) -> (defmulti process (fn [_system params] (:cmd params)))
- Change (run params) -> (run system params)
- All process methods now receive [system params]
- Update all callers: rpc/commands/media, profile, auth, fonts
- Revert shell/exec! to require system with executor (no fallback)
- Fix lint warnings and formatting

Co-authored-by: mimo-v2.5-pro <mimo-v2.5-pro@penpot.app>

* 🔥 Remove unused app.svgo namespace

Co-authored-by: mimo-v2.5-pro <mimo-v2.5-pro@penpot.app>

* 🔥 Remove Node.js from backend Docker image

- Delete unused svgo-cli.js script
- Remove Node.js installation from Dockerfile.backend
- Remove svgo-cli.js copy from backend build script

Co-authored-by: mimo-v2.5-pro <mimo-v2.5-pro@penpot.app>

* 🔥 Remove unused process-error multimethod

- Remove process-error multimethod and its default handler
- Simplify media/run to directly call process
- Fix alignment in main.clj

Co-authored-by: mimo-v2.5-pro <mimo-v2.5-pro@penpot.app>

* 📚 Add ImageMagick resource limits configuration to technical guide

Co-authored-by: mimo-v2.5-pro <mimo-v2.5-pro@penpot.app>

---------

Co-authored-by: mimo-v2.5-pro <mimo-v2.5-pro@penpot.app>
This commit is contained in:
Andrey Antukh 2026-06-18 17:52:01 +02:00 committed by GitHub
parent fe942f9780
commit b4532486e3
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
21 changed files with 414 additions and 417 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
backend/deps.edn
View File

@ -52,10 +52,6 @@
com.github.ben-manes.caffeine/caffeine {:mvn/version "3.2.4"}
org.jsoup/jsoup {:mvn/version "1.22.2"}
org.im4java/im4java
{:git/tag "1.4.0-penpot-2"
:git/sha "e2b3e16"
:git/url "https://github.com/penpot/im4java"}
at.yawk.lz4/lz4-java
{:mvn/version "1.11.0"}

1
backend/scripts/_env
View File

@ -77,7 +77,6 @@ export JAVA_OPTS="\
-Djdk.attach.allowAttachSelf \
-Dlog4j2.configurationFile=log4j2-devenv.xml \
-Djdk.tracePinnedThreads=full \
-Dim4java.useV7=true \
-XX:+UnlockExperimentalVMOptions \
-XX:+UseShenandoahGC \
-XX:+UseCompactObjectHeaders \

1
backend/scripts/build
View File

@ -17,7 +17,6 @@ mv target/penpot.jar target/dist/penpot.jar
cp resources/log4j2.xml target/dist/log4j2.xml
cp scripts/run.template.sh target/dist/run.sh;
cp scripts/manage.py target/dist/manage.py
cp scripts/svgo-cli.js target/dist/scripts/;
chmod +x target/dist/run.sh;
chmod +x target/dist/manage.py

2
backend/scripts/run.template.sh
View File

@ -18,7 +18,7 @@ if [ -f ./environ ]; then
source ./environ
fi
export JAVA_OPTS="-Dim4java.useV7=true -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dlog4j2.configurationFile=log4j2.xml -XX:-OmitStackTraceInFastThrow --sun-misc-unsafe-memory-access=allow --enable-native-access=ALL-UNNAMED --enable-preview $JVM_OPTS $JAVA_OPTS"
export JAVA_OPTS="-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dlog4j2.configurationFile=log4j2.xml -XX:-OmitStackTraceInFastThrow --sun-misc-unsafe-memory-access=allow --enable-native-access=ALL-UNNAMED --enable-preview $JVM_OPTS $JAVA_OPTS"
ENTRYPOINT=${1:-app.main};

214
backend/scripts/svgo-cli.js
View File

File diff suppressed because one or more lines are too long

12
backend/src/app/config.clj
View File

@ -127,6 +127,17 @@
[:media-max-file-size {:optional true} ::sm/int]
[:font-max-file-size {:optional true} ::sm/int]
;; ImageMagick resource limits (PENPOT_IMAGEMAGICK_*)
[:imagemagick-thread-limit {:optional true} :string]
[:imagemagick-memory-limit {:optional true} :string]
[:imagemagick-map-limit {:optional true} :string]
[:imagemagick-area-limit {:optional true} :string]
[:imagemagick-disk-limit {:optional true} :string]
[:imagemagick-time-limit {:optional true} :string]
[:imagemagick-width-limit {:optional true} :string]
[:imagemagick-height-limit {:optional true} :string]
[:deletion-delay {:optional true} ::ct/duration]
[:file-clean-delay {:optional true} ::ct/duration]
[:telemetry-enabled {:optional true} ::sm/boolean]
@ -241,7 +252,6 @@
[:assets-path {:optional true} :string]
[:netty-io-threads {:optional true} ::sm/int]
[:executor-threads {:optional true} ::sm/int]
[:nitrate-backend-uri {:optional true} ::sm/uri]

24
backend/src/app/main.clj
View File

@ -38,7 +38,6 @@
[app.storage.gc-deleted :as-alias sto.gc-deleted]
[app.storage.gc-touched :as-alias sto.gc-touched]
[app.storage.s3 :as-alias sto.s3]
[app.svgo :as-alias svgo]
[app.util.cron]
[app.worker :as-alias wrk]
[app.worker.executor]
@ -162,8 +161,8 @@
::wrk/netty-io-executor
{:threads (cf/get :netty-io-threads)}
::wrk/netty-executor
{:threads (cf/get :executor-threads)}
::wrk/executor
{}
:app.migrations/migrations
{::db/pool (ig/ref ::db/pool)}
@ -178,9 +177,6 @@
{::rds/uri
(cf/get :redis-uri)
::wrk/netty-executor
(ig/ref ::wrk/netty-executor)
::wrk/netty-io-executor
(ig/ref ::wrk/netty-io-executor)}
@ -189,12 +185,12 @@
::mtx/metrics (ig/ref ::mtx/metrics)}
::mbus/msgbus
{::wrk/executor (ig/ref ::wrk/netty-executor)
{::wrk/executor (ig/ref ::wrk/executor)
::rds/client (ig/ref ::rds/client)
::mtx/metrics (ig/ref ::mtx/metrics)}
:app.storage.tmp/cleaner
{::wrk/executor (ig/ref ::wrk/netty-executor)}
{::wrk/executor (ig/ref ::wrk/executor)}
::sto.gc-deleted/handler
{::db/pool (ig/ref ::db/pool)
@ -308,12 +304,12 @@
::rpc/climit
{::mtx/metrics (ig/ref ::mtx/metrics)
::wrk/executor (ig/ref ::wrk/netty-executor)
::wrk/executor (ig/ref ::wrk/executor)
::climit/config (cf/get :rpc-climit-config)
::climit/enabled (contains? cf/flags :rpc-climit)}
:app.rpc/rlimit
{::wrk/executor (ig/ref ::wrk/netty-executor)
{::wrk/executor (ig/ref ::wrk/executor)
:app.loggers.mattermost/reporter
(ig/ref :app.loggers.mattermost/reporter)
@ -325,8 +321,8 @@
{::http.client/client (ig/ref ::http.client/client)
::db/pool (ig/ref ::db/pool)
::rds/pool (ig/ref ::rds/pool)
:app.nitrate/client (ig/ref :app.nitrate/client)
::wrk/executor (ig/ref ::wrk/netty-executor)
:app.nitrate/client (ig/ref :app.nitrate/client)
::wrk/executor (ig/ref ::wrk/executor)
::session/manager (ig/ref ::session/manager)
::ldap/provider (ig/ref ::ldap/provider)
::sto/storage (ig/ref ::sto/storage)
@ -356,12 +352,12 @@
{::http.client/client (ig/ref ::http.client/client)
::db/pool (ig/ref ::db/pool)
::rds/pool (ig/ref ::rds/pool)
::wrk/executor (ig/ref ::wrk/netty-executor)
::wrk/executor (ig/ref ::wrk/executor)
::session/manager (ig/ref ::session/manager)
::sto/storage (ig/ref ::sto/storage)
::mtx/metrics (ig/ref ::mtx/metrics)
::mbus/msgbus (ig/ref ::mbus/msgbus)
:app.nitrate/client (ig/ref :app.nitrate/client)
:app.nitrate/client (ig/ref :app.nitrate/client)
::rds/client (ig/ref ::rds/client)
::setup/props (ig/ref ::setup/props)}

147
backend/src/app/media.clj
View File

@ -21,6 +21,7 @@
[app.media.sanitize :as sanitize]
[app.storage :as-alias sto]
[app.storage.tmp :as tmp]
[app.util.shell :as shell]
[buddy.core.bytes :as bb]
[buddy.core.codecs :as bc]
[clojure.java.shell :as sh]
@ -34,9 +35,7 @@
java.io.InputStream
javax.xml.parsers.SAXParserFactory
javax.xml.XMLConstants
org.apache.commons.io.IOUtils
org.im4java.core.ConvertCmd
org.im4java.core.IMOperation))
org.apache.commons.io.IOUtils))
(def schema:upload
[:map {:title "Upload"}
@ -90,25 +89,17 @@
max-size)))
upload))
(defmulti process :cmd)
(defmulti process-error class)
(defmulti process (fn [_system params] (:cmd params)))
(defmethod process :default
[{:keys [cmd] :as params}]
[_system {:keys [cmd] :as params}]
(ex/raise :type :internal
:code :not-implemented
:hint (str/fmt "No impl found for process cmd: %s" cmd)))
(defmethod process-error :default
[error]
(throw error))
(defn run
[params]
(try
(process params)
(catch Throwable e
(process-error e))))
[system params]
(process system params))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; SVG PARSING
@ -152,16 +143,63 @@
;; Related info on how thumbnails generation
;; http://www.imagemagick.org/Usage/thumbnails/
(def ^:private imagemagick-default-env
"Default environment variables for ImageMagick resource limits.
These are the soft ceiling — policy.xml is the hard ceiling."
{"MAGICK_THREAD_LIMIT" "2"
"MAGICK_MEMORY_LIMIT" "256MiB"
"MAGICK_MAP_LIMIT" "512MiB"
"MAGICK_AREA_LIMIT" "128MP"
"MAGICK_DISK_LIMIT" "1GiB"
"MAGICK_TIME_LIMIT" "30"})
(defn- get-imagemagick-env
"Returns environment variables for ImageMagick commands.
Reads individual PENPOT_IMAGEMAGICK_* config values, falling back to defaults."
[]
(let [thread (cf/get :imagemagick-thread-limit)
memory (cf/get :imagemagick-memory-limit)
map-l (cf/get :imagemagick-map-limit)
area (cf/get :imagemagick-area-limit)
disk (cf/get :imagemagick-disk-limit)
time (cf/get :imagemagick-time-limit)
width (cf/get :imagemagick-width-limit)
height (cf/get :imagemagick-height-limit)]
(cond-> imagemagick-default-env
thread (assoc "MAGICK_THREAD_LIMIT" thread)
memory (assoc "MAGICK_MEMORY_LIMIT" memory)
map-l (assoc "MAGICK_MAP_LIMIT" map-l)
area (assoc "MAGICK_AREA_LIMIT" area)
disk (assoc "MAGICK_DISK_LIMIT" disk)
time (assoc "MAGICK_TIME_LIMIT" time)
width (assoc "MAGICK_WIDTH_LIMIT" width)
height (assoc "MAGICK_HEIGHT_LIMIT" height))))
(defn- exec-magick!
"Execute an ImageMagick command with resource limits.
`args` is a vector of string arguments to pass to `magick`."
[system args]
(let [cmd (into ["magick"] args)
result (shell/exec! system
:cmd cmd
:env (get-imagemagick-env)
:timeout 60)]
(when (not= 0 (:exit result))
(ex/raise :type :internal
:code :imagemagick-error
:hint (str "ImageMagick command failed: " (:err result))
:cmd cmd
:exit (:exit result)))
result))
(defn- generic-process
[{:keys [input format operation] :as params}]
[system {:keys [input format convert-args] :as params}]
(let [{:keys [path mtype]} input
format (or (cm/mtype->format mtype) format)
format (or format (cm/mtype->format mtype))
ext (cm/format->extension format)
tmp (tmp/tempfile :prefix "penpot.media." :suffix ext)]
(doto (ConvertCmd.)
(.run operation (into-array (map str [path tmp]))))
tmp (tmp/tempfile :prefix "penpot.media." :suffix ext)
args (into [(str path)] (conj (vec convert-args) (str tmp)))]
(exec-magick! system args)
(assoc params
:format format
:mtype (cm/format->mtype format)
@ -169,38 +207,26 @@
:data tmp)))
(defmethod process :generic-thumbnail
[params]
[system params]
(let [{:keys [quality width height] :as params}
(check-thumbnail-params params)
operation
(doto (IMOperation.)
(.addImage)
(.autoOrient)
(.strip)
(.thumbnail ^Integer (int width) ^Integer (int height) ">")
(.quality (double quality))
(.addImage))]
(generic-process (assoc params :operation operation))))
(check-thumbnail-params params)]
(generic-process system
(assoc params
:convert-args ["-auto-orient" "-strip"
"-thumbnail" (str width "x" height ">")
"-quality" (str quality)]))))
(defmethod process :profile-thumbnail
[params]
[system params]
(let [{:keys [quality width height] :as params}
(check-thumbnail-params params)
operation
(doto (IMOperation.)
(.addImage)
(.autoOrient)
(.strip)
(.thumbnail ^Integer (int width) ^Integer (int height) "^")
(.gravity "center")
(.extent (int width) (int height))
(.quality (double quality))
(.addImage))]
(generic-process (assoc params :operation operation))))
(check-thumbnail-params params)]
(generic-process system
(assoc params
:convert-args ["-auto-orient" "-strip"
"-thumbnail" (str width "x" height "^")
"-gravity" "center"
"-extent" (str width "x" height)
"-quality" (str quality)]))))
(defn get-basic-info-from-svg
[{:keys [tag attrs] :as data}]
@ -230,11 +256,11 @@
{:width (int width)
:height (int height)})))]))
(defn- get-dimensions-with-orientation [^String path]
(defn- get-dimensions-with-orientation [system ^String path]
;; Image magick doesn't give info about exif rotation so we use the identify command
;; If we are processing an animated gif we use the first frame with -scene 0
(let [dim-result (sh/sh "identify" "-format" "%w %h\n" path)
orient-result (sh/sh "identify" "-format" "%[EXIF:Orientation]\n" path)]
(let [dim-result (exec-magick! system ["identify" "-format" "%w %h\n" path])
orient-result (exec-magick! system ["identify" "-format" "%[EXIF:Orientation]\n" path])]
(when (= 0 (:exit dim-result))
(let [[w h] (-> (:out dim-result)
str/trim
@ -249,7 +275,7 @@
{:width w :height h}))))) ; If orientation can't be read, use dimensions as-is
(defmethod process :info
[{:keys [input] :as params}]
[system {:keys [input] :as params}]
(let [{:keys [path mtype] :as input} (check-input input)]
(if (= mtype "image/svg+xml")
(let [info (some-> path slurp parse-svg get-basic-info-from-svg)]
@ -260,7 +286,7 @@
(merge input info {:ts (ct/now) :size (fs/size path)}))
(let [path-str (str path)
identify-res (sh/sh "identify" "-format" "image/%[magick]\n" path-str)
identify-res (exec-magick! system ["identify" "-format" "image/%[magick]\n" path-str])
;; identify prints one line per frame (animated GIFs, etc.); we take the first one
mtype' (if (zero? (:exit identify-res))
(-> identify-res
@ -273,7 +299,7 @@
:code :invalid-image
:hint "invalid image"))
{:keys [width height]}
(or (get-dimensions-with-orientation path-str)
(or (get-dimensions-with-orientation system path-str)
(do
(l/warn "Failed to read image dimensions with orientation" {:path path})
(ex/raise :type :validation
@ -291,13 +317,6 @@
:size (fs/size path)
:ts (ct/now))))))
(defmethod process-error org.im4java.core.InfoException
[error]
(ex/raise :type :validation
:code :invalid-image
:hint "invalid image"
:cause error))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; IMAGE HELPERS
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
@ -371,7 +390,7 @@
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defmethod process :generate-fonts
[{:keys [input] :as params}]
[_system {:keys [input] :as params}]
(letfn [(ttf->otf [data]
(let [finput (tmp/tempfile :prefix "penpot.font." :suffix "")
foutput (fs/path (str finput ".otf"))

5
backend/src/app/redis.clj
View File

@ -43,7 +43,6 @@
io.lettuce.core.ScriptOutputType
io.lettuce.core.SetArgs
io.netty.channel.nio.NioEventLoopGroup
io.netty.util.concurrent.EventExecutorGroup
io.netty.util.HashedWheelTimer
io.netty.util.Timer
java.lang.AutoCloseable
@ -527,7 +526,6 @@
(def ^:private schema:client-params
[:map {:title "redis-params"}
::wrk/netty-io-executor
::wrk/netty-executor
[::uri ::sm/uri]
[::timeout ::ct/duration]])
@ -539,7 +537,7 @@
(check-client-params params))
(defmethod ig/init-key ::client
[_ {:keys [::uri ::wrk/netty-io-executor ::wrk/netty-executor] :as params}]
[_ {:keys [::uri ::wrk/netty-io-executor] :as params}]
(l/inf :hint "initialize redis client" :uri (str uri))
@ -547,7 +545,6 @@
cache (atom {})
resources (.. (DefaultClientResources/builder)
(eventExecutorGroup ^EventExecutorGroup netty-executor)
;; We provide lettuce with a shared event loop
;; group instance instead of letting lettuce to

2
backend/src/app/rpc/commands/auth.clj
View File

@ -320,7 +320,7 @@
(try
(let [storage (sto/resolve cfg)
input (media/download-image cfg uri)
input (media/run {:cmd :info :input input})
input (media/run cfg {:cmd :info :input input})
hash (sto/calculate-hash (:path input))
content (-> (sto/content (:path input) (:size input))
(sto/wrap-with-hash hash))

2
backend/src/app/rpc/commands/fonts.clj
View File

@ -181,7 +181,7 @@
(defn create-font-variant
[{:keys [::sto/storage] :as cfg} {:keys [data] :as params}]
(letfn [(generate-missing [data]
(let [data (media/run {:cmd :generate-fonts :input data})]
(let [data (media/run cfg {:cmd :generate-fonts :input data})]
(when (and (not (contains? data "font/otf"))
(not (contains? data "font/ttf"))
(not (contains? data "font/woff"))

17
backend/src/app/rpc/commands/media.clj
View File

@ -123,11 +123,10 @@
:bucket "file-media-object"}))
(defn- process-thumb-image
[info]
(let [thumb (-> thumbnail-options
(assoc :cmd :generic-thumbnail)
(assoc :input info)
(media/run))
[cfg info]
(let [thumb (media/run cfg (assoc thumbnail-options
:cmd :generic-thumbnail
:input info))
hash (sto/calculate-hash (:data thumb))
data (-> (sto/content (:data thumb) (:size thumb))
(sto/wrap-with-hash hash))]
@ -138,12 +137,12 @@
:bucket "file-media-object"}))
(defn- process-image
[content]
(let [info (media/run {:cmd :info :input content})]
[cfg content]
(let [info (media/run cfg {:cmd :info :input content})]
(cond-> info
(and (not (svg-image? info))
(big-enough-for-thumbnail? info))
(assoc ::thumb (process-thumb-image info))
(assoc ::thumb (process-thumb-image cfg info))
:always
(assoc ::image (process-main-image info)))))
@ -170,7 +169,7 @@
:path (str (:path content))
:origin origin)
(let [result (process-image content)
(let [result (process-image cfg content)
image (sto/put-object! storage (::image result))
thumb (when-let [params (::thumb result)]
(sto/put-object! storage params))

16
backend/src/app/rpc/commands/profile.clj
View File

@ -298,14 +298,14 @@
:file-mtype (:mtype file)}}))))
(defn- generate-thumbnail
[_ input]
(let [input (media/run {:cmd :info :input input})
thumb (media/run {:cmd :profile-thumbnail
:format :jpeg
:quality 85
:width 256
:height 256
:input input})
[cfg input]
(let [input (media/run cfg {:cmd :info :input input})
thumb (media/run cfg {:cmd :profile-thumbnail
:format :jpeg
:quality 85
:width 256
:height 256
:input input})
hash (sto/calculate-hash (:data thumb))
content (-> (sto/content (:data thumb) (:size thumb))
(sto/wrap-with-hash hash))]

38
backend/src/app/svgo.clj
View File

@ -1,38 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
(ns app.svgo
"A SVG Optimizer service"
(:require
[app.common.logging :as l]
[app.util.shell :as shell]
[datoteka.fs :as fs]
[promesa.exec.semaphore :as ps]))
(def ^:dynamic *semaphore*
"A dynamic variable that can optionally contain a traffic light to
appropriately delimit the use of resources, managed externally."
nil)
(set! *warn-on-reflection* true)
(defn optimize
[system data]
(try
(some-> *semaphore* ps/acquire!)
(let [script (fs/join fs/*cwd* "scripts/svgo-cli.js")
cmd ["node" (str script)]
result (shell/exec! system
:cmd cmd
:in data)]
(if (= (:exit result) 0)
(:out result)
(do
(l/raw! :warn (str "Error on optimizing svg, returning svg as-is." (:err result)))
data)))
(finally
(some-> *semaphore* ps/release!))))

27
backend/src/app/util/shell.clj
View File

@ -8,12 +8,14 @@
"A penpot specific, modern api for executing external (shell)
subprocesses"
(:require
[app.common.exceptions :as ex]
[app.worker :as-alias wrk]
[datoteka.io :as io]
[promesa.exec :as px])
(:import
java.io.InputStream
java.io.OutputStream
java.util.concurrent.TimeUnit
java.util.List
org.apache.commons.io.IOUtils))
@ -39,16 +41,18 @@
[penv k v]
(.put ^java.util.Map penv
^String k
^String v))
^String v)
penv)
(defn exec!
[system & {:keys [cmd in out-enc in-enc env]
[system & {:keys [cmd in out-enc in-enc env timeout]
:or {out-enc "UTF-8"
in-enc "UTF-8"}}]
(assert (vector? cmd) "a command parameter should be a vector")
(assert (every? string? cmd) "the command should be a vector of strings")
(let [executor (::wrk/executor system)
_ (assert (some? executor) "executor is required, check ::wrk/executor")
builder (ProcessBuilder. ^List cmd)
env-map (.environment ^ProcessBuilder builder)
_ (reduce-kv set-env env-map env)
@ -63,9 +67,22 @@
(with-open [stdout (.getInputStream ^Process process)
stderr (.getErrorStream ^Process process)]
(let [out (px/submit! executor (fn [] (read-with-enc stdout out-enc)))
err (px/submit! executor (fn [] (read-as-string stderr)))
ext (.waitFor ^Process process)]
(let [out (px/submit! executor (fn [] (try (read-with-enc stdout out-enc)
(catch java.io.IOException _ ""))))
err (px/submit! executor (fn [] (try (read-as-string stderr)
(catch java.io.IOException _ ""))))
ext (if timeout
(let [completed (.waitFor ^Process process (long timeout) TimeUnit/SECONDS)]
(if completed
(.exitValue ^Process process)
(do
(.destroyForcibly ^Process process)
(ex/raise :type :internal
:code :process-timeout
:hint (str "process timed out after " timeout " seconds")
:cmd cmd
:timeout timeout))))
(.waitFor ^Process process))]
{:exit ext
:out @out
:err @err}))))

32
backend/src/app/worker/executor.clj
View File

@ -15,9 +15,7 @@
[promesa.exec :as px])
(:import
io.netty.channel.nio.NioEventLoopGroup
io.netty.util.concurrent.DefaultEventExecutorGroup
java.util.concurrent.ExecutorService
java.util.concurrent.ThreadFactory
java.util.concurrent.TimeUnit))
(set! *warn-on-reflection* true)
@ -36,13 +34,6 @@
{:title "executor"
:description "Instance of NioEventLoopGroup"}})
(sm/register!
{:type ::wrk/netty-executor
:pred #(instance? DefaultEventExecutorGroup %)
:type-properties
{:title "executor"
:description "Instance of DefaultEventExecutorGroup"}})
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; IO Executor
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
@ -58,7 +49,7 @@
nthreads (or threads (mth/round (/ (px/get-available-processors) 2)))
nthreads (max 2 nthreads)]
(l/inf :hint "start netty io executor" :threads nthreads)
(NioEventLoopGroup. (int nthreads) ^ThreadFactory factory)))
(NioEventLoopGroup. (int nthreads) ^java.util.concurrent.ThreadFactory factory)))
(defmethod ig/halt-key! ::wrk/netty-io-executor
[_ instance]
@ -68,22 +59,15 @@
TimeUnit/MILLISECONDS)))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; IO Offload Executor
;; Executor
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defmethod ig/assert-key ::wrk/netty-executor
[_ {:keys [threads]}]
(assert (or (nil? threads) (int? threads))
"expected valid threads value, revisit PENPOT_EXEC_THREADS environment variable"))
(defmethod ig/init-key ::wrk/executor
[_ _]
(let [factory (px/thread-factory :prefix "penpot/exec/")]
(l/inf :hint "start cached executor")
(px/cached-executor :factory factory)))
(defmethod ig/init-key ::wrk/netty-executor
[_ {:keys [threads]}]
(let [factory (px/thread-factory :prefix "penpot/exec/")
nthreads (or threads (mth/round (/ (px/get-available-processors) 2)))
nthreads (max 2 nthreads)]
(l/inf :hint "start default executor" :threads nthreads)
(DefaultEventExecutorGroup. (int nthreads) ^ThreadFactory factory)))
(defmethod ig/halt-key! ::wrk/netty-executor
(defmethod ig/halt-key! ::wrk/executor
[_ instance]
(px/shutdown! instance))

126
backend/test/backend_tests/media_test.clj Normal file
View File

@ -0,0 +1,126 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
(ns backend-tests.media-test
(:require
[app.common.exceptions :as ex]
[app.media :as media]
[backend-tests.helpers :as th]
[clojure.test :as t]
[datoteka.fs :as fs]))
(t/use-fixtures :once th/state-init)
(t/deftest info-jpeg
(t/testing "info on valid JPEG returns dimensions and mime type"
(let [path (th/tempfile "backend_tests/test_files/sample.jpg")
info (media/run th/*system* {:cmd :info
:input {:path path
:mtype "image/jpeg"}})]
(t/is (pos? (:width info)))
(t/is (pos? (:height info)))
(t/is (= "image/jpeg" (:mtype info)))
(t/is (pos? (:size info)))
(t/is (some? (:ts info))))))
(t/deftest info-png
(t/testing "info on valid PNG returns dimensions and mime type"
(let [path (th/tempfile "backend_tests/test_files/sample.png")
info (media/run th/*system* {:cmd :info
:input {:path path
:mtype "image/png"}})]
(t/is (pos? (:width info)))
(t/is (pos? (:height info)))
(t/is (= "image/png" (:mtype info))))))
(t/deftest info-webp
(t/testing "info on valid WebP returns dimensions and mime type"
(let [path (th/tempfile "backend_tests/test_files/sample.webp")
info (media/run th/*system* {:cmd :info
:input {:path path
:mtype "image/webp"}})]
(t/is (pos? (:width info)))
(t/is (pos? (:height info)))
(t/is (= "image/webp" (:mtype info))))))
(t/deftest info-svg
(t/testing "info on valid SVG returns dimensions from viewBox"
(let [path (th/tempfile "backend_tests/test_files/sample1.svg")
info (media/run th/*system* {:cmd :info
:input {:path path
:mtype "image/svg+xml"}})]
(t/is (pos? (:width info)))
(t/is (pos? (:height info))))))
(t/deftest info-invalid-image
(t/testing "info on invalid image raises error"
(let [path (fs/create-tempfile :prefix "penpot-test-" :suffix ".jpg")]
;; Write garbage data
(spit (str path) "not an image")
(try
(media/run th/*system* {:cmd :info
:input {:path path
:mtype "image/jpeg"}})
(t/is false "should have thrown")
(catch Exception e
(let [data (ex-data e)]
;; Could be validation or imagemagick-error depending on what magick does
(t/is (contains? #{:validation :internal} (:type data)))))
(finally
(fs/delete path))))))
(t/deftest generic-thumbnail
(t/testing "generic-thumbnail produces a file of expected format"
(let [path (th/tempfile "backend_tests/test_files/sample.jpg")
info (media/run th/*system* {:cmd :info
:input {:path path
:mtype "image/jpeg"}})
thumb (media/run th/*system* {:cmd :generic-thumbnail
:input info
:format :jpeg
:quality 80
:width 200
:height 200})]
(t/is (some? (:data thumb)))
(t/is (pos? (:size thumb)))
(t/is (= :jpeg (:format thumb)))
(t/is (= "image/jpeg" (:mtype thumb)))
;; Verify the thumbnail file exists
(t/is (fs/exists? (:data thumb))))))
(t/deftest profile-thumbnail
(t/testing "profile-thumbnail produces a center-cropped file"
(let [path (th/tempfile "backend_tests/test_files/sample.jpg")
info (media/run th/*system* {:cmd :info
:input {:path path
:mtype "image/jpeg"}})
thumb (media/run th/*system* {:cmd :profile-thumbnail
:input info
:format :jpeg
:quality 85
:width 128
:height 128})]
(t/is (some? (:data thumb)))
(t/is (pos? (:size thumb)))
(t/is (= :jpeg (:format thumb)))
(t/is (= "image/jpeg" (:mtype thumb)))
;; Verify the thumbnail file exists
(t/is (fs/exists? (:data thumb))))))
(t/deftest generic-thumbnail-webp
(t/testing "generic-thumbnail can produce WebP format"
(let [path (th/tempfile "backend_tests/test_files/sample.jpg")
info (media/run th/*system* {:cmd :info
:input {:path path
:mtype "image/jpeg"}})
thumb (media/run th/*system* {:cmd :generic-thumbnail
:input info
:format :webp
:quality 80
:width 200
:height 200})]
(t/is (= :webp (:format thumb)))
(t/is (= "image/webp" (:mtype thumb))))))

78
backend/test/backend_tests/shell_test.clj Normal file
View File

@ -0,0 +1,78 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
(ns backend-tests.shell-test
(:require
[app.common.exceptions :as ex]
[app.util.shell :as shell]
[clojure.string :as str]
[clojure.test :as t]))
(t/deftest exec-normal-completes
(t/testing "normal process completes within timeout"
(let [result (shell/exec! {}
:cmd ["echo" "hello"]
:timeout 10)]
(t/is (= 0 (:exit result)))
(t/is (str/includes? (:out result) "hello")))))
(t/deftest exec-captures-stderr
(t/testing "stderr is captured separately"
(let [result (shell/exec! {}
:cmd ["bash" "-c" "echo out; echo err >&2"]
:timeout 10)]
(t/is (= 0 (:exit result)))
(t/is (str/includes? (:out result) "out"))
(t/is (str/includes? (:err result) "err")))))
(t/deftest exec-non-zero-exit
(t/testing "non-zero exit code is captured"
(let [result (shell/exec! {}
:cmd ["bash" "-c" "exit 42"]
:timeout 10)]
(t/is (= 42 (:exit result))))))
(t/deftest exec-with-env
(t/testing "environment variables are passed to the process"
(let [result (shell/exec! {}
:cmd ["bash" "-c" "echo $MY_VAR"]
:env {"MY_VAR" "test-value"}
:timeout 10)]
(t/is (= 0 (:exit result)))
(t/is (str/includes? (:out result) "test-value")))))
(t/deftest exec-with-input
(t/testing "stdin input is passed to the process"
(let [result (shell/exec! {}
:cmd ["cat"]
:in "hello from stdin"
:timeout 10)]
(t/is (= 0 (:exit result)))
(t/is (str/includes? (:out result) "hello from stdin")))))
(t/deftest exec-timeout-kills-process
(t/testing "process that exceeds timeout is killed and raises exception"
(let [start (System/currentTimeMillis)]
(try
(shell/exec! {}
:cmd ["sleep" "60"]
:timeout 1)
(t/is false "should have thrown")
(catch Exception e
(let [elapsed (- (System/currentTimeMillis) start)
data (ex-data e)]
;; Should complete quickly due to timeout, not wait 60s
(t/is (< elapsed 10000) "process should be killed within ~1 second")
(t/is (= :internal (:type data)))
(t/is (= :process-timeout (:code data)))
(t/is (= 1 (:timeout data)))))))))
(t/deftest exec-no-timeout-waits
(t/testing "without timeout, process runs to completion"
(let [result (shell/exec! {}
:cmd ["sleep" "0.1"]
:timeout nil)]
(t/is (= 0 (:exit result))))))

30
docker/images/Dockerfile.backend
View File

@ -5,7 +5,6 @@ ENV LANG='C.UTF-8' \
LC_ALL='C.UTF-8' \
JAVA_HOME="/opt/jdk" \
DEBIAN_FRONTEND=noninteractive \
NODE_VERSION=v24.16.0 \
TZ=Etc/UTC
RUN set -ex; \
@ -19,30 +18,6 @@ RUN set -ex; \
apt-get clean; \
rm -rf /var/lib/apt/lists/*
RUN set -eux; \
ARCH="$(dpkg --print-architecture)"; \
case "${ARCH}" in \
aarch64|arm64) \
OPENSSL_ARCH='linux-aarch64'; \
BINARY_URL="https://nodejs.org/dist/${NODE_VERSION}/node-${NODE_VERSION}-linux-arm64.tar.gz"; \
;; \
amd64|x86_64) \
OPENSSL_ARCH='linux-x86_64'; \
BINARY_URL="https://nodejs.org/dist/${NODE_VERSION}/node-${NODE_VERSION}-linux-x64.tar.gz"; \
;; \
*) \
echo "Unsupported arch: ${ARCH}"; \
exit 1; \
;; \
esac; \
curl -LfsSo /tmp/nodejs.tar.gz ${BINARY_URL}; \
mkdir -p /opt/node; \
cd /opt/node; \
tar -xf /tmp/nodejs.tar.gz --strip-components=1; \
chown -R root /opt/node; \
find /opt/node/include/node/openssl/archs -mindepth 1 -maxdepth 1 ! -name "$OPENSSL_ARCH" -exec rm -rf {} \; ; \
rm -rf /tmp/nodejs.tar.gz;
RUN set -eux; \
ARCH="$(dpkg --print-architecture)"; \
case "${ARCH}" in \
@ -78,7 +53,7 @@ LABEL maintainer="Penpot <docker@penpot.app>"
ENV LANG='C.UTF-8' \
LC_ALL='C.UTF-8' \
JAVA_HOME="/opt/jre" \
PATH=/opt/jre/bin:/opt/node/bin:/opt/imagick/bin:$PATH \
PATH=/opt/jre/bin:/opt/imagick/bin:$PATH \
DEBIAN_FRONTEND=noninteractive \
TZ=Etc/UTC
@ -125,9 +100,10 @@ RUN set -ex; \
chown -R penpot:penpot /opt/data;
COPY --from=build /opt/jre /opt/jre
COPY --from=build /opt/node /opt/node
COPY --from=penpotapp/imagemagick:7.1.2-13 /opt/imagick /opt/imagick
COPY files/imagemagick-policy.xml /opt/imagick/etc/ImageMagick-7/policy.xml
ARG BUNDLE_PATH="./bundle-backend/"
COPY --chown=penpot:penpot $BUNDLE_PATH /opt/penpot/backend/

17
docker/images/files/imagemagick-policy.xml Normal file
View File

@ -0,0 +1,17 @@
<policymap>
<policy domain="resource" name="memory" value="256MiB"/>
<policy domain="resource" name="map" value="512MiB"/>
<policy domain="resource" name="area" value="128MP"/>
<policy domain="resource" name="disk" value="1GiB"/>
<policy domain="resource" name="file" value="768"/>
<policy domain="resource" name="thread" value="4"/>
<policy domain="resource" name="time" value="30"/>
<policy domain="resource" name="width" value="16KP"/>
<policy domain="resource" name="height" value="16KP"/>
<policy domain="coder" rights="none" pattern="PS"/>
<policy domain="coder" rights="none" pattern="PS2"/>
<policy domain="coder" rights="none" pattern="PS3"/>
<policy domain="coder" rights="none" pattern="EPS"/>
<policy domain="coder" rights="none" pattern="PDF"/>
<policy domain="coder" rights="none" pattern="XPS"/>
</policymap>

36
docs/technical-guide/configuration.md
View File

@ -593,6 +593,42 @@ PENPOT_AUTO_FILE_SNAPSHOT_TIIMEOUT: "1h" # How often is an automatic save
Setting custom values for auto-file-snapshot does not change the behaviour for manual versions.
### ImageMagick Resource Limits
Penpot uses ImageMagick for image processing (thumbnail generation, MIME detection, dimension extraction).
You can configure resource limits for ImageMagick child processes to prevent a single image operation
from consuming unbounded server resources.
These environment variables override the default resource limits passed to ImageMagick via `MAGICK_*`
environment variables. They can make limits tighter than the Docker `policy.xml` but never looser.
```bash
# Backend
PENPOT_IMAGEMAGICK_THREAD_LIMIT: 2
PENPOT_IMAGEMAGICK_MEMORY_LIMIT: 256MiB
PENPOT_IMAGEMAGICK_MAP_LIMIT: 512MiB
PENPOT_IMAGEMAGICK_AREA_LIMIT: 128MP
PENPOT_IMAGEMAGICK_DISK_LIMIT: 1GiB
PENPOT_IMAGEMAGICK_TIME_LIMIT: 30
PENPOT_IMAGEMAGICK_WIDTH_LIMIT:
PENPOT_IMAGEMAGICK_HEIGHT_LIMIT:
```
| Variable | Default | Description |
|----------|---------|-------------|
| `PENPOT_IMAGEMAGICK_THREAD_LIMIT` | `2` | Max threads per ImageMagick process |
| `PENPOT_IMAGEMAGICK_MEMORY_LIMIT` | `256MiB` | Max heap memory per process |
| `PENPOT_IMAGEMAGICK_MAP_LIMIT` | `512MiB` | Max memory-mapped area (disk-backed pixel cache) |
| `PENPOT_IMAGEMAGICK_AREA_LIMIT` | `128MP` | Max total pixels (128 megapixels ≈ 11584×11096) |
| `PENPOT_IMAGEMAGICK_DISK_LIMIT` | `1GiB` | Max pixel cache on disk |
| `PENPOT_IMAGEMAGICK_TIME_LIMIT` | `30` | Max seconds per ImageMagick operation |
| `PENPOT_IMAGEMAGICK_WIDTH_LIMIT` | *(not set)* | Max width in pixels |
| `PENPOT_IMAGEMAGICK_HEIGHT_LIMIT` | *(not set)* | Max height in pixels |
The Docker image also includes a `policy.xml` that acts as a hard ceiling — these env vars
cannot exceed the limits set in `policy.xml`. The policy also blocks dangerous coders (PS,
EPS, PDF, XPS) that invoke Ghostscript.
## Frontend
In comparison with backend, frontend only has a small number of runtime configuration