* 🐳 Add ImageMagick policy.xml resource limits to backend Docker image
Add a restrictive policy.xml to the backend Docker image that caps
ImageMagick resource usage: 256MiB memory, 512MiB map, 128MP area,
30s time limit, 16KP max dimensions. Blocks PS/EPS/PDF/XPS coders
to prevent Ghostscript attack surface.
Co-authored-by: mimo-v2.5-pro <mimo-v2.5-pro@penpot.app>
* ✨ Add timeout support to shell/exec!
Add optional :timeout parameter (in seconds) that uses
Process.waitFor(long, TimeUnit). On timeout, the process is
destroyed forcibly and an :internal/:process-timeout exception
is raised. Stdout/stderr readers handle IOException from closed
streams when the process is killed.
Co-authored-by: mimo-v2.5-pro <mimo-v2.5-pro@penpot.app>
* ♻️ Rename ::wrk/netty-executor to ::wrk/executor with cached pool
Replace DefaultEventExecutorGroup (fixed Netty thread pool) with a
cached thread pool (px/cached-executor) for general async task
offloading. The cached pool creates threads on demand and reuses
idle ones, which is more appropriate for blocking I/O workloads
(shell commands, message bus, rate limiting, etc.).
Changes:
- Rename ::wrk/netty-executor to ::wrk/executor in worker/executor.clj
- Switch implementation from DefaultEventExecutorGroup to px/cached-executor
- Update all ig/ref wiring in main.clj (msgbus, tmp cleaner, climit, rlimit, rpc)
- Remove ::wrk/netty-executor from redis.clj (let lettuce create its own
eventExecutorGroup instead of sharing a Netty executor)
- Assert executor is present in shell/exec! to prevent silent nil usage
- Remove executor-threads config (no longer needed for cached pool)
The ::wrk/netty-io-executor (NioEventLoopGroup) remains unchanged as it
handles actual non-blocking network I/O for Redis and S3.
Co-authored-by: mimo-v2.5-pro <mimo-v2.5-pro@penpot.app>
* 🔥 Remove im4java dependency and replace with direct ImageMagick CLI calls
- Replace im4java Java library with direct 'magick' CLI calls via shell/exec!
- Add PENPOT_IMAGEMAGICK_* config env vars for resource limits (thread, memory, map, area, disk, time, width, height)
- Use configurable ImageMagick environment with sensible defaults matching policy.xml
- Remove -Dim4java.useV7=true JVM flag from startup scripts
- Remove org.im4java/im4java from deps.edn
- All ImageMagick commands now use shell/exec! with 60s timeout and resource limits
Co-authored-by: mimo-v2.5-pro <mimo-v2.5-pro@penpot.app>
* 💄 Rename imagemagick env functions and optimize config reads
- Rename imagemagick-defaults -> imagemagick-default-env
- Rename imagemagick-env -> get-imagemagick-env
- Optimize to avoid double cf/get calls per config key
Co-authored-by: mimo-v2.5-pro <mimo-v2.5-pro@penpot.app>
* ✨ Add tests for shell/exec! timeout and media processing
- Add shell_test.clj: tests for exec! timeout, env vars, stdin, stderr
- Add media_test.clj: tests for info, generic-thumbnail, profile-thumbnail
- Fix generic-process to prefer explicit format over input mtype
- Fix shell/exec! to use cached executor when system has no executor
- Fix reduce-kv accumulator in set-env (must return penv)
Co-authored-by: mimo-v2.5-pro <mimo-v2.5-pro@penpot.app>
* ♻️ Refactor media/process to take system as first argument
- Change (defmulti process :cmd) -> (defmulti process (fn [_system params] (:cmd params)))
- Change (run params) -> (run system params)
- All process methods now receive [system params]
- Update all callers: rpc/commands/media, profile, auth, fonts
- Revert shell/exec! to require system with executor (no fallback)
- Fix lint warnings and formatting
Co-authored-by: mimo-v2.5-pro <mimo-v2.5-pro@penpot.app>
* 🔥 Remove unused app.svgo namespace
Co-authored-by: mimo-v2.5-pro <mimo-v2.5-pro@penpot.app>
* 🔥 Remove Node.js from backend Docker image
- Delete unused svgo-cli.js script
- Remove Node.js installation from Dockerfile.backend
- Remove svgo-cli.js copy from backend build script
Co-authored-by: mimo-v2.5-pro <mimo-v2.5-pro@penpot.app>
* 🔥 Remove unused process-error multimethod
- Remove process-error multimethod and its default handler
- Simplify media/run to directly call process
- Fix alignment in main.clj
Co-authored-by: mimo-v2.5-pro <mimo-v2.5-pro@penpot.app>
* 📚 Add ImageMagick resource limits configuration to technical guide
Co-authored-by: mimo-v2.5-pro <mimo-v2.5-pro@penpot.app>
---------
Co-authored-by: mimo-v2.5-pro <mimo-v2.5-pro@penpot.app>
The main idea behind this refactor is make the
API less especialized for specific use of out internal
submidules and make it more general and usable
for more general purposes (per example cache)
Replace general usage of virtual threads with platform threads
and use virtual threads for lightweight procs such that websocket
connections. This decision is made mainly because virtual threads
does not appear on thread dumps in an easy way so debugging issues
becomes very difficult.
The threads requirement of penpot for serving http requests
is not very big so having so this decision does not really affects
the resource usage.
This upgrade also includes complete elimination of use spec
from the backend codebase, completing the long running migration
to fully use malli for validation and decoding.
Mainly the followin changes:
- Pass majority of code to the old and plain synchronous style
and start using virtual threads for the RPC (and partially some
HTTP server middlewares).
- Make some improvements on how CLIMIT is handled, simplifying code
- Improve considerably performance reducing the reflection and
unnecesary funcion calls on the whole stack-trace of an RPC call.
- Improve efficiency reducing considerably the total threads number.
- makes the profile access more efficient (replace in-app joins to a
simple select query on profile table
- add partial support for access-tokens (still missing some RPC methods)
- move router definitions to specific modules and simplify the main http
module definitions to simple includes
- simplifiy authentication code related to access-tokens and sessions
- normalize db parameters with proper namespaced props
- more work on convert all modules initialization to use proper specs
with fully-qualified keyword config props
As a replacement for the current pubsub approach.
It now uses a single connection for multiple
subscriptions (instead of conn per subscription);
has asynchronous publish and uses more efficient
blob encoding for message encoding (the same used
as page storage).
