* ✨ Add media-processor service for image and font processing
Externalizes ImageMagick and FontForge subprocess invocations into a
separate Node.js HTTP service (media-processor/). Backend dispatches
via feature flag :use-remote-media-processing.
Key changes:
- media-processor module (TypeScript, Express 5, Sharp, FontForge/woff)
- POST /api/image/info, /api/image/thumbnail, /api/font/generate
- Resource limits: 128MP rejection, prlimit (512MB + 30s CPU)
- Streaming multipart via SequenceInputStream
- app.media split into validation (leaf), local (shell impls), remote (HTTP)
- Schema enforcement: :upload and :input schemas in validation namespace
- Configurable timeout (PENPOT_MEDIA_PROCESSING_SERVICE_TIMEOUT)
- 78 tests across 4 files (image, font, middleware, config)
- FontForge path escaping for command injection prevention
- Parallel font variant conversions with Promise.all
AI-assisted-by: mimo-v2.5-pro
* 🐳 Revert docker-compose changes from media-processor commit
Remove docker-compose.yaml modifications that were part of the media-processor
service commit. The media-processor service definition, flags, and environment
variables are reverted to their previous state.
AI-assisted-by: qwen3.7-plus
* ⬆️ Update dependencies
* 🐛 Fix PR review issues in media-processor
- Font path bug: sfntToWoff and woff2ToSfnt now copy input to temp dir
when input is a file path, ensuring output lands in expected location
- Error preservation: execCommand preserves killed/signal/code properties
from child process errors for OOM detection
- Content-Length: service-multipart-request calculates and includes
Content-Length header for streaming multipart requests
AI-assisted-by: qwen3.7-plus
* 🐛 Fix code review issues in media-processor
- Rename PENPOT_MEDIA_PROCESSOR_SECRET_KEY to PENPOT_MEDIA_PROCESSOR_SHARED_KEY
in devenv to match backend config key
- Fix timeout middleware to destroy request AFTER response finishes,
preventing truncated 504 responses
- Fix quality=0 parsing to preserve explicit zero (was silently overridden to 85)
- Replace require('fs') with proper ES module import in upload-storage.ts
- Refactor font conversion temp-dir boilerplate into withTempInput helper
- Document FontForge escaping limitations (single quotes only)
- Fix misleading comment in image.ts about sharp metadata decoding
AI-assisted-by: qwen3.7-plus
* 🐛 Fix code review issues in media-processor (round 2)
- Fix queue middleware to skip next() when response already ended,
preventing orphaned work after timeout
- Fix hybrid storage to use disk when Content-Length is absent (chunked
transfer), preventing unbounded memory allocation
- Add source image format validation in generateThumbnail to reject
unsupported formats (TIFF, BMP, etc.) with 400 instead of 500
- Remove dead code in convertFont for unreachable woff→woff path
- Remove unused isEnabled() method from LokiLogTransport
- Fix sfntToWoff to use correct extension (.ttf/.otf) based on source type
- Extract queue middleware to separate file for testability
- Add comprehensive tests for queue middleware and upload storage
AI-assisted-by: qwen3.7-plus
* 🐛 Fix code review issues in media-processor (round 3)
- Fix disk-backed upload cleanup after successful requests by adding
cleanup middleware that removes temp files on response finish/close
- Wrap sharp metadata/decoding errors as 400 validation errors instead
of 500 internal errors
- Only apply flatten() for JPEG output to preserve alpha channel in
PNG and WebP outputs
AI-assisted-by: qwen3.7-plus
* ✨ Add comprehensive tests for media-processor
Phase 1 - Cleanup verification:
- Add cleanup middleware unit tests (6 tests)
- Add HTTP upload cleanup integration tests (5 tests)
Phase 2 - Error handling & alpha preservation:
- Add sharp error wrapping tests (4 tests)
- Add HTTP malformed image tests (2 tests)
- Add alpha preservation tests (3 tests)
Phase 3 - Edge cases:
- Add upload storage edge case tests (3 tests)
- Add queue middleware edge case tests (4 tests)
Phase 4 - Backend mock verification:
- Fix backend mocks to include :mtype field in image info responses
- Verify all error codes match actual service behavior
Total: 27 new tests added (160 tests passing)
AI-assisted-by: qwen3.7-plus
* 🐛 Fix code review issues in media-processor (round 4)
- Add Zod validation constraints for config values (int, positive, min)
- Fix auth middleware to compare Buffer byte lengths instead of string lengths
- Validate requested output dimensions in generateThumbnail (crop mode)
- Change queue middleware to release slot via callback in finally block
- Add comprehensive tests for all fixes
AI-assisted-by: qwen3.7-plus
* 🐛 Close HTTP response streams in backend media remote
- Wrap stream consumption in try/finally with .close() calls
- Add tests to verify stream closure for info, font-convert, and thumbnail
AI-assisted-by: qwen3.7-plus
* 🐛 Fix queue slot leak on upload failures
Make releaseQueue idempotent and attach fallback listener to release
slot when response finishes. This covers Multer errors that bypass
the route handler's finally block, preventing permanent queue stall.
AI-assisted-by: qwen3.7-plus
* 🐛 Cancel processing on timeout
Create AbortController in timeout middleware and abort signal when
timeout fires. Pass signal to Sharp and FontForge to cancel ongoing
processing and release resources when request is cancelled.
AI-assisted-by: qwen3.7-plus
* 🐛 Fix code review issues in media-processor (round 6)
- Error handler: check headersSent before writing response to prevent
ERR_HTTP_HEADERS_SENT when timeout already sent 504
- Timeout config: increase default requestTimeout from 60s to 180s to
match font processing timeout (120s) and backend request timeout
- Image processing: check abort signal before starting Sharp operations
to cancel processing when timeout fires
- Queue lifecycle: remove res.on('close', release) fallback to hold
queue slot until processing completes, preventing concurrency limit
violation when client disconnects
AI-assisted-by: qwen3.7-plus
* 🐛 Close HTTP response stream in download-image
Wrap response body in with-open to ensure stream is closed after
writing to temp file, preventing HTTP connection leaks on repeated
URL imports.
AI-assisted-by: qwen3.7-plus
* 🐛 Close HTTP response stream on validation errors in download-image
Move with-open to wrap the entire validation and processing block,
ensuring the response body stream is closed even when validation fails
(non-2xx status, missing size, invalid media type). This prevents
HTTP connection leaks on repeated failed downloads.
Add test to verify stream closure on validation errors.
AI-assisted-by: qwen3.7-plus
* 🐛 Pass abort signal to Sharp toBuffer for timeout cancellation
Wrap Sharp's toBuffer() with Promise.race to check abort signal during
processing. This ensures large thumbnails stop processing when the
request times out, preventing wasted CPU/memory and queue capacity.
Add test to verify abort during toBuffer operation.
AI-assisted-by: qwen3.7-plus
* 🐛 Hold queue slot until Sharp completes and handle client disconnect
- Remove Promise.race from generateThumbnail — Sharp processing now
completes fully before queue slot is released, preventing concurrency
limit violations under timeout conditions
- Remove res.on("finish", release) fallback from queue middleware —
error handler now explicitly calls releaseQueue in all error paths
- Add res.on("close") handler in timeout middleware to abort signal
when client disconnects, ensuring processing stops early
- Add tests for client disconnect handling and queue slot lifecycle
AI-assisted-by: qwen3.7-plus
* 🐛 Address round 9 review findings
- Document Sharp 0.35.3 cancellation limitation in image.ts
- Add integration test for timeout cleanup with large images
- Fix font tools (sfntToWoff, woffToSfnt, woff2ToSfnt) to throw
ProcessingError on resource limit kills instead of returning null
- Validate font signatures for same-format conversions to prevent
arbitrary files from being persisted as valid fonts
- Fix concurrent mkdtemp race in upload-storage by using shared
initialization promise
AI-assisted-by: qwen3.7-plus
* 🐛 Address round 10 review findings
- Add tmpdir assertion in font.ts to prevent path injection
- Preserve original error in queue middleware catch handler
- Change auth middleware response type from "internal" to "authorization"
- Add cleanup flag to prevent double cleanup in cleanup middleware
- Move quality clamping into parseQuality function for consistency
- Add integration tests for quality parameter clamping at route level
- Update existing tests to match new auth response type
AI-assisted-by: qwen3.7-plus
* 🐛 Address round 11 review findings
- Extract releaseSlot helper in error-handler to reduce duplication
- Remove redundant try/catch in font.ts withTempDir cleanup
- Improve font path validation error message for clarity
- Move path validation before try/catch to prevent swallowing
- Add debug logging for cleanup failures in cleanup middleware
- Inline TransportTargetSpec type alias in logger.ts
- Extract logging middleware to separate file for consistency
- Remove duplicate MIME validation in image thumbnail route
- Add test for font path validation (outside tmpdir rejection)
- Add tests for error handler queue release across all branches
AI-assisted-by: qwen3.7-plus
* 🐛 Remove Content-Length header from multipart requests
The JDK's HttpClient rejects Content-Length as a restricted header,
causing IllegalArgumentException when sending multipart requests to the
media-processor. Remove the explicit Content-Length header and let the
JDK use chunked transfer encoding. The media-processor will use disk
storage for all multipart requests (safe default behavior).
Remove unused size computations (file-size, header-bytes, footer-bytes,
total-size) that were only used for Content-Length.
Update test to verify Content-Length is not present in request headers.
AI-assisted-by: qwen3.7-plus
* 🐛 Fix pino ESM bundling for media-processor
Mark pino and its transports (pino-pretty, pino-loki) as external to
avoid bundling issues with worker thread modules that reference
__dirname (not available in ES modules).
AI-assisted-by: qwen3.7-plus
Move all development tools from tools/ to scripts/ for consistency.
Rename lint/fmt/check-fmt to lint-clj/fmt-clj/check-fmt-clj to clarify
they target Clojure specifically. Remove unused scripts (attach-opencode,
start-opencode, start-opencode-server) and the backport-commit skill.
Update all internal references across .serena/, AGENTS.md, and
CONTRIBUTING.md to point to the new script locations. Simplify
CONTRIBUTING.md by delegating module-specific fmt/lint instructions
to the respective serena memories.
AI-assisted-by: deepseek-v4-flash
* 📎 Ignore .iml files (IntelliJ module files)
* 🎉 Enable multi-instance horizontal scaling for MCP server
Allow the MCP server to run as multiple instances behind a plain
round-robin load balancer, removing the previous requirement that a
user's plugin WebSocket and MCP client connection terminate on the same
instance. Behaviour is unchanged when run as a single instance or
without Redis.
Cross-instance MCP sessions: when a request arrives with an
mcp-session-id that was initialised on another instance, the session is
adopted locally instead of rejected. The user token is read from the
query parameter (present on every request, as the configured endpoint
URL is never rewritten), so no shared session store is needed; the
transport is pre-initialised so the SDK's validateSession() accepts it.
Cross-instance task routing: when a Redis URI is configured in
multi-user mode, plugin task requests are routed via Redis pub/sub keyed
by user token. The instance holding a plugin's WebSocket subscribes to
that token's request channel; any instance handling a tool call
publishes the request and awaits the response on a per-request channel.
RedisBridge is a pure transport for the existing serialised
PluginTaskRequest/Response objects. PluginTask is split into an abstract
base plus a local (promise-backed) PluginTask and a RemotePluginTask
whose resolve/reject publish the outcome back over Redis, so the
existing local dispatch and response-correlation paths are reused
unchanged on the executing instance.
Refs #10000
* 🐳 Split devenv compose for parallel workspaces
Move shared services into an infra compose file and keep the main devenv container plus Valkey in a separate compose file driven by defaults.env. Parameterize host-side ports, container names, source path, and runtime env while keeping container-internal ports fixed for same-origin proxying.
Make tmux startup idempotent, add attach-devenv for the live instance, move shared MinIO user setup to infra startup, and let exporter scripts load backend _env.local overrides.
Co-authored-by: Codex <codex@openai.com>
* 🐳 Run parallel devenv instances against shared infra
Add support for running N parallel devenv instances under separate compose
projects sharing Postgres, MinIO, mailer, and LDAP. Each instance has its
own main container, Valkey, source checkout, tmux session, and host port
range offset by 10000 (3449 -> 13449 -> 23449, etc.).
./manage.sh run-devenv-agentic --n-instances N reconciles the running set
to exactly {ws0..ws(N-1)}: missing instances are created (workspace sync
from the live repo via git ls-files + per-instance env-file generation
under docker/devenv/instances/ + detached tmux startup), surplus instances
are stopped highest-first via compose down (never -v), already-running
instances are left untouched. ws0 binds the live repo at PWD; ws1+ are
scratch clones under ~/.penpot/penpot_workspaces/.
Backend workers (enable-backend-worker) are gated on PENPOT_BACKEND_WORKER
in backend/scripts/_env; ws1+ overlays disable them so async-task
notifications stay bound to a single Valkey Pub/Sub instance.
Compose helpers wrap docker compose with env -i so per-instance overlay
--env-file actually overrides defaults.env -- without the strip, the shell
env from sourcing defaults.env at startup would shadow the overlay (Compose
gives shell precedence over --env-file).
Other:
- Drop network aliases (- main, - redis); use container_name for
cross-container DNS so multiple instances on the shared network don't
fight over the same DNS name.
- Pin volume names via name: (PENPOT_*_VOLUME) so volumes survive project
renames; ws0 keeps the pre-existing physical names (penpotdev_*).
- Remove cross-project depends_on from main.yml (postgres/minio-setup now
live in penpotdev-infra); manage.sh ensure-infra-up docker-waits on the
minio-setup one-shot.
- Strict arg parsing in run-devenv / run-devenv-agentic; --n-instances 0
rejected.
- Remove unused Host-matched server block from the Caddyfile.
Memory mem:devenv/core and developer docs updated.
Co-authored-by: Codex <codex@openai.com>
* ✨ Document and stabilise the parallel-workspace CLI; wire AI agents
Improve parallel-workspaces developer CLI,
and add an opt-in layer that lets four AI
coding agents (Claude Code, opencode, VS Code Copilot, OpenAI Codex CLI)
drive a specific workspace through a single launcher command.
Parallel-workspace semantics
----------------------------
each run-devenv-agentic call brings up one wsN;
--ws N (integer; default 0) targets a specific workspace and auto-starts
ws0 first when N>=1 so the worker invariant holds. --sync is forbidden on
ws0 and re-seeds the workspace from the live repo for ws1+. Stop semantics
mirror the start invariant -- ws0 is the last to stop, shared infra stops
with it, --all walks every instance highest-first. The worker policy
section explains why workers run only on ws0 (Postgres FOR UPDATE
SKIP LOCKED is safe across many workers but the cron dedup primitive is
best-effort, and :telemetry / :audit-log-archive are not idempotent).
Per-instance Valkey Pub/Sub isolation, msgbus topology, and the
"async task notifications miss ws1+ tabs" caveat are stated explicitly.
The mem:prod-infra/core memory captures the same external-services and
task-queue / Pub-Sub topology in agent-readable form, and
mem:backend/core and mem:critical-info now cross-link it so backend work
surfaces the horizontal-scaling constraints from the start.
AI coding agent integration
---------------------------
New top-level .devenv/ directory holds committed templates
(templates/{claude-code,opencode,vscode}.json and templates/codex.toml,
each with \${PENPOT_MCP_PORT} and \${SERENA_MCP_PORT} placeholders) plus
committed shared entries (matching shared/* files for Playwright, the
only workspace-independent server we ship today).
./manage.sh start-coding-agent <claude|opencode|vscode|codex> [--ws N]
launches the chosen client against one workspace. It cd's into the
target's directory (the live repo for ws0; workspace-path "wsN" for ws1+)
and refuses to launch unless (a) the binary is on PATH, (b) the
workspace directory exists for ws1+, and (c) the instance is up
(devenv-main-running) -- the MCP servers only exist while the devenv is
running. The agentic-devenv guide is restructured around this Quick
start path, with a per-client table and a Manual configuration fallback
for clients we don't cover.
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* ♻️ Scope the shadow devtools to the dev build
---------
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Exclude files like CONTRIBUTING.md or README.md from being ignored by /*.md pattern,
as this can influence agent behaviour (configurations that disallow ignored files
from being edited)
Update agent configurations: change commiter mode to all, rename
engineer agent to "Penpot Engineer", and remove obsolete testing agent.
Add new read-only planner agent for architecture analysis and planning.
Add four new skills: bat-cat (syntax-highlighted cat clone), fd-find
(fast file finder), jq-json-processor (JSON processor), and ripgrep
(fast text search).
Add fd-find and bat packages to devenv Dockerfile.
Update .gitignore to exclude opencode package-lock and plans directory.
Signed-off-by: Andrey Antukh <niwi@niwi.nz>
